# Multimodal Web Navigation with Instruction-Finetuned Foundation Models

Hiroki Furuta1,2 Ofir Nachum2 Kuang-Huei Lee2 Yutaka Matsuo1

Shixiang Shane Gu2,1 Izzeddin Gur2

1The University of Tokyo 2Google Research, Brain Team
<EMAIL_ADDRESS>
Work done as Student Researcher at Google.

###### Abstract

The progress of _autonomous web navigation_ has been hindered by the
dependence on billions of exploratory interactions via online reinforcement
learning, and domain-specific model designs that make it difficult to leverage
generalization from rich out-of-domain data. In this work, we study data-
driven offline training for web agents with vision-language foundation models.
We propose an instruction-following multimodal agent, WebGUM, that observes
both webpage screenshots and HTML pages and outputs web navigation actions,
such as _click_ and _type_. WebGUM is trained by jointly finetuning an
instruction-finetuned language model and a vision transformer on a large
corpus of demonstrations. We empirically demonstrate this recipe improves the
agent’s ability of grounded visual perception, HTML comprehension and multi-
step reasoning, outperforming prior works by a significant margin. On the
MiniWoB benchmark, we improve over the previous best offline methods by more
than 31.9%, being close to reaching online-finetuned SoTA. On the WebShop
benchmark, our 3-billion-parameter model achieves superior performance to the
existing SoTA, PaLM-540B. We also collect 347K high-quality demonstrations
using our trained models, 38 times larger than prior work, and make them
available to promote future research in this direction.

##  1 Introduction

Web navigation is a sequential decision making problem where agents use
computers or crawl browsers following user instructions [70, 55, 29]. Common
web navigation tasks include, for example, form filling [21], information
retrieval [61, 1], or sending emails via a sequence of interactions with
computer interface such as click or type (Figure 1). Recently, there has been
a growing interest in developing autonomous web agents to automate these
actions and release humans from repetitive interactions with computer
interfaces [58, 53, 73, 36, 86].

Prior works have studied web navigation problems as online RL to learn the
optimal action distribution with task-specific models from scratch [55, 29,
40, 36]. However, online RL requires massive trials-and-errors and is often
infeasible in practice since the failure in web navigation would result in
undesirable consequences; for instance, wrong password may lead to account
freeze, and sending email to the wrong person could be problematic in a
business scene. Offline training from the static dataset is a promising
alternative for safe development of web agents, but the performance has been
sub-optimal compared to those online RL counterparts [36, 31]. Another issue
with many of the prior works has been unable to generalize from rich out-of-
domain data, as they usually use specialized models to explicitly handle the
hierarchical structures of document object model (DOM) and their dependencies,
for example, with LSTM [29, 30], self-attention [55], or GNN [40].
Furthermore, many of them only output a fixed set of categorical actions [36],
which is unfavorable for truly open-ended web navigation in the real world.

Recently, foundation models [8], especially large language models (LLM) [10,
16], have demonstrated superior performance in commonsense, symbolic,
arithmetic, and multi-step logical reasoning [83, 84, 44]. Many prior works
have shown that these models enable transformative generalization and are
capable of solving wide ranges of interactive decision making problems in the
wild, including but not limited to task planning in robotics [34, 35, 69, 4],
board game [59], web-based retrieval and browser crawling [60, 87, 88].

Figure 1:  Example episode on MiniWoB++ [70, 55] (email-inbox-forward-nl). The
agent clicks the email from the proper sender, and types the correct receiver
to forward that email, to satisfy the given instruction (e.g. Find Gisele’s
email and forward it to Siana, please). WebGUM makes use of both HTML and
image screenshot information to adapt a pre-trained instruction-finetuned
foundation model to solve challenging web-based tasks.

In this work, we leverage pre-trained foundation models and introduce a
competitive offline learning recipe for autonomous web agents: First, we
hypothesize that grounded spatial understanding is important for web
navigation [36, 77] and thus enables our agent to observe both HTML and
screenshots by combining a language model and a ViT [22], from semantically
rich multimodal tokens. Second, we observe that web navigation tasks are by
nature instruction-following and thus base the language model on an
instruction-tuned LLM [82, 17, 62, 37] instead of self-supervisedly pre-
trained LLMs [66, 10] as in Gur et al. [31]. Third, we collect a large
multimodal corpus, with both HTML and screenshots, to finetune the language
model and ViT jointly. Fourth, our model outputs action in free-form text.
These four key pieces together give us a multimodal web agent, which we call
_Web navigation via Grounded Understanding Models_ or WebGUM in short. As
shown in Figure 1, our model takes in a command for a web-based task via a
natural language instruction (e.g., in an email client, Find Gisele’s email
and forward it to Siana, please.) and uses multimodal observations of the
computer interface to complete the task via a sequence of computer actions.

Through evaluation on MiniWoB++ [70, 55], a representative web navigation
benchmark with simulated websites, our multimodal model outperforms previous
best offline approaches trained with HTML inputs [31] by 31.9%. Moreover,
WebGUM closes the performance gap to the best existing online RL approaches
[36], despite being trained fully offline with much fewer experiences. We
perform extensive ablations and analysis to demonstrate WebGUM’s advantages in
(1) multimodal vision-and-HTML observations, (2) better HTML understanding,
(3) ability of multi-step reasoning, and (4) dataset and model size scaling.
WebGUM could leverage multimodal tokens to ground vision and HTML
understanding on the computer interface, especially to solve multi-step tasks
with dynamic page transitions or tasks that require global contexts, such as
booking flights, crawling social media, or shape recognition. Using
instruction-finetuned language models [17], compared to using the self-
supervised pre-trained counterpart [66], improves the success rate on
MiniWoB++ by over 25%, and is especially adept at handling the unknown
composition of the tasks or out-of-distribution HTML inputs synthesized with
realistic perturbations. On the WebShop benchmark [86], we demonstrate that
the capability of multi-step reasoning in language models surprisingly
enhances downstream decision making by achieving superior performance even to
the existing state-of-the-art few-shot PaLM-540B [87, 16], while our model
only has 3 billion parameters. Finally, our results confirm the importance of
both model and dataset scaling in multimodal web navigation; we collect 347K
multimodal expert demonstrations on MiniWoB++, 38 times larger than the
existing unimodal dataset [55], and make these publicly available for future
research 111https://github.com/google-research/google-
research/tree/master/mm_webnav.

Methods | Architecture | Pre-trained | Input | Output | Offline
---|---|---|---|---|---
WGE [55] | LSTM, self-attention | ✗${}^{~{}}$ | DOM | Logit of action | ✗
CoDE [29, 30] | Bi-LSTM | ✗${}^{~{}}$ | DOM | Logit of action | ✗
DOM-Q-NET[40] | GNN | ✗${}^{~{}}$ | DOM | Logit of action | ✗
CC-Net [36] | LSTM, Transformer, ResNet | ✗∗ | DOM, Screenshot | Logit of action | ✗
WebShop [86] | BERT, BART | ✔${}^{~{}}$ | Text (from HTML) | Logit of action |  ✗ / ✔
WebGUM (Ours) | T5 Transformer, ViT | ✔${}^{~{}}$ | HTML, Screenshot | Text | ✔

Table 1: Prior works have studied web navigation problem as online RL to learn
the optimal action distribution with task-specific model architectures from
scratch (∗or partially using pre-trained vision encoder). We omit the web-
specialized architecture and input-output space, and convert web navigation
into visual question-answering format (text, image $\rightarrow$ text), which
allows us to learn the agents offline by leveraging pre-trained foundation
models [66, 17, 22] in vision or language domains as strong inductive bias for
web environments.

##  2 Related Work

Web Navigation Among many proposed benchmarks for autonomous web navigation
[77, 11, 86], one of the most inclusive and representative benchmark to test
the capability of autonomous agents is MiniWoB++ [70, 55], which consists of a
set of simulated websites with various user instructions from primitive tasks
to complex multi-step decision making tasks, such as sending emails or booking
flights. Prior works have tried to solve this benchmark using a variety of
techniques; Liu et al. [55] and Gur et al. [29, 30] leverage the guidance
during online RL from high-level workflow [55] or curriculum learning [29,
30], which should be, however, designed per task, and then would not be
scalable methods. Other approaches have employed supervised learning (SL) with
a large million-scale dataset and following RL-finetuning [36], or SL with
LLM-based agents [31]. Offline SL agents often suffer from sub-optimal
behavior, and online RL with tremendous exploratory experiences has been
critical for proficient navigation on the web [36, 31], which is, however,
difficult to conduct in real websites as there is typically no reward signal
and interactions are prohibitively costly. As shown in Table 1, many of these
approaches depend on task-specific hierarchical structures of DOM [40, 33],
tailored architectures to encode their dependencies such as LSTM [29, 30],
self-attention [55], or GNN [40], and task-dependent categorical output space
[36], which could not handle open-ended multi-task settings similar to real
world, or incorporate pre-trained models. In contrast, we remove such web-
specific architectures and convert web navigation into visual question-
answering format (text, image $\rightarrow$ text), which allows us to leverage
pre-trained foundation models [17, 22] as rich prior knowledge on the web, and
then to learn the capable agents even with offline training.

Multimodal Large-scale Models Large language models have demonstrated
extraordinary emergent abilities on a variety of NLP tasks, such as
commonsense question answering, arithmetic, logical reasoning, open-ended text
generation [63, 10, 16, 83, 76], or code completion [13, 7, 54]. In addition,
some works have investigated vision-and-language understanding to improve the
accuracy of common vision-based tasks such as open-ended image/object
classification [64, 28, 42], image captioning, or visual question-answering
[56, 5, 15, 67]. Several works also have tackled document understanding with
(multimodal) transformer models [85, 49, 51, 6, 75, 80, 81], including markup
languages such as HTML [2, 3, 50, 46] for summarization of the documents or
question answering on the contents. Despite the great efforts on document
understanding, these works are less connected to interactive decision making
problems. Our model obtains not only a grounded understanding of websites in a
multimodal manner but also the ability to decide the optimal actions to
achieve given instructions in web navigation, helping multi-step decisions and
global context perception.

Foundation Models for Decision Making Recently, the ability of multi-step
reasoning and inductive bias in foundation models have been leveraged to solve
text-based interactive tasks via sequential decisions considering few-shot in-
context examples [4, 34, 35, 89, 87, 59]. Even in continuous control [12, 38,
25, 9] or computer games [67, 47, 23], high-capacity transformer models are
trained with a large amount of diverse dataset via multi-task behavioral
distillation [14, 27, 18, 24, 72, 41]. To build autonomous web navigation
agents, we also leverage pre-trained LLM [66, 17], by finetuning with
massively-curated multimodal demonstrations, and we point out that the better
content comprehension and multi-step reasoning abilities, obtained through
instruction-finetuning of LLM [17], are essential for the notable performance
on downstream decision making aligned with human instructions.

Figure 2:  Overview of WebGUM, our multimodal encoder-decoder transformer
model. It takes recent $H$-step screenshots ($H=2$), action history,
instruction, and HTML as inputs. Image observations are embedded to tokens per
$16\times 16$-size patch via pre-trained vision transformer (ViT) [22].
Multimodal language-image tokens are fed into pre-trained T5 encoder-decoder
transformer [66], and then predict executable actions in text formats.

##  3 Preliminaries

We formulate autonomous web navigation as a deterministic sequential decision
making problem; composed of a state space $\mathcal{S}$, action space
$\mathcal{A}$, deterministic transition function
$T:\mathcal{S}\times\mathcal{A}\xrightarrow{}\mathcal{S}$, instruction space
$\mathcal{G}$, reward function (or episodic success criteria)
$r:\mathcal{S}\times\mathcal{G}\times\mathcal{A}\xrightarrow{}\\{0,1\\}$. At
each time step $t$, the agent follows a parameterized policy conditioned on
previous states and actions
$\pi:\underbrace{\mathcal{S}\times\cdots\times\mathcal{S}}_{\times
t}\times\underbrace{\mathcal{A}\times\cdots\times\mathcal{A}}_{\times
t}\times\mathcal{G}\rightarrow{}\mathcal{A}$, and transits to the next state:
$s_{t+1}=T(s_{t},a_{t})$. This process continues until the agent reaches the
terminal state (e.g. Submit button is clicked) or the max time step is
exceeded. The episode is treated as a success if given instruction $g$ is
satisfied (i.e. $r(s_{t},g,a_{t})=1$), and as a failure if the agent takes a
invalid action or reaches a wrong terminal state.

In autonomous web navigation, the state $s_{t}\in\mathcal{S}$ is a web page
consisting of the raw HTML as a text sequence and a screenshot as an image.
Following prior works [70, 55, 29, 30], we assume the constraint action space:
function(selector, text). function is either click or type, selector is an
integer index that can uniquely specify the element, and text is a text input
for type function.

Figure 1 presents an example episode of MiniWoB++ [70, 55], which involves
multi-step decision making. To meet the given instruction, the agent clicks an
email from the proper sender and types the correct receiver to forward that
email. On the other hand, MiniWoB++ also has primitive behavioral tasks such
as clicking buttons or entering texts. See Appendix H for the examples of
WebShop [86].

##  4 WebGUM

###  4.1 Multimodal Transformer Models

In this work, we follow Gur et al. [31] to use an encoder-decoder architecture
for HTML-based web navigation, as its bi-directional nature could be a good
fit for the tree structure of HTML and the architecture has been shown to
scale well [66]. We extend T5, an encoder-decoder transformer-based language
model [78] proposed by Raffel et al. [66] for multimodality. As shown in
Figure 2, the encoder considers a set of ViT-encoded visual tokens from a
history of screenshots ($H=2$ steps) and a set of text tokens from action
history, user instruction, and raw HTML. The encoder transformer handles both
visual and text tokens in a unified manner, and decoder predicts actions in
text. See Appendix B for further implementation details.

Image Encoder for Visual Tokens To map image observations (screenshots) into
visual tokens for T5 encoder, we use a vision transformer (ViT) [22], pre-
trained on ImageNet-21K classification [19]. To better extract spatial and
semantic information from the screenshots of websites, T5 encoder considers
one token per patch rather than per image (i.e. CLS-token). We divide an input
image into $16\times 16$ patches – giving a total of $14\times
14~{}(\text{number of patches})\times 2~{}(\text{context window})=392$ visual
tokens. We crop the screenshots of MiniWoB++ to remove the yellow instruction
part (as shown in Figure 1), and the image size becomes 160 $\times$ 160\. We
pad cropped images with white pixels to fit them into 224 $\times$ 224; the
default input size for ViT.

Methods | Modality | Pre-trained Models | Offline | Dataset | Success Rate
---|---|---|---|---|---
CC-Net (SL) [36] | DOM+Image | ResNet | ✔ | 2.4M | 32.0%
WebN-T5 [31] | HTML | T5-XL | ✔ | 12K | 48.4%
MM-WebN-T5 | HTML+Image | T5-XL,ViT-B16 | ✔ | 347K | 55.6%
WebGUM (Ours) | HTML+Image | Flan-T5-Base,ViT-B16 | ✔ | 2.8K | 61.1%
HTML | Flan-T5-XL | ✔ | 347K | 75.5%
HTML+Image | Flan-T5-XL,ViT-B16 | ✔ | 347K |  80.3%
WGE [55] | DOM | – | ✗ | 12K+ | 64.6%
CC-Net (SL+RL) [36] | DOM+Image | ResNet | ✗ | 2.4M+ |  96.4%
Table 2:  Average success rate on MiniWoB++ among 56 tasks. We recalculate the
baseline performances referring prior works [36, 31]. See Appendix E for the
detailed scores per task. WebGUM significantly outperforms the previous
finetuned-LLM with 3B parameters [31], which is the recent best among offline
methods, or its multimodal variant (MM-WebN-T5) even with 2.8K dataset and
Base-size model (310M parameters). Scaling dataset and model size, WebGUM is
close to reaching the online RL-finetuned state-of-the-art [36], despite fully
offline training. “+” in Dataset column means extra billions of frames are
required during the online RL phase.

###  4.2 Instruction-Finetuned Large Language Models

We base our language model on an instruction-finetuned T5, Flan-T5 [17],
instead of using a vanilla pre-trained T5 as in Gur et al. [31]. Flan-T5 is
finetuned with large-scale instruction-following format problems and few/zero-
shot chain-of-thought examples across a variety of domains, including
reasoning or programming. Considering that web navigation is inherently an
instruction-following task, we hypothesize that carefully trained instruction-
finetuned models could generalize well to enhance the alignment with user
instruction and zero-shot reasoning in the web-navigation, interactive
decision making context. For the same reason, we also hypothesize that these
high-performing instruction-finetuned models enable better sample efficiency
and downstream performance, and thus are well-suited for offline learning. We
further finetune the Flan-T5 language model and the ViT vision encoder jointly
(Figure 2) on a large corpus of instruction-following multimodal web
navigation data, which we describe in Section 4.3. In Section 5, we
empirically demonstrate that this instruction-finetuned recipe improves HTML
comprehension and multi-step reasoning and decision making significantly.

###  4.3 Large-scale Data Collection with Finetuned LLM

Recent successes of foundation models are largely powered by internet-scale
data [10, 64, 15, 79]. While large amount of data is critical, for web
navigation domain, there is only a small public dataset for MiniWoB++,
consisting of 12K episodes of human demonstration [55]. Moreover, the dataset
only consists of DOM observations and lacks any visual features, which might
limit the spatial perception of the elements on the page. A large-scale
multimodal dataset, including screenshots of websites, is required to build a
better navigation policy at scale.

To collect a huge amount of multimodal behavioral dataset on MiniWoB++, we
leverage a public finetuned-LLM policy [31] trained with human demonstrations
[55] for data collection, instead of hiring human demonstrators [55, 36],
which significantly reduces the cost to construct a new dataset by leveraging
the prior success of autonomous agents. We gradually increase the dataset
size. We first rollout a LLM policy with 100 episodes per task, which results
in a 2.8K successful episodes. Then, we finetune Flan-T5-XL models with this
small dataset and run those with 10,000 episodes per task. Such efforts result
in a multi-task 347K-episode dataset with HTML and screenshots at each time
step, generated by proficient LLM-based agents. See Appendix D for further
details.

Methods | Modality | Success Rate
---|---|---
WebGUM | HTML | 75.5%
WebGUM (white) | HTML+Image | 76.6%
WebGUM (random) | HTML+Image | 78.5%
WebGUM | HTML+Image | 80.3%
WebGUM (single, $H=2$) | HTML+Image | 64.2%
WebGUM (multiple, $H=1$) | HTML+Image | 64.0%
WebGUM (multiple, $H=2$) | HTML+Image | 66.1%

Figure 3:  (Left) Average success rate with white/random image inputs,
single/multiple visual tokens, and history length ($H=1,2$). We use Flan-T5-XL
for white/random image ablations, and Flan-T5-Base for visual tokens and
history length ablations. The results imply that WebGUM successfully leverages
spatial and temporal information from image modality, and multiple visual
tokens from patches could extract much richer features than a single visual
token per image. (Right) Top-10 performance improvement among MiniWoB++ by
adding image modality to HTML. We subtract the success rates to compute
absolute improvement: (Success Rate of WebGUM(HTML+Image)) - (Success Rate of
WebGUM(HTML)). Image modality is leveraged for multi-step tasks with dynamic
page transitions or tasks that require global perception (e.g. book-flight or
tic-tac-toe). See Appendix E and H for the details.

##  5 Results

We test our method on the MiniWoB++ benchmark [70, 55] with 100 evaluation
episodes per task, taking the average success rate over 56 tasks taken from
Gur et al. [31]. Due to the huge computational requirements, we run one seed
to train each model throughout the paper [36, 31]. Table 2 shows that WebGUM,
with a small 2.8K dataset and Base-size model (310M parameters), significantly
outperforms previous offline methods for web navigation [36, 31], including
the multimodal variant of the recent best method (MM-WebN-T5) we prepared for
comparison. While they used 2.4 million episodes [36] or 3 billion parameters
[31], WebGUM could improve the data and parameter efficiency to achieve
superior performance in offline regime, which is realized by the problem
simplification of web navigation in order to leverage visual modalities and
instruction-finetuned LLMs as strong inductive bias on web environments. In
addition, scaling dataset and model size, WebGUM achieves 80.3% success
rate222Videos are available at https://sites.google.com/view/mm-webnav/,
exceeding the previous best offline model, WebN-T5 [31], by over 31.9% and
being close to reaching the online RL-finetuned SoTA, CC-Net [36] (-16.1%),
despite our fully offline training and much fewer data. The per-task
comparison (see Appendix E) implies online-finetuned CC-Net is still better at
some multi-step, long-horizon tasks (e.g. choose-date or guess-number),
although WebGUM also shows improvements on them. While there is still room for
performance improvement, we believe that simplifying the web navigation
problem to incorporate foundation models for efficient offline training is
quite an important contribution as a scalable and deployable approach towards
real-world web automation where online interactions are prohibitively costly.

In the following sections, we perform extensive and precise ablations of
WebGUM to clearly identify the source of improvement. Especially, we will
demonstrate the contribution of (1) multimodal vision-and-HTML observations
(Section 5.1), (2) better HTML comprehension (Section 5.2) and (3) capability
of multi-step reasoning (Section 5.3) from instruction-finetuned LLMs, and (4)
dataset and model size scaling (Section 5.4).

Methods | Modality | Success Rate
---|---|---
WebN-T5 [31] | HTML | 51.0%
WebGUM | HTML | 67.8%
WebGUM | HTML+Image |  75.5%

Figure 4:  (Left) Example of compositional evaluation on MiniWoB++. We combine
two different tasks (click-link and click-button) into a single-page
sequential task (click-link_click-button) at test time. See Appendix F for the
details of combinations. (Right) Average success rate on 6 compositional
MiniWoB tasks. WebGUM generalizes combinational tasks better than Gur et al.
[31], achieving better success rate by over 24.5% (HTML+Image) or 16.8%
(HTML).

Methods | Modality | Perturbation | Success Rate
---|---|---|---
WebN-T5 | HTML | Top | 24.7%
[31] |  | Bottom | 42.8%
|  | Coordinates | 6.4%
WebGUM | HTML | Top | 53.6%
|  | Bottom | 48.0%
|  | Coordinates | 39.8%
WebGUM | HTML+Image | Top |  71.8%
|  | Bottom |  64.7%
|  | Coordinates |  62.6%

Figure 5:  (Left) Example of input perturbation for MiniWoB++ evaluation,
taken from click-button. We prepare three different types of perturbations at
test time: adding extra HTML at the top of the original input HTML (left) or
at the bottom (middle), and adding task-irrelevant attributes in each element
(right) such as coordinate information (left, right, top, bottom). We randomly
sample extra HTML from the 12K dataset [55]. (Right) Average success rate of
perturbation evaluation over MiniWoB++ 56 tasks. The results reveal that while
all the methods are affected by input corruptions to some extent, WebGUM,
especially with multimodality, achieves significantly better performances than
previous finetuned-LLM.

###  5.1 Image Modality for Grounded Understanding and Temporal Task
Information

To examine if the models actually leverage image modality for a grounded
understanding of websites, we design two ablations at test time: replacing
image observations with completely white images, and with randomly sampled
MiniWoB++ screenshots taken in the initial states. In addition, we also
investigate whether our design choices for image observations (multiple tokens
from patches, historical observations with $H=2$) are suitable ones or not. We
use XL-size models for white/random image ablations, and Base-size models for
visual tokens and history length ablations.

Figure 3 (left) reveals that the performance of the model with white images is
comparable to the unimodal HTML model. Because the model with randomly-taken
images may accidentally contain the images from the target task, WebGUM
(random) slightly surpasses WebGUM (white). These results prove WebGUM
successfully obtains grounded vision and HTML understanding for web navigation
by leveraging semantic and spatial information from image modality. Multiple
visual tokens from patches outperform a single visual token per image, which
means they extract much richer task-relevant features. Besides, we find that
historical image observations ($H=2$) contribute to the improvement more than
single-step observation ($H=1$), which encodes temporal task information.

We also compare per-task performance gaps caused by adding image modality to
instruction-finetuned LLM. Figure 3 (right) presents the top-10 absolute
performance improvement, which suggests WebGUM leverages visual inputs for
multi-step tasks with dynamic page transitions (e.g. book-flight or search-
engine) or the tasks that require global context perception of the page (e.g.
tic-tac-toe or click-shape). See Appendix E and H for further details.

###  5.2 Better HTML Comprehension from Instruction-Finetuned LLMs

Table 2 shows that, thanks to instruction-finetuned LLMs, WebGUM significantly
outperforms MM-WebN-T5 by 24.7%. To analyze the effect of instruction-
finetuning more precisely, we first focus on the capability of HTML
understanding. Since instruction-finetuned LLMs perform well on many NLP tasks
with content comprehension [17, 62, 37], web navigation should also benefit
from them. As a test bed for HTML comprehension, we investigate (1)
generalization to unseen compositions of known tasks, and (2) robustness to
the realistic input perturbations, which are also important challenges for the
web agents to be deployed on the real-world internet.

For the compositional tasks, we pick up 4 click-“something” (link, button,
checkboxes, dialog) tasks and make 6 combinations of these by naively
stitching with 2 or 3 tasks (e.g. Figure 4). These tasks should be resolved in
order. See Appendix F for further details. The results show that WebGUM with
HTML and image inputs outperforms prior finetuned-LLM [31] by over 24.5%,
which implies WebGUM has obtained better reading skills to control computers
and could transfer them to handle unseen HTML in compositional tasks.

To check the robustness against input corruptions, we test three different
realistic perturbations; adding extra HTML at the top or bottom of the
original HTML, and adding attributes of coordinates (left, right, top, bottom;
they are unrelated to solving the tasks) in each element of HTML at test time.
These perturbations often happen in the real world due to the renewal or API
changes, not to mention unknown websites, but rule-based pre-processing may
not fully cover them. The results show that while all the methods are affected
by the input corruptions to some extent, WebGUM, with both HTML and HTML plus
image modalities, achieves significantly better performances than Gur et al.
[31]. Notably, WebGUM outperforms prior finetuned LLM (+ 56.2% in multimodal
and +33.4% in unimodal models) even when extra distracted attributes are added
to HTML. They support our hypothesis: instruction-finetuning imporves HTML
comprehension in LLMs, which enables the downstream agents to deal with out-
of-distribution inputs or tasks robustly.

Methods | Training | Models | Score | Success Rate
---|---|---|---|---
Rule | – | – | 45.6 | 9.6%
IL | SL | BART, BERT | 59.9 | 29.1%
IL+RL | SL+RL | BART, BERT | 62.4 | 28.7%
Act | In-context | PaLM-540B | 62.3 | 30.1%
ReAct | In-context | PaLM-540B | 66.6 | 40.0%
WebN-T5 | SL | T5-XL | 61.0 | 29.8%
WebGUM | SL | Flan-T5-XL | 67.5 | 45.0%

Table 3: Average score and success rate on WebShop [86]. WebGUM achieves 45.0%
success, outperforming baseline approaches including ReAct, a prompted
PaLM-540B. We refer Yao et al. [87] for the baselines.

###  5.3 Ability of Multi-Step Reasoning as a Prior for Interactive Decision
Making

Another notable feature in instruction-finetuned LLMs is an ability of multi-
step reasoning [17]. We hypothesize this reasoning capability would play an
important role as a prior for interactive decision making. To decouple the
evaluation of reasoning capability from visual page perception, HTML
understanding, and the benchmark simulator (MiniWoB++), we extensively
evaluate our WebGUM on WebShop [86], another online-shopping website simulator
with a large amount of real-world product data. Because it requires complex
multi-step decisions considering previous contexts for item comparison,
WebShop is suitable for investigating the capability of multi-step reasoning
from instruction-finetuned LLM in depth [86, 87]. WebShop provides a user
instruction that describes the features of item (e.g. I need a long clip-in
hair extension which is natural looking, and price lower than 20.00 dollars).
The agents should search, compare and choose a proper product that matches the
given instruction. The performance score is evaluated by the percentage of
required attributes covered by the chosen product, and if the product meets
all the requirements, that episode is labeled a success. See Appendix G for
further details of the experiment.

Table 3 shows that WebGUM achieves 45.0% success, significantly outperforming
not only simple baselines, such as supervised imitation learning (IL), IL plus
RL-finetuing and WebN-T5 (by more than 15%), but also recent prompt-based LLM
agents, including ReAct [87] (i.e. PaLM-540B [16] with one-shot prompt and
reasoning annotations), while our model only has 3 billion parameters. Due to
the consistent reasoning and enhanced alignment with user intentions, WebGUM
could compare the products with backtracking, and choose proper options (see
Appendix H). Because BERT, BART, or T5 do not have reasoning capabilities
while Flan-T5 have, our results imply that ability of multi-step reasoning
works as strong and transferable prior knowledge for downstream decision
making.

Figure 6:  Average success rate of WebGUM with different dataset (left) and
model sizes (right). X-axis is a logarithmic scale. As for both HTML and
multimodal models, we could observe the scaling effect: the larger the dataset
and model size are, the higher the success rates are. See Appendix C for
further details.

###  5.4 Scaling Effect in Dataset and Model Size

Lastly, we present how critical it is to scale up the dataset and model size
in WebGUM, as large-scale models often show their superb capability by scaling
tremendous data size and high-capacity model size [71, 10, 43, 65, 64, 83,
16]. To investigate the scalability to the dataset size, we prepare three
dataset: minimal 2.8K demonstrations, 347K demonstrations, and its 20%-size
demonstrations (68K), and then finetune Flan-T5-Base with them. Figure 6
(left) proves that increasing dataset size leads to the improvement of success
rate. It also implies the larger dataset size is more crucial in multimodal
models, which also supports our attempts to construct large-scale multimodal
dataset for web navigation. Notably, Base-size WebGUM even with 2.8K HTML
episodes already achieves 55.7%, surpassing previous SL state-of-the-art
(48.4% by Gur et al. [31]); that dataset size is about 25% of the previous one
released by Liu et al. [55]. This surprising data efficiency comes from the
sufficient inductive bias and alignment with the user intentions in
instruction-finetuned LLMs. WebGUM could fully leverage them to solve
sequential web automation problems.

In addition to dataset size, Figure 6 (right) shows that the performance of
WebGUM improves as the number of parameters in T5 model increases from Base
(220M) to XXL (11B). These results also reveal that scaling the models might
be more important than the dataset; the low-capacity model may cap the
performance at a lower level. This observation encourages our simplification
to apply large-scale foundation models in web navigation easily. See Appendix
C for further details.

##  6 Discussion and Limitation

Throughout the paper, we present an effective and practical methodology to
simplify web navigation into offline training in order to leverage inductive
bias of web environments in instruction-finetuned LLMs. Despite the
substantial improvement among offline-trained approaches, we still leave room
for finetuning large-scale multimodal transformers with online RL [55, 39, 90,
74, 62, 36] in a scalable manner as future work, which might be required for
several long-horizon or data-shortage tasks.

We collect and release a multimodal expert dataset with 347K episodes on
MiniWoB++. However, this is still far from internet-scale dataset that is
necessary for generalist models. Collecting behavioral data at scale by
iterative data-collection and deployment [26, 57, 52] might be a key for
practical interactive agents. Since our approach – taking raw HTML and
screenshots as inputs and predicting parsable actions in text – only has
minimal assumptions which constraint model architectures, it might be
applicable to any advanced LLMs or open-ended situations. While WebGUM could
deal with out-of-distribution compositional and perturbed tasks in a robust
manner, human-level broader generalization to the diverse real-world websites
or instructions is still a hard problem to be resolved.

##  7 Conclusion

Through the transformation of web navigation into data-driven offline training
regime, we develop Web navigation via Grounded Understanding Models (WebGUM)
by finetuning an instruction-finetuned foundation model with multimodal and
proficient demonstrations. WebGUM significantly improves the success rate on
MiniWoB, compared to previous offline-trained SoTA from 48.4% to 80.3%. Our
detailed ablations reveal that multiple visual tokens extract temporal and
spatial information to capture dynamic transition and global context of the
page, and that instruction-finetuned language models remarkably boost web
navigation performance due to the better HTML comprehension and capability of
multi-step reasoning, which deals with out-of-distribution tasks more
robustly, and outperforms PaLM-540B in WebShop. Furthermore, we scale the
existing MiniWoB dataset into multimodal 347K expert demonstrations, about 38
times larger than before. We hope our work would be a step towards building
more capable and scalable models for autonomous web navigation.

## Acknowledgements

HF was supported by JSPS KAKENHI Grant Number JP22J21582. We thank Yusuke
Iwasawa, Mustafa Safdari, Austin Huang, Aleksandra Faust, Heiga Zen for
helpful feedback on this work, and Shunyu Yao for setting up WebShop
experiments.

## References

  * Adolphs et al. [2022] Leonard Adolphs, Benjamin Boerschinger, Christian Buck, Michelle Chen Huebscher, Massimiliano Ciaramita, Lasse Espeholt, Thomas Hofmann, Yannic Kilcher, Sascha Rothe, Pier Giuseppe Sessa, and Lierni Sestorain Saralegui.  Boosting search engines with interactive agents.  In _Transactions on Machine Learning Research_ , 2022.
  * Aghajanyan et al. [2021] Armen Aghajanyan, Dmytro Okhonko, Mike Lewis, Mandar Joshi, Hu Xu, Gargi Ghosh, and Luke Zettlemoyer.  Htlm: Hyper-text pre-training and prompting of language models.  _arXiv preprint arXiv:2107.06955_ , 2021.
  * Aghajanyan et al. [2022] Armen Aghajanyan, Bernie Huang, Candace Ross, Vladimir Karpukhin, Hu Xu, Naman Goyal, Dmytro Okhonko, Mandar Joshi, Gargi Ghosh, Mike Lewis, and Luke Zettlemoyer.  Cm3: A causal masked multimodal model of the internet.  _arXiv preprint arXiv:2201.07520_ , 2022.
  * Ahn et al. [2022] Michael Ahn, Anthony Brohan, Noah Brown, Yevgen Chebotar, Omar Cortes, Byron David, Chelsea Finn, Chuyuan Fu, Keerthana Gopalakrishnan, Karol Hausman, Alex Herzog, Daniel Ho, Jasmine Hsu, Julian Ibarz, Brian Ichter, Alex Irpan, Eric Jang, Rosario Jauregui Ruano, Kyle Jeffrey, Sally Jesmonth, Nikhil J Joshi, Ryan Julian, Dmitry Kalashnikov, Yuheng Kuang, Kuang-Huei Lee, Sergey Levine, Yao Lu, Linda Luu, Carolina Parada, Peter Pastor, Jornell Quiambao, Kanishka Rao, Jarek Rettinghouse, Diego Reyes, Pierre Sermanet, Nicolas Sievers, Clayton Tan, Alexander Toshev, Vincent Vanhoucke, Fei Xia, Ted Xiao, Peng Xu, Sichun Xu, Mengyuan Yan, and Andy Zeng.  Do as i can, not as i say: Grounding language in robotic affordances.  _arXiv preprint arxiv:2204.01691_ , 2022.
  * Alayrac et al. [2022] Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katie Millican, Malcolm Reynolds, Roman Ring, Eliza Rutherford, Serkan Cabi, Tengda Han, Zhitao Gong, Sina Samangooei, Marianne Monteiro, Jacob Menick, Sebastian Borgeaud, Andrew Brock, Aida Nematzadeh, Sahand Sharifzadeh, Mikolaj Binkowski, Ricardo Barreira, Oriol Vinyals, Andrew Zisserman, and Karen Simonyan.  Flamingo: a visual language model for few-shot learning.  _arXiv preprint arxiv:2204.14198_ , 2022.
  * Appalaraju et al. [2021] Srikar Appalaraju, Bhavan Jasani, Bhargava Urala Kota, Yusheng Xie, and R. Manmatha.  Docformer: End-to-end transformer for document understanding.  In _International Conference on Computer Vision_ , 2021.
  * Austin et al. [2021] Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, and Charles Sutton.  Program synthesis with large language models.  _arXiv preprint arXiv:2108.07732_ , 2021.
  * Bommasani et al. [2021] Rishi Bommasani, Drew A. Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S. Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, Erik Brynjolfsson, Shyamal Buch, Dallas Card, Rodrigo Castellon, Niladri Chatterji, Annie Chen, Kathleen Creel, Jared Quincy Davis, Dora Demszky, Chris Donahue, Moussa Doumbouya, Esin Durmus, Stefano Ermon, John Etchemendy, Kawin Ethayarajh, Li Fei-Fei, Chelsea Finn, Trevor Gale, Lauren Gillespie, Karan Goel, Noah Goodman, Shelby Grossman, Neel Guha, Tatsunori Hashimoto, Peter Henderson, John Hewitt, Daniel E. Ho, Jenny Hong, Kyle Hsu, Jing Huang, Thomas Icard, Saahil Jain, Dan Jurafsky, Pratyusha Kalluri, Siddharth Karamcheti, Geoff Keeling, Fereshte Khani, Omar Khattab, Pang Wei Kohd, Mark Krass, Ranjay Krishna, Rohith Kuditipudi, Ananya Kumar, Faisal Ladhak, Mina Lee, Tony Lee, Jure Leskovec, Isabelle Levent, Xiang Lisa Li, Xuechen Li, Tengyu Ma, Ali Malik, Christopher D. Manning, Suvir Mirchandani, Eric Mitchell, Zanele Munyikwa, Suraj Nair, Avanika Narayan, Deepak Narayanan, Ben Newman, Allen Nie, Juan Carlos Niebles, Hamed Nilforoshan, Julian Nyarko, Giray Ogut, Laurel Orr, Isabel Papadimitriou, Joon Sung Park, Chris Piech, Eva Portelance, Christopher Potts, Aditi Raghunathan, Rob Reich, Hongyu Ren, Frieda Rong, Yusuf Roohani, Camilo Ruiz, Jack Ryan, Christopher Ré, Dorsa Sadigh, Shiori Sagawa, Keshav Santhanam, Andy Shih, Krishnan Srinivasan, Alex Tamkin, Rohan Taori, Armin W. Thomas, Florian Tramèr, Rose E. Wang, William Wang, Bohan Wu, Jiajun Wu, Yuhuai Wu, Sang Michael Xie, Michihiro Yasunaga, Jiaxuan You, Matei Zaharia, Michael Zhang, Tianyi Zhang, Xikun Zhang, Yuhui Zhang, Lucia Zheng, Kaitlyn Zhou, and Percy Liang.  On the opportunities and risks of foundation models.  _arXiv preprint arXiv:2108.07258_ , 2021.
  * Brohan et al. [2022] Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Joseph Dabis, Chelsea Finn, Keerthana Gopalakrishnan, Karol Hausman, Alex Herzog, Jasmine Hsu, Julian Ibarz, Brian Ichter, Alex Irpan, Tomas Jackson, Sally Jesmonth, Nikhil J Joshi, Ryan Julian, Dmitry Kalashnikov, Yuheng Kuang, Isabel Leal, Kuang-Huei Lee, Sergey Levine, Yao Lu, Utsav Malla, Deeksha Manjunath, Igor Mordatch, Ofir Nachum, Carolina Parada, Jodilyn Peralta, Emily Perez, Karl Pertsch, Jornell Quiambao, Kanishka Rao, Michael Ryoo, Grecia Salazar, Pannag Sanketi, Kevin Sayed, Jaspiar Singh, Sumedh Sontakke, Austin Stone, Clayton Tan, Huong Tran, Vincent Vanhoucke, Steve Vega, Quan Vuong, Fei Xia, Ted Xiao, Peng Xu, Sichun Xu, Tianhe Yu, and Brianna Zitkovich.  Rt-1: Robotics transformer for real-world control at scale.  _arXiv preprint arXiv:2212.06817_ , 2022.
  * Brown et al. [2020] Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei.  Language models are few-shot learners.  _arXiv preprint arXiv:2005.14165_ , 2020.
  * Burns et al. [2022] Andrea Burns, Deniz Arsan, Sanjna Agrawal, Ranjitha Kumar, Kate Saenko, and Bryan A. Plummer.  A dataset for interactive vision-language navigation with unknown command feasibility.  In _European Conference on Computer Vision_ , 2022.
  * Chen et al. [2021a] Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee, Aditya Grover, Michael Laskin, Pieter Abbeel, Aravind Srinivas, and Igor Mordatch.  Decision transformer: Reinforcement learning via sequence modeling.  In _Advances in Neural Information Processing Systems_ , 2021a.
  * Chen et al. [2021b] Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cummings, Matthias Plappert, Fotios Chantzis, Elizabeth Barnes, Ariel Herbert-Voss, William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang, Igor Babuschkin, Suchir Balaji, Shantanu Jain, William Saunders, Christopher Hesse, Andrew N. Carr, Jan Leike, Josh Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob McGrew, Dario Amodei, Sam McCandlish, Ilya Sutskever, and Wojciech Zaremba.  Evaluating large language models trained on code.  _arXiv preprint arXiv:2107.03374_ , 2021b.
  * Chen et al. [2021c] Tao Chen, Jie Xu, and Pulkit Agrawal.  A system for general in-hand object re-orientation.  In _Conference on Robot Learning_ , 2021c.
  * Chen et al. [2022] Xi Chen, Xiao Wang, Soravit Changpinyo, AJ Piergiovanni, Piotr Padlewski, Daniel Salz, Sebastian Goodman, Adam Grycner, Basil Mustafa, Lucas Beyer, Alexander Kolesnikov, Joan Puigcerver, Nan Ding, Keran Rong, Hassan Akbari, Gaurav Mishra, Linting Xue, Ashish Thapliyal, James Bradbury, Weicheng Kuo, Mojtaba Seyedhosseini, Chao Jia, Burcu Karagol Ayan, Carlos Riquelme, Andreas Steiner, Anelia Angelova, Xiaohua Zhai, Neil Houlsby, and Radu Soricut.  Pali: A jointly-scaled multilingual language-image model.  _arXiv preprint arxiv:2209.06794_ , 2022.
  * Chowdhery et al. [2022] Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam Shazeer, Vinodkumar Prabhakaran, Emily Reif, Nan Du, Ben Hutchinson, Reiner Pope, James Bradbury, Jacob Austin, Michael Isard, Guy Gur-Ari, Pengcheng Yin, Toju Duke, Anselm Levskaya, Sanjay Ghemawat, Sunipa Dev, Henryk Michalewski, Xavier Garcia, Vedant Misra, Kevin Robinson, Liam Fedus, Denny Zhou, Daphne Ippolito, David Luan, Hyeontaek Lim, Barret Zoph, Alexander Spiridonov, Ryan Sepassi, David Dohan, Shivani Agrawal, Mark Omernick, Andrew M. Dai, Thanumalayan Sankaranarayana Pillai, Marie Pellat, Aitor Lewkowycz, Erica Moreira, Rewon Child, Oleksandr Polozov, Katherine Lee, Zongwei Zhou, Xuezhi Wang, Brennan Saeta, Mark Diaz, Orhan Firat, Michele Catasta, Jason Wei, Kathy Meier-Hellstern, Douglas Eck, Jeff Dean, Slav Petrov, and Noah Fiedel.  Palm: Scaling language modeling with pathways.  _arXiv preprint arXiv:2204.02311_ , 2022.
  * Chung et al. [2022] Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, Albert Webson, Shixiang Shane Gu, Zhuyun Dai, Mirac Suzgun, Xinyun Chen, Aakanksha Chowdhery, Sharan Narang, Gaurav Mishra, Adams Yu, Vincent Zhao, Yanping Huang, Andrew Dai, Hongkun Yu, Slav Petrov, Ed H. Chi, Jeff Dean, Jacob Devlin, Adam Roberts, Denny Zhou, Quoc V. Le, and Jason Wei.  Scaling instruction-finetuned language models.  _arXiv preprint arxiv:2210.11416_ , 2022.
  * DeepMind Interactive Agents Team et al. [2021] DeepMind Interactive Agents Team, Josh Abramson, Arun Ahuja, Arthur Brussee, Federico Carnevale, Mary Cassin, Felix Fischer, Petko Georgiev, Alex Goldin, Mansi Gupta, Tim Harley, Felix Hill, Peter C Humphreys, Alden Hung, Jessica Landon, Timothy Lillicrap, Hamza Merzic, Alistair Muldal, Adam Santoro, Guy Scully, Tamara von Glehn, Greg Wayne, Nathaniel Wong, Chen Yan, and Rui Zhu.  Creating multimodal interactive agents with imitation and self-supervised learning.  _arXiv preprint arXiv:2112.03763_ , 2021.
  * Deng et al. [2009] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei.  Imagenet: A large-scale hierarchical image database.  In _Conference on Computer Vision and Pattern Recognition_ , 2009\.
  * Devlin et al. [2019] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.  Bert: Pre-training of deep bidirectional transformers for language understanding.  _arXiv preprint arXiv:1810.04805_ , 2019.
  * Diaz et al. [2013] Oscar Diaz, Itziar Otaduy, and Gorka Puente.  User-driven automation of web form filling.  In _International Conference on Web Engineering_ , 2013.
  * Dosovitskiy et al. [2020] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al.  An image is worth 16x16 words: Transformers for image recognition at scale.  _arXiv preprint arXiv:2010.11929_ , 2020.
  * Fan et al. [2022] Linxi Fan, Guanzhi Wang, Yunfan Jiang, Ajay Mandlekar, Yuncong Yang, Haoyi Zhu, Andrew Tang, De-An Huang, Yuke Zhu, and Anima Anandkumar.  Minedojo: Building open-ended embodied agents with internet-scale knowledge.  _arXiv preprint arXiv:2206.08853_ , 2022.
  * Furuta et al. [2022a] Hiroki Furuta, Yusuke Iwasawa, Yutaka Matsuo, and Shixiang Shane Gu.  A system for morphology-task generalization via unified representation and behavior distillation.  _arXiv preprint arXiv:2211.14296_ , 2022a.
  * Furuta et al. [2022b] Hiroki Furuta, Yutaka Matsuo, and Shixiang Shane Gu.  Generalized decision transformer for offline hindsight information matching.  In _International Conference on Learning Representations_ , 2022b.
  * Ghosh et al. [2021] Dibya Ghosh, Abhishek Gupta, Ashwin Reddy, Justin Fu, Coline Manon Devin, Benjamin Eysenbach, and Sergey Levine.  Learning to reach goals via iterated supervised learning.  In _International Conference on Learning Representations_ , 2021.
  * Gu et al. [2021a] Shixiang Shane Gu, Manfred Diaz, Daniel C. Freeman, Hiroki Furuta, Seyed Kamyar Seyed Ghasemipour, Anton Raichuk, Byron David, Erik Frey, Erwin Coumans, and Olivier Bachem.  Braxlines: Fast and interactive toolkit for rl-driven behavior engineering beyond reward maximization.  _arXiv preprint arXiv:2110.04686_ , 2021a.
  * Gu et al. [2021b] Xiuye Gu, Tsung-Yi Lin, Weicheng Kuo, and Yin Cui.  Open-vocabulary object detection via vision and language knowledge distillation.  _arXiv preprint arxiv:2104.13921_ , 2021b.
  * Gur et al. [2019] Izzeddin Gur, Ulrich Rueckert, Aleksandra Faust, and Dilek Hakkani-Tur.  Learning to navigate the web.  In _International Conference on Learning Representations_ , 2019.
  * Gur et al. [2021] Izzeddin Gur, Natasha Jaques, Yingjie Miao, Jongwook Choi, Manoj Tiwari, Honglak Lee, and Aleksandra Faust.  Environment generation for zero-shot compositional reinforcement learning.  In _Advances in neural information processing systems_ , 2021.
  * Gur et al. [2022] Izzeddin Gur, Ofir Nachum, Yingjie Miao, Mustafa Safdari, Austin Huang, Aakanksha Chowdhery, Sharan Narang, Noah Fiedel, and Aleksandra Faust.  Understanding html with large language models.  _arXiv preprint arxiv:2210.03945_ , 2022.
  * He et al. [2016] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In _Conference on Computer Vision and Pattern Recognition_ , 2016\.
  * He et al. [2020] Zecheng He, Srinivas Sunkara, Xiaoxue Zang, Ying Xu, Lijuan Liu, Nevan Wichers, Gabriel Schubiner, Ruby Lee, Jindong Chen, and Blaise Agüera y Arcas.  Actionbert: Leveraging user actions for semantic understanding of user interfaces.  _arXiv preprint arXiv:2012.12350_ , 2020.
  * Huang et al. [2022a] Wenlong Huang, Pieter Abbeel, Deepak Pathak, and Igor Mordatch.  Language models as zero-shot planners: Extracting actionable knowledge for embodied agents.  _arXiv preprint arXiv:2201.07207_ , 2022a.
  * Huang et al. [2022b] Wenlong Huang, Fei Xia, Ted Xiao, Harris Chan, Jacky Liang, Pete Florence, Andy Zeng, Jonathan Tompson, Igor Mordatch, Yevgen Chebotar, Pierre Sermanet, Noah Brown, Tomas Jackson, Linda Luu, Sergey Levine, Karol Hausman, and Brian Ichter.  Inner monologue: Embodied reasoning through planning with language models.  _arXiv preprint arxiv:2207.05608_ , 2022b.
  * Humphreys et al. [2022] Peter C Humphreys, David Raposo, Toby Pohlen, Gregory Thornton, Rachita Chhaparia, Alistair Muldal, Josh Abramson, Petko Georgiev, Alex Goldin, Adam Santoro, and Timothy Lillicrap.  A data-driven approach for learning to control computers.  In _International Conference on Machine Learning_ , 2022.
  * Iyer et al. [2022] Srinivasan Iyer, Xi Victoria Lin, Ramakanth Pasunuru, Todor Mihaylov, Daniel Simig, Ping Yu, Kurt Shuster, Tianlu Wang, Qing Liu, Punit Singh Koura, Xian Li, Brian O’Horo, Gabriel Pereyra, Jeff Wang, Christopher Dewan, Asli Celikyilmaz, Luke Zettlemoyer, and Ves Stoyanov.  Opt-iml: Scaling language model instruction meta learning through the lens of generalization.  _arXiv preprint arXiv:2212.12017_ , 2022.
  * Janner et al. [2021] Michael Janner, Qiyang Li, and Sergey Levine.  Offline reinforcement learning as one big sequence modeling problem.  In _Advances in Neural Information Processing Systems_ , 2021.
  * Jaques et al. [2019] Natasha Jaques, Asma Ghandeharioun, Judy Hanwen Shen, Craig Ferguson, Agata Lapedriza, Noah Jones, Shixiang Gu, and Rosalind Picard.  Way off-policy batch deep reinforcement learning of implicit human preferences in dialog.  _arXiv preprint arXiv:1907.00456_ , 2019.
  * Jia et al. [2019] Sheng Jia, Jamie Ryan Kiros, and Jimmy Ba.  DOM-q-NET: Grounded RL on structured language.  In _International Conference on Learning Representations_ , 2019.
  * Jiang et al. [2022] Yunfan Jiang, Agrim Gupta, Zichen Zhang, Guanzhi Wang, Yongqiang Dou, Yanjun Chen, Li Fei-Fei, Anima Anandkumar, Yuke Zhu, and Linxi Fan.  Vima: General robot manipulation with multimodal prompts.  _arXiv preprint arXiv:2210.03094_ , 2022.
  * Kamath et al. [2021] Aishwarya Kamath, Mannat Singh, Yann LeCun, Gabriel Synnaeve, Ishan Misra, and Nicolas Carion.  Mdetr – modulated detection for end-to-end multi-modal understanding.  _arXiv preprint arXiv:2104.12763_ , 2021.
  * Kaplan et al. [2020] Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei.  Scaling laws for neural language models.  _arXiv preprint arXiv:2001.08361_ , 2020.
  * Kojima et al. [2022] Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa.  Large language models are zero-shot reasoners.  In _Advances In Neural Information Processing Systems_ , 2022.
  * Kudo and Richardson [2018] Taku Kudo and John Richardson.  Sentencepiece: A simple and language independent subword tokenizer and detokenizer for neural text processing.  _arXiv preprint arXiv:1808.06226_ , 2018.
  * Lee et al. [2022a] Kenton Lee, Mandar Joshi, Iulia Turc, Hexiang Hu, Fangyu Liu, Julian Eisenschlos, Urvashi Khandelwal, Peter Shaw, Ming-Wei Chang, and Kristina Toutanova.  Pix2struct: Screenshot parsing as pretraining for visual language understanding.  _arXiv preprint arXiv:2210.03347_ , 2022a.
  * Lee et al. [2022b] Kuang-Huei Lee, Ofir Nachum, Mengjiao Yang, Lisa Lee, Daniel Freeman, Winnie Xu, Sergio Guadarrama, Ian Fischer, Eric Jang, Henryk Michalewski, and Igor Mordatch.  Multi-game decision transformers.  _arXiv preprint arxiv:2205.15241_ , 2022b.
  * Lewis et al. [2019] Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Ves Stoyanov, and Luke Zettlemoyer.  Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension.  _arXiv preprint arXiv:1910.13461_ , 2019.
  * Li et al. [2021a] Chenliang Li, Bin Bi, Ming Yan, Wei Wang, Songfang Huang, Fei Huang, and Luo Si.  Structurallm: Structural pre-training for form understanding.  _arXiv preprint arxiv:2105.11210_ , 2021a.
  * Li et al. [2021b] Junlong Li, Yiheng Xu, Lei Cui, and Furu Wei.  Markuplm: Pre-training of text and markup language for visually-rich document understanding.  _arXiv preprint arxiv:2110.08518_ , 2021b.
  * Li et al. [2021c] Peizhao Li, Jiuxiang Gu, Jason Kuen, Vlad I. Morariu, Handong Zhao, Rajiv Jain, Varun Manjunatha, and Hongfu Liu.  Selfdoc: Self-supervised document representation learning.  In _Conference on Computer Vision and Pattern Recognition_ , 2021c.
  * Li et al. [2022a] Shuang Li, Xavier Puig, Chris Paxton, Yilun Du, Clinton Wang, Linxi Fan, Tao Chen, De-An Huang, Ekin Akyürek, Anima Anandkumar, Jacob Andreas, Igor Mordatch, Antonio Torralba, and Yuke Zhu.  Pre-trained language models for interactive decision-making.  In _Advances In Neural Information Processing Systems_ , 2022a.
  * Li et al. [2020] Yang Li, Jiacong He, Xin Zhou, Yuan Zhang, and Jason Baldridge.  Mapping natural language instructions to mobile ui action sequences.  In _Annual Conference of the Association for Computational Linguistics_ , 2020.
  * Li et al. [2022b] Yujia Li, David Choi, Junyoung Chung, Nate Kushman, Julian Schrittwieser, Remi Leblond, Tom Eccles, James Keeling, Felix Gimeno, Agustin Dal Lago, Thomas Hubert, Peter Choy, Cyprien de, Masson dAutume, Igor Babuschkin, Xinyun Chen, Po-Sen Huang, Johannes Welbl, Sven Gowal, Alexey Cherepanov, James Molloy, Daniel J. Mankowitz, Esme Sutherland Robson, Pushmeet Kohli, Nando de Freitas, Koray Kavukcuoglu, and Oriol Vinyals.  Competition-level code generation with alphacode, 2022b.
  * Liu et al. [2018] Evan Zheran Liu, Kelvin Guu, Panupong Pasupat, and Percy Liang.  Reinforcement learning on web interfaces using workflow-guided exploration.  In _International Conference on Learning Representations_ , 2018.
  * Lu et al. [2022] Jiasen Lu, Christopher Clark, Rowan Zellers, Roozbeh Mottaghi, and Aniruddha Kembhavi.  Unified-io: A unified model for vision, language, and multi-modal tasks.  _arXiv preprint arXiv:2206.11795_ , 2022.
  * Matsushima et al. [2021] Tatsuya Matsushima, Hiroki Furuta, Yutaka Matsuo, Ofir Nachum, and Shixiang Gu.  Deployment-efficient reinforcement learning via model-based offline optimization.  In _International Conference on Learning Representations_ , 2021.
  * Mazumder and Riva [2020] Sahisnu Mazumder and Oriana Riva.  Flin: A flexible natural language interface for web navigation.  _arXiv preprint arXiv:2010.12844_ , 2020.
  * Meta Fundamental AI Research Diplomacy Team et al. [2022] Meta Fundamental AI Research Diplomacy Team, Anton Bakhtin, Noam Brown, Emily Dinan, Gabriele Farina, Colin Flaherty, Daniel Fried, Andrew Goff, Jonathan Gray, Hengyuan Hu, Athul Paul Jacob, Mojtaba Komeili, Karthik Konath, Minae Kwon, Adam Lerer, Mike Lewis, Alexander H. Miller, Sasha Mitts, Adithya Renduchintala, Stephen Roller, Dirk Rowe, Weiyan Shi, Joe Spisak, Alexander Wei, David Wu, Hugh Zhang, and Markus Zijlstra.  Human-level play in the game of diplomacy by combining language models with strategic reasoning.  _Science_ , 378(6624):1067–1074, 2022.
  * Nakano et al. [2021] Reiichiro Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu, Long Ouyang, Christina Kim, Christopher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, Xu Jiang, Karl Cobbe, Tyna Eloundou, Gretchen Krueger, Kevin Button, Matthew Knight, Benjamin Chess, and John Schulman.  Webgpt: Browser-assisted question-answering with human feedback.  _arXiv preprint arXiv:2112.09332_ , 2021.
  * Nogueira and Cho [2016] Rodrigo Nogueira and Kyunghyun Cho.  End-to-end goal-driven web navigation.  In _Advances In Neural Information Processing Systems_ , 2016.
  * Ouyang et al. [2022] Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe.  Training language models to follow instructions with human feedback.  _arXiv preprint arxiv:2203.02155_ , 2022.
  * Radford et al. [2019] Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever.  Language models are unsupervised multitask learners, 2019.
  * Radford et al. [2021] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever.  Learning transferable visual models from natural language supervision.  _arXiv preprint arXiv:2103.00020_ , 2021.
  * Rae et al. [2021] Jack W. Rae, Sebastian Borgeaud, Trevor Cai, Katie Millican, Jordan Hoffmann, Francis Song, John Aslanides, Sarah Henderson, Roman Ring, Susannah Young, Eliza Rutherford, Tom Hennigan, Jacob Menick, Albin Cassirer, Richard Powell, George van den Driessche, Lisa Anne Hendricks, Maribeth Rauh, Po-Sen Huang, Amelia Glaese, Johannes Welbl, Sumanth Dathathri, Saffron Huang, Jonathan Uesato, John Mellor, Irina Higgins, Antonia Creswell, Nat McAleese, Amy Wu, Erich Elsen, Siddhant Jayakumar, Elena Buchatskaya, David Budden, Esme Sutherland, Karen Simonyan, Michela Paganini, Laurent Sifre, Lena Martens, Xiang Lorraine Li, Adhiguna Kuncoro, Aida Nematzadeh, Elena Gribovskaya, Domenic Donato, Angeliki Lazaridou, Arthur Mensch, Jean-Baptiste Lespiau, Maria Tsimpoukelli, Nikolai Grigorev, Doug Fritz, Thibault Sottiaux, Mantas Pajarskas, Toby Pohlen, Zhitao Gong, Daniel Toyama, Cyprien de Masson d’Autume, Yujia Li, Tayfun Terzi, Vladimir Mikulik, Igor Babuschkin, Aidan Clark, Diego de Las Casas, Aurelia Guy, Chris Jones, James Bradbury, Matthew Johnson, Blake Hechtman, Laura Weidinger, Iason Gabriel, William Isaac, Ed Lockhart, Simon Osindero, Laura Rimell, Chris Dyer, Oriol Vinyals, Kareem Ayoub, Jeff Stanway, Lorrayne Bennett, Demis Hassabis, Koray Kavukcuoglu, and Geoffrey Irving.  Scaling language models: Methods, analysis &; insights from training gopher.  _arXiv preprint arXiv:2112.11446_ , 2021.
  * Raffel et al. [2020] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu.  Exploring the limits of transfer learning with a unified text-to-text transformer.  _Journal of Machine Learning Research_ , 21(140):1–67, 2020.
  * Reed et al. [2022] Scott Reed, Konrad Zolna, Emilio Parisotto, Sergio Gomez Colmenarejo, Alexander Novikov, Gabriel Barth-Maron, Mai Gimenez, Yury Sulsky, Jackie Kay, Jost Tobias Springenberg, Tom Eccles, Jake Bruce, Ali Razavi, Ashley Edwards, Nicolas Heess, Yutian Chen, Raia Hadsell, Oriol Vinyals, Mahyar Bordbar, and Nando de Freitas.  A generalist agent.  _arXiv preprint arxiv:2205.06175_ , 2022.
  * Roberts et al. [2022] Adam Roberts, Hyung Won Chung, Anselm Levskaya, Gaurav Mishra, James Bradbury, Daniel Andor, Sharan Narang, Brian Lester, Colin Gaffney, Afroz Mohiuddin, Curtis Hawthorne, Aitor Lewkowycz, Alex Salcianu, Marc van Zee, Jacob Austin, Sebastian Goodman, Livio Baldini Soares, Haitang Hu, Sasha Tsvyashchenko, Aakanksha Chowdhery, Jasmijn Bastings, Jannis Bulian, Xavier Garcia, Jianmo Ni, Andrew Chen, Kathleen Kenealy, Jonathan H. Clark, Stephan Lee, Dan Garrette, James Lee-Thorp, Colin Raffel, Noam Shazeer, Marvin Ritter, Maarten Bosma, Alexandre Passos, Jeremy Maitin-Shepard, Noah Fiedel, Mark Omernick, Brennan Saeta, Ryan Sepassi, Alexander Spiridonov, Joshua Newlan, and Andrea Gesmundo.  Scaling up models and data with t5x and seqio.  _arXiv preprint arXiv:2203.17189_ , 2022.
  * Shah et al. [2022] Dhruv Shah, Blazej Osinski, Brian Ichter, and Sergey Levine.  Lm-nav: Robotic navigation with large pre-trained models of language, vision, and action.  In _Conference on Robot Learning_ , 2022.
  * Shi et al. [2017] Tianlin Shi, Andrej Karpathy, Linxi Fan, Jonathan Hernandez, and Percy Liang.  World of bits: An open-domain platform for web-based agents.  In _International Conference on Machine Learning_ , 2017.
  * Shoeybi et al. [2019] Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro.  Megatron-lm: Training multi-billion parameter language models using model parallelism.  _arXiv preprint arXiv:1909.08053_ , 2019.
  * Shridhar et al. [2022] Mohit Shridhar, Lucas Manuelli, and Dieter Fox.  Perceiver-actor: A multi-task transformer for robotic manipulation.  In _Conference on Robot Learning_ , 2022.
  * Shvo et al. [2021] Maayan Shvo, Zhiming Hu, Rodrigo Toro Icarte, Iqbal Mohomed, Allan D. Jepson, and Sheila A. McIlraith.  Appbuddy: Learning to accomplish tasks in mobile apps via reinforcement learning.  In _Canadian Conference on Artificial Intelligence_ , 2021.
  * Stiennon et al. [2020] Nisan Stiennon, Long Ouyang, Jeff Wu, Daniel M. Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul Christiano.  Learning to summarize from human feedback.  In _Advances in Neural Information Processing Systems_ , 2020.
  * Tang et al. [2022] Zineng Tang, Ziyi Yang, Guoxin Wang, Yuwei Fang, Yang Liu, Chenguang Zhu, Michael Zeng, Cha Zhang, and Mohit Bansal.  Unifying vision, text, and layout for universal document processing.  _arXiv preprint arxiv:2212.02623_ , 2022.
  * Tay et al. [2022] Yi Tay, Mostafa Dehghani, Vinh Q. Tran, Xavier Garcia, Jason Wei, Xuezhi Wang, Hyung Won Chung, Dara Bahri, Tal Schuster, Huaixiu Steven Zheng, Denny Zhou, Neil Houlsby, and Donald Metzler.  Ul2: Unifying language learning paradigms.  _arXiv preprint arXiv:2205.05131_ , 2022.
  * Toyama et al. [2021] Daniel Toyama, Philippe Hamel, Anita Gergely, Gheorghe Comanici, Amelia Glaese, Zafarali Ahmed, Tyler Jackson, Shibl Mourad, and Doina Precup.  Androidenv: A reinforcement learning platform for android.  _arXiv preprint arXiv:2105.13231_ , 2021.
  * Vaswani et al. [2017] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  In _Advances in neural information processing systems_ , 2017.
  * Wang et al. [2023] Chengyi Wang, Sanyuan Chen, Yu Wu, Ziqiang Zhang, Long Zhou, Shujie Liu, Zhuo Chen, Yanqing Liu, Huaming Wang, Jinyu Li, Lei He, Sheng Zhao, and Furu Wei.  Neural codec language models are zero-shot text to speech synthesizers.  _arXiv preprint arXiv:2301.02111_ , 2023.
  * Wang et al. [2022a] Jiapeng Wang, Lianwen Jin, and Kai Ding.  LiLT: A simple yet effective language-independent layout transformer for structured document understanding.  In _Annual Meeting of the Association for Computational Linguistics_ , 2022a.
  * Wang et al. [2022b] Qifan Wang, Yi Fang, Anirudh Ravula, Fuli Feng, Xiaojun Quan, and Dongfang Liu.  Webformer: The web-page transformer for structure information extraction.  _arXiv preprint arXiv:2202.00217_ , 2022b.
  * Wei et al. [2022a] Jason Wei, Maarten Bosma, Vincent Y. Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M. Dai, and Quoc V. Le.  Finetuned language models are zero-shot learners.  In _International Conference on Learning Representations_ , 2022a.
  * Wei et al. [2022b] Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, Ed H. Chi, Tatsunori Hashimoto, Oriol Vinyals, Percy Liang, Jeff Dean, and William Fedus.  Emergent abilities of large language models.  _arXiv preprint arXiv:2206.08853_ , 2022b.
  * Wei et al. [2022c] Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc Le, and Denny Zhou.  Chain of thought prompting elicits reasoning in large language models.  _arXiv preprint arXiv:2201.11903_ , 2022c.
  * Xu et al. [2019] Yiheng Xu, Minghao Li, Lei Cui, Shaohan Huang, Furu Wei, and Ming Zhou.  LayoutLM: Pre-training of text and layout for document image understanding.  _arXiv preprint arxiv:1912.13318_ , 2019.
  * Yao et al. [2022a] Shunyu Yao, Howard Chen, John Yang, and Karthik Narasimhan.  Webshop: Towards scalable real-world web interaction with grounded language agents.  _arXiv preprint arxiv:2207.01206_ , 2022a.
  * Yao et al. [2022b] Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao.  React: Synergizing reasoning and acting in language models.  _arXiv preprint arXiv:2210.03629_ , 2022b.
  * Zaheer et al. [2022] Manzil Zaheer, Kenneth Marino, Will Grathwohl, John Schultz, Wendy Shang, Sheila Babayan, Arun Ahuja, Ishita Dasgupta, Christine Kaeser-Chen, and Rob Fergus.  Learning to navigate wikipedia by taking random walks.  _arXiv preprint arXiv:2211.00177_ , 2022.
  * Zeng et al. [2022] Andy Zeng, Maria Attarian, Brian Ichter, Krzysztof Choromanski, Adrian Wong, Stefan Welker, Federico Tombari, Aveek Purohit, Michael Ryoo, Vikas Sindhwani, Johnny Lee, Vincent Vanhoucke, and Pete Florence.  Socratic models: Composing zero-shot multimodal reasoning with language.  _arXiv preprint arXiv:2204.00598_ , 2022.
  * Ziegler et al. [2019] Daniel M. Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B. Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving.  Fine-tuning language models from human preferences.  _arXiv preprint arxiv:1909.08593_ , 2019.

## Appendix

##  Appendix A Broader Impacts

While WebGUM is evaluated only in realistic web simulators [70, 55, 86], we
should carefully conduct it if we deploy the autonomous web agent on the real-
world Internet because of security and safety reasons. For instance, the wrong
password may cause an account freeze, and emailing the wrong person is
problematic in a business scene. Training with online RL may often be
infeasible for this reason, while we demonstrate an alternative approach;
data-driven, fully offline training by leveraging inductive bias in foundation
models. Autonomous agents, well-grounded with the user’s intention, should be
helpful in our daily lives by reducing our burden on computer tasks.

##  Appendix B Implementation Details

We adopt the encoder-decoder models proposed by Raffel et al. [66] as
multimodal transformers, and vision transformer [22] pre-trained with
ImageNet-21K [19] as an image encoder for the visual
tokens333https://github.com/google-research/scenic. We especially use ViT-B16,
a small-size transformer with 86 million parameters, which divides an input
image into $16\times 16$-size patches. We use publicly available checkpoints
of T5 [66]444https://github.com/google-
research/t5x/blob/main/docs/models.md#t5-11-checkpoints, Flan-T5
[17]555https://github.com/google-
research/t5x/blob/main/docs/models.md#flan-t5-checkpoints, and T5-XL finetuned
with MiniWoB++ demonstrations
[31]666https://console.cloud.google.com/storage/browser/gresearch/webllm/webn_t5_3b
for the experiments. As suggested in Gur et al. [31], we focus on encoder-
decoder architectures to solve HTML-based web navigation tasks. Applying our
method to other architectures, such as auto-regressive decoder-only models
[63, 10, 16] remains as future work. To construct the training pipeline, we
leverage SeqIO [68] library, and use SentencePiece [45] vocabulary with 32K
tokens from C4 dataset [66] for text tokenization. The batch size for training
is 128, and input sequence length is set to 4096 tokens. We use cloud TPU-v4,
which has a 32 GiB HBM memory space for the experiments. Base-size models
require 256 cores and XL-size models do 512 cores, which takes 1-2 days for
finetuning.

###  B.1 Input Length

Gur et al. [31] set the max input length to 512, while we observe that some
tasks in MiniWoB++ may exceed it due to the complex webpages with many
contents (Table 4). We investigate the different input lengths in Table 5.
WebGUM already surpasses WebN-T5 [31] with 512 tokens by 17.7%, and
consistently improves the performance in both HTML and HTML+image inputs, as
the input length increases from 512 to 4096 tokens (+14.2%). We set 4096
tokens as a default length in this work.

Min | Mean | 90% Percentile | 95% Percentile | Max
---|---|---|---|---
77 | 507 | 870 | 1823 | 4072

Table 4:  Statistics of input tokens in our 347K dataset. While the average is
507 tokens, the maximum length is over 4K tokens, since some tasks in
MiniWoB++ have complex webpages with many HTML elements.

Methods | Modality | Input Length | Success Rate
---|---|---|---
WebN-T5 [31] | HTML | 512 | 48.4%
WebGUM | HTML | 512 | 61.5%
WebGUM | HTML | 1024 | 68.8%
WebGUM | HTML | 2048 | 73.5%
WebGUM | HTML | 4096 | 75.5%
WebGUM | HTML+Image | 512 | 66.1%
WebGUM | HTML+Image | 1024 | 70.7%
WebGUM | HTML+Image | 2048 | 75.0%
WebGUM | HTML+Image | 4096 | 80.3%

Table 5:  Average success rate with different input sequence length. The
models are initialized with Flan-T5-XL (+ViT-B16) and trained with our
347K-episode dataset. WebGUM consistently improves the performance in both
HTML and HTML+image inputs, as the input length increases from 512 to 4096
tokens.

##  Appendix C Details on Dataset and Model Size Scaling

We here present how critical it is to scale up the dataset and model size in
WebGUM. For the dataset size ablation, we use Flan-T5-Base and ViT-B16. As for
both HTML and multimodal models, we could observe the scaling effects in web
navigation: the larger the dataset (Table 6) and model (Table 7) size are, the
higher the success rates are. Surprisingly, our approach even with only 2.8K
HTML episodes (about 25% of the previous one curated by Liu et al. [55]) and
Base-size model (about 7.3% parameters) already achieves 55.7%, surpassing
previous SL state-of-the-art (48.4% by Gur et al. [31]). This surprising
efficiency might come from the sufficient inductive bias and alignment with
the user intentions in instruction-finetuned LLMs, and WebGUM could fully
leverage them for web automation problems. The margin of improvement might be
smaller than expected due to the limited capacity of transformer to obtain the
grounded understanding of natural language instructions, HTML, and
screenshots. In fact, the results also reveal that scaling the models might be
more important than the dataset; the low-capacity model may cap the
performance at a lower level.

Pre-Trained Models | Modality | Dataset | Success Rate
---|---|---|---
T5-XL [31] | HTML | 12K | 48.4%
T5-XL | HTML | 347K | 49.8%
Flan-T5-Base | HTML | 2.8K | 55.7%
Flan-T5-Base | HTML | 68K | 56.3%
Flan-T5-Base | HTML | 347K | 57.2%
Flan-T5-Base, ViT-B16 | HTML+Image | 2.8K | 61.1%
Flan-T5-Base, ViT-B16 | HTML+Image | 68K | 62.3%
Flan-T5-Base, ViT-B16 | HTML+Image | 347K | 66.1%

Table 6:  Average success rate of WebGUM with different dataset sizes. We
observe the larger the dataset size is, the higher the success rate is.
Surprisingly, our approach outperforms previous state-of-the-art by over 7.3%
even with 2.8K-episode dataset (about 25% of the previous dataset curated by
Liu et al. [55]).

Pre-Trained Models | # of Params | Modality | Success Rate
---|---|---|---
Flan-T5-Base | 220M | HTML | 57.2%
Flan-T5-Large | 770M | HTML | 72.4%
Flan-T5-XL | 3B | HTML | 75.5%
Flan-T5-XXL | 11B | HTML | 79.0%
Flan-T5-Base, ViT-B16 | 310M | HTML+Image | 66.1%
Flan-T5-Large, ViT-B16 | 860M | HTML+Image | 77.4%
Flan-T5-XL, ViT-B16 | 3B | HTML+Image | 80.3%

Table 7:  Average success rate of WebGUM with different model sizes. As for
both HTML-only and multimodal models, we could observe the performance
increases as the model size does.

##  Appendix D Dataset Details

To construct a large-scale multimodal behavioral dataset on MiniWoB++, we
leverage a public finetuned-LLM policy [31] trained with multi-task human
demonstration dataset [55]777https://github.com/stanfordnlp/miniwob-plusplus-
demos as a demonstrator. We run such LLM policies with 10,000 episodes per
task and only keep successful trajectories to maintain the quality of dataset,
following Humphreys et al. [36]. In addition, to collect expert demonstrations
on a harder task that finetuned-LLM struggles to solve, we write a scripted
policy for book-flight task. Table 8 shows the details of our multimodal
dataset, consisting of HTML, screenshots, actions, and instructions at each
time step.

Task | # of episodes | # of steps | Ratio (episode)
---|---|---|---
book-flight | 9999 | 90177 | 2.88%
choose-date | 383 | 1508 | 0.11%
choose-date-easy | 3353 | 12946 | 0.97%
choose-date-medium | 2222 | 8733 | 0.64%
choose-list | 1861 | 3724 | 0.54%
click-button | 9782 | 9909 | 2.82%
click-button-sequence | 10000 | 20000 | 2.88%
click-checkboxes | 9761 | 28904 | 2.81%
click-checkboxes-large | 1962 | 19072 | 0.57%
click-checkboxes-soft | 9228 | 36384 | 2.66%
click-checkboxes-transfer | 10000 | 59793 | 2.88%
click-collapsible | 5947 | 13077 | 1.71%
click-collapsible-2 | 2199 | 5627 | 0.63%
click-color | 2554 | 2554 | 0.74%
click-dialog | 10000 | 10000 | 2.88%
click-dialog-2 | 3285 | 3285 | 0.95%
click-link | 9961 | 9961 | 2.87%
click-menu | 3238 | 3243 | 0.93%
click-option | 9998 | 20000 | 2.88%
click-pie | 3724 | 8548 | 1.07%
click-scroll-list | 0 | 0 | 0.00%
click-shades | 0 | 0 | 0.00%
click-shape | 6116 | 6117 | 1.76%
click-tab | 9978 | 13177 | 2.88%
click-tab-2 | 1844 | 2109 | 0.53%
click-tab-2-hard | 1574 | 1916 | 0.45%
click-test | 10000 | 10000 | 2.88%
click-test-2 | 10000 | 10000 | 2.88%
click-widget | 9963 | 9963 | 2.87%
count-shape | 5849 | 5893 | 1.69%
email-inbox | 5159 | 14258 | 1.49%
email-inbox-forward-nl | 9995 | 39980 | 2.88%
email-inbox-forward-nl-turk | 4900 | 20165 | 1.41%
email-inbox-nl-turk | 4346 | 11416 | 1.25%
enter-date | 10000 | 20000 | 2.88%
enter-password | 9980 | 29940 | 2.88%
enter-text | 10000 | 20000 | 2.88%
enter-text-dynamic | 9983 | 19966 | 2.88%
enter-time | 0 | 0 | 0.00%
focus-text | 10000 | 10000 | 2.88%
focus-text-2 | 10000 | 10000 | 2.88%
grid-coordinate | 8353 | 8353 | 2.41%
guess-number | 1021 | 2042 | 0.29%
identify-shape | 9007 | 9010 | 2.60%
login-user | 9793 | 29379 | 2.82%
login-user-popup | 9786 | 39170 | 2.82%
multi-layouts | 10000 | 40000 | 2.88%
multi-orderings | 10000 | 40000 | 2.88%
navigate-tree | 9864 | 15140 | 2.84%
search-engine | 8872 | 35095 | 2.56%
social-media | 2631 | 4407 | 0.76&
social-media-all | 95 | 208 | 0.03%
social-media-some | 319 | 893 | 0.09&
tic-tac-toe | 3947 | 13773 | 1.14%
use-autocomplete | 3465 | 6930 | 1.00%
use-spinner | 530 | 532 | 0.15%
Total | 346827 | 867277 | 100%

Table 8: Details of our multimodal dataset. It contains about 347K episodes in
total.

##  Appendix E Per-Task Performance of MiniWoB++

In this section, we present per-task success rate on MiniWoB++, 56 tasks
(Table 10) and absolute performance improvement by adding image modality to
HTML input for WebGUM (Figure 7).

As for Table 10, we refer to Gur et al. [31] and Humphreys et al. [36] for the
baseline performance. We use 56 tasks as benchmark, while removing some
duplicated tasks (e.g. “-nodelay” tasks) from 62 tasks adopted in Gur et al.
[31], which might cause slight difference between the performance presented in
this paper and one reported in prior works. During the evaluation on
MiniWoB++, we ignore the time limit due to the computational constraints.

Figure 7 presents full results of the absolute performance improvement,
subtracting the success rates: (Success Rate of WebGUM(HTML+Image)) - (Success
Rate of WebGUM(HTML)). The results suggest WebGUM leverages visual inputs for
multi-step tasks with dynamic page transitions (e.g. book-flight or search-
engine) or the tasks that require global contexts of the page (e.g. tic-tac-
toe or click-shape). See Appendix H for the visualization.

Figure 7:  Performance improvement by adding image modality to HTML on 56
tasks from MiniWoB++. We subtract the success rates: (Success Rate of
WebGUM(HTML+Image)) - (Success Rate of WebGUM(HTML)).

Task | WebGUM (HTML) | WebGUM (HTML+Image) | WebN-T5 | WGE | CC-Net (SL) | CC-Net (SL&RL)
---|---|---|---|---|---|---
book-flight | 0.48 | 0.98 | 0.00 | 0.00 | 0.00 | 0.87
choose-date | 0.08 | 0.13 | 0.00 | 0.00 | 0.12 | 0.97
choose-date-easy | 1.00 | 1.00 | 0.03 | – | 0.42 | 0.99
choose-date-medium | 0.57 | 0.60 | 0.00 | – | 0.26 | 0.99
choose-list | 0.16 | 0.24 | 0.26 | 0.16 | 0.19 | 0.99
click-button | 0.98 | 1.00 | 1.00 | 1.00 | 0.78 | 1.00
click-button-sequence | 1.00 | 1.00 | 1.00 | 0.99 | 0.47 | 1.00
click-checkboxes | 1.00 | 1.00 | 0.96 | 0.98 | 0.32 | 0.98
click-checkboxes-large | 0.98 | 0.99 | 0.22 | 0.68 | 0.00 | 0.71
click-checkboxes-soft | 1.00 | 0.98 | 0.54 | 0.51 | 0.04 | 0.95
click-checkboxes-transfer | 0.99 | 0.99 | 0.63 | 0.64 | 0.36 | 0.99
click-collapsible | 1.00 | 0.98 | 0.00 | 1.00 | 0.81 | 1.00
click-collapsible-2 | 0.94 | 0.95 | 0.00 | 0.65 | 0.17 | 0.98
click-color | 0.27 | 0.34 | 0.27 | 1.00 | 0.82 | 1.00
click-dialog | 1.00 | 1.00 | 1.00 | 1.00 | 0.95 | 1.00
click-dialog-2 | 0.34 | 0.43 | 0.24 | 1.00 | 0.88 | 1.00
click-link | 1.00 | 1.00 | 1.00 | 1.00 | 0.59 | 0.99
click-menu | 0.41 | 0.37 | 0.37 | – | 0.22 | 0.94
click-option | 1.00 | 1.00 | 0.37 | 1.00 | 0.21 | 0.99
click-pie | 0.99 | 0.99 | 0.51 | 0.32 | 0.15 | 0.97
click-scroll-list | 0.00 | 0.00 | 0.00 | – | 0.01 | 0.60
click-shades | 0.00 | 0.00 | 0.00 | 0.22 | 0.04 | 1.00
click-shape | 0.58 | 0.72 | 0.53 | 0.64 | 0.11 | 0.95
click-tab | 1.00 | 1.00 | 0.74 | 0.55 | 0.95 | 1.00
click-tab-2 | 0.94 | 0.95 | 0.18 | 0.64 | 0.27 | 0.98
click-tab-2-hard | 0.57 | 0.95 | 0.12 | – | 0.19 | 0.98
click-test | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00
click-test-2 | 1.00 | 1.00 | 1.00 | 1.00 | 0.95 | 1.00
click-widget | 1.00 | 1.00 | 1.00 | 0.93 | 0.56 | 1.00
count-shape | 0.64 | 0.68 | 0.41 | 0.59 | 0.21 | 0.85
email-inbox | 0.99 | 0.99 | 0.38 | 0.43 | 0.09 | 1.00
email-inbox-forward-nl | 1.00 | 1.00 | 0.60 | – | 0.00 | 1.00
email-inbox-forward-nl-turk | 1.00 | 1.00 | 0.33 | – | 0.00 | 1.00
email-inbox-nl-turk | 0.92 | 0.98 | 0.23 | 0.77 | 0.05 | 1.00
enter-date | 1.00 | 1.00 | 0.00 | 0.00 | 0.02 | 1.00
enter-password | 1.00 | 1.00 | 0.97 | 0.99 | 0.02 | 1.00
enter-text | 1.00 | 1.00 | 0.89 | 1.00 | 0.35 | 1.00
enter-text-dynamic | 1.00 | 1.00 | 0.98 | 1.00 | 0.39 | 1.00
enter-time | 0.00 | 0.00 | 0.00 | 0.52 | 0.04 | 0.97
focus-text | 1.00 | 1.00 | 1.00 | 1.00 | 0.99 | 1.00
focus-text-2 | 1.00 | 1.00 | 1.00 | 1.00 | 0.96 | 1.00
grid-coordinate | 1.00 | 1.00 | 0.49 | 1.00 | 0.66 | 1.00
guess-number | 0.10 | 0.11 | 0.00 | 0.00 | 0.21 | 1.00
identify-shape | 0.90 | 1.00 | 0.88 | 0.90 | 0.68 | 1.00
login-user | 1.00 | 1.00 | 0.82 | 0.99 | 0.00 | 1.00
login-user-popup | 0.97 | 0.99 | 0.72 | – | 0.02 | 1.00
multi-layouts | 1.00 | 1.00 | 0.83 | 0.99 | 0.00 | 1.00
multi-orderings | 1.00 | 1.00 | 0.88 | 0.99 | 0.00 | 1.00
navigate-tree | 1.00 | 1.00 | 0.91 | 0.99 | 0.32 | 0.99
search-engine | 0.59 | 0.96 | 0.34 | 0.26 | 0.15 | 1.00
social-media | 0.99 | 1.00 | 0.21 | 0.39 | 0.03 | 0.90
social-media-all | 0.09 | 0.31 | 0.00 | 0.01 | 0.00 | 0.75
social-media-some | 0.39 | 0.68 | 0.02 | 0.01 | 0.01 | 0.85
tic-tac-toe | 0.42 | 0.56 | 0.48 | 0.37 | 0.32 | 0.83
use-autocomplete | 0.98 | 0.98 | 0.22 | 0.78 | 0.07 | 1.00
use-spinner | 0.03 | 0.11 | 0.07 | 0.04 | 0.47 | 1.00
Ave. | 0.755 | 0.803 | 0.484 | 0.646 | 0.343 | 0.964
# of Tasks | 56 | 56 | 56 | 48 | 56 | 56

Table 9: Per-task average success rate on 56 tasks from MiniWoB++. Because we
omit some duplicated tasks (e.g. “-nodelay” tasks) from 62 tasks adopted in
Gur et al. [31], we recalculate the baseline performances referring Humphreys
et al. [36] and Gur et al. [31].

##  Appendix F Compositional Evaluation on MiniWoB++

For the compositional evaluation, we pick up 4 click-“something” (link,
button, checkboxes, dialog) tasks and make some combinations of those by
naively stitching with 2 or 3 tasks. Then, we prepare the following 6
combinational tasks,

  * •

click-button_click-checkboxes

  * •

click-button_click-dialog

  * •

click-button_click-link

  * •

click-link_click-button

  * •

click-link_click-button_click-dialog

  * •

click-link_click-dialog

These tasks should be resolved in order of the name: for instance, in click-
link_click-button_click-dialog task, the agent should click the proper link,
click the proper button, click the proper dialog, and then the task results in
success. In click-button_click-link task, the agent should click the proper
button, and then click the proper link. The instructions for compositional
tasks are also simply combined among original task instructions in order of
the name. This evaluation could test the ability to transfer primitive skills
to control computers to solve unseen tasks. Table 10 shows the per-task
average success rate among 6 combinations above. WebGUM can solve the
compositional tasks much better than baseline [31].

Compositional Task |  WebN-T5 [31] | WebGUM (HTML) | WebGUM (HTML+Image)
---|---|---|---
click-button_click-checkboxes | 0.26 | 0.21 | 0.27
click-button_click-dialog | 0.95 | 0.72 | 0.87
click-button_click-link | 0.87 | 0.51 | 0.83
click-link_click-button | 0.35 | 0.95 | 0.90
click-link_click-button_click-dialog | 0.08 | 0.73 | 0.73
click-link_click-dialog | 0.55 | 0.95 | 0.93
Ave. | 0.510 | 0.678 | 0.755

Table 10: Per-task average success rate on 6 tasks from compositional
MiniWoB++.

##  Appendix G Evaluation on WebShop

In addition to MiniWoB++, we extensively evaluate our WebGUM on WebShop [86]
benchmark, another online-shopping websites simulator with a large amount of
real-world product data. WebShop provides user instruction that describes the
feature of items (e.g. I need a long clip-in hair extension which is natural
looking, and price lower than 20.00 dollars). The agents should search,
compare and choose a proper product that matches the given instruction. Since
WebShop requires complex multi-step reasoning considering previous contexts
for comparison [86, 87], we can test the capability of instruction-finetuned
LLM in decision making tasks in depth. The performance score is evaluated by
the percentage of required attributes covered by the chosen product (from 0 to
100), and if the product meets all the requirements, that episode is labeled a
success.

Because WebShop does not have API to get the screenshot of rendered websites,
we focus on WebGUM with text inputs, parsed from noisy HTML in the real
world.888WebShop just provides visual features of item pictures when the
agents reach the product page. These features are extracted by ResNet-50 [32],
rather than raw images or screenshots of the website. Some baseline agents (IL
and IL+RL) incorporate such embeddings. We convert the actions from raw texts
(e.g. search[a long clip-in hair extension] or click[<item id>]) to
dictionary-like format (e.g. {"action": "search", "ref": "a long clip-in hair
extension"} or {"action": "click", "ref": "<item id>"}), as we use in
MiniWoB++, to improve the prediction accuracy. We finetune Flan-T5-XL with
about 1K human demonstrations curated by Yao et al.
[86]999https://github.com/princeton-
nlp/WebShop/tree/master/baseline_models/data, using only high-score
demonstrations. The score threshold is $\texttt{score}\geq 50$ and we have 840
episodes in total (Table 12). We construct the model input with action
history, instruction, and text observation, the same as MiniWoB++ experiments.
We evaluate our method with 500 user instructions in the test set.

Table 11 shows that WebGUM achieves 45.0% success, significantly outperforming
not only simple baselines, such as supervised imitation learning (IL) and IL
plus RL-finetuing (by more than 15%), but also recent prompt-based LLM agents,
including ReAct [87] (i.e. PaLM-540B [16] with one-shot prompt and reasoning
annotations), while our model only has 3 billion parameters. IL and IL plus
RL-finetuning baselines use BART [48] model for the search policy, and BERT
[20] model for the click policy. The better performance of WebGUM proves the
hypothesis that the ability of multi-step reasoning in instruction-finetuned
language models works as a prior for decision making problems.

Methods | Training | Model | Modality | Score | Success Rate
---|---|---|---|---|---
Rule | – | – | Text | 45.6 | 9.6%
IL | SL | BART, BERT | Text(+Image) | 59.9 | 29.1%
IL+RL | SL+RL | BART, BERT | Text(+Image) | 62.4 | 28.7%
Act | In-context | PaLM-540B | Text | 62.3 | 30.1%
ReAct | In-context | PaLM-540B | Text | 66.6 | 40.0%
WebN-T5 | SL | T5-XL | Text | 61.0 | 29.8%
WebGUM | SL | Flan-T5-XL | Text | 67.5 | 45.0%
Human | – | – | Text+Image | 82.1 | 59.6%
Table 11: Average score and success rate on WebShop [86] benchmark. WebGUM based on Flan-T5-XL achieves 45.0% success, outperforming most baseline approaches including ReAct, a prompted PaLM-540B with reasoning annotations. We refer Yao et al. [87] for the baselines.  Threshold | # of Episodes | Score | Success Rate
---|---|---|---
$\texttt{score}\geq 0$ | 1026 | 67.2 | 44.4%
$\texttt{score}\geq 50$ | 840 | 67.5 | 45.0%
$\texttt{score}=100$ | 497 | 65.3 | 44.4%
Table 12: Average score and success rate on WebShop with different score
thresholds. Because we should balance the dataset coverage and proficiency, we
choose 50 as a threshold.

##  Appendix H Example Episodes of WebGUM

Figure 8:  Example of successful episodes demonstrated by multimodal WebGUM on
MiniWoB++ [70, 55]. The time step goes from left to right. As discussed in
Section 5.1, image modality seems to be leveraged for multi-step tasks with
dynamic page transitions (e.g. search-engine) or tasks that require global
visual contexts (e.g. tic-tac-toe).

Instruction: i am looking for dark denim color ethylene |  Instruction: i would like a xlarge plus red camellia fleece
---|---
vinyl ultra train of size 10, 3rd generation for men, | jacket that can be machine washed, and price lower
and price lower than 160.00 dollars | than 130.00 dollars
Action: search[dark denim color ethylene vinyl ultra |  Action: search[xlarge plus red camellia fleece jacket that
train of size 10, 3rd generation for men] |  can be machine washed]
Observation: | Observation:
[back to search] page 1 (total results: 50) [next] |  [back to search] page 1 (total results: 50) [next]
[b08b1mcfkl] | [b09h2qsj5p]
salewa ultra train 3 - men’s | men’s thermal underwear pants usb heated warm cotton
$104.0 to $182.99 | with zone 3 heating long trousers leggings bottoms
[b07j5mc2x1] | $56.77
skechers men’s low-top trainers, us men | [b09hcddf1x]
$53.39 to $140.35 | men’s thermal underwear pants usb heated warm cotton
[b07x5hr3tz] | long trousers leggings bottoms for winter washable baselayer
bruno marc men’s linen canvas stretch loafer shoes | $56.77 to $56.99
$39.99 | [b09jgmr43j]
| premium soft and smooth faux fur loose plush vest jackets
Action: click[b08b1mcfkl] | fashion sexy fuzzy sleeveless outwear
Observation: | $9.47 to $12.47
[back to search] [prev] |
size [7][7.5][8][8.5][9][9.5][10][10.5][11][11.5] |  Action: click[next]
[12][12.5][13] | Observation:
color [ombre blue | red orange][raw green | black out] |  [back to search] page 2 (total results: 50) [prev] [next]
[dark denim | red orange] |
salewa ultra train 3 - men’s | [b07cqt8rbl]
price: $104.0 to $182.99 | king and queen write hoodie couple matching sweater
rating: n.a. | pullover hooded sweatshirt jacket
[description] [features] [reviews] [buy now] | $26.99 to $29.99
| [b09jmvfpzg]
Action: click[dark denim | red orange] | mens sherpa quilted lined flannel jacket plus size plaid
Observation: you have clicked dark denim | red orange. | button down fleece shirts coats thicken warm
| cotton outwear
Action: click[10] | $36.79 to $55.99
Observation: you have clicked 10. | [b09kxfgbn2]
| mankalvxy womens jacket cardigan flannel cowl neck
Action: click[buy now] | asymmetric loose outerwear plus size xs-xxl 0-22
Termination: Your score (min 0.0, max 1.0): 1.0 | $20.3 to $26.85
|  Action: click[b07cqt8rbl]
| Observation:
|  [back to search] [prev]
| size [king-small][queen-small][king-medium]
| [queen-medium][king-large][queen-large][king-xxl]
| [king-xlarge][queen-xlarge][queen-xxlarge]
| king and queen write hoodie couple matching sweater
| pullover hooded sweatshirt jacket
| price: $26.99 to $29.99
| rating: n.a.
|  [description] [features] [reviews] [buy now]
|  Action: click[prev]
| Observation:
|  [back to search] page 2 (total results: 50) [prev] [next]
| [b07cqt8rbl]
| king and …
| [b09jmvfpzg]
| mens sherpa …
| [b09kxfgbn2]
| mankalvxy womens…
|  Action: click[b09jmvfpzg]
| Observation:
|  [back to search] [prev]
| color [army green][black][coffee][gray][green][red]
| [a-army green][a-red]
| size [small][medium][large][x-large][xx-large]
| [3x-large]
| mens sherpa quilted lined flannel jacket plus size plaid button
| down fleece shirts coats thicken warm cotton outwear
| price: $36.79 to $55.99
| rating: n.a.
|  [description] [features] [reviews] [buy now]
|  Action: click[red]
|  Observation: you have clicked red.
|  Action: click[x-large]
|  Observation: you have clicked x-large.
|  Action: click[buy now]
|  Termination: Your score (min 0.0, max 1.0): 1.0

Table 13: Example of successful episodes demonstrated by WebGUM on WebShop
[86]. WebGUM could search and choose the required product and choose proper
options (left). In addition, WebGUM could also compare the products with
browsing and backtracking (i.e. clicking “next” or “prev” buttons) during the
episodes (right).