Diverse evaluation benchmarks play a crucial role to assess a wide
range of capabilities of large language models (LLM). Although plenty of
endeavors have been dedicated
to building valuable benchmarks, there is still little work aiming at evaluating
the capability of LLM in multistep interactive environments.
Noticing that LLM requires a text representation of the environment observations
for interaction, we choose to fill such a blank by building a novel benchmark
based on the information user interface (InfoUI). InfoUI consists of rich text contents
and can be represented in some text formats, thus is suitable for the assessment of
interaction ability of LLM. Additionally, the complex structures of InfoUI can
further raise a challenge for LLM to understand structured texts rather than plain texts.

An interaction platform is always used to evaluate an agent, however, there is still
a lack of a satisfactory interaction platform dedicated to InfoUI.
Consequently, we propose to build a novel easily-extendable,
adaptable, and close-to-reality interaction platform, Mobile-Env, to provide a
base for an appropriate benchmark.

Based on Mobile-Env, an InfoUI task set WikiHow is then built to establish
a benchmark for the multistep interaction capability of LLM in structured
text-based environments. Agents based on a series of LLMs
are tested on the task set to obtain an insight into the potential and challenge
of LLM for InfoUI interaction.

It is sincerely welcome that the community contribute new environments and
new task sets for Mobile-Env to provide better test benchmarks and facilitate
the development of the corresponding domains.

§ INTRODUCTION

Along with the remarkable advancement of large language models
(LLM) <cit.>,
LLM evaluation benchmark is emerging as an important topic to assess the incredible
capability and intelligence of LLM. Numbers of benchmarks have been proposed
to assess the capacity of LLM for
reasoning <cit.>,
mathematics <cit.>,
and coding <cit.>.
Moreover, some
work <cit.>
combines a wide range of natural language processing (NLP) tasks and
even human exams to conduct a comprehensive evaluation. Nevertheless, there
is still a shortage of benchmarks to assess the capability of LLM to interact
with RL (Reinforcement Learning) environments. Different from the conventional
interaction models, LLM requires a text representation of the environment
observations. Thus, the existing interaction benchmarks based on embodied or visual-based
environments <cit.>
are inapplicable to LLM assessment.
Aiming at filling in such a blank, we choose to build
a benchmark based on the information user interfaces (InfoUI). InfoUI comprises rich
text information and can be conveyed in some text formats, thus is highly suitable for
the assessment of interaction ability of LLM. Besides, compared to pure text-based
environments like ALFWorld <cit.>, InfoUI consists
of complex structures as well. Hence, two challenges can be highlighted for LLM on
InfoUI: (1) multistep interaction (2) in structured text-based environments.
(See the comparison in Fig. <ref>.)

Embodied simulation environment

Visual game UI

Text-based game interface

Information UI

[]Comparison of
different types of interaction environments. Compared to the
environments consisting of pure visual-based or text-based
information, InfoUI comprises both rich texts and complex spatial
structures, and constitutes an appropriate challenge to LLM-based agents.
Fig. <ref>,
Fig. <ref>, and Fig. <ref> are extracted
from <cit.>,

<cit.>, and
<cit.> respectively.

An interaction platform is always adopted to evaluate the interaction capability
of an agent.
However, there is still a lack of a satisfactory interaction platform dedicated
to InfoUI to assess the capability of LLM or other agents.
Existing work like World of Bits <cit.>
and WebShop <cit.> suffers from the difficulty of
extending to new environments or new tasks, as well as the synthetic
simulated pages that are far different from real-world UIs. A comparison can be
referred to in Tab. <ref>. A satisfactory interaction platform is
expected to have the following features: (1) can be easily extended with new
task goals, (2) is adaptable to a variety of environments (i.e.,
different apps or websites) to help to evaluate an agent comprehensively,
(3) is close
to real-world scenarios to assess the most practical capability. To achieve
task extensibility, the task definitions
should be decoupled from the program internals. A universal interaction
interface and multi-class event sources should be enabled to achieve
environment adaptability and build environments close to real problems.

In order to offer a suitable benchmark for assessment of the InfoUI interaction
capability of LLM, we design
an InfoUI interaction platform Mobile-Env with the aforementioned
features in this paper. To be specific, we separate the task definition from the program
internals into an
external task definition file, and build our platform upon
Android ™ operating system (OS), which offers
millions of apps that can serve as diverse environments. Multiple
types of OS feedback are exploited to parse the episode signals (i.e., step
instructions, rewards, and the episode end) to further improve the adaptability
of the platform to in-the-wild apps.
Thus, large amounts of real-world apps can be exploited beyond the
customized dedicated apps to provide the closest-to-reality environments.
In this way, Mobile-Env offers a universal
interface for interaction with different Android app environments, enabling an
opportunity for comprehensive evaluation of LLM-based agents. Training and evaluating
conventional RL agents are supported as well.
Furthermore, an InfoUI task set based on WikiHow app is crafted to demonstrate
the usage of Mobile-Env and constitute a preliminary benchmark for the InfoUI
interaction capability of LLM. Finally, we tested the agents based on a series
of LLMs (i.e., ChatGLM-6B <cit.>,
LLaMA-13B <cit.>,
GPT-3.5 <cit.>, and
ChatGPT <cit.>)on
WikiHow task set to evaluate the potential and challenge of LLM for multistep
interaction in structured text-based environments.

Our contributions are summarized as follows: (1) A novel InfoUI interaction platform
Mobile-Env is designed. Mobile-Env is designed to be
easily-extendable, highly adaptable, and close to real-world scenarios to provide
a universal and flexible evaluation for the interaction capability of LLMs or
other agents. (2) A novel InfoUI task set WikiHow is built upon Mobile-Env to
establish a benchmark for assessment of InfoUI interaction capability of LLMs.
(3) A series of LLMs are tested on the proposed WikiHow task set to obtain an
insight into the potential and challenge of LLM for multistep interaction in structured
text-based environments.
[Mobile-Env code repository: <https://github.com/X-LANCE/Mobile-Env>.
Task set repository: <https://huggingface.co/datasets/zdy023/WikiHow-taskset>.]

§ RELATED WORK

Comparison of Mobile-Env with existing UI-based interaction
platforms. The table tells about whether the platform is designed for
InfoUI interaction, whether a new task can be added through an external
definition file, the mechanism of triggering episode signals, and whether
the environments are based on real pages or apps.

Platform      InfoUI        Task via Def.     Episode Signal⁎	    Real Page/App

World of Bits <cit.>       51      55     Internal	    55

WebShop <cit.>       51      51     Internal	    55

AndroidEnv <cit.>        55      51     Log    	55

Mobile-Env (Ours)      51      51     Multi	    51

⁎ “Internal” indicates that the episode signals
are generated by the hard-coded internal mechanism of the platform.
“Log” indicates parsing episode signals from
the OS runtime log (one type of available OS feedback).	“Multi” means
that the platform can take advantage of multiple types of OS feedback.

LLM evaluation benchmark As LLM has demonstrated remarkable
performance in a wide range of NLP tasks, various benchmarks
are proposed to assess the ability and intelligence of LLM comprehensively and deeply.
Reasoning is considered to be a pivotal mark of the intelligence and constitutes
one of the most outstanding features of LLM compared to the conventional models. Thus,
Most work focuses on the assessment of LLM's reasoning ability.
<cit.> proposes CoTHub to evaluate the capacity of LLM for
free-form reasoning with chains-of-thought (CoT). Other
work <cit.>
chooses to assess reasoning ability through mathematic problems.
Some work <cit.>
offers a benchmark for the capacity
of LLM for coding, which is always regarded as a typical logical activity. Further
work <cit.>
attempts to combine multiple tasks and even human exams like the admission tests
and bar exams to establish a comprehensive benchmark. Nevertheless, little
existing work investigates the assessment of the interaction capability of LLM. We argue
that multistep interaction in structured text-based environments still constitutes
a challenge to LLM. Consequently,
to fill such a blank, this work aims at establishing an interactive platform
for the assessment of LLM
based on InfoUI interaction. Based on our platform, WikiHow
task set is crafted as an evaluation benchmark, and several insights are drawn
through a series of experiments.

Interaction platform for InfoUI There is already a little work for
the InfoUI interaction platform. Some typical platforms are compared in
Tab. <ref>.  Existing work usually suffers from simplified
simulated pages and is difficult to add a new task owing to the coupling of
task definitions with the program internals.
World of Bits (WoB) <cit.> is an early interaction platform for
UI interaction comprising numbers of single-page tasks,
which are built on synthetic simulated mobile web pages.
WebShop <cit.> designs multipage
tasks in a scene of a shopping website with handcrafted simulated pages.
Information of new products can be appended to extend new task goals.  However,
the action space designed by WebShop is dedicated to the shopping website and
will cause extra efforts to migrate it to other domains.
AndroidEnv <cit.> is capable of importing a new
task through an external definition file. However, AndroidEnv parses the episode
signals through the system log, which requires the app to print several
characteristic lines into the log just at the time an episode signal is triggered.
This feature
prevents most real-world apps from being used in AndroidEnv, and seriously
harms its adaptability. The proposed Mobile-Env is built based on AndroidEnv
and preserves the extensibility brought by the external task definition file.
types of operating system feedback are further adopted to parse the episode
signals to seize better adaptability and take advantage of voluminous real-world

LLM for UI understanding & interaction Prior work has investigated
the potential of LLM for UI understanding and interaction.
<cit.> applies LLM to four conversational
mobile screen understanding tasks and depicts both the potential and room for
improvement of the LLM on the screen tasks. ReAct <cit.> injects
reasoning into the interaction trajectory and achieves a promising performance.
<cit.> combines LLM with iterative self-improvement to complete
WoB tasks. This work aims at InfoUI interaction and evaluated a series of LLM-based
agents on Mobile-Env to acquire perspectives on
the potential and challenge of LLMs.

§ PLATFORM FEATURES

Main components of Mobile-Env environment. The simulator serves as
a proxy to the Android OS, offers observations and accepts actions from the
agent. Task managers manage the episode signals according to the loaded task definition
files. Multiple task managers can be
loaded during launch, however, only one can be active for interaction at one time.

§.§ Overview

Mobile-Env is designed as a universal interaction platform supporting
various environments and different tasks to support the evaluation of LLM-based or
conventional InfoUI agents. An environment is usually
referred to as a specific app running on Mobile-Env, enabling agents to interact
with it and accomplish the task goal. A task corresponds to a specific
goal and several configurations, which will be detailed later in
<ref>. Based on Mobile-Env, agent designers can train and evaluate
their agents with various environments and tasks, meanwhile, environment and task
designers are welcome to build new environments and tasks as well.

A runtime instance of Mobile-Env consists of two types of components: simulator and task manager, as shown in Fig. <ref>. The
simulator wraps Android
Emulator ™[<https://developer.android.com/about>]
and provides the agent with a suite of interaction interface with the Android
OS. The simulator predefines basic observation space and action space
and supports multiple kinds of wrappers to customize the interaction interface,
which will be elaborated on in <ref>. A task manager
is corresponding to a specific task goal. During the launch of Mobile-Env instance,
the task manager will be loaded by reading and parsing a task definition file.
Then, during the episode, the task manager will control the triggering of the episode
signals according to the loaded task definition. The task definition file will be
detailed in <ref>. Multiple task managers can be loaded during
launch to support multiple task goals that can be switched at runtime. But there will
be only one task manager active for the current episode at one time.

During interaction, the agent will receive the observation from the simulator
and the episode signals (i.e., step instructions, rewards, and the episode
from the active task manager. The action decision will then be made accordingly and
sent to the OS through the simulator.

Both continuous control and discrete control agents
can be evaluated on Mobile-Env with the help of different wrappers.  And both
image-based and text-based agents (e.g., LLM-based agents) are supported,
as the screenshots and the
view hierarchy XMLs are enabled as the observation simultaneously.

§.§ Observation & action spaces

Observation space The observation space of Mobile-Env comprises
four components: a screenshot, the screen orientation, a time delta, and an
XML-format view hierarchy (VH). The screenshot is an RGB image
captured from the gRPC[<https://grpc.io/>] interface of the
emulator. It conveys exactly the same information as the human perceives. The
time delta is the amount of time passed since the last fetch of
observation. Both screen orientation and time delta are proposed originally in
<cit.> to offer the agent some extra hints at the
environment state. The view hierarchy is a tree description of the Android screen,
just like the HTML DOM (Document Object Model) of the web page.

The properties of a VH element can provide helpful information like the
content, name,
and purpose of the element in many cases. Meanwhile, the text format of VH
enables the text-based agents (e.g., LLM-based agents)
to interact with Mobile-Env

Action space The action space of Mobile-Env comprises three types
of actions: , , and .  indicates
to touch a specific point on the screen, and  means to lift the
“finger” off from the screen. The coordinates of the touchpoint are measured
from the
top-left of the screen and are normalized to $[0, 1]$.  action will
input a token from a predefined vocabulary. Sub-word tokens are
supported as well. It is proposed by the previous
work <cit.> that it is too challenging for the
agent to learn to type texts through the screen
keyboard character by character. Consequently,  is implemented to
reduce the difficulty.

Wrappers for observation & action spaces Besides the basic
observation and action
spaces, a suite of wrappers is supported to customize the
observation and action spaces the agent deals with. For example, a
discretization wrapper will discretize the screen into grids to enable
agents with discrete action space, or a VH wrapper
can be used to convert the VH-element-wise interaction to the combination of
the basic actions so that the text-based agents (e.g., LLM-based
agents) can interact with Android OS through Mobile-Env.

§.§ Task definition

Demonstration of event triggering logic $f$ with diverse operating system
The triggering state $t$ of the event slot depends on a
combinatorial boolean logic of the matching states $s_i$ of multiple event sources.
Text,  VH,
and  Log indicate different types of OS feedback.

A task corresponds to a particular task goal and several configurations defined
in a definition file. The task definition file is supposed to contain the following

* How the environment should be set up and reset, e.g., to
install and launch a particular app on the device.

* A task description describing the task goal, e.g.,
“find me an article about how to use ChatGPT”.

* Step instructions and how the step instructions are triggered.

* Rewards and how the rewards are triggered.

* Episode end and how the episode end is triggered.

The step instructions, rewards, and episode end constitute the episode signals
the agent will have a perception to. These episode signals
will be triggered during
interaction after some crucial steps in accordance with the task definition file.
A so-called crucial step may be opening a
target page, scrolling to the correct area, etc. The step instruction
is proposed to provide some auxiliary information during the episode,
e.g., the detailed tour information for a ticket booking task.
Intermediate rewards can be defined to mitigate the sparsity of the single
reward at the episode end during training conventional RL agents.

The timing of the episode signals are defined in the task definition file.
To be specific,
the step instruction, reward, and episode end are referred to as predefined
event slots in the task definition file. The triggering state of an event
slot depends on the event sources defined in the task definition file.
An event source indicates a specific pattern for the OS feedback. Given a triggering
state of an event slot $t$, matching states of a group of event sources
$S = \{s_1, s_2, \dots, s_n\}$, the triggering logic can be formalized as

\begin{equation}
t = f(s_1, s_2, \dots, s_n),
\label{eqn:evt_trg}
\end{equation}

where $f$ is a boolean function defined in the task definition file.
Fig. <ref> offers an intuitive
illustration. Among all the types of available OS feedback,
the VH and the system log are supported internally, while the screen texts and icons
should be enabled with an external OCR system and icon recognition system.

§ WIKIHOW TASK SET

To establish a preliminary benchmark for the multistep interaction capability of LLM
in structured text-based environments,
a navigation task set based on WikiHow app is built. This section will elaborate
on the construction of WikiHow task set, including
app data collection in <ref> and task
definition generation in <ref>.

§.§ App data collection for WikiHow

WikiHow task set instructs the agent to follow the instructions and navigate
through the pages in the app. WikiHow app relies on varying web contents,
therefore, we need to crawl and store the app data to achieve a consistent
evaluation. The app data will be replayed at runtime to mimic the real scenario.

WikiHow app comprises two types of pages: static pages and dynamic pages. The
static pages are referred to as the pages independent of user
inputs[Static pages may contain contents that vary according to spatio-temporal
conditions as well, e.g., dynamic recommendations, statistics for visits,
and ratings. However, these contents do not depend on instant user inputs, and thus are
considered “static” resources.] such as the home page, articles,
author information, etc. The dynamic pages are the pages sensitive to
user inputs, such as the pages of search
results. App data for static pages are directly collected from WikiHow
website[<https://www.wikihow.com/Main-Page>]. As observed that the pages
in the app are different from the pages in the browser, we checked the HTTP
request headers of all the types of pages and mocked the app requests to obtain
the correct contents. Considering that the user always explores the app from
the home page, the crawler also starts from the home page and accesses the
links in breadth-first order. For simplicity, the links to the outside of
WikiHow are omitted. All the links including all types of media are requested
and stored for a perfect replay. Finally, 856,045 resources are dumped, in
which there are 107,448 distinct pages. These website resources totally occupy
about 88 GiB. In contrast to the static resources, it is nearly impossible to
traverse all the probable combinations of keywords to dump the pages of search
results.  Consequently, an open-source information retrieval engine,
Pyserini <cit.>, is adopted following
<cit.> to mimic the WikiHow search engine. Then the
synthetic search pages are built dynamically during the replay upon a handcrafted
template of the search result page on WikiHow.

§.§ Automatic task generation for WikiHow

It is noticed that most pages in WikiHow app can be categorized into 4 common
classes and 2 special pages. The 4 common classes are search result page,
article page, author information page, and category content page. The 2 special
pages are the category listing page and the about page. According to the
characteristics of these pages, a group of single-page task definition
templates is designed. Such a single-page task definition emits a single
instruction requesting the agent to access the target page. The page-specific
keywords such as the article title and author name are made into filling slots in the
templates. The templates can be instantiated with the real keywords sampled
from the dumped app data.
Subsequently, the single-page task definitions are randomly combined sequentially
under the constraint that guarantees the target page in the succeeding step
is directly
referenced on the page in the preceding step. The agent is expected to
follow the instructions and traverse the target pages sequentially. Through
this approach, about 5,000 navigation tasks are auto-generated, from which 178
tasks are sampled and validated manually to form a canonical set.

§ LLM-BASED AGENT & EXPERIMENT RESULTS

This section will elaborate on the design of the LLM-based agent, as well as the
experiment results of the agent based on different LLMs.

§.§ LLM-based agent

To assess the capability of LLM to conduct multistep interaction in structured text-based
environments, we design a simple in-context
learning (ICL) agent to complete WikiHow tasks. Two steps of interaction are selected
and annotated as the exemplars. The certain exemplars and full prompts are presented
in the supplementary.

Framework of the LLM-based agent. The LLM is fed with the task description
(prompted with “Task:”), the screen representation
(prompted with “Screen:”), the
step instruction (prompted with “Instruction:”), and
the action history (prompted with
“Action History:”). Then the LLM will decide a proper
action according to the input.

The input and output framework of the LLM-based agent is depicted in
Fig. <ref>. The
observation from Mobile-Env is fed into the LLM together with the task
description and a few episode information to decide a proper action for the current step.
To be specific,
the input comprises the task description, a screen representation, the current
step instruction, and the action history. The screen is represented in a really
simplified format, which is just a list of HTML elements inspired by
<cit.>. HTML is chosen rather than the
original view hierarchy out of the consideration that HTML codes are directly included
in the training corpora and are more familiar to LLM.  All the visible leaf VH
nodes are converted to HTML elements according to a mapping rule detailed
in the supplementary.

The  property of the VH nodes is converted to the text content
of the HTML elements
or the  attribute of  element. A few other properties
are converted as well according to a mapping rule detailed in the supplementary.

An additional integer attribute  is added for LLM to reference the element to

The LLM-based agent can take one of these element-wise interactions:

*  (Available directions are ,
, , .)

The chosen interaction will be translated into a sequence of basic Mobile-Env
actions (i.e., , , and  introduced
in <ref>) and be executed by the platform.

Results of the agents based on different LLMs on WikiHow task set.

“GPT-3.5” denotes OpenAI GPT-3.5 text-davinci-003. “ChatGPT” denotes
OpenAI GPT-3.5 gpt-3.5-turbo.

LLM       Model Size (#Parameters)       Avg #Steps       Avg Reward        Success Rate (%)

ChatGLM-6B        6B        11.76     1.59      29

LLaMA-13B     13B     9.11      2.04      51

GPT-3.5       175B     5.27        2.50      89

ChatGPT       Undisclosed       4.84      2.57      84

§.§ Interaction capability of different LLMs

We evaluated four popular LLMs on the proposed WikiHow task set:
ChatGLM-6B <cit.>, LLaMA-13B <cit.>,
GPT-3.5 <cit.>,
and ChatGPT <cit.>. ChatGLM-6B and LLaMA-13B
are accessed through the open-sourced weights, while GPT-3.5 and ChatGPT are accessed
through the online APIs[<https://openai.com/api/>] of OpenAI.
The experiments are conducted with a 2-shot in-context learning under sampling
temperature of 0.1. Owing to the limit of budget, only 70 tasks from the canonical
set of
WikiHow are evaluated for a more affordable cost. The trajectory will be
regarded as failed if the number of steps reaches 15, as the normal number of
interaction iterations is around 5 according to the advance human test. The
results are shown in Tab. <ref>.

It is clearly illustrated that
larger models achieve better performance and demonstrate stronger interaction
capability. Despite the outstanding performance of GPT-3.5 and ChatGPT, WikiHow still
constitutes a significant challenge for other LLMs.
ChatGPT achieves worse performance than GPT-3.5 by degradation of 5 points for the success
rate. This may be due
to that the task based on structured texts benefits less from continuous RLHF (Reinforcement
Learning from Human Feedback), which further aligns the model with natural human
Besides, several interesting phenomena are observed. ChatGLM-6B prefers
actions, while other models rarely predict . LLaMA-13B is more prone to
violate the output format and generate invalid predictions or free-form language
Several cases are provided in the supplementary.

§.§ Ablation studies

Results of ablation studies based on GPT-3.5 on WikiHow task set.
The table
depicts the result under the normal setting as well as those under three
ablation settings: results with VH input instead of HTML, without action
history input, and without HTML structural information.

Setting       Avg #Steps        Avg Reward        Success Rate (%)

GPT-3.5     5.27      2.50      89

w/ VH    	4.34	   	2.50    	86

w/o history        5.81      2.44      74

w/o HTML      14.99     0.03      1

Further ablation studies are conducted based on GPT-3.5 to draw deeper insights
into the potential and challenge of LLM for multistep interaction with structured
text-based InfoUI.
We conducted experiments with screen represented in view
hierarchy instead of HTML. The screen is represented as a sequence of the
visible leaf VH nodes.  For simplicity and to control the input length, only
the node properties used in HTML conversion are preserved and the other
redundant properties are ignored. This simplification leads to a fair
comparison to the result with HTML input. As depicted in Tab. <ref>,
the LLM-based agent with VH input achieves a success rate of 86%, which is lower
than the result with HTML input by 3 points. This
result is consistent with the viewpoint stated in <ref> that
the LLM is more familiar with HTML than with VH.

It is noticed that the LLM-based agent sometimes gets stuck at some steps and
repeats some wrong actions. Nevertheless, it may succeed in getting out of the
cycle and then making the correct decision. A case is
depicted in the supplementary.
The action
history input is believed to assist the agent in overcoming this difficulty.
The result of the ablation study by removing the action history input proves
this guess.  After removing the action history input, the agent fell into the
cycle of wrong actions more often, and it seems to be impossible for the agent
to get rid of it under this setting for the identical inputs. As shown in
Tab. <ref>, under the ablation setting, the average number of steps
increases from 5.27 to 5.81, while the average reward decreases from 2.50 to 2.44,
and a remarkable decline of 15 points is observed for the success rate.

Another ablation study is conducted to investigate the effect of the HTML
structural information. The HTML tag is destructed and the tag name, the values
of the concerned attributes, and the text content of an element are
concatenated as plain texts.  The results in Tab. <ref> depict that
with such a plain text representation, the LLM-based agent can hardly retrieve
any valid information and seldom manages to complete a task. This can be
attributed to that the HTML syntax presents the screen information in a
structured way, which helps the LLM-based agent to understand the screen
comprehensively.  Consequently, such structural information is crucial for
InfoUI interaction.

§ CONCLUSION

Aiming at providing a benchmark for the assessment of the multistep interaction
capability of LLM in structured text-based environments, a novel easily-extendable,
adaptable, and close to reality interaction platform,  Mobile-Env, is
designed in this work. Based on the platform, a task set based on WikiHow
app is crafted to establish a preliminary benchmark. Agents based on a series of
popular LLMs are evaluated on WikiHow. The experiment results depict the potential
of the LLMs for InfoUI interaction, the scaling rule, and the impact of RLHF, and
reveal several challenges for future work. We sincerely welcome that the community
contribute new environments and task sets for Mobile-Env to provide better benchmarks
and facilitate the development of the corresponding domains.

§ CHECKLIST

* For all authors...

* Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

* Did you describe the limitations of your work?
The limitations and future work are discussed in the supplementary.

* Did you discuss any potential negative societal impacts of your work?

* Have you read the ethics review guidelines and ensured that your paper conforms to them?

* If you are including theoretical results...

* Did you state the full set of assumptions of all theoretical results?

* Did you include complete proofs of all theoretical results?

* If you ran experiments (e.g. for benchmarks)...

* Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
The Mobile-Env code repository and WikiHow task set are open-sourced at GitHub and Hugging Face, respectively. The links are offered in the supplementary. The full prompts to reproduce the experiment results are represented in the supplementary as well.

* Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?

* Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
Duplicated experiments are not conducted owing to the limit of budget.

* Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
See the supplementary.

* If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...

* If your work uses existing assets, did you cite the creators?

* Did you mention the license of the assets?
The license is mentioned in the supplementary.

* Did you include any new assets either in the supplemental material or as a URL?
The links to the Mobile-Env code repository and WikiHow task set are provided in the supplementary.

* Did you discuss whether and how consent was obtained from people whose data you're using/curating?
See the datasheet in the supplementary.

* Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?
See the datasheet in the supplementary.

* If you used crowdsourcing or conducted research with human subjects...

* Did you include the full text of instructions given to participants and screenshots, if applicable?

* Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?

* Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?

§ OVERVIEW & AUTHOR STATEMENT

This work releases Mobile-Env platform and WikiHow task set under Apache-2.0
License. The authors bear all responsibility in case of violation of rights.
The codes and documents of Mobile-Env is available at
<https://github.com/X-LANCE/Mobile-Env>. WikiHow task set is made public at
<https://huggingface.co/datasets/zdy023/WikiHow-taskset>. Both platform and
task set will be preserved for a long time. The platform will be maintained and
developed continuously. Updates will be released from time to time. Errata and
correction will be announced if mistakes are found in WikiHow task set. We are
also planning to build more task sets and establish a website tracking and
indexing the task sets available for Mobile-Env.

This supplementary is organized in seven sections.
Sections <ref>, <ref>, <ref> will
introduce more details about Mobile-Env platform, WikiHow task set, and the
experiments with LLM-based agents, respectively. Following NeurIPS Datasets and
Benchmarks Track guidelines, Section <ref> will present the
datasheet according to the datasheet
format <cit.>, and
Section <ref> will present the main concerns raised by the
previous venue and our revision. Section <ref> will discuss the
limitations and future work.

§ MORE MOBILE-ENV DETAILS

Both code repository and detailed documents of Mobile-Env are hosted on GitHub.
This section will supplement several details about the design and usage of
Mobile-Env. Specifically, a brief usage guideline will be presented in
<ref>. Then we will introduce the problem about the compatibility
of apps on Android Emulator in <ref>, certificate pinning
problem of apps with varying contents in <ref>, and the
limitations of VH observation in <ref>. <Ref> will
further elaborate on the episode signal triggering mechanism. Finally, a few
miscellaneous auxiliary tools will be introduced in <ref>.

§.§ Usage guideline

§.§.§ Evaluation & training with Mobile-Env

Structure of observation object The basic observation object
consists of four components: screenshot in pixels, the screen orientation in
one-hot vector, time delta in microseconds, and VH as an XML tree object. The
detailed field names and shapes are depicted in Tab. <ref>.

Structure of action object The basic action object consists of
three (optional) components: action type, touch position (for
type), and token index (for ). The detailed field names and shapes
are depicted in Tab. <ref>.

Observation & action spaces customization with wrappers A suite of
wrappers are designed to alter the observation and action spaces to adapt to
different types of agents. A few typical wrappers are listed in Tab. <ref>.
The users can also define their own wrappers. Several hook methods are declared
in the base wrapper class. The designer just needs to inherit the base class
and implement necessary hooks. The key hooks are sketched in Tab. <ref>.

For the detailed usage, we refer the reader to the online documents as well as
the code repository.

Observation structure

Field    	Shape/Type    	Description

RGB screenshot.

One-hot vector for 4 different

scalar    	Time passed since the last observation
in microseconds.

object    	XML tree object of VH.

Action structure

Field    	Shape/Type    	Description

int enum    	, , or .

Touch position normalized to
$[0, 1]$ for .

scalar    	Token index for .

Typical wrappers

Wrapper    	Description

Gridizes the screen to enable discrete

Resizes the raw screen pixels.

Gymnasium-compatible <cit.> interfaces.

Provides VH-based interfaces for text-based

Key hooks

Hook    	Description

Invoked before resetting the environment or
switching task goals.

Invoked after switching task goals.

Alters the observation before returning it
to agent.

Alters the action before sending it to Android

§.§.§ Extending new environments and new tasks for Mobile-Env

To extend a new environment (i.e., a new app) for Mobile-Env, the
environment designer needs to prepare the app package and ensure that the
package manages to launch and run on some versions of Android Emulator. A brief
explanation of the app compatibility is given in <ref>. If
the app requires varying online data, the necessary data should be crawled and
dumped and then be replayed for a consistent evaluation. In such case, the
designer is supposed to validate the certain effectiveness of certificate
unpinning plan for the package. Certificate pinning problem is discussed in
<ref>.As regards to extend new tasks, task definition files are
just required. Detailed instructions for extending new environments and new
tasks are available in the online documents.

§.§ Compatibility of apps on Android Emulator

Most Android devices run on -arch hardware, while Android
Emulator usually runs on  infrastructure. Consequently, an
in-the-wild Android app package may be incompatible with the emulator. We
consulted the documents[<https://developer.android.com/about>] of
Android Emulator and conducted a series of tests accordingly. It is finally
affirmed that the image of Android 11 (API Level 30) offers the best
compatibility for -arch packages on  infrastructure.
Besides, a few third-party package stores are discovered. The details are
referred to in the online documents.

§.§ Certificate pinning problem

As lots of information apps relies on varying online contents, the environment
designer has to download and fix the contents in advance and replay them to the
app for a consistent evaluation. However, most apps adopted SSL for network
connections and give no trust to the user-added certificate of the MITM
(Man-in-the-Middle) proxy, which is required for SSL replay. Such a problem is
called certificate pinning. To handle this problem, we sought help from the
reverse engineers and validated three solutions: mocking system certificate,
runtime verifier replacement, and APK repackaging. It is found that neither of
these methods is able to deal with all the apps with certificate pinning,
however, most apps can be handled by at least one of them. The detailed
implementations are documented online.

§.§ Limitations of view hierarchy

Compared to the screenshot observation, view hierarchy suffers a few
limitations. The most crucial limitation of view hierarchy is the significant
overhead of acquisition. According to our test on a host with CPU of Intel Core
i7-10700 (16 logic cores of 4.8GHz) with KVM (Kernal-based Virtual Machine)
enabled, and 64 GB of memory, it takes about 2–3 seconds to acquire an XML
representation of VH through ADB (Android Debug Bridge), while it takes only 20
microseconds to acquire a screenshot with size of 1,080$\times$1,920 through
gRPC. Besides, the VH nodes may be incomplete in several sensitive apps,
e.g., the messaging and financial apps.

§.§ Details about episode signal triggering

In addition to the combinatorial logic form of boolean function $f$, we
introduce “prerequisite” mechanism to correctly handle the step order for
multistep tasks. To be specific, we call each node (including the event sources
at leaves and the  and  operators inside) on the tree
representation of $f$ in Figure 3 of the main paper a virtual event. If
virtual event $a$ is declared as the prerequisite of virtual event $b$ in
the task definition file, then event $b$ will not be triggered until event $a$
has already been triggered ever in one episode. For example, an order is
supposed to be placed ($b$) only after all the information fields have been
filled in ($a$), thus, information filling can be declared as the prerequisite
of order placing to guarantee such a constraint. After each interaction step
during episode, a virtual event will first check if its prerequisites are
satisfied and then decide its triggering flag according to the matching state
(for event sources) or the triggering flags of its child (for  and

§.§ Miscellaneous auxiliary tools

Annotation tool for human demonstration Touch and typing are close
to human behaviors and are suitable to apply behavior cloning approaches.
Based on Mobile-Env, an annotation tool is developed to collect human
demonstrations for convenience. The annotation tool offers a straightforward
way to collect the human actions. The annotators can directly click on the
screenshot to mock touching on the mobile screen, or click the corresponding
button to directly type a word from a shrunk vocabulary. The actions,
observations, and task information will be recorded for behavior cloning.

Task definition template toolkit A simple suite of tools is
developed to auto-generate task definition files from templates. Such a task
definition template declares a group of value slots. The value slots can be
instantiated through a simple configuration file.  Besides, the definition
instantiated from a single template can be combined sequentially to form a
“multistep” task definition. WikiHow task set is built with the help of this

§ DETAILS ABOUT WIKIHOW TASK SET

§.§ Data format & flow replay

The crawled data are marked with the requested URL paths. Each response is
dumped in an individual file. Both HTTP headers and raw payload are saved. The
flow data can be replayed through mitmproxy[<https:
//mitmproxy.org/>].  The saved static resources are directly read and replayed
with the timestamps refreshed. The search result page is crafted with the
results from Pyserini engine <cit.> and the page
template. Pyserini builds the indices of the articles from the article titles.
The page template is built from a real search result page from WikiHow website
by making the search result list a filling slot. All the crawled flow data, the
pyserini indices, and the replay script for mitmproxy are released together
with the task definitions on Hugging Face.

§ DETAILS ABOUT LLM-BASED AGENTS AND EXPERIMENTS

This section will introduce details about the implementation of the LLM-based
agents and the experiments. <Ref> and <ref>
will introduce the conversion of VH-based observation and action. The full
prompts used in the agents are presented in <ref>. Compute
resources are introduced in <ref>. <Ref> will give
some case studies.

§.§ Detailed mapping rule from VH to HTML

Tab. <ref> depicts how the different classes of VH nodes are
mapped to the HTML tags.  Tab. <ref> depicts how the VH
node properties are mapped to the HTML element attributes. An example of
conversion from VH to HTML is illustrated in Fig. <ref>.

Mapping from VH Node Class to HTML Tag

VH node class     HTML tag class

Mapping from VH Properties to HTML Attributes. The resource-id property usually consists of three parts: the package
name, the element class, and the element name. Only the element name is
used for the converted class attribute, which is following

VH property     HTML attribute

Example of conversion from VH to HTML. Fig. (a) depicts the raw
view hierarchy extracted from the OS. Fig. (b) depicts the visible leaf
nodes from Fig. (a). Additional  property is appended. Fig. (c)
depicts the converted HTML elements from Fig. (b) according the the conversion

§.§ Details about translation from VH-element-wise actions to basic actions

Sliding trajectory of  (sliding up)

Sliding trajectory of  (sliding left)

Demonstration of sliding trajectories translated from
. The starting point is at 20% of the screen height
(width), while the terminating point is at 80%.

Three types of VH-based actions (i.e., , , and
) are supposed to be translated into basic actions (i.e.,
, , and ) to be executed in Mobile-Env. In
practice, the translation is implemented in  introduced in
<ref>. Particularly,  is translated into a
sequence of s appended with a :

\begin{equation}
\mathtt{CLICK}(\mathtt{eid}) \mapsto \underbrace{\mathtt{TOUCH}(c), \cdots, \mathtt{TOUCH}(c)}_{\text{$n$ \texttt{TOUCH}s}}, \mathtt{LIFT}.
\label{eqn:click_trslt}
\end{equation}

Here  is the integer element id and $c$ is the center point of the
element bounding box. $n$ is set to 3 in our experiments.  is
translated into a , a sequence of s, and an keyboard
action “Enter”:

\begin{equation}
\mathtt{INPUT}(\mathtt{eid}, \mathtt{text}) \mapsto \mathtt{CLICK}(\mathtt{eid}), \mathtt{TEXT}(\mathtt{tkn}_1), \cdots, \mathtt{TEXT}(\mathtt{tkn}_2), \mbox{Enter},
\label{eqn:input_trslt}
\end{equation}

where $\mathtt{tkn}_i$ denotes the $i$-th token from the input . The
keyboard action “Enter” is a special action beyond the basic actions and
indicates typing a carriage return into the OS.  is translated
into $m$ consecutive s followed by a , where $m$ is set
to 10 in our experiments. These s form a consecutive sliding of the
“finger” on the screen.  Fig. <ref> demonstrates the
translated sliding trajectories.

§.§ Prompts for LLM-based agents

The prompts used for LLM-based agents are listed in this subsection. The
prompts for GPT-3.5 and ChatGPT is the full versions. The versions for
ChatGLM-6B and LLaMA-13B are shrunk to fit into the limit of input length, as
these two LLMs are pretrained to accept only 2,048 tokens.

to p
1cPrompt for GPT-3.5

Given a task desciption, a screen representation in simplified html, and an instruction sentence at the current step, I need to take an appropriate action according to the given information to finish the underlying task. Available actions are:

INPUT(element_id, text)

Usually I will click the correct link to access the willing contents or search or scroll down if it is not present on the current screen. For example, given the input:

Search an article to learn how to hide gauges.

Then, access the article "How to Hide Gauges"

<button alt="Open navigation drawer" id="0" clickable="true"></button>

<img class="wikihow toolbar logo" id="1" clickable="false">

<img class="search button" alt="Search" id="2" clickable="true">

<div class="webView" id="3" clickable="true"></div>

<div class="statusBarBackground" id="4" clickable="false"></div>

Action History:

I will answer as

INPUT(2, hide gauges)

Another yet example:

Search an article to learn how to do ruby rose hair.

Then, access the article "How to Do Ruby Rose Hair"

Then, access the about page to learn why people trust wikihow.

<button alt="Open navigation drawer" id="0" clickable="true"></button>

<input class="search src text" value="Do ruby rose hair " type="text" id="1" clickable="true">

<img class="search close btn" alt="Clear query" id="2" clickable="true">

<div id="3" clickable="true">How to Do Ruby Rose Hair</div>

<div id="4" clickable="true">• </div>

<p id="5" clickable="true">41,446 views</p>

<div id="6" clickable="true">• </div>

<p id="7" clickable="true">Updated</p>

<p id="8" clickable="true">2 years ago</p>

<div id="9" clickable="true">• </div>

<p id="10" clickable="true">Expert Co-Authored</p>

<div id="11" clickable="true">How to Dye Your Hair Rose Gold</div>

<div id="12" clickable="true">• </div>

<p id="13" clickable="true">48,548 views</p>

<div id="14" clickable="true">• </div>

<p id="15" clickable="true">Updated</p>

<p id="16" clickable="true">3 years ago</p>

<div id="17" clickable="true">• </div>

<p id="18" clickable="true">Expert Co-Authored</p>

<div id="19" clickable="true">How to Do a Hair Mask for Frizzy Hair</div>

<div id="20" clickable="true">• </div>

<p id="21" clickable="true">29,415 views</p>

<div id="22" clickable="true">• </div>

<p id="23" clickable="true">Updated</p>

<p id="24" clickable="true">2 years ago</p>

<div id="25" clickable="true">• </div>

<p id="26" clickable="true">Expert Co-Authored</p>

<div id="27" clickable="true">How to Do 1920s Hair</div>

<div id="28" clickable="true">• </div>

<p id="29" clickable="true">194,834 views</p>

<div id="30" clickable="true">• </div>

<p id="31" clickable="true">Updated</p>

<p id="32" clickable="true">1 year ago</p>

<div id="33" clickable="true">• </div>

<p id="34" clickable="true">Expert Co-Authored</p>

<div id="35" clickable="true">How to Do 50s Hair</div>

<div id="36" clickable="true">• </div>

<p id="37" clickable="true">39,890 views</p>

<div id="38" clickable="true">• </div>

<p id="39" clickable="true">Updated</p>

<p id="40" clickable="true">1 year ago</p>

<div id="41" clickable="true">• </div>

<p id="42" clickable="true">Expert Co-Authored</p>

<div id="43" clickable="true">How to Do 70s Hair</div>

<div id="44" clickable="true">• </div>

<p id="45" clickable="true">69,458 views</p>

<div id="46" clickable="true">• </div>

<p id="47" clickable="true">Updated</p>

<p id="48" clickable="true">1 year ago</p>

<div id="49" clickable="true">• </div>

<p id="50" clickable="true">Expert Co-Authored</p>

<div id="51" clickable="true">How to Do Goku Hair</div>

<div id="52" clickable="true">• </div>

<p id="53" clickable="true">44,049 views</p>

<div class="statusBarBackground" id="54" clickable="false"></div>

Access the article "How to Do Ruby Rose Hair"

Action History:

INPUT(2, do ruby rose hair)

The answer will be:

Now the new input:

to rp12.1cm
2cPrompt for ChatGPT

Given a task desciption, a screen representation in simplified html, and an instruction sentence at the current step, you need to take an appropriate action according to the given information to finish the underlying task. Available actions are:

INPUT(element_id, text)

Usually you will click the correct link to access the willing contents or search or scroll down if it is not present on the current screen.

Search an article to learn how to hide gauges.

Then, access the article "How to Hide Gauges"

<button alt="Open navigation drawer" id="0" clickable="true"></button>

<img class="wikihow toolbar logo" id="1" clickable="false">

<img class="search button" alt="Search" id="2" clickable="true">

<div class="webView" id="3" clickable="true"></div>

<div class="statusBarBackground" id="4" clickable="false"></div>

Action History:

INPUT(2, hide gauges)

Search an article to learn how to do ruby rose hair.

Then, access the article "How to Do Ruby Rose Hair"

Then, access the about page to learn why people trust wikihow.

<button alt="Open navigation drawer" id="0" clickable="true"></button>

<input class="search src text" value="Do ruby rose hair " type="text" id="1" clickable="true">

<img class="search close btn" alt="Clear query" id="2" clickable="true">

<div id="3" clickable="true">How to Do Ruby Rose Hair</div>

<div id="4" clickable="true">• </div>

<p id="5" clickable="true">41,446 views</p>

<div id="6" clickable="true">• </div>

<p id="7" clickable="true">Updated</p>

<p id="8" clickable="true">2 years ago</p>

<div id="9" clickable="true">• </div>

<p id="10" clickable="true">Expert Co-Authored</p>

<div id="11" clickable="true">How to Dye Your Hair Rose Gold</div>

<div id="12" clickable="true">• </div>

<p id="13" clickable="true">48,548 views</p>

<div id="14" clickable="true">• </div>

<p id="15" clickable="true">Updated</p>

<p id="16" clickable="true">3 years ago</p>

<div id="17" clickable="true">• </div>

<p id="18" clickable="true">Expert Co-Authored</p>

<div id="19" clickable="true">How to Do a Hair Mask for Frizzy Hair</div>

<div id="20" clickable="true">• </div>

<p id="21" clickable="true">29,415 views</p>

<div id="22" clickable="true">• </div>

<p id="23" clickable="true">Updated</p>

<p id="24" clickable="true">2 years ago</p>

<div id="25" clickable="true">• </div>

<p id="26" clickable="true">Expert Co-Authored</p>

<div id="27" clickable="true">How to Do 1920s Hair</div>

<div id="28" clickable="true">• </div>

<p id="29" clickable="true">194,834 views</p>

<div id="30" clickable="true">• </div>

<p id="31" clickable="true">Updated</p>

<p id="32" clickable="true">1 year ago</p>

<div id="33" clickable="true">• </div>

<p id="34" clickable="true">Expert Co-Authored</p>

<div id="35" clickable="true">How to Do 50s Hair</div>

<div id="36" clickable="true">• </div>

<p id="37" clickable="true">39,890 views</p>

<div id="38" clickable="true">• </div>

<p id="39" clickable="true">Updated</p>

<p id="40" clickable="true">1 year ago</p>

<div id="41" clickable="true">• </div>

<p id="42" clickable="true">Expert Co-Authored</p>

<div id="43" clickable="true">How to Do 70s Hair</div>

<div id="44" clickable="true">• </div>

<p id="45" clickable="true">69,458 views</p>

<div id="46" clickable="true">• </div>

<p id="47" clickable="true">Updated</p>

<p id="48" clickable="true">1 year ago</p>

<div id="49" clickable="true">• </div>

<p id="50" clickable="true">Expert Co-Authored</p>

<div id="51" clickable="true">How to Do Goku Hair</div>

<div id="52" clickable="true">• </div>

<p id="53" clickable="true">44,049 views</p>

<div class="statusBarBackground" id="54" clickable="false"></div>

Access the article "How to Do Ruby Rose Hair"

Action History:

INPUT(2, do ruby rose hair)

Access the article "How to Do Ruby Rose Hair"

Action History:

INPUT(2, do ruby rose hair)

to p
1cPrompt for LLaMA-13B

Given a task desciption, a screen representation in simplified html, and an instruction sentence at the current step, I need to take an appropriate action according to the given information to finish the underlying task. Available actions are:

INPUT(element_id, text)

Usually I will click the correct link to access the willing contents or search or scroll down if it is not present on the current screen.

Search an article to learn how to hide gauges.

Then, access the article "How to Hide Gauges"

<button alt="Open navigation drawer" id="0" clickable="true"></button>

<img class="wikihow toolbar logo" id="1" clickable="false">

<img class="search button" alt="Search" id="2" clickable="true">

<div class="webView" id="3" clickable="true"></div>

<div class="statusBarBackground" id="4" clickable="false"></div>

Action History:

INPUT(2, hide gauges)

Search an article to learn how to do ruby rose hair.

Then, access the article "How to Do Ruby Rose Hair"

Then, access the about page to learn why people trust wikihow.

<button alt="Open navigation drawer" id="0" clickable="true"></button>

<input class="search src text" value="Do ruby rose hair " type="text" id="1" clickable="true">

<img class="search close btn" alt="Clear query" id="2" clickable="true">

<div id="3" clickable="true">How to Do Ruby Rose Hair</div>

<div id="4" clickable="true">• </div>

<p id="5" clickable="true">41,446 views</p>

<div id="6" clickable="true">• </div>

<p id="7" clickable="true">Updated</p>

<p id="8" clickable="true">2 years ago</p>

<div id="9" clickable="true">• </div>

<p id="10" clickable="true">Expert Co-Authored</p>

<div id="11" clickable="true">How to Dye Your Hair Rose Gold</div>

<div id="12" clickable="true">• </div>

<p id="13" clickable="true">48,548 views</p>

<div id="14" clickable="true">• </div>

<p id="15" clickable="true">Updated</p>

<p id="16" clickable="true">3 years ago</p>

<div id="17" clickable="true">• </div>

<p id="18" clickable="true">Expert Co-Authored</p>

<div class="statusBarBackground" id="19" clickable="false"></div>

Access the article "How to Do Ruby Rose Hair"

Action History:

INPUT(2, do ruby rose hair)

to rp12.1cm
2cPrompt for ChatGLM

Available responses are:

INPUT(element_id, text)

Search an article to learn how to hide gauges.

Then, access the article "How to Hide Gauges"

<button alt="Open navigation drawer" id="0" clickable="true"></button>

<img class="wikihow toolbar logo" id="1" clickable="false">

<img class="search button" alt="Search" id="2" clickable="true">

<div class="webView" id="3" clickable="true"></div>

<div class="statusBarBackground" id="4" clickable="false"></div>

Action History:

INPUT(2, hide gauges)

Search an article to learn how to do ruby rose hair.

Then, access the article "How to Do Ruby Rose Hair"

Then, access the about page to learn why people trust wikihow.

<button alt="Open navigation drawer" id="0" clickable="true"></button>

<input class="search src text" value="Do ruby rose hair " type="text" id="1" clickable="true">

<img class="search close btn" alt="Clear query" id="2" clickable="true">

<div id="3" clickable="true">How to Do Ruby Rose Hair</div>

<div id="4" clickable="true">• </div>

<p id="5" clickable="true">41,446 views</p>

<div id="6" clickable="true">• </div>

<p id="7" clickable="true">Updated</p>

<p id="8" clickable="true">2 years ago</p>

<div id="9" clickable="true">• </div>

<p id="10" clickable="true">Expert Co-Authored</p>

<div id="11" clickable="true">How to Dye Your Hair Rose Gold</div>

<div id="12" clickable="true">• </div>

<p id="13" clickable="true">48,548 views</p>

<div id="14" clickable="true">• </div>

<p id="15" clickable="true">Updated</p>

<p id="16" clickable="true">3 years ago</p>

<div id="17" clickable="true">• </div>

<p id="18" clickable="true">Expert Co-Authored</p>

<div class="statusBarBackground" id="19" clickable="false"></div>

Access the article "How to Do Ruby Rose Hair"

Action History:

INPUT(2, do ruby rose hair)

Access the article "How to Do Ruby Rose Hair"

Action History:

INPUT(2, do ruby rose hair)

§.§ Compute resources

In our experiments, Android Emulator runs on a Linux host with KVM
(Kernal-based Virtual Machine) enabled to conduct efficient enough emulation.
ChatGLM-6B <cit.> is deployed on an NVIDIA GeForce RTX
3090 with 24 GiB of memory. The codes and the weights are from the official
repository[<https://huggingface.co/THUDM/chatglm-6b>] on Hugging
Face. LLaMA-13B <cit.> is deployed on two NVIDIA TITAN
RTXs with 24 GiB of memory. The codes and the weights are from the official
repository[<https://github.com/facebookresearch/llama>] on GitHub.
GPT-3.5 <cit.> and
ChatGPT <cit.> are accessed through the
online APIs[<https: //openai.com/api/>] of OpenAI. GPT-3.5 is
accessed through text-davinci-003, while ChatGPT is accessed through

§.§ Case study

Example of ChatGLM-6B trajectory

Examples from invalid responses of LLaMA-13B

Several cases of ChatGLM-6B and LLaMA-13B

Example of ChatGPT trajectory

Example of trajectory with VH input

Several cases of ChatGPT and GPT-3.5 with VH input. The element id
is replaced by the element text (icon) in the figures for a clear

Example trajectory of ChatGPT. ChatGPT is misled by the
exemplar and mistakenly regards element 2 as the search button. However,
the search button is actually element 1 on this page.

Example trajectory of GPT-3.5. GPT-3.5 is not misled in this
case and correctly locates the search button.

Comparison between behaviors of ChatGPT and GPT-3.5. The element
id is replaced by the element text (icon) in the figures for a clear

A Case that the LLM-based agent gets out of the cycle of wrong
actions. After repeating some wrong CLICKs for several steps, the
agent manages to get out of the cycle and perform the proper action to reach
the next page from the current one.

A Case that the LLM-based agent failed because of improper search
keywords. The agent refuses to try some different keywords after failing to
reach the target.

Cases of responses of LLMs It is observed that ChatGLM-6B prefers
actions, and LLaMA-13B is more prone to violate the output format
and generate free-form language responses. One of the trajectories of
ChatGLM-6B is demonstrated in Fig. <ref>. Several examples
from the invalid responses of LLaMA-13B are illustrated in Fig. <ref>.
Fig. <ref> demonstrates a trajectory of ChatGPT, which reveals
that ChatGPT seems to be less capable of jumping out of the wrong action cycle
than GPT-3.5.  Another example in Fig. <ref> offers a
comparison between the behaviors of ChatGPT and GPT-3.5. It is more often than
GPT-3.5 that ChatGPT is misled by the first exemplar in the prompt and
mistakenly corresponds the search button to id 2.

Cases from ablation study An example from the trajectories of
GPT-3.5 with VH input is illustrated in Fig. <ref>. A degradation
of the capacity for understanding the screen is observed in this case compared
to the trajectory of the full model in Fig. <ref>. Additionally,
Fig. <ref> depicts a case that the LLM-based agent with action
history feedback succeeds in getting out of the wrong action cycle. The example
in Fig. <ref> demonstrates a case that the LLM-based agent fails
owing to improper search keywords.  The LLM-based agent always repeats the
originally selected keywords and never tries another keyword combination. This
fact reveals that the current LLM-based agent has a weak ability to adopt the
interaction feedback during multistep episode. Thus, a more effective feedback
mechanism should be designed to improve the performance of the LLM.

§ DATASHEET

§.§ Motivation

* For what purpose was the dataset created? Was there a specific
task in mind? Was there a specific gap that needed to be filled? Please
provide a description.

Mobile-Env platform and WikiHow task set are created to fill in the
blank of a proper evaluation benchmark for the interaction capability
of LLM-based agents. This work constructs an InfoUI interaction
platform to establish two challenges for LLM-based agents: (1)
multistep interaction (2) in structured text-based environments.

* Who created the dataset (e.g., which team, research
group) and on behalf of which entity (e.g., company,
institution, organization)?

Danyang Zhang, Lu Chen, Zihan Zhao, Ruisheng Cao, and Kai Yu from
X-Lance Lab of SJTU AI Institute create the platform and the task set.

§.§ Composition

* What do the instances that comprise the dataset represent
(e.g., documents, photos, people, countries)? Are there
multiple types of instances (e.g., movies, users, and ratings;
people and interactions between them; nodes and edges)? Please provide
a description.

The task definitions of WikiHow task set are provided in the text
format of Protocol Buffers[<https://protobuf.dev/>]. They
can be directly loaded and parsed by Mobile-Env. The app data of
WikiHow are given as the captured HTTP flows. All types of media
(e.g., web pages, images, and videos) are provided as raw
bytes for convenience during replay.

* How many instances are there in total (of each type, if

We totally instantiate 5,522 task definitions for WikiHow task set.
178 tasks are sampled from these, validated manually, and called the
canonical set. There are totally 856,045 resources crawled from
WikiHow website, in which there are 107,448 distinct web pages.

* Does the dataset contain all possible instances or is it a
sample (not necessarily random) of instances from a larger set? If the
dataset is a sample, then what is the larger set? Is the sample
representative of the larger set (e.g., geographic coverage)?
If so, please describe how this representativeness was validatedverified. If it is not representative of the larger set, please
describe why not (e.g., to cover a more diverse range of
instances, because instances were withheld or unavailable).

There are over 340,000 articles on WikiHow, from which about 100,000
ones are crawled. The pages are accessed in breadth-first order to
mimic the browsing behavior of human users and agents. We argue that
the current saved flow data are sufficient for the exploration of
agents in the app. The task definitions are instantiated with the
sampled keyword information from the crawled WikiHow resources.

* What data does each instance consist of? “Raw” data
(e.g., unprocessed text or images) or features? In either case,
please provide a description.

The app data of WikiHow are stored as raw flow. Each flow file consists
of the HTTP headers in text and the payload in raw bytes.

* Is there a label or target associated with each instance? If so,
please provide a description.

The task definition files in WikiHow task set define task goals, step
instructions, and rewards.

* Is any information missing from individual instances? If so,
please provide a description, explaining why this information is
missing (e.g., because it was unavailable). This does not
include intentionally removed information, but might include,
e.g., redacted text.

* Are there any errors, sources of noise, or redundancies in the
dataset? If so, please provide a description.

Redundancies may exist in the crawled app data of WikiHow task set, as
some URLs may point to the same resource.

* Is the dataset self-contained, or does it link to or otherwise
rely on external resources (e.g., websites, tweets, other datasets)?
If it links to or relies on external resources, a) are there guarantees that
they will exist, and remain constant, over time; b) are there official archival
versions of the complete dataset (i.e., including the external
resources as they existed at the time the dataset was created); c) are there
any restrictions (e.g., licenses, fees) associated with any of the
external resources that might apply to a dataset consumer? Please provide
descriptions of all external resources and any restrictions associated with
them, as well as links or other access points, as appropriate.

WikiHow task set is considered self-contained. All the required app
data for WikiHow task set are saved.

* Does the dataset contain data that might be considered
confidential (e.g., data that is protected by legal privilege
or by doctor–patient confidentiality, data that includes the content of
individuals' non-public communications)? If so, please provide a

No. All the crawled app data are published publicly on WikiHow website.

* Does the dataset contain data that, if viewed directly, might be
offensive, insulting, threatening, or might otherwise cause anxiety?
If so, please describe why.

No. All the crawled app data have already been published according to
the content policy of WikiHow website.

* Does the dataset identify any subpopulations (e.g., by
age, gender)? If so, please describe how these subpopulations are
identified and provide a description of their respective distributions
within the dataset.

* Is it possible to identify individuals (i.e., one or
more natural persons), either directly or indirectly (i.e., in
combination with other data) from the dataset? If so, please describe how.

The article authors may be identified through the pages of article and
author information, which are publicly available on WikiHow website.

* Does the dataset contain data that might be considered sensitive
in any way (e.g., data that reveals race or ethnic
origins, sexual orientations, religious beliefs, political
opinions or union memberships, or locations; financial or
health data; biometric or genetic data; forms of government
identification, such as social security numbers; criminal history)? If
so, please provide a description.

No. All the crawled app data have already been published according to
the content policy of WikiHow website.

§.§ Collection Process

* How was the data associated with each instance acquired? Was
the data directly observable (e.g., raw text, movie ratings),
reported by subjects (e.g., survey responses), or indirectly
inferredderived from other data (e.g., part-of-speech
tags, model-based guesses for age or language)?  If the data was
reported by subjects or indirectly inferredderived from other
data, was the data validatedverified? If so, please describe

All the crawled app data and information used in the task definitions
of WikiHow task set are directly observable on WikiHow website.

* What mechanisms or procedures were used to collect the data
(e.g., hardware apparatuses or sensors, manual human curation,
software programs, software APIs)? How were these mechanisms or procedures

Mobile-Env platform is developed with Python based on Androin Emulator.
Several auxiliary tools and scripts are created with Tcl, JavaScript,
etc. The task definitions of WikiHow task set are generated
with the template toolkit (introduced in <ref>) of
Mobile-Env. The app data of WikiHow task set are crawled with a Python
crawler on a host with 8 CPU threads and 15GiB of memory.

* If the dataset is a sample from a larger set, what was the
sampling strategy (e.g., deterministic, probabilistic with
specific sampling probabilities)?

The task definitions of WikiHow task set are combined from the
single-page task definitions randomly under the constraint that
guarantees the target page in the succeeding step is directly
referenced on the page in the preceding step. The app data are crawled
in breadth-first order starting from the home page to mimic the
browsing behavior of agents and offer the largest coverage of the
agents' exploration.

* Who was involved in the data collection process (e.g.,
students, crowdworkers, contractors) and how were they compensated
(e.g., how much were crowdworkers paid)?

All the development of platform, app data collection, and task
definition creation are completed by the authors.

* Over what timeframe was the data collected? Does this timeframe
match the creation timeframe of the data associated with the instances
(e.g., recent crawl of old news articles)? If not, please
describe the timeframe in which the data associated with the instances
was created.

The app data of WikiHow task set are collected from WikiHow website
from Jul. 6th, 2022 to Aug. 18th, 2022. Wiki articles published on
the website before the date may be included.

* Did you collect the data from the individuals in question
directly, or obtain it via third parties or other sources (e.g.,

The app data including the published wiki articles and public author
information are crawled from WikiHow website.

* Were the individuals in question notified about the data
collection? If so, please describe (or show with screenshots or other
information) how notice was provided, and provide a link or other
access point to, or otherwise reproduce, the exact language of the
notification itself.

The individuals as the wiki article authors are not notified.

* Did the individuals in question consent to the collection and
use of their data? If so, please describe (or show with screenshots or
other information) how consent was requested and provided, and provide
a link or other access point to, or otherwise reproduce, the exact
language to which the individuals consented.

Only the publicly available data on WikiHow website are crawled. We
obey the terms of use and  of WikiHow website. Thus,
it is considered that the individuals as the wiki article authors
consent to the collection and usage of their works.

* If consent was obtained, were the consenting individuals
provided with a mechanism to revoke their consent in the future or for
certain uses? If so, please provide a description, as well as a link or
other access point to the mechanism (if appropriate).

The crawled app data are regared as a snapshot of WikiHow website.
Currently, no mechanism for revocation is designed. However, if there
is really a need to revoke the particular resources, the indidual can
contact the paper authors directly.

§.§ Uses

* Has the dataset been used for any tasks already? If so, please
provide a description.

Agents based on different LLMs have been evaluated on WikiHow task set
in this paper.

* Is there a repository that links to any or all papers or systems
that use the dataset? If so, please provide a link or other access

Currently, no. We plan to develop a website tracking and indexing the
available task sets for Mobile-Env.

* Is there anything about the composition of the dataset or the
way it was collected and preprocessedcleanedlabeled that
might impact future uses? For example, is there anything that a dataset
consumer might need to know to avoid uses that could result in unfair
treatment of individuals or groups (e.g., stereotyping, quality of
service issues) or other risks or harms (e.g., legal risks,
financial harms)? If so, please provide a description. Is there anything a
dataset consumer could do to mitigate these risks or harms?

§.§ Distribution

* Will the dataset be distributed to third parties outside of the
entity (e.g., company, institution, organization) on behalf of
which the dataset was created? If so, please provide a description.

Yes. Both Mobile-Env platform and WikiHow task set are open-sourced.

* How will the dataset will be distributed (e.g., tarball
on website, API, GitHub)? Does the dataset have a digital object
identifier (DOI)?

The platform is open-sourced at GitHub. The task set is released at
Hugging Face. We do not apply for a DOI.

* When will the dataset be distributed?

Both the platform and the task set have already been made public.

* Will the dataset be distributed under a copyright or other
intellectual property (IP) license, andor under applicable terms
of use (ToU)? If so, please describe this license andor ToU, and
provide a link or other access point to, or otherwise reproduce, any
relevant licensing terms or ToU, as well as any fees associated with these

Both Mobile-Env platform and WikiHow task set are open-sourced under
Apached-2.0 license.

* Have any third parties imposed IP-based or other restrictions on
the data associated with the instances? If so, please describe these
restrictions, and provide a link or other access point to, or otherwise
reproduce, any relevant licensing terms, as well as any fees associated
with these restrictions.

The copyright of the app data in WikiHow task set are owned by the
original wiki authors and WikiHow website. No other third parties own
the copyright of Mobile-Env platform and the task definitions of
WikiHow task set.

* Do any export controls or other regulatory restrictions apply to
the dataset or to individual instances? If so, please describe these
restrictions, and provide a link or other access point to, or otherwise
reproduce, any supporting documentation.

§.§ Maintenance

* Who will be supportinghostingmaintaining the

The authors will support, host, and maintain both Mobile-Env platform
and WikiHow task set.

* How can the ownercuratormanager of the dataset be
contacted (e.g., email address)?

Issues and discussions on GitHub and Hugging Face are welcome. One can
also seek help from Danyang Zhang (zhang-dy20@sjtu.edu.cn), Lu
Chen (chenlusz@sjtu.edu.cn), and Kai Yu (kai.yu@sjtu.edu.cn).

* Is there an erratum? If so, please provide a link or other
access point.

Currently, no. Errata will be anounced if there is any in future.

* Will the dataset be updated (e.g., to correct labeling
errors, add new instances, delete instances)? If so, please describe
how often, by whom, and how updates will be communicated to dataset
consumers (e.g., mailing list, GitHub)?

Mobile-Env platform will be continuously developed and maintained.
Updates will be released on GitHub from time to time. Currently, there
is no remarkable updating plan for WikiHow task set. Errata may be
released to correct errors if there is any in future.

* If the dataset relates to people, are there applicable limits on
the retention of the data associated with the instances (e.g.,
were the individuals in question told that their data would be retained for
a fixed period of time and then deleted)? If so, please describe these
limits and explain how they will be enforced.

* Will older versions of the dataset continue to be
supportedhostedmaintained? If so, please describe how.
If not, please describe how its obsolescence will be communicated to
dataset consumers.

Old versions of Mobile-Env and WikiHow task set can be accessed through
GitHub and Hugging Face, respectively.

* If others want to extendaugmentbuild oncontribute to the dataset, is there a mechanism for them to do so? If
so, please provide a description. Will these contributions be
validated/verified? If so, please describe how. If not, why not? Is
there a process for communicatingdistributing these
contributions to dataset consumers? If so, please provide a

We sincerely welcome that one can contribute features or report bugs
for Mobile-Env through the mechanisms like pull request, issues,
etc. on GitHub. If new environments or task sets are crafted,
it will be welcome that the creators notify the authors through e-mail
or GitHub, so that we can update the indices of available task sets.

§ REVIEW FROM PREVIOUS VENUE & REVISION

This work is previously submitted to ACL Demo 2023. The main concerns of the
reviewers are (1) the particular motivation of this work, (2) how the platform
is used, (3) and the structure of the task definition file. Accepted these
suggestions, we first highlight two challenges for LLM-based agents: (1)
multistep interaction (2) in structured text-based environments. Then we
further clarify our motivation: to fill in a blank and build an evaluation
platform for interactive agents in the LLM era. Moreover, in the main paper, we
supplement clear descriptions of a few terms in Mobile-Env and the structure of
the task definition file. Meanwhile, a brief usage guideline (<ref>)
involving the structure of observation and action objects and usage of wrappers
is supplemented in the supplementary.

§ LIMITATIONS & FUTURE WORK

There are still several limitations of this work. More LLMs are expected to be
evaluated to draw deeper insights into the interaction capability of LLM in
structured text-based environments. Meanwhile, more environments and task sets
are supposed to be constructed to establish more challenging and comprehensive
benchmarks. It is also sincerely welcome that the community contribute diverse
environments and task sets based on Mobile-Env. We will continue maintaining
and developing Mobile-Env platform in future work as well. We plan to develop a
distributed version of Mobile-Env, so that the host with hardware
virtualization (e.g., KVM) for launching Android Emulator can be
separated from the GPU host for deploying agents, on which hardware
virtualization is usually turned off by default. Other new functions are going
to be developed, aiming at supporting more diverse tasks and agents.