# AutoTask: Executing Arbitrary Voice Commands by Exploring and Learning from
Mobile GUI

Lihang Pan  0000-0001-8856-0309<EMAIL_ADDRESS>,  Bowen Wang
<EMAIL_ADDRESS>0009-0009-1358-722X Department of Computer Science
and Technology, Tsinghua UniversityBeijingChina ,  Chun Yu
0000-0003-2591-7993 Department of Computer Science and Technology, Tsinghua
UniversityBeijingChina ,  Yuxuan Chen  Department of Computer Science and
Technology, Tsinghua UniversityBeijingChina ,  Xiangyu Zhang
0009-0006-2183-3054 Department of Computer Science and Technology, Tsinghua
UniversityBeijingChina and  Yuanchun Shi  0000-0003-2273-6927 Department of
Computer Science and Technology, Tsinghua UniversityBeijingChina
<EMAIL_ADDRESS>
(2024)

###### Abstract.

Voice command interfaces (VCIs) have gained increasing importance, enabling
hands-free and eyes-free interaction with digital devices. However, the
inherent complexity in constructing effective voice interfaces has limited the
VCIs’ functionalities to only a small fraction of GUI applications and tasks.
This paper presents AutoTask, a VCI capable of automating any task in any
mobile application without configuration or modification from developers or
end users. The primary challenge for AutoTask is the lack of knowledge, as it
needs to accomplish unknown tasks (e.g., user commands) within an unknown
environment (e.g., GUI). To address this challenge, AutoTask employs two
strategies: (1) trial and error: AutoTask explores the GUI, attempts potential
operation sequences, and recovers from errors through backtracking; (2)
learning from the environment: AutoTask accumulates experiences during
exploration and summarizes correct knowledge from these experiences. We
implemented AutoTask on Android devices and conducted an evaluation study,
which proved the feasibility of AutoTask.

voice command interface, large language model, UI automation

††copyright: acmcopyright††journalyear: 2024††doi:
XXXXXXX.XXXXXXX††conference: Proceedings of the ACM on Interactive, Mobile,
Wearable and Ubiquitous Technologies; October 5–9 2024; Melbourne,
Australia††booktitle: IMWUT ’24: Proceedings of the ACM on Interactive,
Mobile, Wearable and Ubiquitous Technologies, October 5–9 2024, Melbourne,
Australia††price: 15.00††isbn: 978-1-4503-XXXX-X/18/06††ccs: Human-centered
computing Natural language interfaces††ccs: Human-centered computing User
interface programming††copyright: none Figure 1. To execute the command
”import contacts from contacts.vcf” in the Contacts application (version
4.8.17), AutoTask first clicks the ”Add” button, transitioning from Page 1 to
Page 2. AutoTask finds that it can only manually add a contact on Page 2;
hence, it reverts to Page 1 and clicks another button labeled ”Fix & Manager”.
It then completes the import process through subsequent steps (4 & 5). After
finishing the task, AutoTask synthesizes knowledge from its experiences,
improving its ability for future commands.

##  1\. Introduction

Voice interaction can effectively enhance the interactivity of applications,
enabling end users to automate multi-step GUI tasks (e.g., setting an alarm
for 9 AM) eyes-freely and hands-freely (Pan et al., 2022; Wulf et al., 2014;
Zhong et al., 2014; Luger and Sellen, 2016; Cowan et al., 2017). However,
constructing voice command interfaces (VCIs) for existing GUI tasks is
challenging and requires significant effort (Pan et al., 2022; Huang et al.,
2023). Consequently, existing VCIs support only a limited set of predefined
intents, failing to cover the actual needs of users (Luger and Sellen, 2016;
Pan et al., 2022).

Large language models (LLMs) have been applied to numerous domains and have
significantly reduced the cost of system development and deployment (Ross et
al., 2023; Zheng et al., 2023; Fan et al., 2023; Fiannaca et al., 2023; Chung
et al., 2022). While LLMs help understand user commands and alleviate the
development burden of VCIs, their application is limited in scope (Wang et
al., 2023a). Developers are still required to pre-define a set of intents for
the voice interface (Arsan et al., 2021; Pan et al., 2022). Additionally, they
must configure how these intents are executed (Li and Riva, 2018; Li et al.,
2017) and address potential errors arising from LLMs (Tandon et al., 2022).

In this paper, we present AutoTask, a ready-to-use VCI that operates without
any modifications or configurations by either developers or end users. It is
capable of executing any intent within any application. AutoTask accomplishes
arbitrary tasks (i.e., user commands) in an unknown environment (i.e., the
GUI), the primary challenge of which is the lack of necessary knowledge. To
overcome this, AutoTask (1) engages in trial and error: exploring the GUI,
attempting possible operation sequences, and recovering from errors through
backtracking; and (2) learns from the environment by accumulating experiences
during exploration and summarizing knowledge from them.

As illustrated in Figure 1, AutoTask comprehends the semantics of the GUI and
the user command and determines an operations sequence to carry out the given
task. For example, to execute the command ”Import contacts from contacts.vcf”,
AutoTask first chooses to click the ”Add” button on Page 1. It then emulates
this operation on the GUI, leading to content updates and a transition from
Page 1 to Page 2. AutoTask subsequently evaluates the correctness of the
executed operation sequence. If an error is detected, AutoTask revokes its
actions to rectify it. For instance, upon reaching Page 2, AutoTask recognizes
that it can only manually add a single contact there and cannot perform a
batch import from a file. Consequently, it reverts to Page 1 and selects
another button labeled ”Fix & Manager”. This process continues until the task
is successfully completed. Additionally, AutoTask improves its performance by
accumulating experiences while navigating the GUI and summarizing knowledge
from these experiences, including:

  1. (1)

Environmental knowledge: for example, in Figure 1, AutoTask can learn that
clicking the ”Add” button does not lead to batch importing of contacts. This
knowledge can expedite AutoTask’s execution of subsequent commands.

  2. (2)

Task knowledge: for instance, in Figure 1, AutoTask can learn that the intent
”Import contacts from a file” requires a parameter specifying the file name.
This knowledge aids AutoTask in understanding the semantics of the commands.

  3. (3)

Execution knowledge: as shown in Figure 1, AutoTask can learn the correct
operation sequence for importing contacts from a file. This sequence can be
directly replayed to accomplish similar tasks and help execute other commands.

This paper makes two main contributions:

  1. (1)

We introduce a new paradigm in which an agent accomplishes unknown tasks in an
unknown environment. The agent explores the environment to find a solution and
summarizes its experiences into knowledge to enhance its capabilities.

  2. (2)

We present a ready-to-use VCI named AutoTask, where end users can automate any
intent with a single command. Experimental results proved its usability. We
implemented AutoTask on Android smartphones and conducted an evaluation study.
The experimental results proved its usability.

##  2\. Related Work

Voice command interfaces can effectively reduce the interaction burden,
enabling users to interact with devices hands-freely and eyes-freely (Pan et
al., 2022; Wulf et al., 2014; Zhong et al., 2014; Luger and Sellen, 2016;
Cowan et al., 2017). However, constructing a VCI for mobile devices requires
significant effort (Pan et al., 2022; Li and Riva, 2018; Huang et al., 2023),
leading to the existing VCIs covering only a limited set of GUI
functionalities. The workload primarily encompasses determining the supported
intent set, understanding user natural language commands accurately, and
executing tasks correctly. Self-improvement of VCIs during runtime is a
crucial approach to reducing effort (Pan et al., 2022) but has yet to gain
widespread support. Table 1 compares existing VCIs with AutoTask in these four
aspects.

Table 1. Comparison of AutoTask and existing VCIs. The content within parentheses indicates the workload required by developers or end users. PBD stands for programming by demonstration, and CCG stands for combinatory categorial grammar. | Intent Set | Command Understanding | Task Execution | Self-Improvement
---|---|---|---|---
| SUGILITE
---
(Li et al., 2017)
| Predefined
---
(PBD)
| CCG
---
(Handcrafted rules)
| Replay operations
---
(PBD)
| Improve executing
---
(PBD)
| SAVANT
---
(Arsan et al., 2021)
| Predefined
---
(specify a list)
| Dialogflow agents
---
(provide examples)
| Search app screen
---
(collect traces)
Not supported
| AutoVCI
---
(Pan et al., 2022)
| Predefined
---
(PBD)
| BERT
---
(extra dialogues)
| Replay operations
---
(PBD)
| Improve understanding
---
(extra dialogues)
| LLM4Mobile
---
(Wang et al., 2023a)
| Intents in the GUI
---
(No effort)
| LLM
---
(No effort)
N/A | Not supported
AutoTask |  | Any intent
---
(No effort)
| LLM
---
(No effort)
| Explore the GUI
---
(No effort)
| Improve understanding
---
Improve executing
(No effort)

###  2.1. Supported intents of VCIs

The earliest voice interfaces on smartphones only supported single-step GUI
operations (Ashok et al., 2015, 2017; Zhong et al., 2014; Wang et al., 2023a;
Vu et al., 2023), for example, clicking a button already present on the GUI.
The intent sets of this kind of VCI are limited to the current GUI contents.
Although the VCIs can be applied to any mobile application without any
configuration or modification, GUI tasks typically require multiple operations
(e.g., clicking several buttons sequentially), and providing voice commands
for each step would impose a significant interaction burden. Therefore, this
approach is mainly used for accessibility purposes (Grussenmeyer and Folmer,
2017; Zhong et al., 2014; Wagner, 2013) and has not been widely adopted by
ordinary users.

Task-oriented VCIs (Li et al., 2017; Arsan et al., 2021; Li and Riva, 2018;
Pan et al., 2022; Sereshkeh et al., 2020) (e.g., Siri) address the
aforementioned issues and reduce the interaction burden. End users only
provide a single voice command, and the virtual assistant can automatically
complete a multi-step task on the GUI (e.g., setting a 9:00 am alarm).
However, existing task-oriented voice assistants only support a limited set of
intents. For instance, Siri does not support sending WhatsApp messages. This
results in two challenges: on one hand, developers need to invest a
significant amount of effort (e.g., conducting formative studies (Li et al.,
2017; Arsan et al., 2021)) to determine a useful intent set; on the other
hand, end users often complain about the discoverability of functionalities
(Murad et al., 2019; Srinivasan et al., 2019; Corbett and Weber, 2016;
Yankelovich, 1996) and the lack of support for necessary intents (Luger and
Sellen, 2016).

AutoTask differs significantly from the two categories of voice interfaces
mentioned above. AutoTask is a task-oriented voice assistant capable of
automating multi-step tasks with a single command. However, AutoTask does not
rely on a predefined set of intents and can accommodate any intent that can be
executed on the GUI without requiring additional overhead from developers or
end users.

###  2.2. Understanding User Commands

A task-oriented voice assistant understands the user command to: (1) classify
the command into a specific intent (e.g., sending a message); and (2) identify
parameters for the intent (e.g., message content and message recipient) (Pan
et al., 2022; Li et al., 2017; Li and Riva, 2018). The most traditional
approach to command understanding is using context-free grammar (e.g., regular
expressions (Ashok et al., 2017; Zhong et al., 2014) and combinatory
categorial grammar (CCG) (Azaria et al., 2016; Li et al., 2017, 2018, 2019)).
Additionally, researchers have employed more sophisticated algorithms (e.g.,
word dependency (Sereshkeh et al., 2020), n-grams (Gao et al., 2015; Kim et
al., 2019a) and word embedding (Sereshkeh et al., 2020)) to extract features
from commands and create natural language processing scripts. These solutions
heavily rely on handcrafted rules (Ward and Issar, 1994; Ashok et al., 2015),
necessitating significant developer effort; however, their performance is
relatively poor (Azaria et al., 2016; Luger and Sellen, 2016; Xu et al., 2020;
Yankelovich et al., 1995) and cannot satisfy users’ needs.

Nowadays, many systems utilize deep neural networks to comprehend user
commands, achieving satisfactory results (Qu et al., 2019; Seo et al., 2019).
Before the widespread adoption of pre-trained models, researchers needed to
collect a large amount of training data (Brown et al., 2020; Devlin et al.,
2018; Thomason et al., 2015) and meticulously fine-tune the model’s
architecture and parameters to achieve good results (Zhang and Wallace, 2015).
This process involved a significant workload. With the development of pre-
trained models, researchers only provide prompts and a few optional examples,
and large language models (LLMs) can successfully understand the commands
(Wang et al., 2023a; Liu et al., 2023a).

An often overlooked issue is that interactive systems may inaccurately
interpret user commands, leading to conversation breakdowns (Beneteau et al.,
2019; Grudin and Jacques, 2019; Pan et al., 2022). This problem increases the
user’s burden (Jiang et al., 2013; Pelikan and Broth, 2016) and reduces their
willingness to engage in interactions (Jain et al., 2018; Luger and Sellen,
2016; Zamora, 2017). Existing solutions all entail additional costs. For
instance, developers can programmatically address breakdowns (Bohus and
Rudnicky, 2005; Kim et al., 2019b). AutoVCI (Pan et al., 2022) and SOVITE (Li
et al., 2020a) require additional user interactions to ensure accurate command
understanding.

AutoTask utilizes LLMs to comprehend user commands, enabling support for any
task in any application with minimal development effort. To address the issue
of LLM errors, AutoTask does not require developer or user involvement;
instead, it automatically learns from the mobile GUI and continually adjusts
its command understanding results.

###  2.3. Executing the Command

Existing VCIs search for and execute scripts in databases based on the command
understanding results; the scripts can automatically carry out the user
commands. These scripts can be categorized into two types based on their
origins: those created by developers and those generated by end users.

The scripts for the majority of commercial voice assistants are manually
created by developers (Ravindranath et al., 2012). For example, Siri directly
invokes functions implemented by developers to execute voice commands. Because
developers need to write a script for each intent, this approach significantly
limits the number of functionalities available in the voice interface (Pan et
al., 2022).

Since end users have a strong demand for voice assistants that can support
their personalized needs, researchers have proposed different methods to
collect execution scripts from users. One typical approach is program by
demonstration (PBD) (Pan et al., 2022; Allen et al., 2007; Cypher and Halbert,
1993; Sugiura and Koseki, 1996; Myers, 1986; Li et al., 2017; Li and Riva,
2018; Sereshkeh et al., 2020; Dey et al., 2004; Veuskens et al., 2020). Users
can demonstrate how they complete tasks on the GUI; this process is recorded
and automatically transformed into execution scripts. Researchers have also
attempted to extract execution scripts from users’ historical behavior records
automatically (Arsan et al., 2021; Antila et al., 2012; Lau et al., 2010).
While this approach avoids the explicit burden of demonstration, it also
results in unpredictable system capabilities and relies on the time and
quality of data accumulation.

Note that scripts collected from users are not always correct. For example,
replaying the operation sequence demonstrated by users may fail due to pop-up
windows or application version updates (Li et al., 2017). Existing solutions
require users to handle exceptions manually (Li et al., 2017), which
introduces additional burdens.

AutoTask does not require any predefined scripts, whether they originate from
developers or end users. It dynamically calculates a potential operation
sequence on the GUI at runtime. Furthermore, it assesses whether the sequence
is correct and automatically handles errors. The entire process does not
necessitate any user intervention.

###  2.4. Self-Improvement

Many systems can learn and improve themselves through interactions with users.
A typical application scenario is learning user preferences to provide better
services (e.g., recommendations (Massimo et al., 2017; Tu et al., 2019),
navigation (Rana and Bridge, 2020), and scheduling (Gervasio et al., 2005)).
AutoVCI (Pan et al., 2022) can enhance its semantic understanding ability
through multi-turn dialogues with users (Thomason et al., 2015). In the field
of machine learning, this approach is known as ”human-in-the-loop” (Stiennon
et al., 2020; Ziegler et al., 2019; Wu et al., 2022), in which users
contribute to improving machine capabilities by providing annotations. This
approach has achieved significant success in training large language models
(OpenAI, 2023; Ouyang et al., 2022).

Reinforcement learning is a common approach for improving machine capabilities
without user intervention: AI-driven agents enhance themselves based on
rewards provided by the environment (Kaelbling et al., 1996). This approach
has been widely applied in fields such as gaming (Kaiser et al., 2019) and
autonomous driving (Kiran et al., 2021), but it has seen limited application
in executing user commands on GUIs (Branavan et al., 2010). AppBuddy (Shvo et
al., 2021) is a preliminary attempt in this direction; however, it suffers
from issues like sparse rewards (Li et al., 2023) and excessive trial-and-
error steps, making it unsuitable for direct application in interactive
systems. AutoTask shares a similar concept with reinforcement learning: it
autonomously summarizes knowledge from its explorations of the GUI, all
without requiring user intervention.

##  3\. Problem Formulation & Solution

This paper focuses on problems of the following form: an intelligent agent is
required to complete an unknown task in an unknown environment. This type of
problem is prevalent in applying artificial intelligence (AI) to daily tasks
for two primary reasons. Firstly, it is impossible for the developers to
collect corpora and pre-train an agent for every real-world scenario.
Secondly, end users often struggle to, or choose not to, provide
comprehensive, structured descriptions and step-by-step procedures for tasks.
Solving problems characterized by these patterns can significantly broaden the
application of AI in everyday tasks.

AutoTask is a solution to the problem in the field of VCIs: it automates the
execution of interaction intents (i.e., tasks) expressed through natural
language commands by simulating user operation sequences within the GUI (i.e.,
the environment) of mobile devices. AutoTask supports any applications and
intents, ensuring comprehensive coverage of GUI tasks for voice assistants.

The core challenge of ”completing unknown tasks in an unknown environment”
lies in the lack of knowledge, which includes:

  1. (1)

Lack of environmental knowledge. Although the intelligent agent can observe
and interact with the environment, there is no prior knowledge about how the
environment will change after interactions. For example, AutoTask can acquire
GUI content; however, it lacks knowledge about how the GUI will change after
simulating user actions (e.g., clicking a button on the screen).

  2. (2)

Lack of task knowledge. The agent does not possess a set of supported tasks or
have any predefined understanding related to task semantics or the
interpretation of external inputs. For example, we have not provided AutoTask
with a predefined set of intents or information about intent parameters.
Additionally, we have not provided models or scripts to assist AutoTask in
recognizing the intents and parameters within user commands.

  3. (3)

Lack of execution knowledge. Given the absence of environmental and task
knowledge, the agent lacks the knowledge of how to execute tasks. For
AutoTask, this is reflected in the absence of execution scripts (whether
provided by developers or end users) for the intent.

To address this challenge, we propose an ”explore-learn” strategy that
comprises:

  1. (1)

Trial and error: The agent explores the environment and attempts to execute
the task. It recovers from errors through backtracking when necessary.

  2. (2)

Learn from the environment: The agent accumulates experiences (records of
actions and observations) during the exploration of the environment. From
these experiences, the agent learns knowledge, enabling it to (a) directly
execute intents that have been completed in the past and (b) expedite the
exploration when executing unknown intents.

The system design of AutoTask is an application of this strategy to the field
of voice assistants, which will be elaborated upon in the next section.

##  4\. System Design

Figure 2. AutoTask’s pipeline. It first checks if the intent has been
executed. If so, it replays the recorded operation sequence. Otherwise,
AutoTask explores the GUI and learns. It understands the GUI and command
semantics, deciding the most probable operation. This operation is executed by
simulating user actions. AutoTask checks the completed actions with the new
GUI content. If an error is detected, it backtracks to undo previous actions.
Throughout, AutoTask accumulates experience and summarizes knowledge to
enhance its capabilities. The arrows pointing to ”Experiences” are omitted in
the figure for simplicity.

Figure 2 illustrates the AutoTask pipeline. Upon receiving a user command,
AutoTask checks whether the intent expressed in the command has been
previously executed. If it has, AutoTask automatically executes the task by
replaying the operation sequence, with adjustments based on the current
command’s parameter values (Pan et al., 2022). If the command has not been
executed previously or if the replay of the sequence fails, AutoTask enters
the ”explore-learn” mode, which can be divided into two parts:

  1. (1)

Trail and error, which can be further subdivided into forward exploration and
backward backtracking:

    1. (a)

During the forward exploration, AutoTask selects an optimal operation within
the current GUI content (the understanding module and the deciding module),
which is then automated by programmatically injecting an event into the GUI
(the executing module). After obtaining the resulting GUI content, AutoTask
assesses whether the current task is completed and whether the executed
operation sequence is correct (the checking module). Based on this assessment,
it decides whether to terminate the execution, continue forward exploration,
or initiate backward backtracking.

    2. (b)

During the backward backtracking, AutoTask undoes the last action (the
backtracking module) and evaluates whether the current task is completed and
whether the operation sequence is correct (the checking module). AutoTask
decides accordingly to terminate the execution, continue backward
backtracking, or start forward exploration.

  2. (2)

Learning from the environment, which encompasses:

    1. (a)

Accumulating experiences: AutoTask records all its decisions (e.g., outputs of
the modules) and GUI observations.

    2. (b)

Summarizing knowledge: AutoTask extracts correct knowledge from experiences at
appropriate times. The details of AutoTask’s knowledge are illustrated in
Table 2.

    3. (c)

Applying knowledge: AutoTask utilizes its knowledge during task execution to
directly execute previously completed intents or expedite the exploration
process.

Table 2. AutoTask’s knowledge Knowledge | Type | Content | Goal
---|---|---|---
| Directly execute
---
completed intents
| Expedite Exploration
---
for unknown intents
Environmental Knowledge | 1 |  | Contents and transitions
---
within the GUI
(used in understanding)
- | ✓
2 |  | Contents or transitions
---
that do not exist within the GUI
(used in deciding and checking)
- | ✓
| Task
---
Knowledge
- |  | Intents and parameters of commands
---
(used in replaying and understanding)
✓ | -
Execution Knowledge | 1 |  | Correct operation sequences
---
(used in replaying, deciding, and checking)
✓ | ✓
2 |  | Lessons that prevent execution errors
---
(used in deciding and checking)
- | ✓

###  4.1. Learning from the GUI: Summarizing Experiences into Knowledge

####  4.1.1. Experiences of AutoTask

AutoTask automatically records its experiences during runtime, which assists
the system in executing the current task and is also summarized into knowledge
to support subsequent tasks. As depicted in Figure 3, we represent AutoTask’s
experiences as a graph, where nodes represent GUI pages, and edges record the
results of the modules. Please refer to the corresponding sections for
detailed results of each module.

Figure 3. AutoTask’s execution experience for the command ”Import Contacts
from contacts.vcf”. (1) AutoTask clicks the ”Add” button on Page 1. (2)
AutoTask detects an incorrect history operation based on the content of the
new page (i.e., Page 2) and backtracks to Page 1, as indicated by the orange
arrow. (3) AutoTask determines that the task remains incomplete and selects
the ”Fix & Manage” button. (4) AutoTask clicks ”Import from file” on the new
page. (5) AutoTask selects ”contacts.vcf” following the user command. It is a
parameter of the intent. (6) AutoTask concludes that the task is completed and
finishes the execution.

####  4.1.2. Environmental knowledge

AutoTask’s environment is the GUI, and its environmental knowledge describes
the contents of GUI pages and the transitions between pages. This knowledge
can be categorized into the following two types:

  1. (1)

(Type-1) Contents and page transitions of the GUI, which are stored in the
form of triplets (S, O, D). S and D represent the GUI pages before and after
the operation (represented through HTML, as indicated in Figure 5). O = (E, A,
P), which denotes elements, actions, and parameters (e.g., the texts for the
”text input” actions), respectively; these three components together describe
a GUI operation. For example, the Edge (5) (in Figure 3) corresponds to the
following triplet: (Page 4, (button labeled ”contacts.vcf”, click, null), Page
3).

  2. (2)

(Type-2) Contents or transitions that do not exist in the GUI. This kind of
knowledge is in natural language. For example, one piece of environmental
knowledge (Type-2) summarized for Edge (1) (in Figure 3) could be, ”By
clicking the ’Add’ button on the home page, you can only manually adding a
single contact. Importing contacts from files is not supported”. This
knowledge may prevent AutoTask from attempting to click the ”Add” button in
future commands (e.g., ”Import contacts from cloud backup”), thereby
expediting the execution of subsequent tasks.

####  4.1.3. Task knowledge

Task knowledge helps understand the semantics of the tasks. AutoTask’s task is
to execute natural language commands, the semantics of which are typically
described by intents and parameters (also called slots in some works (Li and
Riva, 2018)). A piece of task knowledge comprises (1) an intent name, (2)
parameter names, (3) a command corresponding to the intent, and (4) values of
the parameters in that command. An intent may appear in multiple pieces of
task knowledge, as the commands inside are different. For example, the task
knowledge from Figure 3 is (1) intent name - ”import contacts from file”; (2)
parameter name list - ”file name”; (3) command - ”import contacts from
contacts.vcf”; (4) parameter values - ”file name = ’contacts.vcf’”.

####  4.1.4. Execution knowledge

Execution knowledge describes how to execute the command in the GUI. It can be
categorized into the following two types:

  1. (1)

(Type-1) The correct operation sequence for the command. For example, the
first type of execution knowledge for ”import contacts from contacts.vcf” is
”click ’Fix & manage’, click ’Import from file’, click ¡file name¿”. The angle
brackets (¡¿) denote parameter values in the command.

  2. (2)

(Type-2) Describing how to avoid incorrect operations. This knowledge
comprises lessons summarized in natural language, corresponding to errors made
by AutoTask during execution. As exemplified in Figure 4, AutoTask makes a
mistake when executing ”save Alice, 2122000000”: it clicks ”Save” without
adding any information about the contact. A possible piece of knowledge
summarized from this error could be, ”You should pay attention to the order of
steps; actions that may finalize a task (e.g., clicking the Save button)
should be performed last”.

Figure 4. AutoTask’s execution experiences for the command ”Save Alice,
2122000000 to contact”. AutoTask makes a mistake: clicking the Save button
(Edge 1) without entering the name and the phone number. This error is
corrected through backtracking (Edge 2). AutoTask also summarizes a piece of
knowledge to avoid similar mistakes: you should pay attention to the order of
steps; actions that may finalize a task (e.g., clicking the Save button)
should be performed last.

####  4.1.5. From experiences to knowledge

AutoTask’s experiences need to be further summarized into knowledge because
some experiences may be redundant or erroneous.

Environmental knowledge (Type-1): When AutoTask simulates an operation on the
GUI and obtains the contents of the resulting page, the experience is
immediately transformed into environmental knowledge (Type-1). AutoTask’s
observations of the environment are always correct; as a result, AutoTask can
directly and instantaneously convert the related experiences into knowledge.

Execution Knowledge (Type-1): AutoTask identifies the shortest path in its
experiences that (1) connects the starting point and the endpoint111The GUI
screen where AutoTask thinks the task is completed and (2) encompasses all
parameters. For example, in Figure 3, the path ”3-4-5” satisfies the
aforementioned criteria. This path is considered the correct operation
sequence for the task and is added to the database. Such knowledge is only
summarized after the task is completed because AutoTask can only determine the
task’s endpoint at that time.

Task Knowledge: AutoTask records the command understanding result (to be
discussed in 4.2.2) at the final step of the correct path as a piece of task
knowledge. This type of knowledge will be summarized once AutoTask completes
the task; otherwise, the command understanding result may be incorrect.

Environmental Knowledge (Type-2) & Execution Knowledge (Type-2): The purpose
of these two types of knowledge is to prevent errors during task execution.
AutoTask compares its experiences with the correct path to identify erroneous
steps (e.g., Step 1 in Figure 3 & 4). AutoTask categorizes the reasons for
errors into two types:

  1. (1)

Lack of environmental knowledge. For example, the error in step 1 of Figure 3
occurs because AutoTask does not know whether there will be an ”import from
file” option after clicking the ”Add” button. Since importing from a file is a
way to ”add” a batch of contacts, AutoTask considers trying the ”Add” button
worthwhile.

  2. (2)

Lack of execution knowledge. For example, the error in step 1 of Figure 4
happens because AutoTask does not know how to determine the order of
operations when multiple GUI actions are related to the command.

AutoTask summarizes a lesson in natural language for each error and utilizes
it to avoid future errors. This knowledge can only be summarized after
completing the task. Otherwise, AutoTask cannot accurately determine whether a
step is correct. We employ LLM to compile this type of knowledge. The prompt
will be discussed in section 5.3.

###  4.2. The Understanding Module

In the understanding module, AutoTask comprehends the GUI and the user
command, the results of which serve as inputs for subsequent modules. This
module enables AutoTask to augment information about the environment (i.e.,
the GUI) and the task (i.e., the user command) with its knowledge.

####  4.2.1. Understanding the GUI

Figure 5. An example result of AutoTask understanding the GUI. (A) The
operation sequence for enabling SIM lock. We omit some operations, such as
scrolling the screen. AutoTask learns environmental knowledge (Type-1) from
the sequence. (B) The HTML representation of Page 1. We omit some GUI
elements. (C) The HTML representation after AutoTask understanding the GUI
when executing the command ”enable app pinning”. The understanding result
corresponds to the ”targets” property of the ”Security & privacy” button.

The GUI semantics are formed by the contents of GUI pages and the transitions
between pages. AutoTask can obtain page data through APIs provided by the
operating system (e.g., Android
AccessibilityService222https://developer.android.com/reference/android/accessibilityservice/AccessibilityService).
However, the incompleteness of GUI semantics arises because the contents after
GUI operations cannot be foreseen. AutoTask addresses this issue by querying
environmental knowledge to infer the elements ”hidden” behind the current GUI
elements. This process can be divided into two steps:

  1. (1)

AutoTask retrieves elements from the environmental knowledge (Type-1) that can
be reached through one or multiple operations starting from the current GUI
elements.

  2. (2)

To filter out irrelevant elements, these elements are transformed into
vectors, and their similarities to the user command are computed (details are
discussed in section 5.2). The semantic understanding result includes only
elements with similarities exceeding a threshold. As exemplified in Figure 5
(C), the button ”App pinning” is very related to the user command ”enable app
pinning” and is reachable by operating an element on the current GUI (clicking
”Security & privacy” and then clicking ”More security & privacy”). As a
result, it is added to the ”target” property of the button ”Security &
privacy”.

The GUI semantics effectively guide AutoTask in selecting the correct
operations. As illustrated in Figure 5, AutoTask successfully executes the
command ”enable SIM lock”. This is not challenging since SIM lock and security
are highly semantically related. AutoTask also learns GUI-related knowledge
during execution. Next, AutoTask executes the command ”enable App pinning”.
Without relevant GUI knowledge, executing this command is challenging:
AutoTask may blindly attempt to click on different buttons on the Android
Settings homepage, such as ”Application”, ”Display”, and ”Safety”. However,
GUI knowledge can assist AutoTask in directly choosing ”Security & privacy”
without additional explorations. During GUI understanding, AutoTask discovers
a high semantic similarity between the ”App pinning” button and the user
command, and there exists an operation sequence from ”Security & privacy” to
”App pinning”.

####  4.2.2. Understanding the command

During the process of understanding command semantics, AutoTask generates a
natural language phrase that describes the intent conveyed in the user
command. It also detects the parameters and their values from the command.
Command understanding needs to be performed in each iteration of AutoTask
because the understanding results may be updated as experiences accumulate
(Pan et al., 2022).

The command semantics can guide AutoTask in taking correct operations in the
GUI. For example, in the command ”Save Alice, 2122000000 to contact” (Figure
4), if AutoTask realizes that ”Alice” and ”2122000000” are parameters for
”create a new contact”, it will use the two parameters during task execution,
that is, entering them into text boxes333Another way to use a parameter is to
select an item with corresponding text in a list (Pan et al., 2022).. The
results of command understanding may be stored as task knowledge (as already
discussed in 4.1.5), which can assist AutoTask in comprehending subsequent
commands.

AutoTask employs task knowledge and an LLM to understand the semantics of
commands. We calculate the semantic similarities between the historical
commands in the task knowledge and the current user command (details will be
discussed in 5.2). Historical commands with similarities greater than a
threshold will be selected as examples. The LLM will utilize these examples,
the executed operation sequence, and current GUI contents to calculate the
command understanding results. The prompts will be discussed in 5.3.

###  4.3. The Deciding Module

Table 3. Types of operations supported by AutoTask. Click, text input, and scroll forward are used in the deciding module, while the other three are used in the backtracking module. Action | GUI Elements | Parameters | Usage
---|---|---|---
Click | Clickable elements | N/A | The Deciding Module
Text Input | Editable elements | Text content
Scroll Forward | Scrollable elements | N/A
Navigate Up | N/A | N/A |  | The Backtracking Module
---
(to undo a click operation)
Clear Text | Editable elements | N/A |  | The Backtracking Module
---
(to undo a text input operation)
Scroll Backward | Scrollable elements | N/A |  | The Backtracking Module
---
(to undo a scroll forward operation)

In the deciding module, AutoTask calculates the most possible operation in the
current GUI to complete the user command. AutoTask identifies all
operations444The parameter for text input will be determined later available
in the GUI. Table 3 provides an overview of the types of supported operations.
Subsequently, AutoTask assigns scores to these operations, with higher scores
indicating a greater possibility of being the next operation. Each operation’s
score is calculated based on a basic score and a penalty factor:
$score=basic\\_score/(1+penalty)$:

  1. (1)

Basic Score (1.0 - 8.0), which is calculated by adding the following two
components together:

    1. (a)

Likert scale (1.0 - 7.0). We employ an LLM to assess the relevance of the
operations to the user’s command. We use a 7-point Likert scale, where 1
indicates extremely low relevance, and 7 denotes very high relevance. To
assist the LLM in calculating the scores, we retrieve relevant environmental
knowledge (Type-2) and execution knowledge (Type-1 & 2) from the knowledge
base, as discussed in 5.2. Further details about the prompt will be provided
in 5.3.

    2. (b)

Tie-breaking score (0.0 - 1.0). We calculate the semantic similarities between
operations and tasks (please refer to section 5.2 for more details) as the
tie-breaking scores. The tie-breaking scores increase the differentiation
between different operations. While Likert scales are typically effective at
identifying the most relevant operation, they may lack granularity when
scoring less relevant options. The tie-breaking scores prevent operations from
receiving identical scores, thereby avoiding AutoTask being reduced to brute-
force searching.

  2. (2)

Penalty factor (0.0 - positive infinity). It is important to note that
penalizing an operation does not necessarily mean that the final score of the
element will not be the highest. For example, when an operation is penalized
by the checking module, AutoTask may attempt other operations and find that
these operations are even less relevant to the user command. In this case,
AutoTask may retry the penalized operation. The penalty factor consists of two
components:

    1. (a)

Repetition penalty, used to penalize operations that have already appeared in
the executed operation sequence555Note that a combination of an action, a GUI
element, and a parameter describes an operation. A repetitive operation
implies that AutoTask has arrived at the current GUI screen.. The repetition
penalty is fixed at 10.

    2. (b)

Backtracking penalty, used to penalize operations considered incorrect by the
checking module. The backtracking penalty is initialized at 0 and can be
updated by the checking module (see 4.5 for details).

AutoTask selects the operation with the highest score as the next to be
executed. If the current operation is text input, AutoTask utilizes an LLM to
calculate the text content. Please refer to 5.3 for more details about the
prompt.

###  4.4. The Executing Module & the Backtracking Module

In the executing module, we utilize the accessibility API to inject operations
into the GUI based on the results from the deciding module. Conversely, the
backtracking module injects interaction events (e.g., scrolling backward) to
undo previous operations (e.g., scrolling forward, as indicated in Table 3).
Both modules retrieve the GUI hierarchy after injecting events, which will be
converted into HTML format (as shown in Figure 5(B)) and used by other
modules. Operations may only cause some minor localized changes in the GUI, so
we compare the GUI pages before and after the operations to identify newly
appeared elements, which are then marked with a boolean property named ”new”
(as shown in Figure LABEL:fig:_appendix-context in the Appendix).

We remove elements that meet both of the following two criteria from the GUI
hierarchy:

  1. (1)

Low interaction importance. This criterion applies when the element itself and
its descendant elements (if any) cannot be interacted with and do not contain
text or descriptions.

  2. (2)

Low layout importance. This criterion applies when the element has no sibling
elements or all sibling elements are considered to have ”low interaction
importance”.

###  4.5. The Checking Module

After AutoTask performs GUI operations (both in the executing module and the
backtracking module), the checking module conducts two checks on the completed
operation sequence: completeness and correctness.

####  4.5.1. completeness check

The checking module employs an LLM to determine whether the current task is
completed. This check is also performed during the backtracking process to
address ”overshoot” issues, where unnecessary operations are executed after
task completion. For more prompt details, please refer to 5.3. The checking
module also considers the task completed when the number of executed steps
exceeds a threshold (set to 20 in our implementation).

When AutoTask considers the task as completed, we present the user with a list
describing the shortest execution path (as discussed in 4.1.5). Each item in
the list includes (1) a screenshot, (2) a rectangular bounding box used to
highlight the operated element in the screenshot, and (3) a text description
of the operation, e.g., ”Text input: Alice”. The user can choose one of the
following options:

  1. (1)

Confirming the correctness of the execution process. The system summarizes the
knowledge and terminates.

  2. (2)

Confirming that the task is not yet completed. The system continues running
and starts the correctness check. The current page will not be considered as
the endpoint for the current command.

  3. (3)

Forcing termination. The system stops running directly without knowledge
summarization. Note that the first type of environmental knowledge is still
summarized and accumulated.

  4. (4)

Ignoring (default666In the evaluation study, we assumed that users would
select this option.). If the step threshold is exceeded, AutoTask stops
running without knowledge summarization. Otherwise, the system summarizes
knowledge and terminates.

####  4.5.2. correctness check

If a task is not completed, AutoTask checks whether the last
step777Backtracking steps or the steps being undone will not be considered as
last steps. For example, if ”A-B-C” forms an operation sequence and AutoTask
uses operation D to undo operation C, then, even though the sequence is ”A-B-
C-D”, we regard B as the last step. currently being executed is correct, i.e.,
whether AutoTask can continue to fulfill the user’s instruction. The essence
of a correctness check is checking the correctness of the deciding module with
more experiences and knowledge accumulated from the GUI. For example, in
Figure 3, AutoTask clicks the ”Add” button to import contacts from the file.
However, after simulating user interaction and obtaining new GUI contents, the
checking module can discover that the result of the deciding module is
erroneous.

AutoTask applies an LLM to conduct the correctness check. Please refer to 5.3
for the detailed prompt of LLM. It is worth noting that AutoTask takes into
account the backtracking penalty of the last step. If an operation has a high
backtracking penalty but is still executed, the possibility of other
operations may be lower. In such a case, the checking module should be more
tolerant and consider it correct, providing an opportunity for further
exploration.

If the last operation is considered to be incorrect, the checking module will
calculate a penalty (0-9) to describe the severity of the error: 0 indicates
that the error in the last operation is due to preceding steps already being
incorrect; 9 indicates a very serious error in the last operation itself. This
penalty will be accumulated into the current backtracking penalty of the last
operation. Consequently, the backtracking penalty of an operation may be
greater than 9, which indicates that it has been rejected by the checking
module several times. We use LLM to calculate the penalty, with details of its
prompt discussed in 5.3.

##  5\. Implementation

In this section, we describe how AutoTask utilizes LLMs for computational
purposes. Please refer to the Appendix for more detailed examples.

###  5.1. AutoTask’s context

AutoTask’s context describes its state and is widely used throughout the
computational process, as shown in Figure LABEL:fig:_appendix-context in the
Appendix. It encompasses (1) the user command, (2) the executed operation
sequence, and (3) the current GUI contents represented in HTML. The GUI
content is augmented with environmental knowledge (Type-1), as discussed in
4.2.1. (4) the latest semantic comprehension result of the instruction (if
any).

###  5.2. Embedding & Similarity: Choosing One or More Answers from Several
Candidates

AutoTask utilizes the Embedding & Similarity approach to select one or
multiple answers from several candidates for a given question. Both the
question and the candidates are transformed into vectors using the embedding
API provided by OpenAI. We regard each candidate’s cosine similarity with the
question as its score. A higher score indicates that the candidate is more
likely to be the correct answer. Table 4 summarizes the usage of this method.
It is worth noting that the description of elements includes their surrounding
elements, as they may exhibit strong semantic relevance.

Table 4. Usage of the Embedding & Similarity approach. To prevent the table from becoming too wide, we use ”env.” as an abbreviation for ”environmental” and ”exe.” as an abbreviation for ”execution”. Purpose | Question | Candidate Answers
---|---|---
| Retrieve elements from
---
env. knowledge (Type-1)
(4.2.1)
| (1) context
---
(2) ”what element is related”
| Elements
---
in env. knowledge (Type-1)
| Filter relevant pieces of
---
task knowledge
(4.2.1)
| (1) context
---
(2) ”what task is similar to the command”
| Tasks and its semantics
---
from task knowledge
| Filter relevant pieces of
---
env. knowledge and
exe. knowledge
(4.3 & 4.5)
| (1) context
---
(2) ”what knowledge is related”
| Env. knowledge (Type-2)
---
Exe. knowledge (Type-1 & 2)
| Compute tie-breaking score
---
(4.3)
| (1) context
---
(2) ”what element should be operated”
Current GUI elements

###  5.3. Text Completion: Answering Questions

AutoTask utilizes the text completion approach to generate an answer for a
given question. The question is passed as part of the prompt to the LLM
(gpt-4), and the response generated by the LLM serves as the answer. Table 5
summarizes the usage of this method. The output template specifies the JSON
format the LLM response should adhere to, which AutoTask can parse easily.

Table 5. Usage of the Text Completion method Module | Purpose (question) | Prompt Composition
---|---|---
Understanding | Understand command semantics |  | (1) Purpose
---
(2) Task knowledge
(3) Context
(4) Output template
Deciding | Calculate the Likert scale |  | (1) Purpose
---
(2) Execution knowledge
(3) Context
(4) Output template
| Calculate parameter for
---
text input
| (1) Purpose
---
(2) Context
(3) Textbox to be edited
(4) Output template
Check |  | Checking completeness
---
Check correctness
Calculate penalty
| (1) Purpose
---
(2) Execution knowledge
(3) GUI before last operation
(4) Context
(5) Output template
N/A | Summary Knowledge |  | (1) Purpose
---
(2) AutoTask experiences
(3) The erroneous step
(4) The ground truth
(5) Output template

##  6\. Evaluation Study

The evaluation study has two goals: (1) to validate that AutoTask can
correctly execute user instructions without predefined knowledge, and (2) to
validate that knowledge accumulation can effectively accelerate AutoTask’s
execution of user commands. We did not evaluate AutoTask’s feasibility
regarding executing intents that have been carried out before. In such cases,
AutoTask only replays the recorded operation sequences (with parameter
adjustments), and the feasibility has already been validated in previous works
(Pan et al., 2022).

###  6.1. Apparatus

We conducted the evaluation on an Android virtual machine (Pixel_XL running
Android 11). No modifications were made to the virtual machine or system, and
AutoTask can run on commercial physical devices.

###  6.2. Tasks

We validated the capabilities of AutoTask on the following datasets:

  1. (1)

PixelHelp (Li et al., 2020b). Consisting of natural language commands and
their corresponding operation sequences, PixelHelp was revised for this study.
Due to system and application upgrades, some outdated instructions were
manually removed, leaving a total of 67 instructions.

  2. (2)

UGIF (Venkatesh et al., 2022). This dataset also comprises natural language
commands and their corresponding operation sequences. We concentrated on
instructions related to Android Settings, as knowledge accumulation is more
pronounced within the same application. We used these tasks to assess the
impact of knowledge accumulation on AutoTask’s performance. Similar to
PixelHelp, outdated instructions were removed, resulting in 100 remaining
instructions.

We made adjustments to the commands in the datasets, including:

  1. (1)

Changing the tone from inquiry to command. Both datasets were compiled from
tutorials on the internet, where instructions were mostly presented in the
form of inquiries. We modified the instructions to align them with how users
typically interact with voice assistants. For example, we changed ”how to turn
off WiFi” to ”turn off WiFi”.

  2. (2)

Supplementing missing parameters. Some commands lacked parameters and were not
executable. We added parameters to these instructions. For example, we changed
”how to delete a Google account” to ”delete the Google account named Alice”.

###  6.3. Metrics

Success rate, i.e., the ratio of the number of successfully completed tasks to
the total number of tasks.

The step accuracy of a task, i.e., the ratio of the number of correct steps to
the minimum number of steps required to complete the task. The correct steps
are defined as the longest subsequence (instead of substring) of the actually
executed steps that satisfy the following criterion: the minimum steps needed
to complete the task start with the subsequence. For completed tasks, this
metric is always 1. For tasks that were not successfully completed, this
metric indicates the proximity of AutoTask to successful completion.

The step redundancy rate of a task, i.e., the ratio of the difference between
the number of executed steps and the number of correct steps to the number of
executed steps. This metric evaluates the system’s efficiency. Tasks that
succeeded with a step redundancy rate of 0 are referred to as ”tasks completed
without redundancy”.

Non-redundant completion rate, i.e., the ratio of the number of tasks
completed without redundancy to the total number of tasks.

###  6.4. Baseline

We utilized the LLM approach proposed by Android in the Wild (AITW) (Rawles et
al., 2023) as the baseline. Similar to AutoTask, this approach employs an LLM
to automate user instructions without requiring any configuration or
modifications, making it applicable to arbitrary GUI intents. However, the
baseline solution lacks an explicit self-checking and backtracking mechanism,
although it can undo previous actions by performing certain actions (e.g.,
clicking the back button). Furthermore, it does not summarize and accumulate
knowledge from the execution process. Note that in the original baseline
solution, the prompt used only included information about the most recent five
operations. We adjusted it to include the complete operation sequence to
ensure consistency with AutoTask. When the number of execution steps in the
baseline approach exceeds 20, we also forcibly terminate it.

###  6.5. Procedure

AutoTask and the baseline are executed in the same order for the commands from
the two datasets (shuffled beforehand). After completing each task (normal
completion or exceeding the maximum number of steps), the next instruction is
automatically executed. Throughout this process, AutoTask accumulates
knowledge when a task ends successfully but does not utilize knowledge derived
from other tasks888AutoTask still utilizes the first type of environment
knowledge accumulated during the execution of the current task.. After
completing all tasks, experimenters manually check whether each task has been
executed correctly.

We categorized the tasks in UGIF into three types based on AutoTask’s
execution results: (Type-1) tasks that AutoTask can complete without
redundancy; (Type-2) tasks that AutoTask can complete with redundancy;
(Type-3) tasks that AutoTask is unable to complete. Tasks of Type-1 and 2 are
collectively referred to as Type-A tasks; AutoTask has already summarized
knowledge for Type-A tasks in Phase 1. Tasks of Type-2 and 3 are collectively
referred to as Type-B tasks; there is room for improvement in AutoTask’s
performance on Type-B tasks.

We then evaluate the improvement of AutoTask’s performance with the
accumulated knowledge. For each Type-B task, we randomly select a certain
number of tasks from Type-A tasks999We guarantee that the selected Type-A
tasks do not include the Type-B task to be tested.. We evaluate the
performance of AutoTask after accumulating the knowledge from these tasks. We
repeat the aforementioned random selection process ten times and compute the
average results to mitigate the influence of random noise. To explore the
impact of the amount of knowledge on AutoTask’s performance, we repeated the
process several times with different percentages of the selected Type-A tasks:
20%, 40%, 60%, 80%, and 100%.

###  6.6. Results

####  6.6.1. Success rate

The accuracy of AutoTask in PixelHelp is 91.2% (6 errors in 67 commands) and
that in UGIF is 93.0% (7 errors in 100 commands). The two metrics for the
baseline are 52.2% (32 errors in 67 commands) and 67.0% (33 errors in 100
commands), respectively. The chi-square test (p ¡ 0.001) proved that AutoTask
significantly outperformed the baseline in both datasets. We identified the
following two reasons:

  1. (1)

AutoTask demonstrates excellent capability in detecting task completion, with
precision at 99.4% and recall at 100%. Although the baseline achieves 100%
accuracy in verifying task completion, its recall rate is relatively low:
36.1% (15 tasks from PixelHelp and 11 tasks from UGIF) of the errors are due
to ”overshoot”, that is, the baseline executes unnecessary steps after the
tasks had already completed.

  2. (2)

AutoTask exhibits higher accuracy in its behavior on the GUI (results from the
deciding and checking modules). The step accuracy of AutoTask is 93.5%
(PixelHelp: 93.6%, UGIF: 93.4%), whereas the baseline stands at 82.9%
(PixelHelp: 77.4%, UGIF: 87.2% ).

We analyzed tasks that AutoTask did not complete correctly and categorized the
reasons for errors into three main types:

  1. (1)

AutoTask failed to properly ground instructions to the GUI, leading to blind
attempts on the interface. A typical example is the command ”check my
chromebook if any”, where AutoTask did not recognize ”chromebook” as a
connected device and instead kept clicking irrelevant buttons such as
”Display” and ”System”. Three tasks failed due to this reason.

  2. (2)

AutoTask was misled by information in the command, continuously trying to
accomplish irrelevant tasks. For instance, with the command ’lock screen when
app unpinning’, AutoTask focused on what it perceived as the keyword ”lock”
and repeatedly tried to add a personal identification number (PIN) to the
phone. Nine tasks failed due to this misunderstanding.

  3. (3)

AutoTask mistakenly believed the task was completed. This occurred once in our
experiments with the command ”show system applications”. AutoTask successfully
navigated to the ”Installed Applications” page and thought the task was
finished. However, it was expected to click the ”More Options” button and then
select the ”Show System” option.

####  6.6.2. Redundancy

The step redundancy rates of AutoTask across both datasets are 8.54% and
8.96%, significantly lower than the corresponding baseline results (46.0%,
p¡0.001; 32.0%, p¡0.001). AutoTask necessitates backtracking in only 12
(PixelHelp: 7, UGIF: 5) (7.14%) tasks, with an average backtrack count of 2.83
(min=1, max=8, sd = 5.18) within these tasks. This indicates that AutoTask
requires minimal backtracking to accomplish tasks.

Model | Dataset | Success Rate | Step Accuracy | Step Redundancy Rate | Non-redundant Completion Rate
---|---|---|---|---|---
Baseline | PixelHelp | 52.2% | 77.4% | 46.0% | 65.3%
UGIF | 67.0% | 87.2% | 32.0% | 78.4%
AutoTask | PixelHelp | 91.2% | 93.6% | 8.54% | 89.7%
UGIF | 93.0% | 93.4% | 8.96% | 95.0%
Table 6. Evaluation Results of Baseline LLM (AITW) and AutoTask on PixelHelp
dataset and UGIF dataset.

####  6.6.3. Performance improvement through knowledge accumulation

Figure 6. The success rate (A) and step accuracy (B) of Type-3 tasks increase
as the percentage of Type-A tasks from which AutoTask accumulates knowledge
increases. The shadow represents the standard deviation.

The quantities of the three task types in UGIF are 88, 5, and 7, respectively.
Figure x illustrates how the success rate and the step accuracy increase with
the accumulation of knowledge. When AutoTask learns all the knowledge, these
metrics can reach 85.7% and 97.6%, respectively. Only one Type-3 task cannot
be completed even after accumulating all the knowledge.

Furthermore, knowledge accumulation effectively improves the efficiency of
task completion. Figure 7 & 8demonstrates how the average step redundancy rate
and non-redundant completion rate for Type-2 and Type-3 tasks change as the
accumulated knowledge increases. Upon acquiring all available knowledge, the
step redundancy rates for these two task types decrease from 46.7% to 16.1%
and from 97.1% to 4.76%, respectively, with only 3 tasks (2 Type-2, 1 Type-3)
still requiring backtracking.

Figure 7. The step redundancy rate (A) and non-redundant completion rate (B)
of Type-2 tasks change as the percentage of Type-A tasks from which AutoTask
accumulates knowledge increases. The shadow represents the standard deviation.
Figure 8. The step redundancy rate (A) and non-redundant completion rate (B)
of Type-3 tasks change as the percentage of Type-A tasks from which AutoTask
accumulates knowledge increases. The shadow represents the standard deviation.

##  7\. Discussion

###  7.1. Generalization of Knowledge: Across Versions and Applications

Figure 9. How AutoTask executes ”Import contacts from contacts.vcf” in version
1.7.31 of the Contacts application. Note that the location of this
functionality has changed compared to Figure 3.

While the study evaluated the performance improvement brought about by
knowledge accumulation within the same application, this knowledge can be
applied across different versions of the same application or even across
applications with similar functionalities (e.g., iMessage vs. WhatsApp). These
applications follow similar design principles and semantics (Liu et al.,
2018). For instance, Figure 9 depicts an earlier version of a contact
application (version 1.7.31). In contrast to Figure 3, its home page does not
contain ”Fix & manage”, and ”Import from file” is hidden under ”Settings”.
However, AutoTask can still benefit from the knowledge it has summarized. For
example, it will refrain from attempting to click the ”Add” button, thereby
expediting the execution of the command.

The generalization of knowledge across versions and different applications
also carries some risks, as the accuracy of such knowledge cannot be
guaranteed. For example, if AutoTask first imports contacts from a file in
version 1.7.31 of the Contacts application, it may summarize the knowledge
that this functionality can be found by clicking the ”Settings” button.
However, when it attempts the same task in version 4.8.17 of the Contacts
application based on this knowledge, it might experience decreased efficiency,
even though it can still complete the task correctly through backtracking.
This is because the functionality of importing contacts from a file has been
relocated in the new version. One solution is to estimate the confidence level
of its knowledge. While this goes beyond the scope of this paper, it is a
promising direction for future research.

###  7.2. The Generalization of AutoTask: to Other Devices and Tasks

While we implemented AutoTask on Android smartphones and conducted evaluation
experiments, AutoTask can generalize to other devices and platforms (e.g., web
browsers (Leshed et al., 2008; Little et al., 2007)) as long as they provide
APIs, with which AutoTask can (1) retrieve the current contents on the GUI and
(2) simulate user interactions in the GUI. AutoTask may also generalize to
non-graphical interfaces, such as command line interfaces (Liu et al., 2023b),
to reduce the learning curve and interaction burdens. Besides, AutoTask can be
viewed as a proxy for existing GUIs (Huang et al., 2023; Zhang et al., 2017),
and its interaction modality is not limited to voice interaction. GUI mapping
(Lee et al., 2021) (e.g., mapping a smartphone GUI to a smartwatch GUI (Zhou
et al., 2020)) is a typical example. Developers only need to be concerned with
the visual mapping rules (Chen et al., 2018) instead of the execution logic of
the applications. AutoTask can automatically operate the original GUI (e.g.,
smartphone GUI) based on the user’s interaction behaviors on the new GUI
(e.g., smartwatch GUI).

AutoTask ”accomplishes unknown tasks in an unknown environment”, and all tasks
that align with this pattern can benefit from the ”explore-learn” strategy
outlined in this paper with little effort required from developers or end
users. For example, crafting prompts that yield high-performance results can
be challenging for end users (Dang et al., 2022; Jiang et al., 2022; Deng et
al., 2022; Liu et al., 2023a; Wang et al., 2023b). In this problem, the
environment is the LLM that has not been fully explored and the task is to
generate an effective prompt. However, the effectiveness is not well-defined.
AutoTask can attempt an initial prompt to determine the capability boundaries
of the LLM and then refine it (similar to the backtracking process described
in this paper) based on the responses while accumulating knowledge to enhance
its prompt-generation capabilities. Similar concepts (Li et al., 2023) are
found in ReAct (Yao et al., 2022) and Reflexion (Shinn et al., 2023). However,
these approaches do not explicitly summarize knowledge to enhance their
capabilities. AutoTask can also extend beyond the digital world into the
physical world and be applied in various fields, such as embodied intelligence
(Brohan et al., 2023; Driess et al., 2023; Murali et al., 2023).

##  8\. Limitation & Future Work

AutoTask does not interact with users during its execution. However, user
commands may be incomplete or ambiguous (Pan et al., 2023), and AutoTask
should request clarification or additional information when necessary.
Additionally, it can proactively ask questions to prune its GUI exploration
based on the user’s answers. For example, when a user needs to enable App
pinning (Figure 5, Page 3), AutoTask may not be familiar with this feature and
attempt various incorrect operations. AutoTask can significantly expedite
command execution if it proactively asks the user questions like, ”Is this
feature related to Security?” to gain relevant insights.

AutoTask utilizes an online LLM (gpt-4) service provided by OpenAI via HTTPS
requests, and the computational process is slow. Although optimizing the
efficiency of LLMs is beyond the scope of this paper, future work could
construct smaller and faster models, for example, through techniques such as
knowledge distillation (Gou et al., 2021), to reduce waiting time for end
users.

Users often have complex intents that cannot be covered by a single GUI task
(Grudin and Jacques, 2019; Luger and Sellen, 2016; Fast et al., 2018). For
example, in the command ”Send my schedule to Alice”, a voice interface is
expected to accomplish two tasks (retrieve the schedule and send a message)
sequentially. In future work, AutoTask can be combined with complex task
decomposition systems (Teevan et al., 2016; Yang et al., 2023) to satisfy
users’ complex needs.

##  9\. Conclusion

In this paper, we present AutoTask, a voice command interface that automates
voice commands by simulating GUI interactions. To make it applicable across
different applications and GUI tasks, AutoTask requires no configuration or
modification from developers or end users. Instead, AutoTask explores the GUI
to attempt different operation sequences and accumulates knowledge from these
explorations to enhance its capabilities. The evaluation study proves the
feasibility of this approach: AutoTask performs significantly better than the
baseline when no knowledge is accumulated, and knowledge accumulation further
improves its performance. AutoTask addresses the special case in the voice
assistant domain of accomplishing unknown tasks in an unknown environment, a
problem pattern that applies to many other similar scenarios. We hope that
AutoTask can inspire future work to apply general artificial intelligence to
everyday tasks with little effort from the developers or the end users.

## References

  * (1)
  * Allen et al. (2007) James Allen, Nathanael Chambers, George Ferguson, Lucian Galescu, Hyuckchul Jung, Mary Swift, and William Taysom. 2007.  Plow: A collaborative task learning agent. In _AAAI_ , Vol. 7. 1514–1519.
  * Antila et al. (2012) Ville Antila, Jussi Polet, Arttu Lämsä, and Jussi Liikka. 2012.  RoutineMaker: Towards end-user automation of daily routines using smartphones. In _2012 IEEE International Conference on Pervasive Computing and Communications Workshops_. IEEE, 399–402.
  * Arsan et al. (2021) Deniz Arsan, Ali Zaidi, Aravind Sagar, and Ranjitha Kumar. 2021.  App-Based Task Shortcuts for Virtual Assistants. In _The 34th Annual ACM Symposium on User Interface Software and Technology_. 1089–1099.
  * Ashok et al. (2015) Vikas Ashok, Yevgen Borodin, Yury Puzis, and IV Ramakrishnan. 2015.  Capti-speak: a speech-enabled web screen reader. In _Proceedings of the 12th International Web for All Conference_. 1–10.
  * Ashok et al. (2017) Vikas Ashok, Yury Puzis, Yevgen Borodin, and IV Ramakrishnan. 2017.  Web screen reading automation assistance using semantic abstraction. In _Proceedings of the 22nd International Conference on Intelligent User Interfaces_. 407–418.
  * Azaria et al. (2016) Amos Azaria, Jayant Krishnamurthy, and Tom Mitchell. 2016.  Instructable intelligent personal agent. In _Proceedings of the AAAI Conference on Artificial Intelligence_ , Vol. 30.
  * Beneteau et al. (2019) Erin Beneteau, Olivia K Richards, Mingrui Zhang, Julie A Kientz, Jason Yip, and Alexis Hiniker. 2019.  Communication breakdowns between families and Alexa. In _Proceedings of the 2019 CHI conference on human factors in computing systems_. 1–13.
  * Bohus and Rudnicky (2005) Dan Bohus and Alexander Rudnicky. 2005.  Error handling in the RavenClaw dialog management architecture. In _Proceedings of Human Language Technology Conference and Conference on Empirical Methods in Natural Language Processing_. 225–232.
  * Branavan et al. (2010) SRK Branavan, Luke Zettlemoyer, and Regina Barzilay. 2010.  Reading between the lines: Learning to map high-level instructions to commands. In _Proceedings of the 48th annual meeting of the association for computational linguistics_. 1268–1277.
  * Brohan et al. (2023) Anthony Brohan, Yevgen Chebotar, Chelsea Finn, Karol Hausman, Alexander Herzog, Daniel Ho, Julian Ibarz, Alex Irpan, Eric Jang, Ryan Julian, et al. 2023\.  Do as i can, not as i say: Grounding language in robotic affordances. In _Conference on Robot Learning_. PMLR, 287–318.
  * Brown et al. (2020) Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020\.  Language models are few-shot learners.  _Advances in neural information processing systems_ 33 (2020), 1877–1901.
  * Chen et al. (2018) Chunyang Chen, Ting Su, Guozhu Meng, Zhenchang Xing, and Yang Liu. 2018.  From ui design image to gui skeleton: a neural machine translator to bootstrap mobile gui implementation. In _Proceedings of the 40th International Conference on Software Engineering_. 665–676.
  * Chung et al. (2022) John Joon Young Chung, Wooseok Kim, Kang Min Yoo, Hwaran Lee, Eytan Adar, and Minsuk Chang. 2022.  TaleBrush: Sketching stories with generative pretrained language models. In _Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems_. 1–19.
  * Corbett and Weber (2016) Eric Corbett and Astrid Weber. 2016.  What can I say? addressing user experience challenges of a mobile voice user interface for accessibility. In _Proceedings of the 18th international conference on human-computer interaction with mobile devices and services_. 72–82.
  * Cowan et al. (2017) Benjamin R Cowan, Nadia Pantidi, David Coyle, Kellie Morrissey, Peter Clarke, Sara Al-Shehri, David Earley, and Natasha Bandeira. 2017.  ” What can i help you with?” infrequent users’ experiences of intelligent personal assistants. In _Proceedings of the 19th international conference on human-computer interaction with mobile devices and services_. 1–12.
  * Cypher and Halbert (1993) Allen Cypher and Daniel Conrad Halbert. 1993.  _Watch what I do: programming by demonstration_.  MIT press.
  * Dang et al. (2022) Hai Dang, Lukas Mecke, Florian Lehmann, Sven Goller, and Daniel Buschek. 2022.  How to prompt? Opportunities and challenges of zero-and few-shot learning for human-AI interaction in creative applications of generative models.  _arXiv preprint arXiv:2209.01390_ (2022).
  * Deng et al. (2022) Mingkai Deng, Jianyu Wang, Cheng-Ping Hsieh, Yihan Wang, Han Guo, Tianmin Shu, Meng Song, Eric P Xing, and Zhiting Hu. 2022.  Rlprompt: Optimizing discrete text prompts with reinforcement learning.  _arXiv preprint arXiv:2205.12548_ (2022).
  * Devlin et al. (2018) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018.  Bert: Pre-training of deep bidirectional transformers for language understanding.  _arXiv preprint arXiv:1810.04805_ (2018).
  * Dey et al. (2004) Anind K Dey, Raffay Hamid, Chris Beckmann, Ian Li, and Daniel Hsu. 2004.  a CAPpella: programming by demonstration of context-aware applications. In _Proceedings of the SIGCHI conference on Human factors in computing systems_. 33–40.
  * Driess et al. (2023) Danny Driess, Fei Xia, Mehdi SM Sajjadi, Corey Lynch, Aakanksha Chowdhery, Brian Ichter, Ayzaan Wahid, Jonathan Tompson, Quan Vuong, Tianhe Yu, et al. 2023\.  Palm-e: An embodied multimodal language model.  _arXiv preprint arXiv:2303.03378_ (2023).
  * Fan et al. (2023) Angela Fan, Beliz Gokkaya, Mark Harman, Mitya Lyubarskiy, Shubho Sengupta, Shin Yoo, and Jie M Zhang. 2023.  Large Language Models for Software Engineering: Survey and Open Problems.  _arXiv preprint arXiv:2310.03533_ (2023).
  * Fast et al. (2018) Ethan Fast, Binbin Chen, Julia Mendelsohn, Jonathan Bassen, and Michael S Bernstein. 2018.  Iris: A conversational agent for complex tasks. In _Proceedings of the 2018 CHI conference on human factors in computing systems_. 1–12.
  * Fiannaca et al. (2023) Alexander J Fiannaca, Chinmay Kulkarni, Carrie J Cai, and Michael Terry. 2023.  Programming without a Programming Language: Challenges and Opportunities for Designing Developer Tools for Prompt Programming. In _Extended Abstracts of the 2023 CHI Conference on Human Factors in Computing Systems_. 1–7.
  * Gao et al. (2015) Tong Gao, Mira Dontcheva, Eytan Adar, Zhicheng Liu, and Karrie G Karahalios. 2015.  Datatone: Managing ambiguity in natural language interfaces for data visualization. In _Proceedings of the 28th annual acm symposium on user interface software & technology_. 489–500.
  * Gervasio et al. (2005) Melinda T Gervasio, Michael D Moffitt, Martha E Pollack, Joseph M Taylor, and Tomas E Uribe. 2005.  Active preference learning for personalized calendar scheduling assistance. In _Proceedings of the 10th international conference on Intelligent user interfaces_. 90–97.
  * Gou et al. (2021) Jianping Gou, Baosheng Yu, Stephen J Maybank, and Dacheng Tao. 2021.  Knowledge distillation: A survey.  _International Journal of Computer Vision_ 129 (2021), 1789–1819.
  * Grudin and Jacques (2019) Jonathan Grudin and Richard Jacques. 2019.  Chatbots, humbots, and the quest for artificial general intelligence. In _Proceedings of the 2019 CHI conference on human factors in computing systems_. 1–11.
  * Grussenmeyer and Folmer (2017) William Grussenmeyer and Eelke Folmer. 2017.  Accessible touchscreen technology for people with visual impairments: a survey.  _ACM Transactions on Accessible Computing (TACCESS)_ 9, 2 (2017), 1–31.
  * Huang et al. (2023) Tian Huang, Chun Yu, Weinan Shi, Bowen Wang, David Yang, Yihao Zhu, Zhaoheng Li, and Yuanchun Shi. 2023.  Interaction Proxy Manager: Semantic Model Generation and Run-Time Support for Reconstructing Ubiquitous User Interfaces of Mobile Services.  _Proc. ACM Interact. Mob. Wearable Ubiquitous Technol._ 7, 3, Article 99 (sep 2023), 39 pages.  https://doi.org/10.1145/3610929
  * Jain et al. (2018) Mohit Jain, Pratyush Kumar, Ramachandra Kota, and Shwetak N Patel. 2018.  Evaluating and informing the design of chatbots. In _Proceedings of the 2018 designing interactive systems conference_. 895–906.
  * Jiang et al. (2022) Ellen Jiang, Kristen Olson, Edwin Toh, Alejandra Molina, Aaron Donsbach, Michael Terry, and Carrie J Cai. 2022.  Promptmaker: Prompt-based prototyping with large language models. In _CHI Conference on Human Factors in Computing Systems Extended Abstracts_. 1–8.
  * Jiang et al. (2013) Jiepu Jiang, Wei Jeng, and Daqing He. 2013.  How do users respond to voice input errors? Lexical and phonetic query reformulation in voice search. In _Proceedings of the 36th international ACM SIGIR conference on Research and development in information retrieval_. 143–152.
  * Kaelbling et al. (1996) Leslie Pack Kaelbling, Michael L Littman, and Andrew W Moore. 1996.  Reinforcement learning: A survey.  _Journal of artificial intelligence research_ 4 (1996), 237–285.
  * Kaiser et al. (2019) Lukasz Kaiser, Mohammad Babaeizadeh, Piotr Milos, Blazej Osinski, Roy H Campbell, Konrad Czechowski, Dumitru Erhan, Chelsea Finn, Piotr Kozakowski, Sergey Levine, et al. 2019\.  Model-based reinforcement learning for atari.  _arXiv preprint arXiv:1903.00374_ (2019).
  * Kim et al. (2019b) Jihyun Kim, Meuel Jeong, and Seul Chan Lee. 2019b.  ” Why did this voice agent not understand me?” error recovery strategy for in-vehicle voice user interface. In _Proceedings of the 11th International Conference on Automotive User Interfaces and Interactive Vehicular Applications: Adjunct Proceedings_. 146–150.
  * Kim et al. (2019a) Yea-Seul Kim, Mira Dontcheva, Eytan Adar, and Jessica Hullman. 2019a.  Vocal shortcuts for creative experts. In _Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems_. 1–14.
  * Kiran et al. (2021) B Ravi Kiran, Ibrahim Sobh, Victor Talpaert, Patrick Mannion, Ahmad A Al Sallab, Senthil Yogamani, and Patrick Pérez. 2021.  Deep reinforcement learning for autonomous driving: A survey.  _IEEE Transactions on Intelligent Transportation Systems_ 23, 6 (2021), 4909–4926.
  * Lau et al. (2010) Tessa Lau, Julian Cerruti, Guillermo Manzato, Mateo Bengualid, Jeffrey P Bigham, and Jeffrey Nichols. 2010.  A conversational interface to web automation. In _Proceedings of the 23nd annual ACM symposium on User interface software and technology_. 229–238.
  * Lee et al. (2021) Sunjae Lee, Hayeon Lee, Hoyoung Kim, Sangmin Lee, Jeong Woon Choi, Yuseung Lee, Seono Lee, Ahyeon Kim, Jean Young Song, Sangeun Oh, Steven Y. Ko, and Insik Shin. 2021.  FLUID-XP: Flexible User Interface Distribution for Cross-Platform Experience. In _Proceedings of the 27th Annual International Conference on Mobile Computing and Networking_ (New Orleans, Louisiana) _(MobiCom ’21)_. Association for Computing Machinery, New York, NY, USA, 762–774.  https://doi.org/10.1145/3447993.3483245
  * Leshed et al. (2008) Gilly Leshed, Eben M Haber, Tara Matthews, and Tessa Lau. 2008.  CoScripter: automating & sharing how-to knowledge in the enterprise. In _Proceedings of the SIGCHI Conference on Human Factors in Computing Systems_. 1719–1728.
  * Li et al. (2023) Tao Li, Gang Li, Zhiwei Deng, Bryan Wang, and Yang Li. 2023.  A Zero-Shot Language Agent for Computer Control with Structured Reflection.  arXiv:2310.08740 [cs.CL]
  * Li et al. (2017) Toby Jia-Jun Li, Amos Azaria, and Brad A Myers. 2017.  SUGILITE: creating multimodal smartphone automation by demonstration. In _Proceedings of the 2017 CHI conference on human factors in computing systems_. 6038–6049.
  * Li et al. (2020a) Toby Jia-Jun Li, Jingya Chen, Haijun Xia, Tom M Mitchell, and Brad A Myers. 2020a.  Multi-modal repairs of conversational breakdowns in task-oriented dialogs. In _Proceedings of the 33rd Annual ACM Symposium on User Interface Software and Technology_. 1094–1107.
  * Li et al. (2018) Toby Jia-Jun Li, Igor Labutov, Xiaohan Nancy Li, Xiaoyi Zhang, Wenze Shi, Wanling Ding, Tom M Mitchell, and Brad A Myers. 2018.  Appinite: A multi-modal interface for specifying data descriptions in programming by demonstration using natural language instructions. In _2018 IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC)_. IEEE, 105–114.
  * Li et al. (2019) Toby Jia-Jun Li, Marissa Radensky, Justin Jia, Kirielle Singarajah, Tom M Mitchell, and Brad A Myers. 2019.  Pumice: A multi-modal agent that learns concepts and conditionals from natural language and demonstrations. In _Proceedings of the 32nd annual ACM symposium on user interface software and technology_. 577–589.
  * Li and Riva (2018) Toby Jia-Jun Li and Oriana Riva. 2018.  KITE: Building conversational bots from mobile apps. In _Proceedings of the 16th Annual International Conference on Mobile Systems, Applications, and Services_. 96–109.
  * Li et al. (2020b) Yang Li, Jiacong He, Xin Zhou, Yuan Zhang, and Jason Baldridge. 2020b.  Mapping natural language instructions to mobile UI action sequences.  _arXiv preprint arXiv:2005.03776_ (2020).
  * Little et al. (2007) Greg Little, Tessa A Lau, Allen Cypher, James Lin, Eben M Haber, and Eser Kandogan. 2007.  Koala: capture, share, automate, personalize business processes on the web. In _Proceedings of the SIGCHI conference on Human factors in computing systems_. 943–946.
  * Liu et al. (2023a) Michael Xieyang Liu, Advait Sarkar, Carina Negreanu, Benjamin Zorn, Jack Williams, Neil Toronto, and Andrew D Gordon. 2023a.  “What It Wants Me To Say”: Bridging the Abstraction Gap Between End-User Programmers and Code-Generating Large Language Models. In _Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems_. 1–31.
  * Liu et al. (2018) Thomas F. Liu, Mark Craft, Jason Situ, Ersin Yumer, Radomir Mech, and Ranjitha Kumar. 2018.  Learning Design Semantics for Mobile Apps. In _Proceedings of the 31st Annual ACM Symposium on User Interface Software and Technology_ (Berlin, Germany) _(UIST ’18)_. Association for Computing Machinery, New York, NY, USA, 569–579.  https://doi.org/10.1145/3242587.3242650
  * Liu et al. (2023b) Xiao Liu, Hao Yu, Hanchen Zhang, Yifan Xu, Xuanyu Lei, Hanyu Lai, Yu Gu, Hangliang Ding, Kaiwen Men, Kejuan Yang, et al. 2023b.  Agentbench: Evaluating llms as agents.  _arXiv preprint arXiv:2308.03688_ (2023).
  * Luger and Sellen (2016) Ewa Luger and Abigail Sellen. 2016.  ”Like Having a Really Bad PA” The Gulf between User Expectation and Experience of Conversational Agents. In _Proceedings of the 2016 CHI conference on human factors in computing systems_. 5286–5297.
  * Massimo et al. (2017) David Massimo, Mehdi Elahi, and Francesco Ricci. 2017.  Learning user preferences by observing user-items interactions in an IoT augmented space. In _Adjunct Publication of the 25th Conference on User Modeling, Adaptation and Personalization_. 35–40.
  * Murad et al. (2019) Christine Murad, Cosmin Munteanu, Benjamin R Cowan, and Leigh Clark. 2019.  Revolution or evolution? Speech interaction and HCI design guidelines.  _IEEE Pervasive Computing_ 18, 2 (2019), 33–45.
  * Murali et al. (2023) Prasanth Murali, Ian Steenstra, Hye Sun Yun, Ameneh Shamekhi, and Timothy Bickmore. 2023.  Improving multiparty interactions with a robot using large language models. In _Extended Abstracts of the 2023 CHI Conference on Human Factors in Computing Systems_. 1–8.
  * Myers (1986) Brad A Myers. 1986.  Visual programming, programming by example, and program visualization: a taxonomy.  _ACM sigchi bulletin_ 17, 4 (1986), 59–66.
  * OpenAI (2023) OpenAI. 2023.  GPT-4 Technical Report.  arXiv:2303.08774 [cs.CL]
  * Ouyang et al. (2022) Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022\.  Training language models to follow instructions with human feedback.  _Advances in Neural Information Processing Systems_ 35 (2022), 27730–27744.
  * Pan et al. (2023) Lihang Pan, Chun Yu, Zhe He, and Yuanchun Shi. 2023.  A Human-Computer Collaborative Editing Tool for Conceptual Diagrams. In _Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems_. 1–29.
  * Pan et al. (2022) Lihang Pan, Chun Yu, JiaHui Li, Tian Huang, Xiaojun Bi, and Yuanchun Shi. 2022.  Automatically generating and improving voice command interface from operation sequences on smartphones. In _Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems_. 1–21.
  * Pelikan and Broth (2016) Hannah RM Pelikan and Mathias Broth. 2016.  Why that nao? how humans adapt to a conventional humanoid robot in taking turns-at-talk. In _Proceedings of the 2016 CHI conference on human factors in computing systems_. 4921–4932.
  * Qu et al. (2019) Chen Qu, Liu Yang, Minghui Qiu, W Bruce Croft, Yongfeng Zhang, and Mohit Iyyer. 2019.  BERT with history answer embedding for conversational question answering. In _Proceedings of the 42nd international ACM SIGIR conference on research and development in information retrieval_. 1133–1136.
  * Rana and Bridge (2020) Arpit Rana and Derek Bridge. 2020.  Navigation-by-preference: a new conversational recommender with preference-based feedback. In _Proceedings of the 25th International Conference on Intelligent User Interfaces_. 155–165.
  * Ravindranath et al. (2012) Lenin Ravindranath, Arvind Thiagarajan, Hari Balakrishnan, and Samuel Madden. 2012.  Code in the air: simplifying sensing and coordination tasks on smartphones. In _Proceedings of the Twelfth Workshop on Mobile Computing Systems & Applications_. 1–6.
  * Rawles et al. (2023) Christopher Rawles, Alice Li, Daniel Rodriguez, Oriana Riva, and Timothy Lillicrap. 2023.  Android in the wild: A large-scale dataset for android device control.  _arXiv preprint arXiv:2307.10088_ (2023).
  * Ross et al. (2023) Steven I Ross, Fernando Martinez, Stephanie Houde, Michael Muller, and Justin D Weisz. 2023.  The programmer’s assistant: Conversational interaction with a large language model for software development. In _Proceedings of the 28th International Conference on Intelligent User Interfaces_. 491–514.
  * Seo et al. (2019) Minjoon Seo, Jinhyuk Lee, Tom Kwiatkowski, Ankur P Parikh, Ali Farhadi, and Hannaneh Hajishirzi. 2019.  Real-time open-domain question answering with dense-sparse phrase index.  _arXiv preprint arXiv:1906.05807_ (2019).
  * Sereshkeh et al. (2020) Alborz Rezazadeh Sereshkeh, Gary Leung, Krish Perumal, Caleb Phillips, Minfan Zhang, Afsaneh Fazly, and Iqbal Mohomed. 2020.  VASTA: a vision and language-assisted smartphone task automation system. In _Proceedings of the 25th international conference on intelligent user interfaces_. 22–32.
  * Shinn et al. (2023) Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik R Narasimhan, and Shunyu Yao. 2023.  Reflexion: Language agents with verbal reinforcement learning. In _Thirty-seventh Conference on Neural Information Processing Systems_.
  * Shvo et al. (2021) Maayan Shvo, Zhiming Hu, Rodrigo Toro Icarte, Iqbal Mohomed, Allan D Jepson, and Sheila A McIlraith. 2021.  AppBuddy: Learning to Accomplish Tasks in Mobile Apps via Reinforcement Learning.. In _Canadian Conference on AI_.
  * Srinivasan et al. (2019) Arjun Srinivasan, Mira Dontcheva, Eytan Adar, and Seth Walker. 2019.  Discovering natural language commands in multimodal interfaces. In _Proceedings of the 24th International Conference on Intelligent User Interfaces_. 661–672.
  * Stiennon et al. (2020) Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F Christiano. 2020.  Learning to summarize with human feedback.  _Advances in Neural Information Processing Systems_ 33 (2020), 3008–3021.
  * Sugiura and Koseki (1996) Atsushi Sugiura and Yoshiyuki Koseki. 1996.  Simplifying macro definition in programming by demonstration. In _Proceedings of the 9th annual ACM symposium on User interface software and technology_. 173–182.
  * Tandon et al. (2022) Niket Tandon, Aman Madaan, Peter Clark, and Yiming Yang. 2022.  Learning to repair: Repairing model output errors after deployment using a dynamic memory of feedback. In _Findings of the Association for Computational Linguistics: NAACL 2022_ , Marine Carpuat, Marie-Catherine de Marneffe, and Ivan Vladimir Meza Ruiz (Eds.). Association for Computational Linguistics, Seattle, United States, 339–352.  https://doi.org/10.18653/v1/2022.findings-naacl.26
  * Teevan et al. (2016) Jaime Teevan, Shamsi T Iqbal, Carrie J Cai, Jeffrey P Bigham, Michael S Bernstein, and Elizabeth M Gerber. 2016.  Productivity decomposed: Getting big things done with little microtasks. In _Proceedings of the 2016 CHI Conference Extended Abstracts on Human Factors in Computing Systems_. 3500–3507.
  * Thomason et al. (2015) Jesse Thomason, Shiqi Zhang, Raymond J Mooney, and Peter Stone. 2015.  Learning to interpret natural language commands through human-robot dialog. In _Twenty-Fourth International Joint Conference on Artificial Intelligence_.
  * Tu et al. (2019) Zhen Tu, Yali Fan, Yong Li, Xiang Chen, Li Su, and Depeng Jin. 2019.  From fingerprint to footprint: Cold-start location recommendation by learning user interest from app data.  _Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies_ 3, 1 (2019), 1–22.
  * Venkatesh et al. (2022) Sagar Gubbi Venkatesh, Partha Talukdar, and Srini Narayanan. 2022.  UGIF: UI Grounded Instruction Following.  _arXiv preprint arXiv:2211.07615_ (2022).
  * Veuskens et al. (2020) Tom Veuskens, Kris Luyten, and Raf Ramakers. 2020.  Rataplan: Resilient Automation of User Interface Actions with Multi-Modal Proxies.  _Proc. ACM Interact. Mob. Wearable Ubiquitous Technol._ 4, 2, Article 60 (jun 2020), 23 pages.  https://doi.org/10.1145/3397329
  * Vu et al. (2023) Minh Duc Vu, Han Wang, Zhuang Li, Gholamreza Haffari, Zhenchang Xing, and Chunyang Chen. 2023.  Voicify Your UI: Towards Android App Control with Voice Commands.  _Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies_ 7, 1 (2023), 1–22.
  * Wagner (2013) Amber Wagner. 2013.  Automation of VUIto GUI Mapping. In _CHI ’13 Extended Abstracts on Human Factors in Computing Systems_ (Paris, France) _(CHI EA ’13)_. Association for Computing Machinery, New York, NY, USA, 1941–1944.  https://doi.org/10.1145/2468356.2468706
  * Wang et al. (2023a) Bryan Wang, Gang Li, and Yang Li. 2023a.  Enabling conversational interaction with mobile ui using large language models. In _Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems_. 1–17.
  * Wang et al. (2023b) Yunlong Wang, Shuyuan Shen, and Brian Y Lim. 2023b.  RePrompt: Automatic Prompt Editing to Refine AI-Generative Art Towards Precise Expressions. In _Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems_. 1–29.
  * Ward and Issar (1994) Wayne Ward and Sunil Issar. 1994.  Recent improvements in the CMU spoken language understanding system. In _Human Language Technology: Proceedings of a Workshop held at Plainsboro, New Jersey, March 8-11, 1994_.
  * Wu et al. (2022) Xingjiao Wu, Luwei Xiao, Yixuan Sun, Junhang Zhang, Tianlong Ma, and Liang He. 2022.  A survey of human-in-the-loop for machine learning.  _Future Generation Computer Systems_ 135 (2022), 364–381.
  * Wulf et al. (2014) Linda Wulf, Markus Garschall, Julia Himmelsbach, and Manfred Tscheligi. 2014.  Hands free-care free: elderly people taking advantage of speech-only interaction. In _Proceedings of the 8th Nordic Conference on Human-Computer Interaction: Fun, Fast, Foundational_. 203–206.
  * Xu et al. (2020) Liang Xu, Hai Hu, Xuanwei Zhang, Lu Li, Chenjie Cao, Yudong Li, Yechen Xu, Kai Sun, Dian Yu, Cong Yu, et al. 2020\.  CLUE: A Chinese language understanding evaluation benchmark.  _arXiv preprint arXiv:2004.05986_ (2020).
  * Yang et al. (2023) Hui Yang, Sifu Yue, and Yunzhong He. 2023.  Auto-GPT for Online Decision Making: Benchmarks and Additional Opinions.  _arXiv preprint arXiv:2306.02224_ (2023).
  * Yankelovich (1996) Nicole Yankelovich. 1996.  How do users know what to say?  _interactions_ 3, 6 (1996), 32–43.
  * Yankelovich et al. (1995) Nicole Yankelovich, Gina-Anne Levow, and Matt Marx. 1995.  Designing SpeechActs: Issues in speech user interfaces. In _Proceedings of the SIGCHI conference on Human factors in computing systems_. 369–376.
  * Yao et al. (2022) Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. 2022.  React: Synergizing reasoning and acting in language models.  _arXiv preprint arXiv:2210.03629_ (2022).
  * Zamora (2017) Jennifer Zamora. 2017.  I’m sorry, dave, i’m afraid i can’t do that: Chatbot perception and expectations. In _Proceedings of the 5th international conference on human agent interaction_. 253–260.
  * Zhang et al. (2017) Xiaoyi Zhang, Anne Spencer Ross, Anat Caspi, James Fogarty, and Jacob O Wobbrock. 2017.  Interaction proxies for runtime repair and enhancement of mobile application accessibility. In _Proceedings of the 2017 CHI conference on human factors in computing systems_. 6024–6037.
  * Zhang and Wallace (2015) Ye Zhang and Byron Wallace. 2015.  A sensitivity analysis of (and practitioners’ guide to) convolutional neural networks for sentence classification.  _arXiv preprint arXiv:1510.03820_ (2015).
  * Zheng et al. (2023) Zibin Zheng, Kaiwen Ning, Jiachi Chen, Yanlin Wang, Wenqing Chen, Lianghong Guo, and Weicheng Wang. 2023.  Towards an understanding of large language models in software engineering tasks.  _arXiv preprint arXiv:2308.11396_ (2023).
  * Zhong et al. (2014) Yu Zhong, TV Raman, Casey Burkhardt, Fadi Biadsy, and Jeffrey P Bigham. 2014.  JustSpeak: enabling universal voice control on Android. In _Proceedings of the 11th Web for All Conference_. 1–4.
  * Zhou et al. (2020) Zhilan Zhou, Jian Xu, Aruna Balasubramanian, and Donald E. Porter. 2020.  A Survey of Patterns for Adapting Smartphone App UIs to Smart Watches. In _22nd International Conference on Human-Computer Interaction with Mobile Devices and Services_ (Oldenburg, Germany) _(MobileHCI ’20)_. Association for Computing Machinery, New York, NY, USA, Article 2, 11 pages.  https://doi.org/10.1145/3379503.3403564
  * Ziegler et al. (2019) Daniel M Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. 2019.  Fine-tuning language models from human preferences.  _arXiv preprint arXiv:1909.08593_ (2019).