# Chatting with GPT-3 for Zero-Shot Human-Like Mobile Automated GUI Testing

Zhe Liu1,3,Chunyang Chen4, Junjie Wang1,2,3,∗, Mengzhuo Chen1,3, Boyu Wu3,
Xing Che1,3, Dandan Wang1,3, Qing Wang1,2,3,∗ 1Laboratory for Internet
Software Technologies, 2State Key Laboratory of Computer Sciences, Science &
Technology on Integrated Information System LaboratoryInstitute of Software
Chinese Academy of Sciences, Beijing, China;
3University of Chinese Academy of Sciences, Beijing, China; ∗Corresponding
author
4Monash University, Melbourne, Australia<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract.

Mobile apps are indispensable for people’s daily life, and automated GUI
(Graphical User Interface) testing is widely used for app quality assurance.
There is a growing interest in using learning-based techniques for automated
GUI testing which aims at generating human-like actions and interactions.
However, the limitations such as low testing coverage, weak generalization,
and heavy reliance on training data, make an urgent need for a more effective
approach to generate human-like actions to thoroughly test mobile apps.
Inspired by the success of the Large Language Model (LLM), e.g., GPT-3 and
ChatGPT, in natural language understanding and question answering, we
formulate the mobile GUI testing problem as a Q&A task. We propose GPTDroid,
asking LLM to chat with the mobile apps by passing the GUI page information to
LLM to elicit testing scripts, and executing them to keep passing the app
feedback to LLM, iterating the whole process. Within it, we extract the static
context of the GUI page and the dynamic context of the iterative testing
process, design prompts for inputting this information to LLM, and develop a
neural matching network to decode the LLM’s output into actionable steps to
execute the app. We evaluate GPTDroid on 86 apps from Google Play, and its
activity coverage is 71%, with 32% higher than the best baseline, and can
detect 36% more bugs with faster speed than the best baseline. GPTDroid also
detects 48 new bugs on the Google Play with 25 of them being confirmed/fixed.
We further summarize the capabilities of GPTDroid behind the superior
performance, including semantic text input, compound action, long meaningful
test trace, and test case prioritization.

Automated GUI testing, Large language model

††copyright: none

##  1\. Introduction

Mobile apps have become increasingly popular over the past decade, with
millions of apps available for download from app stores like the Apple App
Store (App, 2023) and Google Play Store (Goo, 2023). With the rise of app
importance in our daily life, it has become increasingly critical for app
developers to ensure that their apps are of high quality and perform as
expected for users. To avoid time-consuming and labour-extensive manual
testing, automated GUI (Graphical User Interface) testing is widely used for
quality assurance of mobile apps (Mirzaei et al., 2016; Yang et al., 2018;
Yang et al., 2013; Machiry et al., 2013; Zeng et al., 2016; Mao et al., 2016)
i.e., dynamically exploring mobile apps by executing different actions such as
scrolling, clicking based on the program analysis to verify the app
functionality.

However, existing GUI testing tools such as probability-based or model-based
ones (Su et al., 2021; Kong et al., 2018; Rubinov and Baresi, 2018) suffer
from low testing coverage when testing practical commercial apps, meaning that
they may miss important bugs and issues. This is because of the complex and
dynamic nature of modern mobile apps (Rubin et al., 2015; Deka et al., 2017;
Pecorelli et al., 2022; Rubinov and Baresi, 2018; Kong et al., 2018; Fan et
al., 2018), which can have hundreds or even thousands of different screens,
each with its own unique set of interactions and possible user actions and
logic. In addition, test inputs generated by these methods are significantly
different from real users’ interaction traces (Peng et al., 2022), resulting
in the low testing coverage. To address these limitations, there has been a
growing interest in using deep learning (DL) (Li et al., 2019b; Dong et al.,
2020; Yang et al., 2022b; Yasin et al., 2021) and reinforcement learning (RL)
(Pan et al., 2020; Romdhana et al., 2022; Collins et al., 2021; Lv et al.,
2022) methods for automated mobile GUI testing. By learning from human
testers’ behavior, these methods aim to generate human-like actions and
interactions that can be used to test the app’s GUI more thoroughly and
effectively. These approaches are based on the idea that the more closely the
actions performed by the testing algorithm mimic those of a human user, the
more comprehensive and effective the testing will be.

Nevertheless, there are still some limitations with these DL and RL based GUI
testing methods. First, learning algorithms require large amounts of data
which is difficult to collect real-world users’ interactions. Second, learning
algorithms are designed to learn and predict from training data, so they may
not generalize well to the new, unseen situations, as apps are constantly
evolving and updating. Third, mobile apps can be non-deterministic, meaning
that the outcome of an action may not be the same every time, it is performed
(e.g., clicking the “delete” button from a list with the last content would
produce an empty list for which the delete button no longer works) which
specifically makes it difficult for RL algorithms to learn and make accurate
predictions. Therefore, another more effective approach to generate human-like
actions is highly needed to thoroughly test mobile apps.

The emerging Large Language Model (LLM) (Vaswani et al., 2017; Brown et al.,
2020; Chowdhery et al., 2022; Zhang et al., 2022) trained on ultra-large-scale
corpus, which shows promising performance in natural language understanding,
logical reasoning and question answering in recent years. For example, GPT-3
(Brown et al., 2020) (Generative Pre-trained Transformer-3) is one LLM from
OpenAI with 175 billion parameters trained on a massive dataset including
existing test scripts and bug reports, which makes it capable of understanding
and generating text across a wide range of topics and domains. The success of
ChatGPT111https://openai.com/blog/chatgpt/ based on GPT-3 demonstrates that
LLM can understand human knowledge and interact with humans as a knowledgeable
expert. Inspired by ChatGPT222ChatGPT cannot be directly applied for mobile
GUI testing, so we adopt GPT-3 which provides official API and easy-to-control
results., we formulate GUI testing problem as a questions & answering (Q&A)
task, i.e., asking the LLM to play a role as a human tester to test the target
app.

In detail, we propose a new approach, GPTDroid , asking LLM to chat with
mobile apps by passing the GUI page information to LLM to elicit testing
scripts, and executing them to keep passing the app feedback to LLM, iterating
the whole process. To convert the visual information of the app GUI into the
corresponding natural language description, we first extract the semantic
information of the app and GUI page by decompiling the target app and view
hierarchy files, as well as the dynamic context information of the iterative
testing process. Similar to the screenreader through which the blind interact
with mobile apps (Tal, 2023; Voi, 2023), we design linguistic patterns to
generate prompts for describing the current GUI page as the input to LLM,
which provides ways for LLM to interact with the app. Given the natural
language described answer from LLM, we decode it into actionable steps to
execute the target app by developing a neural matching network model.

Figure 1. Demonstrated example of how GPTDroid works.

Compared with conventional learning-based algorithms mentioned above, our
approach is based on LLM as a zero-shot tester that does not require any
training data or corresponding computational resources for training the model.
One example chat log can be seen in Figure 1. LLM can understand the app GUI,
and provide detailed actions to navigate the app (e.g., A1-A5 at Figure 1). To
compensate for its wrong prediction (A2 at Figure 1), the real-time feedback
by our approach guides it to regenerate the input until triggering a valid
page transition. It remains clear testing logic even after a long testing
trace to make complex reasoning of actions (A3, A4 at Figure 1), and it can
prioritize to test important functions earlier (e.g., A5 at Figure 1). A more
detailed analysis of our approach’s capability and reasons behind it is in
Section 6.

To evaluate the effectiveness of GPTDroid, we carry out an experiment on 86
popular Android apps in Google Play with 129 bugs. Compared with 9 common-used
and state-of-the-art baselines, GPTDroid can achieve more than 32% boost in
activity coverage than the best baseline, resulting in 71% activity coverage.
As GPTDroid can cover more activities, the method can detect 36% more bugs
with faster speed than the best baseline. Apart from the accuracy of our
GPTDroid, we also evaluate the usefulness of our GPTDroid by detecting unseen
crash bugs in real-world apps from Google Play. Among 216 apps, we obtain 48
crash bugs with 25 of them being confirmed and fixed by developers, while the
remaining are still pending. To reveal reasons behind the promising
performance of our approach, we further investigate the experiment results
qualitatively and summarize 4 findings in discussion including semantic text
input, compound action, long meaningful test trace, and test case
prioritization.

The contributions of this paper are as follows:

  * •

Vision. The first work to formulate the automated GUI testing problem to a
question & answering task by bringing LLM into GUI testing domain, as far as
we know.

  * •

Technique. A novel approach GPTDroid based on “pre-train, prompt and predict”
paradigm of the LLM by understanding the GUI semantic information and dynamic
context of the iterative testing process, for automatically inferring possible
operation steps.

  * •

Evaluation. Effectiveness and usefulness evaluation of GPTDroid in the real-
world apps with practical bugs detected.

  * •

Insight. Detailed qualitative analysis in discussion revealing the reasons why
LLM can generate human-like actions for app testing.

##  2\. Background

###  2.1. Android GUI and GUI event

For a mobile app, the user interface (UI) is the place where interactions
between humans and machines occur. App developers design UI to help users
understand the features of their apps, and users interact with the apps
through the UI. To help developers manipulate views flexibly, Android Software
Development Kit (SDK) (And, 2023) allows developers to build UI in Android
source code using the View and ViewGroup objects. The View objects are usually
called “widget” (e.g., ImageView, TextView), while the ViewGroup objects are
usually called “layout” which provides various layout structures (e.g.,
LinearLayout, RelativeLayout). Every widget and layout in the Android source
code has its specific attributes, which are used to set its boundaries,
clickable and referenced external resources, etc. During the running process,
developers can obtain the view hierarchy file corresponding to the current UI
page (screenshot) through “uiautomator dump” of the Android Debug Bridge (ADB)
command (And, 2023). The view hierarchy file includes the widget information
(coordinate information, ID, widget type, text description, etc.), and the
layout information on the current UI page (Vie, 2023), which can be used by
automated GUI testing tools to obtain the information about widgets.

The graphical user interface (GUI) is the most important type of UI for most
mobile apps, where apps present content and actionable widgets on the screen
and users interact with the widgets using actions (GUI event) such as clicks,
swipes, and text inputs. In the process of using apps, users often need
complex GUI events to jump from one page to another, such as filling in
multiple text input widgets, sliding widgets to the left to operate, and long
pressing widgets to delete.

###  2.2. Large Language Model & Prompt

Pre-trained Large Language Models (LLMs) have been shown effective in many
natural language processing (NLP) tasks. It is trained on ultra-large-scale
corpus and can understand the input prompts (sentences with prepending
instructions or a few examples) and generate reasonable text. When pre-trained
on billions of samples from the Internet, recent LLMs (like GPT-3 (Brown et
al., 2020), PaLM (Chowdhery et al., 2022) and OPT (Zhang et al., 2022)) encode
enough information to support many NLP tasks (Lucy and Bamman, 2021; Sharples,
2022; Yang et al., 2022a). GPT-3 (Brown et al., 2020) is one of the most
popular and powerful LLM which has great performance in many text generation
tasks. It is based on the transformer model (Vaswani et al., 2017) including
input embedding layers, masked multi-self attention, normalizaiton layers, and
feed-forward in Fig 2. Given a sentence, the input embedding layer encodes it
through the word embedding. The multi-self attention layer is used to divide a
whole high-dimensional space into several different subspaces to calculate the
similarity. The normalization layer is implemented through a normalization
step that fixes the mean and variance of each layer’s inputs. The feed-forward
layer compiles the data extracted by previous layers to form the final output.

Figure 2. The model structure of GPT-3.

Recently, prompt engineering has been proposed to close the gap between pre-
training and downstream tasks. Instead of designing a new training objective
for each downstream task, prompt engineering rewrites the input by adding a
natural language instruction such as “This is XX app, On its xxx page, it has
xxx. What to do next?” to reuse the masking objective for downstream tasks.
Formally, a popular prompt engineering employs a prompt template
$T_{prompt}(.)$ to convert the input $X$ to prompt input
$X_{prompt}=T_{prompt}(X)$. The prompt template is a textual string with
unfilled slots to fill the input $X$ and a question.

For automated GUI testing test script generation, the filled input $X$ is the
GUI information of the page, and LLM attempts to generate the operation steps
to be executed. Although achieving promising results in various NLP tasks, the
standard prompt is ineffective for automated GUI testing, because LLM cannot
understand the visual information of the GUI page or the source code of the
app. Therefore, this paper focuses on how to describe the GUI page information
to let LLM better understand, and how to decode the LLM’s feedback to
actionable operation steps to execute the app.

##  3\. Approach

We model the GUI testing as a Question & Answering (Q&A) problem, i.e., asking
the LLM to play a role as a human tester, and enabling the interactions
between the LLM and the app under testing. To realize this, we propose
GPTDroid, as demonstrated in Figure 3. It extracts the static and dynamic
context of the current GUI page, encodes them into prompt questions for LLM,
decodes LLM’s feedback answer into actionable operation scripts to execute the
app; and iterates the whole process. Armed with the knowledge learned from
large-scale training corpus, GPTDroid would have the potential to guide the
testing in exploring more diversified pages, conducting more complex
operational actions, and covering more meaningful operational sequences.

Figure 3. Overview of GPTDroid. Table 1. Extracted information and examples. Id |  Attribute |  Description |  Examples
---|---|---|---
Static context - App information
1 |  AppName |  Name of the app under testing |  AppName = “Money Tracker”
2 |  Activities |  List of names for all activities of the app, obtained from AndroidManifest.xml file |  Activities = [“Main”, “AddAccount”, “Import”, “Income”, …]
Static context - page GUI information
3 |  ActivityName |  Activity name of the current GUI page |  ActivityName = “AddPersonalInformation”
4 |  Widgets |  List of all widgets in current page, represented with text/id |  Widgets = [“Edit Account”, “btn_income”, …]
5 |  Position |  Relative position of widgets, obtained through their coordinates |  Upper = [“Welcome”, …], Lower = [“Add Income”, …]
Static context - widget information
6 |  WidgetText |  Widget text, obtained by field ‘text’ or ‘hint-text’ |  WidgetText = “Welcome to the Money Tracker!”
7 |  WidgetID |  Widget ID, obtained by field ‘resource-id’. |  WidgetID = “add_account”
8 |  WidgetCategory |  Category: TextView, EditText, ImageView, etc, obtained by field ‘class’ |  WidgetCategory = “TextView”
9 |  WidgetAction |  Widget action, obtained by field ‘clickable’, such as click, input, etc. |  WidgetAction = “Click”
10 |  NearbyWidget |  Nearby widgets, obtained by the text of parent widgets and sibling widgets |  NearbyWidget = “your income: [SEP] $ ”
Dynamic context
11 |  PageVisits |  Set of tested GUI pages with page visits number |  PageVisits = [{“Main”: “5” , “Account”: “3”, “Setting”: “1”, …}]
12 |  WidgetVisits |  Set of tested widgets of current GUI page with widget visits number |  WidgetVisits = [{“Income”: “2”, “Add”: “2”, “Delete”: “3”, …}]

Specifically, in each iteration of the testing, GPTDroid first obtains the
view hierarchy file of the mobile app and extracts the static context
including the app information, information of the current GUI page, and
details of each widget in the page. It also maintains a testing operation
memorizer, and extracts the latest dynamic information from it to indicate
current testing progress. Based on this contextual information, we design
linguistic patterns for generating the prompts as input of LLM, and the LLM
would output the operational steps described in natural language. We then
design a natural matching network to match the operational steps with the GUI
events (i.e., widgets) of the app to enable it to be automatically executed.

###  3.1. Context Extraction

Despite of its excellence on various tasks, the performance of LLM can be
significantly influenced by the quality of its input, i.e., whether the input
can precisely describe what to ask (Chen et al., 2022; Liao et al., 2022; Zhou
et al., 2022). In the scenario of this interactive mobile GUI testing, we need
to accurately depict the GUI page currently under test, as well as its
contained widgets information from a more micro perspective, and the app
information from a more macro perspective. Furthermore, to act like a human
tester, GPTDroid should also capture current testing progress so as to
recommend the testing operations from a more global viewpoint to potentially
cover more activities and avoid duplicate explorations. This section describes
which information will be extracted, organized into static context and dynamic
context to facilitate reading. And Section 3.2 will describe how we organize
this information into the style that LLM can better understand.

####  3.1.1. Static Context Extraction

Static context relates to the information of the app, the GUI page currently
tested, and all the widgets on the page. The app information is extracted from
the AndroidMaincast.xml file, while the other two types of information are
extracted from the view hierarchy file, which is obtainable with UIAutomator
(UIAutomator, 2021). Table 1 presents the summarized view of them.

App information provides the macro-level semantics of the app under testing,
which facilitates the LLM to gain a general perspective about the functions of
the app. The extracted information includes the name of the app and the name
of all its activities.

Page GUI information provides the semantics of the current page under testing
during the interactive process, which facilitates the LLM to capture the
current snapshot. We extract the activity name of the page, all the widgets
represented by the “text” field or “resource-id” field (the first non-empty
one in order), and the widget position of the page. For the position, inspired
by the screen reader (Zhang et al., 2021; Stangl et al., 2021; Wang et al.,
2021), we first obtain the coordinates of each widget in order from top to
bottom and from left to right, and the widgets whose ordinate is below the
middle of the page is marked as lower, and the rest is marked as upper.

Widget information denotes the micro-level semantics of the GUI page, i.e.,
the inherent meaning of all its widgets, which facilitates the LLM in
providing actionable operational steps related to these widgets. The extracted
information includes “text”, “hint-text”, and “resource-id” field (the first
non-empty one in order), “class” field, and “clickable” field. We also extract
the information from nearby widgets to provide a more thorough perspective,
which includes the “text” of parent node widgets and sibling node widgets.

Table 2. The example of linguistic patterns of prompts and prompt generation
rules.

Id |  Sample of linguistic patterns/rules |  Instantiation
---|---|---
Static context patterns:  $\langle StaticContext\rangle$
1 |  We want to test the ‘AppName’ App. It has the following activities, including ‘Activities’. |  We want to test “Money tracker” App. It has the following activities, including “Main”, “AddAccount”, “Import”, “Setting”, … .
2 |  The current page is ‘ActivityName’, it has ‘Widgets’. The upper part of the app is ‘Position’, the lower part is ‘Position’. |  The current page is “Main”, it has “Income”, “Add”, … . The upper part of the app is “Welcome to …, delete, …”, the lower part of the app is “Income, add, …”.
3 |  The widgets which can operation are ‘WidgetText / WidgetID’. ‘WidgetText / WidgetID’ is ‘WidgetCategory’ which can ‘WidgetAction’ and its nearby widget is ‘NearbyWidget’. |  The widgets which can operation are “add”, “delete”, … . “add” is Button which can click and its nearby widget is “Add account, …” , “delete” is TextView which can click and its nearby widget is … .
Dynamic context patterns:  $\langle DynamicContext\rangle$
4 |  We have tested ‘PageVisits’ with visit time. We have tested the ‘WidgetVisits’ on the current page. |  We have tested “Main: 7”, “About: 2”, “Account: 5”, … with visit time. We have tested the “Add: 1”, “Delete: 2”, “Edit the order: 1” on the current page.
Operation & feedback question patterns:  $\langle OperationQuestion\rangle$
5 |  What operation is required? |  What operation is required?
6 |  There is no ‘WidgetText / WidgetID’ on the current page, please reselect. |  There is no “Input” on the current page, please reselect.
Prompt generation rules
1 |  Start Prompt: $\langle StaticContext\rangle$[1,2,3] +  $\langle OperationQuestion\rangle$[5] |  We want to test “Money tracker” App, It has the following activities, including “Main”, “AddAccount”, “Exchange”, … . The current page is “Main”, it has “Income”, “Add”, … . The upper of the app is “Welcome to …, delete, …”, the lower of the app is “Income, add, …”. The “Income” is Button which can click and its nearby widget is ”Please input …”, … .  What operation is required?
2 |  Test Prompt: We successfully did the above operation.  $\langle DynamicContext\rangle$[4] +  $\langle StaticContext\rangle$[2,3] +  $\langle OperationQuestion\rangle$[5] |  We successfully did the above operation.  We have tested the “Main: 4” and “Exchange: 2” with visit time. We have tested “Add Exchange: 2” on the current page.  The current page is “AddAccount”, it has “Exchange”, “Edit”, … . The upper part of the app is “delete, Income info …”, the lower part of the app is … The “Exchange” is Button which can click and its nearby widget is ”Exchange the ….”.  What operation is required?
3 |  Feedback Prompt: Sorry,  $\langle OperationQuestion\rangle$[6] +  $\langle StaticContext\rangle$[3] +  $\langle OperationQuestion\rangle$[5] |  Sorry,  there is no “Enter” button on this page, please reselect.  The current page is “AddAccount”, it has “Exchange”, “Edit”, … . The “Exchange” is TextView which can click and its nearby is ”Exchange the ….”.  What operation is required?

  * •

Notes: “[1,2, …, 5]” means the id of each pattern.

####  3.1.2. Dynamic Context Extraction

Dynamic context relates to the detailed testing progress, which facilitates
the LLM being well aware of the process context and making informed decisions.
We design an operation memorizer to keep the record of this information, i.e.,
whether and how many times a GUI page has been explored or a widget has been
operated, as shown in Table 1.

Specifically, during the iteration, when an operation is conducted, we can
obtain the widget information of the operation, and the GUI pages information
after the operation, and then the operation memorizer is updated accordingly.
In detail, the visit number of the widget is updated through finding the same
widget in the operation memorizer by the “text” field and “resource-id” field
of the widget. The visit number of the GUI page is updated through finding the
same activity in the memorizer with the “ActivityName” field of the page.

###  3.2. Prompt Generation

With the extracted information, we design linguistic patterns to generate
prompts for inputting into the LLM.

We first conduct preprocessing for the extracted information, to facilitate
the follow-up design. For each static attribute in Table 1, we tokenize it by
the underscore and Camel Case (pas, 2023) (e.g. Capital letters of each word)
considering the naming convention in app development and remove the stop words
to reduce noise. We then conduct the part-of-speech (POS) tagging with
Standford NLP parser (De Marneffe and Manning, 2008), and only retain the
noun, verb and prepositions for the linguistic patterns.

####  3.2.1. Linguistic Patterns of Prompt

To design the patterns, each of the five authors is asked to write the prompt
sentence following regular prompt template (Chen et al., 2022; Cantino, 2016;
Gu et al., 2021), and questions the LLM for generating the operation steps.
He/she then checks to what extent the recommended operation is reasonable
considering the whole testing process. Each author can access a random-chosen
100 apps from Google play, and he/she can obtain the preprocessed static
context information and the dynamic context information. After 10 hours of
trial, he/she is required to provide the most promising and diversified 20
prompt sentences, which are served as the seeds for designing patterns. With
the prompt sentences, the five authors then conduct card sorting (Spencer,
2009) and discussion to derive the linguistic patterns. As shown in Table
3.1.1, this process comes out with 6 linguistic patterns corresponding with
the four sub-types of information in Table 1 and two operation & feedback
patterns.

Pattern related to static context: We design three patterns to describe the
overview of the GUI page currently under testing, respectively corresponding
to the app information, page GUI information, and widget information in Table
1.

Pattern related to dynamic context: We design one pattern to describe the
testing progress with the dynamic context as shown in Table 1.

Pattern related to operation & feedback question: We design two patterns to
describe the operational and feedback question. For the operational question,
we ask the LLM what operation is required. And for the feedback question,
after deciding the previous operation is not applicable (as described in
Section 3.3), we inform the LLM that there is no such widget on the current
page, and let it re-try.

####  3.2.2. Prompt Generation Rules:

Since the designed patterns describe information from different points of
view, we combine the patterns from different viewpoints and generate the
prompt rules as shown in Table 3.1.1. We design three kinds of prompts
respectively for starting the test, routine inquiry, and the feedback in case
of error occurred. Note that, due to the robustness of the LLM, the generated
prompt sentence does not need to follow the grammar completely.

Test prompt is the most commonly used prompt for informing the LLM of the
current status and query for the next operation. Specifically, we first tell
the LLM the dynamic context, i.e., how many times each GUI page and widget has
been explored; followed by the static context, i.e., the information about the
current GUI page and detailed widget information; then ask the LLM which
operation is required.

Feedback prompt is used for informing the LLM error occurred and re-try for
querying the next operation. Specifically, we first tell LLM its generation
operation cannot correspond to the widget on the page; re-provide it the
detailed widget information of the page and let the LLM recommend the
operation again.

Besides the above two kinds of prompts, we additionally design start prompt to
start the testing of the app. Different from the test prompt, it provides the
LLM with the information about the app with all activities for a global
overview; and it does not have dynamic context since the testing just begins.
This prompt is only used once since the LLM can somehow remember this global
app information during the testing process (Su et al., 2017; Li et al.,
2019b).

###  3.3. Operation Matching

After inputting the generated prompt, LLM would output a natural language
sentence describing the operation steps for the testing, e.g., click the save
button. We need to convert the natural language described operation steps to
the GUI events (i.e., widgets) of the app to enable it to be automatically
executed. This is non-trivial considering the natural language description can
be arbitrary, and inherently imprecise. We design a neural matching network to
predict which widget can be most likely to be mapped to the operation step.
Since training the neural network usually requires a large amount of labeled
data, we develop a heuristic-based automated training data generation method
to facilitate the model training in Section 3.3.2.

Figure 4. The matching network architecture.

####  3.3.1. Neural Matching Network

As shown in Figure 4, one input of the neural matching network is the LLM’s
feedback answer, which is the natural language described operation step
$C_{step}$, while the other input is the textual information $C_{text}$ of
app’s widget. We choose the first non-empty “text”, “ID”, and “description”
fields of the widget in turn. The operation step $C_{step}$ and the textual
information of the widget $C_{text}$ are concatenated with symbol
$\textless$SEP$\textgreater$, then input into the pre-trained transformer
encoder to generate the hidden state of the text. Finally, the textual hidden
state is input into a fully connected layer for obtaining the matching score
of the feedback answer and the widget.

During the iterative testing process, each time when the LLM provides the
feedback answer, GPTDroid would first separate the answer with “and”, “,”, “.”
to derive the atomic operation step, considering the compound operations can
be recommended by the LLM. Then for each atomic operation step, we compute the
matching score with all candidate widgets in the current GUI page, and choose
the widget with the highest matching score as the target widget. We then use
the WidgetAction attribute of the widget in Table 1 to performance the action
on the target widget for executing the app. Besides, if the matching score of
all widgets is less than 0.5, we determine there is no satisfied widget on the
page. It indicates an error occurred in the LLM’s feedback answer, and would
activate the feedback prompt in the next iteration.

####  3.3.2. Heuristic-based Training Data Generation

Training such a neural matching network requires a large amount of labeled
data with natural language described operation step and GUI event, e.g., press
the back button and corresponding event click back button in the app. However,
there is no such open dataset, and collecting it from scratch is time- and
effort-consuming. Meanwhile, by examining the operation step of LLM’s output,
we observe that they tend to follow certain linguistic patterns. This
motivates us in developing a heuristic-based automated training data
generation method for collecting the satisfied training data.

The primary idea is that for each interactive widget in a GUI page, we
heuristically generate the operation step which can operate on it for
transitioning to the next state; meanwhile, generate the negative data
instances between the operation step and the irrelevant widgets. To derive the
heuristic rules, the five authors examine 400 operation steps of LLM’s output,
summarizing the linguistic patterns for writing the operation steps. We have
summarized 31 non-repetitive operation descriptions and their variants, such
as click, enter, press, etc. We uploaded the complete table to our
websiteLABEL:github. For each iterative widget, we randomly generate three
different operation descriptions to serve as the positive data instances in
the training data. For the negative data instances, we follow the hard
negative mining strategy (Liu et al., 2016) to enhance the discriminability of
the model.

###  3.4. Implementation

Our GPTDroid is implemented as a fully automated GUI app testing tool, which
uses or extends the following tools: VirtualBox (vir, 2023) and the Python
library pyvbox (pyv, 2023) for running and controlling the Android-x86 OS,
Android UIAutomator (UIAutomator, 2021) for extracting the view hierarchy
file, and Android Debug Bridge (ADB) (Adb, 2023) for interacting with the app
under test (Section 3.1).

For the LLM (Section 3.2), we use the pre-trained GPT-3 model which was
released on the OpenAI website333https://beta.openai.com/docs/models/gpt-3.
The basic model of GPT-3 is the text-davinci-003 model which is extremely
powerful and good at answering questions. We build our neural matching network
model (Section 3.3) based on PyTorch (pyt, 2023) and Sentence Transformers
(Reimers and Gurevych, 2019). The text processing module is loaded with
DistilBert (Sanh et al., 2019), a 12-layer transformer-based pre-training
model. We use AdamW as the optimizer, BCEWithLogitsLoss as the loss function,
and train the model with the batch size set to 20.

##  4\. Effectiveness Evaluation

In order to verify the performance of GPTDroid, we evaluate it by
investigating the activity coverage (RQ1) and the number of detected bugs
(RQ2). In addition, we present the performance of operation matching in RQ3.
Note that, this section utilizes the previously-detected bugs in the app’s
repositories to demonstrate the effectiveness of our approach, and the next
section will further evaluate the usefulness of GPTDroid in detecting new
bugs.

###  4.1. Experimental Setup

The experimental dataset comes from two sources. The first is from the apps in
Themis benchmark (Su et al., 2021), which contains 20 open-source apps with 34
bugs in GitHub. Considering the small number of apps in the benchmark, we
collect a second dataset following similar procedures as the benchmark.

In detail, we crawl the 50 most popular apps of each category from Google Play
(Goo, 2023), and we keep the ones with at least one update after May. 2022,
resulting in 317 apps in 12 Google Play categories. Then, we use 9 common-used
and state-of-the-art automated GUI testing tools (details are in Section 4.2)
to run these apps in turn to ensure that they work properly. We then filter
out the unusable apps by the following criteria: (1) UIAutomator (UIAutomator,
2021) can’t obtain the view hierarchy file; (2) they would constantly crash on
the emulator; (3) one or more tools can’t run on them; (4) The registration
and login functions cannot be skipped with scripts (Su et al., 2021; Lv et
al., 2022; Dong et al., 2020); (5) They don’t have issue records or pull
requests on GitHub.

There are 66 apps (with 93 bugs) remaining for this effectiveness evaluation.
Note that, same as the benchmark, all bugs are crash bugs. Specifically, for
each app, we select the version in which the bugs are confirmed by developers
(merged GitHub pull requests) as our experimental data, following the practice
of the benchmark. The details of all 86 experimental apps (20 + 66) and
related bugs are shown in Table 3.

Table 3. Dataset of effectiveness evaluation. Statistics |  #Activities |  #Bugs |  #Download |  #Update
---|---|---|---|---
Min |  7 |  1 |  50K+ |  05/22
Max |  21 |  9 |  50M+ |  11/22
Median |  10 |  3 |  1M+ |  -
Average |  9 |  1.5 |  10M+ |  -
All |  790 |  129 |  - |  -

Note that, there are 101 apps that are filtered out for effectiveness
evaluation, yet can successfully run with our proposed approach. We apply them
to the manual prompt generation in Section 3.2.1 and heuristic training data
generation in Section 3.3.2. And this ensures there is no overlapping between
the apps in approach design and evaluation.

We employ activity coverage and the number of detected bugs, which are widely
used metrics for evaluating GUI testing (He et al., 2020; Liu et al., 2017;
Arnatovich et al., 2018). We also present the number of covered activities and
widgets which are also commonly-used metrics (Li et al., 2019b; Pan et al.,
2020; Su et al., 2021) in Table 4.

###  4.2. Baselines

To demonstrate the advantage of GPTDroid, we compare it with 9 common-used and
state-of-the-art baselines. We roughly divide the GUI testing approaches into
random-/rule-based methods, model-based methods, and learning-based methods,
to facilitate understanding.

For random-/rule-based methods, we use Monkey (Developers, 2012) and Droidbot
(Li et al., 2017). For model-based methods, we use Stoat (Su et al., 2017),
Ape (Gu et al., 2019), Fastbot (Cai et al., 2020), ComboDroid (Wang et al.,
2020), TimeMachine (Dong et al., 2020). For learning-based methods, we use
Humanoid (Li et al., 2019b) and Q-testing (Pan et al., 2020).

We deploy the baselines and our approach on a 64-bit Ubuntu 18.04 machine (64
cores, AMD 2990WX CPU, and 128GB RAM) and evaluate them on Google Android 7.1
emulators (API level 25). Each emulator is configured with 2GB RAM, 1GB
SDCard, 1GB internal storage, and X86 ABI image. Different types of external
files (including PNGs / MP3s / PDFs / TXTs / DOCXs) are stored on the SDCard
to facilitate file access from apps. Following common practice (Gu et al.,
2019; Li et al., 2017), we registered separate accounts for each bug that
requires login and wrote the login scripts, and during testing reset the
account data before each run to avoid possible interference. In order to
ensure fair and reasonable use of resources, we set up the running time of
each tool in one app to 30 minutes, which is widely used in other GUI testing
studies (Fan et al., 2018; Su et al., 2021; Gu et al., 2019; Li et al., 2017).
We run each tool three times and obtain the highest performance to mitigate
potential bias.

Table 4. Performance of activity coverage (RQ1). Metric | Random-/rule-based | Model-based | Learning-based
---|---|---|---
|  Monkey |  Droidbot |  Stoat |  Ape |  Fastbot |  ComboDroid |  TimeMachine |  Humanoid |  Q-testing |  GPTDroid
#Widgets |  351 |  893 |  1337 |  1582 |  1437 |  1388 |  1701 |  1453 |  1398 |  1989
#Activities |  104 |  269 |  333 |  370 |  391 |  383 |  401 |  340 |  323 |  477
Avg. activity coverage |  0.16 |  0.34 |  0.44 |  0.51 |  0.56 |  0.53 |  0.57 |  0.49 |  0.45 |  0.71

###  4.3. Results and Analysis

####  4.3.1. Performance of Activity Coverage (RQ1)

Table 4 shows the number of covered widgets, number of covered activities, and
average activity coverage of GPTDroid and the baselines. We can see that
GPTDroid covers far more widgets and activities than the baselines, and the
average activity coverage achieves 71% across the 86 apps. It is 32% (0.71 vs.
0.54) higher even compared with the best baseline (TimeMachine). This
indicates the effectiveness of GPTDroid in covering more activities and
widgets, thus bring higher confidence to the app quality and potentially
uncovering more bugs.

Figure 5. Activity coverage with varying time (RQ1).

Figure 5 additionally demonstrates the average activity coverage with varying
time. We can see that, in every time point, GPTDroid achieves higher activity
coverage than the baselines, and it achieves high coverage within about 13
minutes. This again indicates the effectiveness and efficiency of GPTDroid in
covering more activities with less time, which is valuable considering the
testing budget.

Among the baselines, the model-based and learning-based approaches have
relatively higher performance. Yet the model-based approaches cannot capture
the GUI semantic information and the exploration could not well understand the
inherent business logic of the app. In addition, existing learning-based
approaches only use few context information for guiding the exploration, and
the learners only have limited intelligence restricted by the model
architecture and amount of labeled training data.

We further analyze the potential reasons for the uncovered cases. First, some
widgets or inputs do not have meaningful “text” or “resource-id”, which
hinders the approach of effectively understanding the GUI page. Second, some
app requires specific operations, e.g., database connection, long press and
drag widgets to a fixed location, which is difficult if not impossible to be
automatically achieved.

####  4.3.2. Performance of Bug Detection (RQ2)

Figure 6 shows the overall number of detected bugs of GPTDroid and baselines
with varying times. GPTDroid detects 72 bugs for the 86 apps, 36% (72 vs. 53)
higher than the best baseline (Stoat). We also compare the similarities and
differences of the bugs between Stoat and our approach, and the results show
that its detected bugs are a subset of our detected bugs. This indicates the
effectiveness of GPTDroid in detecting bugs and helps to ensure app quality.

We can also see that, in every time point, GPTDroid detects more bugs than the
baselines, and reaches the highest value in about 17 minutes, saving 35% (17
vs. 26) of the testing time compared with the best baseline (also with more
detected bugs). This again proves its effectiveness and efficiency of
GPTDroid, which is valuable for saving more time for the follow-up bug fixing.
We will conduct a further discussion about the reason behind the superior
performance in Section 4.3.3.

Figure 6. Bug detection with varying time (RQ2).

####  4.3.3. Performance of Operation Matching (RQ3)

When testing the experimental apps in RQ1, we randomly choose 1,000 LLM’s
feedback answer, i.e., natural language described operations outputted by LLM,
and evaluate whether the operations can successfully match the correct GUI
widgets. Specifically, the two authors follow the principle of open-coding,
analyze the feedback answer and find the matching widget on the current page.
For the inconsistent labeling, the third author will judge until all the
authors reach an agreement.

Results show that the operation matching can achieve 0.96 accuracy, indicating
that most of the LLM’s feedback answers can be accurately matched to the GUI
widget. This lays a solid foundation for the high activity coverage of our
GPTDroid.

Figure 7. Examples of our finding.

##  5\. Usefulness Evaluation

###  5.1. Experimental Setup

This section further evaluates the usefulness of GPTDroid in detecting new
crash bugs. We employ a similar experimental setup to the previous section. To
make it brief, we only compare the best baselines for bug detection in the
last section, i.e., Droidbot, Stoat and Humanoid, the best one from each type
of methods (random-/rule-based methods, model-based methods and learning-based
methods).

We begin with the most popular and recently updated 317 apps from 12
categories as in the previous section. Then we reuse the five criteria in the
previous section for filtering the unusable apps. Differently, we loosen the
criteria 5, which only requires the app to have ways for bug reporting, since
the issue records or pull requests are not mandatory in this section. This
results in 216 apps for our usefulness evaluation. Note that, this section
aims at evaluating whether GPTDroid can detect new bugs on these apps, thus
the overlap between the apps of this section and the previous section is
allowed.

We use the same hardware and the software configurations as the previous
evaluation section. When the crash bugs are detected, we report them to the
app development team through online issue reports or email.

Table 5. confirmeded or fixed bugs Id |  APP Name |  Category |  Download |  Status
---|---|---|---|---
1 |  PerfectPia |  Music |  50M+ |  confirmed
2 |  MusicPlayer |  Music |  50M+ |  confirmed
3 |  NoxSecu |  Tool |  10M+ |  fixed
4 |  Degoo |  Tool |  10M+ |  fixed
5 |  Proxy |  Tool |  10M+ |  confirmed
6 |  Secure |  Tool |  10M+ |  confirmed
7 |  Thunder |  Tool |  10M+ |  confirmed
8 |  ApowerMir |  Tool |  5M+ |  confirmed
9 |  MediaFire |  Product |  5M+ |  confirmed
10 |  Postegro |  Commun |  500K+ |  fixed
11 |  Deezer MP |  Music |  500K+ |  fixed
12 |  MTG |  Utilities |  500K+ |  fixed
13 |  OFF |  Health |  500K+ |  confirmed
14 |  Yucata |  Tool |  500K+ |  confirmed
15 |  ClassySha |  Tool |  500K+ |  confirmed
16 |  Linphone |  Commun |  500K+ |  confirmed
17 |  Paytm |  Finance |  100K+ |  confirmed
18 |  Transdroid |  Tool |  100K+ |  confirmed
19 |  Transistor |  Music |  10K+ |  fixed
20 |  Onkyo |  Music |  10K+ |  fixed
21 |  Democracy |  News |  10K+ |  confirmed
22 |  NewPipe |  Media |  10K+ |  confirmed
23 |  LessPass |  Product |  10K+ |  confirmed
24 |  CEToolbox |  Medical |  10K+ |  confirmed
25 |  OSM |  Health |  10K+ |  fixed

###  5.2. Results and Analysis

For the 216 apps, GPTDroid detects 135 bugs involving 115 apps, of which 48
bugs involving 39 apps are newly-detected bugs. Furthermore, these new bugs
are not detected by the three baselines. We submit these 48 bugs to
developers, and 25 of them have been fixed/confirmed so far (8 fixed and 17
confirmed), while the remaining are still pending (none of them is rejected).
This further indicates the effectiveness of our proposed GPTDroid in bug
detection. Due to space limit, Table 5 presents these fixed/confirmed bugs,
and the full lists can be found on our websiteLABEL:github.

We further analyze the details of our found bugs, and 17 of them involve
multiple text inputs or compound operations. Besides, we also observe that
there are 11 bugs with more than 20 operation steps before triggering the bug,
counting from the MainActivity page, which indicates the ability of GPTDroid
in testing deeper features. Furthermore, we find at least 28 bugs related to
the main business logic of the app, for example, a bug about the health data
statistics is revealed for a digital health app.

##  6\. Discussion

Despite the superior performance of GPTDroid in the last section, it is still
unclear the reason behind it. To fully understand the testing capability of
the LLM, we carry out a qualitative study by investigating the cases in which
our model outperforms baselines. We summarize four kinds of capabilities
including low-level (i.e., valid text input, and compound actions) and high-
level ones (i.e., long meaningful test trace, and test case prioritization).
These findings pave the way for further research in this area.

Valid text inputs. Our approach can automatically fill in valid text content
to the input widget which is essentially the key for passing the page as seen
in Figure 7 (a). More importantly, our model can generate semantic text input
(e.g., income, date, ID number, searching item, etc) accordingly. Besides
single text input, it can also successfully fill in multiple input widgets at
the same time which are correlated to each other like the departure and
arrival cities and dates in the flight booking app. It may be due to GPT-3’s
language generation capabilities (Brown et al., 2020) which was well learned
during the training phase.

Compound actions. GPTDroid can conduct complex compound operations guided by
the LLM. As shown in the above example (Figure 7 (a)), to add the “Cable
crunch” information, it first inputs the text, selects the date, sets the
“SETS” and “REPS” by clicking the upper or lower button, then click the submit
button in the lower right corner. Since there are many tutorials or bug
reports on the web including natural-language descriptions of how certain
actions can lead to specific outcomes which may be adopted into GPT-3’s
training corpus. It may help with a better understanding of the causal
relationships between actions and outcomes, resulting in LLM’s compound action
ability in GUI testing.

Long meaningful test trace. GPTDroid can automatically generate the test cases
with a long sequence of operations which together accomplish a business logic
of the app, and this is quite important for covering the app features and
ensuring its quality. As shown in Figure 7 (b), in SmartMeter app (Sma, 2023),
to test a commonly-used app feature “delete equipment”, the automatic tool
first needs to click “find devices” in the device page, then select a device
(Bluetooth is turned on and there are candidate devices) and click “add
device” for adding it in the device page; input the related information and
click “start” to start the device; then turn off this device in the device
page, long press it and click “delete” from the pop-up menu. Only with this
long sequence of operations which touches the “deleting equipment” feature, a
crash can be revealed. It may be because of GPT-3’s exposure to tutorial or
bug reports which contain step-by-step instructions or descriptions of how to
trigger a certain feature or reproduce a certain bug (Su et al., 2021; Wang et
al., 2022) in the training corpus. Therefore, provided with low-level semantic
information (i.e., current GUI page) and high-level testing history, GPTDroid
can capture long-term dependencies among GUI pages for generating long
meaningful exploration sequences.

Test case prioritization. We also observe that GPTDroid usually prioritizes
testing the “important” widgets, which is valuable for reaching a higher
activity coverage and covering more key activities with relatively less time.
As shown in Figure 7(c), in the Main page of the Moni app (Mon, 2023), the
baseline tools tend to first click the “Setting” button following the
exploration order from upper to lower, which leads the testing easily trapped
into the setting page cycle. Our GPTDroid chooses to first test the
“AddIncome” activity, i.e., click the “add” button, which is based on the
semantics of the GUI page and app information, and can quickly explore the
activities related to the key features of the app. This may be because that
GPT-3’s training corpus contains a wide variety of software-related
information, including user manuals, release notes, and app/software
descriptions where developers often highlight the most important features at
the front.

##  7\. Related Work

###  7.1. Mobile App GUI and GUI Analysis.

GUI provides a visual bridge between apps and users, and is drawing increasing
attention (Speicher et al., 2015; Nebeling et al., 2013; Huang et al., 2021).
The graphical user interface (GUI) is the most important type of UI for most
mobile apps, where apps present content and actionable widgets on the screen
and users interact with the widgets using actions such as clicks, swipes, and
text inputs. GUI is an indispensable part of software on most major platforms
including Android. Analyzing the app’s GUI is of great interest to many
researchers and practitioners. Related studies included automatic GUI search
(Reiss et al., 2018; Behrang et al., 2018; Chen et al., 2019), GUI code
generation (Nguyen and Csallner, 2015; Chen et al., 2018; Moran et al.,
2018a), GUI changes detection and summarization (Moran et al., 2018b, c), GUI
design (Chen et al., 2021a; Yang et al., 2021), etc. Gao et al. (Gao et al.,
2017) and Li et al. (Li et al., 2019a) analyzed the possible problems in UI
rendering, and developed automatic approaches to detect them. Nayebi et al.
(Nayebi et al., 2012) and Holzinger et al. (Holzinger et al., 2012) found that
different resolutions of mobile devices have brought challenges in Android app
design and implementation. Huang et al. (Huang et al., 2014) and Rubin et al.
(Rubin et al., 2015) proposed to detect stealthy behaviors in Android apps by
comparing the actual behaviors with the UI. Chen et al. (Chunyang et al.,
2020) introduced a machine learning-based method to extract UI skeletons from
UI images, in order to facilitate GUI development. In human-computer
interaction research, software GUI is mainly used to mine UI design practices
(Kumar et al., 2013; Alharbi and Yeh, 2015) and interaction patterns (Deka et
al., 2016) at scale. The mined knowledge can further be used to guide UI and
UX (user experience) design. Due to these challenges, how to reasonably
summarize the app is a problem to be solved. Our focus is to use the
reasonable natural language to describe the GUI information of the app page,
so that the large language model can understand the GUI information.

###  7.2. Automated GUI testing

To ensure the quality of mobile apps, many researchers study the automatic
generation of large-scale test scripts to test apps (Xie and Memon, 2007).
Monkey (Developers, 2012) is the popular random-based automated GUI testing
tool, which emits pseudo-random streams of UI events and some system events.
It is easy to use and compatible with different Android versions. However, the
random-based testing strategy cannot formulate a reasonable testing path
according to the characteristics of the app, resulting in low test coverage.
To improve the test coverage, researchers propose model-based (Dong et al.,
2020; Mirzaei et al., 2016; Yang et al., 2018; Yang et al., 2013; Zeng et al.,
2016) automated GUI testing methods, design corresponding models through the
research and analysis of large-scale apps. Sapienz (Mao et al., 2016) used
genetic algorithms as the model and Stoat (Su et al., 2017) used the
stochastic model learned from an app to optimize test suite generation. Ape
(Gu et al., 2019) used the runtime information to dynamically evolve its
abstraction criterion via a decision tree and generated UI events via a random
and greedy depth-first state exploration strategy. ComboDroid (Wang et al.,
2020) obtained such use cases either from humans or automatically generates
from a GUI model constructed by GUI exploration and analyzed the data flow
between obtained use cases, and combined them to generate final tests.
Although model-based automated GUI testing tools can improve test coverage,
the coverage is still low because it does not consider the semantic
information of the app’s GUI and Page. Researchers further proposed human-like
testing strategies and designed learning-based automated GUI testing methods.
Humanoid (Li et al., 2019b) used a deep neural network model that predicts
which UI elements on the current GUI page are more likely to be interacted
with by users and how to interact with them. Q-testing (Pan et al., 2020) used
a reinforcement learning-based method to compare GUI pages and give rewards.
These rewards are used and iteratively updated to guide the testing to cover
more functionalities of apps. Although the learning-based approach can improve
the test coverage by learning a large number of interactive processes or using
the idea of reinforcement learning. However, it is still unable to better
understand the semantic information of the page and plan the path according to
the actual situation of the app, and is greatly affected by the training data.
This study aims at proposing a more effective approach to generate human-like
actions for thoroughly and more effectively testing the app, and accomplishing
it with LLM.

###  7.3. Large Language Model

Recently, there has been a great success of pre-trained Large Language Models
(e.g., RoBERTa (Liu et al., 2019), GPT-3 (Brown et al., 2020), PaLM (Chowdhery
et al., 2022), OPT (Zhang et al., 2022)) in a variety of NLP tasks. Due to the
large amounts of available pre-training data from the internet, research shows
that LLMs can already be used for very specific downstream tasks through the
new paradigm “pre-train, prompt and prediction” (Liu et al., 2023) without any
fine-tuning of special data sets. This paradigm for LLM was widely used in
many works and achieved state-of-the-art performance on downstream tasks
(Huang et al., 2022; Deng et al., 2022). The core of this paradigm is to use
prompt engineering (Liu et al., 2023; Branwen, 2020; Cantino, 2016; Ge and
Parikh, 2021), where a natural language description of the task is provided to
the LLM. Considering the powerful performance of LLM, researchers try to use
LLM to solve relevant tasks in the field of software engineering. Supported by
code naturalness (Hindle et al., 2016), researchers applied the LLMs to code
writing in different programming languages (Chen et al., 2021b; Feng et al.,
2020; Fried et al., 2022; Xu et al., 2022). Huang et al. (Huang et al., 2022)
used LLM for the type inference in statically-typed partial Code. In testing,
LLMFuzz (Deng et al., 2022) used LLMs to generate input programs for fuzzing
Deep Learning libraries. Xia et al. (Xia et al., 2022; Xia and Zhang, 2022)
applied LLM to automatic program repair to improve the accuracy of the
generated repair patches. Austin et al. (Austin et al., 2021) and Jain et al.
(Jain et al., 2022) used LLM for program synthesis in general-purpose
programming languages. This paper opens a new dimension for automated GUI
testing by formulating it as a Q&A task with LLM.

##  8\. Conclusion

As one of the most important quality assurance activities for mobile apps,
automated GUI testing has made much progress, yet still suffers from low
activity coverage and may miss critical bugs. This paper aims at generating
human-like actions to facilitate app testing more thoroughly and effectively.
Inspired by the success of LLM like ChatGPT, we formulate the GUI testing
problem as a Q&A task and propose GPTDroid. It extracts the static and dynamic
context of the current GUI page, encodes them into prompt question to ask the
LLM, decodes the LLM’s feedback answer into actionable operations to execute
the app, and iterates the whole process. Results on 86 popular apps
demonstrate that GPTDroid can achieve 71% activity coverage, with 32% higher
than the best baseline, and can detect 36% more bugs with faster speed than
the best baseline. GPTDroid also detects 48 new bugs on Google Play with 25 of
them being confirmed/fixed, with the remaining pending. The capability of
GPTDroid in generating semantic text input and compound actions, guiding to
explore the long meaningful test trace, and prioritizing test cases, further
proves the effectiveness and human-like aspects of our proposed GPTDroid.

In the future, we will fine-tune the LLM to improve the performance and
conduct a systematic study to understand reasons why LLM can help the GUI
testing.

## References

  * (1)
  * Adb (2023) 2023\.  Android Debug Bridge (adb).  https://developer.android.com/studio/command-line/adb.html#forwardports.
  * And (2023) 2023\.  Android development.  http://developer.android.com/reference/android.
  * App (2023) 2023\.  App Store.  https://www.apple.com.cn/app-store/.
  * Goo (2023) 2023\.  Google play.  https://play.google.com/store/apps/.
  * Mon (2023) 2023\.  Moni.  https://play.google.com/store/apps/details?id=Moni.
  * pas (2023) 2023\.  pascal case.  https://en.wikipedia.org/wiki/Camel_case.
  * pyt (2023) 2023\.  Pytorch.  https://pytorch.org/.
  * pyv (2023) 2023\.  pyvbox.  https://pypi.org/project/pyvbox/.
  * Sma (2023) 2023\.  SmartMeter.  https://play.google.com/store/apps/details?id=SmartMeter.
  * Tal (2023) 2023\.  Talkback.  https://play.google.com/store/apps/details?id=talkback.
  * Vie (2023) 2023\.  View hierachy.  https://developer.android.google.cn/topic/performance/.
  * vir (2023) 2023\.  virtualbox.  https://www.virtualbox.org/.
  * Voi (2023) 2023\.  VoiceOver.  https://play.google.com/store/apps/details?id=voiceover.
  * Alharbi and Yeh (2015) Khalid Alharbi and Tom Yeh. 2015.  Collect, decompile, extract, stats, and diff: Mining design pattern changes in Android apps. In _Proceedings of the 17th international conference on human-computer interaction with mobile devices and services_. 515–524.
  * Arnatovich et al. (2018) Yauhen Leanidavich Arnatovich, Lipo Wang, Ngoc Minh Ngo, and Charlie Soh. 2018.  Mobolic: An automated approach to exercising mobile application GUIs using symbiosis of online testing technique and customated input generation.  _Software: Practice and Experience_ 48, 5 (2018), 1107–1142.
  * Austin et al. (2021) Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, et al. 2021\.  Program synthesis with large language models.  _arXiv preprint arXiv:2108.07732_ (2021).
  * Behrang et al. (2018) Farnaz Behrang, Steven P Reiss, and Alessandro Orso. 2018\.  GUIfetch: supporting app design and development through GUI search. In _Proceedings of the 5th International Conference on Mobile Software Engineering and Systems_. ACM, 236–246.
  * Branwen (2020) Gwern Branwen. 2020\.  Gpt-3 creative fiction.  https://www.gwern.net/GPT-3.
  * Brown et al. (2020) Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020\.  Language models are few-shot learners.  _Advances in neural information processing systems_ 33 (2020), 1877–1901.
  * Cai et al. (2020) Tianqin Cai, Zhao Zhang, and Ping Yang. 2020.  Fastbot: A Multi-Agent Model-Based Test Generation System Beijing Bytedance Network Technology Co., Ltd.. In _Proceedings of the IEEE/ACM 1st International Conference on Automation of Software Test_. 93–96.
  * Cantino (2016) Andrew Cantino. 2016\.  Prompt Engineering Tips and Tricks with GPT-3.  https://blog.andrewcantino.com/blog/2021/04/21/prompt-engineering-tips-and-tricks/.
  * Chen et al. (2019) Chunyang Chen, Sidong Feng, Zhenchang Xing, Linda Liu, Shengdong Zhao, and Jinshui Wang. 2019\.  Gallery DC: Design Search and Knowledge Discovery through Auto-created GUI Component Gallery.  _CSCW_ (2019).
  * Chen et al. (2018) Chunyang Chen, Ting Su, Guozhu Meng, Zhenchang Xing, and Yang Liu. 2018.  From ui design image to gui skeleton: a neural machine translator to bootstrap mobile gui implementation. In _ICSE_. ACM.
  * Chen et al. (2021b) Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. 2021b.  Evaluating large language models trained on code.  _arXiv preprint arXiv:2107.03374_ (2021).
  * Chen et al. (2021a) Qiuyuan Chen, Chunyang Chen, Safwat Hassan, Zhengchang Xing, Xin Xia, and Ahmed E Hassan. 2021a.  How Should I Improve the UI of My App? A Study of User Reviews of Popular Apps in the Google Play.  _TOSEM_ 30, 3 (2021), 1–38.
  * Chen et al. (2022) Xiang Chen, Ningyu Zhang, Xin Xie, Shumin Deng, Yunzhi Yao, Chuanqi Tan, Fei Huang, Luo Si, and Huajun Chen. 2022.  Knowprompt: Knowledge-aware prompt-tuning with synergistic optimization for relation extraction. In _Proceedings of the ACM Web Conference 2022_. 2778–2788.
  * Chowdhery et al. (2022) Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. 2022\.  Palm: Scaling language modeling with pathways.  _arXiv preprint arXiv:2204.02311_ (2022).
  * Chunyang et al. (2020) Chen Chunyang, Feng Sidong, Liu Zhengyang, Xing Zhenchang, Zhao Shengdong, and Liu Linda. 2020\.  From Lost to Found: Discover Missing UI Design Semantics through Recovering Missing Tags. In _CSCW_.
  * Collins et al. (2021) Eliane Collins, Arilo Neto, Auri Vincenzi, and José Maldonado. 2021.  Deep reinforcement learning based Android application GUI testing. In _Proceedings of the XXXV Brazilian Symposium on Software Engineering_. 186–194.
  * De Marneffe and Manning (2008) Marie-Catherine De Marneffe and Christopher D Manning. 2008.  The Stanford typed dependencies representation. In _Coling 2008: proceedings of the workshop on cross-framework and cross-domain parser evaluation_. 1–8.
  * Deka et al. (2017) Biplab Deka, Zifeng Huang, Chad Franzen, Joshua Hibschman, Daniel Afergan, Yang Li, Jeffrey Nichols, and Ranjitha Kumar. 2017\.  Rico: A Mobile App Dataset for Building Data-Driven Design Applications. In _UIST_.
  * Deka et al. (2016) Biplab Deka, Zifeng Huang, and Ranjitha Kumar. 2016\.  ERICA: Interaction mining mobile apps. In _Proceedings of the 29th annual symposium on user interface software and technology_. 767–776.
  * Deng et al. (2022) Yinlin Deng, Chunqiu Steven Xia, Haoran Peng, Chenyuan Yang, and Lingming Zhang. 2022\.  Fuzzing Deep-Learning Libraries via Large Language Models.  _arXiv preprint arXiv:2212.14834_ (2022).
  * Developers (2012) Android Developers. 2012\.  Ui/application exerciser monkey.
  * Dong et al. (2020) Zhen Dong, Marcel Böhme, Lucia Cojocaru, and Abhik Roychoudhury. 2020.  Time-travel testing of android apps. In _ICSE_. IEEE.
  * Fan et al. (2018) Lingling Fan, Ting Su, Sen Chen, Guozhu Meng, Yang Liu, Lihua Xu, Geguang Pu, and Zhendong Su. 2018\.  Large-scale analysis of framework-specific exceptions in android apps. In _ICSE_. IEEE, 408–419.
  * Feng et al. (2020) Zhangyin Feng, Daya Guo, Duyu Tang, Nan Duan, Xiaocheng Feng, Ming Gong, Linjun Shou, Bing Qin, Ting Liu, Daxin Jiang, et al. 2020\.  Codebert: A pre-trained model for programming and natural languages.  _EMNLP_ (2020).
  * Fried et al. (2022) Daniel Fried, Armen Aghajanyan, Jessy Lin, Sida Wang, Eric Wallace, Freda Shi, Ruiqi Zhong, Wen-tau Yih, Luke Zettlemoyer, and Mike Lewis. 2022\.  Incoder: A generative model for code infilling and synthesis.  _arXiv preprint arXiv:2204.05999_ (2022).
  * Gao et al. (2017) Yi Gao, Yang Luo, Daqing Chen, Haocheng Huang, Wei Dong, Mingyuan Xia, Xue Liu, and Jiajun Bu. 2017\.  Every pixel counts: Fine-grained UI rendering analysis for mobile applications. In _IEEE INFOCOM 2017-IEEE Conference on Computer Communications_. IEEE, 1–9.
  * Ge and Parikh (2021) Songwei Ge and Devi Parikh. 2021.  Visual Conceptual Blending with Large-scale Language and Vision Models.  _arXiv preprint arXiv:2106.14127_ (2021).
  * Gu et al. (2019) Tianxiao Gu, Chengnian Sun, Xiaoxing Ma, Chun Cao, Chang Xu, Yuan Yao, Qirun Zhang, Jian Lu, and Zhendong Su. 2019.  Practical GUI testing of Android applications via model abstraction and refinement. In _ICSE_. IEEE.
  * Gu et al. (2021) Yuxian Gu, Xu Han, Zhiyuan Liu, and Minlie Huang. 2021\.  Ppt: Pre-trained prompt tuning for few-shot learning.  (2021).
  * He et al. (2020) Yuyu He, Lei Zhang, Zhemin Yang, Yinzhi Cao, Keke Lian, Shuai Li, Wei Yang, Zhibo Zhang, Min Yang, Yuan Zhang, et al. 2020\.  TextExerciser: feedback-driven text input exercising for android applications. In _2020 IEEE Symposium on Security and Privacy (SP)_. IEEE, 1071–1087.
  * Hindle et al. (2016) Abram Hindle, Earl T Barr, Mark Gabel, Zhendong Su, and Premkumar Devanbu. 2016.  On the naturalness of software.  _Commun. ACM_ 59, 5 (2016), 122–131.
  * Holzinger et al. (2012) Andreas Holzinger, Peter Treitler, and Wolfgang Slany. 2012\.  Making apps useable on multiple different mobile platforms: On interoperability for business application development on smartphones. In _International Conference on Availability, Reliability, and Security_. Springer, 176–189.
  * Huang et al. (2021) Huaxun Huang, Ming Wen, Lili Wei, Yepang Liu, and Shingchi Cheung. 2021.  Characterizing and Detecting Configuration Compatibility Issues in Android Apps. In _ASE_.
  * Huang et al. (2014) Jianjun Huang, Xiangyu Zhang, Lin Tan, Peng Wang, and Bin Liang. 2014.  Asdroid: Detecting stealthy behaviors in android applications by user interface and program behavior contradiction. In _ICSE_. 1036–1046.
  * Huang et al. (2022) Qing Huang, Zhiqiang Yuan, Zhenchang Xing, Xiwei Xu, Liming Zhu, and Qinghua Lu. 2022\.  Prompt-tuned Code Language Model as a Neural Knowledge Base for Type Inference in Statically-Typed Partial Code. In _ICSE_.
  * Jain et al. (2022) Naman Jain, Skanda Vaidyanath, Arun Iyer, Nagarajan Natarajan, Suresh Parthasarathy, Sriram Rajamani, and Rahul Sharma. 2022\.  Jigsaw: Large language models meet program synthesis. In _ICSE_.
  * Kong et al. (2018) Pingfan Kong, Li Li, Jun Gao, Kui Liu, Tegawendé F Bissyandé, and Jacques Klein. 2018.  Automated testing of android apps: A systematic literature review.  _IEEE Transactions on Reliability_ 68, 1 (2018), 45–66.
  * Kumar et al. (2013) Ranjitha Kumar, Arvind Satyanarayan, Cesar Torres, Maxine Lim, Salman Ahmad, Scott R Klemmer, and Jerry O Talton. 2013.  Webzeitgeist: design mining the web. In _Proceedings of the SIGCHI Conference on Human Factors in Computing Systems_. 3083–3092.
  * Li et al. (2019a) Wenjie Li, Yanyan Jiang, Chang Xu, Yepang Liu, Xiaoxing Ma, and Jian Lü. 2019a.  Characterizing and detecting inefficient image displaying issues in Android apps. In _2019 IEEE 26th International Conference on Software Analysis, Evolution and Reengineering (SANER)_. IEEE, 355–365.
  * Li et al. (2017) Yuanchun Li, Ziyue Yang, Yao Guo, and Xiangqun Chen. 2017\.  Droidbot: a lightweight ui-guided test input generator for android. In _ICSE_. IEEE.
  * Li et al. (2019b) Yuanchun Li, Ziyue Yang, Yao Guo, and Xiangqun Chen. 2019b.  Humanoid: a deep learning-based approach to automated black-box Android app testing. In _2019 34th IEEE/ACM International Conference on Automated Software Engineering (ASE)_. IEEE, 1070–1073.
  * Liao et al. (2022) Jinzhi Liao, Xiang Zhao, Jianming Zheng, Xinyi Li, Fei Cai, and Jiuyang Tang. 2022\.  PTAU: Prompt Tuning for Attributing Unanswerable Questions. In _Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval_. 1219–1229.
  * Liu et al. (2023) Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig. 2023\.  Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing.  _Comput. Surveys_ 55, 9 (2023), 1–35.
  * Liu et al. (2017) Peng Liu, Xiangyu Zhang, Marco Pistoia, Yunhui Zheng, Manoel Marques, and Lingfei Zeng. 2017\.  Automatic text input generation for mobile testing. In _ICSE_. IEEE.
  * Liu et al. (2016) Wei Liu, Dragomir Anguelov, Dumitru Erhan, Christian Szegedy, Scott Reed, Cheng-Yang Fu, and Alexander C Berg. 2016.  Ssd: Single shot multibox detector. In _European conference on computer vision_. Springer, 21–37.
  * Liu et al. (2019) Y. Liu, M. Ott, N. Goyal, J. Du, M. Joshi, D. Chen, O. Levy, M. Lewis, L. Zettlemoyer, and V. Stoyanov. 2019.  RoBERTa: A Robustly Optimized BERT Pretraining Approach.  (2019).  http://arxiv.org/abs/1907.11692
  * Lucy and Bamman (2021) Li Lucy and David Bamman. 2021.  Gender and representation bias in GPT-3 generated stories. In _Proceedings of the Third Workshop on Narrative Understanding_. 48–55.
  * Lv et al. (2022) Zhengwei Lv, Chao Peng, Zhao Zhang, Ting Su, Kai Liu, and Ping Yang. 2022.  Fastbot2: Reusable Automated Model-based GUI Testing for Android Enhanced by Reinforcement Learning. In _ICSE_.
  * Machiry et al. (2013) Aravind Machiry, Rohan Tahiliani, and Mayur Naik. 2013\.  Dynodroid: An input generation system for android apps. In _Proceedings of the 2013 9th Joint Meeting on Foundations of Software Engineering_. 224–234.
  * Mao et al. (2016) Ke Mao, Mark Harman, and Yue Jia. 2016.  Sapienz: Multi-objective automated testing for Android applications. In _Proceedings of the 25th International Symposium on Software Testing and Analysis_. 94–105.
  * Mirzaei et al. (2016) Nariman Mirzaei, Joshua Garcia, Hamid Bagheri, Alireza Sadeghi, and Sam Malek. 2016.  Reducing combinatorics in GUI testing of android applications. In _ICSE_. IEEE.
  * Moran et al. (2018a) Kevin Moran, Carlos Bernal-Cárdenas, Michael Curcio, Richard Bonett, and Denys Poshyvanyk. 2018a.  Machine Learning-Based Prototyping of Graphical User Interfaces for Mobile Apps.  _arXiv preprint arXiv:1802.02312_ (2018).
  * Moran et al. (2018b) Kevin Moran, Boyang Li, Carlos Bernal-Cárdenas, Dan Jelf, and Denys Poshyvanyk. 2018b.  Automated reporting of GUI design violations for mobile apps. In _ICSE_. ACM.
  * Moran et al. (2018c) Kevin Moran, Cody Watson, John Hoskins, George Purnell, and Denys Poshyvanyk. 2018c.  Detecting and Summarizing GUI Changes in Evolving Mobile Apps.  _ASE_ (2018).
  * Nayebi et al. (2012) Fatih Nayebi, Jean-Marc Desharnais, and Alain Abran. 2012\.  The state of the art of mobile application usability evaluation. In _CCECE_. IEEE, 1–4.
  * Nebeling et al. (2013) Michael Nebeling, Maximilian Speicher, and Moira C. Norrie. 2013.  W3touch: metrics-based web page adaptation for touch. In _2013 ACM SIGCHI Conference on Human Factors in Computing Systems, CHI ’13, Paris, France, April 27 - May 2, 2013_. ACM, 2311–2320.  https://doi.org/10.1145/2470654.2481319
  * Nguyen and Csallner (2015) Tuan Anh Nguyen and Christoph Csallner. 2015.  Reverse engineering mobile application user interfaces with remaui (t). In _ASE_. IEEE, 248–259.
  * Pan et al. (2020) Minxue Pan, An Huang, Guoxin Wang, Tian Zhang, and Xuandong Li. 2020.  Reinforcement learning based curiosity-driven testing of Android applications. In _Proceedings of the 29th ACM SIGSOFT International Symposium on Software Testing and Analysis_. 153–164.
  * Pecorelli et al. (2022) Fabiano Pecorelli, Gemma Catolino, Filomena Ferrucci, Andrea De Lucia, and Fabio Palomba. 2022\.  Software testing and Android applications: a large-scale empirical study.  _Empirical Software Engineering_ 27, 2 (2022), 1–41.
  * Peng et al. (2022) Chao Peng, Zhao Zhang, Zhengwei Lv, and Ping Yang. 2022\.  MUBot: Learning to Test Large-Scale Commercial Android Apps like a Human. In _2022 IEEE International Conference on Software Maintenance and Evolution (ICSME)_. IEEE, 543–552.
  * Reimers and Gurevych (2019) Nils Reimers and Iryna Gurevych. 2019.  Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks. In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing_. Association for Computational Linguistics.  http://arxiv.org/abs/1908.10084
  * Reiss et al. (2018) Steven P Reiss, Yun Miao, and Qi Xin. 2018.  Seeking the user interface.  _Automated Software Engineering_ 25, 1 (2018), 157–193.
  * Romdhana et al. (2022) Andrea Romdhana, Alessio Merlo, Mariano Ceccato, and Paolo Tonella. 2022.  Deep reinforcement learning for black-box testing of android apps.  _TOSEM_ (2022).
  * Rubin et al. (2015) Julia Rubin, Michael I Gordon, Nguyen Nguyen, and Martin Rinard. 2015.  Covert communication in mobile applications (t). In _2015 30th IEEE/ACM International Conference on Automated Software Engineering (ASE)_. IEEE, 647–657.
  * Rubinov and Baresi (2018) Konstantin Rubinov and Luciano Baresi. 2018.  What are we missing when testing our android apps?  _Computer_ 51, 4 (2018), 60–68.
  * Sanh et al. (2019) Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. 2019.  DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter.  _arXiv preprint arXiv:1910.01108_ (2019).
  * Sharples (2022) Mike Sharples. 2022\.  Automated Essay Writing: An AIED Opinion.  _International Journal of Artificial Intelligence in Education_ (2022), 1–8.
  * Speicher et al. (2015) Maximilian Speicher, Andreas Both, and Martin Gaedke. 2015\.  S.O.S.: Does Your Search Engine Results Page (SERP) Need Help?. In _CHI_. ACM.  https://doi.org/10.1145/2702123.2702568
  * Spencer (2009) Donna Spencer. 2009\.  _Card sorting: Designing usable categories_.  Rosenfeld Media.
  * Stangl et al. (2021) Abigale Stangl, Nitin Verma, Kenneth R Fleischmann, Meredith Ringel Morris, and Danna Gurari. 2021.  Going Beyond One-Size-Fits-All Image Descriptions to Satisfy the Information Wants of People Who are Blind or Have Low Vision. In _The 23rd International ACM SIGACCESS Conference on Computers and Accessibility_. 1–15.
  * Su et al. (2017) Ting Su, Guozhu Meng, Yuting Chen, Ke Wu, Weiming Yang, Yao Yao, Geguang Pu, Yang Liu, and Zhendong Su. 2017.  Guided, stochastic model-based GUI testing of Android apps. In _Proceedings of the 2017 11th Joint Meeting on Foundations of Software Engineering_. 245–256.
  * Su et al. (2021) Ting Su, Jue Wang, and Zhendong Su. 2021.  Benchmarking automated GUI testing for Android against real-world bugs. In _Proceedings of the 29th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering_. 119–130.
  * UIAutomator (2021) UIAutomator. 2021\.  Python wrapper of Android uiautomator test tool.  https://github.com/xiaocong/uiautomator.
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017.  Attention is all you need.  _Advances in neural information processing systems_ (2017).
  * Wang et al. (2021) Bryan Wang, Gang Li, Xin Zhou, Zhourong Chen, Tovi Grossman, and Yang Li. 2021\.  Screen2words: Automatic mobile UI summarization with multimodal learning. In _The 34th Annual ACM Symposium on User Interface Software and Technology_. 498–510.
  * Wang et al. (2022) Jue Wang, Yanyan Jiang, Ting Su, Shaohua Li, Chang Xu, Jian Lu, and Zhendong Su. 2022.  Detecting non-crashing functional bugs in Android apps via deep-state differential analysis. In _FSE_.
  * Wang et al. (2020) Jue Wang, Yanyan Jiang, Chang Xu, Chun Cao, Xiaoxing Ma, and Jian Lu. 2020\.  Combodroid: generating high-quality test inputs for android apps via use case combinations. In _ICSE_. 469–480.
  * Xia et al. (2022) Chunqiu Steven Xia, Yuxiang Wei, and Lingming Zhang. 2022\.  Practical Program Repair in the Era of Large Pre-trained Language Models.  _arXiv preprint arXiv:2210.14179_ (2022).
  * Xia and Zhang (2022) Chunqiu Steven Xia and Lingming Zhang. 2022.  Less training, more repairing please: revisiting automated program repair via zero-shot learning. In _Proceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering_. 959–971.
  * Xie and Memon (2007) Qing Xie and Atif M Memon. 2007.  Designing and comparing automated test oracles for GUI-based software applications.  _TOSEM_ (2007).
  * Xu et al. (2022) Frank F Xu, Uri Alon, Graham Neubig, and Vincent Josua Hellendoorn. 2022.  A systematic evaluation of large language models of code. In _Proceedings of the 6th ACM SIGPLAN International Symposium on Machine Programming_. 1–10.
  * Yang et al. (2021) Bo Yang, Zhenchang Xing, Xin Xia, Chunyang Chen, Deheng Ye, and Shanping Li. 2021\.  Don’t Do That! Hunting Down Visual Design Smells in Complex UIs against Design Guidelines. In _ICSE_. IEEE, 761–772.
  * Yang et al. (2018) Shengqian Yang, Haowei Wu, Hailong Zhang, Yan Wang, Chandrasekar Swaminathan, Dacong Yan, and Atanas Rountev. 2018.  Static window transition graphs for Android.  _Automated Software Engineering_ 25, 4 (2018), 833–873.
  * Yang et al. (2013) Wei Yang, Mukul R Prasad, and Tao Xie. 2013.  A grey-box approach for automated GUI-model generation of mobile applications. In _International Conference on Fundamental Approaches to Software Engineering_. Springer, 250–265.
  * Yang et al. (2022b) Yanming Yang, Xin Xia, David Lo, and John Grundy. 2022b.  A survey on deep learning for software engineering.  _ACM Computing Surveys (CSUR)_ 54, 10s (2022), 1–73.
  * Yang et al. (2022a) Zhengyuan Yang, Zhe Gan, Jianfeng Wang, Xiaowei Hu, Yumao Lu, Zicheng Liu, and Lijuan Wang. 2022a.  An empirical study of gpt-3 for few-shot knowledge-based vqa. In _Proceedings of the AAAI Conference on Artificial Intelligence_ , Vol. 36. 3081–3089.
  * Yasin et al. (2021) Husam N Yasin, Siti Hafizah Ab Hamid, and Raja Jamilah Raja Yusof. 2021\.  Droidbotx: Test case generation tool for android applications using Q-learning.  _Symmetry_ (2021).
  * Zeng et al. (2016) Xia Zeng, Dengfeng Li, Wujie Zheng, Fan Xia, Yuetang Deng, Wing Lam, Wei Yang, and Tao Xie. 2016\.  Automated test input generation for android: Are we really there yet in an industrial case?. In _Proceedings of the 2016 24th ACM SIGSOFT International Symposium on Foundations of Software Engineering_. 987–992.
  * Zhang et al. (2022) Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. 2022\.  Opt: Open pre-trained transformer language models.  _arXiv preprint arXiv:2205.01068_ (2022).
  * Zhang et al. (2021) Xiaoyi Zhang, Lilian de Greef, Amanda Swearngin, Samuel White, Kyle Murray, Lisa Yu, Qi Shan, Jeffrey Nichols, Jason Wu, Chris Fleizach, et al. 2021\.  Screen recognition: Creating accessibility metadata for mobile applications from pixels. In _Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems_. 1–15.
  * Zhou et al. (2022) Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. 2022\.  Learning to prompt for vision-language models.  _International Journal of Computer Vision_ (2022), 1–12.