# XUAT-Copilot: Multi-Agent Collaborative System for Automated User Acceptance
Testing with Large Language Model

Zhitao Wang∗, Wei Wang∗, Zirao Li, Long Wang, Can Yi, Xinjie Xu, Luyang Cao,
Hanjing Su, Shouzhi Chen, Jun Zhou  zhitaowang, dylanwwang, ziraoli,
<EMAIL_ADDRESS>WeChat Pay, TencentShenzhenChina

(2023)

###### Abstract.

In past years, we have been dedicated to automating user acceptance testing
(UAT) process of WeChat Pay, one of the most influential mobile payment
applications in China. A system titled XUAT has been developed for this
purpose. However, there is still a human-labor-intensive stage, i.e., test
scripts generation, in the current system. Therefore, in this paper, we
concentrate on methods of boosting the automation level of the current system,
particularly the stage of test scripts generation. With recent notable
successes, large language models (LLMs) demonstrate significant potential in
attaining human-like intelligence and there has been a growing research area
that employs LLMs as autonomous agents to obtain human-like decision-making
capabilities. Inspired by these works, we propose an LLM-powered multi-agent
collaborative system, named XUAT-Copilot, for automated UAT. The proposed
system mainly consists of three LLM-based agents responsible for action
planning, state checking and parameter selecting, respectively, and two
additional modules for state sensing and case rewriting. The agents interact
with testing device, make human-like decision and generate action command in a
collaborative way. The proposed multi-agent system achieves a close
effectiveness to human testers in our experimental studies and gains a
significant improvement of $Pass@1$ accuracy compared with single-agent
architecture. More importantly, the proposed system has launched in the formal
testing environment of WeChat Pay mobile app, which saves a considerable
amount of manpower in the daily development work.

Automated UAT, AI Agents, Large Language Model

††copyright: acmcopyright††journalyear: 2023††doi:
XXXXXXX.XXXXXXX††conference: 2023; 978-1-4503-XXXX-X/18/06††isbn:
;**footnotetext: These authors contributed equally to this work

##  1\. Introduction

User Acceptance Testing (UAT) is a critical phase in the software development
life cycle that involves assessing the functionality, usability, and
performance of a software application or system. The main goal of UAT is to
evaluate whether the developed software meets the predefined requirements from
the end-user’s perspective. It validates that the software is fit for purpose
and ready for deployment in a real-world environment. However, traditional UAT
methodologies, which rely heavily on manual testing, are time-consuming,
resource-intensive, and susceptible to human errors. It is thus non-trivial to
explore the way of automating UAT to improve efficiency, accuracy, and overall
software quality.

WeChat Pay111https://pay.weixin.qq.com/index.php/public/wechatpay_en, as one
of the most influential mobile payment applications in China, has become an
integral part of daily life for billions of users. Ensuring the seamless
functionality, usability, and compatibility of the application is of paramount
importance for maintaining customer satisfaction and trust. Therefore, in past
years, we have been dedicated to the development of an effective,
comprehensive and low-cost UAT system, named XUAT. The workflow of the XUAT
system is illustrated in Fig.1. At the initial stage, business requirements
are represented as use case flow charts, which are intended to ensure the
completeness, accuracy and conformity of the requirements. Given the flow
charts, the system enumerates all possible flows and transfers each flow as
one specific test case in semi-structured natural language. At the following
stage, these test cases are distributed to a number of human testers, who are
responsible for implementing test scripts consisting of a series of Android
Debug Bridge (ADB) commands to simulate end-users’ actions expected in the
test cases on the App (in this paper, we only focus on testing on Android
system). In this process, human testers often make trial and errors on test
devices to guarantee the script can be executed in expectation of the test
case accurately. The final scripts submitted by testers (also called
”persistent test scripts”) will be automatically ran in routine by the system
in the future and reports are recorded in the system. Based on the test
reports, a group of acceptance engineers are asked to validate the results of
test cases against the expectations of the requirements, and document any
defects, issues or improvements.

Figure 1. The overview of XUAT system and XUAT-Copilot system

Apparently, current version of XUAT is still a semi-automated system, which is
particularly labor-intensive at the stage of test script implementation.
Understanding contents of test cases and making trial and error are inevitable
in this human-engaged process. This motivates us to seek for more automated
and efficiency approaches of implementing test scripts.

Recent advancements in large language models (LLMs) like ChatGPT, have
demonstrated remarkable capabilities in not just linguistic tasks, but also
various general tasks especially embodied agents planning, where LLMs are
applied for making goal-driven decisions in interactive and embodied
environments (Song et al., 2022; Singh et al., 2023; Huang et al., 2022; Lin
et al., 2023). Test script generation in XUAT is also a task-oriented,
interactive and embodied scenario, where end-users are expected to be
represented by autonomous agents who automatically interact with the app
through ADB commands to achieve the goal defined in the test case step-by-
step. Inspired by the previous work, we are interested to investigate the
potential of employing LLM powered agents to automate the test script
generation process in the XUAT system.

Despite of the power of LLMs, this work still faces a set of unique
challenges:

  * •

Highly condensed expression. Each step of test cases is expressed in a highly
brief way but often implies a series of actions. For example, the step of
”Submitting Personal Information” expects a long action-sequence: clicking
name input box, entering text of name, clicking ID input box, entering ID
number, etc.

  * •

Context sensitive actions. Similar or even same contents ask for different
actions at different graphical user interface (GUI) pages or with different
expectations. For example, the step of ”Confirming the Information” indicates
a single action of clicking the confirmation button on some feedback pages,
but multiple actions of scrolling the screen, selecting the checkbox and
clicking the confirmation button on some agreement pages.

  * •

Many test parameters. In the WeChat Pay app, around 80 percent of actions
belong to the type of ”submitting information”, which indicates hundreds of
test parameters are maintained. In current system, a list of all test
parameters is provided and human testers need to select appropriate
parameters. This brings difficulties for LLM to recognize required parameters
without experience of human testers.

  * •

Step-by-step correctness. Different from random test methods, UAT needs to
strictly follow the provided test case such that the final goal can be
achieved. If one step is wrong, the whole test case will not be passed.
Therefore, it is crucial for agents to self-check step-by-step correctness of
the test case.

In this paper, we attempt to resolve above challenges and propose XUAT-
Copilot, an LLM-powered multi-agent collaborative system for automating UAT.
Fig 2 illustrates the proposed multi-agent collaborative system, which mainly
consists of three LLM-based agents responsible for action planning, state
checking and parameter selecting, respectively, and two additional modules for
state sensing and case rewriting. Specifically, Rewriting Module is applied to
make the contents of test cases more understandable by the agents. On the
other hand, Perception Module is utilized to provide precise and concise GUI
information for the agents due to context sensitivity of this scenario. The
three agents are associated in a collaborative way. Given the rewritten
current step instruction of the test case, Operation Agent is responsible for
planning expected user actions and generating corresponding action commands.
If the commands require parameters, Parameter Selection Agent is asked by the
operation agent for choosing appropriate parameters from hundreds of
candidates. With returned parameters, the operation agent regenerates action
commands and submits to execute them on testing devices simulating the
expected user actions. Since each step may involve a series of user actions,
Inspection Agent is devised to verify whether the state achieves the
expectation of current step. Generally, if the APP navigates to the GUI page
where actions of next step to be performed, it satisfies expectation of
current step. Therefore, the inspection agent makes a decision based on both
GUI information after actions and contents of next step.

Based on the above collaborative multi-agent architecture, the proposed system
achieves a close effectiveness to human testers in our experimental studies.
Compared with single-agent architecture, our multi-agent architecture gains a
significant improvement of Pass@1 and Complete@1 metrics. More importantly,
the proposed XUAT-Copilot system has launched in the formal testing
environment of WeChat Pay mobile app, which saves a considerable amount of
manpower in the daily development work.

Figure 2. The proposed multi-agent collaborative framework

The contributions of this paper could be concluded as follows:

  * •

To the best of our knowledge, we are the first to employ LLMs for UAT. We
demonstrate the possibility and feasibility of LLMs to automate the test
script generation in the UAT process.

  * •

We propose a novel multi-agent collaborative approach to UAT system that
harnesses the power of pre-trained LLMs for both high-level human language
instruction and low-level command planning and develop a range of innovative
approaches aimed at enhancing the understanding and planning capabilities of
LLMs in the context of the UAT scenario.

  * •

The proposed system has been applied to real research and development
environment of WeChat Pay, a mobile payment app with a billion-level user
base, with a significant improvement on testing efficiency.

##  2\. Background

###  2.1. Automated UAT and XUAT System

UAT stands for User Acceptance Testing. It is a phase in the software
development process where the end users or clients test the software to ensure
it meets their requirements and expectations. The main goal of UAT is to
validate that the software is functioning as intended and is ready for
deployment in a real-world environment. Automated UAT aims to reduce the
manual effort required to execute test cases, increase the efficiency and
accuracy of the testing process, and shorten the overall testing cycle. It
allows for faster feedback, easier identification of defects, and improved
consistency in testing compared to manual UAT.

XUAT system is an instantiation of automated UAT on the WeChat Pay mobile app.
As shown in Fig. 1, if business requirements are represented in predefined
semi-structure, the XUAT system could automatically generate the test cases.
However, the stage of transferring test cases to executable scripts requires
intensive human participation. Human testers initially write the script, i.e.,
a series of ADB commands, and make a run trial on the development test devices
to check whether the script simulates the expected actions and achieve the
goal of the given test case. If there are some errors, testers need to make
some revisions and a trial again, until return results satisfy the
expectations. The final scripts submitted by testers are called as ”persistent
test scripts”, since these scripts would be run in routine by the system and
test reports are recorded automatically. In order to improve the system’s
automation level, in this paper, we focus on the most labor-intensive stage,
i.e., test script implementation (generation), and apply large language models
to make this stage automated.

It should be noted that our work on automated UAT is very different from
existing works on automated graphical user interface (GUI) testing (Li et al.,
2019; Pan et al., 2020; Liu et al., 2023a, b). First, the two testing
processes have different purposes. Automated GUI testing is focused on testing
the graphical user interface of an application, while automated UAT aims to
validate the overall functionality and usability of the application from an
end-user perspective. Second, the two testing processes have distinct
paradigms. Automated GUI testing is often defined as a black-box task, thus
methods on this task pay much attention to exploration of all UI events under
a high degree of freedom. In contrast, automate UAT is a kind of white-box
task, where testing behaviors are strictly constrained by well-predefined test
cases such that the specific function points can be tested. Last, the two
testing processes concentrate on different testing metrics. Previous works on
automated GUI testing evaluate the performance by activity coverage, while UAT
performance, in this paper, is evaluated by pass rates of test cases.

###  2.2. Large Language Models

Large Language Models (LLMs) are the latest breakthroughs in natural language
processing. LLMs acquire the ability to achieve general-purpose language
understanding and generation by training on massive amounts of data to learn
billions of parameters. As autoregressive models, the training of LLMs is
taking an input text and repeatedly predicting the next token or word. Most
popular LLMs are based on the transformer architecture (Vaswani et al., 2017).

At the beginning of the emergence of LLMs, fine tuning was the only way make a
LLM could be adapt to specific tasks. Recent LLMs, such as GPT-3 (Brown et
al., 2020), GPT-4 (OpenAI, 2023), LLaMA (Touvron et al., 2023a, b) and PaLM
(Chowdhery et al., 2022; Anil et al., 2023), however, can achieve similar
task-specific results without requiring additional training through prompt-
engineering. Prompt engineering is the process of formulating a piece of text,
which often describes the task and can be understood by LLMs. Prompt
engineering is enabled by in-context learning (Dong et al., 2022), defined as
a model’s ability to temporarily learn from prompts, which is an emergent
ability of LLMs (Wei et al., 2022a). Based on the remarkable general
intelligence of LLMs, the application of in-context learning ability has
shifted to zero-shot learning, where LLMs make predictions by directly
describing the desired output. On the other hand, few-shot learning is
utilized to augment the context with a few examples of desired inputs and
outputs, which enables LLMs to recognize the input prompt syntax and patterns
of the output. Instead of few-shot learning, which easily reaches to token
limitation of LLMs, we apply zero-shot paradigm for the agents, which has
shown satisfactory performance in the UAT scenario.

Chain-of-thought (CoT) (Wei et al., 2022b) prompting is a technique that
allows LLM to solve a problem as a series of intermediate steps. CoT prompting
improves reasoning ability by inducing the model to answer a multi-step
problem with steps of reasoning that mimic a train of thought. Several follow-
up works have been extended to more sophisticated reasoning architecture
beyond simple prompting. For example, Selection-Inference (Creswell et al.,
2022) divides the reasoning process into two steps of “selection” and
“inference”. Tree-of-Thought (Yao et al., 2023) generalizes over CoT prompting
and encourages exploration over thoughts in a tree structure. In contrast,
ReAct (Yao et al., 2022b) integrates model actions and corresponding
observations into a coherent stream of inputs for the model to reason more
accurately and tackle tasks beyond reasoning. UAT is also a kind of
interactive decision making, therefore, in this paper, we employ a prompting
schema similar to the ReACT with more complex structure using personas and a
more comprehensive CoT prompting. Additionally, in order to reduce
hallucinations and inefficient planning, we apply self-reflection mechanism
(Shinn et al., 2023), which prompts agents to self-reflect on task feedback
and maintain reflective information in memory buffer to induce better
decision-making in subsequent trials.

###  2.3. Autonomous Agent and LLM-Based Agent

An early definition of autonomous agent is a system situated within and a part
of an environment that senses that environment and acts on it, over time, in
pursuit of its own agenda and so as to effect what it senses in the future
(Franklin and Graesser, 1996). Characteristics of ideal autonomous agents
include: (a) Autonomy: They can operate independently, without human
intervention, and make decisions based on their own internal logic; (b)
Perception: They can perceive their environment, using sensors, cameras, or
other input devices, and process the data; (c) Action: They can take actions
in the physical or digital world, using actuators, motors, or other output
devices, to achieve their goals; (d) Learning: They can learn from experience,
adapt to changing circumstances, and improve their performance over time; (e)
Communication: They can communicate with other agents, humans, or systems,
using various protocols and interfaces. Therefore, invention of autonomous
agents has long been recognized as a promising approach to achieving
artificial general intelligence (AGI).

In previous studies, autonomous agents are assumed to act based on policy
functions, and learned in isolated and restricted environments (Mnih et al.,
2015; Lillicrap et al., 2015; Schulman et al., 2017; Haarnoja et al., 2018).
The survey (Wang et al., 2023a) has pointed out that such assumptions
significantly differ from the human learning process, since individuals can
learn from a much wider variety of environments. Due to these gaps, the agents
learned from the previous studies are usually far from replicating human-level
decision processes, especially in unconstrained, open-domain settings. As
mentioned above, with notable successes, LLMs has demonstrated potential in
achieving human-like intelligence and leads to a trending research area that
employs LLMs to build autonomous agents to obtain human-like decision-making
capabilities. In a very short time, researchers have developed numerous agents
(Nakano et al., 2021; Yao et al., 2022a; Wang et al., 2023c; Park et al.,
2023; Schick et al., 2023; Shen et al., 2023; Deng et al., 2023; Wang et al.,
2023d; Qian et al., 2023). The key idea is equipping agents with crucial human
capabilities like memory and planning by prompting, tuning or augmenting LLMs
to complete various tasks. The initial inspiration of this paper comes from a
group of LLM-based embodied agents (Song et al., 2022; Singh et al., 2023;
Huang et al., 2022; Lin et al., 2023), which are able to make goal-driven
decisions and execute corresponding actions in interactive and embodied
environments.

A very recent series of research focus on developing LLM-powered multi-agent
system. Most common interaction schema in these works is ordered cooperation
(Chen et al., 2023; Wu et al., 2023; Hong et al., 2023; Talebirad and Nadiri,
2023). Different agents participate the task in an ordered sequence, i.e.,
agent of current step receives the output from previous agent and generate its
own output for next agent. Another schema is disordered cooperation, where
agents cooperates with each other in a complex way instead of a simple
pipeline and the cooperation structure may vary from to different tasks (Mandi
et al., 2023; Wang et al., 2023b). In contrast to cooperation, some works
design an adversarial interaction schema, which put agents in adversarial
relationship and push the agents to make advancements themselves (Chan et al.,
2023; Du et al., 2023). In this paper, we employ disordered cooperation schema
and carefully devise the interaction structure for the proposed multi-agent
system.

Figure 3. Image Processing Pipeline For Widgets With Hyperlink

##  3\. XUAT-Copilot

###  3.1. Problem Statement

As shown in Fig. 2, the system receives two types of inputs and interacts with
test environment through a predefined skill library. They are defined as
follows:

  * •

Test Case ($C$): Test case is an ordered sequence of instructions defined as
$C=\left[c_{1},...,c_{n}\right]$, where $n$ is total steps of the test case
$C$. Each piece of text instruction $c_{i}$ describes the expected goal
implying user actions at $i$th step of the test case.

  * •

Parameter List ($P$): Parameter list is associated with a specific test case
denoted as $P=\\{(p_{1},v_{1}),...,(p_{m},v_{m})\\}$, where $(p_{i},v_{i})$
denotes each possible test parameters and their corresponding values in the
test case $S$ and $m$ is the number of all parameters.

  * •

Environment ($E$): Environment in this scenario is a (virtual) mobile device,
which is represented as $E$. The system needs to continuously interactive with
the environment $E$ to execute actions as well as update the state of agents.

  * •

Skill Library ($F$): Skill library represents possible agents’ interactive
actions with the test environment, which are expected users’ operations on the
APP. We define a set of functions, which wrap ADB commands and simulate users’
actions in the environment, as the skill library:
$F=\\{f_{click},f_{scroll},...\\}$. The details of skill library will be
introduced at the following subsection.

Based on the above definitions, the problem that the proposed XUAT-Copilot
aims to solve is: given a raw test case $C$ associated with an associated
parameter list $P$, an interactive test environment $E$ and a predefined skill
library $F$, the goal of the proposed system is generate a sequence of actions
$A=\left[A_{1},...,A_{n}\right]$ by continuously interacting with the
environment $E$. Since there may be a series of actions for each step of the
test case, the generated $A_{i}$ at $i$th step is still a sequence of actions
$A_{i}=\left[a^{1}_{i}...,a^{l}_{i}\right]=\left[f^{1}_{i}(p_{i}^{1}),...,f^{l}_{i}(p_{i}^{l})\right]$,
where each operation $a_{i}^{j}$ can be specified as a call of skill function
$f_{i}^{j}\in F$ associated with a required parameter $p_{i}^{j}\in P$.

###  3.2. Perception Module

|  Raw Contents |  Rewritten Contents
---|---|---
Template |  System requests User to [PHRASE1], and System validates the result feedback from User is [PHRASE2]. |  [PHRASE2’]
Example 1 |  System requests User to select a bank for card-less binding, and System validates the result feedback from User is selecting bank. |  Select ${bank_name}
Example 2 |  System requests User to set payment password, and System validates the result feedback from User is submitting payment password. |  Submit payment password (Use numeric keyboard)
Table 1. Templates and Examples of Test Case Before and After Rewriting

Similar to the use of sensors in the field of robotics to observe the state of
the agents in the environment, we define a perception module to collect and
represent the testing state interactively. In this scenario, the state of the
agents in the environment is the GUI page where the current step is located in
the APP. Therefore, the function of the perception module is to collect
information from the current GUI page and represent it in a certain form.
Generally, there are two ways to represent App GUI:

  * •

GUI View Hierarchy: A view hierarchy file presenting objects of current GUI
page in XML format.

  * •

GUI Image: A screenshot image of current GUI page.

In this paper, we use information of both view hierarchy and screenshot image
for state perception.

The structured text representation of view hierarchy naturally suits LLM
Agents. Models such as GPT-3.5 and GPT-4 can understand view hierarchy very
well in the setting of in-context learning. The view hierarchy file includes
the widget information (coordinates, ID, widget type, text description, etc.),
and the layout information on the current GUI page. The state extractor could
easily obtain the view hierarchy file of current GUI page through “uiautomator
dump” of the Android Debug Bridge (ADB) command [2]. However, the raw view
hierarchy has too many tokens easily reaching to the limitation of LLMs.
Additionally, redundant information can interfere with LLM’s decision-making
and reduce accuracy. We define filtering rules in the state extractor to
extract concise and informative representation. For each widget node, six
attributes, i.e., class, resource-id, content-desc, text, clickable,
scrollable, bounds are extracted. class denotes the widget type, resource-id
is the ID of the widget, content-desc and text are text description of the
widget, and bounds represents the coordinates of the widget. Only widget nodes
with non-empty text or content-desc attribute are kept. Meanwhile, resource-id
is mocked if the original one is empty since it is very important for LLMs to
locate the widget. After filtering, the state is represented as a set of
widget node information: $s=\\{w_{1},...,w_{k}\\}$, where a widget node
$w_{i}$ is a string with following structure:
”<node class=, resource-id=, clickable=, scrollable= , content-desc=, text=,
bounds= node/>”.

There is a special kind of clickable widgets containing a piece of hyperlinked
text and valid operation for these widgets is clicking the hyperlinked text.
For example, in Fig.3, there are two widgets with hyperlinked text and their
original bounds defined in the view hierarchy are two rectangle areas
containing the whole paragraphs with the hyperlinked text. Given this
information, LLMs would infer to click the center point based on common sense,
but the operation would not be executed successfully since the hyperlinked
text are not located at the center of the widget. Therefore, in addition to
view hierarchy, we make use of screen images of GUI pages to extract correct
clickable bounds of these special widgets. The idea is illustrated as Fig.3.
Hyperlinked text always has a unified and unique color different from that of
normal text. Based on this prior knowledge, a preprocessing is designed to
extract coarse-grained regions of the hyperlinked text. The raw image is
firstly filtered and transferred from RGB color space to HSV color space that
allows for easy manipulation of hue, saturation, and value, making it suitable
for color selection in various applications. Afterwards, the image is
binarized by computing the differences between HSV values the image and those
of target color and then dilated to get rectangle-like bright regions of the
hyperlinked text. Subsequently, rectangle contours are detected in the
processed image and used to mask areas without hyperlink in the raw image.
Given the masked image, a detection model and a recognition model are used to
find the exact bounds, i.e., regions of interest (ROIs), of the hyperlinked
text, and recognize the text in each ROI, respectively. As for detection, a
pretrained SegLink++ (Tang et al., 2019) model is employed. SegLink++ adopts a
bottom-up approach, first detecting the attraction-repulsion relationships
between text blocks and text lines, then clustering the text blocks into
lines, and finally outputting the coordinates of the bounding boxes of the
text lines. Based on the detected bounding boxes, a ConvNeXts (Liu et al.,
2022) model is adopted for text recognition. The architecture of ConvNeXts
consists of a convolutional backbone and a series of ConvTransformer Blocks.
If a widget has more than one detected bounding boxes after image processing,
the coordinate of the bounding box with the largest width are used as the new
”bounds” attribute of the widget. The recognized text is also used to update
the ”content-desc” and ”text” attributes of the widget.

###  3.3. Rewriting Module

As defined above, a raw test case is an ordered sequence of instructions
$C=\left[c_{1},...,c_{n}\right]$. As shown in Table 1, each instruction
$c_{i}$ is a piece of text with a fixed template. This kind of expression
limits ”natural” degree of the instructions. According to the above example,
both [PHRASE1] and [PHRASE2] seem to describe the expected users’ actions,
this may make LLMs confused. In essence, [PHRASE1] in the template is designed
for expressing the ”process”, while [PHRASE2] is supposed to emphasis the
”result” of the actions. It is observed that [PHRASE1] often has a very
similar description to the title of the current UI page. In other words,
[PHRASE1] can be regarded as context information while [PHRASE2] can be
regarded as actual user actions expected by the test case. Therefore,
rewriting module extracts reformulate the instruction with the following
process:

  * •

Extracting [PHRASE1] and [PHRASE2] based on regular expression matching.

  * •

Rewriting [PHRASE2] as [PHRASE2’] with a series of rules based on the
information of original [PHRASE1] and [PHRASE2].

  * •

Use the rewritten [PHRASE2’] as the input instruction for the agents.

Due to the page limitation, the details of rewriting rules would not be
discussed in this paper. Two typical examples of rewriting are given in Table
1. The first example involves a specific parameter, thus the rewriting uses a
special mark to label the expression of the parameter. For the second example,
a rule of information supplementary is utilized. In the WeChat Pay APP, the
numeric keyboard pop-up on the screen is the only acceptable way for inputting
the payment password. But LLMs may not know this piece of specific knowledge,
thus the rewriting rule makes it more clear avoiding LLMs responds with an
illegal command.

###  3.4. Multi-Agent Collaborative Framework

The motivation behind the multi-agent framework is the limited capability of a
single LLM-based agent on this task. First, inputs of the task, e.g., filtered
hierarchy file or parameter list, are long text, which occupy the working
memory of the single agent in a large extent. This may easily lead the agent
to forgetting, hallucinating and inefficient planning due to the token
limitation of LLMs. Additionally, compared with other random-based black-box
testing, UAT on WeChat Pay requires for more complex and accurate actions,
e.g., selecting a given bank card or inputting a given ID number at correct
input widget. Therefore, in order to reduce the complexity of the task and the
length of the inputs given to the agents, we decompose the task as three sub-
tasks, i.e., action generation, parameter selection and state inspection, and
carefully devise the corresponding agents. Specifically, Operation Agent is
responsible for planning expected user actions and generating corresponding
action commands. Parameter Selection Agent has obligation of choosing
appropriate parameters from hundreds of candidates. Inspection Agent is
proposed to verify whether the state achieves the goal of current step.

Given : Instruction sequence of a test case $C=\left[c_{1},...,c_{n}\right]$,
parameter list $P$, language descriptions $D_{F}$ of skill library $F$, test
environment $E$, a backbone large language model $\pi$.

Output : Action sequence
$A=\left[f_{1}^{1}(p_{1}^{1}),...,f_{i}^{t}(p_{i}^{t}),...\right]$.

1 Initialize action sequence: $A\leftarrow\left[\right]$ ;

2 for  _$i=1,...,n$_ do

3 Initialize action count within step: $t\leftarrow 1$;

4 Initialize step goal accomplishment flag: $g\leftarrow 0$;

5 Initialize system memory: $M\leftarrow\emptyset$ ;

6 Initialize action sequence: $A_{i}\leftarrow\left[\right]$ ;

7 Observe state $s_{i}^{t}$ from environment $E$ ;

8 while  _$g\neq 1$_ do

9 $f_{i}^{t}\leftarrow\pi(M,D_{F},c_{i},s_{i}^{t},prompt_{op})$, $f_{i}^{t}\in
F$;

10 $p_{i}^{t}\leftarrow\pi(P,c_{i},f_{i}^{t},prompt_{para})$, $p_{i}^{t}\in
P$;

11 Execute action $f_{i}^{t}(p_{i}^{t})$ in environment $E$;

12 Initialize invalid history: $H\leftarrow\left[\right]$ ;

13 while  _$f_{i}^{t}(p_{i}^{t})$ is invalid_ do

14 $H\leftarrow H.\textup{append}(f_{i}^{t}(p_{i}^{t}))$;

15 $prompt^{\prime}_{op}\leftarrow prompt_{op}\cup prompt_{relf}\cup H$;

16 $f_{i}^{t}\leftarrow\pi(M,D_{F},c_{i},s_{i}^{t},prompt^{\prime}_{op})$,
$f_{i}^{t}\in F$;

17 $p_{i}^{t}\leftarrow\pi(P,c_{i},f_{i}^{t},prompt_{para})$, $p_{i}^{t}\in
P$;

18 Execute action $f_{i}^{t}(p_{i}^{t})$ in environment $E$;

19

20 end while

21 $A_{i}\leftarrow A_{i}.\textup{append}(f_{i}^{t}(p_{i}^{t}))$;

22 $t\leftarrow t+1$;

23 Observe state $s_{i}^{t}$ from environment $E$ ;

24 $g\leftarrow\pi(A_{i},c_{i+1},s_{i}^{t},prompt_{insp})$;

25 $M\leftarrow\pi(M,f_{i}^{t}(p_{i}^{t}),s_{i}^{t},g,prompt_{sum})$;

26

27 end while

28 $A\leftarrow A.\textup{append}(A_{i})$;

29

30 end for

return $A$

Algorithm 1 Multi-Agent Instruct2Act Skill Functions |  Descriptions
---|---
click(rid: str) |  Click a UI element specified by the resource id (rid) of view hierarchy
input_text(rid: str, text: str) |  Input the content string into a text box specified by the resource id of view hierarchy
input_by_numeric_keyboard(digits: str) |  Use the numeric keyboard to enter the digits
press_adb_back_key() |  Press back key on Android device, this is different from clicking back button on screen
scroll(direction: str) |  Scroll on screen
swipe_selector(rid: str, direction: str) |  Swipe a selector specified by specified by the resource id of view hierarchy
Table 2. Skill Library

####  3.4.1. Multi-Agent Instruct2Act Algorithm

The detailed collaborative mechanism of the proposed multi-agent system is
summarized as Alg.1. Apart from the above-mentioned definitions, the backbone
LLM that the three agents are founded on is denoted as $\pi$. At the beginning
of each step, the algorithm initializes the system memory $M$ and the
corresponding action sequence of this step $A_{i}$ as empty and sets the step
goal as unaccomplished state. Afterwards, the algorithm continuously generates
actions for the current step until the goal is accomplished. At each iteration
of action generation, the operation agent, which is designed by prompting the
backbone LLM $\pi$ with $prompt_{op}$, firstly generates a skill(operation)
function $f_{i}^{t}$ given the memory $M$, the descriptions of the skill
library $D_{F}$, the instruction of the current step $c_{i}$, and the state
$s_{i}^{t}$. Based on $c_{i}$ and the generated $f_{i}^{t}$, the parameter
selection agent is activated by prompting $\pi$ with $prompt_{para}$ and
select an appropriate parameter for $f_{i}^{t}$ from the parameter list $P$. A
complete action is the inferred operation function associated with the
selected parameter, i.e., $f_{i}^{t}(p_{i}^{t})$. However, actions generated
by LLM may not always be valid in the test environment, e.g., non-predefined
functions. Therefore, a self-reflection mechanism is introduced to make the
operation agent correct itself based on the reflection prompt $prompt_{relf}$
and the invalid history until a valid action $f_{i}^{t}(p_{i}^{t})$ is
generated. Afterwards, the action sequence of the current step $A_{i}$ is
appended with $f_{i}^{t}(p_{i}^{t})$, and a new state $s_{i}^{t}$ is captured.
Given the updated $A_{i}$, $s_{i}^{t}$ and the instruction of next step
$c_{i+1}$, the inspection agent makes a judgement on whether the step goal is
accomplished by prompting $\pi$ with $prompt_{insp}$. Subsequently, the
operation agent updates the memory $M$ by summarizing the conversation with
$prompt_{sum}$. If the step goal is accomplished, the algorithm will start a
new iteration for action generation of next step. It is worth mentioning that,
in spite of using different prompts, summarization and self-reflection are
still the behavior of the operation agent, since they continuously maintain
and use the common memory $M$ in one conversational session. Therefore, the
operation agent is the core controller of the system. After goals of all steps
are accomplished, the algorithm will return a full sequence of all recorded
actions as the final output.

####  3.4.2. Skill Library

A skill is defined as a high-level operation function wrapping ADB commands to
simulate users’ actions on the APP. Natural language descriptions are
carefully prompted to make these skills more understandable for the agent.
Skill functions and their descriptions are defined as Table 2. As for click,
input_text, swipe_selector, the operation agent needs to identify the resource
id of the target widget, which is an argument for these skill functions. When
using input functions, i.e., input_text and input_by_numeric_keyboard, the
operation agent asks the parameter selection agent for test parameters as
inputs of the two functions. We use the following template to embed the
information of skill library into the prompts of the operation agent:
”Exclusively use commands listed below, e.g. command_name. Commands:
[NAME]: [DESCRIPTION], args: ([ARGUMENT]: [TYPE], ...);
……
[NAME]: [DESCRIPTION], args: ([ARGUMENT]: [TYPE], ...).”

####  3.4.3. Prompt Structure

We use a shared prompt structure for consistency, yet with agent-specific
content. The prompt is composed of the following key components:

  * •

Profiling: indicates a specific role of the agent.

  * •

Task: describes detailed task of the agent including requirements and
constraints.

  * •

Capability: represents the agent’s available skills or prior knowledge.

  * •

Instruction: is rewritten instruction of a specific test case step.

  * •

Observation: denotes observed state (GUI information) in the environment.

  * •

Context: includes all useful context information, e.g., past dialog or
information from other agents.

  * •

Response Format: defines a specific JSON template of the agent response, which
has the following elements:

    * –

Reasoning: asks for a detailed reasoning process of the inferred answer.

    * –

Plan: (optional) requires for detailed action plan, which is only used for the
operation agent.

    * –

Answer: specifies the structure and elements of the answer. This varies among
agents.

For each agent, profiling, task, capability and response format are fixed all
the time, while instruction, observation and context are dynamically
collected.

####  3.4.4. Operation Agent

As the core of the system, operation agent is responsible for planning
expected user actions and generating corresponding action functions. The
contents of “Capability” component are skill functions and corresponding
descriptions. “Instruction” is the rewritten text of current step, and
“Observation” is the text representation of current GUI page. “Context”
component of operation agent is relatively complex, including:

  * •

Working Memory: past dialog and executed actions from previous rounds.

  * •

Invalid Actions: (optional) invalid actions and corresponding results at the
current step.

  * •

Reflection: (optional) prompts the agent to review invalid actions and provide
self reflection.

The expected response is an action through a calling of a specific skill
function, thus “Answer” of operation agent requires for skill function name,
corresponding arguments and natural language description of this action.

####  3.4.5. Parameter Selection Agent

Parameter selection agent is responsible for choosing appropriate parameters
from hundreds of candidates. “Capability” of parameter selection agent is the
parameter list. Since parameter selection agent only need to interact with
operation agent, “Observation” is empty and “Context” is information of
inferred skill function from operation agent. Only when inferred skills are
inputting functions, parameter selection agent is activated. “Answer” of
parameter selection agent specifies the structure should be the name of the
selected parameter.

####  3.4.6. Inspection Agent

Inspection agent aims at verifying whether the state achieves the goal of
current step. Different from operation agent, “Instruction” and “Observation”
of inspection agent are the content of next step and the information of GUI
page after actions, respectively. “Context” of inspection agent is the
sequence of performed actions of previous steps, Since the expected answer of
inspection agent is binary, i.e., goal-accomplished or not, “Answer” of
inspection agent is either ”yes” or ”no”.

##  4\. Experiments

###  4.1. Evaluation Settings

Number of Test Cases | Average Steps | Average Actions
---|---|---
450 | 7 | 15
Table 3. Statistics of Emperimental Data

##### Experimental Data

The objective of the proposed model is to generate correct scripts of test
cases in the UAT system. Test cases without bugs or other system problems are
sampled from the system as evaluation data. This means that all the
experimental cases have passed the UAT with human implemented scripts. The
statistics of the experimental data is summarized in Table 3.

##### Metrics

The main evaluation metrics are $Pass@1$ and $Complete@1$. $@1$ denotes
allowing the method to generate candidate script for each test case only once.
$Pass@1$ represents case-level pass rate, i.e., the ratio of successfully
passed cases among all test cases:

$Pass@1=\frac{\sum_{k=1}^{N}\mathbbm{1}[C^{k}\textup{ is passed}]}{N}$

where $C^{k}$ is $k$th test case in the experimental dataset.
$\mathbbm{1}(C^{k})$ is an indicator function returning 1 if the whole test
case $C^{k}$ is passed and 0 otherwise.

$Complete@1$ represents step-level complete rate, i.e., the average ratio of
successfully completed steps on all test cases:

$Complete@1=\frac{\sum_{k=1}^{N}\sum_{j=1}^{L_{k}}\mathbbm{1}[c_{j}^{k}\textup{
is passed}]}{\sum_{k=1}^{N}L_{k}}$

where $N$ is the total number of test cases, $L_{k}$ represents the length
(number of steps) of $k$th test case and $c_{j}^{k}$ denotes $j$th step of
$k$th test case. Similarly, $\mathbbm{1}(c_{j}^{k})$ is 1 if $c_{j}^{k}$ is
passed and 0 otherwise.

##### Compared Methods

In the literature, there is few research works or public methods for automated
UAT, particularly for the stage of test script generation. Therefore, in this
paper, we compare the proposed method with its variants.

  * •

Single Agent: this is a variant that only one agent is used and it should
fulfill all sub-tasks when generation actions. The prompt consists of all
agents’ prompts from XUAT-Copilot.

  * •

Without Reflection: this is a variant that reflection part is removed from the
prompt of operation agent.

###  4.2. Evaluation Performance

Methods | $Pass@1$ | $Complete@1$
---|---|---
Single Agent | 22.65% | 25.25%
Without Reflection | 81.96% | 89.39%
XUAT-Copilot | 88.55% | 93.03%
Table 4. Evaluation Performance

The evaluation performance is listed in Table 4. Compared with the single
agent version, XUAT-Copilot gains significant improvements on both metrics.
This demonstrates the superiority of the proposed multi-agent framework based
on LLM. As mentioned above, single agent cannot handle multiple tasks in the
UAT processes simultaneously, and long prompts may easily lead to forgetting,
hallucinating and inefficient planning. Additionally, the experimental results
prove the importance of reflections in prompts. When the reflection is
removed, the evaluation performance decreases.

##  5\. Conclusion

In this paper, we investigate the possibility and feasibility of LLMs to
automate the test script generation in the UAT process. A multi-agent system
XUAT-Copilot is proposed for this purpose. Three agents in the proposed system
handle different tasks for test script generation and collaborate with each
other. The superiority of the proposed system is demonstrated by the
experiments. The system has launched the formal testing environment of WeChat
Pay APP to improve the efficiency of test script generation. As for future
work, we plan to further improve the generalization ability of the system by
removing some rule-based parts and introducing more LLM agents for more
specific tasks.

## References

  * (1)
  * Anil et al. (2023) Rohan Anil, Andrew M Dai, Orhan Firat, Melvin Johnson, Dmitry Lepikhin, Alexandre Passos, Siamak Shakeri, Emanuel Taropa, Paige Bailey, Zhifeng Chen, et al. 2023\.  Palm 2 technical report.  _arXiv preprint arXiv:2305.10403_ (2023).
  * Brown et al. (2020) Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020\.  Language models are few-shot learners.  _Advances in neural information processing systems_ 33 (2020), 1877–1901.
  * Chan et al. (2023) Chi-Min Chan, Weize Chen, Yusheng Su, Jianxuan Yu, Wei Xue, Shanghang Zhang, Jie Fu, and Zhiyuan Liu. 2023\.  Chateval: Towards better llm-based evaluators through multi-agent debate.  _arXiv preprint arXiv:2308.07201_ (2023).
  * Chen et al. (2023) Guangyao Chen, Siwei Dong, Yu Shu, Ge Zhang, Jaward Sesay, Börje F Karlsson, Jie Fu, and Yemin Shi. 2023\.  AutoAgents: A Framework for Automatic Agent Generation.  _arXiv preprint arXiv:2309.17288_ (2023).
  * Chowdhery et al. (2022) Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. 2022\.  Palm: Scaling language modeling with pathways.  _arXiv preprint arXiv:2204.02311_ (2022).
  * Creswell et al. (2022) Antonia Creswell, Murray Shanahan, and Irina Higgins. 2022\.  Selection-inference: Exploiting large language models for interpretable logical reasoning.  _arXiv preprint arXiv:2205.09712_ (2022).
  * Deng et al. (2023) Xiang Deng, Yu Gu, Boyuan Zheng, Shijie Chen, Samuel Stevens, Boshi Wang, Huan Sun, and Yu Su. 2023\.  Mind2Web: Towards a Generalist Agent for the Web.  _arXiv preprint arXiv:2306.06070_ (2023).
  * Dong et al. (2022) Qingxiu Dong, Lei Li, Damai Dai, Ce Zheng, Zhiyong Wu, Baobao Chang, Xu Sun, Jingjing Xu, and Zhifang Sui. 2022.  A survey for in-context learning.  _arXiv preprint arXiv:2301.00234_ (2022).
  * Du et al. (2023) Yilun Du, Shuang Li, Antonio Torralba, Joshua B Tenenbaum, and Igor Mordatch. 2023.  Improving Factuality and Reasoning in Language Models through Multiagent Debate.  _arXiv preprint arXiv:2305.14325_ (2023).
  * Franklin and Graesser (1996) Stan Franklin and Art Graesser. 1996.  Is it an Agent, or just a Program?: A Taxonomy for Autonomous Agents. In _International workshop on agent theories, architectures, and languages_. Springer, 21–35.
  * Haarnoja et al. (2018) Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. 2018.  Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In _International conference on machine learning_. PMLR, 1861–1870.
  * Hong et al. (2023) Sirui Hong, Xiawu Zheng, Jonathan Chen, Yuheng Cheng, Ceyao Zhang, Zili Wang, Steven Ka Shing Yau, Zijuan Lin, Liyang Zhou, Chenyu Ran, et al. 2023\.  Metagpt: Meta programming for multi-agent collaborative framework.  _arXiv preprint arXiv:2308.00352_ (2023).
  * Huang et al. (2022) Wenlong Huang, Pieter Abbeel, Deepak Pathak, and Igor Mordatch. 2022.  Language models as zero-shot planners: Extracting actionable knowledge for embodied agents. In _International Conference on Machine Learning_. PMLR, 9118–9147.
  * Li et al. (2019) Yuanchun Li, Ziyue Yang, Yao Guo, and Xiangqun Chen. 2019\.  Humanoid: A deep learning-based approach to automated black-box android app testing. In _2019 34th IEEE/ACM International Conference on Automated Software Engineering (ASE)_. IEEE, 1070–1073.
  * Lillicrap et al. (2015) Timothy P Lillicrap, Jonathan J Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. 2015.  Continuous control with deep reinforcement learning.  _arXiv preprint arXiv:1509.02971_ (2015).
  * Lin et al. (2023) Kevin Lin, Christopher Agia, Toki Migimatsu, Marco Pavone, and Jeannette Bohg. 2023.  Text2motion: From natural language instructions to feasible plans.  _arXiv preprint arXiv:2303.12153_ (2023).
  * Liu et al. (2023a) Zhe Liu, Chunyang Chen, Junjie Wang, Xing Che, Yuekai Huang, Jun Hu, and Qing Wang. 2023a.  Fill in the blank: Context-aware automated text input generation for mobile gui testing. In _2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE)_. IEEE, 1355–1367.
  * Liu et al. (2023b) Zhe Liu, Chunyang Chen, Junjie Wang, Mengzhuo Chen, Boyu Wu, Xing Che, Dandan Wang, and Qing Wang. 2023b.  Chatting with GPT-3 for Zero-Shot Human-Like Mobile Automated GUI Testing.  _arXiv preprint arXiv:2305.09434_ (2023).
  * Liu et al. (2022) Zhuang Liu, Hanzi Mao, Chao-Yuan Wu, Christoph Feichtenhofer, Trevor Darrell, and Saining Xie. 2022\.  A convnet for the 2020s. In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_. 11976–11986.
  * Mandi et al. (2023) Zhao Mandi, Shreeya Jain, and Shuran Song. 2023.  Roco: Dialectic multi-robot collaboration with large language models.  _arXiv preprint arXiv:2307.04738_ (2023).
  * Mnih et al. (2015) Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. 2015\.  Human-level control through deep reinforcement learning.  _nature_ 518, 7540 (2015), 529–533.
  * Nakano et al. (2021) Reiichiro Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu, Long Ouyang, Christina Kim, Christopher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, et al. 2021\.  Webgpt: Browser-assisted question-answering with human feedback.  _arXiv preprint arXiv:2112.09332_ (2021).
  * OpenAI (2023) OpenAI. 2023.  GPT-4 Technical Report.  arXiv:2303.08774 [cs.CL]
  * Pan et al. (2020) Minxue Pan, An Huang, Guoxin Wang, Tian Zhang, and Xuandong Li. 2020.  Reinforcement learning based curiosity-driven testing of Android applications. In _Proceedings of the 29th ACM SIGSOFT International Symposium on Software Testing and Analysis_. 153–164.
  * Park et al. (2023) Joon Sung Park, Joseph O’Brien, Carrie Jun Cai, Meredith Ringel Morris, Percy Liang, and Michael S Bernstein. 2023.  Generative agents: Interactive simulacra of human behavior. In _Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology_. 1–22.
  * Qian et al. (2023) Chen Qian, Xin Cong, Cheng Yang, Weize Chen, Yusheng Su, Juyuan Xu, Zhiyuan Liu, and Maosong Sun. 2023\.  Communicative agents for software development.  _arXiv preprint arXiv:2307.07924_ (2023).
  * Schick et al. (2023) Timo Schick, Jane Dwivedi-Yu, Roberto Dessì, Roberta Raileanu, Maria Lomeli, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. 2023.  Toolformer: Language models can teach themselves to use tools.  _arXiv preprint arXiv:2302.04761_ (2023).
  * Schulman et al. (2017) John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. 2017.  Proximal policy optimization algorithms.  _arXiv preprint arXiv:1707.06347_ (2017).
  * Shen et al. (2023) Yongliang Shen, Kaitao Song, Xu Tan, Dongsheng Li, Weiming Lu, and Yueting Zhuang. 2023\.  Hugginggpt: Solving ai tasks with chatgpt and its friends in huggingface.  _arXiv preprint arXiv:2303.17580_ (2023).
  * Shinn et al. (2023) Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik R Narasimhan, and Shunyu Yao. 2023\.  Reflexion: Language agents with verbal reinforcement learning. In _Thirty-seventh Conference on Neural Information Processing Systems_.
  * Singh et al. (2023) Ishika Singh, Valts Blukis, Arsalan Mousavian, Ankit Goyal, Danfei Xu, Jonathan Tremblay, Dieter Fox, Jesse Thomason, and Animesh Garg. 2023.  Progprompt: Generating situated robot task plans using large language models. In _2023 IEEE International Conference on Robotics and Automation (ICRA)_. IEEE, 11523–11530.
  * Song et al. (2022) Chan Hee Song, Jiaman Wu, Clayton Washington, Brian M Sadler, Wei-Lun Chao, and Yu Su. 2022\.  Llm-planner: Few-shot grounded planning for embodied agents with large language models.  _arXiv preprint arXiv:2212.04088_ (2022).
  * Talebirad and Nadiri (2023) Yashar Talebirad and Amirhossein Nadiri. 2023.  Multi-Agent Collaboration: Harnessing the Power of Intelligent LLM Agents.  _arXiv preprint arXiv:2306.03314_ (2023).
  * Tang et al. (2019) Jun Tang, Zhibo Yang, Yongpan Wang, Qi Zheng, Yongchao Xu, and Xiang Bai. 2019\.  Seglink++: Detecting dense and arbitrary-shaped scene text by instance-aware component grouping.  _Pattern recognition_ 96 (2019), 106954.
  * Touvron et al. (2023a) Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023a.  Llama: Open and efficient foundation language models.  _arXiv preprint arXiv:2302.13971_ (2023).
  * Touvron et al. (2023b) Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023b.  Llama 2: Open foundation and fine-tuned chat models.  _arXiv preprint arXiv:2307.09288_ (2023).
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017.  Attention is all you need.  _Advances in neural information processing systems_ 30 (2017).
  * Wang et al. (2023c) Guanzhi Wang, Yuqi Xie, Yunfan Jiang, Ajay Mandlekar, Chaowei Xiao, Yuke Zhu, Linxi Fan, and Anima Anandkumar. 2023c.  Voyager: An open-ended embodied agent with large language models.  _arXiv preprint arXiv:2305.16291_ (2023).
  * Wang et al. (2023a) Lei Wang, Chen Ma, Xueyang Feng, Zeyu Zhang, Hao Yang, Jingsen Zhang, Zhiyuan Chen, Jiakai Tang, Xu Chen, Yankai Lin, et al. 2023a.  A survey on large language model based autonomous agents.  _arXiv preprint arXiv:2308.11432_ (2023).
  * Wang et al. (2023d) Lei Wang, Jingsen Zhang, Xu Chen, Yankai Lin, Ruihua Song, Wayne Xin Zhao, and Ji-Rong Wen. 2023d.  RecAgent: A Novel Simulation Paradigm for Recommender Systems.  _arXiv preprint arXiv:2306.02552_ (2023).
  * Wang et al. (2023b) Zhenhailong Wang, Shaoguang Mao, Wenshan Wu, Tao Ge, Furu Wei, and Heng Ji. 2023b.  Unleashing cognitive synergy in large language models: A task-solving agent through multi-persona self-collaboration.  _arXiv preprint arXiv:2307.05300_ (2023).
  * Wei et al. (2022a) Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, et al. 2022a.  Emergent abilities of large language models.  _arXiv preprint arXiv:2206.07682_ (2022).
  * Wei et al. (2022b) Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. 2022b.  Chain-of-thought prompting elicits reasoning in large language models.  _Advances in Neural Information Processing Systems_ 35 (2022), 24824–24837.
  * Wu et al. (2023) Qingyun Wu, Gagan Bansal, Jieyu Zhang, Yiran Wu, Shaokun Zhang, Erkang Zhu, Beibin Li, Li Jiang, Xiaoyun Zhang, and Chi Wang. 2023\.  Autogen: Enabling next-gen llm applications via multi-agent conversation framework.  _arXiv preprint arXiv:2308.08155_ (2023).
  * Yao et al. (2022a) Shunyu Yao, Howard Chen, John Yang, and Karthik Narasimhan. 2022a.  Webshop: Towards scalable real-world web interaction with grounded language agents.  _Advances in Neural Information Processing Systems_ 35 (2022), 20744–20757.
  * Yao et al. (2023) Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Thomas L Griffiths, Yuan Cao, and Karthik Narasimhan. 2023.  Tree of thoughts: Deliberate problem solving with large language models.  _arXiv preprint arXiv:2305.10601_ (2023).
  * Yao et al. (2022b) Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. 2022b.  React: Synergizing reasoning and acting in language models.  _arXiv preprint arXiv:2210.03629_ (2022).


Author Name1Contributed equally1,2<EMAIL_ADDRESS>
1 Address 1
2 Address 2 and Author Name21<EMAIL_ADDRESS>
Author Name32<EMAIL_ADDRESS>
Author Name4Contributed equally3<EMAIL_ADDRESS>
3 Address 3 and Author Name54<EMAIL_ADDRESS>
4 Address 4

# Utilising Weak-to-Strong Consistency for Semi-Supervised Glomeruli
Segmentation

###### Abstract

This is a great paper and it has a concise abstract.

##  1 Introduction

Chronic kidney disease (CKD) involves the progressive loss of kidney function
over time with minimal chance for reversal. CKD affects more than 800 million
people and is expected to become the fifth-largest cause of death worldwide by
2040 [vanholder2021fighting]. The terminal stage of CKD is known as End-stage
kidney disease (ESKD) when kidneys lose their ability to function, filter
waste, and maintain fluid. Glomeruli are specialized structures of capillaries
in kidneys that serve as the primary filtration units to remove excess fluid
from the blood. Understanding the role of glomeruli is crucial for disease
pathogenesis and management to prevent ESKD. Histopathology assessment of
renal specimens is indispensable in providing assessment of morphological
changes for accurate diagnosis, prognosis, patient response to treatment, and
monitoring of patients with CKD. Segmentation and localization of glomeruli
instances from histopathology whole-slide images (WSIs) is an essential step
toward computer-assisted analysis of renal biopsies.

In this study, we validate the proposed method with relatively large datasets
containing both healthy and malignant cases from three independent sources.
This also includes NURTuRE data (the National Unified Renal Translational
Research Enterprise), which contains renal biopsies from 13 NHS trusts across
the UK. a)

##  2 Related Work

###  2.1 Glomeruli localisation

Andreini2024 show that the segmentation of glomeruli in animals models as a
pre-training step for application on human tissue improves model
generalisibility. Other works demonstrate the utility of cycle-consistent
generative adversarial networks (cycle GANs) in improving the performance of
glomeruli segmentation Lo2021; Bouteldja2022. Hölscher2023 show, in a large-
scale study of automated kidney morphometry, how features from histology can
predict long-term clinical outcomes.

###  2.2 Semi-supervised learning

##  3 Methods

Figure 1: Model Diagram. blueCan you mark directly on the figure where the
weak and strong augmentation is applied, and possibly include examples of the
augmented patches?

We propose the framework shown in Fig. 1 for semi-supervised semantic
segmentation of glomeruli in Periodic Acid Schiff (PAS) stained renal
biopsies. The proposed framework was aimed to leverage a limited number of
labelled images $D^{l}$ and a vast number of unlabelled images $D^{u}$ to
train a generalisable glomeruli segmentation model that can be applied to
diverse multi-centre renal histopathology datasets. Patches and corresponding
ground-truth masks from labelled dataset $D^{l}$ are passed to the supervised
branch. Image patches from unlabelled dataset $D^{u}$ are transformed with a
weak augmentation and two strong augmentations first and then passed into the
encoder.

###  3.1 Supervised baseline models

The attention U-Net was the first-place-winning solution for HuBMAP – Hacking
the Kidney competition on Kaggle in 2021 hubmap-kidney-segmentation, which is
based on the U-Net architecture, the previous state-of-the-art model for
medical image segmentation. SegFormer is a semantic segmentation model that
uses multi-scale Vision Transformers (ViTs) as encoders and light-weight
multi-layer perception decoders. In order to find an ideal baseline
segmentation model, we compared the performance of Attention U-Net and
SegFormer in a fully supervised manner on our labelled datasets with 5-fold
cross validation, before performing external validation on hold-out datasets
to evaluate the generalisability of both models. We selected the model with
better external performance for semi-supervised training in the following
steps.

###  3.2 Weak-to-strong augmentation

In semi-supervised learning, there are a supervised branch and an unsupervised
branch. In the supervised branch, the model trained to make predictions close
to the ground truth. Given the lack of ground truth labels in the unsupervised
branch, the model is trained to make consistent predictions on weakly
augmented versions and strongly augmented versions of the same unlabelled
input image. Weak augmentations usually include random crop and rotation,
while strong augmentations include colour jitters.

###  3.3 Semi-supervised glomeruli segmentation

In FixMatch, one weak augmentation and one strong augmentation are applied to
each unlabelled image. UniMatch expands the pertubation space by introducing
two separate strong augmentations and image-level feature perturbations. We
applied FixMatch and UniMatch semi-supervised learning to leverage the vast
amount of unlabelled images. We followed the setup proposed by xx for UniMatch
semi-supervised training.

##  4 Experimental Settings

###  4.1 Data

We used four data sets in this study. The characteristics of these data sets
are summarised in Table 1. As the images from NURTuRE do not have pathological
annotations, we found publicly available data sets with annotations for
glomeruli to use as labelled data.

fig:example

Figure 2: tiles to show heterogeneity across these datasets

####  4.1.1 HuBMAP - Hacking the Kidney

In 2020, Howard et al published a dataset of PAS-stained renal biopsies for
HuBMAP - Hacking the Kidney Competition (referred as HuBMAP Kidney). We
downloaded the public training set together with the pathological annotations
for glomeruli as our first labelled data set. The whole slide images from
HuBMAP Kidney were scanned at 20x magnification. The tissues in whole slide
images from HuBMAP Kidneys are wedge resections that consist of both kidney
cortex and medulla.

####  4.1.2 HuBMAP - Hacking the Human Vasculature

In 2023, Howard et al launched another competition on Kaggle, HuBMAP - Hacking
the Human Vasculature (referred as HuBMAP Vasculature). They published a data
set consisted of tiles extracted from whole slide images of PAS-stained renal
tissues. Pathological annotations for glomeruli were provided in this data set
as well. However, it should be noted that the images in HuBMAP Vasculature
were extracted from only 2 whole slide images. Given the small size of this
data set, we used HuBMAP Vasculature as an external validation data set.

####  4.1.3 Kidney Precision Medicine Project

The Kidney Precision Medicine Project (KPMP) is a prospective cohort study
aimed to develop new disease ontologies, classification systems, and
treatments for acute kidney injury (AKI) and chronic kidney disease (CKD)
kpmprationale. KPMP provides deep molecular phenotypes of kidney biopsies and
longitudinally collected clinical phenotypic data. We used pathologically
annotated whole slide images of PAS-stained renal biopsies from KPMP in this
study.

####  4.1.4 NURTuRE

Should we introduce an official introduction of NURTuRE project here???

Table 1: Data Sets Data | # of WSIs | # of tiles
---|---|---
HuBMAP Kidney | 15 | 3706
HuBMAP Vasculature | 2 | 416
KPMP | 221 | 20868
NURTuRE | 219 | 14845

###  4.2 Implementation Details

##  5 Results

###  5.1 Comparative Analysis

####  5.1.1 Supervised baseline models

  * •

No significant difference in performance on HuBMAP kidney dataset between
U-Net and SegFormer

  * •

SegFormer has only one-tenth of the parameters of U-Net so it’s much more
light weight and easier to fine-tune or retrain

  * •

SegFormer worked slightly better on external datasets

Table 2: Dice Scores of Supervised Baseline Models with 5-fold Cross Validation Method | # of Parameters | HuBMAP Kidney
---|---|---
Attention U-Net | 134.1M | $0.93\pm 0.01$
SegFormer_b1 | 13.7M | $0.93\pm 0.01$
Table 3: External Validation of Supervised Baseline Models Method | HuBMAP Vasculature | KPMP | NURTuRE Labelled
---|---|---|---
Attention U-Net | 0.64 | 0.74 | 0.63
SegFormer_b1 | 0.66 | 0.74 | 0.64

####  5.1.2 Semi-supervised pipelines

Table 4: Segmentation Metrics with Semi-Supervised Learning Dataset | Method | Precision | Recall | Dice
---|---|---|---|---
HuBMAP Vasculature | Supervised baseline | 0.95 | 0.50 | 0.66
FixMatch | 0.85 | 0.74 | 0.79
UniMatch | 0.85 | 0.75 | 0.80
KPMP | Supervised baseline | 0.95 | 0.61 | 0.74
FixMatch | 0.94 | 0.63 | 0.76
UniMatch | 0.94 | 0.64 | 0.76
NURTuRE Labelled | Supervised baseline | 0.82 | 0.52 | 0.64
FixMatch | 0.82 | 0.56 | 0.67
UniMatch | 0.82 | 0.57 | 0.68

fig:example

Figure 3: segmentation results of different models

###  5.2 Ablation Study

####  5.2.1 Semi-supervised learning with few labeled images

Table 5: Dice scores of FixMatch and UniMatch on different fractions of labeled images from HuBMAP Kidney Method | 1/16 | 1/8 | 1/4 | 1/2
---|---|---|---|---
Supervised | 0.15 | 0.17 | 0.46 | 0.60
FixMatch | 0.22 | 0.25 | 0.53 | 0.63
UniMatch | 0.22 | 0.25 | 0.54 | 0.65

In the original paper, UniMatch performed better than FixMatch in low data
regime. However, we didn’t see the same tendency on renal histopathology
images. Nevertheless, both semi-supervised learning methods consistently
performed better than supervised baseline.

####  5.2.2 Semi-supervised learning with unlabeled images from different
number of sources

We randomly sample 100 patches from each NURTuRE centre and explored whether
the model would benefit from using unlabeled images from a larger number of
sources with more diversity.

Table 6: Dice scores of UniMatch on HuBMAP Vasculature using unlabeled images from different numbers of centres in NURTuRE Sources of unlabeled images | Dice Score
---|---
1 centre | 0.79
3 centres | 0.78
5 centres | 0.78
7 centres | 0.80
9 centres | 0.78
11 centres | 0.79
13 centres | 0.78
15 centres | 0.79

####  5.2.3 Different backbones for the encoder

  * •

More complex SegFormer backbone didn’t necessarily lead to improvements in
performance

  * •

SegFormer b4 performed better than SegFormer b1 but given the much larger
number of parameters, we chose to use SegFormer b1 as the default model in
this paper

Table 7: Segmentation Metrics of SegFormer Models with Different Backbones on HuBMAP Vaculature Dataset Backbone | # of Parameters | Precision | Recall | Dice
---|---|---|---|---
SegFormer_b0 | 3.7M | 0.83 | 0.76 | 0.79
SegFormer_b1 | 13.7M | 0.85 | 0.75 | 0.80
SegFormer_b2 | 24.7M | 0.87 | 0.74 | 0.80
SegFormer_b3 | 44.6M | 0.75 | 0.70 | 0.73
SegFormer_b4 | 61.4M | 0.85 | 0.79 | 0.82
SegFormer_b5 | 82.0M | 0.78 | 0.72 | 0.75

##  6 Discussion

###  6.1 Generalizable approach

  * •

We found a light-weight baseline model SegFormer as a substatitue for previous
SOTA U-Net

  * •

We proposed using semi-supervised learning to model performance on new data

###  6.2 Multi-centred validation

  * •

We validated our approach on several external datasets collected from real-
world settings. KPMP and NURTuRE are multi-centre cohort studies which
provides diverse data. The digital pathology images from these sources are of
high clinical relevance

  * •

Challenges with multi-centre real-world histopathology data: large cohort
size, high variability, potential mislabelling, artefacts on WSIs, etc

##  7 Conclusion and Future Work

###  7.1 Correlate glomerular features to disease progression

Our next goal for AI-guided analyses of renal biopsies is to correlate tissue-
based biomarkers to disease progression. One potential approach is to extract
glomerular features such as glomerular volume, cell density from glomerular
regions, and the ratio between healthy glomeruli and sclerotic glomeruli and
explore whether these features are correlated with the risk for fast
progression of CKD.

###  7.2 Expand to multi-class segmentation

  * •

Segmenting glomeruli as well as classifying different types of glomerular
diseases or morphological changes would further assist pathological
examination of renal biopsies. However, this will require expert annotations
from certified pathologists.

  * •

Other functional tissue units are also of interest when evaluating CKD
biopsies. For example, tubular atrophy is an important indicator of CKD.
Therefore, segmenting other structures such as tubuli should also be useful.

###  7.3 Develop a stain agnostic segmentation model

  * •

Other stains such as Masson’s Trichrome, Jone’s Silver Stain, and Sirius Red
are also commonly used on renal biopsies. The morphology highlighted by
different stains are all informative for the diagnosis for CKD. Therefore,
training a model that is agnostic of the staining assay could potentially
enable us to build more automated image processing tools to assist renal
pathology analyses.

Test citation hubmap-hacking-the-human-vasculature—note the tilde in LaTeX
source.

  * •

Overview of chronic kidney disease (CKD), prevalence of CKD, economic burden
of CKD

  * •

The role of pathology in CKD, challenges

  * •

Significance of glomeruli segmentation

  * •

What others have done in the way of glomeruli segmentation

  * •

Introduce NURTuRE (multi-centre biobank, multi-modal data, etc)

  * •

Explain the proposed approach (semi-supervised learning, overview, uni-match
approach)

  * •

What has been achieved (results, multiple validation cohorts etc): light-
weight model SegFormer, Semi-supervised learning to utilise unlabeled images
in training, improved recall and dice

Figure 4: Model Diagram. blueCan you mark directly on the figure where the
weak and strong augmentation is applied, and possibly include examples of the
augmented patches?