language = Java, basicstyle = , stringstyle = , keywordstyle = , keywordstyle
= [2], keywordstyle = [3], keywordstyle = [4], morekeywords = [3]<<, >>,
morekeywords = [4]++, basicstyle=, commentstyle=, morecomment=[f][red!20]-,
morecomment=[f][green!20]+, morecomment=[f][yellow!20]++,
morecomment=[f][yellow!20]--, morecomment=[f][]@@, texcl=false

# Guiding ChatGPT to Fix Web UI Tests via Explanation-Consistency Checking

Zhuolin Xu  Concordia UniversityCanada<EMAIL_ADDRESS>,
Yuanzhang Lin  Beihang UniversityChina<EMAIL_ADDRESS>,  Qiushi Li
Concordia UniversityCanada<EMAIL_ADDRESS>and  Shin Hwei Tan
Concordia UniversityCanada<EMAIL_ADDRESS>
###### Abstract.

The rapid evolution of Web UI incurs time and effort in maintaining UI tests.
Existing techniques in Web UI test repair focus on finding the target elements
on the new web page that match the old ones so that the corresponding broken
statements can be repaired. We present the first study that investigates the
feasibility of using prior Web UI repair techniques for initial local matching
and then using ChatGPT to perform global matching. Our key insight is that
given a list of elements matched by prior techniques, ChatGPT can leverage the
language understanding to perform global view matching and use its code
generation model for fixing the broken statements. To mitigate hallucination
in ChatGPT, we design an explanation validator that checks whether the
provided explanation for the matching results is consistent, and provides
hints to ChatGPT via a self-correction prompt to further improve its results.
Our evaluation on a widely used dataset shows that the ChatGPT-enhanced
techniques improve the effectiveness of existing Web test repair techniques.
Our study also shares several important insights in improving future Web UI
test repair techniques.

##  1\. Introduction

When developers change the attributes of user interfaces (UI) of a Web
application due to the rapidly changing requirements, the corresponding Web UI
tests need to be manually updated for test maintenance. To reduce manual
efforts in repairing broken Web UI tests, several automated approaches have
been proposed (Choudhary et al., 2011; Stocco et al., 2018; Lin et al., 2023).
The key step in automated repair of Web UI tests is to modify the broken
statements containing outdated element locators by matching the element
$e_{old}$ in the old version of a Web application with the element in the new
version $e_{new}$ (Hammoudi et al., 2016). Prior Web UI test repair techniques
mostly rely on a set of Document Object Model (DOM) attributes (e.g.,
identifiers and XPath) (Choudhary et al., 2011) or visual information (Stocco
et al., 2018) to determine whether the two elements $e_{old}$ and $e_{new}$
match. These techniques extract and compute the similarity of these
information to select the most similar element as the result of the matching.

As there can be many attributes of a Web UI element which can be used for
matching, these techniques may prioritize certain attributes. For example,
Water, one of the classical Web UI test repair techniques, performs matching
via several steps by using different sets of attributes in each step. First,
it searches for elements that are exactly the same by matching five attributes
(_id_ , _XPath_ , _class_ , _linkText_ , _name_). If the first step fails to
find the matching element in the new version that makes the test pass, it then
finds similar DOM nodes using other additional attributes. Specifically, it
finds the element with the same _tagname_ , and then computes the similarity
between the element with the same _tagname_ using normalized Levenshtein
distance between the XPaths of $e_{old}$ and $e_{new}$. Then, it further
matches using other attributes (e.g., screen position of a DOM node), and
computes a similarity score based on the weighted sum of XPath and other
attributes where it prioritizes XPath similarity based on the heuristic that
the XPaths of the nodes “should be very similar across versions” (Choudhary et
al., 2011)). As the prioritization and the predefined order used for matching
these set of attributes are usually based on heuristic made by the tool
developers, _the matching algorithm may not accurately reflect the evolution
of the Web element_ , causing inaccuracy in the matching step, and
subsequently unable to find the repair for the broken statement. Meanwhile,
prior learning-based techniques show promising results in combining different
types of information for repairing broken GUI tests in Android apps (e.g.,
combining word and layout embeddings (Yoon et al., 2022), or fusing GUI
structure and visual information (Xu et al., 2021)). The richer representation
used by these learning-based techniques has been shown to help improving the
accuracy of the UI matching step.

To solve the aforementioned problem of Web UI test repair and to hinge on
richer representation in learning-based approach, we present the _first
feasibility study_ of combining ChatGPT with traditional Web UI test repair
approaches for solving the aforementioned element matching problem in prior
techniques. Our use of ChatGPT is based on the encouraging results shown in
prior studies for solving related software maintenance tasks, (e.g., (1) test
generation (Feldt et al., 2023), and (2) automated program repair (Sobania et
al., 2023; Fan et al., 2023)). However, the Web UI test repair problem is
different from these tasks as it mainly involves Web element matching where
accurate matching results will usually lead to the correct repairs being
generated. Specifically, our study evaluates the effectiveness of integrating
ChatGPT into two representative Web test repair techniques (Water and Vista).
To further evaluate the heuristic used in Water that prioritizes XPath
similarity, we also design a simplified variant of Water that performs
matching using only Levenshtein distance between XPaths of the old element
$e_{old}$ and candidate new element $e_{new}$ (in this paper, we call this
approach _Edit Distance_). Our study focuses on Java Selenium UI tests. The
key insight of our study is that _we can improve the overall matching results
by first using existing Web test repair approaches to obtain an initial list
of candidate matched elements (which may be biased by the prioritization used
by a particular technique), and then use ChatGPT to perform global matching to
further select the best matched element in the new version_.

The key insights of our approach are twofold: (1) we first rely on traditional
Web UI test repair approaches to obtain an initial list of candidate matched
elements (which may be biased by the prioritization used by a particular
technique), and then use ChatGPT to perform global matching to further select
the best matched element in the new version, and (2) as ChatGPT may suffer
from the hallucination problem (Feldt et al., 2023), we design our prompt
based on OpenAI’s official documentation by asking ChatGPT to generate an
explanation along with each selection, and propose an _explanation validator_
that automatically checks for the consistency of the provided explanation to
detect the hallucination, and inform ChatGPT for any incorrect explanation to
provide a chance for ChatGPT to self-correct the initial selection.

Our study of the three approaches (Water,Vista, and Edit Distance) with
ChatGPT aims to answer the following question:

RQ1::

Can ChatGPT help in improving the accuracy of Web element matching of prior
Web test repair approaches?

RQ2::

What is the effectiveness of ChatGPT in repairing broken Selenium statements
for Web test repair?

RQ3::

When ChatGPT explains about the element matching result, what is the quality
of the explanation?

RQ4::

Can our proposed explanation validator guide ChatGPT in self-correction to
improve the overall matching and repair results?

Contributions. Our contributions are summarised as follows:

Study::

We present the first feasibility study of combining traditional test repair
approaches with ChatGPT for Web UI test repair. Our study reveals several
findings: (1) the combination with ChatGPT helps to improve the accuracy of
all evaluated approaches; (2) to our surprise, although Edit Distance performs
the worst individually, its combination with ChatGPT outperforms all the
evaluated approaches in element matching and repair; (3) the repair
performance of all evaluated approaches is generally similar to the matching
performance but there are two cases where the repair performance decreases,
which shows the limitations of evaluated approaches; (4) our proposed workflow
of checking for explanation consistency and generating self-correct prompt as
hint to ChatGPT could further improve the effectiveness of matching and repair
for certain combinations (e.g., Water+ChatGPT).

Technique::

We proposed a novel Web UI test repair technique that uses traditional Web UI
test repair approach for initial matching and then uses ChatGPT for global
matching to further improve the UI matching accuracy. To combat the
hallucination problem in ChatGPT generated response, we also design an
explanation validator that automatically checks for the consistency of the
generated explanation to provide a mechanism for self-correction.

Evaluation::

We evaluate our proposed approaches against three baselines (Water,Vista,Edit
Distance) and their combinations with ChatGPT. Our evaluation on a widely used
dataset (Imtiaz et al., 2021a) shows that the proposed workflow further
improves the effectiveness of prior approaches in Web UI element matching and
test repair.

##  2\. Background And Related Work

Automatic UI test repair. Different from traditional unit tests, UI tests are
typically used in UI applications (e.g., Web apps and mobile apps) to validate
the functionality of a particular component through a sequence of UI event
actions (e.g., clicking a button) (Choudhary et al., 2011). When the UI
application under test evolves, the corresponding UI tests may crash, leading
to significant effort required to manually fix these broken tests. Several
techniques have been proposed for UI test maintenance (Choudhary et al., 2011;
Stocco et al., 2018; Lin et al., 2023; Li et al., 2017; Chang et al., 2018;
Gao et al., 2016; Zhang et al., 2013; Yoon et al., 2022; Kirinuki et al.,
2019; Chen et al., 2021). Most of these techniques focus on maintaining UI
tests for mobile applications (Li et al., 2017; Chang et al., 2018; Xu et al.,
2021; Pan et al., 2020; Yoon et al., 2022), Web applications (Choudhary et
al., 2011; Stocco et al., 2018; Kirinuki et al., 2019; Lin et al., 2023; Chen
et al., 2021) or desktop applications (Gao et al., 2016; Zhang et al., 2013).
Existing Web UI test repair techniques (Choudhary et al., 2011; Stocco et al.,
2018; Lin et al., 2023) typically involve (1) executing the test, (2)
extracting information from the Webpage based on various textual attributes
(Choudhary et al., 2011) or visual information (Stocco et al., 2018) of Web
elements, (3) using different matching algorithms to match the element based
on the extracted information, and (4) updating the locator of the matched
element in the UI test to fix the broken test.

Model-based approaches (Li et al., 2017; Chang et al., 2018; Gao et al., 2016;
Zhang et al., 2013; Imtiaz et al., 2021a) build a model of the application
under test and modify the event flow to fix test cases. Meanwhile, several
heuristic-based approaches (Choudhary et al., 2011; Stocco et al., 2018; Lin
et al., 2023) relocate elements using UI matching algorithms and update the
broken tests via replacement of the matched elements. As heuristic-based
approaches do not require building a model which may not be practical for
large-scale applications, our paper mainly evaluates these approaches. A
recent approach (Yoon et al., 2022) first matches elements of two versions of
Android apps using various similarity metrics (e.g., semantic embedding
similarity, and layout similarity based on node embeddings of the GUI layout
tree), and then repairs the tests by updating broken locators. We did not
compare against (Lin et al., 2023; Chen et al., 2021) as their matching
algorithm is not publicly available and exclude model-based tool (Imtiaz et
al., 2021b) as the provided code fails to compile due to missing dependencies.

UI element matching. Given an element from an old version, the goal of a UI
element matching task is to find the corresponding element in the new version.
UI element matching has been applied to several domains (e.g., test reuse
(Mariani et al., 2021), automated compatibility testing (Ren et al., 2022),
and automated maintenance of UI tests). Although there are various techniques
for UI element matching, only a few of them are suitable for our evaluation.
For example, COLOR (Kirinuki et al., 2019) and GUIDER (Xu et al., 2021) use a
combination of attribute and visual information to match elements. However,
COLOR only recommends updated locators to testers instead of repairing broken
tests while GUIDER focuses on Android test repair. As our goal is to study UI
test repair, we didn’t compare with tools (Shao et al., 2021; Brisset et al.,
2023) that perform matching without repair. Water (Choudhary et al., 2011) and
Vista (Stocco et al., 2018) are classic UI test repair tools, which match
elements according to attribute information and visual information,
respectively. Existing repair work mainly relies on designing some strategies
based on element information (Choudhary et al., 2011; Stocco et al., 2018), or
simply combining machine learning algorithms with the repair of UI test cases
(Yoon et al., 2022). Due to the differences between UI test cases and existing
APR work (Xia et al., 2023), it is not suitable to directly use existing APR
approaches to repair UI test cases. In this paper, we focus on Web UI test
repair. Different from these prior approaches, we explore the feasibility of
using prior test repair approaches to obtain an initial ranked list of
elements and use ChatGPT to perform global view matching.

Instead of fixing broken Web UI tests, several approaches focus on improving
the robustness of the element (M. Leotta and Tonella, 2016; Leotta et al.,
2015). Leotta et al. developed _Robula+_ (M. Leotta and Tonella, 2016), which
generates locators to see if an attribute uniquely locates an element based on
a predefined attribute priority. They also later proposed using multiple
locator generators to further increase the robustness (Leotta et al., 2015).
They determine the final result by having different locator generators vote,
where they give higher weight to more reliable locator generators like
_Robula+_. Our design of the experiment is similar in essence with this
approach, letting ChatGPT make a selection among multiple candidates. While
improving the robustness of locators is beyond the scope of this paper, it
could be worthwhile future work.

LLMs in software maintenance. LLMs such as ChatGPT have been applied for
solving various software maintenance tasks (Mashhadi and Hemmati, 2021; Xiao
et al., 2024; To et al., 2023; Feldt et al., 2023; Sobania et al., 2023; Fan
et al., 2023), including related tasks such as (1) test generation (Feldt et
al., 2023; Lemieux et al., 2023; Yu et al., 2023), Android UI testing (Liu et
al., 2023; Feng and Chen, 2023), predicting flaky tests (Fatima et al., 2023),
and (2) automated program repair (Sobania et al., 2023; Fan et al., 2023;
Mashhadi and Hemmati, 2021; Joshi et al., 2023). ChatGPT is an open-source
transformer-based chatbot that assists developers in building conversational
AI applications (Santhosh et al., 2023). Our proposed approach uses ChatGPT
for (1) Web UI element matching, (2) providing explanations for the selected
elements, and (3) updating broken statements in tests. To improve code-related
generation for LLM-based approaches, a recent approach was proposed using
knowledge gained during the pre-training and fine-tuning stage to augment
training data and their results show significant improvement for code
summarization and code generation (To et al., 2023). Similar to this approach,
our proposed workflow also uses the knowledge gained during the selection of
UI elements (i.e., explanation generated by ChatGPT) to improve the UI
matching results. Different from the aforementioned work, this paper focuses
on using ChatGPT for solving the Web UI test repair problem that aims to
perform UI test update by fixing broken locators. To the best of our
knowledge, our study also presents the first attempt that investigates and
improves the quality of explanations provided by ChatGPT by designing an
explanation validator to check the reliability of the explanation to improve
the effectiveness of UI matching and test repair.

##  3\. Motivating Examples

We show two examples where the combination of ChatGPT helps to improve the
matching and test repair results.

Without self-correction. Table 1 shows the target element to be matched by
Waterin the old version of the MantisBT app, and the list of candidate
elements returned by Water. Initially, Water have incorrectly ranked the
“Candidate 1” as the top 1 element. Given the ranked list generated by Water,
ChatGPT can make the correct matching by returning “Candidate 6” as the
correct element. ChatGPT also explains that it selects the element because
“they share the most similar attributes: xpath, text” (its explanation
consistency (EC) is equal to 1). After matching, our approach feeds ChatGPT
with the repair prompt, and it generates the repair in Figure 1. The generated
fix uses (1) a different locator (By.xpath instead of By.name), and (2) the
XPath from “Candidate 6” for updating the broken statement.

⬇

1- driver.findElement(By.name("category")).sendKeys

2- ("Category1");

3+ driver.findElement(By.xpath("/html[1]/body[1]/div[3]

4+ /form[1]/table[1]/tbody[1]/tr[2]/td[2]/input[1]"))

5+ .sendKeys("Category1");

Figure 1. A ChatGPT fix for the broken statement in MantisBT that replaces
with different locators.

With self-correction. Figure 2 shows the fix generated by Water+ChatGPT before
($result1$) and after self-correction ($result2$). The explanation provided by
ChatGPT for $result1$ is “Because they share the most similar attributes:
xpath, text, tagName, linkText.”. Our explanation validator detects that the
explanation of the attributes _text_ and _linkText_ are inconsistent with the
selection (Explanation Consistency=0.5). After self-correction, Water+ChatGPT
repair this broken statement correctly and the result is represented as
$result2$. The explanation for $results2$ is “Because they share the most
similar attributes: xpath, text, tagName, linkText.” (Explanation
Consistency=0.75 because all attributes except for XPath are consistent with
the selection).

⬇

1- driver.findElement(By.xpath("//*[@id=\"content

2- wrapper\"]/div[1]/ul/li[3]/a")).click();

3//result1: before self-correction

4+ driver.findElement(By.xpath("/html[1]/body[1]/div[1]/

5+ div[2]/div[1]/ul[1]/li[1]/a[1]")).click();

6//result2: after self-correction

7+ driver.findElement(By.xpath("/html[1]/body[1]/div[1]/

8+ div[2]/div[2]/div[1]/div[4]/div[1]/h2[1]/a[1]"))

9+ .click();

Figure 2. ChatGPT fixes for the broken statement in Collabtive that shows the
effectiveness of self-correction. Table 1. An example that shows the target
element to be matched, and a list of candidate elements returned by Water

Target Element |  | {numericId=70, id=‘’, name=‘new_category’, class=‘’, xpath=‘/html[1]/body[1]/div[4]/form[1]/table[1]/tbody[1]/tr[2]/td[2]/input[1]’,
---
text=‘Category1’, tagName=‘input’, linkText=‘’, x=363, y=278, width=261,
height=21, isLeaf=true}
Candidate 1 |  | {numericId=20, id=‘’, name=‘’, class=‘button-small’, xpath=‘/html[1]/body[1]/table[1]/tbody[1]/tr[1]/td[3]/form[1]/input[1]’,
---
text=‘Switch’, tagName=‘input’, linkText=‘’, x=951, y=121, width=51,
height=20, isLeaf=true}
… | …
Candidate 6 |  |  {numericId=70, id=‘’, name=‘name’, class=‘’, xpath=‘/html[1]/body[1]/div[3]/form[1]/table[1]/tbody[1]/tr[2]/td[2]/input[1]’,
---
text=‘Category1’, tagName=‘input’, linkText=‘’, x=403, y=295, width=261,
height=21, isLeaf=true}

##  4\. Methodology

Figure 3. The workflow of Web UI test repair using ChatGPT

Figure 3 shows the overall workflow of our proposed approach. Our approach
first extracts information about the corresponding elements on the old Webpage
and all elements on the new Webpage. Then, we use a similarity algorithm
(Water, Vista or Edit Distance) to rank elements extracted from the new
Webpage where top-ranking elements are selected as candidate elements for
ChatGPT to perform element matching. In the matching prompt, we ask ChatGPT to
explain about its selection by listing the most similar attribute it
considers. Given the generated explanation, our explanation validator checks
the consistency of the result. In the repair prompt, we ask ChatGPT to repair
the broken statement based on its selected element. Then, we validate the
repair result by checking if the matching is correct, and if the locator and
assertion (if exist) are updated correctly. Due to the randomness of ChatGPT,
we rerun the matching and repair four times, and use the best results across
the four runs, following the procedure in prior study (Sobania et al., 2023).
For each incorrect repair where the explanation consistency (EC) is less than
1, our approach generates a self-correctness prompt to ChatGPT as a hint to
encourage it to provide another answer based on the inconsistent explanation
of their matching result. After feeding the repair prompt to ChatGPT, we
obtain the ChatGPT’s repaired broken statement.

###  4.1. Prompt Design

We interact with ChatGPT via the API of the gpt-3.5-turbo model (gpt, 2022).
To obtain prompt with the best results, we design our prompt based on the
official OpenAI documentations, including: (1) the official ChatGPT API Guide
(OpenAI Help, 2023) and (2) OpenAI six strategies for achieving better results
(OpenAI, 2023a). Table 2 shows for each sentence of the prompt (the “Prompt
Content” column), the corresponding rule (the “Rule in OpenAI official
documentation” column) that inspires the design.

The ChatGPT API Guide emphasizes the importance of system instructions in
providing high-level guidance for conversations so we use system instructions
to design the Web UI test repair context part by telling ChatGPT that (1) it
is a UI test repair tool, and (2) outlining the steps in fixing a broken
statement.

Due to token limits, we summarize ChatGPT’s matching results from the previous
dialogue to inform ChatGPT of the specific element information it should use
in the subsequent repair process, aligning with the tactic “For dialogue
applications that require very long conversations, summarize or filter
previous dialogue.” (OpenAI, 2023a). Following the rule “Use delimiters to
clearly indicate distinct parts of the input” (OpenAI, 2023a), we use angle
brackets to separate sections of text. This aids ChatGPT in recognizing the
content of the _target element_ (element on the old Webpage to be matched) and
the broken statement.

Table 2 shows the patterns of prompt and used rules. Our complete prompt
generation rules are as follow:

:

Matching Prompt: Context[p1, p2, p3] + Input[p8]

:

Repair Prompt: Context[p1, p4, p5] + Input[p9]

:

Self-Correction Prompt: Context[p6, p7]

Table 2. The patterns of prompt and generation rules

PID | Rule in OpenAI’s official documentation | Prompt Content
---|---|---
Web UI test repair context patterns: Context
p1 | Use system instruction to give high level instructions (OpenAI Help, 2023) | You are a web UI test script repair tool.
p2 | Split complex tasks into simpler subtasks (OpenAI, 2023b) | To repair the broken statement, you need to choose the element most similar to the target element from the given candidate element list firstly.
|  | Give me your selected element’s numericId and a brief explanation containing the attributes that are most similar to the target element.
p3 | Provide examples (OpenAI, 2023a) | Your answer should follow the format of this example:
|  | “The most similar element’s numericId: 1. Because they share the most similar attributes: id, xpath, text.”
p4 | Summarize or filter previous dialogue (OpenAI, 2023a) | To repair the broken statement, you chose the element <selected element>as the most similar to the target element from the given candidate element list.
p5 | Specify the steps required to complete a task (OpenAI, 2023a) | Now based on your selected element, update the locator and outdated assertion of the broken statement. Give the result of repaired statement.
p6 | Use delimiters to clearly indicate distinct parts | This is a previous prompt: <Matching Prompt>
| of the input (OpenAI, 2023a) | This is your previous answer: <Corresponding Answer>
p7 |  | But your explanation for attributes <attributes>are inconsistent with your selection and this will influence the correctness of your answer. Please answer again.
Input pattern: Input
p8 | Use delimiters to clearly indicate distinct parts | Target element: <target element>
| of the input (OpenAI, 2023a) | Candidate elements: <candidate element list>
p9 |  | Broken statement: <broken statement>

###  4.2. Information Extraction

Table 3. Extracted attributes and the tools that use these attributes. The
superscript number for the last column indicates the priority of an attribute
for a tool.

Attribute | Description (Example) | Used by Tools
---|---|---
id | Unique identifier for an element () | $\textsc{Water}^{E1}$
name | Name for an element (submit) | $\textsc{Water}^{E5}$
class | Class name for an element () | $\textsc{Water}^{E3}$
XPath | Path of an element in the DOM tree (/html[1]/body[1]/div[1]/div[4]/form[1]/input[1]) |  $\textsc{Water}^{E2}$, $\textsc{Water}^{L6}$, $\textsc{Edit Distance}^{L1}$
text | Textual information of an element (Enter) |
tagName | Name of the HTML tag (input) |
linkText | The text content of a hyperlink () | $\textsc{Water}^{E4}$
x | X-coordinate for an element (66) | $\textsc{Water}^{6}$
y | Y-coordinate for an element (183) | $\textsc{Water}^{6}$
width | Width of an element (48) | $\textsc{Vista}^{1}$
height | Height of an element (21) | $\textsc{Vista}^{1}$
isLeaf | True if the element is a leaf node in the DOM tree, and false otherwise (true) |

  * •

1 The superscript ‘E’ means the tool checks whether the candidate element’s
attribute is exactly the same as the target element.

  * •

2 The superscript ‘L’ indicates the tool use Levenshtein Distance to calculate
the similarity.

  * •

3 The number in superscript indicates the priority of the attribute.

  * •

4 We refer to the version of Water implemented in prior work (Stocco et al.,
2018) to determine the precise priority of attributes in Water.

  * •

5 We refer to VISUAL mode of Vista (Stocco et al., 2018) for Vista’s priority.

We used UITESTFIX (Lin et al., 2023) to extract the information about the Web
elements. UITESTFIX retrieves the HTML source code of the target Webpage
through a Web browser, and then employs the _Jsoup_ library (Houston, 2013) to
analyze the source code and extract the Web elements’ information in it. Note
that although we use UITESTFIX for information extraction, we did not compare
it with UITESTFIX as its matching algorithm is not publicly available.
Specifically, we extract information about the following attributes: _id_ ,
_name_ , class, _XPath_ , _text_ , _tagName_ , _linkText_ , _x_ , _y_ ,
_width_ , _height_ , and _isLeaf_. The first two columns of Table 3 show the
descriptions and examples of these attributes, whereas the “Used by Tools”
column shows whether Water, Vista consider these attributes when matching
elements and how these tools use these attributes.

###  4.3. Candidate Selection

To address the token length limitation of ChatGPT, our approach fed ChatGPT
with a list of pre-selected candidate Web elements. Specifically, our approach
first obtains 10 top-ranked candidate elements from the matching results of
prior approaches (Water,Vista, Edit Distance). We only chose 10 elements
because (1) the gpt-3.5-turbo model that we use has a limitation of 4096
tokens, and (2) each element is represented by 12 attributes, which can be
lengthy. We briefly introduce the three approaches below:

Vista::

Vista uses Fast Normalized Cross Correlation algorithm (Briechle and Hanebeck,
2001) to calculate the template matching results. It returns a list where
elements of a particular screen position are ranked in descending order based
on their similarity scores. Instead of using the top 1 element, we modify
Vista to retrieve the top 10 elements based on the matching point coordinates
as candidates for the pre-selection.

Water::

As shown in Table 3, Water first finds whether there are candidate elements
that have the same _id_ , _XPath_ , _class_ , _linkText_ , or _name_ to the
target elements. If so, Water add them into the candidate list. Then Water
ranks the elements on the new Webpage according to their similarity score. If
there are candidates with same first scores, Water calculates the second score
with _tagName_ , _coordinates_ , _clickable_ , _visible_ , _zindex_ (the
screen stack order of the element), _hash_ (checksum of the textual
information of the element). During the second similarity score calculation,
Water gives a greater weight (0.9) to the _XPath_ similarity because the
authors of Water assume that the _XPaths_ of the matched elements are usually
very similar after the version update. Instead of its original setting that
returns one best matched element, as Water keeps the candidates in a ranked
list, we use the top 10 elements in the ranked list as candidates for the pre-
selection.

Edit Distance::

Inspired by the key idea of prioritizing XPath similarity in Water, we design
a simplified element matching algorithm to only consider the XPath similarity.
Similar to Water, we use Levenshtein distance (Ristad and Yianilos, 1998) to
measure the difference between _XPaths_. Levenshtein distance measures the
minimum number of insertions, deletions, and substitutions required to
transform one string into another. Greater _XPath_ Edit Distance means that
the two elements are less similar. This algorithm is also employed in other
field of Web testing (e.g., prior work have used the Levenshtein distance to
detect conflicting and outdated data on Webpages(Jnoub et al., 2018)). This
variant returns the top 10 elements ranked in descending order according to
their XPath similarities.

###  4.4. Explanation Validator

As shown in the highlighted texts in Table 2, we instruct ChatGPT to generate
a brief explanation for describing the attributes that it uses for selecting
the best matched element. Our intuition is that _if the provided explanation
is consistent with its selection, then the selection is more likely to be
accurate_ (and a repaired statement is more likely to be successfully
generated). Based on this intuition, we have developed an explanation
validator for checking whether the explanation given by ChatGPT is consistent
with the actual selection. Specifically, for each attribute $a$ mentioned in
the explanation, our explanation validator calculates the following to
determine the most similar element for the consistency calculation
$cons(a_{i},R)$ where $cons(a_{i},R)$=1 if the most similar element has been
selected:

Position::

We use Euclidean distance for calculating position-related attributes (e.g., x
and y coordinates), and we choose the element with the minimum distance as the
most similar.

Size::

We calculate the product of the width and the height of an element, and
consider the element with the smallest size difference as the most similar.

isLeaf::

We check if the value of _isLeaf_ is the same (true or false).

Other attributes::

We use Levenshtein edit distance to measure similarity, with the smallest
distance considered as the most similar.

In the cases where multiple candidate elements and the target element share
the same similarity for a particular attribute, we will retain the results of
multiple candidate elements and consider the explanation given by ChatGPT to
be consistent if either one of these candidate elements has selected.

###### Definition 4.1 (Explanation Consistency (EC)).

Given the _target element_ $t$ (element in the old version of the Webpage to
be matched), the selection results $R$ and an explanation $e$ where $e$
mentioned one or more attributes $A=a_{1},a_{2},...,a_{n}$, we calculate the
_Explanation Consistency (EC)_ of $e$ by computing

cons($a_{i},R$) for each attribute $a_{i}$ where $cons(a_{i},R)$ checks
whether each attribute $a_{i}$ of $R$ is most similar to that of the target
element $t$ ($cons(a_{i},R)$=1 if the most similar element based on $a\\_{i}$
is selected in $R$, and $cons(a_{i},R)$=0 otherwise.

$EC(e)=\frac{\sum_{i=1}^{n}cons(a_{i}\in A,R)}{|A|}$

Def. 4.1 presents the definition for Explanation Consistency (EC). For each
explanation generated by ChatGPT, our explanation validator checks whether the
attributes mentioned in the explanation that are similar between the selected
element and the target element are consistent with the calculated values of
all mentioned attributes. If the calculated values for all mentioned
attributes are consistent, then our explanation validator considers the
provided explanation as _consistent_ across all mentioned attributes ($EC$=1).

##  5\. Experimental Setup

Dataset. We use an existing dataset (Imtiaz et al., 2021a) for evaluating the
effectiveness of Web test repair approaches. The dataset contains Java
Selenium UI tests from five open-source Web applications (3 of which are used
in the experiment of VISTA (Stocco et al., 2018)). All of these open-source
applications are hosted in Sourceforge (except for MantisBT that is hosted in
GitHub (man, 2022)). We follow the same filtering process of a prior
evaluation (Lin et al., 2023) to remove duplicated tests and non-broken tests.
Subsequently, we obtained 62 test cases containing 139 broken statements as
our dataset.

Preparing ground truth dataset. As our evaluation dataset only has tests for
the old versions of Web applications, we need to manually label the ground
truths for the matching UI elements for the new versions. Specifically, the
first two authors (i.e., annotators) independently labelled the ground truths
for each UI element located in the broken statement in the dataset for the
three individual baselines (Water,Vista,Edit Distance). Both annotators are
graduate students with more than one year of experience in relevant research
of Web UI test repair. For 12 cases out of 3*139=417 cases (139 for each of
the three individual baselines), the annotators have disagreements so they
meet to resolve the disagreement (Cohen’s Kappa=0.94 which indicates almost
perfect agreement).

Table 4 shows the old and the new updated versions of the open-source Web
applications in our dataset. The “$\Delta$V” column denotes the number of
versions that occur between the old and the new versions (a greater number of
versions usually denotes more substantial UI changes between the two versions,
leading to more challenging matching and repair). The “Test” column represents
the number of tests for each app, whereas the “Broken Stmt” denotes the number
of broken statements for each app. Note that one test could have multiple
broken statements. In our experiment, we assume that each broken statement is
independent of subsequent statements (a repair failure of a preceding
statement will not be propagated to subsequent statements) so that we can
measure the effectiveness of ChatGPT in repairing each broken statement. This
assumption is similar to prior evaluations of learning-based automated program
repair techniques where the correct fault location is provided (perfect
localization) (Li et al., 2022; Lutellier et al., 2020).

All experiments are run on a computer with an Intel Core i5 processor (1.6GHz)
and 12 GB RAM. For the experiments related to ChatGPT, we use the API of
gpt-3.5-turbo model and set the temperature to 0.8 as used in prior work (Fan
et al., 2023). Our experiment with ChatGPT costs USD 10.96 in total (which
uses 8,429 API requests, and 7,178,802 tokens).

Table 4. Statistics of open-source Web apps in our dataset

Application |  $\Delta$V | Old Version | Updated Version | Tests | Broken Stmt
---|---|---|---|---|---
AddressBook | 8 | 4.0 | 6.1 | 2 | 2
Claroline | 29 | 1.10.7 | 1.11.5 | 27 | 53
Collabtive | 5 | 0.65 | 1 | 4 | 11
MantisBT | 38 | 1.1.8 | 1.2.0 | 25 | 66
MRBS | 24 | 1.2.6.1 | 1.4.9 | 4 | 7

##  6\. RQ1: Effectiveness of UI Matching

###  6.1. Matching Result and Analysis

Table 5. The number of correct matching and the number of correct repairs of
different approaches

Applications | Vista |  Vista+ChatGPT | Water |  Water+ChatGPT | Edit Distance |  Edit Distance+ChatGPT
---|---|---|---|---|---|---
Matching | Repair | Matching | Repair | Matching | Repair | Matching | Repair | Matching | Repair | Matching | Repair
AddressBook | 1 | 1 | 1 | 1 | 2 | 2 | 2 | 2 | 0 | 0 | 2 | 2
Claroline | 48 | 47 | 51 | 51 | 18 | 18 | 18 | 18 | 1 | 1 | 47 | 47
Collabtive | 1 | 1 | 1 | 1 | 10 | 10 | 7 | 7 | 8 | 8 | 8 | 8
MantisBT | 11 | 11 | 37 | 37 | 50 | 50 | 56 | 56 | 34 | 34 | 63 | 62
MRBS | 7 | 7 | 7 | 7 | 1 | 1 | 3 | 3 | 0 | 0 | 2 | 2
Total | 68 | 67 | 97 | 97 | 81 | 81 | 86 | 86 | 43 | 43 | 122 | 121

Our evaluation includes a total of six approaches, including individual
baselines (Vista, Water, Edit Distance), and their respective combinations
with ChatGPT. For each approach, we recorded its respective abilities to
correctly match the ground truth elements of the broken statements in our
dataset.

The “Matching” columns in Table 5 show the comparison results for the six
approaches on the evaluated applications. If we compare the overall matching
results for the individual baseline with its corresponding combination with
ChatGPT, we observe that all combinations outperform the individual baseline
(i.e., 97 versus 68 for Vista, 86 versus 81 for Water, and 122 versus 43 for
Edit Distance). This result confirms our hypothesis that the combinations of
prior test repair approaches with ChatGPT help to improve the UI matching
results.

Finding 1: All the combinations of prior Web test repair approaches with
ChatGPT outperforms the corresponding standalone approach (without ChatGPT).
Implication 1: Our suggested workflow of using prior test repair approaches
for pre-selecting a ranked list of candidate elements and then using ChatGPT
for a global matching is effective in improving the overall accuracy of UI
matching.

We notice that Vista’s matching algorithm that uses visual information is more
effective for certain applications (e.g., Claroline and MRBS). However, the
overall results show that Edit Distance+ChatGPT performs the best (fixing 122
out of a total of 139 broken statements), outperforming all the other
approaches. Specifically, the combination of Edit Distance+ChatGPT yields the
greatest improvement (183.72%) over the individual Edit Distance approach.
After combining with ChatGPT, it matches 79 broken statements that originally
could not be matched and 43 broken statements that could originally be
matched, without matching any originally matchable broken statements
incorrectly. Considering the fact the individual Edit Distance performs the
worst among all the individual baselines, we think this result is a bit
counterintuitive because one would select the tool that performs well
individually (i.e., Water) for combining with ChatGPT to achieve further
improvement.

Finding 2: Although the individual Edit Distance approach performs the worst
among all the individual baselines, its combination with ChatGPT outperforms
all the evaluated approaches. Implication 2: Edit Distance, a simplified
variant of Water, integrates well with ChatGPT compared to other combinations.
By ranking elements using only XPath similarity, Edit Distance delegates the
responsibility of matching other attributes to ChatGPT which later performs
global view matching.

###  6.2. Ranking performance of the baselines

As the combination of Edit Distance+ChatGPT outperforms all evaluated
approaches, we are interested in analyzing the reasons behind the improvement.
Specifically, as the baselines Water and Vista originally return only one
element as the best matching result, we analyze the ranking performance of
each baseline approach. Given a selected element $s_{e}$ by a tool $t$ and the
correct element $t_{e}$ (i.e., the corresponding element in the ground truth),
we consider $t$ _hits_ if $s_{e}$ is exactly the same as $t_{e}$. If one of
the elements in ranked list produced by a tool $t$ hits, we record its ranking
to evaluate the ranking performance.

Figure 4. Comparison of Hit Ratios for the three baselines

We employ metrics commonly used in evaluating top-N recommendation task (Li et
al., 2020): Top-K Hit Ratio (HR, or Recall, in this study, the proportion of
experimental instances in which the top N candidates selected by each baseline
contain ground truth in the candidate list). Figure 4 depicts the variation of
Hit Ratio with increasing values of N for the three baselines.

We observe several interesting trends in Figure 4: 1) When N=1, the Hit Ratio
for Vista and Water has already reached around 50%, while Edit Distance’s Hit
Ratio is only 30.9%. This indicates that the top candidate selected by Vista
and Water already effectively hits ground truth, whereas Edit Distance’s
advantage is less evident. 2) When N increases, the Hit Ratios for Vista and
Water increase slowly. Even when N grows to 10, their Hit Ratios only increase
by less than 25% compared to when N is 1. In contrast, Edit Distance’s Hit
Ratio gradually increases, reaching 68.3% when N=5; and when N=9 and 10, Edit
Distance’s Hit Ratio reaches 84.2% and 90.1%, respectively. Hence, if the
number of selected candidates is expanded to the top 10 elements in the
candidate list, Edit Distance is more likely to hit the ground truth element
compared to Vista and Water, leading to more potential improvement when
combining with ChatGPT.

##  7\. RQ2: Effectiveness of Test Repair

Before checking for the repair correctness of each repaired statement, we
first check if ChatGPT has the correct matching result for the broken
statement because a correct repair can only be generated after a correct
matching.

Repair Correctness. As the ground truth repaired statements (the repair
written by developers of the Web apps) are unavailable in our dataset and the
repaired statements generated by ChatGPT may have diverse fix patterns, we
need to manually validate the correctness of all generated repaired
statements. To reduce the manual effort in validating the repair correctness
for the generated repaired statements, we use a semi-automated approach to
verify the correctness of each generated repaired statement. Specifically,
given the original broken statement $o$, the repaired statement $r$, and the
ground truth element $e$, we design a parser that automatically parses the
locator type (e.g., By.name and By.xpath), and the expression within the
locator (e.g., the XPath value) in the repaired statement $r$ to verify their
correctness with respect to the ground truth element $e$. For example, if
ChatGPT uses By.xpath as the locator, it should use the XPath information of
the element rather than the value of other attributes. To identify cases that
require subsequent manual analysis, our parser also finds repaired statements
where more substantial changes have been introduced by ChatGPT, including (1)
different types of locators are used in $o$ and $r$ are used or (2) additional
statements have been added to the repaired statement $r$.

The “Repair” columns in Table 5 shows the repair results for all evaluated
approaches. We can observe that the number of correct repair is almost equal
to the number of correct matching (except for one case in Vista and Edit
Distance+ChatGPT, respectively). Vista fails to repair one statement in
Claroline after correctly matching the element as it currently does not
support the repair of assertion values (Stocco et al., 2018). Meanwhile, for
MantisBT, Edit Distance+ChatGPT correctly matches the element but generates a
repair that modify the original intention of the broken statements.
Specifically, we consider the repair as incorrect because the broken statement
sends a text input “” to the element, but Edit Distance’s repaired statement
sends “Test” to the element, which change the intention of original test
statement. Nevertheless, the Edit Distance+ChatGPT combination still excels
with the best performance with 121 correct repairs.

Finding 3: The repair performance of all evaluated approaches is generally
similar to the matching performance, except for two cases where the generated
repairs are incorrect. Implication 3: After matching the correct element, most
approaches could generate the correct repairs. For two cases, we identify some
limitations of evaluated approaches, including lack of diverse fix patterns
(Vista) and introducing non-intention perserving changes (Edit
Distance+ChatGPT).

Fix Patterns used by ChatGPT. Our prompt only directs ChatGPT to modify the
locator and outdated assertion in the broken statement based on the elements
it selects for repair because we want to keep the expressions outside the
locator unchanged. It is crucial that ChatGPT refrains from altering
expressions outside the broken statement locator to preserve the original
intention of the test. For example, actions such as changing a click operation
to entering text on the element should not take place. During the manual
validation of the repair result, we examine both the correctness of the
repaired locator and ensure that the content outside the locator remains
unaltered. To investigate the diversity of the generated repairs, we manually
analyze and derive some common fix patterns used by ChatGPT. Overall, we found
five fix patterns in the repair of ChatGPT, described below:

Single statement change: Modify locator values . 73.61% of correct repairs
share this fix pattern. Figure 5 shows an example of ChatGPT generated repair
where it modifies the value of the locator for updating the values of the
XPath.

⬇

1- driver.findElement(By.xpath("//*[@id=\"

2- content\"]/form[2]/div[4]/input")).click();

3+ driver.findElement(By.xpath("/html[1]/body[1]/

4+ div[1]/div[4]/form[2]/div[3]/input[1]")).click();

Figure 5. A ChatGPT fix for the broken statement in AddressBook that only
modifies the value of locator.

Single statement change: Modify assertion values. Figure 6 shows an example of
ChatGPT generated repair where it modifies the value of the assertion for
updating the values of the XPath. As Vista does not currently support update
of assertion values, ChatGPT could help Vista in generating repaired statement
that use more diverse fix patterns in this case. We observe that 13.82%
correct repairs involve assertion. In 7.79% of these cases, ChatGPT modified
assertion values to update outdated expected values.

⬇

1- assertEquals(driver.findElement(By.xpath("//div[@id=’

2- claroBody’]/div[2]/div")).getText(), "This username

3- is already taken");

4+ assertEquals(driver.findElement(By.xpath("/html[1]/body[1]/

5+ div[1]/div[2]/div[1]/div[2]")).getText(), "This user

6+ name is already taken");

Figure 6. A ChatGPT fix for the broken statement in Claroline that changes the
assertion value for XPath update.

Single statement change: Use different assertions and values. Figure 7 shows
an example of ChatGPT generated repair where it uses a different type of
assertion (replacing assertEquals with assertTrue) and assertion value (update
the XPath). In this case, the statement that replaces assertTrue with
assertEquals preserves the semantic of assertion as the original statement
still invoke .equals() within the assertion. In 10.39% of correct repairs that
involve assertions, ChatGPT used different assertions.

⬇

1- Assert.assertTrue(driver.findElement(By.xpath("//*[@id=\"

2- loginBox\"]/h3")).getText().equals("Authentication"));

3+ Assert.assertEquals("Authentication", driver.findElement

4+ (By.xpath("/html[1]/body[1]/div[1]/div[2]/div[1]/div[1]

5+ /div[1]/div[1]")).getText());

Figure 7. A ChatGPT fix for the broken statement in Claroline that uses
different assertions (replacing assertEquals with assertTrue) and assertion
value (update XPath).

Single statement change: Use different locators and values. Figure 1 shows an
example of ChatGPT generated repair where it uses a different type of locator
(replacing By.name with By.xpath) and locator value (update the XPath).
Compared to other locators (By.name and By.className), we observe that most
broken statements tend to use By.xpath locators (70.50%). Among the generated
correct repair, we observed that ChatGPT often use “By.xpath” when updating
the broken locator (99.82%). In 26.39% correct repair, ChatGPT changed the
locators and values. Among these cases, we only observe one case where ChatGPT
changes By.xpath into another locator (By.className). For the remaining cases,
ChatGPT replaces other locators with By.xpath.

Multi-statements change: Add statements and modify locators. 1.26% of correct
repairs have multi-statements changes. Figure 8 shows an example of ChatGPT
generated repair where (1) it stores the element into a temporary variable
element, (2) performs the click action on the variable, and (3) add an
assertion to check for the validity of the text of the element. Since the test
where the broken statement is located has other assertion that serves as test
oracle, adding the assertions is unnecessary in this example. However, this
example demonstrates the ability of ChatGPT in generating assertions which
could be helpful for test case generation.

⬇

1- driver.findElement(By.xpath("//div[@id=’claroBody’]/

2- div[2]/div")).click();

3+ WebElement element = driver.findElement(By.xpath("/

4+ html[1]/body[1]/div[1]/div[2]/div[1]/div[1]"));

5+ element.click();

6+ String actualText = element.getText();

7+ String expectedText = "The new user has been sucessfully

8+ created";

9+ assertEquals(expectedText, actualText);

Figure 8. A ChatGPT generated fix for the broken statement in Claroline that
adds statements.

##  8\. RQ3: Quality of ChatGPT’s Explanation on Matching Result

Table 6. The number of times an attribute is mentioned (M) and the number of
times where the mentioned attribute is consistent (C) in the generated
explanation for each approach

Approach | id | name | class | XPath | text | tagName | linkText | location | size | isLeaf
---|---|---|---|---|---|---|---|---|---|---
C | M | C | M | C | M | C | M | C | M | C | M | C | M | C | M | C | M | C | M
Vista | 1 | 16 | 11 | 30 | 98 | 111 | 304 | 542 | 392 | 498 | 390 | 412 | 111 | 122 | 74 | 201 | 113 | 201 | 164 | 169
Water | 12 | 122 | 28 | 55 | 148 | 161 | 254 | 555 | 346 | 484 | 299 | 410 | 176 | 192 | 154 | 221 | 213 | 221 | 201 | 201
Edit Distance | 121 | 121 | 23 | 131 | 159 | 164 | 291 | 556 | 452 | 510 | 316 | 347 | 126 | 149 | 128 | 306 | 179 | 304 | 178 | 179
Total | 134 | 259 | 62 | 216 | 405 | 436 | 849 | 1653 | 1190 | 1492 | 1005 | 1169 | 413 | 463 | 356 | 728 | 505 | 726 | 543 | 549

To assess the quality of ChatGPT’s explanations for element matching results,
we use two metrics: (1) _mention frequency_ (the number of times where an
attribute $a$ has been mentioned $M$), and (2) _mention consistency_ (the
number of times where an attribute $a$ has been consistently mentioned $C$
where the consistency is determined by our explanation validator described in
Section 4.4). These two metrics helps in answering the research questions
below:

RQ3a::

What are the frequently mentioned attributes in ChatGPT’s explanation?

RQ3b::

What are the mentioned attributes that are consistent in ChatGPT’s
explanation?

Table 6 presents the mention frequency and mention consistency of ChatGPT for
each attribute for each approach. It can be observed in the table that ChatGPT
mentions two attributes most frequently: _XPath_ (1653) and _text_ (1492),
wheares the least mentioned attribute is _name_ (216). Compared to the
priority imposed by prior test repair approaches (shown in Table 3), this
result show that ChatGPT has certain preference towards particular attributes
since it often mention the _XPath_ and _text_ attributes regardless of the
baseline used for selecting the list of candidate elements.

Finding 4: ChatGPT frequently mentions the the _XPath_ and _text_ attributes
in the provided explanations. Implication 4: Similar to other approaches that
imposed certain priority, ChatGPT has certain preferences towards the _XPath_
and _text_ attributes.

Table 6 also shows that the three attributes with the greatest mention
consistency are _text_ (1190), _tagName_ (1005), and _XPath_ (849). Among the
two most frequently mentioned attributes (_XPath_ and _text_), we can observe
that although ChatGPT often prefers using these attributes for matching
elements, prioritizing the _text_ attribute over the _XPath_ attribute will
lead to better matching results (as the _text_ attribute is often mentioned
consistently).

Finding 5: The most frequently mentioned attribute by ChatGPT (i.e., XPath)
tends to lead to incorrect matching (low mentioned consistency). In contrast,
the _text_ attribute is high in mention frequency and mention consistency.
Implication 5: Prioritizing the _text_ attribute over the _XPath_ attribute
may be a better strategy for ChatGPT as it will lead to more accurate
matching.

Correlation between EC and correctness of the matching. Our explanation
validator measures the explanation consistency (EC) as a mechanism to check
and improve the reliability of the matching results provided by ChatGPT.
However, even if our explanation validator can accurately assess EC, the final
matching result may not be correct (i.e., may not retrieve the target element
in the labeled ground truth). To investigate the correlation between our
proposed EC and the _correctness of the final matching result_ (i.e., whether
it selects the element in ground truth), we measure the correlation between
these two variables. As these two variables belong to discrete (whether it
selects the element in ground truth is a binary variable) and continuous (EC
is a value between 0 and 1) categories respectively, we compute the Point-
Biserial Correlation Coefficient ($r_{pbi}$) (Tate, 1954) for all
explanations. Since this is a special case of the Pearson Correlation, we
assess the strength of the relationship by evaluating the numerical value of
the correlation coefficient. For the three combinations with ChatGPT, we
obtain the following values for the Point-Biserial Correlation Coefficient:
$r_{pbi}$=0.51 for Vista+ChatGPT, $r_{pbi}$=0.84 for Water+ChatGPT, and
$r_{pbi}$=0.49 for Edit Distance+ChatGPT. These values indicate that EC and
the correctness of the matching for the Vista+ChatGPT and Edit
Distance+ChatGPT are only moderately correlated but _strongly correlated for
the Water+ChatGPT combination_. The strong correlation between EC and
Water+ChatGPT’s correctness of the matching implies that our explanation
validator will be most effective in improving the matching results for the
Water+ChatGPT combination among all the three evaluated combinations.

Finding 6: Among the three evaluated combinations, EC for the Water+ChatGPT
combination is strongly correlated with the matching correctness. Implication
6: The strong correlation indicates that our explanation validator that checks
and generates the self-correction prompt will be most effective in helping the
Water+ChatGPT combination to further improve the matching correctness.

##  9\. RQ4: Improving reliability of ChatGPT’s generated explanation

Table 7 presents the results of VISTA, WATER, and XPath tools combined with
ChatGPT before and after self-correction (SC), comparing the correct matches
and average explanation consistency (EC). We can observe that except for Edit
Distance+ChatGPT, the number of correct matches has increased for the other
approaches. Notably, the improvement over Water+ChatGPT is the greatest (it
gets 5 more correct matches and repair after SC). Meanwhile, we think that
Edit Distance+ChatGPT does not show any improvement due to the moderately
positive correlation between EC and matching accuracy (0.49).

Finding 7: The improvement offered by the self-correction mechanism varies
across different baselines. Among the three combinations, Water benefits the
most from self-correction (5 more correct matching after the correction).
Implication 7: Our proposed workflow of checking for explanation consistency
and generating self-correct prompt as hint to ChatGPT to improve the
reliability of its explanation could further improve the effectiveness of
matching and repair for certain combinations (e.g., Water+ChatGPT).

Table 7. The matching and repair results before and after self-correction

Approach | VISTA+ChatGPT | WATER+ChatGPT | XPath+ChatGPT
---|---|---|---
before SC | after SC | before SC | after SC | before SC | after SC
Matching | 97 | 97 | 86 | 91 | 122 | 123
Repair | 97 | 97 | 86 | 91 | 121 | 122

##  10\. Implications and Discussions

Our study identifies several key implications and suggestions for future test
repair and ChatGPT research.

Prioritization of attributes by Web UI test repair tools. Table 3 shows that
prior test repair tools have different ways of prioritizing attributes when
matching elements (e.g., Water gives higher priority to XPath similarity,
whereas Vista uses the position and the size information of an element to
extract visual information for matching). As the initial selected priorities
may be bias, our study shows that a global matching using ChatGPT can improve
over prior approaches (Finding 1) by mitigating this bias, leading to more
accurate matching. In fact, our study of the frequently mentioned attributes
in ChatGPT’s explanation also reveals that ChatGPT has preferences towards
certain attributes, e.g., _XPath_ and _text_ (Finding 5). Although this paper
only studies the prioritization of attributes used by two traditional Web test
repair approaches, we foresee that other UI matching techniques that measure
similarities using multiple properties may share similar bias. In future, it
is worthwhile to study (1) the characteristics of the selected prioritization
in other tasks where UI matching algorithms are used (e.g., compatibility
testing (Ren et al., 2022)), and (2) improving the effectiveness of other UI
matching techniques via subsequent matching using LLMs.

Test repair techniques used for selecting candidates elements. Our study that
compares three baseline approaches (Water,Vista, and Edit Distance) for the
pre-selection of candidate elements shows that a simplified version of Water
(i.e., Edit Distance) integrates well with ChatGPT where the combination of
Edit Distance+ChatGPT outperforms all the evaluated approaches (Finding 2).
Compared to Water that matches using multiple attributes (e.g., _id_ ,
_XPath_) and Vista that uses complex information for matching (position
information when extracting visual information), Edit Distance that matches
using only _XPath_ information is not effective as a standalone matching
algorithm (it performs worst among all individual baselines). However, our
study shows that by using only XPath similarity for matching, Edit Distance
delegates the responsibility of matching using other attributes to ChatGPT
which can later perform better global view matching. Intuitively, one may
think that Water that performs the best among the individual baselines would
lead to the best combination with ChatGPT. _As the best individual baseline
may not be the most effective combination with ChatGPT_ , our study urges
researchers to perform thorough evaluation when choosing appropriate baseline
to combine with ChatGPT for solving other software maintenance tasks.

LLM-based test repair. Our study shows promising results in using LLMs like
ChatGPT for Web test repair (Finding 3). Our manual analysis shows that it can
generate correct repairs for the broken statements using a diverse set of fix
patterns, including modifications and generations of assertions. This shows
the promises of using LLM for improving general purpose test repair technique
that fixes broken assertions (Daniel et al., 2011).

Improving reliability of ChatGPT output. Prior study has expressed concerns
regarding the tendency of ChatGPT to “hallucinate” when solving specific tasks
(Feldt et al., 2023), our workflow that checks whether the explanations
provided by ChatGPT are consistent along with the selected elements aims to
improve the reliability of ChatGPT generated outputs shows promising results
for improving the matching accuracy for certain combinations (Finding 7).
Although our study are limited to solving the tasks of Web UI element matching
and repair, we believe that our proposed way of checking for explanation
consistencies is general and can be applied to improve the reliability of
ChatGPT for other software maintenance tasks (e.g., test generation and
automated program repair).

##  11\. threats to validity

External Threats. During the ground truth construction and evaluating our way
of calculating EC, we mitigate potential bias by first asking two annotators
(two authors of the paper) to manually construct and cross-validate the
“ground truth target element and patch”, the two annotators meet to resolve
any disagreement during the annotation, and further discuss with other authors
until a consensus is reached. For verifiability, we also release our dataset.
Due to limited resources and cost budget, our experiments use the cost-
effective GPT 3.5-Turbo model which may be less effective than newer models in
solving the Web test repair task. As ChatGPT performance may varies in
different settings, our experiments may not generalize beyond the studied
settings, and fixing other forms of UI tests (this paper only focuses on Java
Selenium Web UI tests). We mitigate this threat by reusing settings and
suggestions in prior work (e.g., referring to OpenAPI official documentations
for prompt design and using temperature 0.8 as prior paper suggested (Fan et
al., 2023)), and evaluating on several test repair tools that use different
algorithms (e.g., text-based and visual-based). As our study only evaluates
the test repair capability of ChatGPT, the findings may not apply for other
LLMs. Nevertheless, our suggested workflow of using LLMs to perform global
matching and rematching based on inconsistent explanation are still generally
applicable.

Internal Threats. Our experimental scripts may have bugs that can affect our
results. To mitigate this threat, we made our scripts and results publicly
available.

Conclusion Threats. Conclusion threats of our study include (1) overfitting of
our dataset and (2) subjectivity of ground truth construction. To ensure that
the matching and repair task do not overlap with the training dataset of
ChatGPT to minimize (1), we manually analyze the updated UI tests (2) we have
manually labeled and created a ground truth dataset which can be used to
support future research in Web UI test repair. We mitigate (2) by cross-
validating between two annotators during ground truth construction.

##  12\. Conclusions

This paper presents the first feasibility study that investigates the
effectiveness of using prior Web UI repair techniques for initial local
matching and then using ChatGPT to perform global matching to mitigate the
bias in prioritization of attributes used by prior approaches. To mitigate
hallucination in ChatGPT, we propose an explanation validator that checks the
consistency of the provided explanation, and provides hints to ChatGPT via a
self-correction prompt to further improve its results. Our evaluation on a
widely used dataset shows that the combination with ChatGPT improve the
effectiveness of existing Web test repair techniques. Our study also reveals
several findings and implications. As an initial study that focuses on LLM-
based Web test repair, we hope that our study could shed light in improving
future Web UI test repair approaches.

## References

  * (1)
  * gpt (2022) 2022\.  https://platform.openai.com/docs/api-reference
  * man (2022) 2022\.  https://github.com/mantisbt/mantisbt
  * Briechle and Hanebeck (2001) Kai Briechle and Uwe D Hanebeck. 2001.  Template matching using fast normalized cross correlation. In _Optical Pattern Recognition XII_ , Vol. 4387. SPIE, 95–102.
  * Brisset et al. (2023) Sacha Brisset, Romain Rouvoy, Lionel Seinturier, and Renaud Pawlak. 2023.  SFTM: Fast matching of web pages using Similarity-based Flexible Tree Matching.  _Information Systems_ 112 (2023), 102126.
  * Chang et al. (2018) Nana Chang, Linzhang Wang, Yu Pei, Subrota K Mondal, and Xuandong Li. 2018.  Change-based test script maintenance for android apps. In _2018 IEEE International Conference on Software Quality, Reliability and Security (QRS)_. IEEE, 215–225.
  * Chen et al. (2021) Wei Chen, Hanyang Cao, and Xavier Blanc. 2021.  An Improving Approach for DOM-Based Web Test Suite Repair. In _Web Engineering_. Springer International Publishing, 372–387.
  * Choudhary et al. (2011) Shauvik Roy Choudhary, Dan Zhao, Husayn Versee, and Alessandro Orso. 2011.  Water: Web application test repair. In _Proceedings of the First International Workshop on End-to-End Test Script Engineering_. 24–29.
  * Daniel et al. (2011) Brett Daniel, Danny Dig, Tihomir Gvero, Vilas Jagannath, Johnston Jiaa, Damion Mitchell, Jurand Nogiec, Shin Hwei Tan, and Darko Marinov. 2011.  Reassert: a tool for repairing broken unit tests. In _Proceedings of the 33rd International Conference on Software Engineering_. 1010–1012.
  * Fan et al. (2023) Zhiyu Fan, Xiang Gao, Martin Mirchev, Abhik Roychoudhury, and Shin Hwei Tan. 2023.  Automated repair of programs from large language models. In _2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE)_. IEEE, 1469–1481.
  * Fatima et al. (2023) Sakina Fatima, Taher A. Ghaleb, and Lionel Briand. 2023.  Flakify: A Black-Box, Language Model-Based Predictor for Flaky Tests.  _IEEE Transactions on Software Engineering_ 49, 4 (2023), 1912–1927.  https://doi.org/10.1109/TSE.2022.3201209
  * Feldt et al. (2023) Robert Feldt, Sungmin Kang, Juyeon Yoon, and Shin Yoo. 2023.  Socratest: Towards Autonomous Testing Agents via Conversational Large Language Models. In _Proceedings of the 38th IEEE/ACM International Conference on Automated Software Engineering_.
  * Feng and Chen (2023) Sidong Feng and Chunyang Chen. 2023.  Prompting Is All Your Need: Automated Android Bug Replay with Large Language Models.  _arXiv preprint arXiv:2306.01987_ (2023).
  * Gao et al. (2016) Zebao Gao, Zhenyu Chen, Yunxiao Zou, and Atif M. Memon. 2016.  SITAR: GUI Test Script Repair.  _IEEE Transactions on Software Engineering_ 42, 2 (2016), 170–186.  https://doi.org/10.1109/TSE.2015.2454510
  * Hammoudi et al. (2016) Mouna Hammoudi, Gregg Rothermel, and Paolo Tonella. 2016.  Why do Record/Replay Tests of Web Applications Break?. In _2016 IEEE International Conference on Software Testing, Verification and Validation (ICST)_. 180–190.  https://doi.org/10.1109/ICST.2016.16
  * Houston (2013) Pete Houston. 2013.  _Instant jsoup How-to_.  Packt Publishing Ltd.
  * Imtiaz et al. (2021b) Javaria Imtiaz, Muhammad Zohaib Iqbal, et al. 2021b.  An automated model-based approach to repair test suites of evolving web applications.  _Journal of Systems and Software_ 171 (2021), 110841.
  * Imtiaz et al. (2021a) Javaria Imtiaz, Muhammad Zohaib Iqbal, and et al. 2021a.  An automated model-based approach to repair test suites of evolving web applications.  _Journal of Systems and Software_ 171 (2021), 110841.
  * Jnoub et al. (2018) Nour Jnoub, Wolfgang Klas, Peter Kalchgruber, and Elaheh Momeni. 2018.  A Flexible Algorithmic Approach for Identifying Conflicting/Deviating Data on the Web. In _2018 International Conference on Computer, Information and Telecommunication Systems (CITS)_. 1–5.  https://doi.org/10.1109/CITS.2018.8440185
  * Joshi et al. (2023) Harshit Joshi, José Cambronero Sanchez, Sumit Gulwani, Vu Le, Gust Verbruggen, and Ivan Radiček. 2023.  Repair Is Nearly Generation: Multilingual Program Repair with LLMs.  _Proceedings of the AAAI Conference on Artificial Intelligence_ 37, 4 (Jun. 2023), 5131–5140.  https://doi.org/10.1609/aaai.v37i4.25642
  * Kirinuki et al. (2019) Hiroyuki Kirinuki, Haruto Tanno, and Katsuyuki Natsukawa. 2019.  COLOR: correct locator recommender for broken test scripts using various clues in web application. In _2019 IEEE 26th International Conference on Software Analysis, Evolution and Reengineering (SANER)_. IEEE, 310–320.
  * Lemieux et al. (2023) Caroline Lemieux, Jeevana Priya Inala, Shuvendu K. Lahiri, and Siddhartha Sen. 2023.  CodaMosa: Escaping Coverage Plateaus in Test Generation with Pre-Trained Large Language Models. In _Proceedings of the 45th International Conference on Software Engineering_ (Melbourne, Victoria, Australia) _(ICSE ’23)_. IEEE Press, 919–931.  https://doi.org/10.1109/ICSE48619.2023.00085
  * Leotta et al. (2015) Maurizio Leotta, Andrea Stocco, Filippo Ricca, and Paolo Tonella. 2015.  Using multi-locators to increase the robustness of web test cases. In _2015 IEEE 8th International Conference on Software Testing, Verification and Validation (ICST)_. IEEE, 1–10.
  * Li et al. (2020) Dong Li, Ruoming Jin, Jing Gao, and Zhi Liu. 2020.  On Sampling Top-K Recommendation Evaluation. In _Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining_ (Virtual Event, CA, USA) _(KDD ’20)_. Association for Computing Machinery, New York, NY, USA, 2114–2124.  https://doi.org/10.1145/3394486.3403262
  * Li et al. (2017) Xiao Li, Nana Chang, Yan Wang, Haohua Huang, Yu Pei, Linzhang Wang, and Xuandong Li. 2017.  ATOM: Automatic Maintenance of GUI Test Scripts for Evolving Mobile Applications. In _2017 IEEE International Conference on Software Testing, Verification and Validation (ICST)_. 161–171.  https://doi.org/10.1109/ICST.2017.22
  * Li et al. (2022) Yi Li, Shaohua Wang, and Tien N. Nguyen. 2022.  DEAR: A Novel Deep Learning-based Approach for Automated Program Repair. In _2022 IEEE/ACM 44th International Conference on Software Engineering (ICSE)_. 511–523.  https://doi.org/10.1145/3510003.3510177
  * Lin et al. (2023) Yuanzhang Lin, Guoyao Wen, and Xiang Gao. 2023.  Automated Fixing of Web UI Tests via Iterative Element Matching. In _38th IEEE/ACM International Conference on Automated Software Engineering_.
  * Liu et al. (2023) Zhe Liu, Chunyang Chen, Junjie Wang, Xing Che, Yuekai Huang, Jun Hu, and Qing Wang. 2023.  Fill in the Blank: Context-aware Automated Text Input Generation for Mobile GUI Testing. In _2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE)_. 1355–1367.  https://doi.org/10.1109/ICSE48619.2023.00119
  * Lutellier et al. (2020) Thibaud Lutellier, Hung Viet Pham, Lawrence Pang, Yitong Li, Moshi Wei, and Lin Tan. 2020.  Coconut: combining context-aware neural translation models using ensemble for program repair. In _Proceedings of the 29th ACM SIGSOFT international symposium on software testing and analysis_. 101–114.
  * M. Leotta and Tonella (2016) F. Ricca M. Leotta, A. Stocco and P. Tonella. 2016.  Robula+: An algorithm for generating robust XPath locators for web testing. In _J. Softw. Evol. Process_ , Vol. 28. 177–204.
  * Mariani et al. (2021) Leonardo Mariani, Ali Mohebbi, Mauro Pezzè, and Valerio Terragni. 2021.  Semantic Matching of GUI Events for Test Reuse: Are We There Yet?. In _Proceedings of the 30th ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA)_ (Virtual, Denmark). Association for Computing Machinery, New York, NY, USA, 177–190.  https://doi.org/10.1145/3460319.3464827
  * Mashhadi and Hemmati (2021) Ehsan Mashhadi and Hadi Hemmati. 2021.  Applying CodeBERT for Automated Program Repair of Java Simple Bugs. In _2021 IEEE/ACM 18th International Conference on Mining Software Repositories (MSR)_. 505–509.  https://doi.org/10.1109/MSR52588.2021.00063
  * OpenAI (2023a) OpenAI. 2023a.  _Six Strategies for Getting Better Results with GPT_.  https://platform.openai.com/docs/guides/prompt-engineering/six-strategies-for-getting-better-results Accessed: November 15, 2023.
  * OpenAI (2023b) OpenAI. 2023b.  _Strategy: Split Complex Tasks into Simpler Subtasks_.  https://platform.openai.com/docs/guides/prompt-engineering/strategy-split-complex-tasks-into-simpler-subtasks Accessed: November 15, 2023.
  * OpenAI Help (2023) OpenAI Help. 2023.  _ChatGPT API Transition Guide_.  https://help.openai.com/en/articles/7042661-chatgpt-api-transition-guide Accessed: November 15, 2023.
  * Pan et al. (2020) Minxue Pan, Tongtong Xu, Yu Pei, Zhong Li, Tian Zhang, and Xuandong Li. 2020.  GUI-Guided Test Script Repair for Mobile Apps.  _IEEE Transactions on Software Engineering_ (2020), 1–1.  https://doi.org/10.1109/TSE.2020.3007664
  * Ren et al. (2022) Yanwei Ren, Youda Gu, Zongqing Ma, Hualiang Zhu, and Fei Yin. 2022.  Cross-Device Difference Detector for Mobile Application GUI Compatibility Testing. In _2022 IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW)_. IEEE, 253–260.
  * Ristad and Yianilos (1998) Eric Sven Ristad and Peter N Yianilos. 1998.  Learning string-edit distance.  _IEEE Transactions on Pattern Analysis and Machine Intelligence_ 20, 5 (1998), 522–532.
  * Santhosh et al. (2023) R. Santhosh, M. Abinaya, V. Anusuya, and D. Gowthami. 2023.  ChatGPT: Opportunities, Features and Future Prospects. In _2023 7th International Conference on Trends in Electronics and Informatics (ICOEI)_. 1614–1622.  https://doi.org/10.1109/ICOEI56765.2023.10125747
  * Shao et al. (2021) Fei Shao, Rui Xu, Wasif Haque, Jingwei Xu, Ying Zhang, Wei Yang, Yanfang Ye, and Xusheng Xiao. 2021.  WebEvo: taming web application evolution via detecting semantic structure changes. In _Proceedings of the 30th ACM SIGSOFT International Symposium on Software Testing and Analysis_. 16–28.
  * Sobania et al. (2023) Dominik Sobania, Martin Briesch, Carol Hanna, and Justyna Petke. 2023.  An Analysis of the Automatic Bug Fixing Performance of ChatGPT. In _2023 IEEE/ACM International Workshop on Automated Program Repair (APR)_. 23–30.  https://doi.org/10.1109/APR59189.2023.00012
  * Stocco et al. (2018) Andrea Stocco, Rahulkrishna Yandrapally, and Ali Mesbah. 2018.  Visual web test repair. In _Proceedings of the 2018 26th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering_. 503–514.
  * Tate (1954) Robert F Tate. 1954.  Correlation between a discrete and a continuous variable. Point-biserial correlation.  _The Annals of mathematical statistics_ 25, 3 (1954), 603–607.
  * To et al. (2023) Hung Quoc To, Nghi DQ Bui, Jin Guo, and Tien N Nguyen. 2023.  Better Language Models of Code through Self-Improvement.  _arXiv preprint arXiv:2304.01228_ (2023).
  * Xia et al. (2023) Chunqiu Steven Xia, Yuxiang Wei, and Lingming Zhang. 2023.  Automated program repair in the era of large pre-trained language models. In _Proceedings of the 45th International Conference on Software Engineering (ICSE 2023). Association for Computing Machinery_.
  * Xiao et al. (2024) Tao Xiao, Christoph Treude, Hideaki Hata, and Kenichi Matsumoto. 2024.  DevGPT: Studying Developer-ChatGPT Conversations. In _Proceedings of the International Conference on Mining Software Repositories (MSR 2024)_.
  * Xu et al. (2021) Tongtong Xu, Minxue Pan, Yu Pei, Guiyin Li, Xia Zeng, Tian Zhang, Yuetang Deng, and Xuandong Li. 2021.  Guider: Gui structure and vision co-guided test script repair for android apps. In _Proceedings of the 30th ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA)_. 191–203.
  * Yoon et al. (2022) Juyeon Yoon, Seungjoon Chung, Kihyuck Shin, Jinhan Kim, Shin Hong, and Shin Yoo. 2022.  Repairing Fragile GUI Test Cases Using Word and Layout Embedding. In _2022 IEEE Conference on Software Testing, Verification and Validation (ICST)_. 291–301.  https://doi.org/10.1109/ICST53961.2022.00038
  * Yu et al. (2023) Shengcheng Yu, Chunrong Fang, Yuchen Ling, Chentian Wu, and Zhenyu Chen. 2023.  LLM for Test Script Generation and Migration: Challenges, Capabilities, and Opportunities.  _arXiv preprint arXiv:2309.13574_ (2023).
  * Zhang et al. (2013) Sai Zhang, Hao Lü, and Michael D Ernst. 2013.  Automatically repairing broken workflows for evolving GUI applications. In _Proceedings of the 2013 International Symposium on Software Testing and Analysis_. 45–55.