# Dichromatic Number and Cycle Inversions

Pierre Charbit and Stéphan Thomassé

(3 Janvier 2020)

###### Abstract

The results of this note were stated in the first author PhD manuscript in
2006 but never published. The writing of the proof given there was slightly
careless and the proof itself scattered accross the document, the goal of this
note is to give a short and clear proof using Farkas Lemma. The first theorem
is a characterization of the acyclic chromatic number of a digraph in terms of
cyclic ordering. Using this theorem we prove that for any digraph, one can
sequentially reverse the orientations of the arcs of a family of directed
cycles so that the resulting digraph has acyclic chromatic number at most $2$.

##  1 Introduction

In this paper all directed graphs (digraphs in short) are simple, i.e. contain
no loop and no multi-arc. Given a digraph $D$, we denote by $V(D)$ its set of
vertices and by $A(D)$ its set of arcs. An acyclic colouring of a digraph is
an assignment of colours to the vertices such that each colour class induces
an acyclic subdigraph, that is a subdigraph containing no directed cycle. The
acyclic chromatic number, or simply _dichromatic number_ , of a digraph $D$,
is defined to be the smallest number of colours required for an acyclic
colouring of $D$ and is denoted $\operatorname{\overrightarrow{\chi}}(D)$.
This notion was first introduced in 1982 by Neumann-Lara [4] and seems to be
the natural generalization for digraphs of the usual chromatic number.

The main result is to give a characterization of the existence of a proper
$k$-acyclic colouring of a digraph. From it we deduce a theorem about circuit
inversions, which was extended in 2020 for infinite digraphs by Ellis in
Soukup in [2].

##  2 The main result and its proof

######  Theorem 2.1.

A digraph $D$ can be partitioned into $k$ acyclic subgraphs (i.e.
$\operatorname{\overrightarrow{\chi}}(D)\leq k$) if and only if there is an
ordering $\sigma$ of the vertices of $D$ such that for any circuit $C$, the
number of arcs of $C$ going forward in $\sigma$ is at least $|C|/k$.

Denote by $n=|V(D)|$ and $m=|A(D)|$ the number of vertices and arcs of $D$. In
order to prove the result we consider the classical incidence matrix of the
digraph which is a $n\times m$ matrix that we will denote $M_{D}$, where each
column corresponds to an arc $a=xy$ of $D$ and contains only two non zero
entries : an entry $(-1)$ for the row corresponding to the tail $x$ and entry
$1$ for one corresponding to the head $y$. For a vector $z\in\mathbb{R}^{n}$
seen as a matrix with one column, ${}^{t}z$ denotes the row vector obtained by
transposition.

######  Proposition 2.2.

Let $D=(V,A)$ be a digraph with $n$ vertices and $m$ arcs, and denote by
$M=M_{D}$ its incidence matrix. Let $p\in R^{m}$ be a weighting of the arcs,
seen as a column vector. The three following propositions are equivalent :

  1. 1.

$\forall C\mbox{ directed circuit of }D,\sum_{a\in A(C)}p(a)\geq 0$

  2. 2.

$\forall c\in\mathbb{R}_{+}^{m}\ \ M.c=0\Rightarrow\ {}^{t}p.c\,\geq 0$

  3. 3.

$\exists z\in\mathbb{R}^{n},\ {}^{t}z.M\leq p$

###### Proof.

The first two are easily seen to be equivalent. Each positive vector in the
kernel is a weighting such that for every vertex the sum of the weights of the
out arcs is equal to the sum of the weights of the arcs entering it. There
must be a directed circuit of positive arcs. One can remove this weighted
circuit and iterate.

The equivalence between (2) and (3) is one version of the classical Farkas
Lemma (see for example [3], page 92) ∎

###### of Theorem 2.1.

The implication from left to right is easy : we first put all vertices of the
first color class in a way that all arcs go forward, then do similarly for a
second color class, and so on. The only backward arcs go from a color class to
another one placed before, so there cannot be more than $k-1$ successive back
arcs in any directed path, which proves the desired inequality for any
circuit.

Let us prove now the converse direction and let $\sigma$ be an ordering of the
vertices satisfying the conditions of the theorem. We define the following
weighting of the arcs :

$p(a)=forw_{\sigma}(a)-1/k$

where $forw_{\sigma}(a)=1$ if $a$ forward arc in $\sigma$ and $0$ otherwise.

The hypothesis on $\sigma$ easily implies that every directed circuit has non
negative weight for $p$. So by Proposition 1, there exists
$z\in\mathbb{R}^{V(D)}$ such that

$\forall a=xy\in A,z(y)-z(x)\leq forw_{\sigma}(a)-1/k,$

For $i\in\mathbb{Z}$ and $t\in\\{0,\ldots,k-1\\}$, let
$I_{i,t}=[i+\frac{t}{k},i+\frac{t+1}{k}[$ et
$D_{t}=z^{-1}(\cup_{i\in\mathbb{Z}}I_{i,t})$.

Now we claim that $D_{t}$ induces an acyclic digraph. Indeed :

  * •

There is no arc from $I_{i,t}$ to $I_{j,t}$ with $i<j$, since
$z(y)-z(x)\leq(1-1/k)$ for every arc.

  * •

If $a=xy$ is an arc with $z(x)$ and $z(y)$ both in $I_{i,t}$ then necessarily
$z(y)-z(x)>-1/k$ so necessarily $forw_{\sigma}(a)=1$. Therefore the digraph
induced by $I(i,t)$ must be acyclic.

This concludes the proof ∎

##  3 A corollary about cycle inversions

######  Corollary 3.1.

For any digraph $D$, it is possible to reverse some of its circuits so that
the resulting digraph $D^{\prime}$ has acylic chromatic number at most $2$.

Preuve :
Pick an arbitrary order. As long as there is one circuit whose number of
backward arcs is larger than its number of forward arcs, reverse it. By doing
this the total number of forward arcs always increase and since it cannot
increase infinitely, one gets the desired ordering. ∎

## References

  * [1] Pierre Charbit (2006) Plongements de Graphes et Étude des Circuits. Thèse de doctorat - Université Claude Bernard, Lyon.
  * [2] Paul Ellis and Dániel T. Soukup.  Cycle reversions and dichromatic number in tournaments.  European Journal of Combinatorics, 77:31-48, 2019.
  * [3] Jiri Matousek and Bernd Gärtner (2007) Understanding and using linear programming, Springer Verlag.
  * [4] V. Neumann-Lara.  The dichromatic number of a digraph.  Journal of Combinatorial Theory, Series B, 33:265–270, 1982.


Leveraging Large Language Models for Automated Web-Form-Test Generation: An Empirical Study

Tao Li, Chenhui Cui, Lei Ma, Dave Towey, Yujie Xie,  Rubing Huang
Tao Li, Chenhui Cui, Yujie Xie, and Rubing Huang are with the School of Computer Science and Engineering, Macau University of Science and Technology, Taipa, Macau 999078, China. E-mail: {3220007015, 3230002105<EMAIL_ADDRESS><EMAIL_ADDRESS>Lei Ma is with The University of Tokyo, Tokyo 113-8654, Japan, and also with the University of Alberta, Edmonton, AB T6G 2R3, Canada. E-mail<EMAIL_ADDRESS>Dave Towey is with the School of Computer Science, University of Nottingham Ningbo China, Ningbo, Zhejiang 315100, China. E-mail<EMAIL_ADDRESS>
The testing of web forms is an essential activity for ensuring the quality of web applications, which mainly involves evaluating the interactions between users and forms.
Automated test-case generation remains a challenge for web-form testing:
Due to the complex, multi-level structure of web pages, it can be difficult to automatically capture their inherent contextual information for inclusion in the tests.
Large Language Models (LLMs) have great potential for contextual text generation.
OpenAI's GPT LLMs have been receiving a lot of attention in software testing, however, they may fail to be applied in practice because of information security concerns.
To the best of our knowledge, no comparative study examining different LLMs has yet been reported for web-form-test generation.
To address this gap in the literature, we conducted a comprehensive empirical study investigating the effectiveness of 11 LLMs on 146 web forms from 30 open-source Java web applications.
According to the experimental results, different LLMs can achieve different testing effectiveness.
Notably, the GPT-4, GLM-4, and Baichuan2 LLMs can generate better web-form tests than the others.
Compared with GPT-4, other LLMs find it difficult to generate appropriate tests for web forms, resulting in decreased successfully-submitted rates (SSRs, measured by the proportions of the LLMs-generated web-form tests that can be successfully inserted into the web forms and submitted) ranging from 9.10% to 74.15%.
Nevertheless, some LLMs (such as GLM-3, GLM-4, Baichuan2, and Spark-3.5) achieve higher SSRs than GPT-3.5, indicating a better ability to generate appropriate tests for web forms.
Our findings also show that, for all LLMs, when the designed prompts include complete and clear contextual information about the web forms, more effective web-form tests were generated.
Finally, we offer some insights for using LLMs to guide automated web-form testing.

Automated Web-Form Testing,
Large Language Models (LLMs),
Web-Form-Test Generation,
Java Web Applications,
Empirical Study.

§ INTRODUCTION

IN the swiftly evolving digital era, web applications have become a cornerstone of daily interactions.
By March 2024, the Internet Archive had saved more than 866 billion web pages <cit.>.
A web application consists of various HyperText Markup Language (HTML) elements, such as links, buttons, and sliders.
Among them, web forms, as a key element composed of the Document Object Model (DOM), are a fundamental interface component.
They not only serve as an interaction bridge between users and web applications <cit.> but also play an important role in improving the user experience and data collection efficiency <cit.>.

Web-form testing has been widely used to ensure the quality of web forms <cit.>.
It aims at simulating user inputs and evaluating user interactions <cit.>.
However, there may be challenges to automatically generating web-form tests, due to the properties of web forms:
(1) Web forms generally have complex structures with various basic and customized components <cit.>.
The basic components, such as tags, elements, attributes, and placeholders, play a significant role in constructing web forms.
In addition, the developers may introduce customized components (for example, control logic code) that could increase the complexity of the web-form structures; and
(2) Web forms also provide diverse contextual information for user interaction <cit.>.
For example, web forms can offer a drop-down list or a set of radio boxes for users to make a single selection from multiple options.
Consequently, web-form testing is crucial to ensure accurate interaction and evaluation of contextual information in complex web structures.

Recently, Large Language Models (LLMs) <cit.> have significantly enhanced Natural Language Processing (NLP) technologies <cit.>, leading to a groundbreaking era of Artificial Intelligence Generated Content (AIGC) <cit.>.
Previous studies have shown that LLMs have the potential to improve software engineering <cit.>.
Due to their reasoning and text-input generation capabilities <cit.>,
LLMs can also use extracted contextual information from complex web-form structures to improve the web-form-test generation process <cit.>.

Among the available LLMs, the GPT series <cit.>, produced by OpenAI, has received a lot of attention in web-form testing <cit.>.
However, due to information security concerns <cit.>, it may not be possible to apply GPT LLMs in practical software projects.
For example, when generating texts, GPT LLMs may leak sensitive user information, such as personal identity information.
Many recent studies have only analyzed a single or very few LLMs <cit.>.
For example, Alian et al. <cit.> proposed a testing method that only compared two kinds of LLMs to guide web-form testing (GPT-4 <cit.> and LLaMa2 <cit.>).
To the best of our knowledge, there is no comprehensive research available for testers to fully understand the different performances of various LLMs.
Motivated by these facts, we conducted a comprehensive empirical study to extensively analyze the effectiveness of various LLMs in generating web-form tests.

Our Work:
(1) We accessed 11 LLMs via publicly available APIs.
(2) We designed three types of prompts based on the HTML content of web forms for LLMs to use: Raw HTML for Task Prompt (RH-P); LLM-Processed HTML for Task  Prompt (LH-P); and Parser-Processed HTML for Task Prompt (PH-P).
(3) We generated 14,454 web-form tests and executed them on 30 Java web open-source projects with 146 web forms from GitHub to evaluate the capabilities of the LLMs.

Key Findings:
(1) Different LLMs achieve different successfully-submitted results.
GPT-4, GLM-4, and Baichuan2 LLMs achieve better successfully-submitted rates (SSRs) — the proportions of the LLMs-generated web-form tests that can be successfully inserted into the web forms and submitted — indicating better web-form test generation (effectiveness) than the others.
In addition, the model size significantly affects the generation performance, such as with the LLaMa2 series of LLMs.
(2) Compared with GPT-4, the other LLMs have difficulty generating appropriate tests for web forms, resulting in a significant decrease in SSRs.
Specifically, the SSR performance of the other LLMS was between 9.10% and 74.15% lower.
Compared with GPT-3.5, some LLMs (such as GLM-3, GLM-4, Baichuan2, and Spark-3.5) were more suitable for generating appropriate web-form tests and achieving higher SSRs.
(3) Different contextual information and prompt constructions have different effects on the effectiveness of the LLMs.
The prompts constructed from parser-processed HTML (PH-P) are generally better than the other two types (RH-P and LH-P).

Practical Implications:
(1) To extract contextual information from web forms, it is necessary to fully understand their properties and ensure accurate parsing of the HTML content.
(2) To prune the HTML content of web forms and reduce complexity, it is necessary to remove redundant components (e.g., the customized components introduced by developers) and avoid requiring complete and complex web-form contextual information.
However, the pruned components should not affect the key contextual information of the web-form (e.g., IDs, which are critical for inserting the generated tests into the corresponding web forms).
(3) To select a specific LLM for generating tests for web forms, if there are no practical testing constraints (such as data privacy issues <cit.>), GPT-4 would be preferable due to its high quality and effective web-form tests.
Otherwise, some alternative LLMs should be selected, such as the LLaMa2 series.

Contributions: This study offers the following significant contributions:

* To the best of our knowledge, this is the first empirical study on multiple LLMs (11 LLMs) that investigates the potential for LLMs to generate automated web-form tests.

* We prune the HTML to reduce the complexity of the web-form structure.
We propose three context construction approaches to extract contextual information from web forms for prompt construction.

* We select 146 web forms from 30 open-source Java web applications to deeply evaluate the effectiveness of the different LLMs.

* We summarize three key findings from the experimental results, and provide two practical implications for future research on web-form testing.

The rest of this paper is organized as follows:
Section <ref> introduces some preliminaries, including the HTML parsing process, web-form testing, and LLMs.
Section <ref> describes the approach to conduct this empirical study.
Section <ref> explains the design of our experiments.
Section <ref> presents and analyzes the experimental results.
Section <ref> outlines some related work.
Section <ref> concludes this paper and discusses some future work.

HTML parsing process.

§ PRELIMINARIES

This section provides a brief overview of the basic concepts of HTML parsing, web-form testing, and LLMs.

§.§ HTML Parsing Process

Listing <ref> shows a basic example HTML structure, which is made up of various components, such as:
title in the  (lines <ref> to <ref>);
content in the  (lines <ref> to <ref>); and
image files in the  (line <ref>).
These components provide instructions to the browser for how to display information <cit.>.

[caption=The basic HTML structure., label=lst:html-script, captionpos=b,escapechar=|]
<!DOCTYPE HTML>
<title>This is a title.</title>

This is a body.

||

The process of parsing an HTML structure is illustrated in Figure <ref>.
It involves the following steps:
(1) The resource loader is initially activated to load the webpage corresponding to the URL.
(2) This loader uses the network module to initiate requests and handle responses.
(3) Data can be obtained from web pages or resources through synchronous and asynchronous methods.
(4) The web page is then handed over to an HTML interpreter and transformed into a series of words or tokens.
(5) Based on these words, the interpreter constructs nodes, forming a DOM tree <cit.>.
(6) If a node is written in JavaScript, then the JavaScript engine is called to interpret and execute it
(7) JavaScript code may modify the structure of the DOM tree.
(8) If a node uses other resources (such as images, Cascading Style Sheets (CSS), videos, etc.), then the resource loader is called to load them.

§.§ Web-Form testing

Web-form testing ensures the interactive functionality, usability, and compatibility of web forms in a web application.
Nowadays, automated testing methods are widely used <cit.>.
For example,
Selenium <cit.> is a popular automated testing framework for web-form testing.
The Selenium web driver, the core of Selenium <cit.>, allows testers to automatically control web applications' behavior on the real browsers by automated test scripts <cit.>.
It uses the native automation support from web browsers <cit.> to enable an end-to-end test execution <cit.>.

Process of using Selenium to control automated testing.

Figure <ref> outlines the main steps in web-form testing using Selenium:
In Step (1), a customized automated test script is developed, which designs the execution logic of the testing task and the initiation request for the Selenium web driver.
In Step (2), the web driver sends an establish connection command to the browser where the web application with the Application Under Test (AUT) has been installed.
After Steps (1) and (2), the pipeline between the test script and the web application is established.
Then, testers can use this pipeline to automatically execute test scripts to test the web-form.
In Step (3), the UI operations (e.g., click a button) for the web-form are defined in the test script and sent to the browser.
In Step (4), the web-form tests are executed within the AUT on the browser with a series of actions.
After executing the web-form tests, in Step (5), the browser returns the test results to the testers.
Finally, in Step (6), the testing environment is reset to the original state.

Web-form testing could be complex due to two properties of web forms:
(1) Web forms can comprise various components, including basic and customized components.
For example, tags are the basic components that specify the input content type;
elements establish the layout of the web-form; and
attributes specify the functionality and characteristics of the elements.
In addition, the customized components introduced by developers can enhance the diversity of the web-form structures, and can include some control logic.
(2) Web forms also provide diverse contextual information for users to interact with the web applications:
For example, web forms may provide different approaches for users to select from alternative options, such as using a selector list or a group of radio boxes.

§.§ LLMs

LLMs are complex deep neural networks trained on various datasets, including books, code, articles, and websites.
This training enables the model to discern and replicate the intricate patterns and relationships inherent in the language it learns.
As a result, LLMs can produce coherent content ranging from grammatically accurate text to syntactically correct code snippets <cit.>.
LLMs <cit.> are advanced technologies in deep learning and NLP, with models including GPT <cit.>, GLM <cit.>, and LLaMa2 <cit.>.
These models learn complex features (such as language structure, grammar, and semantics) by training on large-scale text data.
With this training, LLMs can complete more complex and diverse NLP tasks, such as text generation <cit.>, translation <cit.>, and AI assistance (e.g., contributing to a range of software engineering tasks, including specification generation and the translation of legacy code) <cit.>.

LLMs are used into four key stages of the software engineering lifecycle:
(1) software requirements and design (e.g., software specifications generation <cit.>, GUI layouts <cit.>);
(2) software development (e.g., code generation <cit.>, code summary <cit.>);
(3) software testing (e.g., unit test generation <cit.>, GUI testing <cit.>); and
(4) software maintenance (e.g., code review <cit.>, bug report detection <cit.>).
LLMs can solve complex software engineering problems and promote better software engineering development.

Currently, most generative LLMs use the Transformer architecture, which is composed of two main components: the encoder and the decoder <cit.>.
Here, we take LLaMa2(7B) as an example to illustrate the basic structure of an LLM.
Figure <ref> in the appendix file shows the structure of LLaMa2(7B) along with the input text (prompt).
LLaMa2(7B) only uses the decoder part of the transformer, which is a decoder-only structure and shares 32 decoder layers.

Framework of this empirical study.

§ APPROACH

This study focuses on the automated web-form-test generation task for web-form testing.
Figure <ref> illustrates the framework of this empirical study, which includes the following five steps:
(1) HTML pruning;
(2) context construction;
(3) prompt design;
(4) LLM communication; and
(5) web-form-test insertion.

§.§ HTML Pruning

This section outlines the method of HTML pruning, a critical technique for clarifying the contextual information of web forms.

On the one hand, although HTML has a significant tree-like DOM structure <cit.>, it contains various types of components, which can make it difficult to filter out the key ones.
On the other hand, with the development of web forms, developers generally use essential basic components, such as name and ID.
Additionally, some customized components can be added to make the code logic more transparent, such as data-ID and user-role.
However, these customized components may make the HTML trimming process more complex.
Therefore, designing a method for pruning HTML is becoming increasingly challenging.

After analysis of the web-form structure, some key components (e.g., ID, name, type) are selected while pruning the remaining components.
In other words, the HTML contents of the web forms are simplified while maintaining semantic integrity, which makes the whole HTML clear and the web-form readable.
Meanwhile, selected key components enable us to successfully fill generated web-form tests into web forms, and also ensure the smooth execution of test scripts.
Algorithm <ref> provides the pseudocode for HTML pruning, i.e., the function $\texttt{\textbf{pruneHTML(}}\mathcal{D}, \mathcal{S}\texttt{\textbf{)}}$, where $\mathcal{D}$ is a web driver, and $\mathcal{S}$ is a filter tag set.
In the initialization stage, a web-form $\mathcal{F}$ is generated by searching for elements from $\mathcal{D}$ with the keyword $``form"$, i.e., the function $\texttt{\textbf{findElements(}}\mathcal{D}, ``form"\texttt{\textbf{)}}$.
For each element $\chi$ in $\mathcal{F}$, the algorithm checks each attribute $\alpha$ to find the potential tag $\tau$, i.e., the function $\texttt{\textbf{getTag(}$\alpha$\textbf{)}}$.
If the tag $\tau$ does not belong to the tag set $\mathcal{S}$, the attribute $\alpha$ will be removed from the element $\chi$; otherwise, $\alpha$ will be saved. After that, the updated element $\chi$ is appended to a pruned HTML $\mathcal{H}$.

HTML Pruning $\texttt{\textbf{pruneHTML(}}\mathcal{D}, \mathcal{S}\texttt{\textbf{)}}$

WebDriver $\mathcal{D}$, Filter tag set $\mathcal{S}$.
Pruned HTML $\mathcal{H}$.
$\mathcal{F} \leftarrow \texttt{\textbf{findElements(}}\mathcal{D},  ``form"\texttt{\textbf{)}}$;

$\mathcal{H} \leftarrow \{\}$;

each element $\chi\in \mathcal{F}$

each attribute $\alpha \in \chi$

$\tau \leftarrow \texttt{\textbf{getTag(}}\alpha\texttt{\textbf{)}}$;

$\tau \notin \mathcal{S}$

$\chi \leftarrow \texttt{\textbf{remove(}}\alpha,\chi\texttt{\textbf{)}}$;

$\mathcal{H} \leftarrow \textbf{\texttt{append(}}\mathcal{H}, \chi \textbf{\texttt{)}}$;

return $\mathcal{H}$;

§.§ Context Construction

This section offers an overview of the method of context construction, which involves transforming the contextual information of a web form into task prompts to guide the LLM to generate web-form tests.
LLMs have a strong logical understanding, and the ability to generate high-quality content. These mainly depend on the prompt quality <cit.>.
In this study, we focused on the application of LLMs to web-form-test generation.
To build prompts for LLMs, the contextual information needs to be constructed from the HTML content of the web-form (which includes various components such as placeholders and attributes).
Using the web-form structure, we propose three methods of context construction in this study.

§.§.§ Context from Raw HTML

The first method constructs context by directly using the raw HTML
this was motivated by the fact that the HTML layout effectively represents the relationships among the various components.
Due to the limited amount of context that LLMs can handle <cit.>, it is necessary to “simplify" the raw HTML content by removing some CSS and custom components originally defined by developers
the function $\texttt{\textbf{pruneHTML(}}\mathcal{D}, \mathcal{S}\texttt{\textbf{)}}$ (as shown in Algorithm <ref>).
Then, the pruned HTML is reorganized as a string, which is considered as the contextual information.

Context from LLM-Processed HTML

WebDriver $\mathcal{D}$, Filter tag set $\mathcal{S}$, LLM $\mathcal{L}$.
Context string $s$.

$s \leftarrow \varepsilon$;                 // $s$ is initialized as an empty string.

$\mathcal{H} \leftarrow \texttt{\textbf{pruneHTML(}}\mathcal{D}, \mathcal{S}\texttt{\textbf{)}}$;

$\Omega \leftarrow \texttt{\textbf{parseHTML(}}\mathcal{H}, \mathcal{L}\texttt{\textbf{)}}$;     // Parse the HTML as a list of JSON structured results by the LLM.

each $\chi \in \Omega$

$\tau \leftarrow \texttt{\textbf{getContext(}}\texttt{\textbf{getName(}}\chi\texttt{\textbf{)}},\texttt{\textbf{getValue(}}\chi\texttt{\textbf{)}}\texttt{\textbf{)}}$;

$s \leftarrow \texttt{\textbf{append(}}s,\tau\texttt{\textbf{)}}$;

return $s$;

§.§.§ Context from LLM-Processed HTML

The second method of context construction uses the LLM to pre-process HTML as the targeted format.
This is because LLMs can be very good at parsing HTML content.
We provide the LLM with web-form HTML content, and then ask it to parse the content according to the specific format.
More specifically, LLMs parse the contextual information from the HTML content (which includes accurately assembled elements of the web-form, such input tag name, tag ID, and tag type).
In general, LLMs convert web-form HTML into the JavaScript Object Notation (JSON) format, i.e., a list of JSON structured results.
Then, the context is constructed by traversing the web-form structure JSON, extracting the corresponding JSON information, and concatenating it into the web-form structure context content.

Algorithm <ref> provides the pseudocode for getting context from the LLM-Processed HTML.
At the initialization stage, $s$ is an empty string, and the HTML pruning $\mathcal{H}$ is generated by function $\texttt{\textbf{pruneHTML(}}\mathcal{D}, ``form"\texttt{\textbf{)}}$.
Then, $\mathcal{H}$ is transformed into a JSON structured objects list $\Omega$ through a specific LLM, i.e., the function $\texttt{\textbf{parseLLM(}}\mathcal{H}, \mathcal{L}\texttt{\textbf{)}}$.
For each element $\chi \in \Omega$, the algorithm uses the following functions to capture the contextual information:
$\texttt{\textbf{getName(}}\chi\texttt{\textbf{)}}$ retrieves the name of the JSON object $\chi$ (e.g., “hint text”);
$\texttt{\textbf{getValue(}}\chi\texttt{\textbf{)}}$ obtains the corresponding value of $\chi$ (e.g., “Please enter your name.”); and
$\texttt{\textbf{getContext}}$ concatenates the obtained name and value as natural language sentences (e.g., “The hint text is `Please enter your name.”').
After that, all reconstructed HTML information is reformed into a parsed context string as the context.

Basic framework of the prompt structure.

§.§.§ Context from Parser-Processed HTML

The third method for context construction is similar to the second, with the only difference being that they use different HTML-parsing functions ($\texttt{\textbf{parseHTML}}$ in Algorithm <ref>):
Instead of LLM as the HTML parser, a Java-based HTML parser (Jsoup[<https://jsoup.org/>.]) converts the web-form HTML into a list of JSON structured results. The context is also constructed as a string.

§.§ Prompt Design

This section provides a detailed introduction to the process of prompt design.
The constructed prompts are directly used to guide the web-form-test generation for LLMs.

After collecting and constructing the context (as discussed in Section <ref>), the next stage is to design the prompts for the LLMs.
This includes five steps that produce the following five prompt types:
(1) global prompt;
(2) task prompt;
(3) note prompt;
(4) instruction prompt; and
(5) objective prompt.
Figure <ref> presents the basic framework of the prompt structure, where the “%s” are replaced by the extracted information when testing the AUTs.

§.§.§ Global Prompt

The global prompt contains the most basic information, such as the name of the web application, the title of the web-form, and the number of elements within the form.

§.§.§ Task Prompt

The task prompt includes the core contextual information for the web application.
Based on the three types of context (Section <ref>), three types of task prompts are designed Raw HTML for Task Prompt (RH-P); LLM-Processed HTML for Task Prompt (LH-P); and Parser-Processed HTML for Task Prompt (PH-P).
Two categories of natural language sentences are designed for the LLMs to analyze:
Category 1 asks the LLMs to analyze the structure of the HTML code; and
Category 2 asks the LLMs to analyze the contextual information in natural language expressions.
As shown in Figure <ref>,
RH-P uses Category 1, while LH-P and PH-P use Category 2 (as they are based on the contextual information in the natural language expressions).

§.§.§ Note Prompt

The note prompt attempts to restrict the outputs from the LLMs, guiding them to return a brief output rather than detailed information, such as a step-by-step analysis.

§.§.§ Instruction Prompt

The instruction prompt guides the LLMs to return results in the specified format:
The generated results must be surrounded by a pair of triple quotation marks.

§.§.§ Objective Prompt

The objective prompt asks the LLMs to strictly follow the requirements and return the results.

After collecting the above five sub-prompts, we successively concatenate them into a string as the final prompt for the LLMs.

§.§ LLM Communication

This section elaborates on the communication with the LLM, which is a critical step for the LLMs to generate web-form tests.

Once the communication connection with the LLM is established, the LLM can be used to generate the web-form tests based on the extracted contextual information.

First, we standardize the encapsulation of various LLM APIs, which leads to better management, easier communication, and more convenient collection of experimental data.
We can communicate with a specific LLM by sending the name of the LLM to this encapsulated API.
Second, when running the AUT, we use three types of task prompts to construct the complete prompts (Section <ref>) to guide the LLMs to generate web-form tests:
The test script automatically sends the prompts as a message.
As shown in Figure <ref>, the contextual information is concatenated into three types of prompts (RH-P, LH-P, and PH-P) to communicate with a specific LLM and guide that LLM to generate web-form tests, respectively.
After receiving the response from the LLM, we extract the web-form tests from the response text.
Finally, we convert the extracted web-form tests into a set of key-value pairs for web-form-test insertion: The key stores the selectors of each component (i.e., a series of approaches for detecting the positions of the components in an HTML document such as ID selector,  CSS selector, etc.); and the value stores the corresponding generated web-form tests.

An example to illustrate the whole testing process.

§.§ Web-Form-Test Insertion

This section explains the insertion process of the web-form tests generated by the LLMs.
This step is the key to achieving automation of the entire web-form testing process.

After extracting all the LLM-generated web-form tests, they are stored in a set with the corresponding selectors for all the web-form components.
If the selector detects the web-form component, the corresponding generated web-form-test is inserted.
As shown in Figure <ref>, the function $selector, test$ implements the component selection and web-form-test insertion process.
Once the web-form tests are inserted into the corresponding components,
some UI operations (e.g., click a button) are performed to submit the inserted values to the web application's server.
As illustrated in Figure <ref>, the “Sign in” button is automatically clicked to submit the inserted values.
Finally, a monitor is set to wait for whether the submission triggers a response from the application server:
If a response is caught, then it is considered a successful submission; otherwise, it has failed.

§ STUDY DESIGN

This section introduces the details of our study design, including
the research questions,
the selection of LLMs,
the evaluation setup, and the experiment environment.

§.§ Research Questions

Our research aimed to evaluate the effectiveness of LLMs in generating tests for web forms, guided by the following three research questions:

* RQ1: How effective are web-form tests generated by different LLMs?

* RQ1.1: How well do different types of prompts guide the web-form-test generation?

* RQ1.2: How well do different LLMs generate web-form tests?

* RQ2: What is the quality of the generated web-form tests?

* RQ2.1: Why are some generated web-form tests not submitted?

* RQ2.2: What is the quality of the generated web-form tests, from the perspective of software testers?

* RQ3: What insights/advice can be offered to testers using LLMs for web-form testing?

* RQ3.1: What strategies can be used to design appropriate prompts for testing web forms?

* RQ3.2: What are the criteria for choosing the best LLM for a specific testing scenario?

List of subject LLMs.

No.     Name       Version           Company/Organization            No. of Parameters     Source                             Year of Release
1       GPT-3.5         gpt-3.5-turbo     OpenAI                  175B            <https://platform.openai.com>        2023

2       GPT-4           gpt-4             OpenAI                  Not Reported              <https://platform.openai.com>        2023

3       GLM-3           GLM-3             Zhipuai     Not Reported              <https://www.zhipuai.cn>             2023

4       GLM-4           GLM-4             Zhipuai     Not Reported              <https://www.zhipuai.cn>             2024

5       GLM-4V          GLM-4V            Zhipuai     Not Reported             <https://www.zhipuai.cn>             2024

6       Baichuan2       Baichuan2-53B     Baichuan-inc            53B           <https://api.baichuan-ai.com>       2023

7       LLaMa2(7B)      LLaMa2-7b         Meta                    7B            <https://dashscope.aliyuncs.com>     2023

8       LLaMa2(13B)     LLaMa2-13b        Meta                    13B           <https://dashscope.aliyuncs.com>     2023

9       LLaMa2(70B)     LLaMa2-70b        Meta                    70B           <https://dashscope.aliyuncs.com>     2023

10      Spark-3         Spark-3           IFLYTEK                 Not Reported              <https://spark-api.xf-yun.com>       2023

11      Spark-3.5       Spark-3.5         IFLYTEK                  Not Reported             <https://spark-api.xf-yun.com>       2023

§.§ LLM Selection

After an in-depth investigation and analysis of the current mainstream and widely-used LLMs <cit.>, we selected 11.
These state-of-the-art LLMs were chosen based on the number of parameters <cit.>, the ease of API access and integration <cit.>, and the availability of commercial open-source options <cit.>.

Table <ref> lists some relevant information for these 11 LLMs, including the model name and version, the owning company or organization, the number of parameters, the source website, and the year of its release.
Interestingly, some information about these LLMs was not fully disclosed, such as the number of parameters.
However, when these models were released, their capabilities were often vigorously promoted, which makes the empirical research in this article more important and meaningful.
Due to the space limitations, more details about the subject LLMs can be found in the appendix file.

§.§ Evaluation Setup

We designed different experimental stages to answer each research question.
For RQ1, we identified 300 Java web applications from GitHub using keywords such as “Java web”, “Jobs”, and “Books”.
These keywords were selected from an online statistical resource[<http://5000best.com/websites/>.] that lists the top 5000 famous websites.
We then cloned these selected web applications, and excluded those without web forms and those that could not run properly in the experimental environment.
Finally, 30 Java web applications, with 146 web forms, were selected for the experiment.
To evaluate the effectiveness of the web-form tests, a successfully-submitted rate (SSR)[This metric was originally referred to as the “form passing rate” <cit.>.] was used to measure the proportion of LLMs-generated web-form tests that can be successfully inserted into the web forms and submitted.

We ran each web-form three times under 11 LLMs and three types of prompts:
This was a total of $146 \times 11 \times 3 \times 3 = 14,454$ generated web-form tests.
For RQ1.1 and RQ1.2, we evaluated the SSR from the perspectives of the three types of prompts and 11 LLMs.

For RQ2.1, we conducted a case study to analyze why some generated web-form tests were not passing.
For RQ2.2, we designed an online questionnaire[<https://github.com/abelli1024/web-form-testing-empirical-study>.] to collect user evaluations of LLM-generated web-form tests.
We invited 20 testers with software testing experience from famous Internet enterprises and research institutions.
For each web-form, we presented the screenshot and the corresponding web-form-test text generated by the 11 LLMs and the three types of prompts.
We randomly selected 15 web forms from the 146 web forms for each tester.
We used Kendall’s W <cit.> value to measure the agreement among different testers for responses to the questionnaire statements (which used a 5-point Likert scale:
strongly disagree “1”;
disagree “2”;
neutral “3”;
agree “4”; and
strongly agree “5”).
A higher Kendall's W value (close to 1.0) indicates a higher level of agreement among the testers' evaluation results.
We also collected data on the number of testers who use LLMs in their testing processes.
Additionally, we identified the main concerns for users who currently use or intend to use LLMs for testing.

For RQ3.1 and RQ3.2, based on the findings from other research questions, we provide some advice for constructing prompts for LLMs, and for selecting a specific LLM for a particular testing scenario, respectively.

§.§ Experiment Environment

All experiments were conducted on a MacBook Pro laptop with an Apple M3 Max processor and 64GB of RAM.
The test script was developed in Java.
The version of Google Chrome was 122.0.6261.112 (official version) (arm64), and the version of the Chrome web driver was 122.0.6261.128 (r1250580).

§ RESULTS

This section introduces the experimental results, and answers the research questions.

§.§ RQ1: How effective are web-form tests generated by different LLMs?

This section discusses the effectiveness of the LLM-generated web-form tests.
We provide an analysis from the perspective of the three prompt types, and from the perspective of the 11 LLMs.

The SSR results on 438 test tasks of three types of prompt design.
(The symbol $\#$ denotes the successfully-submitted number;
$!\#$ denotes the non-successfully-submitted number; and
$\%$ denotes the SSR.)

1c|2*No.   1c|2*Methods
3c|RH-P      3c|LH-P      3c|PH-P     3cAverage
1r|$\#$     1r|$!\#$     1r|$\%$     1r|$\#$     1r|$!\#$     1r|$\%$     1r|$\#$     1r|$!\#$     1r|$\%$    1r|$\#$     1r|$!\#$     1r$\%$

1   GPT-3.5     198     240     45.21%     299     139     68.26%     365     73      83.33%    287.33	   150.67	   65.60%

2   GPT-4     433     5     98.86%     425     13      97.03%     436     2     99.54%     431.33	   6.67	   98.48%

3   GLM-3     339     99      77.40%     292     146     66.67%     350     88      79.91%    327.00	   111.00   	74.66%

4   GLM-4     378     60      86.30%     399     39      91.10%     399     39      91.10%    392.00   	46.00   	89.50%

5   GLM-4V      0     438     0.00%      0     438     0.00%      0     438     0.00%    0.00     438.00     0.00%

6   Baichuan2     371     67      84.70%     380     58      86.76%     419     19      95.66%    390.00	   48.00	   89.04%

7   LLaMa2(7B)      151     287     34.47%     1     437     0.23%      185     253     42.24%     112.33	    325.67	    25.65&

8   LLaMa2(13B)     188     250     42.92%     4     434     0.91%      177     261     40.41%     123.00	   315.00	   28.08%

9   LLaMa2(70B)     344     94      78.54%     29      409     6.62%      338     100     77.17%    237.00	   201.00	   54.11%

10   Spark-3     202     236     46.12%     215     223     49.09%     316     122     72.15%    244.33   	193.67	   55.78%

11   Spark-3.5     297     141     67.81%     378     60      86.30%     418     20      95.43%     364.33   	73.67   	83.18%

2c|Average     263.73     174.27     60.21%     220.18     217.82     50.27%     309.36     128.64     70.63%     264.42   	173.58	   60.37%

The sorted SSR results on 438 test tasks of the three types of prompt design.

Comparisons of SSR results of GPT-3.5.

1c|2*No.   1c|2*Methods     2c|RH-P     2c|LH-P     2c|PH-P     2cAverage
$\Delta\#$     $\Delta\%$        $\Delta\#$     $\Delta\%$       $\Delta\#$     $\Delta\%$     $\Delta\#$     $\Delta\%$
1   GPT-4           235      118.69%      126       42.14%       71        19.45%       144          60.09%

2   GLM-3           141      71.21%       -7        -2.34%       -15       -4.11%       39.67        21.59%

3   GLM-4           180      90.91%       100       33.44%       34        9.32%        104.67       44.56%

4   GLM-4V          -198     -100.00%     -299      -100.00%     -365      -100.00%     -287.33      -100.00%

5   Baichuan2       173      87.37%       81        27.09%       54        14.79%       102.67       43.08%

6   LLaMa2(7B)      -47      -23.74%      -298      -99.67%      -180      -49.32%      -175         -57.58%

7   LLaMa2(13B)     -10      -5.05%       -295      -98.66%      -188      -51.51%      -164.33      -51.74%

8   LLaMa2(70B)     146      73.74%       -270      -90.30%      -27       -7.40%       -50.33       -7.99%

9   Spark-3         4        2.02%        -84       -28.09%      -49       -13.42%      -43          -13.16%

10   Spark-3.5       99       50.00%       79        26.42%       53        14.52%       77           30.31%
2c|Average       72.3     36.52%        -86.7     -29.00%       -61.2     -14.93%      -25.20     2.46%

Comparisons of SSR results of GPT-4.

1c|2*No.   1c|2*Methods     2c|RH-P     2c|LH-P     2c|PH-P     2cAverage
$\Delta\#$     $\Delta\%$        $\Delta\#$     $\Delta\%$       $\Delta\#$     $\Delta\%$     $\Delta\#$     $\Delta\%$
1   GPT-3.5         -235       -54.27%      -126       -29.65%      -71        -16.28%      -144        -33.40%

2   GLM-3           -94        -21.71%      -133       -31.29%      -86        -19.72%      -104.33     -24.24%

3   GLM-4           -55        -12.70%      -26        -6.12%       -37        -8.49%       -39.33      -9.10%

4   GLM-4V          -433       -100.00%     -425       -100.00%     -436       -100.00%     -431.33     -100.00%

5   Baichuan2       -62        -14.32%      -45        -10.59%      -17        -3.90%       -41.33      -9.60%

6   LLaMa2(7B)      -282       -65.13%      -424       -99.76%      -251       -57.57%      -319        -74.15%

7   LLaMa2(13B)     -245       -56.58%      -421       -99.06%      -259       -59.40%      -308.33     -71.68%

8   LLaMa2(70B)     -89        -20.55%      -396       -93.18%      -98        -22.48%      -194.33     -45.40%

9   Spark-3         -231       -53.35%      -210       -49.41%      -120       -27.52%      -187        -43.43%

10   Spark-3.5       -136       -31.41%      -47        -11.06%      -18        -4.13%       -67         -15.53%
2c|Average         -186.2     -43.00%      -225.3     -53.01%      -139.3     -31.95%      -183.6      -42.65%

§.§.§ Answer to RQ1.1

Table <ref> and Figure <ref> present the SSR results of the 438 test tasks for the three types of prompts (RH-P, LH-P, and PH-P).
Based on these results, we have the following observations:

* The average SSR results for RH-P, LH-P, and PH-P are 60.21%, 50.27%, and 70.63%, respectively, indicating that PH-P can guide LLMs to generate better web-form tests than RH-P or LH-P.

* The RH-P SSR ranges from 0.00% to 98.86%, the LH-P SSR ranges from 0.00% to 97.03%, and the PH-P SSR ranges from 0.00% to 99.54%.
For all three types of prompts, GPT-4 always has the highest SSR results, indicating the best performance.
GLM-4V always has an SSR of 0.00%, and may not be suitable for generating appropriate web-form tests.

Apart from GLM-4V, LLaMa2(7B) performed worst with the RH-P and LH-P types, with SSRs of 34.47% and 0.23%, respectively; and
LLaMa2(13B) performed worst with the PH-P type, with an SSR of 40.41%.

* For RH-P (Figure <ref>), the SSR results of six LLMs (GPT-4, GLM-4, Baichuan2, LLaMa2(70B), GLM-3, and Spark-3.5) are greater than the average (60.21%), the other five LLMs (Spark-3, GPT-3.5, LLaMa2(7B), LLaMa2(13B), and GLM-4V) have lower results.

* For LH-P (Figure <ref>), the SSR results of six LLMs (GPT-4, GLM-4, Baichuan2, Spark-3.5, GPT-3.5, and GLM-3) are greater than the average (50.27%), the other five LLMs (Spark-3, LLaMa2(7B), LLaMa2(13B), LLaMa2(70B), and GLM-4V) have lower results.

* For PH-P (Figure <ref>), the SSR results of seven LLMs (GPT-4, Baichuan2, Spark-3.5, GLM-4, GPT-3.5, GLM-3, LLaMa2(70B), and Spark-3) are greater than the average (70.63%), the other three LLMs (LLaMa2(7B), LLaMa2(13B), and GLM-4V) have the lower results.

* The average SSR results for RH-P, LH-P, and PH-P are 60.21%, 50.27%, and 70.63%, respectively.
Only PH-P achieved a higher average SSR than the overall average result (60.37%): RH-P and LH-P were both lower than the overall average.

We next discuss the different performance of these three types of prompts.

* With RH-P, we directly feed the raw pruned HTML of the web forms into the LLMs.
This enables the LLMs to use the contextual information from the HTML to generate the web-form tests.
However, we found that some LLMs cannot generate effective web-form tests based on the HTML context.
For example, hint texts in the raw HTML context are descriptions displayed in the web forms that guide users towards what should be entered.
However, some LLMs (such as LLaMa2(13B), LLaMa2(70B), and Baichuan2) only returned the hint texts (e.g., “Please enter the user name”), without generating any web-form tests.

* With LH-P and PH-P, we used two approaches to proceed with the pruned HTML: with the LLMs, and with the automated-testing tool.
We found that the automated-testing tool could perform better for web-form-test generation.
One of the reasons is that LH-P cannot process the HTML into JSON objects that conform to the expectations:
Some key contextual information used for guiding the web-form-test generation may be missed when getting the JSON objects, resulting in the LLMs not successfully generating the web-form tests.
In contrast, the automated testing tool could process the pruned HTML without missing any key contextual information.
Therefore, compared to the automated testing tool, LLMs appear less suitable for parsing the HTML into JSON objects.

Summary of Answers to RQ1.1:
Among the three types of prompts, PH-P performed better than RH-P and LH-P.
Furthermore, the LLMs were found to be less suitable for parsing the HTML of web forms compared to the automated-testing tools.

§.§.§ Answer to RQ1.2

Table <ref> and Figure <ref> also present the SSR results of the 438 tests from the perspective of the 11 LLMs.
Tables <ref> and <ref> compare the SSR results for GPT-3.5 and GPT-4 with other LLMs:
$\Delta\#$ denotes the difference in the  number of successfully submitted tests for an LLM compared to that of GPT-3.5 or GPT-4; and
$\Delta\%$ denotes the change in the SSR.
Based on the results, we have the following observations:

* The average SSR results of the 11 LLMs is 60.37%, ranging from 0.00% to 98.48%.
Among all LLMs, GPT-4 always has the highest average SSR (98.48%), indicating the best ability to generate appropriate tests for web forms.
GLM-4V always has the lowest results (0%).
The performance of the other LLMs ranked according to SSR is:
GLM-4 (89.50%),
Baichuan2 (89.04%),
Spark-3.5 (83.18%),
GLM-3 (74.66%),
GPT-3.5 (65.60%),
Spark-3 (55.78%),
LLaMa2(70B) (54.11%),
LLaMa2(13B) (28.08%), and
LLaMa2(7B) (25.65%).

* Compared with GPT-3.5, GPT-4, GLM-3, GLM-4, Baichuan2, and Spark-3.5 achieved better performance, with SSR improvement rates of 60.9%, 44.56%, 43.08%, 30.31%, and 21.59%, respectively.
GLM-4V, LLaMa2(70B), LLaMa2(13B), LLaMa2(7B), and Spark-3 may not be suitable for generating suitable web-form tests:
Their SSR performances represented degradation of 100.00%, 57.58%, 51.74%, 7.99%, and 13.16%, respectively.

* Compared with GPT-4, all other LLMS were less effective, with decreases in SSRs of 9.10% to 100.00%.
The GPT-3.5, GLM-3, GLM-4, Baichun2, and Spark-3.5 SSR decreases were less than the average reduction (42.65%); while the reductions for GLM-4V, LLaMa2(70B), LLaMa2(13B), LLaMa2(7B), and Spark-3 were greater than the average.

Summary of Answers to RQ1.2:
Some LLMs (such as GPT-4, GLM-4, and Baichuan2) can generate relatively effective web-form tests.
Among the 11 LLMs, GPT-4 always has the best effectiveness;
and GLM-4V always performs worst (with SSRs of 0.00%).
Compared with GPT-4, the remaining LLMs have difficulty generating appropriate tests for web forms, with SSRs ranging from 9.10% to 74.15%.
Nevertheless, some LLMs (such as GLM-3, GLM-4, Baichuan2, and Spark-3.5) achieve higher SSRs than GPT-3.5, indicating better effectiveness.

[A login web-form]
[A restaurant reservation web-form.]
[An airplane booking web-form.]
Examples of web forms that were not successfully submitted.

LLMs and corresponding generated web-form tests for the example in Figure <ref>.

No.     Methods         Generated web-form tests      Status
1       GPT-3.5         json{“label”: [“Username”, “Password”, “Rememberme”], “inputType”:[“text”, “password”, “checkbox”], “inputId”: [“username”, “password”, “exampleCheck1”], “inputName”: [“emailId”, “password”], “inputValue”: [], “placeholder”: [], “additional”: [“We'll never share your email with anyone else.”], “displayStyle”: [], “elementType”: [], “url”: [], “required”: []}      54 (Reason 1)
2       GPT-4<EMAIL_ADDRESS>#password=StrongPassword123!, #exampleCheck1=true      52
3       GLM-3           #username=JohnDoe, #password=Secret123', #exampleCheck1=rememberMe      54 (Reason 2)
4       GLM-4<EMAIL_ADDRESS>#password=yourStrong!Passw0rd, #exampleCheck1=true     52
5       GLM-4V          json{“selectors”: [“#username”, “#password”, “#exampleCheck1”, “#submit”], “values”:
{“#username”: “Enter your username here”, “#password”: “Enter your password here”, “#exampleCheck1”: “”}}      54 (Reason 1)
6       Baichuan2       #username=Your Email Address, #password=Your Password, #exampleCheck1=Remember Me      54 (Reason 1)
7       LLaMa2(7B)      json[`#username=virtuelleFingerabtSport', `#password=securePassword'], [`#emailId=validEmailAddress'], [`input[name=`rememberMe']']     54 (Reason 1)
8       LLaMa2(13B)     [{“label”: “Username”, “inputType”: “text”,“inputId”:“username”,“inputName”: “emailId”,“placeholder”:“Enter your email address”,“additional”: “We'll never share your email with anyone else.”,“displayStyle”: “block”}, {“label”: “Password”,“inputType”:“password      54 (Reason 1)
9       LLaMa2(70B)     [{“label”: “Username”, “inputType”: “text”, “inputId”: “username”, “inputName”: “emailId”, “inputValue”: “”, “placeholder”: “”, “additional”: {}, “displayStyle”: “block”, “elementType”: “div”, “url”: “”, “required”:true}, {“label”: “Password”, “inputType”: “”      54 (Reason 1)
10      Spark-3         javascript[`input[id=username]=' + ` “Username“ `, `input[id=password]=' + ` “Password” ', `input[id=exampleCheck1]' + `=true']     54 (Reason 2)
11      Spark-3.5       #username=emailId, #password=password, input[name=exampleCheck1]=checkbox     54 (Reason 2)

LLMs and corresponding generated web-form tests for the example in Figure <ref>.

No.     Methods         Generated web-form tests      Status
1       GPT-3.5         #time=datetime-local, #message=text, select#tableType[name=“tableType”], #person=number       54 (Reason 2)
2       GPT-4           #time=2022-12-25T18:30, #message=Looking forward to the meal, select[name=tableType]=VIP, #person=4
3       GLM-3           #time=Time, #message=Message, select[name=tableType]=“NORMAL”: “NORMAL”,“VIP”: “VIP”, #person=Number Person
54 (Reason 2)
4       GLM-4           #time=2023-04-01T18:30:00', #message=Please ensure our table has a great view, #tableType=VIP, input[name=person]=6
5       GLM-4V          json{“selectors”: {“time”:“input[type=`datetime-local']”, “message”: “input[type=`text']”, “tableType”: “select[name=`tableType']”, “person”: “input[type=`number']”}, “values”: {“time”: “”, “message”: “”, “tableType”: “NORMAL”, “person”: “”}}      54 (Reason 1)
6       Baichuan2       #time=Select time, #message=Enter message, #tableType=Choose table type, #person=Enter number of people       54 (Reason 2)
7       LLaMa2(7B)      #time=datetime-local();, #message=text();, .person input[type=number();}=person();, .tableType select[name=tableType();)=tableType();      54 (Reason 2)
8       LLaMa2(13B)     #time=datetime-local, #message=text, #tableType=select, #person=number      54 (Reason 2)
9       LLaMa2(70B)     #time=datetime-local, #message=text, #tableType=select, #person=number       54 (Reason 2)
10      Spark-3         “”, “”, “”     54 (Reason 3)
11      Spark-3.5       #time=2023-09-15T18:30', #message=Hello, I would like to book a table., #tableType=VIP, #person=4     52

LLMs and corresponding generated web-form tests for the example in Figure <ref>.

No.     Methods         Generated web-form tests      Status
1       GPT-3.5         #reservationType=, #booking_Date=2023-12-11, #journeyDate=2023-12-11, #fare=10.0, #noSeats=       52
2       GPT-4           #reservationType=Round Trip, #booking_Date=2023-12-11, #journeyDate=2023-12-11, #fare=10.0, #noSeats=2       52
3       GLM-3           #reservationType=Reservation Type, input[name=scheduleID]=1, #booking_Date=2023-12-11, #journeyDate=2023-12-11, #fare=10.0, input[name=noSeats]      54 (Reason 2)
4       GLM-4           [`input[name=addReservationForm]', `form'], [`input#reservationType', `Economy'], [`input#booking_Date', `2023-12-11'], [`input#journeyDate', `2023-12-11'], [`input#fare', `10.0'], [`input#noSeats', `2']
54 (Reason 1)
5       GLM-4V          json{“selectors”: {“#reservationType”: "value from reservationType”, “#booking_Date”: “value from bookingDate”, “#journeyDate”: “value from journeyDate”, “#fare”: “value from fare”, “#noSeats”: “value from noSeats”}, “values”: {
“reservationType”: “”, “bookingDate”: “2023-12-11”, “journeyDate”: “2023-12-11”, “fare”: “10.0", “noSeats”: “"}}      54 (Reason 1)
6       Baichuan2       #reservationType=Economy, #booking_Date=2023-12-11, #journeyDate=2023-12-11, #fare=10.0, #noSeats=10       52
7       LLaMa2(7B)      #reservationType=.$(`#reservationType').val()., #booking_Date=.$(`#bookingDate').val()., #journeyDate=.$(`#journeyDate').val()., #fare=.$(`#fare').val()., #noSeats=.$(`#noSeats').val().      54 (Reason 2)
8       LLaMa2(13B)     $(`input[name=“reservationType”]').val(“Select a reservation type”); $(`input[name=“bookingDate”]').val(“Enter booking date”); $(`input[name=“journeyDate”]').val(“Enter journey date”); $(`input[name=“fare”]').val(“Enter fare for one passenger”); $(`input[name=“noSeats”]').val(“Enter number of seats to book”);      54 (Reason 2)
9       LLaMa2(70B)     #reservationType=roundTrip, #booking_Date=2023-12-11, #journeyDate=2023-12-11, #fare=10.0, #noSeats=1       52
10      Spark-3         [input[name=reservationType]={reservationType},  input[name=bookingDate]={booking_Date}, input[name=journeyDate]={journeyDate}, input[name=fare]={fare}, input[name=noSeats]={noSeats}]      54 (Reason 2)
11      Spark-3.5       #reservationType=Reservation, #booking_Date=2023-12-11, #journeyDate=2023-12-11, #fare=10.0, #noSeats=1      52

§.§ RQ2: What is the quality of the generated web-form tests?

This section discusses the quality of the web-form tests generated by the different LLMs.
The section includes an analysis of the reasons why some tests cannot be submitted successfully.
We also include an analysis of the test quality, from a tester’s perspective.

The quality scores by 20 testers for different LLMs-generated web-form tests.

§.§.§ Answer to RQ2.1

Figure <ref> presents three web-form instances.
Tables <ref> to <ref> show the web-form test information generated with these three web forms by the 11 LLMs.
A “52” indicates that the context of the web-form tests was consistent with the web-form, which can lead to successful submission;
“54” indicates that the generated web-form tests neither satisfied the context, nor were successfully submitted through the web-form.
We categorized the reasons why the LLM-generated web-form tests could not be submitted successfully as follows:

* Reason 1: Some LLMs (such as Baichuan2, Spark, and GLM-4V) were unable to generate the correct web-form test content based on the provided web-form contextual information.
For example, in Figure (<ref>), when we log in to the web-form, we need to provide three inputs: email, password, and whether or not to remember the password.
Some LLMs (such as GLM-3) generated web-form tests with incorrect formatting, such as an email not confirming to regular email formatting.
This may be because some LLMs may not be able to correctly parse the contextual information.

* Reason 2: Some LLMs (such as GLM-4, GLM-4V, and LLaMa2(7B)) were unable to generate web-form tests in the specific format restricted in the designed prompts.
For example, in Figure (<ref>), we constructed restrictive prompts for generating web-form tests (Section <ref>).
However, these LLMs did not return the correct information, which prevented the correct parsing of the information, especially with LH-P.
When using LH-P, the LLMs first parsed the HTML of a given web-form into a list of JSON structures (Algorithm <ref>).
If the LLMs could not return the correct parsed JSON structures, the subsequent testing process would fail.

* Reason 3: Connection problems between the testing environment and the LLM API, or issues with the reasoning process of the LLMs, may also have caused failure or interruption of the web-form-test generation process.
In Figure (<ref>), for example, Spark-3 encountered a request timeout when generating the restaurant reservation information based on the content of the web-form.
Therefore, it was unable to parse and submit the web-form-test information.

Summary of Answers to RQ2.1:
(1) Different LLMs are constrained by training datasets, parameter sizes, and other factors, resulting in significant differences in their effectiveness when generating web-form tests.
(2) Generating web-form tests that are consistent with the web-form context will greatly increase the SSRs.

§.§.§ Answer to RQ2.2

Figure <ref> shows the quality scores given by 20 testers for the different LLMs-generated web-form tests, using the three types of prompts
Table <ref> presents the average quality scores of the 20 testers.
Furthermore, the Kendall's W value for the 20 testers was 0.94 (close to 1.0), indicating a strong agreement among evaluators.
Based on these results, we have the following observations:

From the perspective of the three types of prompts (RH-P, LH-P, and PH-P):

* The average scores for RH-P, LH-P, and PH-P were 2.40, 2.26, and 2.76, respectively.
PH-P achieved a better score than the overall average (2.47), while LH-P and PH-P scored lower than the overall average.

* For the RH-P prompt, only GPT-4 and GLM-4 achieved average scores greater than 3.00 (3.62 and 3.21, respectively), indicating that only these two LLMs achieved a “Neutral” performance from the perspective of the testers.
Testers rejected (sometimes strongly) the quality of the web-form tests generated by other LLMs using RH-P.

* For the LH-P prompt, GPT-4, GLM-4, Baichuan2, and Spark-3.5 achieved a “Neutral” performance according to testers.
They rejected (sometimes strongly) the quality of the web-form tests generated by other LLMs using RH-P.

* For the PH-P prompt, GPT-4, GLM-3, GLM-4, Baichuan2, and Spark-3.5 achieved a “Neutral” performance, according to the testers.

The average score across the 11 LLMs was 2.47.
Five of them (GPT-4, GLM-3, GLM-4, Baichuan2, and Spark-3.5) achieved scores better than the average.
GPT-4 always scored better than other LLMs, and GLM-4V always scored the worst.

Summary of Answers to RQ2.2:
The main findings of the quality evaluation from the 20 testers are consistent with the findings in the effectiveness evaluation (Section <ref>).
GPT-4 received the highest endorsements of the testers.
Other LLMs received poorer evaluations.
In general, there is still room for improving the quality of the LLM-generated web-form tests to meet the expectations of the testers.

The average quality scores by 20 testers.

No.   Methods     RH-P     LH-P     PH-P     Average
1   GPT-3.5     1.82     2.63     2.63     2.36

2   GPT-4     3.62     3.53     3.71     3.62

3   GLM-3     2.76     2.71     3.01     2.83

4   GLM-4     3.21     3.25     3.45     3.31

5   GLM-4V     1.00     1.00     1.00     1.00

6   Baichuan2     2.84     3.30     3.38     3.17

7   LLaMa2(7B)     1.70     1.00     2.16     1.62

8   LLaMa2(13B)     1.98     1.03     1.95     1.65

9   LLaMa2(70B)     2.63     1.11     2.81     2.18

10   Spark-3     2.26     2.20     2.81     2.42

11   Spark-3.5     2.60     3.07     3.43     3.03
2c|Average     2.40     2.26     2.76     2.47

§.§ RQ3: What insights/advice can be offered to testers using LLMs for web-form testing?

This section provides some insights and advice for using LLMs to support the web-form-test generation from two perspectives: prompt design and LLM selection.

§.§.§ Answer to RQ3.1

Based on the experimental results for the three types of prompts (RH-P, LH-P, and PH-P), we found that the following strategies should be followed:

* Insight 1: Ensure accurate extraction of web-form context for prompt guidance.
We can extract the contextual information from the HTML of the web-form and use it as part of the prompt to guide LLMs to generate web-form tests.
However, we need to accurately parse the web-form and accurately extract the contextual information.
For example, we found that using PH-P to construct prompts was 20.36% more successful than using LH-P.

* Insight 2: Simplify web-form HTML for simpler, clearer prompts.
We found that the HTML structure of web forms can be complex.
It is necessary to simplify this when constructing prompts, and to avoid directly using raw HTML to build prompts (which may make the content of the prompts too complex).
For example, we found that using PH-P to construct prompts was 9.94% more successful than using RH-P.

* Insight 3: Simple and clear task requirements should be set for LLMs to help them achieve the expected results more effectively.
The PH-P method extracts web-form contextual information using pruned web-form HTML content, while

RH-P directly uses the pruning HTML (more complex) to construct contextual information:
PH-P provides LLMs with simpler contextual information.
We found that simpler and clearer prompts achieve more effective web-form-test generation.

Summary of Answers to RQ3.1:
We offer the following advice for designing and constructing prompts when testing web forms:
(1) Simplify the HTML content in the web forms;
(2) Accurately extract the contextual information from the web forms; and
(3) Set simple and clear task requirements (simplify the prompt structures).

§.§.§ Answer to RQ3.2

According to our study<ref>, over 50% (11 out of 20) of the testers have started using LLMs to guide quality assurance work, which indicates that LLMs are playing an increasingly important role in testing.
80% (16 out of 20) of the testers were concerned about privacy and security issues when using LLMs for the web-form quality assurance.
Based on the importance of LLMs, the security concerns of testers, and the main findings of our research, we offer the following guidance for selecting LLMs for testing:

* Insight 4: To generate effective web-form tests, GPT-4 is highly recommended.
According to our experimental results (Sections <ref>), GPT-4 can generate the most effective web-form tests.

* Insight 5: If facing testing constraints, the alternative LLMs (not GPT series) should be selected.
Actual web-form testing tasks can involve combining real user data into prompt content.
This helps the LLMs better understand the contextual information of the web forms.
However, this may also lead to private data leakage:
GPT LLMs should, therefore, not be directly used <cit.>.
Alternative LLMs, such as the LLaMa2 series LLMs, should be considered.
In this study, we compared the effectiveness of GPT-3.5 and GPT-4 with other LLMs, which should help testers to more conveniently choose an appropriate LLM based on the actual situation.
For example, the GLM-4 and Baichuan2 effectiveness are only about 10% less than GPT-4.

Summary of Answers to RQ3.2:
We offer the following advice for choosing LLMs:
(1) If data privacy is not an issue when testing, GPT-4 is a good choice for ensuring the effectiveness of the generated web-form tests.
(2) If testers are concerned about data privacy and security, then alternative, open-source LLMs, such as the LLaMa2 series, should be considered.

§.§ Threats to Validity

This section discusses potential threats to the validity of our study.

* The first threat is related to the representativeness of our experimental subject selection.
To mitigate this threat, when selecting the AUTs on Github, we filtered through multiple keywords, such as  “Java Web”, “Jobs”, and “Books”.
This enabled the selection of different types of Java web applications.

* The second threat involves our communication with the LLMs.
As we know, if LLMs are continuously interacted by using a fixed communication role, they may suffer from some biases to generate web-form tests <cit.>.
To avoid such a potential threat, in this study we maintain random access to APIs (i.e., using a custom default role defined by the LLM), and use task guidance based on direct prompts to communicate with the LLMs.

* The third threat involves our research on web-form contextual information.
We designed three types of prompts to better extract contextual information from the web forms.
This helped us to build multiple prompts to guide the LLMs' generation of web-form tests, and to evaluate their effectiveness.

§ RELATED WORK

This section presents some related work, including web-form testing, string data generation, and LLMs for software testing.

§.§ Web-Form Testing

Web-form testing is critical to software quality assurance, impacting both the user experience and the application stability.
As web applications become increasingly complex, researchers have worked to enhance automation, efficiency, and accuracy in web-form testing.

Rothermel et al. <cit.> proposed an automated framework for identifying and validating form elements and their interaction behavior.
Using the consistency between visual elements and back-end logic, their method significantly improved testing coverage and efficiency, setting new standards for form-based visual-program testing.

Ricca et al. <cit.> introduced a UML model to assess static site structures and guide white-box testing.
Applied to real-world scenarios, their approach improved verification and validation, using automatic test-case generation to ensure comprehensive testing and to simplify regression checks.
They emphasized the importance of thorough testing to ensure the quality and performance of web applications, offering a detailed perspective on web-form testing.
Furche et al. <cit.> used ontology tools to automate the understanding of form structure and content:
They optimized the integration and retrieval process of form data, and demonstrated the efficiency and potential advantages of semantic technology for handling complex web forms.
Santiago et al. <cit.> integrated machine learning and constraint-solving techniques to predict the effective input of form fields.
They were able to generate test cases that comply with logical constraints, demonstrating the practical application of machine learning technology in enhancing test automation and improving accuracy.
Cruz-Benito et al. <cit.> explored the adaptability of web forms based on user-feature detection.
They studied the impact of user behavior and preferences on form design through A/B testing and machine learning techniques, and proposed strategies to improve user satisfaction and interaction efficiency through customized experiences.
Lukanov et al. <cit.> used the functional Near Infrared Spectroscopy (fNIRS) measure to study the impact of web-form layout on users' psychological burden.
This provided the scientific basis for understanding how different design schemes affect users' cognitive burden, emphasizing the importance of optimizing the user experience in form design.
Alian et al. <cit.> improved the accuracy and efficiency of automatic filling and validation processes by conducting an in-depth analysis of the semantics of form elements.
They highlighted the critical role of semantic analysis in improving the quality of web-form testing.

§.§ LLMs for Software Testing

With the growing application of LLMs in software engineering, their innovative applications and challenges in the software testing process have become an important research topic.

Wang et al. <cit.> summarized how LLMs are transforming software testing, providing insights into their future roles and challenges based on current practices and research.
Yu et al. <cit.> explored LLMs' roles in improving the generation and migration of automated test scripts, underlining their flexibility and efficiency in complex scenarios, and the potential for enhanced script maintainability and adaptability.
Zhang et al. <cit.> evaluated the effectiveness of LLM-generated security tests.
Through experimental comparisons, they demonstrated LLMs' ability to identify potential security vulnerabilities and generate corresponding test cases, highlighting their potential for automating security testing.
Feldt et al. <cit.> investigated the use of conversational LLMs to develop agents that autonomously conduct testing tasks, introducing innovative approaches to software testing through natural language understanding.

Alshraideh et al. <cit.> proposed a method for generating test data through search-based techniques that incorporate program-specific search operators, showing how LLMs can optimize the search process to improve the efficiency and quality of test-data generation.
Gu et al. <cit.> introduced a method based on LLMs for generating testing code for the Go language compiler.
They used LLMs' ability to understand language structure to generate high-quality code, thus improving the automation level of compiler testing.
Kang et al. <cit.> conducted an experimental study on LLMs' capability to reproduce generic software defects, revealing the ability of LLMs to learn from a few examples, and offering new approaches for software defect diagnosis and repair.
Schäfer et al. <cit.> conducted an empirical evaluation of the effectiveness of using LLMs for automated unit-test generation, highlighting the potential for LLMs to generate high-quality test cases while also noting their limitations in dealing with specific testing challenges.
Fatima et al. <cit.> proposed a black-box approach based on LLMs for predicting flaky tests.
By analyzing the historical data of test executions, this predictor can effectively identify potentially flaky tests, helping to improve the stability and reliability of testing.

To the best of our knowledge, there has been no empirical research assessing the effectiveness of LLMs in generating web-form tests.
Additionally, there is limited understanding of how testers can make optimal use of LLMs for this purpose.
This article aims to address these gaps in the literature.

§ CONCLUSIONS AND FUTURE WORK

This paper has reported on an empirical study to investigate the effectiveness of 11 state-of-the-art LLMs in web-form-test generation.
We evaluated these LLMs using 146 web forms from 30 open-source Java web applications.
Based on our experimental results, we have the following conclusions:
(1) Some LLMs (such as GPT-4, GLM-4, and Baichuan2) can generate relatively efficient and high-quality tests for the web-form.
However, under the same conditions, LLMs such as GLM-4V, LLaMa2(7B), and LLaMa2(13B) did not perform well on the same testing task, indicating that LLMs can still be optimized and improved for automated web-form-test generation.
(2) Some LLMs (such as GLM-3, GLM-4, Baichuan2, and Spark-3.5) may be more suitable for generating appropriate tests for web forms than GPT-3.5, delivering a higher SSR than GPT-3.5.
(3) A comparison of the experimental results for the three different prompt methods (RH-P, LH-P, and PH-P) revealed that clear and concise web-form contextual content could better guide theLLMs to generate appropriate content.
If the contextual information of key information in the HTML elements is missing, it may reduce the effectiveness performance around from 10% to 20%.

(4) Regardless of the prompt, GLM-4V performs poorly.
An analysis of this found that GLM-4V may not be suitable for generating web-form tests,  which indicates that the selection of models should depend on their specific areas of expertise.

Our future research will include the following two directions:

* Based on the experimental results, the average SSR of the selected LLMs is 60.37%, which means that there is room for improvement.
Some approaches can be adopted to improve the LLM effectiveness for generating web-form tests, such as optimizing the design and construction of prompts, and fine-tuning or retaining the LLMs.

* Based on the workflow of our empirical study with various LLMs, an automated web-form-test generation tool can be designed and developed.
Furthermore, the automated generation tool can provide testers with intelligently recommended options (guided by different LLMs) for the testing process.

§ ACKNOWLEDGMENTS

The authors would like to thank the testers from the companies and research institutions who participated in our study questionnaires.
We would also like to thank Yinming Huang from Macau University of Science and Technology for their valuable help in the experiments of this study.

§ ACKNOWLEDGMENT

The authors would like to thank the testers from the companies and research institutions who participated in our study.
We would also like to thank Yinming Huang from the Macau University of Science and Technology for his valuable help in the experiments.

Appendix to “Leveraging Large Language Models for Automated Web-Form-Test Generation: An Empirical Study"

Tao Li, Chenhui Cui, Lei Ma, Dave Towey, Yujie Xie, Rubing Huang

This appendix contains detailed information for the paper “Leveraging Large Language Models for Automated Web-Form-Test Generation: An Empirical Study”, providing a deeper understanding.

§ DETAILS OF THE SUBJECT LLMS

GPT-3.5 <cit.>:
This is an advanced natural language processing model developed by OpenAI, officially launched in 2023.
Utilizing the Transformer architecture, a type of neural network that employs the attention mechanism, this model excels in text understanding, generation, and translation.

It can be effectively used for a wide range of tasks, such as developing chatbots, creating content, and solving complex analytical problems.

GPT-4 <cit.>:
This is the fourth generation of Generative Pretrained Transformers, an advanced natural language processing model developed by OpenAI.
Launched in 2023, it features trillions of parameters, an improved neural network architecture, and utilizes large and diverse data sets.
The model performs well in understanding and generating natural language text.
It can accurately comprehend complex queries and instructions, and demonstrates significant creativity and multi-language processing capabilities.

GLM <cit.>:
This is a large language model launched by Zhipu AI Company, featuring several different parameter sizes and performance characteristics.
This bilingual model, which adopts the GLM model structure proposed by the KEG Laboratory of Tsinghua University, is capable of performing natural language understanding and generation tasks.
Among its variants, GLM-4 is a large-scale basic model with 130 billion parameters.

Baichuan2 <cit.>:
Released in 2023, this series of large-scale multilingual language models includes versions with 7 billion and 13 billion parameters.
Trained from scratch, based on 2.6 trillion tokens, it adopts advanced network structures such as rotational position embedding and the ALiBi activation function.
The models use comprehensive and high-quality data sets in the pre-training stage, including Internet web pages, books, research papers, and code libraries.
With powerful language generation and understanding capabilities, these models perform well in multiple natural language tasks, providing valuable research and application tools for academia and industry.

LLaMa2 <cit.>:
Released by Meta AI in 2023, it is an LLM comprising 70 billion parameters, based on the Transformer network architecture.
This model demonstrates strong language understanding and generation capabilities while maintaining relatively low pre-training costs.

A model structure of LLaMa2(7B).

Spark <cit.>:
Launched in 2023, this large-scale deep neural network model developed by iFlytek is based on the transformer architecture and contains billions of parameters.
It is trained using massive texts, codes, and knowledge, enabling it to handle practical scenarios such as knowledge-based question and answer, code programming, and mathematical calculations.
Its strengths include strong expression and generalization abilities, as well as the capability to understand user needs and perform tasks in a natural conversational manner.