# AXNav: Replaying Accessibility Tests from Natural Language

Maryam Taeb<EMAIL_ADDRESS>Florida State UniversityUSA ,  Amanda Swearngin
<EMAIL_ADDRESS>AppleUSA ,  Eldon Schoop<EMAIL_ADDRESS>AppleUSA ,
Ruijia Cheng<EMAIL_ADDRESS>AppleUSA ,  Yue Jiang<EMAIL_ADDRESS>Aalto UniversityFinland and  Jeffrey Nichols<EMAIL_ADDRESS>AppleUSA

(20 February 2007; 12 March 2009; 5 June 2009)

###### Abstract.

Developers and quality assurance testers often rely on manual testing to test
accessibility features throughout the product lifecycle. Unfortunately, manual
testing can be tedious, often has an overwhelming scope, and can be difficult
to schedule amongst other development milestones. Recently, Large Language
Models (LLMs) have been used for a variety of tasks including automation of
UIs, however to our knowledge no one has yet explored their use in controlling
assistive technologies for the purposes of supporting accessibility testing.
In this paper, we explore the requirements of a natural language based
accessibility testing workflow, starting with a formative study. From this we
build a system that takes as input a manual accessibility test (e.g., “Search
for a show in VoiceOver”) and uses an LLM combined with pixel-based UI
Understanding models to execute the test and produce a chaptered, navigable
video. In each video, to help QA testers we apply heuristics to detect and
flag accessibility issues (e.g., Text size not increasing with Large Text
enabled, VoiceOver navigation loops). We evaluate this system through a 10
participant user study with accessibility QA professionals who indicated that
the tool would be very useful in their current work and performed tests
similarly to how they would manually test the features. The study also reveals
insights for future work on using LLMs for accessibility testing.

accessibility, ui testing, large language models

††ccs: Human-centered computing Accessibility systems and tools††ccs: Human-
centered computing Interactive systems and tools††ccs: Computing methodologies
Multi-agent planning

Figure 1. AXNav interprets accessibility test instructions specified in
natural language, executes them on a remote cloud device using an LLM-based
multiagent planner, and produces a chaptered video of the test annotated with
heuristics that highlight potential accessibility issues. To execute a test,
AXNav provisions a cloud iOS device; stages the device by installing the
target app to be tested and enabling a specified assistive feature;
synthesizes a tentative step-by-step plan to execute the test from the test
instructions; executes each step of the plan, updating the plan as needed; and
annotates a screen recording of the test with chapter markers and visual
elements that point out potential accessibility issues.

The figure shows a flow diagram where on the left, a Cloud Device and Test
instructions are shown. The middle shows a diagram which visualizes the
workflow described in the caption. On the far right, the diagram shows a
mobile device with a Chapters menu open on the right showing the steps marked
by the system.

##  1\. Introduction

Many mobile apps still have incomplete support for accessibility features (Fok
et al., 2022; Ross et al., 2017; Yan and Ramachandran, 2019; Alshayban et al.,
2020; Zhang et al., 2021). Developers of these apps may not implement or test
accessibility support due to a lack of awareness (Alshayban et al., 2020),
organizational support (Ross et al., 2017; Bi et al., 2022), or experience in
accessibility testing (Bi et al., 2022). For apps that do support
accessibility features, developers often work in tandem with experienced
accessibility quality assurance (QA) testers (Bi et al., 2022). Employees in
both roles may use automated tools like accessibility scanners (acc, 2022,
2023b), linters (and, 2023), and test automation (esp, 2023; xct, 2023) to
execute UI test scenarios. However, despite many available tools, the majority
of testing for accessibility is still done manually. This may be sometimes due
to to limitations of the tools themselves—UI tests can be brittle (Pan et al.,
2020; Li et al., 2017) or non-existent (Lin et al., 2020; Kochhar et al.,
2015; Cruz et al., 2019), and scanners can provide false positives (Vigo et
al., 2013). In addition, manual testing is often used for accessibility
services (e.g., VoiceOver (voi, 2023), TalkBack (tal, 2023)) because it can
reveal issues that cannot be detected by automated techniques alone (Mankoff
et al., 2005).

While there are benefits to manual testing, manual testing all possible
accessibility scenarios and features comes at a huge cost to organizations,
and is hard to scale. In a formative study with six accessibility QA testers,
we found they often had difficulties keeping up with the scope of apps and
features they are assigned to test. This causes testers to limit the scope of
their tests, potentially letting bugs slip through. In addition, test passes,
where each team completes a suite of manual tests, were often difficult to
time against release deadlines, and manual test instructions were often
outdated. While research has produced tools aimed to address some of these
challenges through automation (Salehnamadi et al., 2021, 2023), there are
still manual costs associated with writing and recording tests to be replayed.
Recorded tests often need to be updated when the UI or navigation flow
changes, similar to UI automation tests which must specify each step in the
navigation flow in code (Li et al., 2017; Pan et al., 2020).

To address some of these challenges and enable a bridge between a fully
automated testing system and a manual test, we explore the use of natural
language instructions to specify accessibility testing steps to a system.
Manual test instructions are an extremely common artifact within organizations
which often have large databases of manual steps for QA testers and within bug
databases (i.e., repro steps). Our hypothesis is that by giving these
instructions as input to a system to which can flexibly interpret and replay
the steps at both high and low levels of abstraction from existing test cases,
we can provide a valuable bridge and testing tool that preserves some benefits
of manual testing while reducing the cost to manually perform the steps and
keep them updated. Our system, AXNav, translates these instructions to a set
of concrete actions that can be taken in a user interface, which are adapted
automatically as the interface evolves.

Motivated by the abilities of Large Language Models (LLMs) to adapt
instructions to a wide variety of scenarios for generation and tool use (Brown
et al., 2020; Chowdhery et al., 2022; Touvron et al., 2023; Patil et al.,
2023; Schick et al., 2023), we explore their use within accessibility testing
for mobile apps. Prior works have shown how such models can be used to
recreate bug reports (Feng and Chen, 2023), test GUIs (Liu et al., 2023), and
automate tasks for web interfaces (Shaw et al., 2023). However, none of these
works have yet explored the use of LLMs for accessibility testing, or
controlling accessibility services (voi, 2023) and settings (but, 2023).

The contributions of this work are:

  * •

A formative study with 6 professional QA and accessibility testers revealing
motivation and design considerations for a system to support accessibility
testing through natural language instruction based manual tests.

  * •

A novel system, AXNav, that converts manual accessibility test instructions
into replayable, navigable videos by using a large language model and a pixel-
based UI element detection model. The system flags potential issues (e.g.,
non-increasing text, loops) with multiple types of accessibility features
(e.g., Dynamic Text, VoiceOver) to help testers pinpoint issues and replays
tasks through accessibility services (i.e., VoiceOver) to enable testers to
visualize and hear the task as a user of the accessibility service might
perform it.

  * •

A user study with 10 professional QA and accessibility testers revealing key
insights into how accessibility testers can use natural language based
automation within their manual testing workflow.

##  2\. Related Work

Our system is most closely related to work that leverages natural language as
an input to a UI for automation, which is useful beyond accessibility use
cases. In this work, we specifically target UI navigation from natural
language for accessibility testing, thus we also review accessibility testing
tools and approaches our work builds upon.

###  2.1. Large Language Models and UI interaction

A key contribution of AXNav is its LLM-based multi-agent planner that can
navigate the UI of an application for executing a specific task or arriving at
a particular view. Our multi-agent system architecture is loosely based on
ResponsibleTA, which presents a framework for facilitating collaboration
between LLM agents for web UI navigation tasks (Zhang et al., 2023). Since
AXNav is designed for testing rather than end-user automation, it removes some
components (e.g., a system to mask user-specific information), and combines
other modules (e.g., both evaluation and completeness verification are
combined in AXNav, and AXNav determines action feasibility in the same step).
These changes significantly reduce the number of LLM turns taken, which lowers
cost and reduces latency.

Other UI navigation works have recently emerged which navigate web or mobile
apps. Wang, et. al., describe prompting techniques to adapt LLMs for use with
mobile UIs, and evaluates an LLM-based agent on single-step UI tasks (i.e.,
predict the UI element used to perform an action on a given screen) (Wang et
al., 2023). AXNav’s UI navigation system builds upon this work by supporting
more complex, multi-step tasks. Other works map from detailed, multi-step
instructions to actions in mobile apps (Feng and Chen, 2023; Li et al., 2020;
Venkatesh et al., 2023). AutoDroid injects known interaction traces from
random app crawls into an LLM prompt to help execute actions with an LLM agent
(Wen et al., 2023). AXNav can interpret a wide variety of instruction types,
from highly specific step-by-step instructions to unconstrained goals within
an app (“add an item to the cart”), without relying on prior app knowledge.
Furthermore, AXNav is able modify its plan when the UI changes, if it
encounters errors, or if the test instructions are incorrect.

The emergence of LLM-based UI navigation systems has motivated the need for
more interaction datasets. Android in the Wild presents a large dataset of
human demonstrations of tasks on mobile apps for evaluating LLM-based agents
(Rawles et al., 2023). Other datasets, such as PixelHelp (Li et al., 2020) and
Motif (Burns et al., 2022) also collect mobile app instructions and steps.
Unlike prior art, AXNav is designed to work on iOS apps, which can have
different navigation flows and complexities than corresponding Android apps.

Most importantly, none of the above works have been used to interact with
accessibility features or support accessibility testing workflows. This is the
core focus of AXNav’s contribution.

###  2.2. Accessibility Testing Tools

Despite the availability of accessibility guidelines and checklists (acc,
2023a, c; wca, 2023), linters and scanners (acc, 2023b, 2022; and, 2023), and
platforms for test automation (xct, 2023; esp, 2023), developers and QA
testers still often prefer to test their apps manually (Linares-Vásquez et
al., 2017; Lin et al., 2020). Testing manually by using accessibility services
can reveal issues that cannot be revealed by scanners alone (Mankoff et al.,
2005). However, manual testing comes at a huge cost and thus QA and
development have developed test suites for automating UI tests (Kochhar et
al., 2015).

There are also many research, industry, and open source tools to automatically
check accessibility properties of apps (Silva et al., 2018) including
development-time (and, 2023) approaches which use static analysis to examine
code for potential issues. Run-time tools (acc, 2023b; esp, 2023; rob, 2021;
acc, 2022) examine a running app to detect accessibility issues, which enables
them to detect issues beyond static analysis, however, they still must be
activated on each screen of the app to be tested.

Another approach might be to automatically crawl the app to detect issues
(Eler et al., 2018; Chen et al., 2021; Alshayban et al., 2020; Salehnamadi et
al., 2022); however, such tools currently adopt random exploration and thus
may not explore or operate the UI as an end-user might to perform specific
tasks. These crawlers also do not operate through accessibility services which
leaves them unable to evaluate whether navigation paths through the app are
fully accessible.

Latte (Salehnamadi et al., 2021) starts to bridge this gap by converting GUI
tests for navigation flows into accessibility tests that operate using an
accessibility service; however, the majority of apps still lack GUI tests (Lin
et al., 2020) and often require updating the code to new navigation flows when
a UI changes (Pan et al., 2020). Removing the requirements of GUI tests being
available, A11yPuppetry (Salehnamadi et al., 2023) lets developers record UI
flows through their app and replay them using accessibility services (i.e.,
TalkBack (tal, 2023)). This idea has also been explored in prior work for web
applications (Bigham et al., 2010). However, a key challenge with record and
replay approaches is that they can also be brittle and difficult to maintain
as the UI evolves (Pan et al., 2020; Li et al., 2017). Our system instead
takes as input an accessibility test in the form of natural language
instructions of the app, task, and feature to be tested, and automatically
replays the steps on a device either using the accessibility service under
test or in the case of visual features (.e.g, Dynamic Type) toggling on and
off the features to replay the device under the test condition. While our
system does not conduct a full scan for accessibility issues during test
replay, we flag a subset of issues using heuristics. Our system architecture
could also be extended to run accessibility audits during during each step of
the replay, similar to accessibility app crawlers (Salehnamadi et al., 2022;
Eler et al., 2018); however, in this work we focus on navigation and replay
through accessibility services and not on holistic reporting of accessibility
issues.

##  3\. Formative Interviews

To better understand the challenges and benefits of manual accessibility
testing and elicit requirements for AXNav through formative interviews, we
recruited six accessibility QA professionals and testing experts. We recruited
the participants took place through snowball recruiting and our own networks
in a large technology company. Participants spanned four product oriented and
internal services teams across four product organizations. Each had experience
testing iOS mobile apps, and had a minimum of three years of accessibility and
QA testing experience in our company. We conducted 30 minute remote interviews
with each participant.

The participants took part in accessibility testing at various times
throughout the product lifecycle, including annually, as new features were
added, or on regular release cycles of app interfaces. The participants daily
work consists of manually performing tests for accessibility features (e.g.,
VoiceOver, Dynamic Type). To conduct tests for purely visual accessibility
features, they typically toggle on the feature under test and validate the
app’s UI looks correct or behaves correctly based on the setting. For
accessibility services tests (e.g., VoiceOver) they typically enable the
feature, and then either navigate the app to perform a task using the feature
or navigate to a specific screen to validate the navigation order or another
behavior of the feature.

During each session, we asked participants about their experience with the
challenges and benefits of manual accessibility testing, and their process for
writing down and performing manual testing instructions. Based on our
hypothesis that natural language would be a valuable input method to instruct
a testing system, we also elicited desired features for a hypothetical system
by showing the participants a video mockup replaying a manual test from sample
input instructions (“Search for a show using VoiceOver”).

The instructions were for a VoiceOver manual test for an entertainment app
that enables users to view TV shows and movies, and the test instruction was
”Search for a show”. We asked the participants to think aloud as the video
played and imagine any features they would like the system to have.

###  3.1. Challenges & Benefits of Manual Testing

Participants pointed out that there are both benefits and challenges in manual
testing for accessibility. A key benefit noted by participants is that manual
testing enables them to experience the feature as an end user might. One
participant, being a VoiceOver user, mentioned this enables them to more
realistically test the feature as it is meant to be used – “the advantages are
that we can test literally, from the user perspective myself, and a number of
my teammates are, uh, users of the features, because of various accessibility
needs that we have. Um, so we are the foremost experts in the functionality of
those particular tests and what the expected results would be.”

However, participants from three teams brought up challenges, including an
overwhelming scope of features and scenarios, leaving them to target only a
few key features and tasks for testing passes. One participant stated “Um,
it’s not like necessarily difficult. It is just like, repetitive and kind of
boring and, um. And the scope is so big a lot of times like, if you’re looking
at the ¡App Name Anonymized¿ app. There’s so many pages and so many views and
so many buttons and different types of elements and everything that is
overwhelming and you feel you’re going to miss something”

Participants in all four teams also do write down or have existing suites of
manual testing instructions. Participants from multiple teams described that
it is easy for those tests to become outdated when apps are updated, making it
difficult for less experienced QA testers to interpret and follow them.
Finding the right time for accessibility testing was also mentioned by four
participants, as they worked with apps that are frequently updated across
various product milestones. Two participants also mentioned using automated
testing in their work which they described as easily breaking and not covering
all possible scenarios.

Figure 2. Three sample test cases for a video streaming media app: VoiceOver,
Dynamic Type, and Button Shapes

The figure shows three test cases from our dataset.

###  3.2. Granularity and Availability of Manual Accessibility Test
Instructions

Manual testing instructions are an extremely common artifact as noted by prior
work and through our interviews within our organization. One team we
interviewed (two participants) noted they do not write down specific
instructions for accessibility scenarios, but that team also owns a large
database of manual instructions for UI tests. For an additional two
participants on a different team, their accessibility testing instructions
primarily consisted of a large regression test suite across ten apps with 300
individual test cases they perform annually. Among these tests, some had
concrete low-level steps, but many were also abstract, high-level, and assume
the QA tester has a high level of expertise on both the app and the
accessibility feature to be tested. Figure 2 contains three example test cases
for a video streaming app for the accessibility features Voice Over, Dynamic
Type, and Button Shapes. Each test case typically has a title containing the
platform, feature, and app to be tested, but only some test cases have step by
step instructions, and only some test cases have an “Expected Result”
specified.

####  3.2.1. F0: Natural Language Interpretation and Replay

Given that there is benefit in manual testing to experience the feature as an
end user (Mankoff et al., 2005), giving all accessibility testing up to full
automation and simple pass or fail metrics would not be desirable to our
participants who would like to observe the behavior of the interactions
themselves and have limited time to spend writing and updating automated
tests. However, given they already have large databases of manual tests
available, and limited time and resources to perform them as much as they
would like, we aimed to create a system that can be a bridge between a fully
automated test and a manual test that can interpret and replay these natural
language instructions written at both a high and low level of abstraction.

###  3.3. Features in a Natural Language-Based Accessibility Testing Tool

Our interviews revealed key features to include in a natural language based
accessibility testing system through think-aloud feedback while watching our
hypothetical system-generated video. We probed participants for the specific
features they desired by asking them the following questions: “If you could
speak / write down a high level description of an accessibility feature you
want to test and a tool could translate that into a set of instructions and
replay them on a device to demonstrate the test, what would be your reaction
to this?”; and “What features should this system support?”. Here we summarize
the key features revealed by these interviews that we incorporated into the
design of AXNav.

####  3.3.1. F1: Flag Potential Issues with Annotations

Multiple (n=4) participants mentioned that they would like the system to
detect potential issues and include failure reports. When asked what specific
issues to flag would be most helpful, participants mentioned both visual
issues like dynamic type resizing, and accessibility feature navigation issues
(e.g., wrong navigation order, elements missing a label or not available for
navigation). Participants noted that if a system could direct them to target
their testing towards any potential issues, that would save time in bug filing
– “If it could detect, you know, the issue, and, you know, write it down or
like, … , that would be helpful so that I can write bugs or maybe bug can be
automated”. Based on this feedback, we were motivated to develop custom
heuristics in our system that flag three potential accessibility issues

The QA testers also requested the system save screenshots and generate failure
reports to facilitate easier reporting in a bug tracking tool – “if your
product, like, did that, I think that would be like, a huge time saver because
most of my time is taking screenshots and clipping”. AXNav saves screenshots
from each frame, and runs heuristics on stored screenshots during
postprocessing to flag a few common navigational and visual issues.

####  3.3.2. F2: Quickly Navigate and Visualize Steps

While watching the video demonstration of the test, multiple participants
requested to go back and review various steps in the replay. While reviewing,
some participants found it difficult to see what action the system was taking
on each step. To make it easier to review actions and issues flagged by our
custom heuristics (F1), we currently generate a video as the primary output
artifact of the system along with chapter-based navigation to facilitate quick
and easy jumping through various steps of the test, by marking system actions
and flagged heuristics with a chapter label. Our system also annotates each UI
element the system interacts with through the replay with a pink ’+’ symbol
(Figure 1). During the interviews, two participants also requested the system
conduct the replay on a live local device so that they could take control
during various parts of the test; due to constraints with our system
architecture, we will explore the feasibility of this idea in future work.

####  3.3.3. F3: Realistic VoiceOver Navigation and Captioning

The last area of feedback revolved around VoiceOver navigation. However, in
the video recording of the system performing the test, we activated the UI
elements for each step directly like a sighted user might, rather than swiping
through elements on a screen to find UI elements as a non-sighted user might.
Several participants noticed this, and noted that the system should _replay
the test to be as similar as possible to how a user of the accessibility
feature performs the task_. Additionally, our video also included the
VoiceOver captions panel for this task, which three participants mentioned was
an important feature to include in our final system.

####  3.3.4. F4: Perform Tests With and Without Accessibility Features

Many manual tests that our participants shared with us commonly instructed
testers to perform the test with and without a specific accessibility feature
toggled on. QA testers mentioned that this is necessary to verify that the
system returns to the original state and behaves as expected in the default
state of the app after turning off the feature. Thus, AXNav repeats the
navigation steps twice for most tests, first replaying the test with the
feature on and then replaying the test with the feature off.

##  4\. AXNav System

Based on our formative interviews with QA testers, we designed and built a
system, AXNav, that interprets an accessibility test authored in natural
language, and replays the test instructions on a mobile device while
manipulating the accessibility feature to be tested. The main output of our
system is a chaptered video that a tester can navigate and replay along with
heuristics to flag each issue in the video. AXNav currently supports four
accessibility features: VoiceOver, a gesture-based screen reader (voi, 2023);
Dynamic Type, which increases text size; Bold Text, which increases text
weight; and Button Shapes, which ensures clickable elements are
distinguishable without color, typically by adding an underline, or button
background. We selected these features for our system based on feedback from
formative interviews—they provide good coverage of real-world test suites and
span different modalities. AXNav can be extended to other accessibility and
device features provided APIs are available to control them. For each user-
provided test, AXNav executes the test on a specified app both with and
without a specified assistive feature activated for comparison. We based this
design on feedback from our user study that performing tests with and without
accessibility features on is a common manual testing practice (3.3.4)

AXNav consists of three main components that are used to prepare for, execute,
and export test results: (1) Device Allocation and Control, (2) Test Planning
and Execution, and (3) Test Results Export. These components work together to
provision and stage a cloud iOS device for testing, automatically navigate
through an app running on the cloud device to execute the test, and collect
and process test results.

###  4.1. Device Allocation and Control

Before executing a test, AXNav provisions a remote cloud iOS device and
prepares it according to parameters it extracts from the test instructions.
AXNav extracts the name of the app to be tested and the assistive technology
to use in the test (e.g., Dynamic Type) from the instructions to automatically
install the app and select the assistive feature to test. Instructions
typically take the form of those shown in Figure 2.

During setup, AXNav installs a custom binary on a cloud device which silences
several system notifications, controls screen recording, and provides an
interface to interact with assistive technologies during testing. AXNav uses a
device API to toggle and configure the specific accessibility feature under
test (e.g., Dynamic Type size). The system takes additional steps if the test
is for VoiceOver: it activates the caption panel and sets the speaking rate to
0.25 in order to accommodate for speeding up the exported video in the Test
Results Export step.

When the device is ready for the test to be executed, the system launches the
app under test, and begins screen recording. The test execution engine can
interact with the cloud device over VNC and the accessibility-specific
features supported by the custom binary (see subsubsection 4.2.4).

####  4.1.1. Accessibility Feature Control and Replay

AXNav performs tests with and without assistive features on since our
formative studies revealed this need (F4; subsubsection 3.3.4). For tests with
Dynamic Type, the system launches the target application, increases the
Dynamic Type Size, navigates to the target screen specified in the test, takes
a screenshot, kills the application, and repeats this process for all four
Dynamic Type sizes and without Dynamic Type on. This enables testers to
observe the corresponding changes on the screen as the size is increased. For
Bold Text and Button Shapes, the system navigates to the target screen
specified in the test with and without the feature enabled. This enables our
system to save pairwise screenshots of each tested screen with the feature on
and off for comparison. For VoiceOver, our system replays the instructions
once with VoiceOver on and then again with VoiceOver off.

Figure 3. Planning and replanning workflow of our LLM-Based Multi-Agent
Planner

The figure shows the planning and replanning workflow of our system. A Planner
produces a Tentative Plan which directs to Action which steps through the
plan. Action directs to Tools which uses tools to act on the U (e.g., Tap,
Swipe), and Action directs to Evaluation which evaluates the results of taking
an action. Evaluation points to Replan which goes back to Planner. Evaluation
can also direct to Continue which goes back to Tentative to continue the
folow.

###  4.2. Test Planning and Execution

AXNav uses an LLM-based UI navigation system to translate from natural
language test instructions into a set of concrete steps, execute steps in the
plan by calling APIs that interact with a device, and feed back results to the
planner when needed (see Figure 3). We use OpenAI GPT-4 as the LLM in our
implementation because of its proficiency in interpreting and acting on
natural language instructions (OpenAI, 2023). AXNav can be easily adapted to
use other LLMs. Our system architecture is loosely inspired by ResponsibleTA
(Zhang et al., 2023), but eliminates some elements (e.g., having a facility
for masking LLM inputs), and merges other elements (e.g., combining
feasibility with actions). To provide device state to the LLM agents, we use
pixel-based machine learning models to recognize UI elements and text in the
iOS UI and predict descriptive labels for icons (Zhang et al., 2021; Chen et
al., 2022). The system formats detected UI elements as text strings to be
ingested by the LLM, described in subsubsection 4.2.1. To interact with the
device, AXNav provides tools that the LLM invokes to emulate human inputs over
VNC and provides VoiceOver gestures the LLM can invoke with our custom binary
(see subsubsection 4.2.4).

####  4.2.1. Test Plan Proposal

The main coordinator for the multi-agent UI navigation system is the planner
(Figure 3, Planner), which takes as input the accessibility test instructions,
the name of the app under test, and the formatted UI element detections from a
screenshot of an iOS device. The planner prompt contains instructions to
formulate a tentative step-by-step plan (Figure 3, Tentative Plan) to
accomplish the test goal with the current app and the set of actions that can
be taken in a step. To adapt to changes in the UI or unexpected errors (e.g.,
permissions request dialogs), the system instructs the planner to provide
reasonable search queries when the test does not specify them based on the app
name and the current context of the screen, traverses backwards through the
app if it encounters an unexpected state, and accepts an imperfect plan if
needed, since it can be revised later.

The expected output of the planner is a JSON-formatted object that contains a
list of steps. Each step contains a thought designed to facilitate Chain-of-
Thought (CoT) reasoning (Kojima et al., 2022) that answers how the step will
help achieve the user’s goal; evaluation, which suggests criteria to determine
task success; action, a brief, specific description of the action to take
(e.g., tap, swipe, enter text); and a status field, which is initialized as
“todo” and updated to “success” when a step is executed correctly.

####  4.2.2. Representing UI Elements to the LLM

AXNav uses the Screen Recognition model from Zhang, et al. (Zhang et al.,
2021), which predicts bounding boxes, labels, text content, and clickability
of UI elements from screenshot pixels of iOS devices. The system uses
heuristics introduced in (Zhang et al., 2021) to group and sort detected UI
elements in reading order and detect whether an element is a top-left back
button. The system uses an additional heuristic to detect the presence of a
keyboard (to hint that a text field is selected), by detecting the presence of
single-character OCR results on the lower third of a screenshot. If the system
detects a keyboard, it filters all UI elements detected on the keyboard,
except for a submit button (“return”, “search”, “go”, etc.).

To format the detected UI elements as text for use with an LLM, the system
generates a list of items, which each contain an incrementing integer ID of
the UI element in reading order; the UI detection model provided label of the
UI element (e.g., Icon, Toggle); text contained by the UI element, if any; and
the coordinates of the bounding box around the element. The system adds
predicted clickability when formatting detections for the action execution
agent. Unlike prior approaches that format UI detections as JSON or HTML
(Zhang et al., 2023; Feng and Chen, 2023), our system formats UI elements as a
simplified list, which considerably economizes on tokens. For example for an
element with ID 3:

(3) [Button (Clickable)] "Try It Free" (194, 1563) to (1042, 1744)

####  4.2.3. Mapping from Plan Steps to Concrete Actions

For each step in the plan proposed by the planning agent, AXNav uses another
LLM-based agent to map from the text instruction to a concrete action (Figure
3, Task to Action) to take on a particular UI element. This approach is
inspired by prior works which use LLMs to map from instructions to available
UI actions (Li et al., 2020; Feng and Chen, 2023). In effect, our action agent
performs several critical subtasks to navigate UIs in a single step: it
identifies how to map a natural language instruction to the specific context
of a UI, evaluates the feasibility of the requested action, and produces
arguments for a function call to execute the task. Our subtask-to-action
prompt contains instructions to output a specific action to take on a given
screen, represented by the formatted UI detections. The output of the planner
is a JSON-formatted object that contains a thought to elicit CoT reasoning,
relevant UI IDs, a list of UI elements the agent considers relevant (also to
elicit CoT reasoning), and a single action, which specifies a function call in
JSON to execute interactions on the device (Figure 3, Act on UI). The
available actions are:

  * •

Tap: Tap a UI element given its ID. The prompt instructs the agent that
tapping a non-clickable object is acceptable if it is the only reasonable
option on a screen.

  * •

Swipe: Swipe in a cardinal direction (up/down/left/right) from a specified (x,
y) coordinate. The system tells that agent that swiping can be used to scroll
to view more options available on a screen if needed.

  * •

TextEntry: Tap a UI element given its ID and then enter a given text string by
emulating keystrokes. The agent is told to come up with appropriate text if it
is not provided.

  * •

Stop: Abort execution of the current step and prepare feedback for the
replanner to update the plan as needed. The feedback must specify what
information is needed in an updated plan.

The system reads agent-produced actions and executes them (Figure 3, Act on
UI) by either executing inputs over a VNC connection to the remote cloud
device, or, if the test uses VoiceOver, through a connection to a custom
binary running on the device which emulates VoiceOver actions.

####  4.2.4. Executing VoiceOver Actions

For action execution in VoiceOver, the system interacts with the device
through VoiceOver’s accessibility service on the device through a Swift API
(built on top of XCTest (xct, 2023)) to trigger key VoiceOver gestures (voi,
2023). These gestures execute VoiceOver instructions in the same way a user of
VoiceOver would interact with an application (F3; subsubsection 3.3.3).
Supported gestures are as follows:

##### Right swipe through all elements (read-all)

This command mimics the VoiceOver right swipe gesture by navigating through
all exposed elements on the screen, typically in a top-left to bottom-right
ordering. Our system limits the number of elements navigated to 50. After
right swiping through the first 50 elements, the system activates the first
tab, if it exists, and navigates through all tabs from left to right in the
tab bar.

##### Activate an element on a new screen (activate-from-coordinates)

This command combines VoiceOver’s right swipe and double tap gestures to
activate an on-screen element. In our formative interviews, our prototype
video demonstrated the “Search for a Show” task in VoiceOver by directly
navigating to relevant UI elements using the VoiceOver Tap gesture followed by
Double Tap. However, participants gave us feedback that they preferred the
demonstration to be more similar to how a non-sighted user would find and
activate a UI element, by using Right Swipe to navigate through UI elements
(F3; subsubsection 3.3.3). To confirm this, we observed a screen reader user
performing the “Search for a Show” task, who followed a roughly similar
pattern.

activate-from-coordinates takes as input x and y coordinates corresponding to
the center of the UI detection bounding box to be activated; and the UI Type
label from the UI detection model (e.g., Tab). If the UI Type is Tab, the
system navigates the VoiceOver cursor directly to the leftmost tab element,
uses the Right Swipe gesture to find the first tab containing the x and y
coordinates, and then activates it using Double Tap. If UI Type is not Tab,
the system navigates forward from the current element using Right Swipe until
it reaches the last VoiceOver element or finds an element containing x and y
which it activates using Double Tap. If the system does not find the element,
it navigates backwards using Left Swipe until it reaches the first VoiceOver
element containing x and y and if so, activates it using Double Tap. If the
system does not find an element containing the coordinates, the command
returns without activating any element.

##### Scroll (Up/Down/Left/Right) (scroll-<direction>)

This command implements the VoiceOver Three Finger Swipe gesture, which
scrolls the current screen in the given cardinal direction by one page.

To prevent the VoiceOver caption panel from interfering with the UI detection
model’s assessment of the state of the app, the system removes the caption
panel from the formatted UI detections using a heuristic based on a fixed
height from device dimensions. When the input test instructions specify to
perform a task that requires navigating through multiple UI elements and
screens, the system triggers VoiceOver navigation using activate-from-
coordinates when the action agent instructs a TextEntry or Tap action. If the
action agent instructs the system to perform a Scroll action, the system calls
the corresponding scroll-<direction> action in VoiceOver. If the instructions
state to navigate to a specific screen to verify the VoiceOver elements and
navigation order, the system calls read-all once it reaches the final step of
UI navigation, which will swipe through all exposed elements on the screen,
typically in a top-left to bottom-right ordering. This enables testers to
determine whether all elements within that screen are accessible by VoiceOver.

####  4.2.5. Evaluation and Replanning

Once an action is executed on the device, AXNav uses a third LLM agent to
evaluate the results (Figure 3, Evaluation). The system provides the
evaluation agent the test goal, the entire current tentative plan, the action
JSON object (including the function call and “thought”), the UI detections of
the screen before the action was taken, and UI detections of the screen after
the action was taken. The output of the evaluation agent is a JSON object that
contains evaluation_criteria, to encourage CoT reasoning; a result of success,
fail, or task complete; and an explanation, which the system feeds back into
replan if the evaluation fails.

The evaluation agent prompt also includes evaluation hints designed to reduce
navigation errors: if UI elements significantly change, the action likely
succeeded; if the state of the current screen changes, but a new view is not
opened, it errs that the action succeeded; if the last action was a scroll or
swipe, but the screen did not change, the action likely failed; if the target
element is not visible, more scrolling may be required; and if the last action
was to click on a text field, the evaluation should be whether a keyboard is
visible. If the evaluation succeeds, the action agent executes the next step
in the plan. If the evaluation fails, a replanning agent updates the tentative
plan from the current step onwards.

The replanning agent is similar to the original planning agent, but includes
the previous plan, the current step being executed, and a field including a
stop condition or evaluation error. Similar to the planner, it outputs a JSON
object of a tentative plan, revised from the current step onward.

Figure 4. Overview of intermediate steps used by AXNav to interpret natural
language test instructions; provision and stage a device for testing;
formulate and execute a plan to navigate the UI for the test; and export the
test results.

The figure shows the different phases of our system in separate rows,
including Preparatory, which contains the high level goal and prompt; Test
instructions includes the high level test instruction; Device Allocation and
Control includes the steps the system takes to initialize the device; Test
Plan Proposal shows the high level test plan the planner produced;
Representing UI Elements shows formatted UI detections for a example screen;
Take Actions from Plan Steps shows examples of the concrete predicted actions
to take on the UI; Evaluate Action Results shows examples of planner
evaluation results; Finally, Test Results Export shows the output of the test,
which is used to generate the output video.

Figure 5. Examples of issues flagged by our heuristics for Button Shapes
(left) and Dynamic Text (right). The Button Shapes heuristic flags the
Collections row which has a button shape also is underlined (a possible bug).
The Dynamic Type heuristic flags several text elements with red boxes
indicating the size has not increased with the DT size update (a possible
bug).

The figure shows screenshots of UIs on the left; Button Shapes has red box
flagged around Collections row which has a Button Shapes bug with a underlined
UI element inside a button shape. it also has green boxes around UI elements
passing the test; Dynamic Type examples show red boxes around text elements
that have not increased in size, and green boxes around text elements that
have increased in size between the two screenshots.

###  4.3. Test Results Export

AXNav’s primary output is a sped-up video of the test being executed, with
potential issues flagged by heuristics represented by colored bounding boxes
in the affected frames. The system annotates the video with chapter markers
that label each step of the test navigation being performed, and each
heuristic-flagged issue.

####  4.3.1. Accessibility Issue Flagging

Our system flags four different types of potential accessibility issues in the
output video, including VoiceOver navigation loops and missing elements,
Dynamic Type text resizing failures, and Button Shapes failures, visualized in
Figure 5; (F1; subsubsection 3.3.1). We selected these three issues based on
the supported accessibility features by AXNav, and for Dynamic Type and Button
Shapes, these issues can be flagged from the screenshot pixels and UI
detections to verify behavior. We also observed that navigation loops
occurring in some manual test passes through our regression testing dataset
(described in Section 5) and another common issue is that some UI elements are
not available to VoiceOver (Zhang et al., 2021).

During each replay (Phases 3-4), the system records screenshots of each step,
which our system postprocesses to detect potential failures to flag in the
output for Dynamic Type and Button Shapes.

##### VoiceOver loop detection and missing VoiceOver elements

For VoiceOver, our system detects loops in the navigation order during the
activate-from-coordinates and read-all commands. To detect loops, the system
maintains a list of all visited VoiceOver elements, and detects a looping bug
if any element is revisited during the command. To enable the system to
navigate the remaining task steps, the system attempts to break out of the
loop by finding the next VoiceOver element below the element where the looping
was detected, navigating to it and either continuing with read-all or
activate-from-coordinates. The system also flags when a corresponding
VoiceOver element to activate cannot be found during activate-from-
coordinates, and flags this as a missing VoiceOver element.

##### Dynamic Type

The Dynamic Type heuristic determines if text elements and their associated
icons increase in size when the system-wide Dynamic Type size is increased.
The heuristic takes two screenshots of the same view as input: one with a
baseline text size, and another increasing Dynamic Type size by one increment.
The heuristic first uses a UI element detection model (Zhang et al., 2021) on
the screenshot to recognize text elements and perform OCR (Apple, 2020).

The heuristic then uses fuzzy string matching with Levenshtein distance to
find corresponding text elements between the two screenshots, with a partial
similarity threshold set to 50%. The heuristic excludes elements without
matches. For a text element to pass the heuristic, its bounding box in the
test screenshot must increase by an adjustable threshold set to 10% compared
to its corresponding pair in the baseline screenshot.

To identify icons paired with text elements, which should typically scale
along with the text, the heuristic greedily matches icons to text elements in
both screenshots by minimizing the distance between the icon’s right bounding
box coordinate to the text element’s left coordinate. To remove icons that are
not to the immediate left of text, the heuristic excludes icons with a gap of
more than half the icon’s width to the right text element or whose top and
bottom are not bounded by the text element’s bounding box.

The heuristic then matches those icons using paired elements on both screens,
and excludes them if only one text element has a matching icon on both
screens. The heuristic then uses the same size increase threshold to flag
icons as potential bugs (10% increase in bounding box area), with a separately
adjustable threshold.

##### Button Shapes

The Button Shapes heuristic determines, for a given screenshot, whether
clickable text elements outside of clickable container elements are
underlined. This heuristic takes a single screenshot of a view with Button
Shapes activated. The heuristic uses a UI detection model (Zhang et al., 2021)
to locate and classify elements in the UI, along with their predicted
clickability. For every clickable container element (Buttons and Tabs), the
heuristic flags any contained element that is also underlined, which indicates
a bug. For any uncontained text element predicted as clickable, the heuristic
flags it if it is not underlined.

The heuristic detects underlines in text elements by extracting the image
patch of the text bounding box, binarizing the patch using Otsu’s method
(Otsu, 1979), edge-detecting the image with the Canny edge detector (Canny,
1986), and using the Hough Line transform (Duda and Hart, 1972) to detect any
horizontal line that spans at least 75% of the width of the patch. If a text
element is underlined when it should not be (or vice versa), it fails the
heuristic.

####  4.3.2. Output Video Generation

Throughout the replay process, the system records a timestamp of every action
performed on the device, along with the action and the activated UI element.
The system uses these timestamps to generate chapter labels in the output
video (using ffmpeg), which a tester can use to navigate through the recorded
video to find issues more efficiently (F2, 3.3.2). Figure 1 shows example
chapter labels our system generates. For each category of issues our
heuristics flag, the system adds a new chapter in the output video on the
impacted frame where the heuristics flagged the issue.

##  5\. Technical Evaluation

We conducted two evaluations of our system to determine the accuracy of our
test replay. Few datasets currently exist in the literature for UI navigation
tasks for mobile apps from natural language, and none for iOS apps
specifically that we are aware of to provide a baseline for our evaluation.
Instead, we evaluated the system on a regression test suite used within our
company to test a set of media apps, and created a free apps dataset from free
apps within a publicly available app store.

Free Apps | Performance
---|---
Diff. | VO | BT | DT | BS | S | P | F | Acc.
1 | 0 | 4 | 2 | 1 | 5 | 1 | 0 | 83.3%
2 | 5 | 1 | 3 | 4 | 9 | 3 | 2 | 64.3%
Total: | 5 | 5 | 5 | 5 |  |  |  |
Overall Accuracy: | 70% |  |  |  |  |  |  |
Table 1. Total evaluation test case counts for our Free Apps Dataset for each AX feature for VoiceOver (VO), Dynamic Type (DT), Bold Text (BT), and Button Shapes (BS). We report Navigation Replay success (S), Navigation Partial success (P), and Navigation Replay fail (F), along with overall accuracy. Regression Testing Apps | Performance
---|---
Diff. | VO | BT | DT | BS | S | P | F | Acc.
1 | 18 | 3 | 21 | 3 | 42 | 0 | 2 | 95.5%
2 | 16 | 1 | 2 | 0 | 11 | 2 | 5 | 61.1%
Total: | 34 | 4 | 23 | 3 |  |  |  |
Overall Accuracy: | 85.5% |  |  |  |  |  |  |
Table 2. Total evaluation test case counts for our Regression Testing Dataset
for each AX feature for VoiceOver (VO), Dynamic Type (DT), Bold Text (BT), and
Button Shapes (BS). We report Navigation Replay success (S), Navigation
Partial success (P), and Navigation Replay fail (F), along with overall
accuracy.

###  5.1. Regression Testing Dataset

First, we evaluated the system on a large regression manual test suite
consisting of 300 tests, which we introduced in Section 3, Figure 2. As our
system supports only the four features of VoiceOver, Dynamic Type, Button
Shapes, and Bold Text currently, we evaluate our system on the tests for those
features which include 64 test cases across five apps. To note, this
regression test suite is used for manual testing on a semi-annual or annual
basis and was not constructed for the purposes of being used within any
automation system. Many of the tests are very high level and assume the QA
tester has a high level of knowledge and expertise on the feature and the app
under test as the typically performing these tests is a small team of
accessibility testing experts. We chose to use this dataset to showcase the
authentic performance of our system.

###  5.2. Free Apps Dataset

We also constructed a dataset of publicly available apps and accessibility
testing instructions from a publicly available app store top 100 list of free
apps. We first randomly selected an app from this list and then constructed
one manual test for each of our four supported accessibility feature
categories. One researcher on our team drafted the tests using the regression
testing suite as an example, and then validated the tests were realistic by
discussing them with an expert accessibility QA tester in our company. After
drafting the tests for one app, the researcher selected another app randomly.
If the app was from the same category as an already selected app, the
researcher randomly selected another app and so forth until selecting five
apps across five app categories. The final dataset consists of 20 manual tests
across five apps and four accessibility features.

###  5.3. Accuracy Results

We evaluated the difficulty of each test through a rubric based on prior work
(Gur et al., 2023) which tends to rate each type of instruction task into Easy
and Hard categories for evaluation.

  * •

Easy regular expression-based retrieval task: In this level, the tests consist
of one step where the UI element can be matched with a simple one step action.
The action typically involves straightforward operations, such as finding a UI
element, tapping on it, and scrolling on the resulting page, if required.
These are easy tasks because the UI element to accomplish the task can be
identified and matched with a single step.

  * •

Hard structured problem solving or open-loop planning task: In this level,
replaying the test instructions requires the system to make decisions and take
actions based on a predetermined plan and the current state of the screen.
Each task consists of a preplanned set of actions, sequence of actions
previously taken (in case of infeasibility), and the current state and
anticipated future states of the UI. This type of task requires the current
state of the screen, previously infeasible actions that were suggested, and
the high-level goal.

To group the tests into the above categories, two authors independently rated
each test and then met to discuss and resolve any differences. Table 1 and
Table 2 show the total counts for each level across the four supported
accessibility feature categories and the two separate datasets, regression
testing dataset and free apps dataset.

To repeat each test, we input the test instructions into the system, reset the
phone’s current state to match the initial state specified by the test, and
then executed the test instructions on the device. During this process, we
recorded all interactions between the system and the app’s state. For each
dataset, we report navigation replay success which measures whether our system
can follow the instructed steps successfully to reach the destination
instructed in the test, and accessibility test success for whether the
accessibility feature test succeeded. We determined this success based on our
own manual evaluation based on the expected behavior for each accessibility
feature. To ensure consistency, two researchers independently rated each test
case and then met to discuss and resolve any differences.

For the regression testing dataset, our system successfully replayed 95.5% of
easy test cases, and 61.1% of hard test cases for an overall success rate of
85.5%. Table 2 summarizes these results. Within our organization’s apps,
support for the supported accessibility features is already high;
accessibility test success rate across these tests was 78%. We are also
working with the owners of this regression testing dataset to actionably use
these results by filing them into our internal bug tracking system.

For the free apps dataset, our system successfully navigated 83.3% of easy
test cases, and 64.3% hard test cases for an overall success rate of 70%.
Table 1 summarizes these results. We also report partial navigation replay
success which indicates our system was able to replay one or more steps in the
test but did not end up in the correct final state. Support for the
accessibility features of bold text, dynamic type, and button shapes
unfortunately was low across the five apps resulting in an accessibility test
success rate for these apps of 15% across the 20 test cases which further
motivates the use of systems like ours within the app development workflow.

While the navigation replay success of our system is good for both datasets,
our system is fails to replay some test cases. In some cases, the test
navigation replay fails because the test requires tapping on a certain item in
a collection where only some items have the required condition (e.g., have a
subscription available) and the planner typically suggests activating the
first item. In other cases, the planner cannot deduce enough knowledge about
the app and predicts that key functionality for the replay does not exist in
the app. In a few cases, key UI elements needed to be activated for the test
that were scrolled way down on the screen, and our system did not continue
scrolling long enough. Another challenge we have seen is that the planner
sometimes is unable to determine when to stop and gets into an infinite loop.
These are areas we hope to improve our system in future work.

##  6\. User Study

We present our system to 10 professional accessibility testers (explained in
§6.2) in user study sessions. We investigated how well the system replicated
accessibility tests conducted by human testers, how the system can aid testers
in finding accessibility issues, and how testers would like to integrate the
system into their accessibility testing workflow.

###  6.1. Procedure

We conducted a interview-based user study with 10 participants who perform
accessibility tests as part of their daily professional work. Specifically, we
conducted 30-45 minute video call-based interviews with each of the
participants. During the session, we first presented an overview the system to
the participant. We then showed the participant three videos generated by the
system and the natural language instructions that the tests were generated
from. All participants watched the same three videos with randomized orders.

Because of the limited duration of the sessions, we were only able to show
three videos to the participants. We selected the three videos from those
generated by the system for the 64 test cases for technical evaluation
described in §5. Each video shows an accessibility test on iOS media
applications of e-books, news stories, and podcasts respectively, with
different UI elements and layouts. We selected the tests shown in the videos
based on different accessibility features, including Voice Over, Dynamic Text,
and Button Shapes. We selected two of the tests from those with the difficulty
level of Easy, and one test with the difficulty level of Hard (Section §5
explains the difficult levels). Furthermore, all three videos contained some
accessibility issues, which we prompted the users to discover using the
heuristics as part of the system. The specific videos that we used for the
user studies are as follows:

  1. (1)

_VO:_ This video contains a test on the Voice Over feature on an application
that hosts podcasts. The test instruction prompts the system to share an
episode of a podcast show through text message using Voice Over. (Difficulty
level Hard)111This video does not include an issue flagged by the system. In
order to show participants what heuristics in VO look like, we presented them
with a supplementary video of another VO case where the system flags a
VoiceOver navigation loop in the chapters.

  2. (2)

_DT:_ This video contains a test for the Dynamic Text feature on an
application that hosts news articles. The test instruction prompts the system
to increase the size of the text in four different fonts in a specific tab of
the application. (Difficulty level Easy)

  3. (3)

_BS:_ This video contains a test for the Button Shapes feature on a
application for reading books. The test instruction prompts the system to test
the Button Shape feature across all the tabs in the application. (Difficulty
level Easy)

Notably, all three videos contained some levels of pre-discovered mistakes in
test execution or heuristics that attempted to identify accessibility issues.
Specifically, the VO video contains a mistake in test execution where it
shared a podcast show instead of a specific episode; the heuristics in the DT
and BS video failed to identify some of the accessibility issues. We
intentionally presented those imperfections to the participants to showcase
the authentic performance of our system and discuss any limitations and future
directions.

For each video, the researcher asked the participant to think aloud as they
watch the video to 1) point out if they notice any accessibility issues
related to the input test, and 2) point out any places where the test
performed by the system could be improved. After each video, we interviewed
the participants on how the test in the video met their expectation, and how
the heuristics assisted them to find any accessibility issues. Besides
qualitative questions, we also asked the participants to provide 1 to 5 Likert
scale ratings on how similar the tests in the videos are to their manual
tests, and how useful the heuristics are for tests to identify accessibility
bugs.

After all three videos, we asked the participants to provide their general
attitude towards the system, how they would like to incorporate it in their
workflow, and areas of improvement. We also asked the participants to provide
1 to 5 Likert scale ratings on the usefulness of the system in their workflow
in its current form and in an ideal performance.

###  6.2. Participants

We recruited 10 participants who are full time employees in a large tech
company. All 10 participants are what we referred as “professional
accessibility testers”, who perform manual accessibility tests as part of
their professional work. This include workers with professional titles of
accessibility QA testers and accessibility engineers. We recruited
participants via internal communication tools within the company. Since we did
not collect information on the pronouns of our participants, we use the gender
neutral pronoun “they/them” to refer to all participants in our findings.

We include the full interview instruments and profiles of participants in the
Supplemental Materials.

###  6.3. Data Collection and Analysis

The data collected during the study includes audio and video recording of the
study sessions with the consent of the participants. We transcribed all the
recordings into text format using an automated tool. The research team also
took field notes during the session and used the notes to guide the analysis.
The length of the sessions ranged from 29 minutes to 49 minutes, with an
average length of 37 minutes. The interview with P9 only covers two videos (VO
and BS) due to the limitation of the participant’s availability.

The analysis of the qualitative data from the user study followed the
procedure of thematic analysis (Guest et al., 2011). Two authors of the paper
first individually coded all the transcripts. Then the two researchers
presented the codes to each other and collaboratively and iteratively
constructed an affinity diagram of quotes and codes together to develop
themes. The following finding section presents the resulting themes. We also
reported the descriptive statistics of the data collected from the Likert
scale rating questions, including the mean, standard deviation (SD) and sample
size (N), to supplement to our qualitative insights.

###  6.4. Findings

####  6.4.1. Performance of the Automatic Test Navigation

##### Automatic test navigation replicates manual test.

Participants generally agreed that the system navigated the application in a
similar path as they would conduct the test manually. Many thought the system
mimics how a human tester might carry out the BS and VO test cases
particularly well.

For VO, Participants rated $4.60$ (SD = $0.52$, N = $10$) on average in the
similarity regarding the navigation path between human testers and the AI
(between “very good match” and “extremely good match” with their manual
testing procedures). P3 was impressed by the system’s ability to execute the
test: “my mind is blown that it was able to find that [shared button] buried
within that actions menu.” Similarly, in the BS test case, Participants rated
$4.35$ (SD = $0.75$, N = $10$) on average in the similarity regarding the
navigation path between human testers and the AI (between “very good match”
and “extremely good match” with manual testing procedures). In P9’s opinion,
the system might actually out-perform most human testers in BS, since it could
be subjective for a human tester to determine what consists of a button shape.

Compared to BS and VO, the system’s performance in DT received a relatively
lower rating in its similarity to manual testing procedures. On average,
participants rated it to be $3.39$ (SD = $0.78$, N = $9$), (between
“moderately good match” and “good match” with manual testing procedures). A
main difference is that the system always relaunches the application between
the tests of different text sizes, while human testers tend to use the control
center to adjust text sizes within the application without relaunching it in
order to mimic what a real user would do. At the same time, some recognized
the benefit of relaunching, as it added an additional layer of test: “I really
like that launches the app in between changing the tech size, because I think
it’s a separate class of bug, whether or not, it responds to a change in text
size versus having the text size there initially.” (P8)

##### Chapters provides context and comparison in navigation.

Participants reacted positively to the chapter feature, as it enabled
efficient navigation through the video. For those like P10, chapters allowed
them to inspect any potential issues that were not flagged by the heuristics:
“It shows all the different breakdowns,… like, here’s where we turn on
VoiceOver, here’s where we tell VoiceOver to read all. I really like that,
because you can then go to the beginning of where the VoiceOver was enabled,
to see exactly where the issue is. ” Chapters also provided an easy way to
compare between screens, so a tester could catch nuanced changes: “There was
that moment where I wasn’t sure if the text changed between 2 sizes…so you’d
be able to click each of these chapters just to visually jump to that frame
and do a quick comparison.” (P2) P5 and P7 also suggests future versions of
the system to show chapter names on the corresponding screens in the video to
further support testers to navigate through the video.

##### Navigation could be more exploratory and in-depth.

Participants suggested that future version of the system could enable
exploratory and alternative navigation, as well as more in-depth tests into
the UI structure. For example, for BS, many participants mentioned that they
would have explore more content in the application to ensure the Button Shape
feature work for all elements: “I would explore a little more. I would go into
different screens. Like, I would click on on a book, for example, and look at
that screen.” (P6) For VO, participants suggested the system to support
diverse navigation pathways, such as alternative, non-linear pathways that VO
users could go through to share an episode of podcasts (P7) and navigation
using both swiping and tapping gestures (P4). Another common request is the
ability to scroll through the screen of an application when testing display
features like DT and BS. As P10 pointed out, to test DT, “when you jack up the
text size so much, a lot of times I have to scroll more to see more content.”

####  6.4.2. Usefulness of the Heuristics

##### Heuristics aid discovery of accessibility issues.

For all three cases of VO, BS, and DT, all participants were able to spot at
least one accessibility issue with the help of the heuristics, and agreed that
the issues that they discovered were significant enough to be filed in the
internal bug reporting system within their company. Overall, participants
agreed that the heuristics provided by the system assisted them to find the
issues. For VO, BS, and DT respectively, participants on average rated $4.06$
(SD = $1.38$, N = $9$) (between “userful” and “very useful”), $4.75$ (SD =
$0.43$, N = $10$) (between “userful” and “very useful”), and $3.67$ (SD =
$1.09$, N = $9$) (between “moderately userful” and “useful”) on the usefulness
of the heuristics.

Specifically, the potential issues flagged in the chapters allowed
participants to navigate to where the issue was and review it in greater
attention. The heuristics in particular helped direct testers’ attention to
the potential issues, which might otherwise be too subtle to discover:
“Watching it in a video, as opposed to actually interacting with it, I think
it is easier to potentially miss things… So, having some sort of automatic
detection to surface things [is good].” (P8) Even though they sometimes
resulted in false positives, participants still appreciated the heuristics as
they provided an extra layer of caution, as P10 said, “I actively like the red
(annotation boxes around potential issues) because I think the red is like
‘take a look at this’ and then even if it’s not necessarily an issue, that’s
not hurtful.”

##### Human testers face that risks of over-reliance on heuristics.

Participants expressed the concern of over-reliance on the heuristics provided
by the system. In some sessions of our study, participants were able to find
issues that were not marked by the heuristics. Regarding those cases,
participants worried that the false negatives from the heuristics might
actually bias testers: “if things are marked as green, and maybe there
actually is an issue in there, maybe that would dissuade somebody from looking
there.” (P10) This could influence testers of different experience levels
differently. An experienced tester might be able to leverage their expertise
to identify issues, while an in-experienced tester might over-rely on the
suggested bugs (or non-bugs) made by the AI. As P8 explained: “If somebody is
kind of experienced with large text testing, they kind of know what to look
for… If it’s an inexperienced tester, they might not know that the false
positives are false positives and might file bugs.” (P8)

##### Future improvement of heuristics.

Participants mentioned that a mechanism to explain how the heuristics were
generated and applied to the test cases might help with the issue of over-
reliance. For example, P7 imagined it to be a series of “human readable
strings, like what it actually found… human readable descriptions of what the
error is in addition to seeing the, the boxes.” Other suggestions focus on
making the heuristics more digestible for the testers. Currently, we showed
the heuristics as screenshots with annotations that are separated from the
videos. Participants found it would be more easily to comprehend the
heuristics if they were encoded in the video. Some found the screens where the
heuristics were shown a little too “busy” (P1) and suggested only the
potential issues to be annotated. Along the same line, participants like P6
suggested separating the flagged issues and regular chapters so that testers
could “go through all the problem screens to file issues and then go through
the steps for quickly verifying in QA.” P7 brought up the idea to include a
dashboard or summary mechanism in the system, so that a tester “instead of
just having a scrub through this video,” could see “a summary of the errors as
well.”

####  6.4.3. Integration in Accessibility Testing Workflow.

##### Overall usefulness.

Overall, participant reacted positively to our system. Participants rated
$4.70$ (SD = $0.48$, N = $10$) (between “userful” and “very useful”) on
average to the question of how useful the system is in their existing workflow
if it performs extremely well, and $3.95$ (SD = $0.96$, N = $10$) (between
“moderately userful” and “useful”) on average to the system in its current
form. Participant expressed excitement on the potential of integrating the
system and bringing automation to their workflow. For instance, when asked for
a rating on the overall usefulness of the system, P3 answered: “[I will rate]
it like a 5 million… Even with the current limitations, it is very useful…
just being able to feed it some real simple steps and have it do anything at
all is massively powerful. ”

Participants described a range of potential ways of integrating the system
into their existing workflow, including to use the system in test planning,
running tests, reporting bugs, and training new testers.

##### Tests planning.

A major use case of this system is to automate the planning and setup of the
test, which, according to our participants, is a time-consuming part of
accessibility testing as it can involving a excessive amount of manual work to
“go through and find all of the labels to tap through.” (P3) The step-by-step
executable test plan generated from natural language from our system can
reduce the amount of tedious work: “rather than having to like hard code
navigation logic, it seems that this is able to determine those pathways for
you… I think this idea is really awesome and would definitely save a lot of
hours of not having to hard code the setup steps to go through a workflow with
VoiceOver.” (P4) P4 also envisioned using the system as a test authoring tool,
which can generate templates that can be run every day.

##### Running the tests.

Participants found the system helpful in reducing workload and saving time in
running tests. Some participants would like to embrace the automation provided
by the system, keeping the system running a large scale of tests in the
background while the team could focus on more important tasks: “you can run it
in an automated fashion. You don’t need to be there. You can run it overnight.
You can run it continually without scaling up some more people. ” (P7) As P8
imagined, “this could run on each new build [of the software], and then what
all the QA engineer has to do is potentially a review about an hour worth of
videos that were generated by the system, potentially automatically flagging
issues.” The system can also provides consistency and standardization in
tests, which “ensure[s] that everything is run the same way every time.” (P8)

At the same time, some participants are more cautious about the automation and
would like to use the system as a supplement to their manual work. P4 believed
that even with the flagged issues, they would still pay attention to the
system-generated videos in a degree similar to how they would test it
manually. P1 imagined that they would still test manually, but would use the
video as validation of their tests “to see if it could catch things that I
couldn’t catch.” (P1) Some also imagined handing lower risk tests, such as
testing Button Shapes, to the system, while using the time saved by the system
to manually and carefully test higher risk tests that will be a regulatory
blocker. (P2)

##### Downstream bug reporting.

The videos generated by the system can also facilitate bug reporting in the
downstream of the pipeline. Participants agreed that the video along with the
chapters generated by the system could be used to triage any accessibility
issues that they would report to the engineering teams. In their current
practice, testers would sometimes include screenshots or screen recording
video clips to demonstrate the discovered issue. Our system prepared a
navigatable video automatically, streamlining this process: “I thought to be
able to jump to specifically when the issue is and scrub a couple of seconds
back or a couple seconds forward is super useful for engineering.” (P7)
Participants also brought up the idea that future version of the system could
support the entire accessibility QA pipeline, as P4 described: “having some
means of tracking those defects through automatic [internal bug reporting
tool] filing… the next step from this is finding issues, then fixing them and
verifying that they’re fixed.” (P4)

##### Educating novices about accessibility testing.

The system can also serve as an educational tool for those who are new to
accessibility tests. This can not only include new QA professionals, but also
developers from under-resourced teams where there is no dedicated QA teams or
pipelines. For example, P2 found the videos and heuristics helpful in terms of
demonstrating certain accessibility bugs that people should be looking for: “
This will be very useful for some of the folks that never do accessibility
testing and they have a context or starting point for even knowing what a
VoiceOver bug is.” (P2) In a way, our system has the potential to raise the
awareness of accessibility issues among broader developer communities, even
for those who do not have QA resources. Along this line, our participants also
suggested that the future version of the system can include explanation for
the issues that it identified to teach its users: “explanation could also be
helpful for teaching: describe, like, this is the issue we found and here’s
why it’s an issue. That that can be helpful as a teaching moment for people.”
(P7)

##  7\. Discussion and Future Work

Our system employs one workflow for accessibility testing for VoiceOver tests
specifically, where the system uses a forward swipe until finding the element
and then activating it. This may not necessarily reflect how a VoiceOver user
might navigate the task as they may have some prior knowledge of the app
structure enabling them to skip around to various parts of the screen to
activate the desired element. Future versions of such systems may explore how
to simulate more potential patterns of interactions.

In our user study, we only showed the same set of videos and the participants
did not directly use the tool themselves. In future work, we plan to deploy
our system in a longitudinal study so that we can better understand how users
directly interact with it.

Our system also is not currently able to replay all natural language tests
successfully on the datasets we evaluated the system on. In future versions,
we plan to incorporate further sources of app knowledge into our planner,
similar to works such as (Wen et al., 2023) which leverage existing app
datasets.

We have so far evaluated this system only in a QA testing context, but there
are many opportunities beyond this as indicated by our user study and other
work in this area. One that we would like to explore is using this system as a
tool to help novice developers better understand the behaviors of
accessibility features and how they should be tested by generating realistic
simulations of behavior on their own apps. Additionally, natural language
instructions are used in manual UI testing, bug reports, and reproduction
steps (Feng and Chen, 2023), and natural language automation systems may
benefit from the techniques we present in this paper to reconstruct these
types of tests. These are use cases we hope to explore in future work.

Our user studies suggest that our system is able to mimic the pathways of
manual tests, help testers identify issues, and can be used to reduce manual
effort in multiple stages of the accessibility testing pipeline, including
test planning, test execution, and bug reporting, and can help with training
new testers and raising awareness of accessibility issues in software
development teams. While the heuristics can be very helpful in helping testers
pay attention to potential bugs, there is the concern that human testers may
over-rely on the heuristics, especially in the cases where the system makes
false negatives predictions, potentially causing testers to miss critical
bugs. This observation echos with previous studies in other domains, which
human users tend to treat AI-generated as authority and feel reluctant to
challenge AI (Bhat et al., 2021; Chen et al., 2023). A potential way to
address the risks of over-reliance is to provide transparency signals of model
mechanism, such as model confidence scores and textual explanations of how the
predictions were made, echoing design guidelines on transparency and
explainability for human-AI collaboration (Amershi et al., 2019). In addition,
our user study revealed a variety of desire for the level of automation in
test execution (explained in §6.4.3). This speaks to the emerging literature
on calibrating the level of user moderation and control in AI based on the
context and stake of the tasks (Lai et al., 2022). Future development of our
system will explore more in this direction and enable users to customize test
planning and control the system in different granularities based on the
specific tests that the users are running.

We also discovered potential need to support additional features for
accessibility testing beyond the four we currently support. We intend to
expand our system to support both navigational accessibility services (e.g.,
Voice Control) and other accessibility settings (e.g., display features such
as contrast adjustment and motion reduction) in the future.

##  8\. Conclusion

In this paper, we presented a system to support accessibility test
interpretation and replay through natural language instructions. Our system
achieves good technical success on replaying realistic manual test
instructions, achieving 70% and 85% navigation replay success. We evaluated
our system with 10 professional accessibility testers who would find the
system very useful in their work and revealed a number of promising future
opportunities and insights into how we can leverage LLM-based task automation
within accessibility testing.

## References

  * (1)
  * rob (2021) 2021\.  Roboelectric.  http://robolectric.org/
  * acc (2022) 2022\.  Accessibility Programming Guide for OS X: Testing for Accessibility on OS X.  https://developer.apple.com/library/archive/documentation/Accessibility/Conceptual/AccessibilityMacOSX/OSXAXTestingApps.html
  * but (2023) 2023\.  Accessibility (Button Shapes).  https://developer.apple.com/design/human-interface-guidelines/accessibility
  * acc (2023a) 2023a.  Accessibility on iOS.  https://developer.apple.com/accessibility/
  * acc (2023b) 2023b.  Accessibility Scanner.  https://play.google.com/store/apps/details?id=com.google.android.apps.accessibility.auditor&hl=en_US
  * esp (2023) 2023\.  Espresso.  https://developer.android.com/training/testing/espresso
  * tal (2023) 2023\.  Get started on Android with Talkback.  https://support.google.com/accessibility/android/answer/6283677?hl=ens
  * acc (2023c) 2023c.  Google.  https://developer.android.com/guide/topics/ui/accessibility/
  * and (2023) 2023\.  Improve your code with lint checks.  https://developer.android.com/studio/write/lint?hl=en
  * voi (2023) 2023\.  VoiceOver.  https://support.apple.com/guide/iphone/learn-voiceover-gestures-iph3e2e2281/ios
  * wca (2023) 2023\.  WCAG 2 Overview.  https://www.w3.org/WAI/standards-guidelines/wcag/
  * xct (2023) 2023\.  XCTest.  https://developer.apple.com/documentation/xctest
  * Alshayban et al. (2020) Abdulaziz Alshayban, Iftekhar Ahmed, and Sam Malek. 2020\.  Accessibility issues in Android apps: state of affairs, sentiments, and ways forward. In _Proceedings of the ACM/IEEE 42nd International Conference on Software Engineering_. 1323–1334.
  * Amershi et al. (2019) Saleema Amershi, Dan Weld, Mihaela Vorvoreanu, Adam Fourney, Besmira Nushi, Penny Collisson, Jina Suh, Shamsi Iqbal, Paul N. Bennett, Kori Inkpen, Jaime Teevan, Ruth Kikin-Gil, and Eric Horvitz. 2019.  Guidelines for Human-AI Interaction. In _Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems_ (Glasgow, Scotland Uk) _(CHI ’19)_. Association for Computing Machinery, New York, NY, USA, 1–13.  https://doi.org/10.1145/3290605.3300233
  * Apple (2020) Apple. 2020.  Recognizing Text in Images.  https://developer.apple.com/documentation/vision/recognizing_text_in_images/
  * Bhat et al. (2021) Advait Bhat, Saaket Agashe, and Anirudh Joshi. 2021\.  How do people interact with biased text prediction models while writing?. In _HCINLP_.  https://api.semanticscholar.org/CorpusID:233365289
  * Bi et al. (2022) Tingting Bi, Xin Xia, David Lo, John Grundy, Thomas Zimmermann, and Denae Ford. 2022\.  Accessibility in software practice: A practitioner’s perspective.  _ACM Transactions on Software Engineering and Methodology (TOSEM)_ 31, 4 (2022), 1–26.
  * Bigham et al. (2010) Jeffrey P Bigham, Jeremy T Brudvik, and Bernie Zhang. 2010\.  Accessibility by demonstration: enabling end users to guide developers to web accessibility solutions. In _Proceedings of the 12th international ACM SIGACCESS conference on Computers and accessibility_. 35–42.
  * Brown et al. (2020) Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020.  Language Models are Few-Shot Learners.  arXiv:2005.14165 [cs.CL]
  * Burns et al. (2022) Andrea Burns, Deniz Arsan, Sanjna Agrawal, Ranjitha Kumar, Kate Saenko, and Bryan A. Plummer. 2022\.  A Dataset for Interactive Vision-Language Navigation with Unknown Command Feasibility.  arXiv:2202.02312 [cs.CL]
  * Canny (1986) John Canny. 1986\.  A Computational Approach to Edge Detection.  _IEEE Transactions on Pattern Analysis and Machine Intelligence_ PAMI-8, 6 (1986), 679–698.  https://doi.org/10.1109/TPAMI.1986.4767851
  * Chen et al. (2022) Jieshan Chen, Amanda Swearngin, Jason Wu, Titus Barik, Jeffrey Nichols, and Xiaoyi Zhang. 2022\.  Towards Complete Icon Labeling in Mobile Applications. In _Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems_ (New Orleans, LA, USA) _(CHI ’22)_. Association for Computing Machinery, New York, NY, USA, Article 387, 14 pages.  https://doi.org/10.1145/3491102.3502073
  * Chen et al. (2021) Sen Chen, Chunyang Chen, Lingling Fan, Mingming Fan, Xian Zhan, and Yang Liu. 2021\.  Accessible or Not An Empirical Investigation of Android App Accessibility.  _IEEE Transactions on Software Engineering_ (2021), 1–1.  https://doi.org/10.1109/TSE.2021.3108162
  * Chen et al. (2023) Valerie Chen, Q Vera Liao, Jennifer Wortman Vaughan, and Gagan Bansal. 2023.  Understanding the role of human intuition on reliance in human-AI decision-making with explanations.  _arXiv preprint arXiv:2301.07255_ (2023).
  * Chowdhery et al. (2022) Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam Shazeer, Vinodkumar Prabhakaran, Emily Reif, Nan Du, Ben Hutchinson, Reiner Pope, James Bradbury, Jacob Austin, Michael Isard, Guy Gur-Ari, Pengcheng Yin, Toju Duke, Anselm Levskaya, Sanjay Ghemawat, Sunipa Dev, Henryk Michalewski, Xavier Garcia, Vedant Misra, Kevin Robinson, Liam Fedus, Denny Zhou, Daphne Ippolito, David Luan, Hyeontaek Lim, Barret Zoph, Alexander Spiridonov, Ryan Sepassi, David Dohan, Shivani Agrawal, Mark Omernick, Andrew M. Dai, Thanumalayan Sankaranarayana Pillai, Marie Pellat, Aitor Lewkowycz, Erica Moreira, Rewon Child, Oleksandr Polozov, Katherine Lee, Zongwei Zhou, Xuezhi Wang, Brennan Saeta, Mark Diaz, Orhan Firat, Michele Catasta, Jason Wei, Kathy Meier-Hellstern, Douglas Eck, Jeff Dean, Slav Petrov, and Noah Fiedel. 2022.  PaLM: Scaling Language Modeling with Pathways.  arXiv:2204.02311 [cs.CL]
  * Cruz et al. (2019) Luis Cruz, Rui Abreu, and David Lo. 2019.  To the attention of mobile software developers: guess what, test your app!  _Empirical Software Engineering_ 24, 4 (2019), 2438–2468.
  * Duda and Hart (1972) Richard O. Duda and Peter E. Hart. 1972.  Use of the Hough Transformation to Detect Lines and Curves in Pictures.  _Commun. ACM_ 15, 1 (jan 1972), 11–15.  https://doi.org/10.1145/361237.361242
  * Eler et al. (2018) Marcelo Medeiros Eler, José Miguel Rojas, Yan Ge, and Gordon Fraser. 2018.  Automated accessibility testing of mobile apps. In _2018 IEEE 11th International Conference on Software Testing, Verification and Validation (ICST)_. IEEE, 116–126.
  * Feng and Chen (2023) Sidong Feng and Chunyang Chen. 2023.  Prompting Is All You Need: Automated Android Bug Replay with Large Language Models.  arXiv:2306.01987 [cs.SE]
  * Fok et al. (2022) Raymond Fok, Mingyuan Zhong, Anne Spencer Ross, James Fogarty, and Jacob O Wobbrock. 2022.  A Large-Scale Longitudinal Analysis of Missing Label Accessibility Failures in Android Apps. In _Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems_. 1–16.
  * Guest et al. (2011) Greg Guest, Kathleen M MacQueen, and Emily E Namey. 2011\.  _Applied thematic analysis_.  sage publications.
  * Gur et al. (2023) Izzeddin Gur, Hiroki Furuta, Austin Huang, Mustafa Safdari, Yutaka Matsuo, Douglas Eck, and Aleksandra Faust. 2023.  A real-world webagent with planning, long context understanding, and program synthesis.  _arXiv preprint arXiv:2307.12856_ (2023).
  * Kochhar et al. (2015) Pavneet Singh Kochhar, Ferdian Thung, Nachiappan Nagappan, Thomas Zimmermann, and David Lo. 2015\.  Understanding the test automation culture of app developers. In _2015 IEEE 8th International Conference on Software Testing, Verification and Validation (ICST)_. IEEE, 1–10.
  * Kojima et al. (2022) Takeshi Kojima, Shixiang (Shane) Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. 2022\.  Large Language Models are Zero-Shot Reasoners. In _Advances in Neural Information Processing Systems_ , S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (Eds.), Vol. 35. Curran Associates, Inc., 22199–22213.  https://proceedings.neurips.cc/paper_files/paper/2022/file/8bb0d291acd4acf06ef112099c16f326-Paper-Conference.pdf
  * Lai et al. (2022) Vivian Lai, Samuel Carton, Rajat Bhatnagar, Q Vera Liao, Yunfeng Zhang, and Chenhao Tan. 2022\.  Human-ai collaboration via conditional delegation: A case study of content moderation. In _Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems_. 1–18.
  * Li et al. (2017) Xiao Li, Nana Chang, Yan Wang, Haohua Huang, Yu Pei, Linzhang Wang, and Xuandong Li. 2017.  ATOM: Automatic maintenance of GUI test scripts for evolving mobile applications. In _2017 IEEE International Conference on Software Testing, Verification and Validation (ICST)_. IEEE, 161–171.
  * Li et al. (2020) Yang Li, Jiacong He, Xin Zhou, Yuan Zhang, and Jason Baldridge. 2020.  Mapping Natural Language Instructions to Mobile UI Action Sequences. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_. Association for Computational Linguistics, Online, 8198–8210.  https://doi.org/10.18653/v1/2020.acl-main.729
  * Lin et al. (2020) Jun-Wei Lin, Navid Salehnamadi, and Sam Malek. 2020\.  Test automation in open-source android apps: A large-scale empirical study. In _Proceedings of the 35th IEEE/ACM International Conference on Automated Software Engineering_. 1078–1089.
  * Linares-Vásquez et al. (2017) Mario Linares-Vásquez, Carlos Bernal-Cárdenas, Kevin Moran, and Denys Poshyvanyk. 2017.  How do developers test android applications?. In _2017 IEEE International Conference on Software Maintenance and Evolution (ICSME)_. IEEE, 613–622.
  * Liu et al. (2023) Zhe Liu, Chunyang Chen, Junjie Wang, Mengzhuo Chen, Boyu Wu, Xing Che, Dandan Wang, and Qing Wang. 2023\.  Chatting with GPT-3 for Zero-Shot Human-Like Mobile Automated GUI Testing.  _arXiv preprint arXiv:2305.09434_ (2023).
  * Mankoff et al. (2005) Jennifer Mankoff, Holly Fait, and Tu Tran. 2005.  Is your web page accessible? A comparative study of methods for assessing web page accessibility for the blind. In _Proceedings of the SIGCHI conference on Human factors in computing systems_. 41–50.
  * OpenAI (2023) OpenAI. 2023.  GPT-4 Technical Report.  arXiv:2303.08774 [cs.CL]
  * Otsu (1979) Nobuyuki Otsu. 1979\.  A Threshold Selection Method from Gray-Level Histograms.  _IEEE Transactions on Systems, Man, and Cybernetics_ 9, 1 (1979), 62–66.  https://doi.org/10.1109/TSMC.1979.4310076
  * Pan et al. (2020) Minxue Pan, Tongtong Xu, Yu Pei, Zhong Li, Tian Zhang, and Xuandong Li. 2020\.  Gui-guided test script repair for mobile apps.  _IEEE Transactions on Software Engineering_ 48, 3 (2020), 910–929.
  * Patil et al. (2023) Shishir G. Patil, Tianjun Zhang, Xin Wang, and Joseph E. Gonzalez. 2023.  Gorilla: Large Language Model Connected with Massive APIs.  arXiv:2305.15334 [cs.CL]
  * Rawles et al. (2023) Christopher Rawles, Alice Li, Daniel Rodriguez, Oriana Riva, and Timothy Lillicrap. 2023.  Android in the Wild: A Large-Scale Dataset for Android Device Control.  arXiv:2307.10088 [cs.LG]
  * Ross et al. (2017) Anne Spencer Ross, Xiaoyi Zhang, James Fogarty, and Jacob O Wobbrock. 2017.  Epidemiology as a framework for large-scale mobile application accessibility assessment. In _Proceedings of the 19th international ACM SIGACCESS conference on computers and accessibility_. 2–11.
  * Salehnamadi et al. (2021) Navid Salehnamadi, Abdulaziz Alshayban, Jun-Wei Lin, Iftekhar Ahmed, Stacy Branham, and Sam Malek. 2021.  Latte: Use-case and assistive-service driven automated accessibility testing framework for android. In _Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems_. 1–11.
  * Salehnamadi et al. (2023) Navid Salehnamadi, Ziyao He, and Sam Malek. 2023.  Assistive-Technology Aided Manual Accessibility Testing in Mobile Apps, Powered by Record-and-Replay. In _Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems_. 1–20.
  * Salehnamadi et al. (2022) Navid Salehnamadi, Forough Mehralian, and Sam Malek. 2022\.  Groundhog: An Automated Accessibility Crawler for Mobile Apps. In _Proceedings of the 37th IEEE/ACM International Conference on Automated Software Engineering_. 1–12.
  * Schick et al. (2023) Timo Schick, Jane Dwivedi-Yu, Roberto Dessì, Roberta Raileanu, Maria Lomeli, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. 2023.  Toolformer: Language Models Can Teach Themselves to Use Tools.  arXiv:2302.04761 [cs.CL]
  * Shaw et al. (2023) Peter Shaw, Mandar Joshi, James Cohan, Jonathan Berant, Panupong Pasupat, Hexiang Hu, Urvashi Khandelwal, Kenton Lee, and Kristina Toutanova. 2023.  From Pixels to UI Actions: Learning to Follow Instructions via Graphical User Interfaces.  _arXiv preprint arXiv:2306.00245_ (2023).
  * Silva et al. (2018) Camila Silva, Marcelo Medeiros Eler, and Gordon Fraser. 2018.  A survey on the tool support for the automatic evaluation of mobile accessibility. In _Proceedings of the 8th International Conference on Software Development and Technologies for Enhancing Accessibility and Fighting Info-exclusion_. 286–293.
  * Touvron et al. (2023) Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023\.  LLaMA: Open and Efficient Foundation Language Models.  arXiv:2302.13971 [cs.CL]
  * Venkatesh et al. (2023) Sagar Gubbi Venkatesh, Partha Talukdar, and Srini Narayanan. 2023\.  UGIF: UI Grounded Instruction Following.  arXiv:2211.07615 [cs.CL]
  * Vigo et al. (2013) Markel Vigo, Justin Brown, and Vivienne Conway. 2013\.  Benchmarking web accessibility evaluation tools: measuring the harm of sole reliance on automated tests. In _Proceedings of the 10th international cross-disciplinary conference on web accessibility_. 1–10.
  * Wang et al. (2023) Bryan Wang, Gang Li, and Yang Li. 2023.  Enabling Conversational Interaction with Mobile UI Using Large Language Models. In _Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems_ (Hamburg, Germany) _(CHI ’23)_. Association for Computing Machinery, New York, NY, USA, Article 432, 17 pages.  https://doi.org/10.1145/3544548.3580895
  * Wen et al. (2023) Hao Wen, Yuanchun Li, Guohong Liu, Shanhui Zhao, Tao Yu, Toby Jia-Jun Li, Shiqi Jiang, Yunhao Liu, Yaqin Zhang, and Yunxin Liu. 2023\.  Empowering LLM to use Smartphone for Intelligent Task Automation.  _arXiv preprint arXiv:2308.15272_ (2023).
  * Yan and Ramachandran (2019) Shunguo Yan and PG Ramachandran. 2019.  The current status of accessibility in mobile apps.  _ACM Transactions on Accessible Computing (TACCESS)_ 12, 1 (2019), 1–31.
  * Zhang et al. (2021) Xiaoyi Zhang, Lilian de Greef, Amanda Swearngin, Samuel White, Kyle Murray, Lisa Yu, Qi Shan, Jeffrey Nichols, Jason Wu, Chris Fleizach, et al. 2021\.  Screen recognition: Creating accessibility metadata for mobile applications from pixels. In _Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems_. 1–15.
  * Zhang et al. (2023) Zhizheng Zhang, Xiaoyi Zhang, Wenxuan Xie, and Yan Lu. 2023\.  Responsible Task Automation: Empowering Large Language Models as Responsible Task Automators.  arXiv:2306.01242 [cs.AI]

##  Appendix A Appendix

###  A.1. User Study Interview Questions

For each video demo:

  * •

Please think out loud when you are going through the video.

  * •

Do you want to replay the video or scrub through certain parts? You can also
use the chapters to navigate

  * •

What is your first reaction after seeing the video? Did the test go as you
expected? Why or why not? Are there any parts that you found surprising? Are
there any steps that were missed or went wrong?

  * •

How similar or different is this from what you might do to test this manually?
On a scale of 1-5, rate how well this test video replicates how you might
perform this test manually. Please explain your answer (5 - Extremely good
match, 4 - Very good match, 3 - Moderately good match, 2 - Slightly matches, 1
- Not at all)

  * •

What do you think about the similarity/difference?

  * •

Did you find any issues or bugs in the video? How serious is this issue? If
this is a real issue you found, would you file an internal bug report for
this? Why or why not?

  * •

How did the heuristics for flagging issues assist with that? Please rate and
explain your answer. (5 - Very useful, 4 - Useful, 3 - Moderately useful, 2 -
Somewhat useful, 1 - Not useful at all)

  * •

How can the heuristics be improved?

General reflection:

  * •

How do you imagine using this system in your existing practice?

  * •

If the system was able to replicate these tests extremely well, how useful
might it be in your existing work? Please rate and explain your answer. (5 -
Very useful, 4 - Useful, 3 - Moderately useful, 2 - Somewhat useful, 1 - Not
useful at all)

  * •

How useful would the system be to your work in its current form today? Please
rate and explain your answer. (5 - Very useful, 4 - Useful, 3 - Moderately
useful, 2 - Somewhat useful, 1 - Not useful at all)

  * •

What drawbacks or benefits can you think of for this system?

  * •

What other accessibility test types would you like the system to support?

  * •

What other use cases related to accessibility can you imagine for this system?

  * •

What other features would you like the system to have?

###  A.2. Participants information

The profile of our participants are listed in Table 3.

Participant ID | Order of videos | Gender | Accessibility tasks in job duty | Years of experience
---|---|---|---|---
P1 | BS, DT, VO | M |  | Manual accessibility testing;
---
QA testing for accessibility;
Writing accessibility automation tests
3
P2 | DT, VO, BS | M |  | Designing or building accessibility features;
---
Manual accessibility testing;
QA testing for accessibility
4
P3 | BS, VO, DT |  |  |
P4 | VO, DT, BS | F | Writing accessibility automation tests | 6
P5 | BS, DT, VO | F |  | Designing or building accessibility features;
---
Manual accessibility testing
3
P6 | DT, BS, VO | F |  | Manual accessibility testing;
---
QA testing for accessibility
3
P7 | VO, BS, DT | M | Designing or building accessibility features | 6
P8 | DT, VO, BS | M |  | Designing or building accessibility features;
---
Manual accessibility testing;
QA testing for accessibility;
Writing accessibility automation tests
9
P9 | VO, BS, DT | M |  | Designing or building accessibility features;
---
Manual accessibility testing;
QA testing for accessibility
4
P10 | DT, BS, VO | F |  | Manual accessibility testing;
---
QA testing for accessibility;
Writing accessibility automation tests
2
Table 3. Participants in the user study.