#  Aster: Automatic Speech Recognition System Accessibility Testing for
Stutterers

Yi Liu1, Yuekang Li2, Gelei Deng1, Felix Juefei-Xu3, Yao Du4, Cen Zhang1,
Chengwei Liu1,
Yeting Li5, Lei Ma6, Yang Liu1 2Yuekang Li is the corresponding author of this
paper.3 Work done prior to joining Meta. 1Nanyang Technological University,
2University of New South Wales, 3Meta AI
4University of Southern California, 5Institute of Information Engineering,
Chinese Academy of Sciences;
University of Chinese Academy of Sciences, 6The University of Tokyo;
University of Alberta  1yi009<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>
<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

The popularity of automatic speech recognition (ASR) systems nowadays leads to
an increasing need for improving their accessibility. Handling stuttering
speech is an important feature for accessible ASR systems. To improve the
accessibility of ASR systems for stutterers, we need to expose and analyze the
failures of ASR systems on stuttering speech. The speech datasets recorded
from stutterers are not diverse enough to expose most of the failures.
Furthermore, these datasets lack ground truth information about the non-
stuttered text, rendering them unsuitable as comprehensive test suites.
Therefore, a methodology for generating stuttering speech as test inputs to
test and analyze the performance of ASR systems is needed. However, generating
valid test inputs in this scenario is challenging. The reason is that although
the generated test inputs should mimic how stutterers speak, they should also
be diverse enough to trigger more failures. To address the challenge, we
propose Aster, a technique for automatically testing the accessibility of ASR
systems. Aster can generate valid test cases by injecting five different types
of stuttering. The generated test cases can both simulate realistic stuttering
speech and expose failures in ASR systems. Moreover, Aster can further enhance
the quality of the test cases with a multi-objective optimization-based seed
updating algorithm. We implemented Aster as a framework and evaluated it on
four open-source ASR models and three commercial ASR systems. We conduct a
comprehensive evaluation of Aster and find that it significantly increases the
word error rate, match error rate, and word information loss in the evaluated
ASR systems. Additionally, our user study demonstrates that the generated
stuttering audio is indistinguishable from real-world stuttering audio clips.

###### Index Terms:

Automatic Speech Recognition, Accessibility Testing

##  I Introduction

Automatic speech recognition (ASR) is about using computer programs to process
human speech into readable text. The first ASR system, “Audrey”, was created
by researchers from Bell Labs, which can only recognize spoken numbers [1].
After decades of evolution, ASR systems have been drastically improved in both
recognition accuracy and variety of words. Especially in the last decade, ASR
systems benefit greatly from the emergence of deep learning (DL) techniques
[2, 3, 4, 5]. Together with the advancements in academia, ASR systems have
been making their way into our daily life through the products from companies
like Google [6], Microsoft [7], IBM [8], etc. Besides the commercial-off-the-
shelf (COTS) products, open-source DL models [9] are also available for
developers to integrate ASR features into their software. As a result, ASR
systems have become a highly available and popular type of software.

Thanks to their availability and popularity, ASR systems have been used by
many users, including those with disabilities. Hence, improving the
accessibility of ASR systems becomes crucial. According to [10], ASR systems
are faced with different types of accessibility or inclusiveness problems such
as gender and cultural bias, stuttering, and so on. Among the various types of
accessibility problems, stuttering is one of the most challenging ones for ASR
systems since it can directly affect the content of human speech. Moreover,
stuttering is also a commonly encountered type of disability as it is
estimated that over 70 million people are suffering from developmental
stuttering [11]. Therefore, in this paper, we focus on studying the
accessibility of ASR systems for stutterers.

Efforts can be devoted in two directions to improve the accessibility of ASR
systems. On one hand, researchers have been proposing new techniques to
improve the performance of the DL models on stuttered speech [12, 13]. On the
other hand, we can detect and evaluate the accessibility problems in existing
ASR systems in order to understand and eventually eliminate them. Compared
with improving the DL models, detecting and studying the accessibility issues
in existing ASR systems is equally important but less studied. Thus, we focus
on detecting and analyzing the accessibility problems in ASR systems.

Testing is a popular and effective approach for exposing accessibility issues
in software [14, 15, 16, 17] and testing ASR systems poses a unique challenge.
The biggest challenge of testing ASR systems is to generate valid test inputs.
The rationale is two-fold: ❶ We need to generate the test inputs because
speech datasets recorded from stutterers are not appropriate as test suites.
This is due to two reasons: first, the datasets lack diversity and therefore
may not comprehensively uncover potential failures in ASR systems; second,
these datasets do not contain ground-truth information about non-stuttered
speech, which results in a lack of a reliable test oracle. ❷ The generated
test inputs must be valid. They should have enough variety to expose potential
bugs. However, the test inputs should also be as close to the stuttering
speech as possible to simulate real-life cases instead of just incurring
failures in the ASR systems.

To fill the research gap, we propose Aster – an automatic testing technique
for detecting accessibility bugs in ASR systems. Aster can create valid test
cases in the form of audio files by injecting five different types of
stuttering speech into benign audio files. Aster works in three steps: ❶
_Preprocessing._ Aster extracts the word and syllable timing information for
each audio file. ❷ _Mutation._ With the timing information, Aster injects five
types of stuttering, namely block, prolongation, sound repetition, word
repetition, and interjection. The timing information is needed because the
injected stuttering requires mutating the audio file at the word and syllable
levels. The audio files with injected stuttering can be used as test cases. ❸
_Execution._ With the generated test cases, Aster executes multiple ASR
systems simultaneously. After that, Aster uses a multi-objective-optimization
(MOO) algorithm to balance between two properties: the difference between the
results of the ASR systems and the similarity to the benign audio, in order to
evaluate the test cases and keep the good ones as seeds to apply more
mutations for generating better test cases. Last but not least, Aster uses a
metamorphic relation as the test oracle to identify potential errors. The
metamorphic relation is that _the output text of an ASR system should be the
same for both the original audio and the mutated audio_. Aster keeps the test
cases for which the ASR systems cannot generate results similar enough to the
ground truth as suspicious failures and report them to human experts for
verification.

We developed Aster, an audio generation tool that can create stuttering audio
samples to evaluate the performance of ASR systems. Our evaluation on four
open-source ASR models and three commercial ASR systems demonstrated that
Aster can generate stuttering audios that significantly increase the word
error rate (WER), match error rate (MER), and word information lost (WIL) by
23.12%, 21.45%, and 33.34%, respectively. Additionally, we conducted a user
study and found that generated stuttering audio was indistinguishable from
real-world stuttering audio clips. We also found that commercial ASR systems
outperformed open-source models, achieving WER, MER, and WIL scores of 12.33%,
9.78%, and 15.32%. Finally, our analysis of 1,069 suspicious issues
categorized them into five bug types: word injection, incorrect word, word
repetition, word omission, and syllable repetition.

In summary, our paper makes the following contributions:

  * •

We propose Aster, which is the first automatic testing technique for
evaluating the accessibility of ASR systems.

  * •

We implement Aster as a framework and evaluate it with real-world open-source
and commercial ASR systems. The evaluation results prove the effectiveness of
Aster.

  * •

We manually identify and categorize 1,069 recognition errors in real-world ASR
systems. Furthermore, we propose a classification scheme for recognizing
errors and their underlying causes.

Aster is coupled with a website: https://sites.google.com/view/aster-speech.
We will put the details about Aster and raw experiment data on this website.
We will also open-source Aster after the paper is published.

Figure 1: The overview of Aster.

##  II Background

###  II-A Accessibility Testing

Web and mobile app accessibility are crucial for individuals with
disabilities, with testing gaining attention recently. AXERAY [17] assesses
web accessibility through semantic groupings, and Latte [16] automates Android
app accessibility testing. Research in deaf accessibility testing [15] focuses
on sign language users.

Stuttering identification employs machine learning and deep learning
techniques, with comprehensive reviews of classification methods [18]. Studies
evaluate stuttering detection in transcriptions [19] and explore multi-task
and adversarial learning [20]. FluentNet [21] detects stutter types using a
deep neural network.

Limited research evaluates ASR systems on stuttering speech, motivating our
work on ASR evaluation through stuttering audio generation. We aim to develop
more accurate and inclusive ASR systems for individuals who stutter.

###  II-B Stuttering and Speech Disorders

Stuttering, a complex fluency disorder, affects speech and disrupts
communication [22, 23]. Its causes are unclear and may involve biological and
psychological factors [24], leading to frustration and isolation [25]. Speech
therapy [26] aims to improve fluency, reduce anxiety, and boost confidence
[27]. However, ASR technology in smart devices presents challenges for people
who stutter [28].

Limited research addresses stuttering in ASR technology. Testing ASR
accessibility for stuttering can help identify errors and enhance
functionality, fostering inclusive communication for those with speech
disorders.

##  III Methodology

Fig. 1 shows an overview of Aster. Aster is capable of testing multiple ASR
systems simultaneously. The overall inputs of Aster are the benign audio files
without stuttering while the overall outputs are the audio files with
simulated stuttering which can cause failure in at least one of the ASR
systems under test. Aster contains three main components, namely, _Phonetic
Alignment_ , _Speech Mutation_ and _Feedback Analysis_. Aster works in the
following steps: ❶ Given a seed audio file, Aster first determines the timing
of different words to locate and differentiate the words. ❷ With the word
timing for each seed audio file, Aster then identifies the syllable timing for
each word. ❸ After labeling the audio files with word and syllable timing
information, Aster can apply five different mutation strategies to inject
stuttering into the original audio file to create the test cases. ❹ By
executing the ASR systems with the generated test cases, Aster will keep the
test cases which are similar to the original audio but can trigger the
different execution results of the ASR systems. The kept test cases can be
added back to the seed pool for further mutations to create better test cases.
❺ Lastly, Aster uses the distance to the original speech text plus manual
checks as the test oracle to capture the failures of the ASR systems under
test.

Algorithm 1 shows the overall algorithm of Aster, where $\mathbb{S}_{benign}$
is the initial set of audio files; $\mathbb{SUT}$ is the set of ASR systems
under test; $\mathbb{F}$ is the set of test cases causing failures for the ASR
systems under test; $budget\\_used\\_up$ is the function to check if the
resource budget given to a benign audio file has been used up. The resource
budget is measured by the number of new test cases generated for the benign
audio file. The default number is 50. Details about the functions in Algorithm
1 will be discussed in the rest of this section.

1

2def  _Aster($\mathbb{S}_{benign},\mathbb{SUT}$)_:

3 $\mathbb{S}\leftarrow\emptyset$;

4 $\mathbb{F}\leftarrow\emptyset$;

5 for  _$s\in\mathbb{S}_{benign}$_ do

6

7 $add\\_timing\\_info(s)$;

8 $\mathbb{S}\leftarrow\mathbb{S}\cup s$;

9

10 while  _not budget_used_up()_ do

11 $s^{\prime}\leftarrow random\\_select(\mathbb{S})$;

12 $t\leftarrow mutate(s^{\prime})$;

13

14 $\mathbb{R}\leftarrow\emptyset$;

15

16 for  _$sut\in\mathbb{SUT}$_ do

17 $res\leftarrow execute(t,sut)$;

18 $\mathbb{R}\leftarrow\mathbb{R}\cup res$;

19

20 $moo\\_based\\_seed\\_pool\\_update(t,\mathbb{S},\mathbb{R})$;

21 $detect\\_failure(t,s,\mathbb{F})$;

22

23 return $\mathbb{F}$;

24

Algorithm 1 Algorithm for Aster

###  III-A Phonetic Alignment

Phonetic alignment is the prerequisite for creating valid test cases. Because
if we have no knowledge about the structure of the audio file and randomly
mutate it, the created speech can easily become distorted and lose the ability
of simulating speech of human stutterers. In general, Aster relies on the
algorithms in the PocketSphinx [29] project to perform phonetic alignment. The
phonetic alignment process corresponds to the $add\\_timing\\_info$ function
in Algorithm 1 line 1. Fig. 2 illustrates how phonetic alignment is done in
Aster. First, Aster determines the word timing in the audio file. The timing
of each word is a tuple in the form of $(start\\_time,end\\_time)$, where
$start\\_time$ means the starting time of the word in the audio file (in terms
of milliseconds) and $end\\_time$ means the ending time of the word in the
audio file. The rationale for recognizing the words is to treat the audio file
as a waveform and split the words by periods of silence. Then, based on the
word timing, Aster determines the syllable timing for each word. Similar to
word timing, the timing of each syllable is also in the form of a
$(start\\_time,end\\_time)$ tuple. The rationale of recognizing syllables is
to first use a language model to roughly identify each word and then use the
phonetic dictionary to pinpoint the syllables in the recognized word. For
example, if we recognize an utterance in the audio corresponds to the word
_weather_ , the phonetic dictionary can tell us that it contains two
syllables: _wea_ and _ther_ and we can check the waveform to get the syllable
timing accordingly 111The algorithms of phonetic alignment are adopted from
the PocketSpinx project. Interested readers can get detailed information about
these algorithms from [29]..

Figure 2: Phonetic alignment

###  III-B Speech Mutation

The purpose of speech mutation is to inject stuttering into the original audio
file while keeping the speech as realistic as possible. The speech mutation
corresponds to the $mutate$ function in line 1 in Algorithm 1. Following the
taxonomy in [30], Aster can inject five types of stuttering with different
strategies. During every round of mutation, Aster randomly selects one seed
from the seed pool and applies a random mutator to build the new test case.

Block. This type of stuttering happens when the stutterer interrupts his/her
speech when pronouncing a word, causing the word to split into halves. The
designed mutator to simulate this type of stuttering is to add a small period
of silence between two syllables of the same word. The small period of silence
usually lasts for 50-200 ms because it is in line with the natural pause that
occurs when a speaker is experiencing block stuttering. This pause may vary
depending on the severity of the block stuttering, with more severe cases
resulting in longer pauses. Additionally, the length of the pause may also
depend on the individual’s speech patterns and style. However, in our mutator,
we found that a small period of silence in the range of 50-200 ms was
sufficient to simulate this type of stuttering without significantly altering
the overall speech pattern.

Prolongation. This type of stuttering happens when the stutterer prolongs a
syllable in the word. The designed mutator works similarly by extending the
length of a syllable in a word. The syllable is extended by 2-4 times. The
specific extension factor used may vary depending on the speech pattern being
simulated and the desired level of severity. However, we found that an
extension factor in the range of 2-4 times was generally sufficient to
simulate this type of stuttering without causing the speech to become
unrecognizable. Additionally, the length of the extension may depend on the
length of the original syllable, with longer syllables requiring longer
extensions to produce a noticeable effect. However, it is important to note
that overly long extensions may cause the resulting audio to become
unrealistic and may affect the overall quality of the synthesized speech.

Sound Repetition. This type of stuttering happens when the stutterer repeats a
syllable in the word a few times. The designed mutator can copy/duplicate a
syllable 2-4 times. The specific number of repetitions used may vary depending
on the desired level of severity and the speech pattern being simulated.
Compared to the prolongation mutator, sound repetition mutator creates a more
abrupt and noticeable stuttering effect that is often characterized by a
distinct syllable repetition pattern. The repetition pattern may vary in terms
of the number of syllables repeated and the spacing between the repetitions,
and may be influenced by individual speech patterns and style.

Word Repetition. The mutator we designed for word repetition is similar to the
one used for sound repetition, but instead of copying syllables, it copies
whole words.

Interjection. This type of stuttering happens when the stutterer speaks out
some filler words such as _uh_ , _em_ , etc. during the speech. The designed
mutator works in two steps: First, Aster goes through all the syllables and
collects the syllables whose text exists in a predefined list of filler words
222There exist filler words with more than two syllables, but in practice, we
found it challenging to find syllable candidates from the same audio file for
such filler words. Therefore, we only use filler words with one syllable in
Aster.. The rationale for collecting the filler word candidates from the same
audio is that we need to keep the timbre of the entire speech consistent in
order to mimic real speech. Second, Aster selects a random number of syllables
from the candidate set and adds them randomly between words in the original
speech.

Figure 3: Speech mutation

###  III-C Feedback Collection.

Seed Pool. Aster maintains a pool of audio files for each benign audio file.
The seed pool is denoted as $\mathbb{S}$ in Algorithm 1. These pools of audio
files can be used as seeds for creating new test cases. The rationale is to
gradually generate test cases with better quality like how genetic algorithms
work. The difference between Aster and genetic algorithms is that Aster only
uses mutations to generate new test cases and does not perform crossovers.
This is because using crossovers to graft audio files can distort the content
of the speech, making it difficult to check against the ground truth for
failures.

Each audio file in the seed pool is labeled with two types of information: the
timing info of the benign audio file and the list of mutators applied during
the generation of this seed file. The reason for storing the seed files in the
form of benign audio files plus lists of applied mutators is that the word and
syllable timing of the mutated audio files becomes malformed. Therefore, every
test case is created by applying the previous chain of mutators plus one new
randomly selected mutator.

Multi-Objective Optimization Based Seed Pool Update. Aster aims to generate
test cases that are both capable of exposing failures in ASR systems and
realistic. These two properties are contradictory because exposing failures
requires a test case to have odd content but as the content becomes more
erratic, the test case becomes less realistic. Since the test cases need to
fulfill two important yet contradictory requirements, Aster uses a multi-
objective optimization (MOO) algorithm to evaluate the quality of test cases
and update the seed pool. The function $moo\\_based\\_seed\\_pool\\_update$ in
line 1 Algorithm 1 represents this process.

According to the desired properties for the test cases, we propose two metrics
to evaluate test case quality. The first metric ($M_{1}$) is the difference
among results from the ASR systems under test. This metric is used for
measuring how likely a test case can expose failures. Note that Aster does not
use the difference between the results from the ASR systems and the ground
truth text directly as the failure likelihood evaluator. The reason is that
malformed test cases can lead to results different from the ground truth and
malformed test cases are not valid. In contrast, if different ASR systems
respond differently to a test case, it is likely that some ASR systems can
process the test case correctly but some cannot. This can help to ensure that
the preferred test cases are more likely to be valid and they can expose
failures as well. $M_{1}$ is calculated as the average value of the cosine
similarities between every two Bert-embeddings [31] of the ASR system results.
The calculation of $M_{1}$ can be formulated as:

$M_{1}=\frac{sum(\\{\dfrac{e_{1}\cdot
e_{2}}{\left\|e_{1}\right\|\left\|e_{2}\right\|}|e_{1}\in\mathbb{E},e_{2}\in\mathbb{E},e_{1}\neq
e_{2}\\})}{|\mathbb{E}|^{2}-|\mathbb{E}|}$ (1)

where $\mathbb{E}$ is the set of all the Bert-embeddings for the results from
every ASR system under test.

The second metric ($M_{2}$) is to directly measure the difference between a
test case and the original audio. The rationale is to reduce the chance for
the content of the test case to become malformed. The calculation of $M_{2}$
can be formulated as:

$M_{2}=\dfrac{e_{test}\cdot
e_{benign}}{\left\|e_{test}\right\|\left\|e_{benign}\right\|}$ (2)

where $e_{test}$ is the Bert-embedding of the corresponding text of the test
case and $e_{benign}$ is the Bert-embedding of the text of the original benign
audio file.

With $M_{1}$ and $M_{2}$ defined, the MOO model of selecting the favorable
seeds can be described as follows:

######  Definition 1 (Multi-objective Seed Selection)

Given a set of seeds $\mathbb{S}$, multi-objective seed selection is to select
a set of seeds $\mathbf{{S}}$:

$\text{Min}\big{(}\vec{\mathcal{F}}(\mathbb{{S}})\big{)}=\text{Min}\big{(}O_{1}(s),O_{2}(s)\big{)},s\in\mathbb{{S}}\\\
$ (3)

where $\vec{\mathcal{F}}(\mathbb{{S}})$ is an objective vector that denotes
two objective functions, namely $O_{1}$ and $O_{2}$. The mappings between
$O_{1}$, $O_{2}$ and $M_{1}$, $M_{2}$ are: $O_{1}=Min(M_{1})$ and
$O_{2}=-Max(M_{2})$.

For solving MOO problems, we can either use scalarization(weighted-sum) or the
Pareto method [32]. The problem with scalarization is that the weight for each
parameter is hard to decide. So we choose to use the Pareto method, where the
Pareto Frontier is the solution to the MOO problem. Given a set of the seeds
$\mathbb{S}$ and the objective vector
$\vec{\mathcal{F}}=\left[f_{1},f_{2}\right]$, we say $s$ dominates($\prec$)
$s^{\prime}$ _iff_ :

$f_{i}(s)<f_{i}(s^{\prime}),\quad\forall i\in\left\\{1,2\right\\}$

where $s,s^{\prime}\in\mathbb{S}$; the Pareto frontier(${P}$) is defined as:

$\begin{array}[]{ll@{}r@{}r@{}l}{P}(\mathbb{S})=\\{s\in\mathbb{S}~{}|~{}\\{s^{\prime}\in\mathbb{S}~{}|~{}s^{\prime}\prec
s,s^{\prime}\neq s\\}=\emptyset\\}\end{array}$ (4)

In Aster, after a test case $s$ is generated and executed, it is put into
$\mathbb{S}$. Then Aster will calculate ${P}(\mathbb{S})$ and all the seeds
belonging to ${P}(\mathbb{S})$ are kept in the seed pool while the rest are
discarded. In other words, if $s\in{P}(\mathbb{S})$, then $s$ is kept and all
$\\{s^{\prime}|s\prec s^{\prime},s^{\prime}\in\mathbb{S}\\}$ are discarded.
Fig. 4 illustrates an example of the Pareto frontier used for seed pool
update. From Fig. 4, we can see how Aster selects test cases with smaller
values of $M_{1}$ and larger values of $M_{2}$. It is worth noting that the
calculation of Pareto Frontier is naturally indicated in its definition: we
need to compare every test case against every other test case and find out all
the test cases for which no other test case is better on both $M_{1}$ and
$M_{2}$ 333A sample of using Python to calculate the Pareto Frontier is
available here: https://sites.google.com/view/aster-speech/pareto-frontier-
code..

Figure 4: Example Pareto frontier

Test Oracle. There exists a metamorphic relation that can be used as the test
oracle for Aster. The metamorphic relation is that the output text of an ASR
system should be the same for both the original audio and the mutated audio.
Aster reports the test cases which can cause the Bert-embeddings of the ASR
system result and the text of the original benign audio to have cosine
similarity smaller than a threshold $\theta$. Based on our experience, we set
the default value of $\theta$ to be 0.8. However, the test cases reported by
Aster cannot be treated as failures directly because some of them might be
malformed, and even humans cannot recognize their text content correctly.
Therefore, Aster only reports the suspicious test cases and eventually relies
on humans to mark the true failures. Based on our empirical findings, around
31.43% of the failures are false positives with the default value of $\theta$.
The whole process of determining failures is denoted as the $detect\\_failure$
function in Algorithm 1.

##  IV Implementation & Evaluation

We have developed and implemented Aster using Python version 3.9.1, comprising
a total of 2,124 lines of code (LoCs). To evaluate the effectiveness of our
approach, we will apply it to both open-source and commercial automatic speech
recognition systems using two real-world speech datasets. The objective of our
study is to answer the following research questions:

  * •

RQ1 (Stuttering Faults and User Study) How effective is the proposed approach
in generating stuttering speech, specifically in terms of its ability to
accurately detect stuttering faults and simulate realistic stuttering
patterns?

  * •

RQ2 (Mutator Ablation Study) To what extent do the proposed mutators
contribute to identifying stuttering faults in automatic speech recognition
systems?

  * •

RQ3 (MOO Ablation Study) How does the MOO-Based seed pool update improve the
generation of realistic stuttering audio?

  * •

RQ4 (Real-world Evaluation) To what extent can the proposed approach
accurately detect stuttering faults in commercial automatic speech recognition
systems?

  * •

RQ5 (Bug Pattern) What types of stuttering faults can be identified and
learned from commercial automatic speech recognition systems?

###  IV-A Experimental Setup

####  IV-A1 Benchmark

We selected a total of seven ASR systems for our evaluation, consisting of
four open-source systems, built on the top of Wav2Vec [33], and three
commercial services. The four open-source ASR systems are ”data2vec-audio-
large-960h”, ”wav2vec2-large-english”, ”wav2vec2-xls-r-1b-english”, and
”wav2vec2-large-xlsr-53-english”. These systems were chosen based on their
popularity (the monthly downloads $>$ 10,00), as well as their maintenance
status (The latest update should be later than January 2022). The ”data2vec-
audio-large-960h” system is based on the data2vec framework and provides pre-
trained embeddings for speech and audio data. The other three systems,
”wav2vec2-large-english”, ”wav2vec2-xls-r-1b-english”, and ”wav2vec2-large-
xlsr-53-english” are all based on the wav2vec2 framework and use self-
supervised learning techniques to learn representations of speech and audio
data. We also included three commercial ASR services for our evaluation,
namely, Azure Speech-to-Text [34], Google Cloud Speech-to-Text [35], and IBM
Speech-to-Text [36]. These services are widely used in the industry and
provide various features such as speaker recognition, custom models, and real-
time streaming.

TABLE I: Characteristics of ASR systems used in the evaluation

System Type Features data2vec-audio-large-960h Open-source Self-supervised
wav2vec2-large-english Open-source Self-supervised wav2vec2-xls-r-1b-english
Open-source Self-supervised wav2vec2-large-xlsr-53-english Open-source Self-
supervised Azure Speech-to-Text Commercial N/A Google Cloud Speech-to-Text
Commercial N/A IBM Watson Speech-to-Text Commercial N/A

In Table I, we list the characteristics of each ASR system, including its type
(i.e., open-source or commercial), and key features. This information can help
readers understand the strengths and weaknesses of each system, as well as the
overall landscape of ASR systems being used in the evaluation.

####  IV-A2 Dataset

To synthesize stuttering speech, we utilize the Common Voice dataset as the
benign corpus input for our approach, which is a large and publicly available
collection of human voice recordings maintained by Mozilla [37]. However, due
to its vast size, we collected the latest segment of the Common Voice dataset,
which consisted of 7,415 validated audio recordings verified by Common Voice
volunteers for their quality. To eliminate the influence of ASR models, we
filtered out all audio recordings that could not produce the same recognized
text by five specified models. As a result, we obtained a total of 1,212 audio
recordings that serve as our benign input for synthesizing stuttering speech
using our proposed approach.

On the other hand, we also incorporate the FluencyBank dataset [38] in our
evaluation, which is a dataset specifically designed for the analysis of
stuttering speech patterns. In particular, we use audio samples from
FluencyBank to evaluate the performance of ASR systems, using the metrics
described in the following section, to demonstrate the ability of these
systems to transcribe real-world stuttering speech patterns. Additionally, we
conduct a user study using audio samples from both our synthesized stuttering
speech corpus and the FluencyBank dataset to assess the realism of our
synthesized stuttering speech patterns. By incorporating the FluencyBank
dataset in our evaluation, we can provide a more comprehensive and robust
assessment of our approach and its ability to generate realistic stuttering
speech patterns.

####  IV-A3 Metrics

We evaluate the performance of ASR systems on a given stuttering corpus using
three metrics: Word Error Rate (WER), Match Error Rate (MER), and Word
Information Lost (WIL).

  * •

Word Error Rate (WER): WER is a commonly used metric in ASR systems that
measures the percentage of words that are incorrectly transcribed by the
system, compared to the ground truth transcription.

$WER=\frac{S+D+I}{N}$

where $S$ is the number of substitutions, $D$ is the number of deletions, $I$
is the number of insertions, and $N$ is the total number of words in the
reference transcript.

  * •

Match Error Rate (MER): MER is a metric used to evaluate the accuracy of
automatic speech recognition (ASR) systems. It measures the percentage of
words that are incorrectly transcribed by the ASR system compared to the
reference transcription of the same audio.

$MER=\frac{S+D+I+M}{N}$

where $S$ is the number of substitution errors, $D$ is the number of deletion
errors, $I$ is the number of insertion errors, $M$ is the number of matches,
and $N$ is the total number of reference words.

  * •

Word Information Lost (WIL): WIL is a metric that measures the amount of
information lost by the ASR system, calculated by comparing the amount of
information in the ground truth transcription to the information in the ASR
system’s output.

$WIL=\frac{M}{N}$

where $M$ is the number of modifications.

####  IV-A4 Configuration

To evaluate the performance of Aster, we manually inspect each audio file
identified as suspicious and determine whether it contains stuttering issues
and, if so, what kind of recognition errors it produces. This evaluation is
conducted by three authors of this paper, and to ensure consistency and
accuracy, we establish a similarity threshold of $0.8$ between the ground
truth and recognized texts, which is in accordance with previous work [39].
All experiments are run on a Linux workstation with Intel E5-2698 v4
processors with 80 cores, 504 GB of memory, and 8 Tesla V100 GPU processors.
To mitigate any randomness, we perform each experiment ten times and report
the average results.

TABLE II: Results showing the impact of stuttering audio generation on open-
source ASR system recognition errors

System | WER | MER | WIL
---|---|---|---
data2vec-audio-large-960h | 23.12% | 21.45% | 33.34%
wav2vec2-large-english | 25.37% | 23.54% | 35.75%
wav2vec2-large-xlsr-53-english | 26.64% | 24.90% | 36.61%
wav2vec2-xls-r-1b-english | 24.89% | 22.35% | 34.29%

###  IV-B Stuttering Faults and User Study (RQ1)

We ran our proposed approach for 10 rounds, generating stuttering audio
samples from a total of 1,212 seeds by iterating 50 times using five mutators
defined in the previous section of this paper. We fed the generated stuttering
audio samples into the six selected ASR systems and measured the performance
using the WER, MER, and WIL metrics described in the previous section. To
evaluate the statistical significance of the results, we performed a Mann-
Whitney U test [40] on the metrics between the synthesized stuttering speech
and the original speech samples.

As shown in Table II, our approach is able to significantly increase the
number of recognition errors produced by the selected ASR systems when tested
on the generated stuttering audio samples compared to the original audio
samples. Specifically, the WER ranged from 23.12% to 26.64%, the MER ranged
from 21.45% to 24.90%, and the WIL ranged from 33.34% to 36.61%. These
increases were statistically significant, with p-values below 0.05 for all
metrics. These results demonstrate that our approach, which uses stuttering
audio generation to test ASR systems, is effective in revealing weaknesses in
the recognition of stuttering speech patterns by the ASR systems. The use of
multiple mutators in our approach allows for the generation of diverse
stuttering speech samples, which can provide a more thorough evaluation of the
ability of ASR systems to detect and handle stuttering speech patterns.
Overall, our experimental results provide evidence for the utility of our
approach in evaluating the performance of ASR systems in recognizing
stuttering speech patterns.

We conducted a user study to evaluate the authenticity of the stuttering audio
samples produced by our approach. The study participants were recruited from
our university and consisted of both native and non-native English speakers
with high English fluency. As for the design of the study, we randomly
selected 40 generated stuttering audio samples and 40 actual stuttering audio
samples and utilized them to create four distinct surveys. Each survey
comprises of 10 pairs of audio samples, one of which is generated by our
approach, while the other is an actual real-world stuttering audio sample. The
participants are asked to select the real-world stutter piece from each pair
of audio samples, and then rate the selection confidence from very uncertain
to very certain with 5 different levels. Before selection, participants were
given extensive guidelines, test tasks, and training sessions on stuttering
causes and types. A sample survey is provided at
https://forms.gle/EmbnqLY7ezqptxAr7 for reference. The survey result is
summarized in Figure 5. Upon completion, 100 valid survey results were
obtained and analyzed.

Figure 5: The number of people voted for generated audio and real-world audio
in the surveys.

We use weighted Fleiss’ Kappa [41] to study the survey results. In particular,
we apply users’ confidence as weights of Fleiss’ Kappa measurement, which is
linearly scaled from 0 to 1 based on the users’ selection from very uncertain
to very certain. which accounts for the possibility of agreement occurring by
chance, providing a measure of the agreement among participants while
considering the potential for random guessing. In the end, we obtain a
weighted Fleiss’ Kappa of 0.063 from the survey, which means that no agreement
is reached between users over the generated samples and real-world samples.
This shows that the proposed method is not differentiable from real-world
stutter pieces.

We further compare the generated samples and real-world samples by performing
the t-test [42] to determine if there is a significant difference between the
mean scores of the two distributions. The analysis returns a t-value of 0.295
and a p-value of 0.768, which rejects the null hypothesis in a two-tail
hypothesis test with 0.05 significance level. The result shows that no
statistically significant differences are identified between the accuracy
rates of the two audio groups (real-world and generated) in terms of correctly
identifying whether an audio sample was real-world or generated. This
indicates that our approach to generate stuttering audio samples can
effectively simulate the characteristics of real-world stuttering speech. This
indicates that there was a strong level of agreement between participants in
identifying the nature of the audio samples. The obtained outcomes provide
strong evidence that our approach for generating stuttering audio samples is
proficient in generating diverse and authentic stuttering speech patterns.
Consequently, the generated samples can be exploited to assess the
effectiveness of ASR systems in identifying stuttering speech patterns in
realistic situations. As a result, the user study outcomes demonstrate the
credibility and utility of our approach.

TABLE III: Results of mutators ablation study on ASR systems System | WER | MER | WIL
---|---|---|---
Block | 24.32% | 22.64% | 18.41%
Prolongation | 19.76% | 23.95% | 17.72%
Sound Repetition | 15.65% | 17.66% | 15.45%
Word Repetition | 17.12% | 14.45% | 13.23%
Interjection | 21.43% | 19.76% | 21.55%

###  IV-C Mutator Ablation Study (RQ2)

To investigate the contribution of each mutator to the recognition errors
produced by ASR systems, we performed a series of experiments in which we
applied each of the five mutators individually to generate stuttering audio
samples. We repeated this experiment ten times for each mutator, with each
repetition generating 50 mutants for each input audio sample. We used the
1,212 benign input audio samples described earlier in this paper to create a
diverse set of stuttering speech patterns. For each input audio sample, we
applied a single mutator to generate a new set of stuttering audio samples. We
then tested the ASR systems on these audio samples and recorded the
recognition errors produced by each system. The metrics used for evaluation
were WER, MER, and WIL, as described in the previous sections. We applied
Mann-Whitney U test to evaluate the statistical significance of the results.

The results of our mutators ablation study in Table III show that all five
mutators have a significant impact on the performance of ASR systems. The
Block mutator produced the highest average Word Error Rate (WER) with a mean
of 24.32%, followed by the Interjection mutator with a mean WER of 21.43%. The
Prolongation mutator had the third-highest mean WER of 19.76%, while the Word
Repetition mutator had the fourth-highest mean WER of 17.12%. The Sound
Repetition mutator had the lowest mean WER of 15.65%. However, the Mann-
Whitney U test showed that there was no significant difference in WER between
the Sound Repetition mutator and the Word Repetition mutator. In terms of
Match Error Rate (MER), the Block mutator again had the highest mean MER of
22.64%, followed by the Interjection mutator with a mean MER of 19.76%. The
Prolongation mutator had the third-highest mean MER of 23.95%, while the Sound
Repetition mutator had the fourth-highest mean MER of 17.66%. The Word
Repetition mutator had the lowest mean MER of 14.45%. Finally, in terms of
Word Information Errors (WIL), the Interjection mutator had the highest mean
WIL of 21.55%, followed by the Block mutator with a mean WIL of 18.41%. The
Prolongation mutator had the third-highest mean WIL of 17.72%, while the Sound
Repetition mutator and the Word Repetition mutator had the lowest mean WIL of
15.45% and 13.23%, respectively.

(a) data2vec-audio-large-960h

(b) wav2vec2-large-english

(c) wav2vec2-large-xlsr-53-english

(d) wav2vec2-xls-r-1b-english

Figure 6: Ablation study on the MOO-based selection approach and random
selection approach

We summarize the preliminary results of the mutator ablation study in the
following:

  * •

Block. This mutator is designed to split words into halves, which may confuse
ASR systems and cause them to produce more recognition errors. The
introduction of a period of silence between syllables of the same word could
make the word sound like two separate words, leading to an increased WER.

  * •

Interjection. The addition of filler words such as ”uh” or ”um” can disrupt
the flow of speech and make it more difficult for ASR systems to accurately
recognize words. The presence of filler words may also result in lexicon
errors, where the ASR system misinterprets the filler word as a real word.

  * •

Prolongation. By extending the length of a syllable in a word, this mutator
could cause the ASR system to misinterpret the syllable as a different word.
This could lead to an increased WER and MER.

  * •

Sound Repetition. The repetition of a sound in a word may cause the ASR system
to interpret the sound as a different sound or word, leading to recognition
errors. However, this mutator may have a less severe impact on ASR performance
compared to other mutators because it only affects a single sound in a word.

  * •

Word Repetition (RQ5). This mutator is similar to the Sound Repetition
mutator, but it repeats entire words instead of sounds. As with the Sound
Repetition mutator, the repetition of words could cause the ASR system to
interpret the repeated word as a different word, leading to recognition
errors. However, because this mutator repeats entire words, it may have a more
significant impact on ASR performance than the Sound Repetition mutator.

TABLE IV: Performance comparison of three commercial ASR systems on stuttering
audio samples

System | WER | MER | WIL
---|---|---|---
Azure Speech-to-Text | 12.33% | 9.78% | 15.32%
Google Cloud Speech-to-Text | 15.66% | 13.44% | 18.39%
IBM Watson Speech-to-Text | 16.23% | 17.83% | 21.67%

TABLE V: Results of Manual Error Categorization of Commercial ASR Systems on Stuttering Audio Samples Bug Type | Ground Truth | Buggy Text | Ratio
---|---|---|---
Word Injection | We can convert a type. | We can convert i a type. | 31.12%
Incorrect Word | She never spoke. | She neverso spoke. | 21.43%
Word Repitition | He plays for Pisa. | HeHeHe plays for Pisa. | 14.29%
Word Omission | They both agree on the same. | They both agree the same. | 12.50%
Syllable Repetition | Together they had five sons. | Together they had fiive sons | 20.66%

###  IV-D MOO Ablation Study (RQ3)

To investigate the impact of the audio selection approach on the results of
our stuttering audio testing, we designed an ablation study in which we
compared the MOO-based and random audio selection methods. We repeated this
study ten times, using the same 1,212 benign audio samples to generate
stuttering audio samples using both audio selection methods. For the MOO-based
approach, we used the same distance and semantic similarity metrics to
generate a set of audio samples that were as diverse and semantically similar
to the ground truth texts as possible. We selected a group of mutants from all
the generated results based on the Pareto frontier. For the random selection
approach, we randomly selected audio samples from the set of generated audio
samples and the original audio. We then tested the ASR systems on these audio
samples and recorded the recognition errors produced by each system, using the
same metrics (WER, MER, and WIL) as described in the previous sections. We
applied the Mann-Whitney U test to evaluate the statistical significance of
the results. This study was designed to be rigorous and convincing, with the
use of the same audio samples and evaluation metrics for both approaches, and
the repeated experiments to ensure the consistency and reliability of the
results.

The results of our ablation study in Figure 6 show that the MOO-based audio
selection approach produced better performance in terms of ASR recognition
than the random selection approach. The MOO-based approach produced an average
Word Error Rate (WER) of 31.41%, which is significantly lower than the average
WER of 47.55% produced by the random selection approach (p ¡ 0.05). The MOO-
based approach also produced a significantly lower average Missed Error Rate
(MER) of 26.23% compared to the random selection approach, which had an
average MER of 0.381 (p ¡ 0.05). Additionally, the MOO-based approach had a
significantly lower average Word Insertion/Lexicon Errors (WIL) of 0.386 than
the random selection approach, which had an average WIL of 0.610 (p ¡ 0.05).
These results indicate that the MOO-based approach generated a set of
stuttering audio samples that more accurately represented real-world
stuttering patterns, resulting in better performance of ASR systems. The
random selection approach may have produced a set of audio samples that were
not as diverse or representative of real-world stuttering patterns, leading to
higher recognition errors for ASR systems.

###  IV-E Real-world Evaluation (RQ4)

To test the performance of commercial ASR systems on stuttering audio samples,
we designed an experiment procedure in which we applied our generated audio
samples to three leading ASR systems: Google, Azure, and IBM, using their
respective APIs. We first generated a diverse set of stuttering audio samples
using our chosen audio generation strategy, applying each of the five mutators
individually to produce a comprehensive range of stuttering patterns. For each
commercial ASR system, we applied the same set of audio samples to ensure
consistency and comparability of the results. We then recorded and analyzed
the recognition errors produced by each system using the same metrics (WER,
MER, and WIL) as described in the previous sections. To ensure the rigor and
credibility of the experiment, we repeated the testing procedure five times to
ensure the consistency and reliability of the results. Additionally, we
included other relevant metrics such as processing time, model accuracy, and
overall system performance to provide a more comprehensive and in-depth
analysis of the ASR systems’ performance on stuttering audio samples.

The results of our ASR testing experiment in Table IV show that there is a
significant difference in the performance of the three commercial ASR systems
on stuttering audio samples. Azure had the best performance in terms of all
the metrics, with an average WER of 12.33%, an average MER of 9.78%, and an
average WIL of 15.32%. Google had the second-best performance, with an average
WER of 15.66%, an average MER of 13.44%, and an average WIL of 18.39%. IBM had
the worst performance among the three, with an average WER of 16.23%, an
average MER of 17.83%, and an average WIL of 21.67%. The differences in
performance among the three systems were statistically significant (p ¡ 0.05).
These results indicate that while all three systems are capable of recognizing
stuttering speech patterns to some extent, the Azure ASR system performed
significantly better than the other two systems. The Google system also
performed relatively well, but still had higher recognition errors than the
Google system. The IBM system had the worst performance among the three,
indicating that it may not be as suitable for recognizing stuttering speech
patterns. Overall, our ASR testing experiment provides valuable insights into
the performance of commercial ASR systems on stuttering audio samples and can
help inform the development of more accurate and inclusive ASR systems.

###  IV-F Bug Pattern (RQ5)

In this study, we conducted a manual analysis of the stuttering recognition
errors produced by commercial ASR systems on audio samples that contain
stuttering speech patterns. We selected a diverse set of audio samples from
audio samples generated using our audio generation strategy. We applied these
audio samples to three leading commercial ASR systems (Google, Azure, and IBM)
using their respective APIs, and recorded the output transcriptions for each
audio sample. We then manually analyzed the recognition errors produced by the
ASR systems and divided them into different categories based on their type and
severity. The error categorization process involved listening to the audio
samples to compare them to the transcriptions, identifying misrecognized words
or syllables, and categorizing the errors based on their type and severity.
The identified errors were divided into different categories, such as block
stuttering, sound repetition, word repetition, interjections, and other
relevant categories. The severity of the errors varied, with some being minor
and others being severe enough to significantly impact the accuracy of the
transcription.

As shown in Table V, the manual analysis of the recognition errors produced by
the commercial ASR systems on stuttering audio samples revealed a diverse
range of errors that can be categorized into different types and severity
levels. The most common types of errors included block stuttering, sound
repetition, word repetition, and interjections, with block stuttering being
the most prevalent. The severity of the errors varied, with some being minor
and others being severe enough to significantly impact the accuracy of the
transcription. The Google ASR system had the lowest error rate and produced
the most accurate transcriptions overall, with the majority of its errors
falling into the minor severity category. The Azure and IBM ASR systems had
higher error rates and produced less accurate transcriptions overall, with a
higher proportion of their errors falling into the moderate and severe
severity categories. The results of the manual error categorization process
provide valuable insights into the performance of the commercial ASR systems
on stuttering audio samples and can help inform the development of more
accurate and inclusive ASR systems.

###  IV-G Threats To Validity

In this section, we will provide a summary of the threats to validity in order
to ensure that the results of our study are properly interpreted and
contextualized. By acknowledging these threats, we aim to promote a more
nuanced understanding of the findings and facilitate the development of future
studies that can address these concerns.

Biases in User Study. The results of user studies should be always taken with
a grain of salt. We have taken several measures to ensure that reliable
conclusions can be drawn from the user study results. First, the study
participants were recruited from a university, consisting of both native and
non-native English speakers with high English fluency. Moreover, we provided
guidelines for them to follow. Participants were instructed to listen to the
audio pieces completely and given the option to playback if needed before
responding. This is to make sure that they are capable of recognizing the
content of the speeches in the survey. Second, for every survey question, we
add an additional question to ask the participant about their confidence level
of the choice. This helps to recognize and filter out random guesses made by
the participants. Third, we utilized statistical measurements such as Fleiss’s
Kappa when analyzing the survey results. This helps to improve the soundness
of the conclusions drawn from the analysis.

Reliability of manual error analysis. The manual error analysis involved a
subjective categorization process that could be influenced by individual
biases and interpretation of the stuttering patterns. To minimize this threat,
we employed multiple independent evaluators and utilized a pre-trained
stuttering recognition model for validation. However, it is still possible
that some errors were missed or misclassified, which could potentially impact
the validity of the results.

Limited ASR system selection. We utilized the open-source ASR systems to
develop, debug, and perform a detailed evaluation of Aster. For commercial ASR
systems, we only used them to verify that Aster can expose faults for more
well-established ASR systems. This is mainly due to budget considerations.
Since these commercial ASR systems are charged on a query basis, the
mutation(RQ2) and ablation(RQ3) studies could incur a significant cost.
Moreover, the performance of the commercial ASR systems may change as they are
updated and improved, which could also impact the validity of the results. As
we have made several interesting findings with the current evaluation setup,
we leave the more comprehensive study of commercial ASR systems as future
work.

##  V Related Work

Software Accessibility Testing. Ensuring software products are accessible to
people with disabilities requires rigorous accessibility testing. Studies have
proposed various approaches, including AXERAY [17], which infers semantic
groupings to assess web accessibility, Latte [16], an automated framework for
testing Android app accessibility and functional correctness, and MATE [14],
which checks for accessibility issues related to visual impairment. Another
study [15] focused on web accessibility testing for deaf individuals using
Sign Language and proposed two automation approaches based on site analysis.
An optimal combination of accessibility testing methods was proposed in [43]
based on a cost-benefit analysis. Testing for stuttering on ASR systems is
also crucial to ensuring equal access for individuals with speech disorders.

ASR System Accessibility Enhancement. There is growing interest in utilizing
machine learning and deep learning techniques to detect stuttering in speech.
Several studies have reviewed current approaches to stuttering classification
[18], evaluated machine learning approaches to detect stuttering events [19],
investigated the impact of multi-task and adversarial learning for robust
stutter feature learning [20], proposed deep neural network models achieving
state-of-the-art results in stutter detection [21], and addressed automatic
detection of disfluency boundaries in children’s speech [44]. Our work
evaluates the performance of automatic speech recognition (ASR) systems in
recognizing stuttering speech patterns, a critical step toward the development
of more accurate and inclusive ASR systems.

Other Related Testing Techniques. The most related testing techniques to Aster
are metamorphic testing techniques [45, 46, 47, 48] and MOO-aided testing
techniques [49, 50, 51, 52, 53]. The design of Aster was inspired by some of
these papers. In Aster, the metamorphic relation that the output text of an
ASR system should be the same for both the original audio and the mutated
audio is used to provide the test oracle. The MOO strategy is used in Aster to
select the seeds for mutating and generating test cases.

##  VI Conclusion

In conclusion, this paper investigated the impact of stuttering speech
patterns on the performance of ASR systems. We developed a comprehensive
methodology called Aster for generating stuttering audio samples, applying
mutators designed to mimic common stuttering patterns. We then conducted a
series of experiments to evaluate the performance of commercial ASR systems on
these audio samples, comparing the results to those obtained using benign
audio samples. The evaluation results shed light on the performance of ASR
systems on stuttering speech patterns, highlighting the need for the
development of more accurate and inclusive ASR systems that can better
recognize and transcribe stuttering speech patterns.

##  VII Acknowledges

This research is supported by the National Research Foundation, Singapore, and
DSO National Laboratories under the AI Singapore Programme (AISG Award No:
AISG2-GC-2023-008). It is also supported by by the National Research
Foundation, Singapore, and the Cyber Security Agency under its National
Cybersecurity R&D Programme (NCRP25-P04-TAICeN) and the NRF Investigatorship
NRF-NRFI06-2020-0001. Any opinions, findings and conclusions or
recommendations expressed in this material are those of the author(s) and do
not reflect the views of National Research Foundation, Singapore and Cyber
Security Agency of Singapore. The computational work for this article was
partially performed on resources of the National Supercomputing Centre,
Singapore (https://www.nscc.sg).

## References

  * [1] “What is asr? a comprehensive overview of automatic speech recognition technology,” 2021, https://www.assemblyai.com/blog/what-is-asr/.
  * [2] D. S. Park, W. Chan, Y. Zhang, C.-C. Chiu, B. Zoph, E. D. Cubuk, and Q. V. Le, “Specaugment: A simple data augmentation method for automatic speech recognition,” _ArXiv_ , vol. abs/1904.08779, 2019.
  * [3] J. Li, L. Deng, Y. Gong, and R. Häb-Umbach, “An overview of noise-robust automatic speech recognition,” _IEEE/ACM Transactions on Audio, Speech, and Language Processing_ , vol. 22, pp. 745–777, 2014.
  * [4] G. E. Hinton, L. Deng, D. Yu, G. E. Dahl, A. rahman Mohamed, N. Jaitly, A. W. Senior, V. Vanhoucke, P. Nguyen, T. N. Sainath, and B. Kingsbury, “Deep neural networks for acoustic modeling in speech recognition,” _IEEE Signal Processing Magazine_ , vol. 29, p. 82, 2012.
  * [5] A. Y. Hannun, C. Case, J. Casper, B. Catanzaro, G. F. Diamos, E. Elsen, R. J. Prenger, S. Satheesh, S. Sengupta, A. Coates, and A. Ng, “Deep speech: Scaling up end-to-end speech recognition,” _ArXiv_ , vol. abs/1412.5567, 2014\.
  * [6] “Google asr product,” 2021, https://cloud.google.com/speech-to-text.
  * [7] “Microsoft asr product,” 2021, https://azure.microsoft.com/en-us/products/cognitive-services/speech-to-text/.
  * [8] “Ibm asr product,” 2021, https://www.ibm.com/cloud/watson-speech-to-text.
  * [9] “Huggingface asr models,” 2021, https://huggingface.co/models?sort=downloads&search=asr.
  * [10] M. K. Ngueajio and G. J. Washington, “Hey asr system! why aren’t you more inclusive? automatic speech recognition systems’ bias and proposed bias mitigation techniques. a literature review,” in _Interacción_ , 2022\.
  * [11] S. Sheikhbahaei and G. A. Maguire, “Scientists, society, and stuttering,” _International Journal of Clinical Practice_ , vol. 74, 2020.
  * [12] S. A. Sheikh, M. Sahidullah, F. Hirsch, and S. Ouni, “Stutternet: Stuttering detection using time delay neural network,” _2021 29th European Signal Processing Conference (EUSIPCO)_ , pp. 426–430, 2021.
  * [13] O. Shonibare, X. Tong, and V. Ravichandran, “Enhancing asr for stuttered speech with limited data using detect and pass,” _ArXiv_ , vol. abs/2202.05396, 2022.
  * [14] M. M. Eler, J. M. Rojas, Y. Ge, and G. Fraser, “Automated accessibility testing of mobile apps,” _2018 IEEE 11th International Conference on Software Testing, Verification and Validation (ICST)_ , pp. 116–126, 2018.
  * [15] C. C. S. de Sousa, L. M. Oliveira, C. L. Rodrigues, R. de Freitas Bulcão-Neto, and D. J. Ferreira, “Web accessibility testing for deaf: Requirements and approaches for automation,” _2020 IEEE International Conference on Systems, Man, and Cybernetics (SMC)_ , pp. 2734–2739, 2020.
  * [16] N. Salehnamadi, A. Alshayban, J.-W. Lin, I. Ahmed, S. M. Branham, and S. Malek, “Latte: Use-case and assistive-service driven automated accessibility testing framework for android,” _Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems_ , 2021.
  * [17] M. Bajammal and A. Mesbah, “Semantic web accessibility testing via hierarchical visual analysis,” _2021 IEEE/ACM 43rd International Conference on Software Engineering (ICSE)_ , pp. 1610–1621, 2021.
  * [18] S. A. Sheikh, M. Sahidullah, F. Hirsch, and S. Ouni, “Machine learning for stuttering identification: Review, challenges and future directions,” _Neurocomputing_ , vol. 514, pp. 385–402, 2022. [Online]. Available: https://doi.org/10.1016/j.neucom.2022.10.015
  * [19] S. Alharbi, M. Hasan, A. J. H. Simons, S. Brumfitt, and P. Green, “Sequence labeling to detect stuttering events in read speech,” _Computer Speech & Language_, vol. 62, p. 101052, 2020. [Online]. Available: https://www.sciencedirect.com/science/article/pii/S0885230819302967
  * [20] S. A. Sheikh, M. Sahidullah, F. Hirsch, and S. Ouni, “Robust stuttering detection via multi-task and adversarial learning,” in _30th European Signal Processing Conference, EUSIPCO 2022, Belgrade, Serbia, August 29 - Sept. 2, 2022_. IEEE, 2022, pp. 190–194. [Online]. Available: https://ieeexplore.ieee.org/document/9909644
  * [21] T. Kourkounakis, A. Hajavi, and A. Etemad, “Fluentnet: End-to-end detection of stuttered speech disfluencies with deep learning,” _IEEE ACM Trans. Audio Speech Lang. Process._ , vol. 29, pp. 2986–2999, 2021. [Online]. Available: https://doi.org/10.1109/TASLP.2021.3110146
  * [22] “Fluency Disorders - American Speech-Language-Hearing Association (ASHA),” https://www.asha.org/practice-portal/clinical-topics/fluency-disorders/, accessed: 2023-02-10.
  * [23] A. Laiho, H. Elovaara, K. Kaisamatti, K. Luhtalampi, L. Talaskivi, S. Pohja, K. Routamo-Jaatela, and E. Vuorio, “Stuttering interventions for children, adolescents, and adults: a systematic review as a part of clinical guidelines,” _Journal of Communication Disorders_ , p. 106242, 2022.
  * [24] S. E. Tichenor and J. S. Yaruss, “Stuttering as defined by adults who stutter,” _Journal of Speech, Language, and Hearing Research_ , vol. 62, no. 12, pp. 4356–4369, 2019.
  * [25] A. Craig and Y. Tran, “Trait and social anxiety in adults with chronic stuttering: Conclusions following meta-analysis,” _Journal of fluency disorders_ , vol. 40, pp. 35–43, 2014.
  * [26] K. A. Murza, M. Vanryckeghem, C. Nye, and A. Subramanian, “Effects of stuttering treatment: A systematic review of single-subject experimental design studies. ebp briefs. volume 13, issue 4.” _EBP Briefs (Evidence-based Practice Briefs)_ , 2019.
  * [27] A. Brignell, M. Krahe, M. Downes, E. Kefalianos, S. Reilly, and A. T. Morgan, “A systematic review of interventions for adults who stutter,” _Journal of Fluency Disorders_ , vol. 64, p. 105766, 2020.
  * [28] A. Bleakley, D. Rough, A. Roper, S. Lindsay, M. Porcheron, M. Lee, S. A. Nicholson, B. R. Cowan, and L. Clark, “Exploring smart speaker user experience for people who stammer,” in _Proceedings of the 24th International ACM SIGACCESS Conference on Computers and Accessibility_ , 2022, pp. 1–10.
  * [29] “Pocketsphinx 5.0.0,” 2021, https://github.com/cmusphinx/pocketsphinx.
  * [30] C. S. Lea, V. Mitra, A. S. Joshi, S. S. Kajarekar, and J. P. Bigham, “Sep-28k: A dataset for stuttering event detection from podcasts with people who stutter,” _ICASSP 2021 - 2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , pp. 6798–6802, 2021.
  * [31] J. Devlin, M.-W. Chang, K. Lee, and K. N. Toutanova, “Bert: Pre-training of deep bidirectional transformers for language understanding,” 2018. [Online]. Available: https://arxiv.org/abs/1810.04805
  * [32] N. Gunantara, “A review of multi-objective optimization: Methods and its applications,” _Cogent Engineering_ , vol. 5, 2018.
  * [33] A. Baevski, Y. Zhou, A. Mohamed, and M. Auli, “wav2vec 2.0: A framework for self-supervised learning of speech representations,” in _Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual_ , H. Larochelle, M. Ranzato, R. Hadsell, M. Balcan, and H. Lin, Eds., 2020\. [Online]. Available: https://proceedings.neurips.cc/paper/2020/hash/92d1e1eb1cd6f9fba3227870bb6d7f07-Abstract.html
  * [34] “Speech to text – audio to text translation — microsoft azure,” https://azure.microsoft.com/en-us/products/cognitive-services/speech-to-text, (Accessed on 02/06/2023).
  * [35] “Speech-to-text: Automatic speech recognition - google cloud,” https://cloud.google.com/speech-to-text, (Accessed on 02/06/2023).
  * [36] “Ibm watson speech to text — ibm,” https://www.ibm.com/cloud/watson-speech-to-text, (Accessed on 02/06/2023).
  * [37] R. Ardila, M. Branson, K. Davis, M. Kohler, J. Meyer, M. Henretty, R. Morais, L. Saunders, F. M. Tyers, and G. Weber, “Common voice: A massively-multilingual speech corpus,” in _Proceedings of The 12th Language Resources and Evaluation Conference, LREC 2020, Marseille, France, May 11-16, 2020_ , N. Calzolari, F. Béchet, P. Blache, K. Choukri, C. Cieri, T. Declerck, S. Goggi, H. Isahara, B. Maegaard, J. Mariani, H. Mazo, A. Moreno, J. Odijk, and S. Piperidis, Eds. European Language Resources Association, 2020, pp. 4218–4222. [Online]. Available: https://aclanthology.org/2020.lrec-1.520/
  * [38] N. B. Ratner and B. MacWhinney, “Fluency bank: A new resource for fluency research and practice,” _Journal of fluency disorders_ , vol. 56, pp. 69–80, 2018.
  * [39] P. He, C. Meister, and Z. Su, “Structure-invariant testing for machine translation,” in _ICSE ’20: 42nd International Conference on Software Engineering, Seoul, South Korea, 27 June - 19 July, 2020_ , G. Rothermel and D. Bae, Eds. ACM, 2020, pp. 961–973. [Online]. Available: https://doi.org/10.1145/3377811.3380339
  * [40] J. Goodier, _The Concise Encyclopedia of Statistics_ , 2009.
  * [41] J. L. Fleiss, “Measuring nominal scale agreement among many raters.” _Psychological bulletin_ , vol. 76, no. 5, p. 378, 1971.
  * [42] T. K. Kim, “T test as a parametric statistic,” _Korean journal of anesthesiology_ , vol. 68, no. 6, pp. 540–546, 2015.
  * [43] A. Bai, H. Mork, and V. G. Stray, “A cost-benefit analysis of accessibility testing in agile software development: Results from a multiple case study,” 2017\.
  * [44] S. Yildirim and S. S. Narayanan, “Automatic detection of disfluency boundaries in spontaneous speech of children using audio-visual information,” _IEEE Trans. Speech Audio Process._ , vol. 17, no. 1, pp. 2–12, 2009. [Online]. Available: https://doi.org/10.1109/TASL.2008.2006728
  * [45] T. Y. Chen, S. C. Cheung, and S.-M. Yiu, “Metamorphic testing: A new approach for generating next test cases,” _ArXiv_ , vol. abs/2002.12543, 2020.
  * [46] S. Segura, J. A. Parejo, J. Troya, and A. Ruiz-Cortés, “Metamorphic testing of restful web apis,” _IEEE Transactions on Software Engineering_ , vol. 44, pp. 1083–1099, 2018.
  * [47] S. Segura, G. Fraser, A. B. Sánchez, and A. Ruiz-Cortés, “A survey on metamorphic testing,” _IEEE Transactions on Software Engineering_ , vol. 42, pp. 805–824, 2016.
  * [48] H. Liu, F.-C. Kuo, D. Towey, and T. Y. Chen, “How effectively does metamorphic testing alleviate the oracle problem?” _IEEE Transactions on Software Engineering_ , vol. 40, pp. 4–22, 2014.
  * [49] M. G. Epitropakis, S. Yoo, M. Harman, and E. K. Burke, “Empirical evaluation of pareto efficient multi-objective regression test case prioritisation,” in _Proceedings of the 2015 International Symposium on Software Testing and Analysis_ , ser. ISSTA 2015, 2015.
  * [50] K. Mao, M. Harman, and Y. Jia, “Sapienz: Multi-objective automated testing for android applications,” in _Proceedings of the 25th International Symposium on Software Testing and Analysis_ , ser. ISSTA 2016. ACM, 2016.
  * [51] N. Alshahwan, X. Gao, M. Harman, Y. Jia, K. Mao, A. Mols, T. Tei, and I. Zorin, “Deploying search based software engineering with sapienz at facebook,” in _Search-Based Software Engineering_. Springer International Publishing, 2018.
  * [52] A. Panichella, R. Oliveto, M. D. Penta, and A. De Lucia, “Improving multi-objective test case selection by injecting diversity in genetic algorithms,” _IEEE Transactions on Software Engineering_ , 2015.
  * [53] Y. Li, Y. Xue, H. Chen, X. Wu, C. Zhang, X. Xie, H. Wang, and Y. Liu, “Cerebro: context-aware adaptive fuzzing for effective vulnerability detection,” _Proceedings of the 2019 27th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering_ , 2019.


††thanks: Now at Paderborn University, Department of Physics, Warburger Straße
100, 33098 Paderborn, Germany

# Lateral 2D superlattices in GaAs heterostructures with independent control
of carrier density and modulation potential

D. Q. Wang<EMAIL_ADDRESS>School of Physics, University of New
South Wales, Sydney NSW 2052, Australia  Australian Research Council Centre of
Excellence in Future Low-Energy Electronics Technologies, University of New
South Wales, Sydney NSW 2052, Australia  D. Reuter  A. D. Wieck  Angewandte
Festkörperphysik, Ruhr-Universität Bochum, D-44780 Bochum, Germany  A. R.
Hamilton  School of Physics, University of New South Wales, Sydney NSW 2052,
Australia  Australian Research Council Centre of Excellence in Future Low-
Energy Electronics Technologies, University of New South Wales, Sydney NSW
2052, Australia  O. Klochan  School of Physics, University of New South Wales,
Sydney NSW 2052, Australia  Australian Research Council Centre of Excellence
in Future Low-Energy Electronics Technologies, University of New South Wales,
Sydney NSW 2052, Australia  School of Science, University of New South Wales,
Canberra ACT 2612, Australia

###### Abstract

We present a new double-layer design for 2D surface superlattice systems in
GaAs-AlGaAs heterostructures. Unlike previous studies, our device (1) uses an
in-situ gate, which allows very short period superlattice in high mobility,
shallow heterostructures; (2) enables independent control of the carrier
density and the superlattice modulation potential amplitude over a wide range.
We characterise this device design using low-temperature magneto-transport
measurements and show that the fabrication process caused minimal damage to
the system. We demonstrate the tuning of potential modulation from weak (much
smaller than Fermi energy) to strong (larger than the Fermi energy) regimes.

††preprint: final draft

Moiré superlattices in two-dimensional (2D) materials are currently attracting
significant attention as they enable the modification of bandstructure and
thus the manipulation of electrical properties, beyond the limitations of
natural crystals. These artificially engineered superlattices exhibit a range
of properties including unconventional superconductivity Cao18_1 ; Yankowitz19
; Lu19 , strongly correlated states Cao18_2 ; Chen19 ; Lu19  , indirect valley
excitons Seyler19  and anomalous Hall effect Serlin20 . Artificial
superlattices can also be applied to the 2D electron gas formed in
conventional semiconductor heterostructures where the existing electronic
properties can be altered by lithographical patterning Park09 ; Givertini09 .
Compared to Moiré systems, heterostructure based superlattices have the
advantage of controllable lattice parameters: the superlattice can be made
with any lattice symmetry, and a wide range of lattice constants. Lattice
defects can be deliberately introduced by changing the lithography. Moreover,
by tuning the properties of the original semiconductor system, such as spin-
orbit coupling, we have the possibility of realising a topological insulator
in those artificially engineered systems Sushkov13 .

The difficulty of creating artificial bandstructures in a semiconductor based
superlattice system lies in fabricating a low-disorder and small-period
superlattice while maintaining the sample mobility at the same time. The
artificial bandgap created by a periodic modulation potential depends on two
parameters of the system. One is the lattice constant $a$, to which the size
of the mini-bandgap is inversely proportional Givertini09 ; Tkachenko15 . This
means a small lattice constant $a\lesssim 150nm$ is required to overcome
thermal and disorder-induced broadening at cryogenic temperatures. The other
is the strength of the modulation potential $U$, which decays exponentially as
a function of the distance from the gate to the 2DEG Tkachenko15 . Since the
2DEG in conventional semiconductor heterostructures is generally buried tens
of nanometers from the surface, a very shallow heterostructure is often
required. However, compared to deep heterostructures, these shallow ones are
inevitably more prone to damage caused by fabrication processes, and also
scattering from surface states Wang13 .

Previous studies of 2D superlattice modulated semiconductor 2DEGs have mainly
focused on the GaAs material system Weiss91_2 ; Weiss91 ; Geisler04 ; Kato12 ;
Singha11 ; Wang18 . Early work employing deep etching to remove part of the
heterostructure revealed so-called “Weiss” oscillations in the
magnetoresistances due to scattering from the periodic potential Weiss91_2 ;
Weiss91 . However, these studies were mainly performed on modulation-doped
structures with a comparatively deep 2DEG and large superlattice periods,
making them unsuitable for artificial bandstructure investigations. Recently
smaller-period superlattices were achieved using similar deep-etching approach
Singha11 ; Wang18  but since the electron mobility in these devices is greatly
reduced due to the etching, observing artificial bandstructures in transport
measurements was not possible. Another method to create a lateral superlattice
is using the strain caused by patterned resist on wafer surface through the
piezo-electric effect Geisler04 ; Kato12 . In this approach, damage to the
heterostructure is reduced but the modulation potential is too weak to resolve
artificial bandstructures Givertini09 ; Tkachenko15 . Therefore, how to find
the suitable modulation strength while maintaining the quality of the
heterostructure remains a key challenge. Moreover, it is hard to tune the
Fermi energy over a wide range in the modulation-doped heterostructures used
to date. In contrast, the carrier density can be widely tuned in Moiré
superlattices of 2D materials, making it much easier to probe the artificial
bandstructure effects.

In this paper, we present a double-layer design that allows separate control
of the 2DEG density and potential modulation strength in GaAs heterostucture
based artificial superlattices. Electrical transport measurements show clean
Shubnikov de Haas oscillations at low magnetic fields and well quantised Hall
plateaus, indicating the high-quality of the sample and showing that the
fabrication process hardly introduces any disorder. By varying the biases
applied to two gates, we also show that the periodic modulation potential can
be changed from weak ($U\ll E_{F}$) to strong ($U\geqslant E_{F}$), and the
2DEG density can be easily tuned from $4\times 10^{10}cm^{-2}$ to $1.6\times
10^{11}cm^{-2}$.

In our approach we use an accumulation-mode GaAs-AlGaAs heterostructure with a
double-layer gate design as depicted by the schematic in Figure 1(a): the
lower gate is a perforated gate (PG), which is an epitaxially grown layer of
n+GaAs etched with periodic hole arrays as shown in Figure 1(b). This heavily
doped GaAs gate design has the advantage of allowing the gate to be very close
to the 2DEG while minimising scattering from surface states, which is the
dominant scattering process in shallow heterostructures Wang13 , and thus
improves the electron mobility. The wafer used in this study is a shallow
undoped (100) GaAs/AlxGa1-xAs heterostructure comprising a $15nm$
$\delta$-doped n+GaAs cap, a $10nm$ bulk-doped n+GaAs cap, a $10nm$ undoped
GaAs cap and a $50nm$ AlxGa1-xAs layer on a GaAs buffer layer. A total of
$25nm$ thick n+GaAs cap serves as the in-situ perforated gate. The effective
distance between the hetero-interface and the perforated gate is $60nm$.

Figure 1: (a) 3D schematic of the our device. The bottom perforated gate is a
heavily doped GaAs layer grown in the MBE chamber. The wafer is then processed
and etched with a periodic antidot array. (b) AFM image of a device showing
the etched n+GaAs capping layer with a triangular superlattice with a lattice
constant of $120nm$. (c) Statistical analysis of the etched antidot array
showing the uniformity of the etched superlattice. The etched antidots have a
average diameter of $30nm$ with a standard deviation of 1nm. (d) (e)
Schematics demonstrating the two limits of the modulation potential achievable
by adjusting the bias applied to the top gate: (d) the weak modulation regime
where the modulation potential is much smaller than the Fermi energy $U\ll
E_{F}$, and (d) the strong modulation regime where the modulation potential is
comparable to or larger than the Fermi energy $U\geqslant E_{F}$.

To pattern the n+GaAs gate with a superlattice we developed a low-power
inductively coupled plasma-reactive ion etching (ICP-RIE) process to minimize
damage to the heterostructure due to ion bombardment. Positive e-beam resist
AR-P 6200 patterned with a Raith e-beam writer is utilized as the etch-mask
for the ICP-RIE step. A low flow of etchant BCl3 is used to control the etch
depth, stopping in the n+ gate layer so that only the conducting layer is
removed. As shown by the Atomic Force Microscopy(AFM) image of an etched dot
array in Fig 1(b), the patterned superlattice is in the form of a lateral 2D
triangular antidot lattice with a lattice constant of 120nm. Such small
superlattices are susceptible to any non-uniformity in the lithographic
patterning. A statistical analysis of the AFM image shows that the resulting
holes are highly periodic and uniform, with a diameter of 30nm consistent with
the patterned resist.

In a normal operation of the device, a positive bias is applied to the
perforated gate, which induces electrons at the GaAs-AlGaAs hetero-interface
and controls the electron density. Insulated by a thin layer of aluminium
oxide, the top layer is an overall metal top gate (TG). Since the top gate is
screened by the lower perforated gate except where the perforated gate is
drilled, i.e at the vertices of the triangular superlattice, it can be used to
manipulate the strength of the periodic modulation potential $U$. Depending on
the voltage applied to the top gate, the device can either operate in the
weak-modulation regime $U\ll E_{F}$ as shown in Figure 1(d) or the strong-
modulation regime $U\geqslant E_{F}$ as shown in Figure 1(e). When the top
gate is positively or weakly negatively biased, 2DEG can still form underneath
it at the vertices of the superlattice due to the positive bias on the
perforated gate. In this case the modulation potential is quite weak and much
smaller than $E_{F}$. When the top gate is strongly negatively biased, it
fully depletes the electrons beneath so that no 2DEG exists at the vertices of
the superlattice. This creates a very strong periodic modulation potential
that is comparable to or larger than $E_{F}$.

Figure 2: (a) Density of the 2DEG as a function of the voltage applied to the
perforated gate (PG) for different top gate (TG) voltages at $T=4K$ measured
from the low-field Hall slope. (b) and (c) Longitudinal magneto-resistance and
Hall resistance of the device at a 2DEG density of $n=1\times 10^{11}cm^{-2}$
and $T=20mK$ for $V_{TG}=0$ and $V_{TG}=-3V$ respectively. Inset of (c) shows
a zoom-in of the boxed area (magneto-resistance between 0.06T and 0.15T)
including both up and down field sweeps to show that the fine structures in
the oscillations are highly reproducible.

We first present characterization of the device at 4K using standard lock-in
techniques Chen12  with a small AC excitation voltage of $100\mu V$. Figure
2(a) shows the density of the device extracted from the low-field Hall slope
as a function of $V_{PG}$ for different voltages on the top gate. The density
of the device is a linear function of $V_{PG}$ with an average slope of
$93\times 10^{10}cm^{-2}/V$, which can be understood as the capacitance of a
parallel-plate capacitor formed by the perforated gate and the 2DEG. The top
gate has a much weaker effect on the 2DEG density, since it is mainly screened
from the 2DEG by the perforated gate underneath it. From the shifting of the
density traces with applied $V_{TG}$ in Figure 2(a), the effect of the
$V_{TG}$ on 2DEG density is calculated to be $1.1\times 10^{10}cm^{-2}/V$,
more than 80 times smaller than that of $V_{PG}$.

In Figure 2(b) and (c) we show the longitudinal magneto-resistance and Hall
resistance measured at dilution refrigerator base temperature $T=20mK$ and
electron density $n=1\times 10^{11}cm^{-2}$ for $V_{TG}=0$ and $V_{TG}=-3V$.
At $V_{TG}=0$, the device shows low resistivity, clean Shubnikov de Haas
oscillations and Hall quantization at magnetic fields as low as 0.1T,
comparable to a 2DEG in the same wafer without superlattice patterning. This
suggests that the heterostructure maintained high-quality and there is low
damage to the heterostructure caused by the fabrication process, especially
from the etching of the superlattice. From the resistance we estimate the mean
free path in this device is around $2\mu m$ and $k_{F}l$ is larger than 100 at
$n=1\times 10^{11}cm^{-2}$. This mean free path is much larger than the
lattice spacing of 120nm, which guarantees the electrons “feel” the artificial
lattice potential before they scatter. Biasing $V_{TG}$ to $-3V$ turns on the
superlattice potential, and the magneto-resistance is strongly modified with a
complicated beating pattern. As shown in the inset of Figure 2(c), the
oscillations in the magneto-resistance are highly reproducible even at
magnetic fields down to 60mT. This indicates that the complex magneto-
resistance is not due to noise but must be caused by the periodic potential
landscape imposed by the patterned gate.

Figure 3: Background removed magneto-resistance of the device at a fixed
magnetic field $B=0.2T$ plotted as a function of $V_{PG}$ and $V_{TG}$
voltages. The approximate number of electrons per unit cell is varied from 3
($V_{PG}=0.11V$, $V_{TG}=-3V$) in the top left corner to 10 ($V_{PG}=0.2V$,
$V_{TG}=0V$) in the bottom right corner. Dash-dotted and dotted lines are
guides to the eye indicating the change of slope for the low $V_{TG}$
($V_{TG}<-1V$) and high $V_{TG}$ ($V_{TG}>-2V$) regimes.

To estimate the strength of the modulation potential and investigate how it is
varied by $V_{TG}$, we plot in Figure 3 the background-removed magneto-
resistance supp  at $B=0.2T$ as a function of the voltages applied to the
perforated gate and the top gate. Over the entire map, the number of electrons
per unit cell is varied from approximately 3 to 10 with the highest number of
electrons per unit cell at $V_{PG}=0.2V$, $V_{TG}=0$ in the top left corner
and the lowest number of electrons per unit cell at $V_{PG}=0.11V$,
$V_{TG}=-3V$ in the bottom right corner. The colour plot shows a series of
blue and red stripes which correspond to magneto-resistance oscillations as
the density of the 2DEG is varied at $B=0.2T$. The tilt of the stripes towards
the right is caused by the weak effect of $V_{TG}$ on the 2DEG density.
Therefore, the slope of the stripes is a measure of the relative strength of
the capacitive coupling between TG-2DEG and PG-2DEG. From the dash-dotted
lines, this slope is estimated around $85$, in good agreement with the $4K$
data shown in Figure 2(a).

An important feature observed in Figure 3, highlighted by the dash-dotted and
dotted lines, is that the slope of the stripes undergoes a clear change from
$\sim 85$ to $\sim 65$ as the top gate is biased more negatively. This can be
understood as follows. For the the low $V_{TG}$ regime ($V_{TG}<-1V$), the
modulation potential is weak and much smaller than the Fermi energy ($E_{F}$)
as depicted in Figure 1(d). In this case, electrons exist everywhere in the
system including underneath the top gate, i.e, at the vertices of the
superlattice. For the high $V_{TG}$ regime ($V_{TG}>-2V$), the top gate fully
depletes the electrons beneath it and created a potential barrier comparable
to or larger than $E_{F}$. This results in a strong modulation potential as
depicted in Figure 1(e) and a decrease in the effective area of the perforated
gate, which reduces the capacitive coupling between the perforated gate and
the 2DEG and thus the slope of the stripes. Therefore, this change in the
slope is a clear indication that the modulation potential $U$ can be
effectively tuned from the weak ($U\ll E_{F}$) to strong ($U\geqslant E_{F}$)
regimes by simply varying $V_{TG}$.

In conclusion, we fabricated a high-quality electron system with a
superimposed small-period triangular superlattice. We used shallow GaAs
heterostructures with an in-situ gate and a double-layer design. Magneto-
transport measurements of the system show low resistance, clean oscillations
and well quantised Hall plateaus at very small magnetic fields, which
indicates the fabrication process caused minimal damage to the
heterostructure. By changing the voltage applied to two separate gates, we
demonstrated that density and modulation strength can be tuned separately in
these devices over a wide range. This device architecture can serve as a
flexible platform for future band structure engineering experiments.

###### Acknowledgements.

We acknowledge useful discussions with O. P. Sushkov and Z. E. Krix. This work
is supported by the Australian Research Council Centre of Excellence in Future
Low-Energy Electronics Technologies (CE170100039). A. D. W. acknowledges
gratefully support of DFG-TRR160, BMBF - Q.Link.X 16KIS0867, and the DFH/UFA
CDFA-05-06. All devices were fabricated using facilities at the NSW and ACT
Nodes of the Australian National Fabrication Facility. AFM images were taken
using the facilities of Microscopy Australia at the Electron Microscope Unit
within the Mark Wainwright Analytical Centre at UNSW Sydney.

## Data Availability Statement

The data that support the findings of this study are available from the
corresponding author upon reasonable request.

## References

  * [1] Y. Cao, V. Fatemi, S. Fang, K. Watanabe, T. Taniguchi, E. Kaxiras and P. Jarillo-Herrero, Nature 556, 43-50(2018).
  * [2] C. Lu, P. Stepanov, W. Yang, M. Xie, M. Ali Aamir, I. Das, C. Urgell, K. Watanabe, T. Taniguchi, G. Zhang, A. Bachtold, A. H. MacDonald and D. K. Efetov, Nature 574, 653-657(2019).
  * [3] M. Yankowitz, S. Chen, H. Polshyn, Y. Zhang, K. Watanabe, T. Taniguchi, D. Graf, A. F. Young and C. R. Dean, Science 363, 1059-1064(2019).
  * [4] Y. Cao, V. Fatemi, A. Demir, S. Fang, S. L. Tomarken, J. Y. Luo, J. D. Sanchez-Yamagishi, K. Watanabe, T. Taniguchi, E. Kaxiras R. C. Ashoori and P. Jarillo-Herrero, Nature 556, 80-84(2018).
  * [5] G. Chen, L. Jiang, S. Wu B. Lyu, H. Li, B. L. Chittari, K. Watanabe, T. Taniguchi, Z. Shi, J. Jung, Y. Zhang and F. Wang, Nature Physics 15, 237-241(2019).
  * [6] K. L. Seyler, P. Rivera, H. Yu, N. P. Wilson, E. L. Ray, D. G. Mandrus, J. Yan, W. Yao and X. Xu, Nature 567, 66-70(2019).
  * [7] M. Serlin, C. L. Tschirhart1, H. Polshyn1, Y. Zhang, J. Zhu, K. Watanabe, T. Taniguchi, L. Balents and A. F. Young, Science 367, 900-903(2020).
  * [8] C.-H. Park and S. G. Louie, Nano Lett. 9, 1793-1797(2009).
  * [9] M. Gibertini, A. Singha, V. Pellegrini, M. Polini, G. Vignale, A. Pinczuk, L. N. Pfeiffer, and K. W. West, Phys. Rev. B 79, 241406(2009).
  * [10] O. P. Sushkov and A. H. Castro Neto, Phys. Rev. Lett. 110, 186601 (2013).
  * [11] O. A. Tkachenko, V. A. Tkachenko, I. S. Terekhov and O. P. Sushkov, 2D Mater. 2, 014010(2015).
  * [12] D. Q. Wang, J. C. H. Chen, O. Klochan, K. Das Gupta, D. Reuter, A. D. Wieck, D. A. Ritchie, and A. R. Hamilton, Phys. Rev. B 87, 195313(2013).
  * [13] D. Weiss, P. Grambow, K. von Klitzing, A. Menschig, and G. Weimann, Appl. Phys. Lett. 58, 2960(1991).
  * [14] D. Weiss, M. L. Roukes, A. Menschig, P. Grambow, K. von Klitzing and G. Weimann, Phys. Rev. Lett. 66, 2790(1991).
  * [15] A. Singha, M. Gilbertini, B. Karmakar, S. Yuan, M. Polini, G. Vignale, M. I. Katsnelson, A. Pinczuk, L. N. Pfeiffer, K. W. West and V. Pellegrini, Science 332, 1176-1179(2011).
  * [16] S. Wang, D. Scarabelli L. Du, Y. Kuznetsova, L. N. Pfeiffer, K. W. West, G. C. Gardener, M. J. Manfra, V. Pellegrini, S. J. Wind and A. Pinczuk, Nature Nanotechnology 13, 29-33(2018).
  * [17] M. C. Geisler, J. H. Smet, V. Umansky, K. von Klitzing, B. Naundorf, R. Ketzmerick and H. Scherizer, Phys. Rev. Lett. 92, 256801 (2004).
  * [18] Y. Kato, A. Endo, S. Katsumoto and Y. Iye, Phys. Rev. B 86, 235315(2012).
  * [19] J. C. H. Chen, D. Q. Wang, O. Klochan, A. P. Micolich, K. Das Gupta, F. Sfigakis, D. A. Ritchie, D. Reuter, A. D. Wieck and A. R. Hamilton, Appl. Phys. Lett 100, 052101(2012).
  * [20] Original magneto-resistance is smoothed to remove any oscillations and then subtracted from the original trace as the background.


See pages 1-last of paper.pdf