# Muting Whisper: A Universal Acoustic Adversarial Attack on Speech Foundation
Models

Vyas Raina∗ Rao Ma∗ Charles McGhee Kate Knill Mark Gales

###### Abstract

Recent developments in large speech foundation models like Whisper have led to
their widespread use in many automatic speech recognition (ASR) applications.
These systems incorporate ‘special tokens’ in their vocabulary, such as
<endoftext>, to guide their language generation process. However, we
demonstrate that these tokens can be exploited by adversarial attacks to
manipulate the model’s behavior. We propose a simple yet effective method to
learn a universal acoustic realization of Whisper’s <endoftext> token, which,
when prepended to any speech signal, encourages the model to ignore the speech
and only transcribe the special token, effectively ‘muting’ the model. Our
experiments demonstrate that the same, universal 0.64-second adversarial audio
segment can successfully mute a target Whisper ASR model for over 97% of
speech samples. Moreover, we find that this universal adversarial audio
segment often transfers to new datasets and tasks. Overall this work
demonstrates the vulnerability of Whisper models to ‘muting’ adversarial
attacks, where such attacks can pose both risks and potential benefits in
real-world settings: for example the attack can be used to bypass speech
moderation systems, or conversely the attack can also be used to protect
private speech data. 111The code is available at:
https://github.com/rainavyas/prepend_acoustic_attack.

Muting Whisper: A Universal Acoustic Adversarial Attack on Speech Foundation
Models

Vyas Raina∗ Rao Ma∗ Charles McGhee Kate Knill Mark Gales

##  1 Introduction

Figure 1: Universal acoustic adversarial audio segment when prepended to any
speech signal mutes the Automatic Speech Recognition (ASR) system, such that
an empty transcription is generated. The <endoftext> token (EOT) is a special
token in the Whisper vocabulary used to indicate the end of the generated
transcription.

The development of large foundation models has led to rapid advancements in
audio processing, where for example some of the most popular models are of the
Whisper family (Radford et al., 2022). To guide the generation of natural
language, foundation models typically make use of ‘special’ tokens in their
vocabulary that do not exist as real text or real acoustic events. As an
example, most auto-regressive foundation models will have some form of a
<start> token and an <end> token to indicate when to begin generating the
output sequence and when to stop. However, we demonstrate that despite their
need, these ‘special’ tokens can be exploited by adversaries to make
foundational models behave in undesired manners. Specifically, we show that
the <endoftext> special token can be exploited by adversaries to prevent an
Automatic Speech Recognition (ASR) model, such as Whisper, from transcribing
the source audio, i.e., ‘muting’ the model.

Our proposed acoustic adversarial attack method is designed to ‘mute’ Whisper,
by learning an extremely short (0.64-second) adversarial acoustic realization
of the <endoftext> special token (used by Whisper), where the learnt
adversarial audio segment can be prepended to the target speech signal.
Furthermore, our proposed method gives a universal adversarial audio segment,
which allows the same 0.64-second adversarial audio segment to be prepended to
any speech signal, and conceal its contents from the ASR system, as depicted
in Figure 1.

Our experiments, conducted across eight different Whisper ASR models,
demonstrate that the same universal 0.64-second adversarial audio segment can
successfully ‘mute’ Whisper models for more than 97% of unseen speech samples.
We further find that there is a surprising level of transferability of this
universal adversarial audio segment to different speech domains (we consider
four diverse datasets) and can even transfer to different tasks - the
adversarial audio segment can ‘mute’ Whisper when used for speech translation
as well as transcription.

Muting Whisper has significant implications in high stakes settings. Automatic
speech recognition (ASR) systems play a crucial role in detecting and
moderating harmful content such as hate speech (MacAvaney et al., 2019) in
audio or video recordings (Wu and Bhandary, 2020). Muting Whisper poses a risk
of circumventing this moderation process. Adversaries could exploit this
vulnerability to release harmful content to the public audience without
detection. Nevertheless, muting Whisper also has potential positive
implications for speech privacy protection (Cheng et al., 2024). In contexts
where speech recordings are transmitted over a network, malicious actors may
attempt to extract private data through automated transcription. In such
cases, our proposed method of muting Whisper could serve as a form of speech
privacy protection, similar to a ‘jamming’ signal, where the speech remains
unchanged for human listeners but ASR systems are unable to extract its
content. On the whole, through this work we demonstrate both the risks and
benefits of universal acoustic adversarial attacks aimed at muting Whisper.

##  2 Related Work

#### Audio Attacks (early research).

Initial research (Gong and Poellabauer, 2017; Cisse et al., 2017) explored
gradient-based approaches to perturb the input audio to end-to-end ASR systems
(specifically WaveCNN and HMM-DNN architectures) with the objective of
increasing the word error rate (WER) of the generated transcriptions. However,
Yuan et al. (2018); Carlini and Wagner (2018); Das et al. (2018); Qin et al.
(2019) offer methods to perform targeted attacks on ASR systems, such as
DeepSpeech, HMM-DNN and LSTM-based neural networks, where the aim was to
generate a specific output transcription. Other research (Schönherr et al.,
2018, 2018) modified audio adversarial attack methods to better encourage
their imperceptibility.

#### Practical Audio Attacks.

Neekhara et al. (2019) demonstrate that they can generate universal
adversarial perturbations such that the same adversarial audio segment can be
superimposed on different speech signals. However, these attack approaches
cannot be applied to streaming ASR systems, as they have to be superimposed on
the entire speech signal, so Li et al. (2020) attempted to address this issue
by generating universal adversarial perturbations that do not need to be
synchronised with the source speech signal (the carrier audio) when being
superimposed. Lu et al. (2021) extended the targeted universal adversarial
attacks to more recent end-to-end ASR systems including LAS, CTC and RNN-T.
Further, a range of other creative approaches have been proposed for
generating audio adversarial samples in practical settings: transferability
from substitute models (Chen et al., 2020; Fan et al., 2020; Ma et al., 2021);
evolutionary attacks (Alzantot et al., 2018; Khare et al., 2019; Taori et al.,
2019; Du et al., 2019; Zheng et al., 2021); utterance-based attacks (Raina et
al., 2020); and featurization attacks (Carlini et al., 2016; Zhang et al.,
2017; Abdullah et al., 2019).

#### Attacks on Whisper.

All of the above-mentioned methods are designed for traditional ASR systems.
The recent emergence of a powerful foundation model (Whisper) demands an
update to previously developed attack methods. Olivier and Raj (2023) perform
an initial investigation into the vulnerabilities of Whisper to audio
adversarial attacks, where they show that an adversarial signal can be
superimposed on natural speech signals such that Whisper transcribes
incorrectly.

#### Our contributions.

We extend the research on adversarial attacks for modern ASR systems such as
Whisper, by outlining a method to develop a truly practical and effective
adversarial attack with a real-world targeted objective. Specifically, this
work makes the following contributions to adversarial attack literature on ASR
systems:

  * •

We develop a short (0.64-second) adversarial audio segment that can be
prepended to a speech signal. Existing research only considers superimposing
the adversarial audio signal, which is not a practical setting for real-life
attacks.

  * •

Our adversarial audio segment is universal, so the same audio segment can be
prepended to any speech signal.

  * •

Our attack works for a popular, modern and powerful ASR system: Whisper family
of models.

  * •

The objective of our attack is specifically to mute the Whisper model; a
targeted objective not before considered and with real-world implications in
privacy and security.

  * •

Our universal adversarial acoustic attack segment transfers across data
domains and even speech processing tasks.

##  3 Automatic Speech Recognition: Whisper

An Automatic Speech Recognition (ASR) system maps an speech/audio signal of
$N$ frames , $\mathbf{x}=x_{1:N}$, to the text, $\mathbf{y}=y_{1:M}$ uttered
in the speech signal - this is the transcription of the audio with $M$
words/tokens. Whisper’s encoder-decoder architecture, $\mathcal{F}(\cdot)$
with parameters $\theta$ auto-regressively predicts a vector representing the
probability distribution over the vocabulary of tokens, $\mathcal{V}$, for the
next token $y_{m}$, with the speech, $\mathbf{x}=x_{1:N}$ at the encoder input
and the previously decoded tokens, $\mathbf{y}^{*}_{<m}$ at the decoder input,

$P(y_{m}=y|\mathbf{x},\mathbf{y}^{*}_{<m})=\mathcal{F}(\mathbf{x},\mathbf{y}^{*}_{<m};\theta)_{y},\hskip
10.00002pty\in\mathcal{V},$ (1)

where typically a greedy decoding process selects the most likely token to
generate,

$y^{*}_{m}=\operatorname*{arg\,max}_{y}P(y_{m}=y|\mathbf{x},\mathbf{y}^{*}_{<m}).$
(2)

During the decoding process various special tokens are used by the Whisper
model to guide the token generation. The first token (input to the decoder) is
set as <|startoftranscript|>, followed by a token to indicate the language,
for example <en> for English. As the Whisper model is trained to perform two
different speech processing tasks (transcription and speech translation), the
next token is used to indicate the task, e.g., <|transcribe|> or
<|translate|>. Hence we define $\mathbf{y}^{*}_{0}=$ <|startoftranscript|>
<lang tag> <|task tag|>. With this initialization, further tokens are
generated auto-regressively from the vocabulary, $\mathcal{V}$ following
Equation 1 and Equation 2. The auto-regressive decoding ends when the
<|endoftext|> special token is predicted by the decoder.

##  4 Universal Acoustic Prepend Attack

###  4.1 Attack Objective

In this section we propose a practical and effective approach for an adversary
to modify any input speech signal in a manner that results in the Whisper
model being muted (transcribing nothing), without the speech audio sounding
obviously manipulated to human listeners. The objective of muting Whisper is
equivalent to maximizing the probability of the model predicting, $y_{1}$ as
the <|endoftext|> special token. Recall that the decoder is initialized with a
sequence of special tokens, $\mathbf{y}^{*}_{0}=$ <|startoftranscript|> <lang
tag> <|task tag|>.

###  4.2 Prepend Attack

To perturb a speech signal, $\mathbf{x}=x_{1:N}$, it is simplest to prepend a
short, adversarial audio segment of $T$ frames,
$\tilde{\mathbf{x}}=\tilde{x}_{1:T}$, such that the perturbed speech signal is
$\tilde{\mathbf{x}}\oplus\mathbf{x}$, where $\oplus$ represents concatenation
in the raw audio space. Then, given Whisper’s encoder-decoder model in
Equation 1, the optimal adversarial audio segment, $\hat{\tilde{\mathbf{a}}}$,
to ‘mute’ Whisper as per the adversarial objective, can be given as finding
the adversarial audio segment that maximizes the probability of generating the
<endoftext> special token (abbreviated to eot) as the first transcribed token,

$\hat{\tilde{\mathbf{x}}}=\operatorname*{arg\,max}_{\tilde{\mathbf{x}}}P(y_{1}=\texttt{eot}|\tilde{\mathbf{x}}\oplus\mathbf{x},\mathbf{y}^{*}_{0}).$
(3)

###  4.3 Universal Attack

Learning an adversarial audio segment, $\hat{\tilde{\mathbf{x}}}$ that can be
prepended to a speech signal, $\mathbf{x}$ to conceal its contents from a
Whisper ASR model, cannot be achieved in real-time (as the attack segment has
to be prepended before the speech is generated) and requires computational
resources. Therefore, it is not practical to learn an individual adversarial
audio segment $\hat{\tilde{\mathbf{x}}}^{(j)}$ to conceal the contents of each
different speech signal, $\mathbf{x}^{(j)}$. Hence, we propose learning a
universal adversarial audio segment that is agnostic to any speech signal. For
a training dataset of $J$ speech samples $\\{\mathbf{x}^{(j)}\\}_{j=1}^{J}$,
the universal prepend attack aims to maximise the likelihood of predicting
$y_{1}=\texttt{<|endoftext|>}$ over all training samples,

$\hat{\tilde{\mathbf{x}}}=\operatorname*{arg\,max}_{\tilde{\mathbf{x}}}\prod_{j=1}^{J}P(y_{1}=\texttt{eot}|\tilde{\mathbf{x}}\oplus\mathbf{x}^{(j)},\mathbf{y}^{*}_{0}).$
(4)

As the Whisper encoder-decoder model is fully differentiable, standard
gradient-based training approaches can then be used to optimize for the
universal adversarial audio segment, $\hat{\tilde{\mathbf{x}}}$. This
universal adversarial audio segment ‘mutes’ Whisper when prepended to any
speech signal and is thus effectively an acoustic realization of the
<|endoftext|> special token.

###  4.4 Imperceptibility

For a truly practical adversarial attack, it is important for the adversarial
audio segment generated to be sufficiently imperceptible such that it is not
flagged as suspicious when prepended to natural speech signals. We achieve
this imperceptibility in two dimensions. First, we ensure that the adversarial
audio segment is extremely short such that there is little time for a human
listener to detect the abnormal speech. We specifically limit the number of
frames in the adversarial audio segment to $T=10,120$, which corresponds to
0.64-seconds of audio for a 16kHz sampling frequency. Next, we limit the
‘power’ of the adversarial audio segment, to ensure the amplitude of the
adversarial audio segment is not significant relative to natural speech. To
limit the power, we introduce a constraint in the optimization objective of
Equation 4 that limits the amplitude of the adversarial audio,

$||\hat{\tilde{x}}_{1:T}||_{\infty}\leq\epsilon,$ (5)

where $||\cdot||_{\infty}$ represents the l-infinity norm. By default we set
$\epsilon=0.02$, as on the log-mel scale this empirically represents audio
signals with power lower than typical human speech signals (refer to Figure
2). The l-infinity norm constraint is incorporated during gradient-based
learning of the adversarial audio segment $\hat{\tilde{\mathbf{x}}}$, by
clamping the values at $\epsilon$. 222For gradient-based learning we use ADAM
(Kingma and Ba, 2015) optimization with clamping after each gradient update as
in typical Projected Gradient Descent algorithms (Madry et al., 2019). Note
that in practical settings it may be undesirable to have extremely low values
for $\epsilon$, as the adversarial audio segment may then be contaminated by
low-amplitude background noise.

##  5 Prepend Adversarial Attack Evaluation

###  5.1 Attack Performance Evaluation

For a learnt universal acoustic adversarial segment trained to maximize the
probability of the Whisper model generating the <endoftext> special token as
its first token for any speech signal, as per Equation 4, we can evaluate the
performance of the adversarial attack by computing the percentage of unseen
test speech signals, $\varnothing$, for which the attack is able to
successfully ‘mute’ the Whisper model,

$\displaystyle\varnothing$
$\displaystyle=\frac{1}{J}\sum_{j}\mathbbm{1}\\{{\tilde{y}^{*(j)}_{1}=\texttt{eot}}\\}\times
100\%,$ (6) $\displaystyle\tilde{y}^{*(j)}_{1}$
$\displaystyle=\operatorname*{arg\,max}_{y}P(y_{1}=y|\hat{\tilde{\mathbf{x}}}\oplus\mathbf{x}^{(j)},\mathbf{y}^{*}_{0}),$

where $\tilde{y}^{*}_{1}=$ <eot> means that the transcribed sequence has 0
words, i.e., a perfectly successful attack. Hence, the larger the value of
$\varnothing$, approaching 100%, the more effective the acoustic adversarial
attack. A further useful metric to gauge the extent to which a universal
attack is able to ‘mute’ the Whisper model, is the ‘average sequence length’
(asl) of the predicted transcription,

$\texttt{asl}=\frac{1}{J}\sum_{j}\texttt{len}(\tilde{\mathbf{y}}^{*(j)}),$ (7)

where $\texttt{len}(\cdot)$ gives the number of words in the transcribed
sequence. The lower the value of asl, the more effective the adversarial
attack.

###  5.2 Adversarial Sensitivity Analysis

Beyond simply measuring the success of the universal acoustic adversarial
attack in ‘muting’ an ASR system, it is meaningful to have a tool to analyze
the mechanism of the attack that explains its success and lack of success for
specific speech signals. One such tool is in the form of saliency analysis,
which can indicate the sensitivity of the ASR system’s predictions to
different parts of the input audio. The frames in the input audio that the
transcription is most sensitive to are the parts of the audio that dominate
the ASR system’s decisions and this can explain which part of the input the
ASR system is attending to the most.

For an auto-regressive ASR model, $\mathcal{F}(\cdot)$ defined in Equation 1,
we can define the $m$-th saliency of the universal adversarial audio segment,
$\hat{\tilde{\mathbf{x}}}$, as the gradient of the $m$-th transcribed token,
$\tilde{y}^{*}_{m}$, as

$\tilde{s}_{m}=\left|\left|\nabla_{\hat{\tilde{\mathbf{x}}}}\left[\mathcal{F}(\hat{\tilde{\mathbf{x}}}\oplus\mathbf{x},\mathbf{y}^{*}_{<m};\theta)_{\tilde{y}^{*}_{m}}\right]\right|\right|_{2}.$
(8)

Equivalently we can define the saliency of the natural speech signal,
$\mathbf{x}$ as,

$s_{m}=\left|\left|\nabla_{\mathbf{x}}\left[\mathcal{F}(\hat{\tilde{\mathbf{x}}}\oplus\mathbf{x},\mathbf{y}^{*}_{<m};\theta)_{y^{*}_{m}}\right]\right|\right|_{2}.$
(9)

As we are interested primarily in the first generated token, we set $m=1$ in
our saliency analysis.

##  6 Experiments

###  6.1 Experimental Setup

#### Data.

Results are reported across five diverse and popular speech recognition
datasets: LibriSpeech (LBS) (Panayotov et al., 2015), TED-LIUM3 (TED)
(Hernandez et al., 2018), MGB (Bell et al., 2015); Artie Bias (Artie) (Meyer
et al., 2020) and Fleurs (Conneau et al., 2022). Details for each dataset are
provided in Section A.1. The universal acoustic attack segment is learnt using
the development split of the LibriSpeech dataset (for the task of
transcription). The attack is then evaluated on the LibriSpeech test split and
to measure the transferability of the attack it is also evaluated on the other
datasets (TED-LIUM3, MGB and Artie Bias). The universal acoustic segment is
evaluated for task transferability by also evaluating on speech transcription
and speech translation tasks using the Fleurs dataset test splits.

#### Models.

Experimental results are given for the family of Whisper ASR models Radford et
al. (2023). Model details and their performance (Word Error Rate) on the
datasets in this work have been provided for reference in Section A.2.

#### Attack Train Configuration.

The universal acoustic prepend attack segment is trained on the LibriSpeech
development split. The attack segment is trained as per Equation 4, where it
is prepended to speech samples in the raw audio space. The attack segment
length is set to be 0.64 seconds and its maximum amplitude to $\epsilon=0.02$,
to satisfy the constraint of Equation 5. Hyperparameter settings for learning
the universal acoustic attack segment are given in Section A.3.

###  6.2 Results

#### Universal Acoustic Prepend Attack.

The universal prepend attack segment is trained (on the LBS development split)
to make the ASR model generate only an <endoftext> token, i.e. transcribe
nothing. Evaluating on the LBS test-split, Table 1 gives the percentage of
successful attacks, $\varnothing$ and the average sequence length of predicted
transcriptions (asl) for the different target speech recognition models with
the same (per model) trained 0.64-second universal acoustic adversarial
segment prepended to every speech sample. A comparison is made to the no
attack setting, where the speech samples are not modified in any manner. For
every target Whisper model, the universal acoustic prepend attack is extremely
successful in ensuring the model does not transcribe the speech signals, with
the percentage of successful attacks increasing from more than 97% for the
medium models to 99.9% for the tiny models. Similarly, in all cases the asl is
brought to less than 1.0, whereas for the unattacked speech the transcriptions
have nearly 18 words on average. We also compare to a random audio segment
prepended to the speech samples and we find that this behaves identically to
the no attack setting, i.e. a random attack cannot ‘mute’ Whisper. Overall,
Table 1 shows that regardless of the model size, a short 0.64-second universal
acoustic adversarial audio segment can be prepended (imperceptibly) to almost
all speech signals to conceal the contents from Whisper speech recognition
models.

Model | Metric | No Attack | Attack
---|---|---|---
tiny.en |  $\varnothing$ (%) $\uparrow$ | 0.0 | 99.7
$\texttt{asl}\downarrow$ | 17.9 | 0.06
tiny |  $\varnothing$ (%) $\uparrow$ | 0.0 | 99.6
$\texttt{asl}\downarrow$ | 17.9 | 0.04
base.en |  $\varnothing$ (%) $\uparrow$ | 0.0 | 99.0
$\texttt{asl}\downarrow$ | 17.8 | 0.20
base |  $\varnothing$ (%) $\uparrow$ | 0.0 | 99.5
$\texttt{asl}\downarrow$ | 17.8 | 0.05
small.en |  $\varnothing$ (%) $\uparrow$ | 0.0 | 98.6
$\texttt{asl}\downarrow$ | 17.7 | 0.14
small |  $\varnothing$ (%) $\uparrow$ | 0.0 | 98.7
$\texttt{asl}\downarrow$ | 17.3 | 0.15
medium.en |  $\varnothing$ (%) $\uparrow$ | 0.0 | 99.5
$\texttt{asl}\downarrow$ | 17.7 | 0.10
medium |  $\varnothing$(%)$\uparrow$ | 0.0 | 97.8
$\texttt{asl}\downarrow$ | 17.8 | 0.56
Table 1: The percentage of successfully ‘muted’ speech samples, $\varnothing$,
where the first generated token is <endoftext>, and the Average Sequence
Length (asl) of transcriptions, for the LBS dataset. Results are presented for
no attack, and for a trained (per model) universal acoustic adversarial
attack, where the same universal adversarial segment is prepended to each
speech sample.

Figure 2 gives the Mel-spectrogram of a random speech sample from the LBS test
set with a 0.64-second universal acoustic adversarial segment prepended to the
speech signal (learnt for the medium.en model). This validates that
$\epsilon=0.02$ is an appropriate imperceptibility setting as it ensures that
the power of the adversarial segment is always less than $\sim 1.50$dB, which
is significantly lower than a typical human speech signal in the LBS dataset
that can range from 1dB to more than 3.5dB. It is interesting to note that the
acoustic adversarial segment covers the full range of frequencies relatively
uniformly, which means it is likely to sound like static noise to a human
listener. 333We provide the universal acoustic adversarial attack segment
audio files for each Whisper model at
https://github.com/rainavyas/prepend_acoustic_attack/tree/main/audio_attack_segments

Figure 2: Mel spectrogram of universal acoustic segment (0.64s) prepended to a
random speech sample from LBS dataset (truncated to a total length of 3s).

#### Attack Success Analysis.

We now investigate the $<3$% speech samples for which the universal acoustic
attack fails to perfectly mute the Whisper model, i.e., the generated
transcription is not of zero-length. Table 2 gives the average sequence length
(asl) evaluation of the generated transcripts for the failed attack samples
(relative to the successful samples) for LBS. Interestingly, when there is no
adversarial attack, the asl for the failed samples is 2 to 4 times greater
than the average $\sim$17 words in the successful samples’ transcriptions,
suggesting that the universal acoustic attack only struggles to mute the ASR
model for longer input speech signals. Further, for these failed samples, the
attack is still able to reduce the number of generated words significantly (at
least two-fold), highlighting that the attack is still effective in muting the
ASR model to some extent, although not entirely.

Model | Samples | No Attack | Attack
---|---|---|---
tiny | successful | 17.8 | 0.0
failed | 74.6 | 11.0
medium | successful | 17.2 | 0.0
failed | 43.2 | 25.0
Table 2: Average Sequence Length (asl) of generated transcripts for successful
attack samples and failed attack samples. A successful sample is where the
universal acoustic attack causes the Whisper model to generate a zero-length
transcription (perfectly muted).

A natural follow-up question is then, in what manner does the universal attack
shorten the generated transcripts for the failed samples, i.e., is it simple
truncation or is the model generating other tokens unrelated to the original
speech signal. Table 3 gives the breakdown of the word error rate (WER)
contributions from insertions, deletions and substitutions for the failed
samples, where the word error rate is computed between the predicted no attack
transcriptions and the predicted attack transcriptions. We observe that the
attack causes no significant change in the transcriptions other than
deletions, demonstrating the attack is behaving as desired in attempting to
discourage speech transcription. Overall, this analysis shows that even for
the few samples ($<3$%) that the universal attack is not able to perfectly
mute the ASR model, the attack is still able to significantly reduce the
transcription length.

Model | WER | INS | DEL | SUB
---|---|---|---|---
tiny | 88.38 | 0.36 | 85.40 | 2.29
medium | 50.76 | 2.70 | 43.75 | 2.94
Table 3: Word Error Rate (WER) and breakdown (insertions, deletions and
substitutions) between the transcript generated with no attack and the
transcript generated with the universal acoustic attack, for the failed attack
samples only. A failed sample is where the universal acoustic attack is unable
to make Whisper generate a zero-length transcription.

#### Saliency Analysis.

Section 5.2 describes saliency as a tool to measure the sensitivity of the ASR
model to the adversarial and the natural speech segments of the input audio.
The average saliencies for the LBS dataset are given in Table 4, with a
comparison for the successful attack samples and the failed attack samples. It
is clear that a successful attack results in the ASR model being significantly
more sensitive to the adversarial segment, and conversely more sensitive to
the speech signal when the attack fails. This demonstrates that the universal
acoustic attack is operating as intended, as a successful attack encourages
the model to attend more to the acoustic realization of the <endoftext>
special token (the adversarial audio segment). 444Appendix D illustrates the
frame-level saliency for example input audio signals. It is also interesting
to note that for successful attack samples the saliency is significantly
higher, suggesting that success of the adversarial attack is very dependent on
the exact learnt universal adversarial segment.

Model | Samples | Adv, $\tilde{s}$ | Speech, $s$
---|---|---|---
tiny | successful | 835 | 4.80
failed | 101 | 192
medium | successful | 3371 | 143
failed | 314 | 803
Table 4: Average saliency for the adversarial segment and speech segment
(across LBS dataset) for successful and failed samples. A successful sample is
where the universal acoustic attack causes the Whisper model to generate a
zero-length transcription (perfectly muted).

#### Attack Transferability.

The universal attack segment has been trained on a specific domain of data
(LBS data) and there is a risk that the attack may not necessarily transfer to
different, distributionally shifted speech domains. Therefore, in this section
we investigate the impact of transferring the 0.64-second universal acoustic
adversarial segment to different unseen (during training of the attack)
datasets, representing a diverse range of domain distributional shifts. Table
5 presents the results. For all models and datasets, the universal acoustic
attack is able to continue muting the Whisper models for more than 90% of
samples. Although this is slightly lower than 97% success rate for the in-
domain LBS dataset, 90% is still a significant success rate, suggesting that
the adversarial segment truly represents an acoustic realization of the
<endoftext> token, which universally prevents the transcription of different
speech domains.

| Metric | LBS | TED | MGB | Artie
---|---|---|---|---|---
Ref |  $\varnothing$ (%) | 0.0 | 0.0 | 0.0 | 0.0
asl | 17.8 | 24.4 | 8.9 | 8.6
tiny.en |  $\varnothing$ (%) | 99.7 | 99.9 | 99.9 | 100.0
asl | 0.06 | 0.01 | 0.01 | 0.00
tiny |  $\varnothing$ (%) | 99.6 | 99.0 | 99.3 | 99.2
asl | 0.04 | 0.56 | 0.10 | 0.03
base.en |  $\varnothing$ (%) | 99.0 | 98.8 | 99.0 | 99.3
asl | 0.20 | 0.32 | 0.09 | 0.03
base |  $\varnothing$ (%) | 99.5 | 99.9 | 99.5 | 97.4
asl | 0.05 | 0.01 | 0.09 | 0.17
small.en |  $\varnothing$ (%) | 98.6 | 93.1 | 98.3 | 92.4
asl | 0.14 | 1.71 | 0.20 | 0.49
small |  $\varnothing$ (%) | 98.7 | 99.5 | 93.5 | 97.0
asl | 0.15 | 0.21 | 0.43 | 0.16
medium.en |  $\varnothing$ (%) | 99.5 | 99.8 | 99.7 | 99.7
asl | 0.10 | 0.01 | 0.01 | 0.03
medium |  $\varnothing$ (%) | 97.8 | 95.2 | 96.4 | 96.9
asl | 0.56 | 1.05 | 0.29 | 0.24
Table 5: Attack transferability across datasets: the percentage of
successfully ‘muted’ speech samples, $\varnothing$, and the Average Sequence
Length (asl) of generated transcripts with the universal acoustic adversarial
attack learnt on LBS and evaluated on other datasets. Ref is the average
reference transcription length.

Beyond transferability across data distributions, we also investigate how well
the universal acoustic adversarial attacks transfer across different speech
processing tasks. As the multilingual Whisper models can be instructed to
perform transcription or speech translation, we evaluate how effective the
adversarial segment (trained on Whisper for transcription) is in muting
Whisper when used for speech translation. Table 6 presents attack results for
speech translation from French (fr), German (de), Russian (ru) and Korean (ko)
to English, from the Fleurs dataset. Two main trends can be identified. First,
the attack transfers extremely well for the smaller Whisper models, with
attack success rate greater than 94%, but for the larger models the success
rate can drop to less than even 20%. Second, it appears that the ‘further’ the
source language from English, the lower the success rate, e.g., the attack
transfers better for French than Korean in general. As would be expected, this
demonstrates that although the universal attack segment (the acoustic
realization of <endoftext>) is extremely effective for muting Whisper for
English transcription, its potency is increasingly limited as we transfer to a
different speech task (translation) for distant languages.

Model | Metric | fr | de | ru | ko
---|---|---|---|---|---
Ref |  $\varnothing$ (%) | 0.0 | 0.0 | 0.0 | 0.0
| asl | 25.3 | 21.5 | 19.3 | 14.7
tiny |  $\varnothing$ (%) | 99.9 | 94.6 | 96.8 | 94.2
| asl | 0.00 | 0.82 | 0.85 | 1.09
base |  $\varnothing$ (%) | 73.1 | 70.0 | 34.1 | 7.9
| asl | 6.42 | 6.20 | 13.05 | 8.03
small |  $\varnothing$ (%) | 53.4 | 59.1 | 39.2 | 65.7
| asl | 5.01 | 4.45 | 6.11 | 1.68
medium |  $\varnothing$ (%) | 10.5 | 50.7 | 21.7 | 15.5
| asl | 13.04 | 4.44 | 14.46 | 8.18
Table 6: Attack transferability across tasks: the percentage of successfully
‘muted’ speech samples, $\varnothing$, and the Average Sequence Length (asl)
of generated transcripts with the universal acoustic adversarial attack learnt
on LBS for the task of transcription and evaluated on the Fleurs dataset for
the task of speech translation to English. Results are presented for the
multi-lingual Whisper models.

#### Ablations on Imperceptibility.

To ensure the universal acoustic adversarial segment is not easy for a human
listener to detect, we set the maximum amplitude to $\epsilon=0.02$ and the
length of the segment to 0.64-seconds. In this section we explore how much
stricter these imperceptibility constraints can be made during the training of
the universal acoustic attack segments. Figure 3 shows how the attack success
percentage, $\varnothing$ (successfully mute Whisper) changes as the audio
segment length is decreased from 0.64-seconds. The larger a model, the greater
the decay in attack success. Further, the multi-lingual models tend to have a
much greater decay than their English-only counterparts, with the attack
success rate reaching near 0% for every multi-lingual model for a segment of
16-seconds. Figure 4 equivalently presents the impact of reducing the maximum
amplitude, $\epsilon$. A similar trend (although less clear) arises where the
larger and the multi-lingual variants of the models have a greater drop in
success rate with a smaller $\epsilon$. The relative robustness of the multi-
lingual and larger models in extremely constrained attack settings can perhaps
be explained simply by the fact these models have been trained on more data
and thus it is more difficult to find an adversarial acoustic segment that is
a genuine universal realization of the <endoftext> special token.

Figure 3: Ablation on the universal acoustic adversarial attack segment
length. Figure 4: Ablation on the universal acoustic adversarial attack
amplitude constraint, $\epsilon$.

##  7 Conclusion

This work proposes a highly effective and practical method for ‘muting’
Whisper models, achieving a success rate of over 97%. A universal 0.64-second
adversarial audio segment is trained to represent an acoustic realization of
the <endoftext> token used by Whisper, such that when this audio segment is
prepended to any speech signal, Whisper does not transcribe the speech, i.e.,
the model is ‘muted’. Further, this work finds that this universal acoustic
adversarial segment transfer across different data distributions and can even
transfer to different speech processing tasks. While this result offers a
potential for speech privacy protection, it does also reveal the critical
security implications of foundation models’ susceptibility to adversarial
attacks. As speech processing systems continue to develop, addressing these
vulnerabilities is an important direction for future research to ensure their
reliability in real-world applications.

##  8 Limitations

We identify the following potential limitations of our work:

  * •

The scope of this work covers specifically Transformer-based Automatic Speech
Recognition (ASR) systems, such as Whisper. However, due to the recent
popularity and performance of Whisper for ASR, this scope is highly relevant
for a large number of modern speech processing applications.

  * •

We demonstrate that the universal adversarial segment can transfer well across
different data distributions and even sometimes languages. It would be useful
for future work to explore the impact on transferability as specific
dimensions of distributional shift are varied in a controlled manner, e.g.
amplitude of speech (long-distance vs close-distance audio); level of
background noise; or even recording conditions.

  * •

The universal adversarial attack, although very effective, it is Whisper model
specific. This is of course very much expected as each model has a very
different audio-space representation. We discuss this in greater detail in
Appendix C. Although we demonstrate that we can learn a universal attack that
is effective for more than one Whisper model (by considering multiple models
during training), a defence in the future could be to simply transcribe the
text using multiple diverse models. However, we argue that this defence is not
only expensive due to linear inference scaling costs, but is extremely
uncommon in currently deployed ASR systems - it is more common to use a single
ASR system. Hence, if a Whisper model is used for ASR, then an adversary can
use the universal acoustic adversarial segment from this work to mute the
model.

  * •

This work focuses on developing an adversarial attack method to mute the
Whisper model. However, we do not explore detection or defence approaches
explicitly. This is a research area for future work. However, we also
emphasize that it is currently very uncommon in many real-world deployed ASR
settings to perform any form of adversarial detection. Therefore, one primary
aim of this work is to raise awareness around the vulnerability of Whisper ASR
systems to muting universal adversarial attacks. We hope this encourages
future research in defence methods where required. Note that our proposed
muting adversarial attack method can also be used positively by users to
protect the privacy of their audio content.

##  9 Risks and Ethics

This work proposes a method to learn a universal acoustic adversarial attack,
where a 0.64-second audio segment can be prepended to any speech signal and
mute Whisper models. There is the risk that this method could be used by an
adversary to conceal the content of speech signals from speech moderation
systems. However, we argue the aim of this work is to raise awareness around
the vulnerability to such muting adversarial attacks of Whisper ASR models
that have been deployed across many speech processing applications. By raising
this issue, we hope to encourage the research community to develop methods
that improve the robustness and reliability of existing and future ASR
systems. Further, the adversarial attack method proposed in this work can also
be used constructively by users in speech privacy settings, where it is
important to protect the content of audio from malicious actors. On the whole,
this research contributes to the rich adversarial attack literature to
encourage the further development of safe models.

##  10 Acknowledgements

This paper reports on research supported by EPSRC Project EP/V006223/1
(Multimodal Video Search by Examples) and Cambridge University Press &
Assessment. We thank Mengie Qian for the preparation of the MGB dataset used
in the experiments in this work.

## References

  * Abdullah et al. (2019) Hadi Abdullah, Washington Garcia, Christian Peeters, Patrick Traynor, Kevin R. B. Butler, and Joseph Wilson. 2019.  Practical hidden voice attacks against speech and speaker recognition systems.
  * Alzantot et al. (2018) Moustafa Alzantot, Bharathan Balaji, and Mani B. Srivastava. 2018.  Did you hear that? adversarial examples against automatic speech recognition.  _CoRR_ , abs/1801.00554.
  * Ardila et al. (2020) Rosana Ardila, Megan Branson, Kelly Davis, Michael Kohler, Josh Meyer, Michael Henretty, Reuben Morais, Lindsay Saunders, Francis Tyers, and Gregor Weber. 2020.  Common Voice: A Massively-Multilingual Speech Corpus.  In _Proceedings of the Twelfth Language Resources and Evaluation Conference_ , pages 4218–4222.
  * Bell et al. (2015) Peter Bell, Mark JF Gales, Thomas Hain, Jonathan Kilgour, Pierre Lanchantin, Xunying Liu, Andrew McParland, Steve Renals, Oscar Saz, Mirjam Wester, et al. 2015.  The MGB challenge: Evaluating multi-genre broadcast media recognition.  In _2015 IEEE Workshop on Automatic Speech Recognition and Understanding (ASRU)_ , pages 687–693. IEEE.
  * Carlini et al. (2016) Nicholas Carlini, Pratyush Mishra, Tavish Vaidya, Yuankai Zhang, Micah Sherr, Clay Shields, David Wagner, and Wenchao Zhou. 2016.  Hidden voice commands.  In _25th USENIX Security Symposium (USENIX Security 16)_ , pages 513–530, Austin, TX. USENIX Association.
  * Carlini and Wagner (2018) Nicholas Carlini and David A. Wagner. 2018.  Audio adversarial examples: Targeted attacks on speech-to-text.  _CoRR_ , abs/1801.01944.
  * Chen et al. (2020) Yuxuan Chen, Xuejing Yuan, Jiangshan Zhang, Yue Zhao, Shengzhi Zhang, Kai Chen, and XiaoFeng Wang. 2020.  Devil’s whisper: A general approach for physical adversarial attacks against commercial black-box speech recognition devices.  In _29th USENIX Security Symposium (USENIX Security 20)_ , pages 2667–2684. USENIX Association.
  * Cheng et al. (2024) P. Cheng, Y. Wu, Y. Hong, Z. Ba, F. Lin, L. Lu, and K. Ren. 2024.  Uniap: Protecting speech privacy with non-targeted universal adversarial perturbations.  _IEEE Transactions on Dependable and Secure Computing_ , 21(01):31–46.
  * Cisse et al. (2017) Moustapha Cisse, Yossi Adi, Natalia Neverova, and Joseph Keshet. 2017.  Houdini: Fooling deep structured prediction models.
  * Conneau et al. (2022) Alexis Conneau, Min Ma, Simran Khanuja, Yu Zhang, Vera Axelrod, Siddharth Dalmia, Jason Riesa, Clara Rivera, and Ankur Bapna. 2022.  Fleurs: Few-shot learning evaluation of universal representations of speech.  _arXiv preprint arXiv:2205.12446_.
  * Das et al. (2018) Nilaksh Das, Madhuri Shanbhogue, Shang-Tse Chen, Li Chen, Michael E. Kounavis, and Duen Horng Chau. 2018.  ADAGIO: interactive experimentation with adversarial attack and defense for audio.  _CoRR_ , abs/1805.11852.
  * Du et al. (2019) Tianyu Du, Shouling Ji, Jinfeng Li, Qinchen Gu, Ting Wang, and Raheem Beyah. 2019.  Sirenattack: Generating adversarial audio for end-to-end acoustic systems.
  * Fan et al. (2020) Wenshu Fan, Hongwei Li, Wenbo Jiang, Guowen Xu, and Rongxing Lu. 2020.  A practical black-box attack against autonomous speech recognition model.  In _GLOBECOM 2020 - 2020 IEEE Global Communications Conference_ , pages 1–6.
  * Gong and Poellabauer (2017) Yuan Gong and Christian Poellabauer. 2017.  Crafting adversarial examples for speech paralinguistics applications.  _CoRR_ , abs/1711.03280.
  * Hernandez et al. (2018) François Hernandez, Vincent Nguyen, Sahar Ghannay, Natalia Tomashenko, and Yannick Esteve. 2018.  TED-LIUM 3: Twice as much data and corpus repartition for experiments on speaker adaptation.  In _Speech and Computer: 20th International Conference, SPECOM 2018, Leipzig, Germany, September 18–22, 2018, Proceedings 20_ , pages 198–208. Springer.
  * Khare et al. (2019) Shreya Khare, Rahul Aralikatte, and Senthil Mani. 2019.  Adversarial black-box attacks on automatic speech recognition systems using multi-objective evolutionary optimization.
  * Kingma and Ba (2015) Diederik Kingma and Jimmy Ba. 2015.  Adam: A method for stochastic optimization.  In _International Conference on Learning Representations (ICLR)_ , San Diega, CA, USA.
  * Li et al. (2020) Zhuohang Li, Yi Wu, Jian Liu, Yingying Chen, and Bo Yuan. 2020.  Advpulse: Universal, synchronization-free, and targeted audio adversarial attacks via subsecond perturbations.  In _Proceedings of the 2020 ACM SIGSAC Conference on Computer and Communications Security_ , CCS ’20, page 1121–1134, New York, NY, USA. Association for Computing Machinery.
  * Lu et al. (2021) Zhiyun Lu, Wei Han, Yu Zhang, and Liangliang Cao. 2021.  Exploring targeted universal adversarial perturbations to end-to-end asr models.
  * Ma et al. (2021) Chen Ma, Li Chen, and Jun-Hai Yong. 2021.  Simulating unknown target models for query-efficient black-box attacks.
  * MacAvaney et al. (2019) Sean MacAvaney, Hao-Ren Yao, Eugene Yang, Katina Russell, Nazli Goharian, and Ophir Frieder. 2019.  Hate speech detection: Challenges and solutions.  _PloS one_ , 14(8):e0221152.
  * Madry et al. (2019) Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. 2019.  Towards deep learning models resistant to adversarial attacks.
  * Meyer et al. (2020) Josh Meyer, Lindy Rauchenstein, Joshua D Eisenberg, and Nicholas Howell. 2020.  Artie bias corpus: An open dataset for detecting demographic bias in speech applications.  In _Proceedings of the twelfth language resources and evaluation conference_ , pages 6462–6468.
  * Neekhara et al. (2019) Paarth Neekhara, Shehzeen Hussain, Prakhar Pandey, Shlomo Dubnov, Julian J. McAuley, and Farinaz Koushanfar. 2019.  Universal adversarial perturbations for speech recognition systems.  _CoRR_ , abs/1905.03828.
  * Olivier and Raj (2023) Raphael Olivier and Bhiksha Raj. 2023.  There is more than one kind of robustness: Fooling whisper with adversarial examples.
  * Panayotov et al. (2015) Vassil Panayotov, Guoguo Chen, Daniel Povey, and Sanjeev Khudanpur. 2015.  Librispeech: an ASR corpus based on public domain audio books.  In _2015 IEEE international conference on acoustics, speech and signal processing (ICASSP)_ , pages 5206–5210. IEEE.
  * Qin et al. (2019) Yao Qin, Nicholas Carlini, Ian Goodfellow, Garrison Cottrell, and Colin Raffel. 2019.  Imperceptible, robust, and targeted adversarial examples for automatic speech recognition.
  * Radford et al. (2022) Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and Ilya Sutskever. 2022.  Robust speech recognition via large-scale weak supervision.
  * Radford et al. (2023) Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and Ilya Sutskever. 2023.  Robust speech recognition via large-scale weak supervision.  In _International Conference on Machine Learning_ , pages 28492–28518. PMLR.
  * Raina et al. (2020) V Raina, MJF Gales, and K Knill. 2020.  Universal adversarial attacks on spoken language assessment systems.  _Interspeech_.
  * Schönherr et al. (2018) Lea Schönherr, Katharina Siobhan Kohls, Steffen Zeiler, Thorsten Holz, and Dorothea Kolossa. 2018.  Adversarial attacks against automatic speech recognition systems via psychoacoustic hiding.  _ArXiv_ , abs/1808.05665.
  * Schönherr et al. (2018) Lea Schönherr, Katharina Kohls, Steffen Zeiler, Thorsten Holz, and Dorothea Kolossa. 2018.  Adversarial attacks against automatic speech recognition systems via psychoacoustic hiding.
  * Taori et al. (2019) Rohan Taori, Amog Kamsetty, Brenton Chu, and Nikita Vemuri. 2019.  Targeted adversarial examples for black box audio systems.
  * Wu and Bhandary (2020) Ching Seh Wu and Unnathi Bhandary. 2020.  Detection of hate speech in videos using machine learning.  In _2020 International Conference on Computational Science and Computational Intelligence (CSCI)_ , pages 585–590. IEEE.
  * Yuan et al. (2018) Xuejing Yuan, Yuxuan Chen, Yue Zhao, Yunhui Long, Xiaokang Liu, Kai Chen, Shengzhi Zhang, Heqing Huang, Xiaofeng Wang, and Carl A. Gunter. 2018.  Commandersong: A systematic approach for practical adversarial voice recognition.
  * Zhang et al. (2017) Guoming Zhang, Chen Yan, Xiaoyu Ji, Taimin Zhang, Tianchen Zhang, and Wenyuan Xu. 2017.  Dolphinatack: Inaudible voice commands.  _CoRR_ , abs/1708.09537.
  * Zheng et al. (2021) Baolin Zheng, Peipei Jiang, Qian Wang, Qi Li, Chao Shen, Cong Wang, Yunjie Ge, Qingyang Teng, and Shenyi Zhang. 2021.  Black-box adversarial attacks on commercial speech platforms with minimal information.  In _Proceedings of the 2021 ACM SIGSAC Conference on Computer and Communications Security_ , CCS ’21. ACM.

##  Appendix A Experimental Details

This section provides greater detail for the experiments in the main paper.

###  A.1 Data

The LibriSpeech dataset (Panayotov et al., 2015) is derived from English
audio-books and consists of a total of nearly 1000 hours of audio (and
transcriptions). In this work, we use specifically the dev-other split (2864
utterances forming 5.3 hours of audio) and the test-other split (2939
utterances forming 5.1 hours of audio). The TED-LIUM3 dataset (Hernandez et
al., 2018) is formed from English-language TED talks, where the test split
consists of 1155 utterances and 2.6 hours of audio. The Multi-Genre Broadcast
(MGB) Challenge (Bell et al., 2015), an evaluation focused on speech
recognition, speaker diarization, and ‘lightly supervised’ alignment of BBC TV
recordings. The challenge training data covered the whole range of seven weeks
BBC TV output across four channels, resulting in about 1,600 hours of
broadcast audio. In addition several hundred million words of BBC subtitle
text was provided for language modelling. The Artie Bias dataset (Meyer et
al., 2020) is a subset of the Mozilla Common Voice Ardila et al. (2020)
corpus, where it was designed to detect demographic bias in speech
applications. The test-split used in this work consists of 1712 utterances
forming 2.4 hours of audio. The Few-shot Learning Evaluation of Universal
Representation of Speech (Fleurs) (Conneau et al., 2022) is a n-way parallel
speech dataset in 102 languages, with 12 hours of speech per language. For
this work we evaluate on the test splits of specifically French (fr), German
(de), Russian (ru) and Korean (ko).

###  A.2 Models

Whisper model checkpoints are available in a range of sizes: Whisper tiny (39M
parameters); Whisper base (74M); Whisper small (244M); Whisper medium (769M);
and Whisper large (1.55B parameters). The Whisper models are available as
English-only (en) or multilingual models. Whisper large is only available as a
multilingual model. The Whisper models can be prompted to do speech
recognition, voice activity detection, as well as speech translation and
language identification for the multi-lingual model variants. This work
considers a range of sizes of Whisper models for speech recognition and the
multilingual versions are also evaluated for speech translation: tiny(.en),
base(.en), small(.en) and medium(.en). The performance of each model, measured
by the Word Error Rate (WER), for each dataset is given in Table 7.

Model | LBS | TED | MGB | Artie
---|---|---|---|---
tiny.en | 12.8 | 5.4 | 24.5 | 18.4
tiny | 15.0 | 6.3 | 29.5 | 20.8
base.en | 9.6 | 4.6 | 19.7 | 13.2
base | 11.0 | 5.0 | 22.0 | 15.3
small.en | 6.7 | 4.3 | 14.1 | 9.2
small | 7.2 | 4.3 | 15.0 | 9.3
medium.en | 5.7 | 4.3 | 12.4 | 7.4
medium | 5.6 | 4.0 | 12.3 | 6.7
Table 7: Whisper Model Performance - Word Error Rate (WER), %.

###  A.3 Attack Train Configuration

Gradient descent based training is used to learn the acoustic adversarial
segment to minimize the loss, which is defined as the negative of the log-
likelihood of the probability defined in Equation 4. Note that the Whisper
model weights are frozen. The training hyperparamaters for learning the
adversarial attack segment are: the use of an AdamW optimizer; a learning rate
of 1e-3; a batch size of 16 (apart from medium(.en), where a batch size of 4
was used); and parameter clipping in each gradient step, to clamp the learnt
attack segment values of each frame to a maximum absolute value of
$\epsilon=0.02$ to satisfy the imperceptibility constraint, as given in
Equation 5. The larger the target Whisper model, the greater the number of
training epochs are required to guarantee a successful universal attack
segment. The following number of training epochs are used for each Whisper
model: tiny(.en) (40 epochs); base(.en) (40 epochs); small(.en) (120 epochs);
and medium(.en) (160 epochs). Note that for the base and base.en models, runs
over 2 seeds and 3 seeds respectively were required to find a universal
adversarial audio segment that was sufficiently powerful (the seed controls
the initialization of the adversarial audio segment during its training).
Further note that it is empirically observed that increasing the number of
training epochs only increases the strength of the universal attack - there is
no risk of overfitting, which is perhaps expected as there are so few values
being learnt for the universal attack segments.

In typical training setups, there is a risk that excessive training steps can
lead to overfitting, compromising test-time evaluation. However, when learning
the universal prepend attack in this work, this risk does not exist, as the
total number of parameters being learnt are only 10,240 parameters for
0.64-second of audio sampled at 16kHz. This is far smaller than the 100s of
millions of parameters typically being trained in the Whisper speech
recognition models. As a result, we find that the universal prepend attacks
learnt in this work transfer perfectly from the development split of the LBS
data on which they are trained, to the test split on which they are evaluated,
as per the metrics $\varnothing$ and asl, used in this paper.

In the main paper we evaluate the Whisper models in their default setting,
where there is no use of the <notimestamps> special token, such that the first
generated token by the model is always <|0.0|>, and only then the text tokens
follow. However, during training/learning of the universal attack, we
initialized $\mathbf{y}_{0}^{*}$ as <startoftranscript> <language> <task>
<notimestamps> and train to predict $y_{1}=\texttt{<endoftext>}$. The fact
that the attack transfers so well from training time to test time (despite the
mismatch in decoder input initialization), suggests that we have learnt a
genuine acoustic realization of the <endoftext> special token.

A further point to note is that we conducted separate experiments to confirm
that when evaluating the adversarial attack, for no sample is the voice
activity detector (used as part of Whisper’s transcription framework)
returning ‘no speech’, i.e., the universal acoustic adversarial segment is a
genuine realization of the <endoftext> special token. It is unlikely the voice
activity detector would ever be activated at evaluation time as during the
training of the universal attack segment the internal voice activity detector
is not present.

###  A.4 Computational Requirements

Experiments were run on the A100 Nvidia GPU hardware. To learn the 0.64-second
universal acoustic adversarial attack using the development split of the LBS
dataset, the number of GPU hours vary with the target model size and the
number of training epochs used per model. Table 8 summarizes the training
epochs (for a successful attack) and the number of subsequent required GPU
hours for each model size. Further note that the medium models required a
maximum batch size of 4 to fit in the GPU RAM, whilst the other models could
afford a batch size of 16.

Model | Epochs | # GPU hours
---|---|---
tiny | 40 | 0.45
base | 40 | 0.92
small | 120 | 2.6
medium | 160 | 8.4
Table 8: A100 GPU hours to learn a universal acoustic adversarial attack per
target model using the development split of the LBS dataset.

###  A.5 Licensing

All datasets used are publicly available or specifically approved for
experiments in this work (MGB3). Our implementation utilizes the PyTorch 1.12
framework, an open-source library. We observe the MIT license under which the
Whisper’s code and model weights are released.

##  Appendix B Complete Experimental Analysis Results

Experimental results in the main paper are presented for eight Whisper models.
However, the results for the attack success analysis (Table 2 and Table 3) and
the saliency analysis (Table 4) are given for only the tiny and medium model.
Here we present the full results on all eight different models for
completeness. The results maintain the same trends as stated in the analysis
in the main paper. The complete attack success analysis results are given in
Table 9 and Table 10, whereas the the complete saliency analysis results are
given in Table 11.

Model | Samples | No Attack | Attack
---|---|---|---
tiny.en | successful | 17.8 | 0.0
failed | 78.5 | 16.1
tiny | successful | 17.8 | 0.0
failed | 74.6 | 11.0
base.en | successful | 17.5 | 0.0
failed | 50.4 | 19.4
base | successful | 17.6 | 0.0
failed | 60.2 | 11.4
small.en | successful | 17.5 | 0.0
failed | 31.4 | 10.5
small | successful | 17.1 | 0.0
failed | 38.7 | 11.7
medium.en | successful | 17.4 | 0.0
failed | 64.8 | 18.9
medium | successful | 17.2 | 0.0
failed | 43.2 | 25.0
Table 9: Average Sequence Length (asl) of generated transcripts for successful attack samples and failed attack samples. A successful sample is where the universal acoustic attack causes the Whisper model to generate a zero-length transcription (perfectly muted). Model | WER | INS | DEL | SUB
---|---|---|---|---
tiny.en | 80.02 | 0.00 | 79.52 | 0.51
tiny | 88.38 | 0.36 | 85.40 | 2.29
base.en | 64.46 | 0.38 | 61.30 | 2.53
base | 89.57 | 1.97 | 81.30 | 4.53
small.en | 75.50 | 0.24 | 66.46 | 8.62
small | 72.95 | 0.40 | 69.02 | 3.23
medium.en | 72.88 | 0.38 | 70.79 | 1.44
medium | 50.76 | 2.70 | 43.75 | 2.94
Table 10: Word Error Rate (WER) and breakdown (insertions, deletions and substitutions) between the transcript generated with no attack and the transcript generated with the universal acoustic attack, for the failed attack samples only. A failed sample is where the universal acoustic attack is unable to make Whisper generate a zero-length transcription. Model | Samples | Adv, $\tilde{s}$ | Speech, $s$
---|---|---|---
tiny.en | successful | $\underset{\pm 264}{617}$ | $\underset{\pm 15.6}{1.12}$
failed | $\underset{\pm 97.1}{61.0}$ | $\underset{\pm 107}{65.8}$
tiny | successful | $\underset{\pm 332}{835}$ | $\underset{\pm 49.0}{4.80}$
failed | $\underset{\pm 33.1}{101}$ | $\underset{\pm 517}{192}$
base.en | successful | $\underset{\pm 1325}{3527}$ | $\underset{\pm 46.8}{6.05}$
failed | $\underset{\pm 198}{343}$ | $\underset{\pm 246}{91.8}$
base | successful | $\underset{\pm 1480}{4946}$ | $\underset{\pm 140}{13.9}$
failed | $\underset{\pm 183}{483}$ | $\underset{\pm 683}{509}$
small.en | successful | $\underset{\pm 1263}{4339}$ | $\underset{\pm 309}{26.6}$
failed | $\underset{\pm 308}{727}$ | $\underset{\pm 619}{375}$
small | successful | $\underset{\pm 1082}{3502}$ | $\underset{\pm 102}{23.1}$
failed | $\underset{\pm 254}{447}$ | $\underset{\pm 395}{356}$
medium.en | successful | $\underset{\pm 1099}{3205}$ | $\underset{\pm 1185}{123}$
failed | $\underset{\pm 33.4}{114}$ | $\underset{\pm 1950}{812}$
medium | successful | $\underset{\pm 1254}{3371}$ | $\underset{\pm 548}{143}$
failed | $\underset{\pm 170}{314}$ | $\underset{\pm 950}{803}$
Table 11: Average saliency for the adversarial segment and speech segment
(across LBS dataset) for successful and failed samples. A successful sample is
where the universal acoustic attack causes the Whisper model to generate a
zero-length transcription (perfectly muted).

##  Appendix C Transferability Across Models

In this section we explore the transferability of the learnt universal
acoustic adversarial attack segments across different Whisper models. Table 12
shows that there is no transferability across model sizes. This is an expected
result: the attack aims to learn an acoustic realization of a non-acoustic
event (<endoftext> token), but the fact that there is no real audio for the
<endoftext> the attack is unable to find a genuine acoustic realization.
Hence, the acoustic realization being learnt is specific realization of the
<endoftext> token of the target model, such that the attack segment is
specific to the parameter states of the target model. As different models have
very different parameters and values, the learnt universal acoustic segments
are distinct and do not transfer across the models. Nevertheless, the
universal acoustic adversarial attack segment could always be trained with
multiple target models and thus would be effective for multiple target models
simultaneously. Future work will explore this notion further.

src | tgt |  $\varnothing$ (%) | asl
---|---|---|---
tiny | base | 0.0 | 17.8
tiny | small | 0.0 | 17.3
tiny | medium | 0.0 | 17.8
medium | small | 0.0 | 17.3
medium | base | 0.0 | 17.8
medium | tiny | 0.0 | 17.9
Table 12: Transferability of universal acoustic adversarial attack learnt on
the source (src) model and evaluated on the target (tgt) model.

##  Appendix D Saliency Analysis Plots

In the results in the main paper, we conduct a saliency analysis as per
Section 5.2, to better understand the mechanism of the adversarial attack for
when it succeeds relative to when it fails. In Table 4 we report the average
saliency for the adversarial segment, $\tilde{s}$ and the average saliency for
the speech signal, $s$. It is also useful to visualize the frame-level
saliency, to understand how the saliency changes from the adversarial segment
per frame to the speech signal. In Figure 5 we have selected two random speech
samples: one for which the universal acoustic attack succeeded, and one for
which it failed. As we would expect, we observe two very different frame-level
saliency patterns. For a successful attack, the saliency is heavily
concentrated in the adversarial segment and then suddenly decays for the
speech signal, whereas for the failed samples, the converse appears to be
true.

(a) Successful Attack

(b) Unsuccessful Attack

Figure 5: Frame-level saliency plot, where the first 0.64-second represents
the universal acoustic attack segment and the remainder is a randomly sampled
speech signal (truncated to a total length of 3 seconds) for the target model
Whisper medium.en was un/successfully muted by the universal adversarial
attack.

##  Appendix E Spectrogram Plots

Log-mel spectrograms give a frequency-time representation of audio signals in
a manner that can help to interpret the nature of the audio signal. The main
paper gives an example of a log-mel spectrogram for an audio signal where a
universal acoustic segment (learnt for the Whisper medium model) has been
prepended to a specific speech signal. For reference, in this section we
provide the remaining spectrograms. Figure 6 gives the spectrograms for the
universal acoustic adversarial segments learnt for each target Whisper model,
where the adversarial segment is of length 0.64-seconds and a maximum
amplitude of $\epsilon=0.02$, to satisfy the imperceptibility constraint of
Equation 5. Next, in Figure 7 we present the spectrograms for different
universal adversarial attack segments with a different strictness of the
amplitude constraint, $\epsilon$. As would be expected, the stricter the
constraint the lower the relative power of the adversarial segment relative to
the speech signal.

(a) tiny.en

(b) tiny

(c) base.en

(d) base

(e) small.en

(f) small

(g) medium.en

(h) medium

Figure 6: Mel spectrogram of universal acoustic segment (0.64s) prepended to a
random speech sample from LBS dataset (truncated to a total length of 3s) for
different target Whisper models.

(a) $\epsilon=0.02$

(b) $\epsilon=0.01$

(c) $\epsilon=0.005$

Figure 7: Mel spectrogram of universal acoustic segment (0.64s) prepended to a
random speech sample from LBS dataset (truncated to a total length of 3s) for
different amplitude constraints $\epsilon$ for the target model Whisper
tiny.en.


[] [1]

[] [1]

[]

[]

[cor1]Corresponding authors

# Optical Inspection of the Silicon Micro-strip Sensors for the CBM Experiment
employing Artificial Intelligence

E. Lavrik<EMAIL_ADDRESS>Facility for Antiproton and Ion Research in Europe,
Planckstraße 1, 64291 Darmstadt, Germany  GSI Helmholtzzentrum f$\ddot{u}$r
Schwerionenforschung GmbH, Planckstraße 1, 64291 Darmstadt, Germany  Helmholtz
Forschungsakademie Hessen f$\ddot{u}$r FAIR (HFHF), Max-von-Laue-Straße 12,
60438 Frankfurt am Main, Germany  M. Shiroya<EMAIL_ADDRESS>H.R. Schmidt  A.
Toia  Institut f$\ddot{u}$r Kernphysik, Goethe Universit$\ddot{a}$t Frankfurt,
Max-von-Laue-Straße 1, 60438, Frankfurt am Main, Germany  J.M. Heuser
Physikalisches Institut, Universit$\ddot{a}$t T$\ddot{u}$bingen, Auf der
Morgenstelle 14, 72076, T$\ddot{u}$bingen, Germany

###### Abstract

Optical inspection of 1191 silicon micro-strip sensors was performed using a
custom made optical inspection setup, employing a machine-learning based
approach for the defect analysis and subsequent quality assurance.
Furthermore, metrological control of the sensor’s surface was performed. In
this manuscript, we present the analysis of various sensor surface defects.
Among these are implant breaks, p-stop breaks, aluminium strip opens,
aluminium strip shorts, surface scratches, double metallization layer defects,
passivation layer defects, bias resistor defects as well as dust particle
identification. The defect detection was done using the application of
Convolutional Deep Neural Networks (CDNNs). From this, defective strips and
defect clusters were identified, as well as a 2D map of the defects using
their geometrical positions on the sensor was performed. Based on the total
number of defects found on the sensor’s surface, a method for the estimation
of sensor’s overall quality grade and quality score was proposed.

###### keywords:

Silicon micro-strip sensors
Optical inspection
Convolutional deep neural networks
Quality assurance
Quality checks

##  1 Introduction

Silicon sensors play a central role in modern High Energy Particle Physics
experiments for tracking and momentum determination of the charged particles
due to their fast response, high radiation tolerance, and good spatial
resolution. At the same time, the complexity of the experiments and limited
access during the detector operation puts stringent requirements onto the
Quality Assurance (QA) of the silicon sensors. We already have reported on the
devices and methods to perform automated sensor defect searches [1]. In this
manuscript, we report on the application of these procedures to a large sample
of the sensors manufactured by Hamamatsu Photonics for the Compressed Baryonic
Matter (CBM) experiment, comprising 1191 pieces overall. CBM, a fixed-target
heavy-ion physics experiment at the FAIR facility in Darmstadt, Germany, is
designed to reconstruct heavy ion collisions at very high collision rates up
to 10 MHz. Its main tracking detector, the Silicon Tracking System (STS), is
currently under construction, comprising 876 detector modules made out of
double-sided silicon micro-strip sensors of various sizes: 6.2 $\times$ 2.2
cm2, 6.2 $\times$ 4.2 cm2, 6.2 $\times$ 6.2 cm2, 6.2 $\times$ 12.4 cm2, whose
further specifications are detailed in [8, 9]. The compact design of the
tracking system will not allow for extensive maintenance and repair at the
sensor level after the module assembly and installation inside the dipole
magnet. Thus, as a first step in building the silicon tracker, a rigorous
sensor QA is mandatory.

Hamamatsu Photonics, the producer of the silicon micro-strip sensors, could
carry out a certain set of basic quality checks, i.e., detection of pin-holes
and aluminum strip defects, and could guarantee the total amount of defective
strips below 2$\%$.

However, other sensor defects can not be detected by the manufacturer’s
quality control, notably these defects can only be detected with an optical
scanning of the sensor’s surface using a microscope or with a set of specially
designed electrical quality checks [2]. Among these are implant strip breaks,
double metallization layer defects, p-stop implantation breaks, bias resistor
defects, etc. Apart from these defects, active or improper handling of the
sensors might also affect their surface integrity, e.g. introduce surface
scratches, lead to deposition of dust grains, etc. Therefore, it is important
to get a full view of all these defect types before assembling silicon sensor
modules. In the past, the ATLAS and CMS experiments at Large Hadron Collider
(LHC) [3, 4] have reported on their quality control procedures involving
electrical quality inspection along with visual inspection for their
respective large silicon trackers [5, 6, 7].

Figure 1: Optical inspection setup located in a clean room of the Detector
Laboratory department at GSI.

Figure 2: Source images of surface defects taken from the optical inspection
setup. Top panel: Implant break, Aluminium short, Aluminium open, P-stop break
(From left to right) Bottom panel: Double metal line break, Resistor fault,
Passivation open, Scratch (From left to right).

Figure 3: Various surface defects on the silicon sensor identified using
convolutional deep neural network.

In our work, we present a full statistical analysis of the various defect
types observed on the surface of 1191 sensors scanned in an automated fashion
on a dedicated inspection setup and analyzed with advanced methods using
Computer Vision (CV) and Convolutional Deep Neural Networks (CDNNs) [14]. We
also report on the sensors’ metrological measurements, i.e., warp measurement
of the various sensor variants. We believe that this comprehensive analysis
can help optimizing future QA procedures and inspection protocols for similar
applications.

The surface scanning of every sensor was performed using a custom made,
automated optical inspection setup shown in Figure 1. The sensor is placed on
the XY-stage which is connected to the vacuum system to hold the sensors
during their inspection. The XY-stage can move linearly with the help of
Fauhaber/Movtec® SMC-300 servo motors within a range of 200 mm and 70 mm in X
and Y direction respectively, with a precision 0.3 $\mu$m. Detailed technical
information about the setup, sensor alignment procedure and the scanning
process has already been discussed in [1, 10].

Figure 2 shows the surface defects on the silicon sensors which were used as
ground truth for the training of the neural network. The neural network used
in the training the model is based on the Darknet implemented with the
modified R-CNN framework model. It also allows to classify the object and to
locate them in the images against the trained data. The key features of the
neural network are convolutional layers, kernels, activation functions etc [1,
15, 16, 17]. Figure 3 depicts various surface defects detected by the trained
neural network. The efficiency of the CDNN to locate the defects is achieved
to be 91.5 $\%$ with an error of nearly 8.5 $\%$. The equation 1 and 2 are
used to calculate the efficiency and error of the CDNN respectively.

$Accuracy=1-Error$ (1)
$Error=\frac{False~{}Positive+False~{}Negative}{True~{}Positive+True~{}Negative+False~{}Positive+False~{}Negative}$
(2)

##  2 Results

In this section, we present the statistics of defective strips, and the
analysis of the quality grade and quality score based on the total amount of
defect detected on the sensor’s surface. Also, defective strip maps and
defective strip cluster analysis and mapping of the defects of a particular
class based on their positions on the sensor’s surface are presented.

###  2.1 Total defective strips

As per the agreement with Hamamatsu Photonics, up to 30 defective strips
(aluminium defects and pinholes only) per sensor were allowed for the devices
to be accepted by their quality control. However, it is conceivable that there
are more defective strips on the sensor’s surface which had not been detected
by the vendor’s QA.

Therefore, to ensure the sensor’s integrity and guarantee its prescribed
performance it is required to pay attention to the total amount of defective
strips of all kinds on the sensor’s surface. For counting the total number of
bad strips, only four defects, such as implant break, p-stop break, aluminium
open and aluminium short were considered because they lead to a faulty read-
out channel. Also their presence in large amounts on consecutive strips could
create a dead zone on the sensor and the particles traversing the sensor in
such a dead zone could potentially not be detected.

Figure 4: Distribution of the total number of defective strips per sensor
based on the detected defects.

Figure 4 shows the total number of defective strips for individual sensor
variants. It is observed that for the size 6.2 $\times$ 2.2 cm2 and 6.2
$\times$ 4.2 cm2, most of the sensors have less than 15 defective strips per
sensor. On the other hand, for 6.2 $\times$ 6.2 cm2 and 6.2 $\times$ 12.4 cm2
sizes many sensors were found with more than 15 defective strips.

###  2.2 Quality grade and Quality score

For the sensor selection and proper geometrical arrangement, which in turn
ensures the best particle tracking in the final tracking system, we introduce
two metrics. The first one, the quality grade, is based on the overall amount
of the defective strips found on a sensor. The second one, the quality score,
takes into account all kinds of defects found on the sensor and is used to
further rank the sensors within the same quality grade.

For the quality grading scheme, sensors are classified into four different
grade classes by counting the total number of defective strips of the
following types: implant break, p-stop break, aluminium open and aluminium
shorts. The grading scheme is classified into four categories A, B, C and D.
The number of defects allowed per sensor in each grade is presented in Table
1. The sensors in a grade A category are those with very less number of
defects, while those sensors in grade D have large amount of defects. The
resulting grade breakdown for each sensor size is shown in Table 2.

Quality grade A B C D Defects $\leq$ 5 $\leq$ 15 $\leq$ 25 $>$ 25

Table 1: Amount of defect within each of the four grades.

Quality grade Sensor size (cm2) A B C D 6.2 $\times$ 2.2 93 13 2 0 6.2
$\times$ 4.2 296 50 3 0 6.2 $\times$ 6.2 185 144 19 5 6.2 $\times$ 12.4 114
125 107 35 All 688 332 131 40

Table 2: Breakdown of quality grade by sensor size.

It has been found that most of the sensors, 688 specimens, fall in grade A.
This means those sensors in the quality grade A have a minimum amount of the
defects and can be safely used for module production. Moreover, 332 and 131
sensors were found to be in the sensor grades B and C, respectively. Only 40
sensors of the size 6.2 $\times$ 6.2 cm2 and 6.2 $\times$ 12.4 cm2 were
observed in grade D.

The quality score for every sensor was determined, normalized to the value
between 0 and 1, by counting all the possible defect types present on the
sensor’s surface and applying weighted penalty factors to each of those defect
types. Quality score of 0 means the sensors have high number of surface
defects, whereas sensors with a quality score of 1 have a lower amount of
surface defects.

Figure 5: Quality score of the sensor variants 6.2 $\times$ 2.2 cm2, 6.2
$\times$ 4.2 cm2 (Top panel) and 6.2 $\times$ 6.2 cm2, 6.2 $\times$ 12.4 cm2
(Bottom panel).

As can be seen from the Figure 5, the mean value of the quality score for each
sensor variant was attained as 83$\%$, 79$\%$, 70$\%$ and 53$\%$ respectively.
Apparently, the quality score is decreasing with increasing sensor size, with
the amount of defects being evidently proportional to the sensor’s surface
area.

###  2.3 Defective strip maps

There are 1024 strips on each side of the silicon sensor. The defective strip
map represents the distribution of defected strips on the sensor’s surface.

Figure 6: Accumulated defective strip maps for both p-side and n-side of the
sensor variants 6.2 $\times$ 2.2 cm2, 6.2 $\times$ 4.2 cm2 and 6.2 $\times$
6.2 cm2, 6.2 $\times$ 12.4 cm2.

Similarly to the quality grade, only the defective strips of implant break,
p-stop break, aluminium open and aluminium shorts were taken into account.

Figure 6 shows the defective strip maps, individually for both p-side and
n-side, of the sensor variants 6.2 $\times$ 2.2 cm2, 6.2 $\times$ 4.2 cm2, 6.2
$\times$ 6.2 cm2 and 6.2 $\times$ 12.4 cm2. From the defective strip maps, it
is evident that the number of defective strips on the sensor’s p-side was
found to be less than that of the n-side for each sensor variants. On the
other hand, one can deduce that the distribution of defective strips is
relatively uniform without any particular region on the sensor’s surface with
a high number of defective strips. However, there was a substantial amount of
clusters of defective strips found.

###  2.4 Defective strip clusters

Figure 7: Defective strip clusters observed on p- and n-sides of sensor
variants 6.2 $\times$ 2.2 cm2, 6.2 $\times$ 4.2 cm2 (Top panel) and 6.2
$\times$ 6.2 cm2, 6.2 $\times$ 12.4 cm2(Bottom panel).

From defective strip maps shown in Section 2.3, it is clear that some clusters
of defective strips are visible on the sensor’s surface. Therefore, it is
interesting to know how these clusters are sized and distributed.

For finding defective strip clusters, only the defects of implant break,
p-stop break, aluminium open, aluminium shorts on the consecutive strips were
taken into consideration in the algorithm. Figure 7 illustrates the defective
strip clusters for the sensor variants 6.2 $\times$ 2.2 cm2, 6.2 $\times$ 4.2
cm2, 6.2 $\times$ 6.2 cm2 and 6.2 $\times$ 12.4 cm2 respectively. It was found
that the clusters of aluminium open, aluminium shorts defects are of
negligible amount on the surface of all four sensor variants. Furthermore,
only on a few sensors of the sensor size 6.2 $\times$ 2.2 cm2, 6.2 $\times$
4.2 cm2 and 6.2 $\times$ 6.2 cm2 we have found clusters of more than 5 implant
breaks and p-stop breaks defective strips. Many 6.2 $\times$ 12.4 cm2 sensors
were found with more than five consecutive implant breaks and p-stop-breaks
defective strips. Among all, the biggest cluster of implant breaks (28 strips)
and p-stop breaks (24 strips) was found on a 6.2 $\times$ 12.4 cm2 sensor.

###  2.5 2-D map of defects

Figure 8: Defects map of the sensor variant 6.2 $\times$ 12.4 cm2 for p-side
(Left) and n-side (Right).

A two-dimensional map of the defects provides us with a clear view of their
actual position on the sensor surface. The 2D map of defects was obtained by
converting the local pixel coordinate to its global coordinate.

Figure 8 shows the 2D defect map individually for p- and n-side of the sensor
variant 6.2 $\times$ 12.4 cm2. It is striking that some defects like double
metal line and implant break are found to be forming a circular pattern. In a
similar shape, the defects like scratch, p-stop and implant break defects are
observed on the n-side of the sensor’s surface. The pattern is attributed to
the usage of a vacuum tool to handle the sensor during production at the
production site.

##  3 Warp measurement

Metrological measurements such as warp measurement provide information about
the actual geometry and irregularities of the sensor’s surface. In this
section, we present the warp measurement results obtained for different sensor
variants using the same custom made optical inspection setup as shown in
Figure 1. It uses the focus variation technique for the height profile
measurement of the object under test. The conversion ratio motor steps to
micrometer for the focus motor stage was used to convert the values of height
profile obtained in motor steps to real-world measurement. Further details
about the method for contactless metrology measurement has been presented in
[10, 11]. The precise knowledge on the sensors warp is an important
information during the module assembly procedure, where we aim for an
alignment precision of the sensors within $\pm$ 100 µm.

Figure 9: Warp profile of p-side of sensor size 6.2 $\times$ 6.2 cm2.

The warp height is defined as the difference between the maximum and minimum
distances from the reference plane surface. As can be seen in Figure 9 the
actual warp profile of the p-side of one of the sensors variant 6.2 $\times$
6.2 cm2 and its warp height is obtained using the plane fit function.

Table 3 summarizes the warp height for a subset of sensors. It has been
observed that the mean value of maximum warp height for p-side of the sensor
variants 6.2 $\times$ 2.2 cm2, 6.2 $\times$ 4.2 cm2 and 6.2 $\times$ 6.2 cm2
is constrained below 50 $\mu$m. However, for the sensor variant 6.2 $\times$
12.4 cm2, the mean warp height was found slightly above 100 $\mu$m.

Warp height ($\mu$m) Sensor size (cm2) Amount of sensors P-side 6.2 $\times$
2.2 35 23.0 $\pm$ 1.11 6.2 $\times$ 4.2 55 35.2 $\pm$ 1.97 6.2 $\times$ 6.2 50
36.2 $\pm$ 1.94 6.2 $\times$ 12.4 43 107.6 $\pm$ 5.21

Table 3: Mean value of warp height for the different sensor variants.

The warp shape is the result of the mechanical stress induced on the silicon
wafer during different manufacturing and polishing steps [12, 13].

##  4 Summary

Optical inspection is one of the fundamental approaches to detect and study
the silicon sensor’s surface defects. The usage of Convolutional Deep Neural
Networks for automated defect detection and classification yields precision
results with an accuracy rate of 91.5$\%$ with an error of 8.5 $\%$.

The study of defects present on the surface of the sensor allows us to sort
them into different categories by introducing quality grade and quality score
schemes. The quality score was performed by weighting the different defect
types. It was observed that with increasing sensor size, quality score is
decreasing. The quality score is used to rank the sensors within the same
quality grade to distribute the sensors among the tracking stations.

From the defective strips map of all sensor types, we have not observed any
particular sensor area that has a higher number of defective strips. The
defective strips are quite homogeneously distributed on the sensor’s surface.
However, clusters of consecutive defective strips have been found on the
surface of all four sensor variants. From the distribution of the cluster
analysis of consecutive defective strips, it has been found that most of the
sensors from each sensor variant have clusters of less than 5 strips.

The 2D mapping of the defects was performed by converting the defect’s pixel
coordinate to its global geometrical coordinate. It has been observed that
some of the defects appear in circular arrangement originating from a vacuum
tool used to handle sensors during production at Hamamatsu Photonics.

There are as well further quality control activities taking place, e.g.
electrical strip-by-strip characterization and module tests with radioactive
source, which will demonstrate how much the defects observed by the optical
inspection are affecting the sensor’s overall performance. These
investigations are currently ongoing and are yet to be concluded and published
later in the future.

## Acknowledgement

We would like to thank the Detector Laboratory department at GSI for providing
the necessary facility to carry out our work on sensor optical QA. We are in
particular thankful to Dr. Christian Joachim Schmidt and Dr. Andrea Wilms for
their support during this work. From the CBM department, we are grateful to
Dr. Olga Bertini and Marcel Bajdel for helping us with the operation of the
inspection setup for the sensor’s surface scanning. This project has received
funding from the European Union’s Horizon 2020 research and innovation
programme under grant agreement No. 871072 (CREMLINplus). The work was
performed partly under the following grants BMBF 05P16VTFC1. We are also
thankful to the GET-INvolved Programme for International Students and
Researchers and Helmholtz Forschungsakademie Hessen für FAIR (HFHF), GSI
Helmholtzzentrum für Schwerionenforschung, Frankfurt for providing financial
support during the work.

## References

  * [1] E. Lavrik et al., Advanced methods for the optical quality assurance of silicon sensors, NIM A, 922 (2019) 336-344. https://doi.org/10.1016/j.nima.2018.10.210
  * [2] I. Panasenko1 et al., A Custom Probe Station for Microstrip Detector Quality Assurance of the CBM Experiment, Journal of Physics: Conference Series 742 (2016) 012037. https://iopscience.iop.org/article/10.1088/1742-6596/742/1/012037/pdf
  * [3] The ATLAS Collaboration, Technical Design Report for the ATLAS Inner Tracker Strip Detector CERN-LHCC-2017-005. https://cds.cern.ch/record/2257755/files/ATLAS-TDR-025.pdf
  * [4] L. Borrello, A. Messineo, Sensor design for the CMS Silicon Strip Tracker, CMS NOTE 2003/020. http://cdsweb.cern.ch/record/687861/files/note03_020.pdf
  * [5] G. Baumann et al., Quality Assurance of 100 CMS2-OB2 Sensors, LHCb-2004-105, CERN-LHCb-2004-105, 2005. https://inspirehep.net/files/4c02fe4f54ba148cfcb8cad7786e4339
  * [6] J. R. Carter et al., The Silicon Microstrip Sensors of the ATLAS SemiConductor Tracker, Nucl. Instrum. Methods Phys. Res., A 578 (2007) 98-118. https://doi.org/10.1016/j.nima.2007.04.157
  * [7] M. Krammer, Experience with silicon sensor performance and quality control for a large-area detector, Nucl. Instrum. Meth. A 511 (2003) 136-144. https://doi.org/10.1016/S0168-9002(03)01779-0
  * [8] J. Heuser et al., Technical Design Report for the CBM Silicon Tracking System, GSI Report 2013-4, 2013.
  * [9] P. Senger et al., CBM Progress Report 2019, CBM-PR-2019. 10.15120/GSI-2020-00904
  * [10] E. Lavrik, Ph.D. Thesis, Development of quality assurance procedures and methods for the CBM Silicon Tracking System,Universität Tübingen, 2017.
  * [11] E. Lavrik et al., High-precision contactless optical 3D-metrology of silicon sensors, NIM A, 935 (2019) 167-172. https://doi.org/10.1016/j.nima.2019.04.039
  * [12] Y. Ito et al., Warp Measurement for Large-Diameter Silicon Wafer Using Four-Point-Support Inverting Method, Int. J. of Automation Technology, Vol.11 No.5, 2017.
  * [13] D. Quirion et al., Manufacturability and Stress Issues in 3D Silicon Detector Technology at IMB-CNM, Micromachines 2020, 11(12), 1126. https://doi.org/10.3390/mi11121126
  * [14] J. Redmon, You Only Look Once: Unified, Real-Time Object Detection, arXiv:1506.02640v5, 2016. https://arxiv.org/pdf/1506.02640.pdf
  * [15] S. Ren et al., Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks, arXiv:1506.01497v3, 6 Jan, 2016. https://arxiv.org/abs/1506.01497v3
  * [16] Kaiming He et al., Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification, arXiv:1502.01852, 6 Feb, 2015. https://arxiv.org/pdf/1502.01852.pdf
  * [17] C. E. Nwankpa et al., Activation Functions: Comparison of Trends in Practice and Research for Deep Learning, arXiv:1811.03378v1, 8 Nov, 2018. https://arxiv.org/pdf/1811.03378.pdf


††thanks: current address Joint Center for Quantum Information and Computer
Science

# Post-selection-free preparation of high-quality physical qubits

Ben Barber<EMAIL_ADDRESS>Neil I. Gillespie
<EMAIL_ADDRESS>J. M. Taylor<EMAIL_ADDRESS>Riverlane, Cambridge, UK

###### Abstract

Rapidly improving gate fidelities for coherent operations mean that errors in
state preparation and measurement (SPAM) may become a dominant source of error
for fault-tolerant operation of quantum computers. This is particularly acute
in superconducting systems, where tradeoffs in measurement fidelity and qubit
lifetimes have limited overall performance. Fortunately, the essentially
classical nature of preparation and measurement enables a wide variety of
techniques for improving quality using auxiliary qubits combined with
classical control and post-selection. In practice, however, post-selection
greatly complicates the scheduling of processes such as syndrome extraction.
Here we present a family of quantum circuits that prepare high-quality
$|0\rangle$ states without post-selection, instead using CNOT and Toffoli
gates to non-linearly permute the computational basis. We find meaningful
performance enhancements when two-qubit gate fidelities errors go below 0.2%,
and even better performance when native Toffoli gates are available.

Key physical implementations of quantum computers now reliably achieve two-
qubit gate fidelities approaching 99.8% [PhysRevX.11.021058], with multiple
systems reporting fidelities ranging from 99.2% to 99.6% [moskalenko2022high,
PhysRevLett.125.150505, erhard2019characterizing, Google2019]. This leaves
SPAM (state preparation and measurement) errors as a dominant source of error
in many, particularly superconducting, machines. Google’s Sycamore processor
experienced measurement error rates greater than $3$% [Google2019], and IBM
report similar levels of measurement error [ibm_2020] in their devices. Recent
improvements to standard superconducting measurement error rates have not yet
reduced them below 1% [PhysRevX.11.011027], although more is possible using
multilevel qubit encodings [PhysRevX.10.011001]. In contrast, for ions,
neutral atoms, and spins in silicon measurement error rates can be much lower
(cf. IonQ’s 99.3% [wright2019benchmarking] and HRL’s 99.75%
[PRXQuantum.3.010352]), though achieving low rates in dense arrays remain a
substantial challenge [PhysRevLett.127.050501].

There are three broad approaches to reducing the impact of SPAM errors. The
first is to improve the physical processes of preparation and measurement. In
most architectures there are tradeoffs between SPAM errors and speed, as
longer integration or preparation times can improve performance but take
longer; see for example [PhysRevX.10.011001] for the benefits of measuring
slowly up to $T_{1}$ limits. The second is error mitigation techniques which
calibrate and compensate for errors by running the circuit many times and
reconstructing the expected outputs post-facto. This is helpful for NISQ
applications [PhysRevLett.119.180509, Geller_2021] but does little to improve
the entropy extraction critical for quantum error correction, and the
extension of error mitigation into the early fault-tolerant regime
[PRXQuantum.3.010345] requires SPAM errors with similar performance to two-
qubit gate fidelities. Here we examine the third approach: algorithmically
improving the quality of preparation and measurement using quantum logic
gates.

Algorithmic improvement for measurement is straightforward. The simplest
example is used in, e.g., the double species ion clock (see [hume_clock,
jake_clock], and [PhysRevLett.123.033201] for a recent example), where one
readout qubit is used multiple times to estimate the clock qubit—a simple type
of repetition code. More generally, CNOT gates can encode a set of measurement
outcomes in a classical error correcting code on auxiliary measurement qubits,
allowing recovery from sufficiently few measurement errors
[PhysRevA.105.012419]. As these codes have very good performance and efficient
decoders, we recommend their use but do not elaborate on them further here.

A conventional approach to improving preparation using post-selection is the
circuit in Figure LABEL:fig:measurement-based. We prepare two qubits in state
$|0\rangle$, apply a CNOT from the first to the second, then measure the
second in the computational basis. If the measurement result is $0$, we output
the first qubit as a high-quality $|0\rangle$. If not, we reject and begin the
process again.