# Unsupervised Accent Adaptation Through Masked Language Model Correction of
Discrete Self-Supervised Speech Units

###### Abstract

Self-supervised pre-trained speech models have strongly improved speech
recognition, yet they are still sensitive to domain shifts and accented or
atypical speech. Many of these models rely on quantisation or clustering to
learn discrete acoustic units. We propose to correct the discovered discrete
units for accented speech back to a standard pronunciation in an unsupervised
manner. A masked language model is trained on discrete units from a standard
accent and iteratively corrects an accented token sequence by masking
unexpected cluster sequences and predicting their common variant. Small accent
adapter blocks are inserted in the pre-trained model and fine-tuned by
predicting the corrected clusters, which leads to an increased robustness of
the pre-trained model towards a target accent, and this without supervision.
We are able to improve a state-of-the-art HuBERT Large model on a downstream
accented speech recognition task by altering the training regime with the
proposed method.

Index Terms—  self-supervised learning, pre-training, accent invariance,
accent adaptation, speech recognition

##  1 Introduction

Recently, acoustic models pre-trained in a self-supervised fashion have made
big waves in the field of speech processing. Hidden representations learned by
HuBERT [1], wav2vec 2.0 [2], WavLM [3] and others [4, 5] can be used to
achieve strong performance on a variety of speech processing tasks, without
requiring copious amounts of task-specific labeled data. However, prior
research has revealed that while these pre-trained acoustic models are able to
discover informative speech units [6], they are sensitive to domain shifts
between the (pre-)training and target domain [7], to speaker variations [8]
and to contextual factors like co-articulation and allophony [9].

The learned pre-trained representations can be biased towards standardised
canonical data characteristics of the pre-training corpora, and consequently
perform worse on nonnative or accented speakers, as shown for HuBERT in [10].
Following [11], dialectal changes can to some extent be described as a
collection of vocal shifts from a standard pronunciation paradigm. As the
learning process of many self-supervised pre-trained acoustic models crucially
involves quantisation or clustering of latent features in the prospect of
discovering informative speech units, this shift will also be represented in
the discretised features.

In this paper, we propose to improve accent-invariance by learning to remove
the acoustic accent shift from the pre-trained model representations, without
any supervision. We model the standard pronunciation statistics by training a
masked language model [12] on discrete speech units from the source language,
generated by clustering the pre-trained model's hidden features. The extracted
pre-trained discrete clusters of accented speech are then corrected towards
the standard accent by applying an iterative mask-and-decode algorithm to the
language model. Finally, we fine-tune the pre-trained models with these
corrected clusters as self-supervised targets and show improved invariance
towards the target accent.

Our approach requires only unlabeled, non-parallel data from the source
language and the target accent. In this work, we consider North American
English to be the standard source accent, as it is the conventional language
in most work on English speech recognition (e.g. LibriSpeech [13]). We
evaluate the approach for several English accents from the CommonVoice dataset
[14] on an automatic speech recognition (ASR) task. We perform continuous ASR
with fine-tuned HuBERT model features and show improved Word Error Rates (WER)
compared to the uncorrected models on accented speech.

Sentence | WHERE IS …
---|---
Input phones* | W | W | AH | AH | AH | AH | R | R | R | R | IH | IH | IH | Z | Z | Z | Z | Z
Input clusters | 7 | 345 | 181 | 181 | 181 | 181 | 468 | 406 | 406 | 467 | 356 | 356 | 356 | 281 | 281 | 453 | 9 | 9
Masked inputs | 7 | 345 | [M] | [M] | [M] | [M] | 468 | 406 | 406 | 467 | [M] | [M] | [M] | 281 | 281 | 453 | 9 | 9
Output clusters | 7 | 345 | 109 | 109 | 264 | 264 | 468 | 406 | 406 | 467 | 356 | 356 | 356 | 281 | 281 | 453 | 9 | 9
Output phones* | W | W | W | W | EH | EH | R | R | R | R | IH | IH | IH | Z | Z | Z | Z | Z

Table 1: Example of the inputs and outputs of the MLM for part of an accented
utterance in CommonVoice. Note that the input and output phone labels (*) are
not used by the model, but generated afterwards as the most likely cluster to
phone mapping. [M] denotes a mask.

##  2 Preliminaries

###  2.1 BERT

BERT [12] is a language model based on Transformers [15] that learns strong
bidirectional language representations. The main objective is masked language
modeling: part of the input tokens are masked, and the model is optimised to
fill in those masked tokens conditioning on both left and right context with
attention layers. The masks can be applied randomly or in larger spans
comprising several consecutive tokens, as in SpanBERT [16]. The output layer
maps the feature representations back to the vocabulary of tokens. In the
standard BERT model, masked language modeling is combined with a contrastive
next sentence prediction objective. Finally, distilBERT [17] is a smaller,
distilled version of BERT.

###  2.2 HuBERT

As audio and speech are continuous signals, a language model cannot be applied
directly without a discretisation step. HuBERT [1], or Hidden Unit BERT, is a
popular approach for self-supervised speech representation learning from raw
audio. First, an offline clustering step is applied to discover discrete
acoustic units in the audio. Applying the K-means algorithm to MFCC features
for example gives meaningful units. Second, the model is trained to predict
these aligned discrete units from the partly masked output of an acoustic
feature encoder using a bidirectional BERT-like encoder. Hence, HuBERT
intrinsically combines an acoustic and masked language model over continuous
features of units. Finally, the model is improved iteratively by re-training
with better discrete units, obtained from clustering the encoder features of a
previous iteration model. Note that a HuBERT Large model (24 layers) is pre-
trained using the clusters from a 2nd iteration HuBERT Base model (12 layers)
[1].

###  2.3 Adapters

Fine-tuning a large pre-trained model on a new dataset or task can be very
computationally expensive, as the scale of pre-trained models is increasing
rapidly. Instead of fine-tuning the entire model, nearly the same improvements
can be accomplished by only fine-tuning small residual blocks (i.e. adapters)
that are added to every layer [18] and freezing all pre-trained weights,
drastically reducing the number of parameters to optimise. Furthermore,
adapters alleviate catastrophic forgetting of learned knowledge during pre-
training [19], as the pre-trained weights are not altered.

##  3 Approach

This work aims to improve the accent-invariance of pre-trained models using
unlabeled speech from the source and target accent. A masked language model
(MLM) is trained on discrete speech units from the source accent, of which
generally lots of data is available, and afterwards the model is used to
correct the discrete units for an accented utterance.

Fig. 1: Schematic of the proposed method. 1. Features are extracted for an
accented utterance with the HuBERT Base model. 2. K-means quantisation to a
discrete token sequence. 3. Clusters are corrected by the distilBERT model,
trained on US data, using the mask and predict algorithm. 4. HuBERT Large
model is adapted to the accent by optimising small adapter blocks during
continual pre-training with the corrected clusters.

###  3.1 Standardised Unit Language Model

Offline clustering of the hidden features of a self-supervised pre-trained
model leads to informative discrete speech units that correlate well with
actual phonemes as in HuBERT [1], without having seen any labels. Previous
work has shown that these discrete acoustic units are strong building blocks
for modeling spoken language without supervision [20, 21]. To model the
standard pronunciation in the source language, North American English, we
train an MLM on the discretised utterances from that accent. To aid training,
the MLM is initialised from a pre-trained distilBERT
model111https://huggingface.co/distilbert-base-uncased. The token embeddings
are re-initialised as the vocabulary now only consists of 500 clusters id's.
The MLM is optimised with the masked prediction objective, omitting the next
sentence prediction. Following SpanBERT [16], the masks are applied in larger
spans masking several consecutive frames, as one frame corresponds to only
20ms of audio.

###  3.2 Accented Cluster Correction

Accents are characterised by a vocal shift from a standard pronunciation [11].
Vowel shifts are the most obvious examples, but not the only ones. For
example, notice the difference between British and North American
pronunciations for the words dance (/dA:ns/ vs /dæns/), stupid (/”stju:.pId/
vs /”stu:.pId/) and either (/”aI.D@r/ vs /”i:.D/)222According to Cambridge
Dictionary using IPA symbols..

An acoustic model trained in a self-supervised way will identify these sounds
as belonging to separate units, although they represent the same characters.
To achieve accent-invariance, the masked language model trained on the
standard pronunciation should correct this accent shift by changing the
clusters that are affected back to the (statistically most-likely) standard
pronunciation clusters, e.g. the a in the above example dance, taking context
into account.

To correct a sequence of cluster id’s corresponding to an accented utterance
with the MLM, we apply an iterative mask-and-decode algorithm, similar to
Mask-Predict [22]. As the MLM is not conditioned on the audio, conditional
decoding algorithms like Mask-Predict [22], Mask-CTC [23] and BERT-CTC [24]
cannot be trivially applied, and only a small fraction of the input sequence
should be masked, or the model will not be able to recover the original
sentence. Moreover, the masks are applied to groups of identical consecutive
clusters, to correct entire speech units more easily.

The cluster rescoring algorithm works as follows. First, the input cluster
sequence is forwarded through the MLM, and the frame-wise softmax output
scores are used as confidences for the input tokens. Ideally, the
aforementioned accent shifts are given a lower confidence by the LM, as it is
trained to predict an alternative pronunciation for those sounds in a word.
After scoring, consecutive identical clusters are grouped together. The score
of every cluster group is determined by the maximum score of all clusters in
the group, as in [23]. Then, the cluster groups with the lowest scores are
masked, so that at least $N_{k}$ frames are masked. Next, the masked inputs
are digested by the MLM which gives it's predictions for the mask tokens.
Finally, the $M$ most confident predictions of these cluster groups are filled
in. The masks are computed again for the corrected clusters, and the decoding
continues. This procedure is carried out for $K$ iterations, where the total
number of masks adaptively shrinks every iteration $k$ as $N_{k}=\lfloor
N_{max}\cdot\frac{K-k+1}{K}\rfloor$ [23]. The number of masks $M$ that are
filled in every iteration is fixed as $M=\frac{N_{max}}{K}$. The maximum
number of masks is determined by the length $T$ of the utterance and the
masking ratio $P_{mask}$, as $N_{max}=P_{mask}\cdot T$.

The number of iterations $K$ and the masking percentage $P_{mask}$ influence
how much of the input sequence will be altered.

###  3.3 Accent-Invariant Model Adaptation

Previous work [10] has shown that pre-trained HuBERT models benefit from
adaptation to a new accent by continually pre-training on data from the target
accent. To this end, appending accent-specific adapters to the encoder layers
and fine-tuning only the adapter parameters, reaches almost the performance of
full encoder fine-tuning but with a strongly reduced number of parameters.

This work builds upon [10], continually pre-training HuBERT on corrected
cluster sequences by fine-tuning accent-specific adapters, improving accent-
invariance without supervision. We apply Houlsby bottleneck adapters [18]
after every attention and feedforward block in the encoder. The adapters will
learn to normalise the accent shifts that have been corrected by the MLM.

##  4 Experiments

| PER
---|---
Masking strategy | dev | test
Original clusters | 85.69 | 85.76
Adaptive masking cluster groups | 80.55 | 80.61
Adaptive masking phone groups* | 76.09 | 76.14
Adaptive masking phone groups fill-all* | 63.92 | 63.95
Table 2: Phone Error Rates of rescored British English utterances, converted from clusters to phones, w.r.t. a North-American English G2P transcription. For phone groups (*), masks are applied to groups of frames based on their phone class. Fill-all means that all masks are filled in every iteration, instead of a fraction. | ASR w/ LS960 | ASR w/ CV-US
---|---|---
Model / Masking strategy | dev-clean | test-clean | dev-eng | test-eng | dev-clean | test-clean | dev-eng | test-eng
HuBERT Large [1] | 5.77 | 6.01 | 24.39 | 19.82 | 6.84 | 7.22 | 16.79 | 14.26
HuBERT + Adapters [10] | 6.14 | 6.50 | 22.11 | 17.92 | 7.18 | 7.64 | 15.87 | 12.45
Adaptive masking cluster groups | 6.18 | 6.72 | 21.52 | 18.00 | 7.49 | 7.85 | 15.41 | 11.71
Adaptive masking only vowels if $k>3$ | 6.17 | 6.58 | 21.01 | 17.47 | 7.34 | 7.63 | 15.06 | 11.89
Table 3: ASR experiments with WER results. The ASR model is either trained on
LibriSpeech (LS960) or on the North-American English split of CommonVoice (CV-
US), and evaluated on LibriSpeech dev-clean/test-clean or on the British
English (ENG) part of the CommonVoice dev/test set. Row 3 indicates a model
fine-tuned with the outlined algorithm (K=10). Row 4 first applies the masking
algorithm to make basic corrections in the first 3 iterations, and focuses on
vowel shifts by only masking vowels in the following iterations (based on the
cluster id).

| US | ENG | AUS | IND | IRE | SCO
---|---|---|---|---|---|---
Model | dev | test | dev | test | dev | test | dev | test | dev | test | dev | test
HuBERT Large [1] | 17.29 | 18.41 | 16.79 | 14.26 | 10.34 | 13.63 | 24.96 | 25.84 | 14.02 | 10.56 | 15.09 | 16.12
HuBERT + Adapters [10] | 15.09 | 15.82 | 15.87 | 12.45 | 8.93 | 11.97 | 17.87 | 18.04 | 12.20 | 10.54 | 11.67 | 12.93
Ours MLM K=5 | 15.09 | 16.04 | 15.48 | 11.14 | 9.11 | 11.72 | 18.18 | 18.90 | 12.24 | 10.63 | 10.99 | 12.53
Ours MLM K=10 | 15.05 | 16.05 | 15.41 | 11.71 | 8.98 | 11.66 | 17.44 | 17.97 | 12.31 | 10.45 | 11.08 | 12.35

Table 4: ASR experiments with WER results for multiple English accents. The
proposed models are adapted to the target accent in every column separately.
The ASR model is trained on the North-American English (US) split of
CommonVoice, and evaluated on the target accent. The accents are abbreviated
as US (North-American), ENG (British), AUS (Australian), IND (Indian) and SCO
(Scottish).

###  4.1 Experimental Setup

We use a subset of the CommonVoice [14] 7.0 release with 6 well represented
English accents for pre-training. The training set consists of 827k utterances
(611 hours) of English speech, including North American (350k), British
(116k), Indian (71k), Australian (56k), Scottish (11k) and Irish (7k) accents.
There are 1k utterances from each accent for validation and testing.

The baseline HuBERT Large model consists of 7 convolutional feature extraction
layers and 24 Transformer layers, with a total of 316M parameters. It has been
pre-trained on the 60k hour LibriVox corpus with target clusters extracted
from the 9th layer of the 2nd iteration HuBERT Base model (12 Transformer
layers), which was pre-trained on 960 hours of LibriSpeech. The K-means
quantiser that generated the clusters has 500 centroids learned on LibriSpeech
train-clean-100h. Both models and the quantiser are officially released
checkpoints in
fairseq333https://github.com/facebookresearch/fairseq/tree/main/examples/hubert.

For continually pre-training the HuBERT Large model, we insert Houlsby
adapters at every Transformer layer, after the attention block and after the
feed-forward block. The adapters consist of a feedforward down-projection
layer, a ReLU, a feedforward up-projection layer and a layer normalisation
block. The bottleneck dimension is set to 1024, which performed best in [10].
The target clusters are extracted with the same HuBERT Base 2nd iteration
model and the respective quantiser. The learning rate is set to 1.5e-3 and the
model is updated for 30k steps with a linear learning rate schedule with 5k
warm-up steps. We use a batch size of 32 with a maximum of 300k tokens. For a
dataset of 100h, this setup corresponds to 30 full epochs. The adapter-HuBERT
model is fine-tuned on only one accent per experiment.

The distilBERT masked language model consists of 6 Transformer layers. It is
trained on masked token sequences where spans of 10 tokens are masked and on
average 20% of the entire utterance is masked ($P_{mask}=0.2$). Following
BERT, masked tokens have a 20% chance to be replaced with either the original
or a random token, to reduce the train-test mismatch. The MLM is trained for
18 epochs with a batch size of 32, or about 200k steps. The learning rate
follows a linear decay starting from 5e-5. The MLM is trained on the North
American English split of the dataset.

The ASR model consists of a 2-layer BiLSTM (as in the SUPERB challenge [25])
with 512 hidden units, corresponding to around 10M parameters. It is optimised
with a CTC loss [26] to predict characters. The input to the ASR model is a
learnable weighted sum of the pre-trained model's Transformer layer hidden
features. SpecAugment is applied to the inputs and the model is trained for
100k steps with a learning rate of 1e-4. The ASR model is trained on the North
American English split of the dataset, and evaluated on all English accents to
show unsupervised accent adaptation.

###  4.2 Phone Modeling

In this first experiment, we evaluate the cluster corrections of the MLM
towards the North-American accent. As there are only 500 different clusters,
the discrete units are better interpretable. Based on force-aligned phoneme
annotations444https://zenodo.org/record/2619474 for LibriSpeech-100h, we
determine the most likely mapping from the cluster id to the corresponding
phoneme (40 classes). Then, we use a North-American
G2P555https://pypi.org/project/g2p-en/ on the transcripts of the accented dev
and test set to get a reference phoneme transcription for these utterances. We
rescore the original clusters with the MLM and the proposed mask-decode
algorithm to get corrected clusters. Next, we map both the original clusters
and the rescored cluster sequences to their phoneme equivalent based on the
learned cluster-phoneme mapping. Table 1 shows an example of input and output
clusters and their mapping to phonemes. Finally, we evaluate the Phoneme Error
Rate (PER) between the reference G2P phonemes and the original/corrected
phonemes. Results are shown for the British English utterances in the dataset
in Table 2, applying the masking algorithm either to cluster groups or to
phone groups, where the phone group is derived from the cluster id based on
the learned mapping.

###  4.3 Accented Speech Recognition

The main goal of this research is improving the invariance of a pre-trained
model towards an accent without having labeled data for that accent. We train
an ASR model on North-American English data and evaluate it on the other
accents, using the features of the adapted HuBERT models for every accent
separately. For the British English accent, Table 3 displays the results for
different masking schemes, for an ASR model trained on the US part of
CommonVoice (accent mismatch) or on LibriSpeech (accent and dataset mismatch).
For one of the results, the MLM focuses on replacing vowel shifts, if there is
some labeled source data available (as in 4.2). Table 4 shows the WER results
for the experiments on all accents with an ASR model trained on CommonVoice,
where the outlined unsupervised mask-and-predict algorithm is applied to
cluster groups for $K$ iterations.

##  5 Discussion

Phone modeling As the PER w.r.t. the US G2P transcription drops, the clusters
are closer to the North American pronunciation. Empirically, we see the MLM
corrects some vowels and deviant pronunciations, and reduces cluster artifacts
by smoothing similar clusters into one group. However, changing too much can
be harmful, and there is no real ground truth for cluster correctness. There
is a trade-off between smoothing and removing insertions on the one hand, and
keeping temporal detail for ASR purposes on the other hand.

Speech recognition The experiments show that for most accents there is an
improvement over regular fine-tuning of HuBERT with uncorrected clusters of
[10], indicating an improved accent invariance by rescoring the clusters. The
idea of modeling the discrete units with a language model on canonical data
and correct atypical speech is conceptually simple and applicable to many
models. If there is labeled data from the source accent available, the
clusters can be mapped to phones and accent-specific characteristics can be
targeted (e.g. vowel shifts between British and American English).

##  6 Related Work

Several works have built on the idea of combining discretised self-supervised
learning with a language model [20]. In vq-wav2vec [27, 28], the learned
quantiser of a pre-trained wav2vec model is used to generate discrete audio
codes, which are then modelled by a BERT model. Afterwards, the continuous
BERT features are used for downstream tasks like ASR. There are several
differences with our work. First of all, they quantise into 12.5k unique
codes, which makes the BERT training more difficult (128 GPU’s) and the
clusters less interpretable. Furthermore, we use a novel mask-and-decode
algorithm so that the rescored clusters can be used to adapt a HuBERT model or
for discrete lightweight ASR [29]. Finally, we make adaptations to
specifically target the case of accented speech.

Most research on accent robustness has focused on adaptation in multi-accent
speech recognition models [30, 31, 32]. Other recent work [33] has
investigated codebook regularisation to phonemes in wav2vec 2.0 to improve
accent invariance during pre-training without supervision. Finally, [34] uses
discriminators to disentangle accent-specific information during auto-encoder
pre-training.

##  7 Conclusion

This work improves the accent-invariance of a state-of-the-art HuBERT Large
model by correcting the discrete acoustic units towards standard
pronunciation, with a simple adaptation of its training scheme. A masked
language model is trained on the discrete units of a common accent, and
corrects the accented token sequences with an iterative mask-and-decode
algorithm. Fine-tuning the pre-trained model with adapters on these corrected
clusters improves speech recognition for the target accent, without any
labels.

In future work, we plan to explore combining multiple adapters to enable
multi-accent training and be resilient against L2 speakers, and experiment
with bigger language models. It is also interesting to combine speech and MLM
encoders in an end-to-end fashion.

## References

  * [1] Wei-Ning Hsu, Benjamin Bolte, Yao-Hung Hubert Tsai, Kushal Lakhotia, Ruslan Salakhutdinov, and Abdelrahman Mohamed,  ``HuBERT: Self-supervised speech representation learning by masked prediction of hidden units,''  IEEE/ACM Transactions on Audio, Speech, and Language Processing, vol. 29, pp. 3451–3460, 2021.
  * [2] Alexei Baevski, Yuhao Zhou, Abdelrahman Mohamed, and Michael Auli,  ``wav2vec 2.0: A framework for self-supervised learning of speech representations,''  in Proc. NeurIPS, 2020, pp. 12449–12460.
  * [3] Sanyuan Chen et al.,  ``WavLM: Large-scale self-supervised pre-training for full stack speech processing,''  IEEE Journal of Selected Topics in Signal Processing, vol. 16, no. 6, pp. 1505–1518, 2022.
  * [4] Chung-Cheng Chiu, James Qin, Yu Zhang, Jiahui Yu, and Yonghui Wu,  ``Self-supervised learning with random-projection quantizer for speech recognition,''  in Proc. ICML, 2022, pp. 3915–3924.
  * [5] Yu-An Chung et al.,  ``w2v-BERT: Combining contrastive learning and masked language modeling for self-supervised speech pre-training,''  in Proc. ASRU. 2021, pp. 244–250, IEEE.
  * [6] Danni Ma, Neville Ryant, and Mark Liberman,  ``Probing acoustic representations for phonetic properties,''  in Proc. ICASSP. 2021, pp. 311–315, IEEE.
  * [7] Wei-Ning Hsu et al.,  ``Robust wav2vec 2.0: Analyzing domain shift in self-supervised pre-training,''  in Proc. Interspeech, 2021, pp. 721–725.
  * [8] Kaizhi Qian et al.,  ``Contentvec: An improved self-supervised speech representation by disentangling speakers,''  in Proc. ICML, 2022, pp. 18003–18017.
  * [9] Mark Hallap, Emmanuel Dupoux, and Ewan Dunbar,  ``Evaluating context-invariance in unsupervised speech representations,''  in Proc. Interspeech, 2023, pp. 2973–2977.
  * [10] Anshu Bhatia et al.,  ``Don’t stop self-supervision: Accent adaptation of speech representations via residual adapters,''  in Proc. Interspeech, 2023, pp. 3362–3366.
  * [11] John C. Wells,  Accents of English, vol. 2,  Cambridge University Press, 1982.
  * [12] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova,  ``BERT: Pre-training of deep bidirectional transformers for language understanding,''  in Proc. NAACL, 2019, pp. 4171–4186.
  * [13] Vassil Panayotov, Guoguo Chen, Daniel Povey, and Sanjeev Khudanpur,  ``LibriSpeech: An ASR corpus based on public domain audio books,''  in Proc. ICASSP. 2015, pp. 5206–5210, IEEE.
  * [14] Rosana Ardila et al.,  ``Common Voice: A massively-multilingual speech corpus,''  in Proc. LREC, 2020, pp. 4218–4222.
  * [15] Ashish Vaswani et al.,  ``Attention is all you need,''  in Proc. NeurIPS, 2017.
  * [16] Mandar Joshi et al.,  ``SpanBERT: Improving pre-training by representing and predicting spans,''  Transactions of the Association for Computational Linguistics, vol. 8, pp. 64–77, 2020.
  * [17] Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf,  ``DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter,''  in Proc. NeurIPS-EMC², 2019.
  * [18] Neil Houlsby et al.,  ``Parameter-efficient transfer learning for NLP,''  in Proc. ICML, 2019.
  * [19] Steven Vander Eeckt and Hugo Van Hamme,  ``Using adapters to overcome catastrophic forgetting in end-to-end automatic speech recognition,''  in Proc. ICASSP. 2023, IEEE.
  * [20] Kushal Lakhotia et al.,  ``On generative spoken language modeling from raw audio,''  Transactions of the Association for Computational Linguistics, vol. 9, pp. 1336–1354, 2021.
  * [21] Alexei Baevski, Wei-Ning Hsu, Alexis Conneau, and Michael Auli,  ``Unsupervised speech recognition,''  in Proc. NeurIPS, 2021, pp. 27826–27839.
  * [22] Marjan Ghazvininejad, Omer Levy, Yinhan Liu, and Luke Zettlemoyer,  ``Mask-predict: Parallel decoding of conditional masked language models,''  in Proc. EMNLP-IJCNLP, 2019, pp. 6112–6121.
  * [23] Yosuke Higuchi, Shinji Watanabe, Nanxin Chen, Tetsuji Ogawa, and Tetsunori Kobayashi,  ``Mask CTC: Non-autoregressive end-to-end ASR with CTC and mask predict,''  in Proc. Interspeech, 2020, pp. 3655–3659.
  * [24] Yosuke Higuchi et al.,  ``BERT meets CTC: New formulation of end-to-end speech recognition with pre-trained masked language model,''  in Proc. EMNLP, 2022, pp. 5486–5503.
  * [25] Shu wen Yang et al.,  ``SUPERB: Speech processing universal performance benchmark,''  in Proc. Interspeech, 2021, pp. 1194–1198.
  * [26] Alex Graves, Santiago Fernández, Faustino Gomez, and Jürgen Schmidhuber,  ``Connectionist temporal classification: Labelling unsegmented sequence data with recurrent neural networks,''  in Proc. ICML, 2006, p. 369–376.
  * [27] Alexei Baevski, Steffen Schneider, and Michael Auli,  ``vq-wav2vec: Self-supervised learning of discrete speech representations,''  in Proc. ICLR, 2020.
  * [28] Alexei Baevski and Abdelrahman Mohamed,  ``Effectiveness of self-supervised pre-training for ASR,''  in Proc. ICASSP. 2020, pp. 7694–7698, IEEE.
  * [29] Xuankai Chang, Brian Yan, Yuya Fujita, Takashi Maekaku, and Shinji Watanabe,  ``Exploration of efficient end-to-end ASR using discretized input from self-supervised learning,''  in Proc. Interspeech, 2023.
  * [30] Xian Shi et al.,  ``The accented English speech recognition challenge 2020: Open datasets, tracks, baselines, results and methods,''  in Proc. ICASSP. 2021, pp. 6918–6922, IEEE.
  * [31] Yanmin Qian, Xun Gong, and Houjun Huang,  ``Layer-wise fast adaptation for end-to-end multi-accent speech recognition,''  IEEE/ACM Transactions on Audio, Speech, and Language Processing, vol. 30, pp. 2842–2853, 2022.
  * [32] Song Li et al.,  ``End-to-end multi-accent speech recognition with unsupervised accent modelling,''  in Proc. ICASSP. 2021, pp. 6418–6422, IEEE.
  * [33] Chendong Zhao et al.,  ``Learning invariant representation and risk minimized for unsupervised accent domain adaptation,''  in Proc. SLT. 2023, pp. 509–516, IEEE.
  * [34] Yi-Chen Chen et al.,  ``AIPNet: Generative adversarial pre-training of accent-invariant networks for end-to-end speech recognition,''  in Proc. ICASSP. 2020, pp. 6979–6983, IEEE.


# Performance Analysis of Out-of-Distribution Detection on Trained Neural
Networks

Jens Henriksson1, Christian Berger2, Markus Borg3,
Lars Tornberg4, Sankar Raman Sathyamoorthy5, Cristofer Englund3
1Semcon AB, Gothenburg, Sweden, Email<EMAIL_ADDRESS>2University
of Gothenburg and Chalmers Institute of Technology, Sweden, Email:
<EMAIL_ADDRESS>3RISE Research Institutes of Sweden AB, Lund and
Gothenburg, Sweden, Email: {markus<EMAIL_ADDRESS>4Machine Learning and AI
Center of Excellence, Volvo Cars, Gothenburg, Sweden, Email:
<EMAIL_ADDRESS>5QRTech AB, Gothenburg, Sweden, Email:
<EMAIL_ADDRESS>
###### Abstract

Several areas have been improved with Deep Learning during the past years.
Implementing Deep Neural Networks (DNN) for non-safety related applications
have shown remarkable achievements over the past years; however, for using
DNNs in safety critical applications, we are missing approaches for verifying
the robustness of such models. A common challenge for DNNs occurs when exposed
to out-of-distribution samples that are outside of the scope of a DNN, but
which result in high confidence outputs despite no prior knowledge of such
input.

In this paper, we analyze three methods that separate between in- and out-of-
distribution data, called supervisors, on four well-known DNN architectures.
We find that the outlier detection performance improves with the quality of
the model. We also analyse the performance of the particular supervisors
during the training procedure by applying the supervisor at a predefined
interval to investigate its performance as the training proceeds. We observe
that understanding the relationship between training results and supervisor
performance is crucial to improve the model’s robustness and to indicate, what
input samples require further measures to improve the robustness of a DNN. In
addition, our work paves the road towards an instrument for safety
argumentation for safety critical applications. This paper is an extended
version of our previous work presented at 2019 SEAA (cf. [1]); here, we
elaborate on the used metrics, add an additional supervisor and test them on
two additional datasets.

###### Index Terms:

deep neural networks, robustness, out-of-distribution, automotive perception,
safety-critical systems

##  I Introduction

Deep Neural Networks (DNN) constitute a major challenge to safety engineers,
especially for complex and safety-critical functionality such as Autonomous
Driving. Autonomous Driving requires complex perception systems embodying DNNs
for computer vision to process large streams of video, lidar, and radar
sensors. Even though DNNs have revolutionized the field of computer vision
[2], they still pose an enormous challenge for safety verification and
validation since their characteristics and development processes are
completely different compared to conventional software addressed by the
automotive safety standard ISO 26262 – Functional Safety [3]. The behaviour of
a DNN is not explicitly expressed by an engineer in source code following the
principle, where the developer defines an algorithm based on a specification,
but rather instead the developer defines an architecture that learns its
connections. To achieve this learning, developers usually require large
amounts of data complemented with domain-specific labels to train specific
behavior. For autonomous driving, this step usually comprises of data
collection, preprocessing/cleaning, training, and evaluation of large
quantities of annotated camera, lidar, and radar data using Machine Learning
(ML) [4, 5].

###  I-A Background

In January 2019, ISO/PAS 21448 – Safety of the Intended Functionality (SOTIF)
was published to meet a growing industrial need of an automotive safety
standard that is appropriate for ML [6]. A PAS (Publicly Available
Specification) is not an established standard, but a document that closely
resembles what is planned to become a future standard; a PAS is published to
speed up a standardization process in response to market needs. A PAS must be
transformed to a standard within six years, otherwise it will be
withdrawn111https://www.iso.org/deliverables-all.html.

ISO 26262 and SOTIF are meant to be complementary standards: ISO 26262 covers
“absence of unreasonable risk due to hazards caused by malfunctioning
behaviour” [3] by mandating risk reduction activities that is achieved by
following the established V-model. SOTIF, on the other hand, addresses
“hazards resulting from functional insufficiencies of the intended
functionality” [6], e.g., miss-classifications by an object detector in an
automotive perception system. To achieve systematic risk reduction of
functional insufficiencies, ML requires quantitative requirements in addition
to the qualitative instruments that are currently present in ISO 26262.

SOTIF is not organized according to the V-model but instead around four states
defining all possible events: 1) known safe states, 2) known unsafe states, 3)
unknown unsafe states, and 4) unknown safe states. SOTIF presents a process to
minimize the two unsafe states, represented by a flowchart orbiting around
requirements specifications for the functionality under development, where
moving hazards from 3) $\rightarrow$ 2) and from 2) $\rightarrow$ 1)
originates from hazard identification and hazard mitigation, respectively.

###  I-B Problem Domain & Motivation

Following the SOTIF process, we identify that autonomous driving in
environments that differ too much from what the automotive perception was
trained for, constitutes a major hazard; in ML literature, this is referred to
as dealing with out-of-distribution (OOD) samples. In previous work, we have
proposed various DNN supervisors [7] to complement autonomous driving by
detecting such outlier samples as part of a safety cage architecture [8],
i.e., adding a reject option to DNN classifiers when input does not resemble
the training data.

As part of a mitigation strategy for out-of-distribution camera data for
example, we argue that the SOTIF process should result in a safety requirement
mandating the inclusion of a DNN supervisor. However, only a few previous
studies have presented structured evaluations of different DNN supervisors for
image data; most previous work instead focuses on a single supervisor, like,
for example, [9, 10] and [11].

###  I-C Research Goal & Research Questions

In this paper we evaluate three supervisor algorithms: a baseline algorithm
[12], OpenMax [9] and ODIN [10]. In this context, their algorithms have been
adjusted to act as a supervisor and provide an anomaly score of each input
sent to the model to comply with the test setup defined in [7]. Our evaluation
method consists of applying three supervisors on four established DNNs
architectures, namely VGG16 [13], DenseNet-121 [14] and two versions of Wide
ResNets [15]. All networks are trained with identical hyperparameters on the
CIFAR-10 dataset [16] and tested on three different outlier sets. We address
the following two research questions:

  * •

How does the outlier detection performance change with improved training of
the DNNs under supervision?

  * •

How does overfitting of the DNNs under supervision affect the supervisor
performance?

###  I-D Contributions

We find that the performance of the supervisor is almost linear with regards
to the network performance. However, the design and tuning of supervisors can
be a sensitive development process, especially when applied at different
stages during training where network parameters have been shifted. This paper
is an extended version of our previous work presented at 2019 SEAA (cf. [1]);
here, we elaborate on which metrics we use, add an additional supervisor to
the comparison, as well as extend the testing to one additional image
classification dataset and one dataset made out of noise.

###  I-E Structure of the Article

The rest of the paper is organized as follows: Section II introduces related
work on automotive perception and DNN supervisors. In Section III, we describe
our method to evaluate and compare DNN supervisors. Section IV presents our
results and in Section V we discuss our findings in the light of safety for
autonomous driving. Finally, Section VI concludes our paper and outlines
directions for future work.

##  II Related Work

Recent work on AI/ML has gained a lot of attention for the break-through of
large-scale pattern recognition for various applications ranging from popular
old board games, over text and image analysis, up to complete end-to-end
solutions for autonomous driving. Even though the reported results appear
fascinating and underline the potential of AI/ML, the challenges of
commercializing such solutions into safe products is the apparent next
challenge to overcome for a successful roll-out. Hence, recent works are now
exploring and discussing how to test deep learning models. Fei et al. [17] and
the succeeding study from Guo et al. [18] suggest to evaluate testing coverage
of models by evaluating neuron coverage. While such approaches show
similarities to code, statement, or branch coverage in software testing as
defined in ISO 26262 [3], they fall short for evaluating the robustness of
DNNs as increased neuron coverage as achieved by artificially constructed test
samples does not necessarily correlate with out-of-distribution samples that a
DNN would experience in reality. Furthermore, the improvements of these
approaches reported by the authors are in the range of 2-3%. Kim et al. [19]
suggested a different approach, where they analyzed the influence of a
particular stimulus to a DNN with the purpose of providing support when
designing specific test data to evaluate the performance of a DNN.

Czarnecki and Salay postulate in their position paper [20] a framework to
manage uncertainty originating from perception-related components with the
goal of providing a performance metric. Their work provides initial thoughts
and concepts of how to describe uncertainty that is present in ML-based
approaches. However, in contrast to our work here, uncertainty originating
from the trained model itself is only captured briefly and not in a
quantitative approach as outlined here.

A different approach is presented by Ma, Driggs-Campbell, and Kochenderfer,
who investigate in their work the problem of robust learning for autonomous
vehicle control by formulating it as a two-players-game between the autonomous
system and disturbances [21]. They introduce two new concepts to this game
theoretic formulation. In Ma et al. [22], DeepGauge is introduced, which
proposes several multi-granularity testing criteria for DL systems aiming at
rendering a multi-faceted portrayal of the testbed. Initial results from
experiments show that their approach might be applicable to evaluate the
testing adequacy of DNNs.

Several recent papers have discussed the topic of out-of-distribution
detection. Hendrycks and Gimpel [12] described an approach to monitor the
performance of a DNN-based system by evaluating the Softmax layer. This is
used to predict if the network is classifying a given sample correctly; in
addition, they also use this feature as a discriminator between in-
distribution and out-of-distribution from the training data.

Similarly to Hendrycks and Gimpel, Liang et al. [10] also use the Softmax
layer in their approach named ODIN but without the need for any changes in a
given model. Bendale et al. [9] proposed a new model layer OpenMax, which is a
replacement for the Softmax layer. This new layer compares the input sample
towards an unknown probability element, adapted from meta-recognition of the
penultimate layer, thus allowing the layer to learn the behavior of the
training set, and correlated classes.

Carlini and Wagner [23] showed how adversarial attacks can break defensive
distillation [24], a recent approach tailored to reduce the success rate of
adversarial attacks from 95% to 0.5%. Their adversarial attack is tailored to
three distance metrics, which are commonly used when generating adversarial
samples.

Finally, in contrast to only monitoring the Softmax layer, Tranheden and
Landgren [25] compared and extracted information from five different detection
methods to craft their own. Their combined solution suggests to supervise
activations on multiple layers as various sub-models, since the behavior
varies depending on input, thus would benefit from model or class specific
supervisors.

##  III Evaluation Method

The scope of this work consists of comparing three implementations of
supervisors on neural networks, which are particularly trained for image
classification. To achieve a substantiated comparison, the supervisors will be
tested on four image classification models. Additionally, as the training
results may vary on stochastic effects such as weight initialization or random
selection of mini batch gradient descent, we believe by testing every
$10^{th}$ epoch, small variations in the model weight will show how the
performance of the supervisor changes. It is of interest to study this
behavioral change as small perturbations both in network quality or input
space can drastically change the model output. In this section, the model
training, datasets and evaluation metrics are described in detail.

Model training: For our experiments, four state of the art models, based on
three architectural designs are trained from scratch. The first architecture,
VGG16 [13], is a convolutional neural network (CNN) with all layers connected
in series. The second model is the DenseNet-121 architecture as described in
[14], with internal dense blocks that refer to sets of convolutional layers,
where previous layer outputs are concatenated to further improve informational
flow. The last architecture is the Wide Residual Network (WRN) [15], who
introduced a study of depth and width for residual networks [26]. A residual
block consists of a layer that concatenates the feature input such as
$x_{l}=x_{l-1}+f(x_{l-1})$ where $f(\cdot)$ refers to a set of layer
permutations of the input. For our experiments, two model setups are selected
for WRN, namely WRN-28-10 and WRN-40-10, where the depth factor is set to 28
and 40 respectively, and width factor of 10 for both networks.

All four networks are trained with identical hyperparameters, for 200 epochs
with stochastic gradient descent with momentum. The learning rate is
initialized at $10^{-}1$ and decreased by a factor 10 after 100 and 150
epochs. The model weights are stored after every $10^{th}$ epochs, as well as
the best model with highest accuracy on the training set, thus yielding 21
versions each all models.

Datasets: During evaluation, as well as in training the CIFAR-10 dataset will
be used. For the evaluation phase, the test set is used as in-distribution
(i.e., data samples, which are not excluded). For out-of-distribution samples,
we use two datasets with natural images, and one dataset consisting of
artificially constructed noise. All datasets are described more specific in
the following:

  * •

Tiny ImageNet: Tiny ImageNet consists of a selection of images from the
ImageNet [27] dataset that is rescaled to $64x64$. The dataset consist of 200
classes. We further downscale the validation images to match CIFAR-10 image
dimensions, as well as use the validation part of the dataset that consists of
10,000 images.

  * •

SVHN: The Street View House Numbers (SVHN) [28] dataset contains more than
600,000 labeled images of small cropped digits with the goal of recognizing
digits and numbers in natural scene images. The house numbers are obtained
through Google Street View imagery. For our dataset, we use the first 10,000
images from the training set.

  * •

FakeData: The Pytorch222An open source machine learning framework:
https://pytorch.org FakeData dataset generates random samples, where each
pixel is sampled from an i.i.d Gaussian distribution. We generate 10,000
random RGB images with mean 0.5 and standard deviation 1.

Algorithm 1 The Anomaly Score extraction from out-of-distribution methods.
Each method computes an activation vector $\textbf{v}(\textbf{x})$ for a given
input sample $\mathbf{x}$. The activation vector is then normalized into a
probability distribution, that is used to compute the anomaly score.

1:threshold $\epsilon$

2:Out-of-distribution method $F(\cdot)$

3:Compute the output vector v(x) for input sample x with the Out-of-
distribution method $F(\cdot)$

4:Let $P$ be the Softmax of output vector v for all classes $j=1,..,N$

5:
$P(y=j|\textbf{x})=\frac{e^{\textbf{v}_{j}(\textbf{x})}}{\sum_{i=1}^{N}e^{\textbf{v}_{i}(\textbf{x})}}$
(1)

6:Let the anomaly score be $A(\textbf{x})$ = 1 - argmax $P(y=j|\textbf{x})$
where $argmax$ is the function argument that maximizes function value.

7:Let discriminator $D\\{0,1\\}$ be defined as
$D=\left\\{\begin{matrix}\begin{matrix}1&ifA(\textbf{x})<\epsilon\\\
0&otherwise\end{matrix}\end{matrix}\right.$ (2)

Figure 1: The training results for the four models: The orange line represents
the testing accuracy achieved during training, and the blue dashed line
represents the cross entropy loss. All models are trained with stochastic
gradient descent with momentum and $10^{-1}$ learning rate that is reduced by
a factor 10 after 100 and 150 epochs.

Metrics: Based on recent studies [7, 10], we extend on their selection of
metrics and select four metrics that show the discriminative rate of the
supervisor as well as the overall performance of the network at hand. For our
experimental setups, we define a True Positive (TP) as an outlier correctly
being selected for removal and a False Positive (FP) as an inlier that has
been incorrectly removed. Additionally, in this context True Negatives (TN)
refers to inlier samples being accepted and False Negatives (FN) being
outliers that has not been rejected. Thus, True Positive Rate (TPR) and False
Positive Rate (FPR) can be computed as $TPR=TP/(TP+FN)$ and $FPR=FP/(FP+TN)$.
The metrics are explained and motivated below.

  * •

AUROC: Area Under the Receiver Operating Characteristics curve, which plots
the true positive rate (TPR) against the false positive rate (FPR). The curve
is received by creating various thresholds in the anomaly score of the
supervisor.

  * •

FPR at $95\%$ TPR (FPR95): This metric can be interpreted as the probability
that an inlier sample will be removed when the majority (95%) of out-of-
distribution samples are rejected.

  * •

CBPL: Coverage break-point at performance level. The metric relates how
restrictive the threshold has to be to return to the original accuracy that
was received based on test set.

  * •

Cov10: Coverage at $10\%$ Error rate, refers to the coverage break-point but
at a given accepted error rate. This metric does not penalize well-scoring
models due to requiring a more restrictive threshold.

Out-of-distribution methods: This paper studies three methods constructed to
detect out-of-distribution samples. The experiments work on the premise of
Algorithm 1, where each method returns an activation vector that can be used
to compute an anomaly score. In the algorithm, $F(\cdot)$ refers to the OOD
method, and that the threshold $\epsilon$ is varied between the lowest and
highest anomaly score to receive the ROC-curve. It is worth noting that some
of the methods required tuning, which is done once per method, thus not being
optimized per model basis. Additionally, the methods are not built for
thresholding the full anomaly score spectra, but rather finding one threshold.
Thus, results may look worse due to not taking into account the best threshold
per model. The three methods used in this paper are described in the
following:

  * •

Baseline [12]: Hendrycks et al. showed baseline performance by looking at the
softmax output layer of a neural network. They analyzed several inlier and OOD
datasets and showed that by only looking at the most plausible prediction, a
majority of OOD samples could be rejected.

  * •

ODIN [10]: Liang et al. proposed a method that utilizes the gradients inside a
network as well as temperature scaling on the softmax layer to construct a
small perturbation on the image. They showed that their perturbation would
hurt the inlier samples more than OOD samples, thus utilizing this behavior to
differentiate between the distributions.

  * •

OpenMax [9]: Bendale et al. proposed a method to adjust the softmax layer by
incorporating information from the training set. By utilizing meta-recognition
algorithms their approach constructs a Weibull-distribution based on the
training samples that can be used as a distance metric between the sample in
question towards the training set. This information is used to compute a
probability of an unknown unknown class. Additionally, they are not only doing
this revision for the top class, but rather the $n$ top classes, since most
inlier samples follow the same order of top classes.

TABLE I: The performance for supervisor on the three Out-of-Distribution datasets, with the best performing (test set accuracy wise) model. All metrics are shown in percentages, where higher indicates better. For each metric, ODIN, OpenMax and Baseline are presented in order.  |  | ODIN / OpenMax / Baseline
---|---|---
Model |  | Out-of-distribution
---
dataset
AUROC $\uparrow$ | FPR at 95% TPR $\downarrow$ | CBPL $\uparrow$ | Cov10 $\uparrow$
| Tiny ImageNet | 87.39 / 84.80 / 86.68 | 55.75 / 67.07 / 57.22 | 43.34 / 32.52 / 37.98 | 49.25 / 48.34 / 50.14
DenseNet-121 | SVHN | 97.23 / 93.19 / 95.34 | 10.37 / 12.03 / 12.83 | 51.95 / 49.25 / 49.54 | 55.16 / 54.13 / 55.37
| FakeData | 95.45 / 95.23 / 99.00 | 7.68 / 9.54 / 3.28 | 51.75 / 45.23 / 51.93 | 54.79 / 45.23 / 56.96
| Tiny ImageNet | 86.46 / 83.45 / 84.85 | 61.03 / 72.31 / 59.69 | 45.15 / 36.73 / 41.24 | 49.17 / 47.58 / 48.35
VGG 16 | SVHN | 88.18 / 85.37 / 89.26 | 45.42 / 66.91 / 31.37 | 47.04 / 41.77 / 46.34 | 50.18 / 49.78 / 51.16
| FakeData | 5.51 / 5.95 / 32.97 | 99.76 / 99.63 / 97.68 | 8.18 / 7.00 / 8.86 | 10.30 / 10.44 / 13.15
| Tiny ImageNet | 89.07 / 87.47 / 87.64 | 48.07 / 51.78 / 46.33 | 43.65 / 39.97 / 41.70 | 50.97 / 50.72 / 50.68
WRN-28 | SVHN | 83.50 / 84.38 / 88.82 | 41.59 / 43.28 / 26.74 | 40.96 / 38.68 / 44.70 | 46.97 / 47.38 / 51.05
| FakeData | 99.77 / 95.56 / 97.71 | 0.54 / 8.47 / 3.95 | 52.90 / 48.14 / 51.77 | 56.66 / 48.14 / 56.51
| Tiny ImageNet | 87.79 / 85.39 / 86.48 | 62.87 / 70.66 / 58.10 | 43.28 / 33.99 / 39.48 | 50.58 / 49.23 / 50.32
WRN-40 | SVHN | 89.35 / 86.46 / 89.54 | 34.37 / 49.24 / 27.96 | 46.15 / 39.83 / 44.90 | 50.97 / 49.85 / 51.74
| FakeData | 83.53 / 87.51 / 83.81 | 26.71 / 19.74 / 23.51 | 45.15 / 46.27 / 44.65 | 49.08 / 51.84 / 49.95

##  IV Experimental results

Model training: In Fig. 1, the training process can be seen for each of the
four models. The training is conducted with image augmentation that consists
normalization, random cropping, and flipping. These augmentations improve
generalization of the models by forcing it to learn multiple representations
per input. Additionally, batch size was set to 128 and learning rate to
$10^{-1}$ that decreased after 100 and 150 epochs by a factor 10.

The training process contains large variations of accuracy during the first
100 epochs, which is expected due to the high learning rate. Additionally, it
benefits the comparison of supervisors since it indicates variations in
weights and biases inside the network. As the learning rate decreases, we can
see rapid convergence up until a period of overfitting that is occurring
roughly in the same interval of epochs for all models. While overfitting is
generally not desired for training DNNs, we embrace it as it unravels how it
affects the perform ace of supervisors when applied to overtrained models.

Parameter selection: Both ODIN and OpenMax require parameters for their
algorithm to work. As we are interested in a generalized supervisor only one
parameter setup exist per model, which is independent of out-of-distribution
dataset. This way, one parameter search is per model followed by a selection
of the setup that has best AUROC on average. The parameter search is conducted
through a grid search. For ODIN, we iterate over the temperature scaling set
to 50, 100, 200, 500, 700, 1000, 1500, and 2000; and for the noise gradient we
create 21 linear steps between 0.0 and 0.004. For OpenMax, we iterate over
tail length set to 5, 10, 20, 50, and 100; and the top alpha classes are set
to 1, 2, …10.

###  IV-A Results on best epochs

In Table I, the metric performance for the three supervisors can be compared
per OOD-dataset. Regarding the AUROC scores, it is noteworthy that Openmax
does not perform better than baseline. This is due to OpenMax being optimized
for a single threshold, i.e., optimized to create and use their unknown
unknown class score only if it is the most probable outcome. We, however, use
it for the full threshold comparison as described in Algorithm 1. While most
supervisor-model combinations perform similarly on the Tiny ImageNet dataset,
the same cannot be observed for SVHN and FakeData. For SVHN, which is another
image dataset, the results vary far more than on the Tiny ImageNet but the
overall results are better. DenseNet-121 for example, when combined with ODIN,
receives a score of 97.23, which indicates a very good separation between
inlier and outlier datasets. All models except for VGG16 succeeds to detect
samples with random Gaussian noise, whereas VGG seem to completely miss to
separate the in and out distribution. In this specific case, the random noise
creates larger stimulus than the training samples, and thus, the model
believes that outliers are the true distribution.

Figure 2: The AUROC metric plotted against the test set accuracy for each
individual model, on the three out-of-distribution datasets. The coloring
represents which model is used, while the marker represent which supervisor is
used (i.e., an orange square indicates one of the epochs from VGG16 tested
with baseline).

Similarly to AUROC, the FPR, which we measure when the model removes 95% of
the outliers, shows how rapidly the supervisor can exclude the majority of
outliers. On the Tiny ImageNet dataset, it can be seen that most models have
high AUROC but also high FPR95. This shows that the supervisor does a good job
but the last percentages of outliers cannot be separated. This is explained by
the fact that the datasets are not disjoint: Some classes exist in both
datasets, which creates a challenge for separation. For SVHN and FakeData, the
FPR are performing better, DenseNet-121 does a good job, as well as WRN-28
that does almost perfectly on FakeData but performs poorly on SVHN. This
result alone shows how fragile the supervisor performance can be, where
performance on one dataset can be high, while failing on another.

Figure 3: The FPR at 95% TPR (FPR95) metric plotted against the test set
accuracy for each individual model, on the three out-of-distribution datasets.
The coloring represents which model, whilst the marker represent which
supervisor is used. i.e an orange square indicates one of the epochs from
VGG16 tested with Baseline.

For coverage breakpoint on performance level, i.e., how restrictive the
supervisors have to be to receive same overall accuracy as achieved on the
test set, all models except for VGG16 perform similarly, independent on which
supervisor is used. Most models range between 38% and 43.5% CBPL for Tiny
ImageNet, which results in 13-24% of the inlier samples being falsely
rejected. It is noteworthy for DenseNet-121 to achieve $>50\%$ CBPL on SVHN
and FakeData, which indicates the supervisor can exclude the majority outlier
samples, as well as some inliers where the prediction would have been wrong.
When changing the accuracy requirement to 90%, the penalization on higher
accuracy models has been lifted. This change elicit higher coverage, as the
supervisor-model combination has more leeway.

Overall, the DenseNet-121 model outperforms the rest regarding supervisor
performance as well as coverage with and without diminishing requirements.
Even though WRN-28 and WRN-40 receives slightly better accuracy on the test
set, their performance in combination with a supervisor shows the problematic
nature of tuning supervisors. As we will see in the following, these small
variations in network weights can have large effects on the performance of the
supervisor-model pair.

###  IV-B Results during training

In Section III, we explained why it is relevant to study the supervisor
performance as the model is trained. Additionally, in this case, it is also
relevant to study how the performance change as the model coverges and starts
overfitting towards the training set. In this section, our selected metrics
are visualized in Fig. 2-5, respectively, and reviewed in further detail. All
plots are designed the same: Color indicates which model has been used, and
the marker indicates which supervisor has been used, i.e., an orange square
indicates VGG16-Baseline.

AUROC vs Accuracy: The AUROC scores can be seen in Fig. 2. As we presented in
[7], we found a linear relationship between the model accuracy and supervisor
performance. For the Tiny ImageNet dataset this pattern still exists, whereas
SVHN and FakeData does not contain this clear behavior. While SVHN does not
resemble a clear linear trend, it contains a positive lower boundary that
increases as the model accuracy improves. For the FakeData, the results are
instead scattered. It is unclear why this behavior exists, but could be due to
the noise matrices are being randomly generated for experimentation, and not
being kept static. Furthermore, for Tiny ImageNet and SVHN, the supervisor
performance is independent from which model is selected.

Figure 4: The Coverage Breakpoint at Performance Level (CBPL) metric plotted
against the test set accuracy for each individual model, on the three out-of-
distribution datasets. The coloring represents which model, whilst the marker
represent which supervisor is used. i.e an orange square indicates one of the
epochs from VGG16 tested with Baseline. Figure 5: The Coverage at 10% error
rate (Cov10) metric plotted against the test set accuracy for each individual
model, on the three out-of-distribution datasets. The coloring represents
which model, whilst the marker represent which supervisor is used; i.e., an
orange square indicates one of the epochs from VGG16 tested with Baseline.

An interesting phenomenon occurs for small variations in model parameters. As
have been expressed earlier, small subtle changes can have large impacts on
the model output, as well as the performance of the supervisor. Looking at
clusters that are present for both Tiny ImageNet and SVHN in the top right
corners, this impact becomes evident. While a large amount of experiments are
conducted on models with less than a tenth of a percentage difference, the
AUROC score can vary as much as 10% for Tiny ImageNet and up to 20% in SVHN.
This is worrisome, as there is no indication from the training setup, which
yields a more separable end result. An initial explanation of this varying
performance of OpenMax and ODIN can be explained by poor parameter settings,
indicating the need a parameter search for each change of model weights.
However, this does not explain why the behavior is also present for the
Baseline supervisor that does not have any parameters.

FPR at 95% TPR vs Accuracy: False positive rate, i.e., the amount of inlier
data that has falsely been rejected, is a an effective metric to look at in
comparison to True positive rate, i.e., the successful rejection of outlier
samples. In this case, we investigate how many inlier samples have been
rejected when the supervisor catches 95% of the outliers. In combination with
AUROC metric, this tells us several aspects about the model-supervisor
combination: If the AUROC score is close to 1 and FPR95 is close to 0, it
indicates the supervisor achieves a clear separation between the two
distributions, whereas the AUROC decreases and FPR95 increases, there exists
some overlapping of the distributions.

The FPR95 vs. accuracy plots can be seen in Fig. 3. At first, the two image
datasets show a downward trend, i.e., as the model performance increases, the
separation of in and out-of-distribution samples becomes more clear. All
dataset experiments contain large variations of FPR95, see for example Tiny
ImageNet and SVHN for models with $>95\%$ accuracy, where the FPR95 can vary
between 0.45 up to 0.72. This is once again related to small variations in the
model, which causes the robustness of the supervisor to drastically change.
The variations seem to be related to selected models, for example in SVHN
where DenseNet and WRN-28 performs well, but VGG16 and WRN-40 seem to fall
short no matter which supervisor is used. For Tiny ImageNet, FPR95 never
breaks below 0.37. This is similar to the AUROC case due to the datasets not
being disjoint, and thus overlapping in distributions will be present. On the
FakeData, VGG16 struggles more than the other models, especially for later
epochs where the model is more accurate, as the behavior is not present as the
model ranges between 85-92% accuracy. This shows how the effect of overfitting
harms the generalization of the model as well as the possibility for the
supervisor to distinguish between distributions.

Coverage breakpoints vs Accuracy: While AUROC and FPR95 cover the overall
performance of supervisor rejection rate, it is of interest to study what
inlier samples get rejected first. By studying coverage, we can see if
correctly classified samples or mis-classifications are the first ones to be
rejected.

Our previous work (cf. [1]) used CBPL, similarly to this work, see Fig. 4. In
the figure we see similar behavior as reported in our previous work, where the
coverage is decreasing for more accurate models. This is to be expected as the
restriction rate has to be stricter to achieve the high accuracy of the model.
Studying Fig. 4 more in-depth, it can be seen that the average coverage of
Tiny ImageNet is lower than to SVHN and FakeData, which shows how overlapping
distributions increase difficulty in separation.

Having 50% coverage means the model is predicting on the same amount of
samples as in the test set. Having more than 50% coverage would indicate that
the supervisor has managed to not only exclude outlier samples, but also some
inlier samples that would have yielded a mis-classification. Thus the final
result becomes a mix of excluding the majority of outlier samples, as well as
a miniority of inlier samples with high probability of being mis-classified.
We see this behavior for worse performing models in Tiny ImageNet as well as
on several occasions in the SVHN and FakeData datasets.

However, it seems non-intuitive to penalize well-performing models solely on
the basis to reach same error rate as achieved during training. A more test-
oriented approach would be to give the model a fixed target error rate, i.e.,
10%, which is higher than what all models achieved, and tune the threshold to
maximize the coverage for the given error rate. We show this in Fig. 5. In
this case, we can see how the coverage will increase with model performance
and achieve models with more than 50% coverage for the best performing models.
Additionally, for models that have a larger error rate than the fixed target
will now have to be more restrictive to reject outlier samples, as well as
inliers that would be mis-classified to reach the target error rate.

##  V Discussion

As the results improve of DNNs on various tasks, the fact that the confidence
level of out-of-distribution samples are high poses a problem to validity.
Even with more sophisticated outlier detection tools, we have seen here that
small adjustments in model weights can have large effects of the performance.
Thus, the question arises what additional ways of measuring model
generalization are needed in addition to the loss and accuracy metrics that
are used during training.

To deploy DNNs for safety critical applications, it is necessary to
distinguish between known states and unknown states, i.e., it is better to
know the functional limits,and detect scenarios that the system is not capable
at handling than blindly believing it can handle anything. Additionally, since
it is infeasible to record and train on all possible scenarios (cf. [8]),
instead it requires systematic ways of handling them to reduce the risk of the
function to operate on scenarios it was not built for.

To understand and be able to verify the performance of DNNs, additional
experiments and methods are needed that explain the behavior of the model.
This includes outlier detection, but also several additional topics such as
uncertainty estimations and development procedures that argue for development
processes, which aim to reduce the risk of mis-happenings with the model. As
SOTIF was released to cover the quantitative requirements of functions, it
still remains unclear how this should be set up.

##  VI Conclusions and Future Work

In this work we show how performance of supervisors change as the model
performance varies. We test it after every $10^{th}$ epoch, and find that very
small changes in the network, that does not yield any big difference in
accuracy or loss, can affect the out-of-distribution detection significantly.
Additionally, we show how the change of the coverage metric to a fixed error
rate gives the model more leeway to reject samples not only from the outlier
distribution but also from inlier samples where the risk of a miss-
classification is high.

There are many interesting aspects of continuing this research. Initially the
field need to start constructing safety argumentation that describes what and
how methods that support the DNN shall be constructed and tested. SOTIF
provides a starting point for these discussions, but industry needs to join to
describe suitable development processes, and which measures are required to
validate models that are learned from data. Additionally, when it comes to
both model and supervisor exploration, it is desired to study what samples are
rejected for the different outliers to see if an ensemble of models with a
corresponding supervisor would yield a more stable result.

## Acknowledgments

This work was carried out within the SMILE II project financed by Vinnova,
FFI, Fordonsstrategisk forskning och innovation under the grant number:
2017-03066, and partially supported by the Wallenberg AI, Autonomous Systems
and Software Program (WASP) funded by Knut and Alice Wallenberg Foundation.

## References

  * [1] J. Henriksson, C. Berger, M. Borg, L. Tornberg, S. R. Sathyamoorthy, and C. Englund, “Performance analysis of out-of-distribution detection on various trained neural networks,” in _2019 45th Euromicro Conference on Software Engineering and Advanced Applications (SEAA)_. IEEE, 2019, pp. 113–120.
  * [2] Y. LeCun, Y. Bengio, and G. Hinton, “Deep learning,” _Nature_ , vol. 521, no. 7553, pp. 436–444, 2015\.
  * [3] {International Organization for Standardization}, _ISO 26262-2:2018 Road vehicles - Functional safety_. International Organization for Standardization, 2018.
  * [4] R. Salay, R. Queiroz, and K. Czarnecki, “An Analysis of ISO 26262: Using Machine Learning Safely in Automotive Software,” _arXiv:1709.02435_ , 2017, arXiv: 1709.02435.
  * [5] J. Henriksson, M. Borg, and C. Englund, “Automotive safety and machine learning: Initial results from a study on how to adapt the ISO 26262 safety standard,” in _Proceedings of the 1st International Workshop on Software Engineering for AI in Autonomous Systems_. IEEE, 2018, pp. 47–49.
  * [6] {International Organization for Standardization}, _ISO/PAS 21448:2019 Road vehicles – Safety of the intended functionality_. International Organization for Standardization, 2019.
  * [7] J. Henriksson, C. Berger, M. Borg, L. Tornberg, C. Englund, S. R. Sathyamoorthy, and S. Ursing, “Towards structured evaluation of deep neural network supervisors,” in _Proceedings of the 1st IEEE International Conference on Artificial Intelligence Testing_ , 2019.
  * [8] M. Borg, C. Englund, K. Wnuk, B. Duran, C. Levandowski, S. Gao, Y. Tan, H. Kaijser, H. Lönn, and J. Törnqvist, “Safely entering the deep: A review of verification and validation for machine learning and a challenge elicitation in the automotive industry,” _Journal of Automotive Software Engineering (Accepted for publication)_ , 2019.
  * [9] A. Bendale and T. E. Boult, “Towards open set deep networks,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2016, pp. 1563–1572.
  * [10] S. Liang, Y. Li, and R. Srikant, “Enhancing the reliability of out-of-distribution image detection in neural networks,” _arXiv preprint arXiv:1706.02690_ , 2017.
  * [11] D. Hendrycks, M. Mazeika, and T. G. Dietterich, “Deep anomaly detection with outlier exposure,” _arXiv preprint arXiv:1812.04606_ , 2018.
  * [12] D. Hendrycks and K. Gimpel, “A baseline for detecting misclassified and out-of-distribution examples in neural networks,” _arXiv preprint arXiv:1610.02136_ , 2016.
  * [13] K. Simonyan and A. Zisserman, “Very deep convolutional networks for large-scale image recognition,” _CoRR_ , vol. abs/1409.1556, 2014. [Online]. Available: http://arxiv.org/abs/1409.1556
  * [14] G. Huang, Z. Liu, and K. Q. Weinberger, “Densely connected convolutional networks,” _CoRR_ , vol. abs/1608.06993, 2016. [Online]. Available: http://arxiv.org/abs/1608.06993
  * [15] S. Zagoruyko and N. Komodakis, “Wide residual networks,” _CoRR_ , vol. abs/1605.07146, 2016. [Online]. Available: http://arxiv.org/abs/1605.07146
  * [16] A. Krizhevsky and G. Hinton, “Learning multiple layers of features from tiny images,” Citeseer, Tech. Rep., 2009.
  * [17] K. Pei, Y. Cao, J. Yang, and S. Jana, “Deepxplore: Automated whitebox testing of deep learning systems,” _CoRR_ , vol. abs/1705.06640, 2017. [Online]. Available: http://arxiv.org/abs/1705.06640
  * [18] J. Guo, Y. Jiang, Y. Zhao, Q. Chen, and J. Sun, “Dlfuzz: Differential fuzzing testing of deep learning systems,” _CoRR_ , vol. abs/1808.09413, 2018. [Online]. Available: http://arxiv.org/abs/1808.09413
  * [19] J. Kim, R. Feldt, and S. Yoo, “Guiding deep learning system testing using surprise adequacy,” _CoRR_ , vol. abs/1808.08444, 2018. [Online]. Available: http://arxiv.org/abs/1808.08444
  * [20] K. Czarnecki and R. Salay, “Towards a framework to manage perceptual uncertainty for safe automated driving,” _Lecture Notes in Computer Science_ , p. 439–445, 2018. [Online]. Available: http://dx.doi.org/10.1007/978-3-319-99229-7_37
  * [21] X. Ma, K. Driggs-Campbell, and M. J. Kochenderfer, “Improved robustness and safety for autonomous vehicle control with adversarial reinforcement learning,” _2018 IEEE Intelligent Vehicles Symposium (IV)_ , Jun 2018. [Online]. Available: http://dx.doi.org/10.1109/ivs.2018.8500450
  * [22] L. Ma, F. Juefei-Xu, F. Zhang, J. Sun, M. Xue, B. Li, C. Chen, T. Su, L. Li, Y. Liu _et al._ , “Deepgauge: Multi-granularity testing criteria for deep learning systems,” in _Proceedings of the 33rd ACM/IEEE International Conference on Automated Software Engineering_. ACM, 2018, pp. 120–131.
  * [23] N. Carlini and D. Wagner, “Towards evaluating the robustness of neural networks,” in _2017 IEEE Symposium on Security and Privacy (SP)_. IEEE, 2017, pp. 39–57.
  * [24] N. Papernot, P. McDaniel, X. Wu, S. Jha, and A. Swami, “Distillation as a defense to adversarial perturbations against deep neural networks,” in _2016 IEEE Symposium on Security and Privacy (SP)_. IEEE, 2016, pp. 582–597.
  * [25] M. Landgren and L. Tranheden, _Input Verification for Deep Neural Networks – Detectio of data unfamiliar to deep neural networks_ , ser. 2018:EX040. Reading, Massachusetts: Chalmers University of Technology, 2018.
  * [26] K. He, X. Zhang, S. Ren, and J. Sun, “Deep Residual Learning for Image Recognition,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2016, pp. 770–778.
  * [27] O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma, Z. Huang, A. Karpathy, A. Khosla, M. Bernstein, A. C. Berg, and L. Fei-Fei, “ImageNet Large Scale Visual Recognition Challenge,” _International Journal of Computer Vision (IJCV)_ , vol. 115, no. 3, pp. 211–252, 2015.
  * [28] Y. Netzer, T. Wang, A. Coates, A. Bissacco, B. Wu, and A. Y. Ng, “Reading digits in natural images with unsupervised feature learning,” _NIPS workshop on deep learning and unsupervised feature learning_ , vol. 2011, p. 4, 2011.