Max Bain, Jaesung Huh, Tengda Han, Andrew Zisserman

# WhisperX: Time-Accurate Speech Transcription of Long-Form Audio

###### Abstract

Large-scale, weakly-supervised speech recognition models, such as Whisper,
have demonstrated impressive results on speech recognition across domains and
languages. However, their application to long audio transcription via buffered
or sliding window approaches is prone to drifting, hallucination & repetition;
and prohibits batched transcription due to their sequential nature. Further,
timestamps corresponding each utterance are prone to inaccuracies and word-
level timestamps are not available out-of-the-box. To overcome these
challenges, we present WhisperX, a time-accurate speech recognition system
with word-level timestamps utilising voice activity detection and forced
phoneme alignment. In doing so, we demonstrate state-of-the-art performance on
long-form transcription and word segmentation benchmarks. Additionally, we
show that pre-segmenting audio with our proposed VAD Cut & Merge strategy
improves transcription quality and enables a twelve-fold transcription speedup
via batched inference. The code is available open-
source111https://github.com/m-bain/whisperX.

Figure 1: WhisperX: We present a system for efficient speech transcription of
long-form audio with _word-level time alignment_. The input audio is first
segmented with Voice Activity Detection and then cut & merged into
approximately 30-second input chunks with boundaries that lie on minimally
active speech regions. The resulting chunks are then: (i) transcribed in
parallel with whisper and (ii) forced aligned with a phone recognition model
to produce accurate word-level timestamps at high throughput.

##  1 Introduction

With the availability of large-scale web datasets, weakly-supervised and
unsupervised training methods have demonstrated impressive performance on a
multitude of speech processing tasks; including speech recognition [1, 2, 3,
4], speaker recognition [5, 6], speech separation [7] and keyword spotting [8,
9]. Whisper [10] utilises this rich source of data to another scale.
Leveraging 680,000 hours of noisy speech training data, including 96 other
languages and 125,000 hours of English translation data, they showcase that
weakly supervised pretraining of a simple encoder-decoder transformer [11] can
robustly achieve zero-shot multilingual speech transcription on existing
benchmarks.

Most of the academic benchmarks are comprised of short utterances, whereas
real-world applications typically require transcribing long-form audio that
can easily be hours or minutes long, such as meetings, podcasts and videos.
Automatic Speech Recognition (ASR) models are typically trained on short audio
segments (30 seconds for the case of Whisper) and and the transformer
architectures employed prohibit transcription of arbitrarily long input audio
due to memory constraints.

Recent works [12] employ heuristic sliding-window style approaches which are
prone to errors due to (i) overlapping audio, that can lead to inconsistencies
in the transcription when the model processes the same speech twice; (ii)
incomplete audio: if some words lie on at the beginning or end of the input
segment then they can be missed or incorrectly transcribed. Whisper proposes a
buffered transcription approach that relies on accurate timestamp prediction
to determine the amount of shift the input window by. Such a method is prone
to severe drifting since timestamp inaccuracies in one window can accumulate
to subsequent windows. They employ a series of hand-crafted heuristics in
order to reduce these errors with limited success.

Whisper's coupled decoding of both the transcriptions and timestamps with a
single encoder-decoder is prone to the usual challenges faced by auto-
regressive language generation, namely: hallucination and repetition. This has
catastrophic consequences for buffered transcription of long-form and other
timestamp-sensitive tasks such as speaker diarization [13, 14], lip-reading
[15] and audiovisual learning [16]. As the Whisper paper details, partial data
(audio-transcription pairs without timestamp information) makes up a
substantial portion of the training corpus via use of a <|nottimestamps|>
token. Scaling on partial and noisy transcription data naturally trades speech
transcription performance for less accurate timestamp prediction. Therefore
there is a need to correctly align the transcript and speech using additional
modules.

A plethora of works exist on aligning speech transcription with audio
waveforms at the word-level or phoneme-level, so called ``forced alignment''.
A traditional approach is to train the acoustic phone models in Hidden Markov
Model (HMM) [17, 18, 19, 20] framework and using the by-product of possible
state alignments. These word-level or phone-level timestamps are often
adjusted by using external boundary correction models [21, 22]. With the rapid
development in deep learning based methods, a few of recent works employ the
deep learning strategies for forced alignment such as using bi-directional
attention matrix [23] or CTC-segmentation with end-to-end trained model [24].
Further improvement could be made by leveraging a state-of-the-art ASR model
with a light-weight phoneme recognition model, both of which are trained with
substantial large-scale datasets.

To address these challenges, we propose WhisperX, a system for efficient
speech transcription of long-form audio with accurate word-level timestamps.
It consists of three additional stages to Whisper transcription: (i) pre-
segmenting the input audio with an external Voice Activity Detection (VAD)
model; (ii) cut and merging the resulting VAD segments into approximately 30
seconds input chunks with boundaries lying on minimally active speech regions
enabling batched whisper transcription; and finally (iii) forced alignment
with an external phoneme model to provide accurate word-level timestamps.

##  2 WhisperX

In this section we describe WhisperX and its components for long-form speech
transcription with word-level alignment.

###  2.1 Voice Activity Detection

Before transcribing the input audio and performing alignment, we first pre-
segment the audio with VAD. The benefits of this pre-processing stage are two-
fold. First, VAD allows segmentation of the input audio into chunks with
boundaries that do not lie on active speech regions, thereby minimising errors
due to boundary effects and enabling batched transcription of chunks. Second,
the temporal boundaries of each segment can be used to constrain the alignment
to local segments and remove any reliance on Whisper's timestamps – which we
show to be too unreliable when used for alignment.

Formally, given a long-form audio waveform $A$, we apply VAD, resulting in a
list of N non-overlapping segments each corresponding to start and end times
of active speech regions $\mathcal{S}=[S_{1},S_{2},...,S_{N}]$, where
$S_{i}=(t^{i}_{0},t^{i}_{1})$.

###  2.2 VAD Cut & Merge

VAD segments $\mathcal{S}$ can be of varying lengths, shorter or longer than
the input duration that the ASR model (Whisper) was trained on,
$|A_{\text{train}}|$ = 30 seconds. Whilst the transformer architecture can
handle sequences of arbitrary input sizes, the attention operation scales with
its the square of the input length and therefore long segments with no upper
bound in duration can result in impractically high memory consumption.

To address this, we propose a min-cut operation whereby segments longer than
the maximum input duration ($|A_{\text{train}}|$) are divided at the point
with the lowest voice activation score from the VAD model. This ensures that
the newly divided segments do not exist on word boundaries, minimizing
boundary errors during the transcription process.

With an upper bound now set on the duration of input segments, the other
extreme must be considered: very short segments, posing their own set of
unique challenges. Transcribing short speech segment forgoes the wider context
which can be beneficial for modelling speech in challenging settings. In
addition, transcribing many shorter segments increases the time taken for
transcription of the entire audio due to the additional number of forward
passes.

Therefore, we propose the following merge operation:

$S_{i+1}\begin{cases}(t^{i}_{0},t^{i+1}_{1}),&\text{if
}t^{i+1}_{1}-t^{i}_{0}<\tau\\\
(t^{i+1}_{0},t^{i+1}_{1}),&\text{otherwise}\end{cases}$

where $\tau$ is the maximal duration of merged segments, which we show to be
optimal at $\tau=|A_{\text{train}}|$. This essentially merges neighbouring
short speech segments until their total duration is no greater than the input
size of the transcription model, providing the greatest possible context when
transcribing and keep the data distribution similar to that seen during
training.

###  2.3 Whisper Transcription

The resulting speech segments, now with duration approximately equal to the
input size of the model, $|S_{i}|\approx|A_{\text{train}}|~{}\forall i\in N$ ,
and boundaries that do not lie on active speech, can be effectively batch
transcribed with Whisper, outputting text for each segment
$S_{i}\rightarrow\mathcal{T}_{i}$. Transcription is performed without
conditioning on previous text since this would break the independence
assumption of each sample in the batch. Empirically, we find this to be
beneficial for robust transcription since conditioning on previous text tends
to be more prone to hallucination.

###  2.4 Forced Phoneme Alignment

For each segment $S_{i}$, and its resulting text transcription
$\mathcal{T}_{i}$ consisting of a sequence of words
$\mathcal{T}_{i}=[w_{0},w_{1},...,w_{m}]$, our goal is to estimate the start
and end time of each word. For this, we leverage a phoneme recognition model,
trained to classify the smallest unit of speech distinguishing one word from
another, e.g. the element p in ``tap". Let $C=[c_{1},c_{2},...,c_{K}]$ be the
set of phoneme classes in the dictionary. Given an input audio segment, a
phoneme classifier, takes an audio segment $S$ as input and outputs a logits
matrix $L\in\mathbb{R}^{K\times T}$, where $T$ varies depending on the
temporal resolution of the phoneme model.

Formally, for each segment, $S_{i}\in\mathcal{S}$, and its corresponding text
$\mathcal{T}_{i}$:

  1. 1.

Perform phoneme classification over the input segment $S_{i}$, where
classification is restricted to $C^{\prime}=c_{1},..,c_{K^{\prime}}$ classes,
the set of all phonemes in the current segment's transcription
$\mathcal{T}_{i}$.

  2. 2.

Apply Dynamic Time Warping (DTW) on the resulting logits matrix
$L_{i}\in\mathbb{R}^{K^{\prime}\times T}$, to obtain the optimal temporal path
of phonemes in $\mathcal{T}_{i}$.

  3. 3.

Obtain start and end times for each word $w_{i}$ in $\mathcal{T}_{i}$ by
taking the start and end time of the first and last phoneme respectively.

For transcript characters not present in the phoneme dictionary $C$, we assign
the timestamp from the next nearest phoneme in the transcript. The for loop
described aboved can be batch processed in parallel, enabling fast
transcription and word-alignment of long-form audio.

###  2.5 Multilingual Transcription and Alignment

WhisperX can also be applied to multilingual transcription, with the caveat
that (i) the VAD model should be robust to different languages and (ii) the
alignment phoneme model ought to be trained on the language(s) of interest.
Multilingual phoneme recognition models [25] are also be suitable option,
possibly generalising to languages not seen during training – this would just
require an additional mapping from language-independent phonemes to the
phonemes of the target language(s).

###  2.6 Word-level Timestamps without Phoneme Recognition

A natural question is whether word-level timestamps can be extracted from the
Whisper model directly, without the addition of an external phoneme model.
Such a method alleviates the additional inference overhead (although in
practice we find this overhead is minimal, approx. $<$10%), nor requires a
mapping between the Whisper and phoneme dictionaries.

With some tweaking of the forward pass, it is possible to infer word-level
timestamps from the cross-attention scores of the decoded tokens. Attempts of
this nature have been made by the original Whisper authors in their official
open-source repository [26], as well as implementations by others [27].
However, as we show later in the results (Section 3.4), this Whisper-only
method underperforms considerably compared to the proposed external phoneme
approach and is prone to the aforementioned timestamp inaccuracies.

##  3 Evaluation

Our evaluation addresses the following questions: (1) the effectiveness of
WhisperX for long-form transcription and word-level segmentation compared to
state-of-the-art ASR models (namely Whisper and wav2vec2.0); (2) the benefit
of VAD Cut & Merge pre-processing in terms of transcription quality and speed;
and (3) the effect of the choice of phoneme model and whisper model on word
segmentation performance.

###  3.1 Datasets

The AMI Meeting Corpus. We used the test set of the AMI-IHM from the AMI
Meeting Corpus [28] consisting of 16 audio recordings of meetings, each
approximately 30 minutes in duration. Manually verified word-level alignments
are provided for the test set which we use to evaluate word segmentation
performance.

Switchboard-1 Telephone Speech Corpus (SWB). SWB [29] consists of
approximately 2,400 hours of speech of two-sided telephone conversations.
Ground truth transcriptions are provided with manually correct word
alignments. We randomly subsampled a set of 100 conversations to evaluate word
segmentation compared to the manually verified word-level alignments.

TED-LIUM 3. To evaluate transcription quality and speed (no timestamps) of
long-form audio, we follow [10] and report WER on the TEDLIUM test set [30]
consisting of 11 TED talks each approximately 20 minutes in duration. We
additionally report transcription speed.

###  3.2 Metrics

Speech recognition benchmarks typically only measure word error rate (WER) and
do not evaluate the accuracy of the predicted timestamps. Thus, we also
evaluate word segmentation metrics, for datasets that have word-level
timestamps, that jointly evaluate transcription and timestamp quality. We
report Precision and Recall where a true positive is where a predicted word
segment overlaps with a ground truth word segment within a collar, where both
words are an exact string match. For all evaluations we use a collar value of
200 milliseconds to account for differences in annotation and models.

###  3.3 Implementation Details

WhisperX. Unless specified otherwise, we use the default configuration in
Table 1 for all experiments.

Whisper [10]. For Whisper-only transcription and word-alignment we inherit the
default configuration from Table 1, and infer the word-level timestamps from
the cross-attention peaks in the decoded tokens (as in [26, 27]). Timestamp
heuristics must be employed (including clamping negative duration timestamps)
in order to prevent failed alignments.

Wav2vec2.0 [2]. For wav2vec2.0 transcription and word-alignment we use the
default settings in Table 1 unless specified otherwise. We obtain the various
model versions from the official torchaudio
repository222https://pytorch.org/audio/stable/pipelines.html#module-
torchaudio.pipelines. Base_960h, Large_960h and HuBERT [3] models were
finetuned on Librispeech [31] data, whereas the VoxPopuli model was trained on
the Voxpopuli [32] corpus.

For benchmarking inference speed, all models are measured on an NVIDIA A40
gpu.

Table 1: Default configuration for WhisperX. Type | Hyperparameter | Default Value
---|---|---
VAD | Model | pyannote
Onset threshold | 0.767
Offset threshold | 0.377
Min. duration on | 0.136
Min. duration off | 0.067
Whisper | Model version | large-v2
Decoding strategy | greedy
Condition on previous text | False
Phoneme Recognition | Architecture | wav2vec2.0
Model version | BASE_960H
Decoding strategy | greedy

###  3.4 Results

Table 2: State-of-the-art comparison of long-form audio transcription and word segmentation on the TEDLIUM, AMI, and Switchboard corpuses. WER denotes Word Error Rate. †Word-level timestamps are not obtainable out-of-the-box from Whisper and are inferred from the cross-attention scores of the decoded tokens via Dynamic Time Warping. Model | Version | TED-LIUM | AMI | SWB
---|---|---|---|---
WER$\downarrow$ | Speed$\uparrow$ | Precision$\uparrow$ | Recall$\uparrow$ | Precision$\uparrow$ | Recall$\uparrow$
wav2vec2.0 [2] | BASE_960H | 19.8 | 10.3$\times$ | 81.8 | 45.5 | 92.9 | 54.3
Whisper [10]† | large-v2 | 10.5 | 1.0$\times$ | 70.5 | 42.9 | 84.1 | 54.3
WhisperX | large-v2 | 9.7 | 11.8$\times$ | 84.1 | 60.3 | 93.2 | 65.4
Table 3: Effect of WhisperX's VAD Cut & Merge and batched transcription on long-form audio transcription on the TED-LIUM benchmark and AMI corpus. Full audio input corresponds to WhisperX without any VAD pre-processing, VAD-CMτ refers to VAD pre-processing with Cut & Merge, where $\tau$ is the merge duration threshold in seconds. Input |  Batch Size | TED-LIUM | AMI
---|---|---|---
WER$\downarrow$ | Spd.$\uparrow$ | Prec.$\uparrow$ | Rec.$\uparrow$
Full audio | 1 | 10.52 | 1.0$\times$ | 82.6 | 53.4
32 | 78.78 | 7.1$\times$ | 43.2 | 25.7
VAD-CM15 | 1 | 9.72 | 2.1$\times$ | 84.1 | 56.0
32 | 7.9$\times$
VAD-CM30 | 1 | 9.70 | 2.7$\times$ | 84.1 | 60.3
32 | 11.8$\times$

####  3.4.1 Word Segmentation Performance

We compare WhisperX to previous state-of-the-art works in speech
transcription, namely Whisper and wav2vec2.0. In Table 2, we see that Whisper
outperforms both wav2vec2.0 and Whisper by substantial margins on word
segmentation benchmarks, as well as significant improvements to WER and
transcription speed over Whisper. WhisperX with batched transcription is even
faster than the lightweight wav2vec2 model. We see that mining word-level
timestamps from Whisper alone underperforms considerably at word segmentation
precision and recall across both SWB and AMI corpuses, even performing worse
than wav2vec2.0, a smaller model trained on far less data. This suggests that
the large-scale noisy training data of Whisper alone is insufficient to
correctly learn word-level timestamps with the current architecture and
training regime used.

####  3.4.2 Effect of VAD Chunking

Table 3 illustrates the benefits of pre-segmenting the audio with VAD and the
Cut & Merge operations. Interestingly, there is an improvement on on the
transcription-only WER metric, suggesting that VAD preprocessing for general
transcription quality. The benefit of VAD is more pronounced on word
segmentation precision and recall, indicating that whisper timestamps are not
sufficient to bound the alignment window. Batched transcription without VAD
chunking experiences a severe degradation in transcription quality (WER) and
word segmentation.

Batched inference with VAD provides an almost twelve-fold speed increase with
no performance loss, since each segment in the batch can be independently
transcribed. This overcomes the limitations of buffered transcription as in
[10]. We see that setting the merge threshold $\tau$ to the input duration
that whisper was trained on $|A_{\text{train}}|=30$, results in optimal
transcription speed and the lowest WER, compared to lower merge thresholds
such as $\tau=15.0$, suggesting that the greatest amount of context provides
the most accurate transcription.

Table 4: Effect of whisper model and phoneme model on WhisperX on word segmentation. Both the choice of whisper and phoneme model has a significant effect on word segmentation performance. Whisper Model |  Phoneme Model | AMI | SWB
---|---|---|---
Prec. | Rec. | Prec. | Rec.
base.en | Base_960h | 83.7 | 58.9 | 93.1 | 64.5
Large_960h | 84.9 | 56.6 | 93.1 | 62.9
HuBERT | 83.6 | 58.5 | 94.3 | 65.3
VoxPopuli | 87.4 | 60.3 | 86.3 | 60.1
small.en | Base_960H | 84.1 | 59.4 | 92.9 | 62.7
Large_960H | 84.6 | 55.7 | 94.0 | 64.9
HuBERT | 84.0 | 58.9 | 93.4 | 63.0
VoxPopuli | 87.7 | 61.2 | 84.7 | 56.3
large-v2 | Base_960H | 84.1 | 60.3 | 93.2 | 65.4
Large_960H | 84.9 | 57.1 | 93.5 | 65.7
HuBERT | 84.0 | 59.8 | 93.3 | 63.0
VoxPopuli | 87.7 | 61.7 | 84.9 | 58.7

####  3.4.3 Effect of Chosen Whisper and Alignment Models

We compare the effect of different Whisper and phoneme recognition models on
word segmentation performance across the AMI and SWB corpuses in Table 4.
Unsurprisingly, we see consistent improvements in both precision and recall
when using a larger Whisper model. In contrast, the bigger phoneme model is
not necessarily the best is instead more nuanced. The model trained on the
VoxPopuli corpus significantly outperforms other models on AMI, suggesting
that there is a higher degree of domain similarity between the two corpuses.
The lack of consistent high performance from the large model suggests that
increasing the amount of phoneme-supervised finetuning data would realise
further gains. Overall the base model trained on LibriSpeech performs
consistently well and can be concluded to be a suitable default choice of
phoneme alignment for word-level alignment of Whisper.

##  4 Conclusion

To conclude, we introduce WhisperX, an efficient and time-accurate speech
recognition system enabling parallelised and time-aligned transcription with
Whisper utlizing phoneme alignment. We show that the proposed VAD Cut & Merge
preprocessing reduces hallucination (WER) and enables within-audio batched
transcription, resulting in a twelve-fold speed increase without sacrificing
transcription quality. Further, we show that these transcribed VAD segments
can be force aligned with phoneme models, providing accurate word-level
segmentations with minimal inference overhead and resulting in time-accurate
transcriptions benefitting a range of applications (subtitling, diarization
etc.). A promising direction for future work is the training of a single-stage
ASR system that can efficiently transcribe long-form audio with accurate
timestamps.

Acknowledgement This research is funded by the EPSRC VisualAI EP/T028572/1 (M.
Bain, T. Han, A. Zisserman), a Global Korea Scholarship (J. Huh) and the Royal
Society Research Professorship RP$\backslash$R1$\backslash$191132 (A.
Zisserman).

## References

  * [1] Y. Jia, M. Johnson, W. Macherey, R. J. Weiss, Y. Cao, C.-C. Chiu, N. Ari, S. Laurenzo, and Y. Wu, ``Leveraging weakly supervised data to improve end-to-end speech-to-text translation,'' in _ICASSP_. IEEE, 2019, pp. 7180–7184.

  * [2] A. Baevski, Y. Zhou, A. Mohamed, and M. Auli, ``wav2vec 2.0: A framework for self-supervised learning of speech representations,'' _NeurIPS_ , vol. 33, pp. 12 449–12 460, 2020.
  * [3] W.-N. Hsu, B. Bolte, Y.-H. H. Tsai, K. Lakhotia, R. Salakhutdinov, and A. Mohamed, ``Hubert: Self-supervised speech representation learning by masked prediction of hidden units,'' _IEEE/ACM Transactions on Audio, Speech, and Language Processing_ , vol. 29, pp. 3451–3460, 2021.
  * [4] S. Chen, C. Wang, Z. Chen, Y. Wu, S. Liu, Z. Chen, J. Li, N. Kanda, T. Yoshioka, X. Xiao _et al._ , ``Wavlm: Large-scale self-supervised pre-training for full stack speech processing,'' _IEEE Journal of Selected Topics in Signal Processing_ , vol. 16, no. 6, pp. 1505–1518, 2022.
  * [5] J. Huh, H. S. Heo, J. Kang, S. Watanabe, and J. S. Chung, ``Augmentation adversarial training for self-supervised speaker recognition,'' _arXiv preprint arXiv:2007.12085_ , 2020.
  * [6] J. Kang, J. Huh, H. S. Heo, and J. S. Chung, ``Augmentation adversarial training for self-supervised speaker representation learning,'' _IEEE Journal of Selected Topics in Signal Processing_ , vol. 16, no. 6, pp. 1253–1262, 2022.
  * [7] S. Wisdom, E. Tzinis, H. Erdogan, R. Weiss, K. Wilson, and J. Hershey, ``Unsupervised sound separation using mixture invariant training,'' _NeurIPS_ , vol. 33, pp. 3846–3857, 2020.
  * [8] Y. Gong, C.-I. Lai, Y.-A. Chung, and J. Glass, ``Ssast: Self-supervised audio spectrogram transformer,'' in _Proc. AAAI_ , vol. 36, no. 10, 2022, pp. 10 699–10 709.
  * [9] K. Prajwal, L. Momeni, T. Afouras, and A. Zisserman, ``Visual keyword spotting with attention,'' _arXiv preprint arXiv:2110.15957_ , 2021.
  * [10] A. Radford, J. W. Kim, T. Xu, G. Brockman, C. McLeavey, and I. Sutskever, ``Robust speech recognition via large-scale weak supervision,'' _arXiv preprint arXiv:2212.04356_ , 2022.
  * [11] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, ``Attention is all you need,'' _NeurIPS_ , vol. 30, 2017.
  * [12] C.-C. Chiu, W. Han, Y. Zhang, R. Pang, S. Kishchenko, P. Nguyen, A. Narayanan, H. Liao, S. Zhang, A. Kannan _et al._ , ``A comparison of end-to-end models for long-form speech recognition,'' in _2019 IEEE automatic speech recognition and understanding workshop (ASRU)_. IEEE, 2019, pp. 889–896.
  * [13] Q. Wang, C. Downey, L. Wan, P. A. Mansfield, and I. L. Moreno, ``Speaker diarization with lstm,'' in _ICASSP_. IEEE, 2018, pp. 5239–5243.
  * [14] A. Zhang, Q. Wang, Z. Zhu, J. Paisley, and C. Wang, ``Fully supervised speaker diarization,'' in _ICASSP_. IEEE, 2019, pp. 6301–6305.
  * [15] K. Prajwal, T. Afouras, and A. Zisserman, ``Sub-word level lip reading with visual attention,'' in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2022, pp. 5162–5172.
  * [16] A. Miech, D. Zhukov, J.-B. Alayrac, M. Tapaswi, I. Laptev, and J. Sivic, ``Howto100m: Learning a text-video embedding by watching hundred million narrated video clips,'' in _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 2019, pp. 2630–2640.
  * [17] F. Brugnara, D. Falavigna, and M. Omologo, ``Automatic segmentation and labeling of speech based on hidden markov models,'' _Speech Communication_ , vol. 12, no. 4, pp. 357–370, 1993.
  * [18] K. Gorman, J. Howell, and M. Wagner, ``Prosodylab-aligner: A tool for forced alignment of laboratory speech,'' _Canadian Acoustics_ , vol. 39, no. 3, pp. 192–193, 2011.
  * [19] J. Yuan, N. Ryant, M. Liberman, A. Stolcke, V. Mitra, and W. Wang, ``Automatic phonetic segmentation using boundary models.'' in _Interspeech_ , 2013, pp. 2306–2310.
  * [20] M. McAuliffe, M. Socolof, S. Mihuc, M. Wagner, and M. Sonderegger, ``Montreal forced aligner: Trainable text-speech alignment using kaldi.'' in _Interspeech_ , vol. 2017, 2017, pp. 498–502.
  * [21] Y.-J. Kim and A. Conkie, ``Automatic segmentation combining an hmm-based approach and spectral boundary correction,'' in _Seventh International conference on spoken language processing_ , 2002.
  * [22] A. Stolcke, N. Ryant, V. Mitra, J. Yuan, W. Wang, and M. Liberman, ``Highly accurate phonetic segmentation using boundary correction models and system fusion,'' in _ICASSP_. IEEE, 2014, pp. 5552–5556.
  * [23] J. Li, Y. Meng, Z. Wu, H. Meng, Q. Tian, Y. Wang, and Y. Wang, ``Neufa: Neural network based end-to-end forced alignment with bidirectional attention mechanism,'' in _ICASSP_. IEEE, 2022, pp. 8007–8011.
  * [24] L. Kürzinger, D. Winkelbauer, L. Li, T. Watzel, and G. Rigoll, ``Ctc-segmentation of large corpora for german end-to-end speech recognition,'' in _Speech and Computer (SPECOM 2020)_. Springer, 2020, pp. 267–278.
  * [25] A. Conneau, A. Baevski, R. Collobert, A. Mohamed, and M. Auli, ``Unsupervised cross-lingual representation learning for speech recognition,'' _arXiv preprint arXiv:2006.13979_ , 2020.
  * [26] J. W. Kim, ``Whisper word-level timestamps,'' https://github.com/openai/whisper/blob/word-level-timestamps/notebooks/Multilingual_ASR.ipynb, 2022\.
  * [27] J. Louradour, ``whisper-timestamped,'' https://github.com/linto-ai/whisper-timestamped/tree/f861b2b19d158f3cbf4ce524f22c78cb471d6131, 2023\.
  * [28] J. Carletta, S. Ashby, S. Bourban, M. Flynn, M. Guillemot, T. Hain, J. Kadlec, V. Karaiskos, W. Kraaij, M. Kronenthal _et al._ , ``The ami meeting corpus: A pre-announcement,'' in _Machine Learning for Multimodal Interaction: Second International Workshop, MLMI 2005, Edinburgh, UK, July 11-13, 2005, Revised Selected Papers 2_. Springer, 2006, pp. 28–39.
  * [29] J. Godfrey and E. Holliman, ``Switchboard-1 release 2 ldc97s62,'' _Linguistic Data Consortium_ , p. 34, 1993.
  * [30] F. Hernandez, V. Nguyen, S. Ghannay, N. Tomashenko, and Y. Esteve, ``Ted-lium 3: Twice as much data and corpus repartition for experiments on speaker adaptation,'' in _Speech and Computer: 20th International Conference, SPECOM 2018, Leipzig, Germany, September 18–22, 2018, Proceedings 20_. Springer, 2018, pp. 198–208.
  * [31] V. Panayotov, G. Chen, D. Povey, and S. Khudanpur, ``Librispeech: an asr corpus based on public domain audio books,'' in _2015 IEEE international conference on acoustics, speech and signal processing (ICASSP)_. IEEE, 2015, pp. 5206–5210.
  * [32] C. Wang, M. Riviere, A. Lee, A. Wu, C. Talnikar, D. Haziza, M. Williamson, J. Pino, and E. Dupoux, ``Voxpopuli: A large-scale multilingual speech corpus for representation learning, semi-supervised learning and interpretation,'' _arXiv preprint arXiv:2101.00390_ , 2021.


# Interstellar Objects from Broken Dyson Spheres

Abraham Loeb  Astronomy Department, Harvard University, 60 Garden St.,
Cambridge, MA 02138, USA

###### Abstract

Without extensive maintenance, Dyson spheres inevitably disintegrate by
asteroid impacts over billions of years. The resulting fragments would appear
as anomalous interstellar objects, potentially sharing the unusual shape and
motion of 1I/‘Oumuamua or the unusual material strength of the first two
interstellar meteors, IM1 and IM2. If the Dyson sphere’s tiles are light
sails, the number of fragments could exceed that of interstellar asteroids
because of their reduced escape speed from the star and the increase in
stellar luminosity during the red giant phase.

##  1 Introduction

In 1937, Olaf Stapledon published the novel “Star Maker”, in which he imagined
the use of a technologically-manufactored shell of matter to tap the energy
output of a host star. The concept was subsequently formalized by Freeman
Dyson (Dyson, 1960), who reasoned that as the energy needs of humanity will
steadily increase, our civilization might aspire to tap all the energy output
of the Sun. Dyson proposed a shell of orbiting structures that would intercept
and collect the solar luminosity. This so-called Dyson sphere would emit
infrared radiation to balance the heat deposited on it by sunlight. The
resulting infrared signal could flag a Dyson sphere in contrast to the natural
optical emission by Sun-like stars. So far, searches for related infrared
signatures from stars or galaxies did not find evidence for Dyson spheres but
only for emission by natural dust (see references in Wright (2020)).

Below I demonstrate that if Dyson spheres existed to serve their civilizations
for a limited time, most of them would have disintegrated within billions of
years in the absence of extensive maintenance. In that case, their ejected
fragments could appear as interstellar objects (Siraj & Loeb, 2022a).

##  2 Broken Dyson Spheres

Rigid Dyson spheres are not easy to maintain. In compliance with Newton’s iron
sphere theorem, a perfectly spherical and rigid shell is not subjected to any
net gravitational force from a star interior to it, regardless of whether it
is centered on the star. However, the shell experiences destructive
differential forces across its surface, and its material strength must be high
in order to prevent deformation. The required tensile strength, $\Pi$, is
given by:

$\Pi\approx{GM_{\star}\rho_{ds}\over 2R_{ds}}=3.6\times 10^{6}~{}{\rm
MPa}\left({M_{\star}\over M_{\odot}}\right)\left({\rho_{ds}\over 7.9~{}{\rm
g~{}cm^{-3}}}\right)\left({R_{ds}\over 1~{}{\rm au}}\right)^{-1},$ (1)

where $M_{\star}$ is the stellar mass, $R_{ds}$ is the associated Dyson sphere
radius and $\rho_{\rm ds}$ is normalized by the solid density of iron. The
derived value is an order of magnitude above the tensile strength of graphene,
$\Pi_{gr}=1.3\times 10^{5}~{}{\rm MPa}$.

The required mass of a Dyson sphere with a thickness $W_{ds}$,

$M_{ds}=4\pi R_{ds}^{2}W_{ds}\rho_{ds}=2.3M_{\earth}\left({R_{ds}\over
1~{}{\rm au}}\right)^{2}\left({\rho_{ds}\over 7.9~{}{\rm
g~{}cm^{-3}}}\right)\left({W_{ds}\over 1~{}{\rm cm}}\right),$ (2)

exceeds the mass of the asteroid belt, $2.4\times 10^{24}~{}{\rm g}$, by four
orders of magnitude. This may explain the inferred discrepancy between the
required mass in interstellar objects, such as 1I/‘Oumuamua, and the expected
mass in asteroids ejected from planetary systems (Moro-Martín et al., 2009;
Moro-Martín, 2018, 2019).

To circumvent the material strength and mass challenges, Robert Forward
proposed a tiled structure (Forward, 1991), with each component functioning as
a solar sail for which the star’s gravity is exactly balanced by by the star’s
outward radiative push, thus maintaining a fixed position without orbiting the
star. The force balance reads (Loeb, 2022a):

${L_{\star}\over 2\pi R_{ds}^{2}c}={GM_{\star}\Sigma_{ds}\over R_{ds}^{2}},$
(3)

where $L_{\star}$ is the stellar luminosity and the mass per unit area of the
light sail components is given by,

$\Sigma_{ds}=\rho_{ds}W_{ds}=1.5\times 10^{-4}~{}{\rm
g~{}cm^{-2}}\left({L\over L_{\odot}}\right)\left({M_{\star}\over
M_{\odot}}\right)^{-1}~{}.$ (4)

In that case, the required thinness of the Dyson sphere sails could explain
(Loeb, 2022b; Bialy & Loeb, 2018) the flat geometry (Mashchenko, 2019) and
excess push of the interstellar object 1I/‘Oumuamua away from the Sun (Micheli
et al., 2018) without a cometary tail (Trilling et al., 2018).

The repulsive push of the stellar radiation pressure is equivalent to a
reduction in the effective gravitating attraction of the star because both
gravitational and radiative accelerations decline inversely with distance
squared. For a ratio $f_{ds}<1$ between the radiative and gravitational
accelerations (the ration between the left and right sides of equation 3), the
effective gravitating mass of the star is $(1-f_{ds})M_{\star}$. When the two
accelerations balance ($f_{ds}=1$), as considered by equations (3-4), the sail
hovers without any gravitational binding to the star. In that case, the escape
speed of the sail from the star vanishes, and a small gravitational push
outwards by a passing planet could eject it from the planetary system.
Consequently, sail fragments would escape much more easily from a planetary
system, compared to gravitationally bound asteroids which are trapped by the
barrier of a substantial escape speed.

Since sail fragments are thin, they carry much less mass than asteroids of the
same length, thus relieving the mass discrepancy associated with interstellar
asteroids (Moro-Martín et al., 2009; Moro-Martín, 2018, 2019).

Both radiative and gravitational forces are spherically symmetric as long as
the sail maintains a fixed cross-sectional area in the direction of the star.
This can be trivially accomplished by a sail in the shape of a small sphere
which maintains a constant cross-sectional area irrespective of orientation.

Even without damage from asteroids, abandoned sails will eventually be pushed
out of the planetary system as a sun-like star would evolve through its red
giant phase, resulting in an increase in its luminosity-to-mass ratio.

##  3 Implications

If a Dyson sphere disintegrated by asteroid impacts over time, then its
fragments would have been ejected by a gravitational or radiative push out of
the host planetary system more easily than asteroids.

Interstellar fragments of a broken Dyson sphere could potentially share the
unusual shape (Mashchenko, 2019) and light sail characteristics of the
interstellar object 1I/‘Oumuamua (Loeb, 2022b; Bialy & Loeb, 2018), or the
unusually high yield strength ($\gtrsim 10^{2}~{}{\rm Mpa}$, larger than the
value of iron meteorites) exhibited by the first and second interstellar
meteors, IM1 and IM2 (Siraj & Loeb, 2022b). The planned expedition to retrieve
the fragments of IM1 (Siraj et al., 2022) and the subsequent analysis of the
fragments composition, could potentially test this hypothesis.

Irrespective of its architecture, maintenance of a Dyson sphere is extremely
challenging since the structure will be punctured by a few billion human-size
asteroids every year for a planetary system that resembles the Solar system.
Over a few billion years, the structure would have holes of a few meters in
size every hundred meters. Interestingly, the estimated size of 1I/‘Oumuamua
was about a hundred meters (Mashchenko, 2019), potentially dictated by the
asteroid bombardment rate of an old Dyson sphere in a Sun-like planetary
system.

Small holes would be more abundant. The holes would cover a significant
fraction of the Dyson sphere surface from bombardment by micrometeorites on
sub-centimeter scales. This impact statistics is currently measured
empirically, as JWST is being hit by a dust-sized particle every month.
Without repair, a billion-year old Dyson sphere would resemble a fishing net
which lets a substantial fraction of the starlight out.

In summary, once a civilization abandons its Dyson sphere this technosignature
is expected be punctured by micrometeorites and lose its functionality on a
timescale much shorter than the lifetime of the host star. Studies of the
composition of interstellar objects offers a new way to constrain the
abundance of broken Dyson spheres, irrespective of their age and deteriorated
infrared characteristics.

## Acknowledgements

This work was supported in part by Harvard’s Black Hole Initiative, which is
funded by grants from JFT and GBMF.

## References

  * Bialy & Loeb (2018) Bialy, S., & Loeb, A. 2018, ApJ, 868, L1, doi: 10.3847/2041-8213/aaeda8
  * Dyson (1960) Dyson, F. J. 1960, Science, 131, 1667, doi: 10.1126/science.131.3414.1667
  * Forward (1991) Forward, R. L. 1991, Journal of Spacecraft and Rockets, 28, 606, doi: 10.2514/3.26287
  * Loeb (2022a) Loeb, A. 2022a, Research Notes of the American Astronomical Society, 6, 104, doi: 10.3847/2515-5172/ac72ad
  * Loeb (2022b) —. 2022b, Astrobiology, 22, 1392, doi: 10.1089/ast.2021.0193
  * Mashchenko (2019) Mashchenko, S. 2019, MNRAS, 489, 3003, doi: 10.1093/mnras/stz2380
  * Micheli et al. (2018) Micheli, M., Farnocchia, D., Meech, K. J., et al. 2018, Nature, 559, 223, doi: 10.1038/s41586-018-0254-4
  * Moro-Martín (2018) Moro-Martín, A. 2018, ApJ, 866, 131, doi: 10.3847/1538-4357/aadf34
  * Moro-Martín (2019) —. 2019, AJ, 157, 86, doi: 10.3847/1538-3881/aafda6
  * Moro-Martín et al. (2009) Moro-Martín, A., Turner, E. L., & Loeb, A. 2009, ApJ, 704, 733, doi: 10.1088/0004-637X/704/1/733
  * Siraj & Loeb (2022a) Siraj, A., & Loeb, A. 2022a, Astrobiology, 22, 1459, doi: 10.1089/ast.2021.0189
  * Siraj & Loeb (2022b) —. 2022b, ApJ, 941, L28, doi: 10.3847/2041-8213/aca8a0
  * Siraj et al. (2022) Siraj, A., Loeb, A., & Gallaudet, T. 2022, arXiv e-prints, arXiv:2208.00092, doi: 10.48550/arXiv.2208.00092
  * Trilling et al. (2018) Trilling, D. E., Mommert, M., Hora, J. L., et al. 2018, AJ, 156, 261, doi: 10.3847/1538-3881/aae88f
  * Wright (2020) Wright, J. T. 2020, Serbian Astronomical Journal, 200, 1, doi: 10.2298/SAJ2000001W


11institutetext: Institute for Anthropomatics and Robotics, Karlsruhe
Institute of Technology, Karlsruhe, Germany
11email<EMAIL_ADDRESS>11email: {zdravko.marinov,
<EMAIL_ADDRESS>22institutetext: HIDSS4Health - Helmholtz
Information and Data Science School for Health, Karlsruhe/Heidelberg, Germany
**footnotetext: Shared first author

# AutoPET Challenge 2023: Sliding Window-based Optimization of U-Net

Matthias Hadlich,∗ 11 Zdravko Marinov,∗ 1122 Rainer Stiefelhagen 11

###### Abstract

Tumor segmentation in medical imaging is crucial and relies on precise
delineation. Fluorodeoxyglucose Positron-Emission Tomography (FDG-PET) is
widely used in clinical practice to detect metabolically active tumors.
However, FDG-PET scans may misinterpret irregular glucose consumption in
healthy or benign tissues as cancer. Combining PET with Computed Tomography
(CT) can enhance tumor segmentation by integrating metabolic and anatomic
information. FDG-PET/CT scans are pivotal for cancer staging and reassessment,
utilizing radiolabeled fluorodeoxyglucose to highlight metabolically active
regions. Accurately distinguishing tumor-specific uptake from physiological
uptake in normal tissues is a challenging aspect of precise tumor
segmentation. The AutoPET challenge addresses this by providing a dataset of
1014 FDG-PET/CT studies, encouraging advancements in accurate tumor
segmentation and analysis within the FDG-PET/CT domain.
Code: https://github.com/matt3o/AutoPET2-Submission/

###### Keywords:

Semantic Segmentation Sliding Window U-Net

##  1 Introduction

In the domain of oncological diagnostics, the integration of
Fluorodeoxyglucose Positron-Emission Tomography (FDG-PET) and Computed
Tomography (CT) has assumed a pivotal role, facilitating comprehensive
insights into the metabolic dynamics of various malignant solid tumor entities
[1]. FDG-PET, acknowledged for its capacity to delineate glucose consumption
within tissues, holds significant promise in therapy control and monitoring,
owing to the characteristic escalated glucose uptake by tumor lesions [3].
However, the non-specificity of FDG-PET often introduces interpretational
ambiguities, as it may also manifest in benign or healthy tissue [6],
potentially leading to erroneous diagnoses.

To mitigate this diagnostic challenge, the fusion of PET with CT has emerged
as an integrated approach, combining metabolic data with precise anatomical
information. This combination enhances tumor detection accuracy [1], [13],
offering a cohesive synergy particularly valuable in clinical practice [6].

Within this evolving landscape of medical diagnostics, the Automatic Lesion
Segmentation in Whole-Body FDG-PET/CT Challenge (AutoPET)111https://autopet-
ii.grand-challenge.org/ embodies a critical juncture. It motivates researchers
and practitioners to develop automated, bi-modal methodologies for the three-
dimensional segmentation of tumor lesions embedded within FDG-PET and CT scans
[6]. The challenge accelerates advancements in deep learning-based automated
tumor lesion segmentation through the provision of a large densely annotated
dataset of 1014 volumes.

In this work, we propose using the well-known U-Net architecture [15] to
tackle the AutoPET challenge. Despite the ubiquity of U-Net models in medical
segmentation tasks [9], [4], achieving high performance in the domain of
whole-body PET/CT lesion segmentation has remained elusive [12], [17], [8],
[16], [7] largely due to the scarcity of training data in preceding studies
[3]. Drawing upon the insights provided by the AutoPET Challenge U-Net-based
winner from 2022 [16], we undertake a practical investigation to understand
the important training parameters of the U-Net model for segmenting lesions.
We believe that it is possible to achieve a better and more robust model by
focusing on the intricacies of data pre-processing, data augmentation,
learning rate scheduling, and crop-size selection during model training. Our
work and model are based on prior experiments in interactive segmentation
[14]. Thus, for our hyperparameter tuning experiments, we present results
using our interactive model. Nonetheless, for our final submission, we exclude
the integration of interactive clicks into the model and employ its optimal
hyperparameter configuration.

##  2 Methodology

###  2.1 Model Architecture

The model used for the challenge is called DynUNet, which is an adaption of
the UNet for the MONAI library [2]. Contrary to the default UNet, DynUNet does
not use max-pooling for downsampling but instead uses strided convolutions.
Additionally, the residual is passed through a convolutional layer such that
the input size from the downsampling layer matches the output size of this
layer. All of the changes can be traced back to three prior works: [10], [11],
and [5].

Our default configuration of the network consists of six layers of filter size
[32, 64, 128, 256, 320, 320]. As discussed above, the convolutions are strided
with a size of [1, 2, 2, 2, 2, [2, 2, 1]], and the upsampling is thus done in
the inverse order. An architectural diagram can be found in Figure 1.

Figure 1: An overview of the used DynUNet architecture.

###  2.2 Data Pre-processing and Augmentation

Pre-processing. We restrict ourselves by using only the PET volumes from the
paired PET/CT scans. We apply multiple pre-processing transformations to each
batch of data. Apart from changing the channel order, the orientation is set
to a RAS (Right-Anterior-Superiror) coordinate system. As the AutoPET spacing
is $\approx$[2, 2, 3]mm3, the data is resampled accordingly with this fixed
voxel size. The intensity of each PET image is scaled, based on its voxel
intensity statistics, with MONAI’s ScaleIntensityRangePercentiled to the 0.05
and 99.95 percentiles. During training, a random crop of size 224x224x224 is
sampled, with a probability of 0.6 of being centered around a tumor lesion and
0.4 of being centered around the background. To achieve this, we utilize the
RandCropByPosNegLabeld MONAI transform. This crop is balanced by the class
label of the voxel in the crop’s center - in 60% of the cases the voxel is
positive, and in the other 40% it is negative. This ensures that the network
learns about positive and negative samples in a more balanced training regime.

Data Augmentation. We apply two types of data augmentation - random flipping
and random rotation. We apply a random flip on each spatial axis with a
probability of 0.1. We also apply a random 90-degree rotation with a
probability of 0.1 for each axis.

###  2.3 Data Post-processing

Since we are using a sliding window approach, the final prediction volume gets
stitched from the various output patches. This process is done with a user
defined overlap, in our case this was set to 75%.

After the result prediction a softmax is applied.

For the ensemble based solution the two steps mentioned above are done for
each of the five networks prediction separately. After the softmax on each
prediction a voting mechanism combines the different predictions into a single
one.

###  2.4 Hyperparameter Tuning

As explained above most of the different experiments have been run on
interactive code. Nevertheless they should be representative in terms of
general performance of the network. Variations of +/- 0.5% Dice are to be
expected since the guidance signal was non-deterministic.

####  2.4.1 Sliding window versus normal inferer

First of all we compare the sliding window infererence to the normal one,
figure 1. As it can be seen in the table, on the interactive code the sliding
window inferer wins with a lead of 2.81% Dice.

Next different region of interest sizes have been tried out. The best
performing one here was the 128x128x128 crop. Note that the sliding window was
active during training. In the thesis it is shown that training with overlap
active gains about 1% of Dice.

This overlap means for the 128x128x128 instead a window of size 320x320x320
has been calculated, with calculations being equal to a normal inferer of size
384x384x384. As we can see a lot of overhead calculations are being done by
the sliding window inferer. However in the next subsection we will show that
the overhead calculations for the overlap actually lead to a better Dice
score.

Table 1: Interactive run of Sliding Window versus Simple Inferer | Sliding Window | Simple Inferer
---|---|---
Dice | 83.83% | 81.02%
Table 2: Different region of interest sizes compared. Trained on a crop of size 256x256x256. | 64x64x64 | 128x128x128 | 192x192x192 | 256x256x256
---|---|---|---|---
Dice (validation) | 84.74% | 85.22% | 83.66% | 84.75%
Dice (training) | 87.99% | 88.46% | 88.98% | 88.79%

####  2.4.2 Sliding window overlap

Now we will look at the overlap of the sliding window inferer. Table 3 shows
that increasing the overlap also increases the Dice score of the network. In
our experiments the higher the overlap the better the results have been. This
can be seen as a way of creating a mini Ensemble with same weights. The
overlap uses a Gaussian fade away to make the regions closer the center weight
more heavily when stitching together the final output.

Additionally experiments have been run to verify the impact of training with
overlap on. Table 4, which shows a network trained on 0% overlap, overall
shows slightly worse results, especially for the higher overlaps it becomes
significant. As expected running it with 0 overlap returns slightly better
results than the network trained with overlap being forced to use none. We can
thus conclude that activating overlap during training enhances the final
score.

Table 3: Non-interactive validation runs with different settings for the overlap. The network has been trained on 25% overlap. Experiment | Overlap | Dice
---|---|---
201 | 0 | 66.33%
202 | 0.25 | 73.04%
203 | 0.5 | 73.54%
207 | 0.75 | 74.07%
Table 4: Non-interactive validation runs with different settings for the overlap. The network has been trained on 0% overlap. Experiment | Overlap | Dice
---|---|---
v_208_0.0 | 0 | 66.57%
v_208_0.25 | 0.25 | 71.35%
v_208_0.5 | 0.5 | 71.99%
v_208_0.75 | 0.75 |  72.86%

####  2.4.3 Convergence behaviour with different losses

Figure 3(b) shows the convergence behaviour of the Dice loss vs the
DiceCELoss. As it can be seen the DiceCELoss start with a higher initial
validation Dice in epoch 10, 73.62% against 70.09%. Also the final Dice metric
was a little bit higher, 85.47% for DiceCELoss and 84.62% for Dice loss.
However a plateau appears to be reached for both losses. In other experiments
with more iterations it was shown that this method can reach a validation Dice
of up to 87.60%.

We can thus fully recommend the DiceCELoss as a standard choice for training.
It converges faster and also yields higher final scores especially in terms of
Dice.

Figure 2: Comparing the Dice Loss, in MONAI called MeanDice to the DiceCELoss.

(a) Validation Dice of run 158. In this run the DiceCELoss has been used as
the loss.

(b) Validation Dice of run 183. In this run the MeanDice has been used as the
loss.

####  2.4.4 Intensity scaling options

Finally a quick comparison of different intensity scaling options. The base
run was a pre-calculated batch statistics normalization to the 0.005 and 99.95
percentiles of the intensity. The first ScaleIntensityRangePercentiled applied
the same percentiles but this time based on the statistics of a each item. The
last ScaleIntensityRangePercentiled is a base run with no clipping of the
intensities, it only normalizes the intensity from 0 to 1.

As we can see the item-wise statistics outperformed the batch-wise statistics
and the clipless method.

Table 5: Different ScaleIntensity settings compared. | Base run | ScaleIntensity- | ScaleIntensity-
---|---|---|---
| CosineAnnealingLr | RangePercentiled | RangePercentiled 2
| (104) | (148) | (149)
Dice | 85.63% | 86.69% | 85.44%
Table 6: Number of training epochs for each cross-validation model in our final submission of an ensemble and its corresponding best Train Dice. Model | # Epochs | Best Train Dice | Notes
---|---|---|---
1 | 414 | 83.92% | Network did not get included due to NaN errors
2 | 564 | 87.33% |
3 | 447 | 85.92% |
4 | 411 | 85.44% |
5 | 391 | 86.76% |

####  2.4.5 Best parameters

A summary of the best found results can be found in table 7.

Table 7: Best settings Parameter name |  Setting
---|---
Network |  DynUNet with [32, 64, 128, 256, 320, 320] filters and a depth of sex layers
Loss |  DiceCELoss with squared_pred=True
|  and include_background=True
Optimizer |  Adam
Learning rate scheduler |  CosineAnnealingLR (initial lr=2e-4, eta_min=1e-8)
Inferer |  Sliding window inferer with ROI size 128x128x128, sliding window overlap 0.75
Intensity scaling with |  Custom Scaling to 0.05% and 99.95% intensity percentiles using ScaleIntensityRanged
Automatic Mixed Precision |  Active

##  3 Proposed solutions to the AutoPET2 Challenge

We propose two different approaches for the challenge as final submissions:

  * •

A single network with six layers as stated above, trained for 400 epochs.

  * •

An ensemble of five networks, each with same six layers. Four of the networks
were trained with cross-validation on five splits of the data. The network
trained on the first split did not get included, since it ran into NaN errors
very quickly. They were trained without using the validation split for 800
epochs with no validation runs. However, all of the five networks did not
finish in time for the challenge, so the most recent checkpoint was picked
instead, where Table 6 summarizes how many epochs each model was trained for.

Additionally, the best-performing single network was integrated as a teacher,
bringing the total to five networks working collaboratively. The results of
the different networks got combined with an equally weighted voting mechanism.

##  4 Results

The results of our two final submissions can be seen in Table 8. The Dice
score is similar in both approaches but the FPV is significantly reduced in
the ensemble, perhaps due to the smoothening effect on the predictions which
filters outliers outside of the object. However, the single network has a much
lower FNV, signifying a higher sensitivity to tumor detection.

Method | Train dice | Results on the preliminary test set
---|---|---
|  | Dice score | False negative volume | False positive volume
Single network | 86.76% | 56.52% | 0.0249 | 1.8015
Ensemble | 86.16% | 56.60% | 0.0572 | 1.0475
Table 8: The results of our method in the AutoPET2 challenge.

##  5 Post mortem: NaN errors during training if AMP is active

In the preparation for the challenge we ran into NaN errors when training on
A100 GPUs, but only when automated mixed precision was on. During the
debugging we found out our input already contained NaNs.

The reason in our case was a training crop to positive / negative areas of
size 224x224x224. At the borders of the volume this resulted in crops which
contained almost only 0s or even only 0s. Our current hypothesis is that the
normalization on the crop produces division by 0 errors. This would make
especially sense for the intensity scaling which might degrade if the input
tensor only contains 0s. However more debugging is necessary to find out the
exact transform which produces the NaN errors.

The solution is to add a filter after the pre-transform to remap all NaN
values to 0. In our case this fixed the problem and we could resume training
with AMP on the A100 GPUs.

##  6 Acknowledgment

The present contribution is supported by the Helmholtz Association under the
joint research school “HIDSS4Health – Helmholtz Information and Data Science
School for Health“. This work was performed on the HoreKa supercomputer funded
by the Ministry of Science, Research and the Arts Baden-Württemberg and by the
Federal Ministry of Education and Research.

## References

  * [1] Ben-Haim, S., Ell, P.: 18f-fdg pet and pet/ct in the evaluation of cancer treatment response. Journal of Nuclear Medicine 50(1), 88–99 (2009)
  * [2] Cardoso, M.J., Li, W., Brown, R., Ma, N., Kerfoot, E., Wang, Y., Murrey, B., Myronenko, A., Zhao, C., Yang, D., et al.: Monai: An open-source framework for deep learning in healthcare. arXiv preprint arXiv:2211.02701 (2022)
  * [3] Egger, J., Gsaxner, C., Pepe, A., Pomykala, K.L., Jonske, F., Kurz, M., Li, J., Kleesiek, J.: Medical deep learning–a systematic meta-review. Computer Methods and Programs in Biomedicine p. 106874 (2022)
  * [4] Eisenmann, M., Reinke, A., Weru, V., Tizabi, M.D., Isensee, F., Adler, T.J., Ali, S., Andrearczyk, V., Aubreville, M., Baid, U., et al.: Why is the winner the best? In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 19955–19966 (2023)
  * [5] Futrega, M., Milesi, A., Marcinkiewicz, M., Ribalta, P.: Optimized u-net for brain tumor segmentation. In: International MICCAI Brainlesion Workshop. pp. 15–29. Springer (2021)
  * [6] Gatidis, S., Früh, M., Fabritius, M., Gu, S., Nikolaou, K., La Fougère, C., Ye, J., He, J., Peng, Y., Bi, L., et al.: The autopet challenge: Towards fully automated lesion segmentation in oncologic pet/ct imaging (2023)
  * [7] Hallitschke, V.J., Schlumberger, T., Kataliakos, P., Marinov, Z., Kim, M., Heiliger, L., Seibold, C., Kleesiek, J., Stiefelhagen, R.: Multimodal interactive lung lesion segmentation: A framework for annotating pet/ct images based on physiological and anatomical cues. arXiv preprint arXiv:2301.09914 (2023)
  * [8] Heiliger, L., Marinov, Z., Hasin, M., Ferreira, A., Fragemann, J., Pomykala, K., Murray, J., Kersting, D., Alves, V., Stiefelhagen, R., et al.: Autopet challenge: Combining nn-unet with swin unetr augmented by maximum intensity projection classifier. arXiv preprint arXiv:2209.01112 (2022)
  * [9] Isensee, F., Jaeger, P.F., Kohl, S.A., Petersen, J., Maier-Hein, K.H.: nnu-net: a self-configuring method for deep learning-based biomedical image segmentation. Nature methods 18(2), 203–211 (2021)
  * [10] Isensee, F., Jäger, P.F., Kohl, S.A., Petersen, J., Maier-Hein, K.H.: Automated design of deep learning methods for biomedical image segmentation. arXiv preprint arXiv:1904.08128 (2019)
  * [11] Isensee, F., Petersen, J., Klein, A., Zimmerer, D., Jaeger, P.F., Kohl, S., Wasserthal, J., Koehler, G., Norajitra, T., Wirkert, S., et al.: nnu-net: Self-adapting framework for u-net-based medical image segmentation. arXiv preprint arXiv:1809.10486 (2018)
  * [12] Liu, Z., Zhong, S., Mo, J.: Autopet challenge 2022: Step-by-step lesion segmentation in whole-body fdg-pet/ct. arXiv preprint arXiv:2209.09199 (2022)
  * [13] Marinov, Z., Reiß, S., Kersting, D., Kleesiek, J., Stiefelhagen, R.: Mirror u-net: Marrying multimodal fission with multi-task learning for semantic segmentation in medical imaging. arXiv preprint arXiv:2303.07126 (2023)
  * [14] Marinov, Z., Stiefelhagen, R., Kleesiek, J.: Guiding the guidance: A comparative analysis of user guidance signals for interactive segmentation of volumetric images. arXiv preprint arXiv:2303.06942 (2023)
  * [15] Ronneberger, O., Fischer, P., Brox, T.: U-net: Convolutional networks for biomedical image segmentation. In: Medical Image Computing and Computer-Assisted Intervention–MICCAI 2015: 18th International Conference, Munich, Germany, October 5-9, 2015, Proceedings, Part III 18. pp. 234–241. Springer (2015)
  * [16] Ye, J., Wang, H., Huang, Z., Deng, Z., Su, Y., Tu, C., Wu, Q., Yang, Y., Wei, M., Niu, J., et al.: Exploring vanilla u-net for lesion segmentation from whole-body fdg-pet/ct scans. arXiv preprint arXiv:2210.07490 (2022)
  * [17] Zhong, S., Mo, J., Liu, Z.: Autopet challenge 2022: Automatic segmentation of whole-body tumor lesion based on deep learning and fdg pet/ct. arXiv preprint arXiv:2209.01212 (2022)