[affiliation=1]LiangyuNie [affiliation=2]Sudarsana ReddyKadiri
[affiliation=3]RuchitAgrawal

# MMSD-Net: Towards Multi-modal Stuttering Detection

###### Abstract

Stuttering is a common speech impediment that is caused by irregular
disruptions in speech production, affecting over 70 million people across the
world. Standard automatic speech processing tools do not take speech ailments
into account and are thereby not able to generate meaningful results when
presented with stuttered speech as input. The automatic detection of
stuttering is an integral step towards building efficient, context-aware
speech processing systems. While previous approaches explore both statistical
and neural approaches for stuttering detection, all of these methods are uni-
modal in nature. This paper presents MMSD-Net, the first multi-modal neural
framework for stuttering detection. Experiments and results demonstrate that
incorporating the visual signal significantly aids stuttering detection, and
our model yields an improvement of 2-17% in the F1-score over existing state-
of-the-art uni-modal approaches.

###### keywords:

Speech disorders, Stuttering detection, Multimodal neural networks,
Transformer

##  1 Introduction

Recent advancements in machine learning have enabled a wide range of AI
applications across a myriad of sectors such as government, industry,
healthcare and transportation. The automatic recognition and transcription of
speech is an integral task in machine learning, which enables users to
interact seamlessly with machines, lending itself applicable to a variety of
tasks such as automatic translation of speech, subtitling of multimedia
content, audio signal analysis and editing, and so on. In particular, the
massive developments in speech processing have given rise to an unprecedented
surge of virtual/digital assistants such as Siri and Alexa, used by millions
of users across the globe. However, these modern speech processing tools are
not perfect, and are unable to deal with a range of speech impediments.

Stuttered speech is one such common speech impairment, wherein the production
of fluent speech is hindered by a malfunctioning of the central nervous system
of the afflicted person. This makes it highly challenging for people who
stutter to access popular speech recognition tools such as Siri and Alexa. As
an example, Apple's Siri reported an accuracy ranging from 18.2-73% when given
stuttered speech as input, compared to a high accuracy of 92% when presented
with normal speech as input [1], which makes it impractical to be used by
people who stutter. Stuttering can manifest itself in various types of
disfluencies, such as sound repetition, part-word repetition, word repetition,
phrase repetition, revision, interjection, prolongation and block [2]. The
lack of robust speech recognition of stuttered speech is an unfair consequence
for a significant portion of the world population, with over 70 million people
being affected by this condition [3].

Automatic stuttering detection is an integral step towards mitigating this
limitation of existing speech processing tools. While a number of approaches
have been proposed in the recent years for stuttered speech detection, these
are either audio-based or text-based, and therefore uni-modal in nature. The
application of multi-modal neural networks for stuttering detection remains
unexplored. This paper aims to bridge this gap and presents the first multi-
modal neural network framework for stuttered speech detection. The primary
motivation behind our proposed multi-modal approach is that cues to detect
stuttering could be found not only in the audio signal, but also on the
speakers' faces. Our hypothesis is that these visual signals contain relevant
information pertitent to the SD task. We validate our hypothesis by conducting
experiments across a range of settings and demonstrating results on publicly
available datasets.

The primary contributions of this paper are summarized below:

  * •

We present MMSD-Net, the first multi-modal neural network for automatic
stuttering detection.

  * •

Our proposed architecture effectively integrates audio, video, and language
data using a novel multi-modal fusion mechanism, which enhances feature fusion
for superior multi-modal task performance.

  * •

We conduct experimental studies using publicly available datasets and present
extensive comparisons of the results obtained by our model against the state-
of-the-art uni-modal methods for stuttering detection.

  * •

We demonstrate that MMSD-Net outperforms state-of-the-art uni-modal methods by
2-17 % on F1-score.

  * •

We release the code for pre-processing, post-processing as well as the neural
network models publicly to ensure reproducibility of our research.

##  2 Related Work

The automatic detection of stuttering has been approached through various
methodologies, primarily divided into statistical approaches and deep learning
approaches. Statistical methods rely on feature extraction from speech signals
and subsequent classification using machine learning algorithms. These
approaches often employ features such as autocorrelation functions, spectral
measures, and Mel-frequency cepstral coefficients (MFCCs), utilizing
classifiers like support vector machines (SVM), hidden Markov models (HMM),
and artificial neural networks (ANN) [4, 5, 6, 7]. While statistical methods
have shown promising results, they often require manual feature engineering
and may not generalize well across different datasets due to their dependency
on specific feature sets and classifiers. For example, SVMs may struggle with
datasets containing imbalanced classes or noisy features, limiting their
performance in real-world scenarios [8]. Similarly, HMMs have been successful
in modeling temporal sequences in speech, but they may struggle with capturing
complex dependencies in stuttered speech, particularly in distinguishing
between different types of disfluencies [7].

The advancements in deep learning have led to the exploration of deep neural
networks for stutter detection [9, 10, 11]. Recent approaches have explored
the usage of the perceptron model [10] and autoregressive models such as Long-
Short Term Memory networks (LSTMs) [11, 12]. While the former employs LSTMs
with integer linear programming [13], the latter employs bidirectional LSTMs
with attention using the MFCC features. Authors in [9] propose a CNN-based
model to learn stutter-related features, formulating SD as a binary
classification problem. The only input features used in this study are the
spectrograms. Another approach called FluentNet [14] builds upon this method
and explores the use of a residual network along with an LSTM network to learn
frame-level representations. However, these methods only consider a small
subset of disfluent speakers in their studies, and are not tested exhaustively
on their ability to generalize well to a variety of stuttered speakers.

With the advent of the Transformer model [15, 16] propose using a controllable
time-delay transformer architecture, but only for Chinese data. A similar
approach proposed recently, called StutterNet [17], employs a time delay
neural network and formulates the stutter detection task as a multi-class
classification problem. However, this method only consider limited disfluent
behaviours (blocks, repetition, and prolongation) in addition to fluent speech
segments.

A major challenge to train deep neural networks is the availability of large-
scale annotated datasets. To address this problem, [18] curate a large-scale
dataset for stuttering detection, called SEP-28k. They also present
experiments using the ConvLSTM model and demonstrate results on the
FluencyBank as well as SEP-28k datasets. More recently, [19] employ the
Wav2Vec model for stuttering detection and explore self-supervised learning
for this task. The closest to our work is the method called FluentSpeech
proposed by [20]. However, it is uni-modal and does not leverage the visual
signal. Additionally, it considers pauses, non-lexical vocalisations and
interjections such as so, hmmm, umm, like in speech as stuttered segments
[21]. We disagree from their method in that such normal disfluencies could
correspond to useful pauses in speech wherein the speaker can plan their
upcoming discourse; and are notably different from stuttering, which is a
neuro-developmental speech disorder that is characterized by core behaviour
and corresponds to abnormally persistent stoppages in normal speech, often
accompanied by unusual behaviours such as quick eye blinks, lip tremors and
nodding of head [22].

It must be noted that while there have been several approaches proposed for
stuttering detection, these rely on classical machine learning (typically on
the audio signal) or uni-modal deep learning methods. The application of
multi-modal deep learning remains unexplored for this task. We bridge this gap
and present MMSD-Net, the first multi-modal deep learning method for stutter
event detection.

##  3 Proposed Methodology

Figure 1: Illustration of multi-modal cues during stuttering.
Sentence: "I was working on fluency shaping techniques."

This section presents a detailed overview of MMSD-Net. Technically, stutter
presents itself as an audiovisual problem. Figure 1 demonstrates a data sample
from the Adults Who Stutter dataset [23], depicting a snippet of the video,
and the corresponding waveform, spectrogram and textual transcription
respectively. The primary motivation of our method is that cues for stuttering
can be found from the visual signal as well as the audio signal, as
illustrated in Figure 1. It can be observed that facial expressions change
during stuttering and can thereby provide important contributions for the
automatic stutter detection task. To this end, we employ the visual signal in
addition to the audio signal, and present MMSD-Net, the first multi-modal
neural approach towards stuttering detection.

Model architecture: Figure 2 presents an overview of our model architecture.
MMSD-Net comprises three primary modules, described in the subsequent
subsections:

Figure 2: Detailed architecture of our proposed model MMSD-Net

###  3.1 Multi-encoder module

Current Multimodal Language Models are mainly geared towards understanding
video and text-based content. MMSD-Net incorporates specialized modality
encoders to process not just video and text, but also auditory data. These
encoders are designed to extract the most relevant features from each
modality. This upgrade significantly enhances MMSD-Net's ability to process
and interpret information across various modalities efficiently. Unlike
traditional models that might rely on Convolutional Neural Networks (CNNs),
MMSD-Net leverages three transformer encoders for video, audio and text data
respectively. Transformers excel at capturing long-range dependencies within
sequences, making them well-suited for stuttering detection. By capturing
these nuanced details, MMSD-Net gains a richer understanding of the input data
corresponding to different modalities, allowing for more comprehensive
reasoning. The specialized encoders process the video (v), audio (a) and
textual (t) inputs as follows:

$\mathbf{h}_{v}=\text{Model}(\mathbf{v})\in\mathbb{R}^{d_{v}}$ (1)
$\mathbf{h}_{a}=\text{Model}(\mathbf{a})\in\mathbb{R}^{d_{a}}$ (2)
$\mathbf{h}_{t}=\text{Model}(\mathbf{t})\in\mathbb{R}^{d_{t}}$ (3)

where $\mathbf{h}_{v}$, $\mathbf{h}_{a}$, and $\mathbf{h}_{t}$ represent the
extracted features for video, and audio, respectively. $d_{v}$, $d_{a}$, and
$d_{t}$ denote the dimensionality of the features for each modality.

To reduce computational costs and minimize the number of tokens in the prefix,
we employ a 1-D convolutional layer to compress the length of the multi-modal
features to a smaller value. Subsequently, a linear layer is employed to
adjust the hidden size of the features, fusing it with the size of the MLMs'
embeddings as follows:

$\mathbf{h}^{\prime}_{v}=\text{Linear}(\text{Conv1D}(\mathbf{h}_{v}))$ (4)
$\mathbf{h}^{\prime}_{a}=\text{Linear}(\text{Conv1D}(\mathbf{h}_{a}))$ (5)
$\mathbf{h}^{\prime}_{t}=\text{Linear}(\text{Conv1D}(\mathbf{h}_{t}))\\\ $ (6)

where
$\mathbf{h}^{\prime}_{v},\mathbf{h}^{\prime}_{a},\mathbf{h}^{\prime}_{t}$ are
the transformed features with a fixed length of $L^{\prime}$ and an embedding
dimension of $d_{e}$. The value of $L^{\prime}$ is significantly smaller than
$L_{v}$, $L_{a}$, and $L_{t}$, while $d_{e}$ corresponds to the dimensionality
of the embedding matrix $\mathbf{E}\in\mathbb{R}^{v\times d_{e}}$ associated
with the MMSD-Net.

###  3.2 Multimodality Fusion Module

Modality encoders are often trained independently, which can result in
differences in the representations they produce. Therefore, it is essential to
merge these distinct representations into a unified space. We design our
fusion strategy based on the Q-Former [24], and modify it to efficiently
integrate video and audio features with textual features, thereby facilitating
more rapid adaptation. In this context, we designate the video features
derived from our visual modality encoder by $h_{v}\in\mathbb{R}^{L_{v}\times
d_{v}}$, and the audio features extracted from the audio modality encoder are
denoted as $h_{a}\in\mathbb{R}^{L_{a}\times d_{a}}$.

In order to fuse the distinct representations learnt separately by the
individual modality encoders, we consider the transformed visual and audio
modality representations obtained in Equation 4, 5 and 6 as the soft tokens of
our MLM module. The visual and audio representations are fused with the
textual embedding space using the attention mechanism from Equation 1, as
follows:

$\mathbf{h}^{a}=\text{Attn}(\mathbf{h}^{\prime},\mathbf{E},\mathbf{E})$ (7)

where $\mathbf{h}^{\prime}$ is the modality representation obtained in
Equations 4, 5, and 6, E is the embedding matrix as defined in Section 3.1,
and $\mathbf{h}^{a}$ is the corresponding fused representation, specifically
$\mathbf{h}^{a}_{v}$, $\mathbf{h}^{a}_{a}$, and $\mathbf{h}^{a}_{t}$. After
this fusion operation facilitated by the attention mechanism, the MMSD can
seamlessly process the representations from various modalities.

In order to integrate the fused modality representations with the instruction
information, we employ the concatenation operation. Given the fused modality
representations, we define the integration as follows

$x=[h^{a}_{v}:h^{a}_{a}:h^{a}_{t}:Embed(x_{t})]$ (8)

where $[:]$ represents the concatenation operation, $x$ represents the multi-
modal instruction, $x_{t}$ represents the sequence of tokens in the textual
instruction, and $Embed$ ($x_{t}$) represents the sequence of embeddings of
$x_{t}$.

###  3.3 MLM module

Multimodal Language Models (MLM) [25] have demonstrated exceptional aptitude
in comprehending and executing human directives. In particular, cross-modal
approaches have demonstrated improved performance for audio synchronization
tasks [26], [27]. In MMSD-Net, we utilize pretrained MLMs as the core modules,
establishing the basis of MMSD-Net's functionality. The pretrained MLM network
processes three primary inputs: the query vector $Q\in\mathbb{R}^{n_{q}\times
d_{q}}$, the key vector $K\in\mathbb{R}^{n_{k}\times d_{k}}$, and the value
vector $V\in\mathbb{R}^{n_{v}\times d_{v}}$. Through the scaled dot-product
attention, the model evaluates attention scores by comparing each query in $Q$
with all keys in $K$. These scores are then utilized to refine the query
representations by generating a weighted sum of the values in $V$. The
operation is mathematically represented as:

$Attention(Q,K,V)=softmax(\frac{QK^{T}}{\sqrt{d_{k}}})V$

Here, $d_{k}$ denotes the size of the key and query vectors, while $n_{q}$ and
$n_{k}$ represent the counts of queries and keys, respectively.

$MultiHead(Q,K,V)=Concat(head_{1},...,head_{h})W^{O}$

We use stacked multi-head attention as well as positional encodings to model
the complete sequential information encoded by the inputs. The decoder employs
masked multi-head attention followed by softmax normalization for binary
classification. The positional encodings are added to the input as well as
output embeddings, enabling the model to capture the sequentiality of the
input sentence without having recurrence. The encodings are computed from the
position ($pos$) and the dimension ($i$) as follows:

$PE_{(pos,2i)}=sin(pos/10000^{(2i/d_{model})})$ (9)
$PE_{(pos,2i+1)}=cos(pos/10000^{(2i/d_{model})})$ (10)

where $PE$ stands for positional encodings and $d_{model}$ is the
dimensionality of the vectors resulting from the embeddings learned from the
input and output tokens.

##  4 Experiments and Results

We conduct experiments for the detection of stuttered speech using our
proposed model MMSD-Net, and compare the results obtained by our model with
four state-of-the art baseline models, i.e. FluentSpeech [28], ResNet+BiLSTM
[9], ConvLSTM [18] and StutterNet [17].

###  4.1 Datasets

This study uses four publicly available datasets (1 audio-modality and 3
audio-visual modalities) for the experiments. The details of modalities
present in each dataset, along with the amount of data used for training and
testing in each of our experiments, are described in Table 1.

Table 1: Details of datasets used for our experiments.

Dataset Name |  Modality |  Content |  Training set |  Testing
set
---|---|---|---|---
Sept28K [18] |  Audio |  Normal Speech |  28,000 |  0
FluencyBank [23] |  Video,
Audio |  Normal
+stuttered speech |  52,000 |  0
Adults Who
Stutter [23] |  Video,
Audio |  Stuttered Speech |  200 |  500
SpeakingFaces [29] |  Video,
Audio |  Normal speech |  200 |  500

###  4.2 Experimental Setup and Hyperparameters

For training efficiency, we leverage LoRA [30] for optimization on 4 Nvidia
A100 GPUs. Each GPU handles a batch size of 4, with gradients accumulated for
5 steps before updating the model. The training process lasts for 10 epochs,
employing a cosine learning rate scheduler with an initial learning rate of
$5\times 10^{-10}$ and a warmup ratio of 0.02. To promote efficiency, FP16
precision is used for both training and inference. The maximum sequence length
is capped at 512 tokens.

###  4.3 Results and Discussion

Table 2 gives the results (in terms of precision, recall and F1-score) of the
proposed MMSD-Net method along with the four baseline methods. The results are
presented for the dataset obtained by combining the FluencyBank [23], Sept28K
[18] and Adults Who Stutter [23] datasets as described in Table 1. In addition
to model performance, the comparison of modalities used by each method is also
presented in the Table 2. Among the four baseline methods, StutterNet [17]
achieved better results compared to other three methods (FluentSpeech [28],
ResNet+BiLSTM [9] and ConvLSTM [18]), and ResNet+BiLSTM [9] achieved the
lowest scores among the baseline methods. While both StutterNet [17] and
ResNet+BiLSTM [9] are specifically trained on audio data, StutterNet achieves
superior performance using MFCC features extracted from the audio samples.
This suggests that MFCC features are more suitable for stutter detection
compared to the features employed by ResNet+BiLSTM, i.e. the spectrograms.

Our proposed MMSD-Net outperforms all other methods in terms of precision,
recall, and F1-score, achieving the highest scores across all metrics, which
demonstrates its superiority in stuttering detection. Our findings demonstrate
that the fusion module can effectively combine information from three
different modalities. Quantitatively, the proposed MMSD-Net gave an absolute
improvement of 2% in the F1-score (and 3% in the Precision) over the best
baseline method (StutterNet [17]) and 16% in the F1-score (and 17% in the
Precision) over ResNet+BiLSTM [9]. These results validate our hypothesis that
facial expressions serve as an important cues to detect stuttered speech, and
employing the visual signal as part of a multi-modal framework improves the
performance of automatic stuttering detection.

Table 2: Comparison of results obtained by the proposed MMSD-Net with the four
baseline methods. Best result highlighted in bold, second best underlined. P =
Precision, R = Recall, F1 = F1-score.

.

Model Name |  Audio |  Video |  P |  R |  F1
---|---|---|---|---|---
FluentSpeech [28] |  ✓ |  ✗ |  85.73 |  81.82 |  83.72
ResNet+BiLSTMs [9] |  ✓ |  ✗ |  75.28 |  72.73 |  73.98
StutterNet [17] |  ✓ |  ✗ |  89.41 |  87.10 |  88.23
ConvLSTM [18] |  ✓ |  ✗ |  82.63 |  78.32 |  80.41
MMSD-Net |  ✓ |  ✓ |  92.58 |  87.93 |  90.19

##  5 Conclusion

This study presents MMSD-Net, the first multi-modal neural framework crafted
explicitly for stuttered speech detection. We conducted experiments on
publicly available datasets and performed studies comparing against four
existing uni-modal baselines. Our findings showcase noteworthy improvements in
stuttered speech detection accuracy, with enhancements ranging from 2-17% in
F1-score over established baseline models, indicating the effectiveness of our
multi-modal approach for stuttering detection. This paper signifies a major
stride towards augmenting the efficacy of stuttered speech detection, and
highlights the complementarity of multiple modalities for this task. In
particular, it demonstrates that the visual signal carries relevant
information for the stuttering detection task, and lays the groundwork for the
development of further advancements in speech processing tools catered to
individuals suffering from speech impediments. In the future, we would like to
extend our experimentation to larger datasets and conduct a qualitative
analysis of the impact of multi-modality on handling various kinds of
stuttering.

## References

  * [1] E. Mullin, ``Why Siri won’t listen to millions of people with disabilities,'' _Scientific American. Retrieved January_ , vol. 8, p. 2018, 2016.
  * [2] J. E. Prasse and G. E. Kikano, ``Stuttering: an overview,'' _American family physician_ , vol. 77, no. 9, pp. 1271–1276, 2008.
  * [3] E. Yairi and N. Ambrose, ``Epidemiology of stuttering: 21st century advances,'' _Journal of Fluency Disorders_ , vol. 38, no. 2, pp. 66–87, 2013.
  * [4] P. Howell and S. Sackin, ``Automatic recognition of repetitions and prolongations in stuttered speech,'' in _Proceedings of the first World Congress on fluency disorders_ , vol. 2. University Press Nijmegen Nijmegen, The Netherlands, 1995, pp. 372–374.
  * [5] P. Howell, S. Sackin, and K. Glenn, ``Development of a two-stage procedure for the automatic recognition of dysfluencies in the speech of children who stutter: ANN recognition of repetitions and prolongations with supplied word segment markers,'' _Journal of Speech, Language, and Hearing Research_ , vol. 40, no. 5, pp. 1085–1096, 1997.
  * [6] T.-S. Tan, A. Ariff, C.-M. Ting, S.-H. Salleh _et al._ , ``Application of malay speech technology in malay speech therapy assistance tools,'' in _International Conference on Intelligent and Advanced Systems_. IEEE, 2007, pp. 330–334.
  * [7] K. Ravikumar, R. Rajagopal, and H. Nagaraj, ``An approach for objective assessment of stuttered speech using MFCC,'' in _The International Congress for Global Science and Technology_ , vol. 19, 2009.
  * [8] J. Cervantes, F. Garcia-Lamont, L. Rodríguez-Mazahua, and A. Lopez, ``A comprehensive survey on support vector machine classification: Applications, challenges and trends,'' _Neurocomputing_ , vol. 408, pp. 189–215, 2020.
  * [9] T. Kourkounakis, A. Hajavi, and A. Etemad, ``Detecting multiple speech disfluencies using a deep residual network with bidirectional long short-term memory,'' in _IEEE International Conference on Acoustics, Speech and Signal Processing, (ICASSP), Barcelona, Spain, May 4-8, 2020_ , 2020, pp. 6089–6093.
  * [10] B. Villegas, K. M. Flores, K. J. Acuña, K. Pacheco-Barrios, and D. Elias, ``A novel stuttering disfluency classification system based on respiratory biosignals,'' in _IEEE Engineering in Medicine and Biology Society (EMBC)_. IEEE, 2019, pp. 4660–4663.
  * [11] V. Zayats, M. Ostendorf, and H. Hajishirzi, ``Disfluency detection using a bidirectional lstm,'' _Interspeech 2016_ , 2016.
  * [12] J. Santoso, T. Yamada, and S. Makino, ``Classification of causes of speech recognition errors using attention-based bidirectional long short-term memory and modulation spectrum,'' in _Asia-Pacific Signal and Information Processing Association Annual Summit and Conference (APSIPA ASC)_ , 2019, pp. 302–306.
  * [13] K. Georgila, ``Using integer linear programming for detecting speech disfluencies,'' in _Proceedings of Human Language Technologies: The Annual Conference of the North American Chapter of the Association for Computational Linguistics, Companion Volume: Short Papers_ , 2009, pp. 109–112.
  * [14] T. Kourkounakis and A. Etemad, ``FluentNet: End-to-end detection of stuttered speech disfluencies with deep learning,'' _IEEE/ACM Transactions on Audio, Speech, and Language Processing_ , vol. 29, pp. 2986–2999, 2021.
  * [15] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, ``Attention is all you need,'' _Advances in neural information processing systems_ , vol. 30, 2017.
  * [16] Q. Chen, M. Chen, B. Li, and W. Wang, ``Controllable time-delay transformer for real-time punctuation prediction and disfluency detection,'' in _IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , 2020, pp. 8069–8073.
  * [17] S. A. Sheikh, M. Sahidullah, F. Hirsch, and S. Ouni, ``StutterNet: Stuttering detection using time delay neural network,'' in _European Signal Processing Conference (EUSIPCO)_. IEEE, 2021, pp. 426–430.
  * [18] C. Lea, V. Mitra, A. Joshi, S. Kajarekar, and J. P. Bigham, ``SEP-28k: A dataset for stuttering event detection from podcasts with people who stutter,'' in _IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , 2021, pp. 6798–6802.
  * [19] S. P. Bayerl, D. Wagner, E. Nöth, and K. Riedhammer, ``Detecting dysfluencies in stuttering therapy using wav2vec 2.0,'' _arXiv preprint arXiv:2204.03417_ , 2022.
  * [20] Z. Jiang, Q. Yang, J. Zuo, Z. Ye, R. Huang, Y. Ren, and Z. Zhao, ``Fluentspeech: Stutter-oriented automatic speech editing with context-aware diffusion models,'' _arXiv preprint arXiv:2305.13612_ , 2023.
  * [21] P. M. Roberts, A. Meltzer, and J. Wilding, ``Disfluencies in non-stuttering adults across sample lengths and topics,'' _Journal of communication disorders_ , vol. 42, no. 6, pp. 414–427, 2009.
  * [22] P. Riva-Posse, L. Busto-Marolt, Á. Schteinschnaider, L. Martinez-Echenique, Á. Cammarota, and M. Merello, ``Phenomenology of abnormal movements in stuttering,'' _Parkinsonism & related disorders_, vol. 14, no. 5, pp. 415–419, 2008.
  * [23] N. B. Ratner and B. MacWhinney, ``Fluency Bank: A new resource for fluency research and practice,'' _Journal of Fluency Disorders_ , vol. 56, pp. 69–80, 2018.
  * [24] J. Li, D. Li, S. Savarese, and S. Hoi, ``Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models,'' 2023.
  * [25] T. Kourkounakis, A. Hajavi, and A. Etemad, ``Detecting multiple speech disfluencies using a deep residual network with bidirectional long short-term memory,'' 2019.
  * [26] R. Agrawal, D. Wolff, and S. Dixon, ``Structure-aware audio-to-score alignment using progressively dilated convolutional neural networks,'' in _International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. IEEE, 2021, pp. 571–575.
  * [27] ——, ``A convolutional-attentional neural framework for structure-aware performance-score synchronization,'' _IEEE Signal Processing Letters_ , vol. 29, pp. 344–348, 2021.
  * [28] Z. Jiang, Q. Yang, J. Zuo, Z. Ye, R. Huang, Y. Ren, and Z. Zhao, ``FluentSpeech: Stutter-oriented automatic speech editing with context-aware diffusion models,'' in _Findings of the Association for Computational Linguistics (ACL)_ , Toronto, Canada, Jul. 2023.
  * [29] M. Abdrakhmanova, A. Kuzdeuov, S. Jarju, Y. Khassanov, M. Lewis, and H. A. Varol, ``Speakingfaces: A large-scale multimodal dataset of voice commands with visual and thermal video streams,'' _Sensors_ , vol. 21, no. 10, p. 3465, 2021.
  * [30] E. J. Hu, Y. Shen, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang, and W. Chen, ``LoRA: Low-rank adaptation of large language models,'' 2021.


# Improved microwave SQUID multiplexer readout using a kinetic-inductance
traveling-wave parametric amplifier

M. Malnou<EMAIL_ADDRESS>National Institute of Standards and
Technology, Boulder, Colorado 80305, USA  Department of Physics, University of
Colorado, Boulder, Colorado 80309, USA  J. A. B. Mates  National Institute of
Standards and Technology, Boulder, Colorado 80305, USA  M. R. Vissers
National Institute of Standards and Technology, Boulder, Colorado 80305, USA
L. R. Vale  National Institute of Standards and Technology, Boulder, Colorado
80305, USA  D. R. Schmidt  National Institute of Standards and Technology,
Boulder, Colorado 80305, USA  D. A. Bennett  National Institute of Standards
and Technology, Boulder, Colorado 80305, USA  J. Gao  National Institute of
Standards and Technology, Boulder, Colorado 80305, USA  Department of Physics,
University of Colorado, Boulder, Colorado 80309, USA  J. N. Ullom  National
Institute of Standards and Technology, Boulder, Colorado 80305, USA
Department of Physics, University of Colorado, Boulder, Colorado 80309, USA

###### Abstract

We report on the use of a kinetic-inductance traveling-wave parametric
amplifier (KITWPA) as the first amplifier in the readout chain of a microwave
superconducting quantum interference device (SQUID) multiplexer (µmux). This
µmux is designed to multiplex signals from arrays of low temperature detectors
such as superconducting transition-edge sensor microcalorimeters. When
modulated with a periodic flux-ramp to linearize the SQUID response, the flux
noise improves, on average, from $1.6$ µ$\Phi_{0}/\sqrt{\mathrm{Hz}}$ with the
KITWPA off, to $0.77$ µ$\Phi_{0}/\sqrt{\mathrm{Hz}}$ with the KITWPA on. When
statically biasing the µmux to the maximally flux-sensitive point, the flux
noise drops from $0.45$ µ$\Phi_{0}/\sqrt{\mathrm{Hz}}$ to $0.2$
µ$\Phi_{0}/\sqrt{\mathrm{Hz}}$. We validate this new readout scheme by
coupling a transition-edge sensor microcalorimeter to the µmux and detecting
background radiation. The combination of µmux and KITWPA provides a variety of
new capabilities including improved detector sensitivity and more efficient
bandwidth utilization.

Over the past few years, the multiplexed readout of transition-edge sensors
(TES) with microwave superconducting quantum interference device (SQUID)
multiplexers (µmux) has become ubiquitous. For example, this multiplexing
technique will be deployed at the Simons Observatory to read out signals from
tens of thousands of TES bolometers, aimed at measuring the cosmic microwave
background [1]. It is also being used to read out TES microcalorimeter arrays
for x-ray and gamma-ray spectroscopy [2, 3, 4, 5, 6, 7, 8].

A µmux divides the available readout bandwidth by coupling many readout
resonators to a single transmission line. Each resonator is terminated by an
rf-SQUID, inductively coupled to a TES, and this coupling is typically made
large enough to ensure that the TES current noise dominates over the noise of
the readout chain. For pulsed TES signals, this large inductive coupling
results in a high flux slew rate at the SQUIDs, requiring a wide resonator
bandwidth to track[9]. In this context, having a lower readout noise would
allow for a smaller coupling, a slower slew rate, narrower resonators, and
therefore would allow us to increase the multiplexing factor.

Other sensors could benefit from a lower readout noise, in particular metallic
magnetic calorimeters (MMCs). These devices place a magnetically susceptible
calorimeter in the field of a superconducting loop that also passes through
the input coil of a SQUID. With magnetic flux trapped in the loop, a variation
in the MMC susceptibility shifts a fraction of this flux into or out of the
SQUID. This shift of magnetic energy cannot be better resolved by increasing
the inductive coupling to the SQUID. Furthermore, existing µmux readout
techniques substantially degrade the performance of MMCs [10]. The use of a
near-quantum-limited microwave amplifier could mitigate this problem and
advance the use of multiplexed MMC arrays.

Traditionally, the first amplifier in the µmux readout chain is a high
electron-mobility transistor (HEMT) amplifier, placed at 4 kelvin [2, 3, 11,
12]. The HEMT offers several key features that make it compatible with µmux
readout: (i) it provides sufficient gain, (ii) it is wideband, and (iii) it
has a high compression power. However, its noise temperature, usually a few
kelvin, is far from the lower bound imposed by quantum mechanics [13].

In this letter, we use a kinetic-inductance traveling-wave parametric
amplifier (KITWPA) as the first amplifier in the readout chain of a µmux. The
KITWPA gain, bandwidth and power handling are also compatible with µmux
readout, and its wideband noise has been shown to be close to the quantum
limit [14, 15]. It is placed before the HEMT, at millikelvin temperatures.
With this readout scheme, we show that the flux noise attached to a coherent
tone probing one of the resonators in the µmux, whose resonance is modulated
at $3$ MHz, is, on average, $0.77$ µ$\Phi_{0}/\sqrt{\mathrm{Hz}}$. When the
KITWPA is turned off, the flux noise obtained with the HEMT as the first
amplifier is more than doubled, reaching $1.6$ µ$\Phi_{0}/\sqrt{\mathrm{Hz}}$,
a typical noise level for traditional µmux readout chains [16, 12]. When the
resonator is biased at its maximum flux-sensitive point, the open-loop flux
noise drops from $0.45$ µ$\Phi_{0}/\sqrt{\mathrm{Hz}}$ (with the KITWPA off)
to $0.2$ µ$\Phi_{0}/\sqrt{\mathrm{Hz}}$ (with the KITWPA on), equivalent to a
system noise temperature of about $1$ K. Such a low open-loop flux noise
suggests that the demodulated flux noise could be decreased even further with
straightforward improvements. Finally, we validate the use of the KITWPA
coupled to the µmux by measuring pulse events in a TES microcalorimeter caused
by background radiation.

Qualitatively, for microwave SQUID multiplexing, a low amplifier noise
translates into a low flux noise. Each resonator within a µmux is terminated
by an rf-SQUID, whose loop is coupled to a flux-bias line used to modulate the
resonator’s resonant frequency (see Fig. 1a). A probe tone, with frequency
centered within the peak-to-peak frequency shift of the resonance, then
describes a semi-circle trajectory in its rotating frame, at the modulation
frequency. The readout noise attached to this tone can then be thought of as
spreading its instantaneous position over a two-dimensional Gaussian in the
rotating frame, along the in-phase (I) and out-of-phase (Q) quadratures (see
Fig. 1b). It determines the noise on the tone’s angle $\theta$ in this frame,
and in turn, it determines the noise on the phase of $\theta$ (see Fig. 1c).
This phase noise, multiplied by $\Phi_{0}/2\pi$, gives the flux noise.

Figure 1: Schematic of a µmux readout circuit, showing how amplifier noise
translates to flux noise. In traditional µmux readout (a) a TES (shown as a
variable resistor) is inductively coupled to a radio frequency (rf) SQUID
embedded in a microwave resonator. (b) For a fixed microwave probe tone, the
transmission moves along the resonance circle as a periodic function of flux
in the SQUID. The noise associated with the amplifier chain (red disk) spreads
the tone’s position along the I and Q quadratures. (c) Under flux-ramp
modulation, the SQUID is constantly sweeping out its approximately sinusoidal
response. The noise attached to the tone thus translates into some noise on
$\theta(t)$, the phase of the flux-ramp response. This noise impacts how well
one can detect a phase shift $\delta_{a}$ on $\theta(t)$, due to a TES signal.

To reduce the flux noise, one can (i) increase the tone’s power, but this
power is eventually limited by the linearity of the SQUID response, and (ii)
reduce the noise of the readout chain, which is what we propose to do, using a
near-quantum-limited amplifier. In this context, for a given tone’s power we
can ask: what is the lowest flux noise achievable? In other words, what is the
flux noise associated with a quantum-limited amplification chain?

Quantitatively now, starting with a system noise temperature
$T_{\mathrm{sys}}$, the (input-referred) noise power spectral density $S_{N}$
along the I and Q quadratures of the tone is $S_{N}=k_{B}T_{\mathrm{sys}}$,
where $k_{B}$ is the Boltzmann constant. Normalizing by the tone’s power
$P_{t}$, it translates into a spectral density on the rotation angle of the
tone, $\theta$, such that $S_{\theta}=4k_{B}T_{\mathrm{sys}}/P_{t}$ [17].
Assuming a sinusoidal variation of $\theta$ with the flux $\Phi$,
$\theta(\Phi)=A\cos(2\pi\Phi/\Phi_{0})$, with $A$ the variation amplitude and
$\Phi_{0}$ the magnetic flux quantum, the maximum slope is then
$\max\\{d\theta/d\Phi\\}=2\pi A/\Phi_{0}$. At this maximum flux-sensitive
point, the noise power spectral density on the flux is
$\tilde{S}_{\Phi}=S_{\theta}\Phi_{0}^{2}/(2\pi A)^{2}$. Therefore, the flux
noise $\sqrt{\tilde{S}_{\Phi}}$ at the maximum flux-sensitive point, sometimes
called the open-loop flux noise, is:

$\sqrt{\tilde{S}_{\Phi}}=\frac{1}{\pi
A}\sqrt{\frac{k_{B}T_{\mathrm{sys}}}{P_{t}}}.$ (1)

For typical µmux operation, $A\simeq 1$; thus, with a system noise temperature
$T_{\mathrm{sys}}=4$ K, representative of a HEMT, and a probe tone power
$P_{t}=-75$ dBm, we obtain $\sqrt{\tilde{S}_{\Phi}}=0.42$
µ$\Phi_{0}/\sqrt{\mathrm{Hz}}$. Note that the dependence of the flux noise on
the system noise temperature enters as a square root (due to the conversion
from a power noise to an amplitude noise), so in practice, significantly
reducing the flux noise is a difficult task. Note also that Eq. 1 gives the
most honest way to quote $T_{\mathrm{sys}}$ knowing $\sqrt{\tilde{S}_{\Phi}}$,
because here the system noise temperature includes all the possible sources of
noise that contribute to the flux noise.

At the standard quantum limit (SQL), the noise power spectral density is equal
to one photon [13], $S_{N}=\hbar\omega$, where $\hbar$ is the reduced Planck
constant and $\omega$ is the angular frequency of the photon. Thus, at the
SQL,
$\sqrt{\tilde{S}_{\Phi}^{\mathrm{SQL}}}=1/(A\pi)\sqrt{\hbar\omega/P_{t}}$. For
$\omega=2\pi\times 4.5$ GHz and with $P_{t}=-75$ dBm (and $A=1)$ it means that
$\sqrt{\tilde{S}_{\Phi}^{\mathrm{SQL}}}=0.1$ µ$\Phi_{0}/\sqrt{\mathrm{Hz}}$.
This is the quantum limit on the flux noise at the maximum flux-sensitive
point, for this photon frequency and probe tone power.

In practice, the SQUIDs within the µmux are always modulated with a fast ramp
of current[18], transforming the detector signal into a phase shift of the
SQUID modulated flux response. This technique evades multiple sources of low-
frequency noise. In this context, the flux noise $\sqrt{S_{\Phi}}$ on the
demodulated tone is obtained from the open-loop flux noise, degraded twice:
(i) when the peak-to-peak frequency shift of the resonator $\delta_{f}$ is
comparable to its bandwidth B and when the modulation of $\theta$ is
sinusoidal, the flux power spectral density integrated over the full $2\pi$
modulation of $\theta$ increases[18] by a factor of $\approx 2$. (ii) Usually,
the beginning of each ramp of current has a transient, which affects the
modulation of $\theta$. The transient is eliminated by discarding the first
$\Phi_{0}$ of the ramp response. Thus, if the ramp is swept over $n\Phi_{0}$,
the noise increases by a factor $1/\alpha=n/(n-1)$. The general expression of
the tone’s demodulated flux noise is thus:

$\sqrt{S_{\Phi}}=\frac{\sqrt{2/\alpha}}{\pi
A}\sqrt{\frac{k_{B}T_{\mathrm{sys}}}{P_{t}}}.$ (2)

For example, with $A=1$, $\alpha=2/3$, $P_{t}=-75$ dBm, and at the SQL where
$k_{B}T_{\mathrm{sys}}=\hbar\omega$, with $\omega=2\pi\times 4.5$ GHz, we
would predict $\sqrt{S_{\Phi}^{\mathrm{SQL}}}=0.17$
µ$\Phi_{0}/\sqrt{\mathrm{Hz}}$.

Figure 2: Readout of a single resonator within the µmux, disconnected from a
TES. The digitized output (at $60$ MS/s) of the demodulated probe tone is
obtained when the KITWPA is (a) off and (b) on; the origin of the I and Q
quadrature frame is translated to the center of the circle supporting the
tone’s positions. The flux ramp sweeps 3$\Phi_{0}$, at a frequency $f_{r}=1$
MHz (for 60 samples per ramp, 20 samples per $\Phi_{0}$). The tone’s power is
$P_{t}\simeq-75$ dBm, and the tone’s frequency is $f_{t}=4.383$ GHz, to be
compared to the resonator’s maximal resonance $f_{0}=4.392$ GHz and peak-to-
peak frequency shift $\delta_{f}=15$ MHz. (c) For each sample within the ramp
(excluding the 20 first samples corresponding to the first $\Phi_{0}$) we
measure a mean value and standard deviation for $\theta$, the angle of
rotation of the tone in the I and Q quadrature frame, when the KITWPA is off
(gray line) and on (blue line). (d) Extracting a value for the phase of
$\theta$ for each ramp segment, we Fourier transform this phase vector (and
divide by $2\pi$) to obtain the flux noise for the two situations: KITWPA off
(gray line) and KITWPA on (blue line).

Microwave loss and excess noise will prevent the flux noise from reaching the
SQL, so to see how much we can improve the flux noise in practice, we perform
the readout of a µmux, using a KITWPA as our first amplifier, placed at
millikelvin temperatures. Figure 2 shows the results of a single resonator
readout within the µmux, for two situations: when the KITWPA is turned off
(the HEMT is then the first amplifier in the chain), and when the KITWPA is
turned on. This resonator is not connected to any TES, because otherwise the
TES noise would overwhelm the readout noise (due to the engineered SQUID
inductive coupling to the TES). We send a probe tone and apply a $1$ MHz flux
ramp to the SQUID, sweeping 3$\Phi_{0}$ per ramp so that the resonance is
modulated at $f_{m}=3$ MHz. Figure 2a (Fig. 2b) shows a histogram of the
digitized output of the tone in its I and Q quadrature frame, obtained using a
homodyne setup [9], when the KITWPA is off (on). The tone frequency and power
are adjusted in this frame: at the optimal tone frequency the transmission
describes a “figure-8” shape, due to the fact that the resonator is constantly
driven out of equilibrium, and above the optimal probe tone power,
$P_{t}\simeq-75$ dBm, the resonator bifurcates. Qualitatively, when the KITWPA
is turned on, the successive positions taken by the probe tone along the flux
ramp are better defined, indicative of a lower phase noise. Quantitatively,
Figure 2c shows $\langle\theta\rangle$ and $\sigma_{\theta}$, respectively the
mean value and standard deviation of $\theta$ over one ramp period (discarding
the transient-contaminated first $\Phi_{0}$) when the KITWPA is off and on.
Clearly, turning on the KITWPA reduces $\sigma_{\theta}$. It translates into a
lower flux noise [9], $\sqrt{S_{\Phi}}$ (Fig. 2d), which drops from $1.6$
µ$\Phi_{0}/\sqrt{\mathrm{Hz}}$ to $0.77$ µ$\Phi_{0}/\sqrt{\mathrm{Hz}}$ on
average.

Here, we compare the flux noise obtained when the KITWPA is off to when it is
on, but the KITWPA-off situation is not equivalent to a standard µmux readout
chain, with the HEMT as the first amplifier, because our chain contains extra
microwave components [9]. These components insert loss, and therefore increase
the flux noise. Nonetheless, even with these components the flux noise with
the KITWPA off remains low compared to standard values [16, 12]. Furthermore,
we used off-the-shelf microwave components to build the readout chain. These
could be made more efficient, for example by integrating them on-chip,
together with the KITWPA.

Figure 3: Open-loop flux noise measurement. (a) The flux modulation curve of
the resonator shows that $\delta_{f}=15$ MHz, while the bandwidth of the
resonator is $B=4.6$ MHz. We set the tone’s frequency $f_{t}$ at
$\max\\{d\theta/d\Phi\\}$ (dashed line), where $f_{t}=4.384$ GHz. (b) We
measure the flux noise at this particular flux bias point [9], when the KITWPA
is off (gray line) and on (blue line). In comparison, the flux noise of an
amplification chain operating at the SQL is indicated by the dashed red line.

How close are we to the SQL? We cannot directly derive $T_{\mathrm{sys}}$ from
Eq. 2, because in our case $\delta_{f}\gg B$, see Fig. 3a. It degrades
$\sqrt{S_{\Phi}}$ compared to the situation where $\delta_{f}=B$, because when
sweeping the ramp the tone then spends more time away from resonance, in the
flux-insensitive region. Instead, to estimate $T_{\mathrm{sys}}$, we measure
the open-loop flux noise at $\max\\{d\theta/d\Phi\\}$ [9], for the two
situations, KITWPA off and on, see Fig. 3b. At 3 MHz, equal to the modulation
frequency $f_{m}$ previously used, $\sqrt{\tilde{S}_{\Phi}}$ drops from $0.45$
µ$\Phi_{0}/\sqrt{\mathrm{Hz}}$ to $0.2$ µ$\Phi_{0}/\sqrt{\mathrm{Hz}}$.
Indeed, the open-loop flux noise is degraded by more than $\sqrt{2/\alpha}$
(with $\alpha=2/3$) to yield the demodulated flux noise previously obtained.
Using Eq. 1, $\sqrt{\tilde{S}_{\Phi}}$ corresponds to a system noise
temperature $T_{\mathrm{sys}}=4.6$ K and $T_{\mathrm{sys}}=0.9$ K,
respectively (taking $A=1$, and $P_{t}=-75$ dBm). In comparison, at the SQL,
$T_{\mathrm{sys}}^{\mathrm{SQL}}=0.2$ K (at 4.5 GHz), therefore with the
KITWPA turned on we operate $4.5$ times above the quantum limit, whereas with
the KITWPA off we operate more than $20$ times above the quantum limit.

Reaching such low values for $\sqrt{\tilde{S}_{\Phi}}$ suggests two ways
$\sqrt{S_{\Phi}}$ could be further reduced: (i) a resonator for which
$\delta_{f}=B$ should, in principle, yield
$\sqrt{S_{\Phi}}=\sqrt{2/\alpha}\sqrt{\tilde{S}_{\Phi}}$, so with
$\alpha=2/3$, $\sqrt{S_{\Phi}}$ could be as low as $0.35$
µ$\Phi_{0}/\sqrt{\mathrm{Hz}}$ (with the KITWPA on). (ii) With a tone tracking
technique, where the tone’s frequency is also modulated to follow the
resonance, having $\delta_{f}\gg B$ becomes beneficial; in principle, this
technique allows for a lower demodulated flux noise than the one obtained with
a fixed-frequency tone [19].

To demonstrate that this amplification chain can truly be used for sensor
readout, we connected another µmux channel to a TES[9]. When the TES detects a
photon or a particle, it generates a pulse of current in the SQUID loop, which
translates into a dephasing event on the probe tone’s trajectory (see Fig. 1).
Absent any radiation, the flux noise is dominated by the TES noise at
frequencies below 10 kHz, see Fig. 4a, and therefore there is no difference
between the two situations, KITWPA off and on. In fact, the coupling to the
TES within this µmux has been engineered to overwhelm higher readout noises
than those obtained when using the KITWPA. With a redesign of the µmux, where
both the coupling and the resonator bandwidth could be reduced with no penalty
on the readout sensitivity, one would truly benefit from the near-quantum-
limited nature of the readout chain.

Figure 4: Response of a resonator within the µmux, connected to a TES. (a)
When the TES is biased between its normal and superconducting branches, the
flux noise $\sqrt{S_{\Phi}}$ increases below 10 kHz, because the TES noise
overwhelms the readout noise. At higher frequencies, we recover the
improvement in flux noise when turning the KITWPA on (blue curve) compared to
when the KITWPA is off (gray curve). Note that the flux noise values are
higher here than in Fig. 2d, probably because of the unoptimized link (that
includes long wire-bonds) between the TES and the µmux [9]. (b) Pulse events
due to background radiation have been detected by the TES over 3.5 hours.
Focusing on times around 0 ms (inset) and on a single large amplitude pulse,
it is evident that the KITWPA readout chain records the pulses without
distortion even where the derivative of the flux signal is largest.

Continuously acquiring the tone’s excursion in the quadrature frame over
several hours, we record the events for which $\Phi/\Phi_{0}$ significantly
deviates from zero. These events correspond to cosmic rays and other
background radiation hitting the TES [9]. In Fig. 4b, we have overlapped the
226 events where $\Phi/\Phi_{0}\leq 2$, detected over 3.5 hours with the
KITWPA on (events with $\Phi/\Phi_{0}>2$ are present but excluded from the
plot because the TES begins to saturate). This experiment proves that the
readout scheme using the KITWPA is suitable for the detection of pulsed events
from the deposition of energy quanta.

In conclusion, we have demonstrated an unprecedented microwave SQUID
multiplexing readout sensitivity, using a near-quantum-limited KITWPA as our
first amplifier. Modulating a µmux resonance at $3$ MHz with a ramp, we showed
that the flux noise of a demodulated tone is, on average,
$\sqrt{S_{\Phi}}=0.77$ µ$\Phi_{0}/\sqrt{\mathrm{Hz}}$, and it could be
significantly lowered with straightforward improvements. In the context of
µmux readout, the true system noise temperature must be calculated from the
knowledge of the flux noise and the probe tone power. Here, our open-loop flux
noise of $0.2$ µ$\Phi_{0}/\sqrt{\mathrm{Hz}}$ is equivalent to a system noise
temperature of 0.9 K, or about 4.5 times above the quantum limit. Continuously
monitoring over several hours the output of one resonator connected to a TES,
we successfully measured the dynamic response of the sensor to background
radiation, validating the use of this new amplification chain. This
improvement of noise temperature should allow a doubling of µmux multiplexing
factor for TES microcalorimeter readout, as well as enable the useful
application of µmux to MMCs.

We thank D. T. Becker and D. S. Swetz for useful discussions. We gratefully
acknowledge support from the NIST Program on Scalable Superconducting
Computing, the National Aeronautics and Space Administration (NASA) under
Grant No. NNH18ZDA001N-APRA, and the Department of Energy (DOE) Accelerator
and Detector Research Program under Grant No. 89243020SSC000058.

## References

  * McCarrick  _et al._ [2021] H. McCarrick, E. Healy, Z. Ahmed, K. Arnold, Z. Atkins, J. E. Austermann, T. Bhandarkar, J. A. Beall, S. M. Bruno, S. K. Choi, J. Connors, N. F. Cothard, K. D. Crowley, S. Dicker, B. Dober, C. J. Duell, S. M. Duff, D. Dutcher, J. C. Frisch, N. Galitzki, M. B. Gralla, J. E. Gudmundsson, S. W. Henderson, G. C. Hilton, S.-P. P. Ho, Z. B. Huber, J. Hubmayr, J. Iuliano, B. R. Johnson, A. M. Kofman, A. Kusaka, J. Lashner, A. T. Lee, Y. Li, M. J. Link, T. J. Lucas, M. Lungu, J. A. B. Mates, J. J. McMahon, M. D. Niemack, J. Orlowski-Scherer, J. Seibert, M. Silva-Feaver, S. M. Simon, S. Staggs, A. Suzuki, T. Terasaki, R. Thornton, J. N. Ullom, E. M. Vavagiakis, L. R. Vale, J. V. Lanen, M. R. Vissers, Y. Wang, E. J. Wollack, Z. Xu, E. Young, C. Yu, K. Zheng, and N. Zhu, “The simons observatory microwave squid multiplexing detector module design,” The Astrophysical Journal 922, 38 (2021).
  * Noroozian  _et al._ [2013] O. Noroozian, J. A. B. Mates, D. A. Bennett, J. A. Brevik, J. W. Fowler, J. Gao, G. C. Hilton, R. D. Horansky, K. D. Irwin, Z. Kang, D. R. Schmidt, L. R. Vale, and J. N. Ullom, “High-resolution gamma-ray spectroscopy with a microwave-multiplexed transition-edge sensor array,” Applied Physics Letters 103, 202602 (2013).
  * Mates  _et al._ [2017] J. A. B. Mates, D. T. Becker, D. A. Bennett, B. J. Dober, J. D. Gard, J. P. Hays-Wehle, J. W. Fowler, G. C. Hilton, C. D. Reintsema, D. R. Schmidt, D. S. Swetz, L. R. Vale, and J. N. Ullom, “Simultaneous readout of 128 x-ray and gamma-ray transition-edge microcalorimeters using microwave squid multiplexing,” Applied Physics Letters 111, 062601 (2017).
  * Yoon  _et al._ [2018] W. Yoon, J. S. Adams, S. R. Bandler, D. Becker, D. A. Bennett, J. A. Chervenak, A. M. Datesman, M. E. Eckart, F. M. Finkbeiner, J. W. Fowler, J. D. Gard, G. C. Hilton, R. L. Kelley, C. A. Kilbourne, J. A. B. Mates, A. R. Miniussi, S. H. Moseley, O. Noroozian, F. S. Porter, C. D. Reintsema, J. E. Sadleir, K. Sakai, S. J. Smith, T. R. Stevenson, D. S. Swetz, J. N. Ullom, L. R. Vale, N. A. Wakeham, E. J. Wassell, and E. J. Wollack, “Toward large field-of-view high-resolution x-ray imaging spectrometers: Microwave multiplexed readout of 28 tes microcalorimeters,” Journal of Low Temperature Physics 193, 258–266 (2018).
  * Nakashima  _et al._ [2020] Y. Nakashima, F. Hirayama, S. Kohjiro, H. Yamamori, S. Nagasawa, A. Sato, S. Yamada, R. Hayakawa, N. Y. Yamasaki, K. Mitsuda, K. Nagayoshi, H. Akamatsu, L. Gottardi, E. Taralli, M. P. Bruijn, M. L. Ridder, J. R. Gao, and J. W. A. den Herder, “Low-noise microwave squid multiplexed readout of 38 x-ray transition-edge sensor microcalorimeters,” Applied Physics Letters 117, 122601 (2020).
  * Carpenter  _et al._ [2021] M. H. Carpenter, B. Stein, K. E. Koehler, C. J. Fontes, C. M. Smith, G. L. Wagner, Z. K. Baker, M. L. Handley, M. W. Rabin, P. Yang, E. R. Batista, D. G. McNeel, K. A. Schreiber, E. G. Bowes, J. N. Ullom, G. C. O’Neil, C. D. Reintsema, D. A. Bennett, G. C. Hilton, D. S. Swetz, D. R. Schmidt, J. A. B. Mates, D. T. Becker, J. C. Weber, J. D. Gard, K. M. Morgan, J. Imrek, D. Yan, A. L. Wessels, and M. Croce, “Hyperspectral x-ray imaging: Progress towards chemical analysis in the sem,” IEEE Transactions on Applied Superconductivity 31, 1–6 (2021).
  * Szypryt  _et al._ [2021] P. Szypryt, D. A. Bennett, W. J. Boone, A. L. Dagel, G. Dalton, W. B. Doriese, M. Durkin, J. W. Fowler, E. J. Garboczi, J. D. Gard, G. C. Hilton, J. Imrek, E. S. Jimenez, V. Y. Kotsubo, K. Larson, Z. H. Levine, J. A. B. Mates, D. McArthur, K. M. Morgan, N. Nakamura, G. C. O’Neil, N. J. Ortiz, C. G. Pappas, C. D. Reintsema, D. R. Schmidt, D. S. Swetz, K. R. Thompson, J. N. Ullom, C. Walker, J. C. Weber, A. L. Wessels, and J. W. Wheeler, “Design of a 3000-pixel transition-edge sensor x-ray spectrometer for microcircuit tomography,” IEEE Transactions on Applied Superconductivity 31, 1–5 (2021).
  * Szypryt  _et al._ [2022] P. Szypryt, N. Nakamura, D. T. Becker, D. A. Bennett, A. L. Dagel, W. B. Doriese, J. W. Fowler, J. D. Gard, J. Z. Harris, G. C. Hilton, _et al._ , “A tabletop x-ray tomography instrument for nanometer-scale imaging: demonstration of the 1,000-element transition-edge sensor subarray,” arXiv preprint arXiv:2212.12073 (2022).
  * [9] See supplementary information.
  * Wegner  _et al._ [2018] M. Wegner, N. Karcher, O. Krömer, D. Richter, F. Ahrens, O. Sander, S. Kempf, M. Weber, and C. Enss, “Microwave squid multiplexing of metallic magnetic calorimeters: Status of multiplexer performance and room-temperature readout electronics development,” Journal of Low Temperature Physics 193, 462–475 (2018).
  * Dober  _et al._ [2017] B. Dober, D. T. Becker, D. A. Bennett, S. A. Bryan, S. M. Duff, J. D. Gard, J. P. Hays-Wehle, G. C. Hilton, J. Hubmayr, J. A. B. Mates, C. D. Reintsema, L. R. Vale, and J. N. Ullom, “Microwave squid multiplexer demonstration for cosmic microwave background imagers,” Applied Physics Letters 111, 243510 (2017).
  * Dober  _et al._ [2021] B. Dober, Z. Ahmed, K. Arnold, D. T. Becker, D. A. Bennett, J. A. Connors, A. Cukierman, J. M. D’Ewart, S. M. Duff, J. E. Dusatko, J. C. Frisch, J. D. Gard, S. W. Henderson, R. Herbst, G. C. Hilton, J. Hubmayr, Y. Li, J. A. B. Mates, H. McCarrick, C. D. Reintsema, M. Silva-Feaver, L. Ruckman, J. N. Ullom, L. R. Vale, D. D. Van Winkle, J. Vasquez, Y. Wang, E. Young, C. Yu, and K. Zheng, “A microwave squid multiplexer optimized for bolometric applications,” Applied Physics Letters 118, 062601 (2021).
  * Caves [1982] C. M. Caves, “Quantum limits on noise in linear amplifiers,” Phys. Rev. D 26, 1817–1839 (1982).
  * Malnou  _et al._ [2021] M. Malnou, M. Vissers, J. Wheeler, J. Aumentado, J. Hubmayr, J. Ullom, and J. Gao, “Three-wave mixing kinetic inductance traveling-wave amplifier with near-quantum-limited noise performance,” PRX Quantum 2, 010302 (2021).
  * Malnou  _et al._ [2022] M. Malnou, J. Aumentado, M. Vissers, J. Wheeler, J. Hubmayr, J. Ullom, and J. Gao, “Performance of a kinetic inductance traveling-wave parametric amplifier at 4 kelvin: Toward an alternative to semiconductor amplifiers,” Phys. Rev. Applied 17, 044009 (2022).
  * Bennett  _et al._ [2015] D. A. Bennett, J. A. B. Mates, J. D. Gard, A. S. Hoover, M. W. Rabin, C. D. Reintsema, D. R. Schmidt, L. R. Vale, and J. N. Ullom, “Integration of tes microcalorimeters with microwave squid multiplexed readout,” IEEE Transactions on Applied Superconductivity 25, 1–5 (2015).
  * [17] In fact, $S_{N}=R^{2}S_{\theta}$, and when normalizing by the tone’s power, $R=1/2$ is the radius of the semi-circle in Fig. 1b.
  * Mates  _et al._ [2012] J. A. B. Mates, K. D. Irwin, L. R. Vale, G. C. Hilton, J. Gao, and K. W. Lehnert, “Flux-ramp modulation for squid multiplexing,” Journal of Low Temperature Physics 167, 707–712 (2012).
  * Yu  _et al._ [2023] C. Yu, Z. Ahmed, J. C. Frisch, S. W. Henderson, M. Silva-Feaver, K. Arnold, D. Brown, J. Connors, A. J. Cukierman, J. M. D’Ewart, B. J. Dober, J. E. Dusatko, G. Haller, R. Herbst, G. C. Hilton, J. Hubmayr, K. D. Irwin, C.-L. Kuo, J. A. B. Mates, L. Ruckman, J. Ullom, L. Vale, D. D. Van Winkle, J. Vasquez, and E. Young, “Slac microresonator rf (smurf) electronics: A tone-tracking readout system for superconducting microwave resonator arrays,” Review of Scientific Instruments 94, 014712 (2023).

Supplementary information: Improved microwave SQUID multiplexer readout using
a kinetic-inductance traveling-wave parametric amplifier

##  I Bandwidth utilization with flux ramp modulation

In flux-ramp modulated SQUID readout, the flux ramp rate sets the effective
sampling rate of the SQUID[1], which must be high enough that the maximum flux
excursion between samples is less than $\Phi_{0}/2$. The full width at half
maximum $B$ of each resonator is then sized to contain the flux ramp signal,
i.e. we must have

$B>\frac{4n}{\Phi_{0}}M_{\mathrm{in}}\left|\frac{dI_{\mathrm{TES}}}{dt}\right|_{\mathrm{max}},$
(S1)

where $n$ is the number of SQUID oscillations per ramp, $M_{\mathrm{in}}$ is
the input mutual inductance between the detector and the SQUID and
$I_{\mathrm{TES}}$ is the detector current. Reducing the flux noise allows us
to reduce $M_{\mathrm{in}}$ and therefore to reduce $B$.

##  II Experimental setup

Figure S1 presents a schematic of the experimental setup. At millikelvin
temperatures, the input of the chain (left side) consists of a directional
coupler (DC) from which we can send a vector network analyzer (VNA) tone,
along with the µmux interrogating tone. A dc-block follows, preventing the dc
current used to bias the KITWPA from leaking back toward the directional
coupler and dissipating in the $50$ Ω load attached to it. The µmux is then
connected, followed by a low-pass filter (LPF) which provides $40$ dB
rejection at the KITWPA pump frequency, preventing the strong KITWPA pump tone
($P_{t}\simeq-30$ dBm) from perturbing the µmux resonators. In fact, even
though this tone is detuned from the µmux resonances ($f_{p}=8.941$ GHz), we
have seen that it can affect the µmux if not properly filtered. Next, the
circulator provides additional isolation, and we use its third port to send
the KITWPA dc current bias, while the following DC delivers the KITWPA pump.
After the KITWPA, a bias tee separates the dc bias current from the rf signal.
The rf signal is then directed onto an isolator and a LPF, in order to filter
out the KITWPA pump tone, which, reflected off the LPF dissipates into the
isolator. The rf signal is then routed into a HEMT at 4 K, and further
amplification stages at room temperature.

All the connecting cables going in and out of the fridge are microwave coaxial
cables, even those carrying a dc signal. In fact, we have found that they
provide a better isolation against radio frequency noise than twisted pair
cables. For the flux ramp line, the 20 dB attenuator at 4 K is equivalent to a
500 Ohm series resistor, which we use to convert the voltage excitation into a
current excitation. The TES bias source is grounded to the fridge ground, but
it is not connected to the electronic rack’s ground. Again, the cables
delivering the TES bias are microwave coaxial cables. Finally, the KITWPA dc
bias source is fully floating. Care was taken not to connect the ground of the
room temperature electronics to the fridge’s ground through all these cables
(which would create ground loops). That is why most of the microwave cables
going into the fridge contain a dc block before entering the fridge. Those
that don’t are either fully floating, or connected to floating electronics
equipment.

At room temperature, two microwave generators are used: one for the KITWPA
pump and one for the µmux probe tone. This second generator also serves as the
local oscillator (LO) for the in-phase/quadrature (IQ) demodulator. The IQ
demodulator demands a fixed input power for its LO, therefore we placed a
variable attenuator on the probe tone’s path. The VNA allows us to measure the
KITWPA gain, and an arbitrary waveform generator (AWG) is used to generate the
flux ramp. Its second channel serves as a clock for the analog to digital
converter (ADC) so that the ADC can digitize at a specific rate (chosen to
have an integer number of samples per flux ramp). The output trigger of the
AWG is connected to the input trigger of the ADC, so that the digitizing
starts at the beginning of a ramp. At the output, the amplification chain is
routed to the rf-port of the IQ demodulator, and the I and Q channels are then
routed (after further base-band amplification) to channels A and B of the ADC.
Finally, all the microwave generators, VNA and AWG are synchronized with a 10
MHz clock.

Figure S1a shows a close-up picture of the µmux in its packaging, with some of
its resonators connected to a few TESs, via series inductances and shunt
resistances. Obviously, this packaging is not optimized, involving long wire-
bonds in between components. The input of the microwave coplanar waveguide
(CPW) readout line can be seen on the left. Figure S1b presents the full µmux
and TES packaging. The flux ramp and TES biases are fed through the in-line
connectors at the bottom (and through microwave coaxial cables beforehand).

Figure S1: Schematic of the experimental setup. We have represented only 2 of
the 16 microwave resonators that the µmux electronic chip truly contains. In
the schematic, one of the resonators is connected to a TES, while one is left
unconnected. The KITWPA electronic chip is represented by the spiral inside
the square. See the main text for the detailed description. Figure S2:
Photograph of the µmux and TES packaging. (a) Some resonators within the µmux
are connected to a few TES via series inductances, and shunt resistances. (b)
A $1\times 2"$ box contains both the µmux and the TES.

##  III Flux noise measurement, background radiation measurement and
processing

Here we detail how, from the digitized I and Q outputs, we obtain the flux
noise presented in Fig. 2. We send a flux ramp with frequency $f_{r}=1$ MHz,
that sweeps $3\Phi_{0}$ per ramp (therefore the modulation frequency of the
resonator’s resonance is $f_{m}=3$ MHz). We then digitize $0.1$ s of data from
the I and Q channels, at $60$ MS/s, which means that we obtain two arrays of
$6\times 10^{6}$ voltages. It also means that we have 60 samples per ramp, or
20 samples per $\Phi_{0}$. We perform this acquisition twice: when the KITWPA
is off (pump tone off), and when it is on.

The processing of this data consists of several steps. First, we translate all
the coordinates in the IQ rotating frame, such that the center of the frame
corresponds to the center of the semi-circle trajectory described by the
points. In practice, we fit all the points to a circle, and find its center.
When centered, a histogram of the points positions is given by Fig. 2a (Fig.
2b), when the KITWPA is off (on).

Second, we unwrap the angle $\theta$ of these points so that $\theta$
continuously varies between $-\pi$ and $\pi$ as a function of time. We then
extract the phase $\alpha(t)$ from $\theta(t)$, excluding the first 20 values
of $\theta(t)$ that correspond to the first $\Phi_{0}$, using [1]:

$\alpha=\arctan\left(\frac{\sum\theta(t)\sin(\omega_{m}t)}{\sum\theta(t)\cos(\omega_{m}t)}\right),$
(S2)

where $\omega_{m}=2\pi f_{m}$. The vector of phases $\alpha(t)$ runs from
$t=0$ s to $t=0.3$ s, with a step size of $\delta_{t}=1/f_{r}$.

We can now calculate the power spectral density (psd) of $\alpha(t)$ to get
$S_{\alpha}$, in units of $\mathrm{rad}^{2}/\mathrm{Hz}$. The maximum
frequency of this psd is half the ramp frequency, and we chose a bin size of
100 Hz. Finally, the flux noise is given by
$\sqrt{S_{\Phi}}/\Phi_{0}=\sqrt{S_{\alpha}}/(2\pi)$.

When running the experiment continuously to detect background radiation, we
set the fridge temperature to $90$ mK (stable within $\pm 15$ µK). We then
continuously perform the acquisition and processing, and only keep the data
from the I and Q channels if a significant peak in $\alpha(t)$ is detected.
The procedure to look for such a peak consists of calculating the moving
standard deviation $\sigma_{\alpha}$ of $\alpha(t)$ (with a 5 ms window), and
look for peaks in $\sigma_{\alpha}$ that are greater than $0.03$ radians (a
value that gives good compromise between detecting too many false positives
and discarding too many low energy background radiation events).

##  IV Open-loop flux noise measurement and processing

The open-loop flux noise is acquired with the same experimental setup.
However, instead of using the AWG to generate a voltage ramp, we use it as dc
voltage source (which translates into a dc current, due to the 20 dB
attenuator at 4 K). We then digitize the I and Q outputs for $221$ static flux
biases along one full $\Phi_{0}$ (equivalent to $1.2$ mA; we use coarse
current steps of $25$ $\upmu$A in the flux-insensitive region, close to the
maximum resonator frequency, and fine current steps of $5$ $\upmu$A in the
flux-sensitive region). For each flux biasing point, and for the two
configurations KITWPA off/on, we record data over $0.01$ s, with a sampling
rate of $20$ MS/s, therefore we have two arrays of $2\times 10^{5}$ voltages
per bias point, per KITWPA configuration.

The processing steps are as follows. First, we translate the coordinates in
the IQ rotating frame, as in the case of the demodulated flux noise. For each
flux bias, we find the average coordinate (two voltages) in the IQ frame, and
then fit all the averages to a circle to find its center.

Second, we calculate the voltage psd $S_{V}(\Phi)$ (in $V^{2}/\mathrm{Hz}$)
for each flux bias, along the direction tangential to the circle (the unit
vector in the tangential direction is the normalized gradient of the averaged
points coordinates).

Finally, we calculate the open-loop flux noise, as:

$\sqrt{\tilde{S}_{\Phi}}=\frac{\sqrt{S_{V}(\Phi_{M})}}{\lvert\mathrm{max}\\{dV/d\Phi\\}\rvert},$
(S3)

where $\Phi_{M}$ is the flux bias for which $\lvert
dV/d\Phi\rvert=\lvert\mathrm{max}\\{dV/d\Phi\\}\rvert$.

##  V KITWPA gain profile

Figure S3a shows the wideband KITWPA gain profile, obtained with the VNA (pump
on/off ratio), at which the KITWPA was operated during the various
experiments. We chose to run the KITWPA at a relatively modest gain, to ensure
its performance as a function of time be very stable.

Figure S3b shows the KITWPA gain, when zoomed in around the tuning range of
the resonator that we used in the demodulated flux noise measurement. The
feature at $4.392$ GHz comes from the presence of the resonance (the VNA tone
passes through the µmux, see fig. S1). Over the whole resonator’s tuning
range, the KITWPA gain is on average $11.6$ dB, with $2.5$ dB of peak-to-peak
gain ripples. Interestingly, because of the nature of the flux-ramp modulation
scheme, the µmux readout is somewhat insensitive to gain ripples: it may
distort $\theta(t)$ into a non perfect sinusoidal signal, but we only care
about the phase of that signal. So as long as the gain ripples are stable as
function of time, they don’t affect the periodicity of $\theta(t)$, therefore
they don’t affect the determination of $\alpha(t)$.

Figure S3: Gain of the KITWPA (ratio pump on/off). (a) The wideband gain
profile as a function of frequency. (b) Zoom in on the gain around the tuning
range of the resonator used in the flux noise measurements.

## References

  * [1] Mates, J. A. B., Irwin, K. D., Vale, L. R., Hilton, G. C., Gao, J. and Lehnert, K. W., ”Flux-Ramp Modulation for SQUID Multiplexing”, Journal of Low Temperature Physics 167, 707-712 (2012)