:

# Multimodal Sleep Apnea Detection with Missing or Noisy Modalities

Hamed Fayyaz<EMAIL_ADDRESS>
University of Delaware  Abigail Strang<EMAIL_ADDRESS>
Nemours Children’s Health  Niharika S. D’Souza<EMAIL_ADDRESS>
IBM Research Almaden  Rahmatollah Beheshti<EMAIL_ADDRESS>
University of Delaware

###### Abstract

Polysomnography (PSG) is a type of sleep study that records multimodal
physiological signals and is widely used for purposes such as sleep staging
and respiratory event detection. Conventional machine learning methods assume
that each sleep study is associated with a fixed set of observed modalities
and that all modalities are available for each sample. However, noisy and
missing modalities are a common issue in real-world clinical settings. In this
study, we propose a comprehensive pipeline aiming to compensate for the
missing or noisy modalities when performing sleep apnea detection. Unlike
other existing studies, our proposed model works with any combination of
available modalities. Our experiments show that the proposed model outperforms
other state-of-the-art approaches in sleep apnea detection using various
subsets of available data and different levels of noise, and maintains its
high performance (AUROC$>$0.9) even in the presence of high levels of noise or
missingness. This is especially relevant in settings where the level of noise
and missingness is high (such as pediatric or outside-of-clinic scenarios).

#### Data and Code Availability

Our code is publicly available at https://github.com/healthylaife/apnea-
missing-modality. The datasets used in this study are available at
https://sleepdata.org.

#### Institutional Review Board (IRB)

This research did not require IRB approval, as it only needed secondary use of
deidentified data, per the data owner’s (NIH/NHLBI) policy.

##  1 Introduction

Sleep is essential in maintaining and promoting overall health and well-being
(Luyster et al., 2012). Insufficient or poor-quality sleep has been linked to
a wide range of health problems, including cardiovascular disease (Kasasbeh et
al., 2006), obesity (Taheri, 2006), diabetes (Knutson et al., 2006), and
mental health disorders (Zimmerman et al., 2006; Schwartz et al., 2005).

Conditions that affect sleep quality, timing, or duration and impact a
person’s ability to function properly while awake are referred to as sleep
disorders. There are various types of sleep disorders, including insomnia,
circadian rhythm sleep disorders, sleep-disordered breathing, hypersomnia,
parasomnias, and restless legs syndrome (Pavlova and Latreille, 2019).

In particular, sleep apnea and hypopnea syndrome (SAHS) are breathing
disorders that are characterized by recurring pauses in breathing during
sleep, which usually cause fragmentation of sleep and can lead to oxygen
deprivation and disrupt normal physiological processes (Vaquerizo-Villar et
al., 2020). The main types of SAHS are obstructive sleep apnea (the most
common type), central sleep apnea, complex (mixed) sleep apnea, and hypopnea.
It is estimated that 26% of people between 30 and 70 years (Schwartz et al.,
2018) and 1% to 5% of children suffer from SAHS, with the highest prevalence
in age 2 to 8 (Kheirandish-Gozal and Gozal, 2012; Bixler et al., 2009; Marcus
et al., 2012).

Polysomnography (PSG) is the gold standard for diagnosing sleep-related
breathing disorders. It refers to the process used to collect biological
signals and parameters during sleep, which is generally performed in clinical
lab settings and during the night (Rundo and Downey III, 2019). A PSG
generally includes: 1) brain electrical activity (electroencephalogram or
EEG), 2) eye movements during sleep (electrooculogram or EOG), 3) cardiac rate
and rhythm (electrocardiogram or ECG), 4) blood oxygen saturation (pulse
oximetry or SpO2), 5) measurement of exhaled air to indirectly measure blood
CO2 (end-tidal carbon dioxide or ETCO2), 6) respiratory effort in thorax and
abdomen (respiratory inductance plethysmography or RIP), 7) and nasal and oral
airflow. PSG is generally considered effective; however, it presents many
challenges, including complexity, cost, intrusiveness, and the need for
intensive involvement of clinical providers (Spielmanns et al., 2019). The
multimodal nature of PSG data provides a diverse and holistic view of the
subjects (Muhammad et al., 2021; Kline et al., 2022; Liu et al., 2023).

While a fairly large family of studies aiming to detect SAHS from PSG is
present, existing methods have two major limitations. First, conventional
methods are built based on the assumption that all modalities are available
for all subjects. However, in real-world scenarios, certain modalities may be
partially or entirely missing due to technical issues or limitations during
data acquisition. Additionally, the PSG signals can be corrupted by noise from
various sources, including electrode artifacts, electrical interference, or
patient movement. Detecting apnea becomes significantly more challenging when
dealing with missing or noisy modalities (especially the primary signals such
as SpO2). The second limitation is the applicability of existing methods to a
predetermined subset of PSG signals. This issue makes it unclear to what
degree a method developed for a certain set of modalities would work (if at
all) on applications with different sets of available modalities. To address
these limitations, we propose a flexible machine learning-based pipeline for
apnea detection with any combination of available signal modalities.
Specifically, the contributions of our study are:

  * •

We extensively investigate the effects of missing or partially available
modalities in apnea detection.

  * •

We present a novel universal pipeline to predict apnea with any type, length,
or quality of available modality.

  * •

Through extensive experiments, we show that our method is robust to noisy and
missing modalities and outperforms prior methods in various apnea detection
scenarios.

##  2 Related Work

Due to the high relevance of detecting apnea events using machine learning
methods, many attempts have been made to automate apnea detection using such
methods. A large body of prior work focuses on uni-modal methods. Considering
the modalities utilized in prior studies, we highlight four categories in the
following (a non-exhaustive list). (1) Most existing work uses ECG for apnea
detection. while ECG is not the most relevant signal in clinical settings, the
popularity of the methods for ECG is partly due to the availability of public
ECG datasets (Penzel et al., 2000). These methods generally use ‘band-pass’
filters to reduce the noise sourced from the baseline wander, muscle
artifacts, power line interference, and other sources (Urtnasan et al., 2018;
Bahrami and Forouzanfar, 2022). Since ECG signals contain complex patterns,
many studies have used extensive preprocessing steps, including automatic
feature extraction approaches through deep neural networks, in their pipeline
for extracting features (Shen et al., 2021; Chang et al., 2020; Chen et al.,
2022; Zarei et al., 2022). These methods are extensively reviewed by Salari et
al. (2022). (2) Many studies have used SpO2, which is the most clinically
relevant signal for apnea detection (in adults). Similar to the previous
category, various types of manual or automatic feature extraction methods have
been used in this category (Álvarez et al., 2012; Morillo and Gross, 2013;
Uçar et al., 2017; John et al., 2021). (3) The combination of these two
signals (ECG and SpO2) have been used, especially to handle the signals’
imperfection and defects, such as missing data or noise (Tuncer et al., 2019;
Ravelo-García et al., 2015; Pathinarupothi et al., 2017; Xie and Minn, 2012).
(4) Finally, the fourth major category is related to a group of work that uses
EEG signals to detect apneic events (Vimala et al., 2019; Zhao et al., 2021;
Almuhammadi et al., 2015). As discussed earlier, a common limitation of
existing studies is unclear generalizability to other PSG signals. Oftentimes,
these models cannot utilize an additional modality not seen in the training
phase and handle partially available or noisy modalities.

Multimodal learning approaches aim to use information from different sources
to better understand an underlying phenomenon and improve the performance of
downstream tasks. Data fusion is an integral part of multimodal learning,
which is the process of integrating multiple data sources to build a
representation that is more pertinent than any individual source (Stahlschmidt
et al., 2022). Depending on the fusion stage, existing techniques can be
categorized into i) early, ii) intermediate, and iii) late fusion, where the
data from modalities are respectively combined at i) the input (Lim et al.,
2019; Neves et al., 2023), ii) through a middle representation (like a hidden
layer) between the individual modalities and the final model (Hassan et al.,
2022), and iii) using separate models on each modality which are combined
later (Wang et al., 2019; Zhang et al., 2019). When some modalities are
missing or noisy, data fusion enables combining available information from
different sources to create a more comprehensive and reliable representation,
enhancing the robustness of the models (Gaw et al., 2022).

Multimodal information can be especially effective for apnea detection, as
exploiting multimodal data not only can outperform using only uni-modal data
but can also improve the robustness of the system when one or more modalities
are missing. Despite the importance of multimodal learning, there is limited
work (Ye et al., 2023; Fayyaz et al., 2023; Van Steenkiste et al., 2020) on
multimodal data fusion for apnea detection. We are not aware of any previous
study that directly aims at handling missing and noisy data in the context of
sleep apnea detection.

When it comes to working with multi-modal data, the choice of a data fusion
mechanism presents an additional dimension of the problem. In machine learning
pipelines, fusing multimodal data in any of the discussed stages has
advantages and disadvantages. Among the existing fusion techniques, gated
fusion (Arevalo et al., 2017) is a popular method, with applications in
different domains such as computer vision (Hosseinpour et al., 2022; Zhang et
al., 2018; Ren et al., 2018; Feng et al., 2020), speech recognition (Fan et
al., 2020), and natural language processing (Du et al., 2022). By employing
gating mechanisms, gated fusion models selectively weigh and fuse information
and capture complementary aspects. In our study, we propose a modified version
of the gated fusion mechanism that considers the abnormality of the present
modalities (i.e., the extent to which current signal patterns deviate from the
others) to identify the optimized way of “fusing” the present modalities.

##  3 Problem setup

Consider a sleep dataset consisting of ${I}$ sleep studies (e.g., ${I}$
individual nights in the clinic) shown by $\\{S_{i}\\}_{i=1}^{I}$. Each study
can be divided into equal-length non-overlapping epochs (windows), considered
as a sample $X$, and shown by

$S_{i}=\\{X^{i,j}\\}_{j=1}^{J_{i}},$ (1)

where, $J_{i}$ is the number of epochs in $i$-th study.

Each sample can consist of different PSG signals, considered as distinct
modalities:

$X^{i,j}=(X^{i,j}_{1},X^{i,j}_{2},...,X^{i,j}_{M}),$ (2)

where $M$, is the total number of modalities.

Furthermore, each sample has a label $y$, showing if an apnea event occurs
during the sample time window. Thus, $X^{i,j}_{m}$ is a signal from the
modality $m$ that belongs to the epoch $j$ of the study $i$, and is a
multivariate time series consisting of $T$ time points:

$X^{i,j}_{m}=(X^{i,j}_{1,m},X^{i,j}_{2,m},...,X^{i,j}_{T,m}).$ (3)

In this work, we design and train a model $f$ (parameterized with $\theta$)
that can handle missing and noisy modalities and estimates the occurrence of
apnea $\hat{y}$ in a given epoch:

$\hat{y}^{i,j}=f_{\theta}(X^{i,j})$ (4)

##  4 Method

Figure 1: Our proposed pipeline. It consists of four components: (1) Encoder,
(2) Decoder, (3) Classifier, and (4) Anomaly-aware Fusion. In the first
(unimodal) step of training, an encoder for each modality is used to transform
the input into a latent space $Z_{m}$. Besides, a unimodal classifier and
decoder are utilized to detect apnea and reconstruct the input, respectively.
In the second (multimodal) step of training, the Anomaly-Aware Fusion module
classifies the epochs using the unimodal latent representation ($Z$) and the
reconstruction error for each input signal. Notations C and X denote the
concatenation and multiplication operations respectively.

For detecting apnea with missing or noisy modalities, we propose a two-step
semi-supervised learning method in companion with a multimodal autoencoder
network using a transformer backbone (Figure 1). The two steps of our method
include unimodal pre-training and multimodal training. The rationale for
introducing a two-step training process in our design is that the pipeline can
use samples that do not have complete modalities to train the model when
dealing with incomplete modalities (akin to an imputing mechanism paradigm).

#### Unimodal Pre-training

We use unimodal transformer-based autoencoders that reconstruct a unimodal
input $X_{m}$, by mapping the input space to a lower dimensional latent space
$Z_{m}$. More specifically, an encoder $E_{m}$ maps the signal to a latent
representation ($Z_{m}=E_{m}(X_{m})$) and a decoder $D_{m}$ reconstructs the
original signal from the latent representation ($\hat{X}_{m}=D_{m}(Z_{m})$).

The encoder and decoder consist of a series of transformer blocks. Besides
being used for signal reconstruction, the decoder is also used for anomaly
detection by using the reconstruction error as a proxy. Our method also
includes a unimodal classifier $C_{m}$ to detect the epochs with apneic events
using $Z_{m}$ ($\hat{y}_{m}=C_{m}(Z_{m})$).

This way, the encoder learns a mapping to a latent space that preserves the
fundamental characteristics of the signals, while extracting the key features
for apnea detection. Additional details about the design of the transformers
are presented in Appendix A.

For modality $m$, the reconstruction loss $\mathcal{L}_{R}$ can be formulated
as the mean squared error (MSE) of the original signal and its reconstructed
version:

$\mathcal{L}^{m}_{R}=\frac{\sum_{i=1}^{I}\sum_{j=1}^{J_{i}}\sum_{t=1}^{T}(X^{i,j}_{t,m}-\hat{X}^{i,j}_{t,m})^{2}}{N\times
T},$ (5)

where,

$N=\sum_{i=1}^{I}J_{i},\text{ and}$ (6)

$\hat{X}^{i,j}_{t,m}=D_{m}(E_{m}(X^{i,j}_{t,m})).$ (7)

In addition, the classification loss $\mathcal{L}_{C}$ (related to the apnea
detection task) is defined using binary cross entropy (BCE) for modality $m$
as:

$\mathcal{L}^{m}_{C}=\frac{1}{N}\sum_{i=1}^{I}\sum_{j=1}^{J_{i}}BCE(y_{m}^{i,j},\hat{y}_{m}^{i,j}),$
(8)

where,

$\hat{y}^{i,j}=C_{m}(E_{m}(X^{i,j}_{m})).$ (9)

The model (in this first step) is trained by jointly optimizing the two
aforementioned loss functions over all modalities:

$\mathcal{L}_{Unimodal}=\sum_{m=1}^{M}\alpha_{m}\mathcal{L}^{m}_{R}+\beta_{m}\mathcal{L}^{m}_{C}$
(10)

where, the hyperparameters ($\alpha_{m}$ and $\beta_{m}$) are used to tune the
contribution of each loss.

#### Multimodal Training

In the second step of our method, we use a modified version of gated fusion
(Arevalo et al., 2017) to detect apneic events. We refer to this module as
Anomaly Aware Fusion (AAF). It receives the latent representation $Z$ and a
distance measure $A$ for each signal. The distance measure $A$ aims to capture
the abnormality of the signals (the degree to which the signal deviates from
the original), similar to Lee and Kang (2022). Here, $A$ is defined as:

$A=[a_{1},a_{2},...,a_{M}],$ (11)

where,

$a_{m}=[|X_{1,m}-\hat{X}_{1,m}|,...,|X_{T,m}-\hat{X}_{T,m}|],$ (12)

while the anomaly score $a_{m}$ captures the absolute distance between the two
time-series $X_{m}$ and $\hat{X}_{m}$. While closely related, the
reconstruction loss in Eq. 12 yields a single value; however, the distance
measure in Eq. 5 is a vector that shows the degree of abnormality in each time
step.

Specifically, AAF can be shown as a function receiving $Z$ and $A$ to predict
the final label:

$\hat{y}=AAF(Z,A).$ (13)

As the encoder-decoder model learns the distribution of (mostly normal)
signals, the distance measure (captured by the absolute differences) increases
when this model receives an abnormal signal, as the signals containing the
abnormal patterns are not reconstructed well.

Furthermore, $Z$ in AAF (Eq. 13) refers to:

$Z=[Z_{1},Z_{2},...,Z_{M}],$ (14)

where, $Z_{m}$ corresponds to a feature vector in the latent representation
space associated with the modality $m$.

The AAF module (the right box in Figure 1) takes these steps. First, each
feature vector $Z_{m}$ is fed to a fully connected layer with a $Tanh$
activation function, which is intended to encode an internal representation
feature $h_{m}$ based on the particular modality:

$h_{m}=Tanh(W_{m}Z_{m}).$ (15)

where, $W_{m}$s are learnable weights related to the modality $m$. Another
fully connected layer is used for each $Z_{m}$, which controls the
contribution of the feature calculated from $Z_{m}$ to the overall output of
AAF. The output of this layer is another latent representation named
$Z^{{}^{\prime}}$:

$Z^{{}^{\prime}}_{m}=\sigma(W_{Z_{m}}[Z_{1},Z_{2},...,Z_{M},A])$ (16)

where, $W_{Z_{m}}$s are learnable weights related to modality $m$.

When a sample is fed to the gated fusion network, a gate layer associated with
the modality $m$ receives the feature vectors of all modalities and uses them
to decide whether the modality $m$ may contribute or not to the internal
encoding of the particular input sample:

$h=Z^{{}^{\prime}}_{1}h_{1}+Z^{{}^{\prime}}_{2}h_{2}+...+Z^{{}^{\prime}}_{M}h_{M}.$
(17)

Finally, a fully connected layer and a sigmoid function $\sigma$ transform $h$
into a label:

$\hat{y}=\sigma((W_{c}h)+b_{c}),$ (18)

where, $W_{c}$ and $b_{c}$ are learnable weights and biases. We use the BCE
loss (as in the previous step) for the classification loss. The encoder and
decoder weights are frozen during this step.

##  5 Experiments

#### Datasets

A large number of public sleep studies are available, including those
available through the National Sleep Research Resource platform (Mueller,
2024). Most of those, however, are not a good fit for evaluating our method.
For our study, we need sleep recordings that (i) have multimodal data (i.e.,
not unimodal or only a few modalities), (ii) are supervised (i.e., have expert
annotated apnea event labels), and (iii) have a large number of samples (the n
size for the sleep studies and patients), to allow training large deep
learning models. We use two of the largest public sleep datasets that match
the above criteria. These two are the Nationwide Children’s Hospital (NCH)
Sleep Data Bank (Lee et al., 2022) and Childhood Adenotonsillectomy Trial
(CHAT) dataset (Marcus et al., 2013; Redline et al., 2011).

NCH \- The first dataset offers a large and free source that includes both PSG
signals and linked electronic health records for the patients. The linked data
includes demographics and longitudinal clinical data such as encounters,
medication, measurements, diagnoses, and procedures. The dataset was collected
between 2017 and 2019 at Nationwide Children’s Hospital (NCH), Columbus, Ohio,
USA. Sleep studies were annotated in real-time by technicians at the time of
the study, and then were staged and scored by a second technician after the
study was completed. We used all studies that have all of the available six
modalities.

CHAT \- We also use recordings from the CHAT study, which is a randomized,
single-blind, multicenter trial designed to analyze the efficacy of early
removal of adenoids and tonsils (adenotonsillectomy) in children with mild to
moderate obstructive apnea. Physiological measures of sleep were assessed at
baseline and at seven months with standardized full PSG with central scoring
at the Brigham and Women’s Hospital, Boston, MA. In total, 1,447 children had
screened PSG and 464 were randomized to treatment. Children underwent
standardized PSG testing with scoring at a centralized sleep reading center,
cognitive and behavioral testing at baseline, and 7 months after
randomization. We use the PSGs collected in the baseline in our work.

#### Baselines

We chose three related and state of the art (SOTA) studies with different
architectures to compare with the proposed model. These three studies include
architectures based on CNNs (Chang et al., 2020), CNN+LSTM (Zarei et al.,
2022), and transformers (Fayyaz et al., 2023) as described in the following.

CNN \- The first study by Chang et al. (2020) uses a network consisting of 10
CNN layers for feature extraction followed by four fully connected layers for
classification. They also apply batch normalization and dropout for better
generalization and to avoid overfitting. They only utilized ECG for apnea
detection. However, to have a fair comparison, we trained this model with the
same modalities that we used for training our model.

CNN+LSTM \- The second model presented by Zarei et al. (2022) uses an
automatic feature extraction method developed by combining CNN and long short-
term memory (LSTM) modules using ECG. A stack of fully connected layers is
used at the end to classify the events. Similar to the previous baseline, we
trained this model with the same modalities that we used for training our
model.

Transformers \- The last model proposed by Fayyaz et al. (2023) uses
transformers followed by two layers of a fully connected network as a
classifier for detecting apnea using PSG signals.

#### Implementation details

Our model utilizes ECG, EEG, EOG, SpO2, CO2, and respiratory signals to detect
apnea. As the signals have different sampling rates, we resampled all signals
with a frequency of 128Hz. For model input, we divided each sleep study into
30-second non-overlapping epochs. ECG signal was denoised using a band-pass
filter, which allows only a certain range of frequencies to pass through while
blocking others, with lower and upper cutoff frequencies of 3Hz and 45Hz.
Hamilton R-peak detection method (Hamilton and Tompkins, 1986) was utilized to
extract R-R intervals and the amplitude of R-peaks from the ECG signal.

We run experiments in a 5-fold cross-validation manner and report the mean and
standard deviation of performance of the trained models using the F1-score
(harmonic mean of precision and recall) and AUROC (area under the receiver
operating characteristic curve) in the test set.

###  5.1 Research Questions

We study five major research questions in our experiments to investigate the
performance of our model.

#### Q1: How do the methods perform with complete modalities?

At first, we carried out an experiment to find the overall performance of our
method and the baselines when provided with complete data (i.e., without any
added noise or missingness). Table 1 shows the results. Our model has achieved
superior performance, measured in terms of F1 score and AUROC, compared to the
baselines. We also ran an ablation study to evaluate the performance of our
model in the absence of reconstruction loss (in unimodal pretraining) and the
distance measures $A$ (in multimodal training).

Table 1: The overall performance when using complete data. The three baselines are CNNs (Chang et al., 2020), CNN+LSTM (Zarei et al., 2022), and transformers (Fayyaz et al., 2023). Mean ($\pm$ STD). Method | CHAT | NCH
---|---|---
F1 | AUROC | F1 | AUROC
CNN | 77.5 (0.8) | 86.8 (1.0) | 77.2 (1.1) | 86.4 (1.2)
CNN+LSTM | 81.7 (0.6) | 89.7 (0.7) | 81.7 (0.8) | 89.4 (0.6)
Transformer | 83.1 (1.0) | 90.0 (0.8) | 82.6 (0.5) | 90.4 (0.4)
Ours without reconstruction loss | 85.1 (0.7) | 92.3 (0.6) | 83.9 (0.6) | 92.3 (0.4)
Ours | 86.6 (0.7) | 93.3 (0.6) | 85.2 (0.7) | 93.4 (0.4)

#### Q2: How do the methods perform in the presence of missing modalities?

To simulate the effect of missing modalities, we omitted signals within epochs
in a random manner. More specifically, we randomly substituted a predefined
percent of signals in epochs with zero. The algorithm for this purpose is
presented in Appendix D. Figure 2 (NCH data) and 3 (CHAT) show the results.

Figure 2: Model performance with missing modalities on the NCH dataset. Error
bars show the standard deviation. Figure 3: Models Performance with missing
modalities on CHAT dataset.

#### Q3: How do the methods perform with noisy modalities?

We investigated how well our model and other top-performing models handle
noisy data. In the absence of a dataset with controlled non-synthetic or
measured noise, we add synthetic noise to our target datasets. We added white
Gaussian noise to the signals to simulate unremovable types of noises (we
discuss this further in 6), using a process explained further in Appendix D.
Additive Gaussian noise can mimic the effect of many random processes that can
occur in real-life settings. The results are shown in Figures 4 (NCH) and 5
(CHAT). The signal-to-noise ratio (SNR) is a measure of the strength of a
desired signal compared to the background noise. It is often expressed in
decibels (dB), and a higher SNR value indicates a cleaner signal.

Figure 4: Model performance with noisy modalities on the NCH dataset. Figure
5: Models performance with noisy modalities on CHAT dataset.

#### Q4: How do the methods perform when the input has both noisy and missing
signals?

We also fed our model with both noisy and missing modalities at once, which
combines the two studied scenarios in Q3 and Q4. The results on the NCH and
CHAT datasets are shown in Table 2. We report the results of the baseline
models for the Q4’s scenario, in the Appendix (Table 5).

Table 2: Our method’s performance (mean AUROC $\pm$ STD) with the concurrent
occurrence of noise and missingness on NCH dataset.

| NCH | CHAT
---|---|---
Missing Ratio | Signal to noise ratio (dB)
| 10 | 20 | 30 | 40 | 50 | 10 | 20 | 30 | 40 | 50
10% | 84.9(1.9) | 90.4(0.9) | 92.6(0.5) | 92.8(0.5) | 92.8(0.5) | 85.5 (0.7) | 85.5 (0.6) | 90.0 (0.1) | 91.7 (0.4) | 91.8 (0.5)
20% | 82.9(1.9) | 89.0(1.1) | 91.8(0.6) | 92.0(0.5) | 92.1(0.5) | 83.1 (0.7) | 82.6 (1.5) | 87.7 (0.6) | 89.3 (0.1) | 89.9 (0.3)
30% | 80.9(2.0) | 87.3(1.0) | 90.6(0.6) | 91.1(0.5) | 91.1(0.5) | 79.8 (0.8) | 79.7 (0.9) | 85.0 (1.1) | 87.5 (0.1) | 87.3 (0.1)
40% | 78.6(1.9) | 85.4(1.2) | 89.3(0.6) | 89.9(0.5) | 89.8(0.5) | 76.6 (0.5) | 76.8 (1.5) | 81.8 (1.5) | 84.4 (0.8) | 84.6 (0.6)
50% | 76.9(2.0) | 83.4(1.0) | 87.6(0.6) | 88.0(0.6) | 88.2(0.4) | 73.5 (0.1) | 73.6 (1.7) | 78.6 (0.9) | 81.0 (1.1) | 81.1 (0.9)

#### Q5: How do the methods perform when specific modalities are completely
missing?

We also study the methods with one missing modality at a time, to find the
stability of our model when modalities are entirely (versus partially in Q2)
unavailable. Results are shown in Table 3.

Table 3: Performance of our model (mean AUROC $\pm$ STD) when specific
modalities are missing on the NCH dataset.

Missing Modality | NCH | CHAT
---|---|---
CNN | CNN-LSTM | Transformer | Ours | CNN | CNN-LSTM | Transformer | Ours
EOG | 83.3 (2.2) | 83.0 (9.9) | 89.2 (0.0) | 93.0 (0.7) | 68.1 (4.5) | 57.5 (4.1) | 83.9 (3.6) | 91.9 (1.5)
EEG | 84.4 (2.1) | 75.6 (9.4) | 89.2 (0.5) | 92.6 (0.6) | 68.9 (3.5) | 51.5 (7.3) | 84.6 (3.2) | 91.3 (1.7)
Resp | 82.9 (1.9) | 87.3 (2.3) | 86.3 (0.8) | 91.4 (0.5) | 73.1 (11.1) | 56.6 (10.4) | 86.3 (2.7) | 88.3 (1.7)
SpO2 | 64.8 (1.4) | 63.2 (3.0) | 83.5 (1.0) | 91.5 (0.3) | 51.0 (1.6) | 75.1 (5.8) | 83.5 (1.0) | 89.8 (0.7)
CO2 | 81.1 (3.4) | 80.4 (5.9) | 89.4 (0.5) | 92.9 (0.5) | 73.3 (18.6) | 88.6 (0.7) | 89.4 (0.5) | 92.5 (0.9)
ECG | 80.8 (2.3) | 85.6 (2.3) | 88.5 (0.5) | 93.1 (0.6) | 61.8 (6.9) | 54.0 (8.8) | 81.0 (3.6) | 91.8 (0.8)
EOG, EEG | 79.5 (2.9) | 74.7 (12.2) | 88.7 (0.5) | 91.5 (0.8) | 62.3 (2.2) | 47.8 (5.1) | 82.7 (4.0) | 90.3 (2.1)
None | 86.4(1.2) | 89.4(0.6) | 90.4 (0.4) | 93.4 (0.4) | 86.8(1.0) | 89.7(0.7) | 90.0 (0.8) | 93.3 (0.6)

##  6 Discussion

In this study, we presented a machine learning pipeline for apnea detection
with any combination of available PSG modalities. By leveraging the
complementary nature of multiple modalities, data fusion techniques can
mitigate the impact of missing data, reduce uncertainty, improve the overall
robustness of the fused information, and compensate for the absence of certain
modalities. All of these can allow for higher performance in machine learning
tasks (Woo et al., 2023).

We showed that our hybrid pipeline ––which combines a transformer-based
architecture and a gated fusion–– is robust to noisy and missing modalities
and outperforms several recent baselines. Specifically, we investigated the
effect of imperfect (noisy and missing) signals on the performance of apnea
detection models. The performance of the models that are not designed to
handle such conditions can degrade drastically as the imperfection of input
signals increases. As shown in Figures 2 and 4, our model is more resilient to
various degrees of missingness and noise in comparison with the baselines.

In this study, we used six different modalities, each of which with various
noise types. Some of these types of noise have been comprehensively studied,
and efficient removal approaches are available for such noises (Limaye and
Deshmukh, 2016; Lai et al., 2018). As a result, some standard noises are
removable using well-known denoising approaches without losing too much
information and performance degradation. For instance, the “baseline wander”
and “power-line interference” noises in ECG signals are removable using a
notch filter (a filter that eliminates a single frequency from a spectrum of
frequencies) and a high-pass filter (a filter that passes signals with a
frequency higher than a certain cutoff frequency). To demonstrate the
robustness of our proposed pipeline to this type of noise, we report the
performance of our method in the presence of such noise in Appendix C.

In our study, however, we focus on noises that can deeply disturb the signals
so that the original signals are not distinguishable. This type of noise is
not easily reproducible, like the easy-to-model noises we discussed above.
Modeling this type of noise is very hard. Such noise types include patient
movements or electrode detachment, which are events that can introduce severe
noise.

The clinical implications of our study are also worth noting. Our method can
be especially relevant to the ongoing efforts to adopt at-home sleep apnea
testing (HSAT) solutions. While there exist several FDA-approved commercial
HSAT products (Van Pee et al., 2022; Manoni et al., 2020), inside-clinic PSG
is still the gold standard for diagnosing sleep apnea. This is partly due to
the challenges that sleep recording can present.

Research on HSATs is especially relevant to pediatric populations, as such
tools are almost widely available for adults, but not children (Kirk et al.,
2017). In fact, a key barrier to running sleep apnea testing outside the
clinic for children is the presence of noise and missingness, as children can
move more frequently, be less cooperative, and pull the probes, among other
issues.

Additionally, a key difference between apnea detection in adults versus
children relates to using SpO2 (for adults) versus EEG (for children) as the
main signal for apnea detection (Bandla and Gozal, 2000). In children, the
brain activates early on to regulate breathing and sleep disorders, therefore
it functions as a better signal for apnea detection. As we show in Table 3,
even without EEG and EOG (i.e., the two critical but challenging to collect
signals in children), our method performs very well.

The two datasets that we use in this study are related to pediatric sleep
studies, further showcasing the performance of our pipeline on pediatric
cases. We also report the performance of our model across different ages in
Appendix E to further highlight the potential of our pipeline in informing the
efforts targeting younger age patients.

Some limitations of our study are worth noting. A common concern in apnea
detection models relates to the lack of sub-typing. Although apnea and
hypopnea have three different types (i.e., obstructive, central, and mixed)
(Javaheri et al., 2017), our work, similar to most prior studies, only tries
to train a model that detects apnea, not its type and severity. Additionally,
while there are many explanation methods for opaque-box methods (Belle and
Papantonis, 2021), our method (an opaque-box one) would still offer limited
explainability.

Moreover, in this study, we manually added noise to the lab-recorded PSG data.
While we are not aware of any large study collecting at-home sleep data,
evaluating our model using data with actual noise could have demonstrated our
model’s performance more comprehensively.

In the future, we plan to investigate whether applying our method can also
help to improve sleep staging performance in comparison with existing methods.
Sleep staging is mostly done using EEG and EOG. Our study was supported by NIH
awards, P20GM103446 and U54-GM104941.

## References

  * Abadi et al. (2016) Martín Abadi, Paul Barham, Jianmin Chen, Zhifeng Chen, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Geoffrey Irving, Michael Isard, et al.  Tensorflow: A system for large-scale machine learning.  In _12th USENIX symposium on operating systems design and implementation (OSDI )_ , pages 265–283, Savannah, GA, USA, 2016. ACM.
  * Almuhammadi et al. (2015) Wafaa S Almuhammadi, Khald AI Aboalayon, and Miad Faezipour.  Efficient obstructive sleep apnea classification based on EEG signals.  In _2015 Long Island Systems, Applications and Technology_ , pages 1–6. IEEE, 2015.
  * Álvarez et al. (2012) Daniel Álvarez, Roberto Hornero, J Víctor Marcos, and Félix Del Campo.  Feature selection from nocturnal oximetry using genetic algorithms to assist in obstructive sleep apnoea diagnosis.  _Medical engineering & physics_, 34(8):1049–1057, 2012.
  * Arevalo et al. (2017) John Arevalo, Thamar Solorio, Manuel Montes-y Gómez, and Fabio A González.  Gated multimodal units for information fusion.  _arXiv preprint arXiv:1702.01992_ , 2017.
  * Bahrami and Forouzanfar (2022) Mahsa Bahrami and Mohamad Forouzanfar.  Sleep apnea detection from single-lead ECG: A comprehensive analysis of machine learning and deep learning algorithms.  _IEEE Transactions on Instrumentation and Measurement_ , 71:1–11, 2022.
  * Bandla and Gozal (2000) Hari PR Bandla and David Gozal.  Dynamic changes in EEG spectra during obstructive apnea in children.  _Pediatric pulmonology_ , 29(5):359–365, 2000\.
  * Belle and Papantonis (2021) Vaishak Belle and Ioannis Papantonis.  Principles and practice of explainable machine learning.  _Frontiers in big Data_ , page 39, 2021.
  * Bixler et al. (2009) Edward O Bixler, Alexandros N Vgontzas, Hung-Mo Lin, Duanping Liao, Susan Calhoun, Antonio Vela-Bueno, Fred Fedok, Vukmir Vlasic, and Gavin Graff.  Sleep disordered breathing in children in a general population sample: prevalence and risk factors.  _Sleep_ , 32(6):731–736, 2009.
  * Chang et al. (2020) Hung-Yu Chang, Cheng-Yu Yeh, Chung-Te Lee, and Chun-Cheng Lin.  A sleep apnea detection system based on a one-dimensional deep convolution neural network model using single-lead electrocardiogram.  _Sensors_ , 20(15):4157, 2020.
  * Chen et al. (2022) Xianhui Chen, Ying Chen, Wenjun Ma, Xiaomao Fan, and Ye Li.  Toward sleep apnea detection with lightweight multi-scaled fusion network.  _Knowledge-Based Systems_ , 247:108783, 2022.
  * Chollet et al. (2015) François Chollet et al.  Keras.  https://github.com/fchollet/keras, 2015.
  * Du et al. (2022) Yongping Du, Yang Liu, Zhi Peng, and Xingnan Jin.  Gated attention fusion network for multimodal sentiment classification.  _Knowledge-Based Systems_ , 240:108107, 2022.
  * Fan et al. (2020) Cunhang Fan, Jiangyan Yi, Jianhua Tao, Zhengkun Tian, Bin Liu, and Zhengqi Wen.  Gated recurrent fusion with joint training framework for robust end-to-end speech recognition.  _IEEE/ACM Transactions on Audio, Speech, and Language Processing_ , 29:198–209, 2020.
  * Fayyaz et al. (2023) Hamed Fayyaz, Abigail Strang, and Rahmatollah Beheshti.  Bringing at-home pediatric sleep apnea testing closer to reality: A multi-modal transformer approach.  In _Machine Learning for Healthcare Conference_. PMLR, 2023.
  * Feng et al. (2020) Shuanglang Feng, Heming Zhao, Fei Shi, Xuena Cheng, Meng Wang, Yuhui Ma, Dehui Xiang, Weifang Zhu, and Xinjian Chen.  Cpfnet: Context pyramid fusion network for medical image segmentation.  _IEEE transactions on medical imaging_ , 39(10):3008–3018, 2020.
  * Gaw et al. (2022) Nathan Gaw, Safoora Yousefi, and Mostafa Reisi Gahrooei.  Multimodal data fusion for systems improvement: A review.  _IISE Transactions_ , 54(11):1098–1116, 2022\.
  * Hamilton and Tompkins (1986) Patrick S Hamilton and Willis J Tompkins.  Quantitative investigation of qrs detection rules using the mit/bih arrhythmia database.  _IEEE transactions on biomedical engineering_ , (12):1157–1165, 1986.
  * Hassan et al. (2022) Md Rafiul Hassan, Shamsul Huda, Mohammad Mehedi Hassan, Jemal Abawajy, Ahmed Alsanad, and Giancarlo Fortino.  Early detection of cardiovascular autonomic neuropathy: A multi-class classification model based on feature selection and deep learning feature fusion.  _Information Fusion_ , 77:70–80, 2022.
  * Hosseinpour et al. (2022) Hamidreza Hosseinpour, Farhad Samadzadegan, and Farzaneh Dadrass Javan.  Cmgfnet: A deep cross-modal gated fusion network for building extraction from very high-resolution remote sensing images.  _ISPRS journal of photogrammetry and remote sensing_ , 184:96–115, 2022.
  * Javaheri et al. (2017) Shahrokh Javaheri, Ferran Barbe, Francisco Campos-Rodriguez, Jerome A Dempsey, Rami Khayat, Sogol Javaheri, Atul Malhotra, Miguel A Martinez-Garcia, Reena Mehra, Allan I Pack, et al.  Sleep apnea: types, mechanisms, and clinical cardiovascular consequences.  _Journal of the American College of Cardiology_ , 69(7):841–858, 2017.
  * John et al. (2021) Arlene John, Koushik Kumar Nundy, Barry Cardiff, and Deepu John.  Somnnet: An spo2 based deep learning network for sleep apnea detection in smartwatches.  In _2021 43rd Annual International Conference of the IEEE Engineering in Medicine & Biology Society (EMBC)_, pages 1961–1964. IEEE, 2021\.
  * Kasasbeh et al. (2006) E Kasasbeh, David S Chi, and G Krishnaswamy.  Inflammatory aspects of sleep apnea and their cardiovascular consequences.  _Southern medical journal_ , 99(1):58–68, 2006\.
  * Kheirandish-Gozal and Gozal (2012) Leila Kheirandish-Gozal and David Gozal.  _Sleep disordered breathing in children: a comprehensive clinical guide to evaluation and treatment_.  Springer Science & Business Media, 2012.
  * Kingma and Ba (2017) Diederik P. Kingma and Jimmy Ba.  Adam: A method for stochastic optimization, 2017.
  * Kirk et al. (2017) Valerie Kirk, Julie Baughn, Lynn D’Andrea, Norman Friedman, Anjalee Galion, Susan Garetz, Fauziya Hassan, Joanna Wrede, Christopher G Harrod, and Raman K Malhotra.  American academy of sleep medicine position paper for the use of a home sleep apnea test for the diagnosis of osa in children.  _Journal of Clinical Sleep Medicine_ , 13(10):1199–1203, 2017.
  * Kline et al. (2022) Adrienne Kline, Hanyin Wang, Yikuan Li, Saya Dennis, Meghan Hutch, Zhenxing Xu, Fei Wang, Feixiong Cheng, and Yuan Luo.  Multimodal machine learning in precision health: A scoping review.  _npj Digital Medicine_ , 5(1):171, 2022.
  * Knutson et al. (2006) Kristen L Knutson, Armand M Ryden, Bryce A Mander, and Eve Van Cauter.  Role of sleep duration and quality in the risk and severity of type 2 diabetes mellitus.  _Archives of internal medicine_ , 166(16):1768–1774, 2006.
  * Lai et al. (2018) Chi Qin Lai, Haidi Ibrahim, Mohd Zaid Abdullah, Jafri Malin Abdullah, Shahrel Azmin Suandi, and Azlinda Azman.  Artifacts and noise removal for electroencephalogram (EEG): A literature review.  In _2018 IEEE Symposium on Computer Applications & Industrial Electronics (ISCAIE)_, pages 326–332. IEEE, 2018.
  * Lee et al. (2022) Harlin Lee, Boyue Li, Shelly DeForte, Mark L Splaingard, Yungui Huang, Yuejie Chi, and Simon L Linwood.  A large collection of real-world pediatric sleep studies.  _Scientific Data_ , 9(1):1–12, 2022.
  * Lee and Kang (2022) Yunseung Lee and Pilsung Kang.  Anovit: Unsupervised anomaly detection and localization with vision transformer-based encoder-decoder.  _IEEE Access_ , 10:46717–46724, 2022.  10.1109/ACCESS.2022.3171559.
  * Lim et al. (2019) Teck-Yian Lim, Amin Ansari, Bence Major, Daniel Fontijne, Michael Hamilton, Radhika Gowaikar, and Sundar Subramanian.  Radar and camera early fusion for vehicle detection in advanced driver assistance systems.  In _Machine learning for autonomous driving workshop at the 33rd conference on neural information processing systems_ , volume 2, 2019.
  * Limaye and Deshmukh (2016) Hrishikesh Limaye and VV Deshmukh.  Ecg noise sources and various noise removal techniques: A survey.  _International Journal of Application or Innovation in Engineering & Management_, 5(2):86–92, 2016.
  * Liu et al. (2023) Ziyi Liu, Jiaqi Zhang, Yongshuai Hou, Xinran Zhang, Ge Li, and Yang Xiang.  Machine learning for multimodal electronic health records-based research: Challenges and perspectives.  In _Health Information Processing_ , pages 135–155. Springer Nature Singapore, 2023.  ISBN 978-981-19-9865-2.
  * Luyster et al. (2012) Faith S Luyster, Patrick J Strollo Jr, Phyllis C Zee, and James K Walsh.  Sleep: a health imperative.  _Sleep_ , 35(6):727–734, 2012.
  * Manoni et al. (2020) Alessandro Manoni, Federico Loreti, Valeria Radicioni, Daniela Pellegrino, Luigi Della Torre, Alessandro Gumiero, Damian Halicki, Paolo Palange, and Fernanda Irrera.  A new wearable system for home sleep apnea testing, screening, and classification.  _Sensors_ , 20(24):7014, 2020.
  * Marcus et al. (2012) Carole L Marcus, Lee J Brooks, Sally Davidson Ward, Kari A Draper, David Gozal, Ann C Halbower, Jacqueline Jones, Christopher Lehmann, Michael S Schechter, Stephen Sheldon, et al.  Diagnosis and management of childhood obstructive sleep apnea syndrome.  _Pediatrics_ , 130(3):e714–e755, 2012.
  * Marcus et al. (2013) Carole L Marcus, Reneé H Moore, Carol L Rosen, Bruno Giordani, Susan L Garetz, H Gerry Taylor, Ron B Mitchell, Raouf Amin, Eliot S Katz, Raanan Arens, et al.  A randomized trial of adenotonsillectomy for childhood sleep apnea.  _N Engl J Med_ , 368:2366–2376, 2013.
  * Morillo and Gross (2013) Daniel Sánchez Morillo and Nicole Gross.  Probabilistic neural network approach for the detection of sahs from overnight pulse oximetry.  _Medical & biological engineering & computing_, 51(3):305–315, 2013.
  * Mueller (2024) Remo Mueller.  Sleep data - national sleep research resource - nsrr, 2024.  URL http://www.sleepdata.org/.
  * Muhammad et al. (2021) Ghulam Muhammad, Fatima Alshehri, Fakhri Karray, Abdulmotaleb El Saddik, Mansour Alsulaiman, and Tiago H Falk.  A comprehensive survey on multimodal medical signals fusion for smart healthcare systems.  _Information Fusion_ , 76:355–375, 2021.
  * Neves et al. (2023) Francisco Soares Neves, Rafael Marques Claro, and Andry Maykol Pinto.  End-to-end detection of a landing platform for offshore uavs based on a multimodal early fusion approach.  _Sensors_ , 23(5):2434, 2023.
  * Pathinarupothi et al. (2017) Rahul Krishnan Pathinarupothi, Ekanath Srihari Rangan, EA Gopalakrishnan, R Vinaykumar, KP Soman, et al.  Single sensor techniques for sleep apnea diagnosis using deep learning.  In _2017 IEEE international conference on healthcare informatics (ICHI)_ , pages 524–529. IEEE, 2017.
  * Pavlova and Latreille (2019) Milena K Pavlova and Véronique Latreille.  Sleep disorders.  _The American journal of medicine_ , 132(3):292–299, 2019.
  * Penzel et al. (2000) Thomas Penzel, George B Moody, Roger G Mark, Ary L Goldberger, and J Hermann Peter.  The apnea-ECG database.  In _Computers in Cardiology 2000. Vol. 27 (Cat. 00CH37163)_ , pages 255–258. IEEE, 2000.
  * Ravelo-García et al. (2015) Antonio G Ravelo-García, Jan F Kraemer, Juan L Navarro-Mesa, Eduardo Hernández-Pérez, Javier Navarro-Esteva, Gabriel Juliá-Serdá, Thomas Penzel, and Niels Wessel.  Oxygen saturation and rr intervals feature selection for sleep apnea detection.  _Entropy_ , 17(5):2932–2957, 2015.
  * Redline et al. (2011) Susan Redline, Raouf Amin, Dean Beebe, Ronald D Chervin, Susan L Garetz, Bruno Giordani, Carole L Marcus, Renee H Moore, Carol L Rosen, Raanan Arens, et al.  The childhood adenotonsillectomy trial (chat): rationale, design, and challenges of a randomized controlled trial evaluating a standard surgical procedure in a pediatric population.  _Sleep_ , 34(11):1509–1517, 2011.
  * Ren et al. (2018) Wenqi Ren, Lin Ma, Jiawei Zhang, Jinshan Pan, Xiaochun Cao, Wei Liu, and Ming-Hsuan Yang.  Gated fusion network for single image dehazing.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 3253–3261, 2018.
  * Rundo and Downey III (2019) Jessica Vensel Rundo and Ralph Downey III.  Polysomnography.  _Handbook of clinical neurology_ , 160:381–392, 2019.
  * Salari et al. (2022) Nader Salari, Amin Hosseinian-Far, Masoud Mohammadi, Hooman Ghasemi, Habibolah Khazaie, Alireza Daneshkhah, and Arash Ahmadi.  Detection of sleep apnea using machine learning algorithms based on ECG signals: A comprehensive systematic review.  _Expert Systems with Applications_ , 187:115950, 2022.
  * Schwartz et al. (2005) Daniel J Schwartz, William C Kohler, and Gillian Karatinos.  Symptoms of depression in individuals with obstructive sleep apnea may be amenable to treatment with continuous positive airway pressure.  _Chest_ , 128(3):1304–1309, 2005.
  * Schwartz et al. (2018) Martha Schwartz, Luis Acosta, Yuan-Lung Hung, Mariela Padilla, and Reyes Enciso.  Effects of cpap and mandibular advancement device treatment in obstructive sleep apnea patients: a systematic review and meta-analysis.  _Sleep and Breathing_ , 22:555–568, 2018.
  * Shen et al. (2021) Qi Shen, Hengji Qin, Keming Wei, and Guanzheng Liu.  Multiscale deep neural network for obstructive sleep apnea detection using rr interval from single-lead ECG signal.  _IEEE Transactions on Instrumentation and Measurement_ , 70:1–13, 2021.
  * Spielmanns et al. (2019) Marc Spielmanns, David Bost, Wolfram Windisch, Peter Alter, Tim Greulich, Christoph Nell, Jan Henrik Storre, Andreas Rembert Koczulla, and Tobias Boeselt.  Measuring sleep quality and efficiency with an activity monitoring device in comparison to polysomnography.  _Journal of clinical medicine research_ , 11(12):825, 2019.
  * Stahlschmidt et al. (2022) Sören Richard Stahlschmidt, Benjamin Ulfenborg, and Jane Synnergren.  Multimodal deep learning for biomedical data fusion: a review.  _Briefings in Bioinformatics_ , 23(2):bbab569, 2022.
  * Taheri (2006) Shahrad Taheri.  The link between short sleep duration and obesity: we should recommend more sleep to prevent obesity.  _Archives of disease in childhood_ , 91(11):881–884, 2006.
  * Tuncer et al. (2019) Seda Arslan Tuncer, Beyza Akılotu, and Suat Toraman.  A deep learning-based decision support system for diagnosis of osas using ptt signals.  _Medical hypotheses_ , 127:15–22, 2019.
  * Uçar et al. (2017) Muhammed Kürşad Uçar, Mehmet Recep Bozkurt, Cahit Bilgin, and Kemal Polat.  Automatic detection of respiratory arrests in osa patients using ppg and machine learning techniques.  _Neural Computing and Applications_ , 28(10):2931–2945, 2017.
  * Urtnasan et al. (2018) Erdenebayar Urtnasan, Jong-Uk Park, Eun-Yeon Joo, and Kyoung-Joung Lee.  Automated detection of obstructive sleep apnea events from a single-lead electrocardiogram using a convolutional neural network.  _Journal of medical systems_ , 42(6):1–8, 2018\.
  * Van Pee et al. (2022) Bart Van Pee, Frederik Massie, Steven Vits, Pauline Dreesen, Susie Klerkx, Jagdeep Bijwadia, Johan Verbraecken, and Jeroen Bergmann.  A multicentric validation study of a novel home sleep apnea test based on peripheral arterial tonometry.  _Sleep_ , 45(5):zsac028, 2022.
  * Van Steenkiste et al. (2020) Tom Van Steenkiste, Dirk Deschrijver, and Tom Dhaene.  Sensor Fusion using Backward Shortcut Connections for Sleep Apnea Detection in Multi-Modal Data.  In _Proceedings of the Machine Learning for Health NeurIPS Workshop_ , volume 116 of _Proceedings of Machine Learning Research_ , pages 112–125. PMLR, 13 Dec 2020.
  * Vaquerizo-Villar et al. (2020) Fernando Vaquerizo-Villar, Daniel Álvarez, Leila Kheirandish-Gozal, Gonzalo C Gutiérrez-Tobal, Javier Gómez-Pilar, Andrea Crespo, Felix Del Campo, David Gozal, and Roberto Hornero.  Automatic assessment of pediatric sleep apnea severity using overnight oximetry and convolutional neural networks.  In _2020 42nd Annual International Conference of the IEEE Engineering in Medicine & Biology Society (EMBC)_, pages 633–636. IEEE, 2020\.
  * Vimala et al. (2019) V Vimala, K Ramar, and M Ettappan.  An intelligent sleep apnea classification system based on EEG signals.  _Journal of medical systems_ , 43(2):36, 2019\.
  * Wang et al. (2019) Siwei Wang, Xinwang Liu, En Zhu, Chang Tang, Jiyuan Liu, Jingtao Hu, Jingyuan Xia, and Jianping Yin.  Multi-view clustering via late fusion alignment maximization.  In _IJCAI_ , pages 3778–3784, 2019.
  * Woo et al. (2023) Sangmin Woo, Sumin Lee, Yeonju Park, Muhammad Adi Nugroho, and Changick Kim.  Towards good practices for missing modality robust action recognition.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 37, pages 2776–2784, 2023.
  * Xie and Minn (2012) Baile Xie and Hlaing Minn.  Real-time sleep apnea detection by classifier combination.  _IEEE Transactions on information technology in biomedicine_ , 16(3):469–477, 2012.
  * Ye et al. (2023) Pengfei Ye, Han Qin, Xiaojun Zhan, Zhan Wang, Chang Liu, Beibei Song, Yaru Kong, Xinbei Jia, Yuwei Qi, Jie Ji, et al.  Diagnosis of obstructive sleep apnea in children based on the xgboost algorithm using nocturnal heart rate and blood oxygen feature.  _American Journal of Otolaryngology_ , 44(2):103714, 2023.
  * Zarei et al. (2022) Asghar Zarei, Hossein Beheshti, and Babak Mohammadzadeh Asl.  Detection of sleep apnea using deep neural networks and single-lead ECG signals.  _Biomedical Signal Processing and Control_ , 71:103125, 2022\.
  * Zhang et al. (2018) Xinyi Zhang, Hang Dong, Zhe Hu, Wei-Sheng Lai, Fei Wang, and Ming-Hsuan Yang.  Gated fusion network for joint image deblurring and super-resolution.  _arXiv preprint arXiv:1807.10806_ , 2018.
  * Zhang et al. (2019) Yunke Zhang, Lixue Gong, Lubin Fan, Peiran Ren, Qixing Huang, Hujun Bao, and Weiwei Xu.  A late fusion cnn for digital matting.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pages 7469–7478, 2019.
  * Zhao et al. (2021) Xiaoyun Zhao, Xiaohong Wang, Tianshun Yang, Siyu Ji, Huiquan Wang, Jinhai Wang, Yao Wang, and Qi Wu.  Classification of sleep apnea based on EEG sub-band signal characteristics.  _Scientific Reports_ , 11(1):5824, 2021.
  * Zimmerman et al. (2006) Mark Zimmerman, Joseph B McGlinchey, Diane Young, and Iwona Chelminski.  Diagnosing major depressive disorder i: A psychometric evaluation of the dsm-iv symptom criteria.  _The Journal of nervous and mental disease_ , 194(3):158–163, 2006.

##  Appendix A Transformer Backbone

The encoder and decoder consist of multiple layers of transformer, each
consisting of two components. Both components have a residual connection and a
normalization layer. The first is the multi-head self-attention (MHSA), which
allows the different parts of the input sequence to interact:

$X^{{}^{\prime}}=LayerNorm(X)$ (19)

$MHSA(X^{{}^{\prime}})=Concat(h_{1},...,h_{n})W^{C}$ (20)

where:

$h_{i}=Softmax(\frac{W^{K}_{i}X^{{}^{\prime}}(W^{N}_{i}X^{{}^{\prime}})^{T}}{\sqrt{d_{k}}})W^{V}_{i}X^{{}^{\prime}}.$
(21)

The second component is a fully connected network (FCN) which applies a
nonlinear transformation to each position in the sequence:

$X^{{}^{\prime\prime}}=LayerNorm(X^{{}^{\prime}}+MHSA(X^{{}^{\prime}}))$ (22)

$FCN(X^{{}^{\prime\prime}})=ReLU(X^{{}^{\prime\prime}}W_{1}+b_{1})W_{2}+b_{2},$
(23)

$output=X^{{}^{\prime\prime}}+FCN(X^{{}^{\prime\prime}})$ (24)

$W_{i}^{N},W_{i}^{K},W_{i}^{V}$ are learnable weights related to the head $i$
of the self-attention module. $d_{k}$ is the dimension of queries and keys in
self-attention. ($W_{1}$, $b_{1}$) and ($W_{2}$, $b_{2}$) are the learnable
weights and biases for the first and second layers, respectively.

##  Appendix B Additional training details

In experiments, we used Adam optimizer (Kingma and Ba, 2017) with a learning
rate of $10^{-3}$, $\beta_{1}=0.9$, ${\beta}_{2}=0.999$, and
$\epsilon=10^{-7}$. A batch size of 256 was used for training. We use L2
weight regularization with $\lambda=10^{-3}$ and dropout (rate=0.25) to avoid
overfitting. We trained the model for 100 epochs in a 5-fold cross-validation
manner. our model was implemented using Keras (Chollet et al., 2015) inside
the TensorFlow (Abadi et al., 2016) framework.

#### Hyper-parameters tuning

We also tested our model with different hyper-parameters. We ran the
experiments with models with different numbers of transformer layers and heads
in each layer. For our task, the best performance achieved with the model has
5 layers and 4 heads in each layer. in our architecture, each layer consists
of two layers of MLP with 16 and 32 units. Fully connected layers in a gated
fusion network have 8 units, which have been selected among 4, 8, 16, and 32
units.

##  Appendix C Extended Results

The performance of SOTA models in terms of AUROC with the concurrent
occurrence of noise and missingness on the NCH dataset is shown in Table 5.
The performance of the proposed model in terms of AUROC when just utilizing
ECG signals with Baseline wandering and Powerline interference noises are
shown in Table 4.

Noise type | AUROC
---|---
CHAT | NCH
Baseline wandering | 82.2 (0.7) | 82.5 (0.7)
Powerline interference | 82.4 (0.8) | 82.8 (0.7)
Without Noise | 83.1 (0.6) | 83.4 (0.6)
Table 4: Performance of our model with noisy ECG signals. Mean($\pm$ STD) Table 5: Performance of the baselines and our model in terms of AUROC with concurrent occurrence of noise and missingness on NCH dataset. The mean (standard deviation) values are shown. Missing ratio | SNR | CNN-LSTM | CNN | Transformer | Ours
---|---|---|---|---|---
10% | 10 | 57.0(3.3) | 58.8(2.0) | 82.1(0.2) | 84.9(1.9)
20 | 66.6(5.5) | 73.7(1.1) | 83.0(0.3) | 90.4(0.9)
30 | 80.4(0.9) | 80.5(0.7) | 84.6(0.5) | 92.6(0.5)
40 | 81.8(0.9) | 81.4(1.0) | 86.3(0.5) | 92.8(0.5)
50 | 81.8(1.0) | 81.2(0.9) | 86.5(0.3) | 92.8(0.5)
20% | 10 | 57.4(2.4) | 58.8(1.7) | 79.4(0.2) | 82.9(1.9)
20 | 64.6(3.7) | 69.0(0.7) | 80.0(0.5) | 89.0(1.1)
30 | 74.9(0.5) | 74.1(0.5) | 81.8(0.4) | 91.8(0.6)
40 | 75.9(0.7) | 74.7(0.9) | 83.1(0.4) | 92.0(0.5)
50 | 75.9(0.9) | 74.5(1.0) | 83.3(0.4) | 92.1(0.5)
30% | 10 | 56.8(2.0) | 58.6(1.2) | 76.2(0.5) | 80.9(2.0)
20 | 62.3(2.2) | 65.2(0.6) | 77.1(0.3) | 87.3(1.0)
30 | 70.2(0.6) | 68.7(0.5) | 78.3(0.5) | 90.6(0.6)
40 | 70.5(1.2) | 69.3(0.8) | 79.5(0.5) | 91.1(0.5)
50 | 70.4(1.2) | 69.3(0.8) | 80.0(0.5) | 91.1(0.5)
40% | 10 | 56.6(0.9) | 57.6(0.9) | 73.1(0.7) | 78.6(1.9)
20 | 61.0(1.6) | 62.4(0.5) | 73.8(0.9) | 85.4(1.2)
30 | 66.0(0.4) | 64.7(0.3) | 74.8(0.8) | 89.3(0.6)
40 | 66.3(1.1) | 65.0(0.7) | 76.1(1.0) | 89.9(0.5)
50 | 66.4(1.1) | 64.9(0.9) | 76.3(1.1) | 89.8(0.5)
50% | 10 | 56.0(1.0) | 56.6(0.9) | 69.7(0.9) | 76.9(2.0)
20 | 58.6(1.4) | 59.7(0.5) | 70.4(0.9) | 83.4(1.0)
30 | 62.3(0.9) | 60.9(0.4) | 71.3(0.8) | 87.6(0.6)
40 | 62.6(0.8) | 61.1(0.7) | 72.3(1.2) | 88.0(0.6)
50 | 62.5(0.4) | 61.2(0.7) | 72.7(1.2) | 88.2(0.4)

##  Appendix D Noise and missingness algorithms

Additive white Gaussian noise generation and Random epoch-channel omission are
shown in algorithms D and 1, respectively.

Input: $\\{S_{i}\\}_{i=1}^{I},omission\textunderscore ratio$

Output: $\\{S_{i}\\}_{i=1}^{I}$

for  _$i\leftarrow 1$ to $I$_ do

for  _$j\leftarrow 1$ to $J_{i}$_ do

for  _$m\leftarrow 1$ to $M$_ do

$rnd\leftarrow$ Generate a random number in (0,1)

if  _$rnd\leq omission\textunderscore ratio$_ then

$X^{i,j}_{m}=0$

end for

end for

end for

Algorithm 1 Random epoch-channel omission

for  _$j\leftarrow 1$ to $J_{i}$_ do

for  _$m\leftarrow 1$ to $M$_ do

$signal\textunderscore average\textunderscore
power=\frac{\sum_{t=1}^{T}(X^{i,j}_{t,m})^{2}}{T}$
$signal\textunderscore average\textunderscore power\textunderscore
db=10*log_{10}(signal\textunderscore average\textunderscore power)$
$noise\textunderscore average\textunderscore power\textunderscore
db=signal\textunderscore average\textunderscore power\textunderscore db-
target\textunderscore snr$
$noise\textunderscore average\textunderscore power=10^{noise\textunderscore
average\textunderscore power\textunderscore db}$
$noise\sim Normal(0,\sqrt{noise\textunderscore average\textunderscore power})$
$rnd\leftarrow$ Generate a random number in (0,1)
if  _$rnd\leq noise\textunderscore occurrence\textunderscore chance$_ then

$X^{i,j}_{m}\leftarrow X^{i,j}_{m}+noise$

end for

end for

end for

##  Appendix E Model performance across different ages

We separated out patients according to their age range to study the
performance of the methods across different ages. The results for the NCH
dataset are shown in Figure 6. One can observe that the model’s discriminative
performance remains consistently over 90%, while the performance is slightly
lower in younger ages. The patients’ age distribution is shown in Figure 6.

Figure 6: Performance of the proposed and state-of-the-art models across
different age groups on the NCH dataset.

Additive white Gaussian noise generation Input:
$\\{S_{i}\\}_{i=1}^{I},target\textunderscore snr,noise\textunderscore
occurrence\textunderscore chance$

Output: $\\{S_{i}\\}_{i=1}^{I}$

for  _$i\leftarrow 1$ to $I$_ do


11institutetext: Intelligent Process Automation and Robotics Lab (IPR),
Institute for Anthropomatics and Robotics, Karlsruhe Institute of Technology,
Karlsruhe, Germany 11email<EMAIL_ADDRESS>
22institutetext: Karlsruhe University of Applied Sciences, Karlsruhe, Germany

# Adapting the Human: Leveraging Wearable Technology in HRI ††thanks: This
work has been supported from the European Union’s Horizon 2020 research and
innovation programme under grant agreement No 688117 “Safe human-robot
interaction in logistic applications for highly flexible warehouses(SafeLog)”.

David Puljiz 11 0000-0002-8966-3091 Björn Hein 1122 0000-0001-9569-5201

###### Abstract

Adhering to current HRI paradigms, all of the sensors, visualisation and
legibility of actions and motions are borne by the robot or its working cell.
This necessarily makes robots more complex or confines them into specialised,
structured environments. We propose leveraging the state of the art of
wearable technologies, such as augmented reality head mounted displays, smart
watches, sensor tags and radio-frequency ranging, to ”adapt” the human and
reduce the requirements and complexity of robots.

###### Keywords:

Wearables Head-mounted Displays Augmented Reality

Every robot that interacts with humans needs to fulfil many basic
requirements, no matter if in a social, service or industrial context -
compliance, either passive mechanical or active through force-torque sensors;
safety through laser scanners, cameras etc.; legibility of motions and
actions, through more human-like motions or by adding other visual cues; human
intention estimation to facilitate cooperation, and many more. All of these
sensors, visualisations and legibility of actions and motions are borne by the
robot or its working cell. This makes robots that interact with humans complex
and costly.

One possible alternative could be to meet the robots half-way and ”adapt” the
humans too, so the robots have an easier time fulfilling the requirements
without increasing complexity. The human would carry all or part of the
visualisation systems and sensors needed for an intuitive interaction. These
would come in the form of wearables - augmented reality (AR) head-mounted
displays (HMDs) such as the HoloLens, MagicLeap etc., sensor tags or smart
watches, and specialised radio frequency (RF) ranging units for functional
safety. Except for the ranging components the presented wearables are already
used, or will be in the near future, in everyday life, thus not presenting any
extra burden for the human. This trend will only increase as newer
technologies, such as smart textiles, become ever more present. Thus,
leveraging wearables should become a key point in future HRI paradigms.

The wearables here would act as a translators from human-interpretable data to
robot-interpretable data and vice versa, meaning that any interaction between
a human and a robot can then be boiled down to machine-to-machine
communication.

The HMDs present the central component in such a system as they are
responsible for audio and AR cues, carry most of the input modalities and
posses almost all of the sensors. In several works we have shown that HMDs
allow easier programming of robots [1], display of robot intentions [2],
estimation of intentions [3] and tracking of the human coworker [4] etc. This
makes the interaction more intuitive for the human and easier to understand
for the robot. More information means that the robotic system has to deal with
less uncertainty, making decisions simpler. Although HMDs are confined to
enterprises at the moment, they are expect to be available for personal use in
the next few years.

Another interesting use of AR in social robotics in particular comes from
Hirokazu Kato during the 2nd VAM-HRI workshop in 2019 [5]. He proposed that
the agent the human interacts with is disembodied. This agent could then
”inhabit” different robots and systems or be disembodied when the system
cannot be present (e.g. a Roomba cannot climb stairs). Agents can of course be
personalised. This particular modality may be further enhanced with cloud
robotics.

Smart watches with inertial measurement units (IMUs) and/or sensor tags are
used to track a person’s arms when outside the field of view of the HMDs
sensors. This is not strictly necessary but enhances safety and makes gesture
commands more intuitive as they are not restricted by the sensors’ fields of
view. The IMUs accumulate errors fast, thus they still need the sensors of the
HMD to correct them and are not a stand-alone solution.

In the European project SafeLog one of the main challenges is ensuring human
safety inside automated warehouses. Adapting the mobile robot to be safe was
not an option as modifying fleets larger than 500 robots would be financially
unfeasible. The answer was to outfit the robots with simple and cheap
hardware, and let the human wear the complex part of the safety system, based
on RF ranging [6]. As the robots would largely outnumber the humans in the
warehouse, such a system would be much cheaper than e.g. outfitting the robots
with laser sensors. Though the system proved highly effective in scenarios
where the robots significantly outnumber the humans, it also reduces the
number of sensor and complexity of the robots themselves. Thus it is
applicable in all manner of HRI scenarios.

The vision of such a worn systems is purely the realm of research at the
moment, however the enabling technologies are already there or soon to be.
Such an approach could be enhanced in the future with heart rate sensors to
estimate human emotions [7], electromyography sensors for better action
estimation [8] etc. We must move away from our anthropocentric view of HRI,
where we expect the robot to do everything, and move towards paradigms where
humans too wear sensors and hardware to better translate the human world to
the robot world and improve cooperation between them.

## References

  * [1] D. Puljiz, E. Stöhr, K. S. Riesterer, B. Hein, and T. Kröger, “General hand guidance framework using microsoft hololens,” in 2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 5185–5190, 2019.
  * [2] D. Puljiz, G. Gorbachev, and B. Hein, “Implementation of augmented reality in autonomous warehouses: Challenges and opportunities,” CoRR, vol. abs/1806.00324, 2018.
  * [3] T. Petković, D. Puljiz, I. Marković, and B. Hein, “Human intention estimation based on hidden markov model motion validation for safe flexible robotized warehouses,” Robotics and Computer-Integrated Manufacturing, vol. 57, pp. 182 – 196, 2019.
  * [4] D. Puljiz and B. Hein, “Concepts for end-to-end augmented reality based human-robot interaction systems,” 2019.
  * [5] T. Williams, D. Szafir, T. Chakraborti, and E. Phillips, “Virtual, augmented, and mixed reality for human-robot interaction (vam-hri),” in 2019 14th ACM/IEEE International Conference on Human-Robot Interaction (HRI), pp. 671–672, 2019.
  * [6] B. Ivšić, Z. Šipuš, J. Bartolić, and J. Babić, “Analysis of safe ultrawideband human-robot communication in automated collaborative warehouse,” in 2020 14th European Conference on Antennas and Propagation (EuCAP), pp. 1–5, 2020.
  * [7] K. Itoh, H. Miwa, Y. Nukariya, M. Zecca, H. Takanobu, S. Roccella, M. C. Carrozza, P. Dario, and A. Takanishi, “Development of a bioinstrumentation system in the interaction between a human and a robot,” in 2006 IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 2620–2625, 2006.
  * [8] C. Assad, M. Wolf, A. Stoica, T. Theodoridis, and K. Glette, “Biosleeve: A natural emg-based interface for hri,” in 2013 8th ACM/IEEE International Conference on Human-Robot Interaction (HRI), pp. 69–70, 2013.


# Singlet-triplet transitions in highly correlated nanowire quantum dots

Y. T. Chen  Department of Electrophysics, National Chiao Tung University,
Hsinchu 30010, Taiwan, Republic of China  C. C. Chao  S. Y. Huang  C. S. Tang
Department of Mechanical Engineering, National United University, 1, Lienda,
Miaoli 36003, Taiwan, Republic of China  S. J. Cheng<EMAIL_ADDRESS>
###### Abstract

We consider a quantum dot embedded in a three-dimensional nanowire with
tunable aspect ratio $a$. A configuration interaction theory is developed to
calculate the energy spectra of the finite 1D quantum dot systems charged with
two electrons in the presence of magnetic fields $B$ along the wire axis.
Fruitful singlet-triplet transition behaviors are revealed and explained in
terms of the competing exchange interaction, correlation interaction, and spin
Zeeman energy. In the high aspect ratio regime, the singlet-triplet
transitions are shown designable by tuning the parameters $a$ and $B$. The
transitions also manifest the highly correlated nature of long nanowire
quantum dots.

###### keywords:

Nanowire quantum dot , Exchange , Correlation , Singlet-triplet transition

††journal: Physica E

##  1 Introduction

For years, few electron charged quantum dots have attracted extensive
attention due to the controllable electronic and spin properties [1]. However,
only few attempts have so far been made for studies of finite 1D nanowire
quantum dots (NWQDs). More recently, it was shown that the NWQDs formed in the
heterostructures in nanowires can be fabricated as single electron transistors
and successively charged with controlled number of electrons [2]. The
successful experimental works motivate us to explore the possible geometric
effects of NWQDs characterized by their aspect ratios, $a$, on the electronic
and spin properties of two-electron charged NWQDs.

In this work, we focus on the study of the singlet-triplet (ST) transitions in
two-electron charged NWQDs [3], conducted by using a developed configuration
interaction (CI) theory in combination with the exact diagonalization
techniques based on a 3D asymmetric parabolic model. It will be illustrated
that the ST transitions in InAs-based NWQDs driven by an appropriate magnetic
field are associated with the competing effects of large spin-Zeeman energies
as well as the exchange and correlation energies. The correlation-dominated
nature of a long NWQD (i.e. with high aspect ratio) will be identified by the
spin phase diagram with respect to the applied magnetic fields and the tunable
aspect ratio.

##  2 Theoretical Model

###  2.1 single-electron spectrum

We begin with the single electron problem of a NWQD with axial magnetic field
${\bf B}=(0,0,B)$, described by the Hamiltonian [4]

$H_{0}=\frac{1}{2m^{\ast}}({\bf p}+e{\bf A})^{2}+V(x,y,z)+H_{\rm Z}\,,$ (1)

where ${\bf A}=(B/2)(y,-x,0)$ denotes the vector potential and $m^{\ast}$
stands for the effective mass of an electron with charge $-e$. The spin-Zeeman
Hamiltonian $H_{\rm Z}=g^{\ast}\mu_{B}Bs_{z}$ is in terms of the z-component
of electron spin $s_{z}$ and the effective Lande g-factor of electron
$g^{\ast}$ and the Bohr magneton $\mu_{B}$. In addition, the confining
potential
$V(x,y,z)=m^{\ast}\left[\omega_{0}^{2}\left(x^{2}+y^{2}\right)+\omega_{z}^{2}z^{2}\right]/2$
is assumed of the parabolic form with $\omega_{0}$ and $\omega_{z}$
parametrizing, respectively, the transverse and the longitudinal confining
strength.

In this work, we assume a constant $g^{\ast}$ and take the $g^{\ast}=-8.0$ for
InAs [3, 6]. The single electron Hamiltonian (1) leads to the extended Fock-
Darwin single-particle spectrum

$\displaystyle\epsilon_{n,m,q,s_{z}}$ $\displaystyle=$
$\displaystyle\hbar\omega_{+}\left(n+\frac{1}{2}\right)+\hbar\omega_{-}\left(m+\frac{1}{2}\right)$
(2) $\displaystyle+\hbar\omega_{z}\left(q+\frac{1}{2}\right)+E_{\rm Z}$

where $n,m,q=0,1,2\cdots$ denote oscillator quantum numbers,
$s_{z}=+\frac{1}{2}$ ($s_{z}=-\frac{1}{2}$) the projection of electron spin
$\uparrow$ ($\downarrow$) $E_{\rm Z}=g^{\ast}\mu_{B}Bs_{z}$ the spin Zeeman
energy, and $\omega_{\pm}=\omega_{h}\pm\omega_{c}/2$ is defined in terms of
the hybridized frequency
$\omega_{h}\equiv(\omega_{0}^{2}+\omega_{c}^{2}/4)^{1/2}$ and the cyclotron
frequency $\omega_{c}={eB}/{m^{\ast}}$. .

The eigenstate $|n,m,q\rangle$ possesses the orbital angular momentum
$l_{z}=\hbar(n-m)$ and the parity $P=1$ ($P=-1$) with respect to $z-$axis for
the even (odd) $q$ number. The wave function of the lowest orbital is given by
$\psi_{000}({\bf
r})=\exp\left[-\left(\left(x^{2}+y^{2}\right)/{l_{h}^{2}}+{z^{2}}/{l_{z}^{2}}\right)/4\right]/\left(2\pi^{3/4}{l_{h}}\sqrt{l_{z}}\right)$
with the characteristic lengths of the wave function extents
$l_{h}=\sqrt{{\hbar}/{2m^{\ast}\omega_{h}}}$ and
$l_{z}=\sqrt{{\hbar}/{2m^{\ast}\omega_{z}}}$, from which one can generate the
wave functions of any other excited states by successively applying raising
operators [5]. For most synthesized NWQDs, the diameter of the cross section
is of the scales $50$ nm, while the length of wire could be tunable over a
wide range from $10$ nm to $300$ nm [6]. To characterize the geometry of
NWQDs, we define the aspect ratio parameter,

$a\equiv\frac{l_{z}}{l_{0}}=\sqrt{\frac{\omega_{0}}{\omega_{z}}}\sim\frac{L_{z}}{L_{x}}\,.$
(3)

according to the extents of the wave function.

Figure 1: Single-electron energy spectrum as a function of aspect ratio $a$
with no magnetic field. The dot diameter $L_{x}$=$50$ nm and the dot height
$L_{z}$ varies from $40\rm{nm}$ to $150\rm{nm}$. The finite difference results
are shown in the inset for comparison.

Figure 1 shows the single electron energy spectra, as a function of $a$, of
NWQDs at zero magnetic field. To examine the validity of the model, we carry
out a numerical finite difference (FD) simulation for the electronic structure
of InAs/InP heterostructure NWQD, as shown in the inset of Figure 1. The InAs
NWQD is embedded in InP barriers with the diameter $L_{x}=50$ nm and varying
the length from $L_{z}=40$ nm to $L_{z}=150$ nm. The effective mass
$m^{\ast}=0.023m_{0}$ and the barrier offset $V_{b}=0.6$ eV are taken [7]. The
confining strength parameter $\hbar\omega_{0}$ is fit by the ground state
energy from the FD simulation. We set $\hbar\omega_{0}=13.3$ meV for
$L_{x}=50$ nm NWQDs throughout this paper. The numerically calculated energy
spectrum is in good agreement with that given by parabolic model. The
schematic illustration of the engineered single electron energy levels of
NWQDs by tunable $a$ and applied $B$ is shown in Figure 2.

Figure 2: Schematic diagram of the lowest two orbitals occupied by two
relevant electrons for the cases: (a) isotropic ($a=1$) and $B=0$; (b) “rod-
like” ($a>1$) and $B=0$; (c) with nonzero electronic $g$-factor and $B\neq 0$.
$2E_{Z}=g^{\ast}\mu_{B}B$ indicates the spin Zeeman energy splitting.

###  2.2 Interacting NWQD

To investigate the few-electron interaction effects in a NWQD, we express the
few electron Hamiltonian in second quantization as

$\displaystyle H$ $\displaystyle=$
$\displaystyle\sum_{i,\sigma}\epsilon_{i\sigma}c_{i\sigma}^{{\dagger}}c_{i\sigma}$
(4) $\displaystyle+\frac{1}{2}\sum_{ijkl,\sigma\sigma^{\prime}}\langle
ij|V|kl\rangle
c_{i\sigma}^{{\dagger}}c_{j\sigma^{\prime}}^{{\dagger}}c_{k\sigma^{\prime}}c_{l\sigma}\,,$

where $i,j,k,l$ stand for the composite indices of single electron orbitals
(e.g. $|i\rangle=|n_{i},m_{i},q_{i}\rangle$), $\sigma=\uparrow/\downarrow$
denotes the electron spin with $s_{z}=+\frac{1}{2}/-\frac{1}{2}$, and
$c_{i\sigma}^{{\dagger}}$ ($c_{i\sigma}$) is the electron creation
(annihilation) operators. The first (second) term on the right hand side of
Eq.(4) represents the kinetic energy of electrons (the Coulomb interactions
between electrons) and the Coulomb matrix elements are defined as $\langle
ij|V|kl\rangle\equiv e^{2}\left(4\pi\kappa\right)^{-1}\int\int d{\bf
r_{1}}d{\bf r_{2}}\psi_{i}^{*}({\bf r_{1}})\psi_{j}^{*}({\bf
r_{2}})\left(|{\bf r_{1}}-{\bf r_{2}}|\right)^{-1}\psi_{k}({\bf
r_{2}})\psi_{l}({\bf r_{1}})$, where $\kappa$ is the dielectric constant of
dot material ( $\kappa=15.15\epsilon_{0}$ is taken for InAs throughout this
work). After lengthy derivation, for NWQDs with $a\geq 1$, we obtain the
following formulation of the Coulomb matrix elements:

$\displaystyle\langle
n_{i}m_{i}q_{i};n_{j}m_{j}q_{j}|V|n_{k}m_{k}q_{k};n_{l}m_{l}q_{l}\rangle$
$\displaystyle=$ $\displaystyle(\frac{1}{\pi
l_{h}})\frac{\delta_{R_{L},R_{R}}\cdot\delta_{q_{i}+q_{j}+q_{l}+q_{k},\rm{even}}}{\sqrt{n_{i}!m_{i}!q_{i}!n_{j}!m_{j}!q_{j}!n_{k}!m_{k}!q_{k}!n_{l}!m_{l}!q_{l}!}}$
$\displaystyle\times$
$\displaystyle\sum_{p_{1}=0}^{\min(n_{i},n_{l})}\sum_{p_{2}=0}^{\min(m_{i},m_{l})}\sum_{p_{3}=0}^{\min(q_{i},q_{l})}\sum_{p_{4}=0}^{\min(n_{j},n_{k})}\sum_{p_{5}=0}^{\min(m_{j},m_{k})}\sum_{p_{6}=0}^{\min(q_{j},q_{k})}$
$\displaystyle\times$ $\displaystyle p_{1}!p_{2}!p_{3}!p_{4}!p_{5}!p_{6}!$
$\displaystyle\times$ $\displaystyle{n_{i}\choose p_{1}}{n_{l}\choose
p_{1}}{m_{i}\choose p_{2}}{m_{l}\choose p_{2}}{q_{i}\choose
p_{3}}{q_{l}\choose p_{3}}$ $\displaystyle\times$ $\displaystyle{n_{j}\choose
p_{4}}{n_{k}\choose p_{4}}{m_{j}\choose p_{5}}{m_{k}\choose
p_{5}}{q_{j}\choose p_{6}}{q_{k}\choose p_{6}}$ $\displaystyle\times$
$\displaystyle(-1)^{u+v/2+n_{j}+m_{j}+q_{j}+n_{k}+m_{k}+q_{k}}\times(\frac{1}{2})^{u}\times
x^{u+1/2}$ $\displaystyle\times$
$\displaystyle\frac{\Gamma(\frac{1+2u+v}{2})\Gamma(1+u)\Gamma(\frac{1+v}{2})}{\Gamma(\frac{3+2u+v}{2})}$
$\displaystyle\times$
${}_{2}F_{1}(1+u,\frac{1+2u+v}{2};\frac{3+2u+v}{2};1-x)\,,$

where we have defined $u=m_{i}+m_{j}+n_{l}+n_{k}-(p_{1}+p_{2}+p_{4}+p_{5})$,
$v=(q_{i}+q_{l}+q_{j}+q_{k})-2(p_{3}+p_{6})$,
$R_{L}=(m_{i}+m_{j})-(n_{i}+n_{j})=-(L_{z,i}+L_{z,j})$,
$R_{R}=(m_{l}+m_{k})-(n_{l}+n_{k})=-(L_{z,l}+L_{z,k})$,
$x\equiv\omega_{z}/\omega_{h}$, and ${}_{2}F_{1}$ is the hypergeometric
function. The $\delta$-functions $\delta_{q_{i}+q_{j}+q_{l}+q_{k},\rm{even}}$
and $\delta_{R_{L},R_{R}}$ in the formulation ensure the conservation of the
parity with respect to $z$-axis and the $z$-component of angular momentum of
system $L_{z}$, respectively. The formulation of Eq. (2.2) is reexamined by
computing the Coulomb integral numerically.

###  2.3 Exact diagonalization

The energy spectrum of an interacting two-electron NWQD is calculated using
the standard numerical exact diagonalization technique [8]. The numerically
exact results are obtained by increasing the numbers of chosen single electron
orbital basis and the corresponding two-electron configurations until a
numerical convergence is achieved. In our full configuration interaction (FCI)
calculations, we chose the typical orbital number from $20$ to $26$ and the
number of corresponding configurations is from $190$ to $325$. In order to
highlight the Coulomb correlations, we also carry out the partial CI (PCI)
calculations in which only the lowest energy $N_{e}$ configuration is taken
and compare the PCI results with those obtained from the FCI calculations.

##  3 Results and discussion

Figure 3: Correlated two-electron energy spectrum as a function of magnetic
field in a NWQD with diameter $L_{x}=50$ nm and aspect ratio $a=3$.

Figure 3 presents the FCI result of magneto-energy spectrum of two interacting
electrons in a NWQD with $a=3$. The ST transition of the two-electron ground
state is shown to happen as $B_{\rm ST}\sim 0.9$ T. As the applied magnetic
field is weak, the spin Zeeman splitting is small and the two electrons mostly
doubly fill the lowest S-orbital. With increasing magnetic field increases,
the energy difference between triplet and singlet states of the two electrons
decreases because of increasing spin Zeeman and exchange energies, both of
which energetically favor the triplet states $|T^{+}\rangle$. As the applied
field is higher than $B_{c}\sim 0.9$T, the ground state of two electrons
transit from the singlet state to the triplet one.

Figure 4: (a) Singlet-triplet splitting $\Delta_{{\rm ST}^{i}}$ as a function
of aspect ratio $a$ with $B=1$ T. (b) The spin phase diagram for electrons
making singlet-triplet transition with respect to magnetic field $B$ versus
aspect ratio $a$.

Now we turn to study the spin singlet-triplet splitting as a function of $a$,
defined by $\Delta_{{\rm ST}^{i}}\equiv E_{T^{i}}-E_{S}$, with $i=-$, $0$, and
$+$ corresponding to the $T^{-}$, $T^{0}$, and $T^{+}$ triplet states,
respectively. In Figure 4(a), we show the $\Delta_{{\rm ST}^{i}}$ as a
function of aspect ratio $a$ under a fixed magnetic field $B=1$ T. In the non-
interacting case, $\Delta^{0}_{\rm ST^{i}}$ are shown to decrease
monotonically with increasing aspect ratio $a$. Since only $T^{+}$ energy is
decreased by spin-Zeeman term, the ST transition could only occur between $S$
and $T^{+}$ states. That is, only $\Delta_{{\rm ST}^{+}}$ crosses zero as $a$
is very large, while $\Delta_{{\rm ST}^{0}}$ remains positive always. Thus
below we shall only consider $\Delta_{{\rm ST}^{+}}$ for the discussion of ST
transition. The non-interacting ST splitting can be derived as
$\Delta^{0}_{\rm ST^{+}}=\hbar\omega_{0}/a^{2}-2E_{\rm Z}$, explicitly showing
the quadratic decrease of $\Delta^{0}_{\rm ST^{+}}$ with respect to $a$.
Accordingly, in the non-interacting picture, the critical aspect ratio $a_{\rm
ST}$ where the ST transition occurs is predicted as $a_{\rm
ST}=\sqrt{\hbar\omega_{0}/g^{*}\mu_{B}B}$.

However, the PCI calculation predicts a much smaller value of critical aspect
ratio $a_{\rm ST}=2.5$. In the PCI result, the ST splitting is substantially
reduced by the energy reduction of the T state due to the reduced direct
Coulomb interaction and the negative exchange interaction between the two
electrons in the state. The FCI calculation shows $a_{\rm ST}=2.9$, as
indicated by the dashed vertical line in Figure 4(a). In fact, the difference
in the values of $\Delta_{\rm ST^{+}}$ obtained from the PCI and FCI
calculations increases as $a$ increases. This indicates that the Coulomb
correlation effect tends to increase the ST splitting again and becomes even
more pronounced in long NWQD with high $a$.

Figure 4(b) shows the calculated spin phase diagram of two-electron NWQDs with
respect to the aspect ratio $a$ and applied magnetic field $B$. The spin
singlet and triplet phases, appearing in the low $a$-$B$ and high $a$-$B$
regimes, respectively, are distinguished by the curve of $B_{\rm ST}$ which
show a monotonic decrease with $a$. For noninteracting electrons, the critical
magnetic field can be derived as $B_{\rm ST}=\hbar\omega_{0}/g\mu_{B}a^{2}$,
showing a quadratic decay with $a$.

In comparison with the non-interacting cases, the PCI calculations obtain the
$B_{\rm ST}$ that is significantly reduced and goes to zero for $a>2.9$. In
the one-configuration approximation used in the PCI calculation, the ST
splitting is given by $\Delta_{\rm ST^{+}}\approx\Delta^{0}_{\rm
ST^{+}}+\Delta_{\rm ST}^{\rm dir}-V_{\rm T}^{\rm ex}$, where $\Delta^{0}_{\rm
ST^{+}}$ is the ST splitting in the non-interacting cases, $\Delta_{\rm
ST}^{\rm dir}\equiv V_{\rm T}^{\rm dir}-V_{\rm S}^{\rm dir}<0$ is the direct
energy difference between the triplet and the singlet states, and $V_{\rm
T}^{\rm ex}$ is the exchange energy between electrons in the $T^{+}$ state.
Accordingly, we obtain $B_{\rm ST}=(\hbar\omega_{0}/a^{2}+\Delta_{\rm ST}^{\rm
dir}-V_{\rm T}^{\rm ex})/g^{\ast}\mu_{B}$. In the large aspect ratio regime,
the negative $\Delta_{\rm ST}^{\rm dir}$ and $V_{\rm T}^{\rm ex}$ reduce the
$E_{{\rm T}^{+}}$ and $B_{\rm ST}=0$ results for $a>3$. However, the FCI
calculation predict larger and always positive $B_{ST}$. In fact, as
increasing $a$, the relative strength of electronic Coulomb correlations
increases because of reduced $\hbar\omega_{z}$ and strong configuration
interactions. Such $a$-engineered Coulomb correlations energetically favor the
singlet two-electron states and result in the non-zero $B_{ST}$ in the high
aspect ratio regime.

##  4 Summary

In conclusion, a configuration interaction (CI) theory is developed for
studying the magneto-energy spectra and the singlet-triplet transitions of
two-electron NWQDs with longitudinal magnetic field $B$ and tunable aspect
ratio $a$. For short NWQDs of low aspect ratio $a<3$, the ST transition
behaviors are dominated by the spin Zeeman, Coulomb direct and exchange
energies, and can be well studied by using PCI calculation. However, our FCI
calculations show the increasing importance of Coulomb correlations in long
NWQDs with increasing aspect ratio $a$ over $3$. The FCI calculation present
the spin phase diagram of a two-electron NWQD which are highly dependent on
$a$, and suggests the controllability of singlet or triplet spin states by
changing the aspect ratio of NWQD.

##  5 Acknowledgment

This work was supported in part by the National Science Council of the
Republic of China through Contracts No. NSC95-2112-M-009-033-MY3 and No.
NSC97-2112-M-239-003-MY3.

## References

  * [1] S. M. Reimann, M. Manninen, Rev. Mod. Phys. 74 (2002) 1283.
  * [2] M. T. Bj$\rm\ddot{o}$rk, et al., 4 (2004) 1621.
  * [3] C. Fasth, et al., Phys. Rev. Lett. 98 (2007) 266801.
  * [4] Y. T. Chen, Few-Electron Theory of Semiconductor Nanocrystal and Nanorod Systems, MSc thesis submmitted to National Chiao Tung University, 2006.
  * [5] P. Hawrylak, Solid State Comm. 88 (1993) 475.
  * [6] M. T. Bj$\rm\ddot{o}$rk, et al., Phys. Rev. B 72 (2005) 201307.
  * [7] M. T. Bj$\rm\ddot{o}$rk, et al., Appl. Phys. Lett. 81 (2002) 4458.
  * [8] A. Wensauer, M. Korkusinski, P. Hawrylak, Solid State Comm. 130 (2003) 1155.