# PPG to ECG Signal Translation for
Continuous Atrial Fibrillation Detection via
Attention-based Deep State-Space Modeling

Khuong Vo  Department of Computer Science, University of California, Irvine
<EMAIL_ADDRESS>Mostafa El-Khamy  Samsung Device Solutions Research America
Yoojin Choi  Samsung Device Solutions Research America

###### Abstract

An electrocardiogram (ECG or EKG) is a medical test that measures the heart’s
electrical activity. ECGs are often used to diagnose and monitor a wide range
of heart conditions, including arrhythmias, heart attacks, and heart failure.
On the one hand, the conventional ECG requires clinical measurement, which
restricts its deployment to medical facilities. On the other hand, single-lead
ECG has become popular on wearable devices using administered procedures. An
alternative to ECG is Photoplethysmography (PPG), which uses non-invasive,
low-cost optical methods to measure cardiac physiology, making it a suitable
option for capturing vital heart signs in daily life. As a result, it has
become increasingly popular in health monitoring and is used in various
clinical and commercial wearable devices. While ECG and PPG correlate
strongly, the latter does not offer significant clinical diagnostic value.
Here, we propose a subject-independent attention-based deep state-space model
to translate PPG signals to corresponding ECG waveforms. The model is highly
data-efficient by incorporating prior knowledge in terms of probabilistic
graphical models. Notably, the model enables the detection of atrial
fibrillation (AFib), the most common heart rhythm disorder in adults, by
complementing ECG’s accuracy with continuous PPG monitoring. We evaluated the
model on 55 subjects from the MIMIC III database. Quantitative and qualitative
experimental results demonstrate the effectiveness and efficiency of our
approach.

##  1 Introduction

The measurement of the electrical activity generated by an individual’s heart,
known as an electrocardiogram (ECG), typically requires the placement of
several electrodes on the body. ECG is considered the preferred method for
monitoring vital signs and for the diagnosis, management, and prevention of
cardiovascular diseases (CVDs) [1, 2], which are a leading cause of death
globally, accounting for approximately 32% of all deaths in 2017 according to
Global Burden of Disease reports [3]. It has also been demonstrated that
sudden cardiac arrests are becoming more prevalent in young individuals,
including athletes [4]. Regular ECG monitoring has been found to be beneficial
for the early identification of CVDs [5]. Among heart diseases, atrial
fibrillation (AFib) is adults’ most common rhythm disorder. Identifying AFib
at an early stage is crucial for both primary and secondary prevention of
cardioembolic stroke, as it is the leading risk factor for this type of stroke
[6]. Advancements in electronics, wearable technologies, and machine learning
have made it possible to record ECGs more easily and accurately, and to
analyze large amounts of data more efficiently. Despite these developments,
there are still challenges associated with continuously collecting high-
quality ECG data over an extended period, particularly in everyday life
situations. The 12-lead ECG, considered the clinical gold standard, and
simpler versions, such as the Holter ECG, can be inconvenient and bulky due to
the need to place multiple electrodes on the body, which can cause discomfort.
Additionally, the signals may degrade over time as the impedance between the
skin and electrodes changes. Consumer-grade products like smartwatches have
developed solutions to address these issues. However, these products require
users to place their fingers on the watch to form a closed circuit, which
makes continuous monitoring impossible.

One potential solution to these issues is to use a mathematical method to
derive ECG data from an alternative, highly correlated, non-invasive signal,
such as the photoplethysmogram (PPG), which can be easily acquired using
various wearable devices, including smartwatches. PPG is more convenient,
cost-effective, and user-friendly. PPG has been increasingly adopted in
consumer-grade devices. This technique involves the use of a light source,
usually an LED, and a photodetector to measure the changes in light absorption
or reflection as blood flows through the tissue. ECG and PPG signals are
inherently correlated as both are influenced by the same underlying cardiac
activity, namely the depolarization and repolarization of the heart. These
contractions lead to changes in peripheral blood volume, which are measured by
PPG. Although there are established standards for interpreting ECG for
clinical diagnosis, the use of PPG is still mostly limited to measuring heart
rate and oxygen saturation [7]. By translating PPG to ECG signals, clinical
diagnoses of cardiac diseases and anomalies could be made in real-time. Figure
1 shows the relationship between ECG and PPG waveforms.

Figure 1: ECG and PPG waveforms. The pre-ejection period (PEP) is the time
elapsed between the electrical depolarization of the left ventricle and the
beginning of ventricular ejection. Pulse transit time (PTT) is defined as the
period from a relatively proximal site (e.g., arm) to a distal site (e.g.,
finger) or between two distal sites (e.g., figure and toe). The pulse arrival
time (PAT) is the time it takes for the pulse to travel from the heart to a
peripheral artery. The PAT interval includes the PTT interval plus the PEP.

Few research works attempted to synthesize ECG from PPG signals. In [8], a
machine learning-based approach was proposed to estimate ECG parameters,
including the RR, PR, QRS, and QT intervals, using time and frequency domain
features extracted from a fingertip PPG signal. Besides, [9, 10] proposed
models to reconstruct the entire ECG signal from PPG in the frequency domain.
However, the performance of these approaches relied on cumbersome algorithms
for feature crafting. With the recent advances of deep learning, recent works
[11, 12, 13] leveraged neural networks’ expressiveness and structural
flexibility to build end-to-end PPG-to-ECG algorithms. There are data-hungry
problems and non-robustness, as deterministic models do not explicitly model
the underlying sequential structures of the data. Additionally, complex deep
learning models cannot run efficiently on resource-constrained devices (e.g.,
wearables) due to their high computational intensity, which poses a critical
challenge for real-world deployment [14].

To address these challenges, we propose a robust, efficient deep probabilistic
model to accurately estimate ECG waveforms from raw PPG. The contributions of
this work are three-fold:

  * •

We present a generative model incorporating prior knowledge about the data
structures that enable data-efficient learning. Specifically, we develop a
sequential deep generative model combined with a state-space model augmented
by an attention mechanism.

  * •

The model is inherently robust to noise because of its probabilistic nature.
We demonstrate it by evaluating the model on data corrupted with Gaussian and
baseline wandering noise, mimicking real-world scenarios.

  * •

Experimental results demonstrate the success of our model not only on healthy
subjects but also on subjects with AFib, achieving an AFib detection
performance of PR-AUC 0.986. The model is trained only on the first 20% of
each record and tested on the remaining 80%.

The rest of our paper is organized as follows. Section 2 presents the
dynamical model to translate PPG to ECG signals and the latent factors of data
variations. In Section 3, experimental data are presented, and results on
signal translation and Afib detection performance are discussed. Finally,
Section 4 wraps up the paper.

##  2 Methodology

###  2.1 Probabilistic Modeling of ECG from PPG signals

We are given a dataset
$\mathcal{D}:=\left\\{\left(\mathbf{x}^{1},\mathbf{y}^{1}\right),\ldots,\left(\mathbf{x}^{N},\mathbf{y}^{N}\right)\right\\}$
with the $i$-th observation $\mathbf{y}^{i}\in$ $\mathbb{R}^{n_{y}}$, i.e.,
ECG signals of $n_{y}$ time samples, depending on $\mathbf{x}^{i}\in$
$\mathbb{R}^{n_{x}}$, i.e., PPG signals of $n_{x}$ time samples. Throughout
the paper, superscript $i$ is omitted when we refer to only one sequence or
when it is clear from the context.

We aim to learn a generative process with a latent-variable model comprising
of a parametric non-linear Gaussian prior over latents
$p_{\theta_{z}}(\mathbf{z}\mid\mathbf{x})$ and likelihood
$p_{\theta_{y}}(\mathbf{y}\mid\mathbf{z},\mathbf{x})$. The learning process
minimizes a divergence between the true data-generating distribution and the
model w.r.t $\theta$:

$\begin{aligned}
&\underset{\theta}{\arg\min}\mathrm{KL}\left(p_{\mathcal{D}}(\mathbf{y}\mid\mathbf{x})\|p_{\theta}(\mathbf{y}\mid\mathbf{x})\right)\\\
&=\underset{\theta}{\arg\max}\mathbb{E}_{p_{\mathcal{D}}(\mathbf{y}\mid\mathbf{x})}\left[\log
p_{\theta}(\mathbf{y}\mid\mathbf{x})\right]\end{aligned}\ $ (1)

where $p_{\theta}\left(\mathbf{y}\mid\mathbf{x}\right)=\int
p_{\theta_{y}}\left(\mathbf{y}\mid\mathbf{z},\mathbf{x}\right)p_{\theta_{z}}\left(\mathbf{z}\mid\mathbf{x}\right)d\mathbf{z}$
is the conditional likelihood/evidence of data point $\mathbf{y}$ given
condition $\mathbf{x}$, approximated by averaging over the latent
$\mathbf{z}$.

Nevertheless, estimating $p_{\theta}(\mathbf{y}\mid\mathbf{x})$ is typically
intractable. This issue can be mitigated by introducing a parametric inference
model $q_{\phi}(\mathbf{z}\mid\mathbf{x},\mathbf{y})$ to construct a
conditional variational evidence lower bound on the conditional log-likelihood
$\log p_{\theta}(\mathbf{y}\mid\mathbf{x})$ as follows

$\displaystyle\mathcal{L}(\mathbf{x},\mathbf{y};\theta,\phi)$ (2)
$\displaystyle\triangleq\log
p_{\theta}(\mathbf{y}\mid\mathbf{x})-\operatorname{KL}\left(q_{\phi}(\mathbf{z}\mid\mathbf{x},\mathbf{y})\|p_{\theta}(\mathbf{z}\mid\mathbf{x},\mathbf{y})\right)$
$\displaystyle=\mathbb{E}_{q_{\phi}(\mathbf{z}\mid\mathbf{x},\mathbf{y})}\left[\log
p_{\theta_{y}}(\mathbf{y}\mid\mathbf{z},\mathbf{x})\right]-\operatorname{KL}\left(q_{\phi}(\mathbf{z}\mid\mathbf{x},\mathbf{y})\|p_{\theta_{z}}(\mathbf{z}\mid\mathbf{x})\right)$

Taking the likelihood model
$p_{\theta_{y}}(\mathbf{y}\mid\mathbf{z},\mathbf{x})$ to be a decoder, the
latent inference model $q_{\phi}(\mathbf{z}\mid\mathbf{x},\mathbf{y})$ to be
an encoder, and the prior model $p_{\theta_{z}}(\mathbf{z}\mid\mathbf{x})$, a
conditional variational autoencoder (CVAE) [15, 16] considers this objective
from a deep probabilistic autoencoder perspective. Here $\theta$ and $\phi$
are neural network parameters, and learning takes place via stochastic
gradient ascent using unbiased estimates of
$\nabla_{\theta,\phi}\frac{1}{n}\sum_{i=1}^{n}\mathcal{L}\left(\mathbf{x}^{i},\mathbf{y}^{i};\theta_{z},\theta_{y},\phi\right)$.

###  2.2 State-Space Modeling of ECG from PPG Signals

In the previous section, we consider the networks that process the entire time
series as a whole, which do not explicitly model the underlying sequential
natures of the data. This may lead to resource-inefficient learning. Here,
propose to address the problems by leveraging the quasi-periodic nature of the
physiological signals.

####  2.2.1 ECG Generative (Decoding) Process from PPG

We consider non-linear dynamical systems with observations
$\mathbf{y}_{t}\in\mathbb{R}^{n_{rr}}$, i.e., RR intervals or the time elapsed
between two successive R peaks on the ECG, depending on control inputs
$\mathbf{x}_{t}\in\mathbb{R}^{n_{pp}}$, i.e., PP intervals or the time elapsed
between two successive systolic peaks on the PPG. We choose the peaks to
segment the signals as they are the most robust features. Corresponding
discrete-time sequences of length $T$ are denoted as
$\mathbf{y}_{1:T}=\left(\mathbf{y}_{1},\mathbf{y}_{2},\ldots,\mathbf{y}_{T}\right)$
and
$\mathbf{x}_{1:T}=\left(\mathbf{x}_{1},\mathbf{x}_{2},\ldots,\mathbf{x}_{T}\right)$.

Given an input PPG $\mathbf{x}_{1:T}$, we are interested in a probabilistic
model $p\left(\mathbf{y}_{1:T}\mid\mathbf{x}_{1:T}\right)$. Formally, we
assume the graphical model

$p\left(\mathbf{y}_{1:T}\mid\mathbf{x}_{1:T}\right)=\int
p\left(\mathbf{y}_{1:T}\mid\mathbf{z}_{1:T},\mathbf{x}_{1:T}\right)p\left(\mathbf{z}_{1:T}\mid\mathbf{x}_{1:T}\right)\mathrm{d}\mathbf{z}_{1:T}$
(3)

where $\mathbf{z}_{1:T}$ denotes the corresponding latent sequence. That is,
we assume a generative model with an underlying latent dynamical system with
emission model
$p\left(\mathbf{y}_{1:T}\mid\mathbf{z}_{1:T},\mathbf{x}_{1:T}\right)$ and
transition model $p\left(\mathbf{z}_{1:T}\mid\mathbf{x}_{1:T}\right)$.

To obtain state-space models, we impose assumptions on state transition and
emission models, as shown in Figure 2:

$\displaystyle p\left(\mathbf{z}_{1:T}\mid\mathbf{x}_{1:T}\right)$
$\displaystyle=\prod_{t=0}^{T-1}p\left(\mathbf{z}_{t+1}\mid\mathbf{z}_{t},\mathbf{x}_{1:T}\right)$
(4) $\displaystyle
p\left(\mathbf{y}_{1:T}\mid\mathbf{z}_{1:T},\mathbf{x}_{1:T}\right)$
$\displaystyle=\prod_{t=1}^{T}p\left(\mathbf{y}_{t}\mid\mathbf{z}_{t}\right)$
(5)

Equations (4) and (5) assume that the current state $\mathbf{z}_{t}$ contains
all necessary information about the current observation $\mathbf{y}_{t}$, as
well as the next state $\mathbf{z}_{t+1}$ (given the current control input
$\mathbf{x}_{t}$). That is, as opposed to observations, $\mathbf{z}_{t}$
exhibits Markovian behavior.

Figure 2: The graphical model for ECG translation from PPG. Shaded nodes
represent observed variables. Clear nodes represent latent variables. Diamond
nodes denote deterministic variables. Variables
$\mathbf{x}_{t},\mathbf{y}_{t}$, and $\mathbf{c}_{t}$ represent PP intervals,
RR intervals, and context vectors, respectively. $\alpha_{t,i}$ are attention
weights defines how well two intervals $\mathbf{x}_{i}$ and $\mathbf{y}_{t}$
are aligned. The attention mechanism is shown as an example at time step 2.

In contrast to the DKF model of [17, 18], our model takes into account an
entire input signal $\mathbf{x}_{1:T}$ for each output $\mathbf{y}_{t}$ via an
attention mechanism [19]. Note that there usually exists misalignments between
the PPG and ECG cycles. Therefore, it is difficult to construct optimal and
exact sample pairs. This attention mechanism not only helps to add more
context for generating ECG segments but also to cope with the misalignment
problem.

Let us define $\mathbf{c}_{t}$ a sum of features of the input sequence (PP
intervals), weighted by the alignment scores:

$\displaystyle\mathbf{c}_{t}$
$\displaystyle=\sum_{i=1}^{T}\alpha_{t,i}\boldsymbol{x}_{i}$ (6)
$\displaystyle\alpha_{t,i}$
$\displaystyle=\frac{\exp\left(\mathbf{s}\left(\mathbf{z}_{t-1},\boldsymbol{x}_{i}\right)\right)}{\sum_{i^{\prime}=1}^{n}\exp\left(\mathbf{s}\left(\mathbf{z}_{t-1},\boldsymbol{x}_{i^{\prime}}\right)\right)}$
(7)

The alignment function $\mathbf{s}$ assigns a score $\alpha_{t,i}$ to the pair
of input at position $i$ and output at position $t$,
($\mathbf{x}_{i},\mathbf{y}_{t}$), based on how well they match. The set of
$\alpha_{t,i}$ are weights defining how much of each source segment should be
considered for each output interval.

Both state transition (prior) and emission models are non-linear Gaussian
transformations:

$\displaystyle
p_{\theta_{z}}(\mathbf{z}_{t+1}\mid\mathbf{z}_{t},\mathbf{x}_{1:T})$
$\displaystyle=\mathcal{N}(\mathbf{z}_{t+1}\mid\boldsymbol{\mu}_{\theta_{z}}(\mathbf{z}_{t},\mathbf{c}_{t+1}),\boldsymbol{\sigma}^{2}_{\theta_{z}}(\mathbf{z}_{t},\mathbf{c}_{t+1}));$
(8) $\displaystyle p_{\theta_{y}}(\mathbf{y}_{t}\mid\mathbf{z}_{t})$
$\displaystyle=\mathcal{N}(\mathbf{y}_{t}\mid\boldsymbol{\mu}_{\theta_{y}}(\mathbf{z}_{t}),\mathbf{I})$
(9)

where $\boldsymbol{\mu}$ and $\boldsymbol{\sigma}^{2}$ are the parametrized
means and diagonal covariance matrices of the normal distributions
$\mathcal{N}$, $\mathbf{I}$ is the identity covariance matrix.

####  2.2.2 Latent State Inference (Posterior Encoding) Process

Figure 3: The graphical model at latent state inference time. Variables
$\mathbf{y}_{t},\mathbf{h}_{t},\mathbf{g}_{t}$, and $\mathbf{z}_{t}$ represent
respectively RR intervals, backward recurrent states, forward recurrent
states, and latent states.

Unlike a deterministic translation model, the process needs to find meaningful
probabilistic embeddings of the ECG segments in the latent space. We want to
identify the structure of the posterior distribution
$p_{\theta}\left(\mathbf{z}_{1:T}\mid\mathbf{y}_{1:T}\right)$. Notice that we
made a design choice to perform inference using only $\mathbf{y}_{1:T}$. We
chose this with the conditional independence assumption that the PPG segments
do not provide more information than ECG segments alone. Let us first apply
the chain rule that enables us to rewrite this distribution as follows

$p_{\theta}\left(\mathbf{z}_{1:T}\mid\mathbf{y}_{1:T}\right)=p_{\theta}\left(\mathbf{z}_{1}\mid\mathbf{y}_{1:T}\right)\prod_{t=1}^{T-1}p_{\theta}\left(\mathbf{z}_{t+1}\mid\mathbf{z}_{1:t},\mathbf{y}_{1:T}\right)$
(10)

Then, d-separation [20] can be used to simplify each term of the product. The
structure presented in Figure 2 shows that the $\mathbf{z}_{t}$ node blocks
all information coming from the past and flowing to $\mathbf{z}_{t+1}$ (i.e.,
$\mathbf{z}_{1:t-1}$ and $\left.\mathbf{y}_{1:t}\right)$. In other words,
$\mathbf{z}_{t}$ has accumulated this past information or is a summary of this
information. We thus have
$p_{\theta}\left(\mathbf{z}_{t+1}\mid\mathbf{z}_{1:t},\mathbf{y}_{1:T}\right)=p_{\theta}\left(\mathbf{z}_{t+1}\mid\mathbf{z}_{t},\mathbf{y}_{t+1:T}\right)$,
and therefore

$p_{\theta}\left(\mathbf{z}_{1:T}\mid\mathbf{y}_{1:T}\right)=p_{\theta}\left(\mathbf{z}_{1}\mid\mathbf{y}_{1:T}\right)\prod_{t=1}^{T-1}p_{\theta}\left(\mathbf{z}_{t+1}\mid\mathbf{z}_{t},\mathbf{y}_{t+1:T}\right)$
(11)

The variational approximation of the posterior factorizes according to the
structure of the exact posterior as in Figure 3:

$q_{\phi}\left(\mathbf{z}_{1:T}\mid\mathbf{y}_{1:T}\right)=q_{\phi}\left(\mathbf{z}_{1}\mid\mathbf{y}_{1:T}\right)\prod_{t=1}^{T-1}q_{\phi}\left(\mathbf{z}_{t+1}\mid\mathbf{z}_{t},\mathbf{y}_{t+1:T}\right)$
(12)

where

$q_{\phi}(\mathbf{z}_{t+1}\mid\mathbf{z}_{t},\mathbf{y}_{t+1:T})=\mathcal{N}(\mathbf{z}_{t+1}\mid\boldsymbol{\mu}_{\phi}(\mathbf{z}_{t},\mathbf{y}_{t+1:T}),\boldsymbol{\sigma}^{2}_{\phi}(\mathbf{z}_{t},\mathbf{y}_{t+1:T}))$
(13)

####  2.2.3 Training Process

The objective function becomes a timestep-wise conditional variational lower
bound:

$\displaystyle\log
p_{\theta}(\mathbf{y}\mid\mathbf{x})\geq\mathcal{L}(\mathbf{x},\mathbf{y};\theta_{y},\theta_{z},\phi)\triangleq$
(14)
$\displaystyle\sum_{t=1}^{T}\underset{q_{\phi}\left(\mathbf{z}_{t}\mid\mathbf{y}_{t:T}\right)}{\mathbb{E}}[\overbrace{\log\underbrace{p_{\theta_{y}}\left(\mathbf{y}_{t}\mid\mathbf{z}_{t}\right)}_{\text{emission
model}}}^{\text{reconstruction}}]$
$\displaystyle-\beta\overbrace{\operatorname{KL}\left(q_{\phi}\left(\mathbf{z}_{1}\mid\mathbf{y}_{1:T}\right)\|p_{\theta_{z}}\left(\mathbf{z}_{1}\mid\mathbf{x}_{1:T}\right)\right)}^{\text{regularization}}$
$\displaystyle-\beta\sum_{t=1}^{T-1}\underset{q_{\phi}\left(\mathbf{z}_{t}\mid\mathbf{y}_{t:T}\right)}{\mathbb{E}}[\overbrace{\operatorname{KL}(\underbrace{q_{\phi}\left(\mathbf{z}_{t+1}\mid\mathbf{z}_{t},\mathbf{y}_{t:T}\right)}_{\text{posterior
inference
model}}||\underbrace{p_{\theta_{z}}\left(\mathbf{z}_{t+1}\mid\mathbf{z}_{t},\mathbf{x}_{1:T}\right)}_{\text{prior
transition model}})}^{\text{regularization}}]$

where $\beta$ controls the regularization strength. During training, the
Kullback–Leibler (KL) losses in the regularization terms "pull" the posterior
distributions (which encode EEG segments) and the prior distributions (which
embed PPG segments) towards each other. As in the CVAE, we learn the
generative and inference models jointly by maximizing the conditional
variational lower bound with respect to their parameters.

###  2.3 Neural Network Parametrization

Let us denote $\mathbf{W}$, $\mathbf{v}$, and $\mathbf{b}$ the weight matrices
of the neural networks.

Score Model: The alignment score $\alpha$ in Equation 7 is parametrized by a
feed-forward network with a single hidden layer, and this network is jointly
trained with other parts of the model. The score function $\mathbf{s}$ is
therefore in the following form:

$\mathbf{s}\left(\boldsymbol{z}_{t-1},\boldsymbol{x}_{i}\right)=\mathbf{v}_{s}^{\top}\tanh\left(\mathbf{W}_{s}\left[\mathbf{z}_{t-1};\mathbf{W}_{x}\boldsymbol{x}_{i}\right]+\mathbf{b}_{s}\right)$
(15)

Prior Transition Model: We parametrize the transition function in Equation 8
from $z_{t}$ to $z_{t+1}$ using a Gated Transition Function as in [18]. The
model is flexible in choosing a non-linear transition for some dimensions
while having linear transitions for others. The function is parametrized as
follows:

$\displaystyle\mathbf{g}_{t}=\operatorname{sigmoid}\left(\mathbf{W}_{g_{3}}\operatorname{ReLU}\left(\mathbf{W}_{g_{2}}\operatorname{ReLU}\left(\mathbf{W}_{g_{1}}\left[\mathbf{z}_{t};\mathbf{c}_{t+1}\right]+\mathbf{b}_{g_{1}}\right)+\mathbf{b}_{g_{2}}\right)+\mathbf{b}_{g_{3}}\right)$
(16)
$\displaystyle\mathbf{d}_{t}=\mathbf{W}_{d_{3}}\operatorname{ReLU}\left(\mathbf{W}_{d_{2}}\operatorname{ReLU}\left(\mathbf{W}_{d_{1}}\left[\mathbf{z}_{t};\mathbf{c}_{t+1}\right]+\mathbf{b}_{d_{1}}\right)+\mathbf{b}_{d_{2}}\right)+\mathbf{b}_{d_{3}}$
$\displaystyle\boldsymbol{\mu}_{\theta_{z}}(\mathbf{z}_{t},\mathbf{c}_{t+1})=\left(1-\mathbf{g}_{t}\right)\odot\left(\mathbf{W}_{\mu_{z}}\left[\mathbf{z}_{t};\mathbf{c}_{t+1}\right]+\mathbf{b}_{\mu_{z}}\right)+\mathbf{g}_{t}\odot\mathbf{d}_{t}$
$\displaystyle\boldsymbol{\sigma}^{2}_{\theta_{z}}(\mathbf{z}_{t},\mathbf{c}_{t+1})=\operatorname{softplus}\left(\mathbf{W}_{\sigma_{z}^{2}}\operatorname{ReLU}\left(\mathbf{d}_{t}\right)+\mathbf{b}_{\sigma_{z}^{2}}\right)$

where $\mathbb{I}$ denotes the identity function, and $\odot$ denotes element-
wise multiplication.

Emission Model: We parameterize the emission function in Equation 9 using a
two-hidden layer network as:

$\displaystyle\boldsymbol{\mu}_{\theta_{y}}\left(\mathbf{z}_{t}\right)=\mathbf{W}_{e_{3}}\operatorname{ReLU}\left(\mathbf{W}_{e_{2}}\operatorname{ReLU}\left(\mathbf{W}_{e_{1}}\mathbf{z}_{t}+\mathbf{b}_{e_{1}}\right)+\mathbf{b}_{e_{2}}\right)+\mathbf{b}_{e_{3}}$
(17)

Posterior Inference Model: We use a Bi-directional Gated Recurrent Unit
network [21] (GRU) to process the sequential order of RR intervals backward
from $\mathbf{y}_{T}$ to $\mathbf{y}_{t+1}$ and forward from
$\mathbf{y}_{t+1}$ to $\mathbf{y}_{T}$. The GRUs are denoted here as
$\boldsymbol{h}_{t}=\operatorname{GRU}\left(\mathbf{W}_{y}\boldsymbol{y}_{T},\ldots,\mathbf{W}_{y}\boldsymbol{y}_{t+1}\right)$
and
$\boldsymbol{g}_{t}=\operatorname{GRU}\left(\mathbf{W}_{y}\boldsymbol{y}_{t+1},\ldots,\mathbf{W}_{y}\boldsymbol{y}_{T}\right)$,
respectively. The hidden states of the GRUs parametrize the variational
distribution, which are combined with the previous latent states for the
inference in Equation 13 as follows:

$\displaystyle\mathbf{\tilde{h}}_{t}$
$\displaystyle=\frac{1}{3}\left(\tanh\left(\mathbf{W}_{h}\mathbf{z}_{t}+\mathbf{b}_{h}\right)+\mathbf{h}_{t}+\mathbf{g}_{t}\right)$
(18) $\displaystyle\boldsymbol{\mu}_{\phi}(\mathbf{z}_{t},\mathbf{y}_{t+1:T})$
$\displaystyle=\mathbf{W}_{\mu}\mathbf{\tilde{h}}_{t}+\mathbf{b}_{\mu}$
$\displaystyle\boldsymbol{\sigma}^{2}_{\phi}(\mathbf{z}_{t},\mathbf{y}_{t+1:T})$
$\displaystyle=\operatorname{softplus}\left(\mathbf{W}_{\sigma^{2}}\mathbf{\tilde{h}}_{t}+\mathbf{b}_{\sigma^{2}}\right)$

All the hidden layer sizes are 256, and the latent space sizes are 128. Input
and output segments at each timestep are of size 90. We use Adam [22] for
optimization, with a learning rate of 0.0008, exponential decay rates
$\beta_{1}$ = 0.9, and $\beta_{2}$ = 0.999. We train the models for 5000
epochs, with a minibatch size 128. We set the regularization hyperparameter
$\beta=0$ at the beginning of training and gradually increase it until
$\beta=1$ is reached at epoch 1250.

##  3 Implementation Scenarios and Experimental Results

###  3.1 Dataset

The MIMIC-III Waveform Database Matched Subset [23, 24] was used for the
experiments. The database contains recordings collected from patients at
various hospitals. Each session has multiple physiological signals, including
PPG and ECG signals, sampled at a frequency of 125 Hz. We used the records of
43 healthy subjects and 12 subjects having AFib, including 30 males and 25
females, 23-84 years old. Each record duration is 5 minutes. The first 48 s of
each record were used as the training set, the next 12 s as the validation
set, and the remaining 228 s as the test set. The preprocessing steps,
including filtering, alignment, and normalization, were done as described in
[25]. We applied HeartPy [26, 27] to identify peaks in PPG signals. Each long
signal is split into 4-s chunks. All peak-to-peak intervals were linear
interpolated to the length of 90 during training, which is the mean length of
the intervals on the training set. PP interval lengths were used as RR
interval lengths on translated ECG signals during testing. Alternatively, we
can apply padding instead of interpolation. We found that the interpolation
yields the best performance. This can be justified as PPG recordings are used
to analyze heart rate variability as an alternative to ECG [28, 29].
Artificial noise was added to the signals for the robustness evaluation.
Amplitudes of baseline noise signals are 0.3, 0.4, and 0.1, and the
frequencies are 0.3 Hz, 0.2 Hz, and 0.9 Hz, respectively. Gaussian noise of
standard deviation 0.3. Figure 4 shows an example of the preprocessed with
noise-added PPG-ECG waveform pairs.

Figure 4: A preprocessed PPG-ECG waveform pair with the added noise.

###  3.2 Evaluation Metrics

####  3.2.1 ECG Translation from PPG

Pearson’s correlation coefficient $\left(\rho\right)$ measures how much an
original ECG signal $\mathbf{y}_{1:T}$ and its reconstruction
$\hat{\mathbf{y}}_{1:T}$ co-vary:

$\rho=\frac{\left(\mathbf{y}_{1:T}-\bar{y}_{1:T}\right)^{\top}(\hat{\mathbf{y}}_{1:T}-\bar{\hat{y}}_{1:T})}{\left\|\mathbf{y}_{1:T}-\bar{y}_{1:T}\right\|_{2}\|\hat{\mathbf{y}}_{1:T}-\bar{\hat{y}}_{1:T}\|_{2}}$
(19)

Root Mean Squared Error (RMSE) measures the differences between the values of
the original signal and its reconstruction:

$\text{RMSE}=\frac{\left\|\mathbf{y}_{1:T}-\hat{\mathbf{y}}_{1:T}\right\|_{2}}{\sqrt{n_{y}}}$
(20)

Signal-to-Noise Ratio (SNR) compares the level of the desired signal to the
level of undesired noise:

$\text{SNR}=20\log\frac{\left\|\mathbf{y}_{1:T}\right\|_{2}^{2}}{\left\|\mathbf{y}_{1:T}-\hat{\mathbf{y}}_{1:T}\right\|_{2}^{2}}$
(21)

####  3.2.2 AFib Detection

Performance was measured by the Area under the Receiver Operating
Characteristic (ROC-AUC), the Area under the Precision-Recall Curve (PR-AUC),
and the F1 score. The PR-AUC is considered a better measure for imbalanced
data.

###  3.3 Implementation and Results

####  3.3.1 ECG Translation from PPG

|  Parameters |  Correlation |  RMSE (mV) |  SNR (dB)
---|---|---|---|---
Our model |  |  |  |
(healthy sub., |  |  |  |
clean signals) |  645,466 |  0.858 $\pm$ 0.174 |  0.07 $\pm$ 0.047 |  15.365 $\pm$ 11.053
Our model |  |  |  |
(healthy sub., |  |  |  |
noisy signals) |  645,466 |  0.847 $\pm$ 0.174 |  0.076 $\pm$ 0.049 |  13.887 $\pm$ 10.58
Our model |  |  |  |
(healthy & AFib sub.) |  645,466 |  0.804 $\pm$ 0.22 |  0.078 $\pm$ 0.05 |  12.261 $\pm$ 11.328
P2E-WGAN |  4,064,769 |  0.773 $\pm$ 0.242 |  0.091 $\pm$ 0.052 |  9.616 $\pm$ 9.252
LSTM (sub. dep.) |  5,451 |  0.766 $\pm$ 0.234 |  0.093 $\pm$ 0.053 |  8.189 $\pm$ 9.560
Table 1: ECG translation performance on healthy subjects of different models.
The top two rows show our model’s performance in clean and noisy dataset
settings (healthy subjects), while the third row shows the performance on both
the healthy and AFib dataset. If not specified, healthy subjects and clean
signals is the default setting. The LSTM model [25] is subject-dependent,
while the P2E-WGAN [11] and our model are subject-independent.

Table 1 shows the performance of our model and compares it with other models
in terms of the means and standard deviations of $\rho$, RMSE, and SNR. The
correlation between the signals generated by our model and the reference
signals is statistically strong, with the $\rho$ value of 0.858. Also, low
values of RMSE (0.07) and high SNR (15.365) show strong similarities between
them and reference ECG signals. The second row shows our model’s performance
on the noisy dataset. The negligible drop of the metrics from 0.858 to 0.847
($\rho$), 0.07 to 0.76 (RMSE), and 15.365 to 13.887 (SNR) demonstrates the
robustness of our model. We attribute this to the probabilistic nature of the
model that better handles the measurement noise. As expected, the model
performed worse on the AFib subjects because of the erratic patterns of AFib
signals (no visible P waves and an irregularly irregular QRS complex). We show
in the next section that the synthetic AFib signals are beneficial to the
downstream detection task.

The P2E-WGAN model [11], a 1D deep convolutional generative adversarial
network (4,064,769 parameters) for signal-to-signal translation, was recently
proposed to translate PPG to ECG signals from a large number of subjects.
P2E-WGAN achieved significantly lower performance than our model, requiring
almost six times the parameters. Our model is less affected when data is
scarce, which is common in healthcare. On the other hand, the LSTM model [25]
is a deep recurrent neural network that was also recently proposed and built
separately for each subject. Our model outperformed the LSTM model even
trained in the cross-subject setting. These results prove the effectiveness
and the efficiency of our proposed sequential data structure and confirm the
inverse inference strategy.

In Figure 6, translated ECG waveforms are plotted with respect to the
reference ECG waveforms of different heart rates. We can see that the model
closely reconstructed the waveforms and maintained their essential features.
Besides, we can be informed of the translation uncertainty by using a
posterior on the latent embedding to propagate uncertainty from the embedding
to the data. More specifically, with a distribution $p(\mathbf{z})$ on the
latent feature our predictions will be
$p_{\theta}\left(\mathbf{y}\mid\mathbf{x}\right)=\int
p_{\theta_{y}}\left(\mathbf{y}\mid\mathbf{z}\right)p_{\theta_{z}}\left(\mathbf{z}\mid\mathbf{x}\right)d\mathbf{z}$.
This capability makes the model more trustworthy and gives patients and
clinicians higher confidence in using it for medical diagnosis [30].

####  3.3.2 AFib Detection

|  Real ECG |  Translated ECG |
---|---|---|---
|  0.995 $\pm$ 0.006 |  0.99 $\pm$ 0.004 |
|  0.987 $\pm$ 0.013 |  0.986 $\pm$ 0.007 |
|  0.985 $\pm$ 0.009 |  0.944 $\pm$ 0.014 |
Real ECG |  30% missing |  50% missing |  70% missing
ROC-AUC |  0.983 $\pm$ 0.009 |  0.982 $\pm$ 0.016 |  0.958 $\pm$ 0.021
PR-AUC |  0.962 $\pm$ 0.015 |  0.957 $\pm$ 0.044 |  0.931 $\pm$ 0.041
F1 |  0.96 $\pm$ 0.019 |  0.929 $\pm$ 0.017 |  0.871 $\pm$ 0.037
Fusion |  30% missing |  50% missing |  70% missing
|  0.992 $\pm$ 0.006 |  0.99 $\pm$ 0.006 |  0.99 $\pm$ 0.009
|  0.986 $\pm$ 0.011 |  0.982 $\pm$ 0.012 |  0.981 $\pm$ 0.016
|  0.971 $\pm$ 0.01 |  0.969 $\pm$ 0.012 |  0.956 $\pm$ 0.046
Table 2: AFib detection performance. The performance on the translated ECG is
evaluated when the MINA model [31] is trained on real ECG but tested on
synthetic ECG. The fusion performance is when the MINA model is extended to
receive both real ECG and synthetic ECG inputs. x% random time samples are
omitted, simulating intermittent ECG recording, while synthetic ECG is always
available.

We further evaluated our model’s performance on the benefits of the translated
ECG for the AFib detection task. In order to do so, we used a state-of-the-art
AFib detection model, Multilevel Knowledge-Guided Attention (MINA) [31],
trained on the real ECG signals, each of 10 s, and tested against the
synthetic. It should be noted that any pre-trained AFib detection model can be
used in our pipeline. Table 2 reveals the mean detection performance of the
model on the translated ECG closely approximate to that of the real ECG, ROC-
AUC of 0.99 vs. 0.995, PR-AUC of 0.986 vs. 0.987, and F1 of 0.944 vs. 0.985.
This implies that our model allows the combined advantages of ECG’s rich
knowledge base and PPG’s continuous measurement.

Figure 5: Extended MINA model for AFib detections on both original and
translated ECG signals (model fusion).

Figure 5 presents our proposed model fusion method. We extended the MINA model
capability to receive both real and translated ECG signals by incorporating
the frequency channels of the translated into the model. This scenario is when
both ECG and PPG signals can be measured simultaneously. This setting requires
retraining the MINA model on the fused real and synthetic ECG signal dataset.
To simulate the real-life setting where ECG measurement is intermittent while
PPG input is continuous, we randomly zeroed out time samples with different
probabilities: 30%, 50%, and 70%. As shown in the middle and bottom results of
Table 2, the higher the discontinuity, the worse the performance for the
detection on real ECG, but the performance remains almost unchanged in the
fusion mode. The fusion model consistently outperformed the single-modality
model across the omission thresholds. Also, the model learns to utilize sparse
real ECG to marginally improve the performance against only translated ECG.
This suggests our model’s enhancement for the downstream task in real-time
AFib detection.

Another method for fusion is selection fusion. By using a priori information
on whether the ECG signal is available, the selection fusion will utilize the
ECG signal solely when available for AFib detection. However, when the ECG is
not available, the system will switch to AFib monitoring using our translated
ECG signals. As shown from Table 2, if ECG is available $T\%$ of time only,
one expects the average detection accuracy to be

$\text{Selection Fusion Performance}=T\%\times\mbox{Original ECG
perf.}+(1-T\%).\times\mbox{Translated ECG perf.}$ (22)

This selection fusion method can enable continuous AFib monitoring while
achieving satisfactory detection performance. For example, at $T\%=70\%$, AFib
monitoring with ECG available for only $30\%$ of the time will yield an
F1-score of 0.846, while one can expect selection-fusion to yield F1-score
$0.3\times 0.985+0.7\times 0.944=0.9563$. This concludes that continuous AFib
monitoring with selection fusion between available ECG and continuously
translated ECG signals is better than AFib monitoring with only real ECG
signals.

##  4 CONCLUSION

In this work, we present a novel attention-based deep state-space model to
generate ECG waveforms with PPG signals as inputs. The results demonstrate our
model has the potential to provide a paradigm shift in telemedicine by
bringing about ECG-based clinical diagnoses of heart disease via simple PPG
assessment through wearable devices. Our model trained on healthy subjects
achieves an average Pearson’s correlation of 0.858, RMSE of 0.07 mV, and SNR
of 15.365 dB on a small real-world dataset, demonstrating the efficacy of our
approach. Significantly, our model enables the AFib monitoring capability in a
continuous setting, achieving a PR-AUC of 0.986. Being a lightweight model
also facilitates its deployment on resource-constrained devices. In our future
works, we plan to further validate the proposed method with other ECG and PPG
datasets containing noisy PPG signals where the source of noise is from daily-
life activities. Besides, we aim to validate the generalizability of the model
with other physiological signal pairs. Our method allows for the screening and
early detection of cardiovascular diseases in the home environment, which
saves money and labor while supporting society in unusual pandemic situations.

(a) Clean input PPG

(b) Clean input PPG

(c) Noisy input PPG

(d) Noisy input PPG

(e) AFib input PPG

(f) AFib input PPG

Figure 6: Examples of the translated ECG signals. In each subfigure: the top
panel shows the input PPG waveform and the bottom panel shows the
reconstructed ECG waveform compared with the reference waveform. The average
ECG waveform (dark blue) of all possible pulses overlaid on each individual
pulse (light blue).

## Data availability statement

The datasets generated and/or analysed during the current study are available
in the GitHub repository, https://github.com/khuongav/dvae_ppg_ecg

## References

  * [1] Kligfield, P. _et al._ Recommendations for the standardization and interpretation of the electrocardiogram: part i: the electrocardiogram and its technology a scientific statement from the american heart association electrocardiography and arrhythmias committee, council on clinical cardiology; the american college of cardiology foundation; and the heart rhythm society endorsed by the international society for computerized electrocardiology.  _Journal of the American College of Cardiology_ 49, 1109–1127 (2007).
  * [2] Le, T. _et al._ Electrocardiogram: Acquisition and analysis for biological investigations and health monitoring.  In _Interfacing Bioelectronics and Biomedical Sensing_ , 117–142 (Springer, 2020).
  * [3] Allen, J.  Photoplethysmography and its application in clinical physiological measurement.  _Physiol Meas._ 28, 1–39 (2007).
  * [4] sudden.  Sudden death in young people: Heart problems often blamed (Retrieved on September 2020).  https://www.mayoclinic.org/diseases-conditions/sudden-cardiac-arrest/in-depth/sudden-death/art-20047571.
  * [5] Rosiek, A. & Leksowski, K.  The risk factors and prevention of cardiovascular disease: the importance of electrocardiogram in the diagnosis and treatment of acute coronary syndrome.  _Therapeutics and clinical risk management_ 12, 1223 (2016).
  * [6] Olier, I., Ortega-Martorell, S., Pieroni, M. & Lip, G. Y.  How machine learning is impacting research in atrial fibrillation: implications for risk prediction and future management.  _Cardiovascular Research_ 117, 1700–1717 (2021).
  * [7] Reisner, A., Shaltis, P. A., McCombie, D. & Asada, H. H.  Utility of the photoplethysmogram in circulatory monitoring.  _Anesthesiology: The Journal of the American Society of Anesthesiologists_ 108, 950–958 (2008).
  * [8] Banerjee, R., Sinha, A., Choudhury, A. D. & Visvanathan, A.  Photoecg: Photoplethysmographyto estimate ecg parameters.  In _2014 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , 4404–4408 (IEEE, 2014).
  * [9] Zhu, Q., Tian, X., Wong, C.-W. & Wu, M.  Ecg reconstruction via ppg: A pilot study.  In _2019 IEEE EMBS International Conference on Biomedical & Health Informatics (BHI)_, 1–4 (IEEE, 2019).
  * [10] Tian, X., Zhu, Q., Li, Y. & Wu, M.  Cross-domain joint dictionary learning for ecg reconstruction from ppg.  In _ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , 936–940 (IEEE, 2020).
  * [11] Vo, K. _et al._ P2e-wgan: Ecg waveform synthesis from ppg with conditional wasserstein generative adversarial networks.  In _Proceedings of the 36th Annual ACM Symposium on Applied Computing_ , 1030–1036 (2021).
  * [12] Sarkar, P. & Etemad, A.  Cardiogan: Attentive generative adversarial network with dual discriminators for synthesis of ecg from ppg.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , vol. 35, 488–496 (2021).
  * [13] Chiu, H.-Y., Shuai, H.-H. & Chao, P. C.-P.  Reconstructing qrs complex from ppg by transformed attentional neural networks.  _IEEE Sensors Journal_ 20, 12374–12383 (2020).
  * [14] Lee, T.-Y., Vo, K., Baek, W., Khine, M. & Dutt, N.  Stint: selective transmission for low-energy physiological monitoring.  In _Proceedings of the ACM/IEEE International Symposium on Low Power Electronics and Design_ , 115–120 (2020).
  * [15] Kingma, D. P. & Welling, M.  Auto-encoding variational bayes.  _arXiv preprint arXiv:1312.6114_ (2013).
  * [16] Sohn, K., Lee, H. & Yan, X.  Learning structured output representation using deep conditional generative models.  _Advances in neural information processing systems_ 28 (2015).
  * [17] Krishnan, R. G., Shalit, U. & Sontag, D.  Deep kalman filters.  _arXiv preprint arXiv:1511.05121_ (2015).
  * [18] Krishnan, R., Shalit, U. & Sontag, D.  Structured inference networks for nonlinear state space models.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , vol. 31 (2017).
  * [19] Bahdanau, D., Cho, K. & Bengio, Y.  Neural machine translation by jointly learning to align and translate.  _arXiv preprint arXiv:1409.0473_ (2014).
  * [20] Geiger, D., Verma, T. & Pearl, J.  Identifying independence in bayesian networks.  _Networks_ 20, 507–534 (1990).
  * [21] Chung, J., Gulcehre, C., Cho, K. & Bengio, Y.  Empirical evaluation of gated recurrent neural networks on sequence modeling.  _arXiv preprint arXiv:1412.3555_ (2014).
  * [22] Kingma, D. P. & Ba, J.  Adam: A method for stochastic optimization.  _arXiv preprint arXiv:1412.6980_ (2014).
  * [23] Moody, B., Moody, G., Villarroel, M., Clifford, G. & Silva, I.  Mimic-iii waveform database matched subset.  _MIMIC-III Waveform Database Matched Subset v1. 0_ (2020).
  * [24] Johnson, A. E. _et al._ Mimic-iii, a freely accessible critical care database.  _Scientific data_ 3, 1–9 (2016).
  * [25] Tang, Q. _et al._ Robust reconstruction of electrocardiogram using photoplethysmography: A subject-based model.  _Frontiers in Physiology_ 645 (2022).
  * [26] Van Gent, P., Farah, H., Van Nes, N. & Van Arem, B.  Heartpy: A novel heart rate algorithm for the analysis of noisy signals.  _Transportation research part F: traffic psychology and behaviour_ 66, 368–378 (2019).
  * [27] van Gent, P., Farah, H., van Nes, N. & van Arem, B.  Analysing noisy driver physiology real-time using off-the-shelf sensors: Heart rate analysis software from the taking the fast lane project.  _Journal of Open Research Software_ 7 (2019).
  * [28] Lu, G., Yang, F., Taylor, J. & Stein, J. F.  A comparison of photoplethysmography and ecg recording to analyse heart rate variability in healthy subjects.  _Journal of medical engineering & technology_ 33, 634–641 (2009).
  * [29] Aschbacher, K. _et al._ Atrial fibrillation detection from raw photoplethysmography waveforms: A deep learning application.  _Heart rhythm O2_ 1, 3–9 (2020).
  * [30] Begoli, E., Bhattacharya, T. & Kusnezov, D.  The need for uncertainty quantification in machine-assisted medical decision making.  _Nature Machine Intelligence_ 1, 20–23 (2019).
  * [31] Hong, S., Xiao, C., Ma, T., Li, H. & Sun, J.  Mina: multilevel knowledge-guided attention for modeling electrocardiography signals.  _arXiv preprint arXiv:1905.11333_ (2019).