# Non-contact Atrial Fibrillation Detection from Face Videos by Learning
Systolic Peaks

Zhaodong Sun  Juhani Junttila  Mikko Tulppo  Tapio Seppänen  and Xiaobai Li
Manuscript received on November 5, 2021; This study was supported by the
Academy of Finland (project 323287 and 5326291), the Finnish Work Environment
Fund (Project 200414), Sigrid Juselius Foundation and Foundation for
Cardiovascular Research. _Corresponding author: Xiaobai Li_.Zhaodong Sun,
Tapio Seppänen, and Xiaobai Li are with the Center for Machine Vision and
Signal Analysis, University of Oulu, FI-90014 Oulu, Finland (e-mail:
<EMAIL_ADDRESS><EMAIL_ADDRESS>xiaobai.li@oulu.fi). Juhani
Junttila, and Mikko Tulppo are with Research Unit of Internal Medicine,
Medical Research Center Oulu, FI-90014 Oulu, Finland (e-mail:
<EMAIL_ADDRESS>mikko.tulppo@oulu.fi).

###### Abstract

Objective: We propose a non-contact approach for atrial fibrillation (AF)
detection from face videos. Methods: Face videos, electrocardiography (ECG),
and contact photoplethysmography (PPG) from 100 healthy subjects and 100 AF
patients are recorded. Data recordings from healthy subjects are all labeled
as healthy. Two cardiologists evaluated ECG recordings of patients and labeled
each recording as AF, sinus rhythm (SR), or atrial flutter (AFL). We use the
3D convolutional neural network for remote PPG monitoring and propose a novel
loss function (Wasserstein distance) to use the timing of systolic peaks from
contact PPG as the label for our model training. Then a set of heart rate
variability (HRV) features are calculated from the inter-beat intervals, and a
support vector machine (SVM) classifier is trained with HRV features. Results:
Our proposed method can accurately extract systolic peaks from face videos for
AF detection. The proposed method is trained with subject-independent 10-fold
cross-validation with 30s video clips and tested on two tasks. 1)
Classification of healthy versus AF: the accuracy, sensitivity, and
specificity are 96.00%, 95.36%, and 96.12%. 2) Classification of SR versus AF:
the accuracy, sensitivity, and specificity are 95.23%, 98.53%, and 91.12%. In
addition, we also demonstrate the feasibility of non-contact AFL detection.
Conclusion: We achieve good performance of non-contact AF detection by
learning systolic peaks. Significance: non-contact AF detection can be used
for self-screening of AF symptoms for suspectable populations at home or self-
monitoring of AF recurrence after treatment for chronic patients.

Atrial Fibrillation (AF), Remote Photoplethysmography, Photoplethysmography
Imaging, Wasserstein Distance, Heart Rate Variability (HRV)

##  1 Introduction

Atrial fibrillation (AF) is a common heart arrhythmia, and about 2% of the
global population is reported to have this disease [authors20122012]. AF might
cause stroke and heart failure, but most AF episodes are asymptomatic in the
early phase. Early detection of AF episodes is essential to avoid these severe
diseases. The common method for AF diagnosis is to observe electrocardiography
(ECG) signals measured from the electrodes attached to the chest. However,
this requires specific medical equipment and complicated operations to measure
ECG signals. Cardiologists should determine the diagnosis from ECG signals,
which is not practical for daily checks or long-term monitoring. Although some
works [fukunami1991detection, mehta2009detection, guidera1993signal,
couceiro2008detection, tateno2001automatic, dash2009automatic,
islam2016rhythm, islam2019robust, couceiro2008detection, shashikumar2017deep,
fan2018multiscaled, pourbabaee2018deep] designed automatic AF detection
algorithms from ECG and achieve a high accuracy, the acquisition of ECG
signals is complicated and not convenient for daily use. Some works
[poh2018diagnostic, kwon2019deep, eerikainen2019detecting, lee2012atrial,
krivoshei2017smart, bonomi2018atrial, dorr2019watch, corino2017detection,
valiaho2019wrist] propose to use contact photoplethysmography (PPG) from pulse
oximeters, smartwatches or phone cameras on fingers for more convenient and
accessible AF detection. However, all of the ECG and contact PPG methods
require the sensors attached to the body/skin, which might cause irritation
and hygiene issues. In addition, specific biomedical equipment such as ECG
sensors or pulse oximeters also decrease the accessibility of these methods to
more populations.

Compared with ECG and contact PPG, remote PPG111Remote PPG is used to be
literally distinguished from contact PPG. There are other terms equivalent to
remote PPG such as photoplethysmography imaging [wu2000photoplethysmography],
video photoplethysmography [iozzia2016relationships], and videoplethysmography
[couderc2015detection] provides a non-contact way to monitor cardiac signals
by using a camera. Remote PPG is measured from the face color change induced
by the blood volume change in face videos [wu2000photoplethysmography,
verkruysse2008remote], and can reveal cardiac rhythms for AF detection. As
cameras are ubiquitous nowadays, this might lead to potential solutions or
products for convenient non-contact AF detection. There are several situations
where AF detection from face videos is preferred. First, AF detection from
face videos can be used for telemedicine during video conferences without
specific medical instruments. Second, contact AF detection is not applicable
to people with skin burns or other skin diseases on the measurement locations
such as fingers or chest. It has been reported that 486,000 people in the
United States were burned to receive medical treatment in the year 2011
[us_burn]. Third, cameras are ubiquitous and accessible such as in smartphones
or laptops, and much cheaper than the smartwatches with ECG/PPG sensors.
Finally, as people spend a long time on using smart phones or laptops every
day, the AF detection from face videos can be integrated to smart phones or
laptops when people are using these devices.

Recent studies [couderc2015detection, li2018obf, yu2019remote, shi2019atrial,
yan2018contact, corino2020simple] have made some contributions to non-contact
AF detection by remote PPG. Most studies focus on heart rate variability (HRV)
features derived from systolic peaks in remote PPG for AF detection and
achieved promising results. The problem is that remote PPG signals are subtle
compared to noises, and systolic peaks of AF patients have low amplitudes when
the heart rate is high [valiaho2019wrist]. Therefore, it is challenging to
develop a method that can measure systolic peaks accurately enough for AF
detection. Therefore, we develop a method to accurately measure systolic peaks
from face videos to facilitate non-contact AF detection and test the
effectiveness of the method on our large-scale dataset.

Our main contributions are listed below:

  * •

We develop a deep learning-based remote PPG algorithm by using the timing of
the systolic peaks as the ground truth and demonstrate that learning systolic
peaks from face videos can facilitate non-contact AF detection.

  * •

We record the full version of Oulu Bio-face (OBF) dataset [li2018obf] for non-
contact AF detection, including videos from 100 healthy subjects and 100 AF
patients. Two cardiologists evaluated ECG recordings of patients and labeled
each recording as AF, sinus rhythm (SR), or atrial flutter (AFL).

  * •

We test two types of AF detection models for the self-screening of healthy
subjects and patients, respectively, and achieve high accuracy. The
classification between healthy subjects and patients with AF has an accuracy
of 96.00%. The classification between patients with SR and patients with AF
has an accuracy of 95.23%. Additional experiments also demonstrate the
feasibility of non-contact AFL detection.

##  2 Related Works

###  2.1 Contact Methods for AF detection

The AF detection from ECG can be based on 1) P-wave detection
[fukunami1991detection, mehta2009detection, guidera1993signal,
couceiro2008detection], 2) R-R interval variability [tateno2001automatic,
dash2009automatic, islam2016rhythm, islam2019robust, couceiro2008detection],
and 3) deep learning (DL)-based methods [shashikumar2017deep,
fan2018multiscaled, pourbabaee2018deep]. P waves are not prominent features
and are often inﬂuenced by artifacts, while R peaks have higher amplitudes
that can be easily detected. AF patients have a larger R-R interval
variability, which means the cardiac rhythm of AF patients shows larger
irregularity, so the features from R-R intervals can be a good indicator for
AF detection. Some traditional methods [tateno2001automatic,
dash2009automatic, islam2016rhythm] extracted features such as entropy
[islam2016rhythm] or HRV features [dash2009automatic] from R-R intervals and
simply use a threshold to do AF classification. Some methods [islam2017atrial,
andersen2019deep, islam2019robust] applied machine learning methods such as
support vector machine (SVM) [islam2017atrial], convolutional neural network
[andersen2019deep] or neighborhood component analysis [islam2019robust] to R-R
intervals for AF detection and achieved higher performance than traditional
methods. However, it is difficult for rhythm-based methods using R-R intervals
to distinguish AF from other arrhythmias, while the morphology-based methods
such as P-wave absence can avoid false-positive errors in AF detection
[butkuviene2021considerations]. Deep learning-based methods
[shashikumar2017deep, fan2018multiscaled, pourbabaee2018deep] can be directly
applied to ECG signals and achieve good AF detection performance with large
datasets, but they are sensitive to changes in ECG morphology and are unclear
how they generalize to unseen data [butkuviene2021considerations].

Although ECG is a gold standard for AF diagnosis, some studies found contact
PPG [poh2018diagnostic, kwon2019deep, eerikainen2019detecting, lee2012atrial,
krivoshei2017smart, bonomi2018atrial, dorr2019watch, corino2017detection,
valiaho2019wrist] measured from ﬁngertips and ballistocardiograph (BCG)
[bruser2012automatic, hurnanen2016automated, lahdenoja2017atrial] can also be
used for AF detection. PPG and BCG signals do not have the P-wave in ECG, so
the systolic peaks in PPG and BCG are the only clue for AF detection. Similar
to the R-peaks in ECG, the systolic peaks are also an indicator of cardiac
rhythm and the peak-peak intervals can also be used to derive the HRV
features. For the daily use and better accessibility, some work proposed to
use the accelerometer and gyroscope of a smartphone on the chest to get the
BCG signal for AF detection [10], while others proposed to use the camera of a
smartphone [6] on the ﬁngertips or smart watches [bonomi2018atrial,
dorr2019watch, corino2017detection, valiaho2019wrist] to get the contact PPG
signals for AF detection. The AF detection from contact PPG and BCG provides
other alternatives to ECG signals and is more convenient for daily health
monitor.

###  2.2 Remote PPG for Non-contact AF detection

Recent studies [couderc2015detection, yan2018contact, shi2019atrial,
corino2020simple] showed that remote PPG from face videos can be used for non-
contact AF detection. Couderc et al. [couderc2015detection] first used HRV
features from remote PPG for AF detection on a small dataset of 11 patients,
which demonstrates the feasibility of AF detection from face videos. Yan et
al. [yan2018contact] collected a dataset including 217 patients among which 75
patients showed AF. They used a pre-trained SVM with features from
autocorrelation analysis and achieved 95.4% accuracy for classifying AF
patients vs. non-AF patients. Shi et al. [shi2019atrial] proposed a HRV
feature fusion approach by combining three remote PPG algorithms
[wang2016algorithmic, li2018obf, feng2014motion] for AF detection. The method
was tested on the previous version of OBF dataset [li2018obf], including 30 AF
patients and 100 healthy subjects, and achieved an accuracy of 92.56%. Corino
et al. [corino2020simple] used a simple model with three features to do AF
detection from remote PPG on 68 patients and achieved 87% AF detection
accuracy.

There are one or more of the following limitations in the previous works. 1)
Their AF detection accuracy is limited, as the systolic peak information was
not exploited for training remote PPG algorithms. We propose to utilize
systolic peaks to train our remote PPG algorithm, and the reasons are in two
aspects. First, studies [lee2012atrial, eerikainen2019detecting,
couderc2015detection, li2018obf, yu2019remote, shi2019atrial] showed that HRV
features computed using systolic peaks are sufficient for AF detection, so the
waveform is not needed. Second, the systolic peaks are the most prominent and
reliable feature in remote PPG, while other waveform features are subtle and
easily contaminated by noises. 2) Most previous studies were tested on limited
AF cases, and a larger scale of AF data is needed to make a reliable
evaluation of the methods. 3) The low diversity of previous datasets also
limits their application scope. Previous studies mainly focused on classifying
healthy subjects versus before-treatment AF patients. It is also essential to
compare before-treatment (AF) and after-treatment (sinus rhythm, SR) of the
same patients, which was not concerned in any previous study yet. 4) Previous
studies (except [li2018obf]) did not compare the non-contact AF detection
performance with results from ECG or contact PPG. The AF detection results
from ECG or contact PPG can be a performance upper bound from which we can see
the gap between non-contact and contact AF detection.

##  3 Method

In this section, we will introduce the 3-dimensional convolutional neural
network (3DCNN) model for remote PPG monitoring and how to select a loss
function to train our model with the timing of systolic peaks to facilitate AF
detection. We will also explain using the HRV features derived from systolic
peaks for AF detection.

Figure 1: The general framework of non-contact AF detection. The red dots in
the remote PPG are the systolic peaks. 3DCNN = three dimensional convolutional
neural network, PPG = photoplethysmography, AF = atrial fibrillation, HRV =
heart rate variability, SVM = support vector machine.

Figure 2: Three-dimensional convolutional neural network (3DCNN) architecture.
The input is the image sequence of the cropped face. The output is the
systolic peak signal. ”3x3x3 Conv, 64” means the 3D convolution operation with
filter size 3x3x3 and the output channel is 64. ”dconv” is 3D transposed
convolution, which works as upsampling. There is also a batchnorm layer and a
ReLU activation following each convolutional block. There are 4 max pooling
layers. All of them downsample the spatial dimensions, and two of them only
downsample the temporal dimension.

###  3.1 Remote PPG Monitoring

####  3.1.1 Preprocessing

We first need to crop the face region from the face video. The face region is
obtained from the landmarks generated from OpenFace
[baltrusaitis2018openface]. We first get the minimum and maximum horizontal
and vertical coordinates of the landmarks to locate the central face point.
The bounding box size is 1.2 times the range of vertical coordinates of
landmarks and is fixed for each video. After getting the central point and the
size of the bounding box, we can crop the face region from each frame as shown
in Fig. 1. The cropped face is resized to $128\times 128$.

####  3.1.2 3DCNN for Remote PPG Monitoring

The cropped face is fed into the 3DCNN model [yu2019remote] for remote PPG
monitoring. 3DCNN uses the 3D kernels to do the convolution on the video along
the width, height, and time axis. We will use the similar 3DCNN architecture
as [yu2019remote] with one modification that will be illustrated in the next
part. The model is shown in Fig. 2. The input for 3DCNN is the cropped face
video clip $x\in\mathbb{R}^{C\times H\times W\times T}$ where $C$ is the
number of color channels, $H$ is the video height, $W$ is the video width, and
$T$ is the video time length. The output remote PPG is
$p_{r}=G_{\theta}(x)\in\mathbb{R}^{T}$. The input video is a 4D video signal,
while the network eliminates the height, width, and color dimensions and
converts the 4D signal into a 1D remote PPG signal. To train this network, we
need to minimize a loss function. The loss function measures the distance
between the ground truth, which is the contact PPG $p_{c}(t)$, and the network
output, which is the remote PPG $p_{r}(t)$. A previous method [yu2019remote]
used negative Pearson correlation as the loss function with the contact PPG as
the ground truth. This loss function encourages the remote PPG signals and
contact PPG signals to have similar morphology. However, this loss function
also encourages the model to learn redundant information from the contact PPG,
such as diastolic peaks or artifacts.

####  3.1.3 Binary Systolic Peaks for 3DCNN Training

Using contact PPG as the ground truth to train the network is not the best.
The systolic peak timing in the contact PPG is a better ground truth. There
are two reasons. First, [mcduff2014remote] demonstrated that the waveform
morphology of contact PPG signals is not entirely consistent with remote PPG
due to their different measurement locations. This means that contact PPG is
not the actual ground truth for each face video due to the inconsistency of
contact and remote PPG. However, [mcduff2014remote] also showed that the
timing of systolic peaks is the consistent information between contact and
remote PPG. Second, contact PPG from patients usually contain systolic peaks
with much lower amplitude than that from healthy subjects, as shown in Fig. 3.
Väliaho et al. [valiaho2019wrist] also reported this phenomenon and explained
that a higher heart rate in AF causes smaller stroke blood volume and lower
systolic peak amplitudes. If we use the contact PPG as the ground truth for
model training, the model might learn some redundant information and ignore
some systolic peaks with low amplitude. Therefore, we will convert our contact
PPG into binary systolic peaks as shown in Fig. 3. The binary systolic peaks
only have zeros and ones, which means it only keeps the timing of the systolic
peaks and removes other redundant information. This kind of ground truth can
encourage the model only to learn the timing of systolic peaks, which is also
the only information used for AF detection.

(a) Contact PPG from a healthy subject

(b) Binary systolic peaks from a healthy subject

(c) Contact PPG from a patient with AF

(d) Binary systolic peaks from a patient with AF

Figure 3: Contact PPG (with systolic peaks marked with red dots) and
corresponding binary systolic peaks. Binary systolic peaks (b and d) are
extracted from the contact PPG signals (a and c). The PPG signal from the AF
patient contains complex patterns such as irregular heartbeat rhythms and
different systolic peak amplitudes. PPG = photoplethysmography, AF = atrial
fibrillation.

The next question is how to use the binary systolic peaks for our model
training. We can regard our model output $p_{r}=G_{\theta}(x)$ and our binary
systolic peak label $s$ as two probability distributions so that we can use
some probability distribution distances as our loss function. The reason is
that probability distribution distances can compare a binary pulse signal with
a continuous smooth signal. Our systolic peak signal is just a series of
binary signals, and our model output is the continuous smooth signal, so
probability distribution distances might be a good choice. Previous used
negative Pearson correlation loss function can handle continuous values in
contact PPG but cannot work with binary values in binary systolic peak
signals.

To satisfy the requirement of probability distributions, we should ensure that
the sum of a signal is one and the signal is non-negative. We can normalize
the binary systolic peak as $\bar{s}=s/\Sigma_{i}s_{i}$. Since the softmax
layer can make sure the model output is non-negative and the sum is one, we
add a softmax layer to the output of the network $G_{\theta}(x)$ and the new
network is $H_{\theta}(x)=\text{softmax}(G_{\theta}(x))$. Therefore, we have
our new model output $\bar{p}_{r}=H_{\theta}(x)$. Our new loss function
$l(\bar{s},\bar{p}_{r})$ is defined to measure the distance between normalized
systolic peak signal $\bar{s}$ and model output $\bar{p}_{r}$. We use
Wasserstein distance as our loss function $l$ for learning the systolic peaks.
The reason why Wasserstein distance is used as the loss function and the full
analysis about the loss function selection is described in the following.

###  3.2 Loss Function Selection for Learning Systolic Peaks

####  3.2.1 Candidate loss functions

For the proposed 3DCNN model, both the model output signal $\bar{p}_{r}$ and
the normalized binary systolic peaks $\bar{s}$ can be regarded as probability
distributions. Therefore, we can use some probability distribution distances
to measure their similarity. These probability distribution distances can be
used as loss functions for the model training. During training, the loss
function is minimized so that our model output signal is well aligned with the
systolic peak signals. In previous work [cazelles2020wasserstein], probability
distribution distances were used for comparing two time series in the power
spectral density (PSD) domain. However, the accurate representation of
systolic peaks can only be in the time domain. Therefore, we directly use the
probability distribution distances in the time domain. There are some options
for probability distribution distances, such as squared Euclidean distance,
Kullback-Leibler (KL) divergence, Jensen–Shannon (JS) divergence, and
Wasserstein distance.

Squared Euclidean Distance (SED): Squared Euclidean distance is a
straightforward way to compare two time series. It measures the Euclidean
distance between two probability distributions. It is also widely used as a
loss function for deep learning models. It is defined as

$l_{SED}(\bar{p}_{r},\bar{s})=\parallel\bar{p}_{r}-\bar{s}\parallel_{2}^{2}$
(1)

Kullback-Leibler (KL) Divergence: The KL divergence is another probability
distribution distance widely used as the loss function in classification
tasks. In the classification tasks, it measures the distance between the true
classification distribution and the predicted classification distribution. KL
divergence is defined as

$l_{KL}(\bar{p}_{r},\bar{s})=\text{KL}(\bar{s}\parallel\bar{p}_{r})=\sum_{t=1}^{T}\bar{s}(t)\log\Big{(}\frac{\bar{s}(t)}{\bar{p}_{r}(t)}\Big{)}$
(2)

Jensen–Shannon (JS) Divergence: JS divergence is based on KL divergence.
Compared with KL divergence, JS divergence is symmetric and bounded between 0
and 1. It is defined as

$l_{JS}(\bar{p}_{r},\bar{s})=\frac{1}{2}\text{KL}(\bar{p}_{r}\parallel\frac{\bar{p}_{r}+\bar{s}}{2})+\frac{1}{2}\text{KL}(\bar{s}\parallel\frac{\bar{p}_{r}+\bar{s}}{2})$
(3)

Wasserstein Distance: Wasserstein distance is based on optimal transport. It
finds the minimum cost to move the mass of one probability distribution to
turn the probability distribution into another. It is defined as

$l_{WS}(\bar{p}_{r},\bar{s})=\min_{\pi\in\Pi(\bar{p}_{r},\bar{s})}\mathbb{E}_{(x,y)\sim\pi}[\parallel
x-y\parallel]$ (4)

where $\Pi(\bar{p}_{r},\bar{s})$ is a set containing all joint probability
distributions with marginal distribution $\bar{p}_{r}$ and $\bar{s}$.
Wasserstein distance finds the joint probability distribution (transport plan)
$\pi$ so that the movement cost is minimum. $x$ and $y$ are two random
variables following the joint distribution $\pi$. However, the formula above
has a min operation and cannot be directly used as a loss function. Since our
signals $\bar{p}_{r}$ and $\bar{s}$ are one-dimensional, the Wasserstein
distance has a closed-form in the one-dimensional case, as shown below.

$l_{WS}(\bar{p}_{r},\bar{s})=\int_{0}^{1}|F_{\bar{p}_{r}}^{-}(u)-F_{\bar{s}}^{-}(u)|du=\sum_{t=1}^{T}|F_{\bar{p}_{r}}(t)-F_{\bar{s}}(t)|$
(5)

where $F_{\bar{p}_{r}}$ is the cumulative sum of $\bar{p}_{r}$, which is also
the cumulative distribution function. $F_{\bar{p}_{r}}^{-}$ is the inverse of
function $F_{\bar{p}_{r}}$. The Wasserstein distance used for the model
training is the rightmost part of equation 5. Fig. 4 shows the interpretation
of the loss function. To calculate the value of the Wasserstein distance
between two signals, we first get the cumulative sum of each signal. From
equation 5, the Wasserstein distance is the absolute difference between the
two cumulative sum curves, as the gray area shown in the third column of Fig.
4.

(a) epoch 1

(b) epoch 15

(c) epoch 45

Figure 4: Illustration of how Wasserstein distance works. The first, second,
and third rows are at epochs 1, 15, and 45 of model training. The first column
shows the ground truth systolic peaks and our model output. The second column
shows the cumulative sum of the two signals in the first column and also shows
the zoom-in figures. The gray area is the Wasserstein distance value. During
model training, we intend to minimize this gray area so that the model output
is similar to the systolic peak signal. The decreasing of the area makes the
peak signal from the model sharper and more accurately aligned with the true
systolic peaks.

####  3.2.2 Analysis of Peak Misalignment

We first analyze the loss function response when two comparing signals are
misaligned. We first generate a smooth peak signal $p_{s}$ from a truncated
Gaussian distribution curve $\mathcal{N}(0,\sigma^{2})$ with $\sigma^{2}=0.1$
to imitate the 3DCNN model output and normalize it to make sure the sum is
one. We also generate a binary peak signal $p_{b}$ to imitate the ground truth
systolic peak. These two signals are shown in Fig. 5(a). We shift the binary
peak signal $p_{b}(t)$ to $p_{b,\Delta t}(t)=p_{b}(t-\Delta t)$ and get the
loss function values $l(p_{s},p_{b,\Delta t})$. We can plot the loss function
values $l(p_{s},p_{b,\Delta t})$ with respect to peak shift $\Delta t$ to see
the loss functions response to the peak misalignment. The results are shown in
Fig. 5(b-d). For squared Euclidean distance and JS divergence in Fig. 5(b-c),
the loss value becomes saturated and constant when the absolute peak shift is
too large, which means the loss function cannot assign a larger penalty to a
larger peak alignment. This disadvantage might prevent the model from learning
the accurate systolic peaks at these saturated locations. On the other side,
KL divergence and Wasserstein distance in Fig. 5(d) assign a larger penalty
when the peak misalignment is larger and will not be saturated when the peak
shift is large, which means KL divergence and Wasserstein distance could be
the promising options.

(a)

(b)

(c)

(d)

Figure 5: (a) The smooth peak and binary peak. (b) Squared Euclidean Distance
with respect to the peak shift $\Delta t$. (c) JS divergence with respect to
the peak shift $\Delta t$. (d) KL divergence and Wasserstein distance with
respect to the peak shift $\Delta t$.

####  3.2.3 Analysis of Peak Sharpness

We also analyze the loss function response to peak sharpness. We hope the
3DCNN model will output signals with sharper peaks rather than flat ones, as
sharper peaks are less impacted by noises and easier to detect. We use the
binary peak and a series of smooth peak signals with different variances
$\sigma^{2}$ to analyze the response of the loss function to the peak
sharpness. We can change the variance $\sigma^{2}$ of the truncated Gaussian
distribution curve $\mathcal{N}(0,\sigma^{2})$ to control the sharpness of the
smooth peak $p_{s,\sigma^{2}}$. Smaller variance $\sigma^{2}$ means sharper
peaks. This phenomenon is illustrated in Fig. 6(a). The smooth peak with
$\sigma^{2}=10$ is almost flat, while the smooth peak with $\sigma^{2}=0.01$
is very sharp and similar to the binary peak. By changing the variance
$\sigma^{2}$ of the smooth peak, we can plot the loss values
$l(p_{s,\sigma^{2}},p_{b})$ with respect to the variance $\sigma^{2}$ of the
smooth peak in Fig. 6(b). From the plot, Wasserstein distance has larger
penalty values than other loss functions, especially at large variance
positions (small sharpness), which indicates that Wasserstein distance can
assign a much larger penalty value when the output peaks are too flat.
Therefore, Wasserstein distance is the best to motivate the model to produce
sharp peaks.

(a)

(b)

Figure 6: (a) The binary peak and the smooth peak with difference variances
(sharpness). (b) The loss values of the four candidate loss function with
respect to the variance $\sigma^{2}$ of the smooth peak.

###  3.3 AF detection with HRV features

The AF detection part is shown in the right part of Fig. 1. We select 20
features as listed below according to previous studies [lee2012atrial,
eerikainen2019detecting, couderc2015detection, li2018obf, shi2019atrial] to
train a support vector machine (SVM) classifier with radial basis function
(RBF) kernel.

  * •

Time domain: mean inter-beat interval (IBI), standard deviation of IBI (SDNN),
standard deviation of successive difference of IBI (SDSD), percentage of
samples with more than 50 ms difference from the consecutive beat (pNN50),
percentage of samples with more than 20 ms difference from the consecutive
beat (pNN20), the number of samples with more than 50 ms difference from the
consecutive beat (NN50), the number of samples with more than 20 ms difference
from the consecutive beat (NN20), the root mean square of successive
differences of IBI (RMSSD), median of IBI, range of IBI, the coefficient of
variation of successive differences (CVSD), the coefficient of variation
(CVNNI), maximum heart rate, minimum heart rate, and the standard deviation of
heart rate.

  * •

Spectral domain: the power in the low frequency (LF) (0.04Hz-0.15Hz), the
power in the high frequency (HF) (0.15Hz-0.4Hz), and the ratio of LF and HF.

  * •

Geometrical domain: Poincaré plot standard deviations (SD1, SD2).

##  4 Experimental Setup

In this section, we first present our dataset. We will provide the
experimental protocol and evaluation metrics for two experiments. One
experiment is remote PPG monitoring, and another is non-contact AF/AFL
detection.

###  4.1 Dataset

####  4.1.1 Participants

We record the full version of the Oulu Bio-face (OBF) dataset with 100 healthy
subjects and 100 AF patients. The healthy subjects were recruited from the
University of Oulu, and AF patients were from Oulu University Hospital. The
study was performed according to the Declaration of Helsinki, and the local
committee of research ethics of the Northern Ostrobothnia Hospital District
approved the protocol (reference number: 116/2016). All participants gave
their written consent before recording. The statistical information of
participants is summarized in Table 1. The partitions of healthy subjects and
patients in the OBF dataset are OBF-H and OBF-P, respectively.

Table 1: Statistical Information of the participants | OBF-H (n=100) | OBF-P (n=100)
---|---|---
Age (year) | 31.6 ± 8.8, [18, 68] | 64.2 ± 9.2, [43, 88]
Gender |  | 61% Male
---
39% Female
| 83% Male
---
17% Female
Ethnic |  | Caucasian: 32%,
---
Asian: 37%,
Others: 31%.
Caucasian: 100%
Weight (kg) | 71 ± 16 | 96 ± 21
Ratio of wearing eyeglasses | 39% | 45%

####  4.1.2 Data Acquisition

The recording setup is shown in Fig. 7. A participant was seated in front of
an RGB camera at a one-meter distance. Two LED lights on both sides of the RGB
camera face toward the participant at a 45-degree angle at a 1.5-meter
distance. Meanwhile, ECG and contact PPG signals are measured by ECG and PPG
sensors, respectively, and synchronized with the video. The recording
equipment and settings are shown in Tab. 2.

Figure 7: Data Recording Setup. RGB = red green blue, ECG =
electrocardiography, PPG = photoplethysmography, LED = light-emitting diode
Table 2: Recording Equipment and Settings

Devices | Speicifications | Settings | Outputs
---|---|---|---
RGB camera |  | Blackmagic
---
URFA mini
| ISO 400,
---
FPS 60,
Resolution
1920 x 1080
RGB video
ECG sensor | NX-EXG2B |  | Sampling rate
---
256 Hz
ECG signal
| Contact
---
PPG sensor
NX-BVP1C |  | Sampling rate
---
128Hz
| Contact
---
PPG signal
LED lights |  | Aputure,
---
LightStorm LS 1c
| Brightness: 3500 lux,
---
Temperature: 5500 k
N/A

  * •

RGB = red green blue, ECG = electrocardiography, PPG = photoplethysmography,
LED = light-emitting diode

During the recording, the participant was seated facing the camera, and the
ECG/contact PPG sensors were attached to them. For each healthy subject, there
are two recording sessions, and each lasts for five minutes. One session was
recorded at a resting status, and the other was recorded after 5 minutes of
exercise (climbing the stairs) so that a wider range of heart rates is
covered. The mean heart rates before and after exercise are 73.6 bpm and 83.2
bpm, respectively. The heart rate ranges before and after exercise are
50.09-107.2 bpm and 60.64-130.1 bpm, respectively. For each AF patient, there
are also two recording sessions. One was recorded before cardioversion
treatment (AF symptom presented), and the other was recorded after the
treatment (back to sinus rhythm) so that each patient has both AF and SR data.
The mean heart rates before and after treatment are 78.9 bpm and 66.5 bpm,
respectively. The corresponding ECG and contact PPG were recorded
simultaneously with each video. Finally, OBF-H has 200 videos. Due to data
loss, some patients only have one session, and the total number of videos in
OBF-P is 169.

####  4.1.3 Data Labelling

The dataset composition and labels are shown in Table 3. In order to do the AF
detection, all the videos in OBF-H are labeled as healthy. All patients’ data
in OBF-P were labeled by two independent cardiologists by observing ECG
signals. Only persistent AF/SR/AFL (i.e., no rhythm transition during the
recording) cases were labelled as AF/SR/AFL, and complex cases (e.g.,
paroxysmal, ventricular extrasystole, supraventricular extrasystoles) were
labelled as ’Other’ (24 cases). Finally, there are 73 videos labeled as AF, 61
videos labeled as SR, 11 videos labeled as AFL, and 24 videos labeled as other
that are not used for classification experiments.

Table 3: OBF Dataset Composition and Classification Labels
(The numbers of participants are in the brackets)

Modalities |  | Dataset
---
Partitions
Participants | Sessions | Labels
RGB Videos, ECG, Contact PPG | OBF-H |  Healthy Subjects |  | Before
---
Exercise
(100)
Healthy (200)
| After
---
Exercise
(100)
OBF-P |  AF Patients |  | Before
---
Cardioversion
(90)
AF (73) SR (61) AFL (11) Other (24)
| After
---
Cardioversion
(79)

  * •

RGB = red green blue, ECG = electrocardiography, PPG = photoplethysmography,
AF = atrial fibrillation, SR = sinus rhythm, AFL = atrial flutter.

###  4.2 Experiment I: Remote PPG Monitoring

####  4.2.1 Experimental Protocol

We train our models with three dataset partitions: OBF-P, OBF-H, or the whole
OBF dataset. We also test our models on these three dataset partitions. The
model is trained for 45 epochs with a learning rate of 0.0001. Each training
video clip has 512 frames, and the batch size for training is 4. The binary
systolic peaks are obtained by NeuroKit2 [makowski2021neurokit2] from our
contact PPG and will be used as the ground truth to train the 3DCNN model.
Wasserstein distance is used as the loss function. For testing, the videos are
divided into non-overlapping 30-seconds clips and used as inputs. We use
subject independent 10-folds cross-validation in all experiments except two
cases, i.e., training on OBF-H and testing on OBF-P, and training on OBF-P and
testing on OBF-H, which use cross-set protocol. We also compare our method
(3DCNN-PEAK) with four remote PPG methods: 3DCNN-BVP [yu2019remote] is a deep
learning-based method using contact PPG waveforms for training. POS
[wang2016algorithmic], CHROM [de2013robust], and PBV [de2014improved] are
classical remote PPG algorithms without training.

####  4.2.2 Evaluation Metrics

We evaluate accuracy on two levels, i.e., the average heart rate level and IBI
level. Accurate measure on the IBI level is more challenging and essential for
AF detection since it requires accurate systolic peaks.

We use mean absolute error (MAE), root mean squared error (RMSE), and Pearson
correlation (R) to evaluate the error of the average heart rate. MAE for heart
rate is defined as
$\text{MAE}_{HR}=\sum_{n=1}^{N}|\text{HR}_{video}-\text{HR}_{true}|/N$, where
N is the number of samples, $\text{HR}_{video}$ is the heart rate measured
from face videos, and $\text{HR}_{true}$ is the true heart rate obtained from
the contact PPG. RMSE is defined as
$\text{RMSE}_{HR}=\sqrt{\sum_{n=1}^{N}(\text{HR}_{video}-\text{HR}_{true})^{2}/N}$.
Small MAE and RMSE values indicate accurate heart rate estimation. Pearson
correlation is the linear correlation between the heart rates measured from
videos and the true heart rates. When the Pearson correlation is close to 1,
the heart rate estimation is accurate.

We use three metrics of mean absolute error (MAE), standard deviation (STD),
and accuracy as used in [liu2020detecting] to evaluate the error of IBI. We
can first define the absolute error of IBI as
$\text{AE}=\sum_{t=1}^{K}|\text{IBI}_{video}(k)-\text{IBI}_{true}(k)|/K$ where
K is the length of the IBI, $\text{IBI}_{video}$ is the IBI curve from face
video, and $\text{IBI}_{true}$ is the true IBI curve from contact PPG. Since
the original IBI curve is an irregularly spaced time series, we should
resample the original IBI curve to get an evenly spaced time series to
calculate the absolute error between two IBI curves. The MAE and STD of IBI
are the mean and standard deviation of AE for all samples. The accuracy of IBI
is defined as
$\text{AC}_{IBI}=1-\frac{1}{N}\sum_{n=1}^{N}\text{AE}_{n}/(T/(B_{n}-1))$ where
$T$ is the time length of a IBI curve and $B_{n}$ is the number of systolic
peaks in the $n^{th}$ sample.

###  4.3 Experiment II: AF/AFL detection

####  4.3.1 Experimental Protocol

Remote PPG is obtained from the model trained on both OBF-P and OBF-H. HRV
features (as described in Section 3.3) are calculated from the systolic peaks
in remote PPG and used for AF detection experiments. We perform two kinds of
AF detection. 1) We use videos from healthy subjects and videos with AF labels
from patients to classify healthy vs. AF. 2) We use the patient videos with SR
labels and patient videos with AF labels to classify SR vs. AF. We perform the
classification with different clip lengths of 10s, 20s, 30s, 60s, and 120s to
see how the clip length influences the AF detection. 3) We also perform the
classification of SR vs. AFL with clip length 30s to investigate whether AFL
can be detected from face videos.

In the classification experiments of healthy vs. AF and SR vs. AF, we use
subject-independent 10-fold cross-validation 222This means that all subjects
are evenly divided into 10 folds. All data from one subject is only in one
fold. For each testing, one fold is used for testing, and the left nine folds
are used for training, which means data from one subject is never in both
testing and training.. In the classification experiment of SR vs. AFL, the
numbers of two labels are unbalanced, and the number of samples is limited, so
we use another evaluation protocol in [shi2019atrial]. We randomly select 6 SR
patients and 6 AFL patients to form the training set while randomly selecting
another 5 SR patients and the left 5 AFL patients as the testing set. Thus,
subjects in the training and testing sets are balanced and not overlapped. The
experiments are independently performed ten times, and the average performance
is reported.

We also report results achieved by using the ECG signals and contact PPG
signals with the same HRV features in Sec. 3.3, which are the ECG/contact PPG
reference methods. Theoretically, the ECG and contact PPG results should be
the upper bound that the model can achieve and are the reference results
compared with remote PPG results. We also report results of ECG-based AF
detection methods [islam2019robust, fan2018multiscaled] for healthy vs. AF and
SR vs. AF. For remote PPG, we compare our results with four previous
methods[yu2019remote, shi2019atrial, wang2016algorithmic, de2014improved]. For
the baseline methods [yu2019remote, wang2016algorithmic, de2014improved] and
ECG/contact PPG reference methods, we use the same SVM classifier and HRV
features from Sec. 3.3, while only change the input IBI series. For
[islam2019robust, fan2018multiscaled, shi2019atrial], we use the features and
models in their papers.

####  4.3.2 Evaluation Metrics

AF detection results are summarized into true positive (TP), true negative
(TN), false positive (FP), and false negative (FN). We use accuracy,
sensitivity, and specificity as the classification metrics. Accuracy is
defined as $(TP+TN)/(TP+TN+FP+FN)$. Sensitivity is defined as $TP/(TP+FN)$.
Specificity is defined as $TN/(TN+FP)$. We also add metrics of F1 score and
area under curve (AUC) for more convincing comparison.

##  5 Results and Discussion

In this section, we will present the results of our two experiments. One
experiment is remote PPG monitoring, and another is non-contact AF/AFL
detection. Results about computational speed are also reported. Finally,
extensive discussion about the results is provided.

(a)

(b)

(c)

(d)

(e)

(f)

Figure 8: (a-c) The accuracy, sensitivity, and specificity with respect to the
clip length for healthy vs. AF. (d-f) The accuracy, sensitivity, and
specificity with respect to the clip length for SR vs. AF. The red dot lines
are the 30s clip length that we choose for our classification experiments. ECG
= electrocardiography, PPG = photoplethysmography.

###  5.1 Results for Experiment I: Remote PPG Monitoring

We compare these four candidate loss functions mentioned in Sec. 3.2.1. Table
4 shows that Wasserstein distance has the best performance for remote PPG
monitoring. Fig. 4 also shows how Wasserstein distance boosts the predicted
systolic peaks. Therefore, our experimental results validate our analysis
conclusion in Section 3.2 that Wasserstein distance is the best option.

Table 4: Performance Comparison of the candidate loss functions

| Heart Rate | Inter-beat Interval
---|---|---
| MAE
---
(bpm)
| RMSE
---
(bpm)
R |  | MAE
---
(ms)
| STD
---
(ms)
| Accuracy
---
(%)
KL | 8.77 | 16.45 | 0.51 | 189.44 | 435.86 | 83.84
JS | 14.80 | 23.59 | 0.30 | 390.62 | 705.08 | 75.83
SED | 7.18 | 14.39 | 0.58 | 159.77 | 437.77 | 86.07
WS | 1.46 | 3.53 | 0.96 | 50.74 | 68.06 | 94.18

  * •

KL = Kullback-Leibler Divergence, JS = Jensen–Shannon Divergence, SED =
Squared Euclidean Distance, WS = Wasserstein Distance, MAE = mean absolute
error, RMSE = root mean squared error, R = Pearson correlation, STD = standard
deviation.

Table 5 shows the test results on OBF-P. No matter what training set is used,
our proposed method (3DCNN-PEAK) achieves the best performance on heart rate
and IBI measurement. Using both OBF-P and OBF-H for training is preferable
since this can further improve the model performance compared with using OBF-P
or OBF-H alone. The cross dataset test shown in the OBF-H row indicates that
our 3DCNN-PEAK method has better generalization capability than the 3DCNN-BVP
method. Table 6 shows the test results on OBF-H. Our 3DCNN-PEAK method still
outperforms other baseline methods. Using both OBF-P and OBF-H for training
also achieves the best performance on OBF-H. The reason is that a larger
dataset (OBF-P + OBF-H) can include more face videos with different noise
types and facial skin types from both healthy subjects and patients, which can
train the model to be more robust to noise and better capture subtle remote
PPG. It is obvious that the error of heart rate and IBI on OBF-H in Table 6 is
much lower than on OBF-P in Table 5 for all methods, which means the remote
PPG monitoring for patients is more difficult than for healthy subjects. We
provide the final results of training and testing on the whole OBF dataset in
Table 7, and our proposed 3DCNN-PEAK achieves the best results where MAE
values of heart rate and IBI are 1.4658 BPM and 50.7481 ms, respectively.

Table 5: results of heart rate and inter-beat interval on OBF-P

Training Set (Validation Protocol) | Remote PPG Algorithm | Heart Rate | Inter-beat Interval
---|---|---|---
| MAE
---
(bpm)
| RMSE
---
(bpm)
R |  | MAE
---
(ms)
| STD
---
(ms)
| Accuracy
---
(%)
OBF-P and OBF-H (10-folds) | 3DCNN-PEAK | 2.5304 | 5.0275 | 0.9409 | 81.2244 | 88.6318 | 90.91
3DCNN-BVP [yu2019remote] | 3.7356 | 7.1297 | 0.8795 | 95.8779 | 122.6897 | 89.45
OBF-P (10-folds) | 3DCNN-PEAK | 3.3825 | 6.7187 | 0.8916 | 92.8611 | 99.0031 | 89.32
3DCNN-BVP [yu2019remote] | 4.4277 | 8.3049 | 0.8350 | 104.1833 | 107.9021 | 88.21
OBF-H (cross dataset) | 3DCNN-PEAK | 3.6671 | 6.6110 | 0.8957 | 91.9969 | 84.8351 | 89.08
3DCNN-BVP [yu2019remote] | 5.4465 | 9.9317 | 0.7562 | 109.1921 | 107.8691 | 86.68
- | POS [wang2016algorithmic] | 6.2537 | 9.7026 | 0.7836 | 121.1604 | 93.5224 | 85.99
CHROM [de2013robust] | 7.2949 | 10.7810 | 0.7223 | 152.2947 | 90.2355 | 82.40
PBV [de2014improved] | 9.2961 | 13.0999 | 0.6294 | 159.7383 | 115.8414 | 82.28

  * •

MAE = mean absolute error, RMSE = root mean squared error, R = Pearson
Correlation, STD = standard deviation.

Table 6: results of heart rate and inter-beat interval on OBF-H

Training Set (Validation Protocol) | Remote PPG Algorithm | Heart Rate | Inter-beat Interval
---|---|---|---
| MAE
---
(bpm)
| RMSE
---
(bpm)
R |  | MAE
---
(ms)
| STD
---
(ms)
| Accuracy
---
(%)
OBF-P and OBF-H (10-folds) | 3DCNN-PEAK | 0.4534 | 1.2868 | 0.9954 | 25.2568 | 22.5873 | 96.91
3DCNN-BVP [yu2019remote] | 0.7363 | 2.1517 | 0.9873 | 26.8545 | 25.6097 | 96.70
OBF-H (10-folds) | 3DCNN-PEAK | 0.5824 | 1.3059 | 0.9955 | 25.4702 | 21.9592 | 96.79
3DCNN-BVP [yu2019remote] | 1.2912 | 3.4956 | 0.9676 | 33.1952 | 37.6494 | 95.78
OBF-P (cross dataset) | 3DCNN-PEAK | 1.2621 | 3.1478 | 0.9751 | 33.6131 | 29.8656 | 95.78
3DCNN-BVP [yu2019remote] | 1.4451 | 4.3984 | 0.9447 | 39.4681 | 56.3431 | 95.16
- | POS [wang2016algorithmic] | 2.1005 | 5.3898 | 0.9221 | 47.0708 | 50.3133 | 94.13
CHROM [de2013robust] | 4.1686 | 7.6021 | 0.8434 | 93.3947 | 51.5717 | 88.24
PBV [de2014improved] | 3.2652 | 6.2210 | 0.9106 | 63.8180 | 52.8818 | 92.24

  * •

MAE = mean absolute error, RMSE = root mean squared error, R = Pearson
Correlation, STD = standard deviation.

Table 7: Results of heart rate and inter-beat interval on the whole OBF
dataset

| Heart Rate | Inter-beat Interval
---|---|---
| MAE
---
(bpm)
| RMSE
---
(bpm)
R |  | MAE
---
(ms)
| STD
---
(ms)
Accuracy
3DCNN-PEAK | 1.4658 | 3.5351 | 0.9686 | 50.7481 | 68.0636 | 94.18
3DCNN-BVP[yu2019remote] | 2.1371 | 5.0769 | 0.9347 | 58.2923 | 91.6229 | 93.40
POS[wang2016algorithmic] | 4.0234 | 7.6933 | 0.8568 | 81.3753 | 82.3035 | 90.36
CHROM[de2013robust] | 5.6161 | 9.2114 | 0.7900 | 120.6662 | 77.8512 | 85.53
PBV[de2014improved] | 6.0576 | 10.0119 | 0.7711 | 108.2304 | 100.0128 | 87.63

  * •

MAE = mean absolute error, RMSE = root mean squared error, R = Pearson
Correlation, STD = standard deviation.

###  5.2 Results for Experiment II: AF/AFL detection

Fig. 8 shows the accuracy, sensitivity, and specificity with clip lengths of
10s, 20s, 30s, 60s, and 120s for both healthy vs. AF and SR vs. AF. In
general, ECG has the best performance among all, and accuracy is high even
with the shortest 10s clip length. Our proposed 3DCNN-PEAK outperforms other
remote PPG methods at most clip lengths. We can observe that longer clips can
provide better performance. However, longer recording time also decreases the
convenience and application scopes of our method. Therefore, we need to
balance the recording time and classification performance. It can be observed
that the classification performance increases significantly from 10s to 30s
but does not increase too much after 30s for our method and most other
methods. In addition, the 30s recording time is not very long. Therefore, we
select 30s length to show results from our method and other methods in the
following part.

Table 8 shows the classification results for healthy vs. AF when the clip
length is 30s. The accuracy, sensitivity, and specificity for our method are
96.00%, 95.36%, and 96.12%, respectively. For SR vs. AF results in Table 9,
the accuracy, sensitivity, and specificity for our method are 95.23%, 98.53%,
and 91.12%, respectively. For SR vs. AFL in Table 10, the accuracy,
sensitivity, and specificity for our method are 88.43%, 90.04%, and 87.04%,
respectively. Compared with other methods, our method performs the best among
remote PPG methods and is close to ECG and contact PPG methods.

Table 8: Results of classification between healthy subjects and patients with
AF for 30s clips

| Signal
---
types
| Methods
---
| AC
---
(%)
| SE
---
(%)
| SP
---
(%)
| F1
---
(%)
AUC
ECG | Reference | 99.38 | 99.63 | 99.24 | 99.06 | 0.9943
Islam et al. [islam2019robust] | 99.27 | 99.24 | 99.28 | 98.87 | 0.9915
Fan et al. [fan2018multiscaled] | 97.52 | 99.15 | 96.44 | 97.12 | 0.9806
| Contact
---
PPG
Reference | 98.94 | 99.08 | 98.86 | 98.22 | 0.9897
Remote PPG | 3DCNN-PEAK | 96.00 | 95.36 | 96.12 | 92.67 | 0.9574
3DCNN-BVP [yu2019remote] | 95.32 | 94.51 | 95.56 | 91.64 | 0.9504
Shi et al. [shi2019atrial] | 90.53 | 79.88 | 94.51 | 81.96 | 0.8720
POS [wang2016algorithmic] | 91.21 | 90.08 | 91.61 | 85.19 | 0.9084
PBV [de2014improved] | 89.75 | 86.92 | 90.69 | 82.23 | 0.8881

  * •

ECG = electrocardiography, PPG = photoplethysmography, AF = atrial
fibrillation, AC = accuracy, SE = Sensitivity, SP = Specificity, AUC = area
under curve.

Table 9: Results of classification between patients with SR and patients with
AF for 30s clips

| Signal
---
types
| Methods
---
| AC
---
(%)
| SE
---
(%)
| SP
---
(%)
| F1
---
(%)
AUC
ECG | Reference | 97.89 | 99.31 | 95.85 | 98.13 | 0.9758
Islam et al. [islam2019robust] | 98.32 | 99.49 | 96.87 | 98.52 | 0.9762
Fan et al. [fan2018multiscaled] | 96.53 | 97.95 | 94.93 | 96.11 | 0.9617
| Contact
---
PPG
Reference | 95.67 | 98.23 | 92.83 | 96.22 | 0.9553
Remote PPG | 3DCNN-PEAK | 95.23 | 98.53 | 91.12 | 95.87 | 0.9482
3DCNN-BVP [yu2019remote] | 94.38 | 98.48 | 88.95 | 95.17 | 0.9380
Shi et al. [shi2019atrial] | 89.01 | 92.15 | 84.87 | 89.87 | 0.8851
POS [wang2016algorithmic] | 92.87 | 97.18 | 87.93 | 93.66 | 0.9256
PBV [de2014improved] | 88.51 | 95.77 | 79.86 | 90.08 | 0.8782

  * •

ECG = electrocardiography, PPG = photoplethysmography. SR = sinus rhythm. AF =
atrial fibrillation, AC = accuracy, SE = Sensitivity, SP = Specificity, AUC =
area under curve.

Table 10: Results of classification between patients with SR and patients with
AFL for 30s clips

| Signal
---
types
| Methods
---
| AC
---
(%)
| SE
---
(%)
| SP
---
(%)
| F1
---
(%)
AUC
ECG | Reference | 91.96 | 92.31 | 91.36 | 91.75 | 0.9183
| Contact
---
PPG
Reference | 91.33 | 89.79 | 94.67 | 89.50 | 0.9223
Remote PPG | 3DCNN-PEAK | 88.43 | 90.04 | 87.04 | 88.78 | 0.8854
3DCNN-BVP [yu2019remote] | 83.18 | 84.35 | 82.22 | 83.47 | 0.8329
Shi et al. [shi2019atrial] | 79.39 | 78.64 | 77.39 | 76.61 | 0.7801
POS [wang2016algorithmic] | 77.76 | 64.95 | 79.33 | 67.34 | 0.7214
| PBV [de2014improved] | 73.68 | 64.36 | 83.46 | 66.98 | 0.7391

  * •

ECG = electrocardiography, PPG = photoplethysmography. SR = sinus rhythm. AFL
= atrial flutter, AC = accuracy, SE = Sensitivity, SP = Specificity, AUC =
area under curve.

###  5.3 Results for Computational Speed

Our experiments were conducted on an Intel Xeon E5-2650 2.30 GHz CPU and
Nvidia Tesla V100 GPU. The cropped face is first extracted from the original
face videos by OpenFace [baltrusaitis2018openface]. The running speed of this
step is 30 fps. We train our 3DCNN model with the whole OBF dataset (about 31
hours of face videos) for 45 epochs, which costs about 45 hours. The inference
time of the model is about 2 ms ($9\times 10^{5}$ fps) for one 30s video clip.
The training of SVM classification takes 60 ms. For each 30s clip, the
inference time of SVM classification is about 1.2 ms ($1.5\times 10^{6}$ fps).

###  5.4 Discussion

From experiment I: remote PPG monitoring, there are three points we can
conclude from Table 5, 6, and 7. 1) The evaluation of OBF-P, OBF-H, and the
whole OBF dataset shows that our proposed method can achieve the best
performance of remote PPG monitoring on both healthy subject data and patient
data. 2) Patient data is more challenging than healthy subject data, but the
proposed 3DCNN-PEAK can achieve better results than other baselines on patient
data. The results demonstrate that the proposed method with learning systolic
peaks effectively improves remote PPG monitoring performance for patients. 3)
Training with the whole OBF dataset can perform better than training with only
OBF-P or OBF-H. Therefore, we use the remote PPG signals from the model
trained on the whole OBF dataset for our classification tasks.

From experiment II: AF/AFL detection, two points can be observed from the
results. First, the ECG and contact PPG achieve high accuracy, which validates
that the HRV features are effective for AF/AFL detection. Second, our proposed
3DCNN-PEAK achieves slightly lower performance than the contact PPG, but it
works best among all remote PPG methods, which indicates that learning
systolic peaks can facilitate the AF/AFL detection.

We also discuss the performance difference among healthy vs. AF, SR vs. AF,
and SR vs. AFL. It can be seen that the results of SR vs. AF are slightly
lower than that of healthy vs. AF, which means the SR vs. AF classification
task is more challenging. This is expected since both AF and SR samples are
from the patients, while the classification of AF vs. healthy uses data from
two different groups of subjects whose data is more heterogeneous. In
addition, results in Experiment I show that patient data is more challenging
and has lower remote PPG monitoring performance, which may cause low
classification performance between SR patients and AF patients. It is noted
that the performance of all methods for SR vs. AFL is lower than healthy vs.
AF or SR vs. AF. There are two reasons. 1) There is only a limited number of
AFL patients in our dataset, and the numbers of SR (61 subjects) and AFL (11
subjects) are highly unbalanced, which might cause the performance decreasing.
A larger AFL dataset could give a performance improvement and more reliable
evaluation. 2) It was reported that adding PPG morphology features helped
detect AFL compared to only using HRV features [eerikainen2019detecting].
Since only HRV features are used in our experiment, future works could focus
on extracting the remote PPG morphology from face videos to improve AFL
detection.

We checked some cases when the detector fails. One reason is that some facial
videos contain spontaneous facial motions that lead to noises in remote PPG.
Another reason is that some PPG signals from patients contain systolic peaks
with very low amplitudes due to small blood stroke volume [valiaho2019wrist].
The two factors may cause wrongly detected peaks and eventually lead to wrong
classification.

Our study may lead to healthcare products, e.g., self-screening of AF symptoms
of suspectable populations at home or self-monitoring for the chronic patients
to check for recurrence of AF after treatment. Non-contact AF detection can
also be used in public places for rapids screening of AF since this method
does not need time to attach sensors to the subject, and the computational
speed is high enough for real-time application. In addition, the non-contact
way is more hygienic without contact sensors.

##  6 Conclusion

In this paper, we propose a novel method for non-contact AF detection by
utilizing the systolic peaks to train our deep learning model. A large-scale
OBF dataset was collected for non-contact AF detection from face videos. We
achieve high accuracy for non-contact AF detection and demonstrate the
feasibility of non-contact AFL detection.

## Acknowledgements

The authors would like to thank Jingang Shi, Iman Alikhani, and Zitong Yu from
University of Oulu for their important contributions in data collection. The
authors wish to acknowledge CSC – IT Center for Science, Finland, for
computational resources.


# The conditional measures for the determinantal point process with the
Bergman kernel

Alexander I. Bufetov  CNRS, Aix-Marseille Université, Centrale Marseille,
Institut de Mathématiques de Marseille, UMR7373, 39 Rue F. Joliot Curie 13453,
Marseille, France;
Steklov Mathematical Institute of RAS, Moscow, Russia;
Institute for Information Transmission Problems, Moscow, Russia.
<EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract.

This note gives an explicit description of conditional measures for the
determinantal point process with the Bergman kernel.

##  1\. Introduction

###  1.1. Formulation of the main result

The aim of this note is to give an explicit formula for the conditional
measures of the zero set of the Gaussian Analytic Function under the condition
that the configuration be fixed in the complement of a compact set. Recall
that, by the Peres-Viràg theorem [18], the zero set of the Gaussian Analytic
Function is a determinantal point process with the Bergman kernel, cf. (2)
below. The main tool is the explicit representation obtained in [5] of the
Radon—Nikodym derivative of the reduced Palm measure of our determinantal
point process with respect to the process itself; the Radon—Nikodym derivative
is found as a generalized multiplicative functional corresponding to the
divergent Blaschke product over the particles of our configuration.

We proceed to the precise formulations. Let $a_{n}(\omega)$, $n\geq 0$, be
independent standard complex Gaussian random variables, with expectation $0$
and variance $1$. The power series

(1) $\sum_{n=0}^{\infty}a_{n}(\omega)z^{n}$

almost surely has radius of convergence $1$; the zero set of the series (1) is
a configuration on the unit disc whose law $\mathbb{P}_{K}$, by the
Peres—Viràg theorem [18], is the determinantal measure on the space
$\mathrm{Conf}(\mathbb{D})$ of configurations on $\mathbb{D}$ corresponding to
the Bergman kernel

(2) $K(z,w)=\frac{1}{\pi(1-z\bar{w})^{2}},\quad z,w\in\mathbb{D}$

of orthogonal projection in the space $L_{2}(\mathbb{D})$ of square-integrable
functions with respect to the usual Lebesgue measure onto the closed subspace
of square-integrable holomorphic functions.

Consider a decomposition $\mathbb{D}=B\sqcup C$ of the unit disc into two
disjoint Borel sets with $B$ open and having compact closure in $\mathbb{D}$.
The natural restriction map $\pi_{C}\colon X\mapsto X\cap C$ sends the measure
$\mathbb{P}_{K}$ forward to its projection $\overline{\mathbb{P}_{K}^{C}}$;
the $\overline{\mathbb{P}_{K}^{C}}$-almost surely defined conditional measures
of $\mathbb{P}_{K}$ for a configuration $Y$ on $\mathrm{Conf}(\mathbb{D})$
satisfying $Y=Y\cap C$ on the preimage $\pi_{C}^{-1}(Y)$ is denoted
$\mathbb{P}(\,\cdot\,|C;Y)$. Lemma 1.11 in [7] states that if ${\mathbb{P}}$
is a determinantal point process induced by a positive Hermitian contraction,
then so is its conditional measure; Lemma 1.11 also gives a limit procedure
for finding the kernel governing the conditional measure. Our aim in this note
is to give an explicit formula, see (6) below, for the conditional measure
$\mathbb{P}(\,\cdot\,|C;Y)$. The starting point for the argument is Theorem
1.4 in [5] that gives an explicit expression for the Radon-Nikodym derivative
of the Palm measure ${\mathbb{P}}_{K}^{q}$ of ${\mathbb{P}}_{K}$ with respect
to ${\mathbb{P}}_{K}$; the Radon-Nikodym derivative, cf. (10) below, is
expressed in terms of a regularized multiplicative functional
$\overline{\Psi}_{q}$, cf. (5) below, that we now write in a slightly
different way. The next step is an expression of the conditional measure
$\mathbb{P}(\,\cdot\,|C;Y)$ in terms of the multiplicative functional
$\overline{\Psi}_{q}$.

We consider the unit disc as the Poincaré model for the Lobachevsky plane, and
for $p\in\mathbb{D}$, $R>0$ we let $D(p,R)$ stand for the Lobachevskian ball
of radius $R$ centred at $p$.

######  Proposition 1.1.

For $\mathbb{P}_{K}$-almost every $X\in\mathrm{Conf}(\mathbb{D})$ and any
$q\in\mathbb{D}$, the limit

(3) $\tilde{\Psi}_{q}(X)=\\\ =\lim_{R\to\infty}\prod_{x\in X\cap
D(q,R)}\biggl{|}\frac{x-q}{1-\bar{x}q}\biggr{|}^{2}\cdot\exp\biggl{(}\frac{\sqrt{-1}}{2\pi}\int\limits_{D(q,R)}\biggl{(}1-\biggl{|}\frac{z-q}{1-\bar{z}q}\biggr{|}^{2}\biggr{)}\frac{dz\wedge
d\bar{z}}{(1-|z|^{2})}\biggr{)}$

exists in $L_{1}(\mathrm{Conf}(\mathbb{D}),\mathbb{P}_{K})$ as well as
$\mathbb{P}_{K}$-almost surely along a subsequence.

We have

$\int\tilde{\Psi}_{q}(X)\,d\mathbb{P}_{K}(X)=\frac{e^{\gamma-1}}{2},$

where

(4) $\gamma=\lim_{n\to\infty}\Bigl{(}1+\frac{1}{2}+\dots+\frac{1}{n}-\log
n\Bigr{)}$

is the Euler—Mascheroni constant.

Denote

(5) $\overline{\Psi}_{q}(X)=2e^{1-\gamma}\tilde{\Psi}_{q}(X).$

We are now ready to proceed to the formulation of the main result of this
note, an explicit description of the conditional measures of the determinantal
point process with the Bergman kernel.

######  Theorem 1.2.

For $\overline{\mathbb{P}_{K}^{C}}$-almost every configuration $Y$, the
conditional measure $\mathbb{P}_{K}(\,\cdot\,|C;Y)$ has the form

(6) $\eta_{Y,0}\biggl{(}1+\sum_{m=1}^{\infty}\mathop{det}\nolimits
L_{Y}(q_{j},q_{k})_{j,k=1,\dots,m}\cdot\prod_{j=1}^{m}\biggl{(}\frac{\sqrt{-1}}{2\pi}\frac{dq_{j}\wedge
d\bar{q}_{j}}{(1-|q_{j}|^{2})^{2}}\biggr{)}\biggr{)},$

where

(7)
$L_{Y}(q_{1},q_{2})=\frac{\overline{\Psi}_{q_{1}}(Y)\overline{\Psi}_{q_{2}}(Y)}{1-q_{1}\bar{q}_{2}}$

and

$\eta_{Y,0}=\frac{1}{\mathop{det}\nolimits(1+L_{Y})}=\mathbb{P}(\\#_{B}=0\mid
C;Y)$

is the conditional probability that there are no particles in $B$.

Equivalently, the conditional measure is an L-ensemble in the sense of Borodin
[1], and the L-kernel is given by the formula (7).

###  1.2. Palm Measures of the Point Process ${\mathbb{P}}_{K}$.

####  1.2.1. Correlation functions of point processes

Let $E$ be a Polish space. A configuration on $E$ is a countable or finite
collection of points in $E$, called particles, considered without regard to
order and subject to the additional requirement that every compact set contain
only finitely many particles of a configuration. Let ${\mathrm{Conf}}(E)$ be
the space of configurations on $E$. For a bounded Borel set $B\subset E$, let

$\\#_{B}\colon{\mathrm{Conf}}(E)\to\mathbb{N}\cup\\{0\\}$

be the function that to a configuration assigns the number of its particles
belonging to $B$. The random variables $\\#_{B}$ over all bounded Borel sets
$B\subset E$ determine the Borel sigma-algebra on ${\mathrm{Conf}}(E)$. A
Borel probability measure ${\mathbb{P}}$ on ${\mathrm{Conf}}(E)$ is called a
point process with phase space $E$. Recall that the point process
${\mathbb{P}}$ is said to admit correlation measures of order $l$ if for any
continuous compactly supported function $\varphi$ on $E^{l}$ the functional

$\sum\limits_{x_{1},\dots,x_{l}\in X}\varphi(x_{1},\dots,x_{l})$

is ${\mathbb{P}}$-integrable; the sum is taken over all ordered $l$-tuples of
distinct particles in $X$. The $l$-th correlation measure $\rho_{l}$ of the
point process ${\mathbb{P}}$ is then defined by the formula

$\mathbb{E}_{{\mathbb{P}}}\left(\sum\limits_{x_{1},\dots,x_{l}\in
X}\varphi(x_{1},\dots,x_{l})\right)=\displaystyle\int\limits_{E^{l}}\varphi(q_{1},\dots,q_{l})d\rho_{l}(q_{1},\dots,q_{l}).$

If all correlation measures of a point process are well-defined and for any
$m\in\mathbb{N}$ the $m$-th correlation measure is absolutely continuous with
respect to $m$-th tensor power of the first one, then say that our point
process admits correlation functions of all orders.

####  1.2.2. Campbell and Palm Measures.

Following Kallenberg [13], Daley–Vere-Jones [8], we recall the definition of
Campbell measures of point processes; the notation follows [4]. Let
${\mathbb{P}}$ be a point process on $E$ admitting the first correlation
measure $\rho_{1}^{{\mathbb{P}}}$. The Campbell measure ${\EuScript
C}_{{\mathbb{P}}}$ of ${\mathbb{P}}$ is a sigma-finite measure on
$E\times{\mathrm{Conf}}(E)$ such that for any Borel subsets $B\subset E$,
${\mathscr{Z}}\subset{\mathrm{Conf}}(E)$ we have

${\EuScript
C}_{{\mathbb{P}}}(B\times{\mathscr{Z}})=\displaystyle\int\limits_{{\mathscr{Z}}}\\#_{B}(X)d{\mathbb{P}}(X).$

The Palm measure ${\hat{\mathbb{P}}}^{q}$ is the canonical conditional
measure, in the sense of Rohlin [19], of the Campbell measure
${\mathcal{C}}_{{\mathbb{P}}}$ with respect to the measurable partition of the
space $E\times{\mathrm{Conf}}(E)$ into subsets
$\\{q\\}\times{\mathrm{Conf}}(E)$, $q\in E$, cf. [4]. By definition, the Palm
measure ${\hat{\mathbb{P}}}^{q}$ is supported on the subset of configurations
containing a particle at position $q$. Removing these particles, one defines
the reduced Palm measure ${\mathbb{P}}^{q}$ as the push-forward of the Palm
measure ${\hat{\mathbb{P}}}^{q}$ under the erasing map $X\to
X\setminus\\{q\\}$.

Iterating the definition, one arrives at iterated Campbell, Palm and reduced
Palm measures: the $r$-th Campbell measure ${\EuScript C}^{r}_{{\mathbb{P}}}$
of ${\mathbb{P}}$ is a sigma-finite measure on the product
$E\times\dots\times\mathbb{E}\times{\mathrm{Conf}}(E)$ of $r$ copies of $E$
and ${\mathrm{Conf}}(E)$ such that for any disjoint Borel subsets
$B_{1},\dots,B_{r}\subset E$, ${\mathscr{Z}}\subset{\mathrm{Conf}}(E)$ we have

${\EuScript C}_{{\mathbb{P}}}(B_{1}\times\dots\times
B_{r}\times{\mathscr{Z}})=\displaystyle\int\limits_{{\mathscr{Z}}}\\#_{B_{1}}(X)\dots\\#_{B_{r}}(X)d{\mathbb{P}}(X).$

Given distinct $q_{1},\dots,q_{r}\in E$, the Palm measure
${\hat{\mathbb{P}}}^{q_{1},\dots,q_{r}}$ is the canonical conditional measure,
in the sense of Rohlin [19], of the Campbell measure
${\mathcal{C}}^{r}_{{\mathbb{P}}}$ with respect to the measurable partition of
the space $E\times\dots\times E\times{\mathrm{Conf}}(E)$ into subsets
$\\{q_{1},\dots,q_{r}\\}\times{\mathrm{Conf}}(E)$, $q\in E$, cf. [4]. By
definition, the Palm measure ${\hat{\mathbb{P}}}^{q_{1},\dots,q_{r}}$ is
supported on the subset of configurations containing a particle at each
position $q_{1},\dots,q_{r}$. Removing these particles, one defines the
reduced Palm measure ${\mathbb{P}}^{q_{1},\dots,q_{r}}$ as the push-forward of
the Palm measure ${\hat{\mathbb{P}}}^{q}$ under the erasing map $X\to
X\setminus\\{q_{1},\dots,q_{r}\\}$; see Kallenberg [13], whose formalism is
also adopted in [4], for a more detailed exposition. As all conditional
measures, reduced Palm measures ${\mathbb{P}}^{q}$ are a priori only defined
for $\rho_{1}$-almost every $q$. In our context of determinantal point
processes, for any distinct $q_{1},\dots,q_{m}\in E$, the Shirai-Takahashi
Theorem allows us to fix a convenient explicit Borel realization
${\mathbb{P}}^{q_{1},\dots,q_{m}}$ of reduced Palm measures.

####  1.2.3. Determinantal Point Processes

As before, let $E$ be a Polish space, and let $\mu$ be a sigma-finite Borel
measure on $E$. Recall that a Borel probability measure $\mathbb{P}$ on
${\mathrm{Conf}}(E)$ is called determinantal if there exists a locally trace
class operator $K$ acting in $L_{2}(E,\mu)$ such that for any bounded
measurable function $g$, for which $g-1$ is supported in a bounded set $B$, we
have

(8) $\mathbb{E}_{\mathbb{P}}\prod\limits_{x\in
X}g(x)=\mathop{det}\nolimits\biggl{(}1+(g-1)K\chi_{B}\biggr{)}.$

Here and elsewhere in similar formulas, $1$ stands for the identity operator.
The Fredholm determinant in (8) is well-defined since $K$ is locally of trace
class. The equation (8) determines the measure ${\mathbb{P}}$ uniquely. We use
the notation ${\mathbb{P}}_{K}$ for the determinantal measure induced by the
operator $K$. By a theorem due to Macchì and Soshnikov [15], [24] and Shirai-
Takahashi [20], any Hermitian positive contraction that belongs to the local
trace class defines a determinantal point process.

####  1.2.4. Generalized multiplicative functionals

Let $g$ be a Borel function on $E$ and let $\Psi$ be a Borel function defined
on a Borel subset $\mathscr{Z}\subset\mathrm{Conf}(E)$ and satisfying the
following: if $X,Y\in\mathscr{Z}$ and there exist distinct particles
$p_{1},\dots,p_{r},q_{1},\dots,q_{s}\in E$ such that
$X\setminus\\{p_{1},\dots,p_{r}\\}=Y\setminus\\{q_{1},\dots,q_{s}\\}$, then

(9) $\Psi(X)=\frac{g(p_{1})\cdots g(p_{r})}{g(q_{1})\cdots g(q_{s})}\Psi(Y).$

In this case we say that $\Psi$ is a generalized multiplicative functional
corresponding to the function $g$. The regularized multiplicative functional
(3) is a particular case of a generalized multiplicative functional. If the
point process $\mathbb{P}$ has trivial tail $\sigma$-algebra, then a
generalized multiplicative functional corresponding to a function $g$,
provided it exists, is $\mathbb{P}$-almost surely unique up to multiplication
by a constant.

####  1.2.5. The Characterization of Palm Measures for $\mathbb{P}_{K}$

The starting point for the argument is Theorem 1.4 in [5] that, in view of
Proposition 1.1, can be formulated as follows: for any $q\in\mathbb{D}$, the
reduced Palm measure $\mathbb{P}_{K}^{q}$ of our determinantal point process
${\mathbb{P}}_{K}$ with the Bergman kernel is given by the formula

(10) $\mathbb{P}_{K}^{q}=\overline{\Psi}_{q}\mathbb{P}_{K}.$

In other words, the Radon-Nikodym derivative is given by a regularized
multiplicative functional. The argument is completed by a general proposition
describing the conditional measures for a point process whose Palm measures
are expressed as a product of the original measure and a multiplicative
functional.

###  1.3. Palm Measures, Multiplicative Functionals and Conditional Measures.

Let ${\mathbb{P}}$ be a point process with phase space $E$ admitting
correlation functions of all orders. We fix a Borel realization
${\mathbb{P}}^{q}$ of its reduced Palm measures and assume that there exists a
symmetric positive Borel function $B(q_{1},q_{2})$, $q_{1},q_{2}\in E$,
defined on $E\times E$ and such that for any $q\in E$ the Radon—Nikodym
derivative $d\mathbb{P}^{q}/d\mathbb{P}$ is a generalized multiplicative
functional corresponding to the function $B(q,\,\cdot\,)$.

######  Proposition 1.3.

For any decomposition $E=B\sqcup C$ into two Borel sets with
$\rho_{1}^{{\mathbb{P}}}(B)<+\infty$ and for
$\overline{\mathbb{P}_{C}}$-almost every $Y\in\mathrm{Conf}(E;C)$ the
conditional measure $\mathbb{P}_{Y}=\mathbb{P}(\,\cdot\,|Y,C)$ has the form

$\eta_{Y,0}\biggl{(}1+\sum_{m=1}^{\infty}\prod_{1\leq i<j\leq
m}B(q_{i},q_{j})\cdot\prod\limits_{i=1}^{m}\frac{d\mathbb{P}^{q_{i}}}{d\mathbb{P}}(Y)\,d\rho_{1}(q_{i})\biggr{)},$

where $\eta_{Y,0}=\mathbb{P}(\\#_{B}(X)=0|Y;C)$.

Proposition 1.3, together with Proposition 1.1, directly implies Theorem 1.2
in view of the Cauchy identity

$\displaystyle\mathop{det}\nolimits\left(\displaystyle\frac{1}{1-q_{j}\overline{q}_{k}}\right)_{j,k=1,\dots,n}\
=\ \displaystyle\frac{\displaystyle\prod\limits_{1\leq j<k\leq
n}\big{|}q_{j}-q_{k}\big{|}^{2}}{\displaystyle\prod\limits_{1\leq j<k\leq
n}\big{|}1-q_{j}\overline{q}_{k}\big{|}^{2}}.$

It remains to prove Proposition 1.1 and Proposition 1.3.

##  2\. Proof of Proposition 1.3

###  2.1. Conditional measures of point processes.

Let $E$ be a locally compact complete metric space, let ${\mathrm{Conf}}(E)$
be the space of configurations on $E$. Given a configuration
$X\in\textrm{Conf}(E)$ and a subset $C\subset E$, we let $X|_{C}$ stand for
the restriction of $X$ onto the subset $C$. We let $\overline{\mathbb{P}}^{C}$
be the push-forward measure under the natural projection $X\to X|_{C}$, Given
a point process on $E$, that is, a Borel probability measure ${\mathbb{P}}$ on
${\mathrm{Conf}}(E)$, the measure $\mathbb{P}(\cdot|X;C)$ on
${\mathrm{Conf}}(E\setminus C)$ is defined as the conditional measure of
$\mathbb{P}$ with respect to the condition that the restriction of our random
configuration onto $C$ coincides with $X|_{C}$. More formally, we consider the
surjective restriction mapping $X\to X|_{C}$ from $\textrm{Conf}(E)$ to
$\textrm{Conf}(C)$. Fibres of this mapping can be identified with
$\mathrm{Conf}(E\backslash C)$ and conditional measures, in the sense of
Rohlin [19], are precisely the measures $\mathbb{P}(\cdot|X;C)$. Let
${\mathrm{Conf}}(E;C)$ be the subset of those configurations on $E$ all whose
particles lie in $C$; in other words, the image of the natural projection
$X\to X|_{C}$. By definition, we have

${\mathbb{P}}=\int\limits_{{\mathrm{Conf}}(E;C)}\mathbb{P}(\cdot|Y;C)d\overline{\mathbb{P}}^{C}(Y).$

The decomposition into conditional measures is by definition lifted onto the
level of Campbell measures:

(11) ${\EuScript
C}_{{\mathbb{P}}}=\int\limits_{{\mathrm{Conf}}(E;C)}{\EuScript
C}_{\mathbb{P}(\cdot|Y;C)}d\overline{\mathbb{P}}^{C}(Y)$

###  2.2. Palm measures of different orders

As before, let $\mathbb{P}$ be a point process on a Polish space $E$. We
assume that the point process ${\mathbb{P}}$ admits correlation functions of
all orders and that the reduced Palm measures of ${\mathbb{P}}$ are almost
surely absolutely continuous with respect to ${\mathbb{P}}$; it follows that
reduced Palm measures of all orders are also almost surely absolutely
continuous with respect to ${\mathbb{P}}$; almost surely is here understood
with respect to the first correlation measure. It is convenient to think that
the space $E$ is endowed with a sigma-finite Borel measure $\mu$ such that the
first correlation measure of ${\mathbb{P}}$ is absolutely continuous with
respect to $\mu$; the $m$-th correlation measure of ${\mathbb{P}}$ then has
the form $\rho_{m}(q_{1},\dots,q_{m})d\mu(q_{1})\dots d\mu(q_{m})$, where
$\rho_{m}$ is the $m$-th correlation function.

For $\mu$-almost any distinct points $p_{1},\dots,p_{m},q_{1},\dots,q_{r}$ and
$\mathbb{P}$-almost any configuration $X\in\mathrm{Conf}(E)$ not containing
any of the points $p_{1},\dots,p_{m}$, $q_{1},\dots,q_{r}$ the following
identity directly follows from the definition of the Palm measures:

(12)
$\frac{\rho_{m+r}(p_{1},\dots,p_{m},q_{1},\dots,q_{r})}{\rho_{r}(q_{1},\dots,q_{r})}\cdot\frac{d\mathbb{P}^{p_{1},\dots,p_{m},q_{1},\dots,q_{r}}}{d\mathbb{P}^{q_{1},\dots,q_{r}}}(X)={}\\\
{}=\rho_{m}(p_{1},\dots,p_{m})\frac{d\mathbb{P}^{p_{1},\dots,p_{m}}}{d\mathbb{P}}(X,q_{1},\dots,q_{r}),$

###  2.3. Palm measures and conditional measures

As before, we consider a point process ${\mathbb{P}}$ on the phase space $E$;
the point process ${\mathbb{P}}$ is assumed to admit correlation functions of
all orders. Consider a decomposition $E=B\sqcup C$ of our phase space $E$ as a
disjoint union of two Borel sets. As before, we let
$\overline{\mathbb{P}}^{C}$ be the push-forward measure under the natural
projection $X\to X|_{C}$, and, for a configuration $Y$ all whose particles lie
in $C$, in this subsection, we write
$\mathbb{P}_{[Y,C]}=\mathbb{P}(\,\cdot\,|Y;C)$. We take a natural $m$ and let
${\mathbb{P}}$ be a point process whose reduced Palm measures of order $m$ are
$\rho_{m}^{{\mathbb{P}}}$-almost surely absolutely continuous with respect to
${\mathbb{P}}$. From (11) it follows that, $\overline{\mathbb{P}}^{C}$-almost
surely, the $m$-th correlation measure of the measure $\mathbb{P}_{[Y,C]}$ is
absolutely continuous with respect to the $m$-th tensor power of the first
correlation measure of ${\mathbb{P}}$ (note that for determinantal point
processes governed by Hermitian contractions this requirement is automatically
verified by Lemma 1.11 of [7] on the preservation of the determinantal
property under taking conditional measures; observe, however, that Lemma 1.11
is not used in this derivation of the explicit form of the conditional
measures). Let $\rho_{{[Y,C]},m}$ be the $m$-th correlation function of
$\mathbb{P}_{[Y,C]}$. By definition, we have

(13)
$\frac{\rho_{{[Y,C]},m}(q_{1},\dots,q_{m})\,d\mathbb{P}_{[Y,C]}^{q_{1},\dots,q_{m}}}{d\mathbb{P}_{[Y,C]}}(Z)=\frac{\rho_{m}(q_{1},\dots,q_{m})d\mathbb{P}^{q_{1},\dots,q_{m}}}{d\mathbb{P}}(Y\cup
Z).$

We now directly obtain the following

######  Corollary 2.1.

Assume that the first correlation measure of the set $B$ is finite. Then for
$\overline{\mathbb{P}}^{C}$-almost any configuration $Y$ the conditional
measure $\mathbb{P}_{[Y,C]}$ has the form

(14) $\sum_{m=0}^{\infty}\eta_{{[Y,C]},m}(q_{1},\dots,q_{m})\,d\mu(q_{1})\dots
d\mu(q_{m}),$

where $\eta_{{[Y,C]},0}=\mathbb{P}_{[Y,C]}(\varnothing)$ is the conditional
probability of the absence of particles in $B$ and

(15)
$\eta_{{[Y,C]},m}(q_{1},\dots,q_{m})=\eta_{{[Y,C]},0}\cdot\frac{\rho_{m}(q_{1},\dots,q_{m})d\mathbb{P}^{q_{1},\dots,q_{m}}}{d\mathbb{P}}(Y).$

###### Proof.

That the conditional measure $\mathbb{P}_{[Y,C]}$ is absolutely continuous
with respect to the Poisson process of intensity $\mu$ follows from the fact
that the measure $\mathbb{P}_{[Y,C]}$ is, $\overline{\mathbb{P}}_{C}$-almost
surely, supported on the set of configurations with finitely many particles
and that the $m$-th correlation measure of the measure $\mathbb{P}_{[Y,C]}$ is
absolutely continuous with respect to the $m$-th tensor power of the first
correlation measure of ${\mathbb{P}}$. Now take a general measure of the form
(14) for some Borel functions $\eta_{{[Y,C]},m}$, $m=1,\dots$; the $r$-th Palm
measure of our measure at the particles $p_{1},\dots,p_{r}$ takes the form

$M^{-1}(p_{1},\dots,p_{r})\sum_{m=0}^{\infty}\eta_{{[Y,C]},m+r}(p_{1},\dots,p_{r},q_{1},\dots,q_{m})\,d\mu(q_{1})\dots
d\mu(q_{m}),$

where $M(p_{1},\dots,p_{r})$ is a normalization constant. We now write (13)
with $Z=\varnothing$. By definition,

$\rho_{{[Y,C]},m}(q_{1},\dots,q_{m})\cdot\mathbb{P}_{[Y,C]}^{q_{1},\dots,q_{m}}(\varnothing)=\eta_{{[Y,C]},m}(q_{1},\dots,q_{m}),$

and the desired equality (15) follows. ∎

###  2.4. Conclusion of the proof of Proposition 1.3.

Iterating (12), we arrive at the identity

(16)
$\rho_{m}(p_{1},\dots,p_{m})\frac{d\mathbb{P}^{p_{1},\dots,p_{m}}}{d\mathbb{P}}(X)={}\\\
{}=\rho_{1}(p_{1})\frac{d\mathbb{P}^{p_{1}}}{d\mathbb{P}}(X,p_{2},\dots,p_{m})\cdots\rho_{1}(p_{m-1})\frac{d\mathbb{P}^{p_{m-1}}}{d\mathbb{P}}(X,p_{m})\cdot\rho_{1}(p_{m})\frac{d\mathbb{P}^{p_{m}}}{d\mathbb{P}}(X).$

Combining with (15), we obtain the expression

$\eta_{Y,m}(q_{1},\dots,q_{m})={}\\\
{}=\eta_{Y,0}\cdot\rho_{1}(q_{1})\cdots\rho_{m}(q_{m})\cdot\frac{d\mathbb{P}^{q_{1}}}{d\mathbb{P}}(Y)\cdot\frac{d\mathbb{P}^{q_{2}}}{d\mathbb{P}}(q_{1},Y)\cdots\frac{d\mathbb{P}^{q_{m}}}{d\mathbb{P}}(q_{1},\dots,q_{m-1},Y).$

Substituting the expression of the Radon-Nikodym derivative as the
multiplicative functional completes the proof. ∎

##  3\. Proof of Proposition 1.1

Recall that the Hilbert-Carleman regularization $\mathop{det}\nolimits_{2}$ of
the Fredholm determinant is introduced on finite rank operators by the formula

(17)
$\mathop{det}\nolimits_{2}(1+A)=\exp(-{\mathrm{tr}}A)\mathop{det}\nolimits(1+A)$

and then extended by continuity onto the space of Hilbert-Schmidt operators.

We first recall a well-known observation (cf. e.g. [17]). Let
$g\colon\mathbb{D}\to\mathbb{R}_{+}$ be a nonnegative bounded Borel radial
function, in other words, a function depending only on the absolute value of
its argument. The eigenfunctions of the operator $\sqrt{g}K\sqrt{g}$ are
precisely the functions $\sqrt{g}z^{k}$, $k\geq 0$, and the corresponding
eigenvalue is

$\frac{k+1}{\pi}\int_{\mathbb{D}}g(z)|z|^{2k}\,dz=(k+1)\int_{0}^{1}\tilde{g}(\rho)\rho^{k}\,d\rho,$

where $\tilde{g}(\rho)=g(\sqrt{\rho}e^{i\theta})$ for any $\theta$.

It directly follows that the operator

(18) $K_{1}(z,w)=\sqrt{1-|z|^{2}}K(z,w)\sqrt{1-|w|^{2}}$

is Hilbert—Schmidt, as its eigenvalues are

$(k+1)\int_{0}^{1}(1-\rho)\rho^{k}\,d\rho=\frac{1}{k+2},\quad k=0,1,\dots.$

By definition (3), we have

(19)
$\mathbb{E}_{\mathbb{P}_{K}}\tilde{\Psi}=\mathop{det}\nolimits\nolimits_{2}(1+K_{1}).$

Note here that

$\frac{1}{\pi(1-|z|^{2})}=(1-|z|^{2})K(z,z).$

For $r\in(0,1)$, set

$\tilde{\Psi}_{r}(X)=\prod_{x\in
X,|x|<r}|x|^{2}\cdot\exp\biggl{(}\frac{\sqrt{-1}}{2\pi}\int_{\\{z:|z|<r\\}}\frac{dz\wedge
d\bar{z}}{(1-|z|^{2})}\biggr{)}.$

By definition, we have

$\mathbb{E}_{\mathbb{P}_{K}}\tilde{\Psi}_{r}=\mathop{det}\nolimits\nolimits_{2}(1+\chi_{\\{z:|z|<r\\}}K_{1}\chi_{\\{z:|z|<r\\}}).$

Since

$\chi_{\\{z:|z|<r\\}}K_{1}\chi_{\\{z:|z|<r\\}}\to K_{1}$

in the Hilbert—Schmidt norm as $r\to 1$, writing the Cauchy-Bunyakovsky-
Schwartz inequality

$\mathbb{E}_{\mathbb{P}_{K}}\left|\tilde{\Psi}-\tilde{\Psi}_{r}\right|\leq\sqrt{\mathbb{E}_{\mathbb{P}_{K}}\left|\tilde{\Psi}_{r}\right|^{2}}\sqrt{\mathbb{E}_{\mathbb{P}_{K}}\left|\tilde{\Psi}\tilde{\Psi}_{r}^{-1}-1\right|^{2}}$

one directly checks the relation

$\lim\limits_{r\to
1}\mathbb{E}_{\mathbb{P}_{K}}\left|\tilde{\Psi}-\tilde{\Psi}_{r}\right|=0,$

which is to say that the limit

$\tilde{\Psi}=\lim_{r\to 1}\tilde{\Psi}_{r}$

exists in $L_{1}(\mathrm{Conf}(\mathbb{D}),\mathbb{P}_{K})$ as well as almost
surely along a subsequence.

We now compute the right-hand side of (19). In order to do so, we let
$K^{(n)}$ be the orthogonal projection onto $\\{1,z,\dots,z^{n}\\}$ in
$L_{2}(\mathbb{D})$ and write

$K_{1}^{(n)}=\sqrt{1-|z|^{2}}K^{(n)}\sqrt{1-|w|^{2}}.$

We have $K_{1}^{(n)}\to K_{1}$ in the Hilbert—Schmidt norm as $n\to\infty$,
whence

$\mathop{det}\nolimits\nolimits_{2}(1+K_{1})=\lim_{n\to\infty}\mathop{det}\nolimits(1+K_{1}^{(n)})\times\exp\biggl{(}-{\mathrm{tr}}K_{1}^{(n)}\biggr{)}.$

By definition, we have

$\mathop{det}\nolimits(1+K_{1}^{(n)})=\prod_{k=0}^{n}\Bigl{(}1+\frac{1}{k+2}\Bigr{)}=\frac{n+3}{2}.$

and

${\mathrm{tr}}(K_{1}^{(n)})=\frac{1}{2}+\dots+\frac{1}{n+1}.$

Summing up, cf. (4), we obtain

$\mathop{det}\nolimits\nolimits_{2}(1+K_{1})=\frac{e^{\gamma-1}}{2}.$

We therefore obtain an alternative representation of the Palm measure
$\mathbb{P}_{K}^{0}$ with respect to the original measure:

$\frac{d\mathbb{P}_{K}^{0}}{d\mathbb{P}}(X)=\frac{e^{\gamma-1}}{2}\lim\limits_{r\to
1}\prod\limits_{x\in
X:|x|<r}|x|^{2}\exp\biggl{(}\frac{\sqrt{-1}}{2\pi}\int\limits_{\\{z:|z|<r\\}}\frac{dz\wedge
d\bar{z}}{1-|z|^{2}}\biggr{)}.$

Setting $D(z,R)$ to be the Lobachevskian ball centred at $z$ and of
Lobachevskian radius $R$, for any $q\in\mathbb{D}$ we rewrite

$\frac{d\mathbb{P}_{K}^{q}}{d\mathbb{P}}(X)=\frac{e^{\gamma-1}}{2}\lim\limits_{R\to\infty}\prod\limits_{x\in
D(q,R)\cap X}\biggl{|}\frac{x-q}{1-\bar{q}x}\biggr{|}^{2}\times\\\
\times\exp\biggl{(}\frac{\sqrt{-1}}{2}\int\limits_{D(q,R)}\biggl{(}1-\biggl{|}\frac{z-q}{1-\bar{q}z}\biggr{|}^{2}\biggr{)}\biggr{)}K(z,z)\,dz\wedge
d\bar{z}.$

Proposition 1.1 is proved. Theorem 1.2 is proved completely. ∎

Acknowledgements. I am deeply grateful to Dmitrii Khliustov, Alexey Klimenko
and Yanqi Qiu for useful discussions. Part of this work was done during a
visit to the Scuola Internazionale degli Studi Superiori Avanzati in Trieste
and to the Alma Mater Studiorum–University of Bologna. I am deeply grateful to
both institutions for their warm hospitality. This research has received
funding from the European Research Council (ERC) under the European Union’s
Horizon 2020 research and innovation programme, grant agreement No 647133
(ICHAOS), as well as from the ANR grant ANR-18-CE40-0035 REPKA.

## References

  * [1] Alexei Borodin, Determinantal point processes, Oxford Handbook of Random Matrix Theory, Edited by Gernot Akemann, Jinho Baik, and Philippe Di Francesco, Oxford University Press, 2015, pp.231-250.
  * [2] A.M. Borodin, E.M. Rains, Eynard-Mehta theorem, Schur process, and their pfaffian analogs. J. Stat. Phys. 121 (2005), 291–317.
  * [3] A. I. Bufetov, Conditional measures of determinantal point processes, arXiv:1605.01400, Functional Analysis and its Applications, 54:1 (2020), 7–20.
  * [4] A. I. Bufetov. Quasi-Symmetries of Determinantal Point Processes, arxiv:1409.2068, Ann. Probab. 46(2018), 956-1003 .
  * [5] A. I. Bufetov and Y. Qiu, Determinantal point processes associated with Hilbert spaces of holomorphic functions. Commun. Math. Phys., 351(2017), no.1, 1-44.
  * [6] A. I. Bufetov and Y. Qiu, Conditional measures of generalized Ginibre point processes, Journal of Functional Analysis 272 (2017), 11, pp. 4671 – 4708.
  * [7] A.Bufetov, Y.Qiu, A. Shamov, Kernels of conditional determinantal measures, arxiv:1612.06751, to appear in the Journal of the European Mathematical Society, 2021, Volume 23, Issue 5, pp. 1477–1519.
  * [8] D.J.Daley, D. Vere-Jones, An introduction to the theory of point processes, vol.I-II, Springer Verlag 2008.
  * [9] S. Ghosh, Determinantal processes and completeness of random exponentials: the critical case, Probability Theory and Related Fields, December 2015, Volume 163, Issue 3 – 4, pp. 643 – 665.
  * [10] S. Ghosh, Y. Peres. Rigidity and tolerance in point processes: Gaussian zeros and Ginibre eigenvalues, Duke Math. J., Volume 166, Number 10 (2017), 1789 – 1858.
  * [11] S. Ghosh, Rigidity and Tolerance in Gaussian zeros and Ginibre eigenvalues: quantitative estimates, arXiv:1211.3506
  * [12] A.E. Holroyd, T. Soo, Insertion and deletion tolerance for point processes, Electron. J. Probab. 18, 1 – 24, 2013.
  * [13] O. Kallenberg, Random Measures. Akademie-Verlag, Berlin, 1986.
  * [14] R. Lyons, Determinantal probability measures. Pub. Mat. IHÉS. 98 (2003), 167–212.
  * [15] O. Macchi, The coincidence approach to stochastic point processes. Advances in Appl. Probability, 7 (1975), 83–122.
  * [16] G. Olshanski, The quasi-invariance property for the Gamma kernel determinantal measure. Advances in Mathematics, 2011\. Vol. 226. P. 2305 – 2350.
  * [17] H. Osada and T. Shirai, Absolute continuity and singularity of Palm measures of the Ginibre point process. Probab. Theory Related Fields, 165 (2016), no. 3-4, 725–770.
  * [18] Y. Peres and B. Virág. Zeros of the i.i.d. Gaussian power series: a conformally invariant determinantal process. Acta Mathematica, 194(1):1–35, 2005.
  * [19] Rohlin, V. A. On the fundamental ideas of measure theory. (Russian) Mat. Sbornik N.S. 25(67), (1949), 107 – 150.
  * [20] T.Shirai, Y. Takahashi, Fermion process and Fredholm determinant, Proceedings of the Second ISAAC Congress, vol. I, 15 – 23, Kluwer 2000.
  * [21] T. Shirai, Y. Takahashi, Random point fields associated with certain Fredholm determinants. I. Fermion, Poisson and boson point processes. J. Funct. Anal. 205 (2003), no. 2, 414–463.
  * [22] T. Shirai, Y. Takahashi, Random point fields associated with certain Fredholm determinants. II. Fermion shifts and their ergodic and Gibbs properties. Ann. Probab. 31 (2003), no. 3, 1533–1564.
  * [23] B. Simon, Trace class ideals, AMS, 2011.
  * [24] A. Soshnikov, Determinantal random point fields. (Russian) Uspekhi Mat. Nauk 55 (2000), no. 5(335), 107–160; transl. Russian Math. Surv. 55 (2000), no. 5, 923–975.