# On the Out-Of-Distribution Robustness of Self-Supervised Representation
Learning for Phonocardiogram Signals

Aristotelis Ballas  0000-0003-1683-8433 Vasileios Papapanagiotou and Christos
Diou Manuscript created November 28, 2023\. The work leading to these results
has been funded by the European Union under Grant Agreement No. 101057821,
project RELEVIUM. Views and opinions expressed are however those of the
authors and do not necessarily reflect those of the European Union or the
granting authority (HaDEA). Neither the European Union nor the granting
authority can be held responsible for them.Aristotelis Ballas and Christos
Diou are with the Department of Informatics and Telematics, Harokopio
University of Athens, Tavros, 177 78 Greece (e-mail<EMAIL_ADDRESS>cdiou@hua.gr).Vaslieios Papapanagiotou is with the Department of Biosciences
and Nutrition, Karolinska Institutet, Flemingsberg 141 52, Stockholm, Sweden
(e-mail: vasileios.papapanagiotou@ki.se).This paragraph will include the
Associate Editor who handled your paper.This work is a preprint under review.
0000-0001-6834-5548 0000-0002-2461-1928

###### Abstract

Objective: Despite the recent increase in research activity, deep-learning
models have not yet been widely accepted in medicine. The shortage of high-
quality annotated data often hinders the development of robust and
generalizable models, which do not suffer from degraded effectiveness when
presented with newly-collected, out-of-distribution (OOD) datasets. Methods:
Contrastive Self-Supervised Learning (SSL) offers a potential solution to the
scarcity of labeled data as it takes advantage of unlabeled data to increase
model effectiveness and robustness. In this research, we propose applying
contrastive SSL for detecting abnormalities in phonocardiogram (PCG) samples
by learning a generalized representation of the signal. Specifically, we
perform an extensive comparative evaluation of a wide range of audio-based
augmentations and evaluate trained classifiers on multiple datasets across
different downstream tasks. Results: We experimentally demonstrate that,
depending on its training distribution, the effectiveness of a fully-
supervised model can degrade up to 32% when evaluated on unseen data, while
SSL models only lose up to 10% or even improve in some cases. Conclusions:
Contrastive SSL pretraining can assist in providing robust classifiers which
can generalize to unseen, OOD data, without relying on time- and labor-
intensive annotation processes by medical experts. Furthermore, the proposed
extensive evaluation protocol sheds light on the most promising and
appropriate augmentations for robust PCG signal processing. Significance: We
provide researchers and practitioners with a roadmap towards producing robust
models for PCG classification, in addition to an open-source codebase for
developing novel approaches.

Self-Supervised learning, Contrastive learning, Deep learning, Phonocardiogram
classification, OOD representation learning

##  1 Introduction

Abnormal phonocardiogram (PCG) signals are often an important indicator of
heart diseases or deficiencies linked to increased mortality [1]. The
detection of such indicators in a non-invasive, accurate, and robust manner
could assist towards prevention or timely treatment of such diseases,
especially when access to healthcare services is limited.

Automated methods for detecting abnormal samples commonly rely on audio
recordings; typically, audio is processed and then used to train classifiers
that detect abnormal (i.e., non-healthy) samples and sometimes discriminate
between different diseases (e.g., [2]). However, the audio quality can be
affected by several factors. While typical audio-related factors include
microphone type or sampling rate, among others, for the case of PCG recordings
additional factors can significantly affect the quality of their recording,
such as placement of the microphone [3, 4], body size, skin, noise [5, 6],
etc. The above cases can prove even more challenging, such as when measuring
an unborn child/fetus, where PCG can only be captured through the mother’s
belly [7]. In addition to the quality of the recording conditions, the
training of accurate ML models is heavily dependent on annotated data, the
collection of which proves highly time- and cost-consuming. Especially in the
biomedical domain, where available samples are scarce to begin with.

As a result, classification models that have been trained on data originating
from a single source or on low quality signals fail to generalize to
previously unseen data distributions, which do not adhere to the i.i.d.
assumption (i.e they are not idependently and identically distributed),
leading to degraded effectiveness [8, 9]. In this work, we propose training
robust classifiers for PCG signal classification by employing contrastive
self-supervised learning (SSL) and investigate multiple combinations of
augmentations, leveraging both labeled and unlabeled datasets. Specifically,
we attempt to answer the following research questions:

  * •

Does contrastive SSL prove a valid training method for producing robust PCG
classifiers?

  * •

If so, which augmentations or transformations are the most promising in the
above?

  * •

Is an SSL classifier able to maintain adequate effectiveness and demonstrate
generalization capabilities when evaluated on OOD data, in contrast to its
fully supervised counterpart?

To answer the above research questions, we develop and propose an extensive
comparative evaluation in which we: a) investigate multiple combinations of
augmentations, leveraging both labeled and unlabeled datasets, b) examine the
effectiveness on a global “normal vs. abnormal” downstream task, as well as
downstream tasks specific to each dataset (where applicable) and finally, c)
evaluate all trained classifiers on OOD data that are left-out during
training. To facilitate further research in the field and provide future
researchers with a comprehensive baseline, we provide our codebase as open-
source111Code available at:
https://github.com/aristotelisballas/listen2yourheart.

The rest of the paper is organized as follows. Section 2 presents related work
to PCG-based classification, SSL training, and model robustness. Section 3
presents the methodology, including training approach, network architecture,
augmentations, and datasets. Finally, section 4 presents the evaluation
framework and results, while section 5 concludes the paper.

##  2 Related Work

Accurate and robust PCG classification has major implications in the well-
being of patients since it can often be an indicator of heart deficiencies.
However, the limited availability of annotated data has hindered the progress
of research in this specific field, as well as biosignal classification in
general. In this section, we focus on the most important papers and previous
works in DL for PCG classification and in SSL for 1D biosignal processing, and
we finally refer to methods which have been specifically developed for robust
1D biosignal classification.

###  2.1 Deep Learning for Phonocardiogram Classification

The release of publicly available PCG datasets under the scope of the 2016
[10] and 2022 [11] PhysioNet [12] challenges, has a sparked a line of research
in deep learning for PCG classification. The majority of previous works have
mainly proposed either 2D or 1D CNN architectures for tackling this particular
problem. The main difference lies in whether the proposed models are trained
directly on the one dimensional recorded auscultation waves or on spectral
images extracted from the audio recordings. For example, multiple recent
papers [13, 14, 15, 16] propose combining mel-spectrogram images extracted
from PCG signals with encoded patient demographic data, in a dual-branch CNN
model. The authors find that the combination of the above representations
leads to improved accuracy in the heart murmur detection task. Similarly, in
another work utilizing frequency domain features, the authors of [17] design a
hierarchical deep CNN architecture that extracts and combines features from
multiple-scales of PCG Mel-Spectrograms. A more recent work, [18], proposes
training an ensemble of 15 ResNets [19] with channel-wise attention mechanisms
on similar spectrograms, for grading murmur intensity in PCG recordings.

Although few, 1D models also prove promising in the heart-murmur detection
domain. In [20] the authors employ a transformer-based neural network and
train it on 1D wavelet power features extracted from raw PCG signals. On the
other hand, [21] proposes using a U-Net to predict heart states from the raw
PCG signal and combine the predictions with a ResNet model for the final
classification. In a different approach, the authors of [22] suggest that the
combination of a 1D CNN-Bi-LSTM model coupled with attention mechanisms,
trained in conjunction with a feature-based classifier leads to robust
abnormal PCG signal detection.

In this work, we build upon our preliminary research [23] for the 2022
PhysioNet challenge. In particular, we demonstrate the value of adopting self-
supervised contrastive learning for robust PCG classification and also report
the top augmentations which should be implemented in such a framework. To our
knowledge, this is the first study extensively researching the benefits of
each augmentation, and their combinations, for contrastive SSL in 1D PCG
signal processing. The aim of this study is to provide future researchers with
best-practices regarding PCG signal augmentation, while also propose a
structured protocol for the evaluation of developed methods.

Figure 1: Illustration of the proposed experiment pipeline for training and
evaluating the effectiveness and robustness of a model trained via Self-
Supervised Contrastive Learning for PCG classification. The framework has six
steps. In the first step, all datasets are prepared and homogenized into a
common format, as described in Section 3.1. In the second step, each signal is
augmented to two versions. In the next step, the unlabeled and augmented
signals are used to train the backbone encoder (Fig. 2 left) to maximize the
agreement between representations originating from the same initial signal.
Following the pretraining phase, in step 4, a classification head (Fig. 2
right) is attached to the frozen weights of the pretrained encoder. The
classifier weights are fine-tuned on data drawn from one of the Pascal,
PhysioNet2016 or PhysioNet2022 datasets, in a fully supervised manner. The
final model is then evaluated on the test split of the same dataset. In the
fifth step of the framework, the generalization ability of the fine-tuned
model is assesed on signals drawn from datasets which are left-out (”unseen”)
during the training process in step 4. We argue that given sufficient data and
appropriate augmentations, the pretrained encoder will be able to extract a
generalized PCG representation regardless the dataset. Finally, we repeat the
fine-tuning and OOD evaluation steps for each remaining dataset (step 6), to
assess the robustness of our method when the classifier has been trained on
different signal distributions.

###  2.2 Self-Supervised Learning in 1D biosignal classification

Self-supervised representation learning (SSL) [24] was initially proposed for
diminishing the limit posed by the requirement of large annotated datasets for
practical and production-ready deep learning systems. By taking advantage of
common features present in unlabeled data, SSL approaches rely on pretext
tasks [25] to exploit them and ultimately provide models which are able to
extract generalized and robust representations. Inspired by advancements in
contrastive predictive coding [26], the goal of Self-Supervised Contrastive
Learning [27, 28] is to train a model to recognize augmented or transformed
views of the same initial signal. By distinguishing between representations
which originate from different original signals, the resulting model is able
to extract invariant representations from its input data and reason upon
features which remain stable across distinct data distributions [29]. The SSL
paradigm seems to fit quite well in the medical and healthcare domain. In a
field where sufficient annotated datasets are scarcely available, there proves
a need to provide systems which can take advantage of the small amount of data
but also yield adequate and trustworthy inference results. To this end,
several works have proposed SSL approaches for processing biosignals in the
past, the majority of which have studied its advantages when extracting
representations from either electroencephalography (EEG), electrocardiography
(ECG) or electromyography (EMG) signals.

Regarding EEG signals, several prior studies revolve around developing methods
for emotion recognition and/or sleep stage detection [30]. The authors of [31]
propose augmenting multi-channel EEG signals by recombining several channels
into additional channels which represent different aspects of the EEG
recording. On top of the combination, they augment each channel by applying
pre-defined transformations and employ a contrastive loss for improved emotion
recognition and sleep-stage classification. In another work, the authors of
[32] argue that a model can learn informative EEG representations by
predicting whether two windows of a signal are sampled from the same temporal
context and use the above reasoning to design a relevant pretext task. The
above study is extended in [33] where contrastive predictive coding is
implemented on similar SSL-learned features to both predict sleep-stages and
detect pathologies in multi-channel EEG signals. Finally, [34] proposed BENDR,
which combines attention mechanisms [35] and contrastive learning to produce a
model with increased generalization capabilities and fine-tune it for several
downstream tasks in the EEG biosignal domain.

Several SSL methods have been proposed for ECG signal analysis as well [36],
most of which focus on either emotion recognition or pathology detection. For
example, in [37] the authors argue that the training of a model to recognize
different ECG signal transformations can lead to generalized and robust
feature learning. They evaluate their method on four emotion recognition
datasets and report state-of-the art results. In another paper studying
emotion recognition from ECG signals, [38] introduces a transformer model
which is pretrained to predict masked values of the signal and then fine-tuned
to predict the level of emotion in individuals. Regarding the pathology
detection task, most prior works focus on discriminating normal ECG signals
from others indicating a type of arrhythmia. Among the self-supervised
approaches proposed ECG pathology detection, contrastive learning is the most
prevalent choice. Specifically, authors either propose exploiting the temporal
and spatial invariances of the ECG signal [39], implementing attention
mechanisms [40] or combining wavelet transformations and random crops of the
signal [41] for the pretext contrastive task, before fine-tuning their
resulting models for arrhythmia detection. In a different approach, [42]
propose learning good representations by contrasting signals both at the
inter- and intra-subject levels. Specifically, they argue that similarity
should be maximized between signals which share a common label but are
measured from different individuals.

Finally, EMG signal processing is another interesting field which would
greatly benefit from the application of robust prediction models [43]. For
example, [44] introduces NeuroPose, a system for 3D hand pose tracking which
aims to alleviate the problems imposed due to different sensor mounting or
wrist positions, by pretraining an encoder-decoder model to recognize
augmented views of a single user’s signal and then finetuning a downstream
model to a subset of signals from a separate user. Similarly, the authors [45]
employ a contrastive loss to pretrain a model on EMG signals for hang-gesture
recognition.

###  2.3 Model Robustness in 1D biosignal classification

Extracting generalized and robust representations has been one of the most
important long-standing goals of machine learning [46]. To test and improve
the generalizabilty prowess and robustness of deep learning models, several
fields have emerged, such as Domain Adaptation [47], Transfer Learning [48],
and Domain Generalization [49], which use data from non-i.i.d. distributions
or out-of-distribution (OOD)) data for model evaluation. These distinct data
distributions are referred to as data Domains, as they are most often the
outcome of different data generating processes. Here, we reference prior works
which fall under the above research fields.

Domain adaptation (DA) methods are the most popular choice when aiming to
produce models with improved generalization capabilities in 1D biosignal
classification. Similar to Transfer Learning (TL), DA algorithms leverage the
representation learning capabilities of models pretrained on diverse source
data distributions and fine-tune them on previously unseen but similar target
distributions. The end-result is a model which remains robust under the
distribution shift between source and target data. In healthcare, DA methods
are proposed for a plethora of biosignal classification tasks. In [50], the
authors propose pre-training vanilla CNN networks on a large corpus of raw ECG
signals and demonstrate their effectiveness when fine-tuned on a smaller
dataset for arrhythmia detection. In another more recent work, He et al. [51]
propose tackling the distribution shift between different arrhythmia detection
ECG datasets, by employing three distribution alignment mechanisms. To be more
specific, they first extract spatio-temporal features from preprocessed
signals, pass them through a graph convolutional neural network and finally
align the resulting representations between source and target domains. Many
studies have researched the advantages of DA and TL in EEG classification as
well [52]. To name a few, the authors of [53, 54] and [55] propose aligning
source and target distributions via adversarial learning for motor image EEG
classification and epilepsy detection respectively, while [56] investigates
several state-of-the-art DA algorithms for EEG-based emotion recognition.

Finally, Domain Generalization algorithms aim to tackle the model
generalization at its core, as the main goal is to provide generalized models
that perform well across unseen datasets. Although limited, there has been a
recent interest in biosignal DG research. In the field of PCG classification,
the authors of [57] propose a ensemble classifier fusion method yields
improved results in a DG setup for abnormal vs normal signal classification.
To tackle the domain shift and variability present in ECG signals collected
from different patients, [58] and [59] propose domain-adversarial model
training for ECG and EEG classification, respectively. Most recently, the
authors of [9] introduce a DG benchmark and a model which leverages
representations from multiple layers of the network, for 12-lead ECG and
64-channel EEG classification.

Figure 2: The architecture of the CNN encoder trained via SSL contrastive
learning (left) and the classification head trained for the downstream task
(right). The weights of the encoder or ”backbone network” are frozen after the
pretraining phase. During the downstream task, the classification head is
attached to the pretrained encoder and its weights are fine-tuned on the
dataset of said task in a fully-supervised manner. The architecture of the
”downstream model” is used as a baseline in all of our experiments, where all
of its layer weights are adjusted during fully-supervised training.

##  3 Materials and Methods

We introduce an evaluation framework for assessing both the combination of
various transformations and the robustness of a network trained via
contrastive SSL, against a fully supervised baseline model. This allows us to
evaluate and report on the most promising signal transformations or
augmentations, for the PCG classification domain, while also highlight
additional benefits of training an end-to-end classification model via Self-
Supervised Contrastive Learning, such as improved model robustness and
generalization.

For the self-supervised training of all models, we adapt the SimCLR [60]
methodology for 1D PCG representation learning and then evaluate numerous
augmentation combinations designed specifically for 1D signal processing
tasks.

In the following sections, we describe the datasets and preprocessing steps
followed for all signals included in the study, and describe in detail the
protocol followed for evaluating the significance of each combination of
augmentations, as well as the generalization ability of a model pretrained
with self-supervised contrastive learning. Finally, we present the
architecture and relevant implementation details of the 1D CNN network
implemented for PCG classification.

###  3.1 Datasets & Downstream tasks

The PCG signals used in this study originate from a total of 5 distinct public
datasets and are available as “wav” files. These include the Ephnogram [61],
FPCGDB [62], Pascal [63], PhysioNet2016 [10], and PhysioNet2022 [11]
databases. Here we describe each dataset separately and provide all necesary
details regarding the preprocessing steps followed for the homogenization of
all signals into a common format.

The Ephnogram [61] database contains a total of 69 multi-channel, simultaneous
ECG and PCG recordings from 24 healthy adults aged between 23 and 29, acquired
during physical activity and rest conditions. Most recordings (62 during
exercise) are 30min long, while the duration of the remaining 7 (rest) is
about 30 seconds. As specified by the researchers, the dataset includes poor
sample signals for noise research purposes. As this dataset was proposed and
gathered for multi-modal signal processing purposes and not for pathology
detection or classification, it does not include labeled samples. For our
study, we use the unlabeled samples of the PCG signals for the SSL pretraining
of our models. All PCG recordings were sampled at 8 kHz.

The FPCGDB [62] or Fetal PCG Database, as its name suggests, is a dataset
containing 26 20-minute fetal phonocardiographic signals, gathered from
healthy women during their the remaining months of their pregnancy and sampled
at 333 Hz. Similarly to the Ephnogram dataset, the collected signals were
obtained for the development of fetal PCG signal simulation and fetal heart
rate extraction algorithms. Hence, this dataset also does not contain any
labeled samples for classification purposes. Furthermore, most recordings are
somewhat corrupted by noise, either due to the pregnant human body which
contains various noise sources, such as muscular movements or placental blood
turbulence, or even due to the surrounding environment.

The Pascal [63] Heart Sound Classification challenge was introduced for
developing algorithms for the automatic screening of cardiac pathologies. The
challenges contains two tasks; one for heart sound segmentation and one for
classification. In our work, we use the publicly available labeled data for
model training and evaluation which was gathered in both clinical and in-the-
wild settings. Notably, the 832 PCG signals are classified as either Normal,
Murmur, Extra Heart Sound, Artifact or Extrasystole222Each different category
is described in the official challenge description
(https://istethoscope.peterjbentley.com/heartchallenge/index.html).. The
recording lengths vary from 1 to 30 seconds and are sampled at 195 Hz. As each
sample is annotated with a single label, the above task is a multi-class
classification problem.

The remaining two datasets were also introduced in the context of public
challenges for PCG classification. Similarly to the Pascal challenge, the
objective of the PhysioNet2016 [10] challenge aims at urging researchers to
develop robust algorithms for differentiating normal and abnormal heart sound
recordings but instead introduces a binary classification problem. The data
included for model training in the dataset contains a total of 3,126 PCG
recordings collected from a single precordial location from both healthy
subjects and patients suffering from a variety of heart defections, lasting
from 5 to around 120 seconds. Once again, the recordings were gathered in both
clinical and non-clinical environments, with a sample rate of 2 kHz.

Finally, the PhysioNet2022 challenge [11] was designed for the development of
algorithms which could detect heart murmur in PCG signals. Succedent to the
2016 challenge, the 2022 challenge contains multiple heart sound recordings,
collected from multiple auscultation locations. The available data includes
3,163 recordings from 942 patients in total. Each patient has one or more PCG
signals, recorded from one of the four heart valves or from a separate
prominent auscultation location. Each of the recordings can be classified as
”present”, ”absent” or ”unknown”, meaning that heart murmur presence can be
either detected, not detected or can not be determined (e.g due to noise or
corrupted audio) respectively, rendering the challenge a multi-class
classification problem. The sampling rate of the above signals is 4 kHz.

Each dataset contains signals of varying length, sampling rate and label
classes. To accommodate model training in our framework, we follow a similar
process as in [23] for converting all signals into a similar format. As a
first step, to avoid transient noise artifacts, we initially discard the first
and final 2 seconds of each PCG recording. The resulting signal is split into
5 second overlapping windows with a 50% overlap. After resampling all samples
to 2 kHz, we assign two types of labels to each window; one that follows the
annotations of its original dataset (if it is drawn from a labeled dataset)
and one binary label indicating whether it derives from a normal heart sound
recording or not333In this study, all recordings initially labeled as unknown
are considered abnormal.. For the SSL pretraining stage, we assign
pseudolables to each window, as described in Section 3.2. The assignment of
binary labels to each window enables the evaluation of trained models across
datasets, which do not necessarily share similar sample classes. For example,
a classifier trained to detect normal PCG signals in the Pascal dataset, can
be then evaluated without further training on samples from the PhysioNet2022
dataset. The above evaluation process is described in detail in Section 3.4.

###  3.2 Self-Supervised Contrastive Learning for PCG Representation Learning

The main goal of this study is to, first of all, verify the use of SSL
contrastive learning and also evaluate the importance of augmentation
selection, for robust PCG representation learning. The proposed framework
consists of two separate tasks, a Pretraining pretext task and a Downstream
task, as depicted in Figure 1.

Assume a 1D PCG signal $x\in\mathbb{R}^{1\times L}$ sampled from
$\mathcal{X}$, where $L$ is the length of the signal. The aim of the
contrastive SSL pretext task is to learn robust representations of $x$ from
unlabeled PCG signals, by maximizing the agreement between two different
augmented versions of itself (positive pair) and simultaneously minimizing the
agreement between entirely separate augmented windows. Essentially, a model is
trained to predict whether two augmented signals are the result of different
transformations applied on the original sample. Formally, let $t(\cdot)$ be a
transformation or augmentation function drawn from a family of functions
$\mathcal{T}$. During the pretraining phase, an audio window $x$ is augmented
to two different versions $\tilde{x}$ and $\tilde{x}^{\prime}$, after the
application of two transformation functions $t\sim T$ and $t^{\prime}\sim
T^{\prime}$ respectively. Therefore, an initial batch of $N$ sample windows
$(\tilde{x}_{1},\tilde{x}_{2},\dotsc,\tilde{x}_{n})$ results into an augmented
batch of $2N$ signals
$(\tilde{x}_{1},\tilde{x}_{2},\dotsc,\tilde{x}_{n},\tilde{x}_{1}^{\prime},\tilde{x}_{2}^{\prime},\dotsc,\tilde{x}_{n}^{\prime})$,
where $\tilde{x}_{i}=t(\tilde{x})$ and
$\tilde{x}_{i}^{\prime}=t^{\prime}(\tilde{x}_{i})$ for $i\in[1,N]$. As
mentioned, each augmented pair of signals resulting from the same original
window is considered a positive pair, e.g
$(\tilde{x}_{1},\tilde{x}_{1}^{\prime})$, while pairs of different original
windows comprise negative pairs, e.g $(\tilde{x}_{1},\tilde{x}_{2}^{\prime})$.
Following its augmentation, the $2N$ signal batch is passed through a neural
network encoder model $E(\cdot)$ (Figure 2), where each signal is encoded into
a representation $\tilde{h}=E(\tilde{x})$. representations are flattened and
passed through a fully connected linear layer $G(\cdot)$ and projected into
128-dimensional vectors $\tilde{z}=G(\tilde{h})$.

For the contrastive prediction task, we employ the contrastive NT-Xent loss
function introduced in [26] and implemented in the SimCLR framework [64].
Specifically, given a set $\\{\tilde{x}_{k}\\}$ which includes an augmented
positive pair $\tilde{x}_{i}$ and $\tilde{x}_{j}$, a model trained via
contrastive SSL aims to identify $\tilde{x}_{j}$ in
$\\{\tilde{x}_{k}\\}_{k}\neq i$ for a given $\tilde{x}_{i}$. The contrastive
loss for a positive pair $(\tilde{x}_{i},\tilde{x}_{j})$ is formulated as
follows:

$\ell_{i,j}=-\log\frac{\exp(\mathrm{sim}(\bm{z}_{i},\bm{z}_{j})/\tau)}{\sum_{k=1}^{2N}\mathbbm{1}_{k\neq
i}\exp(\mathrm{sim}(\bm{z}_{i},\bm{z}_{k})/\tau)}~{},$ (1)

where $N$ is the initial batch size of the randomly sampled signals,
$\mathbbm{1}\in[0,1]$ is an indicator function and $\tau$ a temperature
parameter. The $\mathrm{sim}(\cdot)$ function calculates the cosine similarity
between the encoded vectors and is defined as:

$\mathrm{sim}(\bm{z}_{i},\bm{\tilde{z}}_{i})=\frac{\bm{z}_{i}.\bm{\tilde{z}}_{i}}{\parallel\bm{z}_{i}\parallel\parallel\bm{\tilde{z}}_{i}\parallel}.$
(2)

The final loss is computed across all positive pairs in a mini-batch.

After the pretraining step has been completed (i.e., step 2 in Figure 1), the
weights of the backbone encoder network $E(\cdot)$ are frozen. At this point,
the model is expected to produce a robust representation $\tilde{h}$. While
keeping the weights of the encoder frozen, a classification head $C(\cdot)$,
with randomly initialized weights, is attached to the pretrained network. The
layers of the classification head are then trained on labeled audio windows
$(x,y)$ in order to predict labels for each downstream task. The
classification layers are trained via fully supervised learning, with the
categorical Cross-Entropy (CE) loss. Depending on the labels of each dataset
in the downstream task, the CE loss is reduced to its binary form.

###  3.3 Augmentations for PCG Representation Learning

Arguably, the most integral part of the proposed framework is the family of
transformation functions $\mathcal{T}$, which is applied on each audio window,
as the type and intensity of each function can drastically affect the learning
of the encoder network. In order to conduct a rigorous evaluation of all
augmentations included in the proposed contrastive SSL framework and also
report the functions which should most likely be applied for robust PCG
representation learning, we select to implement functions that either simulate
factors that can be found in real world scenarios. For example, the first
thing that comes to mind is the volume of noise polluting recordings during
PCG collection. Additionally, the difference in sensor conductance can lead to
different scales of the recorded signals. To this end, it would make sense to
at least either add artificial noise to randomly scale each audio window
during the pretraining task. Following the above rationale, in our study we
select to implement and evaluate the effectiveness of a total of 6 different
augmentations, each of which is described below:

  * •

Noise: Zero-mean Gaussian Noise is added to the PCG signal. Specifically,
given noise
$n(t)=\frac{1}{\sqrt{2\pi\sigma}}e^{-\frac{(x-\mu)^{2}}{2\sigma^{2}}}$, where
$\mu$ is the mean and $\sigma$ the standard deviation, the resulting signal is
$\tilde{x}(t)=x(t)+n(t)$.

  * •

Cut-off Filters: Here either a high-pass or low-pass filter is applied to the
signal, at a 250, 500 or 750 Hz cutoff.

  * •

Scale: The signal is randomly scaled by a factor of $a\in[0.5,2.0]$ to yield
$\tilde{x}(t)=a\times x(t)$.

  * •

Reverse: The signal sequence is reversed.

  * •

Inversion: As its name suggests, this transformation inverts the original
signal by multiplying it by -1, i.e., $\tilde{x}(t)=-x(t)$.

  * •

Random Flip: This function takes as input the signal and concurrently applies
the ‘reverse’ and ‘inversion’ augmentations based on a given probability ($p$)
between 0.3, 0.5 and 0.7.

In the proposed framework, we also evaluate the composition of augmentations
by applying one transformation after the other. Specifically, we consider 4
different augmentation cases: 0vs1, 1vs1, 1vs2 and 2vs2. In each case, the
signal is either not transformed, transformed once or augmented by the
application of 2 concurrent functions.

Figure 3: Number of occurrences of each augmentation type in the top 75
performing models in each downstream task, leading to 150 total augmentation
pairs. This offers and indication on the importance of each augmentation
across all experiments. The plot on the left presents the result for In-
Distribution experiments, while the plot on the right the results for the OOD
experiments.

###  3.4 Evaluation Protocol

The present study aims at providing future researchers with a roadmap for
applying SSL in Phonocardiogram processing problems. To this end, we design a
rigorous and comprehensive protocol for evaluating the importance of each
applied augmentation in robust PCG classification. Each different augmentation
combination is evaluated by repeating the experiment illustrated in Figure 1,
with the exact same data splits for fair comparison.

In the first step of the framework, we create a pool of 5 sec unlabeled audio
windows from all available datasets, namely Pascal, PhysioNet2016,
PhysioNet2022, Ephnogram and Fpcgdb. After the preprocessing step has been
completed and all signals are homogenized in the common format described in
Section 3.1, they are randomly shuffled and split into training and validation
splits (80% and 20% respectively). In the second step, the split signals are
cut into batches. Each sample in the batch is then duplicated and augmented by
applying either one or a combination of the functions described in Section
3.3. The augmented samples are then passed through a Neural Network encoder,
which is trained via the contrastive pretraining task (Step 3). After the SSL
backbone model has been trained, its weights are frozen.

In Step 4, a classification head with learnable weights is attached to the
pretrained encoder which is then trained on data from one of the labeled
datasets (Pascal, PhysioNet2016 or PhysioNet2022). Specifically, each of these
datasets is split into training, validation and test splits with a standard
70%-20%-10% ratio and the trained model is evaluated on the test split only.
Since each model is evaluated on test data originating from the same
distribution as its training data, we call this the ”In-Distribution”
evaluation. Inspired by recent work in biosignal domain generalization [9], we
also design an evaluation protocol for assessing the robustness of a trained
classifier on out-of-distribution data, which are drawn from entirely
different datasets. Since the data has been completely left-out during
downstream training, the above process is followed for evaluating the
generalization ability of a model. Specifically, we now also freeze the
weights of the attached classification layers and test the model on data drawn
from the remaining 2 labeled datasets. For example, if the network is trained
for the Pascal downstream task, it is evaluated on both in-distribution data
(Pascal) and on out-of-distribution data (PhysioNet2016 and PhysioNet2022). We
call this step the out-of-distribution or OOD evaluation step. However, in
order to evaluate the same model on each separate dataset, all data must share
common labels. To that end, we implement two different downstream task types:
‘All’ and ‘Binary‘. In the ‘All‘ downstream task, the original labels of each
dataset are assigned to each audio window, while in the ‘Binary’ task a signal
is labeled as either ‘normal’ or ‘abnormal’, as described in Section 3.1. In
the case of the PhysioNet2016 dataset, the two tasks match as it introduces a
binary classification problem in the first place. For a fair comparison, all
test data splits remain the same across each evaluation scheme. Finally, to
assess the pretrained backbone encoder (and therefore the applied
augmentations) on all available data, we repeat Steps 4 and 5 with the
remaining datasets in a round-robin fashion (step 6 of the proposed
framework).

To thoroughly evaluate the proposed method we also repeat the entire
experiment by leaving data out during Step 1 and pretraining the model with
fewer data. This is also a means to concretely evaluate the proposed method on
true OOD data, as they are completely hidden throughout the training process,
either pretraining or downstream training. The best augmentation combinations
are selected based on the effectiveness of the downstream models on the ‘In-
Distribution’ test split. The same models are then evaluated on the hidden
‘OOD data‘.

###  3.5 Implementation Details

In this section, we describe all the implementation details regarding the
execution of our experiments. First of all, for the backbone encoder model we
implement the same model used in [23] and [65]. The encoder model is a 5 layer
convolutional neural network, the architecture of which is depicted in Figure
2 (left).

For the contrastive SSL pretraining phase, we train the conv layers of the
network by projecting the output of the last layer into a 128-D space via a
dense or fully-connected layer and minimize the NT-Xent loss (eq. 1). For the
cosine similarity (eq. 2) we use a temperature of 0.1, as in [23], while the
initial batch size is set to 256 (leading to 512 after augmentation occurs).
The SSL training epochs are set to a maximum of 200 epochs, based on an early
stopping set on the validation split with a patience of 10 epochs.
Additionally, to accommodate the large batch size we employ the LARS optimizer
[66] with a linear learning rate warm-up of 20 epochs, up to 0.1, followed by
a cosine decay of 1%.

During the downstream task, the dense projection layer of the pretrained
encoder is discarded and all convolutional weights are frozen. Next, a
3-layered classification head is appended to the pretrained network, as shown
in Figure 2 (right). Dropout layers are also included between the dense layers
of the classification head for further improving model robustness. The output
size of the last decision layer depends on the downstream task type and the
label size in each dataset. For the ‘Binary’ type and the PhysioNet2016
dataset, the output size is set to 1. For the ‘All’ type, the output size is
set to 5 or 3, for the Pascal and PhysioNet2022 datasets respectively. In all
cases, the classification head is trained via the appropriate cross-entropy
loss and the Adam optimizer. Additionally, the learning rate is set to
$10^{-4}$ and the batch size is set to 32. Once again, we train the model for
100 epochs at most, using early-stopping with 20 epochs patience. The same CNN
model is used for the fully-supervised baseline. All experiments are
implemented in Tensorflow 2.12 [67] and trained on a SLURM [68] cluster,
containing 4$\times$40GB NVIDIA A100 GPU cards, split into 8 20GB virtual MIG
devices.

Table 1: This table presents the experimental results of the proposed evaluation protocol, as specified in section 3.4 and Figure 1. Top results are specified in bold face. The first column of the table indicates the datasets on which the SSL backbone was pretrained on. As in Fig. 3.4, E, F, Pa, P16 and P22, indicate data from the Ephnogram, Fpcgdb, Pascal, Physionet16 and Physionet22 datasets respectively. The SSL and fully-supervised baseline models are trained and evaluated on In-Distribution data from the respective dataset. The presented models are the best-performing models, based on results from the In-Distribution test data split. Once trained, they are then evaluated on out-of-distribution data originating from unseen datasets. The ‘All’ labels indicate that the PCG signals are classified based on the dataset’s original labels (P16 has only binary labels), whereas ‘Binary’ indicates that the signals are classified between normal and abnormal. The SSL-NoDs model refers to the backbone network which has not ‘seen’ data from the downstream task dataset during pretraining. The adopted augmentation combinations for each model are reported in Section 4.1.  |  | In-Distribution | Out-of-Distribution | Out-of-Distribution
---|---|---|---|---
|  | All | Binary | Binary | Binary
SSL Datasets | Model | Acc | F1 | Acc | F1 | Acc | F1 | Acc | F1
|  | Pascal | PhysioNet2016 | PhysioNet2022
— | Baseline | 79.37 | 48.37 | 67.77 | 40.00 | 51.95 | 62.74 | 51.33 | 59.41
E, F, Pa, P16 | SSL | 76.87 | 42.18 | 31.25 | 47.61 | 63.55 | 74.88 | 53.24 | 64.44
E, F, Pa, P22 | SSL | 76.87 | 42.18 | 32.10 | 42.25 | 61.56 | 72.69 | 61.11 | 73.64
E, F, P16, P22 | SSL-NoDs | 75.62 | 39.06 | 31.25 | 40.00 | 59.88 | 74.97 | 58.33 | 72.06
all | SSL | 81.25 | 53.12 | 35.93 | 47.61 | 67.13 | 79.09 | 62.11 | 74.72
|  | PhysioNet2016 | Pascal | PhysioNet2022
— | Baseline | — | — | 88.64 | 93.09 | 55.00 | 40.00 | 49.85 | 60.59
E, F, Pa, P16 | SSL | — | — | 80.07 | 87.38 | 48.58 | 42.90 | 57.74 | 70.97
E, F, Pa, P22 | SSL-NoDs | — | — | 74.97 | 85.69 | 52.55 | 42.61 | 58.99 | 71.59
E, F, P16, P22 | SSL | — | — | 84.19 | 89.65 | 31.25 | 47.62 | 62.07 | 78.07
all | SSL | — | — | 85.02 | 90.31 | 42.19 | 50.66 | 70.45 | 81.63
|  | PhysioNet2022 | Pascal | PhysioNet2016
— | Baseline | 90.80 | 86.21 | 86.90 | 91.90 | 53.12 | 53.12 | 67.91 | 80.64
E, F, Pa, P16 | SSL-NoDs | 87.94 | 81.91 | 82.49 | 89.71 | 48.29 | 47.09 | 70.45 | 81.63
E, F, Pa, P22 | SSL | 88.55 | 82.83 | 82.83 | 89.96 | 51.42 | 48.64 | 71.07 | 82.94
E, F, P16, P22 | SSL | 85.37 | 78.06 | 78.06 | 87.67 | 31.25 | 47.61 | 74.97 | 85.70
all | SSL | 89.31 | 83.96 | 83.67 | 90.00 | 48.44 | 54.79 | 74.97 | 85.70

##  4 Results and Discussion

###  4.1 Augmentation and Transformation Evaluation

To avoid the combinatorial explosion poised by the evaluation of all possible
augmentation combinations, we designed a protocol to limit the experiments to
a feasible amount. As mentioned in Section 3.3, there are a total of 7
(including not transforming) possible functions that can be applied to the two
copies of the audio sample, resulting in 4 possible augmentation cases; 0vs1,
1vs1, 1vs2 and 2vs2. As a first step, we examined all possible combinations in
the 0vs1 and 1vs1 cases without applying the same transformation twice. After
evaluating the above we selected a subset of the augmentations leading to the
top performing models and continued experimenting with 1vs2 and 2vs2
augmentation schemes. The models with the best results are the ones presented
in Table 1 which are discussed in the next section.

However, the main aim of this study is to provide an evaluation of the most
appropriate augmentations for PCG signal analysis. Due to the fact that by
only presenting the combination of the augmentations leading to the best
classification results could be misleading (since only the specific
combination could pertain to said results), we present a plot depicting the
frequency of augmentation occurrence in the top 75 experiments across all 3
downstream tasks and for both In-Distribution and OOD data. Specifically, we
select the top 25 results for each task and present the occurrences for all 75
total experiments in Fig. 3. The evaluation results are quite interesting as
several conclusions can be made. First of all, the application of low-pass
filters seem to be dominant in both In-Distribution and OOD top experiments.
This is a clear indicator that such filters should be applied towards
increasing the robustness of a classifier. What is more, the application of
uniform noise, random flip and scaling on the signal, also boosts the
effectiveness of classifiers in all cases. Furthermore, we found that the best
performing models in each of the 3 downstream tasks, had adopted the following
augmentations during their pretraining:

  * •

Pascal: high pass filter (pass band 250Hz, stop band 300 Hz) and random flip
(p=0.7) vs inversion and uniform noise (mean 0, std 0.1),

  * •

PhysioNet2016: high pass filter (pass band 250Hz, stop band 300 Hz) vs reverse

  * •

PhysioNet2022: high pass filter (pass band 250Hz, stop band 300 Hz) vs random
flip (p=0.7) and random scaling with $a$ between 1.0 and 1.5

While the above augmentations agree, for the most part, with the distributions
and number of occurrences in Fig. 3, they also support the findings in several
previous contrastive SSL works [60, 69], in which stronger augmentations lead
to improved results. In the next section we present numerical results for all
downstream tasks and experiments.

###  4.2 Results

Table 1 presents the Accuracy and F1-Scores of the best performing models
trained via contrastive SSL for all downstream tasks, including evaluation on
OOD distribution data, as presented in Section 3.4. We perform a total of 4
cycles of SSL experiments, evaluating all augmentation combinations in each
cycle. In each experiment cycle, we either use all available datasets or leave
one labeled dataset out at a time. The first column of Table 1 denotes the
datasets used during SSL training. In the same table, the ‘Baseline’ model
refers to the fully-supervised model trained solely on data from the
downstream ‘In-Distribution’ dataset. Due to the variance in the OOD results
for the baseline model, we present the average of 5 total runs. Finally, the
‘SSL-NoDs’ model refers to the model which has not been pretrained with data
originating from the dataset of the downstream task. All results are rated
based on the F1-Scores on the ‘In-Distribution’ test splits and the adopted
augmentations for each case are the ones mentioned in Section 4.1. The same
models are then evaluated on the test splits of the OOD datasets.

###  4.3 Discussion

At a first glance, the benefits of SSL pretraining admittedly do not seem very
compelling. When evaluating the model on the test data of the In-Distribution
data, the baseline fully-supervised model outperforms the SSL models in 2 out
of 3 cases. When trained on the PhysioNet2016 and PhysioNet2022 datasets the
baseline surpasses the best SSL model by 2.78% and 1.9% respectively. However,
the same does not hold in the case of the Pascal dataset, where the SSL model
is able to outperform the baseline by 4.75% when evaluated on ‘All’ labels and
by 7.61% in the ‘Binary’ task. Since the Pascal dataset is considerably
smaller than the other 2, and arguably the hardest among the datasets, the
above results validate the effectiveness of pretraining a model on a large
corpus of data, when available labeled data is not sufficient.

By diving further into the results, the true benefits of the proposed method
begin to show. First of all, the drop in effectiveness between in-distribution
and OOD data is apparent. Additionally, in all cases the baseline models
appears to have overfit their training data distribution and fails to maintain
its classification ability across distinct datasets. On the other hand, the
models which have been pretrained via SSL yield far superior results. In all
cases, the models trained with the proposed framework surpass the baseline by
11.68% on average and even outperform it by 21.04% in the case of the
PhysioNet22 OOD task. Even more so, the SSL models were suprisingly able to
surpass both the effectiveness of the baseline and their ‘In-Distribution’
counterpart, in the case of the Pascal OOD evaluation. Specifically, the
downstream models trained on data from PhysioNet2016 and PhysioNet2022 were
able to yield better results than the fully-supervised model trained solely on
Pascal data. Furthermore, even though there is a decrease in effectiveness
when the OOD datasets are left out during model pretraining, the SSL models
continue to outshine the baseline across the board, with the exception of the
case where the downstream task is the PhysioNet2022 data and models are
evaluated on OOD data from Pascal. Finally, we would like to note that the
above findings are aligned with the results of our preliminary SSL method [23]
submitted in the PhysioNet2022 challenge, where the proposed model was able to
avoid overfitting on the available training distribution and demonstrated
adequate generalization effectiveness on completely hidden data.

##  5 Conclusions

This research presents a comprehensive and extensive evaluation of
augmentation and transformations for pretraining neural networks, towards
robust PCG classification. In addition to presenting the best performing
combinations of augmentations, we also provide a summary of the most
frequently appearing transformation functions in the top 75 overall
experimental results. Our experiments suggest that the use of cutoff filters,
particularly low pass filters, leads to the best performing models among
several datasets, rendering them almost essential during such tasks.
Furthermore, the addition of small random noise and inversion or reversal of
the signal, also yields improved results. Finally, the numerical results of
our experiments validate our claims that contrastive SSL learning can induce
robust and generalizable classifiers, as the drop in effectiveness when
evaluating on previously unseen data is far less than that of a fully-
supervised model. The proposed method however, does not come without
drawbacks. Most importantly, to pretrain a model via SSL, most likely a large
corpus of data must be available beforehand in order to enhance the
representation extraction capabilities of the model. What’s more, the
implemented contrastive loss expects a large batch size to perform as
expected, adding a computational constraint to the overall framework. In the
future, we aim to enhance the adopted augmentation techniques, by adding novel
functions or researching further parameters of the existing ones, expanding
the downstream tasks by incorporating transfer learning problems, as well as
applying it to other 1D biosignals such as EMG.

## References

  * [1] GBD 2017 Causes of Death Collaborators, “Global, regional, and national age-sex-specific mortality for 282 causes of death in 195 countries and territories, 1980-2017: a systematic analysis for the Global Burden of Disease Study 2017,” _Lancet_ , vol. 392, no. 10159, pp. 1736–1788, Nov. 2018.
  * [2] F. A. Khan, A. Abid, and M. S. Khan, “Automatic heart sound classification from segmented/unsegmented phonocardiogram signals using time and frequency features,” _Physiological Measurement_ , vol. 41, no. 5, p. 055006, jun 2020\.
  * [3] R. Jaros, J. Koutny, M. Ladrova, and R. Martinek, “Novel phonocardiography system for heartbeat detection from various locations,” _Scientific Reports_ , vol. 13, no. 1, p. 14392, 2023.
  * [4] J. Fontecave-Jallon, K. Fojtik, and B. Rivet, “Is there an optimal localization of cardio-microphone sensors for phonocardiogram analysis?” in _2019 41st Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC)_ , 2019, pp. 3249–3252.
  * [5] N. Giordano, S. Rosati, and M. Knaflitz, “Automated assessment of the quality of phonocardographic recordings through signal-to-noise ratio for home monitoring applications,” _Sensors_ , vol. 21, no. 21, p. 7246, Oct. 2021\.
  * [6] C. M. Huisa, C. Elvis Supo, T. Edward Figueroa, J. Rendulich, and E. Sulla-Espinoza, “Pcg heart sounds quality classification using neural networks and smote tomek links for the think health project,” in _Proceedings of Data Analytics and Management_ , A. Khanna, Z. Polkowski, and O. Castillo, Eds. Singapore: Springer Nature Singapore, 2023, pp. 803–811.
  * [7] R. Kahankova _et al._ , “A review of recent advances and future developments in fetal phonocardiography,” _IEEE Reviews in Biomedical Engineering_ , vol. 16, pp. 653–671, 2023.
  * [8] B. Recht, R. Roelofs, L. Schmidt, and V. Shankar, “Do imagenet classifiers generalize to imagenet?” in _International Conference on Machine Learning_. PMLR, 2019, pp. 5389–5400.
  * [9] A. Ballas and C. Diou, “Towards domain generalization for ecg and eeg classification: Algorithms and benchmarks,” _IEEE Transactions on Emerging Topics in Computational Intelligence_ , pp. 1–11, 2023.
  * [10] C. Liu _et al._ , “An open access database for the evaluation of heart sound algorithms,” _Physiological Measurement_ , vol. 37, no. 12, p. 2181, Nov. 2016, publisher: IOP Publishing.
  * [11] M. A. Reyna _et al._ , “Heart murmur detection from phonocardiogram recordings: The George B. Moody PhysioNet Challenge 2022,” Apr. 2023, pages: 2022.08.11.22278688.
  * [12] A. L. Goldberger _et al._ , “PhysioBank, PhysioToolkit, and PhysioNet: components of a new research resource for complex physiologic signals,” _Circulation_ , vol. 101, no. 23, pp. E215–220, Jun. 2000.
  * [13] H. Lu _et al._ , “A lightweight robust approach for automatic heart murmurs and clinical outcomes classification from phonocardiogram recordings,” in _2022 Computing in Cardiology (CinC)_ , vol. 498. IEEE, 2022, pp. 1–4.
  * [14] J. Ding, J.-J. Li, and M. Xu, “Classification of murmurs in pcg using combined frequency domain and physician inspired features,” in _2022 Computing in Cardiology (CinC)_ , vol. 498, 2022, pp. 1–4.
  * [15] M. S. Knorr and J. P. Bremer, “Using mel-spectrograms and 2d-cnns to detect murmurs in variable length phonocardiograms,” in _2022 Computing in Cardiology (CinC)_ , vol. 498, 2022, pp. 1–4.
  * [16] M. Araujo _et al._ , “Maiby’s algorithm: A two-stage deep learning approach for murmur detection in mel spectrograms for automatic auscultation of congenital heart disease,” in _2022 Computing in Cardiology (CinC)_ , vol. 498, 2022, pp. 1–4.
  * [17] Y. Xu, X. Bao, H.-K. Lam, and E. N. Kamavuako, “Hierarchical multi-scale convolutional network for murmurs detection on pcg signals,” in _2022 Computing in Cardiology (CinC)_ , vol. 498, 2022, pp. 1–4.
  * [18] A. Elola _et al._ , “Beyond heart murmur detection: Automatic murmur grading from phonocardiogram,” _IEEE Journal of Biomedical and Health Informatics_ , vol. 27, no. 8, pp. 3856–3866, 2023.
  * [19] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , June 2016.
  * [20] M. Alkhodari, S. Kamarul Azman, L. J. Hadjileontiadis, and A. Khandoker, “Ensemble Transformer-Based Neural Networks Detect Heart Murmur In Phonocardiogram Recordings,” Dec. 2022.
  * [21] S. Choi, H.-C. Seo, C. Kyungmin, G.-W. Yoon, and S. Joo, “Murmur classification with u-net state prediction,” in _2022 Computing in Cardiology (CinC)_ , vol. 498, 2022, pp. 1–4.
  * [22] Z. Imran, E. Grooby, V. V. Malgi, C. Sitaula, S. Aryal, and F. Marzbanrad, “A fusion of handcrafted feature-based and deep learning classifiers for heart murmur detection,” in _2022 Computing in Cardiology (CinC)_ , vol. 498, 2022, pp. 1–4.
  * [23] A. Ballas, V. Papapanagiotou, A. Delopoulos, and C. Diou, “Listen2YourHeart: A Self-Supervised Approach for Detecting Murmur in Heart-Beat Sounds,” in _2022 Computing in Cardiology (CinC)_ , vol. 498, 2022, pp. 1–4.
  * [24] L. Ericsson, H. Gouk, C. C. Loy, and T. M. Hospedales, “Self-supervised representation learning: Introduction, advances, and challenges,” _IEEE Signal Processing Magazine_ , vol. 39, no. 3, pp. 42–62, 2022.
  * [25] I. Misra and L. v. d. Maaten, “Self-supervised learning of pretext-invariant representations,” in _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , 2020, pp. 6707–6717.
  * [26] A. v. d. Oord, Y. Li, and O. Vinyals, “Representation Learning with Contrastive Predictive Coding,” _arXiv:1807.03748 [cs, stat]_ , Jan. 2019, arXiv: 1807.03748.
  * [27] X. Liu _et al._ , “Self-supervised learning: Generative or contrastive,” _IEEE Transactions on Knowledge and Data Engineering_ , vol. 35, no. 1, pp. 857–876, 2023.
  * [28] P. Kumar, P. Rawat, and S. Chauhan, “Contrastive self-supervised learning: review, progress, challenges and future research directions,” _International Journal of Multimedia Information Retrieval_ , vol. 11, no. 4, pp. 461–488, Dec. 2022.
  * [29] Y. Zhang, B. Hooi, D. Hu, J. Liang, and J. Feng, “Unleashing the power of contrastive self-supervised visual models via contrast-regularized fine-tuning,” in _Advances in Neural Information Processing Systems_ , M. Ranzato, A. Beygelzimer, Y. Dauphin, P. Liang, and J. W. Vaughan, Eds., vol. 34. Curran Associates, Inc., 2021, pp. 29 848–29 860.
  * [30] M. H. Rafiei, L. V. Gauthier, H. Adeli, and D. Takabi, “Self-supervised learning for electroencephalography,” _IEEE Transactions on Neural Networks and Learning Systems_ , pp. 1–15, 2022.
  * [31] M. N. Mohsenvand, M. R. Izadi, and P. Maes, “Contrastive Representation Learning for Electroencephalogram Classification,” in _Proceedings of the Machine Learning for Health NeurIPS Workshop_. PMLR, Nov. 2020, pp. 238–253, iSSN: 2640-3498.
  * [32] H. Banville _et al._ , “Self-Supervised Representation Learning from Electroencephalography Signals,” in _2019 IEEE 29th International Workshop on Machine Learning for Signal Processing (MLSP)_. Pittsburgh, PA, USA: IEEE, Oct. 2019, pp. 1–6.
  * [33] H. Banville, O. Chehab, A. Hyvärinen, D.-A. Engemann, and A. Gramfort, “Uncovering the structure of clinical eeg signals with self-supervised learning,” _Journal of Neural Engineering_ , vol. 18, no. 4, p. 046020, 2021\.
  * [34] D. Kostas, S. Aroca-Ouellette, and F. Rudzicz, “BENDR: Using Transformers and a Contrastive Self-Supervised Learning Task to Learn From Massive Amounts of EEG Data,” _Frontiers in Human Neuroscience_ , vol. 15, 2021.
  * [35] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin, “Attention is All you Need,” in _Advances in Neural Information Processing Systems_ , I. Guyon _et al._ , Eds., vol. 30. Curran Associates, Inc., 2017. [Online]. Available: https://proceedings.neurips.cc/paper_files/paper/2017/file/3f5ee243547dee91fbd053c1c4a845aa-Paper.pdf
  * [36] T. Mehari and N. Strodthoff, “Self-supervised representation learning from 12-lead ECG data,” _Computers in Biology and Medicine_ , vol. 141, p. 105114, Feb. 2022.
  * [37] P. Sarkar and A. Etemad, “Self-Supervised ECG Representation Learning for Emotion Recognition,” _IEEE Transactions on Affective Computing_ , vol. 13, no. 3, pp. 1541–1554, 2022.
  * [38] J. Vazquez-Rodriguez, G. Lefebvre, J. Cumin, and J. L. Crowley, “Transformer-based self-supervised learning for emotion recognition,” in _2022 26th International Conference on Pattern Recognition (ICPR)_. IEEE, 2022, pp. 2605–2612.
  * [39] D. Kiyasseh, T. Zhu, and D. A. Clifton, “CLOCS: Contrastive Learning of Cardiac Signals Across Space, Time, and Patients,” in _Proceedings of the 38th International Conference on Machine Learning_. PMLR, Jul. 2021, pp. 5606–5615, iSSN: 2640-3498.
  * [40] J. Oh, H. Chung, J.-m. Kwon, D.-g. Hong, and E. Choi, “Lead-agnostic Self-supervised Learning for Local and Global Representations of Electrocardiogram,” in _Proceedings of the Conference on Health, Inference, and Learning_ , ser. Proceedings of Machine Learning Research, G. Flores, G. H. Chen, T. Pollard, J. C. Ho, and T. Naumann, Eds., vol. 174. PMLR, Apr. 2022, pp. 338–353.
  * [41] H. Chen, G. Wang, G. Zhang, P. Zhang, and H. Yang, “CLECG: A Novel Contrastive Learning Framework for Electrocardiogram Arrhythmia Classification,” _IEEE Signal Processing Letters_ , vol. 28, pp. 1993–1997, 2021.
  * [42] X. Lan, D. Ng, S. Hong, and M. Feng, “Intra-Inter Subject Self-Supervised Learning for Multivariate Cardiac Signals,” _Proceedings of the AAAI Conference on Artificial Intelligence_ , vol. 36, no. 4, pp. 4532–4540, Jun. 2022, number: 4.
  * [43] D. Xiong, D. Zhang, X. Zhao, and Y. Zhao, “Deep learning for emg-based human-machine interaction: A review,” _IEEE/CAA Journal of Automatica Sinica_ , vol. 8, no. 3, pp. 512–533, 2021.
  * [44] Y. Liu, S. Zhang, and M. Gowda, “A Practical System for 3-D Hand Pose Tracking Using EMG Wearables With Applications to Prosthetics and User Interfaces,” _IEEE Internet of Things Journal_ , vol. 10, no. 4, pp. 3407–3427, Feb. 2023.
  * [45] Z. Lai _et al._ , “Contrastive Domain Adaptation: A Self-Supervised Learning Framework for sEMG-Based Gesture Recognition,” in _2022 IEEE International Joint Conference on Biometrics (IJCB)_ , Oct. 2022, pp. 1–7, iSSN: 2474-9699.
  * [46] Y. Bengio, A. Courville, and P. Vincent, “Representation learning: A review and new perspectives,” _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , vol. 35, no. 8, pp. 1798–1828, 2013.
  * [47] M. Wang and W. Deng, “Deep visual domain adaptation: A survey,” _Neurocomputing_ , vol. 312, pp. 135–153, 2018.
  * [48] F. Zhuang _et al._ , “A comprehensive survey on transfer learning,” _Proceedings of the IEEE_ , vol. 109, no. 1, pp. 43–76, 2020.
  * [49] K. Zhou, Z. Liu, Y. Qiao, T. Xiang, and C. C. Loy, “Domain generalization: A survey,” _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , 2022.
  * [50] K. Weimann and T. O. F. Conrad, “Transfer learning for ECG classification,” _Scientific Reports_ , vol. 11, no. 1, p. 5251, Mar. 2021, number: 1 Publisher: Nature Publishing Group.
  * [51] Z. He _et al._ , “A novel unsupervised domain adaptation framework based on graph convolutional network and multi-level feature alignment for inter-subject ECG classification,” _Expert Systems with Applications_ , vol. 221, p. 119711, 2023.
  * [52] Z. Wan, R. Yang, M. Huang, N. Zeng, and X. Liu, “A review on transfer learning in EEG signal analysis,” _Neurocomputing_ , vol. 421, pp. 1–14, Jan. 2021\.
  * [53] Q. She _et al._ , “Improved domain adaptation network based on wasserstein distance for motor imagery eeg classification,” _IEEE Transactions on Neural Systems and Rehabilitation Engineering_ , vol. 31, pp. 1137–1148, 2023\.
  * [54] H. Zhao, Q. Zheng, K. Ma, H. Li, and Y. Zheng, “Deep representation-based domain adaptation for nonstationary eeg classification,” _IEEE Transactions on Neural Networks and Learning Systems_ , vol. 32, no. 2, pp. 535–545, 2021.
  * [55] P. Peng _et al._ , “Domain adaptation for epileptic EEG classification using adversarial learning and Riemannian manifold,” _Biomedical Signal Processing and Control_ , vol. 75, p. 103555, May 2022.
  * [56] Z. Lan, O. Sourina, L. Wang, R. Scherer, and G. R. Müller-Putz, “Domain adaptation techniques for EEG-based emotion recognition: a comparative study on two public datasets,” _IEEE Transactions on Cognitive and Developmental Systems_ , vol. 11, no. 1, pp. 85–94, 2018, publisher: IEEE.
  * [57] T. Dissanayake _et al._ , “Domain generalization in biosignal classification,” _IEEE Transactions on Biomedical Engineering_ , vol. 68, no. 6, pp. 1978–1989, 2021.
  * [58] H. Hasani, A. Bitarafan, and M. S. Baghshah, “Classification of 12-lead ecg signals with adversarial multi-source domain generalization,” in _2020 Computing in Cardiology_ , 2020, pp. 1–4.
  * [59] B.-Q. Ma, H. Li, W.-L. Zheng, and B.-L. Lu, “Reducing the Subject Variability of EEG Signals with Adversarial Domain Generalization,” in _Neural Information Processing_ , ser. Lecture Notes in Computer Science, T. Gedeon, K. W. Wong, and M. Lee, Eds. Cham: Springer International Publishing, 2019, pp. 30–42.
  * [60] T. Chen, S. Kornblith, M. Norouzi, and G. Hinton, “A Simple Framework for Contrastive Learning of Visual Representations,” in _Proceedings of the 37th International Conference on Machine Learning_. PMLR, Nov. 2020, pp. 1597–1607, iSSN: 2640-3498.
  * [61] A. Kazemnejad, P. Gordany, and R. Sameni, “Ephnogram: A simultaneous electrocardiogram and phonocardiogram database,” _PhysioNet_ , 2021.
  * [62] M. Cesarelli, M. Ruffo, M. Romano, and P. Bifulco, “Simulation of foetal phonocardiographic recordings for testing of FHR extraction algorithms,” _Computer Methods and Programs in Biomedicine_ , vol. 107, no. 3, pp. 513–523, Sep. 2012.
  * [63] P. Bentley, G. Nordehn, M. Coimbra, and S. Mannor, “The PASCAL Classifying Heart Sounds Challenge 2011 (CHSC2011) Results,” http://www.peterjbentley.com/heartchallenge/index.html.
  * [64] T. Chen, S. Kornblith, M. Norouzi, and G. Hinton, “A simple framework for contrastive learning of visual representations,” in _International conference on machine learning_. PMLR, 2020, pp. 1597–1607.
  * [65] V. Papapanagiotou, C. Diou, and A. Delopoulos, “Chewing detection from an in-ear microphone using convolutional neural networks,” _Annual International Conference of the IEEE Engineering in Medicine and Biology Society. IEEE Engineering in Medicine and Biology Society. Annual International Conference_ , vol. 2017, pp. 1258–1261, Jul. 2017.
  * [66] B. Ginsburg, I. Gitman, and Y. You, “Large batch training of convolutional networks with layer-wise adaptive rate scaling,” 2018.
  * [67] M. Abadi _et al._ , “TensorFlow: Large-scale machine learning on heterogeneous systems,” 2015, software available from tensorflow.org. [Online]. Available: https://www.tensorflow.org/
  * [68] A. B. Yoo, M. A. Jette, and M. Grondona, “SLURM: Simple Linux Utility for Resource Management,” in _Job Scheduling Strategies for Parallel Processing_ , ser. Lecture Notes in Computer Science, D. Feitelson, L. Rudolph, and U. Schwiegelshohn, Eds. Berlin, Heidelberg: Springer, 2003, pp. 44–60.
  * [69] B. Gopal _et al._ , “3KG: Contrastive Learning of 12-Lead Electrocardiograms using Physiologically-Inspired Augmentations,” in _Proceedings of Machine Learning for Health_. PMLR, Nov. 2021, pp. 156–167, iSSN: 2640-3498.