# Stressor Type Matters! — Exploring Factors Influencing Cross-Dataset
Generalizability of Physiological Stress Detection

Pooja Prajod<EMAIL_ADDRESS>0000-0002-3168-3508 University of
AugsburgAugsburgGermany ,  Bhargavi Mahesh  University of
AugsburgAugsburgGermany<EMAIL_ADDRESS>and  Elisabeth André
University of AugsburgAugsburgGermany<EMAIL_ADDRESS>
###### Abstract.

Automatic stress detection using heart rate variability (HRV) features has
gained significant traction as it utilizes unobtrusive wearable sensors
measuring signals like electrocardiogram (ECG) or blood volume pulse (BVP).
However, detecting stress through such physiological signals presents a
considerable challenge owing to the variations in recorded signals influenced
by factors, such as perceived stress intensity and measurement devices.
Consequently, stress detection models developed on one dataset may perform
poorly on unseen data collected under different conditions. To address this
challenge, this study explores the generalizability of machine learning models
trained on HRV features for binary stress detection. Our goal extends beyond
evaluating generalization performance; we aim to identify the characteristics
of datasets that have the most significant influence on generalizability. We
leverage four publicly available stress datasets (WESAD, SWELL-KW,
ForDigitStress, VerBIO) that vary in at least one of the characteristics such
as stress elicitation techniques, stress intensity, and sensor devices.
Employing a cross-dataset evaluation approach, we explore which of these
characteristics strongly influence model generalizability. Our findings reveal
a crucial factor affecting model generalizability: stressor type. Models
achieved good performance across datasets when the type of stressor (e.g.,
social stress in our case) remains consistent. Factors like stress intensity
or brand of the measurement device had minimal impact on cross-dataset
performance. Based on our findings, we recommend matching the stressor type
when deploying HRV-based stress models in new environments. To the best of our
knowledge, this is the first study to systematically investigate factors
influencing the cross-dataset applicability of HRV-based stress models.

Stress, Generalizability, Cross-dataset, Machine learning, Heart rate
variability, Electrocardiography, Photoplethysmography

††copyright: none††ccs: Computing methodologies Cross-validation††ccs: Human-
centered computing Empirical studies in ubiquitous and mobile computing††ccs:
Computing methodologies Machine learning

##  1\. Introduction

The ability to detect stress in real-time has become increasingly important
within the fields of affective computing and human-machine interaction (HMI)
(Alberdi et al., 2016). Early stress detection offers a valuable tool for
promoting well-being and potentially preventing long-term health consequences
(Greene et al., 2016; Akmandor and Jha, 2017; Alberdi et al., 2016).
Consequently, a growing area of research focuses on developing interactive
systems that can not only detect stress but also provide personalized
interventions to manage it effectively (Yu et al., 2018; Balcombe and De Leo,
2022).

Researchers have explored various modalities for stress detection, including
psychological tools, behavioral patterns, and physiological signals
(Giannakakis et al., 2019; Alberdi et al., 2016). However, physiological
signals are considered more reliable than the other methods as they eliminate
certain measurement biases. Moreover, the growing popularity of unobtrusive
wearable sensors further facilitates continuous stress monitoring through
physiological signals.

Stress manifests differently depending on the context (Alberdi et al., 2016).
For instance, the stress experienced during an exam likely differs from that
of public speaking. While controlled lab settings are often used to develop
stress detection models, real-world HMI scenarios are far more diverse.
Therefore, assessing the generalizability of these models – their ability to
perform well in unseen contexts – is crucial for real-world applications.
Cross-dataset evaluation, where a model trained on one dataset is tested on
another, is a common approach to assess generalizability. Good cross-dataset
performance suggests broader applicability of the model.

The existing literature on stress detection models acknowledges the importance
of generalizability, with a few studies exploring cross-dataset evaluations
(Vos et al., 2023b). These studies typically investigate the applicability of
models across different datasets and sometimes explore combining datasets for
improved performance (refer to Section 2 for details). While prior works often
report limited cross-dataset performance, a key gap exists: a lack of research
systematically investigating the factors influencing this limited
generalizability.

We address this research gap by conducting extensive cross-dataset evaluations
using multiple stress datasets. We train three popular machine learning models
- random forest classifier (RFC), support vector machine (SVM), multi-layer
perceptron (MLP) - using heart rate variability (HRV) features. We aim to
identify the characteristics of these datasets that significantly impact model
generalizability. Our findings are crucial for developing stress detection
models with broader applicability in HMI systems.

##  2\. Related Work

The publicly available datasets such as WESAD (Schmidt et al., 2018) and
SWELL-KW (Koldijk et al., 2014), led to the development of numerous stress
detection models (Can et al., 2019; Haque et al., 2024). Some of these studies
(e.g., (Bobade and Vani, 2020)) focused on comparing multiple models to
determine the best model for stress detection. However, many of these
comparison studies are trained and evaluated on the same datasets. In this
section, we discuss some of the existing works that performed cross-dataset
evaluations on stress detection models.

Mishra et al. (Mishra et al., 2020) conducted cross-dataset evaluations across
four cognitive stress datasets - two had electrocardiogram (ECG) signals, and
the other two had blood volume pulse (BVP) signals. In addition to arithmetic
tasks, the ECG datasets had a startle response test and cold pressor task as
stressors. The authors trained SVM models using HRV features extracted from
the mental stress segments of these datasets. While the ECG-based HRV models
performed well in detecting stress in each other’s arithmetic tasks, they had
a performance drop of 15 - 30% in predicting stress in the BVP datasets. The
authors attributed this drop in performance, despite having the same stressor,
to the difference in sensors. They also noticed an approximately 20 - 40% drop
in the performance when detecting overall stress (including startle and cold
pressor segments), even within the same datasets. Their findings suggest that
the models trained on one type of stressor may not be efficient in detecting
other stress responses.

Prajod and André (Prajod and André, 2022) trained ECG-based deep-learning
models and HRV-based shallow models (RFC, SVM, MLP) on WESAD and SWELL-KW
datasets. While deep learning models outperformed other methods in within-
dataset evaluations, they performed poorly in cross-dataset evaluations (WESAD
models tested on SWELL-KW and vice versa). Although the HRV-based models
performed better than deep learning models in cross-dataset evaluations, their
performances were still low. Moreover, the combining datasets did not improve
the model’s performance on individual datasets. Similarly, Albaladejo-González
et al. (Albaladejo-González et al., 2023) trained HRV-based models on the
WESAD and SWELL-KW datasets. They also observed poor cross-dataset
performances and lack of stress detection improvements by combining datasets.

Benchekroun et al. (Benchekroun et al., 2023) trained two HRV-based models
(RFC, logistic regression) on the MMSD (Benchekroun et al., 2022) and UWS
(Velmovitsky et al., 2021) datasets. They tested the MMSD models using the UWS
data and found that the f1-scores were 12 – 14% lower than the UWS models
(from within-dataset evaluations). They further noted that the f1-score for
stress class was very low (less than 50 %), meaning the models were not very
efficient in detecting stressful instances.

Vos et al. (Vos et al., 2023a) trained shallow models (RFC, SVM, XGBoost)
using heart rate and electro-dermal activity (EDA) features from the SWELL-KW
dataset and evaluated them on WESAD and NEURO (Birjandtalab et al., 2016)
datasets. All three models showed poor cross-dataset performances. They also
implemented an ensemble model and repeated the cross-dataset evaluation.
Although this ensemble model yielded a slight improvement, the performance was
still poor (f1-score $<$ 50%). Furthermore, they trained the ensemble model
using a combined dataset (SWELL-KW, NEURO, UBFC-Phys (Sabour et al., 2021))
and evaluated it on WESAD. While the accuracy increased slightly, the f1-score
dropped further. Their experiments highlight the challenges of developing a
generic stress model.

The above works primarily focused on assessing the generalizability of HRV-
based models. However, Liapis et al. (Liapis et al., 2021) demonstrated that
EDA-based shallow models also struggle with generalizability. They trained
their models on the WESAD dataset and then evaluated them on their own
dataset. Notably, their dataset contained subtle stress instances, unlike
WESAD, implying that the stress intensity might further impact
generalizability.

Baird et al. (Baird et al., 2021) compared models trained on speech features
from three social stress datasets (FAU-TSST, Ulm-TSST and Reg-TSST). These
datasets induced stress following the TSST technique. They predicted cortisol
levels as a proxy for stress levels. In cross-dataset evaluations, the trends
of predicted cortisol levels were aligned for the models, indicating
compatibility between datasets. Due to the dataset compatibility, they
suggested that training models using data from both these datasets could
result in better-performing models.

Table 1 presents an overview of the existing studies on cross-dataset
generalizability of stress detection models. Most studies assess the
generalizability of a model to determine if it is applicable in other stress
scenarios. Some studies take a step further to evaluate whether combining
datasets to train models improves stress detection performances. Although most
studies observe low generalizability of stress models, few studies provide
insights into plausible factors influencing the models’ performance. For
example, Mishra et al. highlighted the poor performance of mental stress
models in detecting physical stress. Similarly, Liapis et al. noted the
difference in stress intensities of the evaluated datasets. Prajod and André
hinted at multiple factors such as stress intensity and measurement devices
that could affect generalizability. However, these studies did not further
investigate these factors. Hence, a crucial question remains relatively
unexplored - What factors or characteristics of the stress datasets need to
match for cross-dataset applicability of models?

Table 1. An overview of the existing works that perform cross-dataset evaluations of their stress models Paper | Input | Datasets | Aim
---|---|---|---
Mishra et al. (Mishra et al., 2020) | HRV |  Own datasets (mainly mental arithmetic tasks) | Assess generalizability
Liapis et al. (Liapis et al., 2021) | EDA |  WESAD, Own UX stress dataset | Assess generalizability
Baird et al. (Baird et al., 2021) | Speech |  Own TSST datasets |  Assess compatibility for combining datasets
Prajod and André (Prajod and André, 2022) | Raw ECG, HRV |  WESAD, SWELL-KW |  Assess generalizability, Combine datasets
Albaladejo-González et al. (Albaladejo-González et al., 2023) | HRV |  WESAD, SWELL-KW |  Assess generalizability, Combine datasets
Benchekroun et al. (Benchekroun et al., 2023) | HRV |  MMSD, UWS | Assess generalizability
Vos et al. (Vos et al., 2023a) | HR, EDA |  WESAD, SWELL-KW, NEURO, UBFC-Phys |  Assess generalizability, Combine datasets
This work | HRV |  WESAD, SWELL-KW, ForDigitStress, VerBIO |  Assess generalizability, Combine datasets, Identify factors influencing generalizability

##  3\. Materials and Methods

###  3.1. Datasets

In this study, we focus on binary stress detection (stress vs. no-stress)
using HRV features. We leverage four publicly available stress datasets: WESAD
(Schmidt et al., 2018), SWELL-KW (Koldijk et al., 2014), ForDigitStress
(Heimerl et al., 2023), and VerBIO (Yadav et al., 2020). While SWELL-KW
contains ECG signals that can be used for extracting HRV, the ForDigitStress
dataset has BVP signals for extracting HRV. The WESAD and VerBIO datasets
contain both ECG and BVP signals. However, we consider only the BVP signals
from the VerBIO dataset to remove some redundant comparisons in our analysis.
A brief overview of the four datasets is presented in Table 2.

####  3.1.1. WESAD

The WESAD dataset is a multimodal stress and affect dataset containing various
physiological signals, including ECG, EDA, and BVP. The data from 15
participants were collected using a chest-worn RespiBan and a wrist-worn
Empatica E4 device. This investigation utilizes the ECG data recorded by the
chest-worn device at 700 Hz.

The participants were subject to three conditions: neutral, amusement, and
stress. In the stress condition, the participants experienced social stress
induced by the TSST technique. The participants engaged in public speaking and
mental arithmetic tasks while being evaluated by a three-member panel. To
induce amusement, the participants watched selected funny video clips. The
experimental sessions began with the neutral condition, followed by the stress
and amusement conditions in alternating order. For each participant, the
neutral condition lasted for approximately 20 minutes, the stress condition
for 10 minutes, and the amusement condition for around 6.5 minutes.

We focus on stress detection, i.e., distinguishing between stress and no-
stress samples. Following the labeling scheme proposed by the dataset
creators, data from both neutral and amusement conditions were considered as
no-stress samples.

####  3.1.2. SWELL-KW

The SWELL-KW dataset is also a multimodal stress dataset that contains two
physiological signals, ECG and EDA. This dataset consists of data from 25
participants who engaged in typical knowledge tasks like writing reports and
presentations. The ECG data was collected using the TMSI Mobi device at a
sampling rate of 2048 Hz.

The participants underwent three experimental conditions: neutral, email
interruptions, and time pressure. During the email interruption session,
participants received eight emails, many irrelevant and some requiring
responses. In the time pressure condition, participants had to complete the
tasks within two-thirds of the allotted neutral session time. Like the WESAD
dataset, the first session was always neutral, followed by the other two
conditions in alternating order. The neutral and email interruption sessions
lasted approximately 45 minutes, while the time pressure session was around 30
minutes long.

Notably, the participants did not report experiencing high stress in any of
the three conditions. However, they indicated a higher temporal demand during
the time pressure session. While training stress detection models, the dataset
creators considered the data from email interruptions and time pressure
sessions as stress samples and the neutral session as no-stress samples
(Koldijk et al., 2016). Therefore, we follow the same labeling scheme for
consistency. However, three participants were excluded due to missing data.

####  3.1.3. ForDigitStress

The ForDigitStress dataset represents another multimodal stress dataset with
various behavioral (facial expression, body pose, etc.) and physiological
(BVP, EDA) signals. The dataset was collected from 40 participants who
attended a mock job interview session. The BVP data was collected using IOM-
biofeedback device, operating at 27 Hz.

The experimental session was divided into three phases: preparation,
interview, and post-interview. The participants were asked to submit their
resumes in advance so that the interviewer could customize the questions
depending on the participant. During the interview phase, the interviewer
questioned the participants on topics such as their strengths/weaknesses,
salary expectations, and hypothetical job-related scenarios. The interview
phase lasted for about 25 mins. In addition to self-reported stress levels,
the participants’ saliva samples were collected for assessing the cortisol
levels. The cortisol levels served as ground truths for the presence of
stress.

Although both preparation and post-interview phases can be considered as no-
stress conditions, the authors suggest using data from later parts of the
post-interview phase based on the cortisol levels. So, we utilize the
interview phase as stress samples and the last segments of post-interview
phase (15 - 20 mins) as no-stress samples.

####  3.1.4. VerBIO

The VerBIO dataset was collected to investigate if exposing participants to
public speaking through virtual reality (VR) training would reduce the public
speaking anxiety.The dataset contains data collected from 55 participants who
were recruited for two real and eight virtual oral presentations over two
days. The physiological data from two wearable sensors, Empatica E4 (BVP, EDA,
skin temperature) and Actiwave Cardio Monitor (ECG) was acquired. Several
self-assessments questionnaires were employed to capture demographics, state-
and trait-based psychological measures.

The experiment was divided into three sessions: pre-interview (Pre), interview
(Test), and post-interview (Post) sessions. The Pre and Post sessions involved
presenting in front of real audience, whereas the Test session involved VR
audience. Each session further constituted relaxation, preparation, and
presentation phases. The presentation topics were of general interest and were
about four minutes long.

We note that the participation dropped through the three sessions, with
maximum participants in the Pre phase and lowest in Post phase. To avoid
drastic imbalances in data between participants, we utilized only the data
from the Pre session. We labeled the data belonging to relaxation phases as
no-stress and those of presentation phase as stressful. In our analysis, we
normalize the data for each participant using few minutes of no-stress data
(see Section 3.3). Hence, 10 participants with low amount of no-stress data
were excluded.

Table 2. An overview of some key characteristics of the four stress datasets |  WESAD |  SWELL-KW |  ForDigitStress |  VerBIO
---|---|---|---|---
Stressor |  TSST |  Interruptions, Time pressure |  Job interview |  Public speaking
Stressor type |  Social |  Cognitive |  Social |  Social
Sensor |  ECG: RespiBan, 700 Hz, BVP: Empatica E4, 64 Hz |  ECG: TMSI Mobi, 2048 Hz |  BVP: IOM, 27 Hz |  BVP: Empatica E4, 64 Hz
Avg. stress level |  18.5/24 (STAI) |  3.5/10 (Likert scale) |  5.4/10 (Likert scale), 6.5/10 (Cortisol) |
Participants |  15 |  22 |  40 |  45
Data duration (per participant) |  stress: 10 mins, no-stress: 26.5 mins |  stress: 75 mins, no-stress: 45 mins |  stress: 25 mins, no-stress: 15 - 20 mins |  stress: 2 - 5 mins, no-stress: 4 - 6 mins

###  3.2. Approach

Our investigations involve the three assessments listed below. The idea is to
run a series of these assessments using the datasets described in Section 3.1
to investigate the dataset characteristics that considerably influence model
generalizability.

  1. (1)

Within-dataset Assessment: involves training and evaluating models on the same
dataset. We utilize the leave-one-subject-out (LOSO) technique to evaluate the
models’ performance on unseen participants of the same dataset.

  2. (2)

Cross-dataset Assessment: involves evaluating the models trained on one
dataset using another dataset. This evaluation assesses to what extent these
models can detect stress in new participants in different settings.

  3. (3)

Combining Datasets: involves training new models on a combined dataset
consisting of data from two or more stress datasets, again using the LOSO
technique. This step investigates potential improvements in models’
performances due to an increase in the number and variations of the training
data.

While ECG-based and BVP-based HRVs reflect similar physiological information
related to heart activity, the models trained on these features might not
perform equally well (Gupta et al., 2023). This discrepancy can be attributed
to the BVP signals being more prone to noise from body movements than ECG
(Martinho et al., 2018). This noise can affect the signal quality, which can,
in turn, impact the stress detection performance. To avoid such influences in
cross-dataset assessment, the above assessments are followed for ECG-based and
BVP-based HRV models separately.

###  3.3. Data Processing

####  3.3.1. ECG signals

Figure 1. An example plot of ECG signal, marked with various repeated
components of the signal.

A woman and a girl in white dresses sit in an open car.

The ECG signals are typically sampled at high frequencies and contain noises
such as baseline wander (low-frequency, 0.5 - 0.6 Hz) and powerline
interference (50 or 60 Hz). Figure 1 illustrates two beats from an ECG signal.
Computing HRV relies on detecting the R peaks in the QRS complex of the ECG
signal. We applied a second-order Butterworth band-pass filter with a
frequency band of 8 - 20 Hz for optimal QRS signal-to-noise ratio (Elgendi et
al., 2010).

For R-peak detection, we utilized the algorithm proposed by (Elgendi et al.,
2010). This algorithm is based on two key assumptions for healthy adults:

  1. (a)

A QRS complex contains one and only one heartbeat

  2. (b)

The duration of a typical QRS complex is in the range of 80 - 120 milliseconds

####  3.3.2. BVP signals

Figure 2. An example plot of BVP signal, marked with various repeated
components of the signal.

A woman and a girl in white dresses sit in an open car.

Like ECG, the BVP signal is susceptible to baseline wander and high-frequency
noise. Hence, a band-pass filter (0.5 - 8 Hz) was applied to reduce these
noises (Elgendi et al., 2013). Figure 2 illustrates two beats from a BVP
signal. To derive the HRV signal from the BVP, the systolic peaks had to be
detected. For this purpose, we employed the peak-finding algorithm from
(Heimerl et al., 2023) to detect points that meet the following criteria:

  1. (a)

Amplitude threshold: The peaks had to be taller than a certain threshold. This
threshold was set based on the distribution of peak heights in the entire
signal.

  2. (b)

Distance between peaks: To avoid identifying every fluctuation as a peak,
consecutive peaks had to be separated by a minimum interval of 0.333 seconds.
This value corresponds to a maximum heart rate of 3 beats per second (180
beats per minute).

####  3.3.3. HRV Features

Once the R-peaks (for ECG) or systolic peaks (for BVP) were identified, the
time intervals between successive peaks were calculated to form the HRV
signals. The features were calculated using 60-second segments with 59 seconds
of overlap between consecutive segments.

A total of 22 well-known features (Schmidt et al., 2018; Pham et al., 2021;
Heimerl et al., 2023; Prajod et al., [n. d.]) were computed from the extracted
HRV signals. These features belonged to the time domain (13 features),
frequency domain (5 features), and poincaré plot characteristics (4 features).
These features were calculated using the NeuroKit2 Python library (Makowski et
al., 2021).

The sensors used in the different datasets differ, potentially resulting in
values recorded on different scales. In addition, the range of physiological
recordings may vary from participant to participant (Braithwaite et al., 2013;
Nkurikiyeyezu et al., 2019; Sarkar and Etemad, 2020). To mitigate the effect
of these differences, participant-specific Min-Max normalization was applied
to each HRV feature. For real-time stress detection, the entire dataset would
not be available for normalization. Similar to (Luong et al., 2020; Prajod and
André, 2022), we used 5 minutes of neutral data to compute normalization
parameters (minimum and maximum values) for each participant. For the VerBIO
dataset, we used neutral data for normalization because many of the
participants had less than 5 minutes of neutral data.

###  3.4. Machine Learning Models

We trained the following three machine learning models using the extracted HRV
features, following the LOSO procedure. To account for the imbalanced sample
distribution of the datasets, the “class_weight” hyperparameter for all models
was set inversely proportional to the sample frequencies.

####  3.4.1. Random Forest Classifier (RFC)

This is an ensemble learning method that combines predictions from multiple
decision trees for improved performance and reduced overfitting. Each tree is
trained on a subset of the available training set. The final prediction is
determined by aggregating the predictions from all the trees (e.g., majority
vote). This strategy often results in a better performance, even if the
individual decision trees are weak predictors. A total of 200 decision trees
(also called estimators) were trained, with a maximum depth of 5 for each
tree.

####  3.4.2. Support Vector Machine (SVM)

This is a commonly used supervised learning method for binary classification
tasks. During the training process of this model, the objective is to find a
hyperplane within the feature space that separates the data points belonging
to different classes. We utilized a linear SVM classifier.

####  3.4.3. Multi-layer Perceptron (MLP)

This is a simple feed-forward neural network (also called simple artificial
neural network), which has been growing in popularity for stress detection
(Bobade and Vani, 2020; Zawad et al., 2023; Albaladejo-González et al., 2023).
Our implementation followed an architecture consisting of an input layer, two
hidden layers, and a prediction layer. The input layer received data
represented as the normalized HRV features. A dropout layer (rate $=$ 0.2) was
included after the input layer to mitigate overfitting of the model. The two
hidden layers (ReLU activation) followed the dropout layer, with 12 nodes for
the first hidden layer and 6 nodes for the second hidden layer. This final
layer outputs the classification result using a Sigmoid activation function.

This model was trained using the SGD optimizer (learning rate $=$ 0.001) and
weighted loss. It was also trained in batches of 256 samples. We utilized the
early stopping technique, where the training stopped if the validation loss
did not decrease for 15 consecutive iterations.

##  4\. Results

We employed three evaluation strategies to assess model performances: within-
dataset, cross-dataset, and combined dataset evaluations. For all evaluations,
accuracy and f1-score metrics were used to quantify model performance. The
detailed results for each evaluation strategy are presented below.

###  4.1. Within-dataset Assessment

We employed LOSO technique to train and evaluate model performance within each
of the four stress datasets. The average f1-score and accuracy for each
dataset are presented in Table 3.

Table 3. Results of within-dataset LOSO evaluation of HRV models conducted on the four stress datasets Model | F1-score | Accuracy
---|---|---
Test on SWELL-KW, ECG
SWELL-KW baseline (Koldijk et al., 2016) | - | 0.589
SWELL-KW RFC | 0.634 | 0.664
SWELL-KW SVM | 0.597 | 0.626
SWELL-KW MLP | 0.667 | 0.692
Test on WESAD, ECG
WESAD (ECG-HRV) baseline (Schmidt et al., 2018) | 0.813 | 0.854
WESAD (ECG-HRV) RFC | 0.819 | 0.863
WESAD (ECG-HRV) SVM | 0.814 | 0.862
WESAD (ECG-HRV) MLP | 0.844 | 0.880
Test on WESAD, BVP
WESAD (BVP-HRV) baseline (Schmidt et al., 2018) | 0.830 | 0.858
WESAD (BVP-HRV) RFC | 0.768 | 0.826
WESAD (BVP-HRV) SVM | 0.763 | 0.792
WESAD (BVP-HRV) MLP | 0.780 | 0.829
Test on ForDigitStress, BVP
ForDigitStress baseline (Heimerl et al., 2023) | 0.784 | 0.797
ForDigitStress RFC | 0.810 | 0.829
ForDigitStress SVM | 0.787 | 0.811
ForDigitStress MLP | 0.814 | 0.831
Test on VerBIO, BVP
VerBIO baseline (Yadav et al., 2020) | - | -
VerBIO RFC | 0.949 | 0.962
VerBIO SVM | 0.879 | 0.904
VerBIO MLP | 0.924 | 0.939

The VerBIO dataset yielded the best overall performance, with models achieving
average accuracies and f1-scores exceeding 90%. On the other hand, models
trained on the SWELL-KW dataset exhibited lower average performance, with
f1-scores and accuracies ranging from 60% to 70%. All other datasets achieved
average f1-scores above 75% and average accuracies greater than 80%.

Except for VerBIO, MLP consistently achieved the highest performance within
each dataset, followed by RFC and then SVM. For VerBIO, RFC outperformed all
other models. Notably, all our models surpassed the baselines established in
the respective dataset papers, except for the WESAD BVP-HRV models. The
performance differences between models were relatively small, typically within
a 5% margin.

Within the WESAD dataset, models trained on ECG-derived HRV features
outperformed those using BVP-HRV features. However, the WESAD dataset paper
(Schmidt et al., 2018) reported slightly higher performance for BVP-HRV
models.

###  4.2. Cross-dataset Assessment

To assess the generalizability of the models, we conducted cross-dataset
evaluations. Each LOSO model trained on one dataset was tested on unseen data
from another dataset. For ECG-derived HRV features, we tested the SWELL-KW
models on data from the WESAD dataset, and vice versa. The results for these
cross-dataset evaluations are presented in Table 4 (SWELL-KW) and Table 5
(WESAD, ECG).

Table 4. Cross-dataset evaluation of SWELL-KW models on WESAD (ECG-HRV) dataset Model | F1-score | Accuracy
---|---|---
Test on WESAD (ECG-HRV)
SWELL-KW RFC | 0.557 | 0.676
SWELL-KW SVM | 0.357 | 0.482
SWELL-KW MLP | 0.427 | 0.574

The SWELL-KW models performed poorly in the cross-dataset evaluation using
WESAD ECG-HRV data. The RFC models achieved the best average performance with
an f1-score of 55.7% and an accuracy of 67.6%. These values are considerably
lower than the worst-performing within-dataset WESAD (ECG-HRV) model, which
achieved f1-score $=$ 81.4% and accuracy $=$ 86.2%.

Table 5. Cross-dataset evaluation of WESAD (ECG-HRV) models on SWELL-KW dataset Model | F1-score | Accuracy
---|---|---
Test on SWELL-KW
WESAD (ECG-HRV) RFC | 0.432 | 0.450
WESAD (ECG-HRV) SVM | 0.421 | 0.439
WESAD (ECG-HRV) MLP | 0.468 | 0.479

Like SWELL-KW models, the WESAD ECG-HRV models also performed poorly in cross-
dataset evaluation. While MLP achieved the highest average performance in this
cross-dataset evaluation, all models yielded f1-scores and accuracies below
50%.

We employed a similar cross-dataset evaluation approach for models trained on
BVP-derived HRV features. Each LOSO model was tested on unseen data from two
different datasets. For instance, the WESAD (BVP-HRV) models were evaluated
using data from the ForDigitStress and VerBIO datasets. The results of cross
dataset evaluation for WESAD (BVP-HRV), ForDigitStress, and VerBIO models are
presented in Tables 6, 7, and 8, respectively.

Table 6. Cross-dataset evaluation of WESAD (BVP-HRV) models on ForDigitStress and VerBIO datasets Model | F1-score | Accuracy
---|---|---
Test on ForDigitStress
WESAD (BVP-HRV) RFC | 0.731 | 0.740
WESAD (BVP-HRV) SVM | 0.774 | 0.775
WESAD (BVP-HRV) MLP | 0.740 | 0.744
Test on VerBIO
WESAD (BVP-HRV) RFC | 0.904 | 0.907
WESAD (BVP-HRV) SVM | 0.873 | 0.877
WESAD (BVP-HRV) MLP | 0.888 | 0.893

The models trained on WESAD BVP-HRV features showed good cross-dataset
performance on ForDigitStress and VerBIO datasets. SVM achieved the best
average f1-score and accuracy on the ForDigitStress dataset, whereas RFC was
better in the VerBIO dataset. The performance drop compared to the best
within-dataset models on ForDigitStress and VerBIO was minimal, ranging from
4% to 6%.

Table 7. Cross-dataset evaluation of ForDigitStress models on WESAD (BVP-HRV) and VerBIO datasets Model | F1-score | Accuracy
---|---|---
Test on WESAD (BVP-HRV)
ForDigitStress RFC | 0.789 | 0.820
ForDigitStress SVM | 0.779 | 0.812
ForDigitStress MLP | 0.763 | 0.810
Test on VerBIO
ForDigitStress RFC | 0.856 | 0.865
ForDigitStress SVM | 0.836 | 0.846
ForDigitStress MLP | 0.784 | 0.805

The ForDigitStress models performed well on data from WESAD (BVP-HRV) and
VerBIO datasets, with all models achieving f1-scores higher than 75% and more
than 80% accuracy. The RFC showed the best performance on both external
datasets.

Table 8. Cross-dataset evaluation of VerBIO models on WESAD (BVP-HRV) and ForDigitStress datasets Model | F1-score | Accuracy
---|---|---
Test on WESAD (BVP-HRV)
VerBIO RFC | 0.744 | 0.759
VerBIO SVM | 0.769 | 0.789
VerBIO MLP | 0.780 | 0.803
Test on ForDigitStress
VerBIO RFC | 0.823 | 0.823
VerBIO SVM | 0.684 | 0.688
VerBIO MLP | 0.747 | 0.752

The VerBIO models exhibited good generalizability to unseen data from both
WESAD (BVP-HRV) and ForDigitStress datasets. A notable exception was the SVM
models tested on ForDigitStress achieved f1-scores slightly below 70%,
representing a drop of 10 - 12% compared to the lowest performing within-
dataset ForDigitStress model. Interestingly, the VerBIO MLP - which performed
the best on ForDigitStress data - achieved a slightly better performance than
the within-dataset ForDigitStress models.

###  4.3. Combining Datasets

To explore the potential benefits of combining datasets, we trained additional
models using the LOSO technique. The SWELL-KW and WESAD (ECG-HRV) data were
combined to train ECG-derived HRV models. Similarly, a combined dataset
consisting of WESAD (BVP-HRV), ForDigitStress, and VerBIO data was used to
train BVP-derived HRV models. The LOSO results for combined ECG-based and BVP-
based HRV models are presented in Tables 9 and 10, respectively.

Table 9. Results of LOSO evaluation of ECG-derived HRV models trained by combining data from SWELL-KW and WESAD (ECG-HRV) datasets Model | F1-score | Accuracy
---|---|---
Test on SWELL-KW
Combined ECG-HRV RFC | 0.644 | 0.671
Combined ECG-HRV SVM | 0.587 | 0.615
Combined ECG-HRV MLP | 0.660 | 0.679
Test on WESAD (ECG-HRV)
Combined ECG-HRV RFC | 0.679 | 0.761
Combined ECG-HRV SVM | 0.526 | 0.652
Combined ECG-HRV MLP | 0.718 | 0.792
Combined ECG-HRV Results
Combined ECG-HRV RFC | 0.658 | 0.707
Combined ECG-HRV SVM | 0.587 | 0.615
Combined ECG-HRV MLP | 0.683 | 0.725

Among the models trained on combined SWELL-KW and WESAD (ECG-HRV), MLP
outperformed others on both datasets. However, the performances on each
dataset were lower than the within-dataset models. While the drop in
performance for SWELL-KW dataset was relatively small, there was around 12%
drop in f1-score and 9% drop in accuracy for WESAD dataset.

Table 10. Results of LOSO evaluation of BVP-derived HRV models trained by combining data from WESAD (BVP-HRV), ForDigitStress, and VerBIO datasets Model | F1-score | Accuracy
---|---|---
Test on WESAD (BVP-HRV)
Combined BVP-HRV RFC | 0.785 | 0.816
Combined BVP-HRV SVM | 0.768 | 0.813
Combined BVP-HRV MLP | 0.823 | 0.863
Test on ForDigitStress
Combined BVP-HRV RFC | 0.809 | 0.828
Combined BVP-HRV SVM | 0.776 | 0.800
Combined BVP-HRV MLP | 0.811 | 0.831
Test on VerBIO
Combined BVP-HRV RFC | 0.909 | 0.934
Combined BVP-HRV SVM | 0.845 | 0.886
Combined BVP-HRV MLP | 0.880 | 0.913
Combined BVP-HRV Results
Combined BVP-HRV RFC | 0.850 | 0.874
Combined BVP-HRV SVM | 0.806 | 0.840
Combined BVP-HRV MLP | 0.844 | 0.873

Combining WESAD (BVP-HRV), ForDigitStress, and VerBIO datasets resulted in
models with good stress detection performance across all three datasets. The
best average f1-scores and accuracies for each dataset was greater than 80%.
MLP trained on the combined dataset outperformed the best within-dataset WESAD
(BVP-HRV) model by around 4%. The best performance of combined models on the
ForDigitStress dataset was similar to the within-dataset results. However, in
the VerBIO dataset, the highest average performance was 3 - 4% lower than the
best within-dataset performance.

##  5\. Discussion

The MLP models achieved the best results in most within-dataset evaluations.
Our observation aligns with the findings of (Bobade and Vani, 2020; Prajod and
André, 2022; Albaladejo-González et al., 2023), where a simple feed-forward
network achieved better performance than other machine learning methods such
as SVM and RFC. We also observed that RFC performed better in many cross-
dataset evaluations. However, this trend is not very consistent across
datasets.

Cross-dataset evaluations revealed significant limitations in generalizability
for models trained on SWELL-KW and WESAD (ECG-HRV) datasets. Combining these
datasets did not improve performance and, in the case of WESAD models, even
led to a decline. This observation highlights the importance of data
compatibility when considering such strategies. Our findings regarding the
cross-dataset performance and combining datasets with respect to SWELL-KW and
WESAD datasets are in line with the observations of previous works (Prajod and
André, 2022; Albaladejo-González et al., 2023).

As highlighted in Table 2, the SWELL-KW and WESAD (ECG-HRV) datasets differ in
many factors including stressors, experienced stress intensity, and the
measurement devices. To pin-point the dataset characteristics which
considerably influences the cross-dataset performance, we considered two
additional datasets: ForDigitStress and VerBIO. Cross-dataset evaluations were
conducted using these additional datasets and WESAD. The WESAD and VerBIO
datasets share more similarities than the ForDigitStress dataset. Both
datasets utilized the Empatica E4 device to measure the BVP signals. While
WESAD employed TSST to elicit stress, VerBIO relied on public speaking task
that is a sub-task of the TSST protocol. On the other hand, the ForDigitStress
dataset utilized a different measurement device and employed a mock job
interview to induce stress. However, we note that the three datasets rely on
social evaluation as a primary source of stress.

The models trained on these three datasets (WESAD, VerBIO, ForDigitStress)
exhibited good cross-dataset performance. This suggests that factors like
brand of measurement device, stress-elicitation technique, and even stress
intensity (within a reasonable range) may not significantly impact
generalizability when the stressor type remains consistent (social stress in
this case). Together with the findings of (Mishra et al., 2020) and (Baird et
al., 2021) \- where good cross-dataset performances were observed in datasets
involving virtually same tasks (mental arithmetic and TSST, respectively) - we
infer that stressor type plays a crucial role in cross-dataset
generalizability of stress models.

While our results suggest stress intensity may not be a critical factor within
a reasonable range, it warrants further exploration. The low stress intensity
in the SWELL-KW dataset might have contributed to the poor performance of
WESAD models trained on high-intensity stress data. Interestingly, SWELL-KW
models, designed for low-intensity stress detection, also struggled with high-
intensity stress from WESAD data.

Overall, this study highlights the importance of considering stressor type and
data compatibility when developing generalizable stress detection models.

##  6\. Conclusion

Stress detection models with broader applicability are crucial due to the
diverse nature of stress experiences across various scenarios. Identifying
factors that influence cross-dataset generalizability is essential for
achieving this goal. This study addressed this gap by conducting cross-dataset
evaluations on four datasets (SWELL-KW, WESAD, ForDigitStress, VerBIO), which
contains both shared and distinct characteristics. We trained HRV-based
machine learning models (RFC, SVM, MLP) using ECG or BVP signals from these
datasets. Our key finding is that stressor type is the most prominent factor
influencing cross-dataset applicability. Models trained on datasets with
similar stressor types exhibited good generalizability, while those with
different stressors showed lower performance. Conversely, factors like stress
elicitation method and stress intensity had a minimal impact within the
explored datasets. Furthermore, matching stressor type proved crucial for
enhancing stress detection performance when combining datasets.

This study focused on ECG- and BVP-derived HRV features. In future works, we
will explore whether these findings extend to other stress-related modalities
(e.g., EDA). Additionally, a more extensive evaluation involving a wider range
of datasets encompassing different stress types (e.g., physical stress) would
further validate these observations.

## References

  * (1)
  * Akmandor and Jha (2017) Ayten Ozge Akmandor and Niraj K Jha. 2017.  Keep the stress away with SoDA: Stress detection and alleviation system.  _IEEE Transactions on Multi-Scale Computing Systems_ 3, 4 (2017), 269–282.
  * Albaladejo-González et al. (2023) Mariano Albaladejo-González, José A Ruipérez-Valiente, and Félix Gómez Mármol. 2023.  Evaluating different configurations of machine learning models and their transfer learning capabilities for stress detection using heart rate.  _Journal of Ambient Intelligence and Humanized Computing_ 14, 8 (2023), 11011–11021.
  * Alberdi et al. (2016) Ane Alberdi, Asier Aztiria, and Adrian Basarab. 2016.  Towards an automatic early stress recognition system for office environments based on multimodal measurements: A review.  _Journal of biomedical informatics_ 59 (2016), 49–75.
  * Baird et al. (2021) Alice Baird, Andreas Triantafyllopoulos, Sandra Zänkert, Sandra Ottl, Lukas Christ, Lukas Stappen, Julian Konzok, Sarah Sturmbauer, Eva-Maria Meßner, Brigitte M Kudielka, et al. 2021\.  An evaluation of speech-based recognition of emotional and physiological markers of stress.  _Frontiers in Computer Science_ 3 (2021), 750284.
  * Balcombe and De Leo (2022) Luke Balcombe and Diego De Leo. 2022.  Human-computer interaction in digital mental health. In _Informatics_ , Vol. 9. MDPI, 14.
  * Benchekroun et al. (2022) Mouna Benchekroun, Dan Istrate, Vincent Zalc, and Dominique Lenne. 2022.  A Multi-Modal Dataset (MMSD) for Acute Stress Bio-Markers. In _International Joint Conference on Biomedical Engineering Systems and Technologies_. Springer, 377–392.
  * Benchekroun et al. (2023) Mouna Benchekroun, Pedro Elkind Velmovitsky, Dan Istrate, Vincent Zalc, Plinio Pelegrini Morita, and Dominique Lenne. 2023.  Cross dataset analysis for generalizability of HRV-based stress detection models.  _Sensors_ 23, 4 (2023), 1807.
  * Birjandtalab et al. (2016) Javad Birjandtalab, Diana Cogan, Maziyar Baran Pouyan, and Mehrdad Nourani. 2016.  A non-EEG biosignals dataset for assessment and visualization of neurological status. In _2016 IEEE International Workshop on Signal Processing Systems (SiPS)_. IEEE, 110–114.
  * Bobade and Vani (2020) Pramod Bobade and M Vani. 2020.  Stress detection with machine learning and deep learning using multimodal physiological data. In _2020 Second International Conference on Inventive Research in Computing Applications (ICIRCA)_. IEEE, 51–57.
  * Braithwaite et al. (2013) Jason J Braithwaite, Derrick G Watson, Robert Jones, and Mickey Rowe. 2013.  A guide for analysing electrodermal activity (EDA) & skin conductance responses (SCRs) for psychological experiments.  _Psychophysiology_ 49, 1 (2013), 1017–1034.
  * Can et al. (2019) Yekta Said Can, Bert Arnrich, and Cem Ersoy. 2019.  Stress detection in daily life scenarios using smart phones and wearable sensors: A survey.  _Journal of biomedical informatics_ 92 (2019), 103139.
  * Elgendi et al. (2010) Mohamed Elgendi, Mirjam Jonkman, and Friso De Boer. 2010.  Frequency Bands Effects on QRS Detection.  _Biosignals_ 2003 (2010), 2002.
  * Elgendi et al. (2013) Mohamed Elgendi, Ian Norton, Matt Brearley, Derek Abbott, and Dale Schuurmans. 2013.  Systolic peak detection in acceleration photoplethysmograms measured from emergency responders in tropical conditions.  _PloS one_ 8, 10 (2013), e76585.
  * Giannakakis et al. (2019) Giorgos Giannakakis, Dimitris Grigoriadis, Katerina Giannakaki, Olympia Simantiraki, Alexandros Roniotis, and Manolis Tsiknakis. 2019.  Review on psychological stress detection using biosignals.  _IEEE transactions on affective computing_ 13, 1 (2019), 440–460.
  * Greene et al. (2016) Shalom Greene, Himanshu Thapliyal, and Allison Caban-Holt. 2016.  A survey of affective computing for stress detection: Evaluating technologies in stress detection for better health.  _IEEE Consumer Electronics Magazine_ 5, 4 (2016), 44–56.
  * Gupta et al. (2023) Rohit Gupta, Amit Bhongade, and Tapan Kumar Gandhi. 2023.  Multimodal Wearable Sensors-based Stress and Affective States Prediction Model. In _2023 9th International Conference on Advanced Computing and Communication Systems (ICACCS)_ , Vol. 1. IEEE, 30–35.
  * Haque et al. (2024) Yeaminul Haque, Rahat Shahriar Zawad, Chowdhury Saleh Ahmed Rony, Hasan Al Banna, Tapotosh Ghosh, M Shamim Kaiser, and Mufti Mahmud. 2024.  State-of-the-Art of Stress Prediction from Heart Rate Variability Using Artificial Intelligence.  _Cognitive Computation_ 16, 2 (2024), 455–481.
  * Heimerl et al. (2023) Alexander Heimerl, Pooja Prajod, Silvan Mertes, Tobias Baur, Matthias Kraus, Ailin Liu, Helen Risack, Nicolas Rohleder, Elisabeth André, and Linda Becker. 2023.  ForDigitStress: A multi-modal stress dataset employing a digital job interview scenario.  _arXiv preprint arXiv:2303.07742_ (2023).
  * Koldijk et al. (2016) Saskia Koldijk, Mark A Neerincx, and Wessel Kraaij. 2016.  Detecting work stress in offices by combining unobtrusive sensors.  _IEEE Transactions on affective computing_ 9, 2 (2016), 227–239.
  * Koldijk et al. (2014) Saskia Koldijk, Maya Sappelli, Suzan Verberne, Mark A Neerincx, and Wessel Kraaij. 2014.  The SWELL knowledge work dataset for stress and user modeling research. In _Proceedings of the 16th international conference on multimodal interaction_. 291–298.
  * Liapis et al. (2021) Alexandros Liapis, Evanthia Faliagka, Christos Katsanos, Christos Antonopoulos, and Nikolaos Voros. 2021.  Detection of subtle stress episodes during UX evaluation: Assessing the performance of the WESAD bio-signals dataset. In _Human-Computer Interaction–INTERACT 2021: 18th IFIP TC 13 International Conference, Bari, Italy, August 30–September 3, 2021, Proceedings, Part III 18_. Springer, 238–247.
  * Luong et al. (2020) Tiffany Luong, Nicolas Martin, Anais Raison, Ferran Argelaguet, Jean-Marc Diverrez, and Anatole Lécuyer. 2020.  Towards real-time recognition of users mental workload using integrated physiological sensors into a VR HMD. In _2020 IEEE international symposium on mixed and augmented reality (ISMAR)_. IEEE, 425–437.
  * Makowski et al. (2021) Dominique Makowski, Tam Pham, Zen J Lau, Jan C Brammer, François Lespinasse, Hung Pham, Christopher Schölzel, and SH Annabel Chen. 2021.  NeuroKit2: A Python toolbox for neurophysiological signal processing.  _Behavior research methods_ (2021), 1–8.
  * Martinho et al. (2018) Miguel Martinho, Ana Fred, and Hugo Silva. 2018.  Towards continuous user recognition by exploring physiological multimodality: An electrocardiogram (ECG) and blood volume pulse (BVP) approach. In _2018 International Symposium in Sensing and Instrumentation in IoT Era (ISSI)_. IEEE, 1–6.
  * Mishra et al. (2020) Varun Mishra, Sougata Sen, Grace Chen, Tian Hao, Jeffrey Rogers, Ching-Hua Chen, and David Kotz. 2020.  Evaluating the reproducibility of physiological stress detection models.  _Proceedings of the ACM on interactive, mobile, wearable and ubiquitous technologies_ 4, 4 (2020), 1–29.
  * Nkurikiyeyezu et al. (2019) Kizito Nkurikiyeyezu, Anna Yokokubo, and Guillaume Lopez. 2019.  The effect of person-specific biometrics in improving generic stress predictive models.  _arXiv preprint arXiv:1910.01770_ (2019).
  * Pham et al. (2021) Tam Pham, Zen Juen Lau, SH Annabel Chen, and Dominique Makowski. 2021.  Heart rate variability in psychology: A review of HRV indices and an analysis tutorial.  _Sensors_ 21, 12 (2021), 3998.
  * Prajod and André (2022) Pooja Prajod and Elisabeth André. 2022.  On the Generalizability of ECG-based Stress Detection Models. In _2022 21st IEEE International Conference on Machine Learning and Applications (ICMLA)_. IEEE, 549–554.
  * Prajod et al. ([n. d.]) Pooja Prajod, Matteo Lavit Nicora, Marta Mondellini, Matteo Meregalli Falerni, Rocco Vertechy, Matteo Malosio, and Elisabeth André. [n. d.].  Flow in Human-Robot Collaboration-Multimodal Analysis and Perceived Challenge Detection in Industrial Scenarios.  _Frontiers in Robotics and AI_ 11 ([n. d.]), 1393795.
  * Sabour et al. (2021) Rita Meziati Sabour, Yannick Benezeth, Pierre De Oliveira, Julien Chappe, and Fan Yang. 2021.  Ubfc-phys: A multimodal database for psychophysiological studies of social stress.  _IEEE Transactions on Affective Computing_ 14, 1 (2021), 622–636.
  * Sarkar and Etemad (2020) Pritam Sarkar and Ali Etemad. 2020.  Self-supervised ECG representation learning for emotion recognition.  _IEEE Transactions on Affective Computing_ 13, 3 (2020), 1541–1554.
  * Schmidt et al. (2018) Philip Schmidt, Attila Reiss, Robert Duerichen, Claus Marberger, and Kristof Van Laerhoven. 2018.  Introducing WESAD, a multimodal dataset for wearable stress and affect detection. In _Proceedings of the 20th ACM international conference on multimodal interaction_. 400–408.
  * Velmovitsky et al. (2021) Pedro Elkind Velmovitsky, Paulo Alencar, Scott T Leatherdale, Donald Cowan, and Plinio Pelegrini Morita. 2021.  Towards real-time public health: a novel mobile health monitoring system. In _2021 IEEE International Conference on Big Data (Big Data)_. IEEE, 6049–6051.
  * Vos et al. (2023a) Gideon Vos, Kelly Trinh, Zoltan Sarnyai, and Mostafa Rahimi Azghadi. 2023a.  Ensemble machine learning model trained on a new synthesized dataset generalizes well for stress prediction using wearable devices.  _Journal of Biomedical Informatics_ 148 (2023), 104556.
  * Vos et al. (2023b) Gideon Vos, Kelly Trinh, Zoltan Sarnyai, and Mostafa Rahimi Azghadi. 2023b.  Generalizable machine learning for stress monitoring from wearable devices: a systematic literature review.  _International Journal of Medical Informatics_ 173 (2023), 105026.
  * Yadav et al. (2020) Megha Yadav, Md Nazmus Sakib, Ehsanul Haque Nirjhar, Kexin Feng, Amir H Behzadan, and Theodora Chaspari. 2020.  Exploring individual differences of public speaking anxiety in real-life and virtual presentations.  _IEEE Transactions on Affective Computing_ 13, 3 (2020), 1168–1182.
  * Yu et al. (2018) Bin Yu, Mathias Funk, Jun Hu, Qi Wang, and Loe Feijs. 2018.  Biofeedback for everyday stress management: A systematic review.  _Frontiers in ICT_ 5 (2018), 23.
  * Zawad et al. (2023) Md Rahat Shahriar Zawad, Chowdhury Saleh Ahmed Rony, Md Yeaminul Haque, Md Hasan Al Banna, Mufti Mahmud, and M Shamim Kaiser. 2023.  A Hybrid Approach for Stress Prediction from Heart Rate Variability.  In _Frontiers of ICT in Healthcare: Proceedings of EAIT 2022_. Springer, 111–121.