11institutetext: Pattern Recognition and Bioinformatics, Delft University of
Technology, Delft, The Netherlands
22institutetext: Department of Cardiology, Haga Teaching Hospital, The Hague,
The Netherlands

# Improving performance of heart rate time series classification by grouping
subjects††thanks: First and second author contributed equally.

Michael Beekhuizen Arman Naseri David Tax Ivo van der Bilt Marcel Reinders 11
0009-0000-3874-6527 11 2 2 0000-0002-5676-6596 11 0000-0002-5153-9087 22
0000-0002-1151-2461 11 0000-0002-1148-1562

###### Abstract

Unlike the more commonly analyzed ECG or PPG data for activity classification,
heart rate time series data is less detailed, often noisier and can contain
missing data points. Using the BigIdeasLab_STEP dataset, which includes heart
rate time series annotated with specific tasks performed by individuals, we
sought to determine if general classification was achievable.

Our analyses showed that the accuracy is sensitive to the choice of
window/stride size. Moreover, we found variable classification performances
between subjects due to differences in the physical structure of their hearts.
Various techniques were used to minimize this variability. First of all,
normalization proved to be a crucial step and significantly improved the
performance. Secondly, grouping subjects and performing classification inside
a group helped to improve performance and decrease inter-subject variability.
Finally, we show that including handcrafted features as input to a deep
learning (DL) network improves the classification performance further.

Together, these findings indicate that heart rate time series can be utilized
for classification tasks like predicting activity. However, normalization or
grouping techniques need to be chosen carefully to minimize the issue of
subject variability.

###### Keywords:

Wearables Deep learning Subject grouping.

##  1 Introduction

In recent years, wearable devices and smartwatches have been equipped with
more sensors, including electrocardiogram (ECG) and photoplethysmography (PPG)
sensors, for the estimation of heart rate and heart rhythm [14]. These
developments enable us to collect long-term heart rate time series data of a
subject’s heart rate in beats per minute (BPM). In the research community,
there are many papers that attempt to perform classification using ECG or PPG
data. While ECG and PPG data shows each heartbeat’s characteristics in detail,
heart rate data summarizes this based on the time elapsed between heartbeats.
For heart rate, we receive a single measurement, representing beats per
minute, at regular intervals—like once every few seconds. This is therefore a
more challenging signal to perform classification tasks on. Research on the
analysis and usage of heart rate time series has been performed for example
for cardiovascular risk detection[1][6]) and sleep analysis [15]. In this
paper, we will look into the classification of heart rate time series data to
predict different activities a subject is doing. This is interesting to test
because it would imply that we can use heart rate data in the future for more
complex classification problems, like heart disease detection. We will make
use of the BigIdeasLab_STEP[4] dataset which contains annotated heart rate
time series of subjects performing different activities.

Magure et al. perform activity classification[9] which was mostly possible due
to the fact that the accelerometer was placed at strategic places to identify
specific movements and the subjects all had the same age and fitness level. In
contrast, Bent and Dunn[4] conducted a study involving subjects of varying
skin tones performing different physical activities while wearing multiple
smartwatches. Their findings revealed no statistically significant difference
in accuracy across skin tones. However, there were notable increases in error
during physical activity compared to rest. Specifically, the absolute error
during physical activity was on average 30% greater than during periods of
rest.

However, constructing a unified classification model is challenging due to the
diverse characteristics between devices and subjects. Therefore we propose to
group similar subjects together and construct a model for each group.

##  2 Results

The BigIdeasLab_STEP dataset contains around 13 minutes of heart rate time
series data per subject. This dataset is annotated with the activity a subject
is performing. The activities were: resting, breathing, performing an activity
(walking), resting after the activity and typing. The data is split up into
windows of fixed size and a specific stride is used between each window. A
window size refers to the number of consecutive samples one takes from a
certain start point. The stride indicates the number of samples the start
point is shifted for the next window. A more detailed description of the
dataset can be found in the methods and section 0.A.1.

###  2.1 Comparison of different window and stride sizes

First, we investigate the influence of varying window and stride sizes. For
that, we trained a Support Vector Machine (SVM) on windows sizes of 50, 80,
100 and 120 and stride sizes of 10, 25, 40, 50, 80, 100 and 120. A short
explanation of the SVM can be found in the methods and appendix 0.A.2. The
input data used was the raw time series. We performed the experiment twice.
During the first time, we used a train and test set where some windows of a
person were in the train set and some were in the test set. The second time we
only used a train and test set where all the windows of a person were either
in the train or the test set (resulting in a "leave-subject-out" validation
procedure). The results for the first two experiments can be found in Figure
1.

(a) Random split

(b) Leave-subject-out

Figure 1: Accuracy when training an SVM using (a): a random split (windows of
the same subject, both in the train and test set) and (b): when windows of
subjects are either in the training or in the test set (leave-subject-out
validation procedure). This is inspected for different window and stride
sizes. For random splitting, the accuracy increases as the window size
increase and stride size decrease, whereas for the leave-subject-out
procedure, the accuracy seems to converge to one point. The achieved
accuracies are plotted on the y-axis and the stride sizes are on the x-axis.
The different window sizes are represented by different coloured lines. 120
(green), 100 (light blue), 80 (dark blue) and 50 (orange).

Both figures clearly illustrate that with every color-coded line, representing
different window sizes, the accuracy increases as the stride size decreases.
As the stride size decreases, there is an increase in the sample size.
However, even though these windows become more dependent (due to larger
overlap) with the reduction in stride size, the effectively larger sample size
still enhances performance. Moreover, as the window sizes get larger, the
accuracy also gets higher. However, there is a difference between the two
figures. For the "leave-subject-out" validation procedure, the accuracies seem
to converge to one point or at least stabilise, whereas in the random split
scenario, the lines show an overall increasing trend.

###  2.2 The effect of clustering subjects

Although the classification performance shown in the previous section is
reasonable, it is known that these data show large inter-person variability
due to physical differences between subjects [11]. To see if more personalised
models improve performance, we cluster the subjects based on various metrics.
The first metric: the average heart rate (in BPM) of every activity resulted
in five values per person. These five values represented a time series of five
points in the order as the activities performed: rest, breath, activity, rest,
and type. When we clustered these time series per person with different
resulting numbers of clusters, a cluster assignment as in Figure 2 was
achieved.

Figure 2: A cluster assignment with the number of clusters equal to 6 using a
time series of a subject’s mean BPM per activity using the TimeSeriesKmeans
clustering procedure[13]. Subplots from left to right represent the six
different clusters and the subjects included. Grey lines represent the
individual time series and thus represent a single subject. Red lines are the
averages of the time series in the cluster. The x-axis shows the different
activities numbered from 0 to 4 and the y-axis shows the heart rate in BPM.

To determine whether there were differences between the cluster groups, we
trained an SVM on one cluster while another cluster was used as a testing set.
The combinations and the corresponding scores achieved are represented in
Table 1.

Train x / Test y |  Averaged balanced accuracy
---|---
Train 1 / Test 2 |  .73
Train 1 / Test 5 |  .40
Train 1 / Test 6 |  .57
Train 5 / Test 6 |  .36
Train 5 / Test 3 |  .44
Table 1: Accuracies of training an SVM and using subjects of one cluster as
training set and subjects of another cluster as testing set. The numbers
indicate the clusters in Figure 2 counted from left to right. Similar clusters
achieve higher accuracy than more dissimilar ones.

We can observe in this table that the clusters that look similar (eg. cluster
1 and 2) achieve a cross-cluster better performance than clusters that look
more dissimilar (eg. 1 and 5). This suggests that there exists inter-subject
variability in this dataset.

To investigate the existence of variability within a cluster, we trained an
SVM on all the data in a cluster except for one subject, which was used for
testing. We performed this for every cluster and for every combination inside
a cluster. We considered two different standardization methods namely
‘Feature’ and ‘Data’ standardization. In Feature standardization, z-score
standardization is applied on the features after windowing and feature
generation. In Data standardization, z-score standardization is applied on the
original heart rate time per person, whereafter windowing and feature
generation is performed. The mean and standard deviation used for
standardising the training data are also used for the standardization of the
features in the testing data. The results of both methods can be found in
Figures 3(a) and 3(b).

(a) Feature standardization

(b) Data standardization

Figure 3: Results of accuracies within a cluster for the Feature and Data
standardization methods when training an SVM with the leave-one-subject out
validation procedure. Yellow and dark blue points represent the mean per
cluster and horizontal lines represent the performance of the SVM when no
clustering is performed. For yellow/orange points, balanced accuracy was used
and for light/dark blue, unbalanced/normal accuracy. With Feature
standardization (a), three of the four larger clusters have a mean accuracy
higher than an SVM trained on all the data. With Data standardization (b),
only two of the four larger clusters, have a mean accuracy higher than an SVM
trained on all the data.

First, these figures show us that the Feature standardization case is
performing better. Next, we see that in three out of four (larger) clusters,
the average accuracy within a cluster is higher than the SVM when no
clustering of subjects is done. Note that clusters 3 and 4 contain an
insufficient number of samples to provide an accurate representation.

Next, we conducted an additional experiment to investigate if the clustering
could be improved by using multiple features instead of only the mean heart
rate per activity. To test this, we evaluated the within-cluster accuracies
using different methods of clustering. The two different methods we
investigated were the use of temporal features and the use of statistical
features instead of mean heart rate. Statistical and temporal features are the
features generated by TSFEL[2]. The results can be seen in Figures 4(a) and
4(b).

(a) Temporal features

(b) Statistical features

Figure 4: Results of accuracies within a cluster for the Feature
standardization method using temporal and statistical features and training an
SVM with the leave-one-subject out validation procedure. Yellow and dark blue
points represent the mean per cluster and horizontal lines represent the
performance of the SVM when no clustering is performed. For yellow/orange
points, balanced accuracy was used and for light/dark blue, unbalanced/normal
accuracy. With temporal features (a), three of the four larger clusters have a
mean accuracy higher than an SVM trained on all the data. With statistical
features (b), all four clusters have a mean accuracy higher than an SVM
trained on all the data.

These figures show that the statistical features are better for clustering
than the temporal features. In all large clusters, it achieves better
performance than the SVM trained when no clustering is performed. In the
temporal case, this is only 3 out of 4 just like with the mean BPM clustering
method.

To demonstrate that it can also help with previously unseen samples, we
conducted several additional experiments using the leave-subject-out
procedure. We used the training set for generating the clustering model and
cluster assignment, as well as to train a model for each cluster. The test set
was used in two different ways. The first approach was per-window
classification. With this approach, a window of a test subject was obtained,
the corresponding cluster was determined, and the model associated with that
cluster was used to classify the window. The results of this approach can be
seen in the first column of Table 2.

|  Per-window |  Per-subject
---|---|---
6 clusters |  .46 |  .74
5 clusters |  .50 |  .63
4 clusters |  .56 |  .75
3 clusters |  .68 |  .72
Table 2: Achieved classification accuracies when using different numbers of
resulting clusters and clustering techniques to find a cluster model for
activity prediction. Grouping the subjects in 4 clusters and using the per-
subject method achieves the highest accuracy.

The second approach was to apply the same personalised classifier to all
windows of one test subject, the per-subject approach. To do so, we used the
clustering model to determine to which cluster each single window of a test
subject belongs to. After this, the cluster with the highest number of
assigned windows was used to obtain the model for classifying all windows of a
specific subject. The result of this experiment is presented in the second
column of Table 2. The per-subject method achieves higher accuracies than the
per-window classification method. It achieves an accuracy of 0.71 while the
per-subject method with 4 clusters achieves 0.75. In the next paragraph, we
delve more into the differences in prediction between both methods, rather
than solely examining the achieved accuracies.

The confusion matrices of the two methods can be found in Figure 5. We can see
a prominent difference within misclassifications occuring between the two
largest classes (Rest and Activity). The per-subject approach exhibits much
less misclassifications between these two classes compared to the per-window
method. Depending on the application, misclassifications between these very
different classes is more severe than misclassifications between similar
classes (Rest vs Breathe and RestAC). Overall, the per-subject model performs
better.

(a) Per-window

(b) Per-subject

Figure 5: Confusion matrices for the per-window and the per-subject approach.
The true/actual labels are shown on the vertical axis and the predicted labels
are on the horizontal axis. The biggest difference can be seen in the
predictions of the Rest and Activity class.

###  2.3 Deep learning with handcrafted features

Current research mostly focuses on deep-learning networks for feature
extraction and classification. Especially for heart rate variability analysis,
there exist some standard features for measuring the variability. Although
they are typically manually constructed, and therefore often interpretable,
one may wonder if these measures capture all information needed for health
diagnosis or activity recognition. In this section, the statistical and
temporal features used in earlier experiments are incorporated into a Deep
Learning method to investigate whether incuding these handcrafted (HC)
features can improve activity classification.

First of all, we compared the performance of the SVM models with the deep
learning models (convolutional neural networks) with and without HC features.
The results can be found in Figure 6 and illustrates that the addition of HC
features results in an increase in balanced accuracies in comparison to the DL
baseline model in certain instances. Additionally, the top four accuracies are
achieved without standardizing the HC features. Furthermore, every DL model
outperformed the SVM. Lastly, the second DL model which makes use of HC
features and a window size of 80 and stride of 10, performs the best compared
to all the models and configurations.

(a) DL model base and MFCC features vs SVM statistical and temporal features.

(b) DL vs SVM, both using statistical and temporal features.

Figure 6: Accuracies of different DL models with base HC features compared to
DL baseline and SVM. The blue line is the baseline DL network that gets as
input the raw standardized data. The red lines are the runs with an SVM and
the green lines are the runs with the proposed DL networks. It can be seen
that all the DL models outperform the SVM and most DL models with HC features
outperform the baseline. The best performing configuration was achieved using
model 2. Models 1, 2, and 3 represent the three different DL models, which are
explained in the method section 0.A.2. Models 1 and 3 make use of late
integration and model 2 of early integration of the DL and HC features. The
parameters column represents which feature set is used. The base feature set
represents the basic HC features like max, min, mean, std and means of
different (first and second-order) derivatives on the heart rate values in a
time series window. Base and MFCC [16] represent the feature set where there
are all the base parameters plus MFCC features. Statistical and temporal
features are the features generated by TSFEL [3]. The column standardized
indicates if the HC features are calculated on the standardized input or not.
The raw heart rate time series data is always standardized. When we talk about
standardized or non-standardized HC features in the next sections, we mean the
features calculated on a standardized or non-standardized input.

Next, we investigated the usage of temporal and statistical features as an
alternative to the base set of HC features. Each DL model was trained with
temporal or statistical features and with or without standardization. The
outcomes can be found in Figure 6(b). Among the top eight accuracies, five
configurations employed standardized input. While in the previous experiment
with the base features, the non-standardized HC features performed better. In
addition to this, we combined both the statistical and temporal features into
a single feature set, resulting in a slight improvement in performance to
58.84 % accuracy. Similarly, in this experiment, the standardized HC feature
set worked better than the non-standardized one.

Besides solely examining the accuracies, it is relevant to investigate whether
the HC features were indeed utilized by the DL model. To this extent, we used
SHAP values to see how important the HC features are in addition to the raw
input data [8]. Figure 7 depicts the top 20 SHAP values with the highest
importance. As we can see, the highest SHAP values correspond to an HC
feature: 0_Autocorrelation. Another observation is that primarily the heart
rate values in the middle or the end of a window input are important (using a
window size of 50).

Figure 7: Resulting top 20 SHAP values of 3rd DL model with the addition of
temporal + statistical features. On the x-axis the SHAP value is shown and on
the y-axis the feature. Features starting with ‘0_’ indicate HC features, and
numbers indicate the index of a heart rate value in the time series window
input.

###  2.4 Misclassification with DL models

To interpret the predictions of the DL model with HC statistical and temporal
features, a time series is plotted where the line colour indicates if the
prediction is correct or incorrect, and to what class it is misclassified. An
example is shown in Figure 8. Most of the misclassifications happen after a
change of class (for instance, at t=270, where the class changes from Breathe
to Activity). Intuitively this makes sense as the heart rate measurements do
not immediately change during activity change and therefore it is difficult to
predict the activity accurately.

Figure 8: Plot which visualizes the misclassification of the DL model. Black
lines correspond to rightly predicted classes and a coloured section indicates
a misprediction of a specific class. The colour corresponds to the class that
is mispredicted. The grey vertical bars correspond to the change in activity
according to the labeling of the data.

##  3 Discussion & Conclusion

We looked into the classification of activity using only heart rate time
series. Results show that there seems to be a relation between the optimal
window and stride size for classification: Higher window sizes and smaller
strides correspond to higher accuracies. In this case, larger window sizes
reveal distinct activity patterns that aren’t discernible at smaller sizes.
The smaller stride sizes ensure that the model is exposed to many time-shifted
variations of an activity pattern, during training. This ensures that the
model is better prepared to recognize any such variations that might be
present during testing. Because of the large variety in characteristics of
wearables and persons (inter-device and inter-subject variability), it is
challenging to build one single prediction model that works on everyone. We
have shown that in the context of activity classification in heart rate time
series, it is helpful to group similar subjects together and thus creating
semi-personalised models, for each separate group of alike persons. This
improvement is particularly evident when using multiple windows from a subject
to assign the person to a specific group. We believe that this is important to
take into consideration when investigating more challenging tasks like heart
disease classification. Furthermore, we have shown that feeding a deep
learning model with handcrafted features improves performance when classifying
heart rate time series. By adding features manually, the network learns to
find patterns in the time series itself but also uses some of the given HC
features to make a decision. This result supports the finding by Eltras et
al.[7], where they use a DL network for feature extraction and concatenate the
learned features with handcrafted features. Here, the best performance was
achieved by concatenating the raw features with handcrafted features before
inputting to the DL network.

Due to the limited number of subjects available, we did not have enough
subjects in some situations to do a proper train/test split within a cluster,
which was a common limitation for all the points mentioned above. This is
particularly true for the the semi-personalised models. For those that were
based on enough subjects, a positive increase in performance was shown.
Another limitation of our work is that the experiments have only been
conducted with one dataset. Finally, other normalization techniques can be
explored to even further reduce the variability among the subjects, for
example incorporating (meta)data like age or fitness.

##  Appendix 0.A Methods

###  0.A.1 BigIdeasLab_STEP

In this paper we used the BigIdeasLab_STEP dataset from PhysioNet [4]. This
dataset includes data from 53 participants and was recorded in July-August
2019. The age of the participants ranged from 18 to 54. Each person needed to
perform three study protocol rounds with different types of wearables. One
study protocol round consisted of five activities in the following order:

  1. 1.

Seated rest (4 min)

  2. 2.

Paced deep breathing (1 min)

  3. 3.

Physical activity (5 min)

  4. 4.

Seated rest ( 2 min)

  5. 5.

Typing (1 min)

In the experiment, every person wore all the available devices spread over
multiple rounds, capturing different amounts of samples. Round 1:Empatica E4
($N\approx 140K$), Apple Watch 4($N\approx 13K$). Round 2:Fitbit Charge
2($N\approx 11K$). Round 3: Garmin Vivosmart 3($N\approx 37K$), Xiaomi
Miband($N\approx 21K$) and Biovotion Everion($N\approx 161K$). During the
whole experiment, the participant always wore a Bittium Faros 180 ECG
device($N\approx 221K$) as a reference.

The dataset consists of a synchronised heart rate value in bpm between the
smartwatch and the ECG device. Moreover, it is annotated with one of the five
activities the person is performing. In the dataset, this is denoted by the
labels Rest, Breathe, Activity, Rest after Activity (RestAC) and Type. In the
experiments, only the heart rate data is used of the Apple Watch because of
its strong correlation with the heart rate time series of the ECG ground truth
in comparison with the other wearables.

###  0.A.2 Classification models

In several experiments, we used a support vector machine (SVM). An SVM tries
to maximize the margin between two classes. The SVM maximizes the
generalization of a model [5]. For multiclass classification one can use
multiple binary SVMs. Two of the methods used for this are One-against-all and
one-against-one [10]. For the experiments, we used the implementation provided
by scikit-learn, which uses the one-against-one method [12].

In addition to the other experiments, we researched the influence of the
addition of handcrafted (HC) features with deep learning models. To
investigate this, three different DL models with the addition of HC features
were used alongside a DL baseline. All of the DL models started with a 1-D
convolution and had three or four fully connected layers.

The baseline model starts with a 1-D convolution where the raw sequence input
will be processed. Next, it goes through a ReLU, Dropout and Max pooling layer
and finally, a flatten layer. After the flattening, it is processed by a fully
connected layer, followed by a ReLU and a last fully connected layer to bring
the output dimension to the required number of classes. A high-level graphical
overview can be seen in Figure 9a.

---
(a) baseline network

---
(b) First network

Figure 9: High-level overview of the internal working of the baseline DL
model(a) and the first DL model (b) that make use of HC features. Yellow
represents the raw input sequence and orange represents the HC features.

The first model is highly similar to the baseline model but it adds an extra
layer between the last two fully connected layers. So after the first fully
connected layer after flattening, the model adds the HC features to the output
of this layer. Next, it processes through another fully connected layer and
thereafter it goes through the last fully connected layer. This layer ensures
that it ends with the correct dimension. A simple graphical representation can
be found in Figure 9b.

The second model is integrating the HC features directly at the beginning of
the DL model. This is achieved by concatenating the HC features with the raw
time series input. This results in a larger input vector than with the
previous model. The third model is very similar to the first one but with one
addition. Instead of adding the HC features directly to the output of the
fully connected layer, the HC features first go through a fully connected
layer and this output is connected to the output of the first fully connected
layer of the model. A graphical representation of both models can be found in
Figure 10.

---
(a) Second network

---
(b) Third network

Figure 10: High-level overview of the internal working of the second(a) and
third(b) DL model that makes use of HC features. Yellow represents the raw
input sequence and orange represents the HC features.

## References

  * [1] Ballinger, B., Hsieh, J., Singh, A., Sohoni, N., Wang, J., Tison, G.H., Marcus, G.M., Sanchez, J.M., Maguire, C., Olgin, J.E.: Deepheart: semi-supervised sequence learning for cardiovascular risk prediction. In: Thirty-Second AAAI Conference on Artificial Intelligence
  * [2] Barandas, M., Folgado, D., Fernandes, L., Santos, S., Abreu, M., Bota, P., Liu, H., Schultz, T., Gamboa, H.: Tsfel: Time series feature extraction library. SoftwareX 11, 100456 (2020)
  * [3] Barandas, M., Folgado, D., Fernandes, L., Santos, S., Abreu, M., Bota, P., Liu, H., Schultz, T., Gamboa, H.: Tsfel: Time series feature extraction library. SoftwareX 11, 100456 (2020)
  * [4] Bent, B., Dunn, J.: “bigideaslab_step”: Heart rate measurements captured by smartwatches for differing skin tones" (version 1.0). https://doi.org/10.13026/cqfy-d860
  * [5] Cervantes, J., Garcia-Lamont, F., Rodríguez-Mazahua, L., Lopez, A.: A comprehensive survey on support vector machine classification: Applications, challenges and trends. Neurocomputing 408, 189–215 (2020)
  * [6] Dahalan, A.J., Razak, T.R., Ismail, M.H., Fauzi, S.S.M., Gining, R.A.J.: Heart rate events classification via explainable fuzzy logic systems. IAES International Journal of Artificial Intelligence 10(4), 1036 (2021)
  * [7] Eltrass, A.S., Tayel, M.B., Ammar, A.I.: Automated ecg multi-class classification system based on combining deep learning features with hrv and ecg measures. Neural Computing and Applications 34(11), 8755–8775 (2022)
  * [8] Lundberg, S.M., Lee, S.I.: A unified approach to interpreting model predictions. Advances in neural information processing systems 30 (2017)
  * [9] Maguire, D., Frisby, R.: Comparison of feature classification algorithm for activity recognition based on accelerometer and heart rate data. In: 9th. IT & T Conference. p. 11
  * [10] Mayoraz, E., Alpaydin, E.: Support vector machines for multi-class classification. In: Engineering Applications of Bio-Inspired Artificial Neural Networks: International Work-Conference on Artificial and Natural Neural Networks, IWANN’99 Alicante, Spain, June 2–4, 1999 Proceedings, Volume II. pp. 833–842. Springer (2006)
  * [11] Niu, J., Tang, Y., Sun, Z., Zhang, W.: Inter-patient ecg classification with symbolic representations and multi-perspective convolutional neural networks. IEEE journal of biomedical and health informatics 24(5), 1321–1332 (2019)
  * [12] Pedregosa, F., Varoquaux, G., Gramfort, A., Michel, V., Thirion, B., Grisel, O., Blondel, M., Prettenhofer, P., Weiss, R., Dubourg, V., Vanderplas, J., Passos, A., Cournapeau, D., Brucher, M., Perrot, M., Duchesnay, E.: Scikit-learn: Machine learning in Python. Journal of Machine Learning Research 12, 2825–2830 (2011)
  * [13] Tavenard, R., Faouzi, J., Vandewiele, G., Divo, F., Androz, G., Holtz, C., Payne, M., Yurchak, R., Rußwurm, M., Kolar, K., Woods, E.: Tslearn, a machine learning toolkit for time series data. Journal of Machine Learning Research 21(118), 1–6 (2020), http://jmlr.org/papers/v21/20-091.html
  * [14] Torres-Soto, J., Ashley, E.A.: Multi-task deep learning for cardiac rhythm detection in wearable devices. NPJ digital medicine 3(1), 1–8 (2020)
  * [15] Walch, O., Huang, Y., Forger, D., Goldstein, C.: Sleep stage prediction with raw acceleration and photoplethysmography heart rate data derived from a consumer wearable device. Sleep 42(12), zsz180 (2019)
  * [16] Xu, M., Duan, L.Y., Cai, J., Chia, L.T., Xu, C., Tian, Q.: Hmm-based audio keyword generation. In: Advances in Multimedia Information Processing-PCM 2004: 5th Pacific Rim Conference on Multimedia, Tokyo, Japan, November 30-December 3, 2004. Proceedings, Part III 5. pp. 566–574. Springer