# Symphony Generation with Permutation Invariant Language Model

###### Abstract

In this work, we propose a permutation invariant language model, SymphonyNet,
as a solution for symbolic symphony music generation. We propose a novel
Multi-track Multi-instrument Repeatable (MMR) representation for symphonic
music and model the music sequence using a Transformer-based auto-regressive
language model with specific 3-D positional embedding. To overcome length
overflow when modeling extra-long symphony tokens, we also propose a modified
Byte Pair Encoding algorithm (Music BPE) for music tokens and introduce a
novel linear transformer decoder architecture as a backbone. Meanwhile, we
train the decoder to learn automatic orchestration as a joint task by masking
instrument information from the input. We also introduce a large-scale
symbolic symphony dataset for the advance of symphony generation research.
Empirical results show that the proposed approach can generate coherent,
novel, complex and harmonious symphony as a pioneer solution for multi-track
multi-instrument symbolic music generation.

##  1 Introduction

Symphony is one of the most complex and brilliant musical composition forms in
human history, where many instruments are intertwined to express rich human
emotions. The past decade has seen the rapid development and tremendous
success of the symbolic music generation in both research and industrial field
[1, 2, 3]. Most current works follow conventional text modeling and generation
method by applying language model to sequences of symbolic musical events [4,
5, 6]. However, symphony modeling and generation still constitutes in itself a
considerable challenge since symphony music sequences differ from text
sequences in various aspects.

Natural language could be modeled as a purely linear sequence constructed
strictly by a sequential order of words. Symphony scores, on the other hand,
are usually viewed as two-dimensional symbolic sequences in which many notes
can be played concurrently. Notes in a symphony score are semi-permutation
invariant. More specifically, as shown in Fig. 1, the blue box indicates the
musical instrument tracks, and the corresponding staves on the right side are
permutation invariant. Similarly, the notes inside the red box are also
permutation invariant. In contrast, notes in the upper yellow box are
permutation variant since each note is played sequentially. Changes in the
order of notes will impair the music itself. The yellow box at the bottom is a
more complicated situation: a permutation variant note sequence in general
containing permutation invariant notes. Simply flattening the score into a 1-D
text-like sequence may damage the local structure of music [7]. To address
this problem, we propose the Multi-track Multi-instrument Repeatable (MMR)
representation with particular 3-D positional embedding in Section 3 which
fully considers the properties of semi-permutation invariance in symbolic
music scores.

Moreover, when comparing music scores with text, conventionally notes could be
considered as characters, while intervals or chords are comparable to words.
Modeling musical events at note level is a common practice [8, 5, 6, 9].
However, this may be confronted with similar problems in char-level text
generation, such as extremely long sequences and less meaningful individual
tokens. Word-level tokenization suffers from large vocabulary size and out of
vocabulary (OOV) problems. Byte Pair Encoding (BPE) [10, 11] subword
tokenization is a tradeoff between word-level and character-level
tokenization. Inspired by BPE, we propose the Music BPE algorithm in Section
4, which could automatically aggregate notes to intervals and chords as
subwords without a pre-defined vocabulary and construct music sequences with
richer semantics.

Figure 1: A simple example of Multi Instruments & Multi Tracks & Repeat
Instruments symphony score.

Generating symphony music with proper instruments for different tracks is
another challenging task. Recent work like Arranger [12] focuses on
instrumentation by learning to separate parts from the mixture in symbolic
multi-track music. However, it does not incorporate music generation task. In
this paper, we present a unique linear transformer decoder architecture for
instrument classification with joint-task training, which allows the model to
learn auto-orchestration rather than relying on instrument information as an
pre-defined input source. [13, 5, 14, 15].

The contributions of this paper are presented as below:

  * •

We propose a novel Multi-track Multi-instrument Repeatable (MMR)
representation for symphony music, including particular 3-D positional
embedding designed to address the semi-permutation invariant challenge in
symphony generation. Our method is also compatible with all existing symbolic
music ensembles, including but not limited to piano solo, quartet and pop band
music.

  * •

We propose a novel algorithm, Music BPE, to model the symbolic music at
subword-level. Furthermore, we found that our Music BPE algorithm could
aggregate notes to intervals and chords, which are consistent with common
chords summarized by human musicians.

  * •

We introduce SymphonyNet, a novel music generation model with joint-task
training for instrument classification based on our proposed MMR
representation and Music BPE. The model can learn the proper orchestration
according to the distribution of the notes.

  * •

We collect a symphony MIDI dataset, consisting of $46,359$ high-quality MIDI
files with multiple instruments and tracks to advance researches on symphony
generation with deep learning.

##  2 Related Work

Name | Time Unit | Representation | Backbone Model | Type of Music | Instruments
---|---|---|---|---|---
DeepBach [1] | Fixed-length grid | N/A | Bi-directional RNN | Chorales | Fixed Ensemble
MuseGAN [16] | Fixed-length grid | N/A | GAN | Multi-track | Fixed Ensemble
Music Transfor. [4] | MIDI-event timeshift | N/A | Vanilla Transformer | Piano | N/A
Pop MT [8] | Beat and note duration | REMI | Transformer-XL | Piano | N/A
CWT [6] | Beat and note duration | Compound Word | Linear Transformer | Piano | N/A
Musenet [13] | MIDI-event timeshift | Token Aggregation | GPT-2 | Multi-track | Not Repeatable
PopMAG [5] | Beat and note duration | MuMIDI | Transformer-XL | Multi-track | Not Repeatable
LakhNES [14] | MIDI-event timeshift | Token Aggregation | Transformer-XL | Multi-track | Fixed Ensemble
MMM [15] | MIDI-event timeshift | Hierarchical | GPT-2 | Multi-track | Repeatable
This work | Beat and note duration | MMR | Linear Transformer | Multi-track | Repeatable
PiRhDy [17] | Fixed-length grid | Fusion module | RNN with attention | Multi-track | Not Repeatable
MusicBert [9] | Beat and note duration | OctupleMIDI | Roberta | Multi-track | Not Repeatable
Table 1: An overview of time unit, representation, backbone model and music
type in existing works, above for generation works and below for understanding
works.

We organize some existing works in Table 1 in terms of five aspects of
symbolic music modeling: time unit, representation method, backbone model,
music type and the ability to model music with repeat instruments. Generation
works are presented above and understanding works are presented below.
Pianoroll, MIDI event timeshift, and Beat-based onset and duration are the
mainstream time units in music generation and understanding tasks. However,
Pianoroll divides music into fixed-length grids, and MIDI format provides
overprecise timeshift events, both suffering from sparsity problems, which
raises another handicap for applying deep learning models in this multi-track
generation. Pop Music Transformer [8] is the first attempt to introduced the
beat-based REMI representation in music generation. It supports variable-
length duration of notes, which is more musically inspired. Compound Word [6],
derived from REMI representation, classifies the sequence of REMI into note-
related or metric-related events, which are then aggregated, greatly
decreasing the sequence length.. This has engendered a new trend of beat-based
symbolic music generation.

Language models are now prevalent in natural language processing tasks [18].
However, applying language models to the creation of multi-track music remains
challenging. MuMIDI [5] and OctupleMIDI [9] models multiple attributes of one
note in one sequence step and also incorporates instrument tokens for multi-
track representation. However, if one musical piece contains more than one
track for the same instrument, their representation could not distinguish them
in different tracks. MMM [15] introduced a MIDI-event-like representation,
creating a time-ordered sequence of musical events for each track and
concatenating several tracks into a single sequence. However, MMM adopts time-
delta tokens and fixed positional encoding which weakens the note-level
correlation and structure between tracks. MuseBert [7] proposes a permutation
invariant bert-like language model with generalized relative position encoding
(RPE) which, however, is not compatible with multi-track music generation.

Though various symbolic music representation strategies have been proposed,
few are compatible with multi-track music with repeatable instruments or
tracks, such as the symphony. Furthermore, permutation invariance of music, as
is discussed in Section 1, has scarcely been considered. To our knowledge,
this work proposes the first representation and tokenization method to encode
music with multiple repeatable instruments and multiple repeatable tracks and
designs a universal and effective strategy for generating symphony music with
permutation invariant language model.

##  3 Multi-track Multi-instrument Repeatable Representation

To further analyze the symphony generation task, it is crucial to understand
the difference between the symphony format and other genres of music.

  * •

Single Instrument in Single Track. No more than one note is played at any
timestep by one instrument. Also called monophonic music. e.g., flute.

  * •

Multi Instruments & Each in Single Track. Only one note for each instrument is
played at any timestep. e.g., quartet singing.

  * •

Single Instrument in Multi Tracks. There are multiple notes played in each
timestep while only one instrument. e.g., piano.

  * •

Multi Instruments & Multi Tracks & No Repeat Instrument. There are multiple
notes played in each timestep. No constraint on the number of instruments and
all instruments are unique. e.g., classical pop band with only drum, electric
guitar and bass.

  * •

Multi Instruments & Multi Tracks & Repeat Instruments. Instruments are not
unique and multiple same instruments can play different notes on different
tracks, e.g., symphony.

For the last case, it’s a common practice to merge the same instruments into a
single track in previous works. However, it may damage the intrinsic structure
of symphony music. For example, this may cause a violin to play polyphonic
notes, or even intermingle multiple melody lines. Our proposed Multi-track
Multi-instrument Repeatable (MMR) representation models repeated instruments
separately, which could capture more heuristic musical information within a
single track. Since our MMR representation is aimed at symphony modeling, it
is also compatible with all existing music ensembles.

Figure 2: An example of MMR representation and illustration of Music BPE
process

###  3.1 Structural and Controlling Token

We consider that special tokens perform two primary functions in a symphony
music generation task: 1) To represent the musical structures of notes. 2) To
control the model output during the inference phase.

Score We use a pair of $[BOS]$ and $[EOS]$ tokens to designate the beginning
and end of a symphony score.

Measure Different from [5, 8], we ascribe a pair of $[BOM_{i}]$ and $[EOM]$ to
indicate the beginning and end of a measure, the character $i$ to represent
the total length of the current measure. The length of a measure is calculated
by time signature, and we choose 32th note as the smallest unit of time. For
example, a 4/4 time signature indicates four quarter notes length per measure,
which is equal to the length of thirty-two 32th notes. In that case, character
$i$ equals 32, and the measure beginning token is $[BOM_{32}]$

Chord The chord token is a valuable indicator of how generally notes are
arranged in the current measure. We pre-define $132$ common types of chord
token and pre-compute chord tokens with a rule-based algorithm proposed in
[6], such as C major seventh chord marked as token $[C_{maj7}]$.

Track Unlike any previous works, we do not explicitly encode the track and
instrument transformation in a single token. A change track token $[CC]$ only
signifies the start of a new track for the latter controlling purpose. Section
5 will further explore the traits of tracks and instruments and the approaches
of differentiating tracks.

Position A position token stands for the _onset_ of a note within the measure,
represented by the token $[POS_{j}]$. The following event tokens are
controlled by the current position token until another position token shows
up. The character $j$ means the number of the current unit of time position.
For example, a $[POS_{48}]$ indicates the 48th unit time position.

To summarize, structural and controlling tokens are designed to specify the
general time-spatial features of notes, such as the time a note is to be
played and the track it locates. In this work, these tokens are mandated with
a sequential order, as a _measure_ token shall be followed by a _chord_ token,
which altogether represents in a explicit way the measure order as shown in
Fig. 3.

###  3.2 Note-Related Tokens

A note in music scores could be defined in five attributes: pitch, duration,
onset, track and instrument. Pitch and duration are content-related and the
others are position-related. The latter will be discussed in Section 5. To
avoid the long-tail problem, we regard pitch and duration to be distinct note
properties and construct two separate vocabularies for model input. Then we
aggregate note pitches with identical duration and onset by our proposed Music
BPE algorithm, as will be described in the next section.

##  4 Music Byte Pair Encoding

As shown in Fig. 2, a complex chord is constructed by several notes at the
same position in a measure, which can be deconstructed into two common and
simple intervals. Unlike natural language, notes played at the same position
are permutation invariant. Changing the order of notes in a chord does not
affect its sound or meaning. For instance, a Chord $C$ consists of ($C4$,
$E4$, $G4$), which is equal to ($G4$, $C4$, $E4$). This intrinsic property may
conflict with the typical natural language processing job, imposing a new
constraint on the use of conventional tokenization methods such as standard
BPE.

In this work, we propose a novel encoding approach, Music Byte Pair Encoding
(Music BPE), for multi-track symbolic music sequence tokenization and
preprocessing to exploit the semantics of music events and minimise the length
of the input context from a representation standpoint. Different from the
original BPE algorithm, our proposed Music BPE is based on concurrence of
notes rather than adjacency of characters.

Our implementation of Music BPE is shown in Algorithm 1. As is mentioned in
Section 1, a note has five attributes: _pi_ tch, _du_ ration, _po_ sition,
_tr_ ack and instrument, while the instrument depends utterly on track within
the same measure. Formally, in a piece of symbolic music, we define a maximum
set of two or more notes

$\\{(pi,du,po,tr)\mid\text{where }du,po,tr\text{ is constant}\\}$

as a _mulpi_ (multiple pitches), i.e., a maximum set of notes that have the
same duration at the same global position and within the same track,
equivalent to a "word" in the BPE algorithm.

Algorithm 1 Music BPE

Input: A multi-set of _mulpies_ $B$
Parameter: desired dictionary size $n$
Output: Merged dictionary $V$

1: Let $V=\\{\\{p\\}\mid p\in[0,128)\\}$.

2: Let $C$ be an empty multi-set

3: for all $mulpi\in B$ do

4: $mulpi\leftarrow\\{\\{p\\}\mid p\in mulpi\\}$

5: for all $\\{P_{1},P_{2}\\}\subseteq mulpi$ do

6: Insert $(P_{1},P_{2})$ into $C$.

7: end for

8: end for

9: while $|V|<n$ do

10: Let $(P_{1},P_{2})$ be the most frequent pair in $C$.

11: $V\leftarrow V\cup\\{P_{1}\cup P_{2}\\}$

12: for all $mulpi\in B$ do

13: if $\\{P_{1},P_{2}\\}\subseteq mulpi$ then

14: for all $Q\in mulpi-\\{P_{1},P_{2}\\}$ do

15: Delete $(Q,P_{1}),(Q,P_{2})$ from $C$.

16: Insert $(Q,P_{1}\cup P_{2})$ into $C$.

17: end for

18: $mulpi\leftarrow(mulpi-\\{P_{1},P_{2}\\})\cup\\{P_{1}\cup P_{2}\\}$

19: end if

20: end for

21: end while

22: return $V$

We collect notes with the same global position and the same duration in the
same track from each music piece to construct a bag of _mulpies_. The
vocabulary list is initialized with $128$ MIDI pitches, where each token
represents a pitch-set containing a single pitch. Every time we locate all
concurrent pairs of tokens in the bag of _mulpies_ , merge the most frequent
pair (’$P_{1}$’, ’$P_{2}$’) into a new symbol $P$ and replace the pair with
the new symbol in each _mulpi_ until the vocabulary size reaches the maximum
limit. A further discussion on the results of the Music BPE algorithm and its
effectiveness on our symphony dataset will be presented in Section 5.

##  5 SymphonyNet Details

###  5.1 The 3-D Positional Embedding

Figure 3: A illustration of the spatial and structural attributes of MMR
sequence (left) and the way it is compressed and organized as model input
(right).

Transformer [19] is the most used backbone for language model, which is
designed _permutation invariant_ : if the positional encoding is not added,
disrupting the order of the inputs will yield the same output, for transformer
model treats inputs as a _set_ during self-attention. Therefore, considering
this property of Transformer, we design a particular 3-D positional embedding
to represent such a semi-permutation invariant feature as shown in Fig. 3.
Event tokens follow a semi-permutation variant order on both the measure order
and the note order axes. For example, notes played on the same position share
the same note positional embedding. In contrast, the track axis is entirely
permutation invariant since we only need track embeddings to differentiate
tracks other than a sequential order. We use the red curves to illustrate the
musical moving trajectory of event tokens to better understand how we compress
the spatial and structural sequence of the event tokens into one dimension and
send them to the model. At last, we add all constructed embeddings vertically
as the model input. To address the extraordinarily long symbolic music
sequences challenge, we employ the linear transformer [20] as the backbone of
our model to satisfy the length constraint. The model follows a decoder-only
fashion, and we design different feed-forward heads for four attributes of
musical events, which are Instrument, Track, Duration, and Event tokens as
shown in Fig. 3.

###  5.2 Joint Task with Instrument Classification

We mask instrument information for every input token at the input side, and
anticipate that the model will learn instrumentation from the output side with
instrument loss. This will turn a succession of simple, blank notes into a
fully orchestrated piece of music, analogous to colouring a black-and-white
painting. This design is motivated by two primary concerns. First, we
investigate the possibility if other instrument may play a certain
instrument’s note track. Therefore, that is a case for the model to determine
to what degree the instrument fits the track’s notes and how instruments
interact with one another across tracks. For instance, it is allowed to
substitute the piano for the marimba in some musical compositions. The
intrinsic nature of a pre-assigned instrument for notes reduces the diversity
of training data.

##  6 Experiments and Results

This section introduces the novel symphony dataset we propose and presents two
stages in the training process111Our code, demos, dataset and further analysis
can be accessed at https://symphonynet.github.io. Secondly, we describe
controllable methods to generate music under certain condition before we
provide findings from Music BPE and compare them with the specific musical
knowledge. Lastly, a human evaluation result analysis and scoring on several
excerpts generated by different models will be presented.

###  6.1 Symphony Dataset

To tackle the obstacles of the symphony generation research, we gather a big
corpus of symphonic music from multiple online sites and conduct a extensive
data cleaning. The average duration of the 46,359 MIDI files containing
multiple instruments and tracks, mostly symphony, is 4.26 minutes. The
collection contains more than 279 million notes and 567 million tokens in MMR
forms. Our symphony dataset is, to the best of our knowledge, the first
worldwide large-scale symbolic symphonic music dataset, which might serve as a
foundation for future work in multi-track music production.

Figure 4: The training and validation curves of different models and the Music
BPE note aggregation results.

###  6.2 Training Details

In our experiment, the model adopts $4096$ as the length of input sequence. We
set the embedding size for event tokens, durations, instruments and 3-D
positional embedding to $512$. The final size of event token vocabulary is
assigned to $1000$ after running Music BPE algorithm and the vocabulary size
of durations, instruments, 3-D positional embeddings are derived from the
dataset. The linear transformer decoder contains $12$ self-attention layers
and each layer consists of 16 attention heads. SymphonyNet is trained with
eight 2080 Ti GPU and we use a batch size of $128$ and an AdamW [21] optimizer
with a learning rate of $3\times 10^{-4}$.

###  6.3 Music BPE Result

After constructing a vocabulary list of length $1,000$ with Music BPE
algorithm, the top-5 merged pairs shown in Fig. 4 with the highest frequency
are: ($D4$, $F4$), ($C4$, $E4$), ($E4$, $G4$), ($D4$, $D5$), and ($G4$, $G5$),
which are usual intervals occurring in symphony _divisi_ passages. After
applying Music BPE on the whole music corpus, the average token length of a
$mulpi$ shortens to half (from $2.28$ to $1.13$), also reducing the burden of
modeling ultra-long symphony sequences.

###  6.4 Ablation Study and Human Evaluations

We train a linear transformer decoder model with the vanilla positional
encoding of GPT-3 [18] as a baseline. Then we train another model of the same
architecture with the training data processed by the proposed Music BPE
algorithm. Finally, we incorporate both 3-D positional embedding and Music BPE
algorithm, which achieves the lowest training and validation loss after the
same total training steps, as is shown in Fig. 4. The objective metric
indicates that our permutation-invariant 3-D positional embedding and Music
BPE algorithm could significantly improve model performance and generalization
ability.

Also, we perform a human evaluation to compare the quality of generated music
excerpts from different models with human composition. Participants include 25
professional musicians and 25 non-musicians. Each participant receives 16
excerpts: four excerpts conditioned on a chord progression, four excerpts
conditioned on a given 4-bar prime, and eight unconditioned excerpts. The
music excerpts are rated in 5 dimensions: Coherence (C), Diversity (D),
Harmoniousness (H), Structureness (S), Orchestration (O) and Overall
Preference (P), in a 5-point Likert scale. As shown in Table 2(a), the model
with 3-D positional embedding and Music BPE beats most of the approaches. It
is worth noting that excerpts generated by our models surpass the human
compositions in the indicator of diversity marked by yellow color.

To further explore the model performance, we retrain SympohyNet on Lakh MIDI
Dataset [22] with the same backbone model architecture as MMM [15], and carry
out another human evaluation to compare with MMM. Each participant receives 10
excerpts: five generated unconditionally from MMM and the others generated
unconditionally from retrained SymphonyNet. The results are presented in Table
2(b), which indicate that SymphonyNet surpass MMM in all indicators. Overall,
the human hearing test suggests that SymphonyNet can construct coherent,
unique, complex, and harmonic symphonies.

| Model | C | D | H | S | O | P
---|---|---|---|---|---|---|---
Chord | Baseline | 3.5 | 3.57 | 3.07 | 3.00 | 3.21 | 3.29
BPE | 3.64 | 3.64 | 3.14 | 3.15 | 3.43 | 3.29
3D + BPE | 3.71 | 3.72 | 3.21 | 3.07 | 3.5 | 3.5
Human | 4.43 | 3.43 | 4.14 | 4.36 | 4.14 | 4.14
|  |  |  |  |  |  |
Prime | Baseline | 3.79 | 2.79 | 3.21 | 3.43 | 3.36 | 3.36
BPE | 3.86 | 3.5 | 3.5 | 3.5 | 3.64 | 3.86
3D + BPE | 3.86 | 3.14 | 3.43 | 3.57 | 3.93 | 3.64
Human | 4.36 | 3.57 | 4.36 | 4.00 | 4.36 | 4.36
|  |  |  |  |  |  |
Uncondi. | Baseline | 3.52 | 3.46 | 3.04 | 3.07 | 3.11 | 3.07
BPE | 3.79 | 3.64 | 3.25 | 3.11 | 3.25 | 3.29
3D + BPE | 3.53 | 3.93 | 3.43 | 3.32 | 3.43 | 3.32
Human | 4.39 | 3.89 | 4.18 | 4.21 | 4.11 | 4.29
(a) Trained on Symphony Dataset

Model | C | D | H | S | O | P
---|---|---|---|---|---|---
MMM | 3.20 | 2.71 | 2.51 | 2.66 | 2.80 | 2.71
±0.13 | ±0.12 | ±0.13 | ±0.12 | ±0.13 | ±0.11
Symph. | 3.33 | 2.89 | 2.76 | 2.69 | 2.99 | 2.87
±0.15 | ±0.13 | ±0.12 | ±0.13 | ±0.12 | ±0.13
(b) Trained on Lakh MIDI Dataset

Table 2: Human evaluation results from 25 musicians and 25 non-musicians, with
mean opinion scores and 95 percent confidence intervals reported.

##  7 Conclusion

In this work, we illustrate the properties of multi-track and multi-instrument
music, like symphony, and propose a novel MMR representation with 3-D
positional embedding for modelling it. To tokenize the ultra-long symbolic
music sequence at sub-word level, we propose the Music BPE algorithm. Besides,
we design a joint task for the model to learn auto-orchestration. Human
evaluation results show that our suggested technique produces competitive
symphonic music when compared to human compositions. In the future, we will
investigate modelling long-term musical structures, since complex music, such
as symphonies, often consists of numerous parts or movements.

##  8 Acknowledgements

Thanks for the anonymous reviewers for their valuable comments. This work is
supported by High-grade, Precision and Advanced Discipline Construction
Project of Beijing Universities, Major Projects of National Social Science
Fund of China (Grant No. 21ZD19), and Nation Culture and Tourism Technological
Innovation Engineering Project.

## References

  * [1] G. Hadjeres, F. Pachet, and F. Nielsen, “DeepBach: a steerable model for bach chorales generation,” in _International Conference on Machine Learning_. PMLR, 2017, pp. 1362–1371.
  * [2] L. Yang, S. Chou, and Y. Yang, “MidiNet: A convolutional generative adversarial network for symbolic-domain music generation,” in _Proceedings of the 18th International Society for Music Information Retrieval Conference, ISMIR 2017, Suzhou, China, October 23-27, 2017_ , 2017, pp. 324–331.
  * [3] A. Roberts, J. Engel, C. Raffel, C. Hawthorne, and D. Eck, “A hierarchical latent vector model for learning long-term structure in music,” in _International conference on machine learning_. PMLR, 2018, pp. 4364–4373.
  * [4] C.-Z. A. Huang, A. Vaswani, J. Uszkoreit, I. Simon, C. Hawthorne, N. Shazeer, A. M. Dai, M. D. Hoffman, M. Dinculescu, and D. Eck, “Music transformer: Generating music with long-term structure,” in _International Conference on Learning Representations_ , 2018.
  * [5] Y. Ren, J. He, X. Tan, T. Qin, Z. Zhao, and T.-Y. Liu, “Popmag: Pop music accompaniment generation,” in _Proceedings of the 28th ACM International Conference on Multimedia_ , 2020, pp. 1198–1206.
  * [6] W.-Y. Hsiao, J.-Y. Liu, Y.-C. Yeh, and Y.-H. Yang, “Compound word transformer: Learning to compose full-song music over dynamic directed hypergraphs,” in _Proceedings of the AAAI Conference on Artificial Intelligence_ , 2021, pp. 178–186.
  * [7] Z. Wang and G. Xia, “MuseBERT: Pre-training music representation for music understanding and controllable generation,” in _Proceedings of the 22nd International Society for Music Information Retrieval Conference, ISMIR 2021, Online, November 7-12, 2021_ , 2021.
  * [8] Y.-S. Huang and Y.-H. Yang, “Pop music transformer: Beat-based modeling and generation of expressive pop piano compositions,” in _Proceedings of the 28th ACM International Conference on Multimedia_ , 2020, pp. 1180–1188.
  * [9] M. Zeng, X. Tan, R. Wang, Z. Ju, T. Qin, and T. Liu, “Musicbert: Symbolic music understanding with large-scale pre-training,” in _Findings of the Association for Computational Linguistics: ACL/IJCNLP 2021, Online Event, August 1-6, 2021_ , ser. Findings of ACL, vol. ACL/IJCNLP 2021. Association for Computational Linguistics, 2021, pp. 791–800. [Online]. Available: https://doi.org/10.18653/v1/2021.findings-acl.70
  * [10] R. Sennrich, B. Haddow, and A. Birch, “Neural machine translation of rare words with subword units,” in _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , 2016, pp. 1715–1725.
  * [11] P. Gage, “A new algorithm for data compression,” _The C Users Journal_ , vol. 12, no. 2, pp. 23–38, 1994.
  * [12] H. Dong, C. Donahue, T. Berg-Kirkpatrick, and J. J. McAuley, “Towards automatic instrumentation by learning to separate parts in symbolic multitrack music,” in _Proceedings of the 22nd International Society for Music Information Retrieval Conference, ISMIR 2021, Online, November 7-12, 2021_ , 2021.
  * [13] C. Payne, “MuseNet,” _OpenAI Blog_ , vol. 3, 2019.
  * [14] C. Donahue, H. H. Mao, Y. E. Li, G. W. Cottrell, and J. J. McAuley, “Lakhnes: Improving multi-instrumental music generation with cross-domain pre-training,” in _Proceedings of the 20th International Society for Music Information Retrieval Conference, ISMIR 2019, Delft, The Netherlands, November 4-8, 2019_ , 2019.
  * [15] J. Ens and P. Pasquier, “Mmm: Exploring conditional multi-track music generation with the transformer,” _arXiv preprint arXiv:2008.06048_ , 2020\.
  * [16] H.-W. Dong, W.-Y. Hsiao, L.-C. Yang, and Y.-H. Yang, “Musegan: Multi-track sequential generative adversarial networks for symbolic music generation and accompaniment,” in _Thirty-second aaai conference on artificial intelligence_ , 2018.
  * [17] H. Liang, W. Lei, P. Y. Chan, Z. Yang, M. Sun, and T.-S. Chua, “Pirhdy: Learning pitch-, rhythm-, and dynamics-aware embeddings for symbolic music,” in _Proceedings of the 28th ACM International Conference on Multimedia_ , 2020, pp. 574–582.
  * [18] T. B. Brown, B. Mann, N. Ryder, M. Subbiah, J. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell _et al._ , “Language models are few-shot learners,” _arXiv preprint arXiv:2005.14165_ , 2020.
  * [19] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, “Attention is all you need,” in _Advances in neural information processing systems_ , 2017, pp. 5998–6008.
  * [20] A. Katharopoulos, A. Vyas, N. Pappas, and F. Fleuret, “Transformers are rnns: Fast autoregressive transformers with linear attention,” in _International Conference on Machine Learning_. PMLR, 2020, pp. 5156–5165.
  * [21] I. Loshchilov and F. Hutter, “Decoupled weight decay regularization,” in _International Conference on Learning Representations_ , 2018.
  * [22] C. Raffel, “Learning-based methods for comparing sequences, with applications to audio-to-midi alignment and matching,” Ph.D. dissertation, COLUMBIA UNIVERSITY, 2016.


# Towards In-Cabin Monitoring: A Preliminary Study on Sensors Data Collection
and Analysis††thanks: This work has been carried out in cooperation with Sleep
Advice Technologies S.r.l. (https://www.satechnology.eu/) and it has been
financially supported by the UE project A-IQ READY that receives funding
within the Key Digital Technologies Joint Undertaking (KDT JU) - the Public-
Private Partnership for research, development and innovation under Horizon
Europe – and National Authorities under grant agreement n. 101096658.

Jacopo Sini, Luigi Pugliese, Sara Groppo, Michele Guagnano, Massimo Violante
Politecnico di Torino
Turin, Italy
{jacopo.sini, luigi.pugliese, sara.groppo, michele.guagnano,
<EMAIL_ADDRESS>
###### Abstract

The last decade’s market has been characterized by wearable devices, mainly
smartwatches, edge, and cloud computing. A possible application of these
technologies is to improve the safety of dangerous activities, especially
driving motor vehicles. Common enabling technologies, such as system-on-chip,
ultra-low-power computational platforms, and wide-band wireless connectivity,
push all these trends. On the one hand, wearable devices, thanks to the
continuous contact with the user’s body, can measure physiological parameters.
On the other hand, edge computing and machine learning techniques, alongside
cameras, allow the implementation of contactless computer vision systems
capable of providing information about the user’s current behavior. Another
trend is the usage of RADARs in automotive applications, both for collision
avoidance and monitoring driver behavior. These technologies can be combined
to develop systems designed to aid the driver. For the sake of this paper, we
are focusing on warning drivers, allowing them to know whenever they are
drowsy and hence risking a sleep onset or are not paying attention to the
road. Developing such systems poses many challenges, such as automatic
classification of physiological signal patterns, facial expression
recognition, head movements and eye gaze detection. These challenges have been
individually addressed in the literature. Anyway, we noticed a need for more
description on implementing data fusion. Two main reasons for adopting the
fusion approach are to improve the quality of the overall representation
(increasing accuracy and specificity against drowsy) and make a more reliable
system due to redundancy.

###### Index Terms:

Physiological data, Sleep, Safety, Vehicle Driving, Camera-based systems,
Image processing, Neural Networks

##  I Introduction

As reported by American National Highway Traffic Safety Administration
(NHTSA), driver loss of attention is a worldwide problem that leads to a high
number of deaths every year. The leading cause of the decrease in attention
while driving is drowsiness. Particularly, sleep at the wheel contributes to
motor vehicle accidents for 20% of all police-reported crashes[1]. Several
methods for monitoring the driver’s state have been developed, mainly focusing
on the way to drive [2], the camera-based measurements[3], and the
physiological parameters of the driver [4]. Detecting methods based on the way
to drive have low reliability because obtained results can change a lot due to
unpredictable factors such as road geometry and traffic conditions. Camera-
based measurements are theoretically very effective in drowsiness detection
but, in real driving conditions, they are subjected to light and skin color
variations [5]. Physiological parameters that can be used for drowsiness study
are electrocardiography (ECG), electromyography (EMG), electrooculography
(EOG), electroencephalography (EEG), and respiration rate (RR) [6]. They can
provide very accurate results but some of them, such as EEG, require very
invasive sensors to be acquired, and this makes it difficult to use them for
driver drowsiness detection. The most used physiological parameters are heart
rate (HR), heart rate variability (HRV), and respiration rate (RR). The best
approach for driver monitoring is combining data from multiple sensors,
obtaining data that are more consistent and reliable [7]. This process is
known as sensor-data fusion and its great reliability is nowadays seen as a
way to increase road safety [8]. Our research group is involved in studying
sleep macro and micro patterns, sleep disorders, and daytime drowsiness
related to sleep onset while driving. In this study, data from different
sensors have been collected and paired as preliminary work for a fusion
algorithm able to give a comprehensive analysis of camera-based and
physiological-based parameters of the driver to evaluate drowsiness and
attention levels. A low attention level is of course dangerous by itself and
could be compromised by drowsiness and/or attention to the road, and these two
factors are crucial for motor vehicle accidents. A vital sign detection RADAR
was used to measure HR and RR, a smartwatch was used to detect HR, RR, and
HRV, and two cameras were used to find eye blinking (EB), eye gaze angles
(EGA), percentage of eyelid closure over the pupil over time (PERCLOS) and
head movements (HM).

##  II State of the Art

Regarding physiological parameters, the most accountable for sleep onset
prediction are:

  * •

Heart Rate (HR), which describes the contractions of the heart per minute.

  * •

Heart Rate Variability (HRV), which represents the change in time intervals
between adjacent heartbeats.

  * •

Respiration Rate (RR), which is the number of breaths a person takes per
minute.

These physiological parameters are the most reliable and accurate in
drowsiness detection as they are concerned with what is happening with the
driver physically; in fact, through their monitoring, it is possible to advise
the driver to stop the vehicle before the physical symptoms of drowsiness
effectively appear [9]. Moreover, they can be extracted both through contact
or contactless technologies[10].
Concerning contact-based solutions, heart signals such as electrocardiogram
(ECG) and photoplethysmography (PPG) are considered accurate measures of
fatigue. However, their use is limited because of the intrusive nature of the
sensors. Nevertheless, novel sensors can be embedded in the steering wheel or
the seat belt [11]. As proposed by Jung et al., electrodes were embedded in
the steering wheel; in this way, HR and HRV were extracted from ECG and
drowsiness was successfully detected. However, very highly accurate sensors
were needed and the position of the hands of the driver on the steering wheel
was crucial for the proper acquisition of the data[12]. In another study, Li
and Chuang developed a PPG sensor placed on the steering wheel of the vehicle.
In this case, HRV was extracted from the raw PPG signal. Then, a Support
Vector Machine (SVM) was trained to classify the state of the driver as
fatigued or alert, thus obtaining a 95% fatigue detection accuracy. Even
considering the non-intrusiveness of this method, it is susceptible to human
error and natural movements[13]. Recently, wrist-worn wearable devices have
been successfully employed as contact-based solutions for monitoring
physiological parameters [14]. As reported by Kundinger et al., by the use of
various smartwatch devices an accuracy of about 92% was reached in detecting
drowsiness, compared to medical-grade device[15]. However, the main limitation
of these devices can be the accuracy of sensing technology; in addition, it is
required that the subject wears the device, relying on the diligence of the
driver itself. Considering the challenges of contact-based methods, various
contactless solutions have been recently explored for monitoring physiological
parameters. Among them, techniques based on radar have shown promising
results. Liu et al. developed a novel radar-based technology for drowsiness
detection, where the accuracy of heart rate was 96.4% and the accuracy of
drowsiness detection was 82.9%, compared to other state-of-the-art
approaches[16]. Moreover, camera-based techniques offer the possibility to
estimate drowsiness through eyes and head tracking. Eye gaze direction is
widely used to detect drowsiness. In-Ho Choi and Yong-Guk Kim [17] track the
driver’s gaze direction by tracing the pupil’s center point. Eye blinking
becomes slower and more frequent while getting drowsier [18]. Shekari
Soleimanloo S et al. [19] used Optalert to detect eye blinking, an infrared
oculography system. Wang X and Xu C used instead the smarteye eye tracker
[20]. PERCLOS express for how much time there is at least 70$\%$ (or 80$\%$)
of eyelid closed during a unit of time of 1 minute (or 30 seconds). In [21],
B. Bakker et al. used a 3-camera smart eye pro system, with infrared lighting.
G. Du et al. used instead an RGB camera for PERCLOS detection [22]. Head
movements are another visual drowsiness sign as, while falling asleep, someone
could start nodding. They can be found by recording with a camera [23], or by
using a magnetic tracker, as the Ascension flock of birds [24].

##  III Proposed methodology

The system has been designed to obtain a time series of physiological
parameters from the previously described sensors. Each sensor communicates
with the experimenter workstation with different protocols and physical
interfaces. For this reason, we decided to design an application developed for
the Microsoft Windows environment to save a CSV file containing the logs from
the current experimental campaign, aligned from the timing point of view, and
to send, as a JSON record via MQTT, each acquisition to a centralized NoSQL
database.

Message Queuing Telemetry Transport (MQTT) allows the creation of a client-
server system. In this protocol, the server (in charge of relaying the record
to the clients) is called _broker_. A client can have two roles: if it
receives the relayed data, it is called _subscriber_ while, if it instead
sends records, it is a _publisher_. For this reason, MQTT is defined as a
publish-subscribe protocol.

JavaScript Object Notation (JSON) is a data-exchange format. It allows to
create a hierarchical data structure, and it is widely used in web development
due to its convenience: thanks to ready-to-use serializer and deserializer
functions, it is possible respectively to encapsulate data contained in a
class into a JSON string or vice-versa to extract the JSON string content
creating novel instances of the contained classes.

The designed system is capable of retrieving data from the following devices:

  * •

Vital Sign Detection Radar (Chuhang Technology Radar), featuring HR, RR, and
distance measurement. The latter value is needed to assess the quality of the
provided data. If the distance changes it means that there is a movement of
the person with respect to the Radar and hence that the measure is unreliable.
The Radar is directly connected via USB (emulated serial port, with data
formatted in binary form) with the experimenter workstation.

  * •

Wearable Device (GARMIN Enduro, Enduro 2, VenuSq, Venu 2) featuring off-the-
shelf HR, RR, HRV, and a drowsiness onset real-time prediction algorithm
developed by our research group. Bluetooth wireless protocol connects the
Device with an Android smartphone, which relays the data to a remote MQTT
broker that transmits a copy to its subscribers and stores the received data
into a persistence layer implemented resorting to MongoDB.

  * •

Camera (Varroc and one developed by our research team) featuring EB, EGA,
PERCLOS, and HM.

As it is possible to observe, some data of these acquisitions are redundant
(HR and RR), but it is important to keep both measurements due to the
different measurement technologies. In particular, the RR is more reliable
when measured by the radar, while the HR is more reliable when measured by the
wearable device.

In Fig.1 it is shown the architecture of the system. The central node is the
workstation running the Data fusion collector. It receives data from the
dashcam (via a Wi-Fi or Ethernet connection), resorting to the MQTT protocol,
and from the vital sign detection radar, via a virtual serial port over USB.
The data from the wearable device are received indirectly: the smartwatch
sends the data to an app running on a smartphone via a virtual serial port
over Bluetooth. The smartphone sends the data via Internet (mobile
communication standards or Wi-Fi), resorting to the MQTT protocol to a remote
server, which runs a Broker that receives the data and send them back to the
Data Fusion Collector (in this case, acting as a Subscriber). Finally, the
Data fusion collector saves the received data into a CSV file in the
workstation’s local file system and sends it to the remote server via MQTT (in
this case, acting as a Publisher). The remote server stores all the received
data in mongoDB. The data transmitted over MQTT are encapsulated, resorting to
the JSON format, allowing to store into the mongoDB directly the received
records.

The variety of devices involved justifies the choice of a NoSQL database: it
is possible to use different dashcams, wearable devices, and vital sign
detection radar, each with a different set of recorded measurements. Thanks to
the NoSQL database (with does not feature fixed-structure tables) and JSON
format, it is possible to change the record structure seamlessly.

Figure 1: The proposed architecture to perform Data Fusion.

Currently, the system is a work in progress, so we expect to add more sensors
to this set in the following months, mainly to measure environmental
conditions like air relative humidity and temperature, ambient light sensors,
and air quality parameters, in particular, carbon dioxide quantity.

Another future expansion of this system will be the possibility of generating
synchronized time series in offline mode: the devices are equipped with
proprietary tools to connect them to the workstation, which can create log
files: for using these for data fusion, it is needed to align all the data
from the timing point of view: the application will read the logs and generate
a newer one with the fusion. Moreover, it can also store in the NoSQL database
the data fusion to make data available for remote access and long-term
reliable storage.

The purpose of the system data-fusion system is twofold:

  1. 1.

in this phase, to aid the experiments described in Section IV, needed for the
development and testing of the data fusion algorithm.

  2. 2.

since the data fusion collector has the availability of all the data from the
sensors acts as a proof-of-concept of the final system running the real-time
warning system.

The data collected in phase 1 are currently used to develop the real-time
algorithm iteratively: the algorithm runs offline, and the results are
compared with respect to the sleep onset detected from the various
polysomnography read by sleep expert medical doctors.

Considering a possible future use in real vehicles (phase 2), the
architectural choices may appear clearer: the dashboard/infotainment computer
system can run the data fusion algorithm and warn the driver. It simplifies
the management of the camera and radar: they can be part of the car itself,
directly connected to the car power supply, and wired to the onboard systems.
This direct connection allows for fulfilling the mandatory need for driver
drowsiness and attention warning (DDAW) (Regulation (EU) 2019/2144). In any
case, when the driver wears a smartwatch, connecting their smartphone to the
system is more convenient by considering the typical commercial use: the
driver connects their phone via Bluetooth to the car’s infotainment system.
Exploiting this link to seamlessly exchange data about drowsiness to the data
fusion algorithm through the infotainment system is possible. Looking forward,
proposing a communication standard to allow different vendors to implement
this functionality in their products appears to be a good idea.

##  IV Experimental results

In this first stage, we conducted some experiments (Mantainance Weakfullness
Tests) involving 16 volunteers, 5 females, 11 males, age range 25-30 to verify
the sensitivity and accuracy of the data fusion algorithm.

Each test lasted about 1.5 h, considering the instrumentation with the
polysomnograph and its pads and wirings and the set-up of the data fusion
system. After the instrumentation phase, the volunteers are asked to take two
test sessions of 20 minutes of Mantainance Weakfullness Test (MWT). Between
the two tests, the volunteers are required to answer two questionnaires. The
first questionnaire, compiled before the tests, aimed to understand the
participants’ clinical status and his/her capability to stay awake (ESS). The
second questionnaire used the Karolinska Sleepiness Scale to evaluate their
drowsiness level during the test.

The fusion approach has undergone thorough evaluation to validate its
effectiveness. This evaluation encompasses two key aspects: firstly, enhancing
the overall representation’s quality, including the information about
distractions, and secondly, bolstering the system’s reliability through
redundancy.

Figure 2: Comparison between physiological-based drowsiness detection and
camera-based attention detection. On the vertical axis on the left (black) are
reported the outcomes from the wearable device, while on the axis on the right
(orange) are the outcomes from the camera.

In Fig.2, it can be observed that a complete understanding of the passenger’s
status requires additional information about distraction, which is not
directly evident from the physiological data.

Figure 3: Comparison between physiological-based (left axis) and camera-based
(right axis) detected drowsiness level.

In Fig.3, it is evident that the physiological-based method provides clearer
and smoother information regarding the drowsiness level. Notably, the blue
line indicates potential alarms for the passenger minutes earlier than the red
line. To further enhance the system’s performance, implementing system
redundancy to establish a clearer initial driver condition would be
beneficial.

##  V Conclusions

In this study, a significant array of sensors was utilized, each capable of
operating independently and improving performance by leveraging information
from other devices.

The results section examined a specific real-life scenario involving a
combination of a camera and a wearable device. This integrated system provided
a comprehensive understanding of the driver’s status in terms of both
drowsiness and attention levels.

Currently, the investigation is focused on merging information from the
various sensors utilized. The future direction of this research involves the
development of a real-time sensor fusion algorithm to effectively synthesize
data from all sensors simultaneously. This advancement aims to further
streamline and enhance the overall system’s performance.

The algorithm development is ongoing as the radar results have not been
presented yet due to lower data accuracy, specifically concerning the wearable
data.

## References

  * [1] WHO Road Traffic Injuries Report, https://www.who.int/news-room/fact-sheets/detail/road-traffic-injuries, last visited 2023/04/26.
  * [2] G. Zhenhai, L. DinhDat, H. Hongyu, Y. Ziwen and W. Xinyu, ”Driver Drowsiness Detection Based on Time Series Analysis of Steering Wheel Angular Velocity,” 2017 9th International Conference on Measuring Technology and Mechatronics Automation (ICMTMA), Changsha, China, 2017, pp. 99-101, doi: 10.1109/ICMTMA.2017.0031.
  * [3] O. Khunpisuth, T. Chotchinasri, V. Koschakosai and N. Hnoohom, ”Driver Drowsiness Detection Using Eye-Closeness Detection,” 2016 12th International Conference on Signal-Image Technology & Internet-Based Systems (SITIS), Naples, Italy, 2016, pp. 661-668, doi: 10.1109/SITIS.2016.110.
  * [4] L. Pugliese, M. Violante and S. Groppo, ”A Novel Algorithm for Detecting the Drowsiness Onset in Real-Time,” in IEEE Access, vol. 10, pp. 42601-42606, 2022, doi: 10.1109/ACCESS.2022.3167708.
  * [5] Sahayadhas A, Sundaraj K, Murugappan M. Detecting driver drowsiness based on sensors: a review. Sensors (Basel). 2012 Dec 7;12(12):16937-53. doi: 10.3390/s121216937. Erratum in: Sensors (Basel). 2021 Jan 11;21(2): PMID: 23223151; PMCID: PMC3571819.
  * [6] Siddiqui HUR, Saleem AA, Brown R, Bademci B, Lee E, Rustam F, Dudley S. Non-Invasive Driver Drowsiness Detection System. Sensors (Basel). 2021 Jul 15;21(14):4833. doi: 10.3390/s21144833. PMID: 34300572; PMCID: PMC8309856.
  * [7] Marceddu, A.C.; Pugliese, L.; Sini, J.; Espinosa, G.R.; Amel Solouki, M.; Chiavassa, P.; Giusto, E.; Montrucchio, B.; Violante, M.; De Pace, F. A Novel Redundant Validation IoT System for Affective Learning Based on Facial Expressions and Biological Signals. Sensors 2022, 22, 2773. https://doi.org/10.3390/s22072773
  * [8] Deo A, Palade V, Huda MN. Centralised and Decentralised Sensor Fusion-Based Emergency Brake Assist. Sensors (Basel). 2021 Aug 11;21(16):5422. doi: 10.3390/s21165422. PMID: 34450863; PMCID: PMC8400951.
  * [9] L. Pugliese, M. Violante and S. Groppo, ”Real-time sleep prediction using a virtual sensor to estimate Heart Rate Variability through Respiratory Rate,” 2022 IEEE 16th International Conference on Application of Information and Communication Technologies (AICT), Washington DC, DC, USA, 2022, pp. 1-4, doi: 10.1109/AICT55583.2022.10013549.
  * [10] M. Ramzan, H. U. Khan, S. M. Awan, A. Ismail, M. Ilyas, and A. Mahmood, “A Survey on State-of-the-Art Drowsiness Detection Techniques,” IEEE Access, vol. 7, pp. 61904–61919, 2019, doi: 10.1109/ACCESS.2019.2914373.
  * [11] Sikander, G., Anwar, S. (2019). Driver Fatigue Detection Systems: A Review. IEEE Transactions on Intelligent Transportation Systems, 2339–2352. https://doi.org/10.1109/TITS.2018.2868499
  * [12] Jung, S. J., Shin, H. S., Chung, W. Y. (2014). Driver fatigue and drowsiness monitoring system with embedded electrocardiogram sensor on steering wheel. IET Intelligent Transport Systems, 43–50. https://doi.org/10.1049/iet-its.2012.
  * [13] Li, G., Chung, W. Y. (2013). Detection of driver drowsiness using wavelet analysis of heart rate variability and a support vector machine classifier. Sensors (Switzerland), 16494–16511. https://doi.org/10.3390/s131216494
  * [14] (In press) L. Pugliese, M. Violante and R. Groppo, ”Real-time Sleep Prediction Algorithm using Commercial Off the Shelf Wearable Devices,” 2023 IEEE Smart World Congress, Porstmouth, UK, 2023, pp. 1-4.
  * [15] Kundinger, T., Sofra, N., Riener, A. (2020). Assessment of the potential of wrist-worn wearable sensors for driver drowsiness detection. Sensors (Switzerland). https://doi.org/10.3390/s20041029
  * [16] Liu, S., Zhao, L., Yang, X., Du, Y., Li, M., Zhu, X., Dai, Z. (2022). Remote Drowsiness Detection Based on the mmWave FMCW Radar. IEEE Sensors Journal, 15222–15234. https://doi.org/10.1109/JSEN.2022.3186486
  * [17] In-Ho Choi and Yong-Guk Kim, ”Head pose and gaze direction tracking for detecting a drowsy driver,” 2014 International Conference on Big Data and Smart Computing (BIGCOMP), Bangkok, 2014, pp. 241-244, doi: 10.1109/BIGCOMP.2014.6741444.
  * [18] B. G. Pratama, I. Ardiyanto and T. B. Adji, ”A review on driver drowsiness based on image, bio-signal, and driver behavior,” 2017 3rd International Conference on Science and Technology - Computer (ICST), Yogyakarta, Indonesia, 2017, pp. 70-75, doi: 10.1109/ICSTC.2017.8011855.
  * [19] Shekari Soleimanloo S, Wilkinson VE, Cori JM, Westlake J, Stevens B, Downey LA, Shiferaw BA, Rajaratnam SMW, Howard ME. Eye-Blink Parameters Detect On-Road Track-Driving Impairment Following Severe Sleep Deprivation. J Clin Sleep Med. 2019 Sep 15;15(9):1271-1284. doi: 10.5664/jcsm.7918. PMID: 31538598; PMCID: PMC6760410.
  * [20] Wang X, Xu C. Driver drowsiness detection based on non-intrusive metrics considering individual specifics. Accid Anal Prev. 2016 Oct;95(Pt B):350-357. doi: 10.1016/j.aap.2015.09.002. Epub 2015 Oct 1. PMID: 26433567.
  * [21] B. Bakker et al., ”A Multi-Stage, Multi-Feature Machine Learning Approach to Detect Driver Sleepiness in Naturalistic Road Driving Conditions,” in IEEE Transactions on Intelligent Transportation Systems, vol. 23, no. 5, pp. 4791-4800, May 2022, doi: 10.1109/TITS.2021.3090272.
  * [22] G. Du, L. Zhang, K. Su, X. Wang, S. Teng and P. X. Liu, ”A Multimodal Fusion Fatigue Driving Detection Method Based on Heart Rate and PERCLOS,” in IEEE Transactions on Intelligent Transportation Systems, vol. 23, no. 11, pp. 21810-21820, Nov. 2022, doi: 10.1109/TITS.2022.3176973.
  * [23] R. Oyini Mbouna, S. G. Kong and M. -G. Chun, ”Visual Analysis of Eye State and Head Pose for Driver Alertness Monitoring,” in IEEE Transactions on Intelligent Transportation Systems, vol. 14, no. 3, pp. 1462-1469, Sept. 2013, doi: 10.1109/TITS.2013.2262098.
  * [24] J. C. Popieul, P. Simon and P. Loslever, ”Using driver’s head movements evolution as a drowsiness indicator,” IEEE IV2003 Intelligent Vehicles Symposium. Proceedings (Cat. No.03TH8683), Columbus, OH, USA, 2003, pp. 616-621, doi: 10.1109/IVS.2003.1212983.