# Heartbeat Classification in Wearables Using Multi-layer Perceptron and Time-
Frequency Joint Distribution of ECG

Anup Das  1234-5678-9012-3456 Department of ECE, Drexel University3141
Chestnut StreetPhiladelphiaPA19104USA<EMAIL_ADDRESS>,  Francky Catthoor
IMECLeuvenBelgium and  Siebren Schaafsma  Stichting Imec NederlandKapeldreef
75, 3001EindhovenNetherlands

(2009; 2018)

###### Abstract.

Heartbeat classification using electrocardiogram (ECG) data is a vital
assistive technology for wearable health solutions. We propose heartbeat
feature classification based on a novel sparse representation using time-
frequency joint distribution of ECG. Fundamental to this is a multi-layer
perceptron, which incorporates these signatures to detect cardiac arrhythmia.
This approach is validated with ECG data from MIT-BIH arrhythmia database.
Results show that our approach has an average 95.7% accuracy, an improvement
of 22% over state-of-the-art approaches. Additionally, ECG sparse distributed
representations generates only 3.7% false negatives, reduction of 89% with
respect to existing ECG signal classification techniques.

Spiking neural network (SNN), global synapse, particle swarm optimization
(PSO), CxQuad, spike disorder count, inter-spike distortion

††journal: TWEB††journalvolume: 9††journalnumber: 4††article: 39††journalyear:
2010††publicationmonth: 3††copyright: acmlicensed††doi:
0000001.0000001††copyright: none††journalyear: 2018††copyright:
acmcopyright††conference: ACM/IEEE International Conference on Connected
Health: Applications, Systems and Engineering Technologies; September 26–28,
2018; Washington, DC, USA††booktitle: ACM/IEEE International Conference on
Connected Health: Applications, Systems and Engineering Technologies (CHASE
’18), September 26–28, 2018, Washington, DC, USA††price: 15.00††doi:
10.1145/3278576.3278598††isbn: 978-1-4503-5958-0/18/09

##  1\. Introduction

With the growing maturity of semiconductor process technology, miniature
sensors are increasingly being integrated into wearable devices, facilitating
vital health monitoring of humans. One important sensor in this context is the
electrocardiogram (ECG) sensor, recording electrical activities associated
with the human heart. ECG signals are useful to detect (1) cardiac arrhythmia,
related to abnormal heart rhythms; (2) ischemia, related to poor blood flow to
the heart muscles, (3) heart abnormalities such as enlarged heart; and (4)
past heart attacks. In this paper, we focus on accurate classification of
heartbeats leading to accurate detection of cardiac arrhythmia, which is a key
technology enabler for wearable health solutions.

Figure 1 shows the QRS complex in a ECG signal. This corresponds to the
depolarization of the left and the right ventricles of a human heart. The P,
Q, R, S and T waves occur in quick succession as shown in the figure.
According to the publicly available ECG database released for cardiac
arrhythmia (MIT-BIH arrhythmia database 1), heartbeats can be classified into
23 different categories such as normal beats, atrial premature beats,
superventicular escape beats, etc. based on the relative position and shape of
these waves. However, according to the ANSI/AAMI/ISO EC57:1998/(R)2008
(American National Standard on testing and reporting performance results of
cardiac rhythm and ST-segment measurement algorithms) 2, only five categories
are sufficient – (1) normal beats, (2) superventricular beats, (3) ventricular
beats, (4) fusion beats and (5) unclassified beats. Additionally, arrhythmia
detection is a direct function of these five heartbeat classes. The arrhythmia
detection problem is therefore reduced to accurate classification of heartbeat
classes. Although our approach considers all 23 classes, the classification
accuracy is improved when a classifier is trained using only these 5 required
classes.

Significant research have been conducted recently to autonomously classify ECG
heartbeats (QRS peaks) 3, 4, 5, 6, 7, 8, 9, 10, 11, 12 (Section 2 provides
details of these approaches.). These studies have the following limitations
that we address in this work. First, some of these studies are based on using
designer specified deterministic QRS features, which in some cases can be
incomplete. Machine learning provides a framework for automatic extraction of
hidden features from ECG signals and is therefore gaining increasing
popularity in this community. Second, existing machine learning based
approaches use either time samples or frequency components for classification.
We show that a time-frequency joint distribution captures time-varying
frequency components, which is critical in anomaly detection of non-stationary
and non-Gaussian signals such as speech and ECG. Third, most works fail to
analyze the impact of noise on heartbeat classification accuracy, which is
important, as noise from data acquisition systems are unavoidable.

In this paper we propose a two-phase approach to real-time heartbeat
classification. Fundamental to this is an offline phase (phase I) that uses
time-frequency joint distribution of QRS complexes and convert it into sparse
distributed signatures that form the basis of classification using multi-layer
perceptron. The trained classifier is then used online (phase II) to classify
live ECG heartbeats detected using Pan & Tompkin’s Algorithm13. Experiments
conducted with the standard MIT-BIH arrhythmia database show that our approach
is able to classify heartbeats with 80.7% – 100% accuracy (average 95.7%) for
all heartbeat classes. Additionally, the proposed ECG sparse distributed
signatures have high noise tolerance (by average 133%) and low false negatives
of only 3.7%, an average 89% lower compared to existing ECG representations.

Contributions: Following are our key contributions.

  * •

a two-phase approach for autonomous and real-time heartbeat classification;

  * •

a time-frequency joint distribution based sparse distributed representation
(signature) for ECG signals;

  * •

a multi-layer perceptron-based classifier to classify these signatures; and

  * •

noise analysis of the proposed approach with Additive White Gaussian Noise.

The remainder of this paper is organized as follows. Related works are
discussed in Section 2. The proposed two-phase approach is discussed in
Section 3 and the problem formulation in Section 4. Results are presented in
Section 5 and conclusions in Section 6.

Figure 1. QRS complex in ECG signal.

##  2\. Related works

Existing studies on heartbeat classification can be categorized into – (1)
classification using deterministic features and (2) classification using
machine learning. Autonomous classification using deterministic heartbeat
features have received research attention with the recent advances and wide-
spread use of wearable devices such as the ECG band and smart watch. In 3,
heartbeats are classified using ECG morphology, heartbeat intervals, and RR-
intervals, extracted from ECG signals. This model is extended in 4 with five
additional features and in 5 with ECG signal amplitude difference as a key
feature. Although these techniques are based on features that are clinically
understood, in some cases these can be incomplete. Supervised machine learning
in the form of classifiers are emerging as a popular alternative to the
classical deterministic approaches due to the ability to autonomously identify
and represent hidden features extracted from ECG signals. In 7, support vector
machine is used to classify ECG signals. A deep neural network based
classifier is used as an alternative in 8. Other alternatives include the
random forest approach of 9 and the decision tree approach of 10. All these
approaches use ECG time samples. There are also some studies using ECG
frequency components. In 11, coefficients of discrete wavelet transform are
used for classification using a neural network. In 12, ECG features are
extracted using discrete Fourier transform of ECG time samples. As discussed
in Section 1, none of these approaches consider time variation of the
frequency components of ECG signals, which is essential to identify anomalies
such as arrhythmia detection.

##  3\. Proposed Approach

Figure 2 outlines our proposed approach, comprising of an offline training
phase followed by a online phase for real-time heartbeat classification. The
offline training phase consists of training a deep neural network using
labeled QRS peaks from the MIT-BIH arrhythmia database 1. Each QRS peak is
first transformed into its time-frequency representation. The ECG signal
together with its real and imaginary components of time-frequency joint
distribution are plotted in Figure 3 for three scenarios – (1) No Beat, (2)
Normal Beat and (3) Atrial Premature Beat. As seen from Figure 3, the real and
imaginary plots differ from each other for all three scenarios indicating the
significance of both these components. We propose to combine these components
to form unique signatures, which are shown on the rightmost subplots of Figure
3. These ECG signatures together with their original labels are then used for
training the neural network as shown in Figure 2.

The offline training phase is carried out on nvidia GPU using python machine
learning library (theano 14). The classifier used in this case is the multi-
layer perceptron (MLP) with error back propagation. This choice is due to the
fact that MLP are computationally less complex (important for real-time use),
yet powerful tool and can be easily ported on a neuromorphic hardware 15. Once
trained, the neural network is validated using a test set. A design space
exploration is then performed to determine dimensions of the neural network
(hidden layers, number of neurons, etc). Online phase of the proposed approach
uses ECG time samples to first detect QRS peaks using Pan & Tompkins Algorithm
13. Detected peaks are then converted into ECG signatures and are classified
using the trained neural network. This online phase can also be integrated
within a ECG acquisition system 16, 17.

Figure 2. Offline and online phases of the proposed approach.

Figure 3. ECG signal and plot for real and imaginary components of time-
frequency joint distribution. Also shown are the sparse distributed
signatures.

##  4\. Problem Formulation

###  4.1. ECG Sparse Distributed Signatures

The classical Fourier analysis assumes that signals are of infinite duration
or periodic in nature. However, non-stationary signals such as the ECG and
speech are of short duration (transient), and change substantially over this
duration. To accurately capture time varying frequency components of these
transient signals, time-frequency joint distributions (such as short-time
Fourier transform and wavelet transform) are frequently used. We use short-
time Fourier transform of ECG signals due to its real-time properties.

The short-time Fourier transform of a discrete-time signal $\small x(t)$ is
given by the following equation 18

(1) $\footnotesize X(t,\omega)=\sum_{\tau=-\infty}^{\infty}h(t-\tau)\cdot
x(\tau)\cdot e^{-j\omega\tau}$

where, $h(t)$ is the analysis window, which is narrow in time and frequency,
and is normalized such that $\small h(0)=1$. This equation is similar to the
classical Fourier transform, except that $\small X(t,\omega)$ is now a
function of both time and frequency, and represents only a local behavior of
$\small x(\tau)$ as viewed through the sliding window $\small h(t-\tau)$.
Using the real and the imaginary components, we create a signature as follows.

(2)
$\footnotesize\mathcal{S}=\left[f\left\\{\mathbb{R}(X)\right\\}~{}~{}~{}~{}f\left\\{\mathbb{I}(X)\right\\}~{}~{}~{}~{}\mathbb{P}\right]$

where $\small f\left\\{x\right\\}$ is a transformation of $\small x$;
$\small\mathbb{R}(X)$ and $\small\mathbb{I}(X)$ are respectively, the real and
the imaginary components of $\small X$ (Equation 1); and $\mathbb{P}$ is the
padded zero component, which is introduced for future extension of the
approach to incorporate additional parameters. The signature generated using
the short-time Fourier transform components are shown in the fourth subplot of
Figure 3, where the unused padded components $\small\mathbb{P}$ (bottom right
corner in the figure) are indicated with circles. Each signature is a
$82\times 82$ matrix with only few elements having values above a certain
threshold (sparsity $<1$%). These elements are shown as different color dots
for each ECG sparse distributed signature. The position of these high value
elements in the matrix distinguishes one signature from another uniquely,
resulting in high classification accuracy and high noise tolerance 19 (see
Section 5).

Figure 4. Single neuron model and neural network architecture.

###  4.2. MLP Classifier

An artificial neural network is a computing paradigm inspired by the
biological computation of the brain. It comprises of processing elements
(neurons) working in unison to solve a specific problem. A simple artificial
neuron is shown in Figure 4. It computes the weighted sum of its inputs (from
external world or from other neurons). If this sum is greater than a
threshold, the neuron is said to _fire_ (i.e., produces an output). This
output is then communicated to external world or connected to the input of
another neuron. A typical feed-forward neural network is shown on the right
side of Figure 4. This consists of an input layer, one or more hidden layers,
and an output layer. The number of neurons in the input layer is equal to the
dimensions (also called _features_) of the input applied to this neural
network. We use ECG sparse distributed signatures $\small\mathcal{S}^{82\times
82}$ as input to the neural network and therefore, the number of input layer
neurons is $\small 6724~{}(=82\times 82)$. The number of hidden layers and
neurons per hidden layer are configurable and determines the training time and
accuracy of classification. Finally, the number of neurons in the output layer
is same as the number of heartbeat classes. It is to be noted that
conceptually, the deep layers of a neural network represents hierarchical and
non-linear combination of features detected from the input. Thus, instead of
hand coding essential features, neural network autonomously selects (and
combines) features resulting in high classification accuracy. This justifies
our choice of neural networks for heartbeat classification.

Neural networks have two modes of operation – training and testing,
facilitating supervised learning. In the training mode, a neural network works
with a set of inputs (ECG sparse distributed signatures) and their known
outputs (QRS labels). In this mode, weights (also called _synaptic weights_)
of the network are adjusted to minimize the error between the expected and
actual output. In the testing mode, the network output is evaluated for an
unknown input. Back propagation algorithm 20 is one of the most popular
approaches for supervised learning in a neural network.

Figure 5. QRS detection using Pan & Tompkin’s Algorithm.

###  4.3. QRS Detection

A real-time implementation of Pan & Tompkin’s algorithm is proposed in 13.
This is shown as a flow chart in Figure 5. It detects QRS complexes based on
the slope, amplitude and width of ECG lead II signals. The digitized signal
from an acquisition system 16 is first processed using a band-pass filter,
which is a cascaded lowpass and highpass filter. The purpose of this filter is
to reduce noise in ECG signals (from muscle contraction-expansion and T-wave
interference) by matching the spectrum of average QRS complexes. Output of the
bandpass filter is processed using a five point differentiator to extract
information on the QRS complex and eliminate baseline drift effects. The next
operation is squaring, which is a nonlinear transformation of the signal and
serves two purposes – (1) makes all data positive and (2) emphasize (and
attenuate) high (and low) frequency components of the signal. The final
transformation is the moving average filtering, which smooths the signal and
performs moving window integration of 150ms.

##  5\. Results and Discussions

We validated our approach using the MIT-BIH Arrhythmia Database 1 (part of
Physionet Database). Offline phase of our approach is implemented using
python: short-time Fourier transform using specgram of python-matplotlib; and
neural network training using lasagne of python-theano14. There are 23
heartbeat classes (provided as annotations in 1). So the output layer of the
neural network consists of 23 neurons. The training set is composed of 6500
labeled QRS complexes selected randomly from 120K QRS complexes for 48
patients in the database, with representatives from every classes. The test
set is composed of all 120K QRS complexes (except the ones used for the
training set). These test QRS complexes are represented from all 48 patients
and 23 classes. Each ECG QRS complex is a sparse distributed signature of
82x82 matrix and therefore, the number of neurons in the input layer is 6724.
The design space exploration determines the number of hidden layers, neurons
per hidden layer and synaptic weights on nvidia GeForce GTX 970 GPU @ 1.05GHz.
The onlne phase is performed on Hashwell CPU.

Figure 6. Prediction results for 23 heartbeat classes.

###  5.1. Classification Accuracy

Figure 6 reports performance of our approach for 23 heartbeat classes. For
comparison, classification results using ECG time samples 6 and ECG frequency
components 11 are also reported in the plot. As can be seen from this figure,
using time samples has higher classification accuracy than using frequency
components for some classes such as 3, 14, 17, 21 & 23\. For these heartbeat
classes, amplitude of ECG signals is sufficient for classification; frequency
components convey little or no information as discussed in 10. For other
classes such as 2, 8, 15, 19 & 20, frequency domain representation carries
more information needed for classification. Compared to these approaches, our
approach not only captures both time and frequency representations, but also
time variations of the frequency components in the sparse representation,
leading to higher classification accuracy for all classes. The accuracy
obtained using our approach is 80.7% to 100% (average 95.7%), $\approx$ 15% –
70% (average 46%) improvement over 11 and 2.5% – 54.4% (average 22%)
improvement over 6.

Table 1. Classification accuracy with different training classes. Heartbeat Class | Training with 23 classes | Training with 5 classes
---|---|---
normal | 91.7% | 95.1%
superventricular | 95.3% | 95.7%
ventricular | 98.9% | 99.5%
fusion | 89.8% | 93.4%
unclassified | 86.8% | 91.2%

Table 1 reports results for the five required classes for two scenarios – (1)
when the classifier is trained with all 23 classes (column 2) and (2) when the
classifier is trained with only 5 classes (column 3). As can be seen from this
table, training with 5 classes improves accuracy by 0.4% – 5.1% (average 2.8%)
as compared to training with all 23 classes. These results reconfirm that
using ECG sparse distributed signatures result in high classification
accuracy, which can be improved further by training with only the heartbeat
classes required for detecting a specific heart abnormality. Thus the approach
proposed in this work is generic and can be extended using disease-specific
learning.

###  5.2. Choice of ECG Preprocessing

Table 2 reports results justifying our choice of ECG sparse distributed
signatures for classification. We report 4 numbers in the table – average
accuracy for 48 patients in the database (column 2); average accuracy for 23
heartbeat classes (column 3); average false positives (column 4); and average
false negatives (column 5). In this context, false positive refers to a
scenario where a heartbeat is actually normal (class 1) but the classifier
classifies it as abnormal, belonging to another class (classes 2 – 23). A
false negative is a scenario in which a heartbeat belongs to a specific class
and the classifier classifies this to be of different class. It is to be noted
that a false positive implies false alarm, while a false negative refers to
miss detection. Therefore, lower are the false positives (and negatives),
better is the quality.

Table 2. Justifying the choice of ECG Sparse Distributed Signatures. Techniques | Patient (Avg.) | Type (Avg.) | False +ve | False -ve
---|---|---|---|---
Wavelet Transform12 | 58.4% | 57.4% | 23.8% | 43.5%
Short-time Fourier Transform | 77.1% | 65.5% | 18.5% | 35.3%
Numenta SDR 21 | 80.3% | 85.1% | 21.5% | 15.6%
Our Sparse Signature | 83.7% | 95.7% | 19.3% | 3.7%

We compare our ECG sparse distributed signatures with (1) wavelet transform
based representation 12, (2) short-time Fourier transform based representation
and (3) Numenta’s sparse representation technique for ECG time sample 21. The
classifier in all these scenarios are the same (i.e., multi-layer perceptron),
only the input to the classifier are different according to representations
used for ECG. As seen from the table, ECG sparse distributed signature-based
representation has higher accuracy than the other three representations, for
all heartbeat classes and also for all patients in the database. Furthermore,
ECG sparse distributed signature-based representation has lower false
positives than 12, but higher than using short time Fourier transform
representation by only 5%. However, our approach has a much lower false
negatives (3.7%), a reduction of 89% and 91% compared to the wavelet and
short-time Fourier transform representations, respectively. These improvements
are due to the fact that we combine the magnitude and phase of time-frequency
joint distribution into sparse distributed signatures, which uniquely
distinguishes QRS complexes, resulting in higher classification accuracy.

Figure 7. Noise tolerance of the proposed approach.

###  5.3. Additive White Noise Tolerance

To determine the noise tolerance of our approach, we injected random noise to
the ECG signal and used our approach to classify heartbeats of this noisy ECG
signal. We used Additive White Gaussian Noise (AWGN) as the basic noise model
to mimic the effect of random processes associated with ECG data acquisition.
Under the assumption of AWGN, let $\small
X=\left\\{x_{1},\cdots,x_{L}\right\\}$, be $\small L$ samples of the input
vector $\small X$. The relative power of the noise is described in terms of
signal-to-noise ratio (in dB) per sample. Let this be $\small SNR$. Power of
the input ECG signal (without noise) is

(3) $\footnotesize E_{s}=\frac{1}{L}\sum_{i=1}^{L}|x_{i}|^{2}$

The noise vector is given by

(4) $\footnotesize N_{s}=\sqrt{\frac{E_{s}}{SNR_{lin}}}\cdot randn(1,L)$

where, $\small SNR_{lin}=10^{SNR/10}$ is the SNR value converted to linear
scale, and $\small randn(1,L)$ is a function that generates $\small L$ random
numbers. The noisy ECG signal is $\small X_{n}=X+N_{s}$.

Figure 7 plots the classification accuracy of our approach, as the SNR per
sample is increased from 0 to 20 dB, for patient id 106 in the database. To
compare the noise tolerance, we have also plotted classification results using
ECG time samples 6. As shown, the accuracy of our approach and that of 6
increases with an increase in the SNR as signal strength increasingly
dominates over noise, resulting in higher classification accuracy. A similar
trend is observed for all other patients in the database. Additionally, our
approach has higher classification accuracy (60% – 170%, average 133% for the
SNR range of 0 to 20 dB) compared to 6. This is because sparse distributed
signatures can tolerate high noise than other representations.

Finally, motion artifacts represent an important class of noise for ambulatory
subjects. Our initial experiments with motion compensated ECG from 22 provides
a comparable accuracy with recently reported wavelet-ICA approach 23.

Table 3. Offline and online execution times. Neurons | Offline | Online (real-time) at 160 MHz
---|---|---
Training | QRS Detect | Sparse Distributed Signature | Classification
2000 | 620.4 s | 10 $\mu$s | 10.4 ms | 1.20 ms
4000 | 907.9 s | 10 $\mu$s | 10.4 ms | 1.70 ms
6000 | 1277.9 s | 10 $\mu$s | 10.4 ms | 2.42 ms
8000 | 1693.3 s | 10 $\mu$s | 10.4 ms | 3.06 ms
10000 | 2245.4 s | 10 $\mu$s | 10.4 ms | 3.97 ms

###  5.4. Real-time Performance

Table 3 reports execution time for the offline training phase and three
components of the online phase – the QRS detection, ECG sparse distributed
signature conversion and heartbeat classification. These are reported in
columns 2, 3, 4 and 5, respectively for different number of neurons. It is to
be noted that the offline phase is executed on nvidia GPU and does not
contribute to real-time performance. As seen from column 2, the offline
training time increases exponentially with increase in the number of neurons.
This is due to an exponential increase in the number of synapses, leading to
an increase in the number of error computations in the back propagation
algorithm. At run-time, the QRS detection algorithm takes on average 10 $\mu$s
to execute the steps listed in Figure 5 and are reported in column 3. The
execution time to convert a QRS peak to ECG sparse distributed signature is
reported in column 4 and is on average 10.4 ms. Finally, the execution time
for the heartbeat classification step is reported in column 5. As shown in
this column, heartbeat classification using a trained neural network increases
with the number of neurons. However, the small execution time validates the
real-time property of the proposed approach.

##  6\. Conclusion

We proposed a real-time approach for classifying heartbeats to detection
abnormal heart conditions such as arrhythmia. The approach is based on multi-
layer perceptron classifier that uses ECG sparse distributed signatures to
classify incoming ECG heartbeats into one of 23 different classes. Experiments
with ECG signals from MIT-BIH arrhythmia database show that the proposed
approach has a classification accuracy of an average 95.7% (22% better than
state-of-the-art). Additionally, the proposed ECG sparse distributed
signatures have higher noise tolerance and reduce false negatives to 3.7%, an
89% improvement over existing ECG data representation techniques.

###### Acknowledgements.

This work is supported in parts by EU-H2020 grant NeuRAM3 Cube (NEUral
computing aRchitectures in Advanced Monolithic 3D-VLSI nano-technologies) and
ITEA3 proposal PARTNER (Patient-care Advancement with Responsive Technologies
aNd Engagement togetheR).

## References

  * [1] G. B. Moody and R. G. Mark, “The impact of the MIT-BIH arrhythmia database,” _IEEE Engineering in Medicine and Biology Magazine_ , 2001.
  * [2] A. for the Advancement of Medical Instrumentation _et al._ , “Testing and reporting performance results of cardiac rhythm and st segment measurement algorithms. american national standards institute, inc.(ansi),” _Inc.(ANSI), ANSI/AAMI/ISO EC57_ , 2008.
  * [3] P. De Chazal, M. O’Dwyer, and R. B. Reilly, “Automatic classification of heartbeats using ECG morphology and heartbeat interval features,” _IEEE Transactions on Biomedical Engineering_ , 2004.
  * [4] M. Llamedo and J. P. Martínez, “Heartbeat classification using feature selection driven by database generalization criteria,” _IEEE Transactions on Biomedical Engineering_ , 2011.
  * [5] J. Park, S. Lee, and K. Kang, “Arrhythmia detection using amplitude difference features based on random forest,” in _IEEE EMBC_ , 2015.
  * [6] V. Queiroz, E. Luz, G. Moreira, Á. Guarda, and D. Menotti, “Automatic cardiac arrhythmia detection and classification using vectorcardiograms and complex networks,” in _EMBC_. IEEE, 2015.
  * [7] Z. Zhang, J. Dong, X. Luo, K.-S. Choi, and X. Wu, “Heartbeat classification using disease-specific feature selection,” _Elsevier Computers in biology and medicine_ , 2014.
  * [8] M. Al Rahhal, Y. Bazi, H. AlHichri, N. Alajlan, F. Melgani _et al._ , “Deep learning approach for active classification of electrocardiogram signals,” _Elsevier Information Sciences_ , 2016.
  * [9] T. Li and M. Zhou, “ECG classification using wavelet packet entropy and random forests,” _MDPI Entropy_ , 2016.
  * [10] J. Park and K. Kang, “PcHD: Personalized classification of heartbeat types using a decision tree,” _Elsevier Computers in biology and medicine_ , 2014\.
  * [11] R. J. Martis, U. R. Acharya, H. Adeli, H. Prasad, J. H. Tan _et al._ , “Computer aided diagnosis of atrial arrhythmia using dimensionality reduction methods on transform domain representation,” _Elsevier Biomedical Signal Processing and Control_ , 2014.
  * [12] S. A. Shufni and M. Y. Mashor, “ECG signals classification based on discrete wavelet transform, time domain and frequency domain features,” in _IEEE Conference on Biomedical Engineering (ICoBE)_ , 2015.
  * [13] J. Pan and W. J. Tompkins, “A real-time QRS detection algorithm,” _IEEE transactions on biomedical engineering_ , 1985.
  * [14] J. Bergstra _et al._ , “Theano: Deep learning on GPUs with python,” in _NIPS BigLearning Workshop_ , 2011.
  * [15] G. Indiveri, F. Corradi, and N. Qiao, “Neuromorphic architectures for spiking deep neural networks,” in _IEEE IEDM_. IEEE, 2015, pp. 4–2.
  * [16] M. Konijnenburg, S. Stanzione _et al._ , “28.4 a battery-powered efficient multi-sensor acquisition system with simultaneous ECG, BIO-Z, GSR, and PPG,” in _IEEE ISSCC_ , 2016.
  * [17] S. Gradl, P. Kugler, C. Lohmüller, and B. Eskofier, “Real-time ECG monitoring and arrhythmia detection using Android-based mobile devices,” in _IEEE EMBC_ , 2012.
  * [18] L. B. Almeida, “The fractional fourier transform and time-frequency representations,” _IEEE Transactions on signal processing_ , 1994.
  * [19] D. L. Donoho, M. Elad, and V. N. Temlyakov, “Stable recovery of sparse overcomplete representations in the presence of noise,” _IEEE Transactions on information theory_ , 2006.
  * [20] D. Williams and G. Hinton, “Learning representations by back-propagating errors,” _Nature_ , 1986.
  * [21] S. Purdy, “Encoding data for HTM systems,” _arXiv preprint arXiv:1602.05925_ , 2016.
  * [22] V. Behravan, N. E. Glover, R. Farry, P. Y. Chiang, and M. Shoaib, “Rate-adaptive compressed-sensing and sparsity variance of biomedical signals,” in _IEEE BSN_ , 2015.
  * [23] S. Abbaspour, M. Lindén, and H. Gholamhosseini, “Ecg artifact removal from surface emg signal using an automated method based on wavelet-ica,” in _International conference on wearable micro and nanotechnologies for personalized health. Sweden: Västerås_ , 2015, pp. 91–7.


# Compound Word Transformer: Learning to Compose Full-Song Music
over Dynamic Directed Hypergraphs

Wen-Yi Hsiao,1 Jen-Yu Liu,1, Yin-Cheng Yeh,1 Yi-Hsuan Yang1, 2

###### Abstract

To apply neural sequence models such as the Transformers to music generation
tasks, one has to represent a piece of music by a sequence of tokens drawn
from a finite set of pre-defined vocabulary. Such a vocabulary usually
involves tokens of various _types_. For example, to describe a musical note,
one needs separate tokens to indicate the note’s pitch, duration, velocity
(dynamics), and placement (onset time) along the time grid. While different
types of tokens may possess different properties, existing models usually
treat them equally, in the same way as modeling words in natural languages. In
this paper, we present a conceptually different approach that explicitly takes
into account the type of the tokens, such as _note_ types and _metric_ types.
And, we propose a new Transformer decoder architecture that uses different
feed-forward heads to model tokens of different types. With an expansion-
compression trick, we convert a piece of music to a sequence of _compound
words_ by grouping neighboring tokens, greatly reducing the length of the
token sequences. We show that the resulting model can be viewed as a learner
over dynamic directed hypergraphs. And, we employ it to learn to compose
expressive Pop piano music of full-song length (involving up to 10K individual
tokens per song), both conditionally and unconditionally. Our experiment shows
that, compared to state-of-the-art models, the proposed model converges 5–10
times faster at training (i.e., within a day on a single GPU with 11 GB
memory), and with comparable quality in the generated music.

## Introduction

To apply neural sequence models such as recurrent neural networks (RNNs) or
Transformers (Vaswani et al. 2017) to automatic music composition (a.k.a.,
symbolic-domain music generation), one has to represent a piece of music as a
sequence of tokens drawn from a pre-defined _vocabulary_ (Oore et al. 2018).
Unlike the case in text, such a vocabulary usually involves tokens of various
_types_. For example, to represent a musical score, we may need tokens that
describe the content of the musical notes (e.g., pitch and duration), their
placement along time, the instrument that plays each note, as well as
indicators of metrical events such as the beginning of a new beat, bar
(measure), or musical phrase (Wu and Yang 2020). We need such a diverse set of
tokens as music is multifaceted; a type alone captures only a certain aspect
of music (e.g., melody, harmony, rhythm, timbre) and cannot faithfully
represent a music piece.

Figure 1: Illustration of the main ideas of the proposed compound word
Transformer: (left) _compound word modeling_ that combines the embeddings
(colored gray) of multiple tokens $\\{w_{t-1,k}\\}_{k=1}^{K}$, one for each
token type $k$, at each time step $t-1$ to form the input
$\vec{\mathbf{x}}_{t-1}$ to the self-attention layers, and (right) _toke type-
specific feed-forward heads_ that predict the list of tokens for the next time
step $t$ at once at the output.

As different types of (musical) tokens may have different properties, modeling
the dependency of these tokens might not be the same as modeling words in
text. However, to our best knowledge, little work has been done to explicitly
account for the heterogeneity of tokens in music. The tokens are mostly
treated equally, in the same way as words in text (Huang et al. 2019; Payne
2019; Huang and Yang 2020).

| Representation | Model | Attn. window | Voc. size | Data type
---|---|---|---|---|---
Music Transformer (Huang et al. 2019) | MIDI-like | Transformer | 2,048 | 388 | Classical piano performance
MuseNet (Payne 2019) | MIDI-like* | Transformer | 4,096 | N/A | Multi-track MIDI
LakhNES (Donahue et al. 2019) | MIDI-like* | Transformer-XL | 512 | 630 | Multi-track MIDI
TR autoencoder (Choi et al. 2020) | MIDI-like | Transformer | 2,048 | 388 | Classical piano performance
Pop Music TR (Huang and Yang 2020) | REMI | Transformer-XL | 512 | 332 | Pop piano performance
Transformer VAE (Jiang et al. 2020) | MIDI-like | Transformer | 128 | 47 | Pop lead sheets
Guitar Transformer (Chen et al. 2020) | REMI* | Transformer-XL | 512 | 221 | Guitar tabs
Jazz Transformer (Wu and Yang 2020) | REMI* | Transformer-XL | 512 | 451 | Jazz lead sheets
MMM (Ens and Pasquier 2020) | MIDI-like* | Transformer | 2,048 |  $>$442 | Multi-track MIDI
This work | CP | linear Transformer | 5,120 | 350 | Pop piano performance
Table 1: A comparison of existing Transformer-based models and the proposed
one for automatic music composition. The representations marked with * are
extensions of either MIDI-like (Oore et al. 2018) or REMI (Huang and Yang
2020).

We are therefore motivated to study in this paper whether we can improve
sequence modeling of music by highlighting the role of token types. Our first
proposal is to _customize the prediction heads for tokens of different types_.
Specifically, using the Transformer as the main architecture of the underlying
sequence model, we approach this by using different feed-forward heads for
tokens of different types.

Our second proposal is to group consecutive and related tokens in a token
sequence into “compound words,” and then _perform sequence modeling over_ the
resulting _sequence of compound words_. This is to capture the co-occurrence
relationship of tokens—e.g., to generate a new musical note, we may need at
least two consecutive tokens to indicate its pitch and duration; to change the
tempo in the middle of a piece of music, we need a token to indicate the
target tempo value, and an co-occurring time-related token to indicate the
time of the tempo change. Under the proposed compound-word modeling, the
individual tokens (e.g., pitch and duration) are still predicted separately
with different heads. Yet, instead of predicting them at different time steps,
we predict multiple tokens of various types _at once_ in a single time step.
The token embeddings of the tokens predicted at the current step are then
combined and fed as the input for the next time step. Namely, the self-
attention is computed over combined embeddings of individual tokens of a
compound word.

From a theoretical point of view, the proposed model can be interpreted as a
learner over discrete-time _dynamic directed hypergraphs_ (Kazemi et al.
2020). Here, a graph consists of nodes that each corresponds to a token in our
vocabulary. A sequence of tokens can then be viewed as a sequence of edges
(each connecting two nodes), or a _walk_ , over this graph. A sequence of
compound words, in contrast, can be viewed as a sequence of _hyperedges_ (each
connecting multiple nodes) (Feng et al. 2019), over the same graph. We discuss
this at greater length later in the paper.

We refer to the proposed representation as the _c ompound word
representation_, or CP for short. CP can be considered as an extension of
existing representations, with the following additional merits. First, it
allows for fine-grained, type-specific control over the prediction heads. For
example, we can now use different loss functions, sampling policies, and token
embedding sizes for different token types.

Second, as a compound word represents multiple tokens at once, it requires
much less time steps to generate a music piece using compound words. Namely,
the sequence length of the same music piece is much shorter in CP than in
existing representations. As the computational complexity of a Transformer is
related to the sequence length (Vaswani et al. 2017), this makes training and
inference faster, and may facilitate learning the long-range dependency in
music.111For example, we can study whether the proposed model creates music
with better “structureness,” or long-term repetitions (Wu and Yang 2020;
Jhamtani and Berg-Kirkpatrick 2019) in the future.

Finally, the sequence length in CP is determined by the number of compound
words in a sequence, not by the number of individual tokens per compound word.
Therefore, it is possible to add new token types (by adding the corresponding
feed-forward head) to increase the expressivity of the representation, without
increasing the sequence length. This makes it easy to extend to underlying
representation, though we do not explore this potential in this work.

For performance study, we consider generating expressive Pop piano music at
full-song scale in both the unconditional setting (i.e., from scratch) and
conditional setting (i.e., generating the piano arrangement given the lead
sheet). This involves modeling fairly long music sequences for up to 10K
individual tokens each. We show that, with CP, we are able to train a linear
Transformer decoder (Katharopoulos et al. 2020) with music quality similar to
that of strong baselines, with faster training and inference time. We provide
audio examples and open source the project at a GitHub
repo.222https://github.com/YatingMusic/compound-word-transformer

## Related Work

Both language and music have principles governing the organization of discrete
structural elements (e.g., words or musical notes) into sequences (Patel
2003). As such, the Transformers, which have been firstly shown to work well
for text generation (Child et al. 2019; Keskar et al. 2019), have been
increasingly applied to music generation in recent years, by treating music
pieces as sequences of discrete tokens akin to text words. We list some
related papers in Table 1.

Table 1 shows that most existing work adopt a music representation derived
from either MIDI-like (Oore et al. 2018) or REMI (Huang and Yang 2020), with
possible addition of track- or structure-related tokens. MIDI-like and REMI
differ mainly in how the advance of time is represented: the former uses
[time_shift] tokens to mark the time interval (in absolute time) between note-
related tokens, whereas the latter assumes symbolic timing and uses [bar] and
[position] tokens to place tokens on a metrical grid that uniformly divides a
bar into a certain number of positions. Neither MIDI-like nor REMI groups the
tokens by token types.333Upon paper completion, we noticed an early but
preliminary attempt of grouping tokens by (Hawthorne et al. 2018b).

Existing work also differs in the length of the _attention window_ (see the
methodology section for definition) and vocabulary size (which is data- and
task-dependent). To our knowledge, our work represents the first one to
consider Pop music modeling at full-song scale (involving 10k tokens per
song), and to use the recently-proposed linear Transformer (Katharopoulos et
al. 2020) as the model backbone.

## Methodology

### Background

For sequence modeling, we need a conversion function $g(\cdot)$ that converts
a music piece $\mathcal{X}$ to a time-ordered sequence of symbolic elements
$\mathcal{S}=g(\mathcal{X})=\\{w_{1},w_{2},\dots,w_{T}\\}$, where $T$ denotes
the resulting sequence length. Given a number of such sequences, we train a
neural sequence model with an architecture such as the Transformer decoder to
learn to generate new sequences $\mathcal{S}^{\prime}$. We then use a
deterministic inverse function $g^{-1}(\cdot)$ to get a new music piece from
such a generated sequence, namely
$\mathcal{X}^{\prime}=g^{-1}(\mathcal{S}^{\prime})$. There can be different
algorithms to implement the conversion function and its inverse, leading to
numerous possible sequence representations of the same music piece, e.g.,
$\mathcal{S}_{\text{MIDI-like}}=g_{\text{MIDI-like}}(\mathcal{X})$ and
$\mathcal{S}_{\text{REMI}}=g_{\text{REMI}}(\mathcal{X})$. Different conversion
functions (or sequence representations) assume different vocabulary sizes $M$,
so $\mathcal{S}_{\text{MIDI-like}}$ and $\mathcal{S}_{\text{REMI}}$ differ in
both $T$ and $M$.

A Transformer decoder comprises a stack of _self-attention_ layers and a stack
of _feed-forward_ layers. The self-attention layers operate on a fixed-length
sub-sequence of $\mathcal{S}$ to learn the dependency among the elements. The
length of such a sub-sequence, a.k.a., the _attention window_ , denoted as
$N$, is usually much smaller than $T$, as $N$ directly affects the space
complexity of the model. For the vanilla Transformer (Vaswani et al. 2017) and
its faster variant Transformer-XL (Dai et al. 2019), it is
$\mathcal{O}(N^{2}M)$; for the linear Transformer (Katharopoulos et al. 2020),
it is $\mathcal{O}(NM)$.

### Individual Tokens vs Compound Words

In this paper, we refer to the elements in either $\mathcal{S}_{\text{MIDI-
like}}$ or $\mathcal{S}_{\text{REMI}}$ as the _individual tokens_. They are
drawn from a pre-defined vocabulary $\mathcal{V}=\\{1,\dots,M\\}$. As
mentioned in the introduction, each token is associated with a _type_ defined
in the type set, $\mathcal{K}=\\{1,\dots,K\\}$. We can partition $\mathcal{V}$
into $K$ subsets by token group, i.e., $\\{\mathcal{V}_{k}\\}_{k=1}^{K}$.

We propose to convert a sequence of tokens (e.g., $\mathcal{S}_{\text{REMI}}$)
into a sequence of compound words $\mathcal{S}_{\text{CP}}$ with the following
procedure. First, _neighboring tokens that define a musical event together are
grouped into a super token_ , i.e., placed on the same time step, as
illustrated in Figures 2(a)–(b). A musical event here can be a _note_ related
one, i.e., to create a new musical note, or a _metrical_ related one, e.g., to
mark the beginning of a new beat, or a new bar. For example, in REMI, a note
is created by consecutive tokens of [pitch], [duration], and [velocity], which
are grouped in CP. And, a tempo or chord change in REMI takes place only at
beat times, so we also group [beat], [chord] and [tempo]. Accordingly, the
model has to make multiple predictions (i.e., generate multiple tokens) at
each time step.

(a) REMI representation

(b) Tokens grouped (c) Compound words

Figure 2: An example illustrating the conversion from a sequence of REMI
tokens (Huang and Yang 2020) into a (shorter) sequence of compound words. A
compound word comprises a number of grouped tokens and the [ignore] tokens,
which are colored white in (c), as well as a family token (N: note-related or
M: metric-related). Best seen in color.

Second, we _fill the missing token types per time step with “[ignore]” tokens_
, so that at each step there are consistently $K$ tokens to be predicted, as
illustrated in Figure 2(c). This is to make computational modeling feasible,
as otherwise the shape and meaning of the target output at each time step
would be uncertain. In other words, a _compound word_ is composed of a list of
$K$ tokens, each drawn from the corresponding subset
$\mathcal{V}_{k}\cup\text{[ignore]}$, that are placed on the same time step
$t$. Formally,
$\mathcal{S}_{\text{CP}}=g_{\text{CP}}(\mathcal{X})=\\{cp_{t}\\}_{t=1}^{T_{\text{cp}}}$,
in which $cp_{t}=\\{w_{t,1},\cdots,w_{t,K}\\}$. We view this conversion
function $g_{\text{CP}}(\cdot)$ as performing an _expansion-compression trick_
, as the original sequence is firstly expanded to a sequence of
$KT_{\text{CP}}$ individual tokens, and then compressed to a sequence of
$T_{\text{CP}}$ compound words; in general
$T_{\text{CP}}<T_{\text{REMI}}<KT_{\text{CP}}$.

To facilitate modeling the CP, we further partition the type set $\mathcal{K}$
into $F$ _families_. For example, if $\mathcal{K}$ can be partitioned into two
families, the _note_ family $\mathcal{K}_{\texttt{N}}$ and _metric_ family
$\mathcal{K}_{\texttt{M}}$ (marked as ‘N’ and ‘M’ in Figure 2(c)), we would
have $\mathcal{K}=\mathcal{K}_{\texttt{N}}\cup\mathcal{K}_{\texttt{M}}$, and
$\mathcal{K}_{\texttt{N}}\cap\mathcal{K}_{\texttt{M}}=\emptyset$. Each
compound word $cp_{t}$ is associated with a _family token_ $f_{t}$. For a
metric-related $cp_{t}$, we would have $w_{t,k}=\text{[ignore]}$, for
$k\in\mathcal{K}_{\texttt{N}}$. Similarly, for a note-related $cp_{t}$,
$w_{t,k}=\text{[ignore]}$, for $k\in\mathcal{K}_{\texttt{M}}$.

### Combining Token Embeddings of Adaptive Sizes

As input to Transformers, an element in a sequence is represented by an
_embedding_ vector, $\mathbf{x}_{t}\in\mathcal{R}^{d}$, and then added with a
positional embedding vector (Ke, He, and Liu 2020). In CP, we propose to form
an embedding vector for a compound word $cp_{t}$ by combining the embedding
vectors $\mathbf{p}_{t,k}$ of the composing tokens $w_{t,k}$, as well as an
embedding vector $\mathbf{q}_{t}$ associated with the family token $f_{t}$.
Specifically, we combine the vectors by firstly concatenating them, and then
linearly projecting the resulting long vector to a $d$-dimensional vector with
a projection matrix $\mathbf{W}_{\text{in}}$. Namely,

$\begin{split}\mathbf{p}_{t,k}&=\text{Embedding}_{k}(w_{t,k})\,,\;k=1,...,K\,,\\\
\mathbf{q}_{t}&=\text{Embedding}_{\mathcal{F}}(f_{t})\,,\\\
\mathbf{x}_{t}&=\mathbf{W}_{\text{in}}\,[\mathbf{p}_{t,1}\oplus...\oplus\mathbf{p}_{t,K}\oplus\mathbf{q}_{t}]\,,\\\
\vec{\mathbf{x}}_{t}&=\text{Positional
Encoding}(\mathbf{x}_{t})\,,\end{split}$ (1)

where $\oplus$ denotes vector concatenation, and $\text{Embedding}_{k}(\cdot)$
and $\text{Embedding}_{\mathcal{F}}(\cdot)$ involve the use of lookup tables.

In essence, $\mathbf{x}_{t}$ can be considered as a _compressive_
representation of the composing tokens $w_{t,k}$ and family token $f_{t}$. We
note the action of compressing the embeddings is reminiscent of the main idea
of the Compressive Transformer (Rae et al. 2020), which proposes to compresses
past memories beyond the attention window for long-range sequence learning.
Unlike it, we compress the memories _within_ the attention window defined over
the individual tokens.

A main merit of CP is that we can customize the settings for different token
types. Being inspired by the _adaptive word representation_ (Baevski and Auli
2018), we use different embedding sizes $d_{k}$ for tokens of different types,
i.e., $\mathbf{p}_{t,k}\in\mathcal{R}^{d_{k}}$. We basically use larger
$d_{k}$ for token types with larger vocabulary size $|\mathcal{V}_{k}|$. See
Table 3 for details.

### Multi-head Output Module

A main proposal of our work is to use different feed-forward heads for tokens
of different types in a Transformer. Specifically, we have $(K+1)$ heads in
total, one for each token type $\mathcal{V}_{k}$ and an additional one for the
token family $\mathcal{F}$.

Instead of working on the $K+1$ heads at the same time, we devise a _two-
stage_ setting that predicts the family token first, and then the remaining
tokens given the family token. Specifically, at the $t$-th time step, the
feed-forward procedure can be summarized as:

$\begin{split}\mathbf{h}_{t}&=\text{Self-
attn}\left(\vec{\mathbf{x}}_{t-1}\right)\,,\\\
\widehat{f_{t}}&=\text{Sample}_{\mathcal{F}}\left(\text{softmax}(\mathbf{W}_{\mathcal{F}}\mathbf{h}_{t})\right)\,,~{}~{}~{}~{}~{}\\\
\mathbf{h}_{t}^{\text{out}}&=\mathbf{W}_{\text{out}}\,[\mathbf{h}_{t}\oplus\text{Embedding}_{\mathcal{F}}(\widehat{f_{t}})]\,,~{}~{}~{}~{}\\\
\widehat{w_{t,k}}&=\text{Sample}_{k}\left(\text{softmax}(\mathbf{W}_{k}\mathbf{h}_{t}^{\text{out}})\right)\,,\;k=1,...,K\,,\end{split}$
(2)

where $\mathbf{W}_{\mathcal{F}}$ and $\\{\mathbf{W}_{k}\\}_{k=1}^{K}$ are the
$K+1$ feed-forward heads, $\text{Self-attn}(\cdot)$ the causal self-attention
layers, and $\text{Sample}(\cdot)$ a sampling function. We empirically find
that this two-stage setting makes it easier for the model to predict
$w_{t,k}=\text{[ignore]}$, for $k$ not in the target family
$\mathcal{K}_{\widehat{f_{t}}}$.

Figure 1 illustrates Eqs. (1)–(2) in work, omitting the first-stage part at
the output for $\widehat{f_{t}}$ due to space limit.

### Adaptive Sampling Policy

At inference time, we use stochastic temperature-controlled sampling (Holtzman
et al. 2020) to avoid degeneration and to increase diversity. With CP, we
employ different sampling policies $\text{Sample}_{k}(\cdot)$ for different
token types; see Table 3.

## Graph Interpretation

We discuss the proposed model from a graph-theoretical point of view below.
Given a vocabulary of tokens, we can construct a fully-connected _static
graph_ $\mathcal{G}=(\mathcal{V},\mathcal{E})$ (Kivelä et al. 2014) comprising
nodes $\mathcal{V}=\\{1,\dots,M\\}$ and edges
$\mathcal{E}=\mathcal{V}\times\mathcal{V}$. Each node corresponds to an
individual token in our vocabulary. This way, a token sequence
$\mathcal{S}_{\text{MIDI-like}}$ or $\mathcal{S}_{\text{REMI}}$ can be viewed
as a sequence of edges (each connecting two nodes), or a _walk_ , over this
graph.

In CP, the vocabulary (and accordingly the graph) is augmented with a set of
_special tokens_ , denoted as $\mathcal{V}^{*}$, that includes for example
type-specific [ignore] tokens and family tokens. And, a compound word consists
of $K+1$ nodes, one from each of the $K$ types and an additional one from the
set of family tokens. A sequence of compound words, namely
$\mathcal{S}_{\text{CP}}$, therefore, involves transitions from $K+1$ nodes to
another $K+1$ nodes per time step. Such a transition can be viewed as a
directed _hyperedge_ (Feng et al. 2019; Jiang et al. 2019), that connects at
once $K+1$ source nodes (e.g., $cp_{t-1}$) to $K+1$ target nodes ($cp_{t}$).
It is directed because the order of the nodes matters (i.e., from $t-1$ to
$t$).

A sequence of compound words also forms a _dynamic directed hypergraph_
(Kazemi et al. 2020):
$\\{\mathcal{G}_{1},\mathcal{G}_{2},\dots,\mathcal{G}_{T}\\}$, where
$\mathcal{G}_{t}=(\mathcal{V},\mathcal{E}_{t})$. Starting from an empty graph
with no edges, at each time step $t>1$ we add a new directed hyperedge,
labeled with the time step $t$, connecting in total $2K+2$ nodes. In practice,
we have a [BOS] token (beginning of sequence) and [EOS] token (end of
sequence), so the hyperedge at $t=1$ and $t=T$ connects to only $K+2$ nodes.

A neural model for graphs, or a _graph neural network_ (GNN), can be regarded
as an encoder-decoder pair (Kazemi et al. 2020; Rossi et al. 2020), where an
_encoder_ is a function that maps from a graph $\mathcal{G}$ to node
embeddings $\mathbf{z}_{i},i=1\dots M$, and a _decoder_ takes as input one ore
more node embeddings and makes a prediction based on these, e.g., node
classification or edge prediction. The proposed CP Transformer can therefore
be regarded as a learner over dynamic directed hypergraphs, as at each time
step $t$ it manages to predict the next hyperedge to be added (i.e.,
$\widehat{w_{t,k}}$ and $\widehat{f_{t}}$) based on the node embeddings
updated from
$\mathcal{G}_{<t}=\\{\mathcal{G}_{1},\mathcal{G}_{2},\dots,\mathcal{G}_{t-1}\\}$,
or the collection of input embeddings
$\mathbf{x}_{<t}=\\{\mathbf{x}_{1},\mathbf{x}_{2},\dots,\mathbf{x}_{t-1}\\}$
marked with positional embeddings (i.e., edge labels on the directed
hyperedges).

We note that, while we introduce the proposed methods in the context of music
modeling, the idea of compound words is generic and may be applicable to
sequences seen in other data domains, when multiple tokens (i.e., a hyperedge)
are needed to represent a single event, entity, or object.

## Implementation

To test the effectiveness of the proposed methods, we implement a CP
Transformer that learns to generate Pop piano music with human performance
characteristics such as expressive variations in velocity (i.e., the force
with which a note is played, which is related to loudness) and tempo (Oore et
al. 2018; Lerch et al. 2019). We consider Pop piano for its richness and
expressivity, and for offering a direct performance comparison with the Pop
Music Transformer (Huang and Yang 2020) (see Table 1).

Specifically, we consider both the conditional and unconditional generation
tasks. In the former, a _lead sheet_ (i.e., a melody line and an accompanying
sequence of chord labels) is given, and the model has to generate a piano
performance according to that. In the latter, the model generates a piano
performance of full-song length from scratch freely.

We intend to compare CP with REMI in our evaluation. We provide the
implementation details below.

Task | Repre. | #words ($T$)
---|---|---
mean ($\pm$ std) | max
Conditional | REMI | 6,432 ($\pm$ 1,689) | 10,240
CP | 3,142 ($\pm$ 821) | 5,120
Unconditional | REMI | 4,873 ($\pm$ 1,311) | 7,680
CP | 2,053 ($\pm$ 580) | 3,584
Table 2: Statistics of the number (#) of words (i.e., tokens in REMI; compound
words in CP) per song in the training set.

### Dataset

We collect the audio files of 1,748 pieces of Pop piano from the Internet. The
average length of the songs is about 4 minutes, and we have about 108 hours in
total. All the songs are in 4/4 time signature (four beats per bar). We
convert each song (an audio) into a symbolic sequence as follows.

  * •

Transcription: We use the state-of-the-art RNN model for automatic piano
transcription, “Onset and Frames” (Hawthorne et al. 2018a), to estimate the
pitch, onset and offset time, and velocity of the musical notes from audio.

  * •

Synchronization: To get symbolic timing from the original wall clock time, we
use the RNN-based model available in the Python package madmom (Böck et al.
2016) to estimate the downbeat and the beat positions, which represent the
state-of-the-art for the task. Then, we interpolate 480 ticks between two
adjacent beats, and map the absolute time into its according tick. By doing
so, we can keep tiny offset. Lastly, we infer the tempo changes from the time
interval between adjacent beats.

  * •

Quantization: We quantize the tempo, velocity, duration and the beat positions
to reduce the size of the vocabulary. For example, we set the 16-th note as
our basic time unit. See Table 3 for the number of tokens per type.

  * •

Analysis: For the conditional generation task, we estimate the melody notes
and chord symbols from the transcription result to form the lead sheets.
Specifically, we develop an in-house rule-based chord recognition
algorithm444https://github.com/joshuachang2311/chorder to recognize 12 roots
and 7 chord qualities. We use the “Skyline algorithm” (Uitdenbogerd and Zobel
1999) to extract the melodies. And, as a lead sheet is usually of coarser time
resolution, we quantize the chord symbols and melody notes to the 4-th notes
(i.e., beat times).

We randomly hold out 50 songs for testing, and use the remaining for training
the Transformers.

Repre. | Token type | Voc. size | Embed. | $\text{Sample}_{k}(\cdot)$
---|---|---|---|---
$|\mathcal{V}_{k}|$ | size ($d_{k}$) | $\tau$ | $\rho$
CP | [track] | 2 (+1) | 3 | 1.0 | 0.90
2-6[1pt/1pt] | [tempo] | 58 (+2) | 128 | 1.2 | 0.90
| [position/bar] | 17 (+1) | 64 | 1.2 | 1.00
| [chord] | 133 (+2) | 256 | 1.0 | 0.99
2-6[1pt/1pt] | [pitch] | 86 (+1) | 512 | 1.0 | 0.90
| [duration] | 17 (+1) | 128 | 2.0 | 0.90
| [velocity] | 24 (+1) | 128 | 5.0 | 1.00
2-6[1pt/1pt] | [family] | 4 | 32 | 1.0 | 0.90
2-6[1pt/1pt] | total | 341 (+9) | — | — | —
REMI | total | 338 | 512 | 1.2 | 0.90
Table 3: Details of the CP representation in our implementation, including
that of the sampling policy ($\tau$-tempered top-$\rho$ sampling). For the
vocabulary size, the values in the parentheses denote the number of special
tokens such as [ignore].

### Vocabulary

To represent the content of a piano performance, the basic setting employs
tokens of six types: three note-related types [pitch], [duration], [velocity],
and three metric-related types [position/bar], [tempo], [chord]. The specific
vocabulary is task-dependent and is introduced below.

_Conditional generation_ —We additionally use [track] tokens to mark whether
it is the _lead sheet_ track (i.e., the condition) or the _piano_ track (the
track to be generated). While the piano track (i.e., the sub-sequence after
the [track=piano] token) involves all the six types of tokens mentioned above,
the lead sheet track only involves the use of composition-related tokens
[position/bar], [chord], [pitch], [duration], not performance-related tokens
[velocity], [tempo]. In CP, we have three family tokens, [family=track],
[family=note], [family=metric]. Moreover, we have type-specific [ignore]
tokens and an additional [conti] token for the beat positions having no tempo
or chord changes.

_Unconditional generation_ —This task only concerns with the piano track so we
do not need the [track] tokens. But, as it concerns with full-song generation,
we add an [EOS] token to signify the end of a sequence. We view it as a family
token, so there are three possible family tokens here: [family=EOS],
[family=note], [family=metric].

Details of the adopted representations are shown in Tables 2 and 3. Table 2
compares the sequence length $T$ of REMI and CP. We can see that
$\mathcal{S}_{\text{CP}}$ is much shorter than $\mathcal{S}_{\text{REMI}}$,
especially under the conditional task.555We set an upper limit of the number
of elements per sequence (e.g., 10,240 tokens in REMI) and remove overly long
songs, which amounts to removing 25–88 songs from the training set depending
on the task and the adopted representation. Table 3 displays the size of each
vocabulary subset $\mathcal{V}_{k}$. We see that CP and REMI have similar
total vocabulary size $M$. REMI does not use the family tokens (except for
[EOS]) and special tokens.

Task | Representation $+$ model@loss | Training | GPU | Inference (/song) | Matchness
---|---|---|---|---|---
time | memory | time (sec) | tokens (#) | melody | chord
Conditional | Training data | — | — | — | — | 0.755 | 0.838
Training data (randomized) | — | — | — | — | 0.049 | 0.239
2-8[1pt/1pt] | REMI $+$<EMAIL_ADDRESS>| 3 days | 4 GB | 88.4 | 4,782 | 0.872 | 0.785
| REMI $+$<EMAIL_ADDRESS>| 7 days | 4 GB | 91.5 | 4,890 | 0.866 | 0.800
2-8[1pt/1pt] | REMI $+$<EMAIL_ADDRESS>| 3 days | 17 GB | 48.9 | 4,327 | 0.779 | 0.709
| CP $+$<EMAIL_ADDRESS>| 0.6 days | 10 GB | 29.2 | 18,200 | 0.829 | 0.733
Unconditional | REMI $+$<EMAIL_ADDRESS>| 3 days | 4 GB | 139.9 | 7,680 | — | —
CP $+$<EMAIL_ADDRESS>| 1.3 days | 9.5 GB | 19.8 | 9,546 | — | —
Table 4: Quantitative evaluation result of different models. REMI$+$XL
represents a re-implementation of the state-of-the-art Pop Music Transformer
(Huang and Yang 2020), while CP$+$linear stands for the proposed CP
Transformer.

### Model Settings

For the backbone architecture of our model, we employ the linear Transformer
(Katharopoulos et al. 2020),666https://github.com/idiap/fast-transformers as
its complexity is a linear function of the length of the attention window $N$.
Moreover, we set $N$ equal to the sequence length $T$ for our model. That is,
_no segmentation_ over the training sequences is done, and thereby _all the
tokens in a sequence can be accessed_ by our model under causal masking,
without using tricks such as memory caching (Dai et al. 2019) or memory
compression (Rae et al. 2020). We refer to our model as CP$+$linear in what
follows.

For the baselines, we employ the Pop Music Transformer (Huang and Yang 2020),
which is open-source and stands for a state-of-the-art for unconditional music
composition.777https://github.com/YatingMusic/remi This REMI$+$XL model adopts
the REMI representation and uses Transformer-XL (Dai et al. 2019) as the model
backbone. As its complexity grows quadratically with $N$, we set $N=512$,
following (Huang and Yang 2020).

Moreover, we consider one more baseline that replaces Transformer-XL by linear
Transformer, using also $N=T$, to offer a sensible performance comparison
between CP and REMI. We refer to this variant as REMI$+$linear.

We use 12 self-attention layers each with 8 attention heads for all the models
for fair comparison. The model hidden size and inner layer of the feed-forward
part are set to 512 and 2,048, respectively. For the token embedding size $d$,
we fix it to 512 for REMI, following (Huang and Yang 2020). For CP, we set it
adaptively based on the vocabulary size of each token type, as shown in Table
3. For sampling, we employ the “nucleus sampling” (Holtzman et al. 2020), a
stochastic method that samples from the smallest subset of tokens whose
cumulative probability mass exceeds a threshold $\rho\in[0,1]$. Before
sampling, we reshape the probability distribution of the tokens (e.g.,
softmax($\mathbf{W}_{k}\mathbf{h}_{t}^{\text{out}}$)) through “temperature”
(Ackley, Hinton, and Sejnowski 1985), with the temperature parameter $\tau>0$.
As Table 3 also shows, we use different $\rho$ and $\tau$ for different token
types. For example, we use a large $\tau$ to encourage diverse velocity
values.

The conditional generation task can be approached with a sequence-to-sequence
model, since we have paired data of lead sheets and piano performances (i.e.,
the former is extracted automatically from the latter). Instead of adding a
Transformer encoder (as done in (Choi et al. 2020)) to realize this, we use
the encoder-free “Prefix LM” method of the Google’s “T5” model (Raffel et al.
2020), and run a single Transformer over an _interleaved_ sequence of lead
sheets and piano performances. Specifically, a sequence of lead sheet and the
corresponding target sequence of piano performance are integrated into one
sequence bar after bar. That is, the integrated sequence would have the form
of {$\dots$, [bar], [track=leadsheet], (content of the lead sheet for a bar),
[track=piano], (content of the piano for the same bar), [bar], (content of the
two tracks of the next bar) $\dots$}. This makes it easy to learn the
dependency of the two tracks, and to impose the pre-given lead sheet at
inference time.

## Quantitative Evaluation

The experiments hereafter are conducted in the interest of a resource-
constrained scenario, assuming that we only have a single GPU with 11 GB
memory and are only willing to train a model for 3 days. We conjecture that
this makes sense for most middle-size academic labs worldwide. Yet, to have an
idea of the model performance when more resources are available, we include to
the evaluation of the conditional task two settings exceeding such a
specification.

We firstly compare the efficiency of the models in terms of training time,
inference time, and GPU memory usage, under the conditional setting. The
average result over the 50 held-out test songs is shown in Table 4.

GPU memory usage. Table 4 shows that both CP$+$linear and REMI$+$XL require
$<$11 GB GPU memory for training. Accordingly, in our implementation, we train
them (separately) on an NVIDIA RTX 2080 Ti GPU (with 11GB memory). In
contrast, REMI$+$linear requires 17 GB GPU memory, so we train it on a TITAN
GPU with 24 GB memory.

Training time. We see that REMI-based models require much longer clock time to
reach a low training loss. While it takes nearly 7 days for REMI$+$XL to
reduce the negative log-likelihood (NLL) of the training data to 0.27, it
takes only 0.6 days for CP$+$linear to reach the same NLL. Such a training
efficiency is desirable (especially given that it is on a single 2080 Ti GPU),
as it makes further extensions and modifications of the model easy and
affordable.

Inference time. CP$+$linear is remarkably fast, taking on average $<$30
seconds to complete the conditional generation of a song. As a song in our
dataset is about 4 minutes, this is much faster than real time. In contrast,
REMI$+$XL and REMI$+$linear are about 3x and 1.7x slower, respectively.
CP$+$linear is fast for it generates in total 8 individual tokens (of
different types) at once each time step.

Table 4 also compares the efficiency of REMI$+$XL and CP$+$linear under the
unconditional setting, for which we generate also 50 songs (from scratch) and
report the average inference time. We see that CP$+$linear is even faster
here, requiring only $<$20 seconds to create a new song at full-song length.
In contrast, REMI$+$XL is on average 7x slower.

Repre. $+$ model@loss | F | R | H | C | O
---|---|---|---|---|---
REMI $+$<EMAIL_ADDRESS>| 4.05 | 3.12 | 3.38 | 3.55 | 3.31
REMI $+$<EMAIL_ADDRESS>| 4.29 | 3.14 | 3.70 | 3.64 | 3.35
REMI $+$<EMAIL_ADDRESS>| 4.03 | 3.09 | 3.48 | 3.46 | 3.29
CP $+$<EMAIL_ADDRESS>| 4.09 | 3.13 | 3.50 | 3.31 | 3.08

(a) Conditional generation

Repre. $+$ model@loss | R | H | S | O
---|---|---|---|---
REMI $+$<EMAIL_ADDRESS>| 3.11 | 3.46 | 2.91 | 3.03
CP $+$<EMAIL_ADDRESS>| 3.33 | 3.68 | 3.11 | 3.34

(b) Unconditional generation

Table 5: Result of subjective evaluation (Fidelity, Richness, Humanness,
Correctness, Structureness, Overall).

Next, we compare the performance of the models in terms of two objective
metrics, also under the conditional setting. As the goal is to generate a song
given a lead sheet, we can measure whether the generated song has a melody
line and chord progression similar to that in the given condition, and take
that as a figure of merit. (In contrast, proper objective evaluation of
unconditional generation models remains an open issue (Yang and Lerch 2020;
Dong et al. 2020; Wu and Yang 2020).) Specifically, we consider:

  * •

Melody matchness. We represent the lead sheet and the correspondingly
generated piano both in the REMI format and compute the bar-wise _longest
common sub-sequence_ (LCS) of the two resulting sequences
$\mathcal{S}_{\text{REMI}}^{\text{LS}}$ and
$\widehat{\mathcal{S}}_{\text{REMI}}^{\text{piano}}$. When two notes (each
from the two sequences) have the same pitch and close onset time (within the
8-th note), we consider that as a match. We divide the length of the LCS by
the number of [pitch] tokens in $\mathcal{S}_{\text{REMI}}^{\text{LS}}$ (i.e.,
the number of target melody notes) of that bar, and take the average value of
such a ratio across all the bars of a song as a simple measure of melody
matchness.

  * •

Chord matchness. The _chroma vector_ (Fujishima 1999) represents a short-time
fragment of music by the distribution of energy across the 12 pitch classes
(C, C#, etc) and offers a simple way to evaluate the harmonic similarity
between two fragments. We calculate the segment-wise cosine similarity between
the chroma vector representing each chord label of a lead sheet (which would
be binary-valued) and the chroma vector of the correspondingly generated piano
segment (normalized by the maximum value so it is $\in[0,1]^{12}$), and treat
the average value across time as a measure of chord matchenss.

Table 4 shows that the evaluated models all have matchness close to that of
the training set, and much higher than that of the random baseline (i.e., the
average matchness between a lead sheet and a random song from the test set).
This suggests, while CP$+$linear is easier and faster to train than REMI$+$XL,
they may generate music of similar quality. We further investigate this
through a user study, which directly assesses the perceptual quality of the
generated music.

(a) REMI$+$XL  (b) CP$+$linear

Figure 3: Piano-rolls of middle 64 bars of random generated pieces of two
models in the unconditional setting. We see richer and diverse content in the
result of CP$+$linear.

## Qualitative Evaluation

We devise an online questionnaire that solicits anonymous response to the
music generated by different models for both the conditional and unconditional
settings. For the former, we present excerpts of 32 bars taking from one-third
location of the music. For the latter, we present the full songs (i.e., when
an [EOS] token is generated).888It turns out that the REMI$+$XL model seldom
generates [EOS] tokens even when the music is already quite long (e.g., 8
minutes), so we stop it each time when it has generated 7,680 tokens. Our
intention is to investigate whether CP$+$linear and REMI$+$XL indeed generate
music of similar perceptual qualities.

The generated music is rendered into audio with a piano synthesizer using a
free, non-professional grade sound font. Each batch comprises the result of
the evaluated models in random order. A subject has to rate the music for
three random batches for each setting separately, in terms of the following
aspects on a five-point Likert scale. 1) Fidelity: is the conditionally
generated piece similar to the reference, from which the condition lead sheet
was taken from? 2) Richness: diversity and interestingness. 3) Humanness: does
the piece sound like expressive human performances? 4) Correctness: perceived
absence of composing or playing mistakes. 5) Structureness: whether there are
structural patterns such as repeating themes or development of musical ideas.
6) Overall. As the music can be long, the questionnaire may take around 30
mins to complete.

Table 5 shows the average result from 18 subjects. We see that REMI$+$XL
performs the best in the conditional setting, yet with only moderate
performance gap between the models.999In the conditional setting, the global
structure of the song to be generated is fairly outlined in the given
condition (i.e., the melody). Thus, it seems sufficient for models to learn
from short segments. In contrast, CP$+$linear performs (slightly) better
consistently across the four metrics in the unconditional setting, suggesting
it a powerful alternative to REMI$+$XL.

## Conclusion

In this paper, we have presented a new variant of the Transformer that
processes multiple consecutive tokens at once at a time step. Each individual
token is associated with a token type, which is exploited by the model to
customize its input and output modules. The proposed model achieves sequence
compression by integrating the embeddings of the tokens, which can be seen as
forming a hyperedge over a dynamic graph. We show that the new Transformer
works remarkably well for modeling music, creating full-song piano of
comparable perceived quality with a competing Transformer-XL based model in
much shorter training and inference time.

## Ethics Statement

Research on automatic music generation may infringe copyright laws and may
raise concerns regarding the role of human musicians in the future. Cares have
to be given regarding the fair use of existing musical material for model
training, and the potential concern of “deepfaking” an existing artist’s style
in computer-generated music.

## Acknowledgement

We are grateful to our interns at the Taiwan AI Labs, Joshua Chang for
developing the symbolic-domain chord recognition algorithm, and Yu-Hua Chen
and Hsiao-Tzu Hung for helping organize the PyTorch code. We also thank the
anonymous reviewers for their valuable comments.

## References

  * Ackley, Hinton, and Sejnowski (1985) Ackley, D. H.; Hinton, G. E.; and Sejnowski, T. J. 1985.  A learning algorithm for Boltzmann machines.  _Cognitive Science_ 9(1): 147–169.
  * Baevski and Auli (2018) Baevski, A.; and Auli, M. 2018.  Adaptive input representations for neural language modeling.  _arXiv preprint arXiv:1809.10853_ .
  * Böck et al. (2016) Böck, S.; Korzeniowski, F.; Schlüter, J.; Krebs, F.; and Widmer, G. 2016\.  Madmom: A new Python audio and music signal processing library.  In _Proc. ACM Multimedia_ , 1174–1178.
  * Chen et al. (2020) Chen, Y.-H.; Huang, Y.-S.; Hsiao, W.-Y.; and Yang, Y.-H. 2020.  Automatic composition of guitar tabs by Transformers and groove modeling.  In _Proc. Int. Soc. Music Information Retrieval Conf._
  * Child et al. (2019) Child, R.; Gray, S.; Radford, A.; and Sutskever, I. 2019.  Generating long sequences with sparse Transformers.  _arXiv preprint arXiv:1904.10509_ .
  * Choi et al. (2020) Choi, K.; Hawthorne, C.; Simon, I.; Dinculescu, M.; and Engel, J. 2020.  Encoding musical style with transformer autoencoders.  In _Proc. Int. Conf. Machine Learning_.
  * Dai et al. (2019) Dai, Z.; Yang, Z.; Yang, Y.; Carbonell, J.; Le, Q.; and Salakhutdinov, R. 2019.  Transformer-XL: Attentive language models beyond a fixed-Length context.  In _Proc. Annual Meeting of the Association for Computational Linguistics_ , 2978–2988.
  * Donahue et al. (2019) Donahue, C.; Mao, H. H.; Li, Y. E.; Cottrell, G. W.; and McAuley, J. 2019.  LakhNES: Improving multi-instrumental music generation with cross-domain pre-training.  In _Proc. Int. Soc. Music Information Retrieval Conf._ , 685–692.
  * Dong et al. (2020) Dong, H.-W.; Chen, K.; McAuley, J.; and Berg-Kirkpatrick, T. 2020.  MusPy: A toolkit for symbolic music generation.  In _Proc. Int. Soc. Music Information Retrieval Conf._
  * Ens and Pasquier (2020) Ens, J.; and Pasquier, P. 2020.  MMM- Exploring conditional multi-track music generation with the Transformer.  _arXiv preprint arXiv:2008.06048_ .
  * Feng et al. (2019) Feng, Y.; You, H.; Zhang, Z.; Ji, R.; and Gao, Y. 2019.  Hypergraph neural networks.  In _Proc. AAAI_ , 3558–3565.
  * Fujishima (1999) Fujishima, T. 1999.  Realtime chord recognition of musical sound: A system using common Lisp.  In _Proc. International Computer Music Conf._ , 464–467.
  * Hawthorne et al. (2018a) Hawthorne, C.; Elsen, E.; Song, J.; Roberts, A.; Simon, I.; Raffel, C.; Engel, J.; Oore, S.; and Eck, D. 2018a.  Onsets and Frames: Dual-objective piano transcription.  In _Proc. Int. Soc. Music Information Retrieval Conf._ , 50–57.
  * Hawthorne et al. (2018b) Hawthorne, C.; Huang, A.; Ippolito, D.; and Eck, D. 2018b.  Transformer-NADE for piano performances.  In _Proc. Machine Learning for Creativity and Design Workshop_.
  * Holtzman et al. (2020) Holtzman, A.; Buys, J.; Du, L.; Forbes, M.; and Choi, Y. 2020.  The curious case of neural text degeneration.  In _Proc. Int. Conf. Learning Representations_.
  * Huang et al. (2019) Huang, C.-Z. A.; Vaswani, A.; Uszkoreit, J.; Simon, I.; Hawthorne, C.; Shazeer, N.; Dai, A. M.; Hoffman, M. D.; Dinculescu, M.; and Eck, D. 2019.  Music Transformer: Generating music with long-term structure.  In _Proc. Int. Conf. Learning Representations_.
  * Huang and Yang (2020) Huang, Y.-S.; and Yang, Y.-H. 2020.  Pop Music Transformer: Beat-based modeling and generation of expressive Pop piano compositions.  In _Proc. ACM Multimedia_.
  * Jhamtani and Berg-Kirkpatrick (2019) Jhamtani, H.; and Berg-Kirkpatrick, T. 2019.  Modeling Self-Repetition in Music Generation using Generative Adversarial Networks.  In _Proc. Machine Learning for Music Discovery Workshop_.
  * Jiang et al. (2019) Jiang, J.; Wei, Y.; Feng, Y.; Cao, J.; and Gao, Y. 2019.  Dynamic hypergraph neural networks.  In _Proc. IJCAI_ , 2635–2641.
  * Jiang et al. (2020) Jiang, J.; Xia, G. G.; Carlton, D. B.; Anderson, C. N.; and Miyakawa, R. H. 2020.  Transformer VAE: A hierarchical model for structure-aware and interpretable music representation learning.  In _Proc. Int. Conf. Acoustics, Speech and Signal Processing_ , 516–520.
  * Katharopoulos et al. (2020) Katharopoulos, A.; Vyas, A.; Pappas, N.; and Fleuret, F. 2020.  Transformers are RNNs: Fast autoregressive Transformers with linear attention.  In _Proc. Int. Conf. Machine Learning_.
  * Kazemi et al. (2020) Kazemi, S. M.; Goel, R.; Jain, K.; Kobyzev, I.; Sethi, A.; Forsyth, P.; and Poupart, P. 2020.  Representation learning for dynamic graphs: A survey.  _Journal of Machine Learning Research_ 21(70): 1–73.
  * Ke, He, and Liu (2020) Ke, G.; He, D.; and Liu, T.-Y. 2020.  Rethinking positional encoding in language pre-training.  _arXiv preprint arXiv:2006.15595_ .
  * Keskar et al. (2019) Keskar, N. S.; McCann, B.; Varshney, L. R.; Xiong, C.; and Socher, R. 2019.  CTRL: A conditional Transformer language model for controllable generation.  _arXiv preprint arXiv:1909.05858_ .
  * Kivelä et al. (2014) Kivelä, M.; Arenas, A.; Barthelemy, M.; Gleeson, J. P.; Moreno, Y.; and Porter, M. A. 2014.  Multilayer networks.  _Journal of Complex Networks_ 2(3): 203–271.
  * Lerch et al. (2019) Lerch, A.; Arthur, C.; Pati, A.; and Gururani, S. 2019.  Music performance analysis: A survey.  In _Proc. Int. Soc. Music Information Retrieval Conf._
  * Oore et al. (2018) Oore, S.; Simon, I.; Dieleman, S.; Eck, D.; and Simonyan, K. 2018.  This time with feeling: Learning expressive musical performance.  _Neural Computing and Applications_ .
  * Patel (2003) Patel, A. D. 2003.  Language, music, syntax and the brain.  _Nature Neuroscience_ 6: 674–681.
  * Payne (2019) Payne, C. M. 2019.  MuseNet.  _OpenAI Blog_ .
  * Rae et al. (2020) Rae, J. W.; Potapenko, A.; Jayakumar, S. M.; Hillier, C.; and Lillicrap, T. P. 2020\.  Compressive Transformers for long-range sequence modelling.  In _Proc. Int. Conf. Learning Representations_.
  * Raffel et al. (2020) Raffel, C.; Shazeer, N.; Roberts, A.; Lee, K.; Narang, S.; Matena, M.; Zhou, Y.; Li, W.; and Liu, P. J. 2020.  Exploring the limits of transfer learning with a unified text-to-text Transformer.  _Journal of Machine Learning Research_ 21(140): 1–67.
  * Rossi et al. (2020) Rossi, E.; Chamberlain, B.; Frasca, F.; Eynard, D.; Monti, F.; and Bronstein, M. 2020.  Temporal graph networks for deep learning on dynamic graphs.  _arXiv preprint arXiv:2006.10637_ .
  * Uitdenbogerd and Zobel (1999) Uitdenbogerd, A.; and Zobel, J. 1999.  Melodic matching techniques for large music databases.  In _Proc. ACM Multimedia_ , 57–66.
  * Vaswani et al. (2017) Vaswani, A.; Shazeer, N.; Parmar, N.; Uszkoreit, J.; Jones, L.; Gomez, A. N.; Kaiser, Ł.; and Polosukhin, I. 2017.  Attention is all you need.  In _Proc. Advances in Neural Information Processing Systems_ , 5998–6008.
  * Wu and Yang (2020) Wu, S.-L.; and Yang, Y.-H. 2020.  The Jazz Transformer on the front line: Exploring the shortcomings of AI-composed music through quantitative measures.  In _Proc. Int. Soc. Music Information Retrieval Conf._
  * Yang and Lerch (2020) Yang, L.-C.; and Lerch, A. 2020.  On the evaluation of generative models in music.  _Neural Computing and Applications_ 32: 4773–4784.