# SCORE DIFFICULTY ANALYSIS FOR PIANO PERFORMANCE EDUCATION
BASED ON FINGERING

###### Abstract

In this paper, we introduce score difficulty classification as a sub-task of
music information retrieval (MIR), which may be used in music education
technologies, for personalised curriculum generation, and score retrieval. We
introduce a novel dataset for our task, Mikrokosmos-difficulty, containing 147
piano pieces in symbolic representation and the corresponding difficulty
labels derived by its composer Béla Bartók and the publishers. As part of our
methodology, we propose piano technique feature representations based on
different piano fingering algorithms. We use these features as input for two
classifiers: a Gated Recurrent Unit neural network (GRU) with attention
mechanism and gradient-boosted trees trained on score segments. We show that
for our dataset fingering based features perform better than a simple baseline
considering solely the notes in the score. Furthermore, the GRU with attention
mechanism classifier surpasses the gradient-boosted trees. Our proposed models
are interpretable and are capable of generating difficulty feedback both
locally, on short term segments, and globally, for whole pieces. Code,
datasets, models, and an online demo are made available for reproducibility.

Index Terms—  Difficulty Analysis, Piano Technique, Music Classification,
Piano Fingering, Symbolic Music Processing & Corpora

##  1 Introduction

Classification of music corpora is a problem well-studied under Music
Information Retrieval (MIR), which is often targeted from the listeners’
perspective as exemplified in genre/style [1, 2] and emotion [3, 4]
classification. In a paradigm shift, music may be classified from the point of
view of the performer by focusing on the required performance skills [5, 6],
which is a newly emerging field of study. This paper focuses on music
classification of performance difficulty, with applications in the formation
of large pedagogical score databases, personalised score recommendation
systems, and as an aid to both individual instrument learners and music
teachers. Towards helping the students in determining where to focus their
effort, and thus, increasing the efficacy in self-induced music studies, we
aim at giving feedback on relative local difficulty over multiple segments of
a piece.

The difficulty, by definition, is a subjective measure with multiple
dimensions (i.e. expressivity, rhythm, sound, and technique [7]). Publishers,
examination boards, and online repositories classify scores based on
performance difficulty. Many exam boards relate these rankings with the school
grades. A widely known publisher, Henle, releases piano difficulty rankings
for their scores in the range 1-9, but the range is more tailored towards
professionals. Certain instrument forums and websites such as 8note also
release difficulty grades in different ranges, generally three or four grades
from beginner to advanced. When it comes to the difficulty datasets in MIR,
both previous datasets [8, 5] are not publicly available.

As a first contribution, we release Mikrokosmos-difficulty 111Dataset
available at:
https://github.com/PRamoneda/Mikrokosmos-difficulty
DOI: 10.5281/zenodo.6092709 – a benchmark dataset for piano score difficulty
analysis derived from a corpus of 147 educational pieces authored by Béla
Bartók for use in piano education. To our best knowledge, this is the first
open dataset containing piano scores ranked in terms of difficulty and
matching different technical levels where all the scores are composed by a
single composer. Alongside the data, we also provide three different
difficulty labels to study the subjectivity of performance difficulty: the
first labels are the order of pieces generated by the composer himself, the
second is the book divisions by the original publisher, and the third is
difficulty labels in the range 1-9 by the publisher Henle, respectively. Since
all the scores are composed by the same composer, the difficulty rankings are
less prone to style biases, and more focused on technique difficulty.

As a second contribution, we introduce several piano technique features and
two classification algorithms capable of giving both score-level and segment-
level difficulty feedback 222Code and models available at:
https://github.com/PRamoneda/ICASSP22, whilst being trained solely using
score-level labels. To that end, we model the score with a novel feature
representation based on piano fingering, and taking that as the input, we
propose two difficulty classification methods: (i) gradient-boosted trees [9]
applied to short-term segments and (ii) GRU neural network [10] with an
attention mechanism. We want the selected classifiers to give feedback related
to the local difficulty of a piece allowing students and teachers to focus and
improve on the most difficult passages. Further, we provide a corpus
visualisation tool for exploring local difficulty representations derived from
both attention weights and the segment-level classification models. The
remainder of this paper is organized as follows: in Section 2, we give the
previous related work, in Section 3 we introduce the Mikrokosmos-difficulty
dataset, in Section 4 we propose our difficulty analysis methods and give the
results of our experiments in Section 5.

##  2 Relation with previous work

In this section we refer to the main methods to model difficulty in piano
repertoire [8, 5, 11]. Sebastien et al. [8] propose a list of different
instrument-agnostic descriptors for difficulty classification. The list of
descriptors was further extended by Chen et al. [5] proposing different
feature spaces for measuring difficulty in the piano repertoire. Although a
pitfall is that neither approach is reproducible without the code or datasets
available, the main drawback is that they use instrument-agnostic attributes.
In contrast, in the present research, we explore features related to the piano
playing technique. Another work targeting piano difficulty is proposed by
Nakamura et al. [11, 12] and deals with fingering frequencies and playing
rate. The rationale for this proposal is that piano fingerings which occur
less often, lead to increased difficulty. This method is extended to other
tasks such as polyphonic transcription [13], rhythm transcription [14] or
score reductions [15, 16], making clear the importance of piano technique in
the creation of music technology systems.

In this research, we aim at exploring several drawbacks of this approach [11],
while looking at piano fingering as a proxy for difficulty. In contrast to
their work which uses a concert-oriented dataset, we use a pedagogically
motivated dataset. In addition, we classify overall difficulty and not solely
instantaneous difficulty. Furthermore, we propose a thorough evaluation which
is lacking in the existing literature [11]. Nevertheless, we also use the
Nakamura et al. approach [11] to derive piano technique related features used
as input for the machine learning models.

##  3 Mikrokosmos-difficulty dataset

Béla Bartók’s Mikrokosmos Sz. 107 is a collection of 153 piano pieces
published in six volumes between 1926 and 1939 and progressively ranked in
terms of difficulty. Individual compositions go from extremely simple and easy
beginner exercises to challenging advanced technical musical works.

| Beginner | Moderate | Professional
---|---|---|---
n scores |  3cm62 |  3cm 54 |  3cm 31
n notes |  3cm $\mu=108.58$ |  |
$\sigma=57.16$ |  3cm $\mu=260.40$ |  |
$\sigma=111.00$ |  3cm $\mu=650.06$ |  |
$\sigma=322.15$ |  |  |
n bars |  3cm $\mu=20.37$ |  |
$\sigma=9.64$ |  3cm $\mu=33.35$ |  |
$\sigma=13.27$ |  3cm $\mu=63.12$ |  |
$\sigma=29.30$ |  |  |
tempo |  3cm $\mu=107.25$ |  |
$\sigma=24.48$ |  3cm $\mu=112.62$ |  |
$\sigma=59.75$ |  3cm $\mu=182.45$ |  |
$\sigma=113.13$ |  |  |

Table 1: Means $\mu$ and the standard deviations $\sigma$ for the number of
scores (n scores), notes (n notes), bars (n bars) and tempo across the three
levels of difficulty in the Mikrokosmos dataset.

We propose the Mikrokosmos-difficulty dataset based on a subset of 146 pieces
from the Mikrokosmos collection, discarding the 7 pieces composed for four
hands performance. We group the pieces on three levels of difficulty according
to the original classification in the the editions by the publishers Wiener
and Henle Verlag. Correspondingly, the beginner level contains the pieces 1-66
(Volumes I and II), the moderate level contains the pieces 67-121 (Volumes III
and IV), the professional level contains the pieces 122-153 (Volumes V and
VI). The 6-volume-long collection is obtained from IMSLP in pdf format. The
conversion to machine-readable symbolic representations in musicXML was done
semi-automatically using a commercial Optical Music Recognition (OMR) software
[17], and manual corrections. The statistics are presented in Table 3, where
we can see that Mikrokosmos-difficulty dataset is biased in lengths and tempo.
Therefore, analyzing the technique difficulty in this dataset requires
robustness to variations in length and tempo.

##  4 Methodology

We present five piano technique-based feature representations to analyse the
performance difficulty and an additional baseline using solely the notes.
Moreover, we use these features as input for two machine learning classifiers.
The two methods are interpretable, and we may derive important feedback for
music education applications.

###  4.1 Feature representation of the piano technique

Automatic piano fingering systems aim to describe the movements of hands and
fingers on the piano departing from the score. This task is related to piano
technique and a proxy to modelling the difficulty of playing a score. The
current approaches in piano fingering go from expert systems [18] to local
search algorithms [19, 20] and, more recently, data-driven methods [11, 12].
Towards modelling difficulty, we derive piano technique features from two
piano fingering approaches, a knowledge-driven system, Pianoplayer [20], and a
data-driven system proposed by Nakamura et al. [12].

Fig. 1: An example of piano fingering computed for a score (top) with
Pianoplayer and Nakamura et al. [12] for which we derive the features P-F,
P-V, N-F, N-P, and then the corresponding matrices $\mathbf{X_{f}}$

(bottom).

Pianoplayer [20] is a local search algorithm grounded in dynamic programming
and combinatorial optimisation, openly available under an MIT license. It
computes which finger is playing each note by considering the following nine
notes and the previous positions of the fingers. Pianoplayer carries out the
optimisation according to hard-coded constraints and a cost function related
to the finger velocity. From the cost function, we derive two features
associated with each note: the finger which plays the note, P-F, and the
velocity associated with the finger, P-V.

Nakamura et al. [12] train a Hidden Markov Model (HMM) on a 150 scores dataset
annotated by more than four professional pianists – data, code and models are
available for noncommercial activities. In contrast to Pianoplayer which has
hard-coded rules, this model learns implicitly the transition probabilities
that model finger movements. We use the transition probability given by the
HMM model to derive two features for a given note: the finger which plays the
note, N-F, and the transition probability associated to the note, N-P.

Given a piano score and the features associated with each note P-F, P-V, N-F,
N-P, we then form the matrices $\mathbf{X_{f}}(i,j)$ containing all the notes
for each of the four features, where $i=1\ldots I$ and $I$ is the total number
of note onsets in the piece, and $j=1\ldots 10$ are the fingers. In order to
reduce the size of $\mathbf{X_{f}}$ and the computational complexity, we do
not consider the note duration. In Figure 1 we present an example of how these
matrices are formed. Note that the matrices $\mathbf{X_{f}}$ associated with
P-F and N-F are binary matrices containing $1$ if the finger $j$ plays at note
onset $i$, while the matrices $\mathbf{X_{f}}$ associated with P-V and N-P
contain either the velocities or the transition probabilities associated with
each note onset. As a baseline, we include a simple note onset representation
K as a feature agnostic to piano technique. We form the matrix
$\mathbf{X_{n}}(i,k)$ containing $1$ if a note $k$ is played at the onset $i$
or $0$ otherwise, where $k=1\ldots 88$ are the notes that can be played on the
piano.

###  4.2 Classification methods

We use the five features introduced in Section 4.1 as input for two machine
learning methods, gradient-boosted trees and a GRU neural network with an
attention mechanism. To that extent, we decided to use machine learning
methods that offer some form of interpretability. This property may
potentially help with understanding the difficult parts of a score.
Correspondingly, the decision trees give us comprehensive explanations while
the attention mechanism in the neural network points out which parts the
network focuses on.

Gradient-boosted trees – _XGBoost_. We perform a two-step classification of a
score into the three difficulty classes using a decision tree classifier,
gradient boosted trees [21]. Since this method works solely on fixed-sized
inputs, we split the input feature matrices $\mathbf{X_{f}}$ and
$\mathbf{X_{n}}$ into windows of size $w$, representing short-term score
segments. The resulting matrices $\mathbf{\hat{X}_{f}}$ of size $(w\times 10)$
and $\mathbf{\hat{X}_{n}}$ of size $(w\times 88)$ are used as input to the
classifier. Next, the predictions corresponding to each segment are averaged
across the whole score to obtain a global difficulty. Note that besides being
interpretable, this method is more robust to noisy labels [21]. In our case,
we deal with a weakly supervised scenario because we have annotations solely
for the whole piece, and not for each segment. Moreover, the difficulty of the
segments may vary across a piece.

GRU network with attention mechanism – _DeepGRU_. Towards modelling time
dependencies in the score and dealing with variable size scores as inputs, we
use a recurrent neural network classifier. We modify an existing architecture
used in multivariate time series classification, _DeepGRU_ [10]. The model
lies in a set of stacked gated recurrent units (GRU), two fully connected (FC)
layers and a global attention model. The final two FC layers, using ReLU
activations, take the output of the attention module and use a softmax
classifier to create the probability distribution of the class labels. The
GRUs stacked layers are able to model the time dependencies while the
attention mechanism selectively attends to specific note onsets which are more
important in the difficulty decision. We use the attention layer to visualise
and understand the important notes.

##  5 Experiments

We evaluate the classification (balanced) accuracy of the machine learning
models _XGBoost_ and _DeepGRU_ trained with proposed features P-F, P-V, N-F,
N-P, K on the Mikrokosmos-difficulty dataset we introduce in Section 3.
Towards assessing the impact of weak labels, we compare the 2-step
classification, _XGBoost (avg)_ with a single step classification, where the
predictions are not averaged across all segments, _XGBoost (window)_. In
addition, we rank the scores using the output probability of the classifiers.
The ranking is calculated by multiplying each class probability by their class
number in the last layer. We then compare this ranking with the original one
proposed by Bartók and the one by the publisher Henle using Spearman’s rank
correlation.

###  5.1 Experimental setup

Because we want to keep the window size consistent with the $9$ notes used as
a temporal context by Pianoplayer, we use segments of window size $w=9$.
Hence, the input matrices are of size $9\times 10$ in the case of
$\mathbf{\hat{X}_{f}}$ and $9\times 88$ in the case of $\mathbf{\hat{X}_{n}}$.
Similarly, we set the stride $s=1$ in order to generate the maximum overlap
between the windows. Note that we perform an ablation study to evaluate the
effect of the window size.

We use the gradient-boosted trees implementation in the XGBoost library [9]
for training the _XGBoost (window)_. We use a random search over the training
set (5-fold cross-validation) to tune seven hyper-parameters. In this case, we
pick the parameters corresponding to the model with the best balanced accuracy
on the validation set. The DeepGRU model is trained with the original
parameters [10]: 20 epochs, an Adam optimizer with a learning rate of 0.002, a
batch-size of 64 samples and using negative log likelihood loss as the
criterion.

In both classification methods, the 80% of data is used for training and 20%
for testing. We repeat the experiments for $50$ different initial random seeds
which control the initialisation of the machine learning models and the train
and test splits. We report means and standard deviations for the considered
metrics across the $50$ seeds.

###  5.2 Results

The effect of different feature representations on the difficulty
classification accuracy is presented in Table 2. The results have to be
interpreted considering the large standard deviations across the 50 seeds,
which the small size of our dataset may cause.

Table 2: 3-class classification accuracy (%) for different features and
machine learning models.

| _XGBoost (window)_ | _XGBoost (avg)_ | _DeepGRU_
---|---|---|---
| train | test | train | test | train | test
K | $77\pm 9$ | $51\pm 7$ | $90\pm 8$ | $65\pm 10$ | $93\pm 6$ | $64\pm 8$
${\small\textsf{N}}-{\scriptsize\textsf{F}}$ | $71\pm 5$ | $51\pm 5$ | $83\pm 5$ | $62\pm 8$ | $85\pm 5$ | $72\pm 8$
${\small\textsf{P}}-{\scriptsize\textsf{F}}$ | $74\pm 7$ | $52\pm 4$ | $86\pm 6$ | $64\pm 8$ | $87\pm 4$ | $73\pm 6$
${\small\textsf{N}}-{\scriptsize\textsf{P}}$ | $80\pm 6$ | $57\pm 5$ | $92\pm 6$ | $67\pm 8$ | $83\pm 5$ | $71\pm 8$
${\small\textsf{P}}-{\scriptsize\textsf{V}}$ | $88\pm 5$ | $62\pm 6$ | $96\pm 4$ | $68\pm 10$ | $80\pm 5$ | $\mathbf{78\pm 6}$

We observe that _XGBoost (avg)_ yields better results than _XGBoost (window)_.
Thus, averaging the results across the whole piece is better than considering
stand-alone segments. We see that using transition probabilities, N-P, and
velocity information, P-V, increases the performance compared to using the
fingers solely, P-F, N-F. Comparing the two finger modelling methods, we see
that Pianoplayer fingerings, P-, perform better than the Nakamura fingerings,
N-. To our best knowledge, it is the first time Pianoplayer is explored in a
research paper, and the results demonstrate its performance is comparable with
the established Nakamura’s algorithm.

The importance of finger modelling in piano difficulty analysis may be seen in
the drastic improvement for the _DeepGRU_ case, where all the fingering
strategies improved the classification accuracy by a large margin compared to
features generated directly from sheet music without finger modelling (K).

The ranking performance using Bartók and Henle difficulty rankings is
presented in Table 3. Even though the models are trained using solely 3-class
labels, the difficulty rankings we derive evaluated against the rankings
provided by the composer (Bartók) and the publisher (Henle) achieve high
correlation.

Table 3: Spearman’s rank correlation coefficient between the rankings derived from the output probabilities of the classifiers and the Bartók and Henle rankings | _XGBoost (avg)_ | _DeepGRU_
---|---|---
| Bartók | Henle | Bartók | Henle
K | $.74\pm.09$ | $.75\pm.08$ | $.71\pm.08$ | $.71\pm.09$
${\small\textsf{N}}-{\scriptsize\textsf{F}}$ | $.70\pm.08$ | $.66\pm.08$ | $.82\pm.06$ | $.81\pm.06$
${\small\textsf{P}}-{\scriptsize\textsf{F}}$ | $.69\pm.09$ | $.64\pm.11$ | $.80\pm.06$ | $.80\pm.06$
${\small\textsf{N}}-{\scriptsize\textsf{P}}$ | $.78\pm.07$ | $.73\pm.09$ | $.84\pm.06$ | $.83\pm.06$
${\small\textsf{P}}-{\scriptsize\textsf{V}}$ | $.81\pm.08$ | $.78\pm.10$ | $\mathbf{.86\pm.05}$ | $\mathbf{.86\pm.05}$

The ablation study of the window size for the _XGBoost_ method is shown in
Table 4. Considering the high standard deviation, we concluded that the change
in window size is not statistically significant. For this reason, we decided
to stick with the window size of the Pianoplayer method, which is $w=9$.

Table 4: Evaluation considering different window size for _XGBoost (avg)_. Results are presented in terms of 3-class classification accuracy (%), Spearman rank correlation coefficient for Bartók and Henle. window size | 3-class acc | Bartók rank | Henle rank
---|---|---|---
$w=1$ | $69\pm 12$ | $.81\pm.08$ | $.77\pm.10$
$w=3$ | $69\pm 10$ | $.81\pm.08$ | $.78\pm.10$
$w=5$ | $69\pm 10$ | $.81\pm.09$ | $.78\pm.10$
$w=9$ | $68\pm 10$ | $.81\pm.08$ | $.78\pm.10$
$w=13$ | $69\pm 10$ | $\mathbf{.83\pm.07}$ | $\mathbf{.82\pm.07}$
$w=19$ | $\mathbf{70\pm 9}$ | $.82\pm.06$ | $.80\pm.07$

###  5.3 Local Difficulty Feedback

As a practical application to music learning, giving feedback related to the
local difficulty of a piece allows students and teachers to focus and improve
on the most difficult passages. Because _XGBoost_ and _DeepGRU_ are
interpretable, we derive local difficulty feedback for both of the methods. We
created an online tool to explore the two different types of feedback for the
Mikrokosmos-difficulty dataset 333At: https://musiccritic.upf.edu/mikrokosmos.

Window-based feedback. The difficulty of the windows pertaining to a score may
vary considerably. In the case of the 2-step _XGBoost (window)_
classification, we obtain an output probability for each window. Considering
the maximum overlapping for the windows for a stride of $s=1$, we derive a
probability for each note onset. Therefore, this feedback is an indicator of
the local difficulty corresponding to that note onset. An example of
visualisation of the local difficulty may be seen in Figure 2. The difficulty
levels in the Mikrokosmos-difficulty dataset are displayed over the score with
three different colours We colour level 1 notes with green, level 2 notes with
yellow and level 3 with red. In the excerpt shown in Figure 2 we note how the
difficulty changes within the same piece. We observe that the part we
highlight as A.1 has asymmetric changes between the two hands while the A.2
part is a simple arpeggio. Consequently, A.1 is marked in red while A.2 is
yellow.

Fig. 2: Mikrokosmos Sz. 107 no. 142. Béla Bartók excerpt. Local difficulty
feedback from _XGBoost (window)_ with P-V as input feature. The score is from
the test set of a random seed.

Attention-based feedback. This method grounds on the weights of the attention
layer in the _DeepGRU_ model. Attention tells us the notes on which the model
is focusing. The most interesting aspect of the present feedback is dealing
with repetitions that occur in the short term. We can see a particular example
in Figure 3.In this plot the attention weights corresponding to each note
control the intensity of the color. The pairs A.1-A.2 and B.1-B.2 are
performed with the same physical movement. Therefore, in the second pattern,
A.2 and B.2 ,respectively, the method pays less attention. The attention
reasoning is similar to how music students should study. When dealing with two
similar patterns, students should study the first one very well and repeat the
second one by imitating the first one.

Fig. 3: Mikrokosmos Sz. 107 no. 109. Béla Bartók excerpt. Local difficulty
feedback from _DeepGRU_ approach with P-V as input feature. The score is from
the test set of a random seed.

##  6 Conclusions and future work

In this work, we proposed a methodology to classify piano scores based on
piano technique features. Namely, we derive piano fingering features from two
approaches [12, 20], and a simple baseline based on using solely the notes in
the score. Towards training and testing machine learning methods for this
task, we distributed the Mikrokosmos-difficulty dataset. We evaluated two
interpretable machine learning methods on the proposed dataset to show that
piano technique features are good predictors for the difficulty of performing
a piece. In addition to the openly available source code, models and data, we
also provide an online demo which visualizes the segmental difficulty, which
is crucial in educational feedback. We acknowledge that the small size of the
dataset may lead to over-fitting biases and large standard deviations for the
bootstrapped experiments. In future, we plan to extend this work with larger
score dataset with difficulty annotations and other instruments.

## References

  * [1] Deepanway Ghosal and Maheshkumar H Kolekar,  “Music genre recognition using deep neural networks and transfer learning.,”  in Interspeech, 2018, pp. 2087–2091.
  * [2] Christof Weiss and Meinard Müller,  “Tonal complexity features for style classification of classical music,”  in 2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2015, pp. 688–692.
  * [3] Satoru Fukayama and Masataka Goto,  “Music emotion recognition with adaptive aggregation of gaussian process regressors,”  in 2016 IEEE international conference on acoustics, speech and signal processing (ICASSP). IEEE, 2016, pp. 71–75.
  * [4] Yi-Hsuan Yang and Homer H Chen,  “Ranking-based emotion recognition for music organization and retrieval,”  IEEE Transactions on audio, speech, and language processing, vol. 19, no. 4, pp. 762–774, 2010.
  * [5] Shih-Chuan Chiu and Min-Syan Chen,  “A study on difficulty level recognition of piano sheet music,”  in 2012 IEEE International Symposium on Multimedia. IEEE, 2012, pp. 17–23.
  * [6] Pedro Ramoneda,  “Computational methods to study piano music in education context,”  Master’s Thesis, Universitat Pompeu Fabra, 2020.
  * [7] Heinrich Neuhaus,  The art of piano playing,  Kahn and Averill, 2008.
  * [8] Véronique Sébastien, Henri Ralambondrainy, Olivier Sébastien, and Noël Conruyt,  “Score analyzer: Automatically determining scores difficulty level for instrumental e-learning,”  in Proceedings of 13th International Society for Music Information Retrieval Conference, Porto, 2012, ISMIR 2021.
  * [9] Tianqi Chen and Carlos Guestrin,  “Xgboost: A scalable tree boosting system,”  in Proceedings of the 22nd acm sigkdd international conference on knowledge discovery and data mining, 2016, pp. 785–794.
  * [10] Mehran Maghoumi and Joseph J LaViola,  “Deepgru: Deep gesture recognition utility,”  in International Symposium on Visual Computing. Springer, 2019, pp. 16–31.
  * [11] Eita Nakamura, Nobutaka Ono, and Shigeki Sagayama,  “Merged-output hmm for piano fingering of both hands.,”  in Proceedings of the 13th International Society for Music Information Retrieval Conference, Taipei, 2014, ISMIR 2014, pp. 531–536.
  * [12] Eita Nakamura, Yasuyuki Saito, and Kazuyoshi Yoshii,  “Statistical learning and estimation of piano fingering,”  Information Sciences, vol. 517, pp. 68–85, 2020.
  * [13] Eita Nakamura, Emmanouil Benetos, Kazuyoshi Yoshii, and Simon Dixon,  “Towards complete polyphonic music transcription: Integrating multi-pitch detection and rhythm quantization,”  in 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2018, pp. 101–105.
  * [14] Eita Nakamura, Kazuyoshi Yoshii, and Shigeki Sagayama,  “Rhythm transcription of polyphonic piano music based on merged-output hmm for multiple voices,”  IEEE/ACM Transactions on Audio, Speech, and Language Processing, vol. 25, no. 4, pp. 794–806, 2017.
  * [15] Eita Nakamura and Shigeki Sagayama,  “Automatic piano reduction from ensemble scores based on merged-output hidden markov model,”  in ICMC, 2015.
  * [16] Eita Nakamura and Kazuyoshi Yoshii,  “Statistical piano reduction controlling performance difficulty,”  APSIPA Transactions on Signal and Information Processing, vol. 7, 2018.
  * [17] Neratron, Sibelius Software division, Avid Technology,  “Photoscore,” v2020.
  * [18] Richard Parncutt, John A Sloboda, Eric F Clarke, Matti Raekallio, and Peter Desain,  “An Ergonomie Model of Keyboard Fingering for Melodic Fragments Sibelius Academy of Music , Helsinski,”  Music perception: An Interdisciplinary Journal, vol. 14, no. 4, pp. 341–382, 1997.
  * [19] Matteo Balliauw, Dorien Herremans, Daniel Palhazi Cuervo, and Kenneth Sörensen,  “A variable neighborhood search algorithm to generate piano fingerings for polyphonic sheet music,”  International Transactions in Operational Research, vol. 24, no. 3, pp. 509–535, 2017.
  * [20] “PianoPlayer automatic piano fingering generator.,” https://github.com/marcomusy/pianoplayer,  Accessed: 2021-02-18.
  * [21] Anabel Gómez-Ríos, Julián Luengo, and Francisco Herrera,  “A study on the noise label influence in boosting algorithms: Adaboost, gbm and xgboost,”  in International Conference on Hybrid Artificial Intelligence Systems. Springer, 2017, pp. 268–280.


# The life and entangled adventures of Schrödinger’s cat

A Sudbery1
Department of Mathematics, University of York,
Heslington, York, England YO10 5DD
1<EMAIL_ADDRESS>
(3 August 2023)

###### Abstract

A lecture given at St. Cross College, Oxford, on 10th June 2017 in a
conference on The Nature of Quantum Reality

In this lecture, intended for a general audience, I describe Schrödinger’s
thought experiment which was designed to show the strange results of extending
the formalism of quantum theory, particularly the idea of superposition,
beyond the subatomic regime. I describe a way to understand superposition in
the terms of formal logic. I trace the development of Schrodinger’s ideas
after this thought experiment, and briefly sketch some work which realises it
in actual experiments, and proposals for taking it further.

In this talk I will need to review some of the basic ideas of quantum theory,
which means that there will be some overlap with the previous talk by Jim
Baggott. And as we all know, in quantum theory where there is overlap there is
likely to be interference.

SUPERPOSITION

One of the basic ideas of quantum mechanics – some might say, _the_ basic idea
– is _superposition_. This is denoted in the mathematical form of the theory
by the simple, innocent symbol $+$. Yet this simple operation takes on a very
mysterious aspect when we consider its physical meaning for objects at very
small, subatomic scales. Consider a single particle at this scale, and ask
“Where is it?” We might know that it is at, or near, one definite place, say
“here”. This could correspond to a function of position like $\Psi_{1}(x)$ in
the first slide, a state of affairs which is represented in quantum theory by
the symbol $|\text{here}\rangle$ in a notation invented by Dirac. The
characters $|$ and $\rangle$ indicate that this is a description of the state
of a quantum object, and between them is put information about this
description. Another possibility is that the particle might be in a different
place, say “there”, corresponding to another function $\Psi_{2}(x)$ or a state
$|\text{there}\rangle$.

But according to quantum mechanics, there is a third possibility: the particle
might be in a _superposition_ of being here and being there, corresponding to
the function $\Psi_{1}(x)+\Psi_{2}(x)$ and represented by the state symbol
$|\text{here}\rangle+|\text{there}\rangle$. You might think that there is
nothing mysterious about this; it simply means that the particle is _either_
here _or_ there. As long as the functions $\Psi_{1}(x)$ and $\Psi_{2}(x)$ do
not overlap, this seems reasonable; but if they start to overlap, the
phenomenon of _interference_ makes itself known, and shows that we cannot
understand superposition this way.

This is demonstrated in the _two-slit experiment_ , shown in the next slide. A
beam of particles, inaccurately collimated so that they cover a range of
directions, is fired at a screen in which two parallel slits have been cut and
can be covered by shutters. Beyond this screen is a second screen which
collects the particles and registers their arrival. If just one slit is open,
the particles reach the second screen in a fuzzy band centred on a line
parallel to the line of the slit. What happens if both slits are open? If the
particles are the sort of thing we are used to in classical physics, like
bullets, then each particle must go through one slit or the other and the
pattern registered on the second screen will be two separated fuzzy lines. But
that is not what is seen with subatomic particles like electrons. Instead,
there is a corrugated pattern, known as an _interference pattern_ , as shown
in the lower picture of the slide.

The description of this experiment in quantum mechanics is as follows. If only
the left-hand slit is open, a particle that goes through that slit will be in
a state $|\text{slit 1}\rangle$ when it reaches the slit, and this will go on
to become a state $|\text{line 1}\rangle$ when it reaches the second slit.
There will be other states $|\text{slit 2}\rangle$ and $|\text{line 2}\rangle$
describing particles that go through the right-hand slit when that is open.
When both slits are open the state of each particle at the second screen will
be $|\text{line 1}\rangle+|\text{line 2}\rangle$. If the symbol $+$ in that
expression meant “or” – if each particle went through _either_ the left-hand
slit _or_ the right-hand slit – then the pattern on the second screen would be
two parallel lines, as it is with bullets. But that is not what is seen. So
$+$ does not mean “or”. It surely doesn’t mean “and” – a particle certainly
isn’t in the left-hand line on the second screen _and_ in the right-hand line.
That familiar symbol $+$, when applied to descriptions of quantum particles,
means something entirely new.

Another new aspect of superposition is that unlike “and” and “or”, which are
both single ways of combining descriptions, superposition encompasses a range
of possibilities: the two states $|\text{here}\rangle$ and
$|\text{there}\rangle$ allow superpositions
$a|\text{here}\rangle+b|\text{there}\rangle$ where $a$ and $b$ are any numbers
(complex numbers, even) though it is only their ratio $a/b$ that counts. Thus
superpositions of $|\text{here}\rangle$ and $|\text{there}\rangle$ range from
the simple $|\text{here}\rangle$ continuously through the equal superposition
$|\text{here}\rangle+|\text{there}\rangle$ to the simple
$|\text{there}\rangle$ – something which has to be taken into account in any
explanation of the quantum meaning of $+$.

ENTER THE CAT

In 1935 Erwin Schrödinger faced up to the implications of this new theory for
what physics has to tell us about the whole world. If quantum mechanics is the
basic theory of how matter behaves, then it should apply to all matter,
including familiar objects like animals. This strange new concept $+$ should
apply to such familiar things; if something can exist in a state
$|\text{here}\rangle$, and it can also exist in a different state
$|\text{there}\rangle$, then this physics says that it can also exist in a
third state $|\text{here}\rangle+|\text{there}\rangle$. He wrote:

> One can even set up quite ridiculous cases. A cat is penned up in a steel
> chamber, along with the following diabolical device (which must be secured
> against direct interference by the cat); in a Geiger counter there is a tiny
> bit of radioactive substance, _so_ small, that _perhaps_ in the course of
> one hour one of the atoms decays, but also, with equal probability, perhaps
> none; if it happens, the counter tube discharges and through a relay
> releases a hammer which shatters a small flask of hydrocyanic acid. If one
> has left this entire system to itself for an hour,, one would say that the
> cat still lives _if_ meanwhile no atom has decayed. The first atomic decay
> would have poisoned it. The $\psi$-function of the entire system would
> express this by having in it the living and the dead cat (pardon the
> expression) mixed or smeared out in equal parts. [6]

In the Dirac notation introduced above: if a cat can exist in a state
$|\text{alive}\rangle$ and can exist in a state $|\text{dead}\rangle$, as it
surely can, then it can also exist in a state
$|\text{alive}\rangle+|\text{dead}\rangle$.

AND YOUR POINT IS …?

The wording at the end of that passage by Schrödinger is a little misleading.
Elsewhere in the same article Schrödinger was at pains to insist that
superposition is _not_ a matter of “smearing”:

> There is a difference between a shaky or out-of-focus photograph and a
> snapshot of clouds and fog banks.

But whatever superposition is, Schrödinger regards this consequence of
orthodox quantum mechanics as “ridiculous”.

This point is often made (but _not_ by Schrödinger) by arguing “If cats can
exist in this strange state, why don’t we ever see such (alive + dead) cats?
The way to answer such questions was indicated by Schrödinger himself; it lies
in the concept of _entanglement_ which he introduced in this same paper.

ENTANGLEMENT

The answer to the question “Why don’t we see cats in superposition states like
$|\text{alive}\rangle+|\text{dead}\rangle$?” is that the physics of looking at
a cat shows that the observer will never see such a thing. When the box
containing the cat in Schrödinger’s diabolical device is opened, the cat is
not in the state $|\text{alive}\rangle+|\text{dead}\rangle$; instead, the
whole system of the cat and the observer (and the rest of the device) is in
the superposition state

$|\text{alive}\rangle|\text{glad}\rangle+|\text{dead}\rangle|\text{sad}\rangle.$
(1)

Here the first term is a state of the cat and the observer in which the cat is
alive and the observer is glad because they see that the cat is alive;
likewise, the second term is a state of the cat and the observer in which the
cat is dead and the observer is sad because they see that the cat is dead.
Nowhere is there a state in which the observer sees the cat in a
superposition. This is the case because we know that the ordinary processes of
everyday physics will cause the cat state $|\text{alive}\rangle$, together
with an observer outside the closed box, to become the first term in eq. (1)
when the box is opened; likewise, the cat state $|\text{dead}\rangle$ with the
observer outside the box will become the second term. The rules of quantum
mechanics then guarantee that the superposition state of the cat, with the
observer outside the box, will become the combined state (1) when the box is
opened.

The state (1) is said to be an _entangled_ state of the combined system of the
cat and the observer. Neither the cat nor the observer has a definite state,
but they are correlated: if the cat is (somehow) found to be alive, then the
observer will be found to be glad, and if the cat is found to be dead, then
the observer will be found to be sad. The word “entanglement” was introduced
by Schrödinger (translating the German Verschränkung) in the paper [6] that
introduced the cat; the concept was already implicit in the paper by Einstein,
Podolsky and Rosen which prompted Schrödinger’s paper – it is just a special
case of superposition – but Schrödinger recognised its special significance,
and it has continued to play a central role in discussions and applications of
quantum mechanics.

MEASUREMENT

I hear impatient mutterings in the audience: “What do you mean, if the cat is
(_somehow_) found to be alive. If someone opens the box and sees the cat
alive, then surely the cat _is_ alive.” Well, you would think so, wouldn’t
you? But the rigorous theory of quantum mechanics, in the form of the equation
of motion (due to – guess who – Schrödinger) which governs the change of the
state vector, only gives the superposition shown in eq. (1). To the founders
of quantum mechanics, who regarded the results of observation, or
_measurement_ , as the only facts that belong in science, this was not a
problem: one calculates using the Schrödinger equation and then uses a well-
defined procedure (known as the _Born rule_) to obtain probabilities for the
possible results of measurement. One can then throw away the state vector as
no longer meaningful. Or if one wants to make further calculations, one has to
allow for an unpredictable change caused by the very fact of measurement,
which cannot be further analysed. This unpredictable change is described by a
supplement to the equation of motion known as the _projection postulate_ (due
to Paul Dirac, though it is often erroneously credited to John von Neumann).

The resulting complete form of quantum theory, which is presented in most
textbooks, presents two different laws for the change of physical objects, or
systems of objects: the Schrödinger equation and the projection postulate. The
first of these applies as long as the objects are not disturbed by a
measurement. This is deterministic: given the state of the objects at one
time, the state at any later time is uniquely determined by the equation. It’s
a beautiful law of motion for any physical system. But, according to official
quantum mechanics, this motion is rudely and randomly interrupted whenever
someone makes a measurement. Only probabilities can be given for the result of
the measurement, and the system changes, depending on the result of the
measurement, according to the projection postulate.

This situation was acceptable to the founding fathers, who were struggling to
understand the strangeness of the subatomic world which was only just being
revealed. But increasingly, modern physicists find it unacceptable. In the
case of Schrödinger’s cat, does it mean that the RSPCA official who rushes
into Schrödinger’s lab to stop this diabolical experiment actually kills the
cat, or possibly saves it, simply by opening the box to see if the cat is
alive? More seriously, what is this “measurement” that causes the system to
behave in a completely different way? Any measurement or observation is
performed by an arrangement of physical apparatus, which should be subject to
the Schrödinger equation like any other physical objects. What makes it behave
differently? This is the _measurement problem_ of quantum mechanics, which
still has no generally accepted solution. It is often raised by students, who
may be met with impatient dismissal and told, as David Mermin put it, to “shut
up and calculate”. I was a student in a class given by Dirac, who, when we
raised this problem, simply shrugged and said “That’s how it is”.

THE HISTORY OF SCHRÖDINGER’S CAT

There’s a fork in the path here. First, let’s follow Schrödinger.

THE HISTORY OF SCHRÖDINGER’S THOUGHT

In 1926 Schrödinger didn’t _invent_ quantum mechanics, but he _explained_
quantum mechanics as an eigenvalue problem in terms of a wave function, which
is the sort of thing that physicists like to deal with. And everybody – no,
not everybody, but a lot of practical people using quantum mechanics, and in
particular chemists – are really happy working with wave functions rather than
the alternative mathematical description of Heisenberg and Dirac in terms of
state vectors, which is often seen as ”abstract” (though I think that on the
contrary, Dirac’s formalism is more concrete). Anyway, Schrödinger was
attached to the wave function, and to the wave equation. In 1935 he presented
the story of the cat as an illustration. (It’s interesting that he talks about
the wave function _of the cat_. That’s an enormously complicated function.) In
1952 [7] he emphasised the importance of the wave function, and in particular
the wave equation. He said there are no quantum jumps 111Schrödinger had
always thought this: in 1929 he grumbled “If I had known we were going to go
on having all this damned quantum-jumping, I would never have got involved in
the subject” [2], nothing changes discontinuously; he even said there are no
particles. He didn’t believe in particles, he only believed in the wave
function. And he made the point (he also made this point in the cat paper)
that if you’ve got a theory of waves, OK you can talk about the waves and the
wave front and so on, but a theory of waves also encompasses something very
particle-like. In most situations, if you’ve got a wave developing, then you
can interpret this in terms of _rays_. You can do geometrical optics as well
as wave optics; and those rays can be interpreted as the trajectories of
particles. For Schrödinger, a wave description is better, more encompassing,
than a particle description. He thought people concentrate far too much on
particles. He prefaced this paper by saying “People are going to think I’m
mad. I’m going to tell you that both of these things are true. You’ve got a
particle here _and_ the same particle there. You’ve got a cat dead _and_ a cat
alive.” Well, that’s a bit mysterious, and in 1955 he cleared it up by saying
“I really do believe that the wave function (of the universe) is all that
there is” [7]. This is getting very close to what, much more famously, was
published by Everett and endorsed by Wheeler in 1957, and which, after being
popularised by de Witt in 1971, has come to be known as the “many worlds
interpretation” of quantum mechanics. According to this, what that
superposition of a live cat (and a happy observer) and a dead cat (and a sad
observer) really means is that there are two worlds: one in which the cat is
alive and one in which the cat is dead. So Schrödinger anticipated the many-
worlds interpretation.

This takes us back to the question of the meaning of superposition. What does
$|\text{here}\rangle+|\text{there}\rangle$ mean? Does it mean that either the
particle is here _or_ the particle is there? That’s what it means for us when
we observe the particle; that’s the way we tend to think of superposition. But
as we saw earlier in the two-slit experiment, it cannot be the true meaning of
superposition. It seems clear that it cannot mean that the particle is here
_and_ the particle is there, though the many-worlds interpretation seems to be
a way of trying to make sense of this. But there are problems with this; who
decides what exactly a “world” is? I don’t think there are many worlds; there
is only one world, and its state is a superposition. I would like to show you
a way to make sense of this.

LOGIC

There is a lot of discussion about the “reality” of the wave function. But
that’s a category error. We’re not talking about things – whether the wave
function is _real_ – we’re talking about propositions: whether the wave
function is a _true_ description. If we have two propositions $P$ and $Q$,
then the compound proposition $P\;or\;Q$ means that just one of $P$ and $Q$ is
true, i.e. has truth value 1. The compound proposition $P\;and\;Q$ means that
they both have truth value 1. In quantum mechanics, with $P$ and $Q$ being
state descriptions (or state vectors) we have this new way, superposition, of
forming compound propositions: $aP+bQ$ where $a$ and $b$ are complex numbers
which we can take to satisfy $|a|^{2}+|b|^{2}=1$. Now the truth value is not
just 0 or 1 for each of these propositions, but the truth is divided between
them. We have to use a many-valued logic in which truth values, just like
probabilities, can take any value, a real number beteen 0 and 1. So if the
universe is described by a superposition of states, in one of which the cat is
alive and in the other it is dead, then it’s that superposition that has truth
value $1$, and the individual components – “cat alive” and “cat dead” – are
not true, at least they’re not completely true; but they’re not completely
false either. If $a|\text{alive}\rangle+b|\text{dead}\rangle$ is true, then
$|\text{alive}\rangle$ has truth value $|a|^{2}$ and $|\text{dead}\rangle$ has
truth value $|b|^{2}$.

There’s another logical point. I’ve been talking about the state of the whole
universe; that’s a sort of God’s eye view, which we can never attain – still,
we might believe that it exists. But that’s not what we actually have. We are
inside the universe. Living in this universe with the cat in it, in the
superposition shown earlier, I am one of these people – that is, I have one of
these brain states. Another way to understand this superposition is to say
that it’s a conditional statement. It says that if I know that one of these
partial states (of part of the universe) is true, however I know it, then I
know that the state of the rest of universe which is attached to it is also
true. If I know that I am happy, then this superposition state tells me that
the cat is alive. And of course, I do know that I am happy, or sad, or
puzzled. But this contradicts the quantum-mechanical view, the God’s-eye view,
of the universe. So we have to acknowledge context, or perspective. From my
perspective, I know that I am happy, or sad, or not knowing; and I know these
statements with the usual bivalent logic. There is only a certain set of
experiences that I can have (determined by the physics of my brain), and I
know that just one of them is true of me at the present moment. Only one of
them has the truth value 1; the others have truth value 0, and there is
nothing in between.

Ordinary bivalent logic applies to statements that I make about my present
experience. But it is different for statements about my future. According to
the Schrödinger equation, my present state of a definite experience can evolve
into a superposition of different experience states. In that situation, what
can I say about what I will experience at some point in the future? My state
will be a superposition; but I can’t experience such a state. I will have some
definite experience, but there is no true statement about which experience it
will be. Each statement “I will have experience $E$” has a truth value between
$0$ and $1$; or, as we usually say, there is only a probability that I will
have that experience. That probability, or truth value, is given by the
squared modulus of the coefficient of $|E\rangle$ in my future superposition
state.

This is not a new idea; it goes back to Aristotle. In a famous passage,
Aristotle discussed the truth or falsity of the proposition “There will be a
sea battle tomorrow”. He concluded that such a statement was neither true or
false. This has been taken by modern logicians to mean that statements in the
future tense obey a many-valued logic; they have augmented the usual values
$1$ or$t$ (true) and $0$ or $f$ (false) by a third truth value $u$
(undecided). For formal reasons, they have restricted truth values to these
three, but it seems much more fitting in a temporal logic [8] to use the full
set of values between $0$ and $1$, which relates them to probability.
Aristotle, indeed, pointed out that future events can be more or less likely.

THE HISTORY OF SCHRÖDINGER’S CAT

Let’s get back to the adventures of the cat. It was 82 years ago that
Schrödinger told the story of the cat. In the first 40 years after that, only
26 people mentioned it in a scientific paper. That might reflect citing
habits, because by 1975, I can assure you, the idea of Schrödinger’s cat was
well known. But in 1986 there was a symposium [3] to celebrate the centenary
of his birth. There were 19 papers, and only one, by John Bell, gave a
citation to the cat paper. The word “cat” does not appear in the index. People
who were interested in Schrödinger knew about his cat – John Bell’s mention of
it makes it clear that he expects people to know what he’s talking about – but
it wasn’t regarded as something of serious scientific interest. But since then
there have been over 300 citations of the original paper in German; I didn’t
count references to the English translation, which appeared in 1980 and was
reprinted in an influential collection of papers edited by Wheeler and Zurek
[9]. Never mind citations: last year alone there were 119 papers with
“Schrödinger’s cat” in the title. And these are not fuzzy philosophy: about
two-thirds of these papers are hard physics, concerned with actual
experiments.

I’ve said that the standard idea of Schrödinger’s cat, i.e. a cat which is, on
its own, in a superposition of alive and dead, is nullified by the fact that
this state cannot, in practice, appear on its own, because it is continually
being entangled with its environment: with molecules in the atmosphere, with
electromagnetic radiation, and so on. But that state of an isolated system is
what has come to be known as a “Schrödinger cat state”: an actual state of a
macroscopic object, separated from the rest of the universe, isolated from any
interaction with anything else, and in a superposition of two states which we
would recognise as being different at a macroscopic level. So there is a
challenge: can experimenters actually make objects which really are in a
superposition in this sense?

DISENTANGLED ADVENTURES

Nobody has done it with a cat (yet), but there is talk of doing it with a
living object. In 1996 Wineland’s group constructed a superposition state of a
single atom [4], demonstrated by the fact that a beam of these atoms, put
through two slits, did show interference as in my second slide. Three years
later Zeilinger’s group did it with a large molecule [1]. If you can do it
with a large molecule, why not a virus [5] – a virus is alive (sort of) – and
if with a virus, why not a microbe?

## References

  * [1] Arndt M, Nairz O, vos Andreae J, Keller C, van der Zouw G, and Zeilinger A, 1999.  Wave-particle duality of C60 molecules.  Nature, 401, 680–682.
  * [2] Jammer M, 1989.  The conceptual development of quantum mechanics.  American Institute of Physics, second edition.  p.344.
  * [3] Kilmister K W, editor, 1987.  Schrödinger: centenary celebration of a polymath. Cambridge University Press.
  * [4] Monroe C, Meekhof D M, King B E, and Wineland D J, 1996.  A ”Schrödinger cat” superposition state of an atom.  Science, 272, 1131–1136.
  * [5] Romero-Isart O, Juan M L, Quidant R, and Cirac J I, 2010.  Toward quantum superposition of living organisms.  New Journal of Physics, 12, 033015.  arXiv:0909.1469.
  * [6] Schrödinger E, 1935.  Die gegenwärtige situation in der quantenmechanik.  Naturwissenschaften, 23, 807.  English translation by J. D. Trimmer in [9].
  * [7] Schrödinger E, 1996.  The Interpretation of Quantum Mechanics: Dublin Seminars (1949-1955) and other unpublished essays.  Ox Bow Press.  ed. M. Bitbol.
  * [8] Sudbery A, 2017.  The logic of the future in quantum theory.  Synthese, 194, 4429–4453.  arXiv:1409.0755.
  * [9] Wheeler J A and Zurek W H, editors, 1983.  Quantum Theory and Measurement.  Princeton University Press.