in each segment are identified. We define a system as a set of subsequent
lines that belong to each other. For example: a tab system is very common in
guitar tablature files and consists of exactly six subsequent tab files. In
chord sheets, a common system is the alteration between chord lines and lyrics
lines. From these systems, decibel can then extract the chord labels. Thereby,
the system retains line information (i.e. the line of the chord in the text
file), as this will be used in the tab-audio alignment step. The steps of line
type classification, segmentation and system and chord extraction are
explained more thoroughly in the remainder of this section.

##### Line type classification

For each line in the tab file, decibel estimates its line type using a set of
heuristics. We distinguish the following line types: empty, chords, tuning
definition, capo change, structural marker, chord definition, tablature,
lyrics and combined chord and lyrics.

  * •

A line has the empty line type if it is empty or consists only of a space.

  * •

To determine if a line has the chords line type, the system first splits the
line by spaces. Then it checks if each element of the line matches the chord
pattern. An element is a chord if:

    * –

It consists of at most 10 characters;

    * –

The first character is a note letter (a, b, c, d, e, f or g);

    * –

The element does not contain sequences of four digits (as this would indicate
a chord definition);

    * –

The element does not contain symbols; and

    * –

Any three letter sequence is in the following list: min, add, aug, dim, maj,
sus, flat.

  * •

A line has the tuning line type if it contains the word “tuning”.

  * •

To find out if a line is a structural marker, our parser searches for words
like “verse”, “chorus” or “bridge”.

  * •

A line is a chord definition if it contains a sequence of exactly 6 digits.

  * •

A line is a tablature line if it contains at least 10 characters which are a
digit, hyphen, vertical bar, slash, letter ‘h’, ‘b’ or ‘p’ or a space and the
number of hyphens is larger than the number of spaces.

  * •

A lyrics line fulfills each of the following conditions:

    * –

It does not contain square brackets, the equals sign or the at sign;

    * –

It contains at most 10 hyphens;

    * –

Either:

      * *

It consists of just one word, which has only letter characters and contains at
least three of the same letters after each other, like ‘ooooh’ or ‘aaah’; or

      * *

It contains at least two words.

  * •

To find out if a line is a combined chords and lyrics line, decicbel first
extracts all characters between square brackets ([ and ]). If these characters
form a chord, it removes all characters between each pair of square brackets.
If the remaining line is a lyrics line, we can conclude that our original line
was a combined chords and lyrics file.

  * •

If a line does not satisfy any of these requirements, the parser assigns the
line type undefined.

##### Segmentation

At this point, we have line types for each line in the tab. As a next step,
our parser divides the tab file into segments. A segment is simply defined as
the lines between lines of the empty line type. In a typical tab, verses and
choruses are separated by empty lines and therefore are designated to their
own segment.

##### System and chord extraction

Now we have our tab file divided into segments, and we know the line type of
each of the lines in a segment. We want to extract the chord labels, but we
only extract them from a suitable system. A system consists of all of lines
that should be played or sung together at the same time. For example, in a
typical chord sheet, we will find a lot of systems consisting of a chord line
above a lyrics line, as the chords in the chord line should be played at the
same time as the word in the lyrics line should be sung. The six different
system types are listed in the first column of Table 6.1.

System type | Chord Extraction
---|---
Combined chords and lyrics line | Parse chords from combined chords and lyrics line
Chord line | Parse chords from chord line
| Chord line
---
Exactly six tablature lines
Parse chords from chord line
| Chord line
---
Exactly six tablature lines
1 to 3 lyrics lines
Parse chords from chord line
Exactly six tablature lines | Parse chords from six tablature lines
| Exactly six tablature lines
---
1 to 3 lyrics lines
Parse chords from six tablature lines
Table 6.1: System types and their chord extraction methods

Decibel assigns each line to the largest possible system. Consider for example
a segment consisting of three chord definition lines, followed by a chord
line, six tablature lines and one lyrics line. Our parser finds only one
system, consisting of a chord line, six tablature lines and a lyrics line. In
this case, the six tablature lines do not form a system on their own, as the
chord line and lyrics line are also played and sung at the same time. This has
to do with the very reason why we consider systems: we only extract one chord
sequence from a system, to prevent duplicate chord sequences. This is
particularly important in guitar tablatures, as they often have a chord line
directly followed by six tab lines with exactly the same chord information. In
that case, the parser ignores the six tab lines and extract the chords from
the chord line only. For each of the systems, the chord extraction method is
displayed in the second column of Table 6.1. We see that there are basically
three chord extraction methods:

  * •

Parse chords from combined chords and lyrics line: our parser extracts all
characters between square brackets ([ and ]), so we have a list of strings, in
which each string represents a chord. Then it parses each string by splitting
it into a root note, chord type and optional bass note using regular
expressions. For example, the chord string “D7/F#” has root note “D”, chord
type “7” (dominant seventh chord) and bass note “F#”. Based on these three
substrings, the parser now converts the chord string to Harte’s chord notation
(Harte et al., 2005). It also saves the line number and character index of the
start of the chord string.

  * •

Parse chords from chord line: the parser extracts chord strings from the chord
line by splitting on spaces and “|” characters. Then it parses each string and
converts it to Harte’s chord notation, line numbers and character indices, as
described above.

  * •

Parse chords from six tablature lines: for each character index ($x$-value),
the parser extracts the six characters that are notated above each other. If
all characters are either a digit of an ‘x’, a chord is played at this
position. The parser then derives the notes played at the same time based on
these characters and their string - for example: if a digit ‘1’ appears on the
B string line, the corresponding chord is a C; the digit ‘2’ on the B string
corresponds to a C#. Finally it finds the nearest chord from our chord
vocabulary (based on cosine distance). Again, it also saves line number (of
the first tab line) and character index.

####  6.1.2 Tab parsing evaluation

In this section, we describe the evaluation of the tab parsing algorithm. For
the evaluation of the tab parsing algorithm, we used a random sample of 25
tabs from the data set, as listed in Appendix C. For these 25 tabs, we
annotated the untimed chord sequence by hand. Then, we ran decibel’s tab
parsing algorithm on these 25 tabs and compared the result to the annotation.

Figure 6.2: Recall of chords in our sample of 25 tab files.

The results are shown in Figure 6.2, in which the height of each bar
corresponds to the number of chords in the annotation of a tab file. The True
Positives (TP) are the chords in the annotation which are also in the parsed
results. In the figure, we see them as the blue part of the bar. The orange
part of the bar corresponds to the False Negatives (FN): the chords that are
in the annotation of a tab file, but not found by our parser. As we see in
this figure, the parser finds (almost) all chords in most of the songs, as the
orange part is very small. There were no chords in either the annotation or
the parser output of the third, tenth and 23rd song. For the other songs,
chords were missing in the tab output for the following reasons:

  * •

The parser does not recognize chords that are glued after each other (without
a separator) in the tab file;

  * •

The parser does not recognize chords in a line with more text or strange
symbols, as such a line is not classified as a chord line;

  * •

The parser does not recognize chords in guitar tablature systems when there
are no digits or ’x’ in all six lines.

In some songs, text that was actually not a chord was misinterpreted as
chords. This happened in one tab file which added some rhythm information by
for example putting a ’Q’ above a chord symbol for quarter note durations and
an ’E’ for eighth note durations. Our parser falsely detected a line of eighth
notes as a sequence of E chords. In another line, chord vocabulary information
was falsely detected as a chord line.

Still, in most songs we see that the tab parser performs quite well, as there
are no missing chords in 15 of the 25 songs.

###  6.2 Jump Alignment

Having completed the tab parsing step, we have extracted the chord labels and
their corresponding line and word numbers from the tab file. However, tab
files retain no timing information, so we need an additional step to align the
chord labels to the audio file. As we have seen in Section 1.3, there already
exist four different algorithms by McVicar et al. (2011b) that incorporate tab
information into a HMM-based system for audio chord estimation. To the best of
our knowledge, these are the only algorithms that use tabs in audio ace. The
most promising of these four algorithms is Jump Alignment. In this section, we
describe and evaluate our implementation of Jump Alignment.

Jump Alignment is based on a Hidden Markov Model (HMM). A HMM models the joint
probability distribution $P(X,y|\Theta)$ over the feature vectors $X$ and the
chord labels $y$, where $\Theta$ are the parameters of the model. In Section
6.2.1, we describe the preprocessing steps that are used for extraction of the
features $X$ and the chord labels $y$. Section 6.2.2 gives an introduction to
the HMM. The details on the Jump Alignment algorithm are treated in Section
6.2.3.

####  6.2.1 Preprocessing

First, the audio file needs to be preprocessed. For this purpose, we use the
python package librosa (McFee et al., 2018). First, we convert the audio file
to mono. Then, we use the HPSS function to separate the harmonic and
percussive elements of the audio. Then, we extract chroma from the harmonic
part, using constant-Q transform with a sampling rate of 22050 and a hop
length of 256 samples. Now we have chroma features for each sample, but we
expect that the great majority of chord changes occurs on a beat. Therefore,
we beat-synchronize the features: we run a beat-extraction function on the
percussive part of the audio and average the chroma features between the
consecutive beat positions. The chord annotations need to be beat-synchronized
as well. We do this by taking the most prevalent chord label between beats.
Each mean feature vector with the corresponding beat-synchronized chord label
is regarded as one frame. Now we have the feature vectors $X$ and chord labels
$y$ for each song, which we feed to our HMM.

####  6.2.2 Hidden Markov Model

As we have seen in Section 4.1.2, a HMM models the joint probability
distribution $P(X,y|\Theta)$ over the feature vectors $X$ and the chord labels
$y$, where $\Theta$ are the parameters of the model. The HMM is an extension
of the Markov chain in which the states are not directly observable (but
hidden). In this section, we give a summary of the Hidden Markov Model, based
on the tutorial by Rabiner (1989) and Section 5.3.2 from Müller (2015).

In a Markov chain, we make the assumption that the probability from the
current state $s_{t}$ to the next state $s_{t+1}$ only depends on the current
state, and not on any of the previous states. Formally:

$P(Y_{t+1}=s_{t+1}|Y_{1}=s_{1},Y_{2}=s_{2},\ldots,Y_{t}=s_{t})=P(Y_{t+1}=s_{t+1}|Y_{t}=s_{t})$
(6.1)

This assumption is called the Markov property. In our implementation, states
are represented by chord labels. Any chord label is an element from the
discrete set of chords from our chord vocabulary. Although we could use any
chord vocabulary, we use the vocabulary with all major and minor chords and
the no-chord symbol. Our vocabulary thus consists of 25 items. Thanks to the
Markov property, we can efficiently model the probability that chord $j$
occurs after chord $i$ by the transition probability
$P_{tr}(y_{t}=j|y_{t-1}=i)$. Given a chord vocabulary of 25 items, we can
model all transition probabilities in $25\cdot 25=625$ probability values,
that can be efficiently stored in the matrix $\mathbf{P_{tr}}$.

The first chord of a chord progression cannot be computed from its previous
chord. This information is specified by the initial state probabilities. There
are 25 initial state probability values, stored in the vector
$\mathbf{P_{ini}}$, in which $P_{ini}(y_{1}=c)$ is the probability that the
first chord is chord $c$.

Based on the Markov chain, we could now compute the probability for a given
chord label given a sequence of chord labels, or compute the probability of
any chord label sequence. However, in a ace scenario, we cannot directly
observe the chord labels, but we do observe a sequence of chroma features that
is related to the chord labels. We thus need to expand our model, by adding
emission probabilities. The resulting model is a Hidden Markov Model. The
emission probability $P_{obs}(X_{t}|y_{t})$ equals the probability of an
observation (chroma feature vector) $X$ given a chord label $y$ at time $t$.
In contrast to transition and initial state probabilities, the emission
probabilities cannot be represented by a matrix, as chroma features are
continuous. Instead, they are modeled by continuous probability density
functions: the observation probability distribution for each of the 25 chord
labels $a_{i}$ in the vocabulary as a 12-dimensional Gaussian with
12-dimensional mean vector $\mu_{i}$ and $12\times 12$ covariance matrix
$\Sigma_{i}$. We can now find our emission probabilities for any observed
chroma feature vector $x_{t}$ and any chord label $a_{i}$, using the Gaussian
with parameters $\mu_{i}$ and $\Sigma_{i}$:

$P_{obs}(x_{t}|\mu_{i},\Sigma_{i})=\frac{1}{(2\pi)^{6}|\Sigma_{i}|^{1/2}}\exp[-\frac{1}{2}(x_{t}-\mu_{i})^{T}\Sigma^{-1}(x_{t}-\mu)]$
(6.2)

We now have seen that a HMM is parametrized by transition, initial state and
emission probabilities. We define $\Theta$ as the set of these parameters:

$\Theta=\\{\mathbf{P_{tr}},\mathbf{P_{ini}},\\{\mu_{i}\\}_{i=1}^{25},\\{\Sigma_{i}\\}_{i=1}^{25}\\}$
(6.3)

There exist different ways of estimating $\Theta$. We adopt the machine
learning approach, that is: we train the probabilities on a subset of our data
set. In our case, the training set is a 100-song random sample of our data
set. The initial state and transition probability matrices are calculated
simply by frequency counting and normalization of all chord labels in the
training set. For calculating $\mu_{i}$ and $\Sigma_{i}$, we take the set of
all feature vectors that are labeled with chord $i$ in our data set, and then
calculate the mean vector and covariance matrix respectively.

Now we know all elements of the HMM, we can represent the joint representation
for the chroma feature vectors $X$ and chord labels $y$ mathematically in the
following equation:

$P(X,y|\Theta)=P_{ini}(y_{1}|\Theta)\cdot
P_{obs}(x_{1}|y_{1},\Theta)\cdot\prod_{t=2}^{|y|}P_{tr}(y_{t}|t_{t-1},\Theta)\cdot
P_{obs}(x_{t}|y_{t},\Theta)$ (6.4)

HMMs can be used to solve three basic problems, as pointed out in Rabiner
(1989). One of these problems, which is called the uncovering problem or
decoding problem, is the relevant problem for our purpose: automatic chord
estimation. The goal of the uncovering problem is to find the state (chord)
sequence that best explains the observed (chroma) feature sequence. Formally:
we want to find the chord label sequence $y^{*}$ where
$y^{*}=\operatorname*{argmax}_{y}P(y|X,\Theta)$. Note that this is equivalent
to finding $y^{*}=\operatorname*{argmax}_{y}P(y,X|\Theta)$.

Note that the optimal chord sequence for an observation sequence of $n$
elements starts with the optimal chord sequence for the observation sequence
of the first $n-1$ elements. The Viterbi algorithm efficiently solves the
uncovering problem by exploiting this property. The pseudocode of Viterbi is
given in Algorithm 2. In each iteration, the algorithm computes the indices
$i_{1},i_{2},\ldots,i_{t}$ to the optimal chord sequence for
$X_{1},X_{2},\ldots,X_{t}$ (i.e. the first $t$ steps of the observation
sequence $X$) ending with chord label $y_{i_{t}}$. The result is then stored
in the matrices $V$ and $B$: in $V$, the algorithm keeps the likelihood of the
current path, while the chord index is stored in $B$. When calculating a chord
sequence of length $n$, the likelihoods of the optimal chord sequences of
length $n-1$ can be efficiently retrieved by reading the corresponding value
from matrix $V$. In the termination phase, we can read the final chord index
$i_{T}$ by maximizing over the last column of $B$ and find the full optimal
chord sequence by following the backpointers in $B$ back in time. We can also
read the likelihood of the full optimal chord sequence from $V[i_{T},T]$.

1:function Viterbi(Observation Sequence $X_{1},X_{2},\ldots,X_{T}$, HMM
specified by $\Theta$, Chord alphabet $y_{1},y_{2},\ldots,y_{25}$)

2: Initialize Viterbi Matrix $V$ of size $25\times T$ $\triangleright$
Initialization

3: Initialize Backpointer Matrix $B$ of size $25\times(T-1)$.

4: for all $i\in[1,25]$ do

5: $V[i,1]=\mathbf{P_{ini}}[y_{i}]\cdot P_{obs}(y_{i}|X_{1})$

6: end for

7: for all $t\in[2,T]$ do $\triangleright$ Recursion step

8: for all $i\in[1,25]$ do

9:
$V[i,t]=\operatorname*{max}_{j=1}^{25}V[j,t-1]\cdot\mathbf{P_{tr}}[y_{j},y_{i}]\cdot
P_{obs}(y_{j}|X_{t})$

10:
$B[i,t-1]=\operatorname*{argmax}_{j=1}^{25}V[j,t-1]\cdot\mathbf{P_{tr}}[y_{j},y_{i}]$

11: end for

12: end for

13: $i_{T}=\operatorname*{argmax}_{j=1}^{25}V[j,T]$ $\triangleright$
Termination

14: for all $t\in[T-1,1]$ do

15: $i_{t}=B[i_{t+1},t]$

16: end for

17: return Chord Sequence $y_{i_{1}},y_{i_{2}},\ldots,y_{i_{T}}$ and
Likelihood Value $V[i_{T},T]$

18:end function

Algorithm 2 Pseudocode of the Viterbi algorithm

####  6.2.3 Jump Alignment

Jump Alignment is an extension to the HMM described in the previous section,
which utilizes the chords that are parsed from tabs. Following McVicar et al.
(2011b), we refer to these chords parsed from tab files as Untimed Chord
Sequences (UCSs). Compared to the original HMM, in the Jump Alignment
algorithm the state space and transition probabilities are altered in such a
way that it can align the UCSs to audio, while allowing for jumps to the start
of other lines. We will explain how this works in the remainder of this
section.

Let us assume that we have an audio file and tab file from the same song. Let
$\mathbf{e}=e_{1},e_{2},\ldots,e_{L}$ be the UCS we extracted from the tab
file, applying the parsing algorithm described in Section 6.1. Because we want
an alignment between the audio file and this UCS $\mathbf{e}$, we define a new
set of hidden states: $Y^{\prime}=\\{1,2,\ldots,L\\}$ corresponds to the
indices of the chords in $\mathbf{e}$. For example, given a (very small) chord
vocabulary $[C,D,E]$ and the UCS $[C,E,D,E,D,C,E]$, our hidden state set
$Y^{\prime}$ would be $[1,3,2,3,2,1,3]$. Note that different hidden states can
now refer to different occurrences of the same chord, in contrast to our
original state space.

We also need to alter the transition probability distribution
$P^{\prime}_{tr}$, as we only want to allow chord sequences that are
alignments of $\mathbf{e}$, our Untimed Chord Sequence. However, McVicar et
al. (2011b) showed that exact alignment of $\mathbf{e}$ to audio results in a
decrease in performance compared to the original HMM, because repetition cues
(like “play this verse twice”) are not interpreted by the tab parser. In
addition, some tabs do not specify the chords of all verses. Therefore, Jump
Alignment allows jumps from the end of a line in the tab file to the beginning
of any line (the current one, of any of the previous or subsequent lines). The
new transition probability distribution is expressed as:

$P^{\prime}_{tr}(j|i,\Theta,e)=\begin{cases}\frac{1}{Z_{i}}P_{tr}(e_{i},e_{j})&\text{if
}j\in\\{i,i+1\\},\\\ \frac{p_{f}}{Z_{i}}P_{tr}(e_{i},e_{j})&\text{if
}i<j\text{ and }i\text{ is the end and }j\text{ is the beginning of a
line},\\\ \frac{p_{b}}{Z_{i}}P_{tr}(e_{i},e_{j})&\text{if }i>j\text{ and
}i\text{ is the end and }j\text{ is the beginning of a line},\\\
0&\text{otherwise}.\end{cases}$ (6.5)

$Z_{i}$ is a normalization factor, which is used to re-normalize
$P^{\prime}_{tr}$ so that the transition probabilities
$P^{\prime}_{tr}(y_{j}|y_{i},\Theta,e)$ sum to one. $p_{f}$ is the probability
of jumping forward when this is allowed. Likewise, $p_{b}$ is the probability
of a jump backwards, provided that this is allowed.

The forward probability $p_{f}$ and backward probability $p_{b}$ need to be
specified. We estimate the optimal $p_{f}$ and $p_{b}$ as follows: first, we
manually select a subset of 22 songs for which we are sure that they are in
the right key (so we do not need to consider transpositions). Appendix D lists
these songs. Then, we run the Jump Alignment algorithm with all combinations
of forward and backward probabilities between 0 and 1 in steps of 0.05 and
evaluate the resulting chord label sequences in terms of csr. We will use the
$p_{f}$ and $p_{b}$ with the highest average csr.

Figure 6.3 shows the average csr for our 22-song subset, given a
$(p_{f},p_{b})$ parameter setting. We see that positive forward or backward
probabilities improve csr compared to a strict alignment (in which $p_{f}=0$
and $p_{b}=0$): particularly non-zero backward probabilities yield
considerably better chord label sequences than $p_{b}$’s of zero. However, if
the forward and backward probabilities become too large, the average csr
decreases as the importance of the order of lines in the tab file decreases.
We find an optimal parameter setting of $p_{f}=0.05$ and $p_{b}=0.05$. This is
the parameter setting that we use in our implementation of Jump Alignment.

Figure 6.3: Average csr for 22-song subset given $p_{f}$ and $p_{b}$. We find
an optimal parameter setting of $p_{f}=0.05$ and $p_{b}=0.05$.

Tabs are often notated in a transposed key compared to the original audio
file, because some keys are easier to play on the guitar than others. In order
to correct for such “simplified” tabs, Jump Alignment considers the tab files
in all 12 transpositions and chooses the transposition with the highest
likelihood.

###  6.3 Tab file selection

The performance of the jump alignment algorithm is dependent on the quality of
the tab file. Following McVicar et al. (2011b), we select for each song the
tab file for which the log-likelihood was the highest. The results for our
full 200 song dataset are shown in Table 6.2. If we would take the average csr
of all tabs for a song, the wcsr is 72.4%. The upper limit for the selection
method is a wcsr of 78.7%: this is what we would get if we selected the best
tab. Since we do not know the csr for unlabelled data, we select the expected
best tab-file for each song based on the log-likelihood, following McVicar et
al. (2011b). By choosing the tab file with the highest log-likelihood for each
song, the resulting wcsr improves to 75.3%. The distribution of csrs of
selected tab files compared to all tab files is also shown in Figure 5.8.

| wcsr
---|---
Worst csr of all tabs | 59.1%
Average csr of all tabs | 72.4%
Best log-likelihood of all tabs | 75.3%
Best csr of all tabs | 78.7%
Table 6.2: wcsr of all songs, with different tab file selection methods

###  6.4 Conclusion

In this section, we have examined the last of the three representation-
specific subsystems of decibel: the tab subsystem. This subsystem consists of
a parser that extracts Untimed Chord Sequences (UCSs) from guitar tablature or
chord sheets, thereby retaining line information. This is then input for the
Jump Alignment algorithm, which aligns the UCSs to the audio recording. When
selecting the estimated best tabs for each song, based on log-likelihood, this
subsystem reaches a wcsr of 75.3% on the Isophonics data set.

##  Chapter 7 Data fusion

Decibel estimates chord label sequences from different music representations,
i.e. audio, midi and tab files, as we have seen in the previous three
chapters. This results in a set of chord label sequences for each song in our
data set. This set of chord labels forms a rich harmonic representation that
is already interesting in itself, as we will see in Section 7.1. However, in
order to answer our research question we need to combine these chord label
sequences into one final sequence (and compare the resulting chord sequence to
the sequence obtained by using only an audio ace method). Decibel achieves
this using a data fusion step. In this chapter, we motivate the chosen data
fusion method by a literature study and an evaluation of various methods.

Midi-based Audio-timed Chord Sequence Audio-based Audio-timed Chord Sequence
Tab-based Audio-timed Chord Sequence Data Fusion Output Chord Sequence Figure
7.1: Diagram of decibel’s data fusion subsystem. The set of midi\- audio- and
tab-based audio-timed chord label sequences forms a rich harmonic
representation. Our data fusion combines all chord label sequences for a song
into a single output chord label sequence.

###  7.1 A rich harmonic representation

In the previous chapters we have seen how decibel’s subsystems estimate chord
label sequences from audio, midi and tab representations of a song. These
three representations store musical content of the same song in fundamentally
different ways: as we have seen in Section 2.5, audio files are obtained by
digitizing the waveform; midi files are a series of note on and note off
events; whereas tab files show the guitar fingering or chord labels, aligned
to the lyrics. The timing information in audio is represented in seconds,
based on the underlying performance; in midi files, timing is measured in
ticks; in tabs, timing information is missing. Also, the source of information
differs: audio files are recordings of a performance; midi files are either
score-based or transcriptions of a recording; tabs can be considered as
(untimed) chord label sequences, manually annotated by music enthusiasts.

However, the chord label estimation and synchronization steps performed by
decibel’s subsystems transform these three heterogeneous representations into
a homogeneous format, i.e. as a series of $\langle$start time, end time, chord
label$\rangle$ 3-tuples. The combination of the set of chord label sequences
from each of the representations of a song results in a rich harmonic
representation.

This rich harmonic representation is a very interesting by-product of the
decibel system, as it allows for cross-version analysis, i.e. comparing
analysis results from different representations. The chord label sequences can
easily be visualized, which makes it very easy to see the consistencies and
inconsistencies in chord labels between different representations. As
identified earlier in research by Ewert et al. (2012) and Konz et al. (2013),
having such a unified view of different analysis results deepens the
understanding of both the algorithm’s behavior and the properties of the
underlying musical material.

Consider for example the visualization of the chord label sequence for the
Beatles song Golden Slumbers in Figure 7.2. In our data set, we have matched
three midi files and eight tab files to the audio version of the song.
Cassette, our midi ace method, analyzes the chords both on a bar and a beat
level, so we have six chord label sequences based on the three midi files. In
combination with the eight chord label sequences based on the tab files and a
single analysis by the Chordify algorithm based on the audio file, we have
$6+8+1=15$ chord label sequences. Each of these estimated chord sequences, as
well as the ground truth, is visually represented by a horizontal bar in which
the color represent the chord label and time in seconds can be read from the
position on the $x$-axis. In this example, we observe there are some parts of
the song for which (most of) the 15 estimated chord sequences agree on the
labels, e.g. the D minor chord starting after 11 seconds; the G major chord
sounding immediately thereafter or the C major chord that starts at 32
seconds. However, in other musical passages there is some disagreement. If
there is a single deviating chord sequence estimation, this is typically
either due to some error in the midi or tab file or due to an error in the
estimation method. For example, when estimating the chord sounding in the 38th
second from our third midi file, using cassette on the bar level, we falsely
find a F major chord instead of a C major chord. In some passages, we observe
multiple “clusters” of possible chord labels, i.e. different perspectives on
the harmonic content of a musical passage, each of them supported by multiple
chord label estimations. For example, we can consider the start of the song
either as a long A minor chord or as a sequence of A minor and C major chords.
Also, there exist different perspectives on the chord in the 23rd second (is
it a C major, E major, F minor or something else?) and the modality of the A
(major or minor) chord starting after 62 seconds.

Figure 7.2: Multiple perspectives on the Beatles song Golden Slumbers: we can
consider the start of the song as a long A minor chord or as a sequence of A
minor and C major chords and there are multiple opinions on e.g. the modality
of the chord starting after 62 seconds.

Midi and tab files implicitly incorporate musical knowledge, as they are (in
the popular music genre) typically transcriptions made by music enthusiasts.
Therefore, the comparison of analysis results from different music
representations reveals passages in the music for which there are multiple
possible perspectives. Studying these passages can give us a deeper
understanding of subjectivity issues in the musical material.

In this section, we have seen that the chord label sequences estimated on
different music representations of the same song together form a rich harmonic
representation. Visualization of this harmonic representation can give
valuable insights in different perspectives on chord labels within a specific
segment of the song. This makes large-scale cross-version analysis feasible.
Moreover, it bridges the gap between technical, audio-oriented, musical signal
processing and non-technical, score-oriented musicology. A detailed analysis
on the different harmonic perspectives on the Beatles and Queen songs from our
data set is beyond the scope of this research, but would be an interesting
direction of future work which would provide better insights in the
subjectivity issues in harmonic analysis of popular music.

###  7.2 Related Work on data fusion

In the remainder of this chapter, we study data fusion methods to combine
chord label sequence obtained from decibel’s audio, midi and tab subsystems
into a single output sequence. The integration of heterogeneous output of
multiple ace algorithms using data fusion is a new idea, which was recently
proposed by Koops et al. (2016). In this study, the authors experiment with
three different techniques to combine chord sequence estimates from different
sources (the mirex 2013 ace submissions, applied to the Billboard data set)
into one final output sequence for each song. They show that their proposed
data fusion method yields the best results in terms of wcsr. Also, they show
that the output sequence found by their data fusion method is an improvement
to the best scoring team, with an increase between 3.6 percentage point and
5.4 percentage point compared to the best team.

The three techniques used for combining chord label sequences are random
picking (rnd), majority voting (mv) and data fusion (df). In all three
techniques, the system proposed in Koops et al. (2016) first samples the chord
label sequence from each source in segments of 10 milliseconds. In the rnd
technique, the output chord label for each segment is found by picking the
corresponding label from a randomly chosen source. For mv, the output chord
label for each segment is the most frequent label of all sources. If multiple
labels are most frequent, the output label is picked randomly from these most
frequent labels. The df technique takes three things into account when
integrating the output of multiple ace algorithms:

  * •

The accuracy of sources;

  * •

The probabilities of the values provided by the sources; and

  * •

The probability of dependency between sources.

The source accuracy $A(S_{i})$ is the arithmetic mean of the probabilities of
all chord labels a source $S_{i}$ provides. Initially, these chord label
probabilities may be based on a frequency count. A source with a high source
accuracy inherently has a lot of chord labels that agree with other sources,
and therefore can be considered more trustworthy. Assuming that the sources
are independent, the source accuracy is the probability that a source provides
the appropriate chord. On the other hand, the probability that a source
provides an inappropriate chord can be computed with $\frac{1-A(S_{i})}{n}$,
given $n$ possible inappropriate chord labels. The source vote count
$VS(S_{i})$ combines these probabilities and is computed as follows:
$VS(S_{i})=\ln\frac{n\cdot A(S_{i})}{1-A(S_{i})}$. Chord labels of sources
with higher source vote counts are more likely to be picked than labels of
sources with low source vote counts, as we will see shortly.

The goal of the data fusion method is to determine the most likely chord label
for each segment, given a number of sources. To this end, the data fusion
method computes a chord label vote count $VC(\mathcal{L})$ for each chord
label $\mathcal{L}$. The chord label vote count is dependent on both the
number of sources with a matching chord label and the source vote count of
these sources, as expressed in the formula $VC(\mathcal{L})=\sum_{\sigma\in
S^{\mathcal{L}}}VS(\sigma)$, in which $S^{\mathcal{L}}$ is the set of sources
with chord label $\mathcal{L}$.

From the label vote counts, we can compute chord label probabilities for each
chord label $\mathcal{L}$ by
$P(\mathcal{L})=\frac{\exp(VC(\mathcal{L}))}{\sum_{l\in D}\exp(VC(l))}$. As we
divide the chord label vote count of our chord label by the chord label vote
count of all possible chord labels $D$, this results in a probability value
between 0 and 1.

The data fusion method proposed by Koops et al. (2016) also takes the
dependency between sources into account, following the intuition that sources
that are dependent of each other share many uncommon chord labels. Based on
the number of shared uncommon chords with other sources, the authors compute a
source dependency weight $\mathcal{I}(S_{i},\mathcal{L})$, which represents
the probability that a source $S_{i}$ provides a chord label $\mathcal{L}$
independently.

We now have seen the definitions of chord label probabilities, source accuracy
and source dependency. Note that they are all defined in terms of each other.
As a solution to this paradoxical situation, Koops et al. (2016) propose to
initialize the chord label probabilities with equal probabilities and
iteratively compute source dependency, chord label probabilities and source
accuracy until convergence is reached. Finally, their data fusion method
selects for each segment the chord label with the highest chord label
probability.

In their experiments, the authors compare the performance of df to rnd and mv
for two subsets of the Billboard data set. Each of these three chord sequence
combination methods is computed on all twelve mirex 2013 ace submissions for
both Billboard subsets. The Billboard annotations serve as ground truth. Their
results show that the df method significantly outperforms both the rnd and mv
methods as well as each of the individual submissions. We can conclude that
combining chord label sequences from multiple sources using data fusion is a
promising strategy to improve ace.

###  7.3 Data fusion experiments

In the previous section, we have seen that the heterogeneous output of
multiple ace systems can be integrated into a single output chord sequence,
using data fusion. Although the results obtained in previous work are solely
based on audio data, the data fusion method can also be applied to chord label
sequences that are based on symbolic data. This is exactly what decibel’s data
fusion subsystem does: the subsystem uses the chord sequences from audio, midi
and tabs and integrates them in a final output sequence. In contrast to
earlier work, in our case the number of sources differs per song. That is why
we experiment with a selection strategy that elects only one source per
representation. In this section, we will explain decibel’s data fusion
implementations.

In order to test if decibel improves current audio ace systems, we need to
combine these chord label sequences into one final sequence. Decibel achieves
this using a data fusion step. In this section, we motivate the chosen data
fusion method by an evaluation of various methods. These methods are based on
earlier work by Koops et al. (2016). In contrast to earlier work, decibel
needs to combine a varying number of sources per song. That is why we
experimented with a selection strategy that selects only one source per
representation. We compare two selection strategies in combination with three
different integration methods. The two selection strategies are all and best:
all takes the chord sequences of all tabs and midi files as sources. Best only
uses the sources of the expected best tab and midi file for each song, as
described in Section 5.3 and 6.3. There are eight songs for which no Midi is
selected, since all midis were badly aligned. For these songs, Best only
combines the audio and tab sequences.

The three integration methods are based on earlier work (Koops et al., 2016).
We first divide each input chord sequence in 10 millisecond samples. Then we
integrate the sources, selected using all/best, with either random picking
(rnd), majority voting (mv) or data fusion (df). The implementations of rnd
and mv are unchanged compared to Koops et al. (2016): given a specific sample,
rnd takes the chord label of an arbitrary source, while mv assigns the chord
label used by most of the sources.

Our implementation of the df technique takes into account both the expected
accuracy of sources and the probability of the labels provided by the sources.
Let $\mathcal{X}$ be the set of samples, $\mathcal{V}$ be the chord
vocabulary, $\mathcal{S}$ be the sources selected by the integration method
and let
$\mathcal{L}:\mathcal{S}\times\mathcal{X}\times\mathcal{V}\mapsto\\{0,1\\}$ be
a labelling function such that $\mathcal{L}(s,x,v)=1$ if source $s$ assigns
chord $v$ to segment $x$ and $0$ otherwise.

The source accuracy $A[s]$ is the probability that a source $s$ provides
appropriate chords; the chord label probability $P[x,v]$ is the probability of
chord label $v$ at segment $x$, given the chord labels of the sources. The
chord label probability is defined based on the labelling $\mathcal{L}$ and
the chord label vote count $VC[x,v]$. This value determines the influence of
each source on the final label, based on its source accuracy. These values are
iteratively computed as follows:

$A[s]=\frac{\sum_{x\in\mathcal{X}}\sum_{v\in\mathcal{V}}P[x,v]\cdot\mathcal{L}(s,x,v)}{|\mathcal{X}|}$
(7.1)
$\textit{VC}[x,v]=\sum_{s\in\mathcal{S}}\mathcal{L}(s,x,v)\cdot\ln{\frac{(|\mathcal{V}|-1)A[s]}{1-A[s]}}$
(7.2)
$P[x,v]=\frac{\exp{\textit{VC}[x,v]}}{\sum_{v^{\prime}\in\mathcal{V}}\exp{\textit{VC}[x,v^{\prime}]}}$
(7.3)

$A[s]$, $\textit{VC}[x,v]$ and $P[x,v]$ are defined in terms of each other.
Therefore, we use an alternative equation for the initial computation of the
chord label probability $P_{0}[x,v]$:

$P_{0}[x,v]=\frac{\sum_{s\in\mathcal{S}}\mathcal{L}(s,x,v)}{|\mathcal{S}|}$
(7.4)

The final chord labelling for df is obtained by first computing Equation 7.4
and then repeatedly updating Equations 7.1–7.3. Whereas Koops et al. (2016)
repeat until a fixed point is reached, we always repeat 5 times. After this,
we assign the chord label $v$ with the highest probability $P[x,v]$ to each
segment $x$ to obtain the final df chord sequence.

###  7.4 Data fusion results

We now compare the performance of each of the six combinations of integration
methods and selection strategies (i.e. rnd-all, rnd-best, mv-all, mv-best, df-
all and df-best), applied to each of our ten audio ace systems. Friedman tests
(Friedman, 1937) for each of these ten audio ace show that the integration
methods and selection strategies give significantly different results in terms
of csr. Consequently, we perform a Tukey’s Honest Significant Difference post-
hoc test (Tukey, 1949) to identify which integration-selection combinations
are significantly different. The results are visually represented in Figure
7.3. In these figures, we can easily observe which differences are significant
because the corresponding horizontal lines do not overlap in the
$y$-direction. Consider for example Figure 7.3a. Here we see that both rnd
methods are significantly worse than the original audio ace system (chf).
There is no significant difference between chf and df-all. Mv-all is better
than chf, but the difference is not found significant by Tukey’s HSD test. Mv-
best and df-best perform significantly better than the original chf system,
and df-best turns out to be the winner.

(a) chf (2017)

(b) cm2 (2017) / cm1 (2018)

(c) jlw1 (2017)

(d) jlw2 (2017)

(e) kbk1 (2017)

(f) kbk2 (2017) / fk2 (2018)

(g) wl1 (2017)

(h) jlcx1 (2018)

(i) jlcx2 (2018)

(j) sg1 (2018)

Figure 7.3: Visual representation of the differences in terms of Chord Symbol
Recall between different data fusion methods, for chf and each of the mirex
2017 and/or 2018 ace submissions. For each of the horizontal lines that do not
overlap in the $y$-direction, the difference in csr between the corresponding
data fusion methods is significant.

When examining the test results for all ten audio ace systems for all ten
state-of-the-art audio ace systems in combination with all two selection
strategies and all three integration methods, we make four observations.

First, the random picking integration method always performs worse than the
original audio ace system, regardless of the chosen selection strategy. This
is consistent with findings in earlier work (Koops et al., 2016). From this
observation, we conclude that agreements between sources should be taken into
account.

Second, the best selection method always performs better than the all
selection method. The explanation for this is as follows: given a poor midi or
tab file, decibel’s midi or tab subsystem will find a poor chord label
sequence. In the all selection method, this chord label sequence will, to a
greater or lesser extent, be integrated in the final output sequence, which
deteriorates the output sequence’s quality, whereas the best selection method
(hopefully) ignores this poor chord label sequence.

Third, we observe that the difference between best and all is even larger for
the df integration method than for mv. A possible explanation for this is that
chord label sequences from tabs are often undersegmented, for example because
chord changes in instrumental parts or very short chords are often not
detected. This is typical for tabs and therefore can occur at multiple
(independent) tabs of the same song. Consequently, suboptimal chord sequences
from tabs could get a high source accuracy in df-all, and therefore
substantially influence the final output sequence. Df-best only considers the
expected best tab and midi file. If the expected best tab is undersegmented,
it will probably get a low source accuracy, because chord sequences from midi
and audio have less undersegmentation issues. The difference between mv-all
and mv-best is smaller, as mv does not use source accuracies.

Our fourth and final observation is that df-best performs better than mv-best,
although the difference is not always significant. However, df-best is always
better than the original audio ace system. As shown in Table 7.1, the
improvement is significant for all tested audio methods except jlcx1 and
jlcx2.

When comparing the oversegmentation measure for our integration methods, we
can do another observation: rand and mv tend to oversegment, in contrast to
df. This becomes clear when looking at the chord sequences for the song Things
We Said Today in Figure 7.4. Note that both rnd chord sequences have way too
many chord changes, resulting in a low oversegmentation score. Both mv chord
sequences are oversegmented as well, albeit to a lower degree. On the other
hand, we see no oversegmentation issues in the df chord label sequence.

Figure 7.4: Chord label sequences for the Beatles song Things We Said Today.
Csr = Chord Symbol Recall; OvS = oversegmentation; UnS = undersegmentation;
and Seg = segmentation. Note that rand and mv tend to oversegment, in contrast
to df.

From these observations, we can conclude that df-best is the best selection-
integration combination. Table 7.1 shows a comparison between the wcsr of each
of the ten audio ace systems and df-best applied to each system. The table
shows that using df-best improves ace wcsr on average by 3.05%.

Audio ACE | Mirex | Audio wcsr | Df-best wcsr | Improvement | Df-gt-best wcsr
---|---|---|---|---|---
chf | - | 82.0% | 84.6% | 2.6% | 86.3%
cm2/cm1 | 2017, 2018 | 75.7% | 81.7% | 6.0% | 84.1%
jlw1 | 2017 | 79.0% | 83.2% | 4.2% | 84.6%
jlw2 | 2017 | 78.5% | 83.0% | 4.5% | 84.3%
kbk1 | 2017 | 82.8% | 85.5% | 2.6% | 86.8%
kbk2/fk2 | 2017, 2018 | 87.3% | 88.2% | 0.8% | 88.9%
wl1 | 2017 | 79.9% | 83.6% | 3.8% | 85.0%
jlcx1 | 2018 | 86.3% | 87.1% | 0.9% | 88.1%
jlcx2 | 2018 | 86.5% | 87.1% | 0.6% | 88.1%
sg1 | 2018 | 79.5% | 84.0% | 4.6% | 85.8%
Table 7.1: wcsr of audio ace systems and df-best. Note that two of the 2017
systems were resubmitted in mirex 2018\. Significant improvements are shown in
bold. Using df-best improves ace wcsr on average by 3.05%.

Finally, let us look at the song-wise performance of df-best compared to the
original audio file. Figure 7.5 shows the difference between the csr of the
chord label sequence obtained by df-best and the csr of the chord label
sequence found by the original audio ace system. The song keys are given on
the $y$-axis, and the length of the horizontal bars correspond to the
difference in csr. If a bar is located on the right side of $x=0$, then the
chord label sequence found by df-best is better than the sequence found by the
original audio ace system. We see that this is the case for the vast majority
of songs in our dataset. On the other hand, if a bar is located on the left
side of $x=0$, df-best performs worse than the original audio ace system. From
all ten plots, we see one song for which df-best performs considerably worse.
This is song number 174, Let it be by The Beatles. This song has an issue
similar to Don’t pass me by, as we saw in Figure 4.5: the audio file that we
used was shifted compared to the audio file that was used in the mirex
competition. That is why the chord label sequence found by df-best is shifted
and therefore is not consistent with the Isophonics annotations any more. We
see another ‘peak’ to the left at song 196, We will rock you by Queen, for the
chf algorithm. The reason for this difference is that chf estimates the chord
label sequence for this song quite well, as shown in Figure 7.6. However, the
best found midi and tab file find very different chord labels for the first 90
seconds of the song and therefore the result found by df-best is based on the
tab file, which apparently had the highest source accuracy. In practice, this
song starts with a lot of percussion and little harmonic content. Therefore,
it is not so surprising that there are multiple views on the chord labels for
this song.

In this section we have seen that df-best is the best combination of the
selection strategy and integration method. Df-best performs better than the
original audio algorithm in terms of wcsr and does not suffer from
oversegmentation.

Figure 7.5: Csr of df-best compared to the csr of the original audio ace
system for each song in the dataset. The difference between the csr of df-best
and the csr of the original audio ace system is given on the $x$-axis; the
$y$-axis gives the index of the song in our data set. Horizontal lines on the
right side of $x=0$ correspond to songs for which df-best improves the
original system in terms of csr, while the horizontal lines on the left side
of $x=0$ correspond to songs for which the original ace system produced better
results in terms of csr. Figure 7.6: Chord label sequences for the Queen song
We will rock you. Note: csr = Chord Symbol Recall; OvS = oversegmentation; UnS
= undersegmentation; and Seg = segmentation.

##  Chapter 8 Conclusion

The goal of this study was to test whether exploiting midis and tabs improves
Automatic Chord Estimation (ace). The ace task is concerned with estimating
chords in audio recordings or symbolic representations of music and has
applications in both music performance and in Music Information Retrieval
(mir). Many systems for audio ace exist, but ace is not yet a solved problem:
as we have identified in Section 1.1, existing methods suffer from stagnation
and overfitting to subjective annotations. There is need for a new strategy
that overcomes existing stagnation in ace without further overfitting to
existing (subjective) data sets. An improved system for ace would have
positive consequences for millions of performers and mir systems for e.g.
cover song identification and genre classification.

For this purpose, we created a new system, decibel, that (1) aligns midi and
tab files to audio recordings and uses representation-specific chord
estimation techniques for both symbolic and audio formats to estimate chord
sequences for each file; and (2) integrates the resulting heterogeneous chord
sequences into one final output sequence. In this concluding chapter, we will
give an overview of our contributions and some directions for future work.

###  8.1 Contributions

The main contributions of this research are twofold: we designed a system that
automatically extracts a rich harmonic representation and used this
representation to improve state-of-the-art audio ace systems. In addition, we
extended the existing Isophonics Reference Annotations with a large set of
midi and tab files, which we manually matched to the annotations. As a final
contribution, we made the implementation of our system available on GitHub.

####  8.1.1 Rich harmonic representation

As we have seen in Chapters 4, 5 and 6, decibel has three subsystems that
estimate chord label sequences from audio, midi and tab files, using
representation-specific ace and alignment techniques. The resulting chord
label sequences form a rich harmonic representation that is a contribution by
itself, as it can easily be visualized and enables large-scale cross-version
analysis of popular music, giving more insights on subjectivity issues. The
potential use of this rich harmonic representation is described in Section
7.1.

####  8.1.2 Improvement of state-of-the-art audio ace systems

In Chapter 7, we described our experiments on data fusion. Decibel uses data
fusion methods to combine the chord label sequences from the aforementioned
rich harmonic representation into a single output sequence. The method df-best
turned out to be the best-performing method. Using this data fusion method,
decibel improves each of the ten original state-of-the-art audio ace systems.
The average wcsr improvement is as much as 3.05%. Supported by statistical
tests, we can confirm the hypothesis that audio ace can be improved by
integrating symbolic music formats.

Although decibel needs a data set of symbolic formats and has a longer
calculation time than the original audio ace systems, it also has two major
advantages beyond the wcsr improvement. As a first advantage, by using midi
and tab files that are scraped from the Internet, decibel implicitly
incorporates musical knowledge: midi and tab files can be considered as human-
made transcriptions of musical notes and chords. As a second advantage,
decibel requires a minimum amount additional training and therefore prohibits
overfitting to subjective data sets. There is little training needed for the
HMM used by Jump Alignment, but apart from that, neither the tab or midi
subsystem needs to be trained.

####  8.1.3 Extended Isophonics data set

As a third contribution, we extended the widely used Isophonics Reference
Annotations with 770 midi files and 1668 tabs. These symbolic music files are
scraped from the Internet and manually matched to the corresponding song in
the Isophonics data set. In this research project, we learned that midi files
and tabs are highly useful for the ace task. We expect that the extended data
set can also contribute to advances in other mir tasks, such as structural
segmentation or lyrics-to-audio alignment.

####  8.1.4 Python implementations

As a fourth and final contribution, we made our Python implementation of
decibel available to the mir community. The complete code repository can be
found on https://github.com/DaphneO/DECIBEL. As part of the complete decibel
system, this repository contains new Python implementations for existing
algorithms, such as Jump Alignment (McVicar et al., 2011b) and data fusion
(Koops et al., 2016), as well as implementations for new algorithms, such as
our midi chord recognizer cassette and a tab parser.

###  8.2 Future work

From the results of this research project, we can conclude that the
integration of multiple symbolic formats and audio for the improvement of ace
is an interesting research direction that deserves more attention in the
future. In this section, we give some directions for future work.

As a first suggestion for future work, it would be interesting to test
decibels performance on a larger data set, for example the Billboard data set.
The Isophonics data set was convenient to use, because there were myriad midi
files and tabs available for music by The Beatles and Queen. Decibel’s
performance may drop when testing on less popular or more modern artists,
because there may be less symbolic formats available for their songs. It would
also be interesting to test decibel on music of another genre, for example
jazz music. Very recently, a new data set for jazz, consisting of 113 tracks,
was released by Eremenko et al. (2018). In the accompanying paper, Eremenko et
al. (2018) evaluate two audio ace methods on their new data set and observe
that the csr’s for these methods were quite low. Extending the data set with
midi and tab files for these jazz songs and running the decibel system on the
extended data set may give better results.

As a second suggestion, we recommend using a larger chord vocabulary in future
work. It would be interesting to see if the integration of symbolic formats
also helps in recognizing more complex chords, such as seventh chords. Using
midi and tab files might be very helpful in recognizing complex chords, as
these representations directly encode notes and chord labels respectively.

Our third suggestion for future work is to experiment various techniques for
decibel’s subtasks. In the current implementation, we decided to use state-of-
the-art audio ace methods in combination with relatively simple methods for
alignment and midi chord estimation. The rationale behind this is as follows:
the audio ace methods should be as “good” as possible, as it may be trivial to
improve a poorly performing audio ace method. On the other hand, the
techniques for tab alignment, midi alignment and midi chord recognition should
be simple: if we can show that simple methods already improve the audio ace
method, we can assume that more sophisticated techniques will give even better
results. Besides, simpler methods are easier to understand and implement, and
typically do not require a lot of training. However, as we now know that
applying simple techniques on midi and tab files already improves audio ace,
it would be interesting to see how much we can improve our results by using
more sophisticated techniques. For example, a midi chord recognizer that takes
the musical context into account will probably give better results than our
proposed cassette algorithm.

As a fourth and final suggestion for future work, we propose to test decibel’s
performance by doing user tests. We saw that decibel improves each of the ten
tested state-of-the-art audio ace systems. Based on these results, we expect
that performers prefer decibel’s estimated chord label sequences to the
sequences found by audio ace systems. However, recall from Section 3.3 that
wcsr is not always a good measure for the quality of a chord sequence.
Therefore, it would be good to verify the improvement by doing user tests.

## References

  * Ajoodha et al. (2015) Ritesh Ajoodha, Richard Klein, and Benjamin Rosman.  Single-labelled music genre classification using content-based features.  In _Pattern Recognition Association of South Africa and Robotics and Mechatronics International Conference (PRASA-RobMech), 2015_ , pages 66–71. IEEE, 2015.
  * Arzt (2016) Andreas Arzt.  _Flexible and Robust Music Tracking_.  PhD thesis, Johannes Kepler University Linz, 2016.
  * Bello and Pickens (2005) Juan Pablo Bello and Jeremy Pickens.  A robust mid-level representation for harmonic content in music signals.  In _Proceedings of the 6th International Society for Music Information Retrieval Conference_ , pages 304–311, 2005.
  * Bonvivi (2014) Alessandro Bonvivi.  Automatic chord recognition using deep learning techniques.  Master’s thesis, Politecnico di Milano, 2014.
  * Bountouridis et al. (2016) Dimitrios Bountouridis, Hendrik Vincent Koops, Frans Wiering, and Remco C. Veltkamp.  A data-driven approach to chord similarity and chord mutability.  In _Second International Conference on Multimedia Big Data_ , pages 275–278. IEEE, 2016.
  * Brown (1991) Judith C. Brown.  Calculation of a constant q spectral transform.  _The Journal of the Acoustical Society of America_ , 89(1):425–434, 1991.
  * Burgoyne et al. (2007) John Ashley Burgoyne, Laurent Pugin, I. Fujinaga, and Kereliuk Ichiro.  A cross-validated study of modelling strategies for automatic chord recognition in audio.  In _Proceedings of the 8th International Society for Music Information Retrieval Conference_ , pages 251–254. ISMIR, 2007.
  * Cannam et al. (2017) Chris Cannam, Emmanouil Benetos, Matthias Mauch, Matthew E.P. Davies, Simon Dixon, Christian Landone, Katy Noland, and Dan Stowell.  MIREX 2018: Vamp plugins from the centre for digital music.  _Proceedings of the Music Information Retrieval Evaluation eXchange (MIREX)_ , 2017.
  * Carabias-Orti et al. (2015) Julio José Carabias-Orti, Francisco J. Rodríguez-Serrano, Pedro Vera-Candeas, Nicolás Ruiz-Reyes, and Francisco J. Cañadas-Quesada.  An audio to score alignment framework using spectral factorization and dynamic time warping.  In _Proceedings of the 16th International Society for Music Information Retrieval Conference_ , pages 742–748. ISMIR, 2015.
  * Cuthbert and Ariza (2010) Michael Scott Cuthbert and Christopher Ariza.  music21: A toolkit for computer-aided musicology and symbolic music data.  In _Proceedings of the 11th International Society for Music Information Retrieval Conference_ , pages 637–642. ISMIR, 2010.
  * Cuvillier and Cont (2014) Philippe Cuvillier and Arshia Cont.  Coherent time modeling of semi-markov models with application to real-time audio-to-score alignment.  In _Machine Learning for Signal Processing (MLSP), 2014 IEEE International Workshop on_ , pages 1–6. IEEE, 2014.
  * Dixon and Widmer (2005) Simon Dixon and Gerhard Widmer.  Match: A music alignment tool chest.  In _Proceedings of the 6th International Society for Music Information Retrieval Conference_ , pages 492–497, 2005.
  * Eerola and Toiviainen (2004) Tuomas Eerola and Petri Toiviainen.  MIR in Matlab: The MIDI toolbox.  In _Proceedings of the 5th International Society for Music Information Retrieval Conference_ , pages 22–27. ISMIR, 2004.
  * Eremenko et al. (2018) Vsevolod Eremenko, Emir Demirel, Baris Bozkurt, and Xavier Serra.  Audio-aligned jazz harmony dataset for automatic chord transcription and corpus-based research.  In _Proceedings of the 19th International Society for Music Information Retrieval Conference_ , pages 483–490. ISMIR, 2018.
  * Ewert et al. (2012) Sebastian Ewert, Meinard Müller, Verena Konz, Daniel Müllensiefen, and Geraint A. Wiggins.  Towards cross-version harmonic analysis of music.  _IEEE Transactions on Multimedia_ , 14(3):770–782, 2012.
  * Friedman (1937) Milton Friedman.  The use of ranks to avoid the assumption of normality implicit in the analysis of variance.  _Journal of the american statistical association_ , 32(200):675–701, 1937.
  * Fujishima (1999) Takuya Fujishima.  Realtime chord recognition of musical sound: a system using Common Lisp Music.  In _Proceedings of the International Computer Music Conference (ICMC)_ , pages 464–467, 1999.
  * Gabor (1946) Dennis Gabor.  Theory of communication. Part 1: The analysis of information.  _Journal of the Institution of Electrical Engineers-Part III: Radio and Communication Engineering_ , 93(26):429–441, 1946\.
  * Gasser and Strasser (2018) Stefan Gasser and Franz Strasser.  Multi objective chord estimation.  _Proceedings of the Music Information Retrieval Evaluation eXchange (MIREX)_ , 2018.
  * Guérin (2009) Robert Guérin.  _MIDI power!: The comprehensive guide_.  Course Technology PTR, 2009.
  * Harte (2010) Christopher Harte.  _Towards automatic extraction of harmony information from music signals_.  PhD thesis, Department of Electronic Engineering, Queen Mary, University of London, 2010.
  * Harte and Sandler (2005) Christopher Harte and Mark Sandler.  Automatic chord identifcation using a quantised chromagram.  In _Proceedings of the 118th Audio Engineering Society Convention_ , pages 745–750. AES, 2005.
  * Harte et al. (2005) Christopher Harte, Mark B. Sandler, Samer A. Abdallah, and Emilia Gómez.  Symbolic representation of musical chords: A proposed syntax for text annotations.  In _Proceedings of the 6th International Society for Music Information Retrieval Conference_ , pages 66–71. ISMIR, 2005.
  * Harte et al. (2006) Christopher Harte, Mark Sandler, and Martin Gasser.  Detecting harmonic change in musical audio.  In _Proceedings of the 1st ACM Workshop on Audio and Music Computing Multimedia_ , pages 21–26. ACM, 2006.
  * Hintze and Nelson (1998) Jerry L. Hintze and Ray D. Nelson.  Violin plots: a box plot-density trace synergism.  _The American Statistician_ , 52(2):181–184, 1998\.
  * Hu et al. (2003) Ning Hu, Roger B. Dannenberg, and George Tzanetakis.  Polyphonic audio matching and alignment for music retrieval.  In _Applications of Signal Processing to Audio and Acoustics, 2003 IEEE Workshop on._ , pages 185–188. IEEE, 2003.
  * Humphrey and Bello (2015) Eric J. Humphrey and Juan Pablo Bello.  Four timely insights on automatic chord estimation.  In _Proceedings of the 16th International Society for Music Information Retrieval Conference_ , pages 673–679, 2015.
  * Itakura (1975) Fumitada Itakura.  Minimum prediction residual principle applied to speech recognition.  _IEEE Transactions on Acoustics, Speech, and Signal Processing_ , 23(1):67–72, 1975.
  * Jiang et al. (2017) Junyan Jiang, Wei Li, and Yiming Wu.  Extended abstract for MIREX 2017 submission: Chord recognition using random forest model.  _Proceedings of the Music Information Retrieval Evaluation eXchange (MIREX)_ , 2017.
  * Jiang et al. (2018) Junyan Jiang, Ke Chen, Wei Li, and Guangyu Xia.  MIREX 2018 submission: A structural chord representation for automatic large-vocabulary chord transcription.  _Proceedings of the Music Information Retrieval Evaluation eXchange (MIREX)_ , 2018.
  * Khadkevich and Omologo (2013) Maksim Khadkevich and Maurizio Omologo.  Large-scale cover song identification using chord profiles.  In _Proceedings of the 14th International Society for Music Information Retrieval Conference_ , pages 233–238, 2013.
  * Kolchinsky et al. (2017) Artemy Kolchinsky, Nakul Dhande, Kengjeun Park, and Yong-Yeol Ahn.  The minor fall, the major lift: inferring emotional valence of musical chords through lyrics.  _Royal Society Open Science_ , 4(11), 2017.
  * Konz et al. (2013) Verena Konz, Meinard Müller, and Rainer Kleinertz.  A cross-version chord labelling approach for exploring harmonic structures: A case study on Beethoven’s Appassionata.  _Journal of New Music Research_ , 42(1):61–77, 2013.
  * Koops et al. (2016) Hendrik Vincent Koops, W. Bas de Haas, Dimitrios Bountouridis, and Anja Volk.  Integration and quality assessment of heterogeneous chord sequences using data fusion.  In _Proceedings of the 17th International Society for Music Information Retrieval Conference_ , pages 178–184. ISMIR, 2016.
  * Koops et al. (2017) Hendrik Vincent Koops, W. Bas de Haas, Jeroen Bransen, and Anja Volk.  Chord label personalization through deep learning of integrated harmonic interval-based representations.  In _Proceedings of the First International Workshop on Deep Learning for Music_ , pages 19–25, 2017.
  * Koops et al. (2019) Hendrik Vincent Koops, W Bas de Haas, John Ashley Burgoyne, Jeroen Bransen, Anna Kent-Muller, and Anja Volk.  Annotator subjectivity in harmony annotations of popular music.  _Journal of New Music Research_ , pages 1–21, 2019.
  * Korzeniowski and Widmer (2016a) Filip Korzeniowski and Gerhard Widmer.  Feature learning for chord recognition: The Deep Chroma Extractor.  In _Proceedings of 17th International Conference on Music Information Retrieval_ , 2016a.
  * Korzeniowski and Widmer (2016b) Filip Korzeniowski and Gerhard Widmer.  A fully convolutional deep auditory model for musical chord recognition.  In _Proceedings of the 26th International Workshop on Machine Learning for Signal Processing_ , pages 1–6. IEEE, 2016b.
  * Lajugie et al. (2016) Rémi Lajugie, Piotr Bojanowski, Philippe Cuvillier, Sylvain Arlot, and Francis Bach.  A weakly-supervised discriminative model for audio-to-score alignment.  In _Acoustics, Speech and Signal Processing (ICASSP), 2016 IEEE International Conference on_ , pages 2484–2488. IEEE, 2016.
  * Lewisohn (1989) Mark Lewisohn.  _Beatles Recording Sessions: The Official Abbey Road Studio Session Notes, 1962-1970_.  Harmony Books, 1989.
  * Macrae (2012) Robert Macrae.  _Linking Music Metadata_.  PhD thesis, Queen Mary University of London, 2012.
  * Macrae and Dixon (2011) Robert Macrae and Simon Dixon.  Guitar tab mining, analysis and ranking.  In _Proceedings of the 12th International Society for Music Information Retrieval Conference_ , pages 453–458. ISMIR, 2011.
  * Masada and Bunescu (2017) Kristen Masada and Razvan Bunescu.  Chord recognition in symbolic music using semi-Markov conditional random fields.  In _Proceedings of the 18th International Society for Music Information Retrieval Conference_ , pages 272–278. ISMIR, 2017.
  * Mauch (2010) Matthias Mauch.  _Automatic Chord Transcription from Audio Using Computational Models of Musical Context_.  PhD thesis, Queen Mary, University of London, 2010.
  * Mauch and Dixon (2008) Matthias Mauch and Simon Dixon.  A discrete mixture model for chord labeling.  In _Proceedings of the 9th International Society for Music Information Retrieval Conference_ , pages 45–50. ISMIR, 2008.
  * Mauch and Dixon (2010) Matthias Mauch and Simon Dixon.  Simultaneous estimation of chords and musical context from audio.  _IEEE Transactions on Audio, Speech, and Language Processing_ , 18(6):1280–1289, 2010.
  * Mauch et al. (2009) Matthias Mauch, Chris Cannam, Matthew Davies, Simon Dixon, Christopher Harte, Sefki Kolozali, Dan Tidhar, and Mark Sandler.  Omras2 metadata project 2009.  In _Proceedings of 10th International Conference on Music Information Retrieval_ , 2009.
  * Mauch et al. (2012) Matthias Mauch, Hiromasa Fujihara, and Masataka Goto.  Integrating additional chord information into HMM-based lyrics-to-audio alignment.  _IEEE Transactions on Audio, Speech, and Language Processing_ , 20(1):200–210, 2012.
  * Maxwell (1992) H. John Maxwell.  An expert system for harmonic analysis of tonal music.  In _Proceedings of the First Workshop on AI and Music_ , pages 20–33. AAAI, 1992.
  * McFee et al. (2018) Brian McFee, Matt McVicar, Stefan Balke, Carl Thomé, Colin Raffel, Dana Lee, Oriol Nieto, Eric Battenberg, Dan Ellis, Ryuichi Yamamoto, Josh Moore, Rachel Bittner, Keunwoo Choi, Pius Friesch, Fabian-Robert Stöter, Vincent Lostanlen, Siddhartha Kumar, Simon Waloschek, Seth, Rimvydas Naktinis, Douglas Repetto, Curtis "Fjord" Hawthorne, CJ Carr, Waldir Pimenta, Petr Viktorin, Paul Brossier, João Felipe Santos, JackieWu, Erik, and Adrian Holovaty.  librosa/librosa: 0.6.1, May 2018.  URL https://doi.org/10.5281/zenodo.1252297.
  * McKay and Fujinaga (2006) Cory McKay and Ichiro Fujinaga.  jSymbolic: A feature extractor for MIDI files.  In _Proceedings of the 2006 International Computer Music Conference_ , pages 302–305. ICMC, 2006.
  * McVicar and De Bie (2010) Matt McVicar and Tijl De Bie.  Enhancing chord recognition accuracy using web resources.  In _Proceedings of the 3rd International Workshop on Machine Learning and Music_ , pages 41–44. ACM, 2010.
  * McVicar et al. (2011a) Matt McVicar, Yizhao Ni, Tijl De Bie, and Raul Santos-Rodriguez.  Leveraging noisy online databases for use in chord recognition.  In _Proceedings of the 12th International Society for Music Information Retrieval Conference_ , pages 639–644. ISMIR, 2011a.
  * McVicar et al. (2011b) Matt McVicar, Yizhao Ni, Raul Santos-Rodriguez, and Tijl De Bie.  Using online chord databases to enhance chord recognition.  _Journal of New Music Research_ , 40(2):139–152, 2011b.
  * McVicar et al. (2014) Matt McVicar, Raúl Santos-Rodríguez, Yizhao Ni, and Tijl De Bie.  Automatic chord estimation from audio: A review of the state of the art.  _IEEE/ACM Transactions on Audio, Speech and Language Processing_ , 22(2):556–575, 2014.
  * Müller (2015) Meinard Müller.  _Fundamentals of Music Processing: Audio, Analysis, Algorithms, Applications_.  Springer International Publishing, 2015.
  * Müller and Ewert (2010) Meinard Müller and Sebastian Ewert.  Towards timbre-invariant audio features for harmony-based music.  _IEEE Transactions on Audio, Speech, and Language Processing_ , 18(3):649–662, 2010.
  * Ni et al. (2012) Yizhao Ni, Matt McVicar, Raul Santos-Rodriguez, and Tijl De Bie.  An end-to-end machine learning system for harmonic analysis of music.  _IEEE Transactions on Audio, Speech, and Language Processing_ , 20(6):1771–1783, 2012.
  * Ni et al. (2013) Yizhao Ni, Matt McVicar, Raul Santos-Rodriguez, and Tijl De Bie.  Understanding effects of subjectivity in measuring chord estimation accuracy.  _IEEE Transactions on Audio, Speech, and Language Processing_ , 21(12):2607–2615, 2013.
  * O’Hanlon et al. (2017) Ken O’Hanlon, Sebastian Ewert, Johan Pauwels, and Mark B. Sandler.  Improved template based chord recognition using the crp feature.  In _Acoustics, Speech and Signal Processing (ICASSP), 2017 IEEE International Conference on_ , pages 306–310. IEEE, 2017.
  * Papadopoulos and Tzanetakis (2012) Hélene Papadopoulos and George Tzanetakis.  Modeling chord and key structure with markov logic.  In _Proceedings of the 13th International Society for Music Information Retrieval Conference_ , pages 127–132, 2012.
  * Pardo and Birmingham (2001) Brian Pardo and William P. Birmingham.  The chordal analysis of tonal music.  Technical Report CSE-TR-439-01, The University of Michigan, Department of Electrical Engineering and Computer Science, 2001.
  * Pardo and Birmingham (2002) Bryan Pardo and William P. Birmingham.  Algorithms for chordal analysis.  _Computer Music Journal_ , 26(2):27–49, 2002\.
  * Pauws (2004) Steffen Pauws.  Musical key extraction from audio.  In _Proceedings of the 5th International Society for Music Information Retrieval Conference_ , pages 96–99. ISMIR, 2004.
  * Prätzlich and Müller (2016) Thomas Prätzlich and Meinard Müller.  Triple-based analysis of music alignments without the need of ground-truth annotations.  In _Acoustics, Speech and Signal Processing (ICASSP), 2016 IEEE International Conference on_ , pages 266–270. IEEE, 2016.
  * Rabiner (1989) Lawrence R Rabiner.  A tutorial on hidden markov models and selected applications in speech recognition.  _Proceedings of the IEEE_ , 77(2):257–286, 1989\.
  * Radicioni and Esposito (2010) Daniele Radicioni and Roberto Esposito.  BREVE: an HMPerceptron-based chord recognition system.  _Advances in Music Information Retrieval_ , pages 143–164, 2010.
  * Raffel (2016) Colin Raffel.  _Learning-Based Methods for Comparing Sequences, with Applications to Audio-to-MIDI Alignment and Matching_.  PhD thesis, Columbia University, 2016.
  * Raffel and Ellis (2014) Colin Raffel and Daniel P.W. Ellis.  Intuitive analysis, creation and manipulation of MIDI data with pretty_midi.  In _15th International Society for Music Information Retrieval Conference Late Breaking and Demo Papers_ , pages 84–93. ISMIR, 2014.
  * Raffel and Ellis (2015) Colin Raffel and Daniel P.W. Ellis.  Large-scale content-based matching of midi and audio files.  In _Proceedings of the 16th International Society for Music Information Retrieval Conference_ , pages 234–240. ISMIR, 2015.
  * Raffel and Ellis (2016a) Colin Raffel and Daniel P.W. Ellis.  Extracting ground-truth information from MIDI files: A MIDIfesto.  In _Proceedings of the 17th International Society for Music Information Retrieval Conference_ , pages 796–802. ISMIR, 2016a.
  * Raffel and Ellis (2016b) Colin Raffel and Daniel P.W. Ellis.  Optimizing dtw-based audio-to-midi alignment and matching.  In _Acoustics, Speech and Signal Processing (ICASSP), 2016 IEEE International Conference on_ , pages 81–85. IEEE, 2016b.
  * Raphael and Stoddard (2003) Christopher Raphael and Joshua Stoddard.  Harmonic analysis with probabilistic graphical models.  In _Proceedings of the 4th International Society for Music Information Retrieval Conference_ , pages 177–181. ISMIR, 2003.
  * Raphael and Stoddard (2004) Christopher Raphael and Joshua Stoddard.  Functional harmonic analysis using probabilistic models.  _Computer Music Journal_ , 28(3):45–52, 2004\.
  * Reed et al. (2009) Jeremy Reed, Yushi Ueda, Sabato Marco Siniscalchi, Yuuki Uchiyama, Shigeki Sagayama, and Chin-Hui Lee.  Minimum classification error training to improve isolated chord recognition.  In _Proceedings of the 10th International Society for Music Information Retrieval Conference_ , pages 609–614. ISMIR, 2009.
  * Sakoe and Chiba (1978) Hiroaki Sakoe and Seibi Chiba.  Dynamic programming algorithm optimization for spoken word recognition.  _IEEE transactions on acoustics, speech, and signal processing_ , 26(1):43–49, 1978.
  * Scholz and Ramalho (2008) Ricardo Scholz and Geber Ramalho.  COCHONUT: Recognizing complex chords from MIDI guitar sequences.  In _Proceedings of the 9th International Society for Music Information Retrieval Conference_ , pages 27–32. ISMIR, 2008.
  * Scholz et al. (2009) Ricardo Scholz, Emmanuel Vincent, and Frédéric Bimbot.  Robust modeling of musical chord sequences using probabilistic n-grams.  In _Proceedings of the 34th International Conference on Acoustics, Speech and Signal Processing_ , pages 53–56. IEEE, 2009.
  * Scholz et al. (2016) Ricardo Scholz, Geber Ramalho, and Giordano Cabral.  Cross task study on MIREX recent results: An index for evolution measurement and some stagnation hypotheses.  In _Proceedings of the 17th International Society for Music Information Retrieval Conference_ , pages 372–378, 2016.
  * Sheh and Ellis (2003) Alexander Sheh and Daniel P.W. Ellis.  Chord segmentation and recognition using em-trained hidden Markov models.  In _Proceedings of the 4th International Society for Music Information Retrieval Conference_ , pages 183–189. ISMIR, 2003.
  * Shenoy and Wang (2005) Arun Shenoy and Ye Wang.  Key, chord, and rhythm tracking of popular music recordings.  _Computer Music Journal_ , 29(3):75–86, 2005\.
  * Sigtia et al. (2015) Siddharth Sigtia, Nicolas Boulanger-Lewandowski, and Simon Dixon.  Audio chord recognition with a hybrid recurrent neural network.  In _Proceedings of the 16th International Society for Music Information Retrieval Conference_ , pages 127–133. ISMIR, 2015.
  * Sleator and Temperley (2001) Daniel Sleator and David Temperley.  The Melisma music analyzer.  _Web:http://www.link.cs.cmu.edu/music-analysis_, 2001.
  * Stowell and Dixon (2011) Dan Stowell and Simon Dixon.  MIR in school? Lessons from ethnographic observation of secondary school music classes.  In _Proceedings of the 12th International Society for Music Information Retrieval Conference_ , pages 347–352. ISMIR, 2011.
  * Taylor (1989) Eric Taylor.  _The AB Guide to Music Theory, Part I_.  The Associated Board of the Royal Schools of Music (Publishing) Ltd, 1989\.
  * Temperley and Sleator (1999) David Temperley and Daniel Sleator.  Modeling meter and harmony: A preference-rule approach.  _Computer Music Journal_ , 23(1):10–27, 1999\.
  * Tukey (1949) John W. Tukey.  Comparing individual means in the analysis of variance.  _Biometrics_ , pages 99–114, 1949.
  * Turetsky and Ellis (2003) Robert J. Turetsky and Daniel P.W. Ellis.  Ground-truth transcriptions of real music from force-aligned midi syntheses.  In _Proceedings of the 4th International Society for Music Information Retrieval Conference_ , pages 135–141, 2003.
  * Wakefield (1999) Gregory H. Wakefield.  Mathematical representation of joint time-chroma distributions.  In _Proceedings of SPIE’s International Symposium on Optical Science, Engineering, and Instrumentation_ , volume 3807, pages 637–646. International Society for Optics and Photonics, 1999.
  * Wang et al. (2014) Siying Wang, Sebastian Ewert, and Simon Dixon.  Robust joint alignment of multiple versions of a piece of music.  In _Proceedings of the 15th International Society for Music Information Retrieval Conference_ , page 83. ISMIR, 2014.
  * Winograd (1968) Terry Winograd.  Linguistics and the computer analysis of tonal harmony.  _Journal of Music Theory_ , 12(1):2–49, 1968\.
  * Wu et al. (2017) Yiming Wu, Xiangyi Feng, and Wei Li.  MIREX 2017 submission: Automatic audio chord recognition with MIDI-trained deep feature and BLSTM-CRF sequence decoding model.  _Proceedings of the Music Information Retrieval Evaluation eXchange (MIREX)_ , 2017.

##  Appendix A Data set

The Beatles - Please Please Me
---
Index | Song title
1 | I Saw Her Standing There
2 | Misery
3 | Anna (Go To Him)
4 | Chains
5 | Boys
6 | Ask Me Why
7 | Please Please Me
8 | Love Me Do
9 | P. S. I Love You
10 | Baby It’s You
11 | Do You Want To Know A Secret
12 | A Taste Of Honey
13 | There’s A Place
14 | Twist And Shout
The Beatles - With the Beatles
Index | Song title
15 | It Won’t Be Long
16 | All I’ve Got To Do
17 | All My Loving
18 | Don’t Bother Me
19 | Little Child
20 | Till There Was You
21 | Please Mister Postman
22 | Roll Over Beethoven
23 | Hold Me Tight
24 | You Really Got A Hold On Me
25 | I Wanna Be Your Man
26 | Devil In Her Heart
27 | Not A Second Time
28 | Money
The Beatles - A Hard Day’s Night
Index | Song title
29 | A Hard Day’s Night
30 | I Should Have Known Better
31 | If I Fell
32 | I’m Happy Just To Dance With You
33 | And I Love Her
34 | Tell Me Why
35 | Can’t Buy Me Love
36 | Any Time At All
37 | I’ll Cry Instead
38 | Things We Said Today
39 | When I Get Home
40 | You Can’t Do That
41 | I’ll Be Back
The Beatles - Beatles for Sale
Index | Song title
42 | No Reply
43 | I’m a Loser
44 | Baby’s In Black
45 | Rock and Roll Music
46 | I’ll Follow the Sun
47 | Mr. Moonlight
48 | Kansas City- Hey, Hey, Hey, Hey
49 | Eight Days a Week
50 | Words of Love
51 | Honey Don’t
52 | Every Little Thing
53 | I Don’t Want to Spoil the Party
54 | What You’re Doing
55 | Everybody’s Trying to Be My Baby
The Beatles - Help!
Index | Song title
56 | Help!
57 | The Night Before
58 | You’ve Got To Hide Your Love Away
59 | I Need You
60 | Another Girl
61 | You’re Going To Lose That Girl
62 | Ticket To Ride
63 | Act Naturally
64 | It’s Only Love
65 | You Like Me Too Much
66 | Tell Me What You See
67 | I’ve Just Seen a Face
68 | Yesterday
69 | Dizzy Miss Lizzy
The Beatles - Rubber Soul
Index | Song title
70 | Drive My Car
71 | Norwegian Wood (This Bird Has Flown)
72 | You Won’t See Me
73 | Nowhere Man
74 | Think For Yourself
75 | The Word
76 | Michelle
77 | What Goes On
78 | Girl
79 | I’m Looking Through You
80 | In My Life
81 | Wait
82 | If I Needed Someone
83 | Run For Your Life
The Beatles - Revolver
Index | Song title
84 | Taxman
85 | Eleanor Rigby
86 | I’m Only Sleeping
87 | Love You To
88 | Here, There And Everywhere
89 | Yellow Submarine
90 | She Said She Said
91 | Good Day Sunshine
92 | And Your Bird Can Sing
93 | For No One
94 | Doctor Robert
95 | I Want To Tell You
96 | Got To Get You Into My Life
97 | Tomorrow Never Knows
The Beatles - Sgt. Pepper’s Lonely Hearts Club Band
Index | Song title
98 | Sgt. Pepper’s Lonely Hearts Club Band
99 | With A Little Help From My Friends
100 | Lucy In The Sky With Diamonds
101 | Getting Better
102 | Fixing A Hole
103 | She’s Leaving Home
104 | Being For The Benefit Of Mr. Kite!
105 | Within You Without You
106 | When I’m Sixty-Four
107 | Lovely Rita
108 | Good Morning Good Morning
109 | Sgt. Pepper’s Lonely Hearts Club Band (Reprise)
110 | A Day In The Life
The Beatles - Magical Mystery Tour
Index | Song title
111 | Magical Mystery Tour
112 | The Fool On The Hill
113 | Flying
114 | Blue Jay Way
115 | Your Mother Should Know
116 | I Am The Walrus
117 | Hello Goodbye
118 | Strawberry Fields Forever
119 | Penny Lane
120 | Baby You’re A Rich Man
121 | All You Need Is Love
The Beatles - The Beatles
Index | Song title
122 | Back in the USSR
123 | Dear Prudence
124 | Glass Onion
125 | Ob-La-Di, Ob-La-Da
126 | Wild Honey Pie
127 | The Continuing Story of Bungalow Bill
128 | While My Guitar Gently Weeps
129 | Happiness is a Warm Gun
130 | Martha My Dear
131 | I’m So Tired
132 | Black Bird
133 | Piggies
134 | Rocky Raccoon
135 | Don’t Pass Me By
136 | Why Don’t We Do It In The Road
137 | I Will
138 | Julia
139 | Birthday
140 | Yer Blues
141 | Mother Nature’s Son
142 | Everybody’s Got Something To Hide Except Me and My Monkey
143 | Sexy Sadie
144 | Helter Skelter
145 | Long Long Long
146 | Revolution
147 | Honey Pie
148 | Savoy Truffle
149 | Cry Baby Cry
150 | Revolution
151 | Good Night
The Beatles - Abbey Road
Index | Song title
152 | Come Together
153 | Something
154 | Maxwell’s Silver Hammer
155 | Oh! Darling
156 | Octopus’s Garden
157 | I Want You
158 | Here Comes The Sun
159 | Because
160 | You Never Give Me Your Money
161 | Sun King
162 | Mean Mr Mustard
163 | Polythene Pam
164 | She Came In Through The Bathroom Window
165 | Golden Slumbers
166 | Carry That Weight
167 | The End
168 | Her Majesty
The Beatles - Let It Be
Index | Song title
169 | Two of Us
170 | Dig a Pony
171 | Across the Universe
172 | I Me Mine
173 | Dig It
174 | Let It Be
175 | Maggie Mae
176 | I’ve Got A Feeling
177 | One After
178 | The Long and Winding Road
179 | For You Blue
180 | Get Back
Queen - Greatest Hits I
Index | Song title
181 | Bohemian Rhapsody
182 | Another One Bites The Dust
184 | Fat Bottomed Girls
185 | Bicycle Race
186 | You’re My Best Friend
187 | Don’t Stop Me Now
188 | Save Me
189 | Crazy Little Thing Called Love
190 | Somebody To Love
192 | Good Old Fashioned Lover Boy
193 | Play The Game
195 | Seven Seas Of Rhye
196 | We Will Rock You
197 | We Are The Champions
Queen - Greatest Hits II
Index | Song title
198 | A Kind Of Magic
201 | I Want It All
202 | I Want To Break Free
206 | Who Wants To Live Forever
211 | Hammer To Fall
212 | Friends Will Be Friends

##  Appendix B Alignment listening test results

midi file | Alignment | Listening test result
---|---|---
| error |
128-005 | 0.723 | 2 - Good alignment, no fade out in midi
014-003 | 0.762 | 2 - Good alignment, midi cut off too soon at the end
095-002 | 0.636 | 2 - Good alignment, but seems to be a bit shifted
047-001 | 0.934 | 1 - Very bad alignment
063-004 | 0.764 | 3 - Good alignment
098-004 | 0.850 | 3 - Good alignment
196-001 | 0.656 | 3 - Good alignment
099-002 | 0.959 | 1 - Very bad alignment
195-001 | 0.723 | 2 - Good alignment with minor issues at the end
036-002 | 0.903 | 1 - First verse and chorus missing in midi, issues at the end
184-001 | 0.720 | 3 - Good alignment
078-003 | 0.654 | 2 - Good alignment, no fade out in midi
042-001 | 0.815 | 2 - Good alignment, but sometimes seems to be a bit shifted
152-002 | 0.705 | 2 - Good alignment, no fade out in midi
181-005 | 0.964 | 1 - Very bad alignment (transcription with only piano)
002-001 | 0.644 | 2 - Good alignment, no fade out in midi
061-002 | 0.762 | 2 - Good alignment, minor issues at the end
155-001 | 0.880 | 1 - Mediocre alignment, last 50s of the audio missing in midi file
073-004 | 0.760 | 2 - Good alignment, but seems to be a bit shifted
055-002 | 0.712 | 3 - Good alignment
003-001 | 0.771 | 2 - Good alignment, but seems to be a bit shifted
010-001 | 0.709 | 2 - Good alignment, no fade out in midi
125-002 | 0.784 | 3 - Good alignment
202-005 | 0.641 | 3 - Good alignment
212-001 | 0.532 | 3 - Good alignment
Table B.1: Results of the alignment listening test

##  Appendix C Tabs in parsing evaluation set

The annotated subset for the tab parsing evaluation consists of these 25
songs:

  1. 1.

Let It Be (ver 6) Tabs

  2. 2.

A Hard Days Night (ver 10) Chords

  3. 3.

You’re My Best Friend solo Tabs

  4. 4.

Savoy Truffle (ver 2) Chords

  5. 5.

Strawberry Fields Forever Acoustic Chords

  6. 6.

A Taste Of Honey (ver 2) Chords

  7. 7.

What You’re Doing Chords

  8. 8.

A Hard Days Night (ver 3) Tabs

  9. 9.

Let It Be (ver 4) Chords

  10. 10.

Love You To Tabs

  11. 11.

Cry Baby Cry (ver 4) Chords

  12. 12.

For You Blue Chords

  13. 13.

Getting Better (ver 5) Chords

  14. 14.

Help (ver 8) Chords

  15. 15.

No Reply (ver 4) Chords

  16. 16.

Happiness Is A Warm Gun Tabs

  17. 17.

I’ll Be Back (ver 2) Tabs

  18. 18.

Tell Me Why Tabs

  19. 19.

Get Back (ver 5) Tabs

  20. 20.

Anna (ver 2) Chords

  21. 21.

A Day In The Life Tabs

  22. 22.

You Never Give Me Your Money Tabs

  23. 23.

Eleanor Rigby (ver 3) Tabs

  24. 24.

I’ll Be Back (ver 7) Chords

  25. 25.

Yesterday Chords

##  Appendix D Tabs in forward and backward training set

The following 22 songs were used for training forward and backward
probabilities:

  1. 1.

Misery Chords

  2. 2.

Please Mister Postman Chords

  3. 3.

You Really Got A Hold On Me (ver 2) Chords

  4. 4.

And I Love Her Tabs

  5. 5.

I’ll Cry Instead Tabs

  6. 6.

I’ll Be Back Tabs

  7. 7.

Mr Moonlight Tabs

  8. 8.

Help (ver 6) Chords

  9. 9.

I Need You Tabs

  10. 10.

You’re Going To Lose That Girl Chords

  11. 11.

Act Naturally (ver 3) Tabs

  12. 12.

Norwegian Wood (ver 4) Chords

  13. 13.

Think For Yourself Chords

  14. 14.

Girl Acoustic Chords

  15. 15.

Good Day Sunshine Chords

  16. 16.

Sgt Peppers Lonely Hearts Club Band Reprise (ver 2) Chords

  17. 17.

Penny Lane (ver 6) Chords

  18. 18.

Come Together Chords

  19. 19.

Mean Mr Mustard (ver 2) Chords

  20. 20.

Golden Slumbers Chords

  21. 21.

A Kind Of Magic Chords

  22. 22.

I Want To Break Free Chords