[1]Antonio Ríos-Vila

[1]Pattern Recognition and Artificial Intelligence Group, University of
Alicante, Spain

2]Instituto Superior de Enseñanzas Artísticas de la Comunidad Valenciana,
Spain

3]LITIS Laboratory - EA 4108, University of Rouen, France

# Sheet Music Transformer++: End-to-End Full-Page Optical Music Recognition
for Pianoform Sheet Music

<EMAIL_ADDRESS>Jorge Calvo-Zaragoza<EMAIL_ADDRESS>David Rizo
<EMAIL_ADDRESS>Thierry Paquet<EMAIL_ADDRESS>*  [  [

###### Abstract

Optical Music Recognition is a field that has progressed significantly,
bringing accurate systems that transcribe effectively music scores into
digital formats. Despite this, there are still several limitations that hinder
OMR from achieving its full potential. Specifically, state of the art OMR
still depends on multi-stage pipelines for performing full-page transcription,
as well as it has only been demonstrated in monophonic cases, leaving behind
very relevant engravings. In this work, we present the Sheet Music
Transformer++, an end-to-end model that is able to transcribe full-page
polyphonic music scores without the need of a previous Layout Analysis step.
This is done thanks to an extensive curriculum learning-based pretraining with
synthetic data generation. We conduct several experiments on a full-page
extension of a public polyphonic transcription dataset. The experimental
outcomes confirm that the model is competent at transcribing full-page
pianoform scores, marking a noteworthy milestone in end-to-end OMR
transcription.

###### keywords:

OMR, SMT++, Transformers, Full Page, GrandStaff

##  1 Introduction

Music represents a crucial element of our cultural legacy, serving as a key to
understanding the social, cultural, and artistic tendencies of various
historical periods. The bulk of historical documents related to music have
been preserved in printed and handwritten formats, known as music scores. Many
of the music archives are stored in physical formats, mostly books or
manuscripts. These resources are subject to permanent loss from degradation,
which creates a considerable need for digitization.

In the same way Optical Character Recognition (OCR) and Handwritten Text
Recognition (HTR) were conceived to address this challenge in text documents,
Optical Music Recognition (OMR) constitutes the research area that handles the
computational information extraction and transcription of music score sheets
[9].

Advancements in OMR have evolved significantly from its inception, which
relied on multi-stage processes [35], towards the incorporation of Deep
Learning technologies. Currently, there are two predominant approaches to
handle music transcription: object-detection workflows [47, 4, 33]—–where
musical notes are individually identified and compiled into a digital
format—–and holistic or end-to-end methods [43, 45, 11, 1], wherein systems
directly produce a sequence of symbols that represent the content of the
document at hand. This holistic approach, which is inspired by the leading-
edge methods in texts, speech, and mathematical expression recognition [15,
14, 49], has demonstrated remarkable effectiveness when transcribing music
scores.

Despite these technological advancements and the urgency to retrieve the
digital music documents of repertories, manual transcription is still the
preferred choice to handle this challenge [3]. The reason behind this is that,
currently, there are still plenty of open problems and limitations that hinder
OMR to be effectively deployed into usable applications. This statement is
backed by two main reasons. First, most state-of-the-art OMR works are only
able to transcribe at the staff level, requiring multistep pipelines to
generate the content of a sheet music score. This complicates the
implementation of production-ready models, since it is required to fine-tune
each pipeline step for every different music repertory available. The other
reason is that, to the extent of our knowledge, holistic OMR literature has
shown scientific progress in the transcription of monophonic documents,
leaving behind relevant music engravings, such as polyphonic and vocal music
scores.

The lack of technical and data capacity of current state of the art to develop
a “universal” end-to-end OMR model highlights the difficulty of producing
applications, as well as it highlights the necessity of developing methods
that go beyond current OMR limits.

The first step to progress towards this “universal” model is to simplify the
transcription method. That is, it is required to develop a single model that
is able to retrieve the content of a page in a single step, avoiding staff
segmentation. This is known in the HTR field as segmentation-free approaches.

This paper introduces the Sheet Music Transformer++ (SMT++) model, a
segmentation-free image-to-sequence transcription model built upon
autoregressive Transformers, which is capable of transcribing entire music
score pages without any layout information, both during training and
inference. In order to unlock this capacity, the model is trained through a
multi-stage curriculum learning strategy, that combines synthetic and real
data, that teaches progressively how to read music pages. To assess this
capacity, our approach is evaluated within the context of polyphonic music
transcription, which is an overlooked texture in the full-page OMR literature.
Specifically, we focus on piano music scores, which are the most complex
documents when addressing OMR [9]. The experiments performed in this work
empirically validate the robustness and effectiveness of our solution for this
challenge. The main scientific contributions of this work can be summarized as
follows:

  * •

We propose the SMT++, the first image-to-sequence-based approach for sheet
music transcription that is able to deal with transcripts beyond single-staff
level without ad-hoc adaptations. In our experiments, we demonstrate that this
approach yields acceptable results for a very complex music score
transcription scenario.

  * •

We propose a curriculum learning strategy to enable the model to deal
gradually with full-page music scores. This approach is based on a synthetic
generation algorithm that, from the resources of a public music scores
dataset, is able to produce a high amount of varied full-page images for
learning the task.

  * •

We explore and analyze different configurations for feature extraction in
order to produce a model that is better suited to complex music document
layouts.

The remainder of the paper is structured as follows: in Section 2, we present
the state of the art in OMR for full-page transcription, as well as we discuss
its limitations and the inability of transcribing beyond single-stave
documents. In Section 3, we describe the SMT++ neural network proposed in this
paper. Section 4 discusses the digital music encoding format for the network
outputs, while in Section 5 the training strategy followed to enable the SMT++
model to transcribe full pages is described. Section 6 describe the
experiments conducted, along with the results and analysis that can be drawn
from them. Sections 7 and 8 report and analyze the retrieved results from both
the main experiments and some ablation studies. Finally, we conclude the paper
in Section 9, in addition to discussing other avenues for future work.

##  2 Full-page OMR transcription

End-to-end state-of-the-art OMR methods typically address the challenge
through neural networks trained with the CTC loss strategy [21]. This
methodology has been largely-proven to address single-staff music
transcription, and is successful for the transcription of both printed and
handwritten scores [4, 45, 11, 10] and has attained very accurate results,
surpassing those of any other image-to-sequence approach [37]. Despite this,
the approach is severely limited. CTC-based methods generally assume a reshape
function that transforms the input image into a sequence-like structure that
enables its decoding processing into a text output. This is known as the
vertical collapse [6, 7]. This operator prevents the network from dealing with
images that contain multiple structures that have to be taken into account at
the same time, such as multiple music staves. This is an issue that directly
affects the processing of full pages.

To address full-page transcription—which suffers from having several music
staves interacting together in a document—state-of-the-art OMR implements a
two-stage pipeline. This workflow is composed of, first, a Layout Analysis
step [32, 12, 13] that detects and extracts all the staves of the music score.
This is typically addressed through object detection algorithms. Then, all the
extracted elements are fed into an end-to-end OMR transcription method, which
follows the implementation described previously. The generated transcriptions
are finally concatenated to represent the content of the full page music
score.

This system is composed of two models that are trained specifically on
independent tasks—that is, each document has to be manually labeled twice.
First, bounding boxes and region classes have to be inserted to train the
object detection model. Once this information has been retrieved, the
corresponding transcription must be written for each staff in the dataset to
train the recognition network. Therefore, labeling a corpus to train a
production-ready OMR for full-page transcription is costly—due to the need for
manual segmentation and transcription of individual music staves. Another
issue is that there exists an accumulated error that propagates through steps,
as some works have studied [32, 13, 38]. That is, the performance of a
specific step is highly related to the effectiveness of the previous one,
which is also dependent on their predecessors, thus creating a snowball
effect. There is a recent segmentation-free method that deals with this
challenge [42]. However, despite presenting promising results, this approach
was not able to surpass state-of-the-art systems.

###  2.1 The challenge of transcribing beyond single staves

In addition to the drawbacks presented before, there is one notable issue in
state-of-the-art OMR for full page transcription. All the approaches are only
valid for monophonic scores. There exist other music sources that are not
monophonic. Figure 1 illustrates an example of some different scores we can
find in music repositories.

(a) Guitar score

(b) Piano score

(c) Orchestra ensemble score

Figure 1: Three examples of music scores that are found in Common Western
Notation.

These music documents present an additional challenge. Specifically, these
engravings must be read in a particular order during interpretation, since
they contain staves that must be read simultaneously. Rather than performing a
line-by-line reading from top to bottom and left to right—such as state-of-
the-art methods do—interpretation is tied to staff groups—referred to as
systems—in which all elements are read simultaneously from left to right and,
then, reading order jumps to the next group, which is located below the
previous one. As it can be seen, the challenge increases when transcribing a
music page, since the reading order is not straightforward to determine.
Currently, there are some end-to-end approaches that are able to deal with
polyphonic transcription at the system level [39, 40]. An adaptation for full-
page polyphonic transcription can be performed if the monophonic transcription
method is replaced by this approach. Despite this, this comes with several
major drawbacks. First, not all polyphonic voices are tied in consistent
groups. Take the example of Figure 2, where some instruments are represented
with one staff, but others—such as the piano—are represented with two. As it
can be observed, jumping between content pieces in a correct reading order is
a non-trivial task, aside from needing to learn how to reproduce all voices at
the same time. Aside from these layout-related problems, information losses
can be produced even in the simplest graphic cases. Pianoform scores
illustrate an example. These sources do not only contain multiple voices
within a single staff, but also tend to present cross-staff interactions—which
are sometimes difficult to distinguish in the document image—that have to be
detected at the whole page level. This language-related complexity is
impossible to assess with segmentation-based pipelines, as this information is
lost when dealing with each system individually.

Figure 2: Example of complex reading order in music scores. In this case, we
have two main voices highlighted: a violin (blue) and a piano (red). Markers
show notes that are reproduce simultaneously (in this case, one for the violin
and two for the piano). Arrows indicate the reading order of both voices, from
left to right and up to down.

This case requires the search of alternative methods that are able to
transcribe the content of the full page and taking into account all the
structure of complex music scores.

##  3 Sheet Music Transformer

In this paper, we present the Sheet Music Transformer++ (SMT++), which
inherits from the Sheet Music Transformer (SMT) network [40]. This model is an
autoregressive end-to-end neural network that generates the transcription of
an input full page music score. The model consists of two fundamental
components: an encoder and a decoder. The encoder is a feature extractor of
the image $x$, which produces a feature map $x_{e}^{\prime}$. The decoder
consists of an autoregressive conditioned language model that predicts the
probability of each symbol of the vocabulary in a timestep given the output
feature vector from the encoder and the previously generated symbols. This is
formalized as:

$\hat{y}_{t}=\underset{y_{t}\in\Sigma}{\text{argmax}}(P(\hat{y}_{t}\mid
x_{e}^{\prime},\left(\hat{y}_{0},\hat{y}_{1},\hat{y}_{2},...,\hat{y}_{t-1}\right)))$
(1)

where $\Sigma$ represents the symbol vocabulary used in order to encode the
music content, $x$ is the input feature map and $t$ is the current timestep. A
graphic scheme of the SMT++ is depicted in Figure 3.

###  3.1 Encoder

Let $x\in\mathbb{R}^{c\times h\times w}$ represent the input image of the
system, where $h$ and $w$ respectively denote its height and width in pixels
and $c$ denotes the number of channels. This block is typically implemented
through Convolutional Neural Networks [16, 44], owing to their capacity to
process image signals. This module therefore results in an image processor
that outputs $c_{e}$ two-dimensional feature maps denoted by
$x_{e}\in\mathbb{R}^{h_{e}\times w_{e}\times c_{e}}$. Note that, $h_{e}$ and
$w_{e}$ respectively relate to the image dimensions $h_{e}=\frac{h}{r_{h}}$
and $w_{e}=\frac{w}{w_{t}}$, where $r_{h}$ and $r_{w}$ represent the
corresponding downscale produced by the pooling operations and strides that
the feature extractor contains.

Figure 3: Graphic scheme of the SMT++ architecture.

###  3.2 Decoder

The second module of the SMT++ is a Transformer [48] decoder, as it is
currently the state-of-the-art method for tasks involving conditional sequence
generation of varying lengths, thanks to its capacity to capture temporal
relationships within the data through the use of multi-head attention
mechanisms.

The decoder consists of a language model that, at each timestep $t$ outputs a
probability distribution $p_{t}\in\mathbb{R}^{\left|\Sigma\right|}$ over the
$\Sigma$ vocabulary symbols, which is conditioned by the input feature vector
produced by the encoder, $x_{e}^{\prime}$ along with the previously predicted
tokens, $\left(\hat{y}_{0},\ldots,\hat{y}_{t-1}\right)$. The predicted token
$\hat{y}_{t}$ is the one that maximizes this probability score. The process
starts with a special start-of-transcription symbol,
$\hat{y}_{0}=\text{\textless{sot}\textgreater}$, and continues until an end-
of-transcription token is predicted,
$\hat{y}_{|\mathbf{\hat{y}}|}=\text{\textless{eot}\textgreater}$.

We should stress that the encoder outputs a two-dimensional feature map to
serve as a context in which to condition the decoder. However, the decoder
works with one less dimension, since it is a sequence generator. In order to
connect these two modules, the two-dimensional feature maps must be converted
into a one-dimensional format that is suitable for the decoder, with a
straightforward approach being that of flattening this structure across height
and width, resulting in sequences of length $h_{e}\times w_{e}$.

Another aspect to take into account is that the Transformer implements a
positional encoding (PE) mechanism with which to model its otherwise order-
agnostic operation. This mechanism adds a position vector to each input
element, determined by its location in the sequence. The addition of a one-
dimensional PE to the unfolded feature maps could initially be considered
possible, but while this might be sufficient for monophonic music sequences,
it would result in a loss of spatial information when targeting polyphonic
scores in which multiple voices are played simultaneously. In order to ensure
that the model is aware of all the dimensions of the image, we, therefore,
incorporate a two-dimensional PE within the feature maps before they are
flattened into a one-dimensional sequence. The two-dimensional PE proposal is
based on sine and cosine functions, akin to the original one-dimensional PE,
in which the first half of the feature dimensions—i.e., $[0,c_{e}/2)$—is meant
for horizontal positions, whereas the second half—i.e., $[c_{e}/2,c_{e})$—is
used for the vertical ones, the same way that it is implemented in other HTR
works [44, 16]. The 2D PE is defined as:

$\displaystyle\begin{split}\text{PE}_{\text{2D}}\left(\text{pos}_{t},2i\right)&=\sin\left(\text{pos}_{t}/10000^{2i/c_{e}}\right)\\\
\text{PE}_{\text{2D}}\left(\text{pos}_{t},2i+1\right)&=\cos\left(\text{pos}_{t}/10000^{2i/c_{e}}\right)\\\
\text{PE}_{\text{2D}}\left(\text{pos}_{f},c_{e}/2+2i\right)&=\sin\left(\text{pos}_{f}/10000^{2i/c_{e}}\right)\\\
\text{PE}_{\text{2D}}\left(\text{pos}_{f},c_{e}/2+2i+1\right)&=\cos\left(\text{pos}_{f}/10000^{2i/c_{e}}\right)\end{split}$
(2)

where $\text{pos}_{t}$ and $\text{pos}_{f}$ respectively specify the
horizontal (width) and vertical (height) pixel positions and
$i\in\left[0,c_{e}/4\right)$ denotes the feature dimension of the output map.

##  4 Output Encoding

The encoding format of the output is a crucial step towards the development of
OMR systems. There exist plenty of options when choosing a digital encoding,
being Music Encoding Initiative (MEI) [22] and MusicXML [20] the most popular
ones. These formats are known for their comprehensive representation of music
score components and metadata through an XML-based markup language. Despite
being such remarkable options, they are not convenient for deep learning-based
solutions. The reason is that they are too verbose and typically require
conversions and adaptations into intermediate languages, which tend to be
lossy.

In this work—based on recent research for both monophonic [36, 41, 39] and
polyphonic [39] music scores—we adopt the text-based Humdrum **kern encoding
format, part of the Humdrum toolkit [25], subsequently referred to as kern.
This encoding stands out in computational music analysis due to its widespread
use, simplicity, and straightforward parsing capabilities, making it
exceptionally well-suited for end-to-end OMR applications. The straightforward
structure of kern files not only facilitates compatibility with various music
software packages [34] but also allows for conversion to and from other music
representations.

A kern file is basically a sequence of lines. Each line is, in turn, another
sequence of columns or _spines_ that are separated by a tab character. Each
column contains an instruction, such as the creation or ending of spines, or
the encoding of musical symbols such as clefs, key signatures, meter, bar
lines, or notes, to name but a few. When interpreting a kern file, all spines
are read simultaneously, thus providing the concept of polyphony to the
format. That is, a line in a kern document should be read from left to
right—interpreting all the symbols that appear simultaneously—and then from
top to bottom, advancing in time through the score. Figure 4 shows an example
on how a kern file is constructed, aligned with its correspondent score.

Figure 4: Example of a kern score (left) aligned by Reading Order with its
rendered music document (right).

Since the SMT is an autoregresive Transformer, we provide the complete kern
document—read line by line— as ground truth. During parsing, we include
aditional markup tokens. One for voice jumping, represented as the tab token,
since there is the case where multiple voices split in the same spine. The
other one is the linebreak token, which indicates time step continuations.
This allows the model to differentiate which notes have to be played
simultaneously and which ones sequentially.

###  4.1 Humdrum Kern preprocessing

Machine Learning approaches for transcription require to divide the ground
truth into element-wise sequences for training and inference. The elements
that compose these sequences are known as tokens. Tokens are arbitrary
structures that highly depend on the output encoding the model gives. In
standard kern, all of the approaches interpret the complete symbol as a token.
In the example from Fig. 4, 8e-J would be a token.

Given this tokenization method, raw kern presents a large vocabulary—which
surpasses $20.000$ tokens—which is inconvenient for this task, since it may
lead to vanishing gradients and training instability. In this work, we propose
two encoding-based alternatives to address this issue. First, we follow the
work of [39] applied the “basic extended kern”, referred to as bekern. This
format reduces significantly the alphabet by decomposing tokens into minimum
semantic elements, such as the duration or the pitch. These elements are
divided by a specific delimiter, which is the ‘·’ character. For example, in
Fig. 4, the 8e-J token is encoded as 8·e·-·J.111Note that the token ‘·’ is
used solely as a separator, and the recognition model is, therefore, does not
provide it explicitly..

This format, although optimal in vocabulary reduction, suffers from a lower
correlation with the image graphic information, as the model should predict
several tokens for the same symbol. This results inconvenient when dealing
with large documents. Therefore, we propose the “Extended Kern
Format”—ekern—which splits the tokens by graphical meaning. This way, music
notes are represented by a unique token, but any other accidentals or features
are represented independently. Continuing with the example of Fig. 4, the 8e-J
token is encoded as 8e·-·J.

###  4.2 Humdrum Kern normalization

Humdrum kern, although bein a suitable encoding to address end-to-end OMR
transcription, has another major drawback for machine learning approaches.
Since this language is compact, it allows different representations for the
same elements, depending only on personal preferences of the transcriber. For
example, in Fig. 4, the ending of a beam is encoded using the ‘J’ character.
In raw kern, it is valid to encode the whole symbol as in the figure, i.e.,
with 8e-J denoting an eighth note (‘8’) of pitch E (’e’) altered by a flat
accidental (‘-’), but also as J8e-. This feature, which is very comfortable
for manual digitization processes, produces both large output vocabularies and
noisy ground truths. Recently, a grammar-based method was proposed to
normalize the data in pianoform music scores [39], where strict rules are set
to avoid ambiguous or non-consistent encodings. In this work, we apply this
method to avoid noisy training.

##  5 Sheet Music Transformer++

The training of Transformer-based applications is known for being a data-
demanding challenging task, mainly because of their low inductive bias [17,
8]. The design of an adequate training process to deal with full page
transcription is key for the success of this model. Inspired by other works in
the HTR field [16, 18], the SMT++ is trained in a three-stage process:

  1. 1.

System-level pretraining: The SMT++ is trained at the system-level music
samples to learn the first reading order step of pianoform music scores.

  2. 2.

Incremental-system curriculum learning: The single system-transcription SMT++
is taught to read full-page pianoform scores. This is achieved through an
incremental curriculum learning method, where the model is progressively fed
with synthetic pages of increasing graphical difficulty.

  3. 3.

Curriculum learning-based fine-tuning: A second curriculum learning strategy
is implemented for fine-tuning the SMT++, where the model is fed with combined
synthetic and real music score pages to learn the task of transcribing the
target corpus without forgetting the general knowledge provided by the
synthetic samples.

###  5.1 Synthetic Generator

The development of the training strategy requires the implementation of a
synthetic music score generator. This software is essential for providing a
sufficient variety of samples to master the task of transcribing full pages,
ensuring robustness and mitigating biases toward specific patterns. Despite
previous efforts [23, 24, 31, 2], there remains a lack of synthetic music
generators capable of producing scores suitable for end-to-end OMR
applications. To address this, we designed a synthetic generator.

The generator is initialized with a collection of single-system kern excerpts,
which the software automatically parses. To maintain rhythmic consistency, it
categorizes these samples by their beat types. Upon request for a score, the
algorithm selects a random number of samples and merges them according to kern
syntax rules.222For a comprehensive explanation of the Humdrum **kern syntax,
please refer to the official documentation
(https://www.humdrum.org/rep/kern/). The clefs and key signatures are
determined by the first chosen system, and subsequent excerpts are adjusted
accordingly. Thanks to the kern syntax agnosticism toward these elements, this
integration process is seamless. The Verovio tool [34] then renders the
generated score, producing the music score page. Upon this algorithm, the
generator also includes the following features:

  * •

Random texture placement to simulate real documents. The generator picks up a
texture from a library of 61 samples. To place a texture, the algorithm crops
the page size and places the generated music image on it.333All the texture
images are greater than standard A4 pages.

  * •

Random title and author names generator.

  * •

Varying spacing between staff lines and notes font size.

  * •

Varying inner score padding in the page.

Note that this system can be run with any single-system score dataset that is
annotated in the Humdrum **kern syntax. However, in this case, we apply it to
the generation of pianoform music scores.

(a)

(b)

(c)

Figure 5: Three synthetic samples produced by the proposed generator. These
music scores have been generated with different hyperparameters. Sample 5(a)
is a basic four-systems page, sample 5(b) is a seven-system page with a random
texture and 5(c) is a seven-system page with a random texture, generated title
and author and random margin values.

In our study, the generator produces music score images through the GrandStaff
dataset [39], which is a public single-system music score dataset designed for
end-to-end pianoform transcription. The generator employs the training set of
the corpus, which consists of musically-distorted scores to enhance the
diversity and variability of both images and transcriptions.444These
distortions are produced through transposition and offset augmentations. This
approach ensures that the data does not mimic any existing music scores,
aligning with the objectives of this work. Figure 5 depicts three music scores
generated through this algorithm.

###  5.2 System-level pretraining

The first step of this training strategy is to pretrain the SMT++ model to
learn the first reading order complexity in full-page polyphonic scores:
system-level transcription. Unlike, the OCR and HTR fields, this task is
already complex, since polyphonic systems are composed of a group of staves
that have to be simultaneously read in order to retrieve adequately their
content. Given the kern syntax, this means that the attention must jump
through staves from top to bottom. This step is crucial to learn the
transcription of full pages, since this first task has already a complex
reading order in the hierarchy of polyphonic transcription. Since the
GrandStaff dataset is composed of single-system music excerpts, the model can
be trained to the task through this dataset.

###  5.3 Curriculum Learning

Once the SMT++ model is trained on transcribing single-system scores, it is
therefore trained through a curriculum learning-based pipeline to be able to
transition from single-system transcription into full-page documents.

####  5.3.1 Incremental Curriculum Learning

The second stage consists in teaching the SMT++ model to learn the reading
order of a multi-system document, i.e. a full page music score. This stage is
performed through an incremental curriculum learning, where the synthetic
generator produces pages of increasing number of systems. The generator starts
producing pages of two systems—as it already knows how to transcribe one—and
it increases until a value of $n$ systems per page is reached, which is
determined by the user. The number of systems per page increases when a
specific number of samples $s$ has been fed into the model, which is also set
up by the user. In order to avoid catastrophic forgetting, the generator does
not focus on producing a specific number of systems, but randomly selects
between the current page complexity and the previous ones during each stage.
For example, in the third step, the model can produce pages up to three
pianoform music systems, including documents with one and two systems. In this
paper, we establish $n=5$ and $s=40.000$.

####  5.3.2 Curriculum Learning-based fine-tuning

To finetune towards the target dataset, a second curriculum learning stage is
implemented. The process feeds training samples through a probabilistic
policy. For each training sample, the model computes a random number. If it is
lower than a synthetic sample threshold $P(g)$, a synthetic sample is
provided. If not, a real one is fed. In order to ensure that the number of
synthetic samples is reduced as the training progresses, $P(g)$ is regularized
through a linear scheduler function:

$P(g)=P_{\text{max}}+\frac{t(P_{\text{min}}-P_{\text{max}})}{N}$ (3)

where $P(g)$ is the probability of providing a synthetic sample from the
generator, $P_{\text{max}}$ and $P_{\text{min}}$ are the maximum and minimum
probabilities of this threshold, $t$ is the current sample global index during
training—also known as training step—and $N$ is the total number of samples
the probabilities have to be distributed in. This scheduler provides mostly
synthetic samples during early stages of the training, while it mostly gives
real samples when the last steps are reached. In our case, we determined the
values of $P_{\text{max}}=0.9$, $P_{\text{min}}=0.2$ and $N=200.000$.

##  6 Experimental setup

In this section, we describe the experimental environment designed to test the
capability of the proposed method in this work555The implementation of the
model and the datasets are available in https://github.com/antoniorv6/SMT-
plusplus.git.

###  6.1 The Full-Page GrandStaff dataset

The performance of the SMT++ model is tested through the Full-Page GrandStaff
dataset, hereinafter referred to as FP-GrandStaff. This corpus consists of
$688$ full-page music transcriptions from six well-known music
composers—provided by the Humdrum repository.666https://github.com/humdrum-
tools/humdrum-data The music score images were processed and rendered in the
same way as described in the work of [39], since they are the full-page
versions of the scores produced in the GrandStaff dataset. Table 1 depicts a
general overview of the corpus.

Table 1: Details of the corpora regarding the features of the FP-GrandStaff dataset, such as sizes in pixels, number of samples and staves per page, number of symbols present and unique symbols. | FP-GrandStaff
---|---
Num pages | 688
Max page size | $2970\times 2100$
Min page size | $2970\times 2100$
Avg systems per page | 5
Max systems per page | 7
Min systems per page | 1
Avg symbols per page |
kern | 1184
ekern | 1363
bekern | 1897
Max symbols per page |
kern | 2316
ekern | 3112
bekern | 4358
Min symbols per page |
kern | 30
ekern | 30
bekern | 34
Unique symbols |
kern | 20578
ekern | 5282
bekern | 184

We should stress that the training and the test samples from the FP-GrandStaff
are not related to the ones from the GrandStaff dataset.

###  6.2 SMT++ configuration

We propose an image-to-sequence approach with two different feature
extractors. The implementation architecture is the same one as in the SMT
[40]. The SMT++ contains a Transformer decoder with eight layers, four
attention heads and an embedding size of 256 features in both the attention
and feed-forward modules. Given this common framework and the results obtained
in the previous work, the two variants of the feature extractor are
implemented:

  * •

CNN As occurs in the recent works of HTR [44, 16, 18], the feature extractor
is a fully-connected Convolutional Neural Network (CNN). The convolutional
backbone of this model rescales the height of the image by 16 and the width by
8, since this attained the best results in the SMT experiments. This
implementation is referred to as $\text{SMT++}_{\text{CNN}}$.

  * •

ConvNeXt The Computer Vision field has recently provided new architectures for
feature extraction. One approach that could be taken is the implementation of
a Swin Transformer [29] as an encoder. However, research on the SMT
demonstrates that this model backbone a higher amount of data variety than the
GrandStaff dataset is able to provide [40]. The ConvNeXt [30] was also tested
as a counterpart to this network. This is an adaptation of traditional
convolutional backbones, specifically the ResNet50, so as to imitate the
behavior of the Swin Transformer. As an advantage, this network does not have
the data-demanding requirement Transformer-based models has. We follow the
same implementation as in the SMT, where we maintain only the first three
layers—with the same ratios—of the network with depths of $64$, $128$ and
$256$. We refer to this model as the $\text{SMT++}_{\text{NeXt}}$.

###  6.3 Evaluation protocol

The assessment of OMR experiments is one of the main challenges of the field.
Despite several efforts [26, 46], it is not clear how to measure the quality
of the results provided by OMR. In this work, however, we measure the
performance of the system when dealing with transcription. Therefore, it is
convenient to employ text-related metrics, as they express the effort a user
would invest into correcting the automatic transcription provided by the
method. This tends, implicitly, to correlate with the quality of the
transcription, making it a coherent approach to retrieve also insights about
the quality of the documents. Specifically, we resort to the same metrics
reported in [39]. All of these measures are based on the normalized mean edit
distance between a hypothesis sequence and a reference sequence by the length
of the latter. These measures are the Character Error Rate (CER), which
evaluates among minimum semantic units in the output encoding, Symbol Error
Rate (SER), a commonly-used metric that takes complete symbols as evaluation
tokens, and Line Error Rate (LER), which approximates the accuracy of the
model as regards both content and **kern document structure
retrieval.777Evaluating the structure of the document is a key aspect in OMR
when the transcription is more complex than the staff level, since the correct
alignment between transcribed notes is relevant for a retrieval of the correct
interpretation We should note that all of the encodings are converted into the
same one, bekern, to provide a fair comparison.

In our setup, the FP-GrandStaff corpus was divided in three partitions, with
$60\%$ of samples for training, $20\%$ for validation and $20\%$ for testing.

##  7 Results

In this section, we report and discuss de retrieved results from our
experiments with the implementations of the SMT++ network.

Table 2 reports the average CER, SER and LER provided in the test set of the
FP-GrandStaff dataset for each network implementation and encoding.

Table 2: Average CER, SER, and LER (%) obtained by the models studied in the test set for the presented datasets in this work. Corpus and encoding |  | FP-SMT CNN | FP-SMT NexT
---|---|---|---
FP-GrandStaff |  |  |
kern | CER | 15.8 | 7.2
SER | 25.2 | 9.7
LER | 44.3 | 19.9
ekern | CER | 17.3 | 6.8
SER | 21.3 | 8.5
LER | 35.7 | 16.6
bekern | CER | 15.2 | 5.6
SER | 19.7 | 6.9
LER | 32.0 | 12.9

The results demonstrate that the SMT++, trained with the curriculum learning
proposed in Section 5, is able to transcribe full-page music scores with
remarkable accuracy. Specifically, the $\text{SMT++}_{\text{NeXt}}$ yields the
best performance in all the encodings. This proves that modern convolution
approaches—although inspired by the advancements in the Transformer
architecture—seem to be capable of retrieving better features and information
from complex sources, such as full-page documents.

From an encoding perspective, the tokenization processes proposed in this work
improve the performance of the model. Both the ekern and the bekern encodings
lead to better results than standard kern, with improvements of $13.2\%$ and
$29.6\%$ respectively. Between these alternative representations, the bekern
notation yields an $18\%$ lower SER than the ekern encoding. This proves that,
in the case of music, it is preferable to perform a granular semantic-aware
tokenization of the ground truth sequences, although some correlation from the
image may be lost.

Errors visualization from the test set of the FP-GrandStaff dataset are shown
in Figures 6 and 7. The highlighted errors were computed using the musicdiff
tool [19] and visualized with the MuseScore editor.888https://musescore.org/es
The example shows that the model is capable of transcribing accurately a
complex full-page music score. Most of the errors are pitch-related, where
some notes and accidentals are misplaced.

(a) Input test image

(b) Prediction from the $\text{SMT++}_{\text{NeXt}}$

Figure 6: Test example from the FP-GrandStaff dataset with the errors
highlighted. This specific sample attained a CER of 5.0%, a SER of 6.3% and a
LER of 11.3%.

Figure 7: Enlarged predictions with highlighted errors from Figure 6. The
input page is on the left and the predictions with highlighted errors on the
right.

From these results, we demonstrate that the SMT++ is an end-to-end method
capable of transcribing full-page pianoform documents with remarkable
accuracy.

##  8 Ablation studies

In addition to the results provided in Section 7, we conducted several
ablation studies in the key features of the SMT++ to assess their relevance in
the task of learning to transcribe full-page documents. All of the experiments
presented in this section were performed with the model that provided the best
results in Table 2, which is the $\text{SMT++}_{\text{NeXt}}$ with the bekern
notation vocabulary.

###  8.1 Curriculum Learning stages

The first evaluation is set on the training pipeline described in Section 5.
This experiment consists in training the $\text{SMT++}_{\text{NeXt}}$ model
alternating different combinations of the two curriculum learning stages: the
system-level pretraining, Sec. 5.2, and the incremental curriculum learning,
Sec. 5.3. Table 3 shows the obtained results in the test set of the FP-
GrandStaff dataset.

Table 3: Ablation study on curriculum learning stages. The steps from our training pipeline are numbered, being (1) the system-level pretraining, (2) the curriculum learning pipeline. | Steps |  | Metrics
---|---|---|---
| (1) | (2) |  | CER | SER | LER
FP-GrandStaff | ✗ | ✗ |  | 32.1 | 42.6 | 71.5
✓ | ✗ |  | 8.1 | 9.8 | 17.9
✗ | ✓ |  | 10.1 | 12.4 | 21.9
✓ | ✓ |  | 5.6 | 6.9 | 12.9

The results give several insights. First, the results on training directly the
model on the data show that it is mandatory to pretrain the model, as very
high CER, SER and LER values are reported. In the individual pretraining
analysis, we observe that a noteworthy improvement is reported when
performing, at least, one of the two steps proposed in this work, reporting a
$76.9\%$ reduction of SER when pretraining on single systems and a $70.82\%$
when training through the curriculum learning and finetuning step. This shows
that both steps are crucial to achieve full-page transcription. Specifically,
it seems that pretraining on single-system images is the most important step.
This is because this pretraining helps the model to understand the first
reading order hierarchy of the pianoform documents. From that point, the model
only has to learn how to read sequentially multiple systems from top to
bottom, as no reading order alterations are found in this document, which
seems to be an easier task once single-systems are learned. Despite this, the
results on the full pipeline reports superior performance than applying
individually each step, improving the baseline result up to $83\%$. This shows
that, to get the better performance from the SMT++ model, it is required to
run the whole pipeline described in Section 5.

###  8.2 Positional Encoding

The other analyzable aspect the SMT++ model is the PE applied to send the
information from a feature map, provided by the feature extractor network, to
a sequence processing module, the Transformer decoder in this case. To do so,
we resort to a 2D PE formula, which is described in Section 3.2. However,
Transformers were originally proposed to use a 1D PE, and has been applied in
other related works [27, 5, 28]. This ablation study is design to analyze the
influence of employing Positional Encoding in this data structure and assess
the difference in performance that has applying a 2D formula instead a 1D one.

Table 4: Ablation study on Position Encoding. Position Encoding |  | FP-GrandStaff
---|---|---
| CER | SER | LER
No Position Encoding |  | 57.3 | 78.4 | 100
Position Encoding 1D |  | 13.6 | 15.57 | 23.7
Position Encoding 2D |  | 5.6 | 6.9 | 12.9

Table 4 depicts the results retrieved in three scenarios: no PE, 1D PE and 2D
PE. The baseline results—where no position information is given—demonstrate
that the application of this formula to the feature map is mandatory to
perform full-page transcription. This was an expected outcome. The application
of Positional Encoding to this feature map is shown to improve on
transcription performance. Specifically, the 2D PE yields better results,
outperforming the 1D formula by a $55.7\%$ in the SER and a $45.5\%$ in the
LER. These results show that, when giving 2D position information, the model
does not only generate better transcription, but significantly more consistent
documents. This is because the reading order in pianoform scores highly
depends on the vertical axis—recall that we have two hierarchies between
system staves and page systems. The explicit encoding this is given in the 2D
PE eases the model to identify this elements along the height of the document.
In the case of 1D PE, this information has to be learned implicitly, which is
harder. This experiment demonstrates that, when transcribing complex full-page
music scores, it is preferrable to apply a 2D formula to the computed feature
map from the encoder. This is supported with other related works on full-page
transcription for HTR [44, 16], where their experiments results lead in the
same way.

##  9 Conclusions

In this paper, we address the challenge of advancing Optical Music Recognition
beyond single-staff transcription, which has traditionally involved pipeline-
based approaches that depend on previous Layout Analysis steps. We introduce
the Sheet Music Transformer++ (SMT++) model, an autoregressive Transformer-
based network that computes attention and language modeling to transcribe
full-page music scores into digital music notation formats. We evaluate this
approach in a full-page polyphonic music scenario. Specifically, we transcribe
pianoform scores, which are the most difficult ones among the music scores
repertorium. We design a three-step curriculum learning-based workflow that
employs synthetic generation to learn how to read these documents. Our results
demonstrate that the SMT++ model effectively transcribes these complex music
scores. Since this topic has not been explored before, this model standas as a
state-of-the-art approach for full-page polyphonic music transcription.

This research also opens avenues for the further improvements to OMR. First,
while this model establishes a new benchmark in OMR, opportunities for
refinement exist in the curriculum learning pipeline, where integrating music
notation learning could improve language model correlation with musical
elements. Additionally, the model, as all Transformer-based applications, is
data demanding and inefficient. Future research can be directed towards
optimization strategies and Self-Supervised Learning (SSL) to enhance
performance and reduce reliance on large, labeled datasets, respectively. The
exploration of a Universal OMR transcription model represents also a
significant advancement, which we believe is unlocked by the SMT++ network.
This initiative aims to develop a neural network capable of transcribing any
music score, independently on its notation language, musical texture or
engraving style. Achieving this ambitious objective requires a focus on
advanced methodologies that extend beyond mere recognition, focusing towards a
deep understanding of music scores. To achieve this, innovative research
approaches are required to delve into the mechanisms of music comprehension.
Such strategies promise to offer not only universal solutions but also to
enhance the current state of the art in the field. If successful, this line of
research could establish the groundwork for a significant milestone in OMR
research.

## References

  *   * Alfaro-Contreras . [2022] Alfaro-Contreras:PRL:2022Alfaro-Contreras, M., Ríos-Vila, A., Valero-Mas, J.J., Iñesta, J.M. Calvo-Zaragoza, J.  2022\.  Decoupling music notation to improve end-to-end Optical Music Recognition Decoupling music notation to improve end-to-end optical music recognition.  Pattern Recognition Letters158157–163,
  * Alfaro-Contreras . [2023] AlfaroContreras:IJMIR:2023Alfaro-Contreras, M., Iñesta, J.M. Calvo-Zaragoza, J.  2023May26.  Optical music recognition for homophonic scores with neural networks and synthetic music generation Optical music recognition for homophonic scores with neural networks and synthetic music generation.  International Journal of Multimedia Information Retrieval12112,
  * Alfaro-Contreras . [2021] Alfaro-Contreras:ISMIR:2021Alfaro-Contreras, M., Rizo, D., Inesta, J.M. Calvo-Zaragoza, J.  202111\.  OMR-assisted transcription: a case study with early prints OMR-assisted transcription: a case study with early prints.  Proceedings of the 22nd International Society for Music Information Retrieval Conference Proceedings of the 22nd International Society for Music Information Retrieval Conference ( 35–41).  OnlineISMIR.
  * Baró . [2022] Baro:ICFHR:2022Baró, A., Riba, P. Fornés, A.  2022\.  Musigraph: Optical Music Recognition Through Object Detection and Graph Neural Network Musigraph: Optical music recognition through object detection and graph neural network.  International Conference on Frontiers in Handwriting Recognition International conference on frontiers in handwriting recognition ( 171–184).
  * Blecher . [2023] Blecher:Arxiv:2023Blecher, L., Cucurull, G., Scialom, T. Stojnic, R.  2023\.  Nougat: Neural Optical Understanding for Academic Documents. Nougat: Neural optical understanding for academic documents.
  * Bluche [2016] Bluche:NIPS:2016Bluche, T.  2016\.  Joint Line Segmentation and Transcription for End-to-End Handwritten Paragraph Recognition Joint line segmentation and transcription for end-to-end handwritten paragraph recognition.  Advances in Neural Information Processing Systems 29: Annual Conference on Neural Information Processing Systems 2016, December 5-10, 2016, Barcelona, Spain Advances in neural information processing systems 29: Annual conference on neural information processing systems 2016, december 5-10, 2016, barcelona, spain ( 838–846).
  * Bluche . [2017] Bluche:ICDAR:2017Bluche, T., Louradour, J. Messina, R.O.  2017\.  Scan, Attend and Read: End-to-End Handwritten Paragraph Recognition with MDLSTM Attention Scan, attend and read: End-to-end handwritten paragraph recognition with MDLSTM attention.  14th IAPR International Conference on Document Analysis and Recognition, ICDAR 2017, Kyoto, Japan, November 9-15, 2017 14th IAPR international conference on document analysis and recognition, ICDAR 2017, kyoto, japan, november 9-15, 2017 ( 1050–1055).  IEEE.
  * Brown . [2020] Brown:NIPS:2020Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J.D., Dhariwal, P.Amodei, D.  2020\.  Language Models are Few-Shot Learners Language models are few-shot learners.  H. Larochelle, M. Ranzato, R. Hadsell, M. Balcan H. Lin (), Advances in Neural Information Processing Systems Advances in neural information processing systems ( 33, 1877–1901).  Curran Associates, Inc.
  * Calvo-Zaragoza . [2020] Calvo-Zaragoza:ACM:2020Calvo-Zaragoza, J., Hajič Jr., J. Pacha, A.  2020\.  Understanding Optical Music Recognition Understanding optical music recognition.  ACM Comput. Surv.534,
  * Calvo-Zaragoza Rizo [2018] Calvo-Zaragoza:ISMIR:2018Calvo-Zaragoza, J. Rizo, D.  201811\.  Camera-PrIMuS: Neural End-to-End Optical Music Recognition on Realistic Monophonic Scores Camera-PrIMuS: Neural End-to-End Optical Music Recognition on Realistic Monophonic Scores.  Proceedings of the 19th International Society for Music Information Retrieval Conference Proceedings of the 19th International Society for Music Information Retrieval Conference ( 248-255).  ISMIR.
  * Calvo-Zaragoza . [2019] Calvo-Zaragoza:PRL:2019Calvo-Zaragoza, J., Toselli, A.H. Vidal, E.  2019\.  Handwritten Music Recognition for Mensural notation with convolutional recurrent neural networks Handwritten Music Recognition for Mensural notation with convolutional recurrent neural networks.  Pattern Recognition Letters128115–121,
  * Campos . [2016] Campos:ICFHR:2016Campos, V.B., Calvo-Zaragoza, J., Toselli, A.H. Ruiz, E.V.  2016\.  Sheet music statistical layout analysis Sheet music statistical layout analysis.  2016 15th International Conference on Frontiers in Handwriting Recognition (ICFHR) 2016 15th international conference on frontiers in handwriting recognition (icfhr) ( 313–318).
  * Castellanos . [2020] Castellanos:ISMIR:2020Castellanos, F.J., Calvo-Zaragoza, J. Iñesta, J.M.  2020\.  A neural approach for full-page optical music recognition of mensural documents A neural approach for full-page optical music recognition of mensural documents.  Proc. of the 21th Int. Society for Music Information Retrieval Conference Proc. of the 21th int. society for music information retrieval conference ( 12–16).
  * Chiu . [2018] Chiu:ICASSP:2018Chiu, CC., Sainath, T.N., Wu, Y., Prabhavalkar, R., Nguyen, P., Chen, Z.Bacchiani, M.  2018\.  State-of-the-Art Speech Recognition with Sequence-to-Sequence Models State-of-the-art speech recognition with sequence-to-sequence models.  2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) 2018 ieee international conference on acoustics, speech and signal processing (icassp) ( 4774–4778).
  * Chowdhury Vig [2018] Chowddhury:BMVC:2018Chowdhury, A. Vig, L.  2018\.  An Efficient End-to-End Neural Model for Handwritten Text Recognition An efficient end-to-end neural model for handwritten text recognition.  British Machine Vision Conference 2018, BMVC 2018, Newcastle, UK, September 3-6, 2018 British machine vision conference 2018, BMVC 2018, newcastle, uk, september 3-6, 2018 ( 202).  BMVA Press.
  * Coquenet . [2023] Coquenet:TPAMI:2023Coquenet, D., Chatelain, C. Paquet, T.  2023\.  DAN: a Segmentation-free Document Attention Network for Handwritten Document Recognition Dan: a segmentation-free document attention network for handwritten document recognition.  IEEE Transactions on Pattern Analysis and Machine Intelligence4578227-8243,
  * Devlin . [2019] Delvin:ACACL:2019Devlin, J., Chang, M., Lee, K. Toutanova, K.  2019\.  BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding BERT: pre-training of deep bidirectional transformers for language understanding.  J. Burstein, C. Doran T. Solorio (), Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume 1 (Long and Short Papers) Proceedings of the 2019 conference of the north american chapter of the association for computational linguistics: Human language technologies, NAACL-HLT 2019, minneapolis, mn, usa, june 2-7, 2019, volume 1 (long and short papers) ( 4171–4186).  Association for Computational Linguistics.
  * Dhiaf . [2023] Dhiaf2023Dhiaf, M., Rouhou, A.C., Kessentini, Y. Salem, S.B.  2023\.  MSdocTr-Lite: A lite transformer for full page multi-script handwriting recognition Msdoctr-lite: A lite transformer for full page multi-script handwriting recognition.  Pattern Recognition Letters16928-34,
  * Foscarin . [2019] Foscarin:DLFM:2019Foscarin, F., Jacquemard, F. Fournier-S’niehotta, R.  2019\.  A diff procedure for music score files A diff procedure for music score files.  6th International Conference on Digital Libraries for Musicology 6th international conference on digital libraries for musicology ( 58–64).
  * Good . [2001] Good:XML:2001Good, M. .  2001\.  MusicXML: An internet-friendly format for sheet music Musicxml: An internet-friendly format for sheet music.  Xml conference and expo Xml conference and expo ( 03–04).
  * Graves . [2006] Graves:ICML:2006Graves, A., Fernández, S., Gomez, F.J. Schmidhuber, J.  2006\.  Connectionist temporal classification: labelling unsegmented sequence data with recurrent neural networks Connectionist temporal classification: labelling unsegmented sequence data with recurrent neural networks.  Proceedings of the Twenty-Third International Conference on Machine Learning, (ICML 2006), Pittsburgh, Pennsylvania, USA, June 25-29, 2006 Proceedings of the twenty-third international conference on machine learning, (ICML 2006), pittsburgh, pennsylvania, usa, june 25-29, 2006 ( 369–376).
  * Hankinson . [2011] Hankinson:ISMIR:2011Hankinson, A., Roland, P. Fujinaga, I.  2011\.  The Music Encoding Initiative as a Document-Encoding Framework The music encoding initiative as a document-encoding framework.  A. Klapuri C. Leider (), Proceedings of the 12th International Society for Music Information Retrieval Conference, ISMIR 2011, Miami, Florida, USA, October 24-28, 2011 Proceedings of the 12th international society for music information retrieval conference, ISMIR 2011, miami, florida, usa, october 24-28, 2011 ( 293–298).  University of Miami.
  * Hernandez-Olivan Beltran [2023] Hernandez:SoftwareX:2023Hernandez-Olivan, C. Beltran, J.R.  2023\.  Musicaiz: A python library for symbolic music generation, analysis and visualization Musicaiz: A python library for symbolic music generation, analysis and visualization.  SoftwareX22101365,
  * Huang . [2018] Huang:Arxiv:2018Huang, CZ.A., Vaswani, A., Uszkoreit, J., Shazeer, N., Simon, I., Hawthorne, C.Eck, D.  2018\.  Music Transformer. Music transformer.
  * Huron [1997] Huron:BMIDI:1997Huron, D.  1997\.  Humdrum and Kern: Selective feature encoding Humdrum and kern: Selective feature encoding.  Beyond MIDI,
  * Jan Hajič Pecina [2017] Hajic:ICDAR:2017Jan Hajič, j. Pecina, P.  2017\.  The MUSCIMA++ Dataset for Handwritten Optical Music Recognition The MUSCIMA++ Dataset for Handwritten Optical Music Recognition.  14th International Conference on Document Analysis and Recognition, ICDAR 2017, Kyoto, Japan, November 13 - 15, 2017 14th international conference on document analysis and recognition, ICDAR 2017, kyoto, japan, november 13 - 15, 2017 ( 39–46).  New York, USAIEEE Computer Society.
  * Kim . [2022] Kim:ECCV:2022Kim, G., Hong, T., Yim, M., Nam, J., Park, J., Yim, J.Park, S.  2022\.  OCR-Free Document Understanding Transformer Ocr-free document understanding transformer.  European Conference on Computer Vision (ECCV). European conference on computer vision (eccv).
  * Li . [2021] Li:Arxiv:2021Li, M., Lv, T., Cui, L., Lu, Y., Florencio, D., Zhang, C.Wei, F.  2021\.  TrOCR: Transformer-based Optical Character Recognition with Pre-trained Models. Trocr: Transformer-based optical character recognition with pre-trained models.
  * Liu . [2021] Liu:ICCV:2021Liu, Z., Lin, Y., Cao, Y., Hu, H., Wei, Y., Zhang, Z.Guo, B.  2021\.  Swin Transformer: Hierarchical Vision Transformer using Shifted Windows Swin transformer: Hierarchical vision transformer using shifted windows.  Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV). Proceedings of the ieee/cvf international conference on computer vision (iccv).
  * Liu . [2022] Liu:CVPR:2022Liu, Z., Mao, H., Wu, CY., Feichtenhofer, C., Darrell, T. Xie, S.  2022June.  A ConvNet for the 2020s A convnet for the 2020s.  Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Proceedings of the ieee/cvf conference on computer vision and pattern recognition (cvpr) ( 11976–11986).
  * Martinez-Sevilla . [2023] MartinezSevilla:ICDAR:2023Martinez-Sevilla, J.C., Ríos-Vila, A., Castellanos, F.J. Calvo-Zaragoza, J.  2023\.  A Holistic Approach for Aligned Music and Lyrics Transcription A holistic approach for aligned music and lyrics transcription.  Document Analysis and Recognition - ICDAR 2023 - 17th International Conference, San José, CA, USA, August 21-26, 2023, Proceedings, Part I Document analysis and recognition - ICDAR 2023 - 17th international conference, san josé, ca, usa, august 21-26, 2023, proceedings, part I ( 14187, 185–201).  Springer.
  * Pacha Calvo-Zaragoza [2018] Pacha:ISMIR:2018Pacha, A. Calvo-Zaragoza, J.  201811\.  Optical Music Recognition in Mensural Notation with Region-based Convolutional Neural Networks Optical Music Recognition in Mensural Notation with Region-based Convolutional Neural Networks.  Proceedings of the 19th International Society for Music Information Retrieval Conference Proceedings of the 19th International Society for Music Information Retrieval Conference ( 240-247).  ISMIR.
  * Pacha . [2019] Pacha:ISMIR:2019Pacha, A., Calvo-Zaragoza, J. Hajič jr., J.  2019\.  Learning Notation Graph Construction for Full-Pipeline Optical Music Recognition Learning notation graph construction for full-pipeline optical music recognition.  20th International Society for Music Information Retrieval Conference 20th international society for music information retrieval conference ( 75–82).
  * Pugin . [2014] Pugin:ISMIR:2014Pugin, L., Zitellini, R. Roland, P.  2014jan.  Verovio - A library for Engraving MEI Music Notation into SVG. Verovio - A library for Engraving MEI Music Notation into SVG.  International Society for Music Information Retrieval. International society for music information retrieval.
  * Rebelo . [2012] Rebelo:IJMIR:2012Rebelo, A., Fujinaga, I., Paszkiewicz, F., Marcal, A.R., Guedes, C. Cardoso, J.S.  2012\.  Optical music recognition: state-of-the-art and open issues Optical music recognition: state-of-the-art and open issues.  International Journal of Multimedia Information Retrieval13173–190,
  * Ríos-Vila . [2020] RiosVila:DLFM:2020Ríos-Vila, A., Calvo-Zaragoza, J. Rizo, D.  2020\.  Evaluating Simultaneous Recognition and Encoding for Optical Music Recognition Evaluating simultaneous recognition and encoding for optical music recognition.  DLfM ’20: 7th International Conference on Digital Libraries for Musicology, Montréal, QC, Canada, October 16, 2020 Dlfm ’20: 7th international conference on digital libraries for musicology, montréal, qc, canada, october 16, 2020 ( 10–17).  ACM.
  * Ríos-Vila . [2022] RiosVila:IBPRIA:2022Ríos-Vila, A., Iñesta, J.M. Calvo-Zaragoza, J.  2022\.  On the Use of Transformers for End-to-End Optical Music Recognition On the use of transformers for end-to-end optical music recognition.  Pattern Recognition and Image Analysis - 10th Iberian Conference, IbPRIA 2022, Aveiro, Portugal, May 4-6, 2022, Proceedings Pattern recognition and image analysis - 10th iberian conference, ibpria 2022, aveiro, portugal, may 4-6, 2022, proceedings ( 13256, 470–481).  Springer.
  * Ríos-Vila . [2021] RiosVila:ICDAR:2021Ríos-Vila, A., Rizo, D. Calvo-Zaragoza, J.  2021\.  Complete Optical Music Recognition via Agnostic Transcription and Machine Translation Complete optical music recognition via agnostic transcription and machine translation.  16th International Conference on Document Analysis and Recognition, ICDAR 2021, Lausanne, Switzerland, September 5-10, 2021, Proceedings, Part III 16th international conference on document analysis and recognition, ICDAR 2021, lausanne, switzerland, september 5-10, 2021, proceedings, part III ( 12823, 661–675).  Springer.
  * Ríos-Vila . [2023] RiosVila:IJDARICDAR:2023Ríos-Vila, A., Rizo, D., Iñesta, J.M. Calvo-Zaragoza, J.  2023\.  End-to-end optical music recognition for pianoform sheet music End-to-end optical music recognition for pianoform sheet music.  Int. J. Document Anal. Recognit.263347–362,
  * Ríos-Vila . [2024] RiosVila:Arxiv:2024Ríos-Vila, A., Calvo-Zaragoza, J. Paquet, T.  2024\.  Sheet Music Transformer: End-To-End Optical Music Recognition Beyond Monophonic Transcription Sheet music transformer: End-to-end optical music recognition beyond monophonic transcription.  arXiv preprint arXiv:2402.07596,
  * Ríos-Vila . [2021] RiosVila:APPSCI:2021Ríos-Vila, A., Esplà-Gomis, M., Rizo, D., Ponce de León, P.J. Iñesta, J.M.  2021\.  Applying Automatic Translation for Optical Music Recognition’s Encoding Step Applying automatic translation for optical music recognition’s encoding step.  Applied Sciences, Special Issue: Advances in Music Reading Systems119,
  * Ríos-Vila . [2022] RiosVila:ISMIR:2022Ríos-Vila, A., Inesta, J.M. Calvo-Zaragoza, J.  202212\.  End-to-End Full-Page Optical Music Recognition for Mensural Notation End-to-End Full-Page Optical Music Recognition for Mensural Notation.  Proceedings of the 23rd International Society for Music Information Retrieval Conference Proceedings of the 23rd International Society for Music Information Retrieval Conference ( 226-232).  Bengaluru, IndiaISMIR.
  * Shi . [2017] Shi:TPAMI:2017Shi, B., Bai, X. Yao, C.  2017\.  An End-to-End Trainable Neural Network for Image-Based Sequence Recognition and Its Application to Scene Text Recognition An End-to-End Trainable Neural Network for Image-Based Sequence Recognition and Its Application to Scene Text Recognition.  IEEE Transactions on Pattern Analysis and Machine Intelligence39112298–2304,
  * Singh Karayev [2021] Singh:ICDAR:2021Singh, S.S. Karayev, S.  2021\.  Full Page Handwriting Recognition via Image to Sequence Extraction Full page handwriting recognition via image to sequence extraction.  J. Lladós, D. Lopresti S. Uchida (), 16th International Conference on Document Analysis and Recognition, ICDAR 2021, Lausanne, Switzerland, September 5-10, 2021, Proceedings, Part III 16th international conference on document analysis and recognition, ICDAR 2021, lausanne, switzerland, september 5-10, 2021, proceedings, part III ( 12823, 55–69).  Springer.
  * Torras . [2021] Torras:ISMIR:2022Torras, P., Baró, A., Kang, L. Fornés, A.  2021\.  On the Integration of Language Models into Sequence to Sequence Architectures for Handwritten Music Recognition On the Integration of Language Models into Sequence to Sequence Architectures for Handwritten Music Recognition.  Proceedings of the 22nd International Society for Music Information Retrieval Conference Proceedings of the 22nd International Society for Music Information Retrieval Conference ( 690–696).  ISMIR.
  * Torras . [2023] Torras:Arxiv:2023Torras, P., Biswas, S. Fornés, A.  2023\.  The Common Optical Music Recognition Evaluation Framework. The common optical music recognition evaluation framework.
  * Tuggener . [2024] Tugenner:UPL:2024Tuggener, L., Emberger, R., Ghosh, A., Sager, P., Satyawan, Y.P., Montoya, J.others  2024\.  Real world music object recognition Real world music object recognition.
  * Vaswani . [2017] Vaswani:NIPS:2017Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N.Polosukhin, I.  2017\.  Attention is All you Need Attention is all you need.  I. Guyon . (), Advances in Neural Information Processing Systems Advances in neural information processing systems ( 30).  Curran Associates, Inc.
  * Zhang . [2017] Zhang:PR:2017Zhang, J., Du, J., Zhang, S., Liu, D., Hu, Y., Hu, J.Dai, L.  2017\.  Watch, Attend and Parse: An End-to-end Neural Network Based Approach to Handwritten Mathematical Expression Recognition Watch, attend and parse: An end-to-end neural network based approach to handwritten mathematical expression recognition.  Pattern Recognition71196–206,

  *[OCR]: Optical Character Recognition
  *[HTR]: Handwritten Text Recognition
  *[OMR]: Optical Music Recognition
  *[SMT++]: Sheet Music Transformer++
  *[SMT]: Sheet Music Transformer
  *[MEI]: Music Encoding Initiative
  *[CNN]: Convolutional Neural Network
  *[CER]: Character Error Rate
  *[SER]: Symbol Error Rate
  *[LER]: Line Error Rate