# Towards Efficient and Real-Time Piano Transcription Using Neural
Autoregressive Models

Taegyun Kwon, Dasaem Jeong, and Juhan Nam T. Kwon and J. Nam are with the
Graduate School of Culture Technology, Korea Advanced Institute of Science and
Technology, Daejeon, South Korea, 34141, e-mail<EMAIL_ADDRESS>Jeong are with Sogang University, Seoul, South Korea, 04107

###### Abstract

In recent years, advancements in neural network designs and the availability
of large-scale labeled datasets have led to significant improvements in the
accuracy of piano transcription models. However, most previous work focused on
high-performance offline transcription, neglecting deliberate consideration of
model size. The goal of this work is to implement real-time inference for
piano transcription while ensuring both high performance and lightweight. To
this end, we propose novel architectures for convolutional recurrent neural
networks, redesigning an existing autoregressive piano transcription model.
First, we extend the acoustic module by adding a frequency-conditioned FiLM
layer to the CNN module to adapt the convolutional filters on the frequency
axis. Second, we improve note-state sequence modeling by using a pitchwise
LSTM that focuses on note-state transitions within a note. In addition, we
augment the autoregressive connection with an enhanced recursive context.
Using these components, we propose two types of models; one for high
performance and the other for high compactness. Through extensive experiments,
we show that the proposed models are comparable to state-of-the-art models in
terms of note accuracy on the MAESTRO dataset. We also investigate the
effective model size and real-time inference latency by gradually streamlining
the architecture. Finally, we conduct cross-data evaluation on unseen piano
datasets and in-depth analysis to elucidate the effect of the proposed
components in the view of note length and pitch range.

###### Index Terms:

Piano Transcription, Autoregressive model, Online Transcription

##  I Introduction

Automatic music transcription (AMT) is a family of tasks that convert musical
audio signals into a piano roll or musical notes [1]. Among others, the
transcription of piano solo music has been the best-studied task because the
note boundaries are well defined in time and it is relatively easy to obtain a
large amount of training data with strongly labeled MIDI by synthesizing
virtual piano sounds from MIDI data [2] or by capturing MIDI data from piano
performance actions on computer-controlled pianos such as the Disklavier [3].
_Onsets and Frames_ is the milestone model that achieved a very high note
accuracy (around 95% in note-level F-score) by harnessing the power of deep
neural networks and the abundance of strongly labeled training data [4].

A common architecture of neural network models in the _Onsets and Frames_ and
the following work is convolutional recurrent neural networks (CRNN) where the
convolutional modules capture local acoustic features from the mel spectrogram
and the recurrent modules use the local features with temporal connections to
detect note onsets or frame-level on/off states (frames) [5, 4, 6, 7] in a
broad context. While they have shown superior performance, we observe two main
limitations. First, they use the CRNN model to detect note states through
binary classification. This approach requires multiple CRNN branches to
independently predict note onset, offset, and frame (on/off state), resulting
in potential redundancy within the model. Second, the bidirectional
connections in the recurrent modules allow the model to capture a broader
temporal context but hinder real-time inference, limiting the practical
applications of piano music transcription [8, 9, 10, 11]. While removing the
backward connection provides a simple solution, it significantly degrades
performance (about 2-3% in F-score) [4]. To address these challenges, we
previously proposed an autoregressive multi-state note model [12]. The model
has a softmax output that predicts multiple note states (onset, offset,
sustain, re-onset, off) from a single CRNN branch as shown in Figure 1 (a) and
(b)) rather than using separate binary outputs from multiple branches, thus
potentially reducing model size. In addition, the model resolved the real-time
inference problem by incorporating an autoregressive connection into a
unidirectional recurrent module, allowing the model to learn a broader context
during real-time note inference [13]. In this paper, we redesign the
autoregressive multi-state note model for lightweight and real-time inference
while improving the performance further.

The first refinement pertains to CNN blocks. In recent years, a standard
prototype for this purpose has been a simple VGG-style CNN configuration with
3x3 filters on mel spectrograms, as used in various studies [11, 5, 6, 14, 3,
12]. However, the conventional 2D convolutional layers assume the filters are
translation invariant on both the time and log-scaled frequency axis. In
reality, time-frequency patterns exhibit variations over different frequency
ranges; for example, high-pitched notes exhibit faster amplitude decay than
low-pitched notes. While increasing the number of filters can help to
accommodate this variation, it also leads to a larger model size. To
effectively capture frequency dependencies, we introduce frequency-conditioned
Feature-wise Linear Modulation (FiLM) layers to the feature maps of the CNN
blocks. Considering that local acoustic feature properties change continuously
with pitch height, we integrate FiLM layers to modulate the feature maps along
the frequency axis.

The second refinement focuses on RNN blocks. We use an autoregressive LSTM to
learn the temporal relations of frame-level note states. Since the model takes
input and output over the entire 88 pitches in an autoregressive manner, it
captures not only the temporal relations of frame-level note states (e.g.,
onset, sustain, and offset) within a single note, but also the harmonic or
sequential relations between different notes. These relations are referred to
as _intra-pitch_ and _inter-pitch_ , respectively, as illustrated in Figure 1
(c) and (d). Learning the _inter-note_ relation is similar to language
modeling in the domain of symbolic music, which is the core concept in
symbolic music modeling [15]. Leveraging such musical knowledge can
potentially improve music transcription in a more musically meaningful manner.
However, Ycart et al. have reported that integrating language models may not
be effective because it requires learning high complexity and dealing with
long-term dependencies when the sequence is given at the frame level [16]. Our
own research has also shown that note decoding based on the musical language
model is computationally expensive and does not outperform a simple rule-based
decoding approach [12]. Based on this observation, the autoregressive model
prioritizes _intra-pitch_ relations while neglecting _inter-pitch_ relations.
To achieve this, we incorporate 88 pitch-wise LSTMs, each operating
independently on every key of the piano. Furthermore, to significantly reduce
the number of parameters, we ensure that these 88 pitch-wise LSTMs share the
same parameters. While this approach requires a large RNN capacity to account
for the variation across the entire pitch range, we assume that the
fundamental role of the RNN, which involves learning the temporal relations at
the frame level, remains relatively consistent across different pitches. In
addition, the frequency-conditioned feature maps produced by the FiLM layers
in the CNN module could compensate for any potential lack of pitch dependence
in the model.

We propose two types of models using the new CNN and RNN components; one for
high performance and the other for high compactness. We validate the two
models through comprehensive experiments including an ablation study to show
the effectiveness of individual components, different model configurations to
investigate the model size and latency, and evaluation on unseen piano music
datasets. We also analyze the prediction errors of the models in the view of
note length and pitch range to better understand the effect of the CNN and RNN
components. We present piano transcription examples and video demos for real-
time inferences at the companion website
https://taegyunkwon.github.io/PARpiano/.

Figure 1: Frame-level note states and their relations across different
pitches. (a) A simplified amplitude envelope of two successive notes while the
sustain pedal is pressed. (b) Multi-state representation of notes. In our
model, each note is in one of the five states. (c) Intra-pitch relation
considers note states within a single pitch only, (d) Inter-pitch relation
considers note states across all pitches. We constrained our model to have
only intra-pitch relations by removing the inter-pitch link in LSTM.

##  II Related Work

AMT is analogous to automatic speech recognition (ASR) in that it predicts a
symbolic sequence from acoustic signals. Since ASR systems typically consist
of an acoustic model and a language model, AMT researchers have borrowed the
concept and have designed the two models separately or jointly. We follow the
same approach, but instead of having an explicit note-level language model, we
use a note-state sequence model, where the sequence model plays a limited
role, learning the transition of note states at the frame level. We review
previous work in this perspective and contrast them to our proposed model.

###  II-A Acoustic Model

The acoustic model in AMT typically takes a spectrogram as input and extracts
local time-frequency patterns to predict frame-level note activations. Early
work used non-negative matrix factorization (NMF) [17] or support vector
machine (SVM)[18, 19]. Kelz et al.[11] showed that high performance can be
achieved by a simple CNN model consisting of convolutional layers with 3x3
filters, max-pool layers, and fully connected layers. Since then, this CNN
configuration has been used as a standard prototype [7, 5, 3, 12, 20]. Other
variations include the use of filters with different sizes [6, 21] and U-net
[22], or the use of CQT instead of mel spectrogram [21, 23]. The CNN
architectures use 2D convolutional layers in common. They assume that the
filters are translation invariant not only along the time axis, but also the
log-scaled frequency axis. However, in musical signals, low and high pitch
ranges have different patterns of acoustic characteristics. In the case of
piano, the timbre is different in the low and high registers. This is usually
ignored in most CNN designs. In this study, we address this issue by applying
the frequency-conditioned FiLM layers to the feature maps in the CNN module.
This modulates the time-frequency feature maps along the frequency axis, thus
adapting the extracted features to the pitch-dependent characteristics. This
also compensates for the limited capacity of the single RNN module, which
shares parameters across the entire pitch range in the note-state sequence
model.

Figure 2: Overall system diagram. A step prediction of the system is
illustrated with the context window required for single step. The system takes
$6$ more frames in forward direction, which corresponds to 320 ms. The output
of the acoustic model are split into 88 segments, corresponding to 88 pitches.
Each segment is processed separately and then combined together. For
simplicity, velocity modules are omitted.

###  II-B Musical Language Model

Unlike ASR, the output in AMT is time-aligned to the input audio and is
usually represented at the frame level in the form of piano rolls. The
language model should therefore learn the note activation patterns in a very
high-dimensional space (e.g., 88$\times$the number of frames in a piano roll).
Previous work has largely tackled this problem using one of three approaches.
The first is to learn the correlation of note activations across the entire
high-dimensional space. Early work used restricted Boltzmann machines (RBM),
RNN and neural autoregressive distribution estimator (NADE) [24, 14]. More
recent work has used adversarial discriminator [5], autoregressive RNN [16],
and autoregressive transformer [25]. In general, these models are
computationally expensive and require a highly complex decoding scheme such as
beam search. Some of them mitigate the complexity by converting the frame-
level input to a note-level input or an event-based input [16, 25]. The second
approach is to learn the correlation of note activations pitch by pitch. In
other words, the model treats the note states at each pitch independently.
This approach prevents the model from learning the inter-note relations and
therefore the music language model focuses on the temporal evolution of each
note. This approach has been used as a post-processing (or smoothing) method
on the frame-level note activations, mainly based on the hidden Markov model
(HMM) [19, 18, 6]. The last approach uses only the acoustic model without any
explicit language model. The _Onsets and Frames_ and the subsequent work
showed that a simple rule-based inference on the output of the acoustic model
is sufficient if the acoustic model is designed to detect different note
states well [4, 7].

In our previous work, we set the CRNN model to cover the entire high-
dimensional pitch activation space in an autoregressive manner (the first
approach) [12]. As a result, the note decoding based on beam search was very
complex and did not even perform better than a simple rule-based decoding. In
this work, we redesign the RNN module to take the high-dimensional pitch
activations pitch by pitch, thus discarding the interactions between different
pitches (the second approach). In addition, we set the 88 pitch-wise RNNs to
share the same parameters. This significantly reduces the number of
parameters, making the model lightweight.

##  III Method

This section introduces the proposed model and describes the details of the
model design.

###  III-A Model Overview

An overview of the proposed model is shown in Figure 2. It is based on the
autoregressive multi-state note model from our previous study [12]. The two
main components of the base model are the recursive connection from the note
states in the previous time step to the output of the acoustic module in the
unidirectional RNN module, and the multi-state softmax output (i.e. onset,
sustain, re-onset, offset, and off), which allows the model to have a single
CRNN branch. The base model is modified in both the CNN and RNN modules.
First, frequency-conditioned FiLM layers are added to the CNN module to
increase the expressive power of the acoustic module in the frequency
dimension. The feature maps from the CNN module are divided into 88 groups
across the channel. In the RNN module, each of the feature map groups is
processed separately by the pitch-wise RNNs. The pitch-wise RNNs are composed
of 88 LSTMs and they share the parameters. With these new components, we
present two specific models. One is the “Pitch-wise AutoRegressive (PAR)”
model, which is relatively large in model size but has high performance. The
other is the “PARCompact” model, which is much smaller in model size but
maintains decent performance. Both models require 6 more frames in the forward
direction. This causes a latency of 320 ms in the real-time inference. This
set was close to the tolerance (300 ms) in the evaluation of score following
systems [26].

Figure 3: Model Diagram. The gray boxes indicate feature maps, and the rounded
white boxes indicate operations. The numbers in the Conv2d block refer to
($channels,filter\ width,filter\ height$) respectively. The Timewise FC refers
to the application of a fully connected layer at each time step. The
PARCompact model differs from the PAR model only in the middle part, as it
reduces the number of channels by applying 1x1 convolution, and splits the
feature maps into 88 pitches earlier to avoid having fully connected layers
with a large number of parameters.

###  III-B Acoustic Module

The model structure is shown in Figure 3. The acoustic module was set up to
take mel spectrogram with 700 mel bins because our preliminary experiments
show that the large size gave better performance than the commonly used 229
bins [7, 4]. The mel spectrogram is processed by three Conv-FiLM blocks and
two max-pooling layers. The Conv-FiLM block is composed of 3x3 convolutions
and a FiLM layer. The max-pooling layer is placed between the Conv-FiLM blocks
and reduces the frequency dimension by 2. All channels of the resulting
feature maps are combined (indicated as 48x175 ) and mapped to 768 neurons per
frame through a time-wise fully connected (FC) layer. The output is then
mapped by another time-wise FC layer with a multiple of 88 nodes, and the
resulting feature maps are split into 88 pitch segments of the same size,
which are then passed to the note state sequence module.

####  III-B1 Frequency-conditioned FiLM

Typical CNN modules used for a transcription model consist of several
convolutional layers with small 2D kernels (3x3) and a fully connected layer.
One drawback of such a design is that all frequency ranges are processed with
the same filters. This setup is good for learning common time-frequency
patterns over the entire frequency range, but ignores differences between low
and high frequencies, such as different decay times of overtones. We
compensate for this limitation by adding learnable linear modulation via the
FiLM layer.

Given a feature map $X$ of a convolution layer, where $i$ is the channel
index, the FiLM layer provides the feature map with a channel-wise
transformation on a condition $c$ as follows:

$FiLM(X_{i}|c)=\gamma_{i,c}X_{i}+\beta_{i,c}.$ (1)

where $\gamma$ and $\beta$ are parameterized by a small trainable neural
network. In the original paper [27], the FiLM layer was used to transform the
feature map of the visual module according to the given question in the visual
question-answer task. We modified the FiLM layer to transform the feature map
based on the relative frequency height of the feature map of the Conv2d layer,
which corresponds to the continuous pitch change from low to high.
Specifically, we set the relative frequency height as $k/F$ where $F$ is the
size of the feature map in height and $k$ is the height index mapped to the
continuous pitch change. Both $\gamma$ and $\beta$ take $k/F$ as the pitch
condition and modulate the feature maps via a small neural network ($f,g$)
(the channel index is omitted) as shown below:

$FiLM(X_{k})=\gamma_{k}X_{k}+\beta_{k}=f(\frac{k}{F})\cdot
X_{k}+g(\frac{k}{F})$ (2)

We used four layers of fully connected networks with the ReLU activation for
$\gamma$ and $\beta$. The size of the last layer was equal to the number of
the channels in the feature map, so that each index corresponds to each
channel.

###  III-C Note-State Sequence Module

####  III-C1 Pitch-wise LSTM

The note-state sequence module takes the 88 segments from the acoustic module
output and processes them independently with pitch-wise LSTMs. The pitch-wise
LSTMs focuses on the temporal transition within a pitched-note (intra-pitch)
rather than learning relationships between multiple notes (inter-pitch). The
same pitch-wise LSTMs are used across the entire 88 segments, assuming that
the temporal evolution within a note is independent of pitch. This parameter
sharing significantly reduces the model size. The output of pitch-wise LSTMs
are projected into a five-dimensional vector using the softmax layer where
each dimension corresponds to the probability of the note state (onset, re-
onset, sustain, offset, and off) at a specific pitch and time step.

####  III-C2 Enhanced Recursive Context

Figure 4: Enhanced Recursive Context. The left piano roll depicts notes states
at the specific step $t-1$ and the highlighted areas indicate the context
being considered. In [12], only the last states were considered as a context
and converted to a 2-dimensional embedding per pitch. In the Enhanced
Recursive Context setting, we additionally consider note velocity and note
duration. Also, the contexts are encoded into a 4-dimensional embedding.

Our previous auto-regressive multi-state note model used note states at the
last frame for the recursive connection in the LSTMs [12]. This single-frame
context does not contain information about how long the activated notes have
been sustained, which often resulted in overly long notes being predicted when
the model missed the note offset state. To overcome this issue, we augment the
recursive context so that the pitch-wise LSTMs catch up the states of the
playing notes effectively. Specifically, we add the duration and velocity of
the playing notes to the note states, as illustrated in Figure 4. The duration
is counted as the number of lasting frames from the onset (starting from 1).
The velocity is the MIDI note velocity of the playing notes. If a note is off,
velocity and duration are simply set to 0. The values are normalized to the
maximum value (for duration, we trimmed it to 5 secs) and embedded with two
layers of small fully connected layer with 16 units, and projected into 4
dimension with a linear layer as a context vector. This is concatenated with
the output of the acoustic module for each pitch as the input of the pitch-
wise LSTMs.

We expect that the duration and velocity information in the recursive context
provides a more complete view of the latest notes helps the model to compare
it to current acoustic features. For example, a note with a long duration and
low velocity indicates that the sound intensity of the last note is weak, so
the LSTM may become sensitive to detecting offset. Note that we used a
teacher-forcing method to train the model. In the inference phase, the
recursive context is calculated from the model prediction.

###  III-D Compact Model

We observed that the fully connected network on top of the Conv-FiLM blocks in
the acoustic module took up most of the parameters in the PAR model. This
issue is found in VGG or other CNN models that have large feature maps beneath
the fully connected layers as well [28]. To address the issue, we additionally
propose a compact version of the PAR model. As shown in Figure 3, the compact
model reduces the number of channels of the output of Conv-FiLM blocks to one
with two $1\times 1$ convolution layers, which was inspired by the inception
module [29]. Then, the feature map is split to 88 pitch segments. Lastly, the
timewise FC layers above operate on each pitch feature separately, sharing the
parameters. These modifications significantly reduce the model size. In the
experiment section, we will investigate the size limit of the compact model by
changing the number of channels in the Conv-FiLM blocks, the number of hidden
units per pitch in the FC layers, and the number of LSTM units.

##  IV Experiments

###  IV-A Datasets

We used five piano music datasets where audio and MIDI are paired. They are
all different in terms of piano quality and recording environments. We used
the MAESTRO dataset[3] as the primary one, as it has been used as a benchmark
in recent years [7, 3, 25, 5, 20]. We trained the model only with the training
split of the MAESTRO dataset and evaluated the model with the rest. We provide
a concise overview of the five datasets below.

####  IV-A1 MAESTRO [3]

This is the largest public dataset for piano music transcription. It contains
piano performances played in International Piano-e-Competition111www.piano-e-
competition.com. The MIDI files were recorded through Yamaha Disklavier pianos
which can capture MIDI messages via built-in sensors. The dataset has three
versions. We used the first (V1) and third (V3) versions for the performance
comparison. The V3 is an updated version of the V2, removing six recordings
with string quartet accompaniment that were mistakenly included.

####  IV-A2 MAPS [30]

This contains 7 sets of synthesized piano and 2 sets of acoustic piano (Yamaha
Disklavier Mark III). We used only the acoustic piano sets which include a
total of 60 pieces in the ‘MUS’ subset.

####  IV-A3 Vienna 4x22 Corpus [31]

This contains recordings of 4 short classical pieces played by 22 pianists on
the Bösendorfer SE290 grand piano. Since the provided MIDI was not aligned
with audio, we manually aligned it.

####  IV-A4 Saarland music dataset (SMD)

We used the piano subset of this dataset [32]. It contains 50 performances
played on a Yamaha Disklavier model by students of piano classes of different
levels.

####  IV-A5 In-house Dataset

We collected our own recordings of piano performances on Yamaha Disklavier C7
Enspire. The recorded pieces consist of brief excerpts (approximately 30 s in
length) extracted from classical piano compositions. These performances were
performed by highly proficient undergraduate students majoring in piano
performance. This dataset contains 461 recordings in total.

Note that we elongated the note offsets when the sustain pedal is pressed,
using the control value of 64 as a threshold [12, 4]. However, on the SMD
dataset, we observed the effective threshold was in a lower range. After a
grid search in the evaluation phase, we selected a threshold value of 21
instead of 64.

###  IV-B Evaluation Metrics

We used standard note-level evaluation metrics including precision (P), recall
(R), and F1-score (F1) and calculated them using the mir_eval library [33].
Following the previous work, note onsets were counted as correct if they are
within $\pm$50 ms of the ground truth and note offsets were as correct if they
are within $\pm$50 ms of the ground truth or $\pm$20% of the note duration For
note velocity, we used $\pm$10% of the ground truth velocity as a tolerance
window. In addition to the standard metrics, we calculated “note duration
accuracy” to measure the accuracy of note offset given the correct note onset.
It was obtained by taking the ratio of the recall of note offset to the recall
of note onset. For the previous studies where we cannot directly calculate
note duration accuracy, we approximated the value by dividing the overall
offset recall by the onset recall (which introduces about $\pm 0.1\%p$
approximation error in our own results).

TABLE I: Overall Results The parameters number of Multi-State Model [12] was
counted from the model without note velocity prediction.

|  | Online | Params | Note |  | Note w/ offsets
---
| Note w/ Offset
---
$\And$ Velocity
| Note Duration
---
Accuracy
|  |  |  | R (%) | P (%) | F1 (%) | R (%) | P (%) | F1 (%) | R (%) | P (%) | F1 (%) | %
MAESTRO V1 | Adversarial[5] |  | 26.9M | 93.2 | 98.1 | 95.6 | 79.3 | 83.5 | 81.3 | 78.2 | 82.3 | 80.2 | 85.1#
Onsets and Frames [3] |  | 24.7M | 92.6 | 98.3 | 95.3 | 78.2 | 83.0 | 80.5 | 75.4 | 79.9 | 77.5 | 84.5#
Seq2Seq[25] |  | 54M | — | — | 96.0 | — | — | 83.5 | — | — | 82.2 | —
Multi-State Model[12] | ✓ | 14.6M$*$ | 91.2 | 98.7 | 94.7 | 76.5 | 82.6 | 79.4 | — | — | — | 83.9
Proposed (PAR) | ✓ | 19.7M | 95.6 | 98.4 | 96.9 | 86.5 | 89.0 | 87.7 | 85.3 | 87.7 | 86.5 | 90.6
| Proposed (PARcompact) | ✓ | 2.7M | 94.9 | 97.2 | 96.0 | 83.7 | 85.7 | 84.7 | 70.5 | 81.5 | 80.6 | 88.3
MAESTRO V3 | Onsets and Frames [reproduced] |  | 24.7M | 89.4 | 99.5 | 94.1 | 75.9 | 84.2 | 79.8 | 72.5 | 80.3 | 76.1 | 84.9
Onsets and Frames [reproduced] | ✓ | 20.9M | 88.8 | 99.4 | 93.7 | 73.9 | 82.5 | 77.8 | 70.6 | 78.8 | 74.4 | 83.2
Regressing Onsets.[7] |  | 20.2M | 95.6 | 98.1 | 96.8 | 83.7 | 85.8 | 84.7 | 82.2 | 84.3 | 83.3 | 87.2
Seq2Seq[25] |  | 54M | — | — | 96.0 | — | — | 83.9 | — | — | 82.8 | —
Neural CRF[20] |  | 9M | 94.0 | 98.7 | 96.1 | 86.5 | 90.8 | 88.4 | 85.5 | 89.7 | 87.4 | 92.0#
Proposed (PAR) | ✓ | 19.7M | 95.6 | 98.5 | 97.0 | 86.7 | 89.2 | 87.9 | 85.5 | 88.1 | 86.8 | 90.7
Proposed (PARcompact) | ✓ | 2.7M | 94.3 | 96.9 | 95.5 | 83.9 | 86.1 | 85.0 | 83.9 | 86.1 | 85.0 | 89.0

###  IV-C Training Details

We used audio with a sample rate of 16 kHz and computed a log mel spectrogram
based on a short-time Fourier transform with a window size of 4096 and a hop
size of 512, covering frequencies from 27.5 Hz to 8372 Hz, corresponding to
the lowest and highest pitches of a piano. The number of mel bins was set to
700. The overall configuration of the proposed model is described in Figure 3.
The PAR model has 48 channels for all convolution layers, 768 nodes for the
fully connected layers, and 48 units for the pitch-wise LSTM layer. For the
modulation functions $\alpha$ and $\beta$ in the frequency-conditioned FiLM,
we used three FC layers with 16 hidden units. The window size ($W$) of the
multistep FC layer was set to 5. For the compact model, we adjusted the size
of the convolution, FC, and LSTM layer to progressively reduce the model size.
We tested a total of 5 models that cover a wide range of model sizes. The
number of parameters, unit size, and performance are compared in Table III.

For estimating note velocity, we used another model with the same
specifications as the note model, as done in the onsets and frames model [4].
The LSTM of the velocity model takes both the output of the acoustic module
from the note model and the velocity model. To prevent interference between
the two models, the gradient was disconnected between the velocity model and
the note model.

All models were trained with the focal loss[34] with $\alpha=1.0,\gamma=2.0$
and the Adabelief optimizer [35] with an initial learning rate of 1e-3, and a
batch size of 12. For note velocity, we linearly mapped the velocity range [0,
127] to [0,1] and applied the L2 loss only on the note onset frame of the
ground truth. We randomly selected 10 s during training but we evaluated the
entire piece at once during inference. We selected a model with the highest F1
validation score in note offset for up to 250k iterations. For the
implementation, we used Pytorch 1.11 and trained the models on two NVIDIA
GeForce RTX 2080 TI GPUs.

##  V Results

###  V-A Performance Comparison

Table I shows an overall comparison between the proposed model and recent
state-of-the-art models evaluated on two versions of the MAESTRO dataset.
Regarding the Onsets and Frames model, we reimplemented it to run both offline
and online and evaluated it on MAESTRO V3. The online model was implemented by
modifying the CRNN model with unidirectional LSTMs222The hyper-parameters were
set in the model in [3]. For the Regressing Onsets[7] model, we re-evaluate it
pre-trained model333We accessed the pre-trained model at
www.github.com/bytedance/piano_transcription on Dec 14, 2021 on MAESTRO V3 as
the original paper reported the result evaluated on MAESTRO V2444We observed
that the results on MAESTRO V3 in Table I are higher than those evaluated on
MAESTRO V2 in the original paper. From the table, we can see that the proposed
PAR model achieves higher or comparable accuracy in note and note with offset.
This indicates that the state-of-the-art level of note onset and offset
accuracy can be achieved without a backward context in the RNNs and learning
inter-pitch relations. The PARcompact model also achieves comparable accuracy
to other state-of-the-art models, demonstrating the effectiveness of the
streamlined model.

Compared to our previous multi-state model[12], the PAR model gained 2.2%p and
8.3%p on note onset and offset F1 scores, respectively. In addition, the
number of notes with excessively long offset prediction where the predicted
duration is more than 3sec longer than the actual duration decreased by about
70% (from 1.08% to 0.34%). As a result, the PAR model shows improvement in
note offset accuracy, which is confirmed by the note duration accuracy,
although it is slightly lower than the note F1 score with offset in the Neural
CRF model. In Section V-B, we will show that this is the result of several
contributing factors.

TABLE II: Ablation Study - PAR Model. The notation W/o indicates that the
component was removed or replaced.

| Note |  | Note w/ offsets
---
| Note w/ offsets
---
$\And$ Velocities
| Note Duration
---
Accuracy
Model | R (%) | P (%) | F1 (%) | R (%) | P (%) | F1 (%) | R (%) | P (%) | F1 (%) | (%)
(a) W/o Pitchwise LSTM | 92.57 | 98.85 | 95.55 | 80.56 | 85.95 | 83.12 | 79.51 | 84.79 | 82.02 | 90.42
(b) W/o Enhanced Context | 95.37 | 98.87 | 97.06 | 85.95 | 89.08 | 87.47 | 84.80 | 87.86 | 86.28 | 90.02
(c) W/o Freq.-conditioned FiLM | 94.81 | 98.82 | 96.74 | 85.36 | 88.92 | 87.08 | 84.24 | 87.73 | 85.92 | 90.08
(d) W/o $\\{$Pitchwise, Enh. Context, FiLM$\\}$ | 90.62 | 98.79 | 94.46 | 76.53 | 83.33 | 79.72 | 75.50 | 82.17 | 78.64 | 84.49
(e) Non-Autoregressive | 93.17 | 97.98 | 95.47 | 82.32 | 86.46 | 84.30 | 81.04 | 85.09 | 82.97 | 88.41
(f) PAR (All Components) | 95.60 | 98.49 | 97.01 | 86.63 | 89.21 | 87.88 | 85.52 | 88.05 | 86.75 | 90.67

###  V-B Ablation Study

We conducted an ablation study to investigate the influence of each component
in the PAR model (Table II). We examined the model by modifying the following
components of the PAR model one by one: (a) replacing the pitch-wise LSTM
layers by a global LSTM layer with 256 nodes covering the entire pitch range,
(b) replacing the enhanced recursive context by a context with the states of
the last frame only, (c) removing the FiLM layers, (d) replacing or removing
the three components (pitch-wise LSTM, enhanced recursive context, frequency-
conditioned FiLM layer) together, (e) removing the autoregressive connection.

The ablation shows that the pitch-wise LSTM is a critical factor in improving
performance on all accuracy metrics. Removing the enhanced context or
frequency-conditioned FiLM alone does not affect the results much, as shown in
(b) and (c), but when they are removed together with the pitch-wise LSTM, the
performance drop is highly significant (d), greater than the sum of the
individual performance drops from (a), (b), and (c). In particular, a
significant difference in note duration accuracy indicates that the
improvement of the PAR model in note offset metrics is not only due to the
improvement in onset but also to independent improvements in note duration
accuracy (offset). Finally, the autoregressive connection is as important as
the pitch-wise LSTM in terms of accuracy scores, as shown in (e).

Figure 5: Model Parameters vs. Note F1 Score. TABLE III: Compact Model
Experiment

| Model
---
| Params
---
| CNN
---
Unit
| RNN
---
Unit
| Hidden Unit
---
per Pitch
Note F1(%) |  | Note F1
---
w/ Offset(%)
Base (PARcompact) | 2.7M | 48 | 48 | 48 | 95.98 | 84.27
Middle | 1.5M | 32 | 32 | 32 | 95.50 | 82.32
Small | 653K | 16 | 16 | 16 | 90.83 | 69.08
Tiny | 299K | 8 | 8 | 8 | 81.31 | 16.47
Single bottleneck | 50K | 8 | 8 | 1 | 66.99 | 15.98

###  V-C Model Size Experiment

Figure 5 shows a comparison of model sizes between the PAR model and the state
of the art models. The PAR model have a similar number of parameters compared
to other CRNN based models, while the PARcompact model has a reduced size of
86% (from 19.7M to 2.7M) at the cost of 1.5%p decrease in the onset F1 score
and 2.9%p decrease in the offset F1 score. We investigated even smaller models
than the PARcompact model, progressively reducing the size of CNN units, RNN
units, and hidden units per pitch. We found that model training tends to be
very unstable when the unit sizes are 16 or less, but we were able to get
results sometimes. Table III shows the performance trajectory. The models can
maintain decent performance (e.g. above 90% note onset accuracy) up to the
“small” size, but they degrade dramatically in the “tiny” and “single
bottleneck’ sizes.

###  V-D Model Latency Experiment

We investigated the model performance by progressively reducing the model
latency. The latency was controlled by reducing the filter widths in the Conv-
FiLM blocks. Table IV shows the performance of both the PAR and PARcompact
models with different combinations of filter widths along the time dimension
as a 6-tuple where left to right corresponds to bottom to top layers. Model
parameters remained stable regardless of filter size. PAR model: 19.59M -
19.73M. Compact model: 2.54M - 2.68M. As expected, both note onset and offset
accuracy decrease with shorter time contexts. Interesting, except for the
extreme case where all filter widths are set to 1 (128 ms latency), the models
show small performance drops in the range of 160 ms to 320 ms latency. Between
the two models, PARcompact has larger performance drops as the latency gets
smaller.

TABLE IV: Model Latency Experiment

|  |  | Note |  | Note w/ offsets
---
|  | CNN Filter
---
Widths
| Latency
---
(ms)
R (%) | P (%) | F1 (%) | R (%) | P (%) | F1(%)
PAR | (1,1,1,1,1,1) | 128 | 90.21 | 96.20 | 93.01 | 72.99 | 77.67 | 75.18
(3,1,1,1,1,1) | 160 | 94.47 | 98.81 | 96.55 | 83.52 | 87.31 | 85.34
(3,1,3,1,1,1) | 192 | 94.93 | 98.81 | 96.81 | 84.96 | 88.40 | 86.62
(3,1,3,1,3,1) | 224 | 95.14 | 98.71 | 96.87 | 85.69 | 88.86 | 87.22
(3,3,3,3,3,3) | 320 | 95.60 | 98.49 | 97.01 | 86.63 | 89.21 | 87.89
PARcompact | (1,1,1,1,1,1) | 128 | 76.31 | 96.76 | 84.25 | 41.18 | 51.41 | 45.18
(3,1,1,1,1,1) | 160 | 92.81 | 96.01 | 94.33 | 77.02 | 79.60 | 78.25
(3,1,3,1,1,1) | 192 | 93.44 | 98.31 | 95.77 | 79.65 | 83.78 | 81.63
(3,1,3,1,3,1) | 224 | 93.56 | 97.77 | 95.58 | 77.00 | 80.43 | 78.64
(3,3,3,3,3,3) | 320 | 94.91 | 97.22 | 96.03 | 83.67 | 85.69 | 84.65

TABLE V: Cross Dataset Evaluation.

| Model |  | MAESTRO
---
V3
MAPS | SMD |  | Vienna
---
Corpus
In-house
Note | PAR | 97.0 | 79.9 | 95.2 | 93.5 | 96.1
PARcompact | 95.6 | 82.0 | 95.9 | 94.8 | 96.1
Regressing Onsets | 96.8 | 82.2 | 96.7 | 96.5 | 96.8
Onsets and Frames (r) | 94.1 | 80.2 | 92.5 | 93.2 | 94.8
Note /off | PAR | 88.0 | 57.3 | 84.3 | 80.0 | 88.8
PARcompact | 85.1 | 60.3 | 82.9 | 76.5 | 88.0
Regressing Onsets | 84.7 | 60.9 | 83.3 | 83.4 | 87.7
Onsets and Frames (r) | 79.8 | 55.1 | 76.2 | 67.2 | 83.4

###  V-E Cross Dataset Evaluation

We evaluated the proposed models (PAR and PARcompact) on four piano music
datasets other than MAESTRO. The unseen datasets during model training have
different recording conditions and piano quality. Therefore, the results can
show the generalization ability of the models. Table V shows the accuracy
scores, comparing the proposed models with the Regressing Onsets model [7] and
the Onsets and Frames model [3]. Note that the two models compared were also
trained in the same MAESTRO v3 training set and Onsets and Frames is the
reproduced model. On Note F1 score, the Regressing Onsets model generally
outperforms the proposed model and the Onsets and Frames model. This is likely
because the regression loss proposed to mitigate the alignment errors in the
audio-MIDI pairs of the MAESTRO dataset increases model generalization.
Interestingly, the PARcompact model is the second best, achieving better
performance than the PARcompact model. This seems to be due to a
regularization effect by the small model size. On Note with offset F1 score,
the trend is different. The PAR model become comparable to the Regressing
Onsets model, archiving best performance on SMD and In-house. This is likely
due to the enhanced recursive context designed to reduce note offset missing.

The three models generally show low performance on MAPS. We suspect that some
of the errors may be due to annotation errors in the dataset itself, as
reported in previous work [36, 4]. Another possibility is that the acoustic
difference between an upright piano and a grand piano is significant, given
that the MAPS dataset is the only one played on an upright piano.

Figure 6: Note Length Vs Note Recall. Solid and dotted lines correspond to
onset and offset, respectively. A thick light blue line indicates the number
of notes in the training set (log scale).

###  V-F Performance Analysis: Note Length View

Note offset accuracy is likely to decrease for longer notes because there are
more time frames to predict a note offset given a note onset. To address this
issue, we incorporated the extended recursive context into the auto-regressive
connection and demonstrated its effectiveness in the ablation study (section
V-B). We examine it further by analyzing the model’s performance in the note
duration view. Figure 6 plots the note with offset recall score over different
note durations for the proposed models, the onsets and frames model, and the
Regressing Onset model. While they have similar trends in note onset accuracy
(solid lines), the two autoregressive models (PAR and PARcompact) have a
slower decline in the note offset recall than the two non-autoregressive
models (dotted lines). In particular, the Regressing Onset model has a rapid
decline in the recall performance for longer notes (dotted green line), being
unable to estimating any note longer than 8s. This is presumably because the
high time-resolution of the model (10ms per frame) hinders learning dependence
between onset and offset in long notes. This result confirms that our proposed
auto-regressive models are more reliable for longer notes.

Figure 7: Pitch Vs Recall. The figure below is an enlarged view of both ends
of pitches. The thick light blue line indicates the number of notes in the
training set (log scale).

###  V-G Performance Analysis: Pitch Range View

The distribution of notes played in most piano music, including MAESTRO, is
unbalanced; low or high notes are played relatively less frequently than
middle notes. The imbalance distribution affects the transcription
performance; the note accuracy is generally proportional to the note
frequency. We investigate the effect of components in our proposed models.
Figure 7 plots note onset recall against note number, comparing the four
models. As expected, note onset recall decreases at both ends of the pitch
range. However, the PAR and PARcompact models have much higher recall at both
ends. This is more easily recognizable in the magnified plots of the very low
and high pitch range in the bottom of Figure 7. We particularly attribute this
to the pitchwise LSTM because the parameters of the pitchwise LSTMs are shared
across all pitches and therefore low and high pitch notes harness the note
state information from middle pitch notes.

We further validate this “pitch generalization” effect by visualizing the
mutual information between LSTM neurons and pitches. Figure 8 contrasts the
mutual information in the normal LSTM (model (c) in Table II) and the second
LSTM layer of the pitchwise LSTM. To compute the mutual information, we
randomly sampled 20,000 frames from the training dataset and used a function
in the scikit-learn555https://scikit-
learn.org/stable/modules/generated/sklearn.feature_selection.mutual_info_classif.html
[37] package based on entropy estimation from k-nearest neighbor distances
between continuous variables (LSTM layer activations) and discrete variables
(pitch labels) [38]. The visualization shows that a significant proportion of
the neurons in the normal LSTM are assigned to a single pitch. The number of
nodes assigned to each pitch was roughly proportional to the number of notes
in that pitch in the dataset. As a result, many pitches on both ends were not
assigned to any neurons in the normal LSTM. In contrast, our PAR model had an
equal and large number of neurons assigned to each pitch, regardless of the
number of notes in each pitch, resulting in better generalization across
pitches.

Figure 8: Mutual information between the LSTM activations (second layer) and
the notes in each pitch. PAR with normal LSTM model ((b) in Tab. II and PAR
model with pitchwise LSTM (f).

##  VI Conclusions

We presented novel CRNN models for piano transcription that can simultaneously
achieve high-performance, lightweight, and real-time inference. Through the
comprehensive experiments, we show that the the proposed models are comparable
to previous state-of-the-art models in performance, while having a compact
model size. We validated the effectiveness of the proposed components
(frequency-conditioned FiLM, pitchwise LSTM, and enhanced recursive context)
through an ablation study. Also, we analyzed the results in the view of note
length and pitch range to elucidate the effect of the proposed components
further.

One of the limitations in this study is that the evaluation on unseen piano
dataset (Section V-E) was conducted only when the models are trained with a
single training set (MAESTRO V3). In order to validate the generalization
power and the effectiveness on real-world examples, we need to augment the
dataset [3] or leverage unlabeled datasets with the aid of semi-supervised
[39] or unsupervised learning method [40, 41]. In addition, our models do not
include an inter-pitch language model. While our intra-pitch-focused models
achieved high performance, inter-pitch connections are still useful if the
time resolution can be reduced [16]. Finally, we look forward to the
development and implementation of a system capable of analyzing a performance
or providing real-time analysis in real-world recording scenarios.

## Acknowledgment

This work was supported by the National Research Foundation of Korea (NRF)
grant funded by the Korea government (MSIT) (No. NRF-2023R1A2C3007605).

## References

  * [1] E. Benetos, S. Dixon, Z. Duan, and S. Ewert, “Automatic music transcription: An overview,” _IEEE Signal Processing Magazine_ , vol. 36, no. 1, pp. 20–30, 2018.
  * [2] V. Emiya, N. Bertin, B. David, and R. Badeau, “Maps-a piano database for multipitch estimation and automatic transcription of music,” 2010.
  * [3] C. Hawthorne, A. Stasyuk, A. Roberts, I. Simon, C.-Z. A. Huang, S. Dieleman, E. Elsen, J. Engel, and D. Eck, “Enabling factorized piano music modeling and generation with the MAESTRO dataset,” in _International Conference on Learning Representations_ , 2019.
  * [4] C. Hawthorne, E. Elsen, J. Song, A. Roberts, I. Simon, C. Raffel, J. Engel, S. Oore, and D. Eck, “Onsets and frames: Dual-objective piano transcription,” in _Proc. of the 19th International Society for Music Information Retrieval Conference (ISMIR)_ , October 2018, pp. 50–57.
  * [5] J. Kim and J. Bello, “Adversarial learning for improved onsets and frames music transcription,” in _Proc. of 20th International Society for Music Information Retrieval Conference (ISMIR)_ , 2019, pp. 670–677.
  * [6] R. Kelz, S. Böck, and G. Widmer, “Deep polyphonic ADSR piano note transcription,” in _Proc. of the 44th International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. IEEE, May 2019, pp. 246–250.
  * [7] Q. Kong, B. Li, X. Song, Y. Wan, and Y. Wang, “High-resolution piano transcription with pedals by regressing onset and offset times,” _IEEE/ACM Transactions on Audio, Speech, and Language Processing_ , vol. 29, pp. 3707–3717, 2021.
  * [8] J. J. Carabias-Orti, F. J. Rodríguez-Serrano, P. Vera-Candeas, F. J. Cañadas-Quesada, and N. Ruiz-Reyes, “Constrained non-negative sparse coding using learnt instrument templates for realtime music transcription,” _Engineering Applications of Artificial Intelligence_ , vol. 26, no. 7, pp. 1671–1680, 2013.
  * [9] M. Pesek, A. Leonardis, and M. Marolt, “Robust real-time music transcription with a compositional hierarchical model,” _PloS one_ , vol. 12, no. 1, p. e0169411, 2017.
  * [10] K. Vaca, A. Gajjar, and X. Yang, “Real-time automatic music transcription (amt) with zync fpga,” in _2019 IEEE Computer Society Annual Symposium on VLSI (ISVLSI)_. IEEE, 2019, pp. 378–384.
  * [11] R. Kelz, M. Dorfer, F. Korzeniowski, S. Böck, A. Arzt, and G. Widmer, “On the potential of simple framewise approaches to piano transcription,” in _Proc. of the 17th International Society for Music Information Retrieval Conference (ISMIR)_ , August 2016, pp. 475–481.
  * [12] T. Kwon, D. Jeong, and J. Nam, “Polyphonic piano transcription using autoregressive multi-state note model,” in _Proc. of the 21th International Society for Music Information Retrieval Conference (ISMIR)_ , 2020\.
  * [13] D. Jeong, “Real-time automatic piano music transcription system,” in _Late Breaking and Demo of the 21st International Society of Music Information Retrieval (ISMIR) Conference_ , 2020.
  * [14] Q. Wang, R. Zhou, and Y. Yan, “Polyphonic piano transcription with a note-based music language model,” _Applied Sciences_ , vol. 8, no. 3, p. 470, 2018.
  * [15] N. Boulanger-Lewandowski, Y. Bengio, and P. Vincent, “Modeling temporal dependencies in high-dimensional sequences: Application to polyphonic music generation and transcription,” in _Proc. of the 29th International Conference on Machine Learning (ICML)_ , 2012, pp. 1881––1888.
  * [16] A. Ycart, A. McLeod, E. Benetos, and K. Yoshii, “Blending acoustic and language model predictions for automatic music transcription,” in _Proc. of the 20th International Society for Music Information Retrieval Conference (ISMIR)_ , 2019, pp. 454–461.
  * [17] E. Vincent, N. Bertin, and R. Badeau, “Adaptive harmonic spectral decomposition for multiple pitch estimation,” _IEEE Transactions on Audio, Speech, and Language Processing_ , vol. 18, no. 3, pp. 528–537, 2009.
  * [18] J. Nam, J. Ngiam, H. Lee, and M. Slaney, “A classification-based polyphonic piano transcription approach using learned feature representations,” in _Proc. of the 12th International Society for Music Information Retrieval Conference (ISMIR)_ , 2011, pp. 175–180.
  * [19] G. E. Poliner and D. P. Ellis, “A discriminative model for polyphonic piano transcription,” _EURASIP Journal on Advances in Signal Processing_ , vol. 2007, pp. 1–9, 2006.
  * [20] Y. Yan, F. Cwitkowitz, and Z. Duan, “Skipping the frame-level: Event-based piano transcription with neural semi-crfs,” _Advances in Neural Information Processing Systems_ , vol. 34, 2021.
  * [21] Q. Wang, R. Zhou, and Y. Yan, “A two-stage approach to note-level transcription of a specific piano,” _Applied Sciences_ , vol. 7, September 2017.
  * [22] Y.-T. Wu, B. Chen, and L. Su, “Multi-instrument automatic music transcription with self-attention-based instance segmentation,” _IEEE/ACM Transactions on Audio, Speech, and Language Processing_ , vol. 28, pp. 2796–2809, 2020.
  * [23] M. Taenzer, S. I. Mimilakis, and J. Abeßer, “Informing piano multi-pitch estimation with inferred local polyphony based on convolutional neural networks,” _Electronics_ , vol. 10, no. 7, p. 851, 2021.
  * [24] N. Boulanger-Lewandowski, Y. Bengio, and P. Vincent, “High-dimensional sequence transduction,” in _2013 IEEE International Conference on Acoustics, Speech and Signal Processing_. IEEE, 2013, pp. 3178–3182.
  * [25] C. Hawthorne, I. Simon, R. Swavely, E. Manilow, and J. Engel, “Sequence-to-sequence piano transcription with transformers,” in _Proc. of the 22th International Society for Music Information Retrieval Conference (ISMIR)_ , 2021.
  * [26] A. Cont, D. Schwarz, N. Schnell, and C. Raphael, “Evaluation of real-time audio-to-score alignment,” in _International Symposium on Music Information Retrieval (ISMIR)_ , 2007.
  * [27] E. Perez, F. Strub, H. De Vries, V. Dumoulin, and A. Courville, “Film: Visual reasoning with a general conditioning layer,” in _Proceedings of the AAAI Conference on Artificial Intelligence_ , vol. 32, no. 1, 2018.
  * [28] K. Simonyan and A. Zisserman, “Very deep convolutional networks for large-scale image recognition,” in _Proceedings of the International Conference on Learning Representations (ICLR)_ , 2015.
  * [29] C. ”Szegedy, W. ”Liu, Y. ”Jia, P. ”Sermanet, S. ”Reed, D. ”Anguelov, D. ”Erhan, V. ”Vanhoucke, and A. ”Rabinovich, “Very deep convolutional networks for large-scale image recognition,” in _Proceedings of the International Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2015.
  * [30] V. Emiya, R. Badeau, and B. David, “Multipitch estimation of piano sounds using a new probabilistic spectral smoothness principle,” _IEEE Transactions on Audio, Speech, and Language Processing_ , vol. 18, no. 6, pp. 1643–1654, 2009.
  * [31] W. Goebl. (1999) The vienna 4x22 piano corpus. http://dx.doi.org/10.21939/4X22.
  * [32] M. Müller, V. Konz, W. Bogler, and V. Arifi-Müller, “Saarland music data (SMD),” in _Late-Breaking and Demo Session of the 12th International Conference on Music Information Retrieval (ISMIR)_ , Miami, USA, 2011.
  * [33] C. Raffel, B. McFee, E. J. Humphrey, J. Salamon, O. Nieto, D. Liang, and D. P. Ellis, “mir_eval: A transparent implementation of common MIR metrics,” in _Proc. of the 15th International Society for Music Information Retrieval Conference (ISMIR)_ , 2014, pp. 367–372.
  * [34] T.-Y. Lin, P. Goyal, R. Girshick, K. He, and P. Dollár, “Focal loss for dense object detection,” in _Proceedings of the IEEE international conference on computer vision_ , 2017, pp. 2980–2988.
  * [35] J. Zhuang, T. Tang, Y. Ding, S. C. Tatikonda, N. Dvornek, X. Papademetris, and J. Duncan, “Adabelief optimizer: Adapting stepsizes by the belief in observed gradients,” _Advances in neural information processing systems_ , vol. 33, pp. 18 795–18 806, 2020.
  * [36] X. Gong, W. Xu, J. Liu, and W. Cheng, “Analysis and correction of maps dataset,” in _the 22nd International Conference on Digital Audio Effects_ , 2019.
  * [37] F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay, “Scikit-learn: Machine learning in Python,” _Journal of Machine Learning Research_ , vol. 12, pp. 2825–2830, 2011.
  * [38] B. C. Ross, “Mutual information between discrete and continuous data sets,” _PloS one_ , vol. 9, no. 2, p. e87357, 2014.
  * [39] K. W. Cheuk, D. Herremans, and L. Su, “Reconvat: A semi-supervised automatic music transcription framework for low-resource real-world data,” in _Proceedings of the 29th ACM International Conference on Multimedia_ , 2021, pp. 3918–3926.
  * [40] K. Choi and K. Cho, “Deep unsupervised drum transcription,” in _20th International Society for Music Information Retrieval Conference, ISMIR 2019_. International Society for Music Information Retrieval, 2019, pp. 183–191.
  * [41] B. Gfeller, C. Frank, D. Roblek, M. Sharifi, M. Tagliasacchi, and M. Velimirović, “Spice: Self-supervised pitch estimation,” _IEEE/ACM Transactions on Audio, Speech, and Language Processing_ , vol. 28, pp. 1118–1128, 2020.

|  Taegyun Kwon is currently working as an Post-doc in the Graduate School of
Culture Technology at the Korea Advanced Institute of Science and Technology
(KAIST) in South Korea. He obtained his Ph.D. and M.S. degrees in culture
technology, and B.S. in Physics from Korea Advanced Institute of Science and
Technology (KAIST). His research primarily focuses on music transcription with
deep learning approach. His research interests also include expressive
performance modeling and human computer interaction in musical performance.
---|---
|  Dasaem Jeong is currently working as an Assistant Professor in the
Department of Art & Technology at Sogang University in South Korea since 2021.
Before joining Sogang University, he worked as a research scientist in T-Brain
X, SK Telecom from 2020 to 2021. He obtained his Ph.D. and M.S. degrees in
culture technology, and B.S. in mechanical engineering from Korea Advanced
Institute of Science and Technology (KAIST). His research primarily focuses on
a diverse range of music information retrieval tasks, including expressive
performance modeling, symbolic music generation, and cross-modal generation.
---|---
|  Juhan Nam is an Associate Professor in the Graduate School of Culture
Technology at the Korea Advanced Institute of Science and Technology (KAIST)
in South Korea. Prior to joining KAIST, he worked as a Staff Research Engineer
at Qualcomm from 2012 to 2014 and as a Software/DSP Engineer at Young Chang
(Kurzweil) from 2001 to 2006. He received his Ph.D. in Music from Stanford
University, where he studied at the Center for Computer Research in Music and
Acoustics (CCRMA). He also holds an M.S. degree in Electrical Engineering from
Stanford University and a B.S. degree in Electrical Engineering from Seoul
National University. His research interests include the application of digital
signal processing and machine learning to music and audio. He is a member of
the IEEE.
---|---