[1]Mickaël Zehren

[1]Department of Computing science, Umeå university, Umeå, Sweden 2]Department
of Music, Universidad EAFIT, Medellín, Colombia

# Analyzing and reducing the synthetic-to-real transfer gap in Music
Information Retrieval: the task of automatic drum transcription

<EMAIL_ADDRESS>Marco Alunno<EMAIL_ADDRESS>Paolo Bientinesi
<EMAIL_ADDRESS>*  [

###### Abstract

Automatic drum transcription is a critical tool in Music Information Retrieval
for extracting and analyzing the rhythm of a music track, but it is limited by
the size of the datasets available for training. A popular method used to
increase the amount of data is by generating them synthetically from music
scores rendered with virtual instruments. This method can produce a virtually
infinite quantity of tracks, but empirical evidence shows that models trained
on previously created synthetic datasets do not transfer well to real tracks.
In this work, besides increasing the amount of data, we identify and evaluate
three more strategies that practitioners can use to improve the realism of the
generated data and, thus, narrow the synthetic-to-real transfer gap. To
explore their efficacy, we used them to build a new synthetic dataset and then
we measured how the performance of a model scales and, specifically, at what
value it will stagnate when increasing the number of training tracks for
different datasets. By doing this, we were able to prove that the
aforementioned strategies contribute to make our dataset the one with the most
realistic data distribution and the lowest synthetic-to-real transfer gap
among the synthetic datasets we evaluated. We conclude by highlighting the
limits of training with infinite data in drum transcription and we show how
they can be overcome.

###### keywords:

Music Information Retrieval, Automatic Drum Transcription, Synthetic-to-real,
Neural Scaling Law

Acknowledgements The computations were enabled by resources provided by the
National Academic Infrastructure for Supercomputing in Sweden (NAISS),
partially funded by the Swedish Research Council through grant agreement no.
2022-06725.

##  1 Introduction

Music information retrieval (MIR) is a research field focused on extracting
features from musical tracks. By retrieving a track’s content, such as its
structure, tempo, or key, we can build software to improve how we study or
listen to music (e.g., providing content-based recommendations or facilitating
music transcriptions for a variety of uses). In this context, several tasks in
MIR are oriented toward the automation of musical, time-demanding operations
that, until recently, we could execute only manually. One of them is automatic
drum transcription (ADT). Specifically, ADT seeks to extract from an audio
track the timing and instrument of the notes played on the drum kit. If this
is already a difficult task because audio recordings are complex and difficult
to analyze, it is even harder to do ADT in the presence of melodic instruments
(DTM) [1] because the drum instruments to be transcribed are obfuscated by
other audio sources.

Due to the difficulty of this challenge, the state-of-the-art methods often
rely on deep learning (DL) models, as they are expressive enough to
disentangle audio mixtures. However, because these models are usually trained
in a supervised manner (i.e., with ground truth labels), their performance is
hindered by the volume of the available datasets used for training. In fact,
labeled datasets are scarce because audio tracks are labor-intensive to
annotate, even for expert musicians, and the process is error-prone. Moreover,
even in the case one makes the effort to create the annotations, tracks are
often copyrighted and cannot be shared. Therefore, to counteract these issues,
synthetic datasets have been generated. In fact, by starting from a target
musical score, such as a MIDI track, the associated audio can be reconstructed
with synthesizers without manual labor and with perfectly accurate labels.
Now, since MIDI tracks are abundant, it is possible to generate a virtually
infinite amount of training data, or, at least, in quantities required by DL
models. However, empirical evidence has shown that these synthetic datasets,
despite their large size, are less effective than non-synthetic datasets in
training models (e.g., [2]).

In fact, the lower standard, diversity, or complexity of synthetic datasets
prevent the trained models from grasping all the nuances of real tracks, which
is what the creators of the synthetic dataset SLAKH acknowledged: “There is no
getting around the fact that these are, in fact, MIDI files. […] it’s
impossible to make the audio not sound cheesy if the MIDI files themselves are
cheesy.”111https://www.slakh.com/ The issue of synthetic-to-real transfer has
been extensively studied outside of ADT and two types of solution emerged to
bridge the gap [3]: 1) Adapting the synthetic datasets so that they are more
realistic or diverse (e.g., through system identification or domain
randomization), and/or 2) adapting the learning algorithm to work on synthetic
data (e.g., via transfer learning). In this work, rather than adapting the
learning algorithm to train on low-quality data, we study how synthetic
datasets can be improved to reduce the transfer gap between them and real
audio datasets.

Specifically, we highlight how the synthetic datasets for drum transcription
are limited by their generation procedure and we propose a new dataset that
does not suffer from the same pitfalls. In fact, three strategies make our
generation procedure different from those used previously: 1) Instead of
following the more common practice of using MIDI files annotated offline, we
increased the realism of the sequences of notes by employing human
performances captured on electronic instruments. 2) Although a previous work
[4] already used human performances to synthesize drum-only audio, we did this
with full tracks containing drums and accompaniment instruments, up to four of
them. 3) Instead of the usual few dozens of synthesizer configurations
(presets), we use hundreds of them to achieve a large diversity of timbres
when rendering the audio. With this generation procedure, we built a new
synthetic dataset which is the only one to our knowledge to feature human
performances, multiple voices and many presets, all in one dataset.

To evaluate the quality of our dataset and identify whether our generation
procedure effectively narrows the gap between synthetic and real tracks more
than what previous methods did, we conducted a set of three investigations.

First, we examined whether the different generation procedures can create
realistic data distributions, that is, we compared the coverage of the real-
world distribution with their synthetic distribution. This allowed us to
discover where synthetic datasets lack in realism, as this lack represents the
gap between synthetic and real, and may be the source of generalization
errors.

Second, we quantified the transfer gap in terms of the training dataset.
Specifically, by measuring how the real-world performance of the model scales
by increasing the amount of training data (scaling laws), we could extrapolate
the minimal loss we would expect to reach with each generation procedure. This
minimal loss will depend on the transfer gap between the training and test
datasets.

Lastly, we performed an ablation study to validate the positive impact of the
three main characteristics of our generation procedure on the transfer gap. To
this end, as we did in the previous experiment, we compared different versions
of our dataset: with and without human performances, with different numbers of
accompaniment instruments, and different numbers of presets.

The remainder of this article is organized as follows. First, we present
related works on the construction of datasets for DTM in Section 2. Then, we
introduce our solution to generate a better synthetic dataset followed by a
comparison of the data distributions resulting from the different generation
procedures, in Section 3. Finally, we describe our experimental design in
Section 4 and we discuss the results of the comparison of the transfer gap
between datasets and ablation study in Section 5. We conclude in Section 6.

##  2 Related works

In many fields of research relying on DL, the amount of training data is the
main limiting factor. This is also true for DTM where the datasets are usually
very small, contain mistakes, or cannot be shared.

To solve this paucity of data, multiple approaches have been studied which we
group into two categories: 1) semi-automatic annotations and 2) synthetic
datasets. The resulting datasets are compared in Table 1.

Table 1: Scalar variables of seven DTM datasets. ADTOS is the dataset introduced in this manuscript. | Synthetic audio |  | Real-world audio
---|---|---|---
Dataset |  TMIDT [5] |  SLAKH [6] |  EGMD [4] |  ADTOS |  |  RBMA [7] |  MDB [8] |  ADTOF-YT [2]
Tracks |  4,228 |  1,710 |  1,059 |  10,250 |  |  27 |  23 |  2915
Duration (h) |  260 |  115 |  $43\times 10$ |  $2\times 250$ |  |  1.6 |  0.4 |  250
Voices |  n.s. |  4-48 |  1 |  4 |  |  - |  - |  -
Drum presets |  57 |  8 |  43 |  512 |  |  - |  - |  -
Other presets |  1 |  179 |  0 |  458 |  |  - |  - |  -

###  2.1 Semi-automatic annotations

While some datasets have been fully annotated by hand, such as RBMA [7], the
general approach is to employ semi-automatic techniques to reduce the manual
labor required to create the labels.

One common technique is to bootstrap the annotations to an intermediate state
and then use human annotators only to refine the labels to the desired level
of detail or precision. For example, both ENST and MDB were created with onset
detection algorithms applied to audio files of isolated instruments (stems).
Then, the annotators verified and labeled the initial set of positions to the
correct instruments [9, 8]. Similarly, in the related field of multi-
instrument transcription, Simon et al. used monophonic recordings, for which
labels can be precisely computed with a pitch detection algorithm, and mixed
them into polyphonic tracks [10]. Since the merged audios come from unrelated
recordings, the data is not musically coherent. However, thanks to its large
diversity, this set made the model generalize better when pre-trained on it.

Other datasets were built with the opposite approach: refining human
annotations with the help of algorithms. This technique is notably required
with crowdsourced datasets where annotations can vary in quality. It is
commonly used to align precisely the annotations to the audio recordings, a
task that can be difficult or tedious for humans when the recording varies in
tempo, as well as to estimate a quality score of the annotations and, thus,
filter out problematic tracks. A2MD was created in this manner, by searching
and aligning MIDI files found in the wild to their corresponding audio
recording on Youtube [11]. The alignment was performed by warping each MIDI
file to minimize its difference, when synthesized, with respect to the
original audio. Then, based on the quality of the alignment, the annotations
were grouped into three levels of fidelity. It is interesting to note that a
dataset of piano recordings such as MusicNetEM was created with a similar
approach to A2MD, with the difference that the annotations were aligned to an
initial algorithmic transcription, rather than to the audio [12]. After
comparing the two kinds of alignments, the authors claimed that the alignment
in the symbolic domain is more robust as it can ignore errors in the original
transcription. Lastly, ADTOF-RGW and ADTOF-YT repurposed annotations meant for
rhythm video games to music transcription by refining the alignment of the
annotated beats to audio cues detected by a beat tracking algorithm [13, 2].
This approach proved to work well, but requires a rough initial alignment
[14], after which the labels are automatically converted from the video game
instruments to their real-world counterpart with a rule-based system. Finally,
to further clean the annotations from human mistakes, a manual check was
performed on those tracks where a preliminary algorithmic transcription
achieved a low score.

###  2.2 Synthetic datasets

Rather than making the annotation process easier, synthetic datasets aim at
scaling the amount of data by automating its generation. Once the generation
process is in place it has the benefit of producing large amounts of data at a
very low cost. Moreover, the generation process can be tuned to control the
data distribution, which can either be used to increase diversity (e.g., to
complement existing real-world datasets by covering underrepresented
occurrences), or to perform controlled experiments (e.g., to perform an
ablation study). Lastly, the generation of original tracks is not affected by
copyright constraints. Despite such advantages, the question of domain
adaptation arises, that is, one could wonder how well a model trained on a
synthetic data distribution will perform in the real world.

The issue of domain adaptation can be tackled in two ways. First, by improving
the data distribution of the synthetically generated dataset to be close to
that of a dataset of real tracks, which is something very difficult to achieve
given the level of complexity of real audio tracks (also, it is not known to
what extent realism, as opposed to diversity from the randomization of the
generation procedure, is necessary [3]). Second, by adapting the algorithm to
effectively learn from synthetic datasets and transfer to the real domain. As
an illustration of these two techniques, we now present different synthetic
datasets.

TMIDT, published in 2018, was generated from 4194 MIDI files (259h) of pop
tracks from a freely available online collection[5]. To ensure diversity of
sounds, each file was synthesized with one of 57 SoundFonts (the equivalent of
synthesizer presets) for the drum kit, but only one general synthesizer for
the non-drum instruments. Moreover, the dataset is available in a normal
version and a balanced version where the MIDI files have been altered to
balance the occurrences of the drum instruments, effectively covering
instruments that are not sufficiently represented in the normal version.
Although this modification has been done musically (i.e., with coherent
permutations of drums and cymbals), the artificially generated patterns did
not help the generalization of the model. Therefore, balancing the dataset did
not have a positive effect when testing on real tracks. To help the model
transfer to real-world audio, multiple training scenarios have been evaluated.
Overall, models trained exclusively on synthetic data can generalize to some
extent to real-world datasets. However, better performances have been achieved
with transfer learning, i.e., by having the model pre-trained on synthetic
data and then refined on real-world data.

SLAKH (redux), published in 2019, was generated from 1709 MIDI files (115h)
from the Lakh MIDI dataset [6, 15]. Compared to TMIDT, fewer (8) synthesizer
presets have been used to render the drum, and more (179) have been used for
the accompaniment. The MIDI files contain a maximum of 48 voices and a minimum
of 4: piano, bass, guitar, and drums. By controlling which voice to include in
the mix [16], Manilow et al. evaluated the impact that the number of
instruments playing has on the transcription and separation tasks. They found
that while the recording becomes more complex, with an increasing number of
voices, the transcription also gets more difficult to achieve for all the
instruments but the drums and the bass.

EGMD, published in 2020, has been synthesized from 1058 human performances
(10h) captured on an electronic drum kit [4, 17]. Thus, compared to the other
datasets which are annotated offline, the timing and dynamics of the notes are
expected to be more human-like. However, no accompanying instruments are
included and only 43 drum synthesizer presets have been used to render each
score (for a total of $43\times 10$h). This resulted in identical sequences
redundantly shared by different tracks, a situation that required data
augmentation to successfully train the models.

Lastly, we mention AAM, published in 2023, which consists of 3000 musical
scores (125h), generated with an automatic procedure based on music theory and
synthesized in a similar way to SLAKH [18]. However, because the aim in the
creation of this dataset was on the fast generation of a large number of
tracks, the authors acknowledge that the drum patterns are very simplistic.
Thus, we did not include it in Table 1.

##  3 Training dataset

The datasets TMIDT, SLAKH, and EGMD have been synthesized with different
procedures and have different limitations. To study the impact of these
limitations on the performance of the trained models and remove them, we
followed a new procedure to generate a synthetic dataset that we named
Automatic Drum Transcription On Synthesizers (ADTOS) and that has the
following characteristics all at once: a large number of MIDI tracks, human
performances with accompaniment instruments, and many synthesizer presets.
This dataset is also presented in Table. 1. Next, we describe the generation
procedure and then we compare the datasets in terms of their data
distributions.

###  3.1 Generation procedure

Similarly to the other synthetic datasets, ADTOS’ tracks are built from a
symbolic representation of notes in MIDI, which is then rendered into audio by
synthesizers. Our procedure works as follows: First, we collected MIDI loops
of performances captured on electronic instruments; then, we assembled the
loops to compose tracks with drums and accompanying instruments; finally, we
synthesized the tracks with virtual instruments and a wide variety of presets.
We now illustrate these three steps in more detail.

To gather the symbolic representations of notes, we used professional MIDI
loops that were expressly generated as building blocks to compose tracks.
These loops are particularly suited for a dataset as they consist of many
short sequences of drums (about 130,000 loops, 250h), piano (about 20,000
loops, 100h), and bass guitar (about 7,000 loops, 30h), are grouped by themes
to build cohesive tracks and cover a wide variety of sections, genres, time
signatures, and tempi. Further, the loops are recorded by professional session
players on electronic instruments. Thus, as opposed to an offline creation
process where the notes are fully quantized, these performances contain slight
variations in the notes’ position and dynamic.

To build full tracks from these MIDI loops, we assembled the MIDI snippets
iteratively, that is, we built a track section by section, in sequence, and a
section is built by layering loops for each voice. This process is repeated
until reaching the desired duration and number of voices. Specifically, to
ensure some degree of coherence in the composition, a track’s section is
initialized with a “master” MIDI loop selected from the same theme as the
previous section. Then, the section is built by layering loops with similar
tempo, length, genre, time signature, and key signature on top of the master.
This process ensures that the different sections of a track are variations of
the same theme, possibly with different characteristics (e.g., a tempo
change), while the loops within a section have uniform characteristics (e.g.,
the same tempo as the master). Loops are selected with replacement (a loop can
be used more than once) based on a distance function that takes into account
both the compatibility of the loop with the desired characteristics and how
many times it has already been selected. This effectively allows us to
generate any number of tracks by evenly sampling the MIDI loops available and
creating different combinations of overlaps when files are reused. With this
method, we generated 10,250 MIDI tracks 222We chose this number of tracks to
build a training set with $2^{13}$ tracks. Each track features drums, piano,
bass guitar, and guitar loops 333Because we do not own loops for the guitar,
we simulated this instrument with loops meant for the piano.. In particular,
drum loops are used twice on average for a total duration of $2\times 250$h.

Finally, to transform the tracks into audio, we followed the approach used to
curate SLAKH. We rendered each instrument with its corresponding synthesizer
and a random preset.444This step is by far the most time-consuming of the
whole generation procedure. Synthesizing the audio for a single instrument is
about ten times faster than real-time when running in parallel on a MacBook
Pro 2019. Because the number of tracks we composed was larger than the number
of presets, each preset was used multiple times. However, we made sure to
reuse it with different presets for the other instruments, thus increasing
diversity in timbres. Finally, before merging all the tracks into a single
audio file, we normalized the loudness of files of isolated instruments with
the EBU R128 algorithm available in Essentia [19, 20]. A total of 512 drum
presets and 458 non-drum presets were used.

###  3.2 Comparing data distributions

By generating audio as close as possible to real tracks, and thus reducing the
synthetic-to-real gap, we were able to curate the largest and most diverse
dataset currently available. However, having a large quantity of information
does not necessarily translate into a better training of the model if this
information is not meaningful.

Consequently, to assess the quality of ADTOS, we compared it with the other
datasets. More specifically, we overlapped and compared their distribution,
because different generation procedures result in different data distributions
and because a source distribution can only train a model that performs well on
a similar (or subsumed) target distribution. Since our synthetic generation
procedure is a cheaper alternative to datasets curated from real-world tracks,
we were also interested in comparing synthetic to real-world distributions in
order to highlight differences that are possible causes of generalization
errors. Both kinds of distributions are respectively plotted on the left and
right side of Figure 1. We now describe and compare the distributions of the
five variables pictured in the rows of Figure 1: tempo, velocity, onset
interval, time signature, and class.

Figure 1: Violin and bar plots representing respectively continuous variables
distributions (tempo, velocity, and onset interval) and discrete variables
distributions (time signature and class), for the synthetic datasets (left
column) and real-world datasets (right column). The distributions are
normalized, so that each plot has the same width.

#### Tempo,

measured in beats per minute (bpm), dictates the speed at which music is
played, with higher tempi indicating smaller note intervals (for the same beat
subdivision). We observe two key differences in the distributions of tempi
among the datasets. First, the lack of representation of some tempi,
especially the lack of fast tracks in all the datasets but ADTOF-YT, suggests
a potential source of generalization error. Second, the tempo of the tracks
follows a normal distribution in the large real-world dataset ADTOF-YT (RBMA
and MDB are too small to confirm this trend for all real-world datasets), but
not in most synthetic datasets. Indeed, only ADTOS is close to a normal
distribution while TMIDT, SLAKH, and EGMD have peaks around some values (e.g.,
90 bpm or 120 bpm). Although it is not clear if those peaks are there to
represent meaningful tempi in specific music genres or are simply due to
limitations in the generation procedure, it is anyway an indication that
increasing the dispersion of tempi in those datasets might improve
generalization.

#### Velocity

indicates how hard the instrument is struck to play a note (dynamic), and
ranges from 0 to 127 in MIDI, with 127 being the loudest. Unfortunately, there
are no accurate velocity annotations on real-world datasets. Still, we can
compare synthetic distributions to identify trends. In fact, all the
distributions are skewed toward the maximal velocity and have a long tail
towards lower values. Despite such a similarity, the long tail is not the same
in every dataset: datasets annotated offline (TMIDT and SLAKH) have peaks
around specific values (e.g., 100), whereas datasets captured on electronic
drum kits (EGMD and ADTOS) are more dispersed. We assume that the peaks are
due to the quantization of dynamic levels in the software used to create the
annotations (e.g., "forte" might correspond to a MIDI velocity of 100).

#### Onset interval

represents the distance, relative to the beat interval, between two
consecutive drum onsets.555To compute this distance, we merged all notes
taking place in a 50ms window, as we assumed they were meant to be played in
unison, and we discarded the values above quarter notes. We can see two trends
in the datasets. First, although there are clear peaks around common beat
subdivisions, some of them are not present in all the datasets. Especially
faster subdivisions (e.g., 16-note triplets or 32nd notes) are missing in
TMIDT and SLAKH, which is possibly due to simplistic MIDI files. Second, a
spread around the common subdivisions is present in captured annotations, both
from real-world distributions (MDB, annotated with an onset detection
algorithm) and synthetic distributions (EGMD and ADTOS, recorded on electronic
drum kits), but not when the annotations are created offline, because in this
way an annotation quantizes the note intervals to the exact beat subdivisions
(as it happens with velocity), whereas the capture will preserve human
deviations. This is a second indication that TMIDT and SLAKH MIDI might be too
simplistic.

#### Time signature

establishes how the beats of a track are organized in bars and how notes in a
bar are grouped. In other words, it tells how rhythm is eventually perceived.
While many time signatures exist, 4/4 is by far the most common in Western
commercial music (e.g., Rock, Metal, EDM), which is the focus of these
datasets. Thus, it is not surprising that 4/4 has the most occurrences in all
datasets. However, it is important to note that ADTOF-YT has a large diversity
of time signatures, most of which are not covered by the other datasets. This
is an indication that the other datasets are more biased than ADTOF-YT.

#### Class

distribution tells what (group of) instruments are more commonly occurring in
each dataset. Since datasets have a different number of classes, to compare
them we mapped their vocabulary down to the largest common vocabulary, which
is the 5-class vocabulary from ADTOF-YT. This effectively grouped some
instruments to the same class (e.g., a “crash cymbal” or a “ride cymbal” are
both mapped to “CY”), as detailed in our previous work [13, p.821]. While some
of those classes are sparse in real-world datasets (i.e., “CY” in RBMA and
“TT” in MDB), the fact that they all appear with high frequency in the
synthetic dataset should facilitate generalization.

##  4 Experimental design

Overall, ADTOS is closer to the real-world distribution compared to the other
synthetic datasets (we further investigate this trend in terms of drum
sequences in Appendix A), which is a good sign that it might have a smaller
synthetic-to-real gap. However, this conclusion is based on the evaluation of
only five features extracted from the drum voice of the MIDI files, which are
not representative of all the characteristics of the audio, for example, they
do not take into account the synthesizer presets or the other instruments.
Furthermore, we do not know to what extent our better coverage effectively
correlates to improvements in the model’s performance.

Therefore, to quantify the transfer gap, we estimated the minimal
generalization error we can achieve when training on data generated with each
procedure. However, since the trained model might benefit from an unknown
amount of data points, and because it is always possible to generate more
synthetic data, we cannot measure the minimal generalization error with
respect to a fixed data size. Instead, we can apply a scaling law, that is, we
can study the scaling of the models’ performance when increasing the quantity
of training data.

Scaling laws are a well-studied phenomenon used to link different properties
of an algorithm to the performance of a model (a survey is available in
Villalobos [21]). The link is used to predict a dependent variable, usually
the performance of the model measured in cross-entropy loss, when tuning
independent variables such as the number of training samples, the number of
parameters in the model, or a computation budget. As an example, by estimating
the parameters of the scaling law from empirical evidence, it is possible to
predict how large the training dataset needs to be to reach a desired accuracy
[22]. After adjusting the number of parameters and the computation budget so
that they are not limiting factors, the loss of the model $L$ achieved by
training on a quantity of data $n$ will form a learning curve that follows a
scaling law [23]:

$L(n)\simeq\alpha n^{-\beta}+\gamma$ (1)

Where $\alpha>0$ is a constant property of the problem, $\beta>0$ is the
scaling exponent that defines the steepness of the learning curve, and
$\gamma\geq 0$ is the lower bound of the loss. Concretely, the learning curve
shows how fast a model improves from adding training data, and at what value
it will plateau.666The learning curve represents the loss of a model trained
until convergence in terms of the training data size. Thus, it is different
from training and validation curves (see [23]).

This lower bound, $\gamma$, depends on two key characteristics of the
algorithm: the irreducible error and the transfer gap. On the one hand, the
loss of the model is bounded by the irreducible errors specific to the test
data, i.e., Bayes errors or errors due to incorrect labels in the ground
truth[23]. On the other hand, the loss of the model is also bounded by the gap
between the training and test datasets [24]. With that in mind, the idea
behind our experiments was to change the training data while keeping the test
data constant. This way, we tempered the impact of the irreducible errors so
that the difference in $\gamma$ is only explained by the transfer gap. We now
describe in more detail the independent, dependent, and control variables we
used to study the scaling laws.

###  4.1 Independent variable

To vary the quantity of training data, we trained a model by increasing the
number of tracks from 1 to 8192. Although the number of tracks is different
from the de facto variable of the number of samples used to study scaling laws
—tracks have different durations and, thus, contain different numbers of
samples— we believe that this gives a better notion of the quantity of
information in the training data. In fact, since a track is uniform in timbre
and has repeated sections, increasing the number of samples while keeping the
same number of tracks will only add redundant information.

In order to evaluate scaling laws of synthetic audio, we trained a model
exclusively on synthetic data, as Fan et al. [25] did. Note that this practice
deviates from the common case where synthetic datasets are employed in
addition to real datasets, for example to cover parts of the data distribution
that are not sufficiently represented (e.g. [24]). In fact, the performance of
the trained model may not depend on the quality of the synthetic dataset, but
on how well that dataset complements real ones. Instead, by training only on
synthetic data we can estimate a more general effectiveness of the generation
procedure. We assume that any improvement in the generation procedure will
also impact models trained with a mix of synthetic and real data.

###  4.2 Dependent variables

To estimate the performance of the model at each training size, we had to
decide 1) the metric used to quantify the performance of the model and 2) the
datasets to use for testing.

The performance of a drum transcription is usually computed with the
F-measure, which is the harmonic mean of precision and recall of the estimated
drum onsets. An estimation is considered correct if it is within a small
window (e.g., 50ms) from the ground truth. Although intuitive, this score is
imprecise and not a direct measure of the performance of the model (e.g., the
F-measure does not capture fine information, like noise in the activation, in
the output of the model) as it depends on the quality of the peak-picking
procedure for the discretization of the model’s activation into onsets.
Therefore, instead of the F-measure, we used the difference between the
model’s activation and the ground truth (the loss), computed with the widely
used and more precise binary cross-entropy metric. Although the loss does not
have an interpretation in absolute terms, a relative reduction in loss means
that the model is improving.

To measure the models’ loss, we use both validation and test datasets. The
validation loss is measured on tracks generated from the same procedure as the
training data (with unseen MIDI and presets) and indicates the “on-domain”
performance of the model. Although the on-domain performance does not express
the synthetic-to-real gap, it is anyway interesting to understand the dynamic
of the system. Particularly, this helps us quantify how much the improvements
in the validation loss transfer to the test loss [26]. Moreover, the
validation loss can tell us about the complexity of the training dataset [24].
The test loss, instead, is measured on a mixture of datasets containing
exclusive real-world audio and indicates the generalization capability of the
model in the real domain. To cover a wide range of music, we tested on four
datasets: ADTOF-RGW [2], ENST [9], MDB [8], and RBMA [7].

###  4.3 Control variables

As the training data grows, different variables have to be controlled, so that
they will not influence the outcome of the experiments. We will talk here
about the size of the model, the number of training steps, and the training
strategy, as these variables were those that had the strongest impact on the
results of the experiments.

The number of parameters of the model has to grow in size to accommodate for
the added information when the training data increases (e.g. [27]), otherwise
the model will saturate and will not be able to learn. However, it is not
known apriori at what ratio the number of parameters has to increase with
respect to the data size, as the ratio depends on the task and the increase of
parameters might not be linearly correlated with the increase in the number of
tracks. Thus, to control the size of the model we tested a range of values
(between $2\times 10^{5}$ and $1.6\times 10^{7}$ parameters) and we kept the
score of the best-performing architecture at each training size. This ensured
that the number of parameters did not represent a limit throughout the
experiments.

Likewise, we tuned the computation budget so that it would not be a limiting
factor. Because different models and datasets require different numbers of
training steps, we used “early stopping” to train models until convergence.
Although it has been shown that training a model to convergence is inefficient
(e.g. [27]), this warrants to find the smallest loss possible for any
combination of parameters and training sizes.

Lastly, we fixed different hyperparameters to secure proper convergence of the
models throughout the experiments. We found that evaluating the validation
loss for early stopping at each complete pass of the training dataset (epoch)
did not work, since we trained on data varying over multiple orders of
magnitude in size and early stopping did not have enough patience to guarantee
the convergence of the model in the low data regime. To solve this issue we
fixed the validation interval to 128 training steps instead of an epoch and we
stopped the training only after no improvement had been noticed for 25
validations in a row. The steps are composed of a batch of $32$ audio
snippets. The audio snippets consist of 4s of audio, taken from random tracks,
which are transformed into a mel-scale spectrogram with a resolution of 100Hz
and 12 bands per octave between 20Hz and 20kHz. Finally, we used Adam
optimizer and a learning rate scheduler with a warm-up phase, with a learning
rate increasing from $0$ to $5\times 10^{-4}$ for $128$ steps and then
decaying by a factor $\times 10$ over $8192$ steps.

##  5 Results

We used the design presented in the previous section to realize two
experiments. First, we quantified the transfer gap on three datasets to
establish a reference for different generation procedures. Then we conducted
an ablation study to evaluate the impact of the different characteristics of
our generation procedure on the transfer gap.

###  5.1 Transfer gap for different generation procedures

Figure 2: Validation and test loss in function of the number of tracks when
training on different datasets. The solid lines represent the learning curves,
fitted in the log space, from equation 1. The dashed lines represent the value
of $\gamma$, the lower bound of the loss. Notice the log-log scale.

In Figure 2, we compared ADTOS with SLAKH and ADTOF-YT, which served as
baselines for the synthetic and the real-world data, respectively. Note that
since ADTOF-YT contains data from the real domain, training on this dataset is
not representative of the synthetic-to-real transfer. In fact, testing on a
new distribution of the same domain is considered easier than testing from the
synthetic to the real domain. Thus, we trained on ADTOF-YT to estimate an
ideal value of the transfer gap, that is, the smallest error we hoped to
achieve with synthetic data.

Training on ADTOF-YT leads to the smallest lower bound of the test loss (red
dashed line in the right plot of Figure 2), thus confirming that this dataset
has the smallest transfer gap. Instead, training on SLAKH leads to the highest
transfer gap (purple dashed line in the right plot), while ADTOS stays in
between (green dashed line in the right plot). This indicates that our
generation procedure yields a dataset that performs better than the other
synthetic dataset, although not as good as a dataset of real audio tracks.

Although training on ADTOF-YT gives the smallest lower bound of the test loss,
it also leads to the highest lower bound of the validation loss (red dashed
line in the left plot). This means that ADTOF-YT has the highest irreducible
errors which, we recall, are due to either errors in the labels or Bayes
errors. This is not surprising considering that the synthetic datasets (green
and purple dashed lines in the left plot) are generated artificially and,
therefore, do not have errors in the labels. Moreover, since the synthetic
datasets are likely less complex than real audio (e.g., there are no audio
effects or vocals), fewer elements obfuscate the onsets in the signal, thereby
reducing the amount of Bayes errors.

We also noticed that the learning curves from training on ADTOF-YT and ADTOS
did not reach the lower bound for the validation loss (red and green solid
lines in the left plot). This means that, in these two cases, the models are
far from the irreducible error and will gain from more training data. However,
this is not the case for SLAKH (purple solid lines in the left plot), most
likely because in this dataset the presence of redundant tracks causes a lack
of diversity that ultimately makes the learning curve plateau after a few
tracks. In the test loss, instead, all the learning curves reached a plateau,
which is an indication that further training data may reduce the validation
loss, but will not transfer to a better performance on the test datasets.

In summary, we deduced that the datasets are not limited by their size, but
only by the diversity they achieve from their generation procedure. Although
our generation procedure is not as good as training on real data, it is better
than the other synthetic procedure we evaluated.

###  5.2 Ablation study

(a) MIDI source

(b) Voices

(c) Drum presets

Figure 3: Learning curves for different versions of ADTOS by modifying: a) the
MIDI source, b) the number of voices, and c) the number of presets.

To validate that the characteristics of our generation procedure are
effectively reducing the transfer gap, we conducted an ablation study by
evaluating different versions of our dataset, as we did in the previous
experiment. We investigated the three main characteristics of our generation
procedure: the MIDI captured from human performances, the accompaniment
instruments, and the number of drum presets. They are presented in Figure 3.

#### Human performance

In order to compare in a fair way the training on MIDI files of human
performances with the training on MIDI annotated offline, it is necessary to
keep everything else constant (e.g., same tracks, same presets). However,
since we had only the human performance version of the tracks at our disposal,
the sole way of keeping everything else constant was to simulate the offline
annotation process with MIDI captured from the drum instruments. To simulate
the offline annotations, we quantized the timing and velocity of the
performances according to the distributions of the offline annotations from
TMIDT and SLAKH presented in Figure 1. Specifically, we quantized both the
notes to the grid of 16th notes, the smallest beat subdivision in the dataset,
and the velocities to the values 127 and 100, the two most common values. The
comparison between the original and quantized version of our dataset is
presented in Figure 3(a).

While the quantized version has a lower validation loss compared to the
original version (red line compared to purple line in the left plot), it also
has a higher test loss (red line compared to purple line in the right plot).
This indicates that, compared to the simulated offline annotation, capturing
human performance to create MIDI increases data complexity and reduces the
transfer gap. Hence, the validation loss increases and the test loss
decreases.

#### Accompaniment instruments

To evaluate the effects of the accompaniment instruments on the transfer gap,
we used the same MIDI tracks to generate four versions of the datasets with an
increasing number of voices. As represented in Figure 3(b), we evaluated the
training of the models with one, two, three and four voices by iteratively
adding piano, bass guitar, and guitar to the drums.

While the validation loss increases with more voices (left plot, going from
purple to red), the test loss decreases (right plot, going from purple to
red). Similarly to what happened in the previous evaluation, the more complex
the training data become by adding new instruments to the audio the lower the
transfer gap. However, while we note a large difference between recordings
containing only drums and drums with piano, there are diminishing returns when
voices are more than two (the curves get closer to each other), to the point
that there is no difference in test loss between three and four voices (the
green and red curves overlap in the right plot). Considering that real-world
audio tracks usually contain many more instruments, this is an indication that
our tracks are too simplistic in the way they overlap voices. Nonetheless, our
semi-coherent overlap of instruments is still better than drum-only
recordings.

#### Number of drum presets

Lastly, to evaluate the benefit of having a large number of presets, we
trained the models on truncated versions of our datasets, as presented in
Figure 3(c). Specifically, since the dataset is synthesized by reusing presets
for different MIDI files, we trained the models by either increasing the
number of presets or the number of times each preset is used. Thus, each
fitted curve represents the scaling achieved, from a fixed number of presets,
by increasing the number of times each preset is used. Note that since each
preset is used between 1 and 20 times, the number of tracks in the dataset
depends on the number of presets.

We noticed that the validation and test losses attained with any number of
presets first decrease and then plateau above the loss of the next number of
presets. In fact, all the curves are going flat above the following curve.
This means that, regardless of the total number of tracks generated, reusing
the presets more than two to four times has diminishing returns. Likewise,
increasing the number of presets for the same number of tracks (i.e., reducing
the number of times each preset is reused) decreases both losses. We conclude
that increasing the number of presets effectively increases the diversity of
the dataset, which helps reduce the transfer gap.

In summary, the three characteristics of our dataset, by either increasing the
data complexity or its diversity, effectively narrow the synthetic-to-real
gap.

##  6 Conclusions

While the use of synthetic datasets in DTM has the potential to solve issues
related to small dataset sizes, annotators’ mistakes, and copyrights, these
datasets have been inefficient at training models able to transfer to real
audio tracks. In fact, by analyzing existing generation procedures, we
identified that synthetic data are possibly limited by their simplistic MIDI
files, the number of presets used, or the lack of non-drum instruments in the
MIDI containing human performances. To overcome those limits, we proposed to
use professional-grade MIDI loops of human performances captured on drums,
piano and bass guitar and more presets than previously experimented to build
complex and diverse tracks with up to four voices. Then, we showed that this
generation procedure has, indeed, a more realistic data distribution on
multiple variables. Further, we confirmed that the transfer gap achieved by
our dataset is better than the other synthetic dataset we evaluated, although
not as good as training on real-world audio. Finally, we validated with an
ablation study that all the advantages of our generation procedure, compared
to the previous ones, are effectively reducing the transfer gap.

With this work, we proved that it is not the amount of data that matters for
datasets in DTM, but its quality, which depends on the generation procedure.
Overall, the more complex and diverse the training data, the smaller the
transfer gap. To pursue this direction, in future works we would like to
assess a finer scaling of the performance of the model (e.g., on different
drum instruments or music genres) to identify points of improvement in the
generation procedure.

##  Appendix A Drum sequences coverage between datasets

(a) Coverage in beats

(b) Coverage in unique beat sequences

Figure 4: Relative frequency at which beats (left) or unique beat sequences
(right) from a target are included in the source. Numbers in parentheses
represent the count of beats or sequences in the datasets.

Due to the repetitive nature of music, the duration of a dataset is not a
precise indication of the quantity of information it contains. Instead, we
argue that the variety of tracks is a better indication since different tracks
have different timbres. However, different tracks might still have duplicated
sequences of notes such as drum patterns typical of a specific music genre.
Thus, in this section, we evaluate the number of unique drum sequences
contained in each dataset, and especially their coverage relative to other
datasets, to estimate the possible generalization of the trained model. The
heatmaps in Figure 4 represent the coverage that a source dataset has on a
target dataset in terms of drum sequences. Because the drum sequences are not
defined in the datasets, we consider the notes played by the drum instruments
in a single beat.777Specifically, the sequences are identified by computing a
”fingerprint” for each beat of the datasets: the notes are mapped down to a
common 5-class vocabulary without velocity information; they are quantized
into 12 evenly-spaced bins to remove humanization; and collisions between
notes are removed (i.e., notes mapped to the same class and quantized to the
same position are not counted). Moreover, because the distribution of beat
sequences has a long tail (i.e., many sequences are played a few times while a
few sequences are played many times), we are interested in both the coverage
in terms of beats and the unique beat sequences between datasets. The reason
of this is because while a source dataset might cover only a small portion of
the unique beat sequences of any target dataset (Figure 4(b)), those beat
sequences may include most of the beats (Figure 4(a)).

We can observe in the figures that ADTOS has the highest coverage of both
beats and unique beat sequences from all the datasets. This is not surprising
considering that it has both the highest number of beats (3.2M) and,
consequently, the highest number of unique beat sequences (204k). Although a
high number of beats does not necessarily indicate diversity, ADTOS has in any
case $\approx 6\%$ of unique beats, and this is despite the fact that it
reuses each drum MIDI loop twice on average. Such a result is higher than in
SLAKH and TMIDT, which do not reuse MIDI, but contain offline annotations and
have only $\approx 3\%$ and $\approx 4\%$ of unique beats, respectively. On
the one hand, the higher diversity of ADTOS largely increases its coverage of
unique sequences compared to the other datasets (e.g., ADTOS covers 28% of the
sequences of ADTOF-YT, while TMIDT covers 13% of them). On the other hand, the
high coverage of sequences does not translate to a much higher coverage of
beats, because of their unbalanced distribution (e.g., ADTOS covers 94% of the
beats of ADTOF-YT, while TMIDT covers 88% of them).

## References

  *   * Wu et al. [2018] Wu, C.-W., Dittmar, C., Southall, C., Vogl, R., Widmer, G., Hockman, J., Muller, M., Lerch, A.: A Review of Automatic Drum Transcription. IEEE/ACM Transactions on Audio, Speech, and Language Processing 26(9), 1457–1483 (2018) https://doi.org/10.1109/TASLP.2018.2830113 . Accessed 2018-09-12
  * Zehren et al. [2023] Zehren, M., Alunno, M., Bientinesi, P.: High-Quality and Reproducible Automatic Drum Transcription from Crowdsourced Data. Signals 4(4), 768–787 (2023) https://doi.org/10.3390/signals4040042 . Accessed 2024-01-09
  * Nikolenko [2019] Nikolenko, S.I.: Synthetic Data for Deep Learning. arXiv. arXiv:1909.11512 [cs] (2019). http://arxiv.org/abs/1909.11512 Accessed 2023-05-16
  * Callender et al. [2020] Callender, L., Hawthorne, C., Engel, J.: Improving Perceptual Quality of Drum Transcription with the Expanded Groove MIDI Dataset. arXiv:2004.00188 [cs] (2020). arXiv: 2004.00188. Accessed 2020-09-21
  * Vogl et al. [2018] Vogl, R., Widmer, G., Knees, P.: Towards multi-instrument drum transcription. In: 21th International Conference on Digital Audio Effects (DAFx-18) (2018). http://arxiv.org/abs/1806.06676 Accessed 2019-01-25
  * Manilow et al. [2019] Manilow, E., Wichern, G., Seetharaman, P., Le Roux, J.: Cutting music source separation some Slakh: A dataset to study the impact of training data quality and quantity. In: IEEE Workshop on Applications of Signal Processing to Audio and Acoustics (WASPAA). IEEE, ??? (2019)
  * Vogl et al. [2017] Vogl, R., Dorfer, M., Widmer, G., Knees, P.: Drum transcription via joint beat and drum modeling using convolutional recurrent neural networks. In: 18th International Society for Music Information Retrieval Conference, Suzhou, China, pp. 150–157 (2017)
  * Southall et al. [2017] Southall, C., Wu, C.-W., Lerch, A., Hockman, J.: MDB drums- An annotated subset of MedleyDB for Automatic Drum Transcription. (Late Breaking/Demos) 18th International Society for Music Information Retrieval Conference (ISMIR), Suzhou, China (2017)
  * Gillet and Richard [2006] Gillet, O., Richard, G.: ENST-Drums: an extensive audio-visual database for drum signals processing. In: 7th International Society for Music Information Retrieval Conference (ISMIR), Victoria, Canada, pp. 156–159 (2006)
  * Simon et al. [2022] Simon, I., Gardner, J., Hawthorne, C., Manilow, E., Engel, J.: Scaling Polyphonic Transcription with Mixtures of Monophonic Transcriptions. In: 23rd International Society for Music Information Retrieval Conference (ISMIR), p. 8 (2022)
  * Wei et al. [2021] Wei, I.-C., Wu, C.-W., Su, L.: Improving Automatic Drum Transcription Using Large-Scale Audio-to-Midi Aligned Data. In: IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 246–250. IEEE, Toronto, ON, Canada (2021). https://doi.org/%****␣main.bbl␣Line␣225␣****10.1109/ICASSP39728.2021.9414409 . https://ieeexplore.ieee.org/document/9414409/ Accessed 2021-11-15
  * Maman and Bermano [2022] Maman, B., Bermano, A.H.: Unaligned supervision for automatic music transcription in the wild. In: Chaudhuri, K., Jegelka, S., Song, L., Szepesvari, C., Niu, G., Sabato, S. (eds.) 39th International Conference on Machine Learning. Proceedings of machine learning research, vol. 162, pp. 14918–14934. PMLR, ??? (2022). https://proceedings.mlr.press/v162/maman22a.html
  * Zehren et al. [2021] Zehren, M., Alunno, M., Bientinesi, P.: ADTOF: A large dataset of non-synthetic music for automatic drum transcription. In: 22nd International Society for Music Information Retrieval Conference (ISMIR), Online, pp. 818–824 (2021)
  * Driedger et al. [2019] Driedger, J., Schreiber, H., Haas, W., Müller, M.: Towards automatically correcting tapped beat annotations for music recordings. In: 20th International Society for Music Information Retrieval Conference (ISMIR), Delft, The Netherlands, pp. 200–207 (2019)
  * Raffel [2016] Raffel, C.: Learning-Based Methods for Comparing Sequences, with Applications to Audio-to-MIDI Alignment and Matching (2016) https://doi.org/10.7916/D8N58MHV . Publisher: Columbia University. Accessed 2022-09-20
  * Manilow et al. [2020] Manilow, E., Seetharaman, P., Pardo, B.: Simultaneous Separation and Transcription of Mixtures with Multiple Polyphonic and Percussive Instruments. In: IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 771–775. IEEE, Barcelona, Spain (2020). https://doi.org/10.1109/ICASSP40776.2020.9054340 . https://ieeexplore.ieee.org/document/9054340/ Accessed 2022-11-11
  * Gillick et al. [2019] Gillick, J., Roberts, A., Engel, J., Eck, D., Bamman, D.: Learning to Groove with Inverse Sequence Transformations, 11 (2019)
  * Ostermann et al. [2023] Ostermann, F., Vatolkin, I., Ebeling, M.: AAM: a dataset of Artificial Audio Multitracks for diverse music information retrieval tasks. EURASIP Journal on Audio, Speech, and Music Processing 2023(1), 13 (2023) https://doi.org/10.1186/s13636-023-00278-7 . Accessed 2023-04-20
  * Bogdanov et al. [2013] Bogdanov, D., Wack, N., Gomez, E., Gulati, S., Herrera, P., Mayor, O., Roma, G., Salamon, J., Zapata, J., Serra, X.: ESSENTIA: AN AUDIO ANALYSIS LIBRARY FOR MUSIC INFORMATION RETRIEVAL. In: 14th International Society for Music Information Retrieval Conference (ISMIR), pp. 493–498 (2013)
  * [20] Loudness Metering: ’EBU Mode’ metering to supplement loudness normalisation in accordance with EBU R 128. Technical report (2011)
  * Villalobos [2023] Villalobos, P.: Scaling laws literature review (2023). https://epochai.org/blog/scaling-laws-literature-review
  * Hestness et al. [2019] Hestness, J., Ardalani, N., Diamos, G.: Beyond Human-Level Accuracy: Computational Challenges in Deep Learning. arXiv. arXiv:1909.01736 [cs] (2019). http://arxiv.org/abs/1909.01736 Accessed 2023-06-14
  * Hestness et al. [2017] Hestness, J., Narang, S., Ardalani, N., Diamos, G., Jun, H., Kianinejad, H., Patwary, M.M.A., Yang, Y., Zhou, Y.: Deep Learning Scaling is Predictable, Empirically. arXiv. arXiv:1712.00409 [cs, stat] (2017). http://arxiv.org/abs/1712.00409 Accessed 2023-06-13
  * Mikami et al. [2023] Mikami, H., Fukumizu, K., Murai, S., Suzuki, S., Kikuchi, Y., Suzuki, T., Maeda, S.-i., Hayashi, K.: A Scaling Law for Syn2real Transfer: How Much Is Your Pre-training Effective? In: Amini, M.-R., Canu, S., Fischer, A., Guns, T., Kralj Novak, P., Tsoumakas, G. (eds.) Machine Learning and Knowledge Discovery in Databases vol. 13715, pp. 477–492. Springer, Cham (2023). https://doi.org/10.1007/978-3-031-26409-2_29 . Series Title: Lecture Notes in Computer Science. https://link.springer.com/10.1007/978-3-031-26409-2_29 Accessed 2024-04-22
  * Fan et al. [2023] Fan, L., Chen, K., Krishnan, D., Katabi, D., Isola, P., Tian, Y.: Scaling Laws of Synthetic Images for Model Training … for Now. arXiv. arXiv:2312.04567 [cs] (2023). http://arxiv.org/abs/2312.04567 Accessed 2024-04-23
  * Abnar et al. [2022] Abnar, S., Dehghani, M., Neyshabur, B., Sedghi, H.: Exploring the limits of large scale pre-training. In: ICLR Spotlight (2022). https://arxiv.org/abs/2110.02095
  * Droppo and Elibol [2021] Droppo, J., Elibol, O.: Scaling Laws for Acoustic Models. arXiv. arXiv:2106.09488 [cs, eess] (2021). http://arxiv.org/abs/2106.09488 Accessed 2024-04-19