# Practical and Reproducible Symbolic Music Generation by Large Language
Models with Structural Embeddings

###### Abstract

Music generation introduces challenging complexities to large language models.
Symbolic structures of music often include vertical harmonization as well as
horizontal counterpoint, urging various adaptations and enhancements for
large-scale Transformers. However, existing works share three major drawbacks:
1) their tokenization requires domain-specific annotations, such as bars and
beats, that are typically missing in raw MIDI data; 2) the pure impact of
enhancing token embedding methods is hardly examined without domain-specific
annotations; and 3) existing works to overcome the aforementioned drawbacks,
such as MuseNet, lack reproducibility. To tackle such limitations, we develop
a MIDI-based music generation framework inspired by MuseNet, empirically
studying two structural embeddings that do not rely on domain-specific
annotations. We provide various metrics and insights that can guide suitable
encoding to deploy. We also verify that multiple embedding configurations can
selectively boost certain musical aspects. By providing open-source
implementations via HuggingFace111https://github.com/anonymous-
submission-351532/examuse-transformers, our findings shed light on leveraging
large language models toward practical and reproducible music generation.

##  1 Introduction

Symbolic music generation is a rapidly growing field in machine learning, as
reflected by the continuously increasing number of musical datasets that
provide a solid foundation for training large music generation models[1, 2,
3]. While the sequential aspect of music naturally led to the application of
the Transformer architecture[4] to the musical domain, the vertical (e.g.,
multiple notes can be played simultaneously) and temporal (e.g., every note
played has a certain duration) structures in musical data call for a careful
tokenization procedure that is distinct from the typical positional encoding
used in text generation[5].

(a) POP909 (b) GiantMIDI

Figure 1: Example of beat detection results using the famous Madmom
algorithm[6] on (a) POP909 and (b) GiantMIDI. The blue and red dotted lines
indicate ground truth and predicted beats, respectively. Figure 2:
Illustration of the reconstructed structural embeddings. Top: An example of a
piano roll where 5 notes appear in different pitches and timings. Bottom: The
corresponding tokens and structural embeddings.

While there exist various tokenization techniques such as REMI[7] and CP[8],
the MIDI-like tokenization[9] is considered most compatible with large-scale
musical data from cloud-based platforms (e.g. MIDI transcription from YouTube
audio). This MIDI data may not include detailed annotations of the temporal
division of bars. While these annotations can be generated manually[10, 11],
this process comes at a high cost. Another approach is to use heuristic
algorithms[7, 12], but these methods can only be applied reliably when the
input music progresses at a constant speed or tempo, with well-separated bars
and beats[13]. Unfortunately, this is not the case for the majority of MIDI
data directly transcribed from live music audio (see Figure 1), due to
performers frequently adding note on- or off-sets that deviate from the
scripted tempo[14, 15].

Nonetheless, enhancing the vanilla MIDI tokenization to effectively encode
musical structure without domain-specific annotations is not yet studied in
depth. One exemplar work is MuseNet, which introduced additional structural
embeddings to better derive the structural context of music from MIDI-based
tokens, demonstrating state-of-the-art performance in prompt-based music
generation[16]. However, the reasoning behind its design choice of structural
embeddings is still unclear. There is no clear experimentation to validate
their effects separately from the overall capacity of the model architecture.
Furthermore, MuseNet has not yet published any open-source implementations to
support further tuning and testing. As a result, it is uncertain whether
incorporating these supplementary embeddings is truly practical and optimal in
capturing structural contexts of raw MIDI music, which hinders potential
applications with data that lacks domain annotations.

In response to such limitations, we focus on the structural embeddings
proposed by MuseNet and deeply investigate their effect on encoding musical
structures based on raw MIDI data. Specifically, we train a vanilla GPT-2
model[17] equipped with our implementation of the structural embeddings from
MuseNet. We build these structural embeddings upon practical methods
considering the noisiness of raw MIDI data. We also perform ablation studies
by adjusting different initialization methods to study their impact on
generation performance. With this paper, we aim to provide practitioners with
insights into how to effectively encode structural information in music
without manual annotations, as well as an accessible pipeline to train models
on large music datasets. Our main contributions are as follows: 1) We focus on
the effect of structural embeddings for learning musical structure from the
raw MIDI data without domain-specific annotations; 2) We implement the
supplementary structural embeddings suggested by MuseNet using practical
methods that fit the raw MIDI data. For public use, we open-source our code
and a demo page; 3) We empirically analyze and clearly validate the impact of
different initializations of structural embeddings via substantial evaluation.

##  2 Method

Figure 3: Illustration of our music generation framework. Left: During
training, four types of structural embeddings are concatenated to the input
tokens, then projected to the input embedding size for next-token prediction.
Right: During inference, structural information inferred via rule-based
modules is used for autoregressive generation.

To encode musical data as a sequence of tokens, we use the event-based
tokenization method of MuseNet[16, 9]. We discard any meta-information and use
only 4,196 different input tokens: 4,096 note-on tokens resulting from 32
quantized velocities and 128 pitch numbers (e.g. <v64:C4>), 128 note-off
tokens with zero velocity (e.g. <v0:C4>), and 100 time-shift tokens
representing 100 divisions from 10 ms to 1000 ms (e.g. <wait:2>). We also use
special <BOS>, <EOS>, and <PAD> tokens as we need to batch pieces with varying
durations into a fixed length.

###  2.1 Structural Embeddings

We revisit the original structural embeddings of MuseNet [16] and reconstruct
them to capture meaningful structures in raw MIDI data. We implement the four
structural embedding layers, which are named as part, type, time, and pitch-
class (PC), modifying the MuseNet version towards better fitness to audio-
based MIDI data. Among the four embeddings, the part embedding is the only one
that is identically reconstructed from MuseNet, while others are modified or
re-interpreted in our application. As demonstrated in Figure 2, four
additional embedding layers construct the structural embeddings which are
concatenated with the token embeddings, projected to the hidden size of the
model, and added to the positional embedding. We expect from this pipeline
that each structural embedding on each token is considered equally important
with the token embedding. Below we provide a detailed description of each type
of structural embedding:

Part. The part embedding informs which temporal part among the entire song
each token is located at. We divide the total time length of a song equally by
128. For each event token, we compute a part class as one of the 128 temporal
parts of the sequence. As a result, each token is indexed starting from 0 to
128, where 0 indicates the non-note tokens (e.g. <BOS>, <EOS>) and 1 indicates
the actual starting part of the song.

Type. The type embedding indicates the type of each token, whether the token
is related to note-on(0), note-off(1), time(2), and other types(3) of event.
This can help the model distinguish different token types more easily and
learn dependencies among tokens within the same token type. Note that without
the type embedding, all tokens are learned independently of each other
regardless of their inherent type.

Time. The time embedding informs the model time difference in 10ms between
each non-time-shift token and its previous non-time-shift token. It is similar
to Musenet, yet we re-interpreted the original to represent the relative time
between the tokens. We directly allocate this information to the note-on or
-off tokens which follow each time-shift token to explicitly encode the
temporal concurrency or progression of notes. If multiple time-shift tokens
are interjected, we only count the closest time-shift token. It is to bound
the class size as the time differences may be unbounded. We expect the
original event tokens to give extra information. Once a time token appears in
a sequence, the corresponding time-shift value (i.e. 2 if <time_2>) is
assigned as a time class to the following non-time-shift tokens. The time-
shift tokens are assigned 0.

Pitch-Class (PC). The pitch-class embedding informs the pitch-class of each
note-on or -off token. At every note-on and note-off token, the corresponding
pitch-class is directly applied as a PC class, while other tokens are assigned
0. MuseNet originally proposed to use the relative position of each note
within the corresponding chord [16]. However, it is tricky to determine the
boundary of a chord due to the notes sequentially intermeshed or sustained by
the piano pedals as shown in Figure 1(b). Therefore, we instead utilize the
pitch-class of each note-related token that can explicitly represent the tonal
color of each token. We conjecture that incorporating pitch-class embedding
allows the model to stack tonal information via self-attention and thus
immediately capture the harmonic context of the input.

###  2.2 Training and Inference

As our backbone architecture, we use GPT-2[17] following MuseNet[16]. However,
we do not utilize sparse attention[18] and mixup[19] to focus on the impact of
adjusting structural embeddings for musical data without any trade-offs in
computational efficiency or regularization. The overall procedure for training
and inference using our model is illustrated in Figure 3. We concatenate the
four structural embeddings with input token embeddings along the feature axis
and project the resulting embeddings into the model’s hidden dimension using a
fully-connected layer. We also add a trainable, randomly initialized sequence
positional encoding to the resulting embeddings.

For training, we use a next-token prediction task, analogous to language
modeling[17]. Let $X=[x_{1},…,x_{N}]$ represent a sequence of input tokens,
where $N$ denotes the sequence’s length. At each step, the model learns to
predict $x_{i}$ from $x_{<i}$ by optimizing
$\mathcal{L}=-\sum{\log{p(x_{n}|x_{<i})}}$. During inference, the model
autoregressively generates the sequence given a prompt of MIDI tokens. After
generating each token, we extract its four structural labels—part, type, time,
and PC—from all subsequent tokens using rule-based modules. These labels are
then fed into the model for the next step in generation.

###  2.3 Initialization Methods

In this paper, we examine the effect of two different initialization methods
of structural embeddings. The first method is to use truncated normal
initialization[20]. The second method involves sinusoidal initialization,
applied only to the time-related embeddings. This approach, inspired by Guo et
al[21], employs sinusoidal encoding to preserve the continuity of ordinal
musical attributes, such as note onset and pitch. We similarly use sinusoidal
lookup tables for temporal embeddings, part and time, as follows:

$\displaystyle SE_{(k,2i)}$ $\displaystyle=\sin(k/(10000/w)^{2i/d})$ (1)
$\displaystyle SE_{(k,2i+1)}$ $\displaystyle=\cos(k/(10000/w)^{2i/d})$

where $k$ is a class index for part or time, $i$ is a feature index, $d$ is
the hidden size, and $w$ is a scaling factor. By using two different values
for $w$, we ensure that the corresponding embeddings are orthogonal, allowing
the part and time attributes to be represented without interference[21].
Considering that Part is composed of a much larger unit than Time which is in
10ms, we manually set $w=10$ and $w=1$, respectively. We also note that the
average length of each part is around 432 ms (the average song length is
$55.39\pm 25.23$ seconds). By testing different initializations, we expect to
examine whether taking temporal continuity into account affects generative
performance under the nature of musical attributes.

Metric | Prompt |  $\mathcal{SI}_{3}^{8}$ |  $\mathcal{SI}_{8}^{15}$ |  $\mathcal{SI}_{15}^{N}$ |  $\mathcal{CPVR}_{2}$ |  $\mathcal{CPVR}_{3}$ |  $\mathcal{CPVR}_{4}$ |  $\mathcal{CPI}_{2}$ |  $\mathcal{CPI}_{3}$ |  $\mathcal{CPI}_{4}$
---|---|---|---|---|---|---|---|---|---|---
GPT2 | 16 |  0.2025 |  0.1801 |  0.1653 |  0.4628 |  0.5011 |  0.4766 |  0.3910 |  0.5367 |  0.6331
GPT2-RE | 16 |  0.1908 |  0.1707 |  0.1585 |  0.4525 |  0.4880 |  0.4630 |  0.4064 |  0.5578 |  0.6570
GPT2-SE | 16 |  0.2169 |  0.1948 |  0.1816 |  0.5216 |  0.5682 |  0.5460 |  0.3162 |  0.4389 |  0.5222
GPT2 | 64 |  0.1985 |  0.1767 |  0.1613 |  0.4576 |  0.4960 |  0.4703 |  0.4034 |  0.5500 |  0.6463
GPT2-RE | 64 |  0.2036 |  0.1832 |  0.1691 |  0.4915 |  0.5318 |  0.5087 |  0.3672 |  0.5057 |  0.5979
GPT2-SE | 64 |  0.2090 |  0.1905 |  0.1774 |  0.5092 |  0.5540 |  0.5305 |  0.3372 |  0.4644 |  0.5502
GPT2 | 256 |  0.1874 |  0.1689 |  0.1555 |  0.4460 |  0.4808 |  0.4544 |  0.4320 |  0.5844 |  0.6834
GPT2-RE | 256 |  0.1941 |  0.1763 |  0.1625 |  0.4755 |  0.5126 |  0.4882 |  0.3977 |  0.5426 |  0.6373
GPT2-SE | 256 |  0.2015 |  0.1816 |  0.1686 |  0.4905 |  0.5321 |  0.5079 |  0.3708 |  0.5064 |  0.5960

Table 1: Results of the objective evaluation scores on the three metrics:
Structureness Indicators (SI), Chord Progression Variation Rationality (CPVR),
and Chord Progression Irregularity (CPI).

(a) SI vs. CPVR

(b) SI vs. CPI

(c) CPVR vs. CPI

Figure 4: Correlation results among SI, CPVR, and CPI computed from the three
metric scores of all models. Pearson’s correlation coefficients are also
reported on the top of each plot.

##  3 Experiments

###  3.1 Experimental Setup

Datasets. For training, we use two datasets comprising 1,748 pieces from
Pop1k7[8] and 5,842 pieces from GiantMIDI-Piano[22]. Both datasets consist of
transcribed MIDI files from large-scale piano performances. For computational
efficiency, we exclude pieces from GiantMIDI-Piano that are longer than 10
minutes. We also apply 7 pitch transpositions up to $\pm 3$ half steps and 5
time stretches up to $\pm 0.05$ [9], consequently using a total of 17,383
hours of transcribed MIDI data. A small portion ($0.01\%$) of the training
dataset is used for validation. For evaluation, we employ a total of 1,276
songs from the MAESTRO dataset[23]. As several state-of-the-art audio
transcription models are trained on MAESTRO[24, 25], it can be considered a
high-quality ground truth testbed for the raw MIDI data.

Implementation. We train the GPT-2 architecture containing 12 hidden layers
implemented by Huggingface[20]. We use the AdamW optimizer with a learning
rate initially set at 1e-4, which is linearly reduced to 0 after 1,000 warm-up
steps. We set the maximum token length as 1024 and run a total of 144,786
training iterations or 6 epochs. We infer all test samples using token prompts
from MAESTRO, sampling each prompt five times to reduce contingency in random
generations. From each song of MAESTRO, we take the beginning of a song as a
prompt and examine the prompt token lengths of $2^{l}$ with $l\in\\{4,6,8\\}$
(The lengths in time are about 2, 5, 15 seconds, respectively). We use top-$k$
sampling with $k=32$ until the sample reaches 1 minute maximum.

Compared Methods. To study the effect of structural embeddings, we train and
test three variants of GPT-2. GPT2 is the vanilla language model without any
structural embeddings. GPT2-RE is GPT2 equipped with four structural
embeddings, all initialized randomly. GPT2-SE is the same as GPT2-RE, but with
part and time embeddings initialized using sinusoidal weights instead.

Objective Evaluation. We use three metrics to evaluate the influence of the
structural embeddings on both temporal and harmonic structures of the
generated music. Structureness Indicator (SI) measures the largest degree of
repeatedness among various intervals in generated music. We exploit this
metric to evaluate how much a generated sequence reflects the repeating nature
of music according to the structural embeddings[10]. It is computed by a
fitness score based on a similarity matrix[26, 10]. We convert the generated
samples to audio, set the sampling rate at 1 Hz, and compute the score in the
unit of seconds on the repeated duration intervals in $3\sim 8$, $8\sim 15$,
and $15\sim N$ seconds, where $N$ is the total length of a song [10]. We also
use two existing metrics to measure the harmonic structure of the generated
sequence. Chord Progression Variation Rationality (CPVR) measures the
rationality of chord changes within the generated harmonic context[12]. It is
computed as the conditional probability of the existence of each unique chord
$n$-gram given its previous $n$-gram within the generated music. We consider
each chord $n$-gram unique if at least one chord element is different from
other $n$-grams. Chord Progression Irregularity (CPI) measures the ratio of
unique chord $n$-grams from generated music, compensating CPVR that yields
high scores with frequently occurring chords [10]. These two chord-related
metrics are examined when $n\in\\{2,3,4\\}$. We utilize the existing rule-
based algorithm [7] to label chords every 500 milliseconds or 1 second
throughout the song. We do not use beat-based intervals which cannot be
determined without beat or bar annotations. For more details on implementing
these metrics, please refer to Appendix B.

Subjective Evaluation. We conduct the common A-vs-B human-rating procedure[9,
27, 2]. We randomly select 12 prompts of length 64 from MAESTRO and generate
the samples in a maximum of 30 seconds. For each prompt, samples from the four
different models including the ground truth from MAESTRO are alternatively
compared under the A/B test procedure. Four raters score each sample pair
based on how natural each sample sounds (Naturalness) and how well each sample
preserves the characteristics of the given prompt (Prompt Maintenance). The
measurement of prompt maintenance aims to validate structural fidelity in
music. Structural fidelity has been defined by how a given phrase or theme is
repeated and developed throughout the composition [10, 12, 28]. Consequently,
each rater listens to a total of 72 pairs where 6 model pairs are compared for
12 samples.

###  3.2 Objective Results

Repeatedness of Musical Patterns. Table 1 shows that GPT2-SE outperforms other
methods in Structureness Indicators (SI) for all interval lengths and prompt
lengths. It demonstrates that sinusoidal initialization may reliably generate
repeated patterns in various lengths, regardless of prompt length. GPT2-RE
shows better performance than GPT2 with prompt lengths longer than 16. It is
unstable compared to other models showing large degradation with prompts at
length 16. It indicates that GPT2-RE may be more beneficial than GPT2 in
generating repeated music only from the prompts longer than 5 seconds. In
contrast, removing supplementary embeddings may be better than preserving them
for creating regular patterns, only from prompt lengths shorter than 5
seconds.

Rationality and Irregularity of Chord Progression. Table 1 shows that GPT2-SE
receives the highest scores in Chord Progression Variation Rationality (CPVR)
among all settings, but scores the lowest in Chord Progression Irregularity
(CPI). It indicates that sinusoidal initialization can induce more common
chords, which can lead to low diversity of the overall harmonic context
compared to the other models. This compromise is further substantiated by the
data in Figure 4, which shows a robust inverse correlation between CPI and
CPVR. Moreover, high SI values may be associated with high CPVR when certain
harmonic contexts are repeated. GPT2-RE shows the worst CPVR scores in prompt
lengths 16, while CPI scores are shown the best (Figure 4). For longer
prompts, GPT2-RE gets better CPVR scores than GPT2 while showing worse CPI
scores. GPT2 reveals the best CPI scores with prompt lengths 64 and 256. It
demonstrates that randomly initialized structural embeddings may benefit in
making rational chords over the vanilla GPT-2 with prompts longer than 5
seconds while creating more unique chords with shorter prompts.

PNSR($s$) | $s<0.25$ | $0.25\leq s<0.5$ | $0.5\leq s<0.75$ | $0.75\leq s$
---|---|---|---|---
Prompt-64 | 59 | 191 | 433 | 593
Prompt-16 | 329 | 397 | 274 | 276

Table 2: Number of prompts in different PNSR ranges. Figure 5: PNSR vs. SI,
CPVR, and CPI plots with prompt lengths 64 (solid) and 16 (dashed) from the
three methods.

Periods of Silence in Prompts. We closely investigate the reason for the large
degradation in performance of GPT2-RE at the prompt length 16. We hypothesize
that prompts of this length become distinguished in some aspect from the
longer prompts, and GPT2-RE is particularly sensitive to this corresponding
aspect. To verify this assumption, we compute the ratio of non-silence periods
within the prompts (shortened as PNSR) with lengths 16 and 64. We convert each
prompt to a $P\times T$ 2-dimensional piano roll, where $P$ is the number of
MIDI pitches, and $T$ is the number of time frames. Then, we obtain PNSR as
the number of non-zero columns divided by the length of the piano roll. Table
2 shows the distribution of the prompts across 4 ranges of the PNSR. The
number of prompts with low PNSR under 0.5 increases when its length is 16,
compared to 64. Figure 5 demonstrates relationships between PSNR and the three
metric scores of all models averaged across the sub-metrics. All metric scores
of GPT2-RE abruptly decline as the PSNR decreases under 0.5 when the prompt
length is 16, in contrast to other models revealing relatively minor changes
across the prompt length. It signifies that GPT2-RE may be more responsive
than the other models to the amount of silence in the input prompt, which
depends on the prompt lengths.

(a) GPT2

(b) GPT2-RE

(c) GPT2-SE

(d) GT

Figure 6: Compiled fitness scape plots across 12 samples from the four models:
Each plot displays the maximum fitness scores among all stacked scores from 12
samples for each entry within the plot. Each plot illustrates the segment
center in frames on the X-axis and the segment duration in frames on the
Y-axis. Every black dotted line marks the average end time (3.77 seconds)
across the prompts. Additionally, average scores for Structureness Indicators
and Prompt Maintenance across the 12 samples are provided at the top of each
plot.

Method | GPT2 | GPT2-RE | GPT2-SE | GT
---|---|---|---|---
Naturalness | 30 | 63 | 37 | 109
Prompt Maintenance | 31 | 57 | 46 | 92

Table 3: Win counts from the subjective evaluation.

Metric | Naturalness
---|---
Method Pair | Wins | Ties | Losses |  $p$-value
GPT2-RE vs. GPT2-SE | 24 | 14 | 10 | 0.0033
GPT2-RE vs. GPT2 | 31 | 10 | 7 | 0.0004
GPT2-RE vs. GT | 8 | 5 | 35 | 0.0011
GPT2-SE vs. GPT2 | 21 | 8 | 19 | 0.3692
GPT2-SE vs. GT | 6 | 7 | 35 | 5.11e-6
GPT2 vs. GT | 4 | 5 | 39 | 1.93e-6
Metric | Prompt Maintenance
Method Pair | Wins | Ties | Losses |  $p$-value
GPT2-RE vs. GPT2-SE | 22 | 11 | 15 | 0.1939
GPT2-RE vs. GPT2 | 24 | 15 | 9 | 0.0115
GPT2-RE vs. GT | 11 | 8 | 29 | 0.0025
GPT2-SE vs. GPT2 | 19 | 13 | 16 | 0.5246
GPT2-SE vs. GT | 12 | 6 | 30 | 0.0064
GPT2 vs. GT | 6 | 9 | 33 | 1.38e-5

Table 4: Pairwise comparison results of the subjective evaluation scores on
Naturalness and Prompt Maintenance.

###  3.3 Subjective Results

Table 3 indicates that GPT2-RE achieves the highest number of wins among the
models, while GPT2 records the lowest. Table 4 presents pairwise comparisons
among the models regarding Naturalness and Prompt Maintenance, including
$p$-values from the Wilcoxon signed-rank test based on the scores collected by
the raters [9, 27]. In Naturalness, all model pairs except for GPT2-SE and
GPT2 demonstrate significant differences ($p<.005$). Concretely, GPT2-RE
significantly beats GPT2, whereas GPT2-SE does not show a significant
difference from GPT2. In Prompt Maintenance, GPT2-RE also significantly
outperforms GPT2 ($p<.05$), while GPT2-SE does not. These results suggest that
incorporating randomly initialized structural embeddings significantly
improves the ability of the basic GPT-2 to create plausible and prompt-
consistent music, whereas sinusoidal initialization shows no meaningful
difference.

These results can give us insight that the recurrence of specific passages,
which can measured by SI, may not simply exhibit a correlation with the sense
of prompt maintenance. Alternatively, scores for Prompt Maintenance are
demonstrated to be associated with those of Naturalness, as evidenced by an
examination of the Pearson’s correlation coefficient between the scores of
these two metrics across the models ($r=0.6559$). Moreover, it may imply that
the least trade-off between CPVR and CPI scores shown in GPT-RE can be
beneficial in reaching higher naturalness and prompt maintenance compared to
the other two models revealing extreme trade-offs. In the following section,
we investigate how GPT2-RE can be superior in listeners’ evaluation to the
other models in terms of prompt maintenance.

###  3.4 Fitness Scape Plots

We conduct a detailed investigation of the fitness scape plot that spans the
entire song [29]. The fitness scape plots are derived from all 12 samples of
each model. We set each frame at 500 ms to examine the plots at a higher
resolution. Then, we compile the plots to calculate the maximum fitness scores
from all stacked scores for each entry within the plot. The X-axis and Y-axis
of the plot denote the center and the length of the repeated segments within
the song, termed ’segment center’ and ’segment duration,’ respectively. We use
the maximum scores rather than the average to clearly observe the model’s peak
performance in creating structural patterns across various segment durations.

Figure 6 displays the resulting fitness scape plots of the three models and
GT. The fitness scape plot for GT is distinctive compared to those of the
other models. It shows clearly defined regular patterns, particularly darker
at the side edges of the triangle. Furthermore, the scores are relatively low
for short segment durations compared to those of other models. This suggests
that GT may feature musical patterns in longer durations, developed from the
given prompts, leading to the highest Prompt Maintenance (PM) scores. While
GPT2-RE and GPT2-SE seem similar, GPT2-RE reveals higher fitness scores at the
upper side of the plot than GPT2-SE and lower fitness scores under the plot.
GPT2-RE also has lower fitness scores for the shorter segments close to the
prompt (at the lower left corner), similar to GT. Conversely, GPT2-SE
demonstrates fitness scores that are broadly distributed below the plot. It
indicates that GPT2-RE may be more beneficial than GPT2-SE to generate longer
patterns derived from the prompt leading to higher PM scores. GPT2 shows a
score pattern that completely neglects the given prompt: the fitness scores
are concentrated in the last half of the song, yielding highly irregular
patterns compared to other models. This aspect clarifies why GPT2 obtains the
worst PM score. We also note that the aligned Structureness Indicators (SI)
scores are not highly correlated to PM scores ($r=-0.1429$). GPT2 and GPT2-SE
show higher average scores for SI than GT, unlike for PM. It may imply that
repetition of the musical patterns cannot solely represent how well the given
prompt is retained in the music. This leads to an intuition suggesting the
necessity for additional structural metrics that can grasp the high-level
fidelity in the musical structure, beyond the simple repetition of musical
patterns.

##  4 Conclusion

We have reconstructed practical structural embeddings for raw MIDI data, which
lacks domain-specific annotations. We have verified that adding structural
embeddings enhances the model’s performance in capturing the structural
aspects of music compared to the model without them. It is encouraging that
structural embeddings can be easily integrated with existing language models
such as GPT-2. Under various embedding setups, we find that: 1) random
initialization generates more plausible music for human listeners but lacks
stability; 2) sinusoidal initialization of temporal embeddings helps produce
repeated patterns and common chords in music. These distinct strengths of the
two settings may enable users to enhance specific aspects of music in certain
ranges when creating their products. Our demonstration222https://free-
pig-6c6.notion.site/ISMIR2024-Demo-Page-7d640a34529646ee91acc6c41ef1ec27?pvs=4
showcases the corresponding examples.

## References

  * [1] J. F. Gemmeke, D. P. W. Ellis, D. Freedman, A. Jansen, W. Lawrence, R. C. Moore, M. Plakal, and M. Ritter, “AudioSet: An ontology and human-labeled dataset for audio events,” in _IEEE international conference on acoustics, speech and signal processing (ICASSP)_. IEEE, 2017, pp. 776–780.
  * [2] A. Agostinelli, T. I. Denk, Z. Borsos, J. Engel, M. Verzetti, A. Caillon, Q. Huang, A. Jansen, A. Roberts, M. Tagliasacchi, M. Sharifi, N. Zeghidour, and C. Frank, “MusicLM: Generating music from text,” _arXiv preprint arXiv:2301.11325_ , 2023.
  * [3] I. Simon, C.-Z. A. Huang, J. Engel, C. Hawthorne, and M. Dinculescu, “Generating piano music with transformer,” _Google AI magenta Blog_ , 2019. [Online]. Available: https://magenta.tensorflow.org/piano-transformer
  * [4] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, “Attention is all you need,” in _Proceedings of the 31st Conference on Neural Information Processing Systems (NIPS)_ , 2017.
  * [5] Z. Wang and G. Xia, “MuseBERT: Pre-training music representation for music understanding and controllable generation,” in _Proceedings of the 22nd International Society for Music Information Retrieval Conference (ISMIR)_ , 2021.
  * [6] S. Böck, F. Korzeniowski, J. Schlüter, F. Krebs, and G. Widmer, “Madmom: A new python audio and music signal processing library,” in _Proceedings of the 24th ACM International Conference on Multimedia_ , 2016.
  * [7] Y.-S. Huang and Y.-H. Yang, “Pop music transformer: Beat-based modeling and generation of expressive pop piano compositions,” in _Proceedings of the 28th ACM International Conference on Multimedia_ , 2020.
  * [8] W.-Y. Hsiao, J.-Y. Liu, Y.-C. Yeh, and Y.-H. Yang, “Compound word transformer: Learning to compose full-song music over dynamic directed hypergraphs,” in _Proceedings of the AAAI Conference on Artificial Intelligence_ , 2021.
  * [9] C.-Z. A. Huang, A. Vaswani, J. Uszkoreit, N. Shazeer, I. Simon, C. Hawthorne, A. M. Dai, M. D. Hoffman, M. Dinculescu, and D. Eck, “Music transformer,” _arXiv preprint arXiv:1809.04281_ , 2018.
  * [10] S.-L. Wu and Y.-H. Yang, “The jazz transformer on the front line: Exploring the shortcomings of ai-composed music through quantitative measures,” _arXiv preprint arXiv:2008.01307_ , 2020.
  * [11] F. Foscarin, A. Mcleod, P. Rigaux, F. Jacquemard, and M. Sakai, “ASAP: A dataset of aligned scores and performances for piano transcription,” in _Proceedings of the 21st International Society for Music Information Retrieval Conference (ISMIR)_ , 2020.
  * [12] X. Zhang, J. Zhang, Y. Qiu, L. Wang, and J. Zhou, “Structure-enhanced pop music generation via harmony-aware learning,” in _Proceedings of the 30th ACM International Conference on Multimedia_ , 2022.
  * [13] Z. Wang, K. Chen, J. Jiang, Y. Zhang, M. Xu, S. Dai, X. Gu, and G. Xia, “POP909: A pop-song dataset for music arrangement generation,” _arXiv preprint arXiv:2008.07142_ , 2020.
  * [14] T. H. Kim, S. Fukayama, T. Nishimoto, and S. Sagayama, “Statistical approach to automatic expressive rendition of polyphonic piano music,” _Guide to Computing for Expressive Music Performance_ , pp. 145–179, 2013.
  * [15] P. Grosche, M. Müller, and C. Sapp, “What makes beat tracking difficult? A case study on Chopin Mazurkas,” in _Proceedings of the 11th International Society for Music Information Retrieval Conference (ISMIR)_ , 2010.
  * [16] C. Payne, “MuseNet,” _OpenAI Blog_ , 2019. [Online]. Available: https://openai.com/blog/musenet/
  * [17] A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, and I. Sutskever, “Language models are unsupervised multitask learners,” _OpenAI Blog_ , vol. 1, no. 8, 2019.
  * [18] R. Child, S. Gray, A. Radford, and I. Sutskever, “Generating long sequences with sparse transformers,” _arXiv preprint arXiv:1904.10509_ , 2019.
  * [19] H. Zhang, M. Cisse, Y. N. Dauphin, and D. Lopez-Paz, “Mixup: Beyond empirical risk minimization,” _arXiv preprint arXiv:1710.09412_ , 2017.
  * [20] T. Wolf, L. Debut, V. Sanh, J. Chaumond, C. Delangue, A. Moi, P. Cistac, T. Rault, R. Louf, M. Funtowicz _et al._ , “HuggingFace’s transformers: State-of-the-art natural language processing,” _arXiv preprint arXiv:1910.03771_ , 2019.
  * [21] Z. Guo, J. Kang, and D. Herremans, “A domain-knowledge-inspired music embedding space and a novel attention mechanism for symbolic music modeling,” _arXiv preprint arXiv:2212.00973_ , 2022.
  * [22] Q. Kong, B. Li, J. Chen, and Y. Wang, “GiantMIDI-piano: A large-scale MIDI dataset for classical piano music,” _arXiv preprint arXiv:2010.07061_ , 2020.
  * [23] C. Hawthorne, A. Stasyuk, A. Roberts, I. Simon, C.-Z. A. Huang, S. Dieleman, E. Elsen, J. Engel, and D. Eck, “Enabling factorized piano music modeling and generation with the MAESTRO dataset,” in _Proceedings of the 7th International Conference on Learning Representations (ICLR)_ , 2019.
  * [24] C. Hawthorne, E. Elsen, J. Song, A. Roberts, I. Simon, C. Raffel, J. Engel, S. Oore, and D. Eck, “Onsets and frames: Dual-objective piano transcription,” _arXiv preprint arXiv:1710.11153_ , 2017.
  * [25] Q. Kong, B. Li, X. Song, Y. Wan, and Y. Wang, “High-resolution piano transcription with pedals by regressing onset and offset times,” _IEEE/ACM Transactions on Audio, Speech, and Language Processing_ , vol. 29, pp. 3707–3717, 2021.
  * [26] M. Müller, P. Grosche, and N. Jiang, “A segment-based fitness measure for capturing repetitive structures of music recordings.” in _Proceedings of the 12th International Society for Music Information Retrieval Conference (ISMIR)_ , 2011.
  * [27] B. Yu, P. Lu, R. Wang, W. Hu, X. Tan, W. Ye, S. Zhang, T. Qin, and T.-Y. Liu, “Museformer: Transformer with fine-and coarse-grained attention for music generation,” _arXiv preprint arXiv:2210.10349_ , 2022.
  * [28] Y.-J. Shih, S.-L. Wu, F. Zalkow, M. Muller, and Y.-H. Yang, “Theme transformer: Symbolic music generation with theme-conditioned transformer,” _IEEE Transactions on Multimedia_ , no. 8, pp. 1–14, 2022.
  * [29] M. Müller and N. Jiang, “A scape plot representation for visualizing repetitive structures of music recordings.” in _Proceedings of the 13th International Society for Music Information Retrieval Conference (ISMIR)_ , 2012.
  * [30] M. Zeng, X. Tan, R. Wang, Z. Ju, T. Qin, and T.-Y. Liu, “MusicBERT: Symbolic music understanding with large-scale pre-training,” in _Proceedings of the 59th Association for Computational Linguistics (ACL)_ , 2021.
  * [31] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “BERT: Pre-training of deep bidirectional transformers for language understanding,” _arXiv preprint arXiv:1810.04805_ , 2018.
  * [32] C. Raffel, N. Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Y. Zhou, W. Li, and P. J. Liu, “Exploring the limits of transfer learning with a unified text-to-text transformer,” _The Journal of Machine Learning Research_ , vol. 21, no. 1, pp. 5485–5551, 2020.
  * [33] Z. Dai, Z. Yang, Y. Yang, J. Carbonell, Q. V. Le, and R. Salakhutdinov, “Transformer-XL: Attentive language models beyond a fixed-length context,” _arXiv preprint arXiv:1901.02860_ , 2019.
  * [34] A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly _et al._ , “An image is worth 16x16 words: Transformers for image recognition at scale,” _arXiv preprint arXiv:2010.11929_ , 2020.
  * [35] V. P. Dwivedi and X. Bresson, “A generalization of transformer networks to graphs,” _arXiv preprint arXiv:2012.09699_ , 2020.
  * [36] M. Müller, N. Jiang, and H. G. Grohganz, “SM toolbox: Matlab implementations for computing and enhancing similarity matrices,” in _Proceedings of 53rd Audio Engineering Society (AES)_ , London, UK, 2014.

##  Appendix A Related Work

###  A.1 Transformer-based Symbolic Music Generation.

Inspired by previous work on large-scale text generation, there exist multiple
frameworks that leverage the Transformer architecture for event-based music
generation. Among many, Music Transformer[9] was the first to successfully
apply Transformers to music generation with long-range dependencies using
relative local attention. In the following year, MuseNet[16] proposed applying
the GPT2[17] architecture to MIDI-based music generation, showing state-of-
the-art performance under prompts with various contexts such as composer and
instrumentation. Several works have also studied music generation in specific
genres such as Pop Music Transformer[7] and Jazz Transformer[10].
MusicBERT[30] proposed using bidirectional masked token prediction as in
BERT[31] with bar-level masking to avoid information leakage among nearby
tokens. Museformer[27] proposed using a coarse- and fine-grained attention
module to reduce the quadratic computational cost of attention to nearly
linear. In our work, we implement a revamped version of MuseNet[16] to cope
with the lack of reproducible code, and also empirically study different
design choices on structural embedding initializations and prompt lengths
during inference.

###  A.2 Structural Embeddings for Music Generation.

For sequence modeling, the original Transformer architecture proposed an
absolute sinusoidal positional encoding[4]. To cope with long sequences,
T5[32] and Transformer-XL[33] have proposed a relative positional encoding
that adds inductive bias directly to the attention-score matrix. Extending to
vision- and graph-data, ViT[34] and Graph Transformer[35] also proposed
positional encoding modules that leverage only topological data structure,
meaning that it can be applied to any large-scale datasets without additional
knowledge. In the musical domain, there exist many variations of event-based
MIDI tokenization pipelines such as REMI[7], CP[8], and OctupleMIDI[30] that
aggregate structural MIDI information into a single token. While they
effectively reduce the computational cost by shortening the overall sequence
length, these require additional structural information that is often missing
in in-the-wild music data such as chord and bar structure. Therefore, we focus
on the structural embedding procedure for event-based MIDI tokenization that
does not require any human annotation or heuristic labeling.

##  Appendix B Metric Details

###  B.1 Structureness Indicator.

We leverage Libfmp
API333https://meinardmueller.github.io/libfmp/build/html/index_c4.html to
compute SI and the corresponding fitness scape plots. We calculate SI as the
overall fitness score following the procedure provided by the
authors444https://www.audiolabs-
erlangen.de/resources/MIR/FMP/C4/C4S3_ScapePlot.html and parameter settings
from[10]: we use the sampling rate of 1 Hz, or 1 second per frame, for
computing the objective metric scores. For the fitness scape plot, we compute
a $N\times N$ fitness matrix from each of the 12 listening samples for each
model, where $N$ is the maximum number of frames derived from downsampling the
original audio files, and each frame denotes 500 ms for fine-level
visualization[36]. Then, we average the fitness matrices by sample.
Empirically, we set $N=61$ as the maximum length of the samples in 30 seconds.
If a sample has a length smaller than $N$, we pad the remaining length with
NaN values.

###  B.2 Chord Progression Variation Rationality.

We follow[12] to compute this metric. The conditional probability $V_{i}$ is
approximated from the appearance frequency of a unique chord $n$-grams for
each $n\in\\{2,3,4\\}$ within the training set, which is the aggregation of
Pop1k7[8] and GiantMIDI-Piano[22], as follows:

$V_{i}=\mathbf{p}\left(C_{i}^{i+n}\mid C_{i}^{i-1+n}\right)$ (2)

where $i$ is the chord index, $C_{i}^{i+n}$ is a current chord $n$-gram
spanning from the $i$-th chord to the $i+n$-th chord, and
$\mathbf{p}\left(\cdot\mid\cdot\right)$ is the conditional probability.