# On the Effectiveness of Speech Self-Supervised Learning for Music

###### Abstract

Self-supervised learning (SSL) has shown promising results in various speech
and natural language processing applications. However, its efficacy in music
information retrieval (MIR) still remains largely unexplored. While previous
SSL models pre-trained on music recordings may have been mostly closed-
sourced, recent speech models such as wav2vec2.0 have shown promise in music
modelling. Nevertheless, research exploring the effectiveness of applying
speech SSL models to music recordings has been limited. We explore the music
adaption of SSL with two distinctive speech-related models, data2vec1.0 and
Hubert, and refer to them as music2vec and musicHuBERT, respectively. We train
$12$ SSL models with 95M parameters under various pre-training configurations
and systematically evaluate the MIR task performances with 13 different MIR
tasks. Our findings suggest that training with music data can generally
improve performance on MIR tasks, even when models are trained using paradigms
designed for speech. However, we identify the limitations of such existing
speech-oriented designs, especially in modelling polyphonic information. Based
on the experimental results, empirical suggestions are also given for
designing future musical SSL strategies and paradigms.

##  1 INTRODUCTION

Deep learning (DL) techniques have shown promising results in a wide range of
auditory tasks, including speech and music information retrieval (MIR).
However, the quantity and quality of labelled data is a bottleneck for
developing algorithms with better generalisation in complex real-world
settings for machine listening. To address this issue, self-supervised
learning (SSL) such as BERT [1] has emerged as a solution to leverage diverse
and representative unlabelled data to train a deep feature extractor with
better generalisation. By combining this pre-trained SSL encoder with a naive
classifier, typically a multi-layer perceptron (MLP) or long short-term memory
(LSTM) with limited hidden layers, the model can achieve strong or state-of-
the-art (SOTA) performance in various downstream tasks including NLP [1, 2,
3], computer vision [4], and audio [5, 6], where well-labelled datasets are
limited. For music, larger datasets can be more expensive due to copyright and
annotation costs, making SSL essential for developing effective MIR systems.
Investigating versatile SSL approaches in MIR can further improve the
performance on many MIR tasks, benefitting the music industry, music
education, and heritage preservation. Although SSL has significantly improved
the performance of models in tasks such as speech recognition, sentiment
analysis, and language modelling, its effectiveness in music information
retrieval (MIR) remains largely unexplored.

There has been much work on SSL for audio representation learning, including
speech, sound events or music. But most results are difficult to evaluate or
fine-tune due to limited access to training data, pre-trained parameters or
training codes. PANN [7] is trained on noisy/weak-label classification and
does not provide promising results in music tasks such as pitch classification
and instrument classification [8]. Besides, it can hardly be re-trained on
music datasets given that the MIR community does not have a weekly labelled
large music dataset. MusiCoder [9], Music PASE [10], and MAP-Music2Vec [11]
use strategies mainly based on masked prediction, where training models
predict the audio waveform manually-designed feature or learnable deep feature
of input removed randomly from the ground truth. Such models trained on music
are not open-sourced except MAP-Music2Vec, which provides pre-trained
parameters on hugging-face111https://huggingface.co/m-a-p/music2vec-v1.
Jukebox [12] uses similar strategies for pop-song recording generation and
demonstrates good potential for multiple MIR tasks [6]. But the training code
for it is unavailable and is hard to fine-tune given its 6 billion parameters.
MAP-MERT v0 [13] mimicks HuBERT [14], which regards the clustering results of
audio as a pseudo label or pseudo spectrum to be reconstructed rather than a
cluster assignment. But it does not provide training codes for further model
evaluation. Furthermore, there are some music SSL models based on instance
discrimination. In this family of approaches, each instance is considered its
class, and models are trained to distinguish among different instances. CLMR
[15] is trained with a limited number of parameters and shows limited capacity
[6]. PEMR [16] does not show promising results besides tagging and is not
open-source for further evaluation.

Although not designed for MIR tasks, some speech SSL models provide promising
results on music tasks, and their training codes are available for fine-tuning
or re-training on musical audio. Mockingjay [17], and PASE [18] use masked
waveform / audio-feature prediction for pre-training. COLR [19] uses
EfficientNet with a limited number of parameters and is designed for general
audio, though it has a promising result on instrument classification. SF-
NFNet-F0 [20] also uses an architecture based on convolution neural networks,
a SlowFast Normalizer-Free ResNet, for audio pre-training. Furthermore, apart
from providing good results on automatic speech recognition (ASR), Wav2Vec2.0
[5], HUBERT and data2vec [21] also provide much better results on pitch
estimation and instrumental classification than PANN, though they are still
far from perfect [8]. All of the speech SSL models are helpful for music SSL
model development.

Previous work on re-training speech SSL systems with music recordings is
limited to the size of training datasets or model structure. Ragano et al.
[22] re-trained wav2vec2.0 on music audio and improved performance on pitch
estimation and instrument classification significantly. But the training set
is less than 100 hours which may be less representative, and the downstream
tasks are limited and not universal. MusiCoder and Music PASE can be regarded
as re-training speech SSL models on music recordings, but the model
performance is not promising. Besides, these models are evaluated with a
limited number of downstream tasks, making the learned embedding less
persuasive. SF-NFNet-F0 is trained on music recordings and provides better
results on multiple music tagging tasks [23]. But its model architecture is
based on CNN, without much room for further scale-up and longer sequence
modelling.

The missing science in the previous studies is as follows. All of the existing
models trained on music are either with a limited number of parameters and
capacity for MIR tasks other than tagging or not open-source for further
evaluation. Some of the systems developed on speech or general audio
recordings demonstrate promising but not satisfying results on MIR tasks.
Besides, previous investigations on the efficacy of applying speech-related
SSL models to music recordings are limited by the size of the training set,
not enough universality on the downstream tasks, or paying less attention to
powerful transformer structures.

Our key contributions are four-fold: (1) exploring two speech-related SSL
models based on transformer structures, data2vec and HuBERT, and comparing the
results with those models pre-trained in speech recordings; (2) carrying out
ablation studies for pre-training, thus providing more intuition for further
music SSL system design; and (3) systematically comparing the performance on
13 downstream tasks, which facilitates comprehensive model evaluation on a
wide range of MIR tasks.

(a) Music2Vec

(b) MusicHuBERT

Figure 1:  Pre-training Paradigms of Selected Models. Both of the models are
fed with masked audio inputs and predict given targets without supervised
information.

##  2 METHOD

In order to keep the pre-training and representation evaluation protocols
comparable, we focus on adapting from the speech self-supervised learning
frameworks that support direct audio input and end-to-end pre-training. Given
our intent of exploring the influence of the pre-training design itself, we
choose two SSL frameworks mainly distinguished by their self-supervised
learning targets while sharing very similar training settings, including model
architecture, training datasets, and evaluation protocols. In this section, we
briefly describe the two selected SSL models – data2vec-1.0 [21] and HuBERT
[14] – in the unified auto-encoding framework (cf. Fig. 1) and discuss the
similarities and differences under music audio pre-training.

###  2.1 Music2Vec: Continuous Target Prediction

We adapt the pre-training paradigm from the speech version of the multi-modal
framework data2vec-1.0 [21], where the prediction targets during pre-training
are continuous representations. We refer to this continuous prediction model
adapted with music recordings as Music2Vec.

Modified from the design of bootstrap your own latent (BYOL) [24], Music2Vec
aims to predict continuous latent representations from the teacher model for
the masked input audios, which is illustrated in Fig. 1(a). The teacher model
and student model share the same architecture, and the parameters of the
teacher model are updated according to the exponential moving average of the
student [21]. The student model takes the partially masked input and is asked
to predict the average pooling of top-$K$ layer outputs from the Transformer
[25] in the teacher model. In contrast, the teacher model takes the unmasked
input and provides contextual prediction targets in the pre-training.

Following the data2vec [5] setting, we train the Music2Vec of 95M parameters
with a comparable 1k hours of music recordings. Since pre-trained speech
models can barely benefit music representation learning [22], we instead train
the base model from scratch to verify its effectiveness in modelling music
audio recordings.

###  2.2 MusicHuBERT: Discrete Target Prediction

Another efficient speech SSL model, HuBERT [14], is chosen as the
representative of discrete target prediction design. We referred to the music
adaption version as MusicHuBERT.

The MusicHuBERT model takes masked music audios as input (Similar to
Music2Vec) and predicts pre-processed discrete labels corresponding to the
masked area, as shown in Fig. 1(b). The discrete targets are pseudo labels
provided by K-means that are trained on the MFCC features of the training
audios. The number of clusters $K$ of the K-means model is a hyperparameter,
and all the centroids are assigned with randomly initialised embeddings and
learned during the MusicHuBERT pre-training. MusicHuBERT can also be trained
for an extra $n$ iterations, where K-means clustering is learned from model
outputs’ previous iteration. We follow the original HuBERT [14] setting to
train a model with 95M parameters of the same size as Music2Vec.

###  2.3 similarities & Differences of SSL frameworks

This subsection will examine the similarities and differences between the SSL
frameworks mentioned above.

Both Music2Vec and MusicHuBERT are annotation-free and utilise SSL techniques;
their most common characteristic is the training task of “reconstructing”
information from masked inputs, making them auto-encoding models. During the
denoising process, these models learn the semantics contained in the audio.
Furthermore, they share similar model architecture designs, which are
inherited from wav2vec-2.0 [5], wherein the audio is initially encoded by a
multi-layer 1-D CNN feature extractor that maps a 16 kHz waveform to 50 Hz
representations. The encoded tokens are then fed into a 12-layer transformer
block with a hidden dimension of $H=768$.

Regarding the differences in the designs, the most notable one is that
Music2Vec is required to predict continuous latent variables, whereas
MusicHuBERT predicts discrete pseudo-labels. The time cost of SSL target
preparation bottleneck varies according to their mechanism. In Music2Vec, the
pre-training consumes twice the model forward time since the target
representations from the teacher model are inferred on-the-fly. In contrast,
MusicHuBERT trains the K-means model and infers all the pseudo-labels before
training, which requires high parallel processing ability when the dataset is
scaled-up.

##  3 DATASET & EVALUATION

###  3.1 Training

We use a private dataset with 1000 hours of music audio recordings for pre-
training; each sample is a 30s-long excerpt from pop-song or instrumental
music. The size of the pre-training dataset is roughly the same as the pre-
training for HuBERT-base and data2vec-audio-base models.

###  3.2 Evaluation

We evaluate the models on 13 downstream tasks, including timbre classification
tasks such as genre and instrumental classification, singing, playing
technique classification, singer classification, and music tagging; emotion-
related tasks like music mood classification and regression; and note-related
tasks such as pitch estimation, key detection; and sequential tasks like beat
tracking.

Music Tagging is a multi-label classification task. We used MagnaTagATune
(MTT) [26] and MTG-Jamendo [27] for this task, tag categories of which include
genre, instrumentation, mood, and tempo (e.g. fast) etc. For both datasets, we
limit the tag vocabulary to the 50 most common tags. We use all clips in MTT
and MTG-Jamendo for evaluation. Since many of the audio recordings among 5.5k
MTG-Jamendo excerpts are longer than the 30s, we averaged the multiple
embeddings computed with a 30s sliding window as the overall embedding. The
metrics are the macro-average of ROC-AUCs and the average precision (AP) / PR-
AUC among all top-50 tags.

Key detection. We use a commonly-used subset of Giantsteps-MTG-keys [28] as
the training and validation set following the data splitting [6], and
Giantsteps (GS) [29] as the test set. The metric is a refined accuracy that
gives partial credit to reasonable errors [30].

Genre classification. We report the multi-class classification accuracy of the
GTZAN [31] dataset, along with ROC and AP on MTG-Genre for multi-label. We
used the standard "fail-filtered" split [32] for GTZAN.

Emotion score regression. The Emomusic dataset [33] contains 744 music clips
of 45 seconds, each reported on a two-dimensional valence-arousal plane after
listening. We use the same dataset split as [6]. The official evaluation
metric is the determination coefficient ($r^{2}$) between the model regression
results and human annotations of arousal (EmoA) and valence (EmoV) [33]. We
split the 45-second clip into a 5-second sliding window for inference and
averaged the prediction.

Instrument classification. We use the Nsynth [34] and MTG-instrument datasets.
The former is a multi-class task on 306k audio samples in 11 instruments with
accuracy as an indicator. The latter is a subset of MTG-Jamendo, containing
25k audio tracks and 41 instrument tags; each track can contain multiple
instruments and is evaluated on ROC and AP.

Pitch classification. Given these audios are short monophonic audio, this task
is multi-class to determine which of the 128 pitch categories, and the
accuracy is used as an evaluation metric.

Vocal technique detection. We use the VocalSet dataset [35], which is the only
publicly available dataset for the study of singing techniques. The dataset
contains the vocals of 17 different singing techniques in various contexts for
a total of 10.1 hours. As the audio clips are divided into 3 seconds, the task
only requires a judgment on the type of technique and not on the start and end
of the technique. We used the same 10 different singing techniques as in [36]
as a subset and used the same 15 singers as the training and validation sets
and 5 singers as the test set. Since there is no accepted division between
training and validation sets, we selected 9 singers as the training set and 6
singers as the validation set. All the 3-second segments originate from the
same recording are allocated to the same part of the split (e.g. all in the
testing set).

Singer identification is to identify the vocal performer from a given
recording. We randomly divided the VocalSet dataset, which contains 20
different professional singers (9 female and 11 male), into a training set,
validation set and testing set based on a ratio of 12:8:5, all containing the
same 20 singers.

Beat tracking. We use an offline approach to the binary classification, i.e.
the model can use the following information from each frame to help with
inference. The model needs to output frame-by-frame predictions at a certain
frequency and post-process them using a dynamic Bayesian network (DBN) [37],
the same methods with supervised SOTA. The DBN is implemented using madmom
[38]. The dataset we use is GTZAN Rhythm [39]. We also label the two adjacent
frames of each label as beat, a common way of smoothing in beat tracking. The
model is evaluated using the f_measure implemented in mir_eval [30], and the
prediction is considered correct if the difference between the predicted event
and the ground truth does not exceed 20ms. In this task, some models were
trained on other datasets, and the full GTZAN set was used as the test set.
For all cases, however, we use GTZAN-train as the training set and GTZAN-test
as the test set.

Emotion Tagging. We use MTG-MoodTheme, another subset of MTG-Jamendo [27] that
contains 18.5k audio tracks and 59 tags. Unlike Emomusic, this is a multi-
label task, with ROC and AP as metrics.

##  4 EXPERIMENTAL RESULTS

We use the fairseq framework222https://github.com/facebookresearch/fairseq
from Meta to train MusicHuBERT and Music2Vec models. All the MusicHuBERT and
Music2Vec models are trained for 400k steps with 8 $\times$ NVIDIA A100-40GB
GPUs. Training with 8 GPUs takes around $2-3$ days. The experimental results
are chiefly as follows.

Our findings suggest these SSL models pre-trained on speech can be helpful for
MIR tasks, but pre-trained on music is generally more helpful, besides some
exceptions. In section 4.2, we identify the strengths along with weaknesses of
training strategies, revealing areas for further improvement. In section 4.3,
we discuss the effect of hyperparameters in pretext tasks.

###  4.1 Pre-trained on Speech and Music

Table 1: Experimental performance of the SSL baseline systems on all
downstream tasks

Downstream | MTT | GS key | GTZAN | EMO | Nsynth | Nsynth | VocalSet | VocalSet | GTzAN | MTG | MTG | MTG | MTG
---|---|---|---|---|---|---|---|---|---|---|---|---|---
dataset |  |  | Genre |  |  | Instr | pitch | tech | singer | Rhythm | Instrument | MoodTheme | Genre | Top50
Metrics | ROC | AP | Refined Acc | Acc | $Emo_{V}$ | $Emo_{A}$ | Acc | Acc | Acc | Acc | F1 (beat) | ROC | AP | ROC | AP | ROC | AP | ROC | AP
HuBERT | 89.8 | 36.4 | 15.0 | 64.8 | 31.0 | 57.5 | 68.2 | 79.4 | 61.0 | 58.8 | 83.5 | 73.2 | 17.0 | 74.0 | 11.6 | 85.0 | 16.3 | 81.8 | 26.5
base
MusicHuBERT | 90.2 | 37.7 | 14.7 | 70.0 | 42.1 | 66.5 | 69.3 | 77.4 | 65.9 | 75.3 | 88.6 | 75.5 | 17.8 | 76.0 | 13.9 | 86.5 | 18.0 | 82.4 | 28.1
base
data2vec | 88.4 | 33.6 | 15.5 | 60.7 | 23.0 | 49.6 | 69.3 | 77.7 | 64.9 | 74.6 | 36.4 | 73.1 | 16.9 | 73.3 | 11.0 | 83.5 | 14.5 | 80.6 | 24.8
audio base
Music2vec | 89.1 | 35.1 | 19.0 | 59.7 | 38.5 | 61.9 | 69.4 | 88.9 | 68.3 | 69.5 | 33.5 | 73.1 | 16.3 | 74.3 | 12.2 | 85.2 | 16.5 | 81.4 | 26.2
vanilla
SOTA | 92.0[40] | 41.4[6] | 74.3[28] | 82.1[41] | 61.7 | 72.1[6] | 78.2[20] | 89.2[23] | 65.6[36] | 80.3[42] | 80.6[43] | 78.8 | 20.2[44] | 78.6 | 16.1[23] | 87.7 | 20.3[44] | 84.3 | 32.1[23]

Table 1 demonstrates the performance of
HuBERT333https://huggingface.co/facebook/HuBERT-base-ls960 and
data2vec444https://huggingface.co/facebook/data2vec-audio-base SSL models that
were pre-trained on speech recordings and music recordings separately. Here,
we only consider the SOTA performance trained with the same dataset
train/valid/test split. All of the models are used as parameter-frozen feature
extractors. The weighted sum of one output of the CNN tokeniser as well as the
12 outputs of all the transformer layers, are combined with an MLP as the back
end. The MLP has only one single 512-dimension hidden layer. The learning rate
of the probing is set to 1e-3.

For the HuBERT model, the results pre-trained on speech recordings are
comparable with SOTA on tasks like music tagging, beat tracking, pitch
estimation and singing technique classification etc., and are surpassed by the
results pre-trained on music audio on most of the downstream tasks besides
pitch estimation on Nsynth and key detection on GS. For pitch detection, the
data samples in Nsynth are a single note played by one single monophonic
instrument, which is similar to speech data. So it is reasonable that HuBERT
pre-trained on speech data is capable of modelling a single pitch. Although
HuBERT surpasses the vanilla MusicHuBERT on GS and Nsynth-pitch, it is
surpassed by the results of MusicHuBERT with an ablation study on pre-training
hyperparameters (shown in Table 2).

For data2vec, the data2vec-audio results are also comparable with SOTA on many
tasks and have a large gap on others, and overall surpassed by Music2Vec or
its ablation study shown in Table 3 on most of the tasks as well. But the
data2vec results of beat tracking on GTZAN-Rhythm and singer identification on
Vocalset surpassed all Music2Vec. Vocalset includes singing of different
phonemes with different singing techniques by different singers. The speech
SSL system is capable of modelling diverse phonemes in ASR and various timbres
of speakers but has less focus on timbre in speaking techniques you may find
in opera. On the contrary, the music SSL models may focus more on phonemes
(lyrics) and singing timbre (techniques) but include less focus on the singer
itself. For beat tracking, we observe that the performance is reduced
significantly when the number of transformer layers increases from 0 to 12.
This shows that the data2vec structure may not be useful for learning temporal
information.

###  4.2 Pre-trained with Different Paradigms

From Table 1, we can tell that MusicHuBERT is more promising than Music2vec
given that it provides better results in most of the downstream tasks,
especially genre classification on GTZAN, emotion regression on EMO and beat
tracking on GTZAN. But it is worse on single-pitch estimation on Nsynth, along
with key detection on GS.

These phenomena suggest pre-training with the HuBERT paradigm is strongly
correlated with the MFCC feature information used for k-means. Therefore, the
quantisation results lack multi-pitch information, including harmony or chord
modelling, that is essential to key detection. The following research can use
the chroma feature to replace MFCCs555For more information on this, please
refer to our following paper MERT: Acoustic Music Understanding Model with
Large-Scale Self-supervised Training at https://arxiv.org/abs/2306.00107. On
the contrary, the mask prediction for the deep feature in the data2vec pre-
training paradigm is clearly better but still has much room for improvement
compared to the SOTA. Although the deep feature still lacks sufficient
harmonic information for key detection, it already contains enough information
for single-pitch estimation, and the MFCCs may focus more on the timbre of
instruments instead of the fundamental frequency. Apparently, Music2Vec can
learn pitch information more freely. Besides, data2vec is generally a bit
worse for tagging than Music2vec, and both are significantly worse on beat
tracking compared to HuBERT and MusicHuBERT.

###  4.3 Ablation Studies on Pretraining Hyperparameters

Here, we carry out an ablation study of hyperparameter search under both pre-
training paradigms. Given the time limitation, we did not extract features on
MTG datasets and only calculated the results in another 9 downstream tasks.

####  4.3.1 Ablation Study on MusicHuBERT

Table 2: Ablation study on MusicHuBERT hyperparameters (k is the number of
MFCC clusters)

Downstream | MTT | GS key | GTZAN | EMO | Nsynth | Nsynth | VocalSet | VocalSet | GTZAN | Average
---|---|---|---|---|---|---|---|---|---|---
dataset |  |  | Genre |  |  | Instr | pitch | tech | singer | Rhythm | Score
Metrics | ROC | AP | Refined Acc | Acc | $Emo_{V}$ | $Emo_{A}$ | Acc | Acc | Acc | Acc | F1 (beat) | score
HuBERT | 89.8 | 36.4 | 15.0 | 64.8 | 31.0 | 57.5 | 68.2 | 79.4 | 61.0 | 58.8 | 83.5 | 59.8
k=2000 MFCC dim=39 | 90.2 | 37.7 | 14.7 | 70.0 | 42.1 | 66.5 | 69.3 | 77.4 | 65.9 | 75.3 | 88.6 | 64.4
k=2000 iter2 | 90.4 | 37.5 | 13.8 | 68.3 | 43.3 | 67.4 | 70.0 | 80.3 | 63.6 | 70.4 | 88.8 | 63.8
k=500 MFCC dim=39 | 89.6 | 36.1 | 15.7 | 64.5 | 41.0 | 67.7 | 66.7 | 76.8 | 60.5 | 72.3 | 87.5 | 62.4
k=500 MFCC dim=60 | 90.3 | 38.0 | 17.6 | 69.7 | 40.8 | 67.5 | 70.3 | 79.0 | 66.2 | 75.5 | 88.6 | 65.0

We use the number of clusters k =500 and k=2000. For the case k=500, we
increase the dimension of MFCC features from 13, which is commonly used in the
speech community, to 20, which is widely used in sound event detection. Thus,
the dimension of MFCCs combined with their delta features and delta-delta
features have 39 and 60 dimensions respectively. For the case of k=2000, we
use the 768-dimension deep feature learned from the first iteration experiment
to carry out the second iteration k-means.

From Table 2, we can see that MusicHuBERT with k=2000 is better than the k=500
case for most of the tasks. Given HuBERT is good for speech when k=100 or
k=500, which is roughly the number of human phonemes, this implies music
tokens or notes are much richer than speech and therefore need a larger number
for quantisation.

The results on k-means for deep features are better than the vanilla
MusicHuBERT besides genre classification on GTZAN, singer identification on
vocalset, and singing techniques classification on vocalset. This implies the
MFCCs features are good for modelling the human voice, regardless of speech or
singing. The results of GTZAN may be due to the randomness as the dataset is
very small.

Besides, increasing the dimension of MFCCs provides no significant difference
among most of the tasks other than tasks on Nsynth and GS. Increased
dimensionality for MFCC features can provide more detailed information on
impulse response for a sound event. Thus, monophonic instrumental notes can be
better modelled with 60-dimension MFCC features. Furthermore, the emotion
regression also provides different results, but the average of the two metrics
is nearly the same, providing no significant improvement.

####  4.3.2 Ablation Study on Music2Vec

Table 3: Ablation study on Music2Vec hyperparameters (span is mask span, prob
is mask probability, step is training steps, target=12 uses all 12 transformer
layers, and crop5s uses 5s music excerpts)

Downstream | MTT | GS key | GTZAN | EMO | Nsynth | Nsynth | VocalSet | VocalSet | GTZAN | Average
---|---|---|---|---|---|---|---|---|---|---
dataset |  |  | Genre |  |  | Instr | pitch | tech | singer | Rhythm | Score
Metrics | ROC | AP | Refined Acc | Acc | $Emo_{V}$ | $Emo_{A}$ | Acc | Acc | Acc | Acc | F1 (beat) | score
data2vec | 88.4 | 33.6 | 15.5 | 60.7 | 23.0 | 49.6 | 69.3 | 77.7 | 64.9 | 74.6 | 36.4 | 55.2
vanilla | 89.1 | 35.1 | 19.0 | 59.7 | 38.5 | 61.9 | 69.4 | 88.9 | 68.3 | 69.5 | 33.5 | 57.8
span=5 | 87.3 | 32.0 | 15.7 | 47.6 | 22.7 | 41.2 | 64.2 | 84.8 | 56.7 | 53.8 | 33.2 | 49.7
span=15 | 88.7 | 34.3 | 16.4 | 56.6 | 39.0 | 58.8 | 67.1 | 88.1 | 63.1 | 61.9 | 33.1 | 55.2
prob=50 | 88.5 | 34.0 | 23.7 | 59.3 | 40.6 | 55.0 | 66.8 | 87.7 | 64.9 | 61.7 | 33.9 | 56.3
prob=80 | 88.2 | 33.9 | 18.4 | 50.3 | 36.7 | 55.7 | 67.9 | 88.9 | 64.2 | 65.2 | 33.7 | 55.1
step=800k | 87.7 | 32.7 | 20.3 | 54.5 | 34.9 | 47.3 | 66.9 | 87.5 | 65.6 | 65.1 | 33.4 | 55.0
target=12 | 89.7 | 35.2 | 26.5 | 64.5 | 41.7 | 64.2 | 71.1 | 89.2 | 71.0 | 73.2 | 34.1 | 60.6
crop5s | 90.0 | 36.6 | 18.5 | 76.6 | 53.4 | 71.6 | 68.3 | 88.9 | 71.3 | 72.4 | 33.9 | 61.8

We use audio files with 30s length, mask span length 10, mask probability 65%,
target top-$8$ transformer layer the teacher model as a deep feature, and
training step 400K as the vanilla setting. We conduct parameter searching and
correlation analysis for Music2Vec pretraining, including masking strategy,
training steps, the learning target layers, and recording length; the results
are shown in Table 3.

We revise the masking strategy by changing the mask span length and mask token
probability in the data2vec-audio-base setting. Mask token probability is the
probability for each token to be chosen as the start of the span to be masked,
the length of which can also be adapted for different data modalities. The
results in Table 3 show that the other span value and other mask token
probability provide a bit worse results on nearly all the tasks. This suggests
that the data2vec hyperparameters for speech pre-training are generally
helpful for music pre-training.

Given the fact that early transformer layer representations generally perform
well on key detection and beat tracking, we modify the prediction target
provided by the teacher model. We change the prediction target in Music2Vec
from the original one, that is, the average of the top-8 layer
representations, to all the 12 layers. The results in Table 3 show that
Music2Vec actually benefits, not only from the potentially preserved key
information shown by a significant increase on GS but all the other tasks as
well.

Furthermore, we use audio length cropping to shorten music excerpts since
longer sequences are more difficult to model. Note that the combined audio
length in a batch on a single GPU is not altered, and the hardware environment
remains the same, making a single training batch contain a larger number of
music samples when clips are cropped. Due to the position embedding in the SSL
systems, the model can get information more than 5 seconds after pre-training
on only 5-second music recordings. But the key detection provides worse
results which may lead to the fact that a local key within a 5-second song may
not be identical to the global key in the whole music sentence.

##  5 CONCLUSION & DISCUSSION

In this paper, we explore the music variants of two distinctive speech-related
transformer-based SSL models, data2vec and HuBERT. Our findings suggest that
pre-training with music recordings rather than speech can generally improve
performance on a wide range of MIR tasks, even when the models and training
are designed for speech. There are exceptions for data2vec, however, such as
singer identification, the dataset of which is similar to the speech dataset
used to pre-train. Thus, when resources are limited, our suggestion is to use
speech pre-training models, given that they can provide helpful information
about music already. Speech data can be beneficial if lacking a sufficient
vocal dataset with different singers, but one should use mainly music data if
possible.

Furthermore, we can use the same speech training hyperparameters for masked
span and masked probability in music pre-training. But some other
hyperparameters, such as the number for pseudo label clustering, might be the
shortage of pretext strategies. We identified some limitations of existing
speech SSL systems, especially in the case of harmonic information and
diversity of music notes. One suggestion is to emphasise key or harmonic in
the pretext task for music SSL models by using more than just MFCC features.
Also, the number of categories for quantisation in k-means should be much
larger if necessary, given the number of pitch, chord, and timbre categories
is much larger than the number of human speech phones. This diversity in music
might be a bottleneck for both speech SSL systems to learn good music
features. For one thing, the larger number of clusters for k-means in HuBERT
is expensive to calculate, making it harder to scale up, preventing
transformer-based models from reaching their potential for better performance
and longer sequence modelling. In addition, it may not be easy for data2vec to
jointly learn deeper features. We may need curriculum learning skills or
manually-designed features to increase training stability.

Another general suggestion for pre-training recognises that batch size should
be as diverse as possible. Given that the memory of one single machine is
limited, it is a good idea to shorten the length of audio to be modelled at
first, allowing for an increase in batch size, and then train another language
model for long sequence modelling.

We believe the findings in this paper to be of value in understanding the
potential for SSL speech models applied to music, and we have offered some
general insights about music modelling that resulted from this study.

##  6 Acknowledgements

Yinghao Ma is a research student at the UKRI Centre for Doctoral Training in
Artificial Intelligence and Music, supported by UK Research and Innovation
[grant number EP/S022694/1]. Yizhi Li is fully funded by an industrial PhD
studentship (Grant number: 171362) from the University of Sheffield, UK. This
work is supported by the National Key R&D Program of China (2020AAA0105200).
We acknowledge IT Services at The University of Sheffield for the provision of
services for High-Performance Computing. We would also like to express great
appreciation for the suggestions from faculties Dr Chris Donahue, and Dr Roger
Dannenberg, as well as the facility support from Mr. Yulong Zhang in the
preliminary stage.

## References

  * [1] J. D. M.-W. C. Kenton and L. K. Toutanova, “Bert: Pre-training of deep bidirectional transformers for language understanding,” in _Proceedings of NAACL-HLT_ , 2019, pp. 4171–4186.
  * [2] S. Gururangan, A. Marasović, S. Swayamdipta, K. Lo, I. Beltagy, D. Downey, and N. A. Smith, “Don’t stop pretraining: Adapt language models to domains and tasks,” _arXiv preprint arXiv:2004.10964_ , 2020.
  * [3] J. Sarzynska-Wawer, A. Wawer, A. Pawlak, J. Szymanowska, I. Stefaniak, M. Jarkiewicz, and L. Okruszek, “Detecting formal thought disorder by deep contextualized word representations,” _Psychiatry Research_ , vol. 304, p. 114135, 2021.
  * [4] A. Newell and J. Deng, “How useful is self-supervised pretraining for visual tasks?” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2020, pp. 7345–7354.
  * [5] A. Baevski, Y. Zhou, A. Mohamed, and M. Auli, “wav2vec 2.0: A framework for self-supervised learning of speech representations,” _Advances in neural information processing systems_ , vol. 33, pp. 12 449–12 460, 2020.
  * [6] R. Castellon, C. Donahue, and P. Liang, “Codified audio language modeling learns useful representations for music information retrieval,” _arXiv preprint arXiv:2107.05677_ , 2021.
  * [7] Q. Kong, Y. Cao, T. Iqbal, Y. Wang, W. Wang, and M. D. Plumbley, “Panns: Large-scale pretrained audio neural networks for audio pattern recognition,” _IEEE/ACM Transactions on Audio, Speech, and Language Processing_ , vol. 28, pp. 2880–2894, 2020.
  * [8] J. Turian, J. Shier, H. R. Khan, B. Raj, B. W. Schuller, C. J. Steinmetz, C. Malloy, G. Tzanetakis, G. Velarde, K. McNally _et al._ , “Hear: Holistic evaluation of audio representations,” in _NeurIPS 2021 Competitions and Demonstrations Track_. PMLR, 2022, pp. 125–145.
  * [9] Y. Zhao and J. Guo, “Musicoder: A universal music-acoustic encoder based on transformer,” in _International Conference on Multimedia Modeling_. Springer, 2021, pp. 417–429.
  * [10] H.-H. Wu, C.-C. Kao, Q. Tang, M. Sun, B. McFee, J. P. Bello, and C. Wang, “Multi-task self-supervised pre-training for music classification,” in _ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. IEEE, 2021, pp. 556–560.
  * [11] Y. Li, R. Yuan, G. Zhang, Y. MA, C. Lin, X. Chen, A. Ragni, H. Yin, Z. Hu, H. He _et al._ , “Map-music2vec: A simple and effective baseline for self-supervised music audio representation learning,” in _ISMIR late braking demo_ , 2022.
  * [12] P. Dhariwal, H. Jun, C. Payne, J. W. Kim, A. Radford, and I. Sutskever, “Jukebox: A generative model for music,” _arXiv preprint arXiv:2005.00341_ , 2020.
  * [13] Y. Li, R. Yuan, G. Zhang, Y. Ma, C. Lin, X. Chen, A. Ragni, H. Yin, Z. Hu, H. He _et al._ , “Large-scale pretrained model for self-supervised music audio representation learning,” in _Digital Music Research Network (DMRN) workshop_ , 2022.
  * [14] W.-N. Hsu, B. Bolte, Y.-H. H. Tsai, K. Lakhotia, R. Salakhutdinov, and A. Mohamed, “Hubert: Self-supervised speech representation learning by masked prediction of hidden units,” _IEEE/ACM Transactions on Audio, Speech, and Language Processing_ , vol. 29, pp. 3451–3460, 2021.
  * [15] J. Spijkervet and J. A. Burgoyne, “Contrastive learning of musical representations,” _arXiv preprint arXiv:2103.09410_ , 2021.
  * [16] D. Yao, Z. Zhao, S. Zhang, J. Zhu, Y. Zhu, R. Zhang, and X. He, “Contrastive learning with positive-negative frame mask for music representation,” in _Proceedings of the ACM Web Conference 2022_ , 2022, pp. 2906–2915.
  * [17] A. T. Liu, S.-w. Yang, P.-H. Chi, P.-c. Hsu, and H.-y. Lee, “Mockingjay: Unsupervised speech representation learning with deep bidirectional transformer encoders,” in _ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. IEEE, 2020, pp. 6419–6423.
  * [18] M. Ravanelli, J. Zhong, S. Pascual, P. Swietojanski, J. Monteiro, J. Trmal, and Y. Bengio, “Multi-task self-supervised learning for robust speech recognition,” in _ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. IEEE, 2020, pp. 6989–6993.
  * [19] A. Saeed, D. Grangier, and N. Zeghidour, “Contrastive learning of general-purpose audio representations,” in _ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. IEEE, 2021, pp. 3875–3879.
  * [20] L. Wang, P. Luc, Y. Wu, A. Recasens, L. Smaira, A. Brock, A. Jaegle, J.-B. Alayrac, S. Dieleman, J. Carreira _et al._ , “Towards learning universal audio representations,” in _ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. IEEE, 2022, pp. 4593–4597.
  * [21] A. Baevski, W.-N. Hsu, Q. Xu, A. Babu, J. Gu, and M. Auli, “Data2vec: A general framework for self-supervised learning in speech, vision and language,” in _International Conference on Machine Learning_. PMLR, 2022, pp. 1298–1312.
  * [22] A. Ragano, E. Benetos, and A. Hines, “Learning music representations with wav2vec 2.0,” _arXiv preprint arXiv:2210.15310_ , 2022.
  * [23] M. C. McCallum, F. Korzeniowski, S. Oramas, F. Gouyon, and A. F. Ehmann, “Supervised and unsupervised learning of audio representations for music understanding,” in _ISMIR_ , 2022.
  * [24] J.-B. Grill, F. Strub, F. Altché, C. Tallec, P. Richemond, E. Buchatskaya, C. Doersch, B. Avila Pires, Z. Guo, M. Gheshlaghi Azar _et al._ , “Bootstrap your own latent-a new approach to self-supervised learning,” _Advances in neural information processing systems_ , vol. 33, pp. 21 271–21 284, 2020.
  * [25] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, “Attention is all you need,” _Advances in neural information processing systems_ , vol. 30, 2017.
  * [26] E. Law, K. West, M. I. Mandel, M. Bay, and J. S. Downie, “Evaluation of algorithms using games: The case of music tagging.” in _ISMIR_. Citeseer, 2009, pp. 387–392.
  * [27] D. Bogdanov, M. Won, P. Tovstogan, A. Porter, and X. Serra, “The mtg-jamendo dataset for automatic music tagging.” ICML, 2019.
  * [28] F. Korzeniowski and G. Widmer, “End-to-end musical key estimation using a convolutional neural network,” in _2017 25th European Signal Processing Conference (EUSIPCO)_. IEEE, 2017, pp. 966–970.
  * [29] P. Knees, Á. Faraldo Pérez, H. Boyer, R. Vogl, S. Böck, F. Hörschläger, M. Le Goff _et al._ , “Two data sets for tempo estimation and key detection in electronic dance music annotated from user corrections,” in _Proceedings of the 16th International Society for Music Information Retrieval Conference (ISMIR); 2015 Oct 26-30; Málaga, Spain.[Málaga]: International Society for Music Information Retrieval, 2015\. p. 364-70._ International Society for Music Information Retrieval (ISMIR), 2015.
  * [30] C. Raffel, B. McFee, E. J. Humphrey, J. Salamon, O. Nieto, D. Liang, D. P. Ellis, and C. C. Raffel, “Mir_eval: A transparent implementation of common mir metrics.” in _ISMIR_ , 2014, pp. 367–372.
  * [31] G. Tzanetakis and P. Cook, “Musical genre classification of audio signals,” _IEEE Transactions on speech and audio processing_ , vol. 10, no. 5, pp. 293–302, 2002.
  * [32] C. Kereliuk, B. L. Sturm, and J. Larsen, “Deep learning and music adversaries,” _IEEE Transactions on Multimedia_ , vol. 17, no. 11, pp. 2059–2071, 2015.
  * [33] M. Soleymani, M. N. Caro, E. M. Schmidt, C.-Y. Sha, and Y.-H. Yang, “1000 songs for emotional analysis of music,” in _Proceedings of the 2nd ACM international workshop on Crowdsourcing for multimedia_ , 2013, pp. 1–6.
  * [34] J. Engel, C. Resnick, A. Roberts, S. Dieleman, M. Norouzi, D. Eck, and K. Simonyan, “Neural audio synthesis of musical notes with wavenet autoencoders,” in _International Conference on Machine Learning_. PMLR, 2017, pp. 1068–1077.
  * [35] J. Wilkins, P. Seetharaman, A. Wahl, and B. Pardo, “Vocalset: A singing voice dataset.” in _ISMIR_ , 2018, pp. 468–474.
  * [36] Y. Yamamoto, J. Nam, and H. Terasawa, “Deformable cnn and imbalance-aware feature learning for singing technique classification,” _arXiv preprint arXiv:2206.12230_ , 2022.
  * [37] S. Böck, F. Krebs, and G. Widmer, “Joint beat and downbeat tracking with recurrent neural networks.” in _ISMIR_. New York City, 2016, pp. 255–261.
  * [38] S. Böck, F. Korzeniowski, J. Schlüter, F. Krebs, and G. Widmer, “madmom: a new Python Audio and Music Signal Processing Library,” in _Proceedings of the 24th ACM International Conference on Multimedia_ , Amsterdam, The Netherlands, 10 2016, pp. 1174–1178.
  * [39] U. Marchand and G. Peeters, “Swing ratio estimation,” in _Digital Audio Effects 2015 (Dafx15)_ , 2015.
  * [40] Q. Huang, A. Jansen, L. Zhang, D. P. Ellis, R. A. Saurous, and J. Anderson, “Large-scale weakly-supervised content embeddings for music recommendation and tagging,” in _ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. IEEE, 2020, pp. 8364–8368.
  * [41] J. Lee, J. Park, K. L. Kim, and J. Nam, “Samplecnn: End-to-end deep convolutional neural networks using very small filters for music classification,” _Applied Sciences_ , vol. 8, no. 1, p. 150, 2018.
  * [42] M. Modrzejewski, P. Szachewicz, and P. Rokita, “Transfer learning with deep neural embeddings for music classification tasks,” in _Artificial Intelligence and Soft Computing: 21st International Conference, ICAISC 2022, Zakopane, Poland, June 19–23, 2022, Proceedings, Part I_. Springer, 2023, pp. 72–81.
  * [43] M. Heydari, F. Cwitkowitz, and Z. Duan, “Beatnet: Crnn and particle filtering for online joint beat downbeat and meter tracking,” _arXiv preprint arXiv:2108.03576_ , 2021.
  * [44] P. Alonso-Jiménez, X. Serra, and D. Bogdanov, “Music representation learning based on editorial metadata from discogs,” in _ISMIR_ , 2022.


# Towards Cross-Cultural Analysis using Music Information Dynamics

###### Abstract

A music piece is both comprehended hierarchically, from sonic events to
melodies, and sequentially, in the form of repetition and variation. Music
from different cultures establish different aesthetics by having different
style conventions on these two aspects. We propose a framework that could be
used to quantitatively compare music from different cultures by looking at
these two aspects.

The framework is based on an Music Information Dynamics model, a Variable
Markov Oracle (VMO), and is extended with a variational representation
learning of audio. A variational autoencoder (VAE) is trained to map audio
fragments into a latent representation. The latent representation is fed into
a VMO. The VMO then learns a clustering of the latent representation via a
threshold that maximizes the information rate of the quantized latent
representation sequence. This threshold effectively controls the sensibility
of the predictive step to acoustic changes, which determines the framework’s
ability to track repetitions on longer time scales. This approach allows
characterization of the overall information contents of a musical signal at
each level of acoustic sensibility.

Our findings under this framework show that sensibility to subtle acoustic
changes is higher for East-Asian musical traditions, while the Western works
exhibit longer motivic structures at higher thresholds of differences in the
latent space. This suggests that a profile of information contents, analyzed
as a function of the level of acoustic detail can serve as a possible cultural
characteristic.

##  1 Introduction

The music information retrieval (MIR) community in recent years is encouraging
more MIR tasks to be applied to non-western music [1, 2, 3], with comparative
analysis between music from different cultures or cross-cultural music
analysis becoming a rapid developing research area. In this paper we prose a
framework extending from a previous work done in [4] on comparing music
recordings from different cultures using a computational framework of Music
Information Dynamics. The proposed framework aims at providing quantitative
measurements of musical structure based on information theoretical
considerations. In this paper we show how this methods could be applied to
show stylistic differences between music from different cultures.

In [5], hand-crafted audio features and traditional machine learning
techniques are used to construct a similarity measurement to find outliers
among music examples from various cultural origins and analyze their
characteristics. In [6], hand-crafted vocal features are devised to study
different singing styles from different cultures, and in [7], the same work is
extended to classify concert non-western music recordings. Various non western
or folk music corpus are evaluated and compared in [8]. To the extent of the
authors’ knowledge, most of the previous works on cross-cultural comparative
music analysis used hand-crafted audio features and traditional machine
learning techniques, and none of the works attempted to model how temporality
of music is realized differently in different cultures. It is a well
established universal observation that music is an temporal art form [9] that
is constructed via sequential relationships, repetitions and variations of
musical elements [10, 11], realized differently in different cultures.

The previous paragraph brings us to the core motivation of the work proposed
in this paper, that is, a music piece is simultaneously comprehended
hierarchically and sequentially. It is hierarchical in the sense that, a basic
“sound unit” could be identified from timbral variations to microtonal
expressions to pitched events in a rigid tuning system. The comprehension is
sequential since it is the temporal relationships of these sound units that
forms larger/longer musical concepts such as phrases, progressions,
repetitions and variations. Given the above discussion, we argue that, to
compare music from different cultures quantitatively, a proposed framework
must takes both hierarchical and sequential relationships into considerations,
to reflect the style and aesthetic differences among them.

We simplify the concurrences of hierarchical and sequential comprehension on a
music piece by dividing the process into two steps. In the first step, a
compressed representation is learned by a variational autoencoder (VAE) [12]
from the music signal to best reconstruct itself. Given an input in time-
frequency format, a VAE assumes little about either the timbral or tonal
structure of the input, which suits the purpose in our case to universally
model the “sound unit” with little presumption beforehand. Then in the second
step, to capture temporal relationships, we adopted the Variable Markov Oracle
(VMO) [13] that is capable of discovering repeated patterns from the input
signal. The VMO is motivated by Music Information Dynamics that maximizes an
approximated mutual information measurement that optimizes the balance between
repetitions and variations embedded in the input signal.

The paper is structured as follows. In section 2 we first describe the Music
Information Dynamics and the VMO algorithm. The framework including the VAE is
elaborated in section 3. Experiments on music examples from different cultures
are presented in section 4. Discussions and conclusions are in section 5.

##  2 Music information dynamics

Assuming the music signal $X=x[n]$ is encoded into a sequence of “sound units”
$Z=z[n]$, with $n$ denotes discrete time step $n$. We would like to
algorithmically discover the sequential structure of $Z$, and be able to
present the structures quantitatively. Music Information Dynamics (MID)[14,
15, 16] provides a theoretical framework utilizes mutual information between
past and present observations to model the predictability of the signal. The
advantage of adopting MID is that it optimizes or calculates an information
theoretic measurements on the input sequence $Z$ and is agnostic of specific
sequence related applications, such as motifs discovery or structure
segmentation. MID was shown to be important for understanding human perception
of music in terms of anticipation and predictability [14, 15].

An efficient formal method for studying MID for sequence $Z[n]$ is the
Information Rate (IR) that considers the relation between the present
measurement $Z=z[n]$ and it’s past
$\overleftarrow{Z}=z[1],z[2],\dots,z[n],\dots,z[N]$, formally defined as the
maximum of mutual information over different quantized level of the sequence
$S=Q(Z)$

$\displaystyle IR(Z)$
$\displaystyle=\max_{Q:S=Q(Z)}I(Q(Z),Q\overleftarrow{(Z)})$ (1)
$\displaystyle=H(S)-H(S|\overleftarrow{S})$ (2)

According to this measure, the maximal value of IR is obtained when the
difference between the uncertainty of $H(S)$ and predictability
$H(S|\overleftarrow{S})$ is at its greatest, meaning that there is a balance
between variation and predictability. Quantization $Q(Z)$ is needed due to the
need to detect inexact repetitions in the sequence $Z$, which in turn
signifies the allowed level of similarities between observations in $Z$, or
the amount of “sound unit” details that is significant when comparing the
present to the past.

One of the realization of MID as an algorithm is the Variable Markov Oracle
(VMO) [13]. Variable Markov Oracle (VMO) is a generative machine learning
method that combines lossy compression with the Factor Oracle (FO) string
matching algorithm[17]. VMO is capable of finding embedded linkages between
samples along the multivariate time series and enables tracking and comparison
between time-series using a Viterbi-like dynamic programming algorithm. In
order to operate on such multivariate time series data, _VMO_ symbolizes a
signal $Z$ sampled at time $n$ into a symbolic sequence
$S=s[1],s[2],\dots,s[n],\dots,s[N]$, with $M$ states and with observation
frame $z[n]$ labeled by $s[n]$ whose value is one of the symbols in a finite
sized alphabet $\Sigma$. The labels are formed by following suffix links along
the states in the suffix tree structure constructed by the VMO algorithm.

The essential step for the construction of VMO is finding a threshold value,
$\theta$ that partitions the space of features into categories. The threshold
$\theta$ is used to determine if the incoming $z[n]$ is similar to one of the
frames following the suffix link started at $n-1$. VMO assigns two frames,
$z[i]$ and $z[j]$, the same label $s[i]=s[j]\in\Sigma$ if
$||z[i]-z[j]||\leq\theta$. In extreme cases, setting $\theta$ too low leads to
VMO assigning different labels to every frame in $Z$ and setting $\theta$ too
high leads to VMO assigning the same label to every frame in $Z$. As a result,
both extreme cases are incapable of capturing any temporal structures
(repeated suffixes) of the time series. To find the optimal threshold
$\theta$, the aforementioned IR is approximated by a deterministic notion of a
compression algorithm $C(\cdot)$ that is used as a computational measure of
time series complexity[18]. This allows estimation of the mutual information
by replacing the the entropy term $H(\cdot)$ in with compression gain for the
best quantization, searching over possible values of the threshold $\theta$
(Eq. 3).

$IR(Z)=\max_{\theta,s[n]\in\Sigma_{\theta}}[C(s[n])-C(s[n]|\overleftarrow{S})].$
(3)

Due to space consideration, we leave out the VMO construction and IR
optimization algorithms and refer the readers to [19, 13]. It should be noted
that the alphabet out of the quantization is constructed dynamically, as new
labels can be added when an input sample cannot be assigned to one of the
existing clusters of samples already labeled by existing labels. We will
denote the resulting alphabet for a given $\theta$ as $\Sigma_{\theta}$.

##  3 Methodology

In this section we describe the comparative framework by introducing each
component. Given a music recording, a time-frequency representation $X$ with
$N$ frames is first extracted and fed into the VAE to learn a compressed
representation $Z$ for each frame. Then $Z$ is quantized and indexed by the
VMO algorithm given a range of thresholds. Two quantitative results are
obtained via VMO. The first is an IR profile by plotting the IR values as a
function of the threshold. The second is motif discovery results obtained by
the optimal threshold $\theta$ that returns the highest IR value. Comparisons
between recordings could then be made by examine these two outputs.

###  3.1 Representation Learning

In this work, as discussed above, we first need a representation learning step
to extract the “sound units” from the audio signal. The high dimensional and
continuous nature of audio signals creates a challenge for finding an
efficient latent representation for music. Several works have utilized the VAE
models for finding disentangled representations in audio [20] and modeling
audio containing speech [21]. In this work we use the variational autoencoder
(VAE) [12] to map standard time-frequency representations into latent
variables $Z$ that captures the intrinsic sonic/tonal structures of the sound.
A normal autoencoder is a deep learning structure that contains an encoder and
a decoder, each of them comprised of trainable non-linear transformations made
by neural networks. During the training process, an input is encoded into a
latent vector, also known as an embedding, and an output is then decoded from
the same latent vector. The goal of training an autoencoder is to make the
input and output as similar as possible, while trying to obtain an embedding
that has some desired properties, such as dimensionality reduction, sparseness
and so on. Broadly speaking, an autoencoders is an unsupervised learning
algorithm, since the goal is to reconstruct itself by learning a compressed
representation.

A VAE explicitly constrains the latent variables $Z$ so that they are
distributed according to a prior $p(z)$. The input $X$ and latent code $Z$ can
then be seen as random variables $Z\sim p(Z),X\sim p(X|Z)$. The VAE consists
of an encoder probability $q_{\lambda}(Z|X)$, which approximates the posterior
probability $p(Z|X)$, and a decoder probability $p_{\theta}(X|Z)$, which
parameterizes the likelihood $p(X|Z)$. In practice, the approximate posterior
and likelihood distributions are parameterized by the weights the neural
networks. Posterior inference is done by minimizing the KL divergence between
the encoder and the true posterior. It can be proved that this optimization
problem is the same as maximizing the evidence lower bound (ELBO):

$\begin{split}ELBO&=E[\log p_{\theta}(X|Z)]-KL(q_{\lambda}(Z|X)||p(Z))\\\
&\leq\log p(X)\end{split}$ (4)

###  3.2 Temporal Structure Discovery

After a sequence of learned representation $Z$ is obtained from the VAE, the
VMO is applied on $Z$ to discover the embedded repetitive structure. As
discussed above, a VMO captures the optimal temporal structure of a signal by
adapting the threshold $\theta$ to distinguish between frames that maximizes
the IR value. This tuning can be considered as a way to extract the most
balanced repetitions and variations from the signal given a trade-off of
tolerance to imprecise matching between repeating signal segment. To put it
differently, the IR value at each threshold represents that total IR at that
level of detail. If then the IR values are plotted as a function of
thresholds, we obtain an “IR profile” that shows how sensitive the IR of the
signal is to the level of quantization. Motifs or repeated patterns and their
statistics from the quantized sequence $S$ from $Z$ obtained via VMO is also
used to compare among different music recordings. The motif discovery
algorithm based on VMO is documented in [13]. In short, the VMO motif
discovery algorithm is a repeated sub-string finding algorithm given a suffix
tree on the original string.

##  4 Experiments

In our experiments, the audio recording is sampled at 22050Hz and then a time-
frequency representation $X$ is obtained by an STFT with a window length of
2048 and a hop length of 256 frames. The spectrogram is then split into short
and non-overlapping frames with frame size of 24, resulting in an input
representation with dimensions $1024\times 24$. The encoder and decoder of our
VAE have a symmetrical structure consisting of convolutional layers and linear
layers. Full details for the architecture parameters are available in Appendix
A. The reconstruction loss function is the mean-squared error between the
input and reconstructed spectrograms, and the whole VAE training loss combines
the reconstruction loss and the KL divergence loss. For the experiment, we
also extracted hand-crafted features such as MFCCs and Chroma features with
the same STFT parameters. The MFCCs are calculated by 128 bands and we take
the first 19 MFCCs omitting the 1st coefficient. Both Chroma features and
MFCCs are mean-aggregated every 24 frames to have the same time resolution as
the input spectrogram frames to the VAE. To train a VAE, we use a batch size
of 50 with total 250 epochs optimized by ADAM [22].

###  4.1 Choice of repertoire

We performed comparative analysis on four musical works for the flute, two
from Western cultures and two from East-Asian traditions. The Western works
considered here are Bach, Partita in A Minor, BWV 1013 and Telemann: Fantasia
No. 3 in B minor. The East-Asian works are one Shakuhachi traditional Japanese
piece and one Nanguan classical Chinese music performed on a Dongxiao flute.

The duration of the pieces were between 4-5 minutes, except for the Bach
Partita that was around 9 minutes in duration. This particular choice of
repertoire is intentionally limited to instruments of similar timbre and
monophonic music material, while at the same time still allowing for different
levels of expressive inflections that are typical to different cultures. In
order to test for differences between different types of flutes, we compared
two performances of Teleman’s Fantasia, one performed on a modern flute
(traverse metal flute) and the other performed on a recorder (wooden beak
flute). The recorder, shakuhachi and dongxiao flutes are end blown, but unlike
the recorder, where the player blows into an internal whistle like duct, the
Shakuhachi and Dongxiao players blow across the flute, which results in
different levels of pitch and breath control.

Figure 1: Information Rate profiles for the three VAE methods, MFCCs, and
Chroma features, for all five flute pieces

###  4.2 Analysis Method

As mentioned earlier, the analysis was done in two separate steps, the first
being a feature extraction that was done by representation learning using VAE,
followed by a second step of temporal structure analysis using VMO. In order
to get a better insight into the role of learned representations in musical
information dynamic (MID), we used Mel-frequency cepstral coefficients (MFCC)
and Chroma features111By librosa https://github.com/librosa/librosa as
baselines for comparing MID analysis between hand-crafted features and learned
representations. Musically speaking, MFCC and Chroma correspond to timbral and
tonal properties of the music recordings, as the MFCCs capture the broad
spectral shape of the timbre of the instrument, while Chroma features wrap the
frequency analysis into bins corresponding to the Western twelve tone system.
In a sense, these hand-crafted features incorporate prior cultural assumptions
as they are designed to capture different aspects of the acoustic signal that
might be prevalent in certain musical practices more than in others.
Specifically, the Chroma features are designed so as to be invariant to
delicate timbral variations by aggregating the spectral features into $12$
bins that correspond to well defined pitches that are predominantly used in
Western musical culture. As such, it is natural to consider if a learned
representation (on the task of reconstructing itself) can be agnostic to
cultural-biases. Furthermore, by training on different sets of the recordings,
it is possible to experiment if learning either a piece, a cultural, or an
instrument specific representations. Accordingly, in our experiments we
compared three different types of VAE representations, one that was trained on
all musical examples across cultures (ALL-method), one that was trained on the
same musical piece that VMO analyzed (SELF-method), and one that was trained
on all of the examples excluding the specific piece that was analyzed by the
VMO (EXCLUDE-method).

###  4.3 IR Profiles

For each of the five music pieces (Bach, Telemann flute, Telemann recorder,
Shakuhachi and Dongxiao), we apply the three (ALL, SELF, and EXCLUDE) methods
of training VAE, and extract MFCCs and Chroma features on it. Each of these
conditions resulted in a sequence $Z$ that we feed it into the VMO algorithm,
and calculate a IR value for a range of threshold to obtain an IR profile.
Cosine distance is used as the distance function in $||z[i]-z[j]||\leq\theta$
for each $\theta$ value.

The result IR profiles are plotted in Figure 1. In each subplot, IR value is
on the vertical axis and thresholds are on the horizontal axis. It is obvious
that Western pieces tend to have a large peak at higher thresholds, while
East-Asian examples tend to have higher IR at lower threshold values.
Implicating that the East-Asian examples utilize more subtle timbral variation
to create phrases and expressions, then build repetition and structures on top
of them. For Western examples, this observation corresponds to the fact that
these pieces are more tonal driven.

###  4.4 Motif Discovery

For each of the five pieces, we also apply the VMO motif discovery algorithm
to obtain a list of repeated sub-sequences. In this case, we only return the
motifs found via the $\theta$ that maximized the IR values(The red vertical
lines in Figure 1). For space consideration, we choose Bach, Telemann Flute,
Dongxiao and only the SELF-method to demonstrate the results. The results are
depicted in Figure 2. In each subplot, the horizontal axis is the time step
and the vertical indices refers to the found motifs. Each horizontal black
line represent an occurrence of a single motif, and the lines on the same
vertical positions are repetitions of one another. The length of lines
represents the length of the motifs. The ordering of the motifs is from the
end of the piece at the top, to the beginning at the bottom. The ordering was
done for plotting conveniences. By looking at the plots it is also noticeable
that both Western examples have longer motifs and are mostly in the tonal
space, while the dongxiao piece has much shorter motifs and the similar amount
of motifs can be found either via timbre or pitch information.

###  4.5 Representations

By looking at the results in section 4.3 and Figure 1, the observation is that
the VAE feature learning is invariant to the three training methods
experimented. The results from SELF, EXCLUDE and ALL are all relatively
consistent with each other. In a way this result is not surprising given that
all the pieces are flute pieces. On the differences between hand-crafted
features and learned representations, by looking at the results from section
4.3 and 4.4, we can verify that although the results from learned
representations are less distinguishable compared to hand-crafted features (IR
peaks between pieces are closer), but they are still consistent with the
results from hand-crafted features. Also by looking at the motif discovery
results, we can observe that although the VAE is not trained toward any
indication of temporal relationships between frames in $X$, we can still find
meaningful motifs on the resulted $Z$ sequence, implicating VAE’s ability as a
universal encoder. The advantage of using learned representations is still
that it posses less algorithmic biases.

Figure 2: Motifs found in three flute pieces using the best VMO for $\text{VAE}_{self}$, Chroma features, and MFCCs. Figure 3: Mean and distribution of the motif lengths for the VAE, Chroma, and MFCC features Feature | F-Value | p-value
---|---|---
$VAE_{all}$ | 20.859 | $5.499\times 10^{-6}$
$VAE_{self}$ | 58.169 | $5.439\times 10^{-14}$
$VAE_{exclude}$ | 49.190 | $4.354\times 10^{-12}$
MFCC | 31.082 | $4.150\times 10^{-8}$
Chroma | 316.287 | $4.179\times 10^{-63}$
Table 1: ANOVA test results comparing the average sequence length in Eastern
and Western music

##  5 Discussion and Conclusion

In order to obtain more quantitative estimate of the differences in motif
structure among the different pieces, the mean and overall distribution of
motifs found at the last step are summarized in Figure 3. Also the ANOVA tests
are summarized in Table 1. We found there to be a significant difference in
the mean motif length between our selections of East-Asian and Western pieces.
Further statistical analysis can be found in Appendix B.These results indicate
that for Western works the VMO algorithm tends to find more structure at the
higher threshold levels, thus favoring coarser structure or units that
quantize larger differences between the element in $z$ under one label. This
results in significantly longer motifs for all Western pieces. It is
interesting to observe that Telemann piece performed with a recorder has a
somewhat larger peak at a lower threshold. We interpret this as an indication
that there is increased structure at the finer sonic details, that might be
related to the nature of the recorder instrument. It should be noted that the
method of motif discovery is based on a single VMO at the optimal threshold.
It is possible that multiple peaks of comparable level in terms of their IR
value would appear for a particular piece. In such a case, it is possible to
interpret such situation as indication that significant musical structure
exists at different threshold levels. In such case it is not clear which of
the thresholds is preferable for performing the motif analysis. In other
words, performing syntagmatic analysis (motif finding) depends on the choice
of the aesthetic criteria (IR optimum). Our experience shows that lower
thresholds that correspond to finer acoustic differences tend to result in
shorter motifs. For instance, analysis of the Telemann piece performed with
the recorder at a low threshold value that corresponds to the secondary peak
results in significantly shorter motifs that is comparable to the lengths
found in the Shakuhachi and Dongxiao recordings.

The next steps are first to examine the different learned $Z$ under different
configurations to understand if we can find meaningful insight on the
differences between music from different cultures on the sound level, and how
much a learned representation (such as the one learned by a VAE) could be
generalized. Then we would like to explore the possibility of combining the
two steps, representation learning and temporal structure discovery, into one
holistic model, to closer mimic the concurrences of hierarchical and
sequential comprehension of a music piece. In parallel we will continue to
experiment on different families of instruments and music from different
cultures.

It is also important to realize that the retrieved motifs using VMO, which is
a technique fundamental to the analysis presented in this paper, might differ
from other motif analysis methods, such as those in MIR challenges for motif
analysis. In [23] the VMO motif analysis results were compared to selected
musical motif discovery systems and showed SOTA performance at the time of the
writing (2015). It is possible that other motif techniques would render
different results that could be then compared based on their musical
significance. To the best of our knowledge, there is little evidence of
analyzing non-Western music of the type we discuss in this paper in terms of
generalized motifs. We consider any sequence of timbral inflections, as
captured by their embeddings and clustering, as motifs. These non-conventional
motif maybe not "musical" motifs in the traditional sense, but rather
"statistical" motifs as they represent sequences of audio features that have
salient repetition statistics. Accordingly, the use of the term "motif" should
be read in a somewhat ambiguous way denoting a structural musical element over
time, rather then engaging in musicological analysis of the
correctness/relevance/meaningfulness of the discovered motifs.

The code with complete graphs can be found in the code
repository222https://github.com/Origamijr/Cross-Cultural-Information-Dynamics.

###  5.1 Cultural Disclaimer

The analysis of the four pieces presented in this paper does not shed light
upon the vastness of Western music and the larger vastness of East Asian
music, so the results presented here should not be interpreted as
characteristics of the whole music cultures. To establish more solid knowledge
of structural and timbral aspects of even just Japanese flute repertory,
repertoires for folk shinobue or theatrical nohkan should be included.
Equally, nanguan might be more representative of the regional style from
South-East China, which is very different from flute music from Nothern China
or Korea. This could be compared with the very music for Daegeum that has
strong characteristics in terms of timbre and structure. Equally, historically
speaking, Telemann and Bach are only minimal representatives of central
European late Baroque music, which is very differnt from romantic, or avant-
garde music for flute, as well as other folk flute traditions in Europe.
Therefore, the results of these experiments should only be considered as an
initial pilot and a limited use case for a method which could be applied to
way larger corpora in order to shed light upon such cultural differences.

## References

  * [1] T. Lidy, C. N. Silla Jr, O. Cornelis, F. Gouyon, A. Rauber, C. A. Kaestner, and A. L. Koerich, “On the suitability of state-of-the-art music information retrieval methods for analyzing, categorizing and accessing non-western and ethnic music collections,” _Signal Processing_ , vol. 90, no. 4, pp. 1032–1048, 2010.
  * [2] X. Serra, “A multicultural approach in music information research,” in _Klapuri A, Leider C, editors. ISMIR 2011: Proceedings of the 12th International Society for Music Information Retrieval Conference; 2011 October 24-28; Miami, Florida (USA). Miami: University of Miami; 2011._ International Society for Music Information Retrieval (ISMIR), 2011.
  * [3] ——, “The computational study of a musical culture through its digital traces,” _Acta Musicologica_ , vol. 89, no. 1, pp. 24–44, 2017.
  * [4] S. Dubnov, K. Burns, and Y. Kiyoki, _Cross-Cultural Multimedia Computing: Semantic and Aesthetic Modeling_. Springer, 2016.
  * [5] M. Panteli, E. Benetos, and S. Dixon, “A computational study on outliers in world music,” _Plos one_ , vol. 12, no. 12, p. e0189399, 2017.
  * [6] M. Panteli, R. Bittner, J. P. Bello, and S. Dixon, “Towards the characterization of singing styles in world music,” in _2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. IEEE, 2017, pp. 636–640.
  * [7] A. Vidwans, P. Verma, and P. Rao, “Classifying cultural music using melodic features,” in _2020 International Conference on Signal Processing and Communications (SPCOM)_. IEEE, 2020, pp. 1–5.
  * [8] P. E. Savage, “An overview of cross-cultural music corpus studies,” 2018.
  * [9] J. Stambaugh, “Music as a temporal form,” _The Journal of Philosophy_ , vol. 61, no. 9, pp. 265–280, 1964.
  * [10] P. Kivy, _The fine art of repetition: Essays in the philosophy of music_. Cambridge University Press, 1993\.
  * [11] E. Campbell, _Boulez, music and philosophy_. Cambridge university press, 2010, vol. 27.
  * [12] D. P. Kingma and M. Welling, “An introduction to variational autoencoders,” _arXiv preprint arXiv:1906.02691_ , 2019.
  * [13] C.-i. Wang, J. Hsu, and S. Dubnov, “Music pattern discovery with variable markov oracle: A unified approach to symbolic and audio representations,” in _International Society for Music Information Retrieval Conference_ , 2015, pp. 176–182.
  * [14] S. Abdallah and M. Plumbley, “Information dynamics: Patterns of expectation and surprise in the perception of music,” _Connect. Sci_ , vol. 21, no. 2-3, pp. 89–117, Jun. 2009.
  * [15] S. Dubnov, “Musical information dynamics as models of auditory anticipation,” in _Machine Audition: Principles, Algorithms and Systems._ , W. Wang, Ed. IGI Global, 2011, pp. 371–397.
  * [16] M. T. Pearce, “Statistical learning and probabilistic prediction in music cognition: mechanisms of stylistic enculturation,” _Annals of the New York Academy of Sciences_ , vol. 1423, no. 1, p. 378, 2018.
  * [17] C. Allauzen, M. Crochemore, and M. Raffinot, “Factor oracle: A new structure for pattern matching,” in _SOFSEMÕ99: Theory and Practice of Informatics_. Springer, 1999, pp. 295–310.
  * [18] A. Lefebvre and T. Lecroq, “Compror: on-line lossless data compression with a factor oracle,” _Information Processing Letters_ , vol. 83, no. 1, pp. 1–6, 2002.
  * [19] S. Dubnov, G. Assayag, and A. Cont, “Audio oracle analysis of musical information rate,” in _Semantic Computing (ICSC), 2011 Fifth IEEE International Conference on_. IEEE, 2011, pp. 567–571.
  * [20] Y.-J. Luo, K. Agres, and D. Herremans, “Learning disentangled representations of timbre and pitch for musical instrument sounds using gaussian mixture variational autoencoders,” 2019.
  * [21] W.-N. Hsu, Y. Zhang, and J. Glass, “Learning latent representations for speech generation and transformation,” 2017.
  * [22] D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” in _3rd International Conference on Learning Representations, ICLR_ , 2015\.
  * [23] C.-i. Wang and S. Dubnov, “Pattern discovery from audio recordings by variable markov oracle: A music information dynamics approach,” in _Acoustics, Speech, and Signal Processing (ICASSP), 2015 IEEE International Conference on_. IEEE, 2015.

##  6 Appendix

##  Appendix A Audio VAE Architecture

Figure 4: Audio VAE Encoder (left) and Decoder (right)

As shown in Figure 4, we use a simple symmetrical convolutional VAE. Both the
encoder and decoder use 3 convolutional layers and max pooling. Fully
connected layers are used to produce the latent distribution parameters, and
to decode the latent representation into a downsampled image for decoding.

##  Appendix B East vs West Statistical Tests

The ANOVA test shown in Table 1 shows significant difference in the mean
pattern lengths between the select Eastern and Western music. Tables 2-6 show
a followup Tukey HSD analysis on the pattern lengths for all pairs of songs in
our selection.

Song Pair | Diff | p-adj | Reject
---|---|---|---
Bach, Dongxiao | -8.30 | 0.001 | True
Bach, Shakuhachi | -10.10 | 0.001 | True
Bach, Telemann F. | -5.53 | 0.081 | False
Bach, Telemann R. | -5.55 | 0.031 | True
Dongxiao, Shakuhachi | -1.80 | 0.900 | False
Dongxiao, Telemann F. | 2.77 | 0.801 | False
Dongxiao, Telemann R. | 2.75 | 0.754 | False
Shakuhachi, Telemann F. | 4.57 | 0.552 | False
Shakuhachi, Telemann R. | 4.55 | 0.504 | False
Telemann F., Telemann R. | -0.02 | 0.900 | False
Table 2: Tukey HSD Comparison of Means for $\text{VAE}_{all}$ Song Pair | Diff | p-adj | Reject
---|---|---|---
Bach, Dongxiao | -9.35 | 0.001 | True
Bach, Shakuhachi | -11.22 | 0.001 | True
Bach, Telemann F. | 0.01 | 0.900 | False
Bach, Telemann R. | -6.24 | 0.001 | True
Dongxiao, Shakuhachi | -1.87 | 0.900 | False
Dongxiao, Telemann F. | 9.36 | 0.001 | True
Dongxiao, Telemann R. | 3.11 | 0.481 | False
Shakuhachi, Telemann F. | 11.23 | 0.001 | True
Shakuhachi, Telemann R. | 4.98 | 0.205 | False
Telemann F., Telemann R. | -6.25 | 0.018 | True
Table 3: Tukey HSD Comparison of Means for $\text{VAE}_{self}$ Song Pair | Diff | p-adj | Reject
---|---|---|---
Bach, Dongxiao | -3.23 | 0.006 | True
Bach, Shakuhachi | -5.04 | 0.001 | True
Bach, Telemann F. | 6.55 | 0.001 | True
Bach, Telemann R. | -1.09 | 0.762 | False
Dongxiao, Shakuhachi | -1.81 | 0.595 | False
Dongxiao, Telemann F. | 9.78 | 0.001 | True
Dongxiao, Telemann R. | 2.14 | 0.308 | False
Shakuhachi, Telemann F. | 11.59 | 0.001 | True
Shakuhachi, Telemann R. | 3.95 | 0.017 | True
Telemann F., Telemann R. | -7.64 | 0.001 | True
Table 4: Tukey HSD Comparison of Means for $\text{VAE}_{exclude}$ Song Pair | Diff | p-adj | Reject
---|---|---|---
Bach, Dongxiao | -2.82 | 0.002 | True
Bach, Shakuhachi | -4.52 | 0.001 | True
Bach, Telemann F. | -3.70 | 0.001 | True
Bach, Telemann R. | 0.85 | 0.839 | False
Dongxiao, Shakuhachi | -1.70 | 0.430 | False
Dongxiao, Telemann F. | -0.89 | 0.894 | False
Dongxiao, Telemann R. | 3.66 | 0.002 | True
Shakuhachi, Telemann F. | 0.82 | 0.900 | False
Shakuhachi, Telemann R. | 5.37 | 0.001 | True
Telemann F., Telemann R. | 4.55 | 0.001 | True
Table 5: Tukey HSD Comparison of Means for MFCC Song Pair | Diff | p-adj | Reject
---|---|---|---
Bach, Dongxiao | -64.22 | 0.001 | True
Bach, Shakuhachi | -63.46 | 0.001 | True
Bach, Telemann F. | -40.82 | 0.001 | True
Bach, Telemann R. | -52.46 | 0.001 | True
Dongxiao, Shakuhachi | 0.76 | 0.900 | False
Dongxiao, Telemann F. | 23.40 | 0.001 | True
Dongxiao, Telemann R. | 11.77 | 0.123 | False
Shakuhachi, Telemann F. | 22.64 | 0.001 | True
Shakuhachi, Telemann R. | 11.01 | 0.318 | False
Telemann F., Telemann R. | -11.63 | 0.043 | True
Table 6: Tukey HSD Comparison of Means for Chroma