∎

11institutetext: J. Kim, J. Urbano, C. C. S. Liem & A. Hanjalic
22institutetext: Multimedia Computing Group
Department of Intelligent Systems
Faculty of Electrical Engineering, Mathematics and Computer Science
Delft University of Technology
Tel.: +31-15-27-87241
22email<EMAIL_ADDRESS>
# One Deep Music Representation to Rule Them All? ††thanks: This work has been
accepted to “Neural Computing and Applications: Special Issue on Deep Learning
for Music and Audio”.

A comparative analysis of different representation learning strategies

Jaehun Kim  Julián Urbano
Cynthia C. S. Liem  Alan Hanjalic

(Received: date / Accepted: date)

###### Abstract

Inspired by the success of deploying deep learning in the fields of Computer
Vision and Natural Language Processing, this learning paradigm has also found
its way into the field of Music Information Retrieval. In order to benefit
from deep learning in an effective, but also efficient manner, deep transfer
learning has become a common approach. In this approach, it is possible to
reuse the output of a pre-trained neural network as the basis for a new
learning task. The underlying hypothesis is that if the initial and new
learning tasks show commonalities and are applied to the same type of input
data (e.g. music audio), the generated deep representation of the data is also
informative for the new task. Since, however, most of the networks used to
generate deep representations are trained using a single initial learning
source, their representation is unlikely to be informative for all possible
future tasks. In this paper, we present the results of our investigation of
what are the most important factors to generate deep representations for the
data and learning tasks in the music domain. We conducted this investigation
via an extensive empirical study that involves multiple learning sources, as
well as multiple deep learning architectures with varying levels of
information sharing between sources, in order to learn music representations.
We then validate these representations considering multiple target datasets
for evaluation. The results of our experiments yield several insights on how
to approach the design of methods for learning widely deployable deep data
representations in the music domain.

###### Keywords:

Representation Learning Music Information Retrieval Multi-Task Learning

##  1 Introduction

In the Music Information Retrieval (MIR) field, many research problems of
interest involve the automatic description of properties of musical signals,
employing concepts that are understood by humans. For this, tasks are derived
that can be solved by automated systems. In such cases, algorithmic processes
are employed to map raw music audio information to humanly understood
descriptors (e.g. genre labels or descriptive tags). To achieve this,
historically, the raw audio would first be transformed into a _representation_
based on _hand-crafted features_ , which are engineered by humans to reflect
dedicated semantic signal properties. The feature representation would then
serve as input to various statistical or Machine Learning (ML) approaches
Casey et al. (2008).

The framing as described above can generally be applied to many applied ML
problems: complex real-world problems are abstracted into a relatively simpler
form, by establishing tasks that can be computationally addressed by automatic
systems. In many cases, the task involves making a prediction based on a
certain observation. For this, modern ML methodologies can be employed, that
automatically can infer the logic for the prediction directly from (a numeric
representation of) the given data, by optimizing an objective function defined
for the given task.

However, music is a multimodal phenomenon, that can be described in many
parallel ways, ranging from objective descriptors to subjective preference. As
a consequence, in many cases, while music-related tasks are well understood by
humans, it often is hard to pinpoint and describe where the truly ‘relevant’
information is in the music data used for the tasks, and how this properly can
be translated into numeric representations that should be used for prediction.
While research into such proper translations can be conducted per individual
task, it is likely that informative factors in music data will be shared
across tasks. As a consequence, when seeking to identify informative factors
that are not explicitly restricted to a single task, Multi-Task Learning (MTL)
is a promising strategy. In MTL, a single learning framework hosts multiple
tasks at once, allowing for models to perform better by sharing commonalities
between involved tasks Caruana (1997). MTL has been successfully used in a
range of applied ML works Bengio et al. (2013); Liu et al. (2015); Bingel and
Søgaard (2017); Li et al. (2015); Zhang et al. (2015, 2014); Kaiser et al.
(2017); Chang et al. (2017), also including the music domain Weston et al.
(2011); Aytar et al. (2016).

Following successes in the fields of Computer Vision (CV) and Natural Language
Processing (NLP), deep learning approaches have recently also gained
increasing interest in the MIR field, in which case _deep representations_ of
music audio data are directly learned from the data, rather than being hand-
crafted. Many works employing such approaches reported considerable
performance improvements in various music analysis, indexing and
classification tasks Hamel and Eck (2010); Boulanger-Lewandowski et al.
(2012); Schlüter and Böck (2014); Choi et al. (2016); van den Oord et al.
(2013); Chandna et al. (2017); Jeong and Lee (2016); Han et al. (2017).

In many deep learning applications, rather than training a complete network
from scratch, pre-trained networks are commonly used to generate deep
representations, which can be either directly adopted or further adapted for
the current task at hand. In CV and NLP, (parts of) certain pre-trained
networks Simonyan and Zisserman (2015); He et al. (2016); Szegedy et al.
(2015); Mikolov et al. (2013) have now been adopted and adapted in a very
large number of works. These ‘standard’ deep representations have typically
been obtained by training a network for a single learning task, such as visual
object recognition, employing large amounts of training data. The hypothesis
on why these representations are effective in a broader of spectrum of tasks
than they originally were trained for, is that _deep transfer learning (DTL)_
is happening: information initially picked up by the network is beneficial
also for new learning tasks performed on the same type of raw input data.
Clearly, the validity of this hypothesis is linked to the extent to which the
new task can rely on similar data characteristics as the task on which the
pre-trained network was originally trained.

Although a number of works deployed DTL for various learning tasks in the
music domainDieleman et al. (2011); Choi et al. (2017); van den Oord et al.
(2014); Liang et al. (2015), to our knowledge, however, transfer learning and
the employment of pre-trained networks are not as standard in the MIR domain
as in the CV domain. Again, this may be due to the broad and partially
subjective range and nature of possible music descriptions. Following the
considerations above, it may then be useful to combine deep transfer learning
with multi-task learning.

Indeed, in order to increase robustness to a larger scope of new learning
tasks and datasets, the concept of MTL also has been applied in training deep
networks for representation learning, both in the music domain Aytar et al.
(2016); Weston et al. (2011) and in general (Bengio et al., 2013, p. 2). As
the model learns several tasks and datasets in parallel, it may pick up
commonalities among them. As a consequence, the expectation is that a network
learned with MTL will yield robust performance across different tasks, by
transferring shared knowledge Caruana (1997); Bengio et al. (2013). A simple
illustration of the conceptual difference between traditional DTL and deep
transfer learning based on MTL (further referred to as _multi-task based deep
transfer learning (MTDTL))_ is shown in Fig. 1.

Figure 1: Simplified illustration of the conceptual difference between
traditional deep transfer learning (DTL) based on a single learning task
(above) and multi-task based deep transfer learning (MTDTL) (below). The same
color used for a learning and an target task indicates that the tasks have
commonalities, which implies that the learned representation is likely to be
informative for the target task. At the same time, this representation may not
be that informative to another future task, leading to a low transfer learning
performance. The hypothesis behind MTDTL is that relying on more learning
tasks increases robustness of the learned representation and its usability for
a broader set of target tasks.

The mission of this paper is to investigate the effect of conditions around
the setup of MTDTL, which are important to yield effective deep music
representations. Here, we understand an ‘effective’ representation to be a
representation that is suitable for a wide range of new tasks and datasets.
Ultimately, we aim for providing a methodological framework to systematically
obtain and evaluate such transferable representations. We pursue this mission
by exploring the effectiveness of MTDTL and traditional DTL, as well as
concatenations of multiple deep representations, obtained by networks that
were independently trained on separate single learning tasks. We consider
these representations for multiple choices of learning tasks and considering
multiple target datasets.

Our work will address the following research questions:

  * •

RQ1: Given a set of learning sources that can be used to train a network, what
is the influence of the number and type of the sources on the effectiveness of
the learned deep representation?

  * •

RQ2: How do various degrees of information sharing in the deep architecture
affect the effectiveness of a learned deep representation?

By answering the RQ1 we arrive at an understanding of important factors
regarding the composition of a set of learning tasks and datasets (which in
the remainder of this work will be denoted as _learning sources_) to achieve
an effective deep music representation, specifically on the number and nature
of learning sources. The answer to RQ2 provides insight in _how to choose the
optimal multi-task network architecture_ under a MTDTL context. For example,
in MTL, multiple sources are considered under a joint learning scheme, that
partially shares inferences obtained from different learning sources in the
learning pipeline. In MTL applications using deep neural networks, this means
that certain layers will be shared between all sources, while at other stages,
the architecture will ‘branch’ out into source-specific layers Caruana (1997);
Bingel and Søgaard (2017); Li et al. (2015); Zhang et al. (2015, 2014); Misra
et al. (2016); Aytar et al. (2016). However, investigation is still needed on
where in the layered architecture branching should ideally happen—if a
branching strategy would turn out beneficial in the first place.

To reach the aforementioned answers, it is necessary to conduct a systematic
assessment to examine relevant factors. For RQ1, we investigate different
numbers and combinations of learning sources. For RQ2, we study different
architectural strategies. However, we wish to ultimately investigate
effectiveness of the representation with respect to new, target learning tasks
and datasets (which in the remainder of this paper will be denoted by _target
datasets_). While this may cause combinatorial explosion with respect to
possible experimental configurations, we will make strategic choices in the
design and evaluation procedure of the various representation learning
strategies.

The scientific contribution of this work can be summarized as follows:

  * $\bullet$

We provide insight into the effectiveness of various deep representation
learning strategies under the multi-task learning context.

  * $\bullet$

We offer in-depth insight into ways to evaluate desired properties of a deep
representation learning procedure.

  * $\bullet$

We propose and release several pre-trained music representation networks,
based on different learning strategies for multiple semantic learning sources.

The rest of this work is presented as following: a formalization of this
problem, as well as the global outline of how learning will be performed based
on different learning tasks from different sources, will be presented in
Section 2. Detailed specifications of the deep architectures we considered for
the learning procedure will be discussed in Section 3. Our strategy to
_evaluate_ the effectiveness of different representation network variants by
employing various _target datasets_ will be the focus of Section 4.
Experimental results will be discussed in Section 5, after which general
conclusions will be presented in Section 6.

##  2 Framework for Deep Representation Learning

In this section, we formally define the deep representation learning problem.
As Fig. 2 illustrates, any domain-specific MTDTL problem can be abstracted
into a formal task, which is instantiated by a specific dataset with specific
observations and labels. Multiple tasks and datasets are involved to emphasize
different aspects of the input data, such that the learned representation is
more adaptable to different future tasks. The learning part of this scheme can
be understood as the MTL phase, which is introduced in Section 2.1.
Subsequently in Section 2.2, we discuss learning sources involved in this
work, which consist of various tasks and datasets to allow investigating their
effects on the transfer learning. Further, we introduce the label
preprocessing procedure that is applied in this work in Section 2.3, ensuring
that the learning sources are more regularized, such that their comparative
analysis is clearer.

(a) Multi-Task Transfer Learning in General Problem Domain

(b) Multi-Task Transfer Learning in Music Information Retrieval Domain

Figure 2: Schematic overview of what this work investigates. The upper scheme
illustrates a general problem solving framework in which multi-task transfer
learning is employed. The tasks $t\in\\{t_{0},t_{1},\cdots,t_{M}\\}$ are
derived from a certain problem domain, which are instantiated by datasets,
that often are represented as sample pairs of observations and corresponding
labels $(X_{t},y_{t})$. Sometimes, the original dataset is processed further
into simpler representation forms $(X_{t},z_{t})$, to filter out undesirable
information and noise. Once a model or system $f_{t}(X_{t})$ has learned the
necessary mappings within the learning sources, this knowledge can be
transferred to another set of target datasets, leveraging commonalities
already obtained by the pre-training. Below the general framework, we show a
concrete example, in which the broad MIR problem domain is abstracted into
various sub-problems with corresponding tasks and datasets.

###  2.1 Problem Definition

A machine learning problem, focused on solving a specific task $t$, can be
formulated as a minimization problem, in which a model function $f_{t}$ must
be learned that minimizes a loss function $\mathcal{L}$ for given dataset
$\mathcal{D}_{t}=\\{\,(x^{(i)}_{t},y^{(i)}_{t})\mid
i\in\\{1,\cdots,I\\}\,\\}$, comparing the model’s predictions given by the
input $x_{t}$ and actual task-specific learning labels $y_{t}$. This can be
formulated using the following expression:

$\hat{\theta}=\operatorname*{arg\,min}\;\mathbb{E}_{\mathcal{D}_{t}}\mathcal{L}(y_{t},f_{t}(x_{t};\theta))$
(1)

where $x_{t}\in\mathbb{R}^{d}$ is, traditionally, a hand-crafted
$d$-dimensional feature vector and $\theta$ is a set of model parameters of
$f$.

When deep learning is employed, the model function $f$ denotes a learnable
network. Typically, the network model $f$ is learned in an end-to-end fashion,
from raw data at the input to the learning label. In the speech and music
field, however, using true end-to-end learning is still not a common practice.
Instead, raw data is typically transformed first, before serving as network
input. More specifically, in the music domain, common input to function $f$
would be $X\in\mathbb{R}^{c\times{n}\times{b}}$, replacing the originally
hand-crafted feature vector $x\in\mathbb{R}^{d}$ from (1) by a time-frequency
representation of the observed music data, usually obtained through the Short-
Time Fourier Transform (STFT), with potential additional filter bank
applications (e.g. mel-filter bank). The dimensions $c$, $n$, $b$ indicate
channels of the audio signal, time steps, and frequency bins respectively.

If such a network still is trained for a specific single machine learning task
$t$, we can now reformulate (1) as follows:

$\hat{\theta}=\operatorname*{arg\,min}\;\mathbb{E}_{\mathcal{D}_{t}}\mathcal{L}(y_{t},f_{t}(X_{t};\theta)).$
(2)

In MTL, in the process of learning the network model $f$, different tasks will
need to be solved in parallel. In case of deep neural networks, this is
usually realized by having a network in which lower layers are shared for all
tasks, but upper layers are task-specific. Given $m$ different tasks $t$, each
having the learning label $y_{t}$, we can formulate the learning objective of
the neural network in a MTL scenario as follows:

$\hat{\theta}^{s},\hat{\theta}^{*}=\operatorname*{arg\,min}\;\mathbb{E}_{t\in{\mathcal{T}}}\mathbb{E}_{\mathcal{D}_{t}}\mathcal{L}(y_{t},f_{t}(X_{t};\theta^{s},\theta^{t}))$
(3)

Here, $\mathcal{T}=\\{t_{1},t_{2},...,t_{m}\\}$ is a given set of tasks to be
learned and $\theta^{*}=\\{\theta^{1},\theta^{2},...,\theta^{m}\\}$ indicates
a set of model parameters $\theta^{t}$ with respect to each task. Since the
deep architecture initially shares lower layers and branches out to task-
specific upper layers, the parameters of shared layers and task-specific
layers are referred to separately as $\theta^{s}$ and $\theta^{t}$,
respectively. Updates for all parameters can be achieved through standard
back-propagation. Further specifics on network architectures and training
configurations will be given in Section 3.

Given the formalizations above, the first step in our framework is to select a
suitable set $\mathcal{T}$ of learning tasks. These tasks can be seen as
multiple concurrent descriptions or transformations of the same input fragment
of musical audio: each will reflect certain semantic aspects of the music.
However, unlike the approach in a typical MTL scheme, solving multiple
specific learning tasks is actually not our main goal; instead, we wish to
learn an effective _representation_ that captures as many semantically
important factors in the low-level music representation as possible. Thus,
rather than using learning labels $y_{t}$, our representation learning process
will employ reduced learning labels $z_{t}$, which capture a reduced set of
semantic factors from $y_{t}$. We then can reformulate (3) as follows:

$\hat{\theta}^{s},\hat{\theta}^{*}=\operatorname*{arg\,min}\;\mathbb{E}_{t\in{\mathcal{T}}}\mathbb{E}_{\mathcal{D}_{t}}\mathcal{L}(z_{t},f_{t}(X_{t};\theta^{s},\theta^{t}))$
(4)

where $z_{t}\in\mathbb{R}^{k}$ is a $k$-dimensional vector that represents
reduced learning label for a specific task $t$. Each $z_{t}$ will be obtained
through task-specific factor extraction methods, as described in Section 2.3.

###  2.2 Learning Sources

In the MTDTL context, a training dataset can be seen as the ‘source’ to learn
the representation, which will be further transferred to the future ‘target’
dataset. Different learning sources of different nature can be imagined, that
can be globally categorized as _Algorithm_ or _Annotation_. As for the
_Algorithm_ category, by employing traditional feature extraction or
representation transformation algorithms, we will be able to automatically
extract semantically interesting aspects from input data. As for the
_Annotation_ category, these include different types of label annotations of
the input data by humans.

The dataset used as resource for our learning experiments is the Million Song
Dataset (MSD)Bertin-Mahieux et al. (2011). In its original form, it contains
metadata and precomputed features for a million songs, with several associated
data resources, e.g. considering Last.fm social tags and listening profiles
from the Echo Nest. While the MSD does not distribute audio due to copyright
reasons, through the API of the 7digital service, 30-second audio previews can
be obtained for the songs in the dataset. These 30-second previews will form
the source for our raw audio input.

Using the MSD data, we consider several subcategories of learning sources
within the _Algorithm_ and _Annotation_ categories; below, we give an overview
of these, and specify what information we considered exactly for the learning
labels in our work.

####  2.2.1 Algorithm

  * •

Self. The music track is the learning source itself; in other words, intrinsic
information in the input music track should be captured through a learning
procedure, without employing further data. Various unsupervised or auto-
regressive learning strategies can be employed under this category, with
variants of Autoencoders, including the Stacked Autoencoder Bengio et al.
(2006); Vincent et al. (2008), Restricted Boltzmann Machines (RBM) Smolensky
(1986), Deep Belief Networks (DBN) Hinton et al. (2006) and Generative
Adversarial Networks (GAN) Goodfellow et al. (2014). As another example within
this category, variants of the Siamese networks for similarity learning can be
considered Han et al. (2015); Arandjelovic and Zisserman (2017); Huang et al.
(2018).

In our case, we will employ the Siamese architecture to learn a metric that
measures whether two input music clips belong to the same track, or two
different tracks. This can be formulated as follows:

$\hat{\theta}^{self},\hat{\theta}^{s}=\operatorname*{arg\,min}\;\mathbb{E}_{X_{l},X_{r}\sim\mathcal{D}_{self}}\mathcal{L}(y_{self},f_{self}(X_{l},X_{r};\theta^{self},\theta^{s}))$
(5)

$y_{self}=\begin{cases}1,&\text{if }X_{l}\text{ and }X_{r}\text{ sampled from
same track}\\\ 0&\text{otherwise}\end{cases}$ (6)

where $X_{l}$ and $X_{r}$ are a pair of randomly sampled short music snippets
(taken from the 30-second MSD audio previews) and $f_{self}$ is a network for
learning a metric between given input representations in terms of the criteria
imposed by $y_{self}$. It is composed of one or more fully-connected layers
and one output layer with softmax activation. An global outline illustration
of our chosen architecture is given in Fig. 3. Further specifications of the
representation network and sampling strategies will be given in Section 3.

Figure 3: Siamese architecture adopted for the _self_ learning task. For
further details of the Representation Network, see Section 3.1 and Fig. 4.

  * •

Feature. Many algorithms exist already for extracting features out of musical
audio, or for transforming musical audio representations. By running such
algorithms on musical audio, learning labels are automatically computed,
without the need for soliciting human annotations. Algorithmically computed
outcomes will likely not be perfect, and include noise or errors. At the same
time, we consider them as a relatively efficient way to extract semantically
relevant and more structured information out of a raw input signal.

In our case, under this category, we use Beat Per Minute (BPM) information,
released as part of the MSD’s precomputed features. The BPM values were
computed by an estimation algorithm, as part of the Echo Nest API.

####  2.2.2 Annotation

  * •

Metadata. Typically, metadata will come ‘for free’ with music audio,
specifying side information, such as a release year, the song title, the name
of the artist, the corresponding album name, and the corresponding album cover
image. Considering that this information describes categorization facets of
the musical audio, metadata can be a useful information source to learn a
music representation. In our experiments, we use release year information,
which is readily provided as metadata with each song in the MSD.

  * •

Crowd. Through interaction with music streaming or scrobbling services, large
numbers of users, also designated as the crowd, left explicit or implicit
information regarding their perspectives on musical content. For example, they
may have created social tags, ratings, or social media mentionings of songs.
With many services offering API access to these types of descriptors, crowd
data therefore offers scalable, spontaneous and diverse (albeit noisy) human
perspectives on music signals.

In our experiments, we use social tags from
Last.fm111https://labrosa.ee.columbia.edu/millionsong/lastfm and user
listening profiles from the Echo Nest.

  * •

Professional. As mentioned in Casey et al. (2008), annotation of music tracks
is a complicated and time-consuming process: annotation criteria frequently
are subjective, and considerable domain knowledge and annotation experience
may be required before accurate and consistent annotations can be made.
Professional experts in categorization have this experience, and thus are
capable of indicating clean and systematic information about musical content.
It is not trivial to get such professional annotations at scale; however,
these types of annotations may be available in existing professional
libraries.

In our case, we use professional annotations from the Centrale Discotheek
Rotterdam (CDR), the largest music library in The Netherlands, holding all
music ever released in the country in physical and digital form in its
collection. The CDR collection can be digitally accessed through the online
Muziekweb222https://www.muziekweb.nl/ platform. For each musical album in the
CDR collection, genre annotations were made by a professional annotator,
according to a fixed vocabulary of 367 hierarchical music genres.

As another professional-level ‘description’, we adopted lyrics information per
each track, which is provided in Bag-of-Words format with the MSD. To filter
out trivial terms such as stop-words, we applied TF-IDFSalton and McGill
(1984).

  * •

Combination. Finally, learning labels can be derived from combinations of the
above categories. In our experiment, we used combination of artist information
and social tags, by making a bag of tags at the artist level as a learning
label.

Not all songs in the MSD actually include learning labels from all the sources
mentioned above. Clearly, it is another advantage of using MTL that one can
use such unbalanced datasets in a single learning procedure, to maximize the
coverage of the dataset. However, on the other hand, if one uses an unbalanced
number of samples across different learning sources, it is not trivial to
compare the effect of individual learning sources. We therefore choose to work
with a subset of the dataset, in which equal numbers of samples across
learning sources can be used. As a consequence, we managed to collect 46,490
clips of tracks with corresponding learning source labels. A 41,841 / 4,649
split was made for training and validation for all sources from both MSD and
CDR. Since we mainly focus on transfer learning, we used the validation set
mostly for monitoring the training, to keep the network from overfitting.

Table 1: Properties of learning sources. Identifier | Category | Data | Dimensionality | Preprocessing
---|---|---|---|---
self | Algorithm | Self | MSD - Track | 1 |
bpm | Feature | MSD - BPM | 1 | GMM
year | Annotation | Metadata | MSD - Year | 1 | GMM
tag | Crowd | MSD - Tag | 174,156 | pLSA
taste | Crowd | MSD - Taste | 949,813 | pLSA
cdr_tag | Professional | CDR - Tag | 367 | pLSA
lyrics | Professional | MSD - Lyrics | 5,000 | pLSA, TF-IDF
artist | Combination | MSD - Artist & Tag | 522,366 | pLSA
Table 2: Examples of Latent Topics extracted with pLSA from MSD social tags Topic | Strongest social tags
---|---
tag1 |  indie rock, indie, british, Scottish
tag2 |  pop, pop rock, dance, male vocalists
tag3 |  soul, rnb, funk, Neo-Soul
tag4 |  Melodic Death Metal, black metal, doom metal, Gothic Metal
tag5 |  fun, catchy, happy, Favorite

###  2.3 Latent Factor Preprocessing

Most learning sources are noisy. For instance, social tags include tags for
personal playlist management, long sentences, or simply typos, which do not
actually show relevant nuances in describing the music signal. The
algorithmically extracted BPM information also is imperfect, and likely
contains octave errors, in which BPM is under- or overestimated by a factor of
2. To deal with this noise, several previous works using the MSD Choi et al.
(2016, 2017) applied a frequency-based filtering strategy along with top-down
domain knowledge. However, this shrinks the available sample size. As an
alternative way to handle noisiness, several other previous works Lamere
(2008); Weston et al. (2011); Hamel et al. (2013); Law et al. (2010); van den
Oord et al. (2014, 2013) apply latent factor extraction using various low-rank
approximation models to preprocess the label information. We also choose to do
this in our experiments.

A full overview of chosen learning sources, their category, origin dataset,
dimensionality and preprocessing strategies is shown in Table 1. In most
cases, we apply probabilistic latent semantic analysis (pLSA), which extracts
latent factors as a multinomial distribution of latent topics Hofmann (1999).
Table 2 illustrates several examples of strong social tags within extracted
latent topics.

For situations in which learning labels are a scalar, non-binary value (BPM
and release year), we applied a Gaussian Mixture Model (GMM) to transform each
value into a categorical distribution of Gaussian components. In case of the
Self category, as it basically is a binary membership test, no factor
extraction was needed in this case.

After preprocessing, learning source labels $y_{t}$ are now expressed in the
form of probabilistic distributions $z_{t}$. Then, the learning of a deep
representation can take place by minimizing the Kullback–Leibler (KL)
divergence between model inferences $f_{t}(X)$ and label factor distributions
$z_{t}$.

Along with the noise reduction, another benefit from such preprocessing is the
regularization of the scale of the objective function between different tasks
involved in the learning, when the resulting factors have the same size. This
regularity between the objective functions is particularly helpful for
comparing different tasks and datasets. For this purpose, we used a fixed
single value $k=50$ for the number of factors (pLSA) and the number of
Gaussians (GMM). In the remainder of this paper, the datasets and tasks
processed in above manner will be denoted by learning sources for coherent
presentation and usage of the terminology.

##  3 Representation Network Architectures

In this section, we present the detailed specification of the deep
representation neural network architecture we exploited in this work. We will
discuss the base architecture of the network, and further discuss the shared
architecture with respect to different fusion strategies that one can take in
the MTDTL context. Also, we introduce details on the preprocessing related to
the input data served into networks.

###  3.1 Base Architecture

Table 3: Configuration of the base CNN. conv and max-pool indicate a 2-dimensional convolution and max-pooling layer, respectively. We set the stride size with 2 on the time dimension of conv1, to compress dimensionality at the early stage. Otherwise, all strides are set as 1 across all the convolution layers. gap corresponds to the global average pooling used in He et al. (2016), which averages out all the spatial dimensions of the filter responses. fc is an abbreviation of fully-connected layer. We use dropout with $p=0.5$ only for the fc-feature layer, where the intermediate latent representation is extracted and evaluated. For simplicity, we omit the batch-size dimension of the input shape. Layer | Input Shape | Weight Shape | Sub-Sampling | Activation
---|---|---|---|---
conv1 | $2\times 216\times 128$ | $2\times 16\times 5\times 5$ | $2\times 1$ | ReLU
max-pool1 | $16\times 108\times 128$ |  | $2\times 2$ |
conv2 | $16\times 54\times 64$ | $16\times 32\times 3\times 3$ |  | ReLU
max-pool2 | $32\times 54\times 64$ |  | $2\times 2$ |
conv3 | $32\times 27\times 32$ | $32\times 64\times 3\times 3$ |  | ReLU
max-pool3 | $64\times 27\times 32$ |  | $2\times 2$ |
conv4 | $64\times 13\times 16$ | $64\times 64\times 3\times 3$ |  | ReLU
max-pool4 | $64\times 13\times 16$ |  | $2\times 2$ |
conv5 | $64\times 6\times 8$ | $64\times 128\times 3\times 3$ |  | ReLU
max-pool5 | $128\times 6\times 8$ |  | $2\times 2$ |
conv61 | $128\times 3\times 4$ | $128\times 256\times 3\times 3$ |  | ReLU
conv62 | $256\times 3\times 4$ | $256\times 256\times 1\times 1$ |  | ReLU
gap | $256$ |  |  |
fc-feature | $256$ | $256\times 256$ |  | ReLU
dropout | $256$ |  |  |
fc-output | $256$ | learning source specific |  | Softmax

As the deep base architecture for feature representation learning, we choose a
Convolutional Neural Network (CNN) architecture inspired by Simonyan and
Zisserman (2015), as described in Fig. 4 and Table 3.

The CNN is one of the most popular architectures in many music-related machine
learning tasks van den Oord et al. (2013); Choi et al. (2016); Han et al.
(2017); Schlüter (2016); Hershey et al. (2017); Lee et al. (2009); Dieleman et
al. (2011); Humphrey and Bello (2012); Nakashika et al. (2012); Ullrich et al.
(2015); Piczak (2015); Simpson et al. (2015); Phan et al. (2016); Pons et al.
(2016); Stasiak and Monko (2016); Su et al. (2016). Many of these works adopt
an architecture having cascading blocks of 2-dimensional filters and max-
pooling, derived from well-known works in image recognition Simonyan and
Zisserman (2015); Krizhevsky et al. (2017). Although variants of CNN using
1-dimensional filters also were suggested by Dieleman and Schrauwen (2014);
van den Oord et al. (2016); Aytar et al. (2016); Jaitly and Hinton (2011) to
learn features directly from a raw audio signal in an end-to-end manner, not
many works managed to use them on music classification tasks successfully Lee
et al. (2017).

The main difference between the base architecture and Simonyan and Zisserman
(2015) is the use of Global Average Pooling (GAP) and the Batch Normalization
(BN) layers. BN is applied to accelerate the training and stabilize the
internal covariate shift for every convolution layer and the fc-feature layer
Ioffe and Szegedy (2015). Also, global spatial pooling is adopted as the last
pooling layer of the cascading convolution blocks, which is known to
effectively summarize the spatial dimensions both in the image He et al.
(2016) and music domain Han et al. (2017). We also applied the approach to
ensure the fc-feature layer not to have a huge number of parameters.

We applied the Rectified Linear Unit (ReLU) Nair and Hinton (2010) to all
convolution layers and the fc-feature layer. For the fc-output layer, softmax
activation is used. For each convolution layer, we applied zero-padding such
that the input and the output have the same spatial shape. As for the
regularization, we choose to apply drop-out Srivastava et al. (2014) on the
fc-feature layer. We added $L2$ regularization across all the parameters with
the same weight $\lambda=10^{-6}$.

####  3.1.1 Audio Preprocessing

We aim to learn a music representation from as-raw-as-possible input data to
fully leverage the capability of the neural network. For this purpose, we use
the dB-scale mel-scale magnitude spectrum of an input audio fragment,
extracted by applying 128-band mel-filter banks on the Short-Time Fourier
Transform (STFT). mel-spectrograms have generally been a popular input
representation choice for CNNs applied in music-related tasks Nam et al.
(2012); Hamel et al. (2013); van den Oord et al. (2013); Choi et al. (2016,
2017); Han et al. (2017); besides, it also was reported recently that their
frequency-domain summarization, based on psycho-acoustics, is efficient and
not easily learnable through data-driven approaches Choi et al. (2018);
Dörfler et al. (2018). We choose a 1024-sample window size and 256-sample hop
size, translating to about 46 ms and 11.6 ms respectively for a sampling rate
of 22 kHz. We also applied standardization to each frequency band of the mel
spectrum, making use of the mean and variance of all individual mel spectra in
the training set.

####  3.1.2 Sampling

During the learning process, in each iteration, a random batch of songs is
selected. Audio corresponding to these songs originally is 30 seconds in
length; for computational efficiency, we randomly crop 2.5 seconds out of each
song each time. Keeping stereo channels of the audio, the size of a single
input tensor $X^{*}$ we used for the experiment ended up with $2\times
216\times 128$, where the first dimension indicates number of channels, and
following dimensions mean time steps and mel-bins, respectively. Along with
the computational efficiency, a number of literatures in MIR field reported
that using a small chunk of the input not only inflates the dataset, but also
shows good performance on the high-level tasks such as music auto-tagging Lee
et al. (2017); Han et al. (2017); Dieleman and Schrauwen (2014). For the self
case, we generate batches with equal numbers of songs for both membership
categories in $y_{self}$.

Figure 4: Default CNN architecture for supervised single-source representation
learning. Details of the Representation Network are presented at the left of
the global architecture diagram. The numbers inside the parentheses indicate
either the number of filters, or the number of units with respect to the type
of layer.

###  3.2 Multi-Source Architectures with Various Degrees of Shared Information

When learning a music representation based on various available learning
sources, different strategies can be taken regarding the choice of
architecture. We will investigate the following setups:

  * •

As a base case, a _Single-Source Representation (SS-R)_ can be learned for a
single source only. As mentioned earlier, this would be the typical strategy
leading to pre-trained networks, that later would be used in transfer
learning. In our case, our base architecture from Section 3.1 and Fig. 4 will
be used, for which the layers in the Representation Network also are
illustrated in Fig. 5a. Out of the fc-feature layer, a $d$-dimensional
representation is obtained.

  * •

If multiple perspectives on the same content, as reflected by the multiple
learning labels, should also be reflected in the ultimate learned
representation, one can learn _SS-R_ representations for each learning source,
and simply concatenate them afterwards. With $d$ dimensions per source and $m$
sources, this leads to a $d\times m$ _Multiple Single-Source Concatenated
Representation (MSS-CR)_. In this case, independent networks are trained for
each of the sources, and no shared knowledge will be transferred between
sources. A layer setup of the corresponding Representation Network is
illustrated in Fig. 5b.

  * •

When applying MTL learning strategies, the deep architecture should involve
shared knowledge layers, before branching out to various individual learning
sources, whose learned representations will be concatenated in the final
$d\times m$-dimensional representation. We call these _Multi-Source
Concatenated Representations (MS-CR)_. As the branching point can be chosen at
different stages, we will investigate the effect of various prototypical
branching point choices: at the second convolution layer (_MS-CR@2_ , Fig.
5c), the fourth convolution layer (_MS-CR@4_ , Fig. 5d), and the sixth
convolution layer (_MS-CR@6_ , Fig. 5e). The later the branching point occurs,
the more shared knowledge the network will employ.

  * •

In the most extreme case, branching would only occur at the very last fully
connected layer, and a Multi-Source Shared Representation (MS-SR) (or, more
specifically, _MS-SR@FC_) is learned, as illustrated in Fig. 5f. As the
representation is obtained from the fc-feature layer, no concatenation takes
place here, and a $d$-dimensional representation is obtained.

A summary of these different representation learning architectures is given in
Table 4. Beyond the strategies we choose, further approaches can be thought of
to connect representations learned for different learning sources in neural
network architectures. For example, for different tasks, representations can
be extracted from different intermediate hidden layers, benefiting from the
hierarchical feature encoding capability of the deep network Choi et al.
(2017). However, considering that learned representations are usually taken
from a specific fixed layer of the shared architecture, we focus on the
strategies as we outlined above.

Table 4: Properties of the various categories of representation learning architectures. | Multi Source | Shared Network | Concatenation | Dimensionality
---|---|---|---|---
SS-R | No | No | No | $d$
MSS-CR | Yes | No | Yes | $d\times{m}$
MS-CR | Yes | Partial | Yes | $d\times{m}$
MS-SR | Yes | Yes | No | $d$

(a) SS-R: Base setup.

(b) MSS-CR: Concatenation of multiple independent SS-R networks.

(c) MS-CR@2: network branches to source-specific layers from 2nd convolution
layer.

(d) MS-CR@4: network branches to source-specific layers from 4th convolution
layer.

(e) MS-CR@6: network branches to source-specific layers from 6th convolution
layer.

(f) MS-SR@FC: heavily shared network, source-specific branching only at final
FC layer.

Figure 5: The various model architectures considered in the current work.
Beyond single-source architectures, multi-source architectures with various
degrees of shared information are studied. For simplification, multi-source
cases are illustrated here for two sources. The fc-feature layer from which
representations will be extracted is the FC(256) layer in the illustrations
(see Table 3).

###  3.3 MTL Training Procedure

1 Initialize $\Theta$: {$\theta^{t}$, $\theta^{s}$} randomly;

2 for  _epoch in 1…N_ do

3 for  _iteration in 1…L_ do

4 Pick a learning source $t$ randomly;

5 Pick batch of samples from learning source $t$;

($X_{l}$, $X_{r}$) for self;

$X$ otherwise;

6 Derive learning label $z_{t}$;

7 Sub-sample chunk $X^{*}$ from track $X$;

8 Forward-pass:;

$\mathcal{L}(y_{self},\Theta,X_{l}^{*},X_{r}^{*})=$Eq. 5 for self;

$\mathcal{L}(z_{t},\Theta,X^{*})=$Eq. 2 otherwise;

9 Backward-pass: $\nabla(\Theta)$;

10 Update model: $\Theta\leftarrow\Theta-\epsilon\nabla(\Theta)$;

Algorithm 1 Training a Multi-Source CNN

Similar to Weston et al. (2011); Liu et al. (2015), we choose to train the MTL
models with a stochastic update scheme as described in Algorithm 1. At every
iteration, a learning source is selected randomly. After the learning source
is chosen, a batch of observation-label pairs $(X,z_{t})$ is drawn. For the
audio previews belonging to the songs within this batch, an input
representation $X^{*}$ is cropped randomly from its super-sample $X$. The
updates of the parameters $\Theta$ are conducted through back-propagation
using the Adam algorithm Kingma and Ba (2015). For each neural network we
train, we set $L=lm$, where $l$ is the number of iterations needed to visit
all the training samples with fixed batch size $b=128$, and $m$ is the number
of learning sources used in the training. Across the training, we used a fixed
learning rate $\epsilon=0.00025$. After a fixed number of epochs $N$ is
reached, we stop the training.

###  3.4 Implementation Details

We used PyTorch Paszke et al. (2017) to implement the CNN models and parallel
data serving. For evaluation of models and cross-validation, we made extensive
use of functionality in Scikit-Learn Pedregosa et al. (2012). Furthermore,
Librosa McFee et al. (2015) was used to process audio files and its raw
features including mel spectrograms. The training is conducted with 8
Graphical Processing Unit (GPU) computation nodes, composed of 2 NVIDIA GRID
K2 GPUs and 6 NVIDIA GTX 1080Ti GPUs.

Figure 6: Overall system framework. The first row of the figure illustrates
the learning scheme, where the representation learning is happening by
minimizing the KL divergence between the network inference $f_{t}(X)$ and the
preprocessed learning label $z_{t}$. The preprocessing is conducted by the
blue blocks which transform the original noisy labels $y_{t}$ to $z_{t}$,
reducing noise and summarizing the high-dimensional label space into a smaller
latent space. The second row describes the entire evaluation scenario. The
representation is first extracted from the representation network, which is
transferred from the upper row. The sequence of representation vectors is
aggregated as the concatenation of their means and standard deviations. The
purple block indicates a machine learning model employed to evaluate the
representation’s effectiveness.

##  4 Evaluation

So far, we discussed the details regarding the learning phase of this work,
which corresponds to the upper row of Fig. 6. This included various choices of
sources for the representation learning, and various choices of architecture
and fusion strategies. In this section, we present the evaluation methodology
we followed, as illustrated in the second row of Fig. 6. First, we will
discuss the chosen target tasks and datasets in Section 4.1, followed in
Section 4.2 by the baselines against which our representations will be
compared. Section 4.3 explains our experimental design, and finally we discuss
the implementation of our evaluation experiments in Section 4.4.

###  4.1 Target Datasets

In order to gain insight into the effectiveness of learned representations
with respect to multiple potential future tasks, we consider a range of
_target datasets_. In this work, our target datasets are chosen to reflect
various semantic properties of music, purposefully chosen semantic biases, or
popularity in the MIR literature. Furthermore, the representation network
should not be configured or learned to explicitly solve the chosen target
datasets.

While for the learning sources, we could provide categorizations on where and
how the learning labels were derived, and also consider algorithmic outcomes
as labels, existing popular research datasets mostly fall in the Professional
or Crowd categories. In our work, we choose 7 evaluation datasets commonly
used in MIR research, which reflect three conventional types of MIR tasks,
namely classification, regression and recommendation:

Table 5: Properties of target datasets used in our experiments. Because of time constraints, we sampled the Lastfm dataset as described in Section 4.1; the original size appears between parentheses. In case particular data splits are defined by an original author or follow up study, we apply the same split, including the reference in which the split is introduced. Otherwise, we applied either a random split stratified by the label (Ballroom), or simple filtering based on reported faulty entries (IRMAS). Task | Data | #Tracks | #Class | Split Method
---|---|---|---|---
Classification | FMADefferrard et al. (2017) | Genre | 25,000 | 16 | Artist Filtered Defferrard et al. (2017)
Classification | GTZANTzanetakis and Cook (2002) | Genre | 1,000 | 10 | Artist Filtered Kereliuk et al. (2015)
Classification | Ext. BallroomGouyon et al. (2006); Marchand and Peeters (2016) | Genre | 3,390 | 13 | N/A
Classification | IRMASBosch et al. (2012) | Instrument | 6,705 | 11 | Song Filtered
Regression | Music EmotionSoleymani et al. (2013) | Arousal | 744 |  | Genre StratifiedSoleymani et al. (2013)
Regression | Music EmotionSoleymani et al. (2013) | Valence | 744 |  | Genre StratifiedSoleymani et al. (2013)
Recommendation | Lastfm*Celma (2010) | Listening Count | 27,093 (961,416) |  | N/A

  * •

Classification. Different types of classification tasks exist in MIR. In our
experiments, we consider several datasets used for genre classification and
instrument classification.

For genre classification, we chose the GTZAN Tzanetakis and Cook (2002) and
FMA Defferrard et al. (2017) datasets as main exemplars. Even though GTZAN is
known for its caveats Sturm (2014), we deliberately used it, because its
popularity can be beneficial when comparing with previous and future work. We
note though that there may be some overlap between the tracks of GTZAN and the
subset of the MSD we use in our experiments; the extent of this overlap is
unknown, due to the lack of a confirmed and exhaustive track listing of the
GTZAN dataset. We choose to use a fault-filtered data split for the training
and evaluation, which is suggested in Kereliuk et al. (2015). The split
originally includes a training, validation and evaluation split; in our case,
we also included the validation split as training data.

Among the various packages provided by the FMA, we chose the top-genre
classification task of FMA-Medium Defferrard et al. (2017). This is a
classification dataset with an unbalanced genre distribution. We used the data
split provided by the dataset for our experiment, where the training is
validation set are combined as the training.

Considering another type of genre classification, we selected the Extended
Ballroom dataset Gouyon et al. (2006); Marchand and Peeters (2016). Because
the classes in this dataset are highly separable with regard to their BPM
Sturm (2016), we specifically included this ‘purposefully biased’ dataset as
an example of how a learned representation may effectively capture temporal
dynamics properties present in a target dataset, as long as learning sources
also reflected these properties. Since no pre-defined split is provided or
suggested by other literature, we used stratified random sampling based on the
genre label.

The last dataset we considered for classification is the training set of the
IRMAS dataset Bosch et al. (2012), which consists of short music clips
annotated with the predominant instruments present in the clip. Compared to
the genre classification task, instrument classification is generally
considered as less subjective, requiring features to separate timbral
characteristics of the music signal as opposed to high-level semantics like
genre. We split the dataset to make sure that observations from the same music
track are not split into training and test set.

As performance metric for all these classification tasks, we used
classification accuracy.

  * •

Regression. As exemplars of regression tasks, we evaluate our proposed deep
representations on the dataset used in the MediaEval Music Emotion prediction
task Soleymani et al. (2013). It contains frame-level and song-level labels of
a two-dimensional representation of emotion, with valence and arousal as
dimensions Posner et al. (2005). Valence is related to the positivity or
negativity of the emotion, and arousal is related to its intensity Soleymani
et al. (2013). The song-level annotation of the V-A coordinates was used as
the learning label. In similar fashion to the approach taken in Choi et al.
(2017), we trained separate models for the two emotional dimensions. As for
the dataset split, we used the split provided by the dataset, which is done by
the random split stratified by the genre distribution.

As evaluation metric, we measured the coefficient of determination $R^{2}$ of
each model.

  * •

Recommendation. Finally, we employed the ‘Last.fm - 1K users’ dataset Celma
(2010) to evaluate our representations in the context of a content-aware music
recommendation task (which will be denoted as _Lastfm_ in the remaining of the
paper). This dataset contains 19 million records of listening events across
$961,416$ unique tracks collected from $992$ unique users. In our experiments,
we mimicked a cold-start recommendation problem, in which items not seen
before should be recommended to the right users. For efficiency, we filtered
out users who listened to less than $5$ tracks and tracks known to less than
$5$ users.

As for the audio content of each track, we obtained the mapping between the
MusicBrainz Identifier (MBID) with the Spotify identifier (SpotifyID) using
the MusicBrainz API333https://musicbrainz.org/. After cross-matching, we
collected 30 seconds previews of all track using the Spotify
API444https://developer.spotify.com/documentation/web-api/. We found that
there is a substantial amount of missing mapping information between the
SpotifyID and MBID in the MusicBrainz database, where only approximately 30%
of mappings are available. Also, because of the substantial amount of inactive
users and unpopular tracks in the dataset, we ultimately acquired a dataset of
$985$ unique users and $27,093$ unique tracks with audio content.

Similar to Liang et al. (2015), we considered the outer matrix performance for
un-introduced songs; in other words, the model’s recommendation accuracy on
the items newly introduced to the system Liang et al. (2015). This was done by
holding out certain tracks when learning user models, and then predicting user
preference scores based on all tracks, including those that were held out,
resulting in a ranked track list per user. As evaluation metric, we consider
Normalized Discounted Cumulative Gain ($nDCG@500$), only treating held-out
tracks that were indeed liked by a user as relevant items. Further details on
how hold-out tracks were chosen are given in Section 4.4.

A summary of all evaluation datasets, their origins and properties, can be
found in Table 5.

###  4.2 Baselines

We examined three baselines to compare with our proposed representations:

  * •

Mel-Frequency Cepstral Coefficients (MFCC). These are some of the most popular
audio representations in MIR research. In this work, we extract and aggregate
MFCC following the strategy in Choi et al. (2017). In particular, we extracted
20 coefficients and also used their first- and second-order derivatives. After
obtaining the sequence of MFCCs and its derivatives, we performed aggregation
by taking the average and standard deviation over the time dimension,
resulting in a 120-dimensional vector representation.

  * •

Random Network Feature (Rand). We extracted the representation at the fc-
feature layer without any representation network training. With random
initialization, this representation therefore gives a random baseline for a
given CNN architecture. We refer to this baseline as Rand.

  * •

Latent Representation from Music Auto-Tagger (Choi). The work in Choi et al.
(2017) focused on a music auto-tagging task, and can be considered as yielding
a state-of-the-art deep music representation for MIR. While the model’s focus
on learning a representation for music auto-tagging can be considered as our
SS-R case, there are a number of issues that complicate direct comparisons
between this work and ours. First, the network in Choi et al. (2017) is
trained with about 4 times more data samples than in our experiments. Second,
it employed a much smaller network than our architecture. Further,
intermediate representations were extracted, which is out of the scope of our
work, as we only consider representations at the fc-feature layer.
Nevertheless, despite these caveats, the work still is very much in line with
ours, making it a clear candidate for comparison. Throughout the evaluation,
we could not fully reproduce the performance reported in the original paper
Choi et al. (2017). When reporting our results, we therefore will report the
performance we obtained with the published model, referring to this as Choi.

###  4.3 Experimental Design

Figure 7: Aliasing among main effects in the final experimental design.

In order to investigate our research questions, we carried out an experiment
to study the effect of the number and type of learning sources on the
effectiveness of deep representations, as well as the effect of the various
architectural learning strategies described in Section 3.2. For the
experimental design we consider the following factors:

  * •

Representation strategy, with 6 levels: _SS-R_ , _MS-SR@FC_ , _MS-CR@6_ , _MS-
CR@4_ , _MS-CR@2_ , and _MSS-CR_).

  * •

8 2-level factors indicating the presence or not of each of the 8 learning
sources: _self_ , _year_ , _bpm_ , _taste_ , _tag_ , _lyrics_ , _cdr_tag_ and
_artist_.

  * •

Number of learning sources present in the learning process (1 to 8). Note that
this is actually calculated as the sum of the eight factors above.

  * •

Target dataset, with 7 levels: Ballroom, FMA, GTZAN, IRMAS, Lastfm, Arousal
and Valence.

Given a learned representation, fitting dataset-specific models is much more
efficient than learning the representation, so we decided to evaluate each
representation on all 7 target datasets. The experimental design is thus
restricted to combinations of representation and learning sources, and for
each such combination we will produce 7 observations. However, given the
constraint of _SS-R_ relying on a single learning source, that there is only
one possible combination for n = 8 sources, as well as the high unbalance in
the number of sources555For instance, from the 255 possible combinations of up
to 8 sources, there are 70 combinations of $n=4$ sources, but 28 with $n=2$,
or only 8 for $n=7$. Simple random sampling from the 255 possible combinations
would lead to a very unbalanced design, that is, a highly non-uniform
distribution of observation counts across the levels of the factor ($n$ in
this case). A balanced design is desired to prevent aliasing and maximize
statistical power. See section 15.2 in Montgomery (2012) for details on
unbalanced designs., we proceeded in three phases:

  1. 1.

We first trained the _SS-R_ representations for each of the 8 sources, and
repeated 6 times each. This resulted in 48 experimental runs.

  2. 2.

We then proceeded to train all five multi-source strategies with all sources,
that is, $n=8$. We repeated this 5 times, leading to 25 additional
experimental runs.

  3. 3.

Finally, we ran all five multi-source strategies with $n=2,\dots,7$. The full
design matrix would contain 5 representations and 8 sources, for a total of
1,230 possible runs. Such an experiment was unfortunately infeasible to run
exhaustively given available resources, so we decided to follow a fractional
design. However, rather than using a pre-specified optimal design with a fixed
amount of runs Goos and Jones (2011), we decided to run sequentially for as
long as time would permit us, generating at each step a new experimental run
on demand in a way that would maximize desired properties of the design up to
that point, such as balance and orthogonality666An experimental design is
orthogonal if the effects of any factor balance out across the effects of the
other factors. In a non-orthogonal design effects may be aliased, meaning that
the estimate of one effect is partially biased with the effect of another, the
extent of which ranges from 0 (no aliasing) to 1 (full aliasing). Aliasing is
sometimes referred to as confounding. See sections 8.5 and 9.5 in Montgomery
(2012) for details on aliasing..

We did this with the greedy Algorithm 2. From the set of still remaining runs
$\mathcal{A}$, a subset $\mathcal{O}$ is selected such that the expected
unbalance in the augmented design $\mathcal{B}\cup\\{o\\}$ is minimal. In this
case, the unbalance of a design is defined as the maximum unbalance found
between the levels of any factor, except for those already exhausted777For
instance, let a design have 20 runs for _SS-R_ , 16 for _MS-SR@FC_ , and 18
for all other representations. The unbalance in the representation factor is
thus $20-16=4$. The total unbalance of the design is defined as the maximum
unbalance found across all factors.. From $\mathcal{O}$, a second subset
$\mathcal{P}$ is selected such that the expected aliasing in the augmented
design is minimal, here defined as the maximum absolute aliasing between main
effects888See section 2.3.7 in Goos and Jones (2011) for details on how to
compute an alias matrix.. Finally, a run $p$ is selected at random from
$\mathcal{P}$, the corresponding representation is learned, and the algorithm
iterates again after updating $\mathcal{A}$ and $\mathcal{B}$.

Following this on demand methodology, we managed to run another 352
experimental runs from all the 1,230 possible.

1 Initialize $\mathcal{A}$ with all possible 1,230 runs to execute;

2 Initialize $\mathcal{B}\leftarrow\emptyset$ for the set of already executed
runs;

3 while  _time allows_ do

4 Select $\mathcal{O}\subseteq\mathcal{A}$ s.t. $\forall o\in\mathcal{O}$, the
unbalance in $\mathcal{B}\cup\\{o\\}$ is minimal;

5 Select $\mathcal{P}\subseteq\mathcal{O}$ s.t. $\forall p\in\mathcal{P}$, the
aliasing in $\mathcal{B}\cup\\{p\\}$ is minimal;

6 Select $p\in\mathcal{P}$ at random;

7 Update $\mathcal{A}\leftarrow\mathcal{A}-\\{p\\}$;

8 Update $\mathcal{B}\leftarrow\mathcal{B}\cup\\{p\\}$;

9 Learn the representation coded by $p$;

Algorithm 2 Sequential generation of experimental runs.

After going through the three phases above, the final experiment contained
$48+25+352=425$ experimental runs, each producing a different deep music
representation. We further evaluated each representation on all 7 target
datasets, leading to a grand total of $42\times 7=2,97$5 datapoints. Fig. 7
plots the alias matrix of the final experimental design, showing that the
aliasing among main factors is indeed minimal. The final experimental design
matrix can be downloaded along with the rest of the supplemental material.

Each considered representation network was trained using the CNN
representation network model from Section 3, based on the specific combination
of learning sources and deep architecture as indicated by the experimental
run. In order to reduce variance, we fixed the number of training epochs to
$N=200$ across all runs, and applied the same base architecture, except for
the branching point. This entire training procedure took approximately 5 weeks
with given computational hardware resources introduced in Section 3.4.

###  4.4 Implementation Details

In order to assess how our learned deep music representations perform on the
various target datasets, transfer learning will now be applied, to consider
our representations in the context of these new target datasets.

As a consequence, new machine learning pipelines are set up, focused on each
of the target datasets. In all cases, we applied the pre-defined split if it
is feasible. Otherwise, we randomly split the dataset in a 80% training and
20% test set. For every dataset, we repeated the training and evaluation for 5
times, using different train/test splits. In most of our evaluation cases,
validation will take place on the test set; in case of the the recommendation
problem, the test set represents a set of tracks to be held out during user
model training, and re-inserted for validation. In all cases, we will extract
representations from evaluation dataset audio as detailed in Section 4.4.1,
and then learn relatively simple models based on them, as detailed in Section
4.4.2. Employing the metrics as mentioned in the previous section, we will
then take average performance scores over the 5 different train-test splits
for final performance reporting.

####  4.4.1 Feature Extraction and Preprocessing

Taking raw audio from the evaluation datasets as input, we take non-
overlapping slices out of this audio with a fixed length of 2.5 seconds. Based
on this, we apply the same preprocessing transformations as discussed in
Section 3.1.1. Then, we extract a deep representation from this preprocessed
audio, employing the architecture as specified by the given experimental run.
As in the case of Section 3.2, representations are extracted from the fc-
feature layer of each trained CNN model. Depending on the choice of
architecture, the final representation may consist of concatenations of
representations obtained by separate representation networks.

Input audio may originally be (much) longer than 2.5 seconds; therefore, we
aggregate information in feature vectors over multiple time slices by taking
their mean and standard deviation values. As a result, we get a representation
with averages per learned feature dimension, and another representation with
standard deviations per feature dimension. These will be concatenated, as
illustrated in Fig. 6.

####  4.4.2 Target Dataset-Specific Models

As our goal is not to over-optimize dataset-specific performance, but rather
perform a comparative analysis between different representations (resulting
from different learning strategies), we keep the model simple, and use fixed
hyper-parameter values for each model across the entire experiment.

To evaluate the trained representations, we used different models according to
the target dataset. For classification and regression tasks, we used Multi
Layer Perceptron (MLP) model Hinton (1989). More specifically, the MLP model
has two hidden layers, whose dimensionality is $256$. As for the non-
linearity, we choose ReLU Nair and Hinton (2010) for all nodes, and the model
is trained with ADAM optimization technique Kingma and Ba (2015) for 200
iterations. In evaluation, we used the Scikit-Learn’s implementation for ease
of distributed computing on multiple CPU computation nodes.

For the recommendation task, we choose a similar model as suggested in Liang
et al. (2015); Hu et al. (2008), in which the learning objective function
$\mathcal{L}$ is defined as

$\hat{U},\hat{V},\hat{W}=\operatorname*{arg\,min}\;||P-UV^{T}||_{C}+\frac{\lambda^{V}}{2}||V-XW||+\frac{\lambda^{U}}{2}||U||+\frac{\lambda^{W}}{2}||W||$
(7)

where $P\in\mathbb{R}^{u\times{i}}$ is a binary matrix indicating whether
there is interaction between users $u$ and items $i$,
$U\in\mathbb{R}^{u\times{r}}$ and $V\in\mathbb{R}^{i\times{r}}$ are $r$
dimensional user factors and item factors for the low-rank approximation of
$P$. $P$ is derived from the original interaction matrix
$R\in\mathbb{R}^{u\times{i}}$, which contains the number of interaction from
users $u$ to items $i$, as follows:

$P_{u,i}=\begin{cases}1,&\text{if }R_{u,i}>0\\\ 0&\text{otherwise}\end{cases}$
(8)

$W\in\mathbb{R}^{d\times{r}}$ is a free parameter for the projection from
$d$-dimensional feature space to the factor space.
$X\in\mathbb{R}^{i\times{d}}$ is the feature matrix where each row corresponds
to a track. Finally, $||\cdot||_{C}$ is the Frobenious norm weighted by the
confidence matrix $C\in\mathbb{R}^{u\times{i}}$, which controls the
credibility of the model on the given interaction data, given as follows:

$C=1+\alpha R$ (9)

where $\alpha$ controls credibility. As for hyper-parameters, we set
$\alpha=0.1$, $\lambda^{V}=0.00001$, $\lambda^{U}=0.00001$, and
$\lambda^{W}=0.1$, respectively. For the number of factors we choose $r=50$ to
focus only on the relative impact of the representation over the different
conditions. We implemented an update rule with the Alternating Least Squares
(ALS) algorithm similar to Liang et al. (2015), and updated parameters during
15 iterations.

##  5 Results and Discussion

In this section, we present results and discussion related to the proposed
deep music representations. In Section 5.1, we will first compare the
performance across the _SS-R_ s, to show how different individual learning
sources work for each target dataset. Then, we will present general
experimental results related to the performance of the multi-source
representations. In Section 5.2, we discuss the effect of the number of
learning sources exploited in the representation learning, in terms of their
general performance, reliability, and model compactness. In Section 5.3, we
discuss effectiveness of different representations in MIR. Finally, we present
some initial evidence for multifaceted semantic explainability of the proposed
MTDTL in Section 5.5.999For the reproducibility, we release all relevant
materials including code, models and extracted features at
https://github.com/eldrin/MTLMusicRepresentation-PyTorch.

###  5.1 Single-Source and Multi-Source Representation

Figure 8: Performance of single source representations. Each point indicates
the performance of a representation learned from the single source. Solid
points indicate the average performance per source. The baselines are
illustrated as horizontal lines.

Fig. 8 presents the performance of _SS-R_ representations on each of the 7
target datasets. We can see that all sources tend to outperform the Rand
baseline on all datasets, except for a handful cases involving sources _self_
and _bpm_. Looking at the top performing sources, we find that _tag_ ,
_cdr_tag_ and _artist_ perform better or on-par with the most sophisticated
baseline, Choi, except for the IRMAS dataset. The other sources are found
somewhere between these two baselines, except for datasets Lastfm and Arousal,
where they perform better than Choi as well. Finally, the MFCC is generally
outperformed in all cases, with the notable exception of the IRMAS dataset,
where only Choi performs better.

Zooming in to dataset-specific observed trends, the _bpm_ learning source
shows a highly skewed performance across target datasets: it clearly
outperforms all other learning sources in the Ballroom dataset, but it
achieves the worst or second worst performance in the other datasets. As shown
in Sturm (2016), this confirms that the Ballroom dataset is well-separable
based on BPM information alone. Indeed, representations trained on the _bpm_
learning source seem to contain a latent representation close to the BPM of an
input music signal. In contrast, we can see that the _bpm_ representation
achieves the worst results in the Arousal dataset, where both temporal
dynamics and BPM are considered as important factors determining the intensity
of emotion.

On the IRMAS dataset, we see that all the _SS-R_ s perform worse than the MFCC
and Choi baselines. Given that they both take into account low-level features,
either by design or by exploiting low-level layers of the neural network, this
suggests that predominant instrument sounds are harder to distinguish based
solely on semantic features, which is the case of the representations studied
here.

Also, we find that there is small variability for each _SS-R_ run within the
training setup we applied. Specifically, in 50% of cases we have within-_SS-R_
variability less than 15% of the within-dataset variability. 90% of the cases
are within 30% of the within-dataset variability.

Figure 9: Performance by representation strategy. Solid points represent the
mean per representation. The baselines are illustrated as horizontal lines.

We now consider how the various representations based on multiple learning
sources perform, in comparison to those based on single learning sources. The
boxplots in Fig. 9 show the distributions of performance scores for each
architectural strategy and per target dataset. For comparison, the gray boxes
summarize the distributions depicted in Fig. 8, based on the _SS-R_ strategy.
In general, we can see that these _SS-R_ obtain the lowest scores, followed by
_MS-SR@FC_ , except for the IRMAS dataset. Given that these representations
have the same dimensionality, these results suggest that adding a single
source-specific layer on top of a heavily shared model may help improving the
adaptability of the neural network models, especially when there is no prior
knowledge regarding the well-matching learning sources for the target
datasets. The _MS-CR_ and _MSS-CR_ representations obtain the best results in
general, which is somewhat expected because of their larger dimensionality.

###  5.2 Effect of Number of Learning Sources and Fusion Strategy

Figure 10: (Standardized) Performance by number of learning sources. Solid
points represent the mean per architecture and number of sources. The black
horizontal line marks the mean performance of the _SS-R_ representations. The
colored lines show linear fits.

While the plots in Fig. 9 suggest that _MSS-CR_ and _MS-CR_ are the best
strategies, the high observed variability makes this statement still rather
unclear. In order to gain better insight of the effects of dataset,
architecture strategies and number and type of learning sources, we further
analyzed the results using a hierarchical or multilevel linear model on all
observed scores Gelman and Hill (2006). The advantage of such a model is
essentially that it accounts for the structure in our experiment, where
observations nested within datasets are not independent.

By Fig. 9 we can anticipate a very large dataset effect because of the
inherently different levels of difficulty, as well as a high level of
heteroskedasticity. We therefore analyzed standardized performance scores
rather than raw scores. In particular, the $i$-th performance score $y_{i}$ is
standardized with the within-dataset mean and standard deviation scores, that
is, $y^{*}_{i}=(y_{i}-\bar{y}_{d[i]})/s_{d[i]}$, where $d[i]$ denotes the
dataset of the $i$-th observation. This way, the dataset effect is effectively
$0$ and the variance is homogeneous. In addition, this will allow us to
compare the relative differences across strategies and number of sources using
the same scale in all datasets.

We also transformed the variable $n$ that refers to the number of sources to
$n^{*}$, which is set to $n^{*}=0$ for _SS-R_ s and to $n^{*}=n-2$ for the
other strategies. This way, the intercepts of the linear model will represent
the average performance of each representation strategy in its simplest case,
that is, _SS-R_ ($n=1$) or non-_SS-R_ with $n=2$. We fitted a first analysis
model as follows:

$\displaystyle y^{*}_{i}$
$\displaystyle=\beta_{0r[i]d[i]}+\beta_{1r[i]d[i]}\cdot n^{*}_{i}+e_{i}$
$\displaystyle e_{i}$ $\displaystyle\sim N(0,\sigma^{2}_{e})$ (10)
$\displaystyle\beta_{0rd}$ $\displaystyle=\beta_{0r}+u_{0rd}$ $\displaystyle
u_{0rd}$ $\displaystyle\sim N(0,\sigma^{2}_{0r})$ (11)
$\displaystyle\beta_{1rd}$ $\displaystyle=\beta_{1r}+u_{1rd}$ $\displaystyle
u_{1rd}$ $\displaystyle\sim N(0,\sigma^{2}_{1r}),$ (12)

where $\beta_{0r[i]d[i]}$ is the intercept of the corresponding representation
strategy within the corresponding dataset. Each of these coefficients is
defined as the sum of a global fixed effect $\beta_{0r}$ of the
representation, and a random effect $u_{0rd}$ which allows for random within-
dataset variation101010We note that hierarchical models do not fit each of the
individual $u_{0rd}$ coefficients (a total of 42 in this model), but the
amount of variability they produce, that is, $\sigma^{2}_{0r}$ (6 in total)..
This way, we separate the effects of interest (ie. each $\beta_{0r}$) from the
dataset-specific variations (ie. each $u_{0rd}$). The effect of the number of
sources is similarly defined as the sum of a fixed representation-specific
coefficient $\beta_{1r}$ and a random dataset-specific coefficient $u_{1rd}$.
Because the slope depends on the representation, we are thus implicitly
modeling the interaction between strategy and number of sources, which can be
appreciated in Fig. 10, specially with _MS-SR@FC_.

Figure 11: Fixed effects and bootstrap 95% confidence intervals estimated for
the first analysis model. The left plot depicts the effects of the
representation strategy ($\beta_{0r}$ intercepts) and the right plot shows the
effects of the number of sources ($\beta_{1r}$ slopes).

Fig. 11 shows the estimated effects and bootstrap 95% confidence intervals.
The left plot confirms the observations in Fig. 9. In particular, they confirm
that _SS-R_ performs significantly worse than _MS-SR@FC_ , which is similarly
statistically worse than the others. When carrying out pairwise comparisons,
_MSS-CR_ outperforms all other strategies except _MS-CR@2_ ($p=0.32$), which
ourperforms all others except _MS-CR@6_ ($p=0.09$). The right plot confirms
the qualitative observation from Fig. 10 by showing a significantly positive
effect of the number of sources except for _MS-SR@FC_ , where it is not
statistically different from 0. The intervals suggest a very similar effect in
the best representations, with average increments of about $0.16$ per
additional source —recall that scores are standardized.

To gain better insight into differences across representation strategies, we
used a second hierarchical model where the representation strategy was modeled
as an ordinal variable $r^{*}$ instead of the nominal variable $r$ used in the
first model. In particular, $r^{*}$ represents the size of the network, so we
coded _SS-R_ as $0$, _MS-SR@FC_ as $0.2$, _MS-CR@6_ as $0.4$, _MS-CR@4_ as
$0.6$, _MS-CR@2_ as $0.8$, and _MSS-CR_ as $1$ (see Fig. 5). In detail, this
second model is as follows:

$\displaystyle y^{*}_{i}$ $\displaystyle=\beta_{0}+\beta_{1d[i]}\cdot
r^{*}_{i}+\beta_{2d[i]}\cdot n^{*}_{i}+\beta_{3d[i]}\cdot r^{*}_{i}\cdot
n^{*}_{i}+e_{i}$ $\displaystyle e_{i}$ $\displaystyle\sim N(0,\sigma^{2}_{e})$
(13) $\displaystyle\beta_{1d}$ $\displaystyle=\beta_{10}+u_{1d}$
$\displaystyle u_{1d}$ $\displaystyle\sim N(0,\sigma^{2}_{1})$ (14)
$\displaystyle\beta_{2d}$ $\displaystyle=\beta_{20}+u_{2d}$ $\displaystyle
u_{2d}$ $\displaystyle\sim N(0,\sigma^{2}_{2})$ (15) $\displaystyle\beta_{3d}$
$\displaystyle=\beta_{30}+u_{3d}$ $\displaystyle u_{3d}$ $\displaystyle\sim
N(0,\sigma^{2}_{3}).$ (16)

In contrast to the first model, there is no representation-specific fixed
intercept but an overall intercept $\beta_{0}$. The effect of the network size
is similarly modeled as the sum of an overall fixed slope $\beta_{10}$ and a
random dataset-specific effect $u_{1d}$. Likewise, this model includes the
main effect of the number of sources (fixed effect $\beta_{20}$), as well as
its interaction with the network size (fixed effect $\beta_{30}$). Fig. 12
shows the fitted coefficients, confirming the statistically positive effect of
the size of the networks and, to a smaller degree but still significant, of
the number of sources. The interaction term is not statistically significant,
probably because of the unclear benefit of the number of sources in _MS-
SR@FC_.

Figure 12: Fixed effects and bootstrap 95% confidence intervals estimated for
the second analysis model, depicting the overall intercept ($\beta_{0}$), the
slope of the network size ($\beta_{10}$), the slope of the number of sources
($\beta_{20}$), and their interaction ($\beta_{30}$).

Overall, these analyses confirm that all multi-source strategies outperform
the single-source representations, with a direct relation to the number of
parameters in the network. In addition, there is a clearly positive effect of
the number of sources, with a minor interaction between both factors.

Fig. 10 also suggests that the variability of performance scores decreases
with the number of learning sources used. This implies that if there are more
learning sources available, one can expect less variability across
instantiations of the network. Most importantly, variability obtained for a
single learning source ($n=1$) is always larger than the variability with 2 or
more sources. The Ballroom dataset shows much smaller variability when BPM is
included in the combination. For this specific dataset, this indicates that
once _bpm_ is used to learn the representation, the expected performance is
stable and does not vary much, even if we keep including more sources. Section
5.3 provides more insight in this regard.

###  5.3 Single-Source vs. Multi-Source

Figure 13: (Standardized) performance by number of learning sources. Solid
points mark representations including the source performing best with _SS-R_
in the dataset; empty points mark representations without it. Solid and dashed
lines represent linear fits, respectively; dashed areas represent 95%
confidence intervals. Figure 14: Correlation between (standardized) _SS-R_
performance and variance component.

The evidence so far tells us that, _on average_ , learning from multiple
sources leads to better performance than learning from a single source.
However, it could be possible that the _SS-R_ representation with the best
learning source for the given target dataset still performs better than a
multi-source alternative. In fact, in Fig. 10 there are many cases where the
best _SS-R_ representation (black circles at $n=1$) already perform quite well
compared to the more sophisticated alternatives. Fig. 13 presents similar
scatter plots, but now explicitly differentiating between representations
using the single best source (filled circles, solid lines) and not using it
(empty circles, dashed lines). The results suggest that even if the strongest
learning source for the specific dataset is not used, the others largely
compensate for it in the multi-source representations, catching up and even
surpassing the best _SS-R_ representations. The exception to this rule is
again _bpm_ in the Ballroom dataset, where it definitely makes a difference.
As the plots shows, the variability for low numbers of learning sources is
larger when not using the strongest source, but as more sources are added,
this variability reduces.

To further investigate this issue, for each target dataset, we also computed
the variance component due to each of the learning sources, excluding _SS-R_
representations Searle et al. (2006). A large variance due to one of the
sources means that, on average and for that specific dataset, there is a large
difference in performance between having that source or not. Table 6 shows all
variance components, highlighting the per-dataset largest. Apart from _bpm_ in
the Ballroom dataset, there is no clear evidence that one single source is
specially good in all datasets, which suggests that in general there is not a
single source that one would use by default. Notably though, sources _artist_
, _tag_ and _self_ tend to have large variance components.

Table 6: Variance components (as percent of total) of the learning sources, within each of the target datasets, and for non-_SS-R_ representations. Largest per dataset in bold face. | Ballroom | FMA | GTZAN | IRMAS | Lastfm | Arousal | Valence
---|---|---|---|---|---|---|---
_self_ | 2 | 32 | 39 | 18 | 29 | 6 | 10
_year_ |  $<$1 | 6 |  $<$1 | 1 | 2 | 2 |  $<$1
_bpm_ | 96 | 3 |  $<$1 | 8 | 16 |  $<$1 | 42
_taste_ |  $<$1 |  $<$1 |  $<$1 |  $<$1 |  $<$1 |  $<$1 | 6
_tag_ | 1 | 17 | 21 | 16 | 20 | 33 | 14
_lyrics_ |  $<$1 |  $<$1 |  $<$1 | 3 |  $<$1 | 11 |  $<$1
_cdr_tag_ |  $<$1 | 9 | 12 | 16 | 2 | 16 | 14
_artist_ | 1 | 32 | 28 | 37 | 32 | 31 | 15

In addition, we observe that the sources with largest variance are not
necessarily the sources that obtain the best results by themselves in an _SS-
R_ representation (see Fig. 8). We examined this relationship further by
calculating the correlation between variance components and (standardized)
performance of the corresponding _SS-R_ s. The Pearson correlation is $0.38$,
meaning that there is a mild association. Fig. 14 further shows this with a
scatterplot, with a clear distinction between poorly-performing sources
(_year_ , _taste_ and _lyrics_ at the bottom) and well-performing sources
(_tag_ , _cdr_tag_ and _artist_ at the right).

This result implies that even if some _SS-R_ is particularly strong for a
given dataset, when considering more complex fusion architectures, the
presence of that one source is not necessarily required because the other
sources make up for its absence. This is especially important in practical
terms, because different tasks generally have different best sources, and
practitioners rarely have sufficient domain knowledge to select them up front.
Also, and unlike the Ballroom dataset, many real-world problems are not easily
solved with a single feature. Therefore, choosing a more general
representation based on multiple sources is a much simpler way to proceed,
which still yields comparable or better results.

In other words, if “a single deep representation to rule them all” is pre-
trained, it is advisable to base this representation on multiple learning
sources. At the same time, given that _MSS-CR_ representations also generally
show strong performance (albeit that they will bring high dimensionality), and
that they will come ‘for free’ as soon as _SS-R_ networks are trained,
alternatively, we could imagine an ecosystem in which the community could pre-
train and release many _SS-R_ networks for different individual sources in a
distributed way, and practitioners can then collect these into _MSS-CR_
representations, without the need for retraining.

###  5.4 Compactness

Figure 15: Number of network parameters by number of learning sources.

Under an MTDTL setup with branching (the _MS-CR_ architectures), as more
learning sources are used, not only the representation will grow larger, but
so will the necessary deep network to learn it: see Fig. 15 for an overview of
necessary model parameters for the different architectures. When using all the
learning sources, _MS-CR@6_ , which for a considerable part encompasses a
shared network architecture and branches out relatively late, has an around
6.3 times larger network size compared to the network size needed for _SS-R_.
In contrast, _MS-SR@FC_ , which is the most heavily shared MTDTL case, uses a
network that is only 1.2 times larger than the network needed for _SS-R_.

Also, while the representations resulting from the _MSS-CR_ and various _MS-
CR_ architectures linearly depend on the chosen number of learning sources $m$
(see Table 4), for _MS-SR@FC_ , which has a fixed dimensionality of $d$
independent of $m$, we do notice increasing performance as more learning
sources are used, except _IRMAS_ dataset. This implies that under MTDTL
setups, the network does learn as much as possible from the multiple sources,
even in case of fixed network capacity.

###  5.5 Multiple Explanatory Factors

Figure 16: Potential semantic explainability of DTMTL music representations.
Here, we provide a visualization using t-SNE Maaten and Hinton (2008),
plotting 2-dimensional coordinates of each sample from the GTZAN dataset, as
resulting from an _MS-CR_ representation trained on 5 sources111111The
specific model used in the visualization is the $232$th model from the
experimental design we introduce in Section 4.3, which is performing better
than 95% of other models on GTZAN target dataset.. In the zoomed-in panes, we
overlay the strongest topic model terms in $z_{t}$, for various types of
learning sources.

By training representation models on multiple learning sources in the way we
did, our hope is that the representation will reflect latent semantic facets
that will ultimately allow for semantic explainability. In Fig. 11, we show a
visualization that suggests this indeed may be possible. More specifically, we
consider one of our _MS-CR_ models trained on 5 learning sources. For each
learning source-specific block of the representation, using the learning
source-specific fc-out layers, we can predict a factor distribution $z_{t}$
for each of the learning sources. Then, from the predicted $z_{t}$, one can
either map this back on the original learning labels $y_{t}$, or simply
consider the strongest predicted topics (which we visualized in Fig. 11), to
relate the representation to human-understandable facets or
descriptions.121212Note that, as soon as a pre-trained representation network
model will be adapted to an new dataset through transfer learning, the fc-out
layer cannot be used to obtain such explanations from the learning sources
used in the representation learning, since the layers will then be fine-tuned
to another dataset. However, we hypothesize it may be possible that the
semantic explainability can still be preserved, if fine-tuning is jointly
conducted with the original learning sources used during the pre-training time
in the multi-objective strategy.

##  6 Conclusion

In this paper, we have investigated the effect of different strategies to
learn music representations with deep networks, considering multiple learning
sources and different network architectures with varying degrees of shared
information. Our main research questions are how the number and combination of
learning sources (RQ1), and different configurations of the shared
architecture (RQ2) affect effectiveness of the learned deep music
representation. As a consequence, we conducted an experiment training 425
neural network models with different combinations of learning sources and
architectures.

After an extensive empirical analysis, we can summarize our findings as
follows:

  * •

RQ1 The number of learning sources positively affects the effectiveness of a
learned deep music representation, although representations based on a single
learning source will already be effective in specialized cases (e.g. BPM and
the Ballroom dataset).

  * •

RQ2 In terms of architecture, the amount of shared information has a negative
effect on performance: larger models with less shared information (e.g.  _MS-
CR@2_ , _MSS-CR_) tend to outperform models where sharing is higher (e.g.
_MS-CR@6_ , _MS-SR@FC_), all of which outperform the base model (_SS-R_).

Our findings give various pointers to useful future work. First of all,
‘generality’ is difficult to define in the music domain, maybe more so than in
CV or NLP, in which lower-level information atoms may be less multifaceted in
nature (e.g. lower-level representations of visual objects naturally extend to
many vision tasks, while an equivalent in music is harder to pinpoint). In
case of clear task-specific data skews, practitioners should be pragmatic
about this.

Also, we only investigated one special case of transfer learning, which might
not be generalized well if one considers the adaptation of the pre-trained
network for further fine-tuning with respect to their target dataset. Since
there are various choices to make, which will bring substantial amount of
variability, we decided to leave the aspects for further future works. We
believe open-sourcing the models we trained throughout this work will be
helpful for such follow-up works. Another limitation of current work is the
selective set of label types in the learning sources. For instance, there are
also a number of MIR related tasks that are using time-variant labels such as
automatic music transcription, segmentation, beat tracking and chord
estimation. We believe that such tasks should be investigated as well in the
future to build a more complete overview of MTDTL problem.

Finally, in our current work, we still largely considered MTDTL as a ‘black
box’ operation, trying to learn _how_ MTDTL can be effective. However, the
original reason for starting this work was not only to yield an effective
general-purpose representation, but one that also would be semantically
interpretable according to different semantic facets. We showed some early
evidence our representation networks may be capable of picking up such facets;
however, considerable future work will be needed into more in-depth analysis
techniques of _what_ the deep representations actually learned.

###### Acknowledgements.

This work was carried out on the Dutch national e-infrastructure with the
support of SURF Cooperative. We further thank the CDR for having provided
their album-level genre annotations for our experiments. We thank Keunwoo Choi
for the discussion and all the help regarding the implementation of his work.
We also thank David Tax for the valuable inputs and discussion. Finally, we
thank editors and reviewers for their effort and constructive help to improve
this work.

Conflict of interest: Jaehun Kim, Julián Urbano, Cynthia C. S. Liem and Alan
Hanjalic state that there are no conflicts of interest.

## References

  * Casey et al. [2008] Michael A. Casey, Remco C. Veltkamp, Masataka Goto, Marc Leman, Christophe Rhodes, and Malcolm Slaney.  Content-based music information retrieval: Current directions and future challenges.  _Proceedings of the IEEE_ , 96(4):668–696, 2008\.  doi: 10.1109/JPROC.2008.916370.
  * Caruana [1997] Rich Caruana.  Multitask learning.  _Machine Learning_ , 28(1):41–75, July 1997.  ISSN 1573-0565.  doi: 10.1023/A:1007379606734.
  * Bengio et al. [2013] Yoshua Bengio, Aaron C. Courville, and Pascal Vincent.  Representation learning: A review and new perspectives.  _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , 35(8):1798–1828, August 2013.  ISSN 0162-8828.  doi: 10.1109/TPAMI.2013.50.
  * Liu et al. [2015] Wu Liu, Tao Mei, Yongdong Zhang, Cherry Che, and Jiebo Luo.  Multi-task deep visual-semantic embedding for video thumbnail selection.  In _IEEE Conference on Computer Vision and Pattern Recognition CVPR_ , pages 3707–3715, Boston, MA, USA, June 2015.  doi: 10.1109/CVPR.2015.7298994.
  * Bingel and Søgaard [2017] Joachim Bingel and Anders Søgaard.  Identifying beneficial task relations for multi-task learning in deep neural networks.  In _Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 2_ , pages 164–169, Valencia, Spain, April 2017. Association for Computational Linguistics.
  * Li et al. [2015] Sijin Li, Zhi-Qiang Liu, and Antoni B. Chan.  Heterogeneous multi-task learning for human pose estimation with deep convolutional neural network.  _International Journal of Computer Vision_ , 113(1):19–36, May 2015.  ISSN 1573-1405.  doi: 10.1007/s11263-014-0767-8.
  * Zhang et al. [2015] Wenlu Zhang, Rongjian Li, Tao Zeng, Qian Sun, Sudhir Kumar, Jieping Ye, and Shuiwang Ji.  Deep model based transfer and multi-task learning for biological image analysis.  In _Proceedings of the 21th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining KDD_ , pages 1475–1484, Sydney, NSW, Australia, 2015. ACM.  ISBN 978-1-4503-3664-2.  doi: 10.1145/2783258.2783304.
  * Zhang et al. [2014] Zhanpeng Zhang, Ping Luo, Chen Change Loy, and Xiaoou Tang.  Facial landmark detection by deep multi-task learning.  In _Computer Vision - ECCV 13th European Conference, Proceedings, Part VI_ , pages 94–108, Zurich, Switzerland, September 2014. Springer.  doi: 10.1007/978-3-319-10599-4˙7.
  * Kaiser et al. [2017] Lukasz Kaiser, Aidan N. Gomez, Noam Shazeer, Ashish Vaswani, Niki Parmar, Llion Jones, and Jakob Uszkoreit.  One model to learn them all.  _CoRR_ , abs/1706.05137, 2017.  URL https://arxiv.org/abs/1706.05137.
  * Chang et al. [2017] Jen-Hao Rick Chang, Chun-Liang Li, Barnabás Póczos, and B. V. K. Vijaya Kumar.  One network to solve them all - solving linear inverse problems using deep projection models.  In _IEEE International Conference on Computer Vision, ICCV_ , pages 5889–5898, Venice, Italy, October 2017. IEEE Computer Society.  doi: 10.1109/ICCV.2017.627.
  * Weston et al. [2011] Jason Weston, Samy Bengio, and Philippe Hamel.  Multi-tasking with joint semantic spaces for large-scale music annotation and retrieval.  _Journal of New Music Research_ , 40(4):337–348, 2011.  doi: 10.1080/09298215.2011.603834.
  * Aytar et al. [2016] Yusuf Aytar, Carl Vondrick, and Antonio Torralba.  Soundnet: Learning sound representations from unlabeled video.  In _Advances in Neural Information Processing Systems 29: Annual Conference on Neural Information Processing Systems_ , pages 892–900, Barcelona, Spain, December 2016.
  * Hamel and Eck [2010] Philippe Hamel and Douglas Eck.  Learning features from music audio with deep belief networks.  In _Proceedings of the 11th International Society for Music Information Retrieval Conference, ISMIR_ , pages 339–344, Utrecht, Netherlands, August 2010.
  * Boulanger-Lewandowski et al. [2012] Nicolas Boulanger-Lewandowski, Yoshua Bengio, and Pascal Vincent.  Modeling temporal dependencies in high-dimensional sequences: Application to polyphonic music generation and transcription.  In _Proceedings of the 29th International Conference on Machine Learning, ICML_ , Edinburgh, Scotland, UK, June 2012. Omnipress.
  * Schlüter and Böck [2014] Jan Schlüter and Sebastian Böck.  Improved musical onset detection with convolutional neural networks.  In _IEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP_ , pages 6979–6983, Florence, Italy, May 2014. IEEE.  doi: 10.1109/ICASSP.2014.6854953.
  * Choi et al. [2016] Keunwoo Choi, György Fazekas, and Mark B. Sandler.  Automatic tagging using deep convolutional neural networks.  In _Proceedings of the 17th International Society for Music Information Retrieval Conference, ISMIR_ , pages 805–811, New York City, USA, August 2016.
  * van den Oord et al. [2013] Aäron van den Oord, Sander Dieleman, and Benjamin Schrauwen.  Deep content-based music recommendation.  In _Advances in Neural Information Processing Systems 26 NIPS_ , pages 2643–2651, Lake Tahoe, NV, USA, December 2013.
  * Chandna et al. [2017] Pritish Chandna, Marius Miron, Jordi Janer, and Emilia Gómez.  Monoaural audio source separation using deep convolutional neural networks.  In _Latent Variable Analysis and Signal Separation - 13th International Conference, LVA/ICA, Proceedings_ , pages 258–266, Grenoble, France, February 2017.  ISBN 978-3-319-53547-0.  doi: 10.1007/978-3-319-53547-0˙25.
  * Jeong and Lee [2016] Il-Young Jeong and Kyogu Lee.  Learning temporal features using a deep neural network and its application to music genre classification.  In _Proceedings of the 17th International Society for Music Information Retrieval Conference, ISMIR_ , pages 434–440, New York City, USA, August 2016.
  * Han et al. [2017] Yoonchang Han, Jae-Hun Kim, and Kyogu Lee.  Deep convolutional neural networks for predominant instrument recognition in polyphonic music.  _IEEE/ACM Transactions on Audio, Speech and Language Processing_ , 25(1):208–221, Jan 2017.  ISSN 2329-9290.  doi: 10.1109/TASLP.2016.2632307.
  * Simonyan and Zisserman [2015] Karen Simonyan and Andrew Zisserman.  Very deep convolutional networks for large-scale image recognition.  In _3th International Conference on Learning Representations, ICLR_ , San Diego, CA, USA, May 2015.
  * He et al. [2016] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In _IEEE Conference on Computer Vision and Pattern Recognition, CVPR_ , pages 770–778, Las Vegas, NV, USA, June 2016. IEEE Computer Society.  doi: 10.1109/CVPR.2016.90.
  * Szegedy et al. [2015] Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott E. Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich.  Going deeper with convolutions.  In _IEEE Conference on Computer Vision and Pattern Recognition, CVPR_ , pages 1–9, Boston, MA, USA, June 2015. IEEE Computer Society.  doi: 10.1109/CVPR.2015.7298594.
  * Mikolov et al. [2013] Tomas Mikolov, Ilya Sutskever, Kai Chen, Gregory S. Corrado, and Jeffrey Dean.  Distributed representations of words and phrases and their compositionality.  In _Advances in Neural Information Processing Systems 26 NIPS_ , pages 3111–3119, Lake Tahoe, NV, USA, December 2013.
  * Dieleman et al. [2011] Sander Dieleman, Philémon Brakel, and Benjamin Schrauwen.  Audio-based music classification with a pretrained convolutional network.  In _Proceedings of the 12th International Society for Music Information Retrieval Conference, ISMIR_ , pages 669–674, Miami, FL, USA, October 2011. University of Miami.  ISBN 9780615548654.
  * Choi et al. [2017] Keunwoo Choi, György Fazekas, Mark B. Sandler, and Kyunghyun Cho.  Transfer learning for music classification and regression tasks.  In _Proceedings of the 18th International Society for Music Information Retrieval Conference, ISMIR_ , pages 141–149, Suzhou, China, October 2017.
  * van den Oord et al. [2014] Aäron van den Oord, Sander Dieleman, and Benjamin Schrauwen.  Transfer learning by supervised pre-training for audio-based music classification.  In _Proceedings of the 15th International Society for Music Information Retrieval Conference, ISMIR_ , pages 29–34, Taipei, Taiwan, October 2014.
  * Liang et al. [2015] Dawen Liang, Minshu Zhan, and Daniel P. W. Ellis.  Content-aware collaborative music recommendation using pre-trained neural networks.  In _Proceedings of the 16th International Society for Music Information Retrieval Conference, ISMIR_ , pages 295–301, Málaga, Spain, October 2015.
  * Misra et al. [2016] Ishan Misra, Abhinav Shrivastava, Abhinav Gupta, and Martial Hebert.  Cross-stitch networks for multi-task learning.  In _IEEE Conference on Computer Vision and Pattern Recognition, CVPR_ , pages 3994–4003, Las Vegas, NV, USA, June 2016. IEEE Computer Society.
  * Bertin-Mahieux et al. [2011] Thierry Bertin-Mahieux, Daniel P. W. Ellis, Brian Whitman, and Paul Lamere.  The million song dataset.  In _Proceedings of the 12th International Society for Music Information Retrieval Conference, ISMIR_ , pages 591–596, Miami, FL, USA, October 2011. University of Miami.
  * Bengio et al. [2006] Yoshua Bengio, Pascal Lamblin, Dan Popovici, and Hugo Larochelle.  Greedy layer-wise training of deep networks.  In _Advances in Neural Information Processing Systems 19, NIPS_ , pages 153–160, Vancouver, BC, Canada, December 2006. MIT Press.
  * Vincent et al. [2008] Pascal Vincent, Hugo Larochelle, Yoshua Bengio, and Pierre-Antoine Manzagol.  Extracting and composing robust features with denoising autoencoders.  In _Proceedings of the 25th International Conference on Machine Learning ICML_ , pages 1096–1103, Helsinki, Finland, June 2008. ACM.  doi: 10.1145/1390156.1390294.
  * Smolensky [1986] Paul Smolensky.  Information processing in dynamical systems: Foundations of harmony theory.  Technical report, University of Colorado, Boulder, Dept. of Computer Science, 1986.
  * Hinton et al. [2006] Geoffrey E. Hinton, Simon Osindero, and Yee-Whye Teh.  A fast learning algorithm for deep belief nets.  _Neural Computation_ , 18(7):1527–1554, 2006\.  doi: 10.1162/neco.2006.18.7.1527.  PMID: 16764513.
  * Goodfellow et al. [2014] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio.  Generative adversarial nets.  In _Advances in Neural Information Processing Systems 27, NIPS_ , pages 2672–2680. Curran Associates, Inc., Montreal, QC, Canada, December 2014.
  * Han et al. [2015] Xufeng Han, Thomas Leung, Yangqing Jia, Rahul Sukthankar, and Alexander C. Berg.  Matchnet: Unifying feature and metric learning for patch-based matching.  In _IEEE Conference on Computer Vision and Pattern Recognition, CVPR_ , pages 3279–3286, Boston, MA, USA, June 2015. IEEE Computer Society.  doi: 10.1109/CVPR.2015.7298948.
  * Arandjelovic and Zisserman [2017] Relja Arandjelovic and Andrew Zisserman.  Look, listen and learn.  In _IEEE International Conference on Computer Vision, ICCV_ , pages 609–617, Venice, Italy, October 2017. IEEE Computer Society.  doi: 10.1109/ICCV.2017.73.
  * Huang et al. [2018] Yu-Siang Huang, Szu-Yu Chou, and Yi-Hsuan Yang.  Generating music medleys via playing music puzzle games.  In _Proceedings of the Thirty-Second Conference on Artificial Intelligence, AAAI_ , pages 2281–2288, New Orleans, LA, USA, February 2018. AAAI Press.
  * Salton and McGill [1984] Gerard Salton and Michael McGill.  _Introduction to Modern Information Retrieval_.  McGraw-Hill Book Company, 1984.  ISBN 0-07-054484-0.
  * Lamere [2008] Paul Lamere.  Social tagging and music information retrieval.  _Journal of New Music Research_ , 37(2):101–114, 2008.  ISSN 0929-8215.  doi: 10.1080/09298210802479284.
  * Hamel et al. [2013] Philippe Hamel, Matthew E. P. Davies, Kazuyoshi Yoshii, and Masataka Goto.  Transfer learning in MIR: Sharing learned latent representations for music audio classification and similarity.  In _Proceedings of the 14th International Society for Music Information Retrieval Conference, ISMIR_ , pages 9–14, Curitiba, Brazil, November 2013.
  * Law et al. [2010] Edith Law, Burr Settles, and Tom M. Mitchell.  Learning to tag from open vocabulary labels.  In _Machine Learning and Knowledge Discovery in Databases, European Conference, ECML PKDD, Proceedings, Part II_ , pages 211–226, Barcelona, Spain, September 2010. Springer.
  * Hofmann [1999] Thomas Hofmann.  Probabilistic latent semantic analysis.  In _UAI: Proceedings of the Fifteenth Conference on Uncertainty in Artificial Intelligence_ , pages 289–296, Stockholm, Sweden, July 1999. Morgan Kaufmann.
  * Schlüter [2016] Jan Schlüter.  Learning to pinpoint singing voice from weakly labeled examples.  In _Proceedings of the 17th International Society for Music Information Retrieval Conference, ISMIR_ , pages 44–50, New York City, USA, August 2016.
  * Hershey et al. [2017] Shawn Hershey, Sourish Chaudhuri, Daniel P. W. Ellis, Jort F. Gemmeke, Aren Jansen, R. Channing Moore, Manoj Plakal, Devin Platt, Rif A. Saurous, Bryan Seybold, Malcolm Slaney, Ron J. Weiss, and Kevin W. Wilson.  CNN architectures for large-scale audio classification.  In _IEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP_ , pages 131–135, New Orleans, LA, USA, March 2017\. IEEE.  doi: 10.1109/ICASSP.2017.7952132.
  * Lee et al. [2009] Honglak Lee, Peter T. Pham, Yan Largman, and Andrew Y. Ng.  Unsupervised feature learning for audio classification using convolutional deep belief networks.  In _Advances in Neural Information Processing Systems 22, NIPS_ , pages 1096–1104, Vancouver, BC, Canada, December 2009. Curran Associates, Inc.
  * Humphrey and Bello [2012] Eric J. Humphrey and Juan Pablo Bello.  Rethinking automatic chord recognition with convolutional neural networks.  In _11th International Conference on Machine Learning and Applications, ICMLA_ , pages 357–362, Boca Raton, FL, USA, December 2012. IEEE.  doi: 10.1109/ICMLA.2012.220.
  * Nakashika et al. [2012] Toru Nakashika, Christophe Garcia, and Tetsuya Takiguchi.  Local-feature-map integration using convolutional neural networks for music genre classification.  In _INTERSPEECH, 13th Annual Conference of the International Speech Communication Association_ , pages 1752–1755, Portland, OR, USA, September 2012. ISCA.
  * Ullrich et al. [2015] Karen Ullrich, Jan Schlüter, and Thomas Grill.  Boundary detection in music structure analysis using convolutional neural networks.  In _Proceedings of the 16th International Society for Music Information Retrieval Conference, ISMIR_ , pages 417–422, Málaga, Spain, October 2015.
  * Piczak [2015] Karol J. Piczak.  Environmental sound classification with convolutional neural networks.  In _25th IEEE International Workshop on Machine Learning for Signal Processing, MLSP_ , pages 1–6, Boston, MA, USA, September 2015. IEEE.  doi: 10.1109/MLSP.2015.7324337.
  * Simpson et al. [2015] Andrew J. R. Simpson, Gerard Roma, and Mark D. Plumbley.  Deep karaoke: Extracting vocals from musical mixtures using a convolutional deep neural network.  In _Latent Variable Analysis and Signal Separation - 12th International Conference, LVA/ICA, Proceedings_ , pages 429–436, Liberec, Czech Republic, August 2015. Springer.  ISBN 978-3-319-22482-4.  doi: 10.1007/978-3-319-22482-4˙50.
  * Phan et al. [2016] Huy Phan, Lars Hertel, Marco Maaß, and Alfred Mertins.  Robust audio event recognition with 1-max pooling convolutional neural networks.  In _INTERSPEECH 17th Annual Conference of the International Speech Communication Association_ , pages 3653–3657, San Francisco, CA, USA, September 2016. ISCA.  doi: 10.21437/Interspeech.2016-123.
  * Pons et al. [2016] Jordi Pons, Thomas Lidy, and Xavier Serra.  Experimenting with musically motivated convolutional neural networks.  In _14th International Workshop on Content-Based Multimedia Indexing, CBMI_ , pages 1–6, Bucharest, Romania, June 2016. IEEE.  doi: 10.1109/CBMI.2016.7500246.
  * Stasiak and Monko [2016] Bartlomiej Stasiak and Jedrzej Monko.  Analysis of time-frequency representations for musical onset detection with convolutional neural network.  In _Proceedings of the Federated Conference on Computer Science and Information Systems, FedCSIS_ , pages 147–152, Gdańsk, Poland, September 2016.  doi: 10.15439/2016F558.
  * Su et al. [2016] Hong Su, Hui Zhang, Xueliang Zhang, and Guanglai Gao.  Convolutional neural network for robust pitch determination.  In _IEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP_ , pages 579–583, Shanghai, China, March 2016. IEEE.  doi: 10.1109/ICASSP.2016.7471741.
  * Krizhevsky et al. [2017] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E. Hinton.  Imagenet classification with deep convolutional neural networks.  _Communications of the ACM_ , 60(6):84–90, May 2017.  doi: 10.1145/3065386.
  * Dieleman and Schrauwen [2014] Sander Dieleman and Benjamin Schrauwen.  End-to-end learning for music audio.  In _IEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP_ , pages 6964–6968, Florence, Italy, May 2014. IEEE.  doi: 10.1109/ICASSP.2014.6854950.
  * van den Oord et al. [2016] Aäron van den Oord, Sander Dieleman, Heiga Zen, Karen Simonyan, Oriol Vinyals, Alex Graves, Nal Kalchbrenner, Andrew W. Senior, and Koray Kavukcuoglu.  Wavenet: A generative model for raw audio.  In _The 9th ISCA Speech Synthesis Workshop, SSW_ , page 125, Sunnyvale, CA, USA, September 2016. ISCA.
  * Jaitly and Hinton [2011] Navdeep Jaitly and Geoffrey E. Hinton.  Learning a better representation of speech soundwaves using restricted boltzmann machines.  In _IEEE International Conference on Acoustics, Speech, and Signal Processing, ICASSP_ , pages 5884–5887, Prague, Czech Republic, May 2011\. IEEE.  doi: 10.1109/ICASSP.2011.5947700.
  * Lee et al. [2017] Jongpil Lee, Jiyoung Park, Keunhyoung Luke Kim, and Juhan Nam.  Sample-level deep convolutional neural networks for music auto-tagging using raw waveforms.  In _14th Sound and Music Computing Conference, SMC_ , Espoo, Finland, July 2017.
  * Ioffe and Szegedy [2015] Sergey Ioffe and Christian Szegedy.  Batch normalization: Accelerating deep network training by reducing internal covariate shift.  In _Proceedings of the 32nd International Conference on Machine Learning, ICML_ , pages 448–456, Lille, France, July 2015. JMLR, Inc.
  * Nair and Hinton [2010] Vinod Nair and Geoffrey E. Hinton.  Rectified linear units improve restricted boltzmann machines.  In _Proceedings of the 27th International Conference on Machine Learning ICML_ , pages 807–814, Haifa, Israel, June 2010. Omnipress.
  * Srivastava et al. [2014] Nitish Srivastava, Geoffrey E. Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov.  Dropout: A simple way to prevent neural networks from overfitting.  _Journal of Machine Learning Research_ , 15(1):1929–1958, 2014.
  * Nam et al. [2012] Juhan Nam, Jorge Herrera, Malcolm Slaney, and Julius O. Smith.  Learning sparse feature representations for music annotation and retrieval.  In _Proceedings of the 13th International Society for Music Information Retrieval Conference, ISMIR_ , pages 565–570, Porto, Portugal, October 2012. FEUP Edições.
  * Choi et al. [2018] Keunwoo Choi, György Fazekas, Mark B. Sandler, and Kyunghyun Cho.  A comparison of audio signal preprocessing methods for deep neural networks on music tagging.  In _26th European Signal Processing Conference, EUSIPCO_ , pages 1870–1874, Roma, Italy, September 2018. IEEE.
  * Dörfler et al. [2018] Monika Dörfler, Thomas Grill, Roswitha Bammer, and Arthur Flexer.  Basic filters for convolutional neural networks applied to music: Training or design?  _Neural Computing and Applications_ , Sep 2018.  ISSN 1433-3058.  doi: 10.1007/s00521-018-3704-x.
  * Kingma and Ba [2015] Diederik P. Kingma and Jimmy Ba.  Adam: A method for stochastic optimization.  In _3th International Conference on Learning Representations, ICLR_ , San Diego, CA, USA, May 2015.
  * Paszke et al. [2017] Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer.  Automatic differentiation in PyTorch.  In _NIPS-W_ , 2017.
  * Pedregosa et al. [2012] Fabian Pedregosa, Gaël Varoquaux, Alexandre Gramfort, Vincent Michel, Bertrand Thirion, Olivier Grisel, Mathieu Blondel, Peter Prettenhofer, Ron Weiss, Vincent Dubourg, Jake Vanderplas, Alexandre Passos, David Cournapeau, Matthieu Brucher, Matthieu Perrot, and Édouard Duchesnay.  Scikit-learn: Machine Learning in Python.  _Journal of Machine Learning Research_ , 12:2825–2830, 2012\.  ISSN 15324435.  doi: 10.1007/s13398-014-0173-7.2.
  * McFee et al. [2015] Brian McFee, Colin Raffel, Dawen Liang, Daniel P. W. Ellis, Matt McVicar, Eric Battenberg, and Oriol Nieto.  librosa: Audio and music signal analysis in python.  In Kathryn Huff and James Bergstra, editors, _Proceedings of the 14th Python in Science Conference SciPy_ , pages 18 – 24, Austin, TX, USA, July 2015.  doi: 10.25080/Majora-7b98e3ed-003.
  * Defferrard et al. [2017] Michaël Defferrard, Kirell Benzi, Pierre Vandergheynst, and Xavier Bresson.  FMA: A dataset for music analysis.  In _Proceedings of the 18th International Society for Music Information Retrieval Conference, ISMIR_ , pages 316–323, Suzhou, China, October 2017.
  * Tzanetakis and Cook [2002] George Tzanetakis and Perry R. Cook.  Musical genre classification of audio signals.  _IEEE Transactions on Speech and Audio Processing_ , 10(5):293–302, July 2002.  ISSN 1063-6676.  doi: 10.1109/TSA.2002.800560.
  * Kereliuk et al. [2015] Corey Kereliuk, Bob L. Sturm, and Jan Larsen.  Deep learning and music adversaries.  _IEEE Transactions on Multimedia_ , 17(11):2059–2071, November 2015.  ISSN 1520-9210.  doi: 10.1109/TMM.2015.2478068.
  * Gouyon et al. [2006] Fabien Gouyon, Anssi Klapuri, Simon Dixon, M. Alonso, George Tzanetakis, C. Uhle, and Pedro Cano.  An experimental comparison of audio tempo induction algorithms.  _IEEE Transactions on Audio, Speech, and Language Processing_ , 14(5):1832–1844, September 2006.  ISSN 1558-7916.  doi: 10.1109/TSA.2005.858509.
  * Marchand and Peeters [2016] Ugo Marchand and Geoffroy Peeters.  Scale and shift invariant time/frequency representation using auditory statistics: Application to rhythm description.  In _26th IEEE International Workshop on Machine Learning for Signal Processing, MLSP_ , pages 1–6, Salerno, Italy, September 2016. IEEE.  doi: 10.1109/MLSP.2016.7738904.
  * Bosch et al. [2012] Juan J. Bosch, Jordi Janer, Ferdinand Fuhrmann, and Perfecto Herrera.  A comparison of sound segregation techniques for predominant instrument recognition in musical audio signals.  In _Proceedings of the 13th International Society for Music Information Retrieval Conference, ISMIR_ , pages 559–564, Porto, Portugal, October 2012. FEUP Edições.
  * Soleymani et al. [2013] Mohammad Soleymani, Michael N. Caro, Erik M. Schmidt, Cheng-Ya Sha, and Yi-Hsuan Yang.  1000 songs for emotional analysis of music.  In _Proceedings of the 2nd ACM International Workshop on Crowdsourcing for Multimedia CrowdMM@ACM Multimedia_ , pages 1–6, Barcelona, Spain, 2013. ACM.  ISBN 978-1-4503-2396-3.  doi: 10.1145/2506364.2506365.
  * Celma [2010] Òscar Celma.  _Music Recommendation and Discovery - The Long Tail, Long Fail, and Long Play in the Digital Music Space_.  Springer, 2010.  ISBN 978-3-642-13286-5.  doi: 10.1007/978-3-642-13287-2.
  * Sturm [2014] Bob L. Sturm.  The state of the art ten years after a state of the art: Future research in music information retrieval.  _Journal of New Music Research_ , 43(2):147–172, 2014.  doi: 10.1080/09298215.2014.894533.
  * Sturm [2016] Bob L. Sturm.  The “Horse” inside: Seeking causes behind the behaviors of music content analysis systems.  _Computers in Entertainment_ , 14(2):3:1–3:32, 2016.  doi: 10.1145/2967507.
  * Posner et al. [2005] Jonathan Posner, James A. Russell, and Bradley S. Peterson.  The circumplex model of affect: An integrative approach to affective neuroscience, cognitive development, and psychopathology.  _Development and Psychopathology_ , 17(3):715–734, 2005.  ISSN 1469-2198.  doi: 10.1017/S0954579405050340.
  * Montgomery [2012] Douglas C. Montgomery.  _Design and Analysis of Experiments_.  Wiley, 8th edition, 2012.
  * Goos and Jones [2011] Peter Goos and Bradley Jones.  _Optimal Design of Experiments: A Case Study Approach_.  Wiley, 1st edition, 2011.
  * Hinton [1989] Geoffrey E. Hinton.  Connectionist learning procedures.  _Artificial Intelligence_ , 40(1):185 – 234, 1989\.  ISSN 0004-3702.  doi: 10.1016/0004-3702(89)90049-0.
  * Hu et al. [2008] Yifan Hu, Yehuda Koren, and Chris Volinsky.  Collaborative filtering for implicit feedback datasets.  In _Proceedings of the 8th IEEE International Conference on Data Mining (ICDM)_ , pages 263–272, Pisa, Italy, December 2008. IEEE Computer Society.  doi: 10.1109/ICDM.2008.22.
  * Gelman and Hill [2006] Andrew Gelman and Jennifer Hill.  _Data Analysis Using Regression and Multilevel/Hierarchical Models_.  Press, Cambridge University, 2006.
  * Searle et al. [2006] Shayle R. Searle, George Casella, and Charles E. McCulloch.  _Variance components_.  Wiley, 2006.
  * Maaten and Hinton [2008] Laurens van der Maaten and Geoffrey Hinton.  Visualizing data using t-SNE.  _Journal of machine learning research_ , 9(November):2579–2605, 2008.