* Boselli, Fossati & Sun (2021) Boselli, A., Fossati, M., & Sun, M. 2021, in press
  * Bothun & Dressler (1986) Bothun, G. D., & Dressler, A. 1986, ApJ, 301, 57
  * Bravo-Alfaro et al. (2000) Bravo-Alfaro H., Cayatte V., van Gorkom J. H., & Balkowski C. 2000, AJ, 119, 580
  * Brown & Rudnick (2011) Brown, S., & Rudnick, L. 2011, MNRAS, 412, 2
  * Brunetti, Setti & Comastri (1997) Brunetti, G., Setti, G., & Comastri, A. 1997, A&A, 325, 898
  * Brunetti, & Lazarian (2011) Brunetti, G., & Lazarian, A. 2011, MNRAS, 410, 127
  * Brunetti & Jones (2014) Brunetti, G., & Jones, T. W. 2014, Int. Jl. of Modern Physics D, 23
  * Brunetti, & Lazarian (2016) Brunetti, G., & Lazarian, A. 2016, MNRAS, 458, 2584
  * Burns (1990) Burns, J. O. 1990, AJ, 99, 14
  * Bykov et al. (2019) Bykov, A. M., Vazza, F., Kropotina, J. A., et al. 2019, Space Sci. Rev., 215, 14
  * Capetti et al. (2000) Capetti, A., de Ruiter, H. R., Fanti, R., et al. 2000, A&A, 362, 871
  * Chen et al. (2020) Chen, H., Sun, M., Yagi, M., et al. 2020, MNRAS, 496, 4654
  * Churazov et al. (2003) Churazov, E., Forman, W. R., Jones, C., & Böhringer, H. 2003, ApJ, 590, 225
  * Churazov et al. (2012) Churazov, E., Vikhlinin, A., Zhuravleva, I., et al. 2012, MNRAS, 421, 1123
  * Churazov et al. (2021) Churazov, E., Khabibullin, I., Lyskova, N., Sunyaev, R., & Bykov, A. M. 2021, A&A, 651, A41
  * Crowl et al. (2005) Crowl H. H., Kenney J. D. P., van Gorkom J. H., & Vollmer B. 2005, AJ, 130, 65
  * Curtis (1918) Curtis, H. D. 1918, Publ. of Lick Obs., 13, 9
  * Fanti et al. (1990) Fanti, R., Fanti, C., Schilizzi, R. T., et al. 1990, A&A, 231, 333
  * Feretti et al. (1990) Feretti, L., Dallacasa, D., Giovannini, G., & Venturi, T. 1990, A&A, 232, 337
  * Feretti & Venturi (2002) Feretti, L., & Venturi, T. 2002, in Merging Processes in Galaxy Clusters 272, Astrophysics and Space Science Library, ed. L. Feretti, I. M. Gioia & G. Giovannini (Dordrecht), 163
  * Gavazzi (1989) Gavazzi, G. 1989, AJ, 346, 59
  * Giovannini, Feretti & Stanghellini (1991) Giovannini, G., Feretti, L, & Stanghellini, C. 1991, A&A, 252, 528
  * Giovannini et al. (1993) Giovannini, G., Feretti, L., Venturi, T., Kim, K. -T., & Kronberg, P. P. 1993, ApJ, 406, 399
  * Grishin et al. (2021) Grishin, K. A., Chilingarian, I., Afanasiev, A. V., et al. 2021, Nat. Astron. 5, 1308
  * Gunn & Gott (1972) Gunn, J. E., & Gott, J. R. III 1972, ApJ, 176, 1
  * Gupta et al. (2017) Gupta, Y., Ajithkumar, B., Kale, H., et al. 2017, Current Science, 113, 707
  * Herschel (1785) Herschel, W. 1785, Phil. Trans. of the Roy. Soc. of London, 75, 213
  * Huchra et al. (1990) Huchra, J. P., Geller, M. J., de Lapparent, V., & Corwin, H. G. Jr. 1990 ApJS 72, 433
  * Jaffe & Perola (1973) Jaffe, W. J., & Perola, G. C. 1973, A&A, 26, 423
  * Joye & Mandel (2003) Joye, W. A., & Mandel, E. 2003, in ASP Conf. Ser., 295, Astronomical Data Analysis Software and Systems, ed. H. E. Payne, R. I. Jedrzejewski, & R. N. Hook (San Francisco, CA: ASP), 489.
  * Kenney, Abramson & Bravo-Alfaro (2015) Kenney, J. D. P., Abramson, A., & Bravo-Alfaro, H. 2015 AJ 150, 59
  * Kent & Gunn (1982) Kent, S. M., & Gunn, J. E. 1982 AJ 87, 945
  * Kim et al. (1989) Kim, K.-T., Kronberg, P. P., Giovannini, G., & Venturi, T. 1989, Nat., 341, 720
  * Kim et al. (1994) Kim, K.-T., Kronberg, P. P., Dewdney, P. E., & Landecker, T. L. 1994, ApJS, 105, 385
  * Koopmann & Kenney (2004) Koopmann R. A., & Kenney J. D. P. 2004, ApJ, 613, 866
  * Lal & Rao (2004) Lal, D. V., & Rao, A. P. 2004, A&A, 420, 491
  * Lal (2020a) Lal, D. V. 2020a, ApJS, 250, 22
  * Lal (2020b) Lal, D. V. 2020b, AJ, 160, 161
  * Lyskova et al. (2019) Lyskova, N. Churazov, E., Zhang, C., et al. 2019, MNRAS, 485, 2922
  * Mazzarella & Balzano (1986) Mazzarella, J. M., & Balzano V. A. 1986, ApJS, 62, 751
  * McMullin et al. (2007) McMullin, J. P., Waters, B., Schiebel, D., Young, W., & Golap, K. 2007, in ASP Conf. Ser., 376, Astronomical Data Analysis Software and Systems, ed. R. A. Shaw, F. Hill, & D. J. Bell (San Francisco, CA: ASP), 127
  * Miley (1980) Miley, G. K. 1980, ARA&A, 18, 165
  * Miller & Owen (2002) Miller, N. A., & Owen, F. N. 2002, AJ, 124, 2453
  * Miller, Hornschemeier & Mobasher (2009) Miller, N. A., Hornschemeier, A. E., Mobasher, B. 2009, AJ, 137, 4436
  * Neumann et al. (2001) Neumann, D. M., Arnaud, M., Gastaud, R., et al. 2001, A&A, 365, L74
  * Nulsen (1982) Nulsen P. E. J. 1982, MNRAS, 198, 1007
  * O’Dea (1998) , O’Dea, C. 1998, PASP, 110, 493
  * Oemler (1976) Oemler, A. 1976, ApJ, 209, 693
  * Ozawa et al. (2015) Ozawa, T., Nakanishi, H., Akahori, T., et al. 2015, PASJ, 67, 110
  * Perley & Butler (2017) Perley, R. A., & Butler, B. J. 2017, ApJS 230, 7
  * Pinzke, Oh & Pfrommer (2017) Pinzke, A., Oh, S. P., & Pfrommer, C. 2017, MNRAS, 465, 4800
  * Planck Collaboration (2013) Planck Collaboration 2013, A&A, 554, A140
  * Roberts & Parker (2020) Roberts, I. D., & Parker, L. C. 2020, MNRAS, 495, 554
  * Roberts et al. (2021) Roberts, I. D., van Weeren, R. J., McGee, S. L., et al. 2021, A&A, 650, A111
  * Robitaille et al. (2013) Robitaille, T. P., Tollerud, E. J., Greenfield, P., et al. 2013, A&A, 558, A33
  * Rudnick (2021) Rudnick, L. 2021, Galaxies, 9, 85
  * Sanders et al. (2014) Sanders J. S., Fabian A. C., Sun M., et al. 2014, MNRAS, 439, 1182
  * Sheardown et al. (2019) Sheardown, A., Fish, T. M., Roediger, E., et al. 2019, ApJ, 874, 112
  * Simionescu et al. (2013) Simionescu A., Werner N., Urban O., et al., 2013, ApJ, 775, 4
  * Smith et al. (2010) Smith R. J., Lucey, J. R., Hammer, D., et al. 2010, MNRAS 408, 1417
  * Sun et al. (2005) Sun, M., Vikhlinin, A., Forman, W., Jones, C., & Murray, S. S. 2005, ApJ, 619, 169
  * Sun et al. (2007) Sun, M., Jones, C., Forman, W., et al. 2007, ApJ, 657, 197
  * Sun et al. (2021) Sun, M., Ge, C., Luo, R., et al. 2021, Nat. Astron., 243
  * Swarup et al. (1991) Swarup, G., Ananthakrishnan, S., Kapahi, V. K., et al. 1991, Current Science, 60, 95
  * Vikhlinin et al. (2001) Vikhlinin, A., Markevitch, M. M., Forman, W. R., & Jones, C. 2001, ApJ, 555, L87
  * Vikhlinin et al. (2005) Vikhlinin, A., Markevitch, M. M., Murray, S. S., et al. 2005, ApJ, 628, 655
  * Venturi, Feretti & Giovannini (1989) Venturi, T., Feretti, L., & Giovannini, G. 1989, A&A, 213, 49
  * Venturi, Giovannini & Feretti (1990) Venturi, T., Giovannini, G., & Feretti, L. 1990, AJ, 99, 1381
  * Venturi et al. (2022) Venturi, T., Giacintucci, S., Merluzzi, P., et al. 2022, A&A, in press
  * van Moorsel, Kemball & Greisen (1996) van Moorsel, G., Kemball, A., & Greisen, E. 1996, in ASP Conf. Ser. 101, Astronomical Data Analysis Software and Systems, ed. G. H. Jacoby & J. Barnes J. (San Francisco, CA: ASP), 37
  * van Weeren et al. (2009) van Weeren, R. J., Röttgering, H. J. A., Bagchi, J., et al. 2009, A&A, 506, 1083
  * van Weeren et al. (2017) van Weeren, R. J., Andrade-Santos, F., Dawson, W. A., et al. 2017, Nat. Astron. 1, 5
  * van Weeren et al. (2019) van Weeren, R. J., de Gasperin, F., Akamatsu, H., et al. 2019, Sp. Sc. Rev. 215, 16
  * Wagner et al. (1999) Wegner, G., Colless, M., Saglia, R. P., et al. 1999, MNRAS, 305, 259
  * Willson (1970) Willson, M. A. G. 1970, MNRAS, 151, 1
  * Yagi et al. (2010) Yagi M., Yoshida, M., Komiyama, Y., et al. 2010 AJ 140, 1814
  * Zhang et al. (2019) Zhang, C., Churazov, E., Forman, W. R., & Lyskova, N. 2019, MNRAS, 488, 5259
  * Zwicky (1933) Zwicky, F. 1933, Helvetica Physica Acta, 6, 110

---
Figure A2: Images of extended ($\simeq$12.6 kpc) radio sources in the Coma
cluster field that are either foreground/background galaxies or do not have
redshift data. The source_ID as tabulated in the Table A1 is labelled in the
upper-right corner of each image. These radio sources are in the order
presented in Table A1, from left to right and top to bottom. The magenta and
blue colour radio contours correspond to 250–500 MHz (band 3) and 550–850 MHz
(band 4) at the angular resolutions of 6$\farcs$1 and 3$\farcs$7, respectively
of the uGMRT that are are overlaid on the grayscale SDSS (DR12) $r$-band
image. The lowest radio contour plotted is three times the local rms noise and
increasing by factors of 2. The local rms noise are tabulated in Table A1 for
250–500 MHz band (Col. 5) and 550–850 MHz band (Col. 6) of the uGMRT. The
grayscale images are displayed in logarithmic scales to emphasize the optical
hosts associated with the respective radio galaxies.
---
Figure A2: Continued
---
Figure A2: Continued


# JS Fake Chorales: a Synthetic Dataset of Polyphonic Music with Human
Annotation

Omar A. Peracha
Humtap, Inc.
London, U.K.
<EMAIL_ADDRESS>

###### Abstract

High-quality datasets for learning-based modelling of polyphonic symbolic
music remain less readily-accessible at scale than in other domains, such as
language modelling or image classification. Deep learning algorithms show
great potential for enabling the widespread use of interactive music
generation technology in consumer applications, but the lack of large-scale
datasets remains a bottleneck for the development of algorithms that can
consistently generate high-quality outputs. We propose that models with narrow
expertise can serve as a source of high-quality scalable synthetic data, and
open-source the JS Fake Chorales, a dataset of 500 pieces generated by a new
learning-based algorithm, provided in MIDI form.

We take consecutive outputs from the algorithm and avoid cherry-picking in
order to validate the potential to further scale this dataset on-demand. We
conduct an online experiment for human evaluation, designed to be as fair to
the listener as possible, and find that respondents were on average only 7%
better than random guessing at distinguishing JS Fake Chorales from real
chorales composed by JS Bach. Furthermore, we make anonymised data collected
from experiments available along with the MIDI samples. Finally, we conduct
ablation studies to demonstrate the effectiveness of using the synthetic
pieces for research in polyphonic music modelling, and find that we can
improve on state-of-the-art validation set loss for the canonical JSB Chorales
dataset, using a known algorithm, by simply augmenting the training set with
the JS Fake Chorales.

##  1 Introduction

Representation learning for application to music has become an increasingly
active field of research in recent years, as Figure 1 demonstrates. Despite
growth in the domain, certain bottlenecks have persisted which hinder the rate
of breakthroughs, particularly regarding problems related to symbolic music.
The most obvious of these concerns the volume of high-quality datasets
available for researchers to work with cheaply and with low effort; datasets
comprising millions of samples [1] or even tens of billions of canonical
input/output pairs [2] have been available for many years in the fields of
computer vision and natural language processing. Some of these are even
licensed with sufficient permissiveness as to allow commercial use of any
technology whose development relied upon said data [3], a factor which plays
an important role in determining the ultimate viability of deploying these
algorithms into the real world where they may generate value for businesses
and customers alike, naturally promoting further investment. Unsurprisingly,
the aforementioned two domains continue to see incredible progress in research
and increasingly wide adoption of the resulting technology in consumer-facing
products.

Figure 1: Number of results returned by [4] when filtering for papers in the
Computer Science, Mathematics or Statistics categories with both "Music" and
"Learning" in their abstracts for the years 2010-2020 inclusive. For years
before 2015, we further filter out any irrelevant results manually.

A second bottleneck more pertinent to generative music modelling is the lack
of automated methods of assessing outputs which strongly correspond to good
qualitative performance. This difficulty in measuring success naturally leads
to slower progress. Few alternatives have been proposed to human evaluation,
which is typically slow or costly, for assessing the quality of
algorithmically-generated music. The process of acquiring human feedback
itself tends to vary in implementation details between different published
works [5, 6, 7], which further convolutes the process of reliably comparing
results in the literature.

We propose a dataset of synthetic polyphonic music, generated by our neural
network-based KS_Chorus algorithm, in symbolic form, and aim to validate both
its usefulness for aiding in symbolic music modelling tasks and its ability to
scale on demand. We achieve the latter by building the dataset from 500
consecutive generated outputs of KS_Chorus, with zero cherry-picking of
samples, and performing an experiment designed to solicit human evaluation in
a context entirely free from curation, which may otherwise serve to skew
results in the algorithm’s favour. 500 samples is small enough that conducting
such an experiment remains fairly manageable, but is large enough relative to
the set of 382 chorales by Johann Sebastian Bach [8], whose style the JS Fake
Chorales seek to imitate, that the results may be deemed significant. After
6,810 responses collected from an estimated 446 unique respondents, our
experiment shows that human listeners are only 7.3% better than a coin flip at
telling apart our synthesised samples from chorales composed by Bach himself.
We facilitate scalability by making a web app freely available for researchers
to generate new samples from KS_Chorus,111Web app to generate new samples is
available at https://omarperacha.github.io/make-js-fake/ enabling them to grow
the dataset, which is further described in Section 3.3.

We then conduct experiments using the JS Fake Chorales as training data or
auxiliary data on a common benchmark task, namely modelling the canonical JS
Bach Chorales dataset. Using the current state-of-the-art model, TonicNet [9],
without any tuning of parameters, we show that training purely on JS Fake
Chorales achieves performance close to state-of-the-art on the JS Bach
Chorales in terms of validation set loss, even without training on any pieces
from the actual JS Bach training set. By combining the JS Bach Chorales
training set with the JS Fake Chorales, using the latter as a form of dataset
augmentation, we can improve on state-of-the-art results. We make all code to
run our ablation studies publicly available, for reproducibility.222Code to
reproduce results from experiments is available at
https://github.com/omarperacha/TonicNet

In evidencing the scalability of the JS Fake Chorales and their usefulness in
common music modelling tasks, we hope to highlight the possibility for
synthetic data to help alleviate the current bottleneck of low data volume in
this domain. Finally, to help address the bottleneck of weak methods for
measuring qualitative performance, we make anonymised metadata for each of the
6,810 human responses available as part of the dataset,333Dataset and usage
information are available at https://github.com/omarperacha/js-fakes which
includes information about the evaluator’s level of music education and which
specific pieces were guessed correctly or incorrectly by which evaluator,
among other data further detailed in Section 3. Future work may see attempts
at modelling these data as a means to automatically infer qualities which make
music more readily-perceived as human-composed.

Ours is the first dataset in this domain that provides such data for such a
large number of samples without cherry-picking, to our knowledge. Similarly,
Section 4.1 shows that KS_Chorus is the first deep learning algorithm proven
to consistently approach, though not yet achieve, a production-level
consistency of high-quality unconditioned outputs in Bach’s, or indeed any
clearly-defined polyphonic, style. The value of this is that the generated
samples can already be very useful as auxiliary training data in downstream
tasks, as shown in Section 4.2.

In a data-starved domain, models with narrower focus are able to squeeze
performance on specific genres from the lack of a requirement to generalise to
other styles, for example by making style-specific adjustments in the
representation. As a concrete example, KS_Chorus does not need to handle
individual voices capable of producing more than one note simultaneously,
allowing for a more efficient representation of Bach’s chorales than cross-
genre polyphonic music models, such as MuseNet [18]. We propose that narrow-
focussed models, by generating high quality synthetic data in a single or
small number of styles, may play a role in significantly increasing the
availability of training data, thus empowering future generative algorithms to
generalise better across styles while improving on both quality and
consistency.

##  2 Related Work

Synthetic datasets in the music domain have so far seen wider use in audio
contexts rather than symbolic music. For example, both [12] and [13] trained
algorithms on synthetic data to learn how to accurately detect the fundamental
frequency of monophonic musical material. While algorithms which process audio
have seen more benefit from synthetic data up until now, examples do exist of
such data playing a role in symbolic music contexts; in [14], extra training
data is synthesised by performing a conditional nonlinear transform on the
original training corpus, in a naive 1-dimensional token embedding space. This
data is used to augment the training set and is fed to the generator and
discriminator in a generative adversarial learning context, with an input
label corresponding to the intensity of the transform performed on the
original data sample so as to allow both networks to learn features expressed
in the extra data which are also present in the original data. It is worth
noting that the intent in this case was not to create perfect stylistic
imitations of the original dataset, but to introduce novelties in perceivably-
systematic ways. Nonetheless, [14] serves as a precursor for using synthetic
data to improve the quality of generated symbolic music.

With 10,854 complete pieces of solo piano music in MIDI form, [15] is among
the larger symbolic music datasets available. It could be considered a
synthetic dataset in that the authors used an automated method of
transcription [16] to convert over 1000 hours of audio recordings into a
symbolic representation. The authors report validation onset and offset F1
score of 0.825, and a score of 0.967 for onsets alone. These are impressive
results in the current landscape, yet clearly this may translate to many
transcription errors across a dataset comprising millions of note events. The
severity of these inconsistencies with respect to the stylistic integrity of
the original human-composed piece can be hard to measure, and further work is
required to determine the areas where this dataset will ultimately enable
progress in the field.

Both [5] and [6] propose algorithms intended to generate stylistic imitations
of Bach’s chorales, and use human evaluation to support claims of the
effectiveness of their solutions. The experiment conducted by [5] first takes
50 pieces from the validation set of the Bach chorales and uses their DeepBach
algorithm to reharmonise them. They repeat the process with other baseline
algorithms. They then take 12-second extracts of each and play these samples
one-by-one, asking listeners to determine if they think each is by Bach or is
a generated piece. In contrast, we generate 500 pieces entirely from scratch
with no conditioning and always play the entire piece to listeners, which can
be over one minute in length. We also make a ground truth chorale by Bach
available to the listener to play back at any time alongside the sample
currently being evaluated.

The experiment conducted by [6] is similar to ours in that it allows to listen
to two samples simultaneously. Unlike ours, however, the listener must simply
pick which of the two samples they feel is more Bach-like. In our case, the
evaluator knows that one sample is definitely composed by Bach, and must
decide if the second sample is also by Bach or if it is generated by
KS_Chorus. A second significant difference is that [6] only use a set of 12
compositions fully-generated by their BachBot algorithm in their experiments.
This is too small a sample to substantiate the model’s general performance. In
our case, we generate 500 samples in a row and include all of these in the
human evaluation experiment without cherry-picking. Neither [5] nor [6] make a
substantial dataset of generated compositions available, while we make these
initial 500 available as the JS Fake Chorales dataset, and offer evidence that
the dataset can be further scaled while maintaining the same quality
consistently as a result of avoiding curation during human evaluation.

Perhaps the most relevant work to ours is the Bach Doodle Dataset [17],
obtained by allowing users to interact with the CoCoNet algorithm [7]; the
user could enter a melody via a web interface, and the algorithm would
harmonise this melody in the style of Bach. Comprising 21.6 million samples,
the size far exceeds any that is commonly-used in symbolic music, to our
knowledge. The web interface also collects a user rating on a three-point
scale to reflect their satisfaction with the output. This rating can not be
said to reflect stylistic consistency with any intended target, however, but
simply provides a general measure of the user’s experience. This rating is
provided as part of the dataset, along with more metadata about the user’s
session. Because the melodies are user-entered, only constrained to fit within
a selected key and within a 2-bar duration, the quality of the final samples
in the Bach Doodle Dataset will be inherently unpredictable. Furthermore, all
samples are fixed to 2 bars in duration, and melodies are limited to quaver
resolution. Our samples consist only of pieces in their entirety, with a
median bar duration of 11.375 and maximum of 35.5, and rhythmic resolution can
fully capture that seen in the original Bach chorales.

Several style-agnostic polyphonic models have been proposed in recent years.
Three examples notable for being particularly high-performing are MuseNet
[18], MMM [19] and MusicBERT [20]. While MMM and MusicBERT are highly
promising, MuseNet is the only one of these to be evaluated on full song
generation. This evaluation is not performed quantitatively, and provided
samples are cherry-picked, though an interface for novel song generation is
available. It is likely that all three models have been trained on the Bach
Chorales, though not known for certain as not all the datasets are public. In
any case, none of these models are able to demonstrate the capability to
generate a dataset with quality matching the JS Fake Chorales.

##  3 Dataset

###  3.1 Synthesising 4-part Chorales

To create the JS Fake Chorales dataset, we first develop a learning-based
sequence modelling algorithm, KS_Chorus, and train it on the Bach chorales as
made available by the music21 toolkit [21]. Bach’s chorales are a good choice
as the basis for a synthetic dataset mainly because they are widely-cited in
the literature regarding algorithmic modelling of polyphonic music [5, 6, 7,
8, 9], and so downstream quantitative analysis against good benchmarks is
readily possible.

KS_Chorus is a generative algorithm for polyphonic music of any number of
instruments, where each instrument is itself monophonic. It consists of an
ensemble of deep RNN-based networks with some domain-specific architectural
additions, totalling roughly 130 million parameters, and a sampling routine
idiosyncratic both to this architecture and the data representation utilised.
We do not detail the specifics of the architecture, input representation or
training process here, and instead leave this to a separate work for the
future.

Once trained, we then sample 500 compositions from the model consecutively and
fully unconditionally, i.e using absolutely no priming or similar input to
specify parameters such as the length of the compositions. The only exception
to this relates to metre; while a small handful of pieces exist in the
original training corpus which are not in 4/4 time, we found during
development that KS_Chorus falls somewhat below par when generating
compositions in other time signatures. We therefore manually restrict all
generated samples to be in 4/4, because the likelihood of being correctly
identified as algorithmically-composed seemed to be consistently higher in
other time signatures.

Each generated composition is compared to each of the 344 Bach chorales in the
training set seen by the model during development to ensure originality; in
particular, the edit distance to every single sample in the training corpus
must be greater than 50%, measured separately across both the entire piece and
across the opening few bars. An edit distance below 50% in either case would
invalidate the sample and trigger the algorithm to run once more. Furthermore,
all samples in the original dataset were transposed as far as possible in
either direction while maintaining a singable range for each voice, and the
edit distance was similarly validated to these transpositions. A single
composition takes roughly 10 minutes on average to generate when running on
CPU. The algorithm was implemented and trained using the PyTorch library [22],
and sampling was also performed in Python.

While we did not measure the edit distance of generated pieces to all other
generated pieces at sample time, we perform post-hoc analysis of this metric.
Since we do not detail the representation used by KS_Chorus in this work, for
the sake of reproducibility we perform these comparisons using a standard
representation seen in the literature [9, 23]; the piece is split into 16th-
note time-steps and represented as a matrix $x\in\mathbb{Z}^{4\times T}$,
where $T$ is the number of time-steps in that sample, the four rows represent
the four different voice parts and the value of each element is the pitch
observed in the respective voice at the respective time-step. This matrix is
typically converted into a linear sequence before being input to a learning
algorithm, by end-concatenating the columns. We also make the JS Fake Chorales
available in this representation.

Using the above representation, we compare the intra-set edit distance (i.e.
comparing each sample with every sample from the same dataset) for both the
Bach chorales and the JS Fake Chorales, and we repeat the comparison of each
JS Fake chorale with each Bach chorale. Table 1 shows the results of taking
the smallest edit distance from each sample in dataset 1 to each sample in
dataset 2 for the three different dataset configurations, and computing the
mean of these smallest distances. We also provide the minimum in each case. We
can see that there is in fact more intra-set similarity among the Bach
chorales than among the JS Fakes. Of course, the edit distance between the two
was enforced at generation time, hence the distribution of inter-set least
edit distances trending much higher than that of either intra-set least edit
distances. We can deduce from this post-hoc analysis that 50% was perhaps an
unnecessarily strict threshold for edit distance during the process of
sampling from KS_Chorus.

Table 1: Results of taking the smallest edit distance from each sample in
dataset 1 to each sample in dataset 2 and computing both the mean and the
minimum of these smallest distances.
Dataset 1 | Dataset 2 | Minimum | Mean
---|---|---|---
JSF | JSB | 55.078% | 72.818%
JSF | JSF | 25.625% | 66.657%
JSB | JSB | 12.755% | 57.792%

###  3.2 Human Annotation

We conduct a human evaluation experiment hosted on the internet to investigate
how these generated compositions are perceived in comparison to Bach’s
chorales. We first present the experiment design in this section and describe
the data obtained and provided as part of the JS Fake Chorales dataset, while
the actual results of this experiment will be discussed in Section 4. We use
$\mathcal{G}$ to notate the set of 500 pieces which comprise the JS Fake
Chorales dataset, and $\mathcal{C}$ to refer to the training corpus of 344
chorales by J.S. Bach.

####  3.2.1 Listening Experiment

Participants are first prompted to self-report their highest level of musical
education, selecting from a set of six coarse pre-defined options: 0, no
musical education; 1, some musical education, but no formal qualifications; 2,
high School level qualifications directly related to music; 3, undergraduate
degree directly related to music; 4, postgraduate degree directly related to
music; and 5, postgraduate degree specialising in the music of J.S. Bach.

Once they have confirmed their choice, they are taken to a page with
instructions describing how to complete the experiment. On this page, the
participants are provided with a reference chorale by Bach to listen to,
chosen randomly per session from the 344 pieces
$[C_{1},...,C_{344}]\in\mathcal{C}$. Participants are informed that the given
chorale is a genuine example of Bach’s music, and they must play it through to
the end in order to advance further. Upon doing so, a second piece will
appear, which is selected at random from the combined set of all pieces
$\mathcal{G}\cup\mathcal{C}$ (excluding whichever Bach chorale was used as the
reference), with equal probability of any sample in this set being selected.

Participants are not informed whether this second piece is composed by Bach or
KS_Chorus, but are tasked to determine this. Once more they can not proceed
without listening to the given sample in its entirety, at which point a menu
will appear through which they can provide their response and submit. All
pieces are rendered using the same piano synthesiser, and can be played as
many times as the participant wants before submitting a response, including
the reference track. KS_Chorus does not currently include tempo data as part
of the representation it learns to model, therefore all piece are played back
at a fixed tempo of 120bpm.

Upon submitting a response, participants are taken to a new page where they
are immediately shown the correct answer, and given the option to try another
round. They are also shown a running score which increments by 1 for each
correct response in the session, which carries over if they choose to try a
new sample. Should they choose to do so, they will be taken back to the
previous page where the same reference piece will be available for them to
play back as desired, and a new sample will be loaded for the participant to
identify as belonging to $\mathcal{G}$ or $\mathcal{C}$, ensuring this new
sample has not previously been seen in the current session. In this case,
participants are not forced to replay the reference track, but must once more
listen through the new sample entirely before proceeding.

Responses were solicited both organically via social media, and through Amazon
MTurk [24]. In the latter case, participants were required to obtain a score
of 10 in order to fulfil the conditions for payment, and were given a 30
minute limit to do so. This was done to motivate participants to try their
utmost to provide correct responses; mandating that the sample be played
through entirely for each round vastly increases the time cost for every wrong
answer, meaning that random guessing is not an effective strategy for MTurk
respondents to earn the available reward. The mean time taken by MTurk workers
to complete the task was 19m45s, and we estimate these individuals make up
roughly 66% of all unique respondents. Overall we received 6,810 responses
from 446 unique IP address hashes. Responses were collected throughout the
month of February 2021, and the website remains live for anyone to interact
with,444Listening test available at https://omarperacha.github.io/listening-
test while data is no longer being collected from sessions.

####  3.2.2 Metadata

Besides a simple binary value denoting the accuracy of a participant’s
response, we collect several other data points designed to offer insight on
how complex a given sample was to identify by the participant. Examples
include how many times the sample was played before submitting a response and
how long the participant took to submit once they had heard the sample for the
first time. We aggregate these and make them available as part of the dataset
in the hope that this information might provide value to researchers.

For each generated chorale $G_{i}\in\mathcal{G}$, we make four additional
pieces of metadata available: (1) Total responses concerning sample $G_{i}$;
(2) responses which correctly identified $G_{i}$ as composed by an algorithm;
(3) mean number of times the sample was played before a response was
submitted; and (4) mean time in seconds between hearing the sample and
submitting a response. We provide these data separately for each skill level,
to provide further granularity on how the samples were perceived by different
demographics. This allows such custom analysis as examining which pieces were
likely to be misidentified by people with university degrees in music, and
which pieces did people with limited musical education find simplest to
correctly identify. We also provide the data aggregated over all skill levels.
Further usage explanation is provided in the documentation.

###  3.3 Web Application for Generating New JS Fake Chorales

We make a web application freely available for anyone to sample new chorales
with KS_Chorus. The application consists of a single page, including text and
a button to trigger sample generation. Once the sampling process is complete,
the button is replaced with a MIDI player where the user can preview the
newly-generated sample, rendered with a piano synthesiser. The MIDI file will
also be automatically downloaded to the user’s local hard drive from their
browser at that time. The main JS Fake Chorales repository will be
periodically updated to include all samples generated from the web app, and
users are informed of this. No user data of any kind is captured from this
interface, besides the generated sample itself. While we commit to the
persistency of the JS Fake Chorales dataset, and all samples generated from
this web app in future included therein, we may in due course choose to
discontinue the availability of this app as we see fit, for example if the
cost of upkeep is deemed no longer sustainable.

##  4 Experimental Results

###  4.1 Human Evaluation

####  4.1.1 Sample Category Identification Task

Table 2: Number of samples by Bach and by KS_Chorus seen over all participant
sessions during human evaluation (columns 2 & 5), the number of times these
were correctly identified (columns 3 & 6) and the resulting rate of correct
responses (columns 4 & 7), shown across all skill levels.
Skill | Bach | A.I. | Total Samples
---|---|---|---
| Samples | Correct | Ratio | Samples | Correct | Ratio |
0 | 705 | 339 | 48.1% | 1081 | 633 | 58.6% | 1786
1 | 943 | 445 | 47.2% | 1351 | 748 | 55.4% | 2294
2 | 515 | 270 | 52.4% | 732 | 409 | 55.9% | 1247
3 | 264 | 98 | 37.1% | 366 | 225 | 61.5% | 630
4 | 215 | 97 | 45.1% | 348 | 223 | 64.1% | 563
5 | 120 | 65 | 54.2% | 170 | 83 | 48.8% | 290
ALL | 2762 | 1314 | 47.6% | 4048 | 2321 | 57.3% | 6810

We compare the likelihood of participants determining a sample to be human-
composed, and present results across skill levels for samples by both Bach and
KS_Chorus. Given an ideal model whose distribution of generated compositions
$\widetilde{G}$ exactly equals the distribution of training samples
$\widetilde{C}$, we would expect the likelihood of any $G_{i}\in\mathcal{G}$
and $C_{i}\in\mathcal{C}$ being deemed as human-composed to also be equal.
Given a fair experiment which does not incur bias, for example by
disproportionately highlighting certain samples of Bach’s pieces or
suppressing specific qualities present in $\mathcal{C}$ from being observed by
the evaluators, this expected likelihood should be 50%.

Table 2 shows the proportion of all responses which correctly identified
samples from $\mathcal{G}$ and $\mathcal{C}$, broken down by skill level. We
see that the total rate of correct responses for $\mathcal{C}$ is 47.6%, while
for $\mathcal{G}$ it is 57.3%. These values, though unequal, are close to each
other, and in particular they suggest evaluators did not differ greatly from
random chance regarding the likelihood of a correct guess for either corpus.
This is despite receiving instant feedback after each round and a reference
$C_{i}\in\mathcal{C}$ being made accessible at all times while prompting for a
response, which should both aid in increasing the participants’ success rate.

We perform Fisher’s exact test on a contingency table consisting of rows
$\mathcal{C}$, $\mathcal{G}$ and columns voted Bach, voted A.I. across all
skill levels to interpret whether samples from $\mathcal{G}$ did indeed have a
similar likelihood to those from $\mathcal{C}$ of being considered human-
composed. We test each skill group separately, the combination of all skill
groups together, and two subgroups dividing the respondents into those who
self-reported holding a formal qualification in music, and those who did not.
We also compute the F1 score for each group as a measure of overall
performance on the classification task. We report these results in Table 3,
and see that at the 99.9% critical value we can accept the null hypothesis
that $\mathcal{C}$ and $\mathcal{G}$ are equally likely to be voted as being
written by Bach, for every individual skill group and for the two larger
subgroups. However, the value of $p$ varies greatly across these groups, and
for the overall sample comprising all responses $p=6.432e^{-5}$, so we must
reject the null hypothesis. Therefore while at a high level the JS Fakes are
certainly difficult to distinguish from Bach Chorales, performing further
tests makes it clear there is room for improvement in the quality of the
generated samples. Closing this gap is precisely an example of an application
which this dataset might serve in future.

We can also see from Table 3 that there is no apparent trend in F1 score
across respondent skill levels, which seems surprising. One obvious candidate
for this is the inherent noise and unreliability introduced by enabling
respondents to self-report their skill. A stronger method for a future
experiment would involve assessing user skill level via some short test.
However, the fact that $p>0.001$ consistently across all skill groups suggests
participants within each group did in fact respond similarly to their peers.
Another criticism we can make is that while likelihood of considering a piece
from either sample as human-composed was similar, this observed value was not
50%, but 55.7% as measured by weighted average recall. While we have theories
for why this may be the case, we must conclude that there are likely some
aspects of the experimental design which are not free from introducing small
bias. We leave the experiment fully accessible online for transparency and
leave it for future work to analyse and improve on the fairness of designing
such an experiment.

Table 3: Results of performing Fisher’s exact test and computing F1 score for
each skill group, all skill groups combined, and two subgroups dividing the
respondents into those who do and do not hold a formal qualification in music.
SKILL | 0 | 1 | 2 | 3 | 4 | 5 | NO QUAL | QUAL | ALL
---|---|---|---|---|---|---|---|---|---
p | 0.006 | 0.233 | 0.004 | 0.740 | 0.033 | 0.635 | 0.006 | 0.003 | < 0.001
F1 | 0.609 | 0.576 | 0.590 | 0.594 | 0.647 | 0.540 | 0.591 | 0.600 | 0.594

####  4.1.2 Ranking

One possibility offered by the data captured is the ability to rate samples
with respect to the likelihood that a listener might perceive the pieces as
human-composed. This could prove a useful feature to researchers whose goal is
to understand the qualities of music which people tend to perceive as "human",
or to implicitly model them as part of a generative system designed to output
increasingly higher-quality symbolic music. Moreover, this can be done in a
manner stratified by listener music education level, which may provide further
insight into which qualities are preferred or more telling for which people.
We use the following equation to obtain such a rating $R$ for any given sample
$S_{i}\in\mathcal{G}\cup\mathcal{C}$.

$R_{S_{i}}=N_{S_{i}}^{-0.5}*(\beta_{S_{i}}+\frac{\epsilon}{N_{S_{i}}})$ (1)

Equation 1 is designed to balance the absolute ratio of responses deeming the
sample to belong to $\mathcal{C}$ with the number of responses received for
that particular sample in total, since this ratio becomes more reliably
indicative of quality as the number of responses increases. We use
$\beta_{S_{i}}$ to denote the number of responses categorising a sample as
Bach’s music, and $N_{S_{i}}$ for the total number of responses for that
sample. We use 0.001 for $\epsilon$, which serves to preserve a logical
ranking for any samples where $\beta_{S_{i}}=0$, by giving those which
received more negative responses overall a lower score.

We plot $R$ as histograms for both $\mathcal{G}$ and $\mathcal{C}$ in Figure
2. We can see that samples from $\mathcal{C}$ trend towards slightly higher
scores than those from $\mathcal{G}$, suggesting that pieces by Bach are
indeed still more likely to be perceived as human-composed than those by
KS_Chorus. Having said that, the distributions of $R$ for the two sets show
strong similarities on aggregate. The maximum $R$ for the generated pieces
$\mathcal{G}$ is 2.89, while the mean and standard deviation are 1.19 and 0.55
respectively. For $\mathcal{C}$, the maximum, mean and standard deviation are
2.94, 1.33 and 0.57 respectively.

We perform a Mann-Whitney U test on the $R$ scores for the two sets and find
$p=0.0493$, which supports our claims that there are similarities between the
way the two sets are perceived. Furthermore, we perform the same test on a
naive scoring method, for transparency, where the rating is given by the
number of "Bach" votes for each sample divided by the total number of votes
for that sample. In this case, $p=0.303$. We believe this motivates the use of
Equation 1 to calculate rating, as it implicitly factors in a form of
confidence by multiplying by a term proportional to $N_{S_{i}}$, offering a
more nuanced impression of listener perception.

(a) JS Fakes Dataset.

(b) Bach training dataset.

Figure 2: Histogram of scores $R$ for pieces in the JS Fakes Dataset (a) and
by Bach in the training dataset (b).

###  4.2 Algorithmic Evaluation

We conduct experiments to ascertain the benefit of using the JS Fakes Dataset
to help with music modelling tasks. We use the canonical JSB Chorales dataset
to perform ablation studies, and measure performance in terms of negative log
likelihood on the validation set, using the same dataset split as [8]. The
task involves element-wise autoregressive modelling of chorales represented in
sequence form, as described in Section 3.1. We use the TonicNet_Z algorithm to
perform experiments, using the same hyperparameters, training schedule and
specific data representation as [9]. We explore the effect of training only on
the JS Fake Chorales and of combining them with the training set of the JSB
Chorales as a form of data augmentation. We show how the JS Fake Chorales
impact results both with and without augmentation by transposition, and also
investigate the impact of limiting the set of JS Fake Chorales seen by the
model to those samples with the highest $R$ ratings. Table 4 shows the results
of these experiments, alongside two further strong-performing algorithms.

Table 4: Results when modelling the J.S. Bach Chorales dataset with two strong
baselines and TonicNet_Z trained on various combinations and subsets of the JS
Fake Chorales and J.S. Bach Chorales datasets. Rows marked with asterisks are
taken directly from [9], while rows marked with a cross are taken from [23].
Datasets | Model | Aug. | Val. NLL | NCL Val. NLL
---|---|---|---|---
Bach† | CoCoNet (ordered) | None | - | 0.436
Bach† | Music Transformer | None | - | 0.335
JSF-top | TonicNet_Z | None | 0.474 | 0.363
JSF | TonicNet_Z | None | 0.424 | 0.319
Bach* | TonicNet_Z | None | 0.422 | -
JSF-top-s | TonicNet_Z | Trnsps | 0.364 | 0.272
JSF-top | TonicNet_Z | Trnsps | 0.364 | 0.269
JSF | TonicNet_Z | Trnsps | 0.328 | 0.234
Bach* | TonicNet_Z | Trnsps | 0.321 | 0.224
Bach* | TonicNet_Z | Trnsps+MM | 0.317 | 0.220
Bach+JSF-top | TonicNet_Z | Trnsps | 0.309 | 0.215
Bach+JSF-top-s | TonicNet_Z | Trnsps | 0.307 | 0.213
Bach+JSF | TonicNet_Z | Trnsps | 0.300 | 0.208

In the first column of Table 4, we denote the combinations and subsets of
datasets used to train TonicNet_Z. Bach here refers to the 229 samples in the
JSB Chorales training set and $JSF$ refers to the full set of 500 pieces in
the JS Fake Chorales datatet. We also use two special subsets of JSF in
experiments, namely JSF-top and JSF-top-s; these represent the subset of JSF
obtained by ordering the samples by their $R$ scores and taking the top 229.
In the case of JSF-top-s, we first consider only responses by participants
with a skill level of 3 or higher before calculating $R$, to perform an
initial investigation into whether insights specifically from individuals with
higher levels of musical education might impact downstream results.

The third column shows whether any augmentation techniques were performed on
the datasets in question. Trnsps refers to transposing the training set only,
as far as possible in both direction while all voices in the sample remain
within a singable range for their respective voice type. MM refers to a crude
technique of converting pieces in major keys to minor, and vice versa, as
described in [9], roughly doubling the dataset size. The remaining columns
show experimental results on the validation set, where NCL denotes evaluation
on the notes only, as is typical in experiments on the JSB Chorales, omitting
elements containing chord tokens specifically included as part of the TonicNet
representation when averaging the loss for each sample.

We can see that even without fine-tuning parameters, training the model only
on the JS Fake Chorales obtains results close to training on the original set
Bach chorales, whether augmented or not. Remarkably, we are even able to
perform better than strong baselines [7, 23] without using any real training
data. Naturally, there are far more samples in JSF than Bach, so we use JSF-
top for a more balanced comparison and find that the model trained on this
dataset indeed performs worse, but still within a close range compared to
training on the canonical dataset. We see that there is no significant
difference between using JSF-top and JSF-top-s during training, but that using
either in combination with Bach provides better results than augmenting via
MM. The dataset size is roughly doubled from the original in all three cases
when using JSF-top, JSF-top-s and MM, meaning that the final training set size
is close to equal for these cases, in turn suggesting that using the JS Fake
Chorales is a higher quality form of augmentation than MM. We are therefore
able to set a new state-of-the-art result of 0.208 validation set NLL on the
JSB Chorales by combining the original training set with the JS Fake Chorales.

##  5 Conclusion

We develop an algorithm capable of generating chorales in the style of Bach
and sample 500 pieces from it consecutively with no human intervention. We
devise an experiment for human evaluation designed to be fair and free from
bias, and through this find that these samples are perceived to be composed by
Bach almost as readily as pieces which are in fact by Bach. In so doing, we
collect data about human interaction with these pieces and create a dataset of
500 synthetic chorales with human annotation. By avoiding cherry-picking when
conducting human evaluation, we demonstrate that the number of pieces in the
JS Fake Chorales dataset could be readily scaled while expecting quality to be
maintained. We further provide a web interface to readily enable such scaling
of the dataset.

We run experiments to demonstrate the effectiveness of this dataset in a
common MIR task. We show that training an algorithm on our dataset can be
nearly as effective at modelling Bach’s music than training on his actual
pieces, outperforming several strong baselines. Finally, by combining both the
JS Fake Chorales and Bach’s chorales into a training corpus, we achieve state-
of-the-art results on JSB Chorales dataset.

In general, the motivation for any field of research is to see it lead to
real-world applications where it enables people to do what they previously
could not, or to to do things with fewer resources than was previously
possible. In order to see widespread adoption of generative music algorithms
in consumer applications, they must consistently perform well in at least one
relevant task. Such tasks in the symbolic music domain include, but are not
limited to, melody generation, melody continuation, accompaniment generation,
music in-painting and music style transfer.

Today, no products which enable the aforementioned tasks could truly be
described as being widely-used, with the exception of arpeggiators, a family
of rule-based tools which perform a specific and narrow kind of style transfer
in the symbolic domain. Ultimately this is due to a lack of algorithms which
perform with sufficient consistency and quality to provide meaningful value to
potential users. One of the most promising ways to overcome this is to break
the two bottlenecks mentioned in Section 1 and leverage learning algorithms.

Large datasets of music in varying styles, especially containing polyphonic
music, will be needed to create learning algorithms that perform all of the
above tasks reliably, but these are not easily or cheaply obtainable. Hence we
proposed a synthetic dataset with human evaluation data and on-demand
scalability as one way to alleviate this problem. Conducting a listening
experiment on hundreds of consecutive output allowed us to obtain a good
picture of where samples meet or fall short of typical consumer expectation.
Newly-generated samples from the same algorithm can be expected to be
similarly distributed in this respect, and the function of distance between
these samples and consumer expectation, measured as described in Sections 3
and 4, can be modelled in a semi-supervised way in future works. The ability
to model this gap gives us a quantitative way to measure it for new
algorithms, ultimately allowing researchers to close it more efficiently.

##  6 Acknowledgements

This work was supported by the Polish National Centre for Research and
Development under Grant POIR.01.01.01-00-0322/20 titled “Humtap - research and
development of the technology for audio-video content generation with machine
learning and social sharing”.

## References

  * [1] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei, “Imagenet: A large-scale hierarchical image database,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2009, pp. 248–255.
  * [2] Y. Zhu, R. Kiros, R. Zemel, R. Salakhutdinov, R. Urtasun, A. Torralba, and S. Fidler, “Aligning books and movies: Towards story-like visual explanations by watching movies and reading books,” in _Proceedings of the IEEE International Conference on Computer Vision_ , December 2015.
  * [3] A. Kuznetsova, H. Rom, N. Alldrin, J. Uijlings, I. Krasin, J. Pont-Tuset, S. Kamali, S. Popov, M. Malloci, A. Kolesnikov, T. Duerig, and V. Ferrari, “The open images dataset v4: Unified image classification, object detection, and visual relationship detection at scale,” _IJCV_ , 2020.
  * [4] arxiv search. [Online]. Available: https://arxiv.org/search/
  * [5] G. Hadjeres, F. Pachet, and F. Nielsen, “DeepBach: a steerable model for Bach chorales generation,” in _Proceedings of the 34th International Conference on Machine Learning_ , ser. Proceedings of Machine Learning Research, D. Precup and Y. W. Teh, Eds., vol. 70. PMLR, 2017, pp. 1362–1371. [Online]. Available: http://proceedings.mlr.press/v70/hadjeres17a.html
  * [6] F. Liang, “Bachbot: Automatic composition in the style of bach chorale,” Master’s thesis, University of Cambridge, August 2016.
  * [7] C.-Z. A. Huang, T. Cooijmans, A. Roberts, A. Courville, and D. Eck, “Counterpoint by convolution,” in _Proceedings of the 18th International Society for Music Information Retrieval_ , 2017.
  * [8] N. Boulanger-Lewandowski, Y. Bengio, and P. Vincent, “Modeling temporal dependencies in high-dimensional sequences: Application to polyphonic music generation and transcription,” in _Proceedings of the29th International Conference on Machine Learning, ICML 2012_ , vol. 2, 06 2012.
  * [9] O. Peracha, “Improving polyphonic music models with feature-rich encoding,” in _Proceedings of the 21st International Society for Music Information Retrieval Conference_ , 2020, pp. 169–175.
  * [10] J. Engel, C. Resnick, A. Roberts, S. Dieleman, D. Eck, K. Simonyan, and M. Norouzi, “Neural audio synthesis of musical notes with wavenet autoencoders,” 2017. [Online]. Available: https://arxiv.org/abs/1704.01279
  * [11] J. Engel, L. H. Hantrakul, C. Gu, and A. Roberts, “Ddsp: Differentiable digital signal processing,” in _Proceedings of the International Conference on Learning Representations_ , 2020. [Online]. Available: https://openreview.net/forum?id=B1x1ma4tDr
  * [12] M. Mauch and S. Dixon, “Pyin: A fundamental frequency estimator using probabilistic threshold distributions.” in _Proceedings of ICASSP_. IEEE, May 2014, pp. 659–663. [Online]. Available: https://ieeexplore.ieee.org/abstract/document/6853678
  * [13] J. Kim, J. Salamon, P. Li, and J. Bello, “Crepe: A convolutional representation for pitch estimation,” in _Proceedings of ICASSP_ , 04 2018, pp. 161–165.
  * [14] O. Peracha and S. Head, “Gankyoku: a generative adversarial network for shakuhachi music,” in _Proceedings of the International Computer Music Conference_. ICMA, July 2019.
  * [15] Q. Kong, B. Li, J. Chen, and Y. Wang, “Giantmidi-piano: A large-scale MIDI dataset for classical piano music,” _CoRR_ , vol. abs/2010.07061, 2020. [Online]. Available: https://arxiv.org/abs/2010.07061
  * [16] Q. Kong, B. Li, X. Song, Y. Wan, and Y. Wang, “High-resolution piano transcription with pedals by regressing onsets and offsets times,” _CoRR_ , vol. abs/2010.01815, 2020. [Online]. Available: https://arxiv.org/abs/2010.01815
  * [17] C.-Z. A. Huang, C. Hawthorne, A. Roberts, M. Dinculescu, J. Wexler, L. Hong, and J. Howcroft, “The Bach Doodle: Approachable music composition with machine learning at scale,” in _International Society for Music Information Retrieval (ISMIR)_ , 2019. [Online]. Available: https://goo.gl/magenta/bach-doodle-paper
  * [18] C. Payne. (2019, April) Musenet. [Online]. Available: openai.com/blog/musenet
  * [19] J. Ens and P. Pasquier, “Mmm : Exploring conditional multi-track music generation with the transformer,” 2020. [Online]. Available: https://arxiv.org/abs/2008.06048
  * [20] M. Zeng, X. Tan, R. Wang, Z. Ju, T. Qin, and T. Liu, “Musicbert: Symbolic music understanding with large-scale pre-training,” 2021. [Online]. Available: https://arxiv.org/abs/2106.05630
  * [21] M. S. Cuthbert and C. Ariza, “Music21: A toolkit for computer-aided musicology and symbolic music data.” in _Proceedings of the 11th International Society for Music Information Retrieval Conference_ , J. S. Downie and R. C. Veltkamp, Eds. International Society for Music Information Retrieval, 2010, pp. 637–642.
  * [22] A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga, A. Desmaison, A. Kopf, E. Yang, Z. DeVito, M. Raison, A. Tejani, S. Chilamkurthy, B. Steiner, L. Fang, J. Bai, and S. Chintala, “Pytorch: An imperative style, high-performance deep learning library,” in _Proceedings of Advances in Neural Information Processing Systems 32_. Curran Associates, Inc., 2019, pp. 8024–8035.
  * [23] C.-Z. A. Huang, A. Vaswani, J. Uszkoreit, I. Simon, C. Hawthorne, N. Shazeer, A. M. Dai, M. D. Hoffman, M. Dinculescu, and D. Eck, “Music transformer: Generating music with long-term structure.” in _ICLR (Poster)_ , 2019.
  * [24] K. Crowston, “Amazon mechanical turk: A research tool for organizations and information systems scholars,” in _Shaping the Future of ICT Research. Methods and Approaches_ , A. Bhattacherjee and B. Fitzgerald, Eds. Berlin, Heidelberg: Springer Berlin Heidelberg, 2012, pp. 210–221.