[affiliation=1]HemantYadav [affiliation=2]SunayanaSitaram [affiliation=1]Rajiv
RatnShah

# MS-HuBERT: Mitigating Pre-training and Inference Mismatch in Masked Language
Modelling methods for learning Speech Representations

###### Abstract

In recent years, self-supervised pre-training methods have gained significant
traction in learning high-level information from raw speech. Among these
methods, HuBERT has demonstrated SOTA performance in automatic speech
recognition (ASR). However, HuBERT's performance lags behind data2vec due to
disparities in pre-training strategies. In this paper, we propose (i) a Swap
method to address pre-training and inference mismatch observed in HuBERT and
(ii) incorporates Multicluster masked prediction loss for more effective
utilization of the models capacity. The resulting method is, MS-HuBERT, an
end-to-end self-supervised pre-training method for learning robust speech
representations. It beats vanilla HuBERT on the ASR Librispeech benchmark on
average by a 5% margin when evaluated on different finetuning splits.
Additionally, we demonstrate that the learned embeddings obtained during pre-
training encode essential information for improving performance of content
based tasks such as ASR.

###### keywords:

Automatic speech recognition, Multicluster masked prediction loss, HuBERT

##  1 Introduction

In the recent years, there has been a significant interest in studying self-
supervised pre-training methods to learn/encode high level information present
in the speech data [1, 2, 3, 4, 5]. These SSL methods utilize the input data
itself to learn to encode useful information, with the choice of pretext task
playing a pivotal role in the encoded information. The most popular pretext
task used is masked predictive coding (MPC) [6, 1, 7, 8, 9]. HuBERT [2] is one
such model that popularized the masked language modelling (MLM) technique to
learn high-level speech representations from raw audio by achieving state-of-
the-art (SOTA) on the automatic speech recognition (ASR) task. The underlying
concept of HuBERT revolves around iterative pre-training: starting with a raw
audio/pseudo-label pair (x/y), the model undergoes successive training
iterations where the trained model updates the pseudo-labels, iteratively
refining its representations until a predefined stopping criterion is reached.
However, despite its success, HuBERT falls short compared to data2vec [10] in
ASR performance for two primary reasons: firstly, during pre-training,
data2vec accesses the full context to generate continuous labels, which are
updated after each gradient update step, as opposed to the fixed discrete
labels utilized in HuBERT for the each iteration; and secondly, the ground
truth labels are created by averaging the representations from multiple layers
for loss calculation.

To bridge this gap, we propose two modifications to the HuBERT framework.
Firstly, we introduce the "Swap" method to enable full context access during
pre-training, thus addressing the pre-training and inference mismatch observed
in HuBERT and other MLM-based methods by using both the masked and unmasked
views during pre-training. Swap is motivated by a simple idea, used heavily in
the field of computer vision [11, 12, 13]. Where two augmented views of the
input are used to learn a high level representation. Given $e$ layers in a
encoder, it is a general practice to add a similarity loss on the output
embeddings of the encoder after each layer, as seen in works using U-net type
architectures [14, 15, 16]. In contrast, our proposed Swap method simply swaps
the output embedding, at certain indices, after each layer of the encoder
between the masked and unmasked view of the input. Motivated by the simple
fact, that the learned model is expected to generate exactly the same output
regardless of the two views, Swap method guides the output after each layer to
achieve the same.

Secondly, inspired by the work of Yadav et al. [17], we incorporate a
Multicluster masked prediction loss (MPL) approach. Using multiple cluster
centers, also called multiple resolutions, has been investigated by [18, 19,
17]. In [19], the author introduces down-sampling and up-sampling modules
within the transformer encoder after each layer to facilitate learning
features at multiple resolutions. On the other hand, [18] explores parallel
and hierarchical variations of HuBERT with findings indicating the superiority
of the hierarchical approach. This involves training multiple models, each
model adds almost same parameters as the original HuBERT, at various
resolutions using CNN as a down-sampling module. Lastly [17] uses the fact
that MPL is applied at multiple layers of encoder at different resolutions.
This method does not introduce any additional parameters to the original
HuBERT model, except the linear layers used for loss calculation which are
discarded after the pre-training. In this work, we adopt this approach and
modify it for our use case for loss calculation given its simplicity and
higher performance.

These changes align HuBERT more closely towards data2vec, primarily differing
in their loss functions for pre-training and other minor changes. The goal is
study how much HuBERT can be improved, with these changes, on the ASR task.

Figure 1: Proposed MS-HuBERT approach, an end-to-end self supervised pre-
training method to learn robust speech representations. The input raw audio is
passed to a CNN encoder. Two copies of the output is created i.e., masked and
unmasked. Which is passed through the Swap modified 2nd encoder. Multicluster
Masked prediction loss is calculated, masked indices only, on the output
embeddings from different blocks of the modified 2nd encoder.

Based on these observations. In this work, we propose MulticlusterSwap-HuBERT
(MS-HuBERT) method, incorporates (i) our proposed Swap method to address the
pre-training and inference mismatch issue, as the [MASK] symbol never appears
during the inference, and (ii) the Multicluster MPL similar to [17]. Our
contributions are as follows:

  1. 1.

We propose MS-HuBERT, an end-to-end self supervised pre-training method to
learn robust speech representations. It combines the Swap method and
Multicluster MPL with HuBERT as shown in Figure 1.

  2. 2.

We show that MS-HuBERT outperforms the original HuBERT on the ASR Librispeech
benchmark by a large margin. And matches the performance of data2vec in high-
resource setting.

  3. 3.

We showcase that the embeddings acquired during pre-training encode crucial
information essential for addressing content based tasks such has ASR and
phoneme recognition (PR). This shows the effective utliziation of the modeling
capacity.

##  2 Method

###  2.1 Background

HuBERT is an iterative pre-training SSL method comprising of two encoders
based on CNN (1st) and transformer (2nd), in that order, architectures. The
CNN encoder serves the dual purpose of down-sampling the input data. The
resulting output is passed, denoted as $U$, to the transformer encoder and its
output is used for loss calculation. During the pre-training stage, raw audio
is passed to the CNN encoder and approximately $50\%$ of the output is masked,
using the masking token $[M]$ and is subsequently passed to the transformer
encoder. The network is then trained to optimize to output a discrete target
sequence by minimizing the masked prediction loss. The complete details can be
found in the original paper [2].

###  2.2 MS-HuBERT

MS-HuBERT augments HuBERT model in two ways (i) the Swap method and (ii) the
Multicluster MPL as shown in Figure 1. Swap method is introduced to address
the pre-training and inference mismatch phase in HuBERT i.e, during inference
the model does not use masking. Swap method modifies the 2nd encoder of
HuBERT, such that the updated model now encounters, two views of the input,
both masked and unmasked inputs during pre-training. Lastly, our proposed
method uses modified Multicluster MPL as proposed by [17], because of its
enhanced model capacity utilization in learning features suitable for the ASR
task. These changes aim to improve the ASR performance, as shown in the Table
1.

####  2.2.1 Swap

Given a raw audio as an input, of batch of size 1, to the 1st encoder (CNN),
its output is denoted as $X=x_{1},x_{2},...,x_{t-1},x_{t}$, where $t$
represents the total number of output tokens. Two views of $X$ are created:
(i) masked view, where on average, around 50% of these tokens are masked,
meaning that half of these tokens are replaced with the $[m]$ token, resulting
in an updated output $X^{m}=x_{1},[m],...,[m],x_{t}$ (view 1) and (ii)
unmasked view, a duplicate of the original $X$, denoted as
$X^{c}=x_{1}^{c},x_{2}^{c},...,x_{t-1}^{c},x_{t}^{c}$ (view 2). These two
views are combined, to form a batch of size 2, and is passed to the 2nd
encoder.

The second encoder has $N$ layers, each composed of a transformer layer
followed by a Swap layer as shown in Figure 1. The transformer layer is
exactly similar to the original HuBERT method. The proposed Swap method's
function is to swap the outputs, at the masked indices, of the transformer
layer between the two views. This updated output serves as input to the next
block of encoder layer, and the process repeats till the last layer. For
example, the output of the transformer layer is
$H_{m}=h_{1},h_{2},...,h_{t-1},h_{t}$ and
$H^{c}=h_{1}^{c},h_{2}^{c},...,h_{t-1}^{c},h_{t}^{c}$ for the masked and
unmasked input respectively. The outputs at the masked indices are now swapped
using the swap method i.e., the updated output are
$H^{m}=h_{1},h_{2}^{c},...,h_{t-1}^{c},h_{t}$ and
$H^{c}=h_{1}^{c},h_{2},...,h_{t-1},h_{t}^{c}$ for the masked and unmasked
input, respectively.

It's important to note that there is no associated loss with the "Swap" layer.
This technique indirectly encourages the model to output the same embeddings
irrespective of the masked and unmasked view.

####  2.2.2 Multicluster MPL

The Multicluster MPL, inspired from [17], involves the computation of masked
prediction loss (MPL) across multiple layers of the transformer encoder, using
multiple set of cluster centers as labels. These encoder layers are selected
equidistant in between the last layer and one intermediate layer. For
instance, consider a scenario with three sets of labels as $(500,250,100)$,
where the last layer index is 12 and the intermediate layer index is 8, the
multiple layers are $(12,10,8)$.

The Multicluster MPL is then formulated as the summation of MPL over $a$,
where $a={(12,500),(10,250),(8,100)}$ is a dictionary of which label set to
use with which transformer encoder layer 111In the original paper $a$ would be
calculated in reverse order i.e., ${(8,500),(10,250),(12,100)}$.. MPL is
computed over the masked indices only, as depicted in Figure 1. Furthermore,
given the GPU memory constraints, we randomly drop $d$ items from the
dictionary $a$ for every forward pass.

$Multicluster\ loss=\sum_{a}(MPL)$.

##  3 Experimental Details

For all the experiments, similar to the HuBERT base model configuration [2],
the MS-HuBERT model comprises a CNN encoder and 12 encoder transformer layers
consisting of 768-dimensional hidden states and 8 attention heads. There is no
large model used for training or comparison purposes.

Datasets: The ASR Librispeech benchmark dataset [20], which is derived from
the LibriVox project, is used for pre-training and supervised finetuning
purposes. It has 3 splits (i) Training, comprising train-clean-100, train-
clean-360, and train-other-500, (ii) Development including dev-other and dev-
clean, and (iii) Testing consists test-other and test-clean. Each data
instance comprises an audio and its corresponding transcript. For pre-training
MS-HuBERT, we use only the raw audios from the combined training split
resulting in a total 960 hours audios. For supervised fine-tuning, three sets
of Libri-Light [21]: 1 hour, 10 hour, 100 hour and the full Librispeech 960
hours dataset is used.

pseudo-labels: Six sets of pseudo-labels with varying numbers of
clusters/resolutions are generated using first iteration HuBERT [2].
Initially, a K-means model with 1000 cluster centers is trained using latent
features extracted from the 6th layer of the first iteration HuBERT base.
Subsequently, another K-means model with 500 cluster centers is trained using
the 1000 cluster centers as features obtained in the prior step. This process
is iteratively repeated four times to train four more K-means models with 250,
125, 50, and 25 cluster centers (in that order) utilizing the cluster centers
extracted from the previous step. This results in a total of 6 set of pseudo
labels used to calculate the Multicluster MPL.

Pre-training: Unlike HuBERT, MS-HuBERT base incorporates 6 classification
heads instead of just 1. This is because of the Multicluster MPL. This results
in a total parameter count of 96.01 million, representing an increment of
around 1.25 million parameters compared to HuBERT. MS-HuBERT is trained for
400,000 iterations on 32 GPUs with a batch size of at most 87.5 seconds of
audio per GPU. The best model checkpoint is determined using the dev-other
subset. Pre-trained models and training configurations will be made available
after the acceptance.

Given the memory constraints and to avoid the out-of-memory error, we randomly
drop 2 clusters, and their respective layer indices, in each gradient update
step. Furthermore, the intermediate layer index is chosen using the formula :
$0.25*12$, where $12$ is the number of transformer encoder layers.

Supervised Fine-tuning and inference: We follow the Wav2Vec 2.0 [1] strategy
to fine-tune MS-HuBERT to minimize the Connectionist Temporal Classification
[22] loss using 8 GPUs. The total batch size is of 200 seconds of audio per
GPU and the best model checkpoint is determined by the lowest Word Error Rate
(WER) achieved on the dev-other split. For inference, 4-gram language model
(LM) is used with a beam width of 500 for dev-other, dev-clean and 1500 for
test-clean and test-other. We do a conservative hyper-parameter search for the
1 hour and 10 hour splits and fixed hyper-parameter are used for the 100 and
960 hours training splits during fine-tuning. The inference hyper-parameters
are searched with Ax, a Bayesian optimization toolkit
222https://github.com/facebook/Ax with a beam-width of 500 using 32 trials.

##  4 Results

###  4.1 Main Results: Supervised Fine-tuning and Inference

Table 1: ASR Librispeech benchmark finetuning results using a 4-gram language
model. wav2vec 2.0 and data2vec are not a direct comparison to MS-HuBERT and
is shown only such that the reader has a broader picture. The readers should
ignore these two models until the Discussion section 5.

Method | dev-clean | dev-other | test-clean | test-other
---|---|---|---|---
1hr
wav2vec 2.0 [1] | 5.0 | 10.8 | 5.5 | 11.3
HuBERT [2] | 5.6 | 10.9 | 6.1 | 11.3
WavLM [9] | - | - | 5.7 | 10.8
data2vec [10] | 4.0 | 8.5 | 4.6 | 9.1
MS-HuBERT | 5.6 | 10.9 | 5.9 | 11.3
\- Swap | 5.9 | 11.6 | 6.2 | 12.3
10hr
wav2vec 2.0 | 3.8 | 9.1 | 4.3 | 9.5
HuBERT | 3.9 | 9.0 | 4.3 | 9.4
WavLM | - | - | 4.3 | 9.2
data2vec | 3.3 | 7.5 | 3.9 | 8.1
MS-HuBERT | 3.6 | 8.5 | 4.1 | 8.8
\- Swap | 3.8 | 8.6 | 4.1 | 9.2
100hr
wav2vec 2.0 | 2.7 | 7.9 | 3.4 | 8.0
HuBERT | 2.7 | 7.8 | 3.4 | 8.1
WavLM | - | - | 3.4 | 7.7
data2vec | 2.2 | 6.4 | 2.8 | 6.8
MS-HuBERT | 2.4 | 7.1 | 3.0 | 7.2
\- Swap | 2.6 | 6.9 | 3.1 | 7.4
960hr
wav2vec 2.0 | 2.0 | 5.9 | 2.6 | 6.1
data2vec | - | - | - | 5.5
MS-HuBERT | 1.8 | 5.1 | 2.4 | 5.5

Table 1 presents the outcomes on the Librispeech ASR benchmark, where MS-
HuBERT is compared with two similar approaches, HuBERT and WavLM. It is
evident that MS-HuBERT yields superior results. The margin of improvement
increase and the size of dataset used for fine-tuning has a direct
proportionality. This is a desired property of any training framework i.e., as
the dataset increase the performance should increase.

Notably, upon the removal of the Swap concept, we observed a degradation in
performance, particularly in low-resource settings. This proves that the Swap
method does indeed contribute positively to the performance gains particularly
in low resource setting.

###  4.2 MS-HuBERT as a Feature Extractor

Table 2: SUPERB fine-tuning results.

Method | PR | ASR
---|---|---
wav2vec 2.0 [1] | 5.74 | 6.43
HuBERT [2] | 5.41 | 6.42
WavLM [9] | 4.84 | 6.21
data2vec [10] | 4.69 | 4.94
MS-HuBERT | 4.42 | 5.60

To study the information encoded/learnt at different layers of the MS-HuBERT
model and how it compares to the original HuBERT, we conduct two experiments:
(i) SUPERB benchmark [23] and (ii) canonical correlation analysis (CCA)
similarity with word labels [24, 25].

SUPERB Benchmark: The SUPERB benchmark is designed to evaluate the efficacy of
a pre-trained model without fine-tuning i.e., using the frozen encoder as a
feature extractor. Specifically, a linear weighted sum of the output
embeddings of all the encoder layers serves as a feature for solving any
particular downstream task. In our study, we aim to assess the quality of 2nd
encoder embeddings, from the MS-HuBERT, for tackling the speech recognition
task. Thus, we employ the ASR and phoneme-recognition (PR) tasks within the
SUPERB benchmark. The evaluation is on the clean split of the ASR Librispeech
benchmark. The results are reported in Table 2. Clearly MS-HuBERT surpasses
HuBERT and similar models by a significant margin. This shows the model's
capability in encoding information crucial in solving the ASR and PR task.
Except on the ASR task using data2vec.

Figure 2: Solid lines show the CCA similarity with the word labels. Dotted
lines show the AUC area under the curve for them respectively.

CCA Similarity with Word Labels: Following the layer-wise analysis conducted
by Pasad et al. [24, 25], we use a modified version of canonical correlation
analysis (CCA). Specifically, a projection-weighted CCA (PWCCA) [26]. The
plots are shown in Figure 2. It is clear that MS-HuBERT significantly enhances
the performance of word-level information across the transformer encoder
layers. Additionally, we compute the area under the curve (AUC) and observe
that it consistently surpasses that of Hubert. This increases the model
capacity utilization compared to HuBERT. We also observe that Swap increases
the CCA similarity in the later layers, which could be the reason of
performance gain in low resource settings on ASR.

##  5 Discussion

In comparison to data2vec [10], our performance on the ASR Librispeech
benchmark, as illustrated in Table 1, still falls short, particularly evident
in low resource scenarios. This difference may stem from the inherent nature
of the MLM pre-text task utilizing discrete tokens. For instance, WER metric
for HuBERT and WavLM in a 1-hour setting lag behind even wav2vec 2.0. However,
as the fine-tuning dataset increases, the performance gap diminishes. When
leveraging the entire 960 hours of the Librispeech dataset, our performance
matches that of data2vec. On the SUPERB benchmark, for the PR task, MS-HuBERT
outperforms data2vec and is comparable in the context of ASR.

Given MS-HuBERT is trained using the pseudo labels generated from the the
first iteration HuBERT, we posit that training two iterations using the MS-
HuBERT methodology could potentially surpass data2vec on the ASR Librispeech
benchmark and ASR tasks within the SUPERB benchmark.

Complexity and Computational Cost: MS-HuBERT introduces additional complexity
to the pre-training process, particularly with the incorporation of the Swap
method and Multicluster loss approach. This increased complexity may result in
higher computational costs increasing the total time for pre-training only.
During inference MS-HuBERT and HuBERT follows the same forward pass.

##  6 Conclusion

Our results highlight the potential of MS-HuBERT in bridging the performance
gap between HuBERT and data2vec on the ASR Librispeech benchmark and content
based tasks, ASR and PR, on the SUPERB benchmark. MS-HuBERT is aimed at
mitigating the pre-training and inference mismatch in masked language modeling
for learning. Building upon the HuBERT framework, MS-HuBERT incorporates two
key modifications: our proposed Swap method, enabling full context access
during pre-training, and the Multicluster loss approach for more effective
training. Through empirical evaluation on the ASR Librispeech benchmark, MS-
HuBERT demonstrates significant performance improvements over the original
HuBERT model, achieving state-of-the-art results and matching the performance
of data2vec in high-resource settings. Future research could explore further
enhancements to the MS-HuBERT methodology to avoid iterative pre-training or
improving the quality pseudo labels altogether. Lastly, scaling the model size
is also an open question.

##  7 ACKNOWLEDGMENTS

Hemant Yadav is supported by Microsoft Research India PhD Fellowship program.
Rajiv Ratn Shah is partly supported by the Infosys Center for AI, the Center
of Design and New Media, and the Center of Excellence in Healthcare at IIIT
Delhi.

## References

  * [1] A. Baevski, Y. Zhou, A. Mohamed, and M. Auli, ``wav2vec 2.0: A framework for self-supervised learning of speech representations,'' _Advances in Neural Information Processing Systems_ , vol. 33, pp. 12 449–12 460, 2020.
  * [2] W.-N. Hsu, B. Bolte, Y.-H. H. Tsai, K. Lakhotia, R. Salakhutdinov, and A. Mohamed, ``Hubert: Self-supervised speech representation learning by masked prediction of hidden units,'' _IEEE/ACM Transactions on Audio, Speech, and Language Processing_ , vol. 29, pp. 3451–3460, 2021.
  * [3] T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell _et al._ , ``Language models are few-shot learners,'' _Advances in neural information processing systems_ , vol. 33, pp. 1877–1901, 2020.
  * [4] Y.-A. Chung, W.-N. Hsu, H. Tang, and J. Glass, ``An unsupervised autoregressive model for speech representation learning,'' _arXiv preprint arXiv:1904.03240_ , 2019.
  * [5] A. v. d. Oord, Y. Li, and O. Vinyals, ``Representation learning with contrastive predictive coding,'' _arXiv preprint arXiv:1807.03748_ , 2018.
  * [6] S. Schneider, A. Baevski, R. Collobert, and M. Auli, ``wav2vec: Unsupervised pre-training for speech recognition,'' _arXiv preprint arXiv:1904.05862_ , 2019.
  * [7] Y.-A. Chung, Y. Zhang, W. Han, C.-C. Chiu, J. Qin, R. Pang, and Y. Wu, ``W2v-bert: Combining contrastive learning and masked language modeling for self-supervised speech pre-training,'' in _2021 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU)_. IEEE, 2021, pp. 244–250.
  * [8] A. Baevski, M. Auli, and A. Mohamed, ``Effectiveness of self-supervised pre-training for speech recognition,'' _arXiv preprint arXiv:1911.03912_ , 2019.
  * [9] S. Chen, C. Wang, Z. Chen, Y. Wu, S. Liu, Z. Chen, J. Li, N. Kanda, T. Yoshioka, X. Xiao _et al._ , ``Wavlm: Large-scale self-supervised pre-training for full stack speech processing,'' _IEEE Journal of Selected Topics in Signal Processing_ , 2022.
  * [10] A. Baevski, W.-N. Hsu, Q. Xu, A. Babu, J. Gu, and M. Auli, ``Data2vec: A general framework for self-supervised learning in speech, vision and language,'' in _International Conference on Machine Learning_. PMLR, 2022, pp. 1298–1312.
  * [11] T. Chen, S. Kornblith, M. Norouzi, and G. Hinton, ``A simple framework for contrastive learning of visual representations,'' in _International conference on machine learning_. PMLR, 2020, pp. 1597–1607.
  * [12] K. He, H. Fan, Y. Wu, S. Xie, and R. Girshick, ``Momentum contrast for unsupervised visual representation learning,'' in _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , 2020, pp. 9729–9738.
  * [13] J.-B. Grill, F. Strub, F. Altché, C. Tallec, P. Richemond, E. Buchatskaya, C. Doersch, B. Avila Pires, Z. Guo, M. Gheshlaghi Azar _et al._ , ``Bootstrap your own latent-a new approach to self-supervised learning,'' _Advances in neural information processing systems_ , vol. 33, pp. 21 271–21 284, 2020.
  * [14] O. Ronneberger, P. Fischer, and T. Brox, ``U-net: Convolutional networks for biomedical image segmentation,'' in _Medical Image Computing and Computer-Assisted Intervention–MICCAI 2015: 18th International Conference, Munich, Germany, October 5-9, 2015, Proceedings, Part III 18_. Springer, 2015, pp. 234–241.
  * [15] A. Riahi and É. Plourde, ``Single channel speech enhancement using u-net spiking neural networks,'' in _2023 IEEE Canadian Conference on Electrical and Computer Engineering (CCECE)_. IEEE, 2023, pp. 111–116.
  * [16] C. Macartney and T. Weyde, ``Improved speech enhancement with the wave-u-net,'' _arXiv preprint arXiv:1811.11307_ , 2018.
  * [17] H. Yadav, S. Sitaram, and R. R. Shah, ``Analysing the masked predictive coding training criterion for pre-training a speech representation model,'' in _ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. IEEE, 2023, pp. 1–5.
  * [18] J. Shi, Y. Tang, H. Inaguma, H. GOng, J. Pino, and S. Watanabe, ``Exploration on hubert with multiple resolutions,'' _arXiv preprint arXiv:2306.01084_ , 2023.
  * [19] J. Shi, H. Inaguma, X. Ma, I. Kulikov, and A. Sun, ``Multi-resolution hubert: Multi-resolution speech self-supervised learning with masked unit prediction,'' _arXiv preprint arXiv:2310.02720_ , 2023.
  * [20] V. Panayotov, G. Chen, D. Povey, and S. Khudanpur, ``Librispeech: an asr corpus based on public domain audio books,'' in _2015 IEEE international conference on acoustics, speech and signal processing (ICASSP)_. IEEE, 2015, pp. 5206–5210.
  * [21] J. Kahn, M. Rivière, W. Zheng, E. Kharitonov, Q. Xu, P. E. Mazaré, J. Karadayi, V. Liptchinsky, R. Collobert, C. Fuegen, T. Likhomanenko, G. Synnaeve, A. Joulin, A. Mohamed, and E. Dupoux, ``Libri-light: A benchmark for asr with limited or no supervision,'' in _ICASSP 2020 - 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , 2020, pp. 7669–7673, https://github.com/facebookresearch/libri-light.
  * [22] A. Graves, S. Fernández, F. Gomez, and J. Schmidhuber, ``Connectionist temporal classification: labelling unsegmented sequence data with recurrent neural networks,'' in _Proceedings of the 23rd international conference on Machine learning_ , 2006, pp. 369–376.
  * [23] S.-w. Yang, P.-H. Chi, Y.-S. Chuang, C.-I. J. Lai, K. Lakhotia, Y. Y. Lin, A. T. Liu, J. Shi, X. Chang, G.-T. Lin _et al._ , ``Superb: Speech processing universal performance benchmark,'' _arXiv preprint arXiv:2105.01051_ , 2021.
  * [24] A. Pasad, B. Shi, and K. Livescu, ``Comparative layer-wise analysis of self-supervised speech models,'' in _ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. IEEE, 2023, pp. 1–5.
  * [25] A. Pasad, J.-C. Chou, and K. Livescu, ``Layer-wise analysis of a self-supervised speech representation model,'' in _2021 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU)_. IEEE, 2021, pp. 914–921.
  * [26] A. Morcos, M. Raghu, and S. Bengio, ``Insights on representational similarity in neural networks with canonical correlation,'' _Advances in neural information processing systems_ , vol. 31, 2018.


XX XX XX XX

# Combining Cognitive Styles Matters for Female Software Designers

C. Pretorius  M. Razavian  K. Eling  & F. Langerak  Eindhoven University of
Technology, The Netherlands

(2020)

###### Abstract

Overcoming society’s complex problems requires novel solutions. Applying
different cognitive styles can promote novelty when designing software aimed
at these problems. Through an experiment with 80 software design
practitioners, we found that female practitioners who had a preference for
more than one cognitive style (intuition and rationality) produced the most
novel software features of all participants.

††journal: IEEE Software

There is consensus in the software engineering community that practitioners
sometimes rely on their intuition when designing software. Despite this, the
emphasis in the software development process has generally been on promoting a
rational cognitive style through rationalized processes, tools, and
techniques. Meanwhile, the potential benefits of an intuitive style have been
largely ignored [1]. One such benefit of intuition is novelty [2], which is
crucial for tackling complex societal problems such as inequality, climate
change, and health.

To address this gap, we carried out an experiment in which software design
practitioners with different cognitive styles designed software features for a
mobile application to address a widespread health behavior problem.

We found that female practitioners produced more novel software features than
male practitioners, especially when they were both highly rational and highly
intuitive.

Our study highlights the importance of considering and combining cognitive
styles when designing new software features, but shows that female
practitioners may uniquely benefit from combining intuitive and rational
cognitive styles.

## WHY FEATURE NOVELTY?

When designing software for a complex problem, software design practitioners,
like product designers and requirements engineers, create new features to
(partially) solve the problem. These practitioners tend to start off by
sketching various ideas for a design on a whiteboard or piece of paper [3].
They will then cycle back and forth between their understanding of the problem
and their idea(s) for a potential feature, updating these concurrently as they
go along.

Nowadays, software solutions naturally lend themselves to addressing societal
problems. However, the reality is that such problems demand substantial levels
of novelty in software features [4].

## COGNITIVE STYLE AND GENDER MEET FEATURE NOVELTY

Software design practitioners, like all people, have different cognitive
styles. Cognitive style describes differences in how people obtain, organize,
and process information [5]. Intuition and rationality are two such cognitive
styles. A practitioner designing a software feature through an intuitive style
might do so quickly, and have a gut feeling that their solution is the right
one. Conversely, a practitioner using a rational style would arrive at a
particular feature more slowly, justifying their solution in the context of
available requirements. Both styles can be used by a practitioner at any time,
in a particular order or even simultaneously. Still, all people tend to
usually rely on one or both styles in a specific configuration [6], known as
their dispositional style.

Both intuition and rationality have been positively related to novelty.
Intuition has been shown to result in more novel solutions through holistic
information processing and promoting associative thinking [2]; the “big
picture”. Rationality enables practitioners to assess details, and to
analytically compare potential solutions [6, 7]. Nevertheless, whether this is
specifically true for software design practice remains to be seen.

Although dispositional style is not inherently gender-specific, it has been
shown that the interaction between gender and job type can influence
preference for intuition [7]. Given that female practitioners are often
underrepresented in software engineering [8], endure unique barriers to
entering the field [9], and are subject to a number of different biases [10],
we were particularly curious about whether the novelty of software features
designed by software design practitioners would vary based on their gender and
dispositional style. When speaking of male and female in our study, we take
gender to be a self-identification construct, which may or may not align with
biology or presentation [11].

Given these potential associations between cognitive style and feature
novelty, and gender differences in style preference, our study investigated
whether certain combination(s) of cognitive style and gender led to higher
software feature novelty.

## STUDY DESIGN

We conducted an experiment with practitioners to enable some control, while
still maintaining real-world applicability. Practitioners, whose primary task
involves high level design of features in any software engineering role, were
recruited through the online platform Prolific. Such participants are familiar
with the complexity of the task, and comfortable with producing rough,
wireframe-like sketches. First, participants took part in a feature design
task. Afterwards, the same participants were randomly assigned to evaluate the
novelty of ten features designed by others.

We chose to focus on the health issue of obesity as our context, being a well-
known issue that participants would at least be familiar with.

### Feature design task

Participants were given an explanation of the problem, and instructed to
design at least one feature for a mobile application. They were then given 15
minutes to sketch their software feature(s) on a piece of paper and provide
suitable explanations, using a basic template as per [3]. Figure 1 presents a
selection of the designed features.

Figure 1: A selection of designed software features with high, low, and medium
novelty scores respectively.

Afterwards, we asked the participants to note which of their features, if they
designed more than one, solved the problem best. The participants then
photographed or scanned their features for upload.

To measure participants’ dispositional cognitive style, we used the REI-10
(rational experiential inventory), which consists of five statements about
participants’ use of intuition and five about their use of rationality [6],
measured on a 7-point scale from “completely disagree” to “completely agree.”
We dropped one item from the rationality scale that reduced the scale
validity.

We collected participant’s self-identified gender in the same section in which
we asked control questions about work-relevant experience, industry role, age,
and familiarity with the obesity problem. Participants were paid four English
pounds for completing this part of the study.

### Feature evaluation task

After completing the design task, participants were contacted again, and
randomly formed into groups of five participants. Each group evaluated the
same ten randomly selected features (always excluding their own). For each
feature design sketch, participants were required to answer the question, “How
novel is this feature when compared with existing features from applications
in the market?” Answers were recorded on a five-point scale ranging from “not
novel at all” to “extremely novel”. Participants were paid two English pounds
for completing the evaluation of ten features.

To measure feature novelty, we calculated an average novelty score for each
participant based on the five evaluations of their best (or only) feature.

### The sample

110 practitioners had their top-rated feature evaluated. This was reduced to
80 following data cleaning. 26.25% of the participants were female, and 73.75%
were male. 23.8% of the participants had a high preference for intuition, but
low for rationality; 22.5% had a high preference for rationality, but low for
intuition; 22.5% high preference for both; and 31.2% for neither cognitive
style. Participants’ professional design experience varied from less than one
year to more than 20 years, with the average being 5.44 years.

### Data analysis

We used hierarchical moderated regression analysis to determine whether
gender, intuitive style or rational style could significantly account for
differences in feature novelty in isolation and when taken together. For this
purpose, the three variables were included individually and in all possible
two-way and three-way combinations in our model. To account for other
influences, we initially included the number of features designed, experience,
and age of participants in the model, but these did not correlate with feature
novelty. With an R-squared value of .196 and a coefficient F-value of 2.509,
the variables included in the final model accounted for 19.6% of the variance
in feature novelty among participants (constant 2.510).

## HOW COGNITIVE STYLE AND GENDER EXPLAIN FEATURE NOVELTY

### Cognitive style alone does not matter

We found that cognitive style was unrelated to feature novelty on its own.
Neither a more intuitive nor a more rational dispositional style per se led
participants to design a software feature of higher novelty.

### Female practitioners create more novel features

Gender, in contrast, was positively associated with feature novelty. We found
that the female practitioners in our experiment produced more novel software
features than the male practitioners did.

### Cognitive style matters for female practitioners

Cognitive style and gender taken together are also positively related to
feature novelty. Female practitioners with a higher intuitive preference
designed significantly more novel software features. Additionally, we found
that female practitioners produced the most novel features when they had a
preference for both intuition and rationality.

The two heatmaps in Figure 2 illustrate the relationships between intuition,
rationality, and feature novelty for male and for female practitioners. It is
important to keep in mind that only the high intuition, and high intuition
with high rationality portions in the female practitioner of the regression
model, are statistically significant.

Figure 2: Graphical representation of the relationship between cognitive style
and software feature novelty, separated by gender. Red regions show higher
novelty, whereas blue regions show lower novelty.

## DISCUSSION AND KEY TAKEAWAYS

Our study shows that both cognitive styles (intuition and rationality) as well
as gender matter for software feature novelty. The way that they matter leads
to several important takeaways from our study.

First, since gender was positively associated with software feature novelty in
our study, it is crucial to further investigate the role of female
practitioners in software design activities. Perhaps software teams could
benefit from involving women specifically, but this needs to be empirically
established.

Second, since neither cognitive style was positively related to novelty on its
own, it does not make sense to lean on a single cognitive style, independent
of other factors, to design novel software features. Previous research and
practice in software engineering has generally prescribed the use of
rationality either explicitly (e.g. through design reasoning techniques [12])
or implicitly (e.g. by imposing structured development methods and lifecycle
models [13]). Focusing entirely on rationality or intuition is not supported
by our study. Instead, other factors need to be considered, particularly the
gender of the practitioner.

Indeed, we found that gender has implications for choosing a cognitive style
in software feature design. Female practitioners should not be discouraged
from making use of intuition exclusively, or combining rationality with
intuition, when designing software features. In practice, intuition can even
be promoted through behaviors such as brainstorming and sketching to
intuitively come up with potential solutions, as well as giving female
practitioners incubation time (i.e., distraction from consciously considering
the problem) after being exposed to a problem situation [14].

Currently, we cannot draw certain conclusions from the male portion of our
regression model. However, it is possible that male practitioners with a
dispositional preference for either intuition or rationality design more novel
software features than their male peers in contexts other than our study. This
also raises the question of whether forcing a rational style among intuitively
strong male practitioners is actually beneficial for these practitioners.

## LIMITATIONS AND FUTURE WORK

To our knowledge, this is the first study that investigates the relationship
between the combination of cognitive style and gender, and software-related
performance outcomes. We hope that it will encourage further work on this
important subject. However, in our study design, we have solely considered a
black-box approach to the problem, which does not consider how male and female
practitioners might design software features differently. Such white-box
studies, particularly qualitative design studies, are imperative for
understanding these differences in practice. Perhaps the differences we found
can be explained by the pressure sometimes experienced by female practitioners
to prove themselves [15], as an example.

The black-box nature of our study leads to two further potential limitations.
First, although we controlled for many extraneous variables, there are likely
other variables, such as self-confidence [8], that we did not control for.
Second, although our sample consists of software design practitioners from
many different geographic locations and roles in industry, it is possible that
our sample is not perfectly representative.

Finally, our study focused on the individual level. Although some aspects of
studying individual practitioners can be applied to the team level,
researchers should also investigate the novelty of software designed by teams.
Teams can differ in terms of cognitive style and gender representation.
Interactions between individual practitioners based on these differences could
have unique consequences for software novelty, perhaps through issues like
groupthink and power dynamics invading or supporting the group context.

## References

  * [1] C. Pretorius, M. Razavian, K. Eling, and F. Langerak, “Towards a dual processing perspective of software architecture decision making,” in _IEEE 15th International Conference on Software Architecture Companion (ICSA-C 2018)_. Piscataway: IEEE, 2018, pp. 48–51.
  * [2] J. Pétervári, M. Osman, and J. Bhattacharya, “The role of intuition in the generation and evaluation stages of creativity,” _Frontiers in Psychology_ , vol. 7, no. 1420, pp. 1–12, 2016.
  * [3] R. Mohanani, P. Ralph, and B. Shreeve, “Requirements fixation,” in _Proceedings of the 36th International Conference on Software Engineering (ICSE ’14)_. Hyderabad, India: ACM, 2014, pp. 895–906.
  * [4] D. H. Cropley, _Creativity in engineering: Novel solutions to complex problems_. London, UK: Academic Press, 2015\.
  * [5] I. Aggarwal and A. W. Woolley, “Team creativity, cognition, and cognitive style diversity,” _Management Science_ , vol. 65, no. 4, pp. 1586–1599, 2019\.
  * [6] S. Epstein, R. Pacini, V. Denes-Raj, and H. Heier, “Individual differences in intuitive–experiential and analytical–rational thinking styles.” _Journal of Personality and Social Psychology_ , vol. 71, no. 2, pp. 390–405, 1996.
  * [7] C. Akinci and E. Sadler-Smith, “Assessing individual differences in experiential (intuitive) and rational (analytical) cognitive styles,” _International Journal of Selection and Assessment_ , vol. 21, no. 2, pp. 211–221, 2013.
  * [8] B. Vasilescu, A. Capiluppi, and A. Serebrenik, “Gender, representation and online participation: A quantitative study of StackOverflow,” _Proceedings of the 2012 ASE International Conference on Social Informatics, SocialInformatics 2012_ , pp. 332–338, 2012.
  * [9] A. Wolff, A. Knutas, and P. Savolainen, “What prevents Finnish women from applying to software engineering roles? A preliminary analysis of survey data,” in _Software Engineering Education and Training (ICSE-SEET ’20)_. New York, NY, USA: ACM, 2020.
  * [10] Y. Wang and D. Redmiles, “Implicit gender biases in professional software development: An empirical study,” _Proceedings of the 41st International Conference on Software Engineering: Software Engineering in Society (ICSE-SEIS ’19)_ , pp. 1–10, 2019.
  * [11] M. Burnett, S. Stumpf, J. Macbeth, S. Makri, L. Beckwith, I. Kwan, A. Peters, and W. Jernigan, “Gendermag: A method for evaluating software’s gender inclusiveness,” _Interacting with Computers_ , vol. 28, no. 6, pp. 760–787, 2016.
  * [12] A. Tang, F. Bex, C. Schriek, and J. M. E. van der Werf, “Improving software design reasoning – a reminder card approach,” _Journal of Systems and Software_ , vol. 144, no. February, pp. 22–40, 2018.
  * [13] P. Ralph, “The two paradigms of software development research,” _Science of Computer Programming_ , vol. 156, no. May, pp. 68–89, 2018.
  * [14] K. J. Gilhooly, G. J. Georgiou, J. Garrison, J. D. Reston, and M. Sirota, “Don’t wait to incubate: Immediate versus delayed incubation in divergent thinking,” _Memory and Cognition_ , vol. 40, no. 6, pp. 966–975, 2012.
  * [15] K. Blincoe, O. Springer, and M. R. Wróbel, “Perceptions of gender diversity’s impact on mood in software development teams,” _IEEE Software_ , vol. 36, no. 5, pp. 51–56, 2019.

Carianne Pretorius is a doctoral candidate in the School of Industrial
Engineering at Eindhoven University of Technology in the Netherlands. Her
research interests include software design decision making, cognition, and
human aspects of software engineering. Pretorius received her M.A. in Socio-
Informatics cum laude from Stellenboch University in South Africa. Contact her
at<EMAIL_ADDRESS>
Maryam Razavian is an assistant professor in industrial engineering at
Technical University of Eindhoven. Her research interests include software
design reasoning, human aspects of software design, software architecture, and
service orientation. Razavian received her Ph.D. in Computer Science from VU
University Amsterdam. She’s a member of ACM and IEEE. Contact her at
<EMAIL_ADDRESS>
Katrin Eling is an assistant professor of new product development in the
School of Industrial Engineering at Eindhoven University of Technology in the
Netherlands. She received her Ph.D. from the same school, has an M.Sc. in
strategic product design from Delft University of Technology, and a diploma in
industrial design from the University of Wuppertal, Germany. Dr. Eling’s
research focuses on the successful management of the front end of innovation.
Contact her at<EMAIL_ADDRESS>
Fred Langerak is professor of product development and management in the
Innovation, Technology Entrepreneurship & Marketing Group of the School of
Industrial Engineering at Eindhoven University of Technology in the
Netherlands. He has a M.Sc. and Ph.D. from the Erasmus School of Economics.
His research focuses on managerial interventions to improve the process of
conceiving, designing, developing, and bringing new products to market, and
managing these products post‐launch. Contact him at<EMAIL_ADDRESS>


Lossless compression implementations typically contain two programs,
an encoder and a decoder, which are required to be inverse to one
another. We observe that a significant class of compression methods,
based on asymmetric numeral systems (ANS), have shared structure
between the encoder and decoder—the decoder program is the `reverse'
of the encoder program—allowing both to be simultaneously specified
by a single, reversible function. To exploit this, we have implemented
a small reversible language, embedded in Agda, which we call
`Flipper' (available at github.com/j-towns/flipper). Agda supports
formal verification of program properties, and the compiler for our reversible
language (which is implemented as an Agda macro), produces not just an
encoder/decoder pair of functions but also a proof that they are inverse to one
another. Thus users of the language get formal verification `for free'. We give
a small example use-case of Flipper in this paper, and plan to publish a full
compression implementation soon.

It has been known since the work of Claude Shannon in the late 1940s
that there is a deep connection between probabilistic modelling
and lossless data compression. In <cit.>, Shannon showed
that the best achievable compressed size for random data is equal to
the expected negative log-probability, a quantity which he called the
`entropy'.  Practical approaches to lossless compression work by
assuming a model \(P\) over input data \(x\), and compressing \(x\) to
a size close to the negative log-probability \(-\log_2 P(x)\), using a
coding method such as Huffman coding <cit.>, arithmetic
coding <cit.>, or asymmetric numeral systems (ANS)
Recently, a number of papers have shown that modern deep generative
models can be used to achieve state-of-the-art compression rates
<cit.>. These recent works all use ANS, a method which, due to
its simplicity and superior performance, has also become ubiquitous in
production compression systems.[A list is maintained at
2078-List-of-Asymmetric-Numeral-Systems-implementations, which
includes various ANS programs with state-of-the-art performance.]

Our work on this paper began with an observation: in ANS-based
methods, the encoder and decoder programs appear to have the same
structure.  Indeed, we realised that it made sense to view them as
different interpretations of the same program, one `doing' the
program and the other `undoing' it. Following this idea to its logical
conclusion, we have implemented a small, reversible domain
specific language, called Flipper, embedded in Agda <cit.>,
via an Agda macro. The potential advantages of using Flipper are
significant—the amount of user code is reduced, and accidental
violation of the inverse property (a common cause of bugs in lossless
compression) is no longer possible.

Flipper programs inhabit a record type of bijections between two sets
A and B:[
We would have preferred the names `do' and `undo' to `apply' and
`unapply', but unfortunately `do' is a reserved keyword in Agda.

[43]

The Flipper macro takes a user implementation of
apply in

and compiles it into an inhabitant of

B, complete with an implementation of unapply
and the two necessary correctness proofs.

The compiler only accepts functions expressed in an `obviously'
reversible form, where the reverse interpretation can be seen by
literally rotating the source code by \(180^\circ\) (more detail on
this below). Despite this restriction on form, we have found Flipper
to be a surprisingly effective language for implementing lossless
compression, and surprisingly enjoyable to use.

There is a tradition of `reversible' programming languages going back
at least as far as <cit.>, and we took particular
inspiration from the pure functional reversible languages rFun
<cit.> and Theseus <cit.>. We believe
that Flipper is the first in this tradition to support dependent
types, and as far as we know the first to be formally verified. We
also think that embedding the reversible language in a well
established host language, with excellent editor integration, makes
writing software in Flipper significantly easier than existing
reversible languages. The power of these features is demonstrated by
our implementation of ANS-based compression in Flipper, which to our
knowledge is the most sophisticated compression method to have been
implemented in a reversible language to date (we plan to publicly release this

§ THE FLIPPER LANGUAGE

The Flipper compiler accepts an

expressed as a pattern-matching lambda term with a special
property: rotating the term by \(180^\circ\),
and “re-righting” any upside down symbols, results in a valid
term in

A, which we can use as our unapply.

As a first example, consider the `flippable' function which swaps the
elements of a pair:

Here the Flipper compiler, F, is applied to a lambda
term. Flipping that term gives

Rerighting symbols (including the
and moving the
over to the left, we see that in this case

Note that, in order for the rotated unapply to be a valid
Agda function, it is necessary that each bound variable in a flippable
must be used exactly once.  To emphasize the view of apply
and unapply as different interpretations (or orientations)
of the same program, from here on we will use a more concise,
symmetrical syntax for flippable definitions:

\(x\)                                                                                variables

\(c\)                                                                Agda constructors

\(T\)                                                                                Agda terms

\(p\)       ::=    \(x\) \(\mid\)
(\(c\) \([\) \(p\) \(]\))   patterns

\(f\)       ::=    F { \(bs\) } \(\mid\) \(T\)   flippables

\(bs\)      ::=    \(b\) \(\mid\) \(b\) ; \(bs\)                               branches

\(b\)       ::=    \(p\) \(\AgdaSymbol{\ensuremath\leftrightarrow}\) \(B\)

\(B\)       ::=    \(p\) \(\mid\) \(p_1\)  \(f\)
\(p_2\)
\(B\)

Grammar of the Flipper language. The notation \([\) \(p\)
\(]\) stands for a list of zero or more patterns. Note that
infix Agda constructors, such as
are also allowed in patterns.

New flippables can be built from existing ones: the

is interpreted in apply as “apply the flippable
\(f\) to the variable \(a\) and bind the result to \(b\) in \(T\)”,
equivalent to the expression
T. We can, for example, compose two flippables as follows:

[0]

[28]
[34]
[44]

[28]
[34]
[44]

To allow for conditional branching, flippable expressions can
contain multiple clauses, corresponding to distinct
input/output patterns. For example, the following flippable swaps the
branches of a sum type:

[30]

[30]

In order for a pattern lambda with multiple clauses to be flippable,
the body expressions on the right hand side must partition the output
type: each possible constructor must appear exactly once.

Finally, between being bound and being used, variable names are
considered `in scope', and can be freely referred to inside

(this doesn't count as a `use'), an example is the way that
a is referred to in the flippable
uncurry combinator:

§ BITS BACK CODING IN FLIPPER

As a brief example use case, we show how to implement the `bits-back
with ANS' (BB-ANS) method from <cit.> in Flipper. This is
not an implementation of ANS itself, but a method for composing
ANS-based codecs in order to encode data using a latent variable

var=[circle,draw=black,minimum size=7mm]
latent  =[]
at (0,   0) [var,latent]   (z)  \(z\);
at (1.5, 0) [var,observed] (x)  \(x\);
[->]  (z) – (x);

Graphical model with latent variable \(z\) and observed
variable \(x\).

We assume a model over data \(x\) where we only have access to the
joint distribution \(P(x, z)\) for some `latent' variable \(z\), and
an approximate posterior \(Q(z\given x)\), but not to the

\begin{equation}
P(x) = \int_z P(x, z) \;\mathrm{d}z.
\end{equation}

We define a flippable Encoder type, parameterized by a compressed
message type, C, and the type X of data to be

The apply function of an Encoder accepts
some compressed data in C and a symbol in X
and returns a new element of C from which the original
compressed data and the symbol can both be recovered. An
can be
used to compress a list of elements of X, by (flippably)
folding over the list.

The BB-ANS method can be expressed in Flipper as:

[10]

[62]
[85]
[101]

[62]

[85]
[101]

[62]

[85]
[101]

This matches the Python implementation presented in <cit.>, but with only one function required instead of two.

§ CONCLUSION

We have presented the Flipper language, for which we have implemented
a compiler as an Agda macro. Flipper can be used to reduce the amount
of code, and to avoid a class of bugs, when implementing lossless
compression programs. We hope to release Flipper, as well as the full
ANS compression implementation in Flipper, as soon as possible.

§ ACKNOWLEDGEMENTS

Thanks to Wouter Swierstra and Heiko Zimmerman for their advice and
feedback on drafts of this paper.
This publication is part of the project “neural networks for
efficient storage and communication of information” (with project
number VI.Veni.212.106) of the research programme
“NWO-Talentprogramma Veni ENW 2021” which is financed by the Dutch
Research Council (NWO).

Missing 'biblatex' package
The bibliography requires the 'biblatex' package.

booktitleInternational Conference on Learning Representations (ICLR)
titleIDF++: Analyzing and Improving Integer Discrete Flows for Lossless Compression

eprintclasscs, math
titleAsymmetric Numeral Systems

booktitleAdvances in Neural Information Processing Systems 32
titleInteger Discrete Flows and Lossless Compression

journaltitleProceedings of the IRE
titleA Method for the Construction of Minimum-Redundancy Codes

booktitleConference on Reversible Computation (RC)
noteWork in progress report.
titleTheseus: A High Level Language for Reversible Computing

booktitleAdvances in Neural Information Processing Systems
titleVariational Diffusion Models

booktitleProceedings of the 36th International Conference on Machine Learning
titleBit-Swap: Recursive Bits-Back Coding for Lossless Compression with Hierarchical Latent Variables

noteLetter to R. Landauer.
titleJanus: A Time-Reversible Language

journaltitleACM Transactions on Information Systems (TOIS)
shortjournalACM Trans. Inf. Syst.
titleArithmetic Coding Revisited

Department of Computer Science and Engineering, Chalmers University of Technology

titleTowards a Practical Programming Language Based on Dependent Type Theory

booktitleProceedings of the 38th International Conference on Machine Learning
titleImproving Lossless Compression Rates via Monte Carlo Bits-Back Coding

journaltitleThe Bell System Technical Journal
titleA Mathematical Theory of Communication

New York, NY, USA

booktitleProceedings of the 27th Symposium on the Implementation and Application of Functional Programming Languages
seriesIFL '15
titleInterpretation and Programming of the Reversible Functional Language RFUN

booktitleInternational Conference on Learning Representations (ICLR)
titlePractical Lossless Compression with Latent Variables Using Bits Back Coding

booktitleInternational Conference on Learning Representations (ICLR)
titleHiLLoC: Lossless Image Compression with Hierarchical Latent Variable Models

Berlin, Heidelberg

booktitleReversible Computation
seriesLecture Notes in Computer Science
titleTowards a Reversible Functional Language

booktitleAdvances in Neural Information Processing Systems
titleOn the Out-of-distribution Generalization of Probabilistic Image Modelling


# DiffRoll: Diffusion-based Generative Music Transcription
with Unsupervised Pretraining Capability

###### Abstract

In this paper we propose a novel generative approach, DiffRoll, to tackle
automatic music transcription (AMT). Instead of treating AMT as a
discriminative task in which the model is trained to convert spectrograms into
piano rolls, we think of it as a conditional generative task where we train
our model to generate realistic looking piano rolls from pure Gaussian noise
conditioned on spectrograms. This new AMT formulation enables DiffRoll to
transcribe, generate and even inpaint music. Due to the classifier-free
nature, DiffRoll is also able to be trained on unpaired datasets where only
piano rolls are available. Our experiments show that DiffRoll outperforms its
discriminative counterpart by 19 percentage points (ppt.) and our ablation
studies also indicate that it outperforms similar existing methods by 4.8
ppt.Source code and demonstration are available at
https://sony.github.io/DiffRoll/.

###### keywords:

Automatic Music Transcription, Music Information Retrieval, Signal Processing,
Diffusion, Generative Model, Unsupervised Pretraining

##  1 Introduction

Automatic Music Transcription (AMT) has typically been treated as a
discriminative task [1] in which the frequency bins of spectrograms are
projected onto 88 notes (from A0 to C8) in posteriorgrams. There have been
attempts at modeling AMT using Bayesian inference [2] and probabilistic models
[3], however, these models lack the generative power found in variational
autoencoders (VAEs) [4] and generative adversarial networks (GANs) [5, 6].

In addition, the majority of existing state-of-the-art (SOTA) AMT models are
fully supervised [7, 8, 9, 10, 11]. While there have been attempts to develop
weakly-supervised [12] and semi-supervised AMT models [13, 14], unsupervised
deep learning AMT is still an under-explored direction and thus currently
limited to drum transcription [15] and non-deep learning based approaches [3,
16]. In this paper, we have devised a novel formulation for AMT called
“DiffRoll”, in which the power of diffusion is leveraged, making it a
generative model capable of generating new piano rolls with the potential of
unsupervised pretraining.

##  2 Related Work

Diffusion model in deep learning inspired by the concept of non-equilibrium
thermodynamics [17], has shown promising results in generating highly-fidelity
audio samples [18, 19]. Subsequent improvements have led to better diffusion
methods, for example, better forward process [20], new regularization [21],
faster sampling speed [22], and discrete diffusion for binary data [23]. In
the field of vision, conditional diffusion such as DALL-E [24] enables
generated images to be controlled by an input text, resulting in SOTA text-to-
image performance.

Diffusion is now also being used for music-related tasks such as music
generation [25] and music synthesis [26]. These tasks are intuitively
considered as generative and hence it is reasonable to model them using
diffusion. AMT, on the other hand, has typically been considered a
discriminative task in which an AMT model classifies the notes present in the
spectrograms frame-by-frame. Even though there have been attempts [8, 27] to
model AMT as a token-based task inspired by natural language processing, these
approaches are still discriminative. AMT, however, can be modeled as a
generative task. The spectrograms can be considered as the conditions, and the
piano rolls can be considered as the image to be generated. In other words, we
can think of AMT as a piano roll generation task, which is the reverse process
of music synthesis [26]. In this paper, we take a novel perspective by
modeling AMT as a generative task, with the source code and demonstration
available111https://sony.github.io/DiffRoll/.

##  3 Proposed Method

DiffRoll, as shown in Fig. 1, is designed to convert Gaussian noise $x_{t}$
into a posteriorgram $\hat{x}_{0}$ conditioned on spectrogram
$c_{\text{mel}}$. To handle binary piano rolls $x_{\text{roll}}\in\\{0,1\\}$
during training, we cast them into $[0,1]$, similar to Analog Bits [23].
During sampling, the posteriorgram is binarized back into piano roll with a
threshold of $0.5$ and then exported as a MIDI file.

The DiffRoll model architecture is inspired by DiffWave [18] which is also a
1D convolutional model. In DiffWave, $x_{t}$ is a 1-channel tensor, while in
DiffRoll we consider $x_{t}$ as an 88-channel tensor with a dimension of
$(B,88,\tau)$, where $B$ is the batch size and $\tau$ is the number of frames
in the piano roll. As in DiffWave, $x_{t}$ is projected into a tensor with
shape $(B,512,\tau)$ via a 1D convolutional layer with a kernel size of $1$. A
total of 15 residual layers with the same design as in DiffWave are used,
where the output of the previous residual layer is added to the input of the
next layer. A diffusion time $t$, i.e. a batch of integers with shape $(B,1)$,
is projected into $(B,512)$ and then broadcasted to $(B,512,\tau)$ such that
it can be added to the input of each residual layer. Note that to not confuse
diffusion time $t$ with the time dimension of the spectrograms or piano rolls
$\tau$, $t$ and $\tau$ are completely different from each other. To condition
the posteriorgram generation $\hat{x}_{0}$, there are 1D convolutional layers
projecting the mel spectrograms from $(B,229,\tau)$ to $(B,512,\tau)$ such
that they can be added to the output of the residual layers. Finally, a tensor
with shape $(B,512,\tau)$ is projected back to a tensor with shape
$(B,88,\tau)$ as the generated posteriorgram $\hat{x}_{0}$.

During forward process, denoted as $q(x_{t}|x_{0})$ in Fig. 1, noisy piano
roll $x_{t}$ is directly sampled from clean piano roll $x_{\text{roll}}$ using
Eq. (1).

$x_{t}=\sqrt{\bar{\alpha}_{t}}x_{\text{roll}}+\sqrt{1-\bar{\alpha}_{t}}\epsilon,$
(1)

where $\epsilon\sim\mathcal{N}\left(\textbf{0},\textbf{I}\right)$ is sampled
from a standard Gaussian distribution,
$\bar{\alpha}_{t}=\prod_{s=1}^{t}\alpha_{s}$, and $\alpha_{0}=1$. We used
linear noise scheduling in which $\alpha_{t}\in[0.9999,0.98]$.

During training, model $f_{\theta}(x_{t},t,c_{\text{mel}})$ with weight
$\theta$ directly outputs $\hat{x}_{0}$ given noisy piano roll $x_{t}$,
diffusion step $t$, and mel spectrogram $c_{\text{mel}}$, where $t$ is
uniformly sampled from $[1,2,\dots,T]$. We set $T=200$ as preliminary
experiments demonstrated that it yields the best result. To enable DiffRoll to
transcribe and generate music, we need both a conditional as well as an
unconditional model as defined in Eq. (2).

$\hat{x}_{0}(\cdot)=\begin{cases}f_{\theta}(x_{t},t,c_{\text{mel}}),&\mbox{conditional}\\\
f_{\theta}(x_{t},t,-{1}),&\mbox{unconditional}\end{cases},$ (2)

Although both models should produce the same output $\hat{x}_{0}(\cdot)$, we
use $\hat{x}_{0}(c_{\text{mel}})$ and $\hat{x}_{0}(-1)$ to differentiate their
outputs. To train both models at the same time without introducing extra
parameters, we use the concept of Classifier-Free Guidance (CFG) [28]. More
specifically a dropout layer as shown in Fig. 1, is used to randomly mask
$c_{\text{mel}}$ in each batch by $-1$ with probability $p$. We chose $-1$ as
the dropout value due to the fact that $0$ corresponds to silence in
$c_{\text{mel}}\in[0,1]$ and we want to avoid confusing the model during
training. The model is trained to minimize the L2 loss between model output
$\hat{x}_{0}(\cdot)$ and ground truth label $x_{\text{roll}}$, as shown in Eq.
(3), similar to previous approaches [29, 30]. A full explanation of this loss
is provided in the supplementary material11footnotemark: 1.

$L=\|x_{\text{roll}}-\hat{x}_{0}(\cdot)\|^{2}$ (3)

Although model output $\hat{x}_{0}$ can be easily converted into $x_{t-1}$ via
$q(x_{t-1}|\hat{x}_{0},x_{t})$ (line 4-6 of Algorithm 1), $x_{t-1}$ is not
used during training. The usage of $x_{t-1}$ is discussed further in the next
paragraph.

1$x_{T}\sim\mathcal{N}\left(\textbf{0},\textbf{I}\right)$

2 for  _$t=T,\dots,1$_ do

3 $\epsilon\sim\mathcal{N}\left(\textbf{0},\textbf{I}\right)$ if $t>1$, else
$\epsilon=0$

4 $\hat{x}_{0}=(1+w)\hat{x}_{0}(c_{\text{mel}})-w\hat{x}_{0}(-1)$

5
$\hat{\epsilon}_{\theta}^{t}=(x_{t}-\sqrt{\bar{\alpha}_{t}}\hat{x}_{0})(\sqrt{1-\bar{\alpha}_{t}})^{-1}$

6
$x_{t-1}=\sqrt{\bar{\alpha}_{t-1}}\hat{x}_{0}+\sqrt{1-\bar{\alpha}_{t-1}-\sigma_{t}^{2}}\hat{\epsilon}_{\theta}^{t}+\sigma_{t}\epsilon$

7

8 end for

return $x_{0}$

Algorithm 1 Sampling

We designed Algorithm 1 for sampling as shown in Fig. 2. First, $x_{T}$
($T=200$ in our experiments) is sampled from a standard Gaussian distribution
$\mathcal{N}(\textbf{0},\textbf{I})$. Then Eq. (2) is used to obtain
conditional output $\hat{x}_{0}(c_{\text{mel}})$ and unconditional output
$\hat{x}_{0}(-1)$. Next, a weighting $w$ is applied to both
$\hat{x}_{0}(c_{\text{mel}})$ and $\hat{x}_{0}(-1)$ on line 1 of Algorithm 1
to obtain $\hat{x}_{0}$. Finally, $\hat{x}_{0}$ is used on line 1 to obtain
$x_{t-1}$, which is denoted as $q(x_{t-1}|\hat{x}_{0},x_{t})$. Note that line
1 requires predicted noise $\hat{\epsilon}_{\theta}^{t}$ at diffusion time
step $t$, whereas our model directly predicts $\hat{x}_{0}$. Therefore, we
need line 1 of Algorithm 1 to calculate $\hat{\epsilon}_{\theta}^{t}$ from
$\hat{x}_{0}$. We denote this reverse process step as
$p_{\theta}(x_{t-1}|x_{t},c_{\text{mel}})$ in Fig. 2. The reverse process is
performed iteratively until $x_{0}$ is reached. We experimented with both
$\sigma_{t}=\sqrt{(1-\bar{\alpha}_{t-1})/(1-\bar{\alpha}_{t})}\sqrt{1-\alpha_{t}}$
(DDPM [20]) and $\sigma_{t}=0$ (DDIM [22]) and found that DDIM yields slightly
worse transcription accuracy. Therefore, we used $\sigma_{t}$ for DDPM in our
experiments.

Fig. 1: Training phase of DiffRoll. Within each batch, several spectrograms
are randomly converted to -1 via the dropout layer.

Fig. 2: Diffusion process of DiffRoll. Forward process converts
$x_{\text{roll}}$ into $x_{t}$ for $1\leq t\leq T$ via a Markov process.
Reverse process $p_{\theta}(x_{t-1}|x_{t},c_{\text{mel}})$ is performed
iteratively until $x_{0}$ is reached. A threshold of 0.5 is applied to $x_{0}$
to obtain the predicted piano roll.

##  4 Experiment

The training split and test split of the MAESTRO dataset [31] were used to
train and test our models. During training, we used the validation split to
monitor the loss. For simplicity, we ignored the velocity information of the
piano roll. Following previous work [31, 32], we downsampled all of the audio
clips to $16$kHz. The nnAudio [33] library was used to extract mel
spectrograms with $229$ mel bins ranging from $0$ to $8,000$Hz, with a window
size of $2,048$ samples from the audio segments ($327,680$ samples). The
magnitude of the mel spectrograms was compressed by taking the natural
logarithm and then normalizing it so that $c_{\text{mel}}\in[0,1]$. Although
the original diffusion model was designed to work with $x_{t}\in[-1,1]$
$\forall t\in[0,T]$, our experimental results show that there are no extra
benefits in normalizing the piano rolls into $x_{\text{roll}}\in\\{-1,1\\}$.
DiffRoll is able to learn to output $\hat{x}_{0}\in[0,1]$ matching the
original piano roll values $\\{0,1\\}$. Therefore, we kept
$x_{\text{roll}}\in\\{0,1\\}$ throughout our experiments. We trained our
models for 2,500 epochs on this dataset.

During the evaluation, we reported only the note-wise transcription metric (F1
score) since previous work has shown that the frame-wise transcription metric
does not reflect perceptual accuracy well [34]. We binarized the transcription
$x_{0}$ and extracted note objects from it. Each note object
$\\{\rho,\tau_{\text{on}},\tau_{\text{off}}\\}$ contained three attributes:
pitch $\rho$, onset time $\tau_{\text{on}}$, and offset time
$\tau_{\text{off}}$. Following previous work [31, 34], we used a music
information retrieval library, mir_eval222https://github.com/craffel/mir_eval,
to calculate the note-wise transcription F1 score between the predicted note
objects and the ground truth note objects. The predicted pitch $\hat{\rho}$
needs to match the ground pitch $\rho$ and the difference between the
predicted and ground truth onset
$\lvert\hat{\tau}_{\text{on}}-\tau_{\text{on}}\lvert$ needs to be within 50ms
for the prediction to be considered correct.

Table 1: DiffRoll F1 scores with different $w$ values and dilation settings when $k=9$. | $w$
---|---
Dilation | $0$ | $0.1$ | $0.5$ | $1$
$[1,1,1,1,\dots,]$ | 74.9 | 75.9 | 78.1 | 77.7
$[1,2,1,2,\dots,]$ | 74.0 | 75.3 | 77.1 | 76.9
$[1,2,4,1,\dots,]$ | 73.3 | 74.0 | 76.2 | 75.8
$[1,2,4,8,\dots,]$ | 72.2 | 74.0 | 75.9 | 76.1

###  4.1 DiffRoll versus Discriminative AMT

In this experiment, we trained a traditional discriminative AMT model as the
baseline. To ensure a fair comparison, we kept the model architecture exactly
the same as that of the DiffRoll model, but used a zero tensor as the $x_{t}$,
and a constant $t=1$ to directly predict the posteriorgram $\hat{x}_{0}$.
After training, the discriminative model learned to ignore the $x_{t}$ and $t$
as they contained useless information for the task. This discriminative
baseline (convolutional kernel size $k=3$) had an F1 score of $59.1$ as shown
in both Figs. 3 and 4. Fig. 3 also shows DiffRoll’s performance for different
values of dropout rate $p$. When $p=0$, the model was equivalent to standard
diffusion without CFG. The transcription F1 score decreased as $p$ increased.
This was expected since DiffRoll has a lower probability of obtaining paired
data when $p$ is large. Although the best F1 score ($67.1$) was achieved with
$p=0$, the F1 score rapidly decreased well below $20$ once a generative
component was introduced ($w>0$ on line 1 of Algorithm 1) as shown in Fig. 4.
The results of the ablation study in Fig. 4 also reveal that when $p>0$ and
$w=0.5$, DiffRoll reached its peak performance of $70.0$. This experiment was
done using kernel size $k=3$.

Since piano rolls have long term structures, we also investigated different
receptive field sizes by changing the kernel size ($k$) for DiffRoll. As we
can see from Fig. 4, the performance increased (from $70$ to $78.1$) as we
increased $k$ from $3$ to $9$, which shows that a larger receptive field is
beneficial for improving transcription accuracy. The degree of improvement,
however, decreased as $k$ increased, so we stopped at $k=9$ and used this for
the experiments described below.

In DiffWave [18], dilation is used to create a suitable receptive field to
capture the long-term information present in the waveform. We experimented
with different dilation settings (Table 1) for DiffRoll. Dilation factor
$[1,2,1,2,\dots]$ indicates that the 1st and 2nd layers have dilation factors
of $1$ and $2$, respectively; the same pattern holds for the 3rd and 4th
layers. However, we observed no improvement in the F1 score when changing the
dilation. On the contrary, the larger the dilation factor, the worse the F1
score. Therefore, we believe that a kernel size of $9$ without dilation has a
large enough receptive field for the AMT task.

This experiment shows that modeling AMT as a conditioned generative task is
possible and beneficial. A full animation for the generative trajectory of the
transcription process is available in the supplementary
material11footnotemark: 1. A DiffRoll model with kernel size $k=9$, dropout
rate $p=0.1$, sampling weight $w=0.5$, and without any dilation had the best
performing model and outperformed the discriminative counterpart ($59.1$ F1
scores) by 19 percentage points (ppt.) and even the U-net model [35] ($73.3$
F1 scores) by 4.8 ppt.. The model could not outperform current SOTA AMT models
[31, 9], likely because we have not yet incorporated onset and offset
information during training. We will enhance DiffRoll performance by
incorporating onset and offset information as our future work.

Fig. 3: Result of DiffRoll ablation study with different values of dropout
rate $p$. Baseline was discriminative version of DiffRoll, which directly
predicts $x_{0}$.

Fig. 4: Results of CFG ablation study with different values of sampling weight
$w$. Discriminative AMT model and U-net model [35] were the baseline models.

###  4.2 Unsupervised Pretraining

DiffRoll also supports unsupervised pretraining. Hence, it is possible to
pretrain DiffRoll with piano rolls only (without paired audio). When audio
data is not available, we can simply set $c_{\text{mel}}=-1$ to train only the
unconditional model. To demonstrate this ability, we use MAESTRO [31] as the
unpaired piano roll dataset (967 unpaired piano rolls), and MAPS [36] as the
paired dataset (139 audio clips paired with piano rolls). We first pretrained
DiffRoll for 2,500 epochs using only the piano rolls in the training split of
the MAESTRO dataset. We then continued training DiffRoll with the training
split of the MAPS dataset for 10,000 epochs using paired audio clips and piano
rolls. We removed overlapping samples between the training split and the test
split from the MAPS dataset following previous work [37]. There are two ways
to train on the MAPS dataset. One is to set dropout $p=0.1$ such that the 139
audio and piano roll data paired in the MAPS training split are used to train
both the conditional and unconditional parts of DiffRoll. The other is to load
both the MAPS and MAESTRO datasets, and set $p=0$ for the former and set $p=1$
for the latter. We denote this dropout scheme as $p=0+1$ in Table 2.

Table 2: The performance of DiffRoll when pretrained (denoted as ‘pre-DiffRoll’) using unpaired piano rolls (extracted from MAESTRO). ‘Discriminative’ is the discriminative counterpart of DiffRoll, and ‘DiffRoll’ was trained without unpaired pretraining. ‘ReconVAT’ is an existing semi-supervised AMT model. | Note-wise F1 score
---|---
Model | $w=0$ | $w=0.5$
Discriminative | 50.6
DiffRoll ($p=0.1$) | 55.9 | 57.7
pre-DiffRoll ($p=0.1$) | 59.0 | 62.6
pre-DiffRoll ($p=0+1$) | 60.5 | 63.7
ReconVAT [14] | 64.0

As shown in Table 2, the pretrained DiffRoll (pre-DiffRoll) achieved an F1
score $4.9$ points higher than the DiffRoll trained on the MAPS dataset from
scratch when $p=0.1$ and $w=0.5$. When training scheme $p=0+1$ was used, the
F1 score further improved by 1.1 to 63.7. Although DiffRoll was unable to
outperform the SOTA semi-supervised AMT model ReconVAT [14], its score was
competitive (63.7 v.s. 64.0). We believe that the model architecture of
DiffRoll can be further improved and leave such improvement for future work.
The same as in Section 4.1, performance with $w=0.5$ was always better than
with $w=0$ during sampling.

###  4.3 Generation

Unlike traditional discriminative AMT models, the proposed DiffRoll model is
generative. In other words, it is capable of generating new piano rolls which
can be considered as a form of music generation. To generate new music during
sampling, we can reuse the same model we trained as described above, and set
$w=-1$ during sampling such that the conditional term on line 1 of Algorithm 1
is cancelled out, leaving only the unconditional term. A full animation of the
generation trajectory of $x_{t}$ via line 1 of Algorithm 1 during generation
is available in the supplementary material11footnotemark: 1. In each iteration
of line 1 of Algorithm 1, DiffRoll makes $x_{t}$ less and less noisy. This can
be seen from the animation: as $t$ approaches 0, the value range of $x_{t}$
narrows to around $[0,1]$ even though the output of DiffRoll is unbounded.
Although the generated piano roll is far from perfect, we can still observe
patterns of chords. This shows that our generative model successfully captures
the data distribution for piano rolls $p(x_{\text{roll}})$.

###  4.4 Inpainting

Fig. 5: Piano roll inpainting when part of $c_{\text{mel}}$ is masked by -1
(indicated by red rectangles). DiffRoll generates notes in masked regions and
transcribes notes in unmasked regions. More examples are available in the
demo.

In the previous subsections, we described using DiffRoll for either
transcription or generation. In this subsection, we briefly explore the
inpainting capability of DiffRoll in which the input spectrograms
$c_{\text{mel}}$ are partially masked with $-1$ during sampling. We use the
same sampling scheme as on line 1 of Algorithm 1. As shown in Fig. 5, DiffRoll
performs transcription when $c_{\text{mel}}>0$, and generation when
$c_{\text{mel}}=-1$. More inpainting examples are available in the
supplementary material11footnotemark: 1. Since inpainting is beyond the scope
of this paper, in-depth exploration is left for future research.

##  5 Conclusion

We proposed a novel generative automatic music transcription (AMT) model,
DiffRoll. This model is capable of transcribing, generating, and even
inpainting music. The generative property of DiffRoll enables it to be
pretrained on unpaired piano rolls. Experiments showed that this unsupervised
pretraining improves model performance when the amount of labelled data is
limited. DiffRoll outperformed its discriminative counterpart by $7.1$ in
terms of the F1 score on the MAPS dataset. When unsupervised pretraining was
used (the MAESTRO dataset), the F1 score was further improved by $6$. Although
DiffRoll did not outperform state-of-the-art AMT models, the results presented
provide a successful proof of concept for how diffusion can be used for AMT.
There are number of ways in which DiffRoll can be improved. Future work will
include exploring the possibility of predicting onset and offset locations
using diffusion, which has been shown to improve AMT accuracy. We will also
investigate replacing Gaussian diffusion with discrete diffusion for binary
piano rolls. This novel formulation of AMT should attract more research into
the generative approach to AMT. The source code and generation examples are
available in the supplementary material11footnotemark: 1.

## References

  * [1] Juhan Nam, Jiquan Ngiam, Honglak Lee, Malcolm Slaney, et al.,  “A classification-based polyphonic piano transcription approach using learned feature representations.,”  in ISMIR, 2011, pp. 175–180.
  * [2] Ali Taylan Cemgil,  Bayesian music transcription,  Ph.D. thesis, Radbound University Nijmegen Netherlands, 2004.
  * [3] Taylor Berg-Kirkpatrick, Jacob Andreas, and Dan Klein,  “Unsupervised transcription of piano music,”  in Advances in neural information processing systems, 2014, pp. 1538–1546.
  * [4] Adam Roberts, Jesse Engel, Colin Raffel, Ian Simon, and Curtis Hawthorne,  “Musicvae: Creating a palette for musical scores with machine learning, march 2018,” 2018.
  * [5] Jesse Engel, Kumar Krishna Agrawal, Shuo Chen, Ishaan Gulrajani, Chris Donahue, and Adam Roberts,  “GANSynth: Adversarial neural audio synthesis,”  in International Conference on Learning Representations, 2019.
  * [6] Hao-Wen Dong, Wen-Yi Hsiao, Li-Chia Yang, and Yi-Hsuan Yang,  “Musegan: Multi-track sequential generative adversarial networks for symbolic music generation and accompaniment,”  in Proceedings of the AAAI Conference on Artificial Intelligence, 2018, vol. 32.
  * [7] Curtis Hawthorne, Erich Elsen, Jialin Song, Adam Roberts, Ian Simon, Colin Raffel, Jesse Engel, Sageev Oore, and Douglas Eck,  “Onsets and frames: Dual-objective piano transcription,”  in ISMIR, 2017.
  * [8] Josh Gardner, Ian Simon, Ethan Manilow, Curtis Hawthorne, and Jesse Engel,  “Mt3: Multi-task multitrack music transcription,”  arXiv preprint arXiv:2111.03017, 2021.
  * [9] Qiuqiang Kong, Bochen Li, Xuchen Song, Yuan Wan, and Yuxuan Wang,  “High-resolution piano transcription with pedals by regressing onset and offset times,”  IEEE/ACM Trans. Audio, Speech and Lang. Proc., vol. 29, pp. 3707–3717, jan 2021.
  * [10] Kin Wai Cheuk, Keunwoo Choi, Qiuqiang Kong, Bochen Li, Minz Won, Amy Hung, Ju-Chiang Wang, and Dorien Herremans,  “Jointist: Joint learning for multi-instrument transcription and its applications,”  arXiv preprint arXiv:2206.10805, 2022.
  * [11] Weixing Wei, Peilin Li, Yi Yu, and Wei Li,  “Hppnet: Modeling the harmonic structure and pitch invariance in piano transcription,”  ISMIR, 2022.
  * [12] Ben Maman and Amit H Bermano,  “Unaligned supervision for automatic music transcription in the wild,”  in Proceedings of the 39th International Conference on Machine Learning, Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato, Eds. 17–23 Jul 2022, vol. 162 of Proceedings of Machine Learning Research, pp. 14918–14934, PMLR.
  * [13] Haoran Wu, Axel Marmoret, and Jérémy E Cohen,  “Semi-supervised convolutive nmf for automatic music transcription,”  arXiv preprint arXiv:2202.04989, 2022.
  * [14] Kin Wai Cheuk, Dorien Herremans, and Li Su,  “Reconvat: A semi-supervised automatic music transcription framework for low-resource real-world data,”  in Proceedings of the 29th ACM International Conference on Multimedia, 2021, pp. 3918–3926.
  * [15] Keunwoo Choi and Kyunghyun Cho,  “Deep unsupervised drum transcription,”  in ISMIR 4-8, 2019, Arthur Flexer, Geoffroy Peeters, Julián Urbano, and Anja Volk, Eds., 2019, pp. 183–191.
  * [16] Paris Smaragdis and Judith C Brown,  “Non-negative matrix factorization for polyphonic music transcription,”  in 2003 IEEE Workshop on Applications of Signal Processing to Audio and Acoustics (IEEE Cat. No. 03TH8684). IEEE, 2003, pp. 177–180.
  * [17] Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli,  “Deep unsupervised learning using nonequilibrium thermodynamics,”  in Proceedings of the 32nd International Conference on Machine Learning, Francis Bach and David Blei, Eds., Lille, France, 07–09 Jul 2015, vol. 37 of Proceedings of Machine Learning Research, pp. 2256–2265, PMLR.
  * [18] Zhifeng Kong, Wei Ping, Jiaji Huang, Kexin Zhao, and Bryan Catanzaro,  “Diffwave: A versatile diffusion model for audio synthesis,”  International Conference on Learn- ing Representations, 2020.
  * [19] Nanxin Chen, Yu Zhang, Heiga Zen, Ron J Weiss, Mohammad Norouzi, and William Chan,  “Wavegrad: Estimating gradients for waveform generation,”  International Conference on Learning Representations, 2020.
  * [20] Jonathan Ho, Ajay Jain, and Pieter Abbeel,  “Denoising diffusion probabilistic models,”  Advances in Neural Information Processing Systems, vol. 33, pp. 6840–6851, 2020.
  * [21] Chieh-Hsin Lai, Yuhta Takida, Naoki Murata, Toshimitsu Uesaka, Yuki Mitsufuji, and Stefano Ermon,  “Regularizing score-based models with score fokker-planck equations,”  arXiv, 2022.
  * [22] Jiaming Song, Chenlin Meng, and Stefano Ermon,  “Denoising diffusion implicit models,”  in International Conference on Learning Representations, 2021.
  * [23] Ting Chen, Ruixiang Zhang, and Geoffrey Hinton,  “Analog bits: Generating discrete data using diffusion models with self-conditioning,”  arXiv preprint arXiv:2208.04202, 2022.
  * [24] Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever,  “Zero-shot text-to-image generation,”  in International Conference on Machine Learning. PMLR, 2021, pp. 8821–8831.
  * [25] Gautam Mittal, Jesse Engel, Curtis Hawthorne, and Ian Simon,  “Symbolic music generation with diffusion models,”  2021\.
  * [26] Curtis Hawthorne, Ian Simon, Adam Roberts, Neil Zeghidour, Josh Gardner, Ethan Manilow, and Jesse Engel,  “Multi-instrument music synthesis with spectrogram diffusion,”  ISMIR, 2022.
  * [27] Yujia Yan, Frank Cwitkowitz, and Zhiyao Duan,  “Skipping the frame-level: Event-based piano transcription with neural semi-CRFs,”  in Advances in Neural Information Processing Systems, A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan, Eds., 2021.
  * [28] Jonathan Ho and Tim Salimans,  “Classifier-free diffusion guidance,”  in NeurIPS 2021 Workshop on Deep Generative Models and Downstream Applications, 2021.
  * [29] Bahjat Kawar, Michael Elad, Stefano Ermon, and Jiaming Song,  “Denoising diffusion restoration models,”  arXiv preprint arXiv:2201.11793, 2022.
  * [30] Tim Salimans and Jonathan Ho,  “Progressive distillation for fast sampling of diffusion models,”  in International Conference on Learning Representations, 2022.
  * [31] Curtis Hawthorne, Andriy Stasyuk, Adam Roberts, Ian Simon, Cheng-Zhi Anna Huang, Sander Dieleman, Erich Elsen, Jesse Engel, and Douglas Eck,  “Enabling factorized piano music modeling and generation with the MAESTRO dataset,”  in International Conference on Learning Representations, 2019.
  * [32] Kin Wai Cheuk, Kat Agres, and Dorien Herremans,  “The impact of audio input representations on neural network based music transcription,”  International Joint Conference on Neural Networks, 2020.
  * [33] Kin Wai Cheuk, Hans Anderson, Kat Agres, and Dorien Herremans,  “nnaudio: An on-the-fly gpu audio to spectrogram conversion toolbox using 1d convolution neural networks,”  arXiv preprint arXiv:1912.12055, 2019.
  * [34] Kin Wai Cheuk, Yin-Jyun Luo, Emmanouil Benetos, and Dorien Herremans,  “Revisiting the onsets and frames model with additive attention,”  in 2021 International Joint Conference on Neural Networks (IJCNN), 2021, pp. 1–8.
  * [35] Kin Wai Cheuk, Yin-Jvun Luo, Emmanouil Benetos, and Dorien Herremans,  “The effect of spectrogram reconstruction on automatic music transcription: An alternative approach to improve transcription accuracy,”  in 2020 25th International Conference on Pattern Recognition (ICPR), 2021, pp. 9091–9098.
  * [36] Valentin Emiya, Nancy Bertin, Bertrand David, and Roland Badeau,  “Maps-a piano database for multipitch estimation and automatic transcription of music,”  2010\.
  * [37] Siddharth Sigtia, Emmanouil Benetos, and Simon Dixon,  “An end-to-end neural network for polyphonic piano music transcription,”  IEEE/ACM Transactions on Audio, Speech, and Language Processing, vol. 24, pp. 927–939, 2015.