# Learning Disentangled Representations for Timber and Pitch
in Music Audio

###### Abstract

Timbre and pitch are the two main perceptual properties of musical sounds.
Depending on the target applications, we sometimes prefer to focus on one of
them, while reducing the effect of the other. Researchers have managed to
hand-craft such timbre-invariant or pitch-invariant features using domain
knowledge and signal processing techniques, but it remains difficult to
disentangle them in the resulting feature representations. Drawing upon state-
of-the-art techniques in representation learning, we propose in this paper two
deep convolutional neural network models for learning disentangled
representation of musical timbre and pitch. Both models use encoders/decoders
and adversarial training to learn music representations, but the second model
additionally uses skip connections to deal with the pitch information. As
music is an art of time, the two models are supervised by frame-level
instrument and pitch labels using a new dataset collected from MuseScore. We
compare the result of the two disentangling models with a new evaluation
protocol called “timbre crossover,” which leads to interesting applications in
audio-domain music editing. Via various objective evaluations, we show that
the second model can better change the instrumentation of a multi-instrument
music piece without much affecting the pitch structure. By disentangling
timbre and pitch, we envision that the model can contribute to generating more
realistic music audio as well.

## Introduction

Timbre and pitch are the two main perceptual properties of musical sounds. For
a musical note, they refer to the perception of sound quality and frequency of
the note, respectively. For a musical phrase, the perception of pitch informs
us the notes and their ordering in the phrase (e.g., Do-Do-So-So-La-La-So),
whereas the perception of timbre informs us the instruments that play each
note. Timbre and pitch are interdependent, but they can also be _disentangled_
—we can use different instruments to play the same note sequence, and the same
instrument to play different note sequences. While listening to music, human
beings can selectively pay their attention to either the timbre or pitch
aspect of music. For AI applications in music, we hope machines can do the
same.

Preserving the characteristics of one property while reducing those of the
other has been studied in the literature. When the goal is to build a
computational model that recognizes the note sequences (e.g., for tasks such
as query by humming [1] and cover song identification [2]), we need a feature
representation of music that is not sensitive to changes in timbre and
instrumentation. In contrast, in building an instrument or singer classifier,
we may want to focus more on timbre rather than pitch.

The pursuit of such timbre- or pitch-invariant features has been mostly
approached with domain knowledge and signal processing techniques [3, 4].
However, the effectiveness of these features is usually evaluated by their
performance in the downstream recognition or classification problems, i.e.,
from an _analysis_ point of view. For example, a timbre-invariant feature is
supposed to work better than a non timbre-invariant one for harmonic analysis.
It remains unclear how timbre and pitch are actually disentangled in the
feature representations.

With the recent success in learning disentangled representations for images
using deep autoencoders [5, 6], we see new opportunities to tackle timbre and
pitch disentanglement for music from the _synthesis_ point of view. Taking a
musical audio clip as input, we aim to build a model that processes the timbre
and pitch information in two different streams to arrive at the intermediate
timbre and pitch representations that are disentangled, and that can be
combined to _reconstruct_ the original input. If timbre and pitch are
successfully disentangled, we expect that we can change the instrumentation of
a music clip with this model by manipulating the timbre representation only,
while fixing the pitch representation.111If we think about timbre as _tone
colors_ , this is like _coloring_ the music clip in different ways. When there
are multiple instruments, the model needs to decide which instrument plays
which notes.

To our best knowledge, this work represents the first attempt to disentangle
timber and pitch in music audio with deep encoder/decoder architectures. Our
approach has a few advantages over the conventional analysis approach. First,
we adopt a deep neural network to learn features in a data-driven way, instead
of hand-crafting the features. Second, we can use the learned features to
generate an audio signal, which provides a direct way to evaluate the
effectiveness of disentanglement. Third, accordingly, it enables new
applications in audio-domain _music editing_ —to manipulate the timbre and
pitch content of an existing music clip without re-recording it. Lastly, given
a note sequence generated by human or an AI composer [7, 8, 9], our model can
help decide how to color (i.e., add timbre to) it.

Due to the differences in images and music, we cannot directly apply existing
methods to music. Instead, we propose two ideas that consider the specific
characteristics of music in designing the encoders and decoders.

The first idea is _temporal supervision_. In computer vision, people use
image-level attributes as the supervisory signal to learn disentangled
features. For example, to disentangle face identities and poses [10], or to
disentangle different attributes of faces [6]. For music, we cannot
analogously use clip-level labels, since timbre and pitch are associated with
each individual musical notes, and a music clip is composed of multiple notes.
Therefore, we propose to use the multi-track _pianorolls_ [11] as the learning
target of our encoders/decoders to provide detailed temporal supervision at
the frame level. That is, instead of aiming to reconstruct the input audio, we
aim to generate as the output of the network the pianoroll associated with the
input audio. A pianoroll is a symbolic representation of music that specifies
the timbre and pitch per note. It can be derived from a MIDI file [11]. We
manage to compile a new dataset with 350,000 pairs of audio clips and time-
aligned MIDIs. The temporal supervision provided by the dataset greatly
facilitates timbre and pitch disentanglement, for otherwise the model has to
learn from the audio signals in an unsupervised way. We intend to make public
the dataset and our code for reproducibility.

The second idea is to use different operations to deal with timbre and pitch.
Specifically, we propose to use _convolutions_ in the encoder to learn the
abstract timbre representation in the latent space, and use _symmetric skip
connections_ to allow the pitch information to flow directly from the encoder
to the decoder. This design is based on the intuition that, when we use a
time-frequency representation such as the spectrogram as the input, the timbre
aspect actually affects _how_ the energy of the harmonic partials distributes
along the frequency axis and develops over time [3], whereas the pitch aspect
determines only _what_ the fundamental frequency and duration of the notes
are. To make an analogy between music and images, pitch acts like the boundary
of visual objects, whereas timbre acts like the texture. We therefore aim to
learn an embedding for the texture, while process the discrete pitch
information with skip connections, which have been found effective in image
segmentation [12].

Specifically, we propose two models for disentangling timbre and pitch. The
first model (DuoAE) uses separate encoders for timbre and pitch, whereas the
second model (UnetAE) adopts the aforementioned second idea and use skip
connections to deal with pitch. Both models employ adversarial training [13].
Figure 1 illustrates the two models. We will present the model details later.

As secondary contributions, for music editing and generation purposes, we
additionally train another encoder/decoder sub-network to convert the
pianorolls into audio signals. We find that the use of _binary neurons_ [14]
is critical for this new sub-network. In addition, we propose a new evaluation
protocol called “timbre crossover” to evaluate how well we can create a new
music by exchanging the instrumentation of two existing pieces, using the
timbre representation of one piece in reconstructing the pianoroll or audio of
the other piece. We report systematic objective evaluations of the result of
timbre crossover.

(a) DuoAE: Use separate encoders/decoders for timbre and pitch

(b) UnetAE: Use skip connections to process pitch Fig. 1: The two proposed
encoder/decoder architectures for disentangling timbre and pitch for music
audio. The dashed lines indicate the adversarial training parts. (Notations:
CQT—a time-frequency representation of audio; roll–multi-track pianoroll;
E—encoder; D—decoder; Z—latent code; t—timbre; p—pitch; skip—skip
connections).

## Background

A deep autoencoder (or AE for short) is a network architecture that uses a
stack of encoding layers (known collectively as an _encoder_) to get a low-
dimensional representation of data, which originally resides in a high-
dimensional space [15]. The resulting representation is also known as the
_latent code_. The network is trained such that we can recover the input from
the latent code, by passing the latent code through another stack of decoding
layers (known collectively as a _decoder_). Compared to other representation
learning methods, the AE has the advantages that the training is unsupervised,
and that the obtained representation can be mapped back to the data space.

With the original AE, different properties of data might be entangled in the
latent code, meaning that each entry of the latent code is related to multiple
data properties. For instance, we can train an AE with face images to obtain a
latent vector $\mathbf{z}\in\mathbb{R}^{k}$ for an image, where $k$ denotes
the dimensionality of the latent code. If we add some noise $\epsilon$ to only
a random entry of $\mathbf{z}$ and then decode it, likely many properties of
the decoded face would be different from the original face.

With some labeled data, we can train an AE in a way that different parts of
the latent code correspond to different data properties. For example, given
face images labeled with identities and poses, we can divide $\mathbf{z}$ into
two parts $\mathbf{z}_{1}$ and $\mathbf{z}_{2}$ (i.e.,
$\mathbf{z}=[\mathbf{z}_{1}^{T},\mathbf{z}_{2}^{T}]^{T}$), and use them as the
input to two separate stacks of fully-connected layers to train an identity
classifier $C_{1}$ and a pose classifier $C_{2}$, respectively. We still
require the concatenated code (i.e., $\mathbf{z}$) to reconstruct the input.
With this AE, if we add changes to $\mathbf{z}_{2}$ but keep $\mathbf{z}_{1}$
the same, we may obtain an image with the same identity but a different pose.
In this way, we call $\mathbf{z}_{1}$ and $\mathbf{z}_{2}$ disentangled
representations of face identities and poses.

We can further improve the result of disentanglement by using _adversarial
training_. As usual, we aim to minimize the classification error of $C_{1}$
and $C_{2}$ when their input is $\mathbf{z}_{1}$ and $\mathbf{z}_{2}$,
respectively. However, we additionally use $\mathbf{z}_{2}$ and
$\mathbf{z}_{1}$ as the input to $C_{1}$ and $C_{2}$ respectively, and aim to
_maximize_ the classification error of the two classifiers under such a
scenario. In this way, we promote identity information in $\mathbf{z}_{1}$
while _dispel_ any information related to pose, and similarly for
$\mathbf{z}_{2}$. This idea was proposed by [6].

There are many other ways to achieve disentanglement, e.g., using generative
adversarial networks (GAN) [10, 16, 17], cross-covariance penalties [18], and
latent space arithmetic operations [19]. Some are unsupervised methods. While
the majority of work has been on images, feature disentanglement has also been
studied for video clips [20], speech clips [19], 3D data [21], and MIDIs [22,
23].

Our work distinguishes itself from the existing works mainly in the following
two aspects. First, we use temporal supervision to learn disentangled
representations, while existing work usually use image- or clip-level labels,
such as face identity, speaker identity [19], and genre [23]. Second, to our
best knowledge, little has been done for disentangled representation learning
in music audio. Although MIDI is also a type of music, an disentangling model
for encoding MIDIs cannot be applied to musical audio signals.222For example,
the sounds of different partials from different instruments are mixed in
audio, but this does not happen in MIDIs.333Disentangling musical properties
in MIDIs is a challenging task as well. [22] aimed to disentangle melody and
instrumentation using a variational autoencoder (VAE), but they found that
manipulating the instrumentation code (while fixing the melody code) would
still change the melody.

## Proposed Models for Disentanglement

Figure 1 shows the architecture of the proposed models for disentangling
timbre and pitch. We present the details below.

### Input/Output Data Representation

#### Input

The input to our models is an audio waveform with arbitrary length. To
facilitate timbre and pitch analysis, we firstly convert the waveform into a
time-frequency representation that shows the energy distribution across
different frequency bins for each short-time frame. Instead of using the
short-time Fourier transform (STFT), we use the constant-Q transform (CQT)
here, for the latter adopts a logarithmic frequency scale that better aligns
with our perception of pitch [24]. CQT also provides better frequency
resolution in the low-frequency part, which helps detect the fundamental
frequencies.

As will be shown later, our encoders and decoders are designed to be _fully-
convolutional_ [25], so that our models can deal with input of any length in
testing time. However, for the convenience of training the models with mini-
batches, in the training stage we divide the waveforms in our training set
into 10-second chunks (without overlaps) and use these chunks as the model
input, leading to a matrix $\mathbf{X}_{cqt}\in\mathcal{R}^{F\times T}$ of
fixed size for each input. In our implementation, we compute CQT with the
librosa library [26], with 16,000 Hz sampling rate and 512-sample window size,
again with no overlaps. We use a frequency scale of 88 bins, with 12 bins per
octave to represent each note. Hence, $F=88$ (bins) and $T=312$ (frames).

(a) Pianoroll

(b) Instrument roll

(c) Pitch roll

Fig. 2: Different symbolic representations of music.

#### Output

To provide temporal supervision, we use the pianorolls as the target output of
our models.444Strictly speaking, such a model is no longer an autoencoder,
since the input and target output are different. We abuse the terminology for
the short names of our models, e.g., Duo‘AE.’ As depicted in Figure 2, a
pianoroll is a binary-valued tensor that records the presence of notes (88
notes here) across time for each track (i.e., instrument) [11]. When we
consider $M$ instruments, the target model output would be
$\mathbf{X}_{roll}\in\\{0,1\\}^{F\times T\times M}$. $\mathbf{X}_{roll}$ and
$\mathbf{X}_{cqt}$ are temporally aligned, since we use MIDIs that are time-
aligned with the audio clips to derive the pianorolls, as will be discussed in
Section Dataset.

As shown in Figure 1, besides asking our models to generate
$\mathbf{X}_{roll}$ from the latent code of $\mathbf{X}_{cqt}$, we use the
_instrument roll_ $\mathbf{X}_{t}\in\\{0,1\\}^{M\times T}$ and _pitch roll_
$\mathbf{X}_{p}\in\\{0,1\\}^{F\times T}$ as supervisory signals to disentangle
timbre and pitch. As depicted in Figure 2, these two rolls can be obtained
respectively by marginalizing a certain dimension of the pianoroll.

### The DuoAE Model

The architecture of DuoAE is illustrated in Figure 1(a). The designed is based
on [6], but we use temporal supervision here and adapt the model to encode
music.

Specifically, we train two encoders $E_{t}$ and $E_{p}$ to respectively
convert $\mathbf{X}_{cqt}$ into the _timbre code_
$\mathbf{Z}_{t}=E_{t}(\mathbf{X}_{cqt})\in\mathcal{R}^{\kappa\times\tau}$ and
_pitch code_
$\mathbf{Z}_{p}=E_{p}(\mathbf{X}_{cqt})\in\mathcal{R}^{\kappa\times\tau}$. We
note that, unlike in the case of image representation learning, here the
latent codes are _matrices_ , and we require that the second dimensions (i.e.,
$\tau$) represent time. This way, each column of $\mathbf{Z}_{t}$ and
$\mathbf{Z}_{p}$ is a $\kappa$-dimensional representation of a temporal
segment of the input. For abstraction, we require $\kappa\tau<FT$.

DuoAE also contains three decoders $D_{roll}$, $D_{t}$ and $D_{p}$. The
encoders and decoders are trained such that we can use
$D_{roll}([\mathbf{Z}_{t}^{T},\mathbf{Z}_{p}^{T}]^{T})$ to predict
$\mathbf{X}_{roll}$, $D_{t}(\mathbf{Z}_{t})$ to predict $\mathbf{X}_{t}$, and
$D_{p}(\mathbf{Z}_{p})$ to predict $\mathbf{X}_{p}$. The prediction error is
measured by the _cross entropy_ between the ground truth and the predicted
one. For example, for the timbre classifier $D_{t}$, it is:

$L_{t}=-\textstyle{\sum}~{}[\mathbf{X}_{t}\cdot\ln\sigma(\widehat{\mathbf{X}}_{t})+(1-\mathbf{X}_{t})\cdot\ln(1-\sigma(\widehat{\mathbf{X}}_{t}))]\,,$
(1)

where $\widehat{\mathbf{X}}_{t}=D_{t}(\mathbf{Z}_{t})$, ‘$\cdot$’ denotes the
element-wise product, and $\sigma$ is the sigmoid function that scales its
input to $[0,1]$. We can similarly define $L_{roll}$ and $L_{p}$.

In each training epoch, we optimize _both_ the encoders and decoders by
minimizing $L_{roll}$, $L_{t}$ and $L_{p}$ for the given training batch. We
refer to the way we train the model as using the temporal supervision, since
to minimize the lost terms $L_{roll}$, $L_{t}$ and $L_{p}$, we have to make
accurate prediction for each of the $T$ time frames.

When the adversarial training strategy is employed (i.e., those marked by
dashed lines in Figure 1(a)), we additionally consider the following two lost
terms:

$L_{t}^{n}=-\textstyle{\sum}~{}[\mathbf{0}_{t}\cdot\ln\sigma(\widehat{\mathbf{X}}_{t}^{n})+(1-\mathbf{0}_{t})\cdot\ln(1-\sigma(\widehat{\mathbf{X}}_{t}^{n}))]\,,$
(2)
$L_{p}^{n}=-\textstyle{\sum}~{}[\mathbf{0}_{p}\cdot\ln\sigma(\widehat{\mathbf{X}}_{p}^{n})+(1-\mathbf{0}_{p})\cdot\ln(1-\sigma(\widehat{\mathbf{X}}_{p}^{n}))]\,,$
(3)

where $\widehat{\mathbf{X}}_{t}^{n}=D_{t}(\mathbf{Z}_{p})$,
$\widehat{\mathbf{X}}_{p}^{n}=D_{p}(\mathbf{Z}_{t})$, meaning that we feed the
‘wrong’ input (purposefully) to $D_{t}$ and $D_{p}$. Moreover,
$\mathbf{0}_{t}=\mathbf{0}_{M,T}$ and $\mathbf{0}_{p}=\mathbf{0}_{F,T}$ are
two matrices of _all zeros_. That is to say, when we use the wrong input, we
expect $D_{t}$ and $D_{p}$ can output nothing (i.e., all zeros), since, e.g.,
$\mathbf{Z}_{p}$ is supposed not to contain any timbre-related information.

Please note that, in adversarial training, we use $L_{t}^{n}$ and $L_{p}^{n}$
to update the _encoders only_. This is to preserve the function of the
decoders in making accurate predictions.

### The UnetAE Model

The architecture of UnetAE is depicted in Figure 1(b).555We explain the name
‘Unet’ below. When the design of the encoder (E) and decoder (D) is
_symmetric_ , meaning that they have the same number of layers and that they
use the same kernel sizes and stride sizes in the corresponding layers, we can
add skip connections between the corresponding layers of E and D, by
concatenating the output of the $i$-th layer of E to the input of the $i$-th
last layer of D in the channel-wise direction. In this way, lower-layer
information of E (closer to the input) can be directly passed to the higher-
layer of D (closer to the output), making it easier to train deeper AEs.
Because the resulting architecture has a U-shape, people refer to it as a
U-net [12]. In UnetAE, we learn only one encoder $E_{cqt}$ to get a single
latent representation $\mathbf{Z}_{t}$ of the input $\mathbf{X}_{cqt}$. We add
skip connections between $E_{cqt}$ and $D_{roll}$ and learn $E_{cqt}$ and
$D_{roll}$ by minimizing $L_{roll}$, the cross entropy between
$D_{roll}(\mathbf{Z}_{t})$ and the pianoroll $\mathbf{X}_{roll}$. Moreover, we
promote timbre information in $\mathbf{Z}_{t}$ by refining $E_{cqt}$ and
learning a classifier $D_{t}$ by minimizing $L_{t}$ (see Eq. (1)). When the
adversarial training strategy is adopted, we use the classifier $D_{p}$ pre-
trained from DuoAE to further dispel pitch information from $\mathbf{Z}_{t}$,
by updating $E_{cqt}$ (but fixing $D_{p}$) to minimize $L_{p}^{n}$ (see Eq.
(3)).

In summary, we use $L_{roll}$, $L_{t}$, and optionally $L_{p}^{n}$, to train
the encoder $E_{cqt}$; use $L_{roll}$ to train the decoder $D_{roll}$; and use
$L_{t}$ to train the timbre classifier $D_{t}$. We discard the lost term
$L_{p}^{n}$ when we do not use adversarial training.

The two key design principals of UnetAE are as follows. First, since
$\mathbf{Z}_{t}$ is supposed not to have any pitch information, the only way
to obtain the pitch information needed to predict $\mathbf{X}_{roll}$ is from
the skip connections. Second, there are reasons to believe that the skip
connections can pass along the pitch information, because there is nice one-
to-one time-frequency correspondence between $\mathbf{X}_{cqt}$ and each
frontal slice of $\mathbf{X}_{roll}$,666That is, both $X_{cqt}(i,j)$ and
$X_{roll}(i,j,m)$ refer to the activity of the same musical note $i$ for the
same time frame $j$. and because in $\mathbf{X}_{cqt}$ pitch only affects the
lowest partial of a harmonic series created by a musical note, while timbre
affects all the partials. If we view pitch as the boundary outlining an object
(i.e., the harmonic series) and timbre as the texture of that object, it makes
sense to use a U-net structure, since U-net performs well in image
segmentation [27, 12].

Fig. 3: The architecture of the proposed model for music audio editing, using
UnetAE for disentangling timbre and pitch.

#### Discussion

The major difference between DuoAE and UnetAE is that there is no pitch code
$\mathbf{Z}_{p}$ in UnetAE. We argue below why this may be fine for music
editing and generation applications.

As discussed in the introduction, pitch determines _what_ (i.e., the notes) to
be played in a music piece, whereas timbre determines _how_ they would sound
like to a listener. While the pitch content of a music piece can be largely
specified on the musical score (i.e., symbolic notations of music), the timbre
content manifests itself in the audio sounds. Therefore, we can learn the
timbre code $\mathbf{Z}_{t}$ from an input audio representation such as the
waveform, STFT, or CQT. If we want to learn the pitch code $\mathbf{Z}_{p}$,
we can learn it from symbolic representations of music, as pursued in existing
work on symbolic-domain music generation [28, 22, 23, 29, 8]. Since the pitch
code can be learned from musical scores, we may focus on learning the timbre
code from music audio.

Moreover, for music audio editing, we are interested in manipulating the
instrumentation of a clip without much affecting its pitch content. For this
purpose, it may be sufficient to have the timbre code, since in UnetAE the
pitch content can flow directly from the encoder to the decoder. On the other
hand, to manipulate the pitch content without affecting the timbre, we can
change the pitch in the symbolic domain and then use the same timbre code for
decoding.

#### Implementation Details

Since the input and target output are both matrices (or tensors), we use
convolutional layers in all the encoders and decoders of DuoAE and UnetAE. To
accommodate input of variable length, we adopt a _fully-convolutional_ design
[25], meaning that we do not use pooling layers at all. In the encoders, we
achieve dimension reduction (i.e., reducing $F$ to $\kappa$ and reducing $T$
to $\tau$) by setting the stride sizes of the kernels larger than one. We use
$3\times 3$ kernels in every layers of the encoders. In the decoders, we use
transposed convolution for upsampling. The kernel size is also $3\times 3$.
Moreover, we use leaky ReLU as the activation function and add batch
normalization to all but the last layer of the decoders, where we use the
sigmoid function. Both DuoAE and UnetAE are trained using stochastic gradient
descend (SGD) with momentum 0.9. The initial learning rate is set to 0.01.

## Proposed Model for Music Editing

The model we propose for music editing is shown in Figure 3. We use an
additional encoder-decoder subnet $E_{roll}$ and $D_{stft}$ after UnetAE to
convert pianorolls to the audio domain.777Such a pianoroll-to-audio conversion
can be made with commercial synthesizers, but the resulting sounds tend to be
deadpan. Here we use the STFT spectrograms of the original input audio as the
target output. The model is trained in an end-to-end fashion by additionally
minimizing $L_{MSE}$, the MSE between the groundtruth STFT $\mathbf{X}_{stft}$
and the predicted one $\widehat{\mathbf{X}}_{stft}$. Lastly, we use the
Griffin-Lim algorithm [30] to estimate the phase of the spectorgram and
generate the audio.

Once the model is trained, timbre editing can be done by manipulating the
timbre code $\mathbf{Z}_{t}$ and then generate $\widehat{\mathbf{X}}_{stft}$.
And, pitch editing can be done by manipulating the intermediate pianoroll
$\widehat{\mathbf{X}}_{roll}$ and then generate $\widehat{\mathbf{X}}_{stft}$.

There are three critical designs to make it work. First, we concatenate
$\mathbf{Z}_{t}$ with $\mathbf{Z}_{roll}$, the output of $E_{roll}$. This is
important since the input to $E_{roll}$ is pianorolls, but the pianorolls do
not contain sufficient timbre information to generate realistic audio. Second,
we only use one skip connection between $E_{roll}$ and $D_{stft}$, because if
we use too many skip connections the gradients from the STFT will affect the
training of $D_{roll}$ and lead to noisy pianorolls
$\widehat{\mathbf{X}}_{roll}$. Third, to further prevent the gradients from
the STFT to affect the pianorolls, we use deterministic _binary neurons_ (BN)
[14] to binarize the output of $D_{roll}$ with a hard thresholding function.

#### Discussion

A few other models have been proposed to generate musical audio signals. Many
of them do not take auxiliary conditional signal to condition the generation
process. These include auto-regressive models such as the WaveNet model [31]
and GAN-based models such as the WaveGAN model [32]. Some recent works started
to explore the so-called _score-to-audio_ music generation [33, 34], where the
audio generation model is given a musical score and is asked to render the
score into sounds. While the model proposed by Hawthorne _et al._ [34] deals
with only piano music, both our music editing model and the PerformanceNet
model proposed by Wang and Yang [33] aim to generate music of more other
instruments.

However, our model is different from these two prior arts in that we are not
dealing with score-to-audio music generation but actually audio editing, which
converts an audio into another audio. We use the estimated pianoroll
$\widehat{\mathbf{X}}_{roll}$ in the midway of the generation process for
controllability. As Figure 3 shows, our model takes not only the pianoroll
$\widehat{\mathbf{X}}_{roll}$ but also the timbre code $\mathbf{Z}_{t}$ of the
original audio as inputs. We find such a problem setting interesting and would
try to further improve it in our future work, using for example WaveNet-based
decoder or GAN training for better audio quality.

Method | #Params | Transcription | Crossover (*$\rightarrow$piano) | Crossover (*$\rightarrow$guitar) | Crossover (pia$\rightarrow$str)
---|---|---|---|---|---
Acc | Pitch Acc | Timbre HI | Pitch Acc | Timbre HI | Pitch Acc | Timbre HI
Baseline model | 3,277k | 0.314 | — | — | — | — | — | —
Prior art [17] | 6,533k | — | 0.558 | — | 0.537 | — | — | —
Prior art [35] | 7,729k | — | 0.295 | — | 0.279 | — | — | —
DuoAE w/o adv | 9,438k | 0.395 | 0.595 | 0.990 | 0.664 | 0.983 | 0.743 | 0.998
DuoAE | 9,808k | 0.372 | 0.660 | 0.990 | 0.647 | 0.989 | 0.718 | 0.998
UnetAE w/o adv | 3,499k | 0.396 | 0.563 | 0.869 | 0.611 | 0.911 | 0.681 | 0.966
UnetAE | 3,868k | 0.431 | 0.691 | 0.893 | 0.727 | 0.962 | 0.748 | 0.992
Table 1: The performance of different models for transcription (i.e.,
pianoroll prediction) and timbre crossover, evaluated in terms of pitch
accuracy (Acc) and the timbre histogram intersection (HI) rate. The last two
columns are ‘piano to violin+cello’ conversion. We use ‘w/o adv’ to denote the
cases without adversarial training, and ‘#Params’ the total number of
parameters. The ‘baseline model’ (see Section Evaluation on Pianoroll
Prediction) shown in the first row does not use timbre and pitch classifiers
and adversarial training.

## Dataset

We build a new dataset with paired audio and MIDI files to train and evaluate
the proposed models. This is done by crawling the MuseScore web forum
(https://musescore.com/), obtaining around 350,000 unique MIDI files and the
corresponding MP3 files. Most MP3 files were synthesized from the MIDIs with
the MuseScore synthesizer by the uploaders.888Though synthesized audio may
sounds different than realistic audio, the problem can be solved by domain
adaptation [5]. We take it as a future task and will not further discussed in
this paper. Besides, we also tested the model on realistic music and found
that the model performs well for many of them. Hence, the audio and MIDIs are
already time-aligned. We further ensure temporal alignment by using the method
proposed by [36]. We then convert the time-aligned MIDIs to pianorolls with
the Pypianoroll package [11].

We consider the following nine instruments in this work (i.e., $M=9$): piano,
acoustic guitar, electrical guitar, trumpet, saxphone, bass, violin, cello and
flute. They are chosen based on their popularity in modern music and
MuseScore.999We exclude drums for they are non-pitched. And, we exclude the
singing voices, for they are not transcribed in MIDIs. The average length of
the music clips is $\sim$2 minutes. We randomly pick 585–1000 clips per
instrument for the training set, and 27–50 clips per instrument for the test
set. As a result, the training and test sets are class-balanced.

## Experiment

In what follows, we first evaluate the accuracy of our models in predicting
the pianorolls. We then evaluate the result of timbre and pitch
disentanglement by examining the learned embeddings and by the timbre
crossover evaluation method.

### Evaluation on Pianoroll Prediction

We first evaluate how well we can transcribe the pianorolls from audio. Since
$\widehat{\mathbf{X}}_{roll}$, the output of $D_{roll}$, is a real-valued
tensor in $[0,1]^{F\times T\times M}$, we further binarize it with a simple
threshold picking algorithm so that we can compare it with the groundtruth
pianoroll $\mathbf{X}_{roll}$, which is binary. We select the threshold (from
0.1, 0.15, $\dots$, 0.95, in total 20 candidates) by maximizing the accuracy
on a held-out validation set. The accuracy (denoted as ‘Acc’) is calculated by
comparing $\mathbf{X}_{roll}$ and $\widehat{\mathbf{X}}_{roll}$ per instrument
(by calculating the proportion of true positives among the $FT$ entires) and
then taking the average across the instruments. This way, we can measure the
accuracy for both instrument and pitch prediction, since falsely predicting
the instrument of a note would also reduce the number of true positives for
instruments.

In addition to DuoAE and UnetAE, we consider a baseline model that uses only
an encoder $E_{cqt}$ and a decoder $D_{roll}$ to get the latent code
$\mathbf{Z}_{t}$. That is, not using additional timbre and pitch classifiers
and adversarial training.

The first three columns of Table 1 show the result. Both DuoAE and UnetAE
perform much better than the baseline. In addition, UnetAE outperforms DuoAE,
despite that UnetAE uses much fewer parameters than DuoAE. We attribute this
to the skip connections, which bring detailed information from the input to
the decoding process and thereby help pitch localization. The best accuracy
0.431 is achieved by UnetAE, with adversarial training.

(a) Baseline model

(b) DuoAE

(c) UnetAE

Fig. 4: t-SNE visualization [37] of the timbre code learned by different
models (best viewed in color). Left: the timbre codes of instrument solos.
Right: the timbre code of audio chunks with two instruments, or of manipulated
solo chunks that are six semitones lower than the original (‘6L’). The
instruments are piano, electric guitar, saxphone, bass, violin, cello, flute,
acoustic guitar, trumpet.

### Evaluation on Disentanglement

#### t-SNE visualization of the learned timbre code

The first thing we do to evaluate the performance of timbre and pitch
disentanglement is via examining the timbre code $\mathbf{Z}_{t}$, by
projecting them from the $\kappa$-dimensional space to a 2-D space with
distributed stochastic neighbor embedding (t-SNE) [37]. We implement the t-SNE
algorithm via Sklearn library with learning rate 20, perplexity 30 and
#iteration 1,000. The first column of Figure 4 shows the learned timbre code
for audio chunks of instrument solos randomly picked from MuseScore. We can
see clear clusters of points from the result of DuoAE and UnetAE, which is
favorable since an instrument solo involves only one instrument.

The second column of Figure 4 shows the learned timbre code for two more
cases: 1) audio chunks with two instruments, picked from MuseScore; 2) audio
chunks of instrument solos, but with pitch purposefully shifted lower by us.
For the first case (in cyan), both DuoAE and UnetAE can nicely position the
chunks in the middle of the two clusters of involved instruments. For the
second case (in pink), both DuoAE and UnetAE fail to position the chunks
within the clusters of the involved instruments, suggesting that the learned
timbre code is not perfectly pitch invariant. But, from the distance between
the chunks and the corresponding clusters, it seems UnetAE performs slightly
better.

(a) pia$+$flu

(b) $\rightarrow$pia

(c) $\rightarrow$pia$+$flu$+$bas

(d) $\rightarrow$vio$+$a-g

(e) pia

(f) $\rightarrow$vio$+$a-g

(g) vio$+$cel$+$flu

(h) $\rightarrow$pia

(i) $\rightarrow$vio$+$cel

(j) $\rightarrow$flu$+$a-g$+$bas

(k) flu$+$a-g$+$bas

(l) $\rightarrow$tru

Fig. 5: Demonstration of timbre crossover (best viewed in color). The source
clips are (a), (e), (g) and (k), and the generated ones (i.e., after crossover
by UnetAE) are those to the right of them. We see from (c) that UnetAE finds
the low-pitched notes for the bass to play; from (d) that it knows to use the
violin to play the melody (originally played by the flute) and the acoustic
guitar to play the chords (originally played by the piano); and from (l) that
it does not always work well—it picks new instruments to play when the timbre
palettes of the source and target clips do not match. [Purple: flute (flu),
Red: piano (pia), Black: bass (bas), Green: acoustic guitar (a-g), Light
purple: trumpet (tru), Yellow: cello (cel), Brown: violin (vio)].

#### Timbre Crossover

Secondly, we propose and employ a new evaluation method called _timbre
crossover_ to evaluate the result of disentanglement. Specifically, we
exchange the timbre codes of two existing audio clips and then decode them, to
see whether we can exchange their instrumentation without affecting the pitch
content in the symbolic domain. For example, if clip A (the _source clip_)
plays the flute and clip B (the _target clip_) plays the trumpet, we hope that
after timbre crossover the new clip A’ would use the trumpet to play the
original tune.

For objective evaluation, we compare the pitch between the pianorolls of the
source clip and the new clip to get the pitch Acc. Moreover, we present the
activity of different instruments in a clip as an $M$-bin histogram and
compute the histogram intersection (HI) [38] between the histograms computed
from the pianorolls of the target clip and the new clip. Both pitch accuracy
and timbre HI are the higher the better.

Table 1 tabulates the result of the following crossover scenarios:
‘anything$\rightarrow$piano,’ ‘anything$\rightarrow$guitar,’ and
‘piano$\rightarrow$violin$+$cello.’ We compute the average result for 20 cases
for each scenario. We can see that UnetAE has better pitch accuracy while
poorer HI. We attribute this to the fact that UnetAE does not have control
over the skip connections—some timbre information may still flow through the
skip connections. But, the advantage of UnetAE is its timbre code is more
pitch-invariant. When doing crossover, the pitch content would subject to less
changes. In contrast, DuoAE achieves higher HI, suggesting that its pitch code
is more timbre-invariant. But, as its timbre code is not pitch-invariant, when
doing crossover, some notes might disappear, due to timbre replacement,
causing the low pitch accuracy.

Figure 5 demonstrates the result of UnetAE. In general, UnetAE works well in
changing the timbre without much affecting the pitch. Please read the caption
for details.

Besides, we also adapt and evaluate the models proposed by [17] and [35] for
timbre crossover. For the model proposed by [35], we consider ‘S’ as
instrument and ‘Z’ as pitch and use pianorolls as the target output. For [17],
we replace ‘style’ with pitch and ‘class’ with instrument. The output is also
pianorolls instead of CQT. As Table 1 shows, they can only achieve 0.558 and
0.295 Pitch Acc for ‘anything$\rightarrow$piano,’ and 0.537 and 0.279 Pitch
Acc for ‘anything$\rightarrow$guitar.’ This poor result is expected, since
these models were not designed for music disentanglement.

Fig. 6: The spectrogram of a note from a piano clip (left) and the one after
timbre crossover (right) by the UnetAE-based model shown in Figure 3. The
target timbre is violin.

#### Audio-domain Music Editing

Finally, we demonstrate the result of timbre crossover in the audio domain,
using the model depicted in Figure 3. Figure 6 shows the spectrograms of a
note chunked from the original and generated audio clips, for
piano$\rightarrow$violin crossover. It is known that, compared to piano,
violin has longer sustain after the attack, and stronger energy at the
harmonics [39]. We can see such characteristics in the generated spectrogram.
However, so far we note that the model may not be sophisticated enough (e.g.,
perhaps we can additionally use a GAN-based loss) so the audio quality has
room for improvement, but it shows that music audio editing is feasible.

### Ablation Study

The two models ‘w/o adv’ (i.e., without adversarial training) in Table 1 can
be viewed as ablated versions of the proposed models. We report some more
ablation study below.

We replace the pianorolls with CQT as the target output for DuoAE. In our
study, we found that predicting pianorolls consistently outperforms predicting
CQT in both instrument recognition accuracy (by 9.6%) and pitch accuracy (by
5.3%). Using the pianorolls as the target performs much better, for it
provides the benefits of “temporal supervision” claimed as the first
contribution of the paper in our introduction section. We therefore decided to
use the pianorolls as the target output for both DuoAE and UnetAE.

Moreover, we also remove skip connection from the UnetAE. By doing so, the
transcription accuracy drops to 0.407. Moreover, since there is no skip
connection to help disentangle pitch and timbre, the pitch Acc of timbre
crossover would drop from 0.691 to 0.068.

## Conclusion and Discussion

In this paper, we have presented two encoder/decoder models to learn
disentangled representation of musical timbre and pitch. The core contribution
is that the input to the models is an audio clip, and as a result it can
change the timbre or pitch content of the audio clip by manipulating the
learned timbre and pitch representations. We can evaluate the result of
disentanglement by generating new pianorolls or audio clips. We also proposed
a new evaluation method called “timbre crossover” to analyze timbre and pitch
disentanglement. Through timbre exchanging, analysis shows that UnetAE has
better ability to create new instrumentation without losing pitch information.
We also extend UnetAE to audio-domain editing. Result shows that it is
feasible to change audio timbre through a deep network structure.

Although our study shows that UnetAE can perform better than DuoAE in terms of
transcription accuracy and timbre crossover, a weakness of UnetAE is that it
has limited control over the skip connections and as a result part of the
timbre information may also flow through skip connection. Besides, UnetAE does
not learn a pitch representation, so the model has little control of the pitch
information.

The loss function for adversarial training is another topic for future study.
Currently, we predict the zero-metrics $\mathbf{0}_{t}$ and $\mathbf{0}_{p}$
to dispel information from the embeddings. Other loss function, such as the
one proposed in [21], can also be tested. Besides, it is important to have
further evaluation on the timbre embedding. For example, from the timbre
crossover we learn that timbre embedding somehow learns the relation between
pitch and timbre, but Figure 5(k) shows that it is not feasible to play the
bass and chord with the trumpet. Further experiment can be done to test on
variety of music genre and instrument combination.

The proposed timbre crossover method holds the promise to help human or AI
composer decide the instruments to play different parts of a given lead sheet
or pianoroll. However, a drawback of the current model is that the timbre
embeddings have to be picked from another music piece, which is less
convenient in real-life usage. A more convenient scenario is that we can
interpolate the existing embeddings and directly decide the instruments to
use. A possible way to improve the model regarding this is to add one-hot
vector for conditional learning. The one-hot vector can then be used for
controlling the instrument usage.

The proposed audio editing model, though likely being the first one of its
kind, is not sophisticated enough to synthesize realistic audio. The model can
be improved by using WaveNet [31] as the decoder, or by using the multi-band
structure proposed by Wang and Yang [33].

## References

  * [1] Asif Ghias, Jonathan Logan, David Chamberlin, and Brian C. Smith,  “Query by humming: Musical information retrieval in an audio database,”  in Proc. ACM Multimedia, 1995, pp. 231–236.
  * [2] Joan Serrà, Emilia Gómez, Perfecto Herrera, and Xavier Serra,  “Chroma binary similarity and local alignment applied to cover song identification,”  IEEE Trans. Audio, Speech, and Language Processing, vol. 16, no. 6, pp. 1138–1151, 2008.
  * [3] Meinard Müller, Daniel P. W. Ellis, Anssi Klapuri, and Gaël Richard,  “Signal processing for music analysis,”  IEEE J. Selected Topics in Signal Processing, vol. 5, no. 6, pp. 1088–1110, 2011.
  * [4] Meinard Müller and Sebastian Ewert,  “Towards timbre-invariant audio features for harmony-based music,”  IEEE Trans. Audio, Speech, and Language Processing, vol. 18, no. 3, pp. 649–662, 2010.
  * [5] Konstantinos Bousmalis, George Trigeorgis, Nathan Silberman, Dilip Krishnan, and Dumitru Erhan,  “Domain separation networks,”  in Proc. Advances in Neural Information Processing Systems, 2016, pp. 343–351.
  * [6] Yu Liu, Fangyin Wei, Jing Shao, Lu Sheng, Junjie Yan, and Xiaogang Wang,  “Exploring disentangled feature representation beyond face identification,”  in Proc. IEEE Conf. Computer Vision and Pattern Recognition, 2018, pp. 2080–2089.
  * [7] Jose David Fernández and Francisco Vico,  “AI methods in algorithmic composition: A comprehensive survey,”  J. Artificial Intelligence Research, vol. 48, no. 1, pp. 513–582, 2013.
  * [8] Hao-Wen Dong, Wen-Yi Hsiao, Li-Chia Yang, and Yi-Hsuan Yang,  “MuseGAN: Symbolic-domain music generation and accompaniment with multi-track sequential generative adversarial networks,”  in Proc. AAAI Conf. Artificial Intelligence, 2018.
  * [9] Hao-Min Liu, Meng-Hsuan Wu, and Yi-Hsuan Yang,  “Lead sheet generation and arrangement via a hybrid generative model,”  in Proc. Int. Soc. Music Information Retrieval Conf., 2018,  Late-breaking paper.
  * [10] Luan Tran, Xi Yin, and Xiaoming Liu,  “Disentangled representation learning GAN for pose-invariant face recognition,”  in Proc. IEEE Conf. Computer Vision and Pattern Recognition, 2017, pp. 1283–1292.
  * [11] Hao-Wen Dong, Wen-Yi Hsiao, and Yi-Hsuan Yang,  “Pypianoroll: Open source Python package for handling multitrack pianoroll,”  in Proc. Int. Soc. Music Information Retrieval Conf., 2018,  Late-breaking paper; [Online] https://github.com/salu133445/pypianoroll.
  * [12] Olaf Ronneberger, Philipp Fischer, and Thomas Brox,  “U-net: Convolutional networks for biomedical image segmentation,”  in Proc. Int. Conf. Medical Image Computing and Computer-Assisted Intervention. Springer, 2015, pp. 234–241.
  * [13] Michael Mathieu, Junbo Zhao, Pablo Sprechmann, Aditya Ramesh, and Yann LeCun,  “Disentangling factors of variation in deep representations using adversarial training,”  in Proc. Advances in Neural Information Processing Systems, 2016\.
  * [14] Hao-Wen Dong and Yi-Hsuan Yang,  “Convolutional generative adversarial networks with binary neurons for polyphonic music generation,”  in Proc. Int. Soc. Music Information Retrieval Conf., 2018.
  * [15] Jonathan Masci, Ueli Meier, Dan Cireşan, and Jürgen Schmidhuber,  “Stacked convolutional auto-encoders for hierarchical feature extraction,”  in Proc. Int. Conf. Artificial Neural Networks, 2011, pp. 52–59.
  * [16] Rui Huang, Shu Zhang, Tianyu Li, and Ran He,  “Beyond face rotation: Global and local perception GAN for photorealistic and identity preserving frontal view synthesis,”  arXiv preprint arXiv:1704.04086, 2017.
  * [17] Yang Liu, Zhaowen Wang, Hailin Jin, and Ian Wassell,  “Multi-task adversarial network for disentangled feature learning,”  in Proc. IEEE Conf. Computer Vision and Pattern Recognition, 2018, pp. 3743–3751.
  * [18] Brian Cheung, Jesse A Livezey, Arjun K Bansal, and Bruno A Olshausen,  “Discovering hidden factors of variation in deep networks,”  arXiv preprint arXiv:1412.6583, 2014.
  * [19] Wei-Ning Hsu, Yu Zhang, and James Glass,  “Learning latent representations for speech generation and transformation,”  arXiv preprint arXiv:1704.04222, 2017.
  * [20] Jun-Ting Hsieh, Bingbin Liu, De-An Huang, Fei-Fei Li, and Juan Carlos Niebles,  “Learning to decompose and disentangle representations for video prediction,”  arXiv preprint arXiv:1806.04166, 2018.
  * [21] Feng Liu, Ronghang Zhu, Dan Zeng, Qijun Zhao, and Xiaoming Liu,  “Disentangling features in 3D face shapes for joint face reconstruction and recognition,”  in Proc. IEEE Conf. Computer Vision and Pattern Recognition, 2018, pp. 5216–5225.
  * [22] Adam Roberts, Jesse Engel, Colin Raffel, Curtis Hawthorne, and Douglas Eck,  “A hierarchical latent vector model for learning long-term structure in music,”  in Proc. Int. Conf. Machine Learning, 2018, pp. 4361–4370.
  * [23] Gino Brunner, Andres Konrad, Yuyi Wang, and Roger Wattenhofer,  “MIDI-VAE: Modeling dynamics and instrumentation of music with applications to style transfer,”  in Proc. Int. Soc. Music Information Retrieval Conf., 2018, pp. 23–27.
  * [24] Rachel M. Bittner, Brian McFee, Justin Salamon, Peter Li, and Juan P. Bello.,  “Deep salience representations for $f0$ tracking in polyphonic music,”  in Proc. Int. Soc. Music Information Retrieval Conf., 2017, pp. 63–70.
  * [25] Maxime Oquab, Léon Bottou, Ivan Laptev, and Josef Sivic,  “Is object localization for free?-weakly-supervised learning with convolutional neural networks,”  in Proc. Int. Computer Vision and Pattern Recognition, 2015, pp. 685–694.
  * [26] Brian McFee, Colin Raffel, Dawen Liang, Daniel PW Ellis, Matt McVicar, Eric Battenberg, and Oriol Nieto,  “librosa: Audio and music signal analysis in python.,”  in Proc. Python in Science Conf., 2015, pp. 18–25.
  * [27] Vijay Badrinarayanan, Alex Kendall, and Roberto Cipolla,  “SegNet: A deep convolutional encoder-decoder architecture for image segmentation,”  IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 39, no. 12, pp. 2481–2495, 2017.
  * [28] Li-Chia Yang, Szu-Yu Chou, and Yi-Hsuan Yang,  “MidiNet: A convolutional generative adversarial network for symbolic-domain music generation,”  in Proc. Int. Soc. Music Information Retrieval Conf., 2017.
  * [29] Ian Simon, Adam Roberts, Colin Raffel, Jesse Engel, Curtis Hawthorne, and Douglas Eck,  “Learning a latent space of multitrack measures,”  in Proc. Int. Soc. Music Information Retrieval Conf., 2018.
  * [30] D. Griffin and Jae Lim,  “Signal estimation from modified short-time Fourier transform,”  IEEE Trans. Acoustics, Speech, and Signal Processing, vol. 32, no. 2, pp. 236–243, 1984.
  * [31] Aäron V. D. Oord, Sander Dieleman, Heiga Zen, Karen Simonyan, Oriol Vinyals, Alex Graves, Nal Kalchbrenner, Andrew W Senior, and Koray Kavukcuoglu,  “WaveNet: A generative model for raw audio,”  arXiv preprint arXiv:1609.03499, 2016.
  * [32] Chris Donahue, Julian McAuley, and Miller Puckette,  “Synthesizing audio with generative adversarial networks,”  arXiv preprint arXiv:1802.04208, 2018.
  * [33] Bryan Wang and Yi-Hsuan Yang,  “PerformanceNet: Score-to-audio music generation with multi-band convolutional residual network,”  in Proc. AAAI Conf. Artificial Intelligence, 2019.
  * [34] Curtis Hawthorne, Andriy Stasyuk, Adam Roberts, Ian Simon, Cheng-Zhi Anna Huang, Sander Dieleman, Erich Elsen, Jesse Engel, and Douglas Eck,  “Enabling factorized piano music modeling and generation with the MAESTRO dataset,”  arXiv preprint arXiv:1810.12247, 2018.
  * [35] Naama Hadad, Lior Wolf, and Moni Shahar,  “A two-step disentanglement method,”  in Proc. IEEE Conf. Computer Vision and Pattern Recognition, 2018, pp. 772–780.
  * [36] Colin Raffel and Daniel P. W. Ellis,  “Optimizing DTW-based audio-to-MIDI alignment and matching,”  in Proc. IEEE Int. Conf. Acoustics, Speech and Signal Processing, 2016, pp. 81–85.
  * [37] Laurens van der Maaten and Geoffrey Hinton,  “Visualizing data using t-SNE,”  J. Machine Learning Research, vol. 9, no. Nov, pp. 2579–2605, 2008\.
  * [38] Michael J. Swain and Dana H. Ballard,  “Color indexing,”  Int. J. Computer Vision, vol. 7, no. 1, pp. 11–32, 1991.
  * [39] Kenichi Miyamoto, Hirokazu Kameoka, Takuya Nishimoto, Nobutaka Ono, and Shigeki Sagayama,  “Harmonic-temporal-timbral clustering (HTTC) for the analysis of multi-instrument polyphonic music signals,”  in Proc. IEEE Int. Conf. Acoustics, Speech and Signal Processing, 2008, pp. 113–116.


# Revisiting the ringdown of GW150914

Maximiliano Isi<EMAIL_ADDRESS>Center for Computational
Astrophysics, Flatiron Institute, 162 5th Ave, New York, NY 10010  Will M.
Farr<EMAIL_ADDRESS>Center for Computational Astrophysics, Flatiron
Institute, 162 5th Ave, New York, NY 10010  Department of Physics and
Astronomy, Stony Brook University, Stony Brook NY 11794, USA

###### Abstract

We examine recent claims that evidence for an overtone in the ringdown of the
GW150914 binary black hole merger was a result of noise anomalies. We cannot
reproduce these claims, finding that our previous analysis of this event is
robust to data analysis choices and consistent with the expectation that
strain after the peak is well described as a superposition of quasinormal
modes of the remnant black hole. We discuss the meaning and implications of
establishing that any specific ringdown mode was detected, and argue that it
is misguided to expect actual LIGO-Virgo data to inform the discussion of
whether or why the merger looks linear.

##  I Introduction

Reference Cotesta  _et al._ (2022) revisits the analysis of the ringdown of
the black hole remnant in GW150914, claiming no evidence in favor of an
overtone mode and suggesting that claims of detection of overtone modes in
that signal Isi  _et al._ (2019) (by the present authors and collaborators)
are noise-dominated. The goal of this short paper is to (1) examine the claims
in Cotesta  _et al._ (2022) that the detection of the first overtone in the
ringdown of GW150914 was due to a noise artifact, and (2) briefly touch upon
the question of what it means to detect an individual quasinormal mode, and to
what extent such detections constrain the ‘linearity’ of the spacetime of the
remnant.

Regarding (1), we find that we cannot reproduce the key results reported in
Cotesta  _et al._ (2022): we recover a significant non-zero overtone amplitude
stably over reasonable choices of the analysis starting time, with a decay
rate fully consistent with the expected damping of the overtone; we find that
the constraint on the deviation of the overtone frequency from the general
relativistic prediction is also stable over analysis starting time. We take
this as evidence that the results in Isi  _et al._ (2019) are robust.
Regarding (2), we argue that the question is ill-posed, with much of the focus
that has been devoted to it being misplaced. Answers to questions about the
nature of the source and whether its spacetime behaves linearly or not at a
given point are not to be found in data from gravitational wave detectors Aasi
_et al._ (2015); Acernese  _et al._ (2015); Akutsu  _et al._ (2021) at the
present moment.

Figure 1:  Analysis of the GW150914 ringdown. _Top:_ Posterior for the strain
peak time, $\mathrm{max}(h^{2}_{+}+h^{2}_{\times})$, from full IMR analyses
with either IMRPhenomPv2 Husa  _et al._ (2016); Khan  _et al._ (2016); Schmidt
_et al._ (2011) (blue) or SEOBNRv4ROM Taracchini  _et al._ (2014); Pürrer
(2014, 2016) (orange); dotted lines enclose the 68%-credible symmetric
interval. _Middle:_ Posterior for the overtone amplitude $A_{1}$ obtained with
a two-tone model ($n=0,1$) applied at different starting times $t_{0}$ (blue,
with the preferred start time from Cotesta  _et al._ (2022) in green); the
posterior is summarized by the median (circle) and 68% (95%) symmetric
credible intervals as thick (thin) vertical lines. The dark (light) gray band
shows the 68%-credible (95%-credible) expected decay rate starting from the
measurement at $t_{0}=t_{\rm ref}$. _Bottom:_ The left axis shows the ratio of
the $A_{1}$ posterior median to the standard deviation (blue solid curve); the
right axis shows the standard deviation of the Kerr deviation in the overtone
frequency ($\delta f_{1}$), when both $\delta f_{1}$ and $\delta\tau_{1}$ are
allowed to vary within $[-0.5,0.5]$. We express time from $t_{\rm ref}$ in
seconds (top axis), as well as in units of $t_{M}=GM/c^{3}$ for
$M=69M_{\odot}$, i.e., $t_{M}=0.34\,\mathrm{ms}$ (bottom axis). As in Isi  _et
al._ (2019), the reference time corresponds to GPS time $t_{\rm
ref}=1126259462.423\,\mathrm{s}$ at the LIGO Hanford detector (vertical solid
line).

##  II GW150914 revisited

Here we revisit the GW150914 data to examine the claims in Cotesta  _et al._
(2022), the core of which we understand to be: (1) the chosen analysis start
time in Isi  _et al._ (2019) was too early, (2) the detection of the first
overtone is unstable against small changes in this start time, and, finally,
(3) the overtone amplitude cannot be constrained to be nonzero except
significantly before the peak, when the model is known to be invalid.

The reference time in Isi  _et al._ (2019) (a GPS time of $t_{\rm
ref}=1126259462.423\,\mathrm{s}$ at the Hanford detector) was taken directly
from the original LIGO-Virgo analysis Abbott  _et al._ (2016), where it was
claimed to be the peak time;111Timing posterior information for GW150914 has
not been released by the LIGO and Virgo collaborations, in spite of what is
claimed in Cotesta  _et al._ (2022)—to our knowledge, the statement in Abbott
_et al._ (2016) was the best available at the time of writing Isi  _et al._
(2019). In this work, the peak time reconstructions were obtained from our
reanalysis of the full GW150914 data using LALInference Veitch  _et al._
(2015). it was chosen before carrying out the analysis and not tuned a
posteriori. The actual location of the signal peak is uncertain because it is
measured imperfectly from noisy data. It is true that, as pointed out in
Cotesta  _et al._ (2022), the reference time in Isi  _et al._ (2019); Abbott
_et al._ (2016) happens to fall before the median of the reconstructed
posterior; nevertheless, it lies well within the bulk of the distribution, no
more than ${\sim}1\sigma$ away from the median (Fig. 1, top). The choice is
thus adequate.

Ringdown analyses are, for the most part, conditional on the chosen truncation
time by construction Isi and Farr (2021a). This is because the start time is
not a model parameter, but rather defines the data to be analyzed. The meaning
and implementation of start-time marginalization are not clear. Techniques
like those proposed in Finch and Moore (2021) might offer a possible avenue to
allow the start time of the ringdown analysis to vary in a controlled way.

Instead, to account for uncertainty about the arrival time of the peak of the
gravitational wave strain, we can always repeat the analysis at different
start times. This was the strategy in Cotesta  _et al._ (2022), where it was
found that the posterior for the amplitude of the first overtone quickly drops
to zero unless the start time is chosen to be significantly before the
expected peak, where the quasinormal-mode model is known to be inapplicable.
Furthermore, Cotesta  _et al._ (2022) reported a decrease in the overtone
amplitude immediately before and after the peak time used in Isi  _et al._
(2019). Altogether, this was interpreted as a sign that the positive
identification of the overtone in Isi  _et al._ (2019) was due to a noise
artifact. However, we are unable to reproduce any of these observations.

We carry out a similar exercise as in Cotesta  _et al._ (2022) using the
publicly available code in Isi and Farr (2021b).222We show results from an
analysis using 0.2 s of data sampled at 2048 Hz, but our conclusions did not
change when we increased the sampling rate by factors of two and four. In all
cases, the identification of the data truncation—and therefore analysis
start—time is done at 16 kHz, the highest available sampling rate. This is not
done by the LIGO-Virgo analyses in Abbott  _et al._ (2021a, b, 2020). The
result is summarized in Fig. 1, where the main panel shows the overtone
amplitude ($A_{1}$) posterior obtained in a two-tone analysis (i.e., including
the $n=0,1$ tones of the $\ell=m=2$ angular harmonic) starting at different
times, before and after our previously chosen reference time in Isi  _et al._
(2019) ($t_{0}-t_{\rm ref}=0$ in the plot).333We use a strain model such that
$h=F_{+}h_{+}+F_{\times}h_{\times}$ for detector antenna patterns
$F_{+/\times}$, and polarization quadratures
$h_{+}=\frac{1}{2}(1+\cos^{2}\iota)\sum_{n=0,1}A_{n}\cos(\omega_{n}t+\phi_{n})\exp(-t/\tau_{n})$
and
$h_{\times}=\cos\iota\sum_{n=0,1}A_{n}\sin(\omega_{n}t+\phi_{n})\exp(-t/\tau_{n})$,
where $\omega_{n}$ and $\tau_{n}$ are determined for the $n$th overtone as a
function of $M$ and $\chi$, which are themselves measured from the data
together with $A_{n}$ and $\phi_{n}$. We fix the sky location to the values in
Isi  _et al._ (2019), and set the inclination to $\cos\iota=-0.99$. This
corresponds to the mchi_aligned model in the ringdown package Isi and Farr
(2021b), modulo a factor of two in the definition of the amplitudes. The
amplitude posterior clearly favors $A_{1}>0$ for a broad range of times,
without sharp fluctuations. On the contrary, the posterior decays smoothly in
full consistency with the damping expected from the first overtone; this is
shown by the gray envelopes, which represent credible regions generated by
drawing overtone parameters from the posterior obtained at $t_{0}=t_{\rm ref}$
and projecting the decay forward in time. This result is consistent with the
true peak lying close to $t_{\rm ref}$ and the two-tone model being a good fit
from that point on.

In Isi  _et al._ (2019) we reported that the $A_{1}$ posterior median fell
$3.6\sigma$ away from zero, a measure of overtone detection significance in
reference to a model with the fundamental alone, namely, $A_{1}=0$ (but see
Isi  _et al._ (2019); Isi and Farr (2021a) for an account of other factors
considered, including the effect of allowing for a second overtone). We find a
consistent result in our new analysis here, as shown in the bottom panel of
Fig. 1 ($\mu/\sigma$, blue solid curve). For $t_{0}>t_{\rm ref}$, this figure
of merit decays smoothly, consistent with the evolution of the amplitude
posterior.

The bottom panel of Fig. 1 also shows our constraints on deviations from Kerr
in the overtone frequency ($\delta f_{1}$) when we allow both the frequency
and damping to vary around the Kerr prediction;444We implement the deviation
such that the frequency of the overtone is $f_{1}=f_{1}^{\rm
Kerr}(M,\chi)\exp(\delta f_{1})$ where $f_{1}^{\rm Kerr}(M,\chi)$ is the Kerr
prediction for a given mass and spin, and similar for $\tau_{1}$. This is
slightly different from the parameterization in Isi  _et al._ (2019), where we
had $f_{1}=f_{1}^{\rm Kerr}(M,\chi)\left(1+\delta f_{1}\right)$; we discuss
this in Isi and Farr (2021a). the constraint is summarized by the $\delta
f_{1}$ posterior standard deviation ($\mathrm{std}(\delta f_{1})$, green
dashed curve). As we expect, the frequency is best constrained for start times
around $t_{\rm ref}$. For later start times, the posterior support for
$A_{1}=0$ increases and the frequency is more poorly constrained; this is
clear also from the full $\delta f_{1}$ posteriors in Fig. 2. On the other
hand, including data significantly before $t_{\rm ref}$ can cause the model to
prefer deviations away from the Kerr spectrum, indicating that Kerr
quasinormal modes are not a good model for the data at that point—the
transition between the two regimes occurs sharply around
$t_{0}\approx-1.5t_{M}$ (Fig. 3).

Figure 2:  Posterior for deviations from Kerr in the overtone frequency
($\delta f_{1}$) obtained from a two tone analysis in which both $\delta
f_{1}$ and $\delta\tau 1$ are allowed to vary uniformly over $[-0.5,0.5]$. The
color of the trace corresponds to the analysis start time. The deviation is
implemented such that $f_{1}=f_{1}^{\rm Kerr}\exp(\delta f_{1})$ Isi and Farr
(2021a). The result for $t_{0}=t_{\rm ref}$ is highlighted with a dashed
trace.

A similar conclusion can be drawn from the posterior for the remnant mass
($M$) and dimensionless spin magnitude ($\chi$) obtained assuming a Kerr
spectrum in the analyses of Fig. 1. As shown in Fig. 4, the remnant posterior
stabilizes around $t_{0}\approx t_{\rm ref}$, at which point it begins to show
better consistency with the expectation from the full inspiral-merger-ringdown
analysis (IMR, black dashed). This is not true for a model including the
fundamental mode alone (Fig. 5), which fails to agree with the IMR analysis at
these early times.

Figure 3:  Summary of the posteriors in Fig. 2, plus additional analyses with
start times preceding $t_{\rm ref}$ (shaded area). The left axis shows the
absolute value of the posterior median-to-standard-deviation ratio for $\delta
f_{1}$ (blue solid line); the right axis shows the posterior standard
deviation (green dashed line), and the standard deviation of the prior (green
dotted line). For $t_{0}\lesssim t_{\rm ref}-2t_{M}$, $\delta f_{1}$ is
preferentially negative, indicating lower frequencies are favored.

Besides looking at GW150914 itself, we also inject a number of simulated
signals in real data around the event (Fig. 6). We find results consistent
with those above: the overtone amplitude posterior varies as might be expected
from regular noise fluctuations, and nothing indicates that nonstationarities
in the data are derailing the analysis. This experiment is meant to account
for non-Gaussianities in actual detector noise; injections in Gaussian noise
were already presented in Isi and Farr (2021a).

Figure 4:  Posterior for the remnant mass ($M$ in solar masses $M_{\odot}$)
and dimensionless spin magnitude ($\chi$) obtained with a two-tone analysis
($n=0,1$) beginning at different start times (color). Contours enclose 90% of
the probability mass. The result for $t_{0}=t_{\rm ref}$ is highlighted with a
dashed blue trace. The result from the IMR analysis is also shown for
comparison (dashed black), and was obtained from the LIGO-Virgo samples LIGO
Scientific Collaboration and Virgo Collaboration (2018); Abbott  _et al._
(2021c) as detailed in Isi  _et al._ (2019).

Our results here and in Isi  _et al._ (2019), like in _any_ analysis, are
predicated on a number of choices about data conditioning and noise power-
spectral-density (equivalently, autocovariance-function) estimation. For
different reasonable choices for these settings, we obtain slightly different
results, but this variation is within expectation and do not alter our
conclusions.

Figure 5:  Same as in Fig. 5 but for a model including just the fundamental
mode ($n=0$). This model is a poor description of the data at these early
times.

##  III The meaning and implications of detecting a mode

The problem of establishing whether a specific ringdown mode has been detected
is made difficult by the fact that the meaning of “detection” is not itself
clear in this context. In the regime where we can inarguably expect a
superposition of quasinormal modes to be a good description of the data, one
can reasonably phrase this question as a choice between $A_{1}=0$ and
$A_{1}>0$ within a model including both $n=0,1$—that is, presuming no other
quasinormal modes could be expected to be relevant. Already in that simplified
scenario the question is technically ill-posed: under that model, the
probability that $A_{1}=0$ exactly is effectively zero; this problem appears
in other contexts, e.g., the determination of whether a binary black hole
system is precessing or whether a given angular harmonic has been
detected.555Lack of precession requires the individual black-hole spins to be
_exactly_ perpendicular to the orbital plane, which is practically impossible
to achieve physically. Higher-than-quadrupole multipoles of the radiation are
technically present in all realistic signals, even if their amplitude is
minimal; furthermore, models that simply ignore them while keeping the
quadrupole as in general relativity are unphysical. See, e.g., GW1 (2020);
Abbott  _et al._ (2020) for relevant discussions. The problem is made even
murkier if, like in Cotesta  _et al._ (2022), one allows for the possibility
that data are not a superposition of damped sinusoids and neither of the $n=0$
nor the $n=0,1$ models are valid. In that case, there is no reason to expect a
preference for either model to be informative (whether quantified via the
amplitude posterior or a Bayes factor).

Bayes factors are an especially poor tool for this purpose; the drawbacks of
Bayes factors in the context of this problem were already discussed in Isi and
Farr (2021a), so we do not repeat that here. Nevertheless, Fig. 7 shows the
Bayes factors corresponding to the amplitude posteriors in Fig. 1; the Bayes
factors favor the presence of the overtone even for $t_{0}>t_{\rm ref}$,
unlike in Cotesta  _et al._ (2022). Of course, as remarked in Isi and Farr
(2021a), the overall vertical location of the Bayes factor curve can be
shifted arbitrarily by modifying the maximum allowed value of $A_{1}$ in the
prior (in our case $A_{1}=10^{-20}$). Besdies prior differences, comparing
Bayes factors from different analyses can also be inappropriate when the
models used are not identical—for that reason comparisons to Bayes factors in
Abbott  _et al._ (2021a) are not necessarily interesting or valid.

In any case, the question of whether a mode was detected is inherently
“fuzzy.” To quote Gelman and Stern (2006): the difference between
statistically significant and not statistically significant is not, itself,
statistically significant. This can be seen most clearly in Figure 6, where a
signal of constant amplitude injected into noise with (presumably) identical
statistical properties generates, on average, a ${\sim}3.5\sigma$ overtone
“detection;” but with fluctuations $\sim\pm 1\sigma$, as expected from the
intrinsic variation of the particular noise realization from moment to moment.
Thus the $3.6\sigma$ “detection” in Isi  _et al._ (2019) could just as easily
have been a marginal $2\sigma$ “suggestion” or a $5\sigma$ “confident
detection” of the overtone’s presence. Compounded with the above, this makes
the question of “how significant is the detection of an overtone” not
particularly interesting.

In the end, much of the interest in whether the overtone has positively been
detected in GW150914 seems to be fueled by a wish to empirically test the
results in Giesler  _et al._ (2019) that the gravitational waves from a binary
black hole merger can be described as a superposition of quasinormal modes of
the remnant black hole from the peak of the strain onward; however, this urge
is misguided. As made clear in Isi  _et al._ (2019), ringdown analyses
starting at the peak _presume_ the results of Giesler  _et al._ (2019) to
hold, and are not designed to corroborate them (at least not directly).

Figure 6:  Posterior for the overtone amplitude $A_{1}$ obtained for
injections of the GW150914-like numerical simulation SXS:BBH:0305 Kidder  _et
al._ (2019); Boyle  _et al._ (2019); Schmidt  _et al._ (2017) into actual LIGO
noise around GW150914. We set the total redshifted mass to be $M_{\rm
tot}=72\,M_{\odot}$, with a luminosity distance of 450 Mpc. The inset shows
the value of the median-to-standard-deviation ratio. The variation in the
amplitude posterior agrees with what we expect from Gaussian noise.

As we have argued elsewhere, all that is needed before confronting the data in
an analysis like Isi  _et al._ (2019) is the _empirical_ confirmation from
numerical relativity studies that, starting from the peak of the strain, the
gravitational-wave emission from a binary black hole should be well described
by a superposition of quasinormal modes _with frequencies and damping rates of
the remnant black hole_ (and not others). This is indeed the conclusion of
Giesler  _et al._ (2019) and other studies (particularly Fig. 7 in Giesler
_et al._ (2019)). If, after analyzing real data, the quasinormal modes are not
found to be consistent with a final Kerr remnant, then the observation does
not agree with what we expect from the merger of two Kerr black holes in
general relativity.666Or, like in any other analysis, there is a failure of at
least one of the many underlying assumptions—including that the noise is
stationary, that its spectrum has been correctly estimated, and that the
analysis start time has been properly chosen. See Abbott  _et al._ (2019,
2021a) for relevant discussions of the role of systematics in tests of general
relativity with gravitational waves.

Figure 7:  Base-ten logarithm of Bayes factors comparing a two-tone model
($n=0,1$) to a one-tone model ($n=0$) for runs presented in Fig. 1. The values
are computed from the amplitude posterior via the Savage-Dickey density ratio,
for a prior uniform in $0\leq A_{1}\leq 10^{-20}$.

The ability to carry out such a cross-check is precisely the reason we are
interested in identifying multiple quasinormal modes in the first place. To
the extent that we can carry out this Kerr-consistency test, the mode was
detected. Concretely, as we argued in Isi  _et al._ (2019), the $\delta f_{1}$
(or $\delta\tau_{1}$) posterior can only be informative insofar as the data
support a nonzero overtone amplitude. If that is the case, we can say the
overtone was “detected” and check for one of two scenarios: (1) the deviation
posterior is consistent with zero and thus also with Kerr black holes in
general relativity, or (2) the deviation posterior disfavors zero, indicating
that one or more of the assumptions in the model is broken (including,
potentially, the assumption that general relativity is correct).

##  IV Conclusion

We have revisited the ringdown of GW150914 finding our previous claims in Isi
_et al._ (2019) to be robust against choices in analysis start time, within
the expected location of the signal peak from inspiral-merger-ringdown
analyses. We cannot reproduce the claims to the contrary presented in Cotesta
_et al._ (2022). The robustness of our observations against choices of
sampling rate and power-spectrum estimation methods gives us additional
confidence in our results. We make code to reproduce results in this paper
available in Isi and Farr (2022).

###### Acknowledgements.

The Flatiron Institute is a division of the Simons Foundation, supported
through the generosity of Marilyn and Jim Simons. This material is based upon
work supported by NSF’s LIGO Laboratory which is a major facility fully funded
by the National Science Foundation. This research has made use of data or
software obtained from the Gravitational Wave Open Science Center (gw-
openscience.org), a service of LIGO Laboratory, the LIGO Scientific
Collaboration, the Virgo Collaboration, and KAGRA. LIGO Laboratory and
Advanced LIGO are funded by the United States National Science Foundation
(NSF) as well as the Science and Technology Facilities Council (STFC) of the
United Kingdom, the Max-Planck-Society (MPS), and the State of
Niedersachsen/Germany for support of the construction of Advanced LIGO and
construction and operation of the GEO600 detector. Additional support for
Advanced LIGO was provided by the Australian Research Council. Virgo is
funded, through the European Gravitational Observatory (EGO), by the French
Centre National de Recherche Scientifique (CNRS), the Italian Istituto
Nazionale di Fisica Nucleare (INFN) and the Dutch Nikhef, with contributions
by institutions from Belgium, Germany, Greece, Hungary, Ireland, Japan,
Monaco, Poland, Portugal, Spain. The construction and operation of KAGRA are
funded by Ministry of Education, Culture, Sports, Science and Technology
(MEXT), and Japan Society for the Promotion of Science (JSPS), National
Research Foundation (NRF) and Ministry of Science and ICT (MSIT) in Korea,
Academia Sinica (AS) and the Ministry of Science and Technology (MoST) in
Taiwan. This paper carries LIGO document number LIGO-P2200028.

## References

  * Cotesta  _et al._ (2022) R. Cotesta, G. Carullo, E. Berti, and V. Cardoso, (2022), arXiv:2201.00822 [gr-qc] .
  * Isi  _et al._ (2019) M. Isi, M. Giesler, W. M. Farr, M. A. Scheel, and S. A. Teukolsky, Phys. Rev. Lett. 123, 111102 (2019), arXiv:1905.00869 [gr-qc] .
  * Aasi  _et al._ (2015) J. Aasi _et al._ (LIGO Scientific Collaboration), Classical Quantum Gravity 32, 074001 (2015), arXiv:1411.4547 [gr-qc] .
  * Acernese  _et al._ (2015) F. Acernese _et al._ (Virgo Collaboration), Classical Quantum Gravity 32, 024001 (2015), arXiv:1408.3978 [gr-qc] .
  * Akutsu  _et al._ (2021) T. Akutsu _et al._ (KAGRA), PTEP 2021, 05A101 (2021), arXiv:2005.05574 [physics.ins-det] .
  * Husa  _et al._ (2016) S. Husa, S. Khan, M. Hannam, M. Pürrer, F. Ohme, X. J. Forteza, and A. Bohé, Phys. Rev. D 93, 044006 (2016), arXiv:1508.07250 [gr-qc] .
  * Khan  _et al._ (2016) S. Khan, S. Husa, M. Hannam, F. Ohme, M. Pürrer, X. Jiménez Forteza, and A. Bohé, Phys. Rev. D 93, 044007 (2016), arXiv:1508.07253 [gr-qc] .
  * Schmidt  _et al._ (2011) P. Schmidt, M. Hannam, S. Husa, and P. Ajith, Phys. Rev. D 84, 024046 (2011), arXiv:1012.2879 [gr-qc] .
  * Taracchini  _et al._ (2014) A. Taracchini _et al._ , Phys. Rev. D 89, 061502 (2014), arXiv:1311.2544 [gr-qc] .
  * Pürrer (2014) M. Pürrer, Classical Quantum Gravity 31, 195010 (2014), arXiv:1402.4146 [gr-qc] .
  * Pürrer (2016) M. Pürrer, Phys. Rev. D 93, 064041 (2016), arXiv:1512.02248 [gr-qc] .
  * Abbott  _et al._ (2016) B. P. Abbott _et al._ (LIGO Scientific, Virgo), Phys. Rev. Lett. 116, 221101 (2016), [Erratum: Phys. Rev. Lett.121,no.12,129902(2018)], arXiv:1602.03841 [gr-qc] .
  * Veitch  _et al._ (2015) J. Veitch _et al._ , Phys. Rev. D 91, 042003 (2015), arXiv:1409.7215 [gr-qc] .
  * Isi and Farr (2021a) M. Isi and W. M. Farr, (2021a), arXiv:2107.05609 [gr-qc] .
  * Finch and Moore (2021) E. Finch and C. J. Moore, Phys. Rev. D 104, 123034 (2021), arXiv:2108.09344 [gr-qc] .
  * Isi and Farr (2021b) M. Isi and W. M. Farr, “ringdown software package,”  (2021b).
  * Abbott  _et al._ (2021a) R. Abbott _et al._ (LIGO Scientific, Virgo), Phys. Rev. D 103, 122002 (2021a), arXiv:2010.14529 [gr-qc] .
  * Abbott  _et al._ (2021b) R. Abbott _et al._ (LIGO Scientific, VIRGO, KAGRA), (2021b), arXiv:2112.06861 [gr-qc] .
  * Abbott  _et al._ (2020) R. Abbott _et al._ (LIGO Scientific, Virgo), Astrophys. J. Lett. 900, L13 (2020), arXiv:2009.01190 [astro-ph.HE] .
  * LIGO Scientific Collaboration and Virgo Collaboration (2018) LIGO Scientific Collaboration and Virgo Collaboration, “GWTC-1,” https://doi.org/10.7935/82H3-HH23 (2018).
  * Abbott  _et al._ (2021c) R. Abbott _et al._ (LIGO Scientific, Virgo), SoftwareX 13, 100658 (2021c), arXiv:1912.11716 [gr-qc] .
  * GW1 (2020) (2020), arXiv:2004.08342 [astro-ph.HE] .
  * Gelman and Stern (2006) A. Gelman and H. Stern, The American Statistician 60, 328 (2006).
  * Giesler  _et al._ (2019) M. Giesler, M. Isi, M. A. Scheel, and S. Teukolsky, Phys. Rev. X9, 041060 (2019), arXiv:1903.08284 [gr-qc] .
  * Kidder  _et al._ (2019) L. Kidder, H. Pfeiffer, M. Scheel, M. Boyle, D. Hemberger, G. Lovelace, and B. Szilagyi, “Binary black-hole simulation sxs:bbh:0305,”  (2019).
  * Boyle  _et al._ (2019) M. Boyle _et al._ , Class. Quant. Grav. 36, 195006 (2019), arXiv:1904.04831 [gr-qc] .
  * Schmidt  _et al._ (2017) P. Schmidt, I. W. Harry, and H. P. Pfeiffer, (2017), arXiv:1703.01076 [gr-qc] .
  * Abbott  _et al._ (2019) B. P. Abbott _et al._ (LIGO Scientific, Virgo), Phys. Rev. D100, 104036 (2019), arXiv:1903.04467 [gr-qc] .
  * Isi and Farr (2022) M. Isi and W. M. Farr, “Revisiting the ringdown of gw150914 (data release),”  (2022).