# Toward Fully Self-Supervised Multi-Pitch Estimation

Frank Cwitkowitz  Zhiyao Duan

###### Abstract

Multi-pitch estimation is a decades-long research problem involving the
detection of pitch activity associated with concurrent musical events within
multi-instrument mixtures. Supervised learning techniques have demonstrated
solid performance on more narrow characterizations of the task, but suffer
from limitations concerning the shortage of large-scale and diverse polyphonic
music datasets with multi-pitch annotations. We present a suite of self-
supervised learning objectives for multi-pitch estimation, which encourage the
concentration of support around harmonics, invariance to timbral
transformations, and equivariance to geometric transformations. These
objectives are sufficient to train an entirely convolutional autoencoder to
produce multi-pitch salience-grams directly, without any fine-tuning. Despite
training exclusively on a collection of synthetic single-note audio samples,
our fully self-supervised framework generalizes to polyphonic music mixtures,
and achieves performance comparable to supervised models trained on
conventional multi-pitch datasets.

Multi-Pitch Estimation, Self-Supervised Learning, Music Transcription, Timbre-
Invariance, Geometric-Equivariance

##  1 Introduction

Music is comprised of auditory events generated by exciting the strings,
chambers, or surfaces of an instrument. Pitched events or notes are
characterized by air particle oscillations, each with varying strength and
frequency at an integer multiple (in Hz) of a fundamental frequency (F0)
(Müller, 2015). A note’s F0 directly informs its pitch and captures
articulations such as vibrato, while the presence and relative strength of the
remaining oscillations or harmonics influence its unique timbral properties
(Klapuri & Davy, 2006). Multi-pitch estimation (MPE) seeks to detect all F0
activity across regular time steps within polyphonic audio signals. This is an
important task with broad interest in Music Information Retrieval (MIR)
research and exciting applications in music
analysis111https://www.sonicvisualiser.org,
composition222https://scorecloud.com,
editing333https://www.celemony.com/en/melodyne,
education444https://yousician.com, indexing and
search555https://www.deezer.com/explore/en-us/features/songcatcher/, etc.
Pitch is foundational for various musical concepts surrounding melody and
harmony. As such, MPE is also critical for several downstream MIR tasks,
including automatic music transcription (AMT) (Benetos et al., 2019) and
automatic chord recognition (ACR) (Pauwels et al., 2019). Furthermore, MPE has
the potential to inform systems addressing other challenging MIR tasks like
source separation (Schulze-Forster et al., 2023), polyphonic audio synthesis
(Renault et al., 2023), or bandwidth extension (Grumiaux & Lagrange, 2023).

Figure 1: Example note signal (left) and its corresponding power spectrogram
(right) from the NSynth dataset (Engel et al., 2017), which is used to train
the proposed self-supervised framework.

Although monophonic or single-pitch estimation systems have reached near-
perfect accuracy (Morrison et al., 2023), the polyphonic or multi-pitch case
has proven to be much more difficult. This is largely due to the problem of
overlapping harmonics and ambiguity regarding the number of simultaneously
active notes at a given time (Klapuri, 1998). However, this difficulty is
further compounded by the lack of availability of large-scale comprehensive
datasets for training MPE systems. At present, acquiring multi-pitch
annotations essentially consists of running monophonic pitch trackers on
isolated stems from mixtures of monophonic instruments and performing manual
error-correction (Duan et al., 2010; Bittner et al., 2014; Li et al., 2018).
Alternatively, to avoid manual correction without propagating errors, audio
stems can be re-synthesized directly from the pitch estimates (Salamon et al.,
2017). These procedures have significant limitations in terms of scalability
and the types of musical instruments, passages, and environments that can be
supported.

Additionally, supervision has never been the primary learning paradigm for
humans to learn to transcribe simultaneous pitches. Humans, especially people
without absolute pitch, rely on the intrinsic patterns of auditory
representations and their relations across pitch to transcribe music (Bregman,
1994), requiring only one or a few reference pitches (i.e., labels). This more
closely resembles the self-supervised learning (SSL) paradigm in machine
learning, which has been garnering increasing attention due to its profound
success in learning rich representations for complex data spanning various
modalities (Ericsson et al., 2022). SSL has already been successfully applied
to monophonic pitch estimation, by leveraging the pitch-transposition
equivariance property of the Constant-Q Transform (CQT) (Gfeller et al., 2020;
Riou et al., 2023), a common representation for MIR, or using model-based
approaches (Engel et al., 2020b). However, MPE is a much more complicated
problem, and no attempts have yet been made to adopt an SSL approach for the
task.

In this work, we present a fully self-supervised framework for MPE (SS-MPE),
consisting of a Harmonic Constant-Q Transform (HCQT) (Bittner et al., 2017)
feature extraction stage, a fully convolutional 2D autoencoder (Cwitkowitz et
al., 2024), and a closed-loop suite of self-supervised training objectives.
Our objectives collectively encourage the model to produce multi-pitch
salience-gram estimates directly, bypassing the need for fine-tuning or
annotated data. The first three objectives encourage concentration of energy
around strong F0 candidates, derived via a simple weighted harmonic average
(Klapuri, 2006). The remaining two objectives promote invariance to timbre
transformations and equivariance to geometric transformations, intrinsic
properties of pitch and the CQT, respectively. Our model is trained entirely
on the NSynth dataset (Engel et al., 2017), which comprises 4-second
monophonic note samples (see Figure 1) synthesized from commercial virtual
instruments. While a global pitch label is available for the samples in
NSynth, our framework is completely agnostic to them.

Remarkably, without training on mixtures of the monophonic samples or
performing any sort of music language modeling, SS-MPE is able to generalize
to polyphonic music and yields MPE performance approaching that of supervised
methods, requiring only a simple peak-picking and thresholding procedure at
the output. This result is very promising and merits further investigation
into SSL for MPE and related tasks. Our work is completely open-source666All
code is publicly available at https://github.com/cwitkowitz/ss-mpe., and can
be reproduced with minimal computational resources. This paper seeks only to
establish a foundation for applying SSL techniques to MPE. We suggest several
directions for future work, including scaling the approach to large polyphonic
music datasets and designing self-supervised objectives to exploit additional
properties of music and audio.

##  2 Related Work

MPE for both speech and music signals has been an active area of research for
several decades (de Cheveigné, 2006; Christensen & Jakobsson, 2009).
Traditional methods are largely based around signal processing techniques and
statistical modeling, including spectral analysis (Klapuri, 2003; Yeh et al.,
2010; Zhou et al., 2009), auditory modeling (Klapuri, 2008), maximum a
posteriori (Kameoka et al., 2007) or maximum likelihood (Emiya et al., 2010;
Duan et al., 2010) estimation, and filtering or subspace techniques
(Christensen et al., 2008, 2011). Another class of methods perform
factorization on spectral features (Bertin et al., 2010; Benetos & Weyde,
2015) or the audio waveform (Cogliati et al., 2017) to estimate note templates
and their corresponding activations. While traditional methods are generally
interpretable and yield reasonable performance, they often incorporate several
tunable parameters and fail to generalize to the complexity of arbitrary real-
world music (Benetos et al., 2013). Moreover, most research has investigated
instrument or domain-specific characterizations of MPE in order to simplify
the problem. These approaches tend to be targeted toward piano transcription,
mainly due to the relative convenience of collecting corresponding note
annotations for audio from MIDI-driven acoustic pianos (e.g., Disklavier)
(Emiya et al., 2010; Hawthorne et al., 2019).

Another related task is AMT, where the aim is to directly estimate note events
or pitches quantized to a musical scale (Benetos et al., 2019). Typically, AMT
systems either perform MPE implicitly (Gardner et al., 2021) or consist of a
frame-level MPE module followed by post-processing to produce note-level
predictions (Benetos & Dixon, 2011; Elowsson, 2020). The methodologies and
challenges for MPE and AMT are nearly identical, and their terminology is
frequently used interchangeably in the literature.

With the explosion of deep learning, various convolutional-recurrent neural
networks were subsequently proposed for piano transcription (Böck & Schedl,
2012; Sigtia et al., 2016; Hawthorne et al., 2018) and other instrument-
specific AMT tasks. These models can generally be factored into an acoustic
component which extracts rich features from input spectra and a music language
component which aggregates the features across time before making final
predictions. Currently, sequence modeling (Yan et al., 2021) and transformer-
style architectures (Hawthorne et al., 2021; Toyama et al., 2023) dominate
piano transcription, with systems nearing superhuman performance. In parallel,
there has also been work to build more general MPE systems (Bittner et al.,
2017) based on deep learning and to reuse insights from piano transcription
research for multi-instrument AMT (Bittner et al., 2022). In particular,
transformer-style architectures applied to this domain have demonstrated
remarkable efficacy (Gardner et al., 2021; Lu et al., 2023). U-Net
architectures have been also applied to AMT as frontends to learn richer
feature representations (Pedersoli et al., 2020; Weiß & Peeters, 2022), as
well as to predict multi-pitch activations directly (Wu et al., 2020). These
autoencoders transform input spectrograms into latent codes with reduced
dimensionality, which are then decoded into an output representation matching
the original input size. This type of architecture is especially notable for
MPE, given the typically high correlation between spectrograms and multi-pitch
salience-grams, as well as their matching dimensionality.

While these methods have undoubtedly advanced research on MPE, they are still
fundamentally limited by the availability of large-scale and diverse training
data. As such, alternative methods have been proposed to extract more out of
the data that is currently available. Some works leverage multi-task learning
(Manilow et al., 2020; Tanaka et al., 2020; Hung et al., 2021; Cheuk et al.,
2022) to simultaneously separate and transcribe various sources, while others
employ auxiliary reconstruction objectives (Cheuk et al., 2021b, a; Wu et al.,
2023; Cwitkowitz et al., 2024) to promote a more thorough modeling of music
signals. Additionally, several methods have utilized weak supervision in order
to train models with weakly-aligned data (Weiß & Peeters, 2021; Maman &
Bermano, 2022; Krause et al., 2023b, a). Pre-training methodologies have also
been proposed to learn from random mixtures of pre-transcribed monophonic
audio in-the-wild (Simon et al., 2022) or pitch annotations without
corresponding audio (Cheuk et al., 2023). Finally, many works have leveraged
synthetic datasets (Engel et al., 2017; Manilow et al., 2019; Ostermann et
al., 2023), however it is generally recognized that there exists a significant
domain gap between synthesized and real-world audio.

Following the fields of natural language processing and computer vision, there
has also recently been an increasing focus on designing self-supervised
techniques to learn robust feature representations for downstream tasks from
large databases of unlabeled audio. One such technique is to leverage training
objectives based on masked reconstruction in order to teach models the
important characteristics and structure of audio (Tagliasacchi et al., 2020)
and music (Li et al., 2023) signals. Another is to use a contrastive learning
objective (Chen et al., 2020), which maximizes the similarity between the
learned representations for randomized label-preserving transformations of a
unique training example, while simultaneously maximizing the dissimilarity
between the learned representations for disparate samples. Contrastive
learning has been applied to general audio (Al-Tahan & Mohsenzadeh, 2021;
Saeed et al., 2021) and MIR (Spijkervet & Burgoyne, 2021; McCallum et al.,
2022, 2024) in order to teach models how to organize information about signals
that will be relevant for downstream processing. SSL has also been applied
directly to specific MIR tasks including monophonic pitch estimation (Gfeller
et al., 2020; Riou et al., 2023), singer identity representation learning
(Torres et al., 2023), beat tracking (Desblancs et al., 2023), and tempo
estimation (Henkel et al., 2024). For pitch estimation, the pitch-
transposition equivariance property of the input CQT can be exploited to learn
relative differences in pitch. Self-supervised and unsupervised techniques
based on the principles of differentiable digital signal processing (DDSP)
(Engel et al., 2020a) have also been proposed for monophonic pitch estimation
(Engel et al., 2020b; Hayes et al., 2023; Torres et al., 2024). Finally,
multi-speaker self-supervised pitch estimation based on physical modeling of
the human vocal tract has been carried out (Li et al., 2022), though this
approach is explicitly designed for two speakers.

##  3 Method

In this section, we detail our proposed framework SS-MPE. We begin by
discussing our HCQT feature extraction module, which provides strong a priori
information for MPE and has several important implications for the design of
our self-supervised objectives. We then outline the autoencoder network
employed within our framework and the process of obtaining F0 estimates from
the output multi-pitch salience-grams. Finally, we present our SSL objectives,
which concentrate energy around strong F0 candidates and exploit the
properties of timbre-invariance and geometric equivariance.

###  3.1 Feature Extraction

Figure 2: HCQT power spectrogram (in dB) for an audio signal (top-left), the
corresponding first-harmonic channel (bottom-left) and weighted harmonic
average (top-right), and an example multi-pitch salience-gram estimate
(bottom-right). Self-supervised objectives ($\mathcal{L}_{har}$ and
$\mathcal{L}_{sup}$) are employed to encourage the model to concentrate energy
around strong F0 candidates.

The CQT is a time-frequency representation derived from basis functions with
geometrically spaced center frequencies and a constant Q-factor (Brown, 1991).
It has been applied widely across various MIR tasks to compute the frequency
response for frequencies aligned with the Western music scale (Schörkhuber &
Klapuri, 2010). Illustrations of the CQT spectrogram are provided in Figure 1
(right) and 2 (bottom-left). Like other time-frequency representations, the
CQT is equivariant to time shifting and stretching. However, the CQT is also
pitch-transposition equivariant, meaning that pitch shifting a time-domain
signal will correspond to a frequency (vertical) translation in its CQT
spectrogram. These properties have been leveraged to learn shift-invariant
representations for relevant MIR tasks (Lattner et al., 2019), and to design
self-supervised training objectives for pitch estimation (Gfeller et al.,
2020; Riou et al., 2023).

In this work, we utilize the HCQT, a simple extension which stacks multiple
CQTs along a third channel axis (Bittner et al., 2017), illustrated in Figure
2 (top-left). The individual CQTs $X_{h}$ are computed relative to the
harmonics and subharmonics $h\in\mathcal{H}$ of a reference frequency
$f_{min}$. We follow the originally proposed set
$\mathcal{H}=\\{0.5,1,2,3,4,5\\}$. Although we are only interested in
detecting F0s from the first harmonic channel ($h=1$), the inclusion of prime-
number harmonic channels ($h\in\\{2,3,5\\}$) can offer theoretical advantages
(Klapuri, 1998), and the sub-harmonic ($h=0.5$) and octave ($h\in\\{2,4\\}$)
channels can help to avoid octave errors. Since it is simply a collection of
CQTs, the HCQT maintains the aforementioned equivariance properties.

Apart from providing richer spectral features, the HCQT allows neural
networks, in particular those based on convolution, to leverage the channel
dimension in order to more easily discover harmonic relationships for MIR
tasks (Bittner et al., 2017). Methodologies based on leveraging convolutional
kernels with harmonic structure (Zhang et al., 2020; Wang et al., 2020b, a;
Wei et al., 2022a, b) have also been proposed to exploit these relationships
within a single-channel CQT spectrogram. However, many harmonics do not
perfectly align with the center frequencies of single-channel spectrogram
bins. The HCQT covers these frequencies explicitly by computing a separate CQT
for each harmonic.

We utilize a convolution-based implementation (Cwitkowitz, 2019) to compute
decibel-scale ($[-80,0]$ dB) HCQT777In practice, the Variable-Q Transform
(VQT) (Schörkhuber et al., 2014) is computed for each harmonic in order to
increase computational efficiency and time resolution at lower frequencies.
power spectrograms with a $-80$ dB cutoff for each audio signal. These
features are re-scaled and expressed as $X_{\mathcal{H}}\in[0,1]^{6\times
K\times N}$, where $K=440$ bins with $b_{oct}=60$ bins per octave ($5$ bins
per semitone) and $N$ is the variable frame-length which depends on the input
audio size. The reference frequency for the first-harmonic channel is
$f_{min}=27.5$ Hz, which corresponds to A0, the lowest note on a standard
piano. All audio is resampled to $22,050$ Hz before processing, and we use a
hop size of $256$ samples or approximately $11.6$ ms to compute the HCQT
spectrograms.

###  3.2 Model

We adopt the model from the Timbre-Trap low-resource MPE framework (Cwitkowitz
et al., 2024) as the backbone neural network for SS-MPE. Timbre-Trap employs a
fully convolutional 2D autoencoder inspired by the SoundStream audio
compression model (Zeghidour et al., 2021). The encoder comprises four blocks
with dilated convolutional layers, residual connections, and strided
convolutions. The decoder mirrors the encoder, but with transposed
convolutions in place of strided convolutions. The base model is largely left
unchanged, though we adapt the input and output channel sizes for our
particular usage and remove the latent channel mechanism used in the original
paper to switch between reconstruction and transcription mode. We also insert
layer normalization after each block of the encoder and decoder for improved
training stability. In total, the model contains roughly 545K trainable
parameters.

Our framework leverages this model to process input HCQT spectrograms
$X_{\mathcal{H}}$ by downsampling them along the frequency axis and projecting
them into increasing channels, ultimately producing $N$ latent vectors of size
$128$. The latent vectors are subsequently upsampled to the original
dimensionality and aggregated into a single channel of logits. Through our
self-supervised objectives, the model implicitly learns to produce multi-pitch
salience logits, which are interpreted as multi-pitch salience-grams
$\hat{Y}\in[0,1]^{K\times N}$ by applying $\operatorname{sigmoid}$ activation.
During inference, final F0 estimates are obtained for each frame simply by
performing local peak-picking across the frequency axis and applying an un-
tuned threshold of $0.5$.

###  3.3 Energy Concentration

Our first set of self-supervised objectives is designed to influence the model
to concentrate energy around strong F0 candidates by leveraging the strong a
priori information encoded in the input HCQT spectrogram and to adopt an
implicit bias toward sparsity. We first compute a weighted average across the
linear-scale HCQT power spectrogram

$\bar{X}^{(lin)}=\sum^{5}_{h=1}\frac{1}{h^{4}}X_{h}^{(lin)},$ (1)

excluding the sub-harmonic and loosely following harmonic weighting (12 dB
decay per octave) precedent (Klapuri, 2006). We then convert linear-scale
$\bar{X}^{(lin)}$ back to decibel scale $\bar{X}$ and formulate two separate
losses, the harmonic loss $\mathcal{L}_{har}$ and support loss
$\mathcal{L}_{sup}$. These serve as the positive-only and negative-only
components of a continuous binary cross-entropy (BCE) loss, respectively. The
harmonic loss

$\mathcal{L}_{har}=-\frac{1}{N}\sum^{N-1}_{n=0}\sum^{K-1}_{k=0}\bar{X}[k,n]\log\left(\hat{Y}[k,n]\right)$
(2)

is computed with respect to the positive activations of the weighted harmonic
average of the HCQT, and encourages the model to concentrate energy in output
$\hat{Y}$ around frequencies with strong energy in $\bar{X}$, i.e.,
frequencies that manifest strong harmonics overall. However, $\bar{X}$ can
also contain energy at frequencies that do not manifest strong energy in
$X_{1}$. Such frequencies cannot correspond to F0s, except for in rare cases
where there is a missing fundamental. Therefore, the support loss

$\mathcal{L}_{sup}=-\frac{1}{N}\sum^{N-1}_{n=0}\sum^{K-1}_{k=0}\left(1-X_{1}[k,n]\right)\log\left(1-\hat{Y}[k,n]\right)$
(3)

is computed with respect to the complement of activations in $X_{1}$ to
mitigate this issue. Finally, we introduce an $L_{1}$-norm sparsity loss

$\mathcal{L}_{spr}=\frac{1}{N}\sum^{N-1}_{n=0}\sum^{K-1}_{k=0}\left|\hat{Y}[k,n]\right|,$
(4)

which ideally reduces spurious activations and encourages the output of the
model to utilize less frequency bins overall. An example multi-pitch salience-
gram produced using these three objectives is presented in Figure 2 (bottom-
right).

Figure 3: Examples (by row) of sampled Gaussian equalization curves (center)
applied to CQT spectrograms (left) to produce equalized spectrograms (right)
for the timbre-invariance objective. Figure 4: Examples (by row) of sampled
geometric transformations (center) applied to CQT spectrograms (left) to
produce transformed spectrograms (right) for the geometric-equivariance
objective.

###  3.4 Timbre Invariance

The self-supervised objectives outlined in Section 3.3 provide a reasonable
foundation for estimating multi-pitch salience, but they do not yield
particularly confident or separable output, and distribute energy broadly
across harmonics. Moreover, they are heavily influenced by the timbral
properties, i.e., relative harmonic amplitudes, of the input audio. In order
to alleviate these issues, we introduce a timbre-invariance objective,
motivated by the notion that the same pitch content can be produced by many
different musical instruments. As such, we perform timbre transformations by
applying Gaussian equalization curves $EQ(\cdot,\mu,\sigma,A)$ to scale the
input HCQT features, inspired by previous data augmentation strategies for MIR
(Schlüter & Grill, 2015; Abeßer & Müller, 2021). Gaussian parameters
$\mu\in[0,K-1]$, $\sigma\in[0,2b_{oct}]$, and $A\in[0.625,1.375]$ are
uniformly sampled for each audio signal. Equalization curves are applied
across each channel of $X_{\mathcal{H}}$ by performing nearest-neighbor
interpolation with respect to the scaling factor computed for frequency bins
aligned with the first harmonic, producing equalized features
$X_{\mathcal{H}}^{(EQ)}$, which are clipped to the range $[0,1]$. An
illustration of various sampled equalization curves and their application to
an example CQT spectrogram is presented in Figure 3. Given model
$\mathcal{F}(\cdot)$ and a set of equalized HCQT features
$X_{\mathcal{H}}^{(EQ)}$, a timbre-invariance loss is then computed as

$\mathcal{L}_{tmb}=\frac{1}{N}\sum^{N-1}_{n=0}\sum^{K-1}_{k=0}\mathcal{B}\left(\mathcal{F}\left(X_{\mathcal{H}}^{(EQ)}\right)[k,n],\hat{Y}[k,n]\right),$
(5)

where $\mathcal{B}(\cdot,\cdot)$ is the standard BCE loss, computed over the
multi-pitch salience-gram estimated from the equalized features with respect
to the salience-gram produced for the original features. Applying random
equalizations will inevitably boost and cut the various harmonics within each
signal. In conjunction with the objectives from Section 3.3 this can allow the
model to discover more efficient representations, i.e., the corresponding
implicit multi-pitch salience-grams.

###  3.5 Geometric Equivariance

Our final self-supervised objective is based around the geometric-equivariance
properties of the CQT noted in Section 3.1, which have been exploited in
previous works on SSL for monophonic pitch estimation (Gfeller et al., 2020;
Riou et al., 2023). As the input HCQT features are equivariant to translations
in time and frequency, as well as distortions in time, we adopt a geometric-
equivariance objective to ensure the output representation shares the same
equivariance properties. We perform geometric transformations
$GM(\cdot,\Delta_{k},\Delta_{n},\gamma)$ to shift and stretch the input HCQT
features accordingly. Integer transformation parameters
$\Delta_{k}\in[-2b_{oct},2b_{oct}]$, $\Delta_{n}\in[-\frac{N}{4},\frac{N}{4}]$
($\frac{N}{4}\approx 1$ second for our experiments) are sampled uniformly, and
real-valued parameter $\gamma\in[0.5,2]$ is randomly sampled using two uniform
distributions ($[0.5,1]$ and $[1,2]$) with equal weights. The transformations
are applied to all harmonic channels to produce features
$X_{\mathcal{H}}^{(GM)}$, filling $0$ in for unknown values. An illustration
of various sampled transformation parameters and their application to an
example CQT spectrogram is presented in Figure 4. Given model
$\mathcal{F}(\cdot)$ and a set of transformed HCQT features
$X_{\mathcal{H}}^{(GM)}$, a geometric-equivariance loss is then computed as

$\mathcal{L}_{geo}=\frac{1}{N}\sum^{N-1}_{n=0}\sum^{K-1}_{k=0}\mathcal{B}\left(\mathcal{F}\left(X_{\mathcal{H}}^{(GM)}\right)[k,n],\hat{Y}[k,n]^{(GM)}\right),$
(6)

where $\hat{Y}^{(GM)}$ is the multi-pitch salience-gram produced for the
original features transformed in accordance with $X_{\mathcal{H}}^{(GM)}$.
Apart from inducing properties of geometric equivariance, this objective can
also encourage the allocation of energy to the F0. Specifically, certain
transformations effectively low-pass filter the input features, discarding
higher-order harmonics, while others effectively high-pass filter the input
features, discouraging energy at higher harmonics if the output of the model
adequately represents pitch salience.

##  4 Experiments

| Bach10 | URMP | Su | TRIOS | GuitarSet
---|---|---|---|---|---
Method | $\mathit{P}$ | $\mathit{R}$ | $\mathit{Acc}$ | $\mathit{P}$ | $\mathit{R}$ | $\mathit{Acc}$ | $\mathit{P}$ | $\mathit{R}$ | $\mathit{Acc}$ | $\mathit{P}$ | $\mathit{R}$ | $\mathit{Acc}$ | $\mathit{P}$ | $\mathit{R}$ | $\mathit{Acc}$
Deep-Salience | $86.0$ | $61.0$ | $55.5$ | $91.1$ | $72.8$ | $67.6$ | $74.1$ | $47.9$ | $40.4$ | $92.3$ | $39.4$ | $38.1$ | $77.7$ | $70.6$ | $57.7$
Basic-Pitch | $90.2$ | $75.7$ | $70.0$ | $88.2$ | $71.1$ | $64.9$ | $54.2$ | $44.1$ | $32.8$ | $88.2$ | $44.2$ | $41.5$ | $80.9$ | $75.9$ | $64.0$
Timbre-Trap | $81.2$ | $84.2$ | $70.4$ | $80.5$ | $88.8$ | $73.3$ | $52.1$ | $53.0$ | $34.8$ | $69.4$ | $49.7$ | $40.4$ | $48.6$ | $75.6$ | $41.3$
CREPE(multi) | $82.0$ | $24.8$ | $23.5$ | $88.2$ | $37.3$ | $35.6$ | $63.9$ | $20.4$ | $17.9$ | $87.6$ | $20.3$ | $19.6$ | $70.7$ | $53.9$ | $43.9$
PESTO(multi) | $81.7$ | $23.0$ | $21.9$ | $65.7$ | $27.8$ | $23.9$ | $69.9$ | $21.9$ | $19.6$ | $78.4$ | $18.4$ | $17.3$ | $61.3$ | $50.7$ | $36.5$
SS-MPE ($\mathcal{L}$) | $65.1$ | $82.5$ | $57.1$ | $61.6$ | $63.9$ | $44.4$ | $50.4$ | $52.8$ | $34.1$ | $65.8$ | $51.1$ | $38.9$ | $49.9$ | $67.5$ | $39.2$
SS-MPE ($\mathcal{L}_{t}$) | $65.4$ | $66.5$ | $49.1$ | $66.0$ | $49.5$ | $37.3$ | $52.3$ | $45.4$ | $31.3$ | $65.3$ | $43.7$ | $34.2$ | $54.4$ | $55.4$ | $36.4$
SS-MPE ($\mathcal{L}_{g}$) | $63.0$ | $81.4$ | $55.0$ | $63.8$ | $48.9$ | $36.4$ | $54.4$ | $38.8$ | $28.6$ | $70.3$ | $39.1$ | $31.3$ | $59.5$ | $58.1$ | $40.2$
SS-MPE ($\mathcal{L}_{e}$) | $59.9$ | $78.6$ | $51.4$ | $58.8$ | $49.2$ | $34.5$ | $50.7$ | $39.5$ | $27.7$ | $63.9$ | $40.6$ | $30.3$ | $52.0$ | $60.4$ | $37.0$
Table 1: Comparison of precision (P), recall (R), and accuracy (Acc) scores
(in percentage points) on MPE and AMT datasets for our self-supervised
framework (SS-MPE). Results are also reported for supervised MPE methods Deep-
Salience (Bittner et al., 2017), Basic-Pitch (Bittner et al., 2022), and
Timbre-Trap (Cwitkowitz et al., 2024), as well as multi-pitch estimates
derived from the salience-grams of monophonic models CREPE (Kim et al., 2018)
and PESTO (Riou et al., 2023). Grayed values indicate data was utilized during
training.

In order to demonstrate the efficacy of SS-MPE, we design experiments where
the model is trained on audio from the NSynth (Engel et al., 2017) dataset and
evaluated on various MPE datasets. Our framework is trained with all the
objectives defined in Sections 3.3-3.5, i.e., with overall loss

$\mathcal{L}=\mathcal{L}_{har}+\mathcal{L}_{sup}+\mathcal{L}_{spr}+\mathcal{L}_{tmb}+\mathcal{L}_{geo}.$
(7)

However, we also experiment with various ablations to the training objective
in order to observe the utility of timbre-invariance and geometric-
equivariance. In particular, we also train the framework using the following
loss variations:

$\displaystyle\mathcal{L}_{e}=\mathcal{L}_{har}+\mathcal{L}_{sup}+\mathcal{L}_{spr},$
(8) $\displaystyle\mathcal{L}_{t}=\mathcal{L}_{e}+\mathcal{L}_{tmb},$ (9)
$\displaystyle\mathcal{L}_{g}=\mathcal{L}_{e}+\mathcal{L}_{geo}.$ (10)

These ablations correspond to experiments with the energy concentration
objectives only, the energy concentration objectives with the timbre-
invariance objective, and the energy concentration objectives with the
geometric-invariance objective. Each experiment was executed on a single
GeForce RTX 2080 TI graphics card and took between 3 to 4 days.

###  4.1 Datasets

NSynth is a collection of over 300k music note samples synthesized from
commercial libraries (Engel et al., 2017). It covers $11$ instrument classes,
$1006$ unique instruments, pitches spanning the $88$ keys of a standard MIDI
piano, and several note velocities. Each note sample corresponds to $4$
seconds of audio, with the note onset and release (if applicable) occurring at
$t=0$ and $t=3$, respectively. An example of a note sample from NSynth is
presented in Figure 1. All audio within NSynth is monophonic, and any note
samples with pitch outside of the supported range of our HCQT module are
discarded. We follow the provided splits for training and validation, but use
only a subset of $200$ samples for validation to reduce overall validation
time.

Evaluation is conducted on a variety of MPE and AMT datasets, mostly
comprising classical music. We evaluate on the full datasets, using multitrack
mixtures if applicable.

Bach10 contains multitrack audio along with multi-pitch annotations for $10$
short four-part Bach chorales featuring a violin, clarinet, saxophone, and
bassoon (Duan et al., 2010).

URMP contains multitrack audio along with multi-pitch annotations for $44$
classical music pieces arranged for string, woodwind, and brass instruments
spanning $14$ instrument classes (Li et al., 2018). The dataset features $11$
duets, $12$ trios, $14$ quartets, and $7$ quintets.

The Su dataset contains audio mixtures and note-level annotations for $3$
piano solo, $3$ string quartet, $2$ piano quintet, and $2$ violin sonata
passages (Su & Yang, 2015).

TRIOS contains multitrack audio along with note-level annotations for $4$
classical music trio passages and $1$ jazz trio passage (Fritsch, 2012). The
dataset includes piano, string, woodwind, and brass instruments, and the jazz
track features percussive elements.

GuitarSet contains audio along with multi-pitch and note-level annotations for
360 short monophonic and polyphonic recordings of an acoustic guitar (Xi et
al., 2018).

###  4.2 Training & Evaluation Details

All training is conducted with a batch size of $20$ using AdamW optimizer
(Loshchilov & Hutter, 2019). We set an initial learning rate of $10^{-4}$ and
halve the learning rate whenever the validation loss has not decreased for
$0.5$ epochs. We also perform gradient-norm clipping using a value of $10$. In
each experiment, the model is only trained for $3$ epochs, which is sufficient
to reach convergence. Validation is conducted every $250$ batches, and the
checkpoint with the lowest validation loss across training is selected for
evaluation. We utilize the community-standard mir_eval package (Raffel et al.,
2014) to compute precision (P), recall (R), and accuracy (Acc) of multi-pitch
estimates with respect to the ground-truth $Y$ for the tracks within the MPE
and AMT datasets described in Section 4.1. Accuracy is computed as the ratio
between the number of pitch estimates within $0.5$ semitones of matched
ground-truth pitches (true positives) to the total number of pitch estimates
(true positives and false positives) plus the number of missed ground-truth
pitch estimates (false negatives). Final results are computed by averaging
across all tracks within an individual dataset.

###  4.3 Baselines

We compare our self-supervised framework (SS-MPE) to various supervised MPE
baselines, including Deep-Salience (Bittner et al., 2017), Basic-Pitch
(Bittner et al., 2022), and Timbre-Trap (Cwitkowitz et al., 2024). Deep-
Salience is a convolutional neural network (CNN) with an HCQT frontend that
was trained on mixtures created from F0-annotated stems of MedleyDB (Bittner
et al., 2014). Basic-Pitch extends this methodology and incorporates ideas
from the Onsets & Frames (Hawthorne et al., 2018) model to generate multi-
pitch and note predictions using a single architecture. It is trained on a
variety of AMT and MPE datasets including Slakh (Manilow et al., 2019),
MAESTRO (Hawthorne et al., 2019), MedleyDB stems, GuitarSet (Xi et al., 2018),
and vocal data. Timbre-Trap leverages an audio reconstruction objective to
more efficiently learn from a portion of URMP (Li et al., 2018) (see Section
3.2 for more details). We also obtain multi-pitch estimates from the pitch
salience-grams generated by monophonic pitch estimation models CREPE (Kim et
al., 2018) and PESTO (Riou et al., 2023). CREPE is a supervised CNN trained on
MIR-1K (Hsu & Jang, 2010), Bach10 stems, RWC-Synth (Mauch & Dixon, 2014),
MedleyDB stems, MDB-STEM-Synth (Salamon et al., 2017), and NSynth. PESTO is a
self-supervised framework featuring a CNN trained on MIR-1K. Although these
are monophonic pitch estimation methods, we include them because our framework
is also trained on monophonic audio. For all baselines, frame-level F0
estimates are obtained by performing local peak-picking across frequency and
applying a threshold (0.5 for Timbre-Trap and 0.3 for all others).

Figure 5: CQT spectrogram $X_{1}$ for track 01-AchGottundHerr of Bach10 (Duan
et al., 2010) along with the multi-pitch salience-gram output generated using
our full self-supervised framework (SS-MPE) and the output for the framework
with training objective ablations.

###  4.4 Results & Discussion

Results for all experiments and baselines are presented in Table 1, and an
example of the multi-pitch salience-gram output for SS-MPE and each ablation
is provided in Figure 5. Generally, we observe that the performance of SS-MPE
approaches and in some cases rivals supervised MPE methods, while greatly
outperforming the simple salience thresholding scheme applied to the
monophonic baselines for most datasets. However, we do note that the
monophonic baselines perform comparably to SS-MPE on GuitarSet, roughly half
of which consists of monophonic audio.

These promising results are achieved without applying any weighting to the
objectives, performing any tuning on the threshold for positive predictions,
or calibrating the trained model with annotated data as in (Gfeller et al.,
2020; Riou et al., 2023). Moreover, our framework is trained entirely on
4-second monophonic note samples from NSynth (Engel et al., 2017), yet
remarkably it generalizes to sequences of polyphonic audio with pitch
fluctuations such as vibrato. This is not true for the monophonic baselines,
which have trouble identifying more than one pitch at a time, as evidenced by
their high precision and low recall scores.

The full framework with all objectives ($\mathcal{L}$) achieves the best
overall performance compared to the ablations and self-supervised PESTO. It
also exhibits the most sparsity when predicting multi-pitch salience-grams.
However, the timbre-invariance and geometric-invariance objectives applied
without one another ($\mathcal{L}_{t}$ and $\mathcal{L}_{g}$) still generally
contribute to improved performance over the energy-only objectives
($\mathcal{L}_{e}$). These experiments also suggest the contribution of
timbre-invariance and geometric-invariance is varied on specific datasets. For
instance, geometric-equivariance in isolation ($\mathcal{L}_{g}$) improves
over timbre-invariance ($\mathcal{L}_{t}$) in isolation on Bach10 and
GuitarSet, whereas the opposite is true for URMP, Su, and TRIOS. We also
speculate that NSynth may already intrinsically encode geometric-equivariance
with respect to pitch to some degree, since it comprises notes played over the
entire pitch range of each instrument.

While the presented results are extremely encouraging, there is still plenty
of room for improvement. In particular, the current framework is relatively
susceptible to false-alarm predictions, especially those caused by harmonics
and artifacts (e.g., percussive note attack or release). Additionally,
sometimes the F0 is missed entirely, potentially due to the lack of realistic
training data. Future work will consist of improving the existing formulation,
incorporating more augmentation techniques for increased robustness (e.g.,
noise-invariance to reduce the effect of artifacts or time-varying
transformations to simulate pitch fluctuations in real recordings), and
scaling the framework to polyphonic music datasets.

##  5 Conclusion

In this paper, we have introduced a fully self-supervised framework to
generate multi-pitch estimates. Our framework leverages a harmonic
spectrogram, an autoencoder architecture, and self-supervised objectives
designed to concentrate sparse energy around strong fundamental frequency
candidates and exploit the properties of timbre invariance and geometric
equivariance. The model is trained entirely using isolated synthetic single-
note samples, yet can generalize to conventional polyphonic data. No fine-
tuning is required within our framework, and performance approaches that of
supervised methods. Future work will consist of improving and extending the
current methodology, as well as scaling it to larger and more comprehensive
audio datasets.

## Acknowledgements

This work is partially supported by National Science Foundation (NSF) grants
No. 1846184 and 2222129, and synergistic activities funded by NSF grant
DGE-1922591.

## References

  * Abeßer & Müller (2021) Abeßer, J. and Müller, M.  Jazz bass transcription using a U-net architecture.  _Electronics_ , 10(6):670, 2021.
  * Al-Tahan & Mohsenzadeh (2021) Al-Tahan, H. and Mohsenzadeh, Y.  CLAR: Contrastive learning of auditory representations.  In _Proceedings of AISTATS_ , 2021.
  * Benetos & Dixon (2011) Benetos, E. and Dixon, S.  Joint multi-pitch detection using harmonic envelope estimation for polyphonic music transcription.  _IEEE Journal of Selected Topics in Signal Processing (JSTSP)_ , 5(6):1111–1123, 2011.
  * Benetos & Weyde (2015) Benetos, E. and Weyde, T.  An efficient temporally-constrained probabilistic model for multiple-instrument music transcription.  In _Proceedings of ISMIR_ , 2015.
  * Benetos et al. (2013) Benetos, E., Dixon, S., Giannoulis, D., Kirchhoff, H., and Klapuri, A.  Automatic music transcription: Challenges and future directions.  _Journal of Intelligent Information Systems_ , 41:407–434, 2013.
  * Benetos et al. (2019) Benetos, E., Dixon, S., Duan, Z., and Ewert, S.  Automatic music transcription: An overview.  _IEEE Signal Processing Magazine_ , 36(1):20–30, 2019.
  * Bertin et al. (2010) Bertin, N., Badeau, R., and Vincent, E.  Enforcing harmonicity and smoothness in bayesian non-negative matrix factorization applied to polyphonic music transcription.  _IEEE Transactions on Audio, Speech, and Language Processing (TASLP)_ , 18(3):538–549, 2010.
  * Bittner et al. (2014) Bittner, R. M., Salamon, J., Tierney, M., Mauch, M., Cannam, C., and Bello, J. P.  MedleyDB: A multitrack dataset for annotation-intensive MIR research.  In _Proceedings of ISMIR_ , 2014.
  * Bittner et al. (2017) Bittner, R. M., McFee, B., Salamon, J., Li, P., and Bello, J. P.  Deep salience representations for F0 estimation in polyphonic music.  In _Proceedings of ISMIR_ , 2017.
  * Bittner et al. (2022) Bittner, R. M., Bosch, J. J., Rubinstein, D., Meseguer-Brocal, G., and Ewert, S.  A lightweight instrument-agnostic model for polyphonic note transcription and multipitch estimation.  In _Proceedings of ICASSP_ , 2022.
  * Böck & Schedl (2012) Böck, S. and Schedl, M.  Polyphonic piano note transcription with recurrent neural networks.  In _Proceedings of ICASSP_ , 2012.
  * Bregman (1994) Bregman, A. S.  _Auditory Scene Analysis: The Perceptual Organization of Sound_.  MIT Press, 1994.
  * Brown (1991) Brown, J. C.  Calculation of a constant Q spectral transform.  _Journal of the Acoustical Society of America (JASA)_ , 89(1):425–434, 1991.
  * Chen et al. (2020) Chen, T., Kornblith, S., Norouzi, M., and Hinton, G.  A simple framework for contrastive learning of visual representations.  In _Proceedings of ICML_ , 2020.
  * Cheuk et al. (2021a) Cheuk, K. W., Herremans, D., and Su, L.  ReconVAT: A semi-supervised automatic music transcription framework for low-resource real-world data.  In _Proceedings of ACM MM_ , 2021a.
  * Cheuk et al. (2021b) Cheuk, K. W., Luo, Y.-J., Benetos, E., and Herremans, D.  The effect of spectrogram reconstruction on automatic music transcription: An alternative approach to improve transcription accuracy.  In _Proceedings of ICPR_ , 2021b.
  * Cheuk et al. (2022) Cheuk, K. W., Choi, K., Kong, Q., Li, B., Won, M., Hung, A., Wang, J.-C., and Herremans, D.  Jointist: Joint learning for multi-instrument transcription and its applications.  _arXiv preprint arXiv:2206.10805_ , 2022.
  * Cheuk et al. (2023) Cheuk, K. W., Sawata, R., Uesaka, T., Murata, N., Takahashi, N., Takahashi, S., Herremans, D., and Mitsufuji, Y.  Diffroll: Diffusion-based generative music transcription with unsupervised pretraining capability.  In _Proceedings of ICASSP_ , 2023.
  * Christensen & Jakobsson (2009) Christensen, M. and Jakobsson, A.  _Multi-Pitch Estimation_.  Morgan & Claypool, 2009.
  * Christensen et al. (2008) Christensen, M. G., Stoica, P., Jakobsson, A., and Jensen, S. H.  Multi-pitch estimation.  _Signal Processing_ , 88(4):972–983, 2008.
  * Christensen et al. (2011) Christensen, M. G., Højvang, J. L., Jakobsson, A., and Jensen, S. H.  Joint fundamental frequency and order estimation using optimal filtering.  _EURASIP Journal on Advances in Signal Processing_ , 2011(13), 2011.
  * Cogliati et al. (2017) Cogliati, A., Duan, Z., and Wohlberg, B.  Piano transcription with convolutional sparse lateral inhibition.  _IEEE Signal Processing Letters_ , 24(4):392–396, 2017.
  * Cwitkowitz (2019) Cwitkowitz, F.  End-to-end music transcription using fine-tuned variable-Q filterbanks.  Master’s thesis, Rochester Institute of Technology, 2019.
  * Cwitkowitz et al. (2024) Cwitkowitz, F., Cheuk, K. W., Choi, W., Martínez-Ramírez, M. A., Toyama, K., Liao, W.-H., and Mitsufuji, Y.  Timbre-Trap: A low-resource framework for instrument-agnostic music transcription.  In _Proceedings of ICASSP_ , 2024.
  * de Cheveigné (2006) de Cheveigné, A.  Multiple F0 estimation.  In Wang, D. and Brown, G. J. (eds.), _Computational Auditory Scene Analysis: Principles, Algorithms, and Applications_ , chapter 2, pp. 45–79. Wiley-IEEE Press, 2006.
  * Desblancs et al. (2023) Desblancs, D., Lostanlen, V., and Hennequin, R.  Zero-note samba: Self-supervised beat tracking.  _IEEE/ACM Transactions on Audio, Speech, and Language Processing (TASLP)_ , 31:2922–2934, 2023.
  * Duan et al. (2010) Duan, Z., Pardo, B., and Zhang, C.  Multiple fundamental frequency estimation by modeling spectral peaks and non-peak regions.  _IEEE Transactions on Audio, Speech, and Language Processing (TASLP)_ , 18(8):2121–2133, 2010.
  * Elowsson (2020) Elowsson, A.  Polyphonic pitch tracking with deep layered learning.  _Journal of the Acoustical Society of America (JASA)_ , 148(1):446–468, 2020.
  * Emiya et al. (2010) Emiya, V., Badeau, R., and David, B.  Multipitch estimation of piano sounds using a new probabilistic spectral smoothness principle.  _IEEE Transactions on Audio, Speech, and Language Processing (TASLP)_ , 18(6):1643–1654, 2010.
  * Engel et al. (2017) Engel, J., Resnick, C., Roberts, A., Dieleman, S., Norouzi, M., Eck, D., and Simonyan, K.  Neural audio synthesis of musical notes with wavenet autoencoders.  In _Proceedings of ICML_ , 2017.
  * Engel et al. (2020a) Engel, J., Hantrakul, L., Gu, C., and Roberts, A.  DDSP: Differentiable digital signal processing.  In _Proceedings of ICLR_ , 2020a.
  * Engel et al. (2020b) Engel, J., Swavely, R., Hantrakul, L. H., Roberts, A., and Hawthorne, C.  Self-supervised pitch detection by inverse audio synthesis.  In _ICML Workshop on Self-Supervision in Audio and Speech_ , 2020b.
  * Ericsson et al. (2022) Ericsson, L., Gouk, H., Loy, C. C., and Hospedales, T. M.  Self-supervised representation learning: Introduction, advances, and challenges.  _IEEE Signal Processing Magazine_ , 39(3):42–62, 2022.
  * Fritsch (2012) Fritsch, J.  High quality musical audio source separation.  Master’s thesis, UPMC / IRCAM / Telécom ParisTech, 2012.
  * Gardner et al. (2021) Gardner, J., Simon, I., Manilow, E., Hawthorne, C., and Engel, J.  MT3: Multi-task multitrack music transcription.  In _Proceedings of ICLR_ , 2021.
  * Gfeller et al. (2020) Gfeller, B., Frank, C., Roblek, D., Sharifi, M., Tagliasacchi, M., and Velimirović, M.  SPICE: Self-supervised pitch estimation.  _IEEE/ACM Transactions on Audio, Speech, and Language Processing (TASLP)_ , 28:1118–1128, 2020.
  * Grumiaux & Lagrange (2023) Grumiaux, P.-A. and Lagrange, M.  Efficient bandwidth extension of musical signals using a differentiable harmonic plus noise model.  _EURASIP Journal on Audio, Speech, and Music Processing_ , 2023(51), 2023.
  * Hawthorne et al. (2018) Hawthorne, C., Elsen, E., Song, J., Roberts, A., Simon, I., Raffel, C., Engel, J., Oore, S., and Eck, D.  Onsets and frames: Dual-objective piano transcription.  In _Proceedings of ISMIR_ , 2018.
  * Hawthorne et al. (2019) Hawthorne, C., Stasyuk, A., Roberts, A., Simon, I., Huang, C.-Z. A., Dieleman, S., Elsen, E., Engel, J., and Eck, D.  Enabling factorized piano music modeling and generation with the MAESTRO dataset.  In _Proceedings of ICLR_ , 2019.
  * Hawthorne et al. (2021) Hawthorne, C., Simon, I., Swavely, R., Manilow, E., and Engel, J.  Sequence-to-sequence piano transcription with transformers.  In _Proceedings of ISMIR_ , 2021.
  * Hayes et al. (2023) Hayes, B., Saitis, C., and Fazekas, G.  Sinusoidal frequency estimation by gradient descent.  In _Proceedings of ICASSP_ , 2023.
  * Henkel et al. (2024) Henkel, F., Kim, J., McCallum, M. C., Sandberg, S. E., and Davies, M. E.  Tempo estimation as fully self-supervised binary classification.  In _Proceedings of ICASSP_ , 2024.
  * Hsu & Jang (2010) Hsu, C.-L. and Jang, J.-S. R.  On the improvement of singing voice separation for monaural recordings using the MIR-1K dataset.  _IEEE Transactions on Audio, Speech, and Language Processing (TASLP)_ , 18(2):310–319, 2010.
  * Hung et al. (2021) Hung, Y.-N., Wichern, G., and Le Roux, J.  Transcription is all you need: Learning to separate musical mixtures with score as supervision.  In _Proceedings of ICASSP_ , 2021.
  * Kameoka et al. (2007) Kameoka, H., Nishimoto, T., and Sagayama, S.  A multipitch analyzer based on harmonic temporal structured clustering.  _IEEE Transactions on Audio, Speech, and Language Processing (TASLP)_ , 15(3):982–994, 2007.
  * Kim et al. (2018) Kim, J. W., Salamon, J., Li, P., and Bello, J. P.  Crepe: A convolutional representation for pitch estimation.  In _Proceedings of ICASSP_ , 2018.
  * Klapuri (1998) Klapuri, A.  Number theoretical means of resolving a mixture of several harmonic sounds.  In _Proceedings of EUSIPCO_ , 1998.
  * Klapuri (2003) Klapuri, A.  Multiple fundamental frequency estimation based on harmonicity and spectral smoothness.  _IEEE Transactions on Speech and Audio Processing_ , 11(6):804–816, 2003.
  * Klapuri (2006) Klapuri, A.  Multiple fundamental frequency estimation by summing harmonic amplitudes.  In _Proceedings of ISMIR_ , 2006.
  * Klapuri (2008) Klapuri, A.  Multipitch analysis of polyphonic music and speech signals using an auditory model.  _IEEE Transactions on Audio, Speech, and Language Processing (TASLP)_ , 16(2):255–266, 2008.
  * Klapuri & Davy (2006) Klapuri, A. and Davy, M.  _Signal Processing Methods for Music Transcription_.  Springer, 2006.
  * Krause et al. (2023a) Krause, M., Strahl, S., and Müller, M.  Weakly supervised multi-pitch estimation using cross-version alignment.  In _Proceedings of ISMIR_ , 2023a.
  * Krause et al. (2023b) Krause, M., Weiß, C., and Müller, M.  Soft dynamic time warping for multi-pitch estimation and beyond.  In _Proceedings of ICASSP_ , 2023b.
  * Lattner et al. (2019) Lattner, S., Dörfler, M., and Arzt, A.  Learning complex basis functions for invariant representations of audio.  In _Proceedings of ISMIR_ , 2019.
  * Li et al. (2018) Li, B., Liu, X., Dinesh, K., Duan, Z., and Sharma, G.  Creating a multitrack classical music performance dataset for multimodal music analysis: Challenges, insights, and applications.  _IEEE Transactions on Multimedia_ , 21:522–535, 2018.
  * Li et al. (2022) Li, X., Sun, Y., Wu, X., and Chen, J.  Multi-speaker pitch tracking via embodied self-supervised learning.  In _Proceedings of ICASSP_ , 2022.
  * Li et al. (2023) Li, Y., Yuan, R., Zhang, G., Ma, Y., Chen, X., Yin, H., Lin, C., Ragni, A., Benetos, E., Gyenge, N., et al.  MERT: Acoustic music understanding model with large-scale self-supervised training.  _arXiv preprint arXiv:2306.00107_ , 2023.
  * Loshchilov & Hutter (2019) Loshchilov, I. and Hutter, F.  Decoupled weight decay regularization.  In _Proceedings of ICLR_ , 2019.
  * Lu et al. (2023) Lu, W.-T., Wang, J.-C., and Hung, Y.-N.  Multitrack music transcription with a time-frequency perceiver.  In _Proceedings of ICASSP_ , 2023.
  * Maman & Bermano (2022) Maman, B. and Bermano, A. H.  Unaligned supervision for automatic music transcription in the wild.  In _Proceedings of ICML_ , 2022.
  * Manilow et al. (2019) Manilow, E., Wichern, G., Seetharaman, P., and Le Roux, J.  Cutting music source separation some Slakh: A dataset to study the impact of training data quality and quantity.  In _Proceedings of WASPAA_ , 2019.
  * Manilow et al. (2020) Manilow, E., Seetharaman, P., and Pardo, B.  Simultaneous separation and transcription of mixtures with multiple polyphonic and percussive instruments.  In _Proceedings of ICASSP_ , 2020.
  * Mauch & Dixon (2014) Mauch, M. and Dixon, S.  PYIN: A fundamental frequency estimator using probabilistic threshold distributions.  In _Proceedings of ICASSP_ , 2014.
  * McCallum et al. (2022) McCallum, M. C., Korzeniowski, F., Oramas, S., Gouyon, F., and Ehmann, A. F.  Supervised and unsupervised learning of audio representations for music understanding.  In _Proceedings of ISMIR_ , 2022.
  * McCallum et al. (2024) McCallum, M. C., Davies, M. E., Henkel, F., Kim, J., and Sandberg, S. E.  On the effect of data-augmentation on local embedding properties in the contrastive learning of music audio representations.  In _Proceedings of ICASSP_ , 2024.
  * Morrison et al. (2023) Morrison, M., Hsieh, C., Pruyne, N., and Pardo, B.  Cross-domain neural pitch and periodicity estimation.  _arXiv preprint arXiv:2301.12258_ , 2023.
  * Müller (2015) Müller, M.  _Fundamentals of Music Processing: Audio, Analysis, Algorithms, Applications_.  Springer, 2015.
  * Ostermann et al. (2023) Ostermann, F., Vatolkin, I., and Ebeling, M.  AAM: A dataset of artificial audio multitracks for diverse music information retrieval tasks.  _EURASIP Journal on Audio, Speech, and Music Processing_ , 2023(13), 2023.
  * Pauwels et al. (2019) Pauwels, J., O’Hanlon, K., Gómez, E., Sandler, M., et al.  20 years of automatic chord recognition from audio.  In _Proceedings of ISMIR_ , 2019.
  * Pedersoli et al. (2020) Pedersoli, F., Tzanetakis, G., and Yi, K. M.  Improving music transcription by pre-stacking a U-Net.  In _Proceedings of ICASSP_ , 2020.
  * Raffel et al. (2014) Raffel, C., McFee, B., Humphrey, E. J., Salamon, J., Nieto, O., Liang, D., and Ellis, D. P.  mir_eval: A transparent implementation of common MIR metrics.  In _Proceedings of ISMIR_ , 2014.
  * Renault et al. (2023) Renault, L., Mignot, R., and Roebel, A.  DDSP-Piano: a neural sound synthesizer informed by instrument knowledge.  _Journal of the Audio Engineering Society (AES)_ , 2023.
  * Riou et al. (2023) Riou, A., Lattner, S., Hadjeres, G., and Peeters, G.  PESTO: Pitch estimation with self-supervised transposition-equivariant objective.  In _Proceedings of ISMIR_ , 2023.
  * Saeed et al. (2021) Saeed, A., Grangier, D., and Zeghidour, N.  Contrastive learning of general-purpose audio representations.  In _Proceedings of ICASSP_ , 2021.
  * Salamon et al. (2017) Salamon, J., Bittner, R. M., Bonada, J., Bosch, J. J., Gómez Gutiérrez, E., and Bello, J. P.  An analysis/synthesis framework for automatic F0 annotation of multitrack datasets.  In _Proceedings of ISMIR_ , 2017.
  * Schlüter & Grill (2015) Schlüter, J. and Grill, T.  Exploring data augmentation for improved singing voice detection with neural networks.  In _Proceedings of ISMIR_ , 2015.
  * Schörkhuber & Klapuri (2010) Schörkhuber, C. and Klapuri, A.  Constant-Q transform toolbox for music processing.  In _Proceedings of SMC_ , 2010.
  * Schörkhuber et al. (2014) Schörkhuber, C., Klapuri, A., Holighaus, N., and Dörfler, M.  A matlab toolbox for efficient perfect reconstruction time-frequency transforms with log-frequency resolution.  In _Proceedings of AES_ , 2014.
  * Schulze-Forster et al. (2023) Schulze-Forster, K., Richard, G., Kelley, L., Doire, C. S., and Badeau, R.  Unsupervised music source separation using differentiable parametric source models.  _IEEE/ACM Transactions on Audio, Speech, and Language Processing (TASLP)_ , 31:1276–1289, 2023.
  * Sigtia et al. (2016) Sigtia, S., Benetos, E., and Dixon, S.  An end-to-end neural network for polyphonic piano music transcription.  _IEEE/ACM Transactions on Audio, Speech, and Language Processing (TASLP)_ , 24(5):927–939, 2016.
  * Simon et al. (2022) Simon, I., Gardner, J., Hawthorne, C., Manilow, E., and Engel, J.  Scaling polyphonic transcription with mixtures of monophonic transcriptions.  In _Proceedings of ISMIR_ , 2022.
  * Spijkervet & Burgoyne (2021) Spijkervet, J. and Burgoyne, J. A.  Contrastive learning of musical representations.  In _Proceedings of ISMIR_ , 2021.
  * Su & Yang (2015) Su, L. and Yang, Y.-H.  Escaping from the abyss of manual annotation: New methodology of building polyphonic datasets for automatic music transcription.  In _Proceedings of CMMR_ , 2015.
  * Tagliasacchi et al. (2020) Tagliasacchi, M., Gfeller, B., de Chaumont Quitry, F., and Roblek, D.  Pre-training audio representations with self-supervision.  _IEEE Signal Processing Letters_ , 27:600–604, 2020.
  * Tanaka et al. (2020) Tanaka, K., Nakatsuka, T., Nishikimi, R., Yoshii, K., and Morishima, S.  Multi-instrument music transcription based on deep spherical clustering of spectrograms and pitchgrams.  In _Proceedings of ISMIR_ , 2020.
  * Torres et al. (2023) Torres, B., Lattner, S., and Richard, G.  Singer identity representation learning using self-supervised techniques.  In _Proceedings of ISMIR_ , 2023.
  * Torres et al. (2024) Torres, B., Peeters, G., and Richard, G.  Unsupervised harmonic parameter estimation using differentiable DSP and spectral optimal transport.  In _Proceedings of ICASSP_ , 2024.
  * Toyama et al. (2023) Toyama, K., Akama, T., Ikemiya, Y., Takida, Y., Liao, W.-H., and Mitsufuji, Y.  Automatic piano transcription with hierarchical frequency-time transformer.  In _Proceedings of ISMIR_ , 2023.
  * Wang et al. (2020a) Wang, X., Liu, L., and Shi, Q.  Enhancing piano transcription by dilated convolution.  In _Proceedings of ICMLA_ , 2020a.
  * Wang et al. (2020b) Wang, X., Liu, L., and Shi, Q.  Harmonic structure-based neural network model for music pitch detection.  In _Proceedings of ICMLA_ , 2020b.
  * Wei et al. (2022a) Wei, W., Li, P., Yu, Y., and Li, W.  HarmoF0: Logarithmic scale dilated convolution for pitch estimation.  In _Proceedings of ICME_ , 2022a.
  * Wei et al. (2022b) Wei, W., Li, P., Yu, Y., and Li, W.  HPPNet: Modeling the harmonic structure and pitch invariance in piano transcription.  In _Proceedings of ISMIR_ , 2022b.
  * Weiß & Peeters (2021) Weiß, C. and Peeters, G.  Learning multi-pitch estimation from weakly aligned score-audio pairs using a multi-label CTC loss.  In _Proceedings of WASPAA_ , 2021.
  * Weiß & Peeters (2022) Weiß, C. and Peeters, G.  Comparing deep models and evaluation strategies for multi-pitch estimation in music recordings.  _IEEE/ACM Transactions on Audio, Speech, and Language Processing (TASLP)_ , 30:2814–2827, 2022.
  * Wu et al. (2023) Wu, Y., Zhao, J., Yu, Y., and Li, W.  MFAE: Masked frame-level autoencoder with hybrid-supervision for low-resource music transcription.  In _Proceedings of ICME_ , 2023.
  * Wu et al. (2020) Wu, Y.-T., Chen, B., and Su, L.  Multi-instrument automatic music transcription with self-attention-based instance segmentation.  _IEEE/ACM Transactions on Audio, Speech, and Language Processing (TASLP)_ , 28:2796–2809, 2020.
  * Xi et al. (2018) Xi, Q., Bittner, R. M., Pauwels, J., Ye, X., and Bello, J. P.  GuitarSet: A dataset for guitar transcription.  In _Proceedings of ISMIR_ , 2018.
  * Yan et al. (2021) Yan, Y., Cwitkowitz, F., and Duan, Z.  Skipping the frame-level: Event-based piano transcription with neural semi-CRFs.  In _Proceedings of NeurIPS_ , 2021.
  * Yeh et al. (2010) Yeh, C., Roebel, A., and Rodet, X.  Multiple fundamental frequency estimation and polyphony inference of polyphonic music signals.  _IEEE Transactions on Audio, Speech, and Language Processing (TASLP)_ , 18(6):1116–1126, 2010.
  * Zeghidour et al. (2021) Zeghidour, N., Luebs, A., Omran, A., Skoglund, J., and Tagliasacchi, M.  SoundStream: An end-to-end neural audio codec.  _IEEE/ACM Transactions on Audio, Speech, and Language Processing (TASLP)_ , 30:495–507, 2021.
  * Zhang et al. (2020) Zhang, Z., Wang, Y., Gan, C., Wu, J., Tenenbaum, J. B., Torralba, A., and Freeman, W. T.  Deep audio priors emerge from harmonic convolutional networks.  In _Proceedings of ICLR_ , 2020.
  * Zhou et al. (2009) Zhou, R., Reiss, J. D., Mattavelli, M., and Zoia, G.  A computationally efficient method for polyphonic pitch estimation.  _EURASIP Journal on Advances in Signal Processing_ , 2009, 2009.

##  Appendix A Continuous Binary Cross-Entropy

Figure 6: Positive activation (left), negative activation (center), and
standard (right) BCE loss over continuous-valued targets and estimates.

##  Appendix B Validation Loss Curves

Figure 7: Validation loss computed at checkpoints of SS-MPE and each ablation.

##  Appendix C Additional Inference Examples

Figure 8: Inference for track 03_Dance_fl_cl of URMP (Li et al., 2018). Figure
9: Inference for track PQ03_Farranc of Su (Su & Yang, 2015). Figure 10:
Inference for track take_five of TRIOS (Fritsch, 2012). Figure 11: Inference
for track 00_Funk3-98-A_solo of GuitarSet (Xi et al., 2018).


usvardate [] DAY, 0.85

# Theoretical Study of Kr Hollow Atoms under X-ray Free-Electron
Laser††thanks: Project supported by the Fundamental Research Funds for the
Central Universities (Grant No. 10822041A2038).

Jiaxin Ye Institute of Atomic and Molecular Physics, Sichuan University,
Chengdu 610065, China  Key Laboratory of High Energy Density Physics and
Technology, Ministry of Education, Chengdu 610065, China  Chen Yang
Corresponding author. E-mail<EMAIL_ADDRESS>Institute of Atomic and
Molecular Physics, Sichuan University, Chengdu 610065, China  Key Laboratory
of High Energy Density Physics and Technology, Ministry of Education, Chengdu
610065, China  Gang Jiang Corresponding author. E-mail<EMAIL_ADDRESS>Institute of Atomic and Molecular Physics, Sichuan University, Chengdu 610065,
China  Key Laboratory of High Energy Density Physics and Technology, Ministry
of Education, Chengdu 610065, China

###### Abstract

By calculating the correlation between the total photoionization cross-section
of the ground state of the Kr atom and photon energy, three particular photon
energies close to the near inner orbital energy of 1.75 keV, 1.90 keV, and
14.30 keV are determined in this work. The dynamical simulation under 17.50
keV photon energy in the experimental conditions is achieved by implementing
the Monte Carlo method and optimizing the photon flux modeling parameters. As
a result, our calculated data are more consistent with the experimental
phenomena. The light saturation phenomenon of Kr at 1.75 keV, 1.90 keV, 14.30
keV, and 17.50 keV energies is further calculated and researched using the
optimized photon flux model theory. We statistically compare the main
ionization paths under those four specific photon energies and calculate the
population changes of various hollow atoms. The results demonstrate that the
ratio of hollow atoms produced at the critical ionization photon energy is
high. Furthermore, the change of position is smooth, showing the significant
difference between the generation mode of ions with low photon energy and
those with high photon energy, which has important reference significance for
studying hollow atoms with medium and high charge states.

Keywords: X-ray free-electron lasers, krypton, charge-state population, Monte
Carlo simulation

PACS: 32.80.Hd, 32.80.Aa

##  1 Introduction

X-ray free-electron laser (XFEL) is known as the ”fourth-generation light
source.”[1] XFEL opens up a novel window in exploring substances at the atomic
and molecular scales of space and time, photographing chemical processes
within a few milliseconds or less, and revealing the structure and dynamics of
complex molecular systems.[2, 3] In recent years, free-electron laser
measurements of protein structures have become an increasingly valuable tool
in biology,[4, 5]. At the same time, coherent X-ray diffraction imaging
provides images of small structures, which is a major focus of crystallography
and protein research.[6] It is of great significance to investigate the
dynamical evolution of atoms under a laser in order to assess the radiation
damage phenomenon in sample structure detection using free-electron lasers.[7,
8, 9]

In experiments, X-ray photoionization can produce light transparency in
aluminum.[10] When an atom interacts with the XFEL, the inner electrons are
ionized, resulting in light saturation, reducing the probability of light-
induced damage, and the diffraction image obtained achieves the best
quality.[11, 12, 13, 14] Due to the limitation of photon energy generated by a
laser and the phenomenon of hollow atoms in medium and high Z element atoms
have the high decay probability, the light saturation phenomenon is not
apparent, the research on medium and high Z element hollow atoms is relatively
scarce. Nevertheless, if the photon energy is close to the ionization orbital
energy, when the orbital ionization energy increases and exceeds the photon
energy after the inner electron is ionized, it is possible to make the
phenomenon of light saturation evident and durable, which makes the practical
application of hollow atoms in the characterization of material structure
possible.

The photon energy close to the orbital energy is called the critical
ionization photon energy (CIPE). Consequently, we extend the study of hollow
atoms to Kr and examine the phenomenon of hollow atoms at photon energies
close to their orbital energy. In this work, we performed a genetic algorithm
(GA) to determine laser parameters in the model and simulate the dynamical
evolution of the Kr atom under three CIPEs by theoretical calculation.

##  2 Theoretical method

###  2.1 Theoretical model

Our calculation of the photoionization cross-section is based on the _ab
initio_ calculation framework of non-relativistic quantum electrodynamics and
perturbation theory, using the Hartree-Fock-Slater model.[15] The cross-
section for photoionization of an electron in the $i$th occupied spin orbital
follows as

$\sigma_{i}=\frac{4\pi^{2}}{\omega_{\mathrm{in}}}\alpha\sum_{f}\left|\left\langle\varphi_{f}\left|e^{\mathrm{i}\boldsymbol{k}_{\mathrm{in}}\cdot\boldsymbol{x}}\epsilon_{\boldsymbol{k}_{\mathrm{in}},\lambda_{\mathrm{in}}}\cdot\frac{\nabla}{\mathrm{i}}\right|\varphi_{i}\right\rangle\right|^{2}\delta\left(\varepsilon_{f}-\varepsilon_{i}-\omega_{\mathrm{in}}\right),$
(1)

where $\omega_{\text{in}}$ is the incident photon energy, $\alpha$ is the fine
structure constant, $\boldsymbol{x}$ is the position,
$\epsilon_{\boldsymbol{k}_{\mathrm{in}},\lambda_{\mathrm{in}}}$ is the
polarization vector under the wave vector $\boldsymbol{k}$ and wavelength
$\lambda$, $\varepsilon_{f}$ is the energy of unoccupied orbital,
$\varepsilon_{i}$ is the energy of occupied orbital, $\varphi_{f}$ is the sum
of unoccupied orbital wave functions, and $\varphi_{i}$ is the sum of occupied
orbital wave functions.

###  2.2 Photon flux modeling

Since the calculation of photoionization probability depends on photon flux,
the modeling of photon flux is crucial for us to fit the theoretical
calculation results. In the actual experiment of free-electron laser, the
distribution of light intensity meets not only the Gaussian distribution of
time, but also the Gaussian distribution of position and photon frequency. We
assume that the beam is a circular spot, and the general formula of light
intensity satisfies the Gaussian distribution of space, time, photon
frequency,[16, 17, 18]

$I_{\nu}(r,t)=I_{0}e^{-\ln 2a\left(\frac{r_{1}}{\Delta}\right)^{2}}e^{-\ln
2\left(\frac{t}{T}\right)^{2}}\sqrt{\frac{\ln 2}{\pi\Gamma^{2}}}e^{-\ln
2\left(\frac{h\nu-h\nu_{0}}{\Gamma}\right)^{2}}.$ (2)

For convenience, we define the relative radius position $r$ as

$r=\frac{r_{1}}{\Delta},$ (3)

where $t$ is the time, $r_{1}$ is the relative center position, $\nu$ is the
photon frequency, $\Delta$ is the radius, $\tau$ is the pulse half-width,
$\Gamma$ is the half-width at the half-maximum (HWHM) of the photon frequency
bandwidth, and $h\nu_{0}$ is the central photon energy. The exponential
coefficient $a$ and the peak flux $I_{0}$ are the laser parameters to be
calibrated. The photoionization rate can be obtained by integrating the photon
frequency bandwidth,

$R_{ij}(r,t)=\int\frac{I_{\nu}(r,t)}{h\nu}\sigma_{ij}(h\nu)\mathrm{d}\nu.$ (4)

We can get the function of the photoionization rate by integrating the photon
frequency. The energy of the X-ray pulse can be obtained by integrating the
intensity of space, time, and photon energy. We use a genetic algorithm to
calibrate the peak flux and exponential coefficient of Kr atom at 17.50 keV
photon energy, in which the peak flux is $1.61\text{\times}{10}^{12}$
photons/$\mu$m2, the index coefficient is 1.85, so that the calculated value
can better match the experimental value. For the parameters obtained by the
genetic algorithm, a theoretical calculation of 1.75 keV, 1.90 keV, and 14.30
keV make our theoretical calculation results more reliable.

###  2.3 Numerical method

The transition between electronic configurations can be well described by
establishing the relevant rate equation between configurations. The equation
form is as follows,

$\frac{\mathrm{d}n_{i}}{\mathrm{d}t}=\sum_{j\neq
i}^{N}n_{j}R_{ji}-n_{i}\sum_{j\neq i}^{N}R_{ij},$ (5)

when solving the ionization dynamics of Kr atom at each photon energy, we use
the Monte Carlo method, where $R_{ij}$ is the rate from state $i$ to state
$j$, and $n_{i}$ is the ratio of the total number of events at energy level
$i$ to the total number of events $N$. The total value of events depends on
the parameters set by the Monte Carlo method. This method is often used to
solve the ionization dynamics of atoms. It can give a better evolutionary
relationship of atomic ion population when processing massive data. Moreover,
when the electron number increases, the Monte Carlo method has been proved to
be one of the most practical methods.[19, 20] We ran 106 times to converge the
results. We used the XATOM toolkit[14] to calculate the photoionization cross-
section, Auger decay, and fluorescence decay data of atoms. We calculated the
photoionization cross-section, Auger rate, and fluorescence rate of Kr at 1.75
keV, 1.90 keV, 14.30 keV, and 17.50 keV[21, 22] by using our own developed
Monte Carlo program. A total of $8.19\text{\times}{10}^{4}$ configurations and
$7.45\text{\times}{10}^{7}$ rate equations.

##  3 Results and discussion

###  3.1 Ionization kinetics of Kr in 17.50 keV

The time-dependent population obtained by the Monte Carlo method is about
spatial distribution. Therefore, we integrate the population change of each
ion over space to obtain its time-dependent population change,

$n_{i}(t)=\int_{0}^{1}n_{i}(t,r)\mathrm{d}r,$ (6)

where $i=1,2,3\ldots 10$ is the population of the $i$th ion whose population
changes with time. We made a statistical analysis of the calculation results
of the Monte Carlo method and received the time-dependent change of the
population of ions. The theoretical calculation results at different positions
are integrated over time and space, and the population is obtained from each
charge state compared with the experimental values.

Figure 1: The population of Kr ions varies with time. Pulse width is 30 fs.
Figure 2: Charge state distribution of Kr at 17.50 keV photon energy compared
with experimental data[21] and another theoretical result.[22]

In comparison, our calculated value performed slightly worse in Kr7+, and the
other charge states with experimental values were better than the others.[22]
When compared with the experimental values, there were some differences
between the Kr1+, Kr2+, and Kr4+. The differences are possibly due to the
relatively few physical processes considered, such as the addition of single
photon multi-ionization, double Auger decay, and resonant excitation to the
parameter fitting,[17, 23] which can provide better agreement to the
experimental results.

###  3.2 Selection of CIPEs

Figure 3: Total photoionization cross-section of Kr to photon energy
(1.70–30.00 keV).

From Eq. 1, we presented the relationship between photoionization cross-
section and photon energy. Fig. 3 indicates that the photoionization cross-
section decreases with increasing photon energy. The total photoionization
cross-section changes rapidly at photon energies near 1.70 keV, 1.87 keV, and
14.00 keV. Thus, for the later simulation, we took the photon energies nearby
as 1.75 keV, 1.90 keV, and 14.30 keV. When the photon energy approaches the
ionization energy of each orbital, there will be an increase in the total
ionization cross-section. This increment is because the ionization rate will
increase when the photon energy is close to or more significant than the
orbital energy for single-photon ionization. In XFEL with high photon energy,
this assumption is reasonable. We call an atom a hollow-like atom (HLA) that
ionizes an electron from the electron orbital with the ground state’s most
significant non-inner photoionization cross-section. Table 1 shows that hollow
atoms and HLAs at three CIPEs and one experimental photon energy are compared.

Table 1: The total photoionization cross-section includes the sum of 8 photoionization cross-section channels of $1s$–$4p$. The total rate includes 82 decay modes, 15 fluorescence decay rates and 67 Auger rates. Photon energy | Configuration | Total ionization cross-section (Mb) | Total rates (au)
---|---|---|---
1.75 keV | $1s^{2}2s^{2}2p^{6}$ | 0.59675394 | 0.00000000
| $1s^{2}2s^{2}2p^{5}$ | 0.11423523 | 0.04509832
1.90 keV | $1s^{2}2s^{2}2p^{6}$ | 0.56141388 | 0.00000000
| $1s^{2}2s^{1}2p^{6}$ | 0.50994368 | 0.03101831
14.30 keV | $1s^{2}2s^{2}2p^{6}$ | 0.01822358 | 0.00000000
| $1s^{1}2s^{2}2p^{6}$ | 0.00264105 | 0.10159872
17.50 keV | $1s^{2}2s^{2}2p^{6}$ | 0.01072040 | 0.00000000
| $1s^{1}2s^{2}2p^{6}$ | 0.00628324 | 0.10159872

At 1.75 keV, the total photoionization cross-section decayed from 0.60 Mb to
0.11 Mb before and after ionization, and 14.30 keV decayed from 0.018 Mb to
0.003 Mb. However, the total ionization cross-section at 1.90 keV did not
decay as expected because the ionization energy of $2s$ orbital was similar to
that of $2p$ orbital. After ionization of $2s$ orbital electrons, the
ionization energy of $2p$ orbital increased and approached 1.90 keV, resulting
in the rise of the total ionization cross-section, which did not reach the
expected attenuation value. 17.50 keV is a typical experimental parameter of
photon energy from 1.7 keV to 30 keV. After the electron ionizes the inner
orbital, the time and output of hollow atoms are less than the CIPE. It also
shows that the light saturation effect of general photon energy is much lower
than the CIPEs.

###  3.3 Temporal and spatial distribution of Kr hollow atoms

The generation of hollow atoms depends on time and photon flux,[11, 12]
however, in the actual process, we need to consider the distribution of
spatial position as well. This paper assumes that the laser is a Gaussian
beam; the incident photon beam satisfies the Gaussian distribution in spatial
position. Through different position parameters, three-dimensional images of
spatial and temporal populations of hollow atoms of 1.75 keV, 1.90 keV, 14.30
keV, and 17.50 keV are obtained, which shows the effect on hollow atoms when
the position parameters change.

(a) 1.75 keV

(b) 1.90 keV

(c) 14.30 keV

(d) 17.50 keV

Figure 4: Distribution diagram of hollow atoms and space. Time is from 0–5 fs,
and position parameter is the relative radius position from 0–1 (100%).

From Fig. 4, we determined that 1.75 keV, 1.90 keV, 14.30 keV, and 17.50 keV
reach the maximum average population of hollow atoms at 67%, 35%, 88%, and 26%
percentage of the radius from the center point. Due to the characteristics of
Gaussian distribution, with the distance from the center, the photon flux
decreases, the number of hollow atoms decreases, and the change of population
slow down. The maximum values of three-dimensional images are 40.5%, 33.3%,
2.9%, and 2.4% showing that the hollow atom effect produced by the CIPE of
$1s$ and $2s$ is higher than the CIPE of 17.50 keV and $2p$, which is
consistent with our prediction. By spatially integrating the image data, a
more intuitive change of hollow atoms of Gaussian beams can be obtained.

Figure 5: Time-dependent population change of hollow(-like) atoms. 1.75 keV
and 1.90 keV are HLAs.

Fig. 5 shows that the populations of the two CIPEs of 1.75 keV and 14.30 keV
are much higher than those of 1.90 keV and 17.50 keV photon energies, which
reach the maximum at 0.6 fs and 0.7 fs. In the practical application of hollow
atoms, the photon energy close to the ionization of the $s$ orbital can be
selected to make the diffraction imaging of material structure detection
clearer.

###  3.4 Ionization path diagram of Kr atom

In over $2\text{\times}{10}^{4}$ configurations, studying the comparison of
main ionization paths under each photon energy can help better understand the
direction of hollow atoms. Therefore, we count and plot the main ionization
paths at the four specific photon energies.

Figure 6: Diagram of the dominant photon energy ionization path.
Configurations not shown and not mentioned are full-occupied electron shells;
configurations not shown but mentioned inherit the configuration from the
previous step. The charge state is shown in parentheses on the right side.

Fig. 6 demonstrates that under the photon energy of 1.75 keV and 1.90 keV,
single-photon photoionization and Auger decay account for all channels of ion
generation, and fluorescence decay does not occur. At 14.30 keV, the ion
generation mainly depends on the ion generation mode of inner photoionization
and Auger decay. At 17.50 keV, the inner photoionization cross-section
decreases, and the generation of ions mainly depends on the ion generation
mode of inner photoionization plus fluorescence decay. The photon energy is
close to the orbital energy at low photon energy, and the photoionization
cross-section is large. Thus there will be no fluorescence decay in the
primary ionization process.

On the other hand, the photon energy is no longer close to the orbital energy
at high photon energy, resulting in fluorescence decay after inner
photoionization. For HLAs, 1.75 keV is no longer ionized after ionizing $2p$
orbital electrons, and the main channel is Auger decay; therefore, the
population of HLAs is high. For 1.90 keV, after ionizing the $2s$ orbital
electrons, the photon energy is close to the $2p$ orbital energy. Then the
$2p$ orbital ionization is mainly carried out, which is also the poor effect
of producing hollow atoms in the CIPE. For hollow atoms, at 14.30 keV, because
the photon energy is close to the inner orbital energy, and the ground state
$1s$ photoionization cross-section and fluorescence decay ratio after
ionization are higher than 17.50 keV at 14.30 keV, the hollow atom effect
produced at 14.30 keV is more significant than 17.50 keV.

##  4 Conclusion

We used the Hartree-Fock-Slater model to calculate the total photoionization
cross-section of the Kr atom in the range of 1.70–30.00 keV, calculated the
ionization energy of each orbital of Kr, and finally determined three CIPEs of
1.75 keV, 1.90 keV, and 14.30 keV. We used the genetic algorithm to calibrate
the parameters of the photon flux model in the results of calculating 17.50
keV photon energy Kr by the Monte Carlo method and calculated the dynamical
evolution of hollow atoms of 1.75 keV, 1.90 keV, 14.30 keV, and 17.50 keV. The
conclusion indicates that the number of hollow atoms of 1.75 keV and 14.30 keV
is much higher than that of the other two hollow atoms with ionization energy,
which provides an idea for selecting photon energy in the subsequent
application of hollow atoms. The main ionization paths of Kr atoms at photon
energies of 1.75 keV, 1.90 keV, 14.30 keV, and 17.50 keV were studied. We
suggest that the main ionization paths rarely produce fluorescence decay at
low photon energy. In contrast, the generation of ions at high photon energy
depends on the ionization mode of inner ionization and fluorescence decay,
which provides an idea for understanding the ionization mode of high Z atoms
at different photon energies.

## Acknowledgment

The authors would like to thank all the people who have been helping us in the
theoretical calculation; particularly, our thanks go to Prof. Robin Santra,
Mr. Yongjun Li, and Prof. Jianmin Yuan.

## References

  * [1] Pellegrini C, Marinelli A and Reiche S 2016 _Rev. Mod. Phys._ 88 015006
  * [2] Sobolev E, Zolotarev S, Giewekemeyer K _et al._ 2020 _Commun. Phys._ 3 97
  * [3] Lehmkühler F, Dallari F, Jain A, Sikorski M, Möller J, Frenzel L, Lokteva I, Mills G, Walther M, Sinn H, Schulz F, Dartsch M, Markmann V, Bean R, Kim Y, Vagovic P, Madsen A, Mancuso A P and Grübel G 2020 _Proc. Natl. Acad. Sci. U.S.A._ 117 24110
  * [4] Pellegrini C 2012 _EPJ H_ 37 659
  * [5] Caleman C, Tîmneanu N, Martin A V, Jönsson H O, Aquila A, Barty A, Scott H A, White T A and Chapman H N 2015 _Opt. Express_ 23 1213
  * [6] Kobayashi A, Sekiguchi Y, Takayama Y, Oroguchi T and Nakasako M 2014 _Opt. Express_ 22 27892
  * [7] Ho P J, Kanter E P and Young L 2015 _Phys. Rev. A_ 92 063430
  * [8] Howells M, Beetz T, Chapman H, Cui C, Holton J, Jacobsen C, Kirz J, Lima E, Marchesini S, Miao H, Sayre D, Shapiro D, Spence J and Starodub D 2009 _J. Electron Spectros. Relat. Phenomena_ 170 4
  * [9] Doumy G, Roedig C, Son S K _et al._ 2011 _Phys. Rev. Lett._ 106 083002
  * [10] Nagler B, Zastrau U, Fäustlin R R _et al._ 2009 _Nature Phys._ 5 693
  * [11] Deng P and Jiang G 2019 _Chinese Phys. B_ 28 063203
  * [12] Feng L and Jiang G 2017 _Acta Phys. Sin._ 66 153201 (in Chinese)
  * [13] Yoneda H, Inubushi Y, Yabashi M, Katayama T, Ishikawa T, Ohashi H, Yumoto H, Yamauchi K, Mimura H and Kitamura H 2014 _Nat. Commun._ 5 5080
  * [14] Son S K, Young L and Santra R 2011 _Phys. Rev. A_ 83 033402
  * [15] Santra R 2009 _J. Phys. B: At. Mol. Opt. Phys._ 42 169801
  * [16] Toyota K, Jurek Z, Son S K, Fukuzawa H, Ueda K, Berrah N, Rudek B, Rolles D, Rudenko A and Santra R 2019 _J. Synchrotron Rad._ 26 1017
  * [17] Li Y, Gao C, Zeng J and Yuan J 2022 _J. Phys. B: At. Mol. Opt. Phys._ 55 025002
  * [18] Gao C, Zeng J and Yuan J 2016 _J. Phys. B: At. Mol. Opt. Phys._ 49 044001
  * [19] Son S K and Santra R 2012 _Phys. Rev. A_ 85 063415
  * [20] Rudek B, Son S K, Foucar L _et al._ 2012 _Nature Photon._ 6 858
  * [21] Krause M O and Carlson T A 1967 _Phys. Rev._ 158 18
  * [22] Chernov V E, Dorofeev D L, Elfimov S V, Zon B A, Gavrilov G E and Naryshkin Y G 2015 _Laser Phys. Lett._ 12 036002
  * [23] Ho P J, Bostedt C, Schorb S and Young L 2014 _Phys. Rev. Lett._ 113 253001