# A self-averaging spectral form factor implies unitarity breaking

Apollonas S. Matsoukas-Roubeas  Department of Physics and Materials Science,
University of Luxembourg, L-1511 Luxembourg, G. D. Luxembourg  Mathieu Beau
Department of Physics, University of Massachusetts, Boston, MA 02125, USA  Lea
F. Santos  Department of Physics, University of Connecticut, Storrs,
Connecticut 06269, USA  Adolfo del Campo  Department of Physics and Materials
Science, University of Luxembourg, L-1511 Luxembourg, G. D. Luxembourg
Donostia International Physics Center, E-20018 San Sebastián, Spain

###### Abstract

The complex Fourier transform of the two-point correlator of the energy
spectrum of a quantum system is known as the spectral form factor (SFF). It
constitutes an essential diagnostic tool for phases of matter and quantum
chaos. In black hole physics, it describes the survival probability (fidelity)
of a thermofield double state under unitary time evolution. However, detailed
properties of the SFF of isolated quantum systems with generic spectra are
smeared out by large temporal fluctuations, whose minimization requires
disorder or time averages. This requirement holds for any system size, that
is, the SFF is non-self averaging. Exploiting the fidelity-based
interpretation of this quantity, we prove that using filters, disorder and
time averages of the SFF involve unitarity breaking, i.e., open quantum
dynamics described by a quantum channel that suppresses quantum noise.
Specifically, averaging over Hamiltonian ensembles, time averaging, and
frequency filters can be described by the class of mixed-unitary quantum
channels in which information loss can be recovered. Frequency filters are
associated with a time-continuous master equation generalizing energy
dephasing. We also discuss the use of eigenvalue filters. They are linked to
non-Hermitian Hamiltonian evolution without quantum jumps, whose long-time
behavior is described by a Hamiltonian deformation. We show that frequency and
energy filters make the SFF self-averaging at long times.

The spectral form factor (SFF) is an essential diagnostic tool in the
characterization of complex quantum systems [1, 2, 3, 4, 5, 6]. Given a
Hamiltonian $H$ of a single system with spectrum ${\rm
Sp}(H)=\\{E_{n}|n=1,\dots,d\\}$, the SFF is a real-valued function defined as

$\displaystyle{\rm SFF}(t)$ $\displaystyle=$
$\displaystyle\left|\frac{Z(\beta+it)}{Z(\beta)}\right|^{2}$ $\displaystyle=$
$\displaystyle\frac{1}{Z(\beta)^{2}}\sum_{n,m=1}^{d}e^{-\beta(E_{n}+E_{m})-it(E_{n}-E_{m})},$

where the partition function $Z(\beta)={\rm tr}[\exp(-\beta H)]$ is included
as a normalization such that ${\rm SFF}(0)=1$. Finite values of the inverse
temperature $\beta$ exponentially suppress the contribution from the excited
states. Thus, the Boltzmann factor $\exp(-\beta E_{n})$ acts as an (energy)
eigenvalue filter, where large values of $\beta$ preferentially sample the
low-energy part of the spectrum, and $\beta=0$ gives equal weight to the whole
spectrum.

The SFF admits several information theoretic interpretations. In particular,
it can be expressed as the fidelity [7, 8, 9, 10] between a coherent Gibbs
state $|\psi_{\beta}\rangle=\frac{1}{\sqrt{Z(\beta)}}\sum_{n}e^{-\beta
E_{n}/2}|n\rangle$ and its unitary time-evolution

$\displaystyle{\rm
SFF}(t)=\left|\langle\psi_{\beta}|e^{-itH}|\psi_{\beta}\rangle\right|^{2},$
(2)

or equivalently, as the survival probability of the evolving coherent Gibbs
state. Likewise, in bipartite systems, it is convenient to consider the
entangled state

$\displaystyle|{\rm TFD}\rangle=\frac{1}{\sqrt{Z(\beta)}}\sum_{n}e^{-\beta
E_{n}/2}|n\rangle\otimes|n\rangle,$ (3)

known as the thermofield double state (TFD). In terms of it, ${\rm
SFF}(t)=|\langle{\rm TFD}|e^{-itH\otimes\mathbb{I}}|{\rm TFD}\rangle|^{2}$.
The TFD is the purification of the thermal state of a single copy of the
system, obtained by doubling the Hilbert space. The TFD was first introduced
as a convenient reference state to extract thermal averages in field theory
[11]. The TFD dynamics was used early on to model the “hot” thermal vacuum
observed outside the horizon of a single radiating eternal black hole [12]. In
the context of the AdS/CFT correspondence, it describes an eternal two-sided
black hole in AdS [13, 14]. The SFF captures the survival probability of the
TFD state under unitary time evolution [7, 8, 9, 10]. The conjecture that back
holes are maximally chaotic [15] has led to a surge of activity in the study
of the dynamical manifestations of quantum chaos in the SFF [16, 17, 8, 18,
19, 20].

In theoretical and numerical studies, it is customary to average the SFF by
considering a Hamiltonian ensemble, e.g., in random-matrix theory or in
disordered systems. In such a scenario, a property is said to be self-
averaging when its estimate using a typical member of the ensemble and the
explicit average over the ensemble coincide. Self-averaging largely eases
numerical studies in many-body systems, disposing of the need for Hamiltonian
ensemble averages in characterizing the desirable property of the system.
However, the SFF is not self-averaging [21].

The structure of the SFF in the time domain is well-characterized [16, 17]: it
exhibits a slope-dip-ramp-plateau structure, as shown in Fig. 1a, that is
manifested under averaging over disorder or a Hamiltonian ensemble. In the
absence of averages, erratic time-domain fluctuations appear, making it
difficult to appreciate some of its features. An exception is the SFF computed
using the gauge-gravity duality in the semiclassical approximation, where the
erratic fluctuations are absent [22]. These fluctuations are sometimes
referred to as noise [21], or quantum noise [23], terminology to be
distinguished from the standard one in the theory of open quantum systems
[24]. Erratic wiggles in the time domain are a consequence of the discreteness
of the energy spectrum and can be associated with quantum coherence in the
energy eigenbasis in the time evolution of the coherent Gibbs state or the
TFD. Quantum noise is further responsible for the lack of self-averaging in
the SFF. Fluctuations with respect to the signal do not cancel out upon
averaging, e.g., over a Hamiltonian ensemble [1, 25, 26, 27, 21]. This can be
quantified by the finite value of the relative variance (RV)

$\displaystyle{\rm RV}(t)=\frac{\langle{\rm SFF}^{2}(t)\rangle-\langle{\rm
SFF}(t)\rangle^{2}}{\langle{\rm SFF}(t)\rangle^{2}}\;,$ (4)

that does not vanish as the size of the Hilbert space is increased.

The lack of self-averaging of the SFF and the survival probability for random
matrices and disordered spin models was analytically shown in [28, 29, 30].
This implies that no matter how large the system size is, an ensemble average
is required, adding an extra layer of complexity to numerical studies, which
are generally challenging due to the large Hilbert space involved in analyzing
many-body quantum systems. As an alternative to averaging over a Hamiltonian
ensemble, numerical and analytical studies often resort to running averages
over time that smear ${\rm SFF}(t)$ over intervals of time [1, 27]. A yet
different approach resorts to modifying the definition of the SFF restricting
the Fourier transform of the two-point function over an energy window, or more
generally, using a filter function over an energy or frequency band [31, 32,
33, 21, 18].

In what follows, we build on the interpretation of the SFF as a fidelity
between quantum states related by time evolution and show that suppressing the
erratic wiggles implies the breakdown of unitarity in the dynamics. To this
end, we reformulate as quantum operations described by a (nonunitary) quantum
channel, the different approaches to reduce the time fluctuations in the SFF,
such as ensemble averages, and to enforce self-averaging, such as filters in
the energy and frequency domain. For a particular class of filters, the
resulting channels are of the mixed-unitary class, and the information lost
due to the unitarity breaking can be recovered.

The paper is organized as follows. We review the structure of the unfiltered
SFF in Sec. I, and introduce the generalization of the SFF to arbitrary
physical processes in Sec. II, paving the way to the description of filtering
of the SFF in terms of nonunitary quantum channels in Sec. III. Physical
mechanisms associated with energy-dephasing and giving rise to different
spectral filters are discussed in Sec. IV. Section V discusses the filtered
SFF as a function of the system size, while Sec. VI focuses on information
recovery under mixed-unitary quantum channels and the frequency filter
deconvolution. Fundamental limits to quantum noise associated with the
fidelity-based SFF are presented in Sec. VII. The relation between eigenvalue
filters and Hermitian Hamiltonian deformations is discussed in Sec. VIII.
Time-continuous master equations for frequency filters are derived in Sec. IX
before closing with a discussion and conclusions.

##  I Features of the spectral form factor in an isolated chaotic quantum
system

We start by reviewing the well-known structure of the SFF for a chaotic system
in isolation. The (unfiltered) SFF averaged over a Hamiltonian ensemble can
generally be written down in terms of different contributions. Invoking the
annealed approximation, which replaces the average of a quotient by the ratio
of the averages at high temperature, and in the absence of degeneracies in the
energy spectrum, one finds [16, 8]

$\displaystyle\langle{\rm SFF}(t)\rangle=\frac{1}{\langle
Z(\beta)\rangle^{2}}\bigg{[}|\langle
Z(\beta+it)\rangle|^{2}+g_{c}(\beta,t)+\langle Z(2\beta)\rangle\bigg{]}.$

The first term in brackets is known as the disconnected part as it can be
derived from the average density of states
$\langle\rho(E)\rangle=\langle\sum_{n}\delta(E-E_{n})\rangle$ (one-point
function), as

$\langle Z(\beta+it)\rangle=\int dE\langle\rho(E)\rangle e^{-(\beta+it)E}.$

The second term captures correlations among eigenvalues and is governed by the
Fourier transform $g_{c}(\beta,t)$ of the connected two-level correlation
function of the energy spectrum
$\langle\rho(E)\rho(E^{\prime})\rangle_{c}=\langle\rho(E)\rho(E^{\prime})\rangle-\langle\rho(E)\rangle\langle\rho(E^{\prime})\rangle$.
Specifically,

$g_{c}(\beta,t)=\int
dEdE^{\prime}\langle\rho(E)\rho(E^{\prime})\rangle_{c}e^{-(\beta+it)E}e^{-(\beta-
it)E^{\prime}}.$

The last term is constant and governs the long-time asymptotics. The SFF
reduces to unit value at $t=0$. The short time evolution gives rise to a
parabolic decay $\langle{\rm SFF}(t)\rangle=1-\langle\Delta H^{2}\rangle
t^{2}$ in the time scale fixed by the inverse of the energy fluctuations
$\langle\Delta H^{2}\rangle=\int dE(E-\langle
E\rangle)^{2}\langle\rho(E)\rangle$ and extends, forming a slope. This decay
is governed by the disconnected part of the SFF. In chaotic systems, the decay
reaches a dip below the long-time asymptotics. The region where $\langle{\rm
SFF}(t)\rangle<\langle Z(2\beta)\rangle/\langle Z(\beta)\rangle^{2}$ is known
as correlation hole or dip [1, 3]. The latter is followed by a ramp, governed
by the eigenvalue correlations, and is thus a proxy for quantum chaos. The
ramp extends from the dip time to the plateau time, at which it takes the
constant value $\langle{\rm SFF}\rangle=\langle Z(2\beta)\rangle/\langle
Z(\beta)\rangle^{2}$ in the annealed approximation, in the absence of
degeneracies expected in chaotic systems.

##  II Spectral form factor in arbitrary physical processes

The fidelity-based interpretation of the SFF can be leveraged to consider more
general sorts of time evolution beyond the unitary case. In particular, this
makes it possible to generalize the SFF to non-Hermitian and open quantum
systems characterized by nonunitary evolution [34, 10, 35, 36, 37, 38]. This
section introduces tools used to describe nonunitary evolution that will be
employed in the explanations that follow in the next sections.

Several generalizations of the SFF have been put forward when the dynamics is
not unitary. At variance with alternative proposals with a restricted domain
of applicability [39, 40], the fidelity-based generalization of the SFF has
the advantage of involving only the eigenvalue correlations that govern
quantum dynamics and applies to arbitrary physical processes. Provided that
the evolution is described by a quantum channel $\Phi_{t}(\cdot)$ (i.e., a
completely-positive and trace-preserving map), the fidelity-based SFF is given
by [10, 35, 36, 37, 38]

$\displaystyle{\rm
SFF}(t)=\langle\psi_{\beta}|\Phi_{t}(|\psi_{\beta}\rangle\langle\psi_{\beta}|)|\psi_{\beta}\rangle.$
(6)

An arbitrary quantum channel admits a Kraus representation,
$\Phi_{t}(\rho_{0})=\sum_{\alpha=1}^{r}K_{\alpha}\rho_{0}K_{\alpha}^{\dagger}$,
where $r$ is known as the Choi rank [41]. The case of unitary evolution
corresponds to the case of a single Kraus operator that equals the time
evolution operator, i.e., $K(t)=K_{1}(t)=U(t)$ and $K_{\alpha}(t)=0$ for
$\alpha>1$. Given that Kraus operators need only obey the condition of adding
up to the identity $\sum_{\alpha}K_{\alpha}^{\dagger}K_{\alpha}=\mathbb{I}$
for the dynamics to be trace-preserving and that the Kraus decomposition
involves $1\leq r\leq d^{2}$ Kraus operators in a $d$-dimensional Hilbert
space, it is apparent that the chaotic features of the SFF under unitary
dynamics are generally suppressed under nonunitary time evolution. As a
result, quantum channels with a simple representation in the energy eigenbasis
are singled out to study filtering in quantum chaos and self-averaging of the
SFF.

An important class of channels that will be of relevance in the following is
that of mixed-unitary channels [42]. A channel $\Phi$ is a mixed-unitary
channel if there is an alphabet $\Sigma$, a probability vector $p$ and a
collection of unitaries $\\{U_{y}:y\in\Sigma\\}$ such that

$\displaystyle\Phi(\rho)=\sum_{y\in\Sigma}p(y)U_{y}\rho U_{y}^{\dagger}.$ (7)

The channel is thus a convex combination of unitaries. This kind of quantum
channel is unital and thus preserves the identity $\mathbb{I}$, i.e.,
$\Phi_{t}(\mathbb{I})=\mathbb{I}$.

The fidelity-based interpretation of the SFF extends to higher moments of the
SFF. Indeed, given that the initial state
$\rho_{0}=|\psi_{\beta}\rangle\langle\psi_{\beta}|$ is pure, the $k$-th moment
reads

$\displaystyle{\rm SFF}^{k}={\rm
tr}[\underbrace{\rho_{0}\rho_{t}\dots\rho_{0}\rho_{t}}_{\textrm{$k$
times}}]=\langle\psi_{\beta}|\rho_{t}|\psi_{\beta}\rangle^{k}.$ (8)

The $k$-th moment can be associated with a Zeno sequence in which the time
evolution is interrupted by sequential projective measurements onto the
initial state. Therefore, the RV in Eq. (4) probes the degree of factorization
of the time evolution in a sequence with $k=2$ in the presence of averaging.

Figure 1: Spectral form factor for a single realization (solid red line) and
upon Hamiltonian average (solid black line), together with the corresponding
RV (black dashed line). The averages are taken over a sample of $500$ random
$\mathrm{GOE}(64)$ Hamiltonians $H$, with $\sigma=1$. a In the unfiltered
case, $\kappa=0$, the RV saturates at the unit value after the dip time. b RV
using frequency filtering with the Gaussian function (26) and a finite
dephasing strength $\kappa=0.1$. The RV reaches a maximum value at the dip
time and then drops to a plateau of value
$\text{RV}_{p}=\big{\langle}(Z(2\beta)^{2}/Z(\beta)^{4}\big{\rangle}/\big{\langle}Z(2\beta)/Z(\beta)^{2}\big{\rangle}^{2}$.
c RV with eigenvalue filtering using the Gaussian function (30) with $f(E)=E$.
The RV increases to its maximum at the dip time and then drops to a plateau
given by
$\text{RV}_{p}=\big{\langle}1/Z(\beta)^{2}\big{\rangle}/\big{\langle}1/Z(\beta)\big{\rangle}^{2}$.
In all three panels, the inverse temperature is $\beta=0.1$.

##  III Unitarity breaking: spectrum filtering as a nonunitary quantum channel

In what follows, we consider three approaches frequently used to reduce the
erratic wiggles in the SFF. They involve averaging over a Hamiltonian ensemble
and the use of frequency filters and eigenvalue filters. The last two involve
different kinds of time-averaging and ensure the self-averaging of the SFF at
long times. We show that all three cases involve unitary breaking described by
a nonunitary quantum channel.

###  III.1 Averaging over Hamiltonian ensembles

Averaging over a Hamiltonian ensemble constitutes a popular approach that
smooths out the quantum noise wiggles in the SFF. This approach is at the core
of the random-matrix theory, the study of disordered systems, and matrix
models [6, 5, 43]. Given a Hilbert space $\mathcal{H}$ of dimension $d$,
consider an ensemble of Hamiltonians $\mathcal{E}_{H}$ with a probability
density function $P(H)$ and integration measure $dH$ such that
$\int_{\mathcal{E}_{H}}P(H)dH=1$. The average of the SFF over
${\mathcal{E}_{H}}$ is given by

$\displaystyle\langle{\rm
SFF}(t)\rangle_{{\mathcal{E}_{H}}}=\int_{\mathcal{E}_{H}}dHP(H)\left|\frac{{\rm
tr}\left(e^{-(\beta+it)H}\right)}{{\rm tr}\left(e^{-\beta
H}\right)}\right|^{2}.$ (9)

The fidelity-based interpretation of the SFF illuminates the underlying
physical process involved in such an average. For a specific Hamiltonian
$H=\sum_{n}E_{n}|n\rangle\langle n|$, the initial state is chosen as the
coherent Gibbs state (or the TFD in the case of a bipartite system),

$\displaystyle|\psi_{\beta}(H)\rangle=\sum_{n}\frac{e^{-\beta H/2}}{\sqrt{{\rm
tr}\left(e^{-\beta H}\right)}}|n\rangle.$ (10)

The Hamiltonian ensemble ${\mathcal{E}_{H}}$ provides an alphabet, together
with the collection of unitaries
$\\{U_{H}(t)=e^{-itH}:H\in\mathcal{E}_{H}\\}$. The state
$|\psi_{\beta}(H)\rangle$ is chosen with probability measure $P(H)dH$ and
evolved unitarily into $U_{H}(t)|\psi_{\beta}(H)\rangle$. The SFF is then
computed as the averaged survival probability over the Hamiltonian ensemble,

$\displaystyle\langle{\rm
SFF}(t)\rangle_{{\mathcal{E}_{H}}}=\int_{\mathcal{E}_{H}}dHP(H)|\langle\psi_{\beta}(H)|U_{H}(t)|\psi_{\beta}(H)\rangle|^{2}.$

As a result, averaging the SFF over a Hamiltonian ensemble involves breaking
the unitarity of the dynamics by classically mixing a distribution of states
and unitaries. When the initial state $\rho_{0}$ is fixed and independent of
the Hamiltonian $H$, the process can be associated with a mixed-unitary
channel
$\Phi(\rho_{0})=\int_{\mathcal{E}_{H}}dHP(H)U_{H}(t)\rho_{0}U_{H}(t)^{\dagger}.$
As a relevant instance, this is the case when the initial state is the
coherent Gibbs state in the infinite temperature limit $\beta=0$,
$|\psi_{\beta}\rangle=\sum_{n}\frac{1}{\sqrt{d}}|n\rangle$, where the Hilbert
space dimension fixes the probability amplitudes.

###  III.2 Frequency filtering and the time-averaged SFF

As an alternative to Hamiltonian averaging, in numerical and analytical
studies, it is customary to enforce the SFF’s self-averaging by using a filter
function $w(E_{n}-E_{m})$ that acts on the frequency domain, suppressing
contributions from given eigenvalue differences in the spectrum of a single
Hamiltonian. This is equivalent to filtering eigenvalues of the Liouville
superoperator $\mathbb{L}=-i(H\otimes\mathbb{I}-\mathbb{I}\otimes H^{T})$ that
governs the unitary evolution in the vectorized density matrix $|\rho_{t})$
according to $\frac{d}{dt}|\rho_{t})=\mathbb{L}|\rho_{t})$, i.e., when
representing the Liouville-von Neumann equation as a linear matrix equation.
We assume the frequency filter to be described by a symmetric function
$w(x):\mathbb{R}\rightarrow[0,1]$ satisfying $w(x)=w(-x)$. The frequency-
filtered SFF is then proportional to
$\sum_{nm}e^{-\beta(E_{n}+E_{m})-it(E_{n}-E_{m})}w(E_{n}-E_{m})$. Making use
of the Fourier transform of $w$, the frequency-filtered SFF reads

$\displaystyle{\rm SFF}_{w}(t)$ $\displaystyle=$
$\displaystyle\sum_{n,m=1}^{d}\frac{e^{-\beta(E_{n}+E_{m})-it(E_{n}-E_{m})}}{Z(\beta)^{2}}w(E_{n}-E_{m})$
(12) $\displaystyle=$
$\displaystyle\frac{1}{2\pi}\int_{-\infty}^{\infty}d\tau\widetilde{w}(t-\tau)\left|\frac{Z(\beta+i\tau)}{Z(\beta)}\right|^{2},$

with $\widetilde{w}(y)=\int_{-\infty}^{\infty}dE\exp(-iyE)w(E)$. Filtering in
frequency space is equivalent to time-averaging the canonical SFF associated
with the unitary time evolution. Without degeneracies in the energy spectrum,
the long-time behavior of ${\rm SFF}_{w}$ saturates at the plateau value set
by $w(0)$. Further, in the fidelity-based interpretation of the SFF, frequency
filtering can be recast as the result of a nonunitary time evolution. To this
end, consider a quantum channel $\Phi_{t}$ such that the time-evolution of the
initial coherent Gibbs state
$|\psi_{\beta}\rangle\langle\psi_{\beta}|=\sum_{nm}e^{-\beta(E_{n}+E_{m})/2}/Z(\beta)$
reads

$\displaystyle\rho_{t}$ $\displaystyle=$
$\displaystyle\Phi_{t}(|\psi_{\beta}\rangle\langle\psi_{\beta}|)$
$\displaystyle=$
$\displaystyle\sum_{nm}\frac{e^{-\beta(E_{n}+E_{m})/2-it(E_{n}-E_{m})}}{Z(\beta)}w(E_{n}-E_{m})|n\rangle\langle
m|.$

The latter can be rewritten as

$\displaystyle\Phi_{t}(\rho_{0})=\int_{-\infty}^{\infty}dyK(y)\rho_{0}K(y)^{\dagger},$
(14)

with

$\displaystyle
K(y)=\left(\frac{\widetilde{w}(y)}{2\pi}\right)^{\frac{1}{2}}e^{-i(t+y)H}.$
(15)

For the time evolution to be trace-preserving, it is required that

$\displaystyle\int_{-\infty}^{\infty}dyK(y)^{\dagger}K(y)=\frac{1}{2\pi}\int_{-\infty}^{\infty}dy\widetilde{w}(y)=1,$
(16)

that is, $w(0)=1$. The above equations provide an analog of the Kraus
decomposition with a continuous index [44]. They are associated with energy
diffusion processes. Generally, the Fourier transform $\widetilde{w}(y)$ of
the frequency filter can take both negative and positive values. However,
given Eq. (16), whenever $p(y)=\frac{1}{2\pi}\widetilde{w}(y)\geq 0$, it can
be thought of as a probability distribution. Frequency filtering is then
described by a mixed-unitary channel, i.e., the convex combination of unitary
quantum channels, each with a single Kraus operator that equals the time-
evolution operator shifted as $t\rightarrow t+y$. The collection of unitaries
in this case $\\{U_{y}(t)=e^{-i(t+y)H}:y\in\mathbb{R}\\}$ is generated by one
single Hamiltonian $H$, leading to a time-average of the quantum state at time
$t$, $\overline{\rho_{t}}=\int dyp(y)e^{-i(t+y)H}\rho_{0}e^{i(t+y)H}$, from
which the SFF is obtained as the fidelity ${\rm
SFF}_{w}(t)=\langle\psi_{\beta}|\overline{\rho_{t}}|\psi_{\beta}\rangle$.

An important example concerns the time averaging of the SFF over a time window
of duration $T$,

$\displaystyle\overline{\rm
SFF}(t)=\frac{1}{T}\int_{-T/2}^{+T/2}\left|\frac{Z(\beta+it+iy)}{Z(\beta)}\right|^{2}dy,$
(17)

for which $\widetilde{w}(y)=2\pi/T$ for $y\in[-T/2,T/2]$ and zero otherwise.
This is tantamount to considering the averaged time-dependent state
$\overline{\rho_{t}}=\frac{1}{T}\int_{-T/2}^{T/2}dye^{-i(t+y)H}\rho_{0}e^{i(t+y)H}$.

###  III.3 Eigenvalue filtering

An alternative filtering of the SFF involves expressions of the form
$|\sum_{n}e^{-\beta E_{n}-itE_{n}}w(E_{n})|^{2}$ with a filter function
$w(E)\geq 0$ that acts directly on the eigenvalues. This is equivalent to
selecting an energy band to study the SFF, while disregarding contributions
from other parts of the spectrum [27, 21]. As noted in the introduction, the
Boltzmann factor $\exp(-\beta E_{n})$ can be considered as an exponential
eigenvalue filter acting on the SFF with $\beta=0$. The use of an energy-
eigenvalue filter function can be associated with the evolution governed by a
single nonunitary Kraus operator

$\displaystyle K(t)$ $\displaystyle=$ $\displaystyle e^{-itH}\sqrt{w}(H).$
(18)

The selection of the energy window corresponds to a post-selection represented
by the operation

$\displaystyle|\psi_{\beta}\rangle\langle\psi_{\beta}|\rightarrow\rho_{t}=\frac{K(t)|\psi_{\beta}\rangle\langle\psi_{\beta}|K(t)^{\dagger}}{Z_{w}(\beta)},$
(19)

which is always a pure and normalized state, including the state at $t=0$.
Here, the modified partition function

$\displaystyle Z_{w}(\beta)$ $\displaystyle=$ $\displaystyle{\rm
tr}[K(t)|\psi_{\beta}\rangle\langle\psi_{\beta}|K(t)^{\dagger}]$ (20)
$\displaystyle=$ $\displaystyle{\rm tr}[w(H)e^{-\beta H}].$

This accounts for the correct normalization, so that the SFF at all times
$t\geq 0$ is still given as the Uhlmann fidelity ${\rm SFF}_{w}(t)={\rm
tr}(\rho_{0}\rho_{t})$, i.e., the survival probability of the post-selected
coherent Gibbs state $\rho_{0}$ and its time evolution,

$\displaystyle{\rm
SFF}_{w}(t)=\sum_{nm}e^{-\beta(E_{n}+E_{m})-it(E_{n}-E_{m})}\frac{w(E_{n})w(E_{m})}{Z_{w}(\beta)^{2}}.$
(21)

The choice of the Kraus operator is nonlinear in the quantum state, as it is
tailored for the initial coherent Gibbs state, i.e., ${\rm
tr}[K(t)|\psi_{\beta}\rangle\langle\psi_{\beta}|K(t)^{\dagger}]=1$, making
(only) in this case the dynamics trace-preserving. While this scenario is not
the standard one in the theory of open quantum systems, it admits a natural
interpretation in terms of energy dephasing without quantum jumps, as
discussed in IV.2.

For completeness, we note that in terms of the Fourier transform of
$\widetilde{w}(y)=\int_{-\infty}^{\infty}dE\exp(-iyE)w(E)$ and the definition
$p(y)=\widetilde{w}(y)/(2\pi)$, the filtered SFF can be found in terms of the
analytically continued partition function

$\displaystyle{\rm
SFF}_{w}(t)=\frac{1}{Z_{w}(\beta)^{2}}\left|\int_{-\infty}^{\infty}dyp(t-y)Z(\beta-
iy)\right|^{2}.$ (22)

Naturally, for $w(E)=1$, $Z_{w}(\beta)=Z(\beta)$, $p(t-y)=\delta(t-y)$, one
recovers the canonical SFF in Eq. (2).

Before moving forward, let us characterize the performance of frequency and
energy filters in the SFF. We consider random matrix Hamiltonians as a
paradigm of quantum chaos. We sample the Hamiltonian matrices $H$ from the
Gaussian orthogonal ensemble, $\mathrm{GOE}(d)$, calculate the corresponding
${\rm SFF}(t)$ and ${\rm SFF}^{2}(t)$, and then perform the average over the
different realizations. Specifically, we consider samples of real matrices
$H=(X+X^{\intercal})/2$, where all elements $x\in\mathbb{R}$ of $X$ are
pseudo-randomly generated with probability measure given by the Gaussian,
$\exp(-x^{2}/\big{(}2\sigma^{2}\big{)})/(\sigma\sqrt{2\pi})$, where $\sigma$
is the standard deviation.

Figure 1 shows three panels corresponding to the isolated, unfiltered SFF in
panel a and its modified versions with frequency and energy filters in panels
b and c, respectively. A single realization of the SFF exhibits quantum noise,
manifested in the erratic oscillatory behavior in the time evolution (red line
in Fig. 1a). This is suppressed by performing a Hamiltonian ensemble average
(solid black line in Fig. 1a). Alternatively, the frequency filter can
suppress quantum noise in the SFF for a single random-matrix Hamiltonian
without relying on ensemble averages, as illustrated in panel b. Its effect is
to reduce the oscillatory wiggles and the RV. The use of filters acting on
energy eigenvalues directly provides a different alternative, shown in panel
c. Note that for the unfiltered SFF, RV equals 1 from the time of the dip
onward, as seen in Fig. 1a. This result holds for random matrices of any
dimension [28] and for chaotic many-body quantum systems of any size [28, 29],
which means that the unfiltered SFF is non-self-averaging. $\text{RV}=1$,
because the distribution of the SFF$(t)$ for large times [30] is exponential,
so the square of the mean of the distribution and its variance are equal. In
contrast, the asymptotic values of the RV under frequency and energy filters
become smaller than 1. Furthermore, as we shall see in Sec. V, the long-time
values of the RV of the filtered SFF further decrease as $d$ increases,
indicating that the SFF becomes self-averaging.

##  IV Energy dephasing processes and spectral filtering

This section explores the relationship between energy-dephasing processes and
the effects of different spectral filters.

###  IV.1 Frequency filters from energy dephasing

Energy dephasing processes, also known as energy diffusion processes, arise in
various scenarios [45, 46]. They are postulated in modifications of quantum
mechanics involving wavefunction collapse models [47, 48, 49]. They also arise
in the description of unitary time evolution timed by a realistic clock
subject to errors [50, 51]. They have been used to study the interplay between
quantum chaos and decoherence [52, 10, 35]. Energy dephasing has also been
analyzed in the context of AdS/CFT [44, 53, 54, 55] to explore the relation
between entanglement and spacetime connectedness [13]. It can be described by
the master equation

$\displaystyle d_{t}\rho_{t}=-i[H,\rho_{t}]-\kappa[X,[X,\rho_{t}]],$ (23)

with the condition that $[H,X]=0$, so that both Hermitian operators have a
common set of eigenvectors, i.e., $H=\sum_{n}E_{n}|n\rangle\langle n|$ and
$X=\sum_{n}x_{n}|n\rangle\langle n|$. The nested commutator plays the role of
the dissipator and induces dephasing, suppressing coherent quantum
superpositions in the energy eigenbasis. This is explicitly seen by
considering the time evolution of an initial quantum state
$\rho_{0}=\sum_{nm}\rho_{nm}(0)|n\rangle\langle m|$,

$\displaystyle\rho_{t}=\sum_{nm}\rho_{nm}(0)e^{-it(E_{n}-E_{m})-\kappa
t(x_{n}-x_{m})^{2}}|n\rangle\langle m|.$ (24)

For an initial coherent Gibbs state, the SFF is obtained as the survival
probability

$\displaystyle{\rm
SFF}(t)=\sum_{nm}\frac{e^{-\beta(E_{n}+E_{m})-it(E_{n}-E_{m})}}{Z(\beta)^{2}}e^{-\kappa
t(x_{n}-x_{m})^{2}}.$ (25)

When the Hermitian Lindblad operator is a deformation of the Hamiltonian,
$X=f(H)$, $x_{n}=f(E_{n})$, and $w(E_{n}-E_{m})=\exp\\{-\kappa
t[f(E_{n})-f(E_{m})]^{2}\\}$ in Eq. (12). When they are equal, $X=H$, one
recovers the canonical case of energy dephasing. In this case, one can recast
${\rm SFF}(t)$ in Eq. (25) as the frequency-filtered ${\rm SFF}_{w}$ (12) with
the identification of a time-dependent Gaussian filter function

$\displaystyle w(E_{n}-E_{m})=\exp[-\kappa t(E_{n}-E_{m})^{2}].$ (26)

Figure 2: Frequency filtered SFF and its RV for different dephasing strengths
and inverse temperatures. Panels a and b show the SFF next to the
corresponding RV for inverse temperature $\beta=0.1$ and different dephasing
strengths $\kappa$. Panels c and d show the SFF next to the RV for a dephasing
strength $\kappa=0.01$ and different values of the inverse temperature
$\beta$. In all panels, the Hamiltonian averages were taken over a sample of
$500$ random $\mathrm{GOE}(64)$ Hamiltonians $H$ with $\sigma=1$.

The action of the frequency filter (26) in the SFF is shown in Fig. 2a for
fixed $\beta=0.1$ and varying $\kappa$; see also Fig. 1. Such filtering delays
the onset of the ramp, reduces its span, and decreases the depth of the
correlation hole. In short, it decreases the dynamical manifestations of
quantum chaos. The corresponding RV is shown in Fig. 2b indicating that the
long-time plateau of the RV is independent of $\kappa$ for $\kappa>0$, as
expected from Eq. (25). Figure 2c shows the effect of varying $\beta$ for
fixed $\kappa$, with the corresponding $\beta$-dependent long-time plateau
being associated with the RV of a canonical thermal equilibrium state, as
shown in Fig. 2d.

###  IV.2 Eigenvalue filtering from energy dephasing without quantum jumps

In what follows, we show that eigenvalue filtering can be described as the
non-Hermitian evolution associated with energy-dephasing processes without
quantum jumps. To this end, consider the evolution operator
$U(t)=\exp(-itH_{T})$ generated by the time-independent non-Hermitian
Hamiltonian $H_{T}=H-i\Gamma$, with $H=H^{\dagger}$ and
$\Gamma=\Gamma^{\dagger}$. In this case, the evolution is not trace-
preserving, and one can introduce a single nonlinear Kraus operator dependent
on the initial state $\rho_{0}$

$\displaystyle K=\frac{1}{\sqrt{{\rm
tr}(e^{-itH_{T}}\rho_{0}e^{itH_{T}^{\dagger}})}}e^{-itH_{T}}.$ (27)

The latter is associated with a master equation of the form

$\displaystyle d_{t}\rho_{t}=-i(H_{T}\rho_{t}-\rho_{t}H_{T}^{\dagger})+2{\rm
tr}(\Gamma\rho_{t})\rho_{t},$ (28)

which describes non-Hermitian dynamics subject to balanced norm gain and loss
[56, 57].

In particular, consider a non-Hermitian Hamiltonian in which the Hermitian and
anti-Hermitian parts commute $[H,\Gamma]=0$ and thus have common eigenstates
$\\{|E_{n}\rangle\\}$. The action of the filter function can be identified by
noting that $w(H)=\exp[-it\Gamma]$, i.e.,
$\Gamma|E_{n}\rangle=-\frac{1}{t}\log w(E_{n})|E_{n}\rangle$.

As an illustrative example, consider the master equation for energy dephasing
in Eq. (23) with the condition $[H,X]=0$. This evolution is of the Lindblad
form with a single Hermitian Lindblad operator $\sqrt{2}X$ and is thus
Markovian [41]. As such, it can alternatively be written in terms of a non-
Hermitian Hamiltonian $H_{T}=H-i2\kappa X^{2}$ and a quantum jump term
$\mathcal{J}(\rho)=2\kappa X\rho X$. Disregarding the quantum jump term
induces a non-Hermitian evolution exclusively governed by $H_{T}$. This can be
justified at short times or by post-selection of quantum trajectories to the
absence of quantum jumps [58]. The evolution of the subset of quantum
trajectories exhibiting no quantum jumps from time $t=0$ to time $t$ is
governed by Eq. (28), which is known as the nonlinear Schrödinger equation for
null-measurement conditioning in this context [56]. Specifically, the time
evolution subject to energy dephasing in the absence of quantum jumps is
governed by (28), which admits a closed-form solution [35]. Explicit
computation of the survival probability for the coherent Gibbs state yields
the expression of the SFF

$\displaystyle{\rm
SFF}(t)=\sum_{nm}e^{-\beta(E_{n}+E_{m})-it(E_{n}-E_{m})}\frac{e^{-\kappa
t(x_{n}^{2}+x_{m}^{2})}}{Z(\beta)Z_{w}(\beta,t)},$

where the modified partition function $Z_{w}(\beta,t)={\rm
tr}[w(X)^{2}\exp(-\beta H)]$. The case of the Hamiltonian deformation $X=f(H)$
corresponds to the choice of the time-dependent filter function

$\displaystyle w(E_{n})=\exp[-\kappa tf(E_{n})^{2}],$ (30)

in Eq. (21).

Figure 3: Eigenvalue filtered SFF and its RV for different dephasing strengths
and inverse temperatures. Hamiltonian averages over a sample of $500$ random
$\mathrm{GOE}(64)$ Hamiltonians $H$ with $\sigma=1$. Panels a and b show the
SFF and the corresponding RV with inverse temperature $\beta=0.1$ and
different dephasing strengths $\kappa$. Panels c and d show the SFF and the
associated RV with fixed dephasing strength $\kappa=0.01$ and varying inverse
temperatures $\beta$.

The time dependence of the SFF with eigenvalue filtering in Eq. (30)
engineered through energy dephasing in the absence of quantum jumps is
illustrated in Fig. 3. At fixed $\beta$, increasing $\kappa$ reduces the
correlation hole; see Fig. 3a. For $\kappa>0$, the long-time plateaus of the
SFF and RV differ from the unfiltered case. Increasing $\beta$ for fixed
$\kappa$ favors contributions to the SFF from the low-energy part of the
spectrum and generally reduces the correlation hole and increases the plateau
value of the SFF and the RV, as shown in Figs. 3c and d, respectively.

##  V Self-averaging at long times

Under chaotic quantum dynamics, quantities that are local in space are
expected to be self-averaging at short times [28, 29, 30]. It has further been
suggested that time-locality implies self-averaging at long times. The SFF can
be interpreted as a time auto-correlation function, thus a non-local quantity
in time. The unfiltered SFF lacks the self-averaging property at all
timescales in isolated quantum systems [28].

We have shown that filters ubiquitously used to reduce the erratic wiggles of
the SFF can be associated with quantum channels involving nonunitary dynamics.
The breaking of unitarity contributes to suppressing quantum noise. In what
follows, we numerically investigate the dependence of the RV as a function of
the system size to identify when RV decreases as $d$ increases, thus rendering
the SFF self-averaging.

Figure 1 implies that unitarity breaking can suppress the quantum noise of the
SFF. Nevertheless, the robustness against sample-to-sample fluctuations is
associated with the reduction of the RV as the Hilbert space dimension
increases. Fig. 4a and b show that the frequency- and eigenvalue-filtered SFFs
become self-averaging at the small inverse temperature shown and large times.
Figure 4c-d confirm that the filtered SFFs become self-averaging at times
after the correlation hole, where, according to Fig. 1b-c,
$\langle\text{SFF}(t)\rangle>\text{RV}(t)$.

The effect of the inverse temperature depends on the filter considered, as
shown in Fig. 5. The long-time SFF is only self-averaging for moderate to high
temperatures in the case of frequency filtering; see panel a. By contrast, the
long-time eigenvalue-filtered SFF remains self-averaging as the inverse
temperature varies, as shown in panel b.

Figure 4: Asymptotic self-averaging of the filtered SFF. Hamiltonian averages
over a sample of $1000$ random $\mathrm{GOE}(d)$ Hamiltonians $H$ with
$\sigma=1$. Panels a and b show the plateau value of the frequency-filtered
and the energy-filtered RV, that is independent of $\kappa$, as a function of
the Hilbert space dimension $d$ is shown for different inverse temperatures.
Panel c shows the frequency-filtered RV for inverse temperature $\beta=0.5$,
dephasing strength $\kappa=0.2$, and different Hilbert space dimensions $d$.
In panel d, the corresponding energy-filtered RV for the same parameters is
shown. In both cases, the relative variance plateau decreases with the
dimension increment, i.e., the SFF becomes self-averaging.  Figure 5: Self-
averaging of the filtered SFF as a function of the inverse temperature. Panel
a shows the value of the long-time RV plateau in the frequency-filtered SFF,
reflecting a breakdown of self-averaging as the inverse temperature is
increased. By contrast, panel b indicates that self-averaging remains robust
against variations of the inverse temperature in the case of eigenvalue
filtering.

##  VI Information loss and its recovery

We have shown that the different approaches to suppress quantum noise in the
SFF can be described as quantum channels involving nonunitary physical
processes. In particular, Hamiltonian averaging, frequency filtering, and time
averaging of the SFF are all associated with mixed-unitary channels. The
latter are unital and thus satisfy the necessary conditions for the purity
$P_{t}={\rm tr}[\Phi_{t}(\rho_{0})^{2}]$ of the time-evolving state to decay
monotonically under the action of the channel [59, 52]. Conversely, the linear
entropy $S_{L}=1-P_{t}$ increases monotonically. Thus, these channels lead to
monotonic information loss. Yet, the lost information is fully recoverable
[60, 42]. To appreciate this, it is convenient to consider the Hilbert space
of the system together with the Hilbert space $\mathcal{H}_{E}$ of the
environment with initial density matrix $\rho_{E}$, such that

$\displaystyle\Phi_{t}(\rho_{0})={\rm
tr}_{E}\left(U_{SE}\rho_{0}\otimes\rho_{E}U_{SE}^{\dagger}\right),$ (31)

in terms of a global unitary $U_{SE}$. One can consider a measurement on the
environment associated with a family of operators $M_{y}$, such that
$\sum_{y}M_{y}=\mathbb{I}_{E}$. The expectation value of an operator $A$ on
the system can be described in terms of a family of completely positive maps
$\Phi_{y}$.

$\displaystyle{\rm tr}[\Phi_{t}(\rho_{0})A\otimes\mathbb{I}_{E}]$
$\displaystyle=$ $\displaystyle\sum_{y}{\rm
tr}_{E}\left(U_{SE}\rho_{0}\otimes\rho_{E}U_{SE}^{\dagger}A\otimes
M_{y}\right)$ (32) $\displaystyle=$ $\displaystyle\sum_{y}{\rm
tr}[\Phi_{y}(\rho_{0})A].$

The decomposition of the channel $\Phi_{t}=\sum_{y}\Phi_{y}$ is known as an
instrument. The measurement of $M$ on the environment yields outcome $y$ and
the quantum state $\Phi_{y}(\rho_{0})/{\rm tr}[\Phi_{y}(\rho_{0})]$ with
probability $p(y)={\rm tr}[\Phi_{y}(\rho_{0})]$. It is then possible to select
the reverse operation

$\displaystyle R_{y}=U_{y}^{\dagger}\Phi_{t}(\rho_{0})U_{y},$ (33)

so that the information-recovery channel is

$\displaystyle R=\sum_{y}R_{y}\circ\Phi_{y}.$ (34)

In short, the information acquired by performing a measurement on the
environment can be used to reverse the action of the quantum channel
$\Phi_{t}$ on the system, thus recovering the initial state.

This information-recovery protocol involves access to the degrees of freedom
of an environment, which may be physical or an auxiliary construction,
depending on the context. Any physical system is embedded in an environment
that may give rise to decoherence and filtering through interaction with the
system of interest. By contrast, in an effectively isolated system, one may
still consider using nonunitary operations for filtering as done in numerical
analysis without an explicit physical environment.

In what follows, we tackle a complementary problem, the recovery of
information masked exclusively by the filter. We focus on frequency filtering
and aim at obtaining the unfiltered SFF from the filtered one by undoing the
action of the filter. The filtered SFF is the convolution of the Fourier
transform of the filter function and the canonical SFF, as shown in Eq. (12),
which can be written as

$\displaystyle{\rm SFF}_{w}(t)=\frac{1}{2\pi}\widetilde{w}(t)\ast{\rm
SFF}(t).$ (35)

By the convolution theorem, it is thus possible to retrieve ${\rm SFF}$ from
knowledge of ${\rm SFF}_{w}$ and $w$, using

$\displaystyle\widetilde{{\rm SFF}}(\nu)=\frac{\widetilde{{\rm
SFF}}_{w}(\nu)}{w(\nu)},$ (36)

provided that $w(\nu)$ is nonzero everywhere in the domain of $\widetilde{{\rm
SFF}}_{w}(\nu)$. Even when the inverse frequency filter function $1/w(\nu)$ is
nonsingular, the inversion can be unstable for small values of $w(\nu)$.
Furthermore, knowledge of $\widetilde{{\rm SFF}}_{w}(\nu)$ generally comes
with additive noise, whether resulting from limited machine precision in a
numerical simulation or statistical errors in measured data. This scenario is
common in filter analysis and motivates alternatives to direct deterministic
deconvolution, such as the Wiener deconvolution.

##  VII Intrinsic quantum noise from eigenvalue statistics

In the fidelity-based interpretation, the SFF is the survival probability of
the time-evolving quantum state $\rho_{t}$ in the initial coherent Gibbs (or
TFD) state. As such, one can introduce a projector onto the initial state

$\displaystyle P=\rho_{0}=|\psi_{\beta}\rangle\langle\psi_{\beta}|,$ (37)

satisfying $P^{2}=P$, i.e., with eigenvalues $\pm 1$. Such eigenvalues
correspond to measurement outcomes in a projective measurement of $P$. The
full counting statistics associated with the projective measurement associated
with $P$ is thus that of a discrete random variable, i.e., the Bernoulli
distribution. Its characteristic function reads

$\displaystyle{\rm tr}\left[\rho_{t}e^{i\theta P}\right]=1+(e^{i\theta}-1){\rm
SFF}(t).$ (38)

For any nontrivial evolution, an intrinsic quantum noise cannot be suppressed
(other than by post-selection), whether the dynamics is unitary or not. The
quantum noise associated with the uncertainty in the measurement outcomes of a
projective measurement of $P$ can be quantified by the relative variance of
the eigenvalue statistics encoded in the relation

$\displaystyle\frac{{\rm var}_{\rho_{t}}(P)}{{\rm
tr}(P\rho_{t})^{2}}=\frac{{\rm tr}(P^{2}\rho_{t})-{\rm
tr}(P\rho_{t})^{2}}{{\rm tr}(P\rho_{t})^{2}}=\frac{1-{\rm SFF}}{{\rm SFF}}.$
(39)

For any $t>0$, up to recurrences of zero measure [61, 62], ${\rm
var}_{\rho_{t}}(P)>0$.

##  VIII Eigenvalue filtering as Hamiltonian deformation

We first note the following identity for the modified partition function (20)
with an eigenvalue filter $w(E)$,

$\displaystyle Z_{w}(\beta)={\rm tr}\left[e^{-\beta\left(H-\frac{1}{\beta}\log
w(H)\right)}\right].$ (40)

As a result, $Z_{w}(\beta)$ can be understood as the standard partition
function of the operator

$\displaystyle F_{\beta}=H-\frac{1}{\beta}\log w(H),$ (41)

that describes a one-parameter family of Hermitian Hamiltonian deformations of
$H$ [63, 64]. Formally, this deformation takes the form of a Helmholtz free
energy operator analogous to that introduced to bound the charging power of
quantum batteries [65]. In particular, the filter gives rise to the entropy
(surprisal) term $S(H)=\log w(H)$. The eigenvalue-filtered SFF in Eq. (21) is
then

$\displaystyle{\rm SFF}_{w}(t)=\left|\frac{{\rm tr}\left(e^{-\beta
F_{\beta}-itH}\right)}{{\rm tr}\left(e^{-\beta F_{\beta}}\right)}\right|^{2}.$
(42)

At long times, in the absence of degeneracies, ${\rm SFF}_{w}(t)$ tends to

$\displaystyle\overline{{\rm SFF}_{w}}=\frac{{\rm tr}\left(e^{-2\beta
F_{\beta}}\right)}{{\rm tr}\left(e^{-\beta F_{\beta}}\right)^{2}}.$ (43)

This expression is nothing but the purity $P[\rho_{w}(\beta)]={\rm
tr}[\rho_{w}(\beta)^{2}]$ of the canonical Gibbs thermal state
$\rho_{w}(\beta)$ defined with respect to the deformed Hamiltonian, i.e., the
free energy operator $F_{\beta}$,

$\displaystyle\rho_{w}(\beta)=\frac{e^{-\beta F_{\beta}}}{Z_{w}(\beta)}.$ (44)

Indeed, the asymptotic value of ${\rm SFF}_{w}(t)$ can be written in terms of
the second Rényi entropy $S_{2}[\rho_{w}(\beta)]=-\log{\rm
tr}[\rho_{w}(\beta)^{2}]$ as

$\displaystyle\overline{{\rm
SFF}_{w}}=P[\rho_{w}(\beta)]=e^{-S_{2}[\rho_{w}(\beta)]}.$ (45)

For an eigenvalue filter function $w(E):\mathbb{R}\rightarrow[0,1]$,
$\overline{{\rm SFF}_{w}}\leq\overline{{\rm SFF}}$ and
$S_{2}[\rho_{w}(\beta)]\geq S_{2}[\rho(\beta)]$, where
$\rho(\beta)=\exp(-\beta H)/Z(\beta)$ is the canonical thermal state of the
undeformed Hamiltonian.

##  IX Master equations for frequency filters from Liouvillian deformation

We next show that frequency filters are associated with a family of master
equations that generalize the dynamics related to energy dephasing. Consider
the master equation in which time-evolution is generated by a Liouvillian
$\mathbb{L}$,

$\displaystyle\frac{d}{dt}|\rho_{t})=\mathbb{L}|\rho_{t}),$ (46)

where $|\rho_{t})$ denotes the vectorized density matrix at time $t$. In terms
of it, ${\rm SFF}(t)=(\rho_{0}|\rho_{t})$. Formally, equation (46) is solved
by $|\rho_{t})=e^{\mathbb{L}t}|\rho_{0})$. We focus on the case in which the
Liouvillian is diagonalizable, so that it admits a spectral decomposition of
the form $\mathbb{L}=\sum_{\mu}\lambda_{\mu}|\mu)(\tilde{\mu}|$ using a bi-
orthogonal basis. Here, $|\mu)$ and $(\tilde{\mu}|$ are the right and left
eigenstates, respectively, with complex eigenvalue $\lambda_{\mu}$ [66, 67].
We next consider the Liouvillian of the form

$\displaystyle\mathbb{L}(\cdot)=-i[H,\cdot],$ (47)

associated with an isolated system with Hamiltonian $H$. Its spectrum is
purely imaginary, and left and right eigenvectors coincide. Given a complex
function $W(z):\mathbb{C}\rightarrow\mathbb{C}$ we define the associated
Liouvillian deformation $W(\mathbb{L})=\sum_{n}W(\lambda_{\mu})|\mu)(\mu|$
[36]. By specifying the Liouvillian deformation in terms of the frequency
filter function $w(x):\mathbb{R}\rightarrow[0,1]$ as

$\displaystyle W(\mathbb{L})=\log w(i\mathbb{L}),$ (48)

we consider a physical process in which the initial, unfiltered coherent Gibbs
state $|\psi_{\beta}\rangle\langle\psi_{\beta}|$ evolves into a generalization
of the frequency-filtered time-dependent density matrix in Eq. (III.2).
Specifically, we consider the time evolution for $t\geq 0$ described by the
time-dependent density matrix

$\displaystyle\rho_{t}$ $\displaystyle=$
$\displaystyle\sum_{nm}\frac{e^{-\beta(E_{n}+E_{m})/2-it(E_{n}-E_{m})}}{Z(\beta)}e^{\chi(t)W(E_{n}-E_{m})}|n\rangle\langle
m|,$

where $\chi(t)$ is a real function satisfying $\chi(0)=0$ and
$W(E_{n}-E_{m})=\langle n|W(\mathbb{L})|m\rangle$ are matrix elements in the
Hamiltonian eigenbasis. This evolution fulfills the master equation

$\displaystyle\frac{d}{dt}|\rho_{t})=\left[\mathbb{L}+\dot{\chi}(t)W(\mathbb{L})\right]|\rho_{t}),$
(49)

with the initial condition $\rho_{0}=|\psi_{\beta}\rangle\langle\psi_{\beta}|$
and $\dot{\chi}$ denotes the time-derivative of $\chi$. While $\mathbb{L}$ is
anti-Hermitian, $W(\mathbb{L})$ is Hermitian. Thus, $W(\mathbb{L})$ breaks
unitarity and can be identified as the dissipator in the master equation (49).
Given that the $W(z)=W(-z)$, its Taylor series expansion involves only even
powers of $z$, i.e., $W(z)=\sum_{n=0}^{\infty}W^{(2n)}(0)z^{2n}/(2n!)$. The
master equation can be written as

$\displaystyle\frac{d}{dt}\rho_{t}=-i[H,\rho_{t}]+\dot{\chi}(t)\sum_{n=0}^{\infty}\frac{W^{(2n)}(0)}{(2n)!}{\rm
ad}_{H}^{2n}\rho_{t},$ (50)

where the nested commutators in each term of the Taylor series have been
written compactly in terms of the adjoint map ${\rm ad}_{X}Y=[X,Y]$, ${\rm
ad}_{X}^{2}Y=[X,[X,Y]]$, etc.

The case of a time-independent frequency filter for $t>0$ is described by
choosing $\chi(t)$ as the Heavisde step function,

$\displaystyle\chi(t)=\Theta(t),\qquad\dot{\chi}(t)=\delta(t).$ (51)

The delta function $\delta(t)=\frac{d}{dt}\Theta(t)$ in the master equation is
thus required for the frequency filter to be time-independent. Implementing
this filter relies on a single kick with the dissipator $W(\mathbb{L})$.

Naturally, for the conventional energy-dephasing frequency filter (26), the
master equations (49) and (50) truncate at ${\rm ad}_{H}^{2}\rho$ and reduce
to (23) for the choice $\chi(t)=t$, $\dot{\chi}(t)=1$.

##  X Discussion and conclusions

The lack of self-averaging in the SFF is tied to quantum noise, manifested by
erratic wiggles in the time domain. Analytical and numerical studies of the
SFF enforce the reduction of the wiggles by resorting to Hamiltonian
ensembles, time-averaging, and spectral filters in the energy or frequency
domain. Through scaling analysis of the relative variance of the SFF, we have
shown that the frequency and energy filters ensure that the SFF becomes self-
averaging at long times.

We have established that suppressing the erratic wiggles (quantum noise) in
the SFF implies nonunitary dynamics characterized by information loss and
decoherence. Hamiltonian averaging, time-averaging, and frequency filters can
be described by a mixed-unitary channel representing the application of a
random unitary with a given probability distribution. Mixed-unitary channels
are unital and induce information loss that can, however, be recovered by
environment-assisted channel correction. By contrast, filters acting directly
in the energy eigenvalues can be interpreted as a nonlinear quantum channel
describing the non-Hermitian evolution of an energy-dephasing process
conditioned to the absence of quantum jumps.

The identification of the canonical, filtered SFFs for isolated systems in
terms of the survival probability of a coherent Gibbs state under nonunitary
evolution singles out the fidelity-based generalization of the SFF to open
quantum systems put forward in Refs. [10, 35, 36, 37, 38] with respect to
alternative proposals [39, 40]. The fidelity-based approach makes it possible
to unify SFFs for isolated systems with and without filters and for open
quantum systems in a single framework.

Our results rely on the combination of tools in quantum information science
and quantum chaos, and contribute to the understanding of filters in the
characterization of the spectral properties of many-body systems (e.g., in
numerical studies) as physical operations breaking unitary. In particular, our
results establish how such filters can be implemented in digital or analog
quantum simulation experiments of the noenquilibrium dynamics of many-body
systems. This conclusion should be generalizable to quantities other than the
SFF, such as correlation functions, that admit an information-theoretic
interpretation associated with a quantum evolution. Our results hold for the
dynamics of finite-dimensional systems and thus can be applied to the
description of black hole physics in this framework, where self-averaging SFFs
appear in semiclassical approaches. In view of our findings, the latter
involve unitarity breaking.

###### Acknowledgements.

It is a pleasure to acknowledge discussions with Federico Balducci, Aurelia
Chenu, Julien Cornelius, Íñigo L. Egusquiza, Pablo Martínez-Azcona, Federico
Roccati, Avadh Saxena, and Zhenyu Xu. This project was funded within the
QuantERA II Programme that has received funding from the European Union’s
Horizon 2020 research and innovation programme under Grant Agreement No.
16434093. For open access and in fulfillment of the obligations arising from
the grant agreement, the authors have applied a Creative Commons Attribution
4.0 International (CC BY 4.0) license to any Author Accepted Manuscript
version arising from this submission. L.F.S. was supported by a grant from the
United States National Science Foundation (NSF, Grant No. DMR-1936006).

## References

  * Leviandier  _et al._ [1986] L. Leviandier, M. Lombardi, R. Jost, and J. P. Pique, Fourier transform: A tool to measure statistical level properties in very complex spectra, Phys. Rev. Lett. 56, 2449 (1986).
  * Wilkie and Brumer [1991] J. Wilkie and P. Brumer, Time-dependent manifestations of quantum chaos, Phys. Rev. Lett. 67, 1185 (1991).
  * Alhassid and Levine [1992] Y. Alhassid and R. D. Levine, Spectral autocorrelation function in the statistical theory of energy levels, Phys. Rev. A 46, 4650 (1992).
  * Alhassid and Whelan [1993] Y. Alhassid and N. Whelan, Onset of chaos and its signature in the spectral autocorrelation function, Phys. Rev. Lett. 70, 572 (1993).
  * Haake [2010] F. Haake,  _Quantum Signatures of Chaos_ (Springer, Berlin, 2010).
  * Mehta [2004] M. L. Mehta,  _Random Matrices, 3rd ed._ (Elsevier, San Diego, 2004).
  * Papadodimas and Raju [2015] K. Papadodimas and S. Raju, Local operators in the eternal black hole, Phys. Rev. Lett. 115, 211601 (2015).
  * del Campo  _et al._ [2017] A. del Campo, J. Molina-Vilaplana, and J. Sonner, Scrambling the spectral form factor: Unitarity constraints and exact results, Phys. Rev. D 95, 126008 (2017).
  * del Campo  _et al._ [2018] A. del Campo, J. Molina-Vilaplana, L. F. Santos, and J. Sonner, Decay of a thermofield-double state in chaotic quantum systems, Eur. Phys. J. Spec. Top. 227, 247 (2018).
  * Xu  _et al._ [2021] Z. Xu, A. Chenu, T. Prosen, and A. del Campo, Thermofield dynamics: Quantum chaos versus decoherence, Phys. Rev. B 103, 064309 (2021).
  * Umezawa  _et al._ [1982] H. Umezawa, H. Matsumoto, and M. Tachiki,  _ThermoField Dynamics and Condensed States_ (North-Holland, 1982).
  * Israel [1976] W. Israel, Thermo-field dynamics of black holes, Phys. Lett. A 57, 107 (1976).
  * Maldacena [2003] J. Maldacena, Eternal black holes in anti-de sitter, JHEP 2003 (04), 021.
  * Maldacena and Susskind [2013] J. Maldacena and L. Susskind, Cool horizons for entangled black holes, Fortschr. Phys 61, 781 (2013).
  * Maldacena  _et al._ [2016] J. Maldacena, S. H. Shenker, and D. Stanford, A bound on chaos, JHEP 2016 (8), 106.
  * Cotler  _et al._ [2017] J. S. Cotler, G. Gur-Ari, M. Hanada, J. Polchinski, P. Saad, S. H. Shenker, D. Stanford, A. Streicher, and M. Tezuka, Black holes and random matrices, JHEP 2017 (5), 118.
  * Dyer and Gur-Ari [2017] E. Dyer and G. Gur-Ari, 2d cft partition functions at late times, JHEP 2017 (8), 75.
  * Gharibyan  _et al._ [2018] H. Gharibyan, M. Hanada, S. H. Shenker, and M. Tezuka, Onset of random matrix behavior in scrambling systems, JHEP 2018 (7), 124.
  * Balasubramanian  _et al._ [2022] V. Balasubramanian, P. Caputa, J. M. Magan, and Q. Wu, Quantum chaos and the complexity of spread of states, Phys. Rev. D 106, 046007 (2022).
  * Erdmenger  _et al._ [2023] J. Erdmenger, S.-K. Jian, and Z.-Y. Xian, Universal chaotic dynamics from krylov space (2023), arXiv:2303.12151 [hep-th] .
  * Prange [1997] R. E. Prange, The spectral form factor is not self-averaging, Phys. Rev. Lett. 78, 2280 (1997).
  * Saad  _et al._ [2019] P. Saad, S. H. Shenker, and D. Stanford, A semiclassical ramp in syk and in gravity (2019), arXiv:1806.06840 [hep-th] .
  * Barbón and Rabinovici [2014] J. Barbón and E. Rabinovici, Geometry and quantum noise, Fortschr. Phys 62, 626 (2014).
  * Gardiner and Zoller [2000] C. W. Gardiner and P. Zoller,  _Quantum Noise_ , 2nd ed., edited by H. Haken (Springer, 2000).
  * Argaman  _et al._ [1993a] N. Argaman, Y. Imry, and U. Smilansky, Semiclassical analysis of spectral correlations in mesoscopic systems, Phys. Rev. B 47, 4440 (1993a).
  * Argaman  _et al._ [1993b] N. Argaman, F.-M. Dittes, E. Doron, J. P. Keating, A. Y. Kitaev, M. Sieber, and U. Smilansky, Correlations in the actions of periodic orbits derived from quantum chaos, Phys. Rev. Lett. 71, 4326 (1993b).
  * Eckhardt and Main [1995] B. Eckhardt and J. Main, Semiclassical form factor of matrix element fluctuations, Phys. Rev. Lett. 75, 2300 (1995).
  * Schiulaz  _et al._ [2020] M. Schiulaz, E. J. Torres-Herrera, F. Pérez-Bernal, and L. F. Santos, Self-averaging in many-body quantum systems out of equilibrium: Chaotic systems, Phys. Rev. B 101, 174312 (2020).
  * Torres-Herrera  _et al._ [2020a] E. J. Torres-Herrera, G. De Tomasi, M. Schiulaz, F. Pérez-Bernal, and L. F. Santos, Self-averaging in many-body quantum systems out of equilibrium: Approach to the localized phase, Phys. Rev. B 102, 094310 (2020a).
  * Torres-Herrera  _et al._ [2020b] E. J. Torres-Herrera, I. Vallejo-Fabila, A. J. Martínez-Mendoza, and L. F. Santos, Self-averaging in many-body quantum systems out of equilibrium: Time dependence of distributions, Phys. Rev. E 102, 062126 (2020b).
  * Hammerich  _et al._ [1989] A. D. Hammerich, J. G. Muga, and R. Kosloff, Time-dependent quantum-mechanical approaches to the continuous spectrum: Scattering resonances in a finite box, Isr. J. Chem. 29, 461 (1989).
  * Wall and Neuhauser [1995] M. R. Wall and D. Neuhauser, Extraction, through filter-diagonalization, of general quantum eigenvalues or classical normal mode frequencies from a small number of residues or a short-time segment of a signal. i. theory and application to a quantum-dynamics model, J. Chem. Phys. 102, 8011 (1995).
  * Mandelshtam and Taylor [1997] V. A. Mandelshtam and H. S. Taylor, A low-storage filter diagonalization method for quantum eigenenergy calculation or for spectral analysis of time signals, J. Chem. Phys. 106, 5085 (1997).
  * Tameshtit and Sipe [1992] A. Tameshtit and J. E. Sipe, Survival probability and chaos in an open quantum system, Phys. Rev. A 45, 8280 (1992).
  * Cornelius  _et al._ [2022] J. Cornelius, Z. Xu, A. Saxena, A. Chenu, and A. del Campo, Spectral filtering induced by non-hermitian evolution with balanced gain and loss: Enhancing quantum chaos, Phys. Rev. Lett. 128, 190402 (2022).
  * Matsoukas-Roubeas  _et al._ [2023a] A. S. Matsoukas-Roubeas, F. Roccati, J. Cornelius, Z. Xu, A. Chenu, and A. del Campo, Non-hermitian hamiltonian deformations in quantum mechanics, JHEP 2023 (1), 60.
  * Matsoukas-Roubeas  _et al._ [2023b] A. S. Matsoukas-Roubeas, T. Prosen, and A. del Campo, Quantum chaos and coherence: Random parametric quantum channels (2023b), arXiv:2305.19326 [quant-ph] .
  * Zhou  _et al._ [2023] Y.-N. Zhou, T.-G. Zhou, and P. Zhang, Universal properties of the spectral form factor in open quantum systems (2023), arXiv:2303.14352 [cond-mat.stat-mech] .
  * Li  _et al._ [2021] J. Li, T. Prosen, and A. Chan, Spectral statistics of non-hermitian matrices and dissipative quantum chaos, Phys. Rev. Lett. 127, 170602 (2021).
  * Vikram and Galitski [2022] A. Vikram and V. Galitski, Exact universal bounds on quantum dynamics and fast scrambling (2022), arXiv:2212.14021 [quant-ph] .
  * Breuer and Petruccione [2002] H. P. Breuer and F. Petruccione,  _The Theory of Open Quantum Systems_ (Oxford University Press, 2002).
  * Watrous [2018] J. Watrous,  _The Theory of Quantum Information_ (Cambridge University Press, 2018).
  * Forrester [2010] P. Forrester,  _Log-Gases and Random Matrices (LMS-34)_, London Mathematical Society Monographs (Princeton University Press, 2010).
  * del Campo and Takayanagi [2020] A. del Campo and T. Takayanagi, Decoherence in Conformal Field Theory, JHEP 2020 (2), 170.
  * Gisin [1984] N. Gisin, Quantum measurements and stochastic processes, Phys. Rev. Lett. 52, 1657 (1984).
  * Adler [2003] S. L. Adler, Weisskopf-wigner decay theory for the energy-driven stochastic schrödinger equation, Phys. Rev. D 67, 025007 (2003).
  * Milburn [1991] G. J. Milburn, Intrinsic decoherence in quantum mechanics, Phys. Rev. A 44, 5401 (1991).
  * Bassi and Ghirardi [2003] A. Bassi and G. Ghirardi, Dynamical reduction models, Phys. Rep 379, 257 (2003).
  * Bassi  _et al._ [2013] A. Bassi, K. Lochan, S. Satin, T. P. Singh, and H. Ulbricht, Models of wave-function collapse, underlying theories, and experimental tests, Rev. Mod. Phys. 85, 471 (2013).
  * Egusquiza  _et al._ [1999] I. L. Egusquiza, L. J. Garay, and J. M. Raya, Quantum evolution according to real clocks, Phys. Rev. A 59, 3236 (1999).
  * Egusquiza and Garay [2003] I. n. L. Egusquiza and L. J. Garay, Real clocks and the zeno effect, Phys. Rev. A 68, 022104 (2003).
  * Xu  _et al._ [2019] Z. Xu, L. P. García-Pintos, A. Chenu, and A. del Campo, Extreme decoherence and quantum chaos, Phys. Rev. Lett. 122, 014103 (2019).
  * Verlinde [2020] H. Verlinde, ER $=$ EPR revisited: On the entropy of an einstein-rosen bridge (2020), arXiv:2003.13117 [hep-th] .
  * Verlinde [2021] H. Verlinde, Deconstructing the wormhole: Factorization, entanglement and decoherence (2021), arXiv:2105.02142 [hep-th] .
  * Goto  _et al._ [2021] K. Goto, Y. Kusuki, K. Tamaoka, and T. Ugajin, Product of random states and spatial (half-)wormholes, JHEP 2021 (10), 205.
  * Carmichael [2009] H. Carmichael,  _Statistical Methods in Quantum Optics 2: Non-Classical Fields_ , Theoretical and Mathematical Physics (Springer Berlin Heidelberg, 2009).
  * Brody and Graefe [2012] D. C. Brody and E.-M. Graefe, Mixed-state evolution in the presence of gain and loss, Phys. Rev. Lett. 109, 230405 (2012).
  * Ashida  _et al._ [2020] Y. Ashida, Z. Gong, and M. Ueda, Non-hermitian physics, Adv. Phys. 69, 249 (2020).
  * Lidar  _et al._ [2006] D. Lidar, A. Shabani, and R. Alicki, Conditions for strictly purity-decreasing quantum markovian dynamics, Chem. Phys. 322, 82 (2006), real-time dynamics in complex quantum systems.
  * Gregoratti and Werner [2003] M. Gregoratti and R. F. Werner, Quantum lost and found, J. Mod. Opt. 50, 915 (2003).
  * Bocchieri and Loinger [1957] P. Bocchieri and A. Loinger, Quantum recurrence theorem, Phys. Rev. 107, 337 (1957).
  * Schulman [1978] L. S. Schulman, Note on the quantum recurrence theorem, Phys. Rev. A 18, 2379 (1978).
  * Gross  _et al._ [2020a] D. J. Gross, J. Kruthoff, A. Rolph, and E. Shaghoulian, $t\overline{T}$ in ${\mathrm{ads}}_{2}$ and quantum mechanics, Phys. Rev. D 101, 026011 (2020a).
  * Gross  _et al._ [2020b] D. J. Gross, J. Kruthoff, A. Rolph, and E. Shaghoulian, Hamiltonian deformations in quantum mechanics, $t\overline{T}$, and the syk model, Phys. Rev. D 102, 046019 (2020b).
  * García-Pintos  _et al._ [2020] L. P. García-Pintos, A. Hamma, and A. del Campo, Fluctuations in extractable work bound the charging power of quantum batteries, Phys. Rev. Lett. 125, 040601 (2020).
  * Brody [2013] D. C. Brody, Biorthogonal quantum mechanics, J. Phys. A: Math. Theor. 47, 035305 (2013).
  * Gyamfi [2020] J. A. Gyamfi, Fundamentals of quantum mechanics in liouville space, Eur. J. Phys. 41, 063002 (2020).


# Fundamental Frequency Feature Normalization and
Data Augmentation for Child Speech Recognition

###### Abstract

Automatic speech recognition (ASR) systems for young children are needed due
to the importance of age-appropriate educational technology. Because of the
lack of publicly available young child speech data, feature extraction
strategies such as feature normalization and data augmentation must be
considered to successfully train child ASR systems. This study proposes a
novel technique for child ASR using both feature normalization and data
augmentation methods based on the relationship between formants and
fundamental frequency ($f_{o}$). Both the $f_{o}$ feature normalization and
data augmentation techniques are implemented as a frequency shift in the Mel
domain. These techniques are evaluated on a child read speech ASR task. Child
ASR systems are trained by adapting a BLSTM-based acoustic model trained on
adult speech. Using both $f_{o}$ normalization and data augmentation results
in a relative word error rate (WER) improvement of 19.3% over the baseline
when tested on the OGI Kids' Speech Corpus, and the resulting child ASR system
achieves the best WER currently reported on this corpus.

Index Terms—  child speech recognition, fundamental frequency, feature
normalization, data augmentation

##  1 Introduction

The development of effective child automatic speech recognition (ASR) systems
has become important in recent years. For example, the advancement of child
ASR can facilitate the development of teaching and assessment tools for
children in educational settings [1, 2, 3] using interactive systems such as
social robots [4, 5, 6]. This is especially relevant for kindergarten-aged
children who are just learning to read, write, or type and rely on speech to
interact with technology. Yet, ASR systems for young children still perform
quite poorly when compared to adult ASR [6, 7].

One of the major hurdles facing the development of effective child ASR systems
is the lack of publicly available young child speech databases. This is
especially a concern in an era where deep learning, which requires many hours
of training data, is rapidly becoming the primary method of developing ASR
systems, using data intensive acoustic models such as bidirectional long-short
term memory (BLSTM) networks. Hence, many child ASR systems complement young
child speech with older child speech or even adult speech for training data.
However, there is a large acoustic mismatch between child and adult speech,
further complicated by the fact that children's speech acoustics change quite
dramatically as they grow [8, 9]. These changes include formants and
fundamental frequency ($f_{o}$) [8, 9], two defining features of the speech
signal, especially for vowels. Furthermore, as the age difference between
training and testing speakers grows, ASR performance degrades rapidly [7].

One common strategy to account for the acoustic mismatch between speakers is
frequency normalization. This approach attempts to warp the frequency spectra
of an utterance given a normalization factor for the utterance and a target
acoustic space. For instance, vocal tract length normalization (VTLN) warps
the frequency spectra using a maximum likelihood approach and can be
implemented in several ways with varying degrees of success [10, 11, 12]. An
alternative approach is to use acoustically relevant speech parameters as
normalization factors such as subglottal resonances (SGRs) [13], the third
spectral peak or formant frequency [14], and $f_{o}$ [15].

Another strategy is to augment the training data by creating additional
speech-like features for training data. There are a number of ways to
implement this augmentation such as manipulating the frequency scaling or
adding noise [16, 17]. While data augmentation has not been as readily
explored for child speech compared to adult speech, some techniques that have
been evaluated include adding noise and reverberation [18] and using out-of-
domain adult data [19].

In our previous study, we proposed an $f_{o}$-based normalization technique
for child ASR [15]. In that study, a number of ASR systems were trained using
speech from children of various ages and tested using kindergarten-aged
children. While that study demonstrated that $f_{o}$ normalization was
effective for child ASR, the experiments performed assumed that only older
child speech was available to train a young child ASR system. In a more
practical situation, we may expect that some amount of in-domain child speech
would be available as training data. Furthermore, we expect to be able to
generate additional training data through effective data augmentation
techniques.

In this study, we propose a new data augmentation method for training child
ASR systems based on the $f_{o}$ normalization method we proposed in [15].
This method is capable of generating speech-like features that adhere to the
physical properties of speech defined by the relationship between vowel
formants and $f_{o}$. We show that this data augmentation method is capable of
improving child ASR systems adapted from BLSTM acoustic models trained on
adult speech. Additionally, we demonstrate that this data augmentation method
can be used alongside $f_{o}$ normalization for further improvement.

The remainder of the paper is organized as follows. Section 2 reviews the
$f_{o}$ normalization technique proposed in [15] and formulates the data
augmentation technique. Section 3 describes the databases and experimental
setup. Section 4 presents the experimental results. Section 5 concludes the
paper with a summary and considerations for future work.

##  2 Normalization Technique

Fig. 1: Mel filter bank outputs of an 18 year old male (solid) and 7 year old
male (dashed) saying the vowel /i/, computed with 15 filters and a frequency
range of 20 Hz to 6 kHz. The filter outputs are computed both without (left)
and with (right) $f_{o}$ normalization. When normalization is applied, default
$f_{o}$ is chosen to be $f_{o,def}=100$ Hz. The 18 year old male had
$f_{o,utt}=106$ Hz, and the 7 year old male had $f_{o,utt}=270$ Hz. The Mel
filter bank outputs computed with $f_{o}$ normalization are much more well-
aligned than the filter outputs computed without $f_{o}$ normalization.

###  2.1 Relationship Between $f_{o}$ and Formants

Several past studies have revealed meaningful connections between $f_{o}$ and
the first several formant frequencies ($F1$, $F2$, $F3$, …) of vowels. In Bark
scale, the tonotopic distances between adjacent formants ($F(x+1)-Fx$ for
$x\in\\{1,2,3,\ldots\\}$), along with the tonotopic distance between the first
formant and $f_{o}$ ($F1-f_{o}$), have proven to be effective at modeling
human vowel perception and representing the vowel space [20, 21]. Furthermore,
these tonotopic distances can be equivalently represented by the difference
between any formant and $f_{o}$ ($Fx-f_{o}$ for $x\in\\{1,2,3,\ldots\\}$).
This suggests that the vowel space can be modeled as a linear relationship (in
Bark scale) between formants and $f_{o}$. Additionally, studies have found
that $f_{o}$ and formant locations depend on one another in both vowel
production [22] and vowel perception [23]. A more detailed examination of
these relationships can be found in [15].

###  2.2 Normalization Procedure

The frequency normalization technique we proposed for kindergarten ASR in [15]
was designed to be applied to any spectral-based feature such as Mel-frequency
cepstral coefficients (MFCCs). The linear relationship between $f_{o}$ and
vowel formants is exploited to formulate a normalization technique using
median $f_{o}$ as the only normalization factor. As such, the technique
attempts to map a speaker's acoustic space, governed by the speaker's $f_{o}$,
to some target space, defined by a predetermined target $f_{o}$. Notably, Mel
scale was used in [15] instead of Bark scale as the two scales are highly
correlated, but either scale can be used in practice. For consistency, this
study will also use the Mel scale.

To perform the $f_{o}$-based normalization, a default $f_{o}$, denoted as
$f_{o,def}$, must first be chosen. Before feature computation, $f_{o}$
extraction is performed across the entire utterance using a reliable $f_{o}$
detection algorithm. For this study, we will use the multi-band summary
correlogram (MBSC) pitch detection algorithm [24].

The median $f_{o}$ across all voiced frames of the utterance, denoted as
$f_{o,utt}$, is then chosen as the normalization factor for the utterance. The
discrete Fourier transform (DFT) of the feature extraction procedure is warped
as follows:

$f_{norm}=f_{orig}-(f_{o,utt}-f_{o,def})$ (1)

where all frequencies are in Mel scale, $f_{norm}$ is a normalized frequency
corresponding to some DFT index, and $f_{orig}$ is the frequency from the
original speech spectrum mapped to $f_{norm}$.

An example of this $f_{o}$ normalization is shown in Figure 1. The Mel filter
bank outputs of an 18 year old male and a 7 year old male saying the vowel /i/
are displayed both with and without $f_{o}$ normalization. When both
utterances are normalized to $f_{o,def}=100$ Hz, the Mel filter bank outputs
become more similar.

###  2.3 Data Augmentation Procedure

While the $f_{o}$ normalization procedure attempted to reduce variability
between speakers by using Eq. 1, fixing $f_{o,def}$ to a default value, and
adjusting $f_{o,utt}$, an alternative procedure can use Eq. 1 to create
variability rather than reduce it. This can be accomplished by extracting
features multiple times from the same speech utterance while adjusting
$f_{o,def}$. By perturbing the feature extraction, we can augment the training
data by generating acoustic features that are consistent with the structure of
speech defined by the tonotopic distances. This can be especially useful for
training deep neural networks that require a large amount of speech data. We
will refer to this technique as $f_{o}$ perturbation. When combined with the
$f_{o}$ normalization procedure, we can simultaneously remove larger inter-
speaker variabilities while perturbing the features for additional training
data.

##  3 Database and Experimental Setup

###  3.1 Database

Two child speech databases were used in this study. The first one was the OGI
Kids' Speech Corpus [25]. This corpus contains approximately 100 speakers per
American educational grade level, from kindergarten to 10th grade. Utterances
were recorded with a sampling rate of 16 kHz (8 kHz bandwidth). Both scripted
and spontaneous styles of speech were recorded from each speaker. In this
study, we used the sentence utterances from the scripted speech recordings,
which consisted of a total of 10,072 sentence utterances from children in
grades kindergarten through 5th grade. Approximately 70% of the utterances in
each grade were used as training data for a total of 7,051 training
utterances. The remaining utterances were used for testing.

The second child speech database was the CMU Kids Corpus [26]. This corpus
contains 76 speakers between 1st and 3rd grade with two additional speakers
from kindergarten or 6th grade for a total of 78 speakers. Utterances were
recorded with a sampling rate of 16 kHz. A total of 5,180 read sentence
utterances were recorded across these speakers. Exactly 70% of the utterances
in this corpus were used as training data for a total of 3,626 training
utterances. The remaining utterances were used for testing.

For adult data, the LibriSpeech ASR Corpus was used [27]. For our experiments,
we used all the training data from the corpus, which contains 960 hours of
adults reading audio books in clean and noisy conditions, for adult model
training.

###  3.2 Feature Extraction

The baseline features used for the ASR experiments were 13-dimensional MFCCs
with a window size of 25 ms and shift of 10 ms. These MFCCs were extracted
using a 512-point DFT, 23 Mel filters, and bandwidth from 20 Hz to 8 kHz.

The second set of features was similar to the baseline features except the DFT
was normalized by the procedure presented in Section 2.2. The default $f_{o}$
was chosen to be $f_{o,def}=100$ Hz representing an adult male $f_{o}$, and
$f_{o,utt}$ was chosen to be median $f_{o}$ across the utterance, estimated
using MBSC pitch detection. As the value of $f_{o,utt}$ reached as high as 300
Hz, the bandwidth for these features was limited to 6.2 kHz to compensate for
the frequency shift upwards. Notably, this limits the maximum frequency shift
of the $f_{o}$ normalization procedure to approximately 250 Mels. Both
baseline and $f_{o}$ normalized features were extracted from all utterances.

When applying $f_{o}$ perturbation, $f_{o,def}$ was adjusted such that the DFT
was shifted by $\pm 20$, $\pm 40$, and $\pm 60$ Mels. Along with the original
features, this multiplies the amount of training data by 7. Data augmentation
was applied to the CMU Kids and OGI Kids' training datasets.

Implementation of the feature extraction is relatively simple when using Eq. 1
to compute the DFT frequency shift. We let $f_{o,def}\in\\{$58.52, 72.10,
85.93, 100.00, 114.32, 128.90, 143.74$\\}$ Hz where every value in the set is
used if performing data augmentation and only 100 Hz is used if performing a
standard feature extraction. Similarly, we let $f_{o,utt}=100$ Hz when
performing a non-normalized feature extraction and choose $f_{o,utt}$ as the
median $f_{o}$ across the utterance when normalizing.

###  3.3 Experimental Setup

For both the baseline and $f_{o}$ normalized features, an adult ASR system was
first trained using the LibriSpeech training set. The acoustic model was a
3-layer BLSTM network with 512 cells in each direction followed by a feed-
forward layer that mapped the output of the BLSTMs to senone probabilities.
The input to the BLSTM was 7 frames (3 frames forward and backward in time) to
form a 91-dimensional input for the acoustic model. The output was
approximately 5,700 senone probabilities. The acoustic model was based on
PyKaldi2 [28], while decoding used the Kaldi Speech Recognition Toolkit [29].

The adult ASR systems were adapted to child speech using either the CMU Kids
or OGI Kids' training data. The adult ASR system trained with $f_{o}$
normalization was adapted using child speech features that were extracted with
$f_{o}$ normalization. Similarly, the adult ASR system trained without
normalization was adapted using features that were extracted without
normalization. Additionally, data augmentation using $f_{o}$ perturbation was
applied on the child training datasets.

A 4-gram language model (LM) trained on Project Gutenburg books was chosen for
decoding. This LM is one of the default language models included in Kaldi's
LibriSpeech recipe [29]. Adapted ASR systems were evaluated using the
corresponding CMU Kids or OGI Kids' testing datasets. The ASR system trained
with $f_{o}$ normalization was tested using features extracted with $f_{o}$
normalization.

Table 1: Word error rates (WERs) of the child ASR experiment using a BLSTM-based acoustic model adapted from adult speech. The left two columns indicate whether $f_{o}$ normalization (``Norm?'') and data augmentation using $f_{o}$ perturbation (``Aug?'') were used. WERs for both CMU Kids and OGI Kids' are reported in the latter columns. Norm? | Aug? | CMU Kids | OGI Kids'
---|---|---|---
No | No | 16.88 | 6.84
Yes | No | 16.93 | 6.50
No | Yes | 16.63 | 5.85
Yes | Yes | 16.47 | 5.52

##  4 Results and Discussion

The results of the child ASR experiments described in Section 3 are shown in
Table 1. The top row displays the word error rate (WER) of the baseline system
(i.e., no normalization or augmentation). The second row displays the WER of
the system using $f_{o}$ normalization. The third row displays the system
adapted with $f_{o}$ perturbed child speech features. Finally, the last row
uses both $f_{o}$ normalization and $f_{o}$ perturbed adaptation data.
Additionally, we performed the experiment once more using the acoustic model
trained only on LibriSpeech. This system achieved a WER of 37.49% for CMU Kids
and 59.70% for OGI Kids', significantly worse than any WER in Table 1, which
demonstrates the importance of child speech adaptation data.

When using adaptation and only applying $f_{o}$ normalization, the performance
of the OGI Kids' system saw a small improvement over the baseline, which has
no normalization or data augmentation, but not enough to be significant.
Applying $f_{o}$ perturbation to the training data, a more substantial
improvement was achieved from 6.84% to 5.85%. However, when using the CMU Kids
system, no major improvements were observed.

Using both $f_{o}$ normalization and $f_{o}$ perturbation resulted in the best
performing ASR system for both testing sets. The OGI Kids' testing set saw a
relative WER improvement of 19.3%, reducing the WER to 5.52%, and this result
is statistically significant at $p<0.001$. However, the CMU Kids testing set
only saw a relative WER improvement of 2.4%.

While the OGI Kids' testing set saw a larger improvement than the CMU Kids
testing set, this result may be expected. We note that the CMU Kids testing
set had a narrower age range (approximately 6-9 years old excluding the two
outlier children) compared to the OGI Kids' testing set (approximately 5-11
years old). The $f_{o}$ normalization method has been shown to produce larger
improvements when the range of ages used in training and testing data is wider
[15]. A similar phenomenon may be occurring for $f_{o}$ perturbation. That is,
since there is less variability in the CMU Kids testing set, adding additional
variability to the training set through $f_{o}$ perturbation was unnecessary
to train the BLSTM. Meanwhile, with the larger variability of the OGI Kids'
dataset, both $f_{o}$ normalization and $f_{o}$ perturbation proved helpful.
Furthermore, these techniques may extend to ASR systems using the full OGI
Kids' scripted speech dataset, which has been reported to have a WER of 10.8%
[18]. Preliminary experiments suggest that $f_{o}$ perturbation also performs
better than VTLN-based data augmentation.

##  5 Conclusion

This study proposes a new data augmentation method for training child ASR
systems based on the $f_{o}$ normalization method proposed in [15]. Both
normalization and data augmentation methods adhere to the physical properties
of speech defined by the relationship between vowel formants and $f_{o}$. The
two methods can be formulated as a simple shift of DFT bin frequencies in the
Mel domain and are implemented by manipulating $f_{o,def}$ and $f_{o,utt}$ in
Eq. 1. Child ASR systems were trained using these methods by adapting from a
BLSTM acoustic model trained on adult speech. When using both $f_{o}$
normalization and $f_{o}$ perturbation, a 19.3% relative improvement was
observed on the OGI Kids' Speech Corpus. However, a less substantial
improvement was observed on the CMU Kids Corpus. This suggests, that both
$f_{o}$ normalization and $f_{o}$ perturbation are more effective when the age
range of the speakers is large.

There are a number of possible directions for future work. We plan to evaluate
these methods on ASR systems for both children and adults for a situation with
increased age and $f_{o}$ variability between speakers. We also plan to apply
these systems to other child ASR applications such as conversational speech,
educational applications, and clinical applications.

## References

  * [1] J. Tepperman, J. Silva, A. Kazemzadeh, H. You, S. Lee, A. Alwan, and S. Narayanan,  ``Pronunciation Verification of Children's Speech for Automatic Literacy Assessment,''  in Proc. INTERSPEECH, 2006, pp. 845–848.
  * [2] H. T. Bunnell, D. M. Yarrington, and J. B. Polikoff,  ``STAR: Articulation Training for Young Children,''  in Proc. ICSLP, 2000, pp. 85–88.
  * [3] G. Yeung, A. Afshan, K. E. Ozgun, K. Kaewtip, S. M. Lulich, and A. Alwan,  ``Predicting Clinical Evaluations of Children's Speech with Limited Data Using Exemplar Word Template References,''  in Proc. SLaTE, 2017, pp. 161–166.
  * [4] S. Spaulding, H. Chen, S. Ali, M. Kulinski, and C. Breazeal,  ``A Social Robot System for Modeling Children's Word Pronunciation,''  in Proc. AAMAS, 2018, pp. 1658–1666.
  * [5] G. Yeung, A. L. Bailey, A. Afshan, M. Q. Pérez, A. Martin, S. Spaulding, H. W. Park, A. Alwan, and C. Breazeal,  ``Towards the Development of Personalized Learning Companion Robots for Early Speech and Language Assessment,''  in Proc. AERA, 2019.
  * [6] J. Kennedy, S. Lemaignan, C. Montassier, P. Lavalade, B. Irfan, F. Papadopoulos, E. Senft, and T. Belpaeme,  ``Child Speech Recognition in Human-Robot Interaction: Evaluations and Recommendations,''  in Proc. ACM/IEEE HRI, 2017, pp. 82–90.
  * [7] G. Yeung and A. Alwan,  ``On the Difficulties of Automatic Speech Recognition for Kindergarten-Aged Children,''  in Proc. INTERSPEECH, 2018, pp. 1661–1665.
  * [8] S. Lee, A. Potamianos, and S. Narayanan,  ``Acoustics of Children's Speech: Developmental Changes of Temporal and Spectral Parameters,''  J. Acoust. Soc. Am., vol. 105, no. 3, pp. 1455–1468, 1999.
  * [9] H. K. Vorperian and R. D. Kent,  ``Vowel Acoustic Space Development in Children: A Synthesis of Acoustic and Anatomic Data,''  J. Speech. Lang. Hear., vol. 50, no. 6, pp. 1510–1545, 2007.
  * [10] G. Stemmer, C. Hacker, S. Steidl, and E. Nöth,  ``Acoustic Normalization of Children's Speech,''  in Proc. EUROSPEECH, 2003, pp. 1313–1316.
  * [11] X. Cui and A. Alwan,  ``MLLR-Like Speaker Adaptation Based on Linearization of VTLN with MFCC Features,''  in Proc. INTERSPEECH, 2005, pp. 273–276.
  * [12] R. Serizel and D. Giuliani,  ``Vocal Tract Length Normalization Approaches to DNN-Based Children's and Adults' Speech Recognition,''  in Proc. SLT, 2014, pp. 135–140.
  * [13] J. Guo, R. Paturi, G. Yeung, S. M. Lulich, H. Arsikere, and A. Alwan,  ``Age-Dependent Height Estimation and Speaker Normalization for Children's Speech Using the First Three Subglottal Resonances,''  in Proc. INTERSPEECH, 2015, pp. 1665–1669.
  * [14] X. Cui and A. Alwan,  ``Adaptation of Children's Speech with Limited Data Based on Formant-Like Peak Alignment,''  Comput. Speech Lang., vol. 20, no. 4, pp. 400–419, 2006.
  * [15] G. Yeung and A. Alwan,  ``A Frequency Normalization Technique for Kindergarten Speech Recognition Inspired by the Role of fo in Vowel Perception,''  in Proc. INTERSPEECH, 2019, pp. 6–10.
  * [16] X. Cui, V. Goel, and B. Kingsbury,  ``Data augmentation for deep neural network acoustic modeling,''  in Proc. ICASSP, 2014, pp. 5582–5586.
  * [17] D. S. Park, W. Chan, Y. Zhang, C.-C. Chiu, B. Zoph, E. D. Cubuk, and Q. V. Le,  ``SpecAugment: A simple data augmentation method for automatic speech recognition,''  in Proc. INTERSPEECH, 2019, pp. 2613–2617.
  * [18] F. Wu, L. P. Garcia, D. Povey, and S. Khudanpur,  ``Advances in automatic speech recognition for child speech using factored time delay neural network,''  in Proc. INTERSPEECH, 2019, pp. 1–5.
  * [19] J. Fainberg, P. Bell, M. Lincoln, and S. Renals,  ``Improving children's speech recognition through out-of-domain data augmentation,''  in Proc. of INTERSPEECH, 2016, pp. 1598–1602.
  * [20] A. K. Syrdal and H. S. Gopal,  ``A Perceptual Model of Vowel Recognition Based on the Auditory Representation of American English Vowels,''  J. Acoust. Soc. Am., vol. 79, no. 4, pp. 1086–1100, 1986.
  * [21] H. Traunmüller,  ``Perceptual Dimension of Openness in Vowels,''  J. Acoust. Soc. Am., vol. 69, no. 5, pp. 1465–1475, 1981.
  * [22] S. Barreda and T. M. Nearey,  ``The Perception of Formant-Frequency Range is Affected by Veridical and Judged Fundamental Frequency,''  in Proc. Meetings on Acoustics, 2013, vol. 19, p. 060197.
  * [23] S. Barreda and T. M. Nearey,  ``The Direct and Indirect Roles of Fundamental Frequency in Vowel Perception,''  J. Acoust. Soc. Am., vol. 131, no. 1, pp. 466–477, 2012.
  * [24] L. N. Tan and A. Alwan,  ``Multi-Band Summary Correlogram-Based Pitch Detection for Noisy Speech,''  Speech Comm., vol. 55, no. 7-8, pp. 841–856, 2013.
  * [25] K. Shobaki, J.-P. Hosom, and R. A. Cole,  ``The OGI Kids' Speech Corpus and Recognizers,''  in Proc. ICSLP, 2000, pp. 258–261.
  * [26] M. Eskenazi, J. Mostow, and D. Graff,  ``The CMU Kids Speech Corpus LDC97S63,'' 1997.
  * [27] V. Panayotov, G. Chen, D. Povey, and S. Khudanpur,  ``Librispeech: An ASR corpus based on public domain audio books,''  in Proc. IEEE ICASSP, 2015, pp. 5206–5210.
  * [28] L. Lu, X. Xiao, Z. Chen, and Y. Gong,  ``PyKaldi2: Yet another speech toolkit based on Kaldi and PyTorch,''  arXiv:1907.05955, 2019.
  * [29] D. Povey, A. Ghoshal, G. Boulianne, L. Burget, O. Glembek, N. Goel, M. Hannemann, P. Motlíček, Y. Qian, P. Schwarz, J. Silovský, G. Stemmer, and K. Veselý,  ``The Kaldi Speech Recognition Toolkit,''  in Proc. IEEE ASRU, 2011.