# Birth of Rapidly Spinning, Overmassive Black Holes in the Early Universe

Kohei Inayoshi Kavli Institute for Astronomy and Astrophysics, Peking
University, Beijing 100871, China  Kohei Ichikawa Global Center for Science
and Engineering, Faculty of Science and Engineering, Waseda University
Department of Physics, School of Advanced Science and Engineering, Faculty of
Science and Engineering, Waseda University, 3-4-1, Okubo, Shinjuku, Tokyo
169-8555, Japan

###### Abstract

The James Webb Space Telescope (JWST) has unveiled numerous massive black
holes (BHs) in faint, broad-line active galactic nuclei (AGNs). The discovery
highlights the presence of dust-reddened AGN populations, referred to as
“little red dots (LRDs)”, more abundant than X-ray selected AGNs, which are
less influenced by obscuration. This finding indicates that the cosmic growth
rate of BHs within this population does not decrease but rather increases at
higher redshifts beyond $z\sim 6$. The BH accretion rate density deduced from
their luminosity function is remarkably higher than that from other AGN
surveys in X-ray and infrared bands. To align the cumulative mass density
accreted to BHs with the observed BH mass density at $z\simeq 4-5$, as derived
from the integration of the BH mass function, the radiative efficiency must be
doubled from the canonical 10% value, achieving significance beyond the
$>2\sigma$ confidence level. This suggests the presence of rapid spins with
96% of the maximum limit among these BHs, maintained by prolonged mass
accretion instead of chaotic accretion with randomly oriented inflows.
Moreover, we derive an upper bound for the stellar mass of galaxies hosting
these LRDs, ensuring consistency with galaxy formation in the standard
cosmological model, where the host stellar mass is limited by the available
baryonic reservoir. Our analysis gives a lower bound for the BH-to-galaxy mass
ratio that exceeds the typical value known in the nearby universe and aligns
with that for JWST-detected unobscured AGNs. Accordingly, we propose a
hypothesis that the dense, dust-rich environments within LRDs facilitate the
emergence of rapidly spinning and overmassive BH populations during the epoch
of reionization. This scenario predicts a potential association between
relativistic jets and other high-energy phenomena with overmassive BHs in the
early universe.

Galaxy formation (595); High-redshift galaxies (734); Quasars (1319);
Supermassive black holes (1663)

##  1 introduction

The cosmic evolution of massive black hole (BH) populations is predominantly
driven by mass accretion, powering active galactic nuclei (AGNs) (e.g.,
Lynden-Bell, 1969). Multi-wavelength observations have consistently shown that
AGN activity peaks around $z\sim 2$ and declines toward higher redshifts
(e.g., Ueda et al., 2014; Delvecchio et al., 2014). The analysis of AGN
activity offers insights into the radiative efficiency of accreting BHs by
comparing it with the local mass density of relic BHs (Soltan, 1982; Yu &
Tremaine, 2002).

Recent observations by the James Webb Space Telescope (JWST) have revealed a
new category of dust-reddened, broad-line AGNs, often referred to as “little
red dots” (hereafter LRDs, Matthee et al., 2023). These AGNs are characterized
by their compact morphology and moderate dust obscuration
($A_{\mathrm{V}}\approx 3$) in the spectra (Barro et al., 2023; Kocevski et
al., 2023; Harikane et al., 2023; Labbe et al., 2023). Investigations into the
AGN luminosity function of LRDs at $z=4$–$8$ found an abundance of $\Phi\sim
10^{-5}-10^{-4}~{}{\rm Mpc}^{-3}~{}{\rm mag}^{-1}$ in the observed UV absolute
magnitude range of $-22\lesssim M_{\rm UV}\lesssim-18$ (e.g., Kokorev et al.,
2024), significantly exceeding those predicted by extrapolations from the
unobscured AGN luminosity function in ground-based surveys (e.g., Niida et
al., 2020). Moreover, spectroscopic analysis of LRDs, facilitating direct
measurement of broad H$\alpha$ emissions – a tracer of AGN activity (Greene &
Ho, 2005) – has led to the construction of the AGN bolometric luminosity
function for LRDs. This result suggests that in contrast to expectations based
on AGN surveys in the pre-JWST era, the cosmic growth rate of BHs within this
AGN population does not decline but appears to increase at higher redshifts
($z>6$).

In this Letter, compiling data from extensive high-$z$ AGN and LRD surveys, we
constrain the radiative efficiency of BHs in dominant LRD populations by
comparing the observed BH mass density at $z\simeq 4-5$ to the mass accreted
to BHs calculated from the BH growth rate, i.e., the Soltan-Paczyński argument
at the epoch of reionization. This analysis suggests a radiative efficiency
higher than the canonical 10% value and favors rapid spins of these BHs.
Furthermore, we establish an upper limit for the stellar mass of galaxies
harboring these LRDs, and give a lower bound for the BH-to-galaxy mass that
exceeds the typical value known in the nearby universe and aligns with that
for JWST-detected unobscured AGNs. Consequently, we propose a hypothesis that
the dust-rich environments in LRDs promote the emergence of rapidly spinning
and overmassive BH populations.

##  2 Soltan Argument at $z\geq 5$

Figure 1: Bolometric AGN luminosity functions at $z\simeq 5$. The luminosity
function data obtained from different surveys are shown: the rest-UV-selected
quasars (Niida et al., 2020), the X-ray selected AGNs (Ueda et al., 2014), and
dust-reddened AGNs reported as “little red dots” identified with JWST
photometry and slitless spectroscopy (Matthee et al., 2023; Kokorev et al.,
2024; Greene et al., 2023).  Figure 2: The cosmic BH accretion rate density
(BHAD) as a function of redshift. Each data point and curve represent BHADs
estimated under the assumption of a 10% radiative efficiency ($\epsilon_{\rm
rad}=0.1$) for the three different populations, including LRDs (Matthee et
al., 2023; Kokorev et al., 2024; Greene et al., 2023), X-ray selected AGNs
(Ananna et al., 2019; Peca et al., 2023; Pouliasis et al., 2024), and mid-
infrared selected AGNs (Delvecchio et al., 2014). For comparison, the cosmic
SFRD scaled by a factor of 3,000 is overlaid (Harikane et al., 2022). The BHAD
attributed to LRDs remains significantly dominant at $z>6$.

Early studies of LRDs have indicated that the observed values of $M_{\rm UV}$
are much fainter than the intrinsic UV magnitudes due to dust extinction
(e.g., Kocevski et al., 2023; Matthee et al., 2023; Greene et al., 2023;
Kokorev et al., 2024). To address the underlying AGN activity, these studies
estimated bolometric AGN luminosity from rest-optical emissions. Matthee et
al. (2023) and Greene et al. (2023) conducted JWST/NIRSpec observations on
LRDs and converted the measured H$\alpha$ luminosity to bolometric luminosity,
while Kokorev et al. (2024) relied on continuum luminosity $L_{5100}$ for a
bolometric luminosity estimator by assuming that all the continuum flux
originates from the AGN. The AGN bolometric luminosity derived from the
H$\alpha$ luminosity, particularly its broad-line component emitted from fast-
moving clouds near the AGN, is more accurate than using dust-dereddened
continuum (see also discussion in Section 4.3).

Figure 3: Left: Cosmic evolution of the BH mass density in a comoving volume.
At $z\sim 5$, the BH mass density is derived from the integration of the BHMF
for LRDs (red symbols). From this point, the mass density grows toward lower
redshifts following the BHAD deduced from known AGN populations with a 10%
radiative efficiency (solid curve; Ueda et al. 2014) and reaches the density
of relic BHs in the nearby universe (Shankar et al., 2009). At $z>5$, the
cumulative mass accreted to BHs during the LRD phase,
$\Delta\rho_{\bullet}\equiv{\rm BHAD}\times\Delta t$ inferred from their
bolometric luminosity function over a time span (blue, green, and purple
symbols), assuming a 10% radiative efficiency, substantially exceed the
observed mass density at $z\simeq 5$ as well as the predictions from a BH
growth model calibrated with UV and X-ray selected AGN luminosity function
(dotted curve; Li et al., 2023). Data for LRDs are derived from luminosity
functions and BH mass estimates provided in the literature (circle, Matthee et
al. 2023; diamond, Greene et al. 2023; square, Kokorev et al. 2024). Right:
Summary of the BH mass density and the cumulative mass density during the LRD
phase assuming a 10% radiative efficiency. Shaded areas indicate the BH mass
density $\rho_{\bullet}$ at $z\simeq 5$ (red) and the cumulative mass density
accrued during the LRD stage between $5<z<6$ (blue) and $5<z<8$ (green and
purple). The total cumulative mass within $5<z<8$ is considered under two
different scenarios: one incorporating findings from Greene et al. (2023) and
the other based on Kokorev et al. (2024), to examine the contribution from the
two separated values.

Figure 1 presents the bolometric luminosity function at $z\simeq 5$, combining
data from various sources: rest-UV-selected unobscured quasars (Niida et al.,
2020), X-ray selected AGNs (Ueda et al., 2014), and LRDs identified through
JWST photometry and spectroscopy (Matthee et al., 2023; Kokorev et al., 2024;
Greene et al., 2023). For UV and X-ray selected AGNs, we adopt the bolometric
correction factors calibrated by Duras et al. (2020). The abundance of
unobscured quasars aligns closely with that of X-ray detected AGNs for $L_{\rm
bol}\gtrsim 4\times 10^{45}~{}{\rm erg~{}s}^{-1}$, while the X-ray AGN
abundance further increases at lower luminosity regimes. This suggests that
the obscured AGN fraction increases toward the fainter end (Vito et al.,
2018). In contrast, LRDs exhibit a substantially greater abundance at $L_{\rm
bol}\lesssim 10^{46}~{}{\rm erg~{}s}^{-1}$, nearly one order of magnitude
above that of X-ray AGNs. While the LRD abundance data show some variations
owing to differences in sample size and methods for estimating bolometric
luminosity across the referred studies, the trend of over-abundance relative
to the other AGN populations is consistently observed.

Figure 2 illustrates the BH accretion rate density (BHAD) across various
redshifts, with each data point and curve representing BHADs estimated under
the assumption of a 10% radiative efficiency ($\epsilon_{\rm rad}=0.1$). These
include LRDs (Matthee et al., 2023; Kokorev et al., 2024; Greene et al.,
2023), as well as X-ray selected AGNs (Ananna et al., 2019; Peca et al., 2023;
Pouliasis et al., 2024), and mid-infrared selected AGNs (Delvecchio et al.,
2014). The BHAD estimated from X-ray sources agrees well to that of mid-
infrared AGNs at $z\sim 3$. With the same value of $\epsilon_{\rm rad}=0.1$,
the BHAD inferred from the bolometric luminosity function of LRDs indicates a
persistent or even increasing trend toward higher redshifts ($4<z<8$),
opposite to the declining trend of X-ray selected AGNs.

For comparison, we overlay the cosmic star-formation rate density (SFRD)
scaled by a factor of 3,000 (Harikane et al., 2022). The scaled SFRD matches
well the BHAD based on the mid-infrared selected AGNs at $z\lesssim 3$ and
appears to be consistent with the BHAD of LRDs at $z\simeq 5-6$. On the other
hand, the BHAD attributed to LRDs remains significantly dominant at $z>6$.
This finding, based on the assumption of $\epsilon_{\rm rad}=0.1$, indicates
that rapid growth of BHs at these earlier epochs established a trend of
overmassive BH in terms of the $M_{\bullet}/M_{\star}$ ratio, as observed in
recent JWST AGN studies at $z>6$ (e.g., Harikane et al., 2023; Maiolino et
al., 2023; Pacucci et al., 2023).

Table 1: Significance of the difference between $\rho_{\bullet}(z\simeq 5)$ and $\Delta\rho_{\bullet}(z\simeq 5)$ for different radiative efficiencies. Redshift |  | $\log_{10}\Delta\rho_{\bullet}$ |  |  $p$-value
---|---|---|---|---
|  | $\epsilon_{\rm rad}=0.1$ |  | $\epsilon_{\rm rad}=0.1$ | $\epsilon_{\rm rad}=0.2$ | $\epsilon_{\rm rad}=0.3$ | $\epsilon_{\rm rad}=0.42$
|  | $a_{\bullet}\simeq 0.674$ |  | $a_{\bullet}\simeq 0.674$ | $a_{\bullet}\simeq 0.960$ | $a_{\bullet}\simeq 0.996$ | $a_{\bullet}\simeq 1.00$
$z\simeq 5$ |  | $4.11\pm 0.20$ |  | 0.018 | 0.14 | 0.50 | -
$5<z<8~{}_{\rm G}$ |  | $4.27\pm 0.21$ |  | 0.0083 | 0.048 | 0.21 | -
$5<z<8~{}_{\rm K}$ |  | $4.72\pm 0.24$ |  | 0.0017 | 0.0057 | 0.016 | 0.055

In the left panel of Figure 3, we show the evolution of the BH mass density
within a comoving volume throughout cosmic time. The solid curve represents
the cumulative BH mass density deduced from the AGN bolometric luminosity
functions (primarily X-ray selected populations) at $0<z<5$, under an assumed
10% radiative efficiency (Ueda et al., 2014). This projection agrees closely
with the observed BH mass density at $z\simeq 0$ (Shankar et al., 2009),
concluding the plausibility of the pre-assumed radiative efficiency
($\epsilon_{\rm rad}=0.1$; Soltan, 1982; Yu & Tremaine, 2002). Additionally,
we present the BH mass density directly derived from the integration of the
BHMF for LRDs at $z\simeq 5$, estimated to be $\rho_{\bullet}\simeq
3.3^{+2.5}_{-1.5}\times 10^{3}~{}M_{\odot}~{}{\rm Mpc}^{-3}$ (red symbols;
Matthee et al., 2023; Greene et al., 2023; Kokorev et al., 2024)111We note
that the BHMF from the LRD samples in Kokorev et al. (2024) is derived from
their AGN bolometric luminosity function, assuming an average Eddington ratio
value $\langle\lambda_{\rm Edd}\rangle\simeq 0.3$, which is motivated by the
samples of broad-line RLDs compiled by Greene et al. (2023).. This estimate
remarkably aligns with expectations based on the Soltan argument assuming the
conventional radiative efficiency of 10% at $0<z<5$, further reinforcing the
consistency across the three distinct physical measures.

However, extending the analysis to the universe at $z>5$, while still assuming
the same 10% radiative efficiency, leads to inconsistencies. Specifically, the
cumulative mass of BHs accreted during the LRD phase,
$\Delta\rho_{\bullet}\equiv{\rm BHAD}\times\Delta t$ inferred from their
bolometric luminosity function over a time span at $z>5$, substantially
exceeds the observed mass density at $z\simeq 5$ as well as the predictions
from a BH growth model calibrated with UV and X-ray selected AGN luminosity
functions (dotted curve; Li et al., 2023). This discrepancy raises concerns
about a potential violation of the mass conservation law and suggests the
possible inapplicability of $\epsilon_{\rm rad}=0.1$ for the early universe
beyond $z>5$. Adjusting the radiative efficiency upwards impacts the inferred
BHAD, which follows $\propto(1-\epsilon_{\rm rad})/\epsilon_{\rm rad}$. For
instance, adopting the theoretical upper limit of $\epsilon_{\rm rad}=0.42$
for an extreme Kerr BH with a spin parameter $a_{\bullet}=1$ (e.g., Kerr,
1963; Novikov & Thorne, 1973) resolves the discrepancy between the integrated
BHMF values and the cumulative mass derived from the BHAD.

The right panel of Figure 3 provides a detailed quantitative comparison of BH
mass density values from several studies, using the least-squares method for
fitting $\rho_{\bullet}$ and $\Delta\rho_{\bullet}$ (see also Table 1). While
the results at $z\sim 5$ align across these different studies, a significant
discrepancy is seen in $\Delta\rho_{\bullet}$ at $z\sim 7$ between Greene et
al. (2023) and Kokorev et al. (2024). This divergence is primarily because
Kokorev et al. (2024) examined a larger sample set of LRDs from multiple
survey fields and thus identified more luminous LRDs, substantially impacting
the BHAD estimate. Moreover, Kokorev et al. (2024) included all
photometrically selected LRDs as AGNs due to the lack of spectroscopic
observations. Due to these concerns, we treat these two distinct estimates as
the lower and upper bounds of the cumulative mass density of BHs at $z\sim 7$.

To understand the influence of each contribution of $\Delta\rho_{\bullet}$ on
this analysis and the need for a radiative efficiency beyond the standard 10%
value, we explore three scenarios: one only considering the contribution at
$z\simeq 5$, and two others incorporating the results from Greene et al.
(2023) and Kokorev et al. (2024) at $z\sim 7$. Employing the $t$-test, we
assess the significance of the variance between $\rho_{\bullet}(z\simeq 5)$
and $\Delta\rho_{\bullet}$ for the three cases. The $p$-values, as summarized
in Table 1, indicate that the hypothesis of agreement between the two
quantities at $\epsilon_{\rm rad}=0.1$ is statistically rejected in all the
three cases with a confidence level of $>95\%$ ($p<0.05$). For the scenarios
that include the contribution from $z\sim 7$, a radiative efficiency greater
than 0.2 is concluded with a 95% confidence level. This finding suggests that
the majority of BHs within LRDs or similarly gas/dust-rich environments are
likely to prossess rapid spins with an average $\epsilon_{\rm rad}\geq 0.2$
(the corresponding BH spin is $a_{\bullet}\simeq 0.96$), indicating a
prevalent condition of rapid angular momentum in BH growing environments in
the early universe.

This result suggests that BH growth at these high redshifts, especially in
LRDs, is likely dominated by prolonged accretion episodes with coherent
angular momentum directions (e.g., Volonteri et al., 2005), unlike short-lived
chaotic accretion with randomly oriented inflows that tend to spin BHs down
(e.g., King et al., 2008). Our conclusion on rapid spins of the early BH
population will be directly testable through future gravitational-wave
observations with the space-based detectors such as LISA, TianQin, and Taiji
(e.g., Amaro-Seoane et al., 2023; Torres-Orjuela et al., 2023).

##  3 Potential overmassive BH trends in Little Red Dots

In this section, we explore the possibility that BHs within LRDs are
overmassive relative to the mass correlation with their host mass, as implied
from the BHAD-to-SFRD ratio shown in Figure 2. In general, estimating the
stellar mass of dust-obscured sources poses a significant challenge in the
absence of rest-frame near-infrared data provided by JWST MIRI (e.g., Williams
et al., 2023; Pérez-González et al., 2024). Instead of examining the detailed
spectral energy distribution fitting analysis, we focus on putting an upper
bound for the stellar mass. This approach ensures that the observed abundance
of LRDs does not exceed the theoretical upper bound in the standard $\Lambda$
cold dark matter (CDM) model with a 100% conversion efficiency from gas to
stars (e.g., Boylan-Kolchin, 2023).

The stellar continuum for LRDs can be constrained by the dust-corrected
continuum flux at 5100 Å. Given that broad H$\alpha$ emission indicates AGN
dominance in the continuum (see Section 4.3), we adjust
$L_{\star,5100}=f_{L}L_{5100}$, where $f_{L}$ is significantly less than
unity. To estimate an upper bound of stellar mass, we employ the STARBURST99
population synthesis code (version 7.0.1; Leitherer et al., 1999), adopting a
Kroupa IMF (Kroupa 2001; $0.1-100~{}M_{\odot}$), Padova isochrone models,
constant star formation, and solar metallicity. This approach yields a galaxy
mass$-$luminosity relation:

$\frac{M_{\star}}{10^{9}~{}M_{\odot}}\simeq 1.3~{}f_{\rm
IMF}\left(\frac{L_{\star,5100}}{10^{43}~{}{\rm
erg~{}s}^{-1}}\right)\left(\frac{t_{\rm age}}{1~{}{\rm Gyr}}\right),$ (1)

which is applicable for stellar ages of $t_{\rm age}\sim 0.3-3~{}{\rm Gyr}$.
This estimate is sensitive to the low-mass end of the stellar IMF ($f_{\rm
IMF}=1$ for $m_{\rm\star,min}=0.1~{}M_{\odot}$); for instance, setting the
minimum mass up to $m_{\star,\rm min}=1.0~{}M_{\odot}$ decreases the factor to
$f_{\rm IMF}\sim 0.3$.

Figure 4:  Stellar mass density in galaxies hosting LRDs at various redshifts,
calculated using Eq. (1) and assuming $\mathscr{F}(\equiv f_{\rm
IMF}f_{L})=1.0$ (open symbols) at two redshift ranges of $4.5<z<6.5$ and
$6.5<z<8.5$. For comparison, the stellar mass function derived from the DM
halo mass function at $5\leq z\leq 8$ is shown with a 100% star formation
efficiency. An upper bound of the stellar mass constrains
$\mathscr{F}<0.3f_{\star}$ for $4.5<z<6.5$ and $\mathscr{F}<0.04f_{\star}$ for
$6.5<z<8.5$ (filled symbols).  Figure 5: $M_{\bullet}-M_{\star}$ distribution
for high-redshift AGNs, including LRDs, JWST-detected unobscured ANGs at
$z=4-8$ (purple, Maiolino et al. 2023; green, Harikane et al. 2023; cyan,
Stone et al. 2023; and blue, Ding et al. 2023), and quasars identified in
ground-based surveys (Izumi et al., 2021). For the LRDs at $4.5<z<6.5$ (red)
and $6.5<z<8.5$ (orange), we derive the upper bound of the stellar mass based
on the dust-corrected continuum flux measured by Greene et al. (2023) using
Eqs. (1) and (2). Additionally, a $z=8.5$ LRD with broad H$\beta$ emission,
for which the stellar mass is constrained by ALMA non-detections, is overlaid
(blue, Kokorev et al., 2023). Three different mass correlations are overlaid:
the BHAD/SFRD ratio from Figure 2 (solid), the local relationship (dashed,
Kormendy & Ho, 2013), and the JWST-detected AGNs (dotted, Pacucci et al.,
2023).

Figure 4 presents the stellar mass function (in units of $M_{\odot}~{}{\rm
Mpc}^{-3}$) in galaxies hosting LRDs at various redshifts, calculated by using
Eq. (1) and assuming $\mathscr{F}(\equiv f_{\rm IMF}f_{L})=1.0$ at two
redshift ranges of $4.5<z<6.5$ and $6.5<z<8.5$ (open circles), based on the
LRD luminosity function obtained by Kokorev et al. (2024). We compare the
results to the stellar mass function derived from the halo mass function at
$5\leq z\leq 8$, assuming $M_{\star}=f_{\star}f_{\rm b}M_{\rm h}$, where
$f_{\rm b}=0.16$ is the cosmic baryon fraction and $f_{\star}$ is the star
formation efficiency (see more details in Inayoshi et al., 2022a). Setting
$f_{\star}=1.0$ offers a theoretical upper limit on stellar mass in galaxies
($f_{\star}>1$; the forbidden region), highlighting a mismatch between the
stellar mass density contained in LRDs with $\mathscr{F}=1.0$ and the
$\Lambda$CDM upper limit. As a result, we deduce a stringent constraint
denoted with filled circles,

$\displaystyle\mathscr{F}<0.3~{}f_{\star}$ $\displaystyle\hskip 5.69054pt{\rm
at}~{}~{}4.5<z<6.5,$ $\displaystyle\mathscr{F}<0.04~{}f_{\star}$
$\displaystyle\hskip 5.69054pt{\rm at}~{}~{}6.5<z<8.5.$ (2)

Figure 5 shows the $M_{\bullet}-M_{\star}$ distribution for high-redshift
AGNs, including LRDs (square), JWST-detected unobscured AGNs (cross), and
quasars identified in ground-based surveys (circle). For the LRDs at
$4.5<z<6.5$ (red) and $6.5<z<8.5$ (orange), we derive the stellar mass of
those with broad H$\alpha$ emission from Greene et al. (2023), using Eq. (1).
This calculation incorporates the upper limit for the stellar continuum ratio
$\mathscr{F}$ to the observed continuum (see Eq. 2 and Figure 4), ensuring
consistency with the theoretical upper bound of stellar mass density in the
$\Lambda$CDM universe. The $M_{\bullet}-M_{\star}$ values for LRDs tend to be
overmassive compared to the local relationship (dashed line; Kormendy & Ho,
2013), and aligns well with other AGNs detected by JWST and follows the mass
correlation inferred from JWST AGN data, excluding quasars from ground-based
surveys (dotted line; Pacucci et al., 2023). Moreover, their distribution is
consistent with the locus of a JWST/NIRSpec confirmed $z=8.5$ LRD that
exhibits broad H$\beta$ emission, for which an upper limit on the stellar mass
has been constrained by non-detection in ALMA observations (blue; Kokorev et
al., 2023).

The mass ratios for these LRDs are also consistent with the BHAD/SFRD values
at $z>6$ shown in Figure 2. This suggests a model in which transient rapid
growth phases during the LRD stages elevate these BHs into an overmassive
state. This hypothesis is supported both theoretically (e.g., Inayoshi et al.,
2022b; Hu et al., 2022) and observationally (Fujimoto et al., 2022), providing
a comprehensive insight into the BH growth dynamics in the early universe.

##  4 Discussion

###  4.1 Missing X-ray radiation from LRDs

X-ray AGN surveys are generally effective in identifying obscured AGNs.
However, in the case of LRDs observed with JWST, no X-ray counterparts have
been reported (e.g., Kocevski et al., 2023; Furtak et al., 2023; Matthee et
al., 2023). The optical continuum extinction measurements suggest a gas column
density of $N_{\rm H}\sim 3.3\times 10^{22}(A_{\rm V}/3.0)~{}{\rm cm}^{-2}$
(Maiolino et al., 2001), indicating that the column density outside the broad-
line region is too low to obscure X-rays.

Considering the absence of X-ray counterparts for LRDs, we explore the
possibility that their X-ray emission is intrinsically weak, as compared to
typical X-ray selected AGNs. Figure 6 presents the critical X-ray luminosity
for each bolometric luminosity, so that $\Phi_{\rm X}(L_{\rm
X,crit})\geq\Phi_{\rm LRD}(L_{\rm bol})$, where we adopt the X-ray AGN
luminosity function $\Phi_{\rm X}$ from Ueda et al. (2014) and the LRD
bolometric luminosity function $\Phi_{\rm LRD}$ from Kokorev et al. (2024),
respectively. This condition requires that LRDs must have a bolometric
correction factor to X-rays that prevents them from being classified as X-ray
AGNs and contributing to their abundance (red horizontal lines with arrows).
The critical X-ray luminosity is limited below those derived from comparison
between the X-ray and optical-based AGN luminosity functions at lower
redshifts of $z\lesssim 2$ (Ueda et al., 2003).

Figure 6: The critical X-ray luminosities for LRDs at $z\sim 5$, determined by
the requirement for LRDs not to be classified as X-ray AGNs nor contributing
to the abundance of X-ray AGNs (red horizontal lines with arrows). For
comparison, several models for bolometric correction to X-rays are shown (Ueda
et al., 2003; Nanni et al., 2017; Duras et al., 2020), as well as the cases
with constant optical to X-ray spectral indices of $\alpha_{\rm OX}=-1.5$
(dashed) and $-1.8$ (solid). The shaded area denotes X-ray luminosities below
the detection threshold of current Chandra observations for JWST fields.

To quantify this intrinsic X-ray faintness in LRDs, we utilize the optical to
X-ray spectral index, defined as $\alpha_{\rm
OX}=\log(L_{\rm\nu,2keV}/L_{\nu,2500})/\log(\nu_{\rm 2keV}/\nu_{2500})$, where
$L_{\rm\nu,2keV}$ and $L_{\nu,2500}$ are the extinction corrected luminosity
density at $2~{}{\rm keV}$ and $2500~{}{\rm\AA}$. Our findings support a
constant value of $\alpha_{\rm OX}$ lower than $-1.8$, rather than the
luminosity-dependent $\alpha_{\rm OX}$ values observed in unobscured quasars
across $0\lesssim z\lesssim 6$ (e.g., Steffen et al., 2006; Duras et al.,
2020). The upper bound of $\alpha_{\rm OX}\simeq-1.8$ is seen in the most
luminous quasar populations with $L_{\rm bol}\gtrsim 10^{47}~{}{\rm
erg~{}s}^{-1}$ (Nanni et al., 2017), which does not apply to most LRDs studied
in this paper.

The extensive LRD samples by Kokorev et al. (2024), consisting of 260 dust-
reddened AGN candidates, are compiled from deep JWST/NIRCam fields totaling
$\sim 340~{}{\rm arcmin}^{2}$. This includes observations from the CEERS
field, which spans $\sim 51.9~{}{\rm arcmin}^{2}$ and falls within the
coverage area of the Chandra AEGIS-XD survey coverage (Nandra et al., 2015).
The survey detection limit reaches $1.5\times 10^{-16}~{}{\rm
erg~{}s}^{-2}~{}{\rm cm}^{-1}$ at 0.5–10 keV, which corresponds to $L_{\rm
X}\simeq 1.1\times 10^{43}~{}{\rm erg~{}s}^{-1}$ at the rest-frame 2–10 keV
for $z\sim 5-7$ sources assuming a photon index of $\Gamma=-1.7$ and a
Compton-thin limit ($N_{\mathrm{H}}<10^{24}$ cm-2). Given the average surface
density of these LRDs ($\simeq 0.77~{}{\rm arcmin}^{-2}$), $\sim 40$ LRDs in
the CEERS field show no X-ray detection above this threshold. This suggests
that non-detection of X-rays among LRDs can be explained by the intrinsic
faintness of X-rays as shown in Figure 6. Nevertheless, the most luminous LRDs
with $L_{\rm bol}\gtrsim 10^{47}~{}{\rm erg~{}s}^{-1}$ might still be observed
in X-rays unless classified as Compton-thick AGNs.

Alternatively, if the X-ray emission is not intrinsically faint, the majority
of the LRDs should have highly obscured Compton-thick gas in the centers,
possibly $N_{\mathrm{H}}\gg 10^{24}$ cm-2. In this case, the LRD population is
“dark” even in deep X-ray observations, and there is such a hidden BH
population missed in the cosmic X-ray background radiation.

###  4.2 Dusty young starburst galaxies mimicking LRD-like AGNs?

The classification of LRDs as AGNs has relied on the detection of broad
H$\alpha$ emission. However, such high-velocity gas can also originate from
stellar processes, such as stellar winds or supernova explosions. Wolf-Rayet
(WR) galaxies, characterized by young and massive stellar populations, can
exhibit broad H$\alpha$ emission as well as other high-ionization lines (e.g.,
Ho et al., 1995; Schaerer et al., 1999). Therefore, to conclusively confirm
the AGN nature of LRDs, it is essential to perform line diagnostics that
extend beyond simply identifying broad H$\alpha$ emission, e.g., detection of
broad He II $\lambda 4686$ emission, a signature frequently associated with WR
galaxies (e.g., NGC 4214 discussed in Sargent & Filippenko, 1991).

Spectroscopic observations provide valuable insights into the characteristics
of emission lines in LRDs. Initial studies by Kocevski et al. (2023) and
Greene et al. (2023) have found that the He II $\lambda 4686$ line, commonly
associated with WR stellar activity, is absent in these LRD sources.
Additionally, the spectral signatures typically linked to WR stars have not
been observed within the LRD samples. The lack of He II and other WR
indicative spectral lines suggests that the broad H$\alpha$ emissions detected
in LRDs are not of stellar origin (note that detection of He II $\lambda 4686$
does not necessarily exclude the AGN possibility because the emission is also
observed in AGNs). This finding supports the hypothesis that AGNs are
responsible for these emissions. More detailed spectroscopic analyses of
multiple emission lines would strength this conclusion (Greene et al., 2020;
Reines, 2022).

###  4.3 Bolometric luminosity estimates

Our study is motivated by intriguing discoveries on the high abundance of
LRDs, but an important consideration about the AGN luminosity estimate needs
to be noted. While the UV luminosity functions show similar shapes across
different studies, significant variations in the bolometric luminosity
function arise due to the different methods used for bolometric luminosity
estimates (see Figure 1). Since the observed rest-frame UV flux is heavily
attenuated, earlier studies have employed either the rest-frame continuum flux
at 5100 Å (Kokorev et al., 2024), or the direct H$\alpha$ emission luminosity,
when available, as a proxy for the AGN bolometric luminosity (Matthee et al.,
2023; Greene et al., 2023).

The approach that relies on continuum flux introduces uncertainties of
determining the AGN contribution to the total flux, which might result in
overestimated luminosity. Furthermore, the selection of LRDs based solely on
photometric criteria might include non-AGN sources such as Galactic brown
dwarfs (Langeroodi et al., 2023), thereby possibly overestimating the AGN
abundance. Greene et al. (2023) reported the identification success rate of
AGNs among LRDs in the UNCOVER field to be approximately 60%.

In contrast, spectroscopic data particularly with measurements of broad
H$\alpha$ emission facilitate confirmation of the AGN presence and a more
accurate determination of $L_{\rm bol}$. This method adopts an empirical
relationship derived from local AGN observations (Greene & Ho, 2005). For LRD
sources with detected H$\alpha$ emissions, the continuum fluxes at 5100 Å
calculated through the two methods yield ratios of $L_{5100,\rm
H\alpha}/L_{5100,\rm c}\simeq 1.2\pm 0.2$ (Greene et al., 2023). This result
supports the scenario that the continuum emission at 5100 Å is substantially
AGN-origin, not from dust-reddened stellar continuum of the host galaxy (i.e.,
$f_{L}\ll 1$).

One limitation in the work by Greene et al. (2023) is the use of low-
resolution PRISM spectra for analyzing H$\alpha$ emissions, which complicates
the decomposition of broad and narrow H$\alpha$ emissions in some cases. To
address this challenge, we can use an empirical relationship obtained through
higher-resolution spectroscopic observations by Matthee et al. (2023). This
relationship examines the flux ratio between narrow and broad H$\alpha$
emissions in LRDs; namely, the ratio $F_{\rm H\alpha,broad}/F_{\rm
H\alpha,tot}$ exceeds 0.6 with a positive rest-frame optical spectral index,
with the ratio approaching unity as the spectral index increases. Therefore,
the potential systematic errors in estimating broad H$\alpha$ line luminosity
due to incomplete spectral line decomposition could be alleviated by employing
this relationship.

Despite these complexities found in current analyses, the central conclusion
of our discussion remains valid on a qualitative level. Nevertheless, the
development of more quantitative arguments will benefit from further
observational explorations that refine the understanding of the AGN
characteristics of LRDs and provide a more precise estimate of their cosmic
abundance.

###  4.4 Multi-messenger Counterparts

In this study, we propose a scenario where the dust-rich environments within
LRDs lead to the emergence of rapidly spinning and overmassive BH. The high
spins of these BHs may yield a strong correlation between the presence of
relativistic jets (e.g., Blandford & Znajek, 1977), high-energy emissions and
particles (e.g., Dai & Fang, 2017; Murase et al., 2020), and transient bursts
such as stellar tidal disruption events (e.g., Inayoshi et al., 2023) with
early BHs being overmassive compared to the mass correlation observed in the
nearby universe.

Multi-wavelength surveys including radio, optical, and X-ray bands have
reported that the radio powers (or radio loudness) of obscured AGNs initially
identified by the VLA/FIRST survey as bright radio sources tend to increase
with redshifts at $0.5<z<3.5$, with jet powers exceeding $P_{\rm jet}\gtrsim
10^{46}~{}{\rm erg~{}s}^{-1}$ (Ichikawa et al., 2021, 2023). The inferred jet
production efficiency calculated from $\eta_{\rm jet}\sim\epsilon_{\rm
rad}P_{\rm jet}/L_{\rm bol}$ approaches unity, implying rapid spins of these
nuclear BHs (e.g., Tchekhovskoy et al., 2011). These types of radio AGNs at
intermediate redshifts may offer valuable insights into understanding the
characteristics of LRDs at higher redshifts, thus further supporting our
hypothesis of rapid BH spins.

###  4.5 Implications to BH growth mechanisms at $z\gtrsim 5$

The spin of a massive BH is influenced by a range of physical processes
including BH mergers and mass accretion. The coalescence of two non-spinning
BHs results in a remnant with a significant spin, approximately
$a_{\bullet}\simeq 0.69$ for an equal-mass merger, but the spin diminishes if
the merging BHs have non-zero, misaligned spins relative to the orbital
angular momentum (e.g., Berti & Volonteri, 2008). Gas accreting onto BHs
through a disk aligned with the angular momentum direction of the BH is
expected to enhance the spin during mass accumulation (Bardeen et al., 1972;
Thorne, 1974). However, chaotic accretion characterized by short-lived
episodes with random orientations tends to dampen the BH spin toward average
values of $a_{\bullet}\simeq 0.2$ (e.g., King et al., 2008).

Chaotic accretion and related feeding mechanisms with low angular momentum gas
are considered to promote the efficiency of BH mass growth in the early
universe, due to moderate centrifugal support (Eisenstein & Loeb, 1995) and a
lower radiative efficiency as a result of spin down (King et al., 2008).
However, we show that the majority of the early BH populations growing in
dust-rich environments tend to exhibit high spins with $a_{\bullet}\simeq
0.9$, corresponding to $\epsilon_{\rm rad}\gtrsim 0.2$. This suggests that BH
growth is likely dominated by prolonged accretion episodes with coherent
angular momentum, consistent with an idea pioneered by Volonteri et al.
(2005).

Our conclusion regarding the rapid spins of BHs can be directly testable
through future gravitational-wave observations with the space-based detectors
such as LISA (e.g., Amaro-Seoane et al., 2023). Moreover, if these BHs
frequently merge during galaxy coalescences leading to the LRD phase or
similar activities observed in ultra-luminous infrared galaxies, such events
could significantly contribute to a stochastic gravitational-wave background,
detectable by pulsar-timing array experiments (Inayoshi et al. 2018; see also
Agazie et al. 2023). Therefore, further exploration of the rapidly spinning,
overmassive BHs in LRDs is needed in multiple aspects.

We greatly thank Luis C. Ho for emphasizing the significance of the Soltan-
Paczyński argument and for dedicating time to an inspiring discussion during
the spring festival holiday. We also wish to thank Xiaoyang Chen, Haojie Hu,
Kazumi Kashiyama, Vasily Kokorev, and Masafusa Onoue for constructive
discussions. K. Inayoshi acknowledges support from the National Natural
Science Foundation of China (12073003, 12003003, 11721303, 11991052,
11950410493), and the China Manned Space Project (CMS-CSST-2021-A04 and CMS-
CSST-2021-A06). This work is also supported by Japan Society for the Promotion
of Science (JSPS) KAKENHI (20H01939; K. Ichikawa)

## References

  * Agazie et al. (2023) Agazie, G., Anumarlapudi, A., Archibald, A. M., et al. 2023, ApJ, 952, L37, doi: 10.3847/2041-8213/ace18b
  * Amaro-Seoane et al. (2023) Amaro-Seoane, P., Andrews, J., Arca Sedda, M., et al. 2023, Living Reviews in Relativity, 26, 2, doi: 10.1007/s41114-022-00041-y
  * Ananna et al. (2019) Ananna, T. T., Treister, E., Urry, C. M., et al. 2019, ApJ, 871, 240, doi: 10.3847/1538-4357/aafb77
  * Bardeen et al. (1972) Bardeen, J. M., Press, W. H., & Teukolsky, S. A. 1972, ApJ, 178, 347, doi: 10.1086/151796
  * Barro et al. (2023) Barro, G., Perez-Gonzalez, P. G., Kocevski, D. D., et al. 2023, arXiv e-prints, arXiv:2305.14418, doi: 10.48550/arXiv.2305.14418
  * Berti & Volonteri (2008) Berti, E., & Volonteri, M. 2008, ApJ, 684, 822, doi: 10.1086/590379
  * Blandford & Znajek (1977) Blandford, R. D., & Znajek, R. L. 1977, MNRAS, 179, 433, doi: 10.1093/mnras/179.3.433
  * Boylan-Kolchin (2023) Boylan-Kolchin, M. 2023, Nature Astronomy, doi: 10.1038/s41550-023-01937-7
  * Dai & Fang (2017) Dai, L., & Fang, K. 2017, MNRAS, 469, 1354, doi: 10.1093/mnras/stx863
  * Delvecchio et al. (2014) Delvecchio, I., Gruppioni, C., Pozzi, F., et al. 2014, MNRAS, 439, 2736, doi: 10.1093/mnras/stu130
  * Ding et al. (2023) Ding, X., Onoue, M., Silverman, J. D., et al. 2023, Nature, 621, 51, doi: 10.1038/s41586-023-06345-5
  * Duras et al. (2020) Duras, F., Bongiorno, A., Ricci, F., et al. 2020, A&A, 636, A73, doi: 10.1051/0004-6361/201936817
  * Eisenstein & Loeb (1995) Eisenstein, D. J., & Loeb, A. 1995, ApJ, 443, 11, doi: 10.1086/175498
  * Fujimoto et al. (2022) Fujimoto, S., Brammer, G. B., Watson, D., et al. 2022, Nature, 604, 261, doi: 10.1038/s41586-022-04454-1
  * Furtak et al. (2023) Furtak, L. J., Zitrin, A., Plat, A., et al. 2023, ApJ, 952, 142, doi: 10.3847/1538-4357/acdc9d
  * Greene & Ho (2005) Greene, J. E., & Ho, L. C. 2005, ApJ, 630, 122, doi: 10.1086/431897
  * Greene et al. (2020) Greene, J. E., Strader, J., & Ho, L. C. 2020, ARA&A, 58, 257, doi: 10.1146/annurev-astro-032620-021835
  * Greene et al. (2023) Greene, J. E., Labbe, I., Goulding, A. D., et al. 2023, arXiv e-prints, arXiv:2309.05714, doi: 10.48550/arXiv.2309.05714
  * Harikane et al. (2022) Harikane, Y., Ono, Y., Ouchi, M., et al. 2022, ApJS, 259, 20, doi: 10.3847/1538-4365/ac3dfc
  * Harikane et al. (2023) Harikane, Y., Zhang, Y., Nakajima, K., et al. 2023, arXiv e-prints, arXiv:2303.11946, doi: 10.48550/arXiv.2303.11946
  * Ho et al. (1995) Ho, L. C., Filippenko, A. V., & Sargent, W. L. 1995, ApJS, 98, 477, doi: 10.1086/192170
  * Hu et al. (2022) Hu, H., Inayoshi, K., Haiman, Z., et al. 2022, ApJ, 935, 140, doi: 10.3847/1538-4357/ac7daa
  * Ichikawa et al. (2021) Ichikawa, K., Yamashita, T., Toba, Y., et al. 2021, ApJ, 921, 51, doi: 10.3847/1538-4357/ac1b26
  * Ichikawa et al. (2023) Ichikawa, K., Yamashita, T., Merloni, A., et al. 2023, A&A, 672, A171, doi: 10.1051/0004-6361/202244271
  * Inayoshi et al. (2022a) Inayoshi, K., Harikane, Y., Inoue, A. K., Li, W., & Ho, L. C. 2022a, ApJ, 938, L10, doi: 10.3847/2041-8213/ac9310
  * Inayoshi et al. (2018) Inayoshi, K., Ichikawa, K., & Haiman, Z. 2018, ApJ, 863, L36, doi: 10.3847/2041-8213/aad8ad
  * Inayoshi et al. (2023) Inayoshi, K., Kashiyama, K., Li, W., et al. 2023, arXiv e-prints, arXiv:2312.11602, doi: 10.48550/arXiv.2312.11602
  * Inayoshi et al. (2022b) Inayoshi, K., Nakatani, R., Toyouchi, D., et al. 2022b, ApJ, 927, 237, doi: 10.3847/1538-4357/ac4751
  * Izumi et al. (2021) Izumi, T., Matsuoka, Y., Fujimoto, S., et al. 2021, ApJ, 914, 36, doi: 10.3847/1538-4357/abf6dc
  * Kerr (1963) Kerr, R. P. 1963, Phys. Rev. Lett., 11, 237, doi: 10.1103/PhysRevLett.11.237
  * King et al. (2008) King, A. R., Pringle, J. E., & Hofmann, J. A. 2008, MNRAS, 385, 1621, doi: 10.1111/j.1365-2966.2008.12943.x
  * Kocevski et al. (2023) Kocevski, D. D., Onoue, M., Inayoshi, K., et al. 2023, arXiv e-prints, arXiv:2302.00012.  https://arxiv.org/abs/2302.00012
  * Kokorev et al. (2023) Kokorev, V., Fujimoto, S., Labbe, I., et al. 2023, ApJ, 957, L7, doi: 10.3847/2041-8213/ad037a
  * Kokorev et al. (2024) Kokorev, V., Caputi, K. I., Greene, J. E., et al. 2024, arXiv e-prints, arXiv:2401.09981, doi: 10.48550/arXiv.2401.09981
  * Kormendy & Ho (2013) Kormendy, J., & Ho, L. C. 2013, ARA&A, 51, 511, doi: 10.1146/annurev-astro-082708-101811
  * Kroupa (2001) Kroupa, P. 2001, MNRAS, 322, 231, doi: 10.1046/j.1365-8711.2001.04022.x
  * Labbe et al. (2023) Labbe, I., Greene, J. E., Bezanson, R., et al. 2023, arXiv e-prints, arXiv:2306.07320, doi: 10.48550/arXiv.2306.07320
  * Langeroodi et al. (2023) Langeroodi, D., Hjorth, J., & Zhang, Z. 2023, arXiv e-prints, arXiv:2308.10900, doi: 10.48550/arXiv.2308.10900
  * Leitherer et al. (1999) Leitherer, C., Schaerer, D., Goldader, J. D., et al. 1999, ApJS, 123, 3, doi: 10.1086/313233
  * Li et al. (2023) Li, W., Inayoshi, K., Onoue, M., et al. 2023, arXiv e-prints, arXiv:2306.06172, doi: 10.48550/arXiv.2306.06172
  * Lynden-Bell (1969) Lynden-Bell, D. 1969, Nature, 223, 690, doi: 10.1038/223690a0
  * Maiolino et al. (2001) Maiolino, R., Marconi, A., Salvati, M., et al. 2001, A&A, 365, 28, doi: 10.1051/0004-6361:20000177
  * Maiolino et al. (2023) Maiolino, R., Scholtz, J., Curtis-Lake, E., et al. 2023, arXiv e-prints, arXiv:2308.01230, doi: 10.48550/arXiv.2308.01230
  * Matthee et al. (2023) Matthee, J., Naidu, R. P., Brammer, G., et al. 2023, arXiv e-prints, arXiv:2306.05448, doi: 10.48550/arXiv.2306.05448
  * Murase et al. (2020) Murase, K., Kimura, S. S., Zhang, B. T., Oikonomou, F., & Petropoulou, M. 2020, ApJ, 902, 108, doi: 10.3847/1538-4357/abb3c0
  * Nandra et al. (2015) Nandra, K., Laird, E. S., Aird, J. A., et al. 2015, ApJS, 220, 10, doi: 10.1088/0067-0049/220/1/10
  * Nanni et al. (2017) Nanni, R., Vignali, C., Gilli, R., Moretti, A., & Brandt, W. N. 2017, A&A, 603, A128, doi: 10.1051/0004-6361/201730484
  * Niida et al. (2020) Niida, M., Nagao, T., Ikeda, H., et al. 2020, ApJ, 904, 89, doi: 10.3847/1538-4357/abbe11
  * Novikov & Thorne (1973) Novikov, I. D., & Thorne, K. S. 1973, in Black Holes (Les Astres Occlus), 343–450
  * Pacucci et al. (2023) Pacucci, F., Nguyen, B., Carniani, S., Maiolino, R., & Fan, X. 2023, ApJ, 957, L3, doi: 10.3847/2041-8213/ad0158
  * Peca et al. (2023) Peca, A., Cappelluti, N., Urry, C. M., et al. 2023, ApJ, 943, 162, doi: 10.3847/1538-4357/acac28
  * Pérez-González et al. (2024) Pérez-González, P. G., Barro, G., Rieke, G. H., et al. 2024, arXiv e-prints, arXiv:2401.08782, doi: 10.48550/arXiv.2401.08782
  * Pouliasis et al. (2024) Pouliasis, E., Ruiz, A., Georgantopoulos, I., et al. 2024, arXiv e-prints, arXiv:2401.13515, doi: 10.48550/arXiv.2401.13515
  * Reines (2022) Reines, A. E. 2022, Nature Astronomy, 6, 26, doi: 10.1038/s41550-021-01556-0
  * Sargent & Filippenko (1991) Sargent, W. L. W., & Filippenko, A. V. 1991, AJ, 102, 107, doi: 10.1086/115859
  * Schaerer et al. (1999) Schaerer, D., Contini, T., & Pindao, M. 1999, A&AS, 136, 35, doi: 10.1051/aas:1999197
  * Shankar et al. (2009) Shankar, F., Weinberg, D. H., & Miralda-Escudé, J. 2009, ApJ, 690, 20, doi: 10.1088/0004-637X/690/1/20
  * Soltan (1982) Soltan, A. 1982, MNRAS, 200, 115
  * Steffen et al. (2006) Steffen, A. T., Strateva, I., Brandt, W. N., et al. 2006, AJ, 131, 2826, doi: 10.1086/503627
  * Stone et al. (2023) Stone, M. A., Lyu, J., Rieke, G. H., Alberts, S., & Hainline, K. N. 2023, arXiv e-prints, arXiv:2310.18395, doi: 10.48550/arXiv.2310.18395
  * Tchekhovskoy et al. (2011) Tchekhovskoy, A., Narayan, R., & McKinney, J. C. 2011, MNRAS, 418, L79, doi: 10.1111/j.1745-3933.2011.01147.x
  * Thorne (1974) Thorne, K. S. 1974, ApJ, 191, 507, doi: 10.1086/152991
  * Torres-Orjuela et al. (2023) Torres-Orjuela, A., Huang, S.-J., Liang, Z.-C., et al. 2023, arXiv e-prints, arXiv:2307.16628, doi: 10.48550/arXiv.2307.16628
  * Ueda et al. (2014) Ueda, Y., Akiyama, M., Hasinger, G., Miyaji, T., & Watson, M. G. 2014, ApJ, 786, 104, doi: 10.1088/0004-637X/786/2/104
  * Ueda et al. (2003) Ueda, Y., Akiyama, M., Ohta, K., & Miyaji, T. 2003, ApJ, 598, 886, doi: 10.1086/378940
  * Vito et al. (2018) Vito, F., Brandt, W. N., Yang, G., et al. 2018, MNRAS, 473, 2378, doi: 10.1093/mnras/stx2486
  * Volonteri et al. (2005) Volonteri, M., Madau, P., Quataert, E., & Rees, M. J. 2005, ApJ, 620, 69, doi: 10.1086/426858
  * Williams et al. (2023) Williams, C. C., Alberts, S., Ji, Z., et al. 2023, arXiv e-prints, arXiv:2311.07483, doi: 10.48550/arXiv.2311.07483
  * Yu & Tremaine (2002) Yu, Q., & Tremaine, S. 2002, MNRAS, 335, 965, doi: 10.1046/j.1365-8711.2002.05532.x


Yusheng Tian, Wei Liu, Tan Lee

# Diffusion-Based Mel-Spectrogram Enhancement for Personalized Speech
Synthesis with Found Data

###### Abstract

Creating synthetic voices with found data is challenging, as real-world
recordings often contain various types of audio degradation. One way to
address this problem is to pre-enhance the speech with an enhancement model
and then use the enhanced data for text-to-speech (TTS) model training. This
paper investigates the use of conditional diffusion models for generalized
speech enhancement, which aims at addressing multiple types of audio
degradation simultaneously. The enhancement is performed on the log Mel-
spectrogram domain to align with the TTS training objective. Text information
is introduced as an additional condition to improve the model robustness.
Experiments on real-world recordings demonstrate that the synthetic voice
built on data enhanced by the proposed model produces higher-quality synthetic
speech, compared to those trained on data enhanced by strong baselines. Code
and pre-trained parameters of the proposed enhancement model are available at
https://github.com/dmse4tts/DMSE4TTS

Index Terms: personalized speech synthesis, found data, generalized speech
enhancement, text-informed speech enhancement, conditional diffusion models

##  1 Introduction

Text-to-speech (TTS) models are typically trained using carefully recorded
databases. Collecting such recordings is costly and sometimes impractical. In
real-world applications, sometimes only low-quality recordings from the target
speaker(s) are available. This has motivated the study of speech synthesis
with readily available data, i.e., speech data that are not purposely recorded
for the development of TTS systems[1, 2, 3, 4, 5, 6, 7, 8].

Developing TTS systems in this context is challenging, given the highly
varying degradation of audio quality affecting the found data. Synthetic
voices built directly with low-quality recordings would inevitably produce
distorted speech. Previous studies have attempted to tackle this problem
within the TTS framework, by augmenting the acoustic model with an additional
noise embedding [7, 8]. This enabled TTS model training with noisy speech, but
did not consider the impact of other forms of audio degradation. Background
noise is not the only disturbance present in real-world recordings. With this
in mind, it might be more practical to pre-enhance found data with a separate
module as in [1, 2, 3, 4], especially given the recent progress in generalized
speech enhancement [9, 10, 11, 12, 13], which aims at addressing multiple
types of audio degradation simultaneously.

In the present study, we propose to apply enhancement directly to log Mel-
spectrograms to align with the TTS training objective, and use conditional
diffusion models for generalized speech enhancement. The choice of a diffusion
model is motivated by the work of Palette [14], which used a single generalist
diffusion model to deal with a range of image-to-image translation tasks. Mel-
spectrograms are time-frequency representations that can be treated as images.
Thus we expect that the diffusion model would be effective in Mel-spectrogram
enhancement. To improve the model robustness against unseen forms of audio
degradation, text content of speech is used as an additional condition, which
is typically available in the context of TTS training.

We applied the proposed enhancement model to a real case of speech generation:
developing a personalized synthetic voice for a male Cantonese speaker with
37-minute found recordings. This gentleman lost the ability to speak after
receiving laryngectomy a few years ago. The recordings he provided, though
containing multiple types of degradation, are the only available and precious
record of his voice. TTS models are trained with speech enhanced by different
systems. Subjective evaluation by human listeners show that the synthetic
voice built on speech enhanced by the proposed model is rated higher for both
cleanliness and overall impression, compared to those trained on data enhanced
by strong speech enhancement baselines.

##  2 Related Work

###  2.1 Speech synthesis with found data

A number of previous studies have approached the problem of speech synthesis
with found data. [6, 5] designed algorithms to automatically select clean
recordings from crowd-sourced data. Others consider the situation when there
are simply no high-quality samples in the found data. [7, 8] augmented the TTS
model with an additional noise embedding, which permits the separation of
environmental noise from clean speech during TTS training. Another line of
research [1, 2, 3, 4] approached the problem in two steps: pre-enhance the
low-quality speech audio and then use the enhanced data for TTS model
training.

###  2.2 Generalized speech enhancement

Several prior works [9, 10, 11, 12, 13] have approached the problem of
generalized speech enhancement, i.e., addressing multiple types of audio
degradation simultaneously. Their success relies on well designed simulation
of audio degradation, and advanced neural network architectures. Most of these
models operate on the waveform or magnitude spectrogram. The enhanced speech
might be suboptimal for training TTS acoustic models, which are typically
designed to predict compact acoustic representations such as Mel-spectrograms.

###  2.3 Diffusion-based speech enhancement

The use of diffusion models for speech enhancement has been investigated in
[13, 15, 16, 17]. Most of the models are derived from diffusion-based neural
vocoder[18], by replacing the clean mel-spectrogram input with a degraded one.
This design may not have taken the full advantage of diffusion models for the
task of speech enhancement, as the enhancement work is undertaken largely by
the conditioning network. Our work is most similar to that of [19], in which
diffusion models are used for speech enhancement in the complex short-time
Fourier transform (STFT) domain. However, their model targets only at
denoising, and is not tailored for the development of TTS systems.

##  3 Diffusion-based Mel-spectrogram Enhancement

Suppose we are given a large collection of Mel-spectrogram pairs, denoted as
$\mathcal{D}=\\{\bm{x}^{(i)},\bm{y}^{(i)}\\}_{i=1}^{N}$, where $\bm{x}^{(i)}$
represents the Mel-spectrogram of a high-quality speech sample, and
$\bm{y}^{(i)}$ represents the Mel-spectrogram of a respective degraded sample.
$\bm{y}^{(i)}$ can be created by applying artificial audio degradation to
$\bm{x}^{(i)}$. We are interested in learning the conditional distribution
$P(\bm{x}|\bm{y})$ on $\mathcal{D}$. If $\mathcal{D}$ is constructed to be
representative, Mel-spectrogram enhancement can be achieved by sampling from
the learned conditional distribution. Diffusion models are adopted here to
learn a parametric approximation of $P(\bm{x}|\bm{y})$.

###  3.1 Conditional diffusion process

We consider the Variance Preserving (VP) diffusion model [20, 21]. Suppose
$\bm{x}_{0}\sim P(\bm{x}|\bm{y})$ is one enhanced realization for the degraded
input $\bm{y}$. VP diffusion defines the forward process as

$\displaystyle\mathrm{d}\bm{x}_{t}=-\frac{1}{2}\beta_{t}\bm{x}_{t}\mathrm{d}t+\sqrt{\beta_{t}}\mathrm{d}\bm{w}_{t}\
,$ (1)

where $t\sim\mathcal{U}(0,1)$, $\beta_{t}=\beta_{0}+\beta_{1}t$ is a
predefined linear noise scheduler, and $\bm{w}_{t}$ is a standard Brown
motion.

One important result derived from (1) is the conditional distribution of
$\bm{x}_{t}$ given $\bm{x}_{0}$:

$\displaystyle
P(\bm{x}_{t}|\bm{x}_{0})=\mathcal{N}(\bm{\rho}(\bm{x}_{0},t),\sigma^{2}_{t}\bm{I})\
,$ (2)

where
$\bm{\rho}(\bm{x}_{0},t)=e^{-\frac{1}{2}\int_{0}^{t}\beta_{s}\mathrm{d}s}\bm{x}_{0}$,
and $\sigma_{t}^{2}=1-e^{-\int_{0}^{t}\beta_{s}\mathrm{d}s}$. The nice
property given by (2) suggests that if $\bm{x}_{0}$ is known, we can sample
$\bm{x}_{t}$ using the reparameterization trick:

$\displaystyle\bm{x}_{t}=\bm{\rho}(\bm{x}_{0},t)+\sigma_{t}\bm{\epsilon}_{t},\bm{\epsilon}_{t}\sim\mathcal{N}(\bm{0},\bm{I})\
.$ (3)

Furthermore, as $t\xrightarrow{}1$, with appropriate noise scheduler
$\beta_{t}$ we have $\bm{\rho}(\bm{x}_{0},t)\xrightarrow{}\bm{0}$ and
$\sigma_{t}\xrightarrow{}1$, meaning that the forward process gradually
transforms the data distribution from $P(\bm{x}|\bm{y})$ into a standard
Gaussian distribution $\mathcal{N}(\bm{0},\bm{I})$.

Diffusion models generate samples by reversing the above forward process,
starting with a Gaussian noise:

$\displaystyle\mathrm{d}\bm{x}_{t}=-\frac{1}{2}\beta_{t}\left[\bm{x}_{t}+\nabla_{\bm{x}_{t}}\log
P(\bm{x}_{t}|\bm{y})\right]\mathrm{d}t\ .$ (4)

Note that the reverse process is conditioned on $\bm{y}$ to enable conditional
generation. The core part of a diffusion model is to train a neural network
$S_{\bm{\theta}}$ to estimate the value of $\nabla_{\bm{x}_{t}}\log
P(\bm{x}_{t}|\bm{y})$ (a.k.a the score). Once the score is known for all time
steps, we can draw samples from $P(\bm{x}|\bm{y})$ by simulating the reverse
process from $t=1$ to $0$, typically with an ODE solver[22].

###  3.2 Robust text condition

As mentioned earlier, the Mel-spectrogram enhancement model relies on a
synthetic dataset of paired samples to learn the conditional distribution
$P(\bm{x}|\bm{y})$. Therefore a domain gap between training samples and real-
world degraded recordings is inevitable. Consequently, the trained model may
overfit to in-domain data and fail to generalize well to unseen audio
degradation.

In order to improve the model robustness, we introduce text content of speech
samples as an additional condition. Text transcription is usually available in
the context of TTS development, and has shown to improve the robustness of SE
models[23, 24]. Inspired by GradTTS [25], a diffusion-based TTS model, we use
an average Mel-spectrogram $\bm{\mu}$ to represent text. $\bm{\mu}$ is of the
same shape as $\bm{y}$ and is obtained in three steps. First, text
transcription for each training sample is converted to time-aligned phone
sequence by forced alignment. Second, a phoneme-to-Mel-spectrum dictionary is
created on training data by averaging speech frames that correspond to the
same phoneme. Then given any time-aligned phone sequence, the respective
average Mel-spectrogram $\bm{\mu}$ is obtained by looking up the dictionary.

When text is provided, the reverse process is rewritten as:

$\displaystyle\mathrm{d}\bm{x}_{t}=-\frac{1}{2}\beta_{t}\left[\bm{x}_{t}+\nabla\log
P(\bm{x}_{t}|\bm{y},\bm{\mu})\right]\mathrm{d}t\ .$ (5)

###  3.3 Training and inference

Following [20], we make the score estimator $S_{\bm{\theta}}$ aware of the
time step and train it with a weighted $L2$ loss:

$\displaystyle\mathcal{L}(\bm{\theta})=\mathbb{E}_{t}\sigma_{t}^{2}{\mathbb{E}}_{(\bm{x}_{0},\bm{y})}{\mathbb{E}}_{\bm{\epsilon}_{t}}\left\|S_{\bm{\theta}}(\bm{x}_{t},t,\bm{y},\bm{\mu})+\sigma_{t}^{-1}\bm{\epsilon}_{t}\right\|_{2}^{2}\
,$ (6)

where we have made use of the following results:

$\displaystyle
P(\bm{x}_{t}|\bm{x}_{0},\bm{y},\bm{\mu})=P(\bm{x}_{t}|\bm{x}_{0})=\mathcal{N}(\bm{\rho}(\bm{x}_{0},t),\sigma^{2}_{t}\bm{I})\
,$ (7) $\displaystyle\nabla_{\bm{x}_{t}}\log
P(\bm{x}_{t}|\bm{x}_{0},\bm{y},\bm{\mu})=-\sigma_{t}^{-1}\bm{\epsilon}_{t}\ .$
(8)

Once the score estimator is trained, we can use the predicted score to
generate samples by running equation (5) backward in time from $t=1$ to $0$.
The training and inference procedures of the proposed Mel-spectrogram
enhancement model are illustrated as in Figure 1.

Figure 1: The training (solid lines) and inference (dashed lines) procedure of
the proposed enhancement model.

##  4 Experimental Setup

###  4.1 Baselines for comparison

The proposed enhancement model is named as DMSEtext, abbreviating Diffusion-
based Mel-Spectrogram Enhancement with text conditioning. The model without
text conditioning is named as DMSEbase. We compare them with two baselines:
Demucs[26] and VoiceFixer[12]. Demucs is a denoising model. We use it as a
baseline to investigate the efficacy of a single-task denoising model in the
intended application. VoiceFixer is a regression model trained to address a
range of audio degradation, including additive noise, reverberation, clipping
and low-bandwidth. The motivation of choosing VoiceFixer as a baseline is to
examine whether diffusion-based speech enhancement has advantage over a
regression-based approach.

###  4.2 Network architecture

The GradTTS model architecture[25] is adopted for the diffusion-based Mel-
spectrogram enhancement model. It is augmented to be conditioned on both text
($\bm{\mu}$) and audio $(\bm{y}$). Conditions are provided to the score
estimator by concatenation in the channel dimension. We scale the model depth
to 5-layer, with the output channel dimensions being 32, 64, 128, 256, 256
respectively. The TTS model is similar to DurIAN[27], except that style
control and postnet are not included. We use the HiFi-GAN[28] neural vocoder
to convert log Mel-spectrograms into waveforms. The number of Mel bands is
increased from 80 to 128. We empirically found that this is beneficial for
modelling voices of low vocal range, as is the case of our target speaker.

###  4.3 Data

The found data from our target speaker contain 513 utterances of spontaneous
yet very fluent speech, giving a total duration of approximately 37 minutes,
all manually transcribed. Speech was recorded in six different sessions with
varied room acoustics and sampled at 22.05 kHz. Audio degradation types found
in the recordings include background noise, room reverberation, band limiting
and magnitude clipping. The speech is in Cantonese. This speech corpus is used
for performance evaluation of speech enhancement models. The enhanced speech
is for training the TTS model.

We use CUSENT[29], a multi-speaker Cantonese speech database and artificially
created degraded audio to train DMSEbase and DMSEtext. CUSENT contains about
20 hours read speech from 80 speakers, sampled at 16kHz. It was designed for
automatic speech recognition (ASR) and the recordings contain low-level noise,
making it less than ideal for training speech enhancement models . We
therefore run Demucs on CUSENT to obtain high-quality clean speech. The
resulted dataset, denoted by DenoiseCUSENT, represent the clean reference for
speech enhancement training. Data from speaker cn01m and cn12f are held out
for validation.

To synthesize distorted speech, we consider the four most frequent types of
audio degradation present in found data: noise, reverberation, band limiting
and magnitude clipping. We use the DNS noise dataset[30] and the RIRs[31]
dataset to simulate background noise and reverberation respectively, adopting
their default train/test split. Band limiting and magnitude clipping are
simulated with the Scipy signal processing
toolkit111https://docs.scipy.org/doc/scipy/reference/signal.html. The four
types of degradation are applied following a specific order: reverberation,
noise, clipping and band limiting. Parameters such as SNR or frequency are
randomly set to be within a reasonable range. Details of simulated degradation
are released in the project repository.

For fair comparison with Demucs and VoiceFixer, we also train a diffusion-
based enhancement model on the high-quality English dataset VCTK[32], to rule
out the impact of language mismatch. Data from speaker p232 and p257 are held
out from training, as they appear in a standard denoising testset. Text
conditioning is not included here as the language of found data is Cantonese.
This model is referred to as DMSEbase(VCTK).

Throughout the experiments Mel-spectrograms are computed on audio signals
resampled to 22.05 kHz with a window length of 1024 and a hop size of 256. The
number of mel filterbanks is set to 128 as mentioned earlier.

###  4.4 Implementation details

DMSEbase and DMSEtext are trained on DenoiseCUSENT for 900 epochs, while
DMSEbase(VCTK) is trained on VCTK for 750 epochs, all at batch size of 32. The
number of reverse steps for all three models is 25 with the DPMsolver [22].
Mel-spectrograms are mean-normalized and scaled to be within the range of
$[-1.0,1.0]$. We empirically found that this normalization helps improving the
convergence speed of reverse diffusion.

The TTS model is pre-trained on CUSENT for 600 epochs with a batch size of 32,
then fine-tuned on the target speaker's data (original or enhanced) for 6000
steps with a batch size of 16. The neural vocoder is fine-tuned from a pre-
trained HiFi-GAN222https://github.com/jik876/hifi-gan on the VCTK and CUSENT
dataset for 220,000 steps with a batch size of 32. We insert a pre-processing
block to map the 128-dim log Mel-spectrogram input to 80-dim, to take
advantage of the pre-trained HiFi-GAN parameters.

We use the Adam optimizer[33] in all experiments with the default value
$[0.9,0.999]$ for betas. Learning rate is fixed at 1e-4, 1e-3 and 1e-6 for
training the MDSEbase/base(VCTK)/text, the TTS model, and the HiFi-GAN vocoder
respectively.

Time-aligned phone sequences for training the TTS model and DMSEtext are
obtained using the Montreal Forced Aligner[34]. DMSEtext also need time-
aligned phone sequence during test time. For the found data we apply denoising
with Demucs first before running forced alignment, in order to improve the
alignment accuracy. We should emphasize that the denoised found data are used
only during forced-alignment. All enhancement models take the original found
data as input for evaluation.

We additionally train a CTC-based ASR model for objective evaluation of speech
enhancement, following the recipe from
SpeechBrain333https://github.com/speechbrain/speechbrain/tree/develop/recipes/TIMIT/ASR/CTC.
The model is trained on KingASR086, a commercial Cantonese speech recognition
corpus purchased from
SpeechOcean444https://en.speechocean.com/datacenter/recognition.html, which
contains 80-hour reasonably clean read speech from 136 speakers, sampled at
44.1 kHz. No data augmentation is applied.

##  5 Result

(a) The distribution of MOS results for cleanliness

(b) The distribution of MOS results for overall impression

Figure 2: Score distribution of speech enhanced by different systems. Mean and
95% confidence interval are reported at the bottom. Figure 3: Comparing the
log Mel-spectrograms of speech audio generated by different speech enhancement
models.

###  5.1 Enhancing found speech data

The 513 utterances in the original found data, denoted as Recordings, are
enhanced by the three diffusion-based models as well as the two baselines,
respectively.

Objective evaluation was carried out with the ASR model. Results are
summarized in Table 1. It is noted that VoiceFixer and DMSEbase/base(VCTK)
tend to introduce pronunciation distortions (reflected in the number of
substitutions) and erase speech segments (reflected in the number of
deletions) compared with Demucs. On the other hand, text conditioning helps
DMSEtext to preserve content information and at the same time improve the
audio quality over original recordings.

Table 1: Phone Error Rate (PER%) results on found speech recordings enhanced by different systems with a clean-speech trained ASR model. N, I, D, S stands for the number of phones in the reference text, and the number of insertions, deletions, and substitutions in decoded results, respectively. Source | PER% | Details (N / I, D, S)
---|---|---
Recordings | 22.7 | 18928 / 282, 495, 3525
Demucs | 20.3 | 18928 / 321, 300, 3224
VoiceFixer | 29.7 | 18928 / 352, 921, 4341
DMSEbase | 24.7 | 18928 / 317, 702, 3652
DMSEbase(VCTK) | 24.3 | 18928 / 334, 593, 3664
DMSEtext | 17.6 | 18928 / 232, 258, 2836

Subjective evaluation was conducted through a web-based listening test. We
selected the same 87 sentences with lengths between 4.0 and 5.0 seconds from
the enhanced data produced by each system, as well as the original recordings.
This gives six stimuli for each sentence. The test is evaluated in Mean
Opinion Score (MOS) format. Listeners were asked to rate the cleanliness (no
noise or reverberation) as well as the overall impression of each stimulus on
a scale from 1.0 to 5.0. A clean audio sample from DenoiseCUSENT and a
synthetic distorted sample were provided, serving as the high and low anchors.
We evenly split the 87 sentences into three groups and recruited 18 listeners
per group. Each listener was presented with 29 sentences, and each sentence
was produced by one of the six systems, randomly selected. All listeners are
native Cantonese speakers.

The score distributions depicted in Figure 2 indicate that VoiceFixer and
DMSEbase/base(VCTK) generate speech that is clean but somewhat distorted,
while Demucs gives the opposite pattern. On the other hand, DMSEtext received
the highest score for both cleanliness and overall impression, which conforms
with the objective evaluation results. Notably, all diffusion-based models
outperformed the two baselines in terms of cleanliness. We speculate this to
be related to the generative modeling approach, where the generation process
is influenced by a clean speech prior.

By comparing DMSEbase against DMSEbase(VCTK), we can conclude that there is no
significant gain from using a Cantonese dataset, except for the availability
of text information. Figure 3 gives a specific example of the Mel-spectrogram
comparison between different systems. It can be seen that audio enhanced by
Demucs preserves more speech content as well as disturbances than audio
enhanced by other systems.

###  5.2 Speech synthesis with enhanced data

The purpose of developing the TTS model is to assist the target speaker to
communicate with others. To ensure that the model can produce high-quality
speech that meets the speaker’s personal needs, we asked him to provide text
content that he would like to or most likely say in his daily life for TTS
model evaluation. A total of 30 sentences were randomly selected from a script
written by the target speaker, and synthesized with TTS models trained on
speech enhanced by different systems. We recruited 24 native Cantonese
speakers to participate in the listening test. The test format is similar to
that described in the previous section. Each listener heard all 30 sentences
once, and each sentence generated by one of the six TTS models was evaluated
by exact four listeners.

Table 2: MOS results for cleanliness and overall impression of synthetic voices built on data enhanced by different systems. Source | Cleanliness | Overall impression
---|---|---
Recordings |  2.74 $\pm$ 0.08 |  3.12 $\pm$ 0.06
Demucs |  3.22 $\pm$ 0.08 |  3.38 $\pm$ 0.05
VoiceFixer |  3.97 $\pm$ 0.09 |  3.95 $\pm$ 0.09
DMSEbase |  4.02 $\pm$ 0.09 |  3.98 $\pm$ 0.08
DMSEbase(VCTK) |  4.05 $\pm$ 0.09 |  4.02 $\pm$ 0.09
DMSEtext |  4.32 $\pm$ 0.08 |  4.17 $\pm$ 0.06

The MOS results in Table 2 show an opposite trend to that observed in Figure
2, with the synthetic voice built on speech enhanced by Demucs receiving much
lower scores for overall impression compared with other systems. Two factors
might explain the difference. First, the remaining disturbances in speech
enhanced by Demucs were turned into unpleasant averaged artifacts after TTS
training. Second, the text encoder in the TTS system is shared across
speakers, which might have mitigated the negative impact of content distortion
for VoiceFixer and DMSEbase/base(VCTK). Nevertheless, the synthetic voice
built on speech enhanced by DMSEtext received the highest score for both
cleanliness and overall impression, which is expected given that content
information was well preserved during the enhancement process with the
additional text condition. Readers are encouraged to visit
https://dmse4tts.github.io/ to listen to audio samples.

##  6 Conclusion

We introduced a diffusion-based Mel-spectrogram enhancement model, which is
intended for pre-enhancing found data for TTS model development. It is trained
to tackle multiple types of audio degradation simultaneously, and is
conditioned on text transcriptions to improve its robustness. We empirically
showed that a generalist enhancement model is preferred over a single-task
denoising model for pre-enhancing real-world recordings for TTS model
development. Moreover, the additional text condition helps to preserve the
speech content during the enhancement process. Subjective evaluations by human
listeners demonstrate that the resulted synthetic voice produces higher-
quality synthetic speech compared to those trained on data enhanced by other
generalist speech enhancement baselines.

##  7 Acknowledgements

We thank all the participants in the listening test for their contributions.
Our deepest gratitude belongs to the gentleman, who generously agreed to
reveal his synthetic voice on the demo page.

## References

  * [1] C. Valentini-Botinhao and J. Yamagishi, ``Speech enhancement of noisy and reverberant speech for text-to-speech,'' _IEEE/ACM Transactions on Audio, Speech, and Language Processing_ , vol. 26, no. 8, pp. 1420–1433, 2018\.
  * [2] C. Valentini-Botinhao, X. Wang, S. Takaki, and J. Yamagishi, ``Speech enhancement for a noise-robust text-to-speech synthesis system using deep recurrent neural networks.'' in _Proc. Interspeech_ , 2016, pp. 352–356.
  * [3] N. Adiga, Y. Pantazis, V. Tsiaras, and Y. Stylianou, ``Speech enhancement for noise-robust speech synthesis using wasserstein gan.'' in _Proc. Interspeech_ , 2019, pp. 1821–1825.
  * [4] S. S. Sarfjoo, X. Wang, G. E. Henter, J. Lorenzo-Trueba, S. Takaki, and J. Yamagishi, ``Transformation of low-quality device-recorded speech to high-quality speech using improved SEGAN model,'' _CoRR_ , 2019.
  * [5] P. Baljekar and A. W. Black, ``Utterance selection techniques for tts systems using found speech.'' in _SSW_ , 2016, pp. 184–189.
  * [6] S. Ogun, V. Colotte, and E. Vincent, ``Can we use common voice to train a multi-speaker TTS system?'' in _SLT_ , 2022, pp. 900–905.
  * [7] W.-N. Hsu, Y. Zhang, R. J. Weiss, Y.-A. Chung, Y. Wang, Y. Wu, and J. Glass, ``Disentangling correlated speaker and noise for speech synthesis via data augmentation and adversarial factorization,'' in _Proc. ICASSP_ , 2019, pp. 5901–5905.
  * [8] C. Zhang, Y. Ren, X. Tan, J. Liu, K. Zhang, T. Qin, S. Zhao, and T.-Y. Liu, ``Denoispeech: Denoising text to speech with frame-level noise modeling,'' in _Proc. ICASSP_ , 2021, pp. 7063–7067.
  * [9] J. Su, Z. Jin, and A. Finkelstein, ``Hifi-gan: High-fidelity denoising and dereverberation based on speech deep features in adversarial networks,'' in _Proc. Interspeech_ , 2020, pp. 4506–4510.
  * [10] A. A. Nair and K. Koishida, ``Cascaded time+ time-frequency unet for speech enhancement: Jointly addressing clipping, codec distortions, and gaps,'' in _Proc. ICASSP_ , 2021, pp. 7153–7157.
  * [11] S. Pascual, J. Serrà, and A. Bonafonte, ``Towards generalized speech enhancement with generative adversarial networks,'' in _Proc. Interspeech_ , 2019, pp. 1791–1795.
  * [12] H. Liu, X. Liu, Q. Kong, Q. Tian, Y. Zhao, D. Wang, C. Huang, and Y. Wang, ``Voicefixer: A unified framework for high-fidelity speech restoration,'' in _Proc. Interspeech_ , 2022, pp. 4232–4236.
  * [13] J. Serrà, S. Pascual, J. Pons, R. O. Araz, and D. Scaini, ``Universal speech enhancement with score-based diffusion,'' _CoRR_ , vol. abs/2206.03065, 2022.
  * [14] C. Saharia, W. Chan, H. Chang, C. Lee, J. Ho, T. Salimans, D. Fleet, and M. Norouzi, ``Palette: Image-to-image diffusion models,'' in _ACM SIGGRAPH_ , 2022, pp. 1–10.
  * [15] J. Zhang, S. Jayasuriya, and V. Berisha, ``Restoring degraded speech via a modified diffusion model,'' in _Proc. Interspeech_ , 2021, pp. 221–225.
  * [16] Y.-J. Lu, Y. Tsao, and S. Watanabe, ``A study on speech enhancement based on diffusion probabilistic model,'' in _2021 Asia-Pacific Signal and Information Processing Association Annual Summit and Conference (APSIPA ASC)_. IEEE, 2021, pp. 659–666.
  * [17] Y.-J. Lu, Z.-Q. Wang, S. Watanabe, A. Richard, C. Yu, and Y. Tsao, ``Conditional diffusion probabilistic model for speech enhancement,'' in _Proc. ICASSP_. IEEE, 2022, pp. 7402–7406.
  * [18] Z. Kong, W. Ping, J. Huang, K. Zhao, and B. Catanzaro, ``Diffwave: A versatile diffusion model for audio synthesis,'' in _International Conference on Learning Representations, ICLR_ , 2021.
  * [19] S. Welker, J. Richter, and T. Gerkmann, ``Speech enhancement with score-based generative models in the complex STFT domain,'' in _Proc. Interspeech_. ISCA, 2022, pp. 2928–2932.
  * [20] Y. Song, J. Sohl-Dickstein, D. P. Kingma, A. Kumar, S. Ermon, and B. Poole, ``Score-based generative modeling through stochastic differential equations,'' in _International Conference on Learning Representations, ICLR_ , 2021.
  * [21] J. Ho, A. Jain, and P. Abbeel, ``Denoising diffusion probabilistic models,'' _Advances in Neural Information Processing Systems_ , vol. 33, pp. 6840–6851, 2020.
  * [22] C. Lu, Y. Zhou, F. Bao, J. Chen, C. Li, and J. Zhu, ``DPM-solver: A fast ODE solver for diffusion probabilistic model sampling in around 10 steps,'' in _Advances in Neural Information Processing Systems_ , 2022.
  * [23] K. Kinoshita, M. Delcroix, A. Ogawa, and T. Nakatani, ``Text-informed speech enhancement with deep neural networks,'' in _Proc. Interspeech_ , 2015, pp. 1760–1764.
  * [24] W. Wang, W. Zhang, S. Lin, and Y. Qian, ``Text-informed knowledge distillation for robust speech enhancement and recognition,'' in _Proc. ISCSLP_. IEEE, 2022, pp. 334–338.
  * [25] V. Popov, I. Vovk, V. Gogoryan, T. Sadekova, and M. Kudinov, ``Grad-tts: A diffusion probabilistic model for text-to-speech,'' in _International Conference on Machine Learning_. PMLR, 2021, pp. 8599–8608.
  * [26] A. Défossez, G. Synnaeve, and Y. Adi, ``Real time speech enhancement in the waveform domain,'' in _Proc. Interspeech_ , 2020, pp. 3291–3295.
  * [27] C. Yu, H. Lu, N. Hu, M. Yu, C. Weng, K. Xu, P. Liu, D. Tuo, S. Kang, G. Lei, D. Su, and D. Yu, ``DurIAN: Duration Informed Attention Network for Speech Synthesis,'' in _Proc. Interspeech 2020 – 21 st Annual Conference of the International Speech Communication Association_, 2020, pp. 2027–2031.
  * [28] J. Kong, J. Kim, and J. Bae, ``Hifi-gan: Generative adversarial networks for efficient and high fidelity speech synthesis,'' _Advances in Neural Information Processing Systems_ , vol. 33, pp. 17 022–17 033, 2020.
  * [29] T. Lee, W. K. Lo, P.-C. Ching, and H. Meng, ``Spoken language resources for cantonese speech processing,'' _Speech Communication_ , vol. 36, no. 3-4, pp. 327–342, 2002.
  * [30] C. K. A. Reddy, V. Gopal, R. Cutler, E. Beyrami, R. Cheng, H. Dubey, S. Matusevych, R. Aichner, A. Aazami, S. Braun, P. Rana, S. Srinivasan, and J. Gehrke, ``The INTERSPEECH 2020 deep noise suppression challenge: Datasets, subjective testing framework, and challenge results,'' in _Proc. Interspeech_ , 2020, pp. 2492–2496.
  * [31] T. Ko, V. Peddinti, D. Povey, M. L. Seltzer, and S. Khudanpur, ``A study on data augmentation of reverberant speech for robust speech recognition,'' in _Proc. ICASSP_ , 2017, pp. 5220–5224.
  * [32] J. Yamagishi, C. Veaux, K. MacDonald _et al._ , ``Cstr vctk corpus: English multi-speaker corpus for cstr voice cloning toolkit (version 0.92),'' _University of Edinburgh. The Centre for Speech Technology Research (CSTR)_ , 2019.
  * [33] D. P. Kingma and J. Ba, ``Adam: A method for stochastic optimization,'' in _Proc. ICLR_ , 2015.
  * [34] M. McAuliffe, M. Socolof, S. Mihuc, M. Wagner, and M. Sonderegger, ``Montreal forced aligner: Trainable text-speech alignment using kaldi.'' in _Proc. Interspeech_ , vol. 2017, 2017, pp. 498–502.