11institutetext: Department of Physics, The Applied Math Program, and
Department of Astronomy, The University of Arizona, Tucson, Arizona 85721,
USA; 11email<EMAIL_ADDRESS>
# The Cosmic Timeline Implied by the JWST Reionization Crisis

F. Melia John Woodruff Simpson Fellow

(Received May 31, 2024)

###### Abstract

Context. JWST’s discovery of well-formed galaxies and supermassive black holes
only a few hundred Myr after the big bang, and the identification of
polycyclic aromatic hydrocarbons (PAHs) at $z=6.71$, seriously challenge the
timeline predicted by $\Lambda$CDM. Moreover, the implied bright UV-
irradiation of the early Universe suggests a reionization history much too
short to comply with the observed evolution of the hydrogen ionization
fraction, $x_{HII}(z)$.

Aims. A recent analysis of reionization after JWST by Muñoz et al. (2024) has
concluded that the $\Lambda$CDM timeline simply cannot accommodate the
combined JWST-Planck observations even if exotic fixes are introduced to
modify the standard reionization model. In this paper, we argue that this so-
called ‘photon budget crisis’ is more likely due to flaws in the cosmological
model itself. We aim to re-analyze the data in the context of established
astrophysics with $R_{\rm h}=ct$ as the background cosmology, an approach that
has already been shown to mitigate the tension created by the too-early
appearance of galaxies, quasars and PAHs.

Methods. We employ the standard reionization model using the JWST-measured UV
luminosity function in the early Universe and the timeline and physical
conditions in both $\Lambda$CDM and $R_{\rm h}=ct$. The former has already
been fully probed and discussed by Muñoz et al. (2024), and we here merely
redo the calculation to ensure consistency with their pioneering work. We then
contrast the predicted reionization histories in these two scenarios and
compare them with the data.

Results. We confirm that the reionization history predicted by $\Lambda$CDM is
in significant tension with the observations, and demonstrate that the latter
are instead in excellent agreement with the $R_{\rm h}=ct$ timeline.

Conclusions. Together, the four anomalies uncovered by JWST, including the
newly discovered reionization ‘crisis’ discussed in this paper, provide strong
evidence against the timeline predicted by $\Lambda$CDM and in favor of the
evolutionary history in $R_{\rm h}=ct$.

###### Key Words.:

cosmology: theory – cosmology: observations – gravitation – early universe

††offprints: F. Melia

##  1 Introduction

Observational constraints indicate that the epoch of reionization (EoR)
probably began around redshift $z\sim 10-15$ and ended by $z\sim 6$ (McGreer
et al., 2015; Greig et al., 2017; Sobacchi & Mesinger, 2015; Mason et al.,
2019; Whitler et al., 2020; Wang et al., 2020; Nakane et al., 2024). It is
believed that Lyman continuum radiation from early galaxies (Robertson et al.,
2015) (with a possible contribution from emerging quasars; Madau & Haardt
2015) reionized the expanding gas following the so-called dark ages that began
soon after recombination at $t\sim 380\,000$ yr in the context of
$\Lambda$CDM.

Ironically, the earliest attempts at understanding how reionization came about
struggled to reconcile an apparent paucity of ionizing radiation at high
redshifts with the timescale at which this process took place, beginning with
cosmic microwave background (CMB) constraints from WMAP (Komatsu et al.,
2011), then improving with newer CMB data from Planck (Planck Collaboration et
al., 2016), and even from considerations (Melia & Fatuzzo, 2016) of the
alternative Friedmann-Lemaître-Robertson-Walker (FLRW) cosmology known as the
$R_{\rm h}=ct$ universe (Melia, 2007; Melia & Shevchuk, 2012; Melia, 2020),
the principal focus of this paper.

But JWST has completely reversed this picture, revealing a surprising number
of well-formed galaxies at $z$ approaching 14 and beyond, a mere few hundred
Myr after the big bang—much earlier than expected in the context of
$\Lambda$CDM. These first galaxies were prolific emitters of ionizing photons
which, as we shall confirm shortly, produced far too much UV irradiation of
the intergalactic medium (IGM) in the context of $\Lambda$CDM (Muñoz et al.,
2024) to allow reionization to extend over the observed redshift range
$15\gtrsim z\gtrsim 6$.

The incorrect evolutionary history implied by this finding represents a fourth
strike delivered by JWST against the predictions of the standard model. Most
glaringly, well-formed galaxies (Melia, 2023b; López-Corredoira et al., 2024)
and quasars (Melia, 2024b) have been discovered well before structure
formation should have begun in the early Universe, followed by the detection
of polycyclic aromatic hydrocarbons (PAHs) at $z=6.71$ (Witstok et al., 2023;
Melia, 2024a). These dust grains should have taken about $1$ Gyr to form via
the standard AGB channel, especially at low metallicities, yet they appear a
mere $\sim 500$ Myr after the first stars are supposed to have formed in
$\Lambda$CDM.

In this paper, we shall argue that all four of these significant problems
consistently point to a single flaw in the standard model—its incorrect time
versus redshift relation. We already know from earlier work (Melia, 2023b,
2024a) that the timeline predicted by the $R_{\rm h}=ct$ universe eliminates
the first three of these anomalies. But the motivation for considering whether
this model might also mitigate the reionization crisis extends well beyond
these results. In recent years, we have carried out numerous comparative tests
between $R_{\rm h}=ct$ and $\Lambda$CDM, showing that the data tend to favor
the former over the latter with a likelihood $\sim 90\%$ versus $\sim 10\%$,
based on over 30 different kinds of observations, at both high and low
redshifts. A short summary of these publications may be found in Table 2 of
Melia (2018), and a more complete description of the model’s fundamental
origin and scientific justification may be seen in Melia (2020).

We shall begin by re-analyzing the standard reionization model in
$\Lambda$CDM, solely for the purpose of establishing consistency with the
pioneering work of Muñoz et al. (2024). We assume flat $\Lambda$CDM with
$h=0.7$, $\Omega_{\rm m}=0.3$ and $\Omega_{\rm b}=0.0224/h^{2}$ to match
previous fiducial models, and use AB magnitudes throughout (Oke & Gunn, 1983).
As we shall see, our results fully confirm their conclusions, establishing a
need to consider the alternative $R_{\rm h}=ct$ cosmology. We shall then
repeat the analysis for the timeline and geometry in $R_{\rm h}=ct$, showing
that both the reionization history and CMB optical depth in this model are
well supported by the JWST and Planck data. We shall thereby conclude that the
self-consistent resolution of all four anomalies uncovered by JWST via the
replacement of $\Lambda$CDM with $R_{\rm h}=ct$ argues compellingly in favor
of the latter as the correct cosmological model.

##  2 Standard Reionization History

The three primary factors determining the conventional reionization history
include: (i) the average ionizing photon emissivity, (ii) the fraction $f_{\rm
esc}$ of these photons escaping into the IGM, and (iii) the number of
recombinations per hydrogen atom. The first of these is given by the abundance
of galaxies at high $z$, expressed in terms of the UV luminosity function
(UVLF), $\Phi_{\rm UV}$, in units of the comoving number density of galaxies
per UV magnitude, $M_{\rm UV}$, times the ionizing efficiency $\xi_{\rm ion}$
of each galaxy. The new JWST observations show that early galaxies have higher
ionizing efficiencies than previously assumed, with

$\log_{10}(\xi_{\rm ion}/{\rm Hz\;erg}^{-1})\approx 25.8+0.11(M_{\rm
UV}+17)+0.05(z-7)\,,$ (1)

versus $\sim 25.2$ (Atek et al., 2024; Simmonds et al., 2024; Endsley et al.,
2023; Calabro et al., 2024). Following Muñoz et al. (2024), we shall
conservatively cap $\xi_{\rm ion}$ at $z=9$ and $M_{\rm UV}=-16.5$ to avoid
extrapolation.

The escape fraction in high-$z$ galaxies is not well motivated theoretically,
and difficult to measure while the IGM contains neutral hydrogen. Studies of
reionization-galaxy analogues at low $z$ have found a strong correlation
between $f_{\rm esc}$ and the UV slopes, $\beta_{\rm UV}$, however, which
provides a rough estimate (Flury et al., 2022; Chisholm et al., 2022; Begley
et al., 2022). Based on these studies, the slopes of JWST galaxies imply that
$f_{\rm esc}\sim 5-15\%$ (Mascia et al., 2023; Lin et al., 2024).

We shall use the fit from Chisholm et al. (2022),

$f_{\rm esc}=A_{f}\times 10^{\,b_{f}\beta_{\rm UV}}\;,$ (2)

where $A_{f}=1.3\times 10^{-4}$ and $b_{f}=-1.22$. This expression may be
combined with the $\beta_{\rm UV}-M_{\rm UV}$ measurements in Zhao &
Furlanetto (2024), incorporating data from both JWST and HST (Bouwens et al.,
2014; Topping et al., 2022; Cullen et al., 2023), to estimate $f_{\rm
esc}(M_{\rm UV})$. The UV slopes are capped at $\beta_{\rm UV}=-2.7$ to avoid
extrapolation, since these limits correspond to the bluest galaxies for which
Equation (2) was calibrated.

The volume-averaged hydrogen ionized fraction, $x_{\rm HII}=1-x_{\rm HI}$,
where $x_{\rm HI}\equiv n_{\rm HI}/n_{\rm H}$ is the corresponding neutral
fraction, evolves according to the equation (Madau et al., 1999)

$\dot{x}_{\rm HII}={\dot{n}_{\rm ion}\over n_{\rm H}}-{x_{\rm HII}\over t_{\rm
rec}}\;,$ (3)

where $\dot{n}_{\rm ion}$ is the rate of ionizing-photon production, $n_{\rm
H}=\rho_{\rm b}(1-Y_{\rm He})/m_{\rm H}c^{2}$, $Y_{\rm He}$ is the helium mass
fraction, $m_{\rm H}$ is the proton mass, and $\rho_{\rm b}$ is the baryon
energy density.

The recombination timescale is given as (Shull et al., 2012)

$t_{\rm rec}=\left[C\alpha_{\rm B}(1+x_{\rm He})n_{\rm H}\right]^{-1}\;,$ (4)

where $x_{\rm He}\equiv n_{\rm He}/n_{\rm H}\approx Y_{\rm He}/[4(1-Y_{\rm
He})]$ is the helium fraction, $\alpha_{\rm B}$ is the case-B recombination
coefficient, and $C$ is the clumping factor. We shall follow previous work and
set $C=3$ and evaluate $\alpha_{\rm B}$ at $T=2\times 10^{4}$ K.

The ionizing-photon production rate may be expressed as

$\dot{n}_{\rm ion}=\int dM_{\rm UV}\,(1+z)^{3}\Phi_{\rm UV}\dot{N}_{\rm
ion}f_{\rm esc}\;,$ (5)

where the factor $(1+z)^{3}$ converts $\Phi_{\rm UV}$ into a density per unit
proper volume, and the integral extends down to a cutoff magnitude that is
essentially a free parameter, though its impact on the results is small. The
directly observed galaxies extend down to magnitude $M_{\rm UV}\sim-15$. If
one includes galaxies much fainter than this, the implied escape fraction
$f_{\rm esc}$ would appear to be unrealistically small (Muñoz et al., 2024).
In addition, one expects a turnover due to feedback (Shapiro et al., 2004),
which seems to be confirmed by HST observations, with $M_{\rm UV}\sim 15$
(Atek et al., 2018). In this paper, we shall therefore conservatively take the
cutoff to be $M_{\rm UV}=-14.6$ in $R_{\rm h}=ct$, and assume the same value
for $\Lambda$CDM as well. Our result for the standard model, summarized in
Figure 1 below, is thus somewhat less extreme than that of Muñoz et al.
(2024), who instead integrated Equation (5) down to $M_{\rm UV}=-13$. The UVLF
is taken from the pre-JWST fit in Bouwens et al. (2021) at $z\leq 9$, and from
Donnan et al. (2024) at $z>9$. Lastly, $\dot{N}_{\rm ion}\equiv L_{\rm
UV}\,\xi_{\rm ion}$ is the ionizing-photon production rate per galaxy, in
terms of their UV luminosity $L_{\rm UV}$ and ionizing efficiently $\xi_{\rm
ion}$.

Figure 1: Reionization history in $\Lambda$CDM based on new JWST observations
of structure formation in the early Universe. Lower panel: the neutral
fraction $x_{HI}$ as a function of redshift $z$, for a cutoff $M_{UV}=-14.6$
and a JWST-calibrated ionizing efficiency, $\xi_{\rm ion}$, and escape
fraction ($f_{\rm esc}$) inferred from low-$z$ analogues. The blue data points
represent measured observational constraints (see text for references). Upper
panel: CMB optical depth, $\tau_{\rm CMB}$, as a function of $z$. The yellow
band represents the Planck measurement and its uncertainty (Planck
Collaboration et al., 2020). The $\chi^{2}_{\rm dof}$ for the $x_{\rm HI}$ fit
is $4.36$, and $\tau_{\rm CMB}$ differs from the Planck optimized value by
$\sim 2\sigma$. The new JWST galaxy observations thus create significant
tension for the standard model, both for the reionization history and the CMB
optical depth.

Correspondingly, the reionization history is used to calculate the CMB optical
depth,

$\tau_{\rm CMB}=\int dR\,n_{e}\sigma_{T}\;,$ (6)

where $R$ is proper distance, $\sigma_{T}$ is the Thomson cross section, and

$n_{e}=\left(1+{\alpha Y_{\rm He}\over 4(1-Y_{\rm He})}\right)x_{\rm
HII}\,n_{\rm H}$ (7)

is the proper electron number density, with $\alpha=1$ for $z>4$ and
$\alpha=2$ for $z\leq 4$. For simplicity, we assume that HeI reionization
tracks HI, and that HeII reionization occurs at $z=4$.

The lower panel in Figure 1 shows the hydrogen neutral fraction as a function
of $z$ calculated from Equation (3) with $x_{\rm HII}=0$ at $z=30$, assuming
$\Lambda$CDM is the correct cosmology. This theoretically predicted curve is
compared with the data (blue points), taken from McGreer et al. (2015); Greig
et al. (2017); Sobacchi & Mesinger (2015); Mason et al. (2019); Whitler et al.
(2020); Wang et al. (2020); Nakane et al. (2024). The upper panel shows the
corresponding CMB optical depth, along with the Planck measurement (blue):
$\tau_{\rm CMB}=0.054\pm 0.007$ (Planck Collaboration et al., 2020).

We fully confirm the result, first presented by Muñoz et al. (2024), that the
new JWST observations clearly imply a reionization history in $\Lambda$CDM
ending at $z\sim 9$. The disparity between the data and the predicted $x_{\rm
HI}(z)$ and $\tau_{\rm CMB}(z)$ profiles creates significant tension for the
standard model. In particular, note that the calculated CMB optical depth,
$\tau_{\rm CMB}\sim 0.08$, misses the Planck measurement by almost $4\sigma$,
giving rise to the so-called ‘JWST reionization crisis.’

Muñoz et al. (2024) extended their analysis beyond this point, principally to
discuss all plausible physical mechanisms, theoretical and observational, that
might resolve this photon budget anomaly. The outcome of that work, however,
is that all of the exotic fixes attempted thus far are in conflict with at
least one observational constraint. There is yet no known solution to this
emerging problem with the standard model.

##  3 Reionization History in $R_{\rm h}=ct$

Let us now take the opposite approach, assume that the astrophysics of
reionization is basically correct, and instead redo the calculation for
$R_{\rm h}=ct$. There are several critical differences between the timeline
and geometry of an $R_{\rm h}=ct$ universe and one driven by $\Lambda$CDM.

The first modification arises from the age-redshift relation. In $\Lambda$CDM,
we have

$t^{\Lambda{\rm CDM}}(z)={1\over
H_{0}}\int_{z}^{\infty}{du\over\sqrt{\Omega_{\rm m}(1+u)^{3}+\Omega_{\rm
r}(1+u)^{4}+\Omega_{\Lambda}}}$ (8)

where, in keeping with our assumption of a flat Universe and dark energy in
the form of a cosmological constant, we have simply
$\Omega_{\Lambda}=1-\Omega_{\rm m}-\Omega_{\rm r}$. The corresponding
expression for $R_{\rm h}=ct$ is

$t^{R_{\rm h}}(z)={1\over H_{0}(1+z)}\;.$ (9)

As demonstrated elsewhere, e.g., Melia & Fatuzzo (2016), these relations show
that the Universe is approximately the same age today in both models, but
$t^{R_{\rm h}}$ is roughly twice $t^{\Lambda{\rm CDM}}$ at $z\gtrsim 6$. This
is the reason, of course, why the $R_{\rm h}=ct$ universe completely
eliminates the ‘too early’ galaxy and supermassive black hole problems.

In addition, the two models differ significantly in terms of geometry. The
differential comoving distance at $z$ in $\Lambda$CDM is given as

$dD_{\rm com}^{\Lambda{\rm CDM}}(z)={c\over H_{0}}{dz\over\sqrt{\Omega_{\rm
m}(1+z)^{3}+\Omega_{\rm r}(1+z)^{4}+\Omega_{\Lambda}}}\,,$ (10)

while in $R_{\rm h}=ct$ it is

$dD_{\rm com}^{R_{\rm h}}(z)={c\over H_{0}}{dz\over(1+z)}\;.$ (11)

Surprisingly, the two integrated comoving distances track each other rather
closely up to $z\sim 8$, but then deviate considerably towards higher
redshifts. For example, the ratio of proper volumes at $z=15$ is $dV_{\rm
com}^{\Lambda{\rm CDM}}/dV_{\rm com}^{R_{\rm h}}\sim 0.1$.

Figure 2: Same as Fig. 1, except now for the $R_{\rm h}=ct$ cosmology and a
cutoff $M_{UV}=-14.6$. The $\chi^{2}_{\rm dof}$ for the $x_{\rm HI}$ fit is
$1.1$. The reionization history in this model is thus fully consistent with
the UV irradiation of the early Universe by the newly discovered JWST
galaxies, and the corresponding CMB optical depth falls within $\sim 1\sigma$
of its measured value even though, in principle, the optimized cosmological
parameters in this model need not be identical to those inferred for
$\Lambda$CDM.

Given these two principal differences between the two models, one should also
expect variations in the parameters, such as $H_{0}$ and $\Omega_{\rm b}$,
optimized to fit the full range of cosmological observations. But it turns out
these differences are usually rather small, mainly because it appears that the
formulation in $\Lambda$CDM, which was largely developed empirically, produces
an expansion history mimicking that of $R_{\rm h}=ct$. It is beyond the scope
of the present work to carry out such detailed comparisons, which have
actually already appeared in the literature, as one may see from Table 2 of
Melia (2018). For simplicity (and transparency) in this paper, we shall
therefore adopt the $H_{0}$ and $\Omega_{\rm b}$ values optimized for
$\Lambda$CDM.

Finally, to solve Equation (3) starting with $x_{\rm HII}=0$ at $z=30$, we
also need to recalibrate the UVLF $\Phi_{\rm UV}$ in Equation (5), assembled
for $\Lambda$CDM, to make it compatible with the proper volume differences,
$\propto(dD_{\rm com}^{\Lambda{\rm CDM}}/dD_{\rm com}^{R_{\rm h}})^{3}$,
between $\Lambda$CDM and $R_{\rm h}=ct$. The fits used for $\Phi_{\rm
UV}(M_{\rm UV},z)$ are loosely based on the predictions of $\Lambda$CDM and
$R_{\rm h}=ct$, but are largely empirical, as noted earlier, based on pre-JWST
observations at $z\leq 9$ (Bouwens et al., 2021) and JWST calibrations at
$z>9$ (Donnan et al., 2024). These expressions are inferred primarily from the
observed number of galaxies as a function of $M_{\rm UV}$ and $z$, and the
density is then determined using a conversion of measured redshift to model-
dependent comoving volume. The data are model independent, as they depend only
on $z$. Previous work had already confirmed that the halo mass function may be
formulated in a similar way in these two models (Yennapureddy & Melia, 2019),
the difference arising solely from a recalibration of the volumes. And this is
confirmed by a detailed comparison of the matter power spectra in $\Lambda$CDM
and $R_{\rm h}=ct$, which fit the data equally well once the volume
recalibration is taken into account (Yennapureddy & Melia, 2021). Thus, the
difference in $\Phi_{\rm UV}$ between the two models is due solely to the
difference in their predicted volumes, and for the analysis in this paper, it
is therefore sufficient to merely recalibrate the UV luminosity function used
by Muñoz et al. (2024) via the change in comoving (or, equally, proper) volume
from $\Lambda$CDM to $R_{\rm h}=ct$.

The calculated hydrogen neutral fraction in $R_{\rm h}=ct$ is shown as a
function of $z$ in the lower panel of Figure 2. The corresponding CMB optical
depth in this model is shown in the upper panel. All the other features in
this figure are identical to those in Figure 1.

Though we are not necessarily optimizing any model parameters in this paper,
it is nevertheless helpful to assess how well each of the cosmologies fits the
data in Figures 1 and 2. The $\chi^{2}_{\rm dof}$ of the fit based on the
$\Lambda$CDM prediction (Fig. 1) is $4.36$, while the calculated $\tau_{\rm
CMB}$ is $0.068$, which differs from the Planck value by about $2\sigma$. By
comparison, $\chi^{2}_{\rm dof}$ for $R_{\rm h}=ct$ is 1.1 (Fig. 2), and
$\tau_{\rm CMB}\approx 0.062$, which agrees with Planck to within $\sim
1\sigma$. Were we to use these outcomes as a basis for head-to-head model
selection using, e.g., the Aikake Information Criterion (Akaike, 1974; Melia &
Maier, 2013), we would infer a relative likelihood for $\Lambda$CDM of only
$\sim 10^{-10}$, clearly implying solid support for the $R_{\rm h}=ct$
cosmology by the reionization data.

##  4 Discussion and conclusion

The mitigation of the JWST reionization crisis, from a comparison of Figures 1
and 2, is quite evident and robust. It is essential to understand that only
the cutoff in $M_{\rm UV}$ for the integration in Equation (5) is somewhat
undetermined, as described earlier. All one can say is that the cutoff must be
$>-15$, and could have a value extending perhaps to $-12$ or $-13$, as assumed
by Muñoz et al. (2024). Of course, the larger this value is, the more serious
the ‘crisis’ becomes, because it enhances the UV irradiation of the IGM. We
have here simply taken a conservative approach in using a cutoff as small as
one could reasonably adopt based on the observations, in order to give
$\Lambda$CDM the benefit of the doubt, so its value was not fine-tuned, but
merely assumed to be somewhat below the observational constraints, i.e.,
$-14.6$. All the other factors used in this analysis, including $f_{\rm esc}$,
$\xi_{\rm ion}$, $\Phi_{\rm UV}$ and so forth, were strictly fixed by the
previous analyses.

It is thus rather easy to understand what ‘solves’ the problem. By far, the
most significant difference between the calculation of $x_{\rm HII}(z)$ in
$\Lambda$CDM and $R_{\rm h}=ct$ is the impact on the UVLF $\Phi_{\rm UV}$ of
the proper volume. The Universe was larger at $z\gtrsim 10$ in $R_{\rm h}=ct$
than it would have been in $\Lambda$CDM. As a result, $\Phi_{\rm UV}(z)$ is
smaller in the former than in the latter for the same number of galaxies
observed at $z$. The UV-irradiation of the IGM was therefore less severe in
$R_{\rm h}=ct$, which lowered the reionization rate and thereby delayed the
end of the reionization epoch (to $z\sim 6$).

We have now witnessed at least four major discoveries by JWST in the early
Universe that have created significant tension for $\Lambda$CDM. And while
exotic fixes may be introduced to fix each of them individually, e.g., by
proposing that supermassive black holes were created as $\sim
10^{5}\;M_{\odot}$ seeds (Yoo & Miralda-Escudé, 2004; Latif et al., 2013;
Alexander & Natarajan, 2014)—a process that has never been seen anywhere in
the cosmos—eventually one must acknowledge the fact that a scenario in which a
single modification simultaneously eliminates all of the anomalies ought to be
given serious consideration as being the correct approach. Replacing
$\Lambda$CDM with $R_{\rm h}=ct$ as the background cosmology certainly does
this—and rather impressively—as we have seen in this paper.

Looking forward, the prospect of $R_{\rm h}=ct$ being the correct cosmology is
quite alluring because it solves many, if not all, of the paradoxes and
conflicts accumulated by the standard model. For example, it eliminates all
horizon problems, including those associated with the CMB temperature and the
Electroweak phase transition (Melia, 2013, 2018). And it eliminates the cosmic
entropy anomaly (Melia, 2021), the monopole problem (Melia, 2023a), and
inflation’s violation of the strong energy condition in general relativity
(Melia, 2023c), among many others.

Finally, recent fundamental work with the FLRW metric itself has demonstrated
that the overly simplified FLRW ansatz, with a lapse function $g_{00}=1$, may
in fact be valid only for the zero active mass equation of state, $\rho+3p=0$,
in terms of the total energy density $\rho$ and pressure $p$, which produces
the constant expansion rate in $R_{\rm h}=ct$ (Melia, 2022). In other words,
$R_{\rm h}=ct$ may be the only valid FLRW solution consistent with the
foundational symmetries in the Cosmological principle.

## References

  * Akaike (1974) Akaike, H. 1974, IEEE Transactions on Automatic Control, 19, 716
  * Alexander & Natarajan (2014) Alexander, T. & Natarajan, P. 2014, Science, 345, 1330
  * Atek et al. (2024) Atek, H., Labbé, I., Furtak, L. J., et al. 2024, Nature, 626, 975
  * Atek et al. (2018) Atek, H., Richard, J., Kneib, J.-P., & Schaerer, D. 2018, MNRAS , 479, 5184
  * Begley et al. (2022) Begley, R., Cullen, F., McLure, R. J., et al. 2022, MNRAS , 513, 3510
  * Bouwens et al. (2014) Bouwens, R. J., Bradley, L., Zitrin, A., et al. 2014, ApJ, 795, 126
  * Bouwens et al. (2021) Bouwens, R. J., Oesch, P. A., Stefanon, M., et al. 2021, Astronomical Journal, 162, 47
  * Calabro et al. (2024) Calabro, A., Castellano, M., Zavala, J. A., et al. 2024, arXiv e-prints, arXiv:2403.12683
  * Chisholm et al. (2022) Chisholm, J., Saldana-Lopez, A., Flury, S., et al. 2022, MNRAS , 517, 5104
  * Cullen et al. (2023) Cullen, F., McLure, R. J., McLeod, D. J., et al. 2023, MNRAS , 520, 14
  * Donnan et al. (2024) Donnan, C. T., McLure, R. J., Dunlop, J. S., et al. 2024, arXiv e-prints, arXiv:2403.03171
  * Endsley et al. (2023) Endsley, R., Stark, D. P., Whitler, L., et al. 2023, arXiv e-prints, arXiv:2306.05295
  * Flury et al. (2022) Flury, S. R., Jaskot, A. E., Ferguson, H. C., et al. 2022, ApJ Supplements, 260, 1
  * Greig et al. (2017) Greig, B., Mesinger, A., Haiman, Z., & Simcoe, R. A. 2017, MNRAS , 466, 4239
  * Komatsu et al. (2011) Komatsu, E., Smith, K. M., Dunkley, J., et al. 2011, ApJ Supplements, 192, 18
  * Latif et al. (2013) Latif, M. A., Schleicher, D. R. G., Schmidt, W., & Niemeyer, J. 2013, MNRAS , 433, 1607
  * Lin et al. (2024) Lin, Y.-H., Scarlata, C., Williams, H., et al. 2024, MNRAS , 527, 4173
  * López-Corredoira et al. (2024) López-Corredoira, M., Melia, F., Wei, J.-J., & C.-Y., G. 2024, ApJ, in press
  * Madau & Haardt (2015) Madau, P. & Haardt, F. 2015, ApJ Letters, 813, L8
  * Madau et al. (1999) Madau, P., Haardt, F., & Rees, M. J. 1999, ApJ, 514, 648
  * Mascia et al. (2023) Mascia, S., Pentericci, L., Calabrò, A., et al. 2023, A&A, 672, A155
  * Mason et al. (2019) Mason, C. A., Fontana, A., Treu, T., et al. 2019, MNRAS , 485, 3947
  * McGreer et al. (2015) McGreer, I. D., Mesinger, A., & D’Odorico, V. 2015, MNRAS , 447, 499
  * Melia (2007) Melia, F. 2007, MNRAS , 382, 1917
  * Melia (2013) Melia, F. 2013, A&A, 553, A76
  * Melia (2018) Melia, F. 2018, MNRAS , 481, 4855
  * Melia (2020) Melia, F. 2020, The Cosmic Spacetime (Oxford: Taylor and Francis)
  * Melia (2021) Melia, F. 2021, arXiv e-prints, arXiv:2103.03933
  * Melia (2022) Melia, F. 2022, Modern Physics Letters A, 37, 2250016
  * Melia (2023a) Melia, F. 2023a, Physics of the Dark Universe, 42, 101329
  * Melia (2023b) Melia, F. 2023b, MNRAS , 521, L85
  * Melia (2023c) Melia, F. 2023c, Annalen der Physik, 535, 2300157
  * Melia (2024a) Melia, F. 2024a, Nature Astronomy, submitted
  * Melia (2024b) Melia, F. 2024b, Proceedings of the Royal Society A, submitted
  * Melia & Fatuzzo (2016) Melia, F. & Fatuzzo, M. 2016, MNRAS , 456, 3422
  * Melia & Maier (2013) Melia, F. & Maier, R. S. 2013, MNRAS , 432, 2669
  * Melia & Shevchuk (2012) Melia, F. & Shevchuk, A. S. H. 2012, MNRAS , 419, 2579
  * Muñoz et al. (2024) Muñoz, J. B., Mirocha, J., Chisholm, J., Furlanetto, S. R., & Mason, C. 2024, arXiv e-prints, arXiv:2404.07250
  * Nakane et al. (2024) Nakane, M., Ouchi, M., Nakajima, K., et al. 2024, ApJ, 967, 28
  * Oke & Gunn (1983) Oke, J. B. & Gunn, J. E. 1983, ApJ, 266, 713
  * Planck Collaboration et al. (2016) Planck Collaboration, Ade, P. A. R., Aghanim, N., et al. 2016, A&A, 594, A13
  * Planck Collaboration et al. (2020) Planck Collaboration, Aghanim, N., Akrami, Y., et al. 2020, A&A, 641, A6
  * Robertson et al. (2015) Robertson, B. E., Ellis, R. S., Furlanetto, S. R., & Dunlop, J. S. 2015, ApJ Letters, 802, L19
  * Shapiro et al. (2004) Shapiro, P. R., Iliev, I. T., & Raga, A. C. 2004, MNRAS , 348, 753
  * Shull et al. (2012) Shull, J. M., Harness, A., Trenti, M., & Smith, B. D. 2012, ApJ, 747, 100
  * Simmonds et al. (2024) Simmonds, C., Tacchella, S., Hainline, K., et al. 2024, MNRAS , 527, 6139
  * Sobacchi & Mesinger (2015) Sobacchi, E. & Mesinger, A. 2015, MNRAS , 453, 1843
  * Topping et al. (2022) Topping, M. W., Stark, D. P., Endsley, R., et al. 2022, ApJ, 941, 153
  * Wang et al. (2020) Wang, F., Davies, F. B., Yang, J., et al. 2020, ApJ, 896, 23
  * Whitler et al. (2020) Whitler, L., Mason, C., Ren, K., & Conroy, C. 2020, in American Astronomical Society Meeting Abstracts, Vol. 235, American Astronomical Society Meeting Abstracts #235, 207.04
  * Witstok et al. (2023) Witstok, J., Shivaei, I., Smit, R., et al. 2023, Nature, 621, 267
  * Yennapureddy & Melia (2019) Yennapureddy, M. K. & Melia, F. 2019, European Physical Journal C, 79, 571
  * Yennapureddy & Melia (2021) Yennapureddy, M. K. & Melia, F. 2021, Physics of the Dark Universe, 31, 100752
  * Yoo & Miralda-Escudé (2004) Yoo, J. & Miralda-Escudé, J. 2004, ApJ Letters, 614, L25
  * Zhao & Furlanetto (2024) Zhao, J. & Furlanetto, S. R. 2024, arXiv e-prints, arXiv:2401.07893


# Investigation of Adapter for Automatic Speech Recognition in Noisy
Environment

###### Abstract

Adapting an automatic speech recognition (ASR) system to unseen noise
environments is crucial. Integrating adapters into neural networks has emerged
as a potent technique for transfer learning. This study thoroughly
investigates adapter-based ASR adaptation in noisy environments. We conducted
experiments using the CHiME–4 dataset. The results show that inserting the
adapter in the shallow layer yields superior effectiveness, and there is no
significant difference between adapting solely within the shallow layer and
adapting across all layers. The simulated data helps the system to improve its
performance under real noise conditions. Nonetheless, when the amount of data
is the same, the real data is more effective than the simulated data. Multi-
condition training is still useful for adapter training. Furthermore,
integrating adapters into speech enhancement-based ASR systems yields
substantial improvements.

Index Terms—  Noise-robust ASR, adapter, transfer learning

##  1 Introduction

Automatic speech recognition (ASR) in noisy environments is still challenging
[1]. Deep learning-based ASR models [2, 3], however proficient, exhibit
susceptibility to performance deterioration when exposed to unencountered
noise sources. In today’s large-scale models [4, 5], the imperative to adapt
models to new noisy conditions with minimal data becomes crucial. Domain
adaptation [6] in ASR involves finetuning a model from one (source) domain to
work effectively in another (target) domain, managing challenges arising from
distribution disparities. Data augmentation [7, 8], transfer learning [9],
domain adversarial training [10], and multi-task learning [11] have shown
beneficial for adapting the model to specific tasks.

Adapter, which belongs to transfer learning [12], has been very popular
recently. It has been applied to various ASR tasks, empowering models to adapt
efficiently to specific challenges. Adapters serve as a crucial bridge in
addressing various low-resource ASR tasks [6, 13, 14, 15, 16, 17]. Moreover,
they have shown effectiveness in the disorder and children ASR [6, 16, 17].
Furthermore, when applying the pretrained model [4, 5], how to better transfer
the knowledge in a large model to new scenarios with limited data is
particularly important. Because the adapter meets such requirements, inserting
the adapter into a pretrained large model has also been widely studied [6, 12,
15, 18].

Despite the widespread application of adapters across various ASR tasks,
limited investigation has been conducted towards noise environment adaptation
for ASR. In this paper, we address this problem. Our primary focus encompasses
investigating the adapter insertion points, the data employed for adapter
training, and the synergy with speech enhancement (SE) front-end models.

##  2 Preliminaries

###  2.1 Conformer-based ASR System

Conformer-based ASR systems [19] have demonstrated leading-edge performance
across various benchmark datasets. Typically, these systems comprise two core
components: an encoder and a decoder. The encoder processes the input speech
signal and generates a sequence of feature vectors, encapsulating pertinent
linguistic information. Multiple Conformer layers systematically process the
input sequence, each operating at varying abstraction levels. Meanwhile, the
decoder is tasked with transforming the feature vector sequence, generated by
the encoder, into a character or phone sequence representing the transcription
of the input speech.

###  2.2 Speech Enhancement-based Robust ASR

Robust ASR systems [3, 20, 21] are purpose-built for challenging acoustic
environments. This paper narrows its focus to addressing background noise.
There are two primary ways to enhance ASR robustness. The first approach,
multi-conditioned ASR, involves training the ASR system with diverse noisy
datasets [3]. However, while effective, this method’s performance tends to be
limited in exceedingly noisy or low signal-to-noise ratio (SNR) scenarios. The
second approach involves incorporating an SE front-end [22]. This requires
separate pretraining of the SE front-end and the ASR back-end, often followed
by adopting joint training [23]. The joint training loss function combines the
ASR loss ($L_{ASR}$) and the SE loss ($\mathcal{L}_{SE}$). It should be noted
that computing SE loss ($\mathcal{L}_{SE}$) requires the original clean
speech, making it infeasible for real-world noisy data scenarios. As a result,
during joint training, only the ASR loss ($L_{ASR}$) is utilized.

###  2.3 Adapter

The adapter was initially proposed [24] to enhance transfer learning in
natural language processing tasks, particularly when faced with limited task-
specific data. Its strength is task customization without significantly
altering the model’s structure. This versatility makes it suitable for various
contexts, including accent recognition, emotion detection, and noise-robust
ASR. The design of an adapter depends on task requirements and the model
architecture [13]. The simple yet effective neural network structure of an
adapter consists of two dense layers. It takes input from a chosen layer in
the pretrained model’s output. The primary function of the first dense layer
is to capture the initial transformations of input features from the
pretrained model. The second layer builds on the transformed features from the
first dense layer and captures more intricate task-specific patterns and
representations. The adaptation process is depicted as follows:

$e^{\prime}=e+\text{adapter}(e)$ (1)

Here, $e$ represents the selected layer’s output, and $e^{\prime}$ represents
the adapted feature. The structure of the adapter is shown in Figure 1–(a). We
employ the LoRA [25] structure wherein the central dimension is smaller than
both the input and output dimensions.

Fig. 1: (a) Structure of adapter; (b) flowchart of the adapter-based
adaptation; (c) adapter-based adaptation with speech enhancement front-end.

##  3 Exploration of Adapter for
Noise Robust ASR

Incorporating adapters into ASR models is a prevalent practice for tasks like
accent ASR, children ASR, and multi-lingual ASR. However, noise-robust ASR,
another application that necessitates adaptation, has received limited
attention in adapter studies. Consequently, this paper delves into the
application of adapters for enhancing noise-robust speech recognition. In
order to gain a deeper understanding of the adapter’s role within this
context, our study takes a comprehensive approach, thoroughly investigating
its effect from multiple perspectives.

  * ❏

Where should the adapter be inserted?
The position where the adapter is inserted notably impacts the model’s
performance. To address this, some methods employ machine learning techniques
to automatically identify the optimal layers for adapter integration [18].
However, it is worth noting that the best place to insert the adapter can
differ based on the specific task. To gain a clear and intuitive understanding
of the adapter’s influence on noise-robust ASR, we investigate the most
effective layer for adapter insertion and assess whether stacking adapters
yield improved effect.

  * ❏

How to configure the embedding nodes in the adapter?
Modifications to the network structure can have a substantial impact on model
performance. However, in existing adapter-based research, focus is often
limited to a single chosen embedding dimension. Therefore, the impact of the
embedding dimension in the adapter on noise-robust speech recognition is
explored.

  * ❏

How training data affects the adapter?
Training data is crucial for deep learning-based ASR. To explore the effects
of different data quantities and types on adapter-based noise-robust ASR, we
conducted experiments employing diverse training datasets. We evaluate how
simulation and real training data with varying data quantities affect
adaptation. Furthermore, we investigate whether incorporating simulated data
could enhance the model’s performance on real data. Moreover, we contrast the
impact of models trained on a single noise scene with those trained under
multi-noise conditions.

  * ❏

Can the utilization of the adapter lead to further improvements for SE-based
noise-robust ASR system?
Utilizing a SE front-end can significantly improve the performance of ASR
systems. The SE front-end enhances features and adapts to the system at the
feature level to a certain extent. Hence, while addressing noise-robust ASR
challenges, the emphasis is frequently placed on enhancing the performance of
the SE front-end. This could result in adapters being relatively uncommon in
noise-robust ASR tasks. Hence, in this paper, we investigate whether adapters
can further enhance the performance of models adapted at the feature level.

The adapter is positioned after the encoder layer, shown in Figure 1–(b). We
perform pretraining on the ASR backend at the outset using a substantial
dataset. Following this, we freeze all parameters of the pretrained ASR
backend and insert the adapter after the encoder layer. To examine the
adapter’s impact in unseen noise scenarios, the training noise for the
pretrained ASR back-end differs from the noise used for evaluation. The SE-
based system is shown in Figure 1–(c).

##  4 Experiments

###  4.1 Experimental Settings

The experiments conducted utilized the CHiME-4
dataset111https://spandh.dcs.shef.ac.uk/chime_challenge/CHiME4/index.html. It
includes four different noise conditions: bus (bus), cafe (caf), pedestrian
area (ped), and street junction (str). The audio in the dataset was digitized
at a sampling rate of 16 kHz. Data from channels 1 to 6 were utilized during
the model training phase. The Channel 5 real noisy data from development and
evaluation sets were used for testing.

The Conformer-based ASR backend employed 6 encoder layers. Each encoder layer
utilized the relative positional encoding module for positional encoding,
encompassing a subsampling rate of 4 facilitated by 2 Conv2d layers. The
multi-head attention dimension was set at 512 with 4 attention heads.
Position-wise feedforward units numbered 2048, and the activation function
used was swish. The input LMFB feature was 80-dimension. The decoder, based on
the Transformer architecture [2], comprised 6 decoder layers, each
incorporating 512 dimensions for multi-head attention with 4 attention heads.
Hidden units for position-wise feedforward were specified as 2048. The
dictionary was constructed utilizing transcripts from CHiME-4, WSJ0
222https://catalog.ldc.upenn.edu/LDC93s6a, and WSJ1
333https://catalog.ldc.upenn.edu/LDC94S13A. The BPE vocabulary consisted of
1014 elements, which included $<blank>$, $<unk>$, and $<sos/eos>$. The ASR
backend was pretrained using WSJ0, WSJ1, and Librispeech (960 hours) [26]. In
the pretraining phase, we synthesized noisy speech by combining the MUSAN
dataset [27] with clean speech, randomly selecting signal-to-noise ratios
(SNRs) within the range of 0 to 20 dB. Pretraining was conducted 100 epochs
with the SpecAug [28]. The final pretrained model was an average of ten well-
performing checkpoints on the development set.

“Bi-LSTM” and “DEMUCS” [29] were chosen as the SE front-end. For “Bi-LSTM”,
the feature used was the magnitude of the spectrogram. The architecture
included two Bi-directional LSTM (Bi-LSTM) layers and a fully connected layer.
Each Bi-LSTM layer was equipped with 896 hidden nodes. There were 896 hidden
nodes in each Bi-LSTM layer. For “DEMUCS”, we followed the original neural
network architecture444https://github.com/facebookresearch/denoiser. The SE
frontend was pretrained with the CHiME–4 dataset with 200 epochs.

For the adapter, the input and output dimensions were all 512. During adapter
training, all parameters within the pretrained ASR backend remained frozen. We
positioned the adapters following the encoder layers. Moreover, both the SE
frontend and the adapters were updated simultaneously when utilizing the SE
frontend.

Table 1: The performance of baseline pretrained ASR model. Exp. | System | Development Sets | Evaluation Sets
---|---|---|---
bus | str | ped | caf | avg. | bus | str | ped | caf | avg.
0 | Pretrained | 22.0 | 15.1 | 11.2 | 16.6 | 16.3 | 30.3 | 16.1 | 23.4 | 26.5 | 24.1
Table 2: The effect of placing the adapter into different encoder layers (trained using the entire CHiME–4 training dataset). Exp. | Position of Adapter (Encoder) | Development Sets | Evaluation Sets
---|---|---|---
E1 | E2 | E3 | E4 | E5 | E6 | bus | str | ped | caf | avg. | bus | str | ped | caf | avg.
1 |  |  |  |  |  | ✓ | 17.1 | 10.3 | 8.5 | 11.1 | 11.8 | 25.9 | 12.1 | 17.8 | 20.4 | 19.1
2 |  |  |  |  | ✓ |  | 15.9 | 9.6 | 7.9 | 9.8 | 10.8 | 24.4 | 11.5 | 16.3 | 19.1 | 17.8
3 |  |  |  | ✓ |  |  | 14.3 | 8.8 | 7.2 | 9.2 | 9.9 | 23.1 | 10.6 | 15.0 | 17.1 | 16.5
4 |  |  | ✓ |  |  |  | 14.4 | 8.6 | 6.8 | 8.9 | 9.7 | 23.1 | 10.8 | 15.2 | 17.5 | 16.7
5 |  | ✓ |  |  |  |  | 13.5 | 8.0 | 6.6 | 8.2 | 9.1 | 21.6 | 9.8 | 13.9 | 16.2 | 15.4
6 | ✓ |  |  |  |  |  | 13.4 | 7.8 | 6.7 | 8.0 | 9.0 | 20.8 | 10.2 | 13.7 | 15.9 | 15.1
7 | ✓ | ✓ |  |  |  |  | 13.0 | 7.9 | 6.5 | 8.1 | 8.9 | 20.3 | 9.9 | 13.7 | 16.1 | 15.0
8 | ✓ | ✓ | ✓ |  |  |  | 13.1 | 7.8 | 6.5 | 8.2 | 8.9 | 20.4 | 10.1 | 13.8 | 16.1 | 15.1
9 | ✓ | ✓ | ✓ | ✓ |  |  | 12.9 | 7.9 | 6.8 | 7.9 | 8.9 | 20.5 | 9.6 | 13.3 | 15.6 | 14.7
10 | ✓ | ✓ | ✓ | ✓ | ✓ |  | 12.9 | 7.9 | 6.7 | 8.0 | 8.9 | 20.4 | 9.5 | 13.7 | 15.6 | 14.8
11 | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | 13.1 | 7.9 | 6.9 | 8.1 | 9.0 | 20.3 | 9.6 | 13.3 | 15.6 | 14.7
Table 3: The effect of different embedding dimensions in adapter (trained using the entire CHiME–4 training dataset). Exp. | Emb. Dim. | Development Sets | Evaluation Sets
---|---|---|---
bus | str | ped | caf | avg. | bus | str | ped | caf | avg.
12 | 16 | 13.1 | 7.9 | 6.8 | 8.1 | 9.0 | 20.6 | 9.8 | 13.4 | 15.6 | 14.9
13 | 32 | 13.2 | 7.8 | 6.8 | 8.1 | 9.0 | 20.8 | 9.9 | 13.4 | 15.9 | 15.0
11 | 64 | 13.1 | 7.9 | 6.9 | 8.1 | 9.0 | 20.3 | 9.6 | 13.3 | 15.6 | 14.7
14 | 96 | 13.3 | 8.1 | 6.9 | 7.7 | 9.0 | 20.6 | 9.6 | 13.5 | 15.8 | 14.9
15 | 128 | 12.9 | 7.9 | 6.9 | 8.0 | 8.9 | 20.6 | 9.8 | 13.6 | 15.6 | 14.9

###  4.2 Effect of the Position of Adapter

Table 2 shows the comparison of placing the adapters into different encoder
layers. Compared with the pretrained models in Table 1, inserting the adapter
into any encoder layer can bring considerable performance improvement to the
system. Through experiments 1–6, it is more effective to insert the adapter in
the shallow layer: the performance of the adaptation gradually improves as the
number of layers becomes shallower. Some studies [30] have revealed that the
shallower layers within the ASR model encompass signal-level information, such
as speech structure. In contrast, the deeper layers tend to hold abstract
information. Therefore, the shallow layer may have more noise-related
information, which is why the shallow layer is more effective for adaptation.

In addition, we also compared the effects of multi-layer adaptation through
experiments 7–11. Compared to solely adapting the first encoder layer,
incorporating further adaptations in deeper layers did not result in
substantial performance enhancements. Considering the abovementioned analysis,
leveraging more noise-related information, the self-adaptation at the shallow
layer has already achieved satisfactory effect. Attempting to enhance
information encoding in the deep layer by reducing noise-related information
is challenging, resulting in minimal performance improvements. However, the
best performance was achieved by inserting the adapter after all encoder
layers on the evaluation sets (experiment 11). Thus, in subsequent
experiments, we placed adapters after all encoder layers.

###  4.3 Effect of the Adapter Embedding Dimension

Table 3 shows the comparison of different embedding dimensions in the adapter.
We tried adapters with embedding dimensions of 16, 32, 64, 96, and 128
(experiments 11–15). Despite the wide range of embedding dimensions, the
results demonstrate consistent performance. This indicates a degree of
robustness in the adapter, as it appears unaffected by the embedding
dimension. Based on the results of these models on the evaluation sets, we
ultimately chose to utilize the 64-dimensional embedding adapter for the
subsequent experiments.

Table 4: The effect of different training sets for adapter-based adaptation. “Held” represents whether the specific noise conditions were excluded during model training; when utilizing the held-out approach, both the training and testing sets utilize a single noise type condition. “Real” represents whether the real noisy data is used during the training process. “Simu.” represents whether the simulated noisy data is used during training. “Utt.” represents how many utterances (channels 1 to 6 of the same utterance are considered single utterances) are used during the training process. $\clubsuit$ represents the number of all utterances in the corresponding noisy condition (this is due to the slightly different amounts of simulated data for the four noise conditions). $\bigstar$ represents that 100 sentences are selected from four noise conditions to constitute a training set.  Exp. | Training Data | Development Sets | Evaluation Sets
---|---|---|---
Held | Real | Utt. | Simu. | Utt. | bus | str | ped | caf | avg. | bus | str | ped | caf | avg.
11 | ✗ | ✓ | 1,600 | ✓ | 7,138 | 13.1 | 7.9 | 6.9 | 8.1 | 9.0 | 20.3 | 9.6 | 13.3 | 15.6 | 14.7
16 | ✗ | ✓ | 1,600 | ✗ | - | 13.7 | 8.2 | 7.1 | 8.6 | 9.4 | 22.6 | 9.8 | 14.1 | 16.7 | 15.8
17 | ✓ | ✓ | 400 | ✓ | $\clubsuit$ | 13.6 | 8.2 | 7.0 | 8.4 | 9.3 | 21.3 | 10.2 | 13.6 | 15.6 | 15.2
18 | ✓ | ✓ | 400 | ✗ | - | 14.5 | 8.8 | 7.2 | 9.1 | 9.9 | 23.2 | 10.3 | 14.8 | 17.8 | 16.5
19 | ✓ | ✗ | - | ✓ | 400 | 15.1 | 8.9 | 7.5 | 9.1 | 10.1 | 23.6 | 10.5 | 15.9 | 17.4 | 16.9
20 | ✓ | ✓ | 200 | ✓ | $\clubsuit$ | 13.7 | 8.1 | 7.0 | 8.4 | 9.3 | 21.4 | 10.1 | 13.9 | 15.9 | 15.3
21 | ✓ | ✓ | 200 | ✗ | - | 16.6 | 9.6 | 7.9 | 10.4 | 11.1 | 25.4 | 13.4 | 18.1 | 21.1 | 19.5
22 | ✓ | ✗ | - | ✓ | 200 | 18.1 | 10.0 | 8.9 | 11.7 | 12.2 | 26.4 | 13.6 | 20.2 | 22.0 | 20.5
23 | ✓ | ✓ | 100 | ✓ | $\clubsuit$ | 13.9 | 8.2 | 7.0 | 8.5 | 9.4 | 22.0 | 10.2 | 13.8 | 16.1 | 15.5
24 | ✓ | ✓ | 100 | ✗ | - | 18.4 | 11.2 | 8.8 | 11.7 | 12.5 | 27.2 | 14.7 | 20.3 | 23.1 | 21.3
25 | ✓ | ✗ | - | ✓ | 100 | 20.6 | 11.9 | 10.6 | 13.8 | 14.2 | 29.4 | 15.2 | 22.2 | 24.5 | 22.8
26 | ✗ | ✓ |  $\bigstar$400 | ✗ | - | 14.8 | 8.7 | 7.2 | 8.9 | 9.9 | 23.2 | 10.5 | 14.7 | 17.3 | 16.4
Table 5: The effect of adapter for different SE-based robust ASR systems (trained using the entire CHiME–4 training dataset). Exp. | System | Development Sets | Evaluation Sets
---|---|---|---
bus | str | ped | caf | avg. | bus | str | ped | caf | avg.
27 | Bi-LSTM | 13.3 | 8.1 | 6.8 | 8.4 | 9.2 | 21.4 | 10.1 | 13.5 | 16.1 | 15.3
28 | \+ adapter | 12.4 | 7.5 | 6.7 | 7.9 | 8.6 | 19.7 | 9.8 | 12.4 | 14.1 | 14.0
29 | DEMUCS | 11.7 | 7.7 | 6.3 | 6.9 | 8.2 | 19.2 | 9.2 | 12.8 | 14.9 | 14.0
30 | \+ adapter | 10.7 | 6.7 | 6.4 | 6.5 | 7.6 | 16.8 | 8.3 | 11.4 | 13.2 | 12.4

###  4.4 Effect of the Training Data

Table 4 shows the comparison of different training sets for adapter training.
When comparing experiments 11 and 16, it becomes evident that incorporating
simulated data during training leads to better performance for real noisy
sets: the relative improvement was 4% and 7% for development and evaluation
sets, respectively. This trend becomes more pronounced when the quantity of
real data diminishes: the relative improvements for experiments 17–18, 20–21,
23–24 were 6%, 16%, and 25% in development sets; and 8%, 22%, and 27% in
evaluation sets, respectively. Nevertheless, using simulated data might
constrain the model from achieving further improvements in performance with
real data. When comparing experiments 18–21 (21–24), it becomes evident that
the addition of more real data can lead to substantial performance
enhancements, resulting in relative improvements of 11% (11%) and 15% (9%) for
development and evaluation sets, respectively. On the other hand,
incorporating the same simulated data did not have any effect in the
comparative experiments 17, 20, and 23.

Real data yields better adaptation performance when using the same amount of
data than simulated data. This becomes particularly noticeable, especially
when dealing with smaller amounts of data: the relative improvements for
experiments 18–19, 21–22, 24–25 were 2% (development sets), 9% (development
sets), and 12% (development sets); and 2% (evaluation sets), 5% (evaluation
sets), and 7% (evaluation sets), respectively. This could be attributed to the
distinct distribution of simulated data compared to real data. Nevertheless,
with a substantial volume of simulated data, certain instances might exhibit a
distribution comparable to real data, consequently improving the model’s
performance on the real test sets.

Furthermore, we investigated how multi-condition training influences the
adapter’s effectiveness. The performance of experiments 18 and 26 were the
same. This could be due to shared noises (because each noise condition is
composed of multiple noises) among the four noise scenes in the CHiME–4
dataset. As a result, the noise is only partially unseen, thus partially
limiting the potential effects of multi-condition training. It also serves as
an inspiration that incorporating similar noisy real data as augmented data
can result in substantial performance improvements (compare experiments 24 and
26).

###  4.5 Effect of the Adapter for SE-based robust ASR

Table 5 shows the adapter’s performance for different SE-based robust ASR
systems. Experiments 27 and 29 significantly improved the performance from the
pretrained model (experiment 0) when the SE front-end was used. Incorporating
adapters within the SE-based robust ASR system further improved recognition
performance. While feature enhancement of the SE front-end has been achieved,
significant benefits still arise from adaptation at the backend. This is
because the SE frontend might introduce information loss or distortion and the
adapter plays a role in mitigating these issues.

##  5 Conclusions and Future Works

In this paper, we explored the effect of the adapter on noise-robust ASR. We
conducted a comprehensive exploration from various perspectives, including the
optimal insertion position for the adapter, the quantity and type of data used
for adapter training, and the synergy of the adapter with the SE. We conducted
experiments using the CHiME–4 dataset. The experimental results demonstrate
that incorporating adapters in the shallow layer yields more effectness
compared to the deep layer. Furthermore, the number of embedding nodes in the
adapter does not significantly impact the adaptation process. Moreover, the
training dataset plays a vital role in adapter training: When considering the
same data amount, using real data shows more effective than simulated data but
adding simulated data can enhance the performance of the real test sets.
Combining adapters with the SE frontend leads to further performance
improvement. In the future, we will study on designing the adapter structure
better suited for robust ASR.

## References

  * [1] J. Li, L. Deng, Y. Gong, and R. Haeb-Umbach,  “An Overview of Noise-Robust Automatic Speech Recognition,”  IEEE/ACM TASLP, vol. 22, no. 4, pp. 745–777, 2014.
  * [2] L. Dong, S. Xu, and B. Xu,  “Speech-Transformer: A No-Recurrence Sequence-to-Sequence Model for Speech Recognition,”  in Proc. ICASSP, 2018, pp. 5884–5888.
  * [3] F. Weninger, H. Erdogan, S. Watanabe, E. Vincent, J. Le Roux, J. Hershey, and B. Schuller,  “Speech Enhancement with LSTM Recurrent Neural Networks and its Application to Noise-Robust ASR,”  in Proc. LVA/ICA, 2015, pp. 91–99.
  * [4] A. Baevski, Y. Zhou, A. Mohamed, and M. Auli,  “wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations,”  in Proc. NIPS, 2020, vol. 33, pp. 12449–12460.
  * [5] W. Hsu, B. Bolte, Y. Tsai, K. Lakhotia, R. Salakhutdinov, and A. Mohamed,  “HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units,”  IEEE/ACM TASLP, vol. 29, pp. 3451–3460, 2021.
  * [6] R. Fan, Y. Zhu, J. Wang, and A. Alwan,  “Towards Better Domain Adaptation for Self-Supervised Models: A Case Study of Child ASR,”  IEEE Journal of Selected Topics in Signal Processing, vol. 16, no. 6, pp. 1242–1252, 2022.
  * [7] X. Cui, V. Goel, and B. Kingsbury,  “Data Augmentation for Deep Neural Network Acoustic Modeling,”  IEEE/ACM TASLP, vol. 23, no. 9, pp. 1469–1477, 2015.
  * [8] H. Shi, L. Wang, S. Li, C. Fan, J. Dang, and T. Kawahara,  “Spectrograms Fusion-based End-to-end Robust Automatic Speech Recognition,”  in Proc. APSIPA ASC, 2021, pp. 438–442.
  * [9] S. Wang, W. Li, S. Siniscalchi, and C. Lee,  “A Cross-Task Transfer Learning Approach to Adapting Deep Speech Enhancement Models to Unseen Background Noise Using Paired Senone Classifiers,”  in Proc. ICASSP, 2020, pp. 6219–6223.
  * [10] S. Sun, C. Yeh, M. Hwang, M. Ostendorf, and L. Xie,  “Domain Adversarial Training for Accented Speech Recognition,”  in Proc. ICASSP, 2018, pp. 4854–4858.
  * [11] Y. Hu, C. Chen, R. Li, Q. Zhu, and E. Chng,  “Gradient Remedy for Multi-Task Learning in End-to-End Noise-Robust Speech Recognition,”  in Proc. ICASSP, 2023, pp. 1–5.
  * [12] B. Thomas, S. Kessler, and S. Karout,  “Efficient Adapter Transfer of Self-Supervised Speech Models for Automatic Speech Recognition,”  in Proc. ICASSP, 2022, pp. 7102–7106.
  * [13] Y. Qian, X. Gong, and H. Huang,  “Layer-Wise Fast Adaptation for End-to-End Multi-Accent Speech Recognition,”  IEEE/ACM TASLP, vol. 30, pp. 2842–2853, 2022.
  * [14] Z. Yu, Y. Zhang, K. Qian, C. Wan, Y. Fu, Y. Zhang, and Y. C. Lin,  “Master-ASR: Achieving Multilingual Scalability and Low-Resource Adaptation in ASR with Modular Learning,”  in Proc. Machine Learning Research, 2023, vol. 202, pp. 40475–40487.
  * [15] C. H. H. Yang, B. Li, Y. Zhang, N. Chen, R. Prabhavalkar, T. N. Sainath, and T. Strohman,  “From English to More Languages: Parameter-Efficient Model Reprogramming for Cross-Lingual Speech Recognition,”  in Proc. ICASSP, 2023, pp. 1–5.
  * [16] K. M. Sathyendra, T. Muniyappa, F. J. Chang, J. Liu, J. Su, G. P. Strimel, A. Mouchtaris, and S. Kunzmann,  “Contextual Adapters for Personalized Speech Recognition in Neural Transducers,”  in Proc. ICASSP, 2022, pp. 8537–8541.
  * [17] T. Munkhdalai, Z. Wu, G. Pundak, K. C. Sim, J. Li, P. Rondon, and T. N. Sainath,  “NAM+: Towards Scalable End-to-End Contextual Biasing for Adaptive ASR,”  in Proc. SLT, 2023, pp. 190–196.
  * [18] J. Huang, K. Ganesan, S. Maiti, Y. Min Kim, X. Chang, P. Liang, and S. Watanabe,  “FindAdaptNet: Find and Insert Adapters by Learned Layer Importance,”  in Proc. ICASSP, 2023, pp. 1–5.
  * [19] A. Gulati, J. Qin, C. C. Chiu, N. Parmar, Y. Zhang, J. Yu, W. Han, S. Wang, Z. Zhang, Y. Wu, and R. Pang,  “Conformer: Convolution-augmented Transformer for Speech Recognition,”  in Proc. Interspeech, 2020, pp. 5036–5040.
  * [20] O. Vinyals, S. V. Ravuri, and D. Povey,  “Revisiting Recurrent Neural Networks for robust ASR,”  in Proc. ICASSP, 2012, pp. 4085–4088.
  * [21] Z.-Q. Wang, P. Wang, and D. Wang,  “Complex Spectral Mapping for Single- and Multi-Channel Speech Enhancement and Robust ASR,”  IEEE/ACM TASLP, vol. 28, pp. 1778–1787, 2020.
  * [22] S. Araki, T. Hayashi, M. Delcroix, M. Fujimoto, K. Takeda, and T. Nakatani,  “Exploring multi-channel features for denoising-autoencoder-based speech enhancement,”  in Proc. ICASSP, 2015, pp. 116–120.
  * [23] C. Li, J. Shi, W. Zhang, A. S. Subramanian, X. Chang, N. Kamo, M. Hira, T. Hayashi, C. Boeddeker, Z. Chen, and S. Watanabe,  “ESPnet-SE: End-To-End Speech Enhancement and Separation Toolkit Designed for ASR Integration,”  in Proc. SLT, 2021, pp. 785–792.
  * [24] S.-A. Rebuffi, H. Bilen, and A. Vedaldi,  “Learning multiple visual domains with residual adapters,”  in Advances in Neural Information Processing Systems, 2017, vol. 30.
  * [25] E. J. Hu, Y. Shen, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang, and W. Chen,  “LoRA: Low-Rank Adaptation of Large Language Models,”  arXiv, 2021.
  * [26] V. Panayotov, G. Chen, D. Povey, and S. Khudanpur,  “Librispeech: An ASR corpus based on public domain audio books,”  in Proc. ICASSP, 2015, pp. 5206–5210.
  * [27] D. Snyder, G. Chen, and D. Povey,  “MUSAN: A Music, Speech, and Noise Corpus,” 2015.
  * [28] D. S. Park, W. Chan, Y. Zhang, C. Chiu, B. Zoph, E. D. Cubuk, and Q. V. Le,  “SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognition,”  in Proc. Interspeech, 2019, pp. 2613–2617.
  * [29] A. Défossez, G. Synnaeve, and Y. Adi,  “Real Time Speech Enhancement in the Waveform Domain,”  in Proc. Interspeech, 2020, pp. 3291–3295.
  * [30] C.-Y. Li, P.-C. Yuan, and H.-Y. Lee,  “What does a network layer hear? analyzing hidden representations of end-to-end asr through speech synthesis,”  in Proc. ICASSP, 2020, pp. 6434–6438.