# Leptogenesis Effects on the Gravitational Waves Background: Interpreting the
NANOGrav Measurements and JWST Constraints on Primordial Black Holes

K. El Bourakadi1<EMAIL_ADDRESS>H. Chakir1<EMAIL_ADDRESS>M. Yu.
Khlopov2,3,4<EMAIL_ADDRESS>1Subatomic Research and Applications Team,
Faculty of Science Ben M’sik,
Casablanca Hassan II University, Morocco
2 Institute of Physics, Southern Federal University, 194 Stachki, Rostov-on-
Donu, Russia
3 Virtual Institute of Astroparticle Physics, 75018 Paris, France
4 National Research Nuclear University MEPhI, 115409 Moscow, Russia

###### Abstract

We demonstrate that the leptogenesis mechanisms, which are associated with B-L
symmetry breaking mechanism has notable effects on the production of
gravitational waves. These gravitational waves align well with the recent
observations of a stochastic gravitational wave background by NANOGrav and
pulsar-timing arrays (PTAs). For these gravitational waves to match the recent
measurements, the critical value of the B-L breaking should be around the GUT
scale. Moreover, we consider the generation of primordial gravitational waves
from binary systems of Primordial Black Holes (PBHs) which could be predicted
by the recent detection of gravitational waves. PBHs with specific masses can
be responsible for massive galaxy formation observed at high redshifts
reported by the James Webb Space Telescope (JWST). We contemplate the
potential for a shared source between the NANOGrav and JWST observations,
namely primordial black holes. These black holes could serve as seeds of rapid
galaxy formation, offering an explanation for the galaxies observed by JWST.

##  I Introduction

Gravitational Waves Background (GWB) provides a captivating glimpse into the
early stages of the universe [1]. These waves emerge either from quantum
fluctuations occurring during the inflationary period [2], sourced by the
phase transition of the Universe, results from cosmic strings [3, 4], or
Binary Black Holes (BBH) from the early Universe [5].

During the B-L breaking phase transition ending hybrid inflation, most of the
vacuum energy density gets swiftly transferred to B-L Higgs bosons in a non-
relativistic state and a significant portion also goes into the formation of
cosmic strings. Entropy and baryon asymmetry are produced through thermal and
nonthermal leptogenesis as a result of the decay processes involving massive
Higgs bosons and Majorana neutrinos [6, 7].

The operators responsible for violating the lepton (or B-L) number, emerge as
a result of the exchange of massive Majorana neutrinos $N_{i}$ (where
$i=1,2,3$) [8, 9]. When these heavy neutrinos $N_{i}$ undergo decay, they
inherently generate lepton (or B-L) asymmetry, assuming that the principles of
C and CP conservation do not hold [8]. The resulting lepton asymmetry is then
converted into baryon asymmetry during the early stages of the universe [10].
As a consequence, the process of leptogenesis appears to be the most intrinsic
mechanism for explaining the observed baryon asymmetry in the current
universe.

Up to this point, a range of scenarios for leptogenesis have been put forth,
contingent on the methods of producing the heavy Majorana neutrinos $N_{i}$
[11, 12, 13, 14]. Of these scenarios, the most widely recognized is the
conventional approach involving the thermal generation of $N_{i}$ during the
early phases of the universe. A comprehensive analysis [11] reveals that a
satisfactory lepton asymmetry, capable of accounting for the existing baryon
asymmetry can be achieved when the reheating temperature $T_{re}$ following
inflation is approximately in the range of ${\mathcal{O}}(10^{10})$ $GeV$.
However, delving into leptogenesis within the context of supergravity
introduces a cosmological issue concerning gravitinos: the production of an
excessive number of gravitinos occurs with a $T_{re}$ around $10^{10}$ $GeV$,
to maintain the viability of big-bang nucleosynthesis (BBN) [15, 16]. To
tackle this predicament, one approach is to postulate that the gravitino
serves as the lightest supersymmetry (SUSY) particle with a mass denoted as
$m_{3/2}$, which falls within the range of around $\left(10-100\right)$ $GeV$
[17, 18, 19, 20].

In this work, we calculate the gravitational wave spectrum projected by the
cosmological breaking of B-L symmetry. This spectrum is influenced by various
sources as enumerated earlier, including the inflationary period. Our primary
focus lies in the distinctive traits of the GW spectrum attributed to binary
systems of black holes originating in the early universe. Additionally, we aim
to establish a relationship within the GW spectrum to the values at the phase
transition of the hybrid inflation which can help identify the model
parameters linked to specific characteristics in the spectrum.

Confirming the inflationary paradigm is strongly associated with the
identification of relic gravitational waves (GWs), which are considered a
clear and unambiguous signal. In recent studies, there has been significant
progress in observational cosmology. Various collaborations, such as NANOGrav
[65, 66], European PTA (EPTA)/Indian PTA (InPTA) [21, 22], and Parkes PTA
(PPTA) [23], have presented compelling evidence supporting the existence of a
Stochastic Gravitational Wave Background (SGWB) in the frequency range of
nanohertz (nHz). While these findings are primarily attributed to
astrophysical sources, it remains crucial to investigate their potential
cosmological origins.

The origin of supermassive black holes (SMBHs) remains uncertain, yet numerous
hypotheses have been put forth. The correlation noted earlier between the
masses of black holes and the dispersion of stellar velocities can be
interpreted in diverse manners, ultimately leading to the question of which
came first: the host galaxy or the SMBHs [24, 25]. Thanks to the impressive
capabilities of the James Webb Space Telescope (JWST), we now have the
opportunity to gain a more profound understanding of the conclusion of the
cosmic dark ages. To date, the JWST has successfully pinpointed numerous
promising galaxy candidates situated at remarkably high redshifts [26, 27,
28]. Interestingly, within the JWST Cosmic Evolution Early Release Science
(CEERS) program, a collection of massive galaxy candidates situated in the
redshift range of $6.5\lesssim z\lesssim 9.1$ has been detected. These
candidates exhibit estimated stellar masses $M_{\ast}\gtrsim 109M_{\odot}$
(where $M_{\odot}=1.99\times 10^{30}$ $kg$ represents the solar mass).
Notably, among these candidates, there exist six sources possessing stellar
masses $M_{\ast}\gtrsim 10^{10}M_{\odot}$ within the redshift range of
$7.4\lesssim z\lesssim 9.1$ [29]. The investigation of primordial black holes
has its roots in research conducted over five decades ago [30, 31], and in
recent years, it has garnered growing attention. There are diverse underlying
reasons for this renewed interest. One example is the possibility that the
merging of binary PBHs could produce gravitational waves that are detectable
by collaborations like LIGO/Virgo/KAGRA [32, 33]. Primordial black holes
represent a compelling and inherent contender for the role of dark matter (DM)
within specific mass intervals [34]. In broad terms, there exist two primary
mechanisms through which PBHs can influence the formation of cosmic
structures: the Poisson effect [35] and the seed effect [36, 37]. When PBHs
constitute a substantial proportion of the overall dark matter content, the
Poisson effect becomes prevalent across all scales. Conversely, if PBHs
account for only a minor fraction of the dark matter, the seed effect
dominates on smaller scales.

This article is organized as follows, in Sec. II we discuss the relation of
Leptogenesis to B-L symmetry breaking mechanism, in Sec. III we introduce the
physics of gravitational waves spectrum and the relation to the PTA signals in
light of the B-L breaking, in Sec. IV we Match the PTA signals with
gravitational waves emitted by BBH Mergers and PBH mass predicted by JWST, the
last section is devoted to a conclusion.

##  II  Leptogenesis from B-L symmetry breaking

To illustrate the dynamics of Leptogenesis, we need to examine the B-L
symmetry breaking [38, 39], the presence of baryon asymmetry in the current
universe suggests that B-L violating processes had a significant impact on
creating the baryon asymmetry during the early universe. The lepton number
operators violating B-L symmetry originate from the interaction involving
massive Majorana neutrinos $N_{i}$ $\left(i=1,2,3\right)\ $[40, 41], when $C$
and $CP$ are not conserved, the decays of these massive neutrinos $N_{i}$
naturally give rise to B-L asymmetry. The scalar potential governing the
inflaton field $\varphi$ is is given by

$V\left(\varphi\right)=V_{0}+V_{1l}\left(\varphi/\varphi_{c}\right),$ (1)

with $V_{0}=\frac{\lambda}{4}\upsilon_{B-L}^{4}$ and $\upsilon_{B-L}\
=\varphi_{c}$ is the critical value of the hybrid inflation potential, for
$\varphi\gg\varphi_{c}$

$V_{1l}\left(\varphi/\varphi_{c}\right)\simeq\frac{\lambda}{32\pi^{2}}\upsilon_{B-L}^{4}.$
(2)

When $\varphi$ falls below $\varphi_{c}$, the B-L Higgs boson undergoes a
tachyonic instability, meaning it acquires a negative mass squared. This
sudden occurrence leads to the inflation coming to an end and the spontaneous
breaking of B-L. The slow-roll parameters $\epsilon$ and $\eta$ associated
with hybrid inflation are given as follows

$\displaystyle\epsilon$ $\displaystyle\approx$
$\displaystyle\frac{\lambda}{16\pi^{2}}\left|\eta\right|,$ (3)
$\displaystyle\eta$ $\displaystyle\approx$ $\displaystyle-\frac{\lambda
M_{p}}{32\pi^{3}\varphi_{k}^{2}}\approx-\frac{1}{2N_{k}}$ (4)

Figure 1: $r$ as a function of $n_{s}$ for different values of $\lambda$

Considering our current study, Fig. 1 shows that the tensor-to-scalar ratio,
$r$, shows good consistency with the predicted values of $r$ and decreases
slowly with respect to the spectral index, $n_{s}$. The results show good
consistency for a wide range of model parameters with the latest observations
from Planck data. Moreover, the parameter $\lambda$ provide good consistency
for values bounded as $\lambda\leq 0.5$ following the observational data
constraints on $r$ and $n_{s}$.

The scalar perturbations can be characterized by the curvature power spectrum,
which describes the specific mode with a wave number k. When applying the
slow-roll approximation, one obtains an analytical expression at the point of
horizon crossing.

${\mathcal{P}}_{s}(k)\equiv\left.\frac{k^{3}}{2\pi^{2}}\left|{\mathcal{R}}_{k}\right|^{2}\right|_{k\ll
aH}\simeq\left.\frac{1}{8\pi^{2}\epsilon}\frac{H^{2}}{M_{p}^{2}}\right|_{k=aH},$
(5)

here $3M_{p}^{2}H^{2}=V_{0}\ $and ${\mathcal{R}}_{k}$ represents the Fourier
component of the comoving curvature perturbation. Similarly, the tensor
perturbations $h_{\mathbf{k}}^{\lambda}$ can be quantified using the tensor
power spectrum. By employing the slow-roll approximation, it becomes possible
to derive an analytical expression for the tensor power spectrum as follows

${\mathcal{P}}_{t}(k)\equiv\left.\frac{k^{3}}{2\pi^{2}}\left|h_{\mathbf{k}}^{\lambda}\right|^{2}\right|_{k\ll
aH}\simeq\left.\frac{2}{\pi^{2}}\frac{H^{2}}{M_{p}^{2}}\right|_{k=aH}.$ (6)

For both previous quantities of the power spectrum, the value of
${\mathcal{P}}_{s}\equiv\Delta_{s}^{2}$ is measured by the PLANCK satellite,
$\Delta_{s}^{2}\simeq 2.101\times 10^{-9}$ [42], while for the tensor power
spectrum ${\mathcal{P}}_{t}\equiv\Delta_{t}^{2}=$ $r\Delta_{s}^{2}$ [43]. One
should note that, slow-roll approximated solution for the scalar power spectra
enhanced scalar fluctuations, which means that small-scale fluctuations may
become large enough for PBH to be overproduced. On the other hand, the tensor
power spectra can act as a source for gravitational wave background.

Figure 2: Constraints on $\upsilon_{B-L}$ according the recent Planck’s bound
on $n_{s}.$

Fig. 2 discusses the observational constraints on the critical value that
leads to the B-L symmetry breaking from Planck’s data, the results shows that
$\upsilon_{B-L}\propto 10^{15}GeV$ and must be bounded around
$\sim\left[2\times 10^{15},5\times 10^{15}\right]GeV,$ our findings are also
consistent with with previous results [44, 45] where they proposed that the
aim behind $\upsilon_{B-L}$ value is to adjust its value to be in proximity to
the GUT scale. Specifically, after conducting a thorough analysis of hybrid
inflation, considering the generation of cosmic strings and non-canonical
effects in the Kahler potential, the established values of $\upsilon_{B-L}$
are consistent with all pertinent observations.

The existence of multiple heavy Majorana neutrino species ($N_{i}$) leads to
two mechanisms that reduce the current lepton asymmetry. In previous findings
(Ref. [45]) they considered a normal hierarchy of the right-handed neutrinos,
where the most significant contribution to these processes come from the
lightest right-handed neutrino ($N_{1}$), which leads to the fact that the
time evolution of the reheating process is primarily governed by two key
factors: $\Gamma_{S}^{0}$ and $\Gamma_{N_{1}}^{0}$. These quantities represent
the vacuum decay rate of the B-L Higgs bosons and their superpartners, as well
as the vacuum decay rate of the heavy (s)neutrinos belonging to the first-
generation [45].

$\displaystyle\Gamma_{S}^{0}$ $\displaystyle=$
$\displaystyle\frac{m_{S}}{32}\left(\frac{M_{1}}{\upsilon_{B-L}}\right)^{2}\left[1-\left(\frac{2M_{1}}{m_{S}}\right)^{2}\right]^{1/2}$
(7) $\displaystyle\Gamma_{N_{1}}^{0}$ $\displaystyle=$
$\displaystyle\frac{\tilde{m}_{1}}{4\pi}\left(\frac{M_{1}}{\upsilon_{u}}\right)^{2}$
(8)

In the Froggatt-Nielsen flavor model, which serves as the basis for our
previous study in Ref. [46], the masses of Higgs and (s)neutrinos, denoted as
$m_{S}$, $M_{1}$, and $\tilde{m}_{1}$, are expected to take specific values:
$m_{S}=1.6\times 10^{13}GeV$, $M_{1}=5.4\times 10^{10}GeV$ and
$\tilde{m}_{1}=4.0\times 10^{-2}eV$. The number of free and independent
parameters in this model, represented by the two neutrino masses M1 and m1,
correspond exactly to the two decay rates $\Gamma_{S}^{0}$ and
$\Gamma_{N_{1}}^{0}$. The study was conducted in Refs. [39, 45, 46] aimed to
gain a detailed and time-resolved understanding of the reheating process. A
remarkable finding from this research is the presence of an approximate
plateau in the radiation temperature during reheating after the B-L phase
transition. This plateau occurs around the time when the heavy (s)neutrinos
decay. The constancy of the temperature over an extended period is a direct
result of a temporary balance between entropy production and cosmic expansion.
The temperature at which this plateau occurs serves as a characteristic
temperature scale for both leptogenesis and the thermal production of
gravitinos. The next section will be devoted to studying the impact of B-L
symmetry breaking on the gravitational wave background.

##  III  Gravitational Waves Background

###  III.1 Primordial Gravitational Waves

Detecting the background of primordial gravitational waves would serve as
compelling evidence to support the inflation paradigm, shedding light on the
fundamental physics of the early Universe. In this direction, recent findings
have been dedicated to studying the Primordial Gravitational Waves (PGW) at
the first stages following inflation [47, 48, 49, 50, 51, 52, 53, 54]. In the
case of the inflationary scenario, the occurrence of a kinetic epoch preceding
inflation results in a distinctive blue tilt in the primordial gravitational
wave spectra at higher frequency ranges [55, 56, 57, 58]. Gravitational waves
are characterized as the transverse-trace-less component of the metric
perturbation. In linear order perturbation theory, there is no coupling
between scalar, vector, and tensor modes. Thus, for primary gravitational
waves in a spatially flat FLRW background, we can express the metric element
as follows:

$ds^{2}=-dt^{2}+a^{2}(t)(\delta_{\mu\nu}+h_{\mu\nu})dx^{\mu}dx^{\nu},$ (9)

given that the tensor mode fulfills
$\left(h_{\mu\mu}=h_{00}=\partial^{\mu}h_{\mu\mu}=0\right).$
$h_{\mu\nu}\left(t,x\right)$ can be decomposed into its Fourier mode, each
associated with two polarization tensors which satisfy the equation of motion

$h_{\mathbf{k}}^{\lambda\prime\prime}(\eta)+2{\mathcal{H}}h_{\mathbf{k}}^{\lambda\prime}(\eta)+k^{2}h_{\mathbf{k}}^{\lambda}(\eta)=0,$
(10)

where $\left({}^{\prime}\right)$ represents the derivative with respect to
conformal time $\eta,$ $d\eta=$ $\frac{dt}{a}$, and
${\mathcal{H}}=\frac{a^{\prime}}{a}.$ The normalized gravitational wave energy
density spectrum is defined as the energy density per logarithmic frequency
interval.

$\Omega_{gw}(k)=\frac{1}{\rho_{c}}\frac{d\rho_{gw}}{d\ln k},$ (11)

where $\rho_{c}$ is the total energy density. Moreover,

$\Omega_{gw,0}(k)=\frac{1}{12}\left(\frac{k^{2}}{a_{0}^{2}H_{0}^{2}}\right){\mathcal{P}}_{h}(k),$
(12)

with
${\mathcal{P}}_{h}(k)\equiv\frac{k^{3}}{\pi^{2}}\sum_{\lambda}\left|h_{\mathbf{k}}^{\lambda}\right|^{2}.$
By considering the the scale at horizon re-entry $\left(k=a_{hc}H_{hc}\right)$
and the Hubble parameter at different epochs, we can obtain the primary
gravitational wave spectrum at the present time for mode re-entry during the
matter $\left(M\right)$, radiation $\left(R\right)$, and kinetic
$\left(K\right)$ eras, respectively as follows:

$\displaystyle\Omega_{gw,0}^{\left(M\right)}$ $\displaystyle=$
$\displaystyle\frac{1}{24}\Omega_{m,0}^{2}\frac{a_{0}^{2}H_{0}^{2}}{k^{2}}{\mathcal{P}}_{t}~{}~{}~{}~{}~{}~{}\left(k_{0}<k\leq
k_{eq}\right),$ (13) $\displaystyle\Omega_{gw,0}^{\left(R\right)}$
$\displaystyle=$
$\displaystyle\frac{1}{24}\Omega_{r,0}^{2}\left(\frac{g_{\ast}}{g_{\ast
0}}\right)\left(\frac{g_{\ast s}}{g_{\ast
s0}}\right){\mathcal{P}}_{t}~{}~{}~{}~{}~{}~{}\left(k_{eq}<k\leq
k_{r}\right),$ (14) $\displaystyle\Omega_{gw,0}^{\left(K\right)}$
$\displaystyle=$
$\displaystyle\Omega_{gw,0}^{\left(R\right)}\left(\frac{k}{k_{r}}\right)~{}~{}~{}~{}~{}\left(k_{r}<k\leq
k_{\max}\right),$ (15)

Figure 3:  The evolution of the B-L asymmetry parameter considering the
variation of the scale $k$ and the spectral index $n_{s}$.

Before we go through more details on the gravitational wave density
Considering the effect of the critical value $\upsilon_{B-L}$ and the scale
$k.$ In Fig. 3 we study the evolution of the B-L symmetry breaking parameter
$\upsilon_{B-L}$ as a function of the wavelength $k$ considering the index
spectral observational bound effects on $\upsilon_{B-L}.$ The results are
found considering the form of the tensor power spectrum from the previous
section and the fact that ${\mathcal{P}}_{t}\propto k^{n_{s}-1}$, we conclude
that considering superhorizon scales the potential parameter will decrease as
we take higher values of inflation e-folds number. Additionally, the spectral
index shows that for both smaller and larger scales the $\upsilon_{B-L}$ is
best considered around $2\times 10^{15}GeV\leq\upsilon_{B-L}<3\times
10^{15}GeV.$

Figure 4:  Primordial gravitational wave evolution with respect to the scale
$k$ and $\upsilon_{B-L}$.

Fig. 4 represents the variation of the density of primordial gravitational
waves with respect to the scale $k$ and the $\upsilon_{B-L}$ parameter. The
color scale on the right of the plots represents the logarithm of the density
of PGWs. Regions with higher color intensity (warmer colors) indicate higher
density, while regions with lower color intensity (cooler colors) indicate
lower density. The change in the critical value $\upsilon_{B-L}$ that
corresponds to B-L asymmetry has proportional effects on $\Omega_{gw,0}$ for
all matter, radiation, and kinetic eras. In fact, as we go higher in
$\upsilon_{B-L}$ values specifically around $\upsilon_{B-L}\propto 10^{17}GeV$
we notice a significant transition in the density of PGWs. Moreover, the
transition of $\Omega_{gw,0}$ corresponds to larger scales (e.g., superhorizon
scales) given by $k\propto 10^{6}$ which are noticed only for the matter and
kinetic eras. The peak region in the heat map, where the color intensity is
highest, indicates the preferred combination of $\upsilon_{B-L}$ and $k$ that
leads to the highest density of primordial gravitational waves. This
combination represents the characteristic scale of the gravitational waves
generated during inflation. To conclude, we can see that the proposed scenario
of Leptogenesis has great implications for the observed values of PGWs density
which can be predicted by PTA signals if made fine tuning to the values of the
scale $k$ and $\upsilon_{B-L}$ for the matter, radiation, and kinetic eras.
Next, we will see how much can the NanogGrav predictions reproduce the
theoretical values predicted by the chosen model of the density of PGWs.

###  III.2 Matching Gravitational Waves to the PTA Signals

The time evolution of a gravitational wave field, represented by
$h_{\mathbf{k}}(\eta_{i})$ at an initial conformal time $\eta_{i}$, and
characterized by its tensor spectrum, can be obtained by determining the GW
transfer function
${\mathcal{T}}\mathbf{(\eta,k)=}h_{\mathbf{k}}(\eta)/h_{\mathbf{k}}(\eta_{i})$
[59]. Here, $h_{\mathbf{k}}(\eta)$ is evaluated at a conformal time
$\eta\gg\eta_{i}$ [60, 61]. The relevant quantity for GW direct detection
experiments, such as Pulsar Timing Arrays, are the spectral energy density
parameter $\Omega_{gw}(k)$ [62]:

$\Omega_{gw}(k)\equiv\frac{k^{2}}{12H_{0}}{\mathcal{T}}\mathbf{(\eta}_{0}\mathbf{,k)}{\mathcal{P}}_{t}(k),$
(16)

using $\mathbf{\eta}_{0}$ to represent the present conformal time and $H_{0}$
to denote the Hubble constant [63]. For the purpose of this investigation, we
focus on $\Omega_{gw}(k)$ at PTA scales, where $f\sim{\mathcal{O}}(10-9)Hz$.
The corresponding wavenumbers $k\sim{\mathcal{O}}(10^{6})Mpc^{-1}$ are much
larger than the wavenumber associated with a mode crossing the horizon at
matter-radiation equality. In other words, the modes observed on PTA scales
crossed the horizon deep in the radiation era, well before the matter-
radiation equality. In the regime where $k\gg k_{eq}$, the GW spectral energy
density associated with PTA signals can be expressed as [64, 65, 66, 67, 68,
69]:

$\Omega_{gw}(f)=\frac{2\pi^{2}}{3H_{0}^{2}}A^{2}\frac{f^{5-\gamma}}{f_{yr}^{\gamma-3}}.$
(17)

The anticipated signal of the Stochastic Gravitational Wave Background (SGWB)
generated by merging Black Hole Binaries (BHBs) mergers corresponds to
$\gamma=13/3$ and $\alpha\simeq 5-\gamma.$ The relationship linking the
amplitude of the Pulsar Timing Array (PTA) signal, denoted as $A$, with the
cosmological parameters were found to be [62, 70]

$A=\sqrt{\frac{45\Omega_{m}^{2}A_{s}}{32\pi^{2}(\eta_{0}k_{eq})^{2}}}c\frac{f_{yr}}{\eta_{0}}\left(\frac{f_{yr}^{-1}}{f_{\star}}\right)^{\frac{n_{T}}{2}}\sqrt{r}.$
(18)

The dependence on $n_{T}$ in this context arises from the substantial ”lever
arm” between the Cosmic Microwave Background (CMB) pivot frequency, where As
is constrained, and the frequency of the Pulsar Timing Array (PTA) signal
$\left[1yr^{-1}\right]$.

$\lambda$ | $v_{B-L}(GeV)$ | $r$ | $A$ | $f(Hz)$ | $h^{2}\Omega_{gw}(f)$
---|---|---|---|---|---
$1\times 10^{-5}$ | $2\times 10^{15}$ | $0.0174$ | $6.02\times 10^{-15}$ | $4\times 10^{-8}$ | $5.06\times 10^{-8}$
$1.5\times 10^{-5}$ | $2\times 10^{15}$ | $0.0261$ | $7.37\times 10^{-15}$ | $2\times 10^{-8}$ | $5.36\times 10^{-8}$
$2\times 10^{-5}$ | $2.1\times 10^{15}$ | $0.0423$ | $9.38\times 10^{-15}$ | $1\times 10^{-8}$ | $6.14\times 10^{-8}$
$2\times 10^{-5}$ | $2.2\times 10^{15}$ | $0.0510$ | $1.03\times 10^{-14}$ | $8\times 10^{-7}$ | $6.62\times 10^{-7}$
$2.5\times 10^{-5}$ | $2.3\times 10^{15}$ | $0.0762$ | $1.26\times 10^{-14}$ | $6\times 10^{-7}$ | $8.59\times 10^{-7}$
$3\times 10^{-5}$ | $2.4\times 10^{15}$ | $0.1084$ | $1.50\times 10^{-14}$ | $5\times 10^{-7}$ | $1.11\times 10^{-7}$
Table 1: Testing the density of gravitational waves predicted by PTA as
functions of several parameters related to the potential considered for
describing the Leptogenesis mechanism, here we’ve inserted the best-fit values
for the cosmological parameters as per the Planck results [42],
$\Omega_{m}=0.315,$ $\eta_{0}\simeq 1.38,$ $A_{s}=2,1\times
10^{-9},k_{eq}\simeq 0.01,f_{\star}\approx 7.7\times 10^{-17}Hz,$
$f_{yr}\simeq 3.1\times 10^{-8}Hz$ and $H_{0}\equiv 100~{}h~{}km/s/Mpc.$

Table 1 corresponds to the NanoGrav proposed model of the density
$h^{2}\Omega_{gw}(f)$ as a function of the frequency $f,$ scalar-to-tensor
ratio $r$, the amplitude of PTA signal that is related to the proposed
potential parameters that explain the Leptogenesis phenomena. The model for
$\Omega_{gw}(f)$ depends on various cosmological parameters and other
constants defined previously. We compute $h^{2}\Omega_{gw}(f)$ is an
analytical expression based on the theoretical model for primordial
gravitational waves proposed recently, $h^{2}\Omega_{gw}(f)$ is the the
fractional energy density of gravitational waves in the universe. It
represents the amount of energy carried by gravitational wave background
relative to the critical energy density required to achieve a flat universe.
By looking at the table identify the right parameters that can predict the
higher or lower density of gravitational waves $h^{2}\Omega_{gw}(f)$ which
reproduce the NANOGrav predictions, one can clearly see that for values of
$r\lesssim 0.06$, with the spectrum is blue $\left(n_{T}>0\right)$ which we
consider to take $n_{T}=1.1$, the amplitude of PTA scales will be $A\lesssim
10^{-20}$ which will provide good predictions for $h^{2}\Omega_{gw}(f)$
values. Moreover, the table provides insights into the behavior of GWB density
with respect to frequency, the scalar-to-tensor ratio, and the B-L asymmetry
parameter, we can conclude that the results show good consistency with the
predicted bounds on the density and the frequency once we fine-tune the
inflationary parameter with parameters associated with the PTA signals.

##  IV Matching PTA Signals with Gravitational Waves Emitted by BBH Mergers
and the required PBH mass to explain the JWST Data

###  IV.1 Gravitational waves from BBH mergers and PTA Interpretation

In the early universe, supermassive primordial black holes with masses around
${\mathcal{O}}(10^{18}M_{\odot})$ could potentially emerge through various
mechanisms, particularly during the radiation-dominated era. Nevertheless,
before matter-radiation equality and as a consequence of the universe’s
expansion, multiple primordial black holes (PBHs) have the opportunity to
coexist within the same Hubble patch, allowing for the formation of isolated
pairs of two PBHs [71]. PBH mass can be calculated as follows [72],

$M_{PBH}\simeq M_{\odot}\left(5\times 10^{12}/\left(1+z_{f}\right)\right)^{2}$
(19)

In principle, binary primordial black holes (PBHs) can form either before
[73], or after [74, 75, 76] matter-radiation equality. In the context of the
binary black hole (BBH) merger, our focus will be on examining the observable
implications of this merger rate, which is represented by a power law
distribution model in the following form [77]:

$\frac{dN_{merg}}{dtdV}\equiv{\mathcal{R}}\left(z\right)=R_{0}\left(1+z\right)^{\alpha},$
(20)

for a current merger rate of $R_{0}=9-35Gpc^{-3}yr^{-1}$, the parameter
$\alpha$ is determined to be $2.7_{-1.9}^{+1.8}$. The total energy released in
gravitational waves, accounting for the inspiral, merging, and ringdown phases
is expressed as follows [80, 81]:

$\displaystyle\frac{dE_{gw}}{df}$ $\displaystyle=$
$\displaystyle\frac{\pi^{2/3}M_{c}^{5/3}}{3}\times$ (25)
$\displaystyle\left\\{\begin{array}[]{c}f_{s}^{-1/3}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}for~{}~{}f<f_{1}\\\
f_{1}^{-1}f^{2/3}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}for~{}~{}f_{1}\leq
f<f_{2}\\\
f_{2}^{-3/4}f_{1}^{-1}\frac{\sigma^{4}f}{\left(\sigma^{2}+4\left(f-f_{2}\right)^{2}\right)^{2}}~{}~{}~{}~{}~{}for~{}~{}f_{2}\leq
f<f_{3}\end{array}\right.$

the GW energy spectrum of the merger, denoted as $dE_{gw}/df_{s}$, is
associated with the frequency in the source frame, represented by $f_{s}$.
This frequency in the source frame is related to the observed frequency, $f$,
through the expression $f=f_{obs}\left(1+z\right)$. The function
$E\left(z\right)\equiv
H(z)/H_{0}=\left[\Omega_{r}\left(1+z\right)^{4}+\Omega_{m}\left(1+z\right)^{3}+\Omega_{\Lambda}\right]^{1/2}$,
where $M_{c}$ is the chirp mass, $M_{c}=M_{PBH}/2^{1/5}$ [71]. Analyzing the
spectral attributes of the BBH background typically involves describing the
energy density parameter to characterize the SGWB [82, 83]:

$\left.\Omega_{gw}(f)\right|_{f_{obs}}=\int_{0}^{z_{\sup}}\left.\Theta_{\nu}(f,z)\right|_{f_{obs}}\frac{d{\mathcal{R}}\left(z\right)}{dz}dz,$
(26)

here $d{\mathcal{R}}\left(z\right)/dz$ is the expression for the differential
gravitational wave event rate, while $\Theta_{\nu}$ represents the energy flux
emitted per unit frequency by a source located at a luminosity distance
$d_{L}(z)$:

$\left.\Theta_{\nu}(f,z)\right|_{f_{obs}}=\frac{1}{4\pi
d_{L}(z)^{2}}\frac{dE_{gw}}{df}\left(1+z\right),$ (27)

Ultimately, it proves beneficial to represent the model in the following form
[83]

$\displaystyle\left.\Omega_{gw}(f)\right|_{f_{obs}}$ $\displaystyle\simeq$
$\displaystyle 4.2\times 10^{-10}\left(\frac{r_{0}}{3.1\times
10^{-2}Mpc^{-3}Myr^{-1}}\right)$ (28)
$\displaystyle\times\left(\frac{M_{c}}{8.7M_{\odot}}\right)^{\frac{5}{3}}\left(\frac{{}_{f_{obs}}}{100Hz}\right)^{\frac{2}{3}},$

As our focus is on the current energy density of gravitational waves, it
becomes necessary to transform the earlier GW spectrum into a contemporary
physical parameters. One can obtain the present GW spectra as functions of the
observable density of GWs which can be formulated as follows:

$\Omega_{gw,0}(f)h^{2}=\left.\Omega_{gw}(f)\right|_{f_{obs}}\left(\frac{g_{\ast
j}}{g_{\ast 0}}\right)^{-\frac{1}{3}}\Omega_{r,0}h^{2}$ (29)

here $h$ is the present dimensionless Hubble constant and
$\Omega_{r,0}h^{2}\simeq 4.15\times 10^{-5}$ is the abundance of radiation
today, we choose $g_{\ast j}/g_{\ast 0}=106.75/3.36\simeq 31.$ Because
gravitational waves (GWs) propagate at the speed of light, they are present in
the Universe similar to radiation and add to the overall radiation energy. As
a result, measurements of the total radiation density from the Cosmic
Microwave Background (CMB) and Big Bang Nucleosynthesis (BBN) for species
beyond the standard model can impose limitations on the overall energy density
attributed to gravitational waves.

Figure 5:  Primordial gravitational waves evolution with respect to the scale
and the frequency $f$, the light blue region represents the NANOGrav data, and
$r_{0}$ is represented by three different colors for each value.

In Fig. 5, we show the abundance of the energy density of gravitational waves
as a function of the present value of the frequency. We also display three
expected values of $r_{0}=10^{-5},10^{-4},$ and $10^{-3}$. We numerically
compute the energy spectrum of the GWs to test the effect PBHs mass. The GW
spectrum increases the frequencies and spans over the NANOGrav sensitivity
region for $M_{PBH}\propto 10^{9}M_{\odot}$. We conclude that the BBH system
can partially contribute to the explanation of the energy spectrum at
nanohertz frequencies. However, PBHs mass $M_{PBH}\geq 10^{12}M_{\odot}$ are
not consistent with the expected NANOGrav GWs observatories. Moreover, the
parameter $r_{0}$ of the chosen model can affect the behavior of GWs density,
which means that higher values of $r_{0}$ increase the density
$\Omega_{gw,0}(f)h^{2}$. Here, we mention that the stochastic gravitational
waves from PBH binaries are important stochastic gravitational wave background
sources that are used to probe the Universe and can provide a possible
explanation for the recent NANOGrav results and JWST predictions as we will
see in the rest of the paper.

###  IV.2 The abundance of PBH and its dependence on the B-L Breaking
Mechanism

A process of PBH formation was first discussed in [84, 85, 86, 87] which
hinges on the dynamics of inflation, involving a scalar field with two
distinct vacuum states. These states become unevenly distributed following the
conclusion of inflation. As the universe transitions out of the inflationary
phase and cools down, the expansion rate decreases. Consequently, the scalar
field is allowed to move freely towards the lowest points in its potential
energy. This scenario gives rise to regions within the universe where less
probable vacuum states exist, surrounded by a more favored vacuum state.
Essentially, this implies that once vacuum states become populated during the
Friedmann-Robertson-Walker epoch, enclosed regions of less probable vacuum
form, resembling islands. Over time, these closed vacuum walls can become
causally connected and subsequently collapse, resulting in the formation of
black holes. The mass of these black holes corresponds to that of the
originating vacuum wall. Moreover, PBH formation during the early Universe
generally results from the collapse of significant density fluctuations [88].
Researchers have explored limitations on the prevalence of PBHs through
observations [89, 90], suggesting their abundance to be lower than 10-20 of
the overall energy density of the Universe. A crucial requirement is that the
density contrast needs to surpass the critical threshold of $\delta_{c}=0.414$
[91]. This condition ensures the formation of PBHs during the radiation-
dominated era, precisely when fluctuations come within the horizon. The
estimation of primordial black hole (PBH) generation involves evaluating the
portion of energy density, which is intricately linked to the variance of
density perturbations. For this purpose, we employ the power spectrum
alongside a window function, denoted as $W\left(kR\right)$ [92], to establish
the variance $\sigma(k)$ with a final form given as [95],

$\displaystyle\sigma^{2}$ $\displaystyle=$
$\displaystyle\int_{0}^{\infty}\left(kR\right)^{2}{\mathcal{P}}_{s}(k)W^{2}(\tilde{k}R)\frac{dk}{k},$
(30) $\displaystyle\sigma^{2}$ $\displaystyle\simeq$
$\displaystyle\frac{4\left(1+\omega\right)^{2}}{\left(5+3\omega\right)^{2}}{\mathcal{P}}_{s}(k).$
(31)

The PBH mass fraction can be obtained in an integral form when we assume the
Gaussian fluctuations are given by

$\beta\simeq\frac{1}{\sqrt{2\pi}}\frac{\sigma}{\delta_{c}}\exp\left(-\left(\frac{\delta_{c}}{\sqrt{2}\sigma}\right)^{2}\right).$
(32)

The current density parameter of the PBHs can be obtained from the relation
[95, 96],

$\beta=7.06\times
10^{-18}\Omega_{PBH}\left(\frac{M_{PBH}}{10^{15}g}\right)^{\frac{1}{2}}.$

Figure 6: The evolution of PBH mass fraction as functions of $M_{PBH}$ for
different values of the potential parameter $v_{B-L}$.

Primordial black holes originate from the amplification in the initial power
distribution at smaller scales. Assuming that, during a radiation-dominated
phase in the Universe’s evolution, it has achieved the adiabatic limit, and
the curvature perturbation on larger-than-horizon scales becomes immobilized
in comoving terms. Upon the resumption of a scale entering the Hubble horizon,
certain areas could exhibit a significant positive curvature, analogous to a
closed Universe. During this stage, the expansion halts and a phase of
contraction commences. Subsequently, the Hubble-sized area characterized by
substantial positive curvature will undergo a collapse, leading to the
formation of a black hole. The fraction of energy density can provide an
alternative explanation for investigating the collapses of primordial black
holes in the early Universe. Fig. 6 shows the fraction of the total energy
density for spherically symmetric regions collapsing into PBH as a function of
PBH mass for different values of $\upsilon_{B-L}$ parameter, The results show
that the curves converge toward $10^{1}$ where the overproduction regime
occurs and PBH production is satisfied, knowing that $\upsilon_{B-L}$ has a
notable effect on the behavior of the fraction of the total energy density
where they converge towards values around $M_{PBH}\propto 10^{16}g$. Moreover,
$\beta$ shows a decreasing behavior towards higher values of the fraction
$\beta,$ we should note that lower values $M_{PBH}$ results a convergence of
the fraction of the total energy density towards a maximal value of $\beta$.
It’s important to highlight that in this context, we’ve neglected the
evolution of PBHs through processes like radiation, accretion, and merging, as
these mechanisms fail to account for present-day observations. Thus, when
contemplating PBH accretion as a means to form massive galaxies, we need
supplementary criteria to elucidate the observations made by the JWST.

###  IV.3 Investigating the Mass of Primordial Black Holes using JWST
Observations

In recent studies, there has been significant interest in the investigation of
PBHs, as they have the potential to provide insights into a range of
cosmological occurrences. These include the formation of galaxies with high
redshifts, the characteristics of dark matter, and the origins of supermassive
black holes. After their formation, PBHs have the potential to undergo changes
over cosmic time through the process of accretion. The accumulation of
baryonic matter onto PBHs during this accretion phase can notably influence
their mass [97, 98]. It’s essential to emphasize that PBHs constitute only a
small fraction of the overall dark matter content. Moreover, each PBH is
enveloped by a prevailing dark matter halo, which grows over time, provided
that these PBHs avoid mutual interactions [99]. Sizeable primordial black
holes have the potential to act as origins of fluctuations at a specific mass
scale, denoted as MB, either through the Poisson effect [100, 101] or the seed
effect [102, 103]. Due to the presence of a high-density radiation field
around each PBH during its formation, the fluctuation becomes stagnant in the
radiation-dominated era. However, it starts to amplify during the matter-
dominated era. The mass $M_{B}$, which is bound at the redshift $z_{B}$, is a
key factor in this process [104].

$M_{B}\approx\left\\{\begin{array}[]{c}\frac{M_{PBH}f_{PBH}}{\left[\left(1+z_{B}\right)a_{eq}\right]^{2}}~{}~{}\left(Poisson~{}effect\right),\\\
\\\
\frac{M_{PBH}f_{PBH}}{\left(1+z_{B}\right)a_{eq}}~{}~{}\left(Seed~{}effect\right),\end{array}\right.$
(33)

here $a_{eq}$ is the scale factor at the matter–radiation equality. At this
point we will assume that $M_{B}\sim M_{halo}$ and the galaxy redshift $z\sim
z_{B}$. Furthermore, we should introduce the star formation efficiency
$\varepsilon$ given by,

$\varepsilon=\frac{M_{\ast}}{f_{b}M_{halo}},$ (34)

where $M_{halo}$ is the halo mass, and $f_{b}$ is the fraction of baryons in
matter.

Figure 7:  The evolution of galaxies stellar masses $M_{\ast}$ as a function
of the redshift for different values of the PBH mass $M_{PBH}$, the two dots
represent the two most massive high-z objects, galaxies 35300 and 38094
referred as $G1$ and $G2$ respectively.

Several investigations have imposed limitations on the mass of PBHs based on
the Poisson effect standpoint. Within this context, PBHs with a mass greater
than $10^{11}M_{\odot}$ have been ruled out by multiple studies due to their
non-detection, with the exception of Phoenix A [105]. Moreover, in addition to
the restriction posed by the $\mu$-distortion, PBHs with a mass around
$10^{5}M_{\odot}$ and above also face comparatively milder constraints
stemming from sources such as X-ray binaries [106], PBH infall into the
Galactic center influenced by dynamical friction [107], and statistical
properties of the large-scale cosmic structure [104]. These combined factors
necessitate that the fraction of dark matter composed of PBHs, $f_{PBH}$
should be in the range of $10^{-4}$ to $10^{-3}$ for PBH masses spanning from
around $10^{5}$ to $10^{11}$ $M_{\odot}$ [108]. Additionally, based on data
from the high-redshift Lyman-$\alpha$ forest observations [109], primordial
black holes with $M_{PBH}f_{PBH}$ $\gtrsim$ $170M_{\odot}$ and $f_{PBH}$
$>0.05$ have been excluded. This indicates a significant unfavorable stance
towards the existence of supermassive PBHs capable of contributing to the
Poisson effect. In Fig. 7 we have plotted the variation of galaxies’ stellar
masses $M_{\ast}$ as functions of the redshift $z$, we observe that the
stellar mass decreases slowly with respect to the redshift towards stable
values. Moreover, the galaxy formation efficiency $\varepsilon$ has a notable
effect on the PBH mass $M_{PBH}$ which could get lower as we take higher
values of $\varepsilon.$ Finally, the influence of the seed effect, which
operates on smaller scales, remains a viable explanation for interpreting the
observations made by the JWST observation.

##  V Conclusion

The process of B-L symmetry breaking at the Grand Unified Theory scale can be
counted as an additional factor for the initial conditions of the Universe,
which affects the entropy, the occurrence of baryogenesis through
leptogenesis, and the formation of dark matter due to the thermal generation
of gravitinos. In the preceding sections, we computed the gravitational wave
pattern resulting from the cosmological breaking of B-L symmetry. This process
occurred due to several factors, including the era of inflation, the phase
transition of the Universe, Cosmic strings, and the gravitational waves linked
to binary systems of black holes forming in the early Universe. Additionally,
our focus was directed toward the JWST predictions related to the PBH masses
and the early galaxy formation.

To go into further detail, our findings show first that considering the scalar
potential governing the B-L symmetry breaking mechanism, the model parameter
$\lambda$ shows good consistency for a wide range of model parameters with the
latest observations from Planck data. As for the critical value of the phase
transition parameter $\upsilon_{B-L},$ the results show that it must be
bounded around $\sim[2\times 10^{15},5\times 10^{15}]GeV,$ our findings also
align with earlier findings that suggested tuning the $\upsilon_{B-L}$ value
to be close to the GUT scale. Moreover, when we study the evolution of the B-L
symmetry breaking parameter $\upsilon_{B-L}$ as a function of the wavelength
$k,$ we conclude that considering superhorizon scales the potential parameter
will decreased as we take higher values of inflation e-fold number.
Additionally, the spectral index shows that for both smaller and larger scales
the $\upsilon_{B-L}$ is best considered around the previously observed values.
On the other hand, for the variation of the density of primordial
gravitational waves with respect to the scale $k$ and $\upsilon_{B-L}$ for
different phases, we found that the change in the critical value
$\upsilon_{B-L}$ that corresponds to B-L asymmetry has a proportional effects
on $\Omega_{gw,0}(f)$ for all matter, radiation, and kinetic eras, from the
results, we see that the proposed scenario of Leptogenesis has great
implications of the observed values of PGWs density which can be predicted by
PTA signals.

Moreover, we studied the NanoGrav proposed model of the density
$h^{2}\Omega_{gw}(f)$ as a function of the frequency $f$, scalar-to-tensor
ratio $r$, and the amplitude of the PTA signal that relates to the proposed
potential parameters that explain the Leptogenesis phenomena, we identified
the right parameters that can predict higher or lower density of gravitational
waves that reproduced the NANOGrav predictions, the results show good
consistency with the predicted bounds on the density and the frequency once we
fine-tune the inflationary parameter with parameters associated with the PTA
signals.

We have calculated the density of gravitational waves from BBH mergers that
can be predicted by PTA interpretation, we conclude that the BBH system can
partially contribute to the explanation of the energy spectrum at nanohertz
frequencies. Moreover, PBH mass around $M_{PBH}\propto 10^{9}M_{\odot}$
presents the best fit with recent NANOGrav data. Additionally, PBH binaries
are important stochastic gravitational wave background sources that are used
to probe the Universe and can provide a possible explanation for the recent
NANOGrav results and JWST predictions. On another hand, when we study PBH
formation $\upsilon_{B-L}$ has a notable effect on the behavior of the
fraction of the total energy density where they converge towards values
$M_{PBH}\propto 10^{16}g.$ However, the mass of PBH at the formation time fail
to explain recent data, for this reason, we need to consider that the
evolution of PBHs has already happened which leads to massive galaxies
formation, then supplementary criteria will be considered to explain the
observations made by the JWST. The variation of galaxies stellar masses
$M_{\ast}$ as functions of the redshift $z$, has shown that the galaxy
formation efficiency has a notable effect on the PBH mass, then the influence
of the seed, effect remains a viable tool for interpreting the JWST
observations.

## Acknowledgements

The research by M.K. was carried out in Southern Federal University with
financial support of the Ministry of Science and Higher Education of the
Russian Federation (State contract GZ0110/23-10-IF).

## References

  * [1] Starobinsky, Alexei A. Relict gravitation radiation spectrum and initial state of the universe. JETP lett 30.682-685 (1979): 131-132.
  * [2] Rubakov, V. A., Sazhin, M. V., & Veryaskin, A. V. (1982). Graviton creation in the inflationary universe and the grand unification scale. Physics Letters B, 115(3), 189-192.
  * [3] Blasi, S., Brdar, V., & Schmitz, K. (2021). Has NANOGrav found first evidence for cosmic strings?. Physical Review Letters, 126(4), 041305.
  * [4] Ellis, J., & Lewicki, M. (2021). Cosmic string interpretation of NANOGrav pulsar timing data. Physical Review Letters, 126(4), 041304.
  * [5] Mandic, V., Bird, S., & Cholis, I. (2016). Stochastic gravitational-wave background due to primordial binary black hole mergers. Physical review letters, 117(20), 201102.
  * [6] Fukugita, M., & Yanagida, T. (1986). Barygenesis without grand unification. Physics Letters B, 174(1), 45-47.
  * [7] Lazarides, G., & Shafi, Q. (1991). Origin of matter in the inflationary cosmology. Physics Letters B, 258(3-4), 305-309.
  * [8] Yanagida, T. (1979). Proc. Workshop on Unified theory and the baryon number in the universe. KEK Report No. 79-18, 95.
  * [9] Gell-Mann, M., Ramond, P., & Slansky, R. (1979). Supergravity ed P van Nieuwenhuizen and DZ Freedman. Amsterdam: North-Holland) p, 315, 79-18.
  * [10] Fukugita, M., & Yanagida, T. (1986). Barygenesis without grand unification. Physics Letters B, 174(1), 45-47.
  * [11] Buchmüller, W., & Plümacher, M. (1999). Matter–antimatter asymmetry and neutrino properties. Physics reports, 320(1-6), 329-339.
  * [12] Faessler, A., Kosmas, T. S., Leontaris, G. K., & Lazarides, G. (2000). Leptogenesis in supersymmetric hybrid inflation. Symmetries in intermediate and high energy physics, 227-245.
  * [13] Asaka, T., Hamaguchi, K., Kawasaki, M., & Yanagida, T. (1999). Leptogenesis in inflaton decay. Physics Letters B, 464(1-2), 12-18.
  * [14] Campbell, B. A., Davidson, S., & Olive, K. A. (1993). Inflation, neutrino baryogenesis, and (s) neutrino-induced baryogenesis. Nuclear Physics B, 399(1), 111-136.
  * [15] Khlopov, M. Y., & Linde, A. D. (1984). Is it easy to save the gravitino?. Physics Letters B, 138(4), 265-268.
  * [16] Ellis, J., Kim, J. E., & Nanopoulos, D. V. (1984). Cosmological gravitino regeneration and decay. Physics Letters B, 145(3-4), 181-186.
  * [17] Bolz, M., Buchmüller, W., & Plümacher, M. (1998). Baryon asymmetry and dark matter. Physics Letters B, 443(1-4), 209-213.
  * [18] Khlopov, M. Y., Levitan, Y. L., Sedel’Nikov, E. V., & Sobol, I. M. (1994). Nonequilibrium cosmological nucleosynthesis of light elements: Calculations by the Monte Carlo method. Physics of Atomic Nuclei, 57(8), 1393-1397.
  * [19] Khlopov, M. Y., Barrau, A., & Grain, J. (2006). Gravitino production by primordial black hole evaporation and constraints on the inhomogeneity of the early universe. Classical and Quantum Gravity, 23(6), 1875.
  * [20] Balestra, F. (1984). Annihilation of antiprotons with Helium-4 at low energies and its relationship with the problems of the modern cosmology and models of grand unification. Sov. J. Nucl. Phys, 39, 626-631.
  * [21] Antoniadis, J., Arumugam, P., Arumugam, S., Babak, S., Bagchi, M., Nielsen, A. S. B., … & Wu, Z. (2023). The second data release from the European Pulsar Timing Array III. Search for gravitational wave signals. arXiv preprint arXiv:2306.16214.
  * [22] Antoniadis, J., Arumugam, P., Arumugam, S., Babak, S., Bagchi, M., Nielsen, A. S. B., … & Wu, Z. (2023). The second data release from the European Pulsar Timing Array. Implications for massive black holes, dark matter and the early Universe, 2306.
  * [23] Zic, A., Reardon, D. J., Kapur, A., Hobbs, G., Mandow, R., Curyło, M., … & Zhu, X. J. (2023). The Parkes Pulsar Timing Array Third Data Release. Publications of the Astronomical Society of Australia, 1-15.
  * [24] Rubin, S. G., Sakharov, A. S., & Khlopov, M. Y. (2001). The formation of primary galactic nuclei during phase transitions in the early universe. Journal of Experimental and Theoretical Physics, 92, 921-929.
  * [25] Belotsky, K. M., Dokuchaev, V. I., Eroshenko, Y. N., Esipova, E. A., Khlopov, M. Y., Khromykh, L. A., … & Svadkovsky, I. V. (2019). Clusters of primordial black holes. The European Physical Journal C, 79, 1-20.
  * [26] Naidu, R. P., Oesch, P. A., van Dokkum, P., Nelson, E. J., Suess, K. A., Brammer, G., … & Weibel, A. (2022). Two remarkably luminous galaxy candidates at $z\approx 10-12$ revealed by JWST. The Astrophysical Journal Letters, 940(1), L14.
  * [27] Adams, N. J., Conselice, C. J., Ferreira, L., Austin, D., Trussler, J. A. A., Juodžbalis, I., … & Vijayan, A. P. (2023). Discovery and properties of ultra-high redshift galaxies ($\mathit{9<z<12}$) in the JWST ERO SMACS 0723 Field. Monthly Notices of the Royal Astronomical Society, 518(3), 4755-4766.
  * [28] Yan, H., Ma, Z., Ling, C., Cheng, C., & Huang, J. S. (2022). First batch of candidate galaxies at redshifts 11 to 20 revealed by the James Webb Space Telescope early release observations. arXiv preprint arXiv:2207.11558.
  * [29] Labbe, I., van Dokkum, P., Nelson, E., Bezanson, R., Suess, K., Leja, J., … & Stefanon, M. (2022). A very early onset of massive galaxy formation. arXiv preprint arXiv:2207.12446.
  * [30] Y.B. Zel’dovich and I.D. Novikov, The Hypothesis of Cores Retarded during Expansion and the Hot Cosmological Model, Soviet Astron. AJ (Engl. Transl.), 10 (1967) 602.
  * [31] Hawking, S. (1971). Gravitationally collapsed objects of very low mass. Monthly Notices of the Royal Astronomical Society, 152(1), 75-78.
  * [32] Bavera, S. S., Franciolini, G., Cusin, G., Riotto, A., Zevin, M., & Fragos, T. (2022). Stochastic gravitational-wave background as a tool for investigating multi-channel astrophysical and primordial black-hole mergers. Astronomy & Astrophysics, 660, A26.
  * [33] Abbott, R., Abbott, T. D., Acernese, F., Ackley, K., Adams, C., Adhikari, N., … & Brinkmann, M. (2022). Search for Subsolar-Mass Binaries in the First Half of Advanced LIGO’s and Advanced Virgo’s Third Observing Run. Physical review letters, 129(6), 061104.
  * [34] Carr, B., & Kühnel, F. (2020). Primordial black holes as dark matter: recent developments. Annual Review of Nuclear and Particle Science, 70, 355-394.
  * [35] Hoyle, F., & Narlikar, J. V. (1966). On the formation of elliptical galaxies. Proceedings of the Royal Society of London. Series A. Mathematical and Physical Sciences, 290(1421), 177-185.
  * [36] Carr, B. J., & Silk, J. (1983). Can graininess in the early universe make galaxies?. Astrophysical Journal, Part 1 (ISSN 0004-637X), vol. 268, May 1, 1983, p. 1-16., 268, 1-16.
  * [37] Carr, B. J., & Rees, M. J. (1984). Can pregalactic objects generate galaxies?. Monthly Notices of the Royal Astronomical Society, 206(4), 801-818.
  * [38] Datta, S. (2023). Inflationary gravitational waves, pulsar timing data and low-scale-leptogenesis. arXiv preprint arXiv:2307.00646.
  * [39] Buchmüller, W., Domcke, V., Kamada, K., & Schmitz, K. (2013). The gravitational wave spectrum from cosmological B-L breaking. Journal of Cosmology and Astroparticle Physics, 2013(10), 003.
  * [40] Gell-Mann, M., Ramond, P., & Slansky, R. (1979). Supergravity ed P van Nieuwenhuizen and DZ Freedman. Amsterdam: North-Holland) p, 315, 79-18.
  * [41] Yanagida, T. (1979). Proc. Workshop on Unified theory and the baryon number in the universe. KEK Report No. 79-18, 95.
  * [42] Aghanim, N., Akrami, Y., Ashdown, M., Aumont, J., Baccigalupi, C., Ballardini, M., … & Roudier, G. (2020). Planck 2018 results-VI. Cosmological parameters. Astronomy & Astrophysics, 641, A6.
  * [43] Baumann, D. (2009). TASI lectures on inflation. arXiv preprint arXiv:0907.5424.
  * [44] Nakayama, K., Takahashi, F., & Yanagida, T. T. (2010). Constraint on the gravitino mass in hybrid inflation. Journal of Cosmology and Astroparticle Physics, 2010(12), 010.
  * [45] Buchmüller, W., Domcke, V., & Schmitz, K. (2012). Spontaneous B–L breaking as the origin of the hot early universe. Nuclear Physics B, 862(3), 587-632.
  * [46] Sahu, N., & Yajnik, U. A. (2005). Gauged B-L symmetry and baryogenesis via leptogenesis at TeV scale. Physical Review D, 71(2), 023507\.
  * [47] El Bourakadi, K., Ferricha-Alami, M., Filali, H., Sakhi, Z., & Bennai, M. (2021). Gravitational waves from preheating in Gauss–Bonnet inflation. The European Physical Journal C, 81(12), 1144.
  * [48] El Bourakadi, K., Asfour, B., Sakhi, Z., Bennai, M., & Ouali, T. (2022). Primordial black holes and gravitational waves in teleparallel Gravity. The European Physical Journal C, 82(9), 792.
  * [49] El Bourakadi, K., Bousder, M., Sakhi, Z., & Bennai, M. (2021). Preheating and reheating constraints in supersymmetric braneworld inflation. The European Physical Journal Plus, 136(8), 1-19.
  * [50] Sakhi, Z., El Bourakadi, K., Safsafi, A., Ferricha-Alami, M., Chakir, H., & Bennai, M. (2020). Effect of brane tension on reheating parameters in small field inflation according to Planck-2018 data. International Journal of Modern Physics A, 35(30), 2050191.
  * [51] Bousder, M., El Bourakadi, K., & Bennai, M. (2021). Charged 4d einstein-gauss-bonnet black hole: Vacuum solutions, cauchy horizon, thermodynamics. Physics of the Dark Universe, 32, 100839.
  * [52] El Bourakadi, K., Sakhi, Z., & Bennai, M. (2022). Preheating constraints in $\mathit{\alpha}$-attractor inflation and Gravitational Waves production. International Journal of Modern Physics A, 37(17), 2250117.
  * [53] El Bourakadi, K., Koussour, M., Otalora, G., Bennai, M., & Ouali, T. (2023). Constant-roll and primordial black holes in f (Q, T) gravity. Physics of the Dark Universe, 41, 101246.
  * [54] Bourakadi, K. E., Ferricha-Alami, M., Sakhi, Z., Bennai, M., & Chakir, H. (2023). Dark matter via Baryogenesis: Affleck-Dine Mechanism in the Minimal Supersymmetric Standard Model. arXiv preprint arXiv:2307.15541.
  * [55] Sahni, V., Sami, M., & Souradeep, T. (2001). Relic gravity waves from braneworld inflation. Physical Review D, 65(2), 023518.
  * [56] Figueroa, D. G., & Tanin, E. H. (2019). Inconsistency of an inflationary sector coupled only to Einstein gravity. Journal of Cosmology and Astroparticle Physics, 2019(10), 050.
  * [57] Giovannini, M. (1998). Gravitational wave constraints on post-inflationary phases stiffer than radiation. Physical Review D, 58(8), 083504.
  * [58] Riazuelo, A., & Uzan, J. P. (2000). Quintessence and gravitational waves. Physical Review D, 62(8), 083506.
  * [59] Baumann, D., Steinhardt, P., Takahashi, K., & Ichiki, K. (2007). Gravitational wave spectrum induced by primordial scalar perturbations. Physical Review D, 76(8), 084019.
  * [60] Vagnozzi, S. (2021). Implications of the NANOGrav results for inflation. Monthly Notices of the Royal Astronomical Society: Letters, 502(1), L11-L15.
  * [61] Zhao, W., Zhang, Y., You, X. P., & Zhu, Z. H. (2013). Constraints of relic gravitational waves by pulsar timing arrays: Forecasts for the FAST and SKA projects. Physical Review D, 87(12), 124012.
  * [62] Vagnozzi, S. (2023). Inflationary interpretation of the stochastic gravitational wave background signal detected by pulsar timing array experiments. Journal of High Energy Astrophysics.
  * [63] Kuroyanagi, S., Chiba, T., & Sugiyama, N. (2009). Precision calculations of the gravitational wave background spectrum from inflation. Physical Review D, 79(10), 103501.
  * [64] Afzal, A., Agazie, G., Anumarlapudi, A., Archibald, A. M., Arzoumanian, Z., Baker, P. T., … & NANOGrav Collaboration. (2023). The NANOGrav 15 yr Data Set: Search for Signals from New Physics. The Astrophysical Journal Letters, 951(1), L11.
  * [65] Agazie, G., Anumarlapudi, A., Archibald, A. M., Arzoumanian, Z., Baker, P. T., Bécsy, B., … & NANOGrav Collaboration. (2023). The NANOGrav 15 yr data set: Evidence for a gravitational-wave background. The Astrophysical Journal Letters, 951(1), L8.
  * [66] Agazie, G., Alam, M. F., Anumarlapudi, A., Archibald, A. M., Arzoumanian, Z., Baker, P. T., … & NANOGrav Collaboration. (2023). The NANOGrav 15 yr Data Set: Observations and Timing of 68 Millisecond Pulsars. The Astrophysical Journal Letters, 951(1), L9.
  * [67] Arzoumanian, Z., Baker, P. T., Blumer, H., Bécsy, B., Brazier, A., Brook, P. R., … & NANOGrav Collaboration. (2020). The NANOGrav 12.5 yr data set: search for an isotropic stochastic gravitational-wave background. The Astrophysical Journal Letters, 905(2), L34.
  * [68] Arzoumanian, Z., Brazier, A., Burke-Spolaor, S., Chamberlin, S. J., Chatterjee, S., Christy, B., … & NANOGrav Collaboration. (2016). The NANOGrav nine-year data set: limits on the isotropic stochastic gravitational wave background. The Astrophysical Journal, 821(1), 13.
  * [69] Guo, S. Y., Khlopov, M., Liu, X., Wu, L., Wu, Y., & Zhu, B. (2023). Footprints of Axion-Like Particle in Pulsar Timing Array Data and JWST Observations. arXiv preprint arXiv:2306.17022.
  * [70] Zhao, W., Zhang, Y., You, X. P., & Zhu, Z. H. (2013). Constraints of relic gravitational waves by pulsar timing arrays: Forecasts for the FAST and SKA projects. Physical Review D, 87(12), 124012.
  * [71] Gouttenoire, Y., Trifinopoulos, S., Valogiannis, G., & Vanvlasselaer, M. (2023). Scrutinizing the Primordial Black Holes Interpretation of PTA Gravitational Waves and JWST Early Galaxies. arXiv preprint arXiv:2307.01457.
  * [72] Sasaki, M., Suyama, T., Tanaka, T., & Yokoyama, S. (2018). Primordial black holes-perspectives in gravitational wave astronomy. Classical and Quantum Gravity, 35(6), 063001.
  * [73] Nakamura, T., Sasaki, M., Tanaka, T., & Thorne, K. S. (1997). Gravitational waves from coalescing black hole MACHO binaries. The Astrophysical Journal, 487(2), L139.
  * [74] Quinlan, G. D., & Shapiro, S. L. (1988). Star cluster collapse to a supermassive black hole: binaries and gravitational radiation. In Proceedings of the 2nd Canadian Conference on General Relativity and Relativistic Astrophysics (pp. 335-338).
  * [75] Mouri, H., & Taniguchi, Y. (2002). Runaway merging of black holes: analytical constraint on the timescale. The Astrophysical Journal, 566(1), L17.
  * [76] Bird, S., Cholis, I., Muñoz, J. B., Ali-Haïmoud, Y., Kamionkowski, M., Kovetz, E. D., … & Riess, A. G. (2016). Did LIGO detect dark matter?. Physical review letters, 116(20), 201301.
  * [77] Atal, V., Blanco-Pillado, J. J., Sanglas, A., & Triantafyllou, N. (2022). Constraining changes in the merger history of BH and PBH binaries with the stochastic gravitational wave background. Physical Review D, 105(12), 123522.
  * [78] Abbott, B. P., Abbott, R., Abbott, T. D., Abernathy, M. R., Acernese, F., Ackley, K., … & Chakraborty, R. (2016). GW150914: Implications for the stochastic gravitational-wave background from binary black holes. Physical review letters, 116(13), 131102.
  * [79] Abbott, B. P., Jawahar, S., Lockerbie, N. A., & Tokmakov, K. V. (2016). LIGO Scientific Collaboration and Virgo Collaboration (2016) Directly comparing GW150914 with numerical solutions of Einstein’s equations for binary black hole coalescence. Physical Review D, 94 (6). ISSN 1550-2368, http://dx. doi. org/10.1103/PhysRevD. 94.064035. PHYSICAL REVIEW D Phys Rev D, 94, 064035.
  * [80] Ajith, P., Babak, S., Chen, Y., Hewitson, M., Krishnan, B., Sintes, A. M., … & Thornburg, J. (2008). Template bank for gravitational waveforms from coalescing binary black holes: Nonspinning binaries. Physical Review D, 77(10), 104017.
  * [81] Ajith, P., Hannam, M., Husa, S., Chen, Y., Brügmann, B., Dorband, N., … & Seiler, J. (2011). Inspiral-merger-ringdown waveforms for black-hole binaries with nonprecessing spins. Physical Review Letters, 106(24), 241101.
  * [82] Ajith, P., Babak, S., Chen, Y., Hewitson, M., Krishnan, B., Sintes, A. M., … & Thornburg, J. (2008). Template bank for gravitational waveforms from coalescing binary black holes: Nonspinning binaries. Physical Review D, 77(10), 104017.
  * [83] Zhu, X. J., Howell, E., Regimbau, T., Blair, D., & Zhu, Z. H. (2011). Stochastic gravitational wave background from coalescing binary black holes. The Astrophysical Journal, 739(2), 86.
  * [84] Khlopov, M. Y., Rubin, S. G., & Sakharov, A. S. (2002). Strong primordial inhomogeneities and galaxy formation. arXiv preprint astro-ph/0202505.
  * [85] Khlopov, M. Y., Rubin, S. G., & Sakharov, A. S. (2005). Primordial structure of massive black hole clusters. Astroparticle Physics, 23(2), 265-277.
  * [86] Khlopov, M. (2007, May). Primordial nonlinear structures and massive black holes from early Universe. In Journal of Physics: Conference Series (Vol. 66, No. 1, p. 012032). IOP Publishing.
  * [87] Khlopov, M. Y. (2010). Primordial black holes. Research in Astronomy and Astrophysics, 10(6), 495.
  * [88] Carr, Bernard J. ”The Primordial black hole mass spectrum.” (1975): 1-19.
  * [89] N. Sanchez, A. Zichichi, Current Topics in Astrofundamental Physics edited by N. Sanchez (1997)
  * [90] Green, A. M., & Liddle, A. R. (1997). Constraints on the density perturbation spectrum from primordial black holes. Physical Review D, 56(10), 6166.
  * [91] Bhaumik, N., & Jain, R. K. (2020). Primordial black holes dark matter from inflection point models of inflation and the effects of reheating. Journal of Cosmology and Astroparticle Physics, 2020(01), 037.
  * [92] Zhou, Z., Jiang, J., Cai, Y. F., Sasaki, M., & Pi, S. (2020). Primordial black holes and gravitational waves from resonant amplification during inflation. Physical Review D, 102(10), 103527.
  * [93] Green, A. M., & Malik, K. A. (2001). Primordial black hole production due to preheating. Physical Review D, 64(2), 021301.
  * [94] El Bourakadi, K., et al. ”Primordial black holes and gravitational waves in teleparallel Gravity.” The European Physical Journal C 82.9 (2022): 792.
  * [95] Carr, B., Kohri, K., Sendouda, Y., & Yokoyama, J. I. (2021). Constraints on primordial black holes. Reports on Progress in Physics, 84(11), 116902.
  * [96] Su, B. Y., Li, N., & Feng, L. (2023). An inflation model for massive primordial black holes to interpret the JWST observations. arXiv preprint arXiv:2306.05364.
  * [97] De Luca, V., Franciolini, G., Pani, P., & Riotto, A. (2020). Constraints on primordial black holes: the importance of accretion. Physical Review D, 102(4), 043505.
  * [98] De Luca, V., Franciolini, G., & Riotto, A. (2023). Heavy Primordial Black Holes from Strongly Clustered Light Black Holes. Physical Review Letters, 130(17), 171401.
  * [99] Yuan, G. W., Lei, L., Wang, Y. Z., Wang, B., Wang, Y. Y., Chen, C., … & Fan, Y. Z. (2023). Rapidly growing primordial black holes as seeds of the massive high-redshift JWST Galaxies. arXiv preprint arXiv:2303.09391.
  * [100] Hoyle, F., & Narlikar, J. V. (1966). On the formation of elliptical galaxies. Proceedings of the Royal Society of London. Series A. Mathematical and Physical Sciences, 290(1421), 177-185.
  * [101] Meszaros, P. (1975). Primeval black holes and galaxy formation. Astronomy and Astrophysics, 38, 5-13.
  * [102] Carr, B. J., & Silk, J. (1983). Can graininess in the early universe make galaxies?. Astrophysical Journal, Part 1 (ISSN 0004-637X), vol. 268, May 1, 1983, p. 1-16., 268, 1-16.
  * [103] Carr, B. J., & Rees, M. J. (1984). Can pregalactic objects generate galaxies?. Monthly Notices of the Royal Astronomical Society, 206(4), 801-818.
  * [104] Carr, B., & Silk, J. (2018). Primordial black holes as generators of cosmic structures. Monthly Notices of the Royal Astronomical Society, 478(3), 3756-3775.
  * [105] Brockamp, M., Baumgardt, H., Britzen, S., & Zensus, A. (2016). Unveiling Gargantua: A new search strategy for the most massive central cluster black holes. Astronomy & Astrophysics, 585, A153.
  * [106] Inoue, Y., & Kusenko, A. (2017). New X-ray bound on density of primordial black holes. Journal of Cosmology and Astroparticle Physics, 2017(10), 034.
  * [107] Carr, B. J., & Sakellariadou, M. (1999). Dynamical constraints on dark matter in compact objects. The Astrophysical Journal, 516(1), 195.
  * [108] Su, B. Y., Li, N., & Feng, L. (2023). An inflation model for massive primordial black holes to interpret the JWST observations. arXiv preprint arXiv:2306.05364.
  * [109] Murgia, R., Scelfo, G., Viel, M., & Raccanelli, A. (2019). Lyman-$\alpha$ forest constraints on primordial black holes as dark matter. Physical Review Letters, 123(7), 071102.


# Exploring Spoken Named Entity Recognition: A Cross-Lingual Perspective

Moncef Benaicha$\ast$, David Thulke$\ddagger$, M. A. Tuğtekin Turan$\ast$

###### Abstract

Recent advancements in Named Entity Recognition (NER) have significantly
improved the identification of entities in textual data. However, spoken NER,
a specialized field of spoken document retrieval, lags behind due to its
limited research and scarce datasets. Moreover, cross-lingual transfer
learning in spoken NER has remained unexplored. This paper utilizes transfer
learning across Dutch, English, and German using pipeline and End-to-End (E2E)
schemes. We employ Wav2Vec2-XLS-R models on custom pseudo-annotated datasets
and investigate several architectures for the adaptability of cross-lingual
systems. Our results demonstrate that End-to-End spoken NER outperforms
pipeline-based alternatives over our limited annotations. Notably, transfer
learning from German to Dutch surpasses the Dutch E2E system by 7% and the
Dutch pipeline system by 4%. This study not only underscores the feasibility
of transfer learning in spoken NER but also sets promising outcomes for future
evaluations, hinting at the need for comprehensive data collection to augment
the results.

##  1 Introduction

00footnotetext: This work was carried out while the first author was a
research assistant at Fraunhofer IAIS.

Named Entity Recognition (NER) involves identifying and classifying named
entities such as persons, organizations, locations, and other predefined
categories in a text [1]. While substantial progress has been made in NER for
written text, applying these techniques to spoken language has not been widely
studied because of its unique challenges [2].

Spoken NER is a complex task due to spoken language’s inherent variability and
ambiguity. Factors such as pronunciation variations, speech disfluencies, and
background noise can significantly impact the performance of these systems
[3]. Moreover, the lack of explicit boundaries between words in continuous
speech adds an additional layer of complexity [4]. Despite these challenges,
spoken NER is of great interest due to its potential applications in various
domains, such as voice assistants, automatic transcription services, and
spoken dialogue systems [5, 6, 7].

Recent advancements in Transformer-based approaches have opened new avenues
for this research. In particular, the application of End-to-End (E2E) models,
which directly map speech to transcriptions with entity tags, has shown
promising results [8]. These models can capture the temporal dependencies and
handle the variability in spoken language. However, most of the current
research focuses on high-resource languages like English, and the performance
of these models may drop with the less available data.

This paper addresses the language gap by exploring cross-lingual transfer
learning for spoken NER. We investigate methods that leverage multilingual
language representation models and evaluate their performance on low-resource
setups. Our study focuses on three languages: Dutch, English, and German. We
particularly emphasize different low-resource language transfers,
demonstrating the efficacy of transfer learning when we attempt to apply the
proposed models on zero to low-resource scenarios.

This paper also compares various methods, including pipeline and E2E
approaches. The two-step pipeline architecture is implemented by cascading the
Automatic Speech Recognition (ASR) and NER models. Initially, the ASR system
converts the speech into a text sequence, and then the NER system marks entity
tags based on the sequence. Recent studies have proposed E2E approaches that
jointly perform ASR and NER [9]. This primarily serves two objectives: the
simultaneous optimization of the ASR with Natural Language Understanding (NLU)
and the reduction of error propagation and/or amplification caused by the ASR
bottleneck [10]. However, E2E models require extensive training data, and a
significant challenge lies in the scarcity of paired audio-textual data with
entity annotations. Moreover, generating a substantial volume of annotated
speech data is complex and costly. To solve this issue, Pasad et al. annotated
the external data with a labeling model to produce pseudo-annotation [11].
Following this idea, we prepared our pseudo-annotated Dutch, English, and
German dataset using XLM-RL based [12] NER model.

We further employ the impact of different factors, such as the amount of
training data, the choice of the language model, and the target language, on
the performance of spoken NER systems. This paper contributes to the ongoing
research in spoken NER by providing insights into the challenges and potential
solutions for cross-lingual schemes. Our findings contribute to the ongoing
advancements in spoken document retrieval and pave the way for more
sophisticated and accurate spoken NER systems. In line with the principles of
open research, all the code, data, and results used in this work will be made
publicly accessible111https://github.com/MoncefBenaicha/SpokenNER. To
summarize our contributions,

  * •

We comprehensively compare and analyze the pipeline and E2E approaches for
spoken NER in Dutch, English, and German.

  * •

We investigate different spoken NER transfer learning scenarios on pipeline
and E2E schemes.

  * •

We employ transfer learning from high-resource languages, German, to low-
resource languages, Dutch, to improve recognition of spoken NER by around 10%.

##  2 Related Work

In the literature, NER from the speech is generally performed via a pipeline
method that initially incorporates ASR, followed by NER on the transcriptions
[13]. Despite the prevalent issues associated with this sequential two-step
process, such systems also fail to directly incorporate transcriptions
annotated with entities within the ASR system [14]. Incorporating such data
could potentially enhance the quality of partial hypotheses, often discarded
during the decoding phase. One proposed method to incorporate entity data
involves directly integrating specific entity expressions into the lexicon and
estimating a language model that acknowledges these entity expressions [15].

These challenges motivate the investigation of the E2E spoken NER strategy for
extracting entities directly from speech. By utilizing this method, one can
facilitate joint optimization for both ASR and NER, approached from a NER task
perspective. This architecture is anticipated to be more streamlined than a
conventional pipeline, and we expect to leverage the capabilities of deep
neural architecture to capture long-distance constraints at the sentence
level.

The first study for the E2E spoken NER scheme was applied to French datasets
[5]. The authors adopted an architecture akin to Deep Speech 2 [16], which was
trained via the Connectionist Temporal Classification (CTC) objective [17].
Yadav et al. [18] adopted a similar approach for the English language. Their
study utilized special tokens in the ASR vocabulary to facilitate NER tagging.
Similarly, we also employ special symbols (’{’, ’[’, ’$’, ’]’) in the
transcript to identify the entities (organizations, persons, and locations) as
shown in Figure 1. More recently, [19, 11] used the same approach on top of a
Wav2Vec2 model.

Consequently, the E2E spoken NER learns to align the speech signal with its
annotated transcription, allowing it to extract entities directly from speech.
Results from the French and English datasets demonstrate that the E2E method
could outperform the pipeline approach based on the Long Short-term Memory
based models, which are currently below the state-of-the-art benchmarks [20].

Figure 1: Block diagram of the implementations; left side is a two-stage
pipeline, and right is an E2E spoken NER.

##  3 Methodology

###  3.1 Baseline Models

We establish two distinct baseline systems for each of the three languages -
English, German, and Dutch. The first system is built on a pipeline approach
where we separately train ASR and NER models, which are subsequently combined
during the inference stage. The second system adopts an E2E approach. The ASR
model is fine-tuned in the pipeline scheme over a pre-trained
Wav2Vec2-XLS-R-300M [21] model. Each language is treated separately by
applying the CTC-loss as the objective function. In the same system, the NER
model is a fine-tuned version of the pre-trained language representation model
XLM-RL coupled with a linear layer, which is applied to lower-cased CoNLL 2002
and 2003 [22, 23].

To enhance the ASR model, a 4-gram language model trained on both training and
development sets is used for each language. As for the E2E system,
Wav2Vec2-XLS-R-300M is again employed, but this time it’s fine-tuned on an
augmented corpus where it includes special tokens that denote the beginning of
an entity such as a person (PER), an organization (ORG), or a location (LOC),
and the termination of an entity, as illustrated in Figure 1.

###  3.2 Transfer Learning Models

We utilize the system’s dual-component architecture in the pipeline approach,
substituting the original NER model with a compatible cross-lingual
counterpart. Then, the transfer learning experiments include German as the
source, with English and Dutch as the target languages. We implement a zero-
shot transfer learning by applying the German NER model directly to English
and Dutch transcripts without any prior modifications specific to these
languages. This was followed by a more granular approach involving transfer
learning with fine-tuning. The German NER model served as the foundational
structure, which was then fine-tuned using a dataset size ($k$) representing
either 10% or 20% of the target language’s training set. The model’s
performance is assessed using the target language’s test set.

The E2E model represents a more integrated transfer learning approach. The
experiments begin with a zero-shot transfer learning from the German E2E
spoken NER system to English and Dutch languages. Subsequently, we fine-tuned
the model, employing a more significant portion (20% and 40%) of the target
language’s training set. The performance on the target language test set is
then evaluated, providing insights into the effectiveness of this
comprehensive fine-tuning process.

##  4 Corpus Overview

Table 1: Statistics of Pseudo-annotated Data Across English, German, and Dutch for Training, Validation, and Test Sets. | Training Set | Validation Set | Test Set
---|---|---|---
|  EN |  DE |  NL |  EN |  DE |  NL |  EN |  DE |  NL
#Sentences |  527K |  526K |  42K |  5K |  5K |  5K |  5K |  5K |  5K
#Tokens |  5.4M |  5M |  0.5M |  46K |  47K |  46K |  47K |  45.6K |  45K
#Tokens as LOC |  211K |  156K |  588 |  1.2K |  1.5K |  241 |  1.3K |  1.3K |  314
#Tokens as ORG |  177K |  83K |  95 |  1K |  729 |  66 |  1.1K |  621 |  88
#Tokens as PER |  216K |  144K |  104 |  1.3K |  1.5K |  123 |  1.4K |  1.2K |  155
#Tokens as O |  4.8M |  4.6M |  408K |  43K |  43K |  45K |  43K |  42.4K |  44.5K
Total Hours |  840 |  839 |  54 |  8 |  8.5 |  6.5 |  8.5 |  8.5 |  7

We relied on data from Common Voice
12.0222https://commonvoice.mozilla.org/en/datasets for our experimental work.
We selectively utilized the validated corpus from each language, implementing
further preprocessing measures, including eliminating duplicate sentences and
utterances with audio standard deviation equal to or below the threshold of
$0.001$. All punctuation was removed except for apostrophes in the English
language during the transcription process. In instances where non-Latin
characters, such as Cyrillic or Brahmic, were encountered, we transcribed them
into Latin. Upon completion of the preprocessing stage, the NER model for each
language, constructed on the foundation of XLM-RL with the language-specific
CoNLL dataset [22, 23], was utilized to generate a pseudo-annotation for each
language. Additionally, all transcriptions were uniformly converted to
lowercase. Statistical information for training, development, and testing set
are compiled in Table 1, which highlights the number of sentences, tokens, and
their durations across three languages. We can see different statistics when
looking at the specific tokens in the LOC, ORG, and PER categories. The total
hours of data also reflect a similar distribution. These statistics emphasize
the extensive nature of the dataset and highlight the differences in data
across languages, underlining the importance of considering these variations
when training multilingual models.

Table 2 presents an insightful view into the overlap of entities across
different languages. We observe a perfect overlap within the same language for
training sets, as indicated by 100.0% for English, German and Dutch. However,
when it comes to intra-language comparison, we can observe considerable
overlap between English and German in the training set, 36.5%, whereas the
overlap between English and Dutch is considerably less, only 0.1%.

Table 2: Entity Overlaps in Percentages across Languages. | Train
---|---
|  |  EN |  DE |  NL
Train |  EN |  100.0 |  24.7 |  0.1
|  DE |  36.5 |  100.0 |  0.2
|  NL |  52.4 |  58.0 |  100.0
Test |  EN |  79.6 |  64.0 |  1.7
|  DE |  58.4 |  84.3 |  2.1
|  NL |  64.3 |  68.1 |  27.0

##  5 Evaluation Metrics

We use different metrics to appraise the efficacy of our spoken NER systems.
Alongside the conventional Word Error Rate (WER), the default metric in
evaluating the accuracy of ASR models, we also employ the Entity Error Rate
(EER). The EER provides valuable insights into the system’s ability to
accurately transcribe entities, regardless of subsequent tagging
considerations,

$EER=\frac{N_{\text{{Incorrectly Transcribed Entities}}}}{N_{\text{{Total
Entities}}}}$ (1)

where $N_{\text{{Incorrectly Transcribed Entities}}}$ represents the count of
entities that have been inaccurately transcribed by the system, and
$N_{\text{{Total Entities}}}$ refers to the total count of entities present in
the given set. The ratio essentially quantifies the proportion of entities
that were inaccurately transcribed, thereby providing a measure of the
system’s error rate.

To further evaluate the NER system, we incorporated the micro-average F1 score
similar to [11],

$F_{1}=\frac{TP}{TP+\frac{1}{2}(FP+FN)}$ (2)

where true positives (TP), false positives (FP), and false negatives (FN) are
derived from triples composed of entity transcription, entity type, and the
index of occurrence of the corresponding entity. Our implementation leverages
the SeqScore toolkit [24]. A modified version of this toolkit, adapted for our
specific application, has been made publicly accessible for broader use and
future reference333https://github.com/MoncefBenaicha/seqscore.

##  6 Experiments and Results

During various pipeline baseline experiments, we incorporate pre-trained
models sourced from Hugging Face’s library. For ASR, we leverage the
Wav2Vec2-XLS-R-300M pre-trained
model444https://hf.co/facebook/wav2vec2-xls-r-300m. Simultaneously, the XLM-RL
pre-trained language model555https://hf.co/xlm-roberta-large is employed for
NER. For the ASR model’s fine-tuning process, we prefer AdamW as the optimizer
[25], with $betas=(0.9,0.999)$ and $eps=10^{-8}$. The learning rate is
dynamically controlled by a linear scheduler. Warm-up steps are set to
constitute $1/3$ of total training steps. During this warm-up period, the
learning rate incrementally increases from 0 to $10^{-4}$, then linearly
decreases until the end of the training process. Apart from the freezing of
the feature encoder of the Wav2Vec2-based model, no further modifications are
introduced. In terms of the NER model’s fine-tuning process, the same
optimizer parameters are maintained. However, the maximum learning rate
defined in the scheduler is adjusted to $2\times 10^{-5}$. Lastly, for the E2E
spoken NER model, we adapt the same hyperparameters as those used for the ASR
model within the pipeline system.

###  6.1 Baseline Results

Note that EER and the F1 score have a negative correlation for all three
languages. This phenomenon is expected, given that in case the ASR system
wasn’t able to correctly transcribe the entity, this is counted as an error
for the F1 score even if the entity boundary and category are correct.
Focusing on the German E2E baseline results, we observe that out of a total of
5000 test utterances, 2928 are accurately transcribed. This constitutes
approximately 58.6% of the test set, where nearly 94% of utterances are
correctly labeled. Utilizing a similar approach for the incorrectly
transcribed utterances, we find that in approximately 51.3% of these
utterances, the model is still able to label the entities correctly.
Therefore, the E2E model successfully provided accurate labeling on
approximately 76% of total utterances.

Table 3: Comparative Performance of Baseline Models across Different Configurations and Languages. Lang | E2E | ASR | NER | WER | EER | F1
---|---|---|---|---|---|---
EN | no | no | yes | N/A | N/A | 80.3
no | yes | yes | 16.7 | 48.0 | 40.7
yes | no | no | 16.5 | 46.0 | 41.8
DE | no | no | yes | N/A | N/A | 87.2
no | yes | yes | 9.4 | 29.0 | 61.1
yes | no | no | 9.1 | 27.0 | 61.6
NL | no | no | yes | N/A | N/A | 85.6
no | yes | yes | 9.3 | 49.0 | 40.0
yes | no | no | 9.2 | 47.0 | 37.4

The same pattern is also observed in the pipeline system, with a marginally
greater performance exhibited by the E2E model. The latter is able to
correctly tag entities even with incorrect transcriptions, demonstrating an
improvement of about 2% over the pipeline system. Parallel analyses conducted
on the English and Dutch reveal similar patterns to those in German. A notable
exception is observed in Dutch, where up to 99% of entities are correctly
identified given accurate transcription. It can be observed that the E2E
spoken NER system is able to marginally enhance the WER, EER, and
consequently, the F1 score. The only exception to this trend exists in Dutch,
which is expected due to the smaller volume of available training data, as
compared to English and German.

###  6.2 Transfer Learning Results

Table 4 and Table 5 present the transfer learning outcomes from German-to-
English and German-to-Dutch mirroring those of the baseline pipeline.
Conversely, for the Dutch, identical results are observed using zero-shot
transfer learning, with a notable improvement of 2% upon the inclusion of 40%
of Dutch training data. When examining the E2E system, the German-to-Dutch
transfer yields superior results compared to the German-to-English one. A
possible explanation for this phenomenon can be found in Table 2, which
reveals a significantly higher overlap between German and Dutch entities as
compared to German and English.

Table 4: Performance of Pipeline and E2E Models with German to English Transfer Learning. System | Transfer Learning | WER | EER | F1
---|---|---|---|---
Source to Target | $k$
Pipeline | N/A | N/A | 16.7 | 48.0 | 40.7
DE to EN | 0% | 16.7 | 48.0 | 38.5
20% | 16.7 | 48.0 | 39.6
40% | 16.7 | 48.0 | 40.0
E2E | N/A | N/A | 16.5 | 46.0 | 41.8
DE to EN | 0% | 52.5 | 66.0 | 20.8
20% | 21.8 | 53.0 | 35.8
40% | 22.9 | 54.0 | 35.8

As seen in Table 5, fine-tuning a German E2E system using 40% of the Dutch
training set results in a considerable enhancement of the Dutch E2E system
performance by approximately 7%, and around 4% when compared to the Dutch
pipeline system. Upon investigating the source of this improvement, we
identify that the F1 score for both the PER and LOC classes exhibits an
enhancement of 10% when compared to the Dutch E2E baseline.

Table 5: Performance of Pipeline and E2E Models with German to Dutch Transfer Learning. System | Transfer Learning | WER | EER | F1
---|---|---|---|---
Source to Target | $k$
Pipeline | N/A | N/A | 9.3 | 49.0 | 40.0
DE to NL | 0% | 9.3 | 49.0 | 40.2
20% | 9.3 | 49.0 | 40.3
40% | 9.3 | 49.0 | 42.0
E2E | N/A | N/A | 9.2 | 47.0 | 37.4
DE to NL | 0% | 78.0 | 67.0 | 24.0
20% | 11.8 | 42.0 | 44.4
40% | 10.7 | 40.0 | 44.3

##  7 Conclusions

In this paper, we investigate spoken NER with a particular emphasis on cross-
lingual transfer learning. Our experiments encompass distinct methodologies
grounded in both pipeline and E2E approaches. Clearly, we conclude that the
E2E spoken NER demonstrates superior performance in comparison to the pipeline
approach. This observation is still valid when considering diverse evaluation
metrics as well as the total parameter count between the two systems. Despite
this, the pipeline approach can be used in terms of its practical viability
and flexible application of various ASR or NER components. Our exploration
reveals that using a German NER model without any fine-tuning on a Dutch or
English inside the pipeline, E2E spoken NER gives similar or better results
than using a NER model trained in the pipeline’s corresponding language. Our
research demonstrates the feasibility of transfer learning with E2E spoken NER
systems, which exhibit promising results, even surpassing those observed in
the pipeline system transfer learning. Notably, one critical finding is the
increased performance of the E2E model, which is able to correctly tag
entities even with inaccurate transcriptions, exhibiting a slight enhancement
over the pipeline system.

As future work, there exist several promising investigations. One such
direction involves the refinement of the objective function to enable the ASR
model to give increased attention to specific tokens of interest within the
transcription. Another area worth exploring entails the study of spoken NER
within the context of a multi-lingual model. Such a model, capable of handling
a broad spectrum of languages and dialects, could yield significant
contributions to the field. Lastly, the creation and utilization of human-
annotated datasets, with consistent entity annotation across diverse
languages, emerge as a necessity. Such resources would undoubtedly facilitate
a more robust and reliable evaluation, thereby advancing the field of spoken
NER in a comprehensive and impactful manner.

55footnotetext: This work was supported by the European Union’s Horizon 2020
Research and Innovation Program under Grant Agreement No. 957017 SELMA
(https://selma-project.eu).

## References

  * [1] D. Nadeau and S. Sekine, “A survey of named entity recognition and classification,” Lingvisticae Investigationes, vol. 30, no. 1, pp. 3–26, 2007.
  * [2] N. Tomashenko, A. Caubrière, Y. Estève, A. Laurent, and E. Morin, “Recent advances in end-to-end spoken language understanding,” in International Conference on Statistical Language and Speech Processing, pp. 44–55, Springer, 2019.
  * [3] D. Porjazovski, J. Leinonen, and M. Kurimo, “Attention-based end-to-end named entity recognition from speech,” in International Conference on Text, Speech, and Dialogue, pp. 469–480, Springer, 2021.
  * [4] B. Chen, G. Xu, X. Wang, P. Xie, M. Zhang, and F. Huang, “AISHELL-NER: Named entity recognition from Chinese speech,” in International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 8352–8356, IEEE, 2022\.
  * [5] S. Ghannay, A. Caubrière, Y. Estève, N. Camelin, E. Simonnet, A. Laurent, and E. Morin, “End-to-end named entity and semantic concept extraction from speech,” in Spoken Language Technology Workshop (SLT), pp. 692–699, IEEE, 2018.
  * [6] P. Haghani, A. Narayanan, M. Bacchiani, G. Chuang, N. Gaur, P. Moreno, R. Prabhavalkar, Z. Qu, and A. Waters, “From audio to semantics: Approaches to end-to-end spoken language understanding,” in Spoken Language Technology Workshop (SLT), pp. 720–726, IEEE, 2018.
  * [7] D. Serdyuk, Y. Wang, C. Fuegen, A. Kumar, B. Liu, and Y. Bengio, “Towards end-to-end spoken language understanding,” in International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 5754–5758, IEEE, 2018\.
  * [8] S. Mdhaffar, J. Duret, T. Parcollet, and Y. Estève, “End-to-end model for named entity recognition from speech without paired training data,” in Annual Conference of the International Speech Communication Association (Interspeech), 2022.
  * [9] A. Caubrière, S. Rosset, Y. Estève, A. Laurent, and E. Morin, “Where are we in named entity recognition from speech?,” in Proceedings of the Language Resources and Evaluation Conference, pp. 4514–4520, 2020.
  * [10] M. A. B. Jannet, O. Galibert, M. Adda-Decker, and S. Rosset, “How to evaluate ASR output for named entity recognition?,” in Annual Conference of the International Speech Communication Association (Interspeech), 2015.
  * [11] A. Pasad, F. Wu, S. Shon, K. Livescu, and K. Han, “On the use of external data for spoken named entity recognition,” in Proceedings of the North American Chapter of the Association for Computational Linguistics, pp. 724–737, 2022.
  * [12] N. Goyal, J. Du, M. Ott, G. Anantharaman, and A. Conneau, “Larger-scale transformers for multilingual masked language modeling,” 2021.
  * [13] M. A. B. Jannet, O. Galibert, M. Adda-Decker, and S. Rosset, “Investigating the effect of asr tuning on named entity recognition,” in Annual Conference of the International Speech Communication Association (Interspeech), pp. 2486–2490, 2017.
  * [14] I. Cohn, I. Laish, G. Beryozkin, G. Li, I. Shafran, I. Szpektor, T. Hartman, A. Hassidim, and Y. Matias, “Audio de-identification-a new entity recognition task,” in Proceedings of the North American Chapter of the Association for Computational Linguistics, pp. 197–204, 2019.
  * [15] M. Hatmi, C. Jacquin, E. Morin, and S. Meignier, “Incorporating named entity recognition into the speech transcription process,” in Annual Conference of the International Speech Communication Association (Interspeech), 2013.
  * [16] D. Amodei, S. Ananthanarayanan, R. Anubhai, J. Bai, E. Battenberg, C. Case, J. Casper, B. Catanzaro, Q. Cheng, G. Chen, et al., “Deep Speech 2: End-to-end speech recognition in english and mandarin,” in International conference on machine learning (ICML), pp. 173–182, 2016.
  * [17] A. Graves, S. Fernández, F. Gomez, and J. Schmidhuber, “Connectionist temporal classification: labelling unsegmented sequence data with recurrent neural networks,” in International conference on machine learning (ICML), pp. 369–376, 2006.
  * [18] H. Yadav, S. Ghosh, Y. Yu, and R. R. Shah, “End-to-end named entity recognition from English speech,” Annual Conference of the International Speech Communication Association (Interspeech), pp. 4268–4272, 2020.
  * [19] S. Shon, A. Pasad, F. Wu, P. Brusco, Y. Artzi, K. Livescu, and K. J. Han, “Slue: New benchmark tasks for spoken language understanding evaluation on natural speech,” in International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 7927–7931, IEEE, 2022.
  * [20] S. Vajjala and R. Balasubramaniam, “What do we really know about state of the art ner?,” in Proceedings of the Language Resources and Evaluation Conference, pp. 5983–5993, 2022.
  * [21] A. Babu, C. Wang, A. Tjandra, K. Lakhotia, Q. Xu, N. Goyal, K. Singh, P. von Platen, Y. Saraf, J. Pino, A. Baevski, A. Conneau, and M. Auli, “Xls-r: Self-supervised cross-lingual speech representation learning at scale,” 2021\.
  * [22] E. F. T. K. Sang, “Introduction to the conll-2002 shared task: Language-independent named entity recognition,” 2002.
  * [23] E. F. T. K. Sang and F. D. Meulder, “Introduction to the conll-2003 shared task: Language-independent named entity recognition,” 2003.
  * [24] C. Palen-Michel, N. Holley, and C. Lignos, “SeqScore: Addressing barriers to reproducible named entity recognition evaluation,” in Proceedings of the 2nd Workshop on Evaluation and Comparison of NLP Systems, (Punta Cana, Dominican Republic), pp. 40–50, Association for Computational Linguistics, Nov. 2021.
  * [25] I. Loshchilov and F. Hutter, “Decoupled weight decay regularization,” 2019.