# Implications of the NANOGrav results for primordial black holes and Hubble
tension

M. Bousder1, A.Riadsolh3, A. El Fatimy4,5,
M. El Belkacemi3 and H. Ez-Zahraouy1,2
1Laboratory of Condensend Matter and Interdisplinary Sciences, Department of
physics,
Faculty of Sciences, Mohammed V University in Rabat, Morocco
2CNRST Labeled Research Unit (URL-CNRST), Morocco
3Laboratory of Conception and Systems (Electronics, Signals and Informatics)
Faculty of Sciences, Mohammed V University in Rabat, Morocco
4Central European Institute od Technology,
CEITEC BUT, Purkyňova 656/123, 61200 Brno, Czech Republic
5Departement of Physics, Université Mohammed VI Polytechnique, Ben Guerir
43150, Morocco
<EMAIL_ADDRESS>
###### Abstract

The purpose of this work is to investigate the formation and evaporation of
the primordial black holes in the inflationary scenarios. Thermodynamic
parameters such as mass, temperature and entropy are expressed in terms of
NANOGrav frequency. By numerical calculations we show that the constraint on
the mass range $10^{-5}kg-10^{50}kg$ is well confirmed. We discuss the
relation between the redshift and the probability for gravitational wave
source populations. A new parameter associated with the frequency and Hubble
rate is presented, by which for the spectral index $n_{s}\approx 0.996$ and
the Hubble constant $H_{0}\approx 67.27km.s^{-1}.Mpc^{-1}$, the effective
Hubble constant is calculated to be $H_{eff,0}\approx 73.24km.s^{-1}.Mpc^{-1}$
which is compatible with the observational data. We make a comparison between
the Hubble tension and the primordial perturbations and the expression of the
mass loss rate, chemical potential and central charge needed to describe the
Hawking evaporation will be established.

##  1 Introduction

There has been a recent surge of interest in the 15 yr of pulsar timing data
of stochastic gravitational wave (GW) signal [1, 2], published by the North
American Nanohertz Observatory for Gravitational Waves (NANOGrav)
collaboration, In [3], the authors have been working on 12.5 yrs of pulsar
timing data. Recently, several gravitational wave models have been proposed
[4, 5], in an attempt to explain the NANOGrav results. The observed signal
could be attributed to a stochastic gravitational wave background originating
from primordial black holes (PBHs) formed during inflation [6]. Conversely,
alternative studies propose cosmic strings as the source of these wave
phenomena. [9]. The potential GW signal can be generated by the power-law of
abundance $\Omega_{GW}\propto f^{\zeta}$. The NANOGrav observes the exponent
$\zeta\in\left(-1.5,0.5\right)$ at a frequency $f=5.5nHz$. The LIGO
collaboration and Virgo collaboration [10, 11] detect the GWs from the black
holes. The PBHs can be taken as dark matter candidates and can also have
formed in the early Universe [12], in this regard, we assumed that the dark
matter halo is composed of these PBHs in the edges of the galaxies. From the
constraints on PBHs, the mass range of PBH is $10^{-5}g-10^{50}g$ [13]. The
PBH mass that can form dark matter is around $10^{20}g$ [14]. The PBHs formed
before the end of the radiation-dominated era and are not subject to the big
bang nucleosynthesis (BBN), such that more than 5$\%$ of the critical density
forms the baryons [15]. Inflation predicted a stochastic background of GWs
over a broad range of frequencies, which have a direct relationship with
cosmic microwave background (CMB) measurements [7]. Also, they find a
constraint on the spectral index as $n_{s}>0.9$, which is in good agreement
with the the Planck 2018 results ($n_{s}^{0}\approx 0.965\pm 0.004$) [8].
On the other hand, the Hubble tension is one of the most intriguing problems
in cosmology today. Local measurements of the current expansion rate of the
Universe based on Type Ia Supernovae (SNeIa) observations from the Hubble
Space Telescope (HST), give values that are grouped around
$H_{0}\simeq\left(73.24\pm 1.74\right)km.s^{-1}.Mpc^{-1}$ [16]. While, the
value of the Hubble constant using the CMB measurements comes out at around
$H_{0}\simeq\left(67.27\pm 0.66\right)km.s^{-1}.Mpc^{-1}$ [8], and the
deviation from local measurements is now 4$\sigma$ or more. Among of the more
successful solutions to the Hubble tension is the addition of an early dark
energy component [17] or primordial magnetic fields [18]. There are also
several works that have evaluated this problem by constraints. There has been
lots of discussions about new physics behind this tension in the form of
modifications or additions of energetic components to the standard
cosmological model [19, 20]. Recently, M. R. Visser [21] proposed a novel
holographic central charge $C\sim L^{d-2}$, which has attracted great
attention. He finds that the Euler equation is dual to the generalized Smarr
formula for black holes in the presence of a negative cosmological constant.
This theory sheds light on the holographic and thermodynamic aspects of black
holes. In comparison, authors in [22, 23] have been proposing a restricted
phase space approach, in which the pressure $P$ and the volume $V$ are absent
in the thermodynamic description for black holes. While in [24] through the
Holographic Principle, the limit values of the mass density of PBHs formed in
the early universe can be obtained. Motivated by these and specifically by the
work [6], the purpose of the present letter is to introduce an interpretation
of the Hubble tension from NANOGrav signal in comparison with the PBHs
formation and rotation. We present a new paradigm for explaining the
stochastic gravitational wave background from the PBHs. We also aim to connect
the horizon mass with comoving frequency of the perturbation. The goal of this
paper is to show that there is more information about the early universe
hidden in the thermodynamic properties of PBHs.

This paper is organized as follows. In Sec. 2, we analyze the stochastic
gravitational wave background from the PBHs and will establish relations for
thermodynamical parameters of PBHs. In Sec. 3, we calculate the slow roll
parameter and spectral index in inflationary scenarios for PBH formation.
Also, we explore the Hubble tension from the spectral index. In Sec. 4, the
formation of PBHs and their subsequent evaporation and their relationship with
a new parameter associated with the Hubble rate are studied. In Sec. 5, the
gauge/gravity duality aspect of the PBHs by the chemical potential and central
charge are discussed and we conclude our findings in Sec. 6.

##  2 Method

In order to study the stochastic gravitational wave background from the PBHs
generated during inflation, it is necessary to understand the mechanism by
which there is an accelerated expansion of the Universe. We work in the
framework with units $c\neq G\neq\hbar\neq 1$. The metric of a spatially flat
homogeneous and isotropic universe in the Robertson Walker spacetime with
perturbations is given by [34]:

$ds^{2}=\left(ic^{2}dt\right)^{2}+a^{2}(t)\left(\delta_{ij}+h_{ij}\right)dx^{i}dx^{j},$
(2.1)

where $h_{ij}$ is a $3\times 3$ symmetric matrix of the perturbations of
spacetime, $(i,j=1,2,3)$ and $a\left(t\right)$ is the scale factor that
describes the relative expansion of the Universe. We define the Hubble
parameter as $H=\dot{a}/a$ with $\dot{a}$ denoting derivative of $a$ with
respect to time. In the expanding Robertson Walker space-time the scale factor
$a\left(t\right)$ is related to the wave length $\lambda$ and the comoving
wave number $k$ by [36, 35]:

$k\lambda=2\pi a.$ (2.2)

To make progress, let us introduce the frequency $f=kc/2\pi$, which is
associated with the stochastic gravitational wave background. From Eq. (2.2)
we have

$a(f)=2\pi f\left(\frac{\lambda}{2\pi c}\right).$ (2.3)

Using this and from the complex conformal transformation [36, 37], we assume
that the scale factor associated with gravitational wave background has the
following form

$a\left(f\right)\equiv a_{0}e^{2\pi f\tau}.$ (2.4)

where $a_{0}$ is a constant and $\tau$ is the Euclidean time (Wick rotation
$it\rightarrow\tau$) [38, 39]. One should note that to determine the
temperature associated with respect to e-folds, one must first determine the
entropy of the system in terms of GWs frequency. The entropy $s$ is expressed
as a function of the number of e-folds
$N=\ln\left(a\left(t\right)/a_{0}\right)$ remaining to the end of the
inflation period [53]. The number of e-folds measures the amount of inflation
that occurs. Note that the entropy of the PBH can be modeled by a logarithmic
function [52, 53],

$s=k_{B}\ln\frac{a\left(f\right)}{a_{0}}=k_{B}N,$ (2.5)

where $N$ is the number of e-folds between the horizon exit and the end of
inflation. The quantity $a\left(f\right)/a_{0}$ at the moment of the PBH
formation must be greater than $1$. According to the positivity of $s$, we
find that $a\left(f\right)>a_{0}$. At the end of inflation ($N\approx 0$), the
entropy vanishes. The Wick rotation $it\rightarrow\tau$ [38] links statistical
physics and quantum mechanics by replacing the inverse temperature $1/k_{B}T$
with a periodicity of imaginary time $\tau/\hbar$. So, from Eq. (2.4,2.5) the
entropy is given by

$s\left(f\right)=\frac{2\pi\hbar f}{T},$ (2.6)

This entropy is expressed as a function of $f$. Using the Boltzmann type of
entropy
$s=-k_{B}\ln\mathcal{P}\left(t\right)=-k_{B}\ln\frac{a_{0}}{a\left(f\right)}$,
where

$\mathcal{P}\left(f\right)=e^{-2\pi f\tau}\leq 1,$ (2.7)

is the probability of the redshift $z$ and is a solution of an evolution
Fokker-Planck equation, leading to its current distribution. The probability
distribution is taken to have the non-Gaussian form. This probability is in
good agreement with the estimation of the sensitive volume [42], for GW source
populations at redshift. For the power law form of the scale factor
$a\left(t\right)=a_{0}t^{q}$, the probability distribution reduces to
$\mathcal{P}\left(t\right)=t^{-q}\leq 1$. In the dark-energy-dominated era
($a\left(t\right)=a_{0}e^{H_{0}t}$), the evolution of the probability is
$\mathcal{P}\left(t\right)=e^{-H_{0}t}\leq 1$ with $2\pi f\tau=H_{0}t$.
Considering $t\rightarrow\infty$, the probability distribution becomes zero,
i.e. the boundary condition $\mathcal{P}\left(t\right)=0$. The temperature $T$
determines how fast probability decays with energy.
In addition, from Eq. (2.5) and Eq. (2.6) it is evident that

$T=\frac{2\pi\hbar f}{k_{B}N},$ (2.8)

where $T$ is the temperature of the system, which depends on the frequency $f$
of the stochastic gravitational wave background. At the end of inflation
($N\approx 0$), the temperature tends to infinity. Eq. (2.8) corresponds with
Wien’s displacement law. The temperature fluctuations are projected as
anisotropies on the cosmic microwave background (CMB) sky [40] by the number
of e-folds $N$. The temperature $T$ grows if the conformal time grows quickly.
The radiation energy density today is given by $\Omega_{r}h^{2}=4.18\times
10^{-5}$ for $T_{CMB}=2.725K$ [41]. From Eq. (2.5) and Eq. (2.8) we note that
$T=\frac{2\pi\hbar f}{s}\propto\frac{1}{M}$ with $M$ being PBH mass, which is
in good agreement with the temperature of black holes radiation [57]. The
black holes radiate thermally with temperature $T=\frac{\hbar c^{3}}{8\pi
GMk_{B}}$ [57], which leads to the PBH mass:

$M\left(f\right)=\frac{c^{3}}{G}\frac{N}{16\pi^{2}f}.$ (2.9)

We notice that $M\propto 1/f$, which is compatible with the result found in
[6]. Since the ratio $N/f$ has the dimension of time, which corresponds to the
order of PBHs mass $M\sim\frac{c^{3}}{G}t$ [13]. On the other hand, the PBHs
with the mass less than $10^{15}g$ would have completely been evaporated. Non-
evaporating PBHs are characterized by $M>10^{15}g$. Also, the mass range of
PBH is $10^{-5}g-10^{50}g$ [13]. The e-folds before the end of inflation for
the formation of the visible part of the Universe is $N=60$ [45, 46, 47]. The
PBHs that have been produced in the early universe are referred to as the
production of the galaxies [43]. By using Eq. (2.9), we show that $f$ is the
frequency of the waves emitted by the PBHs. One interesting point of
discussion could then be the following. First, we study the cases between the
moments before the end of inflation ($N=60$) and at the end of inflation
($N=0$). As is very clear from the table 1, we notice that the relation (2.9)
well confirms the constraints on the mass range of PBH [13] in addition to
obtaining the value $f=5.5nHz$ for the frequency which is detected by the
NANOGrav collaboration.

$N$ | $M\left(g\right)$ | $f\left(nHz\right)$
---|---|---
$10^{-47}$ | $10^{-2}$ | $5.500$
$2.15\times 10^{-30}$ | $10^{15}$ | $5.500$
$10$ | $4.65\times 10^{45}$ | $5.507$
$20$ | $9.30\times 10^{45}$ | $5.507$
$30$ | $1.40\times 10^{46}$ | $5.487$
$40$ | $1.86\times 10^{46}$ | $5.507$
$50$ | $2.3\times 10^{46}$ | $5.567$
$60$ | $2.75\times 10^{46}$ | $5.587$
$60$ | $2.78\times 10^{46}$ | $5.527$
$60$ | $2.78\times 10^{47}$ | $1.536$
Table 1: Numerical estimate of the values of the PBH mass, according to the
frequency $f$ and the interval of $N$ between $10^{-47}$ and $60$. Such as
$f=2.561\times 10^{33}\frac{N}{M}.$

Recently, the NANOGrav collaboration published an analysis of 12.5 yrs of
pulsar timing data of stochastic gravitational wave signal [3]. This signal
may be interpreted as a stochastic gravitational wave background from the
primordial black holes (PBHs) generated during inflation [6]. The potential GW
signal can be generated by the power-law of abundance $\Omega_{GW}\propto
f^{\zeta}$. The NANOGrav observes the exponent $\zeta\in\left(-1.5,0.5\right)$
at a frequency $f=5.5nHz$. We assume that $f$ is the frequency of the
stochastic gravitational waves which are emitted in the early Universe. During
phase transitions in the early universe at frequency $f=5.5nHz$, from Eq.
(2.8) we obtain

$N=\frac{T_{0}}{T},$ (2.10)

where $T_{0}=2.64\times 10^{-19}K$. Temperature $T$ must be in the order of
$T_{0}$, which shows that the PBHs are very cold, which is in good agreement
with the cold dark matter (CDM) model. The particles move slowly compared to
the speed of light in the CDM. The scale factor describes the relative
expansion of the Universe $a\left(t\right)$. The scale factor
$a\left(t\right)$ changes according to the chronology of the Universe. Next,
we assume that $H=q/t$, then, the scale factor has the specific form
$a\left(t\right)=a_{0}t^{q}$. In this case, we recall that in the radiation-
dominated era, we have $a\left(t\right)\propto t^{1/2}$. On the other hand,
during the matter-dominated era, the scale factor behaves as
$a\left(t\right)\propto t^{2/3}$. Note that during the dark-energy-dominated
era, the evolution of the scale factor is $a\left(t\right)=a_{0}e^{H_{0}t}$.

Era | $a\left(t\right)\propto$ | $\Omega_{GW}\left(f\right)\propto$
---|---|---
Radiation-dominated era | $t^{1/2}$ | $f^{1/2}$
Matter-dominated era | $t^{2/3}$ | $f^{-3/2}$
Table 2: Two types of the stochastic background of gravitational waves
according to the chronology of the Universe.

Let us briefly discuss the different values of $\Omega_{GW}\left(f\right)$
according to the chronology of the Universe in Table 2. If the PBHs are more
dynamic, then the $\Omega_{GW}$ parameter will be more important in the early
Universe, which means that the case $\Omega_{GW}\propto f^{1/2}$ is the one
that describes the radiation-dominated era. While the case $\Omega_{GW}\propto
f^{-3/2}$ describes the matter-dominated era.

##  3 PBHs formation and Hubble tension

The inflationary scenarios for PBHs formation were proposed in [67, 68]. Now,
when we combine Eq. (2.9) with the PBH mass $M\sim\frac{c^{3}}{G}t$ together,
we find

$f=\frac{N}{16\pi^{2}t}.$ (3.1)

Using Eq. (3.1) and $\dot{N}=-H$ with $H=q/t$, we find the slow roll parameter
$\epsilon_{H}=-\dot{H}/H^{2}\ll 1$ for constant $N$:

$\epsilon_{H}=-\frac{1}{16\pi^{2}qf^{2}}\left(N\dot{f}+Hf\right),$ (3.2)

yielding

$\dot{\epsilon}_{H}=\frac{1}{16\pi^{2}qf^{2}}\left[2\left(N\frac{\dot{f}^{2}}{f}+H\dot{f}\right)-\left(N\ddot{f}-\ddot{N}f\right)\right].$
(3.3)

From this model, the slow-roll parameter
$\eta=\epsilon_{H}-\dot{\epsilon}_{H}/2H\epsilon_{H}$ [48] is given by

$\eta=\frac{1}{2H\left(N\dot{f}+Hf\right)}\left[2\left(N\frac{\dot{f}^{2}}{f}+H\dot{f}\right)-\left(N\ddot{f}-\ddot{N}f\right)\right]-\frac{1}{16\pi^{2}qf^{2}}\left(N\dot{f}+Hf\right).$
(3.4)

In the conventional inflationary scenario, the $\epsilon_{H}$ grows as well
$d\epsilon_{H}/dN=2\epsilon_{H}\left(\eta-\epsilon_{H}\right)$. In this case,
the spectral index of perturbations is equal to $n_{s}=1-6\epsilon_{H}+2\eta$
[48]. We can estimate the spectral index by the ratio

$n_{s}=1+\frac{2H_{f}}{H}+\frac{H+NH_{f}}{4\pi^{2}qf}\left[1-\frac{4\pi^{2}q\left(N\ddot{f}-\ddot{N}f\right)}{H\left(H+NH_{f}\right)^{2}}\right],$
(3.5)

One can define a Hubble parameter $H_{f}=\dot{f}/f$ associated with $f$. The
Planck CMB data implies that $n_{s}\approx 0.965\pm 0.004$ [8]. We introduce
the effective expansion rate $H_{eff}$ [74, 75], which is evaluated as
follows:

$H_{eff}=H+NH_{f}.$ (3.6)

The temperature remains nearly unchanged at the time of formation of the PBHs
[50, 51], i.e. from Eq. (2.8) we have
$\ddot{N}=\left(\frac{2\pi\hbar}{k_{B}T}\right)\ddot{f}$. Thus we obtain
$N\ddot{f}=\ddot{N}f$. Recall that the spectral index is given by Eq. (3.5)
and we use the $N\ddot{f}=\ddot{N}f$, we find the relationship between the
spectral index and the current expansion rate:

$n_{s}=1-\frac{2}{N}+\left(\frac{2}{H_{0}N}+\frac{1}{4\pi^{2}qf}\right)H_{eff,0}.$
(3.7)

From Eqs. (2.9) and (3.7), we find $n_{s}\approx
1+\frac{2H_{eff}}{H_{0}N}-\frac{2}{N}+\frac{4GM}{c^{3}}\frac{H_{eff}}{qN}.$ We
note the presence in this expression the presence of the Schwarzschild radius.
We notice that $H_{eff}$ includes the number of e-folds. In other words, the
term $H_{f}$ affects the evolution of $H_{eff}$. This behavior is very similar
to the one discussed for the Hubble tension [69, 70]. It is useful to
calculate the value of $H_{f,0}$ from the data we have available,
$AsH_{eff,0}\approx\left(73.24\pm 1.74\right)km.s^{-1}.Mpc^{-1}$ based on type
SNeIa observations from the HST [16] and $H_{0}\approx\left(67.27\pm
0.66\right)km.s^{-1}.Mpc^{-1}$ by CMB measurements, for $N=60$, we can
calculate $H_{f,0}$ to be $H_{f,0}\approx\left(0.0995\pm
2.4\right)km.s^{-1}.Mpc^{-1}$ by Eq. (3.6). Next we want to elaborate on the
validity of Eq. (3.6) by numerical calculation which is given in Table 3.
Recall that the spectral index is given by Eq. (3.5) for
$N\ddot{f}\approx\ddot{N}f.$ According to the numerical results and from Eq.
(3.6), we have $H_{0}N\ll 8\pi^{2}qf$. So,

$n_{s}=1+\frac{2}{N}\left(\frac{H_{eff,0}}{H_{0}}-1\right).$ (3.8)

In the limit $H_{eff,0}\approx H_{0}$, one gets $n_{s}\approx 1$.

Measurements | $H_{eff,0}(km.s^{-1}.Mpc^{-1})$ | $H_{0}(km.s^{-1}.Mpc^{-1})$ | $N$ | $n_{s}$ | $n_{s}^{moy}$
---|---|---|---|---|---
CMB with Planck [8] | $67.27\pm 0.60$ | $67.27$ | $60$ | $1_{-0.0002}^{+0.0002}$ | $1$
CMB without Planck [25] | $68.8\pm 1.5$ | $67.27$ | $60$ | $1.0015$ | $1.0015$
No CMB with BBN [26] | $68.5\pm 2.2$ | $67.27$ | $60$ | $1_{-0.0004}^{+0.0016}$ | $1.0006$
CMB lensing [27] | $70.6_{-5.0}^{+3.7}$ | $67.27$ | $60$ | $1_{-0.0008}^{+0.0034}$ | $1.001$
Cepheids - SNIa [28] | $73.2\pm 1.3$ | $67.27$ | $60$ | $1_{-0.0022}^{+0.0035}$ | $1.002$
TRGB -SNIa [29] | $72.1\pm 2.0$ | $67.27$ | $60$ | $1_{-0.0014}^{+0.0033}$ | $1.002$
Miras -SNIa [30] | $73.2\pm 4.0$ | $67.27$ | $60$ | $1_{-0.0009}^{+0.0049}$ | $1.002$
Masers [31] | $73.9\pm 3.0$ | $67.27$ | $60$ | $1_{-0.0017}^{+0.0047}$ | $1.003$
HII galaxies [32] | $71.0\pm 3.5$ | $67.27$ | $60$ | $1_{-0.0001}^{+0.0035}$ | $1.001$
GW related [33] | $73.4_{-10.7}^{+6.9}$ | $67.27$ | $60$ | $1_{-0.0022}^{+0.0064}$ | $1.003$
Table 3: Numerical estimate of the values of $n_{s}$ from Eq. (3.8) with
Hubble constant $H_{eff,0}$ through direct and indirect measurements [49]. We
take $H_{0}=67.27km.s^{-1}.Mpc^{-1}$ as the reference. Note that the values of
$n_{s}^{moy}$ represent the average value of $n_{s}$.

In the radiation-dominated era ($q=\frac{1}{2}$, $N=60$) at the frequency
$f=5.5nHz$, the current effective rate is

$H_{eff,0}=30\left(n_{s}-0.96\right)H_{0}.$ (3.9)

For $H_{0}\approx 67.27km.s^{-1}.Mpc^{-1}$ and $n_{s}^{\ast}\approx 0.996$, we
find $H_{eff,0}\approx 73.24km.s^{-1}.Mpc^{-1}$. This spectral index approves
the constraint $n_{s}>0.9$ [7]. While Blais et al. [71] find that the mass
fluctuation is reduced by $34\%$ for scale-free power law primordial spectra,
and the spectral index is in the range $1\leq n_{s}\leq 1.3$. For that we
widen the domain of existence by $n_{s}$: $0.996\leq n_{s}\leq 1.3$. The
critical value of the spectral index is $n_{s}^{0}=0.96$ which is compatible
with the Planck 2018 results ($n_{s}^{0}\approx 0.965\pm 0.004$) [8], also
with $n_{s}^{0}\approx 0.9569$ found in the inflationary models of PBH with an
inflection point [72, 73].

##  4 PBHs formation and evaporation

In this section, we are interested in studying the formation of PBHs and their
subsequent evaporation. The initial PBH mass $M$ is close to the Hubble
horizon mass $M_{H}$ and therefore given by $M=\gamma M_{H}$ [13] where
$\gamma$ is a factor that depends on the gravitational collapse nature of the
black hole and the radiation quantity in the early Univers. The Hubble horizon
mass is

$M_{H}=\frac{4\pi R_{H}^{3}}{3}\rho,$ (4.1)

where $R_{H}$ is the black hole horizon radius and $V=\frac{4\pi
R_{H}^{3}}{3}$ is the thermodynamic volume. The mass fraction of PBHs at the
time of formation is

$\beta=\frac{\rho_{PBH,i}}{\rho_{r,i}},$ (4.2)

where $\rho_{PBH,i}$ is the PBHs energy density and $\rho_{r,i}$ is the
background radiation density at the PBH formation epoch (index $i$). The
current density parameter $\Omega_{PBH}$ of the PBHs is related to the initial
collapse fraction $\beta$ by

$\Omega_{PBH}=\beta\left(1+z\right)\Omega_{CMB},$ (4.3)

where $\Omega_{CMB}\approx 5\times 10^{-5}$ is the density parameter of the
CMB and $z$ is the redshift. The $(1+z)$ factor can be understood as arising
for the radiation density scales as $(1+z)^{4}$, whereas the PBH density
scales as $(1+z)^{3}$. The PBHs have a mass of order $M\sim\frac{tc^{3}}{G}$.
We introduce the fraction $f_{PBH}(M)$ of the halo in PBHs for
$\Omega_{CDM}\approx 0.264$ [8] by
$f_{PBH}\left(M\right)=\frac{\Omega_{PBH}}{\Omega_{CDM}}\approx
3.79\Omega_{PBH}$. A constraint on $\Omega_{PBH}$ for $M>10^{15}g$ is
$\Omega_{PBH}<\Omega_{CDM}$, this condition describes non-evaporating PBHs.
For the $\gamma$-ray background limit, we have
$\beta\left(10^{15}g\right)\lesssim 10^{-29}$. The PBHs evaporate at the epoch
of cosmological nucleosynthesis if they are characterized by mass $M\approx
10^{10}g$, temperature $T_{PBH}\approx 1TeV$ and lifetime $\tau\approx
10^{3}s$. These PBHs have an effect of these evaporations on the BBN. Using
the relation $\rho=3sT/4$ [13], the fraction of the Universe’s mass in PBHs at
their formation time is then related to their number density $n_{PBH}$:

$\beta=\frac{4M}{3T}\frac{n_{PBH}}{s}=\frac{4n_{PBH}}{3k_{B}^{2}T_{0}}M.$
(4.4)

The black hole mass $M$ is identified with the thermodynamic enthalpy
$\mathcal{H}$ with $d\mathcal{H}=dM$. The black hole first law reads

$dM=Tds+VdP.$ (4.5)

From Eq. (2.5) and Eq. (4.5) we obtain

$\rho=\frac{dM}{dV}=k_{B}T\frac{dN}{dV}+V\frac{dP}{dV}.$ (4.6)

Using $VdP=-PdV$. As a consequence,

$\rho c^{2}+P=nk_{B}T,$ (4.7)

where $n=dN/dV$ is the density of the e-folds number. Using this relation the
e-folds number density is $n=\frac{\rho c^{2}+P}{k_{B}T}$. Certain phase
transitions lead to a sudden reduction in pressure, which enhances the
formation of PBH [63]. In a stable fluid sphere the equation of state
parameters $\omega\equiv P/\rho c^{2}$ should be positive. The pressure $P$
and density $\rho$ should be positive inside the stars and should satisfy the
energy conditions $\rho c^{2}+P\geq 0$ [56]. For this purpose, the number
density $n$ should be positive. As mentioned before, the PBH mass is given by
the horizon mass when the fluctuations reenter the horizon. With this, the
energy $Mc^{2}=\gamma V\rho c^{2}$ Eq. (4.1) is connected with the Smarr
formula, which now reads

$\gamma^{-1}Mc^{2}+PV=T\frac{ds}{d\ln V},$ (4.8)

and for $n=N/V$, we can recover the standard form of Smarr formula

$\gamma^{-1}Mc^{2}+PV=Ts.$ (4.9)

We notice the difference between the black holes which verified the Smarr
formula [62]. Using the general Smarr formula in 4-dimension brought in [62],
we estimate that $\gamma\sim 2$. The mass loss rate of a PBH is obtained from
Eq. (2.9) as

$\frac{dM}{dt}=-\frac{c^{3}}{16\pi^{2}Gf}\left(H+NH_{f}\right)<0.$ (4.10)

This mass loss is due to Hawking evaporation. Using Eqs. (2.9,4.10), we obtain

$\frac{d\ln M}{dt}=-\left(\frac{H}{N}+H_{f}\right).$ (4.11)

The time evolution of the PBH mass with initial mass $M_{i}$ formed at $t_{i}$
is evaluated by relation

$M(t)=M_{i}e^{-\left(\frac{H}{N}+H_{f}\right)t}.$ (4.12)

According to this expression, we see the total disappearance of PBHs for
$t\rightarrow\infty$. Note that the PBHs would have completely evaporated
until today if their mass is lighter than $M=10^{15}g$. At the end of
inflation ($N=0$), the PBH mass vanishes. Some PBHs generated before inflation
are diluted to negligible density [13]. This agrees with Eqs. (4.12), when
$N=60$, we have $M(t)\neq 0$. The effective Hubble rate can be obtained from
Eq. (4.12):

$H_{eff}=-N\frac{d\ln M}{dt}.$ (4.13)

In order to estimate precisely the fluctuations generated in the Hubble rate,
we note that the PBHs evaporation influences the evolution of $H_{eff}$.
Recall that the spectral index is given by Eq. (3.5) and by Eq. (3.9) we
calculate mass ratio as:

$M(n_{s},t=t_{0})/M_{i}=e^{\frac{0.96-n_{s}}{2}}.$ (4.14)

where $t_{0}=1/H_{0}\approx 4.22\times 10^{17}s$, represents the age of the
Universe. For the initial PBH mass $M_{i}=10^{15}g$ [44] and
$n_{s}^{\ast}\approx 0.996$ we find $M^{\ast}=9.8216\times 10^{14}g$. In Fig.
(1), we plot the ratio $M/M_{i}$ as a function of $n_{s}$.

Figure 1: Mass ratio $M(n_{s})/M_{i}$ (or the chemical potential ratio
$\mu(n_{s})/\mu_{i}$ Eq. (5.4)) as a function of the number of the spectral
index $n_{s}$. The orange area represents the validity interval of $n_{s}$.

For a constant $N$, we introduce the effective scale factor:

$a_{eff}=a_{eff,0}\left(\frac{M(t)}{M_{i}}\right)^{-N}.$ (4.15)

The factor $a_{eff,0}$ represents the value of $a_{eff}$ at the end of
inflation ($N=0$). Surprisingly, the $a_{eff}$ is related to the PBH mass,
which can generate a big difference in the measurements of the Hubble rate in
the primordial universe and by this fact, we propose that PBHs which are
formed from primordial perturbations, generate the Hubble tension.

##  5 Chemical potential and central charge

Recently, in the context of the gauge/gravity duality, M. R. Visser [21]
proposed a novel holographic central charge $C=L^{d-2}/l_{p}^{d-2}$ in which
$l_{p}$ is the Planck length, which has attracted great attention. He finds
that the Euler equation is dual to the generalized Smarr formula for black
holes in the presence of a negative cosmological constant. Thus, we replace
the entropy value that depends on $N$ Eq. (2.5) in Eq.(4.5) which gives

$dM=\frac{k_{B}T}{2N}dN^{2}+VdP.$ (5.1)

We note that N is constant at the time of formation of PBHs which is equal to
60, while during the evolution of PBHs the value of N changes according to
this relation. This expression gives an interpretation of the number of
e-folds as the number of microscopic degrees of freedom for the PBH. This
equation involves an AdS space-time and a dual CFT [54, 55] of the basic
structure of Visser’s framework [21]. We have $C\equiv N^{2}$, which
corresponds to the result found by the conformal symmetry in $SU(N)$ [21].
Thus we can uniformly rewrite Eq. (5.1) as

$dM=\mu dC+VdP.$ (5.2)

The volume $V$ and the chemical potential $\mu$ are the conjugate quantities
of the pressure $P$ and the central charge $C$, respectively. This equation
mixes the volume and boundary description of a black hole because $C$ is the
central boundary charge and $P$ is the volume pressure. In d-dimensional CFT
there are several candidates for the central charge of e-folds $C$. In the
dual CFT, the chemical potential $\mu$ is the conjugate quantity of the
central charge, this corresponds to including $\left(\mu,C\right)$ as a new
pair of conjugate thermodynamic variables [21]. This leads to the following
relation

$\mu\left(T\right)=\frac{k_{B}T}{2N}.$ (5.3)

The solution of black hole chemical potential found in [54] reduces to this
solution of PBHs. We notice that the chemical potential in Eq. (5.3) is linked
to the thermal energy of e-folds (equipartition theorem: $E_{th}=Nk_{B}T/2$).
This leads to $E_{th}=\mu C=\pi\hbar f$, i.e., thermal energy exchange occurs
between PBHs during inflation with the emission of gravitational waves of
frequency $f$. Making use of Eq. (4.12), another suggestive expression for the
chemical potential could be

$\mu\left(t\right)=\mu_{i}e^{-\left(\frac{H}{N}+H_{f}\right)t},$ (5.4)

where $\mu_{i}=\frac{M_{i}q}{N^{2}}$. Similarly, the chemical potential
becomes zero as inflation comes to an end through the condition $N=0$. From
Eq. (4.14) we get $\mu(n_{s})/\mu_{i}=e^{\frac{0.96-n_{s}}{2}}$ (see Fig.
(1)). The variation of the chemical potential with time is plotted in Fig.
(2). Using $n_{s}^{\ast}\approx 0.996$, we find $\mu/\mu_{i}\approx 0.98216$
which is in good agreement with the result of $\mu/\mu_{i}$ for CMB with
Planck [8] in Table 4.

Figure 2: Curves of $M/M_{i}$ or $\mu/\mu_{i}$, with respect to time. For $N=60$. Measurements | $H_{eff,0}(km.s^{-1}.Mpc^{-1})$ | $n_{s}$ | $\mu/\mu_{i}$
---|---|---|---
CMB with Planck [8] | $67.27\pm 0.60$ | $1$ | $0.9801$
CMB without Planck [25] | $68.8\pm 1.5$ | $1.0015$ | $0.9794$
No CMB with BBN [26] | $68.5\pm 2.2$ | $1_{-0.0004}^{+0.0016}\left(+0.0006\right)$ | $0.9799$
CMB lensing [27] | $70.6_{-5.0}^{+3.7}$ | $1_{-0.0008}^{+0.0034}\left(+0.001\right)$ | $0.9797$
Cepheids - SNIa [28] | $73.2\pm 1.3$ | $1_{-0.0022}^{+0.0035}\left(+0.002\right)$ | $0.9792$
TRGB -SNIa [29] | $72.1\pm 2.0$ | $1_{-0.0014}^{+0.0033}\left(+0.002\right)$ | $0.9792$
Miras -SNIa [30] | $73.2\pm 4.0$ | $1_{-0.0009}^{+0.0049}\left(+0.002\right)$ | $0.9792$
Masers [31] | $73.9\pm 3.0$ | $1_{-0.0017}^{+0.0047}\left(+0.003\right)$ | $0.9787$
HII galaxies [32] | $71.0\pm 3.5$ | $1_{-0.0001}^{+0.0035}\left(+0.001\right)$ | $0.9797$
GW related [33] | $73.4_{-10.7}^{+6.9}$ | $1_{-0.0022}^{+0.0064}\left(+0.003\right)$ | $0.9787$
Table 4: Numerical estimate of the minimal variation of $\mu/\mu_{i}$
according to the observations.

In the radiation-dominated era ($q=\frac{1}{2}$) we have
$\mu_{i}=\frac{M_{i}}{2N^{2}}$. While, in the matter-dominated era
($q=\frac{3}{2}$) we have $\mu_{i}=\frac{2M_{i}}{3N^{2}}$. Variation of the
Gibbs free energy of PBHs that is held at constant pressure and temperature is
$dG=\mu dN$. The Gibbs free energy is at its minimum if $dG=\mu dN=0$, as a
consequence, the number $N$ is constant (for example $N=60$). With this, we
discuss the chemical potential conjugate to the number of e-folds.
Alternatively, the Hawking-Page temperature is obtained via the vanishing of
the Gibbs free energy.

##  6 Conclusions

This manuscript offers an explanation for the observed gravitational-wave
background signal detected by pulsar timing arrays. The proposed model
suggests that the stochastic gravitational waves originate from primordial
black holes. We extensively explore the implications of NANOGrav data within
the context of inflationary scenarios and draw comparisons between this model
and primordial perturbations. In our study, we present a frequency relation
associated with the stochastic gravitational wave background. Additionally, we
calculate the mass, temperature, and entropy corresponding to the GW frequency
$f$. By establishing a connection between the horizon mass and the comoving
frequency of the perturbation, we derive a relationship between the PBH mass
$M$ and the GW frequency $f$ as $M\propto 1/f$. Through analytical and
numerical analyses, we demonstrate that the frequency $f$ aligns well with the
constraints on the mass range of $10^{-5}kg-10^{50}kg$. Furthermore, we derive
a probability distribution relation as a function of frequency. The obtained
relation aligns well with the estimated sensitive volume for GW source
populations at redshift, indicating a strong agreement. Moreover, we have
established a correlation between entropy and the number of e-folds. Through
the expression of temperature in terms of $N$ (number of e-folds) and
frequency, we have discovered that primordial black holes (PBHs) exhibit
extremely low temperatures, a characteristic that supports the cold dark
matter model. Additionally, we have introduced a new parameter derived from
this frequency. The presence of this parameter signifies the rate of mass loss
for an evaporating black hole. It is possible that this parameter could offer
a resolution to the problem of the Hubble tension. From this perspective, the
effective Hubble constant $H_{eff,0}$ can be expressed in relation to the
number of e-folds $N$ and the Hubble constant $H_{0}\approx
67.27km.s^{-1}.Mpc^{-1}$, and subsequently in terms of the spectral index
$n_{s}$. Numerical calculation led to values: $n_{s}\approx 0.996$ and
$H_{eff,0}\approx 73.24km.s^{-1}.Mpc^{-1}$, which is in good agreement with
the observational data. Subsequently, we suggested the domain $n_{s}$:
$0.996\leq n_{s}\leq 1.3$ for the spectral index. We suggested new
interpretations for the thermodynamic parameters such as chemical potential
and the central charge. We interpreted the number of e-folds as the number of
microscopic degrees of freedom for the PBH and have shown that the temperature
undergoes Wien’s displacement law. Development of the proposed approach gives
ground for a principally new scenario of the formation and evaporation of the
primordial black holes in the early Universe. This opens a new window to study
the relationship between Hubble tension and stochastic gravitational waves.

## References

  * [1] A. Afzal et al. and NANOGrav Collaboration, Astrophys. J. Lett., 951(1), L11 (2023).
  * [2] G. Agazie et al. and NANOGrav Collaboration, Astrophys. J. Lett., 951(1), L8 (2023).
  * [3] Z. Arzoumanian, et al. and NANOGrav Collaboration, Astrophys. J. Lett., 905(2), L34 (2020).
  * [4] J. Ellis and M. Lewicki, Phys. Rev. Lett. 126(4), 041304 (2021).
  * [5] V. Vaskonen and H. Veermäe, Phys. Rev. Lett. 126(5), 051303 (2021).
  * [6] V. De Luca, G. Franciolini, and A. Riotto, Phys. Rev. Lett. 126(4), 041303 (2021).
  * [7] T. L. Smith, M. Kamionkowski, and A. Cooray Phys. Rev. D 73(2) 023504 (2006).
  * [8] N. Aghanim, et al. Astron. Astrophys. (641) A6 (2020).
  * [9] S. Blasi, V. Brdar, and K. Schmitz, Phys. Rev. Lett. 126(4) 041305 (2021).
  * [10] B. P. Abbott, et al., Phys. Rev. Lett. 116(6) 061102 (2016).
  * [11] B. P. Abbott, et al., Phys. Rev. Lett. 116(24) 241103 (2016).
  * [12] B. Carr, F. Kühnel, and M. Sandstad, Phys. Rev. D 94(8) 083504 (2016).
  * [13] B. Carr, K. Kohri, Y. Sendouda, and J. I. Yokoyama, Rep. Prog. Phys. 84(11) 116902 (2021).
  * [14] K. Inomata, M. Kawasaki, K. Mukaida, Y. Tada, and T. T. Yanagida, Phys. Rev. D 96(4) 043504 (2017).
  * [15] R. H. Cyburt, B. D. Fields, and K. A. Olive, Phys. Lett. B 567(3-4) 227 (2003).
  * [16] A. G. Riess, et al. Astrophys. J. 826(1) 56 (2016).
  * [17] V. Poulin, T. L. Smith, T. Karwal and M. Kamionkowski, Phys. Rev. Lett. 122(22), 221301 (2019).
  * [18] K. Jedamzik, and L. Pogosian, Phys. Rev. Lett, 125(18), 181302 (2020).
  * [19] E. Mörtsell and S. Dhawan, J. Cosmol. Astropart. Phys., 2018(09), 025 (2018).
  * [20] G. D’Amico, L. Senatore, P. Zhang and H. Zheng, J. Cosmol. Astropart. Phys., 2021(05), 072 (2021)
  * [21] M. R. Visser, Phys. Rev. D 105(10) 106014 (2022).
  * [22] T. Wang and L. Zhao, Phys. Lett. B 827, 136935, (2022).
  * [23] Z. Gao and L. Zhao, Classical Quant. Grav., 39(7), 075019 (2022).
  * [24] P. S. Custódio and J. E. Horvath, Classical Quant. Grav., 20(5), 813 (2003).
  * [25] D. Dutcher, et al., Phys. Rev. D, 104(2), 022003 (2021).
  * [26] T. Colas, et al., JCAP, 2020(06), 001 (2020).
  * [27] O. H. E. Philcox, B. D. Sherwin, G. S. Farren and E. J. Baxter, Phys. Rev. D, 103(2), 023538 (2021).
  * [28] A. G. Riess, S. Casertano, W. Yuan, J. B. Bowers, L. Macri, J. C. Zinn and D. Scolnic, Astrophys. J. Lett., 908(1), L6 (2021).
  * [29] J. Soltis, , et al., Astrophys. J. Lett., 908(1), L5 (2021).
  * [30] C. D. Huang, et al., Astrophys. J., 889(1), 5 (2020).
  * [31] D. W. Pesce, et al., Astrophys. J. Lett., 891(1), L1 (2020).
  * [32] D. Fernández Arenas, et al., MNRAS, 474(1), 1250-1276, (2018).
  * [33] V. Gayathri, et al., Astrophys. J. Lett., 908(2), L34 (2021).
  * [34] C. Caprini and D. G. Figueroa, Classical Quant. Grav., 35(16), 163001 (2018).
  * [35] Y. Zhang, Y. Yuan, W. Zhao, and Y. T. Chen, Classical Quant. Grav. 22(7) 1383 (2005).
  * [36] F.Briscese, E. Elizalde, S. Nojiri, and S. D. Odintsov, Phys. Lett. B, 646(2-3), 105-111 (2007).
  * [37] K. Bamba, C. Q.Geng, S. I. Nojiri, and S. D. Odintsov, Phys. Rev. D, 79(8), 083014 (2009).
  * [38] S. Longhi, Ann. Phys. 360 150 (2015).
  * [39] M. Beneke and P. Moch, Phys. Rev. D 87(6) 064018 (2013).
  * [40] S. Vagnozzi, Phys. Rev. D 102(2) 023518 (2020).
  * [41] H. Motohashi and W. Hu, Phys. Rev. D 96(6) 063503 (2017).
  * [42] V. Tiwari, Classical Quant. Grav. 35(14) 145009 (2018).
  * [43] M. Bousder, J. Cosmol. Astropart. Phys. (01) 015 (2022).
  * [44] A. R. Liddle, & A. M. Green, Phys. Rep., 307(1-4), 125-131 (1998).
  * [45] S. Clesse, B. Garbrecht and Y. Zhu, Phys. Rev. D, 89(6), 063519 (2014).
  * [46] A.Linde and A. Riotto, Phys. Rev. D, 56(4), R1841 (1997).
  * [47] K. Dimopoulos, Phys. Lett. B, 735, 75-78 (2014).
  * [48] R. Maartens, D. Wands, B. A. Bassett and I. P. C. Heard, Phys. Rev. D, 62(4), 041301(R) (2000).
  * [49] Di Valentino, et al., Classical Quant. Grav., 38(15), 153001 (2021).
  * [50] B. J. Carr, In Quantum Gravity, Springer, Berlin, Heidelberg, 301-321 (2003).
  * [51] K. Jedamzik, Phys. Rev. D, 55(10), R5871 (1997).
  * [52] L. Espinosa-Portalés and J. García-Bellido, Phys. Rev. D 101(4) 043514 (2020).
  * [53] D. Teresi, arXiv preprint arXiv:2112.03922 (2021).
  * [54] T. Wang and L. Zhao Phys. Lett. B 827 136935 (2022).
  * [55] W. Cong, D. Kubizňák, and R. B. Mann, Phys. Rev. Lett 127(9) 091301 (2021).
  * [56] J. Upreti, S. Gedela, N. Pant, and R. P. Pant, New Astron. 80 101403 (2020).
  * [57] J. M. Bardeen, B. Carter, and S. W. Hawking, Commun. Math. Phys. 31(2) 161 (1973).
  * [58] N. Kumarg, S., Sen, and S. Gangopadhyay, arXiv:2206.00440 (2022).
  * [59] T. S. Biró, V. G. Czinner, H. Iguchi, and P. Ván, Phys. Lett. B 782 228 (2018).
  * [60] G. J. Stephens and B. L. Hu Int. J. Theor. Phys. 40(12) 2183 (2001).
  * [61] D. Kastor, S. Ray, and J. Traschen, Class. Quant. Grav. 26(19) 195011 (2009).
  * [62] D. Kubizňák, R. B. Mann, and M. Teo, Class. Quant. Grav. 34(6) 063001 (2017).
  * [63] M. Y. Khlopov and A. G. Polnarev, Phys. Lett. B 97(3-4) 383 (1980).
  * [64] N. Bernal and Ó. Zapata, J. Cosmol. Astropart. Phys. (03) 015 (2021).
  * [65] M. Li, Phys. Lett. B 603(1-2) 1 (2004).
  * [66] Y. Gong, Phys. Rev. D 70(6) 064029 (2004).
  * [67] B. J. Carr and J. E. Lidsey, Phys. Rev. D 48(2) 543 (1993).
  * [68] A. Dolgov and J. Silk, Phys. Rev. D 47(10) 4244 (1993).
  * [69] N. Schöneberg, J., Lesgourgues, and D. C. Hooper, J. Cosmol. Astropart. Phys. (10) 029 (2019).
  * [70] L. Verde, T. Treu, and A. G. Riess, nature Astronomy (2019).
  * [71] D. Blais, T. Bringmann, C. Kiefer and D. Polarski, Phys. Rev. D 67(2) 024024 (2003).
  * [72] J. Garcia-Bellido and E. R. Morales, Phys. Dark Universe, 18, 47-54 (2017).
  * [73] C. Germani and T. Prokopec, Phys. Dark Universe, 18, 6-10 (2017).
  * [74] R. Brandenberger, L. L. Graef, G. Marozzi and G. P. Vacca, Phys. Rev. D, 98(10), 103523 (2018).
  * [75] S. L. Adler, Phys. Rev. D, 100(12), 123503 (2019).


# An Intrinsic Nearest Neighbor Analysis
of Neural Machine Translation Architectures

Hamidreza Ghader
Informatics Institute,
University of Amsterdam,
The Netherlands
<EMAIL_ADDRESS>&Christof Monz
Informatics Institute,
University of Amsterdam,
The Netherlands
<EMAIL_ADDRESS>
###### Abstract

Earlier approaches indirectly studied the information captured by the hidden
states of recurrent and non-recurrent neural machine translation models by
feeding them into different classifiers. In this paper, we look at the encoder
hidden states of both transformer and recurrent machine translation models
from the nearest neighbors perspective. We investigate to what extent the
nearest neighbors share information with the underlying word embeddings as
well as related WordNet entries. Additionally, we study the underlying
syntactic structure of the nearest neighbors to shed light on the role of
syntactic similarities in bringing the neighbors together. We compare
transformer and recurrent models in a more intrinsic way in terms of capturing
lexical semantics and syntactic structures, in contrast to extrinsic
approaches used by previous works. In agreement with the extrinsic evaluations
in the earlier works, our experimental results show that transformers are
superior in capturing lexical semantics, but not necessarily better in
capturing the underlying syntax. Additionally, we show that the backward
recurrent layer in a recurrent model learns more about the semantics of words,
whereas the forward recurrent layer encodes more context.

##  1 Introduction

Neural machine translation (NMT) has achieved state-of-the-art performance for
many language pairs [Bahdanau et al., 2015, Luong et al., 2015b, Jean et al.,
2015, Wu et al., 2016, Vaswani et al., 2017]. Additionally, it is
straightforward to train an NMT system in an end-to-end fashion. This has been
made possible with an encoder-decoder architecture that encodes the source
sentence into a distributed representation and then decodes this
representation into a sentence in the target language. While earlier work has
investigated what information is captured by the attention mechanism of an NMT
system [Ghader and Monz, 2017], it is not exactly clear what linguistic
information from the source sentence is encoded in the hidden distributed
representation themselves. Recently, some attempts have been made to shed some
light on the information that is being encoded in the intermediate distributed
representations [Shi et al., 2016, Belinkov et al., 2017].

Feeding the hidden states of the encoder of different seq2seq systems,
including multiple NMT systems, as the input to different classifiers, ?) aim
to show what syntactic information is encoded in the hidden states. They
provide evidence that syntactic information such as the voice and tense of a
sentence and the part-of-speech (POS) tags of words are being learned with
reasonable accuracy. They also provide evidence that more complex syntactic
information such as the parse tree of a sentence is also learned, but with
lower accuracy.

?) follow the same approach as ?) to conduct more analyses about how syntactic
and morphological information are encoded in the hidden states of the encoder.
They carry out experiments for POS tagging and morphological tagging. They
study the effect of different word representations, different layers of the
encoder and target languages on the accuracy of their classifiers to reveal
the impact of these variables on the amount of the syntactic information
captured in the hidden states.

Additionally, there are recent approaches that compare different state-of-the-
art encoder-decoder architectures in terms of their capabilities to capture
syntactic structures [Tran et al., 2018] and lexical semantics [Tang et al.,
2018]. These works also use some extrinsic tasks to do the comparison. ?) use
subject-verb agreement and logical inference tasks to compare recurrent models
with transformers. On the other hand, ?) use subject-verb agreement and word
sense disambiguation for comparing those architectures in terms of capturing
syntax and lexical semantics respectively. In addition to these tasks, ?)
compare recurrent models with transformers on a multilingual machine
translation task.

Despite the approaches discussed above, attempts to study the hidden states
more intrinsically are still missing. For example, to the best of our
knowledge, there is no work that studies the encoder hidden states from a
nearest neighbor perspective to compare these distributed word representations
with the underlying word embeddings. It seems intuitive to assume that the
hidden state of the encoder corresponding to an input word conveys more
contextual information compared to the embedding of the input word itself. But
what type of information is captured and how does it differ from the word
embeddings? Furthermore, how different is the information captured by
different architectures, especially recurrent vs self-attention architectures
which use entirely different approaches to capture context?

In this paper, we choose to investigate the hidden states from a nearest
neighbors perspective and try to show the similarities and differences between
the hidden states and the word embeddings. We collect statistics showing how
much information from embeddings of the input words is preserved by the
corresponding hidden states. We also try to shed some light on the information
encoded in the hidden states that goes beyond what is transferred from the
word embeddings. To this end, we analyze how much the nearest neighbors of
words based on their hidden state representations are covered by direct
relations in WordNet [Fellbaum, 1998, Miller, 1995]. For our German side
experiments, we use GermaNet [Hamp and Feldweg, 1997, Henrich and Hinrichs,
2010]. From now on, we use WordNet to refer to either WordNet or GermaNet.

This paper does not directly seek improvements to neural translation models,
but to further our understanding of the inside behaviour of these models. It
explains what information is learned in addition to what is already captured
by embeddings. This paper makes the following contributions:

  1. 1.

We provide interpretable representations of hidden states in NMT systems
highlighting the differences between hidden state representations and word
embeddings.

  2. 2.

We compare transformer and recurrent models in a more intrinsic way in terms
of capturing lexical semantics and syntactic structures.

  3. 3.

We provide analyses of the behaviour of the hidden states for each direction
layer and the concatenation of the states from the direction layers.

English-German
---
Model | test2014 | test2015 | test2016 | test2017
Recurrent | 24.65 | 26.75 | 30.53 | 25.51
Transformer | 26.93 | 29.01 | 32.44 | 27.36
German-English
Model | test2014 | test2015 | test2016 | test2017
Recurrent | 28.40 | 29.61 | 34.28 | 29.64
Transformer | 30.15 | 30.92 | 35.99 | 31.80
Table 1:  Performance of our experimental systems in BLEU on WMT [Bojar et
al., 2017] German-English and English-German standard test sets.

##  2 Datasets and Models

We conduct our analysis using recurrent and transformer machine translation
models. Our recurrent model is a two-layer bidirectional recurrent model with
Long Short-Term Memory (LSTM) units [Hochreiter and Schmidhuber, 1997] and
global attention [Luong et al., 2015a]. The encoder consists of a two-layer
unidirectional forward and a two-layer unidirectional backward pass. The
corresponding output representations from each direction are concatenated to
form the encoder hidden state representation for each word. A concatenation
and down-projection of the last states of the encoder is used to initialize
the first state of the decoder. The decoder uses a two-layer unidirectional
(forward) LSTM. We use no residual connection in our recurrent model as they
have been shown to result in performance drop if used on the encoder side of
recurrent model [Britz et al., 2017]. Our transformer model is a 6-layer
transformer with multi-headed attention of 8 heads [Vaswani et al., 2017]. We
choose these settings to obtain competitive models with the relevant core
components from each architecture.

We train our models for two directions, namely English-German and German-
English, both of which use the WMT15 parallel training data. We exclude 100k
randomly chosen sentence pairs which are used as our held-out data. Our
recurrent system has hidden state dimensions of the size of 1,024 (512 for
each direction) and is trained using a batch size of 64 sentences. The
learning rate is set to 0.001 for the Adam optimizer [Kingma and Ba, 2015]
with a maximum gradient norm of 5. A dropout rate of 0.3 has been used to
avoid overfitting. Our transformer model has hidden state dimensions of 512
and a batch size of 4096 tokens and uses layer normalization [Vaswani et al.,
2017]. A learning rate of 2 changed under warm-up strategy with 8000 warm-up
steps is used for Adam optimizer with $\beta_{1}=0.9$, $\beta_{2}=0.998$ and
$\epsilon=10^{-9}$ [Vaswani et al., 2017]. The dropout rate is set to 0.1, and
no gradient clipping is used. The word embedding size of both models is 512.
We apply Byte-Pair Encoding (BPE) [Sennrich et al., 2016] with 32K merge
operation.

We train our models until convergence and then use the trained models to
translate 100K sentences from a held-out dataset and log the hidden states for
later use in our analyses. The 100K held-out data is randomly chosen from the
WMT15 parallel training data. The remaining of the WMT15 parallel training
data is used as our training data.

Table 1 summarizes the performance of our experimental models in BLEU
[Papineni et al., 2002] on different standard test sets. This is to make sure
that the models are trustable.

Figure 1: An example of 5 nearest neighbors of two different occurrences of
the word “deregulation”. Triangles are the nearest neighbors of “deregulation”
shown with the empty triangle. Squares are the nearest neighbors of
“deregulation” shown with the empty square.

##  3 Nearest Neighbors Analysis

Following earlier work on word embeddings [Mikolov et al., 2013, Pelevina et
al., 2016], we choose to look into the nearest neighbors of the hidden state
representations to learn more about the information encoded in them. We treat
each hidden state as the representation of the corresponding input token. This
way, each occurrence of a word has its own representation. Based on this
representation, we compute the list of $n$ nearest neighbors of each word
occurrence. We set $n$ equal to 10 in our experiments. Cosine similarity is
used as the distance measure.

In the case of our recurrent neural model, we use the concatenation of the
corresponding output representations of our two-layer forward and two-layer
backward passes as the hidden states of interest for our main experiments. We
also use the output representations of the forward and the backward passes for
our direction-wise experiments. In the case of our transformer model, we use
the corresponding output of the top layer of the encoder for each word as the
hidden state representation of the word.

Figure 1 shows an example of 5 nearest neighbors for two different occurrences
of the word “deregulation”. Each item in this figure is a specific word
occurrence, but we have removed occurrence information for the sake of
simplicity.

###  3.1 Hidden States vs Embeddings

Here, we count how many of the words in the nearest neighbors lists of hidden
states are covered by the nearest neighbors list based on the corresponding
word embeddings. Just like the hidden states, the word embeddings used for
computing the nearest neighbors are also from the same system and the same
trained model for each experiment. The nearest neighbors of the word
embeddings are also computed using cosine similarity. It should be noted that
we generate the nearest neighbors lists for the embeddings and the hidden
states separately and never compute cosine similarity between word embeddings
and the hidden state representations.

Coverage is formally computed as follows:

$cp^{H,E}_{w_{i,j}}=\frac{\left|{C^{H,E}_{w_{i,j}}}\right|}{\left|{N^{H}_{w_{i,j}}}\right|}$
(1)

where

$C^{H,E}_{w_{i,j}}=N^{H}_{w_{i,j}}\cap N^{E}_{w}$ (2)

and $N^{H}_{w_{i,j}}$ is the set of the $n$ nearest neighbors of word $w$
based on hidden state representations. Since there is a different hidden state
for each occurrence of a word, we use $i$ as the index of the sentence of
occurrence and $j$ as the index of the word in the sentence. Similarly,
$N^{E}_{w}$ is the set of the $n$ nearest neighbors of word $w$, but based on
the embeddings.

Word embeddings tend to capture the dominant sense of a word, even in the
presence of significant support for other senses in the training corpus
[Pelevina et al., 2016]. Additionally, it is reasonable to assume that a
hidden state corresponding to a word occurrence captures more of the current
sense of the word. Comparing the lists can provide useful insights as to which
hidden state-based neighbours are not strongly related to the corresponding
word embedding. Furthermore, it shows in what cases the dominant information
encoded in the hidden states comes from the corresponding word embedding and
to what extent other information has been encoded in the hidden state.

###  3.2 WordNet Coverage

In addition, we also compute the coverage of the list of the nearest neighbors
of hidden states with the directly related words from WordNet. This can shed
further light on the capability of hidden states in terms of learning the
sense of the word in the current context. Additionally, it could play the role
of an intrinsic measure to compare different architectures in their ability to
learn lexical semantics. To this end, we check how many words from the nearest
neighbors list of a word, based on hidden states, are in the list of related
words of the word in WordNet. More formally, we define $R_{w}$ to be the union
of the sets of synonyms, antonyms, hyponyms and hypernyms of word $w$ in
WordNet:

$cp^{H,W}_{w_{i,j}}=\frac{\left|{C^{H,W}_{w_{i,j}}}\right|}{\left|{N^{H}_{w_{i,j}}}\right|}$
(3)

where

$C^{H,W}_{w_{i,j}}=N^{H}_{w_{i,j}}\cap R_{w}$ (4)

and $N^{H}_{w_{i,j}}$ is the set of the $n$ nearest neighbors of word $w$
based on hidden state representations.

(a) $v_{0}$ (b) $v_{1}$ (c) $v_{2}$ (d) $v_{10}$

Figure 2: The figure shows the corresponding word and constituent subtree of query hidden state ($v_{0}$) and the corresponding word and subtree of the first ($v_{1}$) and the second ($v_{2}$) and the last ($v_{10}$) nearest neighbors of it. Model | POS | English-German | $\sigma^{2}$ | German-English | $\sigma^{2}$
---|---|---|---|---|---
Recurrent | All POS | 18% | 4 | 24% | 7
VERB | 29% | 5 | 31% | 5
NOUN | 14% | 3 | 19% | 8
ADJ | 19% | 3 | 31% | 7
ADV | 36% | 5 | 48% | 2
Transformer | All POS | 37% | 14 | 33% | 10
VERB | 39% | 8 | 36% | 7
NOUN | 38% | 16 | 31% | 14
ADJ | 32% | 11 | 36% | 9
ADV | 33% | 12 | 38% | 3
Table 2: Percentage of the nearest neighbors of hidden states covered by the
list of the nearest neighbors of embeddings.

###  3.3 Syntactic Similarity

Recent extrinsic comparisons of recurrent and non-recurrent architectures on
learning syntax [Tran et al., 2018, Tang et al., 2018] also motivate a more
intrinsic comparison. To this end, we also study the nearest neighbors of
hidden states in terms of syntactic similarities. For this purpose, we use the
subtree rooting in the smallest phrase constituent above each word, following
?). This way, we will have a corresponding parse tree for each word occurrence
in our corpus. We parse our corpus using the Stanford constituent parser [Zhu
et al., 2013]. We POS tag and parse our corpus prior to applying BPE
segmentation. Then, after applying BPE, we use the same POS tag and the same
subtree of a word for its BPE segments, following ?).

To measure the syntactic similarity between a hidden state and its nearest
neighbors, we use PARSEVAL standard metric [Sekine and Collins, 1997] as the
similarity metric between the corresponding trees. PARSEVAL computes precision
and recall by counting the correct constituents in a parse tree with respect
to a gold tree and divide the count with the number of constituent in the
candidate parse tree and the gold tree, respectively.

Figure 2a shows the corresponding word and subtree of a hidden state of
interest, and the rest in Figure 2 shows the corresponding words and subtrees
of its three neighbours. The leaves are substituted with dummy “*” labels to
show that they do not influence the computed tree similarities. We compute the
similarity score between the corresponding tree of each word and the
corresponding trees of its nearest nighbors. For example, in Figure 2 we
compute the similarity score between the tree in Figure 2a and each of the
other trees.

###  3.4 Concentration of Nearest Neighbors

Each hidden state with its nearest neighbors behaves like a cluster centered
around the corresponding word occurrence of the hidden state, whereby the
neighboring words give a clearer indication of the captured information in the
hidden state. However, this evidence is more clearly observed in some cases
rather than others.

The stronger the similarities that bring the neighbors close to a hidden
state, the more focused the neighbors around the hidden state are. Bearing
this in mind, we choose to study the relation between the concentration of the
neighbors and the information encoded in the hidden states.

To make it simple but effective, we estimate the population variance of the
neighbors’ distance from a hidden state as the measure of the concentration of
its neighbors. More formally, this is computed as follows:

$v_{w_{i,j}}=\frac{1}{n}\sum_{k=1}^{n}(1-x_{k,w_{i,j}})^{2}$ (5)

Here $n$ is the number of neighbors and $x_{k,w_{i,j}}$ is the cosine
similarity score of the $k$th neighbor of word $w$ occurring as the $j$th
token of the $i$th sentence.

##  4 Empirical Analyses

We train our systems for English-German and German-English and use our trained
model to translate a held-out data of 100K sentences. During translation, we
log the hidden state representations together with the corresponding source
tokens, their sentence and token indices.

We use the logged hidden states to compute the nearest neighbors of the tokens
with frequency of 10 to 2000 in our held-out data. We compute cosine
similarity to find the nearest neighbors.

In addition to hidden states, we also log the word embeddings from the same
systems and the same trained model. Similar to hidden states, we also use
embedding representations to compute the nearest neighbors of words. We have
to note that in the case of embedding representations we have one nearest
neighbor list for each word whereas for hidden states there is one list for
each occurrence of a word.

Model | POS | English-German | $\sigma^{2}$ | German-English | $\sigma^{2}$
---|---|---|---|---|---
Recurrent | All POS | 24% | 6 | 51% | 12
VERB | 49% | 9 | 48% | 10
NOUN | 19% | 3 | 28% | 8
ADJ | 15% | 2 | 60% | 12
ADV | 24% | 4 | 23% | 1
Transformer | All POS | 67% | 16 | 74% | 10
VERB | 77% | 9 | 70% | 9
NOUN | 65% | 18 | 63% | 13
ADJ | 66% | 14 | 81% | 9
ADV | 74% | 10 | 35% | 5
Table 3: Percentage of the nearest neighbors of hidden states covered by the
list of the directly related words to the corresponding word of the hidden
states in WordNet.

###  4.1 Embedding Nearest Neighbors Coverage

As a first experiment, we measure how many of the nearest neighbors based on
the embedding representation would still remain the nearest neighbor of the
corresponding hidden state, as described in Section 3.1, above.

Table 2 shows statistics of the coverage by the nearest neighbors based on
embeddings in general and based on selected source POS tags for each of our
models. To carry out an analysis based on POS tags, we tagged our training
data using the Stanford POS tagger [Toutanova et al., 2003]. We convert the
POS tags to the universal POS tags and report only for POS tags available in
WordNet. We use the same POS tag of a word for its BPE segments, as described
in the Section 3.3.

The first row of Table 2 shows that only 18% and 24% of the information
encoded in the hidden states respectively for English and German is already
captured by the word embeddings, in case of our recurrent model.
Interestingly, in all cases except ADV, the similarity between the hidden
states and the embeddings for the transformer model are much higher, and the
increase for nouns is much higher than for the rest. This may be a product of
the existence of no recurrence in case of transformer which results in a
simpler path from each embedding to the corresponding hidden state. We
hypothesize that this means that the recurrent model uses the capacity of its
hidden states to encode some other information that is encoded to a lesser
extent in the hidden states of the transformer.

###  4.2 WordNet Coverage

Having observed that a large portion of nearest neighbors of the hidden states
are still not covered by the nearest neighbors of the corresponding
embeddings, we look for other sources of similarity that causes the neighbors
to appear in the list. As the next step, we check to see how many of the
neighbors are covered by directly related words of the corresponding word in
WordNet.

This does not yield subsets of the nearest neighbors that are fully disjoint
with the subset covered by the nearest neighbors from the embedding list.
However, it still shows whether this source of similarity is fully covered by
the embeddings or whether the hidden states capture information from this
source that the embeddings miss.

Figure 3: The mean coverage per position of the nearest neighbors of hidden
states of the recurrent model; (i) by the nearest neighbors of the embedding
of the corresponding word (ii) by WordNet related words of the corresponding
word of the hidden state. Figure 4: The mean coverage per position of the
nearest neighbors of hidden states of the transformer model; (i) by the
nearest neighbors of the embedding of the corresponding word (ii) by WordNet
related words of the corresponding word of the hidden state.

(a) English-German, recurrent model  (b) German-English, recurrent model  (c)
English-German, transformer model  (d) German-English, transformer model

Figure 5: The average variance of cosine distance scores of the nearest
neighbors of words per positions.

Table 3 shows the general and the POS-based coverage for our English-German
and German-English systems. The transformer model again has the lead by a
large margin. The jump for nouns is again the highest, as can be seen in Table
2. This basically means that more words from the WordNet relations of the word
of interest are present in the hidden state nearest neighbors of the word. A
simple reason for this could be that the hidden states from transformer
capture more word semantic than hidden states of the recurrent model. Or in
other words, the hidden states from the recurrent model capture some
additional information that brings different words than WordNet relations of
the word of interest to its neighborhood.

To investigate whether recurrency has any direct effect on this, we compute
the mean coverage by direct relations in WordNet per position. Similarly, we
also compute the mean coverage by embedding neighbors per position. More
formally, we write:

$acp_{j}^{H,W}=\frac{\sum_{i=1}^{m}(cp^{H,W}_{w_{i,j}})}{\left|{S_{l(s)\geq
j}}\right|}$ (6)

and

$acp_{j}^{H,E}=\frac{\sum_{i=1}^{m}(cp^{H,E}_{w_{i,j}})}{\left|{S_{l(s)\geq
j}}\right|}$ (7)

respectively for the mean coverage by WordNet direct relations per position
and the mean coverage by embedding neighbors per position.

Here $cp^{H,W}_{w_{i,j}}$ and $cp^{H,E}_{w_{i,j}}$ are the values computed in
Equation 3 and 1, respectively. The function $l(s)$ returns the length of
sentence $s$ and $S_{l(s)\geq j}$ is the set of sentences that are longer than
or equal to $j$.

English-German
---
Model | Precision | Recall | Matched Brackets | Cross Brackets | Tag Accuracy
Recurrent | 0.38 | 0.38 | 0.42 | 0.31 | 0.46
Transformer | 0.31 | 0.31 | 0.35 | 0.28 | 0.40
German-English
Model | Precision | Recall | Matched Brackets | Cross Brackets | Tag Accuracy
Recurrent | 0.12 | 0.12 | 0.30 | 0.80 | 0.32
Transformer | 0.11 | 0.11 | 0.28 | 0.77 | 0.31
Table 4: Average parse tree similarity (PARSEVAL scores) between word
occurrences and their nearest neighbors. Note that the apparent identity of
precision and recall values is due to rounding and the very close number of
constituents in the corresponding parse tree of words (gold parse trees) and
the corresponding parse trees of their nearest neighbors (candidate parse
trees).

Figure 3 shows that the mean coverage in both embedding and WordNet cases is
first increasing by getting farther from the left border of a sentence, but it
starts to decrease from position 5 onwards, in case of the recurrent model.
This is surprising to see the drop in the coverages, taking into account that
the model is a bidirectional recurrent model. However, this may be a
reflection of the fact that the longer a sentence, the less the hidden states
from the recurrent model are encoding information about the corresponding
word.

Figure 4 shows the same mean coverage for the case of hidden states from the
transformer model. No decrease in the coverage per position in the case of
transformer, confirms our hypothesis that the lower coverage in case of
recurrent models is indeed directly in relation with the recurrency.

In order to refine the analysis of the positional behaviour of the hidden
states, we compute the average variance per position of the cosine distance
scores of the nearest neighbors based on hidden states. To compute this value
we use the following definition:

$Av_{j}=\frac{\sum_{i=1}^{m}v_{w_{i,j}}}{\left|{S_{l(s)\geq j}}\right|}$ (8)

Here $v_{w_{i,j}}$ is the variance estimate as defined in Equation 5, $l(s)$
is the function returning the length of sentence $s$ and $S_{l(s)\geq j}$ is
the set of sentences that are longer than or equal to $j$ as mentioned
earlier.

Figures 5a and 5b show the average variance per position for the recurrent
model. One can see that the average variance close to the borders is lower
than the variance in the middle. This means that the nearest neighbors of the
words close to the borders of sentences are more concentrated in terms of
similarity score in general. This could mean the information that captures the
meaning of those words plays less of a role compared to other information
encoded in the corresponding hidden states, especially if we take the
information of coverage per position into account. Interestingly, this does
not happen in the case of the transformer model (See Figures 5c and 5d ).

###  4.3 Syntactic Similarity

The difference in the patterns observed so far between recurrent and
transformer models (with the coverage of embeddings and the WordNet
relations), along with the reported superiority of the recurrent model in
capturing structure in extrinsic tasks [Tran et al., 2018], lead us to
investigate the syntactic similarity between the words of interest and their
nearest neighbors. To this end, we use the approach introduced in the Section
3.3 to study syntactic similarity.

Table 4 shows the average similarity between corresponding constituent
subtrees of hidden states and corresponding subtrees of their nearest
neighbors, computed using PARSEVAL [Sekine and Collins, 1997]. Interestingly,
the recurrent model takes the lead in the average syntactic similarity. This
confirms our hypothesis that the recurrent model dedicates more of the
capacity of its hidden states, compared to transformer, to capture syntactic
structures. It is also in agreement with the results reported on learning
syntactic structures using extrinsic tasks [Tran et al., 2018]. We should add
that our approach may not fully explain the degree to which syntax in general
is captured by each model, but only to the extent to which this is measurable
by comparing syntactic structures using PARSEVAL.

| Embedding | WordNet
---|---|---
POS | Forward | Backward | Forward | Backward
All | 12% | 24% | 18% | 29%
VERB | 19% | 36% | 38% | 52%
NOUN | 9% | 21% | 14% | 25%
ADJ | 13% | 22% | 12% | 17%
ADV | 28% | 34% | 20% | 23%
Table 5: Percentage of the nearest neighbors of hidden states, from the
forward and backward layers, that are covered by the list of the nearest
neighbors of embeddings and the list of the directly related words in WordNet.

(a) Covered by the nearest neighbors of the embedding of the corresponding
word of the hidden state. (b) Covered by the directly related words of the
corresponding word of the hidden state in WordNet.

Figure 6: The mean coverage per position of the nearest neighbors of hidden
states from the forward and backward recurrent layers.

###  4.4 Direction-Wise Analyses

To gain a better understanding of the behaviour of the hidden states in the
recurrent model, we repeat our experiments with hidden states from different
directions. Note that so far the recurrent hidden states in our experiments
were the concatenation of the hidden states from both directions of our
encoder.

Table 5 shows the statistics of embedding coverage and WordNet coverage from
the forward and the backward layers. As shown, the coverage of the nearest
neighbors of the hidden states from the backward recurrent layer is higher
than the nearest neighbors based on those from the forward layer.

Furthermore, Figure 6 shows the mean coverage per position of the nearest
neighbors of hidden states from the forward and the backward recurrent layers.
Figure 6a shows the mean coverage by the nearest neighbors of the
corresponding word embedding of hidden states. As shown for the forward layer
the coverage degrades as it goes forward to the end of sentences. However, the
coverage for the backward layer, except at the very beginning, almost stays
constant through sentences. As shown, the coverage for the backward layer is
much higher than the coverage for the forward layer showing that it keeps more
information from the embeddings compared to the forward layer. The decrease in
the forward layer could mean that it captures more context information when
moving forward in sentences and forgets more of the corresponding embeddings.

Figure 6b shows the mean coverage by the directly related words, in WordNet,
of the corresponding words of hidden states. The difference between the
coverage of the nearest neighbors of hidden states from the backward layer
comparing to those from the forward layer confirms more strongly that the
semantics of words are captured more in the backward layer. This is because
here we check the coverage by the directly related words, in the WordNet, of
the corresponding words of hidden states.

##  5 Conclusion

In this work, we introduce an intrinsic way of comparing neural machine
translation architectures by looking at the nearest neighbors of the encoder
hidden states. Using the method, we compare recurrent and transformer models
in terms of capturing syntax and lexical semantics. We show that the
transformer model is superior in terms of capturing lexical semantics, while
the recurrent model better captures syntactic similarity.

We show that the hidden state representations capture quite different
information than what is captured by the corresponding embeddings. We also
show that the hidden states capture more of WordNet relations of the
corresponding word than they capture from the nearest neighbors of the
embeddings.

Additionally, we provide a detailed analysis of the behaviour of the hidden
states, both direction-wise and for the concatenations. We investigate various
types of linguistic information captured by the different directions of hidden
states in a bidirectional recurrent model. We show that the reverse recurrent
layer captures more lexical and semantic information, whereas the forward
recurrent layer captures more long-distance, contextual information.

## Acknowledgements

This research was funded in part by the Netherlands Organization for
Scientific Research (NWO) under project numbers 639.022.213 and 612.001.218.

## References

  * [Bahdanau et al., 2015]  Bahdanau, Dzmitry, Kyunghyun Cho, and Yoshua Bengio.  2015\.  Neural machine translation by jointly learning to align and translate.  In International Conference on Learning Representations, San Diego, California, USA, May.
  * [Belinkov et al., 2017]  Belinkov, Yonatan, Nadir Durrani, Fahim Dalvi, Hassan Sajjad, and James Glass.  2017\.  What do neural machine translation models learn about morphology?  arXiv preprint arXiv:1704.03471.
  * [Bojar et al., 2017]  Bojar, Ondřej, Rajen Chatterjee, Christian Federmann, Yvette Graham, Barry Haddow, Shujian Huang, Matthias Huck, Philipp Koehn, Qun Liu, Varvara Logacheva, et al.  2017\.  Findings of the 2017 conference on machine translation (wmt17).  In Proceedings of the Second Conference on Machine Translation, pages 169–214.
  * [Britz et al., 2017]  Britz, Denny, Anna Goldie, Minh-Thang Luong, and Quoc Le.  2017\.  Massive exploration of neural machine translation architectures.  In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 1442–1451. Association for Computational Linguistics.
  * [Fellbaum, 1998]  Fellbaum, Christiane.  1998\.  WordNet: An Electronic Lexical Database.  MIT Press, Cambridge, MA.
  * [Ghader and Monz, 2017]  Ghader, Hamidreza and Christof Monz.  2017\.  What does attention in neural machine translation pay attention to?  In Proceedings of the Eighth International Joint Conference on Natural Language Processing (IJCNL-2017), pages 30–39.
  * [Hamp and Feldweg, 1997]  Hamp, Birgit and Helmut Feldweg.  1997\.  Germanet - a lexical-semantic net for german.  In Proceedings of the ACL workshop Automatic Information Extraction and Building of Lexical Semantic Resources for NLP Applications.
  * [Henrich and Hinrichs, 2010]  Henrich, Verena and Erhard Hinrichs.  2010\.  Gernedit - the germanet editing tool.  In Chair), Nicoletta Calzolari (Conference, Khalid Choukri, Bente Maegaard, Joseph Mariani, Jan Odijk, Stelios Piperidis, Mike Rosner, and Daniel Tapias, editors, Proceedings of the Seventh International Conference on Language Resources and Evaluation (LREC’10), Valletta, Malta, may. European Language Resources Association (ELRA).
  * [Hochreiter and Schmidhuber, 1997]  Hochreiter, Sepp and Jürgen Schmidhuber.  1997\.  Long short-term memory.  Neural Comput., 9(8):1735–1780, November.
  * [Jean et al., 2015]  Jean, Sébastien, Kyunghyun Cho, Roland Memisevic, and Yoshua Bengio.  2015\.  On using very large target vocabulary for neural machine translation.  In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 1–10, Beijing, China, July. Association for Computational Linguistics.
  * [Kingma and Ba, 2015]  Kingma, Diederick P and Jimmy Ba.  2015\.  Adam: A method for stochastic optimization.  In International Conference on Learning Representations (ICLR).
  * [Lakew et al., 2018]  Lakew, Surafel Melaku, Mauro Cettolo, and Marcello Federico.  2018\.  A comparison of transformer and recurrent neural networks on multilingual neural machine translation.  In Proceedings of the 27th International Conference on Computational Linguistics, pages 641–652. Association for Computational Linguistics.
  * [Luong et al., 2015a]  Luong, Thang, Hieu Pham, and Christopher D. Manning.  2015a.  Effective approaches to attention-based neural machine translation.  pages 1412–1421, September.
  * [Luong et al., 2015b]  Luong, Thang, Ilya Sutskever, Quoc Le, Oriol Vinyals, and Wojciech Zaremba.  2015b.  Addressing the rare word problem in neural machine translation.  In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 11–19, Beijing, China, July. Association for Computational Linguistics.
  * [Mikolov et al., 2013]  Mikolov, Tomas, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean.  2013\.  Distributed representations of words and phrases and their compositionality.  In Advances in neural information processing systems, pages 3111–3119.
  * [Miller, 1995]  Miller, George A.  1995\.  Wordnet: A lexical database for english.  Communications of the ACM, 38(11):39–41, November.
  * [Papineni et al., 2002]  Papineni, Kishore, Salim Roukos, Todd Ward, and Wei-Jing Zhu.  2002\.  BLEU: a method for automatic evaluation of machine translation.  In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pages 311–318.
  * [Pelevina et al., 2016]  Pelevina, Maria, Nikolay Arefiev, Chris Biemann, and Alexander Panchenko.  2016\.  Making sense of word embeddings.  In Proceedings of the 1st Workshop on Representation Learning for NLP, pages 174–183. Association for Computational Linguistics.
  * [Sekine and Collins, 1997]  Sekine, Satoshi and Michael J. Collins.  1997\.  Evalb – Bracket Scoring Program.
  * [Sennrich and Haddow, 2016]  Sennrich, Rico and Barry Haddow.  2016\.  Linguistic input features improve neural machine translation.  In Proceedings of the First Conference on Machine Translation: Volume 1, Research Papers, pages 83–91. Association for Computational Linguistics.
  * [Sennrich et al., 2016]  Sennrich, Rico, Barry Haddow, and Alexandra Birch.  2016\.  Neural machine translation of rare words with subword units.  In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1715–1725. Association for Computational Linguistics.
  * [Shi et al., 2016]  Shi, Xing, Inkit Padhi, and Kevin Knight.  2016\.  Does string-based neural mt learn source syntax?  In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 1526–1534, Austin, Texas, November. Association for Computational Linguistics.
  * [Tang et al., 2018]  Tang, Gongbo, Mathias Müller, Annette Rios, and Rico Sennrich.  2018\.  Why self-attention? a targeted evaluation of neural machine translation architectures.  In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 4263–4272. Association for Computational Linguistics.
  * [Toutanova et al., 2003]  Toutanova, Kristina, Dan Klein, Christopher D Manning, and Yoram Singer.  2003\.  Feature-rich part-of-speech tagging with a cyclic dependency network.  In Proceedings of the 2003 Conference of the North American Chapter of the Association for Computational Linguistics on Human Language Technology-Volume 1, pages 173–180. Association for Computational Linguistics.
  * [Tran et al., 2018]  Tran, Ke, Arianna Bisazza, and Christof Monz.  2018\.  The importance of being recurrent for modeling hierarchical structure.  In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 4731–4736. Association for Computational Linguistics.
  * [Vaswani et al., 2017]  Vaswani, Ashish, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin.  2017\.  Attention is all you need.  In Guyon, I., U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems 30, pages 5998–6008. Curran Associates, Inc.
  * [Wu et al., 2016]  Wu, Yonghui, Mike Schuster, Zhifeng Chen, Quoc V Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, et al.  2016\.  Google’s neural machine translation system: Bridging the gap between human and machine translation.  arXiv preprint arXiv:1609.08144.
  * [Zhu et al., 2013]  Zhu, Muhua, Yue Zhang, Wenliang Chen, Min Zhang, and Jingbo Zhu.  2013\.  Fast and accurate shift-reduce constituent parsing.  In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 434–443. Association for Computational Linguistics.