# On the backreaction issue for the black hole in de Sitter space-time

E.T.Akhmedov  Moscow Institute of Physics and Technology, Institutskii per. 9,
141700, Dolgoprudny, Russia  National Research Centre «Kurchatov Institute»
<EMAIL_ADDRESS>Moscow Institute of Physics and
Technology, Institutskii per. 9, 141700, Dolgoprudny, Russia  National
Research Centre «Kurchatov Institute»

###### Abstract

We consider quantum real massive scalar field in the de Sitter–Schwarzschild
space-time backround. To have an analytic head way we study in detail the two-
dimensional case, assuming that the situation in four dimensions will not be
much different conceptually. It is assumed, that quantum field is in a thermal
state i.e. described by the planckian distribution for the exact modes in the
geometry under consideration. We calculate approximately the expectation value
of stress-energy tensor near the cosmological and black hole horizons. It is
shown that for a generic temperature backreaction from quantum fields on the
geometry cannot be neglected. Thus, de Sitter–Schwarzschild space-time
geometry inevitably is strongly modified by the quantum fluctuations of the
matter fields.

##  1 Introduction

There is an indirect experimental evidence that our Universe has undergone a
stage of rapid inflationary expansion [1, 2, 3, 4, 5, 6, 7, 8]. It is believed
that the curvature of the Universe at that stage was of the order of GUT
scale. Although formation of black holes at that stage is believed to be
highly unlikely, due to the rapid expansion, we think that possibility of such
a formation strongly depends on the initial quantum state of the fields at the
beginning of the inflation, which is not known to us so far. Furthermore,
there are suggestions about hypothetical type of black holes that formed in
the early Universe [9, 10]. These so-called primordial black holes, their
stability and evaporation are of interests because they are candidates for the
components of the dark matter [11, 12].

So, we would like to consider the situation when there is a black hole present
during that expansion stage. This situation is modelled by the following two-
dimensional part of the four-dimensional metric [13, 14] (see also [15, 16,
17, 18]):

$\displaystyle ds^{2}=f(r)dt^{2}-\frac{dr^{2}}{f(r)},\qquad
f(r)=1-\frac{2M}{r}-H^{2}r^{2}.$ (1.1)

Here $M$ is the mass of the black hole, $H$ is Hubble constant, and
$d\Omega^{2}$ is the line element of the unit sphere. We assume that the
curvature of the de Sitter space-time is parametericaly very large and, hence,
quantum effects for matter fields are essential. But the gravity is still
classical.

The main feature of this metric, which is relevant for our further discussion,
is the presence of two horizons simultaneously — there is the cosmological and
black hole horizon. In the present article we consider two dimensional space-
time since it allows us to find the key properties of the QFT on the metric
(1.1) without unnecessary technical complications. However, the cost of this
simplification is the absence of the dynamical gravity in two dimensions. But,
we expect that the main conclusions remain true in higher dimensions, as it
was in other space-times (see [19, 20] to compare results in two and four
dimensions).

In [21] it was shown that quantum field theory in such a background has
certain peculiarities, which are due to the presence of two horizons. Indeed,
the function $f(r)$ in (1.1) has two separated first order roots, which
correspond to the presence of two horizons. But, after the Wick rotation to
the Euclidian signature one encounters two different conical singularities
[22, 23, 24, 25, 26, 27, 28, 18]. Therefore, it is believed that this system
cannot be in thermodynamic equilibrium and space-time (1.1) is unstable [29,
30, 31, 32, 33, 34]. See also [19, 35, 36, 20, 37] for the discussion of the
physical consequences of the presence of such singularities.

We find it hard to judge about the destiny of quantum field theory on such a
background as (1.1) without addressing the backreaction issue based on the
explicit calculation of the expectation value of the stress-energy tensor
(SET) and its effect on the background metric in the Lorentzian signature. The
goal of the present paper is to calculate the SET. We will do the calculation
for a class of states that are similar to the Hartle-Hawking state, which are
believed to be stationary.

In the sections 2 and 3 we discuss the key ideas of the technical part of the
work and take a closer look at the geometry of the space (1.1)
correspondingly. In the Sec. 4 we quantize the field, and in the Sec. 5 we
find the approximate value of the SET near the horizons and discuss its
properties. In the Sec. 6 we propose a state, which may nullify the SET on
both horizons. However such a state does not obey the fluctuation-dissipation
theorem

##  2 Setup of the problem

We consider the situation, when a quantum field is existing between two
horizons, $r_{c}$ and $r_{b}<r_{c}$, in two dimensional space-time. Namely, in
the space-time (1.1) we consider the free real massive scalar field theory:

$\displaystyle S=\frac{1}{2}\int
d^{2}x\sqrt{g}\Big{(}\partial_{\mu}\varphi\partial^{\mu}\varphi-m^{2}\varphi^{2}\Big{)}.$
(2.1)

In similar situation in four dimensions the presence of the quantum scalar
field leads to the appearance of the expectation value of the corresponding
SET on the right hand side of the Einstein equations:

$\displaystyle G_{\mu\nu}+\Lambda g_{\mu\nu}=8\pi
G\langle:\hat{T}_{\mu\nu}:\rangle,$ (2.2)

where the expectation value, $\langle:\hat{T}_{\mu\nu}:\rangle$, is taken with
respect to a state of the scalar field theory. Of coarse, in two dimensional
space-time, there is no any dynamics of gravity, which can be described by
anything like Einstein equations. However, the situation in higher dimensions
is expected to be qualitatively similar. So, the two-dimensional case allows
us to demonstrate the key features of the theory without unnecessary
complicated calculations. The main goal of our paper is to show that
independently of the choice of the state (within the class of stationary
thermal states) this expectation value of the SET diverges (infinitely grows)
either on cosmological or on the black hole horizon, or simultaneously on both
of them. Thus, quantum field changes the background metric, which signals that
there is a strong backreaction.

The metric (1.1) has the time-like killing vector. Then, among the simplest
possible states there is the class defined by the ‘‘thermal’’ density matrix:

$\displaystyle\hat{\rho}=e^{-\beta\hat{H}}.$ (2.3)

Where, $\hat{H}$ is the Hamiltonian of the scalar field theory under
consideration. Then, the expectation value of an operator $\hat{O}$ is defined
as follows:

$\displaystyle\langle\hat{O}\rangle=\frac{\text{Tr}\hat{O}\hat{\rho}}{\text{Tr}\hat{\rho}}.$
(2.4)

To calculate the expectation value of the SET we express it via the Wightman
two-point function:

$\displaystyle T_{\mu\nu}(x)_{\beta}=\bigg{(}\frac{\partial}{\partial
x_{1}^{\mu}}\frac{\partial}{\partial
x_{2}^{\nu}}-\frac{1}{2}g_{\mu\nu}\Big{[}g^{\alpha\beta}\frac{\partial}{\partial
x_{1}^{\alpha}}\frac{\partial}{\partial
x_{2}^{\beta}}-m^{2}\Big{]}\bigg{)}W_{\beta}(x_{1}|x_{2})\bigg{|}_{x_{1}=x_{2}=x},$
(2.5)

here the Wightman function is defined as follows

$\displaystyle
W_{\beta}(x_{1}|x_{2})=\langle\hat{\varphi}(x_{1})\hat{\varphi}(x_{2})\rangle.$
(2.6)

This expectation value has the standard UV divergence which has to be
regularized.

There are different regularization methods. One of the standard methods is to
use the covariant point-splitting [38, 39, 40, 41] (see also [42, 19, 35]). In
fact, the key feature of the two-dimensional case is the following: the
behavior of the modes at horizon can be expressed in terms of in-going and
out-going plane waves, which are multiplied by the transition and the
reflection coefficients calculated in the corresponding effective potential
due to the background metric. We show this explicitly below.

Moreover, for the class of the states of the field under consideration the
diagonal components of the SET do not depend on the transition and the
reflection coefficients separately. Terms which depend on these coefficients
contribute in the special combination, as we will see in (5.2). It turns out,
that this combination is nothing but the sum of the probability of the
transition and reflection, which always is equal to one. Hence, in two
dimensions one can calculate the behavior of the SET on the horizons without
knowing expressions of the reflection and transition coefficients. As a
result, in two dimensions for the states that we consider the expectation
value contains two terms. The first term depends only on the temperature and
does not depend on the geometry. While the second term, which comes from
regularization does depend only on geometry, but does not depend on the state
of field.

Also we want to stress here that in such a background as we consider here the
expectation value depends on the spatial coordinate $r$ and cannot be found
exactly for any $r$. However, we can calculate the approximate expectation
value of the SET near the horizons, where its behaviour is the most
interesting for the backreaction problem.

##  3 The geometry

In this section we discuss the geometry of the space-time with the metric
(1.1). The function $f(r)$ has two zeros for $r\geq 0$: one, $r_{c}$, is
corresponding to the cosmological horizon, while the other, $r_{b}\leq r_{c}$,
— to the black hole horizon. In (1.1) the range of $r$ is as follows:
$r_{b}\leq r\leq r_{c}$. In the situation under consideration $f(r)$ can be
written as:

$\displaystyle f(r)=H^{2}\frac{(r-r_{b})(r_{c}-r)(r+r_{c}+r_{b})}{r}.$ (3.1)

If we were considering the 4D case the positions of the horizons, $r_{c}$ and
$r_{b}$, could be related to the Hubble constant and black hole mass as
follows:

$\displaystyle H^{2}=\frac{1}{r_{c}^{2}+r_{b}r_{c}+r_{b}^{2}},\qquad
M=\frac{r_{b}r_{c}(r_{b}+r_{c})}{2(r_{c}^{2}+r_{b}r_{c}+r_{b}^{2})}=H^{2}\frac{r_{b}r_{c}(r_{b}+r_{c})}{2}.$

The condition $r_{b}\leq r_{c}$ requires the following restriction on the
black hole mass:

$\displaystyle M\leq\frac{1}{3\sqrt{3}H},$

where the equality corresponds to the extremal case $r_{b}=r_{c}$.

Note that the case $r_{c}\to\infty$ corresponds to the asymptotically flat
Schwarzschild solution. At the same time $r_{b}=0$ corresponds to the empty de
Sitter space-time $r_{c}=H^{-1}$. However, one needs to be careful in taking
such limits. In fact, for example, the space-time with non-zero $M$ (but even
very small) is topologically different from the space with $M=0$.

To quantize a field theory in such a space-time, it is necessary to impose
boundary conditions on the horizons. For these purposes it is more convenient
to use conformal variant of the metric (1.1). Namely, one can define [43]:

$\displaystyle ds^{2}=f(r^{*})\big{(}dt^{2}-dr^{*}{}^{2}\big{)},\quad{\rm
where}\quad r^{*}=\int\frac{dr}{f(r)}.$ (3.2)

As usual to express $r$ via $r^{*}$ one has to solve the following
transcendental equation:

$\displaystyle
r^{*}=-\frac{r_{b}\log(r-r_{b})}{H^{2}(r_{b}-r_{c})(2r_{b}+r_{c})}+\frac{r_{c}\log(r_{c}-r)}{H^{2}(r_{b}-r_{c})(r_{b}+2r_{c})}+\frac{(r_{b}+r_{c})\log(r+r_{b}+r_{c})}{H^{2}(2r_{b}+r_{c})(r_{b}+2r_{c})}.$
(3.3)

It is curious that for a specific values of $r_{b,c}$:

$\displaystyle r_{b}=\frac{\sqrt{3}-1}{2},\quad r_{c}=1,$ (3.4)

one can solve the transcendental eq. (3.3) explicitly

$\displaystyle
r=r_{b}\frac{\left(1+\sqrt{3}\right)\left(2e^{2r^{*}}+1\right)-\left(3+\sqrt{3}\right)\sqrt{2e^{2r^{*}}+1}}{2\left(e^{2r^{*}}-1\right)}.$
(3.5)

For generic values of $r_{b,c}$ the equation can be solved only approximately
in the near horizon limits:

$\displaystyle r\approx
r_{b}+e^{\frac{H^{2}r^{*}(r_{c}-r_{b})(2r_{b}+r_{c})}{r_{b}}}\quad\text{as}\quad
r\to r_{b},\,r^{*}\to-\infty,$

and

$\displaystyle r\approx
r_{c}-e^{-\frac{H^{2}r^{*}(r_{c}-r_{b})(r_{b}+2r_{c})}{r_{c}}}\quad\text{as}\quad
r\to r_{c},\,r^{*}\to+\infty.$

Correspondingly:

$\displaystyle
f(r*)\approx\frac{H^{2}(2r_{b}+r_{c})(r_{c}-r_{b})}{r_{b}}e^{\frac{H^{2}r^{*}(r_{c}-r_{b})(2r_{b}+r_{c})}{r_{b}}}\quad\text{as}\
\ r^{*}\to-\infty,$ (3.6)

and

$\displaystyle
f(r^{*})\approx\frac{H^{2}(r_{b}+2r_{c})(r_{c}-r_{b})}{r_{c}}e^{-\frac{H^{2}r^{*}(r_{c}-r_{b})(r_{b}+2r_{c})}{r_{c}}}\quad\text{as}\
\ r^{*}\to+\infty.$ (3.7)

Thus, one can define the canonical (inverse) temperatures due to the black
hole and cosmological horizons:

$\displaystyle\beta_{b}=\frac{4\pi
r_{b}}{H^{2}(r_{c}-r_{b})(2r_{b}+r_{c})}=\frac{2\pi}{\kappa_{b}},\quad\beta_{c}=\frac{4\pi
r_{c}}{H^{2}(r_{c}-r_{b})(r_{b}+2r_{c})}=\frac{2\pi}{\kappa_{c}}.$ (3.8)

Here $\kappa_{b}$ and $\kappa_{c}$ are the surface gravities on the event and
cosmological horizons, correspondingly. One of the important properties of the
geometry under consideration is that these temperatures are not equal for any
values of $r_{b},r_{c}$:

$\displaystyle\beta_{c}-\beta_{b}=\frac{2\pi(r_{b}+r_{c})}{H^{2}(2r_{b}+r_{c})(r_{c}+2r_{b})}>0$
(3.9)

Furthermore, note that if $r_{b}\to r_{c}$, then both $\beta_{b,c}\to\infty$,
but their difference remains finite.

Below we will show, that in the two dimensional case, to calculate the SET in
the vicinity of the horizons we need to know only the behavior of $f(r)$ and
of the modes near the horizons.

##  4 Quantization

The equations of motion for the scalar field with the action eq. (2.1) in such
a background gravitational field as (3.2) are as follows:

$\Big{[}-\partial_{r^{*}}^{2}+m^{2}f(r^{*})\Big{]}\varphi_{\omega}(r^{*})=\omega^{2}\varphi_{\omega}(r^{*}),\quad{\rm
where}\quad\varphi(t,r^{*})=\frac{1}{\sqrt{2\omega}}e^{i\omega
t}\varphi_{\omega}(r^{*}).$ (4.1)

Thus, in the radial direction one obtains a quantum mechanical scattering
problem. As a result, the full basis of solutions consists of out-going
$\overrightarrow{\varphi}_{\omega}(r^{*})$ and in-going
$\overleftarrow{\varphi}_{\omega}(r^{*})$ modes with the following behaviour
near horizons222Note that the coefficients $R_{\omega}$ depend on the
scattering direction and are related to each other as follows:
$\overrightarrow{R}_{\omega}=-\overleftarrow{R}_{\omega}^{*}\frac{T_{\omega}}{T^{*}_{\omega}}$,
while
$T_{\omega}\equiv\overrightarrow{T}_{\omega}=\overleftarrow{T}_{\omega}$.
However below we will need only absolute values of these coefficients, which
obey the following condition:
$|R_{\omega}|\equiv|\overrightarrow{R}_{\omega}|=|\overleftarrow{R}_{\omega}|$.
That is the reason why we do not distinguish between the coefficients
$\overrightarrow{R}_{\omega}$and $\overleftarrow{R}_{\omega}$ in eq. (4.1) and
below.:

|  $r^{*}\to-\infty$ |  $r^{*}\to+\infty$
---|---|---
$\overrightarrow{\varphi}_{\omega}(r^{*})$ |  $e^{i\omega r^{*}}+R_{\omega}e^{-i\omega r^{*}}$ |  $T_{\omega}e^{i\omega r^{*}}$
$\overleftarrow{\varphi}_{\omega}(r^{*})$ |  $T_{\omega}e^{-i\omega r^{*}}$ |  $e^{-i\omega r^{*}}+R_{\omega}e^{i\omega r^{*}}$

There are the following obvious condition for the reflection and transition
coefficients:

$\displaystyle|T_{\omega}|^{2}+|R_{\omega}|^{2}=1,$ (4.2)

which follows from the normalization conditions that can be related to the
canonical commutation relations between the field operator and its conjugate
momentum and between creation and annihilation operators. The field operator
has the following form:

$\displaystyle\hat{\varphi}(t,r^{*})=\int_{0}^{\infty}\frac{d\omega}{\sqrt{2\pi}}\frac{1}{\sqrt{2\omega}}\bigg{[}e^{i\omega
t}\Big{(}\hat{a}^{\dagger}_{\omega}\overrightarrow{\varphi}_{\omega}(r^{*})+\hat{b}^{\dagger}_{\omega}\overleftarrow{\varphi}_{\omega}(r^{*})\Big{)}+h.c.\bigg{]},$
(4.3)

where the creation and annihilation operators obey the standard commutation
relations:

$\displaystyle[\hat{b}_{\omega},\hat{b}^{\dagger}_{\omega^{\prime}}]=\delta(\omega-\omega^{\prime}),\qquad[\hat{a}_{\omega},\hat{a}^{\dagger}_{\omega^{\prime}}]=\delta(\omega-\omega^{\prime}),\qquad[\hat{a}_{\omega},\hat{b}^{\dagger}_{\omega^{\prime}}]=0.$
(4.4)

Then the thermal state corresponds to the following expectation values:

$\displaystyle\hat{\rho}=e^{-\beta\hat{H}},\ \ \text{then}\ \
\langle\hat{a}^{\dagger}_{\omega},\hat{a}_{\omega^{\prime}}\rangle=\frac{1}{e^{\beta\omega}-1}\delta(\omega-\omega^{\prime}),\
\
\langle\hat{b}^{\dagger}_{\omega},\hat{b}_{\omega^{\prime}}\rangle=\frac{1}{e^{\beta\omega}-1}\delta(\omega-\omega^{\prime}).$
(4.5)

So one obtains the Wightman function as follows:

$\displaystyle
W(t_{2},r_{2}^{*}|t_{1},r_{1}^{*})=\int_{-\infty}^{\infty}\frac{d\omega}{2\pi}\frac{e^{i\omega(t_{2}-t_{1})}}{e^{\beta\omega}-1}\frac{1}{2\omega}\Big{[}\overrightarrow{\varphi}_{\omega}(r_{1}^{*})\overrightarrow{\varphi^{*}}_{\omega}(r_{2}^{*})+\overleftarrow{\varphi}_{\omega}(r_{1}^{*})\overleftarrow{\varphi^{*}}_{\omega}(r_{2}^{*})\Big{]},$
(4.6)

which we will use to calculate the SET expectation value. Please note the
integration limits in (4.6) and in (4.3). The same quantization procedure was
discussed, for example, in [44, 45, 46, 47].

##  5 Stress-energy tensor

Let us start with the calculation of the energy density near say the black
hole horizon (see also [48]). The relevant behaviour of the Wightman function
near the corresponding horizon is:

$W(t_{2},r_{2}^{*}|t_{1},r_{1}^{*})\approx\int_{-\infty}^{\infty}\frac{d\omega}{2\pi}\frac{e^{i\omega(t_{2}-t_{1})}}{e^{\beta\omega}-1}\frac{1}{2\omega}\Big{[}e^{i\omega(r_{2}^{*}-r_{1}^{*})}+|R_{\omega}|^{2}e^{-i\omega(r_{2}^{*}-r_{1}^{*})}+\\\
+R_{\omega}e^{i\omega(r_{1}^{*}+r_{2}^{*})}+R^{*}_{\omega}e^{-i\omega(r_{1}^{*}+r_{2}^{*})}+|T_{\omega}|^{2}e^{-i\omega(r_{2}^{*}-r_{1}^{*})}\Big{]}.$
(5.1)

The approximate form of the Wightman function near the cosmological horizon is
very similar. Note, that near the horizon the Wightman function contains two
types of terms under the integral on the RHS of (5.1). The first type depends
on $r_{2}^{*}-r_{1}^{*}$ while the second type depends on
$r_{1}^{*}+r_{2}^{*}$. By definition (2.5) the energy density can be expressed
via the Wightman function as:

$\displaystyle\langle
T_{00}\rangle=\frac{1}{2}\Big{[}\frac{\partial}{\partial_{t_{1}}}\frac{\partial}{\partial_{t_{2}}}+\frac{\partial}{\partial_{r_{1}^{*}}}\frac{\partial}{\partial_{r_{2}^{*}}}\Big{]}W(t_{2},r_{2}^{*}|t_{1},r_{1}^{*})\Big{|}_{1\to
2}.$ (5.2)

Due to the structure of derivatives in the eq. (5.2) the terms, which depend
on $r_{1}^{*}+r_{2}^{*}$ in (5.1) do not contribute to the $T_{00}$ near the
horizon, and we obtain that:

$\displaystyle\langle T_{00}\rangle\approx\int_{-\infty}^{\infty}\frac{\omega
d\omega}{4\pi}\frac{1}{e^{\beta\omega}-1}\Big{[}1+|R_{\omega}|^{2}+|T_{\omega}|^{2}\Big{]}=\int_{-\infty}^{\infty}\frac{\omega
d\omega}{2\pi}\frac{1}{e^{\beta\omega}-1}.$ (5.3)

Where we have used the properties of the reflection and transition
coefficients (4.2). In (5.3) there is the standard UV divergence due to the
zero-point fluctuations. As we agreed above we use the covariant point
splitting method to obtain that (see the similar calculations in [19, 42,
35]):

$\displaystyle\langle:T_{00}:\rangle\approx\frac{\pi}{6}\frac{1}{\beta^{2}}-\frac{1}{6\pi}f(r^{*})^{1/2}\frac{\partial^{2}}{\partial^{2}r^{*}}f(r^{*})^{-1/2}.$
(5.4)

Concerning $f(r^{*})$, note that it has different approximate behaviour near
the two horizons: (3.6) and (3.7) correspondingly. Hence, the energy density
near the black hole horizon $r^{*}\to-\infty$ and near the cosmological
horizon $r^{*}\to\infty$ has different asymptotics, respectively:

$\displaystyle\begin{matrix}\text{Black hole
horizon}\qquad&\langle:T_{00}:\rangle\approx\frac{\pi}{6}\Big{[}\frac{1}{\beta^{2}}-\frac{1}{\beta_{b}^{2}}\Big{]}\\\
\text{Cosmological
horizon}\qquad&\langle:T_{00}:\rangle\approx\frac{\pi}{6}\Big{[}\frac{1}{\beta^{2}}-\frac{1}{\beta_{c}^{2}}\Big{]}.\end{matrix}$
(5.5)

Absolutely similarly one can obtain $\langle:T_{11}:\rangle$ near both
horizons. Recall that $\beta_{c}\neq\beta_{b}$ according to (3.8). For this
reason, we cannot choose any thermal state that nullifies the SET near the
both horizons simultaneously. Finite expectation value of the SET at least on
one of the horizons signals the presence of strong backreaction. Since the
metric tensor degenerates near the horizon it is very sensitive even to small
perturbations.

On the one hand, this reasoning doesn’t make much sense in two dimensions,
because the gravitational field is not dynamical. On the other hand, in two
dimensions, we obtained the SET by a much simpler calculations than in four
dimensions. In the last section we conclude with the physical predictions for
the backreaction problem in four dimensions.

##  6 Possible candidates for the state with the zero SET expectation value
near the horizons

In the previous section, we have considered only thermal states. It was shown,
that for any $\beta$ the expectation value of the SET is not zero at least on
one of the two horizons. However, if we go beyond the class of thermal states,
we can find the state, which corresponds to the zero SET near both horizons.
In fact, consider the following state:

$\displaystyle\langle\hat{a}^{\dagger}_{\omega},\hat{a}_{\omega^{\prime}}\rangle=n_{out}(\omega)\delta(\omega-\omega^{\prime}),\qquad\langle\hat{b}^{\dagger}_{\omega},\hat{b}_{\omega^{\prime}}\rangle=n_{in}(\omega)\delta(\omega-\omega^{\prime}),\qquad\langle\hat{b}^{\dagger}_{\omega},\hat{a}_{\omega^{\prime}}\rangle=0.$
(6.1)

Where $n_{out}(\omega)$ and $n_{in}(\omega)$ are the distributions of the out-
going and in-going modes, which are generic functions of $\omega$, not
necessarily equal to the planckian distribution. Let us take them in the
following form:

$\displaystyle n_{out}(\omega)=\frac{1}{e^{\beta_{0}\omega}-1}+\delta
n(\omega),\quad n_{in}(\omega)=\frac{1}{e^{\beta_{0}\omega}-1}-\delta
n(\omega),\quad\text{where}\quad\frac{2}{\beta_{0}^{2}}=\frac{1}{\beta_{b}^{2}}+\frac{1}{\beta_{c}^{2}},$
(6.2)

where $\delta n$ satisfies the following condition $\delta n(-\omega)=-\delta
n(\omega)$. Note also that $\beta_{0}$ does not coincide with the effective
temperature of the Schwarzshild de Sitter space [18]. We propose to consider a
class of states which are close to the thermal one, and $\delta n$ plays the
role of the difference in the number of the out-going and the in-going
particles, $n_{out}(\omega)-n_{in}(\omega)=2\delta n(\omega)$.

Because, $n_{out}(\omega)\neq n_{in}(\omega)$ in general there should be
present a non-zero energy flux $T_{01}$. To avoid such apparent non-stationary
situation, we obtain two restricting equations on $\delta n(\omega)$. The
first one comes from the conditions that $\langle:T_{00}:\rangle=0$ and
$\langle:T_{11}:\rangle=0$, while the second one – comes from the condition
$\langle:T_{01}:\rangle=0$. Using expressions similar to (5.3), but with
$n_{out}(\omega)\neq n_{in}(\omega)$, one can find the following restrictions
on $\delta n(\omega)$:

$\displaystyle\int_{0}^{\infty}d\omega\omega\delta
n(\omega)|R_{\omega}|^{2}=\frac{\pi^{2}}{12}\Big{[}\frac{1}{\beta_{b}^{2}}-\frac{1}{\beta_{c}^{2}}\Big{]},\qquad\int_{0}^{\infty}d\omega\omega\delta
n(\omega)|T_{\omega}|^{2}=0.$ (6.3)

The question is whether there is such a solution of these equations which
corresponds to a stationary state — which obeys the fluctuation-dissipation
theorem. In two dimensions, due to the specifics of the scattering processes
there, it is quite plausible to find such a state. But in 4D we do not think
that any of such states is stationary.

##  7 Conclusions

It is stated in many papers [29, 30, 33, 34] that it is impossible to achieve
the thermal equilibrium in Schwarzshild de Sitter space-time. To understand
the latter statement deeper, we study quantum field theory on such a
background. Our conclusions are opposite – we think there is n equilibrium
state, but it strongly affects the backround.

First, we show that there is no thermal state for which the expectation value
of the SET vanishes on both horizons. On the contrary, the expectation value
either blows up on one of the horizons or on both of them simultaneously,
depending on the choice of the temperature.

Although we have considered a specific two-dimensional case, we expect the
situation to be similar in four dimensions (as was the case in [20]). For
example, if we consider the state with the temperature of the cosmological
horizon, then expectation value of the SET on the black hole horizon blows up
and destroys it. I.e. for such a state one will obtain a geometry without BH
horizon [49, 50]. Furthermore, a collapse of a matter into a the black hole in
the de Sitter space may follow a completely different scenario than in the
case of the absence of the cosmological horizon, if one takes into account
quantum fluctuations.

## Acknowledgments

We would like to acknowledge valuable discussions with D.V.Diakonov and
P.A.Anempodistov. This work was supported by the grant from the Foundation for
the Advancement of Theoretical Physics and Mathematics ‘‘BASIS’’ and by the
Russian Ministry of education and science.

## References

  * [1] N. Aghanim et al.  Planck 2018 results. VI. Cosmological parameters.  Astron. Astrophys., 641:A6, 2020.  [Erratum: Astron.Astrophys. 652, C4 (2021)].  arXiv:1807.06209, doi:10.1051/0004-6361/201833910.
  * [2] Alexei A. Starobinsky.  A New Type of Isotropic Cosmological Models Without Singularity.  Phys. Lett. B, 91:99–102, 1980.  doi:10.1016/0370-2693(80)90670-X.
  * [3] Alexei A. Starobinsky.  Dynamics of Phase Transition in the New Inflationary Universe Scenario and Generation of Perturbations.  Phys. Lett. B, 117:175–178, 1982.  doi:10.1016/0370-2693(82)90541-X.
  * [4] Andrei D. Linde.  A New Inflationary Universe Scenario: A Possible Solution of the Horizon, Flatness, Homogeneity, Isotropy and Primordial Monopole Problems.  Phys. Lett. B, 108:389–393, 1982.  doi:10.1016/0370-2693(82)91219-9.
  * [5] Andrei D. Linde.  Chaotic Inflation.  Phys. Lett. B, 129:177–181, 1983.  doi:10.1016/0370-2693(83)90837-7.
  * [6] Alan H. Guth.  The Inflationary Universe: A Possible Solution to the Horizon and Flatness Problems.  Phys. Rev. D, 23:347–356, 1981.  doi:10.1103/PhysRevD.23.347.
  * [7] Alan H. Guth and S. Y. Pi.  Fluctuations in the New Inflationary Universe.  Phys. Rev. Lett., 49:1110–1113, 1982.  doi:10.1103/PhysRevLett.49.1110.
  * [8] Andreas Albrecht and Paul J. Steinhardt.  Cosmology for Grand Unified Theories with Radiatively Induced Symmetry Breaking.  Phys. Rev. Lett., 48:1220–1223, 1982.  doi:10.1103/PhysRevLett.48.1220.
  * [9] Ya. B. Zel’dovich and I. D. Novikov.  The Hypothesis of Cores Retarded during Expansion and the Hot Cosmological Model.  Soviet Astron. AJ (Engl. Transl. ), 43:758, January 1966.
  * [10] Stephen Hawking.  Gravitationally Collapsed Objects of Very Low Mass.  Monthly Notices of the Royal Astronomical Society, 152(1):75–78, 04 1971.  arXiv:https://academic.oup.com/mnras/article-pdf/152/1/75/9360899/mnras152-0075.pdf, doi:10.1093/mnras/152.1.75.
  * [11] George F. Chapline.  Cosmological effects of primordial black holes.  Nature, 253(5489):251–252, 1975.  doi:10.1038/253251a0.
  * [12] P. Meszaros.  Primeval black holes and galaxy formation.  Astron. Astrophys., 38(1):5–13, January 1975.
  * [13] Hidekazu Nariai.  On some static solutions of Einstein’s gravitational field equations in a spherically symmetric case.  Sci. Rep. Tohoku Univ. Eighth Ser., 34:160, January 1950.
  * [14] Hidekazu Nariai.  On a new cosmological solution of einstein’s field equations of gravitation.  General Relativity and Gravitation, 31:963–971, 1951.
  * [15] Paul H. Ginsparg and Malcolm J. Perry.  Semiclassical Perdurance of de Sitter Space.  Nucl. Phys. B, 222:245–268, 1983.  doi:10.1016/0550-3213(83)90636-3.
  * [16] Marcus Spradlin, Andrew Strominger, and Anastasia Volovich.  Les Houches lectures on de Sitter space.  In Les Houches Summer School: Session 76: Euro Summer School on Unity of Fundamental Physics: Gravity, Gauge Theory and Strings, pages 423–453, 10 2001.  arXiv:hep-th/0110007.
  * [17] Kyung-Seok Cha, Bum-Hoon Lee, and Chanyong Park.  dS / CFT correspondence from the brick wall method.  J. Korean Phys. Soc., 42:735, 2003.  arXiv:hep-th/0207194.
  * [18] S. Shankaranarayanan.  Temperature and entropy of Schwarzschild-de Sitter space-time.  Phys. Rev. D, 67:084026, 2003.  arXiv:gr-qc/0301090, doi:10.1103/PhysRevD.67.084026.
  * [19] E. T. Akhmedov, P. A. Anempodistov, K. V. Bazarov, D. V. Diakonov, and U. Moschella.  Heating up an environment around black holes and inside de Sitter space.  Phys. Rev. D, 103(2):025023, 2021.  arXiv:2010.10877, doi:10.1103/PhysRevD.103.025023.
  * [20] K. V. Bazarov.  Notes on peculiarities of quantum fields in space–times with horizons.  Class. Quant. Grav., 39(21):217001, 2022.  arXiv:2112.02188, doi:10.1088/1361-6382/ac8f0e.
  * [21] Bernard S. Kay and Robert M. Wald.  Theorems on the Uniqueness and Thermal Properties of Stationary, Nonsingular, Quasifree States on Space-Times with a Bifurcate Killing Horizon.  Phys. Rept., 207:49–136, 1991.  doi:10.1016/0370-1573(91)90015-E.
  * [22] S. W. Hawking.  Black Holes and Thermodynamics.  Phys. Rev. D, 13:191–197, 1976.  doi:10.1103/PhysRevD.13.191.
  * [23] G. W. Gibbons and S. W. Hawking.  Cosmological Event Horizons, Thermodynamics, and Particle Creation.  Phys. Rev. D, 15:2738–2751, 1977.  doi:10.1103/PhysRevD.15.2738.
  * [24] G. W. Gibbons and S. W. Hawking.  Action integrals and partition functions in quantum gravity.  Phys. Rev. D, 15:2752–2756, May 1977.  URL: https://link.aps.org/doi/10.1103/PhysRevD.15.2752, doi:10.1103/PhysRevD.15.2752.
  * [25] Alejandro Corichi and Andres Gomberoff.  Black holes in de sitter space: Masses, energies, and entropy bounds.  Phys. Rev. D, 69:064016, Mar 2004.  URL: https://link.aps.org/doi/10.1103/PhysRevD.69.064016, doi:10.1103/PhysRevD.69.064016.
  * [26] Claudio Teitelboim.  Gravitational thermodynamics of Schwarzschild-de Sitter space.  In Meeting on Strings and Gravity: Tying the Forces Together, pages 291–299, 2001.  arXiv:hep-th/0203258.
  * [27] Feng-Li Lin and Chopin Soo.  Black hole in de Sitter space.  In 6th International Symposium on Particles, Strings and Cosmology, pages 89–91, 7 1998.  arXiv:hep-th/9807084.
  * [28] Feng-Li Lin and Chopin Soo.  Quantum field theory with and without conical singularities: black holes with a cosmological constant and the multi-horizon scenario.  Classical and Quantum Gravity, 16(2):551–562, jan 1999.  doi:10.1088/0264-9381/16/2/017.
  * [29] Raphael Bousso and Stephen W. Hawking.  Pair creation of black holes during inflation.  Phys. Rev. D, 54:6312–6322, 1996.  arXiv:gr-qc/9606052, doi:10.1103/PhysRevD.54.6312.
  * [30] Raphael Bousso and Stephen W. Hawking.  (Anti)evaporation of Schwarzschild-de Sitter black holes.  Phys. Rev. D, 57:2436–2442, 1998.  arXiv:hep-th/9709224, doi:10.1103/PhysRevD.57.2436.
  * [31] Shin’ichi Nojiri and Sergei D. Odintsov.  Effective action for conformal scalars and anti-evaporation of black holes.  Int. J. Mod. Phys. A, 14:1293–1304, 1999.  arXiv:hep-th/9802160, doi:10.1142/S0217751X9900066X.
  * [32] Shin’ichi Nojiri and Sergei D. Odintsov.  Quantum evolution of Schwarzschild-de Sitter (Nariai) black holes.  Phys. Rev. D, 59:044026, 1999.  arXiv:hep-th/9804033, doi:10.1103/PhysRevD.59.044026.
  * [33] A. J. M. Medved.  Radiation via tunneling from a de Sitter cosmological horizon.  Phys. Rev. D, 66:124009, 2002.  arXiv:hep-th/0207247, doi:10.1103/PhysRevD.66.124009.
  * [34] Rong-Gen Cai and Ru-Keng Su.  Black holes in de sitter space and the stability conjecture of cauchy horizons.  Phys. Rev. D, 52:666–671, Jul 1995.  URL: https://link.aps.org/doi/10.1103/PhysRevD.52.666, doi:10.1103/PhysRevD.52.666.
  * [35] E. T. Akhmedov, K. V. Bazarov, and D. V. Diakonov.  Quantum fields in the future Rindler wedge.  Phys. Rev. D, 104(8):085008, 2021.  arXiv:2106.01791, doi:10.1103/PhysRevD.104.085008.
  * [36] E. T. Akhmedov, K. V. Bazarov, D. V. Diakonov, and U. Moschella.  Quantum fields in the static de Sitter universe.  Phys. Rev. D, 102(8):085003, 2020.  arXiv:2005.13952, doi:10.1103/PhysRevD.102.085003.
  * [37] E. T. Akhmedov and D. V. Diakonov.  Free energy and entropy in Rindler and de Sitter space-times.  Phys. Rev. D, 105(10):105003, 2022.  arXiv:2112.14794, doi:10.1103/PhysRevD.105.105003.
  * [38] P. C. W. Davies, S. A. Fulling, and W. G. Unruh.  Energy-momentum tensor near an evaporating black hole.  Phys. Rev. D, 13:2720–2723, May 1976.  URL: https://link.aps.org/doi/10.1103/PhysRevD.13.2720, doi:10.1103/PhysRevD.13.2720.
  * [39] T. S. Bunch and P. C. W. Davies.  Quantum Field Theory in de Sitter Space: Renormalization by Point Splitting.  Proc. Roy. Soc. Lond. A, 360:117–134, 1978.  doi:10.1098/rspa.1978.0060.
  * [40] P. C. W. Davies and S. A. Fulling.  Quantum vacuum energy in two dimensional space-times.  Proceedings of the Royal Society of London Series A, 354(1676):59–77, April 1977.  doi:10.1098/rspa.1977.0056.
  * [41] Paul R. Anderson, William A. Hiscock, and David A. Samuel.  Stress-energy tensor of quantized scalar fields in static spherically symmetric spacetimes.  Phys. Rev. D, 51:4337–4358, Apr 1995.  URL: https://link.aps.org/doi/10.1103/PhysRevD.51.4337, doi:10.1103/PhysRevD.51.4337.
  * [42] O. Diatlyk.  Hawking radiation of massive fields in 2D.  Phys. Rev. D, 104(6):065011, 2021.  arXiv:2011.03486, doi:10.1103/PhysRevD.104.065011.
  * [43] David Finkelstein.  Past-future asymmetry of the gravitational field of a point particle.  Phys. Rev., 110:965–967, May 1958.  URL: https://link.aps.org/doi/10.1103/PhysRev.110.965, doi:10.1103/PhysRev.110.965.
  * [44] Paul R. Anderson, Shohreh Gholizadeh Siahmazgi, and Zachary P. Scofield.  Infrared Effects and the Unruh State.  10 2022.  arXiv:2210.16397.
  * [45] Paul R. Anderson, Zachary P. Scofield, and Jennie Traschen.  Linear growth of the two-point function for the Unruh state in $1+1$ dimensional black holes.  In 16th Marcel Grossmann Meeting on Recent Developments in Theoretical and Experimental General Relativity, Astrophysics and Relativistic Field Theories, 4 2022.  arXiv:2205.00264.
  * [46] Hassan Firouzjahi.  Cosmological constant and vacuum zero point energy in black hole backgrounds.  Phys. Rev. D, 106(4):045015, 2022.  arXiv:2205.06561, doi:10.1103/PhysRevD.106.045015.
  * [47] Chiranjeeb Singha and Subhashish Banerjee.  Thermal radiation in curved spacetime using influence functional formalism.  Phys. Rev. D, 105(4):045020, 2022.  arXiv:2110.01264, doi:10.1103/PhysRevD.105.045020.
  * [48] Shin-ichi Tadaki and Shin Takagi.  Quantum Field Theory in Two-Dimensional Schwarzschild-de Sitter Spacetime. I: Empty Space.  Progress of Theoretical Physics, 83(5):941–952, 05 1990.  doi:10.1143/PTP.83.941.
  * [49] Pei-Ming Ho, Hikaru Kawai, Yoshinori Matsuo, and Yuki Yokokura.  Back Reaction of 4D Conformal Fields on Static Geometry.  JHEP, 11:056, 2018.  arXiv:1807.11352, doi:10.1007/JHEP11(2018)056.
  * [50] Pei-Ming Ho and Yoshinori Matsuo.  On the Near-Horizon Geometry of an Evaporating Black Hole.  JHEP, 07:047, 2018.  arXiv:1804.04821, doi:10.1007/JHEP07(2018)047.


# Improving Mandarin Speech Recogntion with Block-augmented Transformer

###### Abstract

Recently Convolution-augmented Transformer (Conformer)[1] has shown promising
results in Automatic Speech Recognition (ASR), outperforming the previous best
published Transformer Transducer[2]. In this work, we believe that the output
information of each block in the encoder and decoder is not completely
inclusive, in other words, their output information may be complementary. We
study how to take advantage of the complementary information of each block in
a parameter-efficient way, and it is expected that this may lead to more
robust performance. Therefore we propose the Block-augmented Transformer for
speech recognition, named Blockformer. We have implemented two block ensemble
methods: the base Weighted Sum of the Blocks Output (Base-WSBO), and the
Squeeze-and-Excitation module[3] to Weighted Sum of the Blocks Output (SE-
WSBO). Experiments have proved that the Blockformer significantly outperforms
the state-of-the-art Conformer-based models on AISHELL-1, our model achieves a
CER of 4.29% without using a language model and 4.05% with an external
language model on the testset.

Index Terms: speech recognition, weight, blockformer, conformer block,
transformer block, Squeeze-and-Excitation module

##  1 Introduction

Deep Learning has been applied successfully to Automatic Speech Recognition
(ASR) [4]. A variety of neural network architectures for acoustic modeling
have been explored. For example, DNNs [5], CNNs [6], RNNs [7] and end-to-end
models[8, 9, 10, 11]. Currently, there are mainly three E2E models: Neural
Transducer (NT) [1, 7, 12, 13] models, Attention-Based Encoder-Decoder (AED)
models [9, 10, 14] and Connectionist Temporal Classification [15] (CTC) models
[8, 16]. These E2E models treat ASR as a sequence-to-sequence task that
directly learns speech to text mapping with a neural network. The NT model
consists of an encoder, which maps input acoustic frames into a higher-level
representation, and a prediction and joint network which together correspond
to the decoder network[13]. The decoder is conditioned on the history of
previous predictions. The NT training is unstable and takes more memory which
may limit the training speed. The AED model is composed of an encoder, which
encodes acoustic features, and a decoder, which generates a sentence. The
architectures of many state-of-the-art ASR systems [17] are based on the AED
models. However, the AED model outputs token by token, where each token
depends on previously generated tokens and acoustic context, causing
recognition delays. On the other hand, the CTC model contains only an encoder
and outputs all tokens independently. Although its decoding speed is faster
than the AED model, in term of recognition accuracy it is generally inferior
due to the conditional independence assumption between output tokens. However,
in real environment speech recognition tasks, the attention model performs
poorly because the alignment estimated in the attention mechanism is easily
corrupted by the noise.

In recent years, there are some researches for joint CTC-attention model [18,
19] which use a shared-encoder representation trained by both CTC and
attention model objectives simultaneously within the multi-task learning
framework. In this paper, we focus on the joint CTC-attention models, aiming
at better performance by using more model output information.

Figure 1: Attention information visualization between different layers and
heads, indicates the diversity of information, but also complementarity.

In our work111https://github.com/Mininglamp-Technology/ASR-BlockFormer, we
analyze the contribution of different layers and heads through attention
visualization, as depicted in Fig 1. We consider that the diversity of
information between blocks is useful. We study how to take advantage of the
output information of each block. We believe that the output information of
each block in the encoder and decoder is not completely inclusive, and perhaps
complementary. Combine the output of each block efficiently to make full use
of complementary information. Motivated by this, we introduce a novel model,
named Blockformer, achieves state-of-the-art results on Aishell-1. The
Blockformer adds a block ensemble module, which exploits the output
information of each block in a parameter-efficient way. We have implemented
two block ensemble methods: the base Weighted Sum of the Blocks Output (Base-
WSBO), and the Squeeze-and-Excitation module[3] to Weighted Sum of the Blocks
Output (SE-WSBO). Our experiments are mainly conducted on a public Mandarin
Chinese dataset AISHELL-1. Results show that a CER of 4.29% without using a
language model and 4.05% with an external language model on the testset.

The rest of the paper is organized as follows. Section 2 presents various
components of the Blockformer model in detail. Section 3 and Section 4 present
our experimental settings, detailed results and ablation study. Concluding
remarks and a discussion of future directions are presented in Section 5.

##  2 Blockformer

In this section, we introduce the Blockformer. As shown in Figure 2, the
proposed model is built upon the typical Attention-Based Encoder-Decoders
(AEDs). The Blockformer encoder uses the Conformer block, and the decoder uses
the Transformer block. The conformer block we used is the same as [1]. In the
transformer block, we use the relative positional encoding instead of the
absolute positional encoding. Block ensemble is proposed in this paper, which
is used in encoder and decoder respectively. Moreover, we implement two Block
ensemble methods. It will be described in detail below.

###  2.1 Encoder

Our audio encoder first processes speech features with a convolution
subsampling layer followed by several conformer blocks. The encoder is
composed of a stack of identical Conformer blocks.

####  2.1.1 Conformer block

We use the standard conformer block[1], which is composed of four modules
stacked together, i.e, a feed-forward module, a self-attention module, a
convolution module, and a second feed-forward module, the two Feed Forward
modules sandwiching the Multi-Headed Self-Attention module and the Convolution
module. Mathematically, this means, for input $\mathbf{x}_{i}$ to a Conformer
block $i$, the output $\mathbf{y}_{i}$ of the block is:

$\widetilde{\mathbf{x}}_{i}=LN(\mathbf{x}_{i}+\frac{1}{2}FFN(\mathbf{x}_{i}))$
(1) ${\mathbf{x}_{i}}\\\
^{\prime}=LN(\widetilde{\mathbf{x}}_{i}+MHSA(\widetilde{\mathbf{x}}_{i}))$ (2)
${\mathbf{x}_{i}}\\\ ^{\prime}\\\ ^{\prime}=LN({\mathbf{x}_{i}}\\\
^{\prime}+Conv({\mathbf{x}_{i}}\\\ ^{\prime}))$ (3)
${\mathbf{y}_{i}}=LN({\mathbf{x}_{i}}\\\ ^{\prime}\\\
^{\prime}+\frac{1}{2}FFN({\mathbf{x}_{i}}\\\ ^{\prime}\\\ ^{\prime}))$ (4)

where FFN refers to the Feed-Forward module, MHSA refers to the Multi-Head
Self-Attention module, Conv refers to the Convolution module, and LN refers to
the Layernorm module .

####  2.1.2 Relative positional encoding

We employ the relative sinusoidal positional encoding scheme which is an
important technique from Transformer-XL[20]. The relative positional encoding
allows the self-attention module to generalize better on different input
lengths so that the resulting encoder is more robust to the variance of the
utterance length.

Figure 2: Blockformer model architecture with Base-WSBO block ensemble .
Figure 3: SE-WSBO

####  2.1.3 Block ensemble

We have implemented two block ensemble methods. The first method is very
intuitive which is the base Weighted Sum of the Blocks Output (Base-WSBO). The
structure is depicted in Figure 2. The weight of the block output is a
learnable parameter. The second method is the Squeeze-and-Excitation module
[3] to Weighted Sum of the Blocks Output (SE-WSBO). The structure is depicted
in Figure 3. We can improve the quality of representations produced by the SE
module[3] by explicitly modeling the interdependencies between the output of
all blocks. We allow the network to perform feature recalibration, through
which it can learn to use global information to selectively emphasize
informative block output and suppress unuseful ones.

Base-WSBO. The base Weighted Sum of the Blocks Output, $\mathbf{y}_{i}$ is the
block output, $\alpha_{i}$ is a scalar .

$\widetilde{\mathbf{y}}=\sum_{i=1}^{N}\hat{\alpha}_{i}\mathbf{y}_{i}$ (5)
$\hat{\alpha}_{i}=\frac{e^{\alpha_{i}}}{\sum_{j}e^{\alpha_{j}}}=Softmax(\alpha_{i})$
(6)

where in order to make the sum of the coefficients equal to 1, we optionally
use softmax to constrain the coefficient $\alpha_{i}$.

SE-WSBO. Since Base-WSBO does not fully establish the correlation between
blocks, each output $\mathbf{y}_{i}$ of the block is unable to exploit
contextual information outside of this region. In order to tackle the issue of
exploiting blocks dependencies, we first consider how to extract each block
output feature. We squeeze global block output information into a channel
descriptor. This is achieved by using global average pooling to generate
channel-wise statistics. The c-th element of $\mathbf{z}$ is calculated by:

$z_{c}=\mathbf{F}_{sq}(\mathbf{y}_{c})=\frac{1}{T\times
D}\sum_{i=1}^{T}\sum_{j=1}^{D}\mathbf{y}_{c}(i,j)$ (7)

where $\mathbf{y}_{c}\in\mathbb{R}^{T\times D}$ refers to the c-th block
output, T and D are the dimensions.

$\mathbf{s}=\mathbf{F}_{ex}(\mathbf{z},\mathbf{W})=\sigma(\mathbf{W}_{2}\delta(\mathbf{W}_{1}\mathbf{z}))$
(8)

where $\sigma$ refers to the sigmoid function, $\delta$ refers to the ReLU
[21] function, $\mathbf{W}_{1}\in\mathbb{R}^{\frac{C}{r}\times C}$ and
$\mathbf{W}_{2}\in\mathbb{R}^{C\times\frac{C}{r}}$. $\frac{C}{r}$ is the
bottleneck dim. Usually, we set $C_{encoder}=12,C_{decoder}=6,r=1$.

$\widetilde{\mathbf{y}}_{c}=\mathbf{F}_{scale}(\mathbf{y}_{c},s_{c})=s_{c}\mathbf{y}_{c}$
(9)

where $\mathbf{F}_{scale}$ refers to channel-wise multiplication by the scalar
$s_{c}$.

$\widetilde{\mathbf{y}}=\mathbf{F}_{sum}(\widetilde{\mathbf{y}}_{c})=\sum_{c=1}^{N}\widetilde{\mathbf{y}}_{c}$
(10)

where $\widetilde{\mathbf{y}}$ is the final block ensemble output .

###  2.2  Attention based decoder

The decoder is also composed of a stack of identical Transformer blocks. In
addition to the two modules (a feed-forward module, a self-attention module) ,
the decoder inserts a third module, named multi-Head Cross-Attention module
(MHCA) which performs multi-head attention over the output of encoder block
ensemble. Similar to the encoder, we employ Relative positional encoding and
Block ensemble.

####  2.2.1 Transformer block

For input $\mathbf{x}_{i}$ to the decoder Transformer block i, the output
$\mathbf{y}_{i}$ of the block is:

${\mathbf{x}_{i}}\\\ ^{\prime}=LN(\mathbf{x}_{i}+MHSA(\mathbf{x}_{i}))$ (11)
${\mathbf{x}_{i}}\\\ ^{\prime}\\\ ^{\prime}=LN({\mathbf{x}_{i}}\\\
^{\prime}+MHCA({\mathbf{x}_{i}}\\\ ^{\prime},\widetilde{\mathbf{y}}))$ (12)
${\mathbf{y}_{i}}=LN({\mathbf{x}_{i}}\\\ ^{\prime}\\\
^{\prime}+FFN({\mathbf{x}_{i}}\\\ ^{\prime}\\\ ^{\prime}))$ (13)

where $\widetilde{\mathbf{y}}$ refers to the encoder output after block
ensemble.

###  2.3  Hybrid CTC-Attention Objective

With the aim to take advantage of ctc and attention, the CTC and attention
loss can be combined [22]. Both CTC and attention-based methods have their own
drawbacks. CTC often has poor results due to the assumption of conditional
independence between output tokens. Since the estimated alignment in the
attention mechanism is easily corrupted by noise, the performance of the
attention model is also generally poor.

The combination of these two not only helps the model to converge, but also
enables the model to take full advantage of token dependencies. The hybrid
CTC-Attention objective is defined in Equation 14, where x is the acoustic
feature, y is the corresponding annotation. $L_{CTC}(x,y)$, $L_{AED}(x,y)$are
the CTC and AED loss respectively, $\lambda\in(0,1)$ is a hyperparameter which
balance the importance of CTC and AED loss:

$L_{hybrid}=\lambda
L_{CTC}(\mathbf{x},\mathbf{y})+(1-\lambda)L_{AED}(\mathbf{x},\mathbf{y})$ (14)

##  3 Experimental setting

###  3.1 Datasets

In this paper, we validated the proposed two Blockformer methods (Base-WSBO
and SE-WSBO) on two Mandarin speech recognition datasets: public AISHELL-1
corpus [23], internal 1400 hours corpus. The AISHELL-1 corpus consists of 178
hours of labeled speech collected from 400 speaker with high fidelity
microphone. We construct the decoding graph TLG using its annotated text as a
corpus for language model. The 1400-hour internal corpus is collected from
some service domains, such as medicine, cosmetic and estate, which is more
diverse in data and more challenging in speech recognition. For the AISHELL-1
task, we use the 150-hour for training and the 18-hour development set for
early-stopping. The character error rate (CER%) is reported in the
7176-sentence test set (about 10 hours). For the 1400-hour internal corpus, we
use a 1400-hour for training, a 6-hour development set for early-stopping, and
a 13-hour test set for evaluation.

###  3.2 Experimental Setup

For all experiments, the input features are 80-dimensional log Mel-
filterbank(FBank) computed on 25ms window with 10ms shift. We use the open-
source WeNet toolkit[24] to build both the vanilla hybrid CTC/attention
Conformer baseline and our proposed Blockformer. We use SpecAugment[25, 26]
for data augmentation with the frequency mask parameter (F = 10), the time
mask parameter(T=50), and the number of frequency and time masks(mF = mT = 2).
We choose 4233 and 4599 characters (including $\langle pad\rangle,\langle
eos\rangle,\langle sos\rangle$ labels) as model units for AISHELL-1, 1400-hour
internal corpus respectively.

We build the baseline model with a 12-layer encoder and a 6-layer decoder
following the WeNet recipe[24]. We employ h = 4 parallel attention heads in
the blockformer models. For every layer, we use $d_{k}=d_{v}=d_{model}/h=64$,
$d_{ffn}=2048$ . The base model has about 46M parameters. Our Base-WSBO model
only adds some scaler weight( Increased number of parameters: 18 ), SE-WSBO
model adds SE-module( Increased number of parameters: 360 ).

AdamOptimizer[27] is used with $learning\\_rate=0.002,warm\\_up=50000$, and
gradient clipping at 5.0. Moreover, we employ label smoothing of value
$\epsilon_{ls}=0.1$[28] and dropout rate of $P_{drop}=0.1$. For
regularization, we apply dropout [29] in each residual unit of the conformer
block and transformer block, i.e, to the output of each module, before it is
added to the module input. We set the weight $\lambda$ of the CTC branch
during joint training to 0.3. During joint decoding, we set the CTC-weight
$\lambda$ to 0.5. We also train an external n-gram LM followed by WeNet
recipe[24]. To avoid overfitting, we averaged the 30 best model parameters in
the development dataset. For the optimal combination of SE-WSBO acoustic model
and language model TLG for Aishell-1 testset, the parameters detail as
follows:
$acoustic\\_scale=3.26,ngram\\_weight=0.93,lattice\\_beam=16,beam=32$.

Furthermore, we use the gradient accumulation[30, 31] during training, where
the gradients are updated every 4 batches. The baseline dataloader sorts all
the utterances according to the frames length, packs them in sequence by the
batch size, and randomly select to be passed to the model. The benefit of
sorting all utterances makes training more efficient, which can occupy as much
GPU memory as possible. The disadvantage is that the package of each batch is
fixed, which may not allow the model to learn a better combination of
information. In view of the above shortcomings, our batch data are selected
from all utterances instead of packages. We train models using 4 Nvidia A100
GPUs for at most 120 epochs with a batch size of 20.

##  4 Experimental results

We firstly present our results on the Aishell-1 test dataset to provide a deep
insight into our method. The effectiveness of the proposed method is further
verified on the larger corpus (1400-hours internal corpus). To evaluate the
effectiveness of the two Blockformer methods Base-WSBO and SE-WSBO, we conduct
some experiments to compare differences. The performance of the models is
evaluated based on character error rates (CERs) both without and with external
language models. All our experimental results are based on the attention-
rescore two-step decoding method [32, 19].

###  4.1 Results of Aishell-1

Table 1: Experimental results on the Aishell-1 test dataset (CER%)  Method | No LM | With LM
---|---|---
AEDs(_previous work_) |  |
Espnet[33] | 4.90% | 4.70%
WeNet[24] | 4.61% | 4.36%
K2[34] |  – | 4.26%
NTs(_previous work_) |  |
Neural Transducer+LFMMI[35] |  – | 4.18%
Blockformer(_our work_) |  |
Base-WSBO | 4.48% | 4.22%
Base-WSBO+Softmax | 4.54% | 4.28%
SE-WSBO | 4.29% |  4.05%

Table 1 compares the (CER) result of our model on the Aishell-1 test dataset
with a few public models include: Espnet[33], WeNet[24], K2[34] and Neural
Transducer+LFMMI[35]. The first three models are all AED model structures, and
the last is NT based. All our evaluation results round up to 2 digit after
decimal point. From Table 1, we can see that our proposed Base-WSBO model is
better than previous three AEDs model, but it is still a little worse compared
with the NT model. In the experiment, we additionally learn that the effect of
adding softmax to Base WSBO became worse. This case may indicate that the
fixed sum of coefficients is not effective.

Without a language model, the performance of our SE-WSBO model already
achieves competitive results of CER 4.29% on testset outperforming the known
WeNet model. With the language model, our model achieves the lowest CER 4.05%
among all the existing models. This clearly demonstrates the effectiveness of
block ensemble in a single neural network.

###  4.2 Ablation Study

Table 2 shows the impact of each change to SE-WSBO on the Aishell-1 test
dataset. We perform experiments to study the effect of varying number of
blocks at block ensemble. E12D6 means utilizing all 12 block output
information of the encoder and all 6 block output information of the decoder.
E5D5 means use the last 5 block output of encoder and the ones of decoder
respectively.

We can see the major performance drops in Table 2 (1) replacing E12D6 with
E5D5, which reduce the number of encoder block ensemble more than the one of
the decoder. Comparison (4) can also clearly see the importance of the encoder
block ensemble. The result of (2) shows that the use of relative position
encoding in the decoder is not more effective than absolute position encoding.
It may be that this replacement only affects the attention decoding result. In
the two-step decoding method, the CTC is the first pass decoding, and the
attention is only used for revision in the second pass. So the replacement of
the positional encoding method on the decoder does not show the advantage. As
can be seen from the results of experiment (3), using the random data reading
method does not seem to be very critical, which may be affected by random
factors. To verify the importance of encoder and decoder in block ensemble
respectively, we design (4) and (5) experiments. The results show that the
block ensemble of the encoder is more important than the one of decoder in SE-
WSBO model. The experiments in (6) want to discuss how much performance can be
improved by considering only relative position encoding in the decoder in
combination with unsorted and unpacked training data.

Table 2: Ablation study of the SE-WSBO (CER%). Starting from the SE-WSBO, we remove its features: (1)replacing E12D6 with E5D5; (2)replacing decoder self-attention with relative positional encoding with a vanilla self-attention layer [27] with absolute positional encoding; (3)replacing the data preprocessing method used in SE-WSBO to randomly extract the training data one by one with which is packaged after the overall sorting of the training utterance length in the baseline; (4)replacing E12D6 with only E12; (5)replacing E12D6 with only D6; (6)without using E12D6. All ablation study results are evaluated on the Aishell-1 test dataset without the external LM. Method | No LM
---|---
SE-WSBO | 4.29%
– E12D6 + E5D5 | 4.48%
– decoder Relative Pos + Abs Pos | 4.49%
– no_pack no_sort + packsort | 4.47%
– E12D6 + E12 | 4.39%
– E12D6 + D6 | 4.46%
– E12D6 | 4.52%

###  4.3 Results of internal 1400 hours

As in Table 3, our SE-WSBO model is still beneficial for internal 1400 hours
training. The CER value of SE-WSBO model decreased by 3% in the 13-hour
testset.

Moreover, comparing the No LM and With LM columns in Table 2, we also observe
that the performance of the language model improves slightly when the amount
of training data increases. It is possible that the acoustic model has already
learned enough knowledge of the language model in the large amount of training
data.

Table 3: Experimental results on 1400-hour internal dataset (CER%)  Method | No LM | With LM
---|---|---
WeNet-baseline | 23.49% | 23.06%
SE-WSBO | 22.78% | 22.37%

##  5 CONCLUSIONS AND FUTURE WORK

In this work, we introduced the Blockformer, an architecture that integrated
block ensemble for end-to-end speech recognition. We studied and discussed two
block ensemble methods, and demonstrated that it was beneficial to the
performance of the Blockformer model. The model achieved better results with a
few extra parameters than previous work on the Mandarin dataset Aishell-1, and
achieved a new state-of-the-art performance at 4.29%/4.05% for test dataset.
The Blockformer on a much larger and more challenging dataset also confirmed
our findings. Our method was easy to implement and could also be applied to
other models.

There were some works we had not explored yet. For example, the decoder used
in our experiment is still based on the transformer block. In fact, we did the
experiment based on the conformer block, but the CER only dropped by about 3%.
We would like to write another paper to discuss this issue in detail in our
future work.

## References

  * [1] A. Gulati, J. Qin, C.-C. Chiu, N. Parmar, Y. Zhang, J. Yu, W. Han, S. Wang, Z. Zhang, Y. Wu, and R. Pang, “Conformer: Convolution-augmented transformer for speech recognition,” 2020.
  * [2] Q. Zhang, H. Lu, H. Sak, A. Tripathi, E. McDermott, S. Koo, and S. Kumar, “Transformer transducer: A streamable speech recognition model with transformer encoders and rnn-t loss,” _ICASSP 2020 - 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , pp. 7829–7833, 2020.
  * [3] J. Hu, L. Shen, S. Albanie, G. Sun, and E. Wu, “Squeeze-and-excitation networks,” _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , vol. 42, pp. 2011–2023, 2020.
  * [4] G. E. Hinton, L. Deng, D. Yu, G. E. Dahl, A. rahman Mohamed, N. Jaitly, A. W. Senior, V. Vanhoucke, P. Nguyen, T. N. Sainath, and B. Kingsbury, “Deep neural networks for acoustic modeling in speech recognition,” _IEEE Signal Processing Magazine_ , vol. 29, p. 82, 2012.
  * [5] G. E. Dahl, D. Yu, L. Deng, and A. Acero, “Context-dependent pre-trained deep neural networks for large-vocabulary speech recognition,” _IEEE Transactions on Audio, Speech, and Language Processing_ , vol. 20, pp. 30–42, 2012\.
  * [6] T. N. Sainath, A. rahman Mohamed, B. Kingsbury, and B. Ramabhadran, “Deep convolutional neural networks for lvcsr,” _2013 IEEE International Conference on Acoustics, Speech and Signal Processing_ , pp. 8614–8618, 2013.
  * [7] A. Graves, A. rahman Mohamed, and G. E. Hinton, “Speech recognition with deep recurrent neural networks,” _2013 IEEE International Conference on Acoustics, Speech and Signal Processing_ , pp. 6645–6649, 2013.
  * [8] A. Graves and N. Jaitly, “Towards end-to-end speech recognition with recurrent neural networks,” in _ICML_ , 2014.
  * [9] W. Chan, N. Jaitly, Q. V. Le, and O. Vinyals, “Listen, attend and spell: A neural network for large vocabulary conversational speech recognition,” _2016 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , pp. 4960–4964, 2016.
  * [10] D. Bahdanau, J. Chorowski, D. Serdyuk, P. Brakel, and Y. Bengio, “End-to-end attention-based large vocabulary speech recognition,” _2016 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , pp. 4945–4949, 2016.
  * [11] C.-C. Chiu, T. N. Sainath, Y. Wu, R. Prabhavalkar, P. Nguyen, Z. Chen, A. Kannan, R. J. Weiss, K. Rao, K. Gonina, N. Jaitly, B. Li, J. Chorowski, and M. Bacchiani, “State-of-the-art speech recognition with sequence-to-sequence models,” _2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , pp. 4774–4778, 2018.
  * [12] A. Graves, “Sequence transduction with recurrent neural networks,” _ArXiv_ , vol. abs/1211.3711, 2012.
  * [13] K. Rao, H. Sak, and R. Prabhavalkar, “Exploring architectures, data and units for streaming end-to-end speech recognition with rnn-transducer,” _2017 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU)_ , pp. 193–199, 2017.
  * [14] L. Dong, S. Xu, and B. Xu, “Speech-transformer: A no-recurrence sequence-to-sequence model for speech recognition,” _2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , pp. 5884–5888, 2018.
  * [15] A. Graves, S. Fernández, F. J. Gomez, and J. Schmidhuber, “Connectionist temporal classification: labelling unsegmented sequence data with recurrent neural networks,” _Proceedings of the 23rd international conference on Machine learning_ , 2006.
  * [16] S. Kriman, S. Beliaev, B. Ginsburg, J. Huang, O. Kuchaiev, V. Lavrukhin, R. Leary, J. Li, and Y. Zhang, “Quartznet: Deep automatic speech recognition with 1d time-channel separable convolutions,” _ICASSP 2020 - 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , pp. 6124–6128, 2020.
  * [17] S. Karita, N. Yalta, S. Watanabe, M. Delcroix, A. Ogawa, and T. Nakatani, “Improving transformer-based end-to-end speech recognition with connectionist temporal classification and language model integration,” in _INTERSPEECH_ , 2019.
  * [18] S. Kim, T. Hori, and S. Watanabe, “Joint ctc-attention based end-to-end speech recognition using multi-task learning,” _2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , pp. 4835–4839, 2017.
  * [19] B. Zhang, D. Wu, Z. Yao, X. Wang, F. Yu, C. Yang, L. Guo, Y. Hu, L. Xie, and X. Lei, “Unified streaming and non-streaming two-pass end-to-end model for speech recognition,” _ArXiv_ , vol. abs/2012.05481, 2020.
  * [20] Z. Dai, Z. Yang, Y. Yang, J. G. Carbonell, Q. V. Le, and R. Salakhutdinov, “Transformer-xl: Attentive language models beyond a fixed-length context,” _ArXiv_ , vol. abs/1901.02860, 2019.
  * [21] V. Nair and G. E. Hinton, “Rectified linear units improve restricted boltzmann machines,” in _ICML_ , 2010.
  * [22] S. Watanabe, T. Hori, S. Kim, J. R. Hershey, and T. Hayashi, “Hybrid ctc/attention architecture for end-to-end speech recognition,” _IEEE Journal of Selected Topics in Signal Processing_ , vol. 11, pp. 1240–1253, 2017\.
  * [23] H. Bu, J. Du, X. Na, B. Wu, and H. Zheng, “Aishell-1: An open-source mandarin speech corpus and a speech recognition baseline,” _2017 20th Conference of the Oriental Chapter of the International Coordinating Committee on Speech Databases and Speech I/O Systems and Assessment (O-COCOSDA)_ , pp. 1–5, 2017.
  * [24] Z. Yao, D. Wu, X. Wang, B. Zhang, F. Yu, C. Yang, Z. Peng, X. Chen, L. Xie, and X. Lei, “Wenet: Production oriented streaming and non-streaming end-to-end speech recognition toolkit,” _Interspeech 2021_ , 2021.
  * [25] D. S. Park, W. Chan, Y. Zhang, C.-C. Chiu, B. Zoph, E. D. Cubuk, and Q. V. Le, “Specaugment: A simple data augmentation method for automatic speech recognition,” _Interspeech 2019_ , Sep 2019. [Online]. Available: http://dx.doi.org/10.21437/Interspeech.2019-2680
  * [26] D. S. Park, Y. Zhang, C.-C. Chiu, Y. Chen, B. Li, W. Chan, Q. V. Le, and Y. Wu, “Specaugment on large scale datasets,” _ICASSP 2020 - 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , pp. 6879–6883, 2020.
  * [27] A. Vaswani, N. M. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin, “Attention is all you need,” _ArXiv_ , vol. abs/1706.03762, 2017.
  * [28] C. Szegedy, V. Vanhoucke, S. Ioffe, J. Shlens, and Z. Wojna, “Rethinking the inception architecture for computer vision,” _2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , pp. 2818–2826, 2016.
  * [29] N. Srivastava, G. E. Hinton, A. Krizhevsky, I. Sutskever, and R. Salakhutdinov, “Dropout: a simple way to prevent neural networks from overfitting,” _J. Mach. Learn. Res._ , vol. 15, pp. 1929–1958, 2014.
  * [30] J. Hermans, G. Spanakis, and R. Moeckel, “Accumulated gradient normalization,” _ArXiv_ , vol. abs/1710.02368, 2017.
  * [31] “https://towardsdatascience.com/what-is-gradient-accumulation-in-deep-learning-ec034122cfa.”
  * [32] T. N. Sainath, R. Pang, D. Rybach, Y. He, R. Prabhavalkar, W. Li, M. Visontai, Q. Liang, T. Strohman, Y. Wu, I. McGraw, and C.-C. Chiu, “Two-pass end-to-end speech recognition,” in _INTERSPEECH_ , 2019.
  * [33] S. Watanabe, T. Hori, S. Karita, T. Hayashi, J. Nishitoba, Y. Unno, N. Yalta, J. Heymann, M. Wiesner, N. Chen, A. Renduchintala, and T. Ochiai, “Espnet: End-to-end speech processing toolkit,” _ArXiv_ , vol. abs/1804.00015, 2018\.
  * [34] “k2: Fsa/fst algorithms, differentiable, with pytorch compatibility., 2021. https://github.com/k2-fsa/k2.”
  * [35] J. Tian, J. Yu, C. Weng, S.-X. Zhang, D. Su, D. Yu, and Y. Zou, “Consistent training and decoding for end-to-end speech recognition using lattice-free mmi,” _ArXiv_ , vol. abs/2112.02498, 2021.


# A Deep Reinforcement Learning Approach
for Improving Age of Information in Mission-Critical IoT

Hossam Farag  Department of Electronic Systems
Aalborg University
Copenhagen, Denmark
<EMAIL_ADDRESS>Mikael Gidlund  Department of Information Systems and
Technology
Mid Sweden University
Sundsvall, Sweden
<EMAIL_ADDRESS>Čedomir Stefanović  Department of Electronic Systems
Aalborg University
Copenhagen, Denmark
<EMAIL_ADDRESS>
###### Abstract

The emerging mission-critical Internet of Things (IoT) play a vital role in
remote healthcare, haptic interaction, and industrial automation, where timely
delivery of status updates is crucial. The Age of Information (AoI) is an
effective metric to capture and evaluate information freshness at the
destination. A system design based solely on the optimization of the average
AoI might not be adequate to capture the requirements of mission-critical
applications, since averaging eliminates the effects of extreme events. In
this paper, we introduce a Deep Reinforcement Learning (DRL)-based algorithm
to improve AoI in mission-critical IoT applications. The objective is to
minimize an AoI-based metric consisting of the weighted sum of the average AoI
and the probability of exceeding an AoI threshold. We utilize the actor-critic
method to train the algorithm to achieve optimized scheduling policy to solve
the formulated problem. The performance of our proposed method is evaluated in
a simulated setup and the results show a significant improvement in terms of
the average AoI and the AoI violation probability compared to the related-
work.

###### Index Terms:

IoT, deep reinforcement learning, neural networks, age of information,
mission-critical communication.

##  I Introduction

Future Internet of Things (IoT) enables ubiquitous sensing and connected
systems that will provide enhanced situational awareness, data driven decision
analytics, and automated response without human intervention [1]. Mission-
critical IoT applications, such as first-responders monitoring, process
automation and control, and intelligent transportation, are characterized by
stringent communication requirements in terms of real-time and reliable
information delivery [2]. For such scenarios, packet delay cannot be
considered a primary performance metric, but rather the freshness of the
sensor data received at the destination side. Packets received with stale
sensor data are useless to the destination, and may result in wrong
actuation/control decisions as well. To this end, the Age of Information (AoI)
[3], is a relevant metric in quantifying the information freshness, which
accounts for the time elapsed since the generation instant of the latest
received update at the destination. The AoI depends both on the traffic
generation pattern (frequent messages offering fresher information), as well
as on the time spent by an update in the service. As a result, the metric is
intrinsically different from classical performance criteria like throughput
and latency, which focuses on a single packet and only captures its service
time [4].

Most previous works [5, 6, 7] focused on the analysis of the average AoI and
peak AoI. However, these two metrics of AoI can not fully characterize the
precise performance of real-time status updates in mission-critical
applications, as they do cannot account for extreme AoI events that occur with
very low probabilities [8]. Instead, the AoI distribution needs to be
considered especially when dealing with time-critical IoT applications.
Specifically, minimizing the violation probability that the AoI of each sensor
node exceeds a predetermined age constraint is of great significance for
guaranteeing information freshness in mission-critical networks. Therefore,
the minimization problem should accounts for a combined AoI-based metric that
considers the trade-off between minimizing the average AoI of each node and
maintaining the violation probability at its minimum. This is a stochastic
optimization problem with non-convex constraints that is known to be NP-hard
even in deterministic settings. Deep Reinforcement Learning (DRL) is
introduced as a viable way to solve NP-hard and non-convex problems [9], where
an agent can be trained offline to choose the action that maximizes the system
reward.

In this paper, we introduce a DRL-based algorithm to minimize the average AoI
and its threshold violation in mission-critical IoT networks. We consider a
remote monitoring and control scenario where a set of sensor nodes are
deployed to transmit fresh updates to a central controller. We develop the
problem as a NP-hard and non-convex problem aiming to minimize an objective
function that constitutes the weighted sum of the average AoI and the
probability of AoI threshold violation. Then, we utilize DRL to solve the
problem and produce the scheduling policy, i.e., which node is to be scheduled
for transmitting fresh data at a given time instant that minimizes the
formulated AoI-based metric. The DRL algorithm is trained using the actor-
critic algorithm with a non-zero probability of packet failure. Performance
evaluations are carried out via simulations and the obtained results show that
our proposed algorithm outperforms existing work in terms of the the average
AoI and the AoI violation probability.

The remainder of the text is organized as follows. Section II presents related
work. Section III introduces the proposed DRL method. Performance evaluations
are given in Section IV, followed by the conclusions in Section V.

##  II Related Work

The problem of AoI minimization has been addressed in the literature
considering various setups. The authors in [10] investigates the AoI
minimization problem with minimum weighted sum of average peak ages for one-
hop battery-free wireless sensor networks. In [11], the authors consider the
problem of scheduling real-time traffic to minimize the weighted average AoI
under timely throughput constraint. An age-independent stationary randomized
policy is proposed in [12] to minimize the long-term weighted AoI subject to a
long-term average power constraint in fading multiple access channels. The
authors of [13] also address the average AoI minimization problem under
deterministic and random packet deadlines. In [14] the authors aim to minimize
the long-term average AoI of industrial IoT networks under average
transmission power constraints at the source. Despite their interesting
results, all the aforementioned works overlook the AoI violation probability
in the formulated optimization problem. The work in [8] introduces a
theoretical basis for characterizing the violation probability of AoI in IoT
systems under First-Come-First-Served (FCFS) discipline. The authors in [15]
consider the minimization of the AoI violation probability in vehicular
communication networks through an online learning approach. However, the
proposed approach is not applicable in the considered industrial scenario of
remote monitoring and control.

Figure 1: The remote monitoring system model.

##  III The Proposed Method

In this work, we consider the remote monitoring system shown in Fig.1, in
which $N$ sensor nodes are distributed at the field network side. The
controller selects one node at a time to generate a fresh update and transmit
it to a central controller. The monitoring process is modelled as a set of
tasks, where $i\in\\{0,1,...,T-1\\}$ denotes the task ID, and the scheduling
policy implies that the controller selects one node $n\in\\{0,1,...,N-1\\}$ at
a time to transmit its update at task ID $i$ at a rate $\lambda(i)$, i.e.,
$\lambda(i)$ represents the throughput achieved at the $i-th$ task. The
variable $I_{n}(i)$ represents the output of the scheduler where $I_{n}(i)=1$
if sensor $n$ is selected to sample and transmit, and $I_{n}(i)=0$ otherwise.
The sensors transmit updates with different sizes, where $L_{n}$ is the size
of the transmitted packet by sensor $n$. at task number $i$. The packets are
received successfully at the controller with probability $p\in(0,1]$, where
$r_{n}(i)$ denotes the number of transmission attempts until sensor $n$
successfully transmits its packet at task $i$. Consequently, the age of the
$n-th$ sensor’s data at the controller can be expressed as

$\begin{split}A_{n}(i+1)&=I_{n}(i)\sum_{k=1}^{r_{n}(i)}\frac{L_{n}}{\lambda_{k}(i)}\\\
&+(1-I_{n}(i))\left(A_{n}(i)+\sum_{m=1}^{N}I_{m}(i)\sum_{k=1}^{r_{m}(i)}\frac{L_{m}}{\lambda_{k}(i)}\right)\end{split}$
(1)

where $\lambda_{k}(i)$ is the average rate at the $k-th$ transmission trial at
the time of task $i$. The calculation of the age of sensor $n$ in (1) can be
explained as follows. When sensor $n$ is selected by the controller at task
$i$, i.e., $I_{n}(i)=1$, it is corresponding AoI at the controller side at
task $i+1$ is equal to the time spent in transmitting its packet. When other
sensor node $m\neq n$ is selected for transmission at task $i$, the AoI of
sensor $n$ at task $i+1$ will be equal to the AoI at task $i$ plus the time
spent in transmitting the packet from sensor $m$.

Our goal is to optimize the information freshness of the sensor nodes at the
controller to improve performance of real-time status updates in mission-
critical scenarios. Therefore, we formulate the optimization problem, in which
the objective function considers both latency and reliability through a
weighted sum of two AoI-based metrics as

$\begin{split}\text{{minimize}}\,&\lim_{T\to\infty}\frac{1}{TN}\sum_{n=0}^{N-1}\sum_{i=0}^{T-1}A_{n}(i)\\\
&+\sum_{n=0}^{N-1}\delta_{n}\sum_{i=0}^{T-1}\Pr[A_{n}(i)>\beta_{n}]\\\
\text{{subject to}}\,\,&\eqref{AoI-n},\\\
&\sum_{n=0}^{N-1}I_{n}(i)=1,\,\forall i\in\\{0,1,...,T-1\\},\\\
&I_{n}(i)\in\\{0,1\\},\,\forall n\in\\{0,1,...,N-1\\},\end{split}$ (2)

where $\beta_{n}$ is the AoI threshold of sensor $n$, where we assume that the
sensors tolerate different AoI limits. $\delta_{n}$ denotes the weight of
sensor $n$ that quantifies the penalty of exceeding its specified AoI
threshold. The problem in (2) is a stochastic optimization problem with non-
convex constraints that is known to be NP-hard. In the following, we introduce
our proposed algorithm to solve the formulated optimization problem using DRL.

In our proposed method, each node (the agent) observes the state of the
environment $s_{i}$, takes an action $a_{i}$ that leads to receiving a reward
$R_{i}$ and the transition to a new state $s_{i+1}$. The state $s_{i}$
includes the value of the AoI $A_{n}(i),\,\forall n,i$, the transmission time
of the last packet denoted as $b(i-1)$ and the throughput that was achieved at
the last $j$ tasks $\lambda(i-j+1),...,\lambda(i+1)$. The action $a(i)$ at the
$i-th$ task represents the selection of the sensor $n$ that will transmit in
the next task. The set of actions is represented by a vector of
probabilities$\text{{P}}=\\{p(0),p(2),...,p(N-1)\\}$ of length $N$, such that
the action $a(i)$ is to select the sensor $n$ corresponds to the maximum
element of P. The reward function $R_{i}$ is formulated as follows

$R_{i}=\underbrace{-\sum_{n=0}^{N-1}A_{n}(i)}_{L_{1}}-\underbrace{\sum_{n=0}^{N-1}\delta_{n}\cdot\text{{1}}_{A_{n(i)>\beta_{n}}}}_{L_{2}},$
(3)

where $\text{{1}}_{(.)}$ is an indicator function. The first term $L_{1}$ in
(3) represents the sum of AoIs of all sensors at the end of the task $i$. The
second term $L_{2}$ is the weighted sum of the penalties when exceeding the
AoI thresholds. Therefore, the reward reflects the performance of each action
with respect to the metric we need to optimize.

Figure 2: Structure of the A3C algorithm in our proposed method.

In our proposed algorithm, first, the agent explores the environment through a
training phase based on the Asynchronous Advantage Actor-Critic (A3C)
algorithm [16]. The structure of the A3C method is shown in Fig. 2 and is
described in details as follows. The A3C is a model-free RL method that is
used for directly updating a stochastic policy which runs multiple agents
asynchronously with each agent having it’s own neural network. A3C algorithm
is characterized by learning separate functions, one for the actor which is
the parameterized policy $\pi_{\theta}(s_{i},a_{i})$ and the other one for the
critic which is the value function $V^{\pi_{\theta}}(s_{i},\theta)$, where
$\theta$ is the policy parameter. The agent observes the environment through a
set of metrics and feeds their values to the neural network as depicted in
Fig. 2. The critic takes the role of the value function and evaluates the
performance of the actor, hereby helping with the estimation of the gradient
to use for the actor’s updates. Hence, for each choice of the parameter
$\theta$, we get the policy (probability over actions)
$\pi_{\theta}(s_{i},a_{i})\leftarrow[0,1]$, where $\pi_{\theta}(s_{i},a_{i})$
is assumed to be differentiable over $\theta$, i.e.,
$\frac{\partial\pi_{\theta}(s_{i},a_{i})}{\partial\theta}$ exists. Gradient
updates are performed to find the parameters that lead to maximal rewards
using the policy gradient method [17]. The method estimates the gradient of
the cumulative discounted reward with respect to the policy parameter as

$\bigtriangledown
E_{\pi_{\theta}}\\{\sum_{i=0}^{T-1}\sum_{j=0}^{T-1}\gamma^{i}R_{i+j}\\}=E_{\pi_{\theta}}\\{\bigtriangledown\text{log}\,\pi_{\theta}(s_{i},a_{i})D^{\pi_{\theta}}(s_{i},a_{i}),$
(4)

where $\gamma\in[0,1]$ is the discount rate and
$D^{\pi_{\theta}}(s_{i},a_{i})$ is the advantage function [16] that captures
how better an action is compared to the expected one selected according to the
policy. The agent estimates $D^{\pi_{\theta}}(s_{i},a_{i})$ by sampling a
trajectory of scheduling decisions and uses the empirically computed advantage
as an unbiased estimate of $D^{\pi(\theta)}(s_{i},a_{i})$. Then, the learning
parameter $\theta$ is updated as follows

$\theta\leftarrow\theta+\alpha\sum_{i=0}^{T-1}\bigtriangledown\text{log}\,\pi_{\theta}(s_{i},a_{i})D^{\pi_{\theta}}(s_{i},a_{i}),$
(5)

where $\alpha$ is the learning rate. The advantage function
$D^{\pi_{\theta}}(s_{i},a_{i})$ is computed based on the estimation of the
value function of the current state $v^{\pi_{\theta}}(s,\theta)$. The value of
$v^{\pi_{\theta}}(s,\theta)$ is calculated by the critic network as the
expected total reward starting at state $s$ and following the policy
$\pi_{\theta}$. The critic network is trained based on the Temporal Difference
(TD) method [18] by bootstrapping from the current estimate of the value
function. Therefore, the update $\theta_{v}$ of the learning parameter is
given as

$\theta_{v}\leftarrow\theta_{v}+\grave{\alpha}\sum_{i=0}^{T-1}\left(R_{i}+\gamma
V^{\pi_{\theta}}(s_{i+1},\theta_{v})-V^{\pi_{\theta}}(s_{i},\theta_{v})\right)^{2},$
(6)

where $\grave{\alpha}$ is the learning rate of the critic network and
$V^{\pi_{\theta}}$ is the estimate of $v^{\pi_{\theta}}$. Hence, for a given
$(R_{i},s_{i},a_{i},s_{i+1})$, the advantage function
$D^{\pi_{\theta}}(s_{i},a_{i})$ is estimated as $R_{i}+\gamma
V^{\pi_{\theta}}(s_{i+1},\theta_{v})-V^{\pi_{\theta}}(s_{i},\theta_{v})$.

##  IV Performance Evaluation

In this section, we first describe the simulation environment of our proposed
method, then we introduce and discuss the simulation results.

TABLE I: Evaluation parameters. Parameter | Value
---|---
Network size ($N$) |  $10$ nodes
Packet length | [$10\text{\,}\mathrm{-}$$100\text{\,}\mathrm{B}\text{/}$]
AoI Threshold | [$20\text{\,}\mathrm{-}$$200\text{\,}\mathrm{ms}\text{/}$]
Packet drop probability |  $10$%
$\alpha$ | $0.01$
$\grave{\alpha}$ | $0.01$
$\gamma$ | $100$
$\rho$ | $5$
TABLE II: Performance Comparison | Proposed method | Benchmark
---|---|---
Normalized objective function | 1.2 | 2.1
$P_{V0}$ | 0.07% | 19.03%
$P_{V1}$ | 0.01% | 8.55%
$P_{V2}$ | 0.016% | 10.12%
$P_{V3}$ | 0.01% | 12.02%
$P_{V4}$ | 0% | 7.24%
$P_{V5}$ | 0% | 4.24%
$P_{V6}$ | 0% | 3.57%
$P_{V7}$ | 0% | 5.22%
$P_{V8}$ | 0% | 3.44%
$P_{V9}$ | 0% | 4.30%
Average AoI ($n=0$) | 11.02 | 26.34
Average AoI ($n=1$) | 18.25 | 33.66
Average AoI ($n=2$) | 20.11 | 40.73
Average AoI ($n=3$) | 28.39 | 44.18
Average AoI ($n=4$) | 30.64 | 51.22
Average AoI ($n=5$) | 34.07 | 58.68
Average AoI ($n=6$) | 30.89 | 61.48
Average AoI ($n=7$) | 37.14 | 70.29
Average AoI ($n=8$) | 40.01 | 81.98
Average AoI ($n=9$) | 39.71 | 89.53
Figure 3: CDF of the AoI of each sensor; (a) $n=0$, (b) $n=1$, …, (j) $n=9$

###  IV-A Simulation Model

We evaluate the performance of our proposed method via MATLAB simulations
using the parameters listed in Table I. We consider a field network consists
of 10 sensors (i.e., $N=10$) where each node generates packets of sizes 10 to
100 bytes (with a step size of 10). Each sensor node is attached with a
different AoI threshold within the range
[$20\text{\,}\mathrm{-}$$200\text{\,}\mathrm{ms}\text{/}$] with a step size of
$20\text{\,}\mathrm{ms}\text{/}$, where node $n=0$ has the strictest threshold
and node $n=9$ has the loosest one. We set $\delta_{n}=1000\frac{N-n}{N}$, so
that nodes with tighter AoI thresholds get more penalty. The actor-critic pair
is implemented in each node (agent) with the same neural network structure. We
use a 1D Convolutional Neural Network (1D-CNN) layer with 128 filters each of
size 8 and stride 1, followed by a hidden layer consists of 256 neurons with a
Rectified Linear Unit (ReLU) activation function [19]. The throughput of the
last 10 tasks ($j=10$) is passed to the 1D-CNN, then its output along with the
other inputs is aggregated in the hidden layer. Finally, the model has two
sets of outputs, a single linear output for the value function and 10
terminals representing the output of the policy function based on the softmax
approach. To ensure that each agent explores the action space adequately
during training, we add an entropy regularization term [16] with a weight
$\rho$ to the actor’s update rule. This term encourages exploration by pushing
$\theta$ in the direction of higher entropy. $\rho$ is set to 5 at the start
of training and decays until it reaches zero. The work proposed in [11] is
used as a benchmark to evaluate the effectiveness of our proposed method. In
the referred work, a sensor $n$ is selected at task $i$ with a probability
that is inversely proportional to its AoI threshold.

###  IV-B Results and Discussion

Here we present the simulation results and the performance comparison with
regard to the work in [11] (referred to as “Benchmark” in the results). The
obtained results are summarized in Table II and Fig. 3.

In Table II, we include three metrics for comparison, the normalized objective
function in (2), the violation probability of each sensor node
($P_{Vn}=\Pr[A_{n}(i)>\beta_{n}]$, and the average AoI of each sensor node.
From Table II, it can be observed that the proposed method achieves a 57%
lower value for the objective function compared to the Benchmark. This is
mainly due to the proposed A3C learning algorithm which guides the scheduling
policy to the minimum sum of both the average AoI and the violation
probability. Moreover, Table II shows that our proposed method outperform the
Benchmark in terms of the average AoI and the AoI violation probability for
all the sensors, even for the ones that have strict deadlines. For instance,
we can observe that node 0 (i.e., $n=0$) exceeds its AoI threshold with a
probability of 0.07% which is 99% lower than that of the same sensor using the
method in the Benchmark. Furthermore, the proposed algorithm eliminates the
AoI violations for nodes $n=4$ up to $n=9$, while the Benchmark continues to
experience high threshold violation for all sensors, e.g., at least the nodes
exceed their AoI threshold $4\%$ of the time. In the proposed method, the
nodes learn how to respect the AoI violation threshold by giving weights that
are directly proportional to the specified AoI limit, of which function is not
defined by the Benchmark.

In order to take a deeper look into the performance of the AoI, the plots in
Fig. 3 show the Cumulative Distribution Function (CDF) of the AoI of each
sensor for the two methods. From these plots, it can be clearly noticed that
the proposed method achieves a reduced AoI all the time for all sensors
compared to the Benchmark. The CDF performance of the proposed method also
confirms the improvements achieved in the AoI violation probability given in
Table II. As a summary, our results show that our proposed learning method
guides the nodes to minimize the average AoI while maintaining low AoI
violation probability as much as possible, which essential in mission-critical
application within the IoT scenarios.

##  V Conclusion

In this paper, we developed a DRL-based algorithm to improve the AoI
performance of mission-critical IoT. We formulate an optimization problem to
minimize the weighted sum of the average AoI and the probability of exceeding
AoI threshold. We employed the A3C approach to train our algorithm and solve
the formulated problem. Compared with existing work, simulation results proved
the effectiveness of our proposed method with respect to the formulated AoI-
based metric. An interesting direction for future work could be investigating
the performance of the proposed method in large-scale networks, and also
integrating a sampling process to the optimization problem and scheduling
decision process.

## Acknowledgement

This paper has received funding from the European Union’s Horizon 2020
research and innovation programme under grant agreement No. 883315.

## References

  * [1] Lin, W. Yu, N. Zhang, X. Yang, H. Zhang and W. Zhao, “A survey on internet of things: Architecture, enabling technologies, security and privacy, and applications,” IEEE Internet of Things J. , vol. 4, no. 5, pp. 1125-1142, Oct. 2017.
  * [2] H. Yang, A. Alphones, W. Zhong, C. Chen and X. Xie, “Learning-based energy-efficient resource management by heterogeneous rf/vlc for ultra-reliable low-latency industrial iot networks,” IEEE Trans. Ind. Informat., vol. 16, no. 8, pp. 5565-5576, Aug 2020.
  * [3] S. Kaul, R. Yates and M. Gruteser, “Real-time status: How often should one update?,” in Proc. IEEE INFOCOM 2012, Mar. 2012, pp. 2731-2735.
  * [4] M. K. Abdel-Aziz, C. Liu, S. Samarakoon, M. Bennis and W. Saad, “”Ultra-reliable low-latency vehicular networks: taming the age of information tail,” in Proc. IEEE Globecomm 2018, Dec. 2018, pp. 1-7.
  * [5] J. Sun, Z. Jiang, B. Krishnamachari, S. Zhou and Z. Niu, “Closed-form whittle’s index-enabled random access for timely status update,” IEEE Trans. Commun., vol. 68, no. 3, pp. 1538-1551, Mar. 2020.
  * [6] M. Moltafet, M. Leinonen and M. Codreanu, “Average aoi in multi-source systems with source-aware packet management,” IEEE Trans. Commun., vol. 69, no. 2, pp. 1121-1133, Feb. 2021.
  * [7] H. Hu, K. Xiong, G. Qu, Q. Ni, P. Fan and K. B. Letaief, “Aoi-minimal trajectory planning and data collection in uav-assisted wireless powered iot networks,” IEEE Internet of Things J. , vol. 8, no. 2, pp. 1211-1223, Jan. 2021.
  * [8] L. Hu, et el., “Optimal status update in iiot systems: An age of information violation probability perspective,” in Proc. IEEE VTC 2020, Dec. 2020, pp. 1-5.
  * [9] M. Akbari, M. R. Abedi, R. Joda, M. Pourghasemian, N. Mokari and M. Erol-Kantarci, “Age of information aware vnf scheduling in industrial iot using deep reinforcement learning,” IEEE J. Sel. Areas Commun., vol. 39, no. 8, pp. 2487-2500, Aug. 2021.
  * [10] T. Zhu, J. Li, H. Gao, Y. Li and Z. Cai, “Aoi minimization data collection scheduling for battery-free wireless sensor networks,” IEEE Trans. Mobile Comput., Early Access, Aug. 2021.
  * [11] I. Kadota, A. Sinha, and E. Modiano, “Scheduling algorithms for optimizing age of information in wireless networks with throughput constraints,” IEEE/ACM Trans. Netw, vol. 27, no. 4, pp. 359-1372, Aug. 2019.
  * [12] R. V. Bhat, R. Vaze and M. Motani, “Minimization of age of information in fading multiple access channels,” IEEE J. Sel. Areas Commun., vol. 39, no. 5, pp. 1471-1484, May 2021.
  * [13] C. Kam, S. Kompella, G. D. Nguyen, J. E. Wieselthier and A. Ephremides, “On the age of information with packet deadlines,” IEEE Trans. Inf. Theory, vol. 64, no. 9, pp. 6419-6428, Sept. 2018.
  * [14] Q. Wang, H. Chen, Y. Li, Z. Pang and B. Vucetic, “Minimizing age of information for real-time monitoring in resource-constrained industrial iot networks,” in Proc. IEEE INDIN 2019, Jul. 2019, pp. 1766-1771.
  * [15] M. K. Abdel-Aziz, S. Samarakoon, M. Bennis and W. Saad, “Ultra-reliable and low-latency vehicular communication: An active learning approach,” IEEE Commun. Lett., vol. 24, no. 2, pp. 367-370, Feb. 2020.
  * [16] V. Mnih, et el., “Asynchronous methods for deep reinforcement learning,” in Proc. ICML 2016, Jun. 2016, pp. 1928-1937.
  * [17] R. S. Sutton, et el., “Policy gradient methods for reinforcement learning with function approximation,” in Proc. NIPS 1999, Nov. 1999, pp. 1057-1063.
  * [18] A. Tan, N. Lu and D. Xiao, “Integrating temporal difference methods and self-organizing neural networks for reinforcement learning With delayed evaluative feedback,” IEEE Trans. Neural Netw., vol. 19, no. 2, pp. 230-244, Feb. 2008.
  * [19] Y. Ying, J. Su, P. Shan, L. Miao, X. Wang and S. Peng, “Rectified exponential units for convolutional neural networks,” IEEE Trans. Neural Netw., vol. 7, pp. 101633-101640, Jul. 2019.