# Outage Probability Analysis of Wireless Paths with Faulty Reconfigurable
Intelligent Surfaces

Mounir Bensalem and Admela Jukan  Technische Universität Braunschweig,
Germany; {mounir.bensalem<EMAIL_ADDRESS>
###### Abstract

We consider a next generation wireless network incorporating a base station a
set of typically low-cost and faulty Reconfigurable Intelligent Surfaces
(RISs). The base station needs to select the path including the RIS to provide
the maximum signal-to-noise ratio (SNR) to the user. We study the effect of
the number of elements, distance and RIS hardware failure on the path outage
probability, and based on the known signal propagation model at high
frequencies, derive the closed-form expression for the said probability of
outage. Numerical results show the path outage likelihood as function of the
probability of hardware failure of RIS elements, the number of elements, and
the distance between mobile users and the RIS.

##  I Introduction

Reconfigurable intelligent surfaces (RISs) are becoming an integral system
component in millimeter-wave (mmW) and Terahertz (THz) communications, aimed
at solving the problem of considerable volatility of transmission due to
physical obstacles by reflecting the wireless links around the obstacles[1,
2]. In addition, RIS partitioning into sub-surfaces, whereby each sub-surface
consists of a number of adjacent reflecting elements, allows the creation of
multiple paths for multi-user and reliability scenarios [3, 4]. On the other
hand, the susceptibility of individual RIS elements, referred to as unit cells
or meta-atoms, to failures is already a well-known concern [5]. Faulty RIS
elements can lead to degradation in the antenna radiation pattern, and in more
severe cases, impact the entire functionality of RIS meta-surface elements. To
ensure reliable RIS-aided networks in practical scenarios, it is critical to
design link and path selection schemes that account for outage due to RIS
meta-surface failures.

In this paper, we provide a novel analysis of the outage probability of
wireless paths due to a) RIS hardware failure and b) connection degradation
due to obstacles, both stationary and mobile. We analyse and obtain a closed-
form of the instantaneous channel state information (CSI) model of an end-to-
end communication system including base station as transmitter, RIS (as an
intermediate node) and user as receiver. In our approach, the base station
does not need to consider an instant CSI, and the "best" RIS can be selected
based on outdated CSI. We formulate the SNR model using the instantaneous CSI
and analyse the partial best path selection. We derive a closed-form
expression of the outage probability for an arbitrary path in terms of the
modified Bessel function of the second kind. We also study the effect of RIS
hardware failure. Thus, the closed-form expression is dependent on the failure
probability of RIS elements, the existence of physical obstacle, devices
characteristics, and the length of the individual links. We show that the
analysis can be used to quantify the impact of the blockages and penetration
losses as well as the RIS hardware failure on the outage probability for
arbitrary paths. We show that at a signal-to-noise-ratio (SNR) approaching
zero, the presence of obstacles and the failure probability of RIS elements
majorly affect the outage probability. A more reliable connection can be
obtained at a large enough RIS device vs. a communication distance, in
consideration of the RIS with strongly correlated elements and their
likelihood of failure. This is due to the outage probability having a
comparably lower values given a certain minimum number of RIS meta-surface
elements.

The rest of the paper is organized as follows. Section II presents the related
work. Section III provides the system model and Section IV the outage
probability analysis. Section V discusses numerical results. Section VI
concludes the paper.

##  II Related Work

Several studies focused on the outage probability in RIS-aided communications,
including [6, 7, 8, 9]. In [6], the authors has analysed the outage
probability of a RIS-aided system, where the RIS with highest SNR value is
selected. Their results showed that the number of RISs and the number of
reflecting elements are crutial for the capacity scaling law of multiple RIS-
aided networks. In our work, a partial path selection approach was adopted,
meaning that the paths with the best SNR value between the RIS and the user
are selected to support the communication.In [9], a large intelligent surfaces
(LISs), which can be placed in structures like walls, has been studied as a
solution for reliable communication. The outage probability of LIS was
characterized, where the closed-form was derived. The authors investigated the
distribution of uplink sum-rate asymptotically in terms of number of LIS
elements. A similar channel modeling is used in our work, and additionally the
failure of elements and links will be integrated into the closed-form of
outage probability. From a channel modelling perspective, the fading channel
is time varying, which make it important to adopt the scenario of an outdated
CSI, as the instant CSI is hard to obtain in real communication systems.
Recent work [10] adopted imperfect with outdated channel estimates to analyse
the outage probability of the best RIS selection in a RIS-assisted network for
vehicular communications, while dealing with the problem of RIS selection in a
single hop system. The closed form of outage probability was derived in
several specific communication scenario including relays [11, 12, 13], and
recently RISs [10, 14]. Similar CSI model of [10] was adopted in our work,
where we assumed equal amplitude between elements of same RIS to obtain a
closed form expression of SNR and thus of the outage. The consideration of
faulty elements for path selection was first introduced in our previous work
[15], and to the best of our knowledge, this work is the first to consider the
failure of RIS elements and connection degradation to analyse the outage
probability of the RIS-aided path. Without loss of generality the obtained
outage probability closed form can be applied to both single hop and multi-hop
system.

##  III System Model

We assume a communication scenario, consisting of a single antenna transmitter
(TX) representing a base station (B), N different RISs, and a single antenna
receiver (RX), representing a mobile user (U), as illustrated in Fig. 1.a. We
assume that an element of a RIS can fail with a probability $p$ and a
connection can degrade due the presence of an obstacle, as shown in Fig.1.b.
It is assumed that each RIS is partitioned into $J$ sub-surface block, where
each block is operating as an independed RIS located in the same position.
This assumption allows the RIS to be used to serve mutiple users by
configuring the phase shift of each RIS block in order to reflect the signal
to a certain location, denoting by $R_{k,j}$ the j${}^{\text{th}}$ block of
the k${}^{\text{th}}$ RIS. A RIS is assumed to have $M$ sub-wavelength-sized
reflecting elements, where each element has a square shape of size $L\times
L$. As the RIS has $J$ equal blocks, we denote by $M^{\prime}=\frac{M}{J}$ the
number of elements in each block. When $J$ is set to $1$, the RIS is operating
as a signle block serving a single user at a time. The direct link between $B$
and $U$ is considered not available due to obstacles, and the line-of-sight
LoS links consist of $U$-to-$R_{k,j}$ and $R_{k,j}$-to-$B$. Let the channel
coefficients of the LoS links from user $U$ by the base station $B$ of RIS
block $R_{k,j}$, be denoted as a vector
$\textbf{h}_{R_{k,j}B}=[h_{R_{k,j}B}^{1},..h_{R_{k,j}B}^{e},..h_{R_{k,j}B}^{M^{\prime}}]$,
where $h_{R_{k,j}B}^{e}$ is the channel coefficient related to the element
$e$, and
$\textbf{h}_{UR_{k,j}}=[h_{UR_{k,j}}^{1},...,h_{UR_{k,j}}^{e},...,h_{UR_{k,j}}^{M^{\prime}}]$,
where $k\in\\{1,...,N\\}$, $j\in\\{1,...,J\\}$, and
$e\in\\{1,...,M^{\prime}\\}$.

The received signal at the user $U$ from base station $B$ through the RIS
block $R_{k,j}$ is expressed as:

$y_{U(R_{k,j})}=\textbf{h}_{UR_{k,j}}^{T}\textbf{C}_{(k,j)}^{\frac{1}{2}}\Psi_{(k,j)}\textbf{C}_{(k,j)}^{\frac{1}{2}}\textbf{h}_{R_{k,j}B}x+N_{U(R_{k,j})}\\\
=\left(\sum_{l=1}^{M^{\prime}}\sum_{s=1}^{M^{\prime}}\sum_{m=1}^{M^{\prime}}\sqrt{a_{l,s(k,j)}a_{l,m(k,j)}}h_{UR_{k,j}}^{s}h_{R_{k,j}B}^{m}e^{i\Psi_{m(k,j)}}\right)\\\
\times x+N_{U(R_{k,j})}$ (1)

where $x$ is the transmitted signal received by the user U and sent from the
base station B with transmit power P, $N_{U(R_{k,j})}$ is the AWGN power at
user U. We denote by
$\textbf{C}_{(k,j)}=\\{a_{l,s(k,j)}\\}_{l,s\in[1,M^{\prime}]^{2}}$ the
correlation coefficients matrix, and $a_{l,s(k,j)}$ is the correlation
coefficient between the $l^{th}$ and $s^{th}$ elements of $j^{th}$ block of
the $k^{th}$ RIS $R_{k,j}$, such that $a_{l,s(k,j)}=1$ if $l=s$, and
$a_{l,s(k,j)}\in[0,1],\forall i,j\in[\\![1,n]\\!]^{2}$. $\Psi_{(k,j)}$ is the
phase matrix induced by $R_{k,j}$, and $\Psi_{m(k,j)}$ describes the $m^{th}$
adjustable phase induced by the $j^{th}$ block of the $k^{th}$ RIS $R_{k,j}$.

Figure 1: The reference RIS-assited wireless network system: a. system
architecture, b. hardware failure and connection degradation case.

###  III-A CSI Model

The channel coefficients of the $U$-to-$R_{k,j}$ and $R_{k,j}$-to-$B$ links
through the element $e$ are defined as follows:

$\begin{split}h_{UR_{k,j}}^{e}=&g_{UR_{k,j}}e^{-i\theta_{k,j}^{e}}\sqrt{(d_{UR_{k,j}})^{-\delta_{k,j}(f)}},\\\
h_{R_{k,j}B}^{e}=&g_{R_{k,j}B}e^{-i\Phi_{k,j}^{e}}\sqrt{(d_{R_{k,j}B})^{-\delta_{k,j}(f)}}\end{split}$
(2)

where $g_{UR_{k,j}}$ and $g_{R_{k,j}B}$ represents, respectively, the gains
for the $U$-to-$R_{k,j}$ and $R_{k,j}$-to-$B$ links, assuming that all
elements of the block has the same gain, $e^{-i\theta_{k,j}^{e}}$ and
$e^{-i\Phi_{k,j}^{e}}$ represents the normalized LoS channel states, with
distances $d_{UR_{k,j}}$ and $d_{R_{k,j}B}$ and the channel phases
$\theta_{k,j}^{e}$ and $\Phi_{k,j}^{e}$ for the $U$-to-$R_{k,j}$ and
$R_{k,j}$-to-$B$ links through the element $e$, $\delta_{k,j}(f)$ is the path
loss exponent for the operating frequency $f$ of the signal.
We assume that the distance between RIS blocks is much lower that the distance
between the base station $B$ and RISs and between the user $U$ and RISs, thus
the differene of power gain between the blocks of the same RIS is negligeable
and can be considered equal.

The channel coefficient at the user $U$ through RIS block $R_{k,j}$ can be
assumed as, similar to [10], but assuming that the gain is similar at all
elements:

$h_{U(R_{k,j})}=g_{UR_{k,j}}g_{R_{k,j}B}\\\
\times\left(\sum_{l=1}^{M^{\prime}}\sum_{s=1}^{M^{\prime}}\sum_{m=1}^{M^{\prime}}\sqrt{a_{l,s(k,j)}a_{l,m(k,j)}(d_{UR_{k,j}}d_{R_{k,j}B})^{-\delta_{k,j}}}\right.\\\
\times\left.e^{i(\Psi_{m(k,j)}-\theta_{k,j}^{s}-\Phi_{k,j}^{m})}\right)$ (3)

###  III-B SNR model

The instantaneous SNR between the user $U$ and the block $j$ of RIS $k$ is
given by:

$\iota_{UR_{k,j}}=\frac{P\nu|h_{UR_{k,j}}|^{2}}{N_{0}}$ (4)

where the transmitter sends with source power $P$ and $N_{0}$ denotes the
additive white Gaussian noise (AWGN) power, $\nu$ is a coefficient related to
the impact of the existence of an obstacle, and $h_{UR_{k,j}}$ is defined by
eq. (2).

Considering the selected path from the base station B to the user U, and using
the channel definition in eq. (3), the instantaneous SNR corresponding to the
block $j$ of RIS $k$ is given as:

$\overline{\iota}_{U(R_{k,j})}=\frac{P\nu|h_{U(R_{k,j})}|^{2}}{N_{0}}\\\
=\Omega\cdot|g_{UR_{k,j}}|^{2}|g_{R_{k,j}B}|^{2}|\sum_{l=1}^{M^{\prime}}\sum_{s=1}^{M^{\prime}}\sum_{m=1}^{M^{\prime}}\zeta_{l,s,m,U}^{k,j}|^{2}$
(5)

where $\Omega$ and $\zeta_{l,s,m,U}^{k,j}$ are defined as follows:

$\Omega=\frac{P\nu}{N_{0}}\;\;\;\;\text{and}\\\
\zeta_{l,s,m,U}^{k,j}=\sqrt{a_{l,s(k,j)}a_{l,m(k,j)}(d_{UR_{k,j}}d_{R_{k,j}B})^{-\delta_{k,j}}}\\\
\times e^{i(\Psi_{m(k)}-\theta_{k,j}^{s}-\Phi_{k,j}^{m})}$ (6)

After simplifying the term with square of summations, the previous form can be
re-written as follows:

$\begin{split}\overline{\iota}_{U(R_{k,j})}=&\Omega\cdot|g_{UR_{k,j}}|^{2}|g_{R_{k,j}B}|^{2}\left[\sum_{l=1}^{M^{\prime}}\sum_{s=1}^{M^{\prime}}\sum_{m=1}^{M^{\prime}}|\zeta_{l,s,m,U}^{k,j}|^{2}\right.\\\
&\left.+2\sum_{(l,s,m,l^{\prime},s^{\prime},m^{\prime})\in\Xi}^{M^{\prime}}\eta_{l,s,m,l^{\prime},s^{\prime},m^{\prime},U}^{k,j}\right]\end{split}$
(7)

where
$|\zeta_{l,s,m,U}^{k,j}|^{2}=a_{l,s(k,j)}a_{l,m(k,j)}(d_{UR_{k,j}}d_{R_{k,j}B})^{-\delta_{k,j}},$

$\eta_{l,s,m,l^{\prime},s^{\prime},m^{\prime},U}^{k,j}=\zeta_{l,s,m,U}^{k,j}\zeta_{l^{\prime},s^{\prime},m^{\prime},U}^{k,j}\\\
=\sqrt{a_{l,s(k,j)}a_{l,m(k,j)}a_{l^{\prime},s^{\prime}(k,j)}a_{l^{\prime},m^{\prime}(k,j)}}(d_{UR_{k,j}}d_{R_{k,j}B})^{-\delta_{k,j}}\\\
\times
cos((\Psi_{m(k,j)}-\theta_{k,j}^{s}-\Phi_{k,j}^{m})-(\Psi_{m^{\prime}(k,j)}-\theta_{k,j}^{s^{\prime}}-\Phi_{k,j}^{m^{\prime}}))$
(8)

and
$\sum_{(l,s,m,l^{\prime},s^{\prime},m^{\prime})\in\Xi}^{M^{\prime}}=\sum_{l=1}^{M^{\prime}}\sum_{s=1}^{M^{\prime}}\sum_{m=1}^{M^{\prime}}\sum_{l^{\prime}=1}^{M^{\prime}}\sum_{s^{\prime}=1}^{M^{\prime}}\sum_{m^{\prime}=1}^{M^{\prime}}$
where $\Xi$ follows the conditions given as:

$\Xi=\begin{cases}(l\neq l^{\prime})\cap(s\neq s^{\prime})\cap(m\neq
m^{\prime})\\\ m<l^{\prime}\\\ (m=l^{\prime})\cap(l<s)\cap(m^{\prime}\leq
s^{\prime})\\\ (m\leq l^{\prime})\cap(l\leq
s)\cap(m^{\prime}>s^{\prime})\end{cases}$ (9)

###  III-C Element failure model

We consider the possibility of RIS element failure. Let $1_{k,j}^{e}$ be an
indicator of whether an element $e$ fails or not; $1_{k,j}^{e}=1$ if element
$e$ in RIS block $R_{k,j}$ does not fail and $0$ otherwise. We denoted by
$P[1_{k,j}^{e}=0]=p,\forall e\in[1,...,M^{\prime}]$ the probability that
element $e$ of a RIS block fails, $Q$ be the number failed elements.

######  Proposition 1.

The probability mass function $P_{Q;M^{\prime}}(q)$ that $q$ element out of
$M^{\prime}$ fails is given as follows:

$P(Q=q)=C_{M^{\prime}}^{q}p^{q}(1-p)^{M^{\prime}-q}$ (10)

###### Proof.

The random variable $Q$ is assumed to follow the binomial distribution with
parameters $M^{\prime}\in\mathbb{N}$ and $p\in[0,1]$,
$Q\sim\mathcal{B}(M^{\prime},p)$. The probability of getting exactly $q$
failed element is equivalent to getting $q$ successes in $M^{\prime}$
independent Bernoulli trials, with the same rate $p$, and can be given by the
probability mass function. ∎

###  III-D Path selection

We assume that the base station $B$ has a perfect knowledge of the RF
channels, which allows it to select the RIS providing the highest CSI. The
base station periodically receives CSIs feedback from the $N$ RISs considering
that some RISs could fail due to an obstacle.

In this paper, we consider that a RIS Controller can read the signal received
at the RIS with an active element, decide the best RIS block to select and
forward the information to the base station $B$ to adjust the arrays
accordingly. We assume that only the user to RIS link is considered for the
selection, and the selection RIS block is given by:

$(k^{*},j^{*})=\arg\max_{k,j}(\overline{\iota}_{UR_{k,j}}(T))$ (11)

Thus, the selected path in a 1 hop scenario, would be $B\rightarrow
R_{k^{*},j^{*}}\rightarrow U$.

##  IV Outage Probability Analysis

The outage probability of the communication channel is defined as the
probability that a certain target data rate cannot be supported by the system.
It can be computed as the the probability that the overall SNR
$\overline{\iota}_{U(R_{k,j})}$ being lower than a threshold,
$\gamma_{T}=2^{2r}-1$, where $r$ represents the target rate.

###  IV-A Outage considering connection degradation

######  Proposition 2.

The outage probability of a RIS-assisted path is given by eq. (17), where
$K_{1}(\cdot)$ is the 1${}^{\text{st}}$ order modified Bessel function of the
second kind.

###### Proof.

In our mobile scenario, the outage probability is defined as the cdf
$F_{\overline{\iota}_{U(R_{k,j})}}(\gamma_{T})$, given by:

$F_{\overline{\iota}_{U(R_{k,j})}}(\gamma_{T})=\text{ Pr
}(\overline{\iota}_{U(R_{k,j})}<\gamma_{T})\\\ =\text{ Pr
}(|g_{UR_{k,j}}|^{2}|g_{R_{k,j}B}|^{2}<\frac{\gamma_{T}}{\Upsilon_{U,k,j}})\\\
=\int_{0}^{\infty}\text{Pr
}\left(|g_{UR_{k,j}}|^{2}y<\frac{\gamma_{T}}{\Upsilon_{U,k,j}}\right)f_{|g_{R_{k,j}B}|^{2}}(y)dy\\\
=\int_{0}^{\infty}F_{|g_{UR_{k,j}}|^{2}}\left(\frac{\gamma_{T}}{y\Upsilon_{U,k,j}}\right)f_{|g_{R_{k,j}B}|^{2}}(y)dy$
(12)

where the parameter $\Upsilon_{U,k,j}$ is defined as:

$\begin{split}\Upsilon_{U,k,j}=&\Omega\left[\sum_{l=1}^{M^{\prime}}\sum_{s=1}^{M^{\prime}}\sum_{m=1}^{M^{\prime}}|\zeta_{l,s,m,U}^{k,j}|^{2}\right.\\\
&\left.+2\sum_{(l,s,m,l^{\prime},s^{\prime},m^{\prime})\in\Xi}^{M^{\prime}}\eta_{l,s,m,l^{\prime},s^{\prime},m^{\prime},U}^{k,j}\right]\end{split}$
(13)

We assume that the outdated CSI $\tilde{h}_{UR_{k,j}}$ and
$\tilde{h}_{R_{k,j}B}$ are following the Rayleigh distribution [16], assuming
that all elements of the same RIS block have the same CSI. Assuming that
$|\tilde{g}_{UR_{k,j}}|^{2}\sim\text{Exp(}\frac{1}{\lambda_{U}}\text{)}$ where
$\frac{1}{\lambda_{U}}$ represents the rate parameter i.e. $\lambda_{U}$
represents the average SNR per link. In addition,
$|\tilde{g}_{R_{k,j}B}|^{2}\sim\text{Exp(}\frac{1}{\lambda_{B}}\text{)}$ where
$\frac{1}{\lambda_{B}}$ is the rate parameter. The PDF and the CDF of the
outdated CSI corresponding to one reflecting element from the block $j$ of RIS
$k$ of the first link from user U to RIS are given by:

$f_{|\tilde{g}_{UR_{k,j}}|^{2}}(x)=\frac{1}{\lambda_{U}}e^{-\frac{x}{\lambda_{U}}}$
(14) $F_{|\tilde{g}_{UR_{k,j}}|^{2}}(x)=1-e^{-\frac{x}{\lambda_{U}}}$ (15)

The distributions of the outdate CSI $|\tilde{g}_{UR_{k,j}}|^{2}$ and the
instantaneous CSI $|g_{UR_{k,j}}|^{2}$ are correlated exponential
distributions [11], thus the joint pdf is given by:

$f_{|g_{UR_{k,j}}|^{2},|\tilde{g}_{UR_{k,j}}|^{2}}(x,y)=\frac{e^{-\frac{x+y}{(1-\rho_{1}^{2})\lambda_{U}}}}{(1-\rho_{1}^{2})\lambda_{U}^{2}}I_{0}\left(\frac{2\sqrt{\lambda_{U}^{2}xy}}{(1-\rho_{1}^{2})\lambda_{U}}\right)$
(16)

where $I_{0}(\^{A}\textperiodcentered)$ is the zero-order modified Bessel
function of the first kind.

$\displaystyle F_{\overline{\iota}_{U(R_{k,j})}}(\gamma_{T})=$
$\displaystyle\frac{N^{2}J^{2}(1-\rho_{1}^{2})}{\lambda_{B}}\sum_{s=0}^{NJ-1}\sum_{s^{\prime}=0}^{NJ-1}\frac{(-1)^{s+s^{\prime}}C^{s}_{NJ-1}C^{s^{\prime}}_{NJ-1}}{(\lambda_{U}^{2}+s+s\rho_{1}^{2}+1)(s^{\prime}+s^{\prime}\rho_{2}^{2}+1)}\left[\frac{(s^{\prime}+s^{\prime}\rho_{2}^{2}+1)(1-\rho_{2}^{2})\lambda_{B}}{(\lambda_{B}^{2}+s^{\prime}+s^{\prime}\rho_{2}^{2}+1)}\right.$
(17)
$\displaystyle-\sqrt{\frac{4(\lambda_{U}^{2}+s+s\rho_{1}^{2}+1)(s^{\prime}+s^{\prime}\rho_{2}^{2}+1)(1-\rho_{2}^{2})\lambda_{B}\gamma_{T}}{(\lambda_{B}^{2}+s^{\prime}+s^{\prime}\rho_{2}^{2}+1)(s+s\rho_{1}^{2}+1)(1-\rho_{1}^{2})\lambda_{U}\Upsilon_{U,k,j}}}$
$\displaystyle\left.\times
K_{1}\left(\sqrt{\frac{4(\lambda_{U}^{2}+s+s\rho_{1}^{2}+1)(\lambda_{B}^{2}+s^{\prime}+s^{\prime}\rho_{2}^{2}+1)\gamma_{T}}{(s+s\rho_{1}^{2}+1)(1-\rho_{1}^{2})(s^{\prime}+s^{\prime}\rho_{2}^{2}+1)(1-\rho_{2}^{2})\lambda_{B}\lambda_{U}\Upsilon_{U,k,j}}}\right)\right]$

The pdf of the instantaneous SNR of the link between the user U and the block
$j$ of RIS $k$ is given as:

$f_{|g_{UR_{k,j}}|^{2}}(x)=\int_{0}^{\infty}f_{|g_{UR_{k,j}}|^{2}|\;|\tilde{g}_{UR_{k,j}}|^{2}}(x|y)f_{|\tilde{g}_{UR_{k,j}}|^{2}}(y)dy$
(18)

The conditional pdfs
$f_{|g_{UR_{k,j}}|^{2}|\;|\tilde{g}_{UR_{k,j}}|^{2}}(.|.)$ are all identical
for all $k,j$ due to the fact that $|g_{UR_{k,j}}|^{2}$ and
$|\tilde{g}_{UR_{k,j}}|^{2}$ are i.i.d. RVs. Thus the conditional pdfs can be
expressed as follows:

$f_{|g_{UR_{k,j}}|^{2}|\;|\tilde{g}_{UR_{k,j}}|^{2}}(x|y)=\frac{f_{|g_{UR_{k,j}}|^{2},|\tilde{g}_{UR_{k,j}}|^{2}}(x,y)}{f_{|\tilde{g}_{UR_{k,j}}|^{2}}(y)}$
(19)

Thus, we substitute (14) and (16) into (19) to obtain the following
expression:

$f_{|g_{UR_{k,j}}|^{2}|\;|\tilde{g}_{UR_{k,j}}|^{2}}(x|y)=\frac{e^{-\frac{x+\rho_{1}^{2}y}{(1-\rho_{1}^{2})\lambda_{U}}}}{(1-\rho_{1}^{2})\lambda_{U}}I_{0}\left(\frac{2\sqrt{\lambda_{U}^{2}xy}}{(1-\rho_{1}^{2})\lambda_{U}}\right)$
(20)

The cdf of outdated CSI $|\tilde{g}_{UR_{k,j}}|^{2}$ can be given as follows:

$F_{|\tilde{g}_{UR_{k,j}}|^{2}}(x)=\sum_{i=1}^{N}\sum_{s=1}^{J}\text{Pr}(|\tilde{g}_{UR_{k,j}}|^{2}\leq
x\text{ }\cap k=i,j=s)\\\ =NJ\text{ Pr}(|\tilde{g}_{UR_{k,j}}|^{2}\leq x\text{
}\cap k=i,j=s)\\\ =NJ\int_{0}^{x}f_{|\tilde{g}_{UR_{k,j}}|^{2}}(y)\text{
Pr}(k=i,j=s|\;|\tilde{g}_{UR_{k,j}}|^{2}\leq y)dy$ (21)

Given that the best selection considers only the link between the user $U$ and
the RIS devices, the link between the base station $B$ and the RISs is not
taken into consideration. In addition, we consider all paths have the same
distribution and using eq. (15), the cdf can be given as:

$F_{|\tilde{g}_{UR_{k,j}}|^{2}}(x)=NJ\int_{0}^{x}f_{|\tilde{g}_{UR_{k,j}}|^{2}}(y)[F_{\tilde{h}_{UR}}(y)]^{NJ-1}dy\\\
=NJ\int_{0}^{x}f_{|\tilde{g}_{UR_{k,j}}|^{2}}(y)[1-e^{-\frac{y}{\lambda_{U}}}]^{NJ-1}dy\\\
=NJ\sum_{s=0}^{NJ-1}(-1)^{s}C^{s}_{NJ-1}\int_{0}^{x}f_{|\tilde{g}_{UR_{k,j}}|^{2}}(y)e^{-\frac{sy}{\lambda_{U}}}dy\\\
=NJ\sum_{s=0}^{NJ-1}\frac{(-1)^{s}C^{s}_{NJ-1}}{s+1}(1-e^{-\frac{(s+1)x}{\lambda_{U}}})$
(22)

Hence, we differentiate the obtained cdf $F_{|\tilde{g}_{UR_{k,j}}|^{2}}(x)$
to get the outdated CSI pdf given by:

$\begin{split}f_{|\tilde{g}_{UR_{k,j}}|^{2}}(x)=&\frac{d}{dx}F_{|\tilde{g}_{UR_{k,j}}|^{2}}(x)\\\
=&\frac{NJ}{\lambda_{U}}\sum_{s=0}^{NJ-1}(-1)^{s}C^{s}_{NJ-1}e^{-\frac{(s+1)x}{\lambda_{U}}}\end{split}$
(23)

By replacing eq. (23) and eq. (20) into eq. (24) we derive the pdf of the
instantaneous CSI as:

$f_{|g_{UR_{k,j}}|^{2}}(x)=\int_{0}^{\infty}f_{|g_{UR_{k,j}}|^{2}|\tilde{g}_{UR_{k,j}}|^{2}}(x|y)f_{|\tilde{g}_{UR_{k,j}}|^{2}}(y)dy\\\
=\frac{NJ}{(1-\rho_{1}^{2})\lambda_{U}^{2}}\sum_{s=0}^{NJ-1}(-1)^{s}C^{s}_{NJ-1}\int_{0}^{\infty}e^{-\frac{x+(s+s\rho_{1}^{2}+1)y}{(1-\rho_{1}^{2})\lambda_{U}}}\\\
\times
I_{0}\left(\frac{2\sqrt{\lambda_{U}^{2}xy}}{(1-\rho_{1}^{2})\lambda_{U}}\right)dy$
(24)

Using the identity in [17] eq. (6.643.4), where we set
$n=0,\nu=0,\alpha=\frac{s+s\rho_{1}^{2}+1}{(1-\rho_{1}^{2})\lambda_{U}},\beta\frac{\sqrt{x}}{1-\rho_{1}^{2}}$,
the closed form of the pdf is given as:

$f_{|g_{UR_{k,j}}|^{2}}(x)=\frac{NJ}{\lambda_{U}}\sum_{s=0}^{NJ-1}\frac{(-1)^{s}C^{s}_{NJ-1}}{s+s\rho_{1}^{2}+1}\times
e^{-\frac{(\lambda_{U}^{2}+s+s\rho_{1}^{2}+1)x}{(s+s\rho_{1}^{2}+1)(1-\rho_{1}^{2})\lambda_{U}}}$
(25)

Using the closed form of the pdf in (25), The cdf of $|g_{UR_{k,j}}|^{2}$ is
given as:

$F_{|g_{UR_{k,j}}|^{2}}(x)=NJ(1-\rho_{1}^{2})\sum_{s=0}^{NJ-1}\frac{(-1)^{s}C^{s}_{NJ-1}}{\lambda_{U}^{2}+s+s\rho_{1}^{2}+1}\\\
\times\left(1-e^{-\frac{(\lambda_{U}^{2}+s+s\rho_{1}^{2}+1)x}{(s+s\rho_{1}^{2}+1)(1-\rho_{1}^{2})\lambda_{U}}}\right)$
(26)

Due to symmetry, the closed form of the CSI pdf and cdf of
$|g_{UR_{k,j}}|^{2}$ and $|\tilde{g}_{UR_{k,j}}|^{2}$ is given by (25) and
(26), respectively, by interchaning $\rho_{1}$ with $\rho_{2}$ and
$\lambda_{U}$ with $\lambda_{B}$.
Hence, we can evaluate the outage probability by substituting (26) and (25)
into (12) and using the identity in [17] eq. (3.324.1), where we set
$\gamma=\frac{(\lambda_{B}^{2}+s^{\prime}+s^{\prime}\rho_{2}^{2}+1)}{(s^{\prime}+s^{\prime}\rho_{2}^{2}+1)(1-\rho_{2}^{2})\lambda_{B}}$
and
$\beta=\frac{4(\lambda_{U}^{2}+s+s\rho_{1}^{2}+1)\gamma_{T}}{(s+s\rho_{1}^{2}+1)(1-\rho_{1}^{2})\lambda_{U}\Upsilon_{U,k,j}}$,
yielding eq. (17).

∎

###  IV-B Outage considering Element failure

Considering that in practical system, the RIS hardware is vulnerable to
hardware failures, we assume that the outage probability can be dependent on
the state of the RIS elements.

Without loss of generality, we assume that the first $q$ elements fail, which
means that we have $M^{\prime}-q$ operating elements in RIS block $R_{k,j}$
starting from element index $q+1$.
Using the theorem of total probability, the outage probability
$F_{\overline{\iota}_{U(R_{k,j})}}(\gamma_{T})$ can be given by:

$F_{\overline{\iota}_{U(R_{k,j})}}(\gamma_{T})=\sum_{q=0}^{M^{\prime}}P(Q=q)F_{\overline{\iota}_{U(R_{k,j})}}^{q}(\gamma_{T})$
(27)

where $F_{\overline{\iota}_{U(R_{k,j})}}^{q}(\gamma_{T})$ is the outage
probability when $k$ elements of the RIS block $R_{k,j}$ are failed.

##  V Numerical Evaluation

In this section, numerical results are presented to verify and validate the
correctness of the obtained closed form expressions and evaluate the impact of
the main system parameters on the performance and reliability. We consider a
set of $N=4$ RISs, with $M=30$ elements, and $J=4$ partitions. We consider a
source power $P=30$ dB, a noise AWGN power $N_{0}=10$ dB, a correlation
coefficient between instantaneous and outdated CSI of user-RIS link and RIS-BS
link $\rho_{1}=\rho_{2}=0.1$, the distance $d_{UR_{k,j}}=d_{R_{k,j}B}=4$ m,
and the phase shifts are random.

Fig. 2 shows the outage of the best RIS as a function of the normalized
average SNR of U-RIS and RIS-B, for different value of element failure
probability. The results shows that the outage probability decreases as the
SNR increases for all values of failure. Also, it increase with the increase
of failure probability $p$ of each RIS element, which quantifies how a lower
number of elements is impacting the transmitted signal.

Figure 2: Outage probability of best RIS vs the normalized average SNR of
U-RIS and RIS-B, with different probability of failures, for $d_{UR_{k,j}}=4$
m.

Fig. 3 shows the outage of the best RIS as a function of the normalized
average SNR of U-RIS and RIS-B, for different value of element failure
probability and considering the presence or abscence of an obstacle with
$\nu=0.1$. It is observed that the presence of obstacle start impacting the
outage when the average SNR is slighly higher than $0.5$ dB, and a higher
element failure cause a higher outage probability.

Figure 3: Outage probability of best RIS vs the normalized average SNR of
U-RIS and RIS-B, with different reliability scenarios, for $d_{UR_{k,j}}=4$ m.

Fig. 4 shows the outage of the best RIS as a function of number of elements,
where $\gamma_{T}=3$ and $\lambda_{U}=0.05$, considering different values of
$\rho$, the correlation coefficient between outdated and instantaneous CSI,
and correlation between elements. For weakly correlated elements and $\rho\leq
0.5$, the outage is high for all sizes of RIS, and low when the correlation is
strong. For strongly correlation elements, we observe that the outage
decreases at a certain number of elements, $M=50$ for a 4 m distance, meaning
that the received power achieves the target SNR at a certain RIS size.

Figure 4: Outage probability of best RIS vs number of elements $M$

Fig. 5 shows the outage of the best RIS as a function of user-RIS distance,
with and without obstacle. For $\rho\leq 0.5$, the outage is high for distance
higher than 3 m in presence of obstacle and 5 m without an obstacle. When
$\rho=0.9$, the correlation is strong and the outage is low for all distances.

Figure 5: Outage probability of best RIS vs distance $d_{UR_{k,j}}$ with and
without obstacle, $\gamma_{T}=3$ and $\lambda_{U}=0.05$

##  VI Conclusion

In this paper, we proposed a novel analysis of the outage probability of
wireless paths, considering RIS hardware failure and connection degradation
due to obstacles. We analysed and obtained a closed-form of the instantaneous
CSI model of a communication system with a base station, RISs and user. We
formulate the SNR model using the instantaneous CSI and analyse the partial
best path selection, using the "best" RIS selected based on outdated CSI. We
derived a closed-form expression of the outage probability for an arbitrary
path, considering the effect of RIS hardware failure and signal degradation
due to obstacles. The analysis can be used to quantify the impact of the
blockages and the RIS failure on the outage probability. The results shows at
a SNR approaching zero, the presence of obstacles and the failure probability
of RIS elements highly affect the outage probability. a large enough RIS
device vs. a communication distance provides a more reliable connection,
considering a RIS with strongly correlated elements and their likelihood of
failure.

## Acknowledgment

This work was partially supported by the DFG Project Nr. JU2757/12-1, and by
the Federal Ministry of Education and Research of Germany, joint project
6GRIC, 16KISK031.

## References

  * [1] A.-A. A. Boulogeorgos and A. Alexiou, “Pathloss modeling of reconfigurable intelligent surface assisted thz wireless systems,” in _ICC 2021 - IEEE International Conference on Communications_ , 2021, pp. 1–6.
  * [2] Q. Wu, S. Zhang, B. Zheng, C. You, and R. Zhang, “Intelligent reflecting surface-aided wireless communications: A tutorial,” _IEEE Transactions on Communications_ , vol. 69, no. 5, pp. 3313–3351, 2021.
  * [3] Z. Mao, W. Wang, Q. Xia, C. Zhong, X. Pan, and Z. Ye, “Element-grouping intelligent reflecting surface: Electromagnetic-compliant model and geometry-based optimization,” _IEEE Transactions on Wireless Communications_ , vol. 21, no. 7, pp. 5362–5376, 2022.
  * [4] C. Cai, X. Yuan, and Y.-J. A. Zhang, “Ris partitioning based scalable beamforming design for large-scale mimo: Asymptotic analysis and optimization,” _IEEE Transactions on Wireless Communications_ , 2023.
  * [5] X. Y. Li and M. S. Tong, “Array failure diagnosis and active compensation for intelligent reflecting surfaces,” in _2021 IEEE International Symposium on Antennas and Propagation and USNC-URSI Radio Science Meeting (APS/URSI)_. IEEE, 2021, pp. 1769–1770.
  * [6] L. Yang and et. al., “Outage probability and capacity scaling law of multiple ris-aided networks,” _IEEE Wireless Communications Letters_ , vol. 10, no. 2, pp. 256–260, 2020.
  * [7] T. Wang and et. al., “Outage probability analysis of ris-assisted wireless networks with von mises phase errors,” _IEEE wireless communications letters_ , vol. 10, no. 12, pp. 2737–2741, 2021.
  * [8] B. Lu, R. Wang, and Y. Liu, “Outage probability of intelligent reflecting surface assisted full duplex two-way communications,” _IEEE Communications Letters_ , vol. 26, no. 2, pp. 286–290, 2021.
  * [9] M. Jung and et. al., “Reliability analysis of large intelligent surfaces (liss): Rate distribution and outage probability,” _IEEE Wireless Communications Letters_ , vol. 8, no. 6, pp. 1662–1666, 2019.
  * [10] N. Mensi and D. B. Rawat, “On the performance of partial ris selection vs. partial relay selection for vehicular communications,” _IEEE Transactions on Vehicular Technology_ , vol. 71, no. 9, pp. 9475–9489, 2022.
  * [11] J. L. Vicario and et. al., “Opportunistic relay selection with outdated csi: Outage probability and diversity analysis,” _IEEE Transactions on Wireless Communications_ , vol. 8, no. 6, pp. 2872–2876, 2009.
  * [12] D. S. Michalopoulos and et. al., “Amplify-and-forward relay selection with outdated channel estimates,” _IEEE Transactions on Communications_ , vol. 60, no. 5, pp. 1278–1290, 2012.
  * [13] S. Kwon and J. Widmer, “Relay selection for mmwave communications,” in _2017 IEEE 28th Annual International Symposium on Personal, Indoor, and Mobile Radio Communications (PIMRC)_. IEEE, 2017, pp. 1–6.
  * [14] C. Guo and et. al., “Outage probability analysis and minimization in intelligent reflecting surface-assisted miso systems,” _IEEE Communications Letters_ , vol. 24, no. 7, pp. 1563–1567, 2020.
  * [15] M. Bensalem, A. Engelmann, and A. Jukan, “Towards optimal path allocation for unreliable reconfigurable intelligent surfaces,” in _2023 19th International Conference on the Design of Reliable Communication Networks (DRCN)_. IEEE, 2023, pp. 1–8.
  * [16] E. Björnson and L. Sanguinetti, “Rayleigh fading modeling and channel hardening for reconfigurable intelligent surfaces,” _IEEE Wireless Communications Letters_ , vol. 10, no. 4, pp. 830–834, 2020.
  * [17] I. S. Gradshteyn and I. M. Ryzhik, “Tables of integrals, series and products,” 1994.


# Adaptive Discounting of Implicit Language Models in RNN-Transducers

###### Abstract

RNN-Transducer (RNN-T) models have become synonymous with streaming end-to-end
ASR systems. While they perform competitively on a number of evaluation
categories, rare words pose a serious challenge to RNN-T models. One main
reason for the degradation in performance on rare words is that the language
model (LM) internal to RNN-Ts can become overconfident and lead to
hallucinated predictions that are acoustically inconsistent with the
underlying speech. To address this issue, we propose a lightweight adaptive LM
discounting technique AdaptLMD, that can be used with any RNN-T architecture
without requiring any external resources or additional parameters. AdaptLMD
uses a two-pronged approach:  1. Randomly mask the prediction network output
to encourage the RNN-T to not be overly reliant on it's outputs.  2.
Dynamically choose when to discount the implicit LM (ILM) based on rarity of
recently predicted tokens and divergence between ILM and implicit acoustic
model (IAM) scores.  Comparing AdaptLMD to a competitive RNN-T baseline, we
obtain up to $4\%$ and $14\%$ relative reductions in overall WER and rare word
PER, respectively, on a conversational, code-mixed Hindi-English ASR task.

Index Terms—  RNN-Transducer, Implicit Language Model, Rare Word ASR

##  1 Introduction

End-to-end RNN Transducer (RNN-T) models [1] are rapidly becoming the de facto
model for streaming speech recognition systems. RNN-T models consist of two
independent encoders: an acoustic encoder that processes an acoustic signal
one frame at a time, and, a language encoder consisting of an auto-regressive
model that encodes previous tokens and acts as an implicit language model.
Outputs from the acoustic and language encoders are combined and input to a
joint network that makes the final RNN-T predictions. Such an architecture
conveniently integrates training of the language model along with the acoustic
model, and enables streaming ASR.

While RNN-T models perform competitively on various benchmarks [2, 3], they
struggle with rare word predictions [4]. For words that rarely appear in the
training data, the language encoder of the RNN-T could lead the network to
favour hypotheses that are unfaithful to the underlying speech. This problem
is compounded when dealing with out-of-domain ASR evaluations where the test
speech is drawn from domains unseen during training. For example, our baseline
RNN-T system hallucinated ``canara bank" as ``amazon" when trained on retail
data but tested on banking data. More such cases of gross overriding of the
acoustic input by LM hallucinations appear in Table 3.

Recent work on RNN-T models have used LM fusion techniques to improve rare
word ASR predictions [5, 6, 4] but have typically relied on external resources
(raw text, lexicons, etc.). In this work, we adopt a more direct approach by
focusing on the RNN-T model itself and proposing model changes that aim at
arresting an overconfident implicit LM from making predictions that are at
odds with the underlying speech. Our proposed technique AdaptLMD has the
following two main features:

  * •

We randomly mask outputs from the language encoder to the joint network, thus
making the RNN-T model more robust to its spurious outputs.

  * •

We propose a new dynamic discounting scheme that identifies regions of
interest by examining discrepancies between the acoustic and language encoders
and computing the rarity of recent tokens in the predicted hypothesis.

On out-of-domain evaluations, we observe statistically significant reductions
in overall WER ($4\%$ relative) and large relative PER improvements of up to
$14\%$ on rare words.

Related Work. Prior work has handled rare word prediction and out-of-
vocabulary detection [7, 8, 9] by modeling at the subword level using hybrid
LMs [10, 11, 12]. More recent work on end-to-end models has mainly tackled the
problem of out-of-domain tokens using techniques such as shallow fusion [13]
where hypotheses are rescored using an external LM trained on text in the
target domain. Many variants of shallow fusion use some form of contextual LMs
[14, 4, 15, 16] that focus on fixing words appearing in specific contexts
(e.g., involving named entities). A popular extension of shallow fusion,
routinely employed in RNN-T models, is the density ratio technique [17] that
additionally discounts the scores using an external LM trained on text from
the training domains. This technique has seen further improvements by
discounting using an implicit LM instead of an external LM [18, 19]. Both
shallow fusion and the density ratio approach are inference-based techniques
and hence preferred more than other techniques like deep fusion [13] and cold
fusion [5] that require retraining with the new LMs. Besides LM rescoring,
other techniques addressing tail performance include the use of subword
regularization [15], using data augmentation via pronunciation dictionaries
[15, 16, 20] or synthetic samples [21] and finetuning the prediction network
using target text [22]. In concurrent work, [23] tries to limit LM
overtraining by combining the two encoded representations using a gate and
using a regularizer to slow the training of the LM. In contrast to most of the
above-mentioned prior work, our technique fits within existing RNN-T
architectures without requiring any additional parameter training or requiring
external resources.

##  2 Background: RNN-Transducers

Consider an acoustic input ${\mathbf{x}}=\\{x_{1},x_{2},x_{3}...x_{T}\\}$ with
corresponding output text transcript
${\mathbf{y}}=\\{y_{1},y_{2},y_{3}...y_{K}\\}$ where each
$y_{j}\in\mathcal{V}$, the output vocabulary. $\mathcal{V}$ also includes a
special blank token $\epsilon$. An RNN-T model comprises a Prediction Network
(PN), a Transcription Network (TN) and a joint network (Figure 1). The PN
takes as input the previously emitted _non-blank_ tokens
${\mathbf{y}}_{<u}=y_{1}\ldots y_{u-1}$, and auto-regressively generates an
encoding for the next output token as $g_{u}$. The TN takes as input the
acoustic signal ${\mathbf{x}}$ and outputs encoder states for each of the time
steps $\mathbf{h}=\\{h_{1},h_{2},h_{3},...h_{T}\\}$. Henceforth, we will also
use implicit acoustic model (IAM) and implicit LM (ILM) to refer to the TN and
PN, respectively. The joint network $J$ for each time step $t\leq T$ and token
position $u\leq K$ takes as inputs the encoded vectors $h_{t}$ and $g_{u}$ and
generates a probability distribution over the vocabulary $\mathcal{V}$ as:

$\displaystyle P_{\text{rnnt}}(y_{t,u}|{\mathbf{y}}_{<u},x_{t})$
$\displaystyle=$ $\displaystyle\text{softmax}\\{J(h_{t}\oplus g_{u})\\}$ (1)
$\displaystyle g_{u}=\text{PN}({\mathbf{y}}_{<u}),$ $\displaystyle
h_{t}=\text{TN}({\mathbf{x}},t)$

Traditionally, the $\oplus$ is an addition operation, however, other
operations can be used too. $J$ is any feed-forward network, PN is typically
an RNN and TN a Transformer. The output probability $P_{\text{rnnt}}$ over the
$[t,u]$ space are marginalized during training to maximize the likelihood of
the token sequence $y_{1},\ldots,y_{K}$ using an efficient dynamic programming
algorithm. During decoding, beam-search is used to find the best possible
$[t,u]$ alignment and output token sequence [1].

(a) Training: LM masking and implicit LM+AM loss

(b) Decoding: Adaptive discounting of implicit LM

Fig. 1: Overview of AdaptLMD

##  3 Our Approach

A limitation of the symmetric integration of the language encoding ($g_{u}$)
with the acoustic encoding ($h_{t}$) is a systematic bias against rare words,
often resulting in hallucination of words that have no connection with the
underlying speech. We seek to temper such overt influence of the ILM when we
expect its predictions to be noisy. Our main challenge is how to detect,
during online decoding, if the ILM is misleading. We propose a light-weight
method called AdaptLMD that can be applied on existing RNN-T architectures
without any external lexicons or additional parameters.

AdaptLMD differs from existing RNN-T based ASR in two ways. First, during
training we impose random masking of the ILM output $g_{u}$, so that the joint
network learns to be robust to spurious embeddings by the ILM. Second, during
decoding we impose an adaptive discounting of the ILM on the output
distribution $P_{\text{rnnt}}$. We dynamically choose when to discount based
on two factors: (1) The discrepancy between the IAM and ILM signals, and (2)
The rarity of recent tokens in the current hypothesis. We compute these terms
using existing RNN-T modules and describe these in Sections 3.1 and 3.2
respectively. Our formula for adaptive ILM discounting based on these two
scores, and the overall decoding algorithm is described in Section 3.3. The
final training algorithm of AdaptLMD appears in Section 3.4.

###  3.1 Discrepancy between LM and AM

The RNN-T architecture allows easy extraction of the independent token
distribution of the TN and PN components by masking the $h_{t}$ and $g_{u}$,
respectively. We call these $P_{\text{ILM}}$ and $P_{\text{IAM}}$ and define
as:

$\displaystyle P_{\text{IAM}}(y_{t}|{\mathbf{x}},t)$ $\displaystyle=$
$\displaystyle\text{softmax}\\{J(h_{t}\oplus\mathbf{0})\\}$ (2) $\displaystyle
P_{\text{ILM}}(y_{t}|{\mathbf{y}}_{<u})$ $\displaystyle=$
$\displaystyle\text{softmax}\\{J(\mathbf{0}\oplus g_{u})\\}$ (3)

where $\mathbf{0}$ denotes an identity vector. These distributions are trained
along with the primary $P_{\text{rnnt}}$ as elaborated in Sec 3.4. The
distance between $P_{\text{ILM}},P_{\text{IAM}}$ can be any divergence metric.
We use _KL-divergence_ :

$\text{D}(P_{\text{ILM}}||P_{\text{IAM}})=\sum_{y\in\mathcal{V}}P_{\text{ILM}}(y|{\mathbf{y}}_{<u})\log\frac{P_{\text{ILM}}(y|{\mathbf{y}}_{<u})}{P_{\text{IAM}}(y|x_{t})}$
(4)

.

###  3.2 Rarity of Recent Tokens

We enable discounting at rare locations within a currently decoded hypothesis.
Instead of external resources we depend on the implicit LM $P_{\text{ILM}}$ to
characterize rare regions. Let the current hypothesis be
${\mathbf{y}}=\\{y_{1},y_{2},y_{3},...y_{u-1}\\}$. We define a quantity
$P_{\text{roll}}({\mathbf{y}})$ that indicates a rolling average of the ILM
probabilities over the recent tokens in the hypothesis ${\mathbf{y}}$. We
choose a $0\leq\rho\leq 1$ to compute $P_{\text{roll}}({\mathbf{y}}+y_{u})$ of
${\mathbf{y}}$ extended by a new token $y_{u}$ at step $u$ incrementally from
the $P_{\text{roll}}({\mathbf{y}})$ as:

$\begin{gathered}P_{\text{roll}}({\mathbf{y}}+y_{u})=\rho
P_{\text{roll}}({\mathbf{y}})+P_{\text{ILM}}(y_{u})\end{gathered}$ (5)

As we want the discounting to be higher for rarer words where the implicit LM
confidence on the hypothesis generated so far is low, we use
$1-P_{\text{roll}}({\mathbf{y}})$ as a measure of the local rarity of recent
tokens of a current hypothesis ${\mathbf{y}}$.

###  3.3 Decoding with Implicit LM Discounting

Using the above two scores, we adaptively discount the influence of the ILM in
regions where the it is less confident and where there is discrepancy between
the ILM and IAM as follows: Let ${\mathbf{y}}=y_{1},\ldots y_{u-1}$ denote the
current hypothesis on the beam. The discounted score of the next token
$y_{t,u}$ by which we extend ${\mathbf{y}}$ at position $u$ from frame $t$ is
then computed as follows:

$\displaystyle\tilde{S}_{\text{disc}}(y_{t,u}|$
$\displaystyle{\mathbf{y}},x_{t})=\log
P_{\text{rnnt}}(y_{t,u}|{\mathbf{y}},x_{t})$
$\displaystyle-\lambda\max(0,D_{\text{adapt}}(y_{t,u},{\mathbf{y}}))\log
P_{\text{ILM}}(y_{u}|{\mathbf{y}})$ $\displaystyle
D_{\text{adapt}}(y,{\mathbf{y}})=$
$\displaystyle\begin{cases}(1-P_{\text{roll}}({\mathbf{y}}))\text{D}(P_{\text{ILM}}(y)||P_{\text{IAM}}(y))&\text{if
}y\neq\epsilon\\\ 0&\text{else.}\end{cases}$

where $\lambda$ is a tunable parameter. While decoding we use the above
discounted scores $\tilde{S}_{\text{disc}}$ as a drop-in replacement for the
original $\log P_{\text{rnnt}}$ scores in the beam-search algorithm. With this
discounted score, we defer to predictions of the implicit AM and discount the
predictions by the $P_{\text{ILM}}$ in regions of low confidence (indicated by
$P_{\text{roll}}$). The only additional state we need to maintain for each
hypothesis on the beam is $P_{\text{roll}}$. The rest of the inference
algorithm stays unchanged. We extended the _Alignment-Length Asynchronous
Decoding (ALSD)_ [24] with the above changes as the decoding algorithm for
AdaptLMD.

###  3.4 Overall Training Objective

Our training strategy differs from RNN-T's in two ways: (1) We use random
masking of the PN encoding to encourage the RNN-T not to be overly reliant on
its output, and (2) we introduce extra losses for training the
$P_{\text{ILM}}$ and $P_{\text{IAM}}$.

$\begin{gathered}\log\sum_{\text{align
t}}P_{\text{rnnt}}(y_{t,u}|h_{t}\oplus\text{maskedLM}_{\eta}(g_{u}))\\\
+\alpha\log\sum_{\text{align t}}P_{\text{ILM}}(y_{u}|\mathbf{0}\oplus
g_{u})\\\ +\beta\log\sum_{\text{align
t}}P_{\text{IAM}}(y_{t,u}|h_{t}\oplus\mathbf{0})\end{gathered}$ (6)

In eqn (6), $\text{maskedLM}_{\eta}(g_{u})$ denotes the random masking of the
TN where with a probability $\eta$ we replace $g_{u}$ with $\mathbf{0}$.

##  4 Experiments and Results

Train (Hours) / Test (Hours) | ASR System | WER | CER | PER |  | Rare PER
---
| Rare CER
---
All-train ($477$ hrs) / All-test ($75$ hrs) | RNNT | 14.5 | 13.5 | 12.0 | 57.7 | 71.2
ILMT | 14.5 | 13.4 | 12.1 | 53.8 | 66.6
AdaptLMD | 14.1 | 13.1 | 11.8 | 52.2 | 63.9
Telco + Retail + Insurance ($327$ hrs) / Test ($58$ hrs) | RNN_T | 14.5 | 13.6 | 12.3 | 66.9 | 68.2
ILMT | 14.6 | 13.4 | 12.0 | 69.3 | 70.5
AdaptLMD | 14.2 | 13.1 | 11.9 | 63.2 | 59.6
Banking + Retail + Insurance ($374$ hrs) / Test ($55$ hrs) | RNN_T | 15.2 | 14.0 | 12.7 | 68.0 | 76.6
ILMT | 15.9 | 14.5 | 12.3 | 63.8 | 74.2
AdaptLMD | 14.9 | 13.6 | 12.3 | 60.2 | 63.3
Banking + Telco + Insurance ($381$ hrs) / Test ($55$ hrs) | RNN_T | 14.2 | 13.2 | 12.0 | 65.3 | 72.7
ILMT | 14.1 | 13.8 | 11.8 | 64.6 | 65.7
AdaptLMD | 13.8 | 12.9 | 11.6 | 63.1 | 63.8
Banking + Telco + Retail ($349$ hrs) / Test ($57$ hrs) | RNN_T | 15.0 | 14.4 | 13.1 | 66.9 | 68.2
ILMT | 15.9 | 14.5 | 13.2 | 69.3 | 70.5
AdaptLMD | 15.1 | 14.0 | 12.7 | 63.5 | 59.6

Table 1: In-domain ASR Results.

Test Data | ASR-System | WER | CER | PER |  | Rare PER
---
| Rare CER
---
Test- Banking (110 hrs) | RNN-T | 22.4 | 20.0 | 18.5 | 70.8 | 75.9
D.R. | 22.2 | 19.9 | 18.4 | 70.8 | 74.6
ILMT | 22.4 | 20.0 | 18.5 | 70.9 | 75.9
AdaptLMD | 21.5 | 18.7 | 17.1 | 67.4 | 71.8
Test- Insurance (95 hrs) | RNN-T | 18.4 | 17.7 | 16.1 | 70.2 | 76.7
D.R. | 18.1 | 17.7 | 16.0 | 69.1 | 75.6
ILMT | 18.7 | 17.6 | 15.9 | 65.7 | 72.2
AdaptLMD | 17.7 | 16.4 | 14.8 | 60.7 | 67.2
Test- Retail (71 hrs) | RNN-T | 24.7 | 22.4 | 20.6 | 76.7 | 81.3
D.R. | 24.6 | 22.1 | 20.4 | 76.0 | 81.3
ILMT | 24.5 | 21.4 | 19.7 | 72.4 | 77.5
AdaptLMD | 23.7 | 21.2 | 19.5 | 72.2 | 78.3
Test- Telco (75 hrs) | RNN-T | 19.4 | 18.6 | 17.1 | 68.3 | 72.6
D.R. | 19.1 | 18.2 | 16.8 | 70.5 | 73.7
ILMT | 19.3 | 18.1 | 16.4 | 63.8 | 68.9
AdaptLMD | 18.6 | 17.5 | 16.0 | 64.3 | 68.6

Table 2: Out-domain ASR Results.

Datasets. We conduct experiments on a proprietary dataset consisting of code-
mixed speech (Hi-En) in a call-centre setting. This dataset contains $628$
hours of speech and is divided into four domains: Banking ($184.87$ hrs),
Insurance ($165.08$ hrs), Retail ($135.87$ hrs) and Telco ($142.33$ hrs).
Domain-wise durations of train/test splis can be found in Table 1.

Architecture. We use a baseline RNN-T model where the transcription network
consists of $6$ Conformer [25] encoder layers with an internal representation
size of $640$ and the prediction network is a single layer LSTM with a $768$
dimensional hidden state. The conformer consists of $8$ attention heads of dim
$27$ and has a kernel size of $31$ for the convolutional layers. The output of
the transcription network is projected to a $256$ dim vector $h_{t}$. The LSTM
hidden state outputs a $10$ dim embedding which is further projected to a
$256$ dim vector $g_{u}$. The joint network consists of an additive operation
between $h_{t}$ and $g_{u}$ followed by a $\tanh$ non-linearity. This is
projected to the final layer of size $123$ that corresponds to the size of our
character-based vocabulary $\mathcal{V}$. We use the _Noam_ optimizer [26] for
training with a learning rate of 0.0001. LM masking while training is
performed with a probability of $\eta=0.2$ and weights for the implicit LM and
AM are $\alpha=\beta=0.125$. The hyperparamters $\lambda$ and $\rho$ are
chosen by tuning on a held-out set.

Baselines. Our in-domain results are compared against a vanilla RNN-T and an
ILMT model trained with an implicit LM [27]. For out-of-domain experiments, we
also compare with the density ratio approach [17] using external RNN-LMs (1024
units, 1 layer) trained on the source and target datasets.

Results. We show WERs/PERs/CERs for both in-domain and out-of-domain settings,
where test utterances are drawn from domains that are either seen or unseen
during training, respectively. Along with evaluations on the entire test set
in each setting, we also separately compute PERs and CERs for rare words that
appear less than $20$ times during training.111We do not report rare word WERs
since they were $100\%$ or higher.

Table 1 shows in-domain results from training on speech from all domains, and
four other training settings where we leave out one domain at a time. AdaptLMD
outperforms both RNN-T and ILMT in terms of WER reductions in almost all
settings. Reductions in PER using AdaptLMD for the rare words are much more
significant. Table 2 shows out-of-domain results on four test domains that
were each held out during training. Relative to the in-domain setting,
AdaptLMD outperforms all the baselines on overall WERs by a statistically
significant margin (at $p<0.001$ using the MAPSSWE test [28]). AdaptLMD gives
large PER improvements on rare words for all four test domains. These PER
improvements predominantly stem from fixing the errors introduced by LM
hallucinations of the RNN-T baseline. Table 3 provides a few examples of such
hallucinations by the RNN-T baseline where the predictions are at times
completely inconsistent with the underlying acoustics (e.g.,
business$\rightarrow$discount, call$\rightarrow$contract, etc.). We also show
examples of AdaptLMD predictions that may not be identical to the ground truth
but are acoustically faithful, unlike the RNN-T baseline (e.g.,
ikatees$\rightarrow$iktees, vicky$\rightarrow$vikkee, etc.).

Table 4 shows an ablation analysis of AdaptLMD. We compare a baseline RNN-T
with variants containing $P_{\text{ILM}}$ and $P_{\text{IAM}}$ losses (as in
eqn (6)) and TN masking. Discounting is particularly beneficial for rare word
predictions.222Discounting the entire utterance with a static discount factor,
instead of our adaptive scheme in AdaptLMD, worsens overall WERs.

| Groundtruth
---
RNN-T
AdaptLMD
|  Ki aap apanaa business (bIzn@s) account kab open (@Up@n) karvana
---
ki aap apanaa discount (dIskaUnt) account kab  bund (b@nd)  karvana
ki aap apanaa business (bIzn@s) account kab open (@Up@n) karvana
| Groundtruth
---
RNN-T
AdaptLMD
|  jisakaa pataa hain plot (pl6t) number ikatees (ikti:s) a..
---
jisakaa pataa hai block (bl6k ) number  tees (ti:s) ek
jisakaa pataa hai plot (pl6t) number iktees (ik t i:s) e..
| Groundtruth
---
RNN-T
AdaptLMD
| ..ab aapki call (kOl ) transfer kar dee jaayegi..
---
..ab aapki contract (kOntr ækt) transfer kar dee jaayegi..
..ab aapki call (kOl ) transfer kar dee jaayegi..
| Groundtruth
---
RNN-T
AdaptLMD
| ..mera pooraa naam hai.. vicky rajak (vikki r@dZ͡@k)..
---
..mera pooraa naam hai.. reeti raagav (ri:ti ra:g@v)..
..mera pooraa naam hai .. vikkee raaj (vikki: ra:dZ͡)..
| Groundtruth
---
RNN-T
AdaptLMD
| ..ye online activate (æktIvejt) karavaa sakataa kee naheen
---
..mujhe online network (nEtwr k) karavaa sakataa hoon
..ye online activate (æktIvejt) karavaa sakataa hoon..
| Groundtruth
---
RNN-T
AdaptLMD
|  namaskaar sir main apane loan (lown) ko fir se
---
namaskaar sir main apane block (bl6k) ko fir se
namaskaar sir main apane loan az(lown) ko fir se

Table 3: Anecdotes illustrating LM hallucinations. Italicized text is
Romanized Hindi. Tokens of interest are in blue.

| WER | CER | PER |  | Rare PER
---
| Rare CER
---
RNN-T | 14.5 | 13.5 | 12.0 | 57.7 | 71.2
RNN-T + $P_{\text{ILM}}$ \+ $P_{\text{IAM}}$ | 14.5 | 13.2 | 11.9 | 53.4 | 66.5
| RNN-T+$P_{\text{ILM}}$ +$P_{\text{IAM}}$
---
\+ TN mask
14.4 | 13.1 | 11.8 | 53.1 | 65.9
AdaptLMD | 14.1 | 13.1 | 11.8 | 52.2 | 63.9

Table 4: Ablations study to demonstrate the effect of TN masking and implicit
LM/AM training in AdaptLMD.

##  5 Conclusion

We propose AdaptLMD, a new adaptive LM discounting technique that is
dynamically invoked and can be used within any RNN-T model without requiring
any additional resources. This technique particularly benefits rare words for
which the LM internal to an RNN-T can generate predictions that do not match
the underlying audio. AdaptLMD yields consistent performance improvements on
overall WERs and rare word predictions in both in-domain and out-of-domain
evaluation settings for code-mixed Hindi-English ASR tasks across four
domains.

##  6 Acknowledgements

The authors from IIT Bombay gratefully acknowledge support from IBM Research,
specifically the IBM AI Horizon Networks-IIT Bombay initiative.

## References

  * [1] Alex Graves,  ``Sequence Transduction with Recurrent Neural Networks,''  2012\.
  * [2] Kanishka Rao, Haşim Sak, and Rohit Prabhavalkar,  ``Exploring architectures, data and units for streaming end-to-end speech recognition with rnn-transducer,''  in ASRU-2017.
  * [3] Yanzhang He, Tara N Sainath, Rohit Prabhavalkar, Ian McGraw, Raziel Alvarez, Ding Zhao, David Rybach, Anjuli Kannan, Yonghui Wu, Ruoming Pang, et al.,  ``Streaming end-to-end speech recognition for mobile devices,''  in ICASSP 2019.
  * [4] Vijay Ravi, Yile Gu, Ankur Gandhe, Ariya Rastrow, Linda Liu, Denis Filimonov, Scott Novotney, and Ivan Bulyko,  ``Improving accuracy of rare words for rnn-transducer through unigram shallow fusion,'' 2020.
  * [5] Anuroop Sriram, Heewoo Jun, Sanjeev Satheesh, and Adam Coates,  ``Cold fusion: Training seq2seq models together with language models,''  2017\.
  * [6] Cal Peyser, Sepand Mavandadi, Tara N Sainath, James Apfel, Ruoming Pang, and Shankar Kumar,  ``Improving tail performance of a deliberation e2e asr model using a large text corpus,''  2020\.
  * [7] Carolina Parada, Mark Dredze, Denis Filimonov, and Frederick Jelinek,  ``Contextual information improves oov detection in speech,''  2010\.
  * [8] Hui Lin, Jeff Bilmes, Dimitra Vergyri, and Katrin Kirchhoff,  ``Oov detection by joint word/phone lattice alignment,''  in ASRU-2007.
  * [9] Ariya Rastrow, Abhinav Sethy, and Bhuvana Ramabhadran,  ``A new method for oov detection using hybrid word/fragment system,''  in ICASSP-2009.
  * [10] Maximilian Bisani and Hermann Ney,  ``Open vocabulary speech recognition with flat hybrid models,''  in Interspeech-2005.
  * [11] A. Yazgan and M. Saraclar,  ``Hybrid language models for out of vocabulary word detection in large vocabulary conversational speech recognition,''  in ICASSP-2004.
  * [12] Jinyu Li, Guoli Ye, Rui Zhao, Jasha Droppo, and Yifan Gong,  ``Acoustic-to-word model without oov,'' 2017.
  * [13] Caglar Gulcehre, Orhan Firat, Kelvin Xu, Kyunghyun Cho, Loic Barrault, Huei-Chi Lin, Fethi Bougares, Holger Schwenk, and Yoshua Bengio,  ``On using monolingual corpora in neural machine translation,'' 2015.
  * [14] Alex Gruenstein et al.,  ``An efficient streaming non-recurrent on-device end-to-end model with improvements to rare-word modeling,''  2021\.
  * [15] Duc Le, Gil Keren, Julian Chan, Jay Mahadeokar, Christian Fuegen, and Michael L. Seltzer,  ``Deep shallow fusion for rnn-t personalization,'' 2020.
  * [16] Ding Zhao, Tara N Sainath, David Rybach, Pat Rondon, Deepti Bhatia, Bo Li, and Ruoming Pang,  ``Shallow-fusion end-to-end contextual biasing.,''  in Interspeech-2019.
  * [17] Erik McDermott, Hasim Sak, and Ehsan Variani,  ``A density ratio approach to language model fusion in end-to-end automatic speech recognition,'' 2020.
  * [18] Ehsan Variani, David Rybach, Cyril Allauzen, and Michael Riley,  ``Hybrid autoregressive transducer (hat),'' 2020.
  * [19] Zhong Meng et al.,  ``Internal language model estimation for domain-adaptive end-to-end speech recognition,'' 2020.
  * [20] Norihide Kitaoka, Bohan Chen, and Yuya Obashi,  ``Dynamic out-of-vocabulary word registration to language model for speech recognition,''  .
  * [21] Jinyu Li et al.,  ``Developing rnn-t models surpassing high-performance hybrid models with customization capability,'' 2020.
  * [22] Janne Pylkkönen, Antti Ukkonen, Juho Kilpikoski, Samu Tamminen, and Hannes Heikinheimo,  ``Fast text-only domain adaptation of rnn-transducer prediction network,'' 2021.
  * [23] Chao Zhang, Bo Li, Zhiyun Lu, Tara N. Sainath, and Shuo yiin Chang,  ``Improving the fusion of acoustic and text representations in rnn-t,''  in ICASSP-2022.
  * [24] George Saon, Zoltán Tüske, and Kartik Audhkhasi,  ``Alignment-length synchronous decoding for rnn transducer,''  in ICASSP-2020.
  * [25] Anmol Gulati et al.,  ``Conformer: Convolution-augmented transformer for speech recognition,'' 2020.
  * [26] Ashish Vaswani et al.,  ``Attention is all you need,'' 2017.
  * [27] Zhong Meng, Naoyuki Kanda, Yashesh Gaur, Sarangarajan Parthasarathy, Eric Sun, Liang Lu, Xie Chen, Jinyu Li, and Yifan Gong,  ``Internal language model training for domain-adaptive end-to-end speech recognition,'' 2021.
  * [28] L. Gillick and S.J. Cox,  ``Some statistical issues in the comparison of speech recognition algorithms,''  in ICASSP-1989.