#  Nonlinear mixed Jordan triple $*$-derivations on $*$-algebras
00footnotetext:  The authors are supported by the Natural Science Foundation
of Shandong Province, China (Grant No. ZR2018BA003) and the National Natural
Science Foundation of China (Grant No. 11801333).00footnotetext:  ∗
Corresponding author. E-mail<EMAIL_ADDRESS>(C. Li).

Dongfang Zhang, Changjing Li$\ {}^{*}$
School of Mathematics and Statistics, Shandong Normal University, Jinan
250014, P. R. China

###### Abstract

Let $\mathcal{A}$ be a unital $\ast$-algebra. For $A,B\in\mathcal{A}$, define
by $[A,B]_{*}=AB-BA^{\ast}$ and $A\bullet B=AB+BA^{\ast}$ the new products of
$A$ and $B$. In this paper, under some mild conditions on $\mathcal{A}$, it is
shown that a map $\Phi:\mathcal{A}\rightarrow\mathcal{A}$ satisfies
$\Phi([A\bullet B,C]_{*})=[\Phi(A)\bullet
B,C]_{*}+[A\bullet\Phi(B),C]_{*}+[A\bullet B,\Phi(C)]_{*}$ for all
$A,B,C\in\mathcal{A}$ if and only if $\Phi$ is an additive $*-$derivation. In
particular, we apply the above result to prime $\ast$-algebras, von Neumann
algebras with no central summands of type $I_{1}$, factor von Neumann algebras
and standard operator algebras.

Keywords: mixed Jordan triple $*$-derivations; $*$-derivations; von Neumann
algebras.

2020 Mathematics Subject Classification: 47B47; 46L10

##  1 Introduction

Let $\mathcal{A}$ be a $*$-algebra over the complex field $\mathbb{C}$. For
$A,B\in\mathcal{A}$, define the skew Lie product of $A$ and $B$ by
$[A,B]_{\ast}=AB-BA^{\ast}$ and the Jordan $*$-product of $A$ and $B$ by
$A\bullet B=AB+BA^{\ast}$. The skew Lie product and the Jordan $*$-product are
fairly meaningful and important in some research topics. They were extensively
studied because they naturally arise in the problem of representing quadratic
functionals with sesquilinear functionals (see [17, 18, 19]) and in the
problem of characterizing ideals (see [2, 16]). Particular attention has been
paid to understanding maps which preserve the skew Lie product or the Jordan
$*$-product on $\ast$-algebras (see [1, 3, 4, 6, 9, 8, 11, 12, 25]).

Recall that an additive map $\Phi:\mathcal{A}\rightarrow\mathcal{A}$ is said
to be an additive derivation if $\Phi(AB)=\Phi(A)B+A\Phi(B)$ for all
$A,B\in\mathcal{A}.$ Furthermore, $\Phi$ is said to be an additive
$*$-derivation if it is an additive derivation and satisfies
$\Phi(A^{*})=\Phi(A)^{*}$ for all $A\in\mathcal{A}.$ A not necessarily linear
map $\Phi:\mathcal{A}\rightarrow\mathcal{A}$ is said to be a nonlinear Jordan
$*$-derivation or a nonlinear skew Lie derivation if

$\Phi(A\bullet B)=\Phi(A)\bullet B+A\bullet\Phi(B)$

or

$\Phi([A,B]_{\ast})=[\Phi(A),B]_{\ast}+[A,\Phi(B)]_{\ast}$

for all $A,B\in\mathcal{A}$. Yu and Zhang in [22] proved that every nonlinear
skew Lie derivation on factor von Neumann algebras is an additive
$*$-derivation. Jing in [7] studied nonlinear skew Lie derivations on standard
operator algebras. Let $\mathcal{A}$ be a standard operator algebra on a
complex Hilbert space $H$ which is closed under the adjoint operation. It was
shown that every nonlinear skew Lie derivation $\Phi$ on $\mathcal{A}$ is
automatically linear. Moreover, $\Phi$ is an inner $*$-derivation. Taghavi et
al. [21] and Zhang [23] independently investigated nonlinear Jordan
$*$-derivations on factor von Neumann algebras, respectively. It turns out
that every nonlinear Jordan $*$-derivation between factor von Neumann algebras
is an additive $*$-derivation. Li et al. in [10] investigated nonlinear skew
Lie derivations and Jordan $*$-derivations on von Neumann algebras with no
central summands of type $I_{1}$.

Given the consideration of nonlinear Jordan $*$-derivations and nonlinear skew
Lie derivations, we can further develop them in one natural way. A map
$\Phi:\mathcal{A}\rightarrow\mathcal{A}$ is said to be a nonlinear Jordan
triple $*$-derivation or a nonlinear skew Lie triple derivation if

$\Phi(A\bullet B\bullet C)=\Phi(A)\bullet B\bullet C+A\bullet\Phi(B)\bullet
C+A\bullet B\bullet\Phi(C)$

or

$\Phi([[A,B]_{\ast},C]_{\ast})=[[\Phi(A),B]_{\ast},C]_{\ast}+[[A,\Phi(B)]_{\ast},C]_{\ast}+[[A,B]_{\ast},\Phi(C)]_{\ast}$

for all $A,B,C\in\mathcal{A}$. Li et al.[13] proved that every nonlinear skew
Lie triple derivation on factor von Neumann algebras is an additive
$*$-derivation. Fu and An [5] proved that $\Phi$ is a nonlinear skew Lie
triple derivation on von Neumann algebras with no central summands of type
$I_{1}$ if and only if $\Phi$ is an additive $*$-derivation. Zhao and Li [24]
proved that every nonlinear Jordan triple $*$-derivation between von Neumann
algebras with no central summands of type $I_{1}$ is an additive
$*$-derivation. Lin [14, 15] studied the nonlinear skew Lie $n$-derivations on
standard operator algebras and von Neumann algebras with no central summands
of type $I_{1}$.

In this paper, we will study the nonlinear mixed Jordan triple $*$-derivations
on $*$-algebras. A map $\Phi:\mathcal{A}\rightarrow\mathcal{A}$ is said to be
a nonlinear mixed Jordan triple $*$-derivation if

$\Phi([A\bullet B,C]_{*})=[\Phi(A)\bullet
B,C]_{*}+[A\bullet\Phi(B),C]_{*}+[A\bullet B,\Phi(C)]_{*}$

for all $A,B,C\in\mathcal{A}$. Under some mild conditions on a $*$-algebra
$\mathcal{A}$, we prove that a map $\Phi:\mathcal{A}\rightarrow\mathcal{A}$ is
a nonlinear mixed Jordan triple $*$-derivation if and only if $\Phi$ is an
additive $*-$derivation. In particular, we apply the above result to prime
$\ast$-algebras, von Neumann algebras with no central summands of type
$I_{1}$, factor von Neumann algebras and standard operator algebras.

##  2 The main result and its proof

Our main result in this paper reads as follows.

######  Theorem 2.1.

Let $\mathcal{A}$ be a unital $\ast$-algebra with the unit $I$. Assume that
$\mathcal{A}$ contains a nontrivial projection $P$ which satisfies

$(\spadesuit)~{}~{}~{}X\mathcal{A}P=0~{}~{}~{}implies~{}~{}~{}X=0$

and

$(\clubsuit)~{}~{}~{}X\mathcal{A}(I-P)=0~{}~{}~{}implies~{}~{}~{}X=0.$

Then a map $\Phi:\mathcal{A}\rightarrow\mathcal{A}$ satisfies $\Phi([A\bullet
B,C]_{*})=[\Phi(A)\bullet B,C]_{*}+[A\bullet\Phi(B),C]_{*}+[A\bullet
B,\Phi(C)]_{*}$ for all $A,B,C\in\mathcal{A}$ if and only if $\Phi$ is an
additive $*-$derivation.

###### Proof.

Let $P_{1}=P$ and $P_{2}=I-P.$ Denote
$\mathcal{A}_{jk}=P_{j}\mathcal{A}P_{k},j,k=1,2.$ Then
$\mathcal{A}=\sum^{2}_{j,k=1}\mathcal{A}_{jk}.$ In all that follows, when we
write $A_{jk},$ it indicates that $A_{jk}\in\mathcal{A}_{jk}$. Clearly, we
only need to prove the necessity. We will complete the proof by several
claims.
Claim 1. $\Phi(0)=0.$

Indeed, we have

$\Phi(0)=\Phi([0\bullet 0,0]_{*})=[\Phi(0)\bullet
0,0]_{*}+[0\bullet\Phi(0),0]_{*}+[0\bullet 0,\Phi(0)]_{*}=0.$

Claim 2. $\Phi$ is additive.

We will complete the proof of Claim 2 by proving several steps.
Step 2.1. For every $A_{12}\in\mathcal{A}_{12},B_{21}\in\mathcal{A}_{21},$ we
have

$\Phi(A_{12}+B_{21})=\Phi(A_{12})+\Phi(B_{21}).$

We only need show that $T=\Phi(A_{12}+B_{21})-\Phi(A_{12})-\Phi(B_{21})=0.$
Since

$[I\bullet(i(P_{2}-P_{1})),A_{12}]_{*}=[I\bullet(i(P_{2}-P_{1})),B_{21}]_{*}=0,$

where $i$ is the imaginary unit, it follows from Claim 1 that

$\displaystyle[\Phi(I)\bullet(i(P_{2}-P_{1})),A_{12}+B_{21}]_{*}+[I\bullet\Phi(i(P_{2}-P_{1})),A_{12}+B_{21}]_{*}$
$\displaystyle+[I\bullet(i(P_{2}-P_{1})),\Phi(A_{12}+B_{21})]_{*}$
$\displaystyle=\Phi([I\bullet(i(P_{2}-P_{1})),A_{12}+B_{21}]_{*})$
$\displaystyle=\Phi([I\bullet(i(P_{2}-P_{1})),A_{12}]_{*})+\Phi([I\bullet(i(P_{2}-P_{1})),B_{21}]_{*})$
$\displaystyle=[\Phi(I)\bullet(i(P_{2}-P_{1})),A_{12}+B_{21}]_{*}+[I\bullet\Phi(i(P_{2}-P_{1})),A_{12}+B_{21}]_{*}$
$\displaystyle+[I\bullet(i(P_{2}-P_{1})),\Phi(A_{12})+\Phi(B_{21})]_{*}.$

From this, we get $[I\bullet(i(P_{2}-P_{1})),T]_{*}=0.$ So $T_{11}=T_{22}=0.$

Since $[I\bullet A_{12},P_{1}]_{*}=0$, it follows that

$\displaystyle[\Phi(I)\bullet(A_{12}+B_{21}),P_{1}]_{*}+[I\bullet\Phi(A_{12}+B_{21}),P_{1}]_{*}$
$\displaystyle+[I\bullet(A_{12}+B_{21}),\Phi(P_{1})]_{*}$
$\displaystyle=\Phi([I\bullet(A_{12}+B_{21}),P_{1}]_{*})$
$\displaystyle=\Phi([I\bullet A_{12},P_{1}]_{*})+\Phi([I\bullet
B_{21},P_{1}]_{*})$
$\displaystyle=[\Phi(I)\bullet(A_{12}+B_{21}),P_{1}]_{*}+[I\bullet(\Phi(A_{12})+\Phi(B_{21})),P_{1}]_{*}$
$\displaystyle+[I\bullet(A_{12}+B_{21}),\Phi(P_{1})]_{*}.$

Hence $[I\bullet T,P_{1}]_{*}=0$, from which we get that $T_{21}=0.$
Similarly, we can prove that $T_{12}=0,$ proving the step.
Step 2.2. For every
$A_{11}\in\mathcal{A}_{11},B_{12}\in\mathcal{A}_{12},C_{21}\in\mathcal{A}_{21},D_{22}\in\mathcal{A}_{22},$
we have

$\Phi(A_{11}+B_{12}+C_{21})=\Phi(A_{11})+\Phi(B_{12})+\Phi(C_{21})$

and

$\Phi(B_{12}+C_{21}+D_{22})=\Phi(B_{12})+\Phi(C_{21})+\Phi(D_{22}).$

Let $T=\Phi(A_{11}+B_{12}+C_{21})-\Phi(A_{11})-\Phi(B_{12})-\Phi(C_{21})$.

It follows from Step 2.1 that that

$\displaystyle[\Phi(I)\bullet(iP_{2}),A_{11}+B_{12}+C_{21}]_{*}+[I\bullet\Phi(iP_{2}),A_{11}+B_{12}+C_{21}]_{*}$
$\displaystyle+[I\bullet(iP_{2}),\Phi(A_{11}+B_{12}+C_{21})]_{*}$
$\displaystyle=\Phi([I\bullet(iP_{2}),A_{11}+B_{12}+C_{21}]_{*})$
$\displaystyle=\Phi([I\bullet(iP_{2}),A_{11}]_{*})+\Phi([I\bullet(iP_{2}),B_{12}+C_{21}]_{*})$
$\displaystyle=\Phi([I\bullet(iP_{2}),A_{11}]_{*})+\Phi([I\bullet(iP_{2}),B_{12}]_{*})+\Phi([I\bullet(iP_{2}),C_{21}]_{*})$
$\displaystyle=[\Phi(I)\bullet(iP_{2}),A_{11}+B_{12}+C_{21}]_{*}+[I\bullet\Phi(iP_{2}),A_{11}+B_{12}+C_{21}]_{*}$
$\displaystyle+[I\bullet(iP_{2}),\Phi(A_{11})+\Phi(B_{12})+\Phi(C_{21})]_{*}.$

From this, we get $[I\bullet(iP_{2}),T]_{*}=0.$ So $T_{12}=T_{21}=T_{22}=0.$

Since

$[I\bullet(i(P_{2}-P_{1})),B_{12}]_{*}=[I\bullet(i(P_{2}-P_{1})),C_{21}]_{*}=0,$

it follows that

$\displaystyle[\Phi(I)\bullet(i(P_{2}-P_{1})),A_{11}+B_{12}+C_{21}]_{*}+[I\bullet\Phi(i(P_{2}-P_{1})),A_{11}+B_{12}+C_{21}]_{*}$
$\displaystyle+[I\bullet(i(P_{2}-P_{1})),\Phi(A_{11}+B_{12}+C_{21})]_{*}$
$\displaystyle=\Phi([I\bullet(i(P_{2}-P_{1})),A_{11}+B_{12}+C_{21}]_{*})$
$\displaystyle=\Phi([I\bullet(i(P_{2}-P_{1})),A_{11}]_{*})+\Phi([I\bullet(i(P_{2}-P_{1})),B_{12}]_{*})+\Phi([I\bullet(i(P_{2}-P_{1})),C_{21}]_{*})$
$\displaystyle=[\Phi(I)\bullet(i(P_{2}-P_{1})),A_{11}+B_{12}+C_{21}]_{*}+[I\bullet\Phi(i(P_{2}-P_{1})),A_{11}+B_{12}+C_{21}]_{*}$
$\displaystyle+[I\bullet(i(P_{2}-P_{1})),\Phi(A_{11})+\Phi(B_{12})+\Phi(C_{21})]_{*},$

from which we get $[I\bullet(i(P_{2}-P_{1})),T]_{*}=0.$ So $T_{11}=0,$ and
then $T=0$. Similarly, we can get that
$\Phi(B_{12}+C_{21}+D_{22})=\Phi(B_{12})+\Phi(C_{21})+\Phi(D_{22}).$

Step 2.3. For every
$A_{11}\in\mathcal{A}_{11},B_{12}\in\mathcal{A}_{12},C_{21}\in\mathcal{A}_{21},D_{22}\in\mathcal{A}_{22},$
we have

$\Phi(A_{11}+B_{12}+C_{21}+D_{22})=\Phi(A_{11})+\Phi(B_{12})+\Phi(C_{21})+\Phi(D_{22}).$

Let
$T=\Phi(A_{11}+B_{12}+C_{21}+D_{22})-\Phi(A_{11})-\Phi(B_{12})-\Phi(C_{21})-\Phi(D_{22}).$
It follows from Step 2.2 that

$\displaystyle[\Phi(I)\bullet(iP_{2}),A_{11}+B_{12}+C_{21}+D_{22}]_{*}+[I\bullet\Phi(iP_{2}),A_{11}+B_{12}+C_{21}+D_{22}]_{*}$
$\displaystyle+[I\bullet(iP_{2}),\Phi(A_{11}+B_{12}+C_{21}+D_{22})]_{*}$
$\displaystyle=\Phi([I\bullet(iP_{2}),A_{11}+B_{12}+C_{21}+D_{22}]_{*})$
$\displaystyle=\Phi([I\bullet(iP_{2}),A_{11}]_{*})+\Phi([I\bullet(iP_{2}),B_{12}+C_{21}+D_{22})$
$\displaystyle=\Phi([I\bullet(iP_{2}),A_{11}]_{*})+\Phi([I\bullet(iP_{2}),B_{12}]_{*})+\Phi([I\bullet(iP_{2}),C_{21}]_{*})+\Phi([I\bullet(iP_{2}),D_{22}]_{*})$
$\displaystyle=[\Phi(I)\bullet(iP_{2}),A_{11}+B_{12}+C_{21}++D_{22}]_{*}+[I\bullet\Phi(iP_{2}),A_{11}+B_{12}+C_{21}+D_{22}]_{*}$
$\displaystyle+[I\bullet(iP_{2}),\Phi(A_{11})+\Phi(B_{12})+\Phi(C_{21})+\Phi(D_{22})]_{*}.$

From this, we get $[I\bullet(iP_{2}),T]_{*}=0.$ So $T_{12}=T_{21}=T_{22}=0.$
Similarly, we can prove $T_{11}=0,$ proving the step.
Step 2.4. For every $A_{jk},B_{jk}\in\mathcal{A}_{jk},1\leq j\neq k\leq 2,$ we
have

$\Phi(A_{jk}+B_{jk})=\Phi(A_{jk})+\Phi(B_{jk}).$

Since

$[\frac{I}{2}\bullet(P_{j}+A_{jk}),P_{k}+B_{jk}]_{*}=(A_{jk}+B_{jk})-A^{\ast}_{jk}-B_{jk}A^{\ast}_{jk},$

we get from Step 2.3 that

$\displaystyle\Phi(A_{jk}+B_{jk})+\Phi(-A^{\ast}_{jk})+\Phi(-B_{jk}A^{\ast}_{jk})$
$\displaystyle=\Phi([\frac{I}{2}\bullet(P_{j}+A_{jk}),P_{k}+B_{jk}]_{*})$
$\displaystyle=[\Phi(\frac{I}{2})\bullet(P_{j}+A_{jk}),P_{k}+B_{jk}]_{*}+[\frac{I}{2}\bullet\Phi(P_{j}+A_{jk}),P_{k}+B_{jk}]_{*}$
$\displaystyle+[\frac{I}{2}\bullet(P_{j}+A_{jk}),\Phi(P_{k}+B_{jk})]_{*}$
$\displaystyle=[\Phi(\frac{I}{2})\bullet(P_{j}+A_{jk}),P_{k}+B_{jk}]_{*}+[\frac{I}{2}\bullet(\Phi(P_{j})+\Phi(A_{jk})),P_{k}+B_{jk}]_{*}$
$\displaystyle+[\frac{I}{2}\bullet(P_{j}+A_{jk}),(\Phi(P_{k})+\Phi(B_{jk}))]_{*}$
$\displaystyle=\Phi([\frac{I}{2}\bullet
P_{j},P_{k}]_{*})+\Phi([\frac{I}{2}\bullet
P_{j},B_{jk}]_{*})+\Phi([\frac{I}{2}\bullet
A_{jk},P_{k}]_{*})+\Phi([\frac{I}{2}\bullet A_{jk},B_{jk}]_{*})$
$\displaystyle=\Phi(B_{jk})+\Phi(A_{jk}-A^{\ast}_{jk})+\Phi(-B_{jk}A^{\ast}_{jk})$
$\displaystyle=\Phi(B_{jk})+\Phi(A_{jk})+\Phi(-A^{\ast}_{jk})+\Phi(-B_{jk}A^{\ast}_{jk}).$

Then

$\Phi(A_{jk}+B_{jk})=\Phi(A_{jk})+\Phi(B_{jk}).$

Step 2.5. For every $A_{jj},B_{jj}\in\mathcal{A}_{jj},1\leq j\leq 2,$ we have

$\Phi(A_{jj}+B_{jj})=\Phi(A_{jj})+\Phi(B_{jj}).$

Let $T=\Phi(A_{jj}+B_{jj})-\Phi(A_{jj})-\Phi(B_{jj}).$ For $1\leq j\neq k\leq
2$, it follows that

$\displaystyle[\Phi(I)\bullet(iP_{k}),A_{jj}+B_{jj}]_{*}+[I\bullet\Phi(iP_{k}),A_{jj}+B_{jj}]_{*}$
$\displaystyle+[I\bullet(iP_{k}),\Phi(A_{jj}+B_{jj})]_{*}$
$\displaystyle=\Phi([I\bullet(iP_{k}),A_{jj}+B_{jj}]_{*})$
$\displaystyle=\Phi([I\bullet(iP_{k}),A_{jj}]_{*})+\Phi([I\bullet(iP_{k}),B_{jj}]_{*})$
$\displaystyle=[\Phi(I)\bullet(iP_{k}),A_{jj}+B_{jj}]_{*}+[I\bullet\Phi(iP_{k}),A_{jj}+B_{jj}]_{*}$
$\displaystyle+[I\bullet(iP_{k}),\Phi(A_{jj})+\Phi(B_{jj})]_{*}.$

From this, we get $[I\bullet(iP_{k}),T]_{*}=0.$ So $T_{jk}=T_{kj}=T_{kk}=0.$
Now we get $T=T_{jj}.$

For every $C_{jk}\in\mathcal{A}_{jk},j\neq k$, it follows from Step 2.4 that

$\displaystyle[\Phi(I)\bullet(A_{jj}+B_{jj}),C_{jk}]_{*}+[I\bullet\Phi(A_{jj}+B_{jj}),C_{jk}]_{*}$
$\displaystyle+[I\bullet(A_{jj}+B_{jj}),\Phi(C_{jk})]_{*}$
$\displaystyle=\Phi([I\bullet(A_{jj}+B_{jj}),C_{jk}]_{*})$
$\displaystyle=\Phi([I\bullet A_{jj},C_{jk}]_{*})+\Phi([I\bullet
B_{jj},C_{jk}]_{*})$
$\displaystyle=[\Phi(I)\bullet(A_{jj}+B_{jj}),C_{jk}]_{*}+[I\bullet\Phi(A_{jj})+\Phi(B_{jj}),C_{jk}]_{*}$
$\displaystyle+[I\bullet(A_{jj}+B_{jj}),\Phi(C_{jk})]_{*}.$

Hence $[I\bullet T_{jj},C_{jk}]_{*}=0$ for all $C_{jk}\in\mathcal{A}_{jk},$
that is, $T_{jj}CP_{k}=0$ for all $C\in\mathcal{A}.$ It follows from
$(\spadesuit)$ and $(\clubsuit)$ that $T=T_{jj}=0,$ proving the step.

Now, it follows from Steps 2.3, 2.4 and 2.5 that $\Phi$ is additive, proving
the Claim 2.
Claim 3. $\Phi(I)$ is a self-adjont central element in $\mathcal{A}$.

On the one hand, we have

$\displaystyle 0$ $\displaystyle=\Phi([I\bullet I,I]_{*})$
$\displaystyle=[\Phi(I)\bullet I,I]_{*}+[I\bullet\Phi(I),I]_{*}+[I\bullet
I,\Phi(I)]_{*}$ $\displaystyle=[2\Phi(I),I]_{*}$
$\displaystyle=2\Phi(I)-2\Phi(I)^{*},$

which implies that $\Phi(I)$ is a self-adjont element in $\mathcal{A}$.

On the other hand, for all $A\in\mathcal{A},$ we get

$\displaystyle 0$ $\displaystyle=\Phi([I\bullet I,A]_{*})$
$\displaystyle=[\Phi(I)\bullet I,A]_{*}+[I\bullet\Phi(I),A]_{*}+[I\bullet
I,\Phi(A)]_{*}$ $\displaystyle=2[2\Phi(I),A]_{*}$
$\displaystyle=4(\Phi(I)A-A\Phi(I)),$

which implies that $\Phi(I)$ is a central element in $\mathcal{A}$.
Claim 4.
$P_{1}\Phi(P_{1})P_{2}=-P_{1}\Phi(P_{2})P_{2},P_{2}\Phi(P_{1})P_{1}=-P_{2}\Phi(P_{2})P_{1},P_{1}\Phi(P_{2})P_{1}=P_{2}\Phi(P_{1})P_{2}=0.$

On the one hand, for $1\leq j\neq k\leq 2$, it follows from Claim 3 that

$\displaystyle 0$ $\displaystyle=\Phi([I\bullet P_{j},P_{k}]_{*})$
$\displaystyle=[\Phi(I)\bullet
P_{j},P_{k}]_{*}+[I\bullet\Phi(P_{j}),P_{k}]_{*}+[I\bullet
P_{j},\Phi(P_{k})]_{*}$
$\displaystyle=[2\Phi(P_{j}),P_{k}]_{*}+[2P_{j},\Phi(P_{k})]_{*}$
$\displaystyle=2\Phi(P_{j})P_{k}-2P_{k}\Phi(P_{j})^{*}+2P_{j}\Phi(P_{k})-2\Phi(P_{k})P_{j}.$

Multiplying both sides of the above equation by $P_{j}$ and $P_{k}$ from the
left and right respectively, we obtain that
$P_{1}\Phi(P_{1})P_{2}=-P_{1}\Phi(P_{2})P_{2}$ and
$P_{2}\Phi(P_{1})P_{1}=-P_{2}\Phi(P_{2})P_{1}$.

On the other hand, we get

$\displaystyle 0$ $\displaystyle=\Phi([I\bullet(iP_{j}),P_{k}]_{*})$
$\displaystyle=[\Phi(I)\bullet(iP_{j}),P_{k}]_{*}+[I\bullet\Phi(iP_{j}),P_{k}]_{*}+[I\bullet(iP_{j}),\Phi(P_{k})]_{*}$
$\displaystyle=[2\Phi(iP_{j}),P_{k}]_{*}+[2iP_{j},\Phi(P_{k})]_{*}$
$\displaystyle=2\Phi(iP_{j})P_{k}-2P_{k}\Phi(iP_{j})^{*}+2i(P_{j}\Phi(P_{k})+\Phi(P_{k})P_{j}).$

Multiplying both sides of the above equation by $P_{j}$ from the left and
right respectively, we obtain that
$P_{1}\Phi(P_{2})P_{1}=P_{2}\Phi(P_{1})P_{2}=0.$
Claim 5. $P_{1}\Phi(P_{1})P_{1}=P_{2}\Phi(P_{2})P_{2}=0.$

For every $A_{12}\in\mathcal{A}_{12}$, on the one hand, it follows from Claim
2 and Claim 3 that

$\displaystyle 2\Phi(A_{12})$ $\displaystyle=\Phi([I\bullet
P_{1},A_{12}]_{*})$ $\displaystyle=[\Phi(I)\bullet
P_{1},A_{12}]_{*}+[I\bullet\Phi(P_{1}),A_{12}]_{*}+[I\bullet
P_{1},\Phi(A_{12})]_{*}$
$\displaystyle=[2\Phi(I)P_{1},A_{12}]_{*}+[2\Phi(P_{1}),A_{12}]_{*}+[2P_{1},\Phi(A_{12})]_{*}$
$\displaystyle=2\Phi(I)A_{12}+2\Phi(P_{1})A_{12}-2A_{12}\Phi(P_{1})^{*}+2P_{1}\Phi(A_{12})-2\Phi(A_{12})P_{1}.$

Multiplying both sides of the above equation by $P_{1}$ and $P_{2}$ from the
left and right respectively, by Claim 4, we get that

$P_{1}\Phi(P_{1})A_{12}+\Phi(I)A_{12}=0.$ (2. 1)

On the other hand, we have

$\displaystyle 2\Phi(A_{12})$ $\displaystyle=\Phi([P_{1}\bullet
P_{1},A_{12}]_{*})$ $\displaystyle=[\Phi(P_{1})\bullet
P_{1},A_{12}]_{*}+[P_{1}\bullet\Phi(P_{1}),A_{12}]_{*}+[P_{1}\bullet
P_{1},\Phi(A_{12})]_{*}$
$\displaystyle=[\Phi(P_{1})P_{1}+P_{1}\Phi(P_{1})^{*},A_{12}]_{*}+[P_{1}\Phi(P_{1})+\Phi(P_{1})P_{1},A_{12}]_{*}+[2P_{1},\Phi(A_{12})]_{*}$
$\displaystyle=\Phi(P_{1})A_{12}+P_{1}\Phi(P_{1})^{*}A_{12}-A_{12}\Phi(P_{1})P_{1}+P_{1}\Phi(P_{1})A_{12}$
$\displaystyle+\Phi(P_{1})A_{12}-A_{12}\Phi(P_{1})^{*}P_{1}+2P_{1}\Phi(A_{12})-2\Phi(A_{12})P_{1}.$

Multiplying both sides of the above equation by $P_{1}$ and $P_{2}$ from the
left and right respectively, we get that

$3P_{1}\Phi(P_{1})A_{12}+P_{1}\Phi(P_{1})^{*}A_{12}=0.$ (2. 2)

Finally, we also have that

$\displaystyle 2\Phi(A_{12})$ $\displaystyle=\Phi([P_{1}\bullet
I,A_{12}]_{*})$ $\displaystyle=[\Phi(P_{1})\bullet
I,A_{12}]_{*}+[P_{1}\bullet\Phi(I),A_{12}]_{*}+[P_{1}\bullet
I,\Phi(A_{12})]_{*}$
$\displaystyle=[\Phi(P_{1})+\Phi(P_{1})^{*},A_{12}]_{*}+[2P_{1}\Phi(I),A_{12}]_{*}+[2P_{1},\Phi(A_{12})]_{*}$
$\displaystyle=(\Phi(P_{1})+\Phi(P_{1})^{*})A_{12}-A_{12}(\Phi(P_{1})+\Phi(P_{1})^{*})+2\Phi(I)A_{12}+2P_{1}\Phi(A_{12})-2\Phi(A_{12})P_{1}.$

Multiplying both sides of the above equation by $P_{1}$ and $P_{2}$ from the
left and right respectively, by Claim 4, we get that

$P_{1}\Phi(P_{1})A_{12}+P_{1}\Phi(P_{1})^{*}A_{12}+2\Phi(I)A_{12}=0.$ (2. 3)

It follows from Eq. (2\. 2) and Eq. (2\. 3) that

$P_{1}\Phi(P_{1})A_{12}-\Phi(I)A_{12}=0.$ (2. 4)

Now, by Eq. (2\. 1) and Eq. (2\. 4), we have $P_{1}\Phi(P_{1})A_{12}=0$, that
is $P_{1}\Phi(P_{1})P_{1}AP_{2}=0$ for all $A\in\mathcal{A}$. It follows from
$(\clubsuit)$ that $P_{1}\Phi(P_{1})P_{1}=0.$ Similarly, we can prove
$P_{2}\Phi(P_{2})P_{2}=0.$
Claim 6. $\Phi(I)=0.$

By Claims 2, 4 and 5, we can get that

$\Phi(I)=\Phi(P_{1})+\Phi(P_{2})=P_{1}\Phi(P_{1})P_{2}+P_{2}\Phi(P_{1})P_{1}+P_{1}\Phi(P_{2})P_{2}+P_{2}\Phi(P_{2})P_{1}=0.$

Claim 7. For all $A,B\in\mathcal{A}$, we have
$\Phi([A,B]_{*})=[\Phi(A),B]_{*}+[A,\Phi(B)]_{*}$.

It follows from Claim 2 and Claim 6 that

$\displaystyle 2\Phi([A,B]_{*})$ $\displaystyle=\Phi([I\bullet A,B]_{*})$
$\displaystyle=[\Phi(I)\bullet A,B]_{*}+[I\bullet\Phi(A),B]_{*}+[I\bullet
A,\Phi(B)]_{*}$ $\displaystyle=[2\Phi(A),B]_{*}+[2A,\Phi(B)]_{*}$
$\displaystyle=2([\Phi(A),B]_{*}+[A,\Phi(B)]_{*}),$

which implies that $\Phi([A,B]_{*})=[\Phi(A),B]_{*}+[A,\Phi(B)]_{*}$.
Claim 8. For all $A\in\mathcal{A}$, $\Phi(A^{*})=\Phi(A)^{*}.$

For every $A\in\mathcal{A}$, by Claims 2, 6 and 7, we have

$\Phi(A)-\Phi(A^{*})=\Phi([A,I]_{*})=[\Phi(A),I]_{*}=\Phi(A)-\Phi(A)^{*}.$

Hence $\Phi(A^{*})=\Phi(A)^{*}.$

Now, let $T=P_{1}\Phi(P_{1})P_{2}-P_{2}\Phi(P_{1})P_{1}.$ Then $T^{*}=-T$.
Defining a map $\phi:\mathcal{A}\rightarrow\mathcal{A}$ by
$\phi(A)=\Phi(A)-(AT-TA)$ for all $A\in\mathcal{A}.$ It is easy to verify that
$\phi$ has the following properties.
Claim 9.

  * (1)

For all $A,B\in\mathcal{A}$,
$\phi([A,B]_{*})=[\phi(A),B]_{*}+[A,\phi(B)]_{*};$

  * (2)

$\phi(P_{i})=0,i=1,2;$

  * (3)

$\phi$ is additive;

  * (4)

$\phi$ is an additive derivation if and only if $\Phi$ is an additive
derivation.

Claim 10. $\phi(\mathcal{A}_{ij})\subseteq\mathcal{A}_{ij},i,j=1,2.$

Let $A_{ij}\in\mathcal{A}_{ij},1\leq i\neq j\leq 2.$ On the one hand, it
follows from $\phi(P_{i})=0$ that

$\phi(A_{ij})=\phi([P_{i},A_{ij}]_{*})=[P_{i},\phi(A_{ij})]_{*}=P_{i}\phi(A_{ij})-\phi(A_{ij})P_{i}.$

Hence
$P_{i}\phi(A_{ij})P_{i}=P_{j}\phi(A_{ij})P_{j}=P_{j}\phi(A_{ij})P_{i}=0.$ So
$\phi({A}_{ij})=P_{i}\phi({A}_{ij})P_{j}\in\mathcal{A}_{ij},1\leq i\neq j\leq
2.$

Let $A_{ii}\in\mathcal{A}_{ii},i=1,2.$ Then

$0=\phi([P_{j},A_{ii}]_{*})=[P_{j},\phi(A_{ii})]_{*}=P_{j}\phi(A_{ii})-\phi(A_{ii})P_{j}.$

Hence $P_{i}\phi(A_{ii})P_{j}=P_{j}\phi(A_{ii})P_{i}=0.$ Now we let
$\phi(A_{ii})=P_{i}\phi(A_{ii})P_{i}+P_{j}\phi(A_{ii})P_{j}.$ For any
$B_{ij}\in\mathcal{A}_{ij},1\leq i\neq j\leq 2,$ it follows from
$\phi(B_{ij})\in\mathcal{A}_{ij}$ that

$0=\phi([B_{ij},A_{ii}]_{*})=[\phi(B_{ij}),A_{ii}]_{*}+[B_{ij},\phi(A_{ii})]_{*}=B_{ij}P_{j}\phi(A_{ii})P_{j}-P_{j}\phi(A_{ii})P_{j}B_{ij}^{*}.$

So $P_{j}\phi(A_{ii})P_{j}B_{ij}^{*}=0$, that is
$P_{j}\phi(A_{ii})P_{j}BP_{i}=0$ holds true for any $B\in\mathcal{A}.$ It
follows from $(\spadesuit)$ and $(\clubsuit)$ that $P_{j}\phi(A_{ii})P_{j}=0.$
Now we get $\phi({A}_{ii})=P_{i}\phi({A}_{ii})P_{i}\in\mathcal{A}_{ii},i=1,2.$
Claim 11. Let $A_{ii},B_{ii}\in\mathcal{A}_{ii}$ and
$A_{ij},B_{ij}\in\mathcal{A}_{ij},1\leq i\neq j\leq 2.$ Then

$\phi(A_{ii}B_{ii})=\phi(A_{ii})B_{ii}+A_{ii}\phi(B_{ii}),\phi(A_{ii}B_{ij})=\phi(A_{ii})B_{ij}+A_{ii}\phi(B_{ij}),$
$\phi(A_{ij}B_{ji})=\phi(A_{ij})B_{ji}+A_{ij}\phi(B_{ji}),\phi(A_{ij}B_{jj})=\phi(A_{ij})B_{jj}+A_{ij}\phi(B_{jj}).$

It follows from Claim 10 that

$\displaystyle\phi(A_{ii}B_{ij})$
$\displaystyle=\phi([A_{ii},B_{ij}]_{*})=[\phi(A_{ii}),B_{ij}]_{*}+[A_{ii},\phi(B_{ij}]_{*})$
$\displaystyle=\phi(A_{ii})B_{ij}+A_{ii}\phi(B_{ij}).$

For any $C_{ij}\in\mathcal{A}_{ij},$ we have

$\displaystyle\phi(A_{ii}B_{ii})C_{ij}+A_{ii}B_{ii}\phi(C_{ij})$
$\displaystyle=\phi(A_{ii}B_{ii}C_{ij})$
$\displaystyle=\phi(A_{ii})B_{ii}C_{ij}+A_{ii}\phi(B_{ii}C_{ij})$
$\displaystyle=\phi(A_{ii})B_{ii}C_{ij}+A_{ii}\phi(B_{ii})C_{ij}+A_{ii}B_{ii}\phi(C_{ij}).$

Then $(\phi(A_{ii}B_{ii})-\phi(A_{ii})B_{ii}-A_{ii}\phi(B_{ii}))C_{ij}=0$ for
any $C_{ij}\in\mathcal{A}_{ij}.$ It follows from $(\spadesuit)$ and
$(\clubsuit)$ that $\Phi(A_{ii}B_{ii})=\Phi(A_{ii})B_{ii}+A_{ii}\Phi(B_{ii}).$

It follows from Claim 10 that

$\displaystyle\phi(A_{ij}B_{ji})$ $\displaystyle=\phi([A_{ij},B_{ji}]_{*})$
$\displaystyle=[\phi(A_{ij}),B_{ji}]_{*}+[A_{ij},\phi(B_{ji})]_{*}$
$\displaystyle=\phi(A_{ij})B_{ji}+A_{ij}\phi(B_{ji}).$

For any $C_{ji}\in\mathcal{A}_{ji}$, we have

$\displaystyle\phi(C_{ji})A_{ij}B_{jj}+C_{ji}\phi(A_{ij}B_{jj})$
$\displaystyle=\phi(C_{ji}A_{ij}B_{jj})$
$\displaystyle=\phi(C_{ji}A_{ij})B_{jj}+C_{ji}A_{ij}\Phi(B_{jj})$
$\displaystyle=\phi(C_{ji})A_{ij}B_{jj}+C_{ji}\phi(A_{ij})B_{jj}+C_{ji}A_{ij}\phi(B_{jj}).$

Hence $C_{ji}(\phi(A_{ij}B_{jj})-\phi(A_{ij})B_{jj}-A_{ij}\phi(B_{jj}))=0$ for
any $C_{ji}\in\mathcal{A}_{ji}.$ It follows from $(\spadesuit)$ and
$(\clubsuit)$ that $\phi(A_{ij}B_{jj})=\phi(A_{ij})B_{jj}+A_{ij}\phi(B_{jj}).$
Claim 12. $\Phi(AB)=\Phi(A)B+A\Phi(B)$ for all $A,B\in\mathcal{A}.$

Write $A=\sum^{2}_{i,j=1}A_{ij},B=\sum^{2}_{i,j=1}B_{ij}\in\mathcal{A}$. Then
$AB=A_{11}B_{11}+A_{11}B_{12}+A_{12}B_{21}+A_{12}B_{22}+A_{21}B_{11}+A_{21}B_{12}+A_{22}B_{21}+A_{22}B_{22}$.
It follows from Claim 11 and the additivity of $\phi$ that
$\phi(AB)=\phi(A)B+A\phi(B).$ So $\Phi(AB)=\Phi(A)B+A\Phi(B).$

Now, by Claims 2, 8 and 12, we have proved that $\Phi$ is an additive
$*-$derivation. This completes the proof of Theorem 2.1. ∎

##  3 Corollaries

In this section, we present some corollaries of the main result. An algebra
$\mathcal{A}$ is called prime if $A\mathcal{A}B=\\{0\\}$ for
$A,B\in\mathcal{A}$ implies either $A=0$ or $B=0$. Observing that prime
$*$-algebras satisfy $(\spadesuit)$ and $(\clubsuit)$, we have the following
corollary.

######  Corollary 3.1.

Let $\mathcal{A}$ be a prime $*$-algebra with unit $I$ and $P$ be a nontrivial
projection in $\mathcal{A}$. Then $\Phi$ is a nonlinear mixed Jordan triple
$*$-derivation on $\mathcal{A}$ if and only if $\Phi$ is an additive
$*-$derivation.

A von Neumann algebra $\mathcal{M}$ is a weakly closed, self-adjoint algebra
of operators on a Hilbert space $\mathcal{H}$ containing the identity operator
$I$. It is shown in [4] and [10] that if a von Neumann algebra has no central
summands of type $I_{1}$, then $\mathcal{M}$ satifies $(\spadesuit)$ and
$(\clubsuit)$. Now we have the following corollary.

######  Corollary 3.2.

Let $\mathcal{M}$ be a von Neumann algebra with no central summands of type
$I_{1}$. Then $\Phi:\mathcal{M}\rightarrow\mathcal{M}$ is a nonlinear mixed
Jordan triple $*$-derivation if and only if $\Phi$ is an additive
$*-$derivation.

$\mathcal{M}$ is a factor von Neumann algebra if its center only contains the
scalar operators. It is well known that a factor von Neumann algebra is prime
and then we have the following corollary.

######  Corollary 3.3.

Let $\mathcal{M}$ be a factor von Neumann algebra with dim$\mathcal{M}\geq 2$.
Then $\Phi:\mathcal{M}\rightarrow\mathcal{M}$ is a nonlinear mixed Jordan
triple $*$-derivation if and only if $\Phi$ is an additive $*-$derivation.

$B(\mathcal{H})$ denotes the algebra of all bounded linear operators on a
complex Hilbert space $\mathcal{H}$. We denote the subalgebra of all bounded
finite rank operators by $\mathcal{F}(H)\subseteq B(\mathcal{H})$. We call a
subalgebra $\mathcal{A}$ of $B(\mathcal{H})$ a standard operator algebra if it
contains $\mathcal{F}(H)$. Now we have the following corollary.

######  Corollary 3.4.

Let $\mathcal{H}$ be an infinite dimensional complex Hilbert space and
$\mathcal{A}$ be a standard operator algebra on $\mathcal{H}$ containing the
identity operator $I$. Suppose that $\mathcal{A}$ is closed under the adjoint
operation. Then $\Phi:\mathcal{A}\rightarrow\mathcal{A}$ is a nonlinear mixed
Jordan triple $*$-derivation if and only if $\Phi$ is a linear $*$-derivation.
Moreover, there exists an operator $T\in B(\mathcal{H})$ satisfying
$T+T^{*}=0$ such that $\Phi(A)=AT-TA$ for all $A\in A$, i.e., $\Phi$ is inner.

###### Proof.

Since $\mathcal{A}$ is prime, we have that $\Phi$ is an additive
$*$-derivation. It follows from [20] that $\Phi$ is a linear inner derivation,
i.e., there exists an operator $S\in B(\mathcal{H})$ such that $\Phi(A)=AS-
SA$. Using the fact $\Phi(A^{*})=\Phi(A)^{*}$, we have

$A^{*}S-SA^{*}=\Phi(A^{*})=\Phi(A)^{*}=-A^{*}S^{*}+S^{*}A^{*}$

for all $A\in A$. This leads to $A^{*}(S+S^{*})=(S+S^{*})A^{*}$. Hence,
$S+S^{*}=\lambda I$ for some $\lambda\in\mathbb{R}$. Let us set
$T=S-\frac{1}{2}\lambda I$. One can check that $T+T^{*}=0$ such that
$\Phi(A)=AT-TA$. ∎

## References

  * [1] Z. Bai, S. Du, Maps preserving products $XY-YX^{*}$ on von Neumann algebras, J. Math. Anal. Appl. 386 (2012) 103-109.
  * [2] M. Bre$\check{s}$ar, A. Fo$\check{s}$ner, On ring with involution equipped with some new product, Publ. Math. Debrecen 57 (2000) 121-134.
  * [3] J. Cui, C. K. Li, Maps preserving product $XY-YX^{\ast}$ on factor von Neumann algebras, Linear algebra Appl. 431 (2009) 833-842.
  * [4] L. Dai, F. Lu, Nonlinear maps preserving Jordan $*$-products, J. Math. Anal. Appl. 409 (2014) 180-188.
  * [5] F. Fu, R. An, Equivalent characterization of $*$-derivations on von Neumann algebras, Linear Multilinear Algebra. 67 (2019) 527-541.
  * [6] D. Huo, B. Zheng and H. Liu, Nonlinear maps preserving Jordan triple $\eta$-$*$-products, J. Math. Anal. Appl. 430 (2015) 830-844.
  * [7] W. Jing, Nonlinear $*$-Lie derivations of standard operator algebras, Quaest. Math. 39 (2016) 1037-1046.
  * [8] C. Li, Q. Chen, T. Wang, Nonlinear maps preserving the Jordan triple $*$-product on factors, Chin. Ann. Math. Ser. B 39(2018) 633-642.
  * [9] C. Li, F. Lu and X. Fang, Mappings preserving new product $XY+YX^{*}$ on factor von Neumann algebras, Linear Algebra Appl. 438 (2013) 2339-2345.
  * [10] C. Li , F. Lu, X. Fang Nonlinear $\xi$-Jordan $*$-derivations on von Neumann algebras, Linear Multilinear Algebra. 62 (2014) 466-473.
  * [11] C. Li, F. Lu, Nonlinear maps preserving the Jordan triple 1-$*$-product on von Neumann algebras, Complex Anal. Oper. Theory. 11 (2017) 109-117.
  * [12] C. Li, F. Lu, Nonlinear maps preserving the Jordan triple $*$-product on von Neumann algebras, Ann. Funct. Anal. 7 (2016) 496-507.
  * [13] C. Li, F. Zhao, Q. Chen, Nonlinear skew Lie triple derivations between factors, Acta Math. Sinica (English Series), 32 (2016) 821-830.
  * [14] W. Lin, Nonlinear $*$-Lie-type derivations on von Neumann algebras, Acta Math. Hungar. 156 (2018) 112-131.
  * [15] W. Lin, Nonlinear $*$-Lie-type derivations on standard operator algebras, Acta Math. Hungar. 154 (2018) 480-500.
  * [16] L. Moln$\acute{a}$r, A condition for a subspace of $\mathcal{B}$(H) to be an ideal, Linear Algebra Appl. 235 (1996) 229-234.
  * [17] P. $\check{S}$emrl, Quadratic functionals and Jordan $\ast$-derivations, Studia Math. 97 (1991) 157-165.
  * [18] P. $\check{S}$emrl, Quadratic and quasi-quadratic functionals, Proc. Amer. Math. Soc. 119 (1993) 1105-1113.
  * [19] P. $\check{S}$emrl, On Jordan $*$-derivations and an application, Colloq. Math. 59 (1990) 241-251.
  * [20] P. $\check{S}$emrl, Additive derivations of some operator algebras, Illinois J. Math. 35 (1991) 234-240.
  * [21] A. Taghavi, H. Rohi and V. Darvish, Non-linear $*$-Jordan derivations on von Neumann algebras, Linear Multilinear Algebra, 64 (2016), 426-439.
  * [22] W. Yu, J. Zhang, Nonlinear *-Lie derivations on factor von Neumann algebras, Linear Algebra Appl. 437 (2012) 1979-1991.
  * [23] F. Zhang, Nonlinear skew Jordan derivable maps on factor von Neumann algebras, Linear Multilinear Algebra, 64 (2016) 2090-2103.
  * [24] F. Zhao, C. Li, Nonlinear $*$-Jordan triple derivations on von Neumann algebras, Math. Slovaca 68 (2018) 163-170.
  * [25] F. Zhao, C. Li, Nonlinear maps preserving the Jordan triple $*$-product between factors, Indag. Math. 29 (2018) 619-627.


# Two-Pass End-to-End ASR Model Compression

###### Abstract

Speech recognition on smart devices is challenging owing to the small memory
footprint. Hence small size ASR models are desirable. With the use of popular
transducer-based models, it has become possible to practically deploy
streaming speech recognition models on small devices [1]. Recently, the two-
pass model [2] combining RNN-T and LAS modules has shown exceptional
performance for streaming on-device speech recognition.

In this work, we propose a simple and effective approach to reduce the size of
the two-pass model for memory-constrained devices. We employ a popular
knowledge distillation approach in three stages using the Teacher-Student
training technique. In the first stage, we use a trained RNN-T model as a
teacher model and perform knowledge distillation to train the student RNN-T
model. The second stage uses the shared encoder and trains a LAS rescorer for
student model using the trained RNN-T+LAS teacher model. Finally, we perform
deep-finetuning for the student model with a shared RNN-T encoder, RNN-T
decoder, and LAS rescorer. Our experimental results on standard LibriSpeech
dataset show that our system can achieve a high compression rate of 55%
without significant degradation in the WER compared to the two-pass teacher
model.

Index Terms—  speech recognition, on-device, online, streaming ASR, model
compression, knowledge distillation

##  1 Introduction

Nowadays, various smart devices such as smart speakers, mobile phones, health
care devices, surveillance devices, smartwatches, and many more have become a
part of daily life. Since most of these devices are small in size, it is
difficult to have a reasonable-sized keyboard. Moreover, speech is a more
natural way to interact with these devices. Hence there is a growing need to
deploy speech recognition models on small devices. Further, it is desirable to
have streaming output (or online output) to improve user’s experience.
Recently, there has been tremendous research interest in the field of speech
recognition for on-device online streaming scenarios [1, 2, 3, 4, 5, 6, 7].

Most smart devices have a low memory footprint. Deploying large size Automatic
Speech Recognition (ASR) models is infeasible on such devices. Hence, many
applications transfer the user’s speech data to the large server, processes
the spoken sentence, and then revert back with the proper response. However,
this protocol incurs high latency due to data transfer time between the device
and the server. Furthermore, transferring user’s data online may create data
privacy concerns. Hence it is desirable to have the ASR model operating on-
device.

End-to-End (E2E) approaches have achieved comparable performances to the
traditional systems by having a single E2E model instead of separate
components [8]. These approaches includes Connectionist Temporal
Classification (CTC) [9], Attention-based Encoder-Decoder (AED) [10], and
Recurrent Neural Network Transducer (RNN-T) [1] based models. The AED models
need the complete audio beforehand to start decoding the audio, making AED
models unsuitable for streaming purposes. The RNN-T and CTC models can be
deployed for streaming scenarios due to their frame-synchronous output nature.
RNN-T model is preferred over CTC because it has token-dependency modeling
with a prediction network. Additionally, CTC also has a strong assumption of
output tokens to be independent, which makes RNN-T more preferable over CTC
networks. However, RNN-T models consume significant memory as it contains
transcription, prediction, joint networks and have higher decoding latency
because of framewise predictions.

Online speech recognition task demands both smaller model size and lower
latency for small memory footprint devices, necessitating the need for model
compression techniques for such models to be deployable for streaming
purposes. Knowledge distillation [11, 12, 13] is a popular approach for
compression of neural models. It refers to the idea of model compression by
training a smaller model (student), step-by-step, with the help of an already
trained bigger model (teacher). The student model can be trained to replicate
the outputs of the teacher model at different levels. The student model with
fewer parameters tries to mimic the teacher model’s behavior (with a large
number of parameters).

Knowledge distillation (KD) has been successfully employed in many speech
recognition tasks. Authors in [14] improves performance of a Convolutional
Neural Network (CNN) [15] with the help of ensemble of VGG [16] and LSTM [17,
18] as the teacher models. KD has also been helpful in attention-based models
for Neural Machine Translation (NMT) task [19] and maximum mutual information-
based training [20]. Recently, authors in [21] also use KD for E2E speech
translation task. In [22], KD is used for distilling knowledge from the
bidirectional to unidirectional CTC models. Authors in [23] compare the
performance of sequence-level and frame-level KD losses for CTC models. In
[24], an efficient strategy to transfer knowledge from the offline RNN-T model
to streaming one is proposed. Recently, authors in [25] proposes a simple and
efficient approach for KD with RNN-T models, which requires less memory
overhead during training, leading to faster convergence.

In this paper, we propose a three-stage approach for compressing a two-pass
streaming ASR model using KD. In the first stage, we perform KD to train a
student RNN-T model using a trained teacher RNN-T model [25]. The second stage
trains an additional Listen, Attend, and Spell (LAS) [26] module with an RNN-T
encoder. Finally, in the third stage, we finetune the model by combining the
RNN-T encoder, the RNN-T decoder, and the LAS module. With the proposed
approach, we achieve 55% of compression without significant degradation in
performance. We also conduct various experimental analyses to study the effect
of KD in different stages of the proposed approach.

##  2 Two-pass E2E Model

The two-pass model [2] consists of the RNN-T and the LAS modules. Each of the
modules is described in the following sections.

###  2.1 Recurrent Neural Network Tranducer

The RNN-T is an encoder-decoder sequence-to-sequence prediction model that is
widely used for end-to-end streaming speech recognition systems [27]. The
RNN-T model comprises the transcription, predication, and a joint network. The
RNN-T training aims at minimizing the negative log posterior probability as
given by the following equation:

$L_{RNN-T}=-\ln~{}P(\boldsymbol{y^{*}}|\boldsymbol{x})$ (1)

where $\boldsymbol{y^{*}}$ is a target sequence and $\boldsymbol{x}$ is input
sequence. The probability $P(\boldsymbol{y^{*}}|\boldsymbol{x})$ is computed
using $T\times U$ probability lattice, where $T$ denotes the input sequence
length and $U$ is the output sequence length [27]. Each node in the lattice
outputs a probability distribution $P(k|t,u)$ over target vocab including a
blank symbol $\phi$, where $k$ represents the output label. The RNN-T model
can be represented with the following set of equations:

$\begin{gathered}h_{t}=LSTMs(x_{t},h_{t-1})\\\ g_{u}=LSTMs(y_{u},g_{u-1})\\\
z_{t,u}=V^{T}\tanh(Wh_{t}+Ug_{u})\\\ P(k|t,u)=softmax(z_{t,u})\end{gathered}$
(2)

Equation (2) correspond to the outputs from the transcription, prediction and
joint networks (in order). The $P(\boldsymbol{y}|\boldsymbol{x})$ is
calculated by summing over all possible alignments $\boldsymbol{a}$.

$P(\boldsymbol{y}|\boldsymbol{x})=\sum_{\boldsymbol{a}\in
B^{-1}(y)}P(\boldsymbol{a}|\boldsymbol{x})$ (3)

where $B$ is a function that removes blank symbols from alignments. Summing
over all possible alignments is intractable, and hence forward-backward
algorithm is used to compute the same. More information on the RNN-T model can
be found in [27].

Fig. 1: Architecture of Two-pass model.

(a) Stage-1: Teacher frozen

(b) Stage-2: Student (encoder) frozen

(c) Stage-3: Deep fine-tuning

Fig. 2: The proposed three-stage approach for compressing the two-pass ASR
model. Blocks colored in orange are kept frozen during training. (a) Stage-1:
KD teacher-student training is used to trained the student RNN-T model.
Coarsed KL-Divergence represents distillation loss given in Equation (7). (b)
Stage-2: An LAS rescorer is added to the RNN-T encoder and trained using KD.
(c) Stage-3: The two-pass student model is deep-finetuned.

###  2.2 Two-pass model with RNN-T and LAS

As shown in Fig. 1, the two-pass ASR model consists of an additional encoder
layer on top of the shared RNN-T encoder. It also has a LAS decoder in
addition to RNN-T model components. The LAS module is used to improve the
streaming ASR output from RNN-T. Training two-pass model [2] is not trivial
because of its known issue of non-converging loss. The training of the two-
pass model is carried out in three stages [2] as follows:

  1. 1.

Train RNN-T model (Section 2.1) [27].

  2. 2.

Train additional two-pass components with cross-entropy loss using pre-trained
shared RNN-T encoder, which remains frozen.

  3. 3.

Deep-finetune the entire network using RNN-T and LAS loss.

##  3 Proposed Two-pass model Compression

Knowledge distillation is used for compression by using a teacher model, where
another student model, smaller in size, is trained to mimic the teacher model.
To mimic the teacher model, KL-Divergence between the probability distribution
of teacher and student model is used as distillation loss which is combined
with original training loss using some trade-off. Similar to training two-pass
model, KD with two-pass models is also not trivial. Motivated from the
training procedure of the two-pass model, the following is the proposed three-
stage approach used for compression of the two-pass model as shown in Fig. 2.

###  3.1 Stage-1: KD training

In the first stage, we train a student RNN-T model using KD from the trained
RNN-T teacher model, which is kept frozen as shown in Fig. 2(a). As discussed
in Section 2.1, the RNN-T loss is computed by summing over all possible
alignments. Thus KL-Divergence between probability distributions of teacher
and student model can be given by:

$L_{distill}=\sum_{t,u}\sum_{k}P^{T}(k|t,u)\ln\frac{P^{T}(k|t,u)}{P^{S}(k|t,u)}$
(4)

where $P^{T}$ and $P^{S}$ are probability distributions of teacher and student
RNN-T models at time step $t$. As we can see, distillation loss computation
requires an additional memory size of $\mathcal{O}(T\times U\times K)$ per
utterance which is too expensive and impractical. Hence the $K$ dimensional
probability distribution is reduced to three dimensions by only taking in
account conditional probabilities of $y_{u+1}$, $\phi$ and remainder tokens at
any lattice node $(t,u)$. The corresponding transition probabilities are as
follows [25]:

$\begin{gathered}P_{y}(t,u)=P(y_{u+1}|t,u)=y(t,u)\\\
P_{\phi}(t,u)=P(\phi|t,u)=\phi(t,u)\\\
P_{r}(t,u)=1-P_{y}(t,u)-P_{\phi}(t,u)\end{gathered}$ (5)

Thus distillation loss can be defined as KL-Divergence between probability
distributions of teacher and student model as follows:

$L_{distill}=\sum_{t,u}\sum_{l\in\\{y,\phi,r\\}}P^{T}_{l}(t,u)\ln\frac{{P^{T}_{l}}(t,u)}{P^{S}_{l}(t,u)}$
(6)

The overall loss in the Stage-1 of the knowledge distillation training is
given by a linear combination of $L_{RNN-T}$ and $L_{distill}$ as follows:

$L_{first}=\beta L_{distill}+(1-\beta)L_{RNN-T}$ (7)

where $\beta$ represents the distillation loss weight.

###  3.2 Stage-2: KD training

As shown in Fig. 2(b), in Stage-2 KD training, a LAS rescorer module is added
to the RNN-T encoder obtained from Stage-1 training. The RNN-T encoder of the
student model is kept frozen, and only the parameters of LAS rescorer modules
are updated during backpropagation. The teacher model is kept frozen during KD
training.

The distillation loss in the Stage-2 KD training is given by KL-Divergence
between two-pass LAS rescorer’s probability distribution of teacher and
student models as follows:

$L_{distill}^{{}^{\prime}}=\sum_{t}\sum_{k}P^{T}_{2}(k|t)\ln\frac{P^{T}_{2}(k|t)}{P^{S}_{2}(k|t)}$
(8)

where $P^{T}_{2}$ and $P^{S}_{2}$ are second pass LAS rescorer’s probability
distributions of teacher and student model respectively at each time step $t$
over target vocab $k$ including a blank symbol. The total loss for the Stage-2
training is given by

$L_{second}=\gamma L_{distill}^{{}^{\prime}}+(1-\gamma)L_{LAS}$ (9)

where $\gamma$ is a tunable hyperparameter which manages trade-off between
these two losses.

###  3.3 Stage-3: Deep finetuning

As shown in Fig. 2(c), in the third stage, we combine the RNN-T encoder
(shared), the RNN-T decoder, and the LAS rescorer. Deep-finetuning is
performed on this model. The combined loss for training used in the Stage-3 is
given by

$L_{combined}=\lambda L_{RNN-T}+(1-\lambda)L_{LAS}$ (10)

where $\lambda$ is a tunable hyperparameter which is set to 0.5 in our
trainings throughout this work.

Evaluation can be done via “$2^{nd}$ beam search” mode or “rescoring” mode
[2]. In this paper, we use the “rescoring” mode for evaluation. In rescoring
mode, we run LAS decoder on the top-K hypotheses predicted by the RNN-T
decoder in teacher-forcing mode [2] with attention on additional encoder layer
output to rescore the beams. We then pick the beam with the highest score from
the LAS module. From this point onwards, we will use the term LAS rescorer for
the additional second pass module, comprising of an additional LSTM layer
followed by LAS decoder, as also shown in Fig. 1.

##  4 Experimental Setup

###  4.1 Dataset, features, and augmentation

We train our models on the standard LibriSpeech dataset [28] containing 960
hours of training speech data and evaluate on test-clean and test-other
datasets containing 2620 and 2939 utterances, respectively. We use Tensorflow
2 Keras API to build the complete training pipeline. The target vocab contains
around 10K Byte Pair Encoding (BPE) [29] units and are embedded in
621-dimensional space for the teacher model and 390-dimensional space for the
smaller model. We use 40 dimensional Mel Frequency Cepstral Coefficients
(MFCCs) [30] computed from a 25 msec window and shifted every 10 msec as an
input to the encoder network. We use a popular SpecAugment [31] technique for
data augmentation during training. The beam size of 8 is used for decoding.
For two-pass models, evaluation is done in rescoring mode.

###  4.2 RNN-T model

As discussed in Section 2.1, the RNN-T model has three components,
transcription network, prediction network, and joint network. It is an
encoder-decoder framework where the transcription network acts as an encoder,
and the prediction network coupled with the joint network acts as a decoder.
The transcription network consists of 6 unidirectional LSTM layers. We use
max-pooling of 2 in the first three layers, with an overall factor of 8, to
reduce the training time and latency of the model. We use dropout [32] of 0.2
after each layer except the first one. To ensure proper convergence, we
gradually add layers in the initial stages of training starting from 2 layers
[33]. The prediction network comprises an embedding layer followed by one LSTM
layer, which acts as a conditional Language Model (LM) model. The joint
network consists of two dense layers, one each for transcription and
prediction output, respectively. It also consists of an additional dense layer
to project the features into vocab space which acts as logits for RNN-T loss
calculation. The initial learning rate of $5\times 10^{-4}$ is used with Adam
optimizer [34], which is decayed exponentially by a factor of $0.9$ after
every 20K steps. Student RNN-T model is trained with the same hyperparameters
with an additional distillation loss linearly combined with the RNN-T loss as
given in Equation (7).

###  4.3 Two-pass model

The two-pass model contains an additional LAS rescorer and is trained in three
stages as discussed in Section 2.2. The first stage model is trained with
architecture details similar to Section 4.2. In the second stage, the shared
encoder, i.e., the transcription network of the RNN-T model is frozen and is
used to train the additional encoder and LAS rescorer as mentioned in Section
3.2. The additional encoder layer consists of one unidirectional LSTM layer.
The LAS rescorer consists of an embedding layer and one unidirectional LSTM
layer. It uses a 4-headed dot attention layer [35]. The learning rate strategy
used to train the second stage is similar to the RNN-T model.

Table 1: Notations used for each of the models are mentioned. The number of
parameters in each of the modules for different models and the percentage
reduction in the number of parameters with respect to the Teacher model are
also mentioned. The subscript $i$ denotes the stage number in knowledge
distillation training.

| Model
---
Description
Model-ID |  | # params in first pass
---
(RNN-T) module
| # params in second pass
---
(additional) module
Tot. # params |  | Reduced size
---
w.r.t. $T_{i}$ (%)
Large size model (also used as Teacher) | $T_{1}$ | 72M | – | 72M | –
$T_{2}$, $T_{3}$ | 72M | 38M | 110M | –
Small size Model | $M_{1}$ | 32M | – | 32M | 55%
$M_{2}$, $M_{3}$ | 32M | 18M | 50M | 55%
Student Model | $S_{1}$ | 32M | – | 32M | 55%
$S_{2}$, $S_{3}$ | 32M | 38M | 70M | 36%

###  4.4 Notations used

We denote the large model, which is also used as the teacher model as $T$. We
also train a smaller size model compared to $T$, which is represented by $M$.
Note that both $T$ and $M$ models are trained independently without using
knowledge distillation, as explained in Section 2. The student model
represented by $S$ is trained with the proposed distillation approach as
explained in Section 3. We compare the performances of the teacher model ($T$)
and the smaller model ($M$), with that of the student model ($S$).

Table 1 shows the notations used to denote different models along with the
corresponding model sizes in terms of the number of parameters. The training
stage number for different models are mentioned as a subscript. For the
Stage-1 training, the RNN-T teacher model is denoted by $T_{1}$, the smaller
RNN-T model is denoted by $M_{1}$, and the student model which is trained from
$T_{1}$ with the loss given in Equation (7) is denoted by $S_{1}$. For Stage-2
of the proposed approach, the teacher model $T_{2}$ is trained using a pre-
trained model $T_{1}$ and the smaller model $M_{2}$ is trained using a pre-
trained model $M_{1}$. Similarly, the student model in Stage-2 trained with
the loss given in Equation (9) is denoted by $S_{2}$. In Stage-3 training of
the proposed approach, the two-pass models minimizes combined loss given in
Equation (10) are denoted by $T_{3}$, $M_{3}$, and $S_{3}$.

Table 2: WERs(%)/ SERs(%) on LibriSpeech for teacher model ($T_{i}$), smaller
model ($M_{i}$), and student model ($S_{i}$).

Model-ID | test-clean | test-other
---|---|---
WER | SER | WER | SER
$T_{1}$ | 6.81 | 59.69 | 18.23 | 83.09
$M_{1}$ | 7.52 | 61.53 | 20.03 | 84.79
$S_{1}$ | 6.92 | 59.08 | 18.83 | 82.58
$T_{2}$ | 5.80 | 53.63 | 16.79 | 79.62
$M_{2}$ | 6.60 | 57.44 | 18.72 | 83.57
$S_{2}$ | 6.24 | 55.95 | 17.77 | 81.35
$T_{3}$ | 5.87 | 53.55 | 16.92 | 80.16
$M_{3}$ | 6.59 | 57.02 | 18.47 | 82.51
$S_{3}$ | 5.81 | 52.67 | 17.27 | 79.72

##  5 Results

In this section, we demonstrate the effectiveness of the proposed compression
approach on the LibriSpeech dataset.

###  5.1 Performance of proposed approach

Table 2 reports the performance of different models in terms of Word Error
Rate (WER) and Sentence Error Rate (SER) for the test-clean and test-other
parts of the LibriSpeech dataset, respectively. The WER/SER mentioned are
slightly higher compared to other offline systems or those using external
language model. We not only report the final performance after completing the
three KD stages but also report the performances in the intermediate stages,
i.e., after Stage-1 and Stage-2 KD training. This gives a detailed overview of
the changes in the performances.

It can be seen from the Table 2 that the smaller models $M_{1}$, $M_{2}$, and
$M_{3}$ shows significantly degraded performance compared to models $T_{1}$,
$T_{2}$, and $T_{3}$ respectively. This clearly shows that directly reducing
the number of parameters may have an adverse effect on WER performance.

It can be seen from Table 2 that the $S_{1}$ RNN-T model achieves around 8%
relative improvement in WER over same-sized model $M_{1}$, by using
distillation loss with teacher model $T_{1}$ on the test-clean part. While
going from $T_{1}$ to $M_{1}$, a relative degradation of 10% in WER with
respect to the model $T_{1}$ is observed due to 55% reduced parameters. Since
this gap is high, the model $M_{1}$ is not usable. After applying knowledge
distillation with teacher-student training (using Teacher model $T_{1}$), our
student model $S_{1}$ performs comparable to the teacher model $T_{1}$ with
minor degradation in WER while maintaining a high compression rate of 55%
compared to $T_{1}$. A similar trend can be observed for the test-other part.

In the second stage, it can be seen that directly reducing the number of
parameters ($M_{2}$) results in degradation in performance for test-clean by
13.7% (relative) while with the proposed KD approach with model $S_{2}$ this
degradation is reduced to 7.5% relative while maintaining the same compression
rate. This shows that KD training is also helpful even in the case of an
attention-based decoder where the student model has the flexibility to attend
to different parts. A similar trend can also be observed for the test-other
part. This further confirms that the knowledge distillation indeed helps to
transfer the teacher’s knowledge.

In the third stage, we combine different modules to form a two-pass model. The
two-pass models have degradation of 12.26% in WER (relative) when model size
is directly reduced to 55% from $T_{3}$ to $M_{3}$. With deep-finetuning, the
student model $S_{3}$ shows performance even better than the Teacher model
($T_{3}$) while having a model size that is 36% less than the Teacher model
($T_{3}$). This shows that the proposed KD training approach is indeed helpful
in compressing the two-pass model without loss in WER performance.

Furthermore, we also conduct an experiment where we first trained the two-pass
teacher model completely as given in Section 2.2. The modules from the fully
trained two-pass teacher model (i.e., $T_{3}$) are taken for performing
knowledge distillation training in different stages in the proposed approach.
We observe a similar trend in the performance.

Table 3: WERs(%) / SERs(%) for student RNN-T model $S_{1}$ with different
values of $\beta$.

$\beta$ value | test-clean | test-other
---|---|---
WER | SER | WER | SER
$0.0$ | 7.52 | 61.53 | 20.03 | 84.79
$1\times 10^{-4}$ | 7.47 | 61.41 | 20.17 | 84.28
$1\times 10^{-3}$ | 7.25 | 60.42 | 19.30 | 84.14
$1\times 10^{-2}$ | 6.92 | 59.08 | 18.83 | 82.58
$1\times 10^{-1}$ | 7.07 | 60.38 | 18.87 | 83.29
$1.0$ | 10.02 | 71.56 | 21.92 | 87.96

Table 4: WERs(%)/ SERs(%) for the case when distillation is omitted from the
Stage-2 in the proposed approach. The KD is performed in the Stage-2 for the
models $S_{2}$ and $S_{3}$, whereas it not performed for the models denoted by
$S_{2}^{{}^{\prime}}$ and $S_{3}^{{}^{\prime}}$.

Model-ID | test-clean | test-other
---|---|---
WER | SER | WER | SER
$T_{2}$ | 5.80 | 53.63 | 16.79 | 79.62
${S}_{2}^{{}^{\prime}}$ | 6.19 | 54.81 | 17.82 | 81.18
$S_{2}$ | 6.24 | 55.95 | 17.77 | 81.35
$T_{3}$ | 5.87 | 53.55 | 16.92 | 80.16
$S_{3}^{{}^{\prime}}$ | 5.92 | 52.82 | 17.52 | 80.81
$S_{3}$ | 5.81 | 52.67 | 17.27 | 79.72

###  5.2 Tuning hyperparameter $\beta$ in Stage-1

As discussed in Section 2.2, $\beta$ is a hyperparameter used for distillation
loss in Stage-1 training of the proposed approach. We found that this
hyperparameter needs to be appropriately tuned to achieve good performance.
Hence, we analyzed the performance of model $S_{1}$ under different values of
$\beta$. The performance on model $S_{1}$ when trained with different values
of $\beta$ is shown in Table 3. It can be seen that extreme values of $\beta$
result in performance degradation. Optimal performance is seen at $1\times
10^{-2}$, which is also used in this work.

###  5.3 Ablation study on knowledge distillation in Stage-2

We conducted an experiment to study the effect of knowledge distillation in
the Stage-2 of the proposed approach. In this experiment we consider two
different variants of student model as follows: (i) Distillation is performed
in Stage-2 (denoted by $S_{2}$ and $S_{3}$), similar to Section 5.1, and (ii)
No distillation is performed in Stage-2 (denoted by $S_{2}^{{}^{\prime}}$ and
$S_{3}^{{}^{\prime}}$).

The results of this experiments are shown in Table 4. In Stage-2, the models
with and without distillation performed comparably. In Stage-3, it can be
observed that the student model that has trained using knowledge distillation
in the second stage (i.e., $S_{3}$) shows consistent improvement. This
improvement is only slightly better than the model when distillation is not
performed (i.e., $S_{3}^{{}^{\prime}}$) in the second stage training. This
could be attributed to the fact that the second-pass mainly focuses on re-
scoring the predicted first pass hypotheses instead of re-decoding them.
Hence, its performance is limited by the first-pass decoder.

Table 5: WERs(%)/ SERs(%) for teacher and smaller student models (denoted by
$SS_{2}$ and $SS_{3}$) in Stage-2 and Stage-3.

Model-ID | test-clean | test-other
---|---|---
WER | SER | WER | SER
$T_{2}$ | 5.80 | 53.63 | 16.79 | 79.62
$SS_{2}$ | 6.25 | 55.99 | 17.99 | 81.69
$S_{2}$ | 6.24 | 55.95 | 17.77 | 81.35
$T_{3}$ | 5.87 | 53.55 | 16.92 | 80.16
$SS_{3}$ | 6.05 | 54.50 | 17.93 | 81.83
$S_{3}$ | 5.81 | 52.67 | 17.27 | 79.72

###  5.4 Reducing parameters for two-pass student model

Since our student model $S_{3}$ performs slightly better than teacher model
$T_{3}$, we also conduct an experiment where we reduce the model size of
$S_{3}$ further from 36% to 55% compared to the $T_{3}$ model. This is done by
reducing the size of the LAS module of the student model in Stage-2 and
Stage-3. These student model with LAS size smaller than $T_{2}$ and $T_{3}$
are denoted by $SS_{2}$ and $SS_{3}$ respectively.

The results obtained with this experiment are given in Table 5. It can be
observed that the smaller student model $SS_{3}$ performs comparable to model
$S_{3}$ and shows degradation of only 3% (relative) while achieving a
significant compression rate of 55% compared to the teacher model $T_{3}$.
This is a reasonable compromise in the performance with an impressive gain
achieved in the compression rate of 55%, which is good enough for medium to
high-end mobile devices.

##  6 Conclusion

In this paper, we propose a model compression technique for the two-pass
streaming ASR model using knowledge distillation. The proposed approach
reduces the two-pass model size by 36% while slightly improving the WER as
compared to the teacher model. Further, with the proposed approach, our
smaller student model is able to achieve a high compression rate of 55%
compared to the two-pass teacher model while performing comparable to the
teacher model in terms of WER. Needless to mention, the two-pass student model
with the proposed compression approach has shown significantly better
performance than that of the smaller two-pass model trained independently. We
also perform detailed analyses of the proposed approach in different stages.
In future, we plan to extend our approach and perform a comparative study with
the low-rank approximation methods.

## References

  * [1] Yanzhang He, Tara N Sainath, Rohit Prabhavalkar, Ian McGraw, Raziel Alvarez, Ding Zhao, David Rybach, Anjuli Kannan, Yonghui Wu, Ruoming Pang, et al.,  “Streaming end-to-end speech recognition for mobile devices,”  in IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2019, pp. 6381–6385.
  * [2] Tara N Sainath, Ruoming Pang, David Rybach, Yanzhang He, Rohit Prabhavalkar, Wei Li, Mirkó Visontai, Qiao Liang, Trevor Strohman, Yonghui Wu, et al.,  “Two-pass end-to-end speech recognition,”  arXiv preprint arXiv:1908.10992, 2019.
  * [3] Ganesh Venkatesh, Alagappan Valliappan, Jay Mahadeokar, Yuan Shangguan, Christian Fuegen, Michael L. Seltzer, and Vikas Chandra,  “Memory-efficient speech recognition on smart devices,”  in IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2021, pp. 8368–8372.
  * [4] Kwangyoun Kim, Kyungmin Lee, Dhananjaya Gowda, Junmo Park, Sungsoo Kim, Sichen Jin, Young-Yoon Lee, Jinsu Yeo, Daehyun Kim, Seokyeong Jung, et al.,  “Attention based on-device streaming speech recognition with large speech corpus,”  in Proc. of ASRU, 2019, pp. 956–963.
  * [5] Abhinav Garg, Dhananjaya Gowda, Ankur Kumar, Kwangyoun Kim, Mehul Kumar, and Chanwoo Kim,  “Improved multi-stage training of online attention-based encoder-decoder models,”  in Proc. of ASRU, 2019, pp. 70–77.
  * [6] Chanwoo Kim, Sungsoo Kim, Kwangyoun Kim, Mehul Kumar, Jiyeon Kim, Kyungmin Lee, Changwoo Han, Abhinav Garg, Eunhyang Kim, Minkyoo Shin, Shatrughan Singh, Larry Heck, and Dhananjaya Gowda,  “End-to-end training of a large vocabulary end-to-end speech recognition system,”  in  Proc. of ASRU , 2019, pp. 562–569.
  * [7] Abhinav Garg, Gowtham Vadisetti, Dhananjaya Gowda, Sichen Jin, Aditya Jayasimha, Youngho Han, Jiyeon Kim, Junmo Park, Kwangyoun Kim, Sooyeon Kim, Youngyoon Lee, Kyungbo Min, and Chanwoo Kim,  “Streaming on-device end-to-end ASR system for privacy-sensitive voicetyping,”  in Proc. of Interspeech, 2020, pp. 3371–3375.
  * [8] Chanwoo Kim, Dhananjaya Gowda, Dongsoo Lee, Jiyeon Kim, Ankur Kumar, Sungsoo Kim, Abhinav Garg, and Changwoo Han,  “A review of on-device fully neural end-to-end automatic speech recognition algorithms,”  arXiv preprint arXiv:2012.07974, 2020.
  * [9] Alex Graves, Santiago Fernández, Faustino Gomez, and Jürgen Schmidhuber,  “Connectionist temporal classification: labelling unsegmented sequence data with recurrent neural networks,”  in Proceedings of the 23rd International Conference on Machine Learning, 2006, pp. 369–376.
  * [10] William Chan, Navdeep Jaitly, Quoc V Le, and Oriol Vinyals,  “Listen, attend and spell,”  arXiv preprint arXiv:1508.01211, 2015.
  * [11] Jianping Gou, Baosheng Yu, Stephen J. Maybank, and Dacheng Tao,  “Knowledge distillation: A survey,”  International Journal of Computer Vision, vol. 129, no. 6, pp. 1789–1819, Mar 2021.
  * [12] Yevgen Chebotar and Austin Waters,  “Distilling knowledge from ensembles of neural networks for speech recognition.,”  in Proc. of Interspeech, 2016, pp. 3439–3443.
  * [13] Geoffrey Hinton, Oriol Vinyals, and Jeff Dean,  “Distilling the knowledge in a neural network,”  arXiv preprint arXiv:1503.02531, 2015.
  * [14] Takashi Fukuda, Masayuki Suzuki, Gakuto Kurata, Samuel Thomas, Jia Cui, and Bhuvana Ramabhadran,  “Efficient knowledge distillation from an ensemble of teachers.,”  in Proc. of Interspeech, 2017, pp. 3697–3701.
  * [15] Saad Albawi, Tareq Abed Mohammed, and Saad Al-Zawi,  “Understanding of a convolutional neural network,”  in International Conference on Engineering and Technology (ICET). IEEE, 2017, pp. 1–6.
  * [16] Karen Simonyan and Andrew Zisserman,  “Very deep convolutional networks for large-scale image recognition,”  arXiv preprint arXiv:1409.1556, 2014.
  * [17] Hasim Sak, Andrew W Senior, and Françoise Beaufays,  “Long short-term memory recurrent neural network architectures for large scale acoustic modeling,”  arXiv preprint arXiv:1402.1128, 2014.
  * [18] Alex Sherstinsky,  “Fundamentals of recurrent neural network (RNN) and long short-term memory (LSTM) network,”  Physica D: Nonlinear Phenomena, vol. 404, pp. 132306, 2020.
  * [19] Yoon Kim and Alexander M Rush,  “Sequence-level knowledge distillation,”  arXiv preprint arXiv:1606.07947, 2016.
  * [20] Jeremy HM Wong and Mark Gales,  “Sequence student-teacher training of deep neural networks,”  in Proc. of Interspeech, 2016, pp. 2761–2765.
  * [21] Hirofumi Inaguma, Tatsuya Kawahara, and Shinji Watanabe,  “Source and target bidirectional knowledge distillation for end-to-end speech translation,”  arXiv preprint arXiv:2104.06457, 2021.
  * [22] Suyoun Kim, Michael L Seltzer, Jinyu Li, and Rui Zhao,  “Improved training for online end-to-end speech recognition systems,”  arXiv preprint arXiv:1711.02212, 2017.
  * [23] Ryoichi Takashima, Sheng Li, and Hisashi Kawai,  “An investigation of a knowledge distillation method for ctc acoustic models,”  in IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2018, pp. 5809–5813.
  * [24] Gakuto Kurata and George Saon,  “Knowledge distillation from offline to streaming RNN-Transducer for end-to-end speech recognition,”  Proc. of Interspeech, pp. 2117–2121, 2020.
  * [25] Sankaran Panchapagesan, Daniel S Park, Chung-Cheng Chiu, Yuan Shangguan, Qiao Liang, and Alexander Gruenstein,  “Efficient knowledge distillation for rnn-transducer models,”  arXiv preprint arXiv:2011.06110, 2020.
  * [26] William Chan, Navdeep Jaitly, Quoc Le, and Oriol Vinyals,  “Listen, attend and spell: A neural network for large vocabulary conversational speech recognition,”  in IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2016, pp. 4960–4964.
  * [27] Alex Graves,  “Sequence transduction with recurrent neural networks,”  arXiv preprint arXiv:1211.3711, 2012.
  * [28] Vassil Panayotov, Guoguo Chen, Daniel Povey, and Sanjeev Khudanpur,  “Librispeech: an asr corpus based on public domain audio books,”  in IEEE International Conference on Acoustics, Speech and Signal processing (ICASSP). IEEE, 2015, pp. 5206–5210.
  * [29] Rico Sennrich, Barry Haddow, and Alexandra Birch,  “Neural machine translation of rare words with subword units,”  arXiv preprint arXiv:1508.07909, 2015.
  * [30] Zhang Wanli and Li Guoxin,  “The research of feature extraction based on mfcc for speaker recognition,”  in Proceedings International Conference on Computer Science and Network Technology. IEEE, 2013, pp. 1074–1077.
  * [31] Daniel S Park, William Chan, Yu Zhang, Chung-Cheng Chiu, Barret Zoph, Ekin D Cubuk, and Quoc V Le,  “Specaugment: A simple data augmentation method for automatic speech recognition,”  arXiv preprint arXiv:1904.08779, 2019.
  * [32] Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov,  “Dropout: a simple way to prevent neural networks from overfitting,”  The Journal of Machine Learning Research, vol. 15, no. 1, pp. 1929–1958, 2014.
  * [33] Albert Zeyer, Kazuki Irie, Ralf Schlüter, and Hermann Ney,  “Improved training of end-to-end attention models for speech recognition,”  arXiv preprint arXiv:1805.03294, 2018.
  * [34] Diederik P Kingma and Jimmy Ba,  “Adam: A method for stochastic optimization,”  arXiv preprint arXiv:1412.6980, 2014.
  * [35] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin,  “Attention is all you need,”  arXiv preprint arXiv:1706.03762, 2017.