[table]capposition=top

# Pseudo-Boolean Functions for Optimal Z-Complementary Code Sets with Flexible
Lengths

Palash Sarkar, Sudhan Majhi, and Zilong Liu Palash Sarkar and Sudhan Majhi are
with the Department of EE, IIT Patna, India, e-mail: {
<EMAIL_ADDRESS>Liu is with the School of CSEE,
University of Essex, UK<EMAIL_ADDRESS>
###### Abstract

This paper aims to construct optimal Z-complementary code set (ZCCS) with non-
power-of-two (NPT) lengths to enable interference-free multicarrier code-
division multiple access (MC-CDMA) systems. The existing ZCCSs with NPT
lengths, which are constructed from generalized Boolean functions (GBFs), are
sub-optimal only with respect to the set size upper bound. For the first time
in the literature, we advocate the use of pseudo-Boolean functions (PBFs)
(each of which transforms a number of binary variables to a real number as a
natural generalization of GBF) for direct constructions of optimal ZCCSs with
NPT lengths.

###### Index Terms:

Multicarrier code-division multiple access (MC-CDMA), generalized Boolean
function (GBF), pseudo-Boolean function (PBF), Z-complementary code set
(ZCCS), zero correlation zone (ZCZ)

##  I Introduction

Multicarrier code-division multiple access (MC-CDMA) has been one of the most
widely adopted wireless techniques in many communication systems/standards
owing to its efficient fast Fourier transform (FFT) based implementation,
resilience against intersymbol interference, and high spectral efficiency [1].
That being said, MC-CDMA may suffer from multiple-access interference (MAI)
[2] and multipath interference (MPI) [3]. A promising way to address both MAI
and MPI is to adopt proper spreading codes, such as complete complementary
codes (CCC) [4] and Z-complementary code sets (ZCCSs) [5]. This paper focuses
on efficient construction of ZCCSs with a new tool, called pseudo-Boolean
functions (PBFs), to enable interference-free quasi-synchronous MC-CDMA
systems.

In 2007, Z-complementary pairs (ZCPs) were introduced by Fan _et al._ [6] to
overcome the limitation on the lengths of Golay complementary pairs (GCPs) [7,
8]. A ZCP refers to a pair of sequences of same length $N$ having zero
aperiodic auto-correlation sums for all time shifts $\tau$ satisfying
$0<|\tau|<Z$, where $Z$ is called zero-correlation zone (ZCZ) width. When
$Z=N$, the resultant sequence pair reduces to a GCP. In the literature, there
are direct constructions of GCPs and ZCPs with the aid of generalized Boolean
functions (GBFs) [9, 10, 11]. The idea of ZCPs introduced in [6] was
generalized to ZCCS by Feng _et al._ in [12]. A ZCCS refers to a set of $K$
codes, each of which consists of $M$ constituent sequences of identical length
$L$, having ideal aperiodic auto- and cross-correlation properties inside the
ZCZ width and satisfying the theoretical upper bound: $K\leq M\lfloor
N/Z\rfloor$ [13]. When $Z=N$, the set is called a mutually orthogonal Golay
complementary sets (MOGCSs) [4], which refers to collection of complementary
codes (CCs) [14, 15, 16] with ideal cross-correlation properties. A set of
CCCs is known as a MOGCSs with the equality $K=M$ [17, 18, 19, 20]. The
theoretical upper bound shows that an optimal ZCCS has larger set size as
compared to CCCs provided $\lfloor\frac{N}{Z}\rfloor\geq 2$. Recently, several
GBFs based constructions of optimal ZCCSs with power-of-two lengths have been
reported in [5, 21, 22, 23]. In the recent literature, two direct
constructions of ZCCSs with NPT lengths can be found in [24] and [22], which
produces sub-optimal ZCCS with $\lfloor\frac{N}{Z}\rfloor=1$ and non-optimal
ZCCSs for NPT lengths with $\lfloor\frac{N}{Z}\rfloor<1$, respectively. To the
best of our knowledge, the construction of optimal ZCCSs of NPT lengths with
$\lfloor\frac{N}{Z}\rfloor\geq 2$, based on GBFs remains open. Other methods
which are dependent on the existence of special sequences, known as indirect
constructions [11], to construct ZCCSs can be found in [25, 26, 27]. The
indirect constructions heavily rely on a series of sequence operations which
may not be feasible for rapid hardware generation, especially, when the
sequence lengths are large [5].

TABLE I: Comparison of the Proposed Construction with [5, 23, 24, 21, 22, 27]

ZCCS | Method | Length ($N$) | $\lfloor\frac{N}{Z}\rfloor$ | Constraints | Optimality
---|---|---|---|---|---
[5] | Direct | $2^{m}$ | $\geq 2$ | $m\geq 2$ | Optimal
[23] | Direct | $2^{m}$ | $\geq 2$ | $m\geq 2$ | Optimal
[24] | Direct | $2^{m}+2$ | $=1$ | $m\geq 4$ | Sub-optimal
[21] | Direct | $2^{m}$ | $\geq 2$ | $m\geq 2$ | Optimal
[22] | Direct | $2^{m}$ | $\geq 2$ | $m\geq 2$ | Optimal
[22] | Direct | $2^{m}+2^{h}$ | $\geq 1$ | $m>0,0<h\leq m$ | Non-optimal
[27] | Indirect | $L$ | $\geq 2$ | $L\geq 1$ | Optimal
Theorem 1 | Direct | $p2^{m}$ | $\geq 2$ |  $m\geq 2$, prime $p$ | Optimal

It is noted that the MAI in MC-CDMA system can be mitigated using zero-
correlation properties of a ZCCS provided that all the received multiuser
signals are roughly synchronous within the ZCZ width [19]. In addition to
their applications in MC-CDMA [18, 19, 27], ZCCSs have also been employed as
optimal training sequences in multiple-input multiple-output (MIMO)
communications [28, 29]. The limitation on the set size of CCCs and the
unavailability of optimal ZCCSs with NPT lengths using direct constructions in
the existing literature are a major motivation of this work. Specifically, for
the first time in the literature, we propose to use PBFs for direct
construction of optimal ZCCS of lengths $p2^{m}$, where $p$ is a prime number
and $m$ is a positive integer. A PBF [30] refers to an arbitrary mapping of
the set of binary $m$-tuples to real numbers. Being a natural generalization
of GBFs, PBFs are also suitable for rapid hardware generation of sequences. A
detailed comparison of the proposed construction with [5, 23, 24, 21, 22, 27]
is given in TABLE I.

##  II Preliminary

In this section, we present some basic definitions and lemmas to be used in
the proposed construction. Let
$\textbf{y}_{1}=(y_{1,0},y_{1,1},\cdots,y_{1,N-1})$ and
$\textbf{y}_{2}=(y_{2,0},y_{2,1},\cdots,y_{2,N-1})$ denote a pair of sequences
with complex components. For an integer $\tau$, define [5]

$\theta(\textbf{y}_{1},\textbf{y}_{2})(\tau)=\begin{cases}\sum_{i=0}^{N-1-\tau}y_{1,i+\tau}y^{*}_{2,i},&0\leq\tau<N,\\\
\sum_{i=0}^{N+\tau-1}y_{1,i}y^{*}_{2,i-\tau},&-N<\tau<0,\\\
0,&\text{otherwise},\end{cases}$ (1)

The functions $\theta(\textbf{y}_{1},\textbf{y}_{2})$ and
$\theta(\textbf{y}_{1},\textbf{y}_{1})$ are called the aperiodic cross-
correlation function (ACCF) between $\textbf{y}_{1}$ and $\textbf{y}_{2}$, and
the aperiodic auto-correlation function (AACF) of $\textbf{y}_{1}$,
respectively. Let
$\mathcal{S}=\\{\mathcal{S}_{0},\mathcal{S}_{1},\cdots,\mathcal{S}_{K-1}\\}$
be a set of $K$ codes or ordered sets defined as

$\begin{split}\mathcal{S}_{\mu}=\left(\textbf{s}_{0}^{\mu},\textbf{s}_{1}^{\mu},\ldots,\textbf{s}_{M-1}^{\mu}\right),\end{split}$
(2)

where $\textbf{s}_{\nu}^{\mu}$ $(0\leq\nu\leq M-1,0\leq\mu\leq K-1)$ is the
$\nu$-th element which we assume is a complex-valued sequence of length $N$ in
$\mathcal{S}_{\mu}$. For $\mathcal{S}_{\mu_{1}}$,
$\mathcal{S}_{\mu_{2}}\in\mathcal{S}$ $(0\leq\mu_{1},\mu_{2}\leq K-1)$, the
ACCF between $\mathcal{S}_{\mu_{1}}$ and $\mathcal{S}_{\mu_{2}}$ is defined as

$\theta(\mathcal{S}_{\mu_{1}},\mathcal{S}_{\mu_{2}})(\tau)=\displaystyle\sum_{\nu=0}^{M-1}\theta(\textbf{s}_{\nu}^{\mu_{1}},\textbf{s}_{\nu}^{\mu_{2}})(\tau).$
(3)

######  Definition 1 ([5])

Code set $\mathcal{S}$ is called a ZCCS if

$\displaystyle\theta(\mathcal{S}_{\mu_{1}},\mathcal{S}_{\mu_{2}})(\tau)=\begin{cases}MN,&\tau=0,\mu_{1}=\mu_{2},\\\
0,&0<|\tau|<Z,\mu_{1}=\mu_{2},\\\ 0,&|\tau|<Z,\mu_{1}\neq\mu_{2},\end{cases}$
(4)

where $Z$ is called ZCZ width. We denote a ZCCS with the parameters $K,N,M$,
and $Z$ by the notation $(K,Z)$-$\text{ZCCS}_{M}^{N}$. For $K=M$ and $Z=N$, a
$(K,Z)$-$\text{ZCCS}_{M}^{N}$ is called a set of CCCs and we denote it by
$(K,K,N)$-CCC.

We call a $(K,Z)$-$\text{ZCCS}_{M}^{N}$ optimal if it achieves the equality in
the theoretical upper-bound, given by $K\leq
M\left\lfloor\frac{N}{Z}\right\rfloor$ [13].

###  II-A Generalized Boolean Functions (GBFs)

Let $x_{0},x_{1},\ldots,x_{m-1}$ denote $m$ variables which take values from
$\mathbb{Z}_{2}$. A monomial of degree $i$ ($0\leq i\leq m$) is defined as the
product of any $i$ distinct variables among $x_{0},x_{1},\ldots,x_{m-1}$. Let
us assume that $\mathcal{A}_{i}$ denotes the set of all monomials of degree
$i$, where

$\begin{split}\mathcal{A}_{i}=&\left\\{x_{0}^{r_{0}}x_{1}^{r_{1}}\cdots
x_{m-1}^{r_{m-1}}:\right.\\\
&\left.r_{0}+r_{1}+\cdots+r_{m-1}=i,(r_{0},r_{1},\ldots,r_{m-1})\in\mathbb{Z}_{2}^{m}\right\\}.\end{split}$
(5)

A function $f:\mathbb{Z}_{2}^{m}\rightarrow\mathbb{Z}_{q}$ is said to be a GBF
if it can uniquely be expressed as a linear combination of the monomials in
$\mathcal{A}_{m}$, where the coefficient of each monomial is drawn from
$\mathbb{Z}_{q}$, where $\mathbb{Z}_{q}$ denotes the set of integers modulo
$q$. The highest degree monomial with non-zero coefficient present in the
expression of $f$ determine the order of $f$. As an example,
$2x_{0}x_{1}+x_{1}+1$ is a second order GBF of two variables $x_{0}$ and
$x_{1}$ over $\mathbb{Z}_{3}$. We denote the graph of a second-order GBF $f$
by $G(f)$ [14]. It contains $m$ vertices which are denoted by the $m$
variables of $f$. The edges in the $G(f)$ are determined by the second-degree
monomials present in the expression of $f$ with non-zero coefficients. There
is an edge of weight $w$ between the vertices $x_{\alpha}$ and $x_{\beta}$ of
$G(f)$ if the expression of $f$ contains the term $wx_{\alpha}x_{\beta}$. Let
$\psi(f)$ denotes the complex-valued sequence corresponding to a GBF $f$ and
it is defined as [14],
$\psi(f)=(\omega_{q}^{f_{0}},\omega_{q}^{f_{1}},\ldots,\omega_{q}^{f_{2^{m}-1}})$,
where $\omega_{q}$ denotes $\exp\left(2\pi\sqrt{-1}/q\right)$,
$f_{r}=f(r_{0},r_{1},\ldots,r_{m-1})$, $(r_{0},r_{1},\ldots,r_{m-1})$ is the
binary vector representation of integer $r$
($r=\displaystyle\sum_{\alpha=0}^{m-1}r_{\alpha}2^{\alpha}$), and $q$ denotes
an even number, no less than 2. We denote by $\bar{x}=1-x$ the binary
complement of $x\in\\{0,1\\}$. For any given GBF $f$ in $m$ variables, we
denote the function $f(1-x_{0},1-x_{1},\ldots,1-x_{m-1})$ or
$f(\bar{x}_{0},\bar{x}_{1},\ldots,\bar{x}_{m-1})$ by $\tilde{f}$. Let
$\mathcal{C}=\left(g_{1},g_{2},\ldots,g_{M}\right)$ be an ordered set of $M$
GBFs. We define the code $\psi(\mathcal{C})$ corresponding to $\mathcal{C}$ as
$\psi(\mathcal{C})=\left(\psi(g_{1}),\psi(g_{2}),\ldots,\psi(g_{M})\right)$.

###### Lemma 1

(Construction of CCC [4])
Let $f:\mathbb{Z}_{2}^{m}\rightarrow\mathbb{Z}_{q}$ be a second-order GBF. Let
us assume that $G(f)$ contains the vertices
$x_{j_{0}},x_{j_{1}},\ldots,x_{j_{k-1}}$ such a way that after performing a
deletion operation on those vertices, the resulting graph reduces to a path.
Let the edges in the path have identical weight of $\frac{q}{2}$ and
$\mathbf{t}=(t_{0},t_{1},\cdots,t_{k-1})$ be the binary representation of the
integer $t$. Define the CC, $C_{t}$ to be

$\Big{\\{}f\\!+\\!\frac{q}{2}\Big{(}(\mathbf{d}+\mathbf{t})\cdot\mathbf{x}\\!+\\!dx_{\gamma}\Big{)}:\mathbf{d}\in\\{0,1\\}^{k},d\in\\{0,1\\}\Big{\\}},$
(6)

and $\bar{C}_{t}$ to be

$\Big{\\{}\tilde{f}\\!+\\!\frac{q}{2}\Big{(}(\mathbf{d}+\mathbf{t})\cdot\bar{\mathbf{x}}\\!+\\!\bar{d}x_{\gamma}\Big{)}:\mathbf{d}\in\\{0,1\\}^{k},d\in\\{0,1\\}\Big{\\}},$
(7)

where $(\cdot)\cdot(\cdot)$ denotes the dot product between two real-valued
vector $(\cdot)$ and $(\cdot)$, $\gamma$ is the label of either end vertex in
the path, $\mathbf{x}=(x_{j_{0}},x_{j_{1}},\ldots,x_{j_{k-1}})$,
$\bar{\mathbf{x}}=(1-x_{j_{0}},1-x_{j_{1}},\ldots,1-x_{j_{k-1}})$, and
$\mathbf{d}=(d_{0},d_{1},\ldots,d_{k-1})$. Then
$\\{\psi(C_{t}),\psi^{*}(\bar{C}_{t}):0\leq t<2^{k}\\}$ forms
$(2^{k+1},2^{k+1},2^{m})$-CCC, where $\psi^{*}(\cdot)$ denotes the complex
conjugate of $\psi(\cdot)$.

###  II-B Pseudo-Boolean Functions (PBFs)

A function $F:\\{0,1\\}^{m}\rightarrow\mathbb{R}$ is said to be a PBF if it
can be uniquely expressed as a linear combination of monomials in
$\mathcal{A}_{m}$ with the coefficients drawn from $\mathbb{R}$, where
$\mathbb{R}$ denotes the set of real numbers. Therefore, PBFs are a natural
generalization of GBFs [30]. As an example, $\frac{2}{3}x_{0}x_{1}+x_{0}+1$ is
a second-order PBF of two variables $x_{0}$ and $x_{1}$ but not a GBF. Let
$f:\mathbb{Z}_{2}^{m}\rightarrow\mathbb{Z}_{q}$ be a GBF of the variables
$x_{0},x_{1},\ldots,x_{m-1}$. Let us assume that $p$ denotes a prime number
and define the following PBFs with the help of the GBF $f$:

$\begin{split}F^{\lambda}&=f+\frac{\lambda
q}{p}(x_{m}+2x_{m+1}+\cdots+2^{s-1}x_{m+s-1}),\\\
G^{\lambda}&=\tilde{f}+\frac{\lambda
q}{p}(x_{m}+2x_{m+1}+\cdots+2^{s-1}x_{m+s-1}),\end{split}$ (8)

where $s\in\mathbb{Z}^{+}$ which denotes the set of all positive integers,
$2\leq p<2^{s+1}$, and $\lambda=0,1,\ldots,p-1$. From (8), it is clear that
$F^{\lambda}$ and $G^{\lambda}$ are PBFs of $m+s$ variables
$x_{0},x_{1},\ldots,x_{m+s-1}$. From (8), it can be observed that the PBFs
$F^{\lambda}$ and $G^{\lambda}$ reduce to $\mathbb{Z}_{q}$-valued GBFs if $p$
divides $q$.

##  III Proposed Construction of Z-Complementary Code Set

In this section, we shall present our proposed construction of ZCCS using
PBFs. To this end, we first present a lemma which will be used in our proposed
construction.

###### Lemma 2

([31]) Let $\lambda$ and $\lambda^{\prime}$ be two non-negative integers,
where $0\leq\lambda\neq\lambda^{\prime}<p$, $p$ is a prime number as defined
in Section-II. Then
$\displaystyle\sum_{\alpha=0}^{p-1}\omega_{p}^{(\lambda-\lambda^{\prime})\alpha}=0.$

For $0\leq t<2^{k}$ and $0\leq\lambda<p$, we define the following sets of
PBFs:

$\begin{split}U_{t}^{\lambda}\\!\\!=\\!\\!\Big{\\{}F^{\lambda}\\!\\!+\\!\frac{q}{2}\Big{(}(\mathbf{d}+\mathbf{t})\cdot\mathbf{x}\\!+\\!dx_{\gamma}\Big{)}:\mathbf{d}\in\\{0,1\\}^{k},d\in\\{0,1\\}\Big{\\}},\end{split}$
(9)

and

$V_{t}^{\lambda}\\!\\!=\\!\\!\Big{\\{}G^{\lambda}\\!+\\!\frac{q}{2}\Big{(}(\mathbf{d}+\mathbf{t})\cdot\bar{\mathbf{x}}\\!+\\!\bar{d}x_{\gamma}\Big{)}:\mathbf{d}\in\\{0,1\\}^{k},d\in\\{0,1\\}\Big{\\}}.$
(10)

Let us assume that
$f^{\mathbf{d},\mathbf{t},d}\\!\\!=\\!\\!f+\frac{q}{2}((\mathbf{d}+\mathbf{t})\cdot\mathbf{x}\\!+dx_{\gamma}),g^{\mathbf{d},\mathbf{t},d}\\!\\!=\\!\\!\tilde{f}+\frac{q}{2}((\mathbf{d}+\mathbf{t})\cdot\bar{\mathbf{x}}\\!+\bar{d}x_{\gamma})$,
in Lemma 1. We also assume
$F^{\mathbf{d},\mathbf{t},d,\lambda}=F^{\lambda}\\!\\!+\\!\frac{q}{2}((\mathbf{d}+\mathbf{t})\cdot\mathbf{x})\\!+\\!dx_{\gamma}$,
in (9), and
$G^{\mathbf{d},\mathbf{t},d,\lambda}=G^{\lambda}\\!+\\!\frac{q}{2}((\mathbf{d}+\mathbf{t})\cdot\bar{\mathbf{x}}\\!+\\!\bar{d}x_{\gamma})$,
in (10). As per our assumption, for any choice of
$\mathbf{d},\mathbf{t}\in\\{0,1\\}^{k}$, and $d\in\\{0,1\\}$, the functions
$f^{\mathbf{d},\mathbf{t},d}$ and $g^{\mathbf{d},\mathbf{t},d}$ are
$\mathbb{Z}_{q}$-valued GBFs of $m$ variables. For any choice of
$\mathbf{d},\mathbf{t}\in\\{0,1\\}^{k},d\in\\{0,1\\}$, and
$\lambda\in\\{0,1,\ldots,p-1\\}$, the functions
$F^{\mathbf{d},\mathbf{t},d,\lambda}$ and
$G^{\mathbf{d},\mathbf{t},d,\lambda}$ are PBFs of $m+s$ variables. We define
$\psi(F^{\mathbf{d},\mathbf{t},d,\lambda})$, the complex-valued sequence
corresponding to $F^{\mathbf{d},\mathbf{t},d,\lambda}$, as

$\psi(F^{\mathbf{d},\mathbf{t},d,\lambda})=(\omega^{F^{\mathbf{d},\mathbf{t},d,\lambda}_{0}}_{q},\omega^{F^{\mathbf{d},\mathbf{t},d,\lambda}_{1}}_{q},\ldots,\omega_{q}^{F^{\mathbf{d},\mathbf{t},d,\lambda}_{2^{m+s}-1}}),$
(11)

where
$F^{\mathbf{d},\mathbf{t},d,\lambda}_{r^{\prime}}=F^{\mathbf{d},\mathbf{t},d,\lambda}(r_{0},r_{1},\cdots,r_{m+s-1})$,
$r^{\prime}=\displaystyle\sum_{\alpha=0}^{m+s-1}r_{\alpha}2^{\alpha}$. The
$r^{\prime}$-th component of $\psi(F^{\mathbf{d},\mathbf{t},d,\lambda})$ is
given by

$\begin{split}\omega^{F^{\mathbf{d},\mathbf{t},d,\lambda}}_{q}&=\omega_{q}^{f^{\mathbf{d},\mathbf{t},d}(r_{0},r_{1},\ldots,r_{m-1})+\frac{q\lambda}{p}(r_{m}+2r_{m+1}+\cdots+2^{s-1}r_{m+s-1})}\\\
&=\omega_{q}^{f^{\mathbf{d},\mathbf{t},d}_{r}}\omega_{p}^{\lambda(r_{m}+2r_{m+1}+\cdots+2^{s-1}r_{m+s-1})}.\end{split}$
(12)

From (12), it can be observed that
$\omega^{F^{\mathbf{d},\mathbf{t},d,\lambda}_{r^{\prime}}}_{q}$ is a root of
the polynomial: $z^{\delta}-1$, where $\delta=lcm(p,q)$, denotes a positive
integer given by the least common multiple (lcm) of $p$ and $q$. Therefore,
the components of $\psi(F^{\mathbf{d},\mathbf{t},d,\lambda})$ are given by the
roots of the polynomial: $z^{\delta}-1$. From (11) and (12), we have

$\begin{split}&\psi(F^{\mathbf{d},\mathbf{t},d,\lambda})=(\underbrace{\omega_{q}^{f^{\mathbf{d},\mathbf{t},d}_{0}}\omega_{p}^{\lambda(0)},\omega_{q}^{f^{\mathbf{d},\mathbf{t},d}_{1}}\omega_{p}^{\lambda(0)},\ldots,\omega_{q}^{f^{\mathbf{d},\mathbf{t},d}_{2^{m}-1}}\omega_{p}^{\lambda(0)}},\\\
&\underbrace{\omega_{q}^{f^{\mathbf{d},\mathbf{t},d}_{0}}\omega_{p}^{\lambda(1)},\omega_{q}^{f^{\mathbf{d},\mathbf{t},d}_{1}}\omega_{p}^{\lambda(1)},\ldots,\omega_{q}^{f^{\mathbf{d},\mathbf{t},d}_{2^{m}-1}}\omega_{p}^{\lambda(1)}},\ldots\\\
&\underbrace{\omega_{q}^{f^{\mathbf{d},\mathbf{t},d}_{0}}\omega_{p}^{\lambda(2^{s}-1)},\omega_{q}^{f^{\mathbf{d},\mathbf{t},d}_{1}}\omega_{p}^{\lambda(2^{s}-1)},\ldots,\omega_{q}^{f^{\mathbf{d},\mathbf{t},d}_{2^{m}-1}}\omega_{p}^{\lambda(2^{s}-1)}}).\end{split}$
(13)

Let us also define
$\psi_{2^{m+s}-p2^{m}}(F^{\mathbf{d},\mathbf{t},d,\lambda})$ which is defined
to be obtained from $\psi(F^{\mathbf{d},\mathbf{t},d,\lambda})$ by removing
its last $2^{m+s}-p2^{m}$ components.

$\begin{split}&\psi_{2^{m+s}-p2^{m}}(F^{\mathbf{d},\mathbf{t},d,\lambda})\\\
=&(\underbrace{\omega_{q}^{f^{\mathbf{d},\mathbf{t},d}_{0}}\omega_{p}^{\lambda(0)},\omega_{q}^{f^{\mathbf{d},\mathbf{t},d}_{1}}\omega_{p}^{\lambda(0)},\ldots,\omega_{q}^{f^{\mathbf{d},\mathbf{t},d}_{2^{m}-1}}\omega_{p}^{\lambda(0)}},\\\
&\underbrace{\omega_{q}^{f^{\mathbf{d},\mathbf{t},d}_{0}}\omega_{p}^{\lambda(1)},\omega_{q}^{f^{\mathbf{d},\mathbf{t},d}_{1}}\omega_{p}^{\lambda(1)},\ldots,\omega_{q}^{f^{\mathbf{d},\mathbf{t},d}_{2^{m}-1}}\omega_{p}^{\lambda(1)}},\ldots\\\
&\underbrace{\omega_{q}^{f^{\mathbf{d},\mathbf{t},d}_{0}}\omega_{p}^{\lambda(p-1)},\omega_{q}^{f^{\mathbf{d},\mathbf{t},d}_{1}}\omega_{p}^{\lambda(p-1)},\ldots,\omega_{q}^{f^{\mathbf{d},\mathbf{t},d}_{2^{m}-1}}\omega_{p}^{\lambda(p-1)}}).\end{split}$
(14)

Similarly, we can also obtain
$\psi_{2^{m+s}-p2^{m}}(G^{\mathbf{d},\mathbf{t},d,\lambda})$ as

$\begin{split}&\psi_{2^{m+s}-p2^{m}}(G^{\mathbf{d},\mathbf{t},d,\lambda})\\\
=&(\underbrace{\omega_{q}^{g^{\mathbf{d},\mathbf{t},d}_{0}}\omega_{p}^{\lambda(0)},\omega_{q}^{g^{\mathbf{d},\mathbf{t},d}_{1}}\omega_{p}^{\lambda(0)},\ldots,\omega_{q}^{g^{\mathbf{d},\mathbf{t},d}_{2^{m}-1}}\omega_{p}^{\lambda(0)}},\\\
&\underbrace{\omega_{q}^{g^{\mathbf{d},\mathbf{t},d}_{0}}\omega_{p}^{\lambda(1)},\omega_{q}^{g^{\mathbf{d},\mathbf{t},d}_{1}}\omega_{p}^{\lambda(1)},\ldots,\omega_{q}^{g^{\mathbf{d},\mathbf{t},d}_{2^{m}-1}}\omega_{p}^{\lambda(1)}},\ldots\\\
&\underbrace{\omega_{q}^{g^{\mathbf{d},\mathbf{t},d}_{0}}\omega_{p}^{\lambda(p-1)},\omega_{q}^{g^{\mathbf{d},\mathbf{t},d}_{1}}\omega_{p}^{\lambda(p-1)},\ldots,\omega_{q}^{g^{\mathbf{d},\mathbf{t},d}_{2^{m}-1}}\omega_{p}^{\lambda(p-1)}}).\end{split}$
(15)

###### Theorem 1

Let $f:\mathbb{Z}_{2}^{m}\rightarrow\mathbb{Z}_{q}^{m}$ be a GBF as defined in
Lemma 1. Then the set of codes

$\left\\{\psi_{2^{m+s}-p2^{m}}(U_{t}^{\lambda}),\psi^{*}_{2^{m+s}-p2^{m}}(V_{t}^{\lambda}):0\leq
t<2^{k},0\leq\lambda<p\right\\},$ (16)

forms $(p2^{k+1},2^{m})$-ZCCS${}_{2^{k+1}}^{p2^{m}}$.

###### Proof:

In (13), (14), and (15), each of the parentheses below a complex-valued
sequence contains $2^{m}$ components of the complex-valued sequence. It can be
observed that the $2^{m}$ components in the $i$-th parentheses of
$\psi_{2^{m+s}-p2^{m}}(F^{\mathbf{d},\mathbf{t},d,\lambda})$ and
$\psi_{2^{m+s}-p2^{m}}(G^{\mathbf{d},\mathbf{t},d,\lambda})$ represent the
complex-valued sequences
$\omega_{p}^{\lambda(i-1)}\psi(f^{\mathbf{d},\mathbf{t}})$ and
$\omega_{p}^{\lambda(i-1)}\psi(g^{\mathbf{d},\mathbf{t}})$, respectively,
where $i=1,2,\ldots,p$. Using (9), (14), Lemma 1, and Lemma 2, the ACCF
between $\psi_{2^{m+s}-p2^{m}}(U_{t}^{\lambda})$ and
$\psi_{2^{m+s}-p2^{m}}(U_{t^{\prime}}^{\lambda^{\prime}})$ for $\tau=0$ can be
derived as follows:

$\begin{split}\theta&(\psi_{2^{m+s}-p2^{m}}(U_{t}^{\lambda}),\psi_{2^{m+s}-p2^{m}}(U_{t^{\prime}}^{\lambda^{\prime}}))(0)\\\
&=\displaystyle\sum_{\mathbf{d},d}\theta(\psi_{2^{m+s}-p2^{m}}(F^{\mathbf{d},\mathbf{t},d,\lambda}),\psi_{2^{m+s}-p2^{m}}(F^{\mathbf{d},\mathbf{t}^{\prime},d,\lambda^{\prime}}))(0)\\\
&=\sum_{\mathbf{d},d}\theta(\psi(f^{\mathbf{d},\mathbf{t},d}),\psi(f^{\mathbf{d},\mathbf{t}^{\prime},d}))(0)\sum_{\alpha=0}^{p-1}\omega_{p}^{(\lambda-\lambda^{\prime})\alpha}\\\
&=\theta(\psi(C_{t}),\psi(C_{t^{\prime}}))(0)\displaystyle\sum_{\alpha=0}^{p-1}\omega_{p}^{(\lambda-\lambda^{\prime})\alpha}\\\
&=\begin{cases}p2^{m+k+1},&t=t^{\prime},\lambda=\lambda^{\prime},\\\
0,&t=t^{\prime},\lambda\neq\lambda^{\prime},\\\ 0,&t\neq
t^{\prime},\lambda=\lambda^{\prime},\\\ 0,&t\neq
t^{\prime},\lambda\neq\lambda^{\prime}.\end{cases}\end{split}$ (17)

Again, Using (9), (14), and Lemma 1, the ACCF between
$\psi_{2^{m+s}-p2^{m}}(U_{t}^{\lambda})$ and
$\psi_{2^{m+s}-p2^{m}}(U_{t^{\prime}}^{\lambda^{\prime}})$ for
$0<|\tau|<2^{m}$ can be derived as

$\begin{split}\theta&(\psi_{2^{m+s}-p2^{m}}(U_{t}^{\lambda}),\psi_{2^{m+s}-p2^{m}}(U_{t^{\prime}}^{\lambda^{\prime}}))(\tau)\\\
=&\theta(\psi(C_{t}),\psi(C_{t^{\prime}}))(\tau)\displaystyle\sum_{\alpha=0}^{p-1}\omega_{p}^{(\lambda-\lambda^{\prime})\alpha}\\\
&+\theta(\psi(C_{t}),\psi(C_{t^{\prime}}))(\tau-2^{m})\displaystyle\sum_{\alpha=0}^{p-2}\omega_{p}^{\lambda(\alpha+1)-\lambda^{\prime}\alpha}.\end{split}$
(18)

From Lemma 1, we have

$\begin{split}\theta(\psi(C_{t}),\psi(C_{t^{\prime}}))(\tau)=0,~{}0<|\tau|<2^{m}.\end{split}$
(19)

From (18) and (19), we have

$\begin{split}\theta(\psi_{2^{m+s}-p2^{m}}(U_{t}^{\lambda}),\psi_{2^{m+s}-p2^{m}}(U_{t^{\prime}}^{\lambda^{\prime}}))(\tau)\\!=\\!0,0<|\tau|<2^{m}.\end{split}$
(20)

From (17) and (20), we have

$\begin{split}\theta&(\psi_{2^{m+s}-p2^{m}}(U_{t}^{\lambda}),\psi_{2^{m+s}-p2^{m}}(U_{t^{\prime}}^{\lambda^{\prime}}))(\tau)\\\
&=\begin{cases}p2^{m+k+1},&t=t^{\prime},\lambda=\lambda^{\prime},\tau=0,\\\
0,&t=t^{\prime},\lambda\neq\lambda^{\prime},0<|\tau|<2^{m},\\\ 0,&t\neq
t^{\prime},\lambda=\lambda^{\prime},0<|\tau|<2^{m},\\\ 0,&t\neq
t^{\prime},\lambda\neq\lambda^{\prime},0<|\tau|<2^{m}.\end{cases}\end{split}$
(21)

Similarly, it can be shown that

$\begin{split}\theta&(\psi^{*}_{2^{m+s}-p2^{m}}(V_{t}^{\lambda}),\psi^{*}_{2^{m+s}-p2^{m}}(V_{t^{\prime}}^{\lambda^{\prime}}))(\tau)\\\
&=\begin{cases}p2^{m+k+1},&t=t^{\prime},\lambda=\lambda^{\prime},\tau=0,\\\
0,&t=t^{\prime},\lambda\neq\lambda^{\prime},0<|\tau|<2^{m},\\\ 0,&t\neq
t^{\prime},\lambda=\lambda^{\prime},0<|\tau|<2^{m},\\\ 0,&t\neq
t^{\prime},\lambda\neq\lambda^{\prime},0<|\tau|<2^{m}.\end{cases}\end{split}$
(22)

From Lemma 1, (9), (10), (14), and (15), the ACCF between
$\psi_{2^{m+s}-p2^{m}}(U_{t}^{\lambda})$ and
$\psi^{*}_{2^{m+s}-p2^{m}}(V_{t^{\prime}}^{\lambda^{\prime}})$ for $\tau=0$
can be derived as

$\begin{split}\theta&(\psi_{2^{m+s}-p2^{m}}(U_{t}^{\lambda}),\psi^{*}_{2^{m+s}-p2^{m}}(V_{t^{\prime}}^{\lambda^{\prime}}))(0)\\\
&=\theta(\psi(C_{t}),\psi^{*}(\bar{C}_{t^{\prime}}))(0)\displaystyle\sum_{\alpha=0}^{p-1}\omega_{p}^{(\lambda+\lambda^{\prime})\alpha}.\end{split}$
(23)

From Lemma 1, we have

$\theta(\psi(C_{t}),\psi^{*}(\bar{C}_{t^{\prime}}))(0)=0.$ (24)

From (23) and (24), we have

$\theta(\psi_{2^{m+s}-p2^{m}}(U_{t}^{\lambda}),\psi^{*}_{2^{m+s}-p2^{m}}(V_{t^{\prime}}^{\lambda^{\prime}}))(0)=0.$
(25)

From Lemma 1, (9), (10), (14), (15), and (24), the ACCF between
$\psi_{2^{m+s}-p2^{m}}(U_{t}^{\lambda})$ and
$\psi^{*}_{2^{m+s}-p2^{m}}(V_{t^{\prime}}^{\lambda^{\prime}})$ for
$0<|\tau|<2^{m}$ can be derived as

$\begin{split}\theta&(\psi_{2^{m+s}-p2^{m}}(U_{t}^{\lambda}),\psi^{*}_{2^{m+s}-p2^{m}}(V_{t^{\prime}}^{\lambda^{\prime}}))(\tau)\\\
=&\theta(\psi(C_{t}),\psi^{*}(\bar{C}_{t^{\prime}}))(\tau)\displaystyle\sum_{\alpha=0}^{p-1}\omega_{p}^{(\lambda+\lambda^{\prime})\alpha}\\\
&+\theta(\psi(C_{t}),\psi^{*}(\bar{C}_{t^{\prime}}))(\tau-2^{m})\displaystyle\sum_{\alpha=0}^{p-2}\omega_{p}^{\lambda(\alpha+1)+\lambda^{\prime}\alpha}\\\
=&0.\end{split}$ (26)

From (25) and (26), we have

$\theta(\psi_{2^{m+s}-p2^{m}}(U_{t}^{\lambda}),\psi^{*}_{2^{m+s}-p2^{m}}(V_{t^{\prime}}^{\lambda^{\prime}}))(\tau)=0,~{}|\tau|<2^{m}.$
(27)

The obtained results in (20), (22), and (27) show that the following set of
codes

$\left\\{\psi_{2^{m+s}-p2^{m}}(U_{t}^{\lambda}),\psi^{*}_{2^{m+s}-p2^{m}}(V_{t}^{\lambda}):0\leq
t<2^{k},0\leq\lambda<p\right\\}$

forms $(p2^{k+1},2^{m})$-ZCCS${}_{2^{k+1}}^{p2^{m}}$. ∎

The proposed $(p2^{k+1},2^{m})$-ZCCS${}_{2^{k+1}}^{p2^{m}}$ is optimal as it
satisfies the equality $K=M\lfloor\frac{N}{Z}\rfloor$.

###### Remark 1

For $p=2$, $\delta=lcm(p,q)=q$, and the PBFs $F^{\lambda}$ and $G^{\lambda}$
become GBFs of $m+s$ variables over $\mathbb{Z}_{q}$. For the same value of
$p$, from Theorem 1, we obtain $(2^{k+2},2^{m})$-ZCCS${}_{2^{k+1}}^{2^{m+1}}$
which is optimal and the components of each codewords from a code in
$(2^{k+2},2^{m})$-ZCCS${}_{2^{k+1}}^{2^{m+1}}$ are drawn from the roots of the
polynomial: $z^{q}-1$. Therefore, the proposed construction also generates
ZCCSs of length in the form of power-of-two over the ring $\mathbb{Z}_{q}$.

Let us illustrate the Theorem 1 with the following example:

###### Example 1

Let us assume that $q=2$, $p=3$, $m=3$, $k=1$ and $s=2$. Let us take the GBF
$f:\\{0,1\\}^{3}\rightarrow\mathbb{Z}_{2}$ as follows: $f=x_{1}x_{2}$, where
$G(f\arrowvert_{x_{0}=0})$ and $G(f\arrowvert_{x_{0}=1})$ give a path with
$x_{2}$ as one of the end vertices. From (8), we have

$\begin{split}F^{\lambda}=x_{1}x_{2}+\frac{2\lambda}{3}(x_{3}+2x_{4}),~{}G^{\lambda}=\bar{x}_{1}\bar{x}_{2}+\frac{2\lambda}{3}(x_{3}+2x_{4}),\end{split}$
(28)

where $\lambda=0,1,2$. From (9) and (10), we have

$\begin{split}U_{t}^{\lambda}&=\left\\{F^{\lambda}+d_{0}x_{0}+t_{0}x_{0}+dx_{2}:d_{0},d\in\\{0,1\\}\right\\}\\\
V_{t}^{\lambda}&=\left\\{G^{\lambda}+d_{0}\bar{x}_{0}+t_{0}\bar{x}_{0}+\bar{d}x_{2}:d_{0},d\in\\{0,1\\}\right\\},\end{split}$
(29)

where $(t_{0})$ is the binary vector representation of $t$. Therefore,
$\\{\psi_{8}(U_{t}^{\lambda}),\psi_{8}^{*}(V_{t}^{\lambda}):0\leq t\leq
1,0\leq\lambda\leq 2\\}$ forms $(12,8)$-ZCCS${}_{4}^{24}$ which also optimal.
The components of each code word from a code in $(12,8)$-ZCCS${}_{4}^{24}$ are
drawn from the roots of the polynomial: $z^{\delta}-1$, where
$\delta\\!=\\!lcm(p,q)\\!=\\!lcm(2,3)\\!\\!=\\!6$.

###### Remark 2

From (14) and (15), we see that $\psi_{2^{m+s}-p2^{m}}(U_{t}^{\lambda})$ and
$\psi^{*}_{2^{m+s}-p2^{m}}(V_{t}^{\lambda})$ can also be expressed as the
concatenation of $\omega_{p}^{\lambda(i-1)}\psi(C_{t})$ and
$\omega_{p}^{-\lambda(i-1)}\psi^{*}(\bar{C}_{t})$, respectively, where
$i=1,2,\ldots,p$. Therefore, the proposed PBF generators establish a link
between the proposed direct construction and the indirect constructions of
ZCCSs which are obtained by performing cocatenation operation on the CCCs from
[4].

##  IV Conclusions

In this paper, we have developed a direct construction of optimal ZCCS with
NPT lengths. Unlike the current state-of-the-art which can only generate sub-
optimal ZCCSs with NPT lengths, the novelty of this work stems from the use of
PBFs.

## References

  * [1] H.-H. Chen, _The Next Generation CDMA Technologies_. Wiley, 2007.
  * [2] D. Carey, D. Roviras, and B. Senadji, “Comparison of multiple access interference in asynchronous MC-CDMA and DS-CDMA systems,” in _Proceedings Seventh International Symposium on Signal Processing and Its Applications._ , vol. 2, 2003, pp. 351–354.
  * [3] P. Nagaradjane, A. Swaminathan, K. S. Dhyaneshwaran, B. R. Narayanasamy, and A. Ramakrishnan, “Multipath interference mitigation technique for MC DS/CDMA systems,” in _International Conference on Control, Automation, Communication and Energy Conservation_ , 2009, pp. 1–3.
  * [4] A. Rathinakumar and A. K. Chaturvedi, “Complete mutually orthogonal Golay complementary sets from Reed-Muller codes,” _IEEE Trans. Inf. Theory_ , vol. 54, no. 3, pp. 1339–1346, Mar. 2008.
  * [5] P. Sarkar, S. Majhi, and Z. Liu, “Optimal Z -complementary code set from generalized Reed-Muller codes,” _IEEE Trans. Commun_ , vol. 67, no. 3, pp. 1783–1796, Mar. 2019.
  * [6] P. Fan, W. Yuan, and Y. Tu, “Z-complementary binary sequences,” _IEEE Signal Process. Lett._ , vol. 14, no. 8, pp. 509–512, Aug. 2007.
  * [7] M. Golay, “Complementary series,” _IRE Trans. Inf. Theory_ , vol. 7, no. 2, pp. 82–87, Apr. 1961.
  * [8] J. A. Davis and J. Jedwab, “Peak-to-mean power control in OFDM, Golay complementary sequences, and Reed-Muller codes,” _IEEE Trans. Inf. Theory_ , vol. 45, no. 7, pp. 2397–2417, Nov. 1999.
  * [9] C. Chen, “A novel construction of Z-complementary pairs based on generalized Boolean functions,” _IEEE Signal Process. Lett._ , vol. 24, no. 7, pp. 987–990, July 2017.
  * [10] C. Pai, S. Wu, and C. Chen, “Z-complementary pairs with flexible lengths from generalized Boolean functions,” _IEEE Commun. Lett._ , vol. 24, no. 6, pp. 1183–1187, 2020.
  * [11] A. R. Adhikary, P. Sarkar, and S. Majhi, “A direct construction of $q$-ary even length Z-complementary pairs using generalized Boolean functions,” _IEEE Signal Process. Lett._ , vol. 27, pp. 146–150, 2020.
  * [12] L. Feng, P. Fan, X. Tang, and K. K. Loo, “Generalized pairwise Z-complementary codes,” _IEEE Signal Process. Lett._ , vol. 15, pp. 377–380, 2008.
  * [13] Z. Liu, Y. L. Guan, B. C. Ng, and H.-H. Chen, “Correlation and set size bounds of complementary sequences with low correlation zone,” _IEEE Trans. Commun._ , vol. 59, no. 12, pp. 3285–3289, Dec. 2011.
  * [14] K. G. Paterson, “Generalized Reed-Muller codes and power control in OFDM modulation,” _IEEE Trans. Inf. Theory_ , vol. 46, no. 1, pp. 104–120, Jan. 2000.
  * [15] P. Sarkar, S. Majhi, and Z. Liu, “A direct and generalized construction of polyphase complementary set with low PMEPR,” in _2019 IEEE International Symposium on Information Theory (ISIT)_ , 2019, pp. 2279–2283.
  * [16] ——, “A direct and generalized construction of polyphase complementary sets with low PMEPR and high code-rate for OFDM system,” _IEEE Trans. Commun._ , 2020.
  * [17] S. Das, S. Budišin, S. Majhi, Z. Liu, and Y. L. Guan, “A multiplier-free generator for polyphase complete complementary codes,” _IEEE Trans. Signal Process._ , vol. 66, no. 5, pp. 1184–1196, Mar. 2018.
  * [18] Z. Liu, Y. L. Guan, and U. Parampalli, “New complete complementary codes for peak-to-mean power control in multi-carrier CDMA,” _IEEE Trans. Commun._ , vol. 62, no. 3, pp. 1105–1113, Mar. 2014.
  * [19] Z. Liu, Y. L. Guan, and H.-H. Chen, “Fractional-delay-resilient receiver design for interference-free MC-CDMA communications based on complete complementary codes,” _IEEE Trans. Wireless Commun._ , vol. 14, no. 3, pp. 1226–1236, Mar. 2015.
  * [20] S. Das, S. Majhi, and Z. Liu, “A novel class of complete complementary codes and their applications for apu matrices,” _IEEE Signal Process. Lett._ , vol. 25, no. 9, pp. 1300–1304, Sept. 2018.
  * [21] P. Sarkar and S. Majhi, “A direct construction of optimal zccs with maximum column sequence PMEPR two for MC-CDMA system,” _IEEE Commun. Lett._ , 2020.
  * [22] S. W. Wu, A. Şahin, Z. M. Huang, and C. Y. Chen, “Z-complementary code sets with flexible lengths from generalized Boolean functions,” _IEEE Access_ , vol. 9, pp. 4642–4652, 2021.
  * [23] S. Wu and C. Chen, “Optimal Z-complementary sequence sets with good peak-to-average power-ratio property,” _IEEE Signal Process. Lett._ , vol. 25, no. 10, pp. 1500–1504, Oct. 2018.
  * [24] P. Sarkar, A. Roy, and S. Majhi, “Construction of Z-complementary code sets with non-power-of-two lengths based on generalized Boolean functions,” _IEEE Commun. Lett._ , pp. 1–5, 2020.
  * [25] S. Das, U. Parampalli, S. Majhi, Z. Liu, and S. Budišin, “New optimal Z-complementary code sets based on generalized paraunitary matrices,” _IEEE Trans. Signal Process._ , vol. 68, pp. 5546–5558, 2020\.
  * [26] A. Adhikary and S. Majhi, “New construction of optimal aperiodic Z-complementary sequence sets of odd-lengths,” _Electron. Lett._ , vol. 55, no. 19, pp. 1043–1045, 2019.
  * [27] J. Li, A. Huang, M. Guizani, and H.-H. Chen, “Inter group complementary codes for interference resistant CDMA wireless communications,” _IEEE Trans. Wireless Commun._ , vol. 7, no. 1, pp. 166–174, Jan. 2008.
  * [28] W. Yuan, Y. Tu, and P. Fan, “Optimal training sequences for cyclic-prefix-based single-carrier multi-antenna systems with space-time block-coding,” _IEEE Trans. Wireless Commun._ , vol. 7, no. 11, pp. 4047–4050, Nov. 2008.
  * [29] H. M. Wang, X. Q. Gao, B. Jiang, X. H. You, and W. Hong, “Efficient MIMO channel estimation using complementary sequences,” _IET Commun._ , vol. 1, no. 5, pp. 962–969, Oct. 2007.
  * [30] V. K. Leont’ev, “On pseudo-boolean polynomials,” _Comput. Math. and Math. Phys._ , vol. 55, pp. 1926–1932, 2015.
  * [31] P. P. Vaidyanathan, “Ramanujan sums in the context of signal processing-Part I: Fundamentals,” _IEEE Trans. Signal Process._ , vol. 62, no. 16, pp. 4145–4157, 2014.


# Learning a Dual-Mode Speech Recognition Model via Self-Pruning

###### Abstract

There is growing interest in unifying the streaming and full-context automatic
speech recognition (ASR) networks into a single end-to-end ASR model to
simplify the model training and deployment for both use cases. While in real-
world ASR applications, the streaming ASR models typically operate under more
storage and computational constraints - e.g., on embedded devices - than any
server-side full-context models. Motivated by the recent progress in Omni-
sparsity supernet training, where multiple subnetworks are jointly optimized
in one single model, this work aims to jointly learn a compact sparse on-
device streaming ASR model, and a large dense server non-streaming model, in a
single supernet. Next, we present that, performing supernet training on both
wav2vec 2.0 self-supervised learning and supervised ASR fine-tuning can not
only substantially improve the large non-streaming model as shown in prior
works, and also be able to improve the compact sparse streaming model.

Index Terms—  Neural network pruning, sparsity optimization, supernet,
recurrent neural network transducer

##  1 Introduction

Thus far, end-to-end automatic speech recognition (ASR) models, which use
neural networks to transduce audio into word sequences, have demonstrated
state-of-the-art results compared to conventional hybrid speech recognizers.
Specifically, recurrent neural network transducer (RNN-T) originally presented
in [1] has shown competitive ASR performance on various benchmarks [2, 3, 4].
Typically based on token emission latency, we categorize ASR models into: (i)
streaming recognizers [5, 6] that emit hypothesized words in real time, with
low latency measured by milliseconds, and (ii) non-streaming models [7, 8]
that only emit word hypotheses after processing the complete speech utterance.
Latest streaming recognizers often employ a transformer/conformer encoder [9,
10], and may use a limited future audio context (also referred to as look-
ahead audio frames) [11, 12]. Non-streaming recognizer takes the entire speech
utterance as input, and scaling up the model size can often improve the model
accuracies [8].

Recently it has been shown favorable to unify the streaming and non-streaming
models, either through a single shared encoder [9, 13, 14, 15, 16], or through
cascaded streaming and non-streaming encoders [10, 17]. The efficacy of such
unified or cascaded encoders includes that the previously two separate
development and deployment workflows can be simplified into one process. Note
that in the two-pass cascaded encoders, input acoustic features are typically
first processed by a streaming encoder, and a non-streaming encoder processes
the streaming encoder outputs and aims to cover the first-pass accuracy loss.
While for the unified dual-mode encoder, the non-streaming encoder directly
processes the entire utterance and is immune from the accuracy degradation of
the streaming encoder; additionally, the accuracy and latency of the streaming
encoder can benefit from the weight sharing, or inplace knowledge distillation
from the more performant non-streaming encoder [13].

This work also focuses on the one-pass dual-mode encoder, while in practice,
various streaming ASR models run on devices under more resource constraints,
like disk size and memory footprint. In contrast, most non-streaming models
run from the server with fewer constraints. Therefore, instead of developing
equally sized encoders, it is preferable to jointly build a compact streaming
model and a large non-streaming model for real-world ASR applications. We note
that even though a single encoder is shared for both modes, we can
substantially prune it into a featherweight, e.g., about 30M parameters as a
streaming model, and use the original copy as a performant non-streaming
encoder. Given the recent progress made in neural network pruning [18, 19, 20,
21], we can specify a target sparsity level during model training, prune the
model weights accordingly before inference, and finally obtain a model of the
target model size. Meanwhile, we also aim to maintain the unpruned encoder’s
performance such that we can keep a copy of the original dense encoder and use
it as a competitive non-streaming encoder.

Prior work [20] has shown success on the ASR training of varying sparsities
jointly in a single model, also known as supernet training. A supernet is a
shared-weight backbone network, where a subnetwork is extracted given each
target sparsity level, and all the subnetworks are jointly optimized during
supernet training. While it can facilitate ASR training of various model
sizes, each sub-model in [20] operates with the same inference latency.
Instead, this work focuses on two sparsity levels and two latency conditions:
a high sparsity and low latency for the streaming model, and a zero sparsity
(i.e., dense or unpruned) and full-utterance latency for the non-streaming
model. Thus, in this case, the dual modes refer to the pruned/sparse streaming
mode and the other unpruned/dense non-streaming mode.

Next, it has been widely shown that the self-supervised acoustic model pre-
training based on wav2vec 2.0 [22] can substantially improve large non-
streaming models; given sufficient unlabeled data, the potential accuracy gain
can be proportional to the growing model size [8]. Similarly, achieving
accuracy gains from pre-training will be difficult given a compact model size.
Also, very few works [23] have shown the self-supervised pre-training efficacy
in streaming models. In this paper, we present that by doing the dual-mode
supernet training, self-supervised pre-training is not only able to
substantially improve the large non-streaming model, and also to improve the
compact sparse streaming model.

##  2 Supernet training of a dual-mode ASR Model

###  2.1 RNN-T with Emformer encoder

In this work we focus on the RNN-T based ASR models with the efficient memory
transformer (Emformer) encoder [11].

####  2.1.1 RNN-T

Each speech utterance is parameterized as an input acoustic feature vector
sequence
$\textbf{x}=\\{\textbf{x}_{1}\ldots\textbf{x}_{T}\\}=\textbf{x}_{1:T}$, where
$\textbf{x}_{t}\in\mathbb{R}^{d}$ and $T$ is the number of frames. Denote a
grapheme set or a wordpiece inventory as $\mathcal{Y}$, and the corresponding
output sequence of length $U$ as $\textbf{y}=\\{y_{1}\ldots
y_{U}\\}=\textbf{y}_{1:U}$, where $y_{u}\in\mathcal{Y}$. We define
$\bar{\mathcal{Y}}$ as $\mathcal{Y}\cup\\{\emptyset\\}$, where $\emptyset$ is
the blank label. Denote $\bar{\mathcal{Y}}^{*}$ as the set of all sequences
over output space $\bar{\mathcal{Y}}$, and the element
$\textbf{a}\in\bar{\mathcal{Y}}^{*}$ as an alignment sequence. Then we have
the posterior probability:

$P(\textbf{y}|\textbf{x})=\sum\limits_{\textbf{a}\in\mathcal{B}^{-1}(\textbf{y})}P(\textbf{a}|\textbf{x})$
(1)

where $\mathcal{B}:\bar{\mathcal{Y}}^{*}\rightarrow\mathcal{Y}^{*}$ is a
function that removes blank symbols from an alignment a. A RNN-T model,
$f(\textbf{x};\theta)$, parameterizes the alignment probability
$P(\textbf{a}|\textbf{x})$ with an encoder, a prediction network (predictor)
and a joint network. The encoder $f^{\text{enc}}$ performs a mapping operation
that converts x into another sequence of representations
$\textbf{h}^{\text{enc}}_{1:T}=\\{\textbf{h}_{1}^{\text{enc}}\ldots\textbf{h}^{\text{enc}}_{T}\\}$:

$\textbf{h}^{\text{enc}}_{1:T}=f^{\text{enc}}(\textbf{x};\theta^{\text{enc}})$
(2)

A prediction network $f^{\text{pred}}$ is to produce the new representation
$\textbf{h}^{\text{pred}}_{u}$:

$\textbf{h}^{\text{pred}}_{1:u}=f^{\text{pred}}(y_{0:(u-1)};\theta^{\text{pred}})$
(3)

where $u$ is output label index and $y_{0}=\emptyset$. The joint network
$f^{\text{join}}$ combines encoder output $\textbf{h}^{\text{enc}}_{t}$ and
prediction network output $\textbf{h}^{\text{pred}}_{u}$ to compute logits
$\textbf{z}_{t,u}$:

$\textbf{z}_{t,u}=f^{\text{join}}(\textbf{h}^{\text{enc}}_{t},\textbf{h}^{\text{pred}}_{u};\theta^{\text{join}})$
(4)
$\begin{split}P(y_{u}|\textbf{x}_{1:t},y_{1:(u-1)})=\text{Softmax}(\textbf{z}_{t,u})\end{split}$
(5)

such that the logits go through a softmax function and produce a posterior
distribution of the next output label $y_{u}$ over $\bar{\mathcal{Y}}$. Note
that, the posterior distribution in Eq. 5 is written as
$P(y_{u}|\textbf{x}_{1:T},y_{1:(u-1)})$, if it uses a non-streaming encoder
and takes each full-context utterance as inputs.

####  2.1.2 Emformer encoder for streaming ASR

Chunk-based methods [24, 14] have been widely applied for streaming ASR, and
in this work, we use the block processing method with transformer encoder
layers [11]. The block processing chunks each whole utterance into a sequence
of non-overlapping segments,
$\textbf{x}=\\{\textbf{C}_{1}\ldots\textbf{C}_{i}\ldots\textbf{C}_{I}\\}$,
where $i$ is the index of a segment.

To leverage the context information around each truncated segment, we
concatenate a left contextual block $\textbf{L}_{i}$ (e.g., 20 acoustic frames
or 120ms audio) and a respective right context block $\textbf{R}_{i}$ (look-
ahead context, e.g., 1 frame or 60ms) to each center block $\textbf{C}_{i}$,
to form a contextual segment
$\hat{\textbf{C}}_{i}=\\{\textbf{L}_{i},\textbf{C}_{i},\textbf{R}_{i}\\}$.
Then during inference, a transformer encoder sequentially takes each
$\hat{\textbf{C}}_{i}$ as input, generates an output corresponding to each
$\textbf{C}_{i}$, and forms a sequence of streaming outputs
$\textbf{h}^{\text{enc}}_{1:t}$ (Eq. 2).

Fig. 1: Illustration of the proposed dual-mode ASR supernet training. When the
encoder operates in the streaming mode, it is pruned by the binary mask
(marked in purple). The predictor is pruned during streaming mode in the
similar way, while intact during the non-streaming mode.

###  2.2 Dual-mode ASR training via dynamic Emformer segment sampling

As in Section 2.1.2, we note that the ASR latency depends on the length of the
center block $\textbf{C}_{i}$, and changing the length of $\textbf{C}_{i}$ can
effectively achieve the target latency. For example, when demanding an ultra-
low latency, we can decrease $\textbf{C}_{i}$ to 100-200ms and use a minimal
$\textbf{R}_{i}$ like 60ms or 0. Instead, to implement non-streaming ASR, we
increase $\textbf{C}_{i}$ to a size as long as the full speech utterance and
pad $\textbf{R}_{i}$ as 0.

Thus to learn a dual-mode ASR model with both competitive streaming and non-
streaming modes, at each training step, we randomly sample an Emformer segment
length $|\textbf{C}_{i}|$, equally probable between a length of the target
latency $\tau_{0}$ and a length equal to the longest utterance, $\tau_{1}$.
Then the input utterances will be chunked differently based on the varying
$|\textbf{C}_{i}|$. Both modes still use the same shared encoder, and only the
query/key/value lengths vary according to $|\textbf{C}_{i}|$ in multi-head
self-attention computations. The RNN-T decoder is also fully shared. This is
similar to the domain-specific Emformer segment used in [6], where it applies
a different segment length to each domain-specific data, though the models of
different domains in [6] are all low-latency streaming.

We implement it with the distributed data-parallel training across multiple
GPUs [25]. Thus each GPU has a copy of the model, samples an
$|\textbf{C}_{i}|$ between $\tau_{0}$ and $\tau_{1}$, and processes a sub-
batch of data, after which gradients are then synchronized between GPUs for
each model update, and the model learns both modes simultaneously.

###  2.3 Dual-mode ASR supernet training

As above, prior work [9, 13] and Section 2.2 have described the joint training
of a streaming and full-context model, in which both modes fully share the
same parameters. Next, we aim to jointly learn a sparse streaming encoder and
a dense full-context encoder. As in Figure 1, during training both modes still
share the same parameters, except that the pruning masks are only applied to
the streaming mode. In this case it is a simplified supernet compared to [20],
as it contains only one performant sub-model for the streaming encoder.

We denote a dense RNN-T model as $f(\textbf{x};\theta)$, and a sub-model can
be derived as $f(\textbf{x};m\odot\theta)$ with a binary pruning mask
$m\in\\{0,1\\}^{|\theta|}$, where $\odot$ is the element-wise product. We
perform layer-wise pruning [20] and prune the encoder Emformer layers and the
predictor LSTM layer. A sparsity level $s$ denotes a percentage of the weights
in each layer are pruned. We use an iterative magnitude pruning approach
similar to [18], following the steps:

  * (i)

Training a unpruned dense model till a certain number of training updates
$t_{0}$ (optionally with group lasso weight decay introduced in Section 3.2
below). As in Section 2.2, at each training step, we dynamically sample either
a streaming or a non-streaming mode, and set the Emformer segment length
$|\textbf{C}_{i}|$ accordingly.

  * (ii)

Starting from $t_{0}$:

    * (a)

in each layer, for every $\Delta T$ training steps (i.e., pruning interval),
prune $p$ (e.g., $p=20\%$) of the weight parameters that have the smallest
magnitudes. Pruning is done by setting the corresponding elements in the
binary mask $m$ to $0$, and $m$ is updated every pruning interval $\Delta T$.

    * (b)

at each training step, when the streaming mode is sampled, the pruning masks
are applied to the model weights during the forward-backward pass - gradients
of the masked weights will be zero, and unmasked nonzero. When the non-
streaming mode is sampled, pruning masks are not applied.

  * (iii)

After $n$ pruning intervals, i.e., $t_{0}+n\Delta T$ training updates,
$(1-p)^{n}$ of weight parameters remain. Once the target sparsity level $s$
has been reached, $s=1-(1-p)^{n}$, the mask $m$ is not updated as in (ii, a)
but fixed onward. The dual-mode training proceeds as in (ii, b).

Note that again the mode sampling (ii, b) is done on each GPU, and the
gradients of each sub-batch are aggregated from all machines for each
optimization step. Also to obtain the sparsity speed-up from on-device
hardware, all this work uses structured pruning, block size $8\times 1$ as in
[20].

##  3 Self-pruning via self-supervised learning

###  3.1 Pre-training for ASR supernet

Prior works on self-supervised acoustic model pre-training is mostly focused
on pre-training a non-streaming dense encoder with self-supervised criterion,
and fine-tuning it with supervised ASR criterion. In this work we examine ways
in which the encoder pre-training can improve the dual-mode ASR supernet, and
the pruning masks learned during self-supervised training can be effective for
the downstream ASR task.

We employ the wav2vec 2.0 pre-training criterion. During pre-training we
either use a standard non-streaming encoder as in prior works [22, 8], or use
the dual-mode encoder as in Section 2.2, after which the pretrained model is
fine-tuned with RNN-T criterion, and then the encoder is always dual-mode to
enable the dual-mode ASR.

Note that the encoder pruning, $(t_{0},t_{0}+n\Delta T)$ in Section 2.3, can
be performed either during pre-training, or during RNN-T fine-tuning. In
practice, we find pruning during RNN-T fine-tuning significantly underperforms
pruning during pre-training. Note that the learning rate in RNN-T fine-tuning
has to be small to maintain the pre-training effect, and we conjecture it is
too small to adapt the encoder to the large sparsity changes. While the
predictor is only used in RNN-T training, the LSTM layer is pruned during
fine-tuning.

###  3.2 Pre-training with group lasso weight decay

Given sufficient unlabeled data, it can be helpful to prune from a converged
model than pruning from scratch, so we consider increasing $t_{0}$ in Section
2.3. However, the model weights learned during the dense model training may
not follow the $8\times 1$ block structure as we use for the subsequent
structured pruning, which results in performance degradation. Therefore, we
particularly develop a block regularization technique below to fit the
structured pruning.

In $8\times 1$ block-wise pruning, essentially we would like the weights in
each $8\times 1$ block to be pruned or kept together. _Group lasso_ [26] is a
regularization method which selects grouped variables by penalizing the sum of
$\ell_{2}$-norm of each group. In our case, we define each $8\times 1$ block
as a group, and specifically add a regularization term to the loss function
$\mathcal{L}$:

$\min_{W}\mathcal{L}+\sum_{i=1}^{l}\lambda_{i}\sum_{g\in\mathcal{G}}\|W_{g}^{(i)}\|_{2},$
(6)

where $l$ is the number of layers, $W_{g}^{(i)}$ is a certain $8\times 1$
block in the $i$-th layer, and $\lambda_{i}$ is a hyper-parameter of penalty
strength. The subgradient with respect to $W_{g}^{(i)}$ in the block lasso
term of Eq. 6 is

${\lambda_{i}\over\|W_{g}^{(i)}\|_{2}}W_{g}^{(i)},$ (7)

and the gradient descent direction pushes $W_{g}^{(i)}$ to zeros as weight
decay, with strength $\lambda_{i}/\|W_{g}^{(i)}\|_{2}$. Thus the block
regularization can push some weight blocks close to zeros, and keep other
blocks almost unchanged.

As in many other regularizations, tuning $\lambda_{i}$ could be nontrivial. We
propose to set it dynamically by the average value of the $\ell_{2}$-norm in
$i$-th layer, i.e.
$\lambda_{i}=\lambda\sum_{g\in\mathcal{G}}\|W_{g}^{(i)}\|_{2}/|\mathcal{G}|$,
where $\lambda$ is a global hyper-parameter shared for all layers, e.g.,
$\lambda=1$. In this way, we can greatly simplify the hyper-parameter tuning
for such block regularization. Finally, we apply such group weight decay to
the wav2vec 2.0 pre-training between $(0,t_{0}+n\Delta T)$ training updates,
and turn it off afterwards.

##  4 Experiments

###  4.1 Experimental setup

####  4.1.1 Data

We use the public LibriSpeech (LS) dataset [27] for all the supervised ASR
experiments. We apply speed perturbation [28] to the LS training data and
produce three versions of each audio with speed factors $0.9$, $1.0$ and
$1.1$. We use the complete unlabeled Libri-Light dataset [29] for self-
supervised pre-training. We do not use the additional LibriSpeech language
model (LM) corpus, and LM fusion is not applied in this work.

####  4.1.2 System implementation details

Input acoustic features are 80-dimensional log-mel filterbank coefficients
with 25 ms window size, and with mean and variance normalization. For all
supervised ASR training, we use RNN-T criterion with alignment restrictions to
improve training throughput [30], and apply the frequency and time masking as
in SpecAugment [31].

RNN-T output labels consist of a blank label and 4096 wordpieces generated by
the unigram language model algorithm from SentencePiece toolkit [32], and the
joint network has 1024 hidden units, and a softmax layer of 4097 units. RNN-T
predictor is a 1-layer LSTM of 512 hidden units, with dropout rate $0.3$.

Six 80-dimensional log-mel features are concatenated with stride 6 to form a
480 dimensional vector, followed by a linear layer and mapped to an input to
the encoder. For differing RNN-T model sizes, we vary the Emformer encoder
parameters as in Table 1. All encoders use relative positional embeddings with
clipping distance 64 (3.84s) in self-attention [33], dropout $0.1$, and the
hybrid layer norm configurations 111 Following [34], we find in each
transformer layer, including the additional third layer norm - that prevents
the features from bypassing the transformer layer entirely - noticeably
improves ASR accuracies.  [34]. Given the input feature stride 6, in streaming
mode, Emformer left/center/right context lengths are 1.2s, 180ms, 60ms, i.e.,
$\textbf{L}_{i}=20,\textbf{C}_{i}=3,\textbf{R}_{i}=1$ (Section 2.1.2). In non-
streaming mode, we set the center segment length as 36s, longer than any
training utterance, to use the full context.

For all neural network implementation, we use an in-house extension of
PyTorch-based _fairseq_ [25] toolkit. All experiments use multi-GPU training,
AdamW optimizer with decoupled weight decay 0.01 [35], $\beta_{1}=0.9$,
$\beta_{2}=0.98$, and tri-stage [31] learning rate schedule. The peak learning
rate is 1e-3 for RNN-T training from scratch, 6e-4 for wav2vec 2.0 pre-
training, tuned over $\\{$2e-5, 5e-5$\\}$ for RNN-T fine-tuning. For RNN-T,
all ASR training uses global batch size 2560, up to 300 epochs on LibriSpeech.
For wav2vec 2.0, pre-training on Libri-Light randomly crops each utterance
into a max length of 15s on-the-fly, and the 181M dense models use global
batch size 3072, for 300K training updates; since for supernet training, each
training step has 50% probability sampling the sparse sub-model on each GPU,
where only a subset of the parameters have nonzero gradients, thus we use a
larger global batch size 3840, and a longer training schedule of 400-450K
updates.

As in Section 2.3, we prune all the encoder Emformer and predictor LSTM
layers, with the following layer-wise sparsity level $s$ and pruning interval
$\Delta T$:

  * •

$s=0.67,\Delta T=10K$ for training the 73M RNN-T model,

  * •

$s=0.87,\Delta T=6K$ for training the 181M RNN-T model,

  * •

$s=0.87,\Delta T=6K$ for pre-training the 181M model,

such that the final sparse models after pruning have about 30M parameters in
all cases. In each pruning interval, we prune out $20\%$ remaining weights,
$p=20\%$ as in [21].

Table 1:  Emformer parameters for differing RNN-T model sizes. RNN-T | # layers | embedding dim | FFN dim | attn heads
---|---|---|---|---
35M | 18 | 384 | 1024 | 4
73M | 20 | 512 | 2048 | 8
181M | 24 | 768 | 3072 | 8
Table 2:  WER results on LibriSpeech test-other. For system D2, we use a non-
streaming encoder for the first 50K updates, and then switch it to the dual-
mode encoder afterwards and perform training the same as D1.

unpruned/pruned unpruned streaming non-streaming B1 35M, streaming dense 11.2
- B2 35M, dual-mode dense 10.9 8.7 1-3[1.0pt/0.5pt] C1 73M, streaming sparsity
0.67 10.9 - C2 73M, non-streaming dense - 6.4 1-3[1.0pt/0.5pt] D1 73M,
streaming sparsity 0.67, 10.6 7.0 non-streaming dense D2 non-streaming dense,
50K + D1 10.4 6.6

Table 3: WER results of 181M dense models on LibriSpeech (LS) test sets. Pre-
training randomly crops each utterance on-the-fly into max length 15s for
system B1 and B2, 30s for B3. All streaming ASR uses center context 180ms,
right context 60ms, and 240ms latency in total (Section 4.1.2). LM fusion is
not used.

dataset system test-clean test-other test-clean test-other 3-6[1.0pt/0.5pt]
unpruned streaming unpruned non-streaming LS B0 181M, dual-mode, dense 4.1 9.7
3.1 7.1 Libri-Light + LS B1 dual-mode wav2vec, pretrain on 15s segment, dense
+ B0 3.3 8.3 2.3 5.3 B2 non-streaming wav2vec, pretrain on 15s segment, dense
+ B0 3.1 7.8 2.1 4.3 B3 non-streaming wav2vec, pretrain on 30s segment, dense
+ B0 3.0 7.4 2.1 4.3

Table 4: WER results of 181M supernet models. Pre-training randomly crops each
utterance into max length 15s in all systems below. As Section 3.1, supernet
training refers to using sparsity 0.87 to learn a sparse sub-model, and using
the unpruned model to learn a dense encoder. All streaming ASR uses center
context 180ms, right context 60ms, and has about 32M parameters after pruning.
LM fusion is not used.

dataset system test-clean test-other test-clean test-other 3-6[1.0pt/0.5pt]
pruned streaming unpruned non-streaming LS C1 181M, streaming sparsity 0.87,
non-streaming dense 4.4 11.2 2.7 6.4 Libri-Light + LS C2 non-streaming wav2vec
supernet training, 400K updates + C1 4.6 12.1 2.5 5.3 2-6[1.0pt/0.5pt] C3.1
non-streaming wav2vec with group lasso, 50K updates + 3.9 10.2 2.3 4.5 non-
streaming wav2vec supernet training, 350K updates + C1 2-6[1.0pt/0.5pt] C3.2
non-streaming wav2vec with group lasso, 150K updates + 3.9 9.4 2.2 4.3 non-
streaming wav2vec supernet training, 300K updates + C1

###  4.2 Results of the dual-mode ASR supernet

We first build a pair of 35M dense model baselines: a streaming single-mode
dense model B1, and a streaming and non-streaming dual-mode model B2. ASR word
error rate (WER) results on LibriSpeech _test-other_ are shown in Table 2. We
find B2 slightly outperforms B1, as observed in [13] similarly. Then we build
a pair of 73M models:

  * (i)

a single-mode sparse streaming model C1 with sparsity 0.67, so after pruning
it has about 29M parameters, less than B1 and B2,

  * (ii)

a single-mode dense non-streaming model C2,

such that respectively, the separate single-mode C1 and C2 use the same number
of parameters as the proposed dual-mode supernet model D1. We find the sparse
streaming mode of D1 outperforms both dense models B1, B2 and the single-mode
C1, but the D1 unpruned non-streaming mode falls behind C2.

D1 uses $t_{0}=\Delta T=10K$ above (Section 2.3), and we find simply
increasing $t_{0}$ is not helpful. Then we try a two-step approach in system
D2:

  * 1.

increase $t_{0}=50K$, and use a single-mode non-streaming encoder, i.e.,
always use the full context between $(0,t_{0})$,

  * 2.

then after $t_{0}$, switch it to the dual-mode encoder, and perform training
the same as D1.

Then we find D2 to provide non-streaming performance on a par with C2.
Overall, we demonstrate the efficacy of jointly learning a sparse streaming
sub-model and a dense non-streaming model in a single supernet.

###  4.3 Results of the pre-training efficacy on dual-mode ASR

Then we scale up the model size to 181M, as in Table 3222We note that,
training the large transformer/Emformer models like system B0 from scratch -
without additional regularization techniques - significantly underperforms.
While applying auxiliary training criteria [36] would substantially improve
baseline B0, they can be applied to other competing systems like B1 and B2 as
well, so we will leave it to future work., and first examine the pre-training
effects on dense models.

As in Section 3.1, we perform the wav2vec 2.0 pre-training on Libri-Light, and
then afterwards use dual-mode encoder during RNN-T fine-tuning, to enable the
dual-mode ASR. We also try using the dual-mode encoder during wav2vec pre-
training as well, referred to as the dual-mode wav2vec in B1 (see Table 3).
However, by comparing B1 and B2, we find pre-training with just the non-
streaming encoder instead is much more effective for both non-streaming and
streaming ASR. Note that system B1 and B2 are pre-trained on audio segments
cropped up to 15s, and we further increase the max segment length to 30s on
system B3. We find B3 can produce further better streaming results compared to
B2.

In all cases above, we present that pre-training can not only substantially
improve the non-streaming ASR results as widely shown in prior works, and also
noticeably improve streaming ASR performance, as one of the contributions in
this work. The proposed dynamic Emformer segment sampling (Section 2.2) allows
for using a non-streaming encoder to maximize the pre-training benefits, and
enabling the high-performing dual-mode ASR afterwards.

###  4.4 Results of supernet training with both self-supervised and supervised
criteria

Next, as in Table 4, we first build a dual-mode supernet model C1 with labeled
data only, and then start to use unlabeled data and examine the pre-training
effects on both the sparse streaming mode and the dense non-streaming mode.

As discussed in Section 3.1, we find any encoder pruning during RNN-T fine-
tuning results in severe streaming ASR degradation, significantly falling
behind the baseline C1. Thus instead we prune the encoder during pre-training.
Note that for the ASR supernet training (Section 2.3), we will sample between
streaming and non-streaming modes; however, given the result comparison
between B1 and B2, we always use non-streaming mode during pre-training - we
sample between the sub-model and the whole model (i.e., apply the mask or
not), and both operate in the non-streaming mode.

Thus the encoder pruning mask is learned completely on the unlabeled data
without supervision, and the encoder mask is fixed during RNN-T fine-tuning,
so we refer to such process as self-pruning. The predictor is also pruned for
streaming ASR, and the predictor mask is learned during RNN-T fine-tuning.
Additionally, after such supernet training, the identified sparse sub-model
will go through different post-processing and specialized hardware for storage
and run-time optimization, therefore, we can choose separate best checkpoints
across epochs for the sparse streaming sub-model and the dense non-streaming
model respectively, based on the ASR accuracies on LS _dev-other_ subset.

Following such training algorithm, although the system C2 gives higher non-
streaming accuracies than the baseline C1 without pre-training, C2 still
trails C1 on the streaming accuracy333Although by comparing the dense model B2
and B3 (Table 3), pretraining on 30s audio segments is more effective for
streaming ASR than on 15s, we find such observation does not hold true for the
supernet training like system C3.1. We conjecture the explanation that
pretraining on longer segments for a highly sparse model results in more
difficult neural network optimization problems, e.g., the training will
diverge using the same learning rate 6e-4, and we have to use 4e-4. Thus
instead, system C2, C3.1 and C3.2 (Table 4) are all pre-trained on segments up
to 15s.. Then we note that C2 performs iterative pruning from scratch, i.e.,
using a small $t_{0}$, $t_{0}=\Delta T=6K$ updates (Section 4.1.2). Instead,
we can increase $t_{0}$ and prune a better converged model, assuming that the
weights will be better initialized for the pruning criterion (i.e., weight
magnitude). However, we find simply increasing $t_{0}$ can only produce
results similar to C2, since as discussed in Section 3.2, weights learned
during $(0,t_{0})$ do not follow the $8\times 1$ block structure, and the
structured sparsity may prune out important weights in each block. Therefore,
next, we not only increase $t_{0}$ and also apply the additional group lasso
weight decay during $(0,t_{0}+n\Delta T)$. We find the resulting system C3.1
with $t_{0}=50K$ outperforms both baseline C1 and C2.

Finally, we increase $t_{0}=150K$ in system C3.2, and find (i) compared to the
dense model B2 without any sparsity (Table 3), C3.2 can match the topline non-
streaming performance, and (ii) compared to baseline C1, C3.2 can effectively
leverage self-supervised learning and provide a significantly improved sparse
streaming model, by 11-16% WER reductions.

##  5 Conclusions

Overall, we first present a dynamic Emformer segment sampling framework to
enable a dual-mode encoder. We demonstrate that, jointly learning a
featherweight sparse streaming ASR model and a large dense non-streaming model
- in a single supernet - can provide competitive accuracies compared to
learning each individually. Second, the proposed dual-mode encoder can
dynamically use the non-streaming mode during the wav2vec 2.0 pre-training and
perform dual-mode ASR thereafter, which allows for self-supervised learning
equally helpful for the non-streaming mode and also to substantially improve
the streaming ASR.

Next, we show that the proposed group lasso weight decay can effectively
address the block patterns as required in structured pruning, such that the
self-supervised pre-training is able to identify a performant and robust sub-
model for the downstream task. Finally, we conclude that for both self-
supervised and supervised learning, the proposed supernet training of a sparse
sub-model and a dense model jointly can provide an equally competitive non-
streaming ASR model and also provide a noticeably improved sparse streaming
model.

## References

  * [1] Alex Graves,  “Sequence transduction with recurrent neural networks,”  Proc. ICML Workshop on Representation Learning, 2012.
  * [2] Chung-Cheng Chiu, Wei Han, Yu Zhang, Ruoming Pang, Sergey Kishchenko, Patrick Nguyen, Arun Narayanan, Hank Liao, Shuyuan Zhang, Anjuli Kannan, et al.,  “A comparison of end-to-end models for long-form speech recognition,”  in Proc. ASRU, 2019.
  * [3] Jinyu Li, Yu Wu, Yashesh Gaur, Chengyi Wang, Rui Zhao, and Shujie Liu,  “On the comparison of popular end-to-end models for large scale speech recognition,”  Proc. Interspeech, 2020.
  * [4] Xiaohui Zhang, Frank Zhang, Chunxi Liu, Kjell Schubert, Julian Chan, Pradyot Prakash, Jun Liu, Ching-feng Yeh, Fuchun Peng, Yatharth Saraf, and Geoffrey Zweig,  “Benchmarking LF-MMI, CTC and RNN-T criteria for streaming ASR,”  in Proc. SLT, 2021.
  * [5] Tara N Sainath, Yanzhang He, Bo Li, Arun Narayanan, Ruoming Pang, Antoine Bruguier, Shuo-yiin Chang, Wei Li, Raziel Alvarez, Zhifeng Chen, et al.,  “A streaming on-device end-to-end model surpassing server-side conventional model quality and latency,”  in Proc. ICASSP, 2020.
  * [6] Jay Mahadeokar, Yangyang Shi, Yuan Shangguan, Chunyang Wu, Alex Xiao, Hang Su, Duc Le, Ozlem Kalinli, Christian Fuegen, and Michael L Seltzer,  “Flexi-transducer: Optimizing latency, accuracy and compute for multi-domain on-device scenarios.,”  in Proc. Interspeech, 2021.
  * [7] Anmol Gulati, James Qin, Chung-Cheng Chiu, Niki Parmar, Yu Zhang, Jiahui Yu, Wei Han, Shibo Wang, Zhengdong Zhang, Yonghui Wu, et al.,  “Conformer: convolution-augmented transformer for speech recognition,”  Proc. Interspeech, 2020.
  * [8] Yu Zhang, James Qin, Daniel S Park, Wei Han, Chung-Cheng Chiu, Ruoming Pang, Quoc V Le, and Yonghui Wu,  “Pushing the limits of semi-supervised learning for automatic speech recognition,”  Proc. of NeurIPS SAS Workshop, 2020.
  * [9] Qian Zhang, Han Lu, Hasim Sak, Anshuman Tripathi, Erik McDermott, Stephen Koo, and Shankar Kumar,  “Transformer transducer: a streamable speech recognition model with transformer encoders and RNN-T loss,”  in Proc. ICASSP, 2020.
  * [10] Bo Li, Anmol Gulati, Jiahui Yu, Tara N Sainath, Chung-Cheng Chiu, Arun Narayanan, Shuo-Yiin Chang, Ruoming Pang, Yanzhang He, James Qin, et al.,  “A better and faster end-to-end model for streaming ASR,”  in Proc. ICASSP, 2021.
  * [11] Yangyang Shi, Yongqiang Wang, Chunyang Wu, Ching-Feng Yeh, Julian Chan, Frank Zhang, Duc Le, and Mike Seltzer,  “Emformer: Efficient memory transformer based acoustic model for low latency streaming speech recognition,”  in Proc. ICASSP, 2021.
  * [12] Yangyang Shi, Chunyang Wu, Dilin Wang, Alex Xiao, Jay Mahadeokar, Xiaohui Zhang, Chunxi Liu, Ke Li, Yuan Shangguan, Varun Nagaraja, et al.,  “Streaming transformer transducer based speech recognition using non-causal convolution,”  in Proc. ICASSP, 2022.
  * [13] Jiahui Yu, Wei Han, Anmol Gulati, Chung-Cheng Chiu, Bo Li, Tara N Sainath, Yonghui Wu, and Ruoming Pang,  “Dual-mode ASR: Unify and improve streaming ASR with full-context modeling,”  Proc. ICLR, 2021.
  * [14] Zhuoyuan Yao, Di Wu, Xiong Wang, Binbin Zhang, Fan Yu, Chao Yang, Zhendong Peng, Xiaoyu Chen, Lei Xie, and Xin Lei,  “Wenet: Production oriented streaming and non-streaming end-to-end speech recognition toolkit,”  Proc. Interspeech, 2021.
  * [15] Kwangyoun Kim, Felix Wu, Prashant Sridhar, Kyu J Han, and Shinji Watanabe,  “Multi-mode transformer transducer with stochastic future context,”  Proc. Interspeech, 2021.
  * [16] Felix Weninger, Marco Gaudesi, Md Akmal Haidar, Nicola Ferri, Jesús Andrés-Ferrer, and Puming Zhan,  “Conformer with dual-mode chunked attention for joint online and offline asr,”  Proc. Interspeech, 2022.
  * [17] Arun Narayanan, Tara N Sainath, Ruoming Pang, Jiahui Yu, Chung-Cheng Chiu, Rohit Prabhavalkar, Ehsan Variani, and Trevor Strohman,  “Cascaded encoders for unifying streaming and non-streaming asr,”  in Proc. ICASSP, 2021.
  * [18] Jonathan Frankle and Michael Carbin,  “The lottery ticket hypothesis: Finding sparse, trainable neural networks,”  Proc. ICLR, 2019.
  * [19] Zhaofeng Wu, Ding Zhao, Qiao Liang, Jiahui Yu, Anmol Gulati, and Ruoming Pang,  “Dynamic sparsity neural networks for automatic speech recognition,”  in Proc. ICASSP, 2021.
  * [20] Haichuan Yang, Yuan Shangguan, Dilin Wang, Meng Li, Pierce Chuang, Xiaohui Zhang, Ganesh Venkatesh, Ozlem Kalinli, and Vikas Chandra,  “Omni-sparsity DNN: fast sparsity optimization for on-device streaming E2E ASR via supernet,”  in Proc. ICASSP, 2022.
  * [21] Shaojin Ding, Tianlong Chen, and Zhangyang Wang,  “Audio lottery: speech recognition made ultra-lightweight, noise-robust, and transferable,”  in Proc. ICLR, 2022.
  * [22] Alexei Baevski, Yuhao Zhou, Abdelrahman Mohamed, and Michael Auli,  “wav2vec 2.0: a framework for self-supervised learning of speech representations,”  Advances in Neural Information Processing Systems, 2020.
  * [23] Tara N Sainath, Yanzhang He, Arun Narayanan, Rami Botros, Weiran Wang, David Qiu, Chung-Cheng Chiu, Rohit Prabhavalkar, Alexander Gruenstein, Anmol Gulati, et al.,  “Improving the latency and quality of cascaded encoders,”  in Proc. ICASSP, 2022.
  * [24] Xie Chen, Yu Wu, Zhenghao Wang, Shujie Liu, and Jinyu Li,  “Developing real-time streaming transformer transducer for speech recognition on large-scale dataset,”  in Proc. ICASSP, 2021.
  * [25] Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli,  “fairseq: A fast, extensible toolkit for sequence modeling,”  in Proceedings of NAACL-HLT 2019: Demonstrations, 2019.
  * [26] Ming Yuan and Yi Lin,  “Model selection and estimation in regression with grouped variables,”  Journal of the Royal Statistical Society: Series B (Statistical Methodology), vol. 68, no. 1, pp. 49–67, 2006.
  * [27] Vassil Panayotov, Guoguo Chen, Daniel Povey, and Sanjeev Khudanpur,  “LibriSpeech: an ASR corpus based on public domain audio books,”  in Proc. ICASSP, 2015.
  * [28] Tom Ko, Vijayaditya Peddinti, Daniel Povey, and Sanjeev Khudanpur,  “Audio augmentation for speech recognition,”  in Proc. Interspeech, 2015.
  * [29] Jacob Kahn, Morgane Rivière, Weiyi Zheng, Evgeny Kharitonov, Qiantong Xu, Pierre-Emmanuel Mazaré, Julien Karadayi, Vitaliy Liptchinsky, Ronan Collobert, Christian Fuegen, et al.,  “Libri-light: A benchmark for asr with limited or no supervision,”  in Proc. ICASSP, 2020.
  * [30] Jay Mahadeokar, Yuan Shangguan, Duc Le, Gil Keren, Hang Su, Thong Le, Ching-Feng Yeh, Christian Fuegen, and Michael L Seltzer,  “Alignment restricted streaming recurrent neural network transducer,”  in Proc. SLT, 2021.
  * [31] Daniel S Park, William Chan, Yu Zhang, Chung-Cheng Chiu, Barret Zoph, Ekin D Cubuk, and Quoc V Le,  “SpecAugment: A simple data augmentation method for automatic speech recognition,”  Proc. Interspeech, 2019.
  * [32] Taku Kudo and John Richardson,  “Sentencepiece: A simple and language independent subword tokenizer and detokenizer for neural text processing,”  arXiv preprint arXiv:1808.06226, 2018.
  * [33] Peter Shaw, Jakob Uszkoreit, and Ashish Vaswani,  “Self-attention with relative position representations,”  Proc. NAACL, 2018.
  * [34] Yongqiang Wang, Abdelrahman Mohamed, Duc Le, Chunxi Liu, Alex Xiao, Jay Mahadeokar, Hongzhao Huang, Andros Tjandra, Xiaohui Zhang, Frank Zhang, et al.,  “Transformer-based acoustic modeling for hybrid speech recognition,”  in Proc. ICASSP, 2020.
  * [35] Ilya Loshchilov and Frank Hutter,  “Decoupled weight decay regularization,”  Proc. ICLR, 2019.
  * [36] Chunxi Liu, Frank Zhang, Duc Le, Suyoun Kim, Yatharth Saraf, and Geoffrey Zweig,  “Improving RNN transducer based ASR with auxiliary tasks,”  in Proc. SLT, 2021.