# Adversarial Joint Training with Self-Attention Mechanism for Robust End-to-
End Speech Recognition

LLujun Li  YYikai Kang  YYuchen Shi  LLudwig Kürzinger  TTobias Watzel
GGerhard Rigoll  Department of Electrical and Computer Engineering, Technical
University of Munich, Munich, Germany.
$\dagger$ Equal contribution, alphabetical order

###### Abstract

Lately, the self-attention mechanism has marked a new milestone in the field
of automatic speech recognition (ASR). Nevertheless, its performance is
susceptible to environmental intrusions as the system predicts the next output
symbol depending on the full input sequence and the previous predictions. A
popular solution for this problem is adding an independent speech enhancement
module as the front-end. Nonetheless, due to being trained separately from the
ASR module, the independent enhancement front-end falls into the sub-optimum
easily. Besides, the handcrafted loss function of the enhancement module tends
to introduce unseen distortions, which even degrade the ASR performance.
Inspired by the extensive applications of the generative adversarial networks
(GANs) in speech enhancement and ASR tasks, we propose an adversarial joint
training framework with the self-attention mechanism to boost the noise
robustness of the ASR system. Generally, it consists of a self-attention
speech enhancement GAN and a self-attention end-to-end ASR model. There are
two highlights which are worth noting in this proposed framework. One is that
it benefits from the advancement of both self-attention mechanism and GANs;
while the other is that the discriminator of GAN plays the role of the global
discriminant network in the stage of the adversarial joint training, which
guides the enhancement front-end to capture more compatible structures for the
subsequent ASR module and thereby offsets the limitation of the separate
training and handcrafted loss functions. With the adversarial joint
optimization, the proposed framework is expected to learn more robust
representations suitable for the ASR task. We execute systematic experiments
on the corpus AISHELL-1, and the experimental results show that on the
artificial noisy test set, the proposed framework achieves the relative
improvements of 66% compared to the ASR model trained by clean data solely,
35.1% compared to the speech enhancement & ASR scheme without joint training,
and 5.3% compared to multi-condition training.

self-attention mechanism,

generative adversarial networks,

speech enhancement,

robust speech recognition,

###### keywords:

Research

$\dagger$ $\dagger$

##  1 Introduction

In recent years, attention-based end-to-end neural networks, which subsume the
acoustic and language models into a single neural network, trigger the
revolution in the field of automatic speech recognition (ASR) [1, 2] and are
challenging the dominance of Hidden Markov Model-based hybrid systems [3].
Furthermore, the self-attention mechanism has made another breakthrough in the
innovation of the attention architecture, which considers the whole sequence
at once to model feature interactions that are arbitrarily distant in time,
leading to faster convergence and state-of-the-art results in ASR [4, 5, 6, 7,
8, 9, 10, 11, 12]. The self-attention system predicts the next output symbol
conditioned on the full sequence of the previous predictions. Once a mistake
occurs in one estimation step due to noise interference, all the subsequent
steps will be disturbed. As speech signals are inevitably interfered by
various background noises in the realistic environment, it is crucial to
improve the robustness of the self-attention mechanism for practical
application.

The mainstream solution to the noise robustness problem is adding an
independent speech enhancement (SE) module as the front-end of ASR. Speech
enhancement aims to transform the interfered speech to its original clean
version, which is achieved by various approaches, i.e., the statistical method
like Wiener filter [13], the time-frequency masking [14, 15, 16], the signal
approximation [17, 18], the spectral mapping [19, 20], etc. No matter what
approach the speech enhancement model adopts to achieve the goal, it is
trained separately from the ASR model on different loss functions (i.e., mean
squared error [21]) and being evaluated by different objective criteria (i.e.,
Mean Opinion Score (MOS) prediction of the intrusiveness of background noise
[22], Segmental SNR [23]). This mismatch between the enhancement training and
the final ASR task leads to a sub-optimum easily [24]. Moreover, the
handcrafted loss functions tend to generate over-smoothed spectra or introduce
unseen distortions, which sometimes even degrade the downstream ASR
performance [25].

To obtain the optimum and circumvent introducing unnecessary distortion, the
idea of a joint training framework is proposed for robust speech recognition
[26, 25, 27, 28]. The fundamental concept of the joint training is
concatenating the speech enhancement front-end and a downstream ASR model to
build an entire neural network and jointly adjust the parameters in each
module. The goal here is that the enhancement front-end tends to produce
enhanced features desired by the ASR component, and the ASR module can guide
the enhancement module to a more discriminative direction. In this way, the
joint framework is optimized on the final ASR objectives, i.e., word/character
error rate (W/CER).

Generative adversarial networks (GANs) aim at mapping samples $\hat{x}$ from
the distribution $\mathcal{\hat{X}}$ to samples $x$ from another distribution
$\mathcal{X}$. There are two components within GANs. One is the generator (G),
which performs the mapping; and the other is the discriminator (D), which
guides the training of the generator. GANs have been applied to various speech
signal processing tasks, such as speech enhancement [29, 30], robust speaker
verification [31], spoken language identification [32], speech emotion
recognition [33], data augmentation [34], and robust speech recognition [35].

Inspired by the advancement of self-attention mechanism and various
applications of GAN in speech-related tasks, we propose an adversarial joint
training framework with self-attention mechanism to boost the robustness of
the self-attention ASR systems, which consists of a self-attention speech
enhancement GAN (SA_SEGAN) and a self-attention end-to-end ASR model (SA_ASR),
where we experiment with Transformer [36] and Conformer [37]. The discriminant
component of SA_SEGAN is first utilized to distinguish the enhanced features
from the original clean features, instructing the enhancement module to output
the clean distribution. When it comes to the stage of the joint training, the
D component acts as the global training guide, and it will shift the direction
for the G component to produce more congruous features for the ASR task. As
the global guide, the discriminator is expected to remedy the limitation of
the separate training and handcrafted loss functions, alleviate the
distortion, and lead the speech enhancement component to the global optimum.
Meanwhile, the enhancement module is supposed to capture more underlying
structural characteristics. With this global guide, the whole framework is
expected to learn more robust representations compatible with the ASR task
automatically.

In summary, the main contributions of this paper are the following:

  * •

We propose a self-attention based jointly-trained adversarial framework
targeting robust speech recognition. This framework benefits from the
advancement of both self-attention mechanism and adversarial training;

  * •

We exert the global adversarial training, where the discriminant component
does not concentrate on the enhancement front-end exclusively, but also plays
the role of the global training guide.

  * •

The proposed framework yields remarkable results, which achieve relative
improvements of 66% compared to the ASR model trained by clean data solely,
35.1% compared to the scheme without joint training, and 5.3% compared to
multi-condition training.

##  2 Related Work

GANs have been applied in speech enhancement tasks without attention [29, 38,
39] and with attention [40, 41]. These works validate the functionality of GAN
in the enhancement task on diverse objective criteria; however, they lack
proofs of the effectiveness of their work for the downstream ASR task.

GANs have also been employed to improve the robustness of the ASR model [35,
42, 43, 44]. A potential limitation lies in the weak matching and
communication between the integrated modules. For instance, speech enhancement
and speech recognition are often designed independently, and the enhancement
system is tuned according to metrics that are not straightly relative to the
final ASR performance.

To address this concern, joint training is a promising approach. An early
attempt was proposed in [45], where a feature extraction front-end and a
Gaussian Mixture Model-Hidden Markov Model back-end are jointly trained on
maximum mutual information. Afterwards, other interesting works are published
in this field [25, 26, 46, 47, 48]. Nevertheless, an effective integration
between the various systems has been difficult for many years, mainly due to
the different nature of the technologies involved at different steps. For
example, in [25, 46], the joint training is actually performed as a fine-
tuning procedure. To tackle this problem, this paper deploys the discriminant
component of GAN as a global guide, leading the enhancement module to match
the downstream ASR module.

##  3 Self-attention Based SE-ASR Scheme

###  3.1 Overview

Fig. 1 illustrates an overview of our proposed joint training framework for
robust end-to-end speech recognition pictorially. The system consists of a
self-attention enhancement front-end and a self-attention ASR model. Given the
raw noisy speech input $\tilde{X}$ and the raw clean input $X^{*}$, we
illustrate the entire procedure of the joint training pipeline in the
following forms:

$\hat{X}=\text{Generator}(\tilde{X}),$ (1) $\hat{F}=\text{FBank}(\hat{X}),$
(2) $P(Y|\hat{F})=\text{SA\\_ASR}(\hat{F}),$ (3)
$P(D|\hat{X},X^{*})=\text{Discriminator}(\hat{X},X^{*}).$ (4)

Here, $\text{Generator}(\cdot)$ acts as a speech enhancement front-end
realized by the generator component of SA_SEGAN [40], which transforms the
noisy raw input $\tilde{X}$ to the enhanced $\hat{X}$. $\text{FBank}(\cdot)$
is a function for extracting the normalized log FBank features $\hat{F}$ from
the enhancement outputs $\hat{X}$. Subsequently, $\text{SA\\_ASR}(\cdot)$ is
an ASR system based on self-attention layers realized by Transformer [36] or
Conformer [37] architecture. $Y$ is the outputs of the whole scheme.
$\text{Discriminator}(\cdot)$ is realized by the discriminator component of
SA_SEGAN [40], which distinguishes enhanced outputs from clean data.

Figure 1: Overview of the SE_ASR joint training framework.

###  3.2 Self-attention Mechanism

Self-attention [49] relates the information over different positions of the
entire input sequence for computing the attention distribution using scaled
dot-product attention:

$\text{Attention}(\bm{Q},\bm{K},\bm{V})=\text{softmax}(\frac{\bm{QK}^{T}}{\sqrt{d_{k}}})\bm{V}.$
(5)

$\bm{Q}\in\mathbb{R}^{t_{q}\times d_{q}}$, $\bm{K}\in\mathbb{R}^{t_{k}\times
d_{k}}$, and $\bm{V}\in\mathbb{R}^{t_{v}\times d_{v}}$ are three inputs of the
self-attention layer: queries, keys, and values, where $t_{q}$, $t_{k}$, and
$t_{v}$ are the element numbers in different inputs while $d_{q}$, $d_{k}$,
and $d_{v}$ denote the corresponding element dimensions. The scalar
$\frac{1}{\sqrt{d_{k}}}$ prevents the softmax function from falling into
regions with tiny gradients. One query’s output is computed as a weighted sum
of the values, where each weight of the value is computed by a designated
function of the query with the corresponding key.

###  3.3 Self-attention Speech Enhancement GANs

####  3.3.1 Speech Enhancement GANs (SEGAN)

Given a dataset
$\mathcal{X}=\\{(\bm{x^{*}_{1}},\bm{\tilde{x}_{1})},(\bm{x^{*}_{2}},\bm{\tilde{x}_{2}}),\cdots,(\bm{x^{*}_{N}},\bm{\tilde{x}_{N}})\\}$
consisting of $N$ pairs of raw signals: clean speech signal $\bm{x}^{*}$ and
noisy speech signal $\bm{\tilde{x}}$. Speech enhancement aims to find a
mapping $f_{\theta}(\bm{\tilde{x}}):\bm{\tilde{x}}\to\bm{\hat{x}}$ to
transform the raw noisy signal $\bm{\tilde{x}}$ to the enhanced signal
$\bm{\hat{x}}$. $\theta$ contains the parameters of the enhancement network.

Conforming to GAN’s principle [50], the generator G is for learning an
effective mapping that can imitate the real data distribution to generate
novel samples related to those of the training set. Hence G acts as the
enhancement function. In contrast, the discriminator D plays the role of a
classifier which distinguishes the real sample, coming from the dataset that G
is imitating, from the fake samples, made up by G. D guides $\theta$ towards
the distribution of clean speech signals. To sum up, SEGAN designates the
generator G for the enhancement mapping, i.e.
$\bm{\hat{x}}=G(\bm{\tilde{x}})$, while designates the discriminator D to
guide the training of G by classifying $(\bm{x^{*}},\bm{\tilde{x}})$ as real
and $(\bm{\hat{x}},\bm{\tilde{x}})$ as fake. Eventually, G learns to produce
enhanced signals $\bm{\hat{x}}$ good enough to fool D such that D classifies
$(\bm{\hat{x}},\bm{\tilde{x}})$ as real.

Figure 2: Illustration of the application of self-attention mechanism in
speech enhancement GANs with $L=9$, $C=6$, $p=3$, and $b=2$.

####  3.3.2 Self-attention Speech Enhancement GANs (SA_SEGAN)

SA_SEGAN [40] is SEGAN with the adoption of the self-attention layer adapted
from non-local attention [51, 52]. Given the feature map
$\bm{F}\in\mathbb{R}^{L\times C}$ output by the 1-dim convolutional layer,
where $L$ is the time dimension, $C$ is the number of channels, the query
matrix $\bm{Q}$, the key matrix $\bm{K}$, and the value matrix $\bm{V}$ are
obtained via transformations:

$\bm{Q}=\bm{FW}^{Q},\bm{K}=\bm{FW}^{K},\bm{V}=\bm{FW}^{V},$ (6)

where $\bm{W}^{Q}$, $\bm{W}^{K}$, and $\bm{W}^{V}$ denote the weight matrices
of the convolutional layer. Furthermore, Phan et al. [40] introduce two
factors, $b$ and $p$, for memory efficiency. $b$ reduces the channel
dimension, while $p$ reduces the number of keys and values by a max pooling
layer with filter width and stride size of $p$. Therefore, the dimension of
the matrices are $\bm{Q}\in\mathbb{R}^{L\times\frac{C}{b}}$,
$\bm{K}\in\mathbb{R}^{\frac{L}{p}\times\frac{C}{b}}$, and
$\bm{V}\in\mathbb{R}^{\frac{L}{p}\times\frac{C}{b}}$. The attention map
$\bm{A}$ and the attentive output $\bm{O}$ are then computed as

$\bm{A}=softmax(\bm{QK}^{T}),\quad\bm{A}\in\mathbb{R}^{L\times\frac{L}{p}},$
(7)
$\bm{O}=(\bm{AV})\bm{W}^{O},\quad\bm{W}^{O}\in\mathbb{R}^{\frac{C}{b}\times
C}.$ (8)

Each element $a_{ij}\in\bm{A}$ indicates the extent to which the model attends
to the $j$th column $\bm{v}_{j}$ of $\bm{V}$ when producing the $i$th output
$\bm{o}_{i}$ of $\bm{O}$. With the weight matrix $\bm{W}^{O}$ realized by a 1
$\times$ 1 convolution layer of $C$ filters, the shape of $\bm{O}$ is restored
to the original shape $L\times C$.

In the end, SA_SEGAN contains a shortcut connection to facilitate information
propagation, and a learnable parameter $\beta$ is employed to balance the
weight between the output $\bm{O}$ and the input feature map $\bm{F}$ as

$\bm{F^{\prime}}=\beta\bm{O}+\bm{F}.$ (9)

We illustrate the diagram of a simplified self-attention layer with $L=9$,
$C=6$, $p=3$, and $b=2$ in Fig. 2.

Figure 3: Illustration of the SA_SEGAN architecture. (a) the generator
component. (b) the discriminator component. [40]

####  3.3.3 Network Architecture

The architectures of the generator $G$ and the discriminator $D$ are depicted
in Fig. 3 (a) and (b). The G component makes use of an encoder-decoder
architecture with fully-convolutional layers [53]. The generator’s encoder
comprises 11 1-dim stridden convolutional layers with a common filter width of
31 and a stride length of 2, followed by parametric rectified linear units
(PReLUs) [54]. The encoder receives a one-second segment of the raw signal
sampled at 16 kHz, approximately 16384 samples as the input. To compensate for
the smaller and smaller convolutional output, the number of filters increases
along the encoder’s depth $\\{16,32,32,64,64,128,128,256,256,512,1024\\}$ ,
resulting in output size of the feature map $\\{8192\times 16,4096\times
32,2048\times 32,1024\times 64,512\times 64,256\times 128,128\times
128,64\times 256,32\times 256,16\times 512,8\times 1024\\}$. At the 11th layer
of the encoder, the encoding vector $\bm{c}\in\mathbb{R}^{8\times 1024}$ is
stacked with the noise sample $\bm{z}\in\mathbb{R}^{8\times 1024}$, sampled
from the distribution $\mathcal{N}(0,I)$, and presented to the decoder.

The decoder component mirrors the encoder architecture with the same number of
filters and the filter width to reverse the encoding process through
deconvolutions. The same as the encoder, each deconvolutional layer is again
followed by a PReLUs. The skip connections are deployed to connect the
encoding layer with its corresponding decoding layer to allow the information
flow between the encoding stage and the decoding stage.

The discriminator is constructed of a similar architecture to the encoder
component of the generator. However, it receives the two-channel input and
utilize virtual batch-norm [55] before LeakyReLU [56] activation with $\alpha$
= 0.3. Moreover, the D network is topped up with a $1\times 1$ convolutional
layer to reduce the dimension of the output of the last convolutional layer
from $8\times 1024$ to 8 for the subsequent classification task with the
softmax layer.

The self-attention layer illustrated in section 3.3.2 couples with the
(de)convolutional layer of both the generator and the discriminator. Fig. 3
(a) and (b) demonstrate an example of the self-attention layer coupling with
the $l$th (de)convolutional layer. As we can see, if we add the self-attention
layer to the $l$th convolutional layer of the encoder, the mirror $l$th
deconvolutional layer of the decoder and the $l$th layer in the discriminator
also couples a self-attention layer. Theoretically, the self-attention layer
can be placed in any number, even all, of the (de)convolutional layers.

###  3.4 FBank Extraction Network

We extract the normalized log FBank features $\bm{\hat{f}}$ as the input of
the subsequent ASR model, which is computed from the enhanced signals
$\bm{\hat{x}}$:

$\bm{\hat{f}}=\text{FBank}(\bm{\hat{x}})=\text{Norm}(\text{log}(\text{Mel}(\text{STFT}(\bm{\hat{x}})))),$
(10)

where STFT($\cdot$) is the operation of short-time Fourier transform (STFT),
Mel($\cdot$) is the operation of Mel matrix multiplication, and Norm($\cdot$)
is for normalizing the mean and variance to 0 and 1, separately. Consequently,
the FBank feature extraction layer is differentiable.

###  3.5 Transformer

####  3.5.1 Multi-head Attention Mechanism

Multi-head attention mechanism [49], as the terminology implies, contains more
than one self-attention module. As the core module of the Transformer [36], it
leverages different attending representations jointly. Before performing each
attention, three linear projections transform the queries, keys, and values to
more discriminated representations, respectively. Afterwards, each dot-product
attention is calculated independently, and their outputs are concatenated and
fed into another linear projection to obtain the final $d_{model}$-dimensional
outputs:

$\text{MultiHead}(\bm{Q},\bm{K},\bm{V})\\\
=\text{Concat}(head_{1},head_{2},\cdots,head_{h})\bm{W}^{OUT},$ (11)

where

$head_{i}=\text{Attention}(\bm{QW}^{Q}_{i},\bm{KW}^{K}_{i},\bm{VW}^{V}_{i}).$
(12)

$h$ refers to the head numbers, and $\bm{Q}$, $\bm{K}$, $\bm{V}$ have the same
dimensions of $d_{model}$. Four projection matrices
$\bm{W}^{Q}_{i}\in\mathbb{R}^{d_{model}\times d_{q}}$,
$\bm{W}^{K}_{i}\in\mathbb{R}^{d_{model}\times d_{k}}$,
$\bm{W}^{V}_{i}\in\mathbb{R}^{d_{model}\times d_{v}}$, and
$\bm{W}^{OUT}\in\mathbb{R}^{hd_{v}\times d_{model}}$. Additionally,
$d_{q}=d_{k}=d_{v}=d_{model}/h$.

####  3.5.2 Positional Encoding

One obvious limitation of the Transformer model is that the output is
invariant to the input order permutation, i.e., the Transformer does not model
the order of the input sequence. Vaswani et al. [49] solve this problem by
injecting information about absolute positions into the input sequence via
sinusoid positional embeddings:

$PE_{(pos,i)}=\left\\{\begin{matrix}sin(pos/10000^{i/d_{model}}))\quad\text{if}\>\,i\>\,\text{is}\>\,\text{even}&\\\
cos(pos/10000^{i/d_{model}}))\quad\text{if}\>\,i\>\,\text{is}\>\,\text{odd}&\end{matrix}\right.,$
(13)

where $pos$ refers to the position and $i$ is the dimension. The sinusoidal
function allows the model to extrapolate from long sequence lengths.

####  3.5.3 Feed-forward Network

The feed-forward network (FFN) is another core module of the Transformer [36].
It is composed of two linear transformations with a ReLU activation in
between. The dimensionality of the input and output is $d_{model}$, and the
inner layer has the dimensionality $d_{ff}$. Specifically,

$\text{FFN}(\bm{x})=\text{max}(0,\bm{x}\bm{W}_{1}+\bm{b}_{1})\bm{W}_{2}+\bm{b}_{2},$
(14)

where the weights $\bm{W}_{1}\in\mathbb{R}^{d_{model}\times d_{ff}}$,
$\bm{W}_{2}\in\mathbb{R}^{d_{ff}\times d_{model}}$ and the biases
$\bm{b}_{1}\in\mathbb{R}^{d_{ff}}$, $\bm{b}_{2}\in\mathbb{R}^{d_{model}}$. The
linear transformations are the same across different positions.

Figure 4: Model architecture of the Transformer. (a) Encoder (b) Decoder [36]

####  3.5.4 Network Architecture

The detailed model architecture of the ASR-Transformer is as follows:

The encoder is shown in Fig. 4 (a). The input-embedding is for extracting
expressive representations of dimension $d_{model}$. Thereafter, to enable the
model to attend on the auxiliary position information, the $d_{model}$-dim
positional encoding (Section 3.5.2) is added to the input encoding. Then the
sum of encoded outputs is fed into a stack of $N_{e}$ encoder blocks, each of
which has two sub-blocks: one is the multi-head attention (Section 3.5.1),
receiving queries, keys, and values from the previous block; the other is the
feed-forward networks (Section 3.5.3). In the meanwhile, layer normalization
and residual connection are introduced to each sub-block for effective
training. Thus, the pipeline of the sub-block is:

$\bm{x}+\text{SubBlock}(\text{Layer\,Norm}(\bm{x})).$ (15)

The decoder is shown in Fig. 4 (b). The output-embedding converts the
character sequence to dimension $d_{model}$. Added with the positional
encoding, the sum of them is fed into a stack of $N_{d}$ decoder blocks, which
consists of three sub-blocks: The first is a masked multi-head attention,
which ensures that the predictions for position $j$ depends only on the known
outputs at positions less than $j$. The second is a multi-head attention whose
keys and values come from the encoder outputs while queries come from the
previous sub-block outputs. The third is also feed-forward networks. Similar
to the encoder, layer normalization and residual connection are also employed
to each sub-block of the decoder. Eventually, the output probabilities are
acquired by a linear projection and a subsequent softmax function.

Figure 5: Illustration of the Conformer encoder model architecture. (i)
Conformer encoder architecture. (ii) Conformer block architecture. (ii-a)
Convolution module of the Conformer block. (ii-b) Multi-headed self-attention
module of Conformer block. (ii-c) Feed forward module of Conformer block.

###  3.6 Conformer

Conformer [37] is a state-of-the-art ASR encoder architecture. Different from
the Transformer block (as described in Section 3.5), it is equipped with a
convolution layer to increase the local information modeling capability of the
Transformer encoder model [49] and a pair of FFN modules sandwiching the
multi-head self-attention module and the integrated convolution module. The
Conformer model consists of a Conformer encoder proposed in [37] and a
Transformer decoder [36]. The encoder first processes the input with a
convolution subsampling layer and then with Conformer blocks, as illustrated
in Fig. 5 (i). The Conformer block (Fig. 5 (ii)) consists of a multi-head
self-attention module (MHSA), a convolution module, sandwiched by a pair of
macron-feedforward module [57]. The layer normalization is applied before each
module and the dropout is followed by a residual connection afterwards (pre-
norm) [58, 59]. Mathematically, let $\bm{x}_{i}$ be the input to the $i$th
Conformer block, the output $\bm{y}_{i}$ of this block is:

$\displaystyle\bm{x}^{\prime}_{i}=\bm{x}_{i}+\frac{1}{2}\text{FFN}(\bm{x}_{i}),$
(16)
$\displaystyle\bm{x}^{\prime\prime}_{i}=\bm{x}^{\prime}_{i}+\text{MHSA}(\bm{x}^{\prime}_{i}),$
(17)
$\displaystyle\bm{x}^{\prime\prime\prime}_{i}=\bm{x}^{\prime\prime}_{i}+\text{Conv}(\bm{x}^{\prime\prime}_{i}),$
(18)
$\displaystyle\bm{y}_{i}=\text{Layer\,Norm}(\bm{x}^{\prime\prime\prime}_{i}+\frac{1}{2}\text{FFN}(\bm{x}^{\prime\prime\prime}_{i})).$
(19)

$\text{FFN}(\cdot)$, $\text{MHSA}(\cdot)$, $\text{Conv}(\cdot)$, and
$\text{Layer\,Norm}(\cdot)$ denote the macron-feedforward module, the multi-
head self-attention module, the convolution module, and the layer
normalization module, respectively. The multi-head self-attention module is
the same as in Section 3.5.1 and is demonstrated in Fig. 5 (ii-b). Section
3.6.1 and 3.6.2 introduce the convolution module and the macron-feedforward
module, respectively.

####  3.6.1 Convolution Module

Fig. 5 (ii-a) demonstrates the details of the convolution module. The
convolution module starts with a 1-dim pointwise convolution layer and a gated
linear units (GLU) activation [60]. The 1-dim pointwise convolution layer
doubles the input channels, and the GLU activation splits the input along the
channel dimension and executes an element-wise product. What follows are a
1-dim depthwise convolution layer, a batch normalization layer, a Swish
activation, and another 1-dim pointwise convolution layer. As mentioned
before, the layer normalization is applied before each module and the dropout
is followed by a residual connection afterwards (pre-norm).

####  3.6.2 Macron-feedforward Module

Unlike the FFN module in Transformer encoder [36], which comprises two linear
transformations with a ReLU activation in between (Equ. 14), Conformer encoder
[37] introduces another FFN module and substitutes the ReLU activation with
the Swish activation. Furthermore, inspired by Macaron-Net [57], this pair of
FFN modules are following a half-step scheme and sandwiching the MHSA and the
convolution modules. The detail of the FFN is illustrated in Fig. 5 (ii-c).

##  4 Adversarial Joint Training

GANs aim at mapping samples $\bm{\hat{x}}$ from the distribution
$\mathcal{\hat{X}}$ to samples $\bm{x}^{*}$ from another distribution
$\mathcal{X^{*}}$. The generator G is tasked to learn an effective mapping
that can imitate the real data distribution to generate novel samples from the
manifold defined in the $\mathcal{X}$, by means of an adversarial training
exerted by the discriminator D. During back-propagation, D classifies real
samples from the fake samples more accurately; in return, G updates its
parameters towards the real data manifold, till the mixed Nash equilibria are
reached [50]. The GAN training process can be formulated as a minimax game
between G and D, with the objective

$\begin{split}\min_{G}\>\max_{D}\>\mathcal{L}(D,G)=&\mathbb{E}_{\bm{x^{*}}\sim
P_{data}(\bm{x^{*}})}[\text{log}\,D(\bm{x^{*}})]+\\\
&\mathbb{E}_{\bm{\hat{x}}\sim
P_{\hat{x}}(\bm{\hat{x}})}[\text{log}(1-D(G((\bm{\hat{x}}))))].\end{split}$
(20)

In our proposed robust end-to-end speech recognition scheme, the discriminant
network first acts as the local guide for the enhancement module, where D
shifts the training of G towards the distribution of clean data; thereafter,
it is deployed as the global guide for the whole scheme, where D instructs G
to output pertinent enhanced data for the subsequent ASR task.

We first train the enhancement module, which contains both the generator and
the discriminator. To solve the problem of vanishing gradients caused by
sigmoid cross-entropy loss for training, the least-squares GAN (LSGAN) with
binary coding (1 for real, 0 for fake) is utilized instead of the cross-
entropy loss. Consequently, the loss function of the discriminator component
changes to

$\begin{split}\min_{D}\>\mathcal{L}(D)=&\frac{1}{2}\mathbb{E}_{\bm{x^{*}},\bm{\tilde{x}}\sim
P_{data}(\bm{x^{*}},\bm{\tilde{x}})}[D(\bm{x^{*}},\bm{\tilde{x}})-1]^{2}+\\\
&\frac{1}{2}\mathbb{E}_{\bm{z}\sim p_{\bm{z}}(\bm{z}),\bm{\tilde{x}}\sim
p_{data}(\bm{\tilde{x})}}[D(G(\bm{z},\bm{\tilde{x}}),\bm{\tilde{x}})]^{2},\end{split}$
(21)

where $\bm{z}$ is a latent variable. To minimize the distance between its
generations and the clean examples, it is beneficial to add a secondary
component to the loss of G. Inspired by the effectiveness of $L_{1}$ norm in
the image manipulation domain [61, 62], we deploy it in G component to gain
more fine-grained and realistic results. The magnitude of the $L_{1}$ norm is
controlled by a new hyper-parameter $\lambda$. Hence, the loss function of the
generator component becomes

$\begin{split}\min_{G}\>\mathcal{L}(G)=&\frac{1}{2}\mathbb{E}_{\bm{z}\sim
p_{\bm{z}}(\bm{z}),\bm{\tilde{x}}\sim
p_{data}(\bm{\tilde{x})}}[D(G(\bm{z},\bm{\tilde{x}}),\bm{\tilde{x}})-1]^{2}\\\
&+\lambda\left\|G(\bm{z},\bm{\tilde{x}})-\bm{x^{*}}\right\|_{1}.\end{split}$
(22)

In the joint training, the enhancement module is initialized from the trained
G component, while the global discriminant module is initialized from the
trained D component. The training of the ASR component is based on the cross
entropy criterion, namely

$\mathcal{L}_{asr}=-\text{ln}P(Y^{*}|F)=-\sum_{n}\text{ln}P(y_{n}^{*}|F,y_{1:n-1}^{*}),$
(23)

where $Y^{*}$ is the ground truth of a whole sequence of output labels and
$y_{1:n-1}^{*}$ is the ground truth from output step 1 to $n-1$. In the
proposed framework, the parameters of all procedures, enhancement, feature
extraction, ASR, and the discriminant network, are updated by stochastic
gradient descent calculated by the loss function of the whole scheme. It is
composed of three losses: $\mathcal{L}_{asr}$, $\mathcal{L}_{enh}$, and
$\mathcal{L}_{gan}$, which correspond to Eqs. 23, 22, and 21, i.e.

$\mathcal{L}=\mathcal{L}_{asr}+\kappa\mathcal{L}_{enh}+\gamma\mathcal{L}_{gan},$
(24)

where $\kappa$ and $\gamma$ are two hyper-parameters weighting the magnitude
of the enhancement loss and adversarial loss. Notably, the scheme targets the
recognition performance, and the loss function of the discriminant network
adapts the enhancement module implicitly. As a result, the discriminant
network guides the enhancement module to serve the subsequent ASR task more
properly. Accordingly, the unnecessary speech distortion caused by the
enhancement process is alleviated.

##  5 Experimental Setups

We systematically evaluate the robustness of the adversarial joint training
framework, and ablation tests are conducted to validate the effects of (i) the
enhancement front-end on the ASR task, (ii) the joint training on the whole
scheme, and (iii) the GAN on the joint training.

###  5.1 Corpus

All experiments are executed on the open source Mandarin speech corpus,
AISHELL-1 [63]. This corpus is 178h long, and its utterances contain 11
domains, e.g., smart home, autonomous driving, industrial production, etc. 400
speakers from different accent areas in China participate in the recording.
The corpus is divided into training, development, and test sets. The training
dataset contains 120,098 utterances from 340 speakers, the development dataset
contains 14,326 utterances from 40 speakers, and the test dataset contains
7,176 utterances from 20 speakers.

For the noisy data, we contaminate clean utterances in AISHELL-1 with 9 sorts
of intrusions from the NOISEX-92 dataset [64] artificially as noisy
utterances. We create noisy training, development, and test sets in the same
manner. Note that besides the “matched” noisy test set, which is contaminated
by the same intrusions as the training dataset, we also corrupt the test set
with the rest 5 sorts of intrusions in the NOISEX-92 dataset as “unmatched”
test materials. Table 1 exhibits the sorts the intrusions mixed in “match” and
“unmatch” cases. All utterances are mixed with the intrusions at SNRs randomly
sampled between [0dB, 20dB]. To sum up, we have two sorts of datasets for
training:

  * •

clean: Clean utterances from the training dataset of AISHELL-1.

  * •

match: Contaminated clean utterances (training dataset) with “matched” noises
of Table 1.

For test datasets, we have:

  * •

clean: clean utterances from the test dataset of AISHELL-1.

  * •

match: Contaminated clean utterances (test dataset) with “matched” noises of
Table 1.

  * •

unmatch: Contaminated clean utterances (test dataset) with “unmatched” noises
of Table 1.

###  5.2 Baseline

For the comparison purpose, we take the work from [28] as the baseline model.

In [28], the mask-based enhancement network is deployed as the front-end. It
estimates a masking function to multiply the frequency-domain feature of the
noisy speech to form an estimate of the clean speech. For the ASR task, Liu et
al. employ the ESPnet model [65]. It consists of an encoder network that maps
the input feature sequence into a higher-level representation. Then a
location-based attention layer integrates the representation into a context
vector with the attention weight vector. In the end, the decoder predicts the
next output conditioned on the full sequence of previous predictions. Besides,
there is an extra discriminant network, whose loss is weighted in the loss
function of the whole scheme to optimize the joint training.

Importantly, the baseline model does not contain any self-attention layer.
Furthermore, the discriminant work in the baseline model is an extra auxiliary
module, which does not participate in the enhancement training directly. By
contrast, our work benefit from self-attention mechanism and the discriminant
module exits innately, which is a component of the enhancement front-end. It
acts as the local guide for the enhancement training, leading the enhancement
network to output towards the distribution of the clean samples.
Simultaneously, it also plays the role of the global guide, instructing the
enhancement module and the ASR module better matched.

Table 1: The demonstration of categories of intrusions utilized in “match” and
“unmatch” cases.  match
---
Intrusion | Description
White Noise | Analog noise generator
Factory Floor Noise 1 | Plate-cutting and electrical welding
Cockpit Noise 1 | Buccaneer jet traveling at 190 knots
Cockpit Noise 3 | F-16
Engine Room Noise | Destroyer
Military Vehicle Noise | Leopard 1 vehicle
Machine Gun Noise | Gun
Vehicle Interior Noise | Volvo 340
HF Channel Noise | HF radio channel
unmatch
Intrusion | Description
Pink Noise | Analog noise generator
Factory Floor Noise 2 | Car production hall
Cockpit Noise 2 | Buccaneer jet traveling at 450 knots
Operations Room Background Noise | Destroyer
Military Vehicle Noise | M109

###  5.3 Configurations

####  5.3.1 Baseline

For the enhancement front-end, the input is the 257-dim logarithmic STFT
features, and all input vectors are normalized to have the zero mean and the
unit variance. The network is composed of 3-layer long short-term memory
(LSTM) with 128 nodes, followed by a linear layer with the sigmoid activation
function. The network outputs the masking estimate, whose size is equal to the
input size, multiplying by the STFT feature of the noisy speech to estimate
the clean speech.

For the ASR network, the input is the 80-dim normalized log FBank features
transformed from the enhanced STFT features. The encoder is composed of
4-layer bidirectional LSTM (BLSTM) with 320 cells, while the decoder is
composed of 1-layer unidirectional LSTM with 320 cells. After each BLSTM
layer, a linear projection layer with 320 nodes is used to combine the forward
and backward LSTM outputs. The location-based attention mechanism comprises 10
centered convolution filters of width 100. Besides, We also adopt a joint
connectionist temporal classification (CTC)-attention multitask loss function
[66] with the CTC loss weight as 0.1.

The discriminant network consists of a 4-layer convolution network, each of
which is followed by the ReLU activation function [67].

For decoding, we use a beam search algorithm with the beam size 12. CTC
rescores the hypotheses with 0.1 weight [66]. Besides, an external recurrent
neural network (RNN) language model is also adopted with 0.2 weight during
decoding.

####  5.3.2 The Proposed Joint Training Scheme

##### SA_SEGAN

The SA_SEGAN is trained for 86 epochs with RMSprop [68] and a learning rate of
0.0002. The batch size is 50. During training, we extract 1-second chunks of
raw waveforms ($L=16,384$ samples) with a 50% overlap. During the test, we
slide the window without overlapping through the whole duration of our test
utterances and concatenate the outputs at the end of the stream. During both
training and test, we employ a high-frequency preemphasis filter with a
coefficient of 0.95 to all inputs. For the self-attention layer in SA_SEGAN,
we use $b=8$ and $p=4$ for memory reduction. Phan et al. [40] suggest that the
placement of the self-attention layer does not show a clear difference on the
performance, which indicates that applying the self-attention to the higher-
level (de)convolutional layer is expected to be as good as to a lower layer.
Compromising between the computation time and memory requirement and the
performance, we place the self-attention layer in the 10th layer ($l$=10).

##### FBank extraction network

The FBank feature extraction network is a linear layer to transform the raw
outputs from the upstream SA_SEGAN to the downstream ASR procedure. We extract
80-dim filterbanks with the window size of 25ms and the window shift of 10ms,
extended with the temporal first- and second-order differences. Thereafter, we
do the logarithmic calculation and global mean and variance normalization
according to Eq. 10.

##### Transformer

For training the Transformer, we adopt Adam optimizer [69] with
$\beta_{1}=0.9$, $\beta_{2}=0.98$, $\epsilon=10^{-9}$, and vary the learning
rate over the course of training according to the formula:

$\displaystyle lr=k^{\prime}\cdot
d^{-0.5}_{model}\cdot\text{min}(n^{-0.5},n\times warmup_{n}^{-1.5})$ (25)

where $n$ denotes the step number. $k^{\prime}$ is a tunable scalar, which is
set to be 10 initially and is declined to 1 when the model converges. The
learning rate increases linearly during the fist $warmup_{n}=25000$ steps, and
afterwards, it decreases proportionally to the inverse square root of the step
number. We apply the residual dropout to each sub-block before adding the
residual information, while the attention dropout is performed on the softmax
activations in each attention. Both of these aforementioned dropouts are set
to be 0.1. Additionally, we guide the system to be more attentive on closer
positions by punishing the attention weights of more distant position-pairs.
Similar to the baseline model, we also adopt a joint CTC-attention multi-task
loss function [66], with the CTC loss weight as 0.3. In the decoding, we set
the beam size to 12 and length penalty $\alpha$ = 1.0 [70]. Besides, we also
integrate an external RNN language model with 0.3 weight. The training
procedure is stopped after 30 epochs.

##### Conformer

The model hyper-parameters of the Conformer are: $N_{e}$=12, $N_{d}$=6, $H$=4,
$d_{k}$=256 and $d_{ff}$=2048. The convolution subsampling layer possesses a
2-layer convolutional neural network (CNN) with 256 channels, stride with 2,
and kernel size with 3. The kernel size of the convolution module is 31. We
apply dropout in each residual unit of the Conformer with the weight 0.1. The
same as the Transformer, we train the network with Adam optimizer [69] with
$\beta_{1}=0.9$, $\beta_{2}=0.98$, $\epsilon=10^{-9}$ and a Transformer
learning rate schedule [49] with 10000 warm-up steps. The learning rate is
peaked at $0.05/\sqrt{d}$, where $d$ is the model dimension in the Conformer
encoder. Note that we do not apply speed perturbation [71] or SpecAugment [72]
for the data augmentation to exclude extra tricks that could cause performance
improvements. The training procedure is stopped after 30 epochs.

##  6 Results

We use character error rate (CER) to quantify the system performance in all
experiments. We report CER of the AISHELL-1 test set on three conditions:
“clean” refers to the original clean test dataset of the corpus, “match”
denotes the noisy test dataset contaminated by “matched” sorts of intrusions
in Table 1, and “unmatch” means the noisy test set corrupted by the rest of
“unmatched” sorts of intrusions in Table 1. To validate the efficacy of the
enhancement front-end, we also introduce multi-condition training (MCT) for
comparison, where we artificially contaminate the training dataset of
AISHELL-1 with background noise at a certain SNR. Note that there are three
randomizations: (i) The utterance to be corrupted is chosen randomly, and in
total 90% of the training utterances are corrupted; (ii) The background noise
is chosen from the “matched” intrusions from Table 1 randomly; (iii) The SNR
is sampled randomly between [0dB, 20dB]. Therefore, MCT training data comprise
10% original clean data and 90% contaminated data, which is corrupted by one
of the “matched” noises at an SNR in the range of [0dB, 20dB].

Firstly, we train the ASR network with the original clean utterance and multi-
condition training strategy. The results are shown in Table 2.

Table 2: CER[%] results of ASR system trained by clean data and multi-condition training (MCT) without the enhancement. ASR | Training | CER[%]
---|---|---
clean | match | unmatch
baseline | clean | 14.0 | 60.0 | 61.6
MCT | 14.6 | 20.8 | 27.3
Transformer | clean | 8.0 | 35.6 | 37.1
MCT | 7.8 | 13.1 | 16.2
Conformer | clean | 6.5 | 32.6 | 31.7
MCT | 6.9 | 12.1 | 13.7

Ranking these three models from the aspect of ASR performance, the first is
Conformer, then Transformer, and the last is the baseline model, consistent
with observations in [36, 37]. However, their performance deteriorates rapidly
in the noisy test set, demonstrating the necessity of the robustness
investigation. The MCT training considerately improves the system’s
robustness. Its performance on the “matched” test set outperforms the clean
training by 63.2% and 62.9% relative, in cases of Transformer and Conformer
model respectively; while on the “unmatched” dataset, it outperforms the clean
training by 56.3% and 56.8% relative, in cases of Transformer and Conformer
model respectively.

Table 3: The impacts of the enhancement front-end on ASR systems trained by clean data and multi-condition training (MCT). Results are in CER[%]. ASR | Training | CER[%]
---|---|---
clean | match | unmatch
baseline | clean | 13.9 | 25.8 | 53.8
MCT | 14.9 | 23.5 | 34.3
Transformer | clean | 8.1 | 19.1 | 33.7
MCT | 7.9 | 14.9 | 20.7
Conformer | clean | 6.5 | 17.6 | 28.8
MCT | 7.0 | 14.2 | 17.9
Table 4: CER[%] results of the SE_ASR system retraining with and without noisy features.  ASR | Retraining | CER[%]
---|---|---
clean | match | unmatch
Transformer_MCT | no | 7.8 | 13.1 | 16.2
enhanced | 7.8 | 12.9 | 15.8
enhanced+noisy | 7.8 | 12.9 | 15.6
Conformer_MCT | no | 6.9 | 12.1 | 13.7
enhanced | 6.7 | 12.0 | 13.5
enhanced+noisy | 6.7 | 11.8 | 13.3
Table 5: The impacts of the joint training with and without GAN on SA-ASR pipeline. Results are in CER[%] SE | ASR | joint training with GANs | CER[%]
---|---|---|---
clean | match | unmatch
SA_SEGAN | baseline | no | 12.8 | 18.7 | 25.3
yes | 12.8 | 18.7 | 24.8
Transformer | no | 7.0 | 12.4 | 15.6
yes | 7.2 | 12.4 | 15.5
Conformer | no | 6.8 | 11.9 | 13.3
yes | 6.9 | 11.8 | 13.0

Secondly, we train SA_SEGAN with the training data contaminated by “matched”
intrusions in Table 1 to enhance the noisy speech. Then the enhanced features
are used for the downstream ASR task. Importantly, the ASR models are taken
over from the same well-trained model as in Table 2, which means that the
enhancement front-end and the ASR back-end are trained separately by different
objectives. As exhibited in Table 3, the enhancement module tremendously
improves the performance of the ASR component, which is trained by the clean
data merely. Compared to Table 2, it outperforms all of the three ASR modules
(baseline, Transformer, Conformer) without the enhancement front-end. The
improvement achieved in the “matched” dataset is more remarkable than that
achieved on the “unmatched” test set. For instance, it outperforms the
Conformer without the enhancement module by 46.0% in the “matched” test set
while 9.1% in the “unmatched” test set. This difference is due to that the
SA_SEGAN is trained with the “matched” intrusions and can enhance the data
contaminated by the same intrusions better during the test. All these
improvements confirm the efficacy of the enhancement module for improving the
robustness of the ASR system. Nevertheless, improving the robustness of the
framework in unseen noisy environments still remains to be a challenge.
Additionally, the speech enhancement module deteriorates the performer of the
ASR_MCT network, which stays in accordance with the observations in [35]. This
degradation may be derived from the latent distortions caused by the
overtraining of the enhancement module.

To remedy the deterioration of the performance of the ASR_MCT, we retrain the
network with the enhanced features. Assuming that the network may also benefit
from the knowledge of the noisy features, we also experiment with ingesting
both enhanced and noisy features. Results are displayed in Table 4. Either the
Transformer_MCT model or the Conformer_MCT model is initialized from the
existing well-trained MCT checkpoints respectively, setting the additional
parameters to zero to ensure the fair training start. As presented in Table 4,
the retraining with the enhanced features improves the performance in both
“matched” and “unmatched” cases, and the retraining with both enhanced and
noisy features improves the performance slightly further.

Lastly, we jointly train the whole scheme with and without adversarial
training according to Eq. 24. In the framework, the enhancement front-end is
initialized from the generator (G component) of SA_SEGAN, the ASR back-end is
initialized from the ASR_MCT checkpoint (without retraining), and the
adversarial module is initialized from the discriminator (D component) of
SA_SEGAN. When the adversarial module participates in the training, we set the
magnitude of the loss function by $\kappa$=6.0 and $\gamma$=0; by contrast,
when it participates in the training, we set $\kappa$=6.0 and $\gamma$=3.0.
Results are presented in Table 5. Compared to Table 2, the joint training
mitigates the distortion problem existing in the MCT strategy. Additionally,
the participance of the adversarial training improves the performance further;
and exceeds the performance of retraining with both enhanced and noisy
features in either Transformer or Conformer case. Taking Conformer for
example, compared to Conformer trained with clean data merely, the adversarial
joint training yields 63.8% relative and 59.0% relative improvements on
“matched” and “unmatched” datasets, respectively; meanwhile, the adversarial
joint training outperforms the MCT strategy by 2.5% relative and 5.2% relative
on “matched” and “unmatched” datasets, separately. These results indicate the
efficacy of the adversarial joint training in improving the robustness of the
end-to-end ASR scheme.

Figure 6: Comparisons of different enhancement models’ performance under
different training conditions.

##  7 Discussion

To analyse the difference between these enhancement modules that are trained
independently, jointly without GANs, and jointly with GANs, we quantify their
performance on the following five objective criteria (the higher the better):

  * •

SSNR: Segmental SNR [23] (in the range of [0 , $+\infty$))

  * •

CBAK: MOS prediction of the intrusiveness of background noises [22] (in the
range of [1 , 5])

  * •

CSIG: MOS prediction of the signal distortion attending only to the speech
signal [22] (in the range of [1 , 5])

  * •

COVL: MOS prediction of the overall effect [22] (in the range of [1 , 5])

  * •

PESQ: Perceptual evaluation of speech quality, using the wide-band version
recommended in ITU-T P.862.2 [73] (in the range of [-0.5 , 4.5])

All criteria are computed based on the implementation in [74], available at
the publisher website111https://www.crcpress.com/downloads/K14513/
K14513_CD_Files.zip. We quantify the performance of the enhancement front-end
that is trained independently, trained jointly with and without GANs in case
of Transformer scheme. As exhibited in Fig. 6, the joint training disgrades
the enhancement module’s performance on SSNR, CBAK, COVL, and PESQ, except for
CSIG. It is safe to draw two conclusions from this result. First, these
results suggest that these objective criteria cannot indicate the suitability
of the enhanced data for ASR task, which verifies the assertion that the
independent training leads the enhancement module into the sub-optimum easily.
Second, the opposite trend on CSIG proves the assumption that the joint
training strategy can mitigate the unseen distortion introduced by the
handcrafted loss function. Another phenomenon which is worth nothing is that
the discrepancies on CBAK and SSNR reveals the conflicts between erasing the
noise contamination and averting the speech distortion. Therefore, the
equilibrium between these two goals is critical. The experimental results in
Section 6 validate the efficacy of the adversarial joint training with a
global discriminant guide for reaching the equilibrium point.

##  8 Conclusion

In this paper, we propose an adversarial joint training framework with the
self-attention mechanism to boost the noise robustness of the end-to-end ASR
system. The jointly compositional scheme consists of an enhancement front-end,
a recognition back-end, and the discriminant network. A highlight of this
proposed framework is the discriminant component first acts as the guide of
the enhancement front-end training; afterwards, it participates in the
adversarial joint training as the global instructor, which leads the
enhancement front-end to output appropriate enhanced features for the
downstream ASR task. Experimental results validate the efficacy of the
proposed adversarial joint training strategy. The next work plan is to
investigate different framework architectures and training strategies for
further improved performance.

## Abbreviations

ASR: automatic speech recognition; SA_ASR: sela-attention automatic speech
recognition; GANs: generative adversarial networks; LSGAN: least-squares
generative adversarial networks; SE: speech enhancement; SEGAN: speech
enhancement generative adversarial networks; SA_SEGAN: self-attention speech
enhancement generative adversarial networks; CER: character error rate;
PReLUs: parametric rectified linear units; FFN: feed-forward network; MHSA:
multi-head self-attention module; GLU: gated linear units; STFT: short-time
Fourier transform; LSTM: long short-term memory; BLSTM: bidirectional long
short-term memory; CTC: connectionist temporal classification; RNN: recurrent
neural network; CNN: convolutional neural network; MCT: multi-conditional
training.

## Availability of data and materials

The dataset is the open source Mandarin speech corpus, AISHELL-1 [63] and can
be found under the following link: http://www.aishelltech.com/kysjcp.

## Competing interests

The authors declare that the research was conducted in the absence of any
commercial or financial relationship that could be construed as potential
competing interests.

## Authors’ contributions

Li, L. conceptualised the study. Li, L., Kang, Y., and Shi, Y. executed the
experiments. All authors did literature analysis, manuscript preparation,
editing, and proofreading, and approved the final manuscript.

## References

  * [1] Chan, W., Jaitly, N., Le, Q.V., Vinyals, O.: Listen, attend and spell. arXiv preprint arXiv:1508.01211 (2015)
  * [2] Chorowski, J.K., Bahdanau, D., Serdyuk, D., Cho, K., Bengio, Y.: Attention-based models for speech recognition. Advances in neural information processing systems 28, 577–585 (2015)
  * [3] Hinton, G., Deng, L., Yu, D., Dahl, G.E., Mohamed, A.-r., Jaitly, N., Senior, A., Vanhoucke, V., Nguyen, P., Sainath, T.N., et al.: Deep neural networks for acoustic modeling in speech recognition: The shared views of four research groups. IEEE Signal processing magazine 29(6), 82–97 (2012)
  * [4] Chiu, C.-C., Sainath, T.N., Wu, Y., Prabhavalkar, R., Nguyen, P., Chen, Z., Kannan, A., Weiss, R.J., Rao, K., Gonina, E., et al.: State-of-the-art speech recognition with sequence-to-sequence models. In: 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 4774–4778 (2018). IEEE
  * [5] Povey, D., Hadian, H., Ghahremani, P., Li, K., Khudanpur, S.: A time-restricted self-attention layer for asr. In: 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 5874–5878 (2018). IEEE
  * [6] Tian, Z., Yi, J., Tao, J., Bai, Y., Wen, Z.: Self-attention transducers for end-to-end speech recognition. arXiv preprint arXiv:1909.13037 (2019)
  * [7] Salazar, J., Kirchhoff, K., Huang, Z.: Self-attention networks for connectionist temporal classification in speech recognition. In: ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 7115–7119 (2019). IEEE
  * [8] Han, K.J., Huang, J., Tang, Y., He, X., Zhou, B.: Multi-stride self-attention for speech recognition. In: INTERSPEECH, pp. 2788–2792 (2019)
  * [9] Han, K.J., Prieto, R., Ma, T.: State-of-the-art speech recognition using multi-stream self-attention with dilated 1d convolutions. In: 2019 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU), pp. 54–61 (2019). IEEE
  * [10] Pham, N.-Q., Nguyen, T.-S., Niehues, J., Müller, M., Stüker, S., Waibel, A.: Very deep self-attention networks for end-to-end speech recognition. arXiv preprint arXiv:1904.13377 (2019)
  * [11] Yeh, C.-F., Mahadeokar, J., Kalgaonkar, K., Wang, Y., Le, D., Jain, M., Schubert, K., Fuegen, C., Seltzer, M.L.: Transformer-transducer: End-to-end speech recognition with self-attention. arXiv preprint arXiv:1910.12977 (2019)
  * [12] Luo, H., Zhang, S., Lei, M., Xie, L.: Simplified self-attention for transformer-based end-to-end speech recognition. arXiv preprint arXiv:2005.10463 (2020)
  * [13] Lim, J., Oppenheim, A.: All-pole modeling of degraded speech. IEEE Transactions on Acoustics, Speech, and Signal Processing 26(3), 197–210 (1978)
  * [14] Narayanan, A., Wang, D.: Ideal ratio mask estimation using deep neural networks for robust speech recognition. In: 2013 IEEE International Conference on Acoustics, Speech and Signal Processing, pp. 7092–7096 (2013). IEEE
  * [15] Wang, Y., Narayanan, A., Wang, D.: On training targets for supervised speech separation. IEEE/ACM transactions on audio, speech, and language processing 22(12), 1849–1858 (2014)
  * [16] Nie, S., Liang, S., Xue, W., Zhang, X., Liu, W., et al.: Two-stage multi-target joint learning for monaural speech separation. In: INTERSPEECH, pp. 1503–1507 (2015)
  * [17] Weninger, F., Hershey, J.R., Le Roux, J., Schuller, B.: Discriminatively trained recurrent neural networks for single-channel speech separation. In: 2014 IEEE Global Conference on Signal and Information Processing (GlobalSIP), pp. 577–581 (2014). IEEE
  * [18] Erdogan, H., Hershey, J.R., Watanabe, S., Le Roux, J.: Phase-sensitive and recognition-boosted speech separation using deep recurrent neural networks. In: 2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 708–712 (2015). IEEE
  * [19] Xu, Y., Du, J., Dai, L.-R., Lee, C.-H.: A regression approach to speech enhancement based on deep neural networks. IEEE/ACM Transactions on Audio, Speech, and Language Processing 23(1), 7–19 (2014)
  * [20] Nie, S., Liang, S., Liu, W., Zhang, X., Tao, J.: Deep learning based speech separation via nmf-style reconstructions. IEEE/ACM Transactions on Audio, Speech, and Language Processing 26(11), 2043–2055 (2018)
  * [21] Ephraim, Y.: A minimum mean square error approach for speech enhancement. In: International Conference on Acoustics, Speech, and Signal Processing, pp. 829–832 (1990). IEEE
  * [22] Hu, Y., Loizou, P.C.: Evaluation of objective quality measures for speech enhancement. IEEE Transactions on audio, speech, and language processing 16(1), 229–238 (2007)
  * [23] Quackenbush, S.R.: Objective measures of speech quality. PhD thesis, Georgia Institute of Technology (1995)
  * [24] Seltzer, M.L.: Bridging the gap: Towards a unified framework for hands-free speech recognition using microphone arrays. In: 2008 Hands-Free Speech Communication and Microphone Arrays, pp. 104–107 (2008). IEEE
  * [25] Wang, Z.-Q., Wang, D.: A joint training framework for robust automatic speech recognition. IEEE/ACM Transactions on Audio, Speech, and Language Processing 24(4), 796–806 (2016)
  * [26] Wang, Z.-q., Wang, D.: Joint training of speech separation, filterbank and acoustic model for robust automatic speech recognition. In: INTERSPEECH (2015)
  * [27] Ochiai, T., Watanabe, S., Hori, T., Hershey, J.R.: Multichannel end-to-end speech recognition. arXiv preprint arXiv:1703.04783 (2017)
  * [28] Bin, L., Shuai, N., Shan, L., Wenju, L., Meng, Y., Lianwu, C., Shouye, P., Changliang, L., et al.: Jointly adversarial enhancement training for robust end-to-end speech recognition. (2019). ISCA
  * [29] Pascual, S., Bonafonte, A., Serra, J.: Segan: Speech enhancement generative adversarial network. arXiv preprint arXiv:1703.09452 (2017)
  * [30] Soni, M.H., Shah, N., Patil, H.A.: Time-frequency masking-based speech enhancement using generative adversarial network. In: 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 5039–5043 (2018). IEEE
  * [31] Michelsanti, D., Tan, Z.-H.: Conditional generative adversarial networks for speech enhancement and noise-robust speaker verification. arXiv preprint arXiv:1709.01703 (2017)
  * [32] Shen, P., Lu, X., Li, S., Kawai, H.: Conditional generative adversarial nets classifier for spoken language identification. In: INTERSPEECH, pp. 2814–2818 (2017)
  * [33] Sahu, S., Gupta, R., Espy-Wilson, C.: On enhancing speech emotion recognition using generative adversarial networks. arXiv preprint arXiv:1806.06626 (2018)
  * [34] Hu, H., Tan, T., Qian, Y.: Generative adversarial networks based data augmentation for noise robust speech recognition. In: 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 5044–5048 (2018)
  * [35] Donahue, C., Li, B., Prabhavalkar, R.: Exploring speech enhancement with generative adversarial networks for robust speech recognition. In: 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 5024–5028 (2018). IEEE
  * [36] Dong, L., Xu, S., Xu, B.: Speech-transformer: a no-recurrence sequence-to-sequence model for speech recognition. In: 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 5884–5888 (2018). IEEE
  * [37] Gulati, A., Qin, J., Chiu, C.-C., Parmar, N., Zhang, Y., Yu, J., Han, W., Wang, S., Zhang, Z., Wu, Y., et al.: Conformer: Convolution-augmented transformer for speech recognition. arXiv preprint arXiv:2005.08100 (2020)
  * [38] Phan, H., McLoughlin, I.V., Pham, L., Chén, O.Y., Koch, P., De Vos, M., Mertins, A.: Improving gans for speech enhancement. IEEE Signal Processing Letters 27, 1700–1704 (2020)
  * [39] Baby, D.: iSEGAN: Improved Speech Enhancement Generative Adversarial Networks (2020)
  * [40] Phan, H., Nguyen, H.L., Chén, O.Y., Koch, P., Duong, N.Q., McLoughlin, I., Mertins, A.: Self-attention generative adversarial network for speech enhancement. arXiv preprint arXiv:2010.09132 (2020)
  * [41] Koizumi, Y., Yatabe, K., Delcroix, M., Masuyama, Y., Takeuchi, D.: Speech enhancement using self-adaptation and multi-head self-attention. In: 2020 IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP), pp. 181–185 (2020)
  * [42] Sriram, A., Jun, H., Gaur, Y., Satheesh, S.: Robust speech recognition using generative adversarial networks. In: 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 5639–5643 (2018)
  * [43] Wang, K., Zhang, J., Sun, S., Wang, Y., Xiang, F., Xie, L.: Investigating generative adversarial networks based speech dereverberation for robust speech recognition. Interspeech 2018 (2018)
  * [44] Liu, B., Nie, S., Zhang, Y., Ke, D., Liang, S., Liu, W.: Boosting noise robustness of acoustic model via deep adversarial training. In: 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 5034–5038 (2018)
  * [45] Droppo, J., Acero, A.: Joint discriminative front end and back end training for improved speech recognition accuracy. In: 2006 IEEE International Conference on Acoustics Speech and Signal Processing Proceedings, vol. 1, p. (2006)
  * [46] Gao, T., Du, J., Dai, L., Lee, C.: Joint training of front-end and back-end deep neural networks for robust speech recognition. In: 2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 4375–4379 (2015)
  * [47] Ravanelli, M., Brakel, P., Omologo, M., Bengio, Y.: Batch-normalized joint training for dnn-based distant speech recognition. In: 2016 IEEE Spoken Language Technology Workshop (SLT), pp. 28–34 (2016)
  * [48] Qian, Y., Tan, T., Yu, D.: Neural network based multi-factor aware joint training for robust speech recognition. IEEE/ACM Transactions on Audio, Speech, and Language Processing 24(12), 2231–2240 (2016)
  * [49] Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, Ł., Polosukhin, I.: Attention is all you need. Advances in neural information processing systems 30, 5998–6008 (2017)
  * [50] Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., Bengio, Y.: Generative adversarial nets. Advances in neural information processing systems 27, 2672–2680 (2014)
  * [51] Wang, X., Girshick, R., Gupta, A., He, K.: Non-local neural networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 7794–7803 (2018)
  * [52] Zhang, H., Goodfellow, I., Metaxas, D., Odena, A.: Self-attention generative adversarial networks. In: International Conference on Machine Learning, pp. 7354–7363 (2019). PMLR
  * [53] Radford, A., Metz, L., Chintala, S.: Unsupervised representation learning with deep convolutional generative adversarial networks. arXiv preprint arXiv:1511.06434 (2015)
  * [54] He, K., Zhang, X., Ren, S., Sun, J.: Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 1026–1034 (2015)
  * [55] Salimans, T., Goodfellow, I., Zaremba, W., Cheung, V., Radford, A., Chen, X.: Improved techniques for training gans. arXiv preprint arXiv:1606.03498 (2016)
  * [56] Maas, A.L., Hannun, A.Y., Ng, A.Y.: Rectifier nonlinearities improve neural network acoustic models. In: Proc. Icml, vol. 30, p. 3 (2013)
  * [57] Lu, Y., Li, Z., He, D., Sun, Z., Dong, B., Qin, T., Wang, L., Liu, T.-Y.: Understanding and improving transformer from a multi-particle dynamic system point of view. arXiv preprint arXiv:1906.02762 (2019)
  * [58] Zeyer, A., Bahar, P., Irie, K., Schlüter, R., Ney, H.: A comparison of transformer and lstm encoder decoder models for asr. In: 2019 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU), pp. 8–15 (2019). IEEE
  * [59] Wang, Q., Li, B., Xiao, T., Zhu, J., Li, C., Wong, D.F., Chao, L.S.: Learning deep transformer models for machine translation. arXiv preprint arXiv:1906.01787 (2019)
  * [60] Dauphin, Y.N., Fan, A., Auli, M., Grangier, D.: Language modeling with gated convolutional networks. In: International Conference on Machine Learning, pp. 933–941 (2017). PMLR
  * [61] Isola, P., Zhu, J.-Y., Zhou, T., Efros, A.A.: Image-to-image translation with conditional adversarial networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1125–1134 (2017)
  * [62] Pathak, D., Krahenbuhl, P., Donahue, J., Darrell, T., Efros, A.A.: Context encoders: Feature learning by inpainting. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2536–2544 (2016)
  * [63] Bu, H., Du, J., Na, X., Wu, B., Zheng, H.: Aishell-1: An open-source mandarin speech corpus and a speech recognition baseline. In: 2017 20th Conference of the Oriental Chapter of the International Coordinating Committee on Speech Databases and Speech I/O Systems and Assessment (O-COCOSDA), pp. 1–5 (2017)
  * [64] Varga, A., Steeneken, H., Jones, D.: The noisex-92 study on the effect of additive noise on automatic speech recognition system. Reports of NATO Research Study Group (RSG. 10) (1992)
  * [65] Watanabe, S., Hori, T., Karita, S., Hayashi, T., Nishitoba, J., Unno, Y., Enrique Yalta Soplin, N., Heymann, J., Wiesner, M., Chen, N., Renduchintala, A., Ochiai, T.: ESPnet: End-to-end speech processing toolkit. In: Proceedings of Interspeech, pp. 2207–2211 (2018)
  * [66] Kim, S., Hori, T., Watanabe, S.: Joint ctc-attention based end-to-end speech recognition using multi-task learning. In: 2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 4835–4839 (2017). IEEE
  * [67] Nair, V., Hinton, G.E.: Rectified linear units improve restricted boltzmann machines. In: ICML (2010)
  * [68] Tieleman, T., Hinton, G.: Lecture 6.5-rmsprop: Divide the gradient by a running average of its recent magnitude. COURSERA: Neural networks for machine learning 4(2), 26–31 (2012)
  * [69] Kingma, D.P., Ba, J.: Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014)
  * [70] Wu, Y., Schuster, M., Chen, Z., Le, Q.V., Norouzi, M., Macherey, W., Krikun, M., Cao, Y., Gao, Q., Macherey, K., et al.: Google’s neural machine translation system: Bridging the gap between human and machine translation. arXiv preprint arXiv:1609.08144 (2016)
  * [71] Ko, T., Peddinti, V., Povey, D., Khudanpur, S.: Audio augmentation for speech recognition. In: Sixteenth Annual Conference of the International Speech Communication Association (2015)
  * [72] Park, D.S., Chan, W., Zhang, Y., Chiu, C.-C., Zoph, B., Cubuk, E.D., Le, Q.V.: Specaugment: A simple data augmentation method for automatic speech recognition. arXiv preprint arXiv:1904.08779 (2019)
  * [73] Rec, I.: P. 862.2: Wideband extension to recommendation p. 862 for the assessment of wideband telephone networks and speech codecs. International Telecommunication Union, CH–Geneva (2005)
  * [74] Loizou, P.C.: Speech Enhancement: Theory and Practice. CRC press, Boca Raton, FL, USA (2013)


# The $C_{2^{n}}$ Borel dual Steenrod algebra

Nick Georgakopoulos

###### Abstract.

In this very short note, we expand the Hu-Kriz computation of the
$G$-equivariant Borel dual Steenrod algebra in characteristic $2$, from the
group $G=C_{2}$ to all power-$2$ cyclic groups $G=C_{2^{n}}$.

###### Contents

  1. 1 Introduction
  2. 2 Conventions and notations
  3. 3 Borel cohomology
  4. 4 The Borel dual Steenrod algebra
  5. 5 Comparison with Greenlees’s description

##  1\. Introduction

In this companion piece to [Geo21a], we show that the $C_{2}$-equivariant
Borel dual Steenrod algebra computation in [HK96] generalizes to all groups
$G=C_{2^{n}}$. More precisely, we give an explicit description of the
$RO(C_{2^{n}})$-graded ring of the homotopy fixed points
$(H\operatorname{\mathbb{F}}_{2}\wedge
H\operatorname{\mathbb{F}}_{2})_{\bigstar}^{hC_{2^{n}}}$ as a Hopf algebroid
over $(H\operatorname{\mathbb{F}}_{2})^{hC_{2^{n}}}_{\bigstar}$, where
$\operatorname{\mathbb{F}}_{2}$ stands for the constant $C_{2^{n}}$-Green
functor associated to the field of two elements. We also compare our
description to the dual description of the Borel Steenrod algebra of [Gre88].

### Acknowledgment

We would like to thank Peter May for his numerous editing suggestions,
including the idea to split off this paper from [Geo21a].

##  2\. Conventions and notations

We will use the letter $k$ to denote the field $\operatorname{\mathbb{F}}_{2}$
with trivial $G=C_{2^{n}}$ action, the constant $G$-Mackey functor
$k=\underline{\operatorname{\mathbb{F}}_{2}}$ and the corresponding
equivariant Eilenberg-MacLane spectrum $Hk$. The meaning should always be
clear from the context.

Henceforth all our co/homology will be in $k$ coefficients. We use
$k_{\bigstar}(X)$ to denote the $RO(G)$-graded Mackey functor of
$G$-equivariant homology in $k$-coefficients. The value of $k_{\bigstar}(X)$
on the $G/H$ orbit is denoted by $k_{\bigstar}^{H}(X)$.

The real representation ring $RO(C_{2^{n}})$ is spanned by the irreducible
representations $1,\sigma,\lambda_{s,k}$ where $\sigma$ is the $1$-dimensional
sign representation and $\lambda_{s,m}$ is the $2$-dimensional representation
given by rotation by $2\pi s(m/2^{n})$ degrees for $1\leq m$ dividing
$2^{n-2}$ and odd $1\leq s<2^{n}/m$. Note that $2$-locally,
$S^{\lambda_{s,m}}\simeq S^{\lambda_{1,m}}$ as $C_{2^{n}}$-equivariant spaces,
by the $s$-power map. Therefore, to compute $k_{\bigstar}(X)$ for $\bigstar\in
RO(C_{2^{n}})$ it suffices to only consider $\bigstar$ in the span of
$1,\sigma,\lambda_{k}:=\lambda_{1,2^{k}}$ for $0\leq k\leq n-2$
($\lambda_{n-1}=2\sigma$ and $\lambda_{n}=2$).

For $V=\sigma$ or $V=\lambda_{m}$, denote by $a_{V}\in k_{-V}^{C_{2^{n}}}$ the
Euler class induced by the inclusion of north and south poles
$S^{0}\hookrightarrow S^{V}$; also denote by $u_{V}\in k_{|V|-V}^{C_{2^{n}}}$
the orientation class generating the Mackey functor $k_{|V|-V}=k$ ([HHR16]).

##  3\. Borel cohomology

Let $EG$ be a contractible free $G$-space and $\tilde{E}G$ be the cofiber of
the collapse map $EG_{+}\to S^{0}$. For a spectrum $X$ we use the notation
$X_{h}=EG_{+}\wedge X$, $X^{h}=F(EG_{+},X)$ and $X^{t}=\tilde{E}G\wedge
X^{h}$; there is a cofiber sequence

$X_{h}\to X^{h}\to X^{t}$

The $G$-fixed points of $X_{h},X^{h},X^{t}$ are the nonequivariant spectra of
homotopy orbits $X_{hG}$, homotopy fixed points $X^{hG}$ and Tate fixed points
$X^{tG}$ respectively.

The orientation classes $u_{V}:k\wedge S^{|V|}\to k\wedge S^{V}$ are
nonequivariant equivalences, hence induce $G$-equivalences in
$X_{h},X^{h},X^{t}$ for a $k$-module $X$, so they act invertibly on
$X_{h\bigstar},X^{h}_{\bigstar}$ and $X^{t}_{\bigstar}$. This implies that

$X_{h\bigstar}\approx X_{h|\bigstar|}\text{ ,
}X^{h}_{\bigstar}=X^{h}_{|\bigstar|}\text{ ,
}X^{t}_{\bigstar}=X^{t}_{|\bigstar|}$ (1)

and the $RO(G)$ graded part is determined by the integer graded part.

######  Proposition 3.1.

For $G=C_{2^{n}}$ and $n>1$:

$\displaystyle
k^{hG}_{\bigstar}=k[a_{\sigma},a_{\lambda_{0}},u_{\sigma}^{\pm},u_{\lambda_{0}}^{\pm},...,u_{\lambda_{n-2}}^{\pm}]/a_{\sigma}^{2}$
(2) $\displaystyle
k^{tG}_{\bigstar}=k[a_{\sigma},a_{\lambda_{0}}^{\pm},u_{\sigma}^{\pm},u_{\lambda_{0}}^{\pm},...,u_{\lambda_{n-2}}^{\pm}]/a_{\sigma}^{2}$
(3)

and $k_{hG\bigstar}=\Sigma^{-1}k^{tG}_{\bigstar}/k^{hG}_{\bigstar}$
(forgetting the ring structure). The map $k_{hG\bigstar}\to k^{hG}_{\bigstar}$
is trivial.

###### Proof.

The homotopy fixed point spectral sequence becomes:

$H^{*}(G;k)[u_{\sigma}^{\pm},u_{\lambda_{0}}^{\pm},...,u_{\lambda_{n-2}}^{\pm}]\implies
k^{hG}_{\bigstar}$ (4)

We have $H^{*}(G;k)=k^{*}BG=k[a]/a^{2}\otimes k[b]$ where $|a|=1$ and $|b|=2$.
The spectral sequence collapses with no extensions and we can identify
$a=a_{\sigma}u_{\sigma}^{-1}$ and $b=a_{\lambda_{0}}u_{\lambda_{0}}^{-1}$.
Finally,
$\tilde{E}G=S^{\infty\lambda_{0}}=\operatorname{\textup{colim}}(S^{\lambda_{0}}\xrightarrow{a_{\lambda_{0}}}S^{\lambda_{0}}\xrightarrow{a_{\lambda_{0}}}\cdots)$
so to get $k^{tG}_{\bigstar}$ we are additionally inverting $a_{\lambda_{0}}$.

∎

For $G=C_{2}$ we have

$\displaystyle k^{hC_{2}}_{\bigstar}=k[a_{\sigma},u_{\sigma}^{\pm}]$ (5)
$\displaystyle k^{tC_{2}}_{\bigstar}=k[a_{\sigma}^{\pm},u_{\sigma}^{\pm}]$ (6)

and
$k_{hC_{2}\bigstar}=\Sigma^{-1}k^{tC_{2}}_{\bigstar}/k^{hC_{2}}_{\bigstar}$
(forgetting the ring structure). The map $k_{hC_{2}\bigstar}\to
k^{hC_{2}}_{\bigstar}$ is trivial.

##  4\. The Borel dual Steenrod algebra

The $G$-Borel dual Steenrod algebra is

$(k\wedge k)_{\bigstar}^{hG}$ (7)

This is a Hopf algebroid over $k_{\bigstar}^{hG}$.

We will implicitly be completing it at the ideal generated by $a_{\sigma}$ for
$G=C_{2}$, and at the ideal generated by $a_{\lambda_{0}}$ for $G=C_{2^{n}}$,
$n>1$ (see [HK96] pg. 373 for more details in the case of $G=C_{2}$). With
this convention, Hu-Kriz computed the $C_{2}$-Borel dual Steenrod algebra to
be

$(k\wedge k)_{\bigstar}^{hC_{2}}=k_{\bigstar}^{hC_{2}}[\xi_{i}]$ (8)

for $|\xi_{i}|=2^{i}-1$ ($\xi_{0}=1$). The generators $\xi_{i}$ restrict to
the Milnor generators in the nonequivariant dual Steenrod algebra and

$\displaystyle\Delta(\xi_{i})=\sum_{j+k=i}\xi_{j}^{2^{k}}\otimes\xi_{k}$ (9)
$\displaystyle\epsilon(\xi_{i})=0\text{ , }i\geq 1$ (10)
$\displaystyle\eta_{R}(a_{\sigma})=a_{\sigma}$ (11)
$\displaystyle\eta_{R}(u_{\sigma})^{-1}=\sum_{i=0}^{\infty}a^{2^{i}-1}_{\sigma}u^{-2^{i}}_{\sigma}\xi_{i}$
(12)

######  Proposition 4.1.

For $G=C_{2^{n}}$, $n>1$,

$(k\wedge k)^{hG}_{\bigstar}=k^{hG}_{\bigstar}[\xi_{i}]$ (13)

for $|\xi_{i}|=2^{i}-1$ restricting to the $C_{2^{n-1}}$ generators $\xi_{i}$,
with

$\displaystyle\Delta(\xi_{i})=\sum_{j+k=i}\xi_{j}^{2^{k}}\otimes\xi_{k}$ (14)
$\displaystyle\epsilon(\xi_{i})=0\text{ , }i\geq 1$ (15)
$\displaystyle\eta_{R}(a_{\sigma})=a_{\sigma}\text{ ,
}\eta_{R}(a_{\lambda_{0}})=a_{\lambda_{0}}$ (16)
$\displaystyle\eta_{R}(u_{\sigma})=u_{\sigma}+a_{\sigma}\xi_{1}$ (17)
$\displaystyle\eta_{R}(u_{\lambda_{m}})=u_{\lambda_{m}}\text{ , }m>0$ (18)
$\displaystyle\eta_{R}(u_{\lambda_{0}})^{-1}=\sum_{i}a^{2^{i}-1}_{\lambda_{0}}u^{-2^{i}}_{\lambda_{0}}\xi_{i}^{2}$
(19)

###### Proof.

The computation of $(k\wedge k)^{hG}_{*}=(k\wedge k)^{*}(BG)$ follows from the
computation of $k^{hG}_{*}=k^{*}(BG)=k[a]/a^{2}\otimes k[b]$ and the fact that
nonequivariantly, $k\wedge k$ is a free $k$-module. To see that the homotopy
fixed point spectral sequence for $k\wedge k$ converges strongly, let
$F^{i}BG$ be the skeletal filtration on the Lens space
$BG=S^{\infty}/C_{2^{n}}$; we can then compute directly that
$\lim_{i}^{1}(k\wedge k)^{*}(F^{i}BG)=\lim_{i}^{1}(k[a]/a^{2}\otimes
k[b]/b^{i})=0$.

Thus we get $(k\wedge k)_{\bigstar}^{hG}=k_{\bigstar}^{hG}[\xi_{i}]$ and the
diagonal $\Delta$ and augmentation $\epsilon$ are the same as in the
nonequivariant case. The Euler classes $a_{\sigma},a_{\lambda_{0}}$ are maps
of spheres so they are preserved under $\eta_{R}$. The action of $\eta_{R}$ on
$u_{\sigma},u_{\lambda_{0}}$ can be computed through the right coaction on
$k_{\bigstar}^{hG}$: The (completed) coaction of the nonequivariant dual
Steenrod algebra on $k^{*}(BG)=k[a]/a^{2}\otimes k[b]$ is

$\displaystyle a\mapsto a\otimes 1$ (20) $\displaystyle
b\mapsto\sum_{i}b^{2^{i}}\otimes\xi_{i}^{2}$ (21)

To verify the formula for the coaction on $b$ we need to check that
$Sq^{1}(b)=0$ (the alternative is $Sq^{1}(b)=ab$). From the long exact
sequence associated to
$0\to\operatorname{\mathbb{Z}}/2\to\operatorname{\mathbb{Z}}/4\to\operatorname{\mathbb{Z}}/2\to
0$, we can see that the vanishing of the Bockstein on $b$ follows from
$H^{2}(C_{2^{n}};\operatorname{\mathbb{Z}}/4)=\operatorname{\mathbb{Z}}/4$
($n>1$).

After identifying $a=a_{\sigma}u_{\sigma}^{-1}$ and
$b=a_{\lambda_{0}}u_{\lambda_{0}}^{-1}$ we get the formula for
$\eta_{R}(u_{\lambda_{0}})$ and also that

$\eta_{R}(u_{\sigma})=u_{\sigma}+\epsilon a_{\sigma}\xi_{1}$ (22)

where $\epsilon$ is either $0$ or $1$. This is equivalent to

$\eta_{R}(u_{\sigma}^{-1})=u_{\sigma}^{-1}+\epsilon
a_{\sigma}u_{\sigma}^{-2}\xi_{1}$

and to see that $\epsilon=1$ we use the map
$k^{hC_{2}}=k^{h(C_{2^{n}}/C_{2^{n-1}})}\to k^{hC_{2^{n}}}$ that sends
$a_{\sigma},u_{\sigma}$ to $a_{\sigma},u_{\sigma}$ respectively. Finally, to
compute $\eta_{R}(u_{\lambda_{m}})$ for $m>0$ note that

$k^{hC_{2^{n-m}}}=k^{hC_{2^{n}}/C_{2^{m}}}\to k^{hC_{2^{n}}}$

sends $a_{\lambda_{0}},u_{\lambda_{0}}$ to $a_{\lambda_{m}}=0,u_{\lambda_{m}}$
respectively. ∎

##  5\. Comparison with Greenlees’s description

We now compare our result with the description of the Borel Steenrod algebra
given in [Gre88], which is dual to our calculation.

In our notation, the $G$-spectrum $b$ of [Gre88] is $b=k^{h}$ and $b^{V}(X)$
corresponds to $(k^{h})^{|V|}_{G}(X)$; to get $(k^{h})^{V}_{G}(X)$ we need to
multiply with the invertible element $u_{V}\in k^{hG}_{|V|-V}$. The Borel
Steenrod algebra is $b^{\bigstar}_{G}b=(k^{h})_{G}^{\bigstar}(k^{h})$ and the
Borel dual Steenrod algebra is
$b_{\bigstar}^{G}b=(k^{h})^{G}_{\bigstar}(k^{h})=(k\wedge k)^{hG}_{\bigstar}$.

Greenlees proves that the Borel Steenrod algebra is given by the Massey-
Peterson twisted tensor product ([MP65]) of the nonequivariant Steenrod
algebra $k^{*}k$ and the Borel cohomology of a point
$(k^{h})_{G}^{\bigstar}=k^{hG}_{-\bigstar}$. The twisting has to do with the
fact that the action of the Borel Steenrod algebra on
$x\in(k^{h})_{G}^{\bigstar}(X)$ is given by:

$(\theta\otimes a)(x)=\theta(ax)$ (23)

where $\theta\in k^{*}k$ and $a\in k^{hG}_{\bigstar}$. The product of elements
$\theta\otimes a$ and $\theta^{\prime}\otimes a^{\prime}$ in the Borel
Steenrod algebra is not $\theta\theta^{\prime}\otimes aa^{\prime}$, since
$\theta$ does not commute with cup-products, but rather satisfies the Cartan
formula:

$\theta(ab)=\sum_{i}\theta_{i}^{\prime}(a)\theta_{i}^{\prime\prime}(b)\text{ ,
}\Delta\theta=\sum_{i}\theta_{i}^{\prime}\otimes\theta^{\prime\prime}_{i}$
(24)

Therefore:

$(\theta\otimes a)(\theta^{\prime}\otimes
a^{\prime})(x)=\theta(a\theta^{\prime}(a^{\prime}x))=\sum_{i}\theta_{i}^{\prime}(a)(\theta^{\prime\prime}_{i}\theta^{\prime})(a^{\prime}x)$
(25)

so

$(\theta\otimes a)(\theta^{\prime}\otimes
a^{\prime})=\sum_{i}\theta_{i}^{\prime}(a)(\theta^{\prime\prime}_{i}\theta^{\prime}\otimes
a^{\prime})$ (26)

(we have ignored signs as we are working in characteristic $2$).
So the Borel Steenrod algebra is $k^{*}k\otimes k^{hG}_{\bigstar}$ with
twisted algebra structured defined by (26).

Moreover, Greenlees expresses the action of $k^{*}k$ on
$(k^{h})^{\bigstar}_{G}(X)$ in terms of the action of $k^{*}k$ on the
orientation classes $u_{V}$ and the usual (nonequivariant) action of $k^{*}k$
on $(k^{h})^{*}_{G}(X)=k^{*}(X\wedge_{G}EG_{+})$. This is done through the
Cartan formula: If $x\in(k^{h})^{V}_{G}(X)$ then
$u_{V}^{-1}x\in(k^{h})^{|V|}_{G}(X)$ and

$\theta(x)=\theta(u_{V}u_{V}^{-1}x)=\sum_{i}\theta_{i}^{\prime}(u_{V})\theta^{\prime\prime}_{i}(u_{V}^{-1}x)$
(27)

What remains to compute is $\theta_{i}^{\prime}(u_{V})$, namely the action of
$k^{*}k$ on orientation classes.

In our case, for $G=C_{2^{n}}$, we can see that:

######  Proposition 5.1.

The action of $k^{*}k$ on orientation classes is determined by:

$\displaystyle Sq^{i}(u_{\sigma})=\begin{cases}u_{\sigma}&i=0\\\
a_{\sigma}&i=1\\\ 0&\text{ otherwise}\end{cases}$ (28) $\displaystyle
Sq^{i}(u_{\lambda_{m}})=\begin{cases}u_{\lambda_{m}}&i=0\\\
a_{\lambda_{0}}&i=2,m=0\\\ 0&\text{ otherwise}\end{cases}$ (29)

###### Proof.

Compare with the proof of Proposition 4.1.∎

The twisting in the case of the Borel dual Steenrod algebra corresponds to the
fact that $(k\wedge k)^{hG}_{\bigstar}$ is a Hopf algebroid and not a Hopf
algebra; computationally this amounts to the formula for $\eta_{R}$ of
Proposition 4.1.

## References

  * [Geo21a] N. Georgakopoulos, _The $RO(C_{4})$ cohomology of the infinite real projective space_, available here
  * [Gre88] J.P.C. Greenlees, _Stable maps into free $G$-spaces_, Transactions of the American Mathematical Society, Volume 310, Number 1, November 1988
  * [HHR16] M. A. Hill, M. J. Hopkins, D. C. Ravenel, _On the non-existence of elements of Kervaire invariant one_ , Annals of Mathematics, Volume 184 (2016), Issue 1
  * [HK96] P. Hu and I. Kriz, _Real-oriented homotopy theory and an analogue of the Adams Novikov spectral sequence_ , Topology 40 (2001), no. 2, 317–399
  * [MP65] W. S. Massey and F. P. Peterson, _The cohomology structure of certain fibre spaces I_ , Topology 4 (1965), 47-65
  * [Wil19] D. Wilson, _$C_{2}$ -equivariant Homology Operations: Results and Formulas_, arXiv:1905.00058

1

Department of Mathematics, University of Chicago
E-mail<EMAIL_ADDRESS>
Website: math.uchicago.edu/$\sim$nickg