# Learning Phone Recognition from Unpaired Audio and Phone Sequences Based on
Generative Adversarial Network

Da-rong Liu, Po-chun Hsu, Yi-chen Chen, Sung-feng Huang, Shun-po Chuang, Da-yi
Wu, and Hung-yi Lee D.-R. Liu, P.-C. Hsu, S.-P. Chuang, Y.-C. Chen and S.-F.
Huang are with the Graduate Institute of Communication Engineering, National
Taiwan University, Taipei 10617, Taiwan (e-mail<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>f06942045@ntu.edu.tw).D.-Y Wu is with National Taiwan University, Taipei
10617, Taiwan (e-mail: ericwudayi2@gmail.com).H.-Y Lee is with the Department
of Electrical Engineering, National Taiwan University, Taipei 10617, Taiwan
(e-mail: hungyilee@ntu.edu.tw).This work was supported in part by the Ministry
of Science and Technology, R.O.C, under Contract 110-2628-E002-001 and
110-2223-E-002-007-MY3.

###### Abstract

ASR has been shown to achieve great performance recently. However, most of
them rely on massive paired data, which is not feasible for low-resource
languages worldwide. This paper investigates how to learn directly from
unpaired phone sequences and speech utterances. We design a two-stage
iterative framework. GAN training is adopted in the first stage to find the
mapping relationship between unpaired speech and phone sequence. In the second
stage, another HMM model is introduced to train from the generator’s output,
which boosts the performance and provides a better segmentation for the next
iteration. In the experiment, we first investigate different choices of model
designs. Then we compare the framework to different types of baselines: (i)
supervised methods (ii) acoustic unit discovery based methods (iii) methods
learning from unpaired data. Our framework performs consistently better than
all acoustic unit discovery methods and previous methods learning from
unpaired data based on the TIMIT dataset.

###### Index Terms:

unsupervised learning, phone recognition, generative adversarial network

##  I Introduction

With the rapid development of deep learning, automatic speech recognition
(ASR) has achieved remarkable performance and has been widely used. However,
the training of state-of-the-art ASR systems [1, 2, 3, 4] often relies on
massive annotated data. Therefore, for low-resource languages with scarce
annotated data, sufficiently accurate speech recognition is difficult to
achieve. Compared to annotating audio data for low-resource, huge quantities
of unannotated audio data are relatively easy to collect. If the machine can
directly learn an ASR model from unannotated audio with unpaired text only,
building an ASR system will be more feasible for low-resource languages. We
call such a setting ‘unsupervised ASR.’

Unsupervised learning, while challenging, has been successfully applied on
machine translation [5, 6, 7] and performed comparably with supervised
learning results. Unsupervised neural machine translation often comprises two-
step training. In the first step, a transformation from source to target
language word embedding spaces could be learned in an unsupervised manner. In
these papers [5, 6, 7], generative adversarial network (GAN) [8, 9, 10] is
served as the core framework. The GAN consists of a discriminator and a
generator. The generator takes the source language as input and outputs the
target language. The discriminator learns to distinguish the real target
language from the generator output, while the generator learns to ‘fool’ the
discriminator. The generator and the discriminator will train iteratively. In
the end, the generator output will become more and more ‘similar to’ the real
target language. After training, the word translation can be achieved by
selecting the nearest target word embedding compared to the transformed source
word embedding. Then in the second step, a denoising sequence-to-sequence
model is trained based on the word translation acquired in step one to get the
final translation model. In both two steps, the model selection and
hyperparameter tuning are based on the proposed unsupervised metrics instead
of the paired development set to prevent using any supervision information.

The success in unsupervised neural machine translation led to our attempts on
unsupervised ASR, since ASR is also a kind of translation, trying to learn the
mapping relationship from speech space to text space. As being the first step
toward unsupervised ASR, we make three compromises (i) we conduct the research
on the phone-level instead of text-level (ii) we have access to the number of
phones in advance (iii) we have the development set with paired phone
transcriptions to tune the hyperparameters, but not involved in the training
process. Specifically, we aim for the task of phone recognition, where the
learning algorithm only accesses (i) unannotated speech utterances, (ii)
unpaired phone sequences during training while still using the development set
to select the hyperparameters. For the rest of the article, we denote our
setting as ‘unpaired’ phone recognition to make it clear we focus on how to
learn phone recognition from unpaired data.

This paper attempts to use the GAN framework on unpaired phone recognition,
where the generator takes speech utterances as inputs and outputs phone
sequences. After iterative training between generator and discriminator, the
generator will serve as the final phone recognition model. However, there is
some fundamental difference between phone recognition and machine translation.
In unsupervised machine translation, we know that most discrete source tokens
can be mapped to specific target tokens representing the same meaning, and
this mapping can be achieved by the step one described in the second
paragraph. However, in unpaired phone recognition, which learns the mapping
from an utterance (a series of acoustic features) to a discrete sequence, we
do not know which segment of acoustic features should be mapped to a phone. It
is because each phone in a speech utterance consists of a segment of
consecutive frames of variable length. However, the phone boundaries are often
unknown in advance, which has made unpaired phone recognition difficult.

To address the segmentation problem, we include a phone segmentation module
before the generator. This module can segment the original utterance into a
sequence of phone-level segments. Then the generator maps each segment into a
phone and outputs a phone sequence. This phone segmentation module is
performed in an unsupervised manner. Many previous unsupervised phone
segmentation methods [11, 12, 13, 14, 15, 16, 17] can be used here.

After the GAN training, the generator serves as the (first-version) phone
recognition model. We propose to further boost the performance via ‘self re-
training.’ Inputting unpaired speech to the generator, we can generate their
corresponding ‘pseudo transcription’. Then we view the speech utterances and
the pseudo transcriptions as paired data and train a Hidden Markov Model (HMM)
in a supervised manner. Although the pseudo transcriptions have some errors
compared to the oracle transcriptions, the experiment shows that HMM, training
from pseudo transcriptions, can still significantly boost the performance
compared to the first-version model. Moreover, we use the trained HMM to
perform forced alignment on the speech utterances and obtain new segmentation.
This segmentation is more accurate than the result of the unsupervised phone
segmentation method because this segmentation is from a certainly well-trained
HMM (the experiment also supports this fact). With the new, better
segmentation, we can repeat the same procedure: GAN training, self re-
training, and getting new segmentation. The iteration will continue until the
performance converges. With the proposed framework, we get 36.7% phone error
rate on TIMIT. This result is comparable to the supervised method trained with
2.5% to 5% paired data.

The proposed two-stage iterative framework is summarised in figure 1.

Figure 1: Framework overview: blue lines denote GAN training; yellow lines
denote self re-training; black lines denote the paths which are not trainable.

In the training stage, given a large collection of unpaired speech utterances
and texts, speech utterances are segmented into phone-level segments by phone
segmentation module, and texts are transformed into phone sequences by a
lexicon. In figure 1(a), GAN training is then conducted to map speech and
phone sequence domains, which will be described in Section III. In figure
1(b), self re-training is conducted after GAN training. In this stage, another
HMM is trained from the generator’s output to boost the performance further.
HMM also provides more accurate segmentation information via forced alignment,
which can be used for the next iteration of GAN training. The details of self
re-training will be described in Section IV. In the inference stage, we can
either use the generator or the HMM model as the final phone prediction model.
In this framework, the generator plays a vital role because it is used in GAN
training, self re-training, and inference stage. The generator is designed to
have a specific output form under these different scenarios, and this will be
discussed in Section V.

##  II related work

Learning phone recognition from unpaired data is related to unsupervised
spoken term discovery (UTD) and acoustic unit discovery (AUD). The main goal
of UTD is to find clusters of speech segments that match their phonetic
content, which may be syllables, words, or longer phrases. The existing
methods for UTD can be broadly categorized into dynamic-time-warping-based
(DTW-based) and embedding-based approaches. The DTW-based approaches can be
traced back to the segmental DTW algorithm by [18], followed by SWD-model by
[19], DP-ngram model by [20], to name a few. In theory, DTW is very powerful
since it enables pairwise alignment of any arbitrary sequences that consist of
non-linear temporal distortions with respect to each other, and provides a
measure of the degree of similarity. However, the complexity of DTW is
$O(mnk)$ based on dynamic programming, where $m,n$ is the length of two
sequences and $k$ is the dimension of the acoustic feature. When $m,n$ is
large, DTW will become computationally expensive in the inference time.
Besides, DTW approaches often use relatively primitive representation (e.g.,
MFCCs or posteriorgrams) and assume that spectral changes are time-
synchronous, which may also affect the correctness.

Besides DTW, UTD can be done by embedding-based approaches. These approaches
attempt to capture the acoustic information in a speech segment of variable
length by condensing it into a fixed-dimensional speech representation. Speech
segments containing the same contents are hopefully mapped to similar
embeddings. By pre-computing each possible speech segment into a vector, only
dot product is needed when comparing the similarity of two speech segments in
the inference time. The complexity is $O(k)$, where $k$ is the embedding
dimension. Since the complexity of the embedding-based methods is independent
of the segment lengths, they can compute much faster than DTW with long
segments. In the earlier work, embedding approaches were developed primarily
in heuristic ways rather than learned from data. Graph-based embedding
approaches are also used to represent audio segments as fixed-length vectors
[21, 22]. Recently, deep learning has been used to encode acoustic information
as vectors [23, 24, 25, 26, 27, 28, 29, 30, 31]. This transformation
successfully produces vector spaces in which audio segments with similar
phonetic structures are located in close proximity. By training a recurrent
neural network (RNN) with an audio segment as the input and the corresponding
word as the target, the outputs of the hidden layer at the last few time steps
can be taken as the representation of the input segment [24, 32]. In [25], the
authors obtain embeddings by training a neural network that separates same-
word and different-word pairs. In [33], a sequence-to-sequence auto-encoder
training paradigm is proposed, which only uses word boundaries as supervision
information. It has been shown that the representation does contain phonetic
information.

On the other hand, AUD consists of discovering an inventory of phone-like
discrete acoustic units from a set of untranscribed recordings. Nowadays, two
major approaches are widely used in AUD: (i) neural-network-based models,
which typically use an auto-encoder structure with a discretization layer [34,
35, 36, 37, 38, 39]. (ii) non-parametric Bayesian generative-based models,
which can be seen as infinite mixtures time series models [40, 41, 42, 43,
44].

Although all the above approaches can cluster speech segments according to
phonetic structure and generate automatically discovered acoustic tokens,
there is still a gap between mapping the discovered tokens into the human-
defined tokens, for example phone and text. The lack of these mapping
relationships has seriously limited the downstream applications because the
core information of the speech signal is carried by the phones, and many
downstream applications are only based on the transcriptions. [45] tries to
transcribe some tokens into text while with the help of some extra
annotations. Our prior work [46] is the first work to successfully learn phone
recognition with unpaired transcriptions by first clustering audio embeddings
into a set of tokens and then using GAN to learn the mapping relationship
between tokens and phones. However, in [46], the performance still relies on
extremely good phone segmentation. This was properly handled previously by a
specially designed cost function called Segmental Empirical Output
Distribution Matching (Segmental empirical-ODM) [47], which considered both
the n-gram probabilities across all output tokens and the frame-wise
smoothness within each segment [48]. However, the loss term in [47] includes
an empirical average over all dataset inside a logarithmic function, which can
be biased if we sample this empirical average by a mini-batch average. [47]
therefore proposes to use an extremely large batch size during training to
reduce the biasing problem. Besides, the n-gram probabilities considered here
only include local statistics in the output sequences, while other information
such as long-distance dependency is inevitably ignored. These problems are
handled by another prior work [49]. Based on [46], [49] removes the
quantization step and further proposes an iterative training algorithm that
can correct the phone boundaries over the iterations and boost the
performance. Compared to [48], the discriminator in [49] considers all
possible information from the dataset, not limited to the n-gram local
statistics. Besides, [49] works well with a reasonable batch size, which makes
it feasible when the computational resource is limited. [50, 51] also use GANs
to map the word-level speech embedding space [29] to word space, and achieve
promising performance on spoken word classification, speech translation, and
spoken word retrieval. In [50, 51], the word boundaries are also generated
automatically with speech only.

This journal paper is an extension of the previous conference papers [49]. In
this paper, we first propose a training framework, which includes GAN training
and self re-training. In the GAN training, [49] only describes one kind of
architecture design of GAN. While in this paper, all possible generator
architectures are categorized into segment-wise generator and frame-wise
generator. These two kinds of generators are discussed in detail and shown
that the former performs better when we have oracle boundaries, while the
latter performs better when we do not. Different discriminator architectures
are also discussed and compared to each other. Besides, [49] has shown the
effectiveness of self re-training. In this paper, we further give the analysis
and evidence that this improvement is brought from the improvement of phone
segmentation over iterations. Finally, we also analyze where the prediction
errors happen, which has also not been done before.

##  III GAN training

In the training stage, we have a bunch of unpaired speech utterances that are
represented as acoustic features $X$, and text. We transform all text into
phone sequences with the aid of a lexicon, and denote it as ‘real phone
sequences’ $P^{real}$. We assume $P^{real}$ is representative and diverse
enough to capture the distribution of all reasonable phone sequences. Our
target is to learn a function $f:X\rightarrow P$, which maps speech to phone
sequences, such that the distribution of $f(X)$ is close to the distribution
of $P^{real}$.

The overview of GAN training is illustrated in figure 1 (a). The speech
utterances are first represented as acoustic features and segmented into phone
segments, which is described in Section III-A. The segmented acoustic features
are then inputted into the generator, and the generator outputs ‘generated
phone posterior sequence.’ In Section III-B, we only describe the input/output
format of generator while leaving the architecture details to Section V. In
III-C, we talk about the preprocessing of real phone sequences $P^{real}$.
Then in Section III-D, we describe the discriminator details. Finally, the
optimization details of the GAN training process are discussed in Section
III-E.

###  III-A Phone Segmentation Module

The phone segmentation module segment the input acoustic feature sequence $X$
into a sequence of phone-level segments $S=\\{s_{1},...,s_{U}\\}$, where
$s_{i}$ represents a segment of several consecutive acoustic features. In the
first iteration, because no supervised information is provided, this can only
be done by unsupervised phone segmentation methods [11, 12, 13, 14, 15, 16,
17]. After the second iteration, we can perform forced alignment with the
trained HMM model to generate the phone boundaries.

###  III-B Generator

The generator takes a segmented acoustic feature sequence
$S\equiv\\{s_{1},...,s_{U}\\}$ as input, and outputs a phone distribution
sequence $P^{gen}$, which we referred as ‘generated phone posterior sequence.’
Formally speaking, we denote,

$\displaystyle P^{gen}=\mathcal{G}(S)$ (1) $\displaystyle
P^{gen}\equiv(p^{gen}_{1},p^{gen}_{2},...p^{gen}_{U})$ (2)

where $\mathcal{G}$ denotes the generator, $U$ is the length of the output
phone sequence and $p^{gen}_{i}\in\mathcal{R}^{|S|}$ is the $i^{th}$ posterior
over the phone set $\mathcal{S}$, which means:

$\displaystyle\sum^{|\mathcal{S}|}_{j=1}p^{gen}_{i}[j]=1,\forall i\in[1,U]$
(3)

where $p^{gen}_{i}[j]$ is the $j^{th}$ element of $p^{gen}_{i}$. The design of
the generator will be discussed in Section V.

###  III-C Preprocessing of Real Phone Sequence

Corresponds to the generated phone posterior sequences, each real phone
sequence $P^{real}$ is represented as a one-hot encoding sequence. In this
paper, we slightly abuse the notation $P^{real}$ to also refer to the phone
sequence in one-hot encoding form. Hence, we can mathematically denote:

$\displaystyle P^{real}\equiv(p^{real}_{1},p^{real}_{2},...p^{real}_{V})$ (4)

where $V$ is the number of phones in the sequence and
$p^{real}_{i}\in\mathcal{R}^{|\mathcal{S}|}$ is the one-hot encoding of the
$i^{th}$ phone in the sequence.

###  III-D Discriminator

The discriminator $\mathcal{D}$ is trained to distinguish between $P^{gen}$
and $P^{real}$. We follow the Wasserstein GAN (WGAN) framework [9]. The input
of the discriminator is a sequence of phone posteriors, like $P^{gen}$ and
$P^{real}$ ($P^{real}$ as one-hot encoding is also a kind of posterior); the
output of the discriminator is a scalar. The scalar is expected to be high
when the input is $P^{real}$, while to be low when the input is $P^{gen}$.
Multiple discriminator architectures are feasible for this input/output form.
The detailed performance comparisons will be conducted in Section VII-D.

###  III-E Optimization Formulation of GAN Training

####  III-E1 discriminator loss

The loss for training the discriminator $\mathcal{D}$ follows the concept of
WGAN [9, 10] with gradient penalty [10]:

$\mathcal{L}_{\mathcal{D}}=\frac{1}{K}\sum_{k=1}^{K}\mathcal{D}(P^{gen(k)})-\frac{1}{K}\sum_{k=1}^{K}\mathcal{D}(P^{real(k)})+\alpha\mathcal{L}_{gp},$
(5)

where $\mathcal{D}(P)$ is the scalar output of the discriminator for an input
sequence $P$, $K$ is the number of training examples in a batch, and $k$ is
the example index. $\alpha$ is the weight for the gradient penalty
$\mathcal{L}_{gp}$:

$\mathcal{L}_{gp}=\frac{1}{K}\sum_{k=1}^{K}(||\nabla_{P^{inter(k)}}\mathcal{D}(P^{inter(k)})||-1)^{2},$
(6)

where $P^{inter}$ is the interpolation of $P^{real}$ and $P^{gen}$:

$\displaystyle P^{inter}=\epsilon P^{real}+(1-\epsilon)P^{gen}$ (7)
$\displaystyle\epsilon\sim Uniform[0,1]$ (8)

Because $P^{real}$ and $P^{gen}$ may have different sequence lengths,
$P^{real}$ and $P^{gen}$ are first truncated to the shorter length of the two
before calculating $P^{inter}$. This additional term has been proved useful in
stabilizing the GAN training and preventing the gradient vanishing problem
[10].

####  III-E2 generator loss

The generator loss is:

$\mathcal{L}_{\mathcal{G}}=-\frac{1}{K}\sum_{k=1}^{K}\mathcal{D}(P^{gen(k)}),$
(9)

The generator and the discriminator are learned to minimize the loss
iteratively, so the generator can eventually map speech utterances to phone
sequences ‘looking real.’

##  IV Self Re-training

After the GAN is well trained, we decode the training set into phone sequences
by the generator inside the GAN. These GAN-generated phone sequences are taken
as pseudo transcriptions for the training speech utterances to train a new
phone HMM model, which can serve as a new phone recognition model. Besides,
the phone HMM can be used to perform forced alignment for the training set and
get new segmentation boundaries, which are then used to start a new iteration
of GAN training as described in Sections III, then another HMM is trained.
This training procedure can be performed iteratively until convergence. In the
inference stage, we can either use the generator or HMM model as the final
phone recognition model. The whole training/inference algorithm is depicted in
Algorithm 1.

Training Stage:

Input: Real phone sequences $P^{real}$, Speech utterances $X$

Output: generator, HMM

Initialize:

unsupervised phone segmentation boundaries $b$

while  _not converged_ do

GAN Training:

Given $b$, train GAN in an unsupervised manner;

Obtain transcriptions $T$ of each speech utterance using the generator within
the GAN;

Self Re-training:

Given $T$, train the HMMs;

Obtain a new $b$ by forced alignment with the HMMs;

Inference Stage:

Input: Speech utterances $X$, generator, HMM

Output: predicted phone sequences

Obtain predicted phone sequences by the generator or HMM;

Algorithm 1 Whole Framework

##  V Generator

There are two modes for the generator: training mode and evaluation mode. In
the training mode, which is used in the GAN training, the generator outputs
generated phone posterior sequence $P^{gen}$, which is a sequence of phone
posteriors. In the evaluation mode, which is used in the self re-training and
the inference stage, the generator outputs phone sequence, which is a sequence
of discrete tokens. In Sections V-A and V-B, we categorize generators into two
kinds of architectures: segment-wise generator, frame-wise generator. Then
their computation details in both training and evaluation mode are discussed.
In Section V-C, we further review Gumbel-Softmax, a special kind of softmax,
to use in the training mode of generator. Using Gumbel-Softmax is shown to
achieve better performance in the experiments.

Figure 2: Segment-wise generator  Figure 3: Frame-wise generator

###  V-A Segment-wise Generator

Figure 2 illustrates the training and evaluation mode of the segment-wise
generator. Each phone-level segment is first passed through a segment-wise
phone classifier and outputs segment-wise phone posterior. In the training
mode, the segment-wise phone posterior is directly used as generated phone
posterior sequence $P^{gen}$. In the evaluation mode, we have two options to
map the segment-wise phone posterior into phone sequence.

####  V-A1 Select the maximum probability

The output phone sequence is generated by collecting the phone with the
largest probability over all segment-wise phone posteriors.

####  V-A2 WFST

Weighted Finite State Transducer (WFST) is a well-known speech decoder that
can incorporate language model (LM) information and directly output the
recognition result from a series of phone posteriors. Instead of simply
getting the phone with the largest probabilities of each posterior, we decode
the phone sequence with the highest probability while considering phone
n-grams.

###  V-B Frame-wise Generator

Figure 3 illustrates the training and evaluation mode of the frame-wise
generator. Unlike segment-wise generator, in this architecture, frame-wise
phone posteriors are first generated, which implies that phone boundaries are
not used in the phone posterior generation process. In the training mode, each
segment of posterior is passed through a ‘reduce’ process to reduce a segment
of posteriors into one posterior. In this paper, we introduce two types of
reduce methods: average and sample.

####  V-B1 average

Each segment of frame-wise posteriors are averaged into a segment-wise
posterior, which is denoted as:

$\displaystyle
p^{gen}_{i}=\frac{1}{end_{i}-start_{i}+1}\sum_{j=start_{i}}^{end_{i}}y_{j}$
(10)

where $start_{i}$ and $end_{i}$ denote the start and end index of the $i^{th}$
segment and $y_{j}$ is the $j^{th}$ frame posterior.

####  V-B2 sample

Instead of average a series of posteriors within a segment, we simply sample
one of them at each training step.

When using frame-wise generator, to make the phone distribution inside a
phone-level segment being consistent, we further introduce a loss, called
‘intra-segment loss’ $\mathcal{L}_{intra}$:

$\mathcal{L}_{intra}=-\frac{1}{K}\sum_{k=1}^{K}\sum_{i,j\in
s_{k}}(y_{i}-y_{j})^{2},$ (11)

This loss serves as a regularization term for the generator loss described in
equation 9, so the new generator loss will be:

$\mathcal{L}_{G}=-\frac{1}{K}\sum_{k=1}^{K}D(P^{gen(k)})+\lambda\mathcal{L}_{intra},$
(12)

where $\lambda$ is a tunable weight. To compute $\mathcal{L}_{intra}$, instead
of exhaustively averaging over all permutations within a segment, we sample 10
pairs of frames in the implementation.

In the evaluation mode, we can either generate a phone sequence from segment-
wise posterior or directly from frame-wise posterior. If choosing from
segment-wise posterior, then the case will be similar to the segment-wise
generator. To get the phone sequence, we can either select (1) select the
maximum probability or (2) WFST, which are both described in Section V-A. Only
‘average’ method is used for the reduce module in the evaluation mode because
the sampling process will bring randomness, which is not suitable for
evaluation. Another choice is to generate from frame-wise posterior. In this
case, we can also use the WFST decoder because the WFST decoder can also take
posterior with repeated phone as input. In this case, the segmentation
boundaries are even no longer necessary (in the evaluation stage).

###  V-C Gumbel-Softmax

In the training mode, we have an option to add ‘Gumbel-Softmax’ module at the
output of generated phone posterior sequence $P^{gen}$ before passing to the
discriminator, as illustrated in Figure 4.

Figure 4: In Figure 1, generated phone posterior sequence $P^{gen}$ is
directly fed to the discriminator (path a) along with the real phone sequence
$P^{real}$. Due to $P^{gen}$ is the output of softmax, which is a soft
distribution, while $P^{real}$ is a one-hot distribution, the discriminator
may distinguish two sequences easily [10]. Although gradient penalty can
somehow mitigate the problem, we further pass through Gumbel-Softmax [52, 53]
to make $P^{gen}$ a little sharper (path b) or pass through straight-through
Gumbel-Softmax [52, 53] to make $P^{gen}$ also be a one-hot distribution (path
c) and see if the performance improves.

Gumbel-Softmax mimics the categorical sampling procedure while making the
sampling process differentiable, and is only used in the training mode and not
involved in the evaluation mode. Three scenarios are considered in the
experiments.

####  V-C1 Without Gumbel-Softmax

The generated phone posterior sequence $P^{gen}$ is directly fed into
discriminator.

####  V-C2 With Gumbel-Softmax

We replace the softmax layer at the output of $P^{gen}$

$p^{gen}_{i}[j]=\frac{\exp(l_{ij})}{\sum_{j=1}^{|\mathcal{S}|}\exp(l_{ij}))},$
(13)

into

$p^{gen-
gs}_{i}[j]=\frac{\exp((l_{ij}+g_{ij})/\tau)}{\sum_{j=1}^{|\mathcal{S}|}\exp((l_{ij}+g_{ij})/\tau)},$
(14)

where $l_{i1},...l_{i|\mathcal{S}|}$ are the logits before softmax,
$g_{i1},...g_{i|\mathcal{S}|}$ are i.i.d samples drawn from gumbel
distribution $Gumbel(0,1)$ and $\tau$ is the temperature. Samples from Gumbel
distribution $g_{ij}$ can be transformed from samples from uniform
distribution:

$\displaystyle u_{ij}\sim Uniform(0,1)$ (15) $\displaystyle
g_{ij}=-\log(-\log(u_{ij}))$ (16)

####  V-C3 With straight-through(ST) Gumbel-Softmax

After passing through Gumbel-Softmax module, $p^{gen-gs}_{i}$ is sharper while
not being an one-hot vector. We can further convert the $p^{gen-gs}_{i}$ into
one-hot encoding by taking argmax class from vector:

$p^{gen-st}_{i}=onehot(\mathop{\arg\max}_{j}\ p^{gen-gs}_{i}[j]).$ (17)

According to Gumbel-Max trick [54, 53], $p^{gen-st}_{i}$ is exactly a
categorical sample sampling from the posterior $p^{gen}_{i}$ [54, 52]. In the
ST Gumbel-Softmax, because $onehot+argmax$ is a flat function, with zero
gradients at most points and even non-differentiable at some points, we
approximate the gradient:

$\frac{\partial p^{gen-st}_{i}}{\partial p^{gen-
gs}_{i}\hfill}\approx\mathbbm{1}.$ (18)

in the backpropagation [52]. Therefore, in the training of ST Gumbel-Softmax,
we only calculate an approximation of the gradients, not the exact value.

##  VI Experimental Setup

###  VI-A Dataset

We use the TIMIT corpus to evaluate the performance. TIMIT contains recordings
of phonetically-balanced utterances. Each utterance includes manually aligned
phonetic/word transcriptions, as well as a 16-bit, 16kHz waveform file. For
each utterance, 39-dim MFCCs are extracted with utterance-wise cepstral mean
and variance normalization (CMVN) applied. All models are trained based on the
48 phone classes. We evaluate phone error rate (PER) on the standard test set,
which contains 192 utterances, for 39 phone classes mapped from the 48 output
classes of the model.

###  VI-B Training Settings

There are four settings for the following experiments, which are the
permutations of _match/nonmatch_ and _orc/uns_. Each of them is respectively
described below

  1. 1.

_match:_ The utterances and the real phone sequences are matched but not
aligned during training.

  2. 2.

_nonmatch:_ There is no overlapping utterance ID between the utterances and
the real phone sequences. Because TIMIT has different utterances recorded from
the same sentences, there are still recordings of the same content (word
sequence) in two sets.

  3. 3.

_orc:_ the oracle boundaries provided by TIMIT are used.

  4. 4.

_uns:_ the initial boundaries are obtained automatically with GAS [11].

##  VII Architecture Discussion of GAN training

We first discuss the performance of using different architecture designs in
the GAN training. Following [49], we select 4000 utterances in the original
training set for training and others for validation. Under _match_ setting,
all 4000 utterances are used as real phone sequences. Under _nonmatch_
setting, 3000 utterances are taken as speech utterances while the phone
transcriptions of the other 1000 utterances are taken as the real phone
sequences. In this section, we focus on the GAN architecture. Therefore, all
experiments are conducted with training one iteration and without self re-
training.

###  VII-A Comparing Segment-wise and Frame-wise Generator

We first compare the performance of the segment-wise and frame-wise
generators, which are described respectively in Section V-A and Section V-B.
The Gumbel-Softmax is used in all training, and the temperature is set to 0.9.
The discriminator in both settings is a two-layer convolutional network. The
first layer is a 1-D convolution bank [55, 56] with kernel sizes 3,5,7,9, and
channel size 256. The second layer is a convolution layer with kernel size 3
and channel size 1024. The gradient penalty weight $\alpha$ is set to 10. In
the _uns_ setting, we randomly remove 4% of the phones and duplicate 11% of
the phones in the real phone sequences to generate augmented phone sequences
to be used as real phone sequences. We use RAdam [57] optimizer for both
generator and discriminator. The batch size is set to 100. When using the WFST
decoder, we use unaugmented real phone sequences to train a 5-gram LM. The
AM/LM ratio and the self-loop probability are tuned according to the PER of
the development set. The specific GAN setting for each method are listed
below:

####  VII-A1 settings of segment-wise generator

In the segment-wise generator, we first use an LSTM with 512 hidden units to
encode each segment into a fix-dimension vector. Then we use a linear layer to
project the vector into the phone posterior. The learning rate for the
generator and discriminator is set to 1e-2 and 1e-3, respectively, and the
discriminator updates three times per generator update. In the evaluation
stage, we report the PER of 1. selecting the maximum probability from segment-
wise phone posterior (_Max Prob_) 2. using WFST on top of segment-wise phone
posterior (_WFST_).

####  VII-A2 settings of frame-wise generator

For the frame-wise generator, we take the concatenation of 11 windowed frames
of MFCCs as the input feature. The phone classifier is a one-layer DNN with
512 ReLU units with output classes equal to phone numbers. Intra-loss weight
$\lambda$ is set to 0.5. The learning rate for the generator and discriminator
is set to 1e-3 and 2e-3, respectively, and the discriminator updates three
times per generator update. In the evaluation stage, we report the PER of 1.
_Max Prob_ 2\. _WFST_ 3\. using WFST on top of frame-wise phone posterior
(_WFST *_).

TABLE I: Performance of segment-wise and frame-wise generators: _Max Prob_ denotes select the phone with the highest probability of each segment-wise phone posterior. _WFST_ means using WFST decoder on top of segment-wise phone posterior; _WFST *_ means using WFST decoder on top of frame-wise phone posterior. _Evaluation metric: PER in [%]._ |  | _orc_ | _uns_
---|---|---|---
| _match_
---
| _nonmatch_
---
| _match_
---
| _nonmatch_
---
segment-wise generator | (a) _Max Prob_ | 23.07 | 35.83 | 68.66 | 82.73
(b) _WFST_ | 22.35 | 33.89 | 67.10 | 82.44
frame-wise generator | average | (c) _Max Prob_ | 26.91 | 36.12 | 66.16 | 69.35
(d) _WFST_ | 24.24 | 34.10 | 65.18 | 65.82
(e) _WFST *_ | 29.01 | 38.81 | 58.95 | 62.13
sample | (f) _Max Prob_ | 28.63 | 36.97 | 64.02 | 63.33
(g) _WFST_ | 26.38 | 35.34 | 60.80 | 61.83
(h) _WFST *_ | 31.52 | 40.12 | 55.70 | 57.57
|  |  |  |  |  |

Our results are shown in Table I. First, _WFST_ performs consistently better
than _Max Prob_ ((b) vs. (a), (d) vs. (c), (g) vs. (f)), which means
incorporating LM information helps improve the PER. In the evaluation stage,
_WFST *_ ((e)(h)) does not use boundary information. On the other hand, _WFST_
and _Max Prob_ are generated from segment-wise phone posteriors, thus using
boundary information during evaluation. We observe that under _orc_ setting,
where the boundaries are perfect, _WFST_ and _Max Prob_ performs better than
_WFST *_ ((c)(d) vs. (e), (f)(g) vs. (h), (a)(b) vs. (e)(h)). Among them, the
segment-wise generator performs the best ((a)(b) vs. others). On the other
hand, under _uns_ setting, where the boundaries are not accurate, _WFST *_,
which does not use boundary information, performs better ((e)(h) vs. others).

When using frame-wise generator, using ‘average’ as reduce module performs
better than ‘sample’ under _orc_ setting, while worse under _uns_ setting ((e)
vs. (h)).

Figure 5:  The blue lines denote the oracle phone boundaries; the red dashed
lines denote the unsupervised generated phone boundaries. Here take frame-wise
generator as an example and use _uns_ setting. When using ‘average’ as the
reduce module, the generated segment-wise phone posterior may use the acoustic
information from nearby phones. For example, in the left part of the figure,
$p^{gen}_{2}$ contains the acoustic information from $phn_{2}$ and $phn_{3}$.
On the other hand, this will not happen when using ‘sample’ as the reduce
module. For example, in the right part of the figure, each posterior only
contains the acoustic information from one phone.

We hypothesize that during training and under the _uns_ setting, using
‘sample’ is less susceptible to errors in segmentation because it will not
utilize the acoustic information from nearby phone segments, while using
‘average’ will. This is further explained in figure 5. For the following
experiments, we keep using the frame-wise generator with ‘sample’ reduce
module attached with a frame-wise WFST decoder (_WFST *_), which has the best
performance under _uns_ setting.

###  VII-B Discussion of the Capacity of the Frame-wise Generator

The previous section uses one-layer DNN with 512 units, which is a relatively
simple model, as the frame-wise generator. Generally speaking, powerful models
usually achieve better performances in supervised learning if not overfitting.
While in the GAN training, if the generator is powerful enough, the generator
in principle can completely output a phone sequence that is unrelated to the
input acoustic feature and converge to a bad result in the end. In Table II,
we compare two different generators: DNN and LSTM. Both models use 512 hidden
units. Although LSTM has a larger model capacity, the experiments show that
LSTM only achieves similar performance to DNN in _orc_ setting while worse in
_uns_ setting. This result supports the fact that using powerful generators is
not always better in GAN training.

TABLE II: Comparison between different frame-wise generators. _Evaluation metric: PER in [%]._ | _orc_ | _uns_
---|---|---
|  | _match_
---
| _nonmatch_
---
| _match_
---
| _nonmatch_
---
(a) DNN | 31.52 | 40.12 | 55.70 | 57.57
(b) LSTM | 31.36 | 41.89 | 74.14 | 75.63
|  |  |  |

###  VII-C Using Gumbel-Softmax in Generator

Then we investigate the effectiveness of using Gumbel-Softmax at the output of
the generator in the training mode. The temperature is set to 0.9. The results
are shown in Table III.

TABLE III: Performance of using Gumbel-Softmax. ‘ST Gumbel-S.’ stands for straight through Gumbel-Softmax. _Evaluation metric: PER in [%]._ | _orc_ | _uns_
---|---|---
|  | _match_
---
| _nonmatch_
---
| _match_
---
| _nonmatch_
---
(a) w/o Gumbel-S. | 32.03 | 46.10 | 56.52 | 59.41
(b) w/ Gumbel-S. | 31.52 | 40.12 | 55.70 | 57.57
(c) w/ ST Gumbel-S. | 35.12 | 52.21 | 57.41 | 67.63
|  |  |  |

In all cases, using Gumbel-Softmax performs consistently better than w/o using
Gumbel-Softmax ((b) vs. (a)), which validates the effectiveness of making the
generator output sharper. However, using ST Gumbel-Softmax performs worse than
w/o using Gumbel-Softmax ((c) vs. (a)). This is counter-intuitive because ST
Gumbel-Softmax directly converts the generator output into one-hot encoding,
which is the sharpest possible output but performs the worst in the end. We
hypothesize this is because the gradient of ST Gumbel-Softmax is being
approximated that makes the model converge to a bad point.

During evaluation, we also find that the frame-wise phone posterior will
converge to different degrees of sharpness after training with different kinds
of Gumbel-Softmax. Table IV shows the average entropy of each posterior in the
frame-wise phone posterior in the evaluation mode.

TABLE IV: Average entropy of each posterior in frame-wise phone posterior after training with different Gumbel-Softmax under _orc_ /_match_ setting. | _train-set_ | _test-set_
---|---|---
uniform distribution | 3.87 | 3.87
(a) w/o Gumbel-Softmax | 0.10 | 0.11
(b) w/ Gumbel-Softmax | 0.08 | 0.09
(c) w/ st Gumbel-Softmax | 0.56 | 0.59
|  |

After GAN training, the frame-wise phone posterior converges to have the
highest entropy using ST Gumbel-Softmax. There is a large gap between ST
Gumbel-Softmax and any other method. We hypothesize that the model output
before ST Gumbel-Softmax no longer has to be a sharper posterior to fool the
discriminator because the output will be converted into a one-hot distribution
$P^{gen-st}$ during training.

###  VII-D Comparing Discriminator Architecture

In this section, we investigate how well the GAN performs under different
discriminator architectures. Four discriminator architectures are discussed
and listed below111We do not include the LSTM-based discriminator. Because the
framework is based on WGAN [9], we need to calculate second-order derivatives
during backpropagation, while the second differentiation of the LSTM module is
currently not supported in Pytorch..

####  VII-D1 transformer

Transformer [58] comprises positional encoding and stacks of multi-head
attention layer and layer normalization. We use the transformer with 128
hidden units, 8 heads for multi-head attention, and 6 stacks in total.

####  VII-D2 Conv-bank

Conv-bank [55, 56] is the concatenation of a bank of 1-D convolutional filters
with different kernel sizes. Conv-bank has been shown effective in modeling
the input data with multi-resolution properties, which is suitable for speech
data. Speech has multi-resolution properties because speech consists of words,
and words consist of different lengths of phones. We use one layer conv-bank
with kernel sizes 3, 5, 7, 9 to capture the phone patterns with different
resolutions. Hidden units are set to 256.

####  VII-D3 Conv-bank+

Conv-bank is designed to capture different phone patterns, which are possible
words. We want to further model the interaction between these words. The
interaction of words can be viewed as a kind of language model. Therefore, we
stack one more convolutional layer with kernel size 3 and channel size 1024 to
capture the relationship between neighbor words (phonetic pattern).

####  VII-D4 Deep CNN

We also use deep convolutional networks as the discriminator. To compared with
conv-bank based discriminator fairly, we set kernel size to 3 and 8 layers. In
this setting, the receptive field will be 17, which is the same as conv-bank+.

The results are shown in Table V.

TABLE V: Performance of using different discriminators. _Evaluation metric: PER in [%]._ | _orc_ | _uns_
---|---|---
|  | _match_
---
| _nonmatch_
---
| _match_
---
| _nonmatch_
---
(a) transformer | 78.01 | 79.78 | 85.31 | 82.25
(b) conv-bank | 32.23 | 41.77 | 57.24 | 61.74
(c) conv-bank+ | 31.52 | 40.12 | 55.70 | 57.57
(d) deep CNN | 53.89 | 56.50 | 69.83 | 68.19
|  |  |  |

Although the transformer and deep CNN model are considered to be more
expressive models compared to conv-bank, the performance is worse ((a)(d) vs.
(b)(c)). We hypothesize that under the condition of using a simple generator
(following the results of Section VII-B), the discriminator will distinguish
‘real phone sequence’ and ‘generated phone posterior sequence’ easily when it
is too powerful, and can not correctly guide the generator [10]. Finally, the
conv-bank+ performs consistently a little better than conv-bank ((c) vs. (b))
and performs the best among all.

###  VII-E Error Analysis

Under _orc/match_ , we also visualize the heat map of the averaged frame-wise
phone posterior to the corresponding oracle phone, which is illustrated in
Figure 6.

Figure 6: The vertical axis represents the oracle phones of the frames; the
horizontal axis represents the averaged frame-wise phone posterior. The phones
are ordered according to the phone frequency.

We observe that although the prediction of the high-frequency phones is a
little more accurate than the low-frequency phones, the low-frequency phones
still keep a certain degree of accuracy. This means our method is robust to
all phones, not only focus on the high-frequency phones. In Table VI, we also
list the top 6 prediction error phone pairs.

TABLE VI: Top prediction error phone pairs rank | oracle phone | predicted phone | error percentage (%)
---|---|---|---
#1 | ng | n | 19.9
#2 | uh | ih | 18.5
#3 | th | f | 18.4
#4 | th | sil | 17.0
#5 | z | s | 16.2
#6 | aw | aa | 14.3
|  |  |

We observe that the error mostly happens between two phones whose
pronunciations are similar, for example, _n_ and _ng_ , _z_ and _s_ , _aw_ and
_aa_.

##  VIII Compared to previous works

In this section, we compare our framework to previous methods. To compare to
previous works fairly, we follow the standard TIMIT split and the setting in
[48]. Under _match_ setting, all 3696 utterances are used as real phone
sequences. Under _nonmatch_ setting, we follow [48] to split the train and dev
set into 3000 and 1096 to serve as utterances and real phone
sequences222Because [48] does not release the nonmatch split, we follow the
split number while the actual split may be different.. During GAN training,
$\alpha$ is set to 18, and the frame-wise generator is a 256-256 DNN. Other
training details are the same as Section VII-A. In self re-training, the HMM
(monophone and triphone) training followed the standard recipes of Kaldi [59].
Linear Discriminant Analysis (LDA) and Maximum Likelihood Linear Transform
(MLLT) are applied to MFCCs for model training. While decoding, the setting of
LM is the same as the WFST decoder used after GAN training. The framework runs
for one iteration under _orc_ setting, four iterations under _uns_ setting. In
the _uns_ setting, we augment the real phone sequence by randomly removing 4%
and duplicating 11% of phones in the first iteration while do not use
augmentation after the second iteration.

TABLE VII: Comparison of different methods. _Evaluation metric: PER in [%]._
(I) Supervised
---
(a) RNN Transducer [48] | 17.70
(b) standard HMMs | 21.60
(c) _supervised generator_ | 23.24
(d) FS-RNN [60] | 13.81
(II) AUD
(e) HMM [41] | 59.39
(f) SHMM [43] | 55.69
(g) H-SHMM [44] | 55.76
(III) Unpaired
| _orc_ | _uns_
_match_ | _nonmatch_ | _match_ | _nonmatch_
(h) empirical-ODM [48] | 32.50 | 40.10 | 36.50 | 41.60
(i) proposed | 28.74 | 31.81 | 31.03 | 36.71
|  |  |  |

Three types of previous methods are compared, and the results are shown in
Table VII. In (I), all supervised approaches are trained with labeled
transcriptions on the standard TIMIT training set. _Supervised generator_ (row
(c)) and FS-RNN (row (d)) have additional access to the oracle boundary
annotations. In the _supervised generator_ (row (c)), we use the same
architecture as the frame-wise generator and use the same WFST to decode the
phone sequence, except trained by minimizing cross-entropy loss with frame-
wise phone labels. FS-RNN [60] (row (d)) is the previously proposed method,
which uses a stacked frame-level and segment-level recurrent network. In (II),
three AUD models are compared. Because AUD models only generate phone-like
acoustic tokens, which cannot be used to calculate PER, we instead calculate
‘best-mapping PER’ as followed steps:

  1. 1.

AUD models are first trained in an unsupervised manner to discover frame-wise
acoustic tokens for each utterance, including training and testing utterances.
Following [44], the acoustic token truncation number is set to 100.

  2. 2.

We use oracle frame-wise phone labels in the training set to get the best
mapping from acoustic token to phone.

  3. 3.

Utterances in the testing set are mapped into phone sequence and evaluated
PER.

‘Best-mapping PER’ has some advantages in the comparison because it uses the
ground-truth information from the ground-truth frame-level phone sequences. In
(III), row (d) is the previously proposed baseline learning from unpaired
speech and phone sequences, which is based on segmental empirical output
distribution matching [48].

In the _orc/match_ case, the PER in row (i) achieved 28.74%. Although the
performance is still far behind the strong supervised baselines in rows (a)
(b) (d), it performs close to the _supervised generator_ (23.24%) in row (c).
Both row (c) and _orc/match_ in row (i) can access the phone boundaries and
the paired utterances and phone transcriptions. The only difference is that
the utterances and transcriptions are aligned in row (c) while not aligned in
the _orc/match_ setting in row (i). This is where the performance gap comes
from. All settings of the proposed approach perform better than the ‘best-
mapping PER’ of the three AUD models and perform consistently better than the
previous baseline Segmental Empirical-ODM in row (h). Not to mention the prior
work of Segmental Empirical-ODM needs a large batch size (up to 20000 training
examples in a batch) to achieve a satisfactory performance, while the training
process here is done with batch size as small as 100.

Next, we want to determine the quantity of the labeled data required for the
standard HMM to achieve comparable results to the proposed approach,
illustrated in Fig 7. The red curve is for the standard HMMs trained with
different portions of labeled data. The red and blue dashed horizontal lines
represent the proposed method and previous baseline Segmental empirical-ODM
with _uns_ /_nonmatch_ setting.

Figure 7: Comparison of the proposed approaches to standard supervised HMMs
with varying quantity of labeled data.

We see the proposed method equals the situation when around 2.5% to 5% labeled
data is used, while Segmental empirical-ODM is only comparable to use 1% to
2.5% labeled data.

##  IX Discussion

###  IX-A Effectiveness of Self Re-training

TABLE VIII:  Performance of the framework over iterations. _Evaluation metric: PER in [%]._ | _orc_ | _uns_
---|---|---
| _match_
---
| _nonmatch_
---
| _match_
---
| _nonmatch_
---
iteration1 | (a) GAN + WFST | 29.42 | 38.7 | 65.12 | 61.13
(b) GAN + WFST + HMM | 28.74 | 31.81 | 56.26 | 49.89
iteration2 | (c) GAN + WFST | - | 48.69 | 53.42
(d) GAN + WFST + HMM | 33.96 | 41.08
iteration3 | (e) GAN + WFST | - | 43.33 | 52.31
(f) GAN + WFST + HMM | 31.47 | 37.82
iteration4 | (g) GAN + WFST | - | 41.22 | 51.43
(h) GAN + WFST + HMM | 31.03 | 36.71
|  |  |  |  |
TABLE IX: The performance improvement under different PER of the 1st iteration GAN training. _Evaluation metric: PER in [%]._ iteration 1 GAN + WFST | 61.13 | 64.98 | 67.77 | 69.99 | 72.45 | 75.00
---|---|---|---|---|---|---
iteration 2 GAN + WFST | 53.42 | 58.85 | 61.53 | 65.92 | 69.51 | 72.31
improvement | 7.71 | 6.13 | 6.24 | 4.07 | 2.94 | 2.69
|  |  |  |  |  |

In this section, we investigate how our framework benefits from self re-
training. The results over different iterations are shown in Table VIII. The
rows (a)-(h) are for the iteration 1, 2, 3 and 4 of the framework, rows (a)
(c) (e) (g) for GAN alone and decode with WFST decoder, while rows (b) (d) (f)
(h) further use HMM self re-training. We can see the performance improves
consistently after HMM self re-training at each iteration (rows (b) v.s. (a),
(d) v.s. (c), (f) v.s. (e), (h) v.s. (g)). The improvement indicates the HMM
re-training is beneficial even though the training transcription, which is the
output of GAN, may be noisy. Second, there is performance improvement after
each iteration for either GAN alone (rows (a) (c) (e) (g)) or after self re-
training (rows (b) (d) (f) (h)). Because the only difference between
iterations’ training settings is the segmentation, we hypothesize that this
improvement is due to the refinement of the phone boundaries over iterations.

When evaluating the phone segmentation quality, the first thought is to use
the F1-score, the harmonic mean of the precision and recall. However, it is
well-known that the F1-score is not suitable for segmentation because over-
segmentation may give a very high recall leading to a high F1-score, even with
a relatively low precision [12]. A naive periodic predictor, which predicts a
phone boundary for every 40 ms, can still generate boundaries with precision
0.55, recall 0.99, and F1-score 0.71. We can see the high F1 score cannot
reflect the poor quality of the predicted boundaries. Therefore, this paper
further adopts another better evaluation metric, R-value [61], which
appropriately penalizes the over-segmentation phenomenon. The R-value for the
40-ms periodic predictor is only 0.3, which gives a low score to the bad
segmentation. The result is shown in Table X.

TABLE X: phone segmentation qualities over iterations under _uns/match_ setting. Segmentation within 20-ms tolerance window is counted as correct segmentation.  | F1-score | R-value
---|---|---
40-ms periodic predictor | 0.71 | 0.3
iteration1 | 0.76 | 0.72
iteration2 | 0.79 | 0.76
iteration3 | 0.8 | 0.77
iteration4 | 0.81 | 0.78
|  |

Both F1-score and R-value consistently increase and gradually converge over
the iteration, which supports our hypothesis that self-training gives better
and better segmentation over iterations. Figure 8 also illustrates an example
utterance of how the segmentation quality changes over the iterations.

Figure 8: The segmentation refinement of an example utterance over iterations
under _uns/match_ setting. Specifically, the first row denotes the
segmentation generated from GAS; the second row denotes the segmentation after
the 1st iteration self re-training and forced alignment; the last row
represents the oracle phone segmentation. We can see the segmentation quality
increases over the iterations in two different ways: 1. the generated
segmentation is closer and closer to the oracle phone boundaries. 2. the
redundant segmentation decreases.

Under _uns_ setting, because the phone boundaries become better and better
over iterations, the training situation will be more and more similar to _orc_
setting, and finally get closer performance (Table VIII, _orc_ in (b) vs.
_uns_ in (h)).

###  IX-B Robustness of Self Re-training

In this section, we discuss how robust the self re-training is to the 1st
iteration GAN training. First, we run the GAN training nine times under
_uns/nonmatch_ setting. The mean and standard deviation of the performances
are $67.88\pm 4.27$. Then we hand-pick several checkpoints of the 1st
iteration GAN training that performs different PER. The probing range of PER
is roughly equally spaced between 61.13 (best PER) to 75 (approximately
$mean+2*std$). Each selected checkpoint is used to run the next pipeline
iteration (self re-training + 2nd iteration GAN training). The results are
shown in Table IX. We observe that the self re-training can consistently get
PER improvement even as the 1st iteration PER becomes worse (within the
probing range). This improvement becomes smaller as the PER of the 1st
iteration GAN training becomes worse.

###  IX-C Difference with the Concurrent Work: wav2vec-U

This section highlights the main difference in the design between our work and
the concurrent work wav2vec-U [62], which is based on our previous conference
work [49]. We categorize the difference into three points.

####  IX-C1 Feature

Our work uses hand-crafted features, MFCC. Wav2vec-U selects the input feature
from different layers of wave2vec 2.0 [63], a self-supervised model. The
selecting criterion is the PER by training a linear model in a supervised
manner.

####  IX-C2 Unsupervised Segmentation

Our work uses GAS [11] to get the phone-level segmentation. Wav2vec-U uses
k-means to cluster the selected features, and the boundaries are drawn
whenever the clustered index changes.

####  IX-C3 Segmentation Refinement

Our work uses HMM self re-training and forced alignment to refine the phone
boundaries. Wav2vec-U does not have explicit segmentation refinement. While
Wav2vec-U merges the neighboring segments containing the same predicted labels
in each step of GAN training, this design can be viewed as refining the
segmentation implicitly.

##  X Conclusion

In this work, we take a step close to our vision ‘unsupervised ASR’ by
proposing a two-stage iterative framework to learn phone recognition from
unpaired data. First, GAN training is shown able to learn the mapping between
two domains: speech and phone sequence. Then we categorize the generator into
segment-wise and frame-wise generators, which are shown to be good at
different settings. Frame-wise generator performs well in _uns_ setting, while
segment-wise generator, on the other hand, performs well in _orc_ setting.
Finally, self re-training is shown to improves the segmentation quality and
boost the performance, and achieves 36.7% under _uns/nonmatch_ setting over
iterations.

##  XI Acknowledgement

We thank to National Center for High-performance Computing (NCHC) of National
Applied Research Laboratories (NARLabs) in Taiwan for providing computational
and storage resources.

## References

  * [1] J. K. Chorowski, D. Bahdanau, D. Serdyuk, K. Cho, and Y. Bengio, “Attention-based models for speech recognition,” in _Advances in neural information processing systems_ , 2015, pp. 577–585.
  * [2] C.-C. Chiu, T. N. Sainath, Y. Wu, R. Prabhavalkar, P. Nguyen, Z. Chen, A. Kannan, R. J. Weiss, K. Rao, E. Gonina _et al._ , “State-of-the-art speech recognition with sequence-to-sequence models,” in _2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. IEEE, 2018, pp. 4774–4778.
  * [3] A. Gulati, J. Qin, C.-C. Chiu, N. Parmar, Y. Zhang, J. Yu, W. Han, S. Wang, Z. Zhang, Y. Wu _et al._ , “Conformer: Convolution-augmented transformer for speech recognition,” _Proc. Interspeech 2020_ , pp. 5036–5040, 2020\.
  * [4] W. Han, Z. Zhang, Y. Zhang, J. Yu, C.-C. Chiu, J. Qin, A. Gulati, R. Pang, and Y. Wu, “Contextnet: Improving convolutional neural networks for automatic speech recognition with global context,” _Proc. Interspeech 2020_ , pp. 3610–3614, 2020.
  * [5] M. Artetxe, G. Labaka, E. Agirre, and K. Cho, “Unsupervised neural machine translation,” _International Conference on Learning Representations_ , 2018\.
  * [6] A. Conneau, G. Lample, M. Ranzato, L. Denoyer, and H. Jégou, “Word translation without parallel data,” _International Conference on Learning Representations_ , 2018.
  * [7] G. Lample, A. Conneau, L. Denoyer, and M. Ranzato, “Unsupervised machine translation using monolingual corpora only,” _International Conference on Learning Representations_ , 2018.
  * [8] I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio, “Generative adversarial nets,” in _Advances in neural information processing systems_ , 2014, pp. 2672–2680.
  * [9] M. Arjovsky, S. Chintala, and L. Bottou, “Wasserstein gan,” _Proceedings of the 34th International Conference on Machine Learning_ , pp. 214–223, 2017\.
  * [10] I. Gulrajani, F. Ahmed, M. Arjovsky, V. Dumoulin, and A. Courville, “Improved training of wasserstein gans,” _Proceedings of the 31st International Conference on Neural Information Processing Systems_ , pp. 5769–5779, 2017.
  * [11] Y.-H. Wang, C.-T. Chung, and H.-y. Lee, “Gate activation signal analysis for gated recurrent neural networks and its correlation with phoneme boundaries,” _Proc. Interspeech 2017_ , pp. 3822–3826, 2017.
  * [12] P. Michel, O. Räsänen, R. Thiolliere, and E. Dupoux, “Blind phoneme segmentation with temporal prediction errors,” _Proceedings of ACL 2017, Student Research Workshop_ , pp. 62–68, 2017.
  * [13] Y. Qiao, N. Shimomura, and N. Minematsu, “Unsupervised optimal phoneme segmentation: Objectives, algorithm and comparisons,” in _2008 IEEE International Conference on Acoustics, Speech and Signal Processing_. IEEE, 2008, pp. 3989–3992.
  * [14] F. Kreuk, J. Keshet, and Y. Adi, “Self-supervised contrastive learning for unsupervised phoneme segmentation,” _Proc. Interspeech 2020_ , pp. 3700–3704, 2020.
  * [15] J. Franke, M. Mueller, F. Hamlaoui, S. Stueker, and A. Waibel, “Phoneme boundary detection using deep bidirectional lstms,” in _Speech Communication; 12. ITG Symposium_. VDE, 2016, pp. 1–5.
  * [16] O. Rasanen, “Basic cuts revisited: Temporal segmentation of speech into phone-like units with statistical learning at a pre-linguistic level,” in _Proceedings of the Annual Meeting of the Cognitive Science Society_ , vol. 36, no. 36, 2014.
  * [17] H. Kamper and B. van Niekerk, “Towards unsupervised phone and word segmentation using self-supervised vector-quantized neural networks,” _arXiv preprint arXiv:2012.07551_ , 2020.
  * [18] A. Park and J. R. Glass, “Unsupervised word acquisition from speech using pattern discovery,” in _2006 IEEE International Conference on Acoustics Speech and Signal Processing Proceedings_ , vol. 1. IEEE, 2006, pp. I–I.
  * [19] L. ten Bosch and B. Cranen, “A computational model for unsupervised word discovery,” 2007.
  * [20] G. Aimetti, “Modelling early language acquisition skills: Towards a general statistical learning mechanism,” in _Proceedings of the Student Research Workshop at EACL 2009_ , 2009, pp. 1–9.
  * [21] K. Levin, K. Henry, A. Jansen, and K. Livescu, “Fixed-dimensional acoustic embeddings of variable-length segments in low-resource settings,” in _2013 IEEE Workshop on Automatic Speech Recognition and Understanding_. IEEE, 2013, pp. 410–415.
  * [22] K. Levin, A. Jansen, and B. Van Durme, “Segmental acoustic indexing for zero resource keyword search,” in _2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. IEEE, 2015, pp. 5828–5832.
  * [23] S. Bengio and G. Heigold, “Word embeddings for speech recognition,” 2014.
  * [24] G. Chen, C. Parada, and T. N. Sainath, “Query-by-example keyword spotting using long short-term memory networks,” in _2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. IEEE, 2015, pp. 5236–5240.
  * [25] H. Kamper, W. Wang, and K. Livescu, “Deep convolutional acoustic word embeddings using word-pair side information,” in _2016 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. IEEE, 2016, pp. 4950–4954.
  * [26] W. He, W. Wang, and K. Livescu, “Multi-view recurrent neural acoustic word embeddings,” _arXiv preprint arXiv:1611.04496_ , 2016.
  * [27] S. Settle, K. Levin, H. Kamper, and K. Livescu, “Query-by-example search with discriminative neural acoustic word embeddings,” _Proc. Interspeech 2017_ , pp. 2874–2878, 2017.
  * [28] A. L. Maas, S. D. Miller, T. M. O’neil, A. Y. Ng, and P. Nguyen, “Word-level acoustic modeling with convolutional vector regression,” in _Proc. ICML Workshop Representation Learn_ , 2012.
  * [29] Y.-A. Chung and J. Glass, “Speech2vec: A sequence-to-sequence framework for learning word embeddings from speech,” _Proc. Interspeech 2018_ , pp. 811–815, 2018.
  * [30] N. Holzenberger, M. Du, J. Karadayi, R. Riad, and E. Dupoux, “Learning word embeddings: Unsupervised methods for fixed-size representations of variable-length speech segments,” 2018, pp. 2683–2687.
  * [31] H. Kamper, “Truly unsupervised acoustic word embeddings using weak top-down constraints in encoder-decoder models,” in _ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. IEEE, 2019, pp. 6535–3539.
  * [32] S. Settle and K. Livescu, “Discriminative acoustic word embeddings: Tecurrent neural network-based approaches,” in _2016 IEEE Spoken Language Technology Workshop (SLT)_. IEEE, 2016, pp. 503–510.
  * [33] Y.-A. Chung, C.-C. Wu, C.-H. Shen, H.-Y. Lee, and L.-S. Lee, “Audio word2vec: Unsupervised learning of audio segment representations using sequence-to-sequence autoencoder,” _Interspeech 2016_ , pp. 765–769, 2016\.
  * [34] E. Dunbar, R. Algayres, J. Karadayi, M. Bernard, J. Benjumea, X.-N. Cao, L. Miskic, C. Dugrain, L. Ondel, A. W. Black _et al._ , “The zero resource speech challenge 2019: Tts without t,” _Proc. Interspeech 2019_ , pp. 1088–1092, 2019.
  * [35] E. Dunbar, J. Karadayi, M. Bernard, X.-N. Cao, R. Algayres, L. Ondel, L. Besacier, S. Sakti, and E. Dupoux, “The zero resource speech challenge 2020: Discovering discrete subword and word units,” _Proc. Interspeech 2020_ , pp. 4831–4835, 2020.
  * [36] M. Chen and T. Hain, “Unsupervised acoustic unit representation learning for voice conversion using wavenet auto-encoders,” _Proc. Interspeech 2020_ , pp. 4866–4870, 2020.
  * [37] A. Baevski, S. Schneider, and M. Auli, “vq-wav2vec: Self-supervised learning of discrete speech representations,” _International Conference on Learning Representations_ , 2019.
  * [38] R. Eloff, A. Nortje, B. van Niekerk, A. Govender, L. Nortje, A. Pretorius, E. Van Biljon, E. van der Westhuizen, L. van Staden, and H. Kamper, “Unsupervised acoustic unit discovery for speech synthesis using discrete latent-variable neural networks,” _Proc. Interspeech 2019_ , pp. 1103–1107, 2019.
  * [39] J. Chorowski, R. J. Weiss, S. Bengio, and A. van den Oord, “Unsupervised speech representation learning using wavenet autoencoders,” _IEEE/ACM transactions on audio, speech, and language processing_ , vol. 27, no. 12, pp. 2041–2053, 2019.
  * [40] C.-y. Lee and J. Glass, “A nonparametric bayesian approach to acoustic model discovery,” in _Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , 2012, pp. 40–49.
  * [41] L. Ondel, L. Burget, and J. Černockỳ, “Variational inference for acoustic unit discovery,” _Procedia Computer Science_ , vol. 81, pp. 80–86, 2016.
  * [42] H. Chen, C.-C. Leung, L. Xie, B. Ma, and H. Li, “Multilingual bottle-neck feature learning from untranscribed speech,” in _2017 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU)_. IEEE, 2017, pp. 727–733.
  * [43] L. Ondel, H. K. Vydana, L. Burget, and J. Černockỳ, “Bayesian subspace hidden markov model for acoustic unit discovery,” _Proc. Interspeech 2019_ , pp. 261–265, 2019.
  * [44] B. Yusuf, L. Ondel, L. Burget, J. Černockỳ, and M. Saraçlar, “A hierarchical subspace model for language-attuned acoustic unit discovery,” in _ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. IEEE, 2021, pp. 3710–3714.
  * [45] S. Bansal, H. Kamper, A. Lopez, and S. Goldwater, “Towards speech-to-text translation without speech recognition,” _EACL 2017_ , p. 474, 2017.
  * [46] D.-R. Liu, K.-Y. Chen, H.-Y. Lee, and L.-s. Lee, “Completely unsupervised phoneme recognition by adversarially learning mapping relationships from audio embeddings,” _Proc. Interspeech 2018_ , pp. 3748–3752, 2018.
  * [47] Y. Liu, J. Chen, and L. Deng, “Unsupervised sequence classification using sequential output statistics,” in _Advances in Neural Information Processing Systems_ , 2017, pp. 3550–3559.
  * [48] C.-K. Yeh, J. Chen, C. Yu, and D. Yu, “Unsupervised speech recognition via segmental empirical output distribution matching,” _International Conference on Learning Representations_ , 2018.
  * [49] K.-Y. Chen, C.-P. Tsai, D.-R. Liu, H.-Y. Lee, and L.-s. Lee, “Completely unsupervised speech recognition by a generative adversarial network harmonized with iteratively refined hidden markov models,” _Proc. Interspeech 2019_ , pp. 1856–1860, 2019.
  * [50] Y.-A. Chung, W.-H. Weng, S. Tong, and J. Glass, “Unsupervised cross-modal alignment of speech and text embedding spaces,” _Advances in Neural Information Processing Systems_ , vol. 31, pp. 7354–7364, 2018.
  * [51] ——, “Towards unsupervised speech-to-text translation,” in _ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , 2019, pp. 7170–7174.
  * [52] E. Jang, S. Gu, and B. Poole, “Categorical reparameterization with gumbel-softmax,” _arXiv preprint arXiv:1611.01144_ , 2016.
  * [53] C. J. Maddison, A. Mnih, and Y. W. Teh, “The concrete distribution: A continuous relaxation of discrete random variables,” _Proceedings of the international conference on learning Representations_ , 2017.
  * [54] E. Gumbel, _Statistical Theory of Extreme Values and Some Practical Applications: A Series of Lectures_ , ser. Applied mathematics series. U.S. Government Printing Office, 1954. [Online]. Available: https://books.google.com.tw/books?id=SNpJAAAAMAAJ
  * [55] Y. Wang, R. Skerry-Ryan, D. Stanton, Y. Wu, R. J. Weiss, N. Jaitly, Z. Yang, Y. Xiao, Z. Chen, S. Bengio _et al._ , “Tacotron: Towards end-to-end speech synthesis,” _Proc. Interspeech 2017_ , pp. 4006–4010, 2017.
  * [56] J. Lee, K. Cho, and T. Hofmann, “Fully character-level neural machine translation without explicit segmentation,” _Transactions of the Association for Computational Linguistics_ , vol. 5, pp. 365–378, 2017.
  * [57] L. Liu, H. Jiang, P. He, W. Chen, X. Liu, J. Gao, and J. Han, “On the variance of the adaptive learning rate and beyond,” _International Conference on Learning Representations_ , 2019.
  * [58] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, “Attention is all you need,” in _Advances in neural information processing systems_ , 2017, pp. 5998–6008.
  * [59] D. Povey, A. Ghoshal, G. Boulianne, L. Burget, O. Glembek, N. Goel, M. Hannemann, P. Motlicek, Y. Qian, P. Schwarz, J. Silovsky, G. Stemmer, and K. Vesely, “The kaldi speech recognition toolkit,” in _IEEE 2011 Workshop on Automatic Speech Recognition and Understanding_. IEEE Signal Processing Society, Dec. 2011, iEEE Catalog No.: CFP11SRW-USB.
  * [60] M. Ratajczak, S. Tschiatschek, and F. Pernkopf, “Frame and segment level recurrent neural networks for phone classification.” in _Interspeech_ , 2017, pp. 1318–1322.
  * [61] O. J. Räsänen, U. K. Laine, and T. Altosaar, “An improved speech segmentation quality measure: the r-value,” in _Tenth Annual Conference of the International Speech Communication Association_ , 2009.
  * [62] A. Baevski, W.-N. Hsu, A. Conneau, and M. Auli, “Unsupervised speech recognition,” _arXiv preprint arXiv:2105.11084_ , 2021.
  * [63] A. Baevski, H. Zhou, A. Mohamed, and M. Auli, “wav2vec 2.0: A framework for self-supervised learning of speech representations,” _Advances in Neural Information Processing Systems_ , vol. 33, 2020.

|  Da-rong Liu Da-rong Liu, received the Bachelor degree from National Taiwan
University (NTU) in 2016, and is now a P.h. D. student at the Graduate
Institute of Communication Engineering (GICE) at National Taiwan University.
He mainly works on unsupervised learning, speech recognition and speech
generation.
---|---
|  Po-chun Hsu received the B.S. degree from National Taiwan University (NTU)
in 2018 and is now a P.h. D. student at the Graduate Institute of
Communication Engineering (GICE) at NTU. His research focuses on speech
synthesis, including text-to-speech (TTS), voice conversion (VC), and neural
vocoder.
---|---
|  Yi-chen Chen Yi-chen Chen, received the Bachelor degree from National
Taiwan University (NTU) in 2017, and is now a P.h. D. student at the Graduate
Institute of Communication Engineering (GICE) at National Taiwan University,
working on self-supervised/semi-supervised/transfer learning and speech
processing.
---|---
|  Sung-Feng Huang Sung-Feng Huang, received the Bachelor degree from National
Taiwan University (NTU) in 2017, and is now a P.h. D. student at the Graduate
Institute of Communication Engineering (GICE) at National Taiwan University.
He mainly works on learning representations unsupervisedly, speech
recognition, spoken term detection, speech separation, meta learning and
machine learning techniques.
---|---
|  Shun-Po Chuang joined the Speech Processing and Machine Learning Laboratory
in 2016. He is currently working toward the Ph.D. degree at National Taiwan
University, Taipei, Taiwan. His research focuses on speech recognition,
speech-to-text translation and code-switching language modeling.
---|---
|  Da-yi Wu Da-Yi Wu received the B.S and M.S degrees from National Taiwan
University, Taipei, Taiwan, in 2014 and 2018, respectively. He is currently a
research assistant for NTU Speech Processing Laboratory. His research focuses
on voice conversion, singing generation, and machine learning.
---|---
|  Hung-yi Lee received the M.S. and Ph.D. degrees from National Taiwan
University, Taipei, Taiwan, in 2010 and 2012, respectively. From September
2012 to August 2013, he was a Postdoctoral Fellow with Research Center for
Information Technology Innovation, Academia Sinica. From September 2013 to
July 2014, he was a Visiting Scientist with the Spoken Language Systems Group,
MIT Computer Science and Artificial Intelligence Laboratory. He is currently
an Associate Professor with the Department of Electrical Engineering, National
Taiwan University, Taipei, Taiwan with a joint appointment to the Department
of Computer Science and Information Engineering. His research focuses on
spoken language understanding, speech recognition, and machine learning.
---|---