#  Reborn: Reinforcement-Learned Boundary Segmentation
with Iterative Training for Unsupervised ASR

Liang-Hsuan Tseng  En-Pei Hu  Cheng-Han Chiang  Yuan Tseng  Hung-yi Lee  Lin-
shan Lee  Shao-Hua Sun

###### Abstract

Unsupervised automatic speech recognition (ASR) aims to learn the mapping
between the speech signal and its corresponding textual transcription without
the supervision of paired speech-text data. A word/phoneme in the speech
signal is represented by a segment of speech signal with variable length and
unknown boundary, and this segmental structure makes learning the mapping
between speech and text challenging, especially without paired data. In this
paper, we propose Reborn, Reinforcement-Learned Boundary Segmentation with
Iterative Training for Unsupervised ASR. Reborn alternates between (1)
training a segmentation model that predicts the boundaries of the segmental
structures in speech signals and (2) training the phoneme prediction model,
whose input is a segmental structure segmented by the segmentation model, to
predict a phoneme transcription. Since supervised data for training the
segmentation model is not available, we use reinforcement learning to train
the segmentation model to favor segmentations that yield phoneme sequence
predictions with a lower perplexity. We conduct extensive experiments and find
that under the same setting, Reborn outperforms all prior unsupervised ASR
models on LibriSpeech, TIMIT, and five non-English languages in Multilingual
LibriSpeech. We comprehensively analyze why the boundaries learned by Reborn
improve the unsupervised ASR performance.222Our code will be available soon.

Machine Learning, ICML

##  1 Introduction

Automatic speech recognition (ASR) systems convert speech signals into their
transcription texts. Most state-of-the-art (SoTA) ASR systems are supervisedly
trained with a large amount of labeled speech-text paired data (Chen et al.,
2021; Radford et al., 2023). However, more than 80% of languages in the world
have limited access to speech-text paired data (Pratap et al., 2023), and
collecting such paired data requires intensive labor and costs, limiting the
applicability of supervised ASR systems to these languages. This leads to
significant efforts devoted to developing unsupervised ASR (UASR) systems,
which aim to learn the mapping between speech signals and textual
transcriptions (words or phonemes) without any speech-text pairs (Liu et al.,
2018; Chen et al., 2019; Baevski et al., 2021; Liu et al., 2022; Gao et al.,
2023).

UASR models learn to align the distribution of input speech signal and output
text without paired data. Learning to match two distributions of sequences
unsupervisedly has been extensively studied in unsupervised neural machine
translation (NMT) (Artetxe et al., 2018), where the aim is to learn a neural
network that can translate text in a source language to text in a target
language without paired data (Lample et al., 2018a, b). Prior works show that
adversarial training (Goodfellow et al., 2014) can be used to learn such
mappings (Lample et al., 2018a, b), which employs a generator learning to
translate and a discriminator learning to distinguish generated text from the
real text in the target language.

Can we adopt such an adversarial training scheme for UASR? In unsupervised
NMT, text can be easily tokenized into sub-word tokens, so unsupervised NMT
only needs to learn the mapping between the source and target language’s token
embeddings. However, in UASR, a phoneme or word is represented by a variable-
length segment in the speech signal whose boundaries are unknown. Moreover,
the length of a speech signal is much longer than the length of its textual
transcription. The above characteristics of speech make learning the mapping
between the segmental structures in speech and textual transcription
challenging. Existing works in unsupervised ASR rely on heuristics or
handcrafted rules to obtain the boundaries of the segmental structures (Liu et
al., 2018; Yeh et al., 2018; Baevski et al., 2021). Yet, such handcrafted
boundaries are often sub-optimal, bottlenecking the performance of UASR.

This paper focuses on obtaining better segmental boundaries to improve the
mapping between segmental structure and textual transcription. We propose
Reborn (Reinforcement-Learned Boundary Segmentation with Iterative Training),
an unsupervised ASR framework with a segmentation model and a phoneme
prediction model. The segmentation model determines segmental structure
boundaries in speech signals, while the phoneme prediction model assigns a
phoneme to each segmental structure. We use an iterative algorithm that
alternates between two stages to train the segmentation model and the phoneme
prediction model: The first stage trains the segmentation model. Since we do
not have the ground truth segmental boundaries, we use reinforcement learning
(RL) to provide unsupervised guidance for training the segmentation model. We
experiment with various learning objectives and implement them as reward
functions for learning the segmentation model. In the second stage, based on
the boundaries predicted by the segmentation model learned in the previous
stage, we use an adversarial loss similar to generative adversarial networks
(GANs) (Goodfellow et al., 2014) to train the phoneme prediction model.

We conduct extensive experiments on LibriSpeech (Panayotov et al., 2015),
TIMIT (Garofolo, 1993), and Multilingual LibriSpeech (MLS) (Pratap et al.,
2020) to compare the phoneme error rate (PER) and word error rate (WER) with
prior works. We show that our method outperforms all prior UASR methods on
LibriSpeech, TIMIT, and five languages in MLS when using the same amount of
training data. We perform thorough ablation studies to show that iterative
training and the rewards we design are critical to the performance of Reborn.
By analyzing the segmental structure obtained by our segmentation model, we
find that the segmental structures are acoustic units smaller than phonemes,
which helps the phoneme prediction model predict more accurate transcriptions.

##  2 Related Work: Unsupervised ASR (UASR)

An ASR model takes speech signals as the input and predicts the textual
transcriptions. Unsupervised ASR aims to train an ASR model without access to
paired speech-text data. Instead, the only available training data is
unlabeled speech data and unlabeled text data, while the correspondence
between speech and text is not available. In this paper, we follow prior UASR
works (Liu et al., 2018; Baevski et al., 2021) to predict phoneme
transcriptions from speech signals. Learning the mapping between speech
signals and phoneme sequences can be formulated as a distribution matching
problem, where we want to learn an ASR model whose output phoneme sequences
match the distribution of real phoneme sequences.

Yeh et al. (2018) solve this distribution matching problem by an unsupervised
loss function based on the empirical output distribution matching (ODM) (Liu
et al., 2017), which guides the ASR model to output phoneme sequences that
have statistic distributions close to real phoneme sequences.

Liu et al. (2018) propose to use the generative adversarial network (GAN)
(Goodfellow et al., 2014) for UASR. GAN solves the distribution matching
problem by training a generator whose output distribution resembles a target
distribution, and a discriminator is trained to distinguish the outputs of the
generator and the samples from the target distribution. When using GAN to
train unsupervised ASR, the generator is the phoneme prediction model. The
phoneme prediction model takes in speech features of the segmental structures
in the speech signal, and outputs a phoneme transcription. Prior works rely on
heuristics or separately trained unsupervised phoneme segmentation models to
find the boundary of the segmental structures (Wang et al., 2018; Liu et al.,
2018; Chen et al., 2019).

With the emergence of foundation models, wav2vec-U (Baevski et al., 2021)
shows that UASR performance can greatly benefit from using speech foundation
models as the feature extractor. However, wav2vec-U still relies on some
heuristic feature preprocessing steps based on $k$-means to find the
boundaries of the segmental structure in the speech signal. In our paper, we
call the segmentation boundaries obtained by the feature preprocessing in
wav2vec-U as ”$k$-means segmentation”. EURO (Gao et al., 2023) changes the
decoding algorithm used to decode the phoneme prediction sequences and
explores other speech foundation models for feature extraction, including
HuBERT (Hsu et al., 2021) and WavLM (Chen et al., 2022).

In these prior methods, the segmentation is either non-trainable or learned
separately from the phoneme prediction model. Reborn differs from prior works
by introducing a trainable segmentation model tailored for the phoneme
prediction model, and the segmentation model and phoneme prediction model can
be iteratively learned. wav2vec-U 2.0 (Liu et al., 2022) simplifies the
feature preprocessing step in wav2vec-U and does not explicitly consider the
segmentation structure in their model. Reborn has performance better than
wav2vec-U 2.0 on almost all datasets, showing that learning the segmentation
boundary for feature preprocessing is important to the performance of UASR.

Figure 1: (a) and (b): Reborn iterates between using RL to train the
segmentation model and using adversarial training to train the phoneme
prediction model. (c): An illustration of the segmentation post-processing.
Given the original segmentation and the predicted phoneme sequence, we merge
the segments that result in the same phoneme prediction into the same segment,
yielding the post-processed boundary.

##  3 Method: Reborn

The difficulty of mapping speech signals to their corresponding phoneme
transcriptions lies in the segmental structures in speech whose boundaries are
unknown. To tackle this challenge, we propose Reborn, which trains a UASR
system with unpaired speech and text data. Reborn contains a segmentation
model and a phoneme prediction model. The segmentation model takes the speech
feature as input and determines the boundaries of the segmental structures in
the speech signal, and the phoneme prediction model predicts a phoneme for
each segmental structure. We will use the terms segment and segmental
structure interchangeably.

The overall training process of Reborn is outlined in Algorithm 1 and Figure
1. First, we initialize the phoneme prediction model using wav2vec-U (Section
3.3). Next, the training of Reborn iterates through two stages: Stage 1
(Figure 1(a), Section 3.1): Train the segmentation model to learn a better
segmentation until the segmentation model converges while fixing the phoneme
prediction model. Stage 2 (Figure 1(b), Section 3.2): Training the phoneme
predictor based on the segment boundaries predicted by the segmentation model
until the phoneme prediction model converges. The iterative process ceases
when the UASR performance does not improve over the previous iteration. In
UASR, we cannot use PER as a validation metric to select the best model or
determine when to stop training. We use an unsupervised evaluation metric to
select the best model, detailed in Appendix C.2.

###  3.1 Stage 1: Training the Segmentation Model

#### 3.1.1 Segmentation Model

Given an input speech signal, we use a self-supervised feature extractor model
to extract the speech features $X=[x_{1},\cdots,x_{T}]$ from the waveform. The
segmentation model takes the speech features and predicts the boundary of the
segmental structure in the speech features. For each feature $x_{i}$, the
segmentation model assigns a binary value $\hat{b}_{i}$, indicating whether
$x_{i}$ is the first frame of a segment. We split $X$ into segments based on
$\hat{b}_{1:T}$ and mean-pool the features within the same segment. The mean-
pooled features $[s_{1},\cdots,s_{T^{\prime}}]$ are forwarded through the
phoneme prediction model to obtain the phoneme prediction
$[y_{1},\cdots,y_{T^{\prime}}]$. The resulting phoneme prediction will be de-
duplicated into $[y^{\prime}_{1},\cdots,y^{\prime}_{M}]$ by removing the same
consecutive phoneme.

The segmentation model is a one-dimensional CNN, denoted by $\pi_{\theta}$ and
parametrized by $\theta$. Given $[x_{1},\cdots,x_{T}]$, $\pi_{\theta}$
predicts a probability $\pi_{\theta}[i],$ $\forall i\in[1,\cdots,T]$. Let
$\hat{B}_{i}\sim\text{Bernoulli}(\pi_{\theta}[i])$ be a Bernoulli random
variable representing whether $x_{i}$ is the beginning of a segment.
$\hat{B}_{i}=1$ means $x_{i}$ is the beginning of a segment; $\hat{B}_{i}=0$
means $x_{i}$ is not the beginning of a segment. During training, we sample
$\hat{b}_{i}$ from $\text{Bernoulli}(\pi_{\theta}[i])$; during inference, we
take $\hat{b}_{i}=1$ if $\pi_{\theta}[i]\geq 0.5$, otherwise $\hat{b}_{i}=0$.

#### 3.1.2 Training the Segmentation Model with RL

To help the prediction of the phoneme prediction model, we want to train the
segmentation model to capture the segmental structure in speech. However, the
optimal segmentation boundary is not available for training. Recognizing that
the segmentation quality directly affects the phoneme prediction of the
phoneme prediction model, we estimate the quality of the phoneme prediction to
guide the segmentation model training. The phoneme prediction model is from
the previous Reborn iteration or initialized from wav2vec-U in the first
Reborn iteration, and it is fixed when training the segmentation model. Since
the effect of feature segmentation on phoneme prediction is non-
differentiable, we employ RL to train the segmentation model. The estimated
quality of the phoneme sequence serves as the RL reward.

We train the segmentation model using the policy gradient method (Sutton et
al., 1999). For each utterance, we calculate an utterance-wise reward $R$
(defined in Eq. 4 in the next subsection) from the de-duplicated phoneme
prediction to train the segmentation model. Based on the policy gradient
method, the segmentation model $\pi_{\theta}$ is optimized using the following
gradient:
$\mathbb{E}\left[\sum_{t=1}^{T}\nabla_{\theta}\log\pi_{\theta}(\hat{b}_{t}|X)R\right]$,
where $\mathbb{E}$ is taken over the training data distribution and
approximated with the mean of a batch of training sequences.

#### 3.1.3 Reward Designs

Given an utterance in the training set, the utterance-wise reward $R$ is the
weighted sum of perplexity difference reward $R_{\text{ppl}}$, edit-distance
reward $R_{\text{edit}}$, and length difference reward $R_{\text{len}}$.

We introduce some notations: Given an utterance, we use the currently trained
segmentation model $\pi_{\theta}$ for segmentation and the phoneme prediction
model trained in the last Reborn iteration to obtain a deduplicated phoneme
prediction sequence, denoted by $Y^{\prime}_{\theta}$. For the same utterance,
we can use the segmentation model trained in the previous Reborn iteration
$\pi_{\theta-1}$ for features segmentation and the same phoneme prediction
model to obtain another de-duplicated phoneme prediction sequence, denoted as
$Y^{\prime}_{\theta-1}$. In the first Reborn iteration,
$Y^{\prime}_{\theta-1}$ is the de-duplicated phoneme prediction from
wav2vec-U.

##### Perplexity Difference Reward

The perplexity difference reward is designed to favor phoneme segmentation
better than the segmentation learned in the previous iteration. Intuitively, a
better phoneme segmentation prediction $\hat{b}_{1:T^{\prime}}$ should yield a
more reasonable phoneme prediction $y_{1:T^{\prime}}$. We use perplexity
(PPL), the negative likelihood of a phoneme sequence scored by a phoneme
language model (LM), to evaluate how reasonable a phoneme sequence is.
Perplexity measures how likely a phoneme sequence can be observed in the real
world; a phoneme sequence with a lower perplexity means it is more likely to
be observed in real-world phoneme datasets. We use KenLM (Heafield, 2011) as
the phoneme LM, where the LM is trained with the unlabeled text corpora. To
guide the segmentation model to generate a segmentation with a better phoneme
prediction than the phoneme predictions obtained in the previous iteration, we
define perplexity difference reward as follows:

$R_{\text{ppl}}=\text{PPL}_{\theta-1}-\text{PPL}_{\text{$\theta$}},$ (1)

where $\text{PPL}_{\theta-1}$ is the perplexity of $Y^{\prime}_{\theta-1}$ and
$\text{PPL}_{\theta}$ is the perplexity of $Y^{\prime}_{\theta}$. The
perplexity difference reward guides the segmentation model to produce
segmentation that results in a phoneme sequence with lower perplexity.

When only using $R_{\text{ppl}}$, the segmentation model learns some
segmentation method that leads to phoneme predictions with more $<$silence$>$
token, which has a lower perplexity but does not correspond to better phoneme
prediction results. To avoid such unwanted behaviors, we design two
regularization rewards, edit distance reward and length difference reward, to
ensure that the policy learned by the segmentation model does not deviate too
much from the segmentation learned in the previous iteration. The two
regularization rewards punish the segmentation model if $Y^{\prime}_{\theta}$
is too different from $Y^{\prime}_{\theta-1}$. Crucially, regularization works
only if the $Y^{\prime}_{\theta-1}$ in the first iteration is good enough. As
previously mentioned, we ensure this by using the predictions from a trained
wav2vec-U as $Y^{\prime}_{\theta-1}$ in the first iteration.

##### Edit Distance Reward

We use Levenshtein distance $d_{\text{Lev}}$ as the edit distance to calculate
the difference between $Y^{\prime}_{\theta-1}$ and $Y^{\prime}_{\theta}$. We
take the normalized negative edit distance as the reward, which is defined as
follows:

$R_{\text{edit}}=-\frac{d_{\text{Lev}}(Y^{\prime}_{\theta-1},Y^{\prime}_{\theta})}{|Y^{\prime}_{\theta-1}|}$
(2)

##### Length Difference Reward

We use the length difference reward $R_{\text{len}}$ to guide the segmentation
model to predict segmentation such that the length of $Y^{\prime}_{\theta}$
does not differ significantly from the length of $Y^{\prime}_{\theta-1}$. The
length difference reward for an utterance is defined as:

$R_{\text{len}}=1-\frac{\left||Y^{\prime}_{\theta}|-|Y^{\prime}_{\theta-1}|\right|}{|Y^{\prime}_{\theta-1}|},$
(3)

where $|Y^{\prime}|$ is the length of $Y^{\prime}$.

##### Utterance-Wise Reward

The final utterance-wise reward $R$ is the weighted sum of $R_{\text{ppl}}$,
$R_{\text{edit}}$, and $R_{\text{len}}$.

$R=c_{\text{ppl}}\cdot R_{\text{ppl}}+c_{\text{edit}}\cdot
R_{\text{edit}}+c_{\text{len}}\cdot R_{\text{len}},$ (4)

where $c_{\text{ppl}}$, $c_{\text{edit}}$ and $c_{\text{len}}$ are the
weighting coefficients. During training, $R_{\text{ppl}}$, $R_{\text{edit}}$
and $R_{\text{len}}$ are normalized within each training batch.Appendix C.2
discusses how the coefficients are determined without speech-text paired data.

#### 3.1.4 Initializing $\pi_{\theta}$ with Behavior Cloning

Before training the segmentation model $\pi_{\theta}$ with RL, we initialize
it with behavior cloning (BC). BC uses supervised training to train the
segmentation model to predict the boundaries in the speech features. Given
speech features $x_{1:T}$, the segmentation model is trained to predict the
0/1 boundary $\hat{b}_{i}$ by using some boundaries $b_{i}$ as the ground
truth. The ground truth boundary is the post-processed boundary (Section
3.1.5) obtained using $\pi_{\theta-1}$, the segmentation model learned in the
previous iteration. The BC process is a binary classification task that can be
trained with cross-entropy loss. In the first Reborn iteration, we use the
$k$-means segmentation boundary from wav2vec-U as the BC target.

#### 3.1.5 Boundary Post-Processing

After training the segmentation model until convergence, we use the
segmentation model to predict the boundaries of the whole dataset and post-
process those boundaries. The post-processing step uses the phoneme prediction
model trained in the previous iteration to refine the boundary predictions
$\hat{b}_{i}$. Even if some consecutive speech features are split into two
segments by the segmentation model, the mean-pooled features of the two
segments may yield the same phoneme prediction. In this case, the two segments
will be merged into one segment. An illustration of post-processing is shown
in Figure 1(c). Post-processing differs from phoneme sequence de-duplication:
post-processing modifies the segmentation prediction $\hat{b}_{i}$, while de-
duplication modifies the phoneme sequence.

###  3.2 Stage 2: Training the Phoneme Prediction Model

The phoneme predictor takes the mean-pooled features of segmental structures
$S=[s_{1},s_{2},...,s_{T^{\prime}}]$ and predicts a phoneme sequence
$Y=[y_{1},y_{2},...,y_{T^{\prime}}]$. $S$ are obtained by mean-pooling the
features $X$ in the same segmental structure based on
$[\hat{b}^{{}^{\prime}}_{1},\cdots,\hat{b}^{{}^{\prime}}_{T^{\prime}}]$, where
$\hat{b}^{{}^{\prime}}_{i}$ is the post-processed boundaries. We find that it
is essential to perform boundary post-processing to stabilize the training in
this stage.

After obtaining the post-processed phoneme boundaries, we train the phoneme
predictor model using GAN training. The phoneme prediction model is the
generator in GAN training. The generator aims to output phoneme predictions
that look like real phoneme sequences to fool the discriminator. The
discriminator takes in a phoneme sequence, which can be the output of the
generator or a phoneme sequence from the unpaired text corpora, and the goal
of the discriminator is to distinguish whether the input phoneme sequences are
outputs of the generator. The generator and discriminator are updated using
the loss in GAN training. In this stage, the parameters of the segmentation
model are not updated.

###  3.3 Initialization of Reborn

In Stage 1, the segmentation model depends on the phoneme prediction when
calculating the rewards. As a result, Reborn cannot work without properly
initializing the phoneme prediction model. We use wav2vec-U to train a phoneme
prediction model and use it as the initialization for the phoneme prediction
model in Reborn. We briefly introduce wav2vec-U in Appendix A for interested
readers.

Algorithm 1 Training process of Reborn

1:Initialization: Use wav2vec-U to obtain a phoneme predictor (§ 3.3)

2:while True do

3: Stage 1: Initialize segmentation model with behavior cloning (§ 3.1.4) and
then train using RL (§ 3.1)

4: Post-process boundaries (§ 3.1.5)

5: Stage 2: Train phoneme prediction model (§ 3.2)

6: if no improvement (§ C.2) then break

7: end if

8:end while

##  4 Experiment Setup

###  4.1 Dataset

We use three datasets commonly used in ASR to evaluate the performance of
Reborn.

LibriSpeech (Panayotov et al., 2015) is an English speech recognition corpora
that contains 960 hours of training data. Following EURO (Gao et al., 2023),
we use 100 hours of audio from the train-clean-100 set as the unlabeled speech
data. The unlabeled text data does not overlap with the transcription of the
unlabeled speech data.

TIMIT (Garofolo, 1993) is another English speech recognition with the human-
labeled phoneme boundary. We follow the matched setting, which is more broadly
used (Liu et al., 2018; Baevski et al., 2021; Gao et al., 2023), where the
speech and text data come from the same set of utterances.

Multilingual LibriSpeech (MLS) (Pratap et al., 2020) is an ASR dataset
including German (de), Dutch (nl), French (fr), Spanish (es), Italian (it),
and Portuguese (pt). Following (Baevski et al., 2021), we use 100 hours of
speech data for each language and use the LM data provided by the dataset as
unpaired text.

###  4.2 Evaluation Metrics

We use phoneme error rate (PER) and word error rate (WER) to evaluate the ASR
performance. If not specified, we use greedy decoding to obtain phoneme-level
results. For decoding word-level outputs, we perform WFST decoding (Mohri,
1997; Mohri et al., 2002) using PyKaldi (Can et al., 2018). We leave the
details in Appendix C.3.

###  4.3 Implementation Details

For English dataset, we use wav2vec 2.0 (Baevski et al., 2020) to extract
speech features from speech signals; for MLS, we use XLSR-53 (Conneau et al.,
2021) as the feature extractor. The segmentation model is a two-layer one-
dimensional CNN model, detailed in Appendix C.1. For LibriSpeech and TIMIT, we
train Reborn for two iterations. For MLS, we only train one iteration since we
do not find the performance to improve in the second iteration.

##  5 Results

###  5.1 Main Results

We show the PER and WER of LibriSpeech, TIMIT, and MLS in Table 1, Table 2,
and Table 3. We compare Reborn with several prior UASR works: wav2vec-U
(Baevski et al., 2021), wav2vec-U 2.0 (Liu et al., 2022), and EURO (Gao et
al., 2023). We have the following observations:

Table 1: PER/WER on LibriSpeech using 100 hours speech data. $\dagger$: Our
reproduction. (wav2vec-U and wav2vec-U 2.0 only report results of using 960
hours of unlabeled speech).

Approach PER/WER (%) $\downarrow$ dev-clean dev-other test-clean test-other
With Oracle Boundary Train from oracle 6.3/12.8 9.7/16.3 6.4/12.7 10.0/16.8
Baseline EURO (HuBERT) 15.2/23.1 20.7/29.3 15.1/22.8 21.1/29.8 wav2vec-U†
19.3/20.4 22.9/25.6 19.3/21.0 23.2/25.8 wav2vec-U 2.0† 12.2/17.2 16.3/21.7
12.6/17.7 16.3/22.2 wav2vec-U 2.0 + HMM ST † 10.0/15.4 13.1/19.0 10.3/16.0
13.1/19.6 Our Method Reborn 8.3/12.5 11.9/17.6 8.9/13.1 12.5/18.7 Reborn \+
HMM ST 5.2/9.3 8.5/13.5 5.4/9.6 8.5/13.7

#####  Reborn significantly outperforms all prior methods on LibriSpeech

Our experiment on LibriSpeech follows Gao et al. (2023) to use the 100-hour
training split of LibriSpeech as the unlabeled speech data. Compared with all
the baselines in Table 1, Reborn achieves the lowest PER and WER. wav2vec-U
and EURO both use heuristics and hand-crafted rules to obtain the segmental
boundaries, while wav2vec-U 2.0 removes the feature segmentation steps. The
superior performance of Reborn illustrates the importance of learning the
segmental boundaries in UASR. Notably, without using HMM self-training, Reborn
already has PER/WER lower than wav2vec-U 2.0 with HMM self-training, which is
the prior SoTA method on LibriSpeech.

##### Self-training further improves PER/WER of Reborn

Self-training uses the phoneme predictor’s prediction as the pseudo-label and
trains a new phoneme prediction model using the pseudo-label as the training
data. It is commonly used in UASR to boost the performance (Baevski et al.,
2021; Liu et al., 2022). In Table 1, we show that Reborn can be integrated
with HMM self-training to further lower the PER/WER. We reach a new SoTA on
LibriSpeech under the setting of 100-hour speech data, outperforming the prior
SoTA in PER and WER by 5% and 6%, respectively. Surprisingly, Reborn with HMM
self-training outperforms training wav2vec-U with the oracle boundary. This
shows that Reborn can be combined with self-training to improve performance
effectively. Due to limited computation resources, we only conduct self-
training on LibriSpeech.

Table 2: PER results on TIMIT. The cross-mark ( ✗) in the greedy-decoding
column indicates that an additional LM is used during decoding.

Approach Greedy decoding PER (%) core-dev core-test all-test (a) Train from
oracle ✓ 9.1 10.3 9.2 (b) wav2vec-U (reproduced) ✓ 20.2 22.2 20.3 (c)
wav2vec-U+WFST ✗ 17.1 17.8 16.8 (d) EURO (wav2vec 2.0) ✗ 18.5 19.8 - (e) EURO
(WavLM) ✗ 14.3 14.6 - (f) Reborn ✓ 12.4 13.5 12.4

#####  Reborn outperforms all prior UASR methods on TIMIT

Table 2 shows the PER of TIMIT, and we again find that Reborn achieves the
lowest PER compared with all prior UASR methods. This indicates that Reborn
not only works on large datasets like LibriSpeech but also works on small
datasets like TIMIT, which only contains about 3 hours of audio data for
training. Even using greedy decoding only, Reborn outperforms the prior best-
performing UASR model (row (e)), which relies on prefix decoding. EURO shows
that replacing the feature extractor with WavLM (Chen et al., 2022) (row (e))
outperforms using wav2vec 2.0 (row (d)) as the speech feature extractor. Since
Reborn using wav2vec 2.0 already outperforms EURO with WavLM, we leave
changing the feature extractor in Reborn as future work.

Table 3: PER/WER on MLS. $\dagger$: Results from Baevski et al. (2021). $*$:
Our reproduction of wav2vec-U, used as the initialization of the phoneme
prediction model in Reborn. $\ddagger$: Results from Liu et al. (2022).

Approach WER (%) $\downarrow$ de nl fr es it pt Avg. wav2vec-U† 32.5 40.2 39.8
33.3 58.1 59.8 44.0 wav2vec-U∗ 33.9 38.1 37.7 33.1 51.8 59.4 42.3 wav2vec-U
2.0‡ 23.5 35.1 35.7 25.8 46.9 48.5 35.9 Reborn 20.9 26.9 28.2 24.7 39.9 51.5
32.0

Figure 2: Comparing Reborn segmentation (a) with oracle boundaries (c) on an
example (”It’s healthier to cook…”) from TIMIT test set. The ground truth
phoneme transcription for each segment is given in (b). Silences are denoted
with a short dash (-). The red crosses indicate wrong phonemes w.r.t. the
ground truth; while the green ticks indicate the corrected ones. The dotted
lines illustrate the delayed phenomenon as mentioned in Section 5.2.

#####  Reborn performs well on languages other than English

The results on MLS are presented in Table 3. We find out that a single
iteration is enough for performance convergence on MLS. Recall that Reborn is
initialized from wav2vec-U, but Reborn outperforms wav2vec-U by a large margin
in all six languages. This large performance gap underlines the importance of
learning the boundaries of the segmental boundaries. Moreover, Reborn
outperforms wav2vec-U 2.0 except for Portuguese, and the average WER on the
six languages is 3.9% lower than wav2vec-U 2.0.

###  5.2 Boundary Analysis

The core of Reborn is the segmentation model that learns the segmental
structure’s boundaries. In this section, we look at how the quality of
segmental boundaries affects the performance of phoneme prediction. We compare
the boundaries obtained by five methods: (a) the $k$-means segmentation
boundary in wav2vec-U, (b) the oracle phoneme boundaries obtained by forced
alignment (McAuliffe et al., 2017), which requires paired speech-text data to
learn the alignment; (c) the phoneme boundary obtained by the SoTA
unsupervised phoneme segmentation method (Strgar & Harwath, 2022); (d) the
boundaries learned by the segmentation model of Reborn in the first iteration
before post-processing and (e) after post-processing.

Table 4: Boundary evaluation results of different segmentation methods on
LibriSpeech test-clean split. The second-last column (Freq.) is the number of
segments per second. All the methods shares the same phoneme prediction model
trained with the $k$-means segmentation of wav2vec-U.

Boundary method Prec. Rec. F1 Freq. PER (%) (a) Oracle 1.0 1.0 1.0 11.89 13.8
(b) $k$-means 0.64 0.77 0.70 14.27 19.3 (c) Strgar & Harwath (2022) 0.75 0.75
0.75 12.26 23.2 (d) Reborn (no post-processing) 0.57 0.78 0.65 16.29 12.9 (e)
\+ post-processing 0.73 0.71 0.72 11.63 - (f) \+ left-shift 1 frame 0.76 0.74
0.75 11.63 -

First, we focus on the PER in Table 4 when using different segmentation
methods. A segmentation method segments the speech feature into segments, and
the segment-wise mean-pooled features will be used as input to the phoneme
prediction model. In Table 4, all the segmentation methods share the same
phoneme prediction model: the phoneme prediction model trained using the
$k$-means segmentation of wav2vec-U. We find that simply replacing the
$k$-means boundary with the oracle phoneme boundary reduces the PER by 5.5%,
showing the imperfect heuristic segmentation is the bottleneck of UASR. Next,
even when using the boundary predicted by the SoTA unsupervised phoneme
segmentation (Strgar & Harwath, 2022), the PER does not reduce. Conversely,
the boundary learned by Reborn achieves the lowest PER. This is because the
boundaries learned by Reborn are tailored for the phoneme prediction model.

Next, we discuss the phoneme boundary quality in Table 4. The phoneme boundary
quality is evaluated with boundary precision, recall, and F1 with a 20ms
tolerance window, following the harsh scheme from Strgar & Harwath (2022). We
also report the average number of segments per second (Freq. in Table 4).
Interestingly, the boundary F1s of the oracle boundary (row (a)) and Strgar &
Harwath (2022) (row (c)) are much higher than the boundary F1 of Reborn (row
(d)), but the PER of Reborn is much better. This is because Reborn’s
segmentation model predicts more segments than the number of phonemes in the
utterance (before post-processing), indicating that it learns some segmental
structures smaller than the phonemes. This can be seen from the lower
precision (0.57) and higher frequency (16.29) compared to oracle boundaries.
However, overly segmenting the speech feature is not problematic because even
if the consecutive speech features of the same phonemes are split into two
segments by the segmentation model, as long as they have the same phoneme
prediction, the duplicated phoneme prediction will be removed in the de-
duplication step, and thus not affecting the PER.

While Reborn boundaries initially have low precision, with post-processing,
boundary precision can be improved by 0.16. Additionally, after applying
boundary post-processing, the number of segmental structures is closer to the
oracle. In contrast, we find that applying post-processing does not improve
the boundary F1 for all other unsupervised segmentation methods listed in
Table 4.

We show an example utterance in Figure 2 with the oracle boundary and Reborn
boundary (before post-processing). One can observe that Reborn boundaries
produce more segments than the oracle boundaries. We also observe an
interesting boundary delay phenomenon: the phoneme boundaries predicted by
Reborn sometimes lag behind the oracle phoneme boundaries (dotted lines in
Figure 2). This observation motivates us to left-shift the post-processed
Reborn boundary by one frame, and we find that the resulting boundary F1 is
further improved and matches the boundary F1 of the unsupervised phoneme
segmentation SoTA method (Strgar & Harwath, 2022) (row (f) v.s. row (c)).

###  5.3 Ablation Study

#### 5.3.1 Reward Design and Behavior Cloning

Table 5:  We compare different reward functions and the effect of BC
initialization with first iteration results on LibriSpeech. Row (c) is Reborn.

Ablation Stage 1 PER (%) Stage 2 PER (%) Silence Ratio dev-clean test-clean
dev-clean test-clean (a). $R_{\text{ppl}}$ 14.0 14.7 12.2 12.9 0.13 (b).
$R_{\text{ppl}}+R_{\text{edit}}$ 13.3 13.8 11.7 12.1 0.07 (c).
$R_{\text{ppl}}+R_{\text{edit}}+R_{\text{len}}$ 12.7 12.9 11.7 11.9 0.05 (d).
Reborn w/o BC 14.8 14.9 14.1 14.2 0.05

We verify the effectiveness of our three proposed reward functions and
behavior cloning initialization in Table 5. While the perplexity difference
reward alone is sufficient to guide the segmentation model, adding edit
distance and length difference rewards improves performance further. As
previously mentioned, only using the perplexity difference reward will make
the segmentation model predict more segments that lead to phoneme prediction
sequences with more $<$silence$>$ tokens, which have a lower PPL but do not
correspond to better phoneme prediction results. To demonstrate this, we show
the silence ratio in the de-duplicated phoneme prediction sequence per
utterance. We find that the silence ratio when only using $R_{\text{ppl}}$
(row (a)) is 0.13, which is much higher than the silence ratio of ground truth
phoneme transcription sequences, which is 0.07. Adding $R_{\text{edit}}$ (row
(b)) and $R_{\text{len}}$ (row (c)) makes the silent ratio closer to the
ground truth silence ratio.

We also find that removing BC (row (d)) worsens PER compared with using BC
(row (c)). Additionally, BC greatly speeds up RL training convergence, as
shown in Figure 4 (detailed in Appendix B).

Figure 3: PER of each stage during Reborn’s two-stage iterative training on
the test-clean split of LibriSpeech. St.: stage; w2vu: wav2vec-U

#### 5.3.2 Iterative Training

In Figure 3, we show the PER on LibriSpeech test-clean split during the
iterative training process of Reborn. We observe that after training the
segmentation model in the first iteration, the PER drastically drops by 6.4%
compared with wav2vec-U, which is used as the initialization of the phoneme
prediction model in iteration one. This shows that the quality of the
segmental structure’s boundaries is important to the performance of UASR.
Afterward, the PER of Reborn decreases steadily across iterative training,
showing that training the segmentation model and the phoneme prediction model
is critical to improving the performance of Reborn. We find that the PER does
not drop significantly after the fifth iteration.

##  6 Conclusion

In unsupervised ASR, learning the mapping between the segmental structures in
speech and their corresponding transcription is challenging, especially
without paired data. To tackle this, we propose Reborn, an iterative training
method for unsupervised ASR. Reborn iteratively trains the segmentation model
and the phoneme prediction model to improve the ASR performance. The
segmentation model is trained using RL with carefully designed rewards to
guide the segmentation model to find the boundaries of the segmental
structures in speech signals. Extensive experiments on three datasets spanning
seven languages show that Reborn outperforms prior best-performing
unsupervised ASR models in all datasets except one language in MLS. We conduct
comprehensive ablation studies to show that each component in Reborn is
critical to the performance. We also explain the effectiveness of Reborn by
analyzing its segmentation patterns and find that Reborn tends to produce
segmental structures smaller than phonemes, which helps the generators to
better predict the phoneme transcription.

## Impact Statements

Unsupervised automatic speech recognition (UASR) - predicting the textual
transcriptions for the given speech signals using only unpaired speech and
text data - is highly attractive. The existence of thousands of low-resourced
languages spoken globally is the major reason. This paper presents Reborn, a
novel unsupervised ASR training algorithm. We foresee that Reborn has a great
potential to make low-resource languages more accessible to train ASR systems,
making speech technology more accessible and boundaryless.

However, our training method may reinforce the bias in the dataset. This paper
presents work whose goal is to advance the field of Machine Learning. There
are many potential societal consequences of our work, none of which we feel
must be specifically highlighted here.

## References

  * Artetxe et al. (2018) Artetxe, M., Labaka, G., Agirre, E., and Cho, K.  Unsupervised neural machine translation.  In _International Conference on Learning Representations_ , 2018.
  * Baevski et al. (2020) Baevski, A., Zhou, Y., Mohamed, A., and Auli, M.  wav2vec 2.0: A framework for self-supervised learning of speech representations.  In _Advances in Neural Information Processing Systems_ , 2020.
  * Baevski et al. (2021) Baevski, A., Hsu, W.-N., Conneau, A., and Auli, M.  Unsupervised speech recognition.  _Advances in Neural Information Processing Systems_ , 2021.
  * Can et al. (2018) Can, D., Martinez, V. R., Papadopoulos, P., and Narayanan, S. S.  Pykaldi: A python wrapper for kaldi.  In _IEEE International Conference on Acoustics, Speech and Signal Processing_ , 2018.
  * Chen et al. (2021) Chen, G., Chai, S., Wang, G., Du, J., Zhang, W., Weng, C., Su, D., Povey, D., Trmal, J., Zhang, J., Jin, M., Khudanpur, S., Watanabe, S., Zhao, S., Zou, W., Li, X., Yao, X., Wang, Y., You, Z., and Yan, Z.  Gigaspeech: An evolving, multi-domain ASR corpus with 10, 000 hours of transcribed audio.  In _Interspeech_ , 2021.
  * Chen et al. (2019) Chen, K.-Y., Tsai, C.-P., Liu, D.-R., Lee, H.-Y., and Lee, L.-S.  Completely Unsupervised Phoneme Recognition by a Generative Adversarial Network Harmonized with Iteratively Refined Hidden Markov Models.  In _Interspeech_ , 2019.
  * Chen et al. (2022) Chen, S., Wang, C., Chen, Z., Wu, Y., Liu, S., Chen, Z., Li, J., Kanda, N., Yoshioka, T., Xiao, X., et al.  Wavlm: Large-scale self-supervised pre-training for full stack speech processing.  _IEEE Journal of Selected Topics in Signal Processing_ , 2022.
  * Conneau et al. (2021) Conneau, A., Baevski, A., Collobert, R., Mohamed, A., and Auli, M.  Unsupervised Cross-Lingual Representation Learning for Speech Recognition.  In _Interspeech_ , 2021.
  * Gao et al. (2023) Gao, D., Shi, J., Chuang, S.-P., Garcia, L. P., Lee, H.-y., Watanabe, S., and Khudanpur, S.  Euro: Espnet unsupervised asr open-source toolkit.  In _IEEE International Conference on Acoustics, Speech and Signal Processing_ , 2023.
  * Garofolo (1993) Garofolo, J. S.  Timit acoustic phonetic continuous speech corpus.  _Linguistic Data Consortium, 1993_ , 1993.
  * Goodfellow et al. (2014) Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., and Bengio, Y.  Generative adversarial nets.  _Advances in Neural Information Processing Systems_ , 2014.
  * Gulrajani et al. (2017) Gulrajani, I., Ahmed, F., Arjovsky, M., Dumoulin, V., and Courville, A. C.  Improved training of wasserstein gans.  _Advances in Neural Information Processing Systems_ , 2017.
  * Heafield (2011) Heafield, K.  Kenlm: Faster and smaller language model queries.  In _Proceedings of the Sixth Workshop on Statistical Machine Translation_ , 2011.
  * Hsu et al. (2021) Hsu, W.-N., Bolte, B., Tsai, Y.-H. H., Lakhotia, K., Salakhutdinov, R., and Mohamed, A.  Hubert: Self-supervised speech representation learning by masked prediction of hidden units.  _IEEE/ACM Transactions on Audio, Speech, and Language Processing_ , 2021.
  * Lample et al. (2018a) Lample, G., Conneau, A., Denoyer, L., and Ranzato, M.  Unsupervised machine translation using monolingual corpora only.  In _International Conference on Learning Representations_ , 2018a.
  * Lample et al. (2018b) Lample, G., Conneau, A., Ranzato, M., Denoyer, L., and Jégou, H.  Word translation without parallel data.  In _International Conference on Learning Representations_ , 2018b.
  * Liu et al. (2022) Liu, A. H., Hsu, W.-N., Auli, M., and Baevski, A.  Towards end-to-end unsupervised speech recognition.  In _IEEE Spoken Language Technology Workshop_ , 2022.
  * Liu et al. (2018) Liu, D.-R., Chen, K.-Y., Lee, H.-Y., and Lee, L.-S.  Completely Unsupervised Phoneme Recognition by Adversarially Learning Mapping Relationships from Audio Embeddings.  In _Interspeech_ , 2018.
  * Liu et al. (2017) Liu, Y., Chen, J., and Deng, L.  Unsupervised sequence classification using sequential output statistics.  _Advances in Neural Information Processing Systems_ , 2017.
  * McAuliffe et al. (2017) McAuliffe, M., Socolof, M., Mihuc, S., Wagner, M., and Sonderegger, M.  Montreal forced aligner: Trainable text-speech alignment using kaldi.  _Interspeech_ , 2017.
  * Mohri (1997) Mohri, M.  Finite-state transducers in language and speech processing.  _Computational Linguistics_ , 1997.
  * Mohri et al. (2002) Mohri, M., Pereira, F., and Riley, M.  Weighted finite-state transducers in speech recognition.  _Computer Speech & Language_, 2002.
  * Panayotov et al. (2015) Panayotov, V., Chen, G., Povey, D., and Khudanpur, S.  Librispeech: an asr corpus based on public domain audio books.  In _IEEE International Conference on Acoustics, Speech and Signal Processing_ , 2015.
  * Pratap et al. (2020) Pratap, V., Xu, Q., Sriram, A., Synnaeve, G., and Collobert, R.  MLS: A Large-Scale Multilingual Dataset for Speech Research.  In _Interspeech_ , 2020.
  * Pratap et al. (2023) Pratap, V., Tjandra, A., Shi, B., Tomasello, P., Babu, A., Kundu, S., Elkahky, A., Ni, Z., Vyas, A., Fazel-Zarandi, M., et al.  Scaling speech technology to 1,000+ languages.  _arXiv preprint arXiv:2305.13516_ , 2023.
  * Radford et al. (2023) Radford, A., Kim, J. W., Xu, T., Brockman, G., McLeavey, C., and Sutskever, I.  Robust speech recognition via large-scale weak supervision.  In _International Conference on Machine Learning_ , 2023.
  * Strgar & Harwath (2022) Strgar, L. and Harwath, D.  Phoneme segmentation using self-supervised speech models.  In _IEEE Spoken Language Technology Workshop_ , 2022.
  * Sutton et al. (1999) Sutton, R. S., McAllester, D., Singh, S., and Mansour, Y.  Policy gradient methods for reinforcement learning with function approximation.  _Advances in Neural Information Processing Systems_ , 12, 1999.
  * Wang et al. (2018) Wang, Y.-H., Lee, H.-y., and Lee, L.-s.  Segmental audio word2vec: Representing utterances as sequences of vectors with applications in spoken term detection.  In _2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , 2018.
  * Yeh et al. (2018) Yeh, C.-K., Chen, J., Yu, C., and Yu, D.  Unsupervised speech recognition via segmental empirical output distribution matching.  In _International Conference on Learning Representations_ , 2018.

##  Appendix A wav2vec-U

wav2vec-U (Baevski et al., 2021) is a popular UASR framework whose variants
(Gao et al., 2023; Liu et al., 2022) have achieved SoTA performance on
multiple UASR benchmarks. Following prior works (Liu et al., 2018), wav2vec-U
also uses adversarial training to train the phoneme prediction model. The
wav2vec-U framework takes the raw waveform as the input and predicts phoneme
transcription via three stages: (1) the feature extraction stage, (2) the
feature preprocessing stage, and (3) the phoneme prediction stage.

Stage (1) extracts speech features using a self-supervised speech foundation
model, wav2vec 2.0 (Baevski et al., 2020). For a spoken utterance, we denote
the extracted feature as $X=[x_{1},x_{2},...,x_{T}]\in\mathbb{R}^{T\times d}$,
where $T$ is the time dimension and $d$ is the feature dimension. Stage (2),
the feature preprocessing stage, uses $k$-means clustering and additional
heuristics, namely, adjacent pooling, to define the boundary of the segmental
structures in the speech features. The features in the same segmental
structure are mean-pooled into one feature embedding to reduce sequence length
to match phoneme sequences better. The pooled features’ dimension is reduced
using PCA. We denote the output from the feature preprocessing stage as
$S=[s_{1},s_{2},...,s_{T^{\prime}}]\in\mathbb{R}^{T^{\prime}\times
d^{\prime}}$. In Stage (3), the phoneme prediction model takes the pooled
speech feature as input and predicts the phoneme sequence
$Y=[y_{1},y_{2},...,y_{T^{\prime}}]$. Last, a de-duplication step removes the
consecutive duplicate phoneme predictions from $Y$, resulting in a shortened
phoneme sequence
$Y^{\prime}=[y^{\prime}_{1},y^{\prime}_{2},...,y^{\prime}_{M}]$ without
consecutive duplicate phonemes. $Y^{\prime}$ will be used as the input to fool
the discriminator in adversarial training, which loss is defined as follows:

$\displaystyle\mathcal{L}_{GAN}=$
$\displaystyle\min_{G}\max_{D}\underset{Z\sim\mathcal{Z}}{\mathbb{E}}[\log(\mathcal{D}(Z))]+\underset{X\sim\mathcal{X}}{\mathbb{E}}[\log(1-\mathcal{D}(\mathcal{G}(X)))]$
(5)

where $Z$ is a phoneme sequence sampled from the text corpora $\mathcal{Z}$
and $X$ is the speech signal from the speech corpora $\mathcal{X}$. Besides
the min-max objective in the adversarial training, wav2vec-U (Baevski et al.,
2021) also adopts the three regularized objectives for better convergence and
performance. First, the gradient penalty $\mathcal{L}_{gp}$ (Gulrajani et al.,
2017) is applied on the discriminator $\mathcal{D}$ to stabilize training,
where the input of $\mathcal{D}$ is the linear combination of a generated
sample $Y$ and a real sample $Z$.

$\mathcal{L}_{gp}=\underset{Y,Z,\alpha\sim
U(0,1)}{\mathbb{E}}\left[\left(\left\|\nabla\mathcal{D}(\alpha
Y+(1-\alpha)Z)\right\|_{2}-1\right)^{2}\right]$ (6)

Next, the smoothness penalty $\mathcal{L}_{sp}$ is computed across consecutive
segments of the generated sequence, promoting the generation of closely
aligned neighboring outputs by the model.

$\mathcal{L}_{sp}=\sum_{(y_{t},y_{t+1})\in Y}\left\|y_{t+1}-y_{t}\right\|^{2}$
(7)

Furthermore, the phoneme diversity loss $\mathcal{L}_{pd}$ is applied to
encourage higher vocabulary usage of the generated sequences. As shown in
Equation 8, the entropy of the averaged softmax distribution of the generator
outputs ($H_{\mathcal{G}}(\mathcal{G}(S))$) is maximized over the phoneme
vocabulary across a batch B.

$\mathcal{L}_{pd}=\frac{1}{|B|}\sum_{S\in B}-H_{\mathcal{G}}(\mathcal{G}(S))$
(8)

Finally, we describe the overall training objectives $\mathcal{L}$ by
weighting with coefficients $\lambda$, $\gamma$, and $\eta$, respectively.

$\mathcal{L}=\mathcal{L}_{GAN}-\lambda\mathcal{L}_{gp}+\gamma\mathcal{L}_{sp}+\eta\mathcal{L}_{pd}$
(9)

In this work, we utilize the same training objective from wav2vec-U to perform
GAN training (§ 3.2)

##  Appendix B Impact of Behavior Cloning on Model Convergence

Figure 4: Training Dynamics of BC Pre-Trained vs. Scratch Models. This figure
depicts the evolution of Phoneme Error Rate (PER) over training epochs, tested
on the test-clean set, to showcase the enhanced convergence speed and
efficiency achieved with BC pre-training.

In this ablation study, we rigorously examine the influence of Behavior
Cloning (BC) pre-training on the convergence dynamics and overall efficacy of
our segmenter model. Our empirical findings reveal a marked improvement in
sample efficiency for models subjected to BC pre-training, which is a critical
aspect in enhancing model performance. Notably, these models demonstrate a
notably faster convergence rate as compared to their counterparts trained from
scratch.

As depicted in Figure 4, the training trajectories of the models, with and
without BC pre-training, are contrasted. This visualization distinctly
underscores the difference in PER across various training epochs. The curves
elucidate not only the expedited convergence but also the superior performance
attainable with BC pre-trained models.

It is observed that segmenters solely pre-trained through BC do not exhibit a
substantial improvement in PER initially. However, this pre-training imbues
the model with a more refined capability for boundary prediction. In
subsequent RL training phases, this enhanced boundary prediction proficiency
translates into a swifter convergence trajectory, culminating in superior
final performance metrics. This phenomenon highlights the nuanced yet
significant role of BC pre-training in the context of unsupervised automatic
speech recognition, underscoring its potential as a pivotal step in the model
optimization process.

##  Appendix C Implementation Details

###  C.1 Segmentaiton model

We parametrize the segmentation model $\pi_{\theta}$ using a two-layered
1-dimensional convolutional neural network (CNN). This choice of architecture
allows us to efficiently extract relevant information from adjacent speech
features, making it particularly well-suited for our segmentation task. The
three convolutional layers in our CNN have different kernel sizes, namely 7
and 3. This is a lightweight model, making it computationally efficient for
applications.

###  C.2 Hyperparameter Searches

We search the optimal hyperparameters of our phoneme predictor and
segmentation models with unsupervised validation metrics. For the phoneme
predictor, we directly adopt the search space of the hyperparameters and the
unsupervised validation metric from wav2vec-U (Baevski et al., 2021).We search
the hyperparameters indicated as $\lambda$, $\gamma$, and $\eta$ in Appendix A
only during the initialization stage of the phoneme predictor (§ 3.3). We
directly adopt the same hyperparameters n the following iterations of the
phoneme predictor GAN training. As for the segmentation model, we take the
idea from Baevski et al. (2021) and derive a similar unsupervised metric.

#### C.2.1 Unsupervised Validation Metric for Segmentation Model Selection

Each segmentation model, denoted as $\pi_{\theta}$, undergoes unsupervised
performance evaluation by predicting the transcriptions of the validation set.
The phoneme predictions generated by the model are processed to form a de-
duplicated set, denoted as
$Y^{\prime}_{\theta}=[y^{\prime}_{1,\theta},\cdots,y^{\prime}_{M,\theta}]$,
where consecutive duplicate phonemes are removed.

The unsupervised metric places significant emphasis on two key aspects:
Language Model (LM) negative log-likelihood (NLL) and vocabulary usage.
Vocabulary usage is calculated as
$U(Y^{\prime}_{\theta})=\frac{1}{|V|}\sum_{v\in V}\mathbb{I}(v\in
Y^{\prime}_{\theta})$, where $V$ denotes the entire vocabulary set. The
validation metric seeks to minimize the NLL while simultaneously maximizing
vocabulary usage. The NLL metric reflects the model’s proficiency in
generating sensible sentences consistent with the dataset distribution, while
vocabulary usage quantifies the diversity of phonemes employed by the model.
This combined approach ensures that our unsupervised ASR system’s predictions
align with the expected LM distribution and exhibit a wide linguistic range,
thus mitigating the risk of favoring repetitive, high-probability sentences.

Additionally, the lengths of transcriptions are taken into account. We employ
NLL without length normalization, thereby favoring predictions that score high
under the language model but without an undue preference for overly lengthy
transcriptions.

The optimization objective is formally expressed as:

$\theta^{*}=\arg\min_{\theta}\left(\frac{{-\sum_{i=1}^{M}\text{ log
}p_{LM}(y^{\prime}_{i,\theta})}}{{U(Y^{\prime}_{\theta})}}\right)$ (10)

captures these considerations and serves as a comprehensive measure for the
performance and generalization capabilities of our unsupervised ASR model.

#### C.2.2 Training Details

For the training of our segmentation model, we initially employ behavioral
cloning, followed by updates using policy gradient methods. In the BC phase,
we configure our model with a batch size of 128 and a learning rate of 0.0005,
training for 20 epochs. We also assign cross-entropy weights of [1, 5] for
boundary classification, to address the imbalance where the class labeled $0$
significantly outnumbers the class labeled $1$.

During the policy gradient update phase, the model is trained with the same
batch size of 128, but with a reduced learning rate of 0.0001 and a weight
decay of 0.0001. We utilize the AdamW optimizer in conjunction with the
CosineAnnealingLR scheduler. For the LS and MLS datasets, the training
proceeds for 40 epochs, while for the TIMIT dataset, owing to its smaller
size, we extend the training to 500 epochs to ensure convergence.

To refine the performance of our reinforcement learning framework, we have
tuned the reward coefficients, leveraging the validation metric (eq. 10), to
secure optimal outcomes across varied datasets. Specifically, the coefficient
for the PPL difference reward, $c_{\text{ppl}}$, is fixed at 1.0 to prioritize
the minimization of perplexity in generated sequences. The coefficient for the
edit-distance reward, $c_{\text{edit}}$, is set at 0.2. And the coefficient
for the length difference reward, $c_{\text{len}}$, is selected from a
predefined set of values within the range [0.0, 0.2, 0.4, 0.6, 0.8]. These
values are carefully chosen to balance the importance of length consistency
with other factors, with the optimal reward configuration for each dataset
detailed in Table 6.

It is noteworthy that, as the iterations progress, the length constraints
become less critical. This observation suggests that the model gradually
learns to generate sequences of appropriate length, reducing the need for
explicit length-based adjustments.

Table 6: Best reward configurations obtained through hyperparameter searches
on each dataset.

Dataset LibriSpeech TIMIT MLS iter. 1 iter. 2-5 iter. 1-3 de nl fr es it pt
$c_{\text{ppl}}$ 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 $c_{\text{edit}}$ 0.2 0.2
0.2 0.2 0.2 0.2 0.2 0.2 0.2 $c_{\text{len}}$ 0.2 0.0 0.0 0.4 0.2 0.8 0.4 0.6
0.4

###  C.3 WFST Decoding for Obtaining Word-Level Outputs

In Reborn, phoneme-level outputs can be generated directly with greedy
decoding. However, obtaining word-level outputs requires additional modules
such as WFST or external language models (Baevski et al., 2021; Liu et al.,
2017; Gao et al., 2023). In this work, we incorporate Reborn with WFST to
generate word-level transcriptions. Our decoding procedure mainly follows
Baevski et al. (2021), which builds the WFST using PyKaldi (Can et al., 2018),
and additional self-loops are introduced to mimic the CTC behavior. Moreover,
two hyperparameters: the acoustic scale $a$, and the additive blank weight $v$
are added during the WFST decoding process. In wav2vec-U, the search interval
of $a$ is $[0,8]$ and $v$ is tuned within $[-3,8]$. As for Reborn, we find
that using a much smaller search space for both of the hyperparameters is
enough for obtaining reasonable outputs. More specifically, We tune the
acoustic scale in $[0.4,1.2]$ with step $0.1$ and the blank weight in $[-5,2]$
with step $1$.

It is worth mentioned that we utilize the original boundary generated by the
segmentation model instead of the post-processed one (§ 3.1.5) for WFST
decoding. We find that the increased amount of segments directly leads to
longer generator outputs, which is beneficial for WFST to generate higher
quality outputs. The phenomenon is also discovered by the prior works (Baevski
et al., 2021; Liu et al., 2022). In wav2vec-U (Baevski et al., 2021), the
segmental feature used for WFST decoding is the one before adjacent pooling;
while in wav2vec-U 2.0 (Liu et al., 2022), they simply lowers the decoding
stride to generate more lengthy outputs for WFST decoding.