Table 3.2: System combination experiments using DOVER-Lap on the AMI evaluation set, reported in terms of MS, FA, SE, and DER. We combined 3 overlap-aware hypotheses: overlap-aware SC, VB-based overlap assignment, and regional proposal networks. Method | MS | FA | SE | DER
---|---|---|---|---
Overlap-aware SC [105] | 11.48 | 2.27 | 9.81 | 23.56
VB-based overlap assignment [106] | 9.84 | 2.06 | 9.60 | 21.50
Region Proposal Networks [90] | 9.49 | 7.68 | 8.25 | 25.42
DOVER-Lap |  |  |  |
Exponential mapping | 9.96 | 2.16 | 7.75 | 19.86
Hungarian mapping | 9.98 | 2.13 | 8.35 | 20.46
Randomized local search | 9.97 | 2.15 | 7.92 | 20.05

For randomized local search (RLS), we set $N$ and $M$ in Algorithm 5 (Appendix
B.4) as 1000 and $2K+1$, respectively, where $K$ is the number of systems to
be combined. As expected, RLS outperformed the Hungarian mapping algorithm,
and the gains come primarily from lower speaker error (7.92% compared with
8.35%). However, this difference in performance is fairly small, especially
when we consider that the RLS method requires a much longer processing time.
This may be because the theoretical bounds are designed to hold in the setting
when the size of inputs is fairly large. In our setting of combining
diarization hypothesis, these “large number” assumptions are violated.
Furthermore, while improving the objective in (3.4) typically leads to an
improvement in DER, the relationship is not strictly monotonic as seen in Fig.
3.3.

Still, these results have important implications. Our experiments with the RLS
method indicates that even with a theoretically stronger algorithm, it may not
be possible to do much better than the fast Hungarian-based mapping algorithm
due to the constraints of our setting. As such, it is unlikely that any
further advances in combination performances under this framework would be
obtained from better label mapping methods.

####  3.5.3 Diarization results on LibriCSS

In Table 3.3, we show the DER results on LibriCSS for four baseline
diarization systems and DOVER-Lap with different label mapping methods, with a
break down by overlap condition. We also report a further break down by missed
speech, false alarm, and speaker confusion, in Table 3.4.

Table 3.3: Diarization performance on LibriCSS evaluation set (sessions 2-10),
evaluated condition-wise, in terms of % DER. 0S and 0L refer to 0% overlap
with short and long inter-utterance silences, respectively. The DL results are
using rank-based weighting.

Method Overlap ratio in % Average 0L 0S 10 20 30 40 VB 3.85 3.84 6.46 8.20
12.60 13.39 8.59 SC 2.57 3.41 6.80 10.03 13.86 15.17 9.34 RPN 4.45 9.11 8.33
6.68 11.59 14.21 9.50 TS-VAD 5.99 4.63 6.62 7.28 10.31 9.54 7.62 DOVER-Lap
Exponential 1.77 1.74 3.47 4.09 6.66 6.83 4.38 Hungarian 1.98 1.66 3.38 4.39
6.72 6.29 4.33 RLS 1.77 1.74 3.39 4.09 6.66 6.83 4.36

Similar to the results on AMI, we found that DOVER-Lap improved the average
DER over the single best system (TS-VAD, in this case) significantly.
Specifically, we obtained a 43.2% relative DER improvement, from 7.62% to
4.33%. This improvement was consistent across the different overlap
conditions, even though the single best system themselves may differ depending
on the condition. For instance, the clustering-based methods were better on
low overlaps, while the supervised methods performed better on high overlap
conditions. DOVER-Lap was able to get the best of both techniques and obtain
the best DERs across the board. If we look at the performances on the
overlapping regions (shown in the smaller font), we see that the Hungarian
mapping outperforms other methods significantly, which results in better
overall DER performance.

Table 3.4: Diarization result break-down on LibriCSS evaluation set, in terms of % missed speech (MS), false alarm (FA), and speaker confusion (Conf.). The numbers in smaller font are the corresponding error rates computed on only the overlapping regions. Method | MS | FA | Conf. | DER
---|---|---|---|---
VB | 1.68 8.77 | 0.48 0.00 | 6.42 19.86 | 8.59 28.63
SC | 2.52 13.35 | 1.10 0.00 | 5.72 17.57 | 9.34 30.92
RPN | 2.87 7.61 | 3.33 3.21 | 3.30 4.16 | 9.50 14.97
TS-VAD | 3.24 11.72 | 1.52 0.78 | 2.86 4.88 | 7.62 17.37
DOVER-Lap |  |  |  |
Exponential | 1.73 8.52 | 0.77 0.02 | 1.88 4.81 | 4.38 13.35
Hungarian |  1.64 8.14 | 0.77 0.12 | 1.92 4.36 |  4.33 12.62
RLS | 1.73 8.52 | 0.77 0.02 |  1.86 4.81 | 4.36 13.36

###  3.6 Late fusion for multi-microphone diarization

The DOVER-Lap algorithm combines diarization hypotheses, irrespective of the
source of these hypotheses. In the experiments above, we have applied it for
combining different diarization systems; however, it can also be applied to
several other use cases. For instance, we may have a single-channel
diarization system, but input signals from an array microphone. In such cases,
the system can be independently run on each channel, and the outputs can be
combined using DOVER-Lap — this is a classic “late fusion” application (in
contrast to early fusion techniques such as beamforming [119, 120]).

In this section, we demonstrate the application of DOVER-Lap for late fusion
on array microphones. We conducted our investigation on the LibriCSS dataset,
which has 7 microphones arranged in a circular array. For our diarization
system, we used the overlap-aware spectral clustering (SC) [105] method
described earlier. As shown in Table 3.4, the method obtained a DER of 9.34%
on LibriCSS using a single microphone.

Table 3.5 shows the results for multichannel diarization using late fusion
with DOVER-Lap (using the exponential mapping method). The single-channel
system obtained a DER of 9.40% on average (with a standard deviation of
0.23%). For early fusion, we applied online weighted prediction error (WPE)
[121] based dereverberation followed by delay-and-sum beamforming on the input
channels. We used the Nara implementation [122] of WPE and the Beamformit tool
[123] for beamforming. The corresponding DER was found to be 9.33%, which is
marginally better than the 7-channel average. Notably, simple beamforming
without dereverberation degraded the DER to 9.71%. Late fusion using DOVER-Lap
improved over the average and best single system by achieving 9.02% DER.
Similar to our earlier results, we found that the improvement was mostly from
reduced false alarms and speaker confusions.

Table 3.5: Diarization results for multichannel LibriCSS evaluation set. Late
fusion using DL achieved better performance compared to early fusion based on
dereverberation and beamforming.

Method MS FA Conf. DER 7-channel avg. 2.58 0.96 5.86 9.40 7-channel best 2.59
0.99 5.53 9.11 WPE + Beamforming 2.91 0.96 5.86 9.33 DL 3.60 0.66 4.76 9.02

###  3.7 Conclusion

We proposed DOVER-Lap, a new method to combine the outputs from overlap-aware
diarization systems. Our method was inspired by the label mapping and label
voting approach in DOVER, but we modified the algorithms used in each of these
stages. We demonstrated through experiments on AMI and LibriCSS that DOVER-Lap
is effective at combining the outputs from different kinds of diarization
systems, such as clustering-based, RPN, and TS-VAD. It provided consistent and
significant improvements over the single best system for both datasets. We
also showed its applicability to multi-channel diarization through late
fusion, where it outperformed early fusion methods.

Since we proposed DOVER-Lap in [124] and [37], it has become the de-facto
approach for combining diarization systems, and has been used by the top teams
in several community challenges pertaining to speaker diarization and multi-
talker ASR. We have summarized these in Table 3.6, where we see that teams
have used DOVER-Lap for combining a variety of systems, such as clustering-
based, EEND, TS-VAD, separation-guided diarization, and multi-channel systems.
Across all such usage, DOVER-Lap has been found to give consistent and
significant DER improvements.

Table 3.6: Summary of DOVER-Lap usage in major speaker diarization and multi-talker ASR community challenges. Challenge/Team | Systems combined | Position
---|---|---
DIHARD-3
USTC-NELSLIP [125] | Clustering, ITS-VAD | 1
Hitachi-JHU [126] | VBx, EEND | 2
BUT [127] | VBx, SC, EEND | 5
VoxSRC 2021
DKU-Duke-Lenovo [128] | AHC, SC, TS-VAD | 1
ByteDance [129] | Multi-scale clustering | 2
Tencent [130] | VBx (multiple extractors) | 3
Huawei [131] | SC, VBx | -
VoxSRC 2022
DKU-SMIIP [132] | AHC, SC, TS-VAD | 1
GIST-AiTeR [133] | Multi-scale clustering | 3
BUCEA [134] | VBx, SC | -
M2MeT at ICASSP 2022
DUKE-DKU [135] | Multi-channel TS-VAD | 1
CUHK-Tencent [136] | VBx, TS-VAD, FFM-TSVAD | 2
USTC-Ximalaya [137] | Multi-channel TS-VAD | 3
RoyalFlush [138] | Separation-guided diarization | 5
CHiME-7 DASR
USTC-NERCSLIP [139] | Multi-channel NSD-MA-MSE | 1
NTT [140] | Multi-channel EEND-VC | 3

In the context of the broader multi-talker ASR problem, DOVER-Lap may be used
to combine channel-wise diarization outputs from multi-channel arrays, as was
done in [139] and [140]. This is relatively efficient for end-to-end
diarization systems which can process batched inputs, and may be particularly
useful when the channels are ad-hoc microphone devices, such as smartphones
placed near the speakers.

##  Chapter 4 Target Speaker Extraction with Guided Source Separation

So far, we have focused exclusively on segmentation of long recordings through
overlap-aware diarization approaches. While this step is important for
identification of speaker-homogeneous segments, we are still required to
extract the speaker’s utterance from possibly mixed audio, such that a regular
speech recognition system may be able to transcribe it. In this chapter, we
will describe this “target speaker extraction” (TSE) task, showing how the
target speaker information can be obtained from various means. For multi-
channel recordings where we have already obtained time-segmented speaker
boundaries (either through an oracle or a diarization system), guided source
separation (GSS) [28] allows for unsupervised modeling of TSE. Towards our
objective for building efficient and accurate multi-talker ASR systems, we
will describe our GPU-accelerated implementation of GSS, which is inspired by
modern deep learning pipelines. This efficient implementation will enable us
to analyze the factors affecting GSS performance in some detail. We will show
how a strong TSE module can result in far-field ASR performance similar to
that obtained using close-talk microphones. In subsequent chapters, we will
combine this extraction step with an overlap-aware diarization system
(described earlier) to build a complete meeting transcription pipeline.

###  4.1 Target speaker extraction

Target speaker extraction (TSE) is the problem of estimating the speech signal
of a target speaker in a mixture of several speakers, given auxiliary cues to
identify the target. In literature, TSE has alternatively been referred to as
informed source separation, personalized speech enhancement, or audio-visual
speech separation, depending on the context and the modalities involved. TSE
is motivated from the human ability of auditory attention in cocktail party
settings, which has been the subject of much research in the last several
decades [141]. As illustrated in Fig. 4.1, neural models for TSE often exploit
cues such as pre-recorded enrolment recordings, spatial information that
provides the direction of the target, or visual tracking of faces, in order to
focus on the target speaker in a multi-source mixture [142, 143, 144, 145,
146]. For more details about neural TSE, we refer the reader to the excellent
review by [1].

Figure 4.1: The target speaker extraction problem, and illustration of
auxiliary cues. Adapted from [1].

Despite the large body of work on supervised neural TSE, current techniques
have several limitations. First, the usage of auxiliary cues imposes
additional data collection requirements. For example, the target speaker may
need to be registered with the device to enable enrollment recordings, or we
may need to measure speaker locations to use spatial cues. Often, such systems
are trained on large amounts of synthetically mixed speech, or using multi-
channel inputs assuming a specific array configuration. This kind of training
limits the model’s performance when used in real-world settings, or makes it
infeasible when the array configuration changes.

For these reasons, beamforming of multi-channel signals using unsupervised
mask estimation remains a strong baseline for target-speaker ASR [147, 148,
149, 150]. Among these, the recently proposed guided source separation (GSS)
technique stands out as a particularly effective approach for handling noisy,
overlapping speech using diarization information [28, 151]. The method was
first proposed for the CHiME-5 challenge, where it provided relative word
error rate (WER) improvement of 21.1% on the multi-array track using oracle
diarization [28]. It was later adopted as the challenge baseline for CHiME-6,
and used by the winning systems on both oracle and unsegmented tracks [6, 39,
152, 153].

GSS relies on fundamental ideas from blind source separation (BSS), using
spatial mixture models to model the sum of short-time Fourier transform (STFT)
bins of multiple speakers [154]. It uses diarization information in two ways.
First, for the unsupervised mask estimation using BSS, speaker activities are
used to (i) estimate the number of mixture components, and (ii) avoid the
speaker-frequency permutation problem when processing different frequency bins
independently. Once the speaker and noise masks are estimated, the speaker
activities are again used to select the “target” speaker — in particular, the
speaker who is active throughout the segment is the target speaker. We will
describe the algorithm in detail in Section 4.2 for the sake of completeness.

Despite its strong performance in the CHiME-5 and CHiME-6 challenges, GSS has
seen limited adoption in other multi-talker benchmarks, most notably offline
meeting transcription, primarily due to its significant computational cost.
For instance, enhancing the CHiME-6 dev set using 80 CPU jobs requires
approximately 20 hours with the original GSS
implementation111https://github.com/fgnt/pb_chime5. There have been some
efforts to adapt the offline GSS algorithm for real-time enhancement by
relying on limited right context [155], but these are also CPU-bound. These
high inference times have also prevented detailed analysis of the method,
particularly in terms of evaluation and impact of the various parameters.

In this chapter, we will describe our new, publicly-available GPU-accelerated
implementation of GSS that aims to remove this computational bottleneck. We
achieve this primarily by porting all the computations on to a GPU, and
applying batching at several levels to maximize GPU memory utilization. Our
implementation is inspired by modern deep learning pipelines where background
CPU-based workers perform data loading of large tensors, while the data
processing is performed by GPUs [156]. We describe our accelerated
implementation in detail in Section 4.3. The resulting 300x speedup allows us
to perform ablation experiments using several benchmarks to analyze the
importance of several factors that impact GSS performance such as WPE, noise
class, context duration, number of BSS iterations, and number of channels,
towards GSS performance. While previous work has only evaluated GSS in terms
of downstream ASR performance, we also measure signal-level enhancement
metrics and the improvement in speaker information in the enhanced signal.

###  4.2 Guided source separation

We first provide an overview of the GSS algorithm, as proposed in [28].
Consider a multi-channel input recording provided in the form of STFT features
$\mathbf{Y}_{t,f}\in\mathbb{C}^{M}$, where $t$ and $f$ are time and frequency
bins, respectively, and $M$ is the number of channels. The GSS algorithm
assumes the following model of the signal:

$\mathbf{Y}_{t,f}=\underbrace{\sum_{k\in
K}\mathbf{X}_{t,f,k}^{\mathrm{early}}}_{\mathbf{X}_{t,f}^{\mathrm{early}}}+\underbrace{\sum_{k\in
K}\mathbf{X}_{t,f,k}^{\mathrm{late}}}_{\mathbf{X}_{t,f}^{\mathrm{late}}}+\mathbf{N}_{t,f},$
(4.1)

where $K$ is the number of speakers in the recording, and “early” and “late”
refer to components of the reverberation. For target-speaker extraction, the
objective is to estimate the de-reverberated signal from a desired speaker
$k$, i.e., $\widehat{\mathbf{X}}_{t,f,k}$. This estimation is performed in
three steps, as described in Fig. 4.2.

Figure 4.2: An overview of the guided source separation (GSS) method for
target-speaker extraction.

####  4.2.1 De-reverberation using WPE

First, we estimate $\mathbf{X}_{t,f}^{\mathrm{late}}$, i.e., the “late” part
of the reverb, using the popular weighted prediction error (WPE) algorithm
[157, 121], which uses multi-channel linear prediction (MCLP).

WPE models the late reverberation component of the signal at the first
microphone ($m=1$) using an auto-regressive model on the received signal
across all $M$ microphones as

$\mathbf{X}_{t,f}^{\mathrm{late}}[1]=\sum_{m=1}^{M}(\mathbf{g}_{f}^{m})^{H}\mathbf{Y}_{t-D,f}^{m}.$
(4.2)

Here, $\mathbf{g}_{f}$ is a regression vector for frequency bin $f$,
$(\cdot)^{H}$ denotes the Hermitian transpose, and $D$ is the signal delay
beyond which speech is not expected to have any residual auto-correlation.
Each STFT coefficient of the desired signal is modeled as a zero-mean Gaussian
random variable with variance $\lambda_{t,f}$. Then, we can solve for
$\mathbf{g}_{f}^{M}$ and $\lambda_{t,f}$ using maximum likelihood estimation
by minimizing the following cost function:

$\mathcal{L}(\Theta_{f})=\sum_{t=1}^{T}\left(\log\lambda_{t,f}+\frac{\lVert\mathbf{Y}_{t,f}^{1}-(\mathbf{g}_{f})^{H}\mathbf{Y}_{t-D,f}\rVert^{2}}{\lambda_{t,f}}\right)$
(4.3)

The function cannot be solved analytically, so WPE solves it in a two-step
iterative process, by alternatively keeping $\mathbf{g}_{f}$ and
$\lambda_{t,f}$ fixed and minimizing w.r.t the other.

Once we have obtained the regression vectors, we estimate the late
reverberation components, i.e., $\mathbf{X}_{t,f}^{\mathrm{late}}$ and remove
it from the signal, followed by normalization to get unit STFT vectors, i.e.,

$\tilde{\mathbf{Y}}_{t,f}=\frac{\mathbf{Y}_{t,f}-\widehat{\mathbf{X}}_{t,f}^{\mathrm{late}}}{\|\mathbf{Y}_{t,f}-\widehat{\mathbf{X}}_{t,f}^{\mathrm{late}}\|}.$
(4.4)

####  4.2.2 Mask estimation using CACGMMs

In the second stage, STFT masks are estimated for each speaker (and noise).
The mask estimation technique is based on the sparsity assumption, which
posits that at most one speaker is active in each time-frequency bin. Using
this assumption, the vector in each T-F bin can be assumed to have been
generated from a mixture model where each component of the mixture belongs to
a different speaker (or noise class). In the case of GSS, each mixture
component is a complex angular central Gaussian (CACG), and hence the mixture
model is a CACGMM [158]. A CACGMM models sums of unit-normalized complex-
valued random variables, and the probability density function at a frequency
index $f$ is given by

$p(\tilde{\mathbf{Y}}_{t,f})=\sum_{k\in
K}\pi_{f,k}\mathcal{A}(\tilde{\mathbf{Y}}_{t,f};\mathbf{B}_{f,k}),$ (4.5)

where $\pi_{f,k}$ is the mixture weight of source $k$ at frequency index $f$,
and $\mathcal{A}(\mathbf{y};\mathbf{B})$ is a CACG distribution parameterized
by a covariance matrix $\mathbf{B}\in\mathbb{C}^{M\times M}$ as

$\mathcal{A}(\mathbf{y};\mathbf{B})=\left(\frac{1}{2\pi}\right)^{M}\frac{(M-1)!}{\lvert\mathbf{B}\rvert}(\mathbf{y}^{H}\mathbf{B}^{-1}\mathbf{y})^{-M}.$
(4.6)

Mixture model parameters are usually estimated using the EM algorithm that
alternates between estimating the state posteriors (in the E-step) and the
parameters ($\mathbf{B}$) of the component model (in the M-step). However,
there are two problems in applying EM independently for each frequency bin:
(i) the number of sources $K$ in equation (4.5) is unknown; and (ii) the same
mixture component may correspond to different sources in different frequency
bins. GSS solves both of these problems by assuming that speaker activities
are known for the recording, either through an oracle or a diarization system.
Given the speaker activities $a_{t,k}\in\\{0,1\\}$, we convert the time-
invariant mixture weights to time-varying weights as

$\pi_{t,f,k}=\frac{\pi_{f,k}a_{t,k}}{\sum_{k^{\prime}\in
K}\pi_{f,k^{\prime}}a_{t,k^{\prime}}}.$ (4.7)

There may still be a permutation problem between the mixture components for
the target speaker and the noise signal, since noise is present throughout the
recording. To solve this problem, the GSS algorithm adds a “context window” to
each utterance in which $a_{t,k}$ is zero for only the target speaker $k$. We
run the EM algorithm on the CACGMM until convergence to obtain the final state
posteriors $\gamma_{t,f,k}$ as the estimated speaker masks.

####  4.2.3 Mask-based MVDR beamforming

Finally, we compute the spatial covariance matrices for the target signal and
background signal as

$\displaystyle\Phi_{k}(f)$
$\displaystyle=\frac{1}{T}\sum_{t}\gamma_{t,f,k}\tilde{\mathbf{Y}}_{t,f}\tilde{\mathbf{Y}}_{t,f}^{H},$
(4.8) $\displaystyle\Phi_{\mathrm{bg}}(f)$
$\displaystyle=\frac{1}{T}\sum_{t}\left(\sum_{k^{\prime}\neq
k}\gamma_{t,f,k^{\prime}}\right)\tilde{\mathbf{Y}}_{t,f}\tilde{\mathbf{Y}}_{t,f}^{H},$
(4.9)

which are then used to compute the minimum-variance distortionless response
(MVDR) filter [159, 160] as

$\mathbf{h}(f)=\frac{\Phi_{\mathrm{bg}}^{-1}(f)\Phi_{k}(f)\mathbf{e}_{\mathrm{ref}}}{\mathrm{tr}\left(\Phi_{\mathrm{bg}}^{-1}(f)\Phi_{k}(f)\right)},$
(4.10)

where $\mathbf{e}_{\mathrm{ref}}\in\\{0,1\\}^{M}$ is a one-hot vector
indicating the reference channel, selected to maximize the signal-to-noise
ratio. Finally, the enhanced STFT signal is computed as

$\widehat{\mathbf{X}}_{t,f,k}=\mathbf{h}(f)^{H}\tilde{\mathbf{Y}}_{t,f}.$
(4.11)

###  4.3 GPU-accelerated inference

The original GSS implementation is slowed down by four key factors:

  1. (i)

All the segments are processed sequentially, so processing time for a
recording increases linearly with number of identified segments.

  2. (ii)

A context window (usually 15s) is used for all segments regardless of the
segment duration, resulting in a lot of wasted computation for short segments.

  3. (iii)

For each segment, the CACGMM-based mask estimation is performed by iterating
over all frequency bins (usually 513) sequentially.

  4. (iv)

All computations (i.e., feature extraction, WPE, mask estimation, beamforming,
and iSTFT) are implemented on the CPU in NumPy [161].

We propose to accelerate GSS-based inference by leveraging the power of modern
GPU hardware and pipelines inspired by neural network training. First, to
address limitation (iv), we use CuPy arrays which speed up array operations
significantly using CUDA kernels, compared with regular NumPy-based array
operations [162]. Since the most computationally intensive operations in the
pipeline (such as CACG probability estimation) involve matrix multiplications
(through einsum), GPU-based CUDA kernels are more efficient. However, simply
transferring all arrays to CuPy is not sufficient — for example, limitations
(i)–(iii) still require sequential processing, which limits GPU utilization.
To maximize GPU utilization and improve real-time factor (RTF), we perform the
following additional optimizations (shown in Fig. 4.3).

Figure 4.3: Overview of batch processing for GPU-accelerated GSS. Solid and
dotted lines denote GPU-bound and CPU-bound operations, respectively. The WPE
module is not shown.

####  4.3.1 Segment batching

Instead of processing each segment independently, we batch together multiple
segments for inference. However, unlike neural network based training
pipelines where batching is performed by stacking sequences in parallel, our
batches are formed by concatenating segments sequentially along the time ($T$)
axis to create “super-segments.” We choose this form of batching because (i)
the einsum-based operations are designed to work with 3-D tensors, and (ii)
parallel batching of segments with padding would result in wasted memory.
Since multiple components of the inference (such as mask estimation and
beamforming) compute statistics over the entire segment, we always create
super-segments from segments of a recording that contain the same target
speaker. Furthermore, we only use a single context window for the entire batch
(instead of segment-wise context), which further amortizes the context window
computations over multiple speaker segments. This batching technique should
work well for the case when optimal reference channels do not vary over the
duration of the recording (i.e., when speakers are stationary, which is common
for meeting scenarios)222We also provide the option for using at most one
segment per batch, for the case when speakers are not stationary (§ 4.5.3.3)..

####  4.3.2 CPU-based data-loaders

We ensure that GPU idle time is minimized by off-loading the batch creation
process to CPU-based data-loaders (possibly containing multiple workers),
similar to deep learning pipelines. These workers collect same-speaker
segments in the background and load the audio from disk while the GPU is busy
processing the previous batch. Section 4.4 provides further details about our
Lhotse-based data pipeline[163].

####  4.3.3 Frequency batching

To address (iii), we modified the CACGMM-based mask estimation to process 3-D
tensors $(F,T,M)$ instead of 2-D arrays $(T,M)$. This simple change allows us
to process all the frequency bins concurrently in a batch, significantly
increasing GPU memory utilization. Such a batching makes sense because all
frequency bins are treated independently in the mask estimation process.

####  4.3.4 Einsum path optimization

As mentioned above, several components in the GSS pipeline are implemented
using einsum, which uses an optimal path contraction technique to find the
path of minimum floating-point operations through the sequence (often
resulting in up to 15x speed-up over a naive computation) [164]. However, the
optimal path finding itself is computationally demanding, with a complexity of
$\mathcal{O}(N!)$ for $N$ arrays, and since it is performed several times
during inference (for example, in each iteration of the CACGMM inference), it
overshadows any speed-ups from the actual contracted sum. To remedy this, we
cache the optimal computed path in the first iteration and re-use it in
subsequent iterations. In practice, since our tensor dimensions often have the
same relative order across all batches (i.e., $M$<$F$<$T$), we can simply fix
the optimal path for all einsum operations. This is because segment batching
avoids very short segments that would otherwise result in $T$<$F$.

Figure 4.4: Representative output of profiler during enhancement of a single
recording. Full stats are available at this https url.

Finally, once the enhanced waveform is obtained for the super-segment, we use
background worker threads to chunk it into the original segments and save them
to disk. With all these speed-ups, we were able to enhance a 10-minute
LibriCSS recording in 159s (as shown in Fig. 4.4), of which mask estimation,
WPE, and beamformer constituted 59.2%, 24.4%, and 8.5% processing time,
respectively. This is equivalent to a real-time factor (RTF) of approximately
0.3. We anticipate that further speed-ups could be obtained using GPUs with
larger memory, by using bigger batches.

###  4.4 Implementation details

From an implementation perspective, we can divide the pipeline into two parts.
The data processing part is tasked with efficiently creating segments and
corresponding speaker activities, while the inference part performs the actual
computations on GPU. We use Lhotse [163] for all data processing, i.e., to
store and read recording metadata, to represent speaker activities, and to
perform segment batching to create super-segments. For batching we create
buckets out of each speaker’s segments on-the-fly, and sample speakers in a
round-robin manner (using Lhotse’s DynamicBucketingSampler and
RoundRobinSampler), so that metadata from all segments do not need to be
stored in memory. The super-segment obtained from the data-loader is converted
to a CuPy array in-place, and all subsequent inference is performed on the
GPU. Since we use Lhotse’s supervision manifests to store speaker activities,
it allows us to use either oracle segments, or read diarization output in the
form of RTTM files with the same data processing pipeline. A typical recipe
for enhancing a corpus using GSS is below:

⬇

#!/bin/bash

# 1. Create Lhotse manifests for corpus

lhotse prepare libricss \--type mdm $corpus_dir $data_dir

# 2. Prepare recording-level cuts

lhotse cut simple -r $data_dir/recordings.jsonl.gz -s
$data_dir/supervisions.jsonl.gz $exp_dir/cuts.jsonl.gz

# 3. Prepare segment-level cuts

lhotse cut trim-to-supervisions \--discard-overlapping $exp_dir/cuts.jsonl.gz
$exp_dir/segments.jsonl.gz

# 4. Perform enhancement

gss enhance cuts \--max-batch-duration 50.0 $exp_dir/cuts.jsonl.gz
$exp_dir/segments.jsonl.gz $exp_dir/enhanced

###  4.5 Experiments

####  4.5.1 Setup

We performed evaluations on three publicly-available meeting datasets:
LibriCSS, AMI, and AliMeeting. Detailed statistics for all datasets are in
Section 1.4. We used three different microphone settings for our experiments:
IHM (individual headset microphone), SDM (single distant microphone), and GSS
(GSS-enhanced multi-mic). Since LibriCSS does not provide headset recordings,
we used the corresponding digitally-mixed LibriSpeech utterances to simulate
IHM. For all datasets, the first channel of the first array was used for the
SDM setting. For LibriCSS and AliMeeting, we used all available channels for
GSS, whereas for AMI, we used the first of the two arrays.

We evaluated the target speaker extraction quality using both intrinsic and
extrinsic measures. We consider the IHM recording as the clean signal, the SDM
recording as the mixed signal, and the output of the GSS system as the
enhanced signal. We normalized the loudness of the signals to be equal to that
of the clean signal, and also aligned the samples, before computing these
metrics. We used the pyloudnorm333https://github.com/csteinmetz1/pyloudnorm
and fast-align-audio444https://github.com/nomonosound/fast-align-audio
libraries for loudness normalization and sample alignment, respectively [165].
We report the signal-level enhancement performance in terms of PESQ, SI-SDR,
and STOI, as described in Section 1.5. For extrinsic evaluation, we want to
measure two aspects of the enhanced signal:

  1. 1.

Does the enhanced signal retain target speaker characteristics more than the
mixed signal?

  2. 2.

Is the enhanced signal easier to transcribe than the mixed signal for an ASR
system trained on single-speaker utterances?

For the first question, we used embedding extractors to compute speaker
embeddings for the mixed, clean, and enhanced signals — let us denote the
embeddings as $\mathbf{s}_{m}$, $\mathbf{s}_{c}$, and $\mathbf{s}_{e}$,
respectively. Next, we compute the following increase in cosine similarity

$\Delta_{\text{spk}}=cos(\mathbf{s}_{c},\mathbf{s}_{e})-cos(\mathbf{s}_{c},\mathbf{s}_{m})$
(4.12)

as a measure of the improvement in speaker information when going from the
mixed signal to the enhanced signal, where $cos(\mathbf{x},\mathbf{y})$
denotes the cosine similarity between $\mathbf{x}$ and $\mathbf{y}$. Since the
range of cosine similarity is [-1, 1], the range of $\Delta_{\text{spk}}$ is
[-2, 2], with a higher value denoting a larger improvement in speaker
information. We used two off-the-shelf speaker embedding extractors from the
SpeechBrain toolkit [166] to obtain the embeddings: xvect and ecapa. The
former is a conventional x-vector model using TDNN and stats pooling layers,
and is trained with the regular softmax and cross-entropy loss [63]. The
latter is based on a newer ECAPA-TDNN model architecture and is trained with
additive margin softmax (AM-softmax) [167]. The models obtain equal error
rates (EER) of 3.2% and 0.8% on the VoxCeleb-1 test set, respectively, meaning
that ecapa has a better discriminative capability than xvect.

For the second question, i.e., we measured impact on ASR performance by
training separate transducer-based ASR models for each benchmark. We will
define neural transducers briefly in Chapter 5, when we describe the complete
transcription pipeline. Here, we provide the system configuration details for
these models. For LibriCSS, we used a pretrained Conformer-transducer [168]
trained on LibriSpeech. In this model, the encoder is a 12-layer Conformer
containing 8 attention heads, attention dimension of 256, and feed-forward
dimension of 2048. The prediction network is a 512-dimensional 1D convolution
with a bi-gram context, and the joiner is a 512-dimensional feed-forward
layer. For AMI and AliMeeting, we trained a similar transducer on a
combination of IHM, IHM with simulated reverb, SDM, and GSS-enhanced far-field
recordings of the corresponding train set, and the resulting model was used to
evaluate all microphone settings. This model’s configuration is similar to the
one described earlier, with the exception that we used a Zipformer for the
encoder [169]. The Zipformer encoder consists of 5 blocks, each containing
multiple self-attention and feed-forward layers. Each block is sub-sampled at
different frame rates, with the intermediate ones more strongly down-sampled
(up to 8x, for instance). Within each block, self-attention weights are
computed once and shared for all feed-forward layers in that block. The result
of these optimizations is that the encoder converges faster and better than
the Conformer for the same model size. In all cases, we applied three-fold
speed perturbation and noise augmentation using MUSAN [103] noises. We used a
“stateless” decoder consisting of a convolutional layer with a bi-gram
context. The model was trained using a pruned RNN-T loss [170] implemented in
k2555https://github.com/k2-fsa/{k2,icefall}. For decoding, we used a WFST-
based parallel beam search method with beam size 4 [171]. The ASR performance
is reported in terms of word error rate (WER).

####  4.5.2 Results

#####  4.5.2.1 Signal-level enhancement

First, we measure the improvement in signal-level metrics: PESQ, SI-SDR, and
STOI, as shown in Table 4.1. For LibriCSS, we show a breakdown of these
metrics by overlap ratio, in order to better understand the effect of GSS on
target speaker extraction. We see that GSS-based TSE provides a consistent
improvement across all metrics, as seen in the $\Delta$ for the metrics. The
perceptual quality, as measured by PESQ, decreases with increase in overlap
ratio, and the relative improvement provided by GSS also reduces. The SI-SDR
was found to be negative for both the mixture and enhanced signal, which means
that the signal is weaker than the distortion. However, GSS resulted in a
relative increase in SI-SDR by 19.73 dB, 11.53 dB, and 17.92 dB, respectively,
for LibriCSS, AMI, and AliMeeting. Intelligibility for LibriCSS, as measured
by STOI, decreased for input mixtures with increase in overlap ratio, but the
corresponding output remained consistent. For AMI and AliMeeting, the STOI
improvement was found to be lower, which may be because real meetings may
contain very short, often unintelligible utterances.

Table 4.1: Signal-level enhacement performance of GSS in terms of PESQ, SI-
SDR, and STOI metrics. For each metric, we report the measure on the input
mixture (in), the enhanced signal (out), and the difference ($\Delta$).

Dataset Subset PESQ SI-SDR STOI in out $\Delta$ in out $\Delta$ in out
$\Delta$ LibriCSS 0L $1.58$ $2.12$ $0.54$ $-29.13$ $-10.94$ $18.19$ $0.50$
$0.85$ $0.35$ 0S $1.54$ $2.07$ $0.53$ $-30.40$ $-12.46$ $17.94$ $0.47$ $0.83$
$0.36$ OV10 $1.40$ $1.84$ $0.44$ $-30.24$ $-10.44$ $19.80$ $0.47$ $0.86$
$0.39$ OV20 $1.35$ $1.76$ $0.41$ $-31.60$ $-10.50$ $21.10$ $0.45$ $0.85$
$0.40$ OV30 $1.27$ $1.66$ $0.39$ $-30.76$ $-11.21$ $19.55$ $0.45$ $0.83$
$0.38$ OV40 $1.24$ $1.63$ $0.39$ $-32.37$ $-10.59$ $21.78$ $0.43$ $0.84$
$0.41$ AMI test $1.20$ $1.45$ $0.25$ $-17.87$ $-6.34$ $11.53$ $0.61$ $0.74$
$0.13$ AliMeeting test $1.23$ $1.52$ $0.29$ $-34.14$ $-16.22$ $17.92$ $0.27$
$0.57$ $0.30$

####  4.5.3 Improvement in target-speaker information

Table 4.2: Improvement in target-speaker information for enhanced signal
compared to mixed signal, measured in terms of $\Delta_{\text{spk}}$. The
columns denote the speaker embedding extractor used.

Dataset Subset xvect ecapa LibriCSS 0L $0.013$ $0.055$ 0S $0.012$ $0.057$ OV10
$0.012$ $0.068$ OV20 $0.013$ $0.089$ OV30 $0.015$ $0.124$ OV40 $0.018$ $0.167$
AMI test $0.025$ $0.152$ AliMeeting test $0.034$ $0.185$

For our first extrinsic evaluation, we compute $\Delta_{\text{spk}}$ as
defined in (4.12) for all three datasets using xvect and ecapa, as shown in
Table 4.2. As expected, the $\Delta_{\text{spk}}$ values are positive across
the board, indicating that the target-speaker information improves through
enhancement. On the LibriCSS data, as the overlap ratio increases,
$\Delta_{\text{spk}}$ also increases, primarily because it is very hard to
extract good speaker embeddings from mixed speech with high overlaps. The
improvement is also, in general, larger for AMI and AliMeeting compared to
LibriCSS, since these datasets often contain very short phrases which may not
contain enough speaker information, especially in the presence of background
noise or interfering speakers. Among the embedding extractors, we obtained
larger improvements using ecapa compared to xvect, perhaps since the former is
trained with an additive margin softmax loss. Nevertheless, the two embedding
spaces are likely very different so the absolute values are not comparable
between the extractors.

#####  4.5.3.1 Far-field ASR performance

Table 4.3: Comparison of close-talk and far-field ASR performance for meeting datasets. The GSS setting uses 7 channels for LibriCSS and 8 channels for AMI and AliMeeting. †LibriCSS IHM refers to the corresponding LibriSpeech utterances. #For AliMeeting, the numbers are CER. Dataset | Setting | Ins. | Del. | Sub. | WER/CER
---|---|---|---|---|---
LibriCSS | IHM† | 0.25 | 0.22 | 1.74 | 2.21
SDM | 1.06 | 3.12 | 6.59 | 10.77
GSS | 0.31 | 0.89 | 2.14 | 3.34
AMI | IHM | 2.22 | 4.51 | 11.31 | 18.04
SDM | 4.01 | 9.59 | 18.50 | 32.10
GSS | 2.43 | 6.07 | 14.33 | 22.83
AliMeeting# | IHM | 0.97 | 3.78 | 7.32 | 12.07
SDM | 1.99 | 10.00 | 14.38 | 26.38
GSS | 1.09 | 4.87 | 9.03 | 14.98

Finally, we measure the improvement in far-field ASR performance when using
GSS with oracle segmentation, as shown in Table 4.3. The IHM and SDM settings
may be considered as the lower and upper bounds on WER (or CER), respectively.
We found that across all the datasets, GSS improved ASR performance
significantly, with the recovered error rates, defined as
$\left(W_{\mathrm{SDM}}-W_{\mathrm{GSS}}\right)/\left(W_{\mathrm{SDM}}-W_{\mathrm{IHM}}\right)$,
being 86.8%, 65.9%, and 80.4% for LibriCSS, AMI, and AliMeeting, respectively.
As expected, most of the improvement was obtained from recovered deletion and
substitution errors, possibly from better recognition of overlapped speech
segments. To verify this conjecture, we grouped all segments in the AMI test
set by their overlap ratios to form bins of 20% intervals, and computed the
average WER for each such bin. The resulting bar plot is shown in Fig. 4.5. We
can see that as the overlap ratio increases, the average WER increases more
strongly for the SDM setting as compared to the GSS-enhanced utterances.

Figure 4.5: Average WER for different microphone settings and overlap ratios.

#####  4.5.3.2 Which factors are most important for GSS?

Figure 4.6: Impact of several factors on ASR performance. In each figure, the
left and right y-axes denoten WERs for LibriCSS (test) and AMI (dev),
respectively, with the axes scaled according to the range of corresponding WER
values.

We performed ablation studies to investigate the effect of several GSS
parameters — WPE, noise class, context duration, number of iterations for
CACGMM inference, and number of input channels — on the downstream ASR
performance, as shown in Fig. 4.6. WPE was found to be more important for
LibriCSS, while using an additional noise class was more important for AMI
(Fig. 4.6). This may be because AMI contains occasional background noise,
which is absent in LibriCSS. Increasing the context duration from 5s to 15s
resulted in consistent WER gains (Fig. 4.6). [28] have previously made a
similar observation for CHiME-5 [172], where a 15s context resulted in better
WER compared to a 2s context. However, we found that adding further context
beyond 15s degraded WER performance. If the context is expanded too far, the
context window may also contain target-speaker segments, which would
contaminate the statistics of the noise class.

For both datasets, increasing the number of BSS iterations (for CACGMM
inference) beyond 5 did not result in any WER improvements (Fig. 4.6).
Finally, using more input channels was found to be the single most important
factor for better WER performance. For example, using seven input channels
resulted in relative WER reduction of 50.4% and 21.8% on LibriCSS and AMI,
respectively, compared to using two channels. Nevertheless, it follows the law
of diminishing returns, as evident by the rate of decay in Fig. 4.6.

#####  4.5.3.3 Analysis of speed-up

We compared our GSS implementation with the original GSS on the CHiME-6
development set in terms of wall clock time and ASR performance, as shown in
Table 4.4. For ASR inference, we used the publicly available Kaldi recipe and
pretrained models from JHU-CLSP’s submission to the CHiME-6
challenge666https://github.com/kaldi-
asr/kaldi/blob/master/egs/chime6/s5b_track1,777https://kaldi-
asr.org/models/m12 [39]. We found that our implementation obtained an
effective speed-up of 290 without any degradation in WER. Since CHiME-6 has
non-stationary speakers, we disabled segment batching for this experiment. We
can obtain even further speed-ups by enabling this for meeting-like data where
speakers are stationary.

Table 4.4: Compute time for our GSS implementation compared with original on
CHiME-6 dev set, using all available channels, 15s context, and 20 BSS
iterations. “Time” is the actual wall clock time (in hours), while “cum. time”
is the effective total time for all jobs. Speedup is the ratio of the
cumulative times.

GSS Compute Time Cum. time Speedup WER +RNNLM Original 80 x Xeon 19.3 1542.6
1.0 44.7 43.5 Ours 4 x V100 1.3 5.3 292.2 44.2 43.1

###  4.6 Conclusion

In this chapter, we described the problem of target speaker extraction, and
showed how to do it using guided source separation (GSS). Although GSS was
first proposed in [28] for the CHiME-5 challenge, it has seen limited use for
meeting transcription, partially because of a slow and iterative
implementation. This has also made it challenging to perform detailed ablation
studies for this method. To solve this issue, we proposed a novel GPU-
accelerated implementation of this technique, drawing inspiration from modern
deep learning pipelines. On the CHiME-6 benchmark, it was found to be 300x
faster than the original implementation, thus removing the computational
bottleneck associated with this technique.

Through experiments conducted on LibriCSS, AMI, and AliMeeting, we showed that
GSS provides between 10 and 20 dB SI-SDR improvements compared to single-
channel far-field recordings, when measured against close-talk recordings, and
significant perceptual and intelligibility improvements. Extrinsic evaluation
on ASR and speaker similarity showed that GSS-based enhancement can recover up
to 80% of the WER difference, in going from close-talk to far-field
conditions, and also drastically improve target-speaker information in the
signal. We performed several ablation studies to study the effect of GSS
parameters, and showed that using more input channels is the single most
important factor for better ASR performance.

With this implementation and analysis, we have developed another important
component in our modular system for speaker-attributed multi-talker ASR. A
fast and accurate target-speaker extraction method is essential to be able to
obtain single-speaker segments from multi-talker mixtures, which can then be
transcribed by an ASR module. Furthermore, using GSS for this task allows us
to leverage the overlap-aware diarization system that we have developed
earlier. Our ablation studies show that we can obtain good TSE performance
with GSS with few iterations, provided we have enough input channels. This is
useful because most devices, such as smart speakers, already contain array
microphones which can provide such multi-channel inputs. More importantly, the
GSS method is invariant to the number of input channels — this means that the
same TSE module can be used across a wide variety of scenarios, and the output
can be fed into a universal ASR component. In the next chapter, we will tie
together all of these components to build a complete pipeline for speaker-
attributed transcription.

##  Chapter 5 A Modular Framework for Multi-talker Speech Recognition

In the last several chapters, we formulated and analyzed individual components
that make up the modular framework for multi-talker ASR. With these components
in place, we can build a system that takes as input a multi-channel recording
and generates a speaker-attributed transcription of the recording. In this
chapter, we describe this modular system and use it to perform transcription
for several meeting-style benchmark data. We will then analyze the performance
to understand the role of the different components in causing and propagating
errors in transcription.

###  5.1 Probabilistic formulation of multi-talker ASR

Let $R$ be a long, multi-channel recording, recorded using $M$ microphones.
The objective of the multi-talker ASR (or “who spoke what”) problem is to
obtain a speaker-attributed transcription, $W(R)$, defined as

$W(R)=\\{\mathbf{w}_{1},\ldots,\mathbf{w}_{K}\\},$ (5.1)

where $K$ is the number of speakers in the recording $R$, and $\mathbf{w}_{k}$
denotes the transcription for speaker $k$. For evaluation, we will consider
the concatenated minimum-permutation word error rate (cpWER) metric, described
in Section 1.5, and so all permutations of $\mathbf{w}_{k}$’s are equivalent
under cpWER.

In the modular framework, instead of directly optimizing for $W(R)$, we will
instead compute a proxy solution $h(R)$, which is defined as

$h(R)=\\{(\Delta_{j},u_{j},\mathbf{y}_{j}):1\leq j\leq N\\},$ (5.2)

where $\Delta_{j}=(t_{j}^{\mathrm{st}},t_{j}^{\mathrm{en}})$ denotes the start
and end times for segment $j$, $u_{j}\in[K]$ is the speaker label assigned to
segment $j$, $N$ denotes the number of estimated segments, and
$\mathbf{y}_{j}$ denotes the predicted transcript for segment $j$. It is easy
to show that a deterministic function $\mathcal{Y}$ exists that maps $h(R)$ to
some $W(R)$ by concatenating the transcripts of same-speaker segments in order
of start time. Let $\hat{h}$ maximize $P(h(R)\mid R)$ and $\hat{W}$ maximize
$P(W\mid R)$. Since $\mathcal{Y}$ is many-to-one, we cannot guarantee that
$\hat{W}=\mathcal{Y}(\hat{h})$ in general. Nevertheless, we will use this
proxy task since it provides convenient factorization.

From a probabilistic perspective, finding the optimal $h(R)$ can be formulated
as a maximum a posteriori problem such that

$h(R)=\text{arg}\max_{h^{\prime}}P(h^{\prime}\mid R),\quad\text{where}\quad
h^{\prime}=\\{(\Delta_{j},u_{j},\mathbf{y}_{j}):1\leq j\leq N\\}.$ (5.3)

We can simplify the $P(h^{\prime}\mid R)$ term through basic probability rules
and conditional independence assumptions. Suppose
$\Delta_{1}^{N}=\\{\Delta_{j}:1\leq j\leq N\\}$, and likewise for $u_{1}^{N}$
and $\mathbf{y}_{1}^{N}$. We have

$\displaystyle P(h^{\prime}\mid R)$
$\displaystyle=P\left(\Delta_{1}^{N},u_{1}^{N},\mathbf{y}_{1}^{N}\mid
R\right)$ (5.4) $\displaystyle=P\left(\Delta_{1}^{N},u_{1}^{N}\mid
R\right)P\left(\mathbf{y}_{1}^{N}\mid R,\Delta_{1}^{N},u_{1}^{N}\right).$
(5.5)

Here, the first term may be considered a probabilistic formulation of the
speaker diarization task. For the second term, let us define $N$ continuous
random variables $\mathbf{X}_{1},\ldots,\mathbf{X}_{N}$ representing the
“target speaker signal” for the $N$ segments, respectively. We can then
marginalize over the $\mathbf{X}_{1}^{N}$ to get

$\displaystyle P\left(\mathbf{y}_{1}^{N}\mid
R,\Delta_{1}^{N},u_{1}^{N}\right)$
$\displaystyle=\int_{\mathbf{X}_{1}^{N}}P\left(\mathbf{X}_{1}^{N},\mathbf{y}_{1}^{N}\mid
R,\Delta_{1}^{N},u_{1}^{N}\right)$ (5.6)
$\displaystyle=\int_{\mathbf{X}_{1}^{N}}P\left(\mathbf{X}_{1}^{N}\mid
R,\Delta_{1}^{N},u_{1}^{N}\right)P\left(\mathbf{y}_{1}^{N}\mid
R,\Delta_{1}^{N},u_{1}^{N},\mathbf{X}_{1}^{N}\right)$ (5.7)
$\displaystyle=\int_{\mathbf{X}_{1}^{N}}\prod_{j=1}^{N}P\left(\mathbf{X}_{j}\mid
R,\Delta_{j},u_{j}\right)\prod_{j=1}^{N}P\left(\mathbf{y}_{j}\mid\mathbf{X}_{j}\right),$
(5.8)

where (5.8) is based on the following conditional independence assumptions.

  1. 1.

Given $R$, $\Delta_{j}$, and $u_{j}$, $\mathbf{X}_{j}$ is conditionally
independent of $\mathbf{X}_{i}$, $\Delta_{i}$, and $u_{i}$, $\forall
i\in[N]\setminus\\{j\\}$.

  2. 2.

Given $\mathbf{X}_{j}$, $\mathbf{y}_{j}$ is conditionally independent of all
other random variables.

We will discuss these assumptions in more detail in the following subsections.
Plugging (5.8) into (5.5), we get

$h(R)=\mathrm{arg}\max_{\Delta_{1}^{N},u_{1}^{N},\mathbf{y}_{1}^{N}}\left[P\left(\Delta_{1}^{N},u_{1}^{N}\mid
R\right)\int_{\mathbf{X}_{1}^{N}}\prod_{j=1}^{N}P\left(\mathbf{X}_{j}\mid
R,\Delta_{j},u_{j}\right)\prod_{j=1}^{N}P\left(\mathbf{y}_{j}\mid\mathbf{X}_{j}\right)\right].$
(5.9)

Under the “modular” framework, we make some further modeling choices for
computational tractability. Instead of solving for the joint optimum in (5.9),
we first solve the diarization task to obtain

$\hat{\Delta}_{1}^{N},\hat{u}_{1}^{N}=\underbrace{\arg\max_{\Delta_{1}^{N},u_{1}^{N}}P\left(\Delta_{1}^{N},u_{1}^{N}\mid
R\right)}_{\text{speaker diarization}}.$ (5.10)

Next, we assume that $\mathbf{X}_{j}$ is approximated using a deterministic
function $g(\cdot)$ which is estimated through the target speaker extraction
task as

$\hat{\mathbf{X}}_{j}=\underbrace{g(R,\hat{\Delta}_{j},\hat{u}_{j})}_{\text{target
speaker extraction}}.$ (5.11)

Following this assumption, we can remove the marginalization (since we are
using a point-wise estimate for $\mathbf{X}_{1}^{N}$) and the second term in
(5.8) can be approximated as

$P\left(\mathbf{y}_{1}^{N}\mid
R,\Delta_{1}^{N},u_{1}^{N}\right)\approx\prod_{j=1}^{N}\underbrace{P\left(\mathbf{y}_{j}\mid\hat{\mathbf{X}}_{j}\right)}_{\text{speech
recognition}}.$ (5.12)

Figure 5.1: Flow diagram of the modular system for speaker-attributed
transcription. The input is a multi-channel recording, and the output is
speaker-wise transcripts.

A block diagram representing this modular system is shown in Fig. 5.1. It
consists of three components: (i) overlap-aware speaker diarization; (ii)
target-speaker extraction; and (iii) automatic speech recognition (ASR). In
the previous chapters, we have described (i) and (ii) in some detail. Next, we
will summarize them in the context of the full system, and describe the ASR
component, which is based on neural transducers.

####  5.1.1 Overlap-aware diarization

As described earlier in Section 2.1, a diarization system is first required to
obtain $f(R)=\\{(\Delta_{j},u_{j}):1\leq j\leq N\\}$. Although $R$ is a multi-
channel recording, our diarization system only makes use of the first-channel
to compute $f(R)$. Specifically, we used the multi-class spectral clustering
based diarization system with overlap assignment, described in Section 2.4,
which consists of (i) a speech activity detector (SAD), (ii) a speaker
embedding extractor, (iii) an overlap detector, and (iv) an overlap-aware
spectral clustering module.

As explained in Section 1.5, performing well in diarization — using the
diarization error rate (DER) metric — requires accurate estimation of
$\Delta_{1}^{N}$. However, our eventual objective is to perform speaker-
attributed transcription, which is evaluated using cpWER. While cpWER does not
explicitly penalize incorrect estimates of $\Delta_{1}^{N}$, it is implicitly
penalized since we need accurate time-stamps for two reasons. First, accurate
estimate of speaker activities and segmentation is required for target-speaker
extraction to work well, as evident from (5.11). Second, the final
transcription is obtained by time-ordered concatenation of a speaker’s
transcripts, for which we also need reasonably accurate start times. For these
reasons, downstream cpWER performance is weakly correlated with the
performance of the diarization system.

####  5.1.2 Target-speaker extraction

From (5.11), we have modeled $\mathbf{X}_{j}$ using target-speaker extraction
as

$\hat{\mathbf{X}}_{j}=g(R,\hat{\Delta}_{j},\hat{u}_{j}).$

$\hat{\mathbf{X}}_{j}$ contains only the signal relevant to speaker
$\hat{u}_{j}$, and suppresses background noise or interfering speakers. In our
pipeline, this target-speaker extraction is performed using the guided source
separation (GSS) method described in Chapter 4. By using GSS, our conditional
independence assumptions in (5.8) are valid, since the extraction is performed
for each segment independent of other segments.

####  5.1.3 Automatic speech recognition

Finally, given the output of the targer-speaker extractor, an ASR model is
applied on each segment $\hat{\mathbf{X}}_{j}$ to obtain the corresponding
transcript $\mathbf{y}_{j}$. These are then combined with
$f(R)=\\{(\Delta_{j},u_{j}):1\leq N\\}$ to obtain the desired output, i.e.,
$h(R)=\\{(\Delta_{j},u_{j},\mathbf{y}_{j}):1\leq N\\}$.

In general, any offline ASR model can be used for this component, such as
hybrid HMM-DNN [173], connectionist temporal classification (CTC) [174], or
attention-based encoder-decoder (AED) [175]. We used neural transducers in our
implementation. Neural transducers (using RNNs or transformers) [176] have
become the dominant modeling technique in end-to-end on-device speech
recognition [177, 178, 179, 180], since they allow streaming transcription
similar to CTC models [181, 182], while still retaining conditional
dependence, like AEDs [183, 184]. We will briefly describe the formulation of
neural transducers for ASR.

In conventional single-talker ASR, audio features for a segmented utterance
$\mathbf{X}\in\mathbb{R}^{T\times F}$ (where $T$ and $F$ denote the number of
time frames and the input feature dimension, respectively) are provided as
input to the system, and we are required to predict the transcript
$\mathbf{y}=(y_{1},\ldots,y_{U})$, where $y_{u}\in\mathcal{V}$ denotes output
units such as graphemes or word-pieces, and $U$ is the length of the label
sequence. For the case of discriminative training, this requires computing the
conditional likelihood $P(\mathbf{y}\mid\mathbf{X})$ (or its log for numerical
stability). For inference, we search for
$\hat{\mathbf{y}}=\text{arg}\max_{\mathbf{y}}P(\mathbf{y}\mid\mathbf{X})$,
often in a constrained search space using greedy or beam search. Transducers
achieve this by marginalizing over the set of all alignments
$\mathbf{a}\in\bar{\mathcal{V}}^{T+U}$, where
$\bar{\mathcal{V}}=\mathcal{V}\cup\\{\phi\\}$ and $\phi$ is called the blank
label. Formally,

$P(\mathbf{y}\mid\mathbf{X})=\sum_{\mathbf{a}\in\mathcal{B}^{-1}(\mathbf{y})}P(\mathbf{a}\mid\mathbf{X}),$
(5.13)

where $\mathcal{B}$ is a deterministic mapping from an alignment $\mathbf{a}$
to an output sequence $\mathbf{y}$. In the original transducer, all alignments
$\mathbf{a}$ consist of $\mathbf{y}$ interspersed with $T$ blank tokens,
usually represented as a $T\times U$ lattice with $\phi$ on horizontal arcs
and $\mathbf{y}$ on vertical arcs. Since there may be ${T+U\choose U}$ such
paths on the lattice, some transducer variants (such as recurrent neural
aligner [185] or monotonic RNN-T [186, 187]) restrict the number of non-blank
tokens emitted per time step. External or internal alignments may also be used
to further prune the lattice for marginalization [188, 170].

Figure 5.2: Block diagram of the neural transducer.

Transducers parameterize $P(\mathbf{a}\mid\mathbf{X})$ with an encoder, a
prediction network, and a joiner, as shown in Fig. 5.2. The encoder maps
$\mathbf{X}$ into hidden representations $\mathbf{f}_{1}^{T}$, while the
prediction network maps $\mathbf{y}$ into $\mathbf{g}_{1}^{U}$. The joiner
combines the outputs from the encoder and the prediction network to compute
logits $\mathbf{z}_{t,u}$ which are fed to a softmax function to produce a
posterior distribution over $\bar{\mathcal{V}}$ for each time step. Under
assumptions of full context encoder, we can expand (5.13) as

$\displaystyle P(\mathbf{y}\mid\mathbf{X})$
$\displaystyle=\sum_{\mathbf{a}\in\mathcal{B}^{-1}(\mathbf{y})}\prod_{t=1}^{T+U}P(\mathbf{a}_{t}\mid\mathbf{f}_{1}^{T},\mathbf{g}_{1}^{u(t)-1})$
(5.14)
$\displaystyle=\sum_{\mathbf{a}\in\mathcal{B}^{-1}(\mathbf{y})}\prod_{t=1}^{T+U}\mathrm{Softmax}(\mathbf{z}_{t,u(t)}),$
(5.15)

where $u(t)\in\\{1,\ldots,U\\}$ denotes the index in the label sequence at
time $t$.

###  5.2 Experimental setup

We show speaker-attributed transcription results on three meeting benchmarks:
LibriCSS, AMI, and AliMeeting. For these experiments, we report diarization
error rates (DER) and concatenated minimum-permutation WER (cpWER) in order to
analyze the impact of diarization errors on downstream ASR. In the case of
AliMeeting, we report cpCER (instead of cpWER) since this is a Mandarin
dataset. We did not use any collars to compute DERs for LibriCSS and AMI, but
a collar of 0.25 was used for AliMeeting following the original work.

For speaker embedding extraction, we used a pre-trained ResNet101-based
network [69], which was trained on VoxCeleb [189, 190] and CN-Celeb [191]. For
SAD and overlap detection, we used an end-to-end segmentation method based on
Pyannote [192]. The segmentation model was fine-tuned on the training set
annotations for AMI and AliMeeting, and used off-the-shelf for LibriCSS. For
the ASR component, we used neural transducers with the same configuration as
the ones described in Section 4.5.1. We have reproduced the key results from
Table 4.3 below for reference, as these denotes the minimum achievable cpWER
if we have access to an oracle diarization system.

Table 5.1: WER (or CER) results using an oracle diarization system and GSS. These represent the minimum achievable cpWER (or cpCER) under the modular framework using the same ASR. Dataset | Ins. | Del. | Sub. | WER
---|---|---|---|---
LibriCSS | 0.31 | 0.89 | 2.14 | 3.34
AMI | 2.43 | 6.07 | 14.33 | 22.83
AliMeeting | 1.09 | 4.87 | 9.03 | 14.98

###  5.3 Results & Discussion

Table 5.2: Effect of diarization and GSS components on speaker-attributed ASR,
as measured by cpWER (or cpCER).

Diarizer DER GSS cpWER FA MS Conf. Total Ins. Del. Sub. Total LibriCSS
Spectral 1.19 10.37 3.37 14.93 ✗ 1.00 13.62 3.69 18.30 ✓ 0.73 12.33 2.80 15.86
\+ OVL 2.22 3.79 5.33 11.34 ✗ 2.64 8.09 6.36 17.09 ✓ 1.62 7.13 3.38 12.12 AMI
Spectral 3.24 18.15 4.14 25.53 ✗ 2.64 20.32 15.49 38.45 ✓ 2.59 18.00 12.96
33.55 \+ OVL 7.39 9.63 6.67 23.69 ✗ 4.37 14.47 19.70 38.54 ✓ 3.57 12.24 15.21
31.02 AliMeeting Spectral 0.17 13.60 2.60 16.37 ✗ 1.16 26.38 10.05 37.59 ✓
0.86 24.34 7.23 32.43 \+ OVL 2.83 5.96 5.64 14.43 ✗ 2.32 18.82 14.30 35.44 ✓
1.69 17.00 9.76 28.45

Table 5.2 presents the main results for this chapter. For each of the three
meeting benchmarks, we show the DER and corresponding cpWER results when the
diarization system is (or is not) overlap-aware. Furthermore, for each of
these systems, we compare the performance when using GSS-based target-speaker
extraction, as opposed to simply extracting the corresponding segment from the
input mixture. These results demonstrate the effect of (i) the overlap-aware
spectral clustering and (ii) GSS-based TSE modules, on downstream speaker-
attributed ASR performance.

####  5.3.1 Results for different benchmarks

Let us first consider the results for the different benchmarks. As expected,
LibriCSS is the easiest dataset since it is simulated from LibriSpeech
utterances and contains long utterances. On the other hand, AMI and AliMeeting
are significantly harder due to the presence of natural conversational speech
(such as backchannels). Overall, the breakdown of errors in cpWER follows that
of the diarization errors, i.e., insertion, deletion, and substitutions are
roughly proportional to false alarms, missed speech, and speaker confusion,
respectively. Most errors come from deletions, possibly due to missed
detection of overlapped speech. For the real meeting benchmarks, substitutions
are also significantly higher than LibriCSS. We conjecture that very short
utterances, such as those found in these datasets, are hard for speaker
attribution, since the speaker embedding extractors are usually trained on
longer utterances.

####  5.3.2 Effect of overlap-aware diarization

Next, let us consider the impact of overlap-aware diarization, i.e.,
“Spectral” versus “+OVL“. Overlap assignment improves diarization performance
significantly, as measured by DER. However, when no GSS is performed, the
impact on downstream cpWER is negligible. For example, the cpWER on LibriCSS
improves by 1.21% absolute, while it actually degrades for AMI (from 38.45% to
38.54%). This is due to the fact that although our system detects more
overlapping segments, these cannot be transcribed correctly since the ASR
cannot handle overlapping speech. This finding corroborates the results of the
winning CHiME-6 system [153], which was able to substantially improve ASR
performance on unsegmented recordings using TS-VAD based diarization [80] and
GSS-based enhancement.

####  5.3.3 Effect of target-speaker extraction

Finally, using GSS results in significant improvements, with relative cpWER
(or cpCER) reductions of 29.1%, 19.5%, and 19.7% on LibriCSS, AMI, and
AliMeeting, respectively. Nevertheless, the cpWER performance is still far
from the WERs obtained using an oracle diarizer. While part of the difference
is due to incorrect speaker attribution of the ASR output, other reasons are
undetected overlapping segments and poor mask estimation due to inaccurate
speaker activities.

####  5.3.4 Qualitative analysis

Table 5.3: Examples of speaker-attributed transcription for a short segment of
the session ES2011a from AMI dev set, between 817s and 833s. The first block
shows the reference transcript for this segment. Each line in the transcript
denotes a different speaker. Different segments of the same speaker are
separated with a pipe (“|”) character.

Diarizer GSS Transcript cpWER – – I ALSO THINK THOUGH THAT IT SHOULDN’T HAVE TOO MANY BUTTONS ’CAUSE I HATE THAT WHEN THEY HAVE TOO MANY BUTTONS AND I MEAN I KNOW IT HAS TO HAVE ENOUGH FUNCTIONS BUT LIKE I DON’T KNOW YOU JUST HAVE LIKE EIGHT THOUSAND BUTTONS AND YOU’RE LIKE NO YOU NEVER USE HALF OF THEM | SO YEAH I AGREE | B BUTTON AND THE F BUTTON THEY DON’T DO ANYTHING UM OH WE JUST | YEAH YEAH YEAH YEAH 0% Oracle ✓ I ALSO THINK THOUGH THAT IT SHOULDN’T HAVE TOO MANY BUTTONS ’CAUSE I HATE THAT ONLY HAVE TOO MANY BUTTONS AND I MEAN I KNOW IT HAS TO HAVE ENOUGH FUNCTIONS BUT LIKE JUST HAVE LIKE EIGHT THOUSAND BUTTONS AND YOU’RE LIKE NO YOU NEVER USE HALF OF THEM YEAH I AGREE M THE BUTTON ON THE OFF BUTTON THEY DON’T DO ANYTHING UM OH WE’RE JUST USING THAT | YEAH YEAH YEAH 20.3% Spectral ✓ I ALSO THINK THOUGH THAT IT SHOULDN’T HAVE TOO MANY BUTTONS ’CAUSE I HATE THAT ONLY HAVE TOO MANY BUTTONS AND I MEAN I KNOW IT HAS TO HAVE MANY FUNCTIONS BUT LIKE | I DUNNO JUST HAVE LIKE EIGHT THOUSAND BUTTONS AND YOU’RE LIKE NO YOU NEVER USE HALF OF THEM 40.5% \+ OVL ✓ I ALSO THINK THOUGH THAT IT SHOULDN’T HAVE TOO MANY BUTTONS ’CAUSE I HAD THAT ONLY HAVE TOO MANY BUTTONS AND I MEAN I KNOW IT HAS TO HAVE ENOUGH FUNCTIONS BUT LIKE | I DUNNO JUST HAVE LIKE EIGHT THOUSAND BUTTONS AND YOU’RE LIKE NO YOU NEVER USE HALF OF THEM S YEAH I AGREE M THE BUTTON ON F BUTTON THEY DON’T DO ANYTHING 29.1% ✗ I ALSO THINK THAT IT SHOULDN’T HAVE TOO MANY BUTTONS ’CAUSE I HATED NOT ONLY HAVE TOO MANY BUTTONS AND THINGS BUT I MEAN I KNOW IT HAS TO HAVE NO MANY FUNCTIONS BUT LIKE | I DUNNO JUST HAVE LIKE EIGHT THOUSAND BUTTONS AND YOU’RE LIKE YOU KNOW YOU NEVER USE HALF THE TIME IT SHOULDN’T HAVE TOO MANY BUTTONS ’CAUSE I HATE THAT ONLY HAVE TOO MANY BUTTONS AND THINGS BUT I MEAN I KNOW IT HAS TO HAVE NO MANY FUNCTIONS BUT LIKE 72.2%

In the previous sections, we have quantitatively described the effect of
overlap-aware diarization and GSS on transcription in terms of cpWER. Let us
now consider an illustrative example of the transcriptions produced by the
various systems, in order to qualitatively understand the differences. In
Table 5.3, we show the reference and hypotheses transcriptions for a 16-second
segment (from 817s to 833s) of session ES2011a, which is part of the AMI dev
set. The reference contains several segments of speech from four different
speakers, as shown in the first block of the table. Two of the four speakers
only have very short utterances or back-channels.

The best possible system, obtained using an oracle diarizer and GSS-based TSE,
obtains a cpWER of 20.3% on this segment. We see that most of the segments are
transcribed relatively well with small errors due to conversational speech and
noise. If we replace the oracle diarizer with spectral clustering, we only
obtain a single speaker, which is the dominant speaker in the segment. This is
the most common pattern of error in such systems, where interfering speakers
are completely missed. The resulting cpWER for this segment is 40.5%, of which
35.4% is caused by deletion errors.

On using overlap-aware spectral clustering with GSS, the second speaker is
recovered and transcribed correctly, resulting in a reduction of cpWER to
29.1%. As expected, the deletion error is reduced to 19.0%. However, it is
still difficult to recover very short utterances and backchannels from the
other speakers. We believe that a more powerful diarizer, such as TS-VAD [80],
may be better for these difficult segments. Finally, if we do not use GSS for
target-speaker extraction (and simply cut out the segments from the mixture),
the cpWER jumps to 72.2%. Most of this is caused by repeated transcription,
where the first speaker is transcribed again even in the second speaker’s
segment, possibly because the first speaker is louder. This example
demonstrates the importance of a TSE component even when we have overlap-aware
diarization.

###  5.4 Limitations

In this chapter, we described the complete pipeline for modular transcription
of multi-talker conversations. By formalizing the problem through a
probabilistic lens and making appropriate conditional independence
assumptions, we were able to identify key components of the system: speaker
diarization, target-speaker extraction, and ASR. We then leveraged the methods
proposed in the previous chapters for these components, and stringed them
together for the overall transcription process. Empirical evaluations on
synthetic and real meeting benchmarks demonstrated that this modular
perspective can be a viable solution to the problem of multi-talker ASR.

Nevertheless, our experiments also revealed important limitations about the
approach, most of which result from the approximations made in the
probabilistic formulation. Instead of optimizing the total probability
distribution, we approximated the solution by optimizing the diarization
component and using the result for TSE and ASR. Such an approximation may
result in “error propagation”, where errors made due to inaccurate modeling of
diarization result in degraded performance for the downstream components.

Even if the diarization results are perfect, the system can still produce
suboptimal solutions due to the various conditional independence assumptions
made in the TSE and ASR components. For instance, the transcript
$\mathbf{y}_{j+1}$ is very likely to depend on the dialog history, i.e.,
$\mathbf{y}_{1:j}$, but this dependence is not taken into account in the ASR
modeling. While such assumptions make the modeling more convenient (i.e., the
components can be trained independently on easily available data), they may
also make it impossible to obtain the optimal solution.

Another major limitation of this approach is its inability to perform
streaming transcription. The diarization task, i.e., computing $f(R)$, is
formulated as a clustering problem, which makes it inevitable to have to wait
until the end of the recording when all the $N$ segments are available, in
order to assign the relative speaker labels $u_{j}$. As a result, since the
TSE and ASR components are dependent on the diarization output, they cannot
produce streaming results either.

For these reasons, it is attractive to consider the alternative, “end-to-end”
perspective of multi-talker ASR, which aims to directly optimize to address
the original problem, while avoiding any approximations or modeling
assumptions. Such a framework is driven by the availability of data and
computational resources at scale, and leverages neural networks to model the
distribution. In the second part of this dissertation, we will focus our
attention on one such end-to-end framework —- the Streaming Unmixing and
Recognition Transducer (SURT) — which is based on neural transducers that we
have so far used for ASR. In particular, we will consider various challenges
in designing and training such a model for the task of multi-talker ASR, and
propose ways to jointly perform transcription and speaker attribution in the
same model.

##  Chapter 6 Streaming Unmixing and Recognition Transducers

In the last chapter, we described a modular pipeline for speaker-attributed
transcription. Despite showing promising results on real meeting benchmarks,
the pipeline approach had several limitations, most notably error propagation
and an inability to perform streaming transcription. In the second part of
this dissertation, we will approach the problem from an alternative
perspective, inspired by end-to-end models for speech recognition. In
particular, building on the success of neural transducers for ASR, we will
propose an extension known as the Streaming Unmixing and Recognition
Transducer (SURT). In this chapter, we will describe SURT in some detail,
focusing specifically on aspects such as model design, training data
simulation, loss functions, and training techniques. We will constrain the
problem to multi-talker transcription, delegating the speaker attribution
aspect to the following chapter. Instead, we will focus on problems in
transcription arising due to sparsely overlapped speech, long sequences, and
quick turn-taking.

###  6.1 Introduction

The conventional modeling approach for multi-talker ASR is through a cascade
of separation and recognition systems. This approach leverages advancements in
speech separation research to obtain single-speaker audio [193], which can
then be used with a regular ASR component [24]. However, such a model may be
sub-optimal since the components are independently optimized, and may also
require greater engineering efforts for maintenance [194].

Due to these limitations with cascaded systems, researchers have proposed
jointly optimized models that combine separation and ASR and directly solve
for the task of multi-talker transcription, often using a permutation-
invariant training (PIT) objective [14]. Such a paradigm has been explored in
the context of hybrid HMM-DNN systems [195], and more recently for end-to-end
ASR [196], with most research focusing on attention-based encoder-decoders
(AEDs). For meeting transcription, a well-studied framework is serialized
output training (SOT), wherein multiple references including those that
correspond to overlapped utterances are serialized into a single prediction
sequence, using special tokens to demarcate speaker changes [15]. A detailed
review of related work is presented in Section 6.2.

In this chapter, we describe a framework called Streaming Unmixing and
Recognition Transducer (SURT), which is based on neural transducers. This is
beneficial since neural transducers (using RNNs or transformers) have become
the standard modeling technique for on-device speech recognition [197, 178,
198] in single-speaker settings. The key step in SURT is to separate
overlapping speech into multiple simultaneous branches (or channels), each of
which is transcribed by a shared transducer. SURT was originally proposed by
[199] for the case of two-speaker single-turn conversations, and later
extended to handle long-form multi-turn recordings in [33]. The original SURT
has also been used to jointly perform speaker identification [200],
endpointing [201], and segmentation [202], although these studies have all
been restricted to the single-turn setting. Concurrent to these explorations,
a similar modeling strategy called Multi-turn RNN-T (MT-RNNT) has also been
proposed in [32]. Here, we refer to these class of models as SURT, but the
same ideas should also be applicable to MT-RNNT.

We will begin by describing the formulation of SURT in detail, focusing on its
application to continuous, streaming, multi-talker ASR. A naive implementation
of the model may suffer from several limitations. For example, performance
often degrades on multi-turn sessions due to omission and leakage related
errors. Here, omission refers to the case when an utterance is missed by all
output branches, whereas leakage happens when a non-overlapping segment is
transcribed on multiple branches. Additionally, SURT requires training on long
sessions with the transducer loss, which may be computationally prohibitive,
or even infeasible using typical academic computing resources. Furthermore, it
is not clear whether the models trained using synthetic mixtures, as proposed
in [199] and [32], would transfer well to real-world settings.

We will then discuss several aspects of SURT relating to the model design,
network architecture, training mixture simulation, loss functions, and
training schemes, which are designed to solve one or more of the above
challenges. We will conduct ablation studies to demonstrate the impact of each
of these design choices, and show that SURT is a viable framework even in the
case of real meeting transcription. This will lay the foundation towards
streaming speaker-attributed transcription in the next chapter.

###  6.2 Related Work

The problem of multi-talker speech recognition has traditionally been
addressed using a cascade of separation and transcription systems [107]. Since
these methods pose the same challenges as the modular system described in the
first part of this dissertation, there has been increasing interest in joint
modeling. Early work on joint separation and ASR involved hybrid HMM-DNN
models as the ASR backbone [195, 203]. These models were often trained with
auxiliary speaker information [204] or using transfer learning from single-
speaker acoustic models [205]. With the success and flexibility of end-to-end
ASR systems [174, 176, 206, 175, 207, 197, 198, 208], researchers quickly
adapted these into jointly optimized multi-talker ASR pipelines [196, 209].
Similar training schemes — speaker embeddings, curriculum learning, or
knowledge distillation — were used to improve these pipelines [210, 211, 212].
Nevertheless, these early models transcribed each speaker on a different
output channel, and were thus limited by the knowledge of number of speakers
in the mixture, as we will see in Section 6.3. Furthermore, they were usually
evaluated on the synthetic setting of fully-overlapping two-speaker mixtures,
which does not resemble the sparsely overlapped multi-talker speech in real
settings.

A solution for transcribing arbitrary numbers of overlapping speakers was
proposed by [15], who used existing AED architectures with “serialized output
training”. In this strategy, all the utterances in the mixture are transcribed
on the same output channel by serializing them in order of their start times.
SOT was extended to perform joint speaker counting and speaker identification
using an auxiliary speaker inventory [16, 17]. A token-level variant of SOT
(t-SOT), in conjunction with neural transducers, has shown good performance on
streaming multi-talker ASR [213, 214], and has also been combined with multi-
channel front-ends [215] and large-scale pre-training [216]. An advantage of
t-SOT is that it allows the same model and training scheme to be used for both
single and multi-talker settings. The recently concluded M2MeT challenge used
SOT as the baseline system [217, 218]. Despite its promise, SOT usually
requires large scale training on synthetic mixtures, and its streaming variant
requires complex interleaving of tokens across overlapping utterances.

With the observation that real multi-talker conversations rarely contain
overlaps of 3 or more speakers, researchers have recently proposed the task of
continuous speech separation (CSS) [11]. CSS refers to the task of generating
overlap-free speech signals from a continuous audio stream consisting of
multiple potentially overlapped utterances spoken by different people. This
task has its origins in early work on unmixing transducers [219, 220]. The
original model (which used PIT-based supervised training of BLSTM encoders)
has been improved by leveraging better architectures such as Conformer [221],
by two-stage training [194], and by large-scale semi-supervised and self-
supervised learning [222, 223]. While the original CSS used PIT-based
training, other training methods such as recurrent selective attention network
(RSAN) [224, 225] and Graph-PIT [2, 226] have also been investigated. The CSS
strategy mitigates the problem of transcribing arbitrary number of speakers by
fixing the number of output channels. Furthermore, unlike t-SOT, it does not
require any serialization/deserialization of tokens. However, performing
multi-talker ASR using CSS as a front-end still presents the same issues as
cascaded systems, mainly arising from error propagation. In this chapter, we
will describe the SURT model, which may be regarded as a jointly optimized
version of CSS with transducer-based ASR.

###  6.3 Multi-talker ASR with transducers

In multi-talker ASR, the input $\mathbf{X}\in\mathbb{R}^{T\times F}$ is an
unsegmented mixture containing $N$ utterances from $K$ speakers, i.e.,
$\mathbf{X}=\sum_{n=1}^{N}\mathbf{x}_{n}$, where $\mathbf{x}_{n}$ is the
$n$-th utterance ordered by start time, shifted and zero-padded to the length
of $\mathbf{X}$. The desired output is $\mathbf{Y}=\\{\mathbf{y}_{n}:1\leq
n\leq N\\}$, where $\mathbf{y}_{n}$ is the reference transcript corresponding
to $\mathbf{x}_{n}$. Suppose
$\mathbf{y}_{n}=(y_{1}^{n},\ldots,y_{U_{n}}^{n})$, where $U_{n}$ is the length
of the label sequence and $y$’s are output units, e.g. words or sub-word
units, from some vocabulary $\mathcal{Y}$.

Recall, from Chapter 5, that transducers compute $P(\mathbf{y}|\mathbf{X})$ by
marginalizing over all possible alignments
$\mathbf{a}\in\mathcal{B}^{-1}(\mathbf{y})$, where
$\mathbf{a}=(a_{1},\ldots,a_{T})$, $y_{t}\in\mathcal{Y}\cup\\{\phi\\}$, and
$\mathcal{B}$ is a mapping that removes the blank label $\phi$, i.e.,

$P(\mathbf{y}\mid\mathbf{X})=\sum_{\mathbf{a}\in\mathcal{B}^{-1}(\mathbf{y})}P(\mathbf{a}\mid\mathbf{X})=\sum_{\mathbf{a}\in\mathcal{B}^{-1}(\mathbf{y})}\prod_{t=1}^{T}P(a_{t}\mid\mathbf{X},\mathbf{a}_{1:t-1}).$
(6.1)

The problem we are interested in solving is known as continuous, streaming,
multi-talker ASR. As the name suggests, there are two constraints that our
solution must satisfy. First, the output should be continuous, i.e., no
external segmentation should be required. Second, the transcription must be
streaming: (i) we are not allowed to use right context from the input, and
(ii) overlapping utterances must be transcribed simultaneously. The continuous
requirement is satisfied through end-to-end modeling, i.e., we do not rely on
any diarization/segmentation systems to generate the segments
$\mathbf{x}_{n}$. For the streaming requirement, we first constrain the
encoders to only model the left context, which results in an approximation of
$P(\mathbf{y}\mid\mathbf{X})$ as

$P(\mathbf{y}\mid\mathbf{X})\approx\sum_{\mathbf{a}\in\mathcal{B}^{-1}(\mathbf{y})}\prod_{t=1}^{T}P(a_{t}\mid\mathbf{X}_{1:t-1},\mathbf{a}_{1:t-1}).$
(6.2)

More importantly, transcribing overlapping utterances at the same time
necessitates a multi-branch architecture. Assuming conditional independence of
the utterances given the input, we can approximate
$P(\mathbf{Y}\mid\mathbf{X})$ as

$P(\mathbf{Y}\mid\mathbf{X})=P(\mathbf{y}_{1},\ldots,\mathbf{y}_{N}\mid\mathbf{X})\approx\prod_{n=1}^{N}P(\mathbf{y}_{n}\mid\mathbf{X}).$
(6.3)

Alternatively, if we know the assignment of utterances to speakers, we can
relax the above assumption slightly, and only assume that the transcript of
one speaker is conditionally independent of other speakers. Formally, this is
given as

$P(\mathbf{Y}\mid\mathbf{X})=P(\tilde{\mathbf{y}}_{1},\ldots,\tilde{\mathbf{y}}_{K}\mid\mathbf{X})\approx\prod_{k=1}^{K}P(\tilde{\mathbf{y}}_{k}\mid\mathbf{X}),$
(6.4)

where
$\tilde{\mathbf{y}}_{k}=\mathbf{y}_{1}^{k}\lozenge\ldots\lozenge\mathbf{y}_{N_{k}}^{k}$,
$\lozenge$ denotes concatenation, and $\mathbf{y}_{n}^{k}$ are consecutive
utterances of speaker $k$. This results in multi-output models such as MIMO-
Speech [227], where each output branch transcribes one speaker. Such a
formulation is limited by the requirement of knowing the number of speakers
$K$ in advance. The number of output branches also increases with $K$,
resulting in an increase in model parameters. Furthermore, if permutation-
invariant training is used to resolve the problem of output permutation, this
requires computing the loss for $K^{2}$ pairs, which is computationally
expensive. Instead, we want to constrain the number of output channels to a
small number by making certain assumptions about the nature of overlapping
speech in the mixture. For this, let us formalize the problem in terms of
graph coloring, as proposed by [2].

####  6.3.1 Channel assignment as graph coloring

Consider a graph $\mathcal{G}=(V,E)$ which contains $N$ nodes, each
representing one of the utterances in the mixture. For every pair of
utterances, we connect the corresponding nodes with an edge if the utterances
are overlapping. An example of such a graph is shown in Fig. 6.1 (top).

Figure 6.1: Assigning 5 utterances to 2 output channels. The black dashed
boxes represent different utterance groups. Figure based on [2].

Each connected component in $\mathcal{G}$ is called an utterance group. In
Fig. 6.1, the black dashed boxes represent two utterance groups in the
mixture. We can also define an utterance group in non-graphical terms as
follows.

######  Definition 6.3.1.

An utterance group is a set of utterances connected by speaker overlaps. If
$\mathbf{U}=\\{\mathbf{y}_{1},\ldots,\mathbf{y}_{N}\\}$ is an utterance group
containing $N$ utterances ordered by start time, then we must have, $\forall
n\in[1,N]$,
$t_{n}^{\mathrm{st}}\leq\max_{n^{{}^{\prime}}\in[1,n-1]}t_{n^{{}^{\prime}}}^{\mathrm{en}}$.

As mentioned previously, we want to assign utterances to a fixed number of
channels, $C$, such that no utterances within the same channel are
overlapping. Formally, we can define the channel assignment problem as
follows.

######  Definition 6.3.2 (Channel assignment).

Given a set of $N$ utterances represented by the graph $\mathcal{G}$, and a
fixed number of channels (or colors) $C$, the objective of channel assignment
is to find a $C$-vertex coloring of $\mathcal{G}$, i.e.,

$\displaystyle\zeta:N\rightarrow\\{1,\ldots,C\\},~{}~{}\text{such that}$ (6.5)
$\displaystyle\zeta(u)\neq\zeta(v),~{}~{}\forall u,v\in E.$ (6.6)

The minimum possible $C$ such that such an assignment exists is called the
chromatic number of the graph. For our case, this is equal to the maximum
number of simultaneously active speakers in the mixture.

Recall from Section 1.4 that most of the overlaps in meetings are between at
most 2 speakers, and 3 or more speaker overlaps are very rare. Therefore, for
the rest of our discussion, we will set $C$ as 2, i.e., we will try to map the
$N$ utterances to 2 channels. Suppose each such $\zeta$ creates channel-wise
references $\mathbf{Y}_{1}$ and $\mathbf{Y}_{2}$. Then, for permutation-
invariant training (PIT), we have

$\displaystyle P(\mathbf{y}_{1},\ldots,\mathbf{y}_{N}\mid\mathbf{X})$
$\displaystyle=\max_{\zeta}P(\mathbf{Y}_{1},\mathbf{Y}_{2}\mid\mathbf{X})$
(6.7)
$\displaystyle\approx\max_{\zeta}P(\mathbf{Y}_{1}\mid\mathbf{X})P(\mathbf{Y}_{2}\mid\mathbf{X}),$
(6.8)

where we have again assumed conditional independence of the outputs. Such a
channel assignment of references solves the problem of increase in model size
with number of speakers or utterances. Additionally, the conditional
independence assumptions are less strong, since we only assume the channel
outputs to be independent. The corresponding PIT loss is given as

$\mathcal{L}_{\mathrm{pit}}(\mathbf{y}_{1:N},\mathbf{X};\Theta)=\min_{\zeta}\left[-\log
P_{\Theta}(\mathbf{Y}_{1}\mid\mathbf{X})-\log
P_{\Theta}(\mathbf{Y}_{2}\mid\mathbf{X})\right].$ (6.9)

####  6.3.2 Heuristic error assignment training (HEAT)

Figure 6.2: Example of heuristic error assignment training (HEAT). Each color
represent a different speaker. The utterances
($\mathbf{x}_{u}$,$\mathbf{y}_{u}$) are assigned, in order of start times, to
the next available channel. Such an assignment avoids the exponential
complexity associated with permutation invariant training (PIT).

If there are $\tilde{N}$ utterance groups (or connected components) in the
graph, it is easy to see that we can have $2^{\tilde{N}}$ possible mappings
$\zeta$. This can make it computationally prohibitive to evaluate (6.9) when
$\tilde{N}$ is large. To remedy this problem, we fix the order of the 2
channels and create channel-wise references $\mathbf{Y}_{1}$ and
$\mathbf{Y}_{2}$ by assigning $\mathbf{y}_{n}$’s to the first available
channel, in order of start time, as shown in Fig. 6.2. This technique is known
as heuristic error assignment training or HEAT, and was first introduced in
[199]. Formally, if $t_{n}^{\mathrm{st}}$ is monotonically increasing in $n$,
we have

$\zeta_{\mathrm{heat}}(n)=\begin{cases}1,&\text{if}~{}~{}t_{n}^{\mathrm{st}}\geq\max_{i\in\zeta^{-1}(1)}\theta_{i}^{\mathrm{en}}\\\
2,&\text{otherwise},\end{cases}$ (6.10)

and $\zeta_{\mathrm{heat}}(n)$’s are assigned sequentially. Such an assignment
allows us to avoid permutation-invariant training, reducing the computation
significantly. The corresponding HEAT loss is given as

$\mathcal{L}_{\mathrm{heat}}(\mathbf{y}_{1:N},\mathbf{X};\Theta)=-\log
P_{\Theta}(\mathbf{Y}_{1}\mid\mathbf{X})-\log
P_{\Theta}(\mathbf{Y}_{2}\mid\mathbf{X}).$ (6.11)

####  6.3.3 HEAT vs. PIT

Since HEAT computes training loss for a fixed assignment of references to
channels instead of minimizing over all permutations, one could argue that it
may result in a suboptimal solution. To investigate this phenomenon further,
we set up simple experiments on 2-utterance mixtures generated from
LibriSpeech train-clean set. We prepared two kinds of mixtures with utterance
delays of 2.0 and 0.0 seconds, respectively, and trained a vanilla SURT model
(which will be described in detail in the next section) using both HEAT and
PIT losses. In Fig. 6.3, we show the training dynamics for both the
experiments and also plot the % correct output assignment. This quantity
represents how often the model assigns $\mathbf{Y}_{1}$ to output channel 1.

As expected, for the case of mixtures with 2.0s delay, HEAT quickly learned
the output assignment order. In fact, even when training with PIT, the same
heuristic was learned (albeit slower), and both models started to converge
only after this point was reached (denoted by the vertical line in Fig.
6.3(a). Thereafter, using PIT is wasteful, especially in our case of the
expensive RNN-T loss computation. In the absence of utterance delay (Fig.
6.3(b)), PIT produced a random output assignment. Surprisingly, HEAT still
learned the correct assignment, but on decoding with the trained model, we
found that it learned a degenerate solution where both output channels produce
the exact same hypothesis.

(a) Delay = 2.0 s

(b) Delay = 0.0 s

Figure 6.3: Training dynamics for HEAT versus PIT based loss for different
utterance delays: (a) 2.0 s, and (b) 0.0 s.

###  6.4 The SURT model

Figure 6.4: Block diagram of SURT. $\mathbf{X}$ denotes the input mixture.
$\mathbf{X}_{1}$ and $\mathbf{Y}_{1}$ are concatenated sources and references
for the first branch (only required during training). The “recognition”
component is pre-trained on single-speaker data.

SURT estimates
$\hat{\mathbf{Y}}=[\hat{\mathbf{Y}}_{1},\hat{\mathbf{Y}}_{2}]=f_{\text{surt}}(\mathbf{X})$
by combining an “unmixing” component and a “recognition” component, hence the
name. The overall block diagram of the model is shown in Fig. 6.4. The
unmixing part first produces non-overlapping streams $\mathbf{H}_{1}$ and
$\mathbf{H}_{2}$ from the input mixture $\mathbf{X}$. The original formulation
of SURT by [199] used dual mix/mask encoders that projected input 257-dim
STFTs into high dimensional representations as

$\displaystyle\mathbf{H}_{1}=\mathbf{M}\ast\bar{\mathbf{X}},\quad\mathbf{H}_{2}=(\mathbbm{1}-\mathbf{M})\ast\bar{\mathbf{X}},$
(6.12)
$\displaystyle\mathbf{M}=\sigma(\mathrm{MaskEnc}(\mathbf{X}))~{}\text{and}~{}\bar{\mathbf{X}}=\mathrm{MixEnc}(\mathbf{X}),$

where $\bar{\mathbf{X}},\mathbf{M},\mathbbm{1}\in\mathbb{R}^{T\times D}$ (for
latent dim. $D$) is a matrix of ones, $\sigma$ is the sigmoid function, and
$\ast$ is Hadamard product. Clearly, this design constrains SURT to have
exactly two output branches, and the separated features are not interpretable.
Instead, we use 80-dimensional log Mel filter-banks as inputs, and replace the
mix/mask encoders with a simple masking network that can generate arbitrary
number of masks. Formally, given output channel count $C$, our unmixing module
generates masks

$[\mathbf{M}_{1},\ldots,\mathbf{M}_{C}]^{T}=\mathrm{MaskNet}(\mathbf{X}),$
(6.13)

where $\mathbf{M}_{c}\in\mathbb{R}^{T\times F}$. These masks are applied to
the input $\mathbf{X}$ to obtain channel-specific features:
$\mathbf{H}_{c}=\mathbf{M}_{c}\ast\mathbf{X}$. Such a design has three
advantages: (i) it allows the use of arbitrary number of output branches $C$,
(ii) the masked representations $\mathbf{H}_{c}$ are interpretable as clean
features, and (iii) it allows pre-training of the recognition module on
single-speaker speech.

The recognition module is similar to a conventional single-speaker ASR system
based on transducers. Channel-wise features $\mathbf{H}_{c}$ are fed into an
encoder which generates hidden representations $\mathbf{f}_{1:T}^{c}$. The
corresponding label sequence $\mathbf{Y}_{c}$ (created according to the HEAT
strategy) is fed into a prediction network, generating hidden representations
$\mathbf{g}_{1:U}^{c}$. A joiner combines $\mathbf{f}_{1:T}^{c}$ and
$\mathbf{g}_{1:U}^{c}$ to generate logits $\mathbf{Z}_{c}$ for each branch.
The parameters of the encoder, prediction network, and joiner are shared among
all the output branches, as shown in the figure. Finally,

$\mathcal{L}_{\text{heat}}=\mathcal{L}_{\text{rnnt}}(\mathbf{Y}_{1},\mathbf{Z}_{1})+\mathcal{L}_{\text{rnnt}}(\mathbf{Y}_{2},\mathbf{Z}_{2}),$
(6.14)

where $\mathcal{L}_{\text{rnnt}}$ is the standard RNN-T loss [176]. This is
the same formulation as (6.11), since
$\mathcal{L}_{\text{rnnt}}(\mathbf{Y},\mathbf{Z})=-\log
P_{\Theta}(\mathbf{Y}\mid\mathbf{X})$.

Multi-talker ASR with SURT requires the model to perform well on three
challenging sub-tasks:

  1. 1.

continuous separation of sparsely overlapped speech;

  2. 2.

long-form speech recognition; and

  3. 3.

modeling quick turn-taking among multiple speakers.

For most multi-talker ASR models, failure cases may be linked to model
degeneration on one or more of these sub-tasks. For instance, omission and
leakage errors may be attributed to (1), while high error rates on quick turn-
taking scenarios with short silences (see Section 6.6) may be caused by (3).
Deletion errors on long sequences may be caused by both (1) and (2). In the
following subsections, we will describe several SURT details related to the
network architecture, training objective, mixture simulation, and training
scheme, which are carefully designed to overcome the three challenges.

####  6.4.1 Network architecture

#####  6.4.1.1 Unmixing module

Figure 6.5: Overview of the streaming dual path LSTM model. The intra-chunk
LSTM is bidirectional, whereas the inter-chunk LSTM is unidirectional.

For the masking network, we use dual-path LSTMs (DP-LSTMs) since they provide
strong long range modeling capability for unmixing. First proposed in [228],
DP-LSTM consists of an intra and an inter LSTM per layer, as shown in Fig.
6.5. Input sequences are segmented into (overlapping) chunks and first fed
into the bidirectional intra-LSTM, which processes each chunk independently.
The output is then passed into the inter-LSTM which is unidirectional and
performs strided processing over chunks. By choosing the chunk width to be
approximately square root of the sequence length $l$, we can ensure that both
the LSTMs get similar length inputs. Since the intra-LSTM is bidirectional, a
latency equal to the chunk width is introduced in this model.

#####  6.4.1.2 Chunk width randomization

Dual-path models trained with a fixed chunk width may not be suitable for
evaluation on diverse sequence lengths due to mismatch in train-test input
size for the inter block. We propose training with chunk width randomization
(CWR), wherein we vary the CW between a minimum and maximum value for each
mini-batch. CWR increases the train time diversity in sequence length for both
the intra and inter blocks and makes the model robust to such variations at
test time.

#####  6.4.1.3 Recognition module

As mentioned earlier, the recognition module is a conventional neural
transducer, consisting of an encoder, a prediction network, and a joiner. For
the encoder, we use a recently proposed variant of the Conformer [168], known
as the Zipformer [229]. As shown in Fig. 6.6, the zipformer encoder contains
multiple encoder blocks running at different rates, with the middle ones more
strongly down-sampled (by up to a factor of 8). This makes training more
efficient as there are fewer frames to evaluate. Each block may contain one or
more encoder “layers” operating at the same frame rate. Each layer performs
self-attention twice with shared attention weights, and a trainable bypass is
introduced for each layer dimension.

Figure 6.6: Illustration of the zipformer encoder architecture. The encoder
contains multiple “blocks” running at different frame rates (left). Each block
contains several “layers” (middle), and each layer performs self-attention
twice with shared attention weights (right).

We also propose “branch tying” of the encoder outputs to jointly learn
representation across all branches, i.e.,

$[\hat{\mathbf{h}}_{1}^{\text{enc}},\ldots,\hat{\mathbf{h}}_{C}^{\text{enc}}]^{T}=\mathrm{LSTM}\left([\mathbf{h}_{1}^{\text{enc}},\ldots,\mathbf{h}_{C}^{\text{enc}}]^{T}\right).$
(6.15)

The motivation for branch-tied encoders is to reduce errors from omission and
leakage, which usually happen when output branches do not communicate.

Finally, we use a “stateless” prediction network [230] which uses a 1-D
convolutional layer with a kernel of size 2, thus restricting the context to
the last 2 units. In addition to improving computational efficiency, we
conjecture that a stateless network should also be better suited to handle
quick turn-taking (sub-task (3) described earlier). We set the segment length
for the bi-directional intra-LSTM (of the DP-LSTM network) equal to the chunk
size of the causal Zipformer encoder; this is the overall latency of the SURT
model. Further details about model hyperparameters are given in Section 6.5.2.

####  6.4.2 Training objective

The full-sum transducer loss, which marginalizes $P(\mathbf{y}\mid\mathbf{X})$
over all possible alignments $\mathbf{a}\in\mathcal{B}^{-1}(\mathbf{y})$,
suffers from high memory usage, since the marginalization is done over a logit
tensor of size ($B,T,U,D$). Even with an efficient implementation such as the
one described in [198], this severely limits the training sequence length —
for example, [33] trained SURT with mixtures containing at most 4 speaker
turns. To remedy this issue, we replace the full-sum transducer loss with the
recently proposed pruned transducer loss, which prunes the alignment lattice
using a simple linear joiner before computing the full sum on the pruned
lattice [170]. This idea of pruning the lattice is similar to [188], but it
does not require external forced alignments.

Recall from Section 6.4 that the encoder and the prediction network generate
hidden representations $\mathbf{f}_{1}^{T}$ and $\mathbf{g}_{1}^{U}$,
respectively. In a pruned transducer, these are first projected to
$\mathbb{R}^{|\bar{\mathcal{Y}}|}$, where
$\bar{\mathcal{Y}}=\mathcal{Y}\cup\\{\phi\\}$. Let us denote these projected
representations as $\hat{\mathbf{f}}_{1}^{T}$ and $\hat{\mathbf{g}}_{1}^{U}$,
respectively. A simple additive joiner is then used to compute trivial logits
$\hat{\mathbf{z}}_{t,u}$ as

$\displaystyle\hat{\mathbf{z}}_{t,u}=\hat{\mathbf{f}}_{t}+\hat{\mathbf{g}}_{u}-\hat{\mathbf{z}}^{\mathrm{norm}}_{t,u},$
(6.16) where
$\displaystyle\hat{\mathbf{z}}^{\mathrm{norm}}_{t,u}=\log\sum_{v}\exp\left(\hat{\mathbf{f}}_{t}+\hat{\mathbf{g}}_{u}\right).$
(6.17)

Here, (6.17) may be interpreted as log-space matrix multiplication, and is
easy to implement through simple matmul operations. Gradients from this simple
joiner are used to compute locally optimal pruning bounds for the lattice, and
the full joiner output, $\mathbf{z}_{t,u}$, is only computed on the pruned
lattice. The pruning is “locally” optimal in the sense that the optimization
is performed per-frame, as opposed to a “globally” optimal treatment which
would consider the whole path through the lattice. This local strategy may
result in path discontinuity that is later resolved through adjustments. We
used the open-source implementation available in k2:
https://github.com/k2-fsa/k2.

We use auxiliary loss functions to regularize SURT training and improve
separation of sparsely overlapped speech. For the former, we add a
connectionist temporal classification (CTC) loss [174], which has been shown
to provide regularization capabilities due to monotonicity in alignments [231,
232]. This is given as

$\mathcal{L}_{\text{ctc}}=\log\sum_{\mathbf{a}\in\mathcal{B}_{\text{ctc}}^{-1}(\mathbf{y})}\prod_{t}P(\mathbf{a}_{t}\mid\mathbf{f}_{1}^{T}),$
(6.18)

where $\mathbf{a}$ is a $T$-length sequence that deterministically maps to
$\mathbf{y}$ through transformation $\mathcal{B}_{\text{ctc}}$, which removes
repeated tokens and $\phi$. To improve mask estimation, we apply a masking
loss111Note that including masking loss in model training assumes the
availability of the clean signals $\mathbf{X}_{c}$ and hence applicable only
to training on synthetic mixtures. directly on the outputs $\mathbf{H}_{c}$
generated by the mask encoder as

$\mathcal{L}_{\text{mask}}=\sum_{c\in
C}\mathrm{MSE}(\mathbf{H}_{c},\mathbf{X}_{c}),$ (6.19)

where $\mathrm{MSE}$ denotes mean-squared error, and $\mathbf{X}_{c}$ is
obtained by summing clean inputs $\mathbf{x}_{u}$ assigned to branch $c$ (Fig.
6.2).

The overall training objective is given as

$\mathcal{L}=\mathcal{L}^{\prime}_{\text{rnnt}}+\lambda_{\text{ctc}}\mathcal{L}_{\text{ctc}}+\lambda_{\text{mask}}\mathcal{L}_{\text{mask}},$
(6.20)

where $\mathcal{L}^{\prime}_{\text{rnnt}}$ denotes the pruned transducer loss
and $\lambda$’s are hyperparameters.

####  6.4.3 Mixture simulation

1

Input: $\mathcal{X}$, $\mathcal{M}$, $K$, $T$

Output: $\mathcal{S}$

2

3$D_{=\text{spk}},D_{\neq\text{spk}},D_{\text{ovl}},\mathcal{S}$ = $\phi$

// Fit distributions to $\mathcal{M}$

4 for  _$M$ in $\mathcal{M}$_ do

5 for  _$i$ in range($|M|$)_ do

6 $t$ = $M_{i}.start-M_{i-1}.end$

7 if  _$M_{i}$.spk == $M_{i-1}$.spk_ then

8 $D_{=\text{spk}}$ = $D_{=\text{spk}}\cup\\{t\\}$

9 else

10 if  _$t >0$_ then

11 $D_{\neq\text{spk}}$ = $D_{\neq\text{spk}}\cup\\{t\\}$

12 else

13 $D_{\text{ovl}}$ = $D_{\text{ovl}}\cup\\{-t\\}$

14

15

16

17

18$P_{\text{ovl}}$ =
$\frac{|D_{\text{ovl}}|}{|D_{\neq\text{spk}}|+|D_{\text{ovl}}|}$; $D_{\ast}$ =
histogram($D_{\ast}$)

// Generate mixtures using $\mathcal{X}$

$\mathcal{X}$ = $\\{\mathcal{X}_{1},\ldots,\mathcal{X}_{S}\\}$

// speaker wise bucketing of $\mathcal{X}$

19

20while  _any( $|\mathcal{X}_{s}|>0$)_ do

// Select speakers

21 $k\leftarrow\text{sample}(K)$;
$\mathcal{X}_{s_{1}},\ldots,\mathcal{X}_{s_{k}}\leftarrow\text{sample}(\mathcal{X})$

// Select utterances for each speaker

22 for  _$i$ in range($k$)_ do

23
$U_{s_{k}}\leftarrow\text{sample}(\mathcal{X}_{s_{k}}),\text{s.t.}\left(\sum_{u\in
U_{s_{k}}}u.dur\right)<T$

24 $\mathcal{X}_{s_{k}}\leftarrow\mathcal{X}_{s_{k}}\setminus U_{s_{k}}$

25

26 $U=\text{shuffle}(U_{s_{1}},\ldots,U_{s_{k}})$; offset = 0

// Get offsets for each utterance

$\mathcal{S}_{\mathrm{cur}}\leftarrow\phi$

// initialize empty mixture

27

28 for  _$i$ in range($|U|$)_ do

29 if  _$U_{i}$.spk == $U_{i-1}$.spk_ then

30 ot = sample($D_{=\text{spk}}$)

31 else

32 if  _Bernoulli( $P_{\text{ovl}}$>0.5)_ then

33 ot = –sample($D_{\text{ovl}}$)

34 else

35 ot = sample($D_{\neq\text{spk}}$)

36

37 offset = offset + ot

38
$\mathcal{S}_{\mathrm{cur}}=\mathcal{S}_{\mathrm{cur}}\cup\\{U_{i},\text{offset}\\}$

39 $\mathcal{S}=\mathcal{S}\cup\mathcal{S}_{\mathrm{cur}}$

40

Algorithm 4 Training mixture simulation

Multi-talker ASR models are often trained on synthetic mixtures of full
utterances, which may result in prohibitively long sequences [33]; for e.g.,
LibriSpeech train has an average duration of 12.4s. Instead, we use sub-
segments instead of full utterances as the source for mixture simulation, so
that resulting mixtures are shorter while retaining multiple turns of
conversation. These sub-segments are obtained using word-level alignment
information, by breaking up the utterances at pauses longer than a threshold
$\tau$. As an example, using $\tau=0.2$ for LibriSpeech resulted in sub-
segments that were 2.8s on average. This allowed each training session to
contain up to 9 turns of conversation while still being 36.5% shorter than
those generated using full utterances.

For the simulation process itself, we learn histograms of pause/overlap
distribution statistics from the target sessions, and sample from these
distributions for mixing the segments. Such a strategy has been successfully
applied to improve end-to-end neural diarization [233]. Our mixture simulation
algorithm is described in Algorithm 4, and is similar to the conversation
simulation algorithm from [233]. We assume that the input to the algorithm is
the source segments $\mathcal{X}$, target sessions $\mathcal{M}$ (to learn
statistics), maximum number of speakers $K$ in each mixture, and maximum
duration $T$ of a speaker in a mixture. The algorithm returns the training
mixtures $\mathcal{S}$. For training, the input speech mixture $\mathbf{X}$ is
obtained by digitally adding the utterances in $\mathcal{S}$ with the
specified offsets. The utterance-wise labels $\mathbf{y}_{n}$, along with
$t_{n}^{\mathrm{st}}$ and $t_{n}^{\mathrm{en}}$ (uniquely determined by the
offset and duration of utterance $n$) are used to obtain the reference
transcripts for both channels using (6.10). We optionally convolve
$\mathcal{S}$ with real room impulse responses (RIRs) to train models for far-
field reverberant conditions. The overall simulation workflow is shown in Fig.
6.7.

Figure 6.7: Mixture simulation workflow for LibriSpeech-based training. Gray
cylinders denote external data used during simulation: LibriSpeech utterances,
statistics from LibriCSS, real RIRs, and MUSAN noises. The simulation workflow
can be conceptually divided into three phases: (i) mixing, which creates
anechoic meetings, (ii) reverberation, which convolves with RIRs, and (iii)
augmentation, which perturbs the reverberant mixtures with various schemes.
The dotted path is optional for anechoic training.

####  6.4.4 Pre-training & adaptation

Using sub-segments for mixture simulation improves training efficiency while
allowing multiple speaker turns; however, it creates a train-test mismatch for
duration of individual segments, which could degrade the model’s performance
on sub-task (2), i.e., to recognize long utterances from the same speaker. We
solve this problem by pre-training the transducer module on single-speaker
utterances (e.g., on LibriSpeech train set). Such a pre-training strategy also
decouples the tasks of learning to separate from learning to transcribe, and
helps the SURT model converge faster. Recall from Section 6.4 that this pre-
training is possible in SURT (but not in earlier work such as [33]) because
our masking network generates masked filter-banks instead of high-dimensional
latent representations.

Despite convolving with real RIRs, the acoustic characteristics of the
mixtures used to train SURT may still be mismatched from real meeting
recordings. As a final step of SURT training, we perform model adaptation by
training on in-domain data for a small number of iterations. In this step, we
adjust the training objective in (6.20) to omit the
$\mathcal{L}_{\text{mask}}$ term.

###  6.5 Experimental Setup

####  6.5.1 Evaluation

We performed evaluations on three publicly-available meeting datasets in
English: LibriCSS, AMI, and ICSI. The summary statistics for these datasets
are given in Section 1.4.

From Fig. 6.2, it is clear that SURT generates speaker-agnostic transcription
with no strict correspondence between speaker and channel. This makes it
impossible to evaluate SURT with the conventional word error rate (WER) metric
used for single-speaker ASR systems. Instead, we use the optimal reference
combination WER (ORC-WER) metric proposed independently in [32] and [33], and
described in Section 1.5. ORC-WER computes the minimum word error rate (WER)
based on an optimal assignment of references to the channels, and can be
considered a lower bound on cpWER (which we used in the previous chapter). We
used the polynomial-time implementation of ORC-WER using multi-dimensional
Levenshtein distance, available in the
meeteval222https://github.com/fgnt/meeteval toolkit [34]. In the remainder of
this paper, we will use the abbreviation WER to actually mean ORC-WER, unless
explicitly mentioned. For LibriCSS, we report WERs on the officially provided
(approximately 1 min. long) “segments,” following prior work on continuous
input evaluation [11, 221, 33]. For AMI and ICSI, we use utterance-group based
evaluation similar to [216, 234]. This results in dev/test segments of
duration 7.0s/8.4s and 2.6s/2.9s for AMI and ICSI, respectively.

####  6.5.2 Implementation details

Network architecture. We experimented with two variants of SURT — base and
large. The base model contains four 256-dim DP-LSTM layers as the masking
network trained with chunk width randomization [33]. The encoder consists of 5
zipformer blocks with 2 self-attention layers per block. Each block consists
of a 192-dim attention distributed across 8 heads, and a 768-dim feed-forward
layer. Downsampling factors of (1,2,4,8,2) were used in the zipformer blocks.
The prediction network contains a single 512-dim Conv1D layer. The large model
contains 6 layers in the masking network, and (2,4,3,2,4) self-attention
layers in the 5 zipformer blocks. The chunk size for the intra-LSTM and the
Zipformer is set to 32 frames, resulting in a modeling latency of 320 ms.

Training data. For LibriCSS experiments, we first created anechoic mixtures,
LSMix-clean, using each speed-perturbed LibriSpeech train sub-segment
($\tau=0.2$; cf. § 6.4.3) once, resulting in approx. 2200h of training data.
For this data set, we set $\mathcal{M}$ as the LibriCSS dev set (excluding the
0L and OV10 sessions), $K=3$, and $T=15$ in Algorithm 4. A reverberated copy
of LSMix, named LSMix-reverb, was generated by convolving LSMix with real RIRs
collected from the REVERB [224] dataset. We also added isotropic noises from
the REVERB data, and perturbed the loudness between -20 dB and -25 dB using
the pyloudnorm tool [165]. We will hereafter refer to the combination of
LSMix-clean and LSMix-reverb as LSMix-full. Ablation experiments were
conducted on the anechoic LibriCSS by training SURT on LSMix-clean. For these
experiments, we used on-the-fly noise augmentation using noises from the MUSAN
corpus [103]. For final evaluation, we trained SURT on LSMix-full
($\sim$4400h), so that the same model can be used on both anechoic and
replayed LibriCSS. All models were trained using on-the-fly SpecAugment [235].
We found it beneficial to use high overlap mixtures in the warm-up stage of
SURT training to encourage better mask estimation [236]. For pre-training on
single-speaker data, we used LibriSpeech train set, optionally convolved with
synthetic RIRs (for the final evaluation). This pre-training was done for 10
epochs.

Since AMI and ICSI have similar characteristics, we trained a combined SURT
model for them using synthetic mixtures created from close-talk utterances,
again using sub-segments obtained from forced alignments ($\tau=0.5$). These
mixtures were obtained by setting $D_{=\text{spk}}$, $D_{\neq\text{spk}}$,
$D_{\text{ovl}}$, and $P_{\text{ovl}}$ as 0.5, 0.5, 1.0, and 0.8,
respectively, in Algorithm 4. $K$ and $T$ were set to 3 and 15s, respectively,
similar to LSMix-clean simulation. We refer to these mixtures as AIMix-clean,
their reverberant copy as AIMix-reverb, and the combination as AIMix-full. The
models were subsequently adapted by combining the real train sessions from all
microphone settings.

Hyper-parameters. The auxiliary loss scales, $\lambda_{\text{ctc}}$ and
$\lambda_{\text{mask}}$, were set to 0.2 each. $\mathcal{L}_{\text{mask}}$ was
not used during adaptation since ground-truth separated audio is not available
for real data. We trained the models with the ScaledAdam optimizer following
the standard zipformer-transducer recipes in icefall [169]. This is a variant
of Adam where each parameter’s update is scaled proportional to the norm of
that parameter. The learning rate was warmed up to 0.004 for 5000 iterations,
and decayed exponentially thereafter. All models were trained for 30 epochs
using 4 GPUs. We used either Titan RTX (with batch size 500s) or V100 (with
batch size 650s) depending on availability on our compute cluster.

Decoding. Checkpoints from the last 9 epochs were averaged for decoding. We
conducted experiments with both greedy decoding and beam search. For the
ablation experiments, we used greedy decoding for faster turn-around. For the
final evaluation, we used a “modified” version of beam search with beam size
of 4. This variant constraints the emission of at most 1 non-blank token at
each time step, which allows batched decoding. We normalized the replayed
LibriCSS recordings to -23 dB using pyloudnorm for inference.

###  6.6 Results & Discussion

####  6.6.1 Results on LibriCSS

First, we evaluated the base and large SURT models on the LibriCSS data for
both anechoic and replayed conditions, and compared it against the MT-RNNT
baseline [32]. These comparisons are shown in Table 6.1. We cannot compare
SURT with published results on t-SOT [213] since the latter is evaluated using
the asclite-based speaker-agnostic WER (SAg-WER) metric. SAg-WER requires
token-level time-stamp estimation. More importantly, it does not penalize an
utterance being split into multiple “channels,” which is penalized by ORC-WER,
making them difficult to compare. The models were trained on LSMix-full, and
adapted to LibriCSS using the dev set. For adaptation, we segmented the dev
sessions in 2 ways, by cutting at maximum pause durations of 0.1s and 0.5s,
respectively. This resulted in 381 sub-sessions of average duration 17.9s,
totalling approximately 1.88h each from anechoic and replayed conditions and
containing 18.6% overlapped speech. The model was then trained on the combined
adaptation data for 8 epochs (for base) or 15 epochs (for large). The learning
rate was warmed up to 0.0004 for 2 (or 4) epochs and decayed thereafter. A
single V100 GPU was used for adaptation.

Table 6.1: Performance of SURT on (a) “anechoic” and (b) “replayed” versions
of LibriCSS test set. The SURT 2.0 models were decoded using beam search with
a beam size of 4.

(a) Anechoic

Model Size (M) 0L 0S OV10 OV20 OV30 OV40 Avg. SURT (Base) 26.7 5.2 4.7 14.2
17.8 21.3 23.0 14.4 $\hookrightarrow$ w/ dev adaptation 26.7 5.1 4.2 13.7 18.7
20.5 20.6 13.8 SURT (Large) 37.9 4.6 3.8 14.9 17.3 19.1 23.9 13.9
$\hookrightarrow$ w/ dev adaptation 37.9 4.6 3.8 12.7 14.3 16.7 21.2 12.2

(b) Replayed

Model Size (M) 0L 0S OV10 OV20 OV30 OV40 Avg. Multi-turn RNN-T [32] 81.0 14.8
14.5 18.0 25.8 30.3 32.3 22.6 SURT (Base) 26.7 6.7 8.2 23.0 27.1 28.5 31.8
20.9 $\hookrightarrow$ w/ dev adaptation 26.7 6.8 7.2 21.4 24.5 28.6 31.2 20.0
SURT (Large) 37.9 5.9 7.8 21.2 25.7 27.8 29.9 19.7 $\hookrightarrow$ w/ dev
adaptation 37.9 6.4 6.9 17.9 19.7 25.2 25.5 16.9

As the overlap ratio in Table 6.1 increases from 0% to 40%, the WER also
increases, which is expected. On the anechoic setting, SURT-base obtains a WER
of 14.4% without any adaptation. This may be because the anechoic training
mixtures are well matched to the evaluation condition in the absence of far-
field artifacts. Unlike other multi-talker ASR models which degrade
performance on single-speaker input, SURT obtained very low WERs on the 0L and
0S settings. We attribute this primarily to pre-training on single-speaker
data, which allows the model to handle non-overlapping speech well. On using
model adaptation, the anechoic WER further improved by 0.6% absolute, with
consistent improvements across most overlap conditions. The largest
improvement was obtained for the OV40 sessions, where WER reduced from 23.0%
to 20.6%. When evaluated on replayed LibriCSS, SURT-base was better than MT-
RNNT on average, but slightly worse on overlapped conditions like OV10 and
OV20. This may be because we used a limited set of real RIRs for simulating
reverberant training mixtures, whereas SURT and MT-RNNT used on-the-fly
simulated RIRs. We experimented with using simulated RIRs, but we found that
it consistently degraded WERs on the 0S condition, similar to the observation
in [33]. Adaptation on the dev set improved performance across all settings,
with the resulting average WER reducing to 20.0%.

The large model followed similar trends as the base model, but provided
consistent improvements in WER across most conditions. For unadapted models,
larger improvement was observed on the replayed setting compared to the
anechoic setting (5.7% vs. 3.5% relative). We conjecture that the larger
masking network (6 DP-LSTM layers) may be better suited for unmixing
reverberant features, leading to improved WERs. We also found that the large
model benefited more from adaptation on in-domain data, perhaps due to higher
representation capacity. The relative WER improvement from adaptation was
12.2% and 14.2% for the anechoic and replayed conditions, respectively,
whereas for the base model, the improvements were 4.2% and 4.3%. Overall, our
SURT-large model provided relative WER improvements of 25.2% (22.6%
$\rightarrow$ 16.9%) over an MT-RNNT baseline.

####  6.6.2 Effect of network architecture

Recall from Section 6.4.1 that the SURT network architecture contains several
components which were carefully selected for various reasons. These include:
(i) DP-LSTMs in the unmixing module, (ii) branch-tied encoders, and (iii) a
stateless prediction network. We performed ablation experiments to evaluate
the effect of each choice, as shown in Table 6.2. Each of the first three rows
denote the performance when one of the components is changed (shown in red),
while the last row shows the configuration of the final SURT model. We
selected model configurations such that all models have roughly the same
number of parameters. All models were trained on LSMix-clean until
convergence, and evaluated on the anechoic LibriCSS setting with greedy
decoding. The recognition modules were pre-trained for all setups, but no
auxiliary losses or adaptation were used. We also tried replacing the
zipformer encoder with a DP-LSTM, but this model did not converge. This may be
because the input sequences (utterance groups) for the encoder in SURT are
relatively long, which may affect convergence in architectures that do not use
self-attention.

Table 6.2: Effect of architectural choices for various components, shown on
“anechoic” LibriCSS test set. The last row denotes the final network
architectures for SURT.

Masking network Branch tying Pred. network Size (M) 0L 0S OV10 OV20 OV30 OV40
Avg. Conv2D ✓ Conv1D 25.3 13.9 6.9 27.0 35.4 41.0 45.4 28.3 DP-LSTM ✗ Conv1D
24.6 6.6 5.4 21.3 26.6 33.1 41.2 22.4 DP-LSTM ✓ LSTM 28.1 7.6 6.3 17.2 26.7
26.8 34.7 19.9 DP-LSTM ✓ Conv1D 26.7 6.4 5.1 17.5 23.5 25.4 33.3 18.5

The largest performance degradation was caused by replacing the DP-LSTM based
masking network with Conv2D. This is consistent with speech separation
research where dual-path encoders usually provide large improvements due to
their ability to model long sequences [228]. Without the DP-LSTM masking, the
unmixing module was effectively futile, as evident by the high WERs on the
overlapping sessions.

For the encoder architecture, applying branch tying using equation (6.15)
provided significant improvements on high overlap conditions. For example, the
relative WER reduction with branch tying was 23.3% and 19.2% on the OV30 and
OV40 settings, respectively. The improvement was largely due to a reduction in
deletion errors (at the cost of a small increase in insertions). For example,
on the OV40 setting, the deletion error rate reduced from 20.6% to 14.8%, with
the insertion increasing from 3.5% to 5.5%. This supports our conjecture that
branch tying helps in alleviating errors caused by omissions where parts of
the transcripts “fall through the cracks”, i.e., are skipped by both branches.
We will analyze these errors further in Section 6.6.5.

Finally, we observed small but consistent improvements by replacing the LSTM-
based prediction network (used in the original SURT) with a stateless network
(i.e., using a Conv1D layer). The largest relative improvement for this change
was seen in the 0S setting, where WER improved by 19.0% (compared to 7.0%
relative WER improvement overall). As mentioned in Section 6.4.1, we
conjecture that this may be because a stateless decoder is more suited to
frequent context switching that is required for modeling quick turn-taking.

####  6.6.3 Effect of auxiliary objectives

Table 6.3: Effect of auxiliary objectives on “anechoic” LibriCSS test set. All
models used the SURT-base architecture.

$\mathcal{L}_{\mathrm{ctc}}$ $\mathcal{L}_{\mathrm{mask}}$ 0L 0S OV10 OV20
OV30 OV40 Avg. ✗ ✗ 6.4 5.1 17.5 23.5 25.4 33.3 18.5 ✓ ✗ 6.0 5.2 17.9 23.7 22.8
29.6 17.5 ✗ ✓ 5.6 4.9 16.3 21.3 24.6 29.5 17.1 ✓ ✓ 6.1 5.0 13.6 19.0 21.1 26.5
15.2

We also performed ablation experiments to study the effect of
$\mathcal{L}_{\text{ctc}}$ and $\mathcal{L}_{\text{mask}}$. Similar to Section
6.6.2, we trained SURT-base models on LSMix-clean for this investigation, and
evaluated the models on anechoic LibriCSS using greedy search. The results are
shown in Table 6.3.

First, we see that adding CTC loss on the encoder improved WER mainly for
highly overlapped sessions. We obtained relative improvements of 10.2% and
11.1% on the OV30 and OV40 sessions, respectively. For sessions with more
overlaps and turn-taking, both output branches may contain several speech
segments. We conjecture that an auxiliary CTC objective may be useful in
aligning the segments to the corresponding audio during training, resulting in
better modeling for high overlap sessions.

(a)

(b)

Figure 6.8: Occupation probabilities of nodes in the $T\times U$ lattice for
both output channels, for models trained with and without auxiliary CTC loss
($\mathcal{L}_{\text{ctc}}$. $T$ and $U$ are on $x$ and $y$ axes,
respectively. For this mixture, $T=395$, $U_{1}=79$, and $U_{2}=44$. The
double arrows between the plots denote the reference segments: red for channel
1 and green for channel 2. Brighter colors denote higher occupation
probabilities.

To validate our conjecture, we plotted the the occupation probabilities for
the nodes in the RNN-T lattice (of shape $T\times U$), as obtained from the
gradients of the simple additive joiner used in the pruned transducer loss
[170]. These values should correspond to a soft alignment between the input
and the label sequence333It makes more sense to use this value instead of CTC
alignments because (i) the model trained without $\mathcal{L}_{\text{ctc}}$
cannot provide corresponding alignments, and (ii) we use the transducer head
for the ASR task.. In Fig. 6.8, we show example plots for a randomly selected
mixture from the training set, using models trained with and without
$\mathcal{L}_{\text{ctc}}$. When the auxiliary CTC loss was used, the model
was able to better align the silence region (time frames 100 to 250) in
channel 2, as discernible through the bright horizontal line.

Using auxiliary masking loss $\mathcal{L}_{\text{mask}}$, as defined in
equation (6.19), again improved WER performance over the SURT-base model, as
shown in the third row of Table 6.3. Surprisingly, we observed most
improvements in the low-overlap conditions — for instance, 12.5% relative WER
reduction for 0L. Most of this improvement again resulted from reduction in
deletion errors (2.0% $\rightarrow$ 1.1% for 0L). We conjecture that this may
be a result of fewer leakage-related errors in low overlap regions. We also
experimented with using a graph-PIT based masking loss [2] instead of the
HEAT-based loss, but it did not provide similar improvements. This may be
because our final transducer objective uses the HEAT formulation. Finally, the
best WER results were obtained on combining both the auxiliary objectives. The
resulting model demonstrated a relative WER reduction of 17.8% over the SURT-
base model trained without these objectives.

####  6.6.4 Effect of pre-training

Figure 6.9: Effect of transducer pre-training with LibriSpeech. WERs are shown
for the LibriCSS dev set after each training epoch. The dotted horizontal
lines show the final WERs on test set with model averaging.

Pre-training on single-speaker utterances was found to be one of the most
effective strategies for faster and better convergence of SURT models. Such a
pre-training strategy has also been used in recent work on multi-channel
serialized output training [215]. To quantify this improvement, we computed
the WER on the anechoic LibriCSS dev set (averaged across all overlap
conditions) after each epoch of training a SURT-base model on the LSMix-clean
data, as shown in Fig. 6.9. We observe that when pre-training was used, the
SURT model converged much faster and to a better WER. In fact, it surpassed
the model without pre-training after just 5 epochs. On the anechoic test set,
the models obtained WER of 26.2% and 15.2%, respectively, as shown by the
dotted horizontal lines. The increase in computational time is marginal, since
we only pre-train the transducer for 10 epochs (instead of training to
convergence). Alternatively, an off-the-shelf streaming transducer may also be
plugged in for this purpose, since the branch tying of the encoder is only
added at the time of SURT training. Such a pre-training scheme is analogous to
a curriculum learning strategy using single-speaker utterances, with the masks
fixed as $\mathbf{M}_{1}=J_{T,F}$ and $\mathbf{M}_{c}=0\cdot J_{T,F}$,
$\forall c\neq 1$, where $J_{t,f}=1$, $\forall t,f$.

####  6.6.5 Measuring leakage and omission

(a)

(b)

Figure 6.10: Effect of decoding beam size on insertion and deletion errors,
versus effect on leakage and omission (for $n=4$), for SURT-base models with
and without adaptation, for (a) “anechoic” and (b) “replayed” conditions,
averaged across all overlap settings. The top row shows insertion and deletion
errors, while the bottom row contains leakage@4 and omission@4.

Throughout this paper, we have mentioned leakage and omission, first
identified in [33], as major contributors of errors in SURT. In this section,
we provide a metric for quantifying these error sources, in terms of n-gram
counts on the reference $\mathbf{Y}$ and hypotheses $\hat{\mathbf{Y}}$, as
defined in Section 6.4.

omission@n

For some $n$, the fraction of all unique n-grams in $\mathbf{Y}$ not present
in any $\hat{\mathbf{Y}}_{c}$.

leakage@n

For some $n$, the fraction of all unique n-grams in $\mathbf{Y}$ present in
multiple $\hat{\mathbf{Y}}_{c}$.

If emission time-stamps are known, these definitions may be modified to
include time windows for n-gram search, e.g., an n-gram is considered to be
present in more than one $\mathbf{Y}_{c}$ only if the corresponding time-spans
overlap. In the absence of time marks, the leakage@n and omission@n rates
defined above are upper and lower bounds on the actual leakage and omission,
respectively. Nevertheless, by quantifying these error types explicitly, we
can gain some insights into model behavior. Our SURT model in this chapter
does not predict token time-stamps, so we estimate omission and leakage over
the entire hypotheses.

In Table 6.1, we showed WERs achieved by the SURT models with and without
adaptation, when decoded using beam search with a beam of size 4. Typically,
ASR model performance improves by increasing the beam size, but we found that
for SURT models, the WER first improved (up to a beam size of 4) and then
degraded. This trend can be explained by looking at the leakage and omission
errors, as shown in Fig. 6.10. We used the SURT-base model (with and without
adaptation) for decoding the anechoic and replayed sets, using beam sizes
varying from 1 to 8. In the figure, we show insertion and deletion error
rates, as well as leakage@4 and omission@4. We observe that leakage first
decreased (from beam size 1 to 2) but then increased gradually, while the
opposite trend was observed for omission. Furthermore, adapted models reduce
omissions significantly, at the cost of increase in leakage (particularly in
the replayed setting).

In the analysis above, the overall trend for leakage and omissions followed
those of insertion and deletion errors, which is expected. However, this may
not always be the case. In Table 6.3, we showed results for ablation
experiments done to investigate the effect of auxiliary objectives. We