# Fixing MoE Over-Fitting on Low-Resource Languages in Multilingual Machine
Translation

Maha Elbayad
Meta AI
<EMAIL_ADDRESS>&Anna Sun∗
Meta AI
<EMAIL_ADDRESS>&Shruti Bhosale
Meta AI
<EMAIL_ADDRESS>Equal contribution

###### Abstract

Sparsely gated Mixture of Experts (MoE) models have been shown to be a
compute-efficient method to scale model capacity for multilingual machine
translation. However, for low-resource tasks, MoE models severely over-fit. We
show effective regularization strategies, namely dropout techniques for MoE
layers in EOM and FOM, Conditional MoE Routing and Curriculum Learning methods
that prevent over-fitting and improve the performance of MoE models on low-
resource tasks without adversely affecting high-resource tasks. On a massively
multilingual machine translation benchmark, our strategies result in about +1
chrF++ improvement in very low resource language pairs. We perform an
extensive analysis of the learned MoE routing to better understand the impact
of our regularization methods and how we can improve them.

##  1 Introduction

25k50k75k100k$1.8$$2$$2.2$UpdatesPerplexityDenseMoE6425k50k75k100k$20$$40$$60$UpdatesLow-
resource(eng-kon)High-resource(eng-fra) Figure 1: Validation perplexity of
dense and MoE (64 experts) models. We show a high-resource direction that does
not suffer from over-fitting, when a low resource direction sees extreme over-
fitting.

Training massively multitask models such as multilingual machine translation
models benefit from transfer learning across different tasks. But they also
suffer from reduced model capacity per task and potential interference between
conflicting tasks. Scaling up models has been shown to be a very effective
strategy in many natural language processing tasks such as language modeling,
massively multilingual translation and natural language understanding (Brown
et al., 2020; Kaplan et al., 2020). Most of these advancements have focused on
training increasingly larger dense models. However, dense model scaling is
computationally expensive, as a result, various sparse model architectures
have been proposed to increase model capacity without incurring additional
compute costs; the most commonly used one is the Sparsely-Gated Mixture-of-
Experts (MoE) layer (Shazeer et al., 2017; Lepikhin et al., 2020; Du et al.,
2021; Hwang et al., 2022; Zoph et al., 2022).

MoE models are a type of conditional compute models (Bengio et al., 2013;
Almahairi et al., 2016) that activate a subset of model parameters per input,
as opposed to dense models that activate all model parameters. MoE models
unlock significant representational capacity while maintaining the same
inference and training efficiencies in terms of FLOPs as compared to the core
dense architecture. As a result, past work has demonstrated improved
performance on multitask models such as multilingual machine translation when
using MoE models (Lepikhin et al., 2020; Kim et al., 2021; Fedus et al., 2022;
Zoph et al., 2022).

But we notice that, on imbalanced datasets, MoE models suffer from over-
fitting on low resource tasks i.e., tasks with relatively less training data.
Figure 1 illustrates this phenomenon on a multilingual translation benchmark.
We see that eng-fra, a high-resource translation direction, does not over-fit
with either dense or MoE models. On the other hand, eng-kon, a low-resource
translation direction, extremely over-fits with the MoE model compared to the
dense model.

In this work, we introduce four effective strategies to reduce the over-
fitting of MoE models on low-resource tasks in a massively multilingual MT
benchmark:

  1. 1.

Dropout techniques for MoE layers: we introduce Expert Output Masking (EOM)
and Final Output Masking (FOM), two dropout methods specific to MoE layers
that we apply on top of overall dropout.

  2. 2.

Conditional MoE Routing (CMR): We train an additional gate to decide when to
route a token to an MoE layer vs. a shared dense layer.

  3. 3.

Curriculum Learning (CL): We introduce low-resource pairs that are prone to
over-fitting in the later stages of model training.

On a massively multilingual MT benchmark,11153 languages and 110 translation
directions and approximately 1.7B training examples we experimentally
demonstrate the effectiveness of each of these strategies. Particularly, we
observe close to +1 chrF++ improvements with EOM, FOM, CMR and CL strategies
on very low resource language directions out of English.

##  2 Background

We first describe the multilingual machine translation (MMT) task setup, the
dense backbone architecture, and how we augment it with MoE layers.

#### Multilingual Machine Translation.

We model multilingual neural machine translation as a sequence-to-sequence
task, where we condition on an input sequence in the source language with an
encoder and generate the output sequence in the expected target language with
a decoder (Sutskever et al., 2014). We train to maximize the probability of
the translation sequence in the target language given the source sequence, in
addition to the source language $\ell_{s}$ and the target language $\ell_{t}$.

#### Model Architecture.

Our sequence-to-sequence multilingual machine translation model is based on
the Transformer encoder-decoder architecture (Vaswani et al., 2017).

To prime the model for multilingual translation, we prefix the source sequence
with the source language $\ell_{s}$ and the target sequence with the target
language $\ell_{t}$.

#### Sparsely Gated Mixture of Experts.

In both Transformer encoder and decoder, we replace every other dense FFN
sublayer with an MoE sublayer. The MoE sublayer consists of $E$ feed-forward
networks (FFN), denoted with
$(\operatorname{FFN}_{1},\operatorname{FFN}_{2},\ldots,\operatorname{FFN}_{E})$.
A gating network, consisting of a softmax-normalized linear layer with weights
$W_{g}$, is attached to each MoE sublayer to decide how to route tokens to
experts. Given an input token $x_{t}$ the output of the MoE sublayer is
evaluated as:

$\displaystyle\mathcal{G}_{t}=\text{Top-k-
Gating}(\operatorname{softmax}(W_{g}\cdot x_{t})),$ (1)
$\displaystyle\operatorname{MoE}(x_{t})=\sum\limits_{e=1}^{E}\mathcal{G}_{te}\cdot\operatorname{FFN}_{e}(x_{t}),$
(2)

with $\mathcal{G}_{t}\in\mathbb{R}^{E}$ the routing vector computed by the
gating network, i.e., for each expert, $\mathcal{G}_{t,e}$ is the contribution
of the e${}^{\text{th}}$ expert ($\operatorname{FFN}_{e}$) in the MoE output.
We follow the Top-k-Gating algorithm of Lepikhin et al. (2020) and dispatch
each token to at most $k{=}2$ experts.

The sparse MoE model learns to route input tokens to the corresponding top-2
experts by optimizing a linearly weighted combination of label-smoothed cross
entropy, $L_{\text{MT}}$, ($\epsilon{=}0.1$, Szegedy et al. (2015)) and an
auxiliary load balancing loss, $L_{\text{MoE}}$ (Shazeer et al., 2017),

$\displaystyle L=L_{\text{MT}}+\lambda_{\text{MoE}}L_{\text{MoE}}.$ (3)

This additional loss term ($L_{\text{MoE}}$) pushes the tokens to be uniformly
distributed across experts. We set $\lambda_{\text{MoE}}$ to 0.01 in all our
experiments. We refer the reader to Lepikhin et al. (2020) for more on the
optimization of MoE models.

##  3 Fixing over-fitting on low-resource tasks

$x_{1}$$x_{2}$$x_{3}$gatinggatinggating$e_{1}$$e_{2}$$e_{3}$$e_{1}$$e_{2}$$e_{3}$$e_{1}$$e_{2}$$e_{3}$

(a) Routing tokens

$e_{1}$$e_{2}$$e_{3}$InputExpertoutputFinaloutput

(b) Overall dropout

$e_{1}$$e_{2}$$e_{3}$InputExpertoutputFinaloutput

(c) EOM

$e_{1}$$e_{2}$$e_{3}$InputExpertoutputFinaloutput

(d) FOM

Figure 2: Illustration of Expert/Final Output Masking (EOM/FOM) in contrast to
overall dropout for MoE layers: a color represents a token, and each token is
dispatched to two experts. Faded colors correspond to dropped units or masked
outputs. Note that EOM and FOM are always combined with overall dropout.

The motivation behind MoE models is to allow different parameters to model
different aspects of the input space. The added expert capacity should help
higher resource language pairs that might otherwise be constrained to share
the same capacity with many other language pairs. Besides, increasing model
capacity should reduce interference, thus benefiting tasks of all resource
levels.

Although overall dropout is sufficient to regularize dense models, it is not
enough for MoE models (see Figure 4). To address the issue of over-fitting of
MoE models on low-resource tasks, we propose a series of architectural changes
that improve the performance on low-resource language pairs with MoE models in
Sections 3.1, 3.2 and 3.3. In Section 3.4, we devise and study a simple but
effective curriculum learning strategy as another approach to reduce the over-
fitting on low-resource directions.

LayerNormMoE
Gating$\operatorname{FFN}_{1}$$\ldots$$\operatorname{FFN}_{E}$LayerNormCMR
Gating$\operatorname{FFN}_{\text{shared}}$MoE
Gating$\operatorname{FFN}_{1}$$\ldots$$\operatorname{FFN}_{E}$

Figure 3: Illustration of Conditional MoE Routing (CMR) showing a residual
block in a Transformer layer with regular MoE (left) vs. CMR (right).

###  3.1 MoE Expert Output Masking (EOM).

In this proposed regularization strategy, we mask the expert output for a
random fraction ($p_{\text{eom}}$) of the input tokens. For input tokens with
dropped expert outputs, the first and/or second expert is effectively skipped,
as illustrated in Figure 2(c). Note that although this masking will zero out
some combination weights $\mathcal{G}_{t,e}$ in Equation 2, it will not affect
the weights used in the load balancing loss.

###  3.2 Final Output Masking (FOM).

A simpler alternative to EOM would be to mask the combined expert output for a
random fraction of tokens, i.e., the last stage in Figure 2(d). We denote with
$p_{\text{fom}}$ the fraction of tokens masked with this regularization
method. Note that this type of masking is more generic as it can be applied to
dense models as well.

###  3.3 Conditional MoE Routing (CMR).

Instead of randomly dropping a proportion of activations or masking expert
outputs, we consider the option of letting the model learn which tokens need
the extra capacity or specialization of MoE layers, and which tokens are
better routed to a limited-capacity shared layer. Inspired by Zhang et al.
(2021)’s CLSR-Gate, we design Conditional MoE Routing layers (CMR for short).
As depicted in Figure 3, we augment MoE layers with a binary gate that
determines the weights associated with two branches of the computational
graph: (1) a shared dense FFN sublayer ($\operatorname{FFN}_{\text{shared}}$)
and (2) an MoE layer with its own $E$ expert FFN sublayers. For an input token
$x_{t}$, the output of CMR is evaluated as follows:

$\displaystyle g(x_{t})$
$\displaystyle=\operatorname{sigmoid}(W_{\textsc{CMR}{}}\cdot x_{t}),$ (4)
$\displaystyle\operatorname{\textsc{CMR}{}}(x_{t})$
$\displaystyle=(1-g(x_{t}))\cdot\operatorname{FFN}_{\text{shared}}(x_{t})$ (5)
$\displaystyle+g(x_{t})\cdot\operatorname{MoE}(x_{t}),$ (6)

where $W_{\textsc{CMR}{}}$ are the weights of the CMR’s binary gate.
$W_{\textsc{CMR}{}}$ is trained by optimizing translation accuracy under a
budget constraint $b$. For a mini-batch with $T$ tokens, this amounts to
adding the following auxiliary loss term ($L_{\textsc{CMR}{}}$) to the loss
function in equation 3:

$\displaystyle
L_{\textsc{CMR}{}}=\frac{1}{T}\cdot\sum\limits_{t=1}^{T}\left|g(x_{t})-b\right|,$
(7) $\displaystyle
L=L_{\text{MT}}+\lambda_{\text{MoE}}L_{\text{MoE}}+\lambda_{\textsc{CMR}{}}L_{\textsc{CMR}{}}.$
(8)

We use the budget parameter $b$ to limit the effective capacity of MoE layers,
thus providing a regularizing effect; at $b{=}0$, the model is dense,
practically pushing all tokens through $\operatorname{FFN}_{\text{shared}}$,
and at $b{=}1$, the model is free to always route tokens through the high-
capacity MoE layer.

To reduce over-fitting, we experiment with zeroing out a fraction of the CMR
gates $g(x_{t})$ in the mini-batch; we denote this fraction with
$p_{\text{cmr}}{}$. This means that we force $p_{\text{cmr}}{}\%$ tokens in
the mini-batch to only take the route of $\operatorname{FFN}_{\text{shared}}$.

###  3.4 Curriculum Learning

We next explore alternative methods of regularization by means of Curriculum
Learning (CL). We propose to start training with high-resource pairs first,
then introduce low-resource pairs, prone to over-fit, later in phases. To
derive the phases of the curriculum, we first train a vanilla MoE model
(without CL), then we partition the tasks (translation directions) into $n$
bins $\\{b_{1},\ldots,b_{n}\\}$. If $U$ is the total number of training
updates, we introduce each bin $b_{i}$ after $U-k_{i}$ updates. We compare two
partitioning strategies for when and what directions to add at every phase.

  1. 1.

Count-based: we empirically partition based on training example counts.

  2. 2.

Step-based: partition based on the step where we observed a task to start
over-fitting. See Algorithm 1.

Algorithm 1 Partitioning for step-based CL

1:Input: number of bins $n$, a set of tasks $\mathcal{T}$, the maximum number
of updates $U$, the step corresponding to the best validation perplexity
$s_{\text{best}}:\mathcal{T}\to[0,U]$.

2:$\triangleright$ For $s_{\text{best}}$, we take the max if multiple

3:Output: Partitioning of $\mathcal{T}$ into $n$ bins
$\mathbf{b}{=}(b_{1},\ldots,b_{n})$, characteristic step for each bin
$\mathbf{k}{=}(k_{1},\ldots,k_{n})$.

4:$\triangleright$ The bin $b_{i}$ will be introduced at $U-k_{i}$.

5:$s_{\text{max}}=\max_{t\in\mathcal{T}}s_{\text{best}}(t)$,
$s_{\text{min}}=\min_{t\in\mathcal{T}}s_{\text{best}}(t)$

6:$\Delta=\dfrac{s_{\text{max}}-s_{\text{min}}}{n-1}$.

7:for $i\in\\{1\ldots n\\}$ do

8: $b_{i}=\varnothing,\;k_{i}=s_{\text{max}}-(i-1)\Delta$.

9:end for

10:for $t\in\mathcal{T}$ do

11: $c_{t}=\operatorname*{argmin}_{1\leq i\leq n}|s_{\text{best}}(t)-k_{i}|$

12: $b_{c_{t}}=b_{c_{t}}\cup\\{t\\}$

13: $\triangleright$ assign to the closest bin wrt. its characteristic step.

14:end for

##  4 Experimental Setup

###  4.1 MMT dataset

We construct a multilingual machine translation benchmark consisting of 53
languages and a total of 110 translation directions. Our MMT dataset consists
of 45 directions out of English (aggregated as eng-xx), 45 directions into
English (aggregated as xx-eng) and 20 non-English directions (aggregated as
xx-yy). In terms of resource level, there are 40 high-resource and 70 low-
resource directions, out of which 22 are very low-resource.222 We follow the
categorization in NLLB Team et al. (2022); a language is low-resource if there
are fewer than 1M publicly available, de-duplicated bitext samples with any
other language, very low-resource if fewer than 100K.  The training data is
composed of publicly available bitext in all 110 language directions (primary
data in NLLB Team et al. (2022)) and large-scale mined data (Heffernan et al.,
2022; NLLB Team et al., 2022) in English-centric directions. There are a total
of 2$\times$847M examples in this benchmark. For a detailed listing of the
directions, see Appendix A.

#### Segmentation with SentencePiece.

To tokenize our text sequences, we train a single SentencePiece (SPM) (Kudo
and Richardson, 2018) model for all languages.333202 languages in total,
including the ones not part of our MMT dataset. The vocabulary size of our
trained SPM model is 256,000. For more on this SPM model, see NLLB Team et al.
(2022).

##  5 Results

| eng-xx | xx-eng | xx-yy
---|---|---|---
| all |  | v.low | all |  | v.low | all
Dense 615M |  |  |  |  |  |  |
abDense | 41.7 |  | 30.4 | 51.1 |  | 44.0 | 39.4
abMoE-64 | 43.0 |  | 30.3 | 52.6 |  | 44.2 | 39.8
abDense ($p_{\text{drop}}{=}0.1$) | 41.9 |  | 31.1 | 51.8 |  | 45.3 | 39.6
abMoE-64 ($p_{\text{drop}}{=}0.1$) | 43.6 |  | 32.0 | 53.4 |  | 45.9 | 41.1
Dense 1.3B |  |  |  |  |  |  |
abDense | 43.3 |  | 31.6 | 53.5 |  | 46.5 | 41.3
abMoE-64 | 43.3 |  | 29.7 | 52.9 |  | 43.7 | 39.3
abDense ($p_{\text{drop}}{=}0.1$)$\dagger$ | 43.7 |  | 33.1 | 54.4 |  | 47.9 | 41.9
abMoE-64 ($p_{\text{drop}}{=}0.3$)$\dagger$ | 44.3 |  | 32.5 | 54.4 |  | 47.7 | 41.9

Table 1: Validation set chrF++ of vanilla MoE with and without overall
dropout. $\dagger$ indicates best of sweep.
10k40k70k100k$20$$40$$60$PerplexityDense($p_{\text{drop}}{=}0$)Dense($p_{\text{drop}}{=}0.3$)10k40k70k100k$20$$40$$60$10k40k70k100k$20$$40$$60$MoE64($p_{\text{drop}}{=}0.3$)MoE64($p_{\text{drop}}{=}0.3$
$p_{\text{eom}}{=}0.2$)10k40k70k100k$1.9$$2$$2.1$updatesPerplexity10k40k70k100k$1.9$$2$$2.1$updatesMoE64($p_{\text{drop}}{=}0$)MoE64($p_{\text{drop}}{=}0.3$)10k40k70k100k$1.9$$2$$2.1$updateseng-
koneng-fraDense + dropoutMoE + dropoutMoE + EOM Figure 4: Validation
perplexities with Various dropout strategies for a low-resource direction
(eng-kon in the top row) and a high-resource direction (eng-fra in the bottom
row).

All MoE sub-layers have $E{=}$64 experts444$E=64$ is close to the number of
languages in the benchmark, i.e., 53. All models are trained for 100k updates
with an effective batch size of 1M tokens per update. See Appendix B for
additional details. We use the chrF++ metric (Popović, 2017) to compare the
model performance. We report averages in each set of directions: eng-xx, xx-
eng and xx-yy as all. For eng-xx and xx-eng, and when relevant, we breakdown
the pairs by resource level: high-resource (high), low-resource (low) and very
low resource (v.low).

###  5.1 Vanilla (un-regularized) MoE

When looking at un-regularized models (without overall dropout), we see in
Table 1, that when the backbone dense model has 615M parameters, the MoE
model, while computationally similar, shows +1.3, +1.5 and +0.4 chrF++
improvements on eng-xx, xx-eng and xx-yy respectively. When focusing on the
very low resource pairs (v.low), the performance actually drops on eng-xx
(-0.1 chrF++) signaling an over-fitting issue. When scaling the backbone to
1.3B, we see even more over-fitting on v.low directions (-1.9chrF++ in eng-xx
and -2.8 chrF++ in xx-eng).

Adding overall dropout555sweeping over $p_{\text{drop}}{\in}\\{0.1,0.2,0.3\\}$
significantly improves the performance of MoE models in both the 615M and 1.3B
variants. Importantly, when increasing the dropout to 0.1 for the small MoE
(615M), we see that the relative decline of -0.1 chrF++, turns into an
improvement of +0.9 chrF++ for eng-xx v.low pairs. Once we scale the
computational cost per update (1.3B), tuned overall dropout does not fix the
over-fitting of very low-resource pairs.

In Figure 4, we observe in the case of eng-kon, a very low-resource pair, that
the model continues to face significant over-fitting when trained for 100k
updates. This is unsurprising, as iterating over a small training set with
large capacity causes over-fitting. Training for more steps is important for
high-resource pairs, but we want to avoid negatively affecting low-resource
pairs in the process.

###  5.2 Regularizing MoEs

| eng-xx | xx-eng | xx-yy
---|---|---|---
| all | high | low | v.low | all | high | low | v.low | all
Baseline $p_{\text{drop}}{=}0.3$ $\dagger$ | 44.3 | 56.0 | 39.5 | 32.5 | 54.4 | 63.9 | 50.6 | 47.7 | 41.9
EOM ($p_{\text{drop}}{=}0.3$, $p_{\text{eom}}{=}0.1$)$\dagger$ | 44.7 | 55.9 | 40.1 | 33.4 | 54.8 | 64.3 | 51.0 | 48.3 | 42.5
FOM ($p_{\text{drop}}{=}0.2$, $p_{\text{fom}}{=}0.3$) $\dagger$ | 44.4 | 55.7 | 39.8 | 33.1 | 55.0 | 64.3 | 51.3 | 48.8 | 42.5
Gating Dropout ($p_{\text{drop}}{=}0.3$, $p_{gd}{=}0.2$) (Liu et al., 2022) $\dagger$ | 44.4 | 55.7 | 39.8 | 33.0 | 54.8 | 64.1 | 51.0 | 48.5 | 42.3
CMR top-1 ($p_{\text{drop}}{=}0.3$, $p_{\text{cmr}}{=}0.1$, $b{=}0.6$) $\dagger$ | 44.2 | 55.8 | 39.5 | 33.2 | 54.9 | 64.3 | 51.1 | 48.7 | 42.3
CMR top-2 ($p_{\text{drop}}{=}0.3$, $p_{\text{cmr}}{=}0.2$) $\dagger$ | 46.2 | 56.2 | 41.8 | 35.7 | 55.1 | 64.7 | 51.5 | 49.2 | 42.8

Table 2: Comparison of Various Regularization Strategies applied to an MoE-64
baseline. In each column, we bold the best results out of the first six rows
(computationally comparable), and we bold results from the last row (CMR
top-2) if they outperform the other models. $\dagger$ signals that this model
is best of sweep.

For the rest of this paper, we use the 1.3B variant as our backbone, to which
we add MoE layers with $E{=}64$ experts.

#### Experimental Setup.

We consider the MoE model with an overall dropout rate of 0.3
($p_{\text{drop}}{=}0.3$), best performing after a sweep of
$p_{\text{drop}}\in\\{0.1,0.2,0.3\\}$ to be our baseline.666 Initial
experiments separating the dropout rates of shared and MoE blocks showed that
the best values align.

In each of the sweeps below, we choose the best variant based on the average
chrF++ score on the validation set.

For EOM and FOM, we sweep over the values of
$(p_{\text{drop}},p_{\text{eom/fom}})\in\\{0.1,0.2,0.3\\}^{2}$.

For CMR, and in order to keep the compute equivalent to the baseline MoE, we
use top-1 instead of the top-2 gating used in previous experiments. We fix
$p_{\text{drop}}{=}0.3$ and sweep over the CMR parameters ($p_{\text{cmr}}{}$,
$b{}$). We also train a CMR top-2 model, although not compute-equivalent to
the baseline MoE, it provides insight into performance under a large compute
budget. For CMR top-2, we fix $p_{\text{drop}}{=}0.3$ and sweep over the
values of $p_{\text{cmr}}{}\in\\{0.1,0.2,0.3\\}$. We set
$\lambda_{\text{CMR}}$ to 0.1 in all our CMR experiments.

We additionally compare our methods to Gating Dropout (Liu et al., 2022), a
method in which we route tokens with probability $p_{gd}$ to the local
experts, thus skipping the All-to-All communication between GPUs. We sweep
over the values of $(p_{\text{drop}},p_{gd})\in\\{0.1,0.2,0.3\\}^{2}$.

For each model, we report chrF++ averages on the validation set in 3 groups of
directions: eng-xx, xx-eng and xx-yy, broken down w.r.t. to resource levels:
high, low and very low (v.low) for eng-xx and xx-eng.

| eng-xx | xx-eng | xx-yy
---|---|---|---
| all |  | v.low | all |  | v.low | all
top-1 $b{=}0.4$ | 44.2 |  | 32.8 | 54.4 |  | 48.0 | 42.1
top-1\+ $p_{\text{cmr}}{=}0.1$ | 43.9 |  | 33.0 | 54.9 |  | 48.6 | 42.3
top-1, $b{=}0.8$ | 44.5 |  | 32.9 | 54.3 |  | 47.4 | 42.2
top-1\+ $p_{\text{cmr}}{=}0.1$ | 44.6 |  | 33.5 | 54.3 |  | 47.7 | 42.2
top-1 $p_{\text{cmr}}{=}0.1$ |  |  |  |  |  |  |
top-1 \+ $b{=}0.2$ | 43.8 |  | 32.7 | 54.5 |  | 48.5 | 42.2
top-1 \+ $b{=}0.4$ | 43.9 |  | 33.0 | 54.9 |  | 48.6 | 42.3
top-1 \+ $b{=}0.6$ | 44.2 |  | 33.2 | 54.9 |  | 48.7 | 42.3
top-1 \+ $b{=}0.8$ | 44.6 |  | 33.5 | 54.3 |  | 47.7 | 42.2
top-2 $b{=}0.8$ | 44.6 |  | 33.1 | 54.3 |  | 47.2 | 41.9
top-1\+ $p_{\text{cmr}}{=}0.2$ | 46.2 |  | 35.7 | 55.1 |  | 49.2 | 42.8

Table 3:  Sweep over hyperparameters for MoE-64 CMR: The budget $b$, the CMR
gate dropout $p_{\text{cmr}}$. We bold the best results in each column.

#### Results.

In terms of alleviating the over-fitting issue, the last column of Figure 4
shows that EOM leads to better regularization and less over-fitting on low-
resource tasks compared to overall dropout. In terms of translation quality,
we observe in Table 2 gains of +0.4 chrF++ across all pairs into English and
+0.6 chrF++ across non-English pairs for MoE EOM compared to the MoE baseline.
For out of English, the largest gains are observed on low and very low-
resource languages; +0.6 and 0.9 chrF++ respectively.

With FOM, we see in Table 2 gains over the baseline MoE of +0.1 chrF++ across
eng-xx pairs, +0.6 chrF++ across xx-eng pairs and +0.6 chrF++ across xx-yy
pairs. For into English, the largest gains are observed on low and very low-
resource languages; +0.7 and 1.1 chrF++. Compared to the best EOM model, FOM
under-performs slightly on eng-xx (-0.3 chrF++) but outperforms on xx-eng
(+0.2 chrF++); when averaging over all pairs, the two models achieve the same
chrF++ score of 48.4.

We look in Table 3 at the impact of the budget $b$ and the dropout
$p_{\text{cmr}}$. We observe that $p_{\text{cmr}}$ is a necessary ingredient
in CMR top-2; in the last two rows of Table 3, adding $p_{\text{cmr}}$
improves the performance across the board, particularly in en-xx and xx-en
very low directions (+2.6 and +2.0 chrF++, respectively). With top-1,
$p_{\text{cmr}}$ is less critical as it barely affects the overall
performance, but does help on eng-xx and xx-eng very low pairs. In the middle
section of Table 3, we note that CMR top-1 is not sensitive to the exact value
of $b$, but, at low budget $b$ (less capacity), model performance
significantly drops on eng-xx across all pairs. Pairs in xx-eng, on the other
hand, favor a mid-range budget value.

In Table 2 for CMR top-1, we see +0.4 chrF++ across all pairs into English,
and +0.4 chrF++ across non-English pairs. Improvements are larger for out of
English low and very low-resource languages, with +0.5 and +1.0 chrF++
respectively. For CMR top-2, we see +1.9 chrF++ across all pairs out of
English and +0.9 chrF++ across non-English pairs. The improvements are largest
for low and very low-resource languages, with +2.3 and +3.2 chrF++ out of
English, and +0.9 and +1.5 into English. CMR top-2 is computationally more
expensive by 23% because of the additional shared FFN layer at the level of
each MoE layer in the model.

We find that Gating Dropout performs better than the baseline MoE, but is
outperformed by all of our proposed methods. Overall, these results
demonstrate that EOM, FOM, and CMR strategies help improve on top of vanilla
MoE.

###  5.3 CL

| eng-xx | xx-eng | xx-yy
---|---|---|---
| all |  | v.low | all |  | v.low | all
MoE-64 | 44.3 |  | 32.5 | 54.4 |  | 47.7 | 41.9
\+ CL (count-based) | 43.7 |  | 32.5 | 54.0 |  | 47.1 | 41.1
\+ CL (step-based) | 44.7 |  | 33.3 | 54.6 |  | 47.9 | 42.2
MoE-64 EOM | 44.7 |  | 33.4 | 54.8 |  | 48.3 | 42.5
+CL (step-based) | 44.3 |  | 33.1 | 54.7 |  | 48.4 | 42.2

Table 4:  Results of Curriculum Learning applied to a vanilla MoE model and an
MoE model with EOM.

#### Experimental Setup.

To derive the phases of the curriculum, we train a vanilla MoE model with
$p_{\text{drop}}{=}0.3$ (our baseline), then, based on observed over-fitting
patterns, we partition the tasks in our MMT dataset. For both count and step-
based curricula, we introduce pairs in $n{=}3$ phases over $U{=}100k$. For
count-based curriculum, we partition language pairs into bins w.r.t. the
training examples available for the task ($\mathcal{D}_{t}$): $b_{1}$ if
$|\mathcal{D}_{t}|\geq 5e6$, $b_{2}$ if $8e5\leq|\mathcal{D}_{t}|<5e6$, and
$b_{3}$ if $|\mathcal{D}_{t}|<8e5$. With that we use
$(k_{1},k_{2},k_{3})=(100k,40k,20k)$.777That means $b_{1}$ is introduced at
step $0$, $b_{2}$ at step $U-40k$, and $b_{3}$ at step $U-20k$ For step-based
curriculum, we follow Algorithm 1 with $n=5$ and merge the first 3 buckets
resulting in 3 bins introduced at $(k_{1},k_{2},k_{3})=(100k,40k,20k)$. See
Appendix C for the exact partitioning.

To combine a stronger dropout regularization with Curriculum Learning methods,
we next apply our best CL strategy (step-based) to an MoE model with EOM
($p_{\text{eom}}{=}0.1$).

#### Results.

We show the results of our CL experiments in Table 4. For the baseline MoE-64,
by using step-based CL, we improve the accuracy on very low-resource
directions by 0.8 chrF++ in eng-xx and 0.2 chrF++ in xx-eng . Across all
resource levels, we improve the accuracy in eng-xx and xx-eng by 0.4 and 0.2
chrF++. On non-English directions step-based CL improves the quality by 0.3
chrF++. The count-based CL hurts the model performance in all tasks except
from very low-resource eng-xx directions.

For MoE EOM, training with step-based CL actually hurts performance across all
tasks except for xx-eng very low-resource. We hypothesize that over-fitting on
our MMT dataset is already reduced by EOM, thus, adding a curriculum on top of
that is not needed and has a negligible impact on translation quality.

##  6 Analysis of Multilingual Sparsely Gated MoE Models.

Figure 5:  $E_{50\%}$ for the $p_{\text{drop}}{=}0.1$ model. Each column
correspond to an MoE layer and we aggregate the statistic across 6 task
groups.

(a) $p_{\text{drop}}{=}0.1$

(b) $p_{\text{drop}}{=}0.1$, $p_{\text{eom}}{=}0.3$

(c) $p_{\text{drop}}{=}0.1$, $p_{\text{fom}}{=}0.3$

(d) $p_{\text{drop}}{=}0.3$

(e) $p_{\text{drop}}{=}0.3$, $p_{\text{eom}}{=}0.1$

(f) $p_{\text{drop}}{=}0.3$, $p_{\text{fom}}{=}0.1$

Figure 6: Heatmap of experts’ colocation in consecutive MoE layers of the
encoder-deocder architecture. Darker colors correspond to higher colocation
meaning that an input token routed to expert $i$ in layer $\ell$ will likely
be routed to expert $j$ in layer ${\ell+1}$

MoE theoretically enables models to specialize expert capacity for different
tasks, but what do these models actually learn and how does our regularization
methods affect these learnings?

To evaluate if experts are specialized in any given task (translation
direction), we look at the experts’ usage per task i.e., for a given MoE layer
we report the distribution of the task tokens888source tokens for encoder
layers and target tokens for decoder layers in Flores-200 dev across the $E$
experts. For this metric, each token selects its top-1 expert. The gating
distributions are obtained in teacher-forcing mode, meaning that we feed the
true target prefix to predict the next target token in the decoder side. To
summarize these distributions we report the number of experts it take to cover
50% of the tokens, denoted with $E_{50\%}$. A low $E_{50\%}$ means that a
small number of experts is specialized in this task. We aggregate this
statistic across 6 groups of tasks: English-to-many, Many-to-English and Non-
English, with the suffix high or low for the resource level.

In figure 5, we show this statistic for the model trained with
$p_{\text{drop}}{=}0.1$999similar trends can be observed with other models,
regardless of the regularization hyper-parameters. We observe that translating
into English (xx-en-high/low) uses more experts in the decoder than
translating into non-English (xx-yy and en-xx) and that low resource target
languages (en-xx-low) activate less experts in the decoder.

To check if the specialized experts are only dedicated to a specific
translation direction or if they are activated for any low-resource direction,
we show in Figure 10 of the appendix the heat-map of task experts for all 110
directions in our dataset for the $p_{\text{drop}}{=}0.3$ model. We also look
at the similarities in task experts and show in Figure 11 of the appendix the
similarities heat-maps at different MoE layers.

We next look in Figure 6 at the co-location of experts in consecutive MoE
layers, i.e., if tokens assigned to expert $i$ in $\text{layer}_{\ell}$ will
automatically get routed to expert $j$ in $\text{layer}_{\ell+1}$ If $u_{ti}$
is token $x_{t}$’s usage of expert $i$, then the co-location is evaluated as
follows:

$\displaystyle\text{c}(\ell,{\ell+1})=\max_{i,j}{\text{cor}_{x_{t}\sim\mathcal{D}}(u_{ti}^{\ell},u_{tj}^{\ell+1})}$
(9)

We observe that: (1) There is almost no co-location between the encoder’s MoE
layers whereas the decoder’s MoE experts tend to co-locate. (2)
$p_{\text{drop}}{=}0.3$ leads to a higher co-location in the decoder’s MoE
layers, paired with EOM, and to a lesser extent FOM, this co-location
increases. (3) When the dropout rate is low (0.1), the addition of EOM or FOM
weakens the collocation of the decoder’s experts.

We additionally use the drop in translation quality with random-routing in
Section D.3 as a proxy for experts’ specialization.

##  7 Related work

#### Improved routing in MoE models.

Recent works have proposed alternatives to the commonly used top-2 gating of
Lepikhin et al. (2020): Hash layers (Roller et al., 2021) use random fixed
routing and Lewis et al. (2021) view routing as a linear assignment problem
and drop the load balancing loss. Zuo et al. (2022) suggest to randomly select
experts. Fedus et al. (2022) opt for top-1 routing, and Yang et al. (2021)
split experts into different groups and applies k top-1 routing in each. In
this work, we only use Top-2 gating101010We did use top-1 gating for CMR to
maintain a comparable computational cost with the baseline but our techniques
are orthogonal to the routing method.

#### Regularizing MoE models.

Zoph et al. (2022) tried increasing the dropout within the expert (dubbed
expert dropout) but saw marginal improvement in quality. Kim et al. (2021)
randomize the priority of tokens within a mini-batch as a regularization
method. Liu et al. (2022) propose gating dropout to reduce cross-machine
communication in MoE layers. Xie et al. (2022) propose routing tokens to
expert clusters and a cluster-level expert dropout.

#### Conditional compute and language-specific parameters.

A common solution to relax parameter sharing in MMT models is to use light-
weight language-specific adapters (Rebuffi et al., 2017; Bapna and Firat,
2019). Their size, however, scales linearly in the number of languages.
Baziotis et al. (2022) introduce hyper-adapters to generate the adapters
themselves. To make these language-specific parameters optional, Zhang et al.
(2021) propose CLSR to dynamically select language-specific or shared paths.
These paths are simple linear projections and do not incorporate routing.
Similar to our own CMR’s budget loss, CLSR optimizes the MMT cross-entropy
while constraining the use of the language-specific capacity. Another approach
similar to CMR is Residual-MoE (Rajbhandari et al., 2022). It is a hybrid
dense and MoE model but it does not learn weights for each component.

##  8 Conclusion

In massively multilingual settings with imbalanced datasets, MoE models over-
fit significantly more than dense models on low-resource directions. This work
introduce multiple effective strategies for regularizing MoE models and
achieving better performance across all language pairs, especially low-
resource pairs. With EOM and FOM, we propose dropout methods to further
regularize MoE models. We introduce in CMR a novel architecture to balance the
capacity between MoEs and shared dense paths. Finally, we design curricula for
introducing low-resource languages later during training. These strategies
lead to less over-fitting on low-resource tasks, leading to improvements in
translation quality.

## References

  * Almahairi et al. (2016) Amjad Almahairi, Nicolas Ballas, Tim Cooijmans, Yin Zheng, Hugo Larochelle, and Aaron Courville. 2016.  Dynamic capacity networks.  In _Proceedings of the 33rd International Conference on International Conference on Machine Learning - Volume 48_ , ICML’16, page 2091–2100. JMLR.org.
  * Ba et al. (2016) Lei Jimmy Ba, Jamie Ryan Kiros, and Geoffrey E. Hinton. 2016.  Layer normalization.  _CoRR_ , abs/1607.06450.
  * Bapna and Firat (2019) Ankur Bapna and Orhan Firat. 2019.  Simple, scalable adaptation for neural machine translation.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 1538–1548.
  * Baziotis et al. (2022) Christos Baziotis, Mikel Artetxe, James Cross, and Shruti Bhosale. 2022.  Multilingual machine translation with hyper-adapters.  In _EMNLP_.
  * Bengio et al. (2013) Yoshua Bengio, Nicholas Léonard, and Aaron C. Courville. 2013.  Estimating or propagating gradients through stochastic neurons for conditional computation.  _CoRR_ , abs/1308.3432.
  * Brown et al. (2020) Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020.  Language models are few-shot learners.  _Advances in neural information processing systems_ , 33:1877–1901.
  * Du et al. (2021) Nan Du, Yanping Huang, Andrew M. Dai, Simon Tong, Dmitry Lepikhin, Yuanzhong Xu, Maxim Krikun, Yanqi Zhou, Adams Wei Yu, Orhan Firat, Barret Zoph, Liam Fedus, Maarten Bosma, Zongwei Zhou, Tao Wang, Yu Emma Wang, Kellie Webster, Marie Pellat, Kevin Robinson, Kathy Meier-Hellstern, Toju Duke, Lucas Dixon, Kun Zhang, Quoc V. Le, Yonghui Wu, Zhifeng Chen, and Claire Cui. 2021.  Glam: Efficient scaling of language models with mixture-of-experts.  _CoRR_ , abs/2112.06905.
  * Fedus et al. (2022) William Fedus, Barret Zoph, and Noam Shazeer. 2022.  Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity.  _Journal of Machine Learning Research_ , 23(120):1–39.
  * Heffernan et al. (2022) Kevin Heffernan, Onur Çelebi, and Holger Schwenk. 2022.  Bitext mining using distilled sentence representations for low-resource languages.  In _EMNLP_.
  * Hwang et al. (2022) Changho Hwang, Wei Cui, Yifan Xiong, Ziyue Yang, Ze Liu, Han Hu, Zilong Wang, Rafael Salas, Jithin Jose, Prabhat Ram, et al. 2022.  Tutel: Adaptive mixture-of-experts at scale.  _arXiv preprint arXiv:2206.03382_.
  * Kaplan et al. (2020) Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. 2020.  Scaling laws for neural language models.  _CoRR_ , abs/2001.08361.
  * Kim et al. (2021) Young Jin Kim, Ammar Ahmad Awan, Alexandre Muzio, Andrés Felipe Cruz-Salinas, Liyang Lu, Amr Hendy, Samyam Rajbhandari, Yuxiong He, and Hany Hassan Awadalla. 2021.  Scalable and efficient moe training for multitask multilingual models.  _CoRR_ , abs/2109.10465.
  * Kingma and Ba (2015) Diederik P. Kingma and Jimmy Ba. 2015.  Adam: A method for stochastic optimization.  In _3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings_.
  * Kudo and Richardson (2018) Taku Kudo and John Richardson. 2018.  Sentencepiece: A simple and language independent subword tokenizer and detokenizer for neural text processing.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, EMNLP 2018: System Demonstrations, Brussels, Belgium, October 31 - November 4, 2018_ , pages 66–71. Association for Computational Linguistics.
  * Lepikhin et al. (2020) Dmitry Lepikhin, HyoukJoong Lee, Yuanzhong Xu, Dehao Chen, Orhan Firat, Yanping Huang, Maxim Krikun, Noam Shazeer, and Zhifeng Chen. 2020.  Gshard: Scaling giant models with conditional computation and automatic sharding.  _CoRR_ , abs/2006.16668.
  * Lewis et al. (2021) Mike Lewis, Shruti Bhosale, Tim Dettmers, Naman Goyal, and Luke Zettlemoyer. 2021\.  Base layers: Simplifying training of large, sparse models.  In _International Conference on Machine Learning_ , pages 6265–6274. PMLR.
  * Liu et al. (2022) Rui Liu, Young Jin Kim, Alexandre Muzio, Barzan Mozafari, and Hany Hassan Awadalla. 2022.  Gating dropout: Communication-efficient regularization for sparsely activated transformers.  _arXiv preprint arXiv:2205.14336_.
  * NLLB Team et al. (2022) NLLB Team, Marta R. Costa-jussà, James Cross, Onur Çelebi, Maha Elbayad, Kenneth Heafield, Kevin Heffernan, Elahe Kalbassi, Janice Lam, Daniel Licht, Jean Maillard, Anna Sun, Skyler Wang, Guillaume Wenzek, Al Youngblood, Bapi Akula, Loic Barrault, Gabriel Mejia-Gonzalez, Prangthip Hansanti, John Hoffman, Semarley Jarrett, Kaushik Ram Sadagopan, Dirk Rowe, Shannon Spruit, Chau Tran, Pierre Andrews, Necip Fazil Ayan, Shruti Bhosale, Sergey Edunov, Angela Fan, Cynthia Gao, Vedanuj Goswami, Francisco Guzmán, Philipp Koehn, Alexandre Mourachko, Christophe Ropers, Safiyyah Saleem, Holger Schwenk, and Jeff Wang. 2022.  No language left behind: Scaling human-centered machine translation.  _arXiv preprint arXiv:2207.04672_.
  * Popović (2017) Maja Popović. 2017.  chrf++: words helping character n-grams.  In _Proceedings of the Second Conference on Machine Translation, Volume 2: Shared Task Papers_ , pages 612–618, Copenhagen, Denmark. Association for Computational Linguistics.
  * Rajbhandari et al. (2022) Samyam Rajbhandari, Conglong Li, Zhewei Yao, Minjia Zhang, Reza Yazdani Aminabadi, Ammar Ahmad Awan, Jeff Rasley, and Yuxiong He. 2022.  Deepspeed-moe: Advancing mixture-of-experts inference and training to power next-generation ai scale.  _arXiv preprint arXiv:2201.05596_.
  * Rebuffi et al. (2017) Sylvestre-Alvise Rebuffi, Hakan Bilen, and Andrea Vedaldi. 2017.  Learning multiple visual domains with residual adapters.  _Advances in neural information processing systems_ , 30.
  * Roller et al. (2021) Stephen Roller, Sainbayar Sukhbaatar, Arthur Szlam, and Jason E Weston. 2021.  Hash layers for large sparse models.  In _Advances in Neural Information Processing Systems_.
  * Shazeer et al. (2017) Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, and Jeff Dean. 2017.  Outrageously large neural networks: The sparsely-gated mixture-of-experts layer.  In _Proceedings of International Conference on Learning Representations (ICLR)_.
  * Sutskever et al. (2014) Ilya Sutskever, Oriol Vinyals, and Quoc V. Le. 2014.  Sequence to sequence learning with neural networks.  In _Proc. of NeurIPS_.
  * Szegedy et al. (2015) Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jonathon Shlens, and Zbigniew Wojna. 2015.  Rethinking the inception architecture for computer vision.  _CoRR_ , abs/1512.00567.
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017.  Attention is all you need.  In _Advances in neural information processing systems_ , pages 5998–6008.
  * Xie et al. (2022) Yuan Xie, Shaohan Huang, Tianyu Chen, and Furu Wei. 2022.  Moec: Mixture of expert clusters.  _arXiv preprint arXiv:2207.09094_.
  * Xiong et al. (2020) Ruibin Xiong, Yunchang Yang, Di He, Kai Zheng, Shuxin Zheng, Chen Xing, Huishuai Zhang, Yanyan Lan, Liwei Wang, and Tieyan Liu. 2020.  On layer normalization in the transformer architecture.  In _International Conference on Machine Learning_ , pages 10524–10533. PMLR.
  * Yang et al. (2021) An Yang, Junyang Lin, Rui Men, Chang Zhou, Le Jiang, Xianyan Jia, Ang Wang, Jie Zhang, Jiamang Wang, Yong Li, et al. 2021.  M6-t: Exploring sparse expert models and beyond.  _arXiv preprint arXiv:2105.15082_.
  * Zhang et al. (2021) Biao Zhang, Ankur Bapna, Rico Sennrich, and Orhan Firat. 2021.  Share or not? learning to schedule language-specific capacity for multilingual translation.  In _International Conference on Learning Representations_.
  * Zoph et al. (2022) Barret Zoph, Irwan Bello, Sameer Kumar, Nan Du, Yanping Huang, Jeff Dean, Noam Shazeer, and William Fedus. 2022.  St-moe: Designing stable and transferable sparse expert models.  _arXiv preprint arXiv:2202.08906_.
  * Zuo et al. (2022) Simiao Zuo, Xiaodong Liu, Jian Jiao, Young Jin Kim, Hany Hassan, Ruofei Zhang, Jianfeng Gao, and Tuo Zhao. 2022.  Taming sparsely activated transformer with stochastic experts.  In _International Conference on Learning Representations_.

##  Appendix A Training data

We list in Table 5 the amount of data (bitexts) used to train our models.
Figure 7 shows the data distribution over language pairs sorted by the example
count per pair. The highest resource language pair has 180M examples (English-
French), and the lowest resource language pair has 40K examples (Hindi-Tamil).

$10^{4}$$10^{6}$$10^{8}$Language PairsTrain Count Figure 7: Training data
across all language pairs in our MMT dataset.

##  Appendix B Training details

We train Transformer encoder-decoder models with dimension 1024, FFN dimension
8192, 16 attention heads, 24 encoder layers and 24 decoder layers. We apply
Layer-normalization (Ba et al., 2016) at the beginning of each Transformer
sub-layer (Pre-LN), as opposed to after the residual connection (Post-LN).
This is because Pre-LN is more stable in practice compared to Post-LN (Xiong
et al., 2020). All models are trained for 100k updates with an effective batch
size of 1M tokens per update. We optimize with Adam (Kingma and Ba, 2015)
using ($\beta_{1},\beta_{2},\epsilon)=(0.9,0.98,10^{-6})$. We linearly
increase the learning rate up to 0.004 through 8000 warmup updates, then
follow the inverse square root learning rate schedule. For Top-2-Gating, we
set the expert capacity to $2\times T/E$, i.e., we enforce that each expert
processes, at most, $2\times T/E$ tokens, where $T$ is the number of tokens in
the mini-batch and $E$ is the number of experts. During generation, we set the
capacity to $T$ so that all tokens can be routed to whichever expert they
choose.

code | language
---|---
ace_Latn | Acehnese
afr | Afrikaans
ara_Arab | Arabic
ast | Asturian
ayr | Aymara
bel | Belarussian
bul | Bulgarian
cjk | Chokwe
cym | Welsh
eus | Basque
ewe | Ewe
fas | Persian
fin | Finnish
fon | Fon
fra | French
fuv | Fula
hau | Hausa
hin | Hindi
isl | Icelandic
ita | Italian
jpn | Japanese
kea | Kabuverdianu
kik | Kikuyu
kin | Kinyarwanda
kon | Kongo
kor | Korean
lav | Latvian
lin | Lingala
luo | Luo
mal | Malayalam
mar | Marathi
nso | Northern Sotho
oci | Occitan
por | Portuguese
run | Rundi
rus | Russian
sin | Sinhalese
snd | Sindhi
swh | Swahili
tam | Tamil
tat_Cyrl | Tatar
tel | Telugu
tir | Tigrinya
tsn | Tswana
tso | Tsonga
twi | Twi
urd | Urdu
vie | Vietnamese
wol | Wolof
yor | Yoruba
yue | Yue Chinese
zho_Hans | Chinese

direction |  | primary | mined
---|---|---|---
eng-ace_Latn |  | 36,591 | 1,148,759
eng-afr |  | 1,449,916 | 5,840,012
eng-ara_Arab |  | 36,340,863 | 39,447,939
eng-ast |  | 526 | 874,884
eng-ayr |  | 69,185 | 610,749
eng-bel |  | 47,166 | 892,477
eng-bul |  | 26,706,641 | 35,742,011
eng-cjk |  | 33,038 | 660,404
eng-cym |  | 149,598 | 4,239,464
eng-ewe |  | 534,793 | 739,132
eng-fas |  | 4,402,104 | 19,527,935
eng-fin |  | 34,784,117 | 27,243,736
eng-fon |  | 36,752 | 299,065
eng-fra |  | 37,993,938 | 141,929,009
eng-fuv |  | 18,242 | 189,675
eng-hau |  | 345,481 | 4,598,698
eng-hin |  | 1,688,720 | 24,497,780
eng-isl |  | 1,096,312 | 6,744,150
eng-ita |  | 44,712,431 | 82,724,756
eng-kea |  | 4,727 | 146,254
eng-kik |  | 98,740 | 119,396
eng-kin |  | 376,914 | 2,423,473
eng-kon |  | 188,251 | 213,799
eng-lav |  | 3,867,869 | 10,699,069
eng-lin |  | 666,273 | 555,208
eng-luo |  | 129,000 | 670,367
eng-mal |  | 585,452 | 7,703,121
eng-mar |  | 335,259 | 6,143,242
eng-nso |  | 526,097 | 644,586
eng-oci |  | 5,915 | 585,817
eng-run |  | 454,678 | 1,138,461
eng-rus |  | 30,271,773 | 71,205,569
eng-sin |  | 461,857 | 3,288,143
eng-snd |  | 95,718 | 2,434,012
eng-tam |  | 680,297 | 7,223,944
eng-tel |  | 253,718 | 7,880,705
eng-tir |  | 83,980 | 1,128,918
eng-tso |  | 711,883 | 881,110
eng-twi |  | 508,746 | 1,220,279
eng-urd |  | 875,172 | 2,873,007
eng-vie |  | 3,689,843 | 39,782,690
eng-wol |  | 9,233 | 147,746
eng-yor |  | 397,793 | 2,099,168
eng-yue |  | 54,534 | 0
eng-zho_Hans |  | 228,658 | 33,684,682

direction |  | primary | mined
---|---|---|---
ara_Arab-sin |  | 402,450 | 0
eus-por |  | 432,823 | 0
fra-hau |  | 168,631 | 0
fra-kon |  | 147,886 | 0
fra-lin |  | 397,535 | 0
fra-swh |  | 664,013 | 0
hin-tam |  | 39,992 | 0
jpn-kor |  | 1,009,697 | 0
rus-tat_Cyrl |  | 263,496 | 0
swh-tsn |  | 697,681 | 0

Table 5: List of languages and Data counts between primary (pre-existing
publicly available parallel data) and mined (Heffernan et al., 2022) for the
110 directions of our MMT dataset. 45 languages are paired with English for a
total of 90 English-centric directions. The remaining 20 directions are non-
English centric.

##  Appendix C Curriculum Learning

####  Count-based CL.

We empirically partition based on training example counts. We first train our
baseline model (MoE-64 $(p_{\text{drop}}{=}0.3$) without CL, then we look at
possible correlations between the number of steps before over-fitting and the
count of training examples. In Figure 8 we plot these data points with the
counts on the y-axis and the start-of-over-fitting step on the x-axis. The
horizontal red lines indicate where the count-based curriculum thresholds were
set in order to partition language pairs into bins.

20k40k60k80k100k1e51e61e71e8$s_{\text{best}}(t)$$|\mathcal{D}_{t}|$ Figure 8:
For the baseline MoE model, we plot the steps corresponding to the best
validation perplexity ($s_{\text{best}}$ on the x-axis) against the number of
training examples ($|\mathcal{D}_{t}|$ on the y-axis).

We list in Table 6 the tasks in each bin for the baseline MoE model.

####  Step-based CL.

We partition based on the step where we observed a task to start over-fitting.
Following Algorithm 1, we partition the tasks into $n$ bins. In our
experiments, we started with $n{=}5$ resulting in a $\Delta$ of 20k steps.
However, we merged the first three bins with characteristic steps
$k_{1}=100k$, $k_{2}=80k$ and $k_{3}=60k$ to remain comparable with count-
based CL.

bin $b_{i}$ | #tasks | $k_{i}$ |  Language pairs
---|---|---|---
$b_{1}$ | 17$\times$2 | 100k |  eng-afr, eng-ara_Arab, eng-bul, eng-fas, eng-fin, eng-fra, eng-hin, eng-isl, eng-ita, eng-lav, eng-mal, eng-mar, eng-rus, eng-tam, eng-tel, eng-vie, eng-zho_Hans
$b_{2}$ | 24$\times$2 | 40k |  eng-ace_Latn, eng-ast, eng-ayr, eng-bel, eng-cjk, eng-cym, eng-ewe, eng-hau, eng-kin, eng-lin, eng-luo, eng-nso, eng-oci, eng-run, eng-sin, eng-snd, eng-tir, eng-tso, eng-twi, eng-urd, eng-yor, fra-swh, jpn-kor, swh-tsn
$b_{3}$ | 14$\times$2 | 20k |  eng-fon, eng-fuv, eng-kea, eng-kik, eng-kon, eng-wol, eng-yue, ara_Arab-sin, eus-por, fra-hau, fra-kon, fra-lin, hin-tam, rus-tat_Cyrl

Table 6: Count-based CL bins for the baseline MoE model
($p_{\text{drop}}{=}0.3$). Step represents the number of steps the language
pairs in this bin are trained

bin $b_{i}$ | #tasks | $k_{i}$ |  Language pairs
---|---|---|---
$b_{1}$ | 86 | 100k |  ace_Latn-eng, afr-eng, ara_Arab-eng, ara_Arab-sin, ast-eng, bel-eng, bul-eng, cym-eng, eng-afr, eng-ara_Arab, eng-ast, eng-bel, eng-bul, eng-cym, eng-ewe, eng-fas, eng-fin, eng-fra, eng-hau, eng-hin, eng-isl, eng-ita, eng-kea, eng-kin, eng-lav, eng-luo, eng-mal, eng-mar, eng-nso, eng-oci, eng-run, eng-rus, eng-sin, eng-snd, eng-tam, eng-tel, eng-tir, eng-tso, eng-twi, eng-urd, eng-vie, eng-yor, eng-zho_Hans, eus-por, ewe-eng, fas-eng, fin-eng, fra-eng, fra-hau, fra-swh, hau-eng, hin-eng, hin-tam, isl-eng, ita-eng, jpn-kor, kin-eng, kor-jpn, lav-eng, lin-eng, lin-fra, luo-eng, mal-eng, mar-eng, nso-eng, por-eus, rus-eng, sin-ara_Arab, sin-eng, snd-eng, swh-fra, swh-tsn, tam-eng, tam-hin, tat_Cyrl-rus, tel-eng, tir-eng, tsn-swh, tso-eng, twi-eng, urd-eng, vie-eng, wol-eng, yor-eng, yue-eng, zho_Hans-eng
$b_{2}$ | 12 | 40k |  ayr-eng, cjk-eng, eng-ace_Latn, eng-ayr, eng-kik, eng-lin, fra-lin, fuv-eng, kik-eng, oci-eng, run-eng, rus-tat_Cyrl
$b_{3}$ | 12 | 20k |  eng-cjk, eng-fon, eng-fuv, eng-kon, eng-wol, eng-yue, fon-eng, fra-kon, hau-fra, kea-eng, kon-eng, kon-fra

Table 7: Step-based CL bins for the baseline MoE-64 ($p_{\text{eom}}{=}0.1$)

bin $b_{i}$ | #tasks | $k_{i}$ |  Language pairs
---|---|---|---
$b_{1}$ | 95 | 100k |  ace_Latn-eng, afr-eng, ara_Arab-eng, ara_Arab-sin, ast-eng, ayr-eng, bel-eng, bul-eng, cym-eng, eng-ace_Latn, eng-afr, eng-ara_Arab, eng-ast, eng-bel, eng-bul, eng-cym, eng-ewe, eng-fas, eng-fin, eng-fra, eng-hau, eng-hin, eng-isl, eng-ita, eng-kea, eng-kik, eng-kin, eng-lav, eng-luo, eng-mal, eng-mar, eng-nso, eng-oci, eng-run, eng-rus, eng-sin, eng-snd, eng-tam, eng-tel, eng-tir, eng-tso, eng-twi, eng-urd, eng-vie, eng-yor, eng-zho_Hans, eus-por, ewe-eng, fas-eng, fin-eng, fra-eng, fra-hau, fra-lin, fra-swh, fuv-eng, hau-eng, hau-fra, hin-eng, hin-tam, isl-eng, ita-eng, jpn-kor, kin-eng, kor-jpn, lav-eng, lin-eng, lin-fra, luo-eng, mal-eng, mar-eng, nso-eng, oci-eng, por-eus, run-eng, rus-eng, rus-tat_Cyrl, sin-ara_Arab, sin-eng, snd-eng, swh-fra, swh-tsn, tam-eng, tam-hin, tat_Cyrl-rus, tel-eng, tir-eng, tsn-swh, tso-eng, twi-eng, urd-eng, vie-eng, wol-eng, yor-eng, yue-eng, zho_Hans-eng
$b_{2}$ | 5 | 40k |  eng-ayr, eng-cjk, eng-lin, eng-wol, eng-yue
$b_{3}$ | 10 | 20k |  cjk-eng, eng-fon, eng-fuv, eng-kon, fon-eng, fra-kon, kea-eng, kik-eng, kon-eng, kon-fra

Table 8: Step-based CL bins for the baseline MoE-64 EOM
($p_{\text{drop}}{=}0.3$,, $p_{\text{eom}}{=}0.1$)

##  Appendix D Analysis of Multilingual Sparsely Gated MoE Models

###  D.1 Task Experts

We show in figure 10 the task experts for all 110 directions in our MMT
dataset. These statistics correspond to the MoE-64 $p_{\text{drop}}{=}0.3$
model but similar trends are observed for all other 5 models we analyzed.
Predictably, we observe that in English-to-many directions (en-xx, left side
of the figure) the encoder experts are almost uniformly activated whereas the
encoder experts are specialized per target language. The opposite can observed
to a lesser extent on many-to-English direction (xx-en); we see some variation
in decoder layers, but most importantly we see little variation in the late
encoder layers (encoder.23) likely because the source representation at that
stage is becoming more-or-less language-agnostic.

###  D.2 Language experts.

To obtain statistics on a language’s usage of experts, we combine tokens from
tasks that include our language of interest, ${{<}\ell{>}}$, before computing
the density histogram of the top-1 experts. When looking at encoder MoE
layers, we consider ${{<}\ell{>}}{\to}x$ tasks and when looking at decoder MoE
layers we consider $x{\to}{{<}\ell{>}}$ tasks.

We plot in Figure 11 the cosine similarity scores between all 53 languages of
the MMT dataset, in the first and last encoder MoE layer, and the first and
last decoder MoE layer.

The similarity heatmaps demonstrate that in late decoder layers (see Figure
11(d)), the languages are being separated, i.e., dispatched to different set
of experts. Languages within the same family are highly similar in their
choice of experts, i.e., the late decoder MoE layers are language-specific.
This is particularly the case for languages in the Atlantic-Congo family (the
rows/columns from cjk to yor) and some pairs like
$\\{\texttt{snd\\_Arab},\texttt{urd\\_Arab}\\}$ in the Indo-European family or
$\\{\texttt{yue\\_Hant},\texttt{zho\\_Hans}\\}$ in the Sino-Tibetan family. To
a lesser extent, the early encoder MoE layers (see Figure 11(a)), also show
some language-expert specialization. The late encoder MoE layers and the early
decoder MoE layers (see Figure 11(b) and Figure 11(c)) seem to be language-
agnostic.

###  D.3 Random routing

$p_{\text{drop}}{=}0.1$$p_{\text{drop}}{=}0.3$$p_{\text{drop}}{=}0.3$/$p_{\text{eom}}{=}0.1$$p_{\text{drop}}{=}0.3$/$p_{\text{fom}}{=}0.1$$p_{\text{drop}}{=}0.1$/$p_{\text{eom}}{=}0.3$$p_{\text{drop}}{=}0.1$/$p_{\text{fom}}{=}0.3$$40$$50$$60$chrF++$p_{\text{drop}}{=}0.1$$p_{\text{drop}}{=}0.3$$p_{\text{drop}}{=}0.3$/$p_{\text{eom}}{=}0.1$$p_{\text{drop}}{=}0.3$/$p_{\text{fom}}{=}0.1$$p_{\text{drop}}{=}0.1$/$p_{\text{eom}}{=}0.3$$p_{\text{drop}}{=}0.1$/$p_{\text{fom}}{=}0.3$$20$$40$$60$$80$Random-
routing drop (%)en-xx-highen-xx-lowxx-en-highxx-en-lownon-eng-highnon-eng-low
Figure 9: Performance (left) and relative drop in performance with random
routing (right) of six of our MoE models.

To determine how specialized are MoE experts in any given task (translation
direction), we will evaluate models trained with top-2 gating with random
routing, i.e, we will choose 2 random experts instead of the top-2 experts and
use their weights to compute the final output. We evaluate each model twice
with different random seeds and consider the average of the two evaluation
runs. The drop in translation accuracy with random routing will be a proxy of
how specialized are MoE layers in any given task; if any two random experts
can perform as well as the top-2, then the experts are not specialized. In
Figure 9 We show this drop as a percentage of the top-2 performance in chrF++
(relative chrF++ drop).

A clear and unsurprising trend we see when comparing models is that heavy
regularization leads to less specialization (as shown by the diminishing
relative drop in the right plot of figure 9). Non-English centric directions
suffer the most from random routing followed by directions translating from
English. Translating into English suffers the least, meaning that a
considerable number of decoder experts are dedicated to outputting English.
Additionally, within the same subset of tasks, the low-resource directions
globally suffer more from random routing.

Figure 10: Task experts for all 110 direction of our MMT dataset. Each subplot
corresponds to a specific MoE layer and each column of the subplot show the
experts’ usage in the specific MoE layer for a translation direction. We show
the translation directions in 3 groups: English-to-many (en-xx), many-to-
English (xx-en) and Non-English (xx-yy).

(a) First encoder MoE layer

(b) Last encoder MoE layer

(c) First decoder MoE layer

(d) Last decoder MoE layer

Figure 11: Cosine Similarity Scores between languages of the MMT dataset at
different layers of the encoder-decoder architecture. The similarity is
measured w.r.t. the gating decisions (expert choice) per language (source-side
in the encoder and target-side in the decoder)


Theodoros Kouzelis1, Georgios Paraskevopoulos,1,2, Athanasios Katsamanis1,
Vassilis Katsouros1

# Weakly-supervised forced alignment of disfluent speech using phoneme-level
modeling

###### Abstract

The study of speech disorders can benefit greatly from time-aligned data.
However, audio-text mismatches in disfluent speech cause rapid performance
degradation for modern speech aligners, hindering the use of automatic
approaches. In this work, we propose a simple and effective modification of
alignment graph construction of CTC-based models using Weighted Finite State
Transducers. The proposed weakly-supervised approach alleviates the need for
verbatim transcription of speech disfluencies for forced alignment. During the
graph construction, we allow the modeling of common speech disfluencies, i.e.
repetitions and omissions. Further, we show that by assessing the degree of
audio-text mismatch through the use of Oracle Error Rate, our method can be
effectively used in the wild. Our evaluation on a corrupted version of the
TIMIT test set and the UCLASS dataset shows significant improvements,
particularly for recall, achieving a 23-25% relative improvement over our
baselines.

Index Terms: fluency disorder, phonetic alignment, WFST

##  1 Introduction

Speech disfluencies are common for both children and adults [1]. Stuttering
when reading aloud or speaking spontaneously is an example of a disfluency
that can have a profound impact on an individual's ability to communicate
effectively [2]. Stuttering typically begins in childhood [3] and manifests
itself as interruptions in the flow of speech such as repeating words or
sounds and committing false starts. Diagnosis and assessment of such
disfluencies have traditionally been done by clinicians, who manually count
stuttering events after having transcribed and aligned text and audio of
recorded sessions to classify disfluencies [4]. Disfluencies in both
spontaneous speech and aloud reading are not prevalent only in the speech of
individuals with speech disorders. In [5], Shriberg demonstrates that there is
a $50\%$ probability for a sentence of $10–13$ words to include a disfluency
and that the probability increases with sentence length.

Figure 1: An overview of our proposed approach. Given an approximate
transcription, the disfluent verbatim utterance is aligned with the
corresponding sections of the audio.

In studies of stuttering or other disfluencies, the analysis of large corpora
is essential due to the rich inherent variability in speech production.
However, the traditional method of manually annotating and aligning words and
phonemes is an extremely time-consuming process and can take up to 130 times
real-time [6]. Automatic forced alignment, which detects time boundaries for
each phoneme or word of already transcribed speech, can replace or speed up
manual alignment depending on the required level of accuracy [7].

Figure 2: The composing parts of the alignment graph $\mathcal{A}$. (a), (b)
present the linear FSA and the Modified FSA constructed on the sentence “don’t
ask” with CMU phonetic transcription [D AA N T AE S K]. Part-word repetition
arcs (PW) are illustrated in purple, word repetition arcs (W) in blue, and
word deletion arcs (D) in red. (b), (d) are an example Emissions Graph and the
standard CTC Topology Graph constructed over the CMU phone set. For clarity,
we illustrate the CTC topology over two phones {AA, AE}.

(a) Linear FSA $\mathcal{Y}$

(b) Emissions Graph $\mathcal{E}_{x}$

(c) Modified FSA $\mathcal{Y}_{M}$

(d) CTC Topology Graph $\mathcal{T}$

Manual transcription processes, however, typically correct or remove any
disfluency events, resulting in transcripts that are considerably cleaner than
the actual speech [8]. Further, when speech contains disfluencies, it is
necessary to manually transcribe any corresponding events before the alignment
since the performance of conventional forced alignment approaches can be
significantly affected by mismatches between speech and text [9]. Therefore,
it is crucial for robust speech-to-text alignment to be able to detect
disfluencies and align them accurately with the appropriate parts of the
utterance.

Conventionally, speech-to-text alignment is performed using Hidden Markov
Models - Gaussian Mixture Model (HMM-GMM) systems by application of the
standard Viterbi-based forced alignment. However, very long audio and
potential mismatches between speech and text pose a challenge to these systems
as they are susceptible to error propagation. In the literature these issues
are usually mitigated by iterative approaches [9, 10, 11]. In [11], the
alignment and correction of approximate transcriptions are addressed with an
iterative approach and the use of a modified lattice that allows word
insertions and deletions. A similar approach with a grammar-based decoder has
been used for the alignment of dysarthric speech [12]. Prior research has
explored the effectiveness of using an augmented decoding grammar based on the
original transcription to identify disfluencies, including word and syllable
repetitions, in both stuttering detection [13] and disfluency detection during
oral reading [14, 15]. However, these studies have not addressed the issue of
performance degradation of forced alignment when faced with untranscribed
disfluencies.

While HMM-GMM based techniques have been thoroughly researched for the
alignment of speech and text, recent works are exploring the use of neural
networks with promising results [16, 17, 18, 19]. In most recent neural forced
aligners the final alignment is extracted from the emission probabilities by
Dynamic Time Wrapping (DTW) [16] or by following the procedure in [18] if the
blank token of Connectionist Temporal Classification (CTC) training is present
i.e., cumulative score, blank distribution, and beam search decoding [20].
These methods limit the modifiability made possible by decoding with Weighted
Finite State Transducers (WFSTs). WFST representations enable the efficient
incorporation of lexicons and language models into CTC decoding [21], memory-
efficient variations of the standard CTC [22], and even training with
partially labeled data [23].

In this work, we investigate the performance degradation of state-of-the-art
forced aligners under untranscribed disfluencies. We demonstrate that given
the emission probabilities of a frame classification or CTC-based model, the
forced alignment can be achieved with WFST operations. Building on this, we
propose a weakly-supervised forced alignment that aims at aligning the full
verbatim disfluent utterance without the need for manual transcription. An
overview of our proposed system is presented in Figure 1. To address an in the
wild scenario, where the severity of audio-text mismatch is unknown we
constrain the degrees of freedom of the weakly-supervised forced alignment
leveraging Oracle Error Rate [24]. To test our method we employ a state-of-
the-art neural forced aligner [16] that is based on Wav2Vec2 [25]. We evaluate
our approach on the UCLASS dataset [4] and a corrupted version of the TIMIT
test set with synthesized disfluencies.

Our key contributions can be summarized as follows: (a) We address the
performance degradation of forced alignment under untranscribed disfluencies,
(b) we introduce a weakly supervised forced alignment that is applicable to
modern CTC-based neural forced aligners, (c) we construct and provide
DisfluenTIMIT, a testbed for the evaluation of forced alignment under
untranscribed disfluencies which we hope will inspire further research on the
topic. Our code is available as open-source111https://github.com/zelaki/WSFA/.

##  2 Method

For the methods described in this Section, we assume: (a) an input sequence
$X=[x_{0},...,x_{T-1}]$ corresponding to the speech signal, (b) a phonetic
transcription $Y=[y_{0},...,y_{N-1}]$ , $y_{n}\in\mathcal{P}$, where
$\mathcal{P}$ is the phonetic vocabulary, and (c) a network that given $X$
generates frame-based phone posteriors in the log-domain $\log P(p_{t}|X)$
with $p_{t}\in\mathcal{P}$ and $t\in T$.

Weakly supervised forced alignment with WFSTs: Given the phonetic
transcription $Y$, a label graph $\mathcal{L}_{Y}$ that represents all the
valid alignment paths can be constructed. Further, using the phone posterior
log probabilities, an emission graph $\mathcal{E}_{x}$ (Figure 2(b)) can be
created, where arc weights correspond to $\log P(p_{t}|X)$. The alignment
graph is derived by composing the emission and label graphs,
$\mathcal{A}=\mathcal{L}_{Y}\circ\mathcal{E}_{x}$. The shortest path in
$\mathcal{A}$ is the final frame-level alignment. The label graph can be
further deconstructed to its composing parts
$\mathcal{L}_{Y}=\mathcal{T}\circ\mathcal{Y}$, a topology graph $\mathcal{T}$
(Figure 2(d)) and a linear Finite State Acceptor (FSA) $\mathcal{Y}$ (Figure
2(a)), that accepts only the input phonetic transcription $Y$. Thus the
alignment graph can be written as:

$\displaystyle\mathcal{A}=\mathcal{T}\circ\mathcal{Y}\circ\mathcal{E}_{x}$ (1)

To enable the automated recognition and alignment of disfluencies, we suggest
a straightforward adaptation to the linear FSA. Given an approximate
transcription, a Modified FSA $\mathcal{Y}_{M}$ incorporating non-consuming
$<eps>$ arcs is constructed. Figure 2(c) shows an example Modified FSA for the
sentence “don't ask” [D AA N T AE S K]. Following the horizontal arcs, the
original transcription is obtained. Traversing the blue arcs allows both word
and phrase repetitions, such as “don't ask don't ask”. Red arcs allow the
deletion of a word or phrase. Omissions of words are frequently observed
during oral reading, particularly in children [26]. Part-word repetitions are
allowed by the purple arcs. Unlike the word lattices described in [13] and
[14], our approach entails modeling of part-word repetitions at the phoneme
level rather than the syllable level, alleviating the need for manual
segmentation of words into syllables. Intra-word repetitions are not modeled
since they are not as common in disfluent speech [13].

The arc weights of $\mathcal{Y}_{M}$ are constrained by a prior probability
$\alpha$ which restraints excessive changes to the input phonetic
transcription. For each state, there is a probability of $\alpha$ assigned to
traversing a horizontal arc that follows the path outlined by the
transcription. Conversely, the probability of traversing a non-consuming
$<eps>$ arc is $1-\alpha$. In the case of the start and end word states, the
probability is equally divided between the deletion (red) and the repetition
(blue) arc. When $\alpha=1$ the Modified FSA degenerates to the linear FSA. As
noted by prior studies [13, 14, 27] the weights of $\mathcal{Y}_{M}$ are
effective when $\alpha\geq 0.9$ as the log probabilities of graph
$\mathcal{E}_{x}$ are of larger orders of magnitude. Thus we will denote
$\alpha=1-10^{-\beta}$ with $\beta\in[0,+\inf)$ and consider values of
$\beta$.
Adaptively choosing a value for $\beta$: The modified FSA adds degrees of
freedom in the forced alignment process allowing the recognition of
untranscribed events, but also the possibility of false alarms. Thus, the
value of $\beta$ must balance this trade-off. In previous works [13, 14, 27],
the arc weights are empirically decided to fit the target data. But in a real
word scenario, the mismatch between audio and transcription as well as the
severity of speech disfluencies are unknown. So, the value of $\beta$ should
be proportional to the mismatch between the spoken utterance and the
transcription. A simple method to quantify this mismatch is Oracle Error Rate
(OER) [24]. A low OER indicates that the transcription of the utterance is
mostly correct. We calculate OER as in [24, 28], with a biased bi-gram
language model.

Figure 3 demonstrates that utilizing OER is an effective means of
distinguishing between clean and disfluent samples. Building on this, we
derive an intuitive formula to condition $\beta$, and consequently the degrees
of freedom of the weakly-supervised forced alignment, on OER. For a sampled
audio-transcription pair $S$ we set $\beta_{S}=10^{(1-O_{S})}$, where $O_{S}$
is the OER of sample $S$. Since OER can be greater than 1, we explicitly
restrict $O_{S}\in[0,1]$ and, thus, $\beta\in[1,10]$.

Figure 3: Oracle Error Rates for TIMIT and DisfluenTIMIT test sets.
DisfluenTIMIT is presented in Section 3. Table 1: Evaluation results on UCLASS
and DisfluenTIMIT. WFST and WFST_WS denote the standard and weakly-supervised
force alignment presented in Section 2.

Dataset | Method | P | $\text{P}_{\downarrow\%}$ | R | $\text{R}_{\downarrow\%}$ | F1 | $\text{F1}_{\downarrow\%}$ | R-val | $\text{R-val}_{\downarrow\%}$ | Overlap | $\text{Overlap}_{\downarrow\%}$
---|---|---|---|---|---|---|---|---|---|---|---
DisfluenTIMIT | MFA | $0.56$ | $14.9$ | $0.48$ | $29.4$ | $0.51$ | $23.4$ | $0.58$ | $17.9$ | $0.50$ | $24.4$
DTW | $0.59$ | $9.0$ | $0.47$ | $22.4$ | $0.52$ | $17.4$ | $0.59$ | $14.1$ | $0.57$ | $15.4$
WFST | $0.57$ | $12.3$ | $0.47$ | $21.9$ | $0.51$ | $18.3$ | $0.584$ | $14.4$ | $0.57$ | $15.3$
WFST_WS(Ours) | 0.61 | 5.8 | 0.60 | 1.7 | 0.60 | 3.8 | 0.66 | 3.4 | 0.67 | 1.1
UCLASS | MFA | $0.36$ | $28.4$ | $0.37$ | $33.1$ | $0.36$ | $30.8$ | $0.42$ | $23.4$ | - | -
DTW | $0.54$ | 1.67 | $0.52$ | $4.8$ | $0.53$ | $3.3$ | $0.57$ | $2.36$ | - | -
WFST | $0.55$ | $2.2$ | $0.52$ | $6.0$ | $0.54$ | $4.1$ | 0.58 | $3.1$ | - | -
WFST_WS(Ours) | 0.56 | $1.9$ | 0.54 | 3.8 | 0.54 | 2.9 | 0.58 | 2.3 | - | -

##  3 Data

The scarcity of speech disfluency datasets and the need for manual time-
aligned data pose significant challenges for evaluating speech alignment
methods. To overcome these challenges, we create the DisfluenTimit dataset by
introducing speech disfluencies into the standard Timit test set. Table 2
displays the disfluency types and their percentage of occurrence calculated
over the total number of words in the two datasets.

UCLASS is a commonly used dataset in disfluency-related machine learning
studies, consisting of 457 audio recordings of children with known stuttering
issues. UCLASS Release Two contains five recordings with manually aligned
transcriptions in orthographic format, where only the onset of each word or
disfluency is labeled. Since disfluencies are flagged we are able to extract
the approximate transcription for our experiments.

DisfluenTIMIT is a synthetic dataset that includes various types of
disfluencies, including part-word repetitions, word repetitions, phrase
repetitions, and word deletions. To generate these disfluencies, we followed a
very similar approach to [29], sampling and copying the audio or removing it
in the case of deletions. For an audio file with $n$ words, we randomly sample
a disfluency percentage $p\in[0.1,0.2,0.3]$ and insert $\lceil p\cdot n\rceil$
random disfluencies of types $d\in[PW,W,PH,D]$. The disfluencies are labeled
accordingly, and the ground truth disfluent phonetic transcription is
acquired.

Table 2: Types of disfluencies in the samples from UCLASS and DisfluenTimit.
The $3^{d}$ and $4^{th}$ row present average percentage of each type.

Label | Disfluency Type | DisfluenTimit | UCLASS
---|---|---|---
PW | Part-word repetitions | $4.7\%$ | $0.4\%$
W | Word repetitions | $4.8\%$ | $7.0\%$
PH | Phrase repetitions | $4.7\%$ | $0.7\%$
D | Word deletions | $4.7\%$ | $0\%$
I | Injections | $0\%$ | $2.3\%$

More specifically, to generate word repetitions, we repeat a randomly selected
word one, two, or three times before the original utterance. For phrase
repetitions, we repeat a phrase consisting of two to three words. For part-
word repetitions, we copy and append a random number of phones before the
selected word. For word deletions, we delete one, two, or three consecutive
words from the audio. To ensure a smoother transition between the added audio
and the existing clip, we use interpolation as described in [29]. The original
TIMIT 61 phones are collapsed into the 39 CMU phone set as in [16].

##  4 Experimental Setup

To evaluate the methods described in Section 3 we utilized a pre-trained
state-of-the-art frame classification model called
W2V2-FC222https://huggingface.co/charsiu/en_w2v2_fs_10ms[16] that has been
trained on Librispeech [30]. We compare our proposed method against two strong
baselines: Montreal Forced Aligner (MFA) and W2V2-FC with DTW as in [16]. All
WFST operations described in Section 3 are executed using the k2
framework333https://github.com/k2-fsa/k2. For each audio, we extract the
emission probabilities using W2V2-FC, resulting in a 2D tensor
$E\in\mathbb{R^{N\times T}}$ where T is the number of frames and N=42,
including the 39 CMU phones and special tokens such as [SIL] for silence,
[UNK] for unknown phones, and [PAD] for padding. As samples in the UCLASS
dataset are longer, we divide the audio into 10-second chunks, feed them
through the model, and concatenate the output emission probabilities in the
time dimension. The [SIL] token and the blank token $<b>$ of the topology
graph $\mathcal{T}$ are mapped in the same token id, as W2V2-FC has been fine-
tuned with the cross-entropy loss instead of CTC, and thus the blank token is
absent. In all our experiments the transition weights of the Modified FSA in
the weakly-supervised approach are adaptively chosen for each sample as
described in Section 2.

We evaluate the alignment accuracy using precision, recall, F1 score, and
R-value [31]. A hit is counted for each predicted phone boundary that falls
within a tolerance of $\tau$ and matches the predicted phone. For
DisfluenTIMIT, we only evaluate the phone onsets since each boundary marks the
onset and offset of consecutive phones, while for UCLASS, we evaluate only the
word onsets. We use a tolerance of 40ms and 100ms for DisfluenTIMIT and UCLASS
respectively. Additionally, we measure the percentage of accurately predicted
frame labels at a 10ms timescale for DisfluenTIMIT (Overlap metric in Table
1). The relative performance reduction for each metric $M$ is calculated as:

$M_{\downarrow\%}=\frac{M_{verbatim}-M_{approximate}}{M_{verbatim}}\times 100$
(2)

Table 3: The effect of word repetition $W$, word deletions $D$ and part-word
repetition $PW$ arcs on performance metrics for DisfluenTIMIT dataset. Minus
(-) denotes without.

Method | P | R | F1 | R-val | Overlap
---|---|---|---|---|---
WFST_WS | $0.61$ | $0.60$ | $0.60$ | $0.66$ | $0.67$
-PW | $0.59$ | $0.58$ | $0.58$ | $0.64$ | $0.65$
-D | $0.59$ | $0.59$ | $0.58$ | $0.63$ | $0.66$
-W | $0.59$ | $0.47$ | $0.52$ | $0.60$ | $0.57$
-W-D | $0.57$ | $0.47$ | $0.51$ | $0.59$ | $0.56$
-W-PW | $0.59$ | $0.47$ | $0.52$ | $0.59$ | $0.57$
-PW-D | $0.57$ | $0.58$ | $0.58$ | $0.62$ | $0.64$

Where, $M_{approximate}$ and $M_{verbatim}$ denote the performance metrics
calculated with untranscribed and transcribed disfluencies, respectively.

To evaluate the effectiveness of the adaptive approach presented in Section 2
for adjusting the Modified FSA weights, we conduct an experiment that
simulates a real-world scenario where a combination of disfluent and clear
speech samples are encountered without prior knowledge of the speech-text
mismatch. We create a mixed test set, referred to as MixTIMIT, by randomly
selecting the disfluent or fluent version of a sample from the DisfluenTIMIT
and TIMIT test sets, respectively.

##  5 Results and Discussion

Table 1 displays the alignment performance of all methods on both datasets.
Weakly-supervised forced alignment with WFSTs demonstrates superior
performance across all metrics, with the smallest performance reduction when
disfluencies are not transcribed. However, it is worth noting that while
recall and frame overlap are significantly improved, precision is only
marginally affected due to the presence of false alarms. Interestingly, DTW
outperforms MFA in both datasets indicating that it is less affected by
mismatches between audio and transcription.

To investigate the impact of the number of untranscribed disfluencies on the
performance, we divide uniformly DisfluenTIMIT into three categories based on
the percentage of disfluencies in relation to the total number of words. The
results are presented in Figure 4, which clearly demonstrates that our
proposed approach is robust to untranscribed disfluencies. In contrast, all
other methods and especially MFA, experience significant performance
degradation, particularly in severe cases where the percentage of disfluencies
is high.

Additional arcs: In Table 3 we present the performance gains associated with
each of the additional arcs to gain insight into the relative importance of
these components. Our results indicate that omitting repetition arcs $W$ has
the most significant impact on alignment performance, suggesting that
untranscribed word or phrase repetitions are a primary cause of performance
degradation of forced alignment.

Figure 4: Relative performance reduction of F1 metric under mild, moderate and
severe untranscribed disfluencies.

Effects of adaptive $\beta$: Using the MixTIMIT test set we compared the
impact of adaptively choosing $\beta$ for each sample versus globally setting
$\beta$ to an a priori value for all test samples. The results are presented
in Table 4. While the performance isn't improved substantially, our adaptive
approach succeeded at mitigating the necessity of prior knowledge about the
severity of untranscribed speech disfluencies in target data or an extensive
grid search.

Table 4: Global (G) vs adaptive (A) $\beta$ ablation on MixTIMIT.

$\beta$ | P | R | F1 | R-val | Overlap
---|---|---|---|---|---
1 (G) | $0.69$ | $0.67$ | $0.68$ | $0.72$ | $0.63$
10 (G) | $0.69$ | $0.67$ | $0.68$ | $0.73$ | $0.63$
100 (G) | $0.68$ | $0.65$ | $0.66$ | $0.71$ | $0.62$
1000 (G) | $0.67$ | $0.61$ | $0.64$ | $0.69$ | $0.58$
$10^{(1-O_{S})}$ (A) | $0.70$ | $0.68$ | $0.69$ | $0.74$ | $0.63$

##  6 Conclusions and Future Work

In this paper, we have explored the performance decrease of state-of-the-art
forced aligners under untranscribed disfluencies. We proposed a weakly-
supervised forced alignment, by a straightforward modification of the
alignment graph construction of CTC-based models, utilizing WFSTs. Our method
enables the alignment of common disfluencies without the need for manual
transcription while incurring minimal performance decrease when disfluencies
are not transcribed. Utilizing OER to quantify the severity of untranscribed
disfluencies before alignment enables the effective application of our
approach in the wild. In feature work, we plan to further explore the use of
WFSTs to model pronunciation variants and other disfluency types such as
injections, and also make the Modified FSA's weights learnable through
Expectation Maximization.

## References

  * [1] A. L. Choo, D. Greenberg, H. Li, and A. Talwar, ``Rate of stuttering and factors associated with speech fluency characteristics in adult struggling readers,'' _Journal of Learning Disabilities_ , vol. 56, no. 1, pp. 7–24, 2023, pMID: 35549596. [Online]. Available: https://doi.org/10.1177/00222194221095265
  * [2] R. Lowe, R. Menzies, M. Onslow, A. Packman, and S. O'Brian, ``Speech and anxiety management with persistent stuttering: Current status and essential research,'' _Journal of Speech, Language, and Hearing Research_ , vol. 64, pp. 1–16, 01 2021.
  * [3] E. Yairi and N. Ambrose, ``Epidemiology of stuttering: 21st century advances,'' _Journal of Fluency Disorders_ , vol. 38, no. 2, pp. 66–87, 2013. [Online]. Available: https://www.sciencedirect.com/science/article/pii/S0094730X12001052
  * [4] P. Howell, S. Davis, and J. Bartrip, ``The uclass archive of stuttered speech.'' _Journal of speech, language, and hearing research : JSLHR_ , 03 2009.
  * [5] E. Shriberg, ``Preliminaries to a theory of speech disfluencies,'' 1994.
  * [6] H. Kawai and T. Toda, ``An evaluation of automatic phone segmentation for concatenative speech synthesis,'' _2004 IEEE International Conference on Acoustics, Speech, and Signal Processing_ , vol. 1, pp. I–677, 2004.
  * [7] N. J. Young and M. McGarrah, ``Forced alignment for nordic languages: Rapidly constructing a high-quality prototype,'' _Nordic Journal of Linguistics_ , p. 1–27, 2021.
  * [8] V. Zayats, T. Tran, R. A. Wright, C. Mansfield, and M. Ostendorf, ``Disfluencies and human speech transcription errors,'' in _Interspeech_ , 2019.
  * [9] A. Katsamanis, M. Black, P. Georgiou, L. Goldstein, and S. Narayanan, ``Sailalign: Robust long speech-text alignment,'' in _Proc. of Workshop on New Tools and Methods for Very-Large Scale Phonetics Research_ , 2011.
  * [10] P. J. Moreno, C. F. Joerg, J.-M. V. Thong, and O. Glickman, ``A recursive algorithm for the forced alignment of very long audio segments,'' _5th International Conference on Spoken Language Processing (ICSLP 1998)_ , 1998.
  * [11] T. J. Hazen, ``Automatic alignment and error correction of human generated transcripts for long speech recordings,'' in _Interspeech_ , 2006.
  * [12] Y. T. Yeung, K. H. Wong, and H. Meng, ``Improving automatic forced alignment for dysarthric speech transcription,'' in _Proc. Interspeech 2015_ , 2015, pp. 2991–2995.
  * [13] S. Alharbi, M. Hasan, A. J. H. Simons, S. Brumfitt, and P. D. Green, ``A lightly supervised approach to detect stuttering in children's speech,'' in _Interspeech_ , 2018.
  * [14] J. Proença, C. Lopes, M. Tjalve, A. Stolcke, S. Candeias, and F. Perdigão, ``Mispronunciation detection in children's reading of sentences,'' _IEEE/ACM Transactions on Audio, Speech, and Language Processing_ , vol. 26, no. 7, pp. 1207–1219, 2018.
  * [15] J. Duchateau, Y. O. Kong, L. Cleuren, L. Latacz, J. Roelens, A. Samir, K. Demuynck, P. Ghesquière, W. Verhelst, and H. V. hamme, ``Developing a reading tutor: Design and evaluation of dedicated speech recognition and synthesis modules,'' _Speech Communication_ , vol. 51, no. 10, pp. 985–994, 2009, spoken Language Technology for Education. [Online]. Available: https://www.sciencedirect.com/science/article/pii/S0167639309000697
  * [16] J. Zhu, C. Zhang, and D. Jurgens, ``Phone-to-audio alignment without text: A semi-supervised approach,'' _ICASSP 2022 - 2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , pp. 8167–8171, 2021.
  * [17] K. Schulze-Forster, C. S. J. Doire, G. Richard, and R. Badeau, ``Joint phoneme alignment and text-informed speech separation on highly corrupted speech,'' in _ICASSP 2020 - 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , 2020, pp. 7274–7278.
  * [18] Y. Teytaut and A. Roebel, ``Phoneme-to-Audio Alignment with Recurrent Neural Networks for Speaking and Singing Voice,'' in _Proceedings of Interspeech 2021_ , ser. Proceedings of Interspeech 2021, International Speech Communication Association. Brno, Czech Republic: International Speech Communication Association, Aug. 2021, pp. 61–65. [Online]. Available: https://hal.science/hal-03552964
  * [19] M. C. Kelley and B. V. Tucker, ``A comparison of input types to a deep neural network-based forced aligner,'' in _Interspeech_ , 2018.
  * [20] Y. TEYTAUT, B. Bouvier, and A. Roebel, ``A study on constraining Connectionist Temporal Classification for temporal audio alignment,'' in _Proc. Interspeech 2022_ , 2022, pp. 5015–5019.
  * [21] Y. Miao, M. A. Gowayyed, and F. Metze, ``Eesen: End-to-end speech recognition using deep rnn models and wfst-based decoding,'' _2015 IEEE Workshop on Automatic Speech Recognition and Understanding (ASRU)_ , pp. 167–174, 2015.
  * [22] A. Laptev, S. Majumdar, and B. Ginsburg, ``Ctc variations through new wfst topologies,'' in _Interspeech_ , 2021.
  * [23] V. Pratap, A. Hannun, G. Synnaeve, and R. Collobert, ``Star temporal classification: Sequence classification with partially labeled data,'' _arXiv preprint arXiv:2201.12208_ , 2022.
  * [24] V. Peddinti, V. Manohar, Y. Wang, D. Povey, and S. Khudanpur, ``Far-field asr without parallel data,'' 09 2016, pp. 1996–2000.
  * [25] A. Baevski, H. Zhou, A. Mohamed, and M. Auli, ``Wav2vec 2.0: A framework for self-supervised learning of speech representations,'' in _Proceedings of the 34th International Conference on Neural Information Processing Systems_ , ser. NIPS'20. Red Hook, NY, USA: Curran Associates Inc., 2020.
  * [26] J. Proença, C. Lopes, M. Tjalve, A. Stolcke, S. Candeias, and F. Perdigão, ``Automatic evaluation of reading aloud performance in children,'' _Speech Communication_ , vol. 94, pp. 1–14, 2017. [Online]. Available: https://www.sciencedirect.com/science/article/pii/S0167639316303739
  * [27] P. Bell and S. Renals, ``A system for automatic alignment of broadcast media captions using weighted finite-state transducers,'' in _2015 IEEE Workshop on Automatic Speech Recognition and Understanding (ASRU)_ , 2015, pp. 675–680.
  * [28] J. Yang, L. Ondel, V. Manohar, and H. Hermansky, ``Towards automatic methods to detect errors in transcriptions of speech recordings,'' in _ICASSP 2019 \- 2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , 2019, pp. 3747–3751.
  * [29] T. Kourkounakis, A. Hajavi, and A. Etemad, ``Fluentnet: End-to-end detection of stuttered speech disfluencies with deep learning,'' _IEEE/ACM Transactions on Audio, Speech, and Language Processing_ , vol. 29, pp. 2986–2999, 2021.
  * [30] V. Panayotov, G. Chen, D. Povey, and S. Khudanpur, ``Librispeech: An asr corpus based on public domain audio books,'' in _2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , 2015, pp. 5206–5210.
  * [31] F. Kreuk, J. Keshet, and Y. Adi, ``Self-supervised contrastive learning for unsupervised phoneme segmentation,'' _arXiv preprint arXiv:2007.13465_ , 2020\.