#  : A Large-Scale Mined Corpus
of Multilingual Speech-to-Speech Translations

Paul-Ambroise Duquenne1,2, Equal contributions Hongyu Gong1,∗
Ning Dong1 Jingfei Du1
Ann Lee1 Vedanuj Goswani1 Changhan Wang1
Juan Pino1 Benoît Sagot2 Holger Schwenk1
1 Meta AI Research, 2 Inria
<EMAIL_ADDRESS>
###### Abstract

We present , a large-scale multilingual corpus of speech-to-speech
translations mined from real speech of European Parliament recordings. It
contains speech alignments in 136 language pairs with a total of 418 thousand
hours of speech. To evaluate the quality of this parallel speech, we train
bilingual speech-to-speech translation models on mined data only and establish
extensive baseline results on Europarl-ST, VoxPopuli and FLEURS test sets.
Enabled by the multilinguality of , we also explore multilingual speech-to-
speech translation, a topic which was addressed by few other works. We also
demonstrate that model pre-training and sparse scaling using Mixture-of-
Experts bring large gains to translation performance. The mined data and
models are freely available.

##  1 Introduction

Research has progressed in the area of speech-to-speech translation (S2ST)
with the goal of seamless communication among people who speak different
languages. Direct S2ST models attract increasing research interest, e.g. Jia
et al. (2019). Compared to conventional cascaded models, direct models do not
rely on intermediate text representations which make them applicable to the
translation of languages without a well-defined writing script. Moreover,
direct S2ST have the advantage of higher training and inference efficiency Lee
et al. (2022a).

Despite the benefits of direct approaches, model training is faced with the
major issue of data scarcity. Human labeled speech data is expensive to
create, there are very few data resources providing parallel speech, and the
data amount is quite limited. To mitigate the data scarcity, some works have
leveraged multitask learning Jia et al. (2019); Lee et al. (2022a), data
augmentation with speech variation Jia et al. (2019), or with synthesized
speech Jia et al. (2022a); Popuri et al. (2022), and knowledge from pre-
trained models Lee et al. (2022b); Popuri et al. (2022) such as HuBERT Hsu et
al. (2021), wav2vec 2.0 Baevski et al. (2020) and mBART Liu et al. (2020).

Recently, the multilingual speech/text sentence embedding space from Duquenne
et al. (2021) enabled the first speech mining results, aligning speech and
text in different languages. Using this mined data to train direct speech-to-
text and speech-to-speech translation systems can improve the performance of
such models Duquenne et al. (2021); Lee et al. (2022b). Finally, Duquenne et
al. (2022) showed that such multilingual and multimodal sentence embeddings
could be decoded into different languages and/or modalities in a zero-shot
way, which suggests that multilingual speech content is well encoded in these
fixed-size representations.

In this work, we trained speech encoders for 17 languages111Czech (cs), German
(de), English (en), Spanish (es), Estonian (et), Finnish (fi), French (fr),
Croatian (hr), Hungarian (hu), Italian (it), Lithuanian (lt), Dutch (nl),
Polish (pl), Portuguese (pt), Romanian (ro), Slovak (sk) and Slovenian (sl).
and mined speech-to-speech alignments for all possible language pairs. To the
best of our knowledge, is by far the largest freely available speech-to-speech
translation corpus, with 136 language directions and an average of 1,537 hours
of source speech in each direction for a total of 418 thousand hours. We
demonstrate that strong S2ST models can be trained with these mined data and
validate the good quality of the speech alignments across languages. We are
open-sourcing the mined data, the speech encoders used for mining,
multilingual HuBERT models in four language families for target unit
generation, language-specific vocoders for speech synthesis from discrete
units, and S2S models trained and presented in this work.222
https://github.com/facebookresearch/fairseq/tree/ust/examples/speech_matrix

##  2 Related Works

From bitext mining to speech mining. State-of-the-art machine translation
models are trained on labeled translation data, commonly called bitexts. In
addition to human-labeled data, automatic methods have been proposed to find
parallel sentences in a source and target language in monolingual resources.
This task is known as bitext mining. In order to decide if sentences should be
aligned, some works used document meta-information (Resnik, 1999), cross-
lingual document retrieval (Munteanu and Marcu, 2005) or machine translation
and information retrieval (Abdul-Rauf and Schwenk, 2009; Bouamor and Sajjad,
2018). More recent work use multilingual sentence embeddings to perform bitext
mining, calculating cosine similarity (Schwenk, 2018) or other margin-based
similarity (Artetxe and Schwenk, 2018; Yang et al., 2019a) in the embedding
space to decide which sentences should be aligned. A large variety of methods
has been explored to learn multilingual sentence embedding spaces (España-
Bonet et al., 2017; Schwenk and Douze, 2017; Artetxe and Schwenk, 2019; Yang
et al., 2019a; Reimers and Gurevych, 2019; Yang et al., 2019b; Feng et al.,
2020). Such methods enabled the creation of massively multilingual mined
corpora of bitexts like the CCMatrix project Schwenk et al. (2021a). These
bitexts were successfully used to train state-of-the-art machine translation
models, e.g. Fan et al. (2020); Wang et al. (2022). Finally, existing sentence
embedding spaces can be extended to new languages (Reimers and Gurevych, 2020;
Heffernan et al., 2022) or the speech modality Duquenne et al. (2021); Khurana
et al. (2022) with knowledge distillation, also called teacher-student
approach. These multilingual and multimodal sentence embeddings enabled to
perform large-scale speech-text mining, or speech-speech mining for a small
set of languages.

Speech-to-speech translation. Early works on speech-to-speech translation are
cascaded systems typically consisting of automatic speech recognition (ASR),
machine translation (MT) and text-to-speech synthesis (TTS) Nakamura et al.
(2006); Do et al. (2015). The reliance on intermediate text outputs makes
cascaded models unable to support unwritten languages. Moreover multiple
separate components make the training and inference inefficient. Given these
limitations, there has been a recent surge of research interest in direct
approaches to speech translation without the need of texts. Translatotron (Jia
et al., 2019) is the first end-to-end S2ST model built upon a sequence-to-
sequence architecture, which is trained to generate target spectrograms from
source speech using multitask learning. As an improved version, Translatotron2
Jia et al. (2022b) has the ability to preserve voice in translated speech. It
adopts two decoders: a linguistic decoder to predict phoneme and an acoustic
decoder to predict target spectrograms.

Another line of research replaces the target spectrograms in S2ST modeling
with discrete units which are learned from a large amount of unlabeled speech
Lee et al. (2022a, b). Discrete units are shown to better capture linguistic
content than spectrograms, making the translations more robust to speech
variations from different speakers or prosody Hsu et al. (2021).

Despite these progress on direct S2ST, direct approaches are faced with the
challenge of limited parallel speech. We will describe existing speech-to-
speech datasets in the next part. Data augmentation is a straightforward way
of increasing the data such as speech transformation Jia et al. (2019) and
speech synthesis via TTS Jia et al. (2022a). Multitask learning is a commonly
used technique to better train the model on small amount of labeled data Jia
et al. (2022b); Lee et al. (2022a). Another effective method is to leverage
pre-trained components for model initialization.

Dataset | # of Languages | Avg. duration (h) | Source speech | Target speech
---|---|---|---|---
Fisher Post et al. (2014) | 2 | 127 | Telephone conversation | Synthetic
MaSS Boito et al. (2020) | 8 | 20 | Bible reading | Bible reading
VoxPopuli Wang et al. (2021a) | 15 | 82 | European Parliament speech | Simultaneous interpretation
CVSS (C+T) Jia et al. (2022c) | 21 | 181 | Read | Synthetic
FLEURS Conneau et al. (2022) | 102 | 12 | Read | Read
(ours) | 17 | 1537 | European Parliament speech | European Parliament speech

Table 1: A comparison of existing speech-to-speech datasets.

Speech translation corpora. The Fisher dataset, a collection of approximately
$170$ hours of telephone conversations in Spanish Post et al. (2014), is
commonly used as training data for Spanish-English S2ST, However it does not
have parallel English speech. Previous works generate synthesized English
speech from English text translations provided by Fisher. Another S2S dataset
containing synthesized speech is CVSS which covers parallel S2ST translations
from $21$ languages into English. It is derived from Common Voice Ardila et
al. (2020) and CoVoST 2 Wang et al. (2021b), and synthesizes speech from
translated texts via TTS models. The release of VoxPopuli dataset provided the
largest S2S translations in real speech so far Wang et al. (2021a). It covers
pairwise speech-to-speech translations among $15$ languages, and each
direction has less than $500$ hours of speech. In another initiative named
FLEURS, the text-to-text evaluation data of the FLoRes-101 benchmark Goyal et
al. (2022) was extended to the speech modality. Supporting $102$ languages,
FLEURS has a larger language coverage than VoxPopuli, but it contains speech
of only around $12$ hours per language and it is intended to be used as
$N$-way parallel test set.

In this work, we present , a large-scale multilingual speech-to-speech corpus
mined from VoxPopuli Wang et al. (2021a). It contains speech alignments in
$136{}$ language pairs with an average of $1,537$-hour source speech per
direction. The main characteristics of these speech corpora are summarized in
Table 1.

##  3 Speech-to-Speech Mining

The mining approach used in this work is built upon the core idea of encoding
multilingual speech utterances into a shared embedding space. Speech encoders
project utterances with similar semantic content to fixed-size
representations, the resulting embedding vectors are close in the embedding
space regardless of their languages. The closeness of speech embeddings thus
reflects the similarity of speech content, which serves as the speech
alignment score in the mining process. In this section, we discuss speech
encoders and speech alignment mining.

###  3.1 Speech Encoders

We followed the teacher-student approach introduced in Duquenne et al. (2021)
and trained speech encoders with the supervision of the multilingual LASER
text encoder Schwenk et al. (2021b). The LASER text space has proven to have
interesting semantic properties for mining purposes Schwenk et al. (2021a);
NLLB Team et al. (2022). In order to have similar semantic properties in the
multilingual speech embedding space, the LASER text encoder is used as the
teacher to train speech encoders. Transcriptions or written translation of the
audio utterances are encoded with LASER text encoder as target vectors for
speech encoders. During training, we minimize the cosine loss between fixed-
size representations output by speech encoders, and the outputs of LASER text
encoder (which weights are fixed during training). Speech encoders are
initialized with the 2B parameter XLS-R model Babu et al. (2021), which was
pre-trained on nearly half a million hours of publicly available speech audio
in 128 languages. Following Duquenne et al. (2022), the fixed-size
representation for speech are obtained with max-pooling of the encoder outputs
which appeared to work better compared to other pooling methods. We summarize
the architecture of the speech encoder training in Figure 1.

Figure 1: Architecture of speech encoders training.

Sim Search | cs | de | en | es | et | fi | fr | hr | hu | it | lt | nl | pl | pt | ro | sk | sl
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---
# test sentences | 1k | 1.7k | 1.5k | 1.4k | 47 | 0.4k | 1.5k | 0.3k | 1k | 1k | 39 | 1k | 1.6k | — | 1.3k | 0.6k | 0.3k
Audio vs. transcriptions | 0.6 | 1.0 | 0.2 | 0.7 | 0.0 | 0.7 | 0.5 | 0.3 | 1.1 | 4.9 | 0.0 | 0.8 | 0.9 | — | 0.9 | 0.7 | 3.1

Table 2: Similarity search error rates (in %) on VoxPopuli ASR test set.

Duquenne et al. (2021) presented an ablation study on different text teacher
choices: using transcription embeddings as target, written translation
embeddings as target, or both. Their conclusion was that using both
transcriptions and written translations embeddings as target yield the best
similarity search results. We used various publicly available ASR data sets
which cover our languages to train the speech encoders, like CoVoST 2 Wang et
al. (2020, 2021b), Common Voice (citation), Europarl Ardila et al. (2020),
mTedx Salesky et al. (2021), Must-C Di Gangi et al. (2019) or VoxPopuli Wang
et al. (2021a), as well as speech translation data from the foreign languages
into English and from English into German. We removed training samples where
the transcription or the written translation consisted of multiple sentences,
as LASER has been trained on single sentences only. For better training
efficiency, we trained speech encoders for each language family instead of
each language. The language grouping is provided in Table 6. To better handle
imbalanced training data, we sample the training data from different languages
with the same method and hyper-parameter as Duquenne et al. (2021). For
English (en), Slovenian (sl), Lithuanian (lt) and Dutch (nl), we also trained
separate monolingual speech encoders that appeared to have lower valid cosine
loss compared to multilingual speech encoders, and these ones were used for
mining.

###  3.2 Evaluation of speech encoders

Similarity search is frequently used to evaluate multilingual text encoders,
e.g. (Artetxe and Schwenk, 2018; Feng et al., 2020; Heffernan et al., 2022).
The underlying approach is to first calculate the embeddings of all source and
reference sentences in a test set. Then, for each source sentence, the closest
embedding among all reference sentences is searched, if it is not the expected
one, an error is counted. Following Duquenne et al. (2021), we extend
similarity search to the multimodal setting and use the following score to
measure similarity between the source audio, and the target transcriptions or
translations:

$\displaystyle sim(x,y)$ $\displaystyle=$ $\displaystyle cos(x,y)$
$\displaystyle-\left(\sum\limits_{z\in
NN_{k}(x)}\cfrac{cos(x,z)}{2k}+\sum\limits_{z\in
NN_{k}(y)}\cfrac{cos(y,z)}{2k}\right)$

where $x$ and $y$ are the source and target embeddings, and $NN_{k}(x)$
denotes the $k$ nearest neighbors of $x$. We used $k=4$. We evaluated
similarity search of audio against transcriptions on VoxPopuli ASR test set in
Table 2, which is our target domain as we plan to mine unlabeled speech from
VoxPopuli (see subsection 3.3). We also evaluated similarity search of audio
against written translations or transcriptions on CoVoST 2 test set in Table
3, in order to compare with previous work for German, English, Spanish and
French (details can be found in the Appendix). Finally, we report text-to-text
similarity search using the LASER text encoder as lower bound for the speech
translation similarity search error rate since we use gold transcriptions to
search against written translations. We report error rates (in %) that
correspond to the percentage of audio utterances wrongly matched with text
transcripts from the same test set. We notice that error rates are very low
for all languages (below 5% and around 1 or 2% for most languages), which is
an initial validation of the quality if the speech encoders before large-scale
mining.

| de | en | es | et | fr | it | nl | pt | sl
---|---|---|---|---|---|---|---|---|---
# test sentences | 14k | 16k | 13k | 2k | 15k | 9k | 2k | 4k | 0.4k
Audio |
vs. transcriptions | 1.4 | 2.9 | 0.4 | 0.1 | 0.5 | 0.5 | 1.0 | 1.1 | 1.7
vs. en translations | 3.3 | — | 1.3 | 1.0 | 1.5 | 1.7 | 4.4 | 1.9 | 4.4
Text transcription |
vs. en translations | 2.0 | — | 1.0 | 0.1 | 1.0 | 1.3 | 2.4 | 0.7 | 0.8
|  |  |  |  |  |  |  |  |

Table 3: Similarity search error rates (in %) on CoVoST 2 test set.

###  3.3 Large-scale speech mining

We used VoxPopuli as our source of unlabeled speech for the 17 languages of
focus. We downloaded the full unsegmented parliament session recordings from
VoxPopuli github repository.333https://github.com/facebookresearch/voxpopuli
We present in Table 4 the number of hours of unlabeled speech for each
language.

In principle, performing speech-to-speech or speech-to-text mining can be done
with exactly the same pipeline than text-to-text mining. The calculation of
the embeddings uses of course different encoders, but the rest of the pipeline
is exactly the same. We follow the global mining approach as described in
Schwenk et al. (2021a) and compare all segments in the source language with
all segments in the target language. Similarity scores are calculated in both
directions using the margin as described in subsection 3.2 considering $k=16$
neighbors. Segments are considered to be parallel if the margin score exceeds
a threshold, we use 1.06 if not specified otherwise. The reader is referred to
Schwenk et al. (2021a) for a detailed description of the generic mining
pipeline.

Lang | Hours | Lang | Hours | Lang | Hours
---|---|---|---|---|---
cs | 18.7k | fr | 22.8k | pl | 21.2k
de | 23.2k | hr | 8.1k | pt | 17.5k
en | 24.1k | hu | 17.7k | ro | 17.9k
es | 21.4k | it | 21.9k | sk | 12.1k
et | 10.6k | lt | 14.4k | sl | 11.3k
fi | 14.2k | nl | 19.0k |  |
|  |  |  |  |
Table 4: Number of hours of raw unlabeled speech from VoxPopuli by language.

There is however one important difference when processing speech: it is not
straight-forward to segment the audio signal into parts which have the optimal
granularity for mining. When aligning texts, the monolingual data is usually
segmented into sentences, since a sentence is a well defined semantic concept
and there are reliable algorithms to perform this sentence segmentation. The
VoxPopuli recordings have a rather long duration, e.g. one hour and a half in
average for English. We apply Voice Activity Detection (VAD) using Silero-
VAD444https://github.com/snakers4/silero-vad which supports over 100
languages. The resulting segments do not necessarily correspond to sentences.
On one hand, there may be a silence in the middle of a sentence, e.g. a
hesitation. And on the other hand, two sentences may follow each other without
a long silence separating them. We follow the “over segmentation approach”
outlined in Duquenne et al. (2021): several possible segments are created and
we let the mining algorithm decide which ones match best. Initial experiments
suggest that sentences shorter than 1 sec or longer than 20 sec are unlikely
to be aligned and were excluded.

After mining, the resulting speech alignments may have overlap as we over-
segmented the unlabeled speech. Duquenne et al. (2021) introduced a post-
processing method to remove overlaps between mined speech segments on the
source speech side. We relax a little bit the post-processing of the mined
data, allowing for some overlap between mined speech segments: for two audio
segments that overlap on the source side, if the overlap represent more than
20% of the first segment and of the second segment, we discard the alignment
with the lowest mining score. We did an ablation study on different percentage
thresholds for one low resource, one mid-resource and one high-resource pair
and found that 20% was the best threshold for all settings.

We report the statistics of the mined speech-to-speech translation pairs in
Table 5, with a mining score threshold of $1.06$. The mined data totals 418k
hours of parallel speech with an average of 1,537 hours of source speech on
all translation directions. While some high resource languages like English
(en), Spanish (es) or French (fr) can reach up to 5k hours of aligned speech
with other spoken languages; lower resource languages such as Estonian (et),
Croatian (hr), Slovenian (sl) and Lithuanian (lt) obtain much less aligned
speech, with only a few hours of aligned speech for Lithuanian.

We also performed mining of the source speech in sixteen languages against
more than twenty billion English sentences from Common Crawl. This yielded
between 827 and 3966 hours of speech-text alignments (see the last column of
Table 5). Training and evaluation of speech-to-text translation is left for
future research. An important advantage of our teacher-student approach is
that all speech and text encoders are derived from the same LASER teacher and
are mutually compatible. This enables us to perform speech-to-text mining for
many more languages (NLLB Team et al., 2022).

| Speech targets | Text
---|---|---
Src/Tgt | cs | de | en | es | et | fi | fr | hr | hu | it | lt | nl | pl | pt | ro | sk | sl | en
cs | - | 2381 | 3208 | 2290 | 952 | 1312 | 2476 | 726 | 1396 | 2410 | 84 | 2377 | 2516 | 1867 | 1190 | 2146 | 452 | 2528
de | 2386 | - | 4734 | 3113 | 901 | 1477 | 3536 | 498 | 1871 | 3476 | 41 | 3384 | 2632 | 2250 | 1281 | 1646 | 361 | 3073
en | 3172 | 4676 | - | 4715 | 1585 | 2169 | 5178 | 824 | 2266 | 4897 | 82 | 4422 | 3583 | 3572 | 2258 | 2306 | 586 | -
es | 2240 | 3041 | 4708 | - | 862 | 1373 | 4446 | 528 | 1599 | 4418 | 47 | 3067 | 2646 | 3484 | 1857 | 1603 | 308 | 3966
et | 943 | 892 | 1593 | 877 | - | 1201 | 934 | 265 | 1119 | 1019 | 39 | 1055 | 949 | 721 | 419 | 780 | 196 | 1578
fi | 1296 | 1463 | 2180 | 1393 | 1197 | - | 1449 | 306 | 1473 | 1599 | 47 | 1654 | 1350 | 1128 | 621 | 977 | 260 | 1969
fr | 2424 | 3457 | 5171 | 4455 | 923 | 1435 | - | 560 | 1711 | 4618 | 50 | 3273 | 2822 | 3384 | 1991 | 1657 | 326 | 3966
hr | 736 | 507 | 854 | 553 | 273 | 317 | 588 | - | 328 | 615 | 24 | 546 | 660 | 433 | 277 | 586 | 136 | 1311
hu | 1417 | 1897 | 2346 | 1672 | 1140 | 1507 | 1787 | 328 | - | 1855 | 68 | 1839 | 1566 | 1315 | 808 | 1064 | 311 | 2301
it | 2404 | 3460 | 4948 | 4500 | 1028 | 1614 | 4700 | 607 | 1823 | - | 103 | 3414 | 2848 | 3421 | 1995 | 1656 | 474 | 2891
lt | 78 | 38 | 79 | 46 | 37 | 44 | 48 | 21 | 61 | 95 | - | 77 | 80 | 35 | 18 | 64 | 6 | 827
nl | 2322 | 3305 | 4396 | 3066 | 1040 | 1633 | 3269 | 521 | 1768 | 3355 | 80 | - | 2459 | 2399 | 1352 | 1646 | 458 | 2708
pl | 2530 | 2646 | 3662 | 2735 | 967 | 1378 | 2913 | 656 | 1554 | 2883 | 88 | 2540 | - | 2121 | 1301 | 1892 | 431 | 2871
pt | 1849 | 2224 | 3606 | 3525 | 722 | 1131 | 3421 | 421 | 1279 | 3403 | 37 | 2436 | 2087 | - | 1579 | 1358 | 247 | 3540
ro | 1187 | 1275 | 2290 | 1894 | 423 | 627 | 2024 | 271 | 789 | 1996 | 19 | 1384 | 1288 | 1592 | - | 870 | 125 | 2784
sk | 2127 | 1628 | 2329 | 1631 | 781 | 982 | 1685 | 574 | 1038 | 1650 | 69 | 1676 | 1869 | 1361 | 867 | - | 370 | 2090
sl | 436 | 350 | 579 | 307 | 192 | 254 | 324 | 128 | 295 | 461 | 6 | 454 | 413 | 241 | 121 | 359 | - | 1267
|  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |

Table 5: Duration statistics (hours of source speech) of aligned speech-to-
speech data between each pair of $17{}$ languages (for mining threshold of
$1.06$). The last column provides statistics for alignments of source speech
against 21.5 billion sentences of English texts.

###  3.4 Evaluation Data

Besides the speech-to-speech data mined as the train set, we also leverage
labeled public speech datasets as the evaluation sets.

Test set. In our experiments, we derive test sets in speech translation from
three public corpora, evaluating translation models trained on mined data
across different domains.

  * •

Europarl-ST (EPST) Iranzo-Sánchez et al. (2020). It is a multilingual speech-
to-text translation corpus built from recordings of debates from the European
Parliament, containing $72$ translation directions in $9$ languages.555en, fr,
de, it, es, pt, pl, ro and nl

  * •

VoxPopuli Wang et al. (2021a). S2S data, as part of VoxPopuli release,
provides aligned source and target speech together with source transcription.
We prepare the speech-to-text data with target speech and source transcription
as our test set. To ensure that there is no overlap between the mined data and
VoxPopuli test sets, we need to remove speech from mined alignments which are
from the same session as test samples. In order to keep as much mined data as
possible, we use VoxPopuli test set only when a language direction is not
covered by EPST considering their domain similarity. Moreover, similarity
scores are provided to indicate the quality of VoxPopuli samples. To choose
high-quality data, we sort all sessions in the VoxPopuli S2S data in a
decreasing order of the average similarity score of their samples. We keep
adding samples from highly ranked sessions to the test set until the test size
hits $1000$.

  * •

FLEURS Conneau et al. (2022). Built upon N-way text translations from FLoRes
Goyal et al. (2022), FLEURS provides speech for these texts and creates
speech-to-speech data covering all mined directions. We take its source speech
and target texts as the test data. In the case where multiple utterances
correspond to one piece of source text, we generate one test pair for each
source utterance respectively. FLEURS texts are from English Wikipedia, which
is a different domain from VoxPopuli and EPST.

Valid set. Valid sets are prepared using VoxPopuli and FLEURS data in a
similar way as test sets. For VoxPopuli, we extract a valid set of about
$1000$ samples by adding data from highly scored sessions which are not in the
test set. FLEURS valid set is derived from its valid samples. We prepare
speech-to-unit data from these selected valid samples by transforming the
target speech into target units for speech-to-unit modeling which will be
discussed in section 4.

##  4 Experiments & Results

To evaluate the quality of the mined data, we trained S2ST models on data and
report the translation performance. We hope that these results will serve as
baselines for future studies in speech translation.

###  4.1 Experimental Setup

Figure 2: A Pipeline of Speech-to-Speech Translation and Evaluation.

The training and evaluation pipeline of speech-to-speech translation is shown
in Figure 2. Recent progress in speech-to-speech translation modeling suggests
to discretize the target speech waveform into a unit sequence, relieving
models from the complexity of predicting continuous waveform values. We borrow
the idea of training speech-to-unit (S2U) model where units are pre-generated
from target speech with a pre-trained HuBERT model Lee et al. (2022a). During
S2U training, models are periodically evaluated on the valid set of speech-to-
unit samples, and the best checkpoint with the lowest valid loss is saved for
model inference.

When it comes to inference, speech could be synthesized from the predicted
units with a vocoder, as the output of the S2S pipeline. It is then
transcribed into texts by an off-the-shelf ASR model. The BLEU score is
calculated by comparing the transcriptions against the ground truth target
texts, which serves as the quantitative metric of mined data quality. We note
that the ASR BLEU score is not a perfect metric for data quality, as it is
unavoidably affected by the quality of ASR models. Next we discuss each module
of the pipeline.

Speech-to-Unit. The S2U model takes the source speech and predicts a sequence
of target units. It typically has an encoder-decoder architecture, where the
encoder consists of convolutional and Transformer encoder layers, and the
decoder is a Transformer decoder. We have experimented with different model
variants, and discuss bilingual and multilingual training in section 5 and
section 6 respectively.

HuBERT. We reuse the same HuBERT model and $k$-means clusters for English,
Spanish and French as in Lee et al. (2022b) for a fair comparison with
existing results. We also train HuBERT models for each language family as
shown in Table 6 to cover other languages in . We collect unlabeled VoxPopuli
speech for all languages of the same family as the training data. Each HuBERT
model is trained for three iterations, and more details of HuBERT training can
be found in Appendix A.2. We select the best layer for speech feature
extraction and the best label size (i.e., the number of $k$-means clusters)
using speech resynthesis together with the vocoder, which is discussed in
Appendix A.4. With the optimal layer and label size decided, we could generate
the target unit sequence for the given source speech.

Family | Languages
---|---
Romance | es, fr, it, pt, ro
Slavic | cs, pl, sk, sl, hr, lt
Germanic | en, de, nl
Uralic | fi, et, hu
Table 6: Language families in VoxPopuli data.

Vocoder. Unit-based HiFi-GAN vocoders are trained to synthesize speech from
unit sequence Polyak et al. (2021). In our experiments, vocoders are
separately trained from S2U model. We train vocoders on three datasets:

  * •

CSS10 Park and Mulc (2019). CSS10 is a single-speaker corpus which we use to
train vocoders in German, Finnish, Hungarian and Dutch.

  * •

VoxPopuli Wang et al. (2021a). Given the ASR data with speaker information, we
sort speakers based on their speech duration, and keep adding the top speakers
until the speech is more than $20$ hours.

  * •

Common Voice Ardila et al. (2020). Two languages, Portuguese and Estonian, are
not covered by the two corpora above, and thus we turn to Common Voice. Again,
we select top speakers and prepare 12-hour and 10-hour speech for the vocoder
training in Portuguese and Estonian respectively.

We applied a denoiser666https://github.com/facebookresearch/denoiser Defossez
et al. (2020) to the speech of VoxPopuli and Common Voice as the speech
preprocessing to increase signal-to-noise ratio (SNR) given that they are
noisier than CSS10 audios. Then we prepare vocoder labels with HuBERT models
generating $k$-means cluster labels for each utterance. Single-speaker
vocoders are trained in CSS10, and languages from VoxPopuli and Common Voice
have multi-speaker vocoders where speaker embeddings are learned. During
inference, we select the speaker with the longest speech duration to
synthesize speech from predicted unit sequences, who has the most data for the
vocoder to learn good speaker embeddings.

ASR. We use off-the-shelf ASR models to transcribe the speech generated by
vocoders, and details about the ASR models and their benchmark results are
provided in the subsection A.3.

##  5 Bilingual Speech-to-Speech Baselines

In this part, we discuss the bilingual S2S models trained in each of $272$
language directions in . The architecture of Textless model is used for
bilingual translation in our experiments Lee et al. (2022a). A Textless model
consists of a speech encoder with $2$ convolution layers and $12$ Transformer
encoder layers. Transformer layer has the embedding dimension of $512$ and the
forward dimension of $2048$. It has two unit decoders with $6$ and $2$
Transformer decoder layers for target and source unit prediction respectively.
The target unit decoder has the embedding dimension of $512$ and the forward
dimension of $2048$, and the source unit decoder’s dimensions are $256$ and
$2048$.

Training. For a given direction, we extract units for source and target speech
with their corresponding HuBERT models Hsu et al. (2021). Taking source
speech, the model is trained to predict target unit sequence with cross-
entropy loss as well as to reconstruct source units as an auxiliary task.

Figure 3: Bilingual S2S BLEU by mined data at different thresholds.

For the sake of good translation performance and training efficiency, it is
important to set a reasonable threshold to mined data and select samples above
the threshold as the train set. We performed an analysis of translation
performance varying with thresholds from $1.06$ to $1.09$ on three language
pairs: es-en, ro-en and hr-en. Figure 3 shows the threshold, the corresponding
speech data size and BLEU score.

For low-resource directions such as hr-en, it is best to include all the mined
data. For high-resource directions, es-en and ro-en, the optimal amount of
mined data is around $1$k hours and it does not bring further gains to
increase the data size. Given these observations, we choose the highest
threshold that keeps the source speech duration in mined data more than $1$k
hour for each direction. For example, we use a threshold of $1.09$ for es-en
and of $1.06$ for hr-en.

Comparison with existing results. Since we adopt the same model as the
previous work Lee et al. (2022a) and the difference only lies in the train
set, it is straightforward to compare with its reported S2S translation
results. Table 7 shows the results of S2ST models which are trained on our
mined data compared to VoxPopuli S2S data in each of four language directions:
es-en, fr-en, en-es and en-fr. The threshold of mined data is set as $1.09$ to
these four directions, yielding an average of $1,436$-hour train set. Compared
with $480$-hour labeled speech from VoxPopuli, achieves an an average
improvement of 5.4 BLEU, indicating the good quality and usefulness of the
mined data.

Train set |  | Es-En | Fr-En | En-Es | En-Fr
---|---|---|---|---|---
VoxPopuli S2S | Hours | 532 | 523 | 415 | 451
BLEU | 13.1 | 15.4 | 16.4 | 15.8
($t=1.09$) | Hours | 1,353 | 1,507 | 1,366 | 1,518
BLEU | 20.4 | 20.7 | 21.9 | 19.3

Table 7: BLEU scores on EPST test sets by S2ST models with different training
data.

###  5.1 Large-Scale Bilingual Evaluation

| cs | de | en | es | et | fi | fr | hr | hu | it | lt | nl | pl | pt | ro | sk | sl
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---
cs | \- / - | 12.9 / 2.0 | 22.7 / 4.2 | 16.7 / 4.6 | \- / 0.1 | 0.6 / 0.2 | 21.1 / 7.5 | 4.4 / 2.1 | 0.5 / 0.2 | 10.2 / 2.5 | 0.1 / 0.1 | 6.1 / 1.0 | 8.5 / 2.3 | \- / 2.8 | 4.3 / 1.4 | 16.9 / 3.5 | 3.0 / 1.7
de | 7.3 / 2.3 | \- / - |  16.3 / 8.3 |  11.7 / 3.8 | \- / 0.1 | 1.2 / 0.2 |  10.7 / 6.5 | 4.5 / 2.2 | 0.6 / 0.2 |  3.8 / 1.8 | 0.1 / 0.0 |  10.4 / 1.2 |  3.5 / 0.9 |  7.1 / 3.1 |  5.2 / 2.1 | 3.0 / 0.8 | 4.1 / 1.0
en | 8.2 / 2.7 |  10.1 / 2.7 | \- / - |  21.9 / 6.0 | \- / 0.7 | 1.9 / 0.6 |  19.2 / 10.4 | 8.4 / 2.4 | 1.1 / 0.3 |  11.5 / 3.6 | 0.3 / 0.1 |  15.1 / 3.8 |  8.2 / 1.3 |  11.8 / 5.1 |  7.6 / 2.0 | 5.7 / 1.2 | 5.5 / 1.2
es | 5.2 / 1.9 |  6.1 / 1.8 |  20.4 / 7.5 | \- / - | \- / 0.1 | 1.3 / 0.2 |  16.3 / 9.2 | 3.6 / 1.0 | 0.7 / 0.2 |  11.1 / 4.2 | 0.1 / 0.1 |  8.0 / 1.5 |  3.9 / 1.4 |  13.3 / 5.9 |  5.2 / 2.3 | 2.2 / 0.9 | 2.2 / 0.8
et | \- / 2.1 | \- / 0.7 | \- / 8.2 | \- / 3.0 | \- / - | \- / 0.7 | \- / 6.3 | \- / 1.0 | \- / 0.7 | \- / 2.3 | \- / 0.1 | \- / 1.5 | \- / 1.2 | \- / 1.7 | \- / 1.4 | \- / 0.4 | \- / 0.8
fi | 3.0 / 1.5 | 9.0 / 0.9 | 19.7 / 5.5 | 11.4 / 3.8 | \- / 0.5 | \- / - | 14.1 / 6.2 | 1.5 / 0.5 | 0.0 / 0.0 | 5.8 / 1.2 | 0.1 / 0.0 | 6.6 / 0.8 | 4.5 / 1.2 | \- / 2.0 | 4.4 / 1.1 | 1.7 / 0.7 | 1.6 / 0.7
fr | 5.4 / 1.5 |  6.3 / 2.1 |  20.7 / 9.8 |  18.4 / 7.6 | \- / 0.1 | 0.8 / 0.2 | \- / - | 5.4 / 1.7 | 0.7 / 0.2 |  10.2 / 3.1 | 0.1 / 0.1 |  8.4 / 1.3 |  4.8 / 1.5 |  13.4 / 5.8 |  5.6 / 2.4 | 1.6 / 0.6 | 1.5 / 0.6
hr | \- / 2.5 | \- / 0.9 | \- / 7.7 | \- / 3.1 | \- / 0.2 | \- / 0.1 | \- / 5.8 | \- / - | \- / 0.2 | \- / 1.1 | \- / 0.0 | \- / 0.9 | \- / 1.1 | \- / 2.0 | \- / 0.6 | \- / 0.9 | \- / 0.8
hu | 2.6 / 1.3 | 7.3 / 1.0 | 15.3 / 4.6 | 9.5 / 3.0 | \- / 0.1 | 0.7 / 0.2 | 13.8 / 5.7 | 1.9 / 0.7 | \- / - | 6.3 / 1.2 | 0.1 / 0.0 | 3.0 / 0.1 | 1.6 / 0.4 | \- / 2.3 | 2.4 / 0.9 | 0.9 / 0.2 | 1.2 / 0.3
it | 6.4 / 1.3 |  4.9 / 1.0 |  18.9 / 6.3 |  19.6 / 8.3 | \- / 0.1 | 0.4 / 0.1 |  15.3 / 11.3 | 5.2 / 1.3 | 0.7 / 0.2 | \- / - | 0.1 / 0.0 |  6.5 / 0.9 |  3.6 / 1.1 |  12.4 / 5.6 |  3.7 / 1.9 | 2.1 / 0.4 | 2.8 / 0.6
lt | 0.2 / 0.1 | 0.0 / 0.0 | 3.1 / 0.9 | 0.8 / 0.2 | \- / 0.0 | 0.0 / 0.0 | 0.7 / 0.2 | 0.1 / 0.0 | 0.0 / 0.0 | 0.6 / 0.4 | \- / - | 0.7 / 0.1 | 0.1 / 0.0 | \- / 0.0 | 0.0 / 0.0 | 0.0 / 0.0 | 0.1 / 0.0
nl | 3.5 / 1.4 |  8.1 / 3.1 |  18.0 / 5.7 |  13.2 / 4.9 | \- / 0.2 | 0.5 / 0.2 |  13.0 / 7.5 | 3.3 / 1.8 | 0.4 / 0.2 |  5.2 / 1.7 | 0.1 / 0.0 | \- / - |  3.4 / 0.9 |  6.7 / 3.3 |  4.1 / 1.4 | 1.7 / 0.4 | 2.1 / 1.0
pl | 7.2 / 1.6 |  2.8 / 1.6 |  4.9 / 4.9 |  6.3 / 4.4 | \- / 0.1 | 1.0 / 0.2 |  5.5 / 5.4 | 4.5 / 1.2 | 0.5 / 0.1 |  5.8 / 1.5 | 0.2 / 0.0 |  1.6 / 0.3 | \- / - |  6.1 / 2.5 |  3.2 / 1.2 | 4.7 / 1.1 | 2.4 / 0.7
pt | \- / 1.2 |  4.7 / 1.0 |  21.2 / 6.1 |  23.2 / 8.7 | \- / 0.1 | \- / 0.3 |  18.1 / 11.1 | \- / 1.1 | \- / 0.1 |  4.4 / 1.1 | \- / 0.1 |  5.0 / 0.6 |  3.6 / 0.8 | \- / - |  4.4 / 1.5 | \- / 0.6 | \- / 0.6
ro | 4.6 / 1.9 |  6.5 / 2.2 |  22.6 / 7.8 |  20.1 / 7.0 | \- / 0.4 | 0.8 / 0.3 |  18.6 / 11.3 | 2.4 / 0.9 | 0.4 / 0.2 |  8.7 / 3.8 | 0.1 / 0.1 |  3.5 / 0.9 |  4.6 / 1.1 |  10.3 / 6.0 | \- / - | 2.3 / 0.7 | 0.7 / 0.2
sk | 28.2 / 9.1 | 10.7 / 2.1 | 21.4 / 5.5 | 15.5 / 5.1 | \- / 0.3 | 1.0 / 0.2 | 19.2 / 7.8 | 5.0 / 3.0 | 0.5 / 0.4 | 4.7 / 2.1 | 0.1 / 0.0 | 4.2 / 0.7 | 5.3 / 1.9 | \- / 2.3 | 4.4 / 1.9 | \- / - | 3.6 / 1.5
sl | 4.0 / 2.2 | 11.1 / 2.0 | 19.5 / 7.3 | 8.6 / 3.4 | \- / 0.2 | 0.8 / 0.3 | 13.2 / 4.5 | 4.8 / 1.1 | 0.4 / 0.1 | 6.0 / 1.2 | 0.1 / 0.0 | 4.5 / 1.0 | 6.7 / 1.2 | \- / 1.5 | 1.1 / 0.1 | 1.7 / 0.3 | \- / -
|  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |

Table 8: BLEU scores of bilingual S2S models on three test sets. The first
score is either on EPST or VoxPopuli data, and EPST score is underscored. The
second score is on FLEURS data.

A large-scale evaluation is launched covering $272$ mined languages
directions, and bilingual models are trained for each direction to establish
baseline results in speech-to-speech translation.

Table 8 summarizes performance of bilingual S2ST models on three test sets. In
each direction, the first BLEU score is for European Parliament domain, either
EPST or VoxPopuli set. EPST BLEU is underlined to be distinguished from
VoxPopuli BLEU. The second score is for Wikipedia domain, i.e., FLEURS test
data.

Bilingual results. Empirically we find that translations into high-resource
languages such as en, es and fr outperform those into low-resource languages
such as lt and sl based on the language resource sizes in Table 5. Another
observation is the performance difference across test domains, i.e., BLEU on
FLEURS is lower than that on EPST and VoxPopuli data likely because of the
domain mismatch between train ad test data.

It is also found that translation results are not symmetric for some language
pairs, for example, ro-en has a BLEU of $22.6$ while en-ro BLEU is only $7.6$
on EPST. Besides different complexity levels of target languages and test
sets, such asymmetry also results from the dependency of BLEU score on the
speech synthesis quality of the vocoder and transcription quality of the ASR
model. For languages whose vocoder and ASR models are not well trained, they
are likely to receive low BLEU scores. In this case, Romanian vocoder and ASR
are not as strong as English models as reflected by its higher word error rate
in speech resynthesis as reported in Appendix A.4.

##  6 Multilingual Speech-to-Speech Translation

Multilingual modeling has been explored for the tasks of language models and
machine translation, demonstrating knowledge transfer among languages.
However, to our best knowledge, there are very few studies of multilingual
speech-to-speech translation, partially due to the lack of multilingual
speech-to-speech resources. With the massively multilingual data we have
mined, we are able to explore multilingual S2ST training.

In this work, we focus on many-to-English translation, studying the
translation from $6$ Slavic languages to English in subsection 6.1 and the
translation from all $16$ languages in to English in subsection 6.2. English-
to-many or many-to-many translation are left to future work.

| Bilingual | Multilingual
---|---|---
| 70M | 260M | 70M | 260M | 424M
| EP / VP | FLEURS | EP / VP | FLEURS | EP / VP | FLEURS | EP / VP | FLEURS | EP / VP | FLEURS
cs | 22.7 | 4.2 | 24.7 | 11.2 | 19.7 | 2.3 | 27.5 | 13.7 | 25.3 | 10.2
hr | - | 7.7 | - | 4.6 | - | 3.1 | - | 12.8 | - | 9.2
lt | 3.1 | 0.9 | 0.2 | 0.0 | 2.8 | 0.3 | 14.7 | 4.8 | 10.7 | 3.3
pl | 4.9 | 4.9 | 17.6 | 7.7 | 14.4 | 1.9 | 19.9 | 9.5 | 16.4 | 6.9
sk | 21.4 | 5.5 | 24.4 | 11.0 | 18.9 | 4.1 | 27.2 | 15.4 | 24.9 | 11.1
sl | 19.5 | 7.3 | 16.9 | 4.7 | 14.6 | 3.1 | 22.9 | 10.7 | 21.0 | 7.6
avg | 14.3 | 5.1 | 16.8 | 6.5 | 14.1 | 2.5 | 22.4 | 11.2 | 19.7 | 8.1
|  |  |  |  |  |  |  |  |  |

Table 9: BLEU of Slavic-to-English multilingual Textless model across domains
(for EP / VP column, underlined scores are on EPST data, and others on
VoxPopuli data).

| Bilingual (1.2B) | Multilingual Dense(1.2B) | Multilingual MoE-GShard64 (4.3B)
---|---|---|---
| EP / VP | FLEURS | EP / VP | FLEURS | EP / VP | FLEURS
cs | 28.3 | 17.8 | 29.7 | 18.2 | 30.6 | 19.3
hr | - | 12.1 | - | 17.1 | - | 17.6
lt | 0.0 | 0.0 | 20.9 | 9.6 | 22.2 | 10.2
pl | 17.4 | 7.4 | 21.1 | 12.9 | 21.4 | 12.6
sk | 24.7 | 14.5 | 30.8 | 19.3 | 31.8 | 20.0
sl | 20.1 | 8.5 | 27.4 | 14.0 | 29.1 | 13.0
avg | 18.1 | 10.1 | 26.0 | 15.2 | 27.0 | 15.5
|  |  |  |  |  |

Table 10: BLEU of Slavic-to-English multilingual XM Transformer models in
different domains (for EP / VP column, underlined scores are on EPST data, and
others on VoxPopuli data).

Multilingual models in our experiments include

  * •

Textless model. The same model that we use for bilingual evaluation is reused
in the multilingual experiments. The bilingual Textless model has $70$M
parameters. Given the increased amount and diversity of multilingual data, we
increase the model size for larger model capacity, trying multilingual models
with $70$M, $260$M and $424$M parameters respectively.

  * •

XM Transformer. Inspired by the recent finding that crossmodal pre-training is
beneficial for speech translation Popuri et al. (2022), we introduce XM
Transformer to multilingual training, whose encoder is initialized from pre-
trained XLS-R model with 1B parameters Babu et al. (2021) and decoder is
initialized from a unit decoder pre-trained in an mBART style Popuri et al.
(2022). With multilingual speech-to-unit data, the model is further finetuned
to minimize the cross-entropy loss in unit prediction.

  * •

XM Transformer with Sparsity. Sparse modeling, in particular Mixture-of-
Experts (MoE), has been widely studied in multilingual machine translation as
an efficient approach to encourage knowledge transfer and mitigate
interference among languages. MoE increases the number of parameters of the
model in magnitude without sacrificing computation efficiency. In this work,
we experiment with two variants of sparse modeling, GShard Lepikhin et al.
and Base Layer Lewis et al. (2021).

GShard. GShard is a sparse scaling technique for transformer proposed in
Lepikhin et al. . We replace every other transformer layer with an MoE layer.
FFN layers in an MoE transformer layer are shared across experts. One GPU
could host one or more experts and a learned gating function is used to
determine which expert a token is routed to. Similar approaches have also been
adopted recently in the multilingual text translation area to address language
interference issues NLLB Team et al. (2022). We apply GShard architecture on
the decoder of XM Transformer, to expand the capacity without the need to
retrain larger unit mBART models. All expert weights are initialized with unit
mBART.

Base Layer. Base Layer is a variant of the sparse model proposed by Lewis et
al. (2021), which applies a balanced assignment to expert layers. It
formulates token-to-expert allocation as a linear assignment problem,
encouraging each expert to receive an equal number of tokens. The assignment
scheme improves efficiency by balancing compute loads, and simplifies training
without any new hyperparameters or auxiliary losses. In our experiments, we
add one additional Base Layer in the middle (the 7th layer) of decoder. The
weights of Base Layer experts are randomly initialized.

Appendix B provides more details about these multilingual model
configurations.

###  6.1 Slavic-to-English Translation

The six Slavic languages include Czech (cs), Croatian (hr), Lituanian (lt),
Polish (pl), Slovak (sk), and Slovenian (sl). In the multilingual setting,
mined data into English at the threshold of $1.06$ are combined from each
Slavic language as the train set. As for the evaluation of multilingual
models, we report ASR BLEU in each direction, respectively.

We first extend Textless model from the bilingual to multilingual setting. For
Textless models with different parameter sizes, their multilingual as well as
bilingual translation results are presented in Table 9.

Results. With the Textless model size fixed as $70$M, multilingual training
hurts the performance of most languages compared with bilingual training. This
is due to the insufficient model capacity, and the language interference is
reflected by an average of $-2.6$ BLEU in FLEURS. We increase model parameters
to $260$M in both bilingual and multilingual settings. With a larger model
capacity, bilingual models achieve gains in high-resource languages including
cs, pl and sk, while suffering from performance loss in low-resource
directions such as hr, lt and sl.

| Dense (1.2B) | MoE-GShard64 (4.3B) | Base Layer (1.7B)
---|---|---|---
| EP / VP | FLEURS | EP / VP | FLEURS | EP / VP | FLEURS
cs | 29.9 | 18.7 | 30.9 | 18.2 | 29.9 | 17.3
de | 18.8 | 19.0 | 19.3 | 20.3 | 19.4 | 19.5
es | 22.8 | 15.2 | 23.3 | 15.9 | 22.9 | 14.9
et | - | 16.7 | - | 16.7 | - | 16.4
fi | 26.8 | 14.1 | 28.2 | 14.0 | 28.5 | 13.9
fr | 23.5 | 18.3 | 24.1 | 18.9 | 23.4 | 18.2
hr | - | 16.6 | - | 16.8 | - | 16.3
hu | 20.2 | 12.0 | 21.3 | 12.5 | 20.5 | 12.1
it | 36.3 | 16.2 | 37.8 | 14.9 | 37.4 | 14.0
lt | 21.9 | 9.8 | 23.8 | 10.3 | 23.4 | 10.0
nl | 21.4 | 16.4 | 22.1 | 17.3 | 21.5 | 16.6
pl | 21.2 | 12.4 | 21.3 | 13.4 | 20.9 | 12.5
pt | 23.8 | 21.8 | 24.2 | 22.3 | 23.8 | 21.1
ro | 25.1 | 19.7 | 25.0 | 19.8 | 25.3 | 19.0
sk | 30.8 | 19.6 | 32.2 | 18.2 | 31.5 | 18.4
sl | 28.3 | 13.7 | 29.9 | 13.7 | 28.8 | 13.5
avg | 25.1 | 16.3 | 26.0 | 16.5 | 25.5 | 15.9
|  |  |  |  |  |

Table 11: BLEU of All-to-English multilingual models in different domains (for
EP / VP column, underlined scores are on EPST data, and others on VoxPopuli
data).

Given model sizes of $260$M, we observe consistent gains of multilingual
models over the bilingual models across different language directions and test
domains. An average gain of $5.6$ BLEU is achieved in EP/VP and the gain of
$4.7$ BLEU in FLEURS. It demonstrates the positive transfer enabled by
multilingual training. As the multilingual model size continues to increase to
$424$M, we don’t observe further gains likely due to the bottleneck of
training data amount.

Pre-training has been shown to be beneficial for speech-to-speech translation
Popuri et al. (2022). XM Transformer leveraging pre-trained modules is also
trained on Slavic-to-English data, and results are reported in Table 10.
Comparing against bilingual Textless model (70M), bilingual XM Transformer
outperforms it in all directions except lt-en. The gain in EP/VP is $3.8$ BLEU
on average, and a larger gain of $5.0$ BLEU is achieved in FLEURS.
Multilingual training brings further gains to XM Transformer with +$7.9$ and
+$5.1$ BLEU over bilingual training in EP/VP and FLEURS test set respectively.

Comparing against dense XM Transformer, GShard with 64 experts has +$1.0$ BLEU
gains on average over 5 directions on EP/VP, and +$0.3$ BLEU gains for FLEURS.
We believe it’s due to a phenomena mentioned in Zoph et al. (2022) that MoE
specializes in multilingual settings but not by language. GShard in our
setting brings larger improvements to in-domain test set.

Overall the best Slavic-to-English translation is achieved by XM Transformer
with GShard trained in multilingual setting. This demonstrates that pre-
training, model sparsity and multilinguality are of help to speech-to-speech
translation.

###  6.2 All-to-English Translation

We move forward to a larger-scale multilinguality by extending from Slavic
language family to all languages in . We adopt the best models in Slavic-to-
English translation, i.e., multilingual XM Transformer with both dense and
sparse architectures. In terms of sparse modeling, we try both GShard and Base
Layer in all-to-English translation.

Results. Compared with XM Transformer (1.2B) dense model, MoE-GShard64 (4.3B)
with the same forward computation time brings gains of +$0.9$ and +$0.2$ BLEU
to EP/VP and FLEURS respectively. Similar to our findings in Slavic-to-English
setting, increasing the capacity with sparse modeling benefits more on in-
domain (EP/VP) than out-of-domain FLEURS test set.

With the sparse architecture of XM Transformer with GShard, all-to-English
model shows +$0.6$ and -$0.4$ BLEU difference compared with Slavic-to-English
model on EP/VP and FLEURS respectively, averaged over Slavic languages.
Multilingual sparse model benefits from the additional in-domain data in other
languages when evaluated in EP/VP domain, while sees performance degradation
in out-of-domain data.

The other sparse variant, Base Layer (1.7B) performs comparably to the dense
XM Transformer, with an average of +$0.4$ BLEU in EP/VP test sets and -$0.4$
BLEU in FLEURS. The sparsity in Base Layer does not bring obvious gains to
all-to-English translation. This is likely because we only add one Base Layer
to the decoder with a small expert size. The number of increased model
parameters is only $0.5$B in Base Layer, while it is $3.1$B in GShard. As
suggested by Lewis et al. (2021), the Base Layer performance might improve
with more GPUs and a larger expert size.

##  7 Limitations

The HuBERT model is critical to the speech-to-speech translation performance
as its extracted units are used by both speech-to-unit model and vocoder. We
have not explored the optimal strategy of multilingual HuBERT training. One
research question is how to choose a group of languages that a multilingual
HuBERT model is trained on. For example, it is arguable whether Lithuanian
(lt) should be included in Slavic or Uralic family. Other questions could be
whether a larger HuBERT with more model capacity should be used and how we
should deal with language imbalance in multilingual training.

We provide benchmark results of bilingual speech translation with mined data
selected by heuristics. One of our future directions is to come up with a
better strategy of mined data selection to improve translation performance and
training efficiency.

As mentioned in our results analysis, the reported BLEU scores are heavily
dependent on the ASR quality, which may not reflect the speech translation
performance accurately. Future directions could be improving ASR quality or
exploring other evaluation metrics without reliance on ASR models.

##  8 Conclusion

In this paper, we introduce a large-scale multilingual speech-to-speech corpus
mined from VoxPopuli. It is the largest resource of speech alignments with a
coverage of $17{}$ languages. We perform an extensive evaluation of the mined
parallel speech, showing good quality of the speech alignments. Multilingual
speech-to-speech models can be efficiently trained on this corpus and we
suggest different methods, like sparse scaling using Mixture-of-Experts, to
further boost translation performance in the multilingual setting.

## References

  * Abdul-Rauf and Schwenk (2009) Sadaf Abdul-Rauf and Holger Schwenk. 2009.  On the Use of Comparable Corpora to Improve SMT performance.  In _EACL_ , pages 16–23.
  * Ardila et al. (2020) Rosana Ardila, Megan Branson, Kelly Davis, Michael Kohler, Josh Meyer, Michael Henretty, Reuben Morais, Lindsay Saunders, Francis M. Tyers, and Gregor Weber. 2020.  Common voice: A massively-multilingual speech corpus.  In _Proceedings of The 12th Language Resources and Evaluation Conference, LREC 2020, Marseille, France, May 11-16, 2020_ , pages 4218–4222. European Language Resources Association.
  * Artetxe and Schwenk (2018) Mikel Artetxe and Holger Schwenk. 2018.  Margin-based Parallel Corpus Mining with Multilingual Sentence Embeddings.  _https://arxiv.org/abs/1811.01136_.
  * Artetxe and Schwenk (2019) Mikel Artetxe and Holger Schwenk. 2019.  Massively multilingual sentence embeddings for zero-shot cross-lingual transfer and beyond.  _TACL_ , pages 597–610.
  * Babu et al. (2021) Arun Babu, Changhan Wang, Andros Tjandra, Kushal Lakhotia, Qiantong Xu, Naman Goyal, Kritika Singh, Patrick von Platen, Yatharth Saraf, Juan Pino, Alexei Baevski, Alexis Conneau, and Michael Auli. 2021.  XLS-R: self-supervised cross-lingual speech representation learning at scale.  _CoRR_ , abs/2111.09296.
  * Baevski et al. (2020) Alexei Baevski, Yuhao Zhou, Abdelrahman Mohamed, and Michael Auli. 2020.  wav2vec 2.0: A framework for self-supervised learning of speech representations.  In _Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual_.
  * Boito et al. (2020) Marcely Zanon Boito, William Havard, Mahault Garnerin, Éric Le Ferrand, and Laurent Besacier. 2020.  Mass: A large and clean multilingual corpus of sentence-aligned spoken utterances extracted from the bible.  In _Proceedings of The 12th Language Resources and Evaluation Conference, LREC 2020, Marseille, France, May 11-16, 2020_ , pages 6486–6493. European Language Resources Association.
  * Bouamor and Sajjad (2018) Houda Bouamor and Hassan Sajjad. 2018.  H2@BUCC18: Parallel Sentence Extraction from Comparable Corpora Using Multilingual Sentence Embeddings.  In _BUCC_.
  * Conneau et al. (2022) Alexis Conneau, Min Ma, Simran Khanuja, Yu Zhang, Vera Axelrod, Siddharth Dalmia, Jason Riesa, Clara Rivera, and Ankur Bapna. 2022.  FLEURS: few-shot learning evaluation of universal representations of speech.  _CoRR_ , abs/2205.12446.
  * Defossez et al. (2020) Alexandre Defossez, Gabriel Synnaeve, and Yossi Adi. 2020.  Real time speech enhancement in the waveform domain.  In _Interspeech_.
  * Di Gangi et al. (2019) Mattia A. Di Gangi, Roldano Cattoni, Luisa Bentivogli, Matteo Negri, and Marco Turchi. 2019.  MuST-C: a Multilingual Speech Translation Corpus.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 2012–2017, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Do et al. (2015) Quoc Truong Do, Sakriani Sakti, Graham Neubig, Tomoki Toda, and Satoshi Nakamura. 2015.  Improving translation of emphasis with pause prediction in speech-to-speech translation systems.  In _Proceedings of the 12th International Workshop on Spoken Language Translation: Papers, IWSLT 2015, Da Nang, Vietnam, December 3-4, 2015_.
  * Duquenne et al. (2022) Paul-Ambroise Duquenne, Hongyu Gong, Benoît Sagot, and Holger Schwenk. 2022.  T-modules: Translation modules for zero-shot cross-modal machine translation.  In _EMNLP_.
  * Duquenne et al. (2021) Paul-Ambroise Duquenne, Hongyu Gong, and Holger Schwenk. 2021.  Multimodal and multilingual embeddings for large-scale speech mining.  _Advances in Neural Information Processing Systems_ , 34:15748–15761.
  * España-Bonet et al. (2017) Cristina España-Bonet, Ádám Csaba Varga, Alberto Barrón-Cedeño, and Josef van Genabith. 2017.  An Empirical Analysis of NMT-Derived Interlingual Embeddings and their Use in Parallel Sentence Identification.  _IEEE Journal of Selected Topics in Signal Processing_ , pages 1340–1348.
  * Fan et al. (2020) Angela Fan, Shruti Bhosale, Holger Schwenk, Zhiyi Ma, Ahmed El-Kishky, Siddharth Goyal, Mandeep Baines, Onur Celebi, Guillaume Wenzek, Vishrav Chaudhary, Naman Goyal, Tom Birch, Vitaliy Liptchinsky, Sergey Edunov, Edouard Grave, Michael Auli, and Armand Joulin. 2020.  Beyond english-centric multilingual machine translation.  _JMLR_.
  * Feng et al. (2020) Fangxiaoyu Feng, Yinfei Yang, Daniel Cer, Naveen Arivazhagan, and Wei Wang. 2020\.  Language-agnostic bert sentence embedding.  _arXiv preprint arXiv:2007.01852_.
  * Goyal et al. (2022) Naman Goyal, Cynthia Gao, Vishrav Chaudhary, Peng-Jen Chen, Guillaume Wenzek, Da Ju, Sanjana Krishnan, Marc’Aurelio Ranzato, Francisco Guzmán, and Angela Fan. 2022.  The flores-101 evaluation benchmark for low-resource and multilingual machine translation.  _Trans. Assoc. Comput. Linguistics_ , 10:522–538.
  * Heffernan et al. (2022) Kevin Heffernan, Onur Çelebi, and Holger Schwenk. 2022.  Bitext mining using distilled sentence representations for low-resource languages.  In _EMNLP_.
  * Hsu et al. (2021) Wei-Ning Hsu, Benjamin Bolte, Yao-Hung Hubert Tsai, Kushal Lakhotia, Ruslan Salakhutdinov, and Abdelrahman Mohamed. 2021.  Hubert: Self-supervised speech representation learning by masked prediction of hidden units.  _IEEE ACM Trans. Audio Speech Lang. Process._ , 29:3451–3460.
  * Iranzo-Sánchez et al. (2020) Javier Iranzo-Sánchez, Joan Albert Silvestre-Cerdà, Javier Jorge, Nahuel Roselló, Adrià Giménez, Albert Sanchís, Jorge Civera, and Alfons Juan. 2020.  Europarl-st: A multilingual corpus for speech translation of parliamentary debates.  In _2020 IEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP 2020, Barcelona, Spain, May 4-8, 2020_ , pages 8229–8233. IEEE.
  * Jia et al. (2022a) Ye Jia, Yifan Ding, Ankur Bapna, Colin Cherry, Yu Zhang, Alexis Conneau, and Nobuyuki Morioka. 2022a.  Leveraging unsupervised and weakly-supervised data to improve direct speech-to-speech translation.  _CoRR_ , abs/2203.13339.
  * Jia et al. (2022b) Ye Jia, Michelle Tadmor Ramanovich, Tal Remez, and Roi Pomerantz. 2022b.  Translatotron 2: High-quality direct speech-to-speech translation with voice preservation.  In _International Conference on Machine Learning, ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA_ , volume 162 of _Proceedings of Machine Learning Research_ , pages 10120–10134. PMLR.
  * Jia et al. (2022c) Ye Jia, Michelle Tadmor Ramanovich, Quan Wang, and Heiga Zen. 2022c.  CVSS corpus and massively multilingual speech-to-speech translation.  _CoRR_ , abs/2201.03713.
  * Jia et al. (2019) Ye Jia, Ron J. Weiss, Fadi Biadsy, Wolfgang Macherey, Melvin Johnson, Zhifeng Chen, and Yonghui Wu. 2019.  Direct speech-to-speech translation with a sequence-to-sequence model.  In _Interspeech 2019, 20th Annual Conference of the International Speech Communication Association, Graz, Austria, 15-19 September 2019_ , pages 1123–1127. ISCA.
  * Khurana et al. (2022) Sameer Khurana, Antoine Laurent, and James Glass. 2022.  Samu-xlsr: Semantically-aligned multimodal utterance-level cross-lingual speech representation.  _arXiv preprint arXiv:2205.08180_.
  * Lee et al. (2022a) Ann Lee, Peng-Jen Chen, Changhan Wang, Jiatao Gu, Sravya Popuri, Xutai Ma, Adam Polyak, Yossi Adi, Qing He, Yun Tang, Juan Pino, and Wei-Ning Hsu. 2022a.  Direct speech-to-speech translation with discrete units.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2022, Dublin, Ireland, May 22-27, 2022_ , pages 3327–3339. Association for Computational Linguistics.
  * Lee et al. (2022b) Ann Lee, Hongyu Gong, Paul-Ambroise Duquenne, Holger Schwenk, Peng-Jen Chen, Changhan Wang, Sravya Popuri, Yossi Adi, Juan Pino, Jiatao Gu, and Wei-Ning Hsu. 2022b.  Textless speech-to-speech translation on real data.  In _Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL 2022, Seattle, WA, United States, July 10-15, 2022_ , pages 860–872. Association for Computational Linguistics.
  * (29) Dmitry Lepikhin, HyoukJoong Lee, Yuanzhong Xu, Dehao Chen, Orhan Firat, Yanping Huang, Maxim Krikun, Noam Shazeer, and Zhifeng Chen.  Gshard: Scaling giant models with conditional computation and automatic sharding.  In _9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021_.
  * Lewis et al. (2021) Mike Lewis, Shruti Bhosale, Tim Dettmers, Naman Goyal, and Luke Zettlemoyer. 2021\.  Base layers: Simplifying training of large, sparse models.  In _International Conference on Machine Learning_ , pages 6265–6274. PMLR.
  * Liu et al. (2020) Yinhan Liu, Jiatao Gu, Naman Goyal, Xian Li, Sergey Edunov, Marjan Ghazvininejad, Mike Lewis, and Luke Zettlemoyer. 2020.  Multilingual denoising pre-training for neural machine translation.  _Transactions of the Association for Computational Linguistics_ , 8:726–742.
  * Munteanu and Marcu (2005) Dragos Stefan Munteanu and Daniel Marcu. 2005.  Improving Machine Translation Performance by Exploiting Non-Parallel Corpora.  _Computational Linguistics_ , 31(4):477–504.
  * Nakamura et al. (2006) Satoshi Nakamura, Konstantin Markov, Hiromi Nakaiwa, Gen-ichiro Kikui, Hisashi Kawai, Takatoshi Jitsuhiro, Jinsong Zhang, Hirofumi Yamamoto, Eiichiro Sumita, and Seiichi Yamamoto. 2006.  The ATR multilingual speech-to-speech translation system.  _IEEE Trans. Speech Audio Process._ , 14(2):365–376.
  * NLLB Team et al. (2022) NLLB Team, Marta R. Costa-jussà, James Cross, Onur Çelebi, Maha Elbayad, Kenneth Heafield, Kevin Heffernan, Elahe Kalbassi, Janice Lam, Daniel Licht, Jean Maillard, Anna Sun, Skyler Wang, Guillaume Wenzek, Al Youngblood, Bapi Akula, Loic Barrault, Gabriel Mejia Gonzalez, Prangthip Hansanti, John Hoffman, Semarley Jarrett, Kaushik Ram Sadagopan, Dirk Rowe, Shannon Spruit, Chau Tran, Pierre Andrews, Necip Fazil Ayan, Shruti Bhosale, Sergey Edunov, Angela Fan, Cynthia Gao, Vedanuj Goswami, Francisco Guzmán, Philipp Koehn, Alexandre Mourachko, Christophe Ropers, Safiyyah Saleem, Holger Schwenk, and Jeff Wang. 2022.  No language left behind: Scaling human-centered machine translation.
  * Park and Mulc (2019) Kyubyong Park and Thomas Mulc. 2019.  CSS10: A collection of single speaker speech datasets for 10 languages.  In _Interspeech 2019, 20th Annual Conference of the International Speech Communication Association, Graz, Austria, 15-19 September 2019_ , pages 1566–1570. ISCA.
  * Polyak et al. (2021) Adam Polyak, Yossi Adi, Jade Copet, Eugene Kharitonov, Kushal Lakhotia, Wei-Ning Hsu, Abdelrahman Mohamed, and Emmanuel Dupoux. 2021.  Speech Resynthesis from Discrete Disentangled Self-Supervised Representations.  In _Proc. Interspeech 2021_.
  * Popuri et al. (2022) Sravya Popuri, Peng-Jen Chen, Changhan Wang, Juan Pino, Yossi Adi, Jiatao Gu, Wei-Ning Hsu, and Ann Lee. 2022.  Enhanced direct speech-to-speech translation using self-supervised pre-training and data augmentation.  _CoRR_ , abs/2204.02967.
  * Post et al. (2014) Matt Post, Gaurav Kumar, Adam Lopez, Damianos Karakos, Chris Callison-Burch, and Sanjeev Khudanpur. 2014.  Fisher and callhome spanish–english speech translation.  _LDC2014T23. Web Download. Philadelphia: Linguistic Data Consortium_.
  * Reimers and Gurevych (2019) Nils Reimers and Iryna Gurevych. 2019.  Sentence-BERT: Sentence embeddings using siamese bert-networks.  In _https://arxiv.org/abs/1908.10084_.
  * Reimers and Gurevych (2020) Nils Reimers and Iryna Gurevych. 2020.  Making monolingual sentence embeddings multilingual using knowledge distillation.  In _EMNLP_ , pages 4512–4525.
  * Resnik (1999) Philip Resnik. 1999.  Mining the Web for Bilingual Text.  In _ACL_.
  * Salesky et al. (2021) Elizabeth Salesky, Matthew Wiesner, Jacob Bremerman, Roldano Cattoni, Matteo Negri, Marco Turchi, Douglas W. Oard, and Matt Post. 2021.  The multilingual tedx corpus for speech recognition and translation.  In _Interspeech 2021, 22nd Annual Conference of the International Speech Communication Association, Brno, Czechia, 30 August - 3 September 2021_ , pages 3655–3659. ISCA.
  * Schwenk (2018) Holger Schwenk. 2018.  Filtering and mining parallel data in a joint multilingual space.  In _ACL_ , pages 228–234.
  * Schwenk and Douze (2017) Holger Schwenk and Matthijs Douze. 2017.  Learning joint multilingual sentence representations with neural machine translation.  In _ACL workshop on Representation Learning for NLP_.
  * Schwenk et al. (2021a) Holger Schwenk, Guillaume Wenzek, Sergey Edunov, Edouard Grave, Armand Joulin, and Angela Fan. 2021a.  CCMatrix: Mining billions of high-quality parallel sentences on the web.  In _ACL_ , pages 6490–6500.
  * Schwenk et al. (2021b) Holger Schwenk, Guillaume Wenzek, Sergey Edunov, Edouard Grave, Armand Joulin, and Angela Fan. 2021b.  Ccmatrix: Mining billions of high-quality parallel sentences on the web.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, ACL/IJCNLP 2021, (Volume 1: Long Papers), Virtual Event, August 1-6, 2021_ , pages 6490–6500. Association for Computational Linguistics.
  * Wang et al. (2020) Changhan Wang, Juan Pino, Anne Wu, and Jiatao Gu. 2020.  Covost: A diverse multilingual speech-to-text translation corpus.  In _Proceedings of the 12th Language Resources and Evaluation Conference_ , pages 4197–4203.
  * Wang et al. (2021a) Changhan Wang, Morgane Rivière, Ann Lee, Anne Wu, Chaitanya Talnikar, Daniel Haziza, Mary Williamson, Juan Miguel Pino, and Emmanuel Dupoux. 2021a.  Voxpopuli: A large-scale multilingual speech corpus for representation learning, semi-supervised learning and interpretation.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, ACL/IJCNLP 2021, (Volume 1: Long Papers), Virtual Event, August 1-6, 2021_ , pages 993–1003. Association for Computational Linguistics.
  * Wang et al. (2021b) Changhan Wang, Anne Wu, Jiatao Gu, and Juan Pino. 2021b.  Covost 2 and massively multilingual speech translation.  In _Interspeech_ , pages 2247–2251.
  * Wang et al. (2022) Hongyu Wang, Shuming Ma, Li Dong, Shaohan Huang, Dongdong Zhang, and Furu Wei. 2022\.  Deepnet: Scaling transformers to 1,000 layers.
  * Yang et al. (2019a) Yinfei Yang, Gustavo Hernandez Abrego, Steve Yuan, Qinlan Shen Mandy Guo, Daniel Cer, Brian Strope Yun-hsuan Sun and, and Ray Kurzweil. 2019a.  Improving multilingual sentence embedding using bi-directional dual encoder with additive margin softmax.  In _IJCAI_ , pages 5370–5378.
  * Yang et al. (2019b) Yinfei Yang, Daniel Cer, Amin Ahmad, Mandy Guo, Jax Law, Noah Constant, Gustavo Hernandez Abrego, Steve Yuan, Chris Tar, Yun-Hsuan Sung, Brian Strope, and Ray Kurzweil. 2019b.  Multilingual universal sentence encoder for semantic retrieval.  In _https://arxiv.org/abs/1907.04307_.
  * Zoph et al. (2022) Barret Zoph, Irwan Bello, Sameer Kumar, Nan Du, Yanping Huang, Jeff Dean, Noam Shazeer, and William Fedus. 2022.  Designing effective sparse expert models.  _arXiv preprint arXiv:2202.08906_.

##  Appendix A Appendix

###  A.1 Similarity search on CoVoST

We compared our similarity search results with previous work Duquenne et al.
(2021) in Table 12. We notice that our new speech encoders have lower error
rates compared to previous work.

Audio vs. en translations | de | es | fr
---|---|---|---
Previous work | 3.36 | 1.66 | 2.05
This work | 3.27 | 1.26 | 1.55
|  |  |
Table 12: Similarity search error rates (in %) on CoVoST 2 test set.

###  A.2 HuBERT

We train a multilingual HuBERT model for each family on the collection of
speech in each component language as shown in Table 6. The HuBERT model
consists of $7$ convolutional layers and $12$ Transformer encoder layers. Each
encoder layer has 12 attention heads, the embedding dimension of $768$ and the
forward dimension of $3072$. Models are trained for $3$ iterations, and in
each iteration pseudo-labels are prepared as the training target for
utterances. In the first iteration, the target labels are MFCC features. In
the second iteration, we extract speech features from the $6$-th layer of the
trained HuBERT model and apply $k$-means clustering to derive a set of $500$
labels. In the third iteration, speech features from the $9$-th layer are
clustered into $500$ labels. Lastly after these three iterations, we try
feature extraction from different layers including layer $10$, $11$ and $12$
of trained HuBERT. As for feature clustering, we also try different numbers of
clusters, $800$, $1000$ and $1200$, to derive multiple sets of target units.

To choose the optimal setup, we launch a resynthesis evaluation to select the
HuBERT layer to extract speech features and the number of $k$-means clusters.
We train a vocoder on each set of target units, i.e., vocoder takes the units
and synthesizes target speech. The synthesized speech is sent to off-the-shelf
ASR models, and Word Error Rate (WER) is reported to measure the speech
quality. The set of target units is selected if the corresponding vocoder
achieves lowest WER.

###  A.3 ASR models

We use ASR models publicly released on HuggingFace to transcribe the generated
speech in order to calculate WER or BLEU scores in comparison with ground
truth texts. ASR models used in our evaluation are listed in Table 13.

Lang | cs | de
---|---|---
ASR | comodoro/wav2vec2-xls-r-300m-cs-250 | jonatasgrosman/wav2vec2-xls-r-1b-german
Lang | et | fi
ASR | RASMUS/wav2vec2-xlsr-1b-et | jonatasgrosman/wav2vec2-large-xlsr-53-finnish
Lang | hr | hu
ASR | classla/wav2vec2-xls-r-parlaspeech-hr | jonatasgrosman/wav2vec2-large-xlsr-53-hungarian
Lang | it | lt
ASR | jonatasgrosman/wav2vec2-large-xlsr-53-italian | sammy786/wav2vec2-xlsr-lithuanian
Lang | nl | pl
ASR | jonatasgrosman/wav2vec2-xls-r-1b-dutch | jonatasgrosman/wav2vec2-xls-r-1b-polish
Lang | pt | ro
ASR | jonatasgrosman/wav2vec2-xls-r-1b-portuguese | gigant/romanian-wav2vec2
Lang | sk | sl
ASR | anuragshas/wav2vec2-xls-r-300m-sk-cv8-with-lm | anuragshas/wav2vec2-xls-r-300m-sl-cv8-with-lm
|  |

Table 13: HuggingFace ASR models for each language.

###  A.4 Vocoder

Vocoders are trained to synthesize speech from a given sequence of units. The
train sets are speech data from CSS10, VoxPopuli and Common Voice. As
mentioned before, units are derived from HuBERT models for these speech. Table
14 summarizes WER of ASR models, which reflects the transcription quality in
each language. Besides, we report the training dataset, vocoder WER of
synthesized speech from vocoders, and here we include the vocoder results
obtained from the optimal HuBERT layer and $k$-means cluster size. Layer $11$
is the best HuBERT layer for feature extraction in all languages, and most
languages have the best $k$-means size of $1000$ except Italian (it) whose
best label size is $800$.

Lang | Data | ASR WER | HuBERT | Vocoder WER | Lang | Data | ASR WER | HuBERT | Vocoder WER
---|---|---|---|---|---|---|---|---|---
de | CSS10 | 0.10 |  | Germanic HuBERT
---
layer 11, km 1000
0.16 | nl | CSS10 | 0.19 |  | Germanic HuBERT
---
layer 11, km 1000
0.27
fi | CSS10 | 0.02 |  | Uralic HuBERT
---
layer 11, km 1000
0.15 | hu | CSS10 | 0.21 |  | Uralic HuBERT
---
layer 11, km 1000
0.21
et |  | Common
---
Voice
0.14 |  | Uralic HuBERT
---
layer 11, km 1000
0.44 | it | VoxPopuli | 0.23 |  | Uralic HuBERT
---
layer 11, km 800
0.27
pt |  | Common
---
Voice
0.06 |  | Uralic HuBERT
---
layer 11, km 1000
0.31 | ro | VoxPopuli | 0.42 |  | Uralic HuBERT
---
layer 11, km 1000
0.50
cs | VoxPopuli | 0.15 |  | Slavic HuBERT
---
layer 11, km 1000
0.23 | pl | VoxPopuli | 0.14 |  | Slavic HuBERT
---
layer 11, km 1000
0.23
hr | VoxPopuli | 0.21 |  | Slavic HuBERT
---
layer 11, km 1000
0.29 | lt | VoxPopuli | 0.38 |  | Slavic HuBERT
---
layer 11, km 1000
0.57
sk | VoxPopuli | 0.28 |  | Slavic HuBERT
---
layer 11, km 1000
0.41 | sl | VoxPopuli | 0.37 |  | Slavic HuBERT
---
layer 11, km 1000
0.46
|  |  |  |  |  |  |  |  |

Table 14: Benchmark results of ASR models and vocoder resynthesis.

As shown in Table 14, ASR models are of good quality for high-resource
languages such as de, fi and pt, while suffering from high error rates in
languages such as ro, lt and sl. It is expected to have higher vocoder WER
than ASR WER since the former is for synthesized speech. By measuring the gap
between the two error rates, we can tell how good a vocoder is and also infer
the quality of HuBERT units. For et, pt and lt, the gaps are obviously larger
than other languages. It not surprising since we do not have much good-quality
speech data for these languages, for example, there is only around $10$-hour
noisy speech from Common Voice for et and pt vocoder training.

##  Appendix B Multilingual Speech-to-Speech Translation

We provide details of models and experiment setups in multilingual speech-to-
speech translation.

###  B.1 Slavic-to-English Translation

Textless model. Textless model (260M) has a speech encoder with $4$
convolution layers and $12$ Transformer encoder layers with the embedding
dimension of $1024$ and the forward dimension of $4096$. It has two unit
decoders with $6$ and $2$ Transformer decoder layers for target and source
unit prediction respectively. The target unit decoder has the embedding
dimension of $1024$ and the forward dimension of $4096$, and the source unit
decoder’s dimensions are $256$ and $2048$.

For the Textless model (424M), its speech encoder contains $6$ convolution
layers and $16$ Transformer encoder layers with the embedding dimension of
$1024$ and the forward dimension of $4096$. It has two unit decoders with $12$
and $2$ Transformer decoder layers for target and source unit prediction
respectively. The target unit decoder has the embedding dimension of $1024$
and the forward dimension of $4096$, and the source unit decoder’s dimensions
are $256$ and $2048$.

XM Transformer. XM Transformer (1.2B) is initialized from XLS-R encoder with
$7$ convolution layers and $48$ Transformer encoder layers with the embedding
dimension of $1280$ and the forward dimension of $5120$. Its unit decoder is
initialized from a pre-trained mbart-style decoder with $12$ layers, embedding
dimension of $1024$ and forward dimension of $4096$.

###  B.2 All-to-English Translation

XM Transformer-GShard. XM Transfomer (1.2B) is initialized with the same XLS-R
encoder and unit decoder used in Slavic-English. On the decoder side of XM
Transformer-GShard, each expert is initialized with the same unit decoder. We
set MoE frequency as 2, i.e., every other transformer layer is an MoE layer.

XM Transformer-Base Layer. For our XM Transformer with Base Layer sparsity
(1.7B), the encoder is initialized with the same XLS-R encoder, and the dense
layers of the decoder is initialized with the same unit decoder as GShard. We
add an additional Base Layer which is randomly initialized as the $7$th layer
of decoder. There is one expert in each GPU and we used $64$ GPUs in our
experiments, which means we have $64$ Base Layer experts in total.


# On the two-point function of the one-dimensional KPZ equation

Sergio I. López and Leandro P. R. Pimentel

###### Abstract

In this short communication we show that basic tools from Malliavin calculus
can be applied to derive the two-point function of the slope of the one-
dimensional KPZ equation, starting from a two-sided Brownian motion with an
arbitrary diffusion parameter, in terms of the polymer end-point annealed
distribution associated to the stochastic heat equation. We also prove that
this distribution is given in terms of the derivative of the variance of the
solution of the KPZ equation.

Keywords— KPZ equation, Two-point function, Malliavin calculus

##  1 Introduction and Results

The one-dimensional Kardar-Parisi-Zhang (KPZ) equation describes the growth of
an interface $h=(h_{t}(x)\,,\,x\in{\mathbb{R}})_{t\geq 0}$, modeled by a
height function $h_{t}(x)\in{\mathbb{R}}$ at position $x\in{\mathbb{R}}$ and
time $t\geq 0$, whose evolution satisfies the stochastic partial differential
equation

$\partial_{t}h=\frac{1}{2}\partial_{x}^{2}h+\frac{1}{2}(\partial_{x}h)^{2}+\xi\,,$

where $\xi$ denotes a space-time white noise. The interest in this equation
started when physicists [9, 12] predicted that the solution to the KPZ
equation has a dynamic scaling exponent equals to $3/2$, in the sense that if
one rescales space by $\epsilon^{-1}$ and $h$ diffusively with
$\epsilon^{1/2}$, then one has to rescale time by $\epsilon^{-3/2}$ to see non
trivial fluctuations behavior as $\epsilon\downarrow 0$. Since then, the study
of these fluctuations and its universality class has been a subject of
extensive work [6]. Another interesting aspect of this equation is that its
solutions are expected to be locally Brownian in space, and the understanding
of its non-linearity brought important developments in the theory of
stochastic partial differential equations [11]. Throughout this paper we
assume that $h_{0}(x)=\sigma W(x)$, where $(W(x)\,,\,x\in{\mathbb{R}})$ is a
standard two-sided Brownian motion, with $\sigma>0$ arbitrary, and that the
white noise and the Brownian motion are independent random elements defined on
some probability space $(\Omega,{\mathcal{F}},{\mathbf{P}})$, with
expectation, variance and covariance denoted by ${\mathbf{E}}$, ${\bf Var\,}$
and ${\bf Cov}$, respectively.

The KPZ equation is related to the stochastic heat equation (SHE)

$\partial_{t}Z=\frac{1}{2}\partial_{x}^{2}Z+Z\xi\,,$ (1.1)

by the so called Cole-Hopf transform [4]: $h_{t}(x)=\log Z_{t}(x)$. The Green
function $Z_{t}(x,y)$ associated to the stochastic heat equation is the
solution of (1.1) with the initial condition $Z_{0}(x,y)=\delta(x-y)$, the
Dirac delta function for fixed $y\in{\mathbb{R}}$. The expression

$Z_{t}(x)=\int_{\mathbb{R}}Z_{t}(x,y)e^{\sigma W(y)}dy\,,$ (1.2)

allows us to see $Z_{t}(x)$ as a partition function associated to the (random)
quenched density $p_{x,t}$ of a polymer endpoint $Y$, starting at position
$x\in{\mathbb{R}}$ and time $t\geq 0$, and running backwards in time [1]:

$p_{x,t}(y)=\frac{Z_{t}(x,y)e^{\sigma W(y)}}{Z_{t}(x)}\,\,\mbox{ and
}\,\,E_{x,t}\left[f(Y)\right]=\int_{\mathbb{R}}f(y)p_{x,t}(y)dy,$ (1.3)

see Figure 1.

Figure 1: The time-reversal polymer realisations can be seen as paths starting
at position $x$ at time $t$ and ending at the random endpoint $Y$ at time
zero.

The annealed probability measure of the polymer endpoint $Y$ is defined as

${\mathbb{P}}_{x,t}\left[Y\in
A\right]={\mathbf{E}}\left[\int_{A}p_{x,t}(y)dy\right]\,\,\mbox{ and
}\,\,{\mathbb{E}}_{x,t}\left[f(Y)\right]={\mathbf{E}}\left[E_{x,t}\left[f(Y)\right]\right]\,.$
(1.4)

For simplicity, when $x=0$ we remove the index from the notation. We note that
$Y$ is symmetric under ${\mathbb{P}}_{t}$ and that

${\mathbb{P}}_{x,t}\left[Y\in A\right]={\mathbb{P}}_{t}\left[x+Y\in
A\right]\,.$ (1.5)

In the study of the fluctuations of $h_{t}(x)=\log Z_{t}(x)$, the scaling
function

$g_{t}(x)={\bf Var\,}[h_{t}(x)]\,,$ (1.6)

plays a central rule, as noted in [8, 10], to study KPZ equation evolving on a
large torus and on the line, respectively. We recall that the dependence on
the parameter $\sigma>0$ is hidden but plays an important rule in the analysis
of the behavior of $h_{t}$. In the case $\sigma=1$ we have time stationarity,
in the sense that
$(h_{t}(x)-h_{t}(0)\,,\,x\in{\mathbb{R}})\stackrel{{\scriptstyle
dist.}}{{=}}(h_{0}(x)\,,\,x\in{\mathbb{R}})$, and it has been proven (Theorem
$1.2$ and Proposition $1.6$ in [3], Proposition $2.1$ and Remark $2.2$ in
[10]) that

$g_{t}(x)=\mathbb{E}_{t}|Y|\sim t^{2/3}\,\,\mbox{ and
}\,\,{\mathbf{E}}\left[\partial_{x}h_{t}(z)\partial_{x}h_{0}(y)\right]={\mathbf{E}}\left[p_{t}(z-y)\right]=\frac{g^{\prime\prime}_{t}(z-y)}{2}.$
(1.7)

We also address the reader to [13, 14] for similar non-rigorous results in the
physics literature. The left-hand side of second equality displayed above is
the commonly named KPZ two-point correlation function, and this equality
should be understood in the formal level, meaning that

${\mathbf{E}}\left[X_{t}(\phi_{2})X_{0}(\phi_{1})\right]=\int_{{\mathbb{R}}^{2}}\phi_{2}(x)\phi_{1}(y){\mathbf{E}}\left[p_{t}(x-y)\right]dxdy\,,$

where

$X_{t}(\phi)\equiv\int_{\mathbb{R}}\phi(z)\partial_{x}h_{t}(z)dz=-\int_{\mathbb{R}}\phi^{\prime}(z)h_{t}(z)dz\,,$

is defined for a smooth test function $\phi:{\mathbb{R}}\to{\mathbb{R}}$ of
bounded support.

This work can be seen as a companion communication of [10, 19], by
independently obtaining results in terms of the random polymer endpoint,
including the non-stationary case, and studying the decorrelation between the
initial condition and the configuration of the process as time grows to
infinity. We follow the idea of bringing basic tools from Malliavin calculus
(started in [19] in the context of the KPZ fixed point), to show directly that

${\mathbf{E}}\left[p_{t}(y)\right]=\frac{g_{t}^{\prime\prime}(y)}{2\sigma^{2}}\,\,\mbox{
and that
}\,\,{\mathbf{E}}\left[\partial_{x}h_{t}(z)\partial_{x}h_{0}(y)\right]=\frac{g_{t}^{\prime\prime}(z-y)}{2}\,\,\mbox{for
any $\sigma>0$}\,,$

where the second equality is understood in the formal level. The ideas
contained in the proof strongly rely on the polymer representation of the
solution of the KPZ equation, which also indicates that similar results can be
proved mutatis-mutandis to other polymer models, as in the case of the
O’Connell-Yor polymer [18, 21], or other SHE with different multiplicative
space-time noise.

By using a simple adaptation of Stein-Malliavin method for normal
approximations [16], as developed in [19], we will show that
$g^{\prime\prime}_{t}(y)$ governs not only the decay of correlations but also
the decay of the Wasserstein distance between the joint law $\theta_{t}$ of
$(X_{0}^{\phi_{1}},X_{t}^{\phi_{2}})$ and the product measure $\eta_{t}$
induced by the marginals of $\theta_{t}$:

${\rm
Wass}\left(\theta_{t},\eta_{t}\right)\leq\frac{1}{\sigma\|\phi_{1}\|}\sqrt{\frac{\pi}{2}}\,\int_{{\mathbb{R}}^{2}}|\phi^{\prime}_{2}(x)||\psi_{1}(y)|\frac{g_{t}^{\prime\prime}(x-y)}{2}dxdy\,,$

where $\|\phi_{1}\|$ is the ${\mathbb{L}}^{2}({\mathbb{R}})$ norm of
$\phi_{1}$, $\psi^{\prime}(x)=\phi_{1}(x)$ and $\psi(0)=0$. The Wasserstein
distance between two probability measures defined on ${\mathbb{R}}^{2}$ is
given by

${\rm Wass}(\theta,\eta)=\sup_{\ell\in{\rm
Lip}_{1}}\left\\{\Big{|}\,\int_{{\mathbb{R}}^{2}}\ell\,d\theta-\int_{{\mathbb{R}}^{2}}\ell\,d\eta\,\Big{|}\right\\}\,,$

where ${\rm Lip}_{1}$ is the collection of all functions
$\ell:{\mathbb{R}}^{2}\to{\mathbb{R}}$ with Lipschitz constant bounded by $1$.

The limiting fluctuations of $h_{t}$ are described by the KPZ fixed point [15]
for a wide class of initial profiles [2, 3, 5, 20, 22], which indicates that
$g_{t}(x)\sim t^{2/3}g(xt^{-2/3})$, where $g$ is an universal scaling function
given exactly by the variance of the KPZ fixed point starting from a two-sided
Brownian motion (with a new diffusion coefficient depending on $\sigma$). As a
consequence, one also expects that $g^{\prime\prime}_{t}(x)\sim
t^{-2/3}g^{\prime\prime}(xt^{-2/3})$. Another interesting observation is that
the quenched end-point polymer density $p_{t}(y)=p^{\sigma}_{t}(y)$ also makes
sense for $\sigma=0$, the so called flat initial condition (see also [7] for
localization results). From our formula for $\sigma>0$ we also get that

${\mathbf{E}}\left[p^{0}_{t}(y)\right]=\lim_{\sigma\searrow
0}{\mathbf{E}}\left[p^{\sigma}_{t}(y)\right]=\lim_{\sigma\searrow
0}\frac{g_{t}^{\prime\prime}(y)}{2\sigma^{2}}\,.$

(Recall that $g_{t}(x)=g^{\sigma}_{t}(x)$ also depends on $\sigma>0$.)

Our main results are summarized in the theorem below, in a slightly different
but equivalent formulation that will fit the proofs better. We denote

$\phi_{2}\star\phi_{1}(u)=\int_{\mathbb{R}}\phi_{2}(z)\phi_{1}(z+u)dz\,,$

the cross-correlation between the test functions $\phi_{2}$ and $\phi_{1}$.
Note that (by the symmetry of $g^{\prime\prime}_{t}$)

$\int_{{\mathbb{R}}^{2}}\phi_{2}(x)\phi_{1}(y)\frac{g_{t}^{\prime\prime}(x-y)}{2}dxdy=\int_{\mathbb{R}}\phi_{2}\star\phi_{1}(u)\frac{g_{t}^{\prime\prime}(u)}{2}du\,.$

###### Theorem 1

Recall the annealed end-point probability measure (1.4) and the scaling
function (1.6). We have that

  1. 1.

$g^{\prime}_{t}(y)=\sigma^{2}\left(2{\mathbb{P}}_{t}\left[Y\leq
y\right]-1\right)$;

  2. 2.

${\mathbf{E}}\left[X_{t}(\phi_{2})X_{0}(\phi_{1})\right]=\sigma^{2}{\mathbb{E}}_{t}\left[\phi_{2}\star\phi_{1}(Y)\right]$;

  3. 3.

${\rm
Wass}\left(\theta_{t},\eta_{t}\right)\leq\frac{\sigma}{\|\phi_{1}\|}\sqrt{\frac{\pi}{2}}{\mathbb{E}}_{t}\left[|\phi^{\prime}_{2}|\star|\psi_{1}|\left(Y\right)\right]$.

##  2 Proofs

The Malliavin derivative, with respect to the standard two-sided Brownian
motion $W$, of a random variable $X$ defined on
$(\Omega,{\mathcal{F}},{\mathbf{P}})$ is a
${\mathbb{L}}^{2}(\Omega\times{\mathbb{R}})$ valued random element denoted by
$(DX(u)\,,\,u\in{\mathbb{R}})$ [16]. For instance, for a smooth test function
$\phi:{\mathbb{R}}\to{\mathbb{R}}$, we have that

$D\left(\int_{\mathbb{R}}\phi(z)dW(z)\right)(u)=\phi(u)\,.$ (2.1)

One of the main tools in Malliavin calculus is the integration by parts
formula, which states that

${\mathbf{E}}\left[X\int_{\mathbb{R}}\phi(z)dW(z)\right]={\mathbf{E}}\left[\langle
DX,\phi\rangle\right]\,,$ (2.2)

where $\langle\cdot,\cdot\rangle$ is the usual inner product in
${\mathbb{L}}^{2}({\mathbb{R}})$. Let us define the function

$u\in{\mathbb{R}}\,\mapsto\,{\mathds{1}}_{y}(u)=\left\\{\begin{array}[]{ll}\,\,\,\,{\mathds{1}}_{(0,y]}(u)&\mbox{
if }y>0\,,\\\ \,\,\,\,0&\mbox{ if }y=0\,,\\\ -{\mathds{1}}_{(y,0]}(u)&\mbox{
if }y<0\,.\end{array}\right.$ (2.3)

that will be used several times in the following, as well as the usual
notation $x\vee y=\max\\{x,y\\}$, $x\wedge y=\min\\{x,y\\}$, $x^{+}=x\vee 0$
and $x^{-}=-(x\wedge 0)$.

The Malliavin derivative (2.1) of $\sigma
W(y)=\int_{\mathbb{R}}\sigma{\mathds{1}}_{y}(u)dW(u)$ is given by

$D\left(\sigma W(y)\right)(u)=\sigma{\mathds{1}}_{y}(u)\,,$

and by the chain rule, one gets that

$D(e^{\sigma W(y)})(u)=e^{\sigma W(y)}D\left(\sigma W(y)\right)(u)=\sigma
e^{\sigma W(y)}{\mathds{1}}_{y}(u)\,.$

Using (1.2), one has that $Z_{t}(x)$ is Malliavin differentiable with respect
to $W$ [10]. By switching the derivative with the Lebesgue integral in (1.2)
(Proposition $3.4.3$ in [17]), and using the previous calculation, one obtains
that (notice that $Z_{t}(x,y)$ does not depend on $W$)

$D\left(Z_{t}(x)\right)(u)=\sigma\int_{{\mathbb{R}}}Z_{t}(x,y)e^{\sigma
W(y)}{\mathds{1}}_{y}(u)dy\,,$

and by the chain rule again,

$D\left(h_{t}(x)\right)(u)=D\left(\log
Z_{t}(x)\right)(u)=\frac{1}{Z_{t}(x)}D\left(Z_{t}(x)\right)(u)=\sigma\frac{\int_{{\mathbb{R}}}Z_{t}(x,y)e^{\sigma
W(y)}{\mathds{1}}_{y}(u)dy}{Z_{t}(x)}\,.$

Writing this expression in terms of the polymer end-point density (1.3), one
finally concludes that

$D\left(h_{t}(x)\right)(u)=\sigma E_{x,t}\left[{\mathds{1}}_{Y}(u)\right]\,.$
(2.4)

###### Lemma 1

We have that

$g_{t}(x)={\bf Var\,}\left[h_{t}(x)\right]={\bf
Var\,}\left[h_{t}(0)\right]-\sigma^{2}|x|+2\sigma^{2}G_{t}(x)\,,$

where,

$G_{t}(x):=\left\\{\begin{array}[]{ll}{\mathbb{E}}_{t}\left[\left(x\wedge(x+Y)\right)^{+}\right]&\mbox{
if }x\geq 0\,,\\\
{\mathbb{E}}_{t}\left[\left(x\vee(x+Y)\right)^{-}\right]&\mbox{ if }x\leq
0\,.\end{array}\right.$

### Proof Lemma 1

Use (1.2), together with shift invariance of the Brownian motion and
translation invariance of the white noise, to see that

$h_{t}(x)-h_{0}(x)=\log\int_{\mathbb{R}}Z_{t}(x,y)e^{\sigma(W(y)-W(x))}dy\stackrel{{\scriptstyle
dist.}}{{=}}\log\int_{\mathbb{R}}Z_{t}(0,y)e^{\sigma W(y)}dy=h_{t}(0)\,.$

Since,
${\mathbb{E}}\left[h_{0}(x)\right]=\sigma{\mathbb{E}}\left[W(x)\right]=0$, we
get that

$\displaystyle{\bf Var\,}\left[h_{t}(x)\right]$ $\displaystyle=$
$\displaystyle{\bf Var\,}\left[h_{t}(x)-h_{0}(x)\right]+{\bf
Var\,}\left[h_{0}(x)\right]+2{\bf Cov}\left[h_{t}(x)-h_{0}(x),h_{0}(x)\right]$
$\displaystyle=$ $\displaystyle{\bf Var\,}\left[h_{t}(0)\right]-{\bf
Var\,}\left[h_{0}(x)\right]+2{\bf Cov}\left[h_{t}(x),h_{0}(x)\right]$
$\displaystyle=$ $\displaystyle{\bf
Var\,}\left[h_{t}(0)\right]-\sigma^{2}|x|+2{\mathbf{E}}\left[h_{t}(x)h_{0}(x)\right]\,.$

Integration by parts (2.2) from Malliavin calculus, combined with (2.4),
implies that

${\mathbf{E}}\left[h_{t}(x)h_{0}(x)\right]=\sigma{\mathbf{E}}\left[h_{t}(x)\int_{\mathbb{R}}{\mathds{1}}_{x}(z)dW(z)\right]=\sigma{\mathbf{E}}\left[\langle
Dh_{t}(x)\,,\,{\mathds{1}}_{x}\rangle\right]=\sigma^{2}{\mathbf{E}}\left[\langle
E_{x,t}\left[{\mathds{1}}_{Y}(\cdot)\right]\,,\,{\mathds{1}}_{x}\rangle\right]\,.$

By Fubini’s theorem, for $x\geq 0$,

$\langle
E_{x,t}\left[{\mathds{1}}_{Y}(\cdot)\right]\,,\,{\mathds{1}}_{x}\rangle=\int_{0}^{x}E_{x,t}\left[{\mathds{1}}_{Y}(u)\right]du=E_{x,t}\left[\int_{0}^{x}{\mathds{1}}_{Y}(u)du\right]=E_{x,t}\left[\left(x\wedge
Y\right)^{+}\right]\,,$

while for $x\leq 0$,

$\langle
E_{x,t}\left[{\mathds{1}}_{Y}(\cdot)\right]\,,\,{\mathds{1}}_{x}\rangle=-\int_{x}^{0}E_{x,t}\left[{\mathds{1}}_{Y}(u)\right]du=E_{x,t}\left[-\int_{x}^{0}{\mathds{1}}_{Y}(u)du\right]=E_{x,t}\left[\left(x\vee
Y\right)^{-}\right]\,.$

To conclude the proof, use (1.4) and (1.5). $\Box$

###### Lemma 2

${\mathbf{E}}\left[\langle
DX_{t}(\phi_{2}),\phi_{1}\rangle\right]=\sigma{\mathbb{E}}_{t}\left[\phi_{2}\star\phi_{1}(Y)\right]\,\mbox{
and }\,{\mathbf{E}}\left[\left|\langle
DX_{t}(\phi_{2}),\phi_{1}\rangle\right|\right]\leq\sigma{\mathbb{E}}_{t}\left[|\phi^{\prime}_{2}|\star|\psi_{1}|(Y)\right]\,.$

### Proof Lemma 2

Recall (2.3) and

$\psi_{1}(x)=\int_{\mathbb{R}}\phi_{1}(u){\mathds{1}}_{x}(u)du\,\mbox{ (the
primitive of $\phi_{1}$)}\,.$

By interchanging Malliavin derivative and integral and using (2.4),

$DX_{t}^{\sigma}(\phi_{2})(u)=-\sigma\int_{\mathbb{R}}\phi^{\prime}_{2}(z)E_{z,t}\left[{\mathds{1}}_{Y}(u)\right]dz\,,$

and hence, by Fubini’s theorem,

$\displaystyle\langle DX_{t}(\phi_{2}),\phi_{1}\rangle$ $\displaystyle=$
$\displaystyle\int_{\mathbb{R}}DX_{t}(\phi_{2})(u)\phi_{1}(u)du$ (2.5)
$\displaystyle=$
$\displaystyle-\sigma\int_{\mathbb{R}}\left(\int_{\mathbb{R}}\phi^{\prime}_{2}(z)E_{z,t}\left[{\mathds{1}}_{Y}(u)\right]\phi_{1}(u)dz\right)du$
$\displaystyle=$
$\displaystyle-\sigma\int_{\mathbb{R}}\phi^{\prime}_{2}(z)E_{z,t}\left[\psi_{1}(Y)\right]dz\,.$

On one hand, Fubini’s theorem again, (2.5), (1.4) and (1.5),

${\mathbf{E}}\left[\langle
DX_{t}(\phi_{2}),\phi_{1}\rangle\right]=-\sigma\int_{\mathbb{R}}\phi^{\prime}_{2}(z){\mathbb{E}}_{t}\left[\psi_{1}(z+Y)\right]dz\,.$

On the other hand, by standard integration by parts,

$\int_{\mathbb{R}}\phi^{\prime}_{2}(z)\psi_{1}(z+Y)dz=-\int_{\mathbb{R}}\phi_{2}(z)\phi_{1}(z+Y)dz\,,$

and hence, by Fubini’s theorem once again,

$\displaystyle\int_{\mathbb{R}}\phi^{\prime}_{2}(z){\mathbb{E}}_{t}\left[\psi_{1}(z+Y)\right]dz$
$\displaystyle=$
$\displaystyle{\mathbb{E}}_{t}\left[\int_{\mathbb{R}}\phi^{\prime}_{2}(z)\psi_{1}(z+Y)dz\right]$
$\displaystyle=$
$\displaystyle-{\mathbb{E}}_{t}\left[\int_{\mathbb{R}}\phi_{2}(z)\phi_{1}(z+Y)dz\right]\,,$

and we can conclude that

${\mathbf{E}}\left[\langle
DX_{t}(\phi_{2}),\phi_{1}\rangle\right]=\sigma{\mathbb{E}}_{t}\left[\phi_{2}\star\phi_{1}(Y)\right]\,.$

Taking the absolute value into the integrals, a similar calculation leads to

${\mathbf{E}}\left[\left|\langle
DX_{t}(\phi_{2}),\phi_{1}\rangle\right|\right]\leq\sigma{\mathbb{E}}_{t}\left[|\phi^{\prime}_{2}|\star|\psi_{1}|(Y)\right]\,.$

$\Box$

### Proof Theorem 1

To prove that

$g^{\prime}_{t}(y)=\sigma^{2}\left(2{\mathbb{P}}_{t}\left[Y\leq
y\right]-1\right)\,,$

one uses the formula obtained in Lemma 1, and notice that, for $x\geq 0$,

$\displaystyle{\mathbb{E}}_{t}\left[\left(x\wedge(x+Y)\right)^{+}\right]$
$\displaystyle=$
$\displaystyle{\mathbb{E}}_{t}\left[(x+Y){\mathds{1}}_{\\{Y\in(-x,0]\\}}+x{\mathds{1}}_{\\{Y>0]\\}}\right]$
$\displaystyle=$
$\displaystyle{\mathbb{E}}_{t}\left[Y{\mathds{1}}_{\\{Y\in(-x,0]\\}}\right]+x{\mathbb{P}}_{t}\left[Y>-x\right]\,.$

and that, for $x\leq 0$,

$\displaystyle{\mathbb{E}}_{t}\left[\left(x\vee(x+Y)\right)^{-}\right]$
$\displaystyle=$
$\displaystyle{\mathbb{E}}_{t}\left[-(x+Y){\mathds{1}}_{\\{Y\in(0,-x]\\}}-x{\mathds{1}}_{\\{Y\leq
0]\\}}\right]$ $\displaystyle=$
$\displaystyle-{\mathbb{E}}_{t}\left[Y{\mathds{1}}_{\\{Y\in(0,-x]\\}}\right]-x{\mathbb{P}}_{t}\left[Y\leq-x\right]\,.$

Thus, for $x\geq 0$,

$G^{\prime}_{t}(x)={\mathbb{P}}_{t}\left[Y>-x\right]={\mathbb{P}}_{t}\left[Y\leq
x\right]\,,$

and hence

$g^{\prime}_{t}(x)=-\sigma^{2}+2\sigma^{2}{\mathbb{P}}_{t}\left[Y\leq
x\right]=\sigma^{2}\left(2{\mathbb{P}}_{t}\left[Y\leq x\right]-1\right)\,,$

while for $x\leq 0$,

$G^{\prime}_{t}(x)=-{\mathbb{P}}_{t}\left[Y\leq-x\right]={\mathbb{P}}_{t}\left[Y\leq
x\right]-1\,,$

and hence

$g^{\prime}_{t}(x)=\sigma^{2}+2\sigma^{2}\left({\mathbb{P}}_{t}\left[Y\leq
x\right]-1\right)=\sigma^{2}\left(2{\mathbb{P}}_{t}\left[Y\leq
x\right]-1\right)\,.$

By integration by parts (2.2),

${\mathbf{E}}\left[X_{t}(\phi_{2})X_{0}(\phi_{1})\right]=\sigma{\mathbf{E}}\left[X_{t}(\phi_{2})\int_{\mathbb{R}}\phi_{1}(z)dW(z)\right]=\sigma{\mathbf{E}}\left[\langle
DX_{t}(\phi_{2}),\phi_{1}\rangle\right]\,,$

and the second item of Theorem 1 follows from Lemma 2.

To prove the upper bound on the Wasserstein distance between $\theta_{t}$ and
$\eta_{t}$ we give a brief explanation of the Stein-Malliavin method for
asymptotic independence, as developed in [19]. Let $\sigma_{1}>0$ and
$X_{1}\sim N(0,\sigma_{1}^{2})$ be a normal random variable with mean zero and
variance $\sigma_{1}^{2}$. Define the functional operator ${\mathcal{N}}$,
acting on differentiable functions $f:{\mathbb{R}}^{2}\to{\mathbb{R}}$, by

${\mathcal{N}}f(x_{1},x_{2}):=\sigma_{1}^{2}\partial_{x_{1}}f(x_{1},x_{2})-x_{1}f(x_{1},x_{2})\,.$
(2.6)

Given a random variable $X_{2}$ that is independent of $X_{1}$, the operator
${\mathcal{N}}$ characterizes the joint product measure $\eta$ of
$(X_{1},X_{2})$ in the sense that $X_{1}\sim N(0,\sigma_{1}^{2})$ and $X_{2}$
is independent of $X_{1}$ if and only if
${\mathbf{E}}\left[{\mathcal{N}}f(X_{1},X_{2})\right]=0$ for all nice
functions $f$. In this way, given a random vector $(X_{1},X_{2})$ with
probability measure $\theta$, and such that $X_{1}\sim N(0,\sigma_{1}^{2})$,
we might estimate the distance to independence by looking at
${\mathbf{E}}\left[{\mathcal{N}}f(X_{1},X_{2})\right]$. This idea can be made
precise by using the unique bounded solution $f_{\ell}$ of the partial
differential equation

${\mathcal{N}}f(x_{1},x_{2})=\ell(x_{1},x_{2})-{\mathbf{E}}\left[\ell(X_{1},x_{2})\right]\,,$

in such way that

${\mathbf{E}}\left[{\mathcal{N}}f_{\ell}(X_{1},X_{2})\right]=\int_{{\mathbb{R}}^{2}}\ell\,d\theta-\int_{{\mathbb{R}}^{2}}\ell\,d\eta\,\implies\,{\rm
Wass}(\theta,\eta)\leq\sup_{\ell}\left|{\mathbf{E}}\left[{\mathcal{N}}f_{\ell}(X_{1},X_{2})\right]\right|\,.$

In our context, we have
$X_{1}=X_{0}(\phi_{1})=\sigma\int_{\mathbb{R}}\phi_{1}(z)dW(z)\sim
N(0,\sigma_{1}^{2})$ with $\sigma_{1}^{2}=\sigma^{2}\|\phi_{1}\|^{2}$, and
$X_{2}=X_{t}(\phi_{2})$. By using that

$Df(X_{1},X_{2})=\partial_{x_{1}}f(X_{1},X_{2})DX_{1}+\partial_{x_{2}}f(X_{1},X_{2})DX_{2}\,,$

and integration by parts (2.2), we have that

${\mathbf{E}}\left[X_{1}f(X_{1},X_{2})\right]=\sigma{\mathbf{E}}\left[\langle
Df(X_{1},X_{2}),\phi_{1}\rangle\right]\\\
=\sigma^{2}\|\phi_{1}\|^{2}{\mathbf{E}}\left[\partial_{x_{1}}f(X_{1},X_{2})\right]+\sigma{\mathbf{E}}\left[\partial_{x_{2}}f(X_{1},X_{2})\langle
DX_{2},\phi_{1}\rangle\right]\,,$

and hence

${\mathbf{E}}\left[{\mathcal{N}}f(X_{1},X_{2})\right]=-\sigma{\mathbf{E}}\left[\partial_{x_{2}}f(X_{1},X_{2})\langle
DX_{2},\phi_{1}\rangle\right]\,.$

By Lemma 8 [19],

$\sup_{{\mathbb{R}}^{2}}\left|\partial_{x_{2}}f_{\ell}(x_{1},x_{2})\right|\leq\frac{1}{\sigma_{1}}\sqrt{\frac{\pi}{2}}\sup_{{\mathbb{R}}^{2}}\left|\partial_{x_{2}}\ell(x_{1},x_{2})\right|\leq\frac{1}{\sigma_{1}}\sqrt{\frac{\pi}{2}}\,$

(recall that $\ell$ has Lipschitz constant bounded by $1$), which finally
yields the upper bound

${\rm
Wass}(\theta_{t},\eta_{t})\leq\sup_{\ell}\left|{\mathbf{E}}\left[{\mathcal{N}}f_{\ell}(X_{1},X_{2})\right]\right|\\\
\leq\frac{1}{\|\phi_{1}\|}\sqrt{\frac{\pi}{2}}{\mathbf{E}}\left[\left|\langle
DX_{2},\phi_{1}\rangle\right|\right]\leq\frac{\sigma}{\|\phi_{1}\|}\sqrt{\frac{\pi}{2}}{\mathbb{E}}_{0,t}\left[|\phi^{\prime}_{2}|\star|\psi_{1}|(Y)\right]\,,$

where in the last equality we use Lemma 2.

$\Box$

Acknowledgements Sergio I. López was supported by DGAPA-UNAM grants PAPIIT
IN-102822 and PASPA 2021-II. Leandro P. R. Pimentel was supported by the
National Council of Scientific Researches (CNPQ, Brazil) Grant 305356/2019-4.

## References

  * [1] Alberts, T., Khanin, K. and Quastel, J. (2014). The Continuum Directed Random Polymer. J. Stat. Phys. 154, 305–326.
  * [2] Amir, G., Corwin, I., and Quastel, J. (2011). Probability distribution of the free energy of the continuum directed random polymer in 1 + 1 dimensions. Comm. Pure Appl. Math. 64, 466–537.
  * [3] Balázs, M., Quastel, J., and Seppäläinen, T. (2011). Fluctuation exponent of the KPZ/stochastic Burgers equation. J. Amer. Math. Soc. 24, 683–708.
  * [4] Bertini, L., and Giacomin, G. (1997). Stochastic Burgers and KPZ equations from particle systems. Comm. Math. Phys. 183, 571–607.
  * [5] Borodin, A., Corwin, I., Ferrari, P. L., and Vëto, B. (2015). Height fluctuations for the stationary KPZ equation. Math. Phys. Anal. Geom. 18, 20.
  * [6] Corwin, I. (2012). The Kardar-Parisi-Zhang equation and universality class. Random Matrices: Theory Appl. 1, 1130001.
  * [7] Das, S. and Zhu, W. (2022). Localization of the continuum directed random polymer. Available at https://arxiv.org/abs/2203.03607.
  * [8] Dunlap, A. and Gu, Y., and Komorowski, T. (2021). Fluctuations of the KPZ equation on a large torus. To appear in Comm. Pure and Appl. Math., available at https://arxiv.org/abs/2111.03650.
  * [9] Forster, D., Nelson, D.R., and Stephen, M.J. (1977). Large-distance and long-time properties of a randomly stirred fluid. Phys. Rev. A 16, 732–749.
  * [10] Gu, Y., and Komorowski, T. (2023). Another look at Balázs-Quastel-Seppäläinen theorem. Trans. Amer. Math. Soc. 376, 2947–2962.
  * [11] Hairer, M. (2013). Solving the KPZ equation. Ann. Math. 178, 559–664.
  * [12] Kardar, M., Parisi, G., and Zhang, Y.Z. (1986). Dynamic scaling of growing interfaces. Phys. Rev. Lett. 56, 889–892.
  * [13] Le Doussal, P. (2017). Maximum of an Airy process plus Brownian motion and memory in Kardar-Parisi-Zhang growth. Phys. Rev. E 96 060101\.
  * [14] Maes, C. and Thiery, T. (2017). Midpoint Distribution of Directed Polymers in the Stationary Regime: Exact Result Through Linear Response. J. Stat. Phys. 168 937–963.
  * [15] K. Matetski, K., Quastel, J., and Remenik, D. (2021). The KPZ fixed point. Acta Math. 227, 115–203.
  * [16] Nualart, D. (2019). Malliavin calculus and normal approximations. Ensaios Matemáticos 34, 1–74.
  * [17] Nualart, D., Nualart, E. (2018). Introduction to Malliavin calculus. IMS textbooks, Cambridge University Press.
  * [18] O’Connell, N., Yor, M. (2001). Brownian Analogues of Burke’s Theorem. Stoch. Process Appl. 96 285–304.
  * [19] Pimentel, L.P.R. (2020). Integration by parts and the KPZ two-point function. Ann. Probab. 50(5), 1755–1780.
  * [20] Quastel, J., and Sarkar, S. (2023). Convergence of exclusion processes and the KPZ equation to the KPZ fixed point. J. Amer. Math. Soc. 36 251–289.
  * [21] Seppäläinen, T., Valkó, B. (2010). Bounds for scaling exponents for a 1+1 dimensional directed polymer in a Brownian environment. ALEA 7, 451–476.
  * [22] Virag, B. (2020). The heat and the landscape I. Available at https://arxiv.org/pdf/2008.07241.