$D=2^{10}$ (red) and $2^{100}$ (black).

The two-site partition function corresponding to one replica and one conjugate
replica is given by

$\displaystyle Z_{2}(\beta)$ $\displaystyle=$
$\displaystyle\left\langle\sum_{kl}e^{-\beta(E_{k}+E_{l}^{*})}\right\rangle$
(193) $\displaystyle=$ $\displaystyle\left\langle\sum_{k\neq
l}e^{-\beta(E_{k}+E_{l}^{*})}\right\rangle+\left\langle\sum_{k}e^{-\beta(E_{k}+E_{k}^{*})}\right\rangle$

Since different eigenvalues are uncorrelated, and all eigenvalues have the
same distribution. We obtain in the large $D$ limit

$\displaystyle Z_{2}(\beta)$ $\displaystyle=$
$\displaystyle\frac{D(D-1)}{\pi^{2}R^{4}}\int_{\cal D}d^{2}ze^{-\beta
z}\int_{\cal D}d^{2}ze^{-\beta z^{*}}+\frac{D}{\pi R^{2}}\int_{\cal
D}d^{2}ze^{-\beta(z+z^{*})}$ (194) $\displaystyle=$ $\displaystyle
D(D-1)+\frac{D}{R\beta}I_{1}(2R\beta).$

The large $D$ limit of the free energy density is given by

$\displaystyle\frac{F(T)}{\log D}=-\lim_{D\to\infty}\frac{T}{\log
D}\log(Z_{1}^{2}(\beta)+Z_{2}(\beta))=-\lim_{D\to\infty}\frac{T}{\log
D}\log\left(e^{2\log D}+\frac{D}{R\beta}I_{1}(2R\beta)\right).$ (195)

A nontrivial large $D$ limit is obtained if we scale $R$ as $\log D$, and we
will choose $R=\log D$. At the critical temperature $T_{c}$ the two leading
exponents are equal so that

$\displaystyle 2\log D=\log D+2\beta_{c}\log D$ (196)

resulting in

$\displaystyle T_{c}=2,$ (197)

and the free energy is given by

$\displaystyle\frac{F(T)}{\log D}=(-2-T)\theta(T_{c}-T)-2T\theta(T-T_{c}).$
(198)

In Figure 18 we show the free energy (195) versus the temperature. The
convergence to the asymptotic result (198) is slow, but for $D=2^{100}$ (black
curve) a kink becomes visible.

## References

  * (1) S.F. Edwards and P.W. Anderson, _Theory of spin glasses_ , _Journal of Physics F: Metal Physics_ 5 (1975) 965.
  * (2) G. Parisi, _Toward a mean field theory for spin glasses_ , _Physics Letters A_ 73 (1979) 203.
  * (3) G. Parisi, _Order parameter for spin-glasses_ , _Phys. Rev. Lett._ 50 (1983) 1946.
  * (4) F. Wegner, _The mobility edge problem: continuous symmetry and a conjecture_ , _Zeitschrift für Physik B Condensed Matter_ 35 (1979) 207.
  * (5) A. Kamenev and M. Mezard, _Level correlations in disordered metals: The replica sigma model_ , _Phys. Rev. B_ 60 (1999) 3944 [cond-mat/9903001].
  * (6) M.A. Stephanov, _Random matrix model of QCD at finite density and the nature of the quenched limit_ , _Phys. Rev. Lett._ 76 (1996) 4472 [hep-lat/9604003].
  * (7) G. Akemann, J.C. Osborn, K. Splittorff and J.J.M. Verbaarschot, _Unquenched QCD Dirac operator spectra at nonzero baryon chemical potential_ , _Nucl. Phys. B_ 712 (2005) 287 [hep-th/0411030].
  * (8) H. Nishimori and K.Y.M. Wong, _Statistical mechanics of image restoration and error-correcting codes_ , _Phys. Rev. E_ 60 (1999) 132 [cond-mat/9902312].
  * (9) M. Mézard, G. Parisi, N. Sourlas, G. Toulouse and M. Virasoro, _Nature of the spin-glass phase_ , _Phys. Rev. Lett._ 52 (1984) 1156.
  * (10) Mézard, M., Parisi, G. and Virasoro, M.A., _Random free energies in spin glasses_ , _J. Physique Lett._ 46 (1985) 217.
  * (11) D. Sherrington and S. Kirkpatrick, _Solvable model of a spin-glass_ , _Phys. Rev. Lett._ 35 (1975) 1792.
  * (12) E. Kanzieper, _Replica field theories, Painleve transcendents and exact correlation functions_ , _Phys. Rev. Lett._ 89 (2002) 250201 [cond-mat/0207745].
  * (13) S.M. Nishigaki and A. Kamenev, _Replica treatment of non-hermitian disordered hamiltonians_ , _Journal of Physics A: Mathematical and General_ 35 (2002) 4571–4590 [cond-mat/0109126].
  * (14) J.B. French and S.S.M. Wong, _Validity of random matrix theories for many-particle systems_ , _Physics Letters B_ 33 (1970) 449 .
  * (15) O. Bohigas and J. Flores, _Two-body random hamiltonian and level density_ , _Physics Letters B_ 34 (1971) 261 .
  * (16) A. Kitaev, _A simple model of quantum holography_ , _KITP strings seminar and Entanglement 2015 program, 12 February, 7 April and 27 May 2015,http://online.kitp.ucsb.edu/online/entangled15/_ (2015) .
  * (17) S. Sachdev and J. Ye, _Gapless spin-fluid ground state in a random quantum heisenberg magnet_ , _Phys. Rev. Lett._ 70 (1993) 3339.
  * (18) J. Maldacena and D. Stanford, _Remarks on the Sachdev-Ye-Kitaev model_ , _Phys. Rev. D_ 94 (2016) 106002 [1604.07818].
  * (19) K. Jensen, _Chaos in AdS 2 Holography_, _Phys. Rev. Lett._ 117 (2016) 111601 [1605.06098].
  * (20) J.B. French and S.S.M. Wong, _Some random-matrix level and spacing distributions for fixed-particle-rank interactions_ , _Physics Letters B_ 35 (1971) 5 .
  * (21) T.A. Brody, J. Flores, J.B. French, P.A. Mello, A. Pandey and S.S.M. Wong, _Random-matrix physics: spectrum and strength fluctuations_ , _Rev. Mod. Phys._ 53 (1981) 385.
  * (22) J.J.M. Verbaarschot, H.A. Weidenmuller and M.R. Zirnbauer, _Grassmann Integration in Stochastic Quantum Physics: The Case of Compound Nucleus Scattering_ , _Phys. Rept._ 129 (1985) 367.
  * (23) J. Flores, M. Horoi, M. Muller and T.H. Seligman, _Spectral statistics of the two-body random ensemble revisited_ , _Phys. Rev. E_ 63 (2001) 026204 [cond-mat/0006144].
  * (24) L. Benet and H.A. Weidenmuller, _Review of the k body embedded ensembles of Gaussian random matrices_ , _J. Phys. A_ 36 (2003) 3569 [cond-mat/0207656].
  * (25) V. Zelevinsky and A. Volya, _Nuclear structure, random interactions and mesoscopic physics_ , _Phys. Rept._ 391 (2004) 311 [nucl-th/0309071].
  * (26) V.K.B. Kota and M. Vyas, _Statistical Nuclear Spectroscopy with $q$-normal and bivariate $q$-normal distributions and $q$-Hermite polynomials_, 2201.12509.
  * (27) C. Teitelboim, _Gravitation and hamiltonian structure in two spacetime dimensions_ , _Physics Letters B_ 126 (1983) 41 .
  * (28) R. Jackiw, _Lower dimensional gravity_ , _Nuclear Physics B_ 252 (1985) 343 .
  * (29) J. Maldacena, D. Stanford and Z. Yang, _Conformal symmetry and its breaking in two dimensional Nearly Anti-de-Sitter space_ , _PTEP_ 2016 (2016) 12C104 [1606.01857].
  * (30) J. Engelsöy, T.G. Mertens and H. Verlinde, _An investigation of ads2 backreaction and holography_ , _Journal of High Energy Physics_ 07 (2016) 1 [1606.03438].
  * (31) D. Stanford and E. Witten, _Fermionic Localization of the Schwarzian Theory_ , _JHEP_ 10 (2017) 008 [1703.04612].
  * (32) J.S. Cotler, G. Gur-Ari, M. Hanada, J. Polchinski, P. Saad, S.H. Shenker et al., _Black Holes and Random Matrices_ , _JHEP_ 05 (2017) 118 [1611.04650].
  * (33) A.M. García-García and J.J.M. Verbaarschot, _Analytical spectral density of the sachdev-ye-kitaev model at finite $n$_, _Phys. Rev. D_ 96 (2017) 066012 [1701.06593].
  * (34) A.M. García-García and J.J.M. Verbaarschot, _Spectral and thermodynamic properties of the sachdev-ye-kitaev model_ , _Phys. Rev. D_ 94 (2016) 126010 [1610.03816].
  * (35) Y.-Z. You, A.W.W. Ludwig and C. Xu, _Sachdev-ye-kitaev model and thermalization on the boundary of many-body localized fermionic symmetry-protected topological states_ , _Phys. Rev. B_ 95 (2017) 115150 [1602.06964].
  * (36) A.M. García-García, Y. Jia and J.J.M. Verbaarschot, _Universality and thouless energy in the supersymmetric sachdev-ye-kitaev model_ , _Phys. Rev. D_ 97 (2018) 106003 [1801.01071].
  * (37) A.M. García-García, T. Nosaka, D. Rosa and J.J.M. Verbaarschot, _Quantum chaos transition in a two-site Sachdev-Ye-Kitaev model dual to an eternal traversable wormhole_ , _Phys. Rev. D_ 100 (2019) 026002 [1901.06031].
  * (38) M. Fremling, M. Haque and L. Fritz, _A bipartite Sachdev-Ye-Kitaev model: Conformal limit and level statistics_ , 2111.15215.
  * (39) Z. Cao, Z. Xu and A. del Campo, _Probing quantum chaos in multipartite systems_ , 2111.12475.
  * (40) E. Caceres, A. Misobuchi and R. Pimentel, _Sparse SYK and traversable wormholes_ , _JHEP_ 11 (2021) 015 [2108.08808].
  * (41) J. Maldacena and X.-L. Qi, _Eternal traversable wormhole_ , 1804.00491.
  * (42) P. Gao, D.L. Jafferis and A. Wall, _Traversable Wormholes via a Double Trace Deformation_ , _JHEP_ 12 (2017) 151 [1608.05687].
  * (43) P. Saad, S.H. Shenker and D. Stanford, _JT gravity as a matrix integral_ , 1903.11115.
  * (44) N. Engelhardt, S. Fischetti and A. Maloney, _Free energy from replica wormholes_ , _Phys. Rev. D_ 103 (2021) 046021 [2007.07444].
  * (45) A. Almheiri, T. Hartman, J. Maldacena, E. Shaghoulian and A. Tajdini, _Replica Wormholes and the Entropy of Hawking Radiation_ , _JHEP_ 05 (2020) 013 [1911.12333].
  * (46) A. Almheiri, T. Hartman, J. Maldacena, E. Shaghoulian and A. Tajdini, _The entropy of Hawking radiation_ , _Rev. Mod. Phys._ 93 (2021) 035002 [2006.06872].
  * (47) G. Penington, _Entanglement Wedge Reconstruction and the Information Paradox_ , _JHEP_ 09 (2020) 002 [1905.08255].
  * (48) D.N. Page, _Information in black hole radiation_ , _Phys. Rev. Lett._ 71 (1993) 3743 [hep-th/9306083].
  * (49) J. Maldacena and L. Maoz, _Wormholes in AdS_ , _Journal of High Energy Physics_ 2004 (2004) 053 [hep-th/0401024].
  * (50) P. Saad, S. Shenker and S. Yao, _Comments on wormholes and factorization_ , 2107.13130.
  * (51) A. Belin, J. de Boer, P. Nayak and J. Sonner, _Generalized Spectral Form Factors and the Statistics of Heavy Operators_ , 2111.06373.
  * (52) C.V. Johnson, _The Microstate Physics of JT Gravity and Supergravity_ , 2201.11942.
  * (53) M. Berkooz, N. Brukner, S.F. Ross and M. Watanabe, _Going beyond ER=EPR in the SYK model_ , 2202.11381.
  * (54) J.-M. Schlenker and E. Witten, _No Ensemble Averaging Below the Black Hole Threshold_ , 2202.01372.
  * (55) K. Goto, K. Suzuki and T. Ugajin, _Factorizing Wormholes in a Partially Disorder-Averaged SYK Model_ , 2111.11705.
  * (56) P. Gao and L. Lamprou, _Seeing behind black hole horizons in SYK_ , 2111.14010.
  * (57) A.M. García-García and V. Godet, _Euclidean wormhole in the Sachdev-Ye-Kitaev model_ , _Phys. Rev. D_ 103 (2021) 046014 [2010.11633].
  * (58) A.M. García-García, Y. Jia, D. Rosa and J.J.M. Verbaarschot, _Dominance of Replica Off-Diagonal Configurations and Phase Transitions in a PT Symmetric Sachdev-Ye-Kitaev Model_ , _Phys. Rev. Lett._ 128 (2022) 081601 [2102.06630].
  * (59) C.M. Bender and S. Boettcher, _Real spectra in nonHermitian Hamiltonians having PT symmetry_ , _Phys. Rev. Lett._ 80 (1998) 5243 [physics/9712001].
  * (60) I. Aref’eva, M. Khramtsov, M. Tikhanovskaya and I. Volovich, _Replica-nondiagonal solutions in the SYK model_ , _JHEP_ 07 (2019) 113 [1811.04831].
  * (61) H. Wang, D. Bagrets, A.L. Chudnovskiy and A. Kamenev, _On the replica structure of Sachdev-Ye-Kitaev model_ , _JHEP_ 09 (2019) 057 [1812.02666].
  * (62) J.J.M. Verbaarschot and M.R. Zirnbauer, _Critique of the replica trick_ , _Journal of Physics A: Mathematical and General_ 18 (1985) 1093.
  * (63) M.R. Zirnbauer, _Another critique of the replica trick_ , cond-mat/9903338.
  * (64) K. Splittorff and J.J.M. Verbaarschot, _Factorization of correlation functions and the replica limit of the Toda lattice equation_ , _Nucl. Phys. B_ 683 (2004) 467 [hep-th/0310271].
  * (65) T.A. Sedrakyan, _Toda lattice representation for random matrix model with logarithmic confinement_ , _Nucl. Phys. B_ 729 (2005) 526 [cond-mat/0506373].
  * (66) K. Efetov, _Supersymmetry and theory of disordered metals_ , _Advances in Physics_ 32 (1983) 53.
  * (67) F. Wegner, _Exact density of states for lowest Landau level in white noise potential superfield representation for interacting systems_ , _Zeitschrift fur Physik B Condensed Matter_ 51 (1983) 279.
  * (68) J. Verbaarschot and M. Zirnbauer, _Replica variables, loop expansion, and spectral rigidity of random-matrix ensembles_ , _Annals of Physics_ 158 (1984) 78 .
  * (69) J.J.M. Verbaarschot, H.A. Weidenmüller and M.R. Zirnbauer, _Grassmann integration and the theory of compound-nucleus reactions_ , _Phys. Lett. B_ 149 (1984) 263.
  * (70) T.A. Sedrakyan and K.B. Efetov, _Supersymmetry method for interacting chaotic and disordered systems: The Sachdev-Ye-Kitaev model_ , _Phys. Rev. B_ 102 (2020) 075146 [2002.08963].
  * (71) R. Carlson, _Sur une classe de séries de Taylor_ , Dissertation, Uppsala, Sweden (1914).
  * (72) V.L. Girko, _Theory of random determinants_ , vol. 45, Springer Science & Business Media (2012).
  * (73) S.M. Nishigaki and A. Kamenev, _Replica treatment of non-hermitian disordered hamiltonians_ , _Journal of Physics A: Mathematical and General_ 35 (2002) 4571–4590 [cond-mat/0109126].
  * (74) J. Feinberg and A. Zee, _NonGaussian nonHermitian random matrix theory: Phase transition and addition formalism_ , _Nucl. Phys. B_ 501 (1997) 643 [cond-mat/9704191].
  * (75) R.A. Janik, M.A. Nowak, G. Papp and I. Zahed, _NonHermitian random matrix models. 1._ , _Nucl. Phys. B_ 501 (1997) 603 [cond-mat/9612240].
  * (76) J. Ginibre, _Statistical ensembles of complex, quaternion, and real matrices_ , _Journal of Mathematical Physics_ 6 (1965) 440.
  * (77) Y.V. Fyodorov, B.A. Khoruzhenko and H.-J. Sommers, _Almost hermitian random matrices: Crossover from wigner-dyson to ginibre eigenvalue statistics_ , _Physical Review Letters_ 79 (1997) 557–560 [cond-mat/9703152].
  * (78) Y.V. Fyodorov and H.-J. Sommers, _Random matrices close to hermitian or unitary: overview of methods and results_ , _Journal of Physics A: Mathematical and General_ 36 (2003) 3303–3347 [arXiv:nlin/0207051].
  * (79) Y. Can, A.M. García-García, V. Godet and J.P. Zehn, _to be published_ , .
  * (80) L. Erdős and D. Schröder, _Phase transition in the density of states of quantum spin glasses_ , _Mathematical Physics, Analysis and Geometry_ 17 (2014) 441–464 [arXiv:1407.1552].
  * (81) D. Bagrets, A. Altland and A. Kamenev, _Sachdev–Ye–Kitaev model as Liouville quantum mechanics_ , _Nucl. Phys. B_ 911 (2016) 191 [1607.00694].
  * (82) D. Bagrets, A. Altland and A. Kamenev, _Power-law out of time order correlation functions in the SYK model_ , _Nucl. Phys. B_ 921 (2017) 727 [1702.08902].
  * (83) Y. Jia, D. Rosa and J.J.M. Verbaarschot, _Replica Symmetry Breaking for the Integrable Two-Site Sachdev-Ye-Kitaev Model_ , 2201.05952.
  * (84) L.F. Richardson, _The approximate arithmetical solution by finite differences of physical problems involving differential equations, with an application to the stresses in a masonry dam_ , _Philosophical Transactions of the Royal Society A_ 210 (459-470) (1911) 307.
  * (85) A. Almheiri, T. Hartman, J. Maldacena, E. Shaghoulian and A. Tajdini, _Replica wormholes and the entropy of hawking radiation_ , _Journal of High Energy Physics_ 2020 (2020)  [1911.12333].
  * (86) G. Penington, S.H. Shenker, D. Stanford and Z. Yang, _Replica wormholes and the black hole interior_ , 2020.
  * (87) M.L. Mehta, _Random matrices_ , Academic press (2004).


# XLS-R: Self-supervised Cross-lingual Speech Representation Learning at Scale

Arun Babu△ , Changhan Wang△ , Andros Tjandra△, Kushal Lakhotia◇ , Qiantong
Xu△,

Naman Goyal△, Kritika Singh△, Patrick von Platen♣, Yatharth Saraf△, Juan
Pino△,

Alexei Baevski△, Alexis Conneau□ , Michael Auli△

△ Meta AI □ Google AI ◇ Outreach ♣ Hugging Face
Equal contribution.Work done while at Facebook AI.Equal advising.

###### Abstract

This paper presents XLS-R, a large-scale model for cross-lingual speech
representation learning based on wav2vec 2.0. We train models with up to 2B
parameters on nearly half a million hours of publicly available speech audio
in 128 languages, an order of magnitude more public data than the largest
known prior work. Our evaluation covers a wide range of tasks, domains, data
regimes and languages, both high and low-resource. On the CoVoST-2 speech
translation benchmark, we improve the previous state of the art by an average
of 7.4 BLEU over 21 translation directions into English. For speech
recognition, XLS-R improves over the best known prior work on BABEL, MLS,
CommonVoice as well as VoxPopuli, lowering error rates by 14-34% relative on
average. XLS-R also sets a new state of the art on VoxLingua107 language
identification. Moreover, we show that with sufficient model size, cross-
lingual pretraining can perform as well as English-only pretraining when
translating English speech into other languages, a setting which favors
monolingual pretraining. We hope XLS-R can help to improve speech processing
tasks for many more languages of the world. Models and code are available at
www.github.com/pytorch/fairseq/tree/master/examples/wav2vec/xlsr.111Hugging
Face: https://huggingface.co/models?other=xls_r

##  1 Introduction

Self-supervised learning of generic neural representations has gathered much
recent interest with a large body of work in natural language processing (NLP;
Radford et al. 2018; Baevski et al. 2019; Devlin et al. 2019; Raffel et al.
2019), computer vision (Chen et al., 2020; He et al., 2020; Caron et al.,
2021) as well as speech processing (van den Oord et al., 2018; Schneider et
al., 2019; Baevski et al., 2020b; Hsu et al., 2021b; Chung et al., 2021).
Self-supervised learning provides general representations that can be used
across domains and languages.

Multilingually pretrained NLP models such as mBERT (Devlin et al., 2019),
XLM-R (Conneau et al., 2020) or mT5 (Xue et al., 2020) brought significant
improvements in multilingual language understanding (Conneau et al., 2018; Hu
et al., 2020; Ruder et al., 2021). These models offer a promising path towards
more ubiquitous NLP technology by improving performance for low-resource
languages through leveraging data from high-resource languages. Furthermore,
it is only necessary to maintain a single multilingual model instead of a
myriad of monolingual models.

For speech processing, self-supervised approaches such as wav2vec 2.0 (Baevski
et al., 2020b; Xu et al., 2021) have also been extended to the multilingual
setting (Kawakami et al., 2020; Conneau et al., 2021). The recent XLSR
(Conneau et al., 2021) leverages cross-lingual transfer from high-resource
languages to build better representations for languages with little unlabeled
data. The largest model, XLSR-53, was trained on about 50K hours of public
training data in 53 languages and comprises about 300M parameters (Conneau et
al., 2021). But such models only scratch the surface of self-supervised cross-
lingual speech representation learning.

In natural language processing, language models are trained on very large
datasets, spanning billions of documents such as CC100 (Wenzek et al., 2020)
or mC4 (Xue et al., 2020) to fit models with tens of billions and even
trillions of parameters (Brown et al., 2020; Goyal et al., 2021; Fedus et al.,
2021) with strong results on established benchmarks. In contrast, scaling
efforts in speech have focused either on supervised multilingual models (Li et
al., 2021a) or monolingual self-supervised models, counting a billion or more
parameters (Zhang et al., 2020; 2021), while cross-lingually pretrained speech
models are much smaller in scale.

To this end, we present XLS-R, a large-scale cross-lingually pretrained
wav2vec 2.0 model (see illustration in Figure 1) whose name is inspired by
XLM-R in NLP. It leverages new publicly available VoxPopuli data, comprising
372K hours of unannotated speech (Wang et al., 2021a), the MLS corpus (Pratap
et al., 2020), CommonVoice (Ardila et al., 2020), BABEL (Gales et al., 2014)
and VoxLingua107 (Valk & Alumäe, 2020) to cover 128 different languages from
various regions of the world. To our knowledge, this is the largest effort to
date, in making speech technology accessible for many more languages using
publicly available data.

Figure 1: Self-supervised cross-lingual representation learning. We pre-train
a large multilingual wav2vec 2.0 Transformer (XLS-R) on 436K hours of
unannotated speech data in 128 languages. The training data is from different
public speech corpora and we fine-tune the resulting model for several
multilingual speech tasks.

##  2 Background

Our work builds on Conneau et al. (2021) who pretrain wav2vec 2.0 models on
data from multiple languages. wav2vec 2.0 contains a convolutional feature
encoder $f:\mathcal{X}\mapsto\mathcal{Z}$ to map raw audio $\mathcal{X}$ to
latent speech representations $z_{1},\dots,z_{T}$ which are input to a
Transformer $g:\mathcal{Z}\mapsto\mathcal{C}$ to output context
representations $c_{1},\dots,c_{T}$ (Baevski et al., 2020a). Each $z_{t}$
represents 25ms of audio strided by 20ms and the Transformer architecture
follows BERT (Vaswani et al., 2017; Devlin et al., 2019).

During training, feature encoder representations are discretized to
$q_{1},\dots,q_{T}$ with a quantization module $\mathcal{Z}\mapsto\mathcal{Q}$
to represent the targets in the objective. The quantization module uses a
Gumbel softmax to choose entries form the codebooks and the chosen entries are
concatenated to obtain $q$ (Jegou et al., 2011; Jang et al., 2016; Baevski et
al., 2020a).

The model is trained by solving a contrastive task over masked feature encoder
outputs. At training time, spans of ten time steps with random starting
indices are masked. The objective requires identifying the true quantized
latent $q_{t}$ for a masked time-step within a set of $K=100$ distractors
$\mathbf{Q}_{t}$ sampled from other masked time steps.
$-\log\frac{\exp(sim(c_{t},q_{t}))}{\sum_{\tilde{q}\sim\mathbf{Q}_{t}}\exp(sim(c_{t},\tilde{q}))}$
where $c_{t}$ is the output of the Transformer, and
$sim(\mathbf{a},\mathbf{b})$ denotes cosine similarity. The objective is
augmented by a codebook diversity penalty to encourage the model to use all
codebook entries (Dieleman et al., 2018).

The model is trained on multiple languages to obtain cross-lingual
representations. Specifically, training batches contain samples from multiple
languages $L$ (Devlin et al., 2019; Lample & Conneau, 2019; Conneau et al.,
2021) by sampling from a distribution
$p_{l}\sim\left(\frac{n_{l}}{N}\right)^{\alpha}$ where $l=1,\ldots,L$, while
$n_{l}$ is the amount of unlabeled data for each language, and $\alpha$ is the
upsampling factor which controls the trade-off between high- and low-resource
languages during pretraining.

##  3 Data and Evaluation

In this section, we outline the datasets on which XLS-R is pretrained as well
as the language coverage. We also describe the downstream tasks on which we
evaluate our models.

###  3.1 Training Data

We pretrain our models on a total of 436K hours of publicly available data
from the following sources:

  * •

VoxPopuli (VP-400K) comprises a total of 372K hours of data222There are 400K
hours before removing leading and trailing silences. in 23 European languages
of parliamentary speech from the European parliament (Wang et al., 2021a).
This makes it the largest publicly available speech corpus for semi-supervised
learning.

  * •

Multilingual Librispeech (MLS) contains data in eight European languages
totaling around 50K hours of data (Pratap et al., 2020). The majority of the
data is English (44K hours).

  * •

CommonVoice (CV) is a corpus of read speech. We use the December 2020 release
(v6.1; Ardila et al. 2020) which covers 60 languages and over 7K hours of
speech audio, ranging from over 1.6K hours for English to less than one hour
for languages such as Hindi.

  * •

VoxLingua107 (VL) is a dataset of 6.6K hours of data in 107 languages based on
YouTube content (Valk & Alumäe, 2020) with an average of 62 hours of data per
language.

  * •

BABEL (BBL) is a multilingual corpus of conversational telephone speech of
about 1K hours of data in 17 African and Asian languages (Gales et al., 2014).

110100100010000100000High resource $\leftarrow$ (English, German, …) Low
resource $\rightarrow$ (Sinhala, Zulu, …)Hours of training data Figure 2:
Illustration of the unlabeled training data distribution across the 128
languages of XLS-R.

To the best of our knowledge, this is the largest dataset used for training a
publicly available self-supervised speech model to date. Figure 2 shows the
data distribution across the 128 different languages in our training dataset.
There are about 24 high-resource languages with more than 1K hours of data
each, almost all of which are European, except for Kinyarwanda which is
African. Then there is a small number of 17 mid-resource languages which have
more than 100 hours of data (but less than 1K hours) which includes Catalan,
Persian, Turkish, Russian, and Basque. Finally, the remaining 88 languages are
low-resource and have less than 100 hours of data each. Table 1 lists all the
languages, together with their ISO code, language family, sub-grouping and the
amount of training data.

Language | ISO | Family | Sub-grouping | Train data (h)
---|---|---|---|---
Abkhazian | ab | Abkhaz-Adyge | Abkhaz-Abazin | <1
Afrikaans | af | Indo-European | Germanic | 87
Albanian | sq | Indo-European | Albanian | 56
Amharic | am | Afro-Asiatic | Semitic | 65
Arabic | ar | Afro-Asiatic | Semitic | 95
Armenian | hy | Indo-European | Armenian | 55
Assamese | as | Indo-European | Indo-Iranian | 179
Azerbaijani | az | Turkic | Southern | 47
Bashkir | ba | Turkic | Western | 47
Basque | eu | Language isolate |  | 113
Belarusian | be | Indo-European | Balto-Slavic | 106
Bengali | bn | Indo-European | Indo-Iranian | 100
Bosnian | bs | Indo-European | Balto-Slavic | 83
Breton | br | Indo-European | Celtic | 42
Bulgarian | bg | Indo-European | Balto-Slavic | 17616
Burmese | my | Sino-Tibetan | Tibeto-Burman | 33
Cantonese | yue | Sino-Tibetan | Chinese | 130
Catalan | ca | Indo-European | Italic | 691
Cebuano | ceb | Austronesian | Malayo-Polynesian | 42
Central Khmer | km | Austro-Asiatic | Mon-Khmer | 33
Chinese CN | zh | Sino-Tibetan | Chinese | 90
Chinese HK | zh-HK | Sino-Tibetan | Chinese | 51
Chinese TW | zh-TW | Sino-Tibetan | Chinese | 53
Chuvash | cv | Turkic | Bolgar | 4
Croatian | hr | Indo-European | Balto-Slavic | 2520
Czech | cs | Indo-European | Balto-Slavic | 18514
Danish | da | Indo-European | Germanic | 13588
Divehi | dv | Indo-European | Indo-Iranian | 18
Dutch | nl | Indo-European | Germanic | 20070
English | en | Indo-European | Germanic | 69493
Esperanto | eo | Constructed language |  | 97
Estonian | et | Uralic | Coastal Finnic | 10652
Faroese | fo | Indo-European | Germanic | 54
Finnish | fi | Uralic | Finnic | 13981
French | fr | Indo-European | Italic | 23973
Galician | gl | Indo-European | Italic | 57
Ganda | lg | Atlantic-Congo | Volta-Congo | 3
Georgian | ka | Kartvelian | Georgian | 127
German | de | Indo-European | Germanic | 25378
Greek | el | Indo-European | Greek | 17761
Guarani | gn | Tupian | Tupí-Guaraní | 2
Gujarati | gu | Indo-European | Indo-Iranian | 37
Haitian | ht | Creole | French-based | 138
Hakha Chin | cnh | Sino-Tibetan | Tibeto-Burman | 2
Hausa | ha | Afro-Asiatic | Chadic | 75
Hawaiian | haw | Austronesian | Malayo-Polynesian | 10
Hebrew | he | Afro-Asiatic | Semitic | 77
Hindi | hi | Indo-European | Indo-Iranian | 65
Hungarian | hu | Uralic |  | 17421
Icelandic | is | Indo-European | Germanic | 73
Indonesian | id | Austronesian | Malayo-Polynesian | 41
Interlingua | ia | Constructed language |  | 9
Irish | ga | Indo-European | Celtic | 3
Italian | it | Indo-European | Italic | 21943
Japanese | ja | Japonic |  | 49
Javanese | jv | Austronesian | Malayo-Polynesian | 42
Kabyle | kab | Afro-Asiatic | Berber | 520
Kannada | kn | Dravidian | Southern | 36
Kazakh | kk | Turkic | Western | 98
Kinyarwanda | rw | Atlantic-Congo | Volta-Congo | 1199
Language | ISO | Family | Sub-grouping | Train data (h)
---|---|---|---|---
Kyrgyz | ky | Turkic | Western | 11
Korean | ko | Koreanic |  | 61
Kurmanji | ku | Indo-European | Indo-Iranian | 38
Lao | lo | Kra-Dai | Kam-Tai | 93
Latin | la | Indo-European | Italic | 53
Latvian | lv | Indo-European | Balto-Slavic | 13126
Lingala | ln | Atlantic-Congo | Volta-Congo | 72
Lithuanian | lt | Indo-European | Balto-Slavic | 14423
Luxembourgish | lm | Indo-European | Germanic | 60
Macedonian | mk | Indo-European | Balto-Slavic | 89
Malagasy | mg | Austronesian | Malayo-Polynesian | 87
Malay | ms | Austronesian | Malayo-Polynesian | 66
Malayalam | ml | Dravidian | Southern | 38
Maltese | mt | Afro-Asiatic | Semitic | 9120
Manx | gv | Indo-European | Celtic | 3
Maori | mi | Austronesian | Malayo-Polynesian | 27
Marathi | mr | Indo-European | Indo-Iranian | 68
Mongolian | mn | Mon-Khi | Mongolic | 68
Nepali | ne | Indo-European | Indo-Iranian | 58
Norwegian | no | Indo-European | Germanic | 85
Nynorsk | nn | Indo-European | Germanic | 45
Occitan | oc | Indo-European | Italic | 12
Oriya | or | Indo-European | Indo-Iranian | 1
Pashto | ps | Indo-European | Indo-Iranian | 109
Persian | fa | Indo-European | Indo-Iranian | 321
Polish | pl | Indo-European | Balto-Slavic | 20912
Portuguese | pt | Indo-European | Italic | 17797
Punjabi | pa | Indo-European | Indo-Iranian | 43
Romanian | ro | Indo-European | Italic | 17515
Romansh Sursilvan | rm | Indo-European | Italic | 6
Romansh Vallader | rm | Indo-European | Italic | 2
Russian | ru | Indo-European | Balto-Slavic | 166
Sakha | sah | Turkic | Northern | 4
Sanskrit | sa | Indo-European | Indo-Iranian | 12
Scots | sco | Indo-European | Germanic | 2
Serbian | sr | Indo-European | Balto-Slavic | 40
Shona | sn | Atlantic-Congo | Volta-Congo | 24
Sindhi | sd | Indo-European | Indo-Iranian | 67
Sinhala | si | Indo-European | Indo-Iranian | 54
Slovakian | sk | Indo-European | Balto-Slavic | 11925
Slovenian | sl | Indo-European | Balto-Slavic | 11210
Somali | so | Afro-Asiatic | Cushitic | 82
Sorbian Upper | hsb | Indo-European | Balto-Slavic | 2
Spanish | es | Indo-European | Italic | 22258
Sundanese | su | Austronesian | Malayo-Polynesian | 51
Swahili | sw | Atlantic-Congo | Volta-Congo | 91
Swedish | sv | Indo-European | Germanic | 16325
Tagalog | tl | Austronesian | Malayo-Polynesian | 150
Tajik | tg | Indo-European | Indo-Iranian | 51
Tamil | ta | Dravidian | Dravidian | 118
Tatar | tt | Turkic | Western | 107
Telugu | te | Dravidian | South-Central | 62
Thai | th | Kra-Dai | Kam-Tai | 57
Tibetan | bo | Sino-Tibetan | Tibeto-Burman | 81
Tok | tpi | Creole | English-based | 36
Turkish | tr | Turkic | Southern | 136
Turkmen | tk | Turkic | Southern | 68
Ukrainian | uk | Indo-European | Balto-Slavic | 72
Urdu | ur | Indo-European | Indo-Iranian | 34
Uzbek | uz | Turkic | Eastern | 36
Language | ISO | Family | Sub-grouping | Train data (h)
---|---|---|---|---
Vietnamese | vi | Austro-Asiatic | Mon-Khmer | 131
Votic | vot | Uralic | Finnic | <1
Waray | war | Austronesian | Malayo-Polynesian | 9
Welsh | cy | Indo-European | Celtic | 156
Western Frisian | fy-NL | Indo-European | Germanic | 15
Yiddish | yi | Indo-European | Germanic | 37
Yoruba | yo | Atlantic-Congo | Volta-Congo | 75
Zulu | zu | Atlantic-Congo | Volta-Congo | 56
Table 1: Languages on which XLS-R is trained on together with their ISO code,
language family, sub-grouping and the amount of data for pretraining.

###  3.2 Downstream Evaluation

We evaluate on a broad and diverse set of downstream tasks to showcase the
generalization ability of our pretrained models across tasks, data regimes,
domains and languages.

####  3.2.1 Automatic Speech Translation (AST)

##### CoVoST-2 AST.

For speech translation evaluation we adopt CoVoST-2 (Wang et al., 2020), a
multilingual speech translation benchmark based on CommonVoice (Ardila et al.,
2020).333https://github.com/facebookresearch/covost It provides data for
translating from English into 15 languages (En $\rightarrow$ X) and from 21
languages into English (X $\rightarrow$ En). The En $\rightarrow$ X languages
are: Arabic (ar), Catalan (ca), Welsh (cy), German (de), Estonian (et),
Persian (fa), Indonesian (id), Japanese (ja), Latvian (lv), Mongolian (mn),
Slovenian (sl), Swedish (sv), Tamil (ta), Turkish (tr), Chinese (zh) where
each direction comprises about 430 hours of training data. The X $\rightarrow$
En languages include all target languages of En $\rightarrow$ X as well as
Spanish (es), French (fr), Italian (it), Dutch (nl), Portuguese (pt). We group
the latter into high-resource (136-264h of train data; fr, de, es, ca), mid-
resource (10-49h of train data; fa, it, ru, pt, zh), and low-resource (2-7h of
train data; tr, ar, et , mn, nl, sv, lv, sl, ta, ja, id, cy) for ease of
presentation.

####  3.2.2 Automatic Speech Recognition (ASR)

##### BABEL ASR.

BABEL is a challenging speech recognition benchmark from IARPA consisting of
noisy telephone conversational data.444https://catalog.ldc.upenn.edu/byyear
LDC2016S06, LDC2016S13, LDC2017S05, LDC2017S08, LDC2016S12 We evaluate on five
languages: Assamese (as), Tagalog (tl), Swahili (sw), Lao (lo), and Georgian
(ka). Training sets comprise between 30 and 76 hours of annotated data.
Following Conneau et al. (2021), we use 10% of the training set for
validation, and report test results on the BABEL dev set. We report word error
rate (WER) and use n-gram language models trained on CommonCrawl data.

##### Multilingual LibriSpeech ASR.

Multilingual LibriSpeech (MLS; Pratap et al. 2020) is a large corpus derived
from read audiobooks of Librivox and consists of eight European languages:
Dutch (nl), English (en), French (fr), German (de), Italian (it), Polish (pl),
Portuguese (pt), Spanish
(es).555https://github.com/flashlight/wav2letter/blob/main/recipes/mls
Training sets comprise between 104 hours for Polish and 44.7K hours for
English. We use the 10 hour training splits of Conneau et al. (2021) and
report word error rate with the official n-gram models provided by the MLS
dataset.

##### CommonVoice ASR.

Following Rivière et al. (2020), we use ten languages of CommonVoice for ASR
evaluation: Spanish (es), French (fr), Italian (it), Kyrgyz (ky), Dutch (nl),
Russian (ru), Swedish (sv), Turkish (tr), Tatar (tt) and Chinese-Hong Kong
(zh-
HK).666https://dl.fbaipublicfiles.com/cpc_audio/common_voices_splits.tar.gz
CommonVoice contains read speech primarily from Wikipedia sentences. Following
prior work (Rivière et al., 2020; Conneau et al., 2021), we fine-tune models
on just one hour of labeled data per language, a few-shot scenario. Results
are reported in terms of phoneme error rate (PER) without a language model.

##### VoxPopuli ASR.

Following Wang et al. (2021a), we evaluate on languages which have at least
ten hours of labeled data which is a total of 14 languages: English (en),
German (de), Italian (it), French (fr), Spanish (es), Polish (pl), Romanian
(ro), Hungarian (hu), Dutch (nl), Czech (cs), Slovenian (sl), Finnish (fi),
Croatian (hr), Slovakian (sk).777https://github.com/facebookresearch/voxpopuli
Models are fine-tuned on the full train set, which ranges from 543 hours
(English) to ten hours (Slovenian). We report word error rate without language
models.

##### LibriSpeech ASR.

LibriSpeech is a widely-used evaluation benchmark for speech recognition
research (Panayotov et al., 2015). It consists of 960 hours of English
annotated data. Following Baevski et al. (2020b), we use the 10 minute, 1 hour
and 10 hour training splits. We compare the performance of XLS-R models
against English-only wav2vec 2.0 models. We report word error rate without
language models.

####  3.2.3 Speech classification (LID and Speaker ID)

##### VoxLingua107 LangID.

For spoken language identification we consider VoxLingua107 (Valk & Alumäe,
2020) which spans 107 languages.888http://bark.phon.ioc.ee/voxlingua107/ It
consists of short speech segments automatically extracted from YouTube videos.
The total amount of speech data in the training set is 6,628 hours and the
average per language is 62 hours. We report accuracy on the official test set
which covers 33 different languages.

##### VoxCeleb1 SpeakerID.

We use VoxCeleb1 for speaker identification (Nagrani et al.,
2017).999https://www.robots.ox.ac.uk/~vgg/data/voxceleb/vox1.html VoxCeleb1 is
an audio-visual dataset consisting of short clips of human speech, extracted
from interview videos uploaded to YouTube. It consists of 1,251 unique
speakers and 153K utterances. We use the official dataset splits.

##  4 Experimental Setup

In this section, we give more details on architectures and hyperparameters
used during pretraining and finetuning.

###  4.1 Pretraining

We use the wav2vec 2.0 implementation available in fairseq (Ott et al., 2019)
and evaluate several model architectures detailed in Table 2. We consider
models with between 0.3B parameters to 2B parameters. To optimize GPU memory
usage, we use a fully sharded backend (Rajbhandari et al., 2021) as well as
activation checkpointing (Chen et al., 2016) as implemented in FairScale
(Baines et al., 2021).

Models are optimized with Adam (Kingma & Ba, 2015) and the learning rate is
warmed up for the first 32K steps followed by polynomial decay to zero for the
remainder of training. Training audio sequences are cropped to a maximum of
320K samples, or 20 seconds and all models were pretrained for a total of one
million updates. XLS-R (0.3B) was trained on 128 GPUs with nearly 2M samples
on each GPU, totaling about 4.3h of data in a batch. Larger models were
trained on 200 GPUs with 800K to 1M samples on each GPU giving an effective
batch size of about 2.8-3.6 hours.

Our training data covers 128 languages and five training corpora with
different characteristics. To balance data from the different languages and
corpora we upsample both training corpora and languages. We first upsample the
languages within a particular corpus using the strategy outlined in §2 and
then balance the different corpora using the same strategy by treating each
corpus as a different language. We use $\alpha=0.5$ in all cases.

Model | #lgs | Train datasets | $B$ | $H_{m}$ | $H_{ff}$ | $A$ | #params
---|---|---|---|---|---|---|---
XLSR-53 | 53 | MLS, CV, BBL | 24 | 1024 | 4096 | 16 | 317M
VP-100K | 23 | VP-100K | 24 | 1024 | 4096 | 16 | 317M
XLS-R (0.3B) | 128 | VP-400K, MLS, CV, VL, BBL | 24 | 1024 | 4096 | 16 | 317M
XLS-R (1B) | 128 | VP-400K, MLS, CV, VL, BBL | 48 | 1024 | 4096 | 16 | 965M
XLS-R (2B) | 128 | VP-400K, MLS, CV, VL, BBL | 48 | 1920 | 7680 | 16 | 2162M

Table 2: Model architectures. We show details for prior work, XLSR-53 (Conneau
et al., 2021), VP-100K (Wang et al., 2021a), and the XLS-R models: the number
of languages (#lgs), the pretraining data (Datasets), the number of blocks
($B$), the number of hidden states ($H_{m}$), the inner dimension of feed-
forward blocks ($H_{ff}$), the number of attention heads (A) and the total
number of parameters (#params).

###  4.2 Speech translation

To build speech translation models, we multilingually fine-tune XLS-R models
by training on the combined labeled data of all En $\rightarrow$ X or X
$\rightarrow$ En language directions without upsampling any direction. We
stack a decoder network on top of XLS-R which is a Transformer network with 12
layers, embedding size 1024, 16 attention heads and feed forward network
dimension 4096. The decoder network is initialized with weights from
multilingually fine-tuned mBART (Liu et al., 2020; Li et al., 2021b; Tang et
al., 2021) and uses the same vocabulary with 250K types. The total size of the
decoder network is 459M parameters.

In our ablations, we also consider bilingually fine-tuned speech translation
models which use a much smaller decoder of 16M parameters which has seven
layers, embedding size 256, 4 attention heads and feed forward network
dimension 2048. For this, a 10K byte-pair encoding (BPE; Sennrich et al. 2016)
vocabulary is built on the CoVoST 2 target text for each target language.

We fine-tune with Adam (Kingma & Ba, 2015), a learning rate of 3e-4, label
smoothing with probability 0.1, an effective batch size of 66M samples, or
nearly 68 minutes, layer drop 0.05, a masking strategy similar to wav2vec 2.0
with mask length 5 and mask probability 0.15. During fine-tuning, the wav2vec
2.0 encoder is not updated for the first 10K updates. Models are fine-tuned
for 250K updates in total and the best checkpoint is selected based on
validation BLEU. We choose the learning rate by searching in the interval
$[3e-5,3e-4]$. Translations are generated with a beam size of 5.

###  4.3 Automatic Speech Recognition

For fine-tuning, we follow the settings of Baevski et al. (2020b) by adding a
linear layer on top of the pretrained model to predict the output vocabulary
and train using Connectionist Temporal Classification (CTC; Graves et al.
2006). The output vocabulary is characters for all benchmarks, except for
CommonVoice where we use phonemes. We fine-tune using Adam and the learning
rate is warmed up for the first 10% of total updates, kept constant for the
next 40% and then decayed to zero in the remaining 50% of updates. Since the
amount of labeled data differs widely for each dataset we found the following
number of training updates to be effective: 20K updates for BABEL, 13K updates
for CommonVoice, 20K updates for MLS and 50K updates for VoxPopuli.

We found large batch sizes to be very effective. For XLS-R (0.3B) and XLS-R
(1B), we use an effective batch size of 0.44 hours and for XLS-R (2B) we used
1.06 hours. Learning rate as well as batch size was tuned based on dev error
rate and we searched the range $[2e-5,3e-4]$ for XLS-R (0.3B) and XLS-R (1B)
as well as $[3e-6,3e-5]$ for XLS-R (2B). To reduce overfitting for XLS-R (2B),
we increase stochastic depth to 15% compared to 10% for all other setups
(Huang et al., 2016; Fan et al., 2019). We use a masking probability of
between 30-75%, depending on the setup and which is determined on the
development set.

For MLS and BABEL, we use a language model for decoding. We tune the language
model weight within the interval $[0,5]$ and a word insertion penalty within
the interval $[-5,5]$ using Bayesian
optimization.101010https://github.com/facebook/Ax We run 128 trials with beam
500 and choose the best set of parameters according to the dev error rate. For
Common Voice and Vox Populi we do not use a language model.

##  5 Results

Next, we analyze the results of our pretrained models on all downstream tasks.

###  5.1 Speech Translation

We conduct an extensive study on the CoVoST-2 speech translation benchmark.
The task entails translating speech audio in one language into another
language with text as output. Performance is evaluated in terms of BLEU
(Papineni et al., 2002). Models are simultaneously fine-tuned either on all 21
translation directions with English as target language (X $\rightarrow$ En) or
on all the 15 directions where English is the input language (En $\rightarrow$
X; see §3.2.1), resulting in only two models instead of 36.

####  5.1.1 X $\rightarrow$ English

For X $\rightarrow$ English directions we group languages into high-resource,
mid-resource and low-resource directions (§3.2.1) and compare to several
baselines: in order to directly compare to XLSR-53 (Conneau et al., 2021), and
VP-100K (Wang et al., 2021a), we fine-tune these publicly available models
following the same protocol as XLS-R. We also compare to Li et al. (2021b),
the best known results from the literature who either use an English-
pretrained wav2vec 2.0 model (XMEF-En) for En $\rightarrow$ X directions or
the multilingually pretrained XLSR-53 (XMEF-X) for X $\rightarrow$ En
directions.

Table 3 shows a new state of the art with XLS-R (2B), improving over the
previous best result (Li et al., 2021b), by 7.4 BLEU on average over all 21
directions (14.7 BLEU vs. 22.1 BLEU). This is largely due to improvements on
mid-resource (+7.5 BLEU) and low-resource (+9.2 BLEU) language directions.
Model capacity has a large impact: XLS-R (1B) improves over XLS-R (0.3B) by an
average of 6.1 BLEU and XLS-R (2B) improves by an average of 2.8 BLEU compared
to XLS-R (1B). Appendix A shows detailed results for all translation
directions.

There is a trend of larger capacity in pretrained models enabling few-shot
learning for speech translation, similar to wav2vec 2.0 enabling few-shot
speech recognition (Baevski et al., 2020b; Xu et al., 2020). For example, on
language pairs with only two hours of labeled speech translation data, XLS-R
(2B) improves over XLS-R (0.3B) as follows: from 10.3 BLEU to 29.6 BLEU on
Swedish-English, from 1.4 BLEU to 16.5 BLEU on Indonesian-English and from 3.0
BLEU to 17.1 BLEU on Arabic-English (see Appendix A).

Table 3: Speech translation: results for X $\rightarrow$ English directions on CoVoST-2 in terms of average BLEU for 21 directions grouped into high/mid/low-resource labeled data directions. Appendix A lists detailed results for all languages.  | high | mid | low | Avg.
---|---|---|---|---
Prior work |
XLSR-53 (Conneau et al., 2021) | 30.3 | 11.1 | 3.2 | 10.3
VP-100K (Wang et al., 2021b) | 27.7 | 13.2 | 4.6 | 11.1
XMEF-En (Li et al., 2021b) | 32.4 | 16.8 | 4.0 | 12.4
XMEF-X (Li et al., 2021b) | 34.2 | 20.2 | 5.9 | 14.7
This work |  |  |  |
XLS-R (0.3B) | 30.6 | 18.9 | 5.1 | 13.2
XLS-R (1B) | 34.3 | 25.5 | 11.7 | 19.3
XLS-R (2B) | 36.1 | 27.7 | 15.1 | 22.1

####  5.1.2 English $\rightarrow$ X

For English $\rightarrow$ X directions we compare to previous cross-lingually
pretrained models (XLSR-53, VP-100K) as well as baselines with English-only
pretraining: XMEF JT, the best performing setup of Li et al. (2021b) for En
$\rightarrow$ X directions as well as wav2vec 2.0 pre-trained on 60K hours of
English Libri-light data and fine-tuned following the same protocol as XLS-R
(Kahn et al., 2020; Baevski et al., 2020b). The latter has the advantage of
being pre-trained on exactly the same language as the input data for all
translation directions while cross-lingually pretrained models need to be able
to represent many different languages which puts them at a disadvantage.

Table 4 shows that XLSR-53 now performs similarly to XLS-R (0.3B) while for X
$\rightarrow$ English XLS-R (0.3B) performed much better (see §5.1.1). This is
likely because English data dominates the training corpus of XLSR-53 which is
not the case for XLS-R (§3.1). Both XLS-R (1B) and XLS-R (2B) outperform XMEF
JT showing that larger capacity results in better performance.

We also compare to prior work using the English-only pretrained wav2vec 2.0
LV-60K model (Wang et al., 2021c) which additionally uses self-training and a
language model for decoding. We do not use these techniques. Their results
represent the state of the art on these four directions. Wang et al. (2021c)
achieves an average BLEU of 25.6 on the four directions while as XLS-R (2B)
rivals this at an average BLEU of 25.5. We note that self-training and LM
decoding methods are equally applicable to our approach.

XLS-R (2B) also performs well compared to English-only pretraining at 27.8
average BLEU compared to 26.6 BLEU for a wav2vec 2.0 model pretrained on 60K
hours of Libri-light data and 720M parameters. This confirms that with
sufficient capacity, cross-lingual pretraining can perform as well as strong
monolingual models (Conneau et al., 2021).

Table 4: Speech translation: results for English $\rightarrow$ X directions on
CoVoST-2 in terms of BLEU. We show detailed results for four language pairs:
English-German (en-de), English-Catalan (en-ca), English-Arabic (en-ar) and
English-Turkish (en-tr) as well as the average performance over all 15
directions. For faster experimental turnaround we do not use self-training and
LM-decoding as Wang et al. (2021c) and we expect these methods to be equally
applicable to XLS-R. Appendix A lists detailed results for all translation
directions.
| en-ca | en-ar | en-de | en-tr | Avg. (15 dir)
---|---|---|---|---|---
Prior work |  |
XLSR-53 (Conneau et al., 2021) | 29.0 | 16.5 | 23.6 | 15.3 | 23.4
VP-100K (Wang et al., 2021b) | 26.1 | 14.5 | 20.8 | 13.5 | 20.9
XMEF JT (Li et al., 2021b) | 30.9 | 18.0 | 25.8 | 17.0 | 25.1
wav2vec 2.0 LV-60K (Wang et al., 2021c) | 32.4 | 17.4 | 23.8 | 15.4 | -
\+ self-training + LM (Wang et al., 2021c) | 35.6 | 20.8 | 27.2 | 18.9 | -
This work - monolingual pretraining |  |  |  |  |
wav2vec 2.0 LV-60K (720M) | 32.7 | 19.4 | 27.0 | 17.7 | 26.6
This work - cross-lingual pretraining |  |  |  |  |
XLS-R (0.3B) | 28.7 | 16.3 | 23.6 | 15.0 | 23.2
XLS-R (1B) | 32.1 | 19.2 | 26.2 | 17.1 | 26.0
XLS-R (2B) | 34.2 | 20.7 | 28.3 | 18.6 | 27.8

####  5.1.3 Ablations

We build speech translation models by adopting two design decision from Li et
al. (2021b): multilingual fine-tuning of pretrained models on labeled speech
translation data in multiple translation directions and initializing the
decoder network with mBART (Li et al., 2021b; Tang et al., 2021). In the
following we ablate these two choices to better understand their impact.

We first compare multilingual fine-tuning to bilingual fine-tuning. For faster
experimental turn-around we consider a reduced setup of four English
$\rightarrow$ X language directions (en-ca, en-ar, en-de, en-tr) as well as
all high-resource and mid-resource X $\rightarrow$ English directions.111111We
also did not use mBART initialization for this ablation. We compare bilingual
fine-tuning to models fine-tuned on all 15 English $\rightarrow$ X or all 21 X
$\rightarrow$ English directions.

Table 5 shows that multilingual fine-tuning is particularly effective for X
$\rightarrow$ English directions where the average improvement is 3.3 BLEU
(20.9 BLEU to 24.2 BLEU). The amount of labeled data ranges from 264 hours for
French $\rightarrow$ English to 10 hours for Chinese $\rightarrow$ English and
multilingual fine-tuning leverages supervision from high-resource languages to
improve performance for languages with less labeled data. Languages with less
data benefit both from cross-lingual transfer during pretraining, through
training on unlabeled data in other languages, and fine-tuning, through
labeled data from other languages (Arivazhagan et al., 2019; Conneau et al.,
2021). For English $\rightarrow$ X, multilingual fine-tuning performs roughly
on par to bilingual fine-tuning which is a desirable outcome given that
transfer between language directions is diminished. This is supported by the
larger size of the decoder network in multilingual fine-tuning (§3.2.1).

Table 5: Speech translation: multilingual vs. bilingual fine-tuning. Multilingually fine-tuned models are trained on labeled data from either 21 X $\rightarrow$ En directions or 15 En $\rightarrow$ X directions. Bilingually fine-tuned models are trained on a single pair. We show average BLEU for XLS-R (1B) over four En $\rightarrow$ X directions en-(ca, ar, de, tr), and the average of high- and mid-resource directions for X $\rightarrow$ En. Note: results are not directly comparable to the rest of the paper and are hence denoted by (∗).  | X $\rightarrow$ En∗ | En $\rightarrow$ X∗
---|---|---
Bilingual fine-tuning | 20.9 | 23.4
Multilingual fine-tuning | 24.2 | 23.6

Next, we analyze the impact of initializing the decoder network with mBART
which was pretrained on additional labeled text-to-text machine translation
data (Liu et al., 2020; Li et al., 2021b; Tang et al., 2021). Specifically, we
use MBART-ML50N1 (49 languages to English) for X $\rightarrow$ English
directions and MBART-ML501N (English to 49 languages) for English
$\rightarrow$ X
directions.121212https://github.com/pytorch/fairseq/tree/main/examples/multilingual#mbart50-models
We observe that mBART initialization has little impact on English
$\rightarrow$ X but it leads to large improvements for X $\rightarrow$
English, especially on mid- and low-resource language directions.

Table 6: Speech translation: multilingual fine-tuning performance ablation in terms BLEU. mBART decoder initialization primarily improves mid- and low resource directions which benefit from the labeled translation data mBART was trained on.  | En $\rightarrow$ X | X $\rightarrow$ En
---|---|---
|  | High | Mid | Low | Avg
XLS-R (1B) | 26.0 | 34.3 | 25.5 | 11.7 | 19.3
\- mBART init | 25.1 | 32.4 | 17.7 | 3.6 | 12.4

Initializing the decoder network with mBART resulted in some low-resource
languages moving from 1-3 BLEU to 10+ BLEU. The labeled translation data used
to train mBART helps speech translation to adapt faster to the low supervision
in mid/low-resource language pairs of the CoVoST-2 benchmarks where many
language directions have only a few hours of labeled data. This shows that
pretraining both the encoder and decoder, multilingual fine-tuning, as well as
the use of extra machine translation data through mBART, enables few-shot
learning for some speech translation directions which have only a few hours of
labeled data.

###  5.2 Speech Recognition

Our experiments cover four common speech recognition benchmarks, 26 different
languages, three different domains and both low-data and high-data regimes.
The BABEL dataset evaluates models on noisy speech (§5.2.1), CommonVoice
presents a few-shot setup with just one hour of labeled data per language
(§5.2.2), MLS contains read speech in multiple European languages (§5.2.3),
and VoxPopuli contains parliamentary speech with varying amounts of labeled
data (§5.2.4).

####  5.2.1 BABEL

BABEL consists of the hardest speech recognition setting among our four
benchmarks which results in higher word error rates. Languages are low-
resource, the speech is very noisy and corresponds to natural telephone
conversation. Many competitions have tackled this dataset (Alumäe et al.,
2017; Ragni et al., 2018; Inaguma et al., 2019) and baselines are thus well
tuned. We compare to the best numbers we have found in the literature, as well
as our own best baselines.

Table 7 shows that XLS-R (0.3B) outperforms the equally sized XLSR-53, which
was the previous state of the art on all languages by an average of 1.4 WER,
e.g., on Assamese (as), WER decreases from 44.1 to 42.9, on Swahili (sw) WER
decreases from 26.5 to 24.3 and on Georgian (ka) WER drops from 31.1 to 28.0
WER. XLSR-53 and XLS-R were both pretrained on the same BABEL data, and the
better performance of XLS-R (0.3B) shows that pretraining on additional out-
of-domain datasets such as VoxPopuli does help performance on BABEL. This is
similar to findings for monolingual pretraining (Hsu et al., 2021a).

Using additional capacity, XLS-R (1B) outperforms XLS-R (0.3B) by 2.5 WER on
average. On Georgian (ka), this corresponds to improvements of 6 WER and 7.1
WER compared to Conneau et al. (2021) and Alumäe et al. (2017), respectively.
Compared to results from only three years ago from Ragni et al. (2018) and
Inaguma et al. (2019), XLS-R (1B) reduces word error rate by more than 10 WER,
from 40.6 to 30.6 on Tagalog and from 35.5 to 21.2 on Lao. XLS-R (2B) improves
over XLS-R (1B) by 0.8 BLEU on average showing that additional capacity can
further improve performance.

Table 7: Speech recognition results on BABEL in terms of word error rate (WER) on Assamese (as), Tagalog (tl), Swahili (sw), Lao (lo) and Georgian (ka).  | as | tl | sw | lo | ka
---|---|---|---|---|---
Labeled data | 55h | 76h | 30h | 59h | 46h
Previous work |  |  |  |  |
Alumäe et al. (2017) | - | - | - | - | 32.2
Ragni et al. (2018) | - | 40.6 | 35.5 | - | -
Inaguma et al. (2019) | 49.1 | 46.3 | 38.3 | 45.7 | -
XLSR-10 (Conneau et al., 2021) | 44.9 | 37.3 | 35.5 | 32.2 | -
XLSR-53 (Conneau et al., 2021) | 44.1 | 33.2 | 26.5 | - | 31.1
This work |  |  |  |  |
XLS-R (0.3B) | 42.9 | 33.2 | 24.3 | 31.7 | 28.0
XLS-R (1B) | 40.4 | 30.6 | 21.2 | 30.1 | 25.1
XLS-R (2B) | 39.0 | 29.3 | 21.0 | 29.7 | 24.3

####  5.2.2 CommonVoice

CommonVoice is an easier task than BABEL because it is read-speech but we use
a reduced labeled data setup which introduces a different challenge.
Specifically, we use the train/dev/test splits of Rivière et al. (2020) which
corresponds to a few-shot scenario where only one hour of training data is
available per language.

On English speech recognition, pretraining has been shown to be particularly
beneficial for low labeled data settings (Baevski et al., 2020b). This is
similar to cross-lingual pretraining (Conneau et al., 2021) where pretraining
on the large MLS corpus significantly improved performance over pretraining
only on CommonVoice data, e.g., on Dutch accuracy improved from 14 PER to 5.8
PER.

Table 8 shows that the additional training data of XLS-R compared to XLSR-53
results in better performance of 1.1 PER on average for XLS-R (0.3B). XLS-R
uses the same training data as XLSR-53 plus the very large VP-400K corpus of
parliamentary speech as well as the much smaller VoxLingua-107 which consists
of YouTube data, both of which are out of domain with respect to the read
audiobook domain of CommonVoice. This confirms that pretraining on more out of
domain data can still improve performance (Hsu et al., 2021a).

Furthermore, accuracy improves even on languages for which XLS-R does not add
any pretraining data compared to XLSR-53, e.g., Kyrgyz (ky) improves from 6.1
PER to 5.1 PER for XLS-R (0.3B) and 4.1 PER for XLS-R (1B) and both models are
pretrained on only about 11 hours of Kyrgyz data - 0.003% of the total
pretraining data. This shows that there is cross-lingual transfer that
benefits low-resource languages and that additional capacity is important to
realize this effect.

Chinese improves the least and gains are particularly large for languages for
which the training corpus of XLS-R contains more data due to VoxPopuli, e.g.,
for Swedish VP-400K adds more than 16K hours of unannontated speech and
performance improves from 12.2 PER to 5.5 PER when comparing XLSR-53 to XLS-R
(1B). Finally, XLS-R (2B) performs slightly better than XLS-R (1B) on average
with some languages improving while as others are performing slightly worse.
The modest average improvement is likely because error rates are already low
on this benchmark.

Table 8: Phoneme recognition performance on CommonVoice in terms of phoneme
error rate (PER) when using one hour of labeled data to fine-tune each
language. We compare to m-CPC (Rivière et al., 2020), Fer et al. (2017),
XLSR-10 (Conneau et al., 2021) and XLSR-53 (Conneau et al., 2021).

| es | fr | it | ky | nl | ru | sv | tr | tt | zh-HK | Avg
---|---|---|---|---|---|---|---|---|---|---|---
Labeled data | 1h | 1h | 1h | 1h | 1h | 1h | 1h | 1h | 1h | 1h |
Previous work |  |  |
m-CPC | 38.0 | 47.1 | 40.5 | 41.2 | 42.5 | 43.7 | 47.5 | 47.3 | 42.0 | 55.0 | 44.5
Fer et al. (2017) | 36.6 | 48.3 | 39.0 | 38.7 | 47.9 | 45.2 | 52.6 | 43.4 | 42.5 | 54.3 | 44.9
XLSR-10 | 7.9 | 12.6 | 11.7 | 7.0 | 14.0 | 9.3 | 20.6 | 9.7 | 7.2 | 22.8 | 12.3
XLSR-53 | 2.9 | 5.0 | 5.7 | 6.1 | 5.8 | 8.1 | 12.2 | 7.1 | 5.1 | 18.3 | 7.6
This work |  |  |  |  |  |  |  |  |  |
XLS-R (0.3B) | 3.1 | 5.4 | 4.9 | 5.1 | 5.8 | 6.0 | 7.2 | 6.0 | 4.1 | 17.0 | 6.5
XLS-R (1B) | 2.0 | 3.9 | 3.5 | 4.1 | 4.2 | 4.1 | 5.5 | 4.4 | 3.4 | 15.7 | 5.1
XLS-R (2B) | 2.2 | 4.0 | 3.5 | 4.0 | 4.7 | 3.7 | 5.0 | 4.0 | 2.9 | 14.8 | 4.9

####  5.2.3 Multilingual LibriSpeech

Multilingual LibriSpeech is a common benchmark for evaluating multilingual
speech recognition on eight European languages. We consider a setup where we
use ten hours of labeled data for each language (Conneau et al., 2021) and
compare to the prior work of Pratap et al. (2020) which uses all available
labeled data as well as XLSR-53 (Conneau et al., 2021) which uses the same ten
hour reduced labeled data setup.

Table 9 shows that XLS-R can outperform XLSR-53 on average by 1 WER at equal
capacity and that additional model capacity results in an improvement of 2.9
WER on average for XLS-R (1B). This result rivals the performance of the
supervised models of Pratap et al. (2020) which is based on significantly more
labeled data compared to the ten hour setup of XLS-R. Finally, on average
XLS-R (2B) does not show improvements over XLS-R (1B), which is similar to
CommonVoice.

Table 9: Speech recognition performance on Multilingual LibriSpeech (MLS) in terms of WER. Models are fine-tuned with 10h or all available labeled data (full) for each language. We compare to Pratap et al. (2020) who uses all labeled data (nearly 45K hours for English). Results are based on a 4-gram language model, except for Polish (pl) where we report Viterbi results for all settings since performance with the provided language model data resulted in inferior accuracy, as previously reported (Pratap et al., 2020); we denote this with (∗).  | #ft | en | de | nl | fr | es | it | pt | pl∗ | Avg.
---|---|---|---|---|---|---|---|---|---|---
Full labeled data (h) | 44.7K | 2K | 1.6K | 1.1K | 918 | 247 | 161 | 104 |
Previous work |  |  |  |  |  |  |  |  |  |
Pratap et al. (2020) | full | 5.9 | 6.5 | 12.0 | 5.6 | 6.1 | 10.5 | 19.5 | 19.4 | 10.7
XLSR-53 | 10h | 14.6 | 8.4 | 12.8 | 12.5 | 8.9 | 13.4 | 18.2 | 17.8 | 13.8
This work |  |  |  |  |  |  |  |  |  |
XLS-R (0.3B) | 10h | 15.9 | 9.0 | 13.5 | 12.4 | 8.1 | 13.1 | 17.0 | 13.9 | 12.8
XLS-R (1B) | 10h | 12.9 | 7.4 | 11.6 | 10.2 | 7.1 | 12.0 | 15.8 | 10.5 | 10.9
XLS-R (2B) | 10h | 14.0 | 7.6 | 11.8 | 10.0 | 6.9 | 12.1 | 15.6 | 9.8 | 11.0

####  5.2.4 VoxPopuli

The VoxPopuli corpus provides about 1.8K hours of labeled speech data in 14
languages, ranging from 543 hours for English to 35 hours for Slovakian, as
well as about 400K hours of unlabeled speech. This dataset is representative
of a setting where a lot of unannotated data in the same domain as the labeled
data is available. We compare to the work of Wang et al. (2021a) which used a
cross-lingually pretrained wav2vec 2.0 Base model on an earlier version of
VoxPopuli that contained about 10K hours of unlabeled speech.

Table 10 shows that cross-lingual pretraining (VP-10K) reduces WER from an
average of 37.5 for supervised-only training (No pretraining) to 15.3 WER.
XLS-R uses a lot more unlabeled VoxPopuli data and this results in improved
performance: XLS-R (0.3B) improves over VP-10K by an average of 2.5 WER. The
largest gains are on English, where WER improves from 16.2 to 10.2, likely due
to the use of more English data from MLS during pretraining (over 44K hours).
Increasing model capacity to 1B parameters results in even better performance,
reducing WER from an average of 15.3 for VP-10K to 10.6.

Table 10: VoxPopuli ASR results in terms of WER. We report the supervised-only baseline (No pretraining) of Wang et al. (2021b) as well as their pretrained model (VP-10K). | en | de | it | fr | es | pl | ro | hu
---|---|---|---|---|---|---|---|---
Labeled data | 543h | 282h | 91h | 211h | 166h | 111h | 89h | 63h
Baselines from previous work (Wang et al., 2021b) |
No pretraining | 30.0 | 29.3 | 45.2 | 30.5 | 31.4 | 25.6 | 27.7 | 27.9
VP-10K | 16.2 | 16.2 | 21.5 | 15.4 | 11.0 | 12.5 | 9.4 | 12.0
This work |
XLS-R (0.3B) | 10.2 | 13.0 | 19.2 | 12.6 | 9.8 | 9.6 | 7.9 | 11.6
XLS-R (1B) | 8.8 | 11.5 | 15.1 | 10.8 | 8.2 | 7.7 | 7.3 | 9.6
| nl | cs | sl | fi | hr | sk | Avg |
Labeled data | 53h | 62h | 10h | 27h | 43h | 35h |  |
Baselines from previous work (Wang et al., 2021b) |
No pretraining | 38.3 | 27.7 | 96.5 | 41.6 | 40.2 | 32.7 | 37.5 |
VP-10K | 19.7 | 11.8 | 26.1 | 17.1 | 14.1 | 11.1 | 15.3 |
This work |
XLS-R (0.3B) | 14.8 | 10.5 | 24.5 | 14.2 | 12.3 | 8.9 | 12.8 |
XLS-R (1B) | 12.5 | 8.7 | 19.5 | 11.3 | 10.0 | 7.1 | 10.6 |

####  5.2.5 LibriSpeech

On LibriSpeech English ASR, we compare XLS-R (0.3B) and XLS-R (1B) to the
wav2vec 2.0 English baseline. We see in Table 11 that with the same capacity
and same fine-tuning procedure, the English wav2vec 2.0 significantly
outperforms the XLS-R (0.3B) in all data regimes, showing the capacity
dilution and interference problem of our multilingual model. However, when
increasing the capacity, the model is able to catch up with the monolingual
results. In particular, XLS-R (1B) outperforms wav2vec 2.0 LV-60k on the 10
minute setting, but is at a disadvantage in the 10 hour setting, where the
English-focused monolingual model outperforms it by 0.7 WER on average. This
suggests that higher-capacity models can circumvent the interference problem
and can get strong results on high-resource languages, while still leveraging
their cross-lingual transfer ability for lower-resource languages Conneau et
al. (2021).

Table 11: LibriSpeech ASR results in terms of WER. Models are fine-tuned with 10min, 1h or 10h of annotated data. We compare XLS-R to wav2vec 2.0 (Baevski et al., 2020b) with the same number of parameters (317M). We do not use any language model for these experiments. Cross-lingual training with higher capacity such as for XLS-R (1B) obtains competitive performance.  Model | dev | test
---|---|---
clean | other | clean | other
10 min labeled |  |  |  |
wav2vec 2.0 LV-60K (0.3B) | 31.7 | 35.0 | 32.1 | 34.5
XLS-R (0.3B) | 33.3 | 39.8 | 34.1 | 39.6
XLS-R (1B) | 28.4 | 32.5 | 29.1 | 32.5
1h labeled |  |  |  |
wav2vec 2.0 LV-60K (0.3B) | 13.7 | 16.9 | 13.7 | 17.1
XLS-R (0.3B) | 17.1 | 23.7 | 16.8 | 24.0
XLS-R (1B) | 13.2 | 17.0 | 13.1 | 17.2
10h labeled |  |  |  |
wav2vec 2.0 LV-60K (0.3B) | 5.7 | 9.2 | 5.6 | 9.4
XLS-R (0.3B) | 8.3 | 15.1 | 8.3 | 15.4
XLS-R (1B) | 5.9 | 10.5 | 5.9 | 10.6

###  5.3 Speech Classification

Finally, we evaluate our approach on two speech utterance classification
tasks, language identification and speaker identification. For these tasks we
use our smallest model as these tasks require less capacity given the lower
complexity of the tasks compared to the structured prediction problems of
speech recognition and speech translation.

####  5.3.1 Language Identification

For language identification we adopt the setup of VoxLingua107 (Valk & Alumäe,
2020) which provides data for 107 different languages. We train our model on
the official train set, and report results on the development set, comprising
33 languages.

Table 12 shows that our best model outperforms previous work, improving the
best known prior work of Ravanelli et al. (2020) by 1% absolute, a relative
error reduction of 15%. For comparison, we also fine-tune the English-only
wav2vec 2.0 pretrained on Libri-Light which performs surprisingly well on this
multilingual task but is outperformed by the XLS-R model by 1.5% error rate on
average.

Table 12: Language identification on VoxLingua107. We report the error rate on the development set spanning 33 languages.  | Error Rate (%)
---|---
| 0…5 sec | 5…20 sec | Average
Previous work |  |  |
Valk & Alumäe (2020) | 12.3 | 6.1 | 7.1
Ravanelli et al. (2021) | - | - | 6.7
This work |  |  |
wav2vec 2.0 LV-60K (300M) | 11.5 | 6.3 | 7.2
XLS-R (0.3B) | 9.1 | 5.0 | 5.7

####  5.3.2 Speaker Identification

Finally, we consider speaker identification on VoxCeleb1 where we fine-tune
our model to distinguish between a fixed set of speakers given an utterance.
We compare to prior work including results published as part of the recently
introduced SUPERB benchmark (Yang et al., 2021) but note that their results
are not strictly comparable because they do not fine-tune the underlying pre-
trained model. All parameters of XLS-R are fine-tuned, similar to the
evaluation of all other tasks. The results (Table 13) show that our cross-
lingual model also performs very well for speaker identification, even though
utterances are mostly in English.

Table 13: Speaker identification accuracy on VoxCeleb1 in terms of accuracy. We report baselines from previous work as well as XLS-R (B).  | Accuracy (%)
---|---
Previous work |
CNN (Nagrani et al. (2017)) | 80.5
SUPERB-Hubert Large (Yang et al., 2021) | 90.3
This work |
XLS-R (0.3B) | 95.8

###  5.4 Discussion

##### Single model.

Cross-lingual training results in a single model for multiple languages
compared to a separate model for each language. Training a cross-lingual model
requires more effort than a single monolingual model but the resulting model
can be used for many different languages. Advances in architectures and
training can also be deployed more easily since we only need to retrain a
single model rather than many different ones.

In terms of accuracy, prior work in self-supervised learning for speech
established that cross-lingually pretrained models are very competitive to
monolingually pretrained models for speech recognition (Conneau et al., 2021).
Our experiments show a similar trend for speech-translation: XLS-R can perform
very competitively to English-only pretrained models for English $\rightarrow$
X speech translation where the encoder only needs to encode English speech - a
setting which favors monolingually pretrained models.

##### Performance trends.

Overall, XLS-R performs best for low-resource and mid-resource languages. For
speech translation, we observe strong improvements for low- and mid-resource X
$\rightarrow$ English directions and comparatively smaller gains on high-
resource directions. Many low-resource directions which previously had
performance in the 1-5 BLEU range improve to over 10-20 BLEU due to the better
cross-lingual speech representations. For English $\rightarrow$ X directions,
large enough cross-lingual models can even surpass the performance of English-
only pretrained models.

Similarly, for speech recognition, we see strong improvements on BABEL,
CommonVoice and VoxPopuli, benchmarks which include low- and mid-resource
tasks.131313MLS is a notable exception and we attribute the different
performance pattern to prior work having pretrained on large amounts of in-
domain data. We find that models trained on more data from more languages can
perform as well or better than comparable models of the same size and we we
observe this trend across all speech recognition benchmarks. Keeping
everything else equal, larger capacity models often further improve
performance.

##  6 Conclusion

XLS-R is a new self-supervised cross-lingual speech representation model which
scales the number of languages, the amount of training data as well as model
size. The training corpus is an order of magnitude larger than prior work and
covers 128 languages in 436K hours of recorded speech audio. The resulting
model enables state of the art results for X $\rightarrow$ English speech
translation on CoVoST-2, outperforming prior art by a sizeable margin with the
largest improvements on mid- and low-resource directions. It also performs
competitively to the best English $\rightarrow$ X work, without the use of
equally applicable techniques such as self-training and language model
decoding.

On speech recognition, XLS-R sets a new state of the art on CommonVoice,
VoxPopuli, several languages of BABEL, while performing competitively on MLS
with much less labeled data. These datasets cover a wide range of languages,
data regimes and domains, demonstrating the generalization ability of XLS-R.
Our model also sets a new state of the art on the VoxLingua107 language
identification benchmark. The largest XLS-R model comprises 2B parameters
which enables it to outperform a strong English-only pretrained model on
English $\rightarrow$ X speech translation, a setting which favors
monolingually pretrained models. This shows that cross-lingually trained
models with sufficient capacity can perform as well as specialized
monolingually pretrained models. We hope XLS-R will help catalyze research in
speech technology for many more languages of the world. Models and code are
publicly available on several platforms.

#### Acknowledgments

We thank Morgane Riviere, Ann Lee, Anne Wu, Chaitanya Talnikar, Daniel Haziza,
Mary Williamson, Emmanuel Dupoux for early access to VP-400K, and Christophe
Ropers for his advise on language categorization. We also thank Min Xu, Jacob
Kahn, Shruti Bhosale, Anjali Sridhar, and Tatiana Likhomanenko for help with
infrastructure.

## References

  * Alumäe et al. (2017) T. Alumäe, D. Karakos, W. Hartmann, R. Hsiao, L. Zhang, L. Nguyen, S. Tsakalidis, and R. Schwartz.  The 2016 bbn georgian telephone speech keyword spotting system.  In _ICASSP_ , 2017.
  * Ardila et al. (2020) Rosana Ardila, Megan Branson, Kelly Davis, Michael Henretty, Michael Kohler, Josh Meyer, Reuben Morais, Lindsay Saunders, Francis M. Tyers, and Gregor Weber.  Common voice: A massively-multilingual speech corpus.  _Proc. of LREC_ , 2020.
  * Arivazhagan et al. (2019) Naveen Arivazhagan, Ankur Bapna, Orhan Firat, Dmitry Lepikhin, Melvin Johnson, Maxim Krikun, Mia Xu Chen, Yuan Cao, George Foster, Colin Cherry, et al.  Massively multilingual neural machine translation in the wild: Findings and challenges.  _arXiv_ , abs/1907.05019, 2019.
  * Baevski et al. (2019) Alexei Baevski, Sergey Edunov, Yinhan Liu, Luke Zettlemoyer, and Michael Auli.  Cloze-driven pretraining of self-attention networks.  In _Proc. of ACL_ , 2019.
  * Baevski et al. (2020a) Alexei Baevski, Steffen Schneider, and Michael Auli.  vq-wav2vec: Self-supervised learning of discrete speech representations.  In _Proc. of ICLR_ , 2020a.
  * Baevski et al. (2020b) Alexei Baevski, Yuhao Zhou, Abdelrahman Mohamed, and Michael Auli.  wav2vec 2.0: A framework for self-supervised learning of speech representations.  In _Proc. of NeurIPS_ , 2020b.
  * Baines et al. (2021) Mandeep Baines, Shruti Bhosale, Vittorio Caggiano, Naman Goyal, Siddharth Goyal, Myle Ott, Benjamin Lefaudeux, Vitaliy Liptchinsky, Mike Rabbat, Sam Sheiffer, Anjali Sridhar, and Min Xu.  Fairscale: A general purpose modular pytorch library for high performance and large scale training.  https://github.com/facebookresearch/fairscale, 2021.
  * Brown et al. (2020) Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei.  Language models are few-shot learners.  In _Proc. of NeurIPS_ , 2020.
  * Caron et al. (2021) Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin.  Emerging properties in self-supervised vision transformers.  _arXiv preprint arXiv:2104.14294_ , 2021.
  * Chen et al. (2016) Tianqi Chen, Bing Xu, Chiyuan Zhang, and Carlos Guestrin.  Training deep nets with sublinear memory cost.  _arXiv_ , abs/1604.06174, 2016.
  * Chen et al. (2020) Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton.  A simple framework for contrastive learning of visual representations.  In _International conference on machine learning_ , pp. 1597–1607. PMLR, 2020.
  * Chung et al. (2021) Yu-An Chung, Yu Zhang, Wei Han, Chung-Cheng Chiu, James Qin, Ruoming Pang, and Yonghui Wu.  W2v-bert: Combining contrastive learning and masked language modeling for self-supervised speech pre-training, 2021.
  * Conneau et al. (2018) Alexis Conneau, Guillaume Lample, Ruty Rinott, Adina Williams, Samuel R Bowman, Holger Schwenk, and Veselin Stoyanov.  Xnli: Evaluating cross-lingual sentence representations.  _arXiv preprint arXiv:1809.05053_ , 2018.
  * Conneau et al. (2020) Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov.  Unsupervised cross-lingual representation learning at scale.  In _Proc. of ACL_ , 2020.
  * Conneau et al. (2021) Alexis Conneau, Alexei Baevski, Ronan Collobert, Abdelrahman Mohamed, and Michael Auli.  Unsupervised cross-lingual representation learning for speech recognition.  In _Proc. of Interspeech_ , 2021.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.  Bert: Pre-training of deep bidirectional transformers for language understanding.  _Proc. of NAACL_ , 2019.
  * Dieleman et al. (2018) Sander Dieleman, Aäron van den Oord, and Karen Simonyan.  The challenge of realistic music generation: modelling raw audio at scale.  _Proc of NIPS_ , 2018.
  * Fan et al. (2019) Angela Fan, Edouard Grave, and Armand Joulin.  Reducing transformer depth on demand with structured dropout.  _arXiv_ , abs/1909.11556, 2019.
  * Fedus et al. (2021) William Fedus, Barret Zoph, and Noam Shazeer.  Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity.  _arXiv_ , abs/2101.03961, 2021.
  * Fer et al. (2017) Radek Fer, Pavel Matějka, František Grézl, Oldřich Plchot, Karel Veselỳ, and Jan Honza Černockỳ.  Multilingually trained bottleneck features in spoken language recognition.  _Computer Speech & Language_, 46, 2017.
  * Gales et al. (2014) Mark J. F. Gales, Kate M. Knill, Anton Ragni, and Shakti P. Rath.  Speech recognition and keyword spotting for low-resource languages: Babel project research at cued.  In _n Spoken Language Technologies for Under-Resourced Languages_ , 2014.
  * Goyal et al. (2021) Naman Goyal, Jingfei Du, Myle Ott, Giri Anantharaman, and Alexis Conneau.  Larger-scale transformers for multilingual masked language modeling.  _arXiv_ , abs/2105.00572, 2021.
  * Graves et al. (2006) Alex Graves, Santiago Fernández, and Faustino Gomez.  Connectionist temporal classification: Labelling unsegmented sequence data with recurrent neural networks.  In _Proc. of ICML_ , 2006.
  * He et al. (2020) Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick.  Momentum contrast for unsupervised visual representation learning.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pp. 9729–9738, 2020.
  * Hsu et al. (2021a) Wei-Ning Hsu, Anuroop Sriram, Alexei Baevski, Tatiana Likhomanenko, Qiantong Xu, Vineel Pratap, Jacob Kahn, Ann Lee, Ronan Collobert, Gabriel Synnaeve, et al.  Robust wav2vec 2.0: Analyzing domain shift in self-supervised pre-training.  _arXiv preprint arXiv:2104.01027_ , 2021a.
  * Hsu et al. (2021b) Wei-Ning Hsu, Yao-Hung Hubert Tsai, Benjamin Bolte, Ruslan Salakhutdinov, and Abdelrahman Mohamed.  Hubert: How much can a bad teacher benefit ASR pre-training?  In _Proc. of ICASSP_ , 2021b.
  * Hu et al. (2020) Junjie Hu, Sebastian Ruder, Aditya Siddhant, Graham Neubig, Orhan Firat, and Melvin Johnson.  Xtreme: A massively multilingual multi-task benchmark for evaluating cross-lingual generalisation.  In _International Conference on Machine Learning_ , pp. 4411–4421. PMLR, 2020.
  * Huang et al. (2016) Gao Huang, Yu Sun, Zhuang Liu, Daniel Sedra, and Kilian Q. Weinberger.  Deep networks with stochastic depth.  _arXiv_ , abs/1603.09382, 2016.
  * Inaguma et al. (2019) Hirofumi Inaguma, Jaejin Cho, Murali Karthick Baskar, Tatsuya Kawahara, and Shinji Watanabe.  Transfer learning of language-independent end-to-end asr with language model fusion.  In _ICASSP_ , 2019.
  * Jang et al. (2016) Eric Jang, Shixiang Gu, and Ben Poole.  Categorical reparameterization with gumbel-softmax.  _Proc. of ICLR_ , 2016.
  * Jegou et al. (2011) Herve Jegou, Matthijs Douze, and Cordelia Schmid.  Product quantization for nearest neighbor search.  _IEEE Trans. Pattern Anal. Mach. Intell._ , 33(1):117–128, January 2011.
  * Kahn et al. (2020) Jacob Kahn et al.  Libri-light: A benchmark for asr with limited or no supervision.  In _Proc. of ICASSP_ , 2020.
  * Kawakami et al. (2020) Kazuya Kawakami, Luyu Wang, Chris Dyer, Phil Blunsom, and Aaron van den Oord.  Learning robust and multilingual speech representations.  _Proc. of EMNLP_ , 2020.
  * Kingma & Ba (2015) Diederik P. Kingma and Jimmy Ba.  Adam: A Method for Stochastic Optimization.  In _Proc. of ICLR_ , 2015.
  * Lample & Conneau (2019) Guillaume Lample and Alexis Conneau.  Cross-lingual language model pretraining.  In _Proc. of NeurIPS_ , 2019.
  * Li et al. (2021a) Bo Li, Ruoming Pang, Tara N. Sainath, Anmol Gulati, Yu Zhang, James Qin, Parisa Haghani, W. Ronny Huang, Min Ma, and Junwen Bai.  Scaling end-to-end models for large-scale multilingual asr.  _arXiv_ , 2021a.
  * Li et al. (2021b) Xian Li, Changhan Wang, Yun Tang, Chau Tran, Yuqing Tang, Juan Pino, Alexei Baevski, Alexis Conneau, and Michael Auli.  Multilingual speech translation with efficient finetuning of pretrained models.  In _Proc. of ACL_ , 2021b.
  * Liu et al. (2020) Yinhan Liu, Jiatao Gu, Naman Goyal, Xian Li, Sergey Edunov, Marjan Ghazvininejad, Mike Lewis, and Luke Zettlemoyer.  Multilingual denoising pre-training for neural machine translation.  _arXiv_ , abs/2001.08210, 2020.
  * Nagrani et al. (2017) A. Nagrani, J. S. Chung, and A. Zisserman.  Voxceleb: a large-scale speaker identification dataset.  In _Proc. of Interspeech_ , 2017.
  * Ott et al. (2019) Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli.  fairseq: A fast, extensible toolkit for sequence modeling.  In _Proc. of NAACL System Demonstrations_ , 2019.
  * Panayotov et al. (2015) Vassil Panayotov, Guoguo Chen, Daniel Povey, and Sanjeev Khudanpur.  Librispeech: an asr corpus based on public domain audio books.  In _Proc. of ICASSP_ , pp. 5206–5210. IEEE, 2015.
  * Papineni et al. (2002) K. Papineni, S. Roukos, T. Ward, and W.J. Zhu.  Bleu: a method for automatic evaluation of machine translation.  In _Proc. of ACL_ , 2002.
  * Pratap et al. (2020) Vineel Pratap, Qiantong Xu, Anuroop Sriram, Gabriel Synnaeve, and Ronan Collobert.  Mls: A large-scale multilingual dataset for speech research.  In _Proc. of Interspeech_ , 2020.
  * Radford et al. (2018) Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever.  Improving language understanding by generative pre-training.  _arXiv_ , 2018.
  * Raffel et al. (2019) Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu.  Exploring the limits of transfer learning with a unified text-to-text transformer.  _arXiv preprint arXiv:1910.10683_ , 2019.
  * Ragni et al. (2018) Anton Ragni, Qiujia Li, Mark J. F. Gales, and Yu Wang.  Confidence estimation and deletion prediction using bidirectional recurrent neural networks.  In _SLT_ , Athens, 2018.
  * Rajbhandari et al. (2021) Samyam Rajbhandari, Olatunji Ruwase, Jeff Rasley, Shaden Smith, and Yuxiong He.  Zero-infinity: Breaking the GPU memory wall for extreme scale deep learning.  _arXiv_ , abs/2104.07857, 2021.
  * Ravanelli et al. (2020) Mirco Ravanelli, Jianyuan Zhong, Santiago Pascual, Pawel Swietojanski, Joao Monteiro, Jan Trmal, and Yoshua Bengio.  Multi-task self-supervised learning for robust speech recognition.  _arXiv_ , 2020.
  * Ravanelli et al. (2021) Mirco Ravanelli, Titouan Parcollet, Peter Plantinga, Aku Rouhe, Samuele Cornell, Loren Lugosch, Cem Subakan, Nauman Dawalatabad, Abdelwahab Heba, Jianyuan Zhong, Ju-Chieh Chou, Sung-Lin Yeh, Szu-Wei Fu, Chien-Feng Liao, Elena Rastorgueva, François Grondin, William Aris, Hwidong Na, Yan Gao, Renato De Mori, and Yoshua Bengio.  SpeechBrain: A general-purpose speech toolkit.  _arXiv_ , 2021.
  * Rivière et al. (2020) Morgane Rivière, Armand Joulin, Pierre-Emmanuel Mazaré, and Emmanuel Dupoux.  Unsupervised pretraining transfers well across languages.  In _Proc. of ICASSP_ , 2020.
  * Ruder et al. (2021) Sebastian Ruder, Noah Constant, Jan Botha, Aditya Siddhant, Orhan Firat, Jinlan Fu, Pengfei Liu, Junjie Hu, Graham Neubig, and Melvin Johnson.  Xtreme-r: Towards more challenging and nuanced multilingual evaluation.  _arXiv preprint arXiv:2104.07412_ , 2021.
  * Schneider et al. (2019) Setffen Schneider, Alexei Baevski, Ronan Collobert, and Michael Auli.  wav2vec: Unsupervised pre-training for speech recognition.  In _Proc. of Interspeech_ , 2019.
  * Sennrich et al. (2016) Rico Sennrich, Barry Haddow, and Alexandra Birch.  Neural machine translation of rare words with subword units.  In _Proc. of ACL_ , 2016.
  * Tang et al. (2021) Yuqing Tang, C. Tran, Xian Li, Peng-Jen Chen, Naman Goyal, Vishrav Chaudhary, Jiatao Gu, and Angela Fan.  Multilingual translation from denoising pre-training.  In _Proc. of ACL_ , 2021.
  * Valk & Alumäe (2020) Jörgen Valk and Tanel Alumäe.  Voxlingua107: a dataset for spoken language recognition.  In _Proc. of SLT_ , 2020.
  * van den Oord et al. (2018) Aäron van den Oord, Yazhe Li, and Oriol Vinyals.  Representation learning with contrastive predictive coding.  _Proc. of NIPS_ , 2018.
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  In _Proc. of NIPS_ , 2017.
  * Wang et al. (2020) Changhan Wang, Anne Wu, and Juan Pino.  Covost 2 and massively multilingual speech-to-text translation.  _arXiv_ , 2020.
  * Wang et al. (2021a) Changhan Wang, Morgane Riviere, Ann Lee, Anne Wu, Chaitanya Talnikar, Daniel Haziza, Mary Williamson, Juan Pino, and Emmanuel Dupoux.  VoxPopuli: A large-scale multilingual speech corpus for representation learning, semi-supervised learning and interpretation.  In _Proc. of ACL_ , 2021a.
  * Wang et al. (2021b) Changhan Wang, Morgane Riviere, Ann Lee, Anne Wu, Chaitanya Talnikar, Daniel Haziza, Mary Williamson, Juan Pino, and Emmanuel Dupoux.  VoxPopuli: A large-scale multilingual speech corpus for representation learning, semi-supervised learning and interpretation.  In _Proc. of ACL_ , 2021b.
  * Wang et al. (2021c) Changhan Wang, Anne Wu, Juan Pino, Alexei Baevski, Michael Auli, and Alexis Conneau.  Large-scale self- and semi-supervised learning for speech translation.  In _Proc. of Interspeech_ , 2021c.
  * Wenzek et al. (2020) Guillaume Wenzek, Marie-Anne Lachaux, Alexis Conneau, Vishrav Chaudhary, Francisco Guzmán, Armand Joulin, and Édouard Grave.  Ccnet: Extracting high quality monolingual datasets from web crawl data.  In _Proceedings of the 12th Language Resources and Evaluation Conference_ , pp. 4003–4012, 2020.
  * Xu et al. (2020) Qiantong Xu, Alexei Baevski, Tatiana Likhomanenko, Paden Tomasello, Alexis Conneau, Ronan Collobert, Gabriel Synnaeve, and Michael Auli.  Self-training and pre-training are complementary for speech recognition.  In _Proc. of ICASSP_ , 2020.
  * Xu et al. (2021) Qiantong Xu, Alexei Baevski, Tatiana Likhomanenko, Paden Tomasello, Alexis Conneau, Ronan Collobert, Gabriel Synnaeve, and Michael Auli.  Self-training and pre-training are complementary for speech recognition.  In _ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , pp. 3030–3034. IEEE, 2021.
  * Xue et al. (2020) Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel.  mt5: A massively multilingual pre-trained text-to-text transformer.  _arXiv preprint arXiv:2010.11934_ , 2020.
  * Yang et al. (2021) Shu-Wen Yang, Po-Han Chi, Yung-Sung Chuang, Cheng-I Jeff Lai, Kushal Lakhotia, Yist Y. Lin, Andy T. Liu, Jiatong Shi, Xuankai Chang, Guan-Ting Lin, Tzu-Hsien Huang, Wei-Cheng Tseng, Ko-tik Lee, Da-Rong Liu, Zili Huang, Shuyan Dong, Shang-Wen Li, Shinji Watanabe, Abdelrahman Mohamed, and Hung-yi Lee.  SUPERB: speech processing universal performance benchmark.  _arXiv_ , abs/2105.01051, 2021.
  * Zhang et al. (2020) Yu Zhang, James Qin, Daniel S. Park, Wei Han, Chung-Cheng Chiu, Ruoming Pang, Quoc V. Le, and Yonghui Wu.  Pushing the limits of semi-supervised learning for automatic speech recognition.  In _Proc. of NeurIPS SAS Workshop_ , 2020.
  * Zhang et al. (2021) Yu Zhang, Daniel S. Park, Wei Han, James Qin, Anmol Gulati, Joel Shor, Aren Jansen, Yuanzhong Xu, Yanping Huang, Shibo Wang, Zongwei Zhou, Bo Li, Min Ma, William Chan, Jiahui Yu, Yongqiang Wang, Liangliang Cao, Khe Chai Sim, Bhuvana Ramabhadran, Tara N. Sainath, Françoise Beaufays, Zhifeng Chen, Quoc V. Le, Chung-Cheng Chiu, Ruoming Pang, and Yonghui Wu.  Bigssl: Exploring the frontier of large-scale semi-supervised learning for automatic speech recognition.  _arXiv_ , 2021.

## Appendix

##  Appendix A Detailed Speech Translation Results

| High-resource | Mid-resource |  |
---|---|---|---|---
X $\rightarrow$ English | fr | de | es | ca | fa | it | ru | pt | zh | tr | ar
Train Hours | 264h | 184h | 113h | 136h | 49h | 44h | 18h | 10h | 10h | 4h | 2h
Prior work |  |  |  |
XLSR-53 | 32.3 | 26.9 | 33.3 | 28.6 | 7.1 | 26.2 | 4.9 | 10.5 | 6.9 | 3.7 | 1.2
VP-100K | 30.4 | 23.4 | 31.1 | 25.7 | 2.0 | 26.3 | 21.7 | 13.6 | 2.6 | 0.9 | 0.8
XMEF-En | 35.0 | 28.2 | 35.2 | 31.1 | 3.6 | 27.6 | 22.8 | 24.1 | 6.0 | 4.8 | 2.8
XMEF-X | 36.1 | 30.6 | 38.1 | 31.8 | 8.5 | 31.9 | 30.9 | 20.7 | 8.9 | 9.4 | 6.4
This work |  |  |  |
XLS-R (0.3B) | 32.9 | 26.7 | 34.1 | 28.7 | 5.9 | 29.0 | 26.4 | 28.3 | 4.9 | 4.6 | 3.0
XLS-R (1B) | 36.2 | 31.2 | 37.9 | 31.9 | 9.6 | 33.1 | 37.0 | 39.3 | 8.7 | 12.8 | 12.2
XLS-R (2B) | 37.6 | 33.6 | 39.2 | 33.8 | 12.9 | 34.9 | 39.5 | 41.8 | 9.4 | 16.7 | 17.1

| Low-resource |
---|---|---
X $\rightarrow$ English | et | mn | nl | sv | lv | sl | ta | ja | id | cy | Avg
Train Hours | 3h | 3h | 7h | 2h | 2h | 2h | 2h | 2h | 2h | 2h |
Prior work |  |  |  |
XLSR-53 | 0.7 | 0.6 | 20.5 | 2.8 | 1.9 | 0.5 | 0.1 | 0.4 | 0.6 | 5.6 | 10.3
VP-100K | 4.6 | 0.3 | 18.3 | 11.7 | 9.0 | 8.1 | 0.1 | 0.2 | 0.7 | 0.6 | 11.1
XMEF-En | 1.5 | 0.9 | 14.2 | 5.0 | 4.9 | 5.0 | 0.8 | 1.7 | 3.7 | 2.3 | 12.4
XMEF-X | 2.5 | 1.2 | 24.0 | 4.0 | 5.0 | 5.6 | 0.9 | 1.0 | 2.8 | 8.1 | 14.7
This work |  |  |  |
XLS-R (0.3B) | 3.5 | 0.4 | 22.0 | 10.3 | 6.0 | 6.6 | 0.2 | 0.6 | 1.4 | 2.5 | 13.2
XLS-R (1B) | 8.3 | 0.8 | 28.2 | 24.7 | 16.0 | 16.7 | 0.3 | 1.9 | 10.3 | 8.6 | 19.3
XLS-R (2B) | 11.1 | 1.6 | 31.7 | 29.6 | 19.5 | 19.6 | 0.5 | 3.5 | 16.5 | 14.0 | 22.1

Table 14: CoVoST-2 X $\rightarrow$ English full results. We report baseline
results from XLSR-53 (Conneau et al., 2021), VP-100K Wang et al. (2021a),
XMEF-En and XMEF-X Li et al. (2021b).

English $\rightarrow$ X | ca | ar | de | tr | zh | fa | et | mn
---|---|---|---|---|---|---|---|---
Train Hours | 430h | 430h | 430h | 430h | 430h | 430h | 430h | 430h
Prior work |
XLSR-53 | 29.0 | 16.5 | 23.6 | 15.3 | 33.7 | 19.4 | 19.8 | 13.3
VP-100K | 26.1 | 14.5 | 20.8 | 13.5 | 30.7 | 17.1 | 17.4 | 12.0
XMEF-JT | 30.9 | 18.0 | 25.8 | 17.0 | 33.3 | 21.5 | 22.1 | 14.8
Wav2vec 2.0 (0.3B) | 32.4 | 17.4 | 23.8 | 15.4 | - | - | - | -
\+ self-training + LM | 35.6 | 20.8 | 27.2 | 18.9 | - | - | - | -
This work - monolingual pretraining (LV-60K) |
Wav2vec 2.0 (0.72B) | 32.7 | 19.4 | 27.0 | 17.7 | 37.4 | 21.8 | 22.9 | 15.4
This work - cross-lingual pretraining |
XLS-R (0.3B) | 28.7 | 16.3 | 23.6 | 15.0 | 33.5 | 19.0 | 19.6 | 13.2
XLS-R (1B) | 32.1 | 19.2 | 26.2 | 17.1 | 36.7 | 21.3 | 22.4 | 14.9
XLS-R (2B) | 34.2 | 20.7 | 28.3 | 18.6 | 38.5 | 22.9 | 24.1 | 16.2

English $\rightarrow$ X | sv | lv | sl | ta | ja | id | cy | Avg
---|---|---|---|---|---|---|---|---
Train Hours | 430h | 430h | 430h | 430h | 430h | 430h | 430h |
Prior work |
XLSR-53 | 29.6 | 19.3 | 22.8 | 15.8 | 37.1 | 27.6 | 28.8 | 23.4
VP-100K | 26.3 | 16.7 | 20.2 | 13.9 | 33.9 | 24.9 | 26.0 | 20.9
XMEF-JT | 29.6 | 21.5 | 25.1 | 17.8 | 39.3 | 29.9 | 30.6 | 25.1
Wav2vec 2.0 (0.3B) | - | - | - | - | - | - | - | -
\+ self-training + LM | - | - | - | - | - | - | - | -
This work - monolingual pretraining (LV-60K) |
Wav2vec 2.0 (0.72B) | 33.0 | 22.5 | 26.1 | 18.4 | 40.3 | 31.2 | 32.8 | 26.6
This work - cross-lingual pretraining |
XLS-R (0.3B) | 29.1 | 19.3 | 22.4 | 15.6 | 36.9 | 27.4 | 28.9 | 23.2
XLS-R (1B) | 32.3 | 22.0 | 25.4 | 18.1 | 39.9 | 30.3 | 31.8 | 26.0
XLS-R (2B) | 34.5 | 23.5 | 27.6 | 19.8 | 41.5 | 32.5 | 33.8 | 27.8

Table 15: CoVoST-2 English $\rightarrow$ X full results. We report baselines
results from XLSR-53 (Conneau et al., 2021), VP-100K (Wang et al., 2021a),
XMEF-JT (Li et al., 2021b) and wav2vec 2.0 LV-60K with self-training (Wang et
al., 2021c)