§ INTRODUCTION

German literary critic Friedrich Schlegel once said, “What is lost in the good or excellent translation is precisely the best.” When applied to speech, this sentiment implies that even when a translation accurately renders the semantic meaning of an utterance, certain defining elements of speech may be lost in the process [Schuller et al., 2013].

While the specific constituents of what Schlegel deemed the best are open for interpretation, the speech translation research community has long homed in on two components: the indexical (i.e., components marking the characteristics of a person) and pragmatical (i.e., the way communication works in social situations) components of speech that make human communication what it is. For speech to be natural, it relies on the indexical or revelatory nature of the human voice [Costello, 2000]. A speech translation system that incorporates features that help a listener make inferences about a speaker's personhood bolsters the naturalness of a machine-mediated interaction [Waytz et al., 2014]. Preserving vocal style also involves capturing the prosodic elements of speech (e.g., pitch, stress, rhythm), which are key in facilitating the expression of meaning, emotions, and intent [Aguero et al., 2006, Anumanchipalli et al., 2012]. Next, human speech and translation are sensitive to pragmatic nuances such as turn-taking and timing controls [Cokely, 1986, Levinson, 2016]. Picture how human simultaneous interpreters work: they find just the right balance between low-latency and accurate translations. Waiting too long stifles the flow of communication, while going too fast compromises the overall quality of a translation.

Existing research efforts aimed at preserving these intrinsically human features in translation have led to the independent development of expressive and streaming speech-to-speech translation (S2ST) systems. On the expressive front, recent advances in text-to-speech synthesis have integrated voice style transfer via speech language model [Wang et al., 2023, Kharitonov et al., 2022], flow matching [Le et al., 2023] and diffusion model [Shen et al., 2023]. These approaches subsequently inspired S2ST models designed to preserve the source speech's vocal style and style qualities with a cascaded architecture.
Despite these advances, an open, comprehensive S2ST system capturing semantic translation, rhythm, pauses, and sentence-level preservation of the style of one's voice has yet to be realized.
Streaming wise, recent efforts have explored how different simultaneous translation policies (e.g., rule-based or learnable policies) could be deployed to produce systems that strike a balance between low latency and high-quality translations [Ma et al., 2019, Arivazhagan et al., 2019, Ma et al., 2020].
That said, existing research investments in streaming have homed in on speech-to-text translation (S2TT), and the few that are S2ST compatible are limited in language coverage.
Moreover, most streaming translation systems focus on bilingual communication, limiting their utility in contexts where a group of speakers converse in multiple different languages.

To advance research in multilingual expressive and streaming speech translation, we introduce , , and .
is the foundational multilingual and multimodal model on which the latter two models are initialized.
As an improved version of , delivers state-of-the-art semantic accuracy across different speech and text translation tasks while supporting nearly 100 languages as input speech or text. This new version features multitask-with its non-auto-regressive unit decoder and hierarchical upsampling, making predicting units much more data-efficient. The new speech encoder of was pre-trained on 4.5M hours of unlabeled audio data, and the multitask model was finetuned with more supervision from automatically aligned pairs
to boost 's performance on low-resource languages.
Built using commissioned and publicly available datasets, enables translation that preserves vocal style and prosody (e.g., rhythm and tone). The model supports translations from and into English in five languages.
To our knowledge,
is the first model to enable expressive S2ST from and into English and supports underexplored aspects of prosody such as speech rate and pauses.
Our model leverages the Efficient Monotonic Multihead Attention (EMMA) [Ma et al., 2023] mechanism to generate low-latency target translations without waiting for complete source utterances. As the first of its kind to provide many-to-many translations in a simultaneous manner, supports the same language coverage as the scale of in ASR, S2TT, and S2ST tasks.

To comprehensively evaluate our systems, we combined existing and newly developed metrics (<Ref>). For expressivity, we developed two new automatic metrics that measure prosody—and a rhythm evaluation toolkit. For human evaluation, we used Cross-lingual Semantic Textual Similarity (XSTS) [Licht et al., 2022] to measure semantics, Mean Opinion Score (MOS) to measure the speech quality of all of our models,
and a modified version of the Prosodic Consistency Protocol (PCP) [Huang et al., 2023] to measure the extent to which the expressive qualities in source and target audio are matched.
For latency, we used Ending Offset (see <Ref>) for speech output (i.e., the time between when a person finishes speaking and the last translated speech being generated) and an adapted version of Average Lagging [Ma et al., 2019, Ma et al., 2020] (i.e., a metric that quantifies the degree to which a listener is out of sync with a speaker with regards to the number of seconds in the source speech) and Length-Adaptive Average Lagging [Papi et al., 2022] for text output. Moreover, we used well-known metrics such as , , and to measure translation quality automatically. Lastly, we tested for robustness towards noise and vocal style variations.

To ensure that our models are built safely and ethically, we took a four-pronged approach to Responsible AI by implementing 1) the first known red-teaming effort for machine translation, 2) added-toxicity detection and mitigation, 3) a systematic evaluation of gender bias, and 4) an inaudible, localized watermarking mechanism named .
We also introduce the new concept of a metric card (<Ref>) that compiles details of our evaluation and Responsible AI metrics.

Combining these building blocks, our unified model  (comprising of and ) marks the first publicly available system that unlocks expressive cross-lingual communication in real-time (see <Ref>). Crucially, gives us a pivotal look at the technical foundation needed to transform the Universal Speech Translator from a science fiction concept to a real-world technology. To spur further research into related domains and make our work available to the various communities that could benefit from our effort, we publicly release the following at <https://github.com/facebookresearch/seamless_communication>:

* Models & code: , , , and models

* Automatically aligned data models, code, and metadata: data and speech encoders

* Evaluation tools: , rhythm evaluation toolkit, and a multilingual alignment extraction toolkit based on the UnitY2 aligner

* Responsible AI tools: detector

An overview of the technical components of and how they fit together.

The rest of the article is structured as follows: <Ref> contextualizes the sociotechnical need for expressive and streaming speech translation via an interview study with users who experience language barriers in their day-to-day lives. Then, it outlines existing technical efforts that tackle this issue, followed by a list of tasks and languages our models support. <Ref> details the various improvements made to to create . <Ref> and <Ref> detail the data and modeling techniques devised to train models that supports both expressive and streaming multilingual translations.
<Ref> reports how we bring and together to form .
Subsequently, <Ref> documents the automatic and human evaluation of our translation outputs, and the robustness of our models in various settings.
<Ref> homes in on our Responsible AI effort, where we provide details on our red-teaming, added toxicity detection and mitigation, gender bias evaluation, and watermarking efforts.
Finally, we conclude in <Ref>, where we discuss the social impact of our work and offer a forward-looking perspective on how could spearhead the transformation of multilingual communication in the near future.

§ BEYOND WORDS: EXPRESSIVE AND STREAMING SPEECH-TO-SPEECH TRANSLATION

In this section, we discuss the sociotechnical need for and the current technical landscape behind developing systems that facilitate expressive and streaming speech-to-speech translation. Then, we outline our contributions by summarizing the capabilities and language coverage for each of our models.

§.§ Towards Naturalistic Speech-to-Speech Translation

For long, investments in natural language processing (NLP) and machine translation research have coalesced around the text modality [NLLB Team et al., 2022, Seamless Communication et al., 2023]. While this has given rise to systems that help us translate books, webpages, and text messages, speech translation has lagged behind in terms of language coverage and performance. As a denser modality, the very paralinguistic features (e.g., prosody, tone, timing controls, etc.) that make speech challenging from a computational perspective are also why S2ST systems are filled with promises [Kraut et al., 1992, Nakamura, 2009]. The consummate system, which resembles the fictional Universal Speech Translator in Star Trek, would seamlessly offer expressive and real-time translation without excessive tinkering. Fading into the background, such a tool would provide utility without the drawbacks of existing paradigms—from waiting for translations to begin only after the completion of a sentence (i.e., offline systems that perform consecutive translations)
to monotonic outputs lacking in character.

To better understand user needs when it comes to speech translation, we ground our research on the lived experiences of individuals who are dependent on translation technologies in their everyday lives. While many people use translation technologies while traveling or for other recreational purposes, this group of individuals relies on them for essential information gathering and communication. Accordingly, we interviewed 34 participants from diverse immigrant backgrounds to understand present limitations in real-world deployments of S2ST systems. The goal of this study was to understand how our interviewees, who are either Mandarin or Spanish speakers with limited English proficiency, navigate everyday communication in the United States. The narratives drawn from these interviews not only spotlight the integral role of machine translation in achieving everyday goals, but they give us an empirical window into how S2ST systems designed with naturalistic communication (i.e., with expressivity and streaming) in mind could help this population gain confidence in self-expression and spur further integration into mainstream society.

§.§.§ Meeting translation needs

As well documented by previous research, low proficiency in the languages of the receiving societies is a major source of anxiety and stress for many immigrants [Ding and Hargraves, 2009, Lueck and Wilson, 2011, Delander et al., 2005]. Aside from acquiring language proficiency through learning, many tap into other strategies to bridge communication gaps [Hutchins, 2009, Orellana et al., 2003]. In our interviews, we find that while most participants rely on both their personal networks and translation applications in their everyday lives, most day-to-day translation work is conducted by the latter (especially for those with higher degrees of technological literacy). Moreover, even though many commercially available translation platforms support both text and speech translation, the bulk of the translation tasks our participants perform via apps remain text-centric (i.e., translating emails, work-related documents, etc.).

This observation does not suggest that text-based translation needs supplant speech-based ones. The disparity could largely be attributed to the performance delta between text and speech-based translation tools and the lack of familiarity with speech translation functions in widely adopted translation platforms [Seamless Communication et al., 2023]. Compared to speech translation systems, text-based tools have enjoyed deeper maturity and commercial viability. User familiarity, alongside greater confidence levels in the generated outputs, drives more users to deploy text-based systems even in contexts where speech is used. It is, for example, more common for participants (n=26/34) to translate subtitles rather than audio speech when watching the news or television shows (even though the translation apps they use support both text and speech translation). One participant added that “speech translation just feels foreign” and that they would probably engage with it more if they saw more people using it. This is a sentiment that reverberated across the sample population.

§.§.§ Real-time translation in synchronous contexts

Despite heavy reliance on text-based translation, many participants yearn for reliable speech translation systems to help them in real time. In fact, 20 of the 34 interviewees have previously used commercially available translation platforms supporting speech. That said, using translation apps to perform consecutive speech translations was universally regarded as a workaround in time-sensitive situations, an imperfect solution to a problem. A Mandarin speaker, for example, describes a recent incident when checking out at a local grocer: “I had to give the cashier the phone, ask her to direct her question to it, and then wait for the app to translate. You can tell people around me were a little annoyed.”

For all interviewees, real-time translation would be particularly handy in social situations that require synchronous communication, whether it is face-to-face or digitally mediated. For instance, even though one could rely on text translation to render a menu legible, conversing with or responding to questions from a server poses an issue. Barriers like this not only adversely affect self-esteem but also prevent many participants from partaking in new social or cultural experiences. One participant notes that even if a speech translation system does not enable bidirectional communication, having the capability to interpret a question as it is being asked would be helpful, later adding: “at least I could gesture back or use simple English to tell someone what I want.”

For some, the lack of reliable S2ST compels them to rely on family, friends, or coworkers to help meet cross-lingual conversational needs in both informal and professional settings [Orellana et al., 2003]. However, having network resources to tap into this social workaround is not a given, and even those who have access to cultural brokers [Sánchez and Orellana, 2006] express that this form of linguistic dependency stifles integration into their receiving society. In light of these constraints, one participant fantasizes that having a tool that “translates like a human, especially in circumstances where human interpreters are unavailable, could be a game-changer.”

§.§.§ Expressive translation and the preservation of vocal style & prosody

When probed on what the next generation of speech-translation technologies should look like, many participants stress that beyond simultaneous translation, future systems should enable them to communicate naturally. For them, natural communication could be interpreted in many ways—from using slang or idioms to not slowing down when directing input at a translating app. That said, the most commonly shared conceptualization (n=29/34) of naturalness is for S2ST systems to support prosodic preservation and the preservation of the style of one's voice.

If text directs more attention to the content of a message, then speech more deeply emphasizes the person behind an utterance [Kraut et al., 1992]. The desire for translation outputs to reflect speaker characteristics, as framed by an interviewee, suggests that S2ST systems can do way more than convey semantic information [Huang et al., 2023]. Without encoding the expressive nature of speech, many participants express that a major fear of engaging with S2ST in their day-to-day lives is the risk of misaligned intent. Consider this comment by a Mandarin speaker: “Imagine if I wanted to say something sarcastically. If the system does not translate that properly, it could lead to miscommunication and misunderstandings.”

Extending this sentiment, other participants noted that faithfully reproducing vocal style and prosody in their speech breathes character into their self-expression, giving listeners a more comprehensive sense of their intent [Du et al., 2021]. According to a Spanish-speaking participant, systems that go beyond just words can deeply transform the quality of cross-lingual communication: “Our tone is a part of our personality, and it changes based on context and the language we are speaking. It’s also a matter of candor when we speak Spanish. We get very passionate.”

Reverberating across the interviews is the view that translation systems that deliver language coverage, expressivity, and streaming could serve as a unique tool that helps them better integrate into everyday society. Equipping those with language barriers with the ability to communicate in real-time without erasing their individuality could make prosaic activities like ordering food, communicating with a shopkeeper, or scheduling a medical appointment—all of which abilities non-immigrants take for granted—more ordinary.

§.§ Expressive and Streaming S2ST Today

Having explored the social need behind expressive and streaming S2ST systems, we now review existing efforts directed at these research areas.

§.§.§ Expressive systems

Expressive speech systems have long been of technical interest to researchers in a multidisciplinary context. Combining linguistics insights and computational methods, developing systems that can accurately produce humanlike utterances both at the semantic and paralinguistics levels becomes ever more pressing as the volume of auditory content (i.e., podcasts, audiobooks, short-form videos, etc.) and voice-assisted technologies (e.g., smart home systems, autonomous driving voice controls, etc.) are on the rise. As a technical foundation, expressive speech systems could meaningfully augment the performance of a wide variety of technologies, ranging from robotics to digital assistants.

In the translation context, expressive speech preservation with conventional cascaded S2ST systems can be realized in several ways.
To preserve pre-defined word or token-level paralinguistic characteristics such as emphasis, automatic speech recognition systems (ASR) need to transcribe speech not only into text but also into pre-defined prosody labels. Subsequently, a machine translation model then translates or maps these prosody labels from the source to the target text. Finally, a text-to-speech synthesis (TTS) model synthesizes the speech output with the corresponding labels [Aguero et al., 2006, Do et al., 2017]. For this pipeline to work, parallel data with aligned prosody labels is necessary.

To achieve sentence-level preservation of the style of one's voice, TTS systems supporting cross-lingual transfer through a set of embeddings that disentangle speech nuances such as semantics (i.e., characters or phonemes), stress or tone, vocal styles, and language are typically required [Liu and Mak, 2019, Casanova et al., 2022].
Recent advances in TTS have enabled voice style transfer through prompting via speech language model [Wang et al., 2023], flow matching [Le et al., 2023], and diffusion model [Shen et al., 2023].
Notably, TTS models can now be trained on non-parallel multilingual datasets and achieve cross-lingual transfer when stacked with translation models that predict semantic units [Borsos et al., 2023, Rubenstein et al., 2023, Dong et al., 2023, Wang et al., 2023].
Relatedly, voice-aligned speech could be generated with controllable TTS models, and such data enables the training of direct systems that support translations from source speech into target speech with a consistent vocal style [Jia et al., 2022].

Despite the recent advancements in and direct [Zhang et al., 2023, Rubenstein et al., 2023], a comprehensive system capturing semantic translation, rhythm, pauses,
and sentence-level preservation of the style of one's voice have yet to be realized. Our work explicitly tackles preserving all such features in under a unified framework.
To build our model, we first focused on addressing data paucity with aligned prosodic patterns and systematic evaluation methods.
Signal-based objective metrics, such as mel-cepstral distortion (MCD), exist for systems, but parallel data with aligned prosody and voice style are hard to come by [Neubig et al., 2014, Jia et al., 2022, Ward et al., 2023]. To rectify this, we devised data and textless vocal style conversion strategies to build parallel data with aligned expressivity and reference-free cross-lingual automatic evaluation methods that focus on the prosodic aspects of speech.

§.§.§ Streaming systems

In contrast to offline systems, which only start translating after the completion of a sentence, streaming systems translate as source utterances are being produced [Cho and Esipova, 2016]. The biggest technical challenge of effective streaming is striking a balance between low latency and translation quality. More specifically, a system with very low latency may miss important information, rendering a translation subpar, while a system with high latency creates excessive delays, compromising the flow of a conversation. Typically empowered by simultaneous translation policies, advanced streaming S2ST systems should dynamically decide whether to translate the next token or pause translating to absorb additional contextual information.

Research into simultaneous translation policies may be categorized into two principal categories: rule-based policies [Cho and Esipova, 2016, Dalvi et al., 2018, Ma et al., 2019] and learnable policies.
The main difference between the two policies lies in how a system waits for more input before translating. Rule-based policies rely on heuristics, such as waiting for $k$ tokens to be read before translating, while learnable policies use algorithms such reinforcement  learning  [Gu et al., 2017] or monotonic attention to make this decision.
Among the latter, monotonic-attention based models have been deemed to produce state-of-the-art performance in navigating the latency-quality trade-off [Raffel et al., 2017, Chiu* and Raffel*, 2018, Arivazhagan et al., 2019, Ma et al., 2020].
Recently, there has been a growing interest in adapting simultaneous policies to model speech inputs
[Ren et al., 2020, Ma et al., 2020, Ma et al., 2021, Wang et al., 2020].
To direct further attention to this underexplored area of research, recent shared tasks, such as one focused on simultaneous translation organized by the International Workshop on Spoken Language Technologies, have been established [Agarwal et al., 2023, Anastasopoulos et al., 2022, Anastasopoulos et al., 2021, Ansari et al., 2020]. These shared tasks serve as crucial avenues spurring researchers toward developing state-of-the-art models under standardized conditions.

Despite ongoing efforts dedicated to research on simultaneous translation, certain gaps require further exploration. For one, most research on streaming has focused on speech-to-text rather than speech-to-speech applications. The difference in output modality presents a technical challenge due to data and modeling constraints. Relatedly, most existing streaming models are designed in an ad hoc manner that makes them particularly sensitive to the dynamics of the offline models they are initialized on. For example, if improvements are made to a foundational offline model, it is typically quite challenging to adapt a newer streaming model to take advantage of these technical gains.

Contemporary streaming models predominantly focus on bilingual translations. However, many low-latency application scenarios consist of multiple speakers from diverse language backgrounds, calling for models that can process multilingual inputs and outputs simultaneously in an efficient manner. The development of multilingual streaming models, also an underexplored area of research, has an added advantage—cross-lingual transfer, which allows related languages to learn from one another [NLLB Team et al., 2022, Nguyen and Chiang, 2017].

Moreover, in the domain of streaming S2ST, the research has predominantly focused on a cascaded approach involving a sequential series of processing steps.
However, this approach is suboptimal for real-time streaming applications, a limitation that could be alleviated by direct S2ST models (especially when the scale of training increases).
Moreover, the cascaded model has issues such as compounding errors, additional disk storage, and computation time [Bentivogli et al., 2021, Seamless Communication et al., 2023].
To address these issues, we combine , our multilingual and multimodal foundational model, and Efficient Monotonic Multihead Attention (EMMA), our simultaneous policy, to build a streaming translation model that performs direct translations from speech into
both speech and text for many-to-many directions in real time.

§.§.§ The overarching goals of this effort

In light of the gaps delineated above, our work seeks to advance speech translation in the following ways:

* Developing key data sets and foundational models necessary to create a unified system that enables end-to-end, multilingual, and real-time speech translation that captures a broader range of vocal style and expressive preservation.

* Expanding language coverage both in terms of the number of supported languages and translation directions when it comes to and translation systems (i.e., going beyond translations into English by including translation from English).

* Maintaining systematic evaluations of our systems throughout our workflow to ensure high-quality and safe performance.
This allows us to understand how to direct our efforts to make both current and future iterations of our work more equitable and fair for different user populations.

§.§ Overview of Model Capabilities & Languages

Today, broadly accessible speech translation models cover anywhere between 21 to 113 source languages depending on the wide range of tasks involved [Zhang et al., 2023, Rubenstein et al., 2023].
To build a unified, multimodal, and multitask model that can handle both speech and text,
covers languages as speech input
and languages as text input.
It can output languages as text
and languages as speech.
, capable of preserving rhythm, pauses, and sentence-level style of one's voice, is equipped to handle six languages—English, French, German, Italian, Mandarin, and Spanish.
As for , our low-latency model can handle the same language coverage as on ASR, S2TT, and S2ST tasks.
We summarize information on our models' supported capabilities and languages in <Ref>. Further details on the table header are provided below.

We represent each language with a three-letter ISO 639-3 code.

There may be multiple ways to refer to the same language; due to formatting limitations, only one version is included below. The language names have been cross-referenced with major linguistic information platforms such as Ethnologue [Lewis, 2009] and Glottolog  [Hammarström et al., 2022].

We provide script information in ISO 15924 codes for writing systems.

Resource level. We categorize the speech resource level as high, medium, or low depending on the volume of available primary data for into English (with $x$ the amount of primary data in hours, high if $x>1000$, medium if $x\in[500, 1000]$ and low if $x\in[0, 500]$).

Primary data. Primary data is defined as open-source and pseudo-labeled ASR data. Absent such data, we report the language as zero-shot (when evaluating into English).

We indicate whether a source language is in the speech (Sp) or text (Tx) modality, or both.

We indicate whether a target language is in the speech (Sp) or text (Tx) modality, or both.

left=1cm, right=1cm, bottom=0.5cm, top=0.5cm

5cTask Language Coverage

Coverage of the models. A list of supported tasks and their coverage expressed as $n_s{-}n_t$
where $n_s$ and $n_t$ are the number of languages supported as source or target respectively. $^\dagger$: the task of is evaluated zero-shot.

The first step towards a unified model, capable of expressive cross-lingual translation in real-time, starts with improving to give rise to —a foundational model with state-of-the-art semantic accuracy, wide language coverage, and multitasking capabilities (from and into text or speech). In terms of coverage, supports the same tasks as with the same set of languages
detailed in <Ref> and summarized in <Ref>.

When designing the newer version of , adaptability to simultaneous translation was central.
Due to the length mismatch between discrete acoustic units and text semantic tokens, the model of responsible for generating units tends to hallucinate or truncate the output. This is particularly problematic if the model is only fed partial input and is tasked with generating partial outputs for real-time applications.
For this reason, we pivoted in to non-autoregressive text-to-unit decoding in order to decouple generation from output length prediction. With this non-autoregressive decoder, 's inference speed has improved by 3x (see <Ref>) laying the ground for effective real-time translation with .

We followed the same recipe from and relied on pre-training multiple blocks before finetuning them jointly as a unified model.
Our unified model, previously a multitask-architecture, was upgraded to multitask-, boasting a stronger non-autoregressive model. Compared to its predecessor, delivers stronger T2U performance thanks to its hierarchical upsampling from subwords to characters and then to units. This upsampling makes pre-training multilingual T2U models much more data-efficient. also used 4.5M hours of unlabeled audio data to learn its self-supervised input speech presentation with (4.5x the amount used in v1). was further extended to cover more low-resource languages, enabling increased representation of these languages, ultimately improving the downstream semantic accuracy.

The key ingredients of the recipe are:

* Unlabeled, human-labeled, pseudo-labeled, or automatically aligned data used in the different pre-training and finetuning stages (<Ref>). <Ref> gives a bird's eye view of the different sources of data and how they were used.

* model pre-trained on NLLB data [NLLB Team et al., 2022] in nearly 100 languages [Seamless Communication et al., 2023].

* Conformer speech encoder pre-trained with the algorithm.
We scaled up the amount of unlabeled data from 1 million to 4.5 million hours of audio (<Ref>).

* model trained on different sources of data (human-labeled, pseudo-labeled, and automatically aligned). This model is trained with knowledge distillation to jointly support , , and by combining the models from (a) and (b) (<Ref>).

* based on a novel non-autoregressive decoder architecture with hierarchical modeling of subword, character, and discrete units. relies on unsupervised multilingual character-to-unit alignment learning and introduces a novel span-based glancing for the decoder  (<Ref>).

* Multitask-model finetuned on speech-to-unit data (pseudo-labeled with a teacher or automatically aligned) to build on the model from (c) with a student model (<Ref>).

We evaluated (a -size model) across all its supported tasks [, , , , (zero-shot)] and discuss its results in <Ref>.

Data for speech translation. An overview of the pre-training and finetuning data used in .

§.§ Pre-Training

§.§.§ Self-supervised speech representation

Scaling data size for self-supervised
pre-training has been empirically proven to be a relatively cheap, yet effective way to improve speech representation quality [Zhang et al., 2023]. Following such direction, we continued to add more
unlabeled speech data,
increasing the amount of our pre-training data from 1M hours [Seamless Communication et al., 2023] to approximately 4.5M hours.

Besides leveraging more pre-training data, we removed the random-projection quantizer (RPQ) [Chiu et al., 2022] and its associated loss previously incorporated in v1 [Seamless Communication et al., 2023].[
As we scaled data from 1M to 4.5M hours, we encountered some optimization instability when RPQ was used.  We decided to simply discard RPQ instead of relying on more extensive hyperparameter tuning.]
Akin to v1, the v2 comprises 24 Conformer layers [Gulati et al., 2020] with approximately 600M parameters and the same pre-training hyperparameters.

Model     Languages     Hours     Model type     Open model

USM     over $300^\dagger$     12M     BEST-RQ [Chiu et al., 2022]

MMS     1406     0.5M     wav2vec 2.0 [Baevski et al., 2020]     $\checkmark$

over $\wvbertlangs^\dagger$     1M         $\checkmark$

over $\wvbertlangs^\dagger$     4.5M         $\checkmark$

A comparison of multilingual speech pre-training in state-of-the-art and models. $^\dagger$Estimated from the part of data that has language information.

§.§.§ : Into-text tasks

In , we leveraged foundational models either pre-trained on unlabeled data (for speech encoder pre-training) or trained on supervised high-resource tasks (NLLB model for ) to improve the quality of transfer tasks (speech-to-text and speech-to-speech).
To fuse these pre-trained components and enable meaning transfer through multiple multimodal tasks, we trained an end-to-end model with:
(a) a speech encoder () postfixed with a length adapter,
(b) text encoder (NLLB encoder), and
(c) a text decoder (NLLB decoder).
We used the same length adaptor from  Seamless Communication et al., 2023.
The text encoder was frozen, and the model was finetuned to jointly optimize the following objective functions with respect to the speech encoder parameters $\paramsenc$ and the shared text decoder parameters $\paramtdec$:

\begin{align}
\mathcal L_{\text{\st}}(\paramsenc, \paramtdec)
&= -\log p(\tgttext | \srctext; \paramsenc, \paramtdec)
=- \sum_{t=1}^{|y|} \log p(\tgttextstep | \tgttextprefix, \srcspeech; \paramsenc, \paramtdec),\\
\mathcal L_{\text{\mt}}(\paramtdec)
&= -\log p(\tgttext | \srctext; \paramsenc, \paramtdec)
= - \sum_{t=1}^{|y|} \log p(\tgttextstep | \tgttextprefix, \srctext; \paramtdec),
\end{align}

where $\srctext$ and $\srcspeech$ are the source text and speech in the source language $\srclang$, and $\tgttext$ is the target text in the target language $\tgtlang$.
We additionally optimized an auxiliary objective function in the form of token-level knowledge distillation $\mathcal{L}_{\text{KD}}$ to further transfer knowledge from the strong MT model to the student speech translation task (). This loss function is defined as follows:

\begin{align}
\mathcal{L}_{\text{KD}}(\paramsenc, \paramtdec) & = \sum_{t=1}^{|y|} D_\text{KL}
\left[
p(. |\tgttextprefix, \srctext; \paramtdec)\, \| \,
p(. |\tgttextprefix, \srcspeech; \paramsenc, \paramtdec)
\right].
\end{align}

Our triplets $(\srcspeech, \srctext, \tgttext)$ come mainly from pseudo-labeled data (<Ref>). Since we jointly trained on data, handled as translation with $\srclang=\tgtlang$, we replaced the translation task for the samples with auto-encoding (AE). As such, three additional losses are considered:

\begin{align}
\mathcal L_{\text{\asr}}(\paramsenc, \paramtdec) &= -\log p(\srctext | \srcspeech; \paramsenc, \paramtdec),
\\
\mathcal L_{\text{AE}}(\paramtdec) &= -\log p(\srctext | \srctext; \paramtdec),\\
\mathcal{L}_{\text{KD-ASR}}(\paramsenc, \paramtdec) & = \sum_{t=1}^{|y|} D_\text{KL}
\left[
p(. |\srctextprefix, \srctext; \paramtdec)\, \| \,
p(. |\srctextprefix, \srcspeech; \paramsenc, \paramtdec)
\right].
\end{align}

The final loss is a weighted sum of all six losses:

\begin{align}
\mathcal L \propto  (\mathcal{L}_{\text{\st}} + \mathcal{L}_{\text{\mt}}+ \mathcal{L}_{\text{KD}})
+ \alpha (\mathcal{L}_{\text{\asr}} + \mathcal{L}_{\text{AE}}+ \mathcal{L}_{\text{KD-ASR}}),
\label{eq:offline:x2t}
\end{align}

where $\alpha$ is a scalar hyperparameter dependent on the proportion of data in our mix of training data.

We trained our model in two stages. $\text{Stage}_1$ targeted training on supervised English and into English data. We find that this step is necessary not only for improving the quality of translations but also translations. In fact, we hypothesized that allowing the model to focus on one target language while finetuning multilingual speech representations shields it from the interference that can propagate back from the target side. In $\text{Stage}_2$, we added supervised  and non-English data to the mix.

In , we set $\alpha$ (<Ref>) to 0.04 in the first finetuning stage and 0.13 in the second stage.
Our training batches present
a mix of tasks (or and the associated auxiliary losses),
and languages (source only in the first stage and source-target in the second stage) with temperature sampling ($T=2$).
All speech encoder and text decoder parameters are finetuned for a total of 200K updates—100K in each stage.

Illustration of the model. Panel (1) shows the three main blocks of with its non-autoregressive (NAR) . Panel (2) shows multitask-with its additional text encoder. Panel (3) breaks down the components of (a multitask-model) with a side panel illustration of the teacher model used for pseudo-labeling.

§.§ Training Setup.

Following and modeling in , we trained two NAR models for different purposes:
a teacher model used for unit pseudo-labeling (<Ref>)
and a student model used for initializing the sub-component in and finetuning on data.
Both models are based on the NAR decoder architecture (<Ref>).

Teacher pre-training.
Since discrete unit sequences are much longer than subword sequences, we occasionally observed hallucination during unit pseudo-labeling with an auto-regressive model.
NAR models, on the other hand, rarely hallucinate because duration modeling is decoupled from sequence generation.

The teacher NAR model takes characters as inputs and forgoes the subword-to-character upsampling; it takes ground-truth text for input as opposed to a text decoder output (<Ref>).
The teacher consists of 12 encoder and 12 decoder layers.

Student pre-training.
The student NAR  takes subwords as inputs and consists of six encoder and six decoder layers.
The decoder architecture is exactly the same as the unit decoder in .

Finetuning multitask-.
In the third finetuning stage of , the multitask-model is initialized with the pre-trained and the student NAR models described above. The model is frozen, and only weights corresponding to the model are updated during this finetuning stage. The model is finetuned on a combination of pseudo-labeled and aligned and  data totaling
145K hours (see <Ref>).

The new NAR architecture with the pre-trained alignment module between text and units led to superior performance and faster convergence. Given that all components are pre-trained on related tasks (, , and ), the model converges after less than an epoch.

Multilingual HiFi-GAN unit vocoder.

Unlike , which uses the multitask-architecture, predicts duplicated (non-reduced) units.
As such, we re-trained the unit-based HiFi-GAN vocoder from  [Seamless Communication et al., 2023, Gong et al., 2023] on data to convert the duplicated units to waveform without performing duration prediction.

§.§ Results and Discussion

In this section, we trained , a model in the multitask-architecture with the same coverage (i.e., tasks and languages) as  [Seamless Communication et al., 2023]. A card for this model is available in <Ref>.

We evaluated on all four supervised tasks (, , , and ), as well as the zero-shot task of text-to-speech translation (, also referred to as cross-lingual text-to-speech synthesis [Zhang et al., 2023]).

To generate text hypotheses, we decoded with beam-search (width${=}5$).
We scored with
and with SacreBLEU [default 13a tokenizer and character-level tokenizer for Mandarin Chinese (cmn), Japanese (jpn), Thai (tha), Lao (lao), and Burmese (mya)].
For , following Radford et al., 2022, we scored normalized transcriptions and references with (word error rate). See metric details in <Ref>.

During and inference, we performed two-pass beam-search decoding—the best hypothesis out of the first-pass decoding is embedded with the text decoder and is sent to to search for the best unit sequence hypothesis. We used a beam-width of 5 for both searches.
We evaluated and accuracy with  [Lee et al., 2022] with as the underlying model.[This is different from Seamless Communication] et al., 2023, where was used for directions and was used for directions. We re-evaluated models here with for a direct comparison.
We set the decoding temperature of Whisper at zero and used greedy decoding to ensure a deterministic behavior of the ASR model.
The transcribed hypotheses, as well as the references, are normalized following [Radford et al., 2022] before computing scores (with the tokenization described for ). In the following, we report averages for the per-language scores across all the evaluated tasks (see <Ref>).

Comparison to and cascaded models.
On the set of languages supported by both /and the baselines included as a reference, we compare in <Ref> the performance of our unified and direct model to that of the first version of , as well as cascaded models.
We evaluated on   using <https://github.com/openai/whisper/>. For , we used the results from  Radford et al., 2022.
For , the cascaded models comprise Whisper models and NLLB models.
For , two options were considered for cascading: (1) 3-stage with , , and and (2) 2-stage with and .
We used for English-TTS [Casanova et al., 2022] and MMS's TTS models for non-English[Only 26 of our supported languages are serviced by MMS's models]  [Pratap et al., 2023].

In , achieves state-of-the-art performance in , improving in by 10% over , and by more than 17% over the strongest cascaded model (+ ). When compared against direct models (Whisper and AudioPaLM), significantly outperformed both in directions by more than 35%.[We evaluated the recently released on 's and found it to be worse than .]

In speech-to-speech translation, improves over in by more than 15% in and 25% in . Compared to the strongest cascaded models, this is an improvement of 25% and 15% in and , respectively. Results on show a similar trend and a consistently strong performance with generalizability to other domains.

Mulitasking results.
We compare in <Ref> the performance of to that of state-of-the-art models in and tasks.
Evaluated for  , on the overlapping 77 languages between and ,
improved over by a relative -21% and over by a relative -56% .
For comparison against MMS, we also report the average on -54, where improves over by a relative -19% , closing the gap with MMS's best model (MMS-L61-noLM-LSAH) to -0.4 .
We also compared 's performance to the recently released . Evaluated on 60 languages from (as reported in the release[<https://github.com/openai/whisper/discussions/1762>]), improved over by -4.4% .

Evaluated for , 's performance on drops by -1.6  in both and when compared to .
Its accuracy is still, however, on par with the equally-sized for and for .

Evaluated on  [Wang et al., 2021], a multilingual benchmark dataset, improved over by +2.5 in directions and by +1.1 in directions. In directions still lags behind (-1.2 ).

2c ($\uparrow$)

35.0      22.7

36.4     23.7

26.3     18.4

34.1     21.8

35.9     27.6

Zero-shot  . We report the average of on  .

Zero-shot text-to-speech translation.
We next evaluated on the task of in a zero-shot way.
Given that collected three recordings by three different native
speakers for each sample, we randomly select one for the task of (the input being text).
We report in <Ref> a comparison between models and cascaded models with NLLB and either (English ) or MMS (non-English ) for synthesizing translated text.
We averaged scores over 88 directions (the overlap between and the languages supported by ). We also averaged over 26 directions (the overlap between our and the languages supported by MMS's models).
improved by a large margin over (+1.8 and +5.8 points in and respectively).
Compared to cascaded models, 's zero-shot capability is on par with + in , and outperforms + MMS by more than +3.9 points in .

$\uparrow\Delta$     $\uparrow\Delta$    $\downarrow\Delta$

Low (n=42)    +2.8     +4.3     -7.5

Medium (n=26)     +3.0     +4.5     -4.7

High (n=16)     +1.7     +2.9     -2.9

Improvement from to . Delta of performance in 's  ,  and between and .

Results by resource-level.
We show in <Ref> the improvements in   ,  and achieved in when buttressed with additional supervised data (mostly automatically aligned) and unlabeled data used to train our speech encoder.
Our efforts to increase supervised and self-supervised data targeted low- and medium-resource languages.
Overall, improved on low-resource languages by an average of 2.8 points, 4.3 points and -7.5 in the three tasks respectively. As for medium-resource languages, it improved by an average of 3.0 points, 4.5 points and -4.7 respectively.

Ablation on the input representations for .
We investigated better input and output representations for both AR and NAR models. To do so, we compared subword and character as input with reduced and non-reduced units as output in <Ref>.
We found that the previous setting with subword input and reduced unit was the best for the AR  model, while character input and non-reduced unit were the best for the NAR  model.
The best NAR  model outperformed the best AR  model by 35%
in ASR-WER.

Text input
Output units

Subword     Reduced     20.79

Subword     Non-reduced     24.78

Character     Reduced     35.49

Character     Non-reduced     78.35

Subword     Reduced     16.66

Subword     Non-reduced     16.54

Character     Reduced     13.91

Character     Non-reduced     13.41

A comparison of input and output representations in teacher modeling.

Model     $\downarrow$ASR-WER

NAR     13.41

w/o GLAT     14.97

w/o Span-based masking     15.17

w/o Efficient GLAT     13.54

w/o InterCTC     13.92

Ablation studies in character-level teacher NAR modeling.

Ablation studies in modeling. In each set of experiments, we calculated ASR-WER in 32 of 36 languages since ASR performs poorly (> 50%) for Bengali (ben), Maltese (mlt), Telugu(tel) and Northern Uzbek (uzn).

Ablation on the modeling of NAR .
We next conducted an ablation study of the proposed NAR modeling in <Ref>.
We confirmed that GLAT significantly improved intelligibility and both span-based masking and character-level InterCTC also contributed to further improvement.
Efficient GLAT did not degrade ASR-WER despite a single forward pass.

's  multilingual char-to-unit aligner.
The -based aligner component, used as a duration teacher in the T2U training, presents itself as a universal tool to align arbitrary text-audio pairs for any downstream task.
The presence of extremely large, unlabeled audio corpora makes this tool very attractive for pseudo-labeling.
We release a multilingual aligner component that supports all target languages of , together with a front end for alignment extraction.
The front end uses a character-based Sentence-piece model to tokenize a raw text sequence and a 10K acoustic unit extractor, which outputs a discrete unit sequence from 's unit space.
We found that our aligner also works pretty well when using a normalized text. A model card describing the aligner component can be found in <Ref>.
<Ref> shows an example of a Russian audio sample aligned with its transcription, where the waveform exhibits variable speech rate.
In this work, we utilized this alignment extraction tool as the core component behind the automatic pause alignment evaluation (see <Ref> for more details).

Visualization of an alignment with 's aligner. Example of a Russian audio aligned with its transcription
“проверка нашего элайнера, можно говорить быстро, а можно медленно.” The purple vertical lines show the predicted character boundaries.

Prosody contains rich paralinguistics functions in human communication, such as portraying a speaker's emotional state, attitude, and intent. How a speaker says an utterance can dramatically alter its meaning (holding semantic content constant). For instance, humans leverage variations in pitch (high or low), loudness (strong or soft), and duration (fast or slow) to express themselves in different situations.

In this section, we describe how we built , a model that captures certain underexplored aspects of prosody, such as speech rate and pauses, while preserving the style of one's voice and high content translation quality.
More specifically, we developed with the following techniques:

* we leveraged as a foundational model to achieve high accuracy in translation quality from a semantics standpoint,

* we proposed , integrating an expressivity encoder in to guide unit generation with proper rhythm, speaking rate and pauses, and

* we replaced the unit HiFi-GAN vocoder in with , an expressive unit-to-speech generator conditioned on the source speech for waveform generation to transfer tones, emotional expression, and vocal style.

, which preserves not only sentence-level rhythm and tone but also token-level prosody such as pauses, required prosody-aligned parallel speech data for training. As a result, we describe our effort to collect hours of prosody-aligned parallel speech data in six high-resource languages—English, French, German, Italian, Mandarin, and Spanish.

§.§ Expressive Speech-to-Speech Translation Data

In this section, we introduce our efforts on collecting prosody-aligned parallel speech through data commissioning, automatic alignment, and synthesizing. Commissioned data, including mExpresso and mDRAL, are well-aligned in emotions but limited in data size and diversity. We explore large-scale expressive aligned data—finding expressivity preserving cross-lingual alignments between speech segments from corpora. Finally, synthetic data is part of our data augmentation strategy with , controllable TTS (cTTS), and , which contributed a large amount of aligned expressive speech.

§.§.§ mExpresso

The Expresso corpus [Nguyen et al., 2023] is an English expressive speech dataset that includes both expressively rendered read speech (comprising eight styles) and improvised dialogues. We created mExpresso, a multilingual version of Expresso, by expanding six styles of read speech (i.e., default, happy, sad, confused, enunciated, and whisper) to five other languages—French, German, Italian, Mandarin and Spanish.

To expand the Expresso dataset, bilingual translators first translated the English transcriptions into other languages, including the emphasis markers in the transcription. Second, a different set of gender-matched bilingual speakers (native in the target languages) read the translation in the style suggested by the markers. The speakers had access to the original English recordings to learn how a sentence was uttered initially. To control the quality of the recording, a different set of bilingual reviewers reviewed each recording to check the expressiveness preservation and recording quality, and the speakers re-recorded utterances until all recordings passed the quality check.

§.§.§ mDRAL

Dialogues Re-enacted Across Languages (DRAL) Corpus, proposed in Ward et al., 2023, is a bilingual speech corpus of parallel utterances in Spanish and English created by recording spontaneous conversations and fragments re-enacted by bilingual speakers in a different language.
More specifically, during a recording, two speakers were instructed to carry out unscripted conversations. The moderator then selects “interesting” fragments, which are utterances that elicited more active engagement between the speakers and guided the speakers to re-enact.

We followed the data collection protocols described in Ward et al., 2023, expanded the collection to native speakers of French, German, Italian, Mandarin, or Spanish who are proficient in English, and created the multilingual DRAL corpus dubbed mDRAL (see <Ref> for an overview of the collection protocol).
Unlike the original DRAL collection, we performed the collection remotely with the moderator and the two speakers meeting over Zoom. One challenge in scaling up the data collection effort is the throughput—the number of meaningful speech segments we can acquire from each conversation. We provided the speakers with 32 emotion categories found in EmpatheticDialogues [Rashkin et al., 2019] as topic prompts to increase data collection efficiency. Compared to mExpresso, mDRAL has less exaggerated and performed emotions, while the prosody is more natural.

§.§.§ Automatically extracting expressive audio alignments

Speech-to-speech pairs that are automatically aligned based on semantics (like ),
do not always contain the same expressive characteristics.
While a simple filtering approach
based on heuristics could be devised, the volume of the resulting dataset would likely be drastically reduced as no explicit prosody-preservation goal was enforced to begin with (i.e., data would be expressively aligned by chance). Instead, we chose to modify the algorithm of to not only seek alignments based on semantic preservation but also to incorporate prosodic similarity.

The core algorithm behind relies on the computation of a semantic-based margin score (see <Ref>). We supplement that semantic score with the result of an auxiliary model capable of determining prosodic similarity.
Based on these two components, we introduce a new weighted scoring function defined as:

\begin{equation}
\label{eqn:blended_score}
\text{blended-score}(x, y) = \alpha \cdot \text{margin} + (1 - \alpha) \cdot \text{prosody-score}.
\end{equation}

Given the above formulation, an overview of the expressive audio alignment is shown in <Ref>. We began with the same process as by semantically retrieving the k-nearest neighbors in a multilingual embedding space. Then, instead of choosing a neighbor with the best margin (i.e., semantic score), we applied a prosodic-based auxiliary model to each neighbor and chose a candidate with the highest blended score as defined in <Ref>.

Expressive audio alignment process. Similar to , audio was first language-identified and over-segmented, and then the resulting segments were embedded into a multilingual embedding space. k-nearest neighbor candidates were then retrieved based on the semantic-based margin and subsequently re-ranked with the blended score, resulting in expressively- and semantically-aligned pairs.

In order to tune the $\alpha$ hyperparameter in the blended score, which controls the trade-off between semantic accuracy and prosody preservation, we introduce a new proxy metric for expressive audio alignment: . This new benchmark builds upon , introduced in NLLB Team et al., 2022. Unlike , where the goal is to reconstruct a dataset through semantic-only audio alignment and measure the percentage of incorrect alignments, instead applies the same re-ranking as described in <Ref> and aims to reconstruct a dataset both semantically and expressively. We applied to the mExpresso dataset (see <Ref>). Our choice was driven by a need for variety. For one, mExpresso contains sentences repeated multiple times with varying prosody. This makes it a challenging dataset for expressive audio alignments, as multiple candidates with identical semantics will be retrieved during the $k$-nearest neighbor search. On the contrary, a prosody-aligned dataset with no repetition would offer no such challenge, as alignments could be recovered based on semantic features only.

Results from on the mExpresso benchmark dataset are shown in <Ref>. We tuned $\alpha$ using a grid search. To help our explorations, we also collapsed the mExpresso classes into three coarse emotion labels similar to Parry et al., 2019: positive (happy, laughing), negative (sad, confused), and neutral (default, whisper, enunciated). As a baseline, we used the margin score only and then tried several auxiliary models for the prosody score, namely [For expressive audio alignment, we used an earlier version of the model. It has the same architecture as the model we used for evaluation and it uses embeddings from a different layer of the XSL-R speech encoder.] (<Ref>),
and different embedding layers extracted from w2v-BERT. Adding a prosody-aware component to the audio alignment scoring function clearly boosted performance, and provided significantly higher quality alignments than representations from w2v-BERT.

2* Opt. param.
$\alpha$      2c$\downarrow$
all emotions                      pos/neg/neu

semantic-only baseline     -                                     84.86                             60.57

w2v-BERT - layer 23         0.2                                  84.79                             60.27

w2v-BERT - layer 20         0.1                                 54.40                             36.04

0.1                                  47.06                    28.90

Performance of . Error rate when recreating gold alignments of various prosody-aware auxiliary scorers on the Spanish$\shortrightarrow$English mExpresso test set.

Once the $\alpha$ parameter was optimized using , we ran expressive audio alignment at scale on a curated selection of publicly available data in our target languages. The resulting alignments were used to supplement the final training data for .

Separate to the data used to train , we apply this expressive alignment method at scale on a different publicly available corpus. The total number of hours collected can be found in <Ref>. We release the metadata of this data set to the community as to foster future research in expressive speech-to-speech translation as well as to validate the effectiveness of our expressive alignment method.

Upon manual inspection, we identified several emerging properties when several semantically viable candidates were available:

* expressive audio alignment seems to remove candidates with mismatched background music,

* emotion/intonation imbalance is highly reduced, and

* segments with singing are also much less common in final alignments.

While further analysis of those properties was out of the scope of this study, we hypothesized that expressive audio alignment could also have a net-positive effect on non-expressive speech translation, as it produced much cleaner alignments overall.

§.§.§ Extracting parallel segments from videos

We also processed videos in multiple languages to extract bilingual expressive segments.
This process is different from the standard audio alignment approach described in <Ref> because the audio data is almost parallel in this case.
The task is then to segment and monotonically align the multilingual audio content.

The process was performed as follows: the audio was extracted from the video, segmented, and transcribed with Whisper [Radford et al., 2022].
One issue we faced is that the segmentation provided by Whisper is often inconsistent across languages.
Therefore, the segments cannot be matched directly, leading to a low recall.
To solve this issue, we took advantage of the word boundaries provided by Whisper and adopted a split-merge approach, which consists of first splitting the current segments based on the pauses (available by analyzing the transcriptions) and then concatenating them back together to form new overlapping segments.
Segments longer than 25 seconds and segments having pauses longer than 1.5 seconds were excluded.
Then, the speech segments and their transcriptions were each encoded separately with our encoders
to produce two embeddings per segment.
The next step was to align the segments. If they were disjointed, we could use a simple monotonic alignment algorithm.
Yet, if not the case, finding an optimal solution would be intractable due to the large number of alignments to consider.
Therefore, we used a greedy algorithm that matched bilingual segments having the highest overall score, removed all overlapping segments from the pool, and repeated the process until the candidate pool was empty.
Each segment pair candidate was associated with a score corresponding to an average of the cosine similarities of both the text and speech embeddings.
This score was modified according to an estimation of the lag (i.e., the time gap between the centers of both segments).
Finally, all matching candidates were filtered based on predefined rules (defined by manually inspecting the data), such as similarity threshold, duration mismatch, and time gap.
<Ref> shows the statistics of the resulting aligned data.

Language     2cTotal hours     2*# segments     2c [c]Avg. segment
duration (s)
English      Lang.                             English      Lang.

French      300.7     299.1     499.0k     2.17     2.16

German      118.8     121.8     224.2k     1.91     1.96

Italian       69.3      68.1     122.4k     2.04     2.00

Mandarin      254.9     286.1     268.5k     3.42     3.84

Spanish      242.6     237.9     363.4k     2.40     2.36

Statistics of aligned audio data. The total duration and average segment length per language are reported for data obtained from aligning multilingual videos.

§.§.§ SONAR expressive

SONAR is a multilingual and multimodal fixed-sized sentence embedding space introduced by Duquenne et al., 2023. The modalities cover both text and speech representations. However, this space is primarily grounded in text as it was tuned on speech-to-text
and text-to-text datasets. Given this grounding in text, the space is centered on semantics, so the existing space is not explicitly tuned to encode anything other than semantics from the input text or speech. [Duquenne et al., 2023] extends the capabilities of this space to also include representations for prosodic characteristics.

An overview of the architecture of is shown in <Ref>. It comprises two encoders: a frozen text/speech encoder to capture semantics (embedding) and a trainable speech encoder that captures speech properties other than semantics (embedding). Then, given a combination of both the vector (i.e., prosody, etc.) and the semantic vector, the objective is to reconstruct the input speech, represented using EnCodec units [Défossez et al., 2022].

Model architecture for Expressive.

Given that has the ability to expressively decode input speech, we leveraged this as another data source for model training. We began with unaligned speech segments,
applied the same pre-processing as used for model pre-training [Duquenne et al., 2023], and randomly sampled segments from each non-English language (French, German, Italian, Mandarin, and Spanish). As we observed that semantic preservation for the semantic encoder was higher given an English text-based input [Duquenne et al., 2023], segments from each non-English language were translated into English text using the encoders/decoders. Each non-English speech segment and English text translation were then expressively decoded into English. An overview of the decoded data is shown in <Ref>.

Language     2cTotal hours     2c [c]Avg. segment
duration (s)
English      Lang.                English      Lang.

French      1,651     1,784     2.97     3.22

German      1,622     1,865     2.92     3.36

Italian      1,562     1,891     2.82     3.41

Mandarin       1,672     1,694     3.02     3.06

Spanish      1,567     1,841     2.83     3.33

Statistics of data decoded with Expressive.

§.§.§ Controllable TTS (cTTS) data augmentation

One limitation of automatically aligned expressive data is that the prosody of the audio data may not be perfectly aligned between source and target speech (e.g., speech rate and pause location).
A controllable TTS (cTTS) system is able to control the speech rate and pause location of the synthesized speech from the text prompt. Therefore, we leveraged controllable TTS to synthesize more prosody-aligned speech-to-speech data.

We first sample English monolingual text to augment from. Then, we inserted one paired quote into each English text and ran NLLB [NLLB Team et al., 2022] Dense-3.3B model for translating into all five languages (French, German, Italian, Mandarin, and Spanish).
Followed by further filtering on the translation output to ensure only one paired quote exists, we randomly replaced the paired quote in the source and target text with one of the three augmentation instructions: 1) no augmentation, 2) equal chance to insert the pause at the first or second quote, with a randomly chosen pause duration between 0.3 and 1.5 seconds and the quote converted to a special pause token.
We used an internal controllable TTS system for all languages except Mandarin Chinese.
For Mandarin Chinese, we trained a VITS [Kim et al., 2021] model on 15-hour speech data.
We used these systems to synthesize speech with a random utterance-level speech-rate manipulation between 70% and 130%.

§.§.§ Comparing across data sources

2cCommissioned     2cAutomatically Aligned     2cSynthetic

mExpresso     mDRAL       Expressive
Expressive          cTTS

Style     acted     spontaneous     spontaneous     acted      spontaneous and
synthetic pairs      synthetic

diversity     low     low     high     medium     high     low

Expressiveness     high     medium     medium     high     medium     low

7l[l] Expressivity  alignment

[l]  Sentence-level

speech rate

same voice

Datasets characteristics. We compare commissioned, automatically aligned, and synthetic data on style, speaker diversity, expressiveness, and sentence-level prosody alignment.

<Ref> describes the characteristics of each dataset in four aspects: style, speaker diversity, expressiveness, and expressivity alignment.
Spontaneous style indicates that the speech is more natural, while acted speech implies that the speech can be more expressive yet less natural.
The commissioned datasets have the lowest speaker diversity because the data collection was expensive and time-consuming.
While expressive alignment can provide a large amount of parallel data, such speech pairs are mostly aligned in sentence-level styles due to the choice of prosody score. Further filtering can be done to refine the datasets to be better aligned in speech rate and pauses.
In theory, video alignment should generate speech pairs with the best expressivity alignment. However, we find that due to the constraint of time alignment in videos and the characteristics of different languages, speech for some languages may be much faster than others.
Controllable TTS data provides speech pairs that have the best alignment in speech rate and pauses, but the speech is monotonic and lacks sentence-level expressiveness such as emotions.

§.§.§ Training data pre-processing

Once data was collected, we then performed the following augmentations in order to form (source-target) speaker-aligned, clean speech data with transcriptions: (1) denoising, (2) silence removal, (3) transcription, and (4) vocal style conversion. Since datasets come from various sources (with varying audio qualities), not all preprocessing steps described above must be applied to each. For example, cTTS has no background noise, so no denoising was needed. We have a commissioned dataset with no background noise but with leading and trailing silence, so silence removal is required. Vocal style conversion was applied to all datasets except for Expressive since we observed such qualities were already mostly preserved. Since cTTS and commissioned datasets already had transcriptions available, no ASR was needed.
An overview of which preprocessing steps were applied for each dataset is shown in <Ref>, and an overview of the number of hours collected for each dataset is shown in <Ref>.

Expressive        Video
Alignments       Commissioned      cTTS

Silence Removal

Vocal Style Conversion

Data pre-processing. pre-processing steps applied to each dataset.

In order to perform denoising we used the publicly available Demucs tool[<https://github.com/facebookresearch/demucs>] [Rouard et al., 2023]. Leading and trailing silences were removed using Silero voice activity detection [Silero, 2021], and ASR was run using Whisper[model] [Radford et al., 2022]. Denoising and silence removal were applied in sequence (i.e. once data was denoised, the outputs were fed as input to the silence removal step). Additionally  transcription, where applicable, was performed following silence removal since it is possible in rare cases that some verbal utterances may not be recognized by voice activity detection.

Expressive        Video
Alignments       Commissioned      cTTS

French     4,376     1,784     299     0     1,515

German     2,122     1,865     122     17     1,503

Italian     1,118     1,891     68     18     1,614

Mandarin     116     1,694     286     14     1,402

Spanish     4,242     1,841     237     31     1,637

Total     11,974     9,075     1,012     80     7,671

Data statistics. Number of source hours per dataset.

Vocal style conversion with .

The lack of speaker and prosody-aligned data is one challenge of translation modeling with expressivity. We created such aligned data with a unit-based Voicebox. Voicebox is a flow-matching model supporting preserving the style of one's voice via text-to-speech synthesis [Le et al., 2023]. It takes prompt audio and phoneme sequence as input and then generates speech output with the speaking style of the prompt and semantics of the phonemes. We propose , adapted from the phoneme-based Voicebox framework with the following changes:

* Speech units. To remove the reliance on texts and phonemes, we replaced phonemes with discrete units as semantic representations of speech. These speech units have been used by , which are speech representations from XLS-R quantized with k-means clustering.

* Language embedding. As we aimed to enable multilingual speech synthesis, integrating language information helps the model distinguish languages and generate more natural-sounding speech in various languages. We do this via language embedding. Specifically, each language was assigned a set of learnable embeddings, and the language embeddings were concatenated with the unit embedding when the semantic units were vectorized.

was first pre-trained on large-scale multilingual speech corpora (c.f. <Ref>) to learn prompt-based natural-sounding speech synthesis. To enable the model to better capture speaking styles, we further finetuned it on high-quality emotional speech.

The trained was applied to mExpresso, expressive, and video alignments to boost their prosody matching by explicitly converting paired utterances to the same vocal style.
As for cTTS data aligned in pauses and speed but lacking in emotions, we applied to enhance emotional strength and vocal style variation in speech. We prepared multilingual emotional data as the style prompt and multi-speaker speech as the speaker prompt. takes either style or speaker prompt and transfers its voice style to the aligned speech in the cTTS data. Some heuristics were adopted below to optimize the synthesized speech quality when pairing prompt with cTTS data:

* Speech rate matching. We measured the speech rate of prompt and cTTS speech by the number of syllables per second, and they are paired when the speech rate difference is no larger than one.

* Duration matching. As demonstrated in recent studies [Borsos et al., 2023, Wang et al., 2023], it is harder to transfer the prompt's voice style to a long speech, and thus, a longer prompt is needed to provide more acoustic information and improve the transfer quality. A prompt is paired with cTTS speech when it is no less than one-third of the cTTS duration.

* Mixing style and speaker prompts. Style prompts from multilingual emotional data are usually limited in quantity and further constrained by the number of speakers. To balance the style and speaker diversity in augmented speech, we set the ratio between style and speaker prompt to $0.8$ (i.e., $80\%$ of cTTS speech were paired with style prompts, and the rest were paired with multi-speaker prompts).

§.§ Expressive Modeling

Overview of . Model architecture with two main-submodules, and .

The proposed translation system, as illustrated in <Ref>, incorporates  in both the translation model and speech generator of the 's design. This allows us to maintain high semantic translation quality given by the backbone system. In other words, we propose a cascaded expressive modeling pipeline that consists of two main sub-modules: (1) , which is a prosody-aware speech-to-unit translation model based on architecture, and (2) , a novel textless acoustic model featuring cross-lingual expressivity preservation during unit-to-speech generation.

and are designed to complement each other in transferring the expressiveness of source language speech.
That means, aims to transfer phrase-level prosody such as speech rate or pauses, while transfers utterance-level expressivity like the style of one's voice.
From the following sections, we first introduce and architectures and how they can transfer these expressivity aspects from source language to target language speech.
Then, we discuss how both components are used together to give rise to expressive speech-to-speech translation.

§.§.§ : Expressive unit-to-speech generator

pre-training. Illustration showing the three loss terms of <Ref> used to optimize 's parameters.

For the high-quality cross-lingual expressivity transfer, we propose a Paralinguistic REpresentation-based TextleSS acoustic modEL, or . can efficiently disentangle semantic and expressivity components from source language speech through unsupervised speech reconstruction pretraining.
The overall architecture of and its pretraining process are illustrated in <Ref>.
More specifically, was pretrained to reconstruct 80-dimensional Mel-filterbank features of input speech from the deduplicated (or reduced) XLS-R units with 10k k-means clustering and the same Mel-filterbank features.
After pretraining, the model is capable of expressivity-preserving Mel-filterbank generation by taking a unit sequence in the target language and Mel-filterbank features in source speech as the expressive prompt.
Lastly, the HiFi-GAN vocoder [Kong et al., 2020] synthesizes speech waveform from the Mel-filterbank features.

The proposed is composed of that extracts vector from the source language speech and the textless acoustic model that generates the Mel-filterbank features from and the XLS-R units.
We detail each module in the following subsections.

The extracts a 512-dimensional vector containing high-level paralinguistic representations from the input 80-dimensional Mel-filterbank features.
As a backbone network, we adopted a modified version of the ECAPA-TDNN architecture. The choice of the model architecture was motivated by its high performance in extracting speech's acoustic representation [Desplanques et al., 2020].
In our model, we replaced the batch normalization layer [Ioffe and Szegedy, 2015] with layer normalization [Ba et al., 2016] for consistent performance at different batch sizes.
Once is extracted, we normalized it with the L2 norm to make the training process more stable.

Note that the XLS-R units mainly contain linguistic information of speech [Seamless Communication et al., 2023].
Thus, as mentioned by Skerry-Ryan et al., 2018, the information that learns becomes paralinguistic information, including prosody information and other acoustic properties such as the style of one's voice.
As a result, when it comes to expressive S2ST, both prosody and vocal style characteristics can be efficiently transferred to output speech by extracting from source language speech.

Textless acoustic model.

The textless acoustic model predicts a Mel-filterbank features of output speech for given and XLS-R units.
We adopted a non-autoregressive (NAR) Transformer model derived from FastSpeech2 [Ren et al., 2021], where the XLS-R units are used as a linguistic input instead of text or phoneme sequence.

Specifically, the content encoder first extracts a high-level context representation from XLS-R units using a positional encoding layer followed by feed-forward Transformer (FFT) blocks.
Then, predicts unit-level local prosody features defined by F0 and energy contours and adds them to the context encoder output.
The Gaussian upsampler proposed by Shen et al., 2020 is adopted to upsample the unit-level hidden sequence to the frame-level time scale using unit duration.
Mel-decoder then generates the Mel-filterbank features of output speech from an upsampled hidden sequence using a positional encoding layer followed by FFT blocks.
To predict more naturally generated Mel-filterbank features, we used PostNet, proposed by Shen et al., 2018, to compensate for the residual signal that the decoder's FFT blocks could not capture.

Even though the overall architecture is similar to FastSpeech2, there are clear differences.
First, we actively used and language embedding vectors during the Mel-filterbank generation process for effective language-dependent expressivity transfer.
In particular, inspired by Zaïdi et al., 2022, we applied FiLM conditioning layer [Perez et al., 2018, Oreshkin et al., 2018] for conditioning prosody and language embeddings to every FFT block output and local prosody predictor as:

\begin{align}\label{eq:expressive:film}
\nonumber \text{film}(h,c) &= (\gamma+1) \cdot h + \beta, \\
\gamma &= f_1(c) \cdot \theta_{\gamma},\\
\nonumber \beta &= f_2(c) \cdot \theta_{\beta},
\end{align}

where $h,c$ are respectively the hidden representation and the corresponding conditional embedding; $f_1,f_2$ are linear projections and $\theta_{\gamma}, \theta_{\beta}$ are learnable scalar parameters.
The intuition behind this parameterized layer is to adjust the conditional embedding vector given its value relative to the hidden representation at every time step.

Second, instead of predicting the unit duration with the internal local prosody predictor [Ren et al., 2021], we used predicted duration from for its better unit sequence modeling.
For a given unit duration obtained by , we simply converted it to Mel-duration by scaling it by a factor of two, which is the ratio between intervals of Mel-filterbank features (i.e., 10 ms) and unit extraction (i.e., 20 ms).
Then, we used a Gaussian upsampler [Shen et al., 2020] to upsample unit-scale features to match the Mel-scale features.
Unlike the original Gaussian upsampler that predicts the standard deviation of the Gaussian component, we set it to a constant value $T^2=10$ for simplicity.
As mentioned in Shen et al., 2020, this is more similar to the single Gaussian soft monotonic attention compared to the length regulator of FastSpeech2 mimicking hard monotonic attention.
Thus, the upsampled hidden features show continuous transition at the unit boundary, which is more efficient for representing continuously varying target Mel-filterbank features.

We also propose to split the binary voiced/unvoiced (VUV) flag from the F0 contour and separately predict them during the local prosody prediction process.
After predicting continuous F0 contour and binary VUV flag, we combine them by simply masking F0 values at unvoiced regions to zero.
By externally imposing VUV properties during the local prosody encoding process, the model can more distinctively represent VUV properties in its output Mel-filterbank features.

Unsupervised pretraining.
During pretraining, and textless acoustic model are jointly trained to minimize three loss terms:

\begin{align}
\mathcal{L}_{total} = \mathcal{L}_{mel} + \lambda_{l} \cdot \mathcal{L}_{local} + \lambda_{f} \cdot \mathcal{L}_{film},\label{eq:expressive:ptv}
\end{align}

where $\mathcal{L}_{mel}$, $\mathcal{L}_{local}$, and $\mathcal{L}_{film}$ denote Mel-filterbank prediction loss, local prosody prediction loss, and L2 regularization loss at the FiLM layer, respectively; $\lambda_{v}$ and $\lambda_{v}$ denote weight terms for $\mathcal{L}_{local}$ and $\mathcal{L}_{film}$ that are set to be $1.0$ and $0.001$, respectively.
Each term is formulated as follows:

\begin{align}
\mathcal{L}_{mel} &= \mathcal{L}_1(\hat{{y}}_{before}, {y}) + \mathcal{L}_2(\hat{{y}}_{before}, {y}) + \mathcal{L}_1(\hat{{y}}_{after}, {y}) + \mathcal{L}_2(\hat{{y}}_{after}, {y}), \\
\mathcal{L}_{local} &= \mathcal{L}_2(\hat{{p}}, {p}) + \text{BCE}(\hat{{u}}, {u}) + \mathcal{L}_2(\hat{{e}}, {e}), \\
\mathcal{L}_{film} &= \sum_{\theta_{\gamma},\theta_{\beta}}\left(\theta_{\gamma}^2 + \theta_{\beta}^2\right),
\end{align}

where $\mathcal{L}_1(\cdot, \cdot)$, $\mathcal{L}_2(\cdot, \cdot)$ and $\text{BCE}(\cdot, \cdot)$ denote L1, L2, and binary cross-entropy losses, respectively; $\hat{{y}}$ and ${y}$ denote predicted and reference Mel-filterbank features, respectively; $before$ and $after$ subscripts imply the one before and after PostNet, respectively; ${p}$, ${e}$, and ${u}$ denote log-scale pitch contour where its unvoiced region is linearly interpolated, log-scale energy, and VUV flag, respectively.
We involve $\mathcal{L}_{film}$ for better generalization performance of the textless acoustic model as reported in Oreshkin et al., 2018.
Note that each loss term does not require any text transcriptions or human data annotations, and it is easily scalable to large-scale data and more language directions.

Relationship to prior works.
The key difference of -based expressive S2ST system compared to is that it utilizes source language's during the speech generation process.
The unit-based HiFi-GAN for reconstructs speech waveform from XLSR-R units and language embedding without any connection to expressivity information.
Thus, it tends to produce monotone speech in terms of vocal style and prosody.
In contrast, the proposed overcomes this limitation by explicitly conditioning source language's .

A work similar to is the Prosody2Vec framework [Qu et al., 2022].
It similarly proposed a textless acoustic model based on Tacotron2 [Shen et al., 2018] by replacing phoneme input with HuBERT units [Hsu et al., 2021].
The major differences between our effort and Prosody2vec are: (1) we adopted NAR FastSpeech2-based acoustic model for fast inference, (2) we integrated this model into expressive S2ST system, and (3) we adopted XLS-R units for the easy language scalability.

On the other hand, the PolyVoice [Dong et al., 2023] also adopted a similar cascaded approach for expressive S2ST by incorporating cascaded language models (LMs) at speech-to-unit translation and unit-to-speech generation modules.
In particular, it uses hybrid AR and NAR LM to generate SoundStream units [Zeghidour et al., 2022] at target language from the target language's HuBERT units and the source language's SoundStream units.
Despite its high-quality expressive translation, has strong benefits considering hybrid AR and NAR LM's lower inference speed.
Even though described in <Ref> can be a good alternative as an NAR unit-to-speech generator, it still lags inference speed compared with because its architecture requires heavier model size.
More analysis is included in <Ref>.

§.§.§ : Expressive speech-to-unit translation model

. Illustration of modules building on .

For the expressive speech-to-unit translation, we propose a , which incorporates 's during the unit generation process. As illustrated in <Ref>, the proposed is based on UnitY2's architecture that takes Mel-filterbank features of source language speech as input, and outputs 20-ms interval XLS-R 10K units of target language speech.

Prosody-aware NAR T2U architecture.

To better transfer expressivity information of source speech during the unit generation process, injects extracted by from the source speech into various positions of NAR T2U component: (1) adding to the output of subword-length T2U encoder, and (2) conditioning the unit duration predictor and NAR unit decoder on the transformed by a FiLM layer, as described in <Ref>.

Components in , such as the duration predictor and NAR unit decoder, are capable of word/phrase-level prosody modeling concerning pauses and speech rate. However, it remains a non-trivial task for to preserve them because its T2U component does not predict units explicitly conditioned on acoustic information from source speech.
The proposed prosody-aware T2U component can effectively address this limitation by integrating into unit prediction.

Fine-tuning pretrained components.
We used the model described in <Ref> as the foundation model and deployed it to initialize the speech encoder and the first-pass text decoder of .
The MAdaptor layer of the component of is initialized using the GeLU activation function (where the original model uses ReLU), allowing the model to avoid quick overfitting during finetuning. In contrast to the training design of <Ref>, we finetuned the parameters of the component so that the text decoder is able to learn pause labels located in our text training data.

In addition, for efficient prosody-aware training, we initialized from the parameters of the pretrained .
Then, all model parameters, including and , were jointly trained to optimize conventional training criteria, as explained in <Ref>.

§.§.§ Expressive S2ST with

As illustrated in <Ref>, we separately trained and and constructed the proposed by cascading the two components.
During inference, translates source speech into XLS-R units of target language conditioned by .
Given unit duration and reduced units, the textless acoustic model of synthesizes the Mel-filterbank features of target language speech from XLS-R units.
Specifically, the concatenated vector of the source language's and the target language's language embedding are used as the conditional vector of the acoustic model.
Finally, the HiFi-GAN vocoder [Kong et al., 2020] converts Mel-filterbank features into the speech waveform.

§.§ Experimental Setup

Set     cmn           deu           eng           fra           ita       spa
Training     12,771     1,965      44,571     1,073      247      915

Validation        20         14        16         10         5       10

data statistics. Duration in hours of pretraining datasets per language.

To train , we extracted 80-dimensional Mel-filterbank features and 10K XLS-R units using the same methods as .
Then, we applied zero-mean and unit-variance normalization to input and output Mel-filterbank features to stabilize model training.

To extract F0 and VUV flag, we first extracted F0 in every 5 ms by using DIO algorithm [Morise et al., 2016].
Then, we obtained VUV flag specifying non-zero values of F0, while obtaining continuous F0 contour by linearly interpolating zero values.
To extract energy, we extracted energy contour every 5 ms using a 35-ms Hanning window.
Using the duration of unit, all F0, VUV flag, and energy features were averaged to have a reduced unit-scale.
Finally, we converted linear F0 and energy values into log-scale.

At the pretraining stage, we collected several multilingual datasets covering six languages described at <Ref>.
We summarize the data statistics in <Ref>.
To alleviate language imbalance in training data, we applied temperature-based resampling with the temperature set to 5.
For better generalization of , SpecAugment [Park et al., 2019] with frequency mask with a maximum width of 8 and time mask with a maximum width of 10 was applied on the fly during training.
The model was trained by $500$k iteration using 16 V100 GPUs with a learning rate of $10^{-4}$.

We randomly sampled 10k utterances from each language of training data.
and trained a HiFi-GAN model [Kong et al., 2020] for one million iterations with a fixed batch size of 16, batch length of 8,960, and 8 V100 GPUs.
We modified the original HiFi-GAN's upsampling scales from [8, 8, 2, 2] to [5, 4, 4, 2] because the Mel-filterbank features's 10-ms frame interval equals 160 samples.
We further fine-tuned the HiFi-GAN vocoder by using the output of the for 400k iterations to prevent the quality degradation from over-smoothing problem [Bishop, 1994].
Specifically, we generated over-smoothed Mel-filterbank features by inference under teacher-forcing mode to prepare a pair of Mel-filterbank features and waveform for fine-tuning.
All other training settings follow the original HiFi-GAN training setup [Kong et al., 2020].

is pre-trained on the data in <Ref> as . We include a baseline S2ST system with as the acoustic model (see Model 4 in <Ref>) to compare with in speech synthesis. For data preprocessing introduced in <Ref>, we further finetune on emotional data to synthesize diverse speech. We describe more pre-training and finetuning details in the Appendix.

Expressive        Video
Alignments       Commissioned      cTTS

French     1,949     646     299     0     1,515

German     994     662     122     17     1,503

Italian     286     688     68     18     1,614

Mandarin     110     42     286     14     1,402

Spanish     1,729     866     237     31     1,637

Total     5,068     2,904     1,012     80     7,671

data statistics. Number of source speech hours in training data per data source and language direction.

. We combined data from all training data sources and language directions described in <Ref>.
Given the large amount of data,
we set thresholds and filtered the high-quality data:
for data, we selected source-target pairs with scores greater or equal to 3.2 and cosine scores greater or equal to 0.75 for all directions. The amount of other data sources used for training is the same as reported in <Ref>.
With training data in $10$ directions between English and five languages, we trained a single multilingual speech-to-unit translation model for conducting objective evaluation with automatic metrics and subjective human evaluation (<Ref>).
For the ablation study, we trained five bidirectional models for each language pair (and ), one eng-to-many (E2M) model, and one many-to-eng (M2E) model. Those models help us understand how multilingual training affects translation performance.

All models were trained with a learning rate of $5\times10^{-5}$.
Bidirectional and M2E models were trained in a distributed setup with an effective batch size of 302k tokens; E2M and M2M have a larger batch size of $360$k tokens. We set a maximum of $600$k training steps and adopted an early stop policy such that model training is stopped when validation loss has not been improved for the last $5$ runs. The checkpoint with the best validation loss is used for evaluation.

Automatic metrics. We adopted the following automatic metrics to measure both the semantics and the expressive aspects of .

* . Our expressive translation systems must maintain high-quality content translation. We do model selection and analysis using average for and for (see <Ref>).
This pair of scores allows us to see the final model performance and how much the translation quality alters between text and its audio realization.

* Vocal style similarity (VSim). Embeddings are extracted from generated and source speech using a pretrained WavLM-based encoder [Chen et al., 2022]. We measured the cosine similarity of the embeddings (as in Voicebox [Le et al., 2023]).

* score.
For sentence-level prosody transfer, we relied on the model trained on multilingual data (see <Ref> for more details).

* Rhythm: speech rate Spearman correlation (Rate) and joint pause alignment score (Pause). Speech rate is estimated from source and generated speech as described in <Ref>, and we report Spearman correlation between the rates of the source and the target. Pause preservation was measured using the weighted mean joint pause alignment score from the rhythm evaluation toolkit described in <Ref>.

§.§ Results and Discussion

In this section, we trained models for expressive speech-to-speech translation. A card for these models is available in <Ref>. We also perform empirical evaluation on models together with different baselines, and $10$ translation directions are included:
English$\leftrightarrow${French (fra), German (deu), Italian (ita), Mandarin (cmn), Spanish (spa)}.

Models. <Ref> summarizes the model list for expressive S2ST evaluation. Given current progress in vocal style-preserved text-to-speech such as [<https://huggingface.co/coqui/XTTS-v2>], we included a strong cascaded baseline, Model 1, which consists of speech-to-text modules in and the text-to-speech model .

Our proposed model is Model 5, trained with 5-eng and eng-5 translation data. A natural baseline is Model 2, , which serves as our foundational model. To analyze the performance of , Model 3, which replaced the unit HiFi-GAN in with , is included as another baseline to measure the effectiveness of when compared against Model 5 and evaluates in comparison with Model 2.

As is able to convert units to speech and is also used for training data augmentation, we introduced Model 4 to compare with in speech generation. For a fair comparison with , the pretrained checkpoint of is used in Model 4, and the finetuned is only for the purpose of data augmentation.

ID      Model      Speech-to-Unit/Text      Unit/Text-to-Speech
1     +

2              Unit HiFi-GAN

3      -

4      -

5 (proposed)

List of models for expressive . Model 1 uses text outputs from to synthesize speech, and Model 2-5 connect speech-to-unit and unit-to-speech components with predicted units.

Evaluation data. We prepared three evaluation sets of expressive speech-to-speech translation in mExpresso, mDRAL (<Ref>)[We will release the benchmark sets.] and FLEURS. The data statistics of dev and test splits are reported in <Ref>.

2cmExpresso     2cmDRAL     2cFLEURS
Dev     Test     Dev     Test     Dev     Test
Sample #                  25520      33694      2715     2293     3784     7438

Hours                      28.96     39.67       5.42     5.16     11.31      23.50

Total # Speakers          11        12          53     55     -     -

Total # Male Speakers     5         6           18        19     -      -

Descriptive statistics of evaluation data. Statistics of the development (dev) and test splits of mExpresso, mDRAL and . Note that we do not have speaker information for so these rows are left empty. Since the mExpresso dataset is pivoted out of English, to avoid double counting English volumes we only include the unique English samples in these descriptive statistics (along with the other languages).   <Ref> provides these descriptive statistics for each language pair.

Model     $\uparrow$     $\uparrow$     $\uparrow$Rate

$(n=5)$      1     31.95     2.83     0.35

2     34.47     2.16     0.08

3     34.27     2.76     0.27

4     39.27     3.14     0.64

5     39.00     3.18     0.63

$(n=5)$       1     28.69     2.87     0.39

2     30.35     2.44     0.09

3     30.07     2.93     0.29

4     34.38     3.17     0.65

5     34.21     3.11     0.65

Result on mExpresso test data. Models are compared on , and Rate metrics, and we exclude VSim and Pause as mExpresso is acted short speech with only one or two speakers per set.

Model     $\uparrow$     $\uparrow$VSim     $\uparrow$     $\uparrow$Rate     $\uparrow$Pause

$(n=5)$      1     36.69     0.33     2.78     0.24     0.26

2     38.82     0.05     2.31     0.13     0.14

3     38.59     0.27     2.87     0.15     0.16

4     40.13     0.36     3.13     0.63     0.39

5     40.18     0.28     3.19     0.64     0.39

$(n=5)$       1     23.63     0.39     2.61     0.21     0.21

2     25.32     0.06     2.36     0.06     0.14

3     24.75     0.33     2.76     0.09     0.14

4     34.13     0.40     2.92     0.58     0.35

5     33.82     0.33     2.92     0.59     0.36

Result on mDRAL test data. All evaluation metrics are reported as mDRAL is spontaneous expressive speech which is useful for comparing different aspects of prosody.

Model     [c]
$(n=5)$     [c]

1     31.34     16.59

2     31.99     31.78

3     31.81     29.28

4     30.58     31.58

5     30.47     29.32

Results on FLEURS test data. Model are compared on only as FLEURS is commonly used to evaluate content preservation in translation.

Metric comparison. We empirically evaluated models using both dev and test splits of multiple datasets. Covering multiple evaluation metrics, average results[Watermarking is applied to released models, so they might have slight difference with reported results in the paper.] are reported over five and five directions, respectively. <Ref> reports test results on mExpresso, <Ref> on mDRAL, and <Ref> on FLEURS. These datasets represent different aspects of systems' performance we focus on. Both mDRAL and mExpresso involve explicit expressivity alignment, thus allowing us to monitor score for sentence-level prosody alignment and speech rate, and mDRAL contains spontaneous speech with natural pauses for us to evaluate the quality of pause preservation. FLEURS is mostly used to examine how the model is maintaining content translation quality from .

More detailed results in each language direction can be found in the appendix, where <Ref> reports results on mExpresso, <Ref> on mDRAL, and <Ref> on FLEURS data. In this section,
we focus on comparing the models' performance on test sets because all conclusions we make also hold on development sets.

Expressive speech-to-unit translation.
We compare and by looking at Models 5 and 3, which differ in the speech-to-unit module. Model 5 demonstrates competitive . Averaged over five directions, the gains of Model 5 on test data are +$4.73$ BLEU in mExpresso and +$1.59$ BLEU in mDRAL while falling behind Model 3 by $1.34$ BLEU in FLEURS. The average gains in five directions are +$4.14$ and +$9.07$ BLEU on mExpresso and mDRAL test sets, respectively. Both models have comparable in FLEURS data.

Expressive speech generator.The comparison of Model 3 against Model 2 reflects the effectiveness of . Models 3 and 2 are close in terms of rate and pause metrics across three datasets, as these aspects are mainly controlled by the speech-to-unit translation model.
makes good improvements in preserving prosody and the style of one's voice in generated speech. Given mDRAL test data in <Ref>, yields +$0.22$ vocal style similarity and +$0.56$ in translations, +$0.27$ vocal style similarity and +$0.4$ in translations. Both models have similar , indicating that the preserved expressivity did not severely degrade the intelligibility of the output audio.

Language-specific performance. Evaluation results of speech-to-speech translation systems with the focus on newly proposed automatic metrics and measured on mDRAL set.

Combining into . Combining the strengths of and , (Model 5) demonstrates not only improved content translation but also better preservation of rhythm, tone, and the style of one's voice over the baseline (Model 2).
<Ref> further focuses on prosody-related metrics (Pause, Rate, ) and breaks down the performance of each language measured on mDRAL test data, where we see consistent improvement across different language directions.

Alternative expressive speech generator We examine using as an alternative option for expressive speech generation. We note that , with $329$M parameters, is a much larger model than with $65$M parameters. Furthermore, is faster in speech synthesis: the real-time factor (RTF) of is $0.014$ and $0.089$ for . Given the 6x lower RTF, is a better fit as a speech generator considering the model size and inference efficiency when we equip with streaming capability in <Ref>. Models 5 and 4 show that the speech generator choice mainly results in a trade-off in vocal style similarity (with having higher vocal style similarity), while the performance in is similar across different language pairs (<Ref>).

Expressive +cascade We observe that the strong cross-lingual vocal styles and style-preserving system is sensitive to the noise in the source speech, resulting in a much lower than , which is the most fundamental metric of an system. The system is better in preserving the source speakers' vocal styles, while outperforms in prosody preservation with consistent gains on , speech rate, and pause metrics across language directions (<Ref>).

§.§ Ablation Study

We conducted ablation experiments to study the effectiveness of different training data and modeling choices.
We focused on mDRAL for the ablation studies as it allows us to examine all the semantic and prosodic metrics meaningfully.

§.§.§ Data ablation

We validated the effectiveness of all data sources listed in <Ref> on by ablating each of them from training with a leave-one-out approach, with results presented in <Ref>.
We see the major effect of the data sources is on . A large amount of parallel training data helps achieve high content translation quality, while the prosody preservation performance can be maintained due to prosody-aligned parallel datasets.

$\uparrow$     $\uparrow$VSim     $\uparrow$     $\uparrow$Rate     $\uparrow$Pause
All data             54.35     0.27     3.28     0.64     0.63

w/o Commissioned     53.09     0.26     3.25     0.64     0.64

w/o Video Alignments     53.11     0.27     3.26     0.64     0.61

w/o cTTS             52.60     0.26     3.27     0.60     0.59

w/o Expressive           54.30     0.27     3.25     0.61     0.59

w/o Expressive Alignments           53.54     0.27     3.28     0.66     0.63

Data source ablation. Results on unidirectional spa-eng mDRAL test set.

§.§.§ Semantic and prosodic data filtering

The quality of collected expressive data is variable.
Some segments may contain different semantic content, harmful for translation, while others may evince similar content but with different prosody, harmful for expressive translation.
With this idea in mind, we conducted an analysis to better understand how to characterize and filter data based on their relevance for training an expressive translation system.
For the sake of time, the ablation study described below is conducted only on video-aligned data. We chose this dataset because it is very rich (and also noisy) in terms of both semantic and prosody.

Semantic analysis.

The data is split into three equal-sized parts, one for each semantic quality, i.e., high, medium, and low semantic score.
The semantic score of a sample is a mixture of several semantic scores:

* BLASER speech similarity,

* cosine similarity between source and target text embeddings, and

* cosine similarity between source and target speech embeddings.

Model 5, , is then finetuned with each data split, all other parameters unchanged, and then evaluated (see <Ref>).

Prosody analysis: the case of speech rate.

Another important aspect of training an expressive speech translation model is the prosodic quality of the training data.
To evaluate the finetuning data's impact, we analyzed the expressive training data and trained several models to contrast the results.
For the sake of simplicity, we only considered speech rate in this study, but we evaluated all semantic and prosodic scores.

To compare the speech rates of several languages, it is important to take their relative differences into account since some languages are naturally uttered faster than others.
Thus, we first normalized the speech rate of audio files in a language by the mean for that language and calculated the relative difference between source and target normalized speech rates, by dividing by their average, as follows:

\begin{align}
\textrm{SRD}_\text{norm}(\textrm{S}_{L1}, \textrm{S}_{L2}) = \frac{\textrm{SR}_\text{norm}(\textrm{S}_{L1}) - \textrm{SR}_\text{norm}(\textrm{S}_{L2})}{\textrm{SR}_\text{norm}(\textrm{S}_{L1}) + \textrm{SR}_\text{norm}(\textrm{S}_{L2})},
\end{align}

where $\textrm{S}_{Lx}$ is a segment in language $Lx$, $\textrm{SR}_\text{norm}$ is the normalized speech rate of $\textrm{S}_{Lx}$, and $\textrm{SRD}_\text{norm}$ is the relative speech rate difference.

This measure is then used to split the data into three parts of equal size, similar to what was done for the semantic analysis.
By doing this, we hoped to train systems that better model the relative speech rate difference without considering intra-language variance.
It is worth noting that by looking at the data, we realized that the data labeled as “semantic/low” added too much noise to the statistics and that removing them before performing the prosodic analysis was beneficial.
This means that the three prosodic splits are taken from the high and medium semantic quality data samples only.

Ablation results with semantic and prosodic filtering.

Study     Quality     $\uparrow$     $\uparrow$VSim     $\uparrow$     $\uparrow$Rate     $\uparrow$Pause

$(n=5)$      3*Semantic      Low          32.15     0.26     3.09     0.62     0.25

Medium       35.40     0.27     3.12     0.64     0.20

High         40.19     0.27     3.14     0.59     0.27

3*Prosody       Low          38.90     0.27     3.10     0.50     0.16

Medium       39.61     0.27     3.18     0.62     0.45

High         39.06     0.27     3.18     0.66     0.44

$(n=5)$      3*Semantic      Low          22.56     0.31     2.84     0.51     0.13

Medium       28.19     0.32     2.88     0.56     0.11

High         33.48     0.32     2.84     0.54     0.17

3*Prosody       Low          32.57     0.32     2.83     0.49     0.10

Medium       32.96     0.33     2.88     0.57     0.33

High         33.48     0.32     2.90     0.63     0.31

Results of data filtering based on semantic and prosody alignment quality on mDRAL test sets.

<Ref> shows the results of finetuning the model with the semantic and prosodic splits of video data for and languages pairs, respectively.
Note that the results may not improve upon the baseline system (Model 5, ) since we only use the multilingual video data in this study while the baseline model was trained on much more expressive data.
The comparison of the finetuning results with the three data splits allows us to draw conclusions on the qualitative aspects of the selected data.

Let us first look at the semantic study results. The performance gain is consistent across all datasets and language pairs (see <ref> for language-level breakdown).
We clearly see that the high and medium semantic quality data leads to better scores, with an increase of up to 8% between semantic/high and semantic/low setups.
This motivated us to keep only the average and high semantic quality data splits for the prosodic study.
However, models trained with the semantic splits do not necessarily exhibit better prosodic metric scores.

Looking at the results obtained by finetuning the baseline model with prosodic splits, we observe consistent improvements in speech rate and pause evaluation, which is expected.
By selecting data according to a prosodic criterion (speech rate in our case), we observe improvements in the prosody metrics without hurting the semantic score () (and sometimes slightly improving it).
This makes sense as it tends to confirm that both aspects are correlated and that segments having the same prosody are more prone to be semantically parallel.
We can also notice that the vocal style similarity metric is not sensitive to the data refinement and remains stable in all our experiments, since it is mainly controlled by the speech generator.

The results are also consistent across language directions (and ) and show that all the considered languages can benefit from selecting higher semantic and prosodic quality data.
Those good results suggest that a better expressive model could be trained by carefully selecting the expressive data from all corpora before finetuning the model. We note that models reported in <Ref> and <Ref> have not applied such filtering to the training data due to time limit, so we leave that for future work.

§.§.§ Training ablation

$\uparrow$     $\uparrow$VSim     $\uparrow$     $\uparrow$Rate     $\uparrow$Pause
M2E     39.58     0.28     3.20     0.64     0.39

M2E-Joint     39.02     0.18     2.92     0.65     0.36

Results of jointly trained on mDRAL test sets.

Joint training.

While our main results come from the cascade of and models trained separately, we also explored joint training of the two components with the same initialization as the cascade. Joint training could mitigate
the issue of error propagation in the cascaded model, while it suffers from the constraint of requiring parallel training data fully aligned in prosody and voice style, which we tackled in data pre-processing described in <Ref>. Specifically, we directed hidden states of NAR T2U decoder to and jointly trained and to reconstruct target Mel-filterbank features conditioned on outputs from one single shared encoder.
While encoder should take input from the source speech features during inference, we found that target speech features could help the decoder improve Mel-filterbank prediction during training.
Empirically, during training, we always fed source speech to encoder for conditioning, and randomly fed target or source speech to the encoder with a probability of 80% and 20% respectively for conditioning.

As shown in <Ref>, joint training exhibits degradation in vocal style similarity and . We conjecture that our pseudo-parallel S2ST data still lags on speaker and prosody alignment compared with human speech. When is finetuned on our paired training data, its speaker and prosody preservation performance degrades.

Multilingual training. We conducted an ablation study of models trained with different language directions to quantitatively compare how multilinguality affects model performance. Specifically, the following four multilingual variants are considered:

* : bidirectional models which are trained for each language pair respectively.

* : multidirectional models trained in 5-to-eng directions.

* : multidirectional models trained in eng-to-5 directions.

* : multidirectional model trained in both 5-to-eng and eng-to-5 directions.

$\uparrow$     $\uparrow$VSim     $\uparrow$     $\uparrow$Rate     $\uparrow$Pause
$(n=5)$      Bilingual     40.18     0.27     3.18     0.63     0.36

M2E     39.58     0.27     3.19     0.63     0.38

M2M     40.17     0.27     3.18     0.63     0.38
$(n=5)$      Bilingual     33.08     0.32     2.91     0.54     0.35

E2M     32.37     0.33     2.87     0.48     0.34

M2M     33.82     0.33     2.92     0.58     0.35

Results of different multilingual models on mDRAL test sets.

<Ref> compares the translation performance of multilingual models. In translation, Bilingual, M2E and M2M have similar performances in all metrics except for .
As for translation, M2M outperforms both Bilingual and E2M in .

In this section, we present ,
the first direct simultaneous multilingual and multimodal translation model, initialized from the foundational model model (<Ref>). More specifically, builds on 's language coverage and semantic accuracy to perform direct translations from speech into both speech and text in real time.
Like , supports 101 source languages for speech input, 36 target languages in speech output, and 96 target languages in text output.
also support streaming ASR on 96 languages.
An overview of and its relationship with is shown as <Ref>.
All in all, the highlights of include:

* Simultaneous text decoder empowered by Efficient Monotonic Multihead Attention ()  (<Ref>),

* Fine-tuning from foundational model and streaming inference (<Ref>).

A overview of , and relationship with .

§.§ Efficient Monotonic Multihead Attention (EMMA)

§.§ Experimental Setup

§.§ Results and Discussion

In this section, we introduce ,
which combines two derivatives of , namely
—an offline translation model with comprehensive prosody preservation—and
—a multilingual streaming speech-to-speech translation model—to engender a unified system that provides real time and expressive S2ST.
To the best of our knowledge, marks the first, publicly available system of its kind, paving the way for a myriad of downstream possibilities that can help those experiencing language barriers better communicate in the wild.

Notably, maintains the same semantic accuracy and latency shown in .
In addition, captures a key expressivity transfer features from . More specifically, it focuses to preserve sentence-level expressivity, e.g., tone, emotional expression and the style of one's voice rather than phrase-level one, e.g., speech rate and pauses.
Based on preliminary user testing, it does not appear that not offering the full suite of expressive preservation impacted the overall experience of our test subjects.

The architecture of .

§.§ Architecture

<Ref> provides an overview of the architecture for the model.
First, generates streaming text and discrete units from source speech.
, empowered by EMMA and NAR T2U model, can be quickly adapted from the large-scale trained foundational , as described in <Ref>.
Then, the generated units are fed into the expressive speech generator, to preserve the sentence-level prosody and the style of one's voice from the partial source speech.
Finally, the synthesized Mel-filterbank features from the textless acoustic model will be fed to the HiFi-GAN vocoder to produce expressive translated speech in real-time.

The base version of in supports six high-resource languages. In order to study the behavior of this unified architecture when language support is further scaled, we also trained a version of with an extended set of 36 languages and tested it with . Details on language coverage and training data statistics can be found in <Ref>.

We present two versions of —-6 and -36.
While -6 shares 's coverage for high-resource languages,
-36 extends its language coverage to that of by leveraging the language-extended model.

§.§ Results and Discussion

We only conducted automatic evaluations on the .

§.§.§ extension

We report the automatic measurement of trained with different data.
We document the offline performance of two different models,
namely -6[Note that this model is same model used for .] and -36, in <Ref>.
As evidenced by the results, English output remains of similar quality overall. The significantly expanded set of languages does, however, come at the cost of lowering performance for non-English target languages, as measured by ASR-BLEU and vocal style similarity. Overall, results demonstrate the feasibility of broad language support for , although a larger model might be required for increased capacity.

Language Coverage     6     36     6     36     6     36

[c]ASR-BLEU     25.8      25.5     29.3      28.3     –     22.78

[c]Vocal Style Similarity      0.30      0.30     0.24      0.19      –     0.18

[c]      2.35      2.46     2.80     2.88     –     2.86

The automatic measurement on of the models trained on six and 36 languages. The middle column compares results for the direction limited to the five non-English languages supported by .

§.§.§ Quality-latency

Similar to ,
we report translation quality and latency under different decision thresholds $t_{\emma}$ [For details, see <Ref>]
in <Ref> and <Ref>.
We can see that the model has a small degradation from on ASR-BLEU in both and directions.
has a lower ending offset, which means the model can generate speech with a shorter duration.
-6 has better performance for 's six target language directions than -36, while -36 has the same coverage as .
The full results and metrics per evaluation direction can be found here: <https://github.com/facebookresearch/seamless_communication>.

4*Model        4*[c]Decision
Threshold      2c[c]

ASR-BLEU     [c]Ending

0.4     21.8     2.66

0.5     22.1     2.79

0.6     22.0     2.81

0.7     22.0     2.91

0.4     20.3     1.95

0.5     20.4     2.02

0.6     20.5     2.09

0.7     20.7     2.15

0.4     17.8     2.00

0.5     18.1     2.09

0.6     18.3     2.17

0.7     18.6     2.26

Average translation quality and latency of under different latency decision threshold $t_{\emma}$ on , to English directions.

4*Model        4*[c]Decision
Threshold      2c[c]
(n=5)     2c[c]

ASR-BLEU     [c]Ending
Offset     ASR-BLEU     [c]Ending

0.4     27.7     4.11     20.8     4.59

0.5     27.8     4.15     21.4     4.64

0.6     27.9     4.20     21.5     4.69

0.7     28.0     4.25     21.6     4.73

0.4     25.6     2.77     —     —

0.5     25.6     2.83      —     —

0.6     25.7     2.88     —     —

0.7     25.8     2.95     —     —

0.4     23.4     2.81      16.1     3.38

0.5     23.4     2.87     16.2     3.43

0.6     23.6     2.93     16.1     3.49

0.7     23.7     2.99     16.3     3.55

Average translation quality and latency of under different latency decision threshold $t_{\emma}$ on . The column with $(n=5)$ compares the results for the eng–X direction limited to
the 5 non-English languages supported by .

§.§.§ Expressivity preservation

We present the expressivity preservation at different latency in <Ref> and <Ref>.
Comparing with <Ref>,
there is a degration in both vocal style similarity and due to the partial context used in .

4*Model        4*[c]Decision
Threshold      2c[c]

[c]Vocal Style

0.4     0.21     1.89     1.95

0.5     0.21     1.89     2.02

0.6     0.21     1.89     2.09

0.7     0.21     1.90     2.15

0.4     0.22     1.76     2.01

0.5     0.23     1.76     2.10

0.6     0.23     1.77     2.17

0.7     0.23     1.78     2.26

Average expressivity preservation and latency measurements of under different latency decision threshold $t_{\emma}$ on , to English Directions.

Threshold      3c[c]
(n=5)     3c[c]

[c]Vocal Style
[c]Vocal Style

0.4     0.18     2.48     2.77     —     —     —

0.5     0.18     2.49     2.83     —     —     —

0.6     0.18     2.49     2.88     —     —     —

0.7     0.18     2.49     2.95     —     —     —

0.4     0.19     2.38     2.81    0.19     2.36     3.38

0.5     0.19     2.39     2.87       0.19     2.36     3.43

0.6     0.19     2.39     2.93     0.19     2.37     3.49

0.7     0.19     2.39     2.99     0.19     2.37     3.55

Average expressivity preservation and latency measurements of under different latency decision threshold $t_{\emma}$ on .
The column with $(n=5)$ compares the results for the eng–X direction limited to
the 5 non-English languages supported by .

§ AUTOMATIC AND HUMAN EVALUATION

To properly evaluate our models we relied on a combination of existing and novel metrics which are compiled in the newly proposed concept of metric card <Ref>. In this section, we specifically detailed the novel contributions in automatic expressivity metrics, followed by presenting results of our models in terms of robustness and several human evaluation protocols.

§.§ Automatic Expressivity Metrics

To ensure the quality of , we relied on measures that evaluate the prosodic consistency of source and target speech.

Early work on comparing prosody across languages [Cummins et al., 1999] used an LSTM-based ([Hochreiter and Schmidhuber, 1997]) model with features based on $F_0$ contour and amplitude envelope.
In [Ward et al., 2023, Avila and Ward, 2023], the authors created an English-Spanish corpus (DRAL) and provided an analysis of the prosodic relation between a pair of audios by calculating the Spearman correlation of 100 features (e.g., intensity, speaking rate, pitch) and provided a simple metric corresponding to the Euclidean distance between the two prosodic representations.

We contribute two types of automatic measures of prosodic preservation in speech translation: 1) to evaluate prosody at the sentence level and 2) a rhythm evaluation toolkit.

Our main measure of prosodic preservation, PCP (Huang et al., 2023; see also <Ref>), corresponds to human judgments (using a 4-point Likert scale) of how similarly two spoken utterances sound in prosody. is a neural model trained to predict PCP scores of “sentence-level prosody similarity”. This model has an architecture similar to BLASER [Chen et al., 2023]: embedding vectors of two audios (in our case, obtained by pooling embeddings from the 9th layer of an XLS-R model [Conneau et al., 2020]) are passed into a small fully-connected neural network that predicts the target score.

We trained with two tasks: supervised regression and an unsupervised contrastive task. For regression, we annotated with simplified PCP in which annotators are asked about a single expressive dimension “Overall Manner,” analogous to the dimension “Overall Expressive Intent“ described in <Ref>. In total, we collected nearly 800 sentence pairs for each translation direction (from French, Italian, German, Mandarin, and Spanish to English).
To ensure that the dataset contains audio pairs with diverse prosodic similarity degrees, we compiled it from several sources:

* Audio pairs from multilingual videos (<Ref>), with naturally diverse quality;

* M4T training data (<Ref>), with either original or re-synthesized target speech;

* Audio pairs synthesized using cTTS, with both matching and mismatching speech rate and pauses (following <Ref>);

* mExpresso audio pairs (<Ref>) with matching and mismatching styles.

As a source of contrastive data, we used the parallel corpus from multilingual videos (<Ref>); the model is trained to predict higher scores for positive examples (the original audio pairs) than for hard negative examples (re-combined audio pairs with similar semantic embeddings of their transcriptions).
We return to the discussion of human- and automatic-metric correlation under our human evaluation test sets in <Ref>.

We evaluated the resulting model with three metrics on a test set annotated with PCP: item-level and system-level Spearman correlations and RMSE. As <Ref> shows, the model performs robustly across languages, and its results are comparable to human-level, computed as judgments of a single randomly chosen annotator compared to the median score of the other annotators. We also validate the model by comparing it to a simple baseline: cosine similarity of the same XLS-R speech embeddings, rescaled to minimize RMSE on the test set. The model demonstrates much better item- and system-level correlation with the target than the baseline.

The model card for is given in <Ref>.

6c     baseline    human
(r)2-7(l)8-8 (l)9-9
System     deu     spa     fra     ita     cmn     avg    avg     avg

Item-level correlation $\uparrow$     0.58     0.50     0.44     0.50     0.44     0.49     0.31     0.46

System-level correlation $\uparrow$     0.60     0.58     0.60     0.77     0.72     0.65     0.28     0.84

RMSE $\downarrow$     0.68     0.61     0.88     0.85     0.90     0.79     0.81     0.97

Evaluation of the model. To estimate human-level performance, for each sample, we randomly selected the label from one randomly chosen annotator and compared it to the median label from the other annotators. The same target is used to estimate model performance. By “system” here, we denote a combination of the data source and the method to obtain the target audio.

Rhythm evaluation toolkit.

To complement , a blackbox predictor of overall prosodic similarity, we designed tools for quantifying and comparing several individual aspects of prosody in a more interpretable way. More specifically, we focused on evaluating rhythm as realized in speech rate and pauses.

* Speech rate: the number of syllables per second[Despite syllable-level speech rate showing lower correlation with human judgments on Spanish-English data than with other units of content, we chose this unit based on linguistic considerations—as the most generalizable across diverse languages.]. We obtained the syllables by running the “syllables” python package on the transcription (except for Mandarin, where we counted each character as a syllable) and dividing their number by the net duration of the audio in seconds, computed using Silero VAD [Silero, 2021].

* Pauses: we detected pauses and their durations with Silero VAD and located them between the words of the transcription using the aligner (<Ref>). To evaluate whether a pause is located correctly in the translation, we aligned the source and translation words with AwesomeAlign [Dou and Neubig, 2021], and used the proportion of word alignment edges that do not cross the edge connecting two matched pauses as the metric of pause location. For each source-translation pair, we computed the joint score as the average product of the location score of each pause and its shorter-to-longer duration ratio in the pair. To aggregate these scores over multiple sentence pairs, we computed their average weighted by total pause duration in each pair.

We evaluated these evaluation metrics by computing their correlation with PCP labels on the Expresso-based subset of the English-Spanish data annotated in [Huang et al., 2023], reported in <Ref>. As expected, computed speech rate similarity and pause similarity moderately correlate with overall judgments of prosody preservation.

PCP aspect     Rhythm     Overall manner

pause duration ratio     0.1802     0.1280

pause location score     0.1835     0.1322

pause joint score        0.1820     0.1294

speech rate ratio, word     0.3045     0.2273

speech rate ratio, syllable     0.2513     0.1946

speech rate ratio, character     0.4011     0.2844

speech rate ratio, phoneme     0.4107     0.2994

Spearman correlations of the rhythm metrics with human PCP labels.

These tools are used for annotating training data in <Ref> to apply prosodic filtering and selecting the most relevant samples from the expressive data that has been aligned or generated.
They are also used in <Ref> to evaluate the model's capability to produce prosody-preserving spoken translations.

§.§ Robustness Automatic Evaluation

We evaluated model robustness against background noise and vocal style variations as examples of non-linguistic perturbations in real-world speech inputs. We used benchmarks from Seamless Communication et al., 2023 and compared our models to .

Evaluation results of model robustness against background noise. We report average test and test over four languages (three language families) for  and ASR on with low-to-high input noise level (high-to-low SNR). Simulated noises are sampled from MUSAN [Snyder et al., 2015] on the “noise” and “music” categories.

Robustness against background noise. <Ref> shows the average test and test over the four languages for  and on with low-to-high input noise level (high-to-low SNR). Both -SNR curves for our new model, are consistently above those for and . Similarly, 's -SNR curves are consistently below the ones for and . These indicate the superior robustness of in noisy speaking environments. outperforms by an average of 53.2% and 50.5% over various noise types and levels for  and , respectively. It also outperforms our earlier model, by an average of 14.9% and 14.5% for  and , respectively.

2*Model     2c       2c

chrF$_{MS}$$\uparrow$     CoefVar$_{MS}$$\downarrow$     chrF$_{MS}$$\uparrow$     CoefVar$_{MS}$$\downarrow$

40.8     13.7     58.7     17.0

45.3     9.1     72.5     6.4

51.6     6.3     79.2     4.0