#  DiffNorm: Self-Supervised Normalization for Non-autoregressive Speech-to-
speech Translation

Weiting Tan Jingyu Zhang Lingfeng Shen Daniel Khashabi Philipp Koehn
Department of Computer Science
Johns Hopkins University
{wtan12, jzhan237, lshen30, danielk<EMAIL_ADDRESS>

###### Abstract

Non-autoregressive Transformers (NATs) are recently applied in direct speech-
to-speech translation systems, which convert speech across different languages
without intermediate text data. Although NATs generate high-quality outputs
and offer faster inference than autoregressive models, they tend to produce
incoherent and repetitive results due to complex data distribution (e.g.,
acoustic and linguistic variations in speech). In this work, we introduce
DiffNorm, a diffusion-based normalization strategy that simplifies data
distributions for training NAT models. After training with a self-supervised
noise estimation objective, DiffNorm constructs normalized target data by
denoising synthetically corrupted speech features. Additionally, we propose to
regularize NATs with classifier-free guidance, improving model robustness and
translation quality by randomly dropping out source information during
training. Our strategies result in a notable improvement of about $+7$ ASR-
BLEU for English-Spanish (En-Es) and $+2$ ASR-BLEU for English-French (En-Fr)
translations on the CVSS benchmark, while attaining over $14\times$ speedup
for En-Es and $5\times$ speedup for En-Fr translations compared to
autoregressive baselines.111 Code available at:
https://github.com/steventan0110/DiffNorm.

##  1 Introduction

Speech-to-speech translation (S2ST) systems are essential to bridge
communication gaps and have wide application potential. We focus on non-
autoregressive modeling for direct speech-to-speech translation, converting
source speech to the target without intermediate text data. Such direct S2ST
systems [24, 22, 21, 31, 30, 20, 19] can preserve non-linguistic information,
avoid error propagation from cascaded systems [28, 33] (e.g., a combination of
speech recognition and machine translation systems), and achieve faster
inference speed.

Non-autoregressive Transformers (NAT) [20, 19] has played a central role in
current S2ST work [30, 31, 20]. NAT translates source waveforms into target
speech units via parallel decoding, achieving performance comparable to or
better than autoregressive models while greatly reducing inference time. This
process is often referred to as speech-to-unit (S2UT) translation [30]. The
predicted speech units are then converted to target waveforms via a unit
vocoder in the unit-to-speech synthesis stage [31, 37]. However, NATs suffer
from incoherent and repetitive generations, referred to as the multi-modality
problem [11]. This issue stems from NAT’s assumption of conditional
independence during parallel decoding, worsened by the complex and multi-modal
nature of training data distribution.

In this work, we propose DiffNorm, a self-supervised speech normalization
strategy that alleviates the multi-modality problem of NAT models by
simplifying the target distribution. Instead of distilling training data from
an autoregressive model [9] or utilizing perturbed speech to train a
normalizer [31, 20], we rely on the denoising objective of Denoising Diffusion
Probabilistic Models [14, DDPM] to normalize target speech units. DiffNorm
inject synthetic noise to speech features and use diffusion model to gradually
recover the feature, obtaining a simplified and more consistent data
distribution that obscures non-crucial details. As the denoising objective is
learned in a self-supervised manner over latent speech representations, it
eliminates the necessity for transcription data [31] or manually crafted
perturbation functions [20]. As illustrated in Fig. 1, applying DiffNorm to
the target data obtains normalized speech units that lead to better NAT
training for speech-to-unit translation.

Besides using DiffNorm as a data-centric strategy to mitigate the multi-
modality problem, we also propose a regularization strategy to enhance the NAT
model’s robustness and generalizability when facing complex data distribution
(e.g., linguistic diversity and acoustic variation [20, 19]). Inspired by
classifier-free guidance [16, CG], during training, we occasionally drop out
source information and replace it with a "null" representation, compelling the
models to generate coherent units without conditioning on the source data.
During iterative parallel decoding of the NAT model, we obtain higher-quality
translation by mixing conditional and unconditional generation. Ultimately,
combining DiffNorm and CG results in our top-performing state-of-the-art
system, achieving approximately $+7$ and $+2$ ASR-BLEU increment for En-Es and
En-Fr translation compared to previous non-autoregressive systems on the CVSS
[23] benchmark.

In conclusion, we alleviate the multi-modality problem by proposing (1)
diffusion-based normalization and (2) regularization with classifier-free
guidance. To the best of our knowledge, we are the first to adapt diffusion
models and classifier-free guidance into speech-to-speech translation and NAT
modeling. Our methods obtain notable improvement compared to previous systems
and maintain fast inference speed inherent in non-autoregressive modeling,
achieving speedups of $14\times$ for En-Es and $5\times$ for En-Fr compared to
autoregressive baselines.

Figure 1: Overview of our proposed system. We first normalize the target
speech units with the denoising process from the latent diffusion model. Then
speech-to-unit (S2UT) model is trained to predict normalized units, which are
converted into waveform from an off-the-shelf unit-vocoder.

##  2 Problem formulation and overview

We aim to develop a direct (textless) speech-to-speech translation system that
transduces a source speech $\bm{x}=(x_{1},\cdots,x_{N})$ into target speech.
We follow Lee et al., 2022a [30] to reduce speech-to-speech translation into
two sub-tasks: speech-to-unit translation and unit-to-speech synthesis. In
this work, we focus on speech-to-unit translation and follow prior work [30,
31, 20] to use the same unit-to-speech component for a fair comparison. To
generate speech units for the target language, we first extract speech feature
$\bm{h}=(h_{1},\cdots,h_{M})\in\mathbb{R}^{M\times H}$, where each feature has
$H$ dimensions. Subsequently, a K-means clustering model is trained on
extracted features and used to generate speech units
$\bm{y}=(y_{1},\cdots,y_{M})\in\mathbb{R}^{M\times 1}$. Once the source speech
$\bm{x}$ and target speech unit $\bm{y}$ are prepared, we train sequence-to-
sequence models to translate from source speech into target units. In
practice, we follow Lee et al., 2022b [31] to use a multilingual-HuBERT
(mHuBERT) model to extract $H=768$ dimensional speech features $\bm{h}$. Then
we use a 1000-cluster K-means model to predict speech units given the encoded
features.222 We take the mHuBERT and K-means models off-the-shelf to produce
units consistent with prior work. See
github.com/facebookresearch/fairseq/tree/main/examples/speech_to_speech For
speech-to-unit translation, we follow Huang et al., [20] to adopt Conditional
Masked Language Modeling [9, CMLM], a kind of non-autoregressive transformer
(NAT).

To mitigate NAT models’ multi-modality problem, we propose DiffNorm (section §
3), which denoises synthetically corrupted speech features to construct
normalized speech units $\bm{y}_{\rm norm}$. As shown in Fig. 1, such
normalized units are then used to train the S2UT (CMLM) model, which generates
better-translated units for the unit vocoder to synthesize target speech.

Additionally, we propose to incorporate classifier-free guidance [16, CG], a
widely adopted strategy for diffusion-based image generation, as a
regularization method to improve the non-autoregressive speech-to-unit system
(section § 4). As shown in Fig. 3, by forcing NAT models to unmask target
speech units without conditioning on source information, the model becomes
more robust, generating coherent speech units that result in higher-quality
translations. During inference, we follow classifier-free guidance to mix
conditional and unconditional generation for the NAT model’s iterative
decoding, further enhancing its translation quality.

##  3 DiffNorm: denoising diffusion models for speech normalization

Previous normalization strategy [31, 20] on speech-to-unit translation relies
on connectionist temporal classification (CTC) fine-tuning [3] using the
HuBERT [18] model. For example, Lee et al., 2022b [31] rely on single-speaker
data from VoxPopuli [53] to produce normalized (speaker-invariant) speech
units for HuBERT-CTC fine-tuning. On the other hand, Huang et al., [20]
generate acoustic-agnostic units by perturbing rhythm, pitch, and energy
information with (manually) pre-defined transformation functions. We propose
to normalize speech units using self-supervised denoising objectives from
Denoising Diffusion Probabilistic Models [14, DDPM]. We believe DDPM is more
suitable for speech normalization because: (1) It only requires monolingual
speech data, without the need of transcriptions to create a text-to-unit model
as in [31]. (2) It learns to denoise features in a high-dimensional space
rather than relying on hand-designed perturbation as in [20]. DiffNorm
consists of a variational auto-encoder (VAE) and a diffusion model. Since
vanilla VAE [26] and DDPM [14] are applied to image generation, we modify them
to support token generation and incorporate multitasking objectives. The VAE
model reduces the dimension of the speech feature, mapping feature $\bm{h}$
into lower dimensional latent $\bm{z}$. The diffusion model (visualized in
Fig. 2) is then trained to denoise on the latent representation space, aiming
to recover the original latent $\bm{z}_{0}$ from the standard Gaussian noise
$\bm{z}_{T}$.333 The subscript $z_{0}$ is added to indicate the time step for
diffusion process and $\bm{z}_{0}$ and $\bm{z}$ are equivalent. In the
subsequent sections, we begin by outlining the architecture of our VAE model
(§ 3.1). Then, we delve into how the diffusion model is trained using the
latent variables encoded by the VAE (§ 3.2).

###  3.1 Variational auto-encoders for latent speech representation

Since speech features encoded by mHuBERT have a high dimension ($H=768$), we
compress the feature into lower-dimension latents for the diffusion model. The
VAE model consists of an encoder, a decoder, and a language modeling head.
Following our problem formulation (§ 2), we first prepare a sequence of target
speech features $\bm{h}=(h_{1},\cdots,h_{M})\in\mathbb{R}^{M\times H}$ and
their corresponding speech units $\bm{y}=(y_{1},\cdots,y_{M})$. Our VAE
model’s encoder will map the feature into lower dimension latent
$\bm{z}=f(\bm{h};\theta_{\rm enc})\in\mathbb{R}^{M\times Z}$ where $Z<H$ is
the pre-defined latent dimension. Then VAE model’s decoder reconstructs the
speech feature $\hat{\bm{h}}=f(\bm{z};\theta_{\rm dec})$ and we apply the
language modeling head to convert the reconstructed feature into a
distribution over speech units’ vocabulary $\bm{v}=f(\hat{\bm{h}};\theta_{\rm
lm})\in\mathbb{R}^{M\times V}$. Following prior work [20, 31], we cluster
speech features into $V=1000$ units. The training objective is a weighted
combination of reconstruction loss ($\mathcal{L}_{\text{recon}}$), negative
log-likelihood (NLL) loss ($\mathcal{L}_{\text{nll}}$), and a Kullback–Leibler
(KL) divergence term resulted from the Gaussian constraint [26] to regularize
the representation space:

$\mathcal{L}=\lambda_{1}\mathcal{L}_{\text{recon}}+\lambda_{2}\mathcal{L}_{\text{nll}}+\lambda_{3}\mathcal{L}_{\text{kl}},$
(1)

where the reconstruction loss is defined as
$\mathcal{L}_{\text{recon}}(\hat{\bm{h}},\bm{h})=||\bm{h}-\hat{\bm{h}}||^{2}$,
and the NLL loss is computed as the cross-entropy between ground-truth units
$\bm{y}$ and the predicted vocabulary distribution $\bm{v}$:
$\mathcal{L}_{\text{nll}}(\bm{v},\bm{y})=-\sum_{i=1}^{M}\bm{y}_{i}\log\bm{v}_{i}$,
where $\bm{y}_{i}$ is the one-hot version of $y_{i}$. Lastly,
$\mathcal{L}_{\text{kl}}$ can be solved analytically when we assume Gaussian
distribution for both prior and posterior approximation of latent $\bm{z}$
(more details in [26] and Appendix B). Though the Gaussian constraint has a
small weight $\lambda_{3}$, we show in the ablation study (§ 6.2) that
regularizing latent distribution is critical for the diffusion model. For
details on the architecture of our VAE model, we direct readers to Appendix B.

Figure 2: Visualization of our latent diffusion model’s denoising process for
speech normalization. The clean latent $\bm{z}_{0}$ is synthetically noised
(into $\bm{z}_{T}$) and the reverse diffusion process gradually denoise it to
generate normalized speech units.

###  3.2 Diffusion model for denoising latent speech representation

Training Once the VAE model is trained, we encode speech feature as the first
step feature $\bm{z}_{0}=f(\bm{h};\theta_{\rm enc})$ for the diffusion model.
Diffusion models consists of a (1) forward process that gradually transforms
$\bm{z}_{0}$ into a standard Gaussian distribution, and a (2) reverse process
that denoise and recovers the original feature $\bm{z}_{0}$. Following DDPM
[14], with a pre-defined noise scheduler (let $\beta_{t}\in(0,1)$ be the
scaling of noise variance, define $\alpha_{t}=1-\beta_{t}$ and denote
$\bar{\alpha}_{t}=\prod_{s=1}^{t}\alpha_{t}$ as the noise level for time $t$),
the forward and reverse process can be written as:

$q(\bm{z}_{t}|\bm{z}_{0})=\mathcal{N}(\bm{z}_{t};\sqrt{\bar{\alpha}_{t}}\bm{z}_{0},(1-\bar{\alpha}_{t})\bm{\epsilon})\;\;\;\;p_{\theta}(\bm{z}_{t-1}|\bm{z}_{t})=\mathcal{N}(\bm{z}_{t-1};\bm{\mu}_{\theta}(\bm{z}_{t},t),\bm{\sigma}^{2}\mathbf{I})$
(2)

where mean $\bm{\mu}_{\theta}(\bm{z}_{t},t)$ and variance $\bm{\sigma}^{2}$
are parameterized by trainable models, typically based on U-Net [42] or
Transformer [52]. We follow DDPM to train models using the re-weighted noise
estimation objective:

$\mathcal{L}_{\text{noise}}(\theta,t)=\mathbbm{E}_{\bm{x}_{0},\bm{\epsilon},t}||\bm{\epsilon}-\bm{\epsilon}_{\theta}(\bm{z}_{0},t)||^{2}$
(3)

where the network learns to predict the injected Gaussian noise
$\bm{\epsilon}$ from the forward process. Besides noise estimation, we also
train the model with auxiliary reconstruction and NLL loss using VAE model’s
decoder and language modeling head. Our training procedure is summarized in
Alg. 1.

Algorithm 1 Latent Diffusion Model Training

1:Input: speech feature $\bm{h}$, speech units $\bm{y}$.

2:Pre-compute: $\beta_{t},\alpha_{t},\bar{\alpha}_{t},t\in[1,T]$ not converged

3:$\bm{z}_{0}=f(\bm{h};\theta_{\rm enc})$

4:$t\sim\mathcal{U}[1,T]$, $\bm{\epsilon}\sim\mathcal{N}(0,\mathbf{I})$

5:$\bm{z}_{t}=\sqrt{\bar{\alpha}_{t}}\bm{z}_{0}+\sqrt{1-\bar{\alpha}_{t}}\bm{\epsilon}$

6:$\hat{\bm{\epsilon}}=\bm{\epsilon}_{\theta}(\bm{z}_{t},t)$

7:$\mathcal{L}_{\text{noise}}=||\bm{\epsilon}-\hat{\bm{\epsilon}}||^{2}$

8:$\hat{\bm{z}}_{0}=(\bm{z}_{t}-\sqrt{1-\bar{\alpha}_{t}}\hat{\bm{\epsilon}})/\sqrt{\bar{\alpha}_{t}}$

9:$\hat{\bm{h}}=f(\hat{\bm{z}}_{0};\theta_{\rm dec})$,
$\mathcal{L}_{\text{recon}}=||\bm{h}-\hat{\bm{h}}||^{2}$

10:$\bm{v}=f(\hat{\bm{h}};\theta_{\rm lm})$,
$\mathcal{L}_{\text{nll}}=\textsc{nll}(\bm{v},\bm{y})$

11:Take gradient descent step with loss

12:
$\mathcal{L}=\gamma_{1}\mathcal{L}_{\text{noise}}+\gamma_{2}\mathcal{L}_{\text{recon}}+\gamma_{3}\mathcal{L}_{\text{nll}}$

Algorithm 2 Normalized Units Construction

1:Input: speech feature $\bm{h}$, start time $T$

2:Pre-compute: $\beta_{t},\alpha_{t},\bar{\alpha}_{t},t\in[1,T]$

3:$\bm{z}_{0}=f(\bm{h};\theta_{\rm enc})$

4:$\bm{z}_{T}=\sqrt{\bar{\alpha}_{T}}\bm{z}_{0}+\sqrt{1-\bar{\alpha}_{T}}\bm{\epsilon}$

5:$t=T$ $t>0$

6:$\hat{\bm{\epsilon}}=\bm{\epsilon}_{\theta}(\bm{z}_{t},t)$

7:$\hat{\bm{z}}_{0}=(\bm{z}_{t}-\sqrt{1-\bar{\alpha}_{t}}\hat{\bm{\epsilon}})/\sqrt{\bar{\alpha}_{t}}$

8:$\bm{z}_{t-1}=\sqrt{\bar{\alpha}_{t-1}}\hat{\bm{z}}_{0}+\sqrt{1-\bar{\alpha}_{t-1}}\cdot\hat{\bm{\epsilon}}$

9:$t=t-1$

10:$\hat{\bm{h}}=f(\bm{z}_{0};\theta_{\rm dec})$

11:$\bm{y}_{\rm norm}=\text{argmax}f(\hat{\bm{h}};\theta_{\rm lm})$

12:denoised units $\bm{y}_{\rm norm}$

As shown in Alg. 1, we randomly sample the current timestep $t\in[1,T]$ and
compute corresponding scheduling parameters
$\beta_{t},\alpha_{t},\bar{\alpha}_{t}$444 We follow the cosine schedule
proposed by Nichol and Dhariwal, [34] with a maximum timestep of $T=200$. The
training process involves the regular DDPM objective that injects Gaussian
noise for the model $\bm{\epsilon}_{\theta}$ to perform noise estimation (Alg.
1, line 6-8). Additionally, we generate the pseudo latent $\hat{\bm{z}}_{0}$
by reversing the noise injection process (line 9), which is then decoded by
the VAE into speech features and units for reconstruction loss (line 10) and
NLL loss (line 11). Finally, the objective is a weighted sum of noise
estimation loss, reconstruction loss, and NLL loss:

$\mathcal{L}=\gamma_{1}\mathcal{L}_{\text{noise}}+\gamma_{2}\mathcal{L}_{\text{recon}}+\gamma_{3}\mathcal{L}_{\text{nll}}$
(4)

In our analysis (§ 6.2), we show that adding NLL and reconstruction loss is
indeed helpful in improving the diffusion model’s reconstruction quality.
Lastly, to parameterize our diffusion model for noise estimation, we modify
Diffusion Transformer [36, DiT] to suit our task. For details of our
architecture and hyper-parameters, we refer readers to Appendix B.

Inference  For speech normalization, we choose a start time $T\in[0,200]$ that
decides the amount of noise to inject for the diffusion model to recover.
Given the start time $T$, we follow Denoising Diffusion Implicit Models [46,
DDIM] sampler to reverse noised latent $\bm{z}_{T}$ back to $\bm{z}_{0}$.
Then, our VAE model converts $\bm{z}_{0}$ back to speech units with its
decoder and language modeling head. Our inference procedure is summarized in
Alg. 2 and visualized in Fig. 2.

Note that since our diffusion model is used to normalize speech units as a
data preprocessing strategy, inference speed is not a concern. Therefore we
use step size $\Delta t=1$ for DDIM sampling throughout our experiments.
However, the inference speed could be easily improved by setting a larger step
size.

##  4 Classifier-free guidance for non-autoregressive transformer

In § 3, we proposed DiffNorm to normalize speech units and obtained speech-
unit pairs $(\bm{x},\bm{y}_{\rm norm})$ that benefit NAT training. In this
section, we propose to adapt classifier-free guidance [16] to regularize NAT
models (visualized in Fig. 3), further enhancing NAT-based S2UT model’s
translation quality.

Figure 3: Visualization of CMLM for speech-to-unit translation where the model
is trained with the unmasking objective to recover $\bm{y}_{\rm norm}$. When
classifier-free guidance is used, with probability $p_{\text{drop}}$, we
replace the encoded source speech $\bm{g}$ by a "null" representation
$\bm{g}_{\emptyset}$.

Training  In our CMLM training with classifier-free guidance, we largely
adhere to standard CMLM training [20] but introduce a small dropout
probability $p_{\text{drop}}=0.15$ for the encoder representations.
Specifically, we parameterize the encoder and decoder of the CMLM as
$\phi_{\rm enc}$ and $\phi_{\rm dec}$, respectively. The encoder processes the
source speech into a representation $\bm{g}=f(\bm{x};\phi_{\rm enc})$. The
decoder then predicts the vocabulary distribution
$\bm{v}=f(\hat{\bm{y}}|\bm{g};\phi_{\rm dec})$ from $\bm{g}$ and the randomly
masked target speech units $\hat{\bm{y}}$. We calculate the NLL loss using
$\bm{v}$ and the ground truth units $\bm{y}$ at the masked positions to train
the CMLM model. If dropout is applied, the decoder receives a "null"
representation $\bm{g}_{\emptyset}$—a randomly-initialized learnable
vector—forcing it to rely solely on the target information to unmask units.
For more details of CMLM, we refer readers to Appendix C and previous paper
[9, 20].

Inference  CMLM generates tokens through iterative unmasking. Given the input
sequence $\bm{x}$, CMLM first predicts the length of output sequence $M$ and
initialize a sequence of $M$ masked tokens
$\hat{\bm{y}}_{0}=([\text{mask}]_{1},\cdots,[\text{mask}]_{M})$. Given the
total number of iterations $T$555 $T$ was used in § 3 to denote total
timesteps for diffusion models. We abuse the notation and re-use it here to
represent the number of decoding iterations for CMLM. and current iteration
$t\in[1,T]$, CMLM decodes all tokens $y_{i},i\in[1,M]$ in parallel using their
probabilities:

$y_{i}^{t}=\underset{w}{\text{argmax}}\mathbbm{P}(y_{i}=w|\bm{x},\hat{\bm{y}}_{t-1};\phi_{\rm
dec})\;\;\;\;p_{i}^{t}=\log\mathbbm{P}(y_{i}^{t}|\bm{x},\hat{\bm{y}}_{t-1};\phi_{\rm
dec})$ (5)

From the generated tokens $\hat{\bm{y}}_{t}=(y_{1}^{t},\cdots,y_{M}^{t})$, we
replace $k=M\cdot\frac{T-t}{T}$ tokens with the lowest probabilities
$p_{i}^{t}$ with mask tokens. This re-masked sequence $\hat{\bm{y}}_{t}$ is
then inputted into the CMLM for further decoding iterations. This process
continues until the final step $t=T$, at which point no tokens are re-masked.
With our proposed regularization from classifier-free guidance, we compute two
vocabulary distribution in each iteration step $t$:

$p_{\text{orig}}^{t}=\mathbb{P}(\cdot|\bm{g},\hat{\bm{y}}_{t-1};\phi_{\rm
dec})\;\;\;\;p_{\text{uncond}}^{t}=\mathbb{P}(\cdot|\bm{g}_{\emptyset},\hat{\bm{y}}_{t-1};\phi_{\rm
dec})$ (6)

where $p_{\text{orig}}^{t}$ is the same as vanilla CMLM and
$p_{\text{uncond}}^{t}$ is the unconditional probability obtained with the
"null" representation. Then we select and re-mask tokens using the adjusted
probability

$p^{t}=\omega(p_{\text{orig}}^{t}-p_{\text{uncond}}^{t})+p_{\text{orig}}^{t}$
(7)

where the hyper-parameter $\omega$ is used to control the degree to push away
from the unconditional distribution. In the special case of $\omega=0$, only
conditional distribution is used during iterative decoding, resulting in the
same generation process as standard CMLM. Through our intensive analysis
(Appendix F), we determine that a relatively small $\omega$ (e.g.,
$\omega=0.5$) gives the best result, especially when the number of decoding
iterations is large (i.e., $T>10$). Notably, even when $\omega=0$ — which
corresponds to the traditional CMLM decoding method — CMLMs regularized with
classifier-free guidance during training can consistently outperform their
standard counterparts. This observation further validates the effectiveness of
our proposed regularization.

##  5 Experiments

###  5.1 Experimental setup

Split | En-Es | En-Fr
---|---|---
Size | Length | Size | Length
Train | 79,012 | 256 | 207,364 | 228
Valid | 13,212 | 296 | 14,759 | 264
Test | 13,216 | 308 | 14,759 | 283
Table 1: Data statistics for CVSS benchmarks. Length is the average number of
speech units of the target speech.

Dataset  We perform experiments using the established CVSS-C datasets [23],
which are created from CoVoST2 by employing advanced text-to-speech models to
synthesize translation texts into speech [54]. CVSS-C comprises aligned speech
in multiple languages along with their respective transcriptions. Our methods
are evaluated on two language pairs: English-Spanish (En-Es) and English-
French (En-Fr), with detailed data statistics provided in Table 1. As our
focus is on direct speech-to-speech translation, transcriptions are solely
utilized for evaluation purposes. Utilizing the speech data from CVSS, we
preprocess the target speech to generate speech units using the mHuBERT and
K-means model, as described in our problem formulation (§ 2).

Evaluation  To evaluate the performance of various speech-to-speech
translation systems, we adopt the standard methodology established in previous
studies [31, 30, 20] to compute the ASR-BLEU score. Firstly, our speech-to-
unit translation system generates speech units based on the input speech,
which are then transformed into speech waveforms using a unit-vocoder. The
unit-vocoder is built upon the HifiGAN architecture [27] with customized
objectives, detailed in Appendix D. Once we have the waveforms, we employ an
ASR model to transcribe them and calculate the BLEU score against the
reference transcriptions. This ASR model is fine-tuned based on the Wav2Vec2.0
[4] model with the CTC objective. We direct readers to the original paper [6]
for more details. Both the unit-vocoder and ASR models are sourced from an
off-the-shelf repository, ensuring consistency in evaluation methodology with
previous studies [31, 20].666 Repository available
at:github.com/facebookresearch/fairseq/examples/speech_to_speech/asr_bleu.

Normalized dataset construction  We adhere to the procedure outlined in Alg. 2
to generate normalized speech units. By manipulating start time $T$, we
control the level of noise in the latent
$\bm{z}_{T}=\sqrt{\bar{\alpha}_{t}}\bm{z}_{0}+\sqrt{1-\bar{\alpha}_{t}}\bm{\epsilon}$
for the diffusion model, balancing between the reconstruction quality and
normalization effect. We explore different levels of noise injection and
choose $T=100$ for En-Es and $T=120$ for En-Fr (further details in § 6.1).
Hence, we adopt these settings to construct our normalized dataset, CVSS-norm.

Compared systems  In this section, we provide brief descriptions of evaluated
speech-to-unit models. For autoregressive models, we evaluate the Transformer
model trained following Lee et al., 2022a [30]. We also assess the Conformer
model trained similar to the Transformer, but with its encoder replaced by a
Conformer-Encoder [13]. Lastly, the Norm Transformer shares the same
architecture as the Transformer, but it is trained on normalized speech units
that are speaker-invariant. The normalized dataset is constructed following
the strategy proposed by Lee et al., 2022b [31].

For non-autoregressive systems, we train the Conditional Masked Language Model
(CMLM) following Huang et al., [20], using a Conformer-based encoder and a
Transformer decoder. The CMLM + Bilateral Perturbation (BiP) system retains
the same architecture as CMLM but is trained on normalized speech units
constructed with BiP [20].

For our improved systems, we train CMLM + DiffNorm, which shares the same
architecture as CMLM but is trained on CVSS-norm, the normalized dataset
obtained through DiffNorm. Additionally, we train the CMLM + CG model that
incorporates classifier-free guidance introduced in § 4. Finally, the CMLM +
DiffNorm \+ CG system uses the architecture of CMLM + CG and is trained on our
normalized dataset CVSS-norm.

###  5.2 Results

Table 2 summarizes the (ASR-BLEU) performances of various S2UT systems. Note
that, for non-autoregressive methods, Table 2 shows their results obtained
with 15 decoding iterations. For more details on the inference speedup
achieved through non-autoregressive modeling, we plot Fig. 4 to provide
details on the “quality vs. latency” trade-off. Observations from Table 2 are:

DiffNorm greatly enhances translation quality compared to systems using
original speech units (model 6 vs. 4; 8 vs. 7). For example, CMLM with
DiffNorm achieves about $+7$ BLEU score on En-Es compared to CMLM trained on
original units. DiffNorm also outperforms previous normalization strategies
(model 6 v.s. 2, 5), validating the superior quality of normalized speech
units obtained through our methods. Besides normalization, classifier-free
guidance effectively improves speech-to-unit quality as a regularization
strategy, leading to better translation quality (model 7 v.s. 4; 8 v.s. 6).
Finally, combining both classifier-free guidance and DiffNorm (model 8)
results in the best overall system, outperforming both autoregressive and non-
autoregressive baselines.

ID | System | Quality $\uparrow$ | Inference Speed $\uparrow$
---|---|---|---
abEn-Es | En-Fr | Speed | Speedup
Autoregressive
1 |  Transformer† [30] | 10.07 | 15.28 | 870 | 1.00$\times$
2 |  Norm Transformer† [31] | 12.98 | 15.93 | 870 | 1.00$\times$
3 | Conformer† | 13.75 | 17.07 | 895 | 1.02$\times$
Non-autoregressive Model
4 | CMLM | 12.58 | 15.62 | 4651 | 5.34$\times$
5 | CMLM + BiP†[20] | 12.62 | 16.97
Our Improved Non-autoregressive Model
6 | CMLM + DiffNorm | 18.96 | 17.27 | 4651 | 5.34$\times$
7 | CMLM + CG‡ | 17.06 | 16.89
8 | CMLM + DiffNorm \+ CG‡ | 19.49 | 17.54
Table 2: Comparison of speech-to-speech models evaluated by quality (ASR-BLEU)
and speed (units/seconds). Results with † are taken from the prior work [20].
‡ We use $w=0.5$ for CG. Our NAT models achieve superior translation quality
while maintaining their fast inference speed.

Decoding speed  In Fig. 4, we illustrate the “quality-latency” trade-off of
various non-autoregressive speech-to-unit systems. Quality is measured using
ASR-BLEU, while latency is determined by the relative speedup over the
autoregressive system, calculated as “generated units/second”. For instance,
the first marker on the line plot represents $15$ decoding iterations,
resulting in a speedup of 5.34$\times$ compared to the autoregressive
baseline. Additionally, we include the performance of the Conformer-based
autoregressive model as a horizontal dashed line. We observe that our improved
system consistently outperforms the baseline CMLM model [20] and achieves
better performance than the autoregressive model with more than $14\times$
speedup for En-Es and $5\times$ speedup for En-Fr.

Figure 4: Trade-off between quality (ASR-BLEU) and latency for varying numbers
of decoding iterations. Five markers correspond to {15, 10, 7, 5, 3} decoding
iterations. Decreasing the number of iterations results in a decline in model
performance, traded off for faster speedup. With DiffNorm and CG, our S2UT
model achieves a better quality-latency trade-off than CMLM and outperforms a
strong autoregressive baseline with large speedups.

##  6 Analysis

###  6.1 Effect of synthetic noise injection

In this section, we explore the effects of varying degrees of noise injection
on DiffNorm. Recall (from Alg. 2) that synthetic noise is injected into the
latent representation as
$\bm{z}_{T}=\sqrt{\bar{\alpha}_{T}}\bm{z}_{0}+\sqrt{1-\bar{\alpha}_{T}}\bm{\epsilon}$.
Hence, adjusting the start time $T$ allows us to control the degree of noise
injection.

Setup and Evaluation  We perturb the degree of noise injection by varying $T$,
and assess the reconstruction quality of normalized speech units and
downstream performance of CMLM models trained with different normalized units.
For reconstruction quality, we measure the accuracy (Acc-Rec) of reconstructed
units and ASR-BLEU (BL-Rec) of synthesized speech when using original
units/speech as the reference. For downstream performance, we report the
downstream ASR-BLEU of translated speech produced by CMLM models trained with
the particular noise setup (BL-Dn).777ASR-BLEU is computed by synthesizing
speech from the reconstructed units and transcribing it using the ASR model,
as detailed in § 5.1.

Results  Shown in Table 3, increasing $T$ leads to more noise and a decline in
reconstruction quality, which aligns with expectations that higher noise
levels pose greater challenges for the diffusion model in restoring original
features accurately. Interestingly, we find from Table 4 that when $T$ is too
small or large (e.g., $T=50$ or $T=150$), the normalized units do not result
in an ideal downstream system. We hypothesize that there is barely any
normalization effect when $T$ is too small; and when $T$ is too large, the
reconstructed units are no longer semantically correct. This phenomenon is
visualized in Fig. 5, where we plot the log-mel spectrograms of the
reconstructed speech. As more noise is introduced (larger $T$), the
reconstructed speech becomes smoother (e.g., portions of blank speech are
filled in by diffusion), exhibiting a more pronounced deviation from the
original speech.

Start Step | Objectives | Latent Dimension
---|---|---
| KL | Multitask | 16 | 32 | 128
$T=50$ | ✗ | ✓ | 51.4 | 69.1 | 85.8
✓ | ✗ | 80.9 | 86.4 | 89.3
✓ | ✓ | 80.9 | 86.5 | 89.4
$T=100$ | ✗ | ✓ | 13.3 | 32.5 | 73.1
✓ | ✗ | 64.5 | 76 | 80.8
✓ | ✓ | 65.3 | 76.2 | 81.2
$T=150$ | ✗ | ✓ | 2.9 | 5.3 | 34.6
✓ | ✗ | 19.8 | 38.2 | 56.8
✓ | ✓ | 20.2 | 40 | 57.8
Table 3: Accuray of reconstructed speech units. KL: when applied, the latent
space is regularized to be Gaussian [26]. Multitask: when not applied, the
latent diffusion model is trained only with $\mathcal{L}_{\text{noise}}$.

Start Step | Scheduler Parameters | En-Es | En-Fr
---|---|---|---
| $\beta_{t}$ | $\sqrt{\bar{\alpha}_{t}}$ | $\sqrt{1-\bar{\alpha}_{t}}$ |  Acc-Rec $\uparrow$ |  BL-Rec $\uparrow$ |  BL-Dn $\uparrow$ |  Acc-Rec $\uparrow$ |  BL-Rec $\uparrow$ |  BL-Dn $\uparrow$
original units | N/A | N/A | N/A | 100 | 48.7 | 12.58 | 100 | 40.64 | 15.62
$T=50$ | 0.007 | 0.917 | 0.398 | 89.4 | 48.5 | 18.56 | 91.1 | 40.38 | 17.12
$T=100$ | 0.016 | 0.697 | 0.717 | 81.2 | 47.63 | 18.96 | 83.0 | 39.9 | 16.69
$T=120$ | 0.022 | 0.577 | 0.816 | 75.3 | 46.25 | 16.7 | 77.7 | 39.03 | 17.27
$T=150$ | 0.038 | 0.373 | 0.928 | 57.8 | 31.43 | 14.77 | 60.0 | 28.3 | 14.03

Table 4: For different start steps $T$, we show corresponding noise scheduling
parameter values, reconstruction quality (-Rec columns), and downstream
translation quality (-Dn column). Noise injection that perturbs about 20% of
units (i.e., 80% Acc-Rec) results in the best downstream S2UT performance
(highlighted in bold text).

###  6.2 Ablation on training objectives for DiffNorm

DiffNorm requires an auto-encoder to map speech features into lower-dimension
latents and then train denoising diffusion models using the encoded latents.
In this section, we perform ablation experiments to investigate the effect of
(1) latent dimension (2) Gaussian constraints (3) multitask objective for
diffusion models. Firstly, to investigate the effect of latent dimension, we
train auto-encoders that encode speech features into 16, 32, and 128
dimensions. As shown in Table 3, it comes as no surprise that a larger latent
dimension yields superior reconstruction results, evidenced by higher accuracy
across systems trained with varying objectives. Next, we turn our attention to
the importance of applying Gaussian constraints to the auto-encoder’s latent
representation space. Our results (comparing with and without KL) reveal that
incorporating Gaussian constraints is crucial. Latent spaces not regularized
by $\mathcal{L}_{\text{kl}}$ lead to significantly poorer performance.
Finally, we explore the effectiveness of our proposed multitasking objective
for the diffusion model by training another vanilla DDPM solely with
$\mathcal{L}_{\rm noise}$ (no Multitask). We find that the diffusion model’s
reconstruction capability indeed improves when employing the multitasking
objective, particularly when denoising from a more noisy latent representation
(e.g., $T=150$). Through our ablation analysis, we identify the optimal setup
for our DiffNorm model, which involves (1) mapping speech features to 128
dimensions, (2) regularizing latent space by Gaussian constraints, and (3)
utilizing the multitask objective for diffusion modeling training.

Figure 5: Visualization of reconstructed speech’s log-mel spectrograms.
Noticeable divergence from the original speech is highlighted in the white
bounding boxes.

##  7 Related work

Direct speech-to-speech translation (S2ST)  Direct S2ST aims to directly
translate speech into another language without cascaded systems that rely on
transcriptions or translations. Translatotron [24] and Translatotron 2 [22]
are among the first systems for direct S2ST, which uses sequence-to-sequence
model to map source speech into spectrograms. Then, a spectrogram decoder is
used to synthesize the target language’s speech. Instead of transducing speech
into spectrogram, Tjandra et al., [48], Zhang et al., [56] utilize Vector-
Quantized Variational Auto-Encoder (VQ-VAE) [51] to discretize target speech
and convert speech-to-speech translation into a speech-to-unit task. More
recently, Lee et al., 2022a [30] improved the speech-to-unit models by
obtaining such units with a k-means clustering model trained on self-
supervised representation from HuBERT [18]. To convert units back to speech,
Lee et al., 2022a [30] follows Polyak et al., [37] to train a unit-vocoder
based on HifiGAN [2].

Speech normalization  Inspired by previous work on speech enhancement [49, 1],
Lee et al., 2022b [31] propose to normalize speech by synthesizing speaker-
invariant waveforms through text-to-speech (TTS) systems [55, 44, 40, 39].
Huang et al., [20] propose to normalize speech with Bilateral Perturbation
that focus on the rhythm, pitch, and energy information. Different from
previous normalization methods that requires transcription [31] or manually
designed perturbation [20], our DiffNorm strategy leverages diffusion models.
Diffusion models [14, 46, 47] have achieved remarkable generative ability to
produce high-quality images [7, 15, 41, 36, 43] and audio [32, 38, 45]. Using
self-supervised denoising objectives [14], we train effective denoising models
capable of normalizing speech for training better speech-to-unit systems.

Non-autoregressive speech-to-speech translation  For sequence-to-sequence
modeling, auto-regressive [17, 5, 52] and non-autoregressive [11, 12, 9, 19]
models have been widely explored. Lee et al., 2022a [30] reduce speech-to-
speech into speech-to-unit task and follow the widely-used modeling strategy,
Transformer [52], to predict speech units. Later, non-autoregressive models
[20, 8] have been explored and Huang et al., [20] is the first to use a non-
autoregressive transformer model, Conditional Masked Language Model [9, CMLM],
for speech-to-unit task. Fang et al., [8], on the other hand, adopts Directed
Acyclic Transformer [19] for speech-to-unit translation.

##  8 Conclusion

We improve speech-to-unit translation system through (1) corpus distillation
by constructing normalized speech units with DiffNorm and (2) regularization
with classifier-free guidance. Our improved non-autoregressive (NAR) models,
greatly outperform previous NAR models, yielding an increase of approximately
$+7$ and $+2$ BLEU points for En-Es and En-Fr translation, respectively.
Notably, DiffNorm and classifier-free guidance maintain the inference speed
advantages inherent in NAR models. Consequently, our approach obtains better
performance to autoregressive baselines while achieving over $14\times$
speedup for En-Es and $5\times$ speedup for En-Fr translations.

## References

  * Adiga et al., [2019] Adiga, N., Pantazis, Y., Tsiaras, V., and Stylianou, Y. (2019).  Speech Enhancement for Noise-Robust Speech Synthesis Using Wasserstein GAN.  In Proc. Interspeech 2019, pages 1821–1825.
  * Andreev et al., [2023] Andreev, P., Alanov, A., Ivanov, O., and Vetrov, D. (2023).  Hifi++: A unified framework for bandwidth extension and speech enhancement.  In ICASSP 2023 - 2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1–5.
  * Baevski et al., [2019] Baevski, A., Auli, M., and rahman Mohamed, A. (2019).  Effectiveness of self-supervised pre-training for speech recognition.  ArXiv, abs/1911.03912.
  * Baevski et al., [2020] Baevski, A., Zhou, H., Mohamed, A., and Auli, M. (2020).  wav2vec 2.0: A framework for self-supervised learning of speech representations.
  * Bahdanau et al., [2014] Bahdanau, D., Cho, K., and Bengio, Y. (2014).  Neural machine translation by jointly learning to align and translate.  CoRR, abs/1409.0473.
  * Chen et al., [2023] Chen, P.-J., Tran, K., Yang, Y., Du, J., Kao, J., Chung, Y.-A., Tomasello, P., Duquenne, P.-A., Schwenk, H., Gong, H., Inaguma, H., Popuri, S., Wang, C., Pino, J., Hsu, W.-N., and Lee, A. (2023).  Speech-to-speech translation for a real-world unwritten language.  In Rogers, A., Boyd-Graber, J., and Okazaki, N., editors, Findings of the Association for Computational Linguistics: ACL 2023, pages 4969–4983, Toronto, Canada. Association for Computational Linguistics.
  * Dhariwal and Nichol, [2021] Dhariwal, P. and Nichol, A. (2021).  Diffusion models beat gans on image synthesis.
  * Fang et al., [2023] Fang, Q., Zhou, Y., and Feng, Y. (2023).  DASpeech: Directed acyclic transformer for fast and high-quality speech-to-speech translation.  In Advances in Neural Information Processing Systems.
  * Ghazvininejad et al., [2019] Ghazvininejad, M., Levy, O., Liu, Y., and Zettlemoyer, L. (2019).  Mask-predict: Parallel decoding of conditional masked language models.  In Inui, K., Jiang, J., Ng, V., and Wan, X., editors, Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 6112–6121, Hong Kong, China. Association for Computational Linguistics.
  * Goodfellow et al., [2014] Goodfellow, I. J., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., and Bengio, Y. (2014).  Generative adversarial networks.
  * Gu et al., [2018] Gu, J., Bradbury, J., Xiong, C., Li, V. O., and Socher, R. (2018).  Non-autoregressive neural machine translation.  In International Conference on Learning Representations.
  * Gu et al., [2019] Gu, J., Wang, C., and Zhao, J. (2019).  Levenshtein transformer.  In Wallach, H., Larochelle, H., Beygelzimer, A., d'Alché-Buc, F., Fox, E., and Garnett, R., editors, Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc.
  * Gulati et al., [2020] Gulati, A., Qin, J., Chiu, C., Parmar, N., Zhang, Y., Yu, J., Han, W., Wang, S., Zhang, Z., Wu, Y., and Pang, R. (2020).  Conformer: Convolution-augmented transformer for speech recognition.  CoRR, abs/2005.08100.
  * Ho et al., [2020] Ho, J., Jain, A., and Abbeel, P. (2020).  Denoising diffusion probabilistic models.  In Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M., and Lin, H., editors, Advances in Neural Information Processing Systems, volume 33, pages 6840–6851. Curran Associates, Inc.
  * Ho et al., [2021] Ho, J., Saharia, C., Chan, W., Fleet, D. J., Norouzi, M., and Salimans, T. (2021).  Cascaded diffusion models for high fidelity image generation.
  * Ho and Salimans, [2022] Ho, J. and Salimans, T. (2022).  Classifier-free diffusion guidance.
  * Hochreiter and Schmidhuber, [1997] Hochreiter, S. and Schmidhuber, J. (1997).  Long short-term memory.  Neural computation, 9(8):1735–1780.
  * Hsu et al., [2021] Hsu, W.-N., Bolte, B., Tsai, Y.-H. H., Lakhotia, K., Salakhutdinov, R., and Mohamed, A. (2021).  Hubert: Self-supervised speech representation learning by masked prediction of hidden units.
  * Huang et al., [2022] Huang, F., Zhou, H., Liu, Y., Li, H., and Huang, M. (2022).  Directed acyclic transformer for non-autoregressive machine translation.  In Chaudhuri, K., Jegelka, S., Song, L., Szepesvari, C., Niu, G., and Sabato, S., editors, Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pages 9410–9428. PMLR.
  * Huang et al., [2023] Huang, R., Liu, J., Liu, H., Ren, Y., Zhang, L., He, J., and Zhao, Z. (2023).  Transpeech: Speech-to-speech translation with bilateral perturbation.  In The Eleventh International Conference on Learning Representations.
  * Inaguma et al., [2023] Inaguma, H., Popuri, S., Kulikov, I., Chen, P.-J., Wang, C., Chung, Y.-A., Tang, Y., Lee, A., Watanabe, S., and Pino, J. (2023).  UnitY: Two-pass direct speech-to-speech translation with discrete units.  In Rogers, A., Boyd-Graber, J., and Okazaki, N., editors, Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 15655–15680, Toronto, Canada. Association for Computational Linguistics.
  * [22] Jia, Y., Ramanovich, M. T., Remez, T., and Pomerantz, R. (2022a).  Translatotron 2: High-quality direct speech-to-speech translation with voice preservation.
  * [23] Jia, Y., Ramanovich, M. T., Wang, Q., and Zen, H. (2022b).  Cvss corpus and massively multilingual speech-to-speech translation.
  * Jia et al., [2019] Jia, Y., Weiss, R. J., Biadsy, F., Macherey, W., Johnson, M., Chen, Z., and Wu, Y. (2019).  Direct speech-to-speech translation with a sequence-to-sequence model.  ArXiv, abs/1904.06037.
  * Kingma and Ba, [2017] Kingma, D. P. and Ba, J. (2017).  Adam: A method for stochastic optimization.
  * Kingma and Welling, [2022] Kingma, D. P. and Welling, M. (2022).  Auto-encoding variational bayes.
  * Kong et al., [2020] Kong, J., Kim, J., and Bae, J. (2020).  Hifi-gan: Generative adversarial networks for efficient and high fidelity speech synthesis.
  * Lavie et al., [1997] Lavie, A., Waibel, A., Levin, L., Finke, M., Gates, D., Gavalda, M., Zeppenfeld, T., and Zhan, P. (1997).  Janus-iii: speech-to-speech translation in multiple languages.  In 1997 IEEE International Conference on Acoustics, Speech, and Signal Processing, volume 1, pages 99–102 vol.1.
  * Lecun et al., [1998] Lecun, Y., Bottou, L., Bengio, Y., and Haffner, P. (1998).  Gradient-based learning applied to document recognition.  Proceedings of the IEEE, 86(11):2278–2324.
  * [30] Lee, A., Chen, P.-J., Wang, C., Gu, J., Popuri, S., Ma, X., Polyak, A., Adi, Y., He, Q., Tang, Y., Pino, J., and Hsu, W.-N. (2022a).  Direct speech-to-speech translation with discrete units.  In Muresan, S., Nakov, P., and Villavicencio, A., editors, Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 3327–3339, Dublin, Ireland. Association for Computational Linguistics.
  * [31] Lee, A., Gong, H., Duquenne, P.-A., Schwenk, H., Chen, P.-J., Wang, C., Popuri, S., Adi, Y., Pino, J., Gu, J., and Hsu, W.-N. (2022b).  Textless speech-to-speech translation on real data.  In Carpuat, M., de Marneffe, M.-C., and Meza Ruiz, I. V., editors, Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 860–872, Seattle, United States. Association for Computational Linguistics.
  * Liu et al., [2023] Liu, H., Chen, Z., Yuan, Y., Mei, X., Liu, X., Mandic, D., Wang, W., and Plumbley, M. D. (2023).  AudioLDM: Text-to-audio generation with latent diffusion models.  Proceedings of the International Conference on Machine Learning.
  * Nakamura et al., [2006] Nakamura, S., Markov, K., Nakaiwa, H., Kikui, G., Kawai, H., Jitsuhiro, T., Zhang, J.-S., Yamamoto, H., Sumita, E., and Yamamoto, S. (2006).  The atr multilingual speech-to-speech translation system.  IEEE Transactions on Audio, Speech, and Language Processing, 14(2):365–376.
  * Nichol and Dhariwal, [2021] Nichol, A. Q. and Dhariwal, P. (2021).  Improved denoising diffusion probabilistic models.  In Meila, M. and Zhang, T., editors, Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 8162–8171. PMLR.
  * Ott et al., [2019] Ott, M., Edunov, S., Baevski, A., Fan, A., Gross, S., Ng, N., Grangier, D., and Auli, M. (2019).  fairseq: A fast, extensible toolkit for sequence modeling.  In Ammar, W., Louis, A., and Mostafazadeh, N., editors, Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics (Demonstrations), pages 48–53, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Peebles and Xie, [2023] Peebles, W. and Xie, S. (2023).  Scalable diffusion models with transformers.
  * Polyak et al., [2021] Polyak, A., Adi, Y., Copet, J., Kharitonov, E., Lakhotia, K., Hsu, W.-N., Mohamed, A., and Dupoux, E. (2021).  Speech resynthesis from discrete disentangled self-supervised representations.
  * Popov et al., [2021] Popov, V., Vovk, I., Gogoryan, V., Sadekova, T., and Kudinov, M. (2021).  Grad-tts: A diffusion probabilistic model for text-to-speech.  In Meila, M. and Zhang, T., editors, Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 8599–8608. PMLR.
  * Ren et al., [2022] Ren, Y., Hu, C., Tan, X., Qin, T., Zhao, S., Zhao, Z., and Liu, T.-Y. (2022).  Fastspeech 2: Fast and high-quality end-to-end text to speech.
  * Ren et al., [2019] Ren, Y., Ruan, Y., Tan, X., Qin, T., Zhao, S., Zhao, Z., and Liu, T.-Y. (2019).  Fastspeech: Fast, robust and controllable text to speech.
  * Rombach et al., [2022] Rombach, R., Blattmann, A., Lorenz, D., Esser, P., and Ommer, B. (2022).  High-resolution image synthesis with latent diffusion models.
  * Ronneberger et al., [2015] Ronneberger, O., Fischer, P., and Brox, T. (2015).  U-net: Convolutional networks for biomedical image segmentation.
  * Saharia et al., [2022] Saharia, C., Chan, W., Saxena, S., Li, L., Whang, J., Denton, E., Ghasemipour, S. K. S., Ayan, B. K., Mahdavi, S. S., Lopes, R. G., Salimans, T., Ho, J., Fleet, D. J., and Norouzi, M. (2022).  Photorealistic text-to-image diffusion models with deep language understanding.
  * Shen et al., [2018] Shen, J., Pang, R., Weiss, R. J., Schuster, M., Jaitly, N., Yang, Z., Chen, Z., Zhang, Y., Wang, Y., Skerry-Ryan, R., Saurous, R. A., Agiomyrgiannakis, Y., and Wu, Y. (2018).  Natural tts synthesis by conditioning wavenet on mel spectrogram predictions.
  * Shen et al., [2023] Shen, K., Ju, Z., Tan, X., Liu, Y., Leng, Y., He, L., Qin, T., Zhao, S., and Bian, J. (2023).  Naturalspeech 2: Latent diffusion models are natural and zero-shot speech and singing synthesizers.
  * [46] Song, J., Meng, C., and Ermon, S. (2021a).  Denoising diffusion implicit models.  In International Conference on Learning Representations.
  * [47] Song, Y., Sohl-Dickstein, J., Kingma, D. P., Kumar, A., Ermon, S., and Poole, B. (2021b).  Score-based generative modeling through stochastic differential equations.
  * Tjandra et al., [2019] Tjandra, A., Sakti, S., and Nakamura, S. (2019).  Speech-to-speech translation between untranscribed unknown languages.  2019 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU), pages 593–600.
  * Valentini-Botinhao et al., [2016] Valentini-Botinhao, C., Wang, X., Takaki, S., and Yamagishi, J. (2016).  Speech Enhancement for a Noise-Robust Text-to-Speech Synthesis System Using Deep Recurrent Neural Networks.  In Proc. Interspeech 2016, pages 352–356.
  * van den Oord et al., [2016] van den Oord, A., Dieleman, S., Zen, H., Simonyan, K., Vinyals, O., Graves, A., Kalchbrenner, N., Senior, A., and Kavukcuoglu, K. (2016).  Wavenet: A generative model for raw audio.
  * van den Oord et al., [2017] van den Oord, A., Vinyals, O., and kavukcuoglu, k. (2017).  Neural discrete representation learning.  In Guyon, I., Luxburg, U. V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., and Garnett, R., editors, Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc.
  * Vaswani et al., [2017] Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L. u., and Polosukhin, I. (2017).  Attention is all you need.  In Guyon, I., Luxburg, U. V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., and Garnett, R., editors, Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc.
  * Wang et al., [2021] Wang, C., Riviere, M., Lee, A., Wu, A., Talnikar, C., Haziza, D., Williamson, M., Pino, J., and Dupoux, E. (2021).  VoxPopuli: A large-scale multilingual speech corpus for representation learning, semi-supervised learning and interpretation.  In Zong, C., Xia, F., Li, W., and Navigli, R., editors, Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 993–1003, Online. Association for Computational Linguistics.
  * Wang et al., [2020] Wang, C., Wu, A., and Pino, J. (2020).  Covost 2 and massively multilingual speech-to-text translation.
  * Wang et al., [2017] Wang, Y., Skerry-Ryan, R. J., Stanton, D., Wu, Y., Weiss, R. J., Jaitly, N., Yang, Z., Xiao, Y., Chen, Z., Bengio, S., Le, Q. V., Agiomyrgiannakis, Y., Clark, R. A. J., and Saurous, R. A. (2017).  Tacotron: Towards end-to-end speech synthesis.  In Interspeech.
  * Zhang et al., [2020] Zhang, C., Tan, X., Ren, Y., Qin, T., Zhang, K., and Liu, T.-Y. (2020).  Uwspeech: Speech to speech translation for unwritten languages.

## Supplementary Material

Appendix Sections | Contents
---|---
Appendix A |  | Qualitative Examples of Translated Speech
---
Appendix B |  |  Architecture and Hyper-parameters of Diffusion Model for DiffNorm
---
Appendix C |  | Architecture and Hyper-parameters of Non-autoregressive Models
---
Appendix D |  | Unit-to-Speech Synthesis
---
Appendix E |  | Limitations
---
Appendix F |  | Full Results for Classifier-free Guidance
---
Appendix G |  | Experiments Results Table
---

##  Appendix A Qualitative Examples

From § 5.2 we demonstrate the effectiveness of DiffNorm and classifier-free
guidance by evaluating ASR-BLEU scores. In this section, we provide example
transcriptions of generated Spanish and French speech in Table 5 and Table 6.
Compared to the vanilla CMLM model that has erroneous or incomplete
translation, our method results in higher-quality translation.

Model |  Generated Translation |  English Version
---|---|---
Reference |  este trabajo fue posible en cooperación con otros como alphonse milne-edwards y leon vaillant. |  This work was made possible in cooperation with others such as Alphonse Milne-Edwards and leon vaillant
CMLM |  esta obra fue posible para lugar con otros como alphon y leol vigelante |  This work was possible to place with others such as Alphon and Leol Vigelante
CMLM + CG |  este hora fue hizo por la cooperación con otros con walphonxe mill y leon fallent |  This time was made by cooperation with others with Walphonxe Mill and Leon Fallent
CMLM + DiffNorm |  esta obra fue posible la cooperación con otros como al von milo edwad y leon vallente |  This work was possible through cooperation with others such as Al von Milo Edwad and Leon valente
CMLM + DiffNorm + CG |  este trabajo fue posible por la cooperación con otros como alphonso edos y león valente |  This work was possible through the cooperation with others such as Alphonso Edos and León Valente
Table 5: Example transcription of translated Spanish speech from different systems. Model |  Generated Translation |  English Version
---|---|---
Reference |  mais la principauté est envahie et le prince-évêque joseph-clément de bavière doit s’exiler |  but the principality is invaded and the prince-bishop Joseph-Clement of Bavaria must go into exile
CMLM |  mais la précicipauté et le prince joseph clémont de bavier est fisé en exil |  but the precicipality and prince joseph clemont of bavier is exiled
CMLM + CG |  mais la principauté est lebséve josefh crémend de bavière et contré en exil |  but the principality is lebséve josefh cremend of bavaria and countered in exile
CMLM + DiffNorm |  la princepôuté est envahie et le prince-évêque joser clémant de bavière est contr |  the princepôuté is invaded and the prince-bishop joser clemant of bavaria is opposed
CMLM + DiffNorm + CG |  mais la principauté est envahie et le prend s évêque joseph de bavier et borgé en exile |  but the principality is invaded and takes bishop joseph of bavaria and borgé into exile
Table 6: Example transcription of translated French speech from different
systems.

##  Appendix B Diffusion Model Hyperparameters

###  B.1 Variational Auto-Encoder

Architecture: Our Auto-encoder consists of an encoder, a decoder, and a
language modeling head. The encoder architecture follows WaveNet [50] that
combines Convolutional Neural Networks [29] with residual connections. In
practice, we use 2 stacks of WaveNet Residual Blocks, where each block has 3
layers. Our decoder uses the same architecture as the encoder to revert the
latent dimension back to the original dimension. Additionally, we apply a
Transformer [52] encoder on top of the decoded feature to further enhance
model capacity. The Transformer-encoder has the same configuration as the
vanilla Transformer-base model (6 layers, 8 attention heads, etc.). Lastly, to
decode speech units from the feature, we use a language modeling head which is
parameterized by a feedforward network that converts feature dimension
($H=768$) into vocabulary size ($V=1000$).

Training: To train the VAE model, as described in § 3.1, we use a combination
of three objectives

$\mathcal{L}=\lambda_{1}\mathcal{L}_{\text{recon}}+\lambda_{2}\mathcal{L}_{\text{nll}}+\lambda_{3}\mathcal{L}_{\text{kl}}$
(8)

where $\lambda_{1}=100,\lambda_{2}=1,\lambda_{3}=0.001$. Now we provide more
details about $\mathcal{L}_{\text{kl}}$: in practice, we follow [26] to model
the latent $\bm{z}$ by estimating its mean $\bm{\mu}$ and variance
$\bm{\sigma}$ using the encoder. With the re-parameterization trick, the
latent is sampled as $\bm{z}=\bm{\mu}+\bm{\sigma}\cdot\bm{\epsilon}$ where
$\bm{\epsilon}\sim\mathcal{N}(0,\mathbf{I})$. By constraining the mean and
variance to follow a Gaussian prior, Kingma and Welling, [26] showed that

$\mathcal{L}_{\text{kl}}=\frac{1}{2}\sum_{j=1}^{J}(1+\log((\sigma_{j})^{2})-(\mu_{j})^{2}-(\sigma_{j})^{2})$
(9)

We implement our VAE model on Fairseq [35]. For optimization, we use the Adam
[25] optimizer with betas $(0.9,0.98)$ and we apply gradient clipping by
setting –clip-norm=2.0. During training, we apply dropout with a probability
of 0.1. We train the VAE model using a learning rate of 5e-4 with distributed
data-parallel (DDP) on 4 A100 GPUs, where we set the maximum batch token to be
15000.

###  B.2 Latent Diffusion Model

Architecture: We modified DiT [36] to design a Transformer-based architecture
for noise estimation. Since our input latent is a sequence of encoded speech
feature $\bm{z}\in\mathbb{R}^{M\times Z}$, we first apply a 1D Convolution to
convert the latent to our model’s hidden dimension (set to 512). Then we also
encode the timestep with a learnable Sinusoidal Positional Embedding [52] to
obtain the time embedding.

Subsequently, we feed the latent and time embedding to our modified WaveNet
Block, which applies an affine transformation of the latent using the time
embedding, similar to the "Scale and Shift" operation in DiT’s adaptive layer
norm (adaLN) block. For our diffusion model, we use 4 stacks of such modified
WaveNet Block, each containing 8 layers. The output feature from WaveNet is
then passed through a Transformer-Encoder (12 layers with 8 attention heads)
to further enhance model capacity. Lastly, a projection layer parameterized by
the feedforward network is used to compress the transformed feature to the
latent dimension, which is then used for noise estimation in equation 3.

Training: As described in § 3.2, we train the diffusion model with a multitask
objective:

$\mathcal{L}=\gamma_{1}\mathcal{L}_{\text{noise}}+\gamma_{2}\mathcal{L}_{\text{recon}}+\gamma_{3}\mathcal{L}_{\text{nll}}$
(10)

and we empirically select $\gamma_{1}=1,\gamma_{2}=0.25,\gamma_{3}=0.005$. For
optimization, our latent diffusion model has the same setting as our VAE model
(with Adam optimizer, clip-norm equals 2.0, and dropout with 0.1 probability).
We train the diffusion model using a learning rate of 1e-4 with DDP on 4 A100
GPUs, where we set the maximum batch token to be 12000. The model is warmup-ed
by 10000 steps.

##  Appendix C Non-autoregressive Transformer Details

###  C.1 Background on Non-Autoregressive Transformers

Non-autoregressive Transformers [11, 12, 9, inter alia.] is a family of models
that transduce sequences. In this work, we follow the formalization from the
widely-used NAT: Conditional Masked Language Model [9, CMLM]. CMLM consists of
an encoder $\phi_{\rm enc}$ that represents the input as high-dimensional
features and a decoder $\phi_{\rm dec}$ that generates tokens conditioning on
the encoded features. Different from auto-regressive Transformers [52], CMLM’s
decoder does not apply causal masking and is trained with an unmasking
objective.

CMLM Training Following formulation from § 2, given the input speech
$\bm{x}=(x_{1},\cdots,x_{N})$ and target speech units
$\bm{y}=(y_{1},\cdots,y_{M})$, CMLM first mask the target units into
$\hat{\bm{y}}$. The total amount of masked token is uniformed sampled:
$n\sim\mathcal{U}[1,M]$; subsequently, $n$ of $M$ tokens from the target units
$\bm{y}$ are randomly masked to form noisy target units $\hat{\bm{y}}$. Then,
the decoder generates a distribution over vocabulary conditioning on the
encoder representation and noisy target units
$\bm{v}=f(\hat{\bm{y}}|\bm{g};\phi_{\rm dec})$ where
$\bm{g}=f(\bm{x};\phi_{\rm enc})$ is the encoder representation. Cross-entropy
loss is then computed for the masked positions to update model parameters.
CMLM also trains a length predictor that estimates the output length given
input sequences and we refer readers to [20] for more details.

CMLM Mask-Predict: For CMLM inference, an iterative decoding scheme is used
through unmasking speech units and re-masking low-confident positions. Given
the input sequence $\bm{x}$, CMLM first predicts the length of output sequence
$M$ and initialize a sequence of $M$ masked tokens
$\hat{\bm{y}}_{0}=([\text{mask}]_{1},\cdots,[\text{mask}]_{M})$. Given the
total number of iterations $T$ and current iteration $t\in[1,T]$, CMLM decodes
tokens across all positions and also computes their log-probabilities:

$y_{i}^{t}=\underset{w}{\text{argmax}}\mathbbm{P}(y_{i}=w|\bm{x},\hat{\bm{y}}_{t-1};\phi_{\rm
dec})\;\;\;\;p_{i}^{t}=\log\mathbbm{P}(y_{i}^{t}|\bm{x},\hat{\bm{y}}_{t-1};\phi_{\rm
dec})$ (11)

From the generated tokens $\hat{\bm{y}}_{t}=(y_{1}^{t},\cdots,y_{M}^{t})$, we
replace $k=M\cdot\frac{T-t}{T}$ tokens with the lowest probabilities
$p_{i}^{t}$ with mask tokens. This re-masked sequence $\hat{\bm{y}}_{t}$ is
then inputted into the CMLM for further decoding iterations. This process
continues until the final step $t=T$, at which point no tokens are re-masked.

###  C.2 Architecture

We follow the same architecture and implementation from Huang et al., [20],
where a Conformer-based encoder is used to obtain representation from the
source speech and a Transformer-decoder (with no causal masking) is used to
generate units. The Conformer-encoder has 12 layers with 512 hidden dimensions
and 9 attention heads. The encoder also contains a CNN-based sub-sampler that
has an effective stride size of 320 (i.e., it reduces the length of speech
input by 320$\times$). The Transformer-decoder has 6 layers with 8 attention
heads and uses a fixed positional embedding for speech units. When classifier-
free guidance is used, we randomly dropout encoding from the Conformer model
with a probability of 15% as described in § 4.

###  C.3 Hyper-parameter for Model Training

We train the model with a learning rate of 5e-4, using 10000 warmup steps We
apply Adam optimizer with betas $(0.9,0.98)$ and we clip the gradient by
setting –clip-norm=10. We apply dropout with 0.1 probability and we use label
smoothing of ratio 0.2 when computing the NLL loss. We train the model with
DDP on 4 A100 GPUs, with a maximum batch tokens of 40,000.

##  Appendix D Unit-to-Speech Synthesis

We follow prior work [31, 37, 39] to convert predicted units (from S2UT model)
into speech waveforms. Specifically, given units $\bm{y}$, we use the HifiGAN-
based unit-vocoder from [37] to synthesize speech. The unit-vocoder is trained
using a generator $\mathcal{G}$ and discriminator network $\mathcal{D}$.
During training, the generator is updated with reconstruction loss based on
the mel-spectrogram of the true and synthesized waveforms. Additionally,
adversarial loss and feature-matching loss are added to enhance the fidelity
of generated speech. To train the discriminator, Polyak et al., [37] follows
the minmax objective from GAN [10]. After training, the discriminator is
discarded and only the generator is used to synthesize waveforms.

We follow [31] to use a unit-vocoder that is also trained with a duration
predictor so that it can synthesize speech with reduced speech units. For more
details, we refer readers to the discussion from the original paper [31, 39],
as well as the open-sourced repository:
github.com/facebookresearch/fairseq/examples/speech_to_speech

##  Appendix E Limitations and Broader Impacts

Limitations: The major limitation of the work is the data pre-processing cost
from DiffNorm, as our method requires inferencing diffusion model on all
training samples to obtain better speech units. With the CVSS benchmark, both
En-Es and En-Fr have less than 1M pairs, which can be processed easily with
our resource (4 A100 GPUs). However, when the dataset scales up, the corpus
distillation cost could be large.

Broader Impacts: Our system helps bridge the communication gap between users
of different languages. However, as our system involves speech synthesis, it
also has a risk of being used to mimic a particular speaker.

##  Appendix F Ablation on Classifier-free Guidance

In this section, we compare non-autoregressive transformers trained with and
without classifier-free guidance (CG). We train CMLM models with classifier-
free guidance and evaluate them with $5,10,15$ iterations of decoding. From
Table 7, we observe that CG improves the quality of translation whether the
model is trained on original or normalized speech units. We find CG brings
more improvement on the original units than the normalized units. This happens
because normalized speech units are more conformed and already result in a
large improvement in their translation quality, making the regularization
effect from CG less obvious. Nevertheless, the best-performing system is
achieved with both DiffNorm and CG.

Comparing different hyperparameters $w$, we find a small value like $w=0.5$ or
$w=1$ brings the most improvement empirically, and such improvements are more
noticeable when the number of decoding iterations is larger. For example,
comparing the results under 5 and 15 iterations, we find $w=0$ gives better
results when the number of iterations is small while $w=0.5$ obtains the best
performance with 15 iterations.

Model | En-Es | En-Fr
---|---|---
| 5 | 10 | 15 | 5 | 10 | 15
CMLM | 9.45 | 11.85 | 12.58 | 12.73 | 14.78 | 15.62
CMLM + CG (w=0) | 12.22 | 15.58 | 16.62 | 12.13 | 15.23 | 16.65
CMLM + CG (w=0.5) | 11.37 | 15.51 | 17.06 | 11.63 | 15.39 | 16.89
CMLM + CG (w=1) | 11.27 | 15.4 | 16.99 | 11.52 | 15.44 | 16.65
CMLM + CG (w=2) | 10.98 | 14.97 | 16.57 | 11.12 | 15.04 | 16.50
CMLM + CG (w=3) | 10.63 | 14.7 | 16.22 | 10.80 | 14.71 | 16.17
CMLM + DiffNorm | 14.84 | 17.75 | 18.96 | 14.03 | 16.33 | 17.27
CMLM + DiffNorm + CG (w=0) | 14.69 | 17.57 | 18.63 | 13.03 | 16.05 | 17.13
CMLM + DiffNorm + CG (w=0.5) | 15.02 | 18.24 | 19.49 | 13.48 | 16.53 | 17.54
CMLM + DiffNorm + CG (w=1) | 14.77 | 18.18 | 19.37 | 13.29 | 16.27 | 17.48
CMLM + DiffNorm + CG (w=2) | 14.03 | 17.65 | 18.9 | 12.8 | 15.97 | 17.15
CMLM + DiffNorm + CG (w=3) | 13.45 | 17.09 | 18.5 | 12.23 | 15.44 | 16.52
Table 7: Speech-to-speech translation performance of CMLM models with
different CG hyperparameters.

##  Appendix G Experiment Result Tables

Model | Number of Iterations
---|---
| 3 | 5 | 7 | 19 | 15
CMLM | 7.02 | 9.45 | 10.66 | 11.85 | 12.58
\+ CG | 7.17 | 11.37 | 13.59 | 15.51 | 17.06
\+ DiffNorm | 11.71 | 14.84 | 16.42 | 17.75 | 18.96
\+ DiffNorm + CG | 10.96 | 15.02 | 16.83 | 18.24 | 19.49
Table 8: Experimental Results of different En-Es speech-to-unit translation systems. Model | Number of Iterations
---|---
| 3 | 5 | 7 | 19 | 15
CMLM | 10.1 | 12.73 | 13.64 | 14.78 | 15.62
CMLM + CG | 7.33 | 11.63 | 13.71 | 15.39 | 16.89
CMLM + DiffNorm | 11.6 | 14.03 | 15.24 | 16.33 | 17.27
CMLM + DiffNorm + CG | 9.14 | 13.48 | 15.24 | 16.53 | 17.54
Table 9: Experimental Results of different En-Fr speech-to-unit translation
systems.