Christophe Veaux1, Ranniery Maia1, Spyridoula Papandreou1

# The DeepZen Speech Synthesis System for Blizzard Challenge 2023

###### Abstract

This paper describes the DeepZen text to speech (TTS) system for Blizzard
Challenge 2023. The goal of this challenge is to synthesise natural and high-
quality speech in French, from a large monospeaker dataset (hub task) and from
a smaller dataset by speaker adaptation (spoke task). We participated to both
tasks with the same model architecture. Our approach has been to use an auto-
regressive model, which retains an advantage for generating natural sounding
speech but to improve prosodic control in several ways. Similarly to non-
attentive Tacotron, the model uses a duration predictor and gaussian
upsampling at inference, but with a simpler unsupervised training. We also
model the speaking style at both sentence and word levels by extracting global
and local style tokens from the reference speech. At inference, the global and
local style tokens are predicted from a BERT model run on text. This BERT
model is also used to predict specific pronunciation features like schwa
elision and optional liaisons. Finally, a modified version of HifiGAN trained
on a large public dataset and fine-tuned on the target voices is used to
generate speech waveform. Our team is identified as O in the the Blizzard
evaluation and MUSHRA test results show that our system performs second ex
aequo in both hub task (median score of 0.75) and spoke task (median score of
0.68), over 18 and 14 participants, respectively.

Index Terms: French TTS, multi-level prosody, fine-grained prosody, contextual
prosody, non-attentive Tacotron

##  1 Introduction

The Blizzard Challenge111https://www.synsig.org/index.php/Blizzard_Challenge
is a yearly occurring shared task aiming at advancing TTS by comparing and
understanding different approaches, with extensive evaluation by human
listeners. This year’s challenge consists of a “hub” task and a “spoke” task.
For both tasks, participants are required to build a voice for synthesising
French texts.

  * •

Hub task: to build a voice from about 50h of data from a French female
speaker, training the model using only publicly available data.

  * •

Spoke task: to build a voice from about 2h of data from another French female
speaker, by using speaker adaptation of a pre-trained model. There are no
requirements on the use of external data for training the initial model.

We participated to both tasks with a model architecture derived from our
current production models. Our design choices were led by two aims: achieving
the best possible acoustic and prosodic naturalness, while allowing prosodic
control at both global (utterance) and local (word) levels.

The naturalness of speech produced by TTS systems has significantly improved
with neural-based techniques. A common approach consists of a two-stage
architecture where an acoustic model predicts a mel-spectrogram from the
linguistic input (phonemes with punctuation marks) and a neural vocoder
synthesises speech waveform from it. In this paper we mainly focus on the
acoustic model. Autoregressive acoustic models using an attention-based
encoder-decoder, such as Tacotron2 [1], achieve high naturalness but suffer
from robustness issues and lack of fine-grained prosodic control. Non-
autoregressive acoustic models such as FastSpeech2 [2] or Fastpitch [3] offer
greater robustness and better prosodic control as they use duration and pitch
predictors. However, auto-regressive decoders still have an intrinsic
advantage for generating consistent prosody at the frame level and thus more
natural sounding speech. Therefore we adopt an architecture similar to non-
attentive Tacotron (NAT) [4] with a duration predictor and gaussian upsampling
but modify it to allow simpler unsupervised training. More specifically, we
use a simple gaussian attention with strict monotonic property, similar to
[5], to train the duration predictor and gaussian upsampling. At inference,
the attention module is bypassed and the model relies only on the duration
predictor and gaussian upsampling.

The two-stage TTS architecture using mel-spectrogram as intermediate
representation introduces a training/inference gap since the vocoder is
exposed to ground-truth mel-spectrograms at training but predicted mel-
spectrograms during inference. Recently, single-stage end-to-end architectures
[6, 7, 8] have been introduced to generate speech waveform directly from the
linguistic input and avoid the training/inference gap. However, this gap can
also be greatly reduced by fine-tuning the vocoder on the synthesised mel-
spectrograms and this is the approach followed in this paper.

As predicting prosody from text is a one-to-many problem, there has been
considerable work in learning latent style representations for better prosodic
modeling and controllability. Initial methods [9, 10] obtained latent
representation at the sentence level by encoding the target mel-spectrogram
into a fixed-length embedding. As sentence-level representation is not enough
to model fine-grained prosodic variability, more recent works use the
alignment information to aggregate mel-spectrogram or other prosodic features
at the phone or word levels and extract localised prosody embeddings [11, 12].
It is difficult however to learn disentangled representations in a fully
unsupervised manner [13]. One solution is to estimate the finer-level latent
conditioned on the coarser-level as in [12]. We follow a different approach
which uses distinct features for the sentence and word levels, with the word-
level feature being normalised over the sentence level. At the sentence level,
a reference encoder extracts a global embedding from the mel-spectrogram as in
[9]. At the word level, we consider the pitch contour mean- and variance-
normalised over the sentence and decomposed into a pitch spectrogram by a
continuous wavelet transform (CWT) [14]. A secondary reference encoder is used
to extract a sequence of local embeddings from this pitch spectrogram, at the
onsets of the words as given by the alignment. In this way, the local
reference embeddings encode information such as pitch prominence, whereas the
global reference embedding encodes information such channel conditions, vocal
quality, global pace, pitch register and excursion. Similarly to [9], the
global and local reference embeddings are fed to separate multi-head
attentions with trainable codebooks of style tokens as keys and values in
order to produce global and local style embeddings, respectively. While
reference mel-spectrogram and normalised pitch contour are used at training to
extract global and local style embeddings, we need to select the appropriate
style embeddings from text only at inference. Following an approach inspired
by [15], we use a BERT model to predict the global and local style embeddings
from the input text. Additionally, we use pre-trained speaker embedding to
model speaker information.

Finally, our text frontend is able to generate variants of pronunciations. In
French, most of these pronunciation variants correspond to schwa elision and
optional liaisons. At training, we use a lattice-based forced-alignment that
selects the correct pronunciation variant from the recordings. The forced-
alignment is also used to classify the pauses into broad categories (short,
medium, long), which are transcribed in the linguistic input as "categorical
pause" markers. We found out that these categorical pause markers helps to
stabilise the attention mechanism used to train the duration predictor and
gaussian upsampling222Because of this, the training of our model may be seen
as semi-supervised rather than fully unsupervised.. At inference, the same
BERT model as above is used to predict the pronunciations variants (schwa
elision and optional liaisons) and the categorical pauses. The key elements of
our approach can be summarised as:

  * •

Unsupervised NAT training with single gaussian attention.

  * •

Prosodic variability modeled by global and local style tokens, together with
pre-trained speaker embeddings.

  * •

BERT model used to predict the global and local style embeddings, as well as
pronunciation variants and categorical pauses.

##  2 Acoustic Model

###  2.1 Overview

The schematic architecture of the acoustic model is shown in Figure 1 for
training (a) and inference (b). At training, a gaussian attention learns the
alignment $A_{attn}$ between the target mel-spectrogram and the encoder
outputs. From this alignment, we estimate the durations of each encoder
outputs. These durations and the alignment are fed as targets for the duration
predictor and the gaussian upsampling, respectively. The alignment is also
used to sample the local reference embeddings at the words onsets, in order to
create word-level local style embeddings LSE. These local style embeddings are
concatenated with the global style embedding GSE and the speaker embedding at
the output of the encoder. Since the decoder is autoregressive, estimating the
LSE at the start of the words allows to use them to condition the generation
of each word. The architecture of the encoder and decoder follows that of
Tacotron [1] except that we use layer normalisation instead of batch
normalisation in the PostNet and add a projection layer after it. We detail
the key components of the model in the following subsections.

(a) Training

(b) Inference

Figure 1: Abstract diagram of the acoustic model at training and inference

###  2.2 Attention-driven training

Most TTS systems with explicit duration model are trained in supervised
fashion with "ground-truth" durations extracted from external sources such as
forced-alignment or attention-based TTS models. However, an unsupervised
approach is more practical and consistent because the durations that we need
to model are those of the encoder outputs, and external alignments not
necessarily optimally represent those. An unsupervised learning objective was
proposed in [16] that uses the forward-sum algorithm to maximise the
likelihood over all possible monotonic alignments. This objective makes the
assumption of independence between alignment steps which is valid for parallel
TTS models but doesn't hold for auto-regressive models. Therefore, in the
auto-regressive case, it reduces to an additional loss term encouraging
monotonicity but not enforcing it. Instead, we propose here a modified
gaussian attention to ensure locality, monotonicity and completeness of the
alignment. Similarly to [5], we model the alignment at step $i$ with a single
gaussian:

$\displaystyle\alpha_{i,j}=\exp(-\frac{(j-\mu_{i})^{2}}{2\sigma_{i}^{2}})$ (1)
$\displaystyle\mu_{i}=\mu_{i-1}+\Delta_{i}$ (2)

where the parameters $(\sigma_{i},\Delta_{i})$ are non-linear transforms of
intermediate parameters $(\hat{\sigma_{i}},\hat{\Delta_{i}})$ calculated from
the decoder hidden state $h_{i}$:

$\displaystyle(\hat{\sigma_{i}},\hat{\Delta_{i}})=V\,\mathrm{Relu}(Wh_{i}+b)$
(3) $\displaystyle\sigma_{i}=\mathrm{Softplus}(\hat{\sigma_{i}})$ (4)
$\displaystyle\Delta_{i}=\mathrm{Sigmoid}(\hat{\Delta_{i}})$ (5)

The sigmoid transform constrains the alignment to be monotonic and non-
skipping, which is a desirable property in order to derive encoder outputs
durations. The mean $\mu_{i}$ gives the location of the attention at decoding
step $i$ and can be mapped to the position of the words. Furthermore, using a
purely location-based attention allows to dynamically concatenate the LSE
associated with a given word to the encoder outputs once we start to attend
this word, since the attention mechanism doesn't use the encoder outputs as
precomputed keys.

###  2.3 Duration prediction, Upsampling, Positional encoding

The architecture of the duration predictor and gaussian upsampling follows [4]
with some simplifications. The duration predictor passes the encoder
outputs333In this section we denote as encoder outputs, the concatenation of
the encoder outputs with the GSE, LSE and speaker embedding. through a single
bi-directional LSTM layer followed by a projection layer to predict the log-
durations of each encoder output. These are compared to the log-durations
derived from the attention-based alignment $A_{attn}$ via the L2 loss term
$L_{dur}$. For the gaussian upsampling, the predicted durations are passed
through a 1-D convolutional layer to project them to an embedding space. The
resulting duration embeddings are concatenated with the encoder outputs and
passed through a projection layer and a SoftPlus activation to predict the
range parameter $\sigma$ for each encoder output [4]. The soft alignment
$A_{gauss\\_up}$ produced by the gaussian upsampling is compared to the
attention-based alignment $A_{attn}$ via the KL-divergence loss term
$L_{align}$. The minimisation of the losses $L_{dur}$ and $L_{align}$ only
impacts the weights of the duration predictor and gaussian upsampling networks
since we stop the gradient flow to the encoder outputs and to the gaussian
attention network. Finally, slightly differing from the NAT implementation
[4], we encode the fractional progression of the alignment $\mu_{i}$ with
respect to the current encoder output via a Transformer-style sinusoidal
positional embedding concatenated at the input of the decoder. We found this
positional embedding to be important for the stability of the inference mode,
when the duration predictor and gaussian upsampling are used instead of the
attention module, and where $\mu_{i}$ is derived as the centroid at decoding
step $i$ of the soft alignment $A_{gauss\\_up}$ as shown in Figure 1 (b).

###  2.4 Global and Local Style Layers

Two parallel style layers extract the global and local style embeddings, GSE
and LSE, from the reference speech. As in [9], the embedding spaces are
learned as a convex combination of trainable style tokens. The architecture of
the global style layer follows [9] with a reference encoder consisting of a
2-D convolutional stack followed by a GRU network whose last state is used as
query for the global style tokens layer to generate the global style embedding
GSE. For the local style layer, the GRU network is replaced by a bi-
directional GRU and we consider the outputs at each time-step. Using the
alignment location $\mu_{i}$, we sample these outputs at the time-steps
corresponding to the words onsets to form a sequence of queries for the local
style tokens layer and generate the local style embeddings LSE. The input of
the local style layer is the continuous wavelet transform (CWT) of the pitch
contour normalised over the sentence. We use the Wavelet Prosody
toolkit444https://github.com/asuni/wavelet_prosody_toolkit to extract a
smoothed pitch contour, interpolated over unvoiced gaps, normalise it over the
sentence and apply the CWT. Finally, the speaker characteristics are encoded
via a separate embedding generated by the advanced_gru_network of [17] which
we trained on a combination of LibriTTS [18], VCTK [19] and DAPS [20]
datasets.

##  3 Style Prediction Network

At inference, a separate style prediction network conditioned on both text and
speaker information predicts the GSE at the sentence level, and the LSE as
well as some symbolic linguistic features at the word level. However, rather
than predicting directly the GSE and LSE, we found beneficial to predict the
combination weights over the global and local style tokens, from which we
derive the predicted GSE and LSE. This ensures that the predicted embeddings
are within the embedding spaces learned by the global and local style layers.
A pre-trained BERT model [21] extracts contextualised word-piece embeddings
from the text and these are concatenated with the speaker embedding. Following
an architecture similar to [15], the resulting embeddings are passed through a
bidirectional LSTM and the first and last hidden states are concatenated to
get the GSE combination weights. For the LSE combination weights and other
word-level features, the first word-piece embedding of each words,
concatenated with the speaker embedding, are simply passed through an affine
layer, similarly to [22].

To train the GSE and LSE prediction layers, we use the global and local style
layers of Figure 1 (a) to extract the GSE and LSE over all training sentences.
As in [23], we use the cross-entropy loss between predicted combination
weights and target attention weights for the GSE and LSE.

The word-level linguistic features predicted by the style network are the
pronunciation variants (schwa elision, optional liaisons) and a 5-way
classification of the normalised pause durations555For this categorisation,
the pause durations extracted by the forced-alignment are normalised by the
average speaking rate of the speaker. denoted as "categorical pauses". When
training the acoustic model, these categorical pauses are estimated by forced
alignment and inserted in the phonemes and punctuations sequence at the input
of the encoder to help stabilise the attention. The forced alignment is also
used to select the correct pronunciation variants over the training sentences.
For training the symbolic features prediction layer, we use the cross-entropy
loss between the annotations given by the forced alignment and the predicted
symbolic outputs. However, since the categorical pauses are naturally ordered,
we add a regularisation term to their loss, based on the squared Earth Mover's
Distance [24].

Finally, the losses from the GSE, LSE and symbolic features prediction layers
are combined in an overall loss used to fine-tune the BERT model. The
confusion matrices of the predicted symbolic features for the French female
speaker of the hub task are shown in Figure 2, with full/reduced and 1/0
denoting the realisation/omission of the schwa and optional liaison,
respectively. The normalised pauses durations are classified in an ordinal
fashion from "no pause" (-) to "extra-long" (#).

(a) schwa elision

(b) optional liaison

(c) categorical pauses

Figure 2: Confusion matrices of the predicted symbolic features (ground-truth
value is on ordinate and predicted on abscissa)

##  4 Data Processing, Vocoder and Training

###  4.1 Text processing

We leverage our internal text frontend based on Spacy [25] to convert the
French texts into sequences of phonemes. We added to the Spacy pipeline our
own custom components to perform rule-based text normalisation and basic
homograph disambiguation, using part-of-speech (POS) contexts, as well as
phonetisation. For the phonetic transcription, we first perform a lexicon
lookup using a large lexicon with phonetic variants derived from [26]. Then
out-of-vocabulary words are converted to phonemes with an attention-based
sequence-to-sequence G2P trained on that same lexicon. Finally, a set of post-
lexical rules are applied to enforce mandatory liaisons. When used to process
training data, this pipeline allows to generate phonetic variants associated
mainly to schwa elision and optional liaisons. These variants are used by the
Montreal-Forced Aligner (MFA) [27] to decode the correct phonetic
transcription. At synthesis, the style prediction network is first run on text
to decide the phonetic variants depending on the speaker as well as to predict
the categorical pauses which are inserted in the phonetic sequence generated
by the text pipeline.

###  4.2 Vocoder model

We use a modified version of HiFiGAN [28] that we pre-trained on the same
datasets (LibriTTS, VCTK, DAPS) than the speaker encoder. First, compared to
HiFiGAN-v1, we add one more stack of convolution layers and residual
connection to each ResBlock of the generator in order to capture longer
contexts. Second, in the multi-scale discriminator, we replace the average
pooling by max-pooling as it helps reducing metallic artefacts in unvoiced
segments. As for our mel-spectrogram, we use 128 bins ERB-scale frequency
wrapped spectrograms computed on 50 ms frame size with 10 ms hop size.

###  4.3 Training Strategy

The overall loss function for the acoustic model is

$\displaystyle L=L_{spec}+\lambda(L_{dur}+L_{align})$ (6)

where $L_{spec}$ is the combined L1 loss between ground-truth and predicted
mel-spectrograms before and after the PostNet. We use Adam optimiser with
batch size 32 and a learning rate of $10^{-3}$ exponentially decaying after
50K steps, $\lambda$ is set to 0 until 50K and then to 1.

The style prediction network is trained separately using the pre-trained
uncased Flaubert model [29]. This model is fine-tuned over the combination of
the GSE, LSE and symbolic features prediction losses. We use Adam optimiser
with a learning rate of $2\mathrm{e}{-5}$ and batch size 8.

Both models are trained on the following datasets:

  * •

Hub task: 50h from target French female speaker (NEB), "book" and "parl"
subsets of the Siwis dataset [30] (about 10h from another French female
speaker).

  * •

Spoke task: besides the hub task datasets, we use extra internal dataset of
about 15h (3 French female speakers).

For the spoke task, once the initial acoustic and style prediction models are
trained, we fine-tune them on the unseen target speaker data (AD). The vocoder
model is always fine-tuned to the target speakers for both hub and spoke
tasks. Finally, the embeddings dimensions for the acoustic model are given in
Table 1, otherwise stated similar to [1].

Table 1: Embedding dimensions for the acoustic model. speaker embeddings | 256
---|---
global / local style embedding | 256
global / local style tokens | 32
num global / local style tokens | 10
positional embedding | 32
duration embedding | 32
gaussian attention hidden size | 128

We train our acoustic model on a 32GB NVIDIA V100 GPU while HifiGAN and BERT
fine-tuning are done on a 16GB V100 GPU. Inference is performed on 2.70GHz
Intel Xeon Platinum CPU, where the end-to-end synthesis (including the text
frontend) runs 3.8 times faster than real-time in average.

##  5 Blizzard Evaluation

A total of 18 teams participated in the hub task, and 14 in the spoke task.
Our system is identified with letter O. The letter A denotes the natural
speech from the original speaker. Two baseline systems were added, a
FastSpeech2 + HiFi-GAN benchmark denoted as BF, and a grapheme-based Tacotron2
+ HiFi-GAN benchmark denoted as BT.

The initial evaluation for the hub task includes: naturalness test with MOS
(mean opinion score), similarity test with SMOS (similarity mean opinion
score), SUS intelligibility test with WER (word error rate), and homographs
pronunciation, which is not included here as we focus on the acoustic model.
For the spoke task, the initial evaluation comprises naturalness MOS and
similarity SMOS. All the systems were evaluated in these initial tests,
however the baseline models BF and BT scored very low, which somewhat
compresses the scale for the best models. Therefore, a subset of the best 3-4
models was selected to perform a MUSHRA quality test for both tasks. Our
system was selected for these MUSHRA tests and ranked 2nd ex aequo for both
hub and spoke tasks.

###  5.1 Hub task

The Table 2 reports the initial evaluation scores of our system for the hub
task, compared to the original recording. There were 361 to 228 validated
participants for these initial tests which evaluated all 18 systems. The
naturalness MOS of our system places it among the 3 best systems and close to
natural recordings. In terms of speaker similarity, our system scored in the
middle-range, on par with 10 other of the evaluated systems. However, the
standard deviation of the similarity MOS is relatively high even for the
natural recordings, which makes it difficult to interpret. Further assessment
would be needed to understand if the difference is due to prosody
(prominences, pauses) or speaker quality. The result of the intelligibility
test shows that our system is robust to pronunciations errors and validates
the unsupervised training approach of our non-attentive acoustic model.
Finally, the results of the second evaluation round (MUSHRA test) are shown on
Figure 3. Three systems were evaluated (F, O, I) and our system ranked second,
tied with the system I (no statistically significant differences, as shown in
Blizzard detailed evaluations).

Table 2: Initial scores for our system compared to (original). Test (Hub task) | Median | Mean | Sd | Metric
---|---|---|---|---
Naturalness | 4 (5) | 4.2 (4.4) | 0.85 (0.8) | MOS
Similarity | 3 (4) | 2.9 (3.4) | 1.27 (1.31) | MOS
SUS Intelligibility | 0.0 | 0.11 | 0.2 | WER
Figure 3: MUSHRA scores (all participants). Our System is O.

###  5.2 Spoke task

The Table 3 reports the initial evaluation scores of our system for the spoke
task, compared to the original recording. There were 286 to 282 validated
participants for these tests which evaluated all of the 14 systems for the
spoke task. The naturalness MOS of our system places it among the 2 best
systems, almost on par with natural recordings, although the "compressed
scale" effect due low performing baseline should be accounted here. Like for
the hub task, our system scored in the middle-range in terms of similarity
scores, on par with 7 other systems but with a high variance (also noted for
the natural speech). Our system was selected among the 4 bests models for the
second evaluation test (MUSHRA) whose results are shown on Figure 4. Although
our system appears third on the plot, there is no statistically significant
difference with the second system L, as shown in the Blizzard detailed
evaluations. Among the factors possibly limiting our system for the spoke
task, it could be noted that there is a strong speaker imbalance in the
dataset used to train the initial model (before fine-tuning). Also our speaker
encoder was trained on 1400 speakers, which might not be enough to learn an
exhaustive speaker embedding space.

Table 3: Initial scores for our system compared to (original). Test (Spoke task) | Median | Mean | Sd | Metric
---|---|---|---|---
Naturalness | 5 (5) | 4.4 (4.5) | 0.72 (0.77) | MOS
Similarity | 4 (4) | 3.4 (4) | 1.31 (1.21) | MOS
Figure 4: MUSHRA scores (all participants). Our System is O.

##  6 Conclusions

In this paper, we have presented our proposed two-stage TTS system for the
Blizzard Challenge 2023. Our approach to achieve the best possible acoustic
and prosodic naturalness relies on several key aspects. First, we choose an
auto-regressive model for its high naturalness but use a duration predictor
and gaussian upsampling at inference for better robustness. We devise a simple
unsupervised training using gaussian attention. Second, we model the prosodic
variability at both global and local level using a style token framework.
Finally we use a BERT model to predict the global and local style embeddings,
as well as pronunciation variants, conditioned on the speaker information. Our
system was shortlisted for the second round of the Blizzard evaluation and
performed second best in hub task and third best in spoke task. Further work
would be to improve speaker adaptation and style transfer capabilities,

## References

  * [1] J. Shen, R. Pang, R. J. Weiss, M. Schuster, N. Jaitly, Z. Yang, Z. Chen, Y. Zhang, Y. Wang, R. Skerry-Ryan, R. A. Saurous, Y. Agiomyrgiannakis, and Y. Wu, ``Natural tts synthesis by conditioning wavenet on mel spectrogram predictions,'' in _ICASSP 2018, IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , 2018.
  * [2] Y. Ren, C. Hu, X. Tan, T. Qin, S. Zhao, Z. Zhao, and T.-Y. Liu, ``Fastspeech 2: Fast and high-quality end-to-end text to speech,'' in _International Conference on Learning Representations (ICLR)_ , 2021.
  * [3] A. Łańcucki, ``Fastpitch: Parallel text-to-speech with pitch prediction,'' in _ICASSP 2021, IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , 2021.
  * [4] J. Shen, Y. Jia, M. Chrzanowski, Y. Zhang, I. Elias, H. Zen, and Y. Wu, ``Non-attentive tacotron: Robust and controllable neural tts synthesis including unsupervised duration modeling,'' in _International Conference on Learning Representations (ICLR)_ , 2021.
  * [5] Q. Tian, Z. Zhang, C. Liu, H. Lu, L. Chen, B. Wei, P. He, and S. Liu, ``Feathertts: Robust and efficient attention based neural tts,'' in _11th ISCA Speech Synthesis Workshop (SSW 11)_ , 2021.
  * [6] J. Kim, J. Kong, and J. Son, ``Conditional variational autoencoder with adversarial learning for end-to-end text-to-speech,'' in _International Conference on Machine Learning (ICML)_ , 2021.
  * [7] X. Tan, J. Chen, H. Liu, J. Cong, C. Zhang, Y. Liu, X. Wang, Y. Leng, Y. Yi, L. He, F. Soong, T. Qin, S. Zhao, and T.-Y. Liu, ``Naturalspeech: End-to-end text to speech synthesis with human-level quality,'' 2022.
  * [8] E. Casanova, J. Weber, C. Shulby, A. C. Junior, E. Gölge, and M. A. Ponti, ``Yourtts: Towards zero-shot multi-speaker tts and zero-shot voice conversion for everyone,'' in _International Conference on Machine Learning (ICML)_ , 2022.
  * [9] Y. Wang, D. Stanton, Y. Zhang, R. Skerry-Ryan, E. Battenberg, J. Shor, Y. Xiao, F. Ren, Y. Jia, and R. A. Saurous, ``Style tokens: Unsupervised style modeling, control and transfer in end-to-end speech synthesis,'' in _International Conference on Machine Learning (ICML)_ , 2018.
  * [10] Y.-J. Zhang, S. Pan, L. He, and Z.-H. Ling, ``Learning latent representations for style control and transfer in end-to-end speech synthesis,'' in _ICASSP 2019, IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , 2019.
  * [11] V. Klimkov, S. Ronanki, J. Rohnke, and T. Drugman, ``Fine-grained robust prosody transfer for single-speaker neural text-to-speech,'' in _Interspeech_ , 2019.
  * [12] G. Sun, Y. Zhang, R. J. Weiss, Y. Cao, H. Zen, and Y. Wu, ``Fully-hierarchical fine-grained prosody modeling for interpretable speech synthesis,'' in _ICASSP 2020, IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , 2020.
  * [13] F. Locatello, S. Bauer, M. Lucic, G. Rätsch, S. Gelly, B. Schölkopf, and O. Bachem, ``Challenging common assumptions in the unsupervised learning of disentangled representations,'' in _International Conference on Machine Learning (ICML)_ , 2019.
  * [14] A. Suni, J. Simko, D. Aalto, and M. Vainio, ``Hierarchical representation and estimation of prosody using continuous wavelet transform,'' _Computer Speech & Language_, vol. 45, 11 2016.
  * [15] S. Karlapati, A. Abbas, Z. Hodari, A. Moinet, A. Joly, P. Karanasou, and T. Drugman, ``Prosodic representation learning and contextual sampling for neural text-to-speech,'' _ICASSP 2021, IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , 2021.
  * [16] R. Badlani, A. Lancucki, K. J. Shih, R. Valle, W. Ping, and B. Catanzaro, ``One tts alignment to rule them all,'' _ICASSP 2022, IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , 2022.
  * [17] G. Ruggiero, E. Zovato, L. D. Caro, and V. Pollet, ``Voice cloning: a multi-speaker text-to-speech synthesis approach based on transfer learning,'' 2021\.
  * [18] H. Zen, V. Dang, R. Clark, Y. Zhang, R. J. Weiss, Y. Jia, Z. Chen, and Y. Wu, ``Libritts: A corpus derived from librispeech for text-to-speech,'' in _Interspeech_ , 2019.
  * [19] C. Veaux, J. Yamagishi, and K. MacDonald, ``Cstr vctk corpus: English multi-speaker corpus for cstr voice cloning toolkit,'' 2017.
  * [20] G. J. Mysore, ``Can we automatically transform speech recorded on common consumer devices in real-world environments into professional production quality speech?—a dataset, insights, and challenges,'' _IEEE Signal Processing Letters_ , vol. 22, no. 8, pp. 1006–1010, 2015.
  * [21] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, ``Bert: Pre-training of deep bidirectional transformers for language understanding,'' in _2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, (NAACL-HLT)_ , 2019.
  * [22] A. Talman, A. Suni, H. Celikkanat, S. Kakouros, J. Tiedemann, and M. Vainio, ``Predicting prosodic prominence from text with pre-trained contextualized word representations,'' in _Nordic Conference of Computational Linguistics_ , 2019.
  * [23] D. Stanton, Y. Wang, and R. Skerry-Ryan, ``Predicting expressive speaking style from text in end-to-end speech synthesis,'' in _2018 IEEE Spoken Language Technology Workshop (SLT)_ , 2018.
  * [24] L. Hou, C.-P. Yu, and D. Samaras, ``Squared earth mover's distance-based loss for training deep neural networks,'' in _Conference on Neural Information Processing Systems (NIPS)_ , 2017.
  * [25] M. Honnibal, I. Montani, S. Van Landeghem, and A. Boyd, ``spaCy: Industrial-strength Natural Language Processing in Python,'' 2020.
  * [26] B. New, ``Lexique 3: Une nouvelle base de données lexicales,'' _TALN Conference_ , pp. 892–900, 01 2006.
  * [27] M. McAuliffe, M. Socolof, S. Mihuc, M. Wagner, and M. Sonderegger, ``Montreal forced aligner: Trainable text-speech alignment using kaldi,'' in _Interspeech_ , 2017.
  * [28] J. Kong, J. Kim, and J. Bae, ``Hifi-gan: Generative adversarial networks for efficient and high fidelity speech synthesis,'' _Advances in Neural Information Processing Systems_ , vol. 33, 2020.
  * [29] H. Le, L. Vial, J. Frej, V. Segonne, M. Coavoux, B. Lecouteux, A. Allauzen, B. Crabbé, L. Besacier, and D. Schwab, ``Flaubert: Unsupervised language model pre-training for french,'' in _Language Resources and Evaluation Conference (LREC)_ , 2020.
  * [30] P.-E. Honnet, A. Lazaridis, P. N. Garner, and J. Yamagishi, ``The siwis french speech synthesis database: Design and recording of a high quality french database for speech synthesis,'' 2017.


1]Mozilla, Xiph.Org 2]Microsoft Jean-Marc<EMAIL_ADDRESS>Valin,
Terriberry, Vos

# High-Quality, Low-Delay Music Coding in the Opus Codec

###### Abstract

The IETF recently standardized the Opus codec as RFC6716. Opus targets a wide
range of real-time Internet applications by combining a linear prediction
coder with a transform coder. We describe the transform coder, with particular
attention to the psychoacoustic knowledge built into the format. The result
out-performs existing audio codecs that do not operate under real-time
constraints.

##  1 Introduction

In RFC 6716 [1] the IETF recently standardized Opus [2], a highly versatile
audio codec designed for interactive Internet applications. This means support
for speech and music, operating over a wide range of changing bitrates,
integration with the Real-Time Protocol (RTP), and good packet loss
concealment, with a low algorithmic delay.

Opus scales to delays as low as 5 ms, even lower than AAC-ELD (15 ms).
Applications such as network music performance require these ultra-low delays
[3]. Despite the low delay, Opus is competitive with high-delay codecs
designed for storage and streaming, such as Vorbis and the HE and LC variants
of AAC, as the evaluations in Section 6 show.

Opus supports

  * •

Bitrates from 6 kb/s to 510 kb/s,

  * •

Five audio bandwidths, from narrowband (8 kHz) to fullband (48 kHz),

  * •

Frame sizes from 2.5 ms to 60 ms,

  * •

Speech and music, and

  * •

Mono and stereo coupling.

In-band signaling can dynamically change all of the above, with no switching
artifacts.

We created the Opus codec from two core technologies: Skype’s SILK [4] codec,
based on linear prediction, and Xiph.Org’s CELT codec, based on the Modified
Discrete Cosine Transform (MDCT). Section 2 presents a high-level view of
their unification into Opus. After many major incompatible changes to the
original codecs, the result is open source, with patents licensed under
royalty-free terms111http://opus-codec.org/license/. The reference encoder is
professional-grade, and supports

  * •

Constant bit-rate (CBR) and variable bit-rate (VBR) rate control,

  * •

Floating point and fixed-point arithmetic, and

  * •

Variable encoder complexity.

Its CBR produces packets with exactly the size the encoder requested, without
a bit reservoir to imposes additional buffering delays, as found in codecs
such as MP3 or AAC-LD. It has two VBR modes: Constrained VBR (CVBR), which
allows bitrate fluctuations up to the average size of one packet, making it
equivalent to a bit reservoir, and true VBR, which does not have this
constraint.

This paper focuses on the CELT mode of Opus, which is used primarily for
encoding music. Although it retains the fundamental principles of the original
algorithms published in [5, 6] reviewed in Section 3, the CELT algorithm used
in Opus differs significantly from that work. We have psychoacoustically tuned
the bit allocation and quantization, as Section 4 outlines, and designed
additional tools to conceal artifacts, which Section 5 describes.

##  2 Overview of Opus

Opus operates in one of three modes:

  * •

SILK mode (speech signals up to wideband),

  * •

CELT mode (music and high-bitrate speech), or

  * •

Hybrid mode (SILK and CELT simultaneously for super-wideband and fullband
speech).

Fig. 1 shows a high-level overview of Opus. CELT always operates at a sampling
rate of 48 kHz, while SILK can operate at 8 kHz, 12 kHz, or 16 kHz. In hybrid
mode, the crossover frequency is 8 kHz, with SILK operating at 16 kHz and CELT
discarding all frequencies below the 8 kHz Nyquist rate.

CELT’s look-ahead is 2.5 ms, while SILK’s look-ahead is 5 ms, plus 1.5 ms for
the resampling (including both encoder and decoder resampling). For this
reason, the CELT path in the encoder adds a 4 ms delay. However, an
application can restrict the encoder to CELT and omit that delay. This reduces
the total look-ahead to 2.5 ms.

Figure 1: High-level overview of the Opus codec.

###  2.1 Configuration and Switching

Opus signals the mode, frame size, audio bandwidth, and channel count (mono or
stereo) in-band. It encodes this in a _table-of-contents_ (TOC) byte at the
start of each packet [1]. Additional internal framing allows it to pack
multiple frames into a single packet, up to a maximum duration of 120 ms.
Unlike the rest of the Opus bitstream, the TOC byte and internal framing are
not entropy coded, so applications can easily access the configuration, split
packets into individual frames, and recombine them.

Configuration changes that use CELT on both sides (or between wideband SILK
and Hybrid mode) use the overlap of the transform window to avoid
discontinuities. However, switching between CELT and SILK or Hybrid mode is
more complicated, because SILK operates in the time domain without a window.
For such cases, the bitstream can include an additional 5 ms _redundant CELT
frame_ that the decoder can overlap-add to bridge and gap between the
discontinuous data. Two redundant CELT frames—one on each side of the
transition—allow smooth transitions between modes that use SILK at different
sampling rates. The encoder handles all of this transparently. The application
may not even notice that the mode has changed.

##  3 Constrained-Energy Lapped Transform (CELT)

Like most transform coding algorithms, CELT is based on the MDCT. However, the
fundamental idea behind CELT is that the most important perceptual aspect of
an audio signal is the spectral envelope. CELT preserves this envelope by
explicitly coding the energy of a set of bands that approximates the auditory
system’s critical bands [7]. Fig. 2 illustrates the band layout used by Opus.
The format itself incorporates a significant amount of psychoacoustic
knowledge. This not only reduces certain types of artifacts, it also avoids
coding some parameters.

Figure 2: CELT band layout vs. the Bark scale.

CELT uses flat-top MDCT windows with a fixed overlap of 2.5 ms, regardless of
the frame size, as Fig. 3 shows. The overlapping part of the window is the
Vorbis [8] power-complementary window

$w(n)=\sin\left[\frac{\pi}{2}\sin^{2}\left(\frac{\pi\left(n+\frac{1}{2}\right)}{2L}\right)\right]\
.$ (1)

Unlike the AAC-ELD low-overlap window, the Opus window is still symmetric.
Compared to the full-overlap of MP3 or Vorbis, the low overlap allows lower
algorithmic delay and simplifies the handling of transients, as Section 3.1
describes. The main drawback is increased spectral leakage, which is
problematic for highly tonal signals. We mitigate this in two ways. First, the
encoder applies a first-order pre-emphasis filter
$A_{p}\left(z\right)=1-0.85z^{-1}$ to the input, and the decoder applies the
inverse de-emphasis filter. This attenuates the low frequencies (LF), reducing
the amount of leakage they cause at higher frequencies (HF). Second, the
encoder applies a perceptual prefilter, with a corresponding postfilter in the
decoder, as Section 5.1 describes.

Fig. 4 shows a complete block diagram of CELT. Sections 4 and 5 describe the
various components.

Figure 3: Low-overlap window used for 5 ms frames. Figure 4: Overview of the
CELT algorithm.

###  3.1 Handling of Transients

Like other transform codecs, Opus controls pre-echo primarily by varying the
MDCT size. When the encoder detects a transient, it computes multiple short
MDCTs over the frame and interleaves the output coefficients. For 20-ms
frames, there are 8 MDCTs with full-overlap, 5 ms windows. We constrain the
band sizes to be a multiple of the number of short MDCTs, so that the
interleaved coefficients form bands of the same size, covering the same part
of the spectrum in each block, as the corresponding band of a long MDCT.

##  4 Quantization and Encoding

Opus supports any bitrate that corresponds to an integer number of bytes per
frame. Rather than signal a rate explicitly in the bitstream, Opus relies on
the lower-level transport protocol, such as RTP, to transmit the payload
length. The decoder, not the encoder, makes many bit allocation decisions
automatically based on the number of bits remaining. This means that the
encoder must determine the final rate early in the encoding process, so it can
make matching decisions, unlike codecs such as AAC, MP3, and Vorbis. This has
two advantages. First, the encoder need not transmit these decisions, avoiding
the associated overhead. Second, they allow the encoder to achieve a target
bitrate exactly, without repeated encoding or bit reservoirs. Even though
entropy coding produces variable-sized output, these dynamic adjustments to
the bit allocation ensure that the coded symbols never exceed the number of
bytes allocated for the frame by the encoder earlier in the process. In the
vast majority of cases, the encoder also wastes less than two bits.

Opus encodes most symbols using a range coder [9]. Some symbols, however, have
a power-of-two range and approximately uniform probability. Opus packs these
as _raw bits_ , starting at the end of the packet, back towards the end of the
range coder output, as Fig. 5 illustrates. This allows the decoder to rapidly
switch between decoding symbols with the range coder and reading raw bits,
without interleaving the data in the packet. It also improves robustness to
bit errors, as corruption in the raw bits does not desynchronize the range
coder. A special termination rule for the range coder, described in Section
5.1.5 of RFC 6716 [1], ensures the stream remains decodable regardless of the
values of the raw bits, while using at most 1 bit of padding to separate the
two.

Figure 5: Layout and coding order of the bitstream.

###  4.1 Coarse Energy Quantization ($Q_{1}$)

The most important information encoded in the bitstream is the energy of the
MDCT coefficients in each band. Band energy is quantized using a two-pass
coarse-fine quantizer. The coarse quantizer uses a fixed 6 dB resolution for
all bands, with inter-band prediction and, optionally, inter-frame prediction.
The 2D _z_ -transform of the predictor is

$A\left(z_{\ell},z_{b}\right)=\left(1-\alpha
z_{\ell}^{-1}\right)\cdot\frac{1-z_{b}^{-1}}{1-\beta z_{b}^{-1}}\ ,$ (2)

where $\ell$ is the frame index and $b$ is the band. Inter-frame prediction
can be turned on or off for any frame. When enabled, both $\alpha$ and $\beta$
are non-zero and depend on the frame size. When disabled, $\alpha=0$ and
$\beta=0.15$. Inter-frame prediction is more efficient, but less robust to
packet loss. The encoder can use packet loss statistics to force inter-frame
prediction off adaptively. The prediction residual is entropy-coded assuming a
Laplace probability distribution with per-band variances trained offline.

###  4.2 Bit Allocation

Rather than transmitting scale factors like MP3 and AAC or a floor curve like
Vorbis, CELT mostly allocates bits implicitly. After coarse energy
quantization, the encoder decides on the total number of bytes to use for the
frame. Then both the encoder and decoder run the same _bit-exact_ bit
allocation function to partition the bits among the bands. CELT interpolates
between several static allocation prototypes (see Fig. 6) to achieve the
target rate.

Some bands may not receive any bits. The decoder reconstructs them using only
the energy, generating fine details by spectral folding, as Section 4.4.1
details. When a band receives very few bits, the sparse spectrum that could be
encoded with them would sound worse than spectral folding. Such bands are
automatically skipped, redistributing the bits they would have used to code
their spectrum to the remaining bands. The encoder can also skip more bands
via explicit signaling. This allows it to give the skip decisions some
hysteresis between frames.

After the initial allocation, bands are encoded one at a time. In practice, a
band may use slightly more or slightly fewer bits than allocated. The
difference propagates to subsequent bands to ensure that the final rate still
matches the overall target. Automatically adjusting the allocation based on
the actual bits used makes achieving CBR easy.

Figure 6: Static bit allocation curves in bits/sample for each band and for
multiple bitrates.

The implicit allocation produces a nearly constant signal-to-noise ratio in
each band, with the LF coded at a higher resolution than the HF. It
approximates the real masking curve well without any signaling, and achieves
good quality by itself, as demonstrated by earlier versions of the algorithm
[5, 6]. However, it does not cover two theoretical phenomena:

  1. 1.

Tonality: tones provide weaker masking than noise, requiring a finer
resolution. Since tones usually have harmonics in many bands, the encoder
increases the total rate for these frames.

  2. 2.

Inter-band masking: a band may be masked by neighboring bands, though this is
weaker than intra-band masking.

CELT provides two signaling mechanisms that adjust the implicit allocation:
one that changes the tilt of the allocation, and one that boosts specific
bands.

####  4.2.1 Allocation Tilt

The allocation tilt parameter changes the slope of the bit allocation as a
function of the band index by up to $\pm 5/64$ bit/sample/band, in increments
of $1/64$ bit/sample/band. Although in theory the slope of the masking
threshold should follow the slope of the signal’s spectral envelope, we have
observed that LF-dominated signals require more bits in the LF, with a similar
observation for HF-dominated signals.

####  4.2.2 Band Boost

When a specific band requires more bits, the bitstream includes a mechanism
for increasing its allocation (reducing the allocation of all other bands).
Versions 1.0.x and earlier of the Opus reference implementation rarely use
this band boost. However, newer versions use it to improve quality in the
following circumstances:

  * •

In transients frames, bands dominated by the leakage of the shorter MDCTs
receive more bits.

  * •

Bands that have significantly larger energy than surrounding bands receive
more bits.

CELT does not provide a mechanism to reduce the allocation of a single band
because it would not be worth the signaling cost.

###  4.3 Fine Energy Quantization ($Q_{2}$)

Once the per-band bit allocation is determined, the encoder refines the
coarsely-quantized energy of each band. Let $a$ be the total allocation for a
band containing $N_{DoF}$ degrees of freedom222Usually equal to the number of
coefficients. When stereo coupling is used on a band with more than 2
coefficients, the combined band has an additional degree of freedom.. We
approximate (30) from [10] to obtain the fine energy allocation:

$a_{f}=\frac{a}{N_{DoF}}+\frac{1}{2}\log_{2}N_{DoF}-K_{fine}\ ,$ (3)

where $K_{fine}$ is a tuned _fine allocation offset_. We round the result to
an integer and code the refinement data as raw bits. Bands where $N_{DoF}=2$
get slightly more bits, and we slightly bias the allocation upwards when
adding the first and second fine bit.

If any bits are left unused at the very end of the frame, each band may add
one additional bit per channel to refine the band energy, starting with bands
for which $a_{f}$ was rounded down.

###  4.4 Pyramid Vector Quantization ($Q_{3}$)

Let $\mathbf{X}_{b}$ be the MDCT coefficients for band $b$. We normalize the
band with the unquantized energy,

$\mathbf{x}_{b}=\frac{\mathbf{X}_{b}}{\left\|\mathbf{X}_{b}\right\|}\ ,$ (4)

producing a unit vector on an $N$-sphere, coded with a pyramid vector
quantizer (PVQ) [11] codebook:

$S\left(N,K\right)=\left\\{\frac{\mathbf{y}}{\left\|\mathbf{y}\right\|},\
\mathbf{y}\in\left\\{\mathbb{Z^{N}}:\sum_{i=0}^{N-1}\left|y_{i}\right|=K\right\\}\right\\}\
,$

where $K$ is the $L_{1}$-norm of $\mathbf{y}$, i.e. the number of _pulses_.
The codebook size obeys the recurrence

$V\left(N,K\right)=V\left(N,K-1\right)\\\
+V\left(N-1,K\right)+V\left(N-1,K-1\right)\ ,$ (5)

with $V\left(N,0\right)=1$ and $V\left(0,K\right)=0,\ K>0$. Because
$V\left(N,K\right)$ is rarely a power of two, we use the range coder with a
uniform probability to encode the codeword index, derived from $\mathbf{y}$
using the method of [11]. When $V\left(N,K\right)$ is larger than 255, the
index is renormalized to fall in the range $\left[128,255\right]$ and the
least significant bits are coded using raw bits. The uniform probability
allows both the encoder and the decoder to choose $K$ such that
$\log_{2}V\left(N,K\right)$ achieves allocation determined in Section 4.2.

####  4.4.1 Spectral Folding

When a band receives no bits, the decoder replaces the spectrum of that band
with a normalized copy of MDCT coefficients from lower frequencies. This
preserves some temporal and tonal characteristics from the original band, and
CELT’s energy normalization preserves the spectral envelope. Spectral folding
is far less advanced than spectral band replication (SBR) from HE-AAC and
mp3PRO, but is computationally inexpensive, requires no extra delay, and the
decision to apply it can change frame-by-frame.

###  4.5 Stereo

Opus supports three different stereo coupling modes:

  1. 1.

Mid-side (MS) stereo

  2. 2.

Dual stereo

  3. 3.

Intensity stereo

A coded band index denotes where intensity stereo begins: all bands above it
use intensity stereo, while all bands below it use either MS or dual stereo. A
single flag at the frame level chooses between them.

####  4.5.1 Mid-Side Stereo

We apply MS stereo coupling separately on each band, after normalization.
Because we code the energy of each channel separately, MS stereo coupling
never introduces cross-talk between channels and is safe even when dual stereo
is more efficient. Let $\mathbf{x}_{l}$ and $\mathbf{x}_{r}$ be the normalized
band for the left and right channels, respectively. The orthogonal mid and
side signals are computed as

$\displaystyle\mathbf{M}$
$\displaystyle=\frac{\mathbf{x}_{l}+\mathbf{x}_{r}}{2}\ ,$ (6)
$\displaystyle\mathbf{S}$
$\displaystyle=\frac{\mathbf{x}_{l}-\mathbf{x}_{r}}{2}\ .$ (7)

Opus encodes the mid and side as normalized signals
$\mathbf{m}=\mathbf{M}/\left\|\mathbf{M}\right\|$ and
$\mathbf{s}=\mathbf{S}/\left\|\mathbf{S}\right\|$. To recover $\mathbf{M}$ and
$\mathbf{S}$ from $\mathbf{m}$ and $\mathbf{s}$, we need to know the ratio of
$\left\|\mathbf{S}\right\|$ to $\left\|\mathbf{M}\right\|$, which we encode as
the angle

$\theta_{s}=\arctan\frac{\left\|\mathbf{S}\right\|}{\left\|\mathbf{M}\right\|}\
.$ (8)

Explicitly coding $\theta_{s}$ preserves the stereo width and reduces the risk
of _stereo unmasking_ [12, 7], since it preserves the energy of the difference
signal, in addition to the energy in each channel. We quantize $\theta_{s}$
uniformly, deriving the resolution the same way as the fine energy allocation.
Uniform quantization of $\theta_{s}$ achieves optimal mean-squared error
(MSE).

Let $\hat{\mathbf{m}}$ and $\hat{\mathbf{s}}$ be the quantized versions of
$\mathbf{m}$ and $\mathbf{s}$. We can compute the reconstructed signals as

$\displaystyle\hat{\mathbf{x}}_{l}$
$\displaystyle=\hat{\mathbf{m}}\cos\hat{\theta}_{s}+\hat{\mathbf{s}}\sin\hat{\theta}_{s}\
,$ (9) $\displaystyle\hat{\mathbf{x}}_{r}$
$\displaystyle=\hat{\mathbf{m}}\cos\hat{\theta}_{s}-\hat{\mathbf{s}}\sin\hat{\theta}_{s}\
.$ (10)

As a result of the quantization, $\hat{\mathbf{m}}$ and $\hat{\mathbf{s}}$ may
not be orthogonal, so $\hat{\mathbf{x}}_{l}$ and $\hat{\mathbf{x}}_{r}$ may
not have exactly unit norm and must be renormalized.

The MSE-optimal bit allocation for $\mathbf{m}$ and $\mathbf{s}$ depends on
$\hat{\theta}_{s}$. Let $N$ be the size of the band and $a$ be the total
number of bits available for $\mathbf{m}$ and $\mathbf{s}$. Then the optimal
allocation for $\mathbf{m}$ is

$a_{mid}=\frac{a-\left(N-1\right)\log_{2}\tan\theta_{s}}{2}\ .$ (11)

The larger of $\mathbf{m}$ and $\mathbf{s}$ is coded first, and any unused
bits are given to the other channel. As a special case, when $N=2$ we use the
orthogonality of $\mathbf{m}$ and $\mathbf{s}$ to code one of the channels
using a single sign bit.

####  4.5.2 Dual Stereo

Dual stereo codes the normalized left and right channels independently. We use
this only when the correlation between the channels is not strong enough to
make up for the cost of coding the $\theta_{s}$ angles.

####  4.5.3 Intensity Stereo

Intensity stereo also works in the normalized domain, using a single mid
channel with no side. Instead of $\theta_{s}$, we code a single _inversion_
flag for each band. When set, we invert the right channel, producing two
channels 180 degrees out of phase.

###  4.6 Band splitting

At high bitrates, we allocate some bands hundreds of bits. To avoid arithmetic
on large integers in the PVQ index calculations, we split bands with more than
32 bits, using the same process as MS stereo. $\mathbf{M}$ and $\mathbf{S}$
are set to the first and second half of the band, with $\theta_{s}$ indicating
the distribution of energy between the two halves. If a band contains data
from multiple short MDCTs, we bias the bit allocation to account for pre-echo
or forward masking using $\hat{\theta}_{s}$. If one sub-vector still requires
more than 32 bits, we split it recursively. This recursion stops after 4
levels (1/16th the size of the original band), which puts a hard limit on the
number of bits a band can use. This limit lies far beyond the rate needed to
achieve transparency in even the most difficult samples.

##  5 Psychoacoustic Improvements

We can achieve good audio quality using just the algorithms described above.
However, four different psychoacoustically-motivated improvements make coding
artifacts even less audible.

###  5.1 Prefilter and Postfilter

The low-overlap window increases leakage in the MDCT, resulting in higher
quantization noise on highly tonal signals. Widely-spaced harmonics in
periodic signals provide especially little masking. Opus mitigates this
problem using a pitch-enhancing post-filter. Unlike speech codec postfilters,
we run a matching prefilter on the encoder side. The pair provides perfect
reconstruction (in the absence of quantization), allowing us to enable the
postfilter even at high bitrates. Although the filters look like a pitch
predictor, unlike standard pitch prediction we apply the prefilter to the
unquantized signal, allowing pitch periods shorter than the frame size. The
gain and period are transmitted explicitly. When these change between two
frames, the filter response is interpolated using a 2.5 ms cross-fade window
equal to the square of the $w(n)$ power-complementary window. We use a 5-tap
prefilter with an impulse response of

$A\left(z\right)=1-g\cdot\left[a_{p,2}\left(z^{-T-2}+z^{-T+2}\right)\right.\\\
\left.{}+a_{p,1}\left(z^{-T-1}+z^{-T+1}\right)+a_{p,0}z^{-T}\right]\ ,$ (12)

where $T$ is the pitch period, $g$ is the gain, and $a_{p,i}$ are the
coefficients of tapset $p$. We choose one of three different tapsets to
control the range of frequencies to which we apply the enhancement. They are

$\begin{array}[]{lcl}a_{0,\cdot}&=&\left[0.80\;0.10\;0\right]\ ,\\\
a_{1,\cdot}&=&\left[0.46\;0.27\;0\right]\ ,\\\
a_{2,\cdot}&=&\left[0.30\;0.22\;0.13\right]\ .\end{array}$ (13)

The pitch period lies in the range $\left[15,1022\right]$, and the gain varies
between 0.09 and 0.75. Fig. 7 shows the frequency response of each tapset for
a period of $T=24$ (2 kHz) and a gain $g=0.75$.

Figure 7: Frequency response of the different postfilter tapsets for $T=24$,
$g=0.75$.

Subjective testing conducted by Broadcom on an earlier version of the
algorithm demonstrated the postfilter’s effectiveness [13].

###  5.2 Variable Time-Frequency Resolution

Some frames contain both tones and transients, requiring both good time
resolution and good frequency resolution. Opus achieves this by selectively
modifying the time-frequency (TF) resolution in each band. For example, Opus
can have good frequency resolution for LF tonal content while retaining good
time resolution for a transient’s HF. We change the TF resolution with a
Hadamard transform, a cheap approximation of the DCT. When using multiple
short MDCTs (good time resolution), we increase the frequency resolution of a
band by applying the Hadamard transform to the same coefficient across
multiple MDCTs. This can increase the frequency resolution by a factor of 2 to
8, decreasing the time resolution by the same amount.

The Hadamard transform of consecutive coefficients increases the time
resolution of a long MDCT. This yields more time-localized basis functions,
although they have more ringing than the equivalent short MDCT basis
functions. Fig. 8 illustrates basis functions produced by adaptively modifying
the time-frequency resolution for a 20 ms frame.

Figure 8: Basis functions with modified time-frequency resolution for a 20 ms
frame. Left: fourth basis function of a long MDCT vs. the equivalent basis
function from TF modification of 8 short MDCTs. Right: First (“DC”) basis
function of a short MDCT vs. the equivalent basis function from TF
modification of the first 8 coefficients of a long MDCT.

###  5.3 Spreading Rotations

A common type of artifact in transform codecs is tonal noise, also known as
_birdies_. When quantizing a large number of HF MDCT coefficients to zero, the
few remaining non-zero coefficients sound tonal even when the original signal
did not. This is most noticeable in low-bitrate MP3s. Opus greatly reduces
tonal noise by applying _spreading rotations_. The encoder applies these
rotations to the normalized signal prior to quantization, and the decoder
applies the inverse rotations, as Fig. 9 shows.

We construct the spreading rotations from a series of 2D Givens rotations. Let
$\mathbf{G}\left(m,n,\theta_{r}\right)$ denote a Givens rotation matrix by
angle $\theta_{r}$ between coefficients $m$ and $n$ in some band with $N$
coefficients, with angles near $\pi/4$ implying more spreading. Then the
spreading rotations are

$\mathbf{R}\left(\theta_{r}\right)=\prod_{k=0}^{N-3}\mathbf{G}\left(k,k+1,\theta_{r}\right)\\\
\cdot\prod_{k=2}^{N}\mathbf{G}\left(N-k,N-k+1,\theta_{r}\right)\ .$ (14)

In other words, we rotate adjacent coefficient pairs one at a time from the
beginning of the vector to the end, and then back. We determine $\theta_{r}$
from the band size, $N$, and the number of pulses used, $K$:

$\theta_{r}=\frac{\pi}{4}\left(\frac{N}{N+\delta K}\right)^{2}\ ,$ (15)

where $\delta$ is the _spreading constant_. Once per frame, the encoder
selects $\delta$ from one of three values: 5, 10, or 15, or disables spreading
completely.

In transient frames, we apply the spreading rotations to each short MDCT
separately to avoid pre-echo. When vectors of more than 8 coefficients need to
be rotated, we apply an additional set of rotations to pairs of coefficients
$\bigl{\lfloor}\sqrt{N}\bigr{\rfloor}$ positions apart, using the angle
$\theta_{r}^{{}^{\prime}}=\frac{\pi}{2}-\theta_{r}$. This spreads the energy
within large bands more widely.

Figure 9: Spreading example.

###  5.4 Collapse Prevention

In transients at low bitrates, Opus may quantize all of the coefficients in a
band corresponding to a particular short MDCT to zero. Even though we preserve
the energy of the entire band, this quantization causes audible drop outs, as
Fig. 10 shows on the left. The decoder detects _holes_ that occur when a short
MDCT receives no pulses in a given band, or when folding copies such a hole
into a higher band, and fills them with pseudo-random noise at a level equal
to the minimum band energy over the previous two frames. The encoder transmits
one flag per frame that can disable collapse prevention. We do this after two
consecutive transients to avoid putting too much energy in the holes. Fig. 10
shows the result of collapse prevention on the right. The short drop outs
around each transient are no longer audible.

Figure 10: Extreme collapse prevention example for castanets at 32 kb/s mono.
Top: without collapse prevention. Bottom: with collapse prevention.

##  6 Evaluation and Results

This section presents a quality evaluation of Opus’s CELT mode on music
signals. More complete evaluation data on Opus is available at [14].

###  6.1 Subjective Quality

Volunteers of the HydrogenAudio forum333http://hydrogenaudio.org/ evaluated
the quality of 64 kb/s VBR Opus on fullband stereo music with headphones. 13
listeners evaluated 30 samples using the ITU-R BS.1116-1 methodology [15] with

  * •

The Opus [2] reference implementation (v0.9.2),

  * •

Apple’s HE-AAC444With constrained VBR, as it cannot run unconstrained
(QuickTime v7.6.9),

  * •

Nero’s HE-AAC555http://www.nero.com/enu/company/about-nero/nero-aac-codec.php
(v1.5.4.0), and

  * •

Ogg Vorbis (AoTuV666http://www.geocities.jp/aoyoume/aotuv/ v6.02 Beta).

Apple’s AAC-LC at 48 kb/s served as a low anchor. Fig. 11 shows the results. A
pairwise resampling-based free step-down analysis using the max(T) algorithm
[16, 17] reveals that Opus is better than the other codecs with greater than
99.9% confidence. Apple’s HE-AAC was better than both Nero’s HE-AAC and Vorbis
with greater than 99.9% confidence. Nero’s HE-AAC and Vorbis were
statistically tied. A simple ANOVA analysis gives the same results.

Figure 11: Results of the 64 kb/s evaluation. The low anchor (omitted) was
rated at 1.54 on average.

###  6.2 Cascading Performance

In broadcasting applications, audio streams are compressed and recompressed
multiple times. According to [18], typical broadcast chains may include up to
5 lossy encoding stages. For this reason, we compare the cascading quality of
Opus to both Vorbis and MP3 using PQevalAudio [19], an implementation of the
PEAQ basic model [20]. Fig 12 plots quality as a function of bitrate and the
number of cascaded encodings. Opus performs better than MP3 and Vorbis in the
presence of cascading, with 64 kb/s Opus even out-performing 128 kb/s MP3.
Although the Opus quality with 5 ms frames is lower than for 20 ms frames, it
is still acceptable, and better than MP3.

Figure 12: Cascading quality. Left: Quality degradation vs. number of
cascadings at 128 kb/s. Right: Quality degradation vs. bitrate after 5
cascadings.

##  7 Conclusion and Future Work

By building psychoacoustic knowledge into the Opus format, we minimize the
side information it transmits and the impact of coding artifacts. This allows
Opus to achieve higher music quality than existing non-real-time codecs, even
under cascading. Since Opus was only recently standardized, we are continuing
to improve its encoder, experimenting with such things as look-ahead and
automatic frame size switching for non-real-time encoding.

## References

  * [1] J.-M. Valin, K. Vos, and T. B. Terriberry.  Definition of the Opus Audio Codec.  RFC 6716, http://www.ietf.org/rfc/rfc6716.txt, September 2012.
  * [2] Opus website.  http://opus-codec.org/.
  * [3] A. Carôt.  Musical Telepresence – A Comprehensive Analysis Towards New Cognitive and Technical Approaches.  PhD thesis, University of Lübeck, 2009.
  * [4] K. Vos, S. Jensen, and K. Sørensen.  SILK speech codec.  IETF Internet-Draft http://tools.ietf.org/html/draft-vos-silk-02.
  * [5] J.-M. Valin, T. B. Terriberry, and G. Maxwell.  A full-bandwidth audio codec with low complexity and very low delay.  In Proc. EUSIPCO, 2009.
  * [6] J.-M. Valin, T. B. Terriberry, C. Montgomery, and G. Maxwell.  A high-quality speech and audio codec with less than 10 ms delay.  IEEE Trans. Audio, Speech and Language Processing, 18(1):58–67, 2010.
  * [7] B. C.J. Moore.  An Introduction to the Psychology of Hearing.  fifth edition, 2004.
  * [8] C. Montgomery.  Vorbis I specification.  http://www.xiph.org/vorbis/doc/Vorbis_I_spec.html, 2004.
  * [9] G. Nigel and N. Martin.  Range encoding: An algorithm for removing redundancy from a digitised message.  In Proc. Video and Data Recording Conference, 1979.
  * [10] H. Krüger, R. Schreiber, B. Geiser, and P. Vary.  On logarithmic spherical vector quantization.  In Proc. ISITA, 2008.
  * [11] T. R. Fischer.  A pyramid vector quantizer.  IEEE Trans. on Information Theory, 32:568–583, 1986.
  * [12] J. D. Johnston and A. J. Ferreira.  Sum-difference stereo transform coding.  In Proc. ICASSP, volume 2, pages 569–572, 1992.
  * [13] R. Chen, T. B. Terriberry, J. Skoglund, G. Maxwell, and H. T. M. Nguyet.  Opus testing.  In Proc. codec WG, $80^{th}$ IETF meeting, pages 1–4, Prague, 2011.  http://www.ietf.org/proceedings/80/slides/codec-4.pdf.
  * [14] C. Hoene, J.-M. Valin, K. Vos, and J. Skoglund.  Summary of opus listening test results.  IETF Internet-Draft http://tools.ietf.org/html/draft-ietf-codec-results, 2012.
  * [15] ITU-R.  Recommendation BS.1116-1: Methods for the subjective assessment of small impairments in audio systems including multichannel sound systems, 1997\.
  * [16] Peter H. Westfall and S. Stanley Young.  Resampling-Based Multiple Testing: Examples and Methods for $p$-Value Adjustment.  Wiley Series in Probability and Statistics. John Wiley & Sons, New York, January 1993.
  * [17] Gian-Carlo Pascutto.  Bootstrap.  http://www.sjeng.org/bootstrap.html, 2011.
  * [18] D. Marston and A. Mason.  Cascaded audio coding.  EBU Technical Review, 2005.
  * [19] P. Kabal.  An examination and interpretation of ITU-R BS.1387: Perceptual evaluation of audio quality.  Technical report, TSP Lab, ECE Dept., McGill University, http://www.TSP.ECE.McGill.CA/MMSP/Documents, May 2002.
  * [20] ITU-R.  Recommendation BS.1387: Perceptual Evaluation of Audio Quality (PEAQ) recommendation, 1998.