# RFWave: Multi-band Rectified Flow for Audio Waveform Reconstruction

Peng Liu
<EMAIL_ADDRESS>
&Dongyang Dai
<EMAIL_ADDRESS>

###### Abstract

Recent advancements in generative modeling have led to significant progress in
audio waveform reconstruction from diverse representations. Although diffusion
models have been used for reconstructing audio waveforms, they tend to exhibit
latency issues because they operate at the level of individual sample points
and require a relatively large number of sampling steps. In this study, we
introduce RFWave, a novel multi-band Rectified Flow approach that reconstructs
high-fidelity audio waveforms from Mel-spectrograms. RFWave is distinctive for
generating complex spectrograms and operating at the frame level, processing
all subbands concurrently to enhance efficiency. Thanks to Rectified Flow,
which aims for a flat transport trajectory, RFWave requires only 10 sampling
steps. Empirical evaluations demonstrate that RFWave achieves exceptional
reconstruction quality and superior computational efficiency, capable of
generating audio at a speed 90 times faster than real-time111Code is available
at https://github.com/bfs18/rfwave.

##  1 Introduction

The advent of neural network-based generative models has revolutionized the
generation and synthesis of audio, showcasing remarkable advancements in
artificial intelligence. These models are adept at learning the nuances of
training data distributions, enabling them to reconstruct new, similar data
points. In particular, waveform reconstruction has emerged as a key
application of these generative models.

A key focus in speech technology research is on reconstructing high-quality
waveforms from compact representations, such as Mel-spectrograms or phoneme
sequences. WaveNet [1], a convolution-based autoregressive model, marked a
major breakthrough in this domain through the use of neural network-based
generative models for waveform reconstruction. The quality of waveforms
produced by WaveNet significantly exceeds that of previous signal processing
methods like WORLD [2]. However, the sequential, sample-by-sample waveform
reconstruction approach of WaveNet leads to substantial computational demands.
To more effectively harness the capabilities of recurrent networks, WaveRNN
[3] was developed, employing an RNN in an autoregressive model for waveform
reconstruction, optimizing the use of recurrent network characteristics.

The inherent slowness of autoregressive models, which predict samples one
after another, has prompted interest in parallel waveform reconstruction. This
has led to the development of models like Parallel WaveNet [4] and ClariNet
[5]. Concurrently, the emergence of Generative Adversarial Networks (GANs) [6]
has given rise to parallel GAN-based waveform reconstruction methods, such as
Mel-GAN [7], ParallelWaveGAN [8], and HiFi-GAN [9]. These approaches generally
offer faster reconstruction speeds, attributing to their generators’ simpler
model structures compared to prior parallel methods.

Considering that speech signals consist of a high number of samples per
second, modeling speech waveforms typically involves intricate neural
networks, which incorporate upsampling layers. The incorporation of ISTFT in
replacing some upsampling layers of the model, shifting its task to predicting
the complex spectrograms rather than directly reconstructing the waveform,
effectively refines the model structure and enhances computational efficiency.
Models such as iSTFTNET [10], Vocos [11], and APNet2 [12], which are grounded
in GAN frameworks, have effectively used this method in reconstructing
waveforms. They have achieved faster processing speeds and increased
efficiency, all while maintaining the quality of the speech.

Nevertheless, GAN-based waveform reconstruction models [7, 8, 9, 10, 11] often
require intricate discriminator designs for high-quality output and may face
issues like instability and mode collapse. To address these challenges,
researchers have investigated diffusion models for reconstructing waveforms,
as seen in studies like Diffwave [13], WaveGrad [14], and Multi-Band Diffusion
[15]. However, diffusion-based models typically necessitate multiple sampling
steps for synthesizing high-quality audio, leading to significant
computational costs.

In this paper, we introduce a novel waveform reconstruction model based on
Rectified Flow [16], which ensures high-quality output while also
significantly enhancing overall efficiency. Our main contributions are
summarized as follows:

  1. 1.

By adopting the Rectified Flow and a innovative time-balanced loss, our model
can reconstruct high-quality waveforms with a drastically reduced number of
sampling steps.

  2. 2.

We implement a multi-band approach that generates different sub-bands in
parallel, ensuring audio quality while avoiding cumulative errors and
enhancing synthesis speed.

  3. 3.

Our model operates at the level of STFT frames, not individual waveform sample
points. This approach significantly enhances processing speed.

Our experimental results confirm that our model is capable of generating high-
fidelity audio waveforms and performing inference at speeds up to 90 times
faster than real-time.

##  2 Background

#### Rectified Flow

Rectified Flow [16] presents an innovative ODE-based framework for generative
modeling and domain transfer. It introduces a method to learn a transport
mapping that connects two distributions, $\pi_{0}$ and $\pi_{1}$ on
$\mathbb{R}^{d}$, based on empirical observations:

$\frac{\mathrm{d}Z_{t}}{\mathrm{d}t}=v(Z_{t},t),\leavevmode\nobreak\
\leavevmode\nobreak\ \leavevmode\nobreak\ \text{initialized from
$Z_{0}\sim\pi_{0}$, such that $Z_{1}\sim\pi_{1}$},$ (1)

where $v\colon\mathbb{R}^{d}\times[0,1]\to\mathbb{R}^{d}$ represents a
velocity field. The learning of this field involves minimizing a mean square
objective function,

$\min_{v}\mathbb{E}_{(X_{0},X_{1})\sim\gamma}\left[\int_{0}^{1}\mid\mid\frac{\mathrm{d}}{\mathrm{d}t}X_{t}-v(X_{t},t)\mid\mid^{2}\mathrm{d}t\right],\leavevmode\nobreak\
\leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\
\text{with}\leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\
\leavevmode\nobreak\ X_{t}=\phi(X_{0},X_{1},t),$ (2)

where $X_{t}=\phi(X_{0},X_{1},t)$ represents a time-differentiable
interpolation between $X_{0}$ and $X_{1}$, with
$\frac{\mathrm{d}}{\mathrm{d}t}X_{t}=\partial_{t}\phi(X_{0},X_{1},t)$. The
$\gamma$ represents any coupling of $(\pi_{0},\pi_{1})$. An illustrative
instance of $\gamma$ is the independent coupling
$\gamma=\pi_{0}\times\pi_{1}$, which allows for empirical sampling based on
separately observed data from $\pi_{0}$ and $\pi_{1}$. The authors recommended
a simple choice of

$X_{t}=(1-t)X_{0}+tX_{1}\implies\frac{\mathrm{d}}{\mathrm{d}t}X_{t}=X_{1}-X_{0}.$
(3)

This simplification results in linear trajectories, which are critical for
accelerating the inference process. Typically, the velocity field $v$
represented using a deep neural network. The solution to (2) is approximated
through stochastic gradient methods. To approximate the ODE presented in (1),
numerical solvers are commonly employed. A prevalent technique is the forward
Euler method. This approach computes values using the formula

$Z_{t+\frac{1}{n}}=Z_{t}+\frac{1}{n}v(Z_{t},t),\leavevmode\nobreak\
\leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\
\leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\
\leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\ \forall
t\in\\{0,\ldots,n-1\\}/n,$ (4)

where the simulation is executed with a step size of $\epsilon=1/n$ over $n$
steps.

The velocity field has the capacity to incorporate conditional information.
This is particularly essential in applications like text-to-image generation,
where a text prompt is a critical factor. Consequently, in such contexts,
$v(Z_{t},t)$ in (2) is modified to $v(Z_{t},t\mid\mathcal{C})$, where
$\mathcal{C}$ represents the conditional information pertinent to the
corresponding $X_{1}$.

#### Estimating Complex Spectrograms

Waveform reconstruction from complex spectrograms can be effectively achieved
using the Inverse Short-Time Fourier Transform (ISTFT). Notably, Vocos [11]
and APNet2 [12], utilizing GANs as their model framework, estimate magnitude
and phase spectrograms from the input Mel spectrograms, which can be
transformed to complex spectrograms effortlessly. Both models operate at the
frame level, enabling them to achieve significantly faster inference speeds
compared to HiFi-GAN [9], which uses multiple upsampling layers and operates
at the level of waveform sample points. Morever, these models preserve the
quality of the synthesized waveform, demonstrating their superiority in both
speed and fidelity without a trade-off. In this paper, we directly estimate
complex spectrograms using Rectified Flow and focus on frame-level operations,
aiming to enhance both the efficiency and quality of our waveform synthesis
process.

#### Multi-band Audio Waveform Reconstruction

Both Multi-band MelGAN [17] and Multi-band Diffusion [15] employ multi-band
strategies, albeit for different purposes within their respective frameworks.
Multi-band MelGAN, specifically, uses Pseudo-Quadrature Mirror Filters (PQMF)
[18] to divide frequency bands. This division results in each subband’s
waveform being a fraction of the original waveform’s length, based on the
number of subbands. By reshaping these subbands into feature dimensions and
utilizing a unified backbone for modeling, Multi-band Melgan is able to
operate on considerably shorter signals. This strategy significantly enhances
the efficiency of the model, leading to accelerated training and inference
processes. Multi-band Diffusion utilizes an array of band-pass filters to
separate the frequency bands and models each subband with a distinct model.
This approach ensures that errors in one band do not negatively impact the
others. In our research, we simplify the process of frequency band division by
directly choosing the appropriate dimensions from the complex spectrograms.
Furthermore, we enhance efficiency by modeling all subbands together in
parallel with a single, unified model. This strategy improves the processing
speed and also helps in reducing error accumulation across different subbands.

##  3 Method

Our model utilizes a multi-band Rectified Flow to directly predict the complex
spectrogram. It operates at the STFT frame level and incorporates a highly
efficient ConvNeXtV2 [19] backbone. With only 10 steps of sampling, the model
is capable of producing high-quality waveforms.

(a) RFWave

(b) ConvNeXtV2 backbone

Figure 1: The overall structure for RFWave. [border-style=solid,border-
radius=1ex]Band i is the subband index, [border-style=solid,border-
radius=1ex]Cond is the conditional input, and [border-style=dashed,border-
radius=1ex]Encodec i is the Encodec bandwidth index. The blue boxes represent
modules that contain trainable parameters, while the green boxes symbolize
modules without trainable parameters. Modules enclosed in a dashed box are
considered optional.

###  3.1 Multi-band Rectified Flow

In our initial experiments, we observe that attempts to predict the full-band
complex spectrogram result in a compromised quality of waveform
reconstruction. To address this, we shift our approach to a multi-band
structure. It introduces error accumulation when conditioning higher bands on
lower bands, whereby inaccuracies in the lower bands adversely affects the
higher bands during the inference stage, as noted in [15]. Consequently, we
design our model by not employing the lower band as a conditional input for
the higher band. This structure yields an additional benefit: the ability to
predict all frequency bands concurrently, thereby significantly diminishing
the inference latency. We can batch the different bands of a sample to
facilitate simultaneous training or inference.

The model structure is depicted in Figure 1(a). All frequency bands share the
model, distinguished by a unique subband index. For each subband, the
corresponding noise is fed into the ConvNeXtV2 backbone to predict the
velocity conditioned on time $t$, the subband index, conditional input (the
Mel spectrogram or the Encodec [20] embedding), and an optional Encodec
bandwidth index. The detailed structure of the ConvNeXtV2 backbone is shown in
Figure 1(b). We employ Fourier features as described in [21]. The noise,
Fourier features, and conditional inputs are concatenated and then passed
through a linear layer, forming the input that is fed into a series of
ConvNeXtV2 blocks. The sinusoidal time embedding, along with the optional
Encodec bandwidth index embedding, are element-wise added to the input of each
ConvNeXtV2 block. Furthermore, the subband index is incorporated via an
adaptive layer normalization module, which utilizes learnable embeddings as
described in [22, 11]. The other components are identical to those within the
ConvNeXtV2 architecture, details can be found in the [19].

###  3.2 Operating in Time Domain or Frequency Domain

Our model is designed to function at the STFT frame level, with the
flexibility to operate in either the time or frequency domain. In the time
domain, as shown in Figure 1(a), the signal and noise are both inherently
temporal, necessitating the use of STFT and ISTFT. Conversely, in the
frequency domain, both noise and velocity are represented in this domain,
eliminating the need for STFT and ISTFT. For the time domain, noise and
velocity adhere to dimensions of $[1,T]$, where $T$ represents the waveform
length in sample points222For simplicity, the batch dimension is not included
in the discussion.. Notably, only subband $i$ is processed after STFT and
prior to ISTFT in a single feed-through, in line with our prior discussion. In
the frequency domain, the dimensions of noise and velocity shift to
$[d_{s},F]$, with $d_{s}$ denoting the dimension of a subband’s complex
spectrum and $F$ the number of frames. Here, the real and imaginary parts are
concatenated to form a $d$-dimensional complex spectrum feature. During the
inference stage, the model operating in the time domain includes two
additional operations—STFT and ISTFT—at each step compared to the model
operating in the frequency domain. Despite this, it demonstrates slightly
superior performance, particularly in capturing high-frequency details. The
details of the comparative experiments are provided in Section 5.

###  3.3 Waveform Equalization or STFT Normalization

A white Gaussian noise signal has uniform energy distribution across all
frequency bands. However, the energy profiles of various waveform types vary
markedly among different frequency bands. For instance, the energy in a speech
waveform exhibits an exponential decay with increasing frequency, whereas a
music waveform tends to maintain a more consistent energy distribution across
frequencies. These disparities pose challenges for training diffusion models.
Consequently, it becomes advantageous to equalize the energy across waveform
frequency bands [15].

In the time-domain model, a bank of Pseudo-Quadrature Mirror Filters (PQMF) is
employed to decompose the input waveform into subbands. Subsequently, these
subbands are equalized and then recombined to form the equalized waveform. The
performance of the PQMF bank exhibits a modest enhancement compared to the
array of band-pass filters employed in [15]. In the frequency-domain model,
the waveform is transformed a complex spectrogram without equalization.
Subsequent processing involves the dimension-wise normalization of the complex
spectrogram feature. Mean-variance normalization, utilizing the running
averages of mean and variance computed during training, is applied for both
waveform equalization and STFT normalization. This approach ensures that the
transformation can be effectively inverted using the same running average
statistics.

###  3.4 Time-balanced Loss

In our preliminary experiments, we observe the presence of low-volume noise in
regions that are expected to be silent. We attribute this to the property of
mean square error (MSE) used in (2). The MSE measures the absolute distortion
between the predicted values and the ground truth. Since the values in silent
regions are close to zero, even a minor absolute distortion in predictions can
lead to a significant relative error. Consequently, models trained with the
MSE produce small absolute distortions in silent regions, which are then
perceived as noise.

We propose a time-balanced loss to mitigate this problem. Our time-balanced
loss is designed to weight errors differently depending on the region’s volume
accross the time-axis. Specifically, for each frequency subband, we compute
the standard deviation along the feature dimension of the ground truth
velocity to construct a weighting coefficient of size $[1,F]$. This vector is
reflective of the temporal volume of the respective subband, as depicted in
Figure A.1. Subsequently, both the ground truth and predicted velocity are
divided by this vector before proceeding to the subsequent steps. For the
frequency domain model, the training objective defined in (2) is adjusted as
follows:

$\displaystyle\min_{v}$
$\displaystyle\mathbb{E}_{X_{0}\sim\pi_{0},(X_{1},\mathcal{C})\sim
D}\left[\int_{0}^{1}\mid\mid(X_{1}-X_{0})/\sigma-v(X_{t},t\mid\mathcal{C})/\sigma\mid\mid^{2}\mathrm{d}t\right],$
(5) $\displaystyle\text{with}\leavevmode\nobreak\ \leavevmode\nobreak\
\sigma=\sqrt{\text{Var}_{1}(X_{1}-X_{0})}\leavevmode\nobreak\
\leavevmode\nobreak\ \leavevmode\nobreak\ \text{and}\leavevmode\nobreak\
\leavevmode\nobreak\ \leavevmode\nobreak\ X_{t}=tX_{1}+(1-t)X_{0},$

where $D$ represents the dataset with paired $X_{1}$ and $\mathcal{C}$, and
$\text{Var}_{1}$ calculates the variance along the feature dimension. For the
time domain model, this time balancing operation precedes the ISTFT process.
This approach helps to minimize the relative error in low-volume regions. Our
experimental results demonstrate that this method enhances overall
performance, benefiting not just the silent parts. In an alternative
interpretation, the equalization or normalization discussed previously brings
the features closer to a standard normal distribution along the feature
dimension. Simultaneously, the time-balanced loss aligns the features more
closely with a standard normal distribution along the temporal dimension,
prior to the calculation of the original MSE. This approach provides a more
nuanced adjustment of the features, facilitate the training of Rectified Flow.

##  4 Experiments

#### Overview

Initially, we evaluate models operating in both the time and frequency domains
on the LJSpeech [23] dataset, assessing their performance variations with and
without the incorporation of time-balanced loss. Following this preliminary
assessment, the configuration yielding the best results undergoes further
evaluation in diverse auditory scenarios, including singing, music, and
extensive speech datasets, to determine its comprehensive applicability and
efficiency.

#### Data

For the singing data, we utilize the Opencpop dataset [24], comprising 100
high-quality Mandarin songs performed by a professional female singer.
Regarding the music data, we employ the MTG-Jamendo dataset [25], featuring
over 55,000 full-length audio tracks, annotated with 195 tags spanning genres,
instruments, and mood/theme categories. For a comprehensive collection of
speech data, we resort to the LibriTTS corpus, a multi-speaker English dataset
encompassing roughly 600 hours of recordings made in diverse environments.
Additionally, for speech data, we expand our training to a broader dataset
encompassing LibriTTS-R[26], Aishell-3[27], VCTK[28], and the HQ-TTS mentioned
in [29]. To ensure audio quality, we evaluate each speaker’s recordings within
this dataset using the WADA[30] tool, excluding speakers whose majority of
tracks exhibit a Signal-to-Noise Ratio (SNR) below 25dB. We refer to this
dataset as Universal in the following discussion. For LJSepech, we allocate
250 sentences for testing. We set apart 20 segments for the Opencpop test. The
test-clean is employed for LibriTTS. The LibriTTS-R test-clean is used for
Universial. Lastly, we reserve 1397 audio files for the Jamendo test.

We preserve the original sampling rates: LJSpeech at 22.05 kHz, LibriTTS and
Universal at 24 kHz, and compute the Mel-scaled spectrograms with n_fft =
1024, hop_length = 256, and the number of Mel bins set to 100. For Opencpop
and MTG-Jamendo, we keep the original 44.1 kHz sampling rate and and compute
the Mel-scaled spectrograms with n_fft = 2048, hop_length = 512, and again set
the number of Mel bins to 100. When extracting the complex cofficients
utilized by the model, we use the orthonormal Fast Fourier Transform (FFT) and
its inverse (IFFT), with the normalization convention of dividing by
$1/\sqrt{N}$ for both operations, here $N$ is the number of FFT points. This
approach ensures the spectrogram extracted is within a more reasonable range
for modeling.

#### Implementation

The RFWave backbone contains 8 ConvNeXtV2 blocks. Within each ConvNeXtV2
block, the depth-wise convolutional layer featuring a large kernel utilizes a
kernel size of 7 and maintains a channel dimension of 512. The first and last
1x1 point-wise convolutional layers in the sequence possess channel dimensions
of 512 and 1536, respectively. As described in Subsection 3.1 , the complex
spectrogram is divided into 8 equally spanned subbands. Those subbands are not
related to the waveform equalization subbands mentioned in Subsection 3.3.

During training, audio samples are randomly cropped to lengths of 32512 and
65024 for 22.05/24 kHz and 44.1 kHz waveforms, respectively. This is
equivalent to a crop window of 128 frames for both sampling rates. We use a
batch size of 64. The model optimization is performed using the AdamW
optimizer with a starting learning rate of 2e-4 and beta parameters of (0.9,
0.999). A cosine annealing schedule is applied to reduce the learning rate to
a minimum of 2e-6. For evaluation purposes, we use 10 sampling steps unless
otherwise stated.

#### Baseline and Evaluation Metrics

We benchmark our RFWave model against Vocos [11]. We adopt the original
training details to retrain Vocos for the LJSpeech and Opencpop datasets. For
the LibriTTS dataset, we utilize the pre-trained model. To assess our models,
we employ the UTMOS [31] for automatic prediction of Mean Opinion Scores
(MOS), which acts as a proxy for subjective human assessments. We incorporate
additional metrics into our evaluation framework as well. These metrics
include the Perceptual Evaluation of Speech Quality (PESQ) and Mel Spectral
Signal-to-Noise Ratio (Mel-SNR) as proposed in [15]. Mel-SNR measures the
fidelity of the mel-spectrogram of the reconstructed signal compared with the
ground truth across multiple frequency bands. The results are presented for
low frequencies (Mel-SNR-L), mid frequencies (Mel-SNR-M), high frequencies
(Mel-SNR-H), and an average of these three ranges is provided as the overall
Mel-SNR-A.

##  5 Results

The performance metrics of the model, both in the frequency and time domains,
with and without the implementation of the time-balanced loss, are summarized
in Table 1. The model that operates in the time domain and incorporates the
time-balanced loss exhibits superior performance. Consequently, only the
outcomes from this particular configuration are reported for the other
datasets.

As observed from Table 1 and Table 2, RFWave consistently outperforms in terms
of PESQ scores, while Vocos routinely excels in UTMOS scores across different
datasets. This might be due to the subtle biases inherent in these metrics.
Additionally, it is observed that Vocos achieves superior performance in terms
of Mel-SNR-M and Mel-SHR-H metrics, even when compared with RFWave(100) on the
LibriTTS dataset, which utilizes 100 sampling steps. Nonetheless, upon
examining the spectrograms in Figure 2, it is evident that RFWave produces
more distinct harmonics at medium and high frequencies. The objective metrics
pertaining to the samples can be found in Table A.1. This observation might be
due to the fact that the Mel-SNR metric primarily assesses the accuracy of
energy distribution across different frequencies without taking into account
the precision of the phase information. Simultaneously, models such as Vocos
perform better in spectral distance metrics due to their specific training
aimed at content reconstruction. Conversely, diffusion-based methods, which do
not employ feature or spectrogram matching, tend to generate samples that are
more representative of the distribution, leading to more natural sounding
audio [15].

We have trained the RFWave model on the Universal dataset employing
100-dimensional Mel spectrograms, as detailed in Section 4. Concurrently, we
have also trained the model with 80-dimensional Mel spectrograms, which were
extracted using the Espnet toolkit, a prevalent setup in Text-to-Speech (TTS)
tasks. As evidenced in Table 2, the configuration of Mel spectrograms is
impactful, with the 100-dimensional configuration demonstrating superior
performance to the 80-dimensional one. Furthermore, the model yields
satisfactory reconstructions for the Jamendo dataset. Online demos are
available for further review333https://bfs18.github.io/rfwave/.

We performe inference speed benchmark tests using an Nvidia GeForce RTX 4090
GPU and an Intel Core i7-12700 CPU. The implementation was done in Pytorch
[32], and no specific hardware optimizations were applied. The inference was
carried out with a batch size of 1 sample, utilizing the LJSpeech test set.
Table 1 displays the synthesis speed and model size of RFWave and Vocos. Vocos
stands as a strong baseline given that it requires only a single forward pass
and operates at the frame level.

Table 1: Objective metrics comparing the model’s performance in frequency and time domains with and without the time-balanced loss (frequency noted as freq, time-balanced loss noted as tbl). Setting | UTMOS($\uparrow$) | PESQ($\uparrow$) | Mel-SNR-L($\uparrow$) | Mel-SNR-M($\uparrow$) | Mel-SNR-H($\uparrow$) | Mel-SNR-A($\uparrow$)
---|---|---|---|---|---|---
freq w/o tbl | 3.61 | 3.60 | 15.43 | 16.94 | 18.09 | 16.80
freq w/ tbl | 3.59 | 3.64 | 15.72 | 17.05 | 18.19 | 16.97
time w/o tbl | 3.84 | 3.96 | 16.32 | 17.40 | 19.10 | 17.60
time w/ tbl | 3.86 | 4.00 | 17.40 | 17.73 | 19.63 | 18.24
Vocos(ISTFT) | 4.09 | 3.54 | 16.71 | 18.44 | 20.64 | 18.57
groundtruth | 4.39 | - | - | - | - | -
Table 2: Objective evaluation metrics for RFWave and Vocos across various datasets.RFWave(Espnet) utilizes the 80-dimension mel from Espnet as a conditional input. Meanwhile, RFWave(100) conducts sampling for 100 steps. Dataset | Model | UTMOS($\uparrow$) | PESQ($\uparrow$) | Mel-SNR-L($\uparrow$) | Mel-SNR-M($\uparrow$) | Mel-SNR-H($\uparrow$) | Mel-SNR-A($\uparrow$)
---|---|---|---|---|---|---|---
Opencpop | RFWave | - | 3.30 | 17.69 | 16.42 | 20.54 | 18.19
Vocos(ISTFT) | - | 3.01 | 16.90 | 17.40 | 20.45 | 18.22
LibriTTS | RFWave | 3.41 | 3.67 | 17.06 | 16.43 | 17.96 | 17.14
RFWave(100) | 3.51 | 3.98 | 19.17 | 17.97 | 20.86 | 19.29
Vocos | 3.74 | 3.31 | 17.20 | 18.81 | 20.87 | 18.81
Universal | RFWave | 3.87 | 3.80 | 18.45 | 17.88 | 19.90 | 18.73
RFWave(Espnet) | 3.75 | 3.40 | 16.43 | 16.54 | 17.20 | 16.71
Jamendo | RFWave | - | - | 11.82 | 12.55 | 17.25 | 13.83
Table 3: Model footprint and synthesis speed. xRT stands for the speed at which the model can generate speech in comparison to real-time. A higher xRT value signifies that the model is capable of producing speech quicker than real-time, with a value of 1.0 representing the speed of real-time.  Model | parameters | GPU xRT($\uparrow$) | CPU xRT($\uparrow$)
---|---|---|---
RFWave | 18.1 M | 91.46 | 1.40
Vocos(ISTFT) | 13.5 M | 2078.20 | 143.84

(a) RFWave Opencpop

(b) Vocos Opencpop

(c) RFWave LibriTTS

(d) Vocos LibriTTS

Figure 2: Examples of spectrograms. The differences are emphasized by blue
rectangles. RFWave, in particular, produces clearer harmonics than Vocos,
especially in the higher frequency range.

##  6 Conclusion and Discussion on Text-to-Speech

In this study, we purpose RFWave, a multi-band Rectified Flow approach for
audio waveform reconstruction. The model has been carefully designed to
overcome the latency issues associated with diffusion models. RFWave stands
out for its ability to generate complex spectrograms by operating at the frame
level, processing all subbands concurrently. This concurrent processing
significantly enhances the efficiency of the waveform reconstruction process.
The empirical evaluations conducted in this research have demonstrated that
RFWave achieves exceptional reconstruction quality. Moreover, it has shown
superior computational efficiency by generating audio at a speed that is 90
times faster than real-time.

It would be relatively easy to implement the widely-used cascade pipeline to
develop a text-to-speech (TTS) system. This involves mapping text features to
Mel-spectrograms and then Mel-spectrograms to complex spectrograms using
Rectified Flow for both stages. Nevertheless, it is more advantageous to map
text features directly to complex spectrograms, especially in the context of
rapidly evolving large-scale TTS models. Large-scale TTS models typically
incorporate extensive corpora, and eliminating one stage of processing can
significantly reduce computational resource requirements. Additionally, this
direct approach limits the discrepancies that can arise between the two
stages. The infilling capabilities of Rectified Flow equip it to handle
diverse functions similar to those managed by prominent large-scale TTS
models, for example, replicating the speaker’s voice and speech style from a
provided audio prompt. We have conducted preliminary experiments in developing
a TTS system that maps text features directly to complex spectrograms based on
Rectified Flow. Currently, the results do not match those of the cascade
model. The code and model checkpoints are available in the repository. We
believe this approach warrants further investigation and plan to explore it in
future work.

## References

  * [1] Dario Rethage, Jordi Pons, and Xavier Serra.  A wavenet for speech denoising.  In 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 5069–5073. IEEE, 2018.
  * [2] Masanori Morise, Fumiya Yokomori, and Kenji Ozawa.  World: a vocoder-based high-quality speech synthesis system for real-time applications.  IEICE TRANSACTIONS on Information and Systems, 99(7):1877–1884, 2016.
  * [3] Nal Kalchbrenner, Erich Elsen, Karen Simonyan, Seb Noury, Norman Casagrande, Edward Lockhart, Florian Stimberg, Aaron Oord, Sander Dieleman, and Koray Kavukcuoglu.  Efficient neural audio synthesis.  In International Conference on Machine Learning, pages 2410–2419. PMLR, 2018.
  * [4] Aaron Oord, Yazhe Li, Igor Babuschkin, Karen Simonyan, Oriol Vinyals, Koray Kavukcuoglu, George Driessche, Edward Lockhart, Luis Cobo, Florian Stimberg, et al.  Parallel wavenet: Fast high-fidelity speech synthesis.  In International conference on machine learning, pages 3918–3926. PMLR, 2018.
  * [5] Wei Ping, Kainan Peng, and Jitong Chen.  Clarinet: Parallel wave generation in end-to-end text-to-speech.  arXiv preprint arXiv:1807.07281, 2018.
  * [6] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio.  Generative adversarial nets.  In Z. Ghahramani, M. Welling, C. Cortes, N. Lawrence, and K.Q. Weinberger, editors, Advances in Neural Information Processing Systems, volume 27. Curran Associates, Inc., 2014.
  * [7] Kundan Kumar, Rithesh Kumar, Thibault De Boissiere, Lucas Gestin, Wei Zhen Teoh, Jose Sotelo, Alexandre De Brebisson, Yoshua Bengio, and Aaron C Courville.  Melgan: Generative adversarial networks for conditional waveform synthesis.  Advances in neural information processing systems, 32, 2019.
  * [8] Ryuichi Yamamoto, Eunwoo Song, and Jae-Min Kim.  Parallel wavegan: A fast waveform generation model based on generative adversarial networks with multi-resolution spectrogram.  In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 6199–6203. IEEE, 2020.
  * [9] Jungil Kong, Jaehyeon Kim, and Jaekyoung Bae.  Hifi-gan: Generative adversarial networks for efficient and high fidelity speech synthesis.  Advances in Neural Information Processing Systems, 33:17022–17033, 2020.
  * [10] Takuhiro Kaneko, Kou Tanaka, Hirokazu Kameoka, and Shogo Seki.  istftnet: Fast and lightweight mel-spectrogram vocoder incorporating inverse short-time fourier transform.  In ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 6207–6211. IEEE, 2022.
  * [11] Hubert Siuzdak.  Vocos: Closing the gap between time-domain and fourier-based neural vocoders for high-quality audio synthesis.  CoRR, abs/2306.00814, 2023.
  * [12] Hui-Peng Du, Ye-Xin Lu, Yang Ai, and Zhen-Hua Ling.  Apnet2: High-quality and high-efficiency neural vocoder with direct prediction of amplitude and phase spectra.  arXiv preprint arXiv:2311.11545, 2023.
  * [13] Zhifeng Kong, Wei Ping, Jiaji Huang, Kexin Zhao, and Bryan Catanzaro.  Diffwave: A versatile diffusion model for audio synthesis.  arXiv preprint arXiv:2009.09761, 2020.
  * [14] Nanxin Chen, Yu Zhang, Heiga Zen, Ron J Weiss, Mohammad Norouzi, and William Chan.  Wavegrad: Estimating gradients for waveform generation.  arXiv preprint arXiv:2009.00713, 2020.
  * [15] Robin San Roman, Yossi Adi, Antoine Deleforge, Romain Serizel, Gabriel Synnaeve, and Alexandre Défossez.  From discrete tokens to high-fidelity audio using multi-band diffusion.  In Thirty-seventh Conference on Neural Information Processing Systems, 2023.
  * [16] Xingchao Liu, Chengyue Gong, and Qiang Liu.  Flow straight and fast: Learning to generate and transfer data with rectified flow.  In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net, 2023.
  * [17] Geng Yang, Shan Yang, Kai Liu, Peng Fang, Wei Chen, and Lei Xie.  Multi-band melgan: Faster waveform generation for high-quality text-to-speech.  In 2021 IEEE Spoken Language Technology Workshop (SLT), pages 492–498, 2021.
  * [18] J. Johnston.  A filter family designed for use in quadrature mirror filter banks.  In ICASSP ’80. IEEE International Conference on Acoustics, Speech, and Signal Processing, volume 5, pages 291–294, 1980.
  * [19] Sanghyun Woo, Shoubhik Debnath, Ronghang Hu, Xinlei Chen, Zhuang Liu, In So Kweon, and Saining Xie.  Convnext V2: co-designing and scaling convnets with masked autoencoders.  In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2023, Vancouver, BC, Canada, June 17-24, 2023, pages 16133–16142. IEEE, 2023.
  * [20] Alexandre Défossez, Jade Copet, Gabriel Synnaeve, and Yossi Adi.  High fidelity neural audio compression.  CoRR, abs/2210.13438, 2022.
  * [21] Diederik Kingma, Tim Salimans, Ben Poole, and Jonathan Ho.  Variational diffusion models.  In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems, volume 34, pages 21696–21707. Curran Associates, Inc., 2021.
  * [22] Jingjing Xu, Xu Sun, Zhiyuan Zhang, Guangxiang Zhao, and Junyang Lin.  Understanding and improving layer normalization.  In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019.
  * [23] Keith Ito and Linda Johnson.  The lj speech dataset.  https://keithito.com/LJ-Speech-Dataset/, 2017.
  * [24] Yu Wang, Xinsheng Wang, Pengcheng Zhu, Jie Wu, Hanzhao Li, Heyang Xue, Yongmao Zhang, Lei Xie, and Mengxiao Bi.  Opencpop: A high-quality open source chinese popular song corpus for singing voice synthesis.  In Hanseok Ko and John H. L. Hansen, editors, Interspeech 2022, 23rd Annual Conference of the International Speech Communication Association, Incheon, Korea, 18-22 September 2022, pages 4242–4246. ISCA, 2022.
  * [25] Dmitry Bogdanov, Minz Won, Philip Tovstogan, Alastair Porter, and Xavier Serra.  The mtg-jamendo dataset for automatic music tagging.  In Machine Learning for Music Discovery Workshop, International Conference on Machine Learning (ICML 2019), Long Beach, CA, United States, 2019.
  * [26] Yuma Koizumi, Heiga Zen, Shigeki Karita, Yifan Ding, Kohei Yatabe, Nobuyuki Morioka, Michiel Bacchiani, Yu Zhang, Wei Han, and Ankur Bapna.  Libritts-r: A restored multi-speaker text-to-speech corpus.  arXiv preprint arXiv:2305.18802, 2023.
  * [27] Yao Shi, Hui Bu, Xin Xu, Shaoji Zhang, and Ming Li.  Aishell-3: A multi-speaker mandarin tts corpus and the baselines.  arXiv preprint arXiv:2010.11567, 2020.
  * [28] Junichi Yamagishi, Christophe Veaux, and Kirsten MacDonald.  Cstr vctk corpus: English multi-speaker corpus for cstr voice cloning toolkit (version 0.92).  2019\.
  * [29] Haohe Liu, Qiuqiang Kong, Qiao Tian, Yan Zhao, DeLiang Wang, Chuanzeng Huang, and Yuxuan Wang.  Voicefixer: Toward general speech restoration with neural vocoder.  arXiv preprint arXiv:2109.13731, 2021.
  * [30] Chanwoo Kim and Richard M Stern.  Robust signal-to-noise ratio estimation based on waveform amplitude distribution analysis.  In Ninth Annual Conference of the International Speech Communication Association. Citeseer, 2008.
  * [31] Takaaki Saeki, Detai Xin, Wataru Nakata, Tomoki Koriyama, Shinnosuke Takamichi, and Hiroshi Saruwatari.  UTMOS: utokyo-sarulab system for voicemos challenge 2022.  In Hanseok Ko and John H. L. Hansen, editors, Interspeech 2022, 23rd Annual Conference of the International Speech Communication Association, Incheon, Korea, 18-22 September 2022, pages 4521–4525. ISCA, 2022.
  * [32] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio.  Generative adversarial nets.  In Z. Ghahramani, M. Welling, C. Cortes, N. Lawrence, and K.Q. Weinberger, editors, Advances in Neural Information Processing Systems, volume 27. Curran Associates, Inc., 2014.

##  Appendix A Supplementary Material

Figure A.1: The Volume and $\sigma$ exhibit consistent variation across the frames. Table A.1: Objective metrics for the two sentences as evaluated across different models. Data | Model | UTMOS($\uparrow$) | PESQ($\uparrow$) | Mel-SNR-L($\uparrow$) | Mel-SNR-M($\uparrow$) | Mel-SNR-H($\uparrow$) | Mel-SNR-A($\uparrow$)
---|---|---|---|---|---|---|---
Opencpop | RFWave | - | 3.54 | 18.31 | 16.44 | 20.14 | 18.27
2009000326 | Vocos(ISTFT) | - | 3.25 | 17.40 | 17.16 | 19.39 | 17.96
LibriTTS | RFWave | 3.87 | 3.71 | 17.73 | 16.97 | 17.97 | 17.55
121_121726_000005_000001 | Vocos(ISTFT) | 4.23 | 3.75 | 19.76 | 19.72 | 20.32 | 19.94