#  Frieren: Efficient Video-to-Audio Generation with Rectified Flow Matching

Yongqi Wang, Wenxiang Guo11footnotemark: 1, Rongjie Huang, Jiawei Huang, Zehan
Wang,
Fuming You, Ruiqi Li, Zhou Zhao
Zhejiang University
<EMAIL_ADDRESS>
Equal Contribution.

###### Abstract

Video-to-audio (V2A) generation aims to synthesize content-matching audio from
silent video, and it remains challenging to build V2A models with high
generation quality, efficiency, and visual-audio temporal synchrony. We
propose Frieren, a V2A model based on rectified flow matching. Frieren
regresses the conditional transport vector field from noise to spectrogram
latent with straight paths and conducts sampling by solving ODE, outperforming
autoregressive and score-based models in terms of audio quality. By employing
a non-autoregressive vector field estimator based on a feed-forward
transformer and channel-level cross-modal feature fusion with strong temporal
alignment, our model generates audio that is highly synchronized with the
input video. Furthermore, through reflow and one-step distillation with guided
vector field, our model can generate decent audio in a few, or even only one
sampling step. Experiments indicate that Frieren achieves state-of-the-art
performance in both generation quality and temporal alignment on VGGSound,
with alignment accuracy reaching 97.22%, and 6.2% improvement in inception
score over the strong diffusion-based baseline. Audio samples are available at
http://frieren-v2a.github.io.

##  1 Introduction

Recent advancements in deep generative models have brought revolutionary
enhancements to the quality and diversity of AI-generated content, including
text [25], images [27, 28, 1], videos [30, 21] and audios [17, 18]. Among
various content-generation tasks, video-to-audio (V2A) generation aims to
synthesize semantically relevant and temporally aligned audio from video
frames. Due to its immense potential for application in film dubbing, game
development, YouTube content creation and other areas, the task of V2A has
attracted widespread attention.

A widely applicable V2A solution is expected to have outstanding performance
in the following aspects: 1) audio quality: the generated audio should have
good perceptual quality, which is the fundamental requirement of the audio
generation task; 2) temporal alignment: the generated audio should not only
match the content but also align temporally with the video frames. This has a
significant impact on user experience due to keen human perception of audio-
visual information; and 3) generation efficiency: the model should be
efficient in terms of generation speed and resource utilization, which affects
its practicality for large-scale and high-throughput applications.

Currently, considerable methods have been proposed for this task, including
GAN-based models [3, 7], transformer-based autoregressive models [13, 29], and
a recent latent-diffusion-based model, Diff-Foley [23]. However, these methods
have not yet achieved a balanced and satisfactory performance across the above
aspects. 1) For audio quality, early GAN-based models suffer from poor quality
and lack practicality. Autoregressive and diffusion models make improvements
in generation quality, but still leave room for further advancement. 2) For
temporal alignment, autoregressive models lack the ability to align the
generated audio with the video explicitly. And due to the difficulty of
learning audio-visual alignment with the cross-attention-based conditional
mechanism solely, Diff-Foley relies on additional classifier guidance to
achieve good synchrony, which not only increases the model complexity but also
leads to instability when reducing sampling steps. 3) For generation
efficiency, autoregressive models suffer from high inference latency, while
Diff-Foley requires considerable sampling steps to achieve good generation
quality due to the curved sampling trajectories of diffusion models,
increasing the temporal overhead in inference. In a nutshell, existing methods
still leave significant room for improvement in performance.

In this paper, We introduce another generative modeling approach, namely
rectified flow matching [19], into the V2A task. This method regresses the
conditional transport vector field between noise and data distributions with
as straight trajectories as possible, and conducts sampling by solving the
corresponding ordinary differential equation (ODE). With simpler formulations,
our rectified-flow-based model achieves higher audio quality and diversity. To
improve temporal alignment, we adopt a non-autoregressive vector field
estimator network with a feed-forward transformer with no temporal-dimension
downsampling, thereby preserving temporal resolution. We also employ a
channel-level cross-modal feature fusion mechanism for conditioning,
leveraging the inherent alignment of audio-visual data and achieving strong
alignment. These designs lead to high synchrony between generated audio and
input video while upholding model simplicity. Moreover, through integrating
reflow and one-step distillation techniques, our model can generate decent
audio with a few, or even only one sampling step, significantly improving
generation efficiency.

We name our model Frieren for efficient video-to-audio generation framework
with rectfied flow matching. Experiments indicate that Frieren outperforms
strong baselines in terms of audio quality, generation efficiency, and
temporal alignment on VGGSound [2], achieving a 6.2% improvement in inception
score (IS) and a generation speed 7.3$\times$ that of Diff-Foley, as well as
temporal alignment accuracy of up to 97.22% in 25 steps. Additionally, Frieren
combining reflow and distillation achieves alignment accuracy of up to 97.85%
with just one step, with a 9.3$\times$ acceleration compared to 25-step
sampling, further boosting generation efficiency.

##  2 Related works

###  2.1 Video-to-audio generation

Video-to-audio (V2A) generation aims to synthesize audio of which content
matches the visual information of a video clip. RegNet [3] designs a time-
dependent visual encoder to extract appearance and motion features, which are
then fed to a GAN for audio generation. FoleyGAN [7] also utilizes GAN for
audio generation from visual features, together with a predicted action
category as the conditional input. SpecVQGAN [13] takes RGB and optical flow
of videos and uses a transformer to generate indices of a spectrogram VQVAE
autoregressively. Im2Wav [29] adopts two transformers for different temporal
resolutions and takes CLIP [26] features as the condition to generate VQVAE
indices. Du et al. [4] mimics the real-world foley methodology and introduces
an additional reference audio as the condition. Diff-Foley [23] designs an
audio-visual contrastive feature and adopts a latent diffusion to predict
spectrogram latents, achieving decent audio quality and inference speed.

In addition to training a whole model from scratch, some works integrate off-
the-shelf audio generation models with modality mappers or multimodal encoders
with joint embedding space for conditioning. V2A-Mapper [32] uses a
lightweight mapper to transfer CLIP embeddings of videos to CLAP [33]
embeddings as the condition for audio generation. Xing et al. [34] utilize an
ImageBind[8]-based latent aligner for conditional guidance in audio
generation. Despite the existence of plentiful works on V2A, there is still a
large room left for improvement in quality, synchrony, and efficiency.

###  2.2 Flow matching generative models

Flow matching [16] models the vector field of transport probability path from
noise to data samples. Compared to score-based models like DDPM [10], flow
matching achieves more stable and robust training together with superior
performance. Specifically, rectified flow matching [19] learns the transport
ODE to follow the straight paths connecting the noise and data points as much
as possible, reducing the transport cost, and achieving fewer sampling steps
with the reflow technique. This modeling paradigm has demonstrated excellent
performance in accelerating image generation [20, 5].

In the area of audio generation, Voicebox [14] builds a large-scale multi-task
speech generation model based on flow matching. Its successor, Audiobox [31],
extends the flow-matching-based model to a unified audio generation model with
natural language prompt guidance. Matcha-tts [24] trains an encoder-decoder
TTS model with optimal-transport conditional flow matching. VoiceFlow [9]
introduces rectified flow matching into TTS, achieving speech generation with
fewer inference steps. However, for the task of V2A, there has been no
exploration into utilizing flow matching models to enhance generation quality
or inference efficiency.

##  3 Method

###  3.1 Preliminary: rectified flow matching

We first introduce the basic principles of rectified flow matching (RFM) [19]
that we build our model upon. Conditional generation problems like V2A can be
viewed as a conditional mapping from a noise distribution $\bm{x}_{0}\sim
p_{0}(\bm{x})$ to a data distribution $\bm{x}_{1}\sim p_{1}(\bm{x})$. This
mapping can be further taken as a time-dependent changing process of
probability density (a.k.a. flow), determined by the ODE:

$\mathrm{d}\bm{x}=\bm{u}(\bm{x},t|\bm{c})\mathrm{d}t,t\in[0,1],$ (1)

where $t$ represents the time position, $\bm{x}$ is a point in the probability
density space at time $t$, $\bm{u}$ is the value of the transport vector field
(i.e., the gradient of the probability w.r.t $t$) at $\bm{x}$, and $\bm{c}$ is
the condition. In our case, the condition $\bm{c}$ is the visual features from
the video frames, while the data $\bm{x}_{1}$ is the compressed mel-
spectrogram latent of the corresponding audio from a pre-trained autoencoder.
The fundamental principle of flow matching generative model is to use a neural
network $\theta$ to regress the vector field $\bm{u}$ with the flow matching
objective:

$\mathcal{L}_{\mathrm{FM}}(\theta)=\mathbb{E}_{t,p_{t}(\bm{x})}\left\|\bm{v}(\bm{x},t|\bm{c};\theta)-\bm{u}(\bm{x},t|c)\right\|^{2},$
(2)

Figure 1: Illustration of the sampling process of our rectified-flow based V2A
architecture.

where $p_{t}(\bm{x})$ is the distribution of $\bm{x}$ at timestep $t$.
However, due to a lack of prior knowledge of target distribution
$p_{1}(\bm{x})$ and the forms of $p_{t}$ and $\bm{u}$, it is intractable to
directly compute $\bm{u}(\bm{x},t|c)$. As an alternative, conditional flow
matching objective, which is proven in [16] to have identical gradient as eq.
2 w.r.t $\theta$, is used for regression:

$\mathcal{L}_{\mathrm{CFM}}(\theta)=\mathbb{E}_{t,p_{1}(\bm{x_{1}}),p_{t}(\bm{x}|\bm{x}_{1})}\left\|\bm{v}(\bm{x},t|\bm{c};\theta)-\bm{u}(\bm{x},t|\bm{x}_{1},c)\right\|^{2}.$
(3)

Through designing specific probabilistic paths that enable efficient sampling
from $p_{t}(\bm{x}|\bm{x}_{1})$ and computing of
$\bm{u}(\bm{x},t|\bm{x}_{1},c)$, we achieve an unbiased estimation of
$\bm{u}(\bm{x},t|,c)$ with the CFM objective 3. Specifically, rectified flow
matching attempts to establish straight paths between noise and data, aiming
to facilitate sampling with larger step sizes and fewer steps. Given a noise-
data pair $(\bm{x}_{0},\bm{x}_{1})$, $\bm{x}$ is located at
$(1-t)\bm{x}_{0}+t\bm{x}_{1}$ at timestep $t$, with the vector field being
$\bm{u}(\bm{x},t|\bm{x}_{1},c)=\bm{x}_{1}-\bm{x}_{0}$, pointing from the noise
point to the data point. Hence, for each training step of the vector field
estimator, we simply sample the data point $\bm{x}_{1}$ and noise point
$\bm{x}_{0}$ from $p_{1}(\bm{x})$ and $p_{0}(\bm{x})$, respectively, and
optimize the network with the rectified flow matching (RFM) loss

$\left\|\bm{v}(\bm{x},t|\bm{c};\theta)-(\bm{x}_{1}-\bm{x}_{0})\right\|^{2}.$
(4)

Once the vector estimator network finishes training, we can adopt various
solvers to approximate the solution of the ODE
$\mathrm{d}\bm{x}=\bm{v}(\bm{x},t|\bm{c};\theta)$ at discretized time steps
for sampling. A simple and commonly used ODE solver is the Euler method:

$\bm{x}_{t+\epsilon}=\bm{x}+\epsilon\bm{v}(\bm{x},t|\bm{c};\theta)$ (5)

where $\epsilon$ is the step size. The sampled latent is fed to the decoder of
the spectrogram autoencoder for spectrogram reconstruction, and the result is
further used to reconstruct the audio waveform with a vocoder. Figure 1
provides a simple demonstration of the model’s sampling process.

###  3.2 Model architecture

Figure 2: Illustration of model architecture of Frieren at different levels.

##### Model overview

We illustrate the model architecture of Frieren at different levels in Figure
2. As shown in Figure 2(a), we first utilize a pre-trained visual encoder with
frozen parameters to extract a frame-level feature sequence from the video.
Usually, the video frame rate is lower than the temporal length per second of
the spectrogram latent. To align the visual feature sequence with the mel
latent at the temporal dimension for the cross-modal feature fusion mentioned
below, we adopt a length regulator, which simply duplicates each item in the
feature sequence by the ratio of the latent length per second and the video
frame rate for regulation. The regulated feature sequence is then fed to the
vector field estimator as the condition, together with $\bm{x}$ and $t$, to
get the vector field prediction $\bm{v}$.

##### Visual and audio representations

We conduct experiments with two types of visual representations. For a fair
comparison with Diff-Foley [23], we mainly utilize the CAVP feature proposed
in [23], which is a visual-audio contrastive feature considering both content
and temporal alignment. Meanwhile, to investigate the impact of visual feature
characteristics on model performance, we also attempt the visual feature from
MAViL [12], which is an advanced self-supervised visual-audio representation
learner that employs both masked-reconstruction and contrastive learning, and
exhibits formidable performance in audio-visual understanding (See section
4.3.2 for comparison). For audio representation, we follow a previous text-to-
audio work [11] to train a mel-spectrogram VAE with 1D convolution over the
temporal dimension. Details of the VAE are provided in appendix A

##### Vector field estimator

Figure 2(b) demonstrates the structure of the vector field estimator, which is
composed of a feed-forward transformer and some auxiliary layers. The
regularized visual feature $\bm{c}$ and the point $\bm{x}$ on the transport
path are first processed by stacks of shallow layers separately, with output
dimensions being both half of the transformer hidden dimension, and are then
concatenated along the channel dimension to realize cross-modal feature
fusion. This simple mechanism leverages the inherent alignment within the
video and audio, achieving enforced alignment without relying on learning-
based mechanisms such as attention. As a result, the generated audio and input
video sequences exhibit excellent temporal alignment. After appending the time
step embedding to the beginning, the sequence is added with a learnable
positional embedding and is then fed into the feed-forward transformer. The
structure of the transformer block is illustrated in Figure 2 (c), the design
of which is derived from the spatial transformer in latent diffusion [27],
with the 2D convolution layers replaced by 1D ones. The feed-forward
transformer does not involve temporal downsampling, thus preserving the
resolution of the temporal dimension and further ensuring the preservation of
alignment. The output of the stacked transformer blocks is then passed through
a normalization layer and a 1D convolution layer to finally obtain the
prediction of the vector field.

###  3.3 Re-weighting RFM objective with logit-normal coefficient

The original RFM objective samples uniformly over time span $[0,1]$. However,
for modeling the vector field, positions in the middle of the transport path
(equivalent to time steps in the middle of [0, 1]) present greater difficulty,
as these positions are distant from both noise and data distributions. On the
other hand, positions near the boundaries of the time span typically lie close
to corresponding noise or data points, and their vector field direction tends
to align with the lines connecting these points and the centroid of the
distribution on the opposite side, and therefore relatively easy to regress.
Upon this insight, we introduce time-based re-weighting to the original RFM
objective, allocating more weight to intermediate time steps to achieve better
modeling effectiveness. This is equivalent to increasing the sampling
frequency of intermediate time steps. In practice, logit-normal weighting
coefficients have been proven [5] to yield promising results, with the formula
being

$w(t)=\frac{1}{\sqrt{2\pi}}\frac{1}{t(1-t)}\exp{\left(-\frac{(\ln
t-\ln(1-t))^{2}}{2}\right)}.$ (6)

We re-weight the RFM objective with this weighting function to replace the
original objective.

###  3.4 Classifier-free guidance

Similar to diffusion-based models, we observe that classifier-free guidance
(CFG) is highly important for generating audio that semantically matches and
temporally aligns with the video. During training, we randomly replace the
condition sequence $\bm{c}$ with a zero tensor with a probability of 0.2, and
during sampling, we modify the vector field using the formula

$\bm{v}_{\mathrm{CFG}}(\bm{x},t|\bm{c};\theta)=\gamma\bm{v}(\bm{x},t|\bm{c};\theta)+(1-\gamma)\bm{v}(\bm{x},t|\varnothing;\theta),$
(7)

where $\gamma$ is the guidance scale trading off the sample diversity and
generation quality, and $\bm{v}_{\mathrm{CFG}}$ degenerates into the original
vector field $\bm{v}$ when $\gamma=1$. We set $\gamma$ to 4.5 in our major
experiments.

###  3.5 Reflow and one-step distillation with guided vector field

In this section, we introduce two techniques we adopt for reducing sampling
steps. The first one is reflow, which is a crucial component of the rectified
flow paradigm [19, 20]. Training the estimator network with objective 4 for
once is insufficient to construct straight enough transport paths, and an
extra reflow procedure is needed to strengthen the transport trajectories
without altering the marginal distribution learned by the model, enabling
sampling with larger step sizes and fewer steps. Given a model $\theta$
trained with RFM objective, the reflow procedure applies $\theta$ to conduct
sampling over the entire training dataset to obtain sampled data
$\hat{\bm{x}}_{1}$ and save the corresponding input noise
$\bm{x}_{0}^{\prime}$, finally obtaining triplets
$(\bm{x}_{0}^{\prime},\hat{\bm{x}}_{1},\bm{c})$. The noise-data pair
$(\bm{x}_{0},\bm{x}_{1})$ in the RFM objective 4 is replaced by
$(\bm{x}_{0}^{\prime},\hat{\bm{x}}_{1})$ for a secondary training of $\theta$.
This process can be repeated multiple times to obtain straighter trajectories
with diminishing marginal effects. We conduct reflow for once as it is
sufficient for achieving straight enough trajectories.

While many rectified-flow-based models regress the same velocity field
$\bm{v}$ during both the initial training and the reflow process, we observe
that when incorporating CFG, conducting sampling and reflow with the original
vector field $\bm{v}$ is ineffective in straightening the sampling
trajectories with the guided vector field $\bm{v}_{\mathrm{CFG}}$. Therefore,
we use $\bm{v}_{\mathrm{CFG}}$ for generating $\hat{\bm{x}}_{1}$ and as the
target of regression in reflow. The reflow objective can be written as:

$\mathcal{L}_{\mathrm{reflow}}(\theta^{\prime})=\mathbb{E}_{t,p(\bm{x}_{0}^{\prime},\hat{\bm{x}}_{1}|\bm{c}),p_{t}(\bm{x}|\bm{x}_{0}^{\prime},\hat{\bm{x}}_{1})}\left\|\bm{v}_{\mathrm{CFG}}(\bm{x},t|\bm{c};\theta^{\prime})-(\hat{\bm{x}}_{1}-\bm{x}_{0}^{\prime})\right\|^{2}$
(8)

with same weighting function as eq. 6.

Upon the model $\theta^{\prime}$ obtained from reflow, we further conduct one-
step distillation [19, 20] to enhance the single-step generation performance
of the model. As a type of self-distillation, this procedure tries to reduce
the error between the single-step sampling result
$\bm{x}_{0}^{\prime}+\bm{v}_{\mathrm{CFG}}(\bm{x}_{0}^{\prime},t|\bm{c};\theta)$
and the multi-step sampling result $\hat{\bm{x}}_{1}$. The objective function
can be written as:

$\mathcal{L}_{\mathrm{distill}}(\theta^{\prime\prime})=\mathbb{E}_{t,p(\bm{x}_{0}^{\prime},\hat{\bm{x}}_{1}|\bm{c}),p_{t}(\bm{x}|\bm{x}_{0}^{\prime},\hat{\bm{x}}_{1})}\left\|\bm{x}_{0}^{\prime}+\bm{v}_{\mathrm{CFG}}(\bm{x}_{0}^{\prime},t|\bm{c};\theta^{\prime\prime})-\hat{\bm{x}}_{1}\right\|^{2}$
(9)

Formally, the distillation objective 9 can be viewed as a reflow objective
with the sampling timestep fixed at $t=0$. We observe in the experiment that
due to a limited number of sampling steps in reflow data generation, the model
may experience a decrease in sampling quality after the reflow process.
Therefore, we opt to use the same training data used in reflow for
distillation, rather than re-sampling the training data with the reflow model,
which is based on the theoretical basis that reflow does not alter the
marginal distribution modeled by the estimator.

##  4 Experiments

Table 1: Results of V2A models on VGGSound dataset. R+F and RN50 denote the
RGB+Flow and ResNet50 versions of SpecVQGAN, and CG denotes classifier
guidance in Diff-Foley.

Model |  FD$\downarrow$ |  IS$\uparrow$ |  KL$\downarrow$ |  FAD$\downarrow$ |  KID$(10^{-3})\downarrow$ |  Acc(%) $\uparrow$ |  MOS-Q$\uparrow$ |  MOS-A$\uparrow$
---|---|---|---|---|---|---|---|---
SpecVQGAN (R+F) | 31.69 | 5.23 | 3.37 | 5.42 | 8.53 | 61.83 |  3.30 $\pm$ 0.06 |  2.35 $\pm$ 0.05
SpecVQGAN (RN50) | 32.52 | 5.21 | 3.41 | 5.39 | 9.00 | 56.92 |  3.25 $\pm$ 0.07 |  2.17 $\pm$ 0.05
Im2Wav | 14.98 | 7.20 | 2.57 | 5.49 | 3.35 | 56.70 |  3.39 $\pm$ 0.06 |  2.29 $\pm$ 0.06
Diff-Foley (CG ✓) | 23.94 | 11.11 | 3.38 | 4.72 | 9.58 | 95.03 |  3.57 $\pm$ 0.08 |  3.74 $\pm$ 0.07
Diff-Foley (CG ✗) | 24.97 | 11.69 | 3.23 | 7.10 | 10.32 | 92.53 |  3.64 $\pm$ 0.07 |  3.59 $\pm$ 0.06
DDPM | 11.79 | 10.09 | 2.86 | 1.77 | 2.36 | 95.33 |  3.72 $\pm$ 0.05 |  3.79 $\pm$ 0.07
Frieren (Ours) | 12.25 | 12.42 | 2.73 | 1.32 | 2.49 | 97.22 | 3.78 $\pm$ 0.06 | 3.90 $\pm$ 0.05

###  4.1 Experiment setup

##### Dataset and pre-processing

Following most previous works, we take VGGSound [2] as the benchmark, which
consists of 200k+ 10-second video clips from YouTube spanning 309 categories.
Excluding videos already removed from YouTube, we follow the original train
and test splits of VGGSound, the sizes of which are about 182.6k and 15.3k. We
downsample the audios to 16kHz and transform them to mel-spectrogram with 80
bins and a hop size of 256. We follow [23] to downsample the videos to 4 FPS.
Data samples are truncated to 8-second clips for training and inference.

##### Model configuration

The transformer of the vector field estimator mainly used in the experiments
has 4 layers and a hidden dimension of 576. Each model is trained with 2
NVIDIA RTX-4090 GPUs. We train the estimator for 1.3M steps for the first
training, and 600k and 500k steps for reflow and distillation, with the
learning rate being 5e-5 for all stages. For waveform generation, we train a
BigVGAN [15] vocoder on AudioSet [6]. Details of model parameters are provided
in appendix A.

##### Metrics

We combine objective and subjective metrics to evaluate model performance over
audio quality, diversity, and temporal alignment. For objective evaluation, we
calculate Frechet distance (FD), inception score (IS), Kullback–Leibler
divergence (KL), Frechet audio distance (FAD), kernel inception distance
(KID), and alignment accuracy (Acc). We utilize audio evaluation tools
provided by AudioLDM [17], which are widely used in audio generation tasks, as
well as the alignment classifier provided in [23]. For subjective evaluation,
we conduct crowd-sourced human evaluations with 1-5 Likert scales and report
mean-opinion-scores (MOS) over audio quality (MOS-Q) and content alignment
(MOS-A) with 95% confidence intervals (CI). We sample 10 audios for each test
video for evaluation. Details of subjective evaluation are provided in
appendix B.

##### Baseline models

We adopt three advanced V2A models as baselines, including: 1) SpecVQGAN [13],
a transformer-based autoregressive model generating spectrogram VQVAE indices
from visual features; 2) Im2Wav [29], a hierarchical autoregressive V2A model
predicting audio VQVAE indices conditioned on CLIP features; and 3) Diff-Foley
[23], a strong latent-diffusion-based V2A model. For SpecVQGAN, we evaluate
two versions using RGB+Flow and ResNet features as input visual conditions.
For Diff-Foley, we evaluate its performance with and without classifier
guidance to examine the impact of its complex external alignment mechanism. To
better validate the superiority of our rectified flow model, we also train a
diffusion model sharing the same architecture as Frieren but has a different
prediction target, labeled as DDPM in the following tables. We use DPM-Solver
[22] for all diffusion models, and Euler method 5 for our rectified flow
model.

###  4.2 Results and analysis

##### Video-to-audio generation results

The results of different models are illustrated in table 1. We sample with
diffusion models and Frieren with 25 steps, and report the result of Frieren
without reflow and distillation, which shows the best overall performance with
a high number of sampling steps. It can be seen that Frieren significantly
outperforms other models in IS, FAD, and alignment accuracy, with the values
reaching up to 12.42, 1.32, and 97.22%, together with the highest subjective
scores of 3.78 and 3.90 on quality and alignment. For FD, KL, and KID, the
scores of Frieren are also very close to the best values, with differences of
only 0.46, 0.16, and 0.13$\times 10^{-3}$. This indicates the effectiveness of
our approach. Generally, the performance of Frieren surpasses that of the
DDPM, demonstrating the superiority of rectified flow matching over the score-
based paradigm of diffusion. Additionally, both Frieren and DDPM outperform
Diff-Foley in temporal alignment, proving that our architecture design
achieves strong temporal alignment without the need for complex mechanisms,
and can produce audio that is highly synchronized with visual input.
Additionally, our model also has an advantage in sampling time, with details
provided in appendix C.

Table 2: Results of FRIEREN and the diffusion baselines under different sampling steps. CG denotes classifier guidance, R denotes reflow and D denotes one-step distillation. Model | Steps |  FD$\downarrow$ |  IS$\uparrow$ |  KL$\downarrow$ |  FAD$\downarrow$ |  KID$(10^{-3})\downarrow$ |  Acc(%) $\uparrow$ |  MOS-Q$\uparrow$ |  MOS-A$\uparrow$
---|---|---|---|---|---|---|---|---|---
Diff-Foley (CG ✓) |  1 | 82.61 | 2.31 | 4.44 | 13.64 | 43.96 | 31.60 |  1.28 $\pm$ 0.04 |  1.35 $\pm$ 0.03
Diff-Foley (CG ✗) | 86.97 | 1.86 | 4.17 | 14.66 | 39.73 | 37.02 |  1.17 $\pm$ 0.03 |  1.63 $\pm$ 0.04
DDPM | 66.11 | 2.69 | 3.99 | 12.49 | 25.98 | 52.21 |  2.30 $\pm$ 0.04 |  1.97 $\pm$ 0.03
Frieren (R ✗, D ✗) | 70.48 | 2.95 | 4.21 | 13.07 | 26.99 | 43.18 |  2.12 $\pm$ 0.04 |  1.71 $\pm$ 0.04
Frieren (R ✓, D ✗) | 18.61 | 6.63 | 2.60 | 3.13 | 3.49 | 94.96 |  3.32 $\pm$ 0.07 |  3.74 $\pm$ 0.06
Frieren (R ✓, D ✓) | 17.58 | 8.66 | 2.56 | 1.85 | 2.91 | 97.85 | 3.48 $\pm$ 0.06 | 3.93 $\pm$ 0.05
Diff-Foley (CG ✓) |  5 | 60.99 | 3.42 | 3.62 | 9.61 | 3.60 | 73.30 |  2.66 $\pm$ 0.07 |  2.98 $\pm$ 0.07
Diff-Foley (CG ✗) | 51.52 | 5.14 | 3.45 | 10.96 | 2.66 | 91.30 |  3.03 $\pm$ 0.08 |  3.56 $\pm$ 0.07
DDPM | 18.99 | 7.26 | 2.93 | 3.58 | 3.14 | 91.38 |  3.38 $\pm$ 0.06 |  3.60 $\pm$ 0.07
Frieren (R ✗, D ✗) | 28.78 | 6.69 | 3.02 | 4.34 | 8.56 | 87.69 |  3.30 $\pm$ 0.07 |  3.37 $\pm$ 0.08
Frieren (R ✓, D ✗) | 14.65 | 8.28 | 2.60 | 2.11 | 2.28 | 96.82 | 3.43 $\pm$ 0.06 | 3.83 $\pm$ 0.06
Diff-Foley (CG ✓) |  25 | 23.94 | 11.11 | 3.28 | 4.72 | 9.58 | 95.03 |  3.57 $\pm$ 0.08 |  3.74 $\pm$ 0.07
Diff-Foley (CG ✗) | 24.97 | 11.69 | 3.23 | 7.10 | 10.32 | 92.53 |  3.64 $\pm$ 0.07 |  3.59 $\pm$ 0.06
DDPM | 11.79 | 10.09 | 2.86 | 1.77 | 2.36 | 95.33 |  3.72 $\pm$ 0.05 |  3.79 $\pm$ 0.07
Frieren (R ✗, D ✗) | 12.25 | 12.42 | 2.73 | 1.32 | 2.49 | 97.22 | 3.78 $\pm$ 0.06 | 3.90 $\pm$ 0.05
Frieren (R ✓, D ✗) | 13.38 | 9.79 | 2.64 | 1.65 | 2.70 | 97.36 |  3.61 $\pm$ 0.07 |  3.88 $\pm$ 0.05
Figure 3: One-step generation results of different models. (a): The content of
the input video is a woman playing the violin. (b): Diff-Foley generates
meaningless audio with one step. (c, d): DDPM and Frieren without reflow
generate highly noisy audio. (e, f): reflow enables Frieren to generate
meaningful audio in one step, and distillation further improves the one-step
generation quality.

##### Few and single step generation results

Figure 4: IS and FAD of the models with different steps.

We further demonstrate the results of diffusion models and Frieren on reduced
sampling steps in table 2 to illustrate the impact of reflow and one-step
distillation, together with trend graphs of IS and FAD in figure 4 for
intuitive presentation. The data for reflow are generated with the Euler
method for 25 steps. We observe an obvious drop in performance of the
diffusion baselines as well as Frieren without reflow when sampling with as
few as 5 steps, and their scores become extremely poor when we further reduce
the step number to 1. Figure 3 (b) (c) and (d) illustrate that the audio
generated by these models degrades into unacceptably noisy or meaningless
audio within one step. This is due to the convoluted nature of the sampling
trajectories of these models, which disables them from sampling with large
step sizes and few steps. We also notice that when sampling with 5 steps,
using additional classifier guidance deteriorates the audio quality and
synchrony of Diff-Foley, where alignment accuracy and IS drop by 18.0% and
1.72 respectively, while FD, KL, and KID increase by 9.47, 0.17, and 0.94
$\times 10^{-3}$. This indicates the lack of robustness of the complex
alignment mechanism that Diff-Foley relies on.

In contrast, Frieren with reflow achieves an alignment accuracy of up to
96.82% in just 5 steps, with significant advantages in quality, diversity, and
subjective metrics. Additionally, it maintains an accuracy of 94.96% in
single-step generation, as well as decent quality and diversity. This proves
that reflow functions significantly in straightening the sampling
trajectories, enabling the rectified flow model to generate decent audio with
a small number of sampling steps. Furthermore, single-step distillation
following reflow further improves the model performance with one step, with
alignment accuracy reaching up to 97.85%, and KL, FAD, and KID being very
close to the 25-step results of Frieren trained once, with differences of only
0.08, 0.2 and 2.1$\times 10^{-3}$. It also achieves high MOS-Q and MOS-A of
3.48 and 3.93.

Figure 3 (e) and (f) show that results from Frieren with reflow and
reflow+distillation have distinguishable spectrograms, with the latter showing
higher quality and sharper edges. This fully demonstrates that the combination
of reflow and one-step distillation endows our model with strong single-step
generation capabilities, significantly enhancing the efficiency on the V2A
task. Notice that reflow brings in some quality degradation in sampling with
25 steps. We speculate that this is because the limited number of sampling
steps restricts the data quality when generating data for reflow, resulting in
a shift in the marginal distribution learned by the model. This cumulative
error might be mitigated by increasing the number of sampling steps during
reflow data generation.

###  4.3 Ablation study

####  4.3.1 Model size of vector field estimator

We adjust the number of parameters of the vector field estimator and evaluate
the model performance at different scales. We label the major model as “base”,
and obtain “small” and “large” models by decreasing and increasing the hidden
dimension and / or the number of transformer layers, respectively. The
parameter counts of the estimator and results are presented in table 3.

We observe that when the model parameters are reduced from about 159M to 71M,
performance declines across all metrics, where FD, KL, FAD, and KID increase
by 0.77, 0.05, 0.18, and 0.3$\times 10^{-3}$, and IS, alignment accuracy,
MOS-Q and MOS-A drop by 0.26, 1.18%, 0.07 and 0.07, respectively. On the other
hand, when the parameter number increases to 421M, the model obtains a
performance improvement across multiple metrics, with FD, KL, FAD, and KID
declining by 0.06, 0.05, 0.02, and 0.08$\times 10^{-3}$, and IS, alignment
accuracy, and MOS-Q increasing by 0.03, 0.09%, and 0.03. This result aligns
with our common sense that a larger model capacity leads to better
performance. Moreover, it indicates the potential scalability of our model to
large model size and amount of data, which may bring it broader prospects for
applications. Details of model parameters are provided in appendix A.

Table 3: Ablation results on different model size of vector field estimator
network.

Model Size |  FD$\downarrow$ |  IS$\uparrow$ |  KL$\downarrow$ |  FAD$\downarrow$ |  KID$(10^{-3})\downarrow$ |  Acc(%) $\uparrow$ |  MOS-Q$\uparrow$ |  MOS-A$\uparrow$
---|---|---|---|---|---|---|---|---
Small (70.90 M) | 13.02 | 12.16 | 2.78 | 1.50 | 2.79 | 96.04 |  3.71 $\pm$ 0.07 |  3.83 $\pm$ 0.06
Base (158.88 M) | 12.25 | 12.42 | 2.73 | 1.32 | 2.49 | 97.22 |  3.78 $\pm$ 0.06 |  3.90 $\pm$ 0.05
Large (421.12 M) | 12.19 | 12.45 | 2.68 | 1.30 | 2.41 | 97.31 |  3.81 $\pm$ 0.07 |  3.90 $\pm$ 0.06

####  4.3.2 Visual feature characteristics

In table 4, we compare the results of Frieren using two different types of
visual features from CAVP and MAViL. Intuitively, the MAViL feature should be
more robust and contain richer audio-related semantic information, as it
utilizes masked-reconstruction together with inter-modal and intra-modal
contrastive learning, in contrast to CAVP trained solely with inter-modal
contrastive learning. On the other hand, however, due to MAViL’s convolutional
downsampling in the temporal dimension, its feature sequence has a lower
effective FPS of 2 with the same 4 FPS video input as CAVP. The results in the
table indicate that the model with MAViL feature excels in audio quality and
diversity, with differences of FD, IS, KL, and FAD being 0.17, 0.25, 0.24, and
0.06. Meanwhile, it exhibits a 7.05% decrease in alignment accuracy. This
result yields two insights for V2A tasks: 1) at relatively low frame rates,
the frame rate of features, rather than content, is more likely to become the
bottleneck for generating visual-synchronized audio; 2) compared to high video
frame rates, the semantic information and robustness of visual features are
more crucial for the quality of generated audio.

Table 4: Results on different types visual features.

Type | Feat. FPS |  FD$\downarrow$ |  IS$\uparrow$ |  KL$\downarrow$ |  FAD$\downarrow$ |  KID$(10^{-3})\downarrow$ |  Acc(%) $\uparrow$ |  MOS-Q$\uparrow$ |  MOS-A$\uparrow$
---|---|---|---|---|---|---|---|---|---
CAVP [23] | 4 | 12.25 | 12.42 | 2.73 | 1.32 | 2.49 | 97.22 |  3.78 $\pm$ 0.06 |  3.90 $\pm$ 0.05
MAViL [12] | 2 | 12.08 | 12.67 | 2.49 | 1.26 | 2.52 | 90.17 |  3.87 $\pm$ 0.06 |  3.46 $\pm$ 0.07

####  4.3.3 Classifier-free guidance scale

Figure 5: Model performance of Frieren under different CFG scales.

In figure 5, we illustrate the impact of various CFG scales on the performance
of Frieren. In terms of audio diversity (FD, KL, KID, FAD), the metrics
initially increase with the CFG scale, reaching an optimal value at around 2
and 3. After that, the metrics go down as the increasing CFG scale suppresses
the diversity. For audio quality (IS) and temporal alignment, as larger scales
make the content of the generated audio closer to the visual information, the
metrics initially increase with the scale, reaching an optimal value between 4
and 4.5, and decrease after that due to audio distortion. We prioritize audio
quality and synchrony and adopt a CFG scale of 4.5.

####  4.3.4 Re-weighting RFM objective

We conduct ablation on RFM objective re-weighting and report the results in
table 5. We can see that discarding objective re-weighting leads to a
comprehensive deterioration in FD, IS, KL, FAD, and alignment accuracy, with
differences being 0.13, 0.22, 0.01, and 0.07. It also leads to a drop of MOS-Q
and MOS-A. This validates the positive impact on model performance of
objective re-weighting as an equivalent operation of adjusting sampling
probabilities at different time positions.

Table 5: Ablation results on RFM objective re-weighting.

Re-weighting |  FD$\downarrow$ |  IS$\uparrow$ |  KL$\downarrow$ |  FAD$\downarrow$ |  KID$(10^{-3})\downarrow$ |  Acc(%) $\uparrow$ |  MOS-Q$\uparrow$ |  MOS-A$\uparrow$
---|---|---|---|---|---|---|---|---
✗ | 12.38 | 12.20 | 2.74 | 1.39 | 2.44 | 97.04 |  3.74 $\pm$ 0.07 |  3.82 $\pm$ 0.06
✓ | 12.25 | 12.42 | 2.73 | 1.32 | 2.49 | 97.22 |  3.78 $\pm$ 0.06 |  3.90 $\pm$ 0.05

##  5 Conclusion

In this paper, we propose Frieren, an efficient video-to-audio generation
model based on rectified flow matching. We use a neural network to regress the
conditional transport vector field with straight paths from noise to
spectrogram latents, and conduct sampling by solving ODE, achieving better
performance than diffusion-based and other V2A models. We adopt a vector field
estimator based on a feed-forward transformer as well as channel-level cross-
modal feature fusion to realize strong audio-video synchrony. Through a
combination of reflow and one-step distillation, our model can generate high-
quality audio with a few or even one sampling step, boosting the generation
efficiency significantly. Experiments show that our model achieves state-of-
the-art V2A performance on VGGSound. For future work, we will explore
extending the model to larger scales and larger datasets to achieve V2A
generation on a broader data domain. Besides, we will attempt audio generation
from longer video sequences with variable lengths, rather than being limited
to fixed-length short clips. These efforts aim to build a more versatile and
widely applicable V2A model.

## References

  * Betker et al. [2023] James Betker, Gabriel Goh, Li Jing, Tim Brooks, Jianfeng Wang, Linjie Li, Long Ouyang, Juntang Zhuang, Joyce Lee, Yufei Guo, et al.  Improving image generation with better captions.  _Computer Science. https://cdn. openai. com/papers/dall-e-3. pdf_ , 2(3):8, 2023.
  * Chen et al. [2020a] Honglie Chen, Weidi Xie, Andrea Vedaldi, and Andrew Zisserman.  Vggsound: A large-scale audio-visual dataset.  In _ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , pp. 721–725. IEEE, 2020a.
  * Chen et al. [2020b] Peihao Chen, Yang Zhang, Mingkui Tan, Hongdong Xiao, Deng Huang, and Chuang Gan.  Generating visually aligned sound from videos.  _IEEE Transactions on Image Processing_ , 29:8292–8302, 2020b.
  * Du et al. [2023] Yuexi Du, Ziyang Chen, Justin Salamon, Bryan Russell, and Andrew Owens.  Conditional generation of audio from video via foley analogies.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pp. 2426–2436, 2023.
  * Esser et al. [2024] Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas Müller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, et al.  Scaling rectified flow transformers for high-resolution image synthesis.  _arXiv preprint arXiv:2403.03206_ , 2024.
  * Gemmeke et al. [2017] Jort F Gemmeke, Daniel PW Ellis, Dylan Freedman, Aren Jansen, Wade Lawrence, R Channing Moore, Manoj Plakal, and Marvin Ritter.  Audio set: An ontology and human-labeled dataset for audio events.  In _2017 IEEE international conference on acoustics, speech and signal processing (ICASSP)_ , pp. 776–780. IEEE, 2017.
  * Ghose & Prevost [2022] Sanchita Ghose and John J Prevost.  Foleygan: Visually guided generative adversarial network-based synchronous sound generation in silent videos.  _IEEE Transactions on Multimedia_ , 2022.
  * Girdhar et al. [2023] Rohit Girdhar, Alaaeldin El-Nouby, Zhuang Liu, Mannat Singh, Kalyan Vasudev Alwala, Armand Joulin, and Ishan Misra.  Imagebind: One embedding space to bind them all.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pp. 15180–15190, 2023.
  * Guo et al. [2024] Yiwei Guo, Chenpeng Du, Ziyang Ma, Xie Chen, and Kai Yu.  Voiceflow: Efficient text-to-speech with rectified flow matching.  In _ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , pp. 11121–11125. IEEE, 2024.
  * Ho et al. [2020] Jonathan Ho, Ajay Jain, and Pieter Abbeel.  Denoising diffusion probabilistic models.  _Advances in neural information processing systems_ , 33:6840–6851, 2020.
  * Huang et al. [2023] Jiawei Huang, Yi Ren, Rongjie Huang, Dongchao Yang, Zhenhui Ye, Chen Zhang, Jinglin Liu, Xiang Yin, Zejun Ma, and Zhou Zhao.  Make-an-audio 2: Temporal-enhanced text-to-audio generation.  _arXiv preprint arXiv:2305.18474_ , 2023.
  * Huang et al. [2024] Po-Yao Huang, Vasu Sharma, Hu Xu, Chaitanya Ryali, Yanghao Li, Shang-Wen Li, Gargi Ghosh, Jitendra Malik, Christoph Feichtenhofer, et al.  Mavil: Masked audio-video learners.  _Advances in Neural Information Processing Systems_ , 36, 2024.
  * Iashin & Rahtu [2021] Vladimir Iashin and Esa Rahtu.  Taming visually guided sound generation.  In _The 32st British Machine Vision Virtual Conference_. BMVA Press, 2021.
  * Le et al. [2024] Matthew Le, Apoorv Vyas, Bowen Shi, Brian Karrer, Leda Sari, Rashel Moritz, Mary Williamson, Vimal Manohar, Yossi Adi, Jay Mahadeokar, et al.  Voicebox: Text-guided multilingual universal speech generation at scale.  _Advances in neural information processing systems_ , 36, 2024.
  * Lee et al. [2022] Sang-gil Lee, Wei Ping, Boris Ginsburg, Bryan Catanzaro, and Sungroh Yoon.  Bigvgan: A universal neural vocoder with large-scale training.  In _The Eleventh International Conference on Learning Representations_ , 2022.
  * Lipman et al. [2022] Yaron Lipman, Ricky TQ Chen, Heli Ben-Hamu, Maximilian Nickel, and Matthew Le.  Flow matching for generative modeling.  In _The Eleventh International Conference on Learning Representations_ , 2022.
  * Liu et al. [2023a] Haohe Liu, Zehua Chen, Yi Yuan, Xinhao Mei, Xubo Liu, Danilo Mandic, Wenwu Wang, and Mark D Plumbley.  Audioldm: Text-to-audio generation with latent diffusion models.  _arXiv preprint arXiv:2301.12503_ , 2023a.
  * Liu et al. [2023b] Haohe Liu, Qiao Tian, Yi Yuan, Xubo Liu, Xinhao Mei, Qiuqiang Kong, Yuping Wang, Wenwu Wang, Yuxuan Wang, and Mark D Plumbley.  Audioldm 2: Learning holistic audio generation with self-supervised pretraining.  _arXiv preprint arXiv:2308.05734_ , 2023b.
  * Liu et al. [2022] Xingchao Liu, Chengyue Gong, et al.  Flow straight and fast: Learning to generate and transfer data with rectified flow.  In _The Eleventh International Conference on Learning Representations_ , 2022.
  * Liu et al. [2023c] Xingchao Liu, Xiwen Zhang, Jianzhu Ma, Jian Peng, et al.  Instaflow: One step is enough for high-quality diffusion-based text-to-image generation.  In _The Twelfth International Conference on Learning Representations_ , 2023c.
  * Liu et al. [2024] Yixin Liu, Kai Zhang, Yuan Li, Zhiling Yan, Chujie Gao, Ruoxi Chen, Zhengqing Yuan, Yue Huang, Hanchi Sun, Jianfeng Gao, et al.  Sora: A review on background, technology, limitations, and opportunities of large vision models.  _arXiv preprint arXiv:2402.17177_ , 2024.
  * Lu et al. [2022] Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu.  Dpm-solver: A fast ode solver for diffusion probabilistic model sampling in around 10 steps.  _Advances in Neural Information Processing Systems_ , 35:5775–5787, 2022.
  * Luo et al. [2024] Simian Luo, Chuanhao Yan, Chenxu Hu, and Hang Zhao.  Diff-foley: Synchronized video-to-audio synthesis with latent diffusion models.  _Advances in Neural Information Processing Systems_ , 36, 2024.
  * Mehta et al. [2024] Shivam Mehta, Ruibo Tu, Jonas Beskow, Éva Székely, and Gustav Eje Henter.  Matcha-tts: A fast tts architecture with conditional flow matching.  In _ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , pp. 11341–11345. IEEE, 2024.
  * Radford et al. [2018] Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, et al.  Improving language understanding by generative pre-training.  2018\.
  * Radford et al. [2021] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al.  Learning transferable visual models from natural language supervision.  In _International conference on machine learning_ , pp. 8748–8763. PMLR, 2021.
  * Rombach et al. [2022] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer.  High-resolution image synthesis with latent diffusion models.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pp. 10684–10695, 2022.
  * Saharia et al. [2022] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al.  Photorealistic text-to-image diffusion models with deep language understanding.  _Advances in neural information processing systems_ , 35:36479–36494, 2022.
  * Sheffer & Adi [2023] Roy Sheffer and Yossi Adi.  I hear your true colors: Image guided audio generation.  In _ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , pp. 1–5. IEEE, 2023.
  * Singer et al. [2022] Uriel Singer, Adam Polyak, Thomas Hayes, Xi Yin, Jie An, Songyang Zhang, Qiyuan Hu, Harry Yang, Oron Ashual, Oran Gafni, et al.  Make-a-video: Text-to-video generation without text-video data.  _arXiv preprint arXiv:2209.14792_ , 2022.
  * Vyas et al. [2023] Apoorv Vyas, Bowen Shi, Matthew Le, Andros Tjandra, Yi-Chiao Wu, Baishan Guo, Jiemin Zhang, Xinyue Zhang, Robert Adkins, William Ngan, et al.  Audiobox: Unified audio generation with natural language prompts.  _arXiv preprint arXiv:2312.15821_ , 2023.
  * Wang et al. [2024] Heng Wang, Jianbo Ma, Santiago Pascual, Richard Cartwright, and Weidong Cai.  V2a-mapper: A lightweight solution for vision-to-audio generation by connecting foundation models.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 38, pp. 15492–15501, 2024.
  * Wu et al. [2023] Yusong Wu, Ke Chen, Tianyu Zhang, Yuchen Hui, Taylor Berg-Kirkpatrick, and Shlomo Dubnov.  Large-scale contrastive language-audio pretraining with feature fusion and keyword-to-caption augmentation.  In _ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , pp. 1–5. IEEE, 2023.
  * Xing et al. [2024] Yazhou Xing, Yingqing He, Zeyue Tian, Xintao Wang, and Qifeng Chen.  Seeing and hearing: Open-domain visual-audio generation with diffusion latent aligners.  _arXiv preprint arXiv:2402.17723_ , 2024.

##  Appendix A Model parameters and architecture details

Table 6: Architecture details of 1D VAE for spectrogram compression. Hyperparameter | 1D VAE
---|---
Input tensor shape for 10-sec audio | (80,624)
Embedding dimension | 20
Channels | 224
Channel multiplier | 1, 2, 4
Downsample layer position | after block 1
Attention layer position | after block 3
Output tensor shape for 10-sec audio | (20,312)
Table 7: Hyperparameters of the vector field estimator of Frieren with different sizes. Hyperparameter | Small | Base | Large
---|---|---|---
Layers | 4 | 4 | 6
Hidden dimension | 384 | 576 | 768
Attention heads | 8 | 8 | 8
Conv1D-FFN dimension | 1,536 | 2,304 | 3,072
Number of parameters | 70.90M | 158.88M | 421.12M

In table 6, we provide the architecture details of the mel-spectrogram VAE.
Different from the commonly used 2D VAE for spectrogram, the 1D VAE we adopt
does not involve an extra channel dimension, but takes the frequency axis of
the spectrogram as the channel dimension, and conducts convolution along the
temporal axis. This design is derived from the insight that the spectrogram is
not translation invariant along the frequency axis, and it can better
synergize with the feed-forward transformer. In table 7, we present the
hyperparameters of the vector field estimator networks with different sizes.

Additionally, we observe that although there is no significant difference in
objective metrics, initializing the vector field estimator with the weights of
a diffusion model for text-to-audio (T2A) generation [11] helps improve the
subjective perceptual quality of the generated audio marginally. This
improvement derives from the knowledge of audio generation on a broader data
domain learned by the T2A model. We adopt this trick in our model training.

##  Appendix B Subjective evaluation

Figure 6: Screenshot of subjective evaluation on audio quality. Figure 7:
Screenshot of subjective evaluation on temporal alignment.

For each evaluated model, we randomly select 150 items for subjective
evaluation, accounting for about 1% of the entire test split.

Our subjective evaluation tests are crowd-sourced and conducted via Amazon
Mechanical Turk. For audio quality evaluation, we ask the testers to examine
the audio quality and ignore the content. And for temporal alignment, we
instruct the testers to evaluate the synchrony between the background audio
and the video content, while ignoring the audio quality. The testers rate
scores on 1-5 Likert scales. We provide screenshots of the testing interfaces
in figure 6 and 7. Each data item is rated by 6 testers, and the testers are
paid $8 hourly.

##  Appendix C Time efficiency

Table 8: Inference time per sample of different models with batch size = 1. Model | Inference Time (sec)
---|---
SpecVQGAN | 3.936
Im2Wav | 333.246
Diff-Foley (step=25) | 2.104
Frieren (step=25) | 0.288
Frieren (step=5) | 0.064
Frieren (step=1) | 0.031

In table 8, we compare the inference time per sample of different models. The
inference is conducted on a single RTX-4090 GPU with a batch size of 1. We can
see that the inference procedure of transformer-based autoregressive models,
including SpecVQGAN and Im2Wav, is more time-consuming, especially for Im2Wav,
which takes several minutes to generate a single sample. This is because
Im2Wav conducts a cascaded generation with 2 transformers. Moreover, its use
of high-bitrate audio VQVAE results in very long sequences of audio
representation, significantly increasing the inference time required for the
transformers, which has quadratic time complexity concerning sequence length.
In contrast, Diff-Foley and Frieren require less inference time, and Frieren
enjoys a higher speed, achieving 7.3 times faster than Diff-Foley with 25
sampling steps. This is the result of a combination of multiple factors,
including model architecture, model parameters, sampling methods, and so on.
Furthermore, when using Frieren model with reflow and one-step distillation,
we can generate 5-step sampled audio in 0.064 seconds and 1-step sampled audio
in just 0.031 seconds, achieving 4.5$\times$ and 9.3$\times$ acceleration
compared to 25-step sampling. This demonstrates the extremely high generation
efficiency of our model on the task of V2A.

##  Appendix D Limitations and boarder impacts

##### Limitations

Despite that Frieren achieves outstanding performance on audio quality,
temporal alignment, and generation efficiency, it still has two major
limitations: 1) Currently, experiments have only been conducted on a small-
scale dataset, VGGSound, and we have not yet scaled the model to large-scale
datasets. Therefore, it is still difficult to apply our model to a wide range
of real-world scenarios for now; 2) our current model design only targets
audio generation for fixed-length short video clips, and it lacks the ability
of audio generation for long videos with various lengths. We will explore the
solutions to these issues in future work.

##### Potential positive impacts

The achievements of our model on the V2A task may reduce the cost of sound
effect synthesis, and could potentially drive advancements in the film,
gaming, and social media industries.

##### Potential negative social impacts

The automatic sound effect generation technology may lead to job losses for
related personnel. Additionally, there is a risk of the model being used to
generate harmful content or fake media. Constraints are needed to guarantee
that people will not use the model in illegal cases.