# VidMuse: A Simple Video-to-Music Generation Framework with Long-Short-Term
Modeling

Zeyue Tian1 Zhaoyang Liu1∗ Ruibin Yuan1 Jiahao Pan1 Xiaoqiang Huang1
Qifeng Liu1 Xu Tan2 Qifeng Chen1 Wei Xue1† Yike Guo1 Equal
contributionCorresponding author 1 The Hong Kong University of Science and
Technology
2 Microsoft Research Asia

###### Abstract

In this work, we systematically study music generation conditioned solely on
the video. First, we present a large-scale dataset comprising 190K video-music
pairs, including various genres such as movie trailers, advertisements, and
documentaries. Furthermore, we propose VidMuse, a simple framework for
generating music aligned with video inputs. VidMuse stands out by producing
high-fidelity music that is both acoustically and semantically aligned with
the video. By incorporating local and global visual cues, VidMuse enables the
creation of musically coherent audio tracks that consistently match the video
content through Long-Short-Term modeling. Through extensive experiments,
VidMuse outperforms existing models in terms of audio quality, diversity, and
audio-visual alignment. The code and datasets will be available at
https://github.com/ZeyueT/VidMuse/.

##  1 Introduction

Music is an essential element of video production, as it can enhance emotional
feelings and convey the theme of the video content. Along with the development
of social media platforms _i.e._ , YouTube and TikTok, some studies [53, 12,
56] have shown that a piece of melodious music can vastly attract the
audience’s attention and interest in watching the video. It thus leads to a
great demand for video-to-music generation, which has drawn considerable
attention from some researchers [16, 37, 70, 23, 33, 30].

Nevertheless, music creation for video is a challenging task, which requires
both musical skills and deep understanding of the video semantics. It would be
very time-consuming to produce a piece of suitable music for video in a hand-
crafted manner. Therefore, it is desirable when we can automatically generate
high-quality music for different genres of videos.

Currently, most of works [11, 34, 86, 22, 35, 35, 66] have made significant
achievements, especially in text-to-music generation, but the video-to-music
generation still remains to be further studied. Specifically, existing works
on video-conditioned music generation mainly focus on specific scenarios, such
as dance videos [44, 95], or on the symbolic music, _i.e._ , MIDI [80, 16, 96,
37]. However, these works are unable to generate more diverse musical styles
and are also difficult to generalize to various video genres. Moreover, Hong
_et al._ [33] build a music–video retrieval dataset from YouTube-8M [3],
albeit with limited video genres. Despite that there are also some prominent
works [36, 70] employing multi-modal inputs to generate music for video, it is
still worth studying whether it is possible to generate high-quality and
visually consistent music solely based on the video input, while also
generalizing to diverse genres of videos.

Motivated by this, we first construct a large-scale dataset termed V2M,
equipped with a comprehensive benchmark to evaluate the state-of-the-art works
thoroughly. The video-music pairs are collected from YouTube with various
genres, _e.g._ , movie trailers, advertisements, documentaries, vlogs, _etc_.
In order to ensure the quality of our dataset, we establish a multi-step
pipeline illustrated in Fig. 2 to systematically clean and pre-process data.
The videos with low quality or composed of static images are filtered out. The
proposed dataset contains three subsets: V2M-190K for pretraining, V2M-20K for
finetuning, and V2M-bench for evaluation. We believe that V2M is able to
facilitate the development of video-to-music generation.

Furthermore, on top of V2M, we propose a simple yet effective method, termed
as VidMuse, to generate music only conditioned on the visual input. Instead of
predicting the intermediate musical symbols such as MIDI or retrieving the
music from the database, the proposed VidMuse, integrates both local and
global visual clues to generate background music consistent with the video in
an end-to-end manner. The core techniques in our method are a Long-Short-Term
Visual Module and a Music Token Decoder. Specifically, the long-short-term
visual module aims to learn the spatial-temporal representation of videos,
which is the key to generating music aligned with the video. On the one hand,
the long-term module models the entire video, capturing global context to
understand the whole video. It contributes to the coherence of generated music
at the video level. On the other hand, the short-term module focuses on
learning the fine-grained clues at the clip level, which plays an important
role in generating diverse music. The integration of two modules can improve
the quality and visual consistency of generated music. In addition, the
Transformer-based music token decoder is an autoregressive model, converting
video embeddings obtained by Long-Short-Term Visual Module into music tokens.
We formulate music generation as a task of next token prediction, which has
been widely validated by the NLP community. The predicted music tokens are
further decoded into the music signals by a high-fidelity neural audio
compression model [20].

In summary, the main contributions of our work are as follows:

(a) We construct a large-scale video-to-music dataset, _i.e._ , V2M, which
contains about 190k video-music pairs with high quality, covering various
genres and themes. To the best of our knowledge, this is the largest and most
diverse dataset for this task, which can facilitate future research.

(b) We propose a simple yet effective method, VidMuse, for video-to-music
generation. The proposed method integrates both local and global cues in the
video, enabling the generation of high-fidelity music tracks that are not only
musically coherent but also semantically aligned with the video content.

(c) We benchmark several state-of-the-art works against our method on
V2M-bench with a series of subjective and objective metrics for thorough
evaluation. As demonstrated in experiments, VidMuse achieve state-of-the-art
performance on V2M-bench, outperforming existing models in terms of audio
quality, diversity, and audio-visual consistency.

##  2 Related Work

In this section, we review the existing works related to video-to-music
generation, which mainly fall into four categories:

Video Representation. Various methods have been proposed to learn the spatio-
temporal representation [48, 50, 75, 21, 6, 73, 91, 49] for videos. They aim
to capture the contextual features of video frames, which is beneficial for
video understanding. Recent advances primarily concentrate on developing video
transformers, including ViViT [6, 73, 49]. These transformer-based methods
achieve superior generalized performance on various video understanding tasks,
such as video classification and temporal action localization. Among them,
Tong _et al._ [73] extend masked autoencoders [29] from the image to the
video, exhibiting the strong generalized performance in downstream tasks.
Benefiting from the advance in multi-modal large language models, lots of
works [91, 45, 59] of interactive video understanding have been proposed,
which built upon the large language models (LLMs) [74, 92, 62, 71] and
showcase the visual reasoning capabilities for video understanding.

Audio-Visual Alignment. Audio-visual alignment [5, 65, 68, 10, 26, 94, 87, 78,
82, 84] aims to align the feature between audio, vision in the semantics
level, which plays a vital role in tasks of audio-visual understanding and
generation. For example, CAV-MAE [26] is an audio-visual MAE by integrates the
contrastive learning and masked modeling method. Currently, many works go
beyond exploring audio-visual alignment. ImageBind [25] extends CLIP [61] to
more modalities, including audio, depth, thermal, and IMU data, which paves
the way for cross-modal retrieval and generation. In addition, Wu  _et al._
[82] employ LLMs with multi-modal adaptors to support any modal data as input
and output, showing strong capabilities in universal multi-modal
understanding. These methods transcend audio-visual alignment, and
dramatically advance the development of multi-modal representation learning.

Conditional Music Generation. Despite that there are lots of methods [18, 28,
34, 47, 57, 52, 54] studying unconditional music generation, in this paper, we
mainly focus on reviewing the methods of conditional music generation, which
are more related to our work. Many researchers [66, 4, 86, 22, 35, 11, 88, 14]
make their endeavours on text-to-music generation. Similar to Stable Diffusion
[63], these works [86, 22, 35, 66] try to adapt diffusion models for music
generation. M2UGen [36] is a multi-modal music understanding and generation
system that leverages large language models to process video, audio, and text.
Video2Music[37] can generate music that matches the content and emotion of a
given video. Moreover, the proposed V2Meow [70] not only can generate music
that supports style control via text prompts. In contrast to previous video-
to-music works [23, 37, 96, 70], our VidMuse utilizes a short-term module and
a long-term module to model local and global visual clues in videos. As a
result, it can generate high-fidelity music aligned with the video.

Video-to-music Datasets. Many multi-modal datasets [9, 58, 72, 42, 24, 93, 55,
69, 46, 3, 67, 31, 24] have been released, but there is still a lack of
datasets for video-to-music generation. Hong  _et al._ [33] construct the
HIMV-200K with video-music pairs and aim to retrieve music for the video from
the database. However, this dataset exhibits limited video genres and also
suffers from the issue of data quality, as stated in [96]. We observe that
several works [80, 16, 96] aim to facilitate the MIDI music generation.
However, we argue this musical form imposes restrictions on diversity for the
music generation. Other datasets [95, 44] focus on generating music for dance
videos only and have limited data size, which limits their applicability for
general video-to-music models. As a result, this work constructs a large-scale
video-to-music dataset where the music directly in wav format is diverse. We
establish a rigorous pipeline for data collection and cleaning, which ensures
the quality and diversity of our dataset. We expect the model can learn the
music with more diverse forms from this dataset.

##  3 Dataset

In this section, we build a multi-step pipeline to clean and process source
videos from YouTube to ensure data quality. After that, we construct a large-
scale video-to-music generation dataset, _i.e._ , V2M, with a benchmark. The
constructed dataset stands out for its significant size, high quality and rich
diversity, including a wide range of genres such as movie trailers,
advertisements, documentaries, vlogs, _etc_. This comprehensive and diverse
dataset aims to facilitate the video-to-music generation.

Figure 1: Dataset Construction. To ensure data quality, V2M goes through rule-
based coarse filtering and content-based fine-grained filtering. Music source
separation is applied to remove speech and singing signals in the audio. After
processing, human experts curate the benchmark subset, while the remaining
data is used as the pretraining dataset. The pretrain data is then refined
using Audio-Visual Alignment Ranking to select the finetuning dataset.

###  3.1 Dataset Collection

To quickly collect a large scale of video-music pairs, we curate a series of
query sets to retrieve corresponding videos from YouTube. In addition, we
found that the music in the movie trailer usually showcases rich diversity and
high quality. Therefore, we also aggregate a vast array of video information
from the IMDb Non-Commercial Datasets [1], including video types, names,
release dates, etc. Queries are formulated by appending “official trailer” to
the titles of these selections and retaining the videos released after 2000,
as videos from earlier periods are less likely to be of good quality. In the
process of video crawling, we only keep the top 2 search results, resulting in
a collection of around 400K videos, ranging from movies to documentaries.

###  3.2 Dataset Construction

The 400K raw videos may include many low-quality samples. To address this, we
develop a series of rigorous steps to filter the undesirable data and obtain a
clean set. We further process the source videos from the clean set, separating
the irrelevant vocal tracks while preserving the music tracks. After that, the
final kept videos with high quality can be directly used for training models.
The overall pipeline of data processing is shown in Fig. 1.

Coarse Filtering. We design a rule-based filtering strategy for initial data
screening. First, we perform illegal video and audio filters, which filter out
the video without an audio track or a video track. Next, we apply a duration
filter to filter out videos based on their duration, excluding those that are
either too long (over 480 seconds) or too short (under 30 seconds). Finally,
we we use a domain filter to examine metadata and exclude specific categories
such as Interview, News, Gaming, as the background music in these genres
typically lacks semantic alignment with the visual content.

Fine-grained Filtering. To further ensure the quality of our data, we
conducted additional audio and visual analyses. For the audio analysis, raw
videos may contain audio segments without music, such as speech, silence,
_etc_. To ensure the final dataset consists of high-quality video-music pairs,
we retain only those videos with a larger portion of music content. We utilize
the sound event detection model PANNs [39], which provides frame-level event
labels across the entire video to identify music events. Based on the
observation from a subset of videos, we define two thresholds, _i.e._ , a
confidence threshold and a duration threshold for analyzing the music event.
The confidence threshold is set at $0.5$, indicating an audio frame is
considered a music event if the PANNs model predicts the probability of the
“Music” label to be over $0.5$. The duration threshold of a music event
requires that at least $50\%$ of the audio’s frames are classified as music
events for the video to be considered valid.

For the visual analysis, some videos only consisting of static images will be
removed. Specifically, we uniformly sample multiple temporal windows without
overlap from the video. Within each window, we use Structural Similarity Index
Measure (SSIM) [79] between the first frame and the last frame. By aggregating
average SSIM values from all temporal windows, we remove the videos with
average SSIM values lower than a threshold of 0.8, empirically.

Music Source Separation. Since the irrelevant human speech in videos poses a
negative impact on music generation, we apply music source separation to
process the videos. We employ Demucs [13] as the music source separation model
to filter out the speech signals.

Audio-Video Alignment Ranking. ImageBind-AV [25] scores usually reflect the
semantic correlation between the vision and audio modality. To construct a
high-quality subset with better alignment, we compute the ImageBind-AV scores
for all the data and rank them accordingly.

After filtering and ranking, we split the final videos into the training set,
V2M, from all the paired data. The top 20K pairs are selected to form the
finetuning subset, V2M-20K. In addition, we randomly sample 1,000 videos
excluded from the training set. These 1,000 videos are then further evaluated
by five human experts based on audio quality and the degree of audio-visual
alignment. Ultimately, the top 300 high-quality videos are selected as a test
set, termed as V2M-bench.

###  3.3 Data Analysis

The above data pipeline yields three data splits. Specifically, the training
set comprises $\sim$190 video-music pairs, totaling 6,403 hours. The
finetuning dataset consists of $\sim$20K pairs, totaling 597 hours. The
benchmark dataset contains 300 pairs, with a cumulative duration of 9 hours.
Fig. 1(a) showcases the category and genre distribution of our training data,
highlighting its comprehensive diversity. This diversity ensures a rich and
varied dataset for the model training. As shown in Fig. 1(b), we compare with
other related datasets, demonstrating its advantage in data scale. Through
Fig. 1(c) and Fig. 1(d), we observe that the duration distribution across the
training set, finetuning set, and test set is highly uniform, with the
duration of most videos concentrated around 100 seconds.

Dataset Necessity. Some existing video-music pair datasets have been released
[33, 80, 16, 96, 95, 44], buts some of them [80, 16, 96] aim to facilitate
MIDI music generation, which limits the form of music. Datasets like [95, 44]
focus on generating music for dance videos only and have limited data size.
The dataset constructed by [33] includes video-music pairs but exhibits
limited video genres and suffers from data quality issues. Motivated by this,
we develop a multi-step pipeline and curate a large-scale dataset V2M for the
video-to-music generation.

###  3.4 Evaluation Metrics

To quantitatively evaluate the effectiveness of our model, we employ a series
of metrics to assess different models in terms of quality, fidelity, and
diversity for generated music. Besides, we also examine the alignment degree
between the video and generated music. We believe these metrics can thoroughly
evaluate the performance of different methods.

Frechet Audio Distance (FAD)111https://github.com/haoheliu/audioldm_eval [38]
is a reference-free evaluation metric for assessing audio quality. Similar to
Frechet Image Distance (FID)[32], it compares the embedding statistics of the
generated audio clip with groundtruth audio clips. A shorter distance usually
denotes better human-perceived acoustic-level audio quality. However, this
metric cannot reflect semantic-level information in audio. We report the FAD
based on the VGGish[31] feature extractor.

Frechet Distance (FD).33footnotemark: 3 measures the similarity between
generated samples and target samples in audio generation fields. It’s similar
to FAD but uses a PANNs feature extractor instead. PANNs[39] have been pre-
trained on AudioSet[24], one of the largest audio understanding datasets, thus
resulting in a more robust metric than FAD.

Kullback-Leibler Divergence (KL).33footnotemark: 3 reflects the acoustic
similarity between the generated and reference samples to a certain extent. It
is computed over PANNs’ multi-label class predictions.

Density and Coverage222https://github.com/clovaai/generative-evaluation-prdc
[60] measures the fidelity and diversity aspects of the generated samples.
Fidelity measures how closely the generated samples match the real ones, while
diversity assesses whether the generated samples capture the full range of
variation found in real samples. We use CLAP[83] embeddings for manifold
estimation.

Imagebind Score333https://github.com/facebookresearch/ImageBind [25] assesses
to what extent the generated music aligns with the videos. Despite the fact
that Imagebind extends the CLIP to six modalities, we only use the branches of
audio and vision. Since we use ImageBind to filter out video-audio pairs with
a low matching score during dataset construction, the ImageBind score is
naturally used in our evaluation. We acknowledge that imageBind is not
specifically trained on music data, but there is no better choice currently.

##  4 Method

Figure 2: Overview of the VidMuse Framework.

In this section, we elaborate on the details of our proposed VidMuse. Our
approach leverages Long-Short-Term modeling to generate music that is aligned
with video content. Specifically, the Short-Term Module captures localized
visual features to ensure the generated music aligns with local variations in
the video, while the Long-Term Module models the entire video, providing
global context to guide the Short-Term Module in generating more visually
consistent music. The experiments in Sec. 5 demonstrate the effectiveness of
our proposed method.

###  4.1 Overview

We illustrate the VidMuse framework’s pipeline in Fig. 2, which converts
visual content to music using a structured four-module architecture. The
components include (1) a Visual Encoder, (2) a Long-Short-Term Visual Module,
(3) a Music Token Decoder, and (4) an Audio Codec Decoder.

Given an input video X, the Visual Encoder extracts feature representations
$\bar{\textbf{X}}$. These features are then fed into the Long-Short-Term
Visual Module. Specifically, the Short-Term Module captures localized details,
while the Long-Term Module models the entire video, providing global context,
detailed in Sec. 4.3. This results in the feature representation Z through
multi-head cross-attention. The feature Z is then processed by the Music Token
Decoder to generate music tokens $\bar{\textbf{Y}}$. Finally, these music
tokens are decoded by the Audio Codec to produce the final music
$\bar{\textbf{A}}$.

###  4.2 Visual Encoder

The Visual Encoder processes input video frames, denoted by
$\textbf{X}\in\mathbb{R}^{N\times C\times H\times W}$, where $N$ represents
the frame count and each frame has dimensions $C\times H\times W$. This
component extracts feature representations
$\bar{\textbf{X}}\in\mathbb{R}^{N\times P\times D}$ from the video frames.
Here, $P$ and $D$ denote the sequence length, including the class token, and
the dimensionality of feature vectors, respectively [19, 61, 73, 6].

###  4.3 Long-Short-Term Visual Module

To capture both local and global visual clues, we model the visual features
through the Long-Short-Term Visual Module. Specifically, we evenly samples
$N_{l}$ frames from $\bar{\textbf{X}}$ to obtain global feature
representations $\bar{\textbf{X}}_{l}\in\mathbb{R}^{N_{l}\times P\times D}$,
and extract a continuous segment of $N_{s}$ frames from $\bar{\textbf{X}}$ to
form local feature representations
$\bar{\textbf{X}}_{s}\in\mathbb{R}^{N_{s}\times P\times D}$. The
representations are then refined through Long-term Modeling and Short-term
Modeling, respectively, resulting in long-term features
$\bar{\textbf{F}}_{l}\in\mathbb{R}^{N_{l}\cdot\text{$N_{H}$}\times D}$ and
short-term features
$\bar{\textbf{F}}_{s}\in\mathbb{R}^{N_{s}\cdot\text{$N_{H}$}\times D}$, where
$N_{H}$ represents the number of attention heads.

LST-Fusion integrates long-term and short-term features by utilizing the
cross-attention mechanism in the form of Query ($\mathbf{q}$), Key
($\mathbf{k}$), and Value ($\mathbf{v}$), which is formulated as:

$\mathbf{q}={\bar{\textbf{F}}_{s}}\mathbf{W}_{q},\ \ \
\mathbf{k}=\bar{\textbf{F}}_{l}\mathbf{W}_{k},\ \ \
\mathbf{v}=\bar{\textbf{F}}_{l}\mathbf{W}_{v},$ (1)

$\textbf{Z}^{\prime}=\text{softmax}(\mathbf{q}\mathbf{k}^{T}/\sqrt{C/h})\mathbf{v}+\bar{\textbf{F}}_{s},$
(2)

where $\mathbf{W}_{q}$, $\mathbf{W}_{k}$, $\mathbf{W}_{v}$
$\in\mathbb{R}^{C\times(C/h)}$ denote trainable parameters, with $C$
representing the embedding dimension and $h$ the number of heads. This
mechanism allows the model to query global information rather than generating
music based solely on local visual features. It guarantees that the generated
music is more consistent with the overall video content.

After the cross-attention, a linear projection layer reconfigures
$\textbf{Z}^{\prime}$ to align with the Music Token Decoder’s requirements:

$\textbf{Z}=Linear(\textbf{Z}^{\prime}).$ (3)

The $Linear$ that is a fully connected layer projects $\textbf{Z}^{\prime}$
into a space characterized by $\textbf{Z}\in\mathbb{R}^{N_{s}\cdot N_{H}\times
M}$, where $M$ corresponds to the intermediate layer dimension within the
Music Token Decoder’s transformer architecture.

The Long-Short-Term Visual Module integrates short-term and long-term features
of the video and yields a representation Z, ready for the generation of music
tokens. The experiment results in Sec. 5 show the efficacy of this module.

###  4.4 Music Token Decoder

Music and language are both sequential and structured. We thus can try to
adapt LLMs to music [76, 34, 15, 4, 11]. To apply LLM techniques to the audio
field, several studies have utilized different audio codecs to represent audio
information as discrete tokens [90, 40]. These codecs allow audio segments to
be encoded into a matrix with dimensions $[K,T_{s}]$, where $K$ is the number
of codebooks and $T_{s}$ is the number of sequence timesteps. Some researchers
have also investigated various ways to arrange these codebooks [90, 77, 11,
27, 85, 41].

From the previously described steps, we obtain the feature set
$\textbf{Z}\in\mathbb{R}^{N_{s}\cdot N_{H}\times M}$, which is then input into
the Music Token Decoder’s transformer $\mathcal{G}$ with a linear
classification head. This transformation stage produces an output tensor
$\bar{\textbf{Y}}\in\mathbb{R}^{K\times S}$ as music tokens, where $K$ is the
number of codebooks and $S$ denotes the sequence steps in the musical piece:

$\bar{\textbf{Y}}=Linear(\mathcal{G}(\textbf{Z})),$ (4)

where $\mathcal{G}$ aims to convert video features into a sequence of music
features.

###  4.5 Optimization

Given the codec $\mathcal{C}$, we denote the Audio Encoder as
$\mathcal{C}_{\text{encode}}(\cdot)$ and Audio Decoder as
$\mathcal{C}_{\text{decode}}(\cdot)$, respectively. The process of encoding
the ground truth audio A into its tokenized representation Y can be
represented as

$\textbf{Y}=\mathcal{C}_{\text{encode}}(\textbf{A}).$ (5)

The audio decoder $\mathcal{C}_{\text{decode}}(\cdot)$ is utilized solely
during the inference stage to decode music tokens into music. In the training
stage, an input video X passes through a sequence of operations to generate a
predicted representation $\bar{\textbf{Y}}$. With the target representation,
Y, is derived by encoding the ground-truth audio A by eq. 5. The objective of
our model is to minimize the divergence between $\bar{\textbf{Y}}$ and Y,
formalized as the loss function:

$\mathcal{L}=\mathcal{L}(\textbf{Y},\bar{\textbf{Y}}),$ (6)

where $\mathcal{L}$ is cross-entropy loss in practice.

###  4.6 Inference

When predicting music on videos of arbitrary length, maintaining music
consistency and coherence is particularly important. However, it leads to a
significant challenge on computational resources due to the quadratic
dependency of transformers-based models on sequence length [89, 7]. To address
this problem, similar to methods used in previous works [81, 7], we adopt a
sliding window approach for inferring the whole video.

During inference, given an input video X with the length of $L$, we set the
sliding window length as $L_{s}$ and the length of the window’s overlap as
$O$. The value of $t$ is initially set to $0$. While $t+L_{s}\leq L$, the
inference steps can be summarized as follows: 1) We use the visual encoder to
extract feature embeddings $\bar{\textbf{X}}_{l}$ for a given video X. 2)
Based on $\bar{\textbf{X}}_{l}$, we collect the feature embeddings within the
time window of $[t:t+L_{s}]$ to obtain $\bar{\textbf{X}}_{s}$. 3) Conditioned
on $\bar{\textbf{X}}_{l}$ and $\bar{\textbf{X}}_{s}$, the music token decoder
predicts the music segment $\bar{\textbf{Y}}$ at the time window of
$[t:t+L_{s}-O]$ after the long-short-term visual module. 4) The music tokens
$\bar{\textbf{A}}$ are decoded from $\bar{\textbf{Y}}$ by a audio decoder:
$\bar{\textbf{A}}=\mathcal{C}_{\text{decode}}(\bar{\textbf{Y}})$. 5) The slide
the window moves forward, _i.e._ , $t=t+L_{s}-O$, and we repeat steps 2 to 4
until the end of the video.

After finishing above steps, we can concatenate all musical segments to form a
cohesive piece of music that aligns in duration with the video.

##  5 Experiments

In this section, we first elaborate on the implementation details of our
experiment and then present the main results for both our method and the
baseline methods using objective metrics and a subjective user study. Then, we
demonstrate that our VidMuse exhibits impressive performance on all metrics.
Furthermore, we conduct a comprehensive ablation study to explain our design
choices and their impact on the performance.

###  5.1 Implementation details

We choose CLIP [61] as the visual encoder (refer to experiments in Appendix
A). Since this work does not focus on audio encoding and decoding, we use
Encodec [13] for 32 kHz monophonic audio as our default compression model and
use the pretrained transformer model proposed in MusicGen [11]. The training
stage utilizes the AdamW optimizer [51] for 22,000 steps with a batch size of
240 samples. The hyperparameters are set to $\beta_{1}=0.9$, $\beta_{2}=0.95$,
with a weight decay of 0.1 and gradient clipping at 1.0. A cosine learning
rate schedule is employed, incorporating a warm-up phase of 4,000 steps, and
an exponential moving average decay of 0.99. The training is conducted on 48
H800 GPUs, with float16 mixed precision. This setup leverages $190$K samples
for pre-training and $20$K samples for fine-tuning. A top-k strategy is
applied for sampling, retaining the top $250$ tokens with a temperature
setting of $1.0$. In the inference stage, we set the sliding window size as
$30$s, and the length of the window’s overlap as $5$s.

###  5.2 Main Results

Table 1:  Comparison with naive baselines and state-of-the-art methods. Methods |  Metrics
---|---
| KL$~{}\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}{\downarrow}$ | FD$~{}\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}{\downarrow}$ | FAD$~{}\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}{\downarrow}$ | density$~{}\color[rgb]{0.1953125,0.78515625,0.1953125}\definecolor[named]{pgfstrokecolor}{rgb}{0.1953125,0.78515625,0.1953125}{\uparrow}$ | coverage$~{}\color[rgb]{0.1953125,0.78515625,0.1953125}\definecolor[named]{pgfstrokecolor}{rgb}{0.1953125,0.78515625,0.1953125}{\uparrow}$ | Imagebind$~{}\color[rgb]{0.1953125,0.78515625,0.1953125}\definecolor[named]{pgfstrokecolor}{rgb}{0.1953125,0.78515625,0.1953125}{\uparrow}$
GT | $0.000$ | $0.000$ | $0.000$ | $1.167$ | $1.000$ | $0.241$
Caption2Music | $1.081$ | $40.199$ | $2.485$ | $0.378$ | $0.486$ | $0.191$
Video2Music [37] | $1.782$ | $144.881$ | $18.722$ | $0.103$ | $0.023$ | $0.136$
CMT [16] | $1.220$ | $85.704$ | $8.637$ | $0.080$ | $0.070$ | $0.124$
M2UGen [36] | $0.997$ | $52.246$ | $5.104$ | $0.608$ | $0.433$ | $0.181$
VM-NET∗ [33] | $0.899$ | $67.480$ | $6.252$ | $0.986$ | $0.383$ | $0.147$
VidMuse | 0.738 | 36.171 | 2.369 | 1.175 | 0.710 | 0.207
|  |  |  |  |  |

We benchmark several state-of-the-art methods, serving as baselines to compare
with our method: 1) Caption2Music serving as a naive baseline, which employs
the method [2] to extract the video captions and then outputs the music by
feeding captions into MusicGen [11]. 2) Video2Music [37] and 3) CMT [16] which
botch focus on predicting MIDI notes [64] from videos while our method
directly generates music signals. 4) M2UGen [36], a strong baseline, which
leverages a language model to connect vision and language, then use MusicGen
[11] to generate music from language. 5) VM-NET [33], essentially different
from the above methods, retrieves the piece of music from the database for a
given video, while other methods predict music by learning from video-music
pairs.

In Table 1, VidMuse integrating both global and local visual clues exhibits
impressive performance on all metrics. Specifically, compared with Video2Music
[37] and CMT [16], VidMuse shows the superiority in the diversity of generated
music based on the metrics of density or coverage, which justifies the
advantage of directly predicting music signals compared with MIDI notes. In
addition, our method even outperforms the strong competitors, _i.e._ , M2UGen.
It proves that predicting music directly based on video input can also achieve
better performance. Furthermore, compared with a retrieval-based method _i.e._
, VM-NET [33], VidMuse achieves a higher Imagebind score, indicating that the
music generated by the learning-based method is more consistent with the video
semantics.

###  5.3 User Study

In our user study, we randomly sample 600 video-music pairs from the benchmark
to conduct an A/B test, which is a widely used subjective evaluation method in
the music field [17, 88], across five music generation methods: CMT, M2UGen,
Caption2Music, Ground Truth (GT), and VidMuse. The test was distributed among
40 participants, ensuring each method was compared against another 60 times.
The evaluation criteria are four-fold: 1) Audio quality: Refers to the sound
quality of the audio track; 2) Video-music alignment: Assesses how well the
music matches the visual content, e.g., a scene showing a woman crying should
ideally be paired with music that sounds sad; 3) Musicality: Measures the
aesthetic appeal of music, independent of audio quality, e.g., a piece of
music has high audio quality but is out of tune has poor musicality; 4)
Overall assessment: A comprehensive evaluation that considers all aspects of
the music.

(a) Audio quality.

(b) Video-music alignment.

(a) Musicality.

(b) Overall assessment.

Figure 4: Extended A/B test user study results. Please refer to Sec. 5.3 where
we introduce four criteria, _i.e._ , audio quality, video-music alignment,
musicality and overall assessment.

Participants are asked to choose the better sample for each criterion. The
Fig. 4 illustrates the statistical results of the user study, where the value
at matrix[$i$][$j$] ranges from $0$ to $100$, indicating the % of times
listeners preferred the method correspond to the $i$-row compared to the
method correspond to the $j$-column. For example, in Fig. 4 (c), the value of
matrix[$2$][$4$] represents that VidMuse outperforms CMT in $77\%$ of cases in
terms of Musicality. Across all evaluation criteria, our method surpasses
other methods in more than half of the comparisons, except when compared to
the ground truth. Overall, these results robustly validate VidMuse’s
effectiveness from the subjective evaluation.

###  5.4 Ablation Studies

In this section, we conduct comprehensive ablation studies to determine the
optimal hyperparameters and design choices. First, we examine the impact of
different model design choices on the generation quality. Second, we
investigate the influence of the different video sampling rates and the
duration of video segments in the Short-Term module on the results. After
that, we explore the impact of the model size on performance. More ablation
results are in the Appendix A.

Justification of Design Choices. To validate the impact of different model
design choices on our generation results and verify the effectiveness of our
method, we first design two modules: a short-term modeling module (STM) and a
long-term modeling module (LTM). VidMuse-STM aims to ablate the contribution
of STM by using only STM without LTM while VidMuse-LTM utilizes only LTM. We
present the results in Tab. 2. Based on these results, we learn an insight
that local information plays a more importance role in the generation. Second,
we implement two variants with Cross-Attention with learnable Queries (CAQ) in
our framework. Specifically, CAQ_SL first uses a CAQ where $K$ and $V$ are
short-term features and then uses a CAQ where $K$ and $V$ are long-term
features. CAQ_LS do it in an opposite order. As shown in Tab. 2, VidMuse
outperforms two variants. It seems that the learnable query requires a deeper
architecture, as demonstrated by DETR [8] and Q-Former [43], but this manner
reduces the model’s efficiency. Furthermore, we also try a baseline using
Slowfast [21] and find its performance degenerates in Tab. 2. This may be
because the fast path uses a higher frame rate in inputs than the slow path,
but their temporal receptive fields are identical. Thus, Slowfast lacks
guidance from global clues.

Exploration on model inputs. To explore the impact of different video sampling
rates and the duration of video segments in the Short-Term module on
performance, we conducted ablation studies on input FPS and short-term segment
duration, detailed in Table 3. To intuitively assess the effectiveness of
different settings, we employ an Average Rank (AR) metric. The AR metric ranks
the results for a metric across all methods within the same table. The ranking
result is from $1$ to $N$ (equals to the number of methods within the table),
where $1$ is the best and $N$ is the worst. We eventually obtain AR results by
averaging the ranking results for all metrics. Note that the AR results cannot
be compared across different tables, since this metric is designed to clearly
showcase the dominance for each method within one table. From Table 3, we
observe that increasing both FPS and duration tends to enhance model
capabilities, suggesting that denser frame sampling yields a more detailed
video representation, thereby improving music generation. Nevertheless, to
balance computational costs and performance, we use a 30-second duration at 2
FPS as our optimal setting.

The impact of model size on performance. For the architecture of the music
token decoder, we test three variants with different scales: small ($455$M),
medium ($1.9$B), and large ($3.3$B). The medium-sized VidMuse-M is recognized
as the default setting if not stated, as it achieves the best results across
all metrics, outlined in Table 4. We observe the larger VidMuse-L model fails
to deliver proportional improvements. This discrepancy can be partly
attributed to limited GPU resources. The model’s performance is near
saturation, and the larger model does not exhibit significant advantages. As a
result, VidMuse-M is recognized as the optimal choice, achieving a trade-off
between performance and efficiency.

Table 2:  Ablation studies on design choices. Methods |  Metrics
---|---
| KL$~{}\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}{\downarrow}$ | FD$~{}\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}{\downarrow}$ | FAD$~{}\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}{\downarrow}$ | density$~{}\color[rgb]{0.1953125,0.78515625,0.1953125}\definecolor[named]{pgfstrokecolor}{rgb}{0.1953125,0.78515625,0.1953125}{\uparrow}$ | coverage$~{}\color[rgb]{0.1953125,0.78515625,0.1953125}\definecolor[named]{pgfstrokecolor}{rgb}{0.1953125,0.78515625,0.1953125}{\uparrow}$ | Imagebind$~{}\color[rgb]{0.1953125,0.78515625,0.1953125}\definecolor[named]{pgfstrokecolor}{rgb}{0.1953125,0.78515625,0.1953125}{\uparrow}$
VidMuse-STM | $0.898$ | $45.752$ | $4.915$ | $1.124$ | $0.470$ | $0.196$
VidMuse-LTM | $0.858$ | $53.907$ | $16.074$ | $1.439$ | $0.406$ | $0.205$
VidMuse-CAQ_SL | $0.843$ | $48.940$ | $3.733$ | $0.947$ | $0.547$ | $0.163$
VidMuse-CAQ_LS | $0.919$ | $45.335$ | $2.917$ | $0.562$ | $0.720$ | $0.181$
VidMuse-Slowfast | $1.511$ | $84.683$ | $10.029$ | $0.266$ | $0.285$ | $0.118$
VidMuse | 0.738 | 36.171 | 2.369 | 1.175 | 0.710 | 0.207
|  |  |  |  |  |
Table 3:  Ablation studies on Video Duration and FPS. Duration(s) | FPS |  Metrics
---|---|---
|  | KL$~{}\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}{\downarrow}$ | FD$~{}\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}{\downarrow}$ | FAD$~{}\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}{\downarrow}$ | density$~{}\color[rgb]{0.1953125,0.78515625,0.1953125}\definecolor[named]{pgfstrokecolor}{rgb}{0.1953125,0.78515625,0.1953125}{\uparrow}$ | coverage$~{}\color[rgb]{0.1953125,0.78515625,0.1953125}\definecolor[named]{pgfstrokecolor}{rgb}{0.1953125,0.78515625,0.1953125}{\uparrow}$ | Imagebind$~{}\color[rgb]{0.1953125,0.78515625,0.1953125}\definecolor[named]{pgfstrokecolor}{rgb}{0.1953125,0.78515625,0.1953125}{\uparrow}$ | AR$~{}\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}{\downarrow}$
5 | 2 | $0.820$ | $51.101$ | $4.117$ | $1.430$ | $0.74$ | $0.148$ | $7.00$
15 | 2 | $0.849$ | $41.131$ | $2.709$ | $1.406$ | $0.803$ | $0.181$ | $5.33$
30 | 2 | $0.843$ | $41.354$ | $\underline{2.413}$ | $1.487$ | $\underline{0.840}$ | 0.193 | $3.67$
5 | 4 | 0.800 | $51.540$ | $4.343$ | $1.271$ | $0.787$ | $0.145$ | $7.17$
15 | 4 | $0.830$ | $41.154$ | $2.562$ | $1.278$ | $0.823$ | $0.176$ | $5.17$
30 | 4 | $0.849$ | $\underline{40.032}$ | $2.418$ | $\underline{1.538}$ | 0.843 | 0.193 | $\underline{2.84}$
5 | 8 | $\underline{0.819}$ | $50.667$ | $4.069$ | $1.515$ | $0.743$ | $0.153$ | $5.67$
15 | 8 | $0.857$ | $42.106$ | $2.790$ | $1.476$ | $0.753$ | $\underline{0.187}$ | $6.00$
30 | 8 | $0.824$ | 38.942 | 2.299 | 1.573 | 0.843 | $0.180$ | 2.17
|  |  |  |  |  |  |  |
Figure 5: Qualitative Comparison results on sound spectrograms produced by different methods. Table 4:  Ablation studies on model size. Models | Params |  Metrics
---|---|---
|  | KL$~{}\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}{\downarrow}$ | FD$~{}\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}{\downarrow}$ | FAD$~{}\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}{\downarrow}$ | density$~{}\color[rgb]{0.1953125,0.78515625,0.1953125}\definecolor[named]{pgfstrokecolor}{rgb}{0.1953125,0.78515625,0.1953125}{\uparrow}$ | coverage$~{}\color[rgb]{0.1953125,0.78515625,0.1953125}\definecolor[named]{pgfstrokecolor}{rgb}{0.1953125,0.78515625,0.1953125}{\uparrow}$ | Imagebind$~{}\color[rgb]{0.1953125,0.78515625,0.1953125}\definecolor[named]{pgfstrokecolor}{rgb}{0.1953125,0.78515625,0.1953125}{\uparrow}$
VidMuse-S |  $455$M | $0.854$ | $57.317$ | $4.695$ | $1.349$ | $0.433$ | $0.165$
VidMuse-M |  $1.9$B | 0.843 | 41.354 | 2.413 | 1.487 | 0.840 | 0.193
VidMuse-L |  $3.3$B | $0.873$ | $53.955$ | $4.041$ | $1.330$ | $0.453$ | $0.170$
|  |  |  |  |  |  |

###  5.5 Qualitative Analysis

Our qualitative analysis, presented in Fig. 5, underscores specific
limitations in the methods employed by CMT and M2UGen. For CMT, the method’s
reliance on extracting only low-level visual cues for music generation,
particularly in creating symbolic music, leads to pronounced discontinuities.
As illustrated in Fig. 5, “Abrupt gaps” occur, especially in darker video
segments where the model fails to predict symbolic music notes, resulting in
silence.

M2UGen’s use of LLMs and text embeddings to translate visual content into
music affects its precision in capturing detailed visual nuances, leading to a
noticeable lack of diversity, evidenced by repetitive musical themes.
Additionally, M2UGen’s limitation to 30-second music generation segments
requires the division of longer videos into clips for separate processing.
This approach results in abrupt transitions between segments after stitching,
which compromises the musical continuity.

The last row of Fig. 5 showcases our Long-Short-Term (LST) approach,
highlighting its ability to produce globally diverse music that captures the
essence of the ground truth. This method ensures our music generation remains
contextually rich and aligned with the video.

##  6 Conclusion

In this work, we build a rigorous pipeline to collect high-quality and diverse
video-music pairs, curating a comprehensive dataset V2M. Then, we propose
VidMuse, a simple yet effective method for video-to-music generation. Our
approach utilizes a Long-Short-Term approach to capture both local and global
visual clues in the video, allowing for the generation of contextually rich
and musically diverse outputs. To validate our method, we benchmark a series
of state-of-the-art methods as baselines to compare with VidMuse. Through
comprehensive quantitative studies and qualitative analyses, our method has
demonstrated its superiority over the existing methods.

Limitations. Our work achieves a significant advancement in video-to-music
generation, but it still has some limitations. First, the current
implementation relies on a EnCodec model [20] trained on a dataset that
restricts the system’s capacity to a 32kHz audio sampling rate. This codec
exhibits a noticeable reconstruction loss in the audio signal, potentially
lowering the quality of the generated music. Second, training large models
requires substantial computational resources. Our future work aims to overcome
these limitations by integrating advanced codec technologies to enhance audio
reconstruction fidelity, and exploring a more efficient method design.

## References

  * [1] Imdb non-commercial license dataset.  https://developer.imdb.com/non-commercial-datasets/#imdb-non-commercial-datasets.
  * [2] Spacetimegpt: A spatiotemporal video captioning model.  https://huggingface.co/Neleac/timesformer-gpt2-video-captioning, 2022.
  * [3] Sami Abu-El-Haija, Nisarg Kothari, Joonseok Lee, Paul Natsev, George Toderici, Balakrishnan Varadarajan, and Sudheendra Vijayanarasimhan.  Youtube-8m: A large-scale video classification benchmark.  arXiv preprint arXiv:1609.08675, 2016.
  * [4] Andrea Agostinelli, Timo I Denk, Zalán Borsos, Jesse Engel, Mauro Verzetti, Antoine Caillon, Qingqing Huang, Aren Jansen, Adam Roberts, Marco Tagliasacchi, et al.  Musiclm: Generating music from text.  arXiv preprint arXiv:2301.11325, 2023.
  * [5] Hassan Akbari, Liangzhe Yuan, Rui Qian, Wei-Hong Chuang, Shih-Fu Chang, Yin Cui, and Boqing Gong.  Vatt: Transformers for multimodal self-supervised learning from raw video, audio and text.  Advances in Neural Information Processing Systems, 34:24206–24221, 2021.
  * [6] Anurag Arnab, Mostafa Dehghani, Georg Heigold, Chen Sun, Mario Lučić, and Cordelia Schmid.  Vivit: A video vision transformer.  In Proceedings of the IEEE/CVF international conference on computer vision, pages 6836–6846, 2021.
  * [7] Iz Beltagy, Matthew E Peters, and Arman Cohan.  Longformer: The long-document transformer.  arXiv preprint arXiv:2004.05150, 2020.
  * [8] Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko.  End-to-end object detection with transformers.  In European conference on computer vision, pages 213–229. Springer, 2020.
  * [9] Soravit Changpinyo, Piyush Sharma, Nan Ding, and Radu Soricut.  Conceptual 12m: Pushing web-scale image-text pre-training to recognize long-tail visual concepts.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3558–3568, 2021.
  * [10] Haoyue Cheng, Zhaoyang Liu, Hang Zhou, Chen Qian, Wayne Wu, and Limin Wang.  Joint-modal label denoising for weakly-supervised audio-visual video parsing.  In European Conference on Computer Vision, pages 431–448. Springer, 2022.
  * [11] Jade Copet, Felix Kreuk, Itai Gat, Tal Remez, David Kant, Gabriel Synnaeve, Yossi Adi, and Alexandre Défossez.  Simple and controllable music generation.  Advances in Neural Information Processing Systems, 36, 2024.
  * [12] Johanna N Dasovich-Wilson, Marc Thompson, and Suvi Saarikallio.  Exploring music video experiences and their influence on music perception.  Music & Science, 5:20592043221117651, 2022.
  * [13] Alexandre Défossez, Nicolas Usunier, Léon Bottou, and Francis Bach.  Music source separation in the waveform domain.  arXiv preprint arXiv:1911.13254, 2019.
  * [14] Qixin Deng, Qikai Yang, Ruibin Yuan, Yipeng Huang, Yi Wang, Xubo Liu, Zeyue Tian, Jiahao Pan, Ge Zhang, Hanfeng Lin, et al.  Composerx: Multi-agent symbolic music composition with llms.  arXiv preprint arXiv:2404.18081, 2024.
  * [15] Prafulla Dhariwal, Heewoo Jun, Christine Payne, Jong Wook Kim, Alec Radford, and Ilya Sutskever.  Jukebox: A generative model for music.  arXiv preprint arXiv:2005.00341, 2020.
  * [16] Shangzhe Di, Zeren Jiang, Si Liu, Zhaokai Wang, Leyan Zhu, Zexin He, Hongming Liu, and Shuicheng Yan.  Video background music generation with controllable music transformer.  In Proceedings of the 29th ACM International Conference on Multimedia, pages 2037–2045, 2021.
  * [17] Chris Donahue, Antoine Caillon, Adam Roberts, Ethan Manilow, Philippe Esling, Andrea Agostinelli, Mauro Verzetti, Ian Simon, Olivier Pietquin, Neil Zeghidour, et al.  Singsong: Generating musical accompaniments from singing.  arXiv preprint arXiv:2301.12662, 2023.
  * [18] Hao-Wen Dong, Wen-Yi Hsiao, Li-Chia Yang, and Yi-Hsuan Yang.  Musegan: Multi-track sequential generative adversarial networks for symbolic music generation and accompaniment.  In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32, 2018.
  * [19] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al.  An image is worth 16x16 words: Transformers for image recognition at scale.  arXiv preprint arXiv:2010.11929, 2020.
  * [20] Alexandre Défossez, Jade Copet, Gabriel Synnaeve, and Yossi Adi.  High fidelity neural audio compression.  arXiv preprint arXiv:2210.13438, 2022.
  * [21] Christoph Feichtenhofer, Haoqi Fan, Jitendra Malik, and Kaiming He.  Slowfast networks for video recognition.  In Proceedings of the IEEE/CVF international conference on computer vision, pages 6202–6211, 2019.
  * [22] Seth Forsgren and Hayk Martiros.  Riffusion-stable diffusion for real-time music generation, 2022.  URL https://riffusion. com/about, 6.
  * [23] Chuang Gan, Deng Huang, Peihao Chen, Joshua B Tenenbaum, and Antonio Torralba.  Foley music: Learning to generate music from videos.  In Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XI 16, pages 758–775. Springer, 2020.
  * [24] Jort F Gemmeke, Daniel PW Ellis, Dylan Freedman, Aren Jansen, Wade Lawrence, R Channing Moore, Manoj Plakal, and Marvin Ritter.  Audio set: An ontology and human-labeled dataset for audio events.  In 2017 IEEE international conference on acoustics, speech and signal processing (ICASSP), pages 776–780. IEEE, 2017.
  * [25] Rohit Girdhar, Alaaeldin El-Nouby, Zhuang Liu, Mannat Singh, Kalyan Vasudev Alwala, Armand Joulin, and Ishan Misra.  Imagebind: One embedding space to bind them all.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15180–15190, 2023.
  * [26] Yuan Gong, Andrew Rouditchenko, Alexander H Liu, David Harwath, Leonid Karlinsky, Hilde Kuehne, and James Glass.  Contrastive audio-visual masked autoencoder.  arXiv preprint arXiv:2210.07839, 2022.
  * [27] Hongkun Hao, Long Zhou, Shujie Liu, Jinyu Li, Shujie Hu, Rui Wang, and Furu Wei.  Boosting large language model for speech synthesis: An empirical study.  arXiv preprint arXiv:2401.00246, 2023.
  * [28] Curtis Hawthorne, Andriy Stasyuk, Adam Roberts, Ian Simon, Cheng-Zhi Anna Huang, Sander Dieleman, Erich Elsen, Jesse Engel, and Douglas Eck.  Enabling factorized piano music modeling and generation with the maestro dataset.  arXiv preprint arXiv:1810.12247, 2018.
  * [29] Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick.  Masked autoencoders are scalable vision learners.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 16000–16009, 2022.
  * [30] Yingqing He, Zhaoyang Liu, Jingye Chen, Zeyue Tian, Hongyu Liu, Xiaowei Chi, Runtao Liu, Ruibin Yuan, Yazhou Xing, Wenhai Wang, Jifeng Dai, Yong Zhang, Wei Xue, Qifeng Liu, Yike Guo, and Qifeng Chen.  Llms meet multimodal generation and editing: A survey.  arXiv preprint arXiv:2405.19334, 2024.
  * [31] Shawn Hershey, Sourish Chaudhuri, Daniel PW Ellis, Jort F Gemmeke, Aren Jansen, R Channing Moore, Manoj Plakal, Devin Platt, Rif A Saurous, Bryan Seybold, et al.  Cnn architectures for large-scale audio classification.  In 2017 ieee international conference on acoustics, speech and signal processing (icassp), pages 131–135. IEEE, 2017.
  * [32] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter.  Gans trained by a two time-scale update rule converge to a local nash equilibrium.  Advances in neural information processing systems, 30, 2017.
  * [33] Sungeun Hong, Woobin Im, and Hyun S Yang.  Content-based video-music retrieval using soft intra-modal structure constraint.  arXiv preprint arXiv:1704.06761, 2017.
  * [34] Cheng-Zhi Anna Huang, Ashish Vaswani, Jakob Uszkoreit, Noam Shazeer, Ian Simon, Curtis Hawthorne, Andrew M Dai, Matthew D Hoffman, Monica Dinculescu, and Douglas Eck.  Music transformer.  arXiv preprint arXiv:1809.04281, 2018.
  * [35] Qingqing Huang, Daniel S Park, Tao Wang, Timo I Denk, Andy Ly, Nanxin Chen, Zhengdong Zhang, Zhishuai Zhang, Jiahui Yu, Christian Frank, et al.  Noise2music: Text-conditioned music generation with diffusion models.  arXiv preprint arXiv:2302.03917, 2023.
  * [36] Atin Sakkeer Hussain, Shansong Liu, Chenshuo Sun, and Ying Shan.  M2UGen: Multi-modal Music Understanding and Generation with the Power of Large Language Models.  arXiv preprint arXiv:2311.11255, 2023.
  * [37] Jaeyong Kang, Soujanya Poria, and Dorien Herremans.  Video2music: Suitable music generation from videos using an affective multimodal transformer model.  arXiv preprint arXiv:2311.00968, 2023.
  * [38] Kevin Kilgour, Mauricio Zuluaga, Dominik Roblek, and Matthew Sharifi.  Fr$\backslash$’echet audio distance: A metric for evaluating music enhancement algorithms.  arXiv preprint arXiv:1812.08466, 2018.
  * [39] Qiuqiang Kong, Yin Cao, Turab Iqbal, Yuxuan Wang, Wenwu Wang, and Mark D Plumbley.  Panns: Large-scale pretrained audio neural networks for audio pattern recognition.  IEEE/ACM Transactions on Audio, Speech, and Language Processing, 28:2880–2894, 2020.
  * [40] Rithesh Kumar, Prem Seetharaman, Alejandro Luebs, Ishaan Kumar, and Kundan Kumar.  High-fidelity audio compression with improved rvqgan.  Advances in Neural Information Processing Systems, 36, 2024.
  * [41] Gael Le Lan, Varun Nagaraja, Ernie Chang, David Kant, Zhaoheng Ni, Yangyang Shi, Forrest Iandola, and Vikas Chandra.  Stack-and-delay: a new codebook pattern for music generation.  arXiv preprint arXiv:2309.08804, 2023.
  * [42] Sangho Lee, Jiwan Chung, Youngjae Yu, Gunhee Kim, Thomas Breuel, Gal Chechik, and Yale Song.  Acav100m: Automatic curation of large-scale datasets for audio-visual video representation learning.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 10274–10284, 2021.
  * [43] Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi.  Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models.  In International conference on machine learning, pages 19730–19742. PMLR, 2023.
  * [44] Ruilong Li, Shan Yang, David A Ross, and Angjoo Kanazawa.  Ai choreographer: Music conditioned 3d dance generation with aist++.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 13401–13412, 2021.
  * [45] Bin Lin, Bin Zhu, Yang Ye, Munan Ning, Peng Jin, and Li Yuan.  Video-llava: Learning united visual representation by alignment before projection.  arXiv preprint arXiv:2311.10122, 2023.
  * [46] Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick.  Microsoft coco: Common objects in context.  In Computer Vision–ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13, pages 740–755. Springer, 2014.
  * [47] Jen-Yu Liu, Yu-Hua Chen, Yin-Cheng Yeh, and Yi-Hsuan Yang.  Unconditional audio generation with generative adversarial networks and cycle regularization.  arXiv preprint arXiv:2005.08526, 2020.
  * [48] Zhaoyang Liu, Donghao Luo, Yabiao Wang, Limin Wang, Ying Tai, Chengjie Wang, Jilin Li, Feiyue Huang, and Tong Lu.  Teinet: Towards an efficient architecture for video recognition.  In Proceedings of the AAAI conference on artificial intelligence, volume 34, pages 11669–11676, 2020.
  * [49] Ze Liu, Jia Ning, Yue Cao, Yixuan Wei, Zheng Zhang, Stephen Lin, and Han Hu.  Video swin transformer.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 3202–3211, 2022.
  * [50] Zhaoyang Liu, Limin Wang, Wayne Wu, Chen Qian, and Tong Lu.  Tam: Temporal adaptive module for video recognition.  In Proceedings of the IEEE/CVF international conference on computer vision, pages 13708–13718, 2021.
  * [51] Ilya Loshchilov and Frank Hutter.  Decoupled weight decay regularization.  arXiv preprint arXiv:1711.05101, 2017.
  * [52] Ang Lv, Xu Tan, Peiling Lu, Wei Ye, Shikun Zhang, Jiang Bian, and Rui Yan.  Getmusic: Generating any music tracks with a unified representation and diffusion framework.  arXiv preprint arXiv:2305.10841, 2023.
  * [53] Lin Ma.  Research on the effect of different types of short music videos on viewers’ psychological emotions.  Frontiers in Public Health, 10:992200, 2022.
  * [54] Kinyugo Maina.  Msanii: High fidelity music synthesis on a shoestring budget.  arXiv preprint arXiv:2301.06468, 2023.
  * [55] Antoine Miech, Dimitri Zhukov, Jean-Baptiste Alayrac, Makarand Tapaswi, Ivan Laptev, and Josef Sivic.  Howto100m: Learning a text-video embedding by watching hundred million narrated video clips.  In Proceedings of the IEEE/CVF international conference on computer vision, pages 2630–2640, 2019.
  * [56] Barbara Millet, Juan Chattah, and Soyeon Ahn.  Soundtrack design: The impact of music on visual attention and affective responses.  Applied ergonomics, 93:103301, 2021.
  * [57] Gautam Mittal, Jesse Engel, Curtis Hawthorne, and Ian Simon.  Symbolic music generation with diffusion models.  arXiv preprint arXiv:2103.16091, 2021.
  * [58] Louis-Philippe Morency, Rada Mihalcea, and Payal Doshi.  Towards multimodal sentiment analysis: Harvesting opinions from the web.  In Proceedings of the 13th international conference on multimodal interfaces, pages 169–176, 2011.
  * [59] Shehan Munasinghe, Rusiru Thushara, Muhammad Maaz, Hanoona Abdul Rasheed, Salman Khan, Mubarak Shah, and Fahad Khan.  Pg-video-llava: Pixel grounding large video-language models.  arXiv preprint arXiv:2311.13435, 2023.
  * [60] Muhammad Ferjad Naeem, Seong Joon Oh, Youngjung Uh, Yunjey Choi, and Jaejun Yoo.  Reliable fidelity and diversity metrics for generative models.  In International Conference on Machine Learning, pages 7176–7185. PMLR, 2020.
  * [61] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al.  Learning transferable visual models from natural language supervision.  In International conference on machine learning, pages 8748–8763. PMLR, 2021.
  * [62] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu.  Exploring the limits of transfer learning with a unified text-to-text transformer.  The Journal of Machine Learning Research, 21(1):5485–5551, 2020.
  * [63] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer.  High-resolution image synthesis with latent diffusion models.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10684–10695, 2022.
  * [64] Joseph Rothstein.  MIDI: A comprehensive introduction, volume 7.  AR Editions, Inc., 1995.
  * [65] Andrew Rouditchenko, Angie Boggust, David Harwath, Brian Chen, Dhiraj Joshi, Samuel Thomas, Kartik Audhkhasi, Hilde Kuehne, Rameswar Panda, Rogerio Feris, et al.  Avlnet: Learning audio-visual language representations from instructional videos.  arXiv preprint arXiv:2006.09199, 2020.
  * [66] Flavio Schneider, Ojasv Kamal, Zhijing Jin, and Bernhard Schölkopf.  Mo$\backslash$^ usai: Text-to-music generation with long-context latent diffusion.  arXiv preprint arXiv:2301.11757, 2023.
  * [67] Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al.  Laion-5b: An open large-scale dataset for training next generation image-text models.  Advances in Neural Information Processing Systems, 35:25278–25294, 2022.
  * [68] Bowen Shi, Wei-Ning Hsu, Kushal Lakhotia, and Abdelrahman Mohamed.  Learning audio-visual speech representation by masked multimodal cluster prediction.  arXiv preprint arXiv:2201.02184, 2022.
  * [69] Krishna Srinivasan, Karthik Raman, Jiecao Chen, Michael Bendersky, and Marc Najork.  Wit: Wikipedia-based image text dataset for multimodal multilingual machine learning.  In Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 2443–2449, 2021.
  * [70] Kun Su, Judith Yue Li, Qingqing Huang, Dima Kuzmin, Joonseok Lee, Chris Donahue, Fei Sha, Aren Jansen, Yu Wang, Mauro Verzetti, et al.  V2meow: Meowing to the visual beat via music generation.  arXiv preprint arXiv:2305.06594, 2023.
  * [71] Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto.  Stanford alpaca: An instruction-following llama model.  https://github.com/tatsu-lab/stanford_alpaca, 2023.
  * [72] Yapeng Tian, Dingzeyu Li, and Chenliang Xu.  Unified multisensory perception: Weakly-supervised audio-visual video parsing.  In Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part III 16, pages 436–454. Springer, 2020.
  * [73] Zhan Tong, Yibing Song, Jue Wang, and Limin Wang.  Videomae: Masked autoencoders are data-efficient learners for self-supervised video pre-training.  Advances in neural information processing systems, 35:10078–10093, 2022.
  * [74] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al.  Llama: Open and efficient foundation language models.  arXiv preprint arXiv:2302.13971, 2023.
  * [75] Du Tran, Heng Wang, Lorenzo Torresani, Jamie Ray, Yann LeCun, and Manohar Paluri.  A closer look at spatiotemporal convolutions for action recognition.  In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, pages 6450–6459, 2018.
  * [76] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  Advances in neural information processing systems, 30, 2017.
  * [77] Chengyi Wang, Sanyuan Chen, Yu Wu, Ziqiang Zhang, Long Zhou, Shujie Liu, Zhuo Chen, Yanqing Liu, Huaming Wang, Jinyu Li, et al.  Neural codec language models are zero-shot text to speech synthesizers.  arXiv preprint arXiv:2301.02111, 2023.
  * [78] Peng Wang, Shijie Wang, Junyang Lin, Shuai Bai, Xiaohuan Zhou, Jingren Zhou, Xinggang Wang, and Chang Zhou.  One-peace: Exploring one general representation model toward unlimited modalities.  arXiv preprint arXiv:2305.11172, 2023.
  * [79] Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero P Simoncelli.  Image quality assessment: from error visibility to structural similarity.  IEEE transactions on image processing, 13(4):600–612, 2004.
  * [80] Ziyu Wang, Ke Chen, Junyan Jiang, Yiyi Zhang, Maoran Xu, Shuqi Dai, Xianbin Gu, and Gus Xia.  Pop909: A pop-song dataset for music arrangement generation.  arXiv preprint arXiv:2008.07142, 2020.
  * [81] Zhiguo Wang, Patrick Ng, Xiaofei Ma, Ramesh Nallapati, and Bing Xiang.  Multi-passage bert: A globally normalized bert model for open-domain question answering.  arXiv preprint arXiv:1908.08167, 2019.
  * [82] Shengqiong Wu, Hao Fei, Leigang Qu, Wei Ji, and Tat-Seng Chua.  Next-gpt: Any-to-any multimodal llm.  arXiv preprint arXiv:2309.05519, 2023.
  * [83] Yusong Wu, Ke Chen, Tianyu Zhang, Yuchen Hui, Taylor Berg-Kirkpatrick, and Shlomo Dubnov.  Large-scale contrastive language-audio pretraining with feature fusion and keyword-to-caption augmentation.  In ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1–5. IEEE, 2023.
  * [84] Yazhou Xing, Yingqing He, Zeyue Tian, Xintao Wang, and Qifeng Chen.  Seeing and hearing: Open-domain visual-audio generation with diffusion latent aligners.  arXiv preprint arXiv:2402.17723, 2024.
  * [85] Dongchao Yang, Songxiang Liu, Rongjie Huang, Jinchuan Tian, Chao Weng, and Yuexian Zou.  Hifi-codec: Group-residual vector quantization for high fidelity audio codec.  arXiv preprint arXiv:2305.02765, 2023.
  * [86] Dongchao Yang, Jianwei Yu, Helin Wang, Wen Wang, Chao Weng, Yuexian Zou, and Dong Yu.  Diffsound: Discrete diffusion model for text-to-sound generation.  IEEE/ACM Transactions on Audio, Speech, and Language Processing, 2023.
  * [87] Ziyi Yang, Yuwei Fang, Chenguang Zhu, Reid Pryzant, Dongdong Chen, Yu Shi, Yichong Xu, Yao Qian, Mei Gao, Yi-Ling Chen, et al.  i-code: An integrative and composable multimodal learning framework.  In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pages 10880–10890, 2023.
  * [88] Ruibin Yuan, Hanfeng Lin, Yi Wang, Zeyue Tian, Shangda Wu, Tianhao Shen, Ge Zhang, Yuhang Wu, Cong Liu, Ziya Zhou, et al.  Chatmusician: Understanding and generating music intrinsically with llm.  arXiv preprint arXiv:2402.16153, 2024.
  * [89] Manzil Zaheer, Guru Guruganesh, Kumar Avinava Dubey, Joshua Ainslie, Chris Alberti, Santiago Ontanon, Philip Pham, Anirudh Ravula, Qifan Wang, Li Yang, et al.  Big bird: Transformers for longer sequences.  Advances in neural information processing systems, 33:17283–17297, 2020.
  * [90] Neil Zeghidour, Alejandro Luebs, Ahmed Omran, Jan Skoglund, and Marco Tagliasacchi.  Soundstream: An end-to-end neural audio codec.  IEEE/ACM Transactions on Audio, Speech, and Language Processing, 30:495–507, 2021.
  * [91] Hang Zhang, Xin Li, and Lidong Bing.  Video-llama: An instruction-tuned audio-visual language model for video understanding.  arXiv preprint arXiv:2306.02858, 2023.
  * [92] Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al.  Judging llm-as-a-judge with mt-bench and chatbot arena.  Advances in Neural Information Processing Systems, 36, 2024.
  * [93] Luowei Zhou, Chenliang Xu, and Jason Corso.  Towards automatic learning of procedures from web instructional videos.  In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32, 2018.
  * [94] Qiushi Zhu, Long Zhou, Ziqiang Zhang, Shujie Liu, Binxing Jiao, Jie Zhang, Lirong Dai, Daxin Jiang, Jinyu Li, and Furu Wei.  Vatlm: Visual-audio-text pre-training with unified masked prediction for speech representation learning.  IEEE Transactions on Multimedia, 2023.
  * [95] Ye Zhu, Kyle Olszewski, Yu Wu, Panos Achlioptas, Menglei Chai, Yan Yan, and Sergey Tulyakov.  Quantized gan for complex music generation from dance videos.  In European Conference on Computer Vision, pages 182–199. Springer, 2022.
  * [96] Le Zhuo, Zhaokai Wang, Baisen Wang, Yue Liao, Chenxi Bao, Stanley Peng, Songhao Han, Aixi Zhang, Fei Fang, and Si Liu.  Video background music generation: Dataset, method and evaluation.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 15637–15647, 2023.

##  Appendix A Additional Experiments

Additional experiments focusing on pretraining effects, visual encoders, and
codebook patterns are provided in the appendix. These parts provide insight
into the decision-making process for selecting the experimental configurations
within the VidMuse framework.

Pretraining Effect. Our ablation study on the effects of the data scale during
finetuning, as detailed in Table A1, highlights a balance between data size
and model performance. Despite not performing best in all the metrics, the
model finetuned with 20k pair data emerges as our choice. The 20k data offers
a compelling trade-off: it significantly improves performance across key
metrics without requiring the extensive computational resources that larger
datasets demand.

Visual Encoder. We conduct an ablation experiment to study the impact of
visual encoders. As shown in table A2, we observe that VidMuse is robust in
processing visual information for music generation. To trade-off between
computational costs and quality of generation, we select CLIP [61] as the
default visual encoder if not specified.

Codebook Pattern. The exploration of codebook interleaving patterns has
attracted attention from researchers across several domains [90, 77, 11, 85,
41]. In our ablation study focusing on the patterns, we find that while the
Parallel and Vall-E [77] patterns align with the findings for text-to-music
generation in MusicGen [11], the flattened codebook pattern does not
consistently exceed the performance of the delay pattern in tasks of
generating music from video. The delay pattern, notable for its relatively low
computational cost, is therefore selected for our implementation. The results
of this study are presented in table A3.

Table A1:  Ablation studies on the ratio of finetuning data. Finetuning Data |  Metrics
---|---
| KL$~{}\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}{\downarrow}$ | FD$~{}\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}{\downarrow}$ | FAD$~{}\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}{\downarrow}$ | density$~{}\color[rgb]{0.1953125,0.78515625,0.1953125}\definecolor[named]{pgfstrokecolor}{rgb}{0.1953125,0.78515625,0.1953125}{\uparrow}$ | coverage$~{}\color[rgb]{0.1953125,0.78515625,0.1953125}\definecolor[named]{pgfstrokecolor}{rgb}{0.1953125,0.78515625,0.1953125}{\uparrow}$ | Imagebind$~{}\color[rgb]{0.1953125,0.78515625,0.1953125}\definecolor[named]{pgfstrokecolor}{rgb}{0.1953125,0.78515625,0.1953125}{\uparrow}$
0 | $0.750$ | 35.788 | $18.847$ | $1.088$ | $0.653$ | $0.194$
10k | $0.792$ | $35.824$ | $2.383$ | $1.135$ | $0.653$ | $0.206$
20k | 0.738 | $36.171$ | 2.369 | $1.175$ | $0.710$ | 0.207
40k | $0.771$ | $36.521$ | $2.476$ | $1.150$ | 0.716 | $0.195$
60k | $0.760$ | $36.444$ | $20.519$ | 1.213 | $0.706$ | $0.197$
|  |  |  |  |  |
Table A2:  Ablation studies on video encoders. Encoders |  Metrics
---|---
| KL$~{}\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}{\downarrow}$ | FD$~{}\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}{\downarrow}$ | FAD$~{}\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}{\downarrow}$ | density$~{}\color[rgb]{0.1953125,0.78515625,0.1953125}\definecolor[named]{pgfstrokecolor}{rgb}{0.1953125,0.78515625,0.1953125}{\uparrow}$ | coverage$~{}\color[rgb]{0.1953125,0.78515625,0.1953125}\definecolor[named]{pgfstrokecolor}{rgb}{0.1953125,0.78515625,0.1953125}{\uparrow}$ | Imagebind$~{}\color[rgb]{0.1953125,0.78515625,0.1953125}\definecolor[named]{pgfstrokecolor}{rgb}{0.1953125,0.78515625,0.1953125}{\uparrow}$
ViViT [6] | $0.822$ | $37.167$ | $1.962$ | $1.433$ | $0.870$ | $0.194$
VideoMAE [73] | $0.778$ | $37.900$ | $19.388$ | $1.074$ | $0.663$ | $0.209$
CLIP [61] | $0.843$ | $41.354$ | $2.413$ | $1.487$ | $0.840$ | $0.193$
ViT [19] | $0.876$ | $50.427$ | $4.823$ | $1.081$ | $0.503$ | $0.195$
|  |  |  |  |  |

(a) The distribution of video genres in our dataset.

(b) Comparisons with different datasets.

(c) Duration distributions of pretraining and finetuning data.

(d) Duration distribution of our benchmark data.

Figure A1: The statistics of our dataset. Table A3:  Ablation studies on codebook pattern. Patterns |  Metrics
---|---
| KL$~{}\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}{\downarrow}$ | FD$~{}\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}{\downarrow}$ | FAD$~{}\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}{\downarrow}$ | density$~{}\color[rgb]{0.1953125,0.78515625,0.1953125}\definecolor[named]{pgfstrokecolor}{rgb}{0.1953125,0.78515625,0.1953125}{\uparrow}$ | coverage$~{}\color[rgb]{0.1953125,0.78515625,0.1953125}\definecolor[named]{pgfstrokecolor}{rgb}{0.1953125,0.78515625,0.1953125}{\uparrow}$ | Imagebind$~{}\color[rgb]{0.1953125,0.78515625,0.1953125}\definecolor[named]{pgfstrokecolor}{rgb}{0.1953125,0.78515625,0.1953125}{\uparrow}$
Parallel | $0.921$ | $68.603$ | $18.243$ | $0.562$ | $0.183$ | $0.166$
Flatten | 0.819 | $52.931$ | $4.260$ | $1.351$ | $0.500$ | 0.201
Delay | $0.843$ | 41.354 | 2.413 | 1.487 | 0.840 | $0.193$
Vall-E | $0.866$ | $57.286$ | $4.681$ | $1.148$ | $0.354$ | $0.189$
|  |  |  |  |  |