# MEDIC: Zero-shot Music Editing with Disentangled Inversion Control

Huadai Liu∗
Zhejiang University
<EMAIL_ADDRESS>
&Jialei Wang
Zhejiang University
<EMAIL_ADDRESS>
&Rongjie Huang
Zhejiang University
<EMAIL_ADDRESS>&Yang Liu
Zhejiang University
<EMAIL_ADDRESS>
&Jiayang Xu
Zhejiang University
<EMAIL_ADDRESS>
&Zhou Zhao
Zhejiang University
<EMAIL_ADDRESS>Equal contributionCorresponding Author

###### Abstract

Text-guided diffusion models catalyze a paradigm shift in audio generation,
facilitating the adaptability of source audio to conform to specific textual
prompts. Recent advancements introduce inversion techniques, like DDIM
inversion, to zero-shot editing, exploiting pre-trained diffusion models for
audio modification. Nonetheless, our investigation exposes that DDIM inversion
suffers from an accumulation of errors across each diffusion step, undermining
its efficacy. And the lack of attention control hinders the fine-grained
manipulations of music. To counteract these limitations, we introduce the
Disentangled Inversion technique, which is designed to disentangle the
diffusion process into triple branches, thereby magnifying their individual
capabilities for both precise editing and preservation. Furthermore, we
propose the Harmonized Attention Control framework, which unifies the mutual
self-attention and cross-attention with an additional Harmonic Branch to
achieve the desired composition and structural information in the target
music. Collectively, these innovations comprise the Disentangled Inversion
Control (DIC) framework, enabling accurate music editing whilst safeguarding
structural integrity. To benchmark audio editing efficacy, we introduce ZoME-
Bench, a comprehensive music editing benchmark hosting 1,100 samples spread
across 10 distinct editing categories, which facilitates both zero-shot and
instruction-based music editing tasks. Our method demonstrates unparalleled
performance in edit fidelity and essential content preservation, outperforming
contemporary state-of-the-art inversion techniques.111Audio samples are
available at https://MEDIC-Zero.github.io/.222Benchmark data are available at
https://huggingface.co/datasets/liuhuadai/ZoME-Bench.

##  1 Introduction

Advancements in text-guided diffusion models (Song et al., 2020; Song & Ermon,
2020; Peebles & Xie, 2023) have markedly enhanced audio generation, leveraging
their impressive capability for realistic and varied outputs. These models
(Liu et al., 2023b; Huang et al., 2023; Liu et al., 2024) provide the
foundation for prompt-based audio editing, offering new opportunities to
modify audio landscapes in response to specific textual prompts. Early audio
editing strategies rely on training models from scratch or test-time
optimization, both of which are hampered by intensive computational demands.
Recent advances (Manor & Michaeli, 2024; Zhang et al., 2024) in zero-shot
audio editing have been made through DDIM (Song et al., 2020) and DDPM (Ho et
al., ) inversion techniques, but challenges remain.

Key among these challenges is maintaining editing fidelity - ensuring that the
model’s output matches the given instructions - and preserving essential
content, a hallmark of effective audio editing that targets only necessary
changes. Balancing these objectives involves a careful exchange of information
between the source and target branch in diffusion processes, with existing
inversion methods like DDIM proving suboptimal for conditional diffusions
(Mokady et al., 2023). Enhanced versions of edit-friendly DDPM inversion
(Huberman-Spiegelglas et al., 2024) make strides in preservation by imprinting
the source onto the noise space, yet at the expense of reduced modification
capabilities due to noise reduction.

In this work, we rigorously examine the shortcomings of the DDIM inversion
approach. Our comprehensive analysis, suggests that while techniques like DDIM
inversion offer an editable base, they fall short of precision, potentially
compromising the original’s essence. The implementation of Classifier-free
Guidance (CFG) (Ho & Salimans, 2021) further amplifies the accumulated errors.
Moreover, attention control (Cao et al., 2023; Hertz et al., 2022) has
demonstrated its power in achieving high fidelity and essential content
preservation. Yet, analogous control mechanisms in music editing are
noticeably absent. To bridge this gap, we introduce an innovative Disentangled
Inversion Control technique. This technique is comprised of two main
components: Harmonic Attention Control and Disentangled Inversion. The former
integrates cross-attention and mutual self-attention controls, supplemented by
a novel Harmonic Branch, which facilitates precise, fine-grained manipulations
for both rigid and non-rigid editing scenarios. The latter component
strategically deconstructs the diffusion process into three distinct branches,
enabling each to realize its full potential independently.

Acknowledging the lack of standardized benchmarks in the field of audio
editing, we introduce ZoME-Bench. The first music editing benchmark consists
of 1,100 audio clips, distributing them into 10 rigorously curated editing
categories. Each entry is carefully assembled, comprising a source prompt, a
target prompt, human instruction, and blended words intended for editing.
Experimental results on ZoME-Bench demonstrate that DIC outperforms baselines,
achieving substantial gains in both essential content preservation and editing
fidelity.

Our contributions can be summarized as follows. 1) We introduce a novel,
tuning-free methodology referred to as Disentangled Inversion Control (DIC),
designed to facilitate both consistent manipulations of musical elements and
intricate non-rigid editing tasks. 2) An Harmonic Attention Control framework
is introduced to unify cross-attention control and self-attention control,
which enables both rigid and non-rigid editing. 3) We present Disentangled
Inversion Technique to achieve superior results with negligible inversion
error without optimization by branch disentanglement, aiding in accurately
editing the music while preserving the structural information. 4) We introduce
a tailored benchmark for music editing, named ZoME-Bench, and Experimental
results on ZoME-Bench demonstrate that DIC outperforms baselines, achieving
substantial gains in both essential content preservation and editing fidelity.

##  2 Related Works

###  2.1 Text-based Audio editing

The objective of text-based audio editing (Paissan et al., 2023; Plitsis et
al., 2024; Han et al., 2023a) is to utilize diffusion models to manipulate
audio content based on the provided target prompt. Two primary challenges
command intense focus. The first is the fidelity of editing, which pertains to
the model’s ability to strictly follow the initial instructions. The second
challenge involves essential content preservation, a critical attribute of
competent audio editing models that ensures only suitable regional
modifications are made during the editing process. Existing methodologies for
addressing these intricate challenges typically follow one of three paths. The
first involves attempts to develop end-to-end editing models (Copet et al.,
2023; Agostinelli et al., 2023; Chen et al., 2024)that employ diffusion
processes. However, these efforts are often hampered by indirect training
strategies or a lack of comprehensive datasets. The second path involves test-
time optimization strategies that utilize large pre-trained models for editing
(Paissan et al., 2023; Plitsis et al., 2024). Despite their versatility, these
methods are often burdened by the significant computational demands of fine-
tuning diffusion models or optimizing text-embeddings for signal
reconstruction. Some methods may choose to employ both strategies (Kawar et
al., 2023), further increasing the computational load. The final path involves
inversion techniques, which typically use DDPM (Huberman-Spiegelglas et al.,
2024; Wu & De la Torre, 2023)/DDIM (Song et al., 2020) inversion strategies to
extract diffusion noise vectors that match the source signal. Given its rapid
and intuitive zero-shot editing capabilities, we have chosen inversion
techniques as our primary research framework. In this study, we propose a new
inversion technique named Disentangled Inversion Control. This technique aims
to achieve accurate editing while preserving structural information.

###  2.2 Inversion Techniques

The field of image inversion techniques has experienced significant progress
in recent years (Brooks et al., 2023; Kim et al., 2022; Parmar et al., 2023;
Dhariwal & Nichol, 2021). While the DDIM inversion proves effective for
unconditional diffusion models (Song et al., 2020; Wallace et al., 2023) , its
limitations become apparent when applied to text-guided diffusion models,
particularly when classifier-free guidance is necessary for meaningful
editing. A range of solutions (Mokady et al., 2023; Tumanyan et al., 2022)
have been proposed to address these challenges. For example, Negative-Prompt
Inversion strategically assigns conditioned text embeddings to Null-Text
embeddings, effectively reducing potential deviation during the editing
process. Conversely, Edit Friendly DDPM provides an alternative latent noise
space via modified DDPM sample distributions, promoting the successful
reconstruction of the desired image (Huberman-Spiegelglas et al., 2024).

Recently, optimization-based inversion methods using specific latent variables
have gained popularity (Ju et al., 2023; Kawar et al., 2023). These are
designed to minimize accumulated errors stemming from the DDIM inversion.
Techniques such as Null-Text Inversion (Mokady et al., 2023) are promising,
but they introduce complexity and instability into the optimization process,
making it relatively time-consuming. Differently, we introduce a plug-in-plus
method called Disentangled Inversion Control to separate branches, achieving
superior performance with considerably fewer computational resources.

Figure 1: The framework and rationale behind Disentangled Inversion Control.
Traditional DDIM inversion introduces incremental errors at each step, leading
to only a coarse approximation of the source audio’s distribution. The
implementation of classifier-free guidance exacerbates the error accumulation.
In contrast, our method intricately separates the diffusion process into a
triple-branch system, allowing each branch to fully realize its individual
potential.

##  3 Disentangled Inversion Control

###  3.1 Motivation

Figure 1 and Preliminaries in Appendix 2 reveal that while techniques like
DDIM inversion offer an editable base, they fall short of precision,
potentially compromising the original’s essence. The implementation of
Classifier-free Guidance (CFG) further amplifies the accumulated errors.

In the landscape of prompt-based editing (Dong et al., 2023; Kim et al., 2022;
Feng et al., 2023), the ability to grasp the subtleties of linguistics and
enable more granular cross-modal interactions stands as a formidable
challenge. Hertz et al. (2022) acknowledges that in image editing, the fusion
between text and visual modalities happens within the parameterized noise
prediction network $\epsilon_{\theta}$. This sparks a plethora of attention
control techniques that direct the image domain’s target denoiser network
$\hat{\epsilon}_{\theta}$ to better reflect target prompts. Yet, analogous
control mechanisms in music editing are noticeably absent.

Taking these insights forward, we introduce Disentangled Inversion Control
(DIC), a novel approach poised to achieve fine-grained zero-shot music
editing. DIC strategically disentangles the diffusion process as a triple-
branch, granting each branch to optimize its functionality. At the same time,
the strategy leverages harmonized attention control to facilitate targeted
editing, thus aligning with the dual objectives of preserving the original
audio essence and ensuring edit relevance. We will first introduce Harmonized
Attention Control in Section 3.2 and discuss Disentangled Inversion in Section
3.3.

###  3.2 Harmonized Attention Control Framework

The denoising architecture denoted as $\epsilon_{\theta}$, is structured as a
sequence of fundamental blocks, each comprising a residual block (He et al.,
2015) followed by self-attention and cross-attention modules (Vaswani et al.,
2023; Dosovitskiy et al., 2020; Liu et al., 2023a). At the denoising step $t$,
the output of the $(l-1)$-th block is passed through self-attention and then
aligned with textual cues from prompt $P$ within the cross-attention layer.
The attention mechanism is formalized as:

$\text{Attention}(Q,K,V)=MV=\text{Softmax}\left(\frac{QK^{T}}{\sqrt{d}}\right)V$
(1)

where $Q$ denotes the query features derived from the audio features, $K$, $V$
represent the key and value features, and $M$ is the attention maps. We
explore varying semantic transformations of audio content through bifurcated
attention control strategies—cross-attention for rigid changes and mutual
self-attention for non-rigid adjustments. Finally, we introduce an
intermediate branch to host the desired composition and structural information
in the target music.

Figure 2: The framework of Harmonized attention control (HAC). HAC unifies
cross-attention control and mutual self-attention control with an additional
branch named Harmonic Branch to host the desired composition and structural
information in the target music.

####  3.2.1 Cross-Attention Control

Cross-attention Control (CAC) aims to inject the attention maps that are
obtained from the generation with the original prompt ${\mathcal{P}}$, into a
second generation with the modified prompt ${\mathcal{P}}^{*}$. We follow the
practice of Prompt-to-Prompt and define CAC as Global Attention Refinement and
Local Attention Blend.

##### Global Attention Refinement

At a given time step $t$, the attention map $M_{t}$ for both the origin and
target branches is computed, averaging over all layers with respect to the
noised latent $\mathbf{z}_{t}$. We employ an alignment function $A$ that maps
each token index from the target prompt $\mathcal{P}^{*}$ to its equivalent in
$\mathcal{P}$ or to None for non-aligning tokens. The refinement action is
thus:

$\text{Refine}(M^{src}_{t},M^{tgt}_{t},t)=\begin{cases}(M^{tgt}_{t})_{i,j}&\text{if
}A(j)=\text{None},\\\ (M_{t})_{i,j}&\text{otherwise}.\end{cases}$ (2)

##### Local Attention Blends

Beyond global attention, we incorporate a blending mechanism as suggested by
Hertz et al. (2022) and Mokady et al. (2023). This method selectively
integrates and maintains certain semantics by using target blend words
$w^{tgt}$ for semantic additions and source blend words $w^{src}$ for semantic
preservation. At each denoising step $t$, the mechanism operates on the target
latent ${\bm{z}}^{tgt}_{t}$ as follows:

$\displaystyle m_{\text{tgt}}$
$\displaystyle=\text{Threshold}\left[M^{tgt}_{t},w_{\text{tgt}},k_{\text{tgt}}\right],$
(3) $\displaystyle m_{\text{src}}$
$\displaystyle=\text{Threshold}\left[M^{src}_{t},w_{\text{src}},k_{\text{src}}\right],$
(4) $\displaystyle{\bm{z}}^{tgt}_{t}$
$\displaystyle=(1-m^{tgt}+m^{src})\odot{\bm{z}}^{src}_{t}+(m^{tgt}-m^{src})\odot{\bm{z}}^{tgt}_{t}$
(5)

where $m_{\text{tgt}}$ and $m_{\text{src}}$ are binary masks and threshold
function is as delineated below:

$\text{Threshold}(M,k)=\begin{cases}1&\text{if }M_{i,j}\geq k,\\\ 0&\text{if
}M_{i,j}<k.\end{cases}$ (6)

For simplificity, we define the process of local editing in Equation 10 as:

${\bm{z}}^{tgt}_{t}=\text{LocalEdit}({\bm{z}}^{src}_{t},{\bm{z}}^{tgt}_{t},M^{src}_{t},M^{tgt}_{t},w_{\text{src}},w_{\text{tgt}})$
(7)

##### Scheduling Cross-Attention Control

Applying cross-attention control only at early stages ensures creative
flexibility while maintaining structure. Acting on insights from Hertz et al.
(2022), we limit cross-attention to the initial phases up to a cutoff point
$\tau_{c}$. This moderation allows us to capture the nuances and intended
changes in musical compositions effectively. The approach is defined as:

$\text{CrossEdit}(M_{t},M^{*}_{t},t)=\begin{cases}\text{Refine}(M^{src},M^{tgt})&\text{if}\
t\geq\tau_{c},\\\ M^{tgt}&\text{if}\ t<\tau_{c}.\end{cases}$ (8)

####  3.2.2 Mutual Self-Attention Control

We diverge from the conventional use of cross-attention mechanisms and instead
draw inspiration from the MasaCtrl (Cao et al., 2023) technique to refine
music structure through self-attention queries. These queries adeptly navigate
through non-rigid musical transformations, aligning with the designated
musical theme or instrument (target prompt). The process begins by sketching
the foundational musical theme using the target’s self-attention
components—$Q^{tgt}$, $K^{tgt}$, and $V^{tgt}$. This is followed by enriching
this theme with elements resembling the thematic content from the source
($K^{src}$, $V^{src}$), steered by $Q^{tgt}$. However, applying this attentive
modulation uniformly over all processing layers and through every denoising
step might result in a composition excessively mirroring the source.
Consequently, echoing the ethos of MasaCtrl, our proposed solution selectively
employs mutual self-attention in the decoder portion of our music editing
U-Net, initiated after a set number of denoising iterations.

##### Scheduling Mutual Self-Attention Control

The application of mutual self-attention is meticulously planned, beginning at
a specific denoising step $S$ and extending beyond a designated layer $L$. The
strength and influence of this control mechanism are designed as follows:

$\text{SelfEdit}({Q^{src},K^{src},V^{src}},{Q^{tgt},K^{tgt},V^{tgt}},t)=\begin{cases}{Q^{src},K^{src},V^{src}}&\text{if
}t\geq S\text{ and }l\geq L,\\\ {Q^{tgt},K^{src},V^{src}}&\text{otherwise}\\\
\end{cases}$ (9)

In this framework, $S$ signifies the denoising step from which the mutual
self-attention control commences, serving as a temporal threshold. Similarly,
$L$ distinguishes the layer index below which this nuanced control strategy
becomes operational, tailoring the musical output towards the intended
artistic direction.

####  3.2.3 Harmonic Branch Integretion

The naive combination of cross-attention control and mutual self-attention
control sequentially would lead to a sub-optimal outcome in the original dual-
branch setup, especially failing the global attention refinement. To address
this issue, we introduce an additional latent harmonic branch, which serves as
an intermediate to host the desired composition and structural information in
the target music. Our unified framework is detailed in Algorithm 2 and
illustrated in Figure 2. During each forward step of the diffusion process, we
start with mutual self-attention control on $z^{src}$ and $z^{tgt}$ and assign
the output to the harmonic branch latent $z^{har}$. Following this, cross-
attention control is applied on $M_{har}$ and $M_{tgt}$ to refine the semantic
information for $M_{tgt}$.

Algorithm 1 Harmonized Attention Control in one DDIM Forward Process

1: Input: A source prompt ${\mathcal{P}}$, a target prompt
${\mathcal{P}}^{*}$, a source audio latent ${\bm{z}}_{0}$, denoising network
$\epsilon_{\theta}(\cdot,\cdot,\cdot)$, current time step $\tau$, source and
target blend words $w_{\text{src}},w_{\text{tgt}}$, input latents
$z^{\text{src}}_{\tau},z^{\text{tgt}}_{\tau},z^{\text{har}}_{\tau}$.

2:
$\epsilon_{\text{src}},\\{Q^{\text{src}},K^{\text{src}},V^{\text{src}}\\},M_{\text{src}}=\epsilon_{\theta}(z^{\text{src}}_{\tau},\tau,c_{\text{src}})$

3:
$\epsilon_{\text{tgt}},\\{Q^{\text{tgt}},K^{\text{tgt}},V^{\text{tgt}}\\},M_{\text{tgt}}=\epsilon_{\theta}(z^{\text{tgt}}_{\tau},\tau,c_{\text{tgt}})$

4:
$\\{Q^{\text{har}},K^{\text{har}},V^{\text{har}}\\}=\text{SelfEdit}(\\{Q^{\text{src}},K^{\text{src}},V^{\text{src}}\\},\\{Q^{\text{tgt}},K^{\text{tgt}},V^{\text{tgt}}\\},\tau)$

5:
$\epsilon_{\text{har}},M_{\text{har}}=\epsilon_{\theta}(z^{\text{har}}_{\tau},\tau,c_{\text{src}};\\{Q^{\text{har}},K^{\text{har}},V^{\text{har}}\\})$

6: $\hat{M^{\text{tgt}}}=\text{CrossEdit}(M_{\text{har}},M_{\text{tgt}},\tau)$

7:
$\hat{\epsilon_{\text{tgt}}}=\epsilon_{\theta}(z^{\text{tgt}}_{\tau},\tau,c_{\text{tgt}};\hat{M^{\text{tgt}}})$

8:
$z^{\text{src}}_{\tau-1},z^{\text{tgt}}_{\tau-1},z^{\text{har}}_{\tau-1}=\text{Sample}([z^{\text{src}}_{\tau},z^{\text{tgt}}_{\tau},z^{\text{har}}_{\tau}],[\epsilon_{\text{src}},\epsilon_{c_{\text{tgt}}},\epsilon_{\text{har}}],\tau)$

9: ${\bm{z}}^{tgt}_{\tau-1}$ =
$\text{LocalEdit}({\bm{z}}^{src}_{\tau-1},{\bm{z}}^{tgt}_{\tau-1},M^{src}_{\tau-1},M^{tgt}_{t},w_{src},w_{tgt})$

10: Output:
$z^{\text{src}}_{\tau-1},z^{\text{tgt}}_{\tau-1},z^{\text{har}}_{\tau-1}$

###  3.3 Disentangled Inversion Technique

Recognizing the limitations of using DDIM inversion without classifier-free
guidance, we observe that it yields an easily modifiable but imprecise
approximation of the original audio signal. Increasing the guidance scale
enhances editability but sacrifices reconstruction accuracy due to latent code
deviation during editing.

In order to address this issue, our methodology, which we have termed the
Disentangled Inversion Technique disentangles into three branches: the source,
the target, and the harmonized branch. This bifurcation is designed to
optimize the performance of each branch separately. For the source branch, we
implement a targeted correction mechanism. By reintegrating the distance
$z^{*}_{t}-z^{src}_{t}$ into $z^{src}_{t}$, we directly mitigate the deviation
of the pathway. This straightforward adjustment effectively rectifies the path
and minimizes the accumulated errors introduced by both DDIM inversion and
classifier-free guidance, thereby enhancing consistency in the reconstructed
audio. On the other hand, the target branch and harmonic branch is left
unmodified to fully leverage the innate capabilities of diffusion models in
generating the desired target audio. The branches’ untouched state ensures
that the model’s potential is utilized to its fullest extent, thereby ensuring
the fidelity and integrity of the generated audio. We will further discuss
this in the Section 5.4.2. The algorithm of Disentangled Inversion Technique
has been outlined in Algorithm 2.

Typical diffusion-based editing (Han et al., 2023b; Miyake et al., 2023)
involves two parts: an inversion process to get the diffusion space of the
audio, and a forward process to perform editing on the diffusion space.
Disentangled Inversion can be plug-and-played into the forward process and
rectifies the deviation path step by step. Specifically, Disentangled
Inversion first computes the difference between ${\bm{z}}^{*}_{t-1}$ and
${\bm{z}}^{src}_{t-1}$, then adds the difference back to
${\bm{z}}^{src}_{t-1}$ in DDIM forward. We only add the difference of the
source prompt in latent space and update ${\bm{z}}^{src}_{t-1}$, which is the
key to retaining the editability of the target prompt’s latent space.

Algorithm 2 Disentangled Inversion Technique

1: Input: A source prompt ${\mathcal{P}}$, a target prompt
${\mathcal{P}}^{*}$, a source audio latent ${\bm{z}}_{0}$, and guidance scale
$\omega$.

2: Output: A edited audio latent ${\bm{z}}^{tgt}_{0}$.

3: Compute the intermediate results ${\bm{z}}^{*}_{T},...,{\bm{z}}^{*}_{1}$
using DDIM inversion over ${\bm{z}}_{0}$.

4: Initialize ${\bm{z}}^{src}_{T}\leftarrow{\bm{z}}^{*}_{T}$,
${\bm{z}}^{tgt}_{T}\leftarrow{\bm{z}}^{*}_{T}$,
${\bm{z}}^{har}_{T}\leftarrow{\bm{z}}^{*}_{T}$.

5: for $t=T$ to $1$ do

6:
$[{\bm{d}}^{src}_{t-1},{\bm{d}}^{tgt}_{t-1},{\bm{d}}^{har}_{t-1}]\leftarrow{\bm{z}}^{*}_{t-1}$
\-
DDIM_Forward$({\bm{z}}^{src}_{t},t,[{\mathcal{P}},{\mathcal{P}}^{*},{\mathcal{P}}],\omega)$

7: ${\bm{z}}^{src}_{t-1}\leftarrow$
DDIM_Forward(${\bm{z}}^{src}_{t},t,[{\mathcal{P}},{\mathcal{P}}^{*},{\mathcal{P}}],\omega$)
+ $[{\bm{d}}^{src}_{t-1},{\bm{0}},{\bm{0}}]$

8: ${\bm{z}}^{har}_{t-1}\leftarrow$
DDIM_Forward(${\bm{z}}^{har}_{t},t,[{\mathcal{P}},{\mathcal{P}}^{*},{\mathcal{P}}],\omega$)
+ $[{\bm{d}}^{src}_{t-1},{\bm{0}},{\bm{0}}]$

9: ${\bm{z}}^{tgt}_{t-1}\leftarrow$
DDIM_Forward(${\bm{z}}^{tgt}_{t},t,[{\mathcal{P}},{\mathcal{P}}^{*},{\mathcal{P}}],\omega$)
+ $[{\bm{d}}^{src}_{t-1},{\bm{0}},{\bm{0}}]$

10: end for

11: return ${\bm{z}}^{tgt}_{0}$

##  4 BenchMark Construction

Despite recent significant advancements in music generation utilizing the
latent diffusion model, music editing has received comparably less attention,
especially zero-shot music editing. Hence, to systematically validate our
proposed method as a plug-and-play strategy for editing models and compare our
method with existing zero-shot music editing methods, as well as compensate
for the absence of standardized performance criteria for inversion and editing
techniques, we construct a benchmark dataset, named ZoME-Bench (Zero-shot
Music Editing Benchmark). ZoME-Bench comprises 1,100 audio samples which are
selected from MusicCaps, spanning ten distinct editing types such as add
instrument, delete instrument, change genre, and so on. Each sample in this
collection is accompanied by its corresponding source prompt, target prompt,
human instruction, and the source audio. Additionally, annotations pertinent
to attention control methodologies, such as blended words, are meticulously
included to facilitate detailed evaluations. We attach the details about our
benchmark in Appendix B.

##  5 Experiments

###  5.1 Implementation Details

We perform the inference of different editing methods using the pre-trained
AudioLDM 2 (Liu et al., 2023b) models with 200 inference steps. The setting of
baselines is followed by Manor & Michaeli (2024) and is all evaluated in
NVIDIA 4090Ti GPU for a fair comparison. We evaluate all methods in ZoMo-Bench
across all editing types. Details can be found in Appendix C.

###  5.2 Evaluation Metrics

In order to evaluate the performance of our novel Disentangled Inversion
Control framework, we employ a diverse set of five metrics that encompass
three critical dimensions of audio quality and fidelity: Struture
Distance,CLAP Score,LPAPS,MSE (Mean Squared Error)and FAD (Fréchet Audio
Distance). All the metrics are introduced in detail in Appendix C.1.

###  5.3 Zero-shot Music Editing Results

Method | $\text{Structure Distance}_{\times 10^{3}}\downarrow$ | LPAPS$\downarrow$ | FAD$\downarrow$ | $\text{MSE}_{\times 10^{5}}\downarrow$ | CLAP Score$\uparrow$
---|---|---|---|---|---
SDEdit | 25.87 | 0.22 | 12.18 | 9.82 | 0.40
DDPM-Friendly | 22.36 | 0.21 | 14.08 | 8.49 | 0.41
DDIM Inversion | 17.11 | 0.18 | 6.24 | 6.12 | 0.55
DIC | 11.97 | 0.15 | 2.49 | 4.54 | 0.60
Table 1: Comparison with baselines on ZoMo-Bench about struture and content
preservation and CLAP similiarity. DIC: our proposed Disentangled Inversion
Control.

We present a comparative study of our Disentangled Inversion Control(DIC)
against several established music editing techniques, namely SDEdit (Liu et
al., ), DDIM Inversion Ho et al. , and DDPM-Friendly (Manor & Michaeli, 2024).
Utilizing the ZoMo-Bench test set we released, we assess the quality of the
generated audio samples. The assessment primarily concentrates on two key
aspects: Essential Content Preservation and Editing Fidelity. The results are
summarized in Table 1, leading to two main conclusions:

(1) Our Disentangled Inversion Control demonstrates substantial advancements
in maintaining essential content and enhancing editing fidelity, in comparison
to the baseline methods. (2) Despite DDIM Inversion achieving high CLAP
similarity scores, significantly underperforms in preserving essential content
when compared to our approach.

(a) Spider chart for editing quality.

(b) Spider chart for editing consistency.

Figure 3: A comprehensive performance evaluation on the ZoME-bench. We present
spider charts of editing quality (CLAP Scores) and consistency (LPAPS) across
9 editing tasks for DDPM-Friendly, DDIM Inversion, and Disentangled Inversion
Control methods.

To substantiate the robustness of our method, we delve into fine-grained
comparisons across different types of editing. The LPAPS and CLAP scores are
depicted in Figure 3. The insights gleaned from this analysis are as follows:
(1) Across all editing types, our Disentangled Inversion Control surpasses
other methods, demonstrating its prowess in handling both rigid and non-rigid
editing tasks. (1) While methods like DDIM Inversion exhibit capabilities in
handling certain rigid editing scenarios, they fall short in executing fine-
grained manipulations, as reflected in their inferior performance, especially
in the "Delete Instrument" and "Change Melody" non-rigid editing type.

###  5.4 Ablation Study

The ablation studies presented in this section aim to validate the
contributions of the Unified Attention Control and Refining Dual-path
Inversion to our framework’s overall performance. Additionally, we examine the
impact of the classifier-free guidance scale on editing outcomes, with a
comprehensive analysis included in the Appendix D.

####  5.4.1 Ablation on Attention Control Methods

Method | $\text{Structure Distance}_{\times 10^{3}}\downarrow$ | LPAPS$\downarrow$ | FAD$\downarrow$ | $\text{MSE}_{\times 10^{5}}\downarrow$ | CLAP Score$\uparrow$
---|---|---|---|---|---
HAC | 11.97 | 0.15 | 2.49 | 4.54 | 0.60
w/ CAC | 10.24 | 0.19 | 2.75 | 4.76 | 0.56
w/ MSAC | 9.92 | 0.18 | 2.50 | 4.66 | 0.58
w/ HB | 11.75 | 0.16 | 2.59 | 4.67 | 0.59
Table 2: Ablation results about different control methods. HAC: Harmonized
Attention Control.

To ascertain the impact of our attention control mechanisms, we perform
ablations on the following configurations: Cross-attention Control Only (w/
CAC), Mutual Self-attention Control only (w/ MSAC), without Harmonic Branch
(w/o HB), and any attention control (w/o Control). The findings, detailed in
Table 2, reveal that: (1) Both cross-attention and mutual self-attention
controls individually enhance editing performance. (2) The combined use of CAC
and MSAC further improves results, but the absence of the harmonic branch
yields suboptimal outcomes.

####  5.4.2 Ablation on Disentangled Inversion Technique

To demonstrate the soundness of Algorithm 2 and the effective disentanglement
of triple branches, we can draw conclusions from the Table 3: 1) Incorporating
source distance into the target latent and harmonic branch, this leads to a
decline in both preservation metrics and CLAP similarity score. 2) The
incorporation of target distance into target latent and harmonic branch leads
to an improvement in structure and content integrity but the CLAP score
sharply decreases. 3) Add the harmonic distance into the harmonic latent
witness a distant decrease.

Distance | $\text{Structure Distance}_{\times 10^{3}}\downarrow$ | LPAPS$\downarrow$ | FAD$\downarrow$ | $\text{MSE}_{\times 10^{5}}\downarrow$ | CLAP Score$\uparrow$
---|---|---|---|---|---
$[{\bm{d}}_{src},{\bm{d}}_{src},{\bm{0}}]$ | 14.28 | 0.17 | 2.64 | 4.67 | 0.57
$[{\bm{d}}_{src},{\bm{0}},{\bm{d}}_{src}]$ | 13.17 | 0.17 | 2.55 | 4.59 | 0.58
$[{\bm{d}}_{src},{\bm{d}}_{tgt},{\bm{0}}]$ | 11.24 | 0.15 | 2.44 | 4.32 | 0.56
$[{\bm{d}}_{src},{\bm{0}},{\bm{d}}_{tgt}]$ | 11.12 | 0.14 | 2.41 | 4.29 | 0.56
$[{\bm{d}}_{src},{\bm{0}},{\bm{d}}_{har}]$ | 37.51 | 0.27 | 27.2 | 14.23 | 0.28
$[{\bm{d}}_{src},{\bm{0}},{\bm{0}}]$ | 11.97 | 0.15 | 2.49 | 4.54 | 0.60
Table 3: Ablation results about disentangled inversion technique.
$[\cdot,\cdot,\cdot]$ denotes adding the distance to the corresponding branch
in Algorithm 2.

###  5.5 Qualitative Results

Figure 4: Visualizations of the source audio’s mel and edited mel-spectrograms
by different editing methods.

To complement our quantitative findings, we present a qualitative comparison
in Figure 4. Methods such as SDEdit and inversion-based techniques often
struggle to balance high editability with the preservation of melodic content
and harmonic structure. In contrast, our Disentangled Inversion Control
demonstrates superior performance in precise music editing while preserving
structural integrity. We provide additional qualitative results in the
Appendix G for further examination.

##  6 Conclusion

In this paper, we explored the burgeoning realm of text-guided diffusion
models for audio generation, recognizing their potential to reshape source
audio in alignment with specific textual prompts. Our retrospective analysis
revealed that while recent methods like DDIM inversion have propelled zero-
shot editing forward, they are not without their shortcomings. We identified
that DDIM inversion is plagued by an accumulation of errors throughout the
diffusion steps, which detracts from its overall effectiveness. Moreover, the
absence of a fine-tuned attention control mechanism was found to impede the
nuanced editing of music. To address these issues, we introduced the
Disentangled Inversion framework, a novel approach that meticulously separates
the source and target branches to amplify their individual strengths in
preservation and editing. In tandem, we presented the Harmonized Attention
Control technique, which enhances the preservation of essential content by
integrating the attention maps from the original audio into the diffusion
process. Our comprehensive evaluations, conducted on the ZoME-Bench—a robust
benchmark for music editing comprising 1,000 samples across 10 varied editing
categories—attested to the superiority of our methods.

## References

  * Agostinelli et al. (2023) Agostinelli, A., Denk, T. I., Borsos, Z., Engel, J., Verzetti, M., Caillon, A., Huang, Q., Jansen, A., Roberts, A., Tagliasacchi, M., et al.  Musiclm: Generating music from text.  _arXiv preprint arXiv:2301.11325_ , 2023.
  * Brooks et al. (2023) Brooks, T., Holynski, A., and Efros, A. A.  Instructpix2pix: Learning to follow image editing instructions.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pp. 18392–18402, 2023.
  * Cao et al. (2023) Cao, M., Wang, X., Qi, Z., Shan, Y., Qie, X., and Zheng, Y.  Masactrl: Tuning-free mutual self-attention control for consistent image synthesis and editing.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pp. 22560–22570, 2023.
  * Chen et al. (2024) Chen, K., Wu, Y., Liu, H., Nezhurina, M., Berg-Kirkpatrick, T., and Dubnov, S.  Musicldm: Enhancing novelty in text-to-music generation using beat-synchronous mixup strategies.  In _ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , pp. 1206–1210. IEEE, 2024.
  * Copet et al. (2023) Copet, J., Kreuk, F., Gat, I., Remez, T., Kant, D., Synnaeve, G., Adi, Y., and Défossez, A.  Simple and controllable music generation, 2023.
  * Dhariwal & Nichol (2021) Dhariwal, P. and Nichol, A.  Diffusion models beat gans on image synthesis.  _Advances in neural information processing systems_ , 34:8780–8794, 2021.
  * Dong et al. (2023) Dong, W., Xue, S., Duan, X., and Han, S.  Prompt tuning inversion for text-driven image editing using diffusion models, 2023.
  * Dosovitskiy et al. (2020) Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., et al.  An image is worth 16x16 words: Transformers for image recognition at scale.  _arXiv preprint arXiv:2010.11929_ , 2020.
  * (9) Elizalde, B., Deshmukh, S., Ismail, M. A., and Wang, H.  Clap: Learning audio concepts from natural language supervision.
  * Feng et al. (2023) Feng, W., He, X., Fu, T.-J., Jampani, V., Akula, A., Narayana, P., Basu, S., Wang, X. E., and Wang, W. Y.  Training-free structured diffusion guidance for compositional text-to-image synthesis, 2023.
  * Han et al. (2023a) Han, B., Dai, J., Hao, W., He, X., Guo, D., Chen, J., Wang, Y., Qian, Y., and Song, X.  Instructme: An instruction guided music edit and remix framework with latent diffusion models, 2023a.
  * Han et al. (2023b) Han, L., Wen, S., Chen, Q., Zhang, Z., Song, K., Ren, M., Gao, R., Stathopoulos, A., He, X., Chen, Y., Liu, D., Zhangli, Q., Jiang, J., Xia, Z., Srivastava, A., and Metaxas, D.  Improving tuning-free real image editing with proximal guidance, 2023b.
  * He et al. (2015) He, K., Zhang, X., Ren, S., and Sun, J.  Deep residual learning for image recognition, 2015.
  * Hertz et al. (2022) Hertz, A., Mokady, R., Tenenbaum, J., Aberman, K., Pritch, Y., and Cohen-Or, D.  Prompt-to-prompt image editing with cross attention control.  _arXiv preprint arXiv:2208.01626_ , 2022.
  * Ho & Salimans (2021) Ho, J. and Salimans, T.  Classifier-free diffusion guidance.  In _NeurIPS 2021 workshop on deep generative models and downstream applications_ , 2021.
  * (16) Ho, J., Jain, A., and Abbeel, P.  Denoising diffusion probabilistic models.  In _Proc. of NeurIPS_.
  * Huang et al. (2023) Huang, R., Huang, J., Yang, D., Ren, Y., Liu, L., Li, M., Ye, Z., Liu, J., Yin, X., and Zhao, Z.  Make-an-audio: Text-to-audio generation with prompt-enhanced diffusion models, 2023.
  * Huberman-Spiegelglas et al. (2024) Huberman-Spiegelglas, I., Kulikov, V., and Michaeli, T.  An edit friendly ddpm noise space: Inversion and manipulations, 2024.
  * Iashin & Rahtu (2021) Iashin, V. and Rahtu, E.  Taming visually guided sound generation, 2021.
  * Ju et al. (2023) Ju, X., Zeng, A., Bian, Y., Liu, S., and Xu, Q.  Direct inversion: Boosting diffusion-based editing with 3 lines of code.  _arXiv preprint arXiv:2310.01506_ , 2023.
  * Kawar et al. (2023) Kawar, B., Zada, S., Lang, O., Tov, O., Chang, H., Dekel, T., Mosseri, I., and Irani, M.  Imagic: Text-based real image editing with diffusion models.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pp. 6007–6017, 2023.
  * Kilgour et al. (2018) Kilgour, K., Zuluaga, M., Roblek, D., and Sharifi, M.  Fr$\backslash$’echet audio distance: A metric for evaluating music enhancement algorithms.  _arXiv preprint arXiv:1812.08466_ , 2018.
  * Kim et al. (2022) Kim, G., Kwon, T., and Ye, J. C.  Diffusionclip: Text-guided diffusion models for robust image manipulation.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pp. 2426–2435, 2022.
  * (24) Liu, H., Chen, Z., Yuan, Y., Mei, X., Liu, X., Mandic, D., Wang, W., and Plumbley, M. D.  AudioLDM: Text-to-Audio Generation with Latent Diffusion Models.
  * Liu et al. (2023a) Liu, H., Huang, R., Lin, X., Xu, W., Zheng, M., Chen, H., He, J., and Zhao, Z.  Vit-tts: visual text-to-speech with scalable diffusion transformer.  _arXiv preprint arXiv:2305.12708_ , 2023a.
  * Liu et al. (2023b) Liu, H., Tian, Q., Yuan, Y., Liu, X., Mei, X., Kong, Q., Wang, Y., Wang, W., Wang, Y., and Plumbley, M. D.  Audioldm 2: Learning holistic audio generation with self-supervised pretraining.  _arXiv preprint arXiv:2308.05734_ , 2023b.
  * Liu et al. (2024) Liu, H., Huang, R., Liu, Y., Cao, H., Wang, J., Cheng, X., Zheng, S., and Zhao, Z.  Audiolcm: Text-to-audio generation with latent consistency models.  _arXiv preprint arXiv:2406.00356_ , 2024.
  * Manor & Michaeli (2024) Manor, H. and Michaeli, T.  Zero-shot unsupervised and text-based audio editing using ddpm inversion.  _arXiv preprint arXiv:2402.10009_ , 2024.
  * Miyake et al. (2023) Miyake, D., Iohara, A., Saito, Y., and Tanaka, T.  Negative-prompt inversion: Fast image inversion for editing with text-guided diffusion models, 2023.
  * Mokady et al. (2023) Mokady, R., Hertz, A., Aberman, K., Pritch, Y., and Cohen-Or, D.  Null-text inversion for editing real images using guided diffusion models.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pp. 6038–6047, 2023.
  * OpenAI (2023) OpenAI.  Gpt-4 technical report, 2023.
  * Paissan et al. (2023) Paissan, F., Wang, Z., Ravanelli, M., Smaragdis, P., and Subakan, C.  Audio editing with non-rigid text prompts, 2023.
  * Parmar et al. (2023) Parmar, G., Kumar Singh, K., Zhang, R., Li, Y., Lu, J., and Zhu, J.-Y.  Zero-shot image-to-image translation.  In _ACM SIGGRAPH 2023 Conference Proceedings_ , pp. 1–11, 2023.
  * Peebles & Xie (2023) Peebles, W. and Xie, S.  Scalable diffusion models with transformers, 2023.
  * Plitsis et al. (2024) Plitsis, M., Kouzelis, T., Paraskevopoulos, G., Katsouros, V., and Panagakis, Y.  Investigating personalization methods in text to music generation.  In _ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , pp. 1081–1085. IEEE, 2024.
  * Song et al. (2020) Song, J., Meng, C., and Ermon, S.  Denoising diffusion implicit models.  In _Proc. of ICLR_ , 2020.
  * Song & Ermon (2020) Song, Y. and Ermon, S.  Improved techniques for training score-based generative models.  _Advances in neural information processing systems_ , 33:12438–12448, 2020.
  * Tumanyan et al. (2022) Tumanyan, N., Geyer, M., Bagon, S., and Dekel, T.  Plug-and-play diffusion features for text-driven image-to-image translation, 2022.
  * Vaswani et al. (2023) Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., and Polosukhin, I.  Attention is all you need, 2023.
  * Wallace et al. (2023) Wallace, B., Gokul, A., and Naik, N.  Edict: Exact diffusion inversion via coupled transformations.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pp. 22532–22541, 2023.
  * Wu & De la Torre (2023) Wu, C. H. and De la Torre, F.  A latent space of stochastic diffusion models for zero-shot image editing and guidance.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pp. 7378–7387, 2023.
  * Zhang et al. (2018) Zhang, R., Isola, P., Efros, A. A., Shechtman, E., and Wang, O.  The unreasonable effectiveness of deep features as a perceptual metric, 2018.
  * Zhang et al. (2024) Zhang, Y., Ikemiya, Y., Xia, G., Murata, N., Martínez-Ramírez, M. A., Liao, W.-H., Mitsufuji, Y., and Dixon, S.  Musicmagus: Zero-shot text-to-music editing via diffusion models, 2024.

##  Appendix A Preliminaries And Analyses

This section introduces the foundational concepts of DDIM sampling and
classifier-free guidance as applied in diffusion models for text-guided audio
synthesis. It further explores the challenges associated with these methods.

###  A.1 Diffusion Models

Text-guided diffusion models aim to map a random noise vector ${\bm{z}}_{t}$
and textual condition ${\bm{c}}$ to an output audio ${\bm{z}}_{0}$, which
corresponds to the given conditioning prompt. We train a denoiser network
$\epsilon_{\theta}({\bm{z}}_{t},t,{\bm{c}})$ to predict the gaussian noise
$\epsilon\in{\mathcal{N}}(0,{\mathbf{I}})$ following the objective:

$\underset{\theta}{min}{\mathbf{E}}_{{\bm{z}}_{0},\epsilon\in{\mathcal{N}}(0,{\mathbf{I}}),t\in
Uniform(1,T)}||\epsilon-\epsilon_{\theta}({\bm{z}}_{t},t,{\bm{c}})||^{2}$ (10)

Where noise is added to the sampled data ${\bm{z}}_{0}$ according to timestamp
$t$. At inference, given a noise vector ${\bm{z}}_{T}$, the noise $i$ is
gradually removed by sequentially predicting it using pre-trained network for
$T$ steps. To generate audios from given ${\bm{z}}_{T}$, we employ the
deterministic DDIM samling:

${\bm{z}}_{t-1}=\frac{\sqrt{\alpha_{t-1}}}{\sqrt{\alpha_{t}}}{\bm{z}}_{t}+\sqrt{\alpha_{t-1}}(\sqrt{\frac{1}{\alpha_{t-1}}-1}-\sqrt{\frac{1}{\alpha_{t}}-1})\epsilon_{\theta}({\bm{z}}_{t},t,{\bm{c}})$
(11)

###  A.2 DDIM Inversion

While diffusion models have superior characteristics in the feature space that
can support various down-stream tasks, it is hard to apply them to audios in
the absence of natural diffusion feature space for non-generated audios. Thus,
a simple inversion technique known as DDIM inversion is commonly used for
unconditional diffusion models, predicated on the presumption that the ODE
process can be reversed in the limit of infinitesimally small steps:

${\bm{z}}^{*}_{t}=\frac{\sqrt{\alpha_{t}}}{\sqrt{\alpha_{t-1}}}{\bm{z}}^{*}_{t-1}+\sqrt{\alpha_{t}}(\sqrt{\frac{1}{\alpha_{t}}-1}-\sqrt{\frac{1}{\alpha_{t-1}}-1})\epsilon_{\theta}({\bm{z}}^{*}_{t-1},t-1)$
(12)

However, in most text-based diffusion models, this presumption cannot be
guaranteed, resulting in a perturbation from ${\bm{z}}_{t}$ to
${\bm{z}}^{*}_{t}$. Consequently, an additional perturbation from
${\bm{z}}^{*}_{t}$ to ${\bm{z}}^{src}_{t}$ arises when sampling an audio from
${\bm{z}}^{*}_{T}$ where $\alpha$ is hyper-parameter:

${\bm{z}}_{t}=\sqrt{\alpha_{t}}{\bm{z}}_{0}+\sqrt{1-\alpha_{t}}\epsilon$ (13)

###  A.3 Classifier-free Guidance

Classifier-free Guidance (CFG) (Ho & Salimans, 2021) is proposed to overcome
the limitation of text-conditioned models, where text adherence could be weak.
However, a higher guidance scale, $\omega$, which is intended to strengthen
the model’s fidelity to the text prompt, inadvertently magnifies the
accumulated inversion error. This becomes problematic in editing scenarios
where precise control over the audio synthesis is desired. The modified noise
estimation in CFG can be expressed as:

$\hat{\epsilon}_{\theta}({\bm{z}}_{t},t,{\bm{c}},\varnothing)=\omega\cdot\epsilon_{\theta}({\bm{z}}_{t},t,{\bm{c}})+(1-\omega)\cdot\epsilon_{\theta}({\bm{z}}_{t},t,\varnothing)$
(14)

where $\varnothing$ = ("") is the embedding of a null text. This further leads
to another perturbation from ${\bm{z}}^{{}^{\prime}}_{t}$ to ${\bm{z}}^{"}$
due to the destruction of the DDIM process and causes error augmentation as
demonstrated in Figure 1.

##  Appendix B Benchmark construction

###  B.1 General information

Here are details about our ZoME-Bench dataset (Zero-shot Music Editing
Benchmark).This dataset contains 1,000 audio samples , which are selected from
MusicCaps , and each is 10 seconds long with a sample rate of 16k.

We refactored their captions to express an edit and divide them into 10 parts
, each representing a type of editing. Sample and details are shown in the
following table

Editing type id | Editing type | size |  origin prompt |  editing prompt |  editing instruction
---|---|---|---|---|---
0 | change instrument | 131 |  ambient acoustic [guitar] music |  ambient acoustic [violin] music |  change the instrument from guitar to violin
1 | add instrument | 139 |  metal audio with a distortion guitar [and drums] |  metal audio with a distortion guitar |  add drums to the piece
2 | delete instrument | 133 |  an eerie tense instrumental featuring electronic drums [and synth keyboard] |  an eerie tense instrumental featuring electronic drums |  remove the synth keyboard
3 | change genre | 134 |  a recording of a solo electric guitar playing [blues] licks |  a recording of a solo electric guitar playing [rocks] licks |  change the genre from blues to rock
4 | change mood | 100 |  a recording featuring electric bass with an [upbeat] vibe |  a recording featuring electric bass with an [melancholic] vibe |  turn upbeat mood into melancholic mood
5 | change rhythm | 69 |  a live ukulele performance featuring [fast] strumming and emotional melodies |  a live ukulele performance featuring [slow] strumming and emotional melodies |  change fast rhythm into slow one
6 | change background | 95 |  female voices in unison with [acoustic] guitar |  female voices in unison with [electric] guitar |  switch acoustic guitar to electric guitar
7 | change melody | 121 |  this instrumental song features a [relaxing] melody |  this instrumental song features a [cheerful] melody |  change relaxing melody into cheerful melody
8 | extract instrument | 111 |  a reggae rhythm recording with bongos [djembe congas acoustic drums and electric guitar] |  a reggae rhythm recording with bongos |  extract bongos from the recording
9 | random | 67 |  / |  / |  /
Table 4: Information of ZoME-Bench dataset

###  B.2 Annotation Process

We rebuild our caption from captions for Musiccaps offered by (Agostinelli et
al., 2023). With the help of ChatGPT-4 (OpenAI, 2023), we rebuild the caption
with prompt as follows(take type "change melody" as examples):

"There is a description of a Piece of music ,Please judge whether the
description has information of melody.If not , just answer "Flase" ,else
change its melody properly into the opposite one ,just change the adjective
and don’t replace any instrument! "blended_word" is [origin melody, changed
melody] "emphasize" is [changed melody] "blended_word" and "emphasize" are
tuples.
Question:(A mellow, passionate melody from a noisy electric guitar)
Answer:("original_prompt": "A mellow, [passionate] melody from a noisy
electric guitar", "editing_prompt": "A mellow, [soft] melody from a noisy
electric guitar", "blended_word": ["passionate melody","soft melody"],
"emphasize" :["soft melody"]) Question:(A recording of solo harp music with a
dreamy, relaxing melody.)
Answer:("original_prompt": "A recording of solo harp music with a dreamy,
[relaxing] melody.", "editing_prompt": "A recording of solo harp music with a
dreamy, [nervous] melody.", "blended_word": ["relaxing melody","nervous
melody"], "emphasize" :["nervous melody"])
Question:("A vintage, emotional song with mellow harmonized flute melody and
soft wooden percussions")
Answer:("original_prompt": "A vintage, emotional song with [passionate] flute
melody and soft wooden percussions.", "editing_prompt": "A vintage, emotional
song with [harmonized] flute melody and soft wooden percussions.",
"blended_word": ["harmonized flute melody","passionate flute melody"]),
"emphasize" :["passionate flute melody"])
now we have Question({origin caption})
Answer(?)"

In the same way , instructions were appended by prompt as follows(take type
"change melody" as examples):

"There is two description of different pieces of music divided by &,Please
describe the difference you need to give me the results in the following
format: Question: this instrumental song features a [relaxing] melody with a
country feel accompanied by a guitar piano simple percussion and bass in a
slow tempo & this instrumental song features a [cheerful] melody with a
country feel accompanied by a guitar piano simple percussion and bass in a
slow tempo
Answer: change relaxing melody into cheerful melody
Question:this song features acapella harmonies with a [high pitched] melody
complemented by both high pitched female whistle tones and male low pitch
tones & this song features acapella harmonies with a [smooth] melody
complemented by both high pitched female whistle tones and male low pitch
tones
Answer:turn a high pitched melody into smooth melody
Question: a traditional and hopeful song with a harmonizing throaty male vocal
and [dissonant] background melody from strings albeit presented in low quality
& a traditional and hopeful song with a harmonizing throaty male vocal and
[harmonic] background melody from strings albeit presented in low quality
Answer: change dissonant melody into harmonic melody Question: [’original
prompt’] & [’editing prompt’]
Answer(?)"

Through this method accmopnied with rounds of manual review ,we guarantee the
quality of this Benchmark.

###  B.3 Data Format

Taking the first piece as example , we express our data in json format with
six keys

⬇

{

"000000000000": {

"editing_prompt": "a live recording of ambient acoustic

[violin] music",

"original_prompt": "a live recording of ambient acoustic

[guitar] music",

"blended_word": "(\"guitar\", \"violin\")",

"emphasize": "(\"violin\")",

"audio_path": "wavs/MusicCaps_-4SYC2YgzL8.wav",

"editing_type_id": "0",

"editing_instruction": "change the instrument from guitar

to violin"

}

}

"Editing_prompt" is the edited caption while "Original_prompt" is the origin
one . "Blended_word" is the subject to be edited."Emphasize" is the word which
should be emphasized. "editing_instruction" is the description of
editing.Something else that needs attention is in the editing type "delete
instrument" , we create another key "neg_prompt" , which helps to reduce the
possibility of deleted instruments still appearing.

##  Appendix C Implementation Details

For our evaluation, we utilize the pre-trained AudioLDM 2-Music model (Liu et
al., 2023b). Our assessment employs a comprehensive set of metrics, namely
CLAP, LPAPS, Structure Distance, and FAD. These metrics are calculated using
the CLAP models available in the AudioLDM_eval package, which is accessible at
https://github.com/haoheliu/audioldm_eval. In line with the methodology
described by Manor & Michaeli (2024), we apply a forward guidance of 3 and a
reverse guidance scale of 12 for DDPM inversion. For the DDIM inversion, the
guidance scale is set to 5, while for SDEdit, we employ a guidance scale of
12. The forward guidance of DIC is 1 while the reverse scale is 5. We chose
these values by exploration about different guidance scale, as discussed in
Appendix D. We conduct all experiments in NVIDIA 4090. For our evaluation, we
have selected the public pre-trained AudioLDM 2-Music model (Liu et al.,
2023b). To ensure a thorough and multidimensional assessment, we measure
performance using a suite of metrics that includes CLAP, LPAPS, Structure
Distance, and FAD, and conduct on an NVIDIA 4090. The computation of these
metrics is facilitated by the CLAP models provided within the AudioLDM_eval
package, which is publicly available at
https://github.com/haoheliu/audioldm_eval.

Our methodology is aligned with the protocol established by Manor & Michaeli
(2024), where we have adopted a forward guidance scale of 3 and a reverse
guidance scale of 12 for DDPM inversion. In contrast, the DDIM inversion
employs a guidance scale of 5, and SDEdit utilizes a guidance scale of 12. For
Disentangled Inversion Control, we have determined the forward guidance to be
1 and the reverse scale to be 5. These specific guidance scale values are
selected after extensive experimental exploration, the details of which are
discussed in Appendix D.

###  C.1 Metrics

There are details about five metrics to evaluate the performance of our novel
Disentangled Inversion Control framework: (1) Struture Distance (Ju et al.,
2023): Leveraging self-similarity of audio features to measure the structure
distance between the source and edited audio. (2) CLAP Score (Elizalde et al.,
): This criterion evaluates the degree to which the output conforms to the
specified target prompt. (3) LPAPS (Iashin & Rahtu, 2021; Paissan et al.,
2023): An audio adaptation of the Learned Perceptual Image Patch Similarity
(LPIPS) (Zhang et al., 2018), this measure evaluates the consistency of the
edited audio with the original audio. (4) MSE (Mean Squared Error): This
distance metric quantifies the variance between the source and the edited
audio. (5) FAD (Fréchet Audio Distance) (Kilgour et al., 2018): Analogous to
the FID used in image analysis, this metric calculates the distance between
two distributions of audio signals.

##  Appendix D Quantitative Results

For the lack of systematic experiments which determines the optimal
combination of guidance scales for achieving the best editing performance, and
analysis on how these guidance scales affect final consequence in both
reconstruction and editing, we conduct this experiment to find the best
scales.

Guidance Scale | Structure | Background Preservation | CLIP Similariy
---|---|---|---
Inverse | Forward | $\textbf{Distance}_{\times 10^{3}}\downarrow$ | LPAPS$\downarrow$ | FAD$\downarrow$ | $\text{MSE}_{\times 10^{5}}\downarrow$ |  CLAP Score $\uparrow$
1 | 1 | 8.56 | 0.12 | 1.17 | 3.25 | 0.51
1 | 2.5 | 11.97 | 0.15 | 2.49 | 4.54 | 0.56
1 | 5 | 15.99 | 0.17 | 4.22 | 6.07 | 0.60
1 | 7.5 | 15.99 | 0.17 | 4.22 | 6.07 | 0.59
2.5 | 1 | 22.80 | 0.20 | 6.39 | 8.65 | 0.30
2.5 | 2.5 | 14.24 | 0.16 | 2.50 | 5.40 | 0.46
2.5 | 5 | 14.46 | 0.16 | 3.31 | 5.49 | 0.53
2.5 | 7.5 | 15.51 | 0.17 | 3.94 | 5.89 | 0.53
5 | 1 | 29.94 | 0.24 | 9.81 | 11.36 | 0.20
5 | 2.5 | 29.16 | 0.24 | 9.11 | 11.07 | 0.22
5 | 5 | 22.15 | 0.20 | 5.59 | 8.40 | 0.36
5 | 7.5 | 17.57 | 0.18 | 5.57 | 6.67 | 0.48
7.5 | 1 | 31.41 | 0.25 | 10.62 | 11.92 | 0.20
7.5 | 2.5 | 31.05 | 0.25 | 10.14 | 11.78 | 0.20
7.5 | 5 | 29.20 | 0.24 | 9.32 | 11.08 | 0.24
7.5 | 7.5 | 24.16 | 0.22 | 7.33 | 9.17 | 0.34
Table 5: Ablation Studies on Different Guidance Scale

##  Appendix E Potential Negative Societal Impacts

MEDIC may also lead to potential negative societal impacts that are worthy of
consideration. If the data sample of the training model is not diverse enough
or biased, the AI-generated music may be overly biased toward one style or
element, limiting the diversity of the music and causing discrimination.MEDIC
could be used to create fake audio content, such as faking someone’s voice or
creating fake musical compositions, posing the risk of fraud and
impersonation.Hopefully , all these issues could be taken into consideration
when taking the model for real use to avoid ethical issues.

##  Appendix F Limitations

In spite of remarkable outcome of our method , but due to the limitation of
the generation model we used , we are incapable of instigating a profound
change.

Though our method is faster than est-optimization and end-to-end diffusion
models,but due to the numerous steps it requires (T=200), the duration of
computing distance is quite long.Thus , we will implement a more powerful
text-to-music generation model to support better editing, while trying to use
consistency model or flow matching model to achieve high quality and fast
music generation in future work

##  Appendix G Qualitative Results

For each type in ZoME-Bench , We provide samples to observe the capability of
MEDIC intuitively.

###  G.1 Change Instrument

In the following figures , we show the capability of MEDIC to change the
instrument.Here we edit the ground truth music piece with original prompt "a
live recording of ambient acoustic [guitar] music" and editing prompt "a live
recording of ambient acoustic [violin] music". The difference of instruments
can be observed in the Mel-spectrum

source: a live recording of ambient acoustic [guitar] music

edit: a live recording of ambient acoustic [violin] music

Figure 5: Editing Type 0 :Change Instrument

###  G.2 Add Instrument

In the following figures , we show the capability of MEDIC to add more
instrument.Here we edit the ground truth music piece with original prompt "a
heavy metal instructional audio with a distortion guitar" and editing prompt
"a heavy metal instructional audio with a distortion guitar [and drums]". The
appearance of new instrument can be observed in the Mel-spectrum which present
drum sound of high frequency.

source: a heavy metal instructional audio with a distortion guitar

edit: a heavy metal instructional audio with a distortion guitar [and drums]

Figure 6: Editing Type 1 :Add Instrument

###  G.3 Delete Instrument

In the following figures , we show the capability of MEDIC to delete
instrument.Here we edit the ground truth music piece with original prompt "a
lively ska instrumental featuring keyboard trumpets bass [and percussion] with
a groovy mood" and editing prompt "a lively ska instrumental featuring
keyboard trumpets and bass with a groovy mood". The vanishing of instrument
can be observed in the Mel-spectrum

source: a lively ska instrumental featuring keyboard trumpets bass [and
percussion] with a groovy mood

edit: a lively ska instrumental featuring keyboard trumpets and bass with a
groovy mood

Figure 7: Editing Type 2 :Delete Instrument

###  G.4 Change Genre

In the following figures , we show the capability of MEDIC to change genre of
a music piece. Here we edit the ground truth music piece with original prompt
"a recording of a solo electric guitar playing [blues] licks" and editing
prompt "a recording of a solo electric guitar playing [rock] licks". The
obvious difference of genre can be observed in the Mel-spectrum

source: a recording of a solo electric guitar playing [blues] licks

edit: a recording of a solo electric guitar playing [rock] licks

Figure 8: Editing Type 3 :Change Genre

###  G.5 Change Mood

Mood is an important attribute of a music. In the following figures ,we show
the capability of MEDIC to change mood of a music piece. Here we edit the
ground truth music piece with original prompt "a recording of [aggressive]
electronic and video game music with synthesizer arrangements"and editing
prompt "a recording of [peaceful] electronic and video game music with
synthesizer arrangements". The change of Mood can be observed in the Mel-
spectrum

source: a recording of [aggressive] electronic and video game music with
synthesizer arrangements

edit: a recording of [peaceful] electronic and video game music with
synthesizer arrangements

Figure 9: Editing Type 4 :Change Mood

###  G.6 Change Rhythm

Rhythm represents the speed of the music. In the following figures ,we show
the capability of MEDIC to change Rhythm of a music piece. Here we edit the
ground truth music piece with original prompt "a [slow] tempo ukelele tuning
recording with static"and editing prompt "a [fast] tempo ukelele tuning
recording with static". The change of Rhythm can be observed in the Mel-
spectrum.The edited Mel-spectrum is much intensive.

source: a [slow] tempo ukelele tuning recording with static

edit: a [fast] tempo ukelele tuning recording with static

Figure 10: Editing Type 5 :Change Rhythm

###  G.7 Change Background

In the following figures ,we show the capability of MEDIC to change background
of instrument of a music piece. Here we edit the ground truth music piece with
original prompt "an amateur ukulele recording with a [medium to uptempo]
pace"and editing prompt "an amateur ukulele recording with a [steady and
rhythmic] pace". The change of instrument background can be observed in the
Mel-spectrum.

source: an amateur ukulele recording with a [medium to uptempo] pace"and
editing prompt

edit: an amateur ukulele recording with a [steady and rhythmic] pace

Figure 11: Editing Type 6 :Change Background

###  G.8 Change Melody

In the following figures ,we show the capability of MEDIC to change melody of
a music piece. Here we edit the ground truth music piece with original prompt
"this instrumental song features a [relaxing] melody with a country feel
accompanied by a guitar piano simple percussion and bass in a slow tempo"and
editing prompt "this instrumental song features a [cheerful] melody with a
country feel accompanied by a guitar piano simple percussion and bass in a
slow tempo". The change of Melody can be observed in the Mel-spectrum

source: this instrumental song features a [relaxing] melody with a country
feel accompanied by a guitar piano simple percussion and bass in a slow tempo

edit: this instrumental song features a [cheerful] melody with a country feel
accompanied by a guitar piano simple percussion and bass in a slow tempo

Figure 12: Editing Type 7 :Change Melody

###  G.9 Extract Instrument

In the following figures ,we show the capability of MEDIC to extract one
certain instrument of a music piece. Here we edit the ground truth music piece
with original prompt "a reggae rhythm recording with bongos [djembe congas
acoustic drums and electric guitar]"and editing prompt "a reggae rhythm
recording with bongos". The change of instruments can be observed in the Mel-
spectrum. Much sounds vanished.

source: this instrumental song features a [relaxing] melody with a country
feel accompanied by a guitar piano simple percussion and bass in a slow tempo

edit: this instrumental song features a [cheerful] melody with a country feel
accompanied by a guitar piano simple percussion and bass in a slow tempo

Figure 13: Editing Type 8 :Extract Instrument

##  Appendix H safeguards

In the processing of the data and models involved in this study, we fully
considered the potential risks. We ensure that all data sources are rigorously
screened and vetted,and the model we used is absolutely trained from the safe
dataset to minimize security risks of being misused.

## NeurIPS Paper Checklist

  1. 1.

Claims

  2. Question: Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?

  3. Answer: [Yes]

  4. Justification: We accurately reflect the paper’s contributions and scope in the abstract and section 1 introduction.

  5. Guidelines:

     * •

The answer NA means that the abstract and introduction do not include the
claims made in the paper.

     * •

The abstract and/or introduction should clearly state the claims made,
including the contributions made in the paper and important assumptions and
limitations. A No or NA answer to this question will not be perceived well by
the reviewers.

     * •

The claims made should match theoretical and experimental results, and reflect
how much the results can be expected to generalize to other settings.

     * •

It is fine to include aspirational goals as motivation as long as it is clear
that these goals are not attained by the paper.

  6. 2.

Limitations

  7. Question: Does the paper discuss the limitations of the work performed by the authors?

  8. Answer: [Yes]

  9. Justification: We discuss the limitations in section F limitations

  10. Guidelines:

     * •

The answer NA means that the paper has no limitation while the answer No means
that the paper has limitations, but those are not discussed in the paper.

     * •

The authors are encouraged to create a separate "Limitations" section in their
paper.

     * •

The paper should point out any strong assumptions and how robust the results
are to violations of these assumptions (e.g., independence assumptions,
noiseless settings, model well-specification, asymptotic approximations only
holding locally). The authors should reflect on how these assumptions might be
violated in practice and what the implications would be.

     * •

The authors should reflect on the scope of the claims made, e.g., if the
approach was only tested on a few datasets or with a few runs. In general,
empirical results often depend on implicit assumptions, which should be
articulated.

     * •

The authors should reflect on the factors that influence the performance of
the approach. For example, a facial recognition algorithm may perform poorly
when image resolution is low or images are taken in low lighting. Or a speech-
to-text system might not be used reliably to provide closed captions for
online lectures because it fails to handle technical jargon.

     * •

The authors should discuss the computational efficiency of the proposed
algorithms and how they scale with dataset size.

     * •

If applicable, the authors should discuss possible limitations of their
approach to address problems of privacy and fairness.

     * •

While the authors might fear that complete honesty about limitations might be
used by reviewers as grounds for rejection, a worse outcome might be that
reviewers discover limitations that aren’t acknowledged in the paper. The
authors should use their best judgment and recognize that individual actions
in favor of transparency play an important role in developing norms that
preserve the integrity of the community. Reviewers will be specifically
instructed to not penalize honesty concerning limitations.

  11. 3.

Theory Assumptions and Proofs

  12. Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

  13. Answer: [Yes]

  14. Justification: We provide theory assumptions and proofs in section 2 preliminaries and analyses

  15. Guidelines:

     * •

The answer NA means that the paper does not include theoretical results.

     * •

All the theorems, formulas, and proofs in the paper should be numbered and
cross-referenced.

     * •

All assumptions should be clearly stated or referenced in the statement of any
theorems.

     * •

The proofs can either appear in the main paper or the supplemental material,
but if they appear in the supplemental material, the authors are encouraged to
provide a short proof sketch to provide intuition.

     * •

Inversely, any informal proof provided in the core of the paper should be
complemented by formal proofs provided in appendix or supplemental material.

     * •

Theorems and Lemmas that the proof relies upon should be properly referenced.

  16. 4.

Experimental Result Reproducibility

  17. Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

  18. Answer: [Yes]

  19. Justification: we will open-source our codes(contains model or their access method) and the benchmark we constructed, which is enough to reproduce our experiments.

  20. Guidelines:

     * •

The answer NA means that the paper does not include experiments.

     * •

If the paper includes experiments, a No answer to this question will not be
perceived well by the reviewers: Making the paper reproducible is important,
regardless of whether the code and data are provided or not.

     * •

If the contribution is a dataset and/or model, the authors should describe the
steps taken to make their results reproducible or verifiable.

     * •

Depending on the contribution, reproducibility can be accomplished in various
ways. For example, if the contribution is a novel architecture, describing the
architecture fully might suffice, or if the contribution is a specific model
and empirical evaluation, it may be necessary to either make it possible for
others to replicate the model with the same dataset, or provide access to the
model. In general. releasing code and data is often one good way to accomplish
this, but reproducibility can also be provided via detailed instructions for
how to replicate the results, access to a hosted model (e.g., in the case of a
large language model), releasing of a model checkpoint, or other means that
are appropriate to the research performed.

     * •

While NeurIPS does not require releasing code, the conference does require all
submissions to provide some reasonable avenue for reproducibility, which may
depend on the nature of the contribution. For example

       1. (a)

If the contribution is primarily a new algorithm, the paper should make it
clear how to reproduce that algorithm.

       2. (b)

If the contribution is primarily a new model architecture, the paper should
describe the architecture clearly and fully.

       3. (c)

If the contribution is a new model (e.g., a large language model), then there
should either be a way to access this model for reproducing the results or a
way to reproduce the model (e.g., with an open-source dataset or instructions
for how to construct the dataset).

       4. (d)

We recognize that reproducibility may be tricky in some cases, in which case
authors are welcome to describe the particular way they provide for
reproducibility. In the case of closed-source models, it may be that access to
the model is limited in some way (e.g., to registered users), but it should be
possible for other researchers to have some path to reproducing or verifying
the results.

  21. 5.

Open access to data and code

  22. Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

  23. Answer: [Yes]

  24. Justification: We provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results.

  25. Guidelines:

     * •

The answer NA means that paper does not include experiments requiring code.

     * •

Please see the NeurIPS code and data submission guidelines
(https://nips.cc/public/guides/CodeSubmissionPolicy) for more details.

     * •

While we encourage the release of code and data, we understand that this might
not be possible, so “No” is an acceptable answer. Papers cannot be rejected
simply for not including code, unless this is central to the contribution
(e.g., for a new open-source benchmark).

     * •

The instructions should contain the exact command and environment needed to
run to reproduce the results. See the NeurIPS code and data submission
guidelines (https://nips.cc/public/guides/CodeSubmissionPolicy) for more
details.

     * •

The authors should provide instructions on data access and preparation,
including how to access the raw data, preprocessed data, intermediate data,
and generated data, etc.

     * •

The authors should provide scripts to reproduce all experimental results for
the new proposed method and baselines. If only a subset of experiments are
reproducible, they should state which ones are omitted from the script and
why.

     * •

At submission time, to preserve anonymity, the authors should release
anonymized versions (if applicable).

     * •

Providing as much information as possible in supplemental material (appended
to the paper) is recommended, but including URLs to data and code is
permitted.

  26. 6.

Experimental Setting/Details

  27. Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

  28. Answer: [Yes]

  29. Justification: we specify all the training and test details in section 5 Experiments and section C Implementation Details

  30. Guidelines:

     * •

The answer NA means that the paper does not include experiments.

     * •

The experimental setting should be presented in the core of the paper to a
level of detail that is necessary to appreciate the results and make sense of
them.

     * •

The full details can be provided either with the code, in appendix, or as
supplemental material.

  31. 7.

Experiment Statistical Significance

  32. Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

  33. Answer: [No]

  34. Justification: We only use one random seed.

  35. Guidelines:

     * •

The answer NA means that the paper does not include experiments.

     * •

The authors should answer "Yes" if the results are accompanied by error bars,
confidence intervals, or statistical significance tests, at least for the
experiments that support the main claims of the paper.

     * •

The factors of variability that the error bars are capturing should be clearly
stated (for example, train/test split, initialization, random drawing of some
parameter, or overall run with given experimental conditions).

     * •

The method for calculating the error bars should be explained (closed form
formula, call to a library function, bootstrap, etc.)

     * •

The assumptions made should be given (e.g., Normally distributed errors).

     * •

It should be clear whether the error bar is the standard deviation or the
standard error of the mean.

     * •

It is OK to report 1-sigma error bars, but one should state it. The authors
should preferably report a 2-sigma error bar than state that they have a 96%
CI, if the hypothesis of Normality of errors is not verified.

     * •

For asymmetric distributions, the authors should be careful not to show in
tables or figures symmetric error bars that would yield results that are out
of range (e.g. negative error rates).

     * •

If error bars are reported in tables or plots, The authors should explain in
the text how they were calculated and reference the corresponding figures or
tables in the text.

  36. 8.

Experiments Compute Resources

  37. Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

  38. Answer: [Yes]

  39. Justification: We provide sufficient information on the computer resources in section 5.1 Implementation Details

  40. Guidelines:

     * •

The answer NA means that the paper does not include experiments.

     * •

The paper should indicate the type of compute workers CPU or GPU, internal
cluster, or cloud provider, including relevant memory and storage.

     * •

The paper should provide the amount of compute required for each of the
individual experimental runs as well as estimate the total compute.

     * •

The paper should disclose whether the full research project required more
compute than the experiments reported in the paper (e.g., preliminary or
failed experiments that didn’t make it into the paper).

  41. 9.

Code Of Ethics

  42. Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines?

  43. Answer: [Yes]

  44. Justification: The research conducted in the paper conforms, in every respect, with the NeurIPS Code of Ethics

  45. Guidelines:

     * •

The answer NA means that the authors have not reviewed the NeurIPS Code of
Ethics.

     * •

If the authors answer No, they should explain the special circumstances that
require a deviation from the Code of Ethics.

     * •

The authors should make sure to preserve anonymity (e.g., if there is a
special consideration due to laws or regulations in their jurisdiction).

  46. 10.

Broader Impacts

  47. Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

  48. Answer: [Yes]

  49. Justification: Positive impacts is in section 6 Conclusion.Negative impacts in section E Potential Negative Societal Impacts

  50. Guidelines:

     * •

The answer NA means that there is no societal impact of the work performed.

     * •

If the authors answer NA or No, they should explain why their work has no
societal impact or why the paper does not address societal impact.

     * •

Examples of negative societal impacts include potential malicious or
unintended uses (e.g., disinformation, generating fake profiles,
surveillance), fairness considerations (e.g., deployment of technologies that
could make decisions that unfairly impact specific groups), privacy
considerations, and security considerations.

     * •

The conference expects that many papers will be foundational research and not
tied to particular applications, let alone deployments. However, if there is a
direct path to any negative applications, the authors should point it out. For
example, it is legitimate to point out that an improvement in the quality of
generative models could be used to generate deepfakes for disinformation. On
the other hand, it is not needed to point out that a generic algorithm for
optimizing neural networks could enable people to train models that generate
Deepfakes faster.

     * •

The authors should consider possible harms that could arise when the
technology is being used as intended and functioning correctly, harms that
could arise when the technology is being used as intended but gives incorrect
results, and harms following from (intentional or unintentional) misuse of the
technology.

     * •

If there are negative societal impacts, the authors could also discuss
possible mitigation strategies (e.g., gated release of models, providing
defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms
to monitor how a system learns from feedback over time, improving the
efficiency and accessibility of ML).

  51. 11.

Safeguards

  52. Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

  53. Answer: [Yes]

  54. Justification: We take actions for safeguards , which is described in section H Safeguards

  55. Guidelines:

     * •

The answer NA means that the paper poses no such risks.

     * •

Released models that have a high risk for misuse or dual-use should be
released with necessary safeguards to allow for controlled use of the model,
for example by requiring that users adhere to usage guidelines or restrictions
to access the model or implementing safety filters.

     * •

Datasets that have been scraped from the Internet could pose safety risks. The
authors should describe how they avoided releasing unsafe images.

     * •

We recognize that providing effective safeguards is challenging, and many
papers do not require this, but we encourage authors to take this into account
and make a best faith effort.

  56. 12.

Licenses for existing assets

  57. Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

  58. Answer: [Yes]

  59. Justification: Creators or original owners of assets, used in the paper, are properly credited and the license and terms of use explicitly mentioned and are properly respected.

  60. Guidelines:

     * •

The answer NA means that the paper does not use existing assets.

     * •

The authors should cite the original paper that produced the code package or
dataset.

     * •

The authors should state which version of the asset is used and, if possible,
include a URL.

     * •

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

     * •

For scraped data from a particular source (e.g., website), the copyright and
terms of service of that source should be provided.

     * •

If assets are released, the license, copyright information, and terms of use
in the package should be provided. For popular datasets,
paperswithcode.com/datasets has curated licenses for some datasets. Their
licensing guide can help determine the license of a dataset.

     * •

For existing datasets that are re-packaged, both the original license and the
license of the derived asset (if it has changed) should be provided.

     * •

If this information is not available online, the authors are encouraged to
reach out to the asset’s creators.

  61. 13.

New Assets

  62. Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

  63. Answer: [Yes]

  64. Justification: Our assets is well documented and the document is provided

  65. Guidelines:

     * •

The answer NA means that the paper does not release new assets.

     * •

Researchers should communicate the details of the dataset/code/model as part
of their submissions via structured templates. This includes details about
training, license, limitations, etc.

     * •

The paper should discuss whether and how consent was obtained from people
whose asset is used.

     * •

At submission time, remember to anonymize your assets (if applicable). You can
either create an anonymized URL or include an anonymized zip file.

  66. 14.

Crowdsourcing and Research with Human Subjects

  67. Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

  68. Answer: [N/A]

  69. Justification: Our work do not involve crowdsourcing nor research with human subjects.

  70. Guidelines:

     * •

The answer NA means that the paper does not involve crowdsourcing nor research
with human subjects.

     * •

Including this information in the supplemental material is fine, but if the
main contribution of the paper involves human subjects, then as much detail as
possible should be included in the main paper.

     * •

According to the NeurIPS Code of Ethics, workers involved in data collection,
curation, or other labor should be paid at least the minimum wage in the
country of the data collector.

  71. 15.

Institutional Review Board (IRB) Approvals or Equivalent for Research with
Human Subjects

  72. Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

  73. Answer: [N/A]

  74. Justification: Our work do not involve crowdsourcing nor research with human subjects.

  75. Guidelines:

     * •

The answer NA means that the paper does not involve crowdsourcing nor research
with human subjects.

     * •

Depending on the country in which research is conducted, IRB approval (or
equivalent) may be required for any human subjects research. If you obtained
IRB approval, you should clearly state this in the paper.

     * •

We recognize that the procedures for this may vary significantly between
institutions and locations, and we expect authors to adhere to the NeurIPS
Code of Ethics and the guidelines for their institution.

     * •

For initial submissions, do not include any information that would break
anonymity (if applicable), such as the institution conducting the review.