# Instruct-MusicGen: Unlocking Text-to-Music Editing for Music Language Models
via Instruction Tuning

Yixiao Zhang1, Yukara Ikemiya2, Woosung Choi2, Naoki Murata2, Marco A.
Martínez-Ramírez2,
Liwei Lin3, Gus Xia3, Wei-Hsiang Liao2, Yuki Mitsufuji2, Simon Dixon1
1C4DM, Queen Mary University of London, 2Sony AI, 3Music X Lab, MBZUAI

###### Abstract

Recent advances in text-to-music editing, which employ text queries to modify
music (e.g. by changing its style or adjusting instrumental components),
present unique challenges and opportunities for AI-assisted music creation.
Previous approaches in this domain have been constrained by the necessity to
train specific editing models from scratch, which is both resource-intensive
and inefficient; other research uses large language models to predict edited
music, resulting in imprecise audio reconstruction. To Combine the strengths
and address these limitations, we introduce Instruct-MusicGen, a novel
approach that finetunes a pretrained MusicGen model to efficiently follow
editing instructions such as adding, removing, or separating stems. Our
approach involves a modification of the original MusicGen architecture by
incorporating a text fusion module and an audio fusion module, which allow the
model to process instruction texts and audio inputs concurrently and yield the
desired edited music. Remarkably, Instruct-MusicGen only introduces $\sim$8%
new parameters to the original MusicGen model and only trains for 5K steps,
yet it achieves superior performance across all tasks compared to existing
baselines, and demonstrates performance comparable to the models trained for
specific tasks. This advancement not only enhances the efficiency of text-to-
music editing but also broadens the applicability of music language models in
dynamic music production environments. 111Work done during Yixiao’s internship
at Sony AI.222Code, model weights and demo are available at:
https://bit.ly/instruct-musicgen.

##  1 Introduction

The rapid advances in text-to-music generation have opened up new
possibilities for AI-assisted music creation (Agostinelli et al., 2023; Copet
et al., 2023; Li et al., 2023; Liu et al., 2023a; Chen et al., 2023). This
paradigm shift has also sparked a growing interest in developing models that
offer greater controllability (Lin et al., 2023; Wu et al., 2023a; Melechovsky
et al., 2023; Lin et al., 2024) and editability (Han et al., 2023; Zhang et
al., 2024; Hussain et al., 2023) over the music generation process. In music
production, a stem—a mixed group of tracks often related by instrument type
(like drums or lead vocals)—is essential for mixing and mastering because it
allows producers to isolate, adjust, and manipulate individual elements of a
song. Following the definition in Zhang et al. (2024), "text-to-music editing"
involves using textual queries to modify various aspects of a music recording,
which can be categorised into two main types: intra-stem editing, which
focuses on modifying a single stem (e.g., changing the instrument, timbre, or
performance style), and inter-stem editing, which involves altering the
relationships among stems (e.g., adding, removing, or separating stems). Our
work mainly focuses on the problem of inter-stem editing.

Previous attempts to develop text-based music editing models have encountered
several challenges. Some approaches (Wang et al., 2023; Han et al., 2023) have
focused on training specialized editing models from scratch, which is
resource-intensive and may not yield results comparable to state-of-the-art
music generation models. Other work (Zhang et al., 2023b; Hussain et al.,
2023; Yang et al., 2023) has sought to leverage existing language models
(LLMs) and MusicGen (Copet et al., 2023), allowing the LLM to interpret
editing instructions without further training the music model. Although this
approach offers flexibility, it often lacks the ability to precisely
reconstruct the conditional audio, leading to unreliable results. To address
these limitations, an ideal solution should harness the knowledge embedded in
pretrained models to ensure high-quality audio output while adapting the
architecture to accommodate the specific requirements of music editing tasks.

Figure 1: Comparison between MusicGen and instruct-MusicGen. Instruct-MusicGen
accepts both audio input and editing instruction text as conditions.

In this paper, we introduce Instruct-MusicGen, a novel approach that applies
an instruction-following tuning strategy to the pretrained MusicGen model,
enhancing its ability to follow editing instructions effectively without
finetuning all its parameters. As shown in Figure 1, by incorporating an audio
fusion module based on LLaMA-adapter (Zhang et al., 2023a; Lin et al., 2023)
and a text fusion module based on LoRA (Hu et al., 2021) into the original
MusicGen architecture, we allow the model to process both precise audio
conditions and text-based instructions simultaneously, while the original
MusicGen does not. This enables Instruct-MusicGen to perform a wide range of
editing tasks, such as adding, separating, and removing stems. To train
Instruct-MusicGen, we synthesize an instructional dataset using the Slakh2100
dataset (Manilow et al., 2019), introducing only 8% additional parameters
compared to the original model, and finetune the model for only 5K steps,
which is less than 1% of training a music editing model from scratch.

We evaluate Instruct-MusicGen on two datasets: the Slakh test set and the out-
of-domain MoisesDB dataset (Pereira et al., 2023). Our model outperforms
existing baselines and achieves performance comparable to models specifically
trained for individual tasks. This demonstrates the effectiveness of our
approach in leveraging pretrained models for text-to-music editing while
maintaining high-quality results. The main contributions of this work are
threefold:

  1. 1.

We propose Instruct-MusicGen, a novel approach that finetunes pretrained large
music language models to follow editing instructions effectively. This method
addresses the limitations of previous approaches by leveraging pretrained
models to ensure high-quality audio output and precise reconstruction,
combining both music and text multi-modal control.

  2. 2.

We enhance the capabilities of a pretrained text-to-music model to perform
multiple tasks, including adding, separating, and extracting stems from music
audio, all within a single training process. This approach significantly
reduces the computing resources cost compared to training specialised editing
models from scratch.

  3. 3.

We extend the benchmarking for text-to-music editing tasks, providing a
comprehensive comparison across various models. Our model outperforms existing
baselines and achieves performance comparable to models specifically trained
for individual tasks, demonstrating the effectiveness and versatility of our
approach.

##  2 Related work

Despite the growing body of research on text-to-music generation (Agostinelli
et al., 2023; Copet et al., 2023; Li et al., 2023; Yao et al., 2023;
Melechovsky et al., 2023; Evans et al., 2024a, b; Chen et al., 2023), the
creation of music remains an inherently iterative process, involving multiple
rounds of editing and revision. This ongoing need for refinement and
adjustment underscores the necessity for specialized models that facilitate
and enhance music editing.

Text-based music editing provides a flexible approach for editing music using
textual queries. This method is similar to other modalities that require
editing, such as image (Brooks et al., 2023; Liu et al., 2023b) and video
(Chai et al., 2023; Ceylan et al., 2023) editing. In text-to-music editing,
tasks leverage text to make precise alterations to existing music
compositions. To address these needs in music editing, AUDIT (Wang et al.,
2023) and InstructME (Han et al., 2023) developed a diffusion model trained
with paired music editing data. Additionally, models like M2UGen Hussain et
al. (2023), Loop Copilot (Zhang et al., 2023b), MusicAgent (Yu et al., 2023),
ComposerX (Deng et al., 2024) and WavCraft (Liang et al., 2024) use large
language models (LLMs) for reasoning and regenerate music with external music
generation models. Furthermore, Postolache et al. (2024) attempt to jointly
model multi-stem distribution of music for text-based generation and
separation. Certain models focus exclusively on specific tasks within music
editing, such as conditioned generation (Lin et al., 2023, 2024) and
separation (Liu et al., 2023c), along with intra-stem editing tasks like text-
based timbre transfer and style transfer (Zhang et al., 2024; Manor and
Michaeli, 2024; Li et al., 2024).

The task of inter-stem music editing is closely related to stem-wise music
generation, which is also crucial in the music production process. While these
efforts are not directly tied to text-based controls, some research focuses on
modeling stem-wise representations to enable simultaneous stem generation and
separation. For instance, Jen-1 Composer (Yao et al., 2023) and MSDM (Mariani
et al., 2023) jointly model the distribution of music with four stems using a
diffusion model. The abilities of most existing stem-wise music models are
restricted to a fixed set of 4 stems, which limits flexibility but enhances
controllability; In contrast, StemGen (Parker et al., 2023) introduces a novel
approach by employing a non-autoregressive Transformer architecture for more
flexible stem-wise generation, moving beyond the constraints of a fixed stem
set.

Our work distinguishes itself from these existing efforts in several key ways.
First, rather than developing a new model from scratch or strictly adhering to
a fixed set of stems, we leverage the power of a pretrained music language
model, MusicGen, and enhance it with instruction tuning. This approach not
only reduces the computational cost but also retains the high audio quality of
the original MusicGen model. By integrating both audio and text fusion
modules, our model, Instruct-MusicGen, can precisely interpret and execute
complex editing instructions involving multiple stems. Furthermore, our method
introduces minimal additional parameters and requires significantly less
training, demonstrating a more efficient and scalable solution for text-based
music editing. This dual approach of using pretrained models with specialized
adapters sets our work apart, providing a flexible and powerful tool for the
evolving needs of AI-assisted music creation.

##  3 Method

###  3.1 MusicGen

The original MusicGen consists of three components: (1) the EnCodec (Défossez
et al., 2022) audio encoder and decoder, which compress music audio waveforms
into latent codes and reconstruct them back into waveforms; (2) a multi-layer
transformer architecture that models sequences of latent codes, capturing
higher-level music representations and efficiently modeling internal
relationships within music audio; and (3) the T5 (Raffel et al., 2020) text
encoder, which converts text descriptions into embeddings for text-conditioned
generation.

MusicGen uses EnCodec (Défossez et al., 2022), a convolutional auto-encoder,
to map real audio waveforms to latent codes. EnCodec employs Residual Vector
Quantization (RVQ) (Zeghidour et al., 2022) to compress audio into tokens
using multiple codebooks, where each quantizer encodes the quantization error
from the previous one. For a reference audio $X\in\mathbb{R}^{d\cdot f_{n}}$,
where $d$ is the duration and $f_{n}$ is the sample rate, EnCodec compresses
$X$ into $Q\in\\{1,...,L\\}^{N\times d\cdot f_{s}}$, where $L$ is the RVQ
codebook size, $N$ is the number of codebooks, and $f_{s}$ is the latent code
sample rate ($f_{s}<<f_{n}$). In MusicGen, $N=4$, $f_{n}=50$, $f_{s}=32000$,
and $L=2048$. Finally, The transformer models the sequence relationships over
latent codes.

###  3.2 Instruct-MusicGen

MusicGen is a text-to-music generation model, capable of generating music
audio from a given text prompt. However, MusicGen cannot edit existing music
audio. To address this limitation, we introduce Instruct-MusicGen, which
transforms MusicGen into a model that can follow editing instructions to
modify existing music audio.

Instruct-MusicGen takes a music audio input $X_{\text{cond}}$ and a text
instruction $X_{\text{instruct}}$ (e.g., "Add guitar") as inputs. The model
then edits the music audio $X_{\text{cond}}$ according to the instruction
$X_{\text{instruct}}$ and generates the desired edited music
$X_{\text{music}}$. As illustrated in Figure 2, Instruct-MusicGen incorporates
two additional modules into the vanilla MusicGen: an audio fusion and a text
fusion module.

Figure 2: The architecture of instruct-MusicGen. The audio fusion module
transforms the conditional music audio into embeddings using a duplicated
encoder and integrates these embeddings into the MusicGen decoder. The text
fusion module modifies the cross-attention mechanism to handle text
instructions by finetuning specific layers while keeping the text encoder
parameters frozen.

####  3.2.1 Audio Fusion Module

The audio fusion module enables Instruct-MusicGen to accept external audio
inputs, which is inspired by Zhang et al. (2023a) and Lin et al. (2023).
Figure 2 presents the diagram of the audio fusion module. Initially, we
convert $X_{\text{cond}}$ into EnCodec tokens, followed by re-encoding these
tokens into the embedding $z_{\text{cond}},z_{\text{cond}}$ through the pre-
trained embedding layers of MusicGen. Similarly, we transform
$X_{\text{music}}$ into the pretrained embedding $z_{\text{music}}$.

The module begins with duplicating self-attention modules of the pretrained
MusicGen model to extract latent representations of $z_{\text{cond}}$. Given
that MusicGen consists of $M$ layers, we denote
$Z_{\text{cond}}=\\{z^{0}_{\text{cond}},z_{\text{cond}}^{1},\ldots,z_{\text{cond}}^{M}\\}$
and
$Z_{\text{music}}=\\{z^{0}_{\text{music}},z_{\text{music}}^{1},\ldots,z_{\text{music}}^{M}\\}$
to represent the hidden states of $z_{\text{cond}}$ and $z_{\text{music}}$
respectively. Note that we use a learnable input embedding as
$z^{0}_{\text{cond}}$ and initialize $z^{0}_{\text{music}}$ with
$z_{\text{music}}$.

Let $W_{\rm Q}$, $W_{\rm K}$, $W_{\rm V}$ and $W_{\rm O}$ denote the query,
key, value, and output projection matrices of the pretrained self-attention
transformer layer respectively, then, $z_{\text{music}}$ is projected into the
latent space,

$K=Z_{\text{music}}W_{\text{K}},\hskip
14.22636ptQ=Z_{\text{music}}W_{\text{Q}},\hskip
14.22636ptV=Z_{\text{music}}W_{\text{V}}.$ (1)

The attention output $O$ of $Q$, $K$, and $V$ is computed as follows,

$O={\rm Attention}(Q,K,V)W_{\rm O},$ (2) ${\rm
Attention}(Q,K,V)=\text{Softmax}\left(\frac{QK^{\mathsf{T}}}{\sqrt{d}}\right)V,$
(3)

where $d$ is the dimension of $K$.

For $z_{\text{cond}}$, we design a learnable linear layer
$\text{Linear}_{\text{cond}}$ to incorporate information from
$X_{\text{cond}}$ into MusicGen, and $M$ layer-wise learnable position
embeddings $\\{e_{1},e_{2},....e_{M}\\}$ to enhance the sequital modeling. Let
$Z^{\prime}_{\text{cond}}=\\{z_{\text{cond}}^{1}+\text{Linear}_{\text{cond}}(z_{\text{cond}})+e_{1},z_{\text{cond}}^{2}+\text{Linear}_{\text{cond}}(z_{\text{cond}})+e_{2},\ldots,z_{\text{cond}}^{M}+\text{Linear}_{\text{cond}}(z_{\text{cond}})+e_{M}\\}$,
then, $Z^{\prime}_{\text{cond}}$ is projected into the latent space,

$K^{\prime}=Z^{\prime}_{\text{cond}}W_{\text{K}},\hskip
14.22636ptQ^{\prime}=Z^{\prime}_{\text{cond}}W_{\text{Q}},\hskip
14.22636ptV^{\prime}=Z^{\prime}_{\text{cond}}W_{\text{V}}.$ (4)

We compute the attention output of $Q^{\prime}$, $K^{\prime}$, and
$V^{\prime}$ and pass it through each layer sequentially. Let
${Q^{\prime}}_{m}$, ${K^{\prime}}_{m}$, and ${V^{\prime}}_{m}$ denote $m$-th
layer value in $Q^{\prime}$, $K^{\prime}$, and $V^{\prime}$, which are
computed from $z_{\text{cond}}^{m-1}$, then,

$z_{\text{cond}}^{m}=\text{Attention}({Q^{\prime}}_{m},{K^{\prime}}_{m},{V^{\prime}}_{m})W^{m}_{\text{O}}.$
(5)

To fuse information of $Z_{\text{cond}}$ into $Z_{\text{music}}$, we compute
the cross attention between $Z_{\text{cond}}$ and $Z_{\text{music}}$,

${\rm
CrossAttention}(Q,K^{\prime},V^{\prime})=\text{Softmax}\left(\frac{(Q+Q^{\prime})(K^{\prime})^{\mathsf{T}}}{\sqrt{d}}\right)V^{\prime}.$
(6)

Finally, the attention output of $Z_{\text{music}}$ is updated as follows,

$O^{\prime\prime}={\rm CrossAttention}(Q,K^{\prime},V^{\prime})W_{\rm O},$ (7)
$O_{\text{fuse}}=O+g\cdot O^{\prime\prime},$ (8)

where $g\in\mathbb{R}$ is a zero-initialized learnable gating factor.

Thus, the total trainable parameters in Instruct-MusicGen include the input
embedding $z^{0}_{\rm cond}$, linear layers ${\rm Linear}_{\rm cond}$,
learnable position embeddings $\\{e_{1},e_{2},....e_{M}\\}$, and learnable
gating factors $g$.

####  3.2.2 Text Fusion Module

To replace the text description input with instruction input, we modify the
behavior of the current text encoder. We achieve this by finetuning only the
cross-attention module between the text embedding and the music
representations while keeping the text encoder’s parameters frozen.

First, the text is embedded and encoded by the T5 text encoder as
$z_{\text{instruct}}=\text{T5}(X_{\text{instruct}})$. Let $W^{\prime}_{\rm
Q}$, $W^{\prime}_{\rm K}$, and $W^{\prime}_{\rm V}$ denote the query, key, and
value projection matrices of the pretrained cross-attention transformer layers
respectively. Then, the query, key, and value of $z_{\text{instruct}}$ and
$Z_{\text{music}}$ for the multi-modal cross-attention are calculated as
follows:

$K_{\rm instruct}=z_{\text{instruct}}W^{\prime}_{\rm K},\hskip
14.22636ptQ_{\rm instruct}=Z_{\text{music}}W^{\prime}_{\rm Q},\hskip
14.22636ptV_{\rm instruct}=z_{\text{instruct}}W^{\prime}_{\rm V}.$ (9)

For efficient finetuning of the cross-attention module, we apply LoRA to the
projection layers $W^{\prime}_{Q}$ and $W^{\prime}_{V}$. LoRA decomposes each
of $W^{\prime}_{Q}$ and $W^{\prime}_{V}$ into two smaller low-rank matrices,
significantly reducing the number of parameters requiring fine-tuning. The
$Q_{\rm instruct}$ and $V_{\rm instruct}$ of cross-attention are then revised
as follows:

$Q_{\rm instruct}=z_{\text{instruct}}(W^{\prime}_{\rm Q}+(W_{\rm
Q}^{\text{A}})^{\mathsf{T}}W_{\rm Q}^{\text{B}}),$ (10) $V_{\rm
instruct}=z_{\text{instruct}}(W^{\prime}_{\rm V}+(W_{\rm
V}^{\text{A}})^{\mathsf{T}}W_{\rm V}^{\text{B}}).$ (11)

During fine-tuning, $W^{\prime}_{Q}$, $W^{\prime}_{K}$, and $W^{\prime}_{V}$
are frozen, while $W^{\text{A}}_{Q}$, $W^{\text{B}}_{Q}$, $W^{\text{A}}_{V}$,
and $W^{\text{B}}_{V}$ are trainable.

##  4 Experiments

###  4.1 Dataset

For our objective evaluations, we utilise two distinct datasets, each serving
a specific purpose in assessing both in-domain and out-of-domain performance
capabilities of various models.

  1. 1.

Slakh2100 dataset (Manilow et al., 2019). The Synthesized Lakh (Slakh)
Dataset, originally derived from the Lakh MIDI Dataset v0.1, comprises high-
quality audio tracks synthesised using sample-based virtual instruments. Known
as Slakh2100, this dataset features 2100 tracks complete with corresponding
MIDI files. These tracks are methodically divided into subsets designated for
training (1500 tracks), validation (375 tracks), and testing (225 tracks),
collectively amounting to approximately 145 hours of audio mixtures. This
dataset primarily supports audio source separation tasks within a controlled,
synthesised environment.

  2. 2.

MoisesDB dataset (Pereira et al., 2023). In contrast to the Slakh2100, the
MoisesDB dataset caters to source separation with a broader scope. It includes
240 real audio tracks sourced from 45 diverse artists spanning twelve musical
genres. Uniquely, MoisesDB organises its tracks into a detailed two-level
hierarchical taxonomy of stems, offering a varied number of stems per track,
each annotated with textual descriptions. This setup facilitates a more
comprehensive evaluation across different genres and stem configurations.

The rationale for selecting two datasets lies in their diverse configurations
and common applications. While the Slakh dataset is traditionally utilized for
training models tailored to a four-stem arrangement (e.g., Postolache et al.,
2024), our model, Instruct-MusicGen, although initially trained on this
dataset, is designed to generalise to various stem configurations. Conversely,
models such as InstructME and AUDIT are trained on private or larger, more
diverse datasets. By employing both Slakh2100 and MoisesDB, we ensure a
comprehensive evaluation, allowing us to fairly compare the adaptability and
performance of different models under varying conditions of data familiarity
and complexity.

###  4.2 Data Preprocessing

We utilized the Slakh2100 dataset to construct an instruction-based dataset
for our experiments, employing the following pipeline:

  1. 1.

A data point was randomly selected from the Slakh training dataset.

  2. 2.

An instruction was chosen from a predefined set {add, remove, extract} along
with a target stem. Subsequently, $n$ other stems were selected from the
remaining stems.

  3. 3.

An offset was randomly determined to cut a 5-second audio clip. If the target
stem contained more than 50% silence, a different offset was selected.

  4. 4.

The stems were mixed according to the specified instructions to create a
triplet consisting of {instruction text, condition audio input, audio ground
truth}.

###  4.3 Experimental setup

For the finetuning of MusicGen, we jointly trained the audio fusion module and
the text fusion module. The optimisation process utilised the AdamW optimiser,
with a learning rate set at $5e^{-3}$. We use L2 loss over latent tokens as
the training objective. Training incorporated a Cosine Annealing scheduler
with an initial warmup of 100 steps. The training regimen extended over 5,000
steps with an accumulated batch size of 32, achieved through setting the batch
size to 8 and using gradient accumulation over 4 iterations. The finetuning
process was executed on a single NVIDIA A100 GPU and was completed within a
span of two days.

###  4.4 Baselines

In this section, we explore various baseline models, each distinguished by
their unique methodologies for handling audio data. We categorise these into
three types: text-based music editing models, stem-wise modelling, and
specialised models for single text-based tasks (see Table 1 for a comparison).
Note we use "ME" to distinguish whether a model is a text-to-music editing
model or not.

####  4.4.1 Text-based Music Editing Models

  1. 1.

AUDIT (Wang et al., 2023): AUDIT is an instruction-guided audio editing model,
consisting of a variational autoencoder (VAE) for converting input audio into
a latent space representation, a T5 text encoder for processing edit
instructions, and a diffusion network that performs the actual audio editing
in the latent space. The system accepts mel-spectrograms of input audio and
edit instructions, and generates the edited audio as output.

  2. 2.

InstructME (Han et al., 2023): InstructME is a novel framework for
instruction-guided music editing and remixing that leverages latent diffusion
models. It enhances a U-Net with multi-scale aggregation to ensure consistency
in the music before and after editing, and incorporates a chord progression
matrix into the semantic space to improve melodic harmony. To handle extended
musical pieces, InstructME utilizes a chunk transformer to recognize long-term
temporal dependencies within music sequences.

  3. 3.

M2UGen (Hussain et al., 2023): The M2UGen framework leverages large language
models to comprehend and generate music across various modalities, integrating
abilities from external models such as MusicGen (Copet et al., 2023) and
AudioLDM 2 (Liu et al., 2023a). It is designed to stimulate creative outputs
from diverse sources, showcasing robust performance in multi-modal music
generation.

####  4.4.2 Specialised models

  1. 1.

GMSDI (Postolache et al., 2024): GMSDI generalizes multi-source diffusion
models to handle compositional music generation tasks without requiring pre-
separated data. It allows for the flexible parameterization of an arbitrary
number of sources and employs text embeddings to enhance semantic control over
the music generation process. GMSDI uses the Slakh and Jamendo datasets for
model training.

  2. 2.

AudioSep (Liu et al., 2023c): AudioSep introduces the LASS-Net, which performs
audio source separation based on natural language queries. This model
effectively combines acoustic and linguistic data to isolate audio sources
that match the language descriptions, significantly improving over
conventional methods. AudioSep uses multiple datasets including AudioSet,
VGGSound, AudioCaps, Clotho v2 and WavCaps dataset for model training.

Model | ME | Param size | Dataset | Hours (h) | Training steps
---|---|---|---|---|---
AUDIT | ✓ | 942M (1.5B) | Multiple |  $\sim$6500 | 0.5M
InstructME | ✓ | 967M (1.7B) | Multiple | 417 | 2M
M2UGen | ✓ | 637M ($\sim$9B) | MUEdit | 60.22 | -
GMSDI | ✗ |  $\sim$1B | Slakh | 145 |  $\sim$1M
AudioSep | ✗ | 238M | Multiple |  $\sim$14000 | 1M
Ours | ✓ | 264M (3.5B) | Slakh | 145 | 5K
Table 1: Comparison of different models.

###  4.5 Metrics

To comprehensively evaluate the quality and alignment of predicted music with
ground truth and textual descriptions, we employ several metrics. The Fr’echet
Audio Distance (FAD)(Kilgour et al.,
2019)333https://github.com/gudgud96/frechet-audio-distance. measures overall
audio quality by comparing the distributions of feature embeddings between two
sets of audio files. The CLAP score (CLAP)(Wu et al.,
2023b)444https://github.com/LAION-AI/CLAP. assesses how well the audio content
aligns with corresponding textual descriptions, indicating semantic coherence.
Kullback-Leibler Divergence (KL)555https://github.com/haoheliu/audioldm_eval.
evaluates the information loss by comparing the probability distributions of
audio features from different sources, ensuring feature similarity between
predicted and ground truth music. The Structural Similarity (SSIM)(Wang et
al., 2004) metric, adapted from image quality assessment, evaluates the
structural similarity between predicted music and the ground truth. Lastly,
the Scale-Invariant Signal-to-Distortion Ratio (SI-SDR)(Roux et al., 2019) and
Scale-Invariant Signal-to-Distortion Ratio improvement (SI-SDRi) (Isik et al.,
2016) measure audio quality, particularly in source separation tasks, by
quantifying the distortion relative to a reference signal and the improvement
after processing, respectively. A detailed explaination can be found in
Appendix A.

###  4.6 Results

Our evaluation of Instruct-MusicGen demonstrates its superior performance
across various tasks compared to existing text-to-music editing baselines
(AUDIT, InstructME, M2UGen) and specialized trained models (GMSDI, AudioSep).
On the Slakh dataset (Table 2), Instruct-MusicGen excelled in adding,
removing, and extracting stems, achieving the lowest Fréchet Audio Distance
(FAD) and highest CLAP and SSIM scores in the addition task. It also
significantly improved the signal-to-noise ratio (SI-SDR) in the removal task,
showing balanced performance across all metrics and proving its robustness in
various editing scenarios. 666Since InstructME’s model weights and evaluation
protocol are not public released, we only record the evaluation results in
Table 5, which were reported in the original paper.

Task | Models | ME | FAD$\downarrow$ | CLAP$\uparrow$ | KL$\downarrow$ | SSIM$\uparrow$ | SI-SDR$\uparrow$ | SI-SDRi$\uparrow$
---|---|---|---|---|---|---|---|---
Add | AUDIT | ✓ | 6.88 | 0.12 | 1.02 | 0.21 | - | -
InstructME* | ✓ | - | - | - | - | - | -
M2UGen | ✓ | 7.24 | 0.22 | 0.99 | 0.20 | - | -
GMSDI | ✗ | 3.29 | 0.17 | 0.46 | 0.31 | - | -
Ours | ✓ | 3.75 | 0.23 | 0.67 | 0.26 | - | -
Remove | AUDIT | ✓ | 15.48 | 0.07 | 2.75 | 0.35 | -45.60 | -47.28
InstructME* | ✓ | - | - | - | - | - | -
M2UGen | ✓ | 8.26 | 0.09 | 1.59 | 0.23 | -44.20 | -46.13
GMSDI | ✗ | 1.30 | 0.12 | 0.23 | 0.78 | 5.67 | 3.74
AudioSep | ✗ | 0.90 | 0.18 | 0.63 | 0.65 | 1.38 | -0.31
Ours | ✓ | 3.35 | 0.12 | 0.66 | 0.45 | -2.09 | -3.77
Extract | AUDIT | ✓ | 15.08 | 0.06 | 2.38 | 0.42 | -52.90 | -50.16
InstructME* | ✓ | - | - | - | - | - | -
M2UGen | ✓ | 8.14 | 0.11 | 2.15 | 0.31 | -46.38 | -43.53
GMSDI | ✗ | 2.91 | 0.12 | 0.28 | 0.74 | 2.73 | 5.58
AudioSep | ✗ | 2.49 | 0.15 | 1.00 | 0.62 | -0.96 | 1.78
Ours | ✓ | 3.24 | 0.12 | 0.54 | 0.52 | -9.00 | -6.15
Table 2: Comparison of text-based music editing models on the Slakh dataset (4
stems).

Similarly, in the MoisesDB dataset evaluations (Table 3), Instruct-MusicGen
continued to demonstrate strong performance. It achieved competitive FAD
scores and showed improvements in CLAP and SSIM metrics for both addition and
removal tasks. While it did not always lead in SI-SDR, it consistently
outperformed baseline models and was comparable to specialized models,
highlighting its efficiency and effectiveness in text-to-music editing
applications.

Task | Models | ME | FAD$\downarrow$ | CLAP$\uparrow$ | KL$\downarrow$ | SSIM$\uparrow$ | SI-SDR$\uparrow$ | SI-SDRi$\uparrow$
---|---|---|---|---|---|---|---|---
Add | AUDIT | ✓ | 4.06 | 0.12 | 0.84 | 0.21 | - | -
InstructME* | ✓ | - | - | - | - | - | -
M2UGen | ✓ | 5.00 | 0.18 | 0.83 | 0.20 | - | -
GMSDI | ✗ | 5.40 | 0.12 | 0.60 | 0.30 | - | -
Ours | ✓ | 3.79 | 0.18 | 0.35 | 0.35 | - | -
Remove | AUDIT | ✓ | 10.72 | 0.10 | 2.46 | 0.34 | -44.32 | -57.10
InstructME* | ✓ | - | - | - | - | - | -
M2UGen | ✓ | 3.75 | 0.13 | 1.27 | 0.19 | -43.94 | -56.73
GMSDI | ✗ | 1.42 | 0.09 | 0.41 | 0.65 | -2.35 | -15.13
AudioSep | ✗ | 1.25 | 0.13 | 0.49 | 0.68 | 4.97 | -7.81
Ours | ✓ | 5.05 | 0.10 | 0.84 | 0.34 | -13.70 | -26.48
Extract | AUDIT | ✓ | 6.67 | 0.07 | 1.97 | 0.45 | -54.53 | -56.17
InstructME* | ✓ | - | - | - | - | - | -
M2UGen | ✓ | 5.74 | 0.08 | 1.91 | 0.25 | -42.84 | -44.49
GMSDI | ✗ | 4.95 | 0.08 | 1.32 | 0.54 | -8.19 | -9.83
AudioSep | ✗ | 1.81 | 0.14 | 1.01 | 0.65 | 1.14 | -0.50
Ours | ✓ | 4.96 | 0.11 | 1.36 | 0.40 | -21.39 | -23.03
Table 3: Comparison of text-based music editing models on the MoisesDB
dataset.

Despite these promising results, there are limitations to our approach. One
significant limitation is the reliance on synthetic data from the Slakh2100
dataset for training, which may not fully capture the diversity and complexity
of real-world music recordings. For example, while Instruct-MusicGen achieved
a competitive FAD score of 3.35 and an SSIM of 0.45 in the removal task on the
Slakh dataset, its SI-SDR improvement (SI-SDRi) was -3.77, indicating that
while the audio quality is high, the separation quality could be enhanced.
Similarly, in the extraction task, our model achieved a respectable FAD of
3.24 and SSIM of 0.52, but the SI-SDR and SI-SDRi metrics were -9.00 and
-6.15, respectively, highlighting challenges in accurately isolating stems
without introducing noise. These metrics suggest that while Instruct-MusicGen
performs well in preserving audio fidelity and similarity, it struggles with
signal clarity in more complex editing scenarios.

We provide further ablation studies in the Appendix A. 777Demos are available
at: https://bit.ly/instruct-musicgen.

##  5 Conclusion

In this paper, we introduced Instruct-MusicGen, a novel approach to text-to-
music editing that fosters joint musical and textual controls. By finetuning
the existing MusicGen model with instruction tuning, Instruct-MusicGen
demonstrated its capability of editing music in various ways, including
adding, separating and extracting a stem from music audio using textual
queries, without the need for training specialized models from scratch. Also,
it outperforms various baseline models that are dedicated to specific music
editing tasks. Furthermore, our method uses significantly fewer resources than
previous models, with a requirement of tuning only 8% of the parameters of the
original MusicGen.

Instruct-MusicGen still has several limitations. Firstly, the generative
editing tasks do not guarantee signal-level precision, as the process of
predicting EnCodec tokens inherently involves some inaccuracy. Second, the
finetuning process still relies on paired data, limiting the model’s
applicability to more complex editing tasks where such data is not readily
available. These constraints highlight the need for further research to
improve signal accuracy and expand the model’s capabilities to handle a
broader range of editing scenarios without heavily relying on paired datasets.
Nonetheless, the ability to edit music with fewer computational resources
opens up new avenues for creative expression and automation in music
composition. Instruct-MusicGen represents a significant contribution to the
field of text-to-music research, offering a robust and efficient tool for both
musicians and researchers.

## References

  * Agostinelli et al. [2023] Andrea Agostinelli, Timo I. Denk, Zalán Borsos, Jesse H. Engel, Mauro Verzetti, Antoine Caillon, Qingqing Huang, Aren Jansen, Adam Roberts, Marco Tagliasacchi, Matthew Sharifi, Neil Zeghidour, and Christian Havnø Frank.  MusicLM: Generating music from text.  _CoRR_ , abs/2301.11325, 2023.  doi: 10.48550/arxiv.2301.11325.  URL https://doi.org/10.48550/arxiv.2301.11325.
  * Brooks et al. [2023] Tim Brooks, Aleksander Holynski, and Alexei A. Efros.  InstructPix2Pix: Learning to follow image editing instructions.  In _IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2023, Vancouver, BC, Canada, June 17-24, 2023_ , pages 18392–18402. IEEE, 2023.  doi: 10.1109/CVPR52729.2023.01764.  URL https://doi.org/10.1109/CVPR52729.2023.01764.
  * Ceylan et al. [2023] Duygu Ceylan, Chun-Hao Paul Huang, and Niloy J. Mitra.  Pix2Video: Video editing using image diffusion.  In _IEEE/CVF International Conference on Computer Vision, ICCV 2023, Paris, France, October 1-6, 2023_ , pages 23149–23160. IEEE, 2023.  doi: 10.1109/ICCV51070.2023.02121.  URL https://doi.org/10.1109/ICCV51070.2023.02121.
  * Chai et al. [2023] Wenhao Chai, Xun Guo, Gaoang Wang, and Yan Lu.  StableVideo: Text-driven consistency-aware diffusion video editing.  In _IEEE/CVF International Conference on Computer Vision, ICCV 2023, Paris, France, October 1-6, 2023_ , pages 22983–22993. IEEE, 2023.  doi: 10.1109/ICCV51070.2023.02106.  URL https://doi.org/10.1109/ICCV51070.2023.02106.
  * Chen et al. [2023] Ke Chen, Yusong Wu, Haohe Liu, Marianna Nezhurina, Taylor Berg-Kirkpatrick, and Shlomo Dubnov.  MusicLDM: Enhancing novelty in text-to-music generation using beat-synchronous mixup strategies.  _CoRR_ , abs/2308.01546, 2023.  doi: 10.48550/arxiv.2308.01546.  URL https://doi.org/10.48550/arxiv.2308.01546.
  * Copet et al. [2023] Jade Copet, Felix Kreuk, Itai Gat, Tal Remez, David Kant, Gabriel Synnaeve, Yossi Adi, and Alexandre Défossez.  Simple and controllable music generation.  In Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine, editors, _Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 - 16, 2023_ , 2023.  URL http://papers.nips.cc/paper_files/paper/2023/hash/94b472a1842cd7c56dcb125fb2765fbd-Abstract-Conference.html.
  * Défossez et al. [2022] Alexandre Défossez, Jade Copet, Gabriel Synnaeve, and Yossi Adi.  High fidelity neural audio compression.  _CoRR_ , abs/2210.13438, 2022.  doi: 10.48550/arxiv.2210.13438.  URL https://doi.org/10.48550/arxiv.2210.13438.
  * Deng et al. [2024] Qixin Deng, Qikai Yang, Ruibin Yuan, Yipeng Huang, Yi Wang, Xubo Liu, Zeyue Tian, Jiahao Pan, Ge Zhang, Hanfeng Lin, et al.  ComposerX: Multi-agent symbolic music composition with LLMs.  _arxiv preprint arxiv:2404.18081_ , 2024.
  * Evans et al. [2024a] Zach Evans, CJ Carr, Josiah Taylor, Scott H. Hawley, and Jordi Pons.  Fast timing-conditioned latent audio diffusion.  _CoRR_ , abs/2402.04825, 2024a.  doi: 10.48550/arxiv.2402.04825.  URL https://doi.org/10.48550/arxiv.2402.04825.
  * Evans et al. [2024b] Zach Evans, Julian D Parker, CJ Carr, Zack Zukowski, Josiah Taylor, and Jordi Pons.  Long-form music generation with latent diffusion.  _arxiv preprint arxiv:2404.10301_ , 2024b.
  * Han et al. [2023] Bing Han, Junyu Dai, Xuchen Song, Weituo Hao, Xinyan He, Dong Guo, Jitong Chen, Yuxuan Wang, and Yanmin Qian.  InstructME: An instruction guided music edit and remix framework with latent diffusion models.  _CoRR_ , abs/2308.14360, 2023.  doi: 10.48550/arxiv.2308.14360.  URL https://doi.org/10.48550/arxiv.2308.14360.
  * Hu et al. [2021] Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, and Weizhu Chen.  LoRA: Low-rank adaptation of large language models.  _CoRR_ , abs/2106.09685, 2021.  URL https://arxiv.org/abs/2106.09685.
  * Hussain et al. [2023] Atin Sakkeer Hussain, Shansong Liu, Chenshuo Sun, and Ying Shan.  M${}^{\mbox{2}}$UGen: Multi-modal music understanding and generation with the power of large language models.  _CoRR_ , abs/2311.11255, 2023.  doi: 10.48550/arxiv.2311.11255.  URL https://doi.org/10.48550/arxiv.2311.11255.
  * Isik et al. [2016] Yusuf Ziya Isik, Jonathan Le Roux, Zhuo Chen, Shinji Watanabe, and John R. Hershey.  Single-channel multi-speaker separation using deep clustering.  In Nelson Morgan, editor, _Interspeech 2016, 17th Annual Conference of the International Speech Communication Association, San Francisco, CA, USA, September 8-12, 2016_ , pages 545–549. ISCA, 2016.  doi: 10.21437/INTERSPEECH.2016-1176.  URL https://doi.org/10.21437/Interspeech.2016-1176.
  * Kilgour et al. [2019] Kevin Kilgour, Mauricio Zuluaga, Dominik Roblek, and Matthew Sharifi.  Fréchet audio distance: A reference-free metric for evaluating music enhancement algorithms.  In Gernot Kubin and Zdravko Kacic, editors, _Interspeech 2019, 20th Annual Conference of the International Speech Communication Association, Graz, Austria, 15-19 September 2019_ , pages 2350–2354. ISCA, 2019.  doi: 10.21437/INTERSPEECH.2019-2219.  URL https://doi.org/10.21437/Interspeech.2019-2219.
  * Li et al. [2023] Peike Li, Boyu Chen, Yao Yao, Yikai Wang, Allen Wang, and Alex Wang.  JEN-1: Text-guided universal music generation with omnidirectional diffusion models.  _CoRR_ , abs/2308.04729, 2023.  doi: 10.48550/arxiv.2308.04729.  URL https://doi.org/10.48550/arxiv.2308.04729.
  * Li et al. [2024] Sifei Li, Yuxin Zhang, Fan Tang, Chongyang Ma, Weiming Dong, and Changsheng Xu.  Music style transfer with time-varying inversion of diffusion models.  In Michael J. Wooldridge, Jennifer G. Dy, and Sriraam Natarajan, editors, _Thirty-Eighth AAAI Conference on Artificial Intelligence, AAAI 2024, Thirty-Sixth Conference on Innovative Applications of Artificial Intelligence, IAAI 2024, Fourteenth Symposium on Educational Advances in Artificial Intelligence, EAAI 2014, February 20-27, 2024, Vancouver, Canada_ , pages 547–555. AAAI Press, 2024.  doi: 10.1609/AAAI.V38I1.27810.  URL https://doi.org/10.1609/aaai.v38i1.27810.
  * Liang et al. [2024] Jinhua Liang, Huan Zhang, Haohe Liu, Yin Cao, Qiuqiang Kong, Xubo Liu, Wenwu Wang, Mark D Plumbley, Huy Phan, and Emmanouil Benetos.  WavCraft: Audio editing and generation with large language models.  In _ICLR 2024 Workshop on Large Language Model (LLM) Agents_ , 2024.
  * Lin et al. [2023] Liwei Lin, Gus Xia, Junyan Jiang, and Yixiao Zhang.  Content-based controls for music large language modeling.  _CoRR_ , abs/2310.17162, 2023.  doi: 10.48550/arxiv.2310.17162.  URL https://doi.org/10.48550/arxiv.2310.17162.
  * Lin et al. [2024] Liwei Lin, Gus Xia, Yixiao Zhang, and Junyan Jiang.  Arrange, inpaint, and refine: Steerable long-term music audio generation and editing via content-based controls.  _CoRR_ , abs/2402.09508, 2024.  doi: 10.48550/arxiv.2402.09508.  URL https://doi.org/10.48550/arxiv.2402.09508.
  * Liu et al. [2023a] Haohe Liu, Qiao Tian, Yi Yuan, Xubo Liu, Xinhao Mei, Qiuqiang Kong, Yuping Wang, Wenwu Wang, Yuxuan Wang, and Mark D. Plumbley.  AudioLDM 2: Learning holistic audio generation with self-supervised pretraining.  _CoRR_ , abs/2308.05734, 2023a.  doi: 10.48550/arxiv.2308.05734.  URL https://doi.org/10.48550/arxiv.2308.05734.
  * Liu et al. [2023b] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee.  Visual instruction tuning.  In Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine, editors, _Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 - 16, 2023_ , 2023b.  URL http://papers.nips.cc/paper_files/paper/2023/hash/6dcf277ea32ce3288914faf369fe6de0-Abstract-Conference.html.
  * Liu et al. [2023c] Xubo Liu, Qiuqiang Kong, Yan Zhao, Haohe Liu, Yi Yuan, Yuzhuo Liu, Rui Xia, Yuxuan Wang, Mark D. Plumbley, and Wenwu Wang.  Separate anything you describe.  _CoRR_ , abs/2308.05037, 2023c.  doi: 10.48550/arxiv.2308.05037.  URL https://doi.org/10.48550/arxiv.2308.05037.
  * Manilow et al. [2019] Ethan Manilow, Gordon Wichern, Prem Seetharaman, and Jonathan Le Roux.  Cutting music source separation some Slakh: A dataset to study the impact of training data quality and quantity.  In _2019 IEEE Workshop on Applications of Signal Processing to Audio and Acoustics, WASPAA 2019, New Paltz, NY, USA, October 20-23, 2019_ , pages 45–49. IEEE, 2019.  doi: 10.1109/WASPAA.2019.8937170.  URL https://doi.org/10.1109/WASPAA.2019.8937170.
  * Manor and Michaeli [2024] Hila Manor and Tomer Michaeli.  Zero-shot unsupervised and text-based audio editing using DDPM inversion.  _CoRR_ , abs/2402.10009, 2024.  doi: 10.48550/arxiv.2402.10009.  URL https://doi.org/10.48550/arxiv.2402.10009.
  * Mariani et al. [2023] Giorgio Mariani, Irene Tallini, Emilian Postolache, Michele Mancusi, Luca Cosmo, and Emanuele Rodolà.  Multi-source diffusion models for simultaneous music generation and separation.  _CoRR_ , abs/2302.02257, 2023.  doi: 10.48550/arxiv.2302.02257.  URL https://doi.org/10.48550/arxiv.2302.02257.
  * Melechovsky et al. [2023] Jan Melechovsky, Zixun Guo, Deepanway Ghosal, Navonil Majumder, Dorien Herremans, and Soujanya Poria.  Mustango: Toward controllable text-to-music generation.  _arxiv preprint arxiv:2311.08355_ , 2023.
  * Parker et al. [2023] Julian D. Parker, Janne Spijkervet, Katerina Kosta, Furkan Yesiler, Boris Kuznetsov, Ju-Chiang Wang, Matt Avent, Jitong Chen, and Duc Le.  StemGen: A music generation model that listens.  _CoRR_ , abs/2312.08723, 2023.  doi: 10.48550/arxiv.2312.08723.  URL https://doi.org/10.48550/arxiv.2312.08723.
  * Pereira et al. [2023] Igor Pereira, Felipe Araújo, Filip Korzeniowski, and Richard Vogl.  MoisesDB: A dataset for source separation beyond 4-stems.  In Augusto Sarti, Fabio Antonacci, Mark Sandler, Paolo Bestagini, Simon Dixon, Beici Liang, Gaël Richard, and Johan Pauwels, editors, _Proceedings of the 24th International Society for Music Information Retrieval Conference, ISMIR 2023, Milan, Italy, November 5-9, 2023_ , pages 619–626, 2023.  doi: 10.5281/ZENODO.10265363.  URL https://doi.org/10.5281/zenodo.10265363.
  * Postolache et al. [2024] Emilian Postolache, Giorgio Mariani, Luca Cosmo, Emmanouil Benetos, and Emanuele Rodolà.  Generalized multi-source inference for text conditioned music diffusion models.  _CoRR_ , abs/2403.11706, 2024.  doi: 10.48550/arxiv.2403.11706.  URL https://doi.org/10.48550/arxiv.2403.11706.
  * Raffel et al. [2020] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu.  Exploring the limits of transfer learning with a unified text-to-text transformer.  _J. Mach. Learn. Res._ , 21:140:1–140:67, 2020.  URL http://jmlr.org/papers/v21/20-074.html.
  * Roux et al. [2019] Jonathan Le Roux, Scott Wisdom, Hakan Erdogan, and John R. Hershey.  SDR - half-baked or well done?  In _IEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP 2019, Brighton, United Kingdom, May 12-17, 2019_ , pages 626–630. IEEE, 2019.  doi: 10.1109/ICASSP.2019.8683855.  URL https://doi.org/10.1109/ICASSP.2019.8683855.
  * Wang et al. [2023] Yuancheng Wang, Zeqian Ju, Xu Tan, Lei He, Zhizheng Wu, Jiang Bian, and Sheng Zhao.  AUDIT: Audio editing by following instructions with latent diffusion models.  In Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine, editors, _Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 - 16, 2023_ , 2023.  URL http://papers.nips.cc/paper_files/paper/2023/hash/e1b619a9e241606a23eb21767f16cf81-Abstract-Conference.html.
  * Wang et al. [2004] Zhou Wang, Alan C. Bovik, Hamid R. Sheikh, and Eero P. Simoncelli.  Image quality assessment: From error visibility to structural similarity.  _IEEE Transactions on Image Processing_ , 13(4):600–612, 2004.  doi: 10.1109/TIP.2003.819861.  URL https://doi.org/10.1109/TIP.2003.819861.
  * Wu et al. [2023a] Shih-Lun Wu, Chris Donahue, Shinji Watanabe, and Nicholas J. Bryan.  Music ControlNet: Multiple time-varying controls for music generation.  _CoRR_ , abs/2311.07069, 2023a.  doi: 10.48550/arxiv.2311.07069.  URL https://doi.org/10.48550/arxiv.2311.07069.
  * Wu et al. [2023b] Yusong Wu, Ke Chen, Tianyu Zhang, Yuchen Hui, Taylor Berg-Kirkpatrick, and Shlomo Dubnov.  Large-scale contrastive language-audio pretraining with feature fusion and keyword-to-caption augmentation.  In _IEEE International Conference on Acoustics, Speech and Signal Processing ICASSP 2023, Rhodes Island, Greece, June 4-10, 2023_ , pages 1–5. IEEE, 2023b.  doi: 10.1109/ICASSP49357.2023.10095969.  URL https://doi.org/10.1109/ICASSP49357.2023.10095969.
  * Yang et al. [2023] Dongchao Yang, Jinchuan Tian, Xu Tan, Rongjie Huang, Songxiang Liu, Xuankai Chang, Jiatong Shi, Sheng Zhao, Jiang Bian, Xixin Wu, Zhou Zhao, Shinji Watanabe, and Helen Meng.  UniAudio: An audio foundation model toward universal audio generation.  _CoRR_ , abs/2310.00704, 2023.  doi: 10.48550/arxiv.2310.00704.  URL https://doi.org/10.48550/arxiv.2310.00704.
  * Yao et al. [2023] Yao Yao, Peike Li, Boyu Chen, and Alex Wang.  JEN-1 Composer: A unified framework for high-fidelity multi-track music generation.  _CoRR_ , abs/2310.19180, 2023.  doi: 10.48550/arxiv.2310.19180.  URL https://doi.org/10.48550/arxiv.2310.19180.
  * Yu et al. [2023] Dingyao Yu, Kaitao Song, Peiling Lu, Tianyu He, Xu Tan, Wei Ye, Shikun Zhang, and Jiang Bian.  MusicAgent: An AI agent for music understanding and generation with large language models.  In Yansong Feng and Els Lefever, editors, _Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, EMNLP 2023 - System Demonstrations, Singapore, December 6-10, 2023_ , pages 246–255. Association for Computational Linguistics, 2023.  doi: 10.18653/V1/2023.EMNLP-DEMO.21.  URL https://doi.org/10.18653/v1/2023.emnlp-demo.21.
  * Zeghidour et al. [2022] Neil Zeghidour, Alejandro Luebs, Ahmed Omran, Jan Skoglund, and Marco Tagliasacchi.  SoundStream: An end-to-end neural audio codec.  _IEEE ACM Trans. Audio Speech Lang. Process._ , 30:495–507, 2022.  doi: 10.1109/TASLP.2021.3129994.  URL https://doi.org/10.1109/TASLP.2021.3129994.
  * Zhang et al. [2023a] Renrui Zhang, Jiaming Han, Aojun Zhou, Xiangfei Hu, Shilin Yan, Pan Lu, Hongsheng Li, Peng Gao, and Yu Qiao.  LLaMA-Adapter: Efficient fine-tuning of language models with zero-init attention.  _CoRR_ , abs/2303.16199, 2023a.  doi: 10.48550/arxiv.2303.16199.  URL https://doi.org/10.48550/arxiv.2303.16199.
  * Zhang et al. [2023b] Yixiao Zhang, Akira Maezawa, Gus Xia, Kazuhiko Yamamoto, and Simon Dixon.  Loop Copilot: Conducting AI ensembles for music generation and iterative editing.  _CoRR_ , abs/2310.12404, 2023b.  doi: 10.48550/arxiv.2310.12404.  URL https://doi.org/10.48550/arxiv.2310.12404.
  * Zhang et al. [2024] Yixiao Zhang, Yukara Ikemiya, Gus Xia, Naoki Murata, Marco A. Martínez Ramírez, Wei-Hsiang Liao, Yuki Mitsufuji, and Simon Dixon.  MusicMagus: Zero-shot text-to-music editing via diffusion models.  _CoRR_ , abs/2402.06178, 2024.  doi: 10.48550/arxiv.2402.06178.  URL https://doi.org/10.48550/arxiv.2402.06178.

##  Appendix A Appendix / supplemental material

###  A.1 Metrics

The metrics to evaluate model performance are listed below.

  1. 1.

Fréchet Audio Distance (FAD) [Kilgour et al.,
2019]888https://github.com/gudgud96/frechet-audio-distance. measures the
similarity between two sets of audio files by comparing multivariate Gaussian
distributions fitted to feature embeddings from the audio data. We use the FAD
score to evaluate the overall audio quality of the predicted music.

  2. 2.

CLAP score (CLAP) [Wu et al., 2023b]999https://github.com/LAION-AI/CLAP.
evaluates the alignment and relevance of audio content with corresponding
textual descriptions, using models pretrained on both audio and language tasks
to assess semantic coherence between audio and text. A high CLAP score
indicates the predicted music aligns with the text description well.

  3. 3.

Kullback-Leibler Divergence
(KL)101010https://github.com/haoheliu/audioldm_eval. assesses the difference
between the probability distributions of audio features from two sources,
indicating information loss when approximating one distribution with another.
A low KL score indicates the predicted music shares similar features with the
ground truth.

  4. 4.

Structural Similarity (SSIM) [Wang et al., 2004] is an image quality metric
that we adapt to evaluate structural similarity between predicted music and
ground truth.

  5. 5.

Scale-Invariant Signal-to-Distortion Ratio (SI-SDR) [Roux et al., 2019]
quantifies audio quality, especially in source separation tasks. It is scale-
invariant, useful for varying audio volumes, and measures distortion relative
to a reference signal. We use SI-SDR score to evaluate the signal loss of the
predicted audio.

  6. 6.

Scale-Invariant Signal-to-Distortion Ratio improvement (SI-SDRi) [Isik et al.,
2016] extends SI-SDR, measuring the improvement in signal-to-distortion ratio
before and after processing. It is commonly used in audio enhancement and
separation contexts.

###  A.2 Ablations

####  A.2.1 Hyperparameter selection

The total trainable parameters include the input embedding layer, linear
layers for joint embeddings learning, and gate scalars, and the linear layers
has over 80% parameters in the total parameters. In this ablation we want to
investigate whether the parameters can be less by revising the architecture of
the linear layers.

The original linear layers has the shape of $(48,2048,2048)$, where 48 is the
layer number of MusicGen model, and the $(2048,2048)$ is a linear
transformation from embedding to layer-wise representation. We replace a
single layer with an auto-encoder architecture, consisting of two linear
layers: $(2048,M)$ and $(M,2048)$, where $M\ll 2048$.

We record and compare the loss on the validation set between the variant and
the original instruct-MusicGen model, shown in Figure 3. The variation has
$M=12$, which reduces the parameter size from 201M to 23M. From the
experimental results, we find that even the paramater size becomes less, the
instruction tuning still works for MusicGen, but the small parameter size
limits the model’s representation ability, making the model performs worse
than the original version.

Figure 3: The comparison between the original instruct-MusicGen and its
variant with less parameters.

####  A.2.2 Comparison with original text encoder

We use an audio fusion module to enable MusicGen to accept additional audio
input, allowing the model to process instruction text input without adding a
new adapter. By finetuning the cross-attention module between the text encoder
and the MusicGen transformer, we can fuse instructional knowledge into text
information. In this ablation study, we examine the necessity of finetuning
the text encoder. Since the text encoder is already capable of accepting text
input, can it directly understand the semantic meaning of editing
instructions?

To investigate this, we revised the architecture of Instruct-MusicGen’s text
fusion module, configuring the music decoder to only accept cross-attention
information from the original text encoder.

From Table 4, we observe that removing the text fusion module and relying on
the original text encoder to interpret the instructions significantly
decreases the model’s performance. Qualitative results from the predicted
samples indicate that the original text encoder, which is designed for text
description, struggles with the format of instructional commands. It fails to
fully understand the semantics of keywords such as "extract," "remove," "no,"
and "only," which are not typically used to describe music but rather to
indicate editing operations. Therefore, the presence of the text fusion module
is crucial for MusicGen to correctly interpret the semantic meaning of
instructions.

Task | Models | FAD$\downarrow$ | CLAP$\uparrow$ | KL$\downarrow$ | SSIM$\uparrow$ | SI-SDR$\uparrow$ | SI-SDRi$\uparrow$
---|---|---|---|---|---|---|---
Add | w/o. text fusion | 5.26 | 0.15 | 0.70 | 0.25 | - | -
w. text fusion | 3.75 | 0.23 | 0.67 | 0.26 | - | -
Remove | w/o. text fusion | 5.58 | 0.09 | 1.23 | 0.31 | -10.32 | -12.25
w. text fusion | 3.35 | 0.12 | 0.66 | 0.45 | -2.09 | -3.77
Extract | w/o. text fusion | 7.58 | 0.04 | 1.87 | 0.42 | -23.06 | -20.21
w. text fusion | 3.24 | 0.12 | 0.54 | 0.52 | -9.00 | -6.15
Table 4: Performance comparison between models with and without text fusion
module across different tasks.

###  A.3 Experiments on other datasets

Since GMSDI only handle 4-stem music modelling, we construct a variant
evaluation on Slakh dataset, where all stem can be considered, and the results
are shown in Table 5.

Task | Models | ME | FAD$\downarrow$ | CLAP$\uparrow$ | KL$\downarrow$ | SSIM$\uparrow$ | SI-SDR$\uparrow$ | SI-SDRi$\uparrow$
---|---|---|---|---|---|---|---|---
Add | AUDIT | ✓ | 6.30 | 0.08 | 1.27 | 0.14 | - | -
InstructME* | ✓ | 3.15 | - | - | - | - | -
M2UGen | ✓ | 5.80 | 0.21 | 0.88 | 0.17 | - | -
Ours | ✓ | 3.07 | 0.20 | 0.53 | 0.21 | - | -
Remove | AUDIT | ✓ | 12.19 | 0.06 | 2.58 | 0.27 | -45.40 | -49.32
InstructME* | ✓ | 1.87 | - | - | - | - | -
M2UGen | ✓ | 6.42 | 0.10 | 1.42 | 0.22 | -42.72 | -46.02
AudioSep | ✗ | 0.96 | 0.16 | 0.57 | 0.64 | 2.51 | -0.78
Ours | ✓ | 2.95 | 0.12 | 0.64 | 0.39 | -3.01 | -6.31
Extract | AUDIT | ✓ | 13.78 | 0.04 | 2.86 | 0.43 | -54.95 | -49.10
InstructME* | ✓ | 5.04 | - | - | - | - | -
M2UGen | ✓ | 6.29 | 0.12 | 1.64 | 0.30 | -45.65 | -42.53
AudioSep | ✗ | 2.81 | 0.16 | 1.30 | 0.59 | -2.22 | 0.91
Ours | ✗ | 3.25 | 0.08 | 0.85 | 0.45 | -9.89 | -6.77
Table 5: Comparison of Text-Based Music Editing Models on the Slakh dataset
(full stems).

###  A.4 Social Impact

The introduction of text-based music editing models like Instruct-MusicGen has
the potential to significantly impact the music industry by democratizing
music creation and editing. By lowering the technical barriers to music
editing, these models enable a broader range of individuals, including those
without formal music training, to engage in creative processes. This can
foster greater diversity in music production, as more people from various
backgrounds can contribute their unique perspectives and ideas. Additionally,
the efficiency and accessibility of text-based music editing can streamline
workflows for professional musicians and producers, allowing them to
experiment with different musical elements quickly and with less effort.

However, it is important to recognise the limitations and challenges
associated with these advancements. The reliance on paired data for finetuning
can restrict the model’s versatility in handling more complex editing tasks.
Moreover, the inherent loss in generative editing might affect the precision
of the output, potentially requiring additional finetuning by human experts.
While these models offer new tools for creative expression, they are not a
replacement for the nuanced understanding and expertise of skilled musicians.
The social impact of text-based music editing models will largely depend on
how they are integrated into existing workflows and their acceptance by the
music community.

###  A.5 Audio samples

We provide audio samples in 4. From the results we can find the instruct-
MusicGen performs the adding , removing and extracting tasks.

(a) Input music.

(b) Edited music output.

(c) Ground truth.

(d) Input music.

(e) Edited music output.

(f) Ground truth.

(g) Input music.

(h) Edited music output.

(i) Ground truth.

Figure 4: Audio samples. Fig (a-c): Extracting a stem. Instruction: "Extract
drum."; Fig (d-f): Adding a stem. Instruction: "Add piano."; Fig (g-i):
Removing a stem. Instruction: "Remove bass";