# LLMs Meet Multimodal Generation and Editing:
A Survey

Yingqing He†∗, Zhaoyang Liu†∗, Jingye Chen∗, Zeyue Tian∗, Hongyu Liu∗, Xiaowei
Chi∗,
Runtao Liu∗, Ruibin Yuan∗, Yazhou Xing∗, Wenhai Wang, Jifeng Dai, Yong Zhang,
Wei Xue, Qifeng Liu, Yike Guo, Qifeng Chen
† Project leaders; ∗ Co-first authors. Yingqing He, Zhaoyang Liu, Jingye Chen,
Zeyue Tian, Hongyu Liu, Xiaowei Chi, Runtao Liu, Ruibin Yuan, Yazhou Xing, Wei
Xue, Qifeng Liu, Yike Guo, and Qifeng Chen are with The Hong Kong University
of Science and Technology, Hong Kong SAR. Wenhai Wang is with The Chinese
University of Hong Kong, Hong Kong SAR. Jifeng Dai is with Tsinghua
University, China. Yong Zhang is with the Tencent AI Lab, China.

###### Abstract

With the recent advancement in large language models (LLMs), there is a
growing interest in combining LLMs with multimodal learning. Previous surveys
of multimodal large language models (MLLMs) mainly focus on understanding.
This survey elaborates on multimodal generation across different domains,
including image, video, 3D, and audio, where we highlight the notable
advancements with milestone works in these fields. Specifically, we
exhaustively investigate the key technical components behind methods and
multimodal datasets utilized in these studies. Moreover, we dig into tool-
augmented multimodal agents that can use existing generative models for human-
computer interaction. Lastly, we also comprehensively discuss the advancement
in AI safety and investigate emerging applications as well as future
prospects. Our work provides a systematic and insightful overview of
multimodal generation, which is expected to advance the development of
Artificial Intelligence for Generative Content (AIGC) and world models. A
curated list of all related papers can be found at
https://github.com/YingqingHe/Awesome-LLMs-meet-Multimodal-Generation.

###### Index Terms:

Multimodal Generation, Large Language Models, Text-to-Image, Text-to-Video,
Text-to-3D, Text-to-Audio, Multimodal Agents, AI Safety, Diffusion Models,
Transformers, Generative AI.

Figure 1:  Our main goal is to investigate the roles of LLMs in the task of
language-guided multimodal generation. The modalities we focused on consist of
image, video, 3D, and audio (including sound, music, and speech).

##  1 Introduction

The interactions between human beings and the physical world involve
information from multiple modalities, such as language, vision, and audio.
Therefore, realizing a world simulator also requires the model to be able to
perceive and respond with multimodal information in a flexible manner.
Recently, OpenAI proposed a foundation video generation model Sora [1] that is
capable of generating highly realistic videos as world simulators. It makes
great progress in simulating or generating real-world videos but is unable to
generate other modalities, such as text, 3D, and audio. Also, it lacks the
ability to perceive other modalities such as image, video, 3D, and audio,
making it an incomprehension world simulator.

In the past few years, researchers focused on the generation of each single
modality and have achieved great progress: In terms of text generation, we
have witnessed a qualitative leap in the effectiveness of text generation:
From BERT [2], GPT1 [3], GPT2 [4], GPT3 [5], GPT4 [6] to ChatGPT [7], LLaMA
[8, 9], the number of model parameters and training samples has grown rapidly,
resulting in the continual growth of modal abilities and product deployment.
In the visual generation field, with the rapid progress of diffusion models
and large-scale image-text datasets, image generation has achieved remarkable
achievement and can synthesize high-quality images based on various user-
provided text prompts [10, 11, 12, 13]. Subsequently, significant advancements
have been made in the field of video generation through the utilization of
video diffusion models and large-scale video-language datasets. Notably,
several groundbreaking works have emerged, such as [14, 15, 16, 17, 18, 19,
20, 21, 22] and Sora [1]. For the 3D generation, with the emergence of the
CLIP [23] model, some methods [24, 25, 26] try to bring the text information
to the rendered image of 3D representation (i.e., point cloud, mesh, NeRF [27]
and gaussian splatting [28]), these approaches have led to significant
developments in the text-to-3D generation. Additionally, the integration of
Stable Diffusion (SD) [10] with text-to-image rendering has enabled a series
of works in text-to-3D generation [29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39,
40, 41, 42, 43]. The powerful text-to-image model helps the 3D generation
achieve higher performance and better results. In the area of audio
generation, a series of representative works tackle the different audio
domains such as [44, 45, 46] for text-to-audio, [47, 48, 49] for text-to-
music, and [50, 51, 52, 53, 54, 55] for text-to-speech, and they have achieved
significant performance in generating high-quality natural sounds, music, and
human-level speech.

With the notable progress and significant performance improvements of Large
Language Models (LLMs), other non-text modalities have started harnessing the
power of LLMs to either enhance their generation pipelines or integrate text
generation and non-text generation in a unified system, aiming to achieve more
advanced functionalities and improved generation performance. In the context
of image generation, there are two categories of approaches that have achieved
significant integration with language models. The first category involves
encoding visual information into discrete token IDs, trying to unify visual
understanding and generation [56, 57, 58, 59, 60, 61]. Specifically, visual
information is encoded into token representations, and LLMs directly
comprehend and generate visual tokens, enabling simultaneous visual
understanding and generation. The second category focuses on leveraging LLMs
to enhance the generation quality of existing pretrained text-to-image (T2I)
models: One category of works involves utilizing an LLM as a layout planner to
incorporate knowledge of object spatial positions, quantity, and object size,
enabling the generation of required bounding boxes [62, 63, 64, 65, 66]. After
obtaining the bounding boxes, the images can be generated through a grounded
text-to-image (T2I) model [67]. Another approach involves utilizing LLMs to
expand input user prompts [68]: By providing highly detailed and comprehensive
user prompts, LLMs generate high-quality images by enriching the prompt
information. With the assistance of LLMs, visual generation has achieved
higher generation quality, improved prompt following capabilities, dialogic
function, and user-friendly interface. In video generation, LLMs serve as the
general backbone for unified multimodal joint generation [69, 70], video
layout planning [71, 72, 63, 73, 74] and temporal prompt generation [75, 76,
77, 78, 79] for dynamics guidance. For 3D generation and editing, LLMs serve
as a bridge between users and 3D assets, which improves interaction efficiency
[80, 81] and helps users understand [82, 83] the 3D assets. In the context of
audio generation and editing, the role of language models primarily lies in
serving as coordinated backbones for multimodal audio [84, 85, 86, 87, 88, 89,
90, 91, 92, 93, 94, 95, 96], conditioners for specific tasks [97, 98, 99],
labelers for audio understanding [100, 101, 102], and agents for interactive
generation/editing [103, 104, 105, 106, 107, 108], and as inspiration for
novel approaches [53, 47, 109, 48, 110, 111]. The growing utilization of LLMs
in the audio domain is not only transforming our way of engaging with sound
and music but also expanding the boundaries at the crossroads of AGI and audio
technologies. In addition, multimodal agents integrate multiple modalities
into a single system to develop a universal system, that possesses the ability
of comprehension and generation of non-text modalities. Therefore, LLMs are
increasingly playing an indispensable role in generating various modes of
content.

To empower the world simulator and promote the development of multimodal
generation, in this work, we provide a comprehensive review of works involving
LLMs in the generation of multiple modalities and the roles of LLMs in this
process. As shown in Fig. 1, we summarize the roles of LLMs into several key
aspects, such as evaluator, labeller, instruction processor, planner, provider
of semantic guidance, or as backbone architectures. Also, we discuss the
safety issues in the AIGC era in Sec. 9, with the emerged applications, and
the potential future prospects in Sec. 10 and Sec. 11.

We summarize our contributions as follows:

  * •

We present the first systematic review of LLMs applied to the generation of
multiple modalities, including images, videos, 3D, and audio.

  * •

We discuss the evolution of generative techniques through a comparative
analysis of pre-LLM and post-LLM eras, offering a clear perspective on the
progression and refinement of these approaches.

  * •

We summarize the various roles of LLMs in the generation process in each
modality from a technical view.

  * •

We discuss important AI safety issues, investigate emerging applications and
explore future directions to boost the development of multimodal generation
and world models.

###  1.1 Content Overview

We first review related surveys on modality-specific generation and LLMs in
Sec. 2. We then briefly review the basic techniques of representative
generative models, multi-modality encoder, transformer, and LLMs in Sec. 3.
Next, we review the LLM-based visual generation in different visual
modalities, including image in Sec. 4, video in Sec. 5, 3D in Sec. 6, audio in
Sec. 7, and multimodal agents in Sec. 8, respectively. Lastly, we discuss the
safety aspect of generative AI in Sec. 9 and several potential future
directions for the LLMs-based multimodal generation field in Sec. 11.

###  1.2 Scope

This survey explores the generation of multiple modalities, including images,
videos, 3D models, and audio. The multimodal generation in our survey includes
the separate generation of different modalities as well as the joint
generation of multiple modalities. We will not dig into pure text generation
extensively, as there have already been many surveys specifically focusing on
the advancements in that field [112, 113, 114]. Our primary focus is on how
the recent emergence of large language models in the past few years can assist
in the generation of other visual and audio modalities, especially in the
open-domain generation. This will aid us in designing better unified
generative models for multi-modalities. In detail, we focus on the following
tasks:

  * •

Image generation and editing: Image generation aims to create various open-
domain image contents, including pictures, photos, or stylized drawings from
user-provided textual descriptions. Image editing aims to modify the input
image content and can be based on user instructions.

  * •

Video generation and editing, where models generate or modify arbitrary and
various dynamic visual contents guided by free-form textual descriptions;

  * •

3D generation and editing, which a task for generating and editing 3D objects,
scenes or avatars with user-provided textual descriptions.

  * •

Audio generation and editing, where textual descriptions are used to generate
audio, including general sounds, music, and speech. Audio editing tasks such
as adding, removing, or inpainting involve modifying existing audio using
textual descriptions.

  * •

Multimodal generative agents, which enable LLMs to handle data across
different modalities by utilizing a variety of specialized multimodal tools.

  * •

Generative AI Safety, which focuses on reducing toxic and biased content,
protecting copyright, and addressing the creation of fabricated content by
multimodal generative models.

##  2 Related Surveys

Survey on modality-specific generation. A series of surveys focus on single
modality generation, such as [115] for image generation, [116] for video
generation, [117] for 3D generation, [118] for audio generation. However, the
previous generation paradigm mainly employs pretrained Clip [23], Clip-related
variant [119], or language encoder T5 [120] to achieve open-domain text-guided
generation. With the emergence of LLMs, there are more and more generation
works that leverage the powerful LLM to enhance the generation of each
modality. Our work mainly discusses the role of LLMs in the generation of each
modality, which is absent in previous surveys.

Survey on LLMs Numerous surveys have been conducted to explore various aspects
of LLMs. For example, [121] offers a comprehensive examination of LLMs-based
autonomous agents. Additionally, [122], and [123] look into multimodal LLMs
(MLLMs), introducing papers that combine LLMs with other non-text modalities.
They review papers for both multimodal understanding and generation in a mixed
manner, introduce primarily multimodal understanding works, and focus less on
multimodal generation. In contrast, our work primarily concentrates on the
generation aspect, aiming to investigate the performance and functionality
improvement that LLMs bring to each modality’s generation process, leading to
a better AI-generated world with multiple modalities.

##  3 Preliminaries

In this section, we first review different types of generative models in Sec.
3.1. Then, we present multimodal alignment models in Sec. 3.2. Lastly, we
summarize the foundational architecture behind large language models in Sec.
3.4.

Figure 2: The illustration of generative models. In this picture, $x$ and
$x_{0}$ indicate the sample from the real data distribution, $x^{\prime}$
stands for the sample from the model’s estimated data distribution, and $z$
means the latent sampled from a prior distribution (typically a Gaussian
distribution).

###  3.1 Generative Models

We review the core principles and basic concepts of classic generative models,
including generative adversarial Networks (GANs), variational autoencoders
(VAEs), flow-based models, diffusion models, and autoregressive models.

The generation process of generative models can be characterized as a
transformation from a latent sample $z$ drawn from a prior distribution
$p_{\mathbf{z}}(\mathbf{z})$ to a generated sample $\mathbf{x}^{\prime}$ from
a real data distribution $p_{\text{data}}(\mathbf{x})$ that aligns with the
target data distribution. Specifically, the latent variable $\mathbf{z}$ is
passed through a parametric function, typically implemented as a neural
network, which learns to map the prior distribution to the target data
distribution. The output $\mathbf{x}^{\prime}$ of this transformation is then
regarded as a synthetic instance that mimics the statistical properties of the
original data distribution, which may correspond to various modalities such as
images, videos, 3D representations, audio, or text.

####  3.1.1 Generative Adversarial Networks

GAN [124] has achieved promising results in various tasks during years of
development. As shown in Fig. 2, GAN comprises two crucial components: a
discriminator ($D$) and a generator ($G$). The discriminator is designed to
distinguish between real and fake samples. The generator aims to create fake
samples that are indistinguishable from the real data and try to fool the
discriminator. During training, the $G$ and $D$ are trained simultaneously and
play a two-player minimax game. The optimization objective is formulated as
follows:

$\displaystyle\min_{G}\max_{D}V(D,G)=$
$\displaystyle\mathbb{E}_{\mathbf{x}\sim p_{\text{data}}(\mathbf{x})}[\log
D(\mathbf{x})]+\mathbb{E}_{\mathbf{z}\sim
p_{\mathbf{z}}(\mathbf{z})}[\log(1-D(G(\mathbf{z})))]$

Where $D(\mathbf{x})$ indicates the probability of real sample $\mathbf{x}$
being a real sample, and $D(G(\mathbf{z}))$ indicates the probability of a
generated sample being a fake sample. $\mathbb{E}_{x}$ is the expectation
value over all samples.

####  3.1.2 Variational Auto-Encoder

Variational auto-encoder [125] contains an encoder and a decoder to learn
latent representations from the input data, as shown in the second row in Fig.
2.

Encoder is a neural network that maps the input data $\mathbf{x}$ to the
distribution of the latent space variable $\mathbf{z}$. Then, the variational
posterior distribution $q(\mathbf{z}|\mathbf{x})$ is usually assumed to be a
Gaussian distribution $N(\mu,\sigma^{2})$. In this case, the encoder gives the
$\mu$ and ${\sigma}^{2}$.

Decoder maps the latent space variable $\mathbf{z}$ back to the input space
$\mathbf{x}^{\prime}$, yielding the conditional distribution of the generated
data $x$,$q(x|z)$.

The training optimization target of VAE is to maximize the lower bound of the
marginal log-likelihood of the data. This target can be achieved through
stochastic gradient descent and the reparameterization trick. It is also known
as Evidence Lower Bound (ELBO). Specifically, the ELBO can be written in the
following form:

$\displaystyle\mathcal{L}(\theta;\mathbf{X})=$
$\displaystyle-\left[KL(q(\mathbf{z}|\mathbf{x};\theta)||p(\mathbf{z}))-\mathbb{E}_{q(\mathbf{z}|\mathbf{x};\theta)}[\log
p(\mathbf{x}|\mathbf{z};\theta)]\right]$

where $KL(\textperiodcentered||\textperiodcentered)$ denotes the KL
divergence, which measures the difference between the posterior distribution
inferred by the encoder $q(\mathbf{z}|\mathbf{x})$ and the prior distribution
$p(z)$. The second term is the reconstruction error, which represents the
match between the $\mathbf{x^{\prime}}$ generated from $z$ and the actual data
$x$.

Vector Quantized Variational Auto-Encoder VQ-VAE[126] is a variant of the VAE
that introduces a discrete latent space, which significantly improves the
quality of the generated samples compared to the original VAE. In VQ-VAE, the
encoder maps the continuous output of the encoder to the nearest point in a
predefined discrete codebook and outputs a discrete latent representation
$\mathbf{z}$. The codebook is learned jointly with the rest of the model
parameters. Using a discrete latent space allows VQ-VAE to capture more global
and structured information about the data, leading to better generation
quality.

####  3.1.3 Flow-based Model

Flow-based model, also known as normalizing flows, is a class of generative
models successfully applied in various tasks, including image synthesis,
variational inference, and unsupervised representation learning. The
architecture of a flow-based model consists of a sequence of invertible
transformations (or flows). Each flow is parameterized by a neural network,
which learns to transform the data distribution step by step into the simpler
prior distribution. The objective function for training a flow-based model is
the negative log-likelihood of the data under the model, which can be computed
exactly due to the invertibility of the flows. The function is given by:

$\displaystyle\mathcal{L}(\theta)=-\mathbb{E}_{\mathbf{x}\sim
p_{\text{data}}(\mathbf{x})}[\log p_{\text{model}}(\mathbf{x};\theta)]$ (3)

where $p_{\text{model}}(\mathbf{x};\theta)$ is the probability density
function of the model. In practice, the transformations used in flow-based
models are chosen to be easily invertible and have easily computable
Jacobians, such as affine transformations.

####  3.1.4 Diffusion Model

The diffusion model is proposed in [127], which first gives the prototype of
recent diffusion models. However, the foundational structure of the modern
diffusion model, which led to a revolution of generation paradigms, was
proposed in Denoised Diffusion Probabilistic Models[128]. It is elegant in
training and mythical improvement and only introduced a simple regression
loss. As shown in 2, the diffusion model turned the complex work into a series
of denoising tasks, primarily consisting of two steps: the injection of prior
noise into the data and the denoising prediction.

Forward Noise Injection In the forward noise injection process, the model
sequentially introduces Gaussian noise $\mathbf{\zeta_{t}}$ into the data in
each step $t$ of $T$ steps. The process can be represented as follows:

$\mathbf{x_{t+1}}=\sqrt{1-\alpha_{t}^{2}}x_{t}+\alpha_{t}\mathbf{\zeta_{t}}$
(4)

Where $\mathbf{x_{t}}$ is the data at time $t$, $\mathbf{\zeta_{t}}$ is the
Gaussian noise at time $t$, and $\mathbf{\alpha_{t}}$ is a noise schedule
determining the amount of noise to be added at each step. The noise schedules
$\mathbf{\alpha_{t}}$ typically start close to 0 and gradually increase to 1
over the $T$ steps.

The noise at each step is assumed to follow a Markov transition process, which
means the noise at time $t$, $\mathbf{\zeta_{t}}$, is independent of the
noises at all previous times. This assumption simplifies the model and makes
it tractable to train.

Reverse Denoising After the forward noise injection, the model aims to reverse
the process by predicting the original data from the noisy version. This is
done by learning a denoising function, which is typically parameterized as a
deep neural network. The denoising function inputs the noisy data at time $t$
and attempts to predict the noise added at that step. This process is repeated
for each time step, going backward from $T$ to 1. The denoising function can
be represented as follows:

$\mathbf{\hat{\zeta}_{t}}=D_{\theta}(\mathbf{x_{t}},t)$ (5)

where $D_{\theta}$ is the denoising function parameterized by $\theta$,
$x_{t}$ is the noisy data at time $t$, and $\mathbf{\hat{\zeta}_{t}}$ is the
predicted noise.

The model’s objective during training is to minimize the difference between
the predicted noise $\mathbf{\hat{\zeta}_{t}}$ and the actual noise
$\mathbf{\zeta_{t}}$ added during the forward process. This can be measured
using a simple mean squared error loss.

Training the model to perform this reverse denoising prediction allows it to
generate data similar to the original data distribution, making the diffusion
model a powerful tool for generative tasks.

####  3.1.5 Autoregressive Model

Autoregressive models are another class of generative models widely used in
various tasks, including time series forecasting, speech synthesis, and
natural language processing. Their architecture predicts future values based
on past values.

The objective function for training an autoregressive model is also the
negative log-likelihood of the data under the model, which can be computed
exactly due to the sequential nature of the model. This is given by:

$\displaystyle\mathcal{L}(\theta)=-\mathbb{E}{\mathbf{x}\sim
p_{\text{data}}(\mathbf{x})}[\log
p_{\text{model}}(\mathbf{x_{t+1}}|\mathbf{x_{\leq t}};\theta)]$ (6)

where $p_{\text{model}}(\mathbf{x_{t+1}}|\mathbf{x_{\leq t}};\theta)$ is the
conditional probability density function of the model. In practice, the model
is trained to maximize the likelihood of the next value in the sequence given
the previous values.

###  3.2 Multimodal Alignment Model

CLIP [23] is a ground-breaking image-language alignment model that
simultaneously learns an image encoder and a text encoder to produce visual
and textual representations in a shared semantic space, trained on a diverse
range of internet text and images through contrastive learning [129]. After
its large-scale contrastive pretraining, it is capable of tackling various
downstream tasks, including fine-grained object recognition, video action
recognition, facial emotion recognition, geo-localization, and many others in
a zero-shot manner. Thanks to its web-scale pertaining, it can understand
plenty of semantics. Thus, it has become one of the most widely used visual
and textual encoders in various vision generation and editing works such as
DALLE-2 [12] and LDM [10] for text-to-image generation, VideoCrafter [19] for
text-to-video and Clip-Nerf [130] for 3D.

Besides text and vision alignment, CLAP [131] aligns text and audio
information. The audio-aligned text embedding representation is used as the
condition of AudioLDM [44] for text-guided audio generation.

CAVP makes a further step towards video-audio alignment, which is trained in
Diff-Foley [132] for the video-to-audio generation task. After training CAVP,
Diff-Foley further trains a latent diffusion model, which is conditioned on
the audio-aligned video representation to output synchronized audio signals.

Different from previous methods for the alignment of paired modalities,
ImageBind[133] aligns six different modalities in one shared semantic space.
The supporting modalities include text, image, video, audio, depth, and
thermal. It has been used in multimodal generation tasks such as Next-GPT
[134], Seeing-and-Hearing [135], and also multimodal understanding works such
as PandaGPT [136].

###  3.3 Large Language Models

Modern Large Language Models utilize the transformer architecture to generate
contextually rich embeddings. These models are trained on a large corpus of
text and then fine-tuned for specific tasks. They generate text by predicting
the next word in a sequence, given the previous words.

Typical examples include LlaMA[137] and GPT[138, 139, 140], which are the
autoregressive models that use only the left context to make predictions. They
are mainly built by transformer decoder. The models would be pre-trained on
large, diverse datasets to acquire a strong foundation of language
understanding and generation capabilities and then fine-tuned on datasets that
provide explicit instructions or guidance on specific tasks, such as question-
answering summarization or code generation. Furthermore, tricks like Chain-of-
Thought (CoT) [5] fine-tuning and Reinforcement Learning from Human Feedback
(RLHF) [141] improve the models’ task-specific ability.

###  3.4 Multimodal Large Language Models

Multimodal Large Language Models (MLLMs) are recently emerging models that aim
to equip LLMs with the ability to understand or generate other modalities.
MLLMs usually incorporate a few key components: extra pre-trained modality-
specific encoders for feature extraction and input projectors for multimodal
hidden feature alignment with the LLM backbone. For MLLMs with generation
ability, they usually contain extra output projectors and corresponding
modality generators as the generation ends. A series of works incorporating
extra pre-trained encoders for encoding multimodal information to a pre-
trained LLM and train modality alignment modules to achieve this [142, 143,
134, 144, 145, 146]. Other works train the whole multimodal system in an end-
to-end manner [147]. In the following sections, we will illustrate a range of
recent MLLMs works, especially MLLMs on multimodal generation.

Figure 3: History review on the development trajectory of image generation.
Early works on image generation predominantly concentrated on synthesizing
images within specific narrow domains, such as human faces or bedrooms [148,
149]. Subsequently, DALL-E [150] and Latent Diffusion Models (LDM) [10] have
progressed to generate images through user prompts and support the synthesis
of open-domain images. In the recent two years, powered by LLMs, research has
trended toward achieving a more intuitive and interactive image generation
process, such as iterative generation through conversations [151, 152].  TABLE
I:  Overview of existing methods using LLMs for language-based image
generation. According to the role of LLMs in this task, these methods can be
divided into four categories: multimodal LLMs for generation, image layout
planning, prompt synthesis and refinement, and image quality evaluation. In
the ”Task“ column, the ”T“ and ”I“ are the abbreviations of ”text“ and
”image“, respectively, while ”Any“ represents the universal generation
supporting text, image, video, and audio modality. ”-“ indicates that the
information is not available in the official paper.

Method | Venue | Task | LLM | Generative Model | Training Cost
---|---|---|---|---|---
Multimodal LLMs with image generation
FROMAGe [153] | ICML 2023 | TI$\rightarrow$TI | OPT | Retrieval | 1$\times$A6000, 24Hrs
GILL [154] | NeurIPS 2023 | TI$\rightarrow$TI | OPT | Retrieval/SD | 2$\times$A6000, 48Hrs
SPAE [155] | NeurIPS 2023 | Tokenization | PaLM2/GPT-3.5 | CNN | -
Emu [147] | ICLR 2024 | TI$\rightarrow$TI | LLaMa | SD | -
SEED [56] | ICLR 2024 | Tokenization | OPT | SD | 64$\times$V100, 44Hrs
CM3Leon [156] | arXiv 2023 | TI$\rightarrow$I, I$\rightarrow$T | CM3Leon | CM3Leon | 64$\times$A100
NExT-GPT [134] | arXiv 2023 | Any$\rightarrow$Any | Vicuna | SD etc. | -
DreamLLM [151] | ICLR 2024 | TI$\rightarrow$TI | Vicuna | SD | 128$\times$A800, 17.5Hrs
MiniGPT-5 [157] | arXiv 2023 | TI$\rightarrow$TI | Vicuna | SD | 4$\times$A6000
OpenLEAF [158] | arXiv 2023 | T$\rightarrow$TI | GPT-4 | SDXL | Training-free
Mini-DALLE3 [57] | arXiv 2023 | TI$\rightarrow$TI | GPT-3.5/GPT-4 etc. | SD-XL/DALLE-3 etc. | Training-free
EasyGen [61] | arXiv 2023 | I$\rightarrow$T,T$\rightarrow$I | FlanT5XL/Vicuna | BiDiffuser | 120$\times$A100 Hrs
TEAL [159] | arXiv 2023 | Any$\rightarrow$Any | LLaMa-Adapter | VQGAN | 8$\times$A100
LLMGA [160] | arXiv 2024 | T$\rightarrow$I | LLaVA-1.5 | SD | -
ChatIllusion [161] | arXiv 2023 | TI$\rightarrow$TI | LLaMa-AdapterV2 | SDXL | 4$\times$A6000, 80Hrs
CoDi-2 [58] | CVPR 2024 | Any$\rightarrow$Any | Llama 2 | SD | -
CAFE [162] | CVPR 2024 | T$\rightarrow$I | Llama 2 | SD | 10000$\times$A100 Hrs
StoryGPT-V [163] | arXiv 2024 | Story Generation | OPT/Llama2 | Char-LDM | -
ELLA [164] | arXiv 2024 | T$\rightarrow$I | Llama 2 | SDXL | 1344$\times$A100 Hrs
Lavi-Bridge [165] | arXiv 2024 | T$\rightarrow$I | Llama 2 | SD/PixArt-$\alpha$ | 8$\times$A100, 48Hrs
Image layout planning
LMD [63] | TMLR 2024 | T$\rightarrow$I | GPT-3.5/GPT-4 | SD | Training-free
LayoutGPT [64] | NeurIPS 2023 | T$\rightarrow$I | GPT-3.5/GPT-4/Codex | GLIGEN/SD | -
VP-GEN [166] | NeurIPS 2023 | T$\rightarrow$I | Vicuna | GLIGEN/SD | 4$\times$A6000, 48Hrs
Control-GPT [65] | arXiv 2023 | T$\rightarrow$I | GPT-4 | ControlNet/SD | -
LayoutLLM-T2I [66] | MM 2023 | T$\rightarrow$I | GPT-3.5 | GLIGEN/SD | -
LLM Blueprint [167] | ICLR 2024 | T$\rightarrow$I | GPT-3.5 | LMD | 1$\times$A100
SLD [168] | CVPR 2024 | T$\rightarrow$I | GPT-4 | DALLE3/SD | Training-free
TextDiffuser-2 [169] | arXiv 2023 | T$\rightarrow$I | Vicuna | SD | 8$\times$A100, 168Hrs
COLE [170] | arXiv 2023 | T$\rightarrow$I | Llama 2/LLaVA | IF | -
Prompt synthesis and refinement
SUR-Adapter [171] | MM 2023 | T$\rightarrow$I | LLaMa | SD | -
ChatGenImage [172] | arXiv 2023 | Data Synthesis | GPT-3.5 | SD | 1$\times$GTX3090
SwitchGPT [173] | arXiv 2023 | T$\rightarrow$TI | Llama 2/GPT-3.5 | SD | 4$\times$A100, 3Hrs
TIAC [174] | arXiv 2023 | T$\rightarrow$I | GPT-3.5 | SD | -
Idea2Img [175] | arXiv 2023 | T$\rightarrow$I | GPT-4V | IF/SD | Training-free
WordArt Designer [176] | EMNLP 2023 | T$\rightarrow$I | GPT-3.5 | ControlNet | 1$\times$V100
Image quality evaluation
DreamSync [177] | arXiv 2023 | T$\rightarrow$I | PaLM2/TIFA | SD-XL | -
LLMScore [178] | NeurIPS 2023 | T$\rightarrow$I | GPT-4 | SD | -

TABLE II: Image-language Datasets that can be adopted for language-based image
generation.

Name | Date | Venue | Org | Domain | Source | #Images | Caption
---|---|---|---|---|---|---|---
Im2Text [179] | 12 Dec 2011 | NeurIPS 2011 | SBU | Open | Internet | 1M | Manual
Microsoft-COCO [180] | 1 May 2014 | ECCV 2014 | Microsoft | Common Objects | Internet | 328K | Manual
ALIGN [181] | 11 Feb 2021 | ICML 2021 | Google | Open | Internet | 1.8B | Manual
Conceptual 12M [182] | 17 Feb 2021 | CVPR 2021 | Google | Open | Internet | 12M | Manual
WIT [183] | 2 Mar 2021 | SIGIR 2021 | Google | Open | Wikipedia | 11.5M | Manual
LAION-400M [184] | 3 Nov 2021 | NeurIPS 2021 | LAION | Open | Internet | 400M | Manual
LAION-FACE [185] | 6 Dec 2021 | CVPR 2022 | Microsoft | Face | LAION | 20M | Manual
M3W [186] | 29 Apr 2022 | NeurIPS 2022 | Deepmind | Interleave | Internet | 43M | Manual
LAION-COCO [187] | 15 Sep 2022 | - | LAION | Open | LAION | 600M | Synthetic
LAION-5B [188] | 16 Oct 2022 | NeurIPS 2022 | LAION | Open | Internet | 5B | Manual
Coyo-700M [189] | 31 Aug 2022 | - | Kakao Brain | Open | Internet | 700M | Manual
KOSMOS-1 [190] | 27 Feb 2023 | NeurIPS 2023 | Microsoft | Interleave | Internet | 355M | Manual
Multimodal C4 [191] | 14 Apr 2023 | NeurIPS 2023 | UCSB | Interleave | Internet | 571M | Manual
LLaVA-instruct [192] | 17 Apr 2023 | NeurIPS 2023 | UWM | Instruction | COCO | 150k | Synthetic
DATACOMP [193] | 27 Apr 2023 | NeurIPS 2023 | DATACOMP | Open | Internet | 12.8B | Manual
MARIO-10M [194] | 19 May 2023 | NeurIPS 2023 | Microsoft | Text within image | LAION, TMDB, OpenLibrary | 10M | Manual
LAION-Glyph [195] | 29 May 2023 | NeurIPS 2023 | Microsoft | Text within image | LAION | 10M | Manual
MIMIC-IT [196] | 8 Jun 2023 | arXiv 2023 | NTU | Interleave | Internet | 2.8M | Synthetic

Figure 4: A generic pipeline of integrating image comprehension and generation
ability on LLMs [60, 56, 151, 59]. During inference time, users can input
interleaved multimodal data (e.g., text and images). The image tokenizer
processes the information into image tokens and feeds them into the LLM. LLM
outputs image tokens and then decodes them into textual responses and images.
Figure 5: Pipeline comparisons of (a) standard text-to-image (T2I) [13, 10],
(b) T2I with LLMs as layout planners [64, 66, 62, 63, 166, 65, 167], and (c)
T2I with LLMs for layout suggestions [170, 168].

##  4 Image Generation and Editing

###  4.1 Image Generation

Image generation has long been a fundamental task in the field of computer
vision, playing a vital role in various applications such as digital art,
entertainment, education, and communication [197, 198, 199]. In the beginning
stage of image generation, the generated content is limited to a specific
category, such as human faces, cats, or buildings. Recent advancements in
image generation have been particularly notable due to the introduction of
text guidance and open-domain generation. Most recently, the power of LLMs has
elevated image generation to a new level, enabling interactive or interleaved
generation. In Fig. 3, we summarize the history and development trajectory of
image generation in detail. A curated list of recent image generation methods
is shown in Table I. We also list representative datasets for image generation
in Table II. These works allow for the creation of images that closely align
with textual prompts, providing a powerful tool for visualizing ideas with
creativity.

####  4.1.1 Text-guided Image Generation with CLIP

Previously, the adoption of image-text alignment models, such as CLIP [23],
has played a crucial role in the development of text-guided image generation
[12, 169, 200, 10, 201, 202]. The alignment capabilities of CLIP text encoder
ensure that the generated image aligns with the given textual prompt,
resulting in images that accurately match the intended descriptions, including
desired objects, scenes, or attributes. Given the significant strides made by
CLIP in generating realistic images, it is natural to question: will more
powerful LLMs further benefit the domain of image generation?

It is worth mentioning that, the application of LLMs for the image domain has
been extensively investigated, especially for image understanding. LLMs can
effectively serve as unified processors of visual tokens and language tokens
[203, 204, 205, 206, 207, 142, 208, 209, 9, 210, 211, 212, 134, 213, 8, 214,
215, 216], tool coordinators [217, 103, 218, 219], or analysts of upstream
visual model outputs [220, 221, 222]. Inspired by these works, numerous works
further take advantage of LLMs for image generation, and the milestone works
are shown in Figure 3. In the following, we will introduce the progress
achieved in the task of image generation after the emergence of LLMs.

####  4.1.2 Text-guided Image Generation with LLMs

As demonstrated in Fig. 4, MLLMs have emerged as a transformative extension of
LLMs, addressing the inherent limitations of LLMs in handling visual content.
While LLMs excel in flexible text-based interactions, they are confined to
textual inputs and outputs. The introduction of MLLMs stems from the necessity
to bridge this gap and enable language models to comprehend and generate
images. MLLMs offer a two-fold advantage: firstly, they serve as a unified
interface for understanding and generating both textual and visual
information, providing users with a seamless integration of language and
images. Secondly, MLLMs introduce interactive generation capabilities,
allowing users to send commands to modify image content iteratively. This
interactive process empowers users with greater control over the image
generation process, enhancing the overall user experience and controllability
towards user-desired content.

Specifically, CM3Leon [156] is an autoregressive MLLMs designed for the
simultaneous generation of textual and image outputs. Operating within a
decoder-only, retrieval-augmented, token-based framework, CM3Leon offers a
unique approach to multimodal language processing. DreamLLM [151] presents the
first MLLM capable of generating free-form interleaved content, supporting
multi-round dialogue, and achieving remarkable results in image captioning and
video question answering (VQA) without the need for fine-tuning. The entire
framework is trained on interleaved multimodal contents in a truly end-to-end
manner. SEED-Lamma [59], similar to DreamLLM, enables LLMs to comprehend
multimodal instructions and supports multi-turn in-context image and text
generation. Notably, SEED-Lamma emphasizes the design of the Image Tokenizer,
proposing two crucial design principles for its functionality. MiniGPT-5 [157]
introduces visual tokens (referred to as ”voken”) to traditional LLMs,
enabling them to generate images. A two-stage training pipeline is proposed,
involving an unimodal alignment stage and a multimodal learning stage,
allowing LLMs to organically generate both text and images. OpenLeaf [158]
utilizes prompting of LLMs to generate interleaved textual and visual data,
producing entity and style-consistent images and text. It supports various
tasks such as how-to question answering, story-telling, graphical story
rewriting, and webpage/poster generation. EasyGen [61] leverages a
bidirectional conditional diffusion model, BiDiffuser, to endow LLMs with
multimodal understanding and generation capabilities. Unlike previous Clip-
based approaches, EasyGen generates images based on this model. TEAL [159]
utilizes existing tokenizers for different modalities and transforms the
obtained tokens into a joint embedding space, enabling frozen LLMs to
understand and generate across various modalities, including text, image, and
audio. ChatIllusion [161] introduces Genadapter and LLaMa-AdapterV2 to bridge
the hidden embedding space of SD XL and enable LLMs to understand visual
instructions and generate interleaved images and text, supporting image
generation, editing, and storytelling. Emu2 [60] emphasizes the in-context
learning capability of MLLMs, showcasing improved performance through scaling
up the model and unified autoregressive training. It supports tasks such as
visual prompting and object-grounded generation, achieving state-of-the-art
results in question-answering benchmarks and open-ended subject-driven
generation after instruction tuning. ELLA [164] and Lavi-Bridge [165]
incorporate large language models into the T$\rightarrow$I generation
architecture by training several lightweight adapters. LLMGA [160] utilizes
LLaVA to simultaneously encode images and instruction, enabling manipulating
images based on stable diffusion. StoryGPT-V [163] takes advantage of LLM for
coherent story script generation.

####  4.1.3 Image Layout Planning via LLMs

Despite the rapid development of T$\rightarrow$I generation, there remain
several challenging issues that have yet to be fully addressed, including text
rendering, spatial relationships, and quantity representation. Under this
circumstance, some methods seek to utilize LLMs for layout planning and
subsequently generate images based on the obtained layouts (please refer to
Fig. 5). LayoutGPT [64] leverages the inherent reasoning capabilities in LLMs
to facilitate layout generation through contextual demonstrations. It utilizes
GPT3.5/4 to transform user prompts to CSS-style output layouts, in which the
position of each object is specified. LMD [63] improves T$\rightarrow$I
diffusion models by enhancing prompt understanding capabilities. It utilizes a
two-stage approach, leveraging a pre-trained language model to generate scene
layouts and guide image generation. VP-GEN [166] breaks down the T2I task into
object/count generation, layout generation, and image generation steps. By
leveraging GPT-3.5-Turbo fine-tuned on text-layout pairs, VPGEN achieves
better spatial control than end-to-end models. Control-GPT [65] takes
advantage of GPT-4 to output TikZ code to construct the sketch layouts
according to the text descriptions. LayoutLLM-T2I [66] utilizes ChatGPT to
induce the layout based on the user prompt. A prompt encoder module is then
employed to model the text prompt, relation triplets, and the induced layout
separately. To efficiently integrate the layout information, a Layout-aware
Spatial Transformer based on the UNet is introduced. LLM Blueprint [167]
utilizes ChatGPT to produce detailed object descriptions, bounding box
layouts, as well as background prompts. Subsequently, iterative refinement
operations are conducted to remedy regional errors based on the layouts. SLD
[168] improves T$\rightarrow$I generation by iteratively generating images
from input prompts and correcting mistakes using an LLM-based layout planner.
In particular, the layout planner can add, delete, or resize object boxes to
help T2I models produce more accurate images. TextDiffuser-2 [62] employs
Vicuna-7B-1.5 for layout planning, generating the position and content of the
text to be rendered based on user-provided prompts. COLE [170] harnesses large
language models to transform user prompts into detailed JSON files. These
files encompass specifications, such as the content, position, and style, for
the text to be added.

####  4.1.4 Prompt Synthesis and Refinement via LLMs

LLMs can be treated as a huge knowledge base. Some methods [172, 175, 174,
176] have explored LLMs to synthesize or optimize prompts, guiding
T$\rightarrow$I (T2I) models to generate images with rich and detailed
content. For example, ChatGenImage [172] utilizes ChatGPT to generate prompts,
directing AIGC models in generating preliminary images. Subsequently, it
iteratively refines these prompts by incorporating automatically generated
detailed annotations as local constraint prompts, resulting in the production
of diverse and intricate scenes. Inspired by the three-layer artwork theory,
TIAC [174] and WordArt Designer [176] use LLMs to translate abstract concepts
into semantically relevant physical objects, making it easier for downstream
T$\rightarrow$I models. Idea2Img [175] employs a Multimodal LLM to evaluate
the generated images of T$\rightarrow$I models. Subsequently, based on the
obtained feedback, the framework iteratively refines the initial prompts to
generate satisfactory results. DiffusionGPT [223] leverages LLMs to refine
prompts for image generation. By parsing diverse prompts and utilizing domain-
specific Trees-of-Thought, the model selects the most suitable generative
model to generate high-quality images. RPG [224] is a training-free framework
for T$\rightarrow$I generation. It leverages multimodal LLMs to refine the
original prompt, decomposing complex prompts into subregion tasks, and
achieves superior performance in object composition and text-image alignment.
SUR-adapter [171] leverages LLMs to improve its semantic understanding and
reasoning capabilities, enabling it to create better textual semantic
representations for T$\rightarrow$I generation. SwitchGPT [173] introduces an
innovative framework that enables traditional LLMs, like GPT, to interpret the
underlying intent of the given instruction, thus producing a more suitable
response non-text outputs.

####  4.1.5 Image Quality Evaluation via LLMs

A few works focus on using large language models to evaluate the quality of
generated images. For example, DreamSync [177] employs two vision language
models (VLMs) to evaluate the generated results, and select the best-generated
image: one for text alignment and another for aesthetic quality. LoRA [225] is
then used to iteratively fine-tune the T2I model towards the selected best
generations. LLMScore [178] converts images into image-level and object-level
visual descriptions. Subsequently, a set of instructions is given to the LLMs
to check how closely between the images and the descriptions. Finally, a score
is generated with reasons.

###  4.2 Image Editing

Image editing is a closely related task with generation and thus receives
remarkable progress with the development of image generation models. A curated
list of recent image editing methods is shown in Table III.

####  4.2.1 Image Editing with CLIP/T5

CLIP model enables language-based image editing [226, 227, 228, 229, 230, 231,
232, 233, 234, 235, 236, 237, 238, 239]. PAIR-Diffusion [236] identifies
structure and appearance as the two most intuitive aspects of image editing.
Thus, the PAIR-Diffusion is trained using structure and appearance information
explicitly extracted from the training images, which enables editing of the
structure and appearance separately during inference. Instead of relying on
the heavy training process of diffusion models on large-scale datasets,
another line of work [231, 234] slightly tunes the pre-trained diffusion
models to edit the target images. Imagic [231] proposes a generic model for
text-based image editing. Notably, Imagic is free of source prompts, which is
achieved by optimizing the target prompt embedding. After that, the entire
diffusion model is fine-tuned for better reconstruction performance. The
editing ability is achieved by interpolating the optimized prompt embeddings
and the target prompt embeddings. There also exist many tuning-free methods
for text-guided image editing [226, 227, 228, 229, 230, 232, 233, 235, 237,
238, 239]. SDEdit [226] can generate realistic images from user inputs such as
strokes, sketches, or masks, as well as edit existing images with text
instructions. It works by first adding noise to the target image and then
gradually denoising it with text instructions. The denoising process is guided
by a diffusion model generative prior, which is trained on a large-scale image
dataset. SDEdit outperforms state-of-the-art GAN-based methods on multiple
image synthesis and editing tasks, according to a human perception study.
Other tuning-free methods [235, 237, 229, 232] achieve text-guided editing
through the regularization or manipulation of latents, cross-attention maps,
or UNet features. However, most text-guided image editing works rely on the
CLIP model, whose capability limits the editing to simple text prompts and
cannot understand complex human instructions.

TABLE III: An overview of language-based image editing methods based on CLIP
and LLMs. We summarize the involved LLMs and generative models, as well as
whether the method requires training or not.

Method | Venue | LLM | Generative Model | Training
---|---|---|---|---
CLIP for image editing
SDEdit [226] | ICLR 2022 | - | DDPM | ✗
DiffusionCLIP [227] | CVPR 2022 | - | DDPM | ✗
P2P [229] | ICLR 2023 | - | Imagen&SD | ✗
NTI [230] | ICLR 2023 | - | SD | ✗
Imagic [231] | CVPR 2023 | - | Imagen | ✓
PaP [232] | CVPR 2023 | - | SD | ✗
SINE [234] | CVPR 2023 | - | SD | ✓
pix2pix-zero [235] | SIGGRAPH 2023 | - | SD | ✗
PAIR-Diffusion [236] | arXiv 2023 | - | PAIR-Diffusion | ✓
MasaCtrl [237] | SIGGRAPH 2023 | - | SD | ✗
Dragondiffusion [240] | ICLR 2024 | - | SD | ✓
DiffEditor [241] | CVPR 2024 | - | SD | ✓
LLMs for image editing
InstructPix2Pix [242] | CVPR 2023 | GPT-3 | SD | ✓
VisualChatGPT [217] | arXiv 2023 | GPT-3 | SD | ✓
CHATEDIT [243] | EMNLP 2023 | GPT-3 | StyleGAN | ✓
MGIE [244] | ICLR 2024 | LLaVA-7B | SD | ✓
Emu Edit [245] | arXiv 2023 | Llama 2-70B | Emu | ✓
SLD [168] | CVPR 2024 | GPT-4 | DALL-E 3 | ✓

TABLE IV: An overview of video editing methods based on CLIP and LLMs. We
summarize the involved LLMs and generative models, as well as whether the
method requires training or not.

Method | Venue | LLM | Generative Model | Training
---|---|---|---|---
CLIP for video editing
Tune-A-Video [246] | ICCV 2023 | - | SD | ✓
Dreamix [247] | arXiv 2023 | - | Imagen-video | ✓
Video-P2P [248] | arXiv 2023 | - | SD | ✓
FateZero [249] | ICCV 2023 | - | SD | ✗
Pix2Video [250] | ICCV 2023 | - | SD | ✗
StableVideo [251] | ICCV 2023 | - | SD | ✗
Rerender-A-Video [252] | SIGGRAPH Asia 2023 | - | SD | ✓
TokenFlow [253] | ICLR 2024 | - | SD | ✗
CoDeF [254] | CVPR 2024 | - | SD | ✓
MagicEdit [255] | arXiv 2023 | - | SD | ✓
MagicStick [256] | arXiv 2023 | - | SD | ✓
LLMs for language-based video editing
InstructV2V [257] | ICLR 2024 | GPT-3 | SD | ✗
InstructVid2Vid [258] | arXiv 2023 | GPT-3 | SD | ✗

####  4.2.2 Image Editing with LLMs

LLMs provide powerful chat-based or interactive editing capabilities for image
editing [259, 217, 243, 244, 245, 168].

InstructPix2pix [259] proposes to use LLMs to construct the data tuples
(original image, prompt, target image) to train a model that learns to edit
the images following the editing prompt. The model is based on a conditional
diffusion model that can process arbitrarily interleaved image and text inputs
and produce coherent image (and text) outputs. To generate the data tuples,
the authors leverage the knowledge of two large pretrained models: a language
model (GPT-3) and a T$\rightarrow$I model (Stable Diffusion). The language
model generates the editing instructions and the textual descriptions of the
edited images, while the T$\rightarrow$I model renders the edited images based
on the textual descriptions. The authors also introduce a mapping network that
translates the hidden representations of the language model into the embedding
space of the visual models, enabling the model to use the strong text
representations of the LLM for visual outputs.

CHATEDIT [243] further leverages LLMs to contribute an interactive facial
image editing system via dialogue. Specifically, CHATEDIT split the dialogue-
based editing problem into (1) user edit request tracking, (2) image editing,
and (3) response generation subtasks. The user edit request tracking module is
responsible for extracting the user’s editing intentions from the dialogue
history and updating them dynamically. The image editing module is based on a
conditional diffusion model that can process both image and text inputs and
outputs, and perform various editing operations such as changing hair color,
adding glasses, or removing wrinkles. The response generation module is
designed to generate natural and informative responses that reflect the
editing results and guide the user to the next step. CHATEDIT is evaluated on
a novel benchmark dataset proposed by the authors, which contains multi-turn
dialogues and corresponding facial images annotated with user edit requests.

MGIE [244] investigates the MLLMs to do the image editing task. The proposed
MGIE can learn to convert expressive human instructions to editing guidance.
The editing model is also trained to follow the editing guidance in an end-to-
end way. The effectiveness of the MGIE is verified on photoshop-style
manipulations, global photo optimization and local editing.

Emu edit [245] trained the image editing model in a multi-task way. The tasks
include region-based editing, free-form editing and other computer vision
tasks, where all tasks are formulated into generative tasks. Emu edit
leverages LLMs for instruction generation. Concretely, the authors provide the
LLMs with a task description, a few task-specific exemplars, and a real image
caption. Then, the LLM is expected to output an editing instruction, an output
caption for an ideal output image, and which objects should be updated or
added to the original image.

Different from the above works that utilize LLMs to provide editing
instructions, SLD [168] uses LLMs to correct the incorrect generation to
enable object-level image editing.

###  4.3 Image-language datasets

Image-language datasets play a crucial role in the training of T$\rightarrow$I
models, providing the foundational data necessary for these models to learn
how to generate accurate and relevant visual content from textual
descriptions. Over ten years ago, the IM2Text [179] project gathered a huge
collection of photos by searching through Flickr, a popular photo-sharing
website. They sifted through a massive amount of data and carefully picked out
one million images that had clear and directly related captions. MS-COCO [180]
collected images depicting complex everyday scenes with common objects in
their natural settings. The researchers included five written captions to
provide descriptive context. These captions offer a richer understanding of
the scene and the objects within it. In recent years, the academic community
has witnessed a proliferation of large-scale image-text datasets. Typically,
researchers curate these datasets by crawling the internet. For example,
LAION-5B [188] is a massive dataset that was collected by searching the
internet for image-text pairs. Using the CLIP model to filter the results, the
researchers ensured that the text was relevant to the images. This process
resulted in a dataset containing 5.85 billion image-text pairs. Additionally,
some researchers are digging into the LAION-5B to find specific types of
content. For example, the Mario-10M dataset focuses on pulling out parts of
the dataset where the images have text in them to study further. Meanwhile,
LAION-FACE [185] is all about images with faces. These specialized subsets
help researchers focus on particular areas within the massive collection of
image-text pairs. Further, to help image generators follow instructions during
a conversation, the LLaVA Visual Instruct 150K dataset [203] comprises a
collection of multimodal data designed for instruction-following tasks,
generated via a GPT model.

TABLE V:  Overview of existing methods leveraging LLMs for language-based
video generation. We divide these methods into four categories: multimodal
LLMs for video generation, video layout planning, and temporal prompt
generation. In each method, we summarize the input-output of the task, the
involved LLM, and the generative model. In the ”Task“ column, the ”T“ and ”V“
are the abbreviations of ”text“ and ”video“, respectively, while ”Any“
represents the universal generation supporting text, image, video, and audio
modalities. Tokenization is the task of converting video into discrete video
tokens, which can be viewed as a submodule of some video generation pipelines.

Method | Venue | Task | LLM | Generative Model
---|---|---|---|---
Multimodal LLMs for video generation
VideoPoet [69] | arXiv 2023 | Any$\rightarrow$V | VideoPoet | VideoPoet
MAGVIT-v2 [70] | ICLR 2024 | Tokenization | BERT | BERT
Video-LaVIT [70] | arXiv 2024 | TIV$\rightarrow$TIV | Llama 2-7B | SVD img2vid-xt
Video layout planning
Dysen-VDM [71] | CVPR 2024 | T$\rightarrow$V | GPT-4 | Text2Video-Zero
VideoDirectorGPT [72] | arXiv 2023 | TI$\rightarrow$V | GPT-4 | LayoutVid
LVD [63] | ICLR 2024 | T$\rightarrow$V | GPT-3.5/GPT-4 | DSL-grounded generator
GPT4MOTION [73] | arXiv 2023 | T$\rightarrow$V | GPT-4 | SDXL / ControlNet
FlowZero [74] | arXiv 2023 | T$\rightarrow$V | GPT-4 | Gligen
Temporal prompt generation
DirecT2V [75] | arXiv 2023 | T$\rightarrow$V | GPT-4 | Text2Video-Zero
Free-Bloom [76] | NeurIPS 2023 | T$\rightarrow$V | GPT-3.5 | LDM
InterControl [77] | arXiv 2023 | T$\rightarrow$V | GPT-4 | HMDM
PRO-Motion [78] | arXiv 2023 | T$\rightarrow$V | GPT-3.5 | Posture-Diffuser
VideoDrafter [79] | arXiv 2024 | T$\rightarrow$V | ChatGLM-6B | SD-XL

##  5 Video Generation and Editing

###  5.1 Video Generation

While video understanding has been thoroughly investigated [260, 261, 262,
263, 264, 265, 266, 267, 268, 269, 270, 271], the past two years have
witnessed the rapid development or video generation. Particularly in the field
of text-based video generation, numerous works have achieved remarkable
results. We list the milestone works in Fig. 6 including both Clip/T5-based
and LLM-based approaches, and summarize the key technical components in Table
V and commonly-used video-language datasets in Table VI.

####  5.1.1 Text to Video Generation with CLIP

Based on the type of generative models, there are two main paradigms: one is
based on diffusion models, and the other is based on transformer models
combined with discrete codebooks trained by predicting the next token.
Diffusion models have become a mainstream paradigm due to their ease of
training. Within the diffusion framework, there are pixel-level video
diffusion models [272, 16, 17] and latent-level video diffusion models [14,
15, 273, 274]. Pixel-level approaches exhibit better text alignment but
require substantial computational resources. On the other hand, latent-level
models are more efficient as they reduce redundancy in video data. Different
works in video generation have different focuses. Some emphasize
photorealistic or high-definition outputs, aiming to improve the quality of
generated videos. Others focus on controllable generation, such as image-to-
video approaches that enable local control over motion regions, trajectory
control for object and camera movement directions, and the use of sketch,
depth, and pose to control structure. Some works concentrate on generating
longer videos or exploring better network architectures.

####  5.1.2 Text to Video Generation with LLMs

Recently, several works have also taken advantage of multimodal LLMs [69, 70]
for the task of video generation. For example, VideoPoet [69] leverages a pre-
trained auto-regressive transformer model to handle multimodal data for
synthesizing videos with temporal consistency and high-motion fidelity. It
adapts LLMs training techniques, allowing for task-specific video generation,
including tasks such as text-to-video and image-to-video conversion. Another
work MAGVIT-v2 [70] explores the tokenization technique of Multimodal LLMs. It
transforms visual inputs into discrete tokens, enhancing Large Language
Models’ performance on image and video generation tasks. It outperforms
diffusion models on benchmarks like ImageNet and Kinetics, provides video
compression on par with advanced codecs, and improves action recognition
through effective representation learning.

####  5.1.3 Video Layout Planning via LLMs

Numerous studies have validated the efficacy of LLMs in generating image
layouts, and similarly, recent research has sought to explore the potential of
LLMs in crafting video layouts. For instance, some methods employ LLMs to
sequentially generate bounding boxes for objects in each frame to assist in
the video generation process. VideoDirectorGPT [72] utilizes a language model
to generate a video plan that includes object-bound boxes for each scene.
These bounding boxes provide spatial coordinates for the entities, which are
used to maintain object consistency and precise layout control throughout the
video generation process. LLM-grounded Video Diffusion (LVD) [63] enhances
video generation by first using an LLM to create detailed scene layouts from
textual prompts, capturing complex motions. These layouts then guide a
diffusion model through adjusted attention maps to produce videos that
accurately reflect the prompted actions and dynamics, improving upon existing
video generation methods. FlowZero [74] utilizes LLMs to generate a dynamic
scene syntax that includes object bounding boxes, which are critical for
defining object positions and movements within each frame. These bounding
boxes guide the diffusion model to ensure accurate object placement and
coherent motion throughout the video. Another line of works attempts to
utilize layouts beyond conventional bounding boxes. Dysen-VDM [71] enhances
the quality of generated videos by employing a dynamic scene graph (DSG). The
DSG is used to capture and organize the temporal dynamics of actions described
in the text, which are then enriched with details and integrated into a
diffusion model to produce more dynamic and realistic videos. GPT4Motion [73]
uses GPT-4 to script physical scenes in Blender. The simulated scenes are
transformed into intermediate representations, such as depth maps, to serve as
layout conditions. Then these conditions are fed to Stable Diffusion to
produce the final video, ensuring motion consistency and efficiency in
scenarios like object interactions and fluid dynamics.

####  5.1.4 Temporal Prompt Generation via LLMs

In contrast to image generation, video generation requires more elaborate and
dense descriptions. The refinement and expansion of prompts can be facilitated
by leveraging the capabilities of LLMs. For instance, DirecT2V [75] improves
narrative consistency and scene composition from user prompts. It employs
instruction-tuned large language models to decompose a single user prompt into
detailed frame-by-frame descriptions. These descriptions guide the generation
of each video frame, allowing for the seamless integration of time-varying
elements and coherent storytelling. Free-Bloom [76] leverages large language
models LLMs to create a sequence of semantically coherent prompts, guiding the
video’s narrative flow. Pre-trained latent diffusion models (LDMs) are then
used as animators to generate high-fidelity frames that visually represent the
evolving semantic content, like the process of a flower blooming. InterControl
[77] uses an LLM Planner to convert textual interaction descriptions into
detailed contact plans, improving the generation quality of motion videos.
PRO-MOTION [78] uses LLMs to create a sequence of scripts detailing the key
postures needed for the target motion. These scripts are based on simple
templates, which are different from natural languages and are designed to
comprehensively describe all possible postures, thereby simplifying the
process for the subsequent generation process. VideoDrafter [79] utilizes LLMs
to convert an input prompt into a detailed multi-scene script. This script
capitalizes on the logical knowledge of LLMs to ensure the scenes make sense
in sequence.

Figure 6:  Milestone works of Clip/T5-based and LLM-based language-guided
video generation.

###  5.2 Video Editing

####  5.2.1 Text-guided Video Editing with CLIP/T5

CLIP [23] enables language-based video editing. Here, we mainly discuss the
diffusion-model-based video editing approaches. Tune-A-Video [246] presents
the early attempt at text-guided video editing with pre-trained diffusion
models. Instead of training a giant video diffusion model, Wu et al. [246]
propose to inflate and tune the pre-trained text-to-image diffusion model on
the target video in a one-shot way. After tuning, the inflated diffusion
models support versatile video editing capabilities. Despite its simplicity,
Tune-A-Video [246] shows poor temporal stability and the limitation of keeping
the unrelated regions unaltered. Video-P2P [248] and FateZero [249] utilize
better inversion techniques and propose manipulating attention maps to keep
the background unchanged when editing. Pix2Video [250] adopts editing the key
frame first and then propagates the editing to other frames, which achieves
improved temporal consistency and longer video editing capabilities. Different
from these works, Rerender-A-video [252] and CoDeF [254] focus their
applications on video-to-video style translation and achieves impressive
results through optical-flow-based regularization [252] or adoption of
deformable content field [254]. More recently, the developments of CLIP-based
video editing are towards better temporal consistency [253, 251, 255], more
controllable [256], and more computational effective [275]. We list recent
representative video editing works in Table IV.

####  5.2.2 Text-guided Video Editing with LLMs

Existing works that utilize LLMs for video editing are relevantly limited.
Current LLMs-based video editing follows a similar scheme as the
InstructPix2pix [259], i.e., using LLMs to construct the training data more
efficiently.

InstructVid2Vid [258] is one of the works that involve LLMs inefficient
training data construction. The method uses an LLM model to generate synthetic
video-instruction pairs, which are then used to train an editing model for
controllable video editing with natural language instructions. InstructVid2Vid
leverages the pre-trained image generation model, i.e., stable diffusion, and
a conditional 3D Unet to produce high-quality and temporally coherent videos
that match the input video and instruction. To improve the diversity and
realism of the synthetic data, InstructVid2Vid incorporates the knowledge and
expertise of different models, such as ChatGPT, BLIP, and Tune-a-Video, to
synthesize various instructions for the same video. The paper demonstrates the
effectiveness of using LLMs to synthesize training data for complex and
creative tasks, such as attribute editing, background change, and style
transfer.

InsV2V [257] is another approach that extends the paradigm of InstructPix2Pix
to the video editing domain. InsV2V uses a large language model to construct
synthetic data for training a video editing model, which can also follow
natural language instructions to edit videos. InsV2V adopts a one-model-all-
video strategy, eliminating the need for per-video-per-model fine-tuning or
inversion, and simplifying the user interaction by only requiring an editing
prompt. InsV2V leverages a pre-trained image generation model, Stable
Diffusion, and a conditional 3D U-Net architecture to produce high-quality and
temporally coherent videos that match the input video and instruction. InsV2V
demonstrates the versatility and effectiveness of using LLMs to synthesize
training data and perform text-based video editing for various tasks, such as
object replacement, style transfer, and background change.

###  5.3 Video-language Datasets

The availability of captioned video datasets is crucial for text-to-video
generation. To address this challenge, MSR-VTT [276] has introduced a large-
scale open-domain video-language dataset that encompasses a wide range of
categories and diverse content, setting a new benchmark for the video
understanding task in 2016. It contained 200k clip-language pairs from 10K
videos on the web, and every video had 20 human annotations in English. Anna
et al. have presented the large-scale movie description challenge (LSMDC)
[277], which consists of 202 movies accompanied by transcribed audio
descriptions. These descriptions provide a narrative of the significant events
depicted in the visual video. In natural videos, multiple events often occur
within a single video. For instance, a video may feature a man playing the
piano, a girl singing, and a crowd applauding. To identify and describe each
event, Ranjay et al. have proposed the ActivityNet Caption [278] benchmark,
which involves detecting events, describing them using natural language, and
localizing them with start and end times. How2 [279] and VATEX [280] are
multilingual video description datasets. How2 is a large-scale instructional
multimodal and multilingual video dataset that includes English and Portuguese
descriptions, videos, speech, and English video-level summaries. VATEX
comprises both English and Chinese descriptions, covering 600 human
activities. HowTo100M [281] has introduced an automatic video captioning
method that utilizes transcribed narrations from web videos instead of manual
labeling, enabling fast and scalable data collection. Jonathan et al. have
observed that web videos often come with text metadata such as titles and
descriptions. They have proposed a data collection process and gathered 70M
video clips, known as WTS70M [282], using metadata including titles,
descriptions, tags, and channel names. The WebVid [283] dataset has been
created for the text-to-video retrieval task. Recognizing the noise in
previous datasets like HowTo100M, WebVid-2M, and WebVid-10M have been
collected from the internet, featuring weak captions. YT-Temporal-180M [284]
is a dataset that contains a diverse corpus of frames, where ASR is provided
from a filtered set of 6 million YouTube videos, serving as a resource for
multimodal representation learning. HDVILA [285] is a high-resolution large-
scale dataset comprising 370k videos, covering 15 popular YouTube categories
and offering diverse video content. VideoCC3M [286] proposes a method to
transfer captions from existing image captions in CC3M, creating a new weakly
labeled audio-video captioning dataset. VideoFactory [287] introduces HD-
VG-130M, a dataset consisting of 130M high-definition, widescreen, and
watermark-free text-video pairs. InternVid [288] presents a scalable approach
to building a high-quality video-text dataset. They employ a multi-scale
approach that leverages Tag2Text, LLM, and BLIP2 to generate video captions.
Panda-70M [289] is a high-quality and large-scale captioned video dataset
proposed in 2024. It contains 70M video clips from YouTube videos, and the
caption is extracted via multiple teacher models to obtain multiple captions
for one video and a well-trained caption retrieval model to select the best
caption. Vript [290] is a fine-grained video-text dataset proposed in 2024
containing 12K annotated videos. Although the number of videos is limited, the
caption of each video is fine-grained and comprises information on the shot
type, camera movement, content, and scene title.

###  5.4 Summary

In Sec.5, we have introduced the research works on the generation and editing
of the video modality. For each task, we divided the papers into two groups:
Clip/T5-based methods and LLMs-based methods to highlight the advancement
brought by LLMs. We summarize the key technical components of LLMs-based
approaches in Table V, the development of milestone works in the task of
language-guided video generation in Fig.6, and the related video-language
dataset in Table VI

TABLE VI:  Public video-language datasets can be adopted for language-guided
video generation. For each dataset, we list the following information in each
column: dataset name (Dataset), paper conference venue (Venue), dataset domain
(Domain), video source (Vid. Source), video spatial resolution (Res.), average
duration per clip (Dur./Clip), total number of clips ($\\#$Clips), total
number of videos ($\\#$Videos), total number of hours ($\\#$Hours), and
caption source (Cap. Source). The dataset is sorted in ascending order of its
released time.

Dataset | Venue | Domain | Vid. Source | Res. | Dur./Clip | #Clips | #Videos | #Hrs | Cap. Source
---|---|---|---|---|---|---|---|---|---
MSR-VTT [276] | CVPR 2016 | Open | Internet | 240p | 15s | 10K | 7K | 41 | Human
LSMDC [277] | IJCV 2017 | Movie | Amazon | 1080p | 4.8s | 118K | 202 | 158 | Audio Desc.
ActivityNet Captions [278] | ICCV 2017 | Activity | Internet | - | 36s | 100K | 20K | 849 | Human
How2 [279] | NIPS 2018 | Instruction | Youtube | - | 90s | 80K | - | 298 | Human
VATEX [280] | ICCV 2019 | Open | Youtube | 240p | 10s | 41K | 41K | - | Human
HowTo100M [281] | ICCV 2019 | Instruction | Youtube | 240p | 4s | 136M | 1.2M | 134K | ASR
WTS70M [282] | arXiv 2020 | Open | Youtube | - | 10s | 70M | 70M | 194K | Metadata
WebVid-10M [283] | ICCV 2021 | Open | Internet | 360p | 18s | 10.7M | 10.7M | 52K | Alt-text
YT-Temporal-180M [284] | NeurIPS 2021 | Open | Youtube | - | - | 180M | 6M | - | ASR
HD-VILA-100M [285] | CVPR 2022 | Open | Youtube | 720p | 13.4s | 103M | 3.3M | 372 | Algorithm
CelebV-Text [291] | CVPR 2023 | Face | Internet | 5122+ | 14s | 70K | - | 279 | Human+Algorithm
VideoCC3M [286] | ECCV 2022 | Open | CC3M | - | 10s | 6.3M | - | 17.5K | CC3M
HD-VG-130M [287] | arXiv 2023 | Open | Youtube | 720p | - | 130M | - | - | Algorithm
InternVid [288] | ICLR 2024 | Open | Youtube | 720p | 12s | 234M | 7M | 760K | Algorithm
Panda-70M [289] | CVPR 2024 | Open | Youtube | 720p | 8.5s | 70.8M | - | 166.8K | Algorithm
Vript [290] | Github | Open | Youtube | 720p-2K | 11.7s | 400K | 12K | 1.3K | Algorithm

##  6 3D Generation and Editing

Figure 7: A generic pipeline of 3D generation with CLIP and LLMs. The CLIP-
based models optimize the 3D representation by minimizing the distance between
the rendering image and the text prompt. In order to better improve
interaction efficiency, LLMs-based methods try to transfer the language
outputs of LLMs to blender code or 3D representation (i.e. human motion)
directly. Some images are borrowed from [292, 81, 293]

Recent studies have focused on establishing a connection between 3D assets and
texts. There are two types of methods that can bring the text information for
3D assets, including LLMs and CLIP/T5 models. LLMs can make the output of 3D
generation and understanding iterative updates according to user requirements,
thereby facilitating highly effective human-computer interaction (i.e.,
guiding the generation of human motions based on language). Different from
LLMs, which directly influence the 3D assets in an interactive iterative
manner, the CLIP/T5 model integrates the features of both rendered images and
text, enabling the injection of textual information into the 3D assets. In
this section, we will explore various methods that leverage the CLIP model or
LLMs to guide the processes of 3D generation, editing, and understanding. The
generic pipeline is shown in Fig. 7. The corresponding overview of the 3D
generation and editing methods is shown in Table VII.

###  6.1 3D Generation

####  6.1.1 3D Generation with CLIP/T5

By leveraging the multi-modal representation capabilities of CLIP, researchers
have been able to guide the generation and editing of 3D assets using textual
descriptions or queries, thereby enabling more precise control and
customization. Specifically, CLIP [23] extracts the feature of image and text
with two encoders and aligns them in a contrastive learning manner. This
alignment builds the connection between image and text effectively which
brings significant improvements to text-guided 2D [294, 10, 12, 11, 13, 24],
text-guided 3D [24, 130, 41] generations. There are three typical methods to
utilize the CLIP model to provide text information during 3D generation: 1.
Adopting the pre-trained CLIP model as supervisor to calculate the CLIP loss
between the generation image and text or injecting the text feature to the 3D
assets with CLIP model directly. 2. Leveraging pre-trained text-to-image
generation models as supervisory signals and utilizing distillation loss
functions such as SDS [41, 295] to distill 3D assets [296, 297, 298]. 3.
Utilizing the public annotated 3D Datasets [293, 299, 300, 301] to get the
mesh, NeRF or multiview image, then set these outputs to train a 3D diffusion
model. As mentioned above, we will elaborate on the current methods in two
categories in the following content.

CLIP/T5 Model Supervisor.

Text2Mesh [26] focuses on stylizing 3D meshes by predicting color and local
geometric details that align with target text prompts. The entire process is
guided by a CLIP loss, which helps ensure the generated meshes conform to the
desired textual specifications. This method offers enhanced control over the
visual appearance and geometric attributes of 3D meshes, enabling the creation
of visually appealing and semantically meaningful shapes.

TANGO [302] proposes a pipeline for generating texture on a given mesh. By
leveraging the CLIP model, TANGO aligns the texture generation process with
textual descriptions, allowing for the synthesis of textured meshes that match
specific visual or semantic criteria. This approach facilitates the creation
of realistic and visually coherent 3D models with detailed surface textures.
CLIP-Mesh [25] addresses unsupervised text-guided 3D generation by optimizing
the texture, normal, and vertical position of a 3D object using the CLIP loss.
This approach enables the generation of 3D objects that align with textual
prompts, providing a powerful tool for text-driven content creation and
design. X-mesh [303] further improves the performance of CLIP-Mesh by adopting
an attention-based network, enhancing the fidelity and accuracy of the
generated 3D meshes. CLIP-forge [24] introduces a zero-shot text-to-shape
method for predicting volumetric occupancy by leveraging the text feature of
the CLIP model in conjunction with a conditional normalizing flow network
[304]. This approach enables the generation of 3D shapes based on textual
prompts without the need for explicit supervision or labeled training data.

Some methods employ CLIP guidance to output NeRF representations [296], which
are used to model complex scenes and capture high-frequency spatial
information. DreamFields [305] introduces general-purpose priors to assist in
aligning the optimized NeRF with the given text prompt, improving the quality
and fidelity of the generated scenes. CLIP-NeRF [130] takes a two-step
approach, first training a disentangled conditional NeRF and then utilizing
the text feature to adjust the parameters of the learned NeRF, enabling more
fine-grained control over the generated scenes.

ShapeGPT [306] utilizes a ”word-sentence-paragraph” pipeline to convert shapes
into words. These words are then assembled to form shape sentences, which can
be integrated with instructional text to create multi-modal paragraphs
describing 3D shapes. These multi-modal paragraphs help the ShapeGPT do
several applications, including text-to-shape generation, image-to-shape
generation, and multimodal-to-shape completion and editing.

Moving beyond object generation, MotionCLIP [307] proposes a 3D human motion
auto-encoder that predicts pose sequences. By leveraging CLIP, this method
enables the generation of realistic and contextually coherent human motion
based on textual prompts, providing a means for text-driven animation and
virtual character control. MotionGPT [308] considers human motion as a
distinct language and trains a motion-language model with the T5 model. The
approach incorporates discrete vector quantization to represent human motion
and transforms 3D motion into motion tokens. By establishing a comprehensive
”motion vocabulary,” the model conducts language modeling on both motion and
text in a cohesive manner.

While the aforementioned methods have achieved notable success in text-guided
generation, they still face challenges related to visual artifacts. This can
be attributed to the semantic-level nature of the CLIP loss, which tends to
reduce the high-frequency spatial information in the generated images.
Addressing this limitation remains an active area of research, with the aim of
further improving the visual quality and fidelity of text-guided 3D generation
approaches.

Text-to-image Model Supervisor.

In contrast to the aforementioned methods that directly utilize the CLIP model
or CLIP loss for 3D generation, recent approaches [43, 295, 36, 38, 309, 37,
31, 34, 40, 310, 311] have focused on distilling 3D assets from pre-trained
text-to-image generation models (i.e., Stable Diffusion [10]). These models
employ score distillation sampling (SDS) loss within the framework of
DreamFusion [41]. Specifically, text-to-2D image generation models utilize the
text feature from CLIP to train a generative model (i.e., diffusion model)
within a text-image pair dataset, SDS [41] simulates the training process of
diffusion model and calculates the spatial gradient of the rendered image to
optimize the parameters of NeRF. Many subsequent methods have utilized the SDS
loss function as a supervisory signal and introduced various techniques to
improve the performance of 3D generation.

For instance, Magic3D [42] combines DMTed [298] to transfer the NeRF model to
a mesh representation, enabling high-resolution rendering results. The
experimental results presented in the paper demonstrate the effectiveness of
Magic3D in enhancing the generation quality of both geometry and texture.
Similar to the Magic3D, TextMesh [40] replaces the NeRF model with a textured
mesh representation for 3D asset generation. LatentNeRF[312] introduces a
latent space optimization strategy. It takes advantage of the latent space
structure learned by the text-to-image model, which has already been trained
on a large-scale text-image dataset. By aligning the NeRF optimization process
with the latent space structure, LatentNeRF improves the convergence and
stability of the training process.

Fantasia3D [39] presents a novel approach to generating 3D assets by
separating the geometry and texture components. It first refines the normal
map to generate the geometry and then fixes the geometry to predict the color
field. Moreover, Fantasia3D set the color field as a Physically Based
Rendering (PBR) material model to enhance the generation fidelity. Moreover,
artists and designers can manipulate and modify the geometry and texture
components independently, allowing for a wider range of artistic expression
and customization possibilities.

Despite the significant improvements brought about by SDS, there remain some
limitations in the results generated. These include over-saturation, over-
smoothing, multi-face Janus artifacts, and time-consuming computations. To
address these issues and further enhance the generation performance, some
methods [313, 33, 32] have been proposed.

For example, Perp-Neg [314] debiases the score-distillation framework for
view-consistent text-to-3D generation, aiming to mitigate the multi-face Janus
problem. Prolificdreamer [35] introduces variational score distillation (VSD)
to avoid over-saturation and over-smoothing. It minimizes the KL divergence
between the optimized 3D assets and the target distribution, which makes the
generation process utilize the normal CFG weight. DreamPropeller [315]
presents a drop-in acceleration algorithm to expedite the training process. It
extends the concept of Picard iterations, a well-established algorithm for
parallel sampling of an ODE path, to encompass various scenarios beyond ODEs.
This includes accommodating momentum-based gradient updates and handling
dimensional changes during the optimization process, which are often
encountered in the context of 3D generation. Additionally, some methods [30,
29] have replaced the NeRF model with Gaussian splatting [28] techniques to
improve training efficiency and generate high-quality geometry.

Apart from the methods mentioned, which focus on general object generation,
some methods [316, 317, 318, 319, 320, 321, 322, 323, 324, 325, 326, 327, 328,
329, 330, 331, 332] try to explore 3D generation in 3D avatar generation.

3D Datasets Supervisor.

Some methods [333, 334, 335, 336] try to train the multi-view diffusion model
with diffusion loss or reconstruction model [337, 338, 339] for text-to-3D
generation, and these methods always adopt the text-to-3D dataset [293, 299,
300] as a training label.

Point-E [301] and Shape-E [340] utilize the Blender to collect a point cloud
3D dataset first, then train a diffusion model to generate the point cloud and
mesh with text condition, respectively.

MVDream [333], RichDreamer [336], SPAD [334] and UniDreamer [335] get the
multi-view images from the Objaverse dataset. With these multi-view images,
they Then, fine-tune the 2D text-to-image diffusion to a 3D diffusion model to
predict the multi-view images under the guidance of text and camera poses.

While CLIP-based 3D generation methods have demonstrated impressive progress,
the CLIP model itself cannot maintain flexibility in human-computer
interaction during the 3D generation process. More recently, Large Language
Models (LLMs) have been introduced to enhance human-computer interaction in 3D
generation, which will be discussed in the following section.

####  6.1.2 3D Generation with LLMs

The integration of LLMs with 3D assets has emerged as a promising research
direction recently. By harnessing the powerful language understanding
capabilities of LLMs, researchers aim to directly enhance the performance of
generation, manipulation, or understanding of 3D assets through textual
instructions. These approaches enable users to interact with the 3D
environment more intuitively and naturally, bridging the gap between human
cognition and computer-generated content.

3D-GPT [80] proposes a training-free framework that utilizes LLMs. This
framework consists of three agents: the task dispatch agent, the
conceptualization agent, and the modeling agent. By employing these agents,
3D-GPT can produce the code for blender corresponding to the language, and the
efficiency of end-users engaged in procedural 3D modeling is improved. Similar
to the 3D-GPT, SceneCraft [341] introduces an LLM agent to transfer input text
query into a 3D scene by generating a Blender script. Specifically, sceneCraft
has a dual-loop self-improving pipeline: in the inner-loop, per each scene, an
LLM autonomously writes a script to interact with Blender, receives rendered
images, and keeps improving the script until getting good scenes; in the
outer-loop, SceneCraft summarizes common functions over a batch of written
scripts to maintain a reusable design skill library.

LL3DA [82] proposed a Large Language 3D Assistant that utilizes a transformer
network to predict querying tokens. These tokens are projected to the prefix
of textual instructions as the input of a frozen LLM. Finally, the LLM will
produce the answer to the textual instructions.

PointLLM [342] processes colored point clouds using human instructions and
predicts responses to user questions with the help of LLMs. This enables users
to analyze and interpret point clouds more effectively.

3D-LLM [343] takes 3D points with features and language prompts as input and
performs various 3D-related tasks, leveraging the capabilities of LLMs. The
pipeline involved in this study involves the collection of a comprehensive
dataset comprising over 300,000 instances of 3D-language data. This dataset
encompasses a wide range of diverse 3D-related tasks, which include, but are
not limited to, 3D captioning, dense captioning, 3D question answering, task
decomposition, 3D grounding, 3D-assisted dialog, navigation, and various other
tasks.

In summary, the integration of LLMs and 3D assets has opened up new
possibilities for generating, manipulating, and understanding 3D content
through natural language instructions. These methods have demonstrated
significant advancements in enhancing human-computer interaction in the 3D
domain.

###  6.2 3D Editing

Similar to the generation, we separate the text-to-3D editing into two
aspects. And we will first discuss the CLIP/T5-based editing methods.

####  6.2.1 3D Editing with CLIP/T5

CLIP/T5 model Supervisor.

Blended-NeRF [344] introduced a framework for modifying a specific region of
interest within an existing NeRF scene using CLIP loss. This approach allows
for targeted editing of NeRF scenes by leveraging the power of contrastive
learning with CLIP. NeRF-Art [27] proposed a global-local contrastive learning
strategy to stylize pre-trained NeRF models. By employing contrastive
learning, NeRF-Art enables the creation of artistic and stylized renderings
through the manipulation of pre-existing NeRF scenes. TextDeformer [345]
employed text-to-geometry manipulation by introducing a mesh deformation
technique based on Jacobians. This approach utilizes text descriptions to
deform the geometry of objects, offering a novel way to manipulate and edit 3D
models. Sine [346] presented a prior-guided editing field that encodes fine-
grained geometric and texture modifications. By leveraging this approach, Sine
enables precise and detailed editing of both geometry and texture in 3D
scenes, providing a powerful tool for creative expression.

Text-to-image Model Supervisor.

SKED [347] utilizes sketches as a guiding input for text-to-image generation
models in SDS. By incorporating sketch information, SKED enhances the
generation process, resulting in more accurate and contextually aligned text-
to-image translations. DreamEditor [348] transfers NeRF representations to
meshes and employs a personalized text-to-image generation model called
DreamBooth for mesh editing. This approach allows for interactive and
personalized editing of meshes using text-based instructions, enabling users
to modify and shape 3D scenes according to their preferences. Instruct-
NeRF2NeRF [349] utilizes the InstructPix2Pix model to incorporate SDS loss for
editing NeRF scenes with text-based instructions. By combining the power of
text instructions with NeRF editing, this approach enables users to modify
scenes in a controlled and precise manner, enhancing the interactive editing
capabilities of NeRF. 3D Paintbrush [350] proposes a cascaded score
distillation method for editing the texture of local semantic regions in
meshes. By distilling scores, this approach allows for targeted and localized
texture editing, providing users with a powerful tool for enhancing the visual
appearance of specific regions in 3D models.

####  6.2.2 3D Editing with LLMs

Different from CLIP-based methods, there is no specific method to do the 3D
editing with LLMs. Editing is more like a sub-task of LLMs-based 3D
generation, so some generation methods (i.e., 3D-GPT [80], SceneCraft [341] )
can edit the 3D assets directly. We will follow the latest developments in
LLMs-based 3D editing and discuss them in the future.

###  6.3 Summary

The utilization of CLIP or LLMs in the context of 3D generation and editing
offers several advantages. Firstly, it enables users to express their creative
intentions or desired modifications in natural language, simplifying the
interaction process and reducing the need for specialized software or
technical expertise. Additionally, the incorporation of textual information
into the 3D generation pipeline enhances the interpretability and
explainability of the generated outputs, allowing users to better understand
and fine-tune the results according to their requirements.

In conclusion, the integration of CLIP or LLMs with 3D assets has opened up
new avenues for highly effective human-computer interaction. By aligning
textual information with the visual features of 3D assets, researchers have
been able to facilitate more intuitive and precise control over the generation
and editing of 3D content. These advancements hold great potential for
applications in fields such as computer graphics, virtual reality, and
augmented reality, offering enhanced user experiences and empowering users to
unleash their creativity more seamlessly and efficiently.

TABLE VII: Summary for 3D general object generation. The optimization target means the essential constraint during the learning process. The representation means the type of the 3D output. The method without an optimization target means this method is not guided by CLIP loss or SDS-based loss.  Method | Venue | Optimization Target | Representation | Guided Model
---|---|---|---|---
CLIP/T5 for 3D generation |  |  |  |
MotionCLIP [307] | ECCV 2022 | CLIP Loss | Motion Sequences | CLIP
MotionGPT [308] | NeurIPS 2023 | - | Motion Sequences | T5
MDM [351] | ICLR 2023 | - | Motion Sequences | CLIP
CLIP-Mesh [25] | SIGGRAPH Asia 2022 | CLIP Loss | Mesh | CLIP
TANGO [302] | NeurIPS 2022 Spotlight | CLIP Loss | Mesh | CLIP
DreamFields [305] | CVPR 2022 | CLIP Loss | NeRF | CLIP
Clip-forge [24] | CVPR 2022 | CLIP Loss | Voxel | CLIP
Text2Mesh [26] | CVPR 2022 | CLIP Loss | Mesh | CLIP
TextMesh [40] | 3DV 2023 | CLIP Loss | Mesh | CLIP
X-Mesh [303] | ICCV 2023 | CLIP Loss | Mesh | CLIP
ShapeGPT [306] | arXiv 2023 | - | SDF | T5
Shape-E [340] | arXiv 2023 | Diffusion Loss | Mesh/NeRF | CLIP
Point-E [301] | arXiv 2023 | Diffusion Loss | PointCloud | CLIP
DreamFusion [41] | ICLR 2023 Oral | Score Distillation | NeRF | Imagen
SJC [295] | CVPR 2023 | Score Distillation | NeRF | SD
Magic3D [42] | CVPR 2023 Highlight | Score Distillation | NeRF | SD
Perp-Neg [314] | arXiv 2023 | Score Distillation | NeRF | SD
Latent-NeRF [312] | CVPR 2023 | Score Distillation | NeRF | SD
Fantasia3D [352] | ICCV 2023 | Score Distillation | NeRF | SD
ATT3D [36] | ICCV 2023 | Score Distillation | NeRF | SD
ProlificDreamer [35] | NeurIPS 2023 Spotlight | Score Distillation | NeRF | SD
Text2Room [31] | ICCV 2023 | Score Distillation | Mesh | SD
3DFuse [43] | ICLR 2024 | Score Distillation | NeRF | SD
GaussianDreamer [29] | CVPR 2024 | Score Distillation | Gaussian Splatting | SD
DreamGaussian [30] | ICLR 2024 | Score Distillation | Gaussian Splatting | SD
NFSD [313] | ICLR 2024 | Score Distillation | NeRF | SD
MVDream [333] | ICLR 2024 | Diffusion Loss | Multi-view images | SD
RichDreamer [336] | CVPR 2024 | Diffusion Loss | Multi-view images | SD
SPAD [334] | CVPR 2024 | Diffusion Loss | Multi-view images | SD
UniDreamer [335] | CVPR 2024 | Diffusion Loss | Multi-view images | SD
Enhancing3D [353] | ICLR 2024 | Score Distillation | NeRF | SD
LucidDreamer [32] | CVPR 2024 | Score Distillation | Gaussian Splatting | SD
CSD [33] | ICLR 2024 | Score Distillation | NeRF | SD
SweetDreamer [34] | ICLR 2024 | Score Distillation | NeRF | SD
HiFA [38] | ICLR 2024 | Score Distillation | NeRF | SD
AToM [354] | arXiv 2023 | Score Distillation | Mesh | SD
Consistent3D [355] | arXiv 2023 | Score Distillation | Mesh/NeRF | SD
DreamControl [356] | CVPR 2024 | Score Distillation | NeRF | SD
IT3D [357] | AAAI 2024 | Score Distillation | NeRF | SD
Efficientdreamer [358] | CVPR 2024 | Score Distillation | NeRF | SD
GSGEN [359] | CVPR 2024 | Score Distillation | Gaussian Splatting | SD
X-Dreamer [360] | arXiv 2023 | Score Distillation | Gaussian Splatting | SD
HD-Fusion [361] | WACV 2024 | Score Distillation | Gaussian Splatting | SD
LODS [362] | arXiv 2023 | Score Distillation | Gaussian Splatting | SD
Sherpa3d [363] | CVPR 2024 | Score Distillation | NeRF | SD
DreamPropeller [315] | CVPR 2024 | Score Distillation | NeRF | SD
DreamPolisher [364] | arXiv 2024 | Score Distillation | Gaussian Splatting | SD
LLM for 3D generation |  |  |  |
3D-GPT [80] | arXiv 2023 | - | Blender Code | GPT-3.5
PoseGPT [81] | CVPR 2024 | - | Motion Sequences | LLaVA
HOLODECK [365] | CVPR 2024 | - | Scene | GPT-4
LL3DA [82] | arXiv 2023 | - | PointCloud | GPTV
SceneCraft [341] | arXiv 2023 | - | Blender Code | GPT-3.5
CLIP for 3D editing |  |  |  |
CLIP-NeRF [130] | CVPR 2022 | CLIP Loss | NeRF | CLIP
Blended-NeRF [344] | ICCVW 2023 | CLIP Loss | NeRF | CLIP
SKED [347] | ICCV 2023 | Score Distillation | NeRF | SD
DreamEditor [348] | SIGGRAPH Asia 2023 | Score Distillation | NeRF | SD
Instruct-NeRF2NeRF [349] | SIGGRAPH Asia 2023 | Score Distillation | NeRF | SD
TextDeformer [345] | TVCG 2022 | Score Distillation | Mesh | SD
SINE [346] | CVPR 2023 | Score Distillation | NeRF | SD
Blending-NeRF [366] | ICCV2023 | CLIP Loss | NeRF | CLIP
CustomNeRF [367] | CVPR 2024 | Score Distillation | NeRF | SD
Paint3D [368] | arXiv 2023 | - | Mesh | SD
3D Paintbrush [350] | arXiv 2023 | Score Distillation | NeRF | SD

##  7 Audio Generation, Understanding and Editing

Recently, a surge of innovative works, such as [95, 98, 104, 100, 84, 102],
has demonstrated the utilization of LLMs in various audio-related tasks. These
tasks span across domains including the creation of audio effects, speech
processing, and music composition, showcasing the versatility of LLMs.The
roles of LLMs in these areas are varied, acting as backbones for complex
systems [84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 96], conditioners for
specific tasks [97, 98, 99], labellers for audio content [100, 101, 102],
agents in interactive environments [103, 104, 105, 106, 107, 108], and as
inspiration for some approaches [53, 47, 109, 48, 110, 111]. This surge in the
application of LLMs in the audio field is not only reshaping how we interact
with sound and music but also opening new frontiers in the intersection of AGI
and audio technologies.

TABLE VIII: Audio datasets that can be adopted for language-based audio research. For each dataset, we list the following information in each column: dataset name (Dataset), paper conference venue (Venue), average duration per clip (Dur./Clip), total number of clips ($\\#$Clips), total number of hours ($\\#$Hours), and dataset domain (Domain). Dataset | Venue | Dur./Clip | #Clips | #Hours | Domain
---|---|---|---|---|---
MagnaTagATune [369] | ISMIR 2009 | 29s | 25,863 | 208h | Music
Librispeech [370] | ICASSP 2015 | - | - | 1,000h | Speech
Audioset [371] | ICASSP 2017 | 10s | 2M | - | Audio
MAESTRO [372] | ICLR 2019 | - | - | 200h | Music
Libri-TTS [373] | INTERSPEECH 2019 | - | - | 585h | Speech
MTG-Jamendo [374] | ICMLw 2019 | - | 55,000 | - | Music
Librilight [375] | ICASSP 2020 | - | - | 60,000h | Speech
Vggsound [376] | ICASSP 2020 | 10s | 210,000 | 550h | Audio
WenetSpeech [377] | ICASSP 2022 | - | - | 22,400h | Speech
Libri-heavy [378] | ICASSP 2024 | - | - | 50,000h | Speech
TABLE IX: Summary of approaches for LLMs-related audio tasks: Generation (G), Understanding (U), and Editing (E). We categorize the methods into five types according to the role of LLMs: LLMs as backbone, LLMs inspired backbone, LLMs as conditioner, LLMs as agent, and LLMs as labeller.  Task | Method | Venue | LLM Model | Domain
---|---|---|---|---
LLMs as backbone |  |  |  |
G, U | SongComposer [96] | arXiv 2024 | SongComposer | Audio music, Speech
G, U | ChatMusician [95] | arXiv 2024 | Llama 2 | Symbolic music
G, U | AnyGPT [379] | arXiv 2024 | Llama 2 | Audio, Audio music
G | Boosting Large [380] | arXiv 2023 | LLaMA | Speech
G, U | Unified-IO 2 [381] | arXiv 2023 | Unified-IO 2 | Speech, Audio, Audio music
G, U | M2UGen [93] | arXiv 2023 | Llama 2 | Audio music
G, U | LauraGPT [89] | arXiv 2023 | - | Speech
U | LLaSM [94] | arXiv 2023 | Llama 2 | Speech
G, U | AudioPaLM [87] | arXiv 2023 | PaLM | Speech
U | Pengi [86] | NeurIPS 2023 | - | Speech, Audio, Audio music
G, U | Speechgpt [84] | EMNLP 2023 | LLaMA | Speech
G, U | Sparks [382] | arXiv 2023 | GPT-4 | Symbolic music
U | Qwen-Audio [92] | arXiv 2023 | Qwen-LM | Audio, Speech, Audio music
U | SALMONN [91] | arXiv 2023 | Vicuna | Audio, Speech, Audio music
U | Llark [90] | arXiv 2023 | Llama 2 | Audio music
U | MU-LLaMA [88] | arXiv 2023 | LLaMA | Audio music
U | Speech-LLaMA [383] | ASRU 2023 | LLaMA | Speech
U | LTU [85] | ICLR 2024 | LLaMA | Audio
U | Yu et al. [384] | ICASSP 2024 | Vicuna | Speech
LLMs inspired backbone |  |  |  |
G, E | UniAudio [111] | arXiv 2023 | - | Audio, Speech, Audio music
G | AudioLM [110] | IEEE/ACM TASLP | - | Audio
G | MusicGen [48] | NeurIPS 2023 | - | Audio music
G | Jukebox [109] | arXiv 2020 | - | Audio music
G | MusicLM [47] | arXiv 2023 | - | Audio music
G | VALL-E [53] | arXiv 2023 | - | Speech
U | SICL [385] | arXiv 2023 | - | Speech
LLMs as conditioner |  |  |  |
G | TANGO [98] | arXiv 2023 | FLAN-T5 | Audio
G | Music ControlNet [88] | ICASSP 2024 | ChatGPT | Audio music
U | Wu et al. [99] | ICASSP 2024 | - | Audio
LLMs as agent |  |  |  |
G, E | Loop Copilot [107] | arXiv 2023 | GPT-4 | Audio music
G, U | MusicAgent [106] | EMNLP (Demos) 2023 | ChatGPT | Audio music, Symbolic music
G, U | Audiogpt [104] | AAAI 2024 | GPT-3.5 | Audio, Speech, Audio music
G, U | Hugginggpt [103] | NeurIPS 2023 | ChatGPT | Audio, Speech, Audio music
G | Wavjourney [105] | arXiv 2023 | ChatGPT | Audio, Audio music
LLMs as labeller |  |  |  |
G | Audiobox [102] | arXiv 2023 | LLAMA2 7B | Audio, Speech, Audio music
G | Make-An-Audio 2 [100] | arXiv 2023 | GPT-3.5 | Audio
Figure 8: Milestone works of LLMs-based audio research, including audio
generation, understanding, and editing.  Figure 9:  Method summary of LLMs-
related audio research according to different roles of LLMs. LLMs as backbone:
Language pre-trained LLMs checkpoint serves as the central unit for processing
text and audio tokens, either continuous or discretized. LLMs as inspiration:
Different from LLMs as backbone, this method trains on randomly initialized
LLMs architecture over discretized audio tokens. LLMs as Conditioner: LLMs
encode text prompts into embeddings which serve as the condition of the audio
generator. LLMs as agent: LLMs solve user requests by leveraging external
tools. LLMs as labeller: LLMs convert class labels into audio captions.

###  7.1 Domains

The works that integrate LLMs in the field of audio can be divided into three
categories: general audio sounds, music, and speech. Each category poses the
challenges and prospects from three pivotal perspectives—generation,
understanding, and editing. The summary of the key technical components of
LLMs-based approaches for LLMs-related audio tasks is shown in Tab.
LABEL:tab:audio_method, and the related audio-language dataset in Tab VIII.
The timeline of milestone works is shown in Fig. 8.

####  7.1.1 General Audio Sounds

General Audio Sounds refers to any sound that can be heard. It includes
various auditory experiences, including natural sounds (like birds chirping
and wind rustling through trees), human activities (such as traffic noise and
machinery), and other environmental noises. In the past few months, the field
of general audio has experienced significant advancements through the
application of LLMs [98, 104, 85, 100, 103, 110, 99, 111, 91, 92, 102, 381].
In the following sections, we will explore the specific areas of audio
generation, audio understanding, and audio editing to analyze how these LLMs-
driven developments are reshaping the audio domain.

Audio Understanding. General audio understanding involves the analysis and
interpretation of a wide array of sounds from our environment, beyond just
speech and music. This task includes identifying and classifying sounds (such
as distinguishing a car horn from a dog bark), recognizing patterns in
environmental sounds (like detecting the sound of rainfall or an approaching
vehicle), and even understanding the context or source of sounds.

A suite of groundbreaking models, LTU (Listen, Think, and Understand) [85],
SALMONN (Speech Audio Language Music Open Neural Network) [91], Qwen-Audio
[92], and UNIFIED-IO 2 [381], have utilized LLMs as their backbone for audio
understanding. Unlike LTU [85], which is the first multimodal LLM to focus on
general audio understanding beyond just speech, SALMONN [91] is the first
multimodal LLM capable of perceiving and understanding general audio inputs
with speech, audio events, and music. By integrating audio with other data
modalities, UNIFIED-IO 2 [381] leverages LLMs to enhance the understanding of
complex interactions between various types of input. Qwen-Audio [92] improves
interaction capabilities of pre-trained audio models by covering over 30
different tasks and various audio types, including human speech, natural
sounds, music, and songs, thereby promoting comprehensive audio understanding
capabilities. To improve user interaction, models like AudioGPT [104] and
HuggingGPT [103] also leverage LLMs to serve as intelligent interfaces. The
works [99, 85, 92, 381, 91] demonstrate how LLMs can be used to enhance
automated audio captioning.

LTU [85] combines the audio perception model AST [386] with LLaMA [137] to
improve audio understanding through a perception-to-understanding curriculum.
For this purpose, this work also constructs the OpenAQA-5M dataset, which
includes 1.9 million closed-ended and 3.7 million open-ended tuples. This
dataset facilitates LTU’s training within an autoregressive framework.

SALMONN [91] processes general audio inputs, including speech, events, and
music, by integrating a text-based LLM with speech and audio encoders
together. This fusion improves SALMONN’s understanding ability across various
audio phenomena.

UNIFIED-IO 2 [381] is the first autoregressive multimodal model that
integrates text, images, audio, and actions into a unified framework. Using a
single encoder-decoder transformer model, it tokenizes inputs from different
modalities into a shared semantic space for processing.

Qwen-Audio [92] scales up audio-language pre-training to include over 30
tasks. To address the interference problems that arise from training all tasks
and datasets together, a multi-task training framework was designed. This
framework uses a sequence of hierarchical tags for the decoder, which helps
share knowledge and prevent interference by using both shared and specific
tags. Further developed upon Qwen-Audio, Qwen-Audio-Chat can take input from
different audio and text sources, allowing for multi-turn conversations and
supporting various audio-focused scenarios.

AudioGPT [104] and HuggingGPT [103] showcase the use of LLMs for audio
understanding by coordinating tools through LLMs-driven interfaces. AudioGPT
utilizes ChatGPT as a central node for audio and speech applications,
depending on external audio systems for functionality. HuggingGPT functions as
an agent that combines ChatGPT’s language capabilities with a diverse set of
AI models from the Hugging Face community, improving its ability to understand
audio content.

Wu et al. [99] focuses on advancing automated audio captioning (AAC), a field
dedicated to generating descriptive text for sounds from nature and human
activities. This work pushes this development further by extensively
integrating pretrained models and LLMs. Wu et al. employ BEATS for extracting
detailed audio features and use the INSTRUCTOR LLM for obtaining text
embeddings of captions. Additionally, Wu et al. introduce a data augmentation
technique using ChatGPT to create caption mix-ups and enrich the training data
in terms of quantity, complexity, and diversity.

Audio Generation. Audio generation is an emerging field that focuses on
modeling the creation of diverse audio content. The application of LLMs has
significantly advanced the generation of audio. Significant contributions in
text-to-audio generation include TANGO [98], Make-an-Audio 2 [100], WavJourney
[105], AudioLM [110], and Audiobox [102]. From employing text embedders and
diffusion models in TANGO [98] and Make-an-Audio 2 [100] to integrating multi-
modal approaches in WavJourney [105] and advanced tokenization in AudioLM
[110] and Audiobox [102], these initiatives highlight the versatility and
impact of LLMs in driving forward the capabilities of audio generation
technologies.

TANGO employs FLAN-T5 [387] as a text embedder, while Make-an-Audio 2 uses
pre-trained LLMs for text parsing into structured pairs, both utilizing latent
diffusion-based models for audio synthesis. WavJourney [105] leverages the LLM
agent to integrate various audio models for producing cohesive audio content,
including speech, music, and sound effects, based on textual descriptions.
AudioLM [110] generates high-quality audio with an emphasis on long-term
consistency. The method converts input audio into discrete tokens, effectively
making audio generation a task akin to language modeling within a discrete
space. Audiobox [102] uses LLMs for data construction, including tagging audio
with high-quality, detailed captions and using LLMs to evaluate the quality of
these annotations automatically. Then, Audiobox uses flow-matching techniques
to produce diverse audio types with precise attribute control, from speech to
music and sound effects. UniAudio [111] introduces a versatile system
utilizing LLMs to generate a diverse range of audio types, including speech,
sounds, music, and singing, under various input conditions. Unlike task-
specific models, UniAudio tokenizes different audio types and their respective
conditions into a unified sequence, enabling next-token prediction with LLMs.

####  7.1.2 Music

Music is an art form characterized by the arrangement of sounds in time,
typically including elements such as melody, harmony, rhythm, and timbre. It
is produced using musical instruments and/or the human voice and is often
organized according to pitch (which influences melody and harmony), rhythm
(including tempo, meter, and articulation), dynamics (variations in loudness),
and the sonic properties of timbre and texture. Music serves multiple
functions, including aesthetic enjoyment, ceremonial purposes, and the
expression of cultural identity. In the realm of music research, audio music
refers to the actual recorded sound waves of music, while symbolic music
pertains to the notational representation of music, such as MIDI [388] files.
Each of these forms requires distinct approaches for analysis and
manipulation. We explore the interconnected fields of music understanding,
generation, and editing, each of which utilizes different techniques and
technologies to analyze, create, and refine music, further enriching its
cultural and artistic impact.

Music Understanding. Music understanding involves the analysis and
interpretation of musical elements such as melody, harmony, rhythm, and
timbre, to recognize patterns, genres, emotions, and contextual meanings
within music. It includes the analysis of simple motifs to complex structures.

The field of music understanding has made significant progress with the
development of models like the Music Understanding LLaMA (MU-LLaMA) [88],
LLARK [90] MusicAgent [106], LyricWhiz [108], and ChatMusician [95]. These
models showcase a range of methods, from analyzing detailed music features to
improving lyric transcription. They highlight how LLMs can assist our
understanding and interaction with music in different ways and applications.

MU-LLaMA [88] uses a pretrained MERT encoder for initial music representation,
which is then integrated into the LLaMA model with an adapter. This process
leverages the capabilities of LLMs to understand music by analyzing its
comprehensive features. LLARK [90] improves music understanding by using a
multimodal model that is fine-tuned with instructions from refined annotations
in music datasets. It combines a generative music model with a language model
to analyze music in a unified way. MusicAgent [106] assists music
understanding and generation through LLMs, automating tasks to meet user needs
and using tools for task execution. This simplifies processes and encourages
exploration in music processing. In summary, MU-LLaMA focuses on the analysis
of musical features, LLARK leverages refined labels for a more general
understanding, while MusicAgent emphasizes user-friendly interactions.

LyricWhiz [108] proposes a multilingual, zero-shot method for automatic lyrics
transcription, performing well across various genres. It uses “Whisper” for
speech recognition and “GPT-4” for context-aware annotations, acting as the
transcription’s “ear” and “brain”. This combination greatly reduces the Word
Error Rate in English and provides effective transcription in multiple
languages.

Music Generation. AI music generation, especially with the use of LLMs, is
changing the industry by creating various and complex musical pieces. Notable
examples include MusicLM [47], Jukebox [109], MusicGen [48], Music ControlNet
[97], M2UGen [93], ChatMusician [95], and SongComposer [96]. Specifically,
these models use different techniques, from using LLMs as text embedders to
employing diffusion processes and autoregressive Transformers.

MusicLM [47], Jukebox [109], and MusicGen [48] represent significant strides
in text-to-music generation, each drawing inspiration from the capabilities of
LLMs and employing Transformer architectures to handle complex audio tasks.
MusicLM [47] treats conditional music generation as a hierarchical sequence-
to-sequence task, utilizing decoder-only Transformers to create music in both
the semantic and acoustic stages. Jukebox [109] addresses the challenges of
long audio contexts by compressing raw audio into discrete codes using a
multiscale VQ-VAE and then modeling these codes with autoregressive
Transformers.

On the other hand, MusicGen [48] directly incorporates an LLM as a text
embedder. It combines text tokens and melody conditions into a Transformer
decoder, which then processes the inputs in an autoregressive way. The final
step involves a codec model that converts the processed tokens back into
music.

Music ControlNet [97] uses an LLM as a text embedder and introduces a
diffusion-based model for music generation that offers precise control over
dynamic and temporal aspects of audio. Inspired by image-domain ControlNet’s
pixel-wise control, it applies similar precision to audio through controls for
melody, dynamics, and rhythm derived from training audio.

M2UGen [93] introduces a multi-modal music understanding and generation
framework that leverages LLMs along with pretrained models like MERT, ViT, and
ViViT to analyze and create music from diverse inputs such as music, images,
and videos. The decoder part utilizes AudioLDM 2 [45] and MusicGen [48] for
generating music.

ChatMusician [95] and SongComposer [96] both focus on generating symbolic
music but use different methods and representations. ChatMusician is an open-
source LLM with intrinsic musical abilities, using continual pre-training and
fine-tuning of Llama2 with ABC notation, a text-compatible music
representation. It treats music as a second language, allowing it to
understand and generate music using only a text tokenizer, without the need
for external multi-modal neural structures. In contrast, SongComposer uses
MIDI for its symbolic music representation and introduces a unique tuple
design to format lyrics alongside three note attributes: pitch, duration, and
rest duration. This design ensures the correct interpretation of musical
symbols and precise alignment between lyrics and melodies, differentiating it
from ChatMusician’s approach.

Music editing involves refining and altering musical elements to improve the
sound quality and artistic expression. Loop Copilot [107] combines LLMs with
specialized AI music models to create a conversational interface for
collaborative human-AI music loop creation. It uses a large language model to
interpret user intentions and directs specialized AI models to generate and
refine music through interactive dialogue. Key musical attributes are
centralized and managed to ensure consistent quality throughout the creative
process.

####  7.1.3 Speech

Speech specifically refers to the sounds that humans produce when they speak.
It is the verbal manifestation of language, including a variety of linguistic
elements such as words, sentences, tone, intonation, and rhythm. Speech is a
fundamental mode of human communication and can vary greatly depending on
factors like language, dialect, emotional state, and context. In the realm of
artificial intelligence, LLMs have been advancing both the understanding and
generation of speech, facilitating machines to interpret and replicate human-
like spoken communication with increasing accuracy and naturalness.

Speech Understanding. Speech understanding empowers machines to interpret
spoken language. This aspect of AI captures not just the words but also the
speaker’s intent and nuances, with progress driven by LLMs. Key contributions
in this field include SpeechGPT [84], AudioPaLM [87], and other studies [383,
385, 384], showcasing the improved capabilities of LLMs in recognizing and
processing speech across diverse contexts.

SpeechGPT [84], AudioPaLM [87], and Speech-LLaMA [383] represent pivotal
developments in speech understanding, all utilizing LLMs as the structural
backbone of their frameworks. SpeechGPT [84] and Speech-LLaMA [383]
specifically use LLaMA as their foundation. SpeechGPT not only facilitates
understanding and generating multi-modal content but also supports inter-modal
knowledge transfer. It introduces SpeechInstruct, a large-scale cross-modal
speech instruction dataset built upon discrete speech representations,
highlighting its multi-modal capabilities. Meanwhile, Speech-LLaMA integrates
speech signals with LLMs, emphasizing a mixture of auditory and linguistic
data processing. Similarly, AudioPaLM [87] combines the strengths of PaLM-2
[389] and AudioLM [110] into a unified multimodal framework that performs well
in both speech understanding and generation. It retains paralinguistic
features such as speaker identity and intonation from AudioLM and blends them
with the textual linguistic capabilities of PaLM-2, demonstrating an approach
to multimodal speech processing.

Recent research in Automatic Speech Recognition (ASR) has focused on improving
model accuracy with LLMs. Wang et al. [385] investigate the in-context
learning abilities of the Whisper [390], which is an ASR model released by
OpenAI. SICL [385] is introduced to reduce the Word Error Rates (WERs) with
only a small number of labeled speech samples without gradient descent. Yu et
al. [384] presents a study of structures that include fully connected layers,
multi-head cross-attention, and Q-Former as connectors for integrating ASR
models with LLMs.

Speech Generation. Speech generation, the process of converting text or other
inputs like speech prompts into spoken language, has significantly evolved
with the integration of LLMs. These models facilitate the naturalness and
contextual relevance of generated speech, making it increasingly realistic and
similar to human speech.

Inspired by the capabilities of LLMs, Wang et al. introduced VALL-E [53], a
transformative approach in speech generation. VALL-E utilizes a neural codec
language model that employs discrete codes from an existing neural audio
codec, reframing text-to-speech (TTS) synthesis as a conditional language
modeling task rather than traditional continuous signal regression. Building
on this VALL-E, Hao et al. further advanced the field with their study [380].
They conducted an investigation to improve LLMs’ speech generation
capabilities by integrating the pre-trained LLM frameworks, LLaMA/OPT, with
the TTS model VALL-E. This research showcases a combination of language
modeling and speech synthesis technology, aiming to produce more natural and
effective speech outputs.

Different from previous works, using an LLM as the backbone, LauraGPT [89],
developed by Wang et al., is a unified GPT model capable of handling both
audio and text for recognition, understanding, and generation tasks. It
performs well in a variety of functions including speech recognition,
translation, text-to-speech synthesis, and more.

Another work proposed by Kakouros et al. [391] examines the potential of word
surprisal, a measure of word predictability in context, to improve speech
synthesis prosody.

###  7.2 Roles of LLMs

Language provides a great abstraction of our world. With the flexibility and
rich descriptive power of language, researchers have unified language
understanding and language generation, into a paradigm so-called generative
understanding. Audio research benefits a lot from language in the LLM era,
with LLMs serving as a bridge to collect and process information, the field is
now able to reach a similar generative understanding stage. Broadly, we
categorize the roles of LLMs into the following: LLMs as Backbone, LLMs as
Conditioner, LLMs as Labeller, LLMs as Agent, and LLMs Inspired Backbone. The
method summary is shown in Fig. 9.

####  7.2.1 LLMs as Backbone

Using an LLM as a backbone entails leveraging a pre-trained LLM, like LLaMA,
as the central architecture of a system. These backbones, essential for the
system’s learning and processing abilities, are integrated with various
network components and undergo fine-tuning. In the realm of multimodal LLM
applications within the audio domain, the LLM backbone plays a pivotal role.
It is either coupled with structures specialized for modality-specific
understanding or generation, or it employs a tokenizer to convert audio into
discrete tokens.

Much of the current research in LLMs typically adopts a cascade approach,
which often involves the use of modality-specific encoders and/or decoders. In
LTU [85], Yuan et al. suggest utilizing an Audio Spectrogram Transformer (AST)
[386], a transformer encoder pre-trained with CAV-MAE [392]. This encoder’s
representations are aggregated and input into a LLaMA-7B backbone. These audio
pre-trained representations are then paired with corresponding texts. To fine-
tune the LLM backbone, LoRA adapters [225] are employed, tasked with
predicting text pairs based on the audio representations. Pengi [86] follows a
similar paradigm. Soham et al. call it a ‘audio-text-to-text‘ format. Except
for an CLAP [131] audio encoder, it also uses a text encoder to encode the
task instructions. The audio representation, as well as the text instruction
representation, are together fed into the LLM as a prefix. The LLM is then
trained to predict the paired text output, e.g. a sound description. Both LTU
and Pengi show improvement in close-ended audio understanding tasks and a
certain level of open-ended audio understanding tasks. Similar approaches can
also be found in LLaSM [94], Mu-LLaMA [88], MusicLingo [393], Llark [90],
Qwen-Audio [92]. Popular audio encoder may include CLAP [131], MERT [394],
Whisper [385], AST [386].

Besides the model focused solely on understanding, there has also been
researches extending into generation. Part of these researches adopt the
design philosophy of the cascade approach, incorporating not just an audio
encoder but also introducing an audio decoder. For instance, in M2UGen [93],
Atin et al. adapt both a MERT encoder and an Audioldm2 [45] / MusicGen [48]
decoder. The output projection layer then maps the LLaMA2 model’s output
embeddings to the music decoder. A similar approach is observed in NExT-GPT
[134], a recently proposed any-to-any multimodal language model.

However, the cascade approach requires training heterogeneous neural
structures. In scenarios with abundant data and computational resources, these
heterogeneous neural structures could lead to decreased training efficiency
and lower system scalability. Recently, a unified approach has garnered the
attention of researchers. This method typically necessitates the use of audio
codecs [395, 396], to tokenize raw audio into discrete tokens, which are then
flattened into a one-dimensional sequence for input into an LLM. This requires
the LLM’s vocabulary to include audio tokens, thereby necessitating an
expansion of the LLM vocabulary, akin to integrating audio as a new language
into the LLM. This method employs a uniform LLM structure, facilitating
scalability. AudioPaLM [87], LauraGPT [89], SpeechGPT [84] follows this
paradigm.

####  7.2.2 LLMs as Conditioner

In this setting, LLMs usually act as text embedders, encoding input text to
condition the system’s response or output, thus enabling a more nuanced and
context-aware processing of audio data.

Tango [98] follows this paradigm. It comprises three main components: a text
encoder, a Latent Diffusion Model (LDM), and a Mel-Spectrogram/Audio
Variational Autoencoder (VAE). The text encoder is a Flan-T5, which translates
the audio’s input text prompt into a textual representation. This
representation is then utilized to construct a latent audio representation or
audio prior from standard Gaussian noise through reverse diffusion. Following
this, the Mel-Spectrogram VAE’s decoder generates a Mel-Spectrogram from the
latent audio representation. Finally, this Mel-Spectrogram is input into a
vocoder to produce the final audio output. MusicGen [48] follows a similar
paradigm. The study tested both T5 and Flan-T5 models along with CLAP, and
found that the T5 encoder, serving as a text conditioner, achieved the highest
relevance subjective test score in relation to the text input.

####  7.2.3 LLMs as Labeller

Currently, the majority of large-scale audio datasets, such as AudioSet [371]
and VGGSound [376], are annotated solely with class labels, akin to ImageNet
[397]. Researchers aiming to undertake text-to-audio tasks are compelled to
transform these class labels into full-sentence audio descriptions, also known
as audio captioning. A prevalent method involves utilizing LLMs to achieve
this transformation.

A common pipeline for text description augmentation involves initially
crafting description templates manually for a labeled audio dataset, thereby
parsing audio classes into more uniformly formatted descriptions. Subsequent
steps leverage self-instruction methods, employing LLMs such as ChatGPT [7]
that are capable of following instructions to paraphrase these descriptions,
often making use of self-instruct [398] techniques to enrich the dataset
further.

####  7.2.4 LLMs as Agent

In ‘LLMs as Agent’, LLMs are employed to interface with various tools,
orchestrating multiple functionalities to accomplish diverse tasks. This role
highlights the versatility of LLMs in managing and executing complex, multi-
dimensional operations.

Communicating with LLMs can be approached in various ways. A notably
straightforward yet effective method is through a text interface. During the
nascent phase of multi-modal LLM audio research, Huang et al. introduced
AudioGPT. This system, capitalizing on advanced audio foundation models,
tackles tasks such as sound detection, audio-to-text conversion, speech
recognition, and speech translation. The data gleaned from these audio
processes is then transformed into text, seamlessly integrating with LLM
interactions. Within this framework, task analysis, model assignment, and
response generation all function through textual operations. AudioGPT drew
inspiration from its contemporary work, HuggingGPT, which employs a similar
approach. HuggingGPT uses LLMs to invoke various models on Hugging Face, a
platform hosting a diverse array of machine learning models. Similarly,
MusicAgent is proposed to streamline AI-powered music processing by
integrating a variety of tools and an autonomous workflow, primarily
facilitated by LLMs like ChatGPT. It features a diverse toolset sourced from
platforms like Hugging Face, GitHub, and various web APIs. Tasks like music
classification, music separation, lyric recognition are supported.

####  7.2.5 LLMs Inspired Backbone

With the success of the next token prediction paradigm in language modeling,
the audio domain has also sought to apply this approach by discretizing audio
into tokens for modeling. Researchers aim to achieve emergent capabilities on
audio tokens similar to those observed in LLMs, such as in-context learning
and the chain of thoughts ability. Currently, modest in-context learning
abilities have been confirmed to be attainable through the language modeling
pretraining of audio tokens.

In VALL-E[53], researchers combined autoregressive and non-autoregressive
language models to model encoded tokens. Thanks to the residual vector
quantized (RVQ) modeling[395] of acoustic information, VALL-E can continue
speech with only short audio and text prompts, preserving the speaker’s