timbre, prosody, and acoustic environment while following text constraints. In
AudioLM[110], researchers discovered that unconditional training on RVQ-based
acoustic tokens did not yield semantic-level consistency. Consequently, they
proposed introducing semantic tokens based on Self-Supervised Learning (SSL)
representations. The representations from an SSL-pretrained teacher contain
rich semantic information, and performing k-means clustering on these
representations yields a k-means quantizer, allowing for the extraction of
semantic tokens for the training set. Language modeling on these semantic
tokens achieves better semantic consistency, enabling unconditional speech
continuation to maintain semantic coherence.

##  8 Tool-augmented Multimodal Agents

Figure 10: The pipeline of tool-augmented multimodal agents.  Figure 11: The
milestone of multimodal agents that focus on multimodal generation and
eidting.

Over the past few months, lots of works [399, 400, 401, 103, 218, 402, 403,
404, 405, 406, 407, 408, 409, 410, 411, 412, 413, 414, 415, 416, 417, 217]
known as tool-augmented LLMs have been emerged as a promising direction in
human-computer interaction. They empower LLMs to use external tools to enhance
the models’ capabilities. Among them, several works [103, 217, 218, 402, 418,
407] which extend LLMs to other modalities beyond the natural language stand
out. In contrast to those expert models which only focus on optimizing one
specific task, _e.g._., image generation or video generation, these pioneering
works built upon LLMs can interactively generate or edit images, videos, and
audio by invoking corresponding tools. In this section, we mainly focus on
reviewing the recent works that aim to extend LLMs to multimodal generation by
augmenting them with external tools.

###  8.1 Motivation

It is known that LLMs have limitations in accessing and processing information
that is not available in their training data, such as ephemeral, changing, or
private data. For example, LLMs may not be able to answer questions that
involve factual knowledge updated frequently, such as the current weather or
stock prices.

To overcome these limitations, several works have proposed to augment LLMs
with external tools or APIs, such as retrieval-augmented generation (RAG),
calculator or visual foundation models, that can provide additional
information or functionality for LLMs. These tools can be invoked by LLMs
through natural language instructions, and the results can be integrated into
the LLMs’ outputs. For example, an LLM can use a weather API to obtain the
current temperature and humidity of a given location, and use them to generate
a natural language response. The tool-augmented paradigm have been validated
the efficacy by many works [399, 400]. In practice, Microsoft
Copilot111https://www.microsoft.com/en/microsoft-copilot augmented by various
tools has been integrated into their applications including Bing, Edge and
Windows operating system, which dramatically facilitates user experience.
OpenAI also releases a Function
Calling222https://platform.openai.com/docs/assistants/tools service that can
give assistants access to OpenAI-hosted tools like Code Interpreter and
Knowledge Retrieval or build your own tools.

It is well known that LLMs can not generate or edit content in other
modalities, such as images, videos, or audio, which is helpful for creative
purposes. Motivated by tool-augmented LLMs, some pioneers develop multimodal
agents that can control diverse tools across different modalities like image,
video and audio. By augmenting LLMs with external tools, it can enable more
natural and versatile human-computer interaction, as well as more powerful and
creative applications.

###  8.2 Methods

As depicted in Fig. 10, the general framework of tool-augmented LLMs for
multimodal interaction consists of three main stages: 1) Task Planning which
tasks LLM as a controller to interpret the natural language instructions into
a tool invocation scheme. Specifically, the core aim of this stage is to
decide which tools will be used and prepare the arguments for tools. In this
stage, the selected tools are organized into the tool invocation scheme that
specifies the sequence of tool invocation and the inputs of tools. 2) Task
Execution which hosts lots of external multimodal tools, _e.g._ , image
generation, video editing, or audio synthesis. The tools are invoked based on
the invocation scheme obtained in task planning. It is noteworthy that most of
the tools are based on deep learning models including stable diffusion [10],
ControlNet [65], Blip [204], LLaVA [143], _etc_. 3) Response Generation which
can make a user-friendly response by prompting the LLM with execution outputs
from task execution. The overall system connects the LLM and external
multimodal tools, which not only enhances the capabilities of LLM but also
evidently improves the user experience.

The main difference among the existing works lies in how they perform task
planning. To this end, the tool-augmented LLMs for multimodal interaction can
be roughly divided into two categories: (1) training-free methods [217, 103,
218, 419], and (2) instruction-tuning methods [402, 407, 418, 420]. The
evolution path is shown in Fig. 11. In addition, Table XI wraps up works
related to multimodal agents. Next, we will elaborate on the details of two
types of methods.

####  8.2.1 Training-free Methods

Training-free methods [419, 421, 103, 217, 218, 422] primarily reply on prompt
engineering, _e.g._ , ReAct [423], in-context learning [424], to boost the
reasoning capabilities of LLMs. These methods exhibit their effectiveness on
many tasks.

Gupta _et al._ [419] and Surís _et al._ [421] propose a code generation-based
method for solving complex and compositional visual tasks given natural
language instructions. These two methods mainly focus on image perception and
visual question answering. Taking [419] as an example, they develop the
framework of VISPROG, which consists of two main components: a program
generator and a program executor. The program generator uses a large language
model (GPT-3) to generate python-like modular programs from natural language
instructions. The authors prompt GPT-3 with pairs of instructions and the
desired high-level programs, along with a new instruction. GPT-3 then
generates a program that can be executed on the input image(s) to perform the
described task. Each line of the program invokes one of the 20 modules
supported by VISPROG, such as object detection, segmentation, image editing,
knowledge retrieval, etc. Then, the program executor executes the generated
program on the input image(s) and produces the output and a visual rationale.
The executor steps through the program line-by-line and invokes the correct
module with the specified inputs. The modules are implemented as python
classes that use off-the-shelf computer vision models, image-processing
routines, or Python functions. The executor also updates the program state
with the output variable name and value after each step. It is noticed that
VISPROG does not have stage 3 explicitly, and directly returns the output from
tools to users. This work opens up a new direction for multimodal human-
computer interaction and inspires the following studies.

In contrast to VISPROG [419, 421], Visual ChatGPT [217] does not generate
code, directly. It combines a large language model (ChatGPT) with various
visual foundation models (VFMs) to enable conversational interaction with both
text and images. To achieve this, Wu _et al._ [217] design a Prompt Manager
that bridges the gap between ChatGPT and VFMs. The Prompt Manager converts all
non-language signals into language prompts that ChatGPT can understand and
process. Essentially, Visual ChatGPT performs task planning via ReAct [423],
which is the simplest and most straightforward way to augment LLMs with tools.
Instead of generating a full solution containing all related tools once,
ReAct, which extends Chain-of-Thought to tool use, immediately executes the
tool after each thought. It means that the system carries out stage 1 and
stage 2 alternately. In addition, to accurately call the tools, each tool is
equipped with a crafted natural language prompt that instructs the LLMs how to
use it. In practice, one can use a prompt like “Please use the tool named
ImageBind to generate an image of a cat wearing a hat” to invoke an image
generation tool, and expect the system to return an image path as the
response. This approach does not require any modification or retraining of the
LLMs, and can leverage the existing pre-trained LLMs, such as GPT-3 or
ChatGPT. InternGPT [218] shares a similar pipeline to Visual ChatGPT, but it
supports pointing devices. Thus, InternGPT offers more interesting and diverse
modes of interaction, including clicking and drawing. For example, After the
clicking operation is triggered, InternGPT utilizes SAM [425] to select the
chosen semantic region, which can be used to remove or replace the object.
Furthermore, InternGPT supports user directly in drawing the sketch based on
which it can generate a new image.

TABLE X: The instruction samples of multimodal agents, which are used to train
or assess models. Image
---
1\. Can you remove the dog in the image_1.png?
2\. Can you create a new image that depicts a family having a picnic in the
foreground?
3\. Generate a new image based on a segmentation map named seg.png. The new
image should show a beautiful landscape.
4\. Using the pose map pose.png, generate a new image that features a cozy
cabin in the woods with a campfire, trees, and a couple roasting marshmallows.
5\. Using the segmentation from the given image, generate a new image that
showcases a magical forest with glowing mushrooms and fairies flying around.
Video
1\. Can you remove the dog in the video_1.mp4?
2\. Can you provide a video that is related to the given image?
3\. Please give me a new video derived from this file image_2.png?
4\. Make a video that illustrates a calm landscape of a lake surrounded by
vegetation.
5\. I would appreciate it if you can dub the video_1.mp4 with given audio file
named aud_1.wav.
Audio
1\. Can you create a song that visually represents the image?
2\. You need to generate a piece of background music for video_3.mp4.
3\. Please generate a piece of music by given prompt: An 80s driving pop song
with heavy drums in the background.
4\. I want to create a piece of background music for video_4.mp4 and dub the
video with this piece of music.
5\. translate the text into speech: “Hope is the thing with feathers That
perches in the soul, And sings the tune without the words, And never stops at
all”.
Figure 12: We show some case studies on image generation for different
methods.  Figure 13: Interactive image editing by clicking.  Figure 14: An
example of generating audio from the given image.  Figure 15: This example
shows the multimodal agent [407] generate multiple outputs from the
instruction. It first generates an image conditioned on the text prompt and
then creates a video from the generated image.  Figure 16: An example of
multimodal generation that visualizes the weather condition.

In addition, HuggingGPT [103] builds upon a large language model used as the
core controller to manage and organize the cooperation of expert models from
machine learning communities such as Hugging Face. HuggingGPT consists of four
stages: task planning, model selection, task execution, and response
generation. HuggingGPT separates model selection from task planning. In the
task planning stage, Shen _et al._ [103] use ChatGPT to parse the user request
into a list of tasks, and determine the execution order and resource
dependencies among them. In the model selection stage, they use ChatGPT to
assign appropriate models to each task, based on the model descriptions
available on Hugging Face. In the task execution stage, the system invokes and
executes the selected models. Last, the response generation stage uses ChatGPT
to integrate the predictions from all models and generate a response for the
user. It is noticeable that HuggingGPT uses in-context learning in task
planning and model selection. Therefore, it can perform well on some easy
cases, but almost invariably fails to address the hard problems. Different
from Fig. 10, HuggingGPT actually decomposes the task planning into two steps,
namely, one step to parse the task and the other step to identify tools for
each task. HuggingGPT also introduces some techniques to handle resource
dependencies, hybrid endpoints, and prompt design in HuggingGPT.

Limitations. Training-free methods have some disadvantages. First, these
methods rely on the availability and accessibility of the pre-trained LLMs,
which are often proprietary and expensive to use. Second, these methods
require manual design and tuning of the prompts, which can be time-consuming
and error-prone. Last, these methods assume that the LLMs have sufficient
knowledge and capability to use the tools. However, it usually fails to
address complex problems. Furthermore, we found directly using off-the-shelf
LLMs leads to a performance drop when extending the tool set to a large scale.
This is due to the fact that LLMs normally are not trained for this purpose.

####  8.2.2 Instruction-tuning Methods

Instruction-tuning methods[400, 401, 399] involve training a language model to
follow human instructions more accurately, which can vastly improve the
capabilities of tool use for LLMs. As such, several multimodal agents [402,
418, 407] finetune an LLM in the first stage of task planning to use the tools
across different modalities. In this type of method, the key is how to
generate the instruction corpus to train the LLMs.

Taking GPT4Tools [402] as an example, it aims to efficiently enable LLMs to
use multimodal tools, such as visual models, by self-instructing from advanced
LLMs, such as LLaMA [137] and OPT [426]. The first challenge is how to
construct the training corpus. Yang _et al._ [402] utilize a simple yet
effective method to generate an instruction-following dataset by prompting an
advanced LLM such as ChatGPT with various multimodal contexts and tool
descriptions. Next, they filter out similar or invalid instructions from the
raw data, resulting in 41K items. GPT4Tools augments the data by introducing
negative samples (instructions that do not require tool usage) and context
samples (instructions that involve multiple turns or actions). Finally, This
work builds a dataset of 71.4K instruction-response pairs, which covers 31
tools for various visual tasks. We here show some examples of instructions in
Table X. After dataset construction, GPT4Tools incorporates Low-Rank
Adaptation (LoRA) to finetune the open-source LLMs on the generated dataset,
thus adapting them to use tools for various visual tasks, such as visual
comprehension and image generation. Since the LLMs are tuned on the
instruction corpus, the capabilities of tool use are dramatically improved.

Li _et al._ [418] proposes ModelScope-Agent, utilizing the models in
ModelScope to augment the open-sourced LLMs. In this work, the authors also
provide a tool dataset named MSAgent-Bench. Distinct from the above methods,
Li _et al._ [418] design a module of tool retrieval with memory control to
identify tools instead of directly prompting LLMs. Such a design makes the
overall system more flexible and scalable.

Instead of directly training an LLM as a controller to generate the solution,
Liu _et al._ [407] train a language model to perform tool-agnostic task
decomposition and propose Thoughts-on-Graph (ToG) to generate solutions for
each sub-task, which enables LLMs to use various tools across different
modalities, such as text, image, audio, and video, to solve complex real-world
tasks. This paper argues there are three main challenges identified in tool-
augmented LLMs: a) ambiguous user prompts, b) inaccurate tool selection and
parameterization, and c) inefficient tool scheduling. To this end, Liu _et
al._ [407] proposes ControlLLM, a powerful framework that consists of three
components: Task decomposition, which is able to parse the user input into
several subtasks with specific inputs and outputs; Thoughts-on-Graph (ToG)
paradigm, which finds the optimal solution path on a pre-built tool graph by
depth-first search (DFS) algorithm; The execution engine is equipped with a
powerful toolbox, which interprets the solution path and schedules the tools
efficiently on different computational devices. ControlLLM supports lots of
multimodal tools and provides a user-friendly demo interface.

Followed by ControLLM, Wang  _et al._ [420] develop a multimodal tool agent
system called MLLM-Tool. In contrast to prior works [402, 418, 407], this is
the first work to train a large multimodal model for tool learning. In task
planning, they utilize a multimodal encoder based on ImageBind [133] as well
as a projection layer to extract a unified embedding space for six modalities:
text, image, video, audio, speech, and music. Then, the user instruction
combining the multimodal embeddings is fed into a language model to predict
the corresponding API names.

Limitations. Currently, the instruction-training methods still have some
weaknesses. On the one hand, training an LLM is forbiddingly expensive for
most researchers. Despite that lots of works [225, 427, 428], such as LoRA and
its variants, have been proposed to train LLMs efficiently, it still needs to
make a trade-off between performance and training costs. On the other hand, it
needs to generate to diverse instruction corpus for training purposes. Self-
instruct [398] is an effective method that can prompt LLMs to generate more
instruction automatically from seed instruction. However, it is hard to
control the quality of the generated corpus, which unavoidably poses a
negative impact on training LLMs. In addition, there still is an open problem
to be addressed. That is, how to make an LLM that learns from a closed corpus
generalize to unseen instructions. This problem is about whether the language
models are able to generate novel solutions that are unseen in the training
corpus, to solve more complex problems.

###  8.3 Demonstrations

Some works [217, 103, 218, 407] not only have open-sourced code, but also
release online demos. As a result, in this section, we simply showcase the
functionalities of some works by using their available online demos.

Currently, several multimodal agents [217, 103, 218, 407] can interact with
images, either by generating, editing, or understanding them. For example,
they augment LLMs with image generation or editing models, such as Stable
Diffusion [10], ControlNet [65] and InstructPix2Pix [242], that can create or
modify images based on text prompts. As shown in Fig. 12, we make comparisons
between Visual ChatGPT, HuggingGPT, InternGPT, and ControlLLM by an example of
image generation, which is beneficial to study their capabilities directly.
Interestingly, HuggingGPT decomposes the instruction into two tasks and
returns two generated images. The generated image from ControlLLM is more
aligned to the provided instruction. We can find a slight performance gap
between different methods due to the underlying models used in their systems.
In contrast to those text2image models like Stable Diffusion [10], multimodal
agents are able to generate vivid images in an interactive manner instead of
stiffly returning images to the user. In addition, InternGPT and ControlLLM
support pointing devices as input to enhance interactivity. As depicted in Fig
13, taking ControlLLM as an example, the user can click the region of interest
from the image, and the object lying in this region is then segmented into a
mask by using SAM [425]. Next, the user can send an instruction to edit the
image such as removing the masked object in the image. Such a manner not only
can edit the image more precisely and efficiently, but also improves the rate
of success for tool use.

In addition, several works [103, 407] are able to generate the video as shown
in Fig. 14 and also can dub the video with the audio track. However, there is
no multimodal agent available that supports direct editing on the video frame.
This is partly due to the fact that video editing is extremely challenging and
still needs to be further studied. Some agents also support to generate audio
like speech [103, 407] and music [407]. It is really interesting to combine
the tools (_e.g._ , image_captioning and text_to_music) to generate a piece of
music for an image as depicted in Fig. 15. Furthermore, multimodal agents are
not limited to these audio-visual tasks. For example, ControlLLM [407]
supports querying weather and can even visualize the weather condition by an
image as demonstrated in Fig. 16.

TABLE XI:  Multimodal agents. We only showcase the methods that build upon LMMs to solve the user’s question by invoking expert models. Methods | Venue |  Image Editing |  Image Generation |  Video Editing |  Video Generation |  Audio Editing |  Audio Generation |  3D Generation |  Pointing Device
---|---|---|---|---|---|---|---|---|---
Idea-2-3D [429] | arXiv 2024 | ✗ | ✓ | ✗ | ✗ | ✗ | ✗ | ✓ | ✗
MLLM-Tool [420] | arXiv 2024 | ✓ | ✓ | ✓ | ✓ | ✗ | ✓ | ✗ | ✗
ControlLLM [407] | arXiv 2023 | ✓ | ✓ | ✓ | ✓ | ✗ | ✓ | ✗ | ✓
ModelScope-Agent [418] | EMNLP 2023 | ✗ | ✓ | ✗ | ✓ | ✗ | ✓ | ✗ | ✗
GPT4Tools [402] | NeurIPS 2023 | ✓ | ✓ | ✗ | ✗ | ✗ | ✗ | ✗ | ✗
InternGPT [218] | arXiv 2023 | ✓ | ✓ | ✓ | ✓ | ✗ | ✗ | ✗ | ✓
HuggingGPT [103] | NeurIPS 2023 | ✓ | ✓ | ✗ | ✓ | ✗ | ✗ | ✗ | ✗
Visual ChatGPT [217] | arXiv 2023 | ✓ | ✓ | ✗ | ✗ | ✗ | ✗ | ✗ | ✗
VISPROG [419] | CVPR 2022 | ✓ | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ | ✗
|  |  |  |  |  |  |  |  |

###  8.4 Summary

This section investigates LLMs with external tools to enhance their
capabilities, particularly in multimodal interactions. The motivation behind
this integration is to address LLMs’ limitations in processing information not
present in their training data, such as real-time or private data. This is
achieved by augmenting LLMs with tools that can provide additional information
or functionalities, which are invoked through natural language instructions.

The methods fall into two categories: training-free methods, which rely on
prompt engineering and in-context learning, and instruction-tuning methods,
which involve training LLMs to follow instructions more accurately. These
methods generally involve a three-stage framework: task planning, task
execution, and response generation. Task planning interprets instructions into
tool invocation schemes, Task execution involves the use of multimodal tools
for tasks like image generation or audio synthesis, and Response Generation
creates user-friendly responses from the execution outputs.

In summary, tool-augmented multimodal agents represent a significant
advancement in human-computer interaction, enabling more natural and versatile
interactions and fostering creative applications across various modalities.
However, they also present challenges that need to be addressed to realize
their full potential.

##  9 Generative AI Safety

The security concerns of multimodal-generated content are drawing increasing
attention. The research mainly focuses on mitigating biased and toxic content
generation, safeguarding copyright, and alleviating the impact caused by
fabricated content originating from generative models.

The vulnerability of generative models to attacks or malicious usage presents
unique challenges and has attracted significant research attention. Recent
research includes optimization-based attacks, prompt-level manipulations, and
data poisoning methods: (i) Optimization-based attacks demonstrate the
effectiveness of adversarial techniques to degrade model performance[430] or
induce biases and harmful outputs[431][432][433][434]. Adversarial attack and
detection-based defense research are also conducted in the field of audio and
video [435][435][436][437][438][439]. (ii) Prompt-level
attacks[440][441][442][443] reveal the risks at the inference level, where
human-crafted inputs can bypass safeguards and elicit unsafe outputs, bringing
the security challenges. (iii) Data poisoning methods[444][445] expose that
models can be manipulated by injecting malicious data inputs when the
integrity of training data is broken. These research works underscore the need
for comprehensive approaches to enhance model robustness, secure data
integrity, and recognize user’s unsafe interactions, addressing the generative
AI vulnerabilities.

Following the discussion on techniques to attack large generative models,
there are generally two main approaches to defending against undesirable
generation content. The first approach involves not modifying the existing
parameters of the model but employing detection mechanisms or manipulating the
input prompt context. [446] utilizes VLMs to detect and correct hate speech in
multimodal memes. [447][448] manipulate LLMs through in-context learning and
defending against jailbreaking attacks. For text-to-image generation, an
effective and efficient framework named Latent Guard[449] is proposed to
detect the unsafe input prompt. Compared to traditional blacklist-based
approaches, it is much more robust because the input prompt is checked in a
latent space. And the speed is much faster than the LLM-based unsafe detection
method. The second strategy enhances safety by aligning the model with human
preferences or values using alignment algorithms. The Proximal Policy
Optimization (PPO) algorithm, introduced in [450], has been widely used for
aligning LLMs. Recently, Direct Preference Optimization(DPO) and related
methods [451][452] introduce an improvement over [450] by presenting a more
efficient alignment algorithm capable of learning alignment directly from
preference data. For image generative models some preference datasets [453]
and alignment methods [454] have also been proposed.

Some studies focus on preventing multi-modal generative models from
fabricating facts. Powerful generative models that are capable of producing
highly realistic videos have attracted significant attention for their
potential misuse like Deepfakes [455][456][457]. Deepfake is a technique that
can generate realistic content of a certain identity in the form of an image
or video. Methods [458][459][460] focus on distinguishing deepfake videos by
detecting visual artifacts. While these detection methods are still
limited[461] that they do not insist on adversarial attacks so they are hard
to work well on new AIGC models that rely on the pattern seen in training.

Generated content can also bring up copyright issues. To solve these problems,
we may need to use data attribution and embed watermarks in the generated
outputs. Data origin attribution, which involves tracing the model prediction
to its original training data, can help find the data causing copyright
issues[462][463][464]. Watermark technique [465][466][467] can make generated
content distinguishable from real content by integrating the ownership
information in the generated content.

In addition, a variety of datasets also have been proposed to evaluate
different aspects of generative AI safety. The SafetyBench[468] dataset is a
multiple-choice question dataset for assessing unsafe contents which contains
11,435 items spanning 7 safety categories. This effort is complemented by the
GOAT-Bench dataset[469] that evaluates unsafe memes with over 6K diverse
topics, including implicit hate speech, gender discrimination, and
cyberbullying. Furthermore, datasets like ToViLaG and others [470][471][472]
have been developed specifically for visual LLMs, shedding light on the
challenges of addressing the generation of toxic content, such as offensive
texts and inappropriate images. These datasets offer a comprehensive
evaluation for further enhancing the safety of generative models, spanning
across text and image contents.

In summary, safety techniques for generative AI models can mitigate ethical
risks and protect copyrights. Detection and data algorithm technologies are
being utilized by commercial models. And some open-source projects also offer
safety checks for users by default. Watermarking and data tracing technologies
have made significant progress in alleviating copyright protection concerns.
Adopting safety technologies for impactful public projects can enhance the
security and trustworthiness of multimodal generation applications. Table XII
summarizes selected research related to generative AI safety issues.

TABLE XII: Overview of generative AI safety across various modalities and methods. The term ”Adv.” denotes ”adversarial attack”.  Name | Media | Type | Method | Venue
---|---|---|---|---
Wallace et al.[430] | T | Attack | Adv. | EMNLP 2019
Fu et al.[431] | T | Attack | Adv. | arXiv 2023
Image hijacks[432] | I | Attack | Adv. | arXiv 2023
Jones et al.[434] | T | Attack | Adv. | ICML 2023
Wu et al.[440] | T + I | Attack | Prompt | arXiv 2024
Xie et al.[441] | T | Attack | Prompt | NMI 2023
Liu et al.[442] | T | Attack | Prompt | arXiv 2023
Carlini et al.[444] | T | Attack | Data | arXiv 2023
Jia et al.[445] | T | Attack | Data | EMNLP 2017
Latent Guard[449] | T+I | Defense | Detection | arXiv 2023
Van et al.[446] | T+I | Defense | Detection | arXiv 2023
Wei et al.[447] | T | Defense | Prompt | arXiv 2023
Smoothllm[448] | T | Defense | Prompt | arXiv 2023
Rafailov et al.[451] | T | Defense | Alignment | arXiv 2023
Raft[454] | T+I | Defense | Alignment | TMLR 2023
Wodajo et al.[459] | V | Defense | Detection | arXiv 2023
Safetybench[468] | T | Dataset | - | arXiv 2023
GOAT-Bench[469] | T | Dataset | - | arXiv 2024
ToViLaG[470] | T+I | Dataset | - | EMNLP 2023
Figstep[471] | T+I | Dataset | - | arXiv 2023
Liu et al.[472] | T+I | Dataset | - | arXiv 2023

##  10 Applications

The rapid advancements in LLMs from companies like OpenAI, Google, Meta,
Baidu, and Microsoft have led to the development of a wide range of impressive
AI-powered applications. These models, such as GPT-4, Gemini, and Claude, have
demonstrated remarkable capabilities in multimodal tasks, particularly in
multimodal understanding.

The ability of these models to comprehend, interpret, and generate multimodal
content is a significant milestone in artificial intelligence. This multimodal
capability holds immense potential for various industries and showcases the
effectiveness of LLMs in multimodal generation.

In this section, we will review some remarkable applications that have already
been published. Beginning with image generation and progressing to video,
audio, and 3D generation, these showcases demonstrate the remarkable impact of
LLMs in generating content across multiple modalities.

###  10.1 Image

The rapid advancements in diffusion models have witnessed a remarkable
increase in the quality and realism of synthesized images. This has sparked
the emergence of numerous companies developing high-quality text-to-image
generation tools and multimodal conditional image editing or generation
solutions.

Midjourney[473] is making significant strides in the industry. It enables
content creation and design by offering users the ability to generate high-
quality, realistic images from text prompts. Its user-friendly interfaces and
robust performance make it a top choice for professionals and enthusiasts in
image generation.

Moreover, Stability AI[474] has provided a powerful open-source generation
model. The user community has provided various usage methods, indeed handing
over creativity and tools to the users. Opening up fine-tuning has created a
sizeable open-source image usage community. Even artists who are not computer
scientists can easily make their small models based on their basic models.
Users integrate various modal tools for deployment, enabling their image
generation model to play a better role.

DALLE3[152] stands out as a remarkable example of seamlessly integrating image
generation capabilities into the powerful ChatGPT4 chatbot[475]. With DALLE3,
users can generate and modify images through text-based prompts. The success
of DALL-E[150] and DALL-E 2[476] from OpenAI[477] has paved the way for highly
sophisticated image generation capabilities within LLMs. These models can
create detailed, photorealistic images from textual descriptions, allowing for
rapid prototyping and content creation across numerous domains.

In addition to the industry-leading solutions mentioned above, lots of text-
to-image generation tools have emerged that leverage LLMs to enhance the
robustness and overall quality of the user experience. By leveraging LLMs to
expand and refine captions, these tools can improve the quality of the
generated images and the platforms’ overall reliability and user-friendliness.

###  10.2 Video

With the advent of large-scale video generation models, individuals can now
obtain a high-quality video clip by simply inputting a textual description.
Users do not need specialized skills in traditional video production, such as
CG modeling, 3D modeling, or other professional knowledge. Users can generate
desired video clips prompted by a textual description and then assemble them
to create a captivating short film or animated video. Existing prominent tools
in this domain include commercial tools like Pika [478] and Runway’s Gen2
[479], as well as open-source video generation models such as AnimateDiff
[20], VideoCrafter [19] and SVD [251]. Regarding human video generation,
Heygen [480] is a popular tool widely applied in various domains, including
e-commerce, social media, and advertising videos.

After some demo videos generated by Sora[1] were released, significant
advancements have been made in terms of realism and prompt-following
capabilities, instilling greater confidence in the application of large-scale
text-to-video models. Many efforts have been made to reduce the video
production cost of the film and television industry.

###  10.3 Audio

The application of multimodal AI in audio has been explored for a long time.
The use cases are more well-defined, and the demand for customized and
diversified sounds is more established. Technologies such as text-to-speech
generation, sound transfer, music generation, and other audio generation have
demonstrated promising prospects in education, video dubbing, intelligent
terminals, voice assistance, and the medical field.

Microsoft’s Azure platform[481] is taking a leading position in speech
generation and is driving the integration of AI-generated sounds across short-
video platforms. Descript[482], an AI-based audio and video editor, can
transcribe speech in audio and video into text, enabling users to modify the
audio and video akin to editing a Word document. Moreover, numerous video
platforms, video editing software, and audio platforms have devoted
significant attention to applying multimodal models and audio generation.

Besides speech and audio generation, music generation is also a hot spot in
industries. The passion for music has driven countless AI researchers and
scientists to dedicate immense effort to advancing this field. Suno AI[483]
has ushered in the ”Sora era” of music generation, where users can now create
vivid, high-quality songs simply by providing a text prompt describing the
desired lyrical style. Additionally, companies like Stability Audio[484],
Google’s MusicFX[485], Tuneflow[486], and Deepmusic[487] have also provided
their music generation products, further expanding the capabilities in this
domain.

###  10.4 3D

The generation of 3D models is crucial across diverse domains, including film,
gaming, industrial design, architecture, interior design, product design, and
virtual reality. It provides realistic visual experiences and immersive
interactions, facilitating the creation of characters, scenes, products, and
virtual environments to enhance creativity and engagement. Meta[488] has been
heavily investing in 3D modeling and virtual reality technologies. Epic Games’
MetaHuman Creator[489], a cloud-streamed app designed to elevate real-time
digital human creation, is another noteworthy development that can be used in
conjunction with Unreal Engine, a state-of-the-art real-time engine and
editor.

As for 3D reconstruction and generation, Luma AI[490] is making significant
advancements, with their technology capable of generating 3D models from 2D
images, simplifying the process of creating 3D content. Other industry
players, such as Adobe[491] and Kaedim3D[492], are also making substantial
strides in this field. Adobe’s[491] 3D and AR tools enable the creation of
immersive content, while Kaedim3D’s[492] AI technology can convert 2D images
into 3D models.

Wonder Studio[493] is a powerful AI tool for character replacement in videos,
where it can replace the original characters in a video with user-created 3D
models, unlocking exciting possibilities for personalized content creation.

Recent advancements in Language-to-Language Models (LLMs) have revealed the
significant potential of text interaction and generation, opening new
possibilities for creating and manipulating 3D models using natural language
commands, making the process more intuitive and accessible. For instance,
SceneScript [494] from Meta is able to reconstruct environments and represent
the layout of physical spaces based on their powerful language-based model,
Llama[137]. However, compared to the image-to-3D, the text-to-3D is still the
research topic of companies like Meta [488], Google [495], Tencent [496],
_etc_.

Integrating LLMs in the 3D world is transforming how we create and interact
with digital content. As these technologies continue to evolve, we anticipate
even more interesting and practical applications.

####  10.4.1 Others

An AI-driven software normally needs to handle various modalities of input
data. This growing demand for multimodal solutions highlights the importance
of advanced AI models that can seamlessly integrate and process various data
types. For instance, AI-generated movies incorporate 3D technology for video,
music, and speech generation, collaborating with human artists to produce
high-quality cinematic experiences. Digital humans have also emerged as
prominent figures across various industries, from live streaming and gaming to
memorial services and large-scale interactive displays. Furthermore, LLM $+$
multimodal generative tools have found diverse applications in mathematics,
law, education, and robotics fields. In summary, we are currently witnessing
the dawn of multimodal generative models with LLMs, which will undoubtedly
change our lives.

##  11 Future prospects

LLMs augmented multimodal generation stands out as a promising research topic,
which harnesses the linguistic knowledge of LLMs to enhance the generation
across various modalities, such as image, video, 3D, and audio. This series of
approaches not only can improve the quality, diversity, and controllability of
the generated content but also can facilitate interactivity during multimodal
generation. In line with this direction, we intend to present prospects for
future works.

###  11.1 Technical Prospects

In this section, we focus on the technical prospects for multimodal
generation, which are expected to provide more insights and facilitate future
work.

####  11.1.1 High-resolution Generation

High-resolution multimodal generation is pivotal as it directly impacts the
quality and usability of generated content across various domains such as
image [497, 498], video [499, 500], audio and 3D generation [42]. Accordingly,
high fidelity also needs to be taken into account in audio generation[396, 47,
501, 501, 502]. The ability to produce High-resolution multimodal generation
is crucial for applications requiring detailed and realistic representations,
ranging from virtual reality to film production. Because it enhances the
perceptual experience, provides more information for analysis, and improves
the performance of subsequent tasks like object recognition and scene
understanding.

LLMs hold the potential to address the challenges in a high-resolution
multimodal generation. They can provide a more seamless integration of visual
and textual modalities, offering a dialogue-based interface and instruction-
following capabilities [68]. It could enhance the generation process by
improving the understanding of complex instructions and generating more
accurate and diverse outputs. Recent advancements across different modalities,
including image [10, 503, 68, 504], video [19, 273, 505, 506], 3D [353, 507,
508] and audio [44, 46, 100], have led to significant improvements in the
quality of generated content. We are extremely eager to witness the increase
in future works that integrate LLMs, thereby offering enhanced support for
high-resolution generation. Moreover, high-resolution content generation
typically entails substantial hardware expenses and time costs. Consequently,
the efficient generation of high-resolution content is also a topic worthy of
research.

####  11.1.2 Long-term Sequence Generation

Long-term sequence generation is crucial for creating immersive experiences in
the video [506, 509] and the audio [510, 110, 44, 511]. In video, it allows
for the portrayal of evolving scenes and narratives, while in audio, it
supports the development of music and dialogue that can adapt and flow over
time. The ability to generate long sequences over time is not just a technical
challenge but also a creative one, where the model must understand and predict
complex patterns and progressions. It should maintain continuity, prevent
repetition, and introduce novel elements that align with the overarching theme
and input conditions. Only when we are able to generate long sequences for
video and audio, it can potentially lead to practical significance.

Recent advancements in LLMs, such as OpenAI’s GPT series and Meta’s LLaMA
[137], addressing the challenges of long-term sequence generation. LLMs build
upon pre-trained language representations and fine-tuning techniques to
capture intricate patterns and dependencies in text data, enabling them to
generate coherent and contextually relevant sequences over extended lengths.
By harnessing the contextual understanding and generative capabilities of
LLMs, researchers can explore long-term sequence generation. For example,
fine-tuning pre-trained LLMs on multimodal datasets could enable them to
generate coherent and diverse sequences across different modalities, including
video and audio. Additionally, techniques such as prompt engineering and
conditioning can guide the generation process toward desired outcomes,
allowing for the creation of long sequences with specific themes or
narratives. We argue that LLMs can enhance coherence and consistency for
generated long sequence generation.

Generally, long-term sequence generation represents a complex yet compelling
area of research for various domains. By leveraging the capabilities of LLMs
and addressing the associated challenges, researchers can unlock new
opportunities for creating immersive and engaging sequences that captivate
audiences and push the boundaries of content creation and storytelling.

####  11.1.3 More Accurate and Fine-grained Generation Control

Accurate and fine-grained generation control is a significant topic in AIGC
for several reasons. First, it allows for the creation of more realistic and
high-quality multimodal content. This is particularly important in fields such
as entertainment, advertising, and education, where high-quality content can
significantly enhance user experience. Second, fine-grained control can
facilitate more effective communication between humans and AI. For instance,
an AI model with fine-grained control can generate a specific image or sound
based on a user’s detailed description, thereby improving the interaction
between the user and the AI. Last, fine-grained control can also contribute to
the advancement of other AI fields. For example, in reinforcement learning, an
AI agent can learn more effectively if it can generate detailed and accurate
simulations of its environment.

Lots of methods [68, 512, 513, 19] have been proposed to address accurate and
fine-grained generation control. However, these methods still have some
limitations. For instance, they still struggle with generating fine details,
such as fingers or body parts, which can lead to unrealistic outputs.
Moreover, they may also fail to accurately capture the nuances in the control
signals, resulting in a mismatch between the generated content and the control
signal.

Large language models have shown remarkable capabilities in understanding and
generating text. By leveraging these capabilities, we can potentially improve
the accuracy and granularity of generation control. One of prominent examples
is text rendering [514, 194, 62, 515, 195] on images or videos. It has been
observed that by using powerful language models, such as T5-XXL, as the
encoder, the image generation models will exhibit better spelling ability. In
this context, the integration of more potent LLMs into generative models is
worthy of further exploration. Generally, a large language model can be
trained to better understand the nuances in the control signals, thereby
improving the alignment between the control signal and the generated content.

####  11.1.4 Multi-view Consistency

Multi-view consistency (MVC) is a fundamental aspect of visual generation,
particularly in 3D generation, ensuring the coherence and continuity of an
object’s appearance from different viewpoints. This consistency is crucial for
applications in augmented reality (AR), virtual reality (VR), and computer
graphics, where users interact with 3D objects in a seemingly real-world
context. Inconsistent appearances can break immersion and lead to a less
realistic experience. The significance of multi-view consistency lies in its
ability to provide a seamless and integrated perception of 3D objects,
enhancing the user’s experience and interaction with digital content.

MVC is particularly challenging due to the complex nature of translating 2D
images into consistent 3D models, where issues such as occlusions, lighting
variations, and geometric distortions can arise. Recent advancements pay lots
of attention to multi-view consistency. In 3D generation, Sculpt3D [516]
introduces a sparse 3D prior to improve consistency without retraining the 2D
diffusion model. HarmonyView [517] addresses the balance between consistency
and diversity by employing a diffusion sampling technique. Additionally,
MVDream [333] lacks comprehensive multi-view knowledge or 3D-awareness during
score distillation, leading to unstable generation and artifacts. In image and
video generation, the works [518, 519] have contributed to the field by
focusing on novel view synthesis and multi-view image generation based on
large video datasets, respectively.

Despite these advancements, there are still several challenges to be further
studied: 1) Limited Generalization: Many methods struggle to generalize well
across diverse datasets and object categories. 2) Struggling in Complex
Geometries: Accurately rendering objects with complex geometries or
textureless surfaces. Since linguistic prompts can provide more prior
knowledge for generation, we believe it can enhance the multi-view consistency
as well as generation quality by incorporating LLMs in the pipeline.

####  11.1.5 Unified Training for Multimodal Generation

Multimodal generation is defined as the ability to simultaneously create
content across different modalities, including images, videos, 3D objects, and
audio. Currently, most methods [68, 10, 251, 69, 341, 81, 44, 110] only focus
on one aspect such as text-to-image or text-to-video synthesis. This
inevitably prompts consideration: Can a single model possess the capability to
generate multiple modalities?

Several recent works [133, 520, 270, 521, 381, 522] have made notable strides
in feature alignment for text, image, video, audio and other modalities. Some
multimodal agents [103, 217, 402, 218, 407] offer awesome generation
capabilities for various modalities, but atom tools they used are not jointly
trained. Furthermore, the pioneer works [133, 93, 134] have made their
preliminary efforts to explore how to generate multimodal content in one
model.

However, despite these advancements, challenges persist in achieving effective
unified training for multimodal generation. One prominent obstacle lies in
feature alignment across different modalities, as each modality possesses
distinct statistical properties and underlying structures, necessitating
robust alignment mechanisms to ensure consistency and coherence in generated
outputs. Moreover, the mutual interference during training poses a significant
hurdle, as optimizing for multiple modalities concurrently may lead to
conflicts or competition among modal-specific objectives, hindering the
overall training stability and convergence. In addition, the inherent
complexity of multimodal data imposes computational overhead, necessitating
efficient algorithms and scalable architectures to handle the diverse
modalities efficiently.

The pursuit of unified training for multimodal generation represents a crucial
advancement in AI research, offering immense potential for advancing the
capabilities of generative models across diverse domains. In the future, we
even look forward to models capable of generating different modalities in an
interleaved manner.

####  11.1.6 Efficient Training and Deployment Strategies

Efficient training and deployment strategies also still remain to be studied
in multimodal generation. As datasets and models continue to scale
exponentially, the challenge of achieving efficient training and deployment
becomes increasingly significant, in line with the scaling law, which posits
that the computational resources required for training and deploying models
grow rapidly with model size and dataset size [523]. Efficient strategies are
essential not only for reducing computational costs but also for enabling
real-time or resource-constrained applications of multimodal generation
technologies. By minimizing computational overhead and resource utilization,
efficient training and deployment strategies not only reduce time and energy
costs but also enhance scalability and accessibility, democratizing access to
advanced generative capabilities across diverse domains.

Several approaches have been proposed to address the challenge of efficient
training in multimodal generation. Several works study low-rank approximation
techniques, such as LoRA [225] and Q-LoRA [427], which aim to reduce the
computational complexity of model training by approximating weight matrices
with low-rank structures. Additionally, mixed precision training [524], which
involves using reduced precision (e.g., 16-bit floating-point) arithmetic for
certain computations, has emerged as a powerful tool for accelerating training
without sacrificing model accuracy. Despite their effectiveness, these
efficient training techniques still have limitations. Low-rank approximation
methods may introduce approximation errors that degrade the quality of
generated outputs, particularly in scenarios where high-fidelity synthesis is
crucial. Similarly, mixed precision training may encounter numerical
instability issues, especially when dealing with extremely large models or
datasets, leading to suboptimal convergence or even training failures.

Efficient deployment strategies, such as quantization [525, 526, 527, 528] to
int8 or even int4 precision, offer another avenue for reducing the
computational and memory requirements of multimodal generation models during
inference. By quantizing model weights and activations to lower precision
formats, significant savings in memory bandwidth and computational resources
can be achieved, enabling faster inference and deployment on resource-
constrained devices. However, quantization also presents issues, particularly
in preserving model accuracy and generative quality. Lowering the precision of
model parameters and activations can lead to information loss and degradation
in output fidelity, especially in complex multimodal synthesis tasks where
fine-grained details are crucial.

In conclusion, efficient training and deployment strategies are indispensable
for realizing the full potential of multimodal generation technologies across
diverse applications. By overcoming the challenges associated with scalability
and resource constraints, researchers can accelerate the adoption of
multimodal generation systems in real-world scenarios, unlocking new
possibilities for content creation, human-computer interaction, and beyond.

####  11.1.7 Ethically Safe Content Generation

While there have been many works exploring how to strengthen the security of
generative models of text and images[446][447][448], the increasing
capabilities of video generation models should raise security concerns. Due to
the emergence of security issues like Deepfakes[457] even with the use of
previously less powerful video models, the growing strength of video models
magnifies the societal impact of potential risks.

Adversarial attacks have demonstrated effective transferability from open-
source models to commercial closed-source models[433]. Future commercial
closed-source models should consider guarding against attacks from open-source
models, such as by implementing corresponding adversarial token detection
mechanisms. Simultaneously, efforts can also be considered to mitigate the
impact of transferable attacks like minimizing the commercial models
similarities with open-source models, such as in network architecture, data
usage, and weights.

Currently, most research articles focus on ensuring security from individual
perspectives such as detection[446], alignment[450], post-hoc checking[529],
etc. Each of these methods generally has their own advantages and
disadvantages. For instance, detection techniques offer fast checking but may
overlook certain vulnerabilities. Alignment methods also cannot guarantee that
the data used for training alignment covers all security cases. Additionally,
post-hoc checking can be computationally expensive, especially for the
generation of images and videos. There has not been much work integrating
these techniques into a holistic system to ensure the security of large
generative models. For example, the system can first detect user inputs, then
simultaneously apply securely aligned models, and finally conduct security
checks on the output to determine whether to proceed. Integrating these
techniques can lead to higher efficiency and security.

###  11.2 Application Prospects

In this section, we make an effort to build blueprints for the application of
multimodal generative models.

####  11.2.1 Semantic Audio Synthesis

Semantic audio synthesis involves generating of audio signals based on
semantic descriptions or contextual cues, enabling the creation of immersive
auditory experiences with specific characteristics or attributes. Multimodal
generative models offer a promising approach to semantic audio synthesis by
leveraging contextual information from other modalities, such as text or
images. For instance, text-based descriptions of soundscapes or music
compositions can be translated into audio waveforms using generative models
trained on multimodal data. Similarly, images or videos depicting scenes or
environments can inform the generation of corresponding audio accompaniments,
enhancing the realism and richness of multimedia content. By integrating
semantic information across modalities, multimodal generative models enable
the creation of highly personalized and contextually relevant audio
experiences, spanning applications in entertainment, virtual reality, and
assistive technologies.

####  11.2.2 Multi-modal Storytelling.

Multimodal storytelling involves fusing different modalities to craft
compelling narratives that engage multiple senses simultaneously. This
approach not only enriches storytelling experiences but also opens up new
avenues for creative expression and audience engagement. In multimodal
storytelling, the synthesis of content can occur in several directions.

From text prompts to image sequences, multimodal storytelling can begin with a
topic, a script or even a story outline, which serves as the basis for
generating complementary modalities such as text, image sequences. For
instance, given a prompt about a fantastical adventure, a multimodal
generative model could generate vivid imagery depicting characters and scenes,
produce an animated video sequence illustrating key events, or compose a
thematic musical score to accompany the narrative.

From text prompts or images to videos and audio, in this scenario, an image
serves as the starting point for generating accompanying textual descriptions,
video sequences, or audio narratives. For example, given an image depicting a
scenic landscape, a multimodal generative model can generate descriptive text
detailing the setting, produce a video animation depicting the scene in
motion, or create an immersive audio experience capturing ambient sounds and
atmosphere.

Multimodal storytelling holds immense potential for enhancing traditional
narrative formats and creating immersive, multi-sensory experiences that
resonate with audiences across various mediums and platforms. By harnessing
the capabilities of multimodal generative models, storytellers, content
creators, and media producers can unlock new dimensions of creativity and
engagement in the digital age.

####  11.2.3 Interactive Content Design

Interactive content design aims to create and manipulate media elements in
real time, empowering users to actively participate in the creative process.
Traditionally, content creation processes involve iterative steps of ideation,
design, and refinement, often requiring extensive time and resources. However,
with the interactive capabilities afforded by foundation generative models,
creators can swiftly explore a multitude of design possibilities, rapidly
iterate on concepts, and refine compositions in real time, thereby
streamlining the overall creative workflow.

By enabling real-time interaction and manipulation of media elements, it can
improve the efficiency of multimodal generative models. Creators can
efficiently experiment with different visual and auditory elements, explore
diverse artistic styles, and generate high-quality content without the need
for extensive manual labor or specialized expertise. Consequently, this not
only accelerates the production process but also minimizes the expenses
incurred in hiring additional resources or outsourcing tasks. Moreover, the
integration of multimodal generative models in Interactive Content Design
contributes to the democratization of creativity by lowering the barriers to
entry for aspiring artists and designers. Unlike traditional design tools that
often require proficiency in complex software interfaces or artistic skills,
these models offer intuitive and accessible interfaces that empower
individuals from diverse backgrounds to engage in creative expression. By
democratizing access to advanced content creation capabilities, these tools
foster inclusivity and diversity within the creative community, enabling a
broader range of voices to be heard.

Looking ahead, the evolution of multimodal generative models holds exciting
prospects for the future of Interactive Content Design. As advancements
continue to expand the scope and fidelity of generated content across
different modalities, we can anticipate even greater opportunities for
innovation in areas such as virtual reality, augmented reality, and immersive
storytelling. Additionally, ongoing research efforts aimed at enhancing the
interpretability, controllability, and scalability of these models will
further fuel their adoption in diverse creative domains, paving the way for
transformative changes in how we conceive, design, and interact with digital
content.

####  11.2.4 3D Scene Generation

3D scene generation refers to the creation of immersive and realistic
environments in virtual worlds, games, simulations, and architectural
visualization. This application domain leverages multimodal generative models
to synthesize complex 3D scenes comprising objects, textures, lighting, and
spatial arrangements. The ability to generate 3D scenes has profound
implications for various industries, including entertainment, education,
design, and virtual reality.

In the context of games and virtual environments, multimodal generative models
can automate the process of scene creation, reducing the reliance on manual
modeling and asset creation. By inputting textual descriptions or conceptual
sketches, developers can generate entire 3D environments populated with
interactive objects, characters, and atmospheric effects. This not only
accelerates the game development pipeline but also enables the creation of
dynamic and immersive gameplay experiences. Moreover, in architectural
visualization and design, multimodal generative models can assist architects,
urban planners, and designers in visualizing and exploring different design
options. By inputting architectural blueprints or design parameters, designers
can generate realistic 3D renderings of buildings, landscapes, and interior
spaces, allowing for rapid iteration and exploration of design concepts. This
facilitates collaboration, decision-making, and communication among
stakeholders involved in the design process.

By harnessing the capabilities of multimodal generative models, 3D scene
generation might revolutionize how virtual environments are created,
experienced, and interacted with. Whether in games, simulations, or
architectural visualization, the ability to generate immersive and realistic
3D scenes programmatically opens up new possibilities for creativity,
exploration, and storytelling in virtual worlds.

####  11.2.5 Customizable Avatars

Customizable avatars represent digital representations of users that can be
personalized and adapted to reflect individual preferences, identities, and
characteristics.

Multimodal generative models offer a compelling approach to customizable
avatar creation by synthesizing diverse media types such as images, text, and
audio to create lifelike and expressive avatars. For example, generative
models trained on multimodal data can generate photorealistic images of
avatars based on textual descriptions or user preferences, incorporating
details such as facial features, clothing styles, and expressions. Similarly,
audio-based avatars can be generated using voice synthesis techniques,
enabling avatars to communicate with users using natural-sounding voices that
reflect their personalities or preferences. By enabling the creation of
customizable avatars across multiple modalities, multimodal generative models
empower users to express themselves in virtual environments, fostering deeper
engagement and personalization in social interactions, gaming, and virtual
communication platforms.

Currently, there are several aspects that could be further studied: 1)
Personalization and Customization: Multimodal generative models can generate
avatars that closely resemble users based on input parameters such as facial
features, body type, and clothing preferences. Users can interactively
customize their avatars using intuitive interfaces, adjusting attributes such
as hairstyle, facial expression, and accessories in real time. 2) Emotional
Expression and Limb Movements: Avatars generated by multimodal models can
exhibit a wide range of emotional expressions, gestures and physical
movements, enhancing their ability to convey nonverbal communication cues in
virtual environments. Users can dynamically control their avatar’s behaviors,
allowing for more immersive social interactions and collaborative experiences
in virtual worlds. 3) Integration with Virtual Environments: Customizable
avatars can be seamlessly integrated into various virtual environments,
including social platforms, online games, and virtual reality applications.
Users can navigate these environments using their avatars, interacting with
other users and objects in real time, fostering a sense of presence and
belonging in digital spaces.

###  11.3 Towards the World Model

World models [530, 531, 532, 533, 534] have recently emerged as a hotspot
topic. Many renowned researchers have expressed that world models will come
true in the foreseeable future, and researchers around the world hold great
expectations for this development. We found that all the topics mentioned in
the survey correspond precisely to the principal components of the world
modeling, encompassing perceptual modalities such as vision, audition, and
speech, as well as spatial understanding and generation. Once world models
advance to a usable stage, they will endow numerous industries with new
possibilities. We here highlight several core applications for reference.

Multimodal education and communication. World models hold immense promise for
revolutionizing education and communication by facilitating multimodal
learning experiences and immersive interactions. By integrating diverse
sensory modalities such as text, images, audio, and video, these models enable
the creation of rich educational content that caters to different learning
styles and preferences. Furthermore, they empower learners to engage with
complex concepts and environments more intuitively and interactively, thereby
enhancing comprehension and retention. Additionally, world models facilitate
seamless communication by enabling the synthesis of natural and expressive
multimodal dialogue, fostering more engaging and personalized interactions in
virtual learning environments and online collaboration platforms.

Movie Generation. The application of world models in movie generation
represents a paradigm shift in filmmaking, offering filmmakers unprecedented
creative freedom and flexibility. By leveraging multimodal generative
techniques, filmmakers can seamlessly integrate various elements such as
dialogue, visuals, sound effects, and music to craft immersive cinematic
experiences that resonate with audiences on a deeper level. Moreover, world
models enable the generation of dynamic and personalized narratives tailored
to individual viewer preferences, thereby enhancing viewer engagement and
immersion. Furthermore, these models facilitate the exploration of alternative
storytelling formats and experimental filmmaking techniques, pushing the
boundaries of cinematic creativity and expression.

Metaverse. The emergence of the metaverse presents exciting opportunities for
leveraging world models to create immersive and interactive virtual worlds. By
synthesizing multimodal sensory experiences, including visual, auditory, and
haptic feedback, these models enable the creation of highly realistic and
immersive virtual environments that blur the boundaries between physical and
digital reality. Moreover, world models facilitate the development of
intelligent virtual agents and NPCs that exhibit lifelike behaviors and
interactions, enhancing the sense of presence and social immersion within the
metaverse. Additionally, these models empower users to customize and
personalize their virtual experiences, fostering creativity and exploration
within digital worlds.

##  12 Conclusion

In this survey, we systematically review multimodal editing and generation
works augmented by LLMs, delving into the advancements across various
modalities including image, video, 3D and audio. Following that, multimodal
agents that integrate a plethora of state-of-the-art generative models are
thoroughly discussed with diverse case studies. We also investigate the
concern for the safety of multimodal generative models, which play an
indispensable role in practical applications. Our comprehensive review
highlights the significant contributions of LLMs in enhancing the quality and
capabilities of generative systems. Looking forward, we anticipate further
innovations at the intersection of AI and generative content, driving progress
toward a more unified and capable multimodal generation framework. In
conclusion, we ardently expect that our investigation will provide insights
and inspiration for the development of the multimodal generation, particularly
the world models, which have garnered the attention and anticipation of the
majority of researchers.

## References

  * [1] OpenAI, “Video generation models as world simulators,” OpenAI, Tech. Rep., 2024. [Online]. Available: https://openai.com/research/video-generation-models-as-world-simulators
  * [2] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “Bert: Pre-training of deep bidirectional transformers for language understanding,” _arXiv preprint arXiv:1810.04805_ , 2018.
  * [3] A. Radford, K. Narasimhan, T. Salimans, I. Sutskever _et al._ , “Improving language understanding by generative pre-training,” 2018.
  * [4] A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, I. Sutskever _et al._ , “Language models are unsupervised multitask learners,” _OpenAI blog_ , vol. 1, no. 8, p. 9, 2019.
  * [5] T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell _et al._ , “Language models are few-shot learners,” _Advances in neural information processing systems_ , vol. 33, pp. 1877–1901, 2020.
  * [6] J. Achiam, S. Adler, S. Agarwal, L. Ahmad, I. Akkaya, F. L. Aleman, D. Almeida, J. Altenschmidt, S. Altman, S. Anadkat _et al._ , “Gpt-4 technical report,” _arXiv preprint arXiv:2303.08774_ , 2023.
  * [7] OpenAI, “Chatgpt: A language model for conversational ai,” OpenAI, Tech. Rep., 2023. [Online]. Available: https://www.openai.com/research/chatgpt
  * [8] Y. Li, C. Wang, and J. Jia, “Llama-vid: An image is worth 2 tokens in large language models,” _arXiv preprint arXiv:2311.17043_ , 2023.
  * [9] P. Gao, J. Han, R. Zhang, Z. Lin, S. Geng, A. Zhou, W. Zhang, P. Lu, C. He, X. Yue _et al._ , “Llama-adapter v2: Parameter-efficient visual instruction model,” _arXiv preprint arXiv:2304.15010_ , 2023.
  * [10] R. Rombach, A. Blattmann, D. Lorenz, P. Esser, and B. Ommer, “High-resolution image synthesis with latent diffusion models,” in _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , 2022, pp. 10 684–10 695.
  * [11] A. Nichol, P. Dhariwal, A. Ramesh, P. Shyam, P. Mishkin, B. McGrew, I. Sutskever, and M. Chen, “Glide: Towards photorealistic image generation and editing with text-guided diffusion models,” _arXiv preprint arXiv:2112.10741_ , 2021.
  * [12] A. Ramesh, P. Dhariwal, A. Nichol, C. Chu, and M. Chen, “Hierarchical text-conditional image generation with clip latents,” _arXiv preprint arXiv:2204.06125_ , vol. 1, no. 2, p. 3, 2022.
  * [13] C. Saharia, W. Chan, S. Saxena, L. Li, J. Whang, E. L. Denton, K. Ghasemipour, R. Gontijo Lopes, B. Karagol Ayan, T. Salimans _et al._ , “Photorealistic text-to-image diffusion models with deep language understanding,” _Advances in Neural Information Processing Systems_ , vol. 35, pp. 36 479–36 494, 2022.
  * [14] Y. He, T. Yang, Y. Zhang, Y. Shan, and Q. Chen, “Latent video diffusion models for high-fidelity long video generation,” _arXiv preprint arXiv:2211.13221_ , 2022.
  * [15] D. Zhou, W. Wang, H. Yan, W. Lv, Y. Zhu, and J. Feng, “Magicvideo: Efficient video generation with latent diffusion models,” _arXiv preprint arXiv:2211.11018_ , 2022.
  * [16] U. Singer, A. Polyak, T. Hayes, X. Yin, J. An, S. Zhang, Q. Hu, H. Yang, O. Ashual, O. Gafni _et al._ , “Make-a-video: Text-to-video generation without text-video data,” _arXiv preprint arXiv:2209.14792_ , 2022.
  * [17] J. Ho, W. Chan, C. Saharia, J. Whang, R. Gao, A. Gritsenko, D. P. Kingma, B. Poole, M. Norouzi, D. J. Fleet _et al._ , “Imagen video: High definition video generation with diffusion models,” _arXiv preprint arXiv:2210.02303_ , 2022.
  * [18] R. Villegas, M. Babaeizadeh, P.-J. Kindermans, H. Moraldo, H. Zhang, M. T. Saffar, S. Castro, J. Kunze, and D. Erhan, “Phenaki: Variable length video generation from open domain textual descriptions,” in _International Conference on Learning Representations_ , 2022.
  * [19] H. Chen, M. Xia, Y. He, Y. Zhang, X. Cun, S. Yang, J. Xing, Y. Liu, Q. Chen, X. Wang _et al._ , “Videocrafter1: Open diffusion models for high-quality video generation,” _arXiv preprint arXiv:2310.19512_ , 2023.
  * [20] Y. Guo, C. Yang, A. Rao, Y. Wang, Y. Qiao, D. Lin, and B. Dai, “Animatediff: Animate your personalized text-to-image diffusion models without specific tuning,” _arXiv preprint arXiv:2307.04725_ , 2023.
  * [21] O. Bar-Tal, H. Chefer, O. Tov, C. Herrmann, R. Paiss, S. Zada, A. Ephrat, J. Hur, Y. Li, T. Michaeli _et al._ , “Lumiere: A space-time diffusion model for video generation,” _arXiv preprint arXiv:2401.12945_ , 2024.
  * [22] R. Girdhar, M. Singh, A. Brown, Q. Duval, S. Azadi, S. S. Rambhatla, A. Shah, X. Yin, D. Parikh, and I. Misra, “Emu video: Factorizing text-to-video generation by explicit image conditioning,” _arXiv preprint arXiv:2311.10709_ , 2023.
  * [23] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark _et al._ , “Learning transferable visual models from natural language supervision,” in _International conference on machine learning_. PMLR, 2021, pp. 8748–8763.
  * [24] A. Sanghi, H. Chu, J. G. Lambourne, Y. Wang, C.-Y. Cheng, M. Fumero, and K. R. Malekshan, “Clip-forge: Towards zero-shot text-to-shape generation,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2022, pp. 18 603–18 613.
  * [25] N. Mohammad Khalid, T. Xie, E. Belilovsky, and T. Popa, “Clip-mesh: Generating textured meshes from text using pretrained image-text models,” in _SIGGRAPH Asia 2022 conference papers_ , 2022, pp. 1–8.
  * [26] O. Michel, R. Bar-On, R. Liu, S. Benaim, and R. Hanocka, “Text2mesh: Text-driven neural stylization for meshes,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2022, pp. 13 492–13 502.
  * [27] C. Wang, R. Jiang, M. Chai, M. He, D. Chen, and J. Liao, “Nerf-art: Text-driven neural radiance fields stylization,” _IEEE Transactions on Visualization and Computer Graphics_ , 2023.
  * [28] B. Kerbl, G. Kopanas, T. Leimkühler, and G. Drettakis, “3d gaussian splatting for real-time radiance field rendering,” _ACM Transactions on Graphics_ , vol. 42, no. 4, July 2023. [Online]. Available: https://repo-sam.inria.fr/fungraph/3d-gaussian-splatting/
  * [29] T. Yi, J. Fang, J. Wang, G. Wu, L. Xie, X. Zhang, W. Liu, Q. Tian, and X. Wang, “Gaussiandreamer: Fast generation from text to 3d gaussians by bridging 2d and 3d diffusion models,” _arXiv preprint arXiv:2310.08529_ , 2023.
  * [30] J. Tang, J. Ren, H. Zhou, Z. Liu, and G. Zeng, “Dreamgaussian: Generative gaussian splatting for efficient 3d content creation,” _arXiv preprint arXiv:2309.16653_ , 2023.
  * [31] L. Höllein, A. Cao, A. Owens, J. Johnson, and M. Nießner, “Text2room: Extracting textured 3d meshes from 2d text-to-image models,” in _Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)_ , October 2023, pp. 7909–7920.
  * [32] Y. Liang, X. Yang, J. Lin, H. Li, X. Xu, and Y. Chen, “LucidDreamer: Towards high-fidelity text-to-3d generation via interval score matching,” _https://arxiv.org/abs/2311.11284_ , 2023.
  * [33] X. Yu, Y.-C. Guo, Y. Li, D. Liang, S.-H. Zhang, and X. Qi, “Text-to-3d with classifier score distillation,” _https://arxiv.org/abs/2310.19415_ , 2023.
  * [34] W. Li, R. Chen, X. Chen, and P. Tan, “SweetDreamer: Aligning geometric priors in 2d diffusion for consistent text-to-3d,” _https://arxiv.org/abs/2310.02596_ , 2023.
  * [35] Z. Wang, C. Lu, Y. Wang, F. Bao, C. Li, H. Su, and J. Zhu, “Prolificdreamer: High-fidelity and diverse text-to-3d generation with variational score distillation,” _arXiv preprint arXiv:2305.16213_ , 2023.
  * [36] J. Lorraine, K. Xie, X. Zeng, C.-H. Lin, T. Takikawa, N. Sharp, T.-Y. Lin, M.-Y. Liu, S. Fidler, and J. Lucas, “Att3d: Amortized text-to-3d object synthesis,” in _International Conference on Computer Vision ICCV_ , 2023.
  * [37] J. Xu, X. Wang, W. Cheng, Y.-P. Cao, Y. Shan, X. Qie, and S. Gao, “Dream3D: Zero-shot text-to-3d synthesis using 3d shape prior and text-to-image diffusion models,” _https://arxiv.org/abs/2212.14704_ , 2023.
  * [38] J. Zhu and P. Zhuang, “HiFA: High-fidelity text-to-3d with advanced diffusion guidance,” _https://arxiv.org/abs/2305.18766_ , 2023.
  * [39] R. Chen, Y. Chen, N. Jiao, and K. Jia, “Fantasia3d: Disentangling geometry and appearance for high-quality text-to-3d content creation,” in _Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)_ , October 2023.
  * [40] C. Tsalicoglou, F. Manhardt, A. Tonioni, M. Niemeyer, and F. Tombari, “Textmesh: Generation of realistic 3d meshes from text prompts,” _arXiv preprint arXiv:2304.12439_ , 2023.
  * [41] B. Poole, A. Jain, J. T. Barron, and B. Mildenhall, “Dreamfusion: Text-to-3d using 2d diffusion,” _arXiv preprint arXiv:2209.14988_ , 2022.
  * [42] C.-H. Lin, J. Gao, L. Tang, T. Takikawa, X. Zeng, X. Huang, K. Kreis, S. Fidler, M.-Y. Liu, and T.-Y. Lin, “Magic3d: High-resolution text-to-3d content creation,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2023, pp. 300–309.
  * [43] J. Seo, W. Jang, M.-S. Kwak, J. Ko, H. Kim, J. Kim, J.-H. Kim, J. Lee, and S. Kim, “Let 2d diffusion model know 3d-consistency for robust text-to-3d generation,” _arXiv preprint arXiv:2303.07937_ , 2023.
  * [44] H. Liu, Z. Chen, Y. Yuan, X. Mei, X. Liu, D. Mandic, W. Wang, and M. D. Plumbley, “Audioldm: Text-to-audio generation with latent diffusion models,” _arXiv preprint arXiv:2301.12503_ , 2023.
  * [45] H. Liu, Q. Tian, Y. Yuan, X. Liu, X. Mei, Q. Kong, Y. Wang, W. Wang, Y. Wang, and M. D. Plumbley, “Audioldm 2: Learning holistic audio generation with self-supervised pretraining,” _arXiv preprint arXiv:2308.05734_ , 2023.
  * [46] F. Kreuk, G. Synnaeve, A. Polyak, U. Singer, A. Défossez, J. Copet, D. Parikh, Y. Taigman, and Y. Adi, “Audiogen: Textually guided audio generation,” _arXiv preprint arXiv:2209.15352_ , 2022.
  * [47] A. Agostinelli, T. I. Denk, Z. Borsos, J. Engel, M. Verzetti, A. Caillon, Q. Huang, A. Jansen, A. Roberts, M. Tagliasacchi _et al._ , “Musiclm: Generating music from text,” _arXiv preprint arXiv:2301.11325_ , 2023.
  * [48] J. Copet, F. Kreuk, I. Gat, T. Remez, D. Kant, G. Synnaeve, Y. Adi, and A. Défossez, “Simple and controllable music generation,” _Advances in Neural Information Processing Systems_ , vol. 36, 2024.
  * [49] S. Forsgren and H. Martiros, “Riffusion-stable diffusion for real-time music generation, 2022,” _URL https://riffusion. com/about_ , vol. 6, 2022.
  * [50] X. Tan, J. Chen, H. Liu, J. Cong, C. Zhang, Y. Liu, X. Wang, Y. Leng, Y. Yi, L. He _et al._ , “Naturalspeech: End-to-end text-to-speech synthesis with human-level quality,” _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , 2024.
  * [51] K. Shen, Z. Ju, X. Tan, Y. Liu, Y. Leng, L. He, T. Qin, S. Zhao, and J. Bian, “Naturalspeech 2: Latent diffusion models are natural and zero-shot speech and singing synthesizers,” _arXiv preprint arXiv:2304.09116_ , 2023.
  * [52] Z. Ju, Y. Wang, K. Shen, X. Tan, D. Xin, D. Yang, Y. Liu, Y. Leng, K. Song, S. Tang _et al._ , “Naturalspeech 3: Zero-shot speech synthesis with factorized codec and diffusion models,” _arXiv preprint arXiv:2403.03100_ , 2024.
  * [53] C. Wang, S. Chen, Y. Wu, Z. Zhang, L. Zhou, S. Liu, Z. Chen, Y. Liu, H. Wang, J. Li _et al._ , “Neural codec language models are zero-shot text to speech synthesizers,” _arXiv preprint arXiv:2301.02111_ , 2023.
  * [54] Z. Jiang, J. Liu, Y. Ren, J. He, C. Zhang, Z. Ye, P. Wei, C. Wang, X. Yin, Z. Ma _et al._ , “Mega-tts 2: Zero-shot text-to-speech with arbitrary length speech prompts,” _arXiv preprint arXiv:2307.07218_ , 2023.
  * [55] Y. Ren, C. Hu, X. Tan, T. Qin, S. Zhao, Z. Zhao, and T.-Y. Liu, “Fastspeech 2: Fast and high-quality end-to-end text to speech,” _arXiv preprint arXiv:2006.04558_ , 2020.
  * [56] Y. Ge, Y. Ge, Z. Zeng, X. Wang, and Y. Shan, “Planting a seed of vision in large language model,” _arXiv preprint arXiv:2307.08041_ , 2023.
  * [57] L. Zeqiang, Z. Xizhou, D. Jifeng, Q. Yu, and W. Wenhai, “Mini-dalle3: Interactive text to image by prompting large language models,” _arXiv preprint arXiv:2310.07653_ , 2023.
  * [58] Z. Tang, Z. Yang, M. Khademi, Y. Liu, C. Zhu, and M. Bansal, “Codi-2: In-context, interleaved, and interactive any-to-any generation,” _arXiv preprint arXiv:2311.18775_ , 2023.
  * [59] Y. Ge, S. Zhao, Z. Zeng, Y. Ge, C. Li, X. Wang, and Y. Shan, “Making llama see and draw with seed tokenizer,” _arXiv preprint arXiv:2310.01218_ , 2023.
  * [60] Q. Sun, Y. Cui, X. Zhang, F. Zhang, Q. Yu, Z. Luo, Y. Wang, Y. Rao, J. Liu, T. Huang _et al._ , “Generative multimodal models are in-context learners,” _arXiv preprint arXiv:2312.13286_ , 2023.
  * [61] X. Zhao, B. Liu, Q. Liu, G. Shi, and X.-M. Wu, “Making multimodal generation easier: When diffusion models meet llms,” _arXiv preprint arXiv:2310.08949_ , 2023.
  * [62] J. Chen, Y. Huang, T. Lv, L. Cui, Q. Chen, and F. Wei, “Textdiffuser-2: Unleashing the power of language models for text rendering,” _arXiv preprint arXiv:2311.16465_ , 2023.
  * [63] L. Lian, B. Li, A. Yala, and T. Darrell, “Llm-grounded diffusion: Enhancing prompt understanding of text-to-image diffusion models with large language models,” _arXiv preprint arXiv:2305.13655_ , 2023.
  * [64] W. Feng, W. Zhu, T.-j. Fu, V. Jampani, A. Akula, X. He, S. Basu, X. E. Wang, and W. Y. Wang, “Layoutgpt: Compositional visual planning and generation with large language models,” _arXiv preprint arXiv:2305.15393_ , 2023.
  * [65] T. Zhang, Y. Zhang, V. Vineet, N. Joshi, and X. Wang, “Controllable text-to-image generation with gpt-4,” _arXiv preprint arXiv:2305.18583_ , 2023.
  * [66] L. Qu, S. Wu, H. Fei, L. Nie, and T.-S. Chua, “Layoutllm-t2i: Eliciting layout guidance from llm for text-to-image generation,” in _Proceedings of the 31st ACM International Conference on Multimedia_ , 2023, pp. 643–654.
  * [67] Y. Li, H. Liu, Q. Wu, F. Mu, J. Yang, J. Gao, C. Li, and Y. J. Lee, “Gligen: Open-set grounded text-to-image generation,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2023, pp. 22 511–22 521.
  * [68] J. Betker, G. Goh, L. Jing, T. Brooks, J. Wang, L. Li, L. Ouyang, J. Zhuang, J. Lee, Y. Guo _et al._ , “Improving image generation with better captions,” _Computer Science. https://cdn. openai. com/papers/dall-e-3. pdf_ , 2023.
  * [69] D. Kondratyuk, L. Yu, X. Gu, J. Lezama, J. Huang, R. Hornung, H. Adam, H. Akbari, Y. Alon, V. Birodkar _et al._ , “Videopoet: A large language model for zero-shot video generation,” _arXiv preprint arXiv:2312.14125_ , 2023.
  * [70] L. Yu, J. Lezama, N. B. Gundavarapu, L. Versari, K. Sohn, D. Minnen, Y. Cheng, A. Gupta, X. Gu, A. G. Hauptmann _et al._ , “Language model beats diffusion–tokenizer is key to visual generation,” _arXiv preprint arXiv:2310.05737_ , 2023.
  * [71] H. Fei, S. Wu, W. Ji, H. Zhang, and T.-S. Chua, “Empowering dynamics-aware text-to-video diffusion with large language models,” _arXiv preprint arXiv:2308.13812_ , 2023.
  * [72] H. Lin, A. Zala, J. Cho, and M. Bansal, “Videodirectorgpt: Consistent multi-scene video generation via llm-guided planning,” _arXiv preprint arXiv:2309.15091_ , 2023.
  * [73] J. Lv, Y. Huang, M. Yan, J. Huang, J. Liu, Y. Liu, Y. Wen, X. Chen, and S. Chen, “Gpt4motion: Scripting physical motions in text-to-video generation via blender-oriented gpt planning,” _arXiv preprint arXiv:2311.12631_ , 2023.
  * [74] Y. Lu, L. Zhu, H. Fan, and Y. Yang, “Flowzero: Zero-shot text-to-video synthesis with llm-driven dynamic scene syntax,” _arXiv preprint arXiv:2311.15813_ , 2023.
  * [75] S. Hong, J. Seo, S. Hong, H. Shin, and S. Kim, “Large language models are frame-level directors for zero-shot text-to-video generation,” _arXiv preprint arXiv:2305.14330_ , 2023.
  * [76] H. Huang, Y. Feng, C. Shi, L. Xu, J. Yu, and S. Yang, “Free-bloom: Zero-shot text-to-video generator with llm director and ldm animator,” _arXiv preprint arXiv:2309.14494_ , 2023.
  * [77] Z. Wang, J. Wang, D. Lin, and B. Dai, “Intercontrol: Generate human motion interactions by controlling every joint,” _arXiv preprint arXiv:2311.15864_ , 2023.
  * [78] J. Liu, W. Dai, C. Wang, Y. Cheng, Y. Tang, and X. Tong, “Plan, posture and go: Towards open-world text-to-motion generation,” _arXiv preprint arXiv:2312.14828_ , 2023.
  * [79] F. Long, Z. Qiu, T. Yao, and T. Mei, “Videodrafter: Content-consistent multi-scene video generation with llm,” _arXiv preprint arXiv:2401.01256_ , 2024.
  * [80] C. Sun, J. Han, W. Deng, X. Wang, Z. Qin, and S. Gould, “3d-gpt: Procedural 3d modeling with large language models,” _arXiv preprint arXiv:2310.12945_ , 2023.
  * [81] Y. Feng, J. Lin, S. K. Dwivedi, Y. Sun, P. Patel, and M. J. Black, “Posegpt: Chatting about 3d human pose,” _arXiv preprint arXiv:2311.18836_ , 2023.
  * [82] S. Chen, X. Chen, C. Zhang, M. Li, G. Yu, H. Fei, H. Zhu, J. Fan, and T. Chen, “Ll3da: Visual interactive instruction tuning for omni-3d understanding, reasoning, and planning,” _arXiv preprint arXiv:2311.18651_ , 2023.
  * [83] T. Wu, G. Yang, Z. Li, K. Zhang, Z. Liu, L. Guibas, D. Lin, and G. Wetzstein, “Gpt-4v(ision) is a human-aligned evaluator for text-to-3d generation,” _IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2024.
  * [84] D. Zhang, S. Li, X. Zhang, J. Zhan, P. Wang, Y. Zhou, and X. Qiu, “Speechgpt: Empowering large language models with intrinsic cross-modal conversational abilities,” _arXiv preprint arXiv:2305.11000_ , 2023.
  * [85] Y. Gong, H. Luo, A. H. Liu, L. Karlinsky, and J. Glass, “Listen, think, and understand,” _arXiv preprint arXiv:2305.10790_ , 2023.
  * [86] S. Deshmukh, B. Elizalde, R. Singh, and H. Wang, “Pengi: An audio language model for audio tasks,” _arXiv preprint arXiv:2305.11834_ , 2023.
  * [87] P. K. Rubenstein, C. Asawaroengchai, D. D. Nguyen, A. Bapna, Z. Borsos, F. d. C. Quitry, P. Chen, D. E. Badawy, W. Han, E. Kharitonov _et al._ , “Audiopalm: A large language model that can speak and listen,” _arXiv preprint arXiv:2306.12925_ , 2023.
  * [88] S. Liu, A. S. Hussain, C. Sun, and Y. Shan, “Music understanding llama: Advancing text-to-music generation with question answering and captioning,” in _ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. IEEE, 2024, pp. 286–290.
  * [89] Q. Chen, Y. Chu, Z. Gao, Z. Li, K. Hu, X. Zhou, J. Xu, Z. Ma, W. Wang, S. Zheng _et al._ , “Lauragpt: Listen, attend, understand, and regenerate audio with gpt,” _arXiv preprint arXiv:2310.04673_ , 2023.
  * [90] J. Gardner, S. Durand, D. Stoller, and R. M. Bittner, “Llark: A multimodal foundation model for music,” _arXiv preprint arXiv:2310.07160_ , 2023.
  * [91] C. Tang, W. Yu, G. Sun, X. Chen, T. Tan, W. Li, L. Lu, Z. Ma, and C. Zhang, “Salmonn: Towards generic hearing abilities for large language models,” _arXiv preprint arXiv:2310.13289_ , 2023.
  * [92] Y. Chu, J. Xu, X. Zhou, Q. Yang, S. Zhang, Z. Yan, C. Zhou, and J. Zhou, “Qwen-audio: Advancing universal audio understanding via unified large-scale audio-language models,” _arXiv preprint arXiv:2311.07919_ , 2023.
  * [93] A. S. Hussain, S. Liu, C. Sun, and Y. Shan, “M 2 ugen: Multi-modal music understanding and generation with the power of large language models,” _arXiv preprint arXiv:2311.11255_ , 2023.
  * [94] Y. Shu, S. Dong, G. Chen, W. Huang, R. Zhang, D. Shi, Q. Xiang, and Y. Shi, “Llasm: Large language and speech model,” _arXiv preprint arXiv:2308.15930_ , 2023.
  * [95] R. Yuan, H. Lin, Y. Wang, Z. Tian, S. Wu, T. Shen, G. Zhang, Y. Wu, C. Liu, Z. Zhou _et al._ , “Chatmusician: Understanding and generating music intrinsically with llm,” _arXiv preprint arXiv:2402.16153_ , 2024.
  * [96] S. Ding, Z. Liu, X. Dong, P. Zhang, R. Qian, C. He, D. Lin, and J. Wang, “Songcomposer: A large language model for lyric and melody composition in song generation,” _arXiv preprint arXiv:2402.17645_ , 2024.
  * [97] S.-L. Wu, C. Donahue, S. Watanabe, and N. J. Bryan, “Music controlnet: Multiple time-varying controls for music generation,” _arXiv preprint arXiv:2311.07069_ , 2023.
  * [98] D. Ghosal, N. Majumder, A. Mehrish, and S. Poria, “Text-to-audio generation using instruction-tuned llm and latent diffusion model,” _arXiv preprint arXiv:2304.13731_ , 2023.
  * [99] S.-L. Wu, X. Chang, G. Wichern, J.-w. Jung, F. Germain, J. Le Roux, and S. Watanabe, “Improving audio captioning models with fine-grained audio features, text embedding supervision, and llm mix-up augmentation,” in _ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. IEEE, 2024, pp. 316–320.
  * [100] J. Huang, Y. Ren, R. Huang, D. Yang, Z. Ye, C. Zhang, J. Liu, X. Yin, Z. Ma, and Z. Zhao, “Make-an-audio 2: Temporal-enhanced text-to-audio generation,” _arXiv preprint arXiv:2305.18474_ , 2023.
  * [101] Z. Wang, S. Mao, W. Wu, Y. Xia, Y. Deng, and J. Tien, “Assessing phrase break of esl speech with pre-trained language models and large language models,” _arXiv preprint arXiv:2306.04980_ , 2023.
  * [102] A. Vyas, B. Shi, M. Le, A. Tjandra, Y.-C. Wu, B. Guo, J. Zhang, X. Zhang, R. Adkins, W. Ngan _et al._ , “Audiobox: Unified audio generation with natural language prompts,” _arXiv preprint arXiv:2312.15821_ , 2023.
  * [103] Y. Shen, K. Song, X. Tan, D. Li, W. Lu, and Y. Zhuang, “Hugginggpt: Solving ai tasks with chatgpt and its friends in huggingface,” in _Advances in Neural Information Processing Systems_ , 2023.
  * [104] R. Huang, M. Li, D. Yang, J. Shi, X. Chang, Z. Ye, Y. Wu, Z. Hong, J. Huang, J. Liu _et al._ , “Audiogpt: Understanding and generating speech, music, sound, and talking head,” _arXiv preprint arXiv:2304.12995_ , 2023.
  * [105] X. Liu, Z. Zhu, H. Liu, Y. Yuan, M. Cui, Q. Huang, J. Liang, Y. Cao, Q. Kong, M. D. Plumbley _et al._ , “Wavjourney: Compositional audio creation with large language models,” _arXiv preprint arXiv:2307.14335_ , 2023.
  * [106] D. Yu, K. Song, P. Lu, T. He, X. Tan, W. Ye, S. Zhang, and J. Bian, “Musicagent: An ai agent for music understanding and generation with large language models,” _arXiv preprint arXiv:2310.11954_ , 2023.
  * [107] Y. Zhang, A. Maezawa, G. Xia, K. Yamamoto, and S. Dixon, “Loop copilot: Conducting ai ensembles for music generation and iterative editing,” _arXiv preprint arXiv:2310.12404_ , 2023.
  * [108] L. Zhuo, R. Yuan, J. Pan, Y. Ma, Y. LI, G. Zhang, S. Liu, R. Dannenberg, J. Fu, C. Lin _et al._ , “Lyricwhiz: Robust multilingual zero-shot lyrics transcription by whispering to chatgpt,” _arXiv preprint arXiv:2306.17103_ , 2023.
  * [109] P. Dhariwal, H. Jun, C. Payne, J. W. Kim, A. Radford, and I. Sutskever, “Jukebox: A generative model for music,” _arXiv preprint arXiv:2005.00341_ , 2020.
  * [110] Z. Borsos, R. Marinier, D. Vincent, E. Kharitonov, O. Pietquin, M. Sharifi, D. Roblek, O. Teboul, D. Grangier, M. Tagliasacchi _et al._ , “Audiolm: a language modeling approach to audio generation,” _IEEE/ACM Transactions on Audio, Speech, and Language Processing_ , 2023.
  * [111] D. Yang, J. Tian, X. Tan, R. Huang, S. Liu, X. Chang, J. Shi, S. Zhao, J. Bian, X. Wu _et al._ , “Uniaudio: An audio foundation model toward universal audio generation,” _arXiv preprint arXiv:2310.00704_ , 2023.
  * [112] W. X. Zhao, K. Zhou, J. Li, T. Tang, X. Wang, Y. Hou, Y. Min, B. Zhang, J. Zhang, Z. Dong _et al._ , “A survey of large language models,” _arXiv preprint arXiv:2303.18223_ , 2023.
  * [113] S. Minaee, T. Mikolov, N. Nikzad, M. Chenaghlu, R. Socher, X. Amatriain, and J. Gao, “Large language models: A survey,” _arXiv preprint arXiv:2402.06196_ , 2024.
  * [114] Y. Chang, X. Wang, J. Wang, Y. Wu, L. Yang, K. Zhu, H. Chen, X. Yi, C. Wang, Y. Wang, W. Ye, Y. Zhang, Y. Chang, P. S. Yu, Q. Yang, and X. Xie, “A survey on evaluation of large language models,” _ACM Trans. Intell. Syst. Technol._ , vol. 15, no. 3, mar 2024. [Online]. Available: https://doi.org/10.1145/3641289
  * [115] C. Zhang, C. Zhang, M. Zhang, and I. S. Kweon, “Text-to-image diffusion model in generative ai: A survey,” _arXiv preprint arXiv:2303.07909_ , 2023.
  * [116] Z. Xing, Q. Feng, H. Chen, Q. Dai, H. Hu, H. Xu, Z. Wu, and Y.-G. Jiang, “A survey on video diffusion models,” _arXiv preprint arXiv:2310.10647_ , 2023.
  * [117] Z. Shi, S. Peng, Y. Xu, A. Geiger, Y. Liao, and Y. Shen, “Deep generative models on 3d representations: A survey,” _arXiv preprint arXiv:2210.15663_ , 2022.
  * [118] C. Zhang, C. Zhang, S. Zheng, M. Zhang, M. Qamar, S.-H. Bae, and I. S. Kweon, “Audio diffusion model for speech synthesis: A survey on text to speech and speech enhancement in generative ai,” _arXiv preprint arXiv:2303.13336_ , 2023.
  * [119] Q. Sun, Y. Fang, L. Wu, X. Wang, and Y. Cao, “Eva-clip: Improved training techniques for clip at scale,” _arXiv preprint arXiv:2303.15389_ , 2023.
  * [120] C. Raffel, N. Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Y. Zhou, W. Li, and P. J. Liu, “Exploring the limits of transfer learning with a unified text-to-text transformer,” _Journal of machine learning research_ , vol. 21, no. 140, pp. 1–67, 2020.
  * [121] L. Wang, C. Ma, X. Feng, Z. Zhang, H. Yang, J. Zhang, Z. Chen, J. Tang, X. Chen, Y. Lin _et al._ , “A survey on large language model based autonomous agents,” _arXiv preprint arXiv:2308.11432_ , 2023.
  * [122] S. Yin, C. Fu, S. Zhao, K. Li, X. Sun, T. Xu, and E. Chen, “A survey on multimodal large language models,” _arXiv preprint arXiv:2306.13549_ , 2023.
  * [123] J. Wu, W. Gan, Z. Chen, S. Wan, and S. Y. Philip, “Multimodal large language models: A survey,” in _2023 IEEE International Conference on Big Data (BigData)_. IEEE, 2023, pp. 2247–2256.
  * [124] I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio, “Generative adversarial networks,” _Communications of the ACM_ , vol. 63, no. 11, pp. 139–144, 2020.
  * [125] D. P. Kingma and M. Welling, “Auto-encoding variational bayes,” _arXiv preprint arXiv:1312.6114_ , 2013.
  * [126] P. Esser, R. Rombach, and B. Ommer, “Taming transformers for high-resolution image synthesis,” in _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , 2021, pp. 12 873–12 883.
  * [127] J. Sohl-Dickstein, E. Weiss, N. Maheswaranathan, and S. Ganguli, “Deep unsupervised learning using nonequilibrium thermodynamics,” in _International conference on machine learning_. PMLR, 2015, pp. 2256–2265.
  * [128] J. Ho, A. Jain, and P. Abbeel, “Denoising diffusion probabilistic models,” _Advances in neural information processing systems_ , vol. 33, pp. 6840–6851, 2020.
  * [129] Y. Tian, D. Krishnan, and P. Isola, “Contrastive multiview coding,” in _Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XI 16_. Springer, 2020, pp. 776–794.
  * [130] C. Wang, M. Chai, M. He, D. Chen, and J. Liao, “Clip-nerf: Text-and-image driven manipulation of neural radiance fields,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2022, pp. 3835–3844.
  * [131] B. Elizalde, S. Deshmukh, M. Al Ismail, and H. Wang, “Clap learning audio concepts from natural language supervision,” in _ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. IEEE, 2023, pp. 1–5.
  * [132] S. Luo, C. Yan, C. Hu, and H. Zhao, “Diff-foley: Synchronized video-to-audio synthesis with latent diffusion models,” _Advances in Neural Information Processing Systems_ , vol. 36, 2024.
  * [133] R. Girdhar, A. El-Nouby, Z. Liu, M. Singh, K. V. Alwala, A. Joulin, and I. Misra, “Imagebind: One embedding space to bind them all,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2023, pp. 15 180–15 190.
  * [134] S. Wu, H. Fei, L. Qu, W. Ji, and T.-S. Chua, “Next-gpt: Any-to-any multimodal llm,” _arXiv preprint arXiv:2309.05519_ , 2023.
  * [135] Y. Xing, Y. He, Z. Tian, X. Wang, and Q. Chen, “Seeing and hearing: Open-domain visual-audio generation with diffusion latent aligners,” _arXiv preprint arXiv:2402.17723_ , 2024.
  * [136] Y. Su, T. Lan, H. Li, J. Xu, Y. Wang, and D. Cai, “Pandagpt: One model to instruction-follow them all,” _arXiv preprint arXiv:2305.16355_ , 2023.
  * [137] H. Touvron, T. Lavril, G. Izacard, X. Martinet, M.-A. Lachaux, T. Lacroix, B. Rozière, N. Goyal, E. Hambro, F. Azhar, A. Rodriguez, A. Joulin, E. Grave, and G. Lample, “Llama: Open and efficient foundation language models,” _arXiv preprint arXiv:2302.13971_ , 2023.
  * [138] A. Radford, K. Narasimhan, T. Salimans, I. Sutskever _et al._ , “Improving language understanding by generative pre-training,” 2018.
  * [139] A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, I. Sutskever _et al._ , “Language models are unsupervised multitask learners,” _OpenAI blog_ , vol. 1, no. 8, p. 9, 2019.
  * [140] T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell _et al._ , “Language models are few-shot learners,” _Advances in neural information processing systems_ , vol. 33, pp. 1877–1901, 2020.
  * [141] Y. Bai, A. Jones, K. Ndousse, A. Askell, A. Chen, N. DasSarma, D. Drain, S. Fort, D. Ganguli, T. Henighan _et al._ , “Training a helpful and harmless assistant with reinforcement learning from human feedback,” _arXiv preprint arXiv:2204.05862_ , 2022.
  * [142] D. Zhu, J. Chen, X. Shen, X. Li, and M. Elhoseiny, “Minigpt-4: Enhancing vision-language understanding with advanced large language models,” _arXiv preprint arXiv:2304.10592_ , 2023.
  * [143] W.-G. Chen, I. Spiridonova, J. Yang, J. Gao, and C. Li, “Llava-interactive: An all-in-one demo for image chat, segmentation, generation and editing,” _arXiv preprint arXiv:2311.00571_ , 2023.
  * [144] R. Pi, L. Yao, J. Gao, J. Zhang, and T. Zhang, “Perceptiongpt: Effectively fusing visual perception into llm,” _arXiv preprint arXiv:2311.06612_ , 2023.
  * [145] R. Pi, J. Gao, S. Diao, R. Pan, H. Dong, J. Zhang, L. Yao, J. Han, H. Xu, and L. K. T. Zhang, “Detgpt: Detect what you need via reasoning,” _arXiv preprint arXiv:2305.14167_ , 2023.
  * [146] J. Gao, R. Pi, J. Zhang, J. Ye, W. Zhong, Y. Wang, L. Hong, J. Han, H. Xu, Z. Li _et al._ , “G-llava: Solving geometric problem with multi-modal large language model,” _arXiv preprint arXiv:2312.11370_ , 2023.
  * [147] Q. Sun, Q. Yu, Y. Cui, F. Zhang, X. Zhang, Y. Wang, H. Gao, J. Liu, T. Huang, and X. Wang, “Generative pretraining in multimodality,” _arXiv preprint arXiv:2307.05222_ , 2023.
  * [148] F. Yu, A. Seff, Y. Zhang, S. Song, T. Funkhouser, and J. Xiao, “Lsun: Construction of a large-scale image dataset using deep learning with humans in the loop,” _arXiv preprint arXiv:1506.03365_ , 2015.
  * [149] Z. Liu, P. Luo, X. Wang, and X. Tang, “Large-scale celebfaces attributes (celeba) dataset,” _Retrieved August_ , vol. 15, no. 2018, p. 11, 2018.
  * [150] “Dalle-1,” https://openai.com/research/dall-e.
  * [151] R. Dong, C. Han, Y. Peng, Z. Qi, Z. Ge, J. Yang, L. Zhao, J. Sun, H. Zhou, H. Wei _et al._ , “Dreamllm: Synergistic multimodal comprehension and creation,” _arXiv preprint arXiv:2309.11499_ , 2023.
  * [152] “Dalle-3,” https://openai.com/dall-e-3.
  * [153] J. Y. Koh, R. Salakhutdinov, and D. Fried, “Grounding language models to images for multimodal generation,” _arXiv preprint arXiv:2301.13823_ , 2023.
  * [154] J. Y. Koh, D. Fried, and R. Salakhutdinov, “Generating images with multimodal language models,” _arXiv preprint arXiv:2305.17216_ , 2023.
  * [155] L. Yu, Y. Cheng, Z. Wang, V. Kumar, W. Macherey, Y. Huang, D. A. Ross, I. Essa, Y. Bisk, M.-H. Yang _et al._ , “Spae: Semantic pyramid autoencoder for multimodal generation with frozen llms,” _arXiv preprint arXiv:2306.17842_ , 2023.
  * [156] L. Yu, B. Shi, R. Pasunuru, B. Muller, O. Golovneva, T. Wang, A. Babu, B. Tang, B. Karrer, S. Sheynin _et al._ , “Scaling autoregressive multi-modal models: Pretraining and instruction tuning,” _arXiv preprint arXiv:2309.02591_ , 2023.
  * [157] K. Zheng, X. He, and X. E. Wang, “Minigpt-5: Interleaved vision-and-language generation via generative vokens,” _arXiv preprint arXiv:2310.02239_ , 2023.
  * [158] J. An, Z. Yang, L. Li, J. Wang, K. Lin, Z. Liu, L. Wang, and J. Luo, “Openleaf: Open-domain interleaved image-text generation and evaluation,” _arXiv preprint arXiv:2310.07749_ , 2023.
  * [159] Z. Yang, Y. Zhang, F. Meng, and J. Zhou, “Teal: Tokenize and embed all for multi-modal large language models,” _arXiv preprint arXiv:2311.04589_ , 2023.
  * [160] B. Xia, S. Wang, Y. Tao, Y. Wang, and J. Jia, “Llmga: Multimodal large language model based generation assistant,” _arXiv preprint arXiv:2311.16500_ , 2023.
  * [161] X. Chi, Y. Liu, Z. Jiang, R. Zhang, Z. Lin, R. Zhang, P. Gao, C. Fu, S. Zhang, Q. Liu _et al._ , “Chatillusion: Efficient-aligning interleaved generation ability with visual instruction model,” _arXiv preprint arXiv:2311.17963_ , 2023.
  * [162] Y. Zhou, R. Zhang, J. Gu, and T. Sun, “Customization assistant for text-to-image generation,” _arXiv preprint arXiv:2312.03045_ , 2023.
  * [163] X. Shen and M. Elhoseiny, “Storygpt-v: Large language models as consistent story visualizers,” 2023.
  * [164] X. Hu, R. Wang, Y. Fang, B. Fu, P. Cheng, and G. Yu, “Ella: Equip diffusion models with llm for enhanced semantic alignment,” _arXiv preprint arXiv:2403.05135_ , 2024.
  * [165] S. Zhao, S. Hao, B. Zi, H. Xu, and K.-Y. K. Wong, “Bridging different language models and generative vision models for text-to-image generation,” _arXiv preprint arXiv:2403.07860_ , 2024.
  * [166] J. Cho, A. Zala, and M. Bansal, “Visual programming for text-to-image generation and evaluation,” _arXiv preprint arXiv:2305.15328_ , 2023.
  * [167] H. Gani, S. F. Bhat, M. Naseer, S. Khan, and P. Wonka, “Llm blueprint: Enabling text-to-image generation with complex and detailed prompts,” _arXiv preprint arXiv:2310.10640_ , 2023.
  * [168] T.-H. Wu, L. Lian, J. E. Gonzalez, B. Li, and T. Darrell, “Self-correcting llm-controlled diffusion models,” _arXiv preprint arXiv:2311.16090_ , 2023.
  * [169] J. Chen, Y. Huang, T. Lv, L. Cui, Q. Chen, and F. Wei, “Textdiffuser: Diffusion models as text painters,” _arXiv preprint arXiv:2305.10855_ , 2023.
  * [170] P. Jia, C. Li, Z. Liu, Y. Shen, X. Chen, Y. Yuan, Y. Zheng, D. Chen, J. Li, X. Xie _et al._ , “Cole: A hierarchical generation framework for graphic design,” _arXiv preprint arXiv:2311.16974_ , 2023.
  * [171] S. Zhong, Z. Huang, W. Wen, J. Qin, and L. Lin, “Sur-adapter: Enhancing text-to-image pre-trained diffusion models with large language models,” in _Proceedings of the 31st ACM International Conference on Multimedia_ , 2023, pp. 567–578.
  * [172] Q. Yu, J. Li, W. Ye, S. Tang, and Y. Zhuang, “Interactive data synthesis for systematic vision adaptation via llms-aigcs collaboration,” _arXiv preprint arXiv:2305.12799_ , 2023.
  * [173] X. Wang, B. Zhuang, and Q. Wu, “Switchgpt: Adapting large language models for non-text outputs,” _arXiv preprint arXiv:2309.07623_ , 2023.
  * [174] J. Liao, X. Chen, Q. Fu, L. Du, X. He, X. Wang, S. Han, and D. Zhang, “Text-to-image generation for abstract concepts,” _arXiv preprint arXiv:2309.14623_ , 2023.
  * [175] Z. Yang, J. Wang, L. Li, K. Lin, C.-C. Lin, Z. Liu, and L. Wang, “Idea2img: Iterative self-refinement with gpt-4v (ision) for automatic image design and generation,” _arXiv preprint arXiv:2310.08541_ , 2023.
  * [176] J.-Y. He, Z.-Q. Cheng, C. Li, J. Sun, W. Xiang, X. Lin, X. Kang, Z. Jin, Y. Hu, B. Luo _et al._ , “Wordart designer: User-driven artistic typography synthesis using large language models,” _arXiv preprint arXiv:2310.18332_ , 2023.
  * [177] J. Sun, D. Fu, Y. Hu, S. Wang, R. Rassin, D.-C. Juan, D. Alon, C. Herrmann, S. van Steenkiste, R. Krishna _et al._ , “Dreamsync: Aligning text-to-image generation with image understanding feedback,” _arXiv preprint arXiv:2311.17946_ , 2023.
  * [178] Y. Lu, X. Yang, X. Li, X. E. Wang, and W. Y. Wang, “Llmscore: Unveiling the power of large language models in text-to-image synthesis evaluation,” _Advances in Neural Information Processing Systems_ , vol. 36, 2024.
  * [179] V. Ordonez, G. Kulkarni, and T. Berg, “Im2text: Describing images using 1 million captioned photographs,” _Advances in neural information processing systems_ , vol. 24, 2011.
  * [180] T.-Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollár, and C. L. Zitnick, “Microsoft coco: Common objects in context,” in _Computer Vision–ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13_. Springer, 2014, pp. 740–755.
  * [181] C. Jia, Y. Yang, Y. Xia, Y.-T. Chen, Z. Parekh, H. Pham, Q. Le, Y.-H. Sung, Z. Li, and T. Duerig, “Scaling up visual and vision-language representation learning with noisy text supervision,” in _International conference on machine learning_. PMLR, 2021, pp. 4904–4916.
  * [182] S. Changpinyo, P. Sharma, N. Ding, and R. Soricut, “Conceptual 12m: Pushing web-scale image-text pre-training to recognize long-tail visual concepts,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2021, pp. 3558–3568.
  * [183] K. Srinivasan, K. Raman, J. Chen, M. Bendersky, and M. Najork, “Wit: Wikipedia-based image text dataset for multimodal multilingual machine learning,” in _Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval_ , 2021, pp. 2443–2449.
  * [184] C. Schuhmann, R. Vencu, R. Beaumont, R. Kaczmarczyk, C. Mullis, A. Katta, T. Coombes, J. Jitsev, and A. Komatsuzaki, “Laion-400m: Open dataset of clip-filtered 400 million image-text pairs,” _arXiv preprint arXiv:2111.02114_ , 2021.
  * [185] Y. Zheng, H. Yang, T. Zhang, J. Bao, D. Chen, Y. Huang, L. Yuan, D. Chen, M. Zeng, and F. Wen, “General facial representation learning in a visual-linguistic manner,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2022, pp. 18 697–18 709.
  * [186] J.-B. Alayrac, J. Donahue, P. Luc, A. Miech, I. Barr, Y. Hasson, K. Lenc, A. Mensch, K. Millican, M. Reynolds _et al._ , “Flamingo: a visual language model for few-shot learning,” _Advances in Neural Information Processing Systems_ , vol. 35, pp. 23 716–23 736, 2022.
  * [187] LAION-COCO, “https://laion.ai/blog/laion-coco/,” 2022. [Online]. Available: https://laion.ai/blog/laion-coco/
  * [188] C. Schuhmann, R. Beaumont, R. Vencu, C. Gordon, R. Wightman, M. Cherti, T. Coombes, A. Katta, C. Mullis, M. Wortsman _et al._ , “Laion-5b: An open large-scale dataset for training next generation image-text models,” _Advances in Neural Information Processing Systems_ , vol. 35, pp. 25 278–25 294, 2022.
  * [189] Coyo-700M, “https://huggingface.co/datasets/kakaobrain/coyo-700m,” 2022. [Online]. Available: https://huggingface.co/datasets/kakaobrain/coyo-700m
  * [190] S. Huang, L. Dong, W. Wang, Y. Hao, S. Singhal, S. Ma, T. Lv, L. Cui, O. K. Mohammed, B. Patra _et al._ , “Language is not all you need: Aligning perception with language models,” _Advances in Neural Information Processing Systems_ , vol. 36, 2024.
  * [191] W. Zhu, J. Hessel, A. Awadalla, S. Y. Gadre, J. Dodge, A. Fang, Y. Yu, L. Schmidt, W. Y. Wang, and Y. Choi, “Multimodal c4: An open, billion-scale corpus of images interleaved with text,” _Advances in Neural Information Processing Systems_ , vol. 36, 2024.
  * [192] H. Liu, C. Li, Q. Wu, and Y. J. Lee, “Visual instruction tuning,” _Advances in neural information processing systems_ , vol. 36, 2024.
  * [193] S. Y. Gadre, G. Ilharco, A. Fang, J. Hayase, G. Smyrnis, T. Nguyen, R. Marten, M. Wortsman, D. Ghosh, J. Zhang _et al._ , “Datacomp: In search of the next generation of multimodal datasets,” _Advances in Neural Information Processing Systems_ , vol. 36, 2024.
  * [194] J. Chen, Y. Huang, T. Lv, L. Cui, Q. Chen, and F. Wei, “Textdiffuser: Diffusion models as text painters,” _Advances in Neural Information Processing Systems_ , vol. 36, 2024.
  * [195] Y. Yang, D. Gui, Y. Yuan, W. Liang, H. Ding, H. Hu, and K. Chen, “Glyphcontrol: Glyph conditional control for visual text generation,” _Advances in Neural Information Processing Systems_ , vol. 36, 2024.
  * [196] B. Li, Y. Zhang, L. Chen, J. Wang, F. Pu, J. Yang, C. Li, and Z. Liu, “Mimic-it: Multi-modal in-context instruction tuning,” _arXiv preprint arXiv:2306.05425_ , 2023.
  * [197] T. Vetter and T. Poggio, “Linear object classes and image synthesis from a single example image,” _IEEE Transactions on pattern analysis and machine intelligence_ , vol. 19, no. 7, pp. 733–742, 1997.
  * [198] M. F. Cohen and J. R. Wallace, _Radiosity and realistic image synthesis_. Morgan Kaufmann, 1993.
  * [199] D. Kirk and J. Arvo, “Unbiased sampling techniques for image synthesis,” _ACM SIGGRAPH Computer Graphics_ , vol. 25, no. 4, pp. 153–156, 1991.
  * [200] K. Frans, L. Soros, and O. Witkowski, “Clipdraw: Exploring text-to-drawing synthesis through language-image encoders,” _Advances in Neural Information Processing Systems_ , vol. 35, pp. 5207–5218, 2022.
  * [201] S. Gu, D. Chen, J. Bao, F. Wen, B. Zhang, D. Chen, L. Yuan, and B. Guo, “Vector quantized diffusion model for text-to-image synthesis,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2022, pp. 10 696–10 706.
  * [202] Z. Wang, W. Liu, Q. He, X. Wu, and Z. Yi, “Clip-gen: Language-free training of a text-to-image generator with clip,” _arXiv preprint arXiv:2203.00386_ , 2022.
  * [203] H. Liu, C. Li, Q. Wu, and Y. J. Lee, “Visual instruction tuning,” _arXiv preprint arXiv:2304.08485_ , 2023.
  * [204] J. Li, D. Li, S. Savarese, and S. Hoi, “Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models,” _arXiv preprint arXiv:2301.12597_ , 2023.
  * [205] T. Lv, Y. Huang, J. Chen, L. Cui, S. Ma, Y. Chang, S. Huang, W. Wang, L. Dong, W. Luo _et al._ , “Kosmos-2.5: A multimodal literate model,” _arXiv preprint arXiv:2309.11419_ , 2023.
  * [206] Q. Ye, H. Xu, G. Xu, J. Ye, M. Yan, Y. Zhou, J. Wang, A. Hu, P. Shi, Y. Shi _et al._ , “mplug-owl: Modularization empowers large language models with multimodality,” _arXiv preprint arXiv:2304.14178_ , 2023.
  * [207] J. Ye, A. Hu, H. Xu, Q. Ye, M. Yan, Y. Dan, C. Zhao, G. Xu, C. Li, J. Tian _et al._ , “mplug-docowl: Modularized multimodal large language model for document understanding,” _arXiv preprint arXiv:2307.02499_ , 2023.
  * [208] J. Chen, D. Zhu, X. Shen, X. Li, Z. Liu, P. Zhang, R. Krishnamoorthi, V. Chandra, Y. Xiong, and M. Elhoseiny, “Minigpt-v2: large language model as a unified interface for vision-language multi-task learning,” _arXiv preprint arXiv:2310.09478_ , 2023.
  * [209] R. Zhang, J. Han, A. Zhou, X. Hu, S. Yan, P. Lu, H. Li, P. Gao, and Y. Qiao, “Llama-adapter: Efficient fine-tuning of language models with zero-init attention,” _arXiv preprint arXiv:2303.16199_ , 2023.
  * [210] W. Dai, J. Li, D. Li, A. Tiong, J. Zhao, W. Wang, B. Li, P. Fung, and S. Hoi, “Instructblip: Towards general-purpose vision-language models with instruction tuning. arxiv 2023,” _arXiv preprint arXiv:2305.06500_.
  * [211] W. Wang, Z. Chen, X. Chen, J. Wu, X. Zhu, G. Zeng, P. Luo, T. Lu, J. Zhou, Y. Qiao _et al._ , “Visionllm: Large language model is also an open-ended decoder for vision-centric tasks,” _arXiv preprint arXiv:2305.11175_ , 2023.
  * [212] J. Bai, S. Bai, S. Yang, S. Wang, S. Tan, P. Wang, J. Lin, C. Zhou, and J. Zhou, “Qwen-vl: A frontier large vision-language model with versatile abilities,” _arXiv preprint arXiv:2308.12966_ , 2023.
  * [213] W. Wang, Q. Lv, W. Yu, W. Hong, J. Qi, Y. Wang, J. Ji, Z. Yang, L. Zhao, X. Song _et al._ , “Cogvlm: Visual expert for pretrained language models,” _arXiv preprint arXiv:2311.03079_ , 2023.
  * [214] Y. Jin, K. Xu, L. Chen, C. Liao, J. Tan, B. Chen, C. Lei, A. Liu, C. Song, X. Lei _et al._ , “Unified language-vision pretraining with dynamic discrete visual tokenization,” _arXiv preprint arXiv:2309.04669_ , 2023.
  * [215] Z. Chen, J. Wu, W. Wang, W. Su, G. Chen, S. Xing, Z. Muyan, Q. Zhang, X. Zhu, L. Lu _et al._ , “Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks,” _arXiv preprint arXiv:2312.14238_ , 2023.
  * [216] S. Huang, L. Dong, W. Wang, Y. Hao, S. Singhal, S. Ma, T. Lv, L. Cui, O. K. Mohammed, Q. Liu _et al._ , “Language is not all you need: Aligning perception with language models,” _arXiv preprint arXiv:2302.14045_ , 2023.
  * [217] C. Wu, S. Yin, W. Qi, X. Wang, Z. Tang, and N. Duan, “Visual chatgpt: Talking, drawing and editing with visual foundation models,” _arXiv preprint arXiv:2303.04671_ , 2023.
  * [218] Z. Liu, Y. He, W. Wang, W. Wang, Y. Wang, S. Chen, Q. Zhang, Y. Yang, Q. Li, J. Yu _et al._ , “Internchat: Solving vision-centric tasks by interacting with chatbots beyond language,” _arXiv preprint arXiv:2305.05662_ , 2023.
  * [219] Z. Yang, L. Li, J. Wang, K. Lin, E. Azarnasab, F. Ahmed, Z. Liu, C. Liu, M. Zeng, and L. Wang, “Mm-react: Prompting chatgpt for multimodal reasoning and action,” _arXiv preprint arXiv:2303.11381_ , 2023.
  * [220] S. Wang, Z. Zhao, X. Ouyang, Q. Wang, and D. Shen, “Chatcad: Interactive computer-aided diagnosis on medical image using large language models,” _arXiv preprint arXiv:2302.07257_ , 2023.
  * [221] Z. Yang, Z. Gan, J. Wang, X. Hu, Y. Lu, Z. Liu, and L. Wang, “An empirical study of gpt-3 for few-shot knowledge-based vqa,” in _Proceedings of the AAAI Conference on Artificial Intelligence_ , vol. 36, no. 3, 2022, pp. 3081–3089.
  * [222] Z. Gu, B. Zhu, G. Zhu, Y. Chen, M. Tang, and J. Wang, “Anomalygpt: Detecting industrial anomalies using large vision-language models,” _arXiv preprint arXiv:2308.15366_ , 2023.
  * [223] J. Qin, J. Wu, W. Chen, Y. Ren, H. Li, H. Wu, X. Xiao, R. Wang, and S. Wen, “Diffusiongpt: Llm-driven text-to-image generation system,” _arXiv preprint arXiv:2401.10061_ , 2024.
  * [224] L. Yang, Z. Yu, C. Meng, M. Xu, S. Ermon, and B. Cui, “Mastering text-to-image diffusion: Recaptioning, planning, and generating with multimodal llms,” _arXiv preprint arXiv:2401.11708_ , 2024.
  * [225] E. J. Hu, Y. Shen, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang, and W. Chen, “LoRA: Low-rank adaptation of large language models,” in _International Conference on Learning Representations_ , 2022. [Online]. Available: https://openreview.net/forum?id=nZeVKeeFYf9
  * [226] C. Meng, Y. He, Y. Song, J. Song, J. Wu, J.-Y. Zhu, and S. Ermon, “Sdedit: Guided image synthesis and editing with stochastic differential equations,” _arXiv preprint arXiv:2108.01073_ , 2021.
  * [227] G. Kim, T. Kwon, and J. C. Ye, “Diffusionclip: Text-guided diffusion models for robust image manipulation,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2022, pp. 2426–2435.
  * [228] G. Couairon, J. Verbeek, H. Schwenk, and M. Cord, “Diffedit: Diffusion-based semantic image editing with mask guidance,” _arXiv preprint arXiv:2210.11427_ , 2022.
  * [229] A. Hertz, R. Mokady, J. Tenenbaum, K. Aberman, Y. Pritch, and D. Cohen-Or, “Prompt-to-prompt image editing with cross attention control,” _arXiv preprint arXiv:2208.01626_ , 2022.
  * [230] R. Mokady, A. Hertz, K. Aberman, Y. Pritch, and D. Cohen-Or, “Null-text inversion for editing real images using guided diffusion models,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2023, pp. 6038–6047.
  * [231] B. Kawar, S. Zada, O. Lang, O. Tov, H. Chang, T. Dekel, I. Mosseri, and M. Irani, “Imagic: Text-based real image editing with diffusion models,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2023, pp. 6007–6017.
  * [232] N. Tumanyan, M. Geyer, S. Bagon, and T. Dekel, “Plug-and-play diffusion features for text-driven image-to-image translation,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2023, pp. 1921–1930.
  * [233] R. Morita, Z. Zhang, M. M. Ho, and J. Zhou, “Interactive image manipulation with complex text instructions,” in _Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision_ , 2023, pp. 1053–1062.
  * [234] Z. Zhang, L. Han, A. Ghosh, D. N. Metaxas, and J. Ren, “Sine: Single image editing with text-to-image diffusion models,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2023, pp. 6027–6037.
  * [235] G. Parmar, K. Kumar Singh, R. Zhang, Y. Li, J. Lu, and J.-Y. Zhu, “Zero-shot image-to-image translation,” in _ACM SIGGRAPH 2023 Conference Proceedings_ , 2023, pp. 1–11.
  * [236] V. Goel, E. Peruzzo, Y. Jiang, D. Xu, N. Sebe, T. Darrell, Z. Wang, and H. Shi, “Pair-diffusion: Object-level image editing with structure-and-appearance paired diffusion models,” _arXiv preprint arXiv:2303.17546_ , 2023.
  * [237] M. Cao, X. Wang, Z. Qi, Y. Shan, X. Qie, and Y. Zheng, “Masactrl: Tuning-free mutual self-attention control for consistent image synthesis and editing,” _arXiv preprint arXiv:2304.08465_ , 2023.
  * [238] T. Nguyen, Y. Li, U. Ojha, and Y. J. Lee, “Visual instruction inversion: Image editing via visual prompting,” _arXiv preprint arXiv:2307.14331_ , 2023.
  * [239] A. Mirzaei, T. Aumentado-Armstrong, M. A. Brubaker, J. Kelly, A. Levinshtein, K. G. Derpanis, and I. Gilitschenski, “Watch your steps: Local image and scene editing by text instructions,” _arXiv preprint arXiv:2308.08947_ , 2023.
  * [240] C. Mou, X. Wang, J. Song, Y. Shan, and J. Zhang, “Dragondiffusion: Enabling drag-style manipulation on diffusion models,” _arXiv preprint arXiv:2307.02421_ , 2023.
  * [241] ——, “Diffeditor: Boosting accuracy and flexibility on diffusion-based image editing,” _arXiv preprint arXiv:2402.02583_ , 2024.
  * [242] T. Brooks, A. Holynski, and A. A. Efros, “Instructpix2pix: Learning to follow image editing instructions,” _arXiv preprint arXiv:2211.09800_ , 2022.
  * [243] X. Cui, Z. Li, P. Li, Y. Hu, H. Shi, C. Cao, and Z. He, “Chatedit: Towards multi-turn interactive facial image editing via dialogue,” in _Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing_ , 2023, pp. 14 567–14 583.
  * [244] T.-J. Fu, W. Hu, X. Du, W. Y. Wang, Y. Yang, and Z. Gan, “Guiding instruction-based image editing via multimodal large language models,” _arXiv preprint arXiv:2309.17102_ , 2023.
  * [245] S. Sheynin, A. Polyak, U. Singer, Y. Kirstain, A. Zohar, O. Ashual, D. Parikh, and Y. Taigman, “Emu edit: Precise image editing via recognition and generation tasks,” _arXiv preprint arXiv:2311.10089_ , 2023.
  * [246] J. Z. Wu, Y. Ge, X. Wang, S. W. Lei, Y. Gu, Y. Shi, W. Hsu, Y. Shan, X. Qie, and M. Z. Shou, “Tune-a-video: One-shot tuning of image diffusion models for text-to-video generation,” in _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 2023, pp. 7623–7633.
  * [247] E. Molad, E. Horwitz, D. Valevski, A. R. Acha, Y. Matias, Y. Pritch, Y. Leviathan, and Y. Hoshen, “Dreamix: Video diffusion models are general video editors,” _arXiv preprint arXiv:2302.01329_ , 2023.
  * [248] S. Liu, Y. Zhang, W. Li, Z. Lin, and J. Jia, “Video-p2p: Video editing with cross-attention control,” _arXiv preprint arXiv:2303.04761_ , 2023.
  * [249] C. Qi, X. Cun, Y. Zhang, C. Lei, X. Wang, Y. Shan, and Q. Chen, “Fatezero: Fusing attentions for zero-shot text-based video editing,” in _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 2023, pp. 15 932–15 942.
  * [250] D. Ceylan, C.-H. P. Huang, and N. J. Mitra, “Pix2video: Video editing using image diffusion,” in _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 2023, pp. 23 206–23 217.
  * [251] W. Chai, X. Guo, G. Wang, and Y. Lu, “Stablevideo: Text-driven consistency-aware diffusion video editing,” in _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 2023, pp. 23 040–23 050.
  * [252] S. Yang, Y. Zhou, Z. Liu, and C. C. Loy, “Rerender a video: Zero-shot text-guided video-to-video translation,” in _SIGGRAPH Asia 2023 Conference Papers_ , 2023, pp. 1–11.
  * [253] M. Geyer, O. Bar-Tal, S. Bagon, and T. Dekel, “Tokenflow: Consistent diffusion features for consistent video editing,” _arXiv preprint arXiv:2307.10373_ , 2023.
  * [254] H. Ouyang, Q. Wang, Y. Xiao, Q. Bai, J. Zhang, K. Zheng, X. Zhou, Q. Chen, and Y. Shen, “Codef: Content deformation fields for temporally consistent video processing,” _arXiv preprint arXiv:2308.07926_ , 2023.
  * [255] J. H. Liew, H. Yan, J. Zhang, Z. Xu, and J. Feng, “Magicedit: High-fidelity and temporally coherent video editing,” _arXiv preprint arXiv:2308.14749_ , 2023.
  * [256] Y. Ma, X. Cun, Y. He, C. Qi, X. Wang, Y. Shan, X. Li, and Q. Chen, “Magicstick: Controllable video editing via control handle transformations,” _arXiv preprint arXiv:2312.03047_ , 2023.
  * [257] J. Cheng, T. Xiao, and T. He, “Consistent video-to-video transfer using synthetic dataset,” _arXiv preprint arXiv:2311.00213_ , 2023.
  * [258] B. Qin, J. Li, S. Tang, T.-S. Chua, and Y. Zhuang, “Instructvid2vid: Controllable video editing with natural language instructions,” _arXiv preprint arXiv:2305.12328_ , 2023.
  * [259] T. Brooks, A. Holynski, and A. A. Efros, “Instructpix2pix: Learning to follow image editing instructions,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2023, pp. 18 392–18 402.
  * [260] B. Wu, S. Yu, Z. Chen, J. B. Tenenbaum, and C. Gan, “Star: A benchmark for situated reasoning in real-world videos,” in _Thirty-fifth conference on neural information processing systems datasets and benchmarks track (Round 2)_ , 2021.
  * [261] Z. Liu, L. Wang, W. Wu, C. Qian, and T. Lu, “Tam: Temporal adaptive module for video recognition,” in _Proceedings of the IEEE/CVF international conference on computer vision_ , 2021, pp. 13 708–13 718.
  * [262] M. Zhao, B. Li, J. Wang, W. Li, W. Zhou, L. Zhang, S. Xuyang, Z. Yu, X. Yu, G. Li _et al._ , “Towards video text visual question answering: benchmark and baseline,” _Advances in Neural Information Processing Systems_ , vol. 35, pp. 35 549–35 562, 2022.
  * [263] M. Bain, A. Nagrani, G. Varol, and A. Zisserman, “Frozen in time: A joint video and image encoder for end-to-end retrieval,” in _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 2021, pp. 1728–1738.
  * [264] A. Yang, A. Miech, J. Sivic, I. Laptev, and C. Schmid, “Zero-shot video question answering via frozen bidirectional language models,” _Advances in Neural Information Processing Systems_ , vol. 35, pp. 124–141, 2022.
  * [265] ——, “Learning to answer visual questions from web videos,” _arXiv preprint arXiv:2205.05019_ , 2022.
  * [266] K. Lin, L. Li, C.-C. Lin, F. Ahmed, Z. Gan, Z. Liu, Y. Lu, and L. Wang, “Swinbert: End-to-end transformers with sparse attention for video captioning,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2022, pp. 17 949–17 958.
  * [267] J. Lin, C. Gan, and S. Han, “Tsm: Temporal shift module for efficient video understanding,” in _Proceedings of the IEEE/CVF international conference on computer vision_ , 2019, pp. 7083–7093.
  * [268] G. Bertasius, H. Wang, and L. Torresani, “Is space-time attention all you need for video understanding?” in _ICML_ , vol. 2, no. 3, 2021, p. 4.
  * [269] C.-Y. Wu, C. Feichtenhofer, H. Fan, K. He, P. Krahenbuhl, and R. Girshick, “Long-term feature banks for detailed video understanding,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2019, pp. 284–293.
  * [270] H. Zhang, X. Li, and L. Bing, “Video-llama: An instruction-tuned audio-visual language model for video understanding,” _arXiv preprint arXiv:2306.02858_ , 2023.
  * [271] J. Chen, D. Zhu, K. Haydarov, X. Li, and M. Elhoseiny, “Video chatcaptioner: Towards the enriched spatiotemporal descriptions,” _arXiv preprint arXiv:2304.04227_ , 2023.
  * [272] J. Ho, T. Salimans, A. Gritsenko, W. Chan, M. Norouzi, and D. J. Fleet, “Video diffusion models,” _Advances in Neural Information Processing Systems_ , vol. 35, pp. 8633–8646, 2022.
  * [273] A. Blattmann, R. Rombach, H. Ling, T. Dockhorn, S. W. Kim, S. Fidler, and K. Kreis, “Align your latents: High-resolution video synthesis with latent diffusion models,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2023, pp. 22 563–22 575.
  * [274] Y. He, M. Xia, H. Chen, X. Cun, Y. Gong, J. Xing, Y. Zhang, X. Wang, C. Weng, Y. Shan _et al._ , “Animate-a-story: Storytelling with retrieval-augmented video generation,” _arXiv preprint arXiv:2307.06940_ , 2023.
  * [275] Y. Bao, D. Qiu, G. Kang, B. Zhang, B. Jin, K. Wang, and P. Yan, “Latentwarp: Consistent diffusion latents for zero-shot video-to-video translation,” _arXiv preprint arXiv:2311.00353_ , 2023.
  * [276] J. Xu, T. Mei, T. Yao, and Y. Rui, “Msr-vtt: A large video description dataset for bridging video and language,” _2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , pp. 5288–5296, 2016. [Online]. Available: https://api.semanticscholar.org/CorpusID:206594535
  * [277] A. Rohrbach, A. Torabi, M. Rohrbach, N. Tandon, C. J. Pal, H. Larochelle, A. C. Courville, and B. Schiele, “Movie description,” _International Journal of Computer Vision_ , vol. 123, pp. 94 – 120, 2016. [Online]. Available: https://api.semanticscholar.org/CorpusID:18217052
  * [278] R. Krishna, K. Hata, F. Ren, L. Fei-Fei, and J. C. Niebles, “Dense-captioning events in videos,” _2017 IEEE International Conference on Computer Vision (ICCV)_ , pp. 706–715, 2017. [Online]. Available: https://api.semanticscholar.org/CorpusID:1026139
  * [279] R. Sanabria, O. Caglayan, S. Palaskar, D. Elliott, L. Barrault, L. Specia, and F. Metze, “How2: A large-scale dataset for multimodal language understanding,” _ArXiv_ , vol. abs/1811.00347, 2018. [Online]. Available: https://api.semanticscholar.org/CorpusID:53186236
  * [280] X. E. Wang, J. Wu, J. Chen, L. Li, Y. fang Wang, and W. Y. Wang, “Vatex: A large-scale, high-quality multilingual dataset for video-and-language research,” _2019 IEEE/CVF International Conference on Computer Vision (ICCV)_ , pp. 4580–4590, 2019. [Online]. Available: https://api.semanticscholar.org/CorpusID:102352148
  * [281] A. Miech, D. Zhukov, J.-B. Alayrac, M. Tapaswi, I. Laptev, and J. Sivic, “Howto100m: Learning a text-video embedding by watching hundred million narrated video clips,” _2019 IEEE/CVF International Conference on Computer Vision (ICCV)_ , pp. 2630–2640, 2019. [Online]. Available: https://api.semanticscholar.org/CorpusID:182952863