# Multimodal Foundation Models:
From Specialists to General-Purpose Assistants

Chunyuan Li∗♠, Zhe Gan∗, Zhengyuan Yang∗, Jianwei Yang∗, Linjie Li∗,
Lijuan Wang, Jianfeng Gao
Microsoft Corporation
<EMAIL_ADDRESS>∗ Core
Contribution ♠ Project Lead

###### Abstract

This paper presents a comprehensive survey of the taxonomy and evolution of
multimodal foundation models that demonstrate vision and vision-language
capabilities, focusing on the transition from specialist models to general-
purpose assistants. The research landscape encompasses five core topics,
categorized into two classes. $(i)$ We start with a survey of well-established
research areas: multimodal foundation models pre-trained for specific
purposes, including two topics – methods of learning vision backbones for
visual understanding and text-to-image generation. $(ii)$ Then, we present
recent advances in exploratory, open research areas: multimodal foundation
models that aim to play the role of general-purpose assistants, including
three topics – unified vision models inspired by large language models (LLMs),
end-to-end training of multimodal LLMs, and chaining multimodal tools with
LLMs. The target audiences of the paper are researchers, graduate students,
and professionals in computer vision and vision-language multimodal
communities who are eager to learn the basics and recent advances in
multimodal foundation models.

††1 Chunyuan Li initiated the project, and took lead in the writing of Chapter
1, 5 and 7. Zhe Gan, Zhengyuan Yang, Jianwei Yang, Linjie Li took lead in the
writing of Chapter 2, 3, 4 and 6, respectively. Lijuan Wang and Jianfeng Gao
provided comprehensive suggestions and edits of the entire paper. All the
authors provided project advice, and contributed to paper review, editing and
proofreading. ††2 Zhe Gan is currently with Apple AI/ML.

###### Contents

  1. 1 Introduction
    1. 1.1 What are Multimodal Foundation Models?
    2. 1.2 Definition and Transition from Specialists to General-Purpose Assistants
    3. 1.3 Who Should Read this Paper?
    4. 1.4 Related Materials: Slide Decks and Pre-recorded Talks
  2. 2 Visual Understanding
    1. 2.1 Overview
    2. 2.2 Supervised Pre-training
    3. 2.3 Contrastive Language-Image Pre-training
      1. 2.3.1 Basics of CLIP Training
      2. 2.3.2 CLIP Variants
    4. 2.4 Image-Only Self-Supervised Learning
      1. 2.4.1 Contrastive and Non-contrastive Learning
      2. 2.4.2 Masked Image Modeling
    5. 2.5 Synergy Among Different Learning Approaches
    6. 2.6 Multimodal Fusion, Region-Level and Pixel-Level Pre-training
      1. 2.6.1 From Multimodal Fusion to Multimodal LLM
      2. 2.6.2 Region-Level Pre-training
      3. 2.6.3 Pixel-Level Pre-training
  3. 3 Visual Generation
    1. 3.1 Overview
      1. 3.1.1 Human Alignments in Visual Generation
      2. 3.1.2 Text-to-Image Generation
    2. 3.2 Spatial Controllable Generation
    3. 3.3 Text-based Editing
    4. 3.4 Text Prompts Following
    5. 3.5 Concept Customization
    6. 3.6 Trends: Unified Tuning for Human Alignments
  4. 4 Unified Vision Models
    1. 4.1 Overview
    2. 4.2 From Closed-Set to Open-Set Models
      1. 4.2.1 Object Detection and Grounding
      2. 4.2.2 Image Segmentation and Referring
    3. 4.3 From Task-Specific Models to Generic Models
      1. 4.3.1 I/O Unification
      2. 4.3.2 Functionality Unification
    4. 4.4 From Static to Promptable Models
      1. 4.4.1 Multi-modal Prompting
      2. 4.4.2 In-context Prompting
    5. 4.5 Summary and Discussion
  5. 5 Large Multimodal Models: Training with LLM
    1. 5.1 Background
      1. 5.1.1 Image-to-Text Generative Models
      2. 5.1.2 Case Studies
      3. 5.1.3 OpenAI Multimodal GPT-4 and Research Gaps
    2. 5.2 Pre-requisite: Instruction Tuning in Large Language Models
      1. 5.2.1 Instruction Tuning
      2. 5.2.2 Self-Instruct and Open-Source LLMs
    3. 5.3 Instruction-Tuned Large Multimodal Models
    4. 5.4 Advanced Topics
    5. 5.5 How Close We Are To OpenAI Multimodal GPT-4?
  6. 6 Multimodal Agents: Chaining Tools with LLM
    1. 6.1 Overview
    2. 6.2 Multimodal Agent
    3. 6.3 Case Study: MM-ReAct
      1. 6.3.1 System Design
      2. 6.3.2 Capabilities
      3. 6.3.3 Extensibility
    4. 6.4 Advanced Topics
      1. 6.4.1 Comparison to Training with LLM in Chapter 5
      2. 6.4.2 Improving Multimodal Agents
      3. 6.4.3 Diverse Applications of Multimodal Agents
      4. 6.4.4 Evaluation of Multimodal Agents
      5. 6.4.5 Tool Creation
      6. 6.4.6 Retrieval-Augmented Multimodal Agents
  7. 7 Conclusions and Research Trends
    1. 7.1 Summary and Conclusions
    2. 7.2 Towards Building General-Purpose AI Agents

##  Chapter 1 Introduction

Vision is one of the primary channels for humans and many living creatures to
perceive and interact with the world. One of the core aspirations in
artificial intelligence (AI) is to develop AI agents to mimic such an ability
to effectively perceive and generate visual signals, and thus reason over and
interact with the visual world. Examples include recognition of the objects
and actions in the scenes, and creation of sketches and pictures for
communication. Building foundational models with visual capabilities is a
prevalent research field striving to accomplish this objective.

Over the last decade, the field of AI has experienced a fruitful trajectory in
the development of models. We divide them into four categories, as illustrated
in Figure 1.1. The categorization can be shared among different fields in AI,
including language, vision and multimodality. We first use language models in
NLP to illustrate the evolution process. $(i)$ At the early years, task-
specific models are developed for individual datasets and tasks, typically
being trained from scratch. $(ii)$ With large-scale pre-training, language
models achieve state-of-the-art performance on many established language
understanding and generation tasks, such as BERT (Devlin et al.,, 2019),
RoBERTa (Liu et al.,, 2019), T5 (Raffel et al.,, 2020), DeBERTa (He et al.,,
2021) and GPT-2 (Radford et al.,, 2019)). These pre-trained models serve the
basis for downstream task adaptation. $(iii)$ Exemplified by GPT-3 (Brown et
al.,, 2020), large language models (LLMs) unify various language understanding
and generation tasks into one model. With web-scale training and unification,
some emerging capabilities appear, such as in-context-learning and chain-of-
thoughts. $(iv)$ With recent advances in human-AI alignment, LLMs start to
play the role of general-purpose assistants to follow human intents to
complete a wide range of language tasks in the wild, such as ChatGPT (OpenAI,,
2022) and GPT-4 (OpenAI, 2023a, ). These assistants exhibit interesting
capabilities, such as interaction and tool use, and lay a foundation for
developing general-purpose AI agents. It is important to note that the latest
iterations of foundation models build upon the noteworthy features of their
earlier counterparts while also providing additional capabilities.

Inspired by the great successes of LLMs in NLP, it is natural for researchers
in the computer vision and vision-language community to ask the question: what
is the counterpart of ChatGPT/GPT-4 for vision, vision-language and multi-
modal models? There is no doubt that vision pre-training and vision-language
pre-training (VLP) have attracted a growing attention since the birth of BERT,
and has become the mainstream learning paradigm for vision, with the promise
to learn universal transferable visual and vision-language representations, or
to generate highly plausible images. Arguably, they can be considered as the
early generation of multimodal foundation models, just as BERT/GPT-2 to the
language field. While the road-map to build general-purpose assistants for
language such as ChatGPT is clear, it is becoming increasingly crucial for the
research community to explore feasible solutions to building its counterpart
for computer vision: the general-purpose visual assistants. Overall, building
general-purpose agents has been a long-standing goal for AI. LLMs with
emerging properties have significantly reduced the cost of building such
agents for language tasks. Similarly, we foresee emerging capabilities from
vision models, such as following the instructions composed by various visual
prompts like user-uploaded images, human-drawn clicks, sketches and mask, in
addition to text prompt. Such strong zero-shot visual task composition
capabilities can significantly reduce the cost of building AI agents.

Figure 1.1: Illustration of foundation model development trajectory for
language and vision/multi-modality. Among the four categories, the first
category is the task-specific model, and the last three categories belong to
foundation models, where these foundation models for language and vision are
grouped in green and blue blocks, respectively. Some prominent properties of
models in each category are highlighted. By comparing the models between
language and vision, we are foreseeing that the transition of multimodal
foundation models follows a similar trend: from the pre-trained model for
specific purpose, to unified models and general-purpose assistants. However,
research exploration is needed to figure out the best recipe, which is
indicated as the question mark in the figure, as multimodal GPT-4 and Gemini
stay private.

In this paper, we limit the scope of multimodal foundation models to the
vision and vision-language domains. Recent survey papers on related topics
include $(i)$ image understanding models such as self-supervised learning
(Jaiswal et al.,, 2020; Jing and Tian,, 2020; Ozbulak et al.,, 2023), segment
anything (SAM) (Zhang et al., 2023a, ; Zhang et al., 2023c, ), $(ii)$ image
generation models (Zhang et al., 2023b, ; Zhou and Shimada,, 2023), and
$(iii)$ vision-language pre-training (VLP). Existing VLP survey papers cover
VLP methods for task-specific VL problems before the era of pre-training,
image-text tasks, core vision tasks, and/or video-text tasks (Zhang et al.,,
2020; Du et al.,, 2022; Li et al., 2022c, ; Ruan and Jin,, 2022; Chen et al.,
2022a, ; Gan et al.,, 2022; Zhang et al., 2023g, ). Two recent survey papers
cover the integration of vision models with LLM (Awais et al.,, 2023; Yin et
al.,, 2022).

Among them, Gan et al., (2022) is a survey on VLP that covers the CVPR
tutorial series on Recent Advances in Vision-and-Language Research in 2022 and
before. This paper summarizes the CVPR tutorial on Recent Advances in Vision
Foundation Models in 2023. Different from the aforementioned survey papers
that focus on literature review of a given research topic, this paper presents
our perspectives on the role transition of multimodal foundation models from
specialists to general-purpose visual assistants, in the era of large language
models. The contributions of this survey paper are summarized as follows.

  * •

We provide a comprehensive and timely survey on modern multimodal foundation
models, not only covering well-established models for visual representation
learning and image generation, but also summarizing emerging topics for the
past 6 months inspired by LLMs, including unified vision models, training and
chaining with LLMs.

  * •

The paper is positioned to provide the audiences with the perspective to
advocate a transition in developing multimodal foundation models. On top of
great modeling successes for specific vision problems, we are moving towards
building general-purpose assistants that can follow human intents to complete
a wide range of computer vision tasks in the wild. We provide in-depth
discussions on these advanced topics, demonstrating the potential of
developing general-purpose visual assistants.

###  1.1 What are Multimodal Foundation Models?

As elucidated in the Stanford foundation model paper (Bommasani et al.,,
2021), AI has been undergoing a paradigm shift with the rise of models (_e.g._
, BERT, GPT family, CLIP (Radford et al.,, 2021) and DALL-E (Ramesh et al.,
2021a, )) trained on broad data that can be adapted to a wide range of
downstream tasks. They call these models foundation models to underscore their
critically central yet incomplete character: homogenization of the
methodologies across research communities and emergence of new capabilities.
From a technical perspective, it is _transfer learning_ that makes foundation
models possible, and it is _scale_ that makes them powerful. The emergence of
foundation models has been predominantly observed in the NLP domain, with
examples ranging from BERT to ChatGPT. This trend has gained traction in
recent years, extending to computer vision and other fields. In NLP, the
introduction of BERT in late 2018 is considered as the inception of the
foundation model era. The remarkable success of BERT rapidly stimulates
interest in self-supervised learning in the computer vision community, giving
rise to models such as SimCLR (Chen et al., 2020a, ), MoCo (He et al.,, 2020),
BEiT (Bao et al.,, 2022), and MAE (He et al., 2022a, ). During the same time
period, the success of pre-training also significantly promotes the vision-
and-language multimodal field to an unprecedented level of attention.

In this paper, we focus on multimodal foundation models, which inherit all
properties of foundation models discussed in the Stanford paper (Bommasani et
al.,, 2021), but with an emphasis on models with the capability to deal with
vision and vision-language modalities. Among the ever-growing literature, we
categorize multimodal foundation models in Figure 1.2, based on their
functionality and generality. For each category, we present exemplary models
that demonstrate the primary capabilities inherent to these multimodal
foundation models.

Figure 1.2: Illustration of three representative problems that multimodal
foundation models aim to solve in this paper: visual understanding tasks,
visual generation tasks, and general-purpose interface with language
understanding and generation.

  * •

Visual Understanding Models. (Highlighted with orange in Figure 1.2) Learning
general visual representations is essential to build vision foundation models,
as pre-training a strong vision backbone is foundamental to all types of
computer vision downstream tasks, ranging from image-level (_e.g._ , image
classification, retrieval, and captioning), region-level (_e.g._ , detection
and grounding) to pixel-level tasks (_e.g._ , segmentation). We group the
methods into three categories, depending on the types of supervision signals
used to train the models.

    * –

Label supervision. Datasets like ImageNet (Krizhevsky et al.,, 2012) and
ImageNet21K (Ridnik et al.,, 2021) have been popular for supervised learning,
and larger-scale proprietary datasets are also used in industrial labs (Sun et
al.,, 2017; Singh et al., 2022b, ; Zhai et al., 2022a, ).

    * –

Language supervision. Language is a richer form of supervision. Models like
CLIP (Radford et al.,, 2021) and ALIGN (Jia et al.,, 2021) are pre-trained
using a contrastive loss over millions or even billions of noisy image-text
pairs mined from the Web. These models enable zero-shot image classification,
and make traditional computer vision (CV) models to perform open-vocabulary CV
tasks. We advocate the concept of _computer vision in the wild_ ,111Computer-
Vision-in-the-Wild Readings. and encourage the development and evaluation of
future foundation models for this.

    * –

Image-only self-supervision. This line of work aims to learn image
representations from supervision signals mined from the images themselves,
ranging from contrastive learning (Chen et al., 2020a, ; He et al.,, 2020),
non-contrastive learning (Grill et al.,, 2020; Chen and He,, 2021; Caron et
al.,, 2021), to masked image modeling (Bao et al.,, 2022; He et al., 2022a, ).

    * –

Multimodal fusion, region-level and pixel-level pre-training. Besides the
methods of pre-training image backbones, we will also discuss pre-training
methods that allow multimodal fusion (_e.g._ , CoCa (Yu et al., 2022a, ),
Flamingo (Alayrac et al.,, 2022)), region-level and pixel-level image
understanding, such as open-set object detection (_e.g._ , GLIP (Li et al.,
2022e, )) and promptable semgentation (_e.g._ , SAM (Kirillov et al.,, 2023)).
These methods typically rely on a pre-trained image encoder or a pre-trained
image-text encoder pair.

  * •

Visual Generation Models. (Highlighted with green in Figure 1.2) Recently,
foundation image generation models have been built, due to the emergence of
large-scale image-text data. The techniques that make it possible include the
vector-quantized VAE methods (Razavi et al.,, 2019), diffusion-based models
(Dhariwal and Nichol,, 2021) and auto-regressive models.

    * –

Text-conditioned visual generation. This research area focuses on generating
faithful visual content, including images, videos, and more, conditioned on
open-ended text descriptions/prompts. Text-to-image generation develops
generative models that synthesize images of high fidelity to follow the text
prompt. Prominent examples include DALL-E (Ramesh et al., 2021a, ), DALL-E 2
(Ramesh et al.,, 2022), Stable Diffusion (Rombach et al.,, 2021; sta,, 2022),
Imagen (Saharia et al.,, 2022), and Parti (Yu et al., 2022b, ). Building on
the success of text-to-image generation models, text-to-video generation
models generate videos based on text prompts, such as Imagen Video (Ho et
al.,, 2022) and Make-A-Video (Singer et al.,, 2022).

    * –

Human-aligned visual generator. This research area focuses on improving the
pre-trained visual generator to better follow human intentions. Efforts have
been made to address various challenges inherent to base visual generators.
These include improving spatial controllability (Zhang and Agrawala,, 2023;
Yang et al., 2023b, ), ensuring better adherence to text prompts (Black et
al.,, 2023), supporting flexible text-based editing (Brooks et al.,, 2023),
and facilitating visual concept customization (Ruiz et al.,, 2023).

  * •

General-purpose Interface. (Highlighted with blue in Figure 1.2) The
aforementioned multimodal foundation models are designed for specific purposes
– tackling a specific set of CV problems/tasks. Recently, we see an emergence
of general-purpose models that lay the basis of AI agents. Existing efforts
focus on three research topics. The first topic aims to unify models for
visual understanding and generation. These models are inspired by the
unification spirit of LLMs in NLP, but do not explicitly leverage pre-trained
LLM in modeling. In contrast, the other two topics embrace and involve LLMs in
modeling, including training and chaining with LLMs, respectively.

    * –

Unified vision models for understanding and generation. In computer vision,
several attempts have been made to build a general-purpose foundation model by
combining the functionalities of specific-purpose multimodal models. To this
end, a unified model architecture is adopted for various downstream computer
vision and vision-language (VL) tasks. There are different levels of
unification. First, a prevalent effort is to bridge vision and language by
converting all closed-set vision tasks to open-set ones, such as CLIP (Radford
et al.,, 2021), GLIP (Li et al., 2022f, ), OpenSeg (Ghiasi et al., 2022a, ),
etc. Second, the unification of different VL understanding tasks across
different granularity levels is also actively explored, such as I/O
unification methods like UniTAB (Yang et al.,, 2021), Unified-IO (Lu et al.,
2022a, )), Pix2Seq-v2 (Chen et al., 2022d, ) and functional unification
methods like GPV (Gupta et al., 2022a, ), GLIP-v2 (Zhang et al., 2022b, )) and
X-Decoder (Zou et al., 2023a, ). In the end, it is also necessitated to make
the models more interactive and promptable like ChatGPT, and this has been
recently studied in SAM (Kirillov et al.,, 2023) and SEEM (Zou et al., 2023b,
).

    * –

Training with LLMs. Similar to the behavior of LLMs, which can address a
language task by following the instruction and processing examples of the task
in their text prompt, it is desirable to develop a visual and text interface
to steer the model towards solving a multimodal task. By extending the
capability of LLMs to multimodal settings and training the model end-to-end,
multimodal LLMs or large multimodal models are developed, including Flamingo
(Alayrac et al.,, 2022) and Multimodal GPT-4 (OpenAI, 2023a, ).

    * –

Chaining tools with LLM. Exploiting the tool use capabilities of LLMs, an
increasing number of studies integrate LLMs such as ChatGPT with various
multimodal foundation models to facilitate image understanding and generation
through a conversation interface. This interdisciplinary approach combines the
strengths of NLP and computer vision, enabling researchers to develop more
robust and versatile AI systems that are capable of processing visual
information and generating human-like responses via human-computer
conversations. Representative works include Visual ChatGPT (Wu et al., 2023a,
) and MM-ReAct (Yang* et al.,, 2023).

###  1.2 Definition and Transition from Specialists to General-Purpose
Assistants

Based on the model development history and taxonomy in NLP, we group
multimodal foundation models in Figure 1.2 into two categories.

  * •

Specific-Purpose Pre-trained Vision Models cover most existing multimodal
foundation models, including visual understanding models (_e.g._ , CLIP
(Radford et al.,, 2021), SimCLR (Chen et al., 2020a, ), BEiT (Bao et al.,,
2022), SAM (Kirillov et al.,, 2023)) and visual generation models (_e.g._ ,
Stable Diffusion (Rombach et al.,, 2021; sta,, 2022)), as they present
powerful transferable ability for specific vision problems.

  * •

General-Purpose Assistants refer to AI agents that can follow human intents to
complete various computer vision tasks in the wild. The meanings of general-
purpose assistants are two-fold: $(i)$ generalists with unified architectures
that could complete tasks across different problem types, and $(ii)$ easy to
follow human instruction, rather than replacing humans. To this end, several
research topics have been actively explored, including unified vision modeling
(Lu et al., 2022a, ; Zhang et al., 2022b, ; Zou et al., 2023a, ), training and
chaining with LLMs (Liu et al., 2023c, ; Zhu et al., 2023a, ; Wu et al.,
2023a, ; Yang* et al.,, 2023).

for tree= forked edges, grow’=0, draw, rounded corners, node
options=align=center,, text width=2.7cm, s sep=6pt, calign=edge midpoint, ,
[Multimodal
Foundation
Models, fill=gray!45, parent [Specific-Purpose Pre-trained Models, for tree=
top_class [Visual
Understanding §2, for tree=fill=red!45,encoder [Supervised Learning,
encoder_more [BiT (Kolesnikov et al.,, 2020); ViT (Dosovitskiy et al.,, 2021),
encoder_work] ] [Contrastive Language-Image Pre-training, encoder_more [CLIP
(Radford et al.,, 2021); ALIGN (Jia et al.,, 2021), encoder_work] ] [Image-
only Self-supervised Learning, encoder_more [MoCo (He et al.,, 2020); DINO
(Caron et al.,, 2021); MAE (He et al., 2022a, ), encoder_work] ] [Synergy
Among Different Methods, encoder_more [SLIP (Mu et al.,, 2021); UniCL (Yang et
al., 2022b, ), encoder_work] ] [Multimodal Fusion, encoder_more [UNITER (Chen
et al., 2020d, ); CoCa (Yu et al., 2022a, ), encoder_work] ] [Region-level and
Pixel-level Pre-training, encoder_more [GLIP (Li et al., 2022e, ); SAM
(Kirillov et al.,, 2023), encoder_work] ] ] [Visual
Generation §3, for tree=fill=green!45,generation [Overview: Text-to-Image
Generation, generation_more [Stable Diffusion (Rombach et al.,, 2021),
generation_work] ] [Spatial Controllable Generation, generation_more
[ControlNet (Zhang and Agrawala,, 2023), generation_work] ] [Text-based
Editing, generation_more [InstructPix2Pix (Brooks et al.,, 2023),
generation_work] ] [Text Prompts Following, generation_more [DDPO (Black et
al.,, 2023), generation_work] ] [Concept Customization, generation_more
[DreamBooth (Ruiz et al.,, 2023), generation_work] ] ] ] [General-Purpose
Assistants, for tree= top_class [Unified Vision
Models §4, for tree=fill=green!45,gpa [From Closed-set to Open-set Models,
gpa_wide [GLIP (Li et al., 2022f, ); OpenSeg (Ghiasi et al., 2022b, );
OpenSeeD (Zhang et al., 2023e, ), gpa_work] ] [From Task-Specific to Generic
Models, gpa_wide [Unified-IO (Lu et al., 2022a, ); X-Decoder (Zou et al.,
2023a, ), gpa_work] ] [From Static to Promptable Models, gpa_wide [SAM
(Kirillov et al.,, 2023); SEEM (Zou et al., 2023b, ); SegGPT (Wang et al.,
2023j, ), gpa_work] ] ] [Large Multimodal Models:
Training with LLM §5, for tree=fill=green!45,gpa [Image-to-Text Generation,
gpa_wide [Flamingo (Alayrac et al.,, 2022), gpa_work] ] [Instruction Tuning in
LLM, gpa_wide [ChatGPT (OpenAI,, 2022);
Vicuna (Vicuna,, 2023), gpa_work] ] [Instruction Tuning in LMM, gpa_wide
[Multimodal GPT-4 (OpenAI, 2023a, );
LLaVA (Liu et al., 2023c, );
MiniGPT4 (Zhu et al., 2023a, ), gpa_work] ] [Emerging Topics, gpa_wide] ]
[Multimodal Agents:
Chaining Tools with LLM §6, for tree=fill=green!45,gpa [Multimodel Agent,
gpa_wide [VISPROG (Gupta and Kembhavi, 2022a, );
Visual ChatGPT (Wu et al., 2023a, );
MM-REACT (Yang* et al.,, 2023), gpa_work] ] [Advanced Topics, gpa_wide] ] ] ]

Figure 1.3: An overview of the paper’s structure, detailing Chapters 2-6.

###  1.3 Who Should Read this Paper?

This paper is based on our CVPR 2023 tutorial,222https://vlp-
tutorial.github.io/2023/index.html with researchers in the computer vision and
vision-language multimodal communities as our primary target audience. It
reviews the literature and explains topics to those who seek to learn the
basics and recent advances in multimodal foundation models. The target
audiences are graduate students, researchers and professionals who are not
experts of multimodal foundation models but are eager to develop perspectives
and learn the trends in the field. The structure of this paper is illustrated
in Figure 1.3. It consists of 7 chapters.

  * •

Chapter 1 introduces the landscape of multimodal foundation model research,
and presents a historical view on the transition of research from specialists
to general-purpose assistants.

  * •

Chapter 2 introduces different ways to consume visual data, with a focus on
how to learn a strong image backbone.

  * •

Chapter 3 describes how to produce visual data that aligns with human intents.

  * •

Chapter 4 describes how to design unified vision models, with an interface
that is interactive and promptable, especially when LLMs are not employed.

  * •

Chapter 5 describes how to train an LLM in an end-to-end manner to consume
visual input for understanding and reasoning.

  * •

Chapter 6 describes how to chain multimodal tools with an LLM to enable new
capabilities.

  * •

Chapter 7 concludes the paper and discusses research trends.

###### Relations among Chapters 2-6.

Chapter 2-6 are the core chapters of this survey paper. An overview of the
structure for these chapters are provided in Figure 1.2. We start with a
discussion of two typical multimodal foundation models for specific tasks,
including visual understanding in Chapter 2 and visual generation in Chapter
3. As the notion of multimodal foundation models are originally based on
visual backbone/representation learning for understanding tasks, we first
present a comprehensive review to the transition of image backbone learning
methods, evolving from early supervised methods to the recent language-image
contrastive methods, and extend the discussion on image representations from
image-level to region-level and pixel-level (Chapter 2). Recently, generative
AI is becoming increasingly popular, where vision generative foundation models
have been developed. In Chapter 3, we discuss large pre-trained text-to-image
models, and various ways that the community leverage the generative foundation
models to develop new techniques to make them better aligned with human
intents. Inspired by the recent advances in NLP that LLMs serve as general-
purpose assistants for a wide range of language tasks in daily life, the
computer vision community has been anticipating and attempting to build
general-purpose visual assistants. We discuss three different ways to build
general-purpose assistants. Inspired by the spirit of LLMs, Chapter 4 focuses
on unifying different vision models of understanding and generation without
explicitly incorporating LLMs in modeling. In contrast, Chapter 5 and Chapter
6 focus on embracing LLMs to build general-purpose visual assistants, by
explicitly augmenting LLMs in modeling. Specifically, Chapter 5 describes end-
to-end training methods, and Chapter 6 focuses on training-free approaches
that chain various vision models to LLMs.

###### How to read the paper.

Different readers have different backgrounds, and may have different purposes
of reading this paper. Here, we provide a few guidance.

  * •

Each chapter is mostly self-contained. If you have a clear goal and a clear
research direction that you want to focus on, then just jump to the
corresponding chapter. For example, if you are interested in building a mini
prototype using OpenAI’s multimodal GPT-4, then you can directly jump to
Chapter 5.

  * •

If you are a beginner of multimodal foundation models, and are interested in
getting a glimpse of the cutting-edge research, we highly recommend that you
read the whole paper chapter by chapter in order, as the early chapters serve
as the building blocks of later chapters, and each chapter provides the
description of the key concepts to help you understand the basic ideas, and a
comprehensive literature review that to help you grasp the landscape and state
of the art.

  * •

If you already have rich experience in multimodal foundation models and are
familiar with the literature, feel free to jump to specific chapters you want
to read. In particular, we include in most chapters a section to discuss
advanced topics and sometimes provide our own perspectives, based on the up-
to-date literature. For example, in Chapter 6, we discuss several important
aspects of multimodal agents in tool use, including tool creation and its
connection to retrieval-augmented methods.

###  1.4 Related Materials: Slide Decks and Pre-recorded Talks

This survey paper extends what we present in the CVPR 2023 tutorial by
covering the most recent advances in the field. Below, we provide a list of
slide decks and pre-recorded talks, which are related to the topics in each
chapter, for references.

  * •

Chapter 2:  Visual and Vision-Language Pre-training (Youtube, Bilibili)

  * •

Chapter 3: Alignments in Text-to-Image Generation (Youtube, Bilibili)

  * •

Chapter 4: From Representation to Interface: The Evolution of Foundation for
Vision Understanding (Youtube, Bilibili)

  * •

Chapter 5: Large Multimodal Models (Youtube, Bilibili)

  * •

Chapter 6: Multimodal Agents: Chaining Multimodal Experts with LLMs (Youtube,
Bilibili)

##  Chapter 2 Visual Understanding

Over the past decade, the research community has devoted significant efforts
to study the acquisition of high-quality, general-purpose image
representations. This is essential to build vision foundation models, as pre-
training a strong vision backbone to learn image representations is
fundamental to all types of computer vision downstream tasks, ranging from
image-level (_e.g._ , image classification (Krizhevsky et al.,, 2012), image-
text retrieval (Frome et al.,, 2013), image captioning (Chen et al.,, 2015)),
region-level (_e.g._ , object detection (Girshick,, 2015), phrase grounding
(Plummer et al.,, 2015)), to pixel-level (_e.g._ , semantic/instance/panoptic
segmentation (Long et al.,, 2015; Hafiz and Bhat,, 2020; Kirillov et al.,,
2019)) tasks.

In this chapter, we present how image representations can be learned, either
using supervision signals mined inside the images, or through using language
supervision of image-text datasets mined from the Web. Specifically, Section
2.1 presents an overview of different learning paradigms, including supervised
pre-training, contrastive language-image pre-training (CLIP), and image-only
self-supervised learning. Section 2.2 discusses supervised pre-training.
Section 2.3 focuses on CLIP. Section 2.4 discusses image-only self-supervised
learning, including contrastive learning, non-contrastive learning, and masked
image modeling. Given the various learning approaches to training vision
foundation models, Section 2.5 reviews how they can be incorporated for better
performance. Lastly, Section 2.6 discusses how vision foundation models can be
used for finer-grained visual understanding tasks, such as fusion-encoder-
based pre-training for image captioning and visual question answering that
require multimodal fusion, region-level pre-training for grounding, and pixel-
level pre-training for segmentation.

Figure 2.1: An overview of the structure of Chapter 2.

###  2.1 Overview

There is a vast amount of literature on various methods of learning general-
purpose vision backbones. As illustrated in Figure 2.1, we group these methods
into three categories, depending on the types of supervision signals used to
train the models, including:

  * •

Label supervision: Arguably, the most well-studied image representation
learning methods are based on label supervisions (typically in the form of
image classification) (Sun et al.,, 2017), where datasets like ImageNet
(Krizhevsky et al.,, 2012) and ImageNet21K (Ridnik et al.,, 2021) have been
popular, and larger-scale proprietary datasets are also used in industrial
labs (Sun et al.,, 2017; Singh et al., 2022b, ; Zhai et al., 2022a, ; Wu et
al., 2023d, ).

  * •

Language supervision: Another popular approach to learning image
representations leverages weakly supervised signals from text, which is easy
to acquire in large scale. For instance, CLIP (Radford et al.,, 2021) and
ALIGN (Jia et al.,, 2021) are pre-trained using a contrastive loss and
billions of image-text pairs mined from the internet. The resultant models
achieve strong zero-shot performance on image classification and image-text
retrieval, and the learned image and text encoders have been widely used for
various downstream tasks and allow traditional computer vision models to
perform open-vocabulary CV tasks (Gu et al.,, 2021; Ghiasi et al., 2022a, ;
Qian et al.,, 2022; Ding et al., 2022b, ; Liang et al., 2023a, ; Zhang et al.,
2023e, ; Zou et al., 2023a, ; Minderer et al.,, 2022).

  * •

Image-only self-supervision: There is also a vast amount of literature on
exploring image-only self-supervised learning methods to learn image
representations. As the name indicates, the supervision signals are mined from
the images themselves, and popular methods range from contrastive learning
(Chen et al., 2020a, ; He et al.,, 2020), non-contrastive learning (Grill et
al.,, 2020; Chen and He,, 2021; Caron et al.,, 2021), to masked image modeling
(Bao et al.,, 2022; He et al., 2022a, ).

An illustration of these learning methods is shown in Figure 2.2. Besides the
methods of pre-training image backbones, we will also discuss pre-training
methods that allow multimodal fusion (_e.g._ , CoCa (Yu et al., 2022a, ),
Flamingo (Alayrac et al.,, 2022)), region-level and pixel-level image
understanding (_e.g._ , GLIP (Li et al., 2022e, ) and SAM (Kirillov et al.,,
2023)). These methods typically rely on a pre-trained image encoder or a pre-
trained image-text encoder pair. Figure 2.3 shows an overview of the topics
covered in this chapter and some representative works in each topic.

###  2.2 Supervised Pre-training

Supervised pre-training on large-scale human-labeled datasets, such as
ImageNet (Krizhevsky et al.,, 2012) and ImageNet21K (Ridnik et al.,, 2021),
has emerged as a widely adopted approach to acquiring transferable visual
representations. It aims to map an image to a discrete label, which is
associated with a visual concept. This approach has greatly expedited progress
in designing various vision backbone architectures (_e.g._ , AlexNet
(Krizhevsky et al.,, 2012), ResNet (He et al.,, 2016), vision transformer
(Dosovitskiy et al.,, 2021), and Swin transformer (Liu et al.,, 2021)), and is
the testbed for all the modern vision backbones. It also powered computer
vision tasks across the whole spectrum, ranging from image classification,
object detection/segmentation, visual question answering, image captioning, to
video action recognition. However, the effectiveness of learned
representations is often limited by the scale and diversity of supervisions in
pre-training datasets, as human annotation is expensive.

Figure 2.2: A high-level overview of different approaches to learn general
image representations, including supervised learning (Krizhevsky et al.,,
2012), contrastive language-image pre-training (Radford et al.,, 2021; Jia et
al.,, 2021), and image-only self-supervised learning, including contrastive
learning (Chen et al., 2020a, ; He et al.,, 2020), non-contrastive learning
(Grill et al.,, 2020; Chen and He,, 2021), and masked image modeling (Bao et
al.,, 2022; He et al., 2022a, ). Figure 2.3: An overview of the topics covered
in this chapter and representative works in each topic. We start from
supervised learning and CLIP, and then move on to image-only self-supervised
learning, including contrastive learning, non-contrastive learning, and masked
image modeling. Lastly, we discuss pre-training methods that empower
multimodal fusion, region-level and pixel-level image understanding.

###### Large-scale datasets.

For larger-scale pre-training, noisy labels can be derived in large quantities
from image-text pairs crawled from the Web. Using noisy labels, many
industrial labs have successfully constructed comprehensive classification
datasets using semi-automatic pipelines, such as JFT (Sun et al.,, 2017; Zhai
et al., 2022a, ) and I2E (Wu et al., 2023d, ), or by leveraging proprietary
data like Instagram hashtags (Singh et al., 2022b, ). The statistics of
existing large-scale image classification datasets are shown in Table 2.1. The
labels are typically in the form of fine-grained image entities with a long-
tailed distribution. Though classical, this approach has been very powerful
for learning universal image representations. For example, JFT-300M (Sun et
al.,, 2017) has been used for training the BiT (“Big Transfer”) models
(Kolesnikov et al.,, 2020), and JFT-3B (Zhai et al., 2022a, ) has been used to
scale up the training of a plain vision transformer (Dosovitskiy et al.,,
2021) to 22B in model size. LiT (Zhai et al., 2022b, ) proposes to first learn
the image backbone on JFT-3B (Zhai et al., 2022a, ), and keep it frozen and
learn another text tower to align the image and text embedding space to make
the model open-vocabulary and is capable of performing zero-shot image
classification.

###### Model training.

There are many loss functions that can be used to promote embedding properties
(_e.g._ , separability) (Musgrave et al.,, 2020). For example, the large
margin loss (Wang et al.,, 2018) is used for MOFI training (Wu et al., 2023d,
). Furthermore, if the datasets have an immense number of labels (can
potentially be over 2 million as in MOFI (Wu et al., 2023d, )), predicting all
the labels in each batch becomes computationally costly. In this case, a fixed
number of labels is typically used for each batch, similar to sampled softmax
(Gutmann and Hyvärinen,, 2010).

Dataset | # Images | # Classes
---|---|---
ImageNet-1K (Russakovsky et al.,, 2015) | 1.2M | 1K
ImageNet-21K (Ridnik et al.,, 2021) | 14M | 21K
JFT-300M (Sun et al.,, 2017) | 300M |  18K
JFT-3B (Zhai et al., 2022a, ) | 3B |  30K
IG-3.6B (Singh et al., 2022b, ) | 3.6B |  27K
I2E (Wu et al., 2023d, ) | 1.1B | 2M
Table 2.1: Statistics of existing large-scale image classification datasets.
Figure 2.4: Illustration of contrastive language-image pre-training, and how
the learned model can be used for zero-shot image classification. Image
credit: Radford et al., (2021).

###  2.3 Contrastive Language-Image Pre-training

####  2.3.1 Basics of CLIP Training

Language is a richer form of supervision than classical closed-set labels.
Rather than deriving noisy label supervision from web-crawled image-text
datasets, the alt-text can be directly used for learning transferable image
representations, which is the spirit of contrastive language-image pre-
training (CLIP) (Radford et al.,, 2021). In particular, models trained in this
way, such as ALIGN (Jia et al.,, 2021), Florence (Yuan et al.,, 2021), BASIC
(Pham et al.,, 2021), and OpenCLIP (Ilharco et al.,, 2021), have showcased
impressive zero-shot image classification and image-text retrieval
capabilities by mapping images and text into a shared embedding space. Below,
we discuss how the CLIP model is pre-trained and used for zero-shot
prediction.

  * •

Training: As shown in Figure 2.4(1), CLIP is trained via simple contrastive
learning. CLIP is an outstanding example of “ _simple algorithms that scale
well_ ” (Li et al., 2023m, ). To achieve satisfactory performance, model
training needs to be scaled along three dimensions: batch size, data size, and
model size (Pham et al.,, 2021). Specifically, the typical batch size used for
CLIP training can be 16k or 32k. The number of image-text pairs in the pre-
training datasets is frequently measured in billions rather than millions. A
vision transformer trained in this fashion can typically vary from 300M
(Large) to 1B (giant) in model size.

  * •

Zero-shot prediction: As shown in Figure 2.4 (2) and (3), CLIP empowers zero-
shot image classification via reformatting it as a retrieval task and
considering the semantics behind labels. It can also be used for zero-shot
image-text retrieval by its design. Besides this, the aligned image-text
embedding space makes it possible to make all the traditional vision models
open vocabulary and has inspired a rich line of work on open-vocabulary object
detection and segmentation (Li et al., 2022e, ; Zhang et al., 2022b, ; Zou et
al., 2023a, ; Zhang et al., 2023e, ).

####  2.3.2 CLIP Variants

Since the birth of CLIP, there have been tons of follow-up works to improve
CLIP models, as to be discussed below. We do not aim to provide a
comprehensive literature review of all the methods, but focus on a selected
set of topics.

###### Data scaling up.

Data is the fuel for CLIP training. For example, OpenAI’s CLIP was trained on
400M image-text pairs mined from the web, while ALIGN used a proprietary
dataset consisting of 1.8B image-text pairs. In BASIC (Pham et al.,, 2021),
the authors have carefully studied the scaling among three dimensions: batch
size, data size, and model size. However, most of these large-scale datasets
are not publicly available, and training such models requires massive
computing resources.

In academic settings, researchers (Li et al., 2022b, ) have advocated the use
of a few millions of image-text pairs for model pre-training, such as CC3M
(Sharma et al.,, 2018), CC12M (Changpinyo et al.,, 2021), YFCC (Thomee et
al.,, 2016). Relatively small-scale image-text datasets that are publicly
available include SBU (Ordonez et al.,, 2011), RedCaps (Desai et al.,, 2021),
and WIT (Srinivasan et al.,, 2021). Large-scale public available image-text
datasets include Shutterstock (Nguyen et al.,, 2022), LAION-400M (Schuhmann et
al.,, 2021), COYO-700M (Byeon et al.,, 2022), and LAION-2B (Schuhmann et al.,,
2022), to name a few. For example, LAION-2B (Schuhmann et al.,, 2022) has been
used by researchers to study the reproducible scaling laws for CLIP training
(Cherti et al.,, 2023).

Interestingly, in search of the next-generation image-text datasets, in
DataComp (Gadre et al.,, 2023), instead of fixing the dataset and designing
different algorithms, the authors propose to select and rank datasets using
the fixed CLIP training method. Besides paired image-text data mined from the
Web for CLIP training, inspired by the interleaved image-text dataset M3W
introduced in Flamingo (Alayrac et al.,, 2022), there have been recent efforts
of collecting interleaved image-text datasets, such as MMC4 (Zhu et al.,
2023b, ) and OBELISC (Laurençon et al.,, 2023).

Figure 2.5: ImageBind (Girdhar et al.,, 2023) proposes to link a total of six
modalities into a common embedding space via leveraging pre-trained CLIP
models, enabling new emergent alignments and capabilities. Image credit:
Girdhar et al., (2023).

###### Model design and training methods.

CLIP training has been significantly improved. Below, we review some
representative works.

  * •

Image tower: On the image encoder side, FLIP (Li et al., 2023m, ) proposes to
scale CLIP training via masking. By randomly masking out image patches with a
high masking ratio, and only encoding the visible patches as in MAE (He et
al., 2022a, ), the authors demonstrate that masking can improve training
efficiency without hurting the performance. The method can be adopted for all
CLIP training. Cao et al., (2023) found that filtering out samples that
contain text regions in the image improves CLIP training efficiency and
robustness.

  * •

Language tower: On the language encoder side, K-Lite (Shen et al., 2022a, )
proposes to use external knowledge in the form of Wiki definition of entities
together with the original alt-text for contrastive pre-training. Empirically,
the use of enriched text descriptions improves the CLIP performance. LaCLIP
(Fan et al., 2023a, ) shows that CLIP can be improved via rewriting the noisy
and short alt-text using large language models such as ChatGPT.

  * •

Interpretability: The image representation is typically a dense feature
vector. In order to improve the interpretability of the shared image-text
embedding space, STAIR (Chen et al., 2023a, ) proposes to map images and text
to a high-dimensional, sparse, embedding space, where each dimension in the
sparse embedding is a (sub-)word in a large dictionary in which the predicted
non-negative scalar corresponds to the weight associated with the token. The
authors show that STAIR achieves better performance than the vanilla CLIP with
improved interpretability.

  * •

More modalities: The idea of contrastive learning is general, and can go
beyond just image and text modalities. For example, as shown in Figure 2.5,
ImageBind (Girdhar et al.,, 2023) proposes to encode six modalities into a
common embedding space, including images, text, audio, depth, thermal, and IMU
modalities. In practice, a pre-trained CLIP model is used and kept frozen
during training, which indicates that other modality encoders are learned to
align to the CLIP embedding space, so that the trained model can be applied to
new applications such as audio-to-image generation and multimodal LLMs (_e.g._
, PandaGPT (Su et al.,, 2023)).

Figure 2.6: A high-level comparison of contrastive loss and captioning loss
for image encoder pre-training. (a) CLIP (Radford et al.,, 2021) uses
contrastive loss alone for pre-training, which enables zero-shot image
classification and has demonstrated strong scaling behavior. (b) VirTex (Desai
and Johnson,, 2021) uses captioning loss alone for pre-training. SimVLM (Wang
et al., 2022g, ) uses prefix language modeling for pre-training in a much
larger scale. The model architecture is similar to multimodal language models
(_e.g._ , GIT (Wang et al., 2022a, ) and Flamingo (Alayrac et al.,, 2022)),
but VirTex and SimVLM aim to pre-train the image encoder from scratch. (c)
CoCa (Yu et al., 2022a, ) uses both contrastive and captioning losses for pre-
training. The model architecture is similar to ALBEF (Li et al., 2021b, ), but
CoCa aims to pre-train the image encoder from scratch, instead of using a pre-
trained one.

###### Objective function.

The use of contrastive loss alone is powerful, especially when the model is
scaled up. However, other objective functions can also be applied.

  * •

Fine-grained supervision: Instead of using a simple dot-product to calculate
the similarity of an image-text pair, the supervision can be made more fine-
grained via learning word-patch alignment. In FILIP (Yao et al., 2022b, ), the
authors propose to first compute the loss by calculating the token-wise
similarity, and then aggregating the matrix by max-pooling for word-patch
alignment.

  * •

Contrastive captioner: Besides the contrastive learning branch, CoCa (Yu et
al., 2022a, ) (shown in Figure 2.6(c)) adds a generative loss to improve
performance and allow new capabilities that require multimodal fusion (_e.g._
, image captioning and VQA). This is similar to many fusion-encoder-based
vision-language models such as ALBEF (Li et al., 2021b, ), but with the key
difference in that CoCa aims to learn a better image encoder from scratch. A
detailed discussion on multimodal fusion is in Section 2.6.1.

  * •

Captioning loss alone: How about using the captioning loss alone to pre-train
an image encoder? Actually, before CLIP was invented, VirTex (Desai and
Johnson,, 2021) (shown in Figure 2.6(b)) and ICMLM (Sariyildiz et al.,, 2020)
learn encoders using a single image captioning loss, but the scale is very
small (restricted to COCO images) and the performance is poor. CLIP also shows
that contrastive pre-training is a much better choice. In SimVLM (Wang et al.,
2022g, ), the authors found that the learned image encoder was not as
competitive as CLIP. However, in the recent work Cap/CapPa (Tschannen et al.,,
2023), the authors argue that image captioners are scalable vision learners,
too. Captioning can exhibit the same or even better scaling behaviors.

  * •

Sigmoid loss for language-image pre-training: Unlike standard contrastive
learning with softmax normalization, Zhai et al., (2023) uses a simple
pairwise sigmoid loss for image-text pre-training, which operates on image-
text pairs and does not require a global view of the pairwise similarities for
normalization. The authors show that the use of simple sigmoid loss can also
achieve strong performance on zero-shot image classification.

Figure 2.7: Overview of SimCLR (Chen et al., 2020a, ), SimSiam (Chen and He,,
2021), and DINO (Caron et al.,, 2021) for self-supervised image representation
learning. SimCLR uses contrastive learning for model training, while SimSiam
and DINO explores non-contrastive learning methods. Image credit: Chen et al.,
2020a , Chen and He, (2021), Caron et al., (2021).

###  2.4 Image-Only Self-Supervised Learning

Now, we shift our focus to image-only self-supervised learning, and divide the
discussion into three parts: ($i$) contrastive learning, ($ii$) non-
contrastive learning, and ($iii$) masked image modeling.

####  2.4.1 Contrastive and Non-contrastive Learning

###### Contrastive learning.

The core idea of contrastive learning (Gutmann and Hyvärinen,, 2010; Arora et
al.,, 2019) is to promote the positive sample pairs and repulse the negative
sample pairs. Besides being used in CLIP, contrastive learning has also been a
popular concept in self-supervised image representation learning (Wu et al.,,
2018; Ye et al., 2019b, ; Tian et al., 2020a, ; Chen et al., 2020a, ; He et
al.,, 2020; Misra and Maaten,, 2020; Chen et al., 2020c, ). It has been shown
that the contrastive objective, known as the InfoNCE loss (Oord et al.,,
2018), can be interpreted as maximizing the lower bound of mutual information
between different views of the data (Hjelm et al.,, 2018; Bachman et al.,,
2019; Henaff,, 2020).

In a nutshell, all the image-only contrastive learning methods (_e.g._ ,
SimCLR (Chen et al., 2020a, ), see Figure 2.7(a), MoCo (He et al.,, 2020),
SimCLR-v2 (Chen et al., 2020b, ), MoCo-v2 (Chen et al., 2020c, )) share the
same high-level framework, detailed below.

  * •

Given one image, two separate data augmentations are applied;

  * •

A base encoder is followed by a project head, which is trained to maximize
agreement using a contrastive loss (_i.e._ , they are from the same image or
not);

  * •

The project head is thrown away for downstream tasks.

However, a caveat of contrastive learning is the requirement of a large number
of negative samples. These samples can be maintained in a memory bank (Wu et
al.,, 2018), or directly from the current batch (Chen et al., 2020a, ), which
suggests the requirement of a large batch size. MoCo (He et al.,, 2020)
maintains a queue of negative samples and turns one branch into a momentum
encoder to improve the consistency of the queue. Initially, contrastive
learning was primarily studied for pre-training convolutional networks.
However, with the rising popularity of vision transformers (ViT), researchers
have also explored its application in the context of ViT. (Chen et al., 2021b,
; Li et al., 2021a, ; Xie et al.,, 2021).

###### Non-contrastive learning.

Recent self-supervised learning methods do not depend on negative samples. The
use of negatives is replaced by asymmetric architectures (_e.g._ , BYOL (Grill
et al.,, 2020), SimSiam (Chen and He,, 2021)), dimension de-correlation
(_e.g._ , Barlow twins (Zbontar et al.,, 2021), VICReg (Bardes et al.,, 2021),
Whitening (Ermolov et al.,, 2021)), and clustering (_e.g._ , SWaV (Caron et
al.,, 2020), DINO (Caron et al.,, 2021), Caron et al., (2018); Amrani et al.,
(2022); Assran et al., (2022); Wang et al., 2023b ), _etc_.

For example, as illustrated in Figure 2.7(b), in SimSiam (Chen and He,, 2021),
two augmented views of a single image are processed by an identical encoder
network. Subsequently, a prediction MLP is applied to one view, while a stop-
gradient operation is employed on the other. The primary objective of this
model is to maximize the similarity between the two views. It is noteworthy
that SimSiam relies on neither negative pairs nor a momentum encoder.

Another noteworthy method, known as DINO (Caron et al.,, 2021) and illustrated
in Figure 2.7(c), takes a distinct approach. DINO involves feeding two
distinct random transformations of an input image into both the student and
teacher networks. Both networks share the same architecture but have different
parameters. The output of the teacher network is centered by computing the
mean over the batch. Each network outputs a feature vector that is normalized
with a temperature softmax applied to the feature dimension. The similarity
between these features is quantified using a cross-entropy loss. Additionally,
a stop-gradient operator is applied to the teacher network to ensure that
gradients propagate exclusively through the student network. Moreover, DINO
updates the teacher’s parameters using an exponential moving average of the
student’s parameters.

Figure 2.8: Overview of BEiT pre-training for image transformers. Image
credit: Bao et al., (2022).

####  2.4.2 Masked Image Modeling

Masked language modeling (Devlin et al.,, 2019) is a powerful pre-training
task that has revolutionized the NLP research. To mimic the success of BERT
pre-training for NLP, the pioneering work BEiT (Bao et al.,, 2022), as
illustrated in Figure 2.8, proposes to perform masked image modeling (MIM) to
pre-train image transformers. Specifically,

  * •

Image tokenizer: In order to perform masked token prediction, an image
tokenizer is required to tokenize an image into discrete visual tokens, so
that these tokens can be treated just like an additional set of language
tokens. Some well-known learning methods for image tokenziers include VQ-VAE
(van den Oord et al.,, 2017), VQ-VAE-2 (Razavi et al.,, 2019), VQ-GAN (Esser
et al.,, 2021), ViT-VQGAN (Yu et al.,, 2021), _etc_. These image tokenizers
have also been widely used for autoregressive image generation, such as DALLE
(Ramesh et al., 2021a, ), Make-A-Scene (Gafni et al.,, 2022), Parti (Yu et
al., 2022b, ), to name a few.

  * •

Mask-then-predict: The idea of MIM is conceptually simple: models accept the
corrupted input image (_e.g._ , via random masking of image patches), and then
predict the target of the masked content (_e.g._ , discrete visual tokens in
BEiT). As discussed in iBOT (Zhou et al.,, 2021), this training procedure can
be understood as knowledge distillation between the image tokenizer (which
serves as the teacher) and the BEiT encoder (which serves as the student),
while the student only sees partial of the image.

Figure 2.9: Illustration of Masked Autoencoder (MAE) (He et al., 2022a, ) that
uses raw pixel values for MIM training, and MaskFeat (Wei et al.,, 2021) that
uses different features as the targets. HOG, a hand-crafted feature
descriptor, was found to work particularly well in terms of both performance
and efficiency. Image credit: He et al., 2022a  and Wei et al., (2021).

###### Targets.

In Peng et al., 2022b , the authors have provided a unified view of MIM: a
teacher model, a normalization layer, a student model, an MIM head, and a
proper loss function. The most significant difference among all these models
lies in the reconstruction targets, which can be pixels, discrete image
tokens, features from pre-trained models, and outputs from the momentum
updated teacher. Specifically, the targets can be roughly grouped into two
categories.

  * •

Low-level pixels/features as targets: MAE (He et al., 2022a, ), SimMIM (Xie et
al., 2022b, ), ConvMAE (Gao et al.,, 2022), HiViT (Zhang et al., 2022d, ), and
GreenMIM (Huang et al., 2022a, ) leverage either original or normalized pixel
values as the target for MIM. These methods have typically explored the use of
a plain Vision Transformer (Dosovitskiy et al.,, 2021) or the Swin Transformer
(Liu et al.,, 2021) as the backbone architecture. MaskFeat (Wei et al.,, 2021)
introduced the Histogram of Oriented Gradients (HOG) feature descriptor as the
target for MIM (see Figure 2.9(b)). Meanwhile, Ge2-AE (Liu et al., 2023b, )
employed both pixel values and frequency information obtained from the 2D
discrete Fourier transform as the target. Taking MAE (He et al., 2022a, ) as
an example (Figure 2.9(a)), the authors show that using pixel values as
targets works particularly well. Specifically, a large random subset of images
(_e.g._ , 75%) is masked out; then, the image encoder is only applied to
visible patches, while mask tokens are introduced after the encoder. It was
shown that such pre-training is especially effective for object detection and
segmentation tasks, which require finer-grained image understanding.

  * •

High-level features as targets: BEiT (Bao et al.,, 2022), CAE (Chen et al.,
2022g, ), SplitMask (El-Nouby et al.,, 2021), and PeCo (Dong et al.,, 2023)
involve the prediction of discrete tokens using learned image tokenizers.
MaskFeat (Wei et al.,, 2021) takes a different approach by proposing direct
regression of high-level features extracted from models like DINO (Caron et
al.,, 2021) and DeiT (Touvron et al.,, 2021). Expanding this idea, MVP (Wei et
al., 2022b, ) and EVA (Fang et al.,, 2023) make feature prediction using image
features from CLIP as target features. Additionally, other methods such as
data2vec (Baevski et al.,, 2022), MSN (Assran et al.,, 2022), ConMIM (Yi et
al.,, 2022), SIM (Tao et al.,, 2023), and BootMAE (Dong et al.,, 2022) propose
to construct regression feature targets by leveraging momentum-updated teacher
models to enhance online learning. The choice of loss functions depends on the
nature of the targets: cross-entropy loss is typically used when the targets
are discrete tokens, while $\ell_{1}$, $\ell_{2}$, or cosine similarity losses
are common choices for pixel values or continuous-valued features.

###### MIM for video pre-training.

Naturally, there are recent works on extending MIM to video pre-training.
Prominent examples include BEVT (Wang et al., 2022c, ), MAE as spatiotemporal
learner (Feichtenhofer et al.,, 2022), VideoMAE (Tong et al.,, 2022), and
VideoMAEv2 (Wang et al., 2023e, ). Taking Feichtenhofer et al., (2022) as an
example. This paper studies a conceptually simple extension of MAE to video
pre-training via randomly masking out space-time patches in videos and learns
an autoencoder to reconstruct them in pixels. Interestingly, the authors found
that MAE learns strong video representations with almost no inductive bias on
space-time, and spacetime-agnostic random masking performs the best, with an
optimal masking ratio as high as 90%.

###### Lack of learning global image representations.

MIM is an effective pre-training method that provides a good parameter
initialization for further model finetuning. However, the vanilla MIM pre-
trained model does not learn a global image representation. In iBOT (Zhou et
al.,, 2021), the authors propose to enhance BEiT (Bao et al.,, 2022) with a
DINO-like self-distillation loss (Caron et al.,, 2021) to force the [CLS]
token to learn global image representations. The same idea has been extended
to DINOv2 (Oquab et al.,, 2023).

Figure 2.10: Overview of UniCL (Yang et al., 2022a, ) that performs unified
contrastive pre-training on image-text and image-label data. Image credit:
Yang et al., 2022a .

###### Scaling properties of MIM.

MIM is scalable in terms of model size. For example, we can perform MIM pre-
training of a vision transformer with billions of parameters. However, the
scaling property with regard to data size is less clear. There are some recent
works that aim to understand the data scaling of MIM (Xie et al., 2023b, ; Lu
et al., 2023a, ); however, the data scale is limited to millions of images,
rather than billions, except Singh et al., (2023) that studies the
effectiveness of MAE as a so-called “pre-pretraining” method for billion-scale
data. Generally, MIM can be considered an effective regularization method that
helps initialize a billion-scale vision transformer for downstream tasks;
however, whether or not scaling the MIM pre-training to billion-scale image-
only data requires further exploration.

###  2.5 Synergy Among Different Learning Approaches

Till now, we have reviewed different approaches to pre-training image
backbones, especially for vision transformers. Below, we use CLIP as the
anchor point, and discuss how CLIP can be combined with other learning
methods.

###### Combining CLIP with label supervision.

Noisy labels and text supervision can be jointly used for image backbone pre-
training. Some representative works are discussed below.

  * •

UniCL (Yang et al., 2022a, ) proposes a principled way to use image-label and
image-text data together in a joint image-text-label space for unified
contrastive learning, and Florence (Yuan et al.,, 2021) is a scaled-up version
of UniCL. See Figure 2.10 for an illustration of the framework.

  * •

LiT (Zhai et al., 2022b, ) uses a pre-trained ViT-g/14 image encoder learned
from supervised pre-training on the JFT-3B dataset, and then makes the image
encoder open-vocabulary by learning an additional text tower via contrastive
pre-training on image-text data. Essentially, LiT teaches a text model to read
out good representations from a pre-trained image model for new tasks.

  * •

MOFI (Wu et al., 2023d, ) proposes to learn image representations from 1
billion noisy entity-annotated images, and uses both image classification and
contrastive losses for model training. For image classification, entities
associated with each image are considered as labels, and supervised pre-
training on a large number of entities is conducted; for constrastive pre-
training, entity names are treated as free-form text, and are further enriched
with entity descriptions.

###### Combining CLIP with image-only (non-)contrastive learning.

CLIP can also be enhanced with image-only self-supervision. Specifically,

  * •

SLIP (Mu et al.,, 2021) proposes a conceptually simple idea to combine SimCLR
(Chen et al., 2020a, ) and CLIP for model training, and shows that SLIP
outperforms CLIP on both zero-shot transfer and linear probe settings. DeCLIP
(Li et al., 2022g, ) mines self-supervised learning signals on each modality
to make CLIP training data-efficient. In terms of image supervision, the
SimSam framework (Chen and He,, 2021) is used.

  * •

xCLIP (Zhou et al., 2023c, ) makes CLIP non-contrastive via introducing
additional sharpness and smoothness regularization terms borrowed from the
image-only non-contrastive learning literature. However, the authors show that
only non-contrastive pre-training (nCLIP) is not sufficient to achieve strong
performance on zero-shot image classification, and it needs to be combined
with the original CLIP for enhanced performance.

Figure 2.11: Illustration of MVP (Wei et al., 2022b, ), EVA (Fang et al.,,
2023) and BEiTv2 (Peng et al., 2022a, ). (a) & (b) MVP and EVA directly
regress CLIP features for MIM pre-training. (c) BEiTv2 compresses the
information inside CLIP features into discrete visual tokens, and then
performing regular BEiT training. (d) Alternative learning between CLIP and
MIM. Image credit: Wei et al., 2022b , Fang et al., (2023), Peng et al., 2022a
, Fang et al., (2023). Figure 2.12: Overview of BEiT-3 that performs masked
data modeling on both image/text and joint image-text data via a multiway
transformer. Image credit: Wang et al., 2022d .

###### Combining CLIP with MIM.

There are recent works that aim to combine CLIP and MIM for model training. We
group them into two categories.

  * •

Shallow interaction. It turns out that image features extracted from CLIP are
a good target for MIM training, as the CLIP image features potentially capture
the semantics that are missing in MIM training. Along this line of work, as
shown in Figure 2.11, MVP (Wei et al., 2022b, ) proposes to regress CLIP
features directly, while BEiTv2 (Peng et al., 2022a, ) first compresses the
information inside CLIP features into discrete visual tokens, and then
performs regular BEiT training. Similar use of CLIP features as MIM training
target has also been investigated in EVA (Fang et al.,, 2023), CAEv2 (Zhang et
al., 2022c, ), and MaskDistill (Peng et al., 2022b, ). In EVA-02 (Fang et
al.,, 2023), the authors advocate alternative learning of MIM and CLIP
representations. Specifically, an off-the-shelf CLIP model is used to provide
a feature target for MIM training; while the MIM pre-trained image backbone is
used to initialize CLIP training. The MIM representations are used to finetune
various downstream tasks while the learned frozen CLIP embedding enables zero-
shot image classification and other applications.

  * •

Deeper integration. However, instead of using CLIP as targets for MIM
training, if one aims to combine CLIP and MIM for joint model training, MIM
does not seem to improve a CLIP model at scale (Weers et al.,, 2023; Li et
al., 2023m, ).

  * •

Although the combination of CLIP and MIM does not lead to a promising result
at the current stage, the combination of BERT and BEiT is very promising, as
evidenced in BEiT-3 (Wang et al., 2022d, ) (see Figure 2.12), where the
authors show that masked data modeling can be performed on both image/text and
joint image-text data via the design of a multiway transformer, and state-of-
the-art performance can be achieved on a wide range of vision and vision-
language tasks.

###  2.6 Multimodal Fusion, Region-Level and Pixel-Level Pre-training

Till now, we have focused on the methods of pre-training image backbones from
scratch, but not on pre-training methods that power multimodal fusion, region-
level and pixel-level image understanding. These methods typically use a pre-
trained image encoder at the first hand to perform a second-stage pre-
training. Below, we briefly discuss these topics.

Figure 2.13: Illustration of UNITER (Chen et al., 2020d, ) and CoCa (Yu et
al., 2022a, ), which serve as a classical and a modern model that performs
pre-training on multimodal fusion. CoCa also pre-trains the image backbone
from scratch. Specifically, UNITER extracts image features via an off-the-
shelf object detector and treat image features as soft prompts of the text
input to be sent into a multimodal transformer. The model is pre-trained over
a few millions of image-text pairs. For CoCa, an image encoder and a text
encoder is used, with a multimodal transformer stacked on top. Both
contrastive loss and captioning loss are used for model training, and the
model is trained over billions of image-text pairs and JFT data. Image credit:
Chen et al., 2020d , Yu et al., 2022a .

####  2.6.1 From Multimodal Fusion to Multimodal LLM

For dual encoders such as CLIP (Radford et al.,, 2021), image and text are
encoded separately, and modality interaction is only handled via a simple dot
product of image and text feature vectors. This can be very effective for
zero-shot image classification and image-text retrieval. However, due to the
lack of deep multimodal fusion, CLIP alone performs poorly on the image
captioning (Vinyals et al.,, 2015) and visual question answering (Antol et
al.,, 2015) tasks. This requires the pre-training of a fusion encoder, where
additional transformer layers are typically employed to model the deep
interaction between image and text representations. Below, we review how these
fusion-encoder pre-training methods are developed over time.

###### OD-based models.

Most early methods use pre-trained object detectors (ODs) to extract visual
features. Among them, ViLBERT (Lu et al.,, 2019) and LXMERT (Tan and Bansal,,
2019) use co-attention for multimodal fusion, while methods like VisualBERT
(Li et al., 2019b, ), Unicoder-VL (Li et al., 2020a, ), VL-BERT (Su et al.,,
2019), UNITER (Chen et al., 2020d, ), OSCAR (Li et al., 2020b, ), VILLA (Gan
et al.,, 2020) and VinVL (Zhang et al.,, 2021) treat image features as soft
prompts of the text input to be sent into a multimodal transformer.

###### End-to-end models.

Now, end-to-end pre-training methods become the mainstream. Some early methods
use CNNs to extract image features, such as PixelBERT (Huang et al.,, 2020),
SOHO (Huang et al.,, 2021), and CLIP-ViL (Shen et al., 2022b, ), while ViLT
(Kim et al.,, 2021) and ViTCAP (Fang et al.,, 2022) directly feed image patch
features and text token embeddings into a multimodal transformer. Due to the
popularity of vision transformer (ViT), now most methods simply use ViT as the
image encoder (_e.g._ , plain ViT (Dosovitskiy et al.,, 2021) and Swin
transformer (Liu et al.,, 2021)). Prominent examples include ALBEF (Li et al.,
2021b, ), METER (Dou et al., 2022b, ), VLMo (Wang et al., 2021b, ), X-VLM
(Zeng et al.,, 2022), BLIP (Li et al., 2022d, ), SimVLM (Wang et al., 2022g,
), FLAVA (Singh et al., 2022a, ) and CoCa (Yu et al., 2022a, ).

An illustration of UNITER (Chen et al., 2020d, ) and CoCa (Yu et al., 2022a, )
is shown in Figure 2.13. They serve as two examples of a classical model and a
modern model, respectively, which performs pre-training on multimodal fusion.
CoCa also performs image backbone pre-training directly, as all the model
components are trained from scratch. Please refer to Chapter 3 of Gan et al.,
(2022) for a comprehensive literature review.

###### Trend to multimodal LLM.

Instead of using masked language modeling, image-text matching and image-text
contrastive learning, SimVLM (Wang et al., 2022g, ) uses a simple PrefixLM
loss for pre-training. Since then, multimodal language models have become
popular. Early models focus on large-scale pre-training, such as Flamingo
(Alayrac et al.,, 2022), GIT (Wang et al., 2022a, ), PaLI (Chen et al., 2022h,
), PaLI-X (Chen et al., 2023g, ), while recent works focus on using pre-
trained LLMs for instruction tuning, such as LLaVA (Liu et al., 2023c, ) and
MiniGPT-4 (Zhu et al., 2023a, ). A detailed discussion on this topic is
provided in Chapter 5.

####  2.6.2 Region-Level Pre-training

CLIP learns global image representations via contrastive pre-training.
However, for tasks that require fine-grained image understanding such as
object detection, CLIP is not enough. Object detection contains two sub-tasks:
localization and recognition. ($i$) Localization aims to locate the presence
of objects in an image and indicate the position with a bounding box, while
($ii$) recognition determines what object categories are present in the
bounding box. By following the reformulation that converts image
classification to image retrieval used in CLIP, generic open-set object
detection can be achieved.

Specifically, ViLD (Gu et al.,, 2021) and RegionCLIP (Zhong et al., 2022a, )
distill knowledge from CLIP with a two-stage detector for zero-shot object
detection. In MDETR (Kamath et al.,, 2021) and GLIP (Li et al., 2022e, ) (as
shown in Figure 2.14), the authors propose to reformulate detection as a
phrase grounding problem, and perform grounded language-image pre-training.
GLIPv2 (Zhang et al., 2022b, ) and FIBER (Dou et al., 2022a, ) further perform
unified pre-training for both grounding and vision-language understanding
tasks. OVR-CNN (Zareian et al.,, 2021) finetunes an image-text model to
detection on a limited vocabulary and relies on image-text pre-training for
generalization to an open vocabulary setting. Detic (Zhou et al., 2022b, )
improves long-tail detection performance with weak supervision by training
only the classification head on the examples where only image-level
annotations are available. Other works include OV-DETR (Zang et al.,, 2022),
X-DETR (Cai et al.,, 2022), FindIT (Kuo et al.,, 2022), PromptDet (Feng et
al., 2022a, ), OWL-ViT (Minderer et al.,, 2022), GRiT (Wu et al., 2022b, ), to
name a few. Recently, Grounding DINO (Liu et al., 2023h, ) is proposed to
marry DINO (Zhang et al., 2022a, ) with grounded pre-training for open-set
object detection. Please refer to Section 4.2 for a detailed review of this
topic.

Figure 2.14: Overview of GLIP that performs grounded language-image pre-
training for open-set object detection. Image credit: Li et al., 2022f .

####  2.6.3 Pixel-Level Pre-training

The Segment Anything Model (SAM) (Kirillov et al.,, 2023) is a recent vision
foundation model for image segmentation that aims to perform pixel-level pre-
training. Since its birth, it has attracted wide attention and spurred tons of
follow-up works and applications. Below, we briefly review SAM, as a
representative work for pixel-level visual pre-training.

As depicted in Figure 2.15, the objective of the Segment Anything project is
to develop a foundational vision model for segmentation. This model is
designed to be readily adaptable to a wide range of both existing and novel
segmentation tasks, such as edge detection, object proposal generation,
instance segmentation, open-vocabulary segmentation, and more. This
adaptability is seamlessly accomplished through a highly efficient and user-
friendly approach, facilitated by the integration of three interconnected
components. Specifically,

  * •

Task. The authors propose the promptable segmentation task, where the goal is
to return a valid segmentation mask given any segmentation prompt, such as a
set of points, a rough box or mask, or free-form text.

  * •

Model. The architecture of SAM is conceptually simple. It is composed of three
main components: ($i$) a powerful image encoder (MAE (He et al., 2022a, ) pre-
trained ViT); ($ii$) a prompt encoder (for sparse input such as points, boxes,
and free-form text, the CLIP text encoder is used; for dense input such as
masks, a convolution operator is used); and ($iii$) a lightweight mask decoder
based on transformer.

  * •

Data. To acquire large-scale data for pre-training, the authors develop a
_data engine_ that performs model-in-the-loop dataset annotation.

###### Concurrent to SAM.

Parallel to SAM, many efforts have been made to develop general-purpose
segmentation models as well. For example, OneFormer (Jain et al.,, 2023)
develops a universal image segmentation framework; SegGPT (Wang et al., 2023j,
) proposes a generalist in-context learning framework that unifies different
segmentation data formats; SEEM (Zou et al., 2023b, ) further expands the
types of supported prompts that a single segmentation model can handle,
including points, boxes, scribbles, masks, texts, and referred regions of
another image.

Figure 2.15: Overview of the Segment Anything project, which aims to build a
vision foundation model for segmentation by introducing three interconnected
components: a promptable segmentation task, a segmentation model, and a data
engine. Image credit: Kirillov et al., (2023).

###### Extensions of SAM.

SAM has spurred tons of follow-up works that extend SAM to a wide range of
applications, _e.g._ , Inpaint Anything (Yu et al., 2023c, ), Edit Everything
(Xie et al., 2023a, ), Any-to-Any Style Transfer (Liu et al., 2023g, ),
Caption Anything (Wang et al., 2023g, ), Track Anything (Yang et al., 2023b,
), Recognize Anything (Zhang et al., 2023n, ; Li et al., 2023f, ), Count
Anything (Ma et al.,, 2023), 3D reconstruction (Shen et al., 2023a, ), medical
image analysis (Ma and Wang,, 2023; Zhou et al., 2023d, ; Shi et al., 2023b, ;
Zhang and Jiao,, 2023), _etc_. Additionally, recent works have attempted to
develop models for detecting and segmenting anything in the open-vocabulary
scenarios, such as Grounding DINO (Liu et al., 2023h, ) and Grounding-
SAM111https://github.com/IDEA-Research/Grounded-Segment-Anything. For a
comprehensive review, please refer to Zhang et al., 2023a  and some GitHub
repos.222https://github.com/Hedlen/awesome-segment-anything

##  Chapter 3 Visual Generation

Visual generation aims to generate high-fidelity visual content, including
images, videos, neural radiance fields, 3D point clouds, etc.. This topic is
at the core of recently popular artificial intelligence generated content
(AIGC), and this ability is crucial in supporting creative applications such
as design, arts, and multimodal content creation. It is also instrumental in
synthesizing training data to help understand models, leading to the closed
loop of multimodal content understanding and generation. To make use of visual
generation, it is critical to produce visual data that is strictly aligned
with human intents. These intentions are fed into the generation model as
input conditions, such as class labels, texts, bounding boxes, layout masks,
among others. Given the flexibility offered by open-ended text descriptions,
text conditions (including text-to-image/video/3D) have emerged as a pivotal
theme in conditional visual generation.

In this chapter, we describe how to align with human intents in visual
generation, with a focus on image generation. We start with the overview of
the current state of text-to-image (T2I) generation in Section 3.1,
highlighting its limitations concerning alignment with human intents. The core
of this chapter is dedicated to reviewing the literature on four targeted
areas that aim at enhancing alignments in T2I generation, _i.e._ , spatial
controllable T2I generation in Section 3.2, text-based image editing in
Section 3.3, better following text prompts in Section 3.4, and concept
customization in T2I generation in Section 3.5. At the end of each subsection,
we share our observations on the current research trends and short-term future
research directions. These discussions coalesce in Section 3.6, where we
conclude the chapter by considering future trends. Specifically, we envision
the development of a generalist T2I generation model, which can better follow
human intents, to unify and replace the four separate categories of alignment
works.

###  3.1 Overview

####  3.1.1 Human Alignments in Visual Generation

(a) An overview of topics on human alignment for generative foundation models.
Image credit: Yang et al., 2023b ; Brooks et al., (2023); Chefer et al.,
(2023); Ruiz et al., (2023).

(b) Summary and categorization of papers on “Human Alignments in Visual
Generation.”

Figure 3.1: An overview of improving human intent alignments in T2I
generation.

AI Alignment research in the context of T2I generation is the field of study
dedicated to developing image generation models that can easily follow human
intents to synthesize the desired generated visual content. Current literature
typically focuses on one particular weakness of vanilla T2I models that
prevents them from accurately producing images that align with human intents.
This chapter delves into four commonly studied issues, as summarized in Figure
3.1 (a) and follows.

  * •

Spatial controllable T2I generation. Text serves as a powerful medium for
human-computer interaction, making it a focal point in conditional visual
generation. However, text alone falls short in providing precise spatial
references, such as specifying open-ended descriptions for arbitrary image
regions with precise spatial configurations. Spatial controllable T2I
generation (Yang et al., 2023b, ; Li et al., 2023n, ; Zhang and Agrawala,,
2023) aims to combine text inputs with other conditions for better
controllability, thereby facilitating users to generate the desired images.

  * •

Text-based image editing. Editing is another important means for acquiring
human-intended visual content. Users might possess near-perfect images,
whether generated by a model or naturally captured by a camera, but these
might require specific adjustments to meet their intent. Editing has diverse
objectives, ranging from locally modifying an object to globally adjusting the
image style. Text-based image editing (Brooks et al.,, 2023) explores
effective ways to create a versatile editing tool.

  * •

Better following text prompts. Despite T2I models being trained to reconstruct
images conditioned on the paired text input, the training objective does not
necessarily ensure or directly optimize for a strict adherence to text prompts
during image generation. Studies (Yu et al., 2022b, ; Rombach et al.,, 2022)
have shown that vanilla T2I models might overlook certain text descriptions
and generate images that do not fully correspond to the input text. Research
(Feng et al., 2022b, ; Black et al.,, 2023) along this line explores
improvements to have T2I models better following text prompts, thereby
facilitating the easier use of T2I models.

  * •

Visual concept customization. Incorporating visual concepts into textual
inputs is crucial for various applications, such as generating images of one’s
pet dog or family members in diverse settings, or crafting visual narratives
featuring a specific character. These visual elements often encompass
intricate details that are difficult to articulate in words. Alternatively,
studies (Ruiz et al.,, 2023; Chen et al., 2023f, ) explore if T2I models can
be customized to draw those visual concepts with specialized token embeddings
or conditioned images.

Before introducing the alignment works in detail, we first review the basics
of text-to-image generation in the next section.

####  3.1.2 Text-to-Image Generation

Figure 3.2: An overview of representative text-to-image generation models
until July 2023.

T2I generation aims to generate images that are not only of high visual
quality but also semantically correspond to the input text. T2I models are
usually trained with image-text pairs, where text is taken as input
conditions, with the paired image being the targeted output. Abstracted from
the wide range of T2I models shown in Figure 3.2, we give a high-level
overview of the representative image generation techniques.

  * •

Generative adversarial networks (GAN). GANs (Goodfellow et al.,, 2020;
Creswell et al.,, 2018; Kang et al.,, 2023) consist of two key components: a
generator and a discriminator. The generator is tasked with creating synthetic
images from random noise inputs, and it is trained to adjust these noise
inputs based on input text conditions to generate semantically relevant
images. In this adversarial process, the discriminator competes with the
generator, attempting to differentiate between the synthetically generated
images and real ones, thus guiding the generator to improve its image creation
capabilities.

  * •

Variational autoencoder (VAE) Variational Autoencoder (VAE) (Kingma and
Welling,, 2013; van den Oord et al.,, 2017; Vahdat and Kautz,, 2020) is a
probabilistic model that can generate images by employing paired encoder and
decoder network modules. The encoder network optimizes the encoding of an
image into a latent representation, while the decoder refines the process of
converting the sampled latent representations back into a new image. VAEs are
trained by minimizing the reconstruction error between the original and
decoded images, whileregularizing the encoded latent space using the Kullback-
Leibler (KL) divergence. Vector Quantised-VAE (VQ-VAE) (van den Oord et al.,,
2017) further improves VAEs by leveraging the discrete latent space through
vector quantization, enabling improved reconstruction quality and generative
capabilities.

  * •

Discrete image token prediction. At the core of this approach lies a
combination of a paired image tokenizer and detokenizer, like Vector Quantized
Generative Adversarial Networks (VQGAN) (Esser et al.,, 2021), which
efficiently transform continuous visual signals into a finite set of discrete
tokens. In this way, the image generation problem is converted to a discrete
token prediction task. A widely employed strategy for token prediction is to
use an auto-regressive Transformer (Ramesh et al., 2021b, ; Yu et al., 2022b,
) to sequentially generates visual tokens, typically starting from the top
left corner and moving row-by-row towards the bottom right, conditioned on the
text inputs. Alternatively, studies (Chang et al.,, 2022, 2023) also explore
the parallel decoding to speed up the token prediction process. Finally, the
predicted visual tokens are detokenized, culminating in the final image
prediction.

  * •

Diffusion model. Diffusion models (Sohl-Dickstein et al.,, 2015; Song and
Ermon,, 2020; Ho et al.,, 2020) employ stochastic differential equations to
evolve random noises into images. A diffusion model works by initiating the
process with a completely random image, and then gradually refining it over
multiple iterations in a denoising process. Each iteration predicts and
subsequently removes an element of noise, leading to a continuous evolution of
the image, conditioned on the input texts.

We use Stable Diffusion (SD) (Rombach et al.,, 2022) as an example to explain
in detail how T2I models work. We choose this model for a variety of reasons.
Firstly, SD is one of the most widely used open-source T2I models, which makes
it a solid foundation for many alignment techniques we discuss in this
chapter. Additionally, as a diffusion-based generation model, it serves as an
excellent case study for introducing diffusion models. Finally, its cross-
attention-based image-text fusion mechanism is a classic example of various
text-conditioned methods, such as auto-regressive T2I generation (Yu et al.,
2022b, ), helping us gain an in-depth understand of the image-text interaction
in T2I generation.

Figure 3.3: An overview of the latent diffusion model architecture. Image
credit: Rombach et al., (2022).

Stable Diffusion (SD)111We use Stable Diffusion v1 for the introduction. Later
versions such as SD2 and SDXL share the same method but may have different
detailed model configurations, such as a larger text encoder, U-Net, and
latent dimension., and its academic version latent diffusion (Rombach et al.,,
2022), contains mainly three modules, _i.e._ , an image VAE, a denoising
U-Net, and a condition encoder, as shown in the left, center, and right part
of Figure 3.3, respectively. We will introduce each module and the inference
flow for image generation, following the notations in the original latent
diffusion paper (Rombach et al.,, 2022).

  * •

VAE. As introduced in the image generation technique overview, the VAE module
contains a paired encoder $\mathcal{E}$ and decoder $\mathcal{D}$, trained to
encode RGB image $x$ into a latent random variable $z$ and then decode the
latent to best reconstruct the image. Given an RGB image
$x\in\mathbb{R}^{H\times W\times 3}$, the encoder $\mathcal{E}$ encodes it
into a continuous latent representation $z\in\mathbb{R}^{h\times w\times c}$.
With the parameters of $H=W=512$, $h=w=64$, and $c=4$ in SD, latent $z$ is
$48$ times smaller than image $x$, thereby significantly improving the
computational efficiency by performing the denoising process in this
compressed compact latent space.

  * •

Text encoder. SD is a conditional image generation model, where the input text
condition is encoded using a condition encoder $\tau$. Specifically, SD uses
the ViT-L/14 CLIP text encoder (Radford et al.,, 2021) that encodes the
tokenized input text query $y$ into text feature
$\tau(y)\in\mathbb{R}^{N\times d_{\tau}}$, where the maximum length $N$ is
$77$ and text feature dimension $d_{\tau}$ is $768$.

  * •

Denoising U-Net. The denoising U-Net is the core module for the diffusion
image generation process. The module is trained to predict the noise
$\hat{\epsilon}(z_{t},t)$ to subtract in the latent space at each denoising
timestep $t$, such that it can step-by-step evolve the initial random noise
into a meaningful image latent. The module is trained with the L2 loss between
the predicted noise $\hat{\epsilon}(z_{t},t)$ and the target noise $\epsilon$,
which is added to the target image latent encoded by VAE encoder
$\mathcal{E}$. At inference, the iteratively denoised latent $z$, started from
the random noise, is sent through the VAE decoder $\mathcal{D}$ for the final
generated image.

In each denoising step, the U-Net takes the text condition as input to
generate images that are semantically relevant to the text query. We next
detail how the visual stream $z\in\mathbb{R}^{h\times w\times c}$ interacts
with the text stream $\tau(y)\in\mathbb{R}^{N\times d_{\tau}}$. The denoising
U-Net, similar to a classic U-Net (Ronneberger et al.,, 2015; Long et al.,,
2015), consists of a series of spatial downsampling and upsampling blocks with
skip connections in between. In SD’s U-Net, each down/upsampling block has a
cross-attention layer and a 2D convolutional down/upsampling layer. Each block
takes the visual latent feature, text feature, and denoising step as input and
generates the next visual latent feature. The image-text interaction happens
in the image-text cross-attention layer.

$\text{Attention}(Q,K,V)=\text{softmax}\left(\frac{QK^{T}}{\sqrt{d}}\right)\cdot
V,$ (3.1)

where $K$, $V$ are projected from the text stream $\tau(y)$ and $Q$ is
projected from the visual stream $z$ to share the same hidden dimension $d$.
Therefore, the softmax between $Q$ and $K$ produces an attention map $M$ of
size $(hw\times d)\cdot(N\times d)^{T}=hw\times N$. The cross-attention map
$M$ indicates the fine-grained image-text interaction among each one of the
$N$ text words in all spatial positions $hw$. The attention map $M$ then
products $V$ to yield the output of a down/upsampling block.

###  3.2 Spatial Controllable Generation

T2I generation takes open-ended text for users to describe their intended
images. However, text alone is ineffective in certain descriptions, such as
spatial referring. Studies in spatial controllable T2I generation explore
extending T2I models to take extra spatial input conditions to guide image
generation process.

We categorize related studies into three topics. $(i)$ We start with works
(Yang et al., 2023b, ; Li et al., 2023n, ; Avrahami et al., 2023b, ; Cho et
al.,, 2023) that extend the image-level text description in vanilla T2I models
to the region-grounded text description, such that open-ended text
descriptions can precisely operate on a particularly spatial region. $(ii)$ We
then extend from boxes to dense spatial conditions represented as 2D arrays,
such as segmentation masks, edge maps, depth maps, key points. We review
representative works ControlNet (Zhang and Agrawala,, 2023) and many others
(Mou et al.,, 2023; Zeng et al.,, 2023; Zhao et al., 2023b, ; Qin et al.,
2023a, ). $(iii)$The previous two threads of work require finetuning T2I
models to understand the extended spatial condition. We next review techniques
of inference-time guidance (Bansal et al.,, 2023; Chen et al., 2023e, ) that
achieve spatial control without model finetuning.

Figure 3.4: Overview of the ReCo model architecture. Image credit: Yang et
al., 2023b .

Region-controlled T2I generation. Large-scale T2I models have demonstrated
remarkable efficacy in generating high-resolution images. However, the models
lack spatial controllability, _e.g._ , precisely specifying content in a
specified area using a free-form text description. This limitation motivates
the studies on region-controlled T2I generation. As shown in the left side of
Figure 3.4, these studies explore the extra input condition of open-ended text
descriptions on arbitrary regions (_i.e._ , region-controlled text),
augmenting the global image description in T2I models. This new input
condition requires T2I models to understand spatial inputs, and associate them
with grounded texts.

ReCo (Yang et al., 2023b, ) is among the most representative works along this
direction. The core idea is to extend the text vocabulary of the text encoder
$\mathcal{E}$ and arrange different tokens to represent the grounded text
inputs. The study augments text tokens understood using pre-trained T2I models
with an extra set of position tokens, which represent the quantized spatial
coordinates. As shown in Figure 3.4, the position tokens (_e.g._ ,
${<}687{>},{<}204{>},{<}999{>},{<}833{>}$) are seamlessly mixed with the text
tokens and operate as a spatial modifier, indicating that the text to follow
only operates on the specified spatial region, such as the “baseball player
…jersey.” The pre-trained T2I model is then finetuned to support such a new
input interface, thereby facilitating region-controlled T2I generation.

Shared by other approaches along this direction, ReCo discusses several
advantages of region-controlled T2I generation in improving the alignment with
human intents. $(i)$ The grounded texts provide an extra input condition that
allows users to specify the desired image easily, _i.e._ , having a free-form
regional description precisely at a specific location. The box token and the
input sequence design allow users to generate grounded text with the same user
interface as query a T2I model with text, making the extension easy to use.
$(ii)$ The additional region-level controlled texts help better generate
images with correct object count, spatial relationship, and region attributes
such as color/size, which may otherwise confuse the vanilla T2I model (Rombach
et al.,, 2022). $(iii)$ Studies also observe a better image generation
quality, with the conjecture that the grounded text provides object-level
image-text association and therefore simplifies the learning process.

GLIGEN (Li et al., 2023n, ) is another representative work. Alternate to
generating grounded descriptions through the expansion of input tokens and
finetuning the entire T2I model, GLIGEN uses a plug-and-play recipe: freezing
the original T2I model and training extra gated self-attention layers to learn
the new grounding skills. The grounding tokens carry two types of information:
the semantic representation of text words that need to be grounded in and
their spatial configurations. These grounding tokens are then added to the
pre-trained T2I model via a newly added gated self-attention layer, with all
remaining pre-trained parameters frozen. This layer is equipped with a gating
parameter, which is initialized to zero, allowing the pre-trained model to
incrementally incorporate the grounded text inputs. GLIGEN facilitates various
types of grounded controls, including bounding box grounding, keypoint
grounding, image prompting, as well as other types of spatially-aligned dense
conditions.

T2I generation with dense conditions. In addition to spatial coordinates,
there exist other spatial conditions often represented as 2D arrays, such as
segmentation masks, edge maps and depth maps. ControlNet (Zhang and Agrawala,,
2023) is a prominent example of incorporating these dense spatial controls
into T2I models. ControlNet is built upon Stable Diffusion, and introduces an
additional trainable ControlNet branch that adds an extra input condition to
the text prompt. This extra condition can be a canny edge map, hough line, HED
boundary, under sketching, human pose maps, segmentation masks, depth images,
normal maps, or line drawing, each enabled with its distinct model copy. The
added branch is initialized from the pre-trained downsampling blocks in the
SD’s U-Net. This branch takes the added visual latent and the extra dense
condition as input. Before combining input dense conditions with visual latent
in the input and merging the ControlNet branch’s output back to SD’s
upsampling blocks, there is a unique zero-initialized $1\times 1$
convolutional layer. This layer serves as a gated connector to gradually
inject the extra condition into the pre-trained Stable Diffusion model. With
the extra dense spatial control, ControlNet provides an effective channel of
generation controllability.

Figure 3.5: Examples of the dense controls and the corresponding generated
images. Image credit: Zhao et al., 2023b .

Follow-up studies such as Uni-ControlNet (Zhao et al., 2023b, ) and UniControl
(Qin et al., 2023a, ) further improve ControlNet by unifying the input
condition, such that a single model can understand multiple input condition
types or even take a combination of two conditions. Examples of the dense
controls and the corresponding generated images are shown in Figure 3.5.
Moreover, Disco (Wang et al., 2023f, ) exemplifies the efficiency of
ControlNet in the generation of human dancing videos, which aims to generate
videos with controllable elements such as human subjects, video backgrounds,
and motion pose sequences. The study successfully separates the background and
human pose conditions, which are fed into two distinct branches of ControlNet,
which condition on image frames and pose maps, respectively. This
disentanglement of control from all three conditions allows Disco to
accomplish high fidelity in both the human foregrounds and backgrounds. More
importantly, it enables the arbitrary compositionality of human subjects,
backgrounds, and dance movements.

Figure 3.6: Qualitative results of inference-time spatial guidance. Image
credit: Bansal et al., (2023).

Inference-time spatial guidance. The aforementioned works require model
training, either the T2I models or additional modules to understand the extra
spatial conditions. Alternatively, studies (Bansal et al.,, 2023; Chen et al.,
2023e, ) explore providing the inference-time spatial guidance to T2I models
without extra model training. The core idea is similar to classifier guidance
(Dhariwal and Nichol,, 2021), which takes a discriminator loss to guide the
diffusion process as follows:

$\hat{\epsilon}^{\prime}(z_{t},t)=\hat{\epsilon}(z_{t},t)+s(t)\cdot\triangledown_{z_{t}}\ell(c,f(\hat{z}_{0})).$
(3.2)

Taking spatial control as an example, the discriminator can be a Faster-RCNN
object detector (Ren et al.,, 2015) indicated by $f$, which operates on the
intermediate estimated image $\hat{z}_{0}$, and compute the object detection
loss $\ell$ with the desired layout $c$, to guide the generation
$\hat{\epsilon}(z_{t},t)$. $s(t)$ is the guidance strength. This approach
enables the spatial control in T2I generation without extra training, with
qualitative results shown in Figure 3.6. However, it may not yield results as
precise as those from finetuning methods (Yang et al., 2023b, ; Li et al.,
2023n, ; Zhang and Agrawala,, 2023).

Summary and trends. Early research on spatial controllable generation, such as
layout-to-image and mask-to-image generation, was often treated in parallel
with T2I generation. However, with the emergence of advanced large-scale T2I
models, recent studies, as discussed in this subsection, are now leaning
towards integrating spatial conditions with textual conditions. We identify
two primary trends in integrating spatial conditions into T2I models. First,
region-controllable T2I generation, such as ReCo, merges spatial coordinate
control into text inputs by enlarging the text vocabulary with position
tokens. Second, studies extended from ControlNet integrate an additional
“image-like” condition to T2I frameworks, thereby capturing a broad spectrum
of dense conditions. Moving forward, T2I models may have a finetuning stage
that allows them to comprehend both image and text inputs. In such a scenario,
box coordinates could be incorporated through text, while dense controls could
be provided as image inputs. We will explore and elaborate on this idea in
Section 3.5.

###  3.3 Text-based Editing

Text-to-image editing synthesizes new images from an given image and input
text descriptions. Instead of producing an image entirely from scratch, users
might already possess a satisfactory starting point; this could be an image
previously generated from a T2I model or a natural image. The objective is to
retain the majority of the visual content, only modifying specific components.
This could involve altering a local object or the overall image style to
precisely match the user’s intentions. This text-based editing approach offers
users a tool to generate fresh images based on a predecessor, playing a
crucial role in creating visual content that accurately follows human intent.

There are various definitions and task setups in text-based editing. We
introduce the following representative threads. $(i)$ One classic editing
scenario is to change a local image region, such as removing or changing an
object or adding an object in a certain region. Spatially manipulating the
latent in image generation according to the user-generated masks is a simple
but effective method (Avrahami et al., 2022b, ; Avrahami et al., 2022a, ; Meng
et al.,, 2021). Studies (Balaji et al.,, 2022; Hertz et al.,, 2022) also show
that manipulating the image-text cross-attention mask is effective for spatial
editing. $(ii)$ Extended from spatial editing where the language inputs
describe the desired appearance in the spatial region, language can also be
used as editing instruction to tell the machine what to do (Kawar et al.,,
2023; Brooks et al.,, 2023), such as “change object A in the image to object
B.” $(iii)$ Instead of extending a single T2I model for editing, editing
systems (Wu et al., 2023a, ) integrate different specialized modules such as
segmentation models (Kirillov et al.,, 2023; Zou et al., 2023b, ) and large
language models (Brown et al.,, 2020; OpenAI, 2023a, ).

Diffusion process manipulations. The multi-step denoising process in diffusion
image generation naturally supports a certain extent of image editing.
Stochastic Differential Editing (SDEdit) (Meng et al.,, 2021) shows that first
adding noises to the input image to edit and then subsequently denoising the
sample, could produce a meaningful edit. Blended Latent Diffusion (Avrahami et
al., 2022a, ) shows that the diffusion process manipulation can achieve local
object editing with a user-generated object mask $m_{latent}$. In each
diffusion step, the latent $z$ is a spatial blend of the foreground and
background latent: $z=z_{fg}\odot m_{latent}+z_{bg}\odot(1-m_{latent})$, where
$z_{fg}$ is the edited object generated from the text description and $z_{bg}$
is the original backgrund image with noises added.

Figure 3.7: Three types of editing (word swap, adding new phrases, attention
re-weighting) on synthetically generated images, enabled by attention map
manipulation. Image credit: Hertz et al., (2022).

However, there are certain limitations on blending spatial latents. Firstly,
it may not always be feasible to require human-generated masks. Secondly, the
generation process can sometimes result in artifacts at the edges. Instead of
simply blending the latent in a spatial manner, researchers delve into image-
text cross-attention maps to unearth clues for object editing. Specifically,
Prompt2Prompt (Hertz et al.,, 2022) discovers that cross-attention layers
control the interaction among visual regions and text words. Based on this
observation, the study enables three types of editing for images generated by
a diffusion T2I model, including word swap, adding new phrases, and attention
re-weighting, each of which is enabled with corresponding manipulation on the
image-text cross-attention map. Specifically, the Prompt2Prompt tracks both
cross-attention maps generated by the original prompt (namely $M_{t}$) and the
edited prompt (namely $M^{*}_{t}$), and merges the attention maps with pre-
defined rules into the new attention maps $\widehat{M}_{t}$, which is used for
latent computing. For example, while adding a new phrase, attention map
$\widehat{M}_{t}$ remains unaltered for words present in the original prompt.
It only incorporates the modified attention maps $M^{*}_{t}$ for words that
did not exist in the original prompt. Qualitative results of the edits are
shown in Figure 3.7.

Going beyond editing synthetically generated images, Imagic (Kawar et al.,,
2023) explores editing real natural images. The core idea is to represent the
image to be edited as text embedding, and blend this embedding with the target
text embedding describing the desired image. This blend ensures that the
resulting image retains elements from the original while aligning with the
aesthetics detailed in the target textual prompt. In practice, test-time
finetuning is needed to generate high-quality images.

Figure 3.8: Examples of text instruction editing. Image credit: Brooks et al.,
(2023).

Text instruction editing. Instead of repeating the visual contents of the
image to edit in the text prompts, it might be more efficient for users to
directly specify editing instructions using language, such as “swap sunflowers
with roses” in Figure 3.8. The desired text instruction editing model should
work on both model-generated and natural images, and across different types of
editing instructions.

Figure 3.9: The editing data generation pipeline proposed in InstructPix2Pix.
Image credit: Brooks et al., (2023).

InstructPix2Pix (Brooks et al.,, 2023) is designed to accept an image and a
text editing instruction to produce an edited version of the input image. The
goal is to train an image-to-image model that can understand such editing text
instructions. To achieve this, T2I models can be adapted to accept the
additional image input by incorporating more input channels into the SD’s
convolutional layer. The main challenge is how to generate paired editing
data. As shown in Figure 3.9, InstructPix2Pix (Brooks et al.,, 2023) proposes
to use a LMM (Brown et al.,, 2020) to generate a pair of an editing
instruction and an edited caption from the original input caption, _e.g._ ,
“have her ride a dragon,” “photograph of a girl riding a dragon,” and
“photograph of a girl riding a horse.” The study then uses Prompt2Prompt
(Hertz et al.,, 2022) to convert the original and edited caption pair to a
pair of images before and after editing, corresponding to the GPT-generated
editing instruction. The study generates over 450K samples to train the
editing model. This data generation method has also been adopted in subsequent
research, such as CM3Leon (Ge et al.,, 2023) for training general-purpose
image-text-to-image models.

Editing with external pre-trained models. Furthermore, recent studies show the
efficacy of incorporating external language and vision models for editing, as
opposed to relying solely on a single model. Advancements in generalist
segmentation models, such as SAM (Kirillov et al.,, 2023) and SEEM (Zou et
al., 2023b, ), have paved the way for using segmentation models to ground the
region for text-based editing. Representative works include Instruct X-Decoder
(Zou et al., 2023a, ), Grounded SAM inpainting (Liu et al., 2023h, ), Inpaint
anything (Yu et al., 2023c, ), etc.. Another emerging trend is the allocation
of various generation and editing tools through LMM. Studies such as
VisualChatGPT (Wu et al., 2023a, ) can solve complicated visual editing that
requires the collaboration of multiple generation and editing models in
multiple steps.

Summary and trends. Text-based editing models have made significant progress
in their capabilities, leading to improved editing quality, expanded domain
coverage, and more flexible user interface. For example, early studies require
user-generated masks for object editing, while recent models can work on
synthetically generated images without mask inputs, or even directly
understand general text editing instructions. As we look to the future, we
anticipate an all-encompassing generative foundation model that is capable of
processing both image and text inputs. Within this framework, editing
instructions would be a specialized form of text input, seamlessly integrated
with the image description in T2I generation.

###  3.4 Text Prompts Following

Training with image-text pairs encourages T2I models to generate images that
semantically correspond to the input text condition. However, the image
generation training objective does not directly enforce generated images to
exactly follow text prompts. Studies (Feng et al., 2022b, ; Chefer et al.,,
2023) show that T2I models may fail to follow text prompts, especially when
the image description becomes complicated. For example, certain noun phrases
may get omitted, attributes may apply to incorrect objects, and generated
images may have the wrong object count, relationship, styles, etc.. These
limitations motivate work on improving T2I models to better follow text
prompts.

The related literature can be broadly categorized into two main groups. $(i)$
Inference-time manipulation. In the inference stage, the latent and attention
adjustment (Liu et al., 2022a, ; Feng et al., 2022b, ; Chefer et al.,, 2023;
Agarwal et al.,, 2023) design various methods to redistribute the visual
latent or image-text cross-attention, such that all noun phrases in the text
prompts are represented in the generated image. $(ii)$ Alignment tuning. An
extra model learning stage is learned (Black et al.,, 2023; Fan et al., 2023b,
), typically with the image-text similarity as rewards, such that the tuned
T2I model can better follow text prompts.

Figure 3.10: Failure cases of vanilla T2I model in text prompt following.
Image credit: Feng et al., 2022b .

Inference-time manipulation. Training with image-text pairs does not guarantee
that T2I models consistently adhere to the text prompts. There can be multiple
discrepancies, particularly when the text descriptions are lengthy and
intricate. For instance, T2I models may apply attributes to the wrong entity
or miss certain objects, as shown in Figure 3.10. Intuitively, parsing the
text query at inference time and explicitly enforcing T2I models to pay closer
attention to each noun phrase may generate images that better follow text
prompts.

Building upon this intuition, StructureDiffusion (Feng et al., 2022b, )
employs a parsing tree to extract noun phrases and the text prompt’s
linguistic structure. The study then enforces the model to “look at” all
extracted noun phrases. This is implemented by modifying SD’s cross-attention
mechanism introduced in (3.1), written as $O=M\cdot V$ where $M$ is the
softmax cross-attention map. Instead of producing $M$ with the sentence
feature $V$, which may result in words getting overlooked, the study computes
the $O=\frac{1}{k+1}\Sigma_{i=0}^{k}(M\cdot V_{i})$, where $V_{0}$ is the
sentence feature $V$, and $V_{i},i=1,\ldots,k$ is the phrase feature in the
parsing tree. This approach ensures that the visual stream maintains a
balanced attention across all identified noun phrases, fostering more accurate
image generation.

Figure 3.11: Inference time guidance proposed in Attend-and-Excite. Image
credit: Chefer et al., (2023).

Motivated by the same objective to ensure that no object is overlooked,
Attend-and-Excite (Chefer et al.,, 2023) manipulates the attention map. As
shown in the right side equations in Figure 3.11, a regularization loss $\ell$
is computed to amplify the maximal attention towards the most neglected
subject token:

$\ell=\max_{n=1,\ldots,N_{\text{sub}}}(1-max\ G(M_{t}^{n}))$

where $G$ is a Gaussian kernel to smooth the attention map and
$N_{\text{sub}}$ is the number of subject tokens. The loss is then used to
update the latent $z_{t}$ at inference time:

$z_{t}^{\prime}=z_{t}-\alpha\triangledown_{z_{t}}\ell,$

where $\alpha$ is a scalar for the step size. Results show that this
inference-time guidance enables T2I models to focus more on objects described
in the text prompt, resulting in superior image generation. Follow-up studies
(Agarwal et al.,, 2023) further refine the guidance loss to optimize prompt-
following performance.

Model tuning to follow text prompt. Instead of inference-time manipulation,
one may wonder if we can refine a pre-trained T2I model to better follow text
prompts. One promising way to achieve this is via reinforcement learning,
using image-text similarity as reward instead of the image generation
objective used in the main T2I training. This allows the model to be optimized
towards a better image-text alignment.

One work along this direction is the denoising diffusion policy optimization
(DDPO) (Black et al.,, 2023), with the tuning framework shown in Figure 3.12.
Specifically, a vision-language model (Li et al., 2023e, ) is used to convert
the generated image into a text description. This generated caption is
compared with the input text prompt, deriving a similarity reward through the
use of BERTScore (Zhang et al.,, 2019). The similarity reward is then used to
finetune the pre-trained T2I model, such that the model can better follow the
text prompts. The bottom of Figure 3.12 shows the progression of the generated
sample during this similarity-based training. Further, it is worth noting that
other human intent may also be formatted as rewards for model tuning, such as
compressibility, aesthetic quality, etc.

Figure 3.12: DDPO with vision-language-model-based reward function for image-
text alignment tuning. Image credit: Black et al., (2023).

Summary and trends. In this section, we present studies aimed at enhancing the
capability of T2I models to better adhere to text prompts. Despite the good
performance achieved by the inference-time manipulation, the alignment tuning
provides a more intuitive user experience, eliminating the need for extra
modifications. In parallel to instruction tuning in LLMs to align human intent
for text generation, the TI2 model tuning shares a similar goal, but focuses
on image generation. We foresee a similar paradigm emerging in the near future
for generative T2I foundational model development. Specifically, the initial
training phase still relies on the existing image generation objective on
large-scale data, while the subsequent alignment-tuning phase enhances the
model’s prompt adherence and other aspects of human intent, such as
diminishing harmful content. Current RL-based methods show potentials, but
they typically zero in on a single optimization goal. Future research could
extend these methods for more adaptable alignment tuning, amalgamated with
features like accommodating diverse image and text scenarios.

###  3.5 Concept Customization

Though language is an powerful medium to express human intent, it is
inefficient in comprehensively describing all details of a visual concept for
reconstruction. For example, it is challenging to use texts to describe my pet
dog or family members with sufficient details, so that they can be generated
in different visual scenes. In such applications, directly extending T2I
models to understand visual concepts via image inputs is a better option.

We examine relevant research on visual concept customization, which offers
users the ability to generate these personalized concepts. $(i)$ Pioneer
studies (Gal et al.,, 2022; Ruiz et al.,, 2023; Wei et al.,, 2023) start with
single-concept customization that involves test-time finetuning to encode
multiple images of the visual concept into a new token embedding, such that
the learned embedding can be used to refer to the concept during T2I
generation. $(ii)$ Multi-concept customization (Kumari et al.,, 2023; Avrahami
et al., 2023a, ) allows multiple concept tokens to be expanded from the T2I
model’s token vocabulary, enabling multiple concepts to interact with each
other and the remaining visual scene during generation. $(iii)$ Test-time
finetuning requires users to tune T2I models for each new concept to
customize. To simplify the usage, studies (Chen et al., 2022f, ; Shi et al.,
2023a, ; Chen et al., 2023f, ; Yang et al., 2023a, ) explore customization
without test-time finetuning and uses a unified finetuning stage to extend T2I
models for accepting image condition inputs. The models take images of the
visual concept as an extra input condition, and generate images with the
visual concept following the text descriptions.

Figure 3.13: The problem setup and result visualization of the visual concept
customization task. Image credit: Ruiz et al., (2023).

Single-concept customization. The goal of visual concept customization is to
enable T2I models to comprehend additional visual concepts tailored to very
specific cases. The problem setup, studied in Textual Inversion (Gal et al.,,
2022), involves translating visual concepts from a handful of images into
unique token embeddings. As illustrated in the left side of Figure 3.13, the
T2I model processes four images of a distinct dog breed, subsequently learning
the embedding for a new token, denoted as $[\text{V}]$. This $[\text{V}]$
token can be used as a text token to represent this specific dog. The
$[\text{V}]$ token can be seamlessly integrated with other textual
descriptions to render the specific dog in various contexts, such as swimming,
in a bucket, and getting a haircut.

Textual Inversion (Gal et al.,, 2022) learns the $[\text{V}]$ token embedding
via prefix tuning, _i.e._ , freezing all T2I model’s parameters and training
the $[\text{V}]$ token embedding to generate the input images. Later studies
observe that tuning more model parameters leads to significantly better image
generation quality. However, adjusting only the input image may lead to the
risk of overfitting the T2I model for a particular concept, and losing the
capablity to generate diverse images. For instance, the model might become
unable to generate various dog types. To address this, Dreambooth (Ruiz et
al.,, 2023) proposes the class-specific prior preservation loss. Central to
this approach is using the pre-trained T2I model to produce images of the same
class as the targeted customization concept. The model is then jointly
finetuned on both the input image (with the $[\text{V}]$ token) and the model-
generated images (without the $[\text{V}]$ token). This ensures that the model
can differentiate between the unique “$[\text{V}]$ dog” and other general dogs
it was initially trained, thus maintaining its overall T2I capability.
Dreambooth then finetunes all T2I model parameters and achieves better image
generation quality.

Multi-concept customization. Building on studies that focused on learning a
single visual concept $[\text{V}]$, recent research has delved into the
possibility of integrating multiple visual concepts into a single Text-to-
Image (T2I) model, represented as $[\text{V}_{\text{1}}]$,
$[\text{V}_{\text{2}}]$, and so on. Custom Diffusion (Kumari et al.,, 2023)
employs a selective subset of model weights, specifically the key and value
mappings from text to latent features in the cross-attention layers for
concept customization, learned from multiple sets of concept images. The study
facilitates the ability to embed multiple customized visual concepts in a
single text prompt. Instead of learning from multiple sets of input images,
Break-A-Scene (Avrahami et al., 2023a, ) explores extracting multiple visual
concepts in a single image. The study augments input images with segmentation
masks to pinpoint the intended target concepts and subsequently transforms
them into a series of concept embeddings denoted as $[\text{V}_{\text{i}}]$.

Customization without test-time finetuning. While the concept customization
studies, as described above, have achieved good visual quality, the necessity
for test-time finetuning hinders its application in real-world settings. Most
end users and application platforms lack the compute resources required for
finetuning, not to mention the complexities of finetuning process. This
naturally leads to the question: can we take concept images as input
conditions, and achieve concept customization without finetuning?

The input/output format of the imagined system is similar to the retrieval-
augmented generation (Chen et al., 2022f, ), which aims to ease the image
generation by conditioning on a retrieved similar image. The system supports
extra image inputs that contain relevant information for the generation
process. By altering the conditioning images during the training phase, the
model can potentially achieve a broad in-context learning capability,
producing images that align with the given input examples. In line with this
framework, SuTI (Chen et al., 2023f, ) trains a single model to imitate the
finetuned subject-specific experts, and generates images conditioning on both
text and subject input images, as shown in Figure 3.14. As a result, the
trained model can perform in-context concept customization without test-time
finetuning, and generalize to unseen subjects and descriptions. Another
concurrent work, InstantBooth (Shi et al., 2023a, ), also shows remarkable
results in generating images that are not only aligned with language but also
preserve identities, with a single forward pass.

Figure 3.14: Illustration of in-context concept customization without test-
time finetuning. Image credit: Chen et al., 2023f .

Summary and trends. The field of visual concept customization has advanced
from finetuning embeddings during the testing stage, to directly performing
in-context image generation with a frozen model. The in-context generation
pipeline, which incorporates additional image inputs, shows remarkable
potentials in real-world applications. In this subsection, we have explored
two applications of this approach: facilitating generation through the
retrieval of pertinent images (Chen et al., 2022f, ), and personalizing visual
concepts by conditioning them on subject images (Chen et al., 2023f, ; Shi et
al., 2023a, ). An intriguing direction is to unify the diverse uses of image
inputs, directed by descriptive textual instructions. We elaborate on this
idea in the following sub-section.

###  3.6 Trends: Unified Tuning for Human Alignments

In previous subsections, we presented the literature related to tuning T2I
models to more accurately align with human intent. This includes enhancing
spatial controllability, editing existing images for improved alignment, more
effectively following text prompts, and personalizing T2I models for new
visual concepts. A trend observed across these subtopics is the shift towards
integrated alignment solutions that require minimal problem-specific
adjustments. Along this direction, we envision a future T2I model having a
unified alignment tuning stage, which transforms a pre-trained T2I model into
one that resonates more intimately with human intent. Such a model would
seamlessly process both text and image inputs, generating the intended visual
content without the need for multiple models tailored to different alignment
challenges.

Drawing parallels to the established practice of human-alignment tuning in LLM
development, we anticipate that the techniques reviewed in this section will
merge into a holistic second-stage tuning for generative foundation model
development. This alignment tuning phase serves two primary purposes. First,
it extends the T2I’s text inputs to incorporate interleaved image-text inputs,
as illustrated in Figure 3.15. Second, it finetunes the base T2I model, which
has been trained using image generation loss, by the employing data, loss, and
rewards that aim to align with human expectations.

Figure 3.15: Overview of the unified image and text input interface for human
alignments, and its connection to previous sub-sections.

Unified image and text inputs. We begin with the discussion on interface
unification. Specifically, we aim to evolve the textual inputs of T2I models
into a multimodal interface that seamlessly integrates both image and text
inputs. As shown in Figure 3.15, we consider three types of inputs to begin
with: ”content text input” characterizes the visual scene to be produced; the
”image input” accommodates dense 2D inputs such as images and dense
conditions; and the ”instruction text input” explains how the input content
texts and images should be collectively composed as the condition for
generation.

Vanilla T2I models, as shown in the first row of Figure 3.15, take the
“content text input” of the image description and generate the corresponding
image. For the spatial controllable generation in Section 3.2, the extra
spatial condition can be specified via text inputs by expanding text words
with extra box tokens, or via image input by feeding the dense spatial
conditions as an image input. For the text-based editing in Section 3.3, we
examine the efficacy of text instruction editing, a task that finetunes the
T2I model to comprehend editing instruction texts that manipulate the image
input, altering its pixel values accordingly. For visual concept customization
in Section 3.5, the training-free models can now understand customization
instructions to extract visual concepts from the image inputs, and combine the
concept with context text inputs for image generation.

Incorporating the three elements of the input interface, the envisioned
alignment-tuned T2I model can handle all previous tasks described in Section
3.2-3.5. Its behavior is steered by specific text instructions that dictate
how the image and text inputs should be jointly processed as the generation
condition. Given the same image input, different text instructions can invoke
different tasks: “generate a cat image with the same layout” for spatial
control, “change the dog’s color” for editing, “generate the same dog
sleeping” for concept customization, and the arbitrary mixture of the existing
modes. Achieving such a unified interface in generative foundational models
may be possible through training on a consolidated dataset encompassing data
from various tasks, drawing similarities to the success of supervised
instruction tuning observed in LLMs. Furthermore, transitioning from
processing a single image-text pair to handling interleaved image-text pairs
could enable more intriguing capabilities like in-context visual
demonstrations (Sun et al., 2023b, ). Another interesting direction is to
build a generative model that is capable of generating any combination of
output modalities, such as language, image, video, or audio, from any
combination of input modalities, as demonstrated in Composable Diffusion
(CoDi) (Tang et al., 2023b, ).

Tuning with alignment-focused loss and rewards. In addition to the unified
input interface, another noteworthy element deserving consideration is the
alignment-focused loss and rewards. As mentioned in Section 3.4, the image
generation loss based on image-text pairs enables models to produce images
that match the target data distribution. Yet, it doesn’t always perfectly
align with human intent. This is reminiscent of the language model loss in LLM