training, which necessitates a separate alignment tuning phase (Ouyang et
al.,, 2022). The recent success in supervised instruction tuning and
reinforcement learning from human feedback methods (Black et al.,, 2023) on
image generation provides effective tools for similar alignment tuning in
generative foundation models. An intriguing topic left for future exploration
is how to balance the different target losses and rewards, such as jointly
optimizing for higher aesthetic scores, better image-text alignment, fewer
harmful contents, stronger instruction adherence, along with many other
desired properties.

Closed-loop of multimodal content understanding and generation. As we look
ahead, one promising avenue of research is the closed-loop integration of
multimodal content understanding and generation. Preliminary studies have
shown the benefit of using synthesized data to benefit generation from
understanding (Li et al., 2023a, ; He et al., 2022b, ), and vice versa. An
exciting prospect would be the development of an image-text-input, image-text-
output foundational model for both understanding and generation tasks. The
ideal balance in combining these two dimensions, and the most efficient
approach to achieve it, are left for future explorations.

##  Chapter 4 Unified Vision Models

In this chapter, we discuss the unification of vision models. We start with an
overview of the challenges in the unification of vision models and the most
recent efforts towards this goal in Section 4.1. What follows are detailed
discussions on $(i)$ how to transform closed-set models to open-set ones in
Section 4.2; $(ii)$ how to unify different granularities of vision tasks in
Section 4.3; and $(iii)$ how to build a more promptable interface for vision
in Section 4.4. Finally, we summarize the chapter and discuss future trends in
Section 4.5.

###  4.1 Overview

Before talking about general-purpose unified vision systems, we revisit how
language models and natural language processing (NLP) have evolved in the past
years. Before 2018, different NLP tasks are addressed with different task-
specific models, such as translation (Bahdanau et al.,, 2015), semantic
parsing (Berant et al.,, 2013), summarization (Allahyari et al.,, 2017), and
so on. With the emergence of the transformer architecture (Vaswani et al.,,
2017), language models for different NLP tasks are unified with a decoder-only
architecture, e.g., the GPT models (Brown et al.,, 2020). Afterwards, the GPT
models learned using the next word prediction task are further finetuned to
follow human instructions. This leads to ChatGPT 111https://chat.openai.com/,
which fundamentally changes our expectations on what AI systems can do. The
evolution as depicted in Figure 1.1 motivates us to wonder whether we can
build a general-purpose vision system in a similar manner.

###### Challenges.

That computer vision tasks vary greatly presents a great challenge to build a
unified vision model. First, vision tasks have different types of inputs,
ranging from static images (Russakovsky et al.,, 2015) to sequential videos
(Miech et al.,, 2019), from pure vision inputs such as image dehazing (He et
al.,, 2010) to multi-modality inputs that include e.g., vision and language
Antol et al., (2015). Second, different granularities are required for
different tasks, such as image-level tasks like image classification (He et
al.,, 2016) and captioning (Vinyals et al.,, 2016), region-level tasks like
object detection (Girshick,, 2015) and grounding (Plummer et al.,, 2015), and
pixel-level tasks like image segmentation (He et al.,, 2017), super-resolution
(Wang et al.,, 2020), _etc._ As a result, the outputs of vision systems are
also of different formats, such as spatial information like edges, boxes, and
masks, semantic information like class labels, multi-label tags, or detailed
descriptions. In addition to the challenges in modeling, there are also
challenges with data. First, the cost of annotation varies greatly among
different types of labels. As shown in Figure 4.6, these labels are at
different levels of granularity and semantic richness, ranging from whole
images, regions (box annotations), to masks (pixel annotations). Second, it is
in general much more costly to collect image data than text data. So, the
scale of vision data is often much smaller than that of text corpora.

###### Towards a unified vision model.

Despite these challenges, there is a growing interest in the computer vision
community to develop a general-purpose, unified vision system, in particular
for visual understanding tasks. As illustrated in Figure 4.1, we group these
efforts in three categories:

  * •

Bridging vision and language. By extending closed-set classification to open-
world recognition, the contrastive language-image models like CLIP (Radford et
al.,, 2021) demonstrate impressive zero-shot transferability for different
vision tasks. These models learn the mapping between raw visual signals and
rich semantics and can power various open-vocabulary vision recognition tasks
(Zhong et al., 2022b, ; Gu et al.,, 2022; Li et al., 2022f, ; Ghiasi et al.,
2022b, ).

  * •

Unified multi-task modeling. Traditional task-specific vision models are
trained using task-specific data. It is often prohibitively expensive to
develop a model for a new task. Thus, it is desirable to develop a unified
vision model that can perform well across many vision tasks (Yang et al.,
2022c, ; Lu et al., 2022a, ; Zou et al., 2023a, ; Chen et al., 2022c, ).

  * •

LLM-like promptable interface. LLMs can take different language and in-context
prompts as inputs and produce user-desired outputs without finetuning. A
general-purpose vision model should have possessed the same in-context
learning capability to align the output to various user intents without
changing its model parameters (Bar et al.,, 2022; Kirillov et al.,, 2023; Zou
et al., 2023b, ; Wang et al., 2023j, ; Balažević et al.,, 2023).

In what follows, we will elaborate the detailed techniques and methods in each
category.

Figure 4.1: In NLP, we have witnessed a clear trend to build a unified model
such as GPT-3 (Brown et al.,, 2020) and then the sophisticated Human-AI
interaction system ChatGPT, which has ignited the interests of the whole
community and society in AI. A natural question for computer vision (CV) is
whether we can unify all different types of vision tasks such as image
classification, object detection, segmentation and visual question answering,
etc., and likewise build an interaction interface between CV models and
humans. Aspired by this, a lot of attempts have been made recently to crack
the problem from different angles including but not limited to (a) making
vision models open-set; (b) unifying different granularities; and (c) turning
the models more promptable.

###  4.2 From Closed-Set to Open-Set Models

Traditionally, visual recognition is formulated as a classification problem
that maps raw visual data (e.g., images) to discrete text labels. For example,
image classification predicts a label from a pre-defined close set for a whole
image (Deng et al.,, 2009), and object detection identifies the objects,
defined in a close set, within an image (Lin et al.,, 2014). However, such
_closed-set_ models can hardly transfer to other tasks where the close set (or
vocabulary) is insufficient. For example, it is difficult to apply an object
detector trained using the Microsoft COCO object set
222https://cocodataset.org/ to detect Minecraft objects. Recently, CLIP
(Radford et al.,, 2021) addresses the limitation of closed-set models by
introducing a contrastive language-image pre-training method to train an
_open-set_ model. As illustrated in Figure 4.2 (a), instead of learning the
mapping from input to labels, CLIP learns an aligned visual-semantic space
using hundreds of millions of image-text pairs. Mathematically, the
traditional vision tasks optimize the log-likelihood of assigning label $y=c$
to an image, often represented as a feature vector $u\in\mathcal{R}^{P}$:

$\log\mathcal{P}(y=c|u)=\log\frac{\exp^{w_{c}\cdot
u}}{\sum_{i=1}^{K}\exp^{w_{i}\cdot u}},$ (4.1)

where $w\in\mathcal{R}^{K\times P}$ is the projection matrix. Instead of using
a pre-determined project matrix $w$, the CLIP method uses a text encoder
${Enc}_{text}$ to for the projection:

$v_{i}={Enc}_{text}(Concept_{i}),$ (4.2)

where $v$ plays the role of $w$ in Eq. (4.1). The reason why a text encoder
can help achieve open-set recognition is that all textual concepts are
embedded in the same feature space through large-scale pre-training, and the
feature distributions are coherent to the semantic meanings without the need
of a pre-defined vocabulary. As such, the aligned visual-semantic space can be
easily transferred to a wide range of image recognition tasks in a zero-shot
manner. Please refer to Chapter 2 for a detailed discussion. In the following,
we focus our discussion on the region-level and pixel-level models.

(a)

(b)

Figure 4.2: (a) As proposed in CLIP (Radford et al.,, 2021), replacing labels
with textual descriptions and using a text encoder to encode them can feasibly
convert closed-set problems to open-set ones. Image credit from Yang et al.,
2022b . (b) A number of works have been proposed to transform different
computer vision tasks by replacing the label space with language space, such
as UniCL (Yang et al., 2022b, ), GLIP (Li et al., 2022f, ) and OpenSeeD (Zhang
et al., 2023e, ).

After the release of the CLIP model (Radford et al.,, 2021), a number of open-
set vision models have been developed using large amounts of text-image pairs
for visual understanding at different levels of granularity (Yang et al.,
2022b, ; Zhang et al., 2023e, ; Li et al., 2022f, ; Ghiasi et al., 2022a, ),
ranging from image-level tasks (_e.g._ , image classification Deng et al.,
(2009), image-text retrieval, image captioning Chen et al., (2015)), region-
level localization (_e.g._ , object detection and phrase grounding Plummer et
al., (2015)), to pixel-level grouping tasks (e.g., image segmentation and
referring segmentation Long et al., (2015); Kirillov et al., (2019); Hafiz and
Bhat, (2020)). These models can be categorized along the following three
dimensions: model initialization, design and training.

###### Model initialization.

There are different initialization methods for open-set model training.

  * •

CLIP initialized. Many recent open-set models are trained by using a pre-
trained model such as CLIP for initialization since a pre-trained model
already provides a well-aligned (but often coarse-grained) visual-semantic
feature space. For example, OVR-CNN (Zareian et al.,, 2021) and RegionCLIP
(Zhong et al., 2022b, ) use a CLIP-style pre-trained ResNet (He et al.,, 2016)
as the vision encoder and a pre-trained RPN (Ren et al.,, 2015) to extract
regional features. Likewise, MaskCLIP (Zhou et al., 2022a, ) and FreeSeg (Qin
et al., 2023b, ) exploit the CLIP model to extract dense labels for pixels.
FC-CLIP (Yu et al., 2023a, ) uses a frozen convolution network ConvNeXt (Liu
et al., 2022b, ) in CLIP to encode input images of various resolutions.

  * •

CLIP augmented. Instead of initializing a model with CLIP parameters, other
methods initialize the model parameters as usually (e.g., setting random
values to model parameters), but use the pre-trained CLIP to help model
training.For example, ViLD (Gu et al.,, 2022) augments the model with aligned
CLIP features via knowledge-distillation. MaskCLIP (Ding et al., 2022b, ) and
Mask-Adapted CLIP Liang et al., 2023a  rely on the pre-trained CLIP model to
provide features and scores, respectively, during the course of model
training.

  * •

Other works learn a visual-semantic feature space using supervised pre-trained
models or from scratch. For example, GLIP (Li et al., 2022f, ) and OpenSeeD
(Zhang et al., 2023e, ) use a pre-trained BERT model (Devlin et al.,, 2019)
and the CLIP text encoder, respectively, and use a vision backbone pre-trained
on ImageNet for image encoding. Though these separately pre-trained image and
text encoders do not explicitly learn the alignment between image and
language, it turns out that these models still give good representations for
images and texts, and are instrumental to efficient model training.
Differently, GroupViT (Xu et al., 2022a, ) is trained jointly using an open-
set semantic segmentation task and a global image-text alignment task from
scratch. ODISE (Xu et al., 2023a, ) exploits pre-trained Stable Diffusion
models (Rombach et al.,, 2022) to extract compact masks.

###### Model design.

Open-set models can be either multi-stage or end-to-end.

  * •

Two-stage models. These models usually follow the design of the pre-DETR based
models (Ren et al.,, 2015; He et al.,, 2017), which decouples localization and
recognition. For object detection, a region proposal network is typically pre-
trained for localizing the object of interest (Zhong et al., 2022b, ; Gu et
al.,, 2021), and a mask proposal network for extracting masks (Ghiasi et al.,
2022a, ; Yao et al., 2022a, ). Given the localization results, a pre-trained
CLIP model is used to measure the similarity between visual contents and
language concepts. A clear advantage for two-stage models is that they can
inherit the open-set semantic understanding capacity without additional
training so as to devote modeling training to requiring a well-performed
localization network.

  * •

End-to-end models. Different from two-stage models, the end-to-end models
follow the DETR-based methods (Carion et al.,, 2020; Cheng et al.,, 2022) or
other one-stage models (Dai et al.,, 2021). GLIP (Li et al., 2022f, ) is one
of the representative works. GLIP formulates object detection as textual
grounding and is trained end-to-end on image-text pairs with detection and
grounding labels. Follow-up works enhance GLIP by enabling deeper vision-
language interactions (Liu et al., 2023h, ) or using DETR-like model design
(Zang et al.,, 2022; Minderer et al.,, 2022). For segmentation, both ZegFormer
(Ding et al., 2022a, ) and OpenSeeD (Zhang et al., 2023e, ) exploit a DETR-
like architecture and predict the masks and categories based on the outputs of
their decoders.

###### Model pre-training.

There are mainly three learning methods for pre-training open-set vision
models.

  * •

Supervised learning. By converting label supervision to language supervision,
many works directly leverage the existing supervised annotations for training
open-set models. For example, OVR-CNN (Zareian et al.,, 2021) trains a model
with COCO categories and then evaluates its performance on novel categories.
Likewise, ViLD (Gu et al.,, 2021) trains and evaluates two separate models on
COCO and LVIS datasets, respectively. Following a similar protocol, many works
train the open-set segmentation models on a subset of annotated segmentation
data and evaluate the generalization ability on held-out data (Ding et al.,
2022a, ; Ding et al., 2022b, ; Zhang et al., 2023e, ; Xu et al., 2023a, ).

  * •

Semi-supervised learning. One might use both annotated data and unlabeled or
weakly-labeled data. For example, both RegionCLIP (Zhong et al., 2022b, ) and
GLIP (Li et al., 2022f, ) use a teacher model to extract fine-grained region-
text alignments from image-text pairs to augment the training data for better
open-set detection performance. Differently, OpenSeg (Ghiasi et al., 2022b, )
exploits Localized Narrative datasets (Pont-Tuset et al.,, 2020) as weakly-
labeled data, which provides coarse correspondence between language phrases
and strokes in images. Empirically, such semi-supervised learning methods
often help improve models’ generalization ability because they can effectively
leverage rich semantics from noisy data.

  * •

Weakly-supervised learning. Some works solely use weakly-labeled data for
modeling. For example, GroupViT (Xu et al., 2022a, ) uses a contrastive
learning method where all supervisions for model training are from positive
and negative image-text pairs. Following the same contrastive learning method,
SegCLIP (Luo et al., 2023b, ) uses a gathering mechanism to learn to merge
image patches through the training on image-text pairs.

Below, we review recent models developed for region-level and pixel-level
tasks.

(a) DetCLIPv2

(b) Grounding-DINO

Figure 4.3: (a) DetCLIPv2 (Yao et al.,, 2023) learns fine-grained word-region
alignment from object detection and grounding data and large-scale image-text
pairs. (b) Grounding-DINO (Liu et al., 2023h, ) injects text conditions into
different stages of the Transformer encoder-decoder, which significantly
improves the grounding performance. Image credit: Yao et al., (2023) and Liu
et al., 2023h .

####  4.2.1 Object Detection and Grounding

Object detection is a fundamental task in computer vision that involves
identifying and localizing objects of interest within an image or a video
sequence (Viola and Jones,, 2001). Over the years, various techniques and
algorithms have been developed to improve the accuracy and efficiency of
object detection. In the past, region-based approaches such as R-CNN Girshick
et al., (2015), Fast R-CNN (Girshick,, 2015) and Faster R-CNN (Ren et al.,,
2015) have been fostering the development of advanced techniques for object
detection. To improve real-time performance, YOLO (Redmon et al.,, 2016)
proposes a single neural network that simultaneously predicts object classes
and bounding box coordinates. Some improvements are made by either using
multiple feature maps at different scales (Liu et al.,, 2016) or introducing a
focal loss to address the class imbalance problem in dense object detection
scenarios (Lin et al.,, 2017). After the emergence of Transformer (Vaswani et
al.,, 2017), DETR (Carion et al.,, 2020) applies the transformer architecture
to object detection, treating it as a set prediction problem. Since DETR, a
number of methods have been proposed to improve transformer-based detection
models from various aspects, such as DINO (Zhang et al., 2022a, ), Group DETR
(Chen et al., 2022b, ), and Co-DETR (Zong et al.,, 2023).

Open-set object detection models aim to detect arbitrary concepts beyond the
vocabulary provided in training data. Three main evaluation settings have been
developed in the literature:

  * •

Zero-shot object detection. Similar to zero-shot image classification (Xian et
al.,, 2018), zero-shot object detection restricts the object classes used for
training, and evaluates models’ transferrability to novel classes. Methods
falling in this category mainly focus on evaluating how a model leverages pre-
trained concept embeddings (e.g., word2vec (Mikolov et al.,, 2013)) and learns
good visual-semantic alignments (Bansal et al.,, 2018; Rahman et al.,, 2020;
Zhu et al.,, 2019, 2020).

  * •

Strict open-vocabulary object detection. First introduced in OV-RCNN (Zareian
et al.,, 2021), this setting differs from zero-shot object detection in that
there is no limit on the training vocabulary as long as it does not cover any
target classes. Under this protocol, some representative works are ViLD (Gu et
al.,, 2021), RegionCLIP (Zhong et al., 2022a, ) which leverage large-scale
language-image models (Radford et al.,, 2021; Jia et al.,, 2021), and Detic
(Zhou et al., 2022b, ) that learns from image-label data.

  * •

Generalized open-vocabulary object detection. Some recent works like GLIP (Li
et al., 2022f, ), and OWL-VIT (Minderer et al.,, 2022) advocate a more
flexible setting to evaluate the dataset or task transferrability for object
detection models. This setting allows vocabulary overlap between training and
test sets, e.g., Objects365 for training while COCO for evaluation. This is
arguably a more practical setting than the two settings described above in
that models can be trained using any arbitrary set of training data and their
detection performance evaluated in the wild (Li et al., 2022b, ).

Object grounding can be considered as a generalized open-set object detection
task (Plummer et al.,, 2015; Kazemzadeh et al.,, 2014; Chen et al.,, 2019;
Deng et al.,, 2018). In this task, models take a sentence and an image as
input and localize objects that are associated with the noun phrases.
Recently, M-DETR (Kamath et al.,, 2021) employs a transformer-based
architecture to build an end-to-end modulated detector to detect objects in an
image given a raw text query. Unlike previous works where models are trained
on specific datasets, the network is pre-trained with 1.3M pairs of text and
images, sourced from multi-modal datasets where the connections between text
phrases and corresponding image objects are labeled. Inspired by M-DETR, GLIP
(Li et al., 2022f, ) casts object detection as a grounding problem, and
jointly learns a model using object detection and grounding data for open-set
scenarios. Following this line of research, DetCLIPv2 (Yao et al.,, 2023)
proposes a simple joint learning method where multiple tasks are converted
into a word-region alignment task, and then a model is trained end-to-end on a
corpus consisting of object detection data, grounding data and image-text
pairs. Grounding-DINO (Liu et al., 2023h, ) is a state-of-the-art grounded
object detection method, where the object detector is composed of components:
a backbone, a neck, and a head, and inject language conditions at every stage.
A combined text and image backbone is employed to extract features at multiple
scales, which are then passed on to the neck. The text and image
characteristics generated by the neck are subsequently used for language-
driven query selection. Grounding-SAM is developed by combining Grounding-DINO
with SAM (Kirillov et al.,, 2023). As shown in Figure 4.4, an image and a
group of concepts are first fed into Grounding-DINO to produce the boxes, and
then the boxes are used as prompts for SAM to predict masks for each box.

Figure 4.4: Grounding-SAM consisting of Grounding-DINO (Liu et al., 2023h, )
and SAM (Kirillov et al.,, 2023). Image credit: Liu et al., 2023h .

####  4.2.2 Image Segmentation and Referring

Image segmentation is a long-standing and challenging vision problem. There
are mainly three subtasks, including semantic (Long et al.,, 2015), instance
(Hafiz and Bhat,, 2020), and panoptic (Kirillov et al.,, 2019) segmentation.
Semantic segmentation cares about the per-pixel semantic within an image (Long
et al.,, 2015; Chen et al.,, 2017; Chen et al., 2022j, ), whereas instance
segmentation groups pixels of the same semantic meaning into objects. Models
for both tasks have evolved from CNN-based architectures (Long et al.,, 2015)
to transformer-based ones (Chen et al., 2022j, ), and from two-stage models
(He et al.,, 2017) and one-stage models (Bolya et al.,, 2019; Tian et al.,
2020b, ) to the recent query-based approaches (Dong et al.,, 2021; Zou et
al.,, 2022). With the capability of per-pixel and instance-level
understanding, a natural step was taken to formulate panoptic segmentation
(Kirillov et al.,, 2019; Wang et al., 2021a, ; Cheng et al.,, 2022). Most
recently, Mask2Former (Cheng et al.,, 2022) proposed to address all three
tasks with a unified encoder-decoder architecture. Nevertheless, all these
works cope with a limited number of categories. In the following, we will
review the most recent works on open-set image segmentation and referring
segmentation.

Figure 4.5: OpenSeeD (Zhang et al., 2023e, ) leverages both mask and box
supervision for learning a universal open-vocabulary image segmentation model.
Image credit: Zhang et al., 2023e .

###### Open-Vocabulary Segmentation.

Recently, a number of methods have been proposed to transfer or distill the
rich visual-semantic knowledge from foundation models (Radford et al.,, 2021;
Jia et al.,, 2021) to specific segmentation tasks. Prominent examples include
LSeg (Li et al., 2022a, ), OpenSeg (Ghiasi et al., 2022a, ), and Huynh et al.,
(2022). Instead of using existing models, GroupViT Xu et al., 2022a  performs
language-image pre-training from scratch with a bottom-up grouping ViT
(Dosovitskiy et al.,, 2021), while DenseCLIP (Rao et al.,, 2022) demonstrates
the superiority of foundation models in finetuning settings compared with
supervised models. Recently, MaskCLIP (Ding et al., 2022b, ) is proposed to
tackle open-vocabulary panoptic and semantic segmentation simultaneously by
leveraging CLIP, and achieves impressive performance on ADE20K (Zhou et al.,,
2017) and PASCAL (Mottaghi et al.,, 2014; Everingham and Winn,, 2011). Instead
of using the ViT backbone, a recent work called FC-CLIP (Yu et al., 2023a, )
exploits a convolutional CLIP backbone (i.e., ConvNeXt trained by OpenCLIP
(Ilharco et al.,, 2021)) as both a feature extractor and a vision encoder.
Based on a simplified pipeline, FC-CLIP shows plausible efficiency and lefts
the state of the art on various open-vocabulary segmentation benchmarks.
Rather than only using CLIP, a recent work ODISE (Xu et al., 2023a, )
leverages text-to-image diffusion models, and shows that the latent features
in the pre-trained UNet can provide useful compact segmentation information
for open-vocabulary segmentation.

A big challenge in open-vocabulary segmentation is the lack of segmentation
data annotated with semantic labels. Thus far, most of the works are still
using COCO segmentation annotations. A few recent works attempt to leverage
object detection data as the extra supervision to augment the training of
segmentation models, such as OpenSeeD (Zhang et al., 2023e, ) (shown in Figure
4.5) and DataSeg (Gu et al.,, 2023). In addition to these new modeling
techniques, new datasets have been developed to mitigate this problem,
including curating multi-domain segmentation datasets (Lambert et al.,, 2020),
collecting high-quality annotations (Lu et al., 2023c, ) or scaling up to
billions of masks (Kirillov et al.,, 2023).

Referring Segmentation by design is open-vocabulary. Models are usually
designed specifically to learn from target datasets using various multimodal
fusion strategies (Hu et al.,, 2016; Liu et al.,, 2017; Margffoy-Tuay et al.,,
2018; Ye et al., 2019a, ; Yu et al.,, 2016; Wu et al., 2022a, ). CLIPSeg
(Lüddecke and Ecker,, 2022) extends a textual query to a visual query and
shows superior performance not only on referring segmentation but also on
semantic segmentation. Since the emergence of vision transformers, works like
LAVT (Yang et al., 2022e, ) enhance the cross-modal interactions from the very
beginning, which leads to a decent performance on RefCOCO (Yu et al.,, 2016),
RefCOCO+ (Yu et al.,, 2016) and G-Ref (Mao et al.,, 2016; Nagaraja et al.,,
2016). Differently, PolyFormer (Liu et al., 2023e, ) converts masks into
polygons and asks the transformer decoder to decode a sequence of polygon
coordinates. Inspired by Pix2Seq (Chen et al., 2022c, ), a similar method in
object detection, PolyFormer presents an alternative way to represent masks
for state-of-the-art referring segmentation. As we discussed earlier, one can
also compose Grounding DINO (Liu et al., 2023h, ) with SAM (Kirillov et al.,,
2023) for referring segmentation.

###### Unified Segmentation.

Given the above methods for open-vocabulary and referring segmentation, an
open question is how to unify all segmentation tasks in a single framework.
Recently, X-Decoder (Zou et al., 2023a, ) uses a generalized encoder-decoder
architecture to unify all these segmentation tasks. The referring segmentation
task is reformulated as a conditioned panoptic segmentation that takes some
textual phrases as input to the decoder. UNINEXT (Yan et al.,, 2023) is
another work that attempts to unify all instance-level segmentation in images
and videos. Different from X-Decoder, UNINEXT uses early fusion to fuse the
various prompts and vision features, which are then fed to the transformer
encoder-decoder.

###  4.3 From Task-Specific Models to Generic Models

(a)

(b)

Figure 4.6: (a) CV task landscape: CV tasks can span different axes, including
modality, space and time, which renders significant challenges to unify all of
them in a single model. Image credit: Yuan et al., (2021). (b) The data scale
pyramid: In particular, datasets in different tasks usually contain different
types of supervision. Image-level datasets like ImageNet (Deng et al.,, 2009)
and LAION Schuhmann et al., (2021) have annotations that have rich semantics
coverage but are coarse-grained, while pixel-level datasets like COCO panoptic
segmentation (Chen et al.,, 2015) provides fine-grained annotations but with
limited concepts.

Above we have discussed the recent efforts of transforming closed-set models
to open-set ones for detection and segmentation. Until recently, however, most
vision tasks have been separately tackled with specialized model designs,
preventing the synergy of tasks across different granularities or domains from
being exploited. This is arguably due to two reasons:

  * •

Vision tasks are fragmented. As shown in Figure 4.6 (a), computer vision tasks
span across different axes including space, time, and modality. From the space
aspect, it can be image-level, region-level and pixel-level tasks as we
discussed before. Along the time axis, we need to tackle not only static
images but also temporal video sequences. Regarding the modality, the inputs
and outputs can be images, texts, or other types (e.g., human pose, depth
map). Such diverse task formats significantly impede the development of a
unified model for all tasks.

  * •

Data scales are different. In addition to the complicated task landscape, the
scarcity of human annotations and their different scales for different tasks
also make building a unified model challenging. In Figure 4.6 (b), we can see
a clear pyramid of data scale, where different layers of human annotations
have different semantics. More specifically, image-text datasets like LAION
Schuhmann et al., (2021) contain up to 2B samples, while object detection
datasets like Objects365 (Shao et al.,, 2019) have 1.7M images in total. More
significant gap is observed in segmentation datasets due to the high cost of
annotating masks.

Despite the aforementioned challenges, we are now witnessing a growing
interest in building unified, general-purpose models that can learn from and
be applied to a diverse set of vision and vision-language tasks, thanks to the
versatility of transformers (Vaswani et al.,, 2017). These attempts can be
grouped into two main categories:

  * •

I/O Unification. Following the development of unified LLMs, a number of recent
works reformulate many vision tasks as a sequence-to-sequence problem (Wang et
al., 2022b, ; Yang et al., 2022c, ; Chen et al., 2022d, ; Lu et al., 2022a, ).
They typically use a tokenizer to tokenize the original inputs and outputs
(I/O) in different modalities used in various tasks into a coherent sequence
(visual or text) tokens and then exploit a unified, sequence-to-sequence
model.

  * •

Functionality Unification. In addition to I/O unification, one might built a
generic model via functionality unification. Extending multi-task learning
methods (Lu et al.,, 2020; Gupta et al., 2022a, ; Hu and Singh, 2021a, ), many
recent use a coherent encoder-decoder architectures (Yu et al., 2022a, ; Zhang
et al., 2022b, ; Zou et al., 2023a, ). This line of work usually does not need
task-specific or modality-specific tokenizers but requires a sophisticated
model design to accommodate various tasks.

Figure 4.7 illustrates the difference between the two categories of
unification methods. For I/O unification, the I/O unification module always
generates a sequence of tokens, and exploits a separate decoder to decode the
final outputs for different tasks. For functionality unification, the
functional unification module generates heterogeneous outputs for different
task, _e.g._ , semantic outputs and spatial outputs. Then, these different
types of outputs are combined to produce the final task-specific outputs. Both
unification methods strive to make use of synergy across tasks with different
levels of granularity. For example, coarse-grained data is expected to
contribute to rich semantic understanding required by fine-grained tasks,
while fine-trained data to enhance the grounding ability for coarse-grained
tasks. In the following, we review some recent works of these two categories.

Figure 4.7: Side-by-side comparison between I/O unification and functionality
unification. I/O unification is aimed at utilizing a decoder to decode
homogeneous token sequences, which are then decoded by task-specific decoders.
In contrast, functionality unification predicts heterogeneous outputs and then
uses different routers or headers to produce the final outputs for various
tasks.

####  4.3.1 I/O Unification

This line of work is mainly inspired by LLMs that unify many NLP tasks as
sequential modeling. In the vision domain. the methods of building generic
models via I/O unification can be grouped into two categories depending on the
tasks of interest and output formats.

##### Sparse and discrete outputs

For vision tasks that produce sparse or discrete token outputs, we can easily
exploit a language tokenizer, such as byte-pair encoding (BPE) (Sennrich et
al.,, 2016), for I/O unification. In contrast, spatial outputs like boxes,
masks, or human skeletons can be formulated as a sequence of numeric
coordinates which are then tokenized into discrete tokens (Cho et al.,, 2021;
Yang et al., 2022c, ; Liu et al., 2023e, ). As a result, the decoded output
tokens are interleaved with organic textual tokens and numeric textual tokens
to support a wide range of tasks. Without the loss of generality, the decoding
process is formulated as auto-regressive generation and the model trained with
the objective function defined as:

$L(\theta)=-\sum_{t=1}^{T}\log p(s_{t}|s_{<t},v;\theta),$ (4.3)

where $\\{s\\}_{t=1}^{T}$ is the discrete token sequence of length $T$, and
$v$ is the visual feature. Below, we review some representative works.

(a) UniTab

(b) Pix2Seqv2

Figure 4.8: (a) UniTab (Yang et al., 2022c, ) is proposed to unify grounded
captioning, visual grounding, image captioning, VQA, and object localization.
(b) Pix2Seqv2 (Chen et al., 2022d, ) is proposed to unify object detection,
referring segmentation, keypoint detection, and image captioning. Image
credit: Yang et al., 2022c  and Chen et al., 2022d .

UniTab (Yang et al., 2022c, ) unifies text and box output in a sequence
decoding manner. As shown in Figure 4.8 (a), the box coordinates are
represented by numerical numbers with $<>$ and then a special token $<$obj$>$
is used to encompass the location information. In this way, the model can
unify a variety of tasks that require textual and location outputs, including
image captioning (Chen et al.,, 2015), grounded captioning (Plummer et al.,,
2015), visual grounding, object localization and visual question answering
(Antol et al.,, 2015). The model is trained in three stages: pre-training,
multi-task finetuning, and task-specific finetuning.

Pix2SeqV2 (Chen et al., 2022d, ) slightly differs from UniTab in that it
unifies two different vision tasks: referring segmentation and keypoint
detection. Following Pix2Seq (Chen et al., 2022c, ), Pix2SeqV2 represents
objects in an image as $[y_{min},x_{min},y_{max},x_{max},\textit{text}]$.
Then, it introduces a unique task prompt for each task, which contains task
type information or a combination of task types and specific locations. For
mask decoding, a mask contour is converted into a polygon and then its
coordinates extracted from the polygon (Castrejon et al.,, 2017). A similar
strategy is also used for referring segmentation, as in Polyformer (Liu et
al., 2023e, ).

Figure 4.9: VisionLLM (Wang et al., 2023h, ) is proposed to bridge vision
systems with LLMs in a sequential decoding manner. Image credit: Wang et al.,
2023h .

###### LLM-augmented.

Recent works have also explored building a generic decoding interface based on
LLMs, which are pre-trained on large amounts of text data and human
instructions. Kosmos-2 (Peng et al., 2023b, ) exploits the pretrained LLMs of
Kosmos-1 (Huang et al., 2023b, ) and augments the grounded multi-modal data by
collecting a web-scale grounded image-text pair dataset (GRIT) consisting of
91M images. VisionLLM (Wang et al., 2023h, ) appends an even larger LLM (e.g.,
LLaMa (Touvron et al.,, 2023)) on top of an image tokenizer, as shown in
Figure 4.9. The resultant model exhibits a very strong vision-language
reasoning capacity and decent localization ability for object detection,
segmentation, etc. Some other works that combine LLMs with grounding are
DetGPT (Pi et al.,, 2023) and GPT4ROI (Zhang et al., 2023k, ). To further
equip the model with the segmentation capability, both BubaGPT (Zhao et al.,
2023c, ) and LISA (Lai et al.,, 2023) use an extra referring segmentation
model to segment images by taking texts or embeddings as input, respectively.
PaLI-X (Chen et al., 2023g, ) is by far the largest unified model that can
cope with multilingual vision and vision-language tasks.

##### Dense and continuous outputs

(a) VQ-VAE

(b) VQ-GAN

Figure 4.10: Illustration of VQ-VAE (Oord et al.,, 2017) and VQ-GAN (Esser et
al.,, 2021).

There are also some tasks that require dense and continuous outputs, such as
image segmentation (He et al.,, 2017), depth estimation (Mertan et al.,,
2022), image inpainting and editing (Elharrouss et al.,, 2020; Brooks et al.,,
2023). Except for segmentation masks which can be approximated by polygons
(Liu et al., 2023e, ; Chen et al., 2022d, ), most dense and continuous outputs
cannot be easily converted into discrete tokens due to the high-dimensional
space. Thus, we have to resort to an image-oriented tokenizer. Akin to the
language tokenizer, an image tokenizer encodes raw images and extracts
discrete tokens spanning the visual feature space. The most representative
work is VQ-VAE (Oord et al.,, 2017; Razavi et al.,, 2019). As shown in Figure
4.10 (a), VQ-VAE learns an encoder $z_{e}$, a decoder $z_{q}$ and a discrete
codebook $\textbf{e}=\\{e_{1},...,e_{K}\\}$ consisting of $K$ embeddings.
Given the input $x$, the posterior categorical probability $q(z|x)$ is defined
as:

$q(z=k|x)=\begin{cases}1,&\text{for
$k=\operatorname{argmin}_{i}\|z_{e}(x)-e_{i}\|$}.\\\
0,&\text{otherwise}.\end{cases}$ (4.4)

where the decoder $z_{q}$ takes $x$ (or its representation $e_{k}$) as input
to predict class label. As a variant of VQ-VAE, VQ-GAN uses a discriminator
and the perceptual loss (Larsen et al.,, 2016; Lamb et al.,, 2016) to maintain
a good balance between output quality and model efficiency (via high
compression rate). In Figure 4.10 (b), we see that the discriminator is
applied at the patch level to regularize the decoding of images at high
resolution. Below, we discuss some most recent works that attempt to unify
different vision and multi-modal tasks that involve dense outputs.

UViM (Kolesnikov et al.,, 2022) is one of the first works that employ a dense
decoding process to unify various core vision tasks, including panoptic
segmentation, depth estimation and colorization. The learning process consists
of two stages: $(i)$ Base encoder-decoder $f$ and restricted oracle $\Omega$
are learned to predict outputs given input images, where $f$ takes raw image
as input and $\Omega$ takes the desired output as input to decode the oracle
code; $(ii)$ Instead of using the desired output as input to the oracle
$\Omega$, the model learns a language model to produce the oracle code for the
input raw image. Notably, the encoder-decoder model used here is trained with
VQ-VAE objectives. As the first step to unify vision tasks with a single
model, UViM shows promising results on three vision tasks.

Unified-IO (Lu et al., 2022a, ) is another representative work. Compared to
UVIM, it scales to many more vision tasks and datasets. Unlike the training
procedure of UViM, Unified-IO first trains different VQ-VAE models for
different tasks, as depicted in Figure 4.11 left. After obtaining all VQ-VAE
encoder-decoders, 90 datasets are combined to train another transformer
encoder-decoder end-to-end, as shown on the right side. Similar to previous
works, it also uses a language decoder to obtain the organic and numeric texts
to generate coordinate outputs. After the second-stage pre-training, the model
achieves state of the art on the GRIT benchmark (Gupta et al., 2022c, ) and
exhibits compelling compositionality, although the performance still lags
behind the strongest models on common tasks. As a follow-up, a soft-token
strategy is proposed in Ning et al., (2023) to improve the accuracy for next
token decoding. In addition, a masked modeling strategy is proposed to learn
robust representations. Evaluated on instance segmentation and depth
estimation, the model achieves state-of-the-art performance on NYUv2
(Silberman et al.,, 2012) and competitive performance on segmentation. A
recent work uses image inpainting as the general task to unify different
pixel-level vision tasks (Bar et al.,, 2022). Given the target discrete tokens
produced by VQ-GAN, the method exploits a masked autoencoder to decode the
missed image regions, using the task input-output examples as prompts. Painter
(Wang et al., 2023i, ) extends this pipeline to facilitate more vision tasks
and obtains competitive performance on various standard benchmarks.

Figure 4.11: Unified-IO (Lu et al., 2022a, ) unifies different vision tasks by
first pre-training VQ-VAE for each task and then an encoder-decoder for tasks
jointly. Image credit: Lu et al., 2022a .

###### Diffusion-augmented.

Unlike the above works that learn their own decoding models, some recent works
utilize the off-the-shelf stable diffusion model to build generalist vision
models. For example, Prompt Diffusion (Wang et al., 2023m, ) initializes a
model using Stable Diffusion and ControlNet (Zhang and Agrawala,, 2023), and
trains the in-context image-to-image model jointly on six different vision-
language tasks, including segmentation, depth estimation, etc.
InstructDiffusion Geng et al., (2023) also uses the diffusion model but
explicitly introduces task-specific instructions to the diffusion process.
Moreover, it uses task-specific training and human alignment training to
enable a generalist interface for vision tasks.

####  4.3.2 Functionality Unification

Unlike I/O unification, functionality unification attempts to unify different
tasks based on the task characteristics, with the awareness that they are
neither fully isolated nor fully aligned. At a high level, vision tasks
produce three types of outputs: $(i)$ location outputs, $(ii)$ semantic
outputs, and $(iii)$ pixel-level outputs. For example, both object detection
and phrase grounding need to localize objects in the image, while both generic
segmentation and referring segmentation produce masks. On the other hand, many
tasks require semantic (or text) outputs to represent either concept names or
textual descriptions.

##### Multi-task learning

(a) UberNet

(b) Taskonomy

Figure 4.12: (a) UberNet (Kokkinos,, 2017) can be applied to 7 vision tasks
using a unified and budget-controllable CNN architecture. (b) Taskonomy (Zamir
et al.,, 2018) further studies the relationship across vision tasks by
exploiting a multi-task transfer modeling. Image credit: Kokkinos, (2017) and
Zamir et al., (2018).

Some early works explore multi-task learning methods for unifying different
vision or vision-language tasks.

###### Vision models.

A few works explore using CNNs for learning with different vision tasks at
different levels. For example, Cross-stitch Networks (Misra et al.,, 2016)
develops a strategy to split different numbers of layers from the top in CNNs
so as to adapt to different vision tasks. Results show that the best-
performing multi-task architecture depends on the tasks of interest and can
hardly generalize to new tasks. UberNet (Kokkinos,, 2017) takes one step
further to use a single universal CNN architecture and sophisticatedly design
a routing mechanism to save the memory and computing cost, as shown in Figure
4.12 (a). Both works require some tweaking to the CNN architecture so that
they can adapt to different levels of tasks and loss types. But they
unfortunately fail to build the synergy across tasks to improve model
performance. Taskonomy (Zamir et al.,, 2018) specifically studies the
relationship among vision tasks. It first trains task-specific models for each
individual task and then performs transfer modeling across tasks in the latent
space. The task affinity is then calculated in the latent space, providing us
with the taskonomy. The result shows that vision tasks have different
affinities for different groups, as shown in Figure 4.12 (b). For example,
surface normal estimation is heavily related to reshaping and point matching.
Curvature extraction is related to image segmentation tasks. This study
provides deep insights for multi-task vision modeling (Xu et al.,, 2018;
Crawshaw,, 2020).

###### Multi-modal models.

The emergence of Transformers significantly facilitates the advancement of
multi-task multi-modal learning. Among them, 12in1 (Lu et al.,, 2020) is one
of the pioneering works that combine 12 vision-language tasks in a single
BERT-based architecture. It uses task-specific heads for individual tasks and
a commonly shared trunk ViLBERT (Lu et al.,, 2019). Results show that multi-
task learning can achieve substantial improvements over single-task learning
while reducing the model parameters significantly. Later on, UniT (Hu and
Singh, 2021b, ) exploits an encoder-decoder architecture and expands to
vision-only tasks like object detection. Additionally, it allows end-to-end
training on the task pool without relying on pre-trained detectors. Similar to
12in1, it also uses a task-specific head for each task, motivated by the
empirical result that sharing the same head usually hurts performance.
Likewise, E2E-VLP (Xu et al.,, 2021) proposes an end-to-end pipeline for both
localization tasks and text generation. Both UniT and E2E-VLP demonstrate the
versatility of the encoder-decoder architecture of DETR (Carion et al.,,
2020). Following the same spirit, GPV (Gupta et al., 2022b, ) proposes an end-
to-end task-agnostic architecture for different vision and vision-language
tasks. It uses DETR to extract boxes and region features and then exploits a
cross-attention module for fusion, followed by a vision decoder and a language
decoder for decoding different outputs.

The above vision and multi-modal models unify different tasks by incorporating
different modules or heads designed to cope with different tasks, and can
hardly achieve synergy across tasks. In the following, we discuss recent model
unification research that aims to make the best use of synergy among various
vision and multi-modal tasks.

Figure 4.13: A generalist model X-Decoder (Zou et al., 2023a, ) that unifies
different vision and vision-language tasks in a functional manner. It uses a
single decoder with the same suite of parameters, but different routing
mechanisms to tackle different tasks. Image credit: Zou et al., 2023a .

##### Unified learning

The barrier across tasks is gradually blurred thanks to the use of
Transformers (Vaswani et al.,, 2017) and the development of open-set models as
we discussed earlier. It is now possible to bind inputs from different
modalities to learn a shared semantic space. A number of works (Zhang et al.,
2022b, ; Zou et al., 2023a, ; Li et al., 2023g, ) have recently been proposed
to unify vision and vision-language tasks by _using one model for all_. After
pre-training, the single model can be applied to tackle all tasks in a zero-
shot manner and the performance can be further improved via task-specific
finetuning. Note that unified learning in this context differs from previous
works of large-scale pre-training. Like GPT which serves as a universal
language interface after pre-training, a unified vision model is not only a
representation learning engine but also an interface that supports as many
tasks as possible in a zero-shot manner. Below, we review a few representative
works.

GLIPv2 (Zhang et al., 2022b, ) is proposed by extending GLIP (Li et al.,
2022f, ) to support a wide range of vision and vision-language tasks,
including grounded captioning, visual question asnwering, etc. GLIPv2
seamlessly integrates localization pre-training and Vision-Language Pre-
training (VLP) through three distinct pre-training tasks: $(i)$ phrase
grounding, which serves as a vision-language adaptation of detection tasks;
$(ii)$ region-word contrastive learning, introducing a novel contrastive task
at the region-word level; and $(iii)$ masked language modeling. In a zero-shot
manner, this pre-trained model can be applied to different tasks and attain
plausible performance across the board. Unlike previous works (_e.g._ , GPV
(Gupta et al., 2022b, )), it merges the localization module and vision-
language matching module in a coherent manner, which makes model training from
fused data much more efficient and effective.

X-Decoder (Zou et al., 2023a, ) follows the generic design of encoder-decoder
architecture. Given an input image, it first uses an image encoder to extract
features at multiple scales. Afterward, a text encoder is used to encode a
textual query into a sequence of embeddings. The visual features, textual
queries and the non-semantic or latent queries are fed to a decoder to predict
the outputs. Three critical designs are proposed to empower the generalization
ability of X-Decoder to a variety of vision and vision-language tasks: $(i)$
It defines two types of queries and outputs. Specifically, the queries for the
decoder are categorized into latent queries and text queries, which undertake
generic vision and vision-language tasks, respectively. Likewise, the output
is categorized into pixel-level masks and semantic embeddings; $(ii)$ A single
text encoder is exploited to encode the textual corpus from all tasks. The
common text encoder is used to encode referring phrases, text descriptions,
and image captions in the task of referring segmentation, image-text retrieval
and image captioning, respectively; $(iii)$ It fully decouples the image and
text encoder, and use all the outputs as queries. As such, it can learn from
both intra-image supervisions and inter-image ones, which is essential to
learn stronger pixel-level representations and support different granularity
of tasks. As shown in Figure 4.13, the pre-trained model can support different
tasks by taking different routing while sharing the same suite of parameters.

Uni-Perceiver-v2 (Li et al., 2023g, ) is another generalist model that unifies
vision and vision-language tasks. Similar to X-Decoder, the model exploits a
vision encoder, a text encoder and a general decoder. Differently, it
introduces a region proposal network on top of the vision backbone to
explicitly predict the boxes and masks, which are then encoded as “queries”
for the general decoder. To jointly train on datasets at different levels, it
introduces a unified max-likelihood estimation strategy for tasks with
localization and without localization.

###  4.4 From Static to Promptable Models

The success of Large Language Models (LLMs) such as ChatGPT (OpenAI, 2023b, )
have shown the importance of modern AI models in interacting with humans, and
have provided a glimpse of AGI (Bubeck et al.,, 2023). The ability to interact
with humans requires a user-friendly interface that can take as many types of
human inputs as possible and generate responses that humans can easily
understand. In NLP, such a universal interaction interface has emerged and
evolved for a while from early models like GPT (Brown et al.,, 2020) and T5
(Raffel et al.,, 2020), to more advanced techniques like prompting (Shin et
al.,, 2020; Zhao et al.,, 2021; Li and Liang,, 2021) and chain-of-thought (Wei
et al., 2022a, ; Kojima et al.,, 2022; Schick et al.,, 2023). However, most
vision models are still static in that they are less flexible than LLMs to
various prompts. Most recently, a number of works have proposed to enhance the
static vision models with the capabilities to support: $(i)$ multi-modal
prompting; $(ii)$ in-context prompting.

####  4.4.1 Multi-modal Prompting

Vision is different from language by nature. To enable a smooth interaction
between humans and AI, a model requires not only language prompts but also
other types of prompts to complement the missing information or resolve the
ambiguity in language. Recently, a number of works have explored how to
combine or augment language prompts with other types of prompts, such as
spatial prompts (Kirillov et al.,, 2023), visual prompts (Zou et al., 2023b, )
and other modalities (Girdhar et al.,, 2023; Liu et al., 2023f, ). In the
following, we review some representative works.

###### Spatial prompting.

Vision is rooted in the physical world, and as such it is not only semantic
but also spatial by nature. Spatial prompting can be considered as a way to
modulate the vision models through the inputs of location information, which
could be a point, a box, or an arbitrary stroke, etc. Such clues have been
heavily used in UI designs of computers (e.g., mouse) and mobile devices
(e.g., touch screen). In computer vision, interactive segmentation (Mortensen
and Barrett,, 1998; McGuinness and O’connor,, 2010; Chen et al., 2021c, ; Chen
et al., 2022i, ) naturally requires such capability so that the model can take
multiple clicks from users and gradually refine the segmentation mask.
However, most of these works are still designed task-specifically and lack
enough flexibility to support different types of spatial prompts.

SAM (Kirillov et al.,, 2023) is one of the pioneering works that propose a
convenient spatial prompting interface and learn a foundation model for image
segmentation. As shown in Figure 4.14, the model can take points or boxes as
the prompts, and segment images in arbitrary granularity. The ability to
segment images following the user instructions from humans makes it readily a
foundation to build many more models and applications (Zhang et al., 2023c, ).
To name a few, a number of works (Ma and Wang,, 2023; Roy et al.,, 2023) start
with SAM and train a promptable segmentation model for the medical domain.
Spatial prompting is particularly beneficial in that the textual annotations
for medical images are usually limited and hard to interpret. Similar cases
also happen in other industry domains (Tang et al., 2023a, ). To further
improve point prompting, SAMAug (Dai et al., 2023a, ) proposes to refine the
points using the max entropy criterion and saliency map, which can help to
determine the most informative locations the model should look at.

Figure 4.14: SAM (Kirillov et al.,, 2023) proposes a promptable segmentation
model which can take different spatial prompts in addition to text prompts. It
further develops a data annotation engine to scale up the mask-annotated data.
Image credit: Kirillov et al., (2023).  Figure 4.15:  SEEM (Zou et al., 2023b,
) can take different types of prompts as inputs for various image segmentation
tasks. Image credit: Zou et al., 2023b .

###### Visual prompting.

In many cases, textual descriptions of objects are not necessarily clear to
convey the information. For example, given an unrecognizable or indescribable
object, people may fail to express themselves clearly about the object. In
this case, showing one or a few examples would be more informative and
straightforward. With this idea, a lineup of works have studied exemplar-based
visual modeling, such as image-to-image retrieval (Yoon et al.,, 2021; Datta
et al.,, 2008; Zhang et al.,, 2018), image co-segmentation (Joulin et al.,,
2010; Jerripothula et al.,, 2016) and visual object tracking (Yilmaz et al.,,
2006; Luo et al.,, 2021; Wu et al.,, 2013). Most recently, this strategy has
been formulated as visual prompting in that different types of visual inputs
are usually encoded to some unified format and then fed into a Transformer
architecture, as shown in LLMs.

SEEM (Zou et al., 2023b, ) is one of the representative works that enable
visual prompting to a vision model for image segmentation. As shown in Figure
4.15, SEEM differs from the aforementioned SAM and can take visual prompts by
drawing points, boxes, and strokes on an image that can be the target image or
another reference image. It develops a new module called a visual sampler that
can extract visual features from an image according to the locations specified
by users. Based on the visual sampler, the model can even take another
reference image as input without any training like that. As a result, it shows
impressive performance not only for various image segmentation tasks but also
for video object segmentation in a zero-shot manner.

PerSAM (Zhang et al., 2023h, ) develops a personalized segmentation model on
top of SAM and takes one shot as the input. It learns a specific model that
takes a source image plus a mask as input and then predicts the mask for a
target image. To extract the visual prompts, mask pooling is taken and used as
the input tokens to the decoder of PerSAM. It also proposes a way to extract
the positive and negative priors based on feature matching to facilitate pre-
trained SAM models with comprehensive clues. Like most prompt learning methods
in LLMs, a plausible feature for PerSAM is that it can be easily attained by
some off-the-shelf models like SAM. SAM-PT (Rajič et al.,, 2023) further
applies this strategy to video object segmentation. Inspired by the spatial
prompting in SAM, it exploits a point-tracking system (Harley et al.,, 2022)
to track different points (both positive and negative ones) and then ask SAM
to segment the image given the points. It exhibits strong point tracking
performance as well as segmentation.

###### Others.

Some other works combine a wide range of visual prompting types. For example,
Painter (Wang et al., 2023i, ) reformulates different vision tasks (_e.g._ ,
depth estimation, image segmentation) all as prompting and learns a decoder in
an in-context learning manner. The prompts are combinations of raw images and
the corresponding dense annotations (_e.g._ , depth or segmentation maps). In
contrast, Prismer (Liu et al., 2023f, ) makes use of many off-the-shelf vision
models to extract different information from the raw images and then feed the
information to a vision-language model. To facilitate the interplay across
multiple modalities, ImageBind (Girdhar et al.,, 2023) learns a universal
alignment among image/video, language, audio and depth. Once the embedding
space is learned, it can be used to compose different types of prompts by
simply doing the summations.

####  4.4.2 In-context Prompting

The in-context learning capability has been observed in many LLMs such as
GPT-3 (Radford et al.,, 2019), which makes the model more configurable via
prompting without any model parameter updates. In contrast, till now, the in-
context learning capability for vision models is still less studied. Flamingo
(Alayrac et al.,, 2022) is one of the pioneering works that demonstrate in-
context language generation for multi-modal inputs, which is acquired by
learning from interleaved image-text pair data. Likewise, Kosmos-1 (Huang et
al., 2023b, ) is another work that takes visual inputs as a foreign language
so that the in-context learning ability in LLMs can be naturally translated to
multi-modal inputs. However, both methods take multi-modal data as inputs but
merely generate texts as outputs. As we discussed earlier, vision tasks
require outputs of different types beyond texts. How to endow the in-context
learning ability for vision systems is still an open question. Below, we
review some recent attempts toward the goal.

Figure 4.16: Visual prompting via visual inpainting proposed in Bar et al.,
(2022). Image credit: Bar et al., (2022).

Visual prompting via inpainting is proposed in Bar et al., (2022) to teach the
model to predict dense outputs, such as edges, masks, depths, etc. as shown in
Figure 4.16. Given an input image $\textbf{x}\in\mathcal{R}^{H\times W\times
3}$ and a binary mask $\textbf{m}\in\\{0,1\\}^{H\times W}$, an inpainting
model is to predict the missing region $\textbf{y}=f(\textbf{x},\textbf{m})$.
The authors exploit a pre-trained VQ-GAN to encode the original image into
discrete tokens and ask another ViT encoder to predict the masked regions. To
make sure the model understands the visual “context” in the images, the
authors collected a new dataset called Computer Vision Figures dataset which
consists of 88k images from Arxiv papers. After pre-training, the model is
used to predict the content at the bottom-right corner.

Figure 4.17: SegGPT (Wang et al., 2023j, ) proposes to perform in-context
learning for image segmentation. Image credit: Wang et al., 2023j . Figure
4.18: Hummingbird (Balažević et al.,, 2023) is proposed for in-context visual
scene understanding through nearest-neighbor retrieval. Image credit:
Balažević et al., (2023).

Concurrently, Painter (Wang et al., 2023i, ) extends a similar idea of visual
in-context learning to more diverse datasets and benchmarks. Unlike Bar et
al., (2022), it predicts the output in the continuous pixel space instead of
discrete tokens. For different tasks, the authors define rules to convert the
output spaces into image spaces. For example, it uses different colors to
represent different individual instances in the image for the segmentation
task. After unifying the input and output format, the authors use vanilla ViT
as the encoder and masked image modeling (He et al., 2022a, ). A follow-up
work called SegGPT (Wang et al., 2023j, ) is built on top of Painter and
designed specifically for image segmentation tasks. The pre-trained model can
be easily extended for exemplar-based image segmentation tasks.

Hummingbird (Balažević et al.,, 2023) resorts to a different method for in-
context visual learning. Instead of using masked modeling, the authors propose
to leverage attention across target and source images to aggregate the
information. As shown in Figure 4.18, the models take multiple input images
(first row) and corresponding semantic label maps (second row). Given a query
image, it first finds the nearest neighbor feature locations in the prompt
images for the query points and then projects the same matches to the semantic
label maps so as to aggregate the label for the target query. This strategy is
akin to earlier works that build classification models based on K-nearest-
neighbor but differently applied to dense prediction tasks.

###### Discussion.

In-context learning is arguably an appealing feature. On one hand, there are a
number of works that attempt to bridge vision with LLM so as to inherit the
in-context learning capability such as Flamingo (Alayrac et al.,, 2022) and
Kosmos-1 (Huang et al., 2023b, ). On the other hand, researchers resort to
pure vision-based in-context learning to address vision-specific tasks such as
image segmentation, depth estimation, etc. Thus far, there is no single model
that can take multi-modal inputs and predict different types of outputs as
well in an in-context learning manner, which may render a promising future
direction along this line.

###  4.5 Summary and Discussion

Figure 4.19: A summary of topics covered in this chapter. A lot of effort has
been spent to unify vision models from different aspects to enable more
intelligent vision systems.

To the end, an illustrative summary of the works that have been covered in
this chapter is shown in Figure 4.19. There is a clear trend in the vision
community to build _open-world, unified and interactive_ vision models.
Nevertheless, there are still some intrinsic differences between vision and
language. First, vision differs from language in that it captures the physical
world with raw signals. We need to develop some sophisticated tokenization
methods to compress the raw data into compact “tokens”. In the language
domain, this can be easily done by using some well-established heuristic
tokenizers (Sennrich et al.,, 2016). Second, unlike language, vision data
itself is not labeled and thus difficult to convey information or knowledge.
It always requires human labors to annotate the visual contents in either a
semantic or spatial manner. Third, language data is homogeneous while vision
data and tasks are heterogeneous. Last but not least, storing vision data is
much more costly than language data. For example, GPT-3 consumes 45 TB of
training data, while the ImageNet dataset which contains 1.3M images costs
more than hundreds of gigabytes. When it comes to video data like Howto100M
(Miech et al.,, 2019), the storage cost already exceeds that of training
corpus for GPT-3. All these differences cast some open questions that need to
be addressed in the vision community, detailed below.

  * •

Computer vision in the wild. Due to the heterogeneous nature, the current
vision data we use for training models can hardly cover the full picture of
the physical world. Despite the effort in building open-set vision models, we
are still facing significant challenges in coping with novel or long-tail
scenarios.

  * •

Scaling law in vision. As discussed in Kaplan et al., (2020); Hoffmann et al.,
(2022), the performance of large language models improves smoothly with the
increase of model size, data scale, and amount of computes. As the scale
increases, some intriguing emerging properties are further observed in LLMs.
In contrast, it is still not clear what is the right path to scale vision
models, not to mention the emerging properties in such models.

  * •

Vision-centric or language-centric models. Currently, the boundary between
vision and language is gradually dismissed. However, due to intrinsic
differences between vision and language, it is still not clear whether we
should further scale up the vision models and integrate language models or the
combination of moderate vision models and LLMs is sufficient to address most
(if not all) of the problems.

With that being said, we are close yet still far away from an intelligent
vision system that can perceive the world like humans. We hope the literature
review in this chapter could provide an overall picture of the existing
efforts, and inspire the pursuit of next-generation vision models.

##  Chapter 5 Large Multimodal Models:
Training with LLM

In this chapter, we comprehensively explore large multimodal models (Alayrac
et al.,, 2022; OpenAI, 2023a, ). We begin with Section 5.1 to delve into the
background of such models, with the focus on the basics of image-to-text
generative models and their representative model instances in various case
studies. We also discuss the state-of-the-art OpenAI Multimodal GPT-4 (OpenAI,
2023a, ) and identify the existing research gaps in the field. To better
understand the process of instruction tuning in large language models, Section
5.2 examines its importance and its role in self-instruct and open-source
LLMs. Moving forward, we explore instruction-tuned large multimodal models in
Section 5.3, shedding light on their basics, significance and applications.
Additionally, Section 5.4 touches upon advanced topics in the realm of
multimodal models to provide a deeper understanding of the subject. Finally,
we assess the current progress in the field by evaluating how close we are to
achieving the OpenAI Multimodal GPT-4 in Section 5.5, a major milestone in AI
research.

###  5.1 Background

####  5.1.1 Image-to-Text Generative Models

LMMs in their current form is primarily an image-to-text generative model,
which takes images as input, and outputs a text sequence. One example is
illustrated in Figure 5.1 (a) Left. All of the model variants share a very
similar model architecture and training objective.

  * •

Model Architecture. As illustrated in Figure 5.1 (a) Right, the model
typically consists of an image encoder to extract visual features, and a
language model to decode the text sequence. The vision and language modalities
can be optionally connected by trainable connection module. The image encoder
and language model can be either trained from scratch or initialized from pre-
trained models.

  * •

Training Objective. As illustrated in Figure 5.1 (b), it typically employs an
auto-regressive loss on the output text tokens. For the attention map in the
Transformers (Vaswani et al.,, 2017), image tokens can attend to each other,
and the current text token attends to all image tokens and the previous text
tokens.

---
(a) Left: An example of image-to-text generation task; Right: Model
architecture.
(b) Training objective and attention mask. For each row, the yellow elements
indicate that the prediction token attends the tokens on the left.
Figure 5.1: Illustration of image-to-text generation task, architecture, and
training objective.

####  5.1.2 Case Studies

We use some prominent LMMs as examples to illustrate how the aforementioned
network architecture can be instantiated in different models, while
maintaining the same auto-regressive training objective.

###### Case study I: LMM trained with image-text pairwise instances.

Most LMMs are trained on a large number of image-text pairs, where each
training sample is a pair. GIT (Wang et al., 2022a, ) and BLIP2 (Li et al.,
2023h, ) are two large models that achieve state-of-the-art (SoTA) performance
on many datasets. The comparisons are shown in Figure 5.2(a). GIT initializes
the image encoder with contrastively pre-trained Florence model (Yuan et al.,,
2021), and trains the language model from scratch. On the other hand, BLIP2
freezes the weights of a pre-trained image encoder and a pre-trained language
model, while trains a lightweight Q-former module to connect the image encoder
and the language model.

###### Case study II: LMM trained with interleaved image-text sequence
instances.

We use Flamingo (Alayrac et al.,, 2022) as an example, shown in Figure 5.2(b).
It connects the frozen pre-trained image encoder and language model – by
adding novel architectural components in between. Specifically, Perceiver
Sampler module helps reduce computational complexity, and Gated Transformer
module helps to stabilize training during the initial stage. Flamingo is
trained on a mixture of complementary large-scale multimodal data coming only
from the web, without using any data annotated for machine learning purposes.
After this training is done, Flamingo can be directly adapted to vision tasks
via simple few-shot learning without any additional task-specific tuning.

---
(a) Example 1: LMM trained with image-text pairs.
(b) Example 2: LMM trained with image-text pairs and interleaved image-text
data.
Figure 5.2: Examples of image-to-text generation models. Image credit: Wang et
al., 2022a ; Li et al., 2023h ; Alayrac et al., (2022).
Figure 5.3: The emerging property of pre-training on web-scale interleaved
image-text data: multimodal in-context-learning. Examples are adopted from
Alayrac et al., (2022).

###### Multimodal in-context-learning.

Beside the SoTA performance on dozens of academic benchmarks, probably the
most appealing aspect of Flamingo is the emerging property: Multimodal In-
Context-Learning. Specifically, given a couple of image-text pairs as
examples, Flamingo can zero-shot task transfer to unseen problems, such as
solving visual math problems. This means Flamingo can tackle a number of
difficult problems with just a handful of task-specific examples, without any
additional training required. For example in Figure 5.3, two new tasks are
presented to Flamingo. The top row provides two image-text pairs as the
context in the prompt, where the text describes the name of the animal in the
image, followed by the geographical information of the animal. Flamingo is
able to understand the patterns presented in the examples, and output the
corresponding information for a new image. In the bottom row, the text first
shows the optical character recognition (OCR) result of the image, followed by
the answer to the math problem. Flamingo follows the task instruction
illustrated in the multimodal context, and outputs the correct answer for a
new math problem in the third image. This intriguing in-context learning
capability makes Flamingo the GPT-3 moment (Brown et al.,, 2020) in the
multimodal domain.

Figure 5.4: OpenAI Multimodal GPT-4. Visual examples are from OpenAI, 2023a .

####  5.1.3 OpenAI Multimodal GPT-4 and Research Gaps

In March 2023, OpenAI released GPT-4 (OpenAI, 2023a, ), with impressive
capability in visual understanding and reasoning. Though the model details are
not revealed, there is no doubt that GPT-4 enables many new scenarios, based
on the examples highlighted in the technique report. For instance, two popular
visual examples are illustrated in Figure 5.4. The first one identifies the
uncommon visual region and exhibits strong complex reasoning performance. The
second one recognizes text in the image and captures the mere across image-
text. For a while, the research community had no clue how this new ability is
achieved (probably because they are not tightened to any established academic
tasks/datasets), but all are determined that these are exciting results. It
naturally raises a question: how can we build Multimodal GPT-4 like models?

To answer it, we start to review the big models from OpenAI, by highlighting
the most appealing properties for each model in Figure 5.5. There are several
key observations: $(i)$ GPT-2 (Radford et al.,, 2019) is the auto-regressive
counterpart in the BERT era (Devlin et al.,, 2019) for the pre-train-then-
finetune paradigm. Compared with GPT-2, GPT-3 (Brown et al.,, 2020) is a 175B
model trained on web-scale text corpus, which exhibits two emerging properties
with a frozen model: in-context-learning (Brown et al.,, 2020) and chain-of-
thoughts (CoT) reasoning (Wei et al., 2022a, ). This means, without any
additional training, the model can tackle a wide range of new problems with
just a few task-specific examples and by properly prompting it step-by-step,
respectively. It further leads to the modeling paradigm from task-specific
finetuning to prompting frozen models, where the latter shows higher
generalizability and lower adaptation cost in task transfer. $(ii)$ ChatGPT
and InstructGPT (Ouyang et al.,, 2022) show the importance of instruction-
following and alignment with human intents for LLMs, by finetuning the base
language model GPT-3/GPT-3.5 on high quality instruction-following data, and
improving them with a reward model via reinforcement learning with human
feedback. $(iii)$ GPT-4 not only improves the language ability of previous
models, but also allows visual signals as additional input for visual
understanding and reasoning. We see that the newer generation model
maintains/improves the existing properties of the previous ones, and enable
new properties.

In other words, from GPT-3 to GPT-4, we see two new properties: instruction-
following and multimodal input. This reveals the gap between existing LMMs
(e.g., Flamingo) and multimodal GPT-4: how to perform instruction-following
and alignment research in the multimodal space, which is the focus of this
chapter.

###  5.2 Pre-requisite: Instruction Tuning in Large Language Models

Note that instruction-following is a notion originated in NLP. To study the
intuition behind it and have a full picture of its history, we first revisit
instruction tuning with LLMs.

Figure 5.5: Recap on Language Modeling: OpenAI LLM development history. The
unique properties for each generation model are highlighted, from which the
research gap is revealed for LMM.

###### Traditional language data.

As a typical data instance in NLP, sequence-to-sequence (seq2seq)
representation is widely adopted for many language tasks: each data instance
consists of two parts: one sequence as the input and another sequence as the
output. We provide two examples in Figure 5.6 (a). Without any task
instruction specified, we know they are translation and summarization tasks,
respectively.

This seq2seq representation is also the conventional data format in NLP
research, where task instructions are implicit. Based on each data domain,
individual models are trained. Or sometimes one model is trained with multi-
task objectives over multiple data domain without specifying the task
instructions. For both cases, the models are hard to generalize to new tasks
in a zero-shot fashion, as they are not trained to understand task
instructions, thus cannot distinguish and generalize what task to perform
during testing time.

###### Instructional language data.

Recently, researchers have started to explicitly add task instructions into
the model training, as shown in Figure 5.6 (b). Interestingly, the task
instruction of most NLP tasks can be expressed in natural language as well. It
leads a new data format: instruction-input-output triplets. Based on the new
format, one single model can be trained to perform multiple tasks, each with
its specific instructions. Since models have observed many task instructions
and many instances for each task during training, it is more natural and
easier for them to generalize to new tasks by task composition in the
inference stage.

For example, in the evaluation stage, a new task that requires both
summarization and translation is provided in Figure 5.6 (c). Though the model
has never seen this new task during training, it observes individual task
basis, and learns to perform on new tasks. Note that we humans are always
creating new tasks in our daily life, and presumably these new tasks would
never been observed by models. It is thus appealing if a model is able to
solve thousands of new tasks in the wild without training. This is partially
why ChatGPT is becoming popular and prevalent so quickly.

---
(a) Training: Implicit task instructions in traditional language data.
(b) Training: Explicit task instructions in instructional language data.
(c) Inference: Explicit task instructions in instructional language data. The
English meaning of the output is “CVPR’23 visa issue: organizing committee
works to solve and provide virtual and in-person registration switch
services.”
Figure 5.6: Examples of task instructions in traditional and instructional
language data, respectively.

####  5.2.1 Instruction Tuning

How can we collect a diverse set of high-quality instruction-following data?
There are two general schemes. One is through human-human interaction, where
humans (task providers) provide the annotation statement and requirements,
based on which another group of humans complete the annotation tasks. Such a
scheme is typically costly and time consuming. The other scheme is via human-
machine interaction, where similarly humans provide the annotation statement
and requirements, but it is now the machines/models that complete the
annotation tasks.

To enable LLMs to follow natural language instructions and complete real-world
tasks, researchers have been exploring methods to instruction-tune LLMs. This
is implemented by either finetuning the model on a wide range of tasks using
human-annotated prompts and feedback (Ouyang et al.,, 2022), or supervised
finetuning using public benchmarks and datasets augmented with manually or
automatically generated instructions (Wang et al., 2022f, ). Among these
methods, Self-instruct tuning (Wang et al., 2022e, ) is a simple and effective
method of aligning LLMs to human intent, by learning from instruction-
following data generated by SoTA LLMs. It turns out that the line of
instruction-tuning research has produced effective means to improve zero-shot
and few-shot generalization abilities of LLMs. Self-instruct leverages the in-
context-learning ability of LLM. The pipeline is illustrated in Figure 5.7.
Humans create a few examples (i.e. seed examples) as the context, and ask LLM
such as GPT-3 or GPT-4 to create more instructions and responses that follow
the requirements stated in the prompt. The machine-generated instruction-
following data can be further selected to construct with the prompt for in-
context-learning in the next data generation iteration. The procedure iterates
until a given number of samples are collected. Due to the relatively lower
cost and higher response speed of API calls (compared with human annotations),
self-instruct is becoming more favorable in the research community.

####  5.2.2 Self-Instruct and Open-Source LLMs

Figure 5.7: Illustration of the self-instruct pipeline (Wang et al., 2022e, ).

The open-source community has witnessed a surge of open LLMs. The success of
ChatGPT (OpenAI,, 2022) and GPT-4 (OpenAI, 2023a, ) offers tremendous
opportunities to improve open-source LLMs using instruction-tuning. Figure 5.8
compares several open-source instruction-tuned LLMs. LLaMA (Touvron et al.,,
2023) is a series of open-sourced LLMs, which match the performance of
proprietary LLMs such as GPT-3. To teach LLaMA to follow instructions, Self-
instruct tuning has been quickly adopted given its superior performance and
low cost. For example, to name a few early attempts in this line of research,
Stanford Alpaca (Taori et al.,, 2023) uses 52K instruction-following samples
generated by GPT-3.5, while Vicuna (Vicuna,, 2023) uses around 500K high-
quality instruction-following samples (150K conversions) between user and GPT
(ShareGPT,, 2023). To advance the SoTA of instruction-tuning for LLMs, Peng et
al., 2023a  uses GPT-4 as the teacher to generate the responses to the Alpaca
instructions. Many follow-up works (Zhang et al., 2023i, ) improve the
instruction-following data to enable the open LLMs with better alignment
quality in chat. For a comprehensive review, we refer the readers to a recent
paper (Wang et al., 2023k, ), where a LLM Tulu is trained on a mix of several
high-quality instruction data, and comprehensive comparisons are conducted
across multiple benchmarks.

Figure 5.8: Model examples of the LLaMA family.

###### Quick assessment of LLM chatbots.

To study the quality of LLM Chatbots, we consider Vicuna-
Instructions-80111https://github.com/lm-
sys/FastChat/blob/main/fastchat/eval/table/question.jsonl (Vicuna,, 2023), a
dataset with 80 questions that baseline models (Touvron et al.,, 2023) find
challenging. Besides generic instructions, the instructions fall into 8
categories, including knowledge, math, Fermi, counterfactual, roleplay,
generic, coding, writing and common-sense. To quantitatively compare the
performance, GPT-4 is used to rate the response from score 1 to 10 for any two
given chatbots, then compute the relative score. Surprisingly, it turns out
this evaluation metric is quite consistent across different settings. The
open-source LLaMA family seems to perform closely to SoTA proprietary
chatbots.

###### Further discussions.

There are several important topics on LLMs that we have not covered in this
chapter, but are worthwhile future exploring.

  * •

Data-centric AI. We emphasize that the development of these open-source LLMs
is data-centric (Mazumder et al.,, 2022), rather than model-centric, so that
we hope the readers could align with this perspective when discussing the
topic. As the training objectives and network architectures are becoming
similar or even identical to GPT-like models, the key differential factor is
data. For example, behaviors of the aforementioned LLMs are determined by the
instruction tuning data.

  * •

False promise? There is a debate on that the open LLMs could catch up with the
proprietary LLMs is a false promise (Gudibande et al.,, 2023). To align the
discussions, we argue that there are two distinctive abilities for LLMs: the
instruction-following ability to know which task to perform, and massive
knowledge storage to complete the task with high quality. Imitation models are
good at the former, by mimicking ChatGPT’s style but perform poorly in terms
of factuality in their responses. In Gudibande et al., (2023), the authors
conclude that there exists a substantial capabilities gap between open and
closed LLMs that, with current methods, can only be bridged using an unwieldy
amount of imitation data or by using more capable base LLMs. They also
advocate that the highest leverage action for improving open-source models is
to tackle the difficult challenge of developing better base LLMs. However,
unfortunately, the resources to train such base LLMs are only available in a
few industry labs. It seems more promising for most academic research labs to
explore the opportunities in alignment research with affordable resources, or
explore the techniques to reduce the compute barriers.

  * •

Base LLMs. Developing more capable or commercial usable LLMs is of great
value. Besides LLaMA, the open-source community has developed variants of base
LLMs such as LLaMA-2, OpenLLaMA (Geng and Liu,, 2023), MPT (Team,, 2023) and
Falcon (Penedo et al.,, 2023), or released the training recipe (Computer,,
2023).

###  5.3 Instruction-Tuned Large Multimodal Models

In this section, we illustrate how to build the minimum prototype of
multimodal GPT-4 with open-source resources. Specially, we use LLaVA (Liu et
al., 2023c, ) as the running example, a similar idea is also proposed in its
con-current work MiniGPT-4 (Zhu et al., 2023a, ).

The research in the multimodal space has often been inspired by the latest
advances in NLP in recent years. One successful recipe is to explore what
would happen if the most intriguing and successful NLP ideas are borrowed for
the vision-and-language community, for example, self-instruct. However, the
unique challenge with self-instruct in multimodal research is that there is no
strong multimodal teacher publicly available. Therefore, the research question
becomes: how can we use language models such as language-only GPT-4 to create
multimodal instruction following data.

---
(a) The sequence representation of the image data.
(b) Three types of instruction-following data for the given image.
Figure 5.9: Examples of multimodal instruction-following data. Image credit:
Liu et al., 2023c .

##### Data Creation

Instead of directly feeding images into OpenAI GPT-4, we use their symbolic
sequence representations shown in Figure 5.9 (a). In LLaVA, both captions and
bounding boxes are considered, due to the following reasons: $(i)$ it is
empirically found that GPT-4 can understand both well, in contrast to the poor
performance of ChatGPT in understanding bounding box coordinates. $(ii)$ They
are often complementary to each other and hence can represent the image as
informative as possible.

As shown in Figure 5.9 (b), three types of instruction-following data are
considered: $(i)$ multi-turn conversations so that users can chat with the
model; $(ii)$ detailed description so that long-form responses can be
generated from the model; and $(iii)$ complex reasoning, which is more about
the implication of the image, rather than the image content. For example,
“what challenge do these people face?”, which requires to first recognize that
the image is about a SUV in the parking area, and there are quite a few
luggage placed on the ground, and then to infer that the challenge is how the
luggage can be packed into the SUV due to the tight space of the trunk. In
total, 158K samples are collected over three types. To summarize, the spirit
is that whatever tasks one wants the model to perform in the serving stage, it
is important to create the corresponding instruction-following data for
training.

Figure 5.10: Network architecture: Left: General LMM; Right: LLaVA. Image
credit: Liu et al., 2023c .

##### Network Architecture and Training

As illustrated in Figure 5.10, the network architecture of LLaVA is an
instantiation of the general image-to-text generative model framework
introduced in Figure 5.1 of Section 5.1. Specifically, LLaVa connects the pre-
trained CLIP ViT-L/14 visual encoder (Radford et al.,, 2021) and large
language model Vicuna (Vicuna,, 2023), via a simple projection matrix (i.e.,
the linear projection layer). A two-stage instruction-tuning procedure is
adopted to train the model. $(i)$ Stage 1: pre-training for feature alignment.
Only the projection matrix is updated, based on a subset of CC3M (Changpinyo
et al.,, 2021). $(ii)$ Stage 2: finetuning end-to-end. Both the projection
matrix and LLM are updated on the proposed multimodal instruction-following
data for daily user-oriented applications.

##### Performance

###### Visual chat: towards building multimodal GPT-4 level chatbot.

LLaVA is finetuned on the generated multimodal instruction-following data,
which contains a diverse set of task instructions and responses for daily
user-oriented applications. It is empirically observed that finetuning the
linear projection layer only is sufficient for the chat demo/scenarios, though
it requires longer training time. To evaluate the model performance, an
evaluation dataset named LLaVA-Bench is constructed, with two subsets: $(i)$
LLaVA-Bench (COCO): 30 unseen COCO images with 90 new language-image
instructions, $(ii)$ LLaVA-Bench (In-the-Wild): 24 images with 60 questions.
Each image can be associated with three types of instructions: conversation,
detailed description and complex reasoning. The ground-truth answers are
collected by manually re-writing GPT-4 output. We test LLaVA and use language-
only GPT-4 to rate their responses from score 1 to 10. Overall, LLaVA achieves
85.1% relative score compared with ground-truth on LLaVA-Bench (COCO), and
73.5% on LLaVA-Bench (In-the-Wild). On the latter dataset, Google Bard (July
19, 2023) and Microsoft BingChat (June 29, 2023) achieves 77.8% and 71.5%,
respectively. It indicates the effectiveness of the proposed self-instruct
method in multimodal settings. One examples is shown in Table 5.1.

###### Science QA: New SoTA with the synergy of LLaVA with GPT-4.

LLaVA is finetuned on a multimodal reasoning dataset in the science domain (Lu
et al., 2022b, ). LLaVA alone achieves 90.92% in accuracy. We further explores
with language-only GPT-4 as the judge, to predict the final answer based on
its own previous answers and the LLaVA answers. This “GPT-4 as judge” scheme
yields a new SoTA of 92.53%.

###### OCR in the wild: An emerging property.

LLaVA has never been explicitly trained on OCR data, i.e. images that contains
scene text that is described in the corresponding caption. Surprisingly, the
model shows strong zero-shot OCR task transfer ability in the wild.

Visual input example, Chicken Nugget Map:
---
|
User |  Can you explain this meme in detail?
LLaVA |  The meme in the image is a creative and humorous take on food, with a focus on chicken nuggets, as the center of the universe. The meme begins with the phrase ”Sometimes I just look at pictures of the Earth from space and I marvel at how beautiful it all is…” The main image features a tray of cooked chicken nuggets, with the chicken pieces positioned to resemble the continents and islands on a world map. The meme playfully suggests that the chicken nuggets represent the Earth, and the various locations depicted in the photo are actually chicken nugget versions of different places. The punchline of the meme appears in smaller text at the bottom, reading ”I mean, it’s not the real Earth, but how beautiful it is all is.” This lighthearted and entertaining meme combines food and humor, using the image of chicken nuggets to create a funny and imaginative representation of the world.
User |  Can you explain this meme?
GPT-4 |  This meme is a joke that combines two unrelated things: pictures of the earth from space and chicken nuggets. The text of the meme suggests that the image below is a beautiful picture of the earth from space. However, the image is actually of chicken nuggets arranged to vaguely resemble a map of the world. The humor in this meme comes from the unexpected juxtaposition of the text and the image. The text sets up an expectation of a majestic image of the earth, but the image is actually something mundane and silly.
User |  Can you explain this meme in detail?
Bing Chat |  Sure, I can try to explain this meme in detail. This is a **chicken nugget meme**, which is a popular online meme that typically features a picture of a chicken nugget with a caption that is funny or relatable. The meme has been popular since the early 2000s, and continues to be one of the most popular memes today. This particular meme is based on a contrast between the text and the image. The text above the image reads ”Sometimes I just look at pictures of the earth from space and I marvel at how beautiful it all is.” This is a sentiment that many people can relate to, as looking at the earth from space can inspire awe and wonder. However, the image below the text shows a baking tray with chicken nuggets arranged in the shape of the world map. This is a humorous twist, as chicken nuggets are not usually associated with beauty or marvel, but rather with fast food or junk food. The image also implies that the person who made this meme loves chicken nuggets so much that they see them everywhere, even when looking at pictures of the earth from space. The meme is meant to be funny and absurd, and to poke fun at the person’s obsession with chicken nuggets.I hope this explanation helps you understand this meme better.
User |  Can you explain this meme in detail?
Bard |  Sure. The meme you sent me is a humorous take on the idea of looking at pictures of the earth from space and marveling at its beauty. The text in the meme says “Sometimes I just look at pictures of the earth from space and I marvel at how beautiful it all is.” However, the image shows a map of the world made out of chicken nuggets. This juxtaposition of the text and the image is what creates the humor in the meme. The text suggests that the speaker is looking at something beautiful and awe-inspiring, but the image shows something that is more mundane and humorous. The meme also plays on the fact that chicken nuggets are often seen as a junk food. By depicting a map of the world made out of chicken nuggets, the meme suggests that even something as mundane as chicken nuggets can be beautiful, or at least, more beautiful than we might think. Overall, the meme is a humorous way to poke fun at the idea of finding beauty in the mundane. It also shows how our perception of beauty can be influenced by our expectations. Here are some other interpretations of the meme: (1) The meme could be seen as a commentary on the way we consume images of the earth from space. We often see these images as representations of beauty and perfection, but the meme reminds us that the earth is also a messy and imperfect place. (2) The meme could also be seen as a critique of the fast food industry. The use of chicken nuggets to represent the world suggests that we are increasingly seeing the world through the lens of consumerism. Ultimately, the meaning of the meme is up to the interpretation of the viewer. However, it is clear that the meme is a humorous and thought-provoking way to explore the idea of beauty.
User |  Can you explain this meme in detail?
MiniGPT-4 |  This meme is a picture of a pan full of fried chicken. The chicken is cooked and appears to be ready to be served. The caption reads, “Sometimes it’s just like looking at pictures from space and marveling at how beautiful it all is.” This meme is likely meant to be humorous and playful, as it compares the beauty of space to the simple pleasure of eating fried chicken. It is likely meant to be a lighthearted and funny way to express appreciation for both the wonders of space and the simple pleasures of food.
User |  Can you explain this meme in detail?
BLIP-2 |  sometimes i just look at pictures of the earth from space and marvel how beautiful it is
User |  Can you explain this meme in detail?
OpenFlamingo |  It’s a picture of a chicken nugget on the International Space Station.

Table 5.1: Example prompt comparing LLaVA, GPT-4, BingChat, Bard, MiniGPT-4,
BLIP-2, and OpenFlamingo’s visual reasoning capabilities in understanding the
humor. LLaVA and GPT-4 both explain the meme and its humor, while GPT-4
produces a more concise answer. Table credit: Liu et al., 2023c .

###  5.4 Advanced Topics

---
(a) The surge of papers on LMMs from March 14, 2023 to June 19, 2023. Those
with an underline indicate benchmarks, otherwise indicate models.
(b) Summary and categorization of papers on LMMs.
Figure 5.11: Review and summary for the emerged LMM literature. Due to space
constraints, some methods are not displayed visually, but we aim to address
them in the accompanying text.

The history of recent instruction-tuned LMMs are illustrated in Figure 5.11
(a). Due to the popularity of ChatGPT and GPT-4, instruction-tuned LMM appears
as an emerging line of research in the past three months after GPT-4 was
proposed. Alpaca (Taori et al.,, 2023) and Vicuna (Vicuna,, 2023) were
proposed to make LLaMA more instruction-following in the language domain in
March. In two weeks, MiniGPT-4 (Zhu et al., 2023a, ) and LLaVA (Liu et al.,
2023c, ) were proposed to make Vicuna to see and chat about the visual world.
In ten days, LLaMA-Adapter v2 (Gao et al., 2023b, ) and mPlug-OWL (Ye et al.,
2023b, ) started to compare performance with MiniGPT-4/LLaVA, indicating the
beginning of model evolution. The data points in April are relatively sparse.
In May, a large number of LMM papers appeared on arXiv, which improve this
line of research from many different aspects. The momentum is till going in
June.

It is easy to lose track of all the recent papers for the readers, so as well
in our literature review. To better organize the literature, we group them
based on specific research topics, shown in Figure 5.11 (b). The early LMMs
with billions of parameters include GPT-4 (OpenAI, 2023a, ), Flamingo (Alayrac
et al.,, 2022), PaLM-E (Driess et al.,, 2023) and KOSMOS-1 (Huang et al.,
2023b, ). In contrast to these proprietary LMMs, LLaVA and MiniGPT-4 open the
opportunities to build LMMs with open-source resource. We will discuss several
topics as below, in addition to the extensions of RLHF (Gunjal et al.,, 2023),
dense prediction (Wang et al., 2023h, ; Zang et al.,, 2023; Chen et al.,
2023d, ), video (Zhang et al., 2023f, ; Luo et al., 2023c, ; Li et al., 2023i,
), image generation (Koh et al.,, 2023) and embodied agent (Mu et al.,, 2023).

##### More Modalities (Beyond VL)

While LMM extends LLM by adding the vision modality, it is natural to further
extend the framework to include more modalities beyond vision and language.
Following this spirit, several attempts have been made, including ChatBridge
(Zhao et al., 2023e, ), PandaGPT (Su et al.,, 2023), SpeechGPT (Zhang et al.,
2023d, ) and X-LLM (Chen et al., 2023c, ). PandaGPT leverages ImageBind to add
more modalities into LMMs. The ImageBind model (Girdhar et al.,, 2023) learns
a single, shared representation space for text, image/video, audio and sensors
that record depth (3D), thermal (infrared radiation), or inertial measurement
units (IMU), which calculate motion and position. ImageBind provides a
holistic understanding of the visual world that connects objects in a photo
with how they will sound, their 3D shape, how warm or cold they are, and how
they move. By training a projection layer for one modality in LMM, the model
can zero-shot transfer to infer over other modalities, thanks to the shared
multimodal embedding space. Another representative model is SpeechGPT, where
language and speech modalities are enabled for both inputs and outputs.
Despite of rich model variations, the idea to connect diverse modalities is
similar to LMM that adds images into LLMs. NExT-GPT (Wu et al., 2023c, )
connects an LLM with multimodal adaptors and different diffusion decoders,
enabling NExT-GPT to perceive inputs and generate outputs in arbitrary
combinations of text, images, videos, and audio. The LMM framework has also
been successfully extended to speech (Zhao et al., 2023c, ), 3D (Wang et al.,
2023l, ; Hong et al.,, 2023), and point cloud (Xu et al., 2023c, ).

##### Improving Visual Instruction Data Quantity and Quality

Given the convergence of model architectures to GPT-like network, the
performance of LMM is primarily determined by its training data. Therefore, it
is cricial to improve the quantity and quality of visual instruction tuning
data. SVIT (Zhao et al., 2023a, ) follows the same data generation pipeline as
in LLaVA, but further includes region description to prompt GPT-4, in addition
to the caption and box data as shown in Figure 5.9 (a). The data is scaled up
to 3.2 million, which is 20 times larger than the data used in LLaVA.

Unlike existing studies that primarily focus on positive instruction samples,
LRV-Instruction (Liu et al., 2023a, ) includes both positive and negative
instructions for more robust instruction-tuning. Other examples along this
line include LLaVAR (Zhang et al., 2023o, ) that adds OCR-related instruction-
tuning data for text-rich image understanding, and StableLLaVA (Li et al.,
2023o, ) that considers model-synthesized images for image-dialogue data.
Polite Flamingo (Chen et al., 2023b, ) trains LLM to re-write the instruct
data. Instead of leveraging GPT-4 for data generation, VIGC (Wang et al.,
2023a, ) considers to utilize LMM to generate instruction-tuning data and
progressively enhance its quality on-the-fly. Similar to the “less is more”
observation in LIMA (Zhou et al., 2023a, ) from the NLP domain,
InstructionGPT-4 shows that the quality of the instruction-tuning data is more
important than its quantity, where they finetune a better version of MiniGPT-4
with 200 high-quality samples (6%), selected from the 3500 samples used in the
original MiniGPT-4.

##### Multitask Instruct with Established Academic Datasets/Tasks

As discussed earlier in Section 5.2, instruction tuning in the language
domains is implemented in two different ways: finetuning the model on a wide
range of tasks using human-annotated prompts and feedback (Ouyang et al.,,
2022), or supervised finetuning using public benchmarks and datasets augmented
with manually or automatically generated instructions (Wang et al., 2022f, ).
The former is good at user-oriented daily life tasks, and the latter is good
at achieving decent performance on established benchmarks. LLaVA and MiniGPT-4
fall into the former class. Several other works either target for the latter
class or combine both classes, including MultiInstruct (Xu et al., 2022b, ),
mPlug-OWL (Ye et al., 2023b, ), InstructBLIP (Dai et al., 2023b, ),
Multimodal-GPT (Gong et al.,, 2023), Instruction-ViT (Xiao et al.,, 2023) and
Qwen-VL (Bai et al., 2023a, ).

For example, MultiInstruct is an early attempt before open-source LLaMA for
instruction tuning with multimodal datasets. InstructBLIP is a recent work
that combines chat and benchmark instruction-following data. As shown in
Figure 5.12, InstructBLIP transforms 26 publicly available datasets, covering
a wide variety of tasks and capabilities, into instruction tuning format.
Trained on 13 held-in datasets, InstructBLIP attains SoTA zero-shot
performance across all 13 held-out datasets, substantially outperforming
BLIP-2 and larger Flamingo models. Qwen-VL scales up both image-text pair data
for pre-traning and academic datasets for multi-task pre-traning, and achieve
excellent performance on many tasks.

Figure 5.12: The vision-language tasks covered in InstructBLIP. Image credit:
Dai et al., 2023b .

##### Multimodal In-Context-Learning

Similar to the behavior of LLMs, which can address a language task by
processing examples of the task in their text prompt, multimodal in-context-
learning refers to an visual and text interface that can steer the model
towards solving a multimodal task. Given a few example pairs of visual inputs
and expected text responses composed in the multimodal prompt, the model can
be queried with a question about a new image or video, and then generate an
answer. The direction to extend in-context-learning from language to multi-
modalities has been explored, including OpenFlamingo (Awadalla et al.,, 2023),
Otter (Li et al., 2023d, ), M3IT (Li et al., 2023j, ), MetaVL (Monajatipoor et
al.,, 2023) and Sparkles (Huang et al., 2023d, ).

OpenFlamingo (Awadalla et al.,, 2023) is an open source version of DeepMind’s
Flamingo model, trained on Multimodal C4 dataset (Zhu et al., 2023b, ), which
is a billions-scale corpus of interleaved image-text data. To explicitly
enhance the multimodal in-context-learning ability of LMMs, MIMIC-IT (Li et
al., 2023c, ) dataset is constructed, which is 2.4M multimodal instruction
instances with in-context examples. By tuning OpenFlamingo on MIMIC-IT, a new
model Otter is obtained with a stronger instruction-following ability. Using
two image-text pairs as the context, Otter learns the concise answer style
demonstrated by the examples, otherwise a tedious response is generated.

##### Parameter-Efficient Training

While finetuning very large models often leads to high performance, it is
prohibitively expensive; For example, regular 16-bit finetuning of a LLaMA-65B
model (Touvron et al.,, 2023) requires more than 780 GB of GPU memory
(Dettmers et al.,, 2023). Therefore, it is critical to reduce the memory
footprint of LLMs/LMMs, especially when it comes to improve the accessibility
of large models to a wider community.

Parameter-efficient training is an effective approach for LMM adaptation. It
freezes most of the model parameters, and only allows a fraction of trainable
parameters to update with domain-specific data. For example, LLaMA Adapter v2
(Gao et al., 2023b, ) and LAVIN (Luo et al., 2023a, ) only have 14M and 3.8M
trainable parameters, compared with 7B/13B LLM parameters, respectively.
Another efficient training method is quantization. The recent QLoRA (Dettmers
et al.,, 2023) finetunes 65B LLaMA for 24 hours on a single GPU, achieving
99.3% of the performance level of ChatGPT. Since instruction tuning typically
involves a small amount of data, it makes parameter-efficient training or
model quantization the practical approach, especially when with limited GPU
resources. Both LoRA (Hu et al.,, 2021) and QLoRA are supported in LLaVA
codebase to allow LMM training with less GPUs. It is empirically shown in Lu
et al., 2023d  that LoRA/QLoRA can achieve similar performance with full-modal
tuning when scaling LLaVA to 33B and 65B, when training with around 150K
instruct data and evaluating with LLaVA-Bench.

##### Benchmarks

While LMMs have shown excellent visual recognition and reasoning in an open-
set manner with free-form text across many scenarios, the evaluation of LMMs
is becoming an urgent and challenging problem. Several related benchmarks have
been developed to evaluate various aspects of LMMs, ranging from their
specific abilities including OCR (Liu et al., 2023k, ), hallucination (POPE
(Li et al., 2023l, ) and HaELM (Wang et al., 2023d, )) and adversarial
robustness (Zhao et al., 2023d, ), to comprehensive evaluation such as LAMM
(Yin et al.,, 2023), LVLM-eHub (Xu et al., 2023b, ). We summarize the LMM
evaluation benchmarks in Table 5.2. Among them, LLaVA-Bench is the first
attempt to designed open-world visual chat benchmark specifically for LMM.
Recently, early multimodal experiments have been conducted to compare open-
source LMM with commercial ones such as BingChat and Bard and LLaVA-Bench (Liu
et al., 2023c, ) and LVLM-eHub (Shao et al.,, 2023).

Benchmark |  Capability to Evaluate |  Statistics |  Metric
---|---|---|---
LLaVA-Bench (Liu et al., 2023c, ) |  Multi-turn QA, detailed description, reasoning |  Two subsets: 90 samples on COCO and 60 samples on In-the-Wid |  Relative score via GPT-4 evaluation
OCR-Bench (Liu et al., 2023k, ) |  Zero-shot OCR |  A suite of 23 OCR-related academic tasks |  Accuracy
MMBench (Liu et al., 2023j, ) |  Perception (coarse, fine-grained single-instance and cross-instance) and Reasoning (attribute, relation, logic) |  2974 multiple-choice samples in 20 ability dimensions |  Circular evaluation via ChatGPT answer extraction
M3Exam (Zhang et al., 2023l, ) |  Multilingual, multimodal, and multi-level assessment |  12,317 questions in 9 languages, with 2,816 questions involving images |  Accuracy on multiple-choice questions
MME (Fu et al.,, 2023) |  Perception and Cognition |  14 tasks |  Accuracy on “yes” or “no”
LAMM (Yin et al.,, 2023) |  Various 2D/3D vision tasks |  9 image task with 62K samples, and 3 point cloud tasks with 12K samples |  Traditional CV task metrics
LVLM-eHub (Xu et al., 2023b, ) |  six multimodal capabilities such as VQA and embodied AI |  47 standard text-related visual benchmarks |  CIDEr and accuracy; Arena with human judgment
SEED-Bench (Li et al., 2023b, ) |  Comprehension of both the image and video modality |  19K multiple choice questions in 12 dimensions |  Accuracy on multiple-choice questions
VisIT-Bench (Bitton et al.,, 2023) |  Real-life vision-language instructions |  592 samples in 70 tasks |  Elo, matches
MM-VET (Yu et al., 2023d, ) |  Integrated capabilities in recognition, OCR, spatial, knowledge, math, language |  200 samples |  GPT-4 evaluation
TouchStone (Bai et al., 2023b, ) |  Five abilities: basic description, visual recognition, visual comprehension, visual storytelling, and multi-image analysis |  908 dialogues in 27 tasks |  GPT-4 evaluation
SciGraphQA (Li and Tajbakhsh,, 2023) |  Scientific graph question-answering |  3K test samples |  CIDEr, BLEU-4, and ROUGE

Table 5.2: Comparisons of recently proposed LMM evaluation benchmarks.

It is surprising that LMMs shows strong zero-shot OCR performance in the wild,
without explicitly training on text recognition data. To shed light on the
hidden mystery of OCR in LMMs, a comprehensive empirical study is conducted in
Liu et al., 2023k  to compare open-source LMMs on 24 academic text recognition
datasets, shown in Figure 5.13. Three observations are highlighted: $(i)$
LLaVA consistently outperforms MiniGPT-4 on 21 out of 24 datasets, despite
that the training data in LLaVA is an order of magnitude smaller. $(ii)$
Training with significantly more training data leads to higher OCR
performance, as demonstrated by BLIP2 (Li et al., 2023h, ) and mPLUG-Owl.
$(iii)$ In most cases, supervised SoTA results significantly outperform zero-
shot LMM. However, it is worth noting that in the WordArt dataset (Xie et al.,
2022a, ), which primarily features challenging artistic text, BLIP2 surpasses
supervised SoTA. This reveals the potential of LMM in recognizing more complex
text types.

Figure 5.13: Zero-shot OCR performance of LMMs on 24 datasets. Image credit:
Liu et al., 2023k .

##### Applications

The success of ChatGPT/GPT-4 in the general domain has inspired the interests
in building assistants in the vertical domains such as medicine, gaming and
education. Such domain-specific assistants can have the several advantages
over the general domain counterpart: $(i)$ training with high-quality domain-
speicifc data makes the assistants more helpful; $(ii)$ the model size can be
smaller, with lower severing cost; and $(iii)$ the sensitive user prompt data
can be maintained internally by serving the model locally, to avoid privacy
issue.

To improve text recognition ability of LMM, OCR-specific models have been
developed, including BLIVA (Hu et al.,, 2023), LLaVAR (Zhang et al., 2023o, ),
mPlug-DocWL (Ye et al., 2023a, ). LMMs have been recently explored in the
biomedical domain (Sun et al., 2023c, ; Zhang et al., 2023m, ; Li et al.,
2023e, ), where conversational generative AI has demonstrated remarkable
promise for empowering biomedical practitioners. LLaVA-Med (Li et al., 2023e,
) is a cost-efficient approach for training a vision-language conversational
assistant that can answer open-ended research questions about biomedical
images. The key idea is to leverage a large-scale, broad-coverage biomedical
figure-caption dataset extracted from PubMed Central, use GPT-4 to self-
instruct open-ended instruction-following data from the captions, and then
finetune a large general-domain vision-language model LLaVA using a novel
curriculum learning method. Specifically, the model first learns to align
biomedical vocabulary using the image-caption pairs as is, then learns open-
ended conversational semantics using GPT-4 generated instruction-following
data, broadly mimicking how a layperson gradually acquires biomedical
knowledge. In Figure 5.14, we provide examples of the biomed visual
conversations with different chatbots. LLaVA-Med precisely answers the
questions requiring biomedical knowledge, while LLaVA behaves like a
layperson, that hallucinates based on commonsense. LLaVA-Med has inspired
several generalist biomedical AI models, including Google Med-PaLM-M (Tu et
al.,, 2023), Stanford Med-Flamingo (Moor et al.,, 2023) and radiology
generalist (Wu et al., 2023b, ).

Figure 5.14: Application of LMMs to biomedical images. Top: Domain adaptation
from LLaVA to LLaVA-Med. Bottom: The chat behaviors of two chatbots. Image
credit: Li et al., 2023e .

###  5.5 How Close We Are To OpenAI Multimodal GPT-4?

With all the works mentioned above, are we close to (or, even surpassing)
OpenAI Multimodal GPT-4? It is encouraging to see that the open-source
community has quickly developed a variety of models and prototypes for various
new capabilities. For example, LLaVA/Mini-GPT4 paves the way towards building
multimodal chatbots, with some examples that reproduce the results in OpenAI
GPT-4 technique report; CM3leon (Yu and et al,, 2023), Emu (Sun et al., 2023a,
), GILL (Koh et al.,, 2023) extends LMMs for end-to-end image generation, to
the best of our knowledge, this is a capability that the current GPT-4 does
not exhibit. From the perspective of enabling new capabilities with the
minimum prototypes, the open-source community seems close to OpenAI Multimodal
GPT-4, by exploring the baby steps towards building the general-purpose
multimodal assistant.

However, there is still a clear large gap in terms of scaling a given
capability, _e.g._ , for the visual reasoning capability that we have observed
in LLaVA. There are two more visual examples from OpenAI technical report, to
correctly answer the questions, it requires models to understand multiple
high-resolution images and long sequence text depicted in the image, as well
as responding with domain knowledge. It requires much more compute and more
powerful language models, which are not available to most people.

In summary, we have presented the background and strong capabilities of LMM,
reviewed instruction tuning in LLMs, and showed how to build a prototype such
as LLaVA and MiniGPT-4 using open-source resources. We also summarized the
most recent papers emerged on this line of research to help those who are
interested to gain the momentum to start the journey of LMM research. To
discuss the next steps to work on as a community, one sustainable suggestion
can be that those with resources can continue focusing on the scaling success
and study new emerging properties, while others focus on prototypes for new
functionalities and evaluation, as well as developing techniques to reduce the
computational barriers and thus allow easier accessibility to large models.

##  Chapter 6 Multimodal Agents:
Chaining Tools with LLM

Large Language Models (LLMs) (Chowdhery et al.,, 2022; OpenAI, 2023a, ) have
shown intriguing properties generalizing to user prompts in various domains,
and rapidly adapting to new scenarios, using in-context learning with a few
examples. Inspired by such strong capabilities, researchers are now exploring
a new modeling paradigm that shifts from standalone models for solving finite,
pre-defined problems, into synergistically chaining multiple tools or experts
with LLMs to solve complicated, open problems. Unlike what has been introduced
in Chapter 5, such a system can be built without any training involved, just
by using a few demonstration examples to teach the LLM to generate proper
calling to existing tools.

In this chapter, we review the fast-evolving literature on chaining different
multimodal experts with LLMs to solve complicated multimodal understanding
problems, referred to as multimodal agents. We start with an overview on the
evolution of this modeling paradigm in Section 6.1, highlighting the
differences between traditional approaches and the new modeling paradigm of
chaining tools with LLM. Section 6.2 gives a general overview of multimodal
agents. Pivoting on an exemplary multimodal agent MM-ReAct (Yang* et al.,,
2023), Section 6.3 comprehensively reviews how to build a multimodal agent,
its emerging capabilities in multimodal understanding, and how it can be
easily extended to incorporate the latest and strongest LLM and potentially
millions of tools. Finally, in Section 6.4, we end the chapter with
discussions on advanced topics, such as how to improve/evaluate multimodal
agents, the diverse applications powered by multimodal agents.

###  6.1 Overview

We first revisit the evolution of modeling paradigms, from task-specific
models to the most recent large multimodal models, which all require data
curation and model training. We then introduce the new modeling paradigm of
chaining tools with LLM, which may not require any training, but instead
directly takes advantage of a pre-trained LLM and existing tools that are
widely available through open-source platforms or APIs.

Figure 6.1: Evolution of modeling paradigm.

###### Evolution of modeling paradigm.

As summarized in Figure 6.1, we are witnessing the transition from task-
specific models towards general-purpose assistants across language, vision,
and multimodal research.

We started with task-specific models that are trained on small-scale well-
annotated data. This results in dedicated models (Anderson et al.,, 2018; Li
et al., 2019a, ; Yu et al.,, 2019) for each task or even each dataset.

We then transitioned to the phase of pre-trained models, with the pretrain-
then-finetune paradigm widely adopted across both NLP and vision-language (VL)
research. During pre-training, the model can take advantages of large-scale,
web-crawled noisy data, for example, millions to billions of image-text pairs
(Chen et al., 2020d, ; Wang et al., 2022a, ), or billions of text tokens
(Devlin et al.,, 2019; Liu et al.,, 2019). However, it is still mostly task-
specific finetuned, requiring similarly small-scale, well-annotated data as
the ones used in training task-specific models. This paradigm has led to many
well-known models, such as BERT (Devlin et al.,, 2019), RoBERTa (Liu et al.,,
2019) in NLP, and UNITER (Chen et al., 2020d, ), OSCAR (Li et al., 2020b, ) in
VL. These early VL foundation models were considered to be large-scale
(trained with 10M image-text pairs), but may be of intermediate or even small
size in today’s view (billions of pairs).

Nowadays, we are entering a new era of generalist modeling, where the pre-
training has been further scaled up to trillions of text tokens (Gao et al.,
2023b, ). For downstream adaptation, these generalist models have shown strong
performance with in-context few-shot learning on a few demonstration examples,
or even zero-shot evaluation. These models are what we now refer as large
language/multimodal models, including the GPT family (OpenAI,, 2022; OpenAI,
2023a, ), PaLM family (Chowdhery et al.,, 2022; Driess et al.,, 2023), LLaMa
(Touvron et al.,, 2023), Flamingo (Alayrac et al.,, 2022).

Based on the generalist models, the pipeline of building instruction-following
models covered in Chapter 5, similarly follows the pretrain-then-finetune
paradigm. For example, Alpaca (Taori et al.,, 2023), is built on top of the
pre-trained LLaMa (Touvron et al.,, 2023), then finetuned on a smaller-scale
instruction tuning dataset. Similarly, for instruction-following VL models
(e.g.  LLaVA (Li et al., 2023e, )), an additional stage of image-text
alignment pre-training is introduced to align the visual representations to
the frozen LLM first, followed by visual instruction tuning.

###### New modeling paradigm: chaining tools with LLM.

LLMs (Brown et al.,, 2020; Chowdhery et al.,, 2022; OpenAI, 2023a, ) have
demonstrated exceptional abilities to tackle new tasks with only a few
examples or textual instructions, showing the promise of serving as general-
purpose foundations for many applications. Despite being versatile and
impressive, they encounter challenges with the basic functionalities, such as
mathematical reasoning and information retrieval. Furthermore, a fundamental
limitation of not only LLMs but also other large-scale models nowadays, is
that they only represent the world described by their training data, which
will inevitably become outdated over time. Regularly re-training the model
with the latest information is simply not feasible.

Meanwhile, many tasks with real-world impact cannot be readily tackled by by
LLMs alone. For example, accessing up-to-date information and performing
computations, can be done via existing tools (e.g. , search engine or
calculator). Hence, recent research in language modeling has explored a new
modeling paradigm by supplementing LLMs with external NLP tools (Nakano et
al.,, 2021; Huang et al., 2022b, ; Ahn et al.,, 2022), including calculators,
search engines, translation systems, calendars, or even API calls on other
models.

The above studies mainly focus on a single modality, i.e., language, in which
the output of the tools are in text format, thereby can naturally be fed into
LLMs as additional knowledge. However, we live in a multimodal world and a
truly intelligent agent should be able to perform advanced multimodal
reasoning and actions. How to enable LLMs with perception of multimodal
signals via tool using, is the focus of the remaining part of this chapter.

###  6.2 Multimodal Agent

There are several representative works on building multimodal agent with tool
use of vision experts, including VISPROG (Gupta and Kembhavi, 2022b, ), Visual
ChatGPT (Wu et al., 2023a, ) and MM-ReAct (Yang* et al.,, 2023). VISPROG is
the very first work on using programming language to chain different vision
tools with a LLM. Visual ChatGPT enables dialogue-based image editing by
complementing ChatGPT (OpenAI,, 2022) with various image generation tools. MM-
ReAct shows that when collaborating various advanced vision experts, ChatGPT
can perform complex multimodal actions and reasoning. Figure 6.2 presents the
fast-evolving literature in multimodal agents from November 18, 2022 to July
26th, 2023. Among them, we include a few more exemplary multimodal agents in
Table 6.1, along with two representative works in the NLP domain.

Figure 6.2: The surge of multimodal agents from November 18, 2022 to July
26th, 2023.

Model | LLM | Tools | Tool | Tool | Multimodal
---|---|---|---|---|---
Size | Execution
ART (Paranjape et al.,, 2023) | GPT-3 |  | 3 | Program | ✗
Toolformer (Schick et al.,, 2023) | GPT-J |  | 5 | Natural language | ✗
VISPROG (Gupta and Kembhavi, 2022b, ) | GPT-3 |  |  $>$ 10 | Program | ✓
Visual ChatGPT (Wu et al., 2023a, ) | ChatGPT |  |  $>$ 10 | Natural Language | ✓
ViperGPT (Surís et al.,, 2023) | GPT-3 Codex |  |  $>$ 10 | Program | ✓
MM-ReAct (Yang* et al.,, 2023) | ChatGPT/GPT-4 |  |  $>$ 10 | RegExp Match | ✓
HuggingGPT (Shen et al., 2023b, ) | ChatGPT |  |  $>$ 10 | Natural Language | ✓
Chameleon (Lu et al., 2023b, ) | GPT-4 |  |  $>$ 10 | Natural Language | ✓

Table 6.1: Glossary of representative works on chaining tools with LLMs. The
LLMs used in these works include GPT-3 (Brown et al.,, 2020), GPT-J (Wang and
Komatsuzaki,, 2021), ChatGPT (OpenAI,, 2022), GPT-3 Codex (Chen et al., 2021a,
) and GPT-4 OpenAI, 2023a . : search engine. : code. : APIs other than search
engine. : open-source models.

An overview of a typical multimodal agent framework is illustrated in Figure
6.3. The user directly interacts with the Tool Allocator, which functions as
the brain of the agent. In current literature, the tool allocator is usually a
LLM. To achieve the user’s goal, the LLM will outline all the steps necessary
with either a single tool or collaborating multiple tools together.
Subsequently, it will retrieve from all the candidate tools for the needed
tools, and execute possibly multiple rounds of tools to fulfill the human
requirement. Finally, the execution results from the tools are gathered as
inputs of the LLM to generate a response to the user. Next, we cover the three
key components of multimodal agents.

Figure 6.3: An overview of multimodal agent framework.

###### Tools.

Tools are external modules that are callable by the LLM to obtain extra
information that is missing from the model weights, including open-source
models, public/private APIs, or code interpreters. As LLMs only accept
language inputs, one must include tools that can process multimodal inputs to
build a multimodal agent.

###### Planning.

During planning, the LLM decomposes the user requests into smaller, manageable
sub-problems, and outlines a step-by-step solution, each of which involves
calling an external tool. There are two ways to teach LLMs for planning. One
is to prompt the LLM with in-context few-shot examples of all candidate tools.
This approach can extend the general model directly but is limited by the
context length. The other approach relies on large amounts of annotated data
to fine-tune the LLM, which most likely will damage the robustness and
generalizability of the model.

###### Execution.

The generated plan is further translated into executable calls to the required
tools, which can be done via regular expression matching (Yang* et al.,,
2023); directly prompting LLMs to generate executable programs (Surís et al.,,
2023); or leveraging in-context few-shot learning capability of LLMs by
providing natural language instructions that describe the roles of each module
together with a few calling examples (Lu et al., 2023b, ). The execution
results are fed back to the LLM to generate a response to the user.

Figure 6.4: Input/Output modalities of MM-ReAct (Yang* et al.,, 2023).

###  6.3 Case Study: MM-ReAct

We use MM-ReAct (Yang* et al.,, 2023) as a case study to show how to build a
multimodal agent, its emerging capabilities in multimodal understanding, and
how it can be easily extended to incorporate the latest and strongest LLM and
potentially millions of tools.

####  6.3.1 System Design

MM-ReAct designs the system paradigm that composes numerous multimodal
tools111In Yang* et al., (2023), these tools are referred as experts. We unify
the terminology as tools throughout this chapter. with ChatGPT (OpenAI,, 2022)
for multimodal reasoning and action. By augmenting the language-only ChatGPT
with various multimodal tools, MM-ReAct supports both inputs and outputs in
multi-modalities, including text, image and video, as shown in Figure 6.4.

Figure 6.5: System design of MM-ReAct (Yang* et al.,, 2023).

Figure 6.5 shows the system design of MM-ReAct. The multimodal tools explored
in MM-ReAct are mainly computer vision models that take an image as input and
interpret the image content from different perspectives. For instance, the
image captioning model generates a natural description, the OCR model extracts
the scene text in the image, the celebrity recognition model identifies the