celebrity names, and the object detection model extracts the salient object
with bounding box locations. LLMs such as ChatGPT serves as the brain of the
agent, which plans on which tools to use, and in what order, based on the
input image and the user intent. Next, with the example in Figure 6.5, we
unfold the planning and execution of MM-ReAct behind the scene.

###### User prompt.

As ChatGPT only accepts language inputs, to enable image as inputs, we simply
use the file path as the input to ChatGPT. The file path functions as a
placeholder, allowing ChatGPT to treat it as a black box and later seek help
from different tools during the planning stage. Besides the input image, the
user can also provide the intent in text format (e.g. , a question about the
input image). When there is no text input from the user, the goal is to get a
general understanding about the image.

###### Planning.

Upon receiving the input image and user prompt, ChatGPT plans for what tools
to use. Inspired by ReAct (Yao et al., 2022c, ), MM-ReAct instructs ChatGPT to
respond with certain watchwords, such as “Assistant, what objects are there in
the image? $<$file path$>$”, if a specific tool is required (i.e., action
request in Figure 6.5). In practice, one can tell whether a multimodal tool is
needed by simply string-matching the keyword “Assistant,” in ChatGPT’s
response.

MM-ReAct encourages ChatGPT to show the thought (reasoning) process to
highlight why an external tool is needed, which has been proven beneficial in
NLP studies (Yao et al., 2022c, ). In addition, for generating proper calling
to each tool, both instructions and in-context examples are added as the
prefix when prompting ChatGPT. Each tool is described with the model name, a
general description of its capability, the input data format, and the output
information. After describing each tool, a few in-context dialogue examples
are also included for enhanced performance.

###### Execution.

Given the action request from ChatGPT, the tool name and the file path can be
parsed via regular expression matching, which are used to invoke the tool
(action execution).

Take the example shown in Figure 6.5, upon receiving the input image, ChatGPT
first invokes a series of tools for a general understanding about the image.
The invoked tools include image captioning for an overall description of the
image; dense captioning to get the region-level, more detailed description
about the objects in the image; object tagging to get the tags of the objects
in the image; face detection to get the box coordinates of the two faces
mentioned in the object tags. The outputs from the tools (i.e. observations)
are serialized as text, and fed back to ChatGPT.

Combining observations with the chat history, ChatGPT can further invoke
additional experts or return the final answer to the user. In this specific
example, ChatGPT invokes a second round of thought-action-observation over the
two faces detected in the image and calls celebrity recognition to get the
names of these two persons.

###### Response generation.

When ChatGPT decides no external tools are needed, it takes consideration of
all observations gathered and summarize them as the response to the user,
which is “This image contains two celebrities, Kobe Bryant and Paul Pierce.
They are both basketball players.” for the example shown in Figure 6.5.

If the user continues to interact with MM-ReAct, it repeats the process
described above, but with all observations and chat history available when
planning for the tools needed. For instance, if the user then asks “how many
championship rings did the player on the left win in his career”, it is not
available in the existing observations nor chat history, but ChatGPT has the
bounding boxes to decide who is on the left, and also the names of the
players. It plans to invoke Bing Search to find the right answer, which should
be 5.

Figure 6.6: Emerging capabilities of MM-ReAct for multimodal reasoning and
action. Image credit: Yang* et al., (2023).

####  6.3.2 Capabilities

Figure 6.6 shows the representative capabilities and application scenarios
that MM-ReAct demonstrates, including visual math and text reasoning,
understanding visual-conditioned jokes/memes, spatial/coordinate
understanding, visual planning and prediction, multi-image reasoning, multi-
hop document understanding, open-world concept understanding, video analysis
and summarization.

In addition, we show an example of the full response from MM-ReAct on multi-
image reasoning in Figure 6.7, which may not be easily achievable by visual
instruction tuning in Chapter 5. For more comprehensive examples of all
emerging capabilities of MM-REACT, we refer the reader to the original paper.

Figure 6.7: Examples of MM-ReAct’s capabilities and application scenarios:
multi-image reasoning. Image credit: Yang* et al., (2023). Figure 6.8:
Extensibility of multimodal agents: upgrading LLMs. Image credit: Yang* et
al., (2023).

####  6.3.3 Extensibility

One favorable property of tool chaining to build multimodal agents is that the
system can be easily extended and enhanced, from two perspectives. One is to
upgrade the core part of the system, the LLM, and the other is to expand the
number of external tools.

###### Upgrading LLM.

The system design of MM-ReAct allows for upgrading the core part of the
system, the LLM, to newer and more powerful models as they come out, without
the need of re-training. We show an example in Figure 6.8 on upgrading ChatGPT
to language-only GPT-4, which improves MM-ReAct to potentially match the
performance of multimodal GPT-4.

###### Plug-and-play (adding more tools).

Existing multimodal agents incorporates tools via a plug-and-play mechanism,
allowing adding more tools without training. One prominent work along this
direction is HuggingGPT (Shen et al., 2023b, ), which proposes to leverage all
open-source models hosted on huggingface. Chameleon (Lu et al., 2023b, ),
incorporates not only huggingface models, but also open-source models from
GitHub, Bing search API, and python compiler. RestGPT (Song et al.,, 2023)
proposes a multi-level online planning framework that effectively handles the
practical challenges associated with integrating LLMs with more than 100
RESTful APIs. However, it remains challenging in scaling this framework to
thousands to millions of tools, which is the potential future demonstrated in
TaskMatrix.AI (Liang et al., 2023b, ).

Moreover, one can leverage SAM (Kirillov et al.,, 2023) as a tool to allow for
more types of human interaction with the multimodal agent other than text.
Recall in MM-ReAct, the user intent is all captured by the natural language
query from the user. In InternGPT (Liu et al., 2023l, ), by connecting the
tool SAM with GPT, it allows for more ways to interact with the system, for
example, via clicks, scribbles, and drawing bounding boxes. These additional
interactions, to some extent, are mimicking the action of finger-pointing when
we humans are having a conversation.

###  6.4 Advanced Topics

In this section, we discuss more advanced topics and shed light on potential
future directions.

####  6.4.1 Comparison to Training with LLM in Chapter 5

We have covered two directions on building advanced multimodal systems based
on LLMs. As the key distinction, the multimodal agents in this chapter
leverages LLMs’ high-level planning abilities to allocate various multimodal
tools, while training multimodal models with LLMs in Chapter 5 solely
leverages LLMs for text generation conditioned on multimodal inputs.

Nonetheless, both of these methods exhibit their respective advantages and
disadvantages. On one hand, instruction tuning enables an end-to-end model
that directly interprets rich semantics in multimodal inputs, but requires
data curation and training, hence more computationally expensive. However,
limited instruction tuning data may limit its capabilities in certain
scenarios, such as OCR. On the other hand, one can build a multimodal agent
without any training by chaining LLMs with abundant off-the-shelf
models/APIs/code interpreters as tools, and leveraging in-context few-shot
examples to teach LLMs on planning. However, as there is no training, the
system may fail to invoke the right tool. Moreover, weak domain experts may
produce noisy outputs, that can confuse LLM on planning or reasoning, leading
to weak performance.

Though these two approaches exhibit distinct variations,, we envision the
possibility of an intermediate domain that amalgamates the strengths of both
paradigms, and raise the following questions. Now that we have open-source LMM
such as LLaVA (Liu et al., 2023c, ), can we replace the LLM with LLaVA as the
tool allocator? If so, what capabilities would require a tool to be enabled?
And what problems can be solved by instruction tuning. These are interesting
directions that may worth exploring in the near future.

####  6.4.2 Improving Multimodal Agents

Existing multimodal agents mainly rely on in-context few-shot examples to
teach LLM on planning, which can be unreliable, leading to inaccurate tool
using. To improve the accuracy in planning, several works have been proposed
and we group them into three categories below.

###### Composing tools via code generation.

Most existing multimodal agents uses natural language to prompt LLM for
planning which tool to use. Researchers (Gupta and Kembhavi,, 2023; Surís et
al.,, 2023) have also been exploring using programming language for more
accurate execution. Visual programming (Gupta and Kembhavi,, 2023) is a
prominent work along this direction, which uses the in-context learning
ability of GPT-3 (Brown et al.,, 2020) to generate python-like modular
programs from natural language instructions for compositional visual tasks
ViperGPT Surís et al., (2023) instructs GPT-3 Codex (Chen et al., 2021a, ) to
generate Python code to compose multimodal tools for a one-round query
answering. However, as the codes are still generated by a LLM, the problem of
inaccurate tool using still remains.

###### Improving accuracy in tool using: self-assessment.

A recent work AssistGPT (Gao et al., 2023a, ) tries to improve the accuracy in
tool using via self-assessment. It adds a stage of inspection and learning
loop into the system. When the round of plan and execution is finished, the
system inspects the outcome, and determines whether the reasoning path of
calling the tool is a success or not, if so, save it as an in-context example,
to teach LLM for a more accurate tool calling in the future rounds.

###### Improving accuracy in tool using: instruction tuning.

Another thread on improving accuracy in tool using is to combine the system
with instruction tuning (Patil et al.,, 2023; Yang et al., 2023c, ). One can
generate a dataset of instruction-API pairs via self-instruct to tune a
smaller LLM (e.g. , Vicuna-7B (Vicuna,, 2023)).

###### LMM as the tool allocator?

In addition, as LMMs evolve, we envision that the LLM can be replaced by a LMM
as the tool allocator in the system, to enable even more advanced application
scenarios. If the tool allocator can take multimodal inputs, there is no need
to unify the outputs of tools into text sequence, allowing more natural
interactions between the tool allocator and multimodal tools, particularly
those producing multimodal outputs. For instance, one can imagine using
multimodal GPT-4 (OpenAI, 2023a, ) to coordinate various image or video
generation tools to make a short movie by providing it with a sketch of the
storyline and visual examples of the main characters.

####  6.4.3 Diverse Applications of Multimodal Agents

By composing tools from a specific domain, this new system paradigm can also
support diverse domain-specific applications.

Yu et al., 2023b  composes LLMs with image synthesis tools and object-
level/pixel-level image understanding tools to build a data synthesis pipeline
to provide diverse annotations on synthesized image. Instruct2Act (Huang et
al., 2023c, ) complements the LLM with robotic executors, to enable robotic
actions based on multi-modal instructions. When chaining a pool of audio
models with LLM, AudioGPT (Huang et al., 2023a, ) can understand and generate
speech, music, sound and talking head. Similarly, WavJourney (Liu et al.,
2023i, ) further supports compositional audio creation with storylines
encompassing speech, music, and sound effects. With tracking, captioning,
audio understanding models, ChatVideo (Wang et al., 2023c, ) enables ChatGPT
to understand multi-channel videos. Other application scenarios include 3D
scene generation (Lin et al.,, 2023; Feng et al.,, 2023), medical image
understanding (Liu and Zuo,, 2023; Sun et al., 2023c, ) and vision-language
navigation (Zhou et al., 2023b, ).

Figure 6.9: Example of evaluation benchmark focusing on toll using accuracy.
Figure 6.10: Example of evaluation benchmark focusing on emergent
capabilities. MM-Vet focuses on the integration of different core VL
capabilities, including recognition, OCR, knowledge, language generation,
spatial awareness, and math. Image credit: Yu et al., 2023d .

####  6.4.4 Evaluation of Multimodal Agents

###### Multimodal tool using.

Although we have seen qualitative examples of new scenarios enabled by
multimodal agents, it remains unclear how these agents perform in terms of the
accuracy in tool using. API-Bank (Li et al., 2023k, ) is a starting point on
building pipeline in systematically evaluating tool-augmented LLMs.

###### Emergent capabilities.

Existing VL benchmarks focus on specific capabilities of interest, such as
visual recognition (Antol et al.,, 2015), image description (Chen et al.,,
2015; Agrawal et al.,, 2019), as well as other benchmarks for specialized
capabilities such as scene text understanding (Sidorov et al.,, 2020; Gurari
et al.,, 2018), commonsense reasoning (Zellers et al.,, 2019), outside
knowledge (Schwenk et al.,, 2022). The intriguing abilities shown in large
multimodal models and multimodal agents are not examined by existing
benchmarks, such as solving math problems written on the blackboard, reasoning
about events and celebrities in news images, or explaining visual jokes.
Furthermore, the long, chatty outputs from these systems poses challenges to
today’s evaluation metrics. Researchers (Fu et al.,, 2023; Liu et al., 2023j,
) have started to design comprehensive evaluation samples to facilitate the
LMM evaluation. As an attempt to test multimodal systems on integrated
capabilities, MM-Vet (Yu et al., 2023d, ) defines 6 core VL capabilities and
examines the 16 integrations of interest derived from the capability
combination (Figure 6.10). In addition, to accommodate for the open-ended
free-form text outputs, MM-Vet proposes an LLM-based evaluator to enable
evaluation across different question types and answer styles.

####  6.4.5 Tool Creation

Imagine if we have a completely new scenario without a robust tool to use. Can
we create a tool based on the user need on-the-fly? In NLP, CREATOR (Qian et
al.,, 2023) proposes to create tools by writing python code for math
reasoning, as opposed to calling math solver API such as Wolfram Alpha. Cai et
al., (2023) further explores the capabilities of LLMs to make tools, and
experiment with two LLMs, one as the tool maker and the other as the tool user
to collaboratively solve complicated tasks, such as scheduling a meeting. In
terms of multimodal agents, the challenge is how to create a tool that can
process multimodal inputs. One may follow ViperGPT (Surís et al.,, 2023) to
instruct LLMs to generate python programs leveraging pre-existent python
packages such as Open-CV. AutoML GPT (Zhang et al., 2023j, ) envisions that
one can utilize LLMs to automate the model training pipeline. There may be
potential to develop novel multimodal deep learning tools tailored to more
effectively address the requirements of users.

####  6.4.6 Retrieval-Augmented Multimodal Agents

In real-life applications, a substantial portion of information resides within
databases, and user needs may require accurate retrieval of such information.
Meanwhile, it is infeasible to encode all the world knowledge into the weights
of pre-trained models, particularly when it comes to the long-tail concepts
and fast-evolving data.

In NLP, several works augment LLMs with external data encoded with structured
language and relation representations (Peters et al.,, 2019; Guu et al.,,
2020; Lewis et al.,, 2020). Given input texts, such retrieved-augmented models
utilize a retriever that retrieves relevant documents from an external memory,
and uses a generator to generate predictions given the retrieved documents.

Motivated by retrieval-augmented models in NLP, several recent works leverage
visual and / or textual knowledge to improve vision tasks, such as image
classification (Long et al.,, 2022), captioning (Yang et al., 2023a, ),
question answering (Wu et al.,, 2021; Marino et al.,, 2021; Yang et al.,
2022d, ; Chen et al., 2022e, ), image generation (Blattmann et al.,, 2022;
Sheynin et al.,, 2022; Chen et al., 2022f, ; Zhou et al., 2022c, ), and multi-
modal tasks simultaneously (Yasunaga et al.,, 2022). RAC (Long et al.,, 2022)
improves long-tail classification by retrieving from a non-parametric memory
consisting of pre-encoded images and text. K-LITE (Shen et al., 2022a, )
enhances the text prompts with the retrieved external knowledge that is
encoded in natural language. REACT (Liu et al., 2023d, ) retrieve from
billions of the paired knowledge of image-text and aims to improve task
transfer performance for core vision problems. Among them, RA-CM3 (Yasunaga et
al.,, 2022) builds the first retrieval-augmented LMM with a multimodal
retriever to retrieve multimodal documents, and a retrieval-augmented
generator that can generate both text and image. Chaining tools with LLM
shares a strong connection with the retrieval-augmented methods in that both
leverage external knowledge to provide additional information for the core
model to utilize. In the multimodal regime, the image itself can be used as
the query to gain external knowledge, either retrieved from a knowledge base,
or extracted from another pre-trained vision expert models.

##  Chapter 7 Conclusions and Research Trends

Multimodal foundation models have garnered significant interest among scholars
in the fields of computer vision and multimodal vision-language research.
Although prevailing research topics, approaches and methodologies have been
evolving – encompassing image self-supervised learning, language-image
contrastive learning, text-to-image generation, unified vision modeling, and
large language-and-vision assistants – they converge on a common overarching
objective: the creation of general-purpose models and systems capable of
following human intents and effortlessly executing a diverse array of vision
and vision-language tasks in the wild. In this chapter, we provide a concise
summary of what has been reviewed, and delve into the prevailing research
tendencies in the field.

###  7.1 Summary and Conclusions

This paper surveys the most recent advances at the frontier of multimodal
foundation model research, categorized into two classes discussed below.

  * •

Specific-purpose multimodal foundation models. There is a diverse set of
problems to tackle in the computer vision community. To lay a comprehensive
foundation for the introduction of general-purpose visual assistants, we have
discussed many seminar papers in the era of pre-training. The major paradigm
during this period is pre-training on a large amount of problem-related data,
and then transferring to a number of real-world scenarios of the same problem
type in a zero- or few-shot fashion. More specifically, we have presented two
general topics: ($i$) _Visual Understanding_ in Chapter 2: individual
multimodal foundation models have developed to analyze the content of visual
data in the image, region, pixel levels, prospectively. The language-augmented
vision models are a popular family, contributing to the recent success of
visual understanding tasks in the wild. ($ii$) _Visual Generation_ in Chapter
3: text-to-image generation models have served the foundation for image
synthesis, which has been successfully extended to allow user controllability
and customization at more fine-grained manners. The availability and creation
of large amount of problem-related data has played a key role in making these
multimodal foundation models possible.

  * •

General-purpose assistants. We have reviewed recently emerged literature on
building general-purpose assistants, which often possess an unified network
architecture, an unified input-output data format, and a general interface
that facilitates easy interaction with humans. Inspired by the success in NLP
that LLM such as ChatGPT/GPT-4 is a general assistant for a wide range of
language tasks, researchers in computer vision have explored various solutions
to their counterpart for vision tasks. Based on how LLM is leveraged in the
methodology, existing works can be categorized into three topics: ($i$)
_Unified Vision Models_ in Chapter 4: The spirit of unifying modeling in LLM
is borrowed to build unified vision models at different levels and across
different tasks. ($ii$) _Training with LLM_ in Chapter 5: Starting with a pre-
trained LLM, visual data is connected to LLM for end-to-end training. ($iii$)
_Chaining with LLM_ in Chapter 6: By freezing LLM, existing vision experts can
be triggered by prompt engineering LLM to complete specific vision tasks.

The comparisons among these models are summarized in Table 7.1.

| Models |  Advantages |  Disadvantages
---|---|---|---
Specific-Purpose Models | Visual Understanding |  Well studied and scalable solutions on image-level understanding; Emerging interests and success on region-level and pixel-level visual understanding |  High training cost; No successful scalable recipe beyond the billion-image level
Visual Generation |  Well studied and scalable solutions on image-level generation; Emerging interests and success in controllable/customized image generation |  High training and inference cost; Debate between diffusion and auto-regressive solutions for the best recipe; More studies are needed for video generation
General-Purpose Assistants | Unified Vision Models |  Promises to unlock new emerging capabilities and scenarios |  High risks in modeling and high training cost
Training with LLM |  Some new emerging capabilities and scenarios are enabled with light model training |  The performance is bounded by LLM
Chaining with LLM |  Fast system development cycles with low cost as no training is involved |  Low flexibility in improving system performance; No new emerging capabilities

Table 7.1: Comparisons of different multimodal foundation model families
covered in this paper.

###  7.2 Towards Building General-Purpose AI Agents

At the end of each chapter, we have discussed future trends for individual
topics. The paper itself is organized to demonstrate the transition from
specialist multimodal foundation models to general-purpose visual assistants.
Though powerful, existing visual assistants such as Flamingo (Alayrac et al.,,
2022) and multimodal GPT-4 (OpenAI, 2023b, ) are in the preliminary form,
compared with grand vision on building a general-purpose multimodal AI agent
via foundation models. In what follows, we highlight a number of research
trends towards this goal.

###### Generalist agents with multi-modality.

This aligns with the grand goal of building a single generalist agent that
interacts with world like humans through fusing multiple channels such as
language, vision, speech and actions. From this perspective, the notion of
multimodal foundation models becomes somewhat indistinct on its own. Instead,
it serves as a crucial component of the agent for perceiving and synthesizing
visual signals. For example, Gato (Reed et al.,, 2022) and PaLM-E (Driess et
al.,, 2023) perform a wide range of language, multimodal and control tasks
with a single set of model weights, where visual perception is a crucial
component in understanding the environment. It also raises challenges in the
effective and scalable pre-training objectives for unified vision and
multimodal modeling.

###### Alignment with human intents.

AI alignment research focuses on steering AI systems towards humans’ intended
goals, values, or ethical guidelines. An AI system is deemed aligned when it
effectively promotes the desired goals. Though language has exhibited its
generality in expressing human intents, it is not always the best option. As
demonstrated in SAM (Kirillov et al.,, 2023) and ControlNet/GLIGEN (Zhang and
Agrawala,, 2023; Li et al., 2023n, ), human intents can be more precisely and
conveniently represented in visual prompts such as key points, bounding boxes
and sketch drawing, for visual understanding and generation tasks,
respectively. Building foundation models that are well equipped with such a
multimodal human-machine interaction interface is a key step to unlock new use
scenarios, where human intents are best represented visually. For example, the
spatial arrangement of elements within a scene, as well as the artistic style
and visual appeal of a piece of visual art.

###### Planning, memory, and tool use.

It is highlighted in Weng, (2023) that a LLM-powered autonomous agent system
can be built, where LLM functions as the agent’s brain, complemented by
several key components: planning, memory and tool use. Following the
framework, we could foresee the role of multimodal foundation models in this
AI agent system. ($i$) Planning. To complete complex tasks in real-world
scenarios, the agent should be able to decompose large tasks into smaller,
manageable subgoals, enabling efficient handling of complex tasks. In the
ideal case, the AI agent possesses the self-improvement ability, engaging in
self-assessment and introspection regarding previous actions, enabling it to
learn from errors and enhance its approach for subsequent endeavors,
ultimately leading to better outcomes. Visual modality is a common channel to
represent state of the environment. To facilitate planning, it raises
challenges in improving the capability of the current visual understanding
models in perceiving more fine-grained visual details and longer sequence
videos. ($ii$) Memory. For short-term memory, in-context learning (or prompt
engineering) is utilized as short-term memory for the model to learn.
Interleaved multimodal prompts can enable new scenarios to clarify the human
intents. For long-term memory, it provides the agent with the capability to
recall external knowledge over extended sessions, which can be implemented by
fast retrieving from a multi-modal vector space (Liu et al., 2023d, ). In term
of modeling, foundation models are required to learn the new skills to
effectively leverage both types of memory. ($iii$) Tool use. The agent learns
to utilize external APIs for knowledge that is missing from the foundation
model weights. New capabilities are required to deal with the vision modality
in several scenarios. For example, based on both the input visual signal and
instructions, the model decides and plans whether certain external APIs are
needed to complete the goal, such as code execution of
detection/segmentation/OCR/generator experts.

The field of multimodal foundation models is evolving at a rapid speed, with
new directions/methods emerging frequently. There are many important research
topics that are not discussed in this paper, mostly due to the daily-updated
research innovation. We are optimistic about the future of multimodal
foundation models, not only because we are convinced that foreseeable exciting
research innovations/ideas in individual areas are becoming reality by
following the path of LLM in the near future, but also because connecting
computer vision with the broader AI community, and building general-purpose AI
agents is going to significantly advance the daily life of human being.

## Acknowledgments

This book is largely based on our CVPR 2023 tutorial on vision foundation
models. Many people have supported us and provided valuable feedback to the
writing of this book. We thank all the authors who have contributed to the
related papers, which makes the tutorial and book possible. We are also
grateful to Mark de Jongh, the editor from the journal of Foundations and
Trends® in Computer Graphics and Vision, for inspiring and encouraging us to
write the book on multimodal foundation models.

## References

  * sta, (2022) (2022).  Stable diffusion.  https://github.com/CompVis/stable-diffusion.
  * Agarwal et al., (2023) Agarwal, A., Karanam, S., Joseph, K., Saxena, A., Goswami, K., and Srinivasan, B. V. (2023).  A-star: Test-time attention segregation and retention for text-to-image synthesis.  arXiv preprint arXiv:2306.14544.
  * Agrawal et al., (2019) Agrawal, H., Desai, K., Wang, Y., Chen, X., Jain, R., Johnson, M., Batra, D., Parikh, D., Lee, S., and Anderson, P. (2019).  nocaps: novel object captioning at scale.  In ICCV.
  * Ahn et al., (2022) Ahn, M., Brohan, A., Brown, N., Chebotar, Y., Cortes, O., David, B., Finn, C., Gopalakrishnan, K., Hausman, K., Herzog, A., et al. (2022).  Do as i can, not as i say: Grounding language in robotic affordances.  arXiv preprint arXiv:2204.01691.
  * Alayrac et al., (2022) Alayrac, J.-B., Donahue, J., Luc, P., Miech, A., Barr, I., Hasson, Y., Lenc, K., Mensch, A., Millican, K., Reynolds, M., et al. (2022).  Flamingo: a visual language model for few-shot learning.  arXiv preprint arXiv:2204.14198.
  * Allahyari et al., (2017) Allahyari, M., Pouriyeh, S., Assefi, M., Safaei, S., Trippe, E. D., Gutierrez, J. B., and Kochut, K. (2017).  Text summarization techniques: a brief survey.  arXiv preprint arXiv:1707.02268.
  * Amrani et al., (2022) Amrani, E., Karlinsky, L., and Bronstein, A. (2022).  Self-supervised classification network.  In ECCV.
  * Anderson et al., (2018) Anderson, P., He, X., Buehler, C., Teney, D., Johnson, M., Gould, S., and Zhang, L. (2018).  Bottom-up and top-down attention for image captioning and visual question answering.  In CVPR.
  * Antol et al., (2015) Antol, S., Agrawal, A., Lu, J., Mitchell, M., Batra, D., Zitnick, C. L., and Parikh, D. (2015).  Vqa: Visual question answering.  In ICCV.
  * Arora et al., (2019) Arora, S., Khandeparkar, H., Khodak, M., Plevrakis, O., and Saunshi, N. (2019).  A theoretical analysis of contrastive unsupervised representation learning.  arXiv preprint arXiv:1902.09229.
  * Assran et al., (2022) Assran, M., Caron, M., Misra, I., Bojanowski, P., Bordes, F., Vincent, P., Joulin, A., Rabbat, M., and Ballas, N. (2022).  Masked siamese networks for label-efficient learning.  In ECCV.
  * (12) Avrahami, O., Aberman, K., Fried, O., Cohen-Or, D., and Lischinski, D. (2023a).  Break-a-scene: Extracting multiple concepts from a single image.  arXiv preprint arXiv:2305.16311.
  * (13) Avrahami, O., Fried, O., and Lischinski, D. (2022a).  Blended latent diffusion.  arXiv preprint arXiv:2206.02779.
  * (14) Avrahami, O., Hayes, T., Gafni, O., Gupta, S., Taigman, Y., Parikh, D., Lischinski, D., Fried, O., and Yin, X. (2023b).  Spatext: Spatio-textual representation for controllable image generation.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18370–18380.
  * (15) Avrahami, O., Lischinski, D., and Fried, O. (2022b).  Blended diffusion for text-driven editing of natural images.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18208–18218.
  * Awadalla et al., (2023) Awadalla, A., Gao, I., Gardner, J., Hessel, J., Hanafy, Y., Zhu, W., Marathe, K., Bitton, Y., Gadre, S., Jitsev, J., Kornblith, S., Koh, P. W., Ilharco, G., Wortsman, M., and Schmidt, L. (2023).  Openflamingo.
  * Awais et al., (2023) Awais, M., Naseer, M., Khan, S., Anwer, R. M., Cholakkal, H., Shah, M., Yang, M.-H., and Khan, F. S. (2023).  Foundational models defining a new era in vision: A survey and outlook.  arXiv preprint arXiv:2307.13721.
  * Bachman et al., (2019) Bachman, P., Hjelm, R. D., and Buchwalter, W. (2019).  Learning representations by maximizing mutual information across views.  NeurIPS.
  * Baevski et al., (2022) Baevski, A., Hsu, W.-N., Xu, Q., Babu, A., Gu, J., and Auli, M. (2022).  Data2vec: A general framework for self-supervised learning in speech, vision and language.  In ICML.
  * Bahdanau et al., (2015) Bahdanau, D., Cho, K., and Bengio, Y. (2015).  Neural machine translation by jointly learning to align and translate.  In ICLR.
  * (21) Bai, J., Bai, S., Yang, S., Wang, S., Tan, S., Wang, P., Lin, J., Zhou, C., and Zhou, J. (2023a).  Qwen-vl: A frontier large vision-language model with versatile abilities.  arXiv preprint arXiv:2308.12966.
  * (22) Bai, S., Yang, S., Bai, J., Wang, P., Zhang, X., Lin, J., Wang, X., Zhou, C., and Zhou, J. (2023b).  Touchstone: Evaluating vision-language models by language models.
  * Balaji et al., (2022) Balaji, Y., Nah, S., Huang, X., Vahdat, A., Song, J., Kreis, K., Aittala, M., Aila, T., Laine, S., Catanzaro, B., et al. (2022).  ediffi: Text-to-image diffusion models with an ensemble of expert denoisers.  arXiv preprint arXiv:2211.01324.
  * Balažević et al., (2023) Balažević, I., Steiner, D., Parthasarathy, N., Arandjelović, R., and Hénaff, O. J. (2023).  Towards in-context scene understanding.  arXiv preprint arXiv:2306.01667.
  * Bansal et al., (2023) Bansal, A., Chu, H.-M., Schwarzschild, A., Sengupta, S., Goldblum, M., Geiping, J., and Goldstein, T. (2023).  Universal guidance for diffusion models.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 843–852.
  * Bansal et al., (2018) Bansal, A., Sikka, K., Sharma, G., Chellappa, R., and Divakaran, A. (2018).  Zero-shot object detection.  In Proceedings of the European conference on computer vision (ECCV), pages 384–400.
  * Bao et al., (2022) Bao, H., Dong, L., and Wei, F. (2022).  BEiT: Bert pre-training of image transformers.  In ICLR.
  * Bar et al., (2022) Bar, A., Gandelsman, Y., Darrell, T., Globerson, A., and Efros, A. (2022).  Visual prompting via image inpainting.  Advances in Neural Information Processing Systems, 35:25005–25017.
  * Bardes et al., (2021) Bardes, A., Ponce, J., and LeCun, Y. (2021).  Vicreg: Variance-invariance-covariance regularization for self-supervised learning.  arXiv preprint arXiv:2105.04906.
  * Berant et al., (2013) Berant, J., Chou, A., Frostig, R., and Liang, P. (2013).  Semantic parsing on freebase from question-answer pairs.  In Proceedings of the 2013 conference on empirical methods in natural language processing, pages 1533–1544.
  * Bitton et al., (2023) Bitton, Y., Bansal, H., Hessel, J., Shao, R., Zhu, W., Awadalla, A., Gardner, J., Taori, R., and Schimdt, L. (2023).  Visit-bench: A benchmark for vision-language instruction following inspired by real-world use.
  * Black et al., (2023) Black, K., Janner, M., Du, Y., Kostrikov, I., and Levine, S. (2023).  Training diffusion models with reinforcement learning.  arXiv preprint arXiv:2305.13301.
  * Blattmann et al., (2022) Blattmann, A., Rombach, R., Oktay, K., and Ommer, B. (2022).  Retrieval-augmented diffusion models.  arXiv preprint arXiv:2204.11824.
  * Bolya et al., (2019) Bolya, D., Zhou, C., Xiao, F., and Lee, Y. J. (2019).  Yolact: Real-time instance segmentation.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 9157–9166.
  * Bommasani et al., (2021) Bommasani, R., Hudson, D. A., Adeli, E., Altman, R., Arora, S., von Arx, S., Bernstein, M. S., Bohg, J., Bosselut, A., Brunskill, E., et al. (2021).  On the opportunities and risks of foundation models.  arXiv preprint arXiv:2108.07258.
  * Brooks et al., (2023) Brooks, T., Holynski, A., and Efros, A. A. (2023).  Instructpix2pix: Learning to follow image editing instructions.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18392–18402.
  * Brown et al., (2020) Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020).  Language models are few-shot learners.  In NeuIPS.
  * Bubeck et al., (2023) Bubeck, S., Chandrasekaran, V., Eldan, R., Gehrke, J., Horvitz, E., Kamar, E., Lee, P., Lee, Y. T., Li, Y., Lundberg, S., et al. (2023).  Sparks of artificial general intelligence: Early experiments with gpt-4.  arXiv preprint arXiv:2303.12712.
  * Byeon et al., (2022) Byeon, M., Park, B., Kim, H., Lee, S., Baek, W., and Kim, S. (2022).  Coyo-700m: Image-text pair dataset.  https://github.com/kakaobrain/coyo-dataset.
  * Cai et al., (2023) Cai, T., Wang, X., Ma, T., Chen, X., and Zhou, D. (2023).  Large language models as tool makers.  arXiv preprint arXiv:2305.17126.
  * Cai et al., (2022) Cai, Z., Kwon, G., Ravichandran, A., Bas, E., Tu, Z., Bhotika, R., and Soatto, S. (2022).  X-detr: A versatile architecture for instance-wise vision-language tasks.  In ECCV.
  * Cao et al., (2023) Cao, L., Zhang, B., Chen, C., Yang, Y., Du, X., Zhang, W., Lu, Z., and Zheng, Y. (2023).  Less is more: Removing text-regions improves clip training efficiency and robustness.  arXiv preprint arXiv:2305.05095.
  * Carion et al., (2020) Carion, N., Massa, F., Synnaeve, G., Usunier, N., Kirillov, A., and Zagoruyko, S. (2020).  End-to-end object detection with transformers.  In ECCV.
  * Caron et al., (2018) Caron, M., Bojanowski, P., Joulin, A., and Douze, M. (2018).  Deep clustering for unsupervised learning of visual features.  In ECCV.
  * Caron et al., (2020) Caron, M., Misra, I., Mairal, J., Goyal, P., Bojanowski, P., and Joulin, A. (2020).  Unsupervised learning of visual features by contrasting cluster assignments.  NeurIPS.
  * Caron et al., (2021) Caron, M., Touvron, H., Misra, I., Jégou, H., Mairal, J., Bojanowski, P., and Joulin, A. (2021).  Emerging properties in self-supervised vision transformers.  In ICCV.
  * Castrejon et al., (2017) Castrejon, L., Kundu, K., Urtasun, R., and Fidler, S. (2017).  Annotating object instances with a polygon-rnn.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 5230–5238.
  * Chang et al., (2023) Chang, H., Zhang, H., Barber, J., Maschinot, A., Lezama, J., Jiang, L., Yang, M.-H., Murphy, K., Freeman, W. T., Rubinstein, M., et al. (2023).  Muse: Text-to-image generation via masked generative transformers.  arXiv preprint arXiv:2301.00704.
  * Chang et al., (2022) Chang, H., Zhang, H., Jiang, L., Liu, C., and Freeman, W. T. (2022).  Maskgit: Masked generative image transformer.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11315–11325.
  * Changpinyo et al., (2021) Changpinyo, S., Sharma, P., Ding, N., and Soricut, R. (2021).  Conceptual 12m: Pushing web-scale image-text pre-training to recognize long-tail visual concepts.  In CVPR.
  * Chefer et al., (2023) Chefer, H., Alaluf, Y., Vinker, Y., Wolf, L., and Cohen-Or, D. (2023).  Attend-and-excite: Attention-based semantic guidance for text-to-image diffusion models.  arXiv preprint arXiv:2301.13826.
  * (52) Chen, C., Zhang, B., Cao, L., Shen, J., Gunter, T., Jose, A. M., Toshev, A., Shlens, J., Pang, R., and Yang, Y. (2023a).  Stair: Learning sparse text and image representation in grounded tokens.  arXiv preprint arXiv:2301.13081.
  * (53) Chen, D., Liu, J., Dai, W., and Wang, B. (2023b).  Visual instruction tuning with polite flamingo.  arXiv preprint arXiv:2307.01003.
  * (54) Chen, F., Han, M., Zhao, H., Zhang, Q., Shi, J., Xu, S., and Xu, B. (2023c).  X-llm: Bootstrapping advanced large language models by treating multi-modalities as foreign languages.  arXiv preprint arXiv:2305.04160.
  * (55) Chen, F., Zhang, D., Han, M., Chen, X., Shi, J., Xu, S., and Xu, B. (2022a).  Vlp: A survey on vision-language pre-training.  arXiv preprint arXiv:2202.09061.
  * (56) Chen, K., Zhang, Z., Zeng, W., Zhang, R., Zhu, F., and Zhao, R. (2023d).  Shikra: Unleashing multimodal llm’s referential dialogue magic.  arXiv preprint arXiv:2306.15195.
  * Chen et al., (2019) Chen, L., Zhai, M., He, J., and Mori, G. (2019).  Object grounding via iterative context reasoning.  In Proceedings of the IEEE/CVF International Conference on Computer Vision Workshops, pages 0–0.
  * Chen et al., (2017) Chen, L.-C., Papandreou, G., Schroff, F., and Adam, H. (2017).  Rethinking atrous convolution for semantic image segmentation.  arXiv preprint arXiv:1706.05587.
  * (59) Chen, M., Laina, I., and Vedaldi, A. (2023e).  Training-free layout control with cross-attention guidance.  arXiv preprint arXiv:2304.03373.
  * (60) Chen, M., Tworek, J., Jun, H., Yuan, Q., Pinto, H. P. d. O., Kaplan, J., Edwards, H., Burda, Y., Joseph, N., Brockman, G., et al. (2021a).  Evaluating large language models trained on code.  arXiv preprint arXiv:2107.03374.
  * (61) Chen, Q., Chen, X., Zeng, G., and Wang, J. (2022b).  Group detr: Fast training convergence with decoupled one-to-many label assignment.  arXiv preprint arXiv:2207.13085.
  * (62) Chen, T., Kornblith, S., Norouzi, M., and Hinton, G. (2020a).  A simple framework for contrastive learning of visual representations.  In ICML.
  * (63) Chen, T., Kornblith, S., Swersky, K., Norouzi, M., and Hinton, G. E. (2020b).  Big self-supervised models are strong semi-supervised learners.  NeurIPS.
  * (64) Chen, T., Saxena, S., Li, L., Fleet, D. J., and Hinton, G. (2022c).  Pix2seq: A language modeling framework for object detection.  In ICLR.
  * (65) Chen, T., Saxena, S., Li, L., Lin, T.-Y., Fleet, D. J., and Hinton, G. (2022d).  A unified sequence interface for vision tasks.  arXiv preprint arXiv:2206.07669.
  * (66) Chen, W., Hu, H., Chen, X., Verga, P., and Cohen, W. W. (2022e).  Murag: Multimodal retrieval-augmented generator for open question answering over images and text.  arXiv preprint arXiv:2210.02928.
  * (67) Chen, W., Hu, H., Li, Y., Rui, N., Jia, X., Chang, M.-W., and Cohen, W. W. (2023f).  Subject-driven text-to-image generation via apprenticeship learning.  arXiv preprint arXiv:2304.00186.
  * (68) Chen, W., Hu, H., Saharia, C., and Cohen, W. W. (2022f).  Re-imagen: Retrieval-augmented text-to-image generator.  arXiv preprint arXiv:2209.14491.
  * (69) Chen, X., Ding, M., Wang, X., Xin, Y., Mo, S., Wang, Y., Han, S., Luo, P., Zeng, G., and Wang, J. (2022g).  Context autoencoder for self-supervised representation learning.  arXiv preprint arXiv:2202.03026.
  * (70) Chen, X., Djolonga, J., Padlewski, P., Mustafa, B., Changpinyo, S., Wu, J., Ruiz, C. R., Goodman, S., Wang, X., Tay, Y., et al. (2023g).  Pali-x: On scaling up a multilingual vision and language model.  arXiv preprint arXiv:2305.18565.
  * (71) Chen, X., Fan, H., Girshick, R., and He, K. (2020c).  Improved baselines with momentum contrastive learning.  arXiv preprint arXiv:2003.04297.
  * Chen et al., (2015) Chen, X., Fang, H., Lin, T., Vedantam, R., Gupta, S., Dollár, P., and Zitnick, C. L. (2015).  Microsoft COCO captions: Data collection and evaluation server.  arXiv preprint arXiv:1504.00325.
  * Chen and He, (2021) Chen, X. and He, K. (2021).  Exploring simple siamese representation learning.  In CVPR.
  * (74) Chen, X., Wang, X., Changpinyo, S., Piergiovanni, A., Padlewski, P., Salz, D., Goodman, S., Grycner, A., Mustafa, B., Beyer, L., et al. (2022h).  Pali: A jointly-scaled multilingual language-image model.  arXiv preprint arXiv:2209.06794.
  * (75) Chen, X., Xie, S., and He, K. (2021b).  An empirical study of training self-supervised vision transformers.  In ICCV.
  * (76) Chen, X., Zhao, Z., Yu, F., Zhang, Y., and Duan, M. (2021c).  Conditional diffusion for interactive segmentation.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 7345–7354.
  * (77) Chen, X., Zhao, Z., Zhang, Y., Duan, M., Qi, D., and Zhao, H. (2022i).  Focalclick: Towards practical interactive image segmentation.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1300–1309.
  * (78) Chen, Y.-C., Li, L., Yu, L., El Kholy, A., Ahmed, F., Gan, Z., Cheng, Y., and Liu, J. (2020d).  UNITER: Universal image-text representation learning.  In ECCV.
  * (79) Chen, Z., Duan, Y., Wang, W., He, J., Lu, T., Dai, J., and Qiao, Y. (2022j).  Vision transformer adapter for dense predictions.  arXiv preprint arXiv:2205.08534.
  * Cheng et al., (2022) Cheng, B., Misra, I., Schwing, A. G., Kirillov, A., and Girdhar, R. (2022).  Masked-attention mask transformer for universal image segmentation.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1290–1299.
  * Cherti et al., (2023) Cherti, M., Beaumont, R., Wightman, R., Wortsman, M., Ilharco, G., Gordon, C., Schuhmann, C., Schmidt, L., and Jitsev, J. (2023).  Reproducible scaling laws for contrastive language-image learning.  In CVPR.
  * Cho et al., (2021) Cho, J., Lei, J., Tan, H., and Bansal, M. (2021).  Unifying vision-and-language tasks via text generation.  In ICML.
  * Cho et al., (2023) Cho, J., Li, L., Yang, Z., Gan, Z., Wang, L., and Bansal, M. (2023).  Diagnostic benchmark and iterative inpainting for layout-guided image generation.  arXiv preprint arXiv:2304.06671.
  * Chowdhery et al., (2022) Chowdhery, A., Narang, S., Devlin, J., Bosma, M., Mishra, G., Roberts, A., Barham, P., Chung, H. W., Sutton, C., Gehrmann, S., et al. (2022).  Palm: Scaling language modeling with pathways.  arXiv preprint arXiv:2204.02311.
  * Computer, (2023) Computer, T. (2023).  Redpajama-data: An open source recipe to reproduce llama training dataset.
  * Crawshaw, (2020) Crawshaw, M. (2020).  Multi-task learning with deep neural networks: A survey.  arXiv preprint arXiv:2009.09796.
  * Creswell et al., (2018) Creswell, A., White, T., Dumoulin, V., Arulkumaran, K., Sengupta, B., and Bharath, A. A. (2018).  Generative adversarial networks: An overview.  IEEE signal processing magazine, 35(1):53–65.
  * (88) Dai, H., Ma, C., Liu, Z., Li, Y., Shu, P., Wei, X., Zhao, L., Wu, Z., Zhu, D., Liu, W., et al. (2023a).  Samaug: Point prompt augmentation for segment anything model.  arXiv preprint arXiv:2307.01187.
  * (89) Dai, W., Li, J., Li, D., Tiong, A. M. H., Zhao, J., Wang, W., Li, B., Fung, P., and Hoi, S. (2023b).  Instructblip: Towards general-purpose vision-language models with instruction tuning.  arXiv preprint arXiv:2305.06500.
  * Dai et al., (2021) Dai, X., Chen, Y., Xiao, B., Chen, D., Liu, M., Yuan, L., and Zhang, L. (2021).  Dynamic head: Unifying object detection heads with attentions.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 7373–7382.
  * Datta et al., (2008) Datta, R., Joshi, D., Li, J., and Wang, J. Z. (2008).  Image retrieval: Ideas, influences, and trends of the new age.  ACM Computing Surveys (Csur), 40(2):1–60.
  * Deng et al., (2018) Deng, C., Wu, Q., Wu, Q., Hu, F., Lyu, F., and Tan, M. (2018).  Visual grounding via accumulated attention.  In CVPR.
  * Deng et al., (2009) Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., and Fei-Fei, L. (2009).  Imagenet: A large-scale hierarchical image database.  In CVPR.
  * Desai and Johnson, (2021) Desai, K. and Johnson, J. (2021).  Virtex: Learning visual representations from textual annotations.  In CVPR.
  * Desai et al., (2021) Desai, K., Kaul, G., Aysola, Z., and Johnson, J. (2021).  Redcaps: Web-curated image-text data created by the people, for the people.  In NeurIPS, Track on Datasets and Benchmarks.
  * Dettmers et al., (2023) Dettmers, T., Pagnoni, A., Holtzman, A., and Zettlemoyer, L. (2023).  Qlora: Efficient finetuning of quantized llms.  arXiv preprint arXiv:2305.14314.
  * Devlin et al., (2019) Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K. (2019).  Bert: Pre-training of deep bidirectional transformers for language understanding.  In NAACL.
  * Dhariwal and Nichol, (2021) Dhariwal, P. and Nichol, A. (2021).  Diffusion models beat gans on image synthesis.  In NeurIPS.
  * (99) Ding, J., Xue, N., Xia, G.-S., and Dai, D. (2022a).  Decoupling zero-shot semantic segmentation.
  * (100) Ding, Z., Wang, J., and Tu, Z. (2022b).  Open-vocabulary panoptic segmentation with maskclip.  arXiv preprint arXiv:2208.08984.
  * Dong et al., (2021) Dong, B., Zeng, F., Wang, T., Zhang, X., and Wei, Y. (2021).  Solq: Segmenting objects by learning queries.  Advances in Neural Information Processing Systems, 34:21898–21909.
  * Dong et al., (2022) Dong, X., Bao, J., Zhang, T., Chen, D., Zhang, W., Yuan, L., Chen, D., Wen, F., and Yu, N. (2022).  Bootstrapped masked autoencoders for vision bert pretraining.  In ECCV.
  * Dong et al., (2023) Dong, X., Bao, J., Zhang, T., Chen, D., Zhang, W., Yuan, L., Chen, D., Wen, F., Yu, N., and Guo, B. (2023).  Peco: Perceptual codebook for bert pre-training of vision transformers.  In AAAI.
  * Dosovitskiy et al., (2021) Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., et al. (2021).  An image is worth 16x16 words: Transformers for image recognition at scale.  In ICLR.
  * (105) Dou, Z.-Y., Kamath, A., Gan, Z., Zhang, P., Wang, J., Li, L., Liu, Z., Liu, C., LeCun, Y., Peng, N., Gao, J., and Wang, L. (2022a).  Coarse-to-fine vision-language pre-training with fusion in the backbone.  In NeurIPS.
  * (106) Dou, Z.-Y., Xu, Y., Gan, Z., Wang, J., Wang, S., Wang, L., Zhu, C., Liu, Z., Zeng, M., et al. (2022b).  An empirical study of training end-to-end vision-and-language transformers.  In CVPR.
  * Driess et al., (2023) Driess, D., Xia, F., Sajjadi, M. S., Lynch, C., Chowdhery, A., Ichter, B., Wahid, A., Tompson, J., Vuong, Q., Yu, T., et al. (2023).  PaLM-E: An embodied multimodal language model.  arXiv preprint arXiv:2303.03378.
  * Du et al., (2022) Du, Y., Liu, Z., Li, J., and Zhao, W. X. (2022).  A survey of vision-language pre-trained models.  In IJCAI survey track.
  * El-Nouby et al., (2021) El-Nouby, A., Izacard, G., Touvron, H., Laptev, I., Jegou, H., and Grave, E. (2021).  Are large-scale datasets necessary for self-supervised pre-training?  arXiv preprint arXiv:2112.10740.
  * Elharrouss et al., (2020) Elharrouss, O., Almaadeed, N., Al-Maadeed, S., and Akbari, Y. (2020).  Image inpainting: A review.  Neural Processing Letters, 51:2007–2028.
  * Ermolov et al., (2021) Ermolov, A., Siarohin, A., Sangineto, E., and Sebe, N. (2021).  Whitening for self-supervised representation learning.  In ICML.
  * Esser et al., (2021) Esser, P., Rombach, R., and Ommer, B. (2021).  Taming transformers for high-resolution image synthesis.  In CVPR.
  * Everingham and Winn, (2011) Everingham, M. and Winn, J. (2011).  The pascal visual object classes challenge 2012 (voc2012) development kit.  Pattern Analysis, Statistical Modelling and Computational Learning, Tech. Rep, 8(5).
  * (114) Fan, L., Krishnan, D., Isola, P., Katabi, D., and Tian, Y. (2023a).  Improving clip training with language rewrites.  arXiv preprint arXiv:2305.20088.
  * (115) Fan, Y., Watkins, O., Du, Y., Liu, H., Ryu, M., Boutilier, C., Abbeel, P., Ghavamzadeh, M., Lee, K., and Lee, K. (2023b).  Dpok: Reinforcement learning for fine-tuning text-to-image diffusion models.  arXiv preprint arXiv:2305.16381.
  * Fang et al., (2023) Fang, Y., Wang, W., Xie, B., Sun, Q., Wu, L., Wang, X., Huang, T., Wang, X., and Cao, Y. (2023).  Eva: Exploring the limits of masked visual representation learning at scale.  In CVPR.
  * Fang et al., (2022) Fang, Z., Wang, J., Hu, X., Liang, L., Gan, Z., Wang, L., Yang, Y., and Liu, Z. (2022).  Injecting semantic concepts into end-to-end image captioning.  In CVPR.
  * Feichtenhofer et al., (2022) Feichtenhofer, C., Li, Y., He, K., et al. (2022).  Masked autoencoders as spatiotemporal learners.  NeurIPS.
  * (119) Feng, C., Zhong, Y., Jie, Z., Chu, X., Ren, H., Wei, X., Xie, W., and Ma, L. (2022a).  Promptdet: Towards open-vocabulary detection using uncurated images.  In European Conference on Computer Vision, pages 701–717. Springer.
  * (120) Feng, W., He, X., Fu, T.-J., Jampani, V., Akula, A. R., Narayana, P., Basu, S., Wang, X. E., and Wang, W. Y. (2022b).  Training-free structured diffusion guidance for compositional text-to-image synthesis.  In The Eleventh International Conference on Learning Representations.
  * Feng et al., (2023) Feng, W., Zhu, W., Fu, T.-j., Jampani, V., Akula, A., He, X., Basu, S., Wang, X. E., and Wang, W. Y. (2023).  Layoutgpt: Compositional visual planning and generation with large language models.  arXiv preprint arXiv:2305.15393.
  * Frome et al., (2013) Frome, A., Corrado, G. S., Shlens, J., Bengio, S., Dean, J., Ranzato, M., and Mikolov, T. (2013).  Devise: A deep visual-semantic embedding model.  In NeurIPS.
  * Fu et al., (2023) Fu, C., Chen, P., Shen, Y., Qin, Y., Zhang, M., Lin, X., Qiu, Z., Lin, W., Yang, J., Zheng, X., et al. (2023).  Mme: A comprehensive evaluation benchmark for multimodal large language models.  arXiv preprint arXiv:2306.13394.
  * Gadre et al., (2023) Gadre, S. Y., Ilharco, G., Fang, A., Hayase, J., Smyrnis, G., Nguyen, T., Marten, R., Wortsman, M., Ghosh, D., Zhang, J., et al. (2023).  Datacomp: In search of the next generation of multimodal datasets.  arXiv preprint arXiv:2304.14108.
  * Gafni et al., (2022) Gafni, O., Polyak, A., Ashual, O., Sheynin, S., Parikh, D., and Taigman, Y. (2022).  Make-a-scene: Scene-based text-to-image generation with human priors.  arXiv preprint arXiv:2203.13131.
  * Gal et al., (2022) Gal, R., Alaluf, Y., Atzmon, Y., Patashnik, O., Bermano, A. H., Chechik, G., and Cohen-Or, D. (2022).  An image is worth one word: Personalizing text-to-image generation using textual inversion.  arXiv preprint arXiv:2208.01618.
  * Gan et al., (2020) Gan, Z., Chen, Y.-C., Li, L., Zhu, C., Cheng, Y., and Liu, J. (2020).  Large-scale adversarial training for vision-and-language representation learning.  In NeurIPS.
  * Gan et al., (2022) Gan, Z., Li, L., Li, C., Wang, L., Liu, Z., Gao, J., et al. (2022).  Vision-language pre-training: Basics, recent advances, and future trends.  Foundations and Trends® in Computer Graphics and Vision.
  * (129) Gao, D., Ji, L., Zhou, L., Lin, K. Q., Chen, J., Fan, Z., and Shou, M. Z. (2023a).  Assistgpt: A general multi-modal assistant that can plan, execute, inspect, and learn.  arXiv preprint arXiv:2306.08640.
  * (130) Gao, P., Han, J., Zhang, R., Lin, Z., Geng, S., Zhou, A., Zhang, W., Lu, P., He, C., Yue, X., et al. (2023b).  Llama-adapter v2: Parameter-efficient visual instruction model.  arXiv preprint arXiv:2304.15010.
  * Gao et al., (2022) Gao, P., Ma, T., Li, H., Lin, Z., Dai, J., and Qiao, Y. (2022).  Convmae: Masked convolution meets masked autoencoders.  arXiv preprint arXiv:2205.03892.
  * Ge et al., (2023) Ge, Y., Ge, Y., Zeng, Z., Wang, X., and Shan, Y. (2023).  Planting a seed of vision in large language model.  arXiv preprint arXiv:2307.08041.
  * Geng and Liu, (2023) Geng, X. and Liu, H. (2023).  Openllama: An open reproduction of llama.
  * Geng et al., (2023) Geng, Z., Yang, B., Hang, T., Li, C., Gu, S., Zhang, T., Bao, J., Zhang, Z., Hu, H., Chen, D., et al. (2023).  Instructdiffusion: A generalist modeling interface for vision tasks.  arXiv preprint arXiv:2309.03895.
  * (135) Ghiasi, G., Gu, X., Cui, Y., and Lin, T.-Y. (2022a).  Open-vocabulary image segmentation.  In ECCV.
  * (136) Ghiasi, G., Gu, X., Cui, Y., and Lin, T.-Y. (2022b).  Scaling open-vocabulary image segmentation with image-level labels.  In European Conference on Computer Vision, pages 540–557. Springer.
  * Girdhar et al., (2023) Girdhar, R., El-Nouby, A., Liu, Z., Singh, M., Alwala, K. V., Joulin, A., and Misra, I. (2023).  Imagebind: One embedding space to bind them all.  In CVPR.
  * Girshick, (2015) Girshick, R. (2015).  Fast r-cnn.  In ICCV.
  * Girshick et al., (2015) Girshick, R., Donahue, J., Darrell, T., and Malik, J. (2015).  Region-based convolutional networks for accurate object detection and segmentation.  IEEE transactions on pattern analysis and machine intelligence, 38(1):142–158.
  * Gong et al., (2023) Gong, T., Lyu, C., Zhang, S., Wang, Y., Zheng, M., Zhao, Q., Liu, K., Zhang, W., Luo, P., and Chen, K. (2023).  Multimodal-gpt: A vision and language model for dialogue with humans.  arXiv preprint arXiv:2305.04790.
  * Goodfellow et al., (2020) Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., and Bengio, Y. (2020).  Generative adversarial networks.  Communications of the ACM.
  * Grill et al., (2020) Grill, J.-B., Strub, F., Altché, F., Tallec, C., Richemond, P., Buchatskaya, E., Doersch, C., Avila Pires, B., Guo, Z., Gheshlaghi Azar, M., et al. (2020).  Bootstrap your own latent-a new approach to self-supervised learning.  NeurIPS.
  * Gu et al., (2023) Gu, X., Cui, Y., Huang, J., Rashwan, A., Yang, X., Zhou, X., Ghiasi, G., Kuo, W., Chen, H., Chen, L.-C., et al. (2023).  Dataseg: Taming a universal multi-dataset multi-task segmentation model.  arXiv preprint arXiv:2306.01736.
  * Gu et al., (2021) Gu, X., Lin, T.-Y., Kuo, W., and Cui, Y. (2021).  Open-vocabulary object detection via vision and language knowledge distillation.  arXiv preprint arXiv:2104.13921.
  * Gu et al., (2022) Gu, X., Lin, T.-Y., Kuo, W., and Cui, Y. (2022).  Open-vocabulary object detection via vision and language knowledge distillation.  In ICLR.
  * Gudibande et al., (2023) Gudibande, A., Wallace, E., Snell, C., Geng, X., Liu, H., Abbeel, P., Levine, S., and Song, D. (2023).  The false promise of imitating proprietary llms.  arXiv preprint arXiv:2305.15717.
  * Gunjal et al., (2023) Gunjal, A., Yin, J., and Bas, E. (2023).  Detecting and preventing hallucinations in large vision language models.  arXiv preprint arXiv:2308.06394.
  * (148) Gupta, T., Kamath, A., Kembhavi, A., and Hoiem, D. (2022a).  Towards general purpose vision systems: An end-to-end task-agnostic vision-language architecture.  In CVPR.
  * (149) Gupta, T., Kamath, A., Kembhavi, A., and Hoiem, D. (2022b).  Towards general purpose vision systems: An end-to-end task-agnostic vision-language architecture.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 16399–16409.
  * (150) Gupta, T. and Kembhavi, A. (2022a).  Visual programming: Compositional visual reasoning without training.  arXiv preprint arXiv:2211.11559.
  * (151) Gupta, T. and Kembhavi, A. (2022b).  Visual programming: Compositional visual reasoning without training.  ArXiv, abs/2211.11559.
  * Gupta and Kembhavi, (2023) Gupta, T. and Kembhavi, A. (2023).  Visual programming: Compositional visual reasoning without training.  In CVPR.
  * (153) Gupta, T., Marten, R., Kembhavi, A., and Hoiem, D. (2022c).  Grit: General robust image task benchmark.  arXiv preprint arXiv:2204.13653.
  * Gurari et al., (2018) Gurari, D., Li, Q., Stangl, A. J., Guo, A., Lin, C., Grauman, K., Luo, J., and Bigham, J. P. (2018).  Vizwiz grand challenge: Answering visual questions from blind people.  In CVPR.
  * Gutmann and Hyvärinen, (2010) Gutmann, M. and Hyvärinen, A. (2010).  Noise-contrastive estimation: A new estimation principle for unnormalized statistical models.  In AISTATS.
  * Guu et al., (2020) Guu, K., Lee, K., Tung, Z., Pasupat, P., and Chang, M.-W. (2020).  Realm: Retrieval-augmented language model pre-training.  arXiv preprint arXiv:2002.08909.
  * Hafiz and Bhat, (2020) Hafiz, A. M. and Bhat, G. M. (2020).  A survey on instance segmentation: state of the art.  International journal of multimedia information retrieval.
  * Harley et al., (2022) Harley, A. W., Fang, Z., and Fragkiadaki, K. (2022).  Particle video revisited: Tracking through occlusions using point trajectories.  In European Conference on Computer Vision, pages 59–75. Springer.
  * (159) He, K., Chen, X., Xie, S., Li, Y., Dollár, P., and Girshick, R. (2022a).  Masked autoencoders are scalable vision learners.  In CVPR.
  * He et al., (2020) He, K., Fan, H., Wu, Y., Xie, S., and Girshick, R. (2020).  Momentum contrast for unsupervised visual representation learning.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 9729–9738.
  * He et al., (2017) He, K., Gkioxari, G., Dollár, P., and Girshick, R. (2017).  Mask r-cnn.  In ICCV.
  * He et al., (2010) He, K., Sun, J., and Tang, X. (2010).  Single image haze removal using dark channel prior.  IEEE transactions on pattern analysis and machine intelligence, 33(12):2341–2353.
  * He et al., (2016) He, K., Zhang, X., Ren, S., and Sun, J. (2016).  Deep residual learning for image recognition.  In CVPR.
  * He et al., (2021) He, P., Liu, X., Gao, J., and Chen, W. (2021).  DeBERTa: Decoding-enhanced bert with disentangled attention.  In ICLR.
  * (165) He, R., Sun, S., Yu, X., Xue, C., Zhang, W., Torr, P., Bai, S., and Qi, X. (2022b).  Is synthetic data from generative models ready for image recognition?  arXiv preprint arXiv:2210.07574.
  * Henaff, (2020) Henaff, O. (2020).  Data-efficient image recognition with contrastive predictive coding.  In ICML.
  * Hertz et al., (2022) Hertz, A., Mokady, R., Tenenbaum, J., Aberman, K., Pritch, Y., and Cohen-or, D. (2022).  Prompt-to-prompt image editing with cross-attention control.  In The Eleventh International Conference on Learning Representations.
  * Hjelm et al., (2018) Hjelm, R. D., Fedorov, A., Lavoie-Marchildon, S., Grewal, K., Bachman, P., Trischler, A., and Bengio, Y. (2018).  Learning deep representations by mutual information estimation and maximization.  arXiv preprint arXiv:1808.06670.
  * Ho et al., (2022) Ho, J., Chan, W., Saharia, C., Whang, J., Gao, R., Gritsenko, A., Kingma, D. P., Poole, B., Norouzi, M., Fleet, D. J., et al. (2022).  Imagen video: High definition video generation with diffusion models.  arXiv preprint arXiv:2210.02303.
  * Ho et al., (2020) Ho, J., Jain, A., and Abbeel, P. (2020).  Denoising diffusion probabilistic models.  In NeurIPS.
  * Hoffmann et al., (2022) Hoffmann, J., Borgeaud, S., Mensch, A., Buchatskaya, E., Cai, T., Rutherford, E., Casas, D. d. L., Hendricks, L. A., Welbl, J., Clark, A., et al. (2022).  Training compute-optimal large language models.  arXiv preprint arXiv:2203.15556.
  * Hong et al., (2023) Hong, Y., Zhen, H., Chen, P., Zheng, S., Du, Y., Chen, Z., and Gan, C. (2023).  3d-llm: Injecting the 3d world into large language models.  arXiv preprint arXiv:2307.12981.
  * Hu et al., (2021) Hu, E. J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., and Chen, W. (2021).  Lora: Low-rank adaptation of large language models.  arXiv preprint arXiv:2106.09685.
  * Hu et al., (2016) Hu, R., Rohrbach, M., and Darrell, T. (2016).  Segmentation from natural language expressions.  In European Conference on Computer Vision, pages 108–124. Springer.
  * (175) Hu, R. and Singh, A. (2021a).  Unit: Multimodal multitask learning with a unified transformer.  In ICCV.
  * (176) Hu, R. and Singh, A. (2021b).  Unit: Multimodal multitask learning with a unified transformer.  In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 1439–1449.
  * Hu et al., (2023) Hu, W., Xu, Y., Li, Y., Li, W., Chen, Z., and Tu, Z. (2023).  Bliva: A simple multimodal llm for better handling of text-rich visual questions.  arXiv preprint arXiv:2308.09936.
  * (178) Huang, L., You, S., Zheng, M., Wang, F., Qian, C., and Yamasaki, T. (2022a).  Green hierarchical vision transformer for masked image modeling.  NeurIPS.
  * (179) Huang, R., Li, M., Yang, D., Shi, J., Chang, X., Ye, Z., Wu, Y., Hong, Z., Huang, J., Liu, J., et al. (2023a).  Audiogpt: Understanding and generating speech, music, sound, and talking head.  arXiv preprint arXiv:2304.12995.
  * (180) Huang, S., Dong, L., Wang, W., Hao, Y., Singhal, S., Ma, S., Lv, T., Cui, L., Mohammed, O. K., Liu, Q., et al. (2023b).  Language is not all you need: Aligning perception with language models.  arXiv preprint arXiv:2302.14045.
  * (181) Huang, S., Jiang, Z., Dong, H., Qiao, Y., Gao, P., and Li, H. (2023c).  Instruct2act: Mapping multi-modality instructions to robotic actions with large language model.  arXiv preprint arXiv:2305.11176.
  * (182) Huang, W., Abbeel, P., Pathak, D., and Mordatch, I. (2022b).  Language models as zero-shot planners: Extracting actionable knowledge for embodied agents.  In International Conference on Machine Learning, pages 9118–9147. PMLR.
  * (183) Huang, Y., Meng, Z., Liu, F., Su, Y., Collier, N., and Lu, Y. (2023d).  Sparkles: Unlocking chats across multiple images for multimodal instruction-following models.  arXiv preprint arXiv:2308.16463.
  * Huang et al., (2021) Huang, Z., Zeng, Z., Huang, Y., Liu, B., Fu, D., and Fu, J. (2021).  Seeing out of the box: End-to-end pre-training for vision-language representation learning.  In CVPR.
  * Huang et al., (2020) Huang, Z., Zeng, Z., Liu, B., Fu, D., and Fu, J. (2020).  Pixel-BERT: Aligning image pixels with text by deep multi-modal transformers.  arXiv preprint arXiv:2004.00849.
  * Huynh et al., (2022) Huynh, D., Kuen, J., Lin, Z., Gu, J., and Elhamifar, E. (2022).  Open-vocabulary instance segmentation via robust cross-modal pseudo-labeling.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7020–7031.
  * Ilharco et al., (2021) Ilharco, G., Wortsman, M., Wightman, R., Gordon, C., Carlini, N., Taori, R., Dave, A., Shankar, V., Namkoong, H., Miller, J., Hajishirzi, H., Farhadi, A., and Schmidt, L. (2021).  Openclip.  If you use this software, please cite it as below.
  * Jain et al., (2023) Jain, J., Li, J., Chiu, M. T., Hassani, A., Orlov, N., and Shi, H. (2023).  Oneformer: One transformer to rule universal image segmentation.  In CVPR.
  * Jaiswal et al., (2020) Jaiswal, A., Babu, A. R., Zadeh, M. Z., Banerjee, D., and Makedon, F. (2020).  A survey on contrastive self-supervised learning.  Technologies.
  * Jerripothula et al., (2016) Jerripothula, K. R., Cai, J., and Yuan, J. (2016).  Image co-segmentation via saliency co-fusion.  IEEE Transactions on Multimedia, 18(9):1896–1909.
  * Jia et al., (2021) Jia, C., Yang, Y., Xia, Y., Chen, Y.-T., Parekh, Z., Pham, H., Le, Q. V., Sung, Y., Li, Z., and Duerig, T. (2021).  Scaling up visual and vision-language representation learning with noisy text supervision.  In ICML.
  * Jing and Tian, (2020) Jing, L. and Tian, Y. (2020).  Self-supervised visual feature learning with deep neural networks: A survey.  IEEE transactions on pattern analysis and machine intelligence.
  * Joulin et al., (2010) Joulin, A., Bach, F., and Ponce, J. (2010).  Discriminative clustering for image co-segmentation.  In 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pages 1943–1950. IEEE.
  * Kamath et al., (2021) Kamath, A., Singh, M., LeCun, Y., Synnaeve, G., Misra, I., and Carion, N. (2021).  Mdetr-modulated detection for end-to-end multi-modal understanding.  In ICCV.
  * Kang et al., (2023) Kang, M., Zhu, J.-Y., Zhang, R., Park, J., Shechtman, E., Paris, S., and Park, T. (2023).  Scaling up gans for text-to-image synthesis.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10124–10134.
  * Kaplan et al., (2020) Kaplan, J., McCandlish, S., Henighan, T., Brown, T. B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., and Amodei, D. (2020).  Scaling laws for neural language models.  arXiv preprint arXiv:2001.08361.
  * Kawar et al., (2023) Kawar, B., Zada, S., Lang, O., Tov, O., Chang, H., Dekel, T., Mosseri, I., and Irani, M. (2023).  Imagic: Text-based real image editing with diffusion models.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6007–6017.
  * Kazemzadeh et al., (2014) Kazemzadeh, S., Ordonez, V., Matten, M., and Berg, T. (2014).  Referitgame: Referring to objects in photographs of natural scenes.  In EMNLP.
  * Kim et al., (2021) Kim, W., Son, B., and Kim, I. (2021).  ViLT: Vision-and-language transformer without convolution or region supervision.  In ICML.
  * Kingma and Welling, (2013) Kingma, D. P. and Welling, M. (2013).  Auto-encoding variational bayes.  arXiv preprint arXiv:1312.6114.
  * Kirillov et al., (2019) Kirillov, A., He, K., Girshick, R., Rother, C., and Dollár, P. (2019).  Panoptic segmentation.  In CVPR.
  * Kirillov et al., (2023) Kirillov, A., Mintun, E., Ravi, N., Mao, H., Rolland, C., Gustafson, L., Xiao, T., Whitehead, S., Berg, A. C., Lo, W.-Y., et al. (2023).  Segment anything.  arXiv preprint arXiv:2304.02643.
  * Koh et al., (2023) Koh, J. Y., Fried, D., and Salakhutdinov, R. (2023).  Generating images with multimodal language models.  arXiv preprint arXiv:2305.17216.
  * Kojima et al., (2022) Kojima, T., Gu, S. S., Reid, M., Matsuo, Y., and Iwasawa, Y. (2022).  Large language models are zero-shot reasoners.  arXiv preprint arXiv:2205.11916.
  * Kokkinos, (2017) Kokkinos, I. (2017).  Ubernet: Training a universal convolutional neural network for low-, mid-, and high-level vision using diverse datasets and limited memory.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 6129–6138.
  * Kolesnikov et al., (2020) Kolesnikov, A., Beyer, L., Zhai, X., Puigcerver, J., Yung, J., Gelly, S., and Houlsby, N. (2020).  Big transfer (bit): General visual representation learning.  In ECCV.
  * Kolesnikov et al., (2022) Kolesnikov, A., Pinto, A. S., Beyer, L., Zhai, X., Harmsen, J., and Houlsby, N. (2022).  Uvim: A unified modeling approach for vision with learned guiding codes.  arXiv preprint arXiv:2205.10337.
  * Krizhevsky et al., (2012) Krizhevsky, A., Sutskever, I., and Hinton, G. E. (2012).  Imagenet classification with deep convolutional neural networks.  In NeurIPS.
  * Kumari et al., (2023) Kumari, N., Zhang, B., Zhang, R., Shechtman, E., and Zhu, J.-Y. (2023).  Multi-concept customization of text-to-image diffusion.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1931–1941.
  * Kuo et al., (2022) Kuo, W., Bertsch, F., Li, W., Piergiovanni, A., Saffar, M., and Angelova, A. (2022).  Findit: Generalized localization with natural language queries.  In ECCV.
  * Lai et al., (2023) Lai, X., Tian, Z., Chen, Y., Li, Y., Yuan, Y., Liu, S., and Jia, J. (2023).  Lisa: Reasoning segmentation via large language model.  arXiv preprint arXiv:2308.00692.
  * Lamb et al., (2016) Lamb, A., Dumoulin, V., and Courville, A. (2016).  Discriminative regularization for generative models.  arXiv preprint arXiv:1602.03220.
  * Lambert et al., (2020) Lambert, J., Liu, Z., Sener, O., Hays, J., and Koltun, V. (2020).  Mseg: A composite dataset for multi-domain semantic segmentation.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 2879–2888.
  * Larsen et al., (2016) Larsen, A. B. L., Sønderby, S. K., Larochelle, H., and Winther, O. (2016).  Autoencoding beyond pixels using a learned similarity metric.  In International conference on machine learning, pages 1558–1566. PMLR.
  * Laurençon et al., (2023) Laurençon, H., Saulnier, L., Tronchon, L., Bekman, S., Singh, A., Lozhkov, A., Wang, T., Karamcheti, S., Rush, A. M., Kiela, D., et al. (2023).  Obelisc: An open web-scale filtered dataset of interleaved image-text documents.  arXiv preprint arXiv:2306.16527.
  * Lewis et al., (2020) Lewis, P., Perez, E., Piktus, A., Petroni, F., Karpukhin, V., Goyal, N., Küttler, H., Lewis, M., Yih, W.-t., Rocktäschel, T., et al. (2020).  Retrieval-augmented generation for knowledge-intensive nlp tasks.  In NeurIPS.
  * (217) Li, B., Liu, H., Chen, L., Lee, Y. J., Li, C., and Liu, Z. (2023a).  Benchmarking and analyzing generative data for visual recognition.  arXiv preprint arXiv:2307.13697.
  * (218) Li, B., Wang, R., Wang, G., Ge, Y., Ge, Y., and Shan, Y. (2023b).  Seed-bench: Benchmarking multimodal llms with generative comprehension.  arXiv preprint arXiv:2307.16125.
  * (219) Li, B., Weinberger, K. Q., Belongie, S., Koltun, V., and Ranftl, R. (2022a).  Language-driven semantic segmentation.  In ICLR.
  * (220) Li, B., Zhang, Y., Chen, L., Wang, J., Pu, F., Yang, J., Li, C., and Liu, Z. (2023c).  Mimic-it: Multi-modal in-context instruction tuning.  arXiv preprint arXiv:2306.05425.
  * (221) Li, B., Zhang, Y., Chen, L., Wang, J., Yang, J., and Liu, Z. (2023d).  Otter: A multi-modal model with in-context instruction tuning.  arXiv preprint arXiv:2305.03726.
  * (222) Li, C., Liu, H., Li, L. H., Zhang, P., Aneja, J., Yang, J., Jin, P., Lee, Y. J., Hu, H., Liu, Z., et al. (2022b).  Elevater: A benchmark and toolkit for evaluating language-augmented visual models.  In NeurIPS, Track on Datasets and Benchmarks.
  * (223) Li, C., Wong, C., Zhang, S., Usuyama, N., Liu, H., Yang, J., Naumann, T., Poon, H., and Gao, J. (2023e).  Llava-med: Training a large language-and-vision assistant for biomedicine in one day.  arXiv preprint arXiv:2306.00890.
  * (224) Li, C., Yang, J., Zhang, P., Gao, M., Xiao, B., Dai, X., Yuan, L., and Gao, J. (2021a).  Efficient self-supervised vision transformers for representation learning.  arXiv preprint arXiv:2106.09785.
  * (225) Li, F., Zhang, H., Sun, P., Zou, X., Liu, S., Yang, J., Li, C., Zhang, L., and Gao, J. (2023f).  Semantic-sam: Segment and recognize anything at any granularity.  arXiv preprint arXiv:2307.04767.
  * (226) Li, F., Zhang, H., Zhang, Y.-F., Liu, S., Guo, J., Ni, L. M., Zhang, P., and Zhang, L. (2022c).  Vision-language intelligence: Tasks, representation learning, and large models.  arXiv preprint arXiv:2203.01922.
  * (227) Li, G., Duan, N., Fang, Y., Gong, M., and Jiang, D. (2020a).  Unicoder-vl: A universal encoder for vision and language by cross-modal pre-training.  In AAAI.
  * (228) Li, H., Zhu, J., Jiang, X., Zhu, X., Li, H., Yuan, C., Wang, X., Qiao, Y., Wang, X., Wang, W., et al. (2023g).  Uni-perceiver v2: A generalist model for large-scale vision and vision-language tasks.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2691–2700.
  * (229) Li, J., Li, D., Savarese, S., and Hoi, S. (2023h).  Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models.  arXiv preprint arXiv:2301.12597.
  * (230) Li, J., Li, D., Xiong, C., and Hoi, S. (2022d).  Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation.  In ICML.
  * (231) Li, J., Selvaraju, R. R., Gotmare, A. D., Joty, S., Xiong, C., and Hoi, S. (2021b).  Align before fuse: Vision and language representation learning with momentum distillation.  In NeurIPS.
  * (232) Li, K., He, Y., Wang, Y., Li, Y., Wang, W., Luo, P., Wang, Y., Wang, L., and Qiao, Y. (2023i).  Videochat: Chat-centric video understanding.  arXiv preprint arXiv:2305.06355.
  * (233) Li, L., Gan, Z., Cheng, Y., and Liu, J. (2019a).  Relation-aware graph attention network for visual question answering.  In ICCV.
  * (234) Li, L., Yin, Y., Li, S., Chen, L., Wang, P., Ren, S., Li, M., Yang, Y., Xu, J., Sun, X., et al. (2023j).  M3it: A large-scale dataset towards multi-modal multilingual instruction tuning.  arXiv preprint arXiv:2306.04387.
  * (235) Li, L. H., Yatskar, M., Yin, D., Hsieh, C.-J., and Chang, K.-W. (2019b).  VisualBERT: A simple and performant baseline for vision and language.  arXiv preprint arXiv:1908.03557.
  * (236) Li, L. H., Zhang, P., Zhang, H., Yang, J., Li, C., Zhong, Y., Wang, L., Yuan, L., Zhang, L., Hwang, J.-N., et al. (2022e).  Grounded language-image pre-training.  CVPR.
  * (237) Li, L. H., Zhang, P., Zhang, H., Yang, J., Li, C., Zhong, Y., Wang, L., Yuan, L., Zhang, L., Hwang, J.-N., et al. (2022f).  Grounded language-image pre-training.  In CVPR.
  * (238) Li, M., Song, F., Yu, B., Yu, H., Li, Z., Huang, F., and Li, Y. (2023k).  Api-bank: A benchmark for tool-augmented llms.  arXiv preprint arXiv:2304.08244.
  * Li and Tajbakhsh, (2023) Li, S. and Tajbakhsh, N. (2023).  Scigraphqa: A large-scale synthetic multi-turn question-answering dataset for scientific graphs.  arXiv preprint arXiv:2308.03349.
  * (240) Li, X., Yin, X., Li, C., Zhang, P., Hu, X., Zhang, L., Wang, L., Hu, H., Dong, L., Wei, F., Choi, Y., and Gao, J. (2020b).  Oscar: Object-semantics aligned pre-training for vision-language tasks.  In ECCV.
  * Li and Liang, (2021) Li, X. L. and Liang, P. (2021).  Prefix-tuning: Optimizing continuous prompts for generation.  arXiv preprint arXiv:2101.00190.
  * (242) Li, Y., Du, Y., Zhou, K., Wang, J., Zhao, W. X., and Wen, J.-R. (2023l).  Evaluating object hallucination in large vision-language models.  arXiv preprint arXiv:2305.10355.
  * (243) Li, Y., Fan, H., Hu, R., Feichtenhofer, C., and He, K. (2023m).  Scaling language-image pre-training via masking.  In CVPR.
  * (244) Li, Y., Liang, F., Zhao, L., Cui, Y., Ouyang, W., Shao, J., Yu, F., and Yan, J. (2022g).  Supervision exists everywhere: A data efficient contrastive language-image pre-training paradigm.  In ICLR.
  * (245) Li, Y., Liu, H., Wu, Q., Mu, F., Yang, J., Gao, J., Li, C., and Lee, Y. J. (2023n).  Gligen: Open-set grounded text-to-image generation.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 22511–22521.
  * (246) Li, Y., Zhang, C., Yu, G., Wang, Z., Fu, B., Lin, G., Shen, C., Chen, L., and Wei, Y. (2023o).  Stablellava: Enhanced visual instruction tuning with synthesized image-dialogue data.  arXiv preprint arXiv:2308.10253.
  * (247) Liang, F., Wu, B., Dai, X., Li, K., Zhao, Y., Zhang, H., Zhang, P., Vajda, P., and Marculescu, D. (2023a).  Open-vocabulary semantic segmentation with mask-adapted clip.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7061–7070.
  * (248) Liang, Y., Wu, C., Song, T., Wu, W., Xia, Y., Liu, Y., Ou, Y., Lu, S., Ji, L., Mao, S., et al. (2023b).  Taskmatrix. ai: Completing tasks by connecting foundation models with millions of apis.  arXiv preprint arXiv:2303.16434.
  * Lin et al., (2017) Lin, T.-Y., Goyal, P., Girshick, R., He, K., and Dollár, P. (2017).  Focal loss for dense object detection.  In Proceedings of the IEEE international conference on computer vision, pages 2980–2988.
  * Lin et al., (2014) Lin, T.-Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., and Zitnick, C. L. (2014).  Microsoft coco: Common objects in context.  In ECCV.
  * Lin et al., (2023) Lin, Y., Wu, H., Wang, R., Lu, H., Lin, X., Xiong, H., and Wang, L. (2023).  Towards language-guided interactive 3d generation: Llms as layout interpreter with generative feedback.  arXiv preprint arXiv:2305.15808.
  * Liu et al., (2017) Liu, C., Lin, Z., Shen, X., Yang, J., Lu, X., and Yuille, A. (2017).  Recurrent multimodal interaction for referring image segmentation.  In Proceedings of the IEEE International Conference on Computer Vision, pages 1271–1280.
  * (253) Liu, F., Lin, K., Li, L., Wang, J., Yacoob, Y., and Wang, L. (2023a).  Aligning large multi-modal model with robust instruction tuning.  arXiv preprint arXiv:2306.14565.
  * (254) Liu, H., Jiang, X., Li, X., Guo, A., Hu, Y., Jiang, D., and Ren, B. (2023b).  The devil is in the frequency: Geminated gestalt autoencoder for self-supervised visual pre-training.  In AAAI.
  * (255) Liu, H., Li, C., Wu, Q., and Lee, Y. J. (2023c).  Visual instruction tuning.  arXiv preprint arXiv:2304.08485.
  * (256) Liu, H., Son, K., Yang, J., Liu, C., Gao, J., Lee, Y. J., and Li, C. (2023d).  Learning customized visual models with retrieval-augmented knowledge.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition.
  * (257) Liu, J., Ding, H., Cai, Z., Zhang, Y., Satzoda, R. K., Mahadevan, V., and Manmatha, R. (2023e).  Polyformer: Referring image segmentation as sequential polygon generation.
  * (258) Liu, N., Li, S., Du, Y., Torralba, A., and Tenenbaum, J. B. (2022a).  Compositional visual generation with composable diffusion models.  In European Conference on Computer Vision, pages 423–439. Springer.
  * (259) Liu, S., Fan, L., Johns, E., Yu, Z., Xiao, C., and Anandkumar, A. (2023f).  Prismer: A vision-language model with an ensemble of experts.
  * (260) Liu, S., Ye, J., and Wang, X. (2023g).  Any-to-any style transfer.  arXiv preprint arXiv:2304.09728.
  * (261) Liu, S., Zeng, Z., Ren, T., Li, F., Zhang, H., Yang, J., Li, C., Yang, J., Su, H., Zhu, J., et al. (2023h).  Grounding dino: Marrying dino with grounded pre-training for open-set object detection.  arXiv preprint arXiv:2303.05499.
  * Liu et al., (2016) Liu, W., Anguelov, D., Erhan, D., Szegedy, C., Reed, S., Fu, C.-Y., and Berg, A. C. (2016).  Ssd: Single shot multibox detector.  In Computer Vision–ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11–14, 2016, Proceedings, Part I 14, pages 21–37. Springer.
  * Liu and Zuo, (2023) Liu, W. and Zuo, Y. (2023).  Stone needle: A general multimodal large-scale model framework towards healthcare.  arXiv preprint arXiv:2306.16034.
  * (264) Liu, X., Zhu, Z., Liu, H., Yuan, Y., Cui, M., Huang, Q., Liang, J., Cao, Y., Kong, Q., Plumbley, M. D., et al. (2023i).  Wavjourney: Compositional audio creation with large language models.  arXiv preprint arXiv:2307.14335.
  * (265) Liu, Y., Duan, H., Zhang, Y., Li, B., Zhang, S., Zhao, W., Yuan, Y., Wang, J., He, C., Liu, Z., et al. (2023j).  Mmbench: Is your multi-modal model an all-around player?  arXiv preprint arXiv:2307.06281.
  * (266) Liu, Y., Li, Z., Li, H., Yu, W., Huang, M., Peng, D., Liu, M., Chen, M., Li, C., Jin, L., et al. (2023k).  On the hidden mystery of ocr in large multimodal models.  arXiv preprint arXiv:2305.07895.
  * Liu et al., (2019) Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., and Stoyanov, V. (2019).  Roberta: A robustly optimized bert pretraining approach.  arXiv preprint arXiv:1907.11692.
  * (268) Liu, Z., He, Y., Wang, W., Wang, W., Wang, Y., Chen, S., Zhang, Q., Yang, Y., Li, Q., Yu, J., et al. (2023l).  Internchat: Solving vision-centric tasks by interacting with chatbots beyond language.  arXiv preprint arXiv:2305.05662.
  * Liu et al., (2021) Liu, Z., Lin, Y., Cao, Y., Hu, H., Wei, Y., Zhang, Z., Lin, S., and Guo, B. (2021).  Swin transformer: Hierarchical vision transformer using shifted windows.  In ICCV.
  * (270) Liu, Z., Mao, H., Wu, C.-Y., Feichtenhofer, C., Darrell, T., and Xie, S. (2022b).  A convnet for the 2020s.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 11976–11986.
  * Long et al., (2022) Long, A., Yin, W., Ajanthan, T., Nguyen, V., Purkait, P., Garg, R., Blair, A., Shen, C., and van den Hengel, A. (2022).  Retrieval augmented classification for long-tail visual recognition.  In CVPR.
  * Long et al., (2015) Long, J., Shelhamer, E., and Darrell, T. (2015).  Fully convolutional networks for semantic segmentation.  In CVPR.
  * (273) Lu, C.-Z., Jin, X., Hou, Q., Liew, J. H., Cheng, M.-M., and Feng, J. (2023a).  Delving deeper into data scaling in masked image modeling.  arXiv preprint arXiv:2305.15248.
  * Lu et al., (2019) Lu, J., Batra, D., Parikh, D., and Lee, S. (2019).  Vilbert: Pretraining task-agnostic visiolinguistic representations for vision-and-language tasks.  In NeurIPS.
  * (275) Lu, J., Clark, C., Zellers, R., Mottaghi, R., and Kembhavi, A. (2022a).  Unified-io: A unified model for vision, language, and multi-modal tasks.  arXiv preprint arXiv:2206.08916.
  * Lu et al., (2020) Lu, J., Goswami, V., Rohrbach, M., Parikh, D., and Lee, S. (2020).  12-in-1: Multi-task vision and language representation learning.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10437–10446.
  * (277) Lu, P., Mishra, S., Xia, T., Qiu, L., Chang, K.-W., Zhu, S.-C., Tafjord, O., Clark, P., and Kalyan, A. (2022b).  Learn to explain: Multimodal reasoning via thought chains for science question answering.  Advances in Neural Information Processing Systems.
  * (278) Lu, P., Peng, B., Cheng, H., Galley, M., Chang, K.-W., Wu, Y. N., Zhu, S.-C., and Gao, J. (2023b).  Chameleon: Plug-and-play compositional reasoning with large language models.  arXiv preprint arXiv:2304.09842.
  * (279) Lu, Q., Kuen, J., Tiancheng, S., Jiuxiang, G., Weidong, G., Jiaya, J., Zhe, L., and Ming-Hsuan, Y. (2023c).  High-quality entity segmentation.  In ICCV.
  * (280) Lu, Y., Li, C., Liu, H., Yang, J., Gao, J., and Shen, Y. (2023d).  An empirical study of scaling instruction-tuned large multimodal models.  arXiv preprint.
  * Lüddecke and Ecker, (2022) Lüddecke, T. and Ecker, A. (2022).  Image segmentation using text and image prompts.  In CVPR.
  * (282) Luo, G., Zhou, Y., Ren, T., Chen, S., Sun, X., and Ji, R. (2023a).  Cheap and quick: Efficient vision-language instruction tuning for large language models.  arXiv preprint arXiv:2305.15023.
  * (283) Luo, H., Bao, J., Wu, Y., He, X., and Li, T. (2023b).  Segclip: Patch aggregation with learnable centers for open-vocabulary semantic segmentation.  In International Conference on Machine Learning, pages 23033–23044. PMLR.
  * (284) Luo, R., Zhao, Z., Yang, M., Dong, J., Qiu, M., Lu, P., Wang, T., and Wei, Z. (2023c).  Valley: Video assistant with large language model enhanced ability.  arXiv preprint arXiv:2306.07207.
  * Luo et al., (2021) Luo, W., Xing, J., Milan, A., Zhang, X., Liu, W., and Kim, T.-K. (2021).  Multiple object tracking: A literature review.  Artificial intelligence, 293:103448.
  * Ma and Wang, (2023) Ma, J. and Wang, B. (2023).  Segment anything in medical images.  arXiv preprint arXiv:2304.12306.
  * Ma et al., (2023) Ma, Z., Hong, X., and Shangguan, Q. (2023).  Can sam count anything? an empirical study on sam counting.  arXiv preprint arXiv:2304.10817.
  * Mao et al., (2016) Mao, J., Huang, J., Toshev, A., Camburu, O., Yuille, A. L., and Murphy, K. (2016).  Generation and comprehension of unambiguous object descriptions.  In CVPR.
  * Margffoy-Tuay et al., (2018) Margffoy-Tuay, E., Pérez, J. C., Botero, E., and Arbeláez, P. (2018).  Dynamic multimodal instance segmentation guided by natural language queries.  In Proceedings of the European Conference on Computer Vision (ECCV), pages 630–645.
  * Marino et al., (2021) Marino, K., Chen, X., Parikh, D., Gupta, A., and Rohrbach, M. (2021).  Krisp: Integrating implicit and symbolic knowledge for open-domain knowledge-based vqa.  In CVPR.
  * Mazumder et al., (2022) Mazumder, M., Banbury, C., Yao, X., Karlaš, B., Rojas, W. G., Diamos, S., Diamos, G., He, L., Kiela, D., Jurado, D., et al. (2022).  Dataperf: Benchmarks for data-centric ai development.  arXiv preprint arXiv:2207.10062.
  * McGuinness and O’connor, (2010) McGuinness, K. and O’connor, N. E. (2010).  A comparative evaluation of interactive segmentation algorithms.  Pattern Recognition, 43(2):434–444.
  * Meng et al., (2021) Meng, C., He, Y., Song, Y., Song, J., Wu, J., Zhu, J.-Y., and Ermon, S. (2021).  Sdedit: Guided image synthesis and editing with stochastic differential equations.  In International Conference on Learning Representations.
  * Mertan et al., (2022) Mertan, A., Duff, D. J., and Unal, G. (2022).  Single image depth estimation: An overview.  Digital Signal Processing, 123:103441.
  * Miech et al., (2019) Miech, A., Zhukov, D., Alayrac, J.-B., Tapaswi, M., Laptev, I., and Sivic, J. (2019).  Howto100m: Learning a text-video embedding by watching hundred million narrated video clips.  In ICCV.
  * Mikolov et al., (2013) Mikolov, T., Chen, K., Corrado, G., and Dean, J. (2013).  Efficient estimation of word representations in vector space.  arXiv preprint arXiv:1301.3781.
  * Minderer et al., (2022) Minderer, M., Gritsenko, A., Stone, A., Neumann, M., Weissenborn, D., Dosovitskiy, A., Mahendran, A., Arnab, A., Dehghani, M., Shen, Z., Wang, X., Zhai, X., Kipf, T., and Houlsby, N. (2022).  Simple open-vocabulary object detection with vision transformers.
  * Misra and Maaten, (2020) Misra, I. and Maaten, L. v. d. (2020).  Self-supervised learning of pretext-invariant representations.  In CVPR.
  * Misra et al., (2016) Misra, I., Shrivastava, A., Gupta, A., and Hebert, M. (2016).  Cross-stitch networks for multi-task learning.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3994–4003.
  * Monajatipoor et al., (2023) Monajatipoor, M., Li, L. H., Rouhsedaghat, M., Yang, L. F., and Chang, K.-W. (2023).  Metavl: Transferring in-context learning ability from language models to vision-language models.  arXiv preprint arXiv:2306.01311.
  * Moor et al., (2023) Moor, M., Huang, Q., Wu, S., Yasunaga, M., Zakka, C., Dalmia, Y., Reis, E. P., Rajpurkar, P., and Leskovec, J. (2023).  Med-flamingo: a multimodal medical few-shot learner.  arXiv preprint arXiv:2307.15189.
  * Mortensen and Barrett, (1998) Mortensen, E. N. and Barrett, W. A. (1998).  Interactive segmentation with intelligent scissors.  Graphical models and image processing, 60(5):349–384.
  * Mottaghi et al., (2014) Mottaghi, R., Chen, X., Liu, X., Cho, N.-G., Lee, S.-W., Fidler, S., Urtasun, R., and Yuille, A. (2014).  The role of context for object detection and semantic segmentation in the wild.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 891–898.
  * Mou et al., (2023) Mou, C., Wang, X., Xie, L., Zhang, J., Qi, Z., Shan, Y., and Qie, X. (2023).  T2i-adapter: Learning adapters to dig out more controllable ability for text-to-image diffusion models.  arXiv preprint arXiv:2302.08453.
  * Mu et al., (2021) Mu, N., Kirillov, A., Wagner, D., and Xie, S. (2021).  Slip: Self-supervision meets language-image pre-training.  arXiv preprint arXiv:2112.12750.
  * Mu et al., (2023) Mu, Y., Zhang, Q., Hu, M., Wang, W., Ding, M., Jin, J., Wang, B., Dai, J., Qiao, Y., and Luo, P. (2023).  Embodiedgpt: Vision-language pre-training via embodied chain of thought.  arXiv preprint arXiv:2305.15021.
  * Musgrave et al., (2020) Musgrave, K., Belongie, S., and Lim, S.-N. (2020).  A metric learning reality check.  In ECCV.
  * Nagaraja et al., (2016) Nagaraja, V. K., Morariu, V. I., and Davis, L. S. (2016).  Modeling context between objects for referring expression understanding.  In ECCV.
  * Nakano et al., (2021) Nakano, R., Hilton, J., Balaji, S., Wu, J., Ouyang, L., Kim, C., Hesse, C., Jain, S., Kosaraju, V., Saunders, W., et al. (2021).  Webgpt: Browser-assisted question-answering with human feedback.  arXiv preprint arXiv:2112.09332.
  * Nguyen et al., (2022) Nguyen, T., Ilharco, G., Wortsman, M., Oh, S., and Schmidt, L. (2022).  Quality not quantity: On the interaction between dataset design and robustness of clip.  NeurIPS.
  * Ning et al., (2023) Ning, J., Li, C., Zhang, Z., Geng, Z., Dai, Q., He, K., and Hu, H. (2023).  All in tokens: Unifying output space of visual tasks via soft token.  arXiv preprint arXiv:2301.02229.
  * Oord et al., (2018) Oord, A. v. d., Li, Y., and Vinyals, O. (2018).  Representation learning with contrastive predictive coding.  arXiv preprint arXiv:1807.03748.
  * Oord et al., (2017) Oord, A. v. d., Vinyals, O., and Kavukcuoglu, K. (2017).  Neural discrete representation learning.  arXiv preprint arXiv:1711.00937.
  * OpenAI, (2022) OpenAI (2022).  ChatGPT.  https://openai.com/blog/chatgpt/.
  * (315) OpenAI (2023a).  GPT-4 technical report.  https://arxiv.org/abs/2303.08774.
  * (316) OpenAI (2023b).  Gpt-4 technical report.
  * Oquab et al., (2023) Oquab, M., Darcet, T., Moutakanni, T., Vo, H., Szafraniec, M., Khalidov, V., Fernandez, P., Haziza, D., Massa, F., El-Nouby, A., et al. (2023).  Dinov2: Learning robust visual features without supervision.  arXiv preprint arXiv:2304.07193.
  * Ordonez et al., (2011) Ordonez, V., Kulkarni, G., and Berg, T. (2011).  Im2text: Describing images using 1 million captioned photographs.  In NeurIPS.
  * Ouyang et al., (2022) Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. (2022).  Training language models to follow instructions with human feedback.  Advances in Neural Information Processing Systems, 35:27730–27744.
  * Ozbulak et al., (2023) Ozbulak, U., Lee, H. J., Boga, B., Anzaku, E. T., Park, H., Van Messem, A., De Neve, W., and Vankerschaver, J. (2023).  Know your self-supervised learning: A survey on image-based generative and discriminative training.  arXiv preprint arXiv:2305.13689.
  * Paranjape et al., (2023) Paranjape, B., Lundberg, S., Singh, S., Hajishirzi, H., Zettlemoyer, L., and Ribeiro, M. T. (2023).  Art: Automatic multi-step reasoning and tool-use for large language models.  arXiv preprint arXiv:2303.09014.
  * Patil et al., (2023) Patil, S. G., Zhang, T., Wang, X., and Gonzalez, J. E. (2023).  Gorilla: Large language model connected with massive apis.  arXiv preprint arXiv:2305.15334.
  * Penedo et al., (2023) Penedo, G., Malartic, Q., Hesslow, D., Cojocaru, R., Cappelli, A., Alobeidli, H., Pannier, B., Almazrouei, E., and Launay, J. (2023).  The RefinedWeb dataset for Falcon LLM: outperforming curated corpora with web data, and web data only.  arXiv preprint arXiv:2306.01116.
  * (324) Peng, B., Li, C., He, P., Galley, M., and Gao, J. (2023a).  Instruction tuning with GPT-4.  arXiv preprint arXiv:2304.03277.
  * (325) Peng, Z., Dong, L., Bao, H., Ye, Q., and Wei, F. (2022a).  Beit v2: Masked image modeling with vector-quantized visual tokenizers.  arXiv preprint arXiv:2208.06366.
  * (326) Peng, Z., Dong, L., Bao, H., Ye, Q., and Wei, F. (2022b).  A unified view of masked image modeling.  arXiv preprint arXiv:2210.10615.
  * (327) Peng, Z., Wang, W., Dong, L., Hao, Y., Huang, S., Ma, S., and Wei, F. (2023b).  Kosmos-2: Grounding multimodal large language models to the world.  arXiv preprint arXiv:2306.14824.
  * Peters et al., (2019) Peters, M. E., Neumann, M., Logan IV, R. L., Schwartz, R., Joshi, V., Singh, S., and Smith, N. A. (2019).  Knowledge enhanced contextual word representations.  arXiv preprint arXiv:1909.04164.
  * Pham et al., (2021) Pham, H., Dai, Z., Ghiasi, G., Liu, H., Yu, A. W., Luong, M.-T., Tan, M., and Le, Q. V. (2021).  Combined scaling for zero-shot transfer learning.  arXiv preprint arXiv:2111.10050.
  * Pi et al., (2023) Pi, R., Gao, J., Diao, S., Pan, R., Dong, H., Zhang, J., Yao, L., Han, J., Xu, H., and Zhang, L. K. T. (2023).  Detgpt: Detect what you need via reasoning.  arXiv preprint arXiv:2305.14167.
  * Plummer et al., (2015) Plummer, B. A., Wang, L., Cervantes, C. M., Caicedo, J. C., Hockenmaier, J., and Lazebnik, S. (2015).  Flickr30k entities: Collecting region-to-phrase correspondences for richer image-to-sentence models.  In ICCV.
  * Pont-Tuset et al., (2020) Pont-Tuset, J., Uijlings, J., Changpinyo, S., Soricut, R., and Ferrari, V. (2020).  Connecting vision and language with localized narratives.  In ECCV.
  * Qian et al., (2023) Qian, C., Han, C., Fung, Y. R., Qin, Y., Liu, Z., and Ji, H. (2023).  Creator: Disentangling abstract and concrete reasonings of large language models through tool creation.  arXiv preprint arXiv:2305.14318.
  * Qian et al., (2022) Qian, R., Li, Y., Xu, Z., Yang, M.-H., Belongie, S., and Cui, Y. (2022).  Multimodal open-vocabulary video classification via pre-trained vision and language models.  arXiv preprint arXiv:2207.07646.
  * (335) Qin, C., Zhang, S., Yu, N., Feng, Y., Yang, X., Zhou, Y., Wang, H., Niebles, J. C., Xiong, C., Savarese, S., et al. (2023a).  Unicontrol: A unified diffusion model for controllable visual generation in the wild.  arXiv preprint arXiv:2305.11147.
  * (336) Qin, J., Wu, J., Yan, P., Li, M., Yuxi, R., Xiao, X., Wang, Y., Wang, R., Wen, S., Pan, X., and Wang, X. (2023b).  Freeseg: Unified, universal and open-vocabulary image segmentation.
  * Radford et al., (2021) Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. (2021).  Learning transferable visual models from natural language supervision.  In ICML.
  * Radford et al., (2019) Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., Sutskever, I., et al. (2019).  Language models are unsupervised multitask learners.  OpenAI blog.
  * Raffel et al., (2020) Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., and Liu, P. J. (2020).  Exploring the limits of transfer learning with a unified text-to-text transformer.  JMLR.
  * Rahman et al., (2020) Rahman, S., Khan, S., and Barnes, N. (2020).  Improved visual-semantic alignment for zero-shot object detection.  In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 11932–11939.
  * Rajič et al., (2023) Rajič, F., Ke, L., Tai, Y.-W., Tang, C.-K., Danelljan, M., and Yu, F. (2023).  Segment anything meets point tracking.  arXiv preprint arXiv:2307.01197.
  * Ramesh et al., (2022) Ramesh, A., Dhariwal, P., Nichol, A., Chu, C., and Chen, M. (2022).  Hierarchical text-conditional image generation with clip latents.  arXiv preprint arXiv:2204.06125.
  * (343) Ramesh, A., Pavlov, M., Goh, G., Gray, S., Voss, C., Radford, A., Chen, M., and Sutskever, I. (2021a).  Zero-Shot Text-to-Image Generation.  In ICML.
  * (344) Ramesh, A., Pavlov, M., Goh, G., Gray, S., Voss, C., Radford, A., Chen, M., and Sutskever, I. (2021b).  Zero-shot text-to-image generation.  In International Conference on Machine Learning, pages 8821–8831. PMLR.
  * Rao et al., (2022) Rao, Y., Zhao, W., Chen, G., Tang, Y., Zhu, Z., Huang, G., Zhou, J., and Lu, J. (2022).  Denseclip: Language-guided dense prediction with context-aware prompting.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18082–18091.
  * Razavi et al., (2019) Razavi, A., Van den Oord, A., and Vinyals, O. (2019).  Generating diverse high-fidelity images with vq-vae-2.  In NeurIPS.
  * Redmon et al., (2016) Redmon, J., Divvala, S., Girshick, R., and Farhadi, A. (2016).  You only look once: Unified, real-time object detection.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 779–788.
  * Reed et al., (2022) Reed, S., Zolna, K., Parisotto, E., Colmenarejo, S. G., Novikov, A., Barth-Maron, G., Gimenez, M., Sulsky, Y., Kay, J., Springenberg, J. T., et al. (2022).  A generalist agent.  arXiv preprint arXiv:2205.06175.
  * Ren et al., (2015) Ren, S., He, K., Girshick, R., and Sun, J. (2015).  Faster r-cnn: Towards real-time object detection with region proposal networks.  In NeurIPS.
  * Ridnik et al., (2021) Ridnik, T., Ben-Baruch, E., Noy, A., and Zelnik-Manor, L. (2021).  Imagenet-21k pretraining for the masses.  arXiv preprint arXiv:2104.10972.
  * Rombach et al., (2021) Rombach, R., Blattmann, A., Lorenz, D., Esser, P., and Ommer, B. (2021).  High-resolution image synthesis with latent diffusion models.
  * Rombach et al., (2022) Rombach, R., Blattmann, A., Lorenz, D., Esser, P., and Ommer, B. (2022).  High-resolution image synthesis with latent diffusion models.  In CVPR.
  * Ronneberger et al., (2015) Ronneberger, O., Fischer, P., and Brox, T. (2015).  U-net: Convolutional networks for biomedical image segmentation.  In Medical Image Computing and Computer-Assisted Intervention–MICCAI 2015: 18th International Conference, Munich, Germany, October 5-9, 2015, Proceedings, Part III 18, pages 234–241. Springer.
  * Roy et al., (2023) Roy, S., Wald, T., Koehler, G., Rokuss, M. R., Disch, N., Holzschuh, J., Zimmerer, D., and Maier-Hein, K. H. (2023).  Sam. md: Zero-shot medical image segmentation capabilities of the segment anything model.  arXiv preprint arXiv:2304.05396.
  * Ruan and Jin, (2022) Ruan, L. and Jin, Q. (2022).  Survey: Transformer based video-language pre-training.  AI Open.
  * Ruiz et al., (2023) Ruiz, N., Li, Y., Jampani, V., Pritch, Y., Rubinstein, M., and Aberman, K. (2023).  Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 22500–22510.
  * Russakovsky et al., (2015) Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., et al. (2015).  Imagenet large scale visual recognition challenge.  IJCV.
  * Saharia et al., (2022) Saharia, C., Chan, W., Saxena, S., Li, L., Whang, J., Denton, E., Ghasemipour, S. K. S., Ayan, B. K., Mahdavi, S. S., Lopes, R. G., et al. (2022).  Photorealistic text-to-image diffusion models with deep language understanding.  arXiv preprint arXiv:2205.11487.
  * Sariyildiz et al., (2020) Sariyildiz, M. B., Perez, J., and Larlus, D. (2020).  Learning visual representations with caption annotations.  In ECCV.
  * Schick et al., (2023) Schick, T., Dwivedi-Yu, J., Dessì, R., Raileanu, R., Lomeli, M., Zettlemoyer, L., Cancedda, N., and Scialom, T. (2023).  Toolformer: Language models can teach themselves to use tools.  arXiv preprint arXiv:2302.04761.
  * Schuhmann et al., (2022) Schuhmann, C., Beaumont, R., Vencu, R., Gordon, C., Wightman, R., Cherti, M., Coombes, T., Katta, A., Mullis, C., Wortsman, M., et al. (2022).  Laion-5b: An open large-scale dataset for training next generation image-text models.  NeurIPS.
  * Schuhmann et al., (2021) Schuhmann, C., Vencu, R., Beaumont, R., Kaczmarczyk, R., Mullis, C., Katta, A., Coombes, T., Jitsev, J., and Komatsuzaki, A. (2021).  Laion-400m: Open dataset of clip-filtered 400 million image-text pairs.  arXiv preprint arXiv:2111.02114.
  * Schwenk et al., (2022) Schwenk, D., Khandelwal, A., Clark, C., Marino, K., and Mottaghi, R. (2022).  A-okvqa: A benchmark for visual question answering using world knowledge.  arXiv preprint arXiv:2206.01718.
  * Sennrich et al., (2016) Sennrich, R., Haddow, B., and Birch, A. (2016).  Neural machine translation of rare words with subword units.  In ACL.
  * Shao et al., (2019) Shao, S., Li, Z., Zhang, T., Peng, C., Yu, G., Zhang, X., Li, J., and Sun, J. (2019).  Objects365: A large-scale, high-quality dataset for object detection.  In ICCV.
  * Shao et al., (2023) Shao, W., Hu, Y., Gao, P., Lei, M., Zhang, K., Meng, F., Xu, P., Huang, S., Li, H., Qiao, Y., et al. (2023).  Tiny lvlm-ehub: Early multimodal experiments with bard.  arXiv preprint arXiv:2308.03729.
  * ShareGPT, (2023) ShareGPT (2023).  https://sharegpt.com/.
  * Sharma et al., (2018) Sharma, P., Ding, N., Goodman, S., and Soricut, R. (2018).  Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning.  In ACL.
  * (369) Shen, Q., Yang, X., and Wang, X. (2023a).  Anything-3d: Towards single-view anything reconstruction in the wild.  arXiv preprint arXiv:2304.10261.
  * (370) Shen, S., Li, C., Hu, X., Xie, Y., Yang, J., Zhang, P., Rohrbach, A., Gan, Z., Wang, L., Yuan, L., et al. (2022a).  K-lite: Learning transferable visual models with external knowledge.  In NeurIPS.
  * (371) Shen, S., Li, L. H., Tan, H., Bansal, M., Rohrbach, A., Chang, K.-W., Yao, Z., and Keutzer, K. (2022b).  How much can clip benefit vision-and-language tasks?  In ICLR.
  * (372) Shen, Y., Song, K., Tan, X., Li, D., Lu, W., and Zhuang, Y. (2023b).  Hugginggpt: Solving ai tasks with chatgpt and its friends in huggingface.  arXiv preprint arXiv:2303.17580.
  * Sheynin et al., (2022) Sheynin, S., Ashual, O., Polyak, A., Singer, U., Gafni, O., Nachmani, E., and Taigman, Y. (2022).  Knn-diffusion: Image generation via large-scale retrieval.  arXiv preprint arXiv:2204.02849.
  * (374) Shi, J., Xiong, W., Lin, Z., and Jung, H. J. (2023a).  Instantbooth: Personalized text-to-image generation without test-time finetuning.  arXiv preprint arXiv:2304.03411.
  * (375) Shi, P., Qiu, J., Abaxi, S. M. D., Wei, H., Lo, F. P.-W., and Yuan, W. (2023b).  Generalist vision foundation models for medical imaging: A case study of segment anything model on zero-shot medical segmentation.  Diagnostics.
  * Shin et al., (2020) Shin, T., Razeghi, Y., Logan IV, R. L., Wallace, E., and Singh, S. (2020).  Autoprompt: Eliciting knowledge from language models with automatically generated prompts.  arXiv preprint arXiv:2010.15980.
  * Sidorov et al., (2020) Sidorov, O., Hu, R., Rohrbach, M., and Singh, A. (2020).  Textcaps: a dataset for image captioning with reading comprehension.  In ECCV.
  * Silberman et al., (2012) Silberman, N., Hoiem, D., Kohli, P., and Fergus, R. (2012).  Indoor segmentation and support inference from rgbd images.  In Computer Vision–ECCV 2012: 12th European Conference on Computer Vision, Florence, Italy, October 7-13, 2012, Proceedings, Part V 12, pages 746–760. Springer.
  * Singer et al., (2022) Singer, U., Polyak, A., Hayes, T., Yin, X., An, J., Zhang, S., Hu, Q., Yang, H., Ashual, O., Gafni, O., et al. (2022).  Make-a-video: Text-to-video generation without text-video data.  arXiv preprint arXiv:2209.14792.
  * (380) Singh, A., Hu, R., Goswami, V., Couairon, G., Galuba, W., Rohrbach, M., and Kiela, D. (2022a).  Flava: A foundational language and vision alignment model.  In CVPR.
  * Singh et al., (2023) Singh, M., Duval, Q., Alwala, K. V., Fan, H., Aggarwal, V., Adcock, A., Joulin, A., Dollár, P., Feichtenhofer, C., Girshick, R., et al. (2023).  The effectiveness of mae pre-pretraining for billion-scale pretraining.  arXiv preprint arXiv:2303.13496.
  * (382) Singh, M., Gustafson, L., Adcock, A., de Freitas Reis, V., Gedik, B., Kosaraju, R. P., Mahajan, D., Girshick, R., Dollár, P., and Van Der Maaten, L. (2022b).  Revisiting weakly supervised pre-training of visual perception models.  In CVPR.
  * Sohl-Dickstein et al., (2015) Sohl-Dickstein, J., Weiss, E., Maheswaranathan, N., and Ganguli, S. (2015).  Deep unsupervised learning using nonequilibrium thermodynamics.  In International conference on machine learning, pages 2256–2265. PMLR.
  * Song and Ermon, (2020) Song, Y. and Ermon, S. (2020).  Improved techniques for training score-based generative models.  Advances in neural information processing systems, 33:12438–12448.
  * Song et al., (2023) Song, Y., Xiong, W., Zhu, D., Li, C., Wang, K., Tian, Y., and Li, S. (2023).  Restgpt: Connecting large language models with real-world applications via restful apis.  arXiv preprint arXiv:2306.06624.
  * Srinivasan et al., (2021) Srinivasan, K., Raman, K., Chen, J., Bendersky, M., and Najork, M. (2021).  Wit: Wikipedia-based image text dataset for multimodal multilingual machine learning.  arXiv preprint arXiv:2103.01913.
  * Su et al., (2019) Su, W., Zhu, X., Cao, Y., Li, B., Lu, L., Wei, F., and Dai, J. (2019).  VL-BERT: Pre-training of generic visual-linguistic representations.  In ICLR.
  * Su et al., (2023) Su, Y., Lan, T., Li, H., Xu, J., Wang, Y., and Cai, D. (2023).  Pandagpt: One model to instruction-follow them all.  arXiv preprint arXiv:2305.16355.
  * Sun et al., (2017) Sun, C., Shrivastava, A., Singh, S., and Gupta, A. (2017).  Revisiting unreasonable effectiveness of data in deep learning era.  In ICCV.
  * (390) Sun, Q., Yu, Q., Cui, Y., Zhang, F., Zhang, X., Wang, Y., Gao, H., Liu, J., Huang, T., and Wang, X. (2023a).  Generative pretraining in multimodality.  arXiv preprint arXiv:2307.05222.
  * (391) Sun, Y., Yang, Y., Peng, H., Shen, Y., Yang, Y., Hu, H., Qiu, L., and Koike, H. (2023b).  Imagebrush: Learning visual in-context instructions for exemplar-based image manipulation.  arXiv preprint arXiv:2308.00906.
  * (392) Sun, Y., Zhu, C., Zheng, S., Zhang, K., Shui, Z., Yu, X., Zhao, Y., Li, H., Zhang, Y., Zhao, R., et al. (2023c).  Pathasst: Redefining pathology through generative foundation ai assistant for pathology.  arXiv preprint arXiv:2305.15072.
  * Surís et al., (2023) Surís, D., Menon, S., and Vondrick, C. (2023).  Vipergpt: Visual inference via python execution for reasoning.  arXiv preprint arXiv:2303.08128.
  * Tan and Bansal, (2019) Tan, H. and Bansal, M. (2019).  LXMERT: Learning cross-modality encoder representations from transformers.  In EMNLP.
  * (395) Tang, L., Xiao, H., and Li, B. (2023a).  Can sam segment anything? when sam meets camouflaged object detection.  arXiv preprint arXiv:2304.04709.
  * (396) Tang, Z., Yang, Z., Zhu, C., Zeng, M., and Bansal, M. (2023b).  Any-to-any generation via composable diffusion.
  * Tao et al., (2023) Tao, C., Zhu, X., Su, W., Huang, G., Li, B., Zhou, J., Qiao, Y., Wang, X., and Dai, J. (2023).  Siamese image modeling for self-supervised vision representation learning.  In CVPR.
  * Taori et al., (2023) Taori, R., Gulrajani, I., Zhang, T., Dubois, Y., Li, X., Guestrin, C., Liang, P., and Hashimoto, T. B. (2023).  Stanford alpaca: An instruction-following llama model.  https://github.com/tatsu-lab/stanford_alpaca.
  * Team, (2023) Team, M. N. (2023).  Introducing mpt-7b: A new standard for open-source, ly usable llms.  Accessed: 2023-03-28.
  * Thomee et al., (2016) Thomee, B., Shamma, D. A., Friedland, G., Elizalde, B., Ni, K., Poland, D., Borth, D., and Li, L.-J. (2016).  Yfcc100m: The new data in multimedia research.  Communications of the ACM.
  * (401) Tian, Y., Krishnan, D., and Isola, P. (2020a).  Contrastive multiview coding.  In ECCV.
  * (402) Tian, Z., Shen, C., and Chen, H. (2020b).  Conditional convolutions for instance segmentation.  In Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part I 16, pages 282–298. Springer.
  * Tong et al., (2022) Tong, Z., Song, Y., Wang, J., and Wang, L. (2022).  Videomae: Masked autoencoders are data-efficient learners for self-supervised video pre-training.  NeurIPS.
  * Touvron et al., (2021) Touvron, H., Cord, M., Douze, M., Massa, F., Sablayrolles, A., and Jégou, H. (2021).  Training data-efficient image transformers & distillation through attention.  In ICML.
  * Touvron et al., (2023) Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., et al. (2023).  Llama: Open and efficient foundation language models.  arXiv preprint arXiv:2302.13971.
  * Tschannen et al., (2023) Tschannen, M., Kumar, M., Steiner, A., Zhai, X., Houlsby, N., and Beyer, L. (2023).  Image captioners are scalable vision learners too.  arXiv preprint arXiv:2306.07915.
  * Tu et al., (2023) Tu, T., Azizi, S., Driess, D., Schaekermann, M., Amin, M., Chang, P.-C., Carroll, A., Lau, C., Tanno, R., Ktena, I., et al. (2023).  Towards generalist biomedical ai.  arXiv preprint arXiv:2307.14334.
  * Vahdat and Kautz, (2020) Vahdat, A. and Kautz, J. (2020).  Nvae: A deep hierarchical variational autoencoder.
  * van den Oord et al., (2017) van den Oord, A., Vinyals, O., and Kavukcuoglu, K. (2017).  Neural discrete representation learning.  In NeurIPS.
  * Vaswani et al., (2017) Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, Ł., and Polosukhin, I. (2017).  Attention is all you need.  In NeurIPS.
  * Vicuna, (2023) Vicuna (2023).  Vicuna: An open-source chatbot impressing GPT-4 with 90%* chatgpt quality.  https://vicuna.lmsys.org/.
  * Vinyals et al., (2015) Vinyals, O., Toshev, A., Bengio, S., and Erhan, D. (2015).  Show and tell: A neural image caption generator.  In CVPR.
  * Vinyals et al., (2016) Vinyals, O., Toshev, A., Bengio, S., and Erhan, D. (2016).  Show and tell: Lessons learned from the 2015 mscoco image captioning challenge.  IEEE transactions on pattern analysis and machine intelligence, 39(4):652–663.
  * Viola and Jones, (2001) Viola, P. and Jones, M. (2001).  Rapid object detection using a boosted cascade of simple features.  In Proceedings of the 2001 IEEE computer society conference on computer vision and pattern recognition. CVPR 2001, volume 1, pages I–I. Ieee.
  * Wang and Komatsuzaki, (2021) Wang, B. and Komatsuzaki, A. (2021).  GPT-J-6B: A 6 Billion Parameter Autoregressive Language Model.  https://github.com/kingoflolz/mesh-transformer-jax.
  * (416) Wang, B., Wu, F., Han, X., Peng, J., Zhong, H., Zhang, P., Dong, X., Li, W., Li, W., Wang, J., et al. (2023a).  Vigc: Visual instruction generation and correction.  arXiv preprint arXiv:2308.12714.
  * (417) Wang, F., Kong, T., Zhang, R., Liu, H., and Li, H. (2023b).  Self-supervised learning by estimating twin class distribution.  TIP.
  * Wang et al., (2018) Wang, H., Wang, Y., Zhou, Z., Ji, X., Li, Z., Gong, D., Zhou, J., and Liu, W. (2018).  Cosface: Large margin cosine loss for deep face recognition.  CVPR.
  * (419) Wang, H., Zhu, Y., Adam, H., Yuille, A., and Chen, L.-C. (2021a).  Max-deeplab: End-to-end panoptic segmentation with mask transformers.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5463–5474.
  * (420) Wang, J., Chen, D., Luo, C., Dai, X., Yuan, L., Wu, Z., and Jiang, Y.-G. (2023c).  Chatvideo: A tracklet-centric multimodal and versatile video understanding system.  arXiv preprint arXiv:2304.14407.
  * (421) Wang, J., Yang, Z., Hu, X., Li, L., Lin, K., Gan, Z., Liu, Z., Liu, C., and Wang, L. (2022a).  Git: A generative image-to-text transformer for vision and language.  arXiv preprint arXiv:2205.14100.
  * (422) Wang, J., Zhou, Y., Xu, G., Shi, P., Zhao, C., Xu, H., Ye, Q., Yan, M., Zhang, J., Zhu, J., et al. (2023d).  Evaluation and analysis of hallucination in large vision-language models.  arXiv preprint arXiv:2308.15126.
  * (423) Wang, L., Huang, B., Zhao, Z., Tong, Z., He, Y., Wang, Y., Wang, Y., and Qiao, Y. (2023e).  Videomae v2: Scaling video masked autoencoders with dual masking.  In CVPR.
  * (424) Wang, P., Yang, A., Men, R., Lin, J., Bai, S., Li, Z., Ma, J., Zhou, C., Zhou, J., and Yang, H. (2022b).  Ofa: Unifying architectures, tasks, and modalities through a simple sequence-to-sequence learning framework.  In ICML.
  * (425) Wang, R., Chen, D., Wu, Z., Chen, Y., Dai, X., Liu, M., Jiang, Y.-G., Zhou, L., and Yuan, L. (2022c).  Bevt: Bert pretraining of video transformers.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 14733–14743.
  * (426) Wang, T., Li, L., Lin, K., Lin, C.-C., Yang, Z., Zhang, H., Liu, Z., and Wang, L. (2023f).  Disco: Disentangled control for referring human dance generation in real world.  arXiv preprint arXiv:2307.00040.
  * (427) Wang, T., Zhang, J., Fei, J., Ge, Y., Zheng, H., Tang, Y., Li, Z., Gao, M., Zhao, S., Shan, Y., et al. (2023g).  Caption anything: Interactive image description with diverse multimodal controls.  arXiv preprint arXiv:2305.02677.
  * (428) Wang, W., Bao, H., Dong, L., Bjorck, J., Peng, Z., Liu, Q., Aggarwal, K., Mohammed, O. K., Singhal, S., Som, S., and Wei, F. (2022d).  Image as a foreign language: Beit pretraining for all vision and vision-language tasks.  arXiv preprint arXiv:2208.10442.
  * (429) Wang, W., Bao, H., Dong, L., and Wei, F. (2021b).  Vlmo: Unified vision-language pre-training with mixture-of-modality-experts.  arXiv preprint arXiv:2111.02358.
  * (430) Wang, W., Chen, Z., Chen, X., Wu, J., Zhu, X., Zeng, G., Luo, P., Lu, T., Zhou, J., Qiao, Y., et al. (2023h).  VisionLLM: Large language model is also an open-ended decoder for vision-centric tasks.  arXiv preprint arXiv:2305.11175.
  * (431) Wang, X., Wang, W., Cao, Y., Shen, C., and Huang, T. (2023i).  Images speak in images: A generalist painter for in-context visual learning.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6830–6839.
  * (432) Wang, X., Zhang, X., Cao, Y., Wang, W., Shen, C., and Huang, T. (2023j).  Seggpt: Segmenting everything in context.  arXiv preprint arXiv:2304.03284.
  * (433) Wang, Y., Ivison, H., Dasigi, P., Hessel, J., Khot, T., Chandu, K. R., Wadden, D., MacMillan, K., Smith, N. A., Beltagy, I., et al. (2023k).  How far can camels go? exploring the state of instruction tuning on open resources.  arXiv preprint arXiv:2306.04751.
  * (434) Wang, Y., Kordi, Y., Mishra, S., Liu, A., Smith, N. A., Khashabi, D., and Hajishirzi, H. (2022e).  Self-instruct: Aligning language model with self generated instructions.  arXiv preprint arXiv:2212.10560.
  * (435) Wang, Y., Mishra, S., Alipoormolabashi, P., Kordi, Y., Mirzaei, A., Arunkumar, A., Ashok, A., Dhanasekaran, A. S., Naik, A., Stap, D., et al. (2022f).  Benchmarking generalization via in-context instructions on 1,600+ language tasks.  arXiv preprint arXiv:2204.07705.
  * Wang et al., (2020) Wang, Z., Chen, J., and Hoi, S. C. (2020).  Deep learning for image super-resolution: A survey.  IEEE transactions on pattern analysis and machine intelligence, 43(10):3365–3387.
  * (437) Wang, Z., Huang, H., Zhao, Y., Zhang, Z., and Zhao, Z. (2023l).  Chat-3d: Data-efficiently tuning large language model for universal dialogue of 3d scenes.  arXiv preprint arXiv:2308.08769.
  * (438) Wang, Z., Jiang, Y., Lu, Y., Shen, Y., He, P., Chen, W., Wang, Z., and Zhou, M. (2023m).  In-context learning unlocked for diffusion models.
  * (439) Wang, Z., Yu, J., Yu, A. W., Dai, Z., Tsvetkov, Y., and Cao, Y. (2022g).  Simvlm: Simple visual language model pretraining with weak supervision.  In ICLR.
  * Weers et al., (2023) Weers, F., Shankar, V., Katharopoulos, A., Yang, Y., and Gunter, T. (2023).  Masked autoencoding does not help natural language supervision at scale.  In CVPR.
  * Wei et al., (2021) Wei, C., Fan, H., Xie, S., Wu, C.-Y., Yuille, A., and Feichtenhofer, C. (2021).  Masked feature prediction for self-supervised visual pre-training.  In CVPR.
  * (442) Wei, J., Wang, X., Schuurmans, D., Bosma, M., Chi, E., Le, Q., and Zhou, D. (2022a).  Chain of thought prompting elicits reasoning in large language models.  arXiv preprint arXiv:2201.11903.
  * (443) Wei, L., Xie, L., Zhou, W., Li, H., and Tian, Q. (2022b).  Mvp: Multimodality-guided visual pre-training.  In ECCV.
  * Wei et al., (2023) Wei, Y., Zhang, Y., Ji, Z., Bai, J., Zhang, L., and Zuo, W. (2023).  Elite: Encoding visual concepts into textual embeddings for customized text-to-image generation.  arXiv preprint arXiv:2302.13848.
  * Weng, (2023) Weng, L. (2023).  Llm-powered autonomous agents.  lilianweng.github.io.
  * (446) Wu, C., Yin, S., Qi, W., Wang, X., Tang, Z., and Duan, N. (2023a).  Visual chatgpt: Talking, drawing and editing with visual foundation models.  arXiv preprint arXiv:2303.04671.
  * (447) Wu, C., Zhang, X., Zhang, Y., Wang, Y., and Xie, W. (2023b).  Towards generalist foundation model for radiology.  arXiv preprint arXiv:2308.02463.
  * (448) Wu, J., Jiang, Y., Sun, P., Yuan, Z., and Luo, P. (2022a).  Language as queries for referring video object segmentation.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4974–4984.
  * Wu et al., (2021) Wu, J., Lu, J., Sabharwal, A., and Mottaghi, R. (2021).  Multi-modal answer validation for knowledge-based VQA.  arXiv preprint arXiv:2103.12248.
  * (450) Wu, J., Wang, J., Yang, Z., Gan, Z., Liu, Z., Yuan, J., and Wang, L. (2022b).  Grit: A generative region-to-text transformer for object understanding.  arXiv preprint arXiv:2212.00280.
  * (451) Wu, S., Fei, H., Qu, L., Ji, W., and Chua, T.-S. (2023c).  Next-gpt: Any-to-any multimodal llm.  CoRR, abs/2309.05519.
  * (452) Wu, W., Timofeev, A., Chen, C., Zhang, B., Duan, K., Liu, S., Zheng, Y., Shlens, J., Du, X., Gan, Z., et al. (2023d).  Mofi: Learning image representations from noisy entity annotated images.  arXiv preprint arXiv:2306.07952.
  * Wu et al., (2013) Wu, Y., Lim, J., and Yang, M.-H. (2013).  Online object tracking: A benchmark.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2411–2418.
  * Wu et al., (2018) Wu, Z., Xiong, Y., Yu, S. X., and Lin, D. (2018).  Unsupervised feature learning via non-parametric instance discrimination.  In CVPR.
  * Xian et al., (2018) Xian, Y., Lampert, C. H., Schiele, B., and Akata, Z. (2018).  Zero-shot learning—a comprehensive evaluation of the good, the bad and the ugly.  TPAMI.
  * Xiao et al., (2023) Xiao, Z., Chen, Y., Zhang, L., Yao, J., Wu, Z., Yu, X., Pan, Y., Zhao, L., Ma, C., Liu, X., et al. (2023).  Instruction-vit: Multi-modal prompts for instruction learning in vit.  arXiv preprint arXiv:2305.00201.
  * (457) Xie, D., Wang, R., Ma, J., Chen, C., Lu, H., Yang, D., Shi, F., and Lin, X. (2023a).  Edit everything: A text-guided generative system for images editing.  arXiv preprint arXiv:2304.14006.
  * (458) Xie, X., Fu, L., Zhang, Z., Wang, Z., and Bai, X. (2022a).  Toward understanding wordart: Corner-guided transformer for scene text recognition.
  * Xie et al., (2021) Xie, Z., Lin, Y., Yao, Z., Zhang, Z., Dai, Q., Cao, Y., and Hu, H. (2021).  Self-supervised learning with swin transformers.  arXiv preprint arXiv:2105.04553.
  * (460) Xie, Z., Zhang, Z., Cao, Y., Lin, Y., Bao, J., Yao, Z., Dai, Q., and Hu, H. (2022b).  Simmim: A simple framework for masked image modeling.  In CVPR.
  * (461) Xie, Z., Zhang, Z., Cao, Y., Lin, Y., Wei, Y., Dai, Q., and Hu, H. (2023b).  On data scaling in masked image modeling.  In CVPR.
  * Xu et al., (2018) Xu, D., Ouyang, W., Wang, X., and Sebe, N. (2018).  Pad-net: Multi-tasks guided prediction-and-distillation network for simultaneous depth estimation and scene parsing.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 675–684.
  * Xu et al., (2021) Xu, H., Yan, M., Li, C., Bi, B., Huang, S., Xiao, W., and Huang, F. (2021).  E2e-vlp: End-to-end vision-language pre-training enhanced by visual learning.
  * (464) Xu, J., De Mello, S., Liu, S., Byeon, W., Breuel, T., Kautz, J., and Wang, X. (2022a).  Groupvit: Semantic segmentation emerges from text supervision.  In CVPR.
  * (465) Xu, J., Liu, S., Vahdat, A., Byeon, W., Wang, X., and De Mello, S. (2023a).  Open-vocabulary panoptic segmentation with text-to-image diffusion models.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2955–2966.
  * (466) Xu, P., Shao, W., Zhang, K., Gao, P., Liu, S., Lei, M., Meng, F., Huang, S., Qiao, Y., and Luo, P. (2023b).  Lvlm-ehub: A comprehensive evaluation benchmark for large vision-language models.  arXiv preprint arXiv:2306.09265.
  * (467) Xu, R., Wang, X., Wang, T., Chen, Y., Pang, J., and Lin, D. (2023c).  Pointllm: Empowering large language models to understand point clouds.  arXiv preprint arXiv:2308.16911.
  * (468) Xu, Z., Shen, Y., and Huang, L. (2022b).  Multiinstruct: Improving multi-modal zero-shot learning via instruction tuning.  arXiv preprint arXiv:2212.10773.
  * Yan et al., (2023) Yan, B., Jiang, Y., Wu, J., Wang, D., Luo, P., Yuan, Z., and Lu, H. (2023).  Universal instance perception as object discovery and retrieval.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15325–15336.
  * (470) Yang, B., Gu, S., Zhang, B., Zhang, T., Chen, X., Sun, X., Chen, D., and Wen, F. (2023a).  Paint by example: Exemplar-based image editing with diffusion models.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18381–18391.
  * (471) Yang, J., Gao, M., Li, Z., Gao, S., Wang, F., and Zheng, F. (2023b).  Track anything: Segment anything meets videos.  arXiv preprint arXiv:2304.11968.
  * (472) Yang, J., Li, C., Zhang, P., Xiao, B., Liu, C., Yuan, L., and Gao, J. (2022a).  Unified contrastive learning in image-text-label space.  In CVPR.
  * (473) Yang, J., Li, C., Zhang, P., Xiao, B., Yuan, L., Liu, C., and Gao, J. (2022b).  Unicl: Unified contrastive learning in image-text-label space.  In CVPR.
  * (474) Yang, R., Song, L., Li, Y., Zhao, S., Ge, Y., Li, X., and Shan, Y. (2023c).  Gpt4tools: Teaching large language model to use tools via self-instruction.  arXiv preprint arXiv:2305.18752.
  * Yang et al., (2021) Yang, Z., Gan, Z., Wang, J., Hu, X., Ahmed, F., Liu, Z., Lu, Y., and Wang, L. (2021).  Crossing the format boundary of text and boxes: Towards unified vision-language modeling.  arXiv preprint arXiv:2111.12085.
  * (476) Yang, Z., Gan, Z., Wang, J., Hu, X., Ahmed, F., Liu, Z., Lu, Y., and Wang, L. (2022c).  Unitab: Unifying text and box outputs for grounded vision-language modeling.  In European Conference on Computer Vision, pages 521–539. Springer.
  * (477) Yang, Z., Gan, Z., Wang, J., Hu, X., Lu, Y., Liu, Z., and Wang, L. (2022d).  An empirical study of gpt-3 for few-shot knowledge-based vqa.  In AAAI.
  * Yang* et al., (2023) Yang*, Z., Li*, L., Wang*, J., Lin*, K., Azarnasab*, E., Ahmed*, F., Liu, Z., Liu, C., Zeng, M., and Wang, L. (2023).  Mm-react: Prompting chatgpt for multimodal reasoning and action.
  * (479) Yang, Z., Ping, W., Liu, Z., Korthikanti, V., Nie, W., Huang, D.-A., Fan, L., Yu, Z., Lan, S., Li, B., et al. (2023a).  Re-vilm: Retrieval-augmented visual language model for zero and few-shot image captioning.  arXiv preprint arXiv:2302.04858.
  * (480) Yang, Z., Wang, J., Gan, Z., Li, L., Lin, K., Wu, C., Duan, N., Liu, Z., Liu, C., Zeng, M., et al. (2023b).  Reco: Region-controlled text-to-image generation.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14246–14255.
  * (481) Yang, Z., Wang, J., Tang, Y., Chen, K., Zhao, H., and Torr, P. H. (2022e).  Lavt: Language-aware vision transformer for referring image segmentation.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18155–18165.
  * Yao et al., (2023) Yao, L., Han, J., Liang, X., Xu, D., Zhang, W., Li, Z., and Xu, H. (2023).  Detclipv2: Scalable open-vocabulary object detection pre-training via word-region alignment.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 23497–23506.
  * (483) Yao, L., Han, J., Wen, Y., Liang, X., Xu, D., Zhang, W., Li, Z., Xu, C., and Xu, H. (2022a).  Detclip: Dictionary-enriched visual-concept paralleled pre-training for open-world detection.  arXiv preprint arXiv:2209.09407.
  * (484) Yao, L., Huang, R., Hou, L., Lu, G., Niu, M., Xu, H., Liang, X., Li, Z., Jiang, X., and Xu, C. (2022b).  Filip: Fine-grained interactive language-image pre-training.  In ICLR.
  * (485) Yao, S., Zhao, J., Yu, D., Du, N., Shafran, I., Narasimhan, K., and Cao, Y. (2022c).  React: Synergizing reasoning and acting in language models.  arXiv preprint arXiv:2210.03629.
  * Yasunaga et al., (2022) Yasunaga, M., Aghajanyan, A., Shi, W., James, R., Leskovec, J., Liang, P., Lewis, M., Zettlemoyer, L., and Yih, W.-t. (2022).  Retrieval-augmented multimodal language modeling.  arXiv preprint arXiv:2211.12561.
  * (487) Ye, J., Hu, A., Xu, H., Ye, Q., Yan, M., Dan, Y., Zhao, C., Xu, G., Li, C., Tian, J., et al. (2023a).  mplug-docowl: Modularized multimodal large language model for document understanding.  arXiv preprint arXiv:2307.02499.
  * (488) Ye, L., Rochan, M., Liu, Z., and Wang, Y. (2019a).  Cross-modal self-attention network for referring image segmentation.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10502–10511.
  * (489) Ye, M., Zhang, X., Yuen, P. C., and Chang, S.-F. (2019b).  Unsupervised embedding learning via invariant and spreading instance feature.  In CVPR.
  * (490) Ye, Q., Xu, H., Xu, G., Ye, J., Yan, M., Zhou, Y., Wang, J., Hu, A., Shi, P., Shi, Y., et al. (2023b).  mplug-owl: Modularization empowers large language models with multimodality.  arXiv preprint arXiv:2304.14178.