modality, however, there can be multiple types of paired modalities available
in practice. Image Bind [92] includes multiple modalities by learning common
representations for the paired data modalities e.g., (video, audio) or (image,
depth). By aligning visual features with any of the sensory experiences
associated with them, images have this ’binding’ property that can offer many
sources of supervision to learn visual features. For better representation
learning, different sensors should be aligned to a single joint embedding
space to learn visual features. The problem, however, is that it is not
feasible to obtain every type and combination of paired data with the same set
of images. The lack of multimodal data across all modalities is one of the
major obstacles to learning a joint embedding. Using multiple types of image-
paired data, ImageBind learns a single shared representation space, therefore
it is independent of the constraint where datasets from all modalities need to
co-occur with each other for joint learning. ImageBind combines large-scale
paired data (image, text) with other paired data modalities (video, audio) or
(image, depth) to develop a joint feature representation, thereby aligning
each other modality (audio, depth) with textual embedding. ImageBind expands
zero-shot capabilities across four modalities including audio, depth, thermal,
and Inertial Measurement Unit (IMU) readings.

Figure 15: PaLM-E [72] applies visual-language knowledge to embodied
reasoning, to plan robots within dynamic environments, and to answer questions
related to observable reality. PAL-E performs end-to-end training using
multimodal sentences, where inputs from a variety of modalities are added to
text tokens as input. Figure from Reference [72].

Type | Foundational Model | Public | [Link] | Prompts | Training |  Data Type |  Data Size
---|---|---|---|---|---|---|---
Visually Prompted Models | CLIPSeg [185] | ✓ | Link | Text, Image | ✓ |  PhraseCut [290] |  0.34M Images
SegGPT [280] | ✓ | Link | Learnable Image Prompt | ✓ |  Curation of Twelve Modalities [280] |  $\approx$ 0.3M Images
SAM [140] | ✓ | Link | Points, Box, Text | ✓ |  SA-1B [140] |  11M Images, 1.1B Masks
SEEM [366] | ✓ | Link | Text, Points, Scribbles, Boxes, Images | ✓ |  COCO [163] |  118K Images
CAT [274] | ✓ | Link | Text, Points, Boxes, Mask | ✗ |  – |  –
TAM [307] | ✓ | Link | Points, Boxes, Clicks | ✗ |  – |  –
SAM-Track [48] | ✓ | Link | Text | ✗ |  – |  –
SAM-PT [217] | ✓ | Link | Points | ✗ |  – |  –
SAM-DA [162] | ✓ | Link | Boxes | ✓ |  NAT2021 [312] |  0.27M Images
RsPrompter [35] | ✓ | Link | Learnable Prompter | ✓ |  WHU [128], NWPU [46], and SSDD [340] |  $\approx$ 6.6K Images
MedSAM [189] | ✓ | Link | Boxes | ✓ |  Curation of Eleven Modalities [189] |  200k Masks
AutoSAM [233] | ✗ | – | Learnable Prompt | ✓ |  MoNuSeg [145], GlaS [244], Polyp [126, 14, 253, 241], Sun-Seg [127] |  $\approx$ 1.5K Images, 1.1K Videos
DSAM-adapter [93] | ✓ | Link | Points | ✓ |  KiTS21 [110], LiTS17 [16], Pancreas [10], Colon [10] |  $\approx$ 600 CT scans
DeSAM [87] | ✓ | Link | Points, Boxes | ✓ |  Prostate datasets (NCI-ISBI, I2CVB, PROMISE12) [87] |  –
MedLAM [288] | ✓ | Link | Boxes | ✓ |  Curation of Sixteen Modalities [288] |  $\approx$ 14l CT Sans
FasterSAM [328] | ✓ | Link | Points, Box, Text | ✓ |  Subset of SA-1B [140] |  11K Images
Fast Segment [347] | ✓ | Link | Points, Box, Text | ✓ |  Subset of SA-1B [140] |  22K Images
RefSAM [316] | ✓ | Link | Points, Boxes | ✓ |  Ref-Youtu-VOS [229], Ref-DAVIS17 [137] |  $\approx$ 4k Videos
Painter [279] | ✓ | Link | Task-specific Prompts | ✓ |  NYUv2 [240], ADE20K [356], COCO [163], SIDD [4], LOL [285] |  $\approx$ 162K Images
VisionLLM [276] | ✓ | Link | Task Descriptions and Categories | ✓ |  COCO [163], RefCOCO+ [320], RefCOCOg [193] |  238K Images
Prismer [173] | ✓ | Link | Text | ✓ |  COCO [163], Visual Genome [143], Captions [235], SBU-captions [205], Conceptual12M [30] |  11M Images, 12.7M (Image, Text)
Heterogeneous Modalities based Models | CLIP2Video [78] | ✓ | Link | Text | ✓ |  MSR-VTT [298], MSVD [33], VATEX [278] |  $\approx$ 33.7K Videos
AudioCLIP [102] | ✓ | Link | Text, Audio | ✓ |  AudioSet [89] |  1.8M (Video, Audio)
Image Bind [92] | ✓ | Link | Text, Depth, Audio, Thermal | ✓ |  Audioset [89], SUN RGB-D [246], LLVIP [130], Ego4D [95] |  1.8M (Video, Audio), 10.3K (Image, Depth), 15.4K (Image, Thermal), 3,670 hours of (Video, IMU)
MACAW-LLM [188] | ✓ | Link | Text generated using GPT4 | ✓ |  Multi-turn Dialogue |  69K (Image, Text), 50k (Video, Text)
COSA [37] | ✓ | Link | Text | ✓ |  CC14M [37], LAION [226], WebVid [270] |  From 5M to 514M (Image, Text)
Valley [186] | ✓ | Link | Text generated using ChatGPT | ✓ |  Video Conversation and QA [186] |  42K Conversations and 5.8K QA about Videos
Embodied Foundational Agents | Palm-E [72] | ✗ | – | Multi-modal (Image, Text) | ✓ |  Language-Table [187], SayCan [6] |  Pushing Dynamics, Tasks in a Kitchen Environment,
ViMA [131] | ✓ | Link | Multi-modal (Image, Text) | ✓ |  VIMA-BENCH [131] |  600K+ expert trajectories for learning
MineDojo [76] | ✓ | Link | Text | ✓ |  Data collected from Minecraft |  730k Videos, 6k Transcripts, 340K Reddit Posts
VOYAGER [267] | ✓ | Link | Iterative Prompting with Feedback | ✓ |  Collected from within Minecraft |  Minecraft items, Library storing behavior
LM-Nav [231] | ✓ | Link | LandMarks, Text | ✗ |  – |  –

TABLE IV: A summary of publicly available information about visually prompted,
heterogeneous modality-based models and embodied foundational agents, their
prompt design differences, and the nature of their training data type and
size.

Aligning LLM with Heterogeneous Modalities: MACAW-LLM [188] is an instruction-
tuned multi-modal LLM that integrates four different modalities including
image, video, audio, and text into a single model. It aligns the feature
representation of different data modalities into the embeddings of an LLM
thereby bringing those features closer to the textual representation of a
large language model. A large-scale multimodal instruction dataset that
combines image and video modalities is used to train the MACAW-LLM, which
facilitates future work on this type of model learning. MACAW-LLM consists of
three modules including Modality Module integrates different modality (e.g.,
visual and audio data) encoders into MACAW-LLM, Alignment Module which unifies
different modality encoders which are trained independently, and Cognitive
Module which is pre-trained LLM.

For video-language reasoning, temporal consistency and context are important.
The current image-text foundation models which are trained exclusively on
image-text corpora have gained a comprehensive understanding of the semantic
correlations between visual concepts and language, however, they lack the
temporal contexts required for videos. A solution to this problem is to train
on large-scale video-text corpora which is costly to obtain. COSA [37]
proposes to generate a video paragraph corpus from the image-text corpus by
dynamically transforming it into long-form video paragraph samples. It
randomly concatenates a certain number of image-text training samples together
from the same batch at each training step. The images and texts are
concatenated, ensuring that events and sentences correspond explicitly. The
on-the-fly concatenated corpus is superior to short-form video-text corpora
[12] due to its richer scene transformations, reduced vision redundancy, and
finer-grained captions that describe each frame sequentially. Concatenated
samples are used as input during pretraining by COSA, which has a simple
architecture. In addition to visual retrieval, captioning, and answering
questions, this model is capable of handling both discriminative and
generative tasks.

Valley (Video Assistant with a Large Language model) [186] is another multi-
modal framework capable of integrating video, image, and language perceptions.
A simple projection module is used in the Valley model to bridge video, image,
and language modalities, and is further unified with a multi-lingual LLM
through the instructions-tuned pipeline. To obtain a unified vision encoding
of video and image input, Valley also employs a spatiotemporal pooling
strategy. Various video tasks including video question-answering, long
description, casual relationship inference, and action recognition are used to
collect instructions-following data. This data is then used for instruction
fine-tuning to obtain a foundational model for videos.

##  7 Embodied Foundational Agents

The training of LLMs on massive textual data may result in representations
that are relevant to the real world, but to solve a wider range of computer
vision and robotics problems grounded in reality, connecting these
representations to real-world visual and physical sensor modalities is
essential. In this section, we discuss foundational embodied agents for robot
manipulation [72, 131].

For Robot Manipulation: By incorporating continuous inputs from sensor
modalities of the embodied agent directly into embodied language models,
Palm-E [72] (see Fig. 15)solves this challenging task, enabling the language
model itself to make more grounded inferences about sequential decision-
making. An LLM based on a Transformer embeds inputs such as images and state
estimates into the same latent embedding as language tokens and processes them
the same way as texts. The continuous inputs are injected through an encoder
into a pre-trained LLM. These encodings are trained from end to end by
enforcing output sequential decisions in terms of natural text that can be
understood by the embodied agents.

There are various forms of robotic task specification, including imitating
one-shot demonstrations, following linguistic instructions, and reaching
visual targets. Different models are often used for these different tasks.
ViMA [131] showed that multimodal prompting, interleaving textual and visual
tokens, is effective at expressing a wide range of robot manipulation tasks
thus learning robot manipulation through multimodal prompts. ViMA follows the
transformer-based encoder-decoder network proposed by [216]. By encoding
textual and visual prompt tokens using a pre-trained language model [263] and
decoding robot control actions auto-regressively, VIMA encodes and decodes
robot actions for every set of environment interactions. Further, they
developed a new simulation benchmark with multimodal prompts that contains
more than 600K expert trajectories for imitation learning. They also developed
a four-level evaluation protocol for systematic generalization.

For Continual Learners: For such agents, [76] provides a convenient API that
makes it easy for users to specify task specifications, change world settings,
and observe and act on tasks in Minecraft. There are thousands of open-ended
tasks prompted by natural language. As a part of MineDojo, they collected a
wealth of data from Minecraft including 30K+ YouTube videos with time-aligned
transcripts, 6K+ free-form Wiki pages, and 340K+ Reddit posts with multimedia
content. MineDojo [76] then devised a novel learning algorithm for embodied
agents using such data. Their video-text model associates natural language
subtitles with time-aligned YouTube videos from MineDojo. Further, they
proposed a method for evaluating agents by using such a large pre-trained
video-language model that was developed based on Minecraft YouTube videos.
This complements human scoring [232], which is expensive.

Inspired by how humans play Minecraft, VOYAGER [267] argues that lifelong
learning agents should be able to perform similar tasks to humans e.g., it
should solve tasks based on the current level of their skill, adapt to
environmental feedback to refine skills and actively seek out new tasks and
explore the world. VOYAGER is one of the first embodied lifelong learning
agents powered by LLM. It is designed to drive exploration, hone a wide range
of skills, and continuously discover new things in Minecraft. Based on the
automatic curriculum generated by GPT-4, VOYAGER is designed to solve
increasingly difficult tasks so that it discovers as much variety as possible.
This is similar to novelty search [75, 55]. As VOYAGER stores action programs
to complete tasks, a skill library is incrementally built. VOYAGER’s
capability is compounded over time by composing smaller programs, alleviating
the catastrophic forgetting associated with other continual learning methods
[207, 273].

For Navigation Planning: An actionable plan can be derived without prior fine-
tuning in the target environment using LM-Nav [231], which combines pre-
trained vision and language models with a goal-conditioned controller. A pre-
trained navigation model is combined with two robot-agnostic pre-trained
models to achieve this. Using the robot’s observations, we construct a
topological ”mental map” of the environment using a visual navigation model,
ViNG [230]. Then to decode free-form textual instructions into textual
landmarks, LM-Nav uses a large language model, GPT-3 [20]. To ground these
textual landmarks in the topological map, it employs a vision-language model
such as CLIP [215], which infers a joint likelihood over the landmarks and
nodes. Finally, to find a plan for the robot, VNM uses a novel search
algorithm, which maximizes a probabilistic objective. In a nutshell, LM-Nav
provides long-horizon instruction following in complex, real-world
environments by combining three large pre-trained models; a self-supervised
robotic control model, a vision-language model, and a large language model.

##  8 Open Challenges & Research Directions

While individual models discussed in this survey may have respective
shortcomings and open challenges, this section aims to provide a holistic
overview of the shared challenges these approaches (or their subset) face. We
also highlight research directions that can help address these challenges.

Multimodal Open-source Models: In NLP tasks, the transition from GPT3 to
ChatGPT shows the importance of instruction-following and human-feedback-based
reinforcement learning. For multimodal (text and image) inputs, a similar
capability is claimed by GPT4 [204] to allow reasoning and understanding based
on vision-language inputs. However, GPT4 is a closed-source model with
restricted access to date and its training details also remain unknown. To
bridge this gap, multimodal open-source foundation models such as BLIP2 [155],
GIT [269] and Flamingo [7] can be extended with the instruction following and
human intent alignment to have ChatGPT-like capabilities in multimodal spaces.
To this end, initial efforts have been reported such as Intruct-BLIP,
miniGPT4, LLaVA, and Video-ChatGPT. However, matching the capabilities of GPT4
with open-source public models is still a major challenge for multi-modal
foundational models.

Evaluation and Benchmarking: The open-ended nature of large-scale
conversational Vision-Language Models make their comprehensive evaluation
challenging. This challenge is shared with the progress in LLM but more severe
for visual inputs since the possible tasks and reasoning capabilities become
quite diverse for a broad and extensive evaluation. One _quantitative_
approach taken is to define a set of instructions covering multiple reasoning
aspects and forward the responses from two competing chatbot VLMs to GPT4 to
rate them on a scale of 1 to 10. This ‘LLM-as-a-judge’ approach was introduced
by Vicuna-Instruction-80 [50, 350] benchmark for LLM that comprises of 9
instruction categories: _generic, knowledge, math, counterfactual, Fermi,
coding, writing, roleplay, common-sense_. This approach has also been extended
for VLMs e.g., [190] uses four criteria (_correctness of information, detail
orientation, contextual understanding, temporal understanding, consistency_)
scored by GPT4 for benchmarking a VLM tailored for videos. However, the use of
an external GPT4 model as a gold standard is still debatable and new efforts
in LLM for benchmarking and identifying the corner cases have been reported to
address the limitations of existing evaluation measures e.g., [125, 151, 225,
21, 167]. Such efforts are likely to be extended to VLMs with even more
attention to the peculiar visual aspect of VLMs.

Hallucination: Hallucination refers to the phenomenon where the output
generated from a large VLM/LLM is unreal or nonsensical, often based on a
hypothetical scenario. Foundational language and vision models, specifically
those based on Generative Pretrained Models for open-ended conversations [204,
360, 190, 169], can sometimes fabricate answers, even if they are technically
correct in specific contexts. This is because they are trained on massive
datasets of text and/or images, that are often noisy, and they may not be able
to distinguish between what is real and what is not. Specifically for VLMs
where visual data is provided as an input condition, e.g., image-based visual
question answering, one form of hallucination ignores the visual input and can
only provide an answer based on the text prompt. As an example, an image with
a green-colored apple together with a corresponding question _what color is
the apple in this image?_ may result in an answer _red_ due to over-reliance
on training data and ignoring the prompt context. One way to control
hallucinations is to provide explicit instructions (or so-called _system
commands_) to a conversational LLM to provide its answers based on the
provided context e.g., asking the chatbot to provide the missing information
in patient health records while being strictly grounded in the facts available
in the patient data. Other strategies to mitigate hallucination include the
chain of thought prompting [184, 344], self-consistency (voting) [202, 161]
and use of knowledge bases for retrieval augmented generation [218, 171, 206].

Multimodal Alignment: Existing VLMs sometimes also suffer from poor alignment
between vision-language (or other modalities). For instance, Segment anything
[140] performance with text prompts is weaker as compared to the visual
prompts (points, boxes, or masks). For heterogeneous modalities, such an
alignment can be even more challenging. Approaches like ImageBind [92]
demonstrate viable ways in which the alignment between several modalities can
be achieved, however, there is still much room to demonstrate strong alignment
capabilities for even a wider range of related inputs that have the shared
semantic space. For example, when a human sees a picture of a food item, it
not only recognizes tcomputing item’s category but also remembers the taste,
the recipe used to cook it, and the crunchy sound each bite generates while
eating the food. For a unified representation space that can provide a
complete understanding of the world around us, foundational models targeted at
learning joint embedding spaces will be crucial for further development.

Large Data and Compute Requirements: Training large-scale vision and language
models are data and compute-intensive. Labeled data on a large scale can be
costly and time-consuming to acquire, especially for specialized visual
domains or low-resource languages. Similarly, their inference is also costly
due to the many parameters involved. The computational demands of these models
limit their accessibility and scalability in many real-world applications. As
an example, applications requiring real-time inference capability or ones
requiring deployment on the edge and mobile devices with limited on-device
compute and restricted battery times. Similarly, visual prompt-based models
like Segment Anything [140] would benefit from a real-time speed to ensure
intractability [327, 347]. However, the current versions with a high-
performing image encoder do not offer real-time overall processing. Efforts,
such as retentive networks [251], can be integrated within VLMs for high
throughput processing.

Adaptation of FMs: Foundational model training often consumes long training
times and massive compute resources. Therefore, FMs are adapted to several
downstream tasks and applications. The efficient adaptation of FMs without
damaging the extensive knowledge learned by the model is an open-research
question with many interesting initial efforts reported in recent years. Due
to great interest in LLMs and Diffusion models, such Parameter-Efficient Fine-
tuning (PEFT) approaches are primarily explored for these two model classes
but are directly applicable to adapt other vision foundational models as well.
Some representative approaches include Low-rank Adaptation (LoRA) [116] and
its variants such as QLoRA [60] and GLoRA [32], Prefix tuning [157, 176],
Adapters [337, 85], Prompt tuning [149, 38]. Reducing the compute and memory
footprint for quick adaptation of textually and visually prompted foundational
models is still an open research direction since the existing approaches
require careful selection of hyper-parameters (e.g., rank in LoRA or the
placement and dimensions of bottleneck adapters) and can result in loss of
generalization performance.

Vulnerability to Adversarial Attacks: Foundation models, similar to other
neural network-based models, can be fooled by adversarial attacks. These
attacks involve carefully crafted inputs that can cause the model to generate
incorrect or harmful output. However, there are specific ways for adversarial
attacks on foundational models that make them susceptible to unwanted
behavior. As an example, models based on conversational LLMs have been shown
vulnerable to adversarially prompt injection which needs a direct interaction
between the adversary and the LLM-based conversational agent [2, 211, 3].
Greshake et al. [96] demonstrate that even a direct interaction between the
model and adversary is not needed in LLM-integrated applications, and an
adversary can remotely poison the information retrieved by the conversational
agent via indirect prompt injection. This leads to a spectrum of
vulnerabilities for LLMs and VLMs including manipulated content, fraud,
malware, intrusion, personal information leakage, and denial of services via
language and visual prompt injections. Carlini et al. [26] recently showed
that NLP-based optimization attacks are weak and their failure on foundational
models should not be taken as a certification of robustness. They further
demonstrate that in conversational VLMs, an attack can be easily launched by
adversarially perturbing the inputs to get harmful responses from the model.
Maus et al. [195] show how non-meaningful text can be appended with the
prompts to fool generative text and image models. The exemplars (input-output
pairs) provided for In-Context Learning of VLMs can also be changed to fool
the model [271]. Visual prompt-based models e.g., SAM have also been attacked
by corrupting their inputs and associated prompts [329, 121]. Strategies to
robustify foundational VLMs against such attacks is an open research question
of high significance.

Bias and Fairness: Foundational models in vision and language can inherit and
amplify biases present in the data used for training them. Biases,
stereotypes, and prejudices related to race, underrepresented groups, minority
cultures, and gender can make the models output biased predictions or exhibit
skewed behavior. For example, recent work [237] shows the sensitivity of the
CLIP model towards red circles, where simply drawing a red circle around a
person’s face increases its chances of being misclassified as a murderer,
suspect, or a missing person. This behavior potentially emerges from the data
that contain examples from news media outlets that typically put a red circle
around criminals in their broadcasts. In concurrent efforts, new benchmarks
have been developed to assess the capability of existing VLMs towards certain
biases [103]. Addressing biases in foundational AI models is crucial to ensure
fairness, inclusivity, and ethical deployment of these systems.

Interpretablity: Foundation models are often difficult to interpret, which can
make it difficult to understand how they work and why they generate the output
they do. In this direction, existing methods have investigated chain-of-
thought reasoning to explain the outputs generated by vision and language
models [287]. New benchmarks are also developed to evaluate and train
explicitly focusing on providing detailed step-wise rationales for model
choices e.g., ScienceQA [183]. An interesting idea, called Visual Programming,
is to use interpretable neuro-symbolic representation to break down a complex
task into simpler steps that explain the rationale of a particular output from
GPT-3 [100]. While these efforts are promising, they have several failure
cases and can be further improved e.g., by allowing user feedback to improve
the interpretation generated by the model.

Limited Contextual Understanding: While transformer-based foundational models
have shown impressive language and vision understanding capabilities, they can
still struggle with certain contextual nuances. Understanding sarcasm, irony,
or other forms of figurative image and language inputs (e.g., memes) can be
challenging for these models, leading to inaccurate interpretations or
responses. While there have been initial efforts on this topic in language-
only models, similar efforts for large multi-modal models are much needed and
remain an open problem.

Lack of Real-world Understanding: Foundational models for language and vision
lack a deep understanding of the world. They can only generate prompt-
conditioned text, code, or visual outputs that are consistent with the data
they have been trained on. This training is focused on outputting the next
sequence element given previous sequence elements or learning multimodal
alignment. However, such training is much different from human learning and
reasoning which is grounded in physical reality [115]. This means that large
language and vision models may not be able to understand the meaning of what
they are generating or to reason about complex concepts grounded in the real
world. Egocentric perception and embodied AI agents based on foundational
multimodal models need to develop world models and the alignment of
heterogeneous modalities for improved physically grounded reasoning. In this
direction, works on embodied foundational models such as MineDojo [76], VPT
[13] and Voyager [267] use open-ended nature of Minecraft game as a testbed
for embodied agents based on GPT models. However, taking these agents to real-
world tasks and complex environments is a challenging problem requiring much
further work. Google’s Palm-E [72] is a step in this direction that combines
ViTs with Palm [52, 8] LLM model trained with language and sensory inputs to
allow embodied agents to understand commands and take informed actions.

##  9 Conclusion

Models with a foundational understanding of multiple modalities, including
natural language and vision, are essential for developing AI systems that can
effectively perceive and reason about the real world. This survey reviews
vision and language foundational models focusing on their architecture types,
training objectives, downstream task adaption, and their prompting designs. We
provide systematic categorizations of textually-prompted, visually-prompted,
and heterogeneous modality models. We provide a broad coverage of their
applications in a variety of visual tasks including zero-shot recognition and
localization abilities, visual dialogue about an image or a video, cross-
modal, and medical data understanding. We summarize how foundational models in
vision can act as generalist models solving multiple tasks simultaneously and
their combination with large language models gives rise to foundational
embodied agents that can continually learn and navigate in a complex
environment. We hope that this effort will spur further research in leveraging
the potential of foundational models along with addressing their limitations,
e.g. limited contextual understanding, biases, and vulnerability to malicious
uses.

## References

  * [1] Introducing chatgpt.  https://openai.com/blog/chatgpt.  Accessed: 2023-06-30.
  * Cha [2023] How to jailbreak chatgpt, 2023.  URL https://watcher.guru/news/how-to-jailbreak-chatgpt.
  * Red [2023] Red-teaming large-language models, 2023.  URL https://huggingface.co/blog/red-teaming.
  * Abdelhamed et al. [2018] Abdelrahman Abdelhamed, Stephen Lin, and Michael S Brown.  A high-quality denoising dataset for smartphone cameras.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 1692–1700, 2018.
  * Agrawal et al. [2019] Harsh Agrawal, Karan Desai, Yufei Wang, Xinlei Chen, Rishabh Jain, Mark Johnson, Dhruv Batra, Devi Parikh, Stefan Lee, and Peter Anderson.  Nocaps: Novel object captioning at scale.  In _Proceedings of the IEEE/CVF international conference on computer vision_ , pages 8948–8957, 2019.
  * Ahn et al. [2022] Michael Ahn, Anthony Brohan, Noah Brown, Yevgen Chebotar, Omar Cortes, Byron David, Chelsea Finn, Chuyuan Fu, Keerthana Gopalakrishnan, Karol Hausman, et al.  Do as i can, not as i say: Grounding language in robotic affordances.  _arXiv preprint arXiv:2204.01691_ , 2022.
  * Alayrac et al. [2022] Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al.  Flamingo: a visual language model for few-shot learning.  _Advances in Neural Information Processing Systems_ , 35:23716–23736, 2022.
  * Anil et al. [2023] Rohan Anil, Andrew M Dai, Orhan Firat, Melvin Johnson, Dmitry Lepikhin, Alexandre Passos, Siamak Shakeri, Emanuel Taropa, Paige Bailey, Zhifeng Chen, et al.  Palm 2 technical report.  _arXiv preprint arXiv:2305.10403_ , 2023.
  * Antol et al. [2015] Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C Lawrence Zitnick, and Devi Parikh.  Vqa: Visual question answering.  In _Proceedings of the IEEE international conference on computer vision_ , pages 2425–2433, 2015.
  * Antonelli et al. [2022] Michela Antonelli, Annika Reinke, Spyridon Bakas, Keyvan Farahani, Annette Kopp-Schneider, Bennett A Landman, Geert Litjens, Bjoern Menze, Olaf Ronneberger, Ronald M Summers, et al.  The medical segmentation decathlon.  _Nature communications_ , 13(1):4128, 2022.
  * Awadalla et al. [2023] Anas Awadalla, Irena Gao, Joshua Gardner, Jack Hessel, Yusuf Hanafy, Wanrong Zhu, Kalyani Marathe, Yonatan Bitton, Samir Gadre, Jenia Jitsev, Simon Kornblith, Pang Wei Koh, Gabriel Ilharco, Mitchell Wortsman, and Ludwig Schmidt.  Openflamingo, March 2023.  URL https://doi.org/10.5281/zenodo.7733589.
  * Bain et al. [2021] Max Bain, Arsha Nagrani, Gül Varol, and Andrew Zisserman.  Frozen in time: A joint video and image encoder for end-to-end retrieval.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 1728–1738, 2021.
  * Baker et al. [2022] Bowen Baker, Ilge Akkaya, Peter Zhokhov, Joost Huizinga, Jie Tang, Adrien Ecoffet, Brandon Houghton, Raul Sampedro, and Jeff Clune.  Learning to play minecraft with video pretraining (vpt).  _OpenAI Blog, June_ , 23, 2022.
  * Bernal et al. [2015] Jorge Bernal, F Javier Sánchez, Gloria Fernández-Esparrach, Debora Gil, Cristina Rodríguez, and Fernando Vilariño.  Wm-dova maps for accurate polyp highlighting in colonoscopy: Validation vs. saliency maps from physicians.  _Computerized medical imaging and graphics_ , 43:99–111, 2015.
  * Berrios et al. [2023] William Berrios, Gautam Mittal, Tristan Thrush, Douwe Kiela, and Amanpreet Singh.  Towards language models that can see: Computer vision through the lens of natural language.  _arXiv preprint arXiv:2306.16410_ , 2023.
  * Bilic et al. [2023] Patrick Bilic, Patrick Christ, Hongwei Bran Li, Eugene Vorontsov, Avi Ben-Cohen, Georgios Kaissis, Adi Szeskin, Colin Jacobs, Gabriel Efrain Humpire Mamani, Gabriel Chartrand, et al.  The liver tumor segmentation benchmark (lits).  _Medical Image Analysis_ , 84:102680, 2023.
  * Bolya et al. [2019] Daniel Bolya, Chong Zhou, Fanyi Xiao, and Yong Jae Lee.  Yolact: Real-time instance segmentation.  In _Proceedings of the IEEE/CVF international conference on computer vision_ , pages 9157–9166, 2019.
  * Bommasani et al. [2021] Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, et al.  On the opportunities and risks of foundation models.  _arXiv preprint arXiv:2108.07258_ , 2021.
  * Brock et al. [2021] Andy Brock, Soham De, Samuel L Smith, and Karen Simonyan.  High-performance large-scale image recognition without normalization.  In _International Conference on Machine Learning_ , pages 1059–1071. PMLR, 2021.
  * Brown et al. [2020] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al.  Language models are few-shot learners.  _Advances in neural information processing systems_ , 33:1877–1901, 2020.
  * Bubeck et al. [2023] Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, et al.  Sparks of artificial general intelligence: Early experiments with gpt-4.  _arXiv preprint arXiv:2303.12712_ , 2023.
  * Byeon et al. [2022] Minwoo Byeon, Beomhee Park, Haecheon Kim, Sungjun Lee, Woonhyuk Baek, and Saehoon Kim.  Coyo-700m: Image-text pair dataset.  https://github.com/kakaobrain/coyo-dataset, 2022.
  * Caesar et al. [2018] Holger Caesar, Jasper Uijlings, and Vittorio Ferrari.  Coco-stuff: Thing and stuff classes in context.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 1209–1218, 2018.
  * Cao et al. [2022] Hanqun Cao, Cheng Tan, Zhangyang Gao, Guangyong Chen, Pheng-Ann Heng, and Stan Z Li.  A survey on generative diffusion model.  _arXiv preprint arXiv:2209.02646_ , 2022.
  * Carion et al. [2020] Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko.  End-to-end object detection with transformers.  In _Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part I 16_ , pages 213–229. Springer, 2020.
  * Carlini et al. [2023] Nicholas Carlini, Milad Nasr, Christopher A Choquette-Choo, Matthew Jagielski, Irena Gao, Anas Awadalla, Pang Wei Koh, Daphne Ippolito, Katherine Lee, Florian Tramer, et al.  Are aligned neural networks adversarially aligned?  _arXiv preprint arXiv:2306.15447_ , 2023.
  * Caron et al. [2021] Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin.  Emerging properties in self-supervised vision transformers.  _arXiv preprint arXiv: 2104.14294_ , 2021.
  * Carpenter et al. [1990] Patricia A Carpenter, Marcel A Just, and Peter Shell.  What one intelligence test measures: a theoretical account of the processing in the raven progressive matrices test.  _Psychological review_ , 97(3):404, 1990.
  * Chakraborty et al. [2023] Souradip Chakraborty, Kasun Weerakoon, Prithvi Poddar, Pratap Tokekar, Amrit Singh Bedi, and Dinesh Manocha.  Re-move: An adaptive policy design approach for dynamic environments via language-based feedback.  _arXiv preprint arXiv:2303.07622_ , 2023.
  * Changpinyo et al. [2021] Soravit Changpinyo, Piyush Sharma, Nan Ding, and Radu Soricut.  Conceptual 12m: Pushing web-scale image-text pre-training to recognize long-tail visual concepts.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 3558–3568, 2021.
  * Chao et al. [2019] Ping Chao, Chao-Yang Kao, Yu-Shan Ruan, Chien-Hsiang Huang, and Youn-Long Lin.  Hardnet: A low memory traffic network.  In _Proceedings of the IEEE/CVF international conference on computer vision_ , pages 3552–3561, 2019.
  * Chavan et al. [2023] Arnav Chavan, Zhuang Liu, Deepak Gupta, Eric Xing, and Zhiqiang Shen.  One-for-all: Generalized lora for parameter-efficient fine-tuning.  _arXiv preprint arXiv:2306.07967_ , 2023.
  * Chen and Dolan [2011] David Chen and William B Dolan.  Collecting highly parallel data for paraphrase evaluation.  In _Proceedings of the 49th annual meeting of the association for computational linguistics: human language technologies_ , pages 190–200, 2011\.
  * Chen et al. [2023a] Jun Chen, Deyao Zhu, Kilichbek Haydarov, Xiang Li, and Mohamed Elhoseiny.  Video chatcaptioner: Towards the enriched spatiotemporal descriptions.  _arXiv preprint arXiv:2304.04227_ , 2023a.
  * Chen et al. [2023b] Keyan Chen, Chenyang Liu, Hao Chen, Haotian Zhang, Wenyuan Li, Zhengxia Zou, and Zhenwei Shi.  Rsprompter: Learning to prompt for remote sensing instance segmentation based on visual foundation model, 2023b.
  * Chen et al. [2021] Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al.  Evaluating large language models trained on code.  _arXiv preprint arXiv:2107.03374_ , 2021.
  * Chen et al. [2023c] Sihan Chen, Xingjian He, Handong Li, Xiaojie Jin, Jiashi Feng, and Jing Liu.  Cosa: Concatenated sample pretrained vision-language foundation model.  _arXiv preprint arXiv: 2306.09085_ , 2023c.
  * Chen et al. [2023d] Tianrun Chen, Lanyun Zhu, Chaotao Ding, Runlong Cao, Shangzhan Zhang, Yan Wang, Zejian Li, Lingyun Sun, Papa Mao, and Ying Zang.  Sam fails to segment anything?–sam-adapter: Adapting sam in underperformed scenes: Camouflage, shadow, and more.  _arXiv preprint arXiv:2304.09148_ , 2023d.
  * Chen et al. [2020a] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton.  A simple framework for contrastive learning of visual representations.  In _International conference on machine learning_ , pages 1597–1607. PMLR, 2020a.
  * Chen et al. [2020b] Ting Chen, Simon Kornblith, Kevin Swersky, Mohammad Norouzi, and Geoffrey E Hinton.  Big self-supervised models are strong semi-supervised learners.  _Advances in neural information processing systems_ , 33:22243–22255, 2020b.
  * Chen et al. [2022a] Ting Chen, Saurabh Saxena, Lala Li, Tsung-Yi Lin, David J Fleet, and Geoffrey E Hinton.  A unified sequence interface for vision tasks.  _Advances in Neural Information Processing Systems_ , 35:31333–31346, 2022a.
  * Chen et al. [2022b] Xi Chen, Xiao Wang, Soravit Changpinyo, AJ Piergiovanni, Piotr Padlewski, Daniel Salz, Sebastian Goodman, Adam Grycner, Basil Mustafa, Lucas Beyer, et al.  Pali: A jointly-scaled multilingual language-image model.  _arXiv preprint arXiv:2209.06794_ , 2022b.
  * Chen et al. [2015] Xinlei Chen, Hao Fang, Tsung-Yi Lin, Ramakrishna Vedantam, Saurabh Gupta, Piotr Dollár, and C Lawrence Zitnick.  Microsoft coco captions: Data collection and evaluation server.  _arXiv preprint arXiv:1504.00325_ , 2015.
  * Chen et al. [2020c] Yen-Chun Chen, Linjie Li, Licheng Yu, Ahmed El Kholy, Faisal Ahmed, Zhe Gan, Yu Cheng, and Jingjing Liu.  Uniter: Universal image-text representation learning.  In _Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XXX_ , pages 104–120. Springer, 2020c.
  * Cheng et al. [2021] Bowen Cheng, Anwesa Choudhuri, Ishan Misra, Alexander Kirillov, Rohit Girdhar, and Alexander G Schwing.  Mask2former for video instance segmentation.  _arXiv preprint arXiv:2112.10764_ , 2021.
  * Cheng et al. [2014] Gong Cheng, Junwei Han, Peicheng Zhou, and Lei Guo.  Multi-class geospatial object detection and geographic image classification based on collection of part detectors.  _ISPRS Journal of Photogrammetry and Remote Sensing_ , 98:119–132, 2014.
  * Cheng and Schwing [2022] Ho Kei Cheng and Alexander G Schwing.  Xmem: Long-term video object segmentation with an atkinson-shiffrin memory model.  In _Computer Vision–ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23–27, 2022, Proceedings, Part XXVIII_ , pages 640–658. Springer, 2022.
  * Cheng et al. [2023] Yangming Cheng, Liulei Li, Yuanyou Xu, Xiaodi Li, Zongxin Yang, Wenguan Wang, and Yi Yang.  Segment and track anything.  _arXiv preprint arXiv:2305.06558_ , 2023.
  * Cherti et al. [2023] Mehdi Cherti, Romain Beaumont, Ross Wightman, Mitchell Wortsman, Gabriel Ilharco, Cade Gordon, Christoph Schuhmann, Ludwig Schmidt, and Jenia Jitsev.  Reproducible scaling laws for contrastive language-image learning.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 2818–2829, 2023.
  * Chiang et al. [2023] Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing.  Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality, March 2023.  URL https://lmsys.org/blog/2023-03-30-vicuna/.
  * Cho et al. [2021] Jaemin Cho, Jie Lei, Hao Tan, and Mohit Bansal.  Unifying vision-and-language tasks via text generation.  In _International Conference on Machine Learning_ , pages 1931–1942. PMLR, 2021.
  * Chowdhery et al. [2022] Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al.  Palm: Scaling language modeling with pathways.  _arXiv preprint arXiv:2204.02311_ , 2022.
  * Christiano et al. [2017] Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei.  Deep reinforcement learning from human preferences.  _Advances in neural information processing systems_ , 30, 2017.
  * Chung et al. [2022] Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al.  Scaling instruction-finetuned language models.  _arXiv preprint arXiv:2210.11416_ , 2022.
  * Conti et al. [2018] Edoardo Conti, Vashisht Madhavan, Felipe Petroski Such, Joel Lehman, Kenneth Stanley, and Jeff Clune.  Improving exploration in evolution strategies for deep reinforcement learning via a population of novelty-seeking agents.  _Advances in neural information processing systems_ , 31, 2018.
  * Croitoru et al. [2023] Florinel-Alin Croitoru, Vlad Hondru, Radu Tudor Ionescu, and Mubarak Shah.  Diffusion models in vision: A survey.  _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , 2023.
  * Dai et al. [2023] Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven Hoi.  Instructblip: Towards general-purpose vision-language models with instruction tuning.  _arXiv preprint arXiv: 2305.06500_ , 2023.
  * Das et al. [2017] Abhishek Das, Satwik Kottur, Khushi Gupta, Avi Singh, Deshraj Yadav, José MF Moura, Devi Parikh, and Dhruv Batra.  Visual dialog.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 326–335, 2017.
  * Desai et al. [2021] Karan Desai, Gaurav Kaul, Zubin Aysola, and Justin Johnson.  Redcaps: Web-curated image-text data created by the people, for the people.  In Joaquin Vanschoren and Sai-Kit Yeung, editors, _Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks 1, NeurIPS Datasets and Benchmarks 2021, December 2021, virtual_ , 2021.  URL https://datasets-benchmarks-proceedings.neurips.cc/paper/2021/hash/e00da03b685a0dd18fb6a08af0923de0-Abstract-round1.html.
  * Dettmers et al. [2023] Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer.  Qlora: Efficient finetuning of quantized llms.  _arXiv preprint arXiv:2305.14314_ , 2023.
  * Devlin et al. [2018] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.  Bert: Pre-training of deep bidirectional transformers for language understanding.  _arXiv preprint arXiv:1810.04805_ , 2018.
  * Ding et al. [2022] Mingyu Ding, Bin Xiao, Noel Codella, Ping Luo, Jingdong Wang, and Lu Yuan.  Davit: Dual attention vision transformers.  In _European Conference on Computer Vision_ , pages 74–92. Springer, 2022.
  * Ding et al. [2023] Yan Ding, Xiaohan Zhang, Chris Paxton, and Shiqi Zhang.  Task and motion planning with large language models for object rearrangement.  _arXiv preprint arXiv: 2303.06247_ , 2023.
  * Djolonga et al. [2021] Josip Djolonga, Jessica Yung, Michael Tschannen, Rob Romijnders, Lucas Beyer, Alexander Kolesnikov, Joan Puigcerver, Matthias Minderer, Alexander D’Amour, Dan Moldovan, et al.  On robustness and transferability of convolutional neural networks.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 16458–16468, 2021.
  * Dong et al. [2022a] Qingxiu Dong, Lei Li, Damai Dai, Ce Zheng, Zhiyong Wu, Baobao Chang, Xu Sun, Jingjing Xu, and Zhifang Sui.  A survey for in-context learning.  _arXiv preprint arXiv:2301.00234_ , 2022a.
  * Dong et al. [2022b] Xiaoyi Dong, Jianmin Bao, Dongdong Chen, Weiming Zhang, Nenghai Yu, Lu Yuan, Dong Chen, and Baining Guo.  Cswin transformer: A general vision transformer backbone with cross-shaped windows.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 12124–12134, 2022b.
  * Dong et al. [2023] Xiaoyi Dong, Jianmin Bao, Yinglin Zheng, Ting Zhang, Dongdong Chen, Hao Yang, Ming Zeng, Weiming Zhang, Lu Yuan, Dong Chen, et al.  Maskclip: Masked self-distillation advances contrastive language-image pretraining.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 10995–11005, 2023.
  * Dosovitskiy et al. [2020] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al.  An image is worth 16x16 words: Transformers for image recognition at scale.  _arXiv preprint arXiv:2010.11929_ , 2020.
  * Dou et al. [2022a] Zi-Yi Dou, Aishwarya Kamath, Zhe Gan, Pengchuan Zhang, Jianfeng Wang, Linjie Li, Zicheng Liu, Ce Liu, Yann LeCun, Nanyun Peng, et al.  Coarse-to-fine vision-language pre-training with fusion in the backbone.  _arXiv preprint arXiv:2206.07643_ , 2022a.
  * Dou et al. [2022b] Zi-Yi Dou, Yichong Xu, Zhe Gan, Jianfeng Wang, Shuohang Wang, Lijuan Wang, Chenguang Zhu, Pengchuan Zhang, Lu Yuan, Nanyun Peng, et al.  An empirical study of training end-to-end vision-and-language transformers.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 18166–18176, 2022b.
  * Doveh et al. [2023] Sivan Doveh, Assaf Arbelle, Sivan Harary, Amit Alfassy, Roei Herzig, Donghyun Kim, Raja Giryes, Rogerio Feris, Rameswar Panda, Shimon Ullman, et al.  Dense and aligned captions (dac) promote compositional reasoning in vl models.  _arXiv preprint arXiv:2305.19595_ , 2023.
  * Driess et al. [2023] Danny Driess, Fei Xia, Mehdi S. M. Sajjadi, Corey Lynch, Aakanksha Chowdhery, Brian Ichter, Ayzaan Wahid, Jonathan Tompson, Quan Vuong, Tianhe Yu, Wenlong Huang, Yevgen Chebotar, Pierre Sermanet, Daniel Duckworth, Sergey Levine, Vincent Vanhoucke, Karol Hausman, Marc Toussaint, Klaus Greff, Andy Zeng, Igor Mordatch, and Pete Florence.  Palm-e: An embodied multimodal language model.  _arXiv preprint arXiv: 2303.03378_ , 2023.
  * Du et al. [2022] Yifan Du, Zikang Liu, Junyi Li, and Wayne Xin Zhao.  A survey of vision-language pre-trained models.  In _IJCAI_ , 2022.
  * Everingham et al. [2015] Mark Everingham, SM Ali Eslami, Luc Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman.  The pascal visual object classes challenge: A retrospective.  _International journal of computer vision_ , 111:98–136, 2015.
  * Eysenbach et al. [2018] Benjamin Eysenbach, Abhishek Gupta, Julian Ibarz, and Sergey Levine.  Diversity is all you need: Learning skills without a reward function.  _arXiv preprint arXiv:1802.06070_ , 2018.
  * Fan et al. [2022] Linxi Fan, Guanzhi Wang, Yunfan Jiang, Ajay Mandlekar, Yuncong Yang, Haoyi Zhu, Andrew Tang, De-An Huang, Yuke Zhu, and Anima Anandkumar.  Minedojo: Building open-ended embodied agents with internet-scale knowledge.  _arXiv preprint arXiv: 2206.08853_ , 2022.
  * Fan et al. [2023] Lizhou Fan, Lingyao Li, Zihui Ma, Sanggyu Lee, Huizi Yu, and Libby Hemphill.  A bibliometric review of large language models research from 2017 to 2023\.  _arXiv preprint arXiv:2304.02020_ , 2023.
  * Fang et al. [2021] Han Fang, Pengfei Xiong, Luhui Xu, and Yu Chen.  Clip2video: Mastering video-text retrieval via image clip.  _arXiv preprint arXiv: 2106.11097_ , 2021.
  * Fang et al. [2023a] Yuxin Fang, Quan Sun, Xinggang Wang, Tiejun Huang, Xinlong Wang, and Yue Cao.  Eva-02: A visual representation for neon genesis.  _arXiv preprint arXiv:2303.11331_ , 2023a.
  * Fang et al. [2023b] Yuxin Fang, Wen Wang, Binhui Xie, Quan Sun, Ledell Wu, Xinggang Wang, Tiejun Huang, Xinlong Wang, and Yue Cao.  Eva: Exploring the limits of masked visual representation learning at scale.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 19358–19369, 2023b.
  * Fedorov et al. [2012] Andriy Fedorov, Reinhard Beichel, Jayashree Kalpathy-Cramer, Julien Finet, Jean-Christophe Fillion-Robin, Sonia Pujol, Christian Bauer, Dominique Jennings, Fiona Fennessy, Milan Sonka, et al.  3d slicer as an image computing platform for the quantitative imaging network.  _Magnetic resonance imaging_ , 30(9):1323–1341, 2012.
  * Fei-Fei et al. [2009] Li Fei-Fei, Jia Deng, and Kai Li.  Imagenet: Constructing a large-scale image database.  _Journal of vision_ , 9(8):1037–1037, 2009.
  * Gadre et al. [2023] Samir Yitzhak Gadre, Gabriel Ilharco, Alex Fang, Jonathan Hayase, Georgios Smyrnis, Thao Nguyen, Ryan Marten, Mitchell Wortsman, Dhruba Ghosh, Jieyu Zhang, et al.  Datacomp: In search of the next generation of multimodal datasets.  _arXiv preprint arXiv:2304.14108_ , 2023.
  * Gan et al. [2022] Zhe Gan, Linjie Li, Chunyuan Li, Lijuan Wang, Zicheng Liu, Jianfeng Gao, et al.  Vision-language pre-training: Basics, recent advances, and future trends.  _Foundations and Trends® in Computer Graphics and Vision_ , 14(3–4):163–352, 2022.
  * Gao et al. [2023a] Peng Gao, Jiaming Han, Renrui Zhang, Ziyi Lin, Shijie Geng, Aojun Zhou, Wei Zhang, Pan Lu, Conghui He, Xiangyu Yue, Hongsheng Li, and Yu Qiao.  Llama-adapter v2: Parameter-efficient visual instruction model.  _arXiv preprint arXiv:2304.15010_ , 2023a.
  * Gao et al. [2021] Tianyu Gao, Adam Fisch, and Danqi Chen.  Making pre-trained language models better few-shot learners, 2021.
  * Gao et al. [2023b] Yifan Gao, Wei Xia, Dingdu Hu, and Xin Gao.  Desam: Decoupling segment anything model for generalizable medical image segmentation.  _arXiv preprint arXiv:2306.00499_ , 2023b.
  * Ge et al. [2023] Yingqiang Ge, Wenyue Hua, Kai Mei, Jianchao Ji, Juntao Tan, Shuyuan Xu, Zelong Li, and Yongfeng Zhang.  Openagi: When llm meets domain experts.  _arXiv preprint arXiv: 2304.04370_ , 2023.
  * Gemmeke et al. [2017] Jort F Gemmeke, Daniel PW Ellis, Dylan Freedman, Aren Jansen, Wade Lawrence, R Channing Moore, Manoj Plakal, and Marvin Ritter.  Audio set: An ontology and human-labeled dataset for audio events.  In _2017 IEEE international conference on acoustics, speech and signal processing (ICASSP)_ , pages 776–780. IEEE, 2017.
  * Ghiasi et al. [2021a] Golnaz Ghiasi, Xiuye Gu, Yin Cui, and Tsung-Yi Lin.  Scaling open-vocabulary image segmentation with image-level labels.  _arXiv preprint arXiv: 2112.12143_ , 2021a.
  * Ghiasi et al. [2021b] Golnaz Ghiasi, Barret Zoph, Ekin D Cubuk, Quoc V Le, and Tsung-Yi Lin.  Multi-task self-training for learning general representations.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 8856–8865, 2021b.
  * Girdhar et al. [2023] Rohit Girdhar, Alaaeldin El-Nouby, Zhuang Liu, Mannat Singh, Kalyan Vasudev Alwala, Armand Joulin, and Ishan Misra.  Imagebind: One embedding space to bind them all.  _CVPR_ , 2023.
  * Gong et al. [2023] Shizhan Gong, Yuan Zhong, Wenao Ma, Jinpeng Li, Zhao Wang, Jingyang Zhang, Pheng-Ann Heng, and Qi Dou.  3dsam-adapter: Holistic adaptation of sam from 2d to 3d for promptable medical image segmentation.  _ArXiv_ , abs/2306.13465, 2023.
  * Goyal et al. [2016] Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh.  Making the v in vqa matter: Elevating the role of image understanding in visual question answering.  _Computer Vision And Pattern Recognition_ , 2016.  doi: 10.1007/s11263-018-1116-0.
  * Grauman et al. [2022] Kristen Grauman, Andrew Westbury, Eugene Byrne, Zachary Chavis, Antonino Furnari, Rohit Girdhar, Jackson Hamburger, Hao Jiang, Miao Liu, Xingyu Liu, et al.  Ego4d: Around the world in 3,000 hours of egocentric video.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 18995–19012, 2022.
  * Greshake et al. [2023] Kai Greshake, Sahar Abdelnabi, Shailesh Mishra, Christoph Endres, Thorsten Holz, and Mario Fritz.  More than you’ve asked for: A comprehensive analysis of novel prompt injection threats to application-integrated large language models.  _arXiv preprint arXiv:2302.12173_ , 2023.
  * Gu et al. [2021] Xiuye Gu, Tsung-Yi Lin, Weicheng Kuo, and Yin Cui.  Open-vocabulary object detection via vision and language knowledge distillation.  _arXiv preprint arXiv:2104.13921_ , 2021.
  * Guo et al. [2023] Ziyu Guo, Yiwen Tang, Renrui Zhang, Dong Wang, Zhigang Wang, Bin Zhao, and Xuelong Li.  Viewrefer: Grasp the multi-view knowledge for 3d visual grounding with gpt and prototype guidance.  _arXiv preprint arXiv: 2303.16894_ , 2023.
  * Gupta et al. [2019] Agrim Gupta, Piotr Dollar, and Ross Girshick.  Lvis: A dataset for large vocabulary instance segmentation.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pages 5356–5364, 2019.
  * Gupta and Kembhavi [2023] Tanmay Gupta and Aniruddha Kembhavi.  Visual programming: Compositional visual reasoning without training.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 14953–14962, 2023.
  * Gurari et al. [2018] Danna Gurari, Qing Li, Abigale J Stangl, Anhong Guo, Chi Lin, Kristen Grauman, Jiebo Luo, and Jeffrey P Bigham.  Vizwiz grand challenge: Answering visual questions from blind people.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 3608–3617, 2018.
  * Guzhov et al. [2021] Andrey Guzhov, Federico Raue, Jörn Hees, and Andreas Dengel.  Audioclip: Extending clip to image, text and audio.  _arXiv preprint arXiv: 2106.13043_ , 2021.
  * Hall et al. [2023] Siobhan Mackenzie Hall, Fernanda Gonçalves Abrantes, Hanwen Zhu, Grace Sodunke, Aleksandar Shtedritski, and Hannah Rose Kirk.  Visogender: A dataset for benchmarking gender bias in image-text pronoun resolution.  _arXiv preprint arXiv:2306.12424_ , 2023.
  * Hao et al. [2022] Yaru Hao, Haoyu Song, Li Dong, Shaohan Huang, Zewen Chi, Wenhui Wang, Shuming Ma, and Furu Wei.  Language models are general-purpose interfaces.  _arXiv preprint arXiv: 2206.06336_ , 2022.
  * Harley et al. [2022] Adam W Harley, Zhaoyuan Fang, and Katerina Fragkiadaki.  Particle video revisited: Tracking through occlusions using point trajectories.  In _European Conference on Computer Vision_ , pages 59–75. Springer, 2022.
  * He et al. [2016] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 770–778, 2016.
  * He et al. [2017] Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick.  Mask r-cnn.  In _Proceedings of the IEEE international conference on computer vision_ , pages 2961–2969, 2017.
  * He et al. [2020] Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick.  Momentum contrast for unsupervised visual representation learning.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pages 9729–9738, 2020.
  * He et al. [2022] Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick.  Masked autoencoders are scalable vision learners.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 16000–16009, 2022.
  * Heller et al. [2021] Nicholas Heller, Fabian Isensee, Klaus H Maier-Hein, Xiaoshuai Hou, Chunmei Xie, Fengyi Li, Yang Nan, Guangrui Mu, Zhiyong Lin, Miofei Han, et al.  The state of the art in kidney and kidney tumor segmentation in contrast-enhanced ct imaging: Results of the kits19 challenge.  _Medical image analysis_ , 67:101821, 2021.
  * Hendrycks et al. [2020] Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt.  Measuring massive multitask language understanding.  _arXiv preprint arXiv:2009.03300_ , 2020.
  * Hendrycks et al. [2021] Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang, Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, et al.  The many faces of robustness: A critical analysis of out-of-distribution generalization.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 8340–8349, 2021.
  * Hoffmann et al. [2022] Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al.  Training compute-optimal large language models.  _arXiv preprint arXiv:2203.15556_ , 2022.
  * Hossain et al. [2019] MD Zakir Hossain, Ferdous Sohel, Mohd Fairuz Shiratuddin, and Hamid Laga.  A comprehensive survey of deep learning for image captioning.  _ACM Computing Surveys (CsUR)_ , 51(6):1–36, 2019\.
  * Hu [2023] Anthony Hu.  Neural world models for computer vision.  _arXiv preprint arXiv:2306.09179_ , 2023.
  * Hu et al. [2021] Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, and Weizhu Chen.  Lora: Low-rank adaptation of large language models.  _International Conference On Learning Representations_ , 2021.
  * Hu et al. [2023] Mingzhe Hu, Shaoyan Pan, Yuheng Li, and Xiaofeng Yang.  Advancing medical imaging with language models: A journey from n-grams to chatgpt.  _arXiv preprint arXiv: 2304.04920_ , 2023.
  * Hu et al. [2016] Ronghang Hu, Marcus Rohrbach, and Trevor Darrell.  Segmentation from natural language expressions.  In _Computer Vision–ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11–14, 2016, Proceedings, Part I 14_ , pages 108–124. Springer, 2016.
  * Huang and Chang [2023] Jie Huang and Kevin Chen-Chuan Chang.  Towards reasoning in large language models: A survey.  _ACL Findings_ , 2023.
  * Huang et al. [2023a] Shaohan Huang, Li Dong, Wenhui Wang, Yaru Hao, Saksham Singhal, Shuming Ma, Tengchao Lv, Lei Cui, Owais Khan Mohammed, Barun Patra, Qiang Liu, Kriti Aggarwal, Zewen Chi, Johan Bjorck, Vishrav Chaudhary, Subhojit Som, Xia Song, and Furu Wei.  Language is not all you need: Aligning perception with language models.  _arXiv preprint arXiv: 2302.14045_ , 2023a.
  * Huang et al. [2023b] Yihao Huang, Yue Cao, Tianlin Li, Felix Juefei-Xu, Di Lin, Ivor W Tsang, Yang Liu, and Qing Guo.  On the robustness of segment anything.  _arXiv preprint arXiv:2305.16220_ , 2023b.
  * Hudson and Manning [2019] Drew A Hudson and Christopher D Manning.  Gqa: A new dataset for real-world visual reasoning and compositional question answering.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pages 6700–6709, 2019.
  * Huo et al. [2021] Yuqi Huo, Manli Zhang, Guangzhen Liu, Haoyu Lu, Yizhao Gao, Guoxing Yang, Jingyuan Wen, Heng Zhang, Baogui Xu, Weihao Zheng, et al.  Wenlan: Bridging vision and language by large-scale multi-modal pre-training.  _arXiv preprint arXiv:2103.06561_ , 2021.
  * Ilharco et al. [2021] Gabriel Ilharco, Mitchell Wortsman, Ross Wightman, Cade Gordon, Nicholas Carlini, Rohan Taori, Achal Dave, Vaishaal Shankar, Hongseok Namkoong, John Miller, Hannaneh Hajishirzi, Ali Farhadi, and Ludwig Schmidt.  Openclip, July 2021.  URL https://doi.org/10.5281/zenodo.5143773.  If you use this software, please cite it as below.
  * Jain et al. [2023] Neel Jain, Khalid Saifullah, Yuxin Wen, John Kirchenbauer, Manli Shu, Aniruddha Saha, Micah Goldblum, Jonas Geiping, and Tom Goldstein.  Bring your own data! self-supervised evaluation for large language models, 2023.
  * Jha et al. [2020] Debesh Jha, Pia H Smedsrud, Michael A Riegler, Pål Halvorsen, Thomas de Lange, Dag Johansen, and Håvard D Johansen.  Kvasir-seg: A segmented polyp dataset.  In _MultiMedia Modeling: 26th International Conference, MMM 2020, Daejeon, South Korea, January 5–8, 2020, Proceedings, Part II 26_ , pages 451–462. Springer, 2020.
  * Ji et al. [2022] Ge-Peng Ji, Guobao Xiao, Yu-Cheng Chou, Deng-Ping Fan, Kai Zhao, Geng Chen, and Luc Van Gool.  Video polyp segmentation: A deep learning perspective.  _Machine Intelligence Research_ , 19(6):531–549, 2022.
  * Ji et al. [2018] Shunping Ji, Shiqing Wei, and Meng Lu.  Fully convolutional networks for multisource building extraction from an open aerial and satellite imagery data set.  _IEEE Transactions on geoscience and remote sensing_ , 57(1):574–586, 2018.
  * Jia et al. [2021a] Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig.  Scaling up visual and vision-language representation learning with noisy text supervision.  In _International Conference on Machine Learning_ , pages 4904–4916. PMLR, 2021a.
  * Jia et al. [2021b] Xinyu Jia, Chuang Zhu, Minzhen Li, Wenqi Tang, and Wenli Zhou.  Llvip: A visible-infrared paired dataset for low-light vision.  In _Proceedings of the IEEE/CVF international conference on computer vision_ , pages 3496–3504, 2021b.
  * Jiang et al. [2022] Yunfan Jiang, Agrim Gupta, Zichen Zhang, Guanzhi Wang, Yongqiang Dou, Yanjun Chen, Li Fei-Fei, Anima Anandkumar, Yuke Zhu, and Linxi Fan.  Vima: General robot manipulation with multimodal prompts.  _arXiv preprint arXiv: 2210.03094_ , 2022.
  * Jingyi et al. [2023] Zhang Jingyi, Huang Jiaxing, Jin Sheng, and Lu Shijian.  Vision-language models for vision tasks: A survey.  _arXiv preprint arXiv:2304.00685_ , 2023.
  * Jocher et al. [2023] Glenn Jocher, Ayush Chaurasia, and Jing Qiu.  Yolo by ultralytics.  https://github.com/ultralytics/ultralytics, 2023.
  * Kamath et al. [2021] Aishwarya Kamath, Mannat Singh, Yann LeCun, Gabriel Synnaeve, Ishan Misra, and Nicolas Carion.  Mdetr - modulated detection for end-to-end multi-modal understanding.  _arXiv preprint arXiv: 2104.12763_ , 2021.
  * Kazemzadeh et al. [2014] Sahar Kazemzadeh, Vicente Ordonez, Mark Matten, and Tamara Berg.  Referitgame: Referring to objects in photographs of natural scenes.  In _Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP)_ , pages 787–798, 2014.
  * Khan et al. [2022] Salman Khan, Muzammal Naseer, Munawar Hayat, Syed Waqas Zamir, Fahad Shahbaz Khan, and Mubarak Shah.  Transformers in vision: A survey.  _ACM computing surveys (CSUR)_ , 54(10s):1–41, 2022.
  * Khoreva et al. [2019] Anna Khoreva, Anna Rohrbach, and Bernt Schiele.  Video object segmentation with language referring expressions.  In _Computer Vision–ACCV 2018: 14th Asian Conference on Computer Vision, Perth, Australia, December 2–6, 2018, Revised Selected Papers, Part IV 14_ , pages 123–141. Springer, 2019.
  * Kiela et al. [2020] Douwe Kiela, Hamed Firooz, Aravind Mohan, Vedanuj Goswami, Amanpreet Singh, Pratik Ringshia, and Davide Testuggine.  The hateful memes challenge: Detecting hate speech in multimodal memes.  _Advances in neural information processing systems_ , 33:2611–2624, 2020.
  * Kim et al. [2023] Geewook Kim, Hodong Lee, Daehee Kim, Haeji Jung, Sanghee Park, Yoonsik Kim, Sangdoo Yun, Taeho Kil, Bado Lee, and Seunghyun Park.  Cream: Visually-situated natural language understanding with contrastive reading model and frozen large language models.  _arXiv preprint arXiv:2305.15080_ , 2023.
  * Kirillov et al. [2023] Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan-Yen Lo, et al.  Segment anything.  _arXiv preprint arXiv:2304.02643_ , 2023.
  * Koh et al. [2023] Jing Yu Koh, Ruslan Salakhutdinov, and Daniel Fried.  Grounding language models to images for multimodal inputs and outputs.  2023\.
  * Krasin et al. [2017] Ivan Krasin, Tom Duerig, Neil Alldrin, Vittorio Ferrari, Sami Abu-El-Haija, Alina Kuznetsova, Hassan Rom, Jasper Uijlings, Stefan Popov, Andreas Veit, et al.  Openimages: A public dataset for large-scale multi-label and multi-class image classification.  _Dataset available from https://github. com/openimages_ , 2(3):18, 2017.
  * Krishna et al. [2017] Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A Shamma, et al.  Visual genome: Connecting language and vision using crowdsourced dense image annotations.  _International journal of computer vision_ , 123:32–73, 2017.
  * Krizhevsky et al. [2012] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton.  Imagenet classification with deep convolutional neural networks.  _Advances in neural information processing systems_ , 25, 2012.
  * Kumar et al. [2019] Neeraj Kumar, Ruchika Verma, Deepak Anand, Yanning Zhou, Omer Fahri Onder, Efstratios Tsougenis, Hao Chen, Pheng-Ann Heng, Jiahui Li, Zhiqiang Hu, et al.  A multi-organ nucleus segmentation challenge.  _IEEE transactions on medical imaging_ , 39(5):1380–1391, 2019.
  * Kwon et al. [2022] Gukyeong Kwon, Zhaowei Cai, Avinash Ravichandran, Erhan Bas, Rahul Bhotika, and Stefano Soatto.  Masked vision and language modeling for multi-modal representation learning.  _arXiv preprint arXiv:2208.02131_ , 2022.
  * Kwon et al. [2023] Gukyeong Kwon, Zhaowei Cai, Avinash Ravichandran, Erhan Bas, Rahul Bhotika, and Stefano Soatto.  Masked vision and language modeling for multi-modal representation learning.  In _The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023_. OpenReview.net, 2023\.  URL https://openreview.net/pdf?id=ZhuXksSJYWn.
  * Lei et al. [2021] Wenhui Lei, Wei Xu, Ran Gu, Hao Fu, Shaoting Zhang, Shichuan Zhang, and Guotai Wang.  Contrastive learning of relative position regression for one-shot object localization in 3d medical images.  In _Medical Image Computing and Computer Assisted Intervention–MICCAI 2021: 24th International Conference, Strasbourg, France, September 27–October 1, 2021, Proceedings, Part II 24_ , pages 155–165. Springer, 2021.
  * Lester et al. [2021] Brian Lester, Rami Al-Rfou, and Noah Constant.  The power of scale for parameter-efficient prompt tuning.  _arXiv preprint arXiv:2104.08691_ , 2021.
  * Lewis et al. [2019] Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Ves Stoyanov, and Luke Zettlemoyer.  Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension.  _arXiv preprint arXiv:1910.13461_ , 2019.
  * Li et al. [2023a] Bo Li, Gexiang Fang, Yang Yang, Quansen Wang, Wei Ye, Wen Zhao, and Shikun Zhang.  Evaluating chatgpt’s information extraction capabilities: An assessment of performance, explainability, calibration, and faithfulness.  _arXiv preprint arXiv:2304.11633_ , 2023a.
  * Li et al. [2023b] Bo Li, Yuanhan Zhang, Liangyu Chen, Jinghao Wang, Fanyi Pu, Jingkang Yang, Chunyuan Li, and Ziwei Liu.  Mimic-it: Multi-modal in-context instruction tuning.  _arXiv preprint arXiv:2306.05425_ , 2023b.
  * Li et al. [2021] Junnan Li, Ramprasaath Selvaraju, Akhilesh Gotmare, Shafiq Joty, Caiming Xiong, and Steven Chu Hong Hoi.  Align before fuse: Vision and language representation learning with momentum distillation.  _Advances in neural information processing systems_ , 34:9694–9705, 2021.
  * Li et al. [2022a] Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi.  Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation.  In _International Conference on Machine Learning_ , pages 12888–12900. PMLR, 2022a.
  * Li et al. [2023c] Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi.  Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models.  _arXiv preprint arXiv:2301.12597_ , 2023c.
  * Li et al. [2022b] Liunian Harold Li, Pengchuan Zhang, Haotian Zhang, Jianwei Yang, Chunyuan Li, Yiwu Zhong, Lijuan Wang, Lu Yuan, Lei Zhang, Jenq-Neng Hwang, et al.  Grounded language-image pre-training.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 10965–10975, 2022b.
  * Li and Liang [2021] Xiang Lisa Li and Percy Liang.  Prefix-tuning: Optimizing continuous prompts for generation.  _arXiv preprint arXiv:2101.00190_ , 2021.
  * Li et al. [2023d] Xianhang Li, Zeyu Wang, and Cihang Xie.  Clipa-v2: Scaling clip training with 81.1accuracy within a $10,000 budget; an extra $4,000 unlocks 81.8 _arXiv preprint arXiv: 2306.15658_ , 2023d.
  * Li et al. [2023e] Xianhang Li, Zeyu Wang, and Cihang Xie.  An inverse scaling law for clip training.  _arXiv preprint arXiv: 2305.07017_ , 2023e.
  * Li et al. [2023f] Yanghao Li, Haoqi Fan, Ronghang Hu, Christoph Feichtenhofer, and Kaiming He.  Scaling language-image pre-training via masking.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 23390–23400, 2023f.
  * Li et al. [2023g] Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Wayne Xin Zhao, and Ji-Rong Wen.  Evaluating object hallucination in large vision-language models.  _arXiv preprint arXiv:2305.10355_ , 2023g.
  * Liangliang et al. [2023] Yao Liangliang, Zuo Haobo, Zheng Guangze, Fu Changhong, and Pan Jia.  Sam-da: Uav tracks anything at night with sam-powered domain adaptation.  _arXiv:2307.01024_ , 2023.
  * Lin et al. [2014] Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick.  Microsoft coco: Common objects in context.  In _Computer Vision–ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13_ , pages 740–755. Springer, 2014.
  * Lin et al. [2017a] Tsung-Yi Lin, Piotr Dollár, Ross Girshick, Kaiming He, Bharath Hariharan, and Serge Belongie.  Feature pyramid networks for object detection.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 2117–2125, 2017a.
  * Lin et al. [2017b] Tsung-Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár.  Focal loss for dense object detection.  In _Proceedings of the IEEE international conference on computer vision_ , pages 2980–2988, 2017b.
  * Lin et al. [2023] Wei Lin, Leonid Karlinsky, Nina Shvetsova, Horst Possegger, Mateusz Kozinski, Rameswar Panda, Rogerio Feris, Hilde Kuehne, and Horst Bischof.  Match, expand and improve: Unsupervised finetuning for zero-shot action recognition with language knowledge.  _arXiv preprint arXiv:2303.08914_ , 2023.
  * Lin and Chen [2023] Yen-Ting Lin and Yun-Nung Chen.  Llm-eval: Unified multi-dimensional automatic evaluation for open-domain conversations with large language models.  _arXiv preprint arXiv:2305.13711_ , 2023.
  * Liu et al. [2023a] Fan Liu, Delong Chen, Zhangqingyun Guan, Xiaocong Zhou, Jiale Zhu, and Jun Zhou.  Remoteclip: A vision language foundation model for remote sensing.  _arXiv preprint arXiv: 2306.11029_ , 2023a.
  * Liu et al. [2023b] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee.  Visual instruction tuning.  2023b.
  * Liu et al. [2023c] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee.  Visual instruction tuning.  _arXiv preprint arXiv:2304.08485_ , 2023c.
  * Liu et al. [2023d] Jiongnan Liu, Jiajie Jin, Zihan Wang, Jiehan Cheng, Zhicheng Dou, and Ji-Rong Wen.  Reta-llm: A retrieval-augmented large language model toolkit.  _arXiv preprint arXiv:2306.05212_ , 2023d.
  * Liu et al. [2023e] Junjia Liu, Zhihao Li, Sylvain Calinon, and Fei Chen.  Softgpt: Learn goal-oriented soft object manipulation skills by generative pre-trained heterogeneous graph transformer.  _arXiv preprint arXiv:2306.12677_ , 2023e.
  * Liu et al. [2023f] Shikun Liu, Linxi Fan, Edward Johns, Zhiding Yu, Chaowei Xiao, and Anima Anandkumar.  Prismer: A vision-language model with an ensemble of experts.  _arXiv preprint arXiv: 2303.02506_ , 2023f.
  * [174] Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang, Chunyuan Li, Jianwei Yang, Hang Su, Jun Zhu, et al.  Grounding dino: Marrying dino with grounded pre-training for open-set object detection.  _arXiv preprint arXiv:2303.05499_.
  * Liu and Zuo [2023] Weihua Liu and Yong Zuo.  Stone needle: A general multimodal large-scale model framework towards healthcare.  _arXiv preprint arXiv: 2306.16034_ , 2023.
  * Liu et al. [2021a] Xiao Liu, Kaixuan Ji, Yicheng Fu, Weng Lam Tam, Zhengxiao Du, Zhilin Yang, and Jie Tang.  P-tuning v2: Prompt tuning can be comparable to fine-tuning universally across scales and tasks.  _arXiv preprint arXiv:2110.07602_ , 2021a.
  * Liu et al. [2023g] Yihao Liu, Jiaming Zhang, Zhangcong She, Amir Kheradmand, and Mehran Armand.  Samm (segment any medical model): A 3d slicer integration to sam.  _arXiv preprint arXiv:2304.05622_ , 2023g.
  * Liu et al. [2019] Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov.  Roberta: A robustly optimized bert pretraining approach, 2019.
  * Liu et al. [2021b] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo.  Swin transformer: Hierarchical vision transformer using shifted windows.  In _Proceedings of the IEEE/CVF international conference on computer vision_ , pages 10012–10022, 2021b.
  * Long et al. [2022] Siqu Long, Feiqi Cao, Soyeon Caren Han, and Haiqin Yang.  Vision-and-language pretrained models: A survey.  In _IJCAI_ , 2022.
  * Lu et al. [2019] Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee.  Vilbert: Pretraining task-agnostic visiolinguistic representations for vision-and-language tasks.  _Advances in neural information processing systems_ , 32, 2019.
  * Lu et al. [2021] Pan Lu, Liang Qiu, Jiaqi Chen, Tony Xia, Yizhou Zhao, Wei Zhang, Zhou Yu, Xiaodan Liang, and Song-Chun Zhu.  Iconqa: A new benchmark for abstract diagram understanding and visual language reasoning.  _arXiv preprint arXiv:2110.13214_ , 2021.
  * Lu et al. [2022] Pan Lu, Swaroop Mishra, Tony Xia, Liang Qiu, Kai-Wei Chang, Song-Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan.  Learn to explain: Multimodal reasoning via thought chains for science question answering.  In _The 36th Conference on Neural Information Processing Systems (NeurIPS)_ , 2022.
  * Lu et al. [2023] Pan Lu, Baolin Peng, Hao Cheng, Michel Galley, Kai-Wei Chang, Ying Nian Wu, Song-Chun Zhu, and Jianfeng Gao.  Chameleon: Plug-and-play compositional reasoning with large language models.  _arXiv preprint arXiv:2304.09842_ , 2023.
  * Lüddecke and Ecker [2022] Timo Lüddecke and Alexander Ecker.  Image segmentation using text and image prompts.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 7086–7096, 2022.
  * Luo et al. [2023] Ruipu Luo, Ziwang Zhao, Min Yang, Junwei Dong, Minghui Qiu, Pengcheng Lu, Tao Wang, and Zhongyu Wei.  Valley: Video assistant with large language model enhanced ability.  _arXiv preprint arXiv: 2306.07207_ , 2023.
  * Lynch et al. [2022] Corey Lynch, Ayzaan Wahid, Jonathan Tompson, Tianli Ding, James Betker, Robert Baruch, Travis Armstrong, and Pete Florence.  Interactive language: Talking to robots in real time.  _arXiv preprint arXiv:2210.06407_ , 2022.
  * Lyu et al. [2023] Chenyang Lyu, Bingshuai Liu, Minghao Wu, Zefeng Du, Xinting Huang, Zhaopeng Tu, Shuming Shi, and Longyue Wang.  Macaw-llm: Multi-modal language modeling with image, video, audio, and text integration.  https://github.com/lyuchenyang/Macaw-LLM, 2023.
  * Ma and Wang [2023] Jun Ma and Bo Wang.  Segment anything in medical images.  _arXiv preprint arXiv: 2304.12306_ , 2023.
  * Maaz et al. [2023] Muhammad Maaz, Hanoona Rasheed, Salman Khan, and Fahad Shahbaz Khan.  Video-chatgpt: Towards detailed video understanding via large vision and language models.  _arXiv preprint arXiv: 2306.05424_ , 2023.
  * Mahajan et al. [2018] Dhruv Mahajan, Ross Girshick, Vignesh Ramanathan, Kaiming He, Manohar Paluri, Yixuan Li, Ashwin Bharambe, and Laurens Van Der Maaten.  Exploring the limits of weakly supervised pretraining.  In _Proceedings of the European conference on computer vision (ECCV)_ , pages 181–196, 2018.
  * Mai et al. [2023] Jinjie Mai, Jun Chen, Bing Li, Guocheng Qian, Mohamed Elhoseiny, and Bernard Ghanem.  Llm as a robotic brain: Unifying egocentric memory and control.  _arXiv preprint arXiv:2304.09349_ , 2023.
  * Mao et al. [2016] Junhua Mao, Jonathan Huang, Alexander Toshev, Oana Camburu, Alan L Yuille, and Kevin Murphy.  Generation and comprehension of unambiguous object descriptions.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 11–20, 2016.
  * Marino et al. [2019] Kenneth Marino, Mohammad Rastegari, Ali Farhadi, and Roozbeh Mottaghi.  Ok-vqa: A visual question answering benchmark requiring external knowledge.  In _Proceedings of the IEEE/cvf conference on computer vision and pattern recognition_ , pages 3195–3204, 2019.
  * Maus et al. [2023] Natalie Maus, Patrick Chao, Eric Wong, and Jacob Gardner.  Adversarial prompting for black box foundation models.  _arXiv preprint arXiv:2302.04237_ , 2023.
  * Minderer et al. [2022] Matthias Minderer, Alexey Gritsenko, Austin Stone, Maxim Neumann, Dirk Weissenborn, Alexey Dosovitskiy, Aravindh Mahendran, Anurag Arnab, Mostafa Dehghani, Zhuoran Shen, Xiao Wang, Xiaohua Zhai, Thomas Kipf, and Neil Houlsby.  Simple open-vocabulary object detection with vision transformers.  _arXiv preprint arXiv: 2205.06230_ , 2022.
  * Mishra et al. [2019] Anand Mishra, Shashank Shekhar, Ajeet Kumar Singh, and Anirban Chakraborty.  Ocr-vqa: Visual question answering by reading text in images.  In _2019 international conference on document analysis and recognition (ICDAR)_ , pages 947–952. IEEE, 2019.
  * Moor et al. [2023] Michael Moor, Oishi Banerjee, Zahra Shakeri Hossein Abad, Harlan M Krumholz, Jure Leskovec, Eric J Topol, and Pranav Rajpurkar.  Foundation models for generalist medical artificial intelligence.  _Nature_ , 616(7956):259–265, 2023.
  * Mottaghi et al. [2014] Roozbeh Mottaghi, Xianjie Chen, Xiaobai Liu, Nam-Gyu Cho, Seong-Whan Lee, Sanja Fidler, Raquel Urtasun, and Alan Yuille.  The role of context for object detection and semantic segmentation in the wild.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 891–898, 2014.
  * Mu et al. [2022] Norman Mu, Alexander Kirillov, David Wagner, and Saining Xie.  Slip: Self-supervision meets language-image pre-training.  In _Computer Vision–ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23–27, 2022, Proceedings, Part XXVI_ , pages 529–544. Springer, 2022.
  * Mu et al. [2023] Yao Mu, Qinglong Zhang, Mengkang Hu, Wenhai Wang, Mingyu Ding, Jun Jin, Bin Wang, Jifeng Dai, Yu Qiao, and Ping Luo.  Embodiedgpt: Vision-language pre-training via embodied chain of thought.  _arXiv preprint arXiv:2305.15021_ , 2023.
  * Mündler et al. [2023] Niels Mündler, Jingxuan He, Slobodan Jenko, and Martin Vechev.  Self-contradictory hallucinations of large language models: Evaluation, detection and mitigation.  _arXiv preprint arXiv:2305.15852_ , 2023.
  * Nagaraja et al. [2016] Varun K Nagaraja, Vlad I Morariu, and Larry S Davis.  Modeling context between objects for referring expression understanding.  In _Computer Vision–ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11–14, 2016, Proceedings, Part IV 14_ , pages 792–807. Springer, 2016.
  * OpenAI [2023] OpenAI.  Gpt-4 technical report.  _PREPRINT_ , 2023.
  * Ordonez et al. [2011] Vicente Ordonez, Girish Kulkarni, and Tamara Berg.  Im2text: Describing images using 1 million captioned photographs.  _Advances in neural information processing systems_ , 24, 2011.
  * Pan et al. [2023] Junting Pan, Ziyi Lin, Yuying Ge, Xiatian Zhu, Renrui Zhang, Yi Wang, Yu Qiao, and Hongsheng Li.  Retrieving-to-answer: Zero-shot video question answering with frozen large language models.  _arXiv preprint arXiv:2306.11732_ , 2023.
  * Parisi et al. [2019] German I Parisi, Ronald Kemker, Jose L Part, Christopher Kanan, and Stefan Wermter.  Continual lifelong learning with neural networks: A review.  _Neural networks_ , 113:54–71, 2019.
  * Park et al. [2023] Jeongeun Park, Seungwon Lim, Joonhyung Lee, Sangbeom Park, Youngjae Yu, and Sungjoon Choi.  Clara: Classifying and disambiguating user commands for reliable interactive robotic agents.  _arXiv preprint arXiv:2306.10376_ , 2023.
  * Peng et al. [2022] Zhiliang Peng, Li Dong, Hangbo Bao, Qixiang Ye, and Furu Wei.  Beit v2: Masked image modeling with vector-quantized visual tokenizers.  _arXiv preprint arXiv:2208.06366_ , 2022.
  * Peng et al. [2023] Zhiliang Peng, Wenhui Wang, Li Dong, Yaru Hao, Shaohan Huang, Shuming Ma, and Furu Wei.  Kosmos-2: Grounding multimodal large language models to the world.  _arXiv preprint arXiv: 2306.14824_ , 2023.
  * Perez and Ribeiro [2022] Fábio Perez and Ian Ribeiro.  Ignore previous prompt: Attack techniques for language models.  _arXiv preprint arXiv:2211.09527_ , 2022.
  * Pont-Tuset et al. [2020] Jordi Pont-Tuset, Jasper Uijlings, Soravit Changpinyo, Radu Soricut, and Vittorio Ferrari.  Connecting vision and language with localized narratives.  In _Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part V 16_ , pages 647–664. Springer, 2020.
  * Qiu et al. [2023] Zhongxi Qiu, Yan Hu, Heng Li, and Jiang Liu.  Learnable ophthalmology sam.  _arXiv preprint arXiv:2304.13425_ , 2023.
  * Radford et al. [2019] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al.  Language models are unsupervised multitask learners.  _OpenAI blog_ , 1(8):9, 2019.
  * Radford et al. [2021] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al.  Learning transferable visual models from natural language supervision.  In _International conference on machine learning_ , pages 8748–8763. PMLR, 2021.
  * Raffel et al. [2020] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu.  Exploring the limits of transfer learning with a unified text-to-text transformer.  _The Journal of Machine Learning Research_ , 21(1):5485–5551, 2020.
  * Rajič et al. [2023] Frano Rajič, Lei Ke, Yu-Wing Tai, Chi-Keung Tang, Martin Danelljan, and Fisher Yu.  Segment anything meets point tracking.  _arXiv:2307.01197_ , 2023.
  * Ranjit et al. [2023] Mercy Ranjit, Gopinath Ganapathy, Ranjit Manuel, and Tanuja Ganu.  Retrieval augmented chest x-ray report generation using openai gpt models.  _arXiv preprint arXiv:2305.03660_ , 2023.
  * Raven [2003] Jean Raven.  Raven progressive matrices.  In _Handbook of nonverbal assessment_ , pages 223–237. Springer, 2003\.
  * Recht et al. [2019] Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar.  Do imagenet classifiers generalize to imagenet?  In _International conference on machine learning_ , pages 5389–5400. PMLR, 2019.
  * Ren et al. [2023] Allen Z Ren, Anushri Dixit, Alexandra Bodrova, Sumeet Singh, Stephen Tu, Noah Brown, Peng Xu, Leila Takayama, Fei Xia, Jake Varley, et al.  Robots that ask for help: Uncertainty alignment for large language model planners.  _arXiv preprint arXiv:2307.01928_ , 2023.
  * Ren et al. [2015] Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun.  Faster r-cnn: Towards real-time object detection with region proposal networks.  _Advances in neural information processing systems_ , 28, 2015.
  * Rezatofighi et al. [2019] Hamid Rezatofighi, Nathan Tsoi, JunYoung Gwak, Amir Sadeghian, Ian Reid, and Silvio Savarese.  Generalized intersection over union: A metric and a loss for bounding box regression.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pages 658–666, 2019.
  * Ronneberger et al. [2015] Olaf Ronneberger, Philipp Fischer, and Thomas Brox.  U-net: Convolutional networks for biomedical image segmentation.  In _Medical Image Computing and Computer-Assisted Intervention–MICCAI 2015: 18th International Conference, Munich, Germany, October 5-9, 2015, Proceedings, Part III 18_ , pages 234–241. Springer, 2015.
  * Schaeffer et al. [2023] Rylan Schaeffer, Brando Miranda, and Sanmi Koyejo.  Are emergent abilities of large language models a mirage?  _arXiv preprint arXiv:2304.15004_ , 2023.
  * Schuhmann et al. [2021] Christoph Schuhmann, Richard Vencu, Romain Beaumont, Robert Kaczmarczyk, Clayton Mullis, Aarush Katta, Theo Coombes, Jenia Jitsev, and Aran Komatsuzaki.  Laion-400m: Open dataset of clip-filtered 400 million image-text pairs.  _arXiv preprint arXiv:2111.02114_ , 2021.
  * Schuhmann et al. [2022] Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, Patrick Schramowski, Srivatsa Kundurthy, Katherine Crowson, Ludwig Schmidt, Robert Kaczmarczyk, and Jenia Jitsev.  Laion-5b: An open large-scale dataset for training next generation image-text models, 2022.
  * Schwenk et al. [2022] Dustin Schwenk, Apoorv Khandelwal, Christopher Clark, Kenneth Marino, and Roozbeh Mottaghi.  A-okvqa: A benchmark for visual question answering using world knowledge.  In _European Conference on Computer Vision_ , pages 146–162. Springer, 2022.
  * Seo et al. [2020] Seonguk Seo, Joon-Young Lee, and Bohyung Han.  Urvos: Unified referring video object segmentation network with a large-scale benchmark.  In _Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XV 16_ , pages 208–223. Springer, 2020.
  * Shah et al. [2021a] Dhruv Shah, Benjamin Eysenbach, Gregory Kahn, Nicholas Rhinehart, and Sergey Levine.  Ving: Learning open-world navigation with visual goals.  In _2021 IEEE International Conference on Robotics and Automation (ICRA)_ , pages 13215–13222. IEEE, 2021a.
  * Shah et al. [2022] Dhruv Shah, Blazej Osinski, Brian Ichter, and Sergey Levine.  Lm-nav: Robotic navigation with large pre-trained models of language, vision, and action.  _arXiv preprint arXiv: 2207.04429_ , 2022.
  * Shah et al. [2021b] Rohin Shah, Cody Wild, Steven H Wang, Neel Alex, Brandon Houghton, William Guss, Sharada Mohanty, Anssi Kanervisto, Stephanie Milani, Nicholay Topin, et al.  The minerl basalt competition on learning from human feedback.  _arXiv preprint arXiv:2107.01969_ , 2021b.
  * Shaharabany et al. [2023] Tal Shaharabany, Aviad Dahan, Raja Giryes, and Lior Wolf.  Autosam: Adapting sam to medical images by overloading the prompt encoder.  _arXiv preprint arXiv:2306.06370_ , 2023.
  * Shao et al. [2019] Shuai Shao, Zeming Li, Tianyuan Zhang, Chao Peng, Gang Yu, Xiangyu Zhang, Jing Li, and Jian Sun.  Objects365: A large-scale, high-quality dataset for object detection.  In _Proceedings of the IEEE/CVF international conference on computer vision_ , pages 8430–8439, 2019.
  * Sharma et al. [2018] Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut.  Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning.  In _Proceedings of ACL_ , 2018.
  * Shen et al. [2023] Yongliang Shen, Kaitao Song, Xu Tan, Dongsheng Li, Weiming Lu, and Yueting Zhuang.  Hugginggpt: Solving ai tasks with chatgpt and its friends in hugging face.  _arXiv preprint arXiv: 2303.17580_ , 2023.
  * Shtedritski et al. [2023] Aleksandar Shtedritski, Christian Rupprecht, and Andrea Vedaldi.  What does clip know about a red circle? visual prompt engineering for vlms.  _arXiv preprint arXiv:2304.06712_ , 2023.
  * Shukor et al. [2023] Mustafa Shukor, Corentin Dancette, and Matthieu Cord.  ep-alm: Efficient perceptual augmentation of language models.  _arXiv preprint arXiv:2303.11403_ , 2023.
  * Sidorov et al. [2020] Oleksii Sidorov, Ronghang Hu, Marcus Rohrbach, and Amanpreet Singh.  Textcaps: a dataset for image captioning with reading comprehension.  In _Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part II 16_ , pages 742–758. Springer, 2020.
  * Silberman et al. [2012] Nathan Silberman, Derek Hoiem, Pushmeet Kohli, and Rob Fergus.  Indoor segmentation and support inference from rgbd images.  In _Computer Vision–ECCV 2012: 12th European Conference on Computer Vision, Florence, Italy, October 7-13, 2012, Proceedings, Part V 12_ , pages 746–760. Springer, 2012.
  * Silva et al. [2014] Juan Silva, Aymeric Histace, Olivier Romain, Xavier Dray, and Bertrand Granado.  Toward embedded detection of polyps in wce images for early diagnosis of colorectal cancer.  _International journal of computer assisted radiology and surgery_ , 9:283–293, 2014.
  * Singh et al. [2019] Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruv Batra, Devi Parikh, and Marcus Rohrbach.  Towards vqa models that can read.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pages 8317–8326, 2019.
  * Singh et al. [2022] Amanpreet Singh, Ronghang Hu, Vedanuj Goswami, Guillaume Couairon, Wojciech Galuba, Marcus Rohrbach, and Douwe Kiela.  Flava: A foundational language and vision alignment model.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 15638–15650, 2022.
  * Sirinukunwattana et al. [2017] Korsuk Sirinukunwattana, Josien PW Pluim, Hao Chen, Xiaojuan Qi, Pheng-Ann Heng, Yun Bo Guo, Li Yang Wang, Bogdan J Matuszewski, Elia Bruni, Urko Sanchez, et al.  Gland segmentation in colon histology images: The glas challenge contest.  _Medical image analysis_ , 35:489–502, 2017.
  * Skreta et al. [2023] Marta Skreta, Naruki Yoshikawa, Sebastian Arellano-Rubach, Zhi Ji, Lasse Bjørn Kristensen, Kourosh Darvish, Alán Aspuru-Guzik, Florian Shkurti, and Animesh Garg.  Errors are useful prompts: Instruction guided task programming with verifier-assisted iterative prompting, 2023.
  * Song et al. [2015] Shuran Song, Samuel P Lichtenberg, and Jianxiong Xiao.  Sun rgb-d: A rgb-d scene understanding benchmark suite.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 567–576, 2015.
  * Srinivasan et al. [2021] Krishna Srinivasan, Karthik Raman, Jiecao Chen, Michael Bendersky, and Marc Najork.  Wit: Wikipedia-based image text dataset for multimodal multilingual machine learning.  _arXiv preprint arXiv: 2103.01913_ , 2021.
  * Stella et al. [2023] Francesco Stella, Cosimo Della Santina, and Josie Hughes.  Can large language models design a robot?  _arXiv preprint arXiv:2303.15324_ , 2023.
  * Sun et al. [2023a] Quan Sun, Yuxin Fang, Ledell Wu, Xinlong Wang, and Yue Cao.  Eva-clip: Improved training techniques for clip at scale.  _arXiv preprint arXiv:2303.15389_ , 2023a.
  * Sun et al. [2022] Yutao Sun, Li Dong, Barun Patra, Shuming Ma, Shaohan Huang, Alon Benhaim, Vishrav Chaudhary, Xia Song, and Furu Wei.  A length-extrapolatable transformer.  _arXiv preprint arXiv:2212.10554_ , 2022.
  * Sun et al. [2023b] Yutao Sun, Li Dong, Shaohan Huang, Shuming Ma, Yuqing Xia, Jilong Xue, Jianyong Wang, and Furu Wei.  Retentive network: A successor to transformer for large language models.  _arXiv preprint arXiv:2307.08621_ , 2023b.
  * Surís et al. [2023] Dídac Surís, Sachit Menon, and Carl Vondrick.  Vipergpt: Visual inference via python execution for reasoning.  _arXiv preprint arXiv: 2303.08128_ , 2023.
  * Tajbakhsh et al. [2015] Nima Tajbakhsh, Suryakanth R Gurudu, and Jianming Liang.  Automated polyp detection in colonoscopy videos using shape and context information.  _IEEE transactions on medical imaging_ , 35(2):630–644, 2015.
  * Tan and Bansal [2019] Hao Tan and Mohit Bansal.  Lxmert: Learning cross-modality encoder representations from transformers.  _arXiv preprint arXiv:1908.07490_ , 2019.
  * Tan and Le [2019] Mingxing Tan and Quoc Le.  Efficientnet: Rethinking model scaling for convolutional neural networks.  In _International conference on machine learning_ , pages 6105–6114. PMLR, 2019.
  * Tarvainen and Valpola [2017] Antti Tarvainen and Harri Valpola.  Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results.  _Advances in neural information processing systems_ , 30, 2017.
  * Thawkar et al. [2023] Omkar Thawkar, Abdelrahman Shaker, Sahal Shaji Mullappilly, Hisham Cholakkal, Rao Muhammad Anwer, Salman Khan, Jorma Laaksonen, and Fahad Shahbaz Khan.  Xraygpt: Chest radiographs summarization using medical vision-language models.  _arXiv preprint arXiv:2306.07971_ , 2023.
  * Thomee et al. [2016] Bart Thomee, David A Shamma, Gerald Friedland, Benjamin Elizalde, Karl Ni, Douglas Poland, Damian Borth, and Li-Jia Li.  Yfcc100m: The new data in multimedia research.  _Communications of the ACM_ , 59(2):64–73, 2016\.
  * Tong et al. [2023] Shengbang Tong, Erik Jones, and Jacob Steinhardt.  Mass-producing failures of multimodal systems with language models.  _arXiv preprint arXiv:2306.12105_ , 2023.
  * Touvron et al. [2021] Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou.  Training data-efficient image transformers & distillation through attention.  In _International conference on machine learning_ , pages 10347–10357. PMLR, 2021.
  * Touvron et al. [2023] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al.  Llama: Open and efficient foundation language models.  _arXiv preprint arXiv:2302.13971_ , 2023.
  * Tschannen et al. [2023] Michael Tschannen, Manoj Kumar, Andreas Steiner, Xiaohua Zhai, Neil Houlsby, and Lucas Beyer.  Image captioners are scalable vision learners too.  _arXiv preprint arXiv:2306.07915_ , 2023.
  * Tsimpoukelli et al. [2021] Maria Tsimpoukelli, Jacob L Menick, Serkan Cabi, SM Eslami, Oriol Vinyals, and Felix Hill.  Multimodal few-shot learning with frozen language models.  _Advances in Neural Information Processing Systems_ , 34:200–212, 2021.
  * Tu et al. [2005] Zhuowen Tu, Xiangrong Chen, Alan L Yuille, and Song-Chun Zhu.  Image parsing: Unifying segmentation, detection, and recognition.  _International Journal of computer vision_ , 63:113–140, 2005.
  * Vaswani et al. [2017] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  _Advances in neural information processing systems_ , 30, 2017.
  * Wake et al. [2023] Naoki Wake, Atsushi Kanehira, Kazuhiro Sasabuchi, Jun Takamatsu, and Katsushi Ikeuchi.  Chatgpt empowered long-step robot control in various environments: A case application.  _arXiv preprint arXiv: 2304.03893_ , 2023.
  * Wang et al. [2023a] Guanzhi Wang, Yuqi Xie, Yunfan Jiang, Ajay Mandlekar, Chaowei Xiao, Yuke Zhu, Linxi Fan, and Anima Anandkumar.  Voyager: An open-ended embodied agent with large language models.  _arXiv preprint arXiv:2305.16291_ , 2023a.
  * Wang et al. [2022a] Hongyu Wang, Shuming Ma, Shaohan Huang, Li Dong, Wenhui Wang, Zhiliang Peng, Yu Wu, Payal Bajaj, Saksham Singhal, Alon Benhaim, Barun Patra, Zhun Liu, Vishrav Chaudhary, Xia Song, and Furu Wei.  Foundation transformers, 2022a.
  * Wang et al. [2022b] Jianfeng Wang, Zhengyuan Yang, Xiaowei Hu, Linjie Li, Kevin Lin, Zhe Gan, Zicheng Liu, Ce Liu, and Lijuan Wang.  Git: A generative image-to-text transformer for vision and language.  _arXiv preprint arXiv:2205.14100_ , 2022b.
  * Wang et al. [2022c] Jinpeng Wang, Yixiao Ge, Guanyu Cai, Rui Yan, Xudong Lin, Ying Shan, Xiaohu Qie, and Mike Zheng Shou.  Object-aware video-language pre-training for retrieval.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pages 3313–3322, 2022c.
  * Wang et al. [2023b] Jiongxiao Wang, Zichen Liu, Keun Hee Park, Muhao Chen, and Chaowei Xiao.  Adversarial demonstration attacks on large language models.  _arXiv preprint arXiv:2305.14950_ , 2023b.
  * Wang et al. [2023c] Junke Wang, Dongdong Chen, Chong Luo, Xiyang Dai, Lu Yuan, Zuxuan Wu, and Yu-Gang Jiang.  Chatvideo: A tracklet-centric multimodal and versatile video understanding system.  _arXiv preprint arXiv:2304.14407_ , 2023c.
  * Wang et al. [2023d] Liyuan Wang, Xingxing Zhang, Hang Su, and Jun Zhu.  A comprehensive survey of continual learning: Theory, method and application.  _arXiv preprint arXiv:2302.00487_ , 2023d.
  * Wang et al. [2023e] Teng Wang, Jinrui Zhang, Junjie Fei, Hao Zheng, Yunlong Tang, Zhe Li, Mingqi Gao, and Shanshan Zhao.  Caption anything: Interactive image description with diverse multimodal controls.  _arXiv preprint arXiv: 2305.02677_ , 2023e.
  * Wang et al. [2021a] Weiyao Wang, Matt Feiszli, Heng Wang, and Du Tran.  Unidentified video objects: A benchmark for dense, open-world segmentation.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 10776–10785, 2021a.
  * Wang et al. [2023f] Wenhai Wang, Zhe Chen, Xiaokang Chen, Jiannan Wu, Xizhou Zhu, Gang Zeng, Ping Luo, Tong Lu, Jie Zhou, Yu Qiao, and Jifeng Dai.  Visionllm: Large language model is also an open-ended decoder for vision-centric tasks.  _arXiv preprint arXiv: 2305.11175_ , 2023f.
  * Wang et al. [2022d] Wenhui Wang, Hangbo Bao, Li Dong, Johan Bjorck, Zhiliang Peng, Qiang Liu, Kriti Aggarwal, Owais Khan Mohammed, Saksham Singhal, Subhojit Som, et al.  Image as a foreign language: Beit pretraining for all vision and vision-language tasks.  _arXiv preprint arXiv:2208.10442_ , 2022d.
  * Wang et al. [2019] Xin Wang, Jiawei Wu, Junkun Chen, Lei Li, Yuan-Fang Wang, and William Yang Wang.  Vatex: A large-scale, high-quality multilingual dataset for video-and-language research.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 4581–4591, 2019.
  * Wang et al. [2023g] Xinlong Wang, Wen Wang, Yue Cao, Chunhua Shen, and Tiejun Huang.  Images speak in images: A generalist painter for in-context visual learning.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 6830–6839, 2023g.
  * Wang et al. [2023h] Xinlong Wang, Xiaosong Zhang, Yue Cao, Wen Wang, Chunhua Shen, and Tiejun Huang.  Seggpt: Segmenting everything in context.  _arXiv preprint arXiv:2304.03284_ , 2023h.
  * Wang et al. [2022e] Zhaoqing Wang, Yu Lu, Qiang Li, Xunqiang Tao, Yandong Guo, Mingming Gong, and Tongliang Liu.  Cris: Clip-driven referring image segmentation.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pages 11686–11695, 2022e.
  * Wang et al. [2023i] Zhenyu Wang, Yali Li, Xi Chen, Ser-Nam Lim, Antonio Torralba, Hengshuang Zhao, and Shengjin Wang.  Detecting everything in the open world: Towards universal object detection.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 11433–11443, 2023i.
  * Wang et al. [2021b] Zirui Wang, Jiahui Yu, Adams Wei Yu, Zihang Dai, Yulia Tsvetkov, and Yuan Cao.  Simvlm: Simple visual language model pretraining with weak supervision.  _arXiv preprint arXiv:2108.10904_ , 2021b.
  * Wanna et al. [2023] Selma Wanna, Fabian Parra, Robert Valner, Karl Kruusamäe, and Mitch Pryor.  Multimodal grounding for embodied ai via augmented reality headsets for natural language driven task planning.  _arXiv preprint arXiv:2304.13676_ , 2023.
  * Wei et al. [2018] Chen Wei, Wenjing Wang, Wenhan Yang, and Jiaying Liu.  Deep retinex decomposition for low-light enhancement.  _arXiv preprint arXiv:1808.04560_ , 2018.
  * Wei et al. [2022a] Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, et al.  Emergent abilities of large language models.  _arXiv preprint arXiv:2206.07682_ , 2022a.
  * Wei et al. [2022b] Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al.  Chain-of-thought prompting elicits reasoning in large language models.  _Advances in Neural Information Processing Systems_ , 35:24824–24837, 2022b.
  * Wenhui et al. [2023] Lei Wenhui, Wei Xu, Zhang Xiaofan, Li Kang, and Shaoting Zhang.  Medlsam: Localize and segment anything model for 3d medical images.  _arXiv preprint arXiv:_ , 2023.
  * Wu et al. [2023a] Chenfei Wu, Shengming Yin, Weizhen Qi, Xiaodong Wang, Zecheng Tang, and Nan Duan.  Visual chatgpt: Talking, drawing and editing with visual foundation models.  _arXiv preprint arXiv: 2303.04671_ , 2023a.
  * Wu et al. [2020] Chenyun Wu, Zhe Lin, Scott Cohen, Trung Bui, and Subhransu Maji.  Phrasecut: Language-based image segmentation in the wild.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 10216–10225, 2020.
  * Wu et al. [2023b] Jimmy Wu, Rika Antonova, Adam Kan, Marion Lepert, Andy Zeng, Shuran Song, Jeannette Bohg, Szymon Rusinkiewicz, and Thomas Funkhouser.  Tidybot: Personalized robot assistance with large language models.  _arXiv preprint arXiv:2305.05658_ , 2023b.
  * Wu et al. [2023c] Junde Wu, Rao Fu, Huihui Fang, Yuanpei Liu, Zhaowei Wang, Yanwu Xu, Yueming Jin, and Tal Arbel.  Medical sam adapter: Adapting segment anything model for medical image segmentation.  _arXiv preprint arXiv:2304.12620_ , 2023c.
  * Wu et al. [2023d] Zhenyu Wu, Ziwei Wang, Xiuwei Xu, Jiwen Lu, and Haibin Yan.  Embodied task planning with large language models.  _arXiv preprint arXiv:2307.01848_ , 2023d.
  * Xu et al. [2017] Dejing Xu, Zhou Zhao, Jun Xiao, Fei Wu, Hanwang Zhang, Xiangnan He, and Yueting Zhuang.  Video question answering via gradually refined attention over appearance and motion.  In _Proceedings of the 25th ACM international conference on Multimedia_ , pages 1645–1653, 2017.
  * Xu et al. [2022a] Haofei Xu, Jing Zhang, Jianfei Cai, Hamid Rezatofighi, Fisher Yu, Dacheng Tao, and Andreas Geiger.  Unifying flow, stereo and depth estimation.  _arXiv preprint arXiv:2211.05783_ , 2022a.
  * Xu et al. [2022b] Jiarui Xu, Shalini De Mello, Sifei Liu, Wonmin Byeon, Thomas Breuel, J. Kautz, and X. Wang.  Groupvit: Semantic segmentation emerges from text supervision.  _Computer Vision And Pattern Recognition_ , 2022b.  doi: 10.1109/CVPR52688.2022.01760.
  * Xu et al. [2023a] Jiarui Xu, Sifei Liu, Arash Vahdat, Wonmin Byeon, Xiaolong Wang, and Shalini De Mello.  Open-vocabulary panoptic segmentation with text-to-image diffusion models.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 2955–2966, 2023a.
  * Xu et al. [2016] Jun Xu, Tao Mei, Ting Yao, and Yong Rui.  Msr-vtt: A large video description dataset for bridging video and language.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 5288–5296, 2016.
  * Xu et al. [2023b] Peng Xu, Xiatian Zhu, and David Clifton.  Multimodal learning with transformers: A survey.  _TPAMI_ , pages 1–21, 2023b.
  * Xu et al. [2022c] Xiao Xu, Chenfei Wu, Shachar Rosenman, Vasudev Lal, and Nan Duan.  Bridge-tower: Building bridges between encoders in vision-language representation learning.  _arXiv preprint arXiv:2206.08657_ , 2022c.
  * Xue et al. [2020] Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel.  mt5: A massively multilingual pre-trained text-to-text transformer.  _arXiv preprint arXiv:2010.11934_ , 2020.
  * Yang et al. [2021] Antoine Yang, Antoine Miech, Josef Sivic, Ivan Laptev, and Cordelia Schmid.  Just ask: Learning to answer questions from millions of narrated videos.  In _Proceedings of the IEEE/CVF international conference on computer vision_ , pages 1686–1697, 2021.
  * Yang et al. [2023a] Jiange Yang, Wenhui Tan, Chuhao Jin, Bei Liu, Jianlong Fu, Ruihua Song, and Limin Wang.  Pave the way to grasp anything: Transferring foundation models for universal pick-place robots.  _arXiv preprint arXiv:2306.05716_ , 2023a.
  * Yang et al. [2022a] Jianwei Yang, Chunyuan Li, Xiyang Dai, and Jianfeng Gao.  Focal modulation networks.  _Advances in Neural Information Processing Systems_ , 35:4203–4217, 2022a.
  * Yang et al. [2022b] Jianwei Yang, Chunyuan Li, Pengchuan Zhang, Bin Xiao, Ce Liu, Lu Yuan, and Jianfeng Gao.  Unified contrastive learning in image-text-label space.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 19163–19173, 2022b.
  * Yang et al. [2023b] Jingfeng Yang, Hongye Jin, Ruixiang Tang, Xiaotian Han, Qizhang Feng, Haoming Jiang, Bing Yin, and Xia Hu.  Harnessing the power of llms in practice: A survey on chatgpt and beyond.  _arXiv preprint arXiv:2304.13712_ , 2023b.
  * Yang et al. [2023c] Jinyu Yang, Mingqi Gao, Zhe Li, Shang Gao, Fangjing Wang, and Feng Zheng.  Track anything: Segment anything meets videos.  _arXiv preprint arXiv: 2304.11968_ , 2023c.
  * Yang et al. [2022c] Ling Yang, Zhilong Zhang, Yang Song, Shenda Hong, Runsheng Xu, Yue Zhao, Yingxia Shao, Wentao Zhang, Bin Cui, and Ming-Hsuan Yang.  Diffusion models: A comprehensive survey of methods and applications.  _arXiv preprint arXiv:2209.00796_ , 2022c.
  * Yang et al. [2023d] Zhengyuan Yang, Linjie Li, Jianfeng Wang, Kevin Lin, Ehsan Azarnasab, Faisal Ahmed, Zicheng Liu, Ce Liu, Michael Zeng, and Lijuan Wang.  Mm-react: Prompting chatgpt for multimodal reasoning and action.  _arXiv preprint arXiv: 2303.11381_ , 2023d.
  * Yang and Yang [2022] Zongxin Yang and Yi Yang.  Decoupling features in hierarchical propagation for video object segmentation.  _Advances in Neural Information Processing Systems_ , 35:36324–36336, 2022.
  * Yao et al. [2021] Lewei Yao, Runhui Huang, Lu Hou, Guansong Lu, Minzhe Niu, Hang Xu, Xiaodan Liang, Zhenguo Li, Xin Jiang, and Chunjing Xu.  Filip: fine-grained interactive language-image pre-training.  _arXiv preprint arXiv:2111.07783_ , 2021.
  * Ye et al. [2022] Junjie Ye, Changhong Fu, Guangze Zheng, Danda Pani Paudel, and Guang Chen.  Unsupervised domain adaptation for nighttime aerial tracking.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 8896–8905, 2022.
  * Ye et al. [2019] Linwei Ye, Mrigank Rochan, Zhi Liu, and Yang Wang.  Cross-modal self-attention network for referring image segmentation.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pages 10502–10511, 2019.
  * Ye et al. [2023] Qinghao Ye, Haiyang Xu, Guohai Xu, Jiabo Ye, Ming Yan, Yiyang Zhou, Junyang Wang, Anwen Hu, Pengcheng Shi, Yaya Shi, Chaoya Jiang, Chenliang Li, Yuanhong Xu, Hehong Chen, Junfeng Tian, Qian Qi, Ji Zhang, and Fei Huang.  mplug-owl: Modularization empowers large language models with multimodality, 2023.
  * Yoneda et al. [2023] Takuma Yoneda, Jiading Fang, Peng Li, Huanyu Zhang, Tianchong Jiang, Shengjie Lin, Ben Picker, David Yunis, Hongyuan Mei, and Matthew R Walter.  Statler: State-maintaining language models for embodied reasoning.  _arXiv preprint arXiv:2306.17840_ , 2023.
  * Yonglin et al. [2023] Li Yonglin, Zhang Jing, Teng Xiao, and Lan Long.  Refsam: Efficiently adapting segmenting anything model for referring video object segmentation.  _arXiv:2307.00997_ , 2023.
  * You et al. [2023] Hengxu You, Yang Ye, Tianyu Zhou, Qi Zhu, and Jing Du.  Robot-enabled construction assembly with automated sequence planning based on chatgpt: Robogpt.  _arXiv preprint arXiv:2304.11018_ , 2023.
  * Young et al. [2014] Peter Young, Alice Lai, Micah Hodosh, and Julia Hockenmaier.  From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions.  _Transactions of the Association for Computational Linguistics_ , 2:67–78, 2014.
  * Yu et al. [2022] Jiahui Yu, Zirui Wang, Vijay Vasudevan, Legg Yeung, Mojtaba Seyedhosseini, and Yonghui Wu.  Coca: Contrastive captioners are image-text foundation models.  _arXiv preprint arXiv:2205.01917_ , 2022.
  * Yu et al. [2016] Licheng Yu, Patrick Poirson, Shan Yang, Alexander C Berg, and Tamara L Berg.  Modeling context in referring expressions.  In _Computer Vision–ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part II 14_ , pages 69–85. Springer, 2016.
  * Yuan et al. [2021] Lu Yuan, Dongdong Chen, Yi-Ling Chen, Noel Codella, Xiyang Dai, Jianfeng Gao, Houdong Hu, Xuedong Huang, Boxin Li, Chunyuan Li, et al.  Florence: A new foundation model for computer vision.  _arXiv preprint arXiv:2111.11432_ , 2021.
  * Yuan et al. [2023] Zhengqing Yuan, Huiwen Xue, Xinyi Wang, Yongming Liu, Zhuanzhe Zhao, and Kun Wang.  Artgpt-4: Artistic vision-language understanding with adapter-enhanced minigpt-4.  _arXiv preprint arXiv:2305.07490_ , 2023.
  * Zhai and Wu [2018] Andrew Zhai and Hao-Yu Wu.  Classification is a strong baseline for deep metric learning.  _arXiv preprint arXiv:1811.12649_ , 2018.
  * Zhai et al. [2022] Xiaohua Zhai, Alexander Kolesnikov, Neil Houlsby, and Lucas Beyer.  Scaling vision transformers.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 12104–12113, 2022.
  * Zhang et al. [2023a] Ao Zhang, Hao Fei, Yuan Yao, Wei Ji, Li Li, Zhiyuan Liu, and Tat-Seng Chua.  Transfer visual prompt generator across llms.  _arXiv preprint arXiv: 2305.01278_ , 2023a.
  * Zhang et al. [2023b] Binjie Zhang, Yixiao Ge, Xuyuan Xu, Ying Shan, and Mike Zheng Shou.  Taca: Upgrading your visual foundation model with task-agnostic compatible adapter.  _arXiv preprint arXiv: 2306.12642_ , 2023b.
  * Zhang et al. [2023c] Chaoning Zhang, Dongshen Han, Yu Qiao, Jung Uk Kim, Sung-Ho Bae, Seungkyu Lee, and Choong Seon Hong.  Faster segment anything: Towards lightweight sam for mobile applications.  _arXiv preprint arXiv:2306.14289_ , 2023c.
  * Zhang et al. [2023d] Chaoning Zhang, Dongshen Han, Yu Qiao, Jung Uk Kim, Sung-Ho Bae, Seungkyu Lee, and Choong Seon Hong.  Faster segment anything: Towards lightweight sam for mobile applications.  _arXiv preprint arXiv: 2306.14289_ , 2023d.
  * Zhang et al. [2023e] Chenshuang Zhang, Chaoning Zhang, Taegoo Kang, Donghun Kim, Sung-Ho Bae, and In So Kweon.  Attack-sam: Towards evaluating adversarial robustness of segment anything model.  _arXiv preprint arXiv:2305.00866_ , 2023e.
  * Zhang et al. [2023f] Chenshuang Zhang, Chaoning Zhang, Mengchun Zhang, and In So Kweon.  Text-to-image diffusion model in generative ai: A survey.  _arXiv preprint arXiv:2303.07909_ , 2023f.
  * Zhang et al. [2023g] Chunhui Zhang, Li Liu, Yawen Cui, Guanjie Huang, Weilin Lin, Yiqian Yang, and Yuehong Hu.  A comprehensive survey on segment anything model for vision and beyond.  _arXiv preprint arXiv:2305.08196_ , 2023g.
  * Zhang et al. [2022a] Haotian Zhang, Pengchuan Zhang, Xiaowei Hu, Yen-Chun Chen, Liunian Li, Xiyang Dai, Lijuan Wang, Lu Yuan, Jenq-Neng Hwang, and Jianfeng Gao.  Glipv2: Unifying localization and vision-language understanding.  _Advances in Neural Information Processing Systems_ , 35:36067–36080, 2022a.
  * Zhang et al. [2023h] Hongxin Zhang, Weihua Du, Jiaming Shan, Qinhong Zhou, Yilun Du, Joshua B Tenenbaum, Tianmin Shu, and Chuang Gan.  Building cooperative embodied agents modularly with large language models.  _arXiv preprint arXiv:2307.02485_ , 2023h.
  * Zhang et al. [2023i] Jesse Zhang, Karl Pertsch, Jiahui Zhang, and Joseph J Lim.  Sprint: Scalable policy pre-training via language instruction relabeling.  _arXiv preprint arXiv:2306.11886_ , 2023i.
  * Zhang [2023a] Jiawei Zhang.  Graph-toolformer: To empower llms with graph reasoning ability via prompt augmented by chatgpt.  _arXiv preprint arXiv:2304.11116_ , 2023a.
  * Zhang [2023b] Jiawei Zhang.  Graph-toolformer: To empower llms with graph reasoning ability via prompt augmented by chatgpt.  _arXiv preprint arXiv: 2304.11116_ , 2023b.
  * Zhang et al. [2023j] Renrui Zhang, Jiaming Han, Aojun Zhou, Xiangfei Hu, Shilin Yan, Pan Lu, Hongsheng Li, Peng Gao, and Yu Qiao.  Llama-adapter: Efficient fine-tuning of language models with zero-init attention.  _arXiv preprint arXiv:2303.16199_ , 2023j.
  * Zhang et al. [2023k] Shilong Zhang, Peize Sun, Shoufa Chen, Min Xiao, Wenqi Shao, Wenwei Zhang, Kai Chen, and Ping Luo.  Gpt4roi: Instruction tuning large language model on region-of-interest, 2023k.
  * Zhang et al. [2022b] Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al.  Opt: Open pre-trained transformer language models.  _arXiv preprint arXiv:2205.01068_ , 2022b.
  * Zhang et al. [2021] Tianwen Zhang, Xiaoling Zhang, Jianwei Li, Xiaowo Xu, Baoyou Wang, Xu Zhan, Yanqin Xu, Xiao Ke, Tianjiao Zeng, Hao Su, et al.  Sar ship detection dataset (ssdd): Official release and comprehensive data analysis.  _Remote Sensing_ , 13(18):3690, 2021.
  * Zhang et al. [2023l] Xinsong Zhang, Yan Zeng, Jipeng Zhang, and Hang Li.  Toward building general foundation models for language, vision, and vision-language understanding tasks.  _arXiv preprint arXiv:2301.05065_ , 2023l.
  * Zhang et al. [2023m] Yanzhe Zhang, Ruiyi Zhang, Jiuxiang Gu, Yufan Zhou, Nedim Lipka, Diyi Yang, and Tong Sun.  Llavar: Enhanced visual instruction tuning for text-rich image understanding.  _arXiv preprint arXiv: 2306.17107_ , 2023m.
  * Zhang et al. [2023n] Youcai Zhang, Xinyu Huang, Jinyu Ma, Zhaoyang Li, Zhaochuan Luo, Yanchun Xie, Yuzhuo Qin, Tong Luo, Yaqian Li, Shilong Liu, et al.  Recognize anything: A strong image tagging model.  _arXiv preprint arXiv:2306.03514_ , 2023n.
  * Zhang et al. [2023o] Zhuosheng Zhang, Aston Zhang, Mu Li, Hai Zhao, George Karypis, and Alex Smola.  Multimodal chain-of-thought reasoning in language models.  _arXiv preprint arXiv:2302.00923_ , 2023o.
  * Zhao et al. [2023a] Chao Zhao, Shuai Yuan, Chunli Jiang, Junhao Cai, Hongyu Yu, Michael Yu Wang, and Qifeng Chen.  Erra: An embodied representation and reasoning architecture for long-horizon language-conditioned manipulation tasks.  _IEEE Robotics and Automation Letters_ , 2023a.
  * Zhao et al. [2023b] Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, Yifan Du, Chen Yang, Yushuo Chen, Zhipeng Chen, Jinhao Jiang, Ruiyang Ren, Yifan Li, Xinyu Tang, Zikang Liu, Peiyu Liu, Jian-Yun Nie, and Ji-Rong Wen.  A survey of large language models.  _arXiv preprint arXiv:2303.18223_ , 2023b.
  * Zhao et al. [2023c] Xu Zhao, Wenchao Ding, Yongqi An, Yinglong Du, Tao Yu, Min Li, Ming Tang, and Jinqiao Wang.  Fast segment anything.  _arXiv preprint arXiv: 2306.12156_ , 2023c.
  * Zhao et al. [2023d] Xufeng Zhao, Mengdi Li, Cornelius Weber, Muhammad Burhan Hafez, and Stefan Wermter.  Chat with the environment: Interactive multimodal perception using large language models.  _arXiv preprint arXiv: 2303.08268_ , 2023d.
  * Zhao et al. [2023e] Yang Zhao, Zhijie Lin, Daquan Zhou, Zilong Huang, Jiashi Feng, and Bingyi Kang.  Bubogpt: Enabling visual grounding in multi-modal llms.  _arXiv preprint arXiv:2307.08581_ , 2023e.
  * Zheng et al. [2023a] Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al.  Judging llm-as-a-judge with mt-bench and chatbot arena.  _arXiv preprint arXiv:2306.05685_ , 2023a.
  * Zheng et al. [2023b] Mingkai Zheng, Xiu Su, Shan You, Fei Wang, Chen Qian, Chang Xu, and Samuel Albanie.  Can gpt-4 perform neural architecture search?  _arXiv preprint arXiv:2304.10970_ , 2023b.
  * Zheng et al. [2023c] Mingkai Zheng, Xiu Su, Shan You, Fei Wang, Chen Qian, Chang Xu, and Samuel Albanie.  Can gpt-4 perform neural architecture search?  _arXiv preprint arXiv: 2304.10970_ , 2023c.
  * Zhong et al. [2023] Tianyang Zhong, Yaonai Wei, Li Yang, Zihao Wu, Zhengliang Liu, Xiaozheng Wei, Wenjun Li, Junjie Yao, Chong Ma, Xiang Li, et al.  Chatabl: Abductive learning via natural language interaction with chatgpt.  _arXiv preprint arXiv:2304.11107_ , 2023.
  * Zhong et al. [2022] Yiwu Zhong, Jianwei Yang, Pengchuan Zhang, Chunyuan Li, Noel Codella, Liunian Harold Li, Luowei Zhou, Xiyang Dai, Lu Yuan, Yin Li, et al.  Regionclip: Region-based language-image pretraining.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 16793–16803, 2022.
  * Zhou et al. [2017] Bolei Zhou, Hang Zhao, Xavier Puig, Sanja Fidler, Adela Barriuso, and Antonio Torralba.  Scene parsing through ade20k dataset.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 633–641, 2017.
  * Zhou et al. [2019] Bolei Zhou, Hang Zhao, Xavier Puig, Tete Xiao, Sanja Fidler, Adela Barriuso, and Antonio Torralba.  Semantic understanding of scenes through the ade20k dataset.  _International Journal of Computer Vision_ , 127:302–321, 2019.
  * Zhou et al. [2023] Ce Zhou, Qian Li, Chen Li, Jun Yu, Yixin Liu, Guangjing Wang, Kai Zhang, Cheng Ji, Qiben Yan, Lifang He, et al.  A comprehensive survey on pretrained foundation models: A history from bert to chatgpt.  _arXiv preprint arXiv:2302.09419_ , 2023.
  * Zhou et al. [2022] Chong Zhou, Chen Change Loy, and Bo Dai.  Extract free dense labels from clip.  In _European Conference on Computer Vision (ECCV)_ , 2022.
  * Zhu et al. [2023a] Deyao Zhu, Jun Chen, Kilichbek Haydarov, Xiaoqian Shen, Wenxuan Zhang, and Mohamed Elhoseiny.  Chatgpt asks, blip-2 answers: Automatic questioning towards enriched visual descriptions.  _arXiv preprint arXiv:2303.06594_ , 2023a.
  * Zhu et al. [2023b] Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny.  Minigpt-4: Enhancing vision-language understanding with advanced large language models.  _arXiv preprint arXiv:2304.10592_ , 2023b.
  * Zhu et al. [2007] Song-Chun Zhu, David Mumford, et al.  A stochastic grammar of images.  _Foundations and Trends® in Computer Graphics and Vision_ , 2(4):259–362, 2007.
  * Zhu et al. [2023c] Wanrong Zhu, Jack Hessel, Anas Awadalla, Samir Yitzhak Gadre, Jesse Dodge, Alex Fang, Youngjae Yu, Ludwig Schmidt, William Yang Wang, and Yejin Choi.  Multimodal c4: An open, billion-scale corpus of images interleaved with text, 2023c.
  * Zhu et al. [2016] Yuke Zhu, Oliver Groth, Michael Bernstein, and Li Fei-Fei.  Visual7w: Grounded question answering in images, 2016.
  * Zong et al. [2023] Yongshuo Zong, Oisin Mac Aodha, and Timothy Hospedales.  Self-supervised multimodal learning: A survey.  _arXiv preprint arXiv:2304.01008_ , 2023.
  * Zou et al. [2022] Xueyan Zou, Zi-Yi Dou, Jianwei Yang, Zhe Gan, Linjie Li, Chunyuan Li, Xiyang Dai, Jianfeng Wang, Lu Yuan, Nanyun Peng, Lijuan Wang, Yong Jae Lee, and Jianfeng Gao.  Generalized decoding for pixel, image and language.  2022\.
  * Zou et al. [2023] Xueyan Zou, Jianwei Yang, Hao Zhang, Feng Li, Linjie Li, Jianfeng Gao, and Yong Jae Lee.  Segment everything everywhere all at once.  _arXiv preprint arXiv:2304.06718_ , 2023.