# MarineGPT: Unlocking Secrets of “Ocean” to the Public

Ziqiang Zheng1, Jipeng Zhang1, Tuan-Anh Vu1, Shizhe Diao1, Yue Him Wong Tim2,
Sai-Kit Yeung1
1Hong Kong University of Science and Technology, 2Shenzhen University
<EMAIL_ADDRESS><EMAIL_ADDRESS>

###### Abstract

Large language models (LLMs), such as ChatGPT/GPT-4, have proven to be
powerful tools in promoting the user experience as an AI assistant. The
continuous works are proposing multi-modal large language models (MLLM),
empowering LLMs with the ability to sense multiple modality inputs through
constructing a joint semantic space (_e.g._ visual-text space). Though
significant success was achieved in LLMs and MLLMs, exploring LLMs and MLLMs
in domain-specific applications that required domain-specific knowledge and
expertise has been less conducted, especially for marine domain. Different
from general-purpose MLLMs, the marine-specific MLLM is required to yield much
more sensitive, informative, and scientific responses. In this work, we
demonstrate that the existing MLLMs optimized on huge amounts of readily
available general-purpose training data show a minimal ability to understand
domain-specific intents and then generate informative and satisfactory
responses. To address these issues, we propose MarineGPT, the first vision-
language model specially designed for the marine domain, unlocking the secrets
of the ocean to the public. We present our Marine-5M dataset with more than 5
million marine image-text pairs to inject domain-specific marine knowledge
into our model and achieve better marine vision and language alignment. Our
MarineGPT not only pushes the boundaries of marine understanding to the
general public but also offers a standard protocol for adapting a general-
purpose assistant to downstream domain-specific experts. We pave the way for a
wide range of marine applications while setting valuable data and pre-trained
models for future research in both academic and industrial communities. Code
and data will be available at https://github.com/hkust-vgd/MarineGPT

##  1 Introduction

Large language models (LLMs) (Ouyang et al., 2022; OpenAI, 2023) demonstrated
an impressive ability for a large range of user-tailored tasks, endowing them
with immense potential applications (Fu et al., 2023; Cox et al., 2023; Tu et
al., 2023; Hong et al., 2023). As a general-purpose assistant, ChatGPT/GPT-4
(OpenAI, 2023; Ouyang et al., 2022) could understand human intents and
complete various real-world tasks. However, existing LLMs only focus on
unimodal text inputs. Obviously, a text-only Chatbot is less optimal for a
powerful AI assistant. The multi-modal large language models (MLLMs) (Li et
al., 2023c; Liu et al., 2023; Zhao et al., 2023; Zang et al., 2023; Wu et al.,
2023a) empower the LLMs to sense multiple modality inputs, follow multi-modal
instructions, and align with human intent to complete various real-world tasks
in the wild. A vision-language multi-modal model (Zhu et al., 2023) connects a
vision encoder and LLM (Touvron et al., 2023a) for general-purpose visual-and-
language understanding. There has been rapid progress in the MLLM field by
leveraging billions of image-text pairs (Byeon et al., 2022; Schuhmann et al.,
2022) from the public web. Vision-language pre-training (Li et al., 2022;
2023b) is conducted for bridging the frozen pre-trained vision encoders and
frozen language decoders. However, such general-domain vision-language models
still lack sophistication and conception coverage in understanding and
delivering domain-specific knowledge. MiniGPT-4 (Zhu et al., 2023) proposed to
align a frozen visual encoder with a frozen LLM by optimizing the linear layer
to bridge the two modalities. LLaVa (Liu et al., 2023) conducted the first
attempt to perform the visual instruction tuning, enabling the LLMs to
understand the user intent and generate corresponding responses.

Figure 1: MarineGPT could perform auto-recognition of various marine objects
and yield diverse, domain-specific, informative, and scientific responses
associated with the recognized marine object. Best viewed in color.

To enable MLLMs to understand the user intent and generate domain-specific
answers, further efforts and elaborate designs are specially required. The
dominant way is to collect moderate-scale but high-quality image-text pairs
from the specific domains for downstream fine-tuning (Zhang et al., 2023b).
LLaVa-Med (Li et al., 2023a) proposed to fine-tune the LLAVa (Liu et al.,
2023) to the medicine domain to generate the domain-specific responses. The
instruction-following training samples have also been meticulously curated for
the medical domain to enable the model to understand the user intent from the
specific domains. In this work, we propose MarineGPT presented in Fig. 1, the
first marine-specific vision-to-language expert model, which could recognize
marine objects from the given visual inputs and yield corresponding sensitive,
informative, and scientific responses as a powerful marine AI assistant.
Developing an effective marine AI assistant poses three non-trivial
challenges: 1) the existing general-purpose LLMs are less-efficient to
generate reasonable and scientific responses for the marine domain; 2) limited
concept coverage for the marine domain present in existing image-text pairs;
and 3) limited ability to perform the fine-grained object recognition since
many marine species share very similar appearance.

To address these challenges, we first scale up the marine knowledge collection
through crowdsourcing multiple marine sources for collecting huge amounts of
training data. We present our Marine-5M dataset that contains more than 5
million marine image-text pairs with redundant conceptions from the marine
domain. We utilize our Marine-5M for marine-specific continuous pre-training,
which could effectively adapt the general-purpose MLLM to the domain-specific
expert model, aligning images with domain expertise flexibly defined and
managed based on the language descriptions. We then design 50 different
marine-specific instructions based on the expertise and requirements from
marine biologists, which could help MarineGPT understand the user intent. We
scalably generate instruction-following training data based on ChatGPT/GPT-4,
following our constructed marine-specific instructions. Besides, we also
summarized 129 diverse and comprehensive attributes (_e.g._ distribution,
habitat, morphology, reproduction, and etc) of marine objects. We retrieved
and collected corresponding descriptions based on summarized attributes and
crawled category annotation from reliable marine knowledge sources (eol, 2018;
Yang et al., 2020; E. & L.M, 2016; ree, ). Through these ways, we can
effectively inject marine knowledge into the model and enable MarineGPT to
generate informative and domain-specific responses. The constructed
instruction-following training data (with 1.12 million high-quality marine
image-text pairs) are utilized for instruction-following fine-tuning
demonstrated in Fig. 2. Finally, we empirically demonstrate that only
designing the linear layer (Zhu et al., 2023) cannot effectively align the
visual signals and the textual descriptions for fine-grained marine object
recognition. The Q-Former should also be optimized for more effective and
fine-grained marine vision-language alignment.

MarineGPT is unlocking the secrets of the ocean to the public. It provides an
effective way to perform non-invasive automated species recognition, reducing
human labor from the domain experts. We observe that MarineGPT could
effectively identify marine organisms even under a fine-grained setting,
enabling scientific image annotation and decision-making processes. Meanwhile,
MarineGPT could also enable complex visual reasoning, knowledge-grounded image
description, and multi-turn conversations, allowing citizens, scientists, and
the general public to participate actively in marine research and
conservation. In summary, our main contributions are listed as follows.

  * •

MarineGPT empowers marine object recognition with question-answering
capabilities, which could encourage public participation in marine research
and data collection. Both citizens and scientists could contribute to
biodiversity monitoring and research efforts, helping to gather valuable
information for marine studies.

  * •

We propose the largest marine image-text pairs as far as we know for promoting
aligning visual-and-language modalities and enhancing the model’s perception
capabilities. The collected and constructed diverse and comprehensive image-
text pairs could significantly change the way of marine perception, reasoning,
and causal inference.

  * •

We present a marine-specific data generation pipeline to create diverse
(image, instruction, output) training data, by sampling marine image-text
pairs from broad-coverage and open-source websites and using ChatGPT/ChatGPT-4
to create instruction-following data.

  * •

We provide a unified and valuable perspective for performing domain-specific
cross-modal vision-language tasks, which require professional knowledge
injection and user intent understanding. Similar approaches could be adopted
in other fields like botany, entomology, or ornithology, enabling species
recognition and comprehensive data analysis.

##  2 Related Work

###  2.1 LLMs

LLMs have demonstrated remarkable progress across various Natural Language
Processing (NLP) tasks. The impressive performance of ChatGPT (OpenAI, 2022)
and GPT-4 (OpenAI, 2023) has motivated the research community to reproduce
their success, resulting in a rapidly increasing number of foundation models.
These efforts include GPT-3 (Brown et al., 2020), T5 (Raffel et al., 2020),
BLOOM (Scao et al., 2022), OPT (Zhang et al., 2022), LLaMA (Touvron et al.,
2023a), and LLaMA-2 (Touvron et al., 2023b), all of which have get worldwide
significant interest.

Pre-training LLMs for domain-specific applications. In addition to general-
purpose LLMs, several LLMs are specifically pre-trained from scratch to cater
to domain-specific tasks. These domain-specific pre-trained LLMs, such as
Galactica (Taylor et al., 2022) and CodeGen (Nijkamp et al., 2022), are
designed to address specific domains like science and program synthesis (Lu et
al., 2022).

Fine-tuning LLMs for domain-specific applications. These foundation models
have resulted in the exponential growth of fine-tuned variants in task-
oriented settings, such as chatbots. In order to enable these LLMs to follow
human commands and complete tasks, instruction tuning is introduced to obtain
these models’ instruction-following versions like InstructGPT (Ouyang et al.,
2022), Flan-T5 (Chung et al., 2022), and Vicuna (Chiang et al., 2023).
Additionally, various domain-specific applications have emerged based on these
foundation models, including Dr.LLaMA (Guo et al., 2023) and CancerGPT (Li et
al., 2023d) for medical applications, as well as FinGPT (Yang et al., 2023)
and BloombergGPT (Wu et al., 2023b) for finance. Also, several toolboxes for
fine-tuning LLM, like LMFlow (Diao et al., 2023), have been developed. The
explosive emergence of those task-oriented LLMs leads to impressive
performance gains and opens up new possibilities for various LLM applications.

###  2.2 MLLMs

As LLMs grow more intelligent and powerful, there is increasing attention on
extending the frozen LLMs to multi-modal tasks, empowering the LLMs with the
ability to sense multi-modality inputs. Flamingo (Alayrac et al., 2022)
pioneered to harness web-scale image-text data, capitalizing on both vision
and language models. This unveiling showcased remarkable zero-shot image-text
capabilities in a conversational format. BLIP (Li et al., 2022; 2023b)
bootstraps vision-language pre-training from frozen pre-trained image encoders
and frozen language decoders. Based on BLIP-2 (Li et al., 2023b), MiniGPT-4
(Zhu et al., 2023) proposed a projection layer to align pre-trained vision
encoder to frozen LLMs (_e.g._ Vicuna (Chiang et al., 2023)), and exhibited
respectable zero-shot image comprehension in dialogues. Since only linear
layers are required to be optimized, MiniGPT-4 offers high computation
efficiency and flexibility for being fine-tuned to different downstream tasks.
However, we noticed that only optimizing the linear layers cannot effectively
adapt the general-purpose MLLM to a domain-specific expert model.

MLLMs for domain-specific applications. A growing body of research has emerged
to extend general-purpose MLLMs to domain-specific applications. LLaVa-Med (Li
et al., 2023a) proposed to fine-tune the LLAVa to the medicine domain.
Pathasst (Sun et al., 2023) collected over 142K high-quality pathology image-
text pairs from various reliable sources. Pmc-vqa (Zhang et al., 2023b)
proposed to sample biomedical image-text pairs from PMC-15M and adopt GPT-4 to
create instructions from the text alone. EgoCOT (Mu et al., 2023) craft a
large-scale embodied planning dataset, consisting of carefully selected videos
from the Ego4D dataset for effective embodied planning. In this work, we aim
to propose the first marine-specific MLLM MarineGPT, and a large collection of
high-quality marine image-text pairs, which covers a broad spectrum of marine
and marine-relative conceptions.

##  3 Approaches

###  3.1 Framework Overview

The framework overview of MarineGPT is demonstrated in Fig. 2. To better
utilize the powerful zero-shot ability of LLMs and adapt them to domain-
specific experts, we design a two-stage training approach: 1) a marine-
specific continuous pre-training on a large collection of aligned marine
image-text pairs for acquiring marine vision-language alignment; 2) a further
instruction-following fine-tuning based on a moderate scale but high-quality
self-constructed image-text pairs with specifically designed instructions to
generate more informative, reliable and scientific answers, enhancing the
professional ability and usability of MarineGPT. The former continuous pre-
training procedure could effectively inject marine knowledge into the LLM.
Meanwhile, broad coverage of marine conceptions that have been overlooked in
existing general-purpose LLMs and MLLMs, could be introduced and emphasized
through continuous pre-training. Moreover, the constructed instruction-
following data could promote generating domain-specific responses with the
required information delivered and a better understanding of the user intent.

Figure 2: The framework overview of the proposed MarineGPT. There are two main
procedures in our MarineGPT: 1) marine-specific continuous pre-training on 5
million marine image-text pairs; 2) instruction-following fine-tuning based on
constructed high-quality instruction-following image-text pairs to generate
sensitive, informative and scientific responses.

###  3.2 Marine-Specific Continuous Pre-training

The vision-language pre-training based on redundant image-text pairs is an
essential, important, and foundational step to acquiring vision-language
knowledge and constructing a joint feature space. To scale up data collection,
the readily available image-text pairs from open-source websites are used for
vision-language pre-training. We adopt the pre-trained ViT encoder of BLIP-2
(Li et al., 2023b) optimized by CC12M (Changpinyo et al., 2021), COCO captions
(Lin et al., 2014), Visual Genone (Krishna et al., 2017) and sub-part of LAION
dataset (Schuhmann et al., 2021; 2022). In total, there are 14 million image-
text pairs during the initial pre-training stage in (Li et al., 2022). The
existing general-domain LLMs are less effective for marine scenarios because
marine image-text pairs drastically differ from general web content. The
current MLLMs cannot generate or answer the required or accurate descriptions
based on the given image. There are two main drawbacks when directly utilizing
the pre-trained models for instruction-following fine-tuning: 1) the existing
large collection of aligned image-text pairs does not involve too much marine
knowledge or marine visual observations and the conception coverage is not
enough to include most marine organisms; 2) the description of images are
usually short (especially for the LAION dataset (Schuhmann et al., 2021;
2022)), which cannot reveal and deliver too much detail information. Thus, the
pre-trained VIT encoder would still be unable to extract effective feature
representations from the marine images. Meanwhile, the language decoder will
also fail to generate long and meaningful descriptions. The existing general-
purpose LLMs are less effective for marine scenarios because marine image-text
pairs drastically differ from general web content. To address these two
issues, we present Marine-5M dataset with 5 million marine image-text pairs to
promote the ability to extract reliable marine features. Our constructed
Marine-5M is the largest marine image-text dataset specially designed for
marine research.

Different from the pre-training stage of BLIP-2 (Li et al., 2023b), we
optimize the Q-Former and the linear layer architectures to bootstrap the
marine vision-language pre-training based on our Marine-5M dataset.
Considering the annotation of crawled images from public websites only
provides the category annotation, we propose to expand the caption of images
according to $<$category annotation$>$. We summarized 129 diverse,
comprehensive, and hierarchical attributes of the marine objects, including
size, color, shape, feeding diet, distribution, habitat, morphology,
reproduction, and etc. We generate different attribute descriptions based on
texts crawled from reliable marine websites (_e.g._ , EOL (eol, 2018), FishDB
(Yang et al., 2020), Reeflex (ree, ) and so on) for images with the same
category annotation. In this way, we could generate longer image descriptions
and perform the marine knowledge injection. After marine-specific continuous
pre-training, the trained model demonstrates the capacity to possess a wealth
of knowledge in the marine field and offer more accurate and reasonable
responses to human inquiries. To further promote the ability of MarineGPT to
generate more fine-grained and scientific responses, we construct 1.12 million
high-quality image-text pairs with a large range of instruction-following
templates, presenting various tasks of describing marine organisms in the
given image.

###  3.3 Instruction-Following Fine-tuning

Performing continuous pre-training on the marine image-text pairs alone is
insufficient to build a domain-specific and scientific Chatbot for answering
diverse marine questions, as it lacks the ability to follow diverse marine-
specific instructions even though marine concept coverage is highly promoted
after the continuous pre-training. Terribly, the existing LLMs would refrain
from answering biology and marine questions, or worse, produce incorrect
responses or complete hallucinations. In most scenarios, the existing general-
purpose visual assistants may behave like a layperson for domain-specific
questions and cannot understand the user intent. A simple and effective method
of aligning LLMs to human intent is to learn from instruction-following data
generated by state-of-the-art LLMs.

####  3.3.1 Instruction-Following Image-Text Data Construction

MarineGPT adopts a wide range of vision-language instruction data, covering
both template-based converted image captions and ChatGPT-generated question-
answer data following the user intent demonstrated in Fig. 2. Asking
insightful marine questions is crucial for acquiring domain-specific knowledge
and expanding the understanding of the marine world. We propose to prompt
MarineGPT to extend the generation based on different user intent and domain-
specific questions about the identified marine organism, such as its habitat,
behavior, ecological role, conservation status, and etc.

Instruction-following data construction. We generate diverse and comprehensive
multi-modal vision-language instruction-following data. We construct 50
different instructions and scalably invoke instruction-following data based on
crawled marine knowledge from both open-source public marine websites and
ChatGPT/GPT-4. We provide some designed instructions as follows:

  * •

Instruction 1. “please describe the species richness and distribution of
$<$image category$>$.”

  * •

Instruction 2. “please answer what are the predator-prey relationships for the
$<$image category$>$ and how they influence population dynamics.”

  * •

Instruction 3. “please answer how this $<$image category$>$ interacts with
other species in marine ecosystems.”

  * •

Instruction 4. “please answer the conservation status of $<$image category$>$,
including their population trends, threats they face, and the effectiveness of
existing conservation measures.”

The constructed instruction-following data will be utilized for optimizing our
MarineGPT to generate more user-centric and scientific responses.

High-quality image-text pair formulation. Besides leveraging the capabilities
of ChatGPT/GPT-4 to formulate the instruction-following training data. We also
follow our summarized attribute-based image description generation pipeline
(refer to Fig. 3). By combining web-scraped image-text data from open-source
marine websites and domain-specific knowledge, we could enable MarineGPT to
generate more reliable, accurate, and scientific answers aligned with the user
intent. We dedicate substantial efforts to gathering data from diverse
sources, including books, biology blogs, scientific articles, and open-source
marine websites. The constructed image-text pairs from a broad spectrum of
sources, including public websites, various open-source books, guidebooks, and
our privately collected diving and surveying data. After sophisticated and
extensive data cleaning and caption alignment, we generate longer and more
domain-specific image captions.

Figure 3: The procedure of our attribute-based image description generation
pipeline.

We generate 1.12 million scientifically constructed instruction-following
question-answer pairs for marine knowledge foundation model training. This
marine-specific data generation approach enables us to generate more image-
text pairs with detailed and informative image descriptions. To facilitate
marine multi-modal research, we will release our constructed marine-specific
instruction-following data and the trained MarineGPT model. The marine and
biology researchers could then further formulate their own high-quality
vision-language datasets, thereby promoting advancements in the marine
foundation model community.

####  3.3.2 Instruction-Following Fine-tuning

MarineGPT could produce more detailed and scientific image descriptions with
much more diversity and align with the user intent. We provide some prompt
templates as shown below:

  * •

Prompt 1. “Can you answer what ecological roles does the marine organism in
the $<$image$>$ play in their ecosystems?”

  * •

Prompt 2. “Could you describe how do climate changes affect the distribution,
reproduction, and survival of the object in the $<$image$>$?”

  * •

Prompt 3. “Please determine the scientific name of object in this $<$image$>$,
classification within the taxonomic hierarchy, and its relationships to other
known species.”

  * •

Prompt 4. “Take a look at this image and describe how can we mitigate and
human-induced threats to the object in this $<$image$>$.”

Please note that the image prompts are aligned with the previous 50
constructed instructions. We conduct instruction-following fine-tuning to the
marine domain for end-to-end training of a marine visual-to-language
conversational assistant, which could promote the capabilities and usability
of AI-powered systems, making them more versatile, context-aware, and user-
centric.

##  4 Experiments

###  4.1 Implementation Details

MarineGPT aims to achieve cross-modality visual-and-language alignment between
the visual observations and LLMs. To achieve effective visual perception, we
adopt the same visual encoder as used in BLIP-2 (Li et al., 2023b), a ViT
backbone with a pre-trained Q-Former. For the large language model, we adopt
LLaMA-13B (Touvron et al., 2023a) as the decoder to generate responses. Both
language and vision models are open-sourced. We target to bridge the gap
between the visual encoder and LLM based on the Q-Former along with additional
linear layers for computing the similarity between the visual content and
captions. Through such pre-training, we could perform the marine vision-
language alignment.

Additionally, we convert the data type of parameters of the frozen ViT
(Dosovitskiy et al., 2020) and language model to FP16 during the continuous
pre-training to increase computational efficiency. At the first continuous
marine-specific pre-training, we focus on marine image-text alignment, which
involves using our constructed Marine-5M dataset with a large range of marine
image-text pairs. To relieve the computation burden brought by large-scale
image-text pairs, we adopt a low-resolution input ($224\times 224$) for
projecting image-text pairs into the same semantic space. At the second
instruction-following fine-tuning procedure, we adopt a larger image
resolution ($384\times 384$) to provide more detailed information. In the
second fine-tuning procedure, we only adopt the constructed 1.12 million high-
quality marine image-text pairs (with both the template-based image captions
and our instruction-following ChatGPT/GPT-4 generated captions) to enable
MarineGPT more powerful.

###  4.2 Marine-5M Dataset

Dataset construction. To promote the image diversity of our Marine-5M dataset,
we include marine image-text pairs from multiple different sources, including
YouTube, Flickr, Google image engine, open-source marine websites (_e.g._ ,
FishDB (Yang et al., 2020), Fishes-of-Australia (Bray, 2018), Corals-of-World
(E. & L.M, 2016), Reeflex (ree, ), etc) and publicly available marine
datasets. Our Marine-5M dataset contains about 100K marine and marine-relative
object conceptions. For video sequences from YouTube, we especially download
marine documentary videos, which contain high-quality marine image-text pairs.
We utilize the subscripts to clip the keyframes and generate the aligned
marine image-text pairs. The human-based checking and refinement are conducted
to get more accurate and reliable text descriptions. The diving, surveying,
and transect videos are also included to promote the diversity of our
Marine-5M dataset. We extract one keyframe every 10 seconds for those videos
without subscripts. The images from our Marine-5M dataset have a large
diversity, in which the appearance, boundary, shape, and texture of marine
objects vary significantly. Some images are captured in the wild and with low
visibility, background clutter, motion blur, occlusion, dynamic illumination,
color distortion, and optical artifacts. The Marine-5M dataset also contains
web-scraped image-text data from open-source marine websites. These web-
scraped image-text pairs usually contain the category annotation and some
brief summary of the marine object present in the visual images. Such pairs
are usually high-quality and provide clean object conceptions centralized in
images, which could promote the marine vision-language alignment.

As for the text descriptions, we expand the text description based on the
provided category annotation considering that the category annotation or the
image caption is over-simple. We summarize 129 different attributes from the
category annotation as demonstrated in Fig. 3. We generate corresponding
captions based on these diverse attributes and category annotations. It is
worth noting that not all marine object conceptions contain all these
attribute annotations. The images with the same category annotation are
assigned different attribute descriptions to promote the diversity of text
captions and inject marine knowledge. For those images without any annotation,
we propose to borrow the BLIP-2 (Li et al., 2023b) to generate longer image
captions for the marine images. We generate 5 diversified image caption
candidates through Gaussian sampling in the latent space and then we rank all
the generated image captions based on the caption length. Finally, we compute
the sentence-level similarity between these caption candidates. Only the
caption with a similarity less than 0.85 with the longest caption is appended
for concatenation. In this manner, we could generate more diversified and
longer captions for the marine images, describing more fine-grained color,
texture, shape, and geometry information.

Figure 4: The comparison between MiniGPT-4, GPT-4V and our MarineGPT.
MarineGPT could recognize both the common and scientific names of marine
objects and provide diverse information associated with the recognized
objects. Best viewed in color.

###  4.3 Results

Figure 5: MarineGPT could recognize a wide range of marine objects and yield
comprehensive marine and biological knowledge delivered to the users so that
the users could obtain a full understanding of the recognized marine objects.

Comparison with MiniGPT-4 and GPT-4V. We compare our MarineGPT with MiniGPT-4
and GPT-4V in Fig. 4. Compared with MiniGPT-4 and GPT-4V, our MarineGPT could
generate long and detailed responses with corresponding biology information
delivered (_e.g._ , scientific name and common name of recognized objects). We
observed that MiniGPT-4 will generate some descriptions (“a few plants visible
in the foreground” and “in a coral reef”) that are not correct in the given
marine images. We attribute this hallucination to the ineffective marine
visual-language alignment. Directly optimizing the linear layer based on
general-purpose image-text pairs will result in wrong descriptions for the
marine images. GPT-4V cannot yield detailed marine-specific responses, either,
while it tends to follow some fixed templates to describe the physical
appearances of given images. In contrast, our MarineGPT could generate long
and meaningful responses for domain-specific knowledge delivery. Both GPT-4V
and MarineGPT could achieve accurate scientific name generation (“Arothron
nigropunctatus”) and common name generation (“Black-spotted Pufferfish”) about
the given marine object. However, MarineGPT is confused with dog-faced puffer
fish (another common name of Arothron nigropunctatus) and generates a wrong
scientific name for dog-faced puffer fish. In contrast, due to that GPT-4V
could access the Internet, there is less possibility for GPT-4V to make such
mistake. We attribute this failure of MarineGPT to the ineffectiveness of
frozen LLM (LLaMA-13B (Touvron et al., 2023a) used in this work). It may still
make mistakes for the domain-specific knowledge even if we performed the
marine knowledge injection during the continuous vision-language pre-training.
The further fine-tuning of LLMs to domain-specific domain may effectively
alleviate such issues. Besides, Both GPT-4V and MarineGPT demonstrate an
aesthetic metric summarized from common sense. Finally, compared with
MiniGPT-4 and GPT-4V, MarineGPT could generate diverse and relative
information associated with the recognized objects.

Recognizing a wide range of marine objects. We then provide more recognition
results produced by our MarineGPT in Fig. 5. MarineGPT could recognize a wide
range of marine creatures and provide the corresponding common and scientific
names of recognized objects. It is worth noting that our method could generate
diverse and comprehensive image descriptions for the different recognized
marine objects. MarineGPT could also generate the corresponding references to
provide additional user information, as demonstrated in Fig. 5 a). It can
generate informative responses describing how the recognized object benefits
from the physical appearance shown in Fig. 5 b), how can we protect the
recognized threatened marine species demonstrated in Fig. 5 c), the spatial
distribution of “sarpa salpa” in Fig. 5 d), the physical characteristics of
“black-tip reef shark” in Fig. 5 e) and the feeding diet and social behavior
of “bottlenose dolphin” in Fig. 5 f). In this way, MarineGPT could deliver
marine knowledge to the users.

Figure 6: Our MarineGPT could accurately recognize similar marine objects
under the fine-grained setting.

Fine-grained marine object recognition. MarineGPT could discriminate very
similar marine organisms and generate different responses based on them. We
report the fine-grained object recognition results of MarineGPT in Fig. 6. As
demonstrated, MarineGPT could generate the corresponding scientific names for
these three different fish species and provide further detailed descriptions
from various aspects. This novel fine-grained object recognition ability of
MarineGPT introduced by our Marine-5M dataset could enable diversity
monitoring and reduce the human labor from the fish experts. The existing
general-purpose MLLMs will regard all these three fish species as “fish” and
fail to perform fine-grained recognition.

Figure 7: Our MarineGPT could understand diverse the user intent and yield
corresponding informative responses.

Multi-round conversation. Users could upload images of different marine
objects and ask different questions, as demonstrated in Fig. 7. MarineGPT
could recognize the marine objects present in marine images and generate
corresponding responses aligned with the user intent. The generated responses
cover the detailed information, addressing the user query. Through a user-
friendly interface, MarineGPT could enable individuals to contribute data and
increase public awareness about marine biodiversity and its significance.

Figure 8: MarineGPT could yield comprehensive marine and biological knowledge
delivered to the users so that the users could obtain a full understanding of
the recognized marine organism.

Comprehensive analysis. One specific feature of our MarineGPT is that users
can comprehensively understand the recognized marine object. We provide the
illustration by choosing one fish image with “Abramites hypselonotus” as
demonstrated in Fig. 8. In the multi-round conversations, the users could ask
any question they are interested in the recognized marine species. MarineGPT
could generate sensitive, domain-specific, and scientific responses for the
users. Through training based on the instruction-following multi-modal data,
our model could understand the user intent and generate required
correspondences.

Failure cases. There are still some failure cases in our MarineGPT. It will
refrain from generating long and informative responses for images that contain
instances, that are not strictly “marine creatures” as reported in Fig. 9 a),
b) and c). We have also explored whether MarineGPT suffers from knowledge
forgetting in Fig. 9 d) and e). In d), MarineGPT could effectively recognize
the horse and describe its behavior. But it will generate bizarre responses.
As illustrated in Fig. 9 e), MarineGPT could recognize the instance
accurately, but it misattributes the recognized organism as another species
during the explanation procedure, leading to the in-context hallucination.
With the feedback from the user, MarineGPT could provide the corresponding
informative responses for the users. MarineGPT will also make similar mistakes
for the marine images as demonstrated in Fig. 9 f). We attribute this failure
to the inefficient ability of LLMs in the marine field and the lack of
instruction-following data for such conceptions. Fine-tuning the LLMs to the
marine domain with corresponding instruction-following data may effectively
alleviate such issues.

Figure 9: The failure cases of proposed MarineGPT.

###  4.4 Specialized Features of MarineGPT

Our MarineGPT has the following potential applications and broader impact on
the marine field.

  * •

Scale up marine organism recognition. By integrating machine learning
algorithms, MarineGPT could continually learn from user interactions,
enhancing its species recognition accuracy and knowledge base over time.
Scientists in marine biology and related fields can use the system for species
identification, data retrieval, and ecological research. This accelerates the
acquisition of biodiversity data, supporting research and decision-making
processes.

  * •

Monitoring. Our MarineGPT could be effectively integrated for accessing the
marine species diversity, abundance data, and community structure.
Furthermore, we could also perform the long-term marine species morning, which
provides a unique opportunity to quantitatively examine changes in benthic
habitat over time. It not only aids in marine object identification but also
offers valuable insights into marine organisms’ conservation status and
ecological importance. This information could be used to prioritize
conservation actions and monitor endangered or threatened species.

  * •

Centralized platform. Our MarineGPT provides a centralized platform for marine
researchers, biologists, and institutions to share data, enable collaboration
on marine research, and contribute to a global repository of marine species
information. As the system recognizes and answers questions in a nearly real-
time manner, it would facilitate on-the-spot data collection and recognition
during field research or citizen science efforts. In this way, we can enable
rapid and large-scale data collection, surpassing the limitations of manual
expert identification efforts. Such a large-scale marine database is crucial
for understanding species interactions, feeding diets, and ecosystem dynamics,
leading to better conservation and management strategies for marine
ecosystems.

  * •

Interdisciplinary research. The system serves as an educational tool,
fostering greater understanding and appreciation for marine life among users.
It encourages citizen science participation and promotes environmental
awareness and stewardship. Our method and MarineGPT empower users to access
information on endangered or threatened species, aiding conservationists and
policymakers in implementing targeted conservation measures to protect marine
biodiversity. MarineGPT could also be integrated into educational tools,
enhancing marine biology education and fostering a deeper appreciation for
marine life among students and the public. Students, educators, and the
general public could engage with the system, fostering a deeper understanding
and appreciation for marine life and conservation. Collaborative efforts would
improve the quality and accessibility of marine object information globally.

  * •

General public access. Researchers and enthusiasts could quickly access
species information without requiring manual identification or extensive
literature searches. Conservationists can access information on endangered
species and use the system’s data to make informed decisions for marine
conservation efforts. Teachers and students in marine biology or environmental
studies can utilize the system as an educational tool to enhance learning
about marine species and ecosystems. Enthusiastic individuals interested in
marine life can actively contribute to biodiversity monitoring and research by
using the system during their marine observations or expeditions. Scuba
divers, snorkelers, and ocean enthusiasts can utilize the system to identify
marine species encountered during their underwater adventures.

##  5 Discussions

Hallucination. Our MarineGPT inherits the hallucination problem from the
frozen LLMs. Future advancements in more powerful LLMs are expected to
alleviate this issue. Moreover, it is highly worthwhile to explore appropriate
approaches to promote the ability of LLMs for domain-specific understanding.
Through this, we could generate more domain-specific descriptions and
knowledge for the images. We leave this as our future work.

Video-centric MarineGPT. The marine videos have not been considered integrated
into chatting yet. The reasons probably come from the difficulty of accurately
understanding non-static visual scenes. Besides, mitigating the modality gap
between video and text (Li et al., 2023c; Zhang et al., 2023a), which
typically requires handling the temporary information while simultaneously
considering both visual signals and audio signals. It is more challenging than
bridging the gap between image and text. Our next work is to analyze the video
with temporary collections, on how to summarize the relationships between
different marine organisms. The massive video-caption pairs, as well as
higher-quality and domain-specific visual-instruction-tuning datasets, are
emergently required.

##  6 Conclusion

In this paper, we propose the first marine-specific multi-modal large language
model, MarineGPT, which could generate more sensitive, informative, and
scientific responses than existing general-purpose MLLMs. We present the
largest marine image-text dataset with 5 million pairs and instruction-
following cross-modality vision-language data for adapting a general-purpose
AI assistant to a marine expert model. We propose a standard and valuable
pipeline to develop a domain-specific expert model that involves specific
knowledge. We will release our trained models and designed instructions to the
research community to foster future research in both academic and industrial
communities.

## References

  * (1) https://www.reeflex.net/.
  * eol (2018) Encyclopedia of life.  http://eol.org, 2018.
  * Alayrac et al. (2022) Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al.  Flamingo: a visual language model for few-shot learning.  _Advances in Neural Information Processing Systems_ , 35:23716–23736, 2022.
  * Bray (2018) M.F. (eds) Bray, D.J. & Gomon.  Fishes of australia.  http://fishesofaustralia.net.au/, 2018.
  * Brown et al. (2020) Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al.  Language models are few-shot learners.  _Advances in neural information processing systems_ , 33:1877–1901, 2020.
  * Byeon et al. (2022) Minwoo Byeon, Beomhee Park, Haecheon Kim, Sungjun Lee, Woonhyuk Baek, and Saehoon Kim.  Coyo-700m: Image-text pair dataset.  https://github.com/kakaobrain/coyo-dataset, 2022.
  * Changpinyo et al. (2021) Soravit Changpinyo, Piyush Sharma, Nan Ding, and Radu Soricut.  Conceptual 12m: Pushing web-scale image-text pre-training to recognize long-tail visual concepts.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pp. 3558–3568, 2021.
  * Chiang et al. (2023) Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E Gonzalez, et al.  Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality.  _See https://vicuna. lmsys. org (accessed 14 April 2023)_ , 2023.
  * Chung et al. (2022) Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al.  Scaling instruction-finetuned language models.  _arXiv preprint arXiv:2210.11416_ , 2022.
  * Cox et al. (2023) Aram Cox, Ishith Seth, Yi Xie, David J Hunter-Smith, and Warren M Rozen.  Utilizing chatgpt-4 for providing medical information on blepharoplasties to patients.  _Aesthetic Surgery Journal_ , pp. sjad096, 2023.
  * Diao et al. (2023) Shizhe Diao, Rui Pan, Hanze Dong, Ka Shun Shum, Jipeng Zhang, Wei Xiong, and Tong Zhang.  Lmflow: An extensible toolkit for finetuning and inference of large foundation models.  _arXiv preprint arXiv:2306.12420_ , 2023.
  * Dosovitskiy et al. (2020) Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al.  An image is worth 16x16 words: Transformers for image recognition at scale.  _arXiv preprint arXiv:2010.11929_ , 2020.
  * E. & L.M (2016) Turak E. and DeVantier L.M.  Corals of the world.  http://coralsoftheworld.org/v0.01(Beta), 2016.
  * Fu et al. (2023) Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Zhenyu Qiu, Wei Lin, Jinrui Yang, Xiawu Zheng, et al.  Mme: A comprehensive evaluation benchmark for multimodal large language models.  _arXiv preprint arXiv:2306.13394_ , 2023.
  * Guo et al. (2023) Zhen Guo, Peiqi Wang, Yanwei Wang, and Shangdi Yu.  Dr. llama: Improving small language models in domain-specific qa via generative data augmentation.  _arXiv preprint arXiv:2305.07804_ , 2023.
  * Hong et al. (2023) Yining Hong, Haoyu Zhen, Peihao Chen, Shuhong Zheng, Yilun Du, Zhenfang Chen, and Chuang Gan.  3d-llm: Injecting the 3d world into large language models.  _arXiv preprint arXiv:2307.12981_ , 2023.
  * Krishna et al. (2017) Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A Shamma, et al.  Visual genome: Connecting language and vision using crowdsourced dense image annotations.  _International journal of computer vision_ , 123:32–73, 2017.
  * Li et al. (2023a) Chunyuan Li, Cliff Wong, Sheng Zhang, Naoto Usuyama, Haotian Liu, Jianwei Yang, Tristan Naumann, Hoifung Poon, and Jianfeng Gao.  Llava-med: Training a large language-and-vision assistant for biomedicine in one day.  _arXiv preprint arXiv:2306.00890_ , 2023a.
  * Li et al. (2022) Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi.  Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation.  In _International Conference on Machine Learning_ , pp. 12888–12900. PMLR, 2022.
  * Li et al. (2023b) Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi.  Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models.  _arXiv preprint arXiv:2301.12597_ , 2023b.
  * Li et al. (2023c) KunChang Li, Yinan He, Yi Wang, Yizhuo Li, Wenhai Wang, Ping Luo, Yali Wang, Limin Wang, and Yu Qiao.  Videochat: Chat-centric video understanding.  _arXiv preprint arXiv:2305.06355_ , 2023c.
  * Li et al. (2023d) Tianhao Li, Sandesh Shetty, Advaith Kamath, Ajay Jaiswal, Xianqian Jiang, Ying Ding, and Yejin Kim.  Cancergpt: Few-shot drug pair synergy prediction using large pre-trained language models.  _arXiv preprint arXiv:2304.10946_ , 2023d.
  * Lin et al. (2014) Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick.  Microsoft coco: Common objects in context.  In _Proceedings of the European Conference on Computer Vision_ , pp. 740–755. Springer, 2014.
  * Liu et al. (2023) Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee.  Visual instruction tuning.  _arXiv preprint arXiv:2304.08485_ , 2023.
  * Lu et al. (2022) Pan Lu, Swaroop Mishra, Tanglin Xia, Liang Qiu, Kai-Wei Chang, Song-Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan.  Learn to explain: Multimodal reasoning via thought chains for science question answering.  _Advances in Neural Information Processing Systems_ , 35:2507–2521, 2022.
  * Mu et al. (2023) Yao Mu, Qinglong Zhang, Mengkang Hu, Wenhai Wang, Mingyu Ding, Jun Jin, Bin Wang, Jifeng Dai, Yu Qiao, and Ping Luo.  Embodiedgpt: Vision-language pre-training via embodied chain of thought.  _arXiv preprint arXiv:2305.15021_ , 2023.
  * Nijkamp et al. (2022) Erik Nijkamp, Bo Pang, Hiroaki Hayashi, Lifu Tu, Huan Wang, Yingbo Zhou, Silvio Savarese, and Caiming Xiong.  Codegen: An open large language model for code with multi-turn program synthesis.  _arXiv preprint arXiv:2203.13474_ , 2022.
  * OpenAI (2022) OpenAI.  Introducing chatgpt.  2022\.  URL https://openai.com/blog/chatgpt.
  * OpenAI (2023) OpenAI.  Gpt-4 technical report, 2023.
  * Ouyang et al. (2022) Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al.  Training language models to follow instructions with human feedback.  _Advances in Neural Information Processing Systems_ , 35:27730–27744, 2022.
  * Raffel et al. (2020) Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu.  Exploring the limits of transfer learning with a unified text-to-text transformer.  _The Journal of Machine Learning Research_ , 21(1):5485–5551, 2020.
  * Scao et al. (2022) Teven Le Scao, Angela Fan, Christopher Akiki, Ellie Pavlick, Suzana Ilic, Daniel Hesslow, Roman Castagné, Alexandra Sasha Luccioni, François Yvon, Matthias Gallé, Jonathan Tow, Alexander M. Rush, Stella Biderman, Albert Webson, Pawan Sasanka Ammanamanchi, Thomas Wang, Benoît Sagot, Niklas Muennighoff, Albert Villanova del Moral, Olatunji Ruwase, Rachel Bawden, Stas Bekman, Angelina McMillan-Major, Iz Beltagy, Huu Nguyen, Lucile Saulnier, Samson Tan, Pedro Ortiz Suarez, Victor Sanh, Hugo Laurençon, Yacine Jernite, Julien Launay, Margaret Mitchell, Colin Raffel, Aaron Gokaslan, Adi Simhi, Aitor Soroa, Alham Fikri Aji, Amit Alfassy, Anna Rogers, Ariel Kreisberg Nitzav, Canwen Xu, Chenghao Mou, Chris Emezue, Christopher Klamm, Colin Leong, Daniel van Strien, David Ifeoluwa Adelani, and et al.  BLOOM: A 176b-parameter open-access multilingual language model.  _CoRR_ , abs/2211.05100, 2022.  doi: 10.48550/arXiv.2211.05100.  URL https://doi.org/10.48550/arXiv.2211.05100.
  * Schuhmann et al. (2021) Christoph Schuhmann, Richard Vencu, Romain Beaumont, Robert Kaczmarczyk, Clayton Mullis, Aarush Katta, Theo Coombes, Jenia Jitsev, and Aran Komatsuzaki.  Laion-400m: Open dataset of clip-filtered 400 million image-text pairs.  _arXiv preprint arXiv:2111.02114_ , 2021.
  * Schuhmann et al. (2022) Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al.  Laion-5b: An open large-scale dataset for training next generation image-text models.  _Advances in Neural Information Processing Systems_ , 35:25278–25294, 2022.
  * Sun et al. (2023) Yuxuan Sun, Chenglu Zhu, Sunyi Zheng, Kai Zhang, Zhongyi Shui, Xiaoxuan Yu, Yizhi Zhao, Honglin Li, Yunlong Zhang, Ruojia Zhao, et al.  Pathasst: Redefining pathology through generative foundation ai assistant for pathology.  _arXiv preprint arXiv:2305.15072_ , 2023.
  * Taylor et al. (2022) Ross Taylor, Marcin Kardas, Guillem Cucurull, Thomas Scialom, Anthony Hartshorn, Elvis Saravia, Andrew Poulton, Viktor Kerkez, and Robert Stojnic.  Galactica: A large language model for science.  _CoRR_ , abs/2211.09085, 2022.  doi: 10.48550/arXiv.2211.09085.  URL https://doi.org/10.48550/arXiv.2211.09085.
  * Touvron et al. (2023a) Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurélien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample.  Llama: Open and efficient foundation language models.  _CoRR_ , abs/2302.13971, 2023a.  doi: 10.48550/arXiv.2302.13971.  URL https://doi.org/10.48550/arXiv.2302.13971.
  * Touvron et al. (2023b) Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al.  Llama 2: Open foundation and fine-tuned chat models.  _arXiv preprint arXiv:2307.09288_ , 2023b.
  * Tu et al. (2023) Tao Tu, Shekoofeh Azizi, Danny Driess, Mike Schaekermann, Mohamed Amin, Pi-Chuan Chang, Andrew Carroll, Chuck Lau, Ryutaro Tanno, Ira Ktena, et al.  Towards generalist biomedical ai.  _arXiv preprint arXiv:2307.14334_ , 2023.
  * Wu et al. (2023a) Chenfei Wu, Shengming Yin, Weizhen Qi, Xiaodong Wang, Zecheng Tang, and Nan Duan.  Visual chatgpt: Talking, drawing and editing with visual foundation models.  _arXiv preprint arXiv:2303.04671_ , 2023a.
  * Wu et al. (2023b) Shijie Wu, Ozan Irsoy, Steven Lu, Vadim Dabravolski, Mark Dredze, Sebastian Gehrmann, Prabhanjan Kambadur, David Rosenberg, and Gideon Mann.  Bloomberggpt: A large language model for finance.  _arXiv preprint arXiv:2303.17564_ , 2023b.
  * Yang et al. (2023) Hongyang Yang, Xiao-Yang Liu, and Christina Dan Wang.  Fingpt: Open-source financial large language models.  _arXiv preprint arXiv:2306.06031_ , 2023.
  * Yang et al. (2020) Liandong Yang, Zetan Xu, Honghui Zeng, Ning Sun, Baosheng Wu, Cheng Wang, Jing Bo, Lin Li, Yang Dong, and Shunping He.  Fishdb: an integrated functional genomics database for fishes.  _BMC genomics_ , 21(1):1–5, 2020.
  * Zang et al. (2023) Yuhang Zang, Wei Li, Jun Han, Kaiyang Zhou, and Chen Change Loy.  Contextual object detection with multimodal large language models.  _arXiv preprint arXiv:2305.18279_ , 2023.
  * Zhang et al. (2023a) Hang Zhang, Xin Li, and Lidong Bing.  Video-llama: An instruction-tuned audio-visual language model for video understanding.  _arXiv preprint arXiv:2306.02858_ , 2023a.
  * Zhang et al. (2022) Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona T. Diab, Xian Li, Xi Victoria Lin, Todor Mihaylov, Myle Ott, Sam Shleifer, Kurt Shuster, Daniel Simig, Punit Singh Koura, Anjali Sridhar, Tianlu Wang, and Luke Zettlemoyer.  OPT: open pre-trained transformer language models.  _CoRR_ , abs/2205.01068, 2022.  doi: 10.48550/arXiv.2205.01068.  URL https://doi.org/10.48550/arXiv.2205.01068.
  * Zhang et al. (2023b) Xiaoman Zhang, Chaoyi Wu, Ziheng Zhao, Weixiong Lin, Ya Zhang, Yanfeng Wang, and Weidi Xie.  Pmc-vqa: Visual instruction tuning for medical visual question answering.  _arXiv preprint arXiv:2305.10415_ , 2023b.
  * Zhao et al. (2023) Zijia Zhao, Longteng Guo, Tongtian Yue, Sihan Chen, Shuai Shao, Xinxin Zhu, Zehuan Yuan, and Jing Liu.  Chatbridge: Bridging modalities with large language model as a language catalyst.  _arXiv preprint arXiv:2305.16103_ , 2023.
  * Zhu et al. (2023) Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny.  Minigpt-4: Enhancing vision-language understanding with advanced large language models.  _arXiv preprint arXiv:2304.10592_ , 2023.