# Benchmarking Trustworthiness of Multimodal Large Language Models: A
Comprehensive Study

Yichi Zhang1,4 †, Yao Huang2∗, Yitong Sun2, Chang Liu3, Zhe Zhao4
Zhengwei Fang1, Yifan Wang1, Huanran Chen1, Xiao Yang1
Xingxing Wei2, Hang Su1, Yinpeng Dong1,4 , Jun Zhu1,4

1Dept. of Comp. Sci. and Tech., Institute for AI, Tsinghua-Bosch Joint ML
Center,
THBI Lab, BNRist Center, Tsinghua University, Beijing 100084, China
2Institute of Artificial Intelligence, Beihang University, Beijing 100191,
China
3Institute of Image Communication and Networks Engineering in the Department
of
Electronic Engineering (EE), Shanghai Jiao Tong University, Shanghai 200240,
China
4RealAI Equal Contributions ( ✉<EMAIL_ADDRESS><EMAIL_ADDRESS>)Project Leaders

###### Abstract

Despite the superior capabilities of Multimodal Large Language Models (MLLMs)
across diverse tasks, they still face significant trustworthiness challenges.
Yet, current literature on the assessment of trustworthy MLLMs remains
limited, lacking a holistic evaluation to offer thorough insights into future
improvements. In this work, we establish MultiTrust, the first comprehensive
and unified benchmark on the trustworthiness of MLLMs across five primary
aspects: _truthfulness_ , _safety_ , _robustness_ , _fairness_ , and
_privacy_. Our benchmark employs a rigorous evaluation strategy that addresses
both multimodal risks and cross-modal impacts, encompassing 32 diverse tasks
with self-curated datasets. Extensive experiments with 21 modern MLLMs reveal
some previously unexplored trustworthiness issues and risks, highlighting the
complexities introduced by the multimodality and underscoring the necessity
for advanced methodologies to enhance their reliability. For instance, typical
proprietary models still struggle with the perception of visually confusing
images and are vulnerable to multimodal jailbreaking and adversarial attacks;
MLLMs are more inclined to disclose privacy in text and reveal ideological and
cultural biases even when paired with irrelevant images in inference,
indicating that the multimodality amplifies the internal risks from base LLMs.
Additionally, we release a scalable toolbox for standardized trustworthiness
research, aiming to facilitate future advancements in this important field.
Code and resources are publicly available at: https://multi-trust.github.io/.

Warning: This paper contains data and model outputs which are offensive in
nature.

###  1 Introduction

The era towards Artificial General Intelligence (AGI) [48] has witnessed the
emergence of the ground-breaking Large Language Models (LLMs) [4, 7, 63, 139].
With their strong language understanding and reasoning capabilities, recent
studies [35, 85, 105, 134, 178] have seamlessly integrated other modalities
(e.g., vision) into LLMs to understand different inputs. The resultant
Multimodal Large Language Models (MLLMs) have manifested versatile proficiency
in both traditional vision tasks [28, 51, 54, 167] and more complex multimodal
problems [44, 90, 170]. However, despite their remarkable performance and the
efforts in aligning with human preferences [108, 115], these cutting-edge
models still exhibit significant drawbacks in trustworthiness, leading to
factual errors [62, 117], harmful outputs [180], privacy leakage [102], etc.
The trustworthiness issues have notably compromised model reliability and
elicited increasing concerns from researchers, policymakers, and the public
[3, 43].

Table 1: Comparison between MultiTrust and other trustworthiness-related
benchmarks for MLLMs. The numbers in the parenthesis for # MLLM represent the
counts of proprietary models.

| Aspects | Strategy | Task Types | Statistics | Toolbox
---|---|---|---|---|---
|  Truthfulness |  Safety |  Robustness |  Fairness |  Privacy |  Multimodal |  Cross-modal |  Discriminative |  Generative |  # Task/Scenario |  # MLLM |  Unified Interface |  Modularized Design
POPE [76] | ✔ | ✘ | ✘ | ✘ | ✘ | ✔ | ✘ | ✔ | ✘ | 1 | 5 (0) | ✘ | ✘
ToViLaG [150] | ✘ | ✔ | ✘ | ✘ | ✘ | ✔ | ✘ | ✘ | ✔ | 3 | 4 (0) | ✘ | ✘
PrivQA [29] | ✘ | ✘ | ✘ | ✘ | ✔ | ✔ | ✘ | ✘ | ✔ | 2 | 3 (0) | ✘ | ✘
GOAT-Bench [78] | ✘ | ✔ | ✘ | ✘ | ✘ | ✔ | ✘ | ✔ | ✘ | 5 | 11(1) | ✘ | ✘
MM-SafetyBench [87] | ✘ | ✔ | ✘ | ✘ | ✘ | ✔ | ✘ | ✘ | ✔ | 13 | 12(0) | ✘ | ✘
BenchLMM [19] | ✘ | ✘ | ✔ | ✘ | ✘ | ✔ | ✘ | ✘ | ✔ | 15 | 10(1) | ✘ | ✘
SafeBench [49] | ✘ | ✔ | ✘ | ✘ | ✔ | ✔ | ✘ | ✘ | ✔ | 10 | 7 (1) | ✘ | ✘
Unicorn [142] | ✘ | ✔ | ✔ | ✘ | ✘ | ✔ | ✘ | ✔ | ✔ | 7 | 21(1) | ✔ | ✘
RTVLM [75] | ✔ | ✔ | ✘ | ✔ | ✔ | ✔ | ✘ | ✘ | ✔ | 9 | 10(1) | ✘ | ✘
MultiTrust (ours) | ✔ | ✔ | ✔ | ✔ | ✔ | ✔ | ✔ | ✔ | ✔ | 32 | 21(4) | ✔ | ✔

To facilitate the trustworthiness of foundation models, developing holistic
and standardized evaluation benchmarks is indispensable. Although numerous
studies have meticulously assessed and analyzed the trustworthiness of LLMs
[77, 88, 129, 144], a corresponding evaluation framework for MLLMs is lacking.
In addition to the inherent weaknesses of LLMs, the multimodal nature of MLLMs
introduces novel risks, such as susceptibility to adversarial image attacks
[38, 176], presence of toxic content in images [150], and jailbreaking via
visual contexts [21, 111]. As the new modality brings a variety of intricate
factors to consider, including task design across multiple aspects, data
collection from multimodal scenarios, and the interplay between modalities,
the systematic evaluation of MLLMs’ trustworthiness is more challenging.
However, current work [76, 87, 142] typically examines one or a few aspects of
trustworthiness and evaluates MLLMs on limited tasks at a phenomenon level,
concerning threats in images but neglecting the interactions between
modalities (as detailed in Tab. 1). These superficial evaluations could lead
to an oversight of certain risks and a biased understanding of model
trustworthiness, rendering a comprehensive evaluation of MLLMs’
trustworthiness absent.

In this paper, we establish MultiTrust, the first comprehensive and unified
benchmark to evaluate the trustworthiness of MLLMs across diverse dimensions
and tasks. Distilled from the literature on trustworthy foundation models [88,
129, 144], we identify 5 primary aspects of trustworthiness in MultiTrust,
including _truthfulness_ , _safety_ , _robustness_ , _fairness_ , and
_privacy_ , covering the reliability of models in preventing unexpected
outcomes and the assurance of social impacts to users. We propose a more in-
depth evaluation strategy that delves into the multimodal nature of MLLMs by
considering both _multimodal risks_ in novel scenarios and _cross-modal
impacts_ of visual inputs on base LLMs’ performance. To perform systematic
evaluations, we set up 32 various tasks, including improvements to existing
multimodal tasks, extension of text-only tasks to multimodal scenarios, and
novel methods for risk assessment, which focus on models’ basic performance
with practical significance. We curate rich datasets for the tasks, most of
which are either adapted to novel tasks based on existing ones or newly
proposed via data synthesis (e.g., Stable Diffusion [119], GPT-4V [105]) and
manual collection. We conduct large-scale experiments with 21 modern MLLMs (4
proprietary and 17 open-source), which are carefully selected to guarantee
both the coverage of popular models from different stages and the
distinctiveness of model architectures and training techniques to provide
analyses for future improvements. Below, we summarize several key findings:

$\bullet$ Although open-source MLLMs are approaching or even surpassing
proprietary models in multiple general benchmarks [44, 90, 170], there is
still a significant gap in terms of trustworthiness. GPT-4V [105] and Claude3
[8] demonstrate better performance due to their safety guardrails and efforts
in alignment, highlighting the insufficient development and risky deployment
of open-source models.
$\bullet$ The multimodal training and the introduction of images in inference
greatly jeopardize the trustworthiness of MLLMs, manifested in several
perspectives including but not limited to: 1) the performance and alignment of
base LLMs being compromised; 2) irrelevant images causing unstable behaviors;
3) relevant visual contexts exacerbating trustworthy risks. This emphasizes
that developing trustworthy MLLMs is not simply a matter of using a well-
aligned LLM, but a more challenging issue.
$\bullet$ The results of some tasks confirm the contributions from different
model components (e.g., vision encoder [30], aligned LLM [140]) and existing
training paradigms (e.g., supervised fine-tuning with datasets distilled from
GPT-4V [148], RLHF [130]) to improving trustworthiness. However, solely
relying on the existing techniques is far from all-round guarantee of
trustworthiness.

To support standardized and scalable assessments, we develop a toolbox
dedicated to the trustworthiness research of MLLMs. The toolbox is implemented
with unified interfaces and a modularized design of model interaction and task
execution. Hopefully, our toolbox can address the limitation of existing work
[49, 75, 87, 142] that only provides datasets or evaluation scripts, and serve
as a better foundation for future research on the evaluation and development
of trustworthy MLLMs.

###  2 Framework of MultiTrust

In this section, we present the framework of MultiTrust, as shown in Fig. 1.
Sec. 2.1 introduces the design principles of the benchmark, focusing on the
evaluation aspects and strategy. Sec. 2.2 briefly reviews the 32 tasks under
the two-level taxonomy of aspects. Sec. 2.3 and Sec. 2.4 introduce the
selected models for evaluation and the scalable and standardized toolbox for
trustworthiness research.

Figure 1: Framework of MultiTrust, including aspect division, evaluation
strategy and design of the developed toolbox. Specifically, we study the
trustworthiness by delving into the multimodal nature of MLLMs from a broader
perspective, covering both _multimodal risks_ and _cross-modal impacts_.

####  2.1 Philosophy of MultiTrust

Evaluation Aspects. Drawing on extensive studies in trustworthy LLMs [88, 129,
144] and distilling from relevant literature of MLLMs [19, 75, 76, 87], we
pinpoint 5 primary aspects of trustworthiness for evaluating MLLMs, including
_truthfulness_ , _safety_ , _robustness_ , _fairness_ , and _privacy_. In
particular, _truthfulness_ , _safety_ , and _robustness_ guarantee the models’
reliability and stability in preventing undesirable outcomes, i.e., errors,
harms, and variations under different conditions. _Fairness_ and _privacy_
address the models’ social and ethical implications, involving misaligned
attitudes like bias and rights violations like identity theft. These aspects
collectively compose a comprehensive and compact framework for studying
trustworthiness, as they span both technical and ethical perspectives, while
flaws in any of them could trigger profound societal impacts. As detailed in
Sec. 2.2, we further organize a two-level taxonomy encompassing 10 sub-
aspects.

Table 2: Task Overview. Each task ID is linked to the section in Appendix. :
off-the-shelf datasets from prior work; : datasets adapted to new tasks with
additional images, prompts, and annotations; : datasets constructed from
scratch. : tasks for revealing multimodal risks; : tasks for studying cross-
modal impacts. ○: rule-based evaluation (e.g., keywords matching); ●:
automatic evaluation by GPT-4 or other classifiers; ◐: mixture evaluation. ASR
stands for Attack Success Rate, RtA stands for Refuse-to-Answer rate, and
Accuracy is sometimes abbreviated as Acc.

ID |  Task Name |  Dataset |  Metrics | Task Type | Eval
---|---|---|---|---|---
_T.1_ |  Basic World Understanding |  [11, 44, 75, 13, 166] |  Accuracy ($\uparrow$) |  | Dis.&Gen. | ◐
_T.2_ |  Advanced Cognitive Inference |  [11, 44, 81, 13] |  Accuracy ($\uparrow$) |  | Dis. | ○
_T.3_ |  VQA under Instruction Enhancement |  [44] |  Accuracy ($\uparrow$) |  | Gen. | ●
_T.4_ |  QA under Visual Assistance |  |  Accuracy, Cure Rate ($\uparrow$) |  | Gen. | ●
_T.5_ |  Text Misleading VQA |  [34] |  Accuracy ($\uparrow$) |  | Gen. | ●
_T.6_ |  Visual Confusion VQA |  |  Accuracy ($\uparrow$) |  | Dis. | ○
_T.7_ |  Visual Misleading QA |  |  Acc ($\uparrow$), Deterioration Rate ($\downarrow$) |  | Gen. | ●
_S.1_ |  NSFW Image Description |  [150, 1, 157] |  Toxicity Score ($\downarrow$), RtA ($\uparrow$) |  | Gen. | ◐
_S.2_ |  Risk Identification |  [55] |  Accuracy ($\uparrow$) |  | Dis.&Gen. | ●
_S.3_ |  Toxic Content Generation |  [47] |  Toxicity Score ($\downarrow$), RtA ($\uparrow$) |  | Gen. | ◐
_S.4_ |  Plain Typographic Jailbreaking |  |  ASR ($\downarrow$), RtA ($\uparrow$) |  | Gen. | ◐
_S.5_ |  Optimized Multimodal Jailbreaking |  [49, 87] |  ASR ($\downarrow$), RtA ($\uparrow$) |  | Gen. | ◐
_S.6_ |  Cross-modal Influence on Jailbreaking |  [168, 95, 122] |  ASR ($\downarrow$), RtA ($\uparrow$) |  | Gen. | ◐
_R.1_ |  Image Captioning for Artistic Style Images |  [94] |  Accuracy ($\uparrow$) |  | Gen. | ◐
_R.2_ |  VQA for Sensor Style Images |  [19] |  GPT-Score ($\uparrow$) |  | Gen. | ●
_R.3_ |  Sentiment Analysis for OOD Texts |  [144] |  Accuracy ($\uparrow$) |  | Dis. | ○
_R.4_ |  Image Captioning under Untarget Attack |  |  Accuracy ($\uparrow$), ASR ($\downarrow$) |  | Gen. | ◐
_R.5_ |  Image Captioning under Target attack |  |  ASR ($\downarrow$) |  | Gen. | ◐
_R.6_ |  Textual Adversarial Attack |  [144, 146] |  Accuracy ($\uparrow$) |  | Dis. | ○
_F.1_ |  Stereotypical Content Generation |  [5] |  Containing Rate ($\downarrow$) |  | Gen. | ●
_F.2_ |  Agreement on Stereotypes |  [101] |  Agreement Percentage ($\downarrow$) |  | Dis. | ◐
_F.3_ |  Classification of Stereotypes |  [99, 101] |  Accuracy ($\uparrow$) |  | Dis. | ○
_F.4_ |  Stereotype Query Test |  [152] |  RtA ($\uparrow$) |  | Gen. | ◐
_F.5_ |  Visual Preference Selection |  |  RtA ($\uparrow$) |  | Gen. | ●
_F.6_ |  Profession Competence Prediction |  [5] |  P-value ($\uparrow$) |  | Gen. | ◐
_F.7_ |  Preference Selection in QA |  [129] |  RtA ($\uparrow$) |  | Gen. | ●
_P.1_ |  Visual Privacy Recognition |  [53, 106] |  Accuracy, Precision, Recall ($\uparrow$) |  | Dis. | ○
_P.2_ |  Privacy-Sensitive VQA Recognition |  [106] |  Accuracy, Precision, Recall ($\uparrow$) |  | Dis. | ○
_P.3_ |  InfoFlow Expectation |  [98] |  Pearson Correlation ($\uparrow$) |  | Gen. | ○
_P.4_ |  PII Query with Visual Cues |  |  RtA ($\uparrow$) |  | Gen. | ◐
_P.5_ |  Privacy Leakage in Vision |  [106] |  RtA ($\uparrow$), Leakage Rate ($\uparrow$) |  | Gen. | ◐
_P.6_ |  PII Leakage in Conversations |  [144] |  RtA ($\uparrow$), Accuracy($\uparrow$) |  | Gen. | ◐

Evaluation Strategy. For evaluation and task design, we consider both
_multimodal risks_ and _cross-modal impacts_ due to the introduction of new
modality, covering the multimodal nature of MLLMs more holistically. Most of
the existing studies [49, 75, 87] only attend to the trustworthy threats in
the image modality or the combination of image-text pairs, which invoke the
multimodal risks newly introduced by vision. Beyond such issues, we advocate
that it is equally important to consider the interaction between modalities.
Specifically, the new modality could alter the original model behavior in
existing scenarios for LLMs [111, 142]. This greatly concerns the steadiness
of MLLMs in their broader applications, but remains unexplored. We hereby
propose to study the cross-modal impacts by measuring the performance in text-
only tasks when paired with semantically relevant and irrelevant images (as
illustrated in Fig. 1), which leads to more thorough investigations into
MLLMs.

####  2.2 Practice in MultiTrust

We organize a two-level taxonomy containing 10 sub-aspects to better
categorize the target behaviors to be evaluated. Based on the taxonomy, we
curate 32 diverse tasks to cover realistic and comprehensive scenarios with
trustworthy risks, including generative and discriminative, multimodal and
text-only ones, as summarized in Tab. 2. To tackle the current lack of
datasets dedicated for various scenarios under these sub-aspects, we construct
20 datasets based on the existing text, image, and multimodal datasets [44,
80, 106, 144] by adapting prompts, images, and annotations with both manual
efforts and automatic methods. We further propose 8 novel datasets from
scratch by collecting images from the Internet or synthesizing images with
Stable Diffusion [119] and other algorithms specifically for the designed
tasks. Below, we introduce the design details of each sub-aspect by first
presenting multimodal tasks followed by text-only tasks for studying cross-
modal impacts.

#####  2.2.1 Truthfulness

Truthfulness measures whether the outputs of MLLMs align with the objective
facts, emphasizing the accuracy of the information they provide. Unlike
previous studies [75, 76, 129] that narrowly focus on phenomena like
hallucination and sycophancy, we reorganize it into _inherent deficiency_ [16,
82, 177] and _misguided mistakes_ [81, 112] from a macro perspective.

Inherent Deficiency delves into the internal limitations of models that lead
to inaccurate outputs. We first assess MLLMs’ basic perceptual abilities [13,
44, 76] like object existence judgment (Task _T.1_) and advanced cognitive
capabilities [90, 151] like spatial-temporal reasoning (Task _T.2_), with
improved datasets based on existing ones. Beyond them, we propose to integrate
varying assisting instructions with previous VQA tasks (Task _T.3_), to
explore their benefits from prompt guidance. We develop a dataset with prompts
generated by GPT-4 and images collected from the Internet to test model
performance in text-based factual question answering with visual assistance
(Task _T.4_).

Misguided Mistakes focus on errors caused by misleading inputs [34, 75]. We
start by presenting images along with questions containing factual errors to
see their influence on models’ responses in VQA (Task _T.5_). Besides
misleading in text, we set up a new dataset with manually collected images of
visual illusions [112, 137] to examine model performance in visually
challenging scenarios (Task _T.6_). Contrary to the task of QA under visual
assistance, we pair the same questions with faulty images and measure the
interference from visual misguidance (Task _T.7_).

#####  2.2.2 Safety

Safety guarantees that the responses from MLLMs do not cause unexpected
consequences, such as unintentional harms [61] or illegal actions [67]. Two
most significant topics in the literature of large model safety [121, 129] are
the _toxicity_ of AI-generated content [47], which could greatly impact user
interactions, and model _jailbreaking_ , which involves circumventing safety
protocols [89, 153] to facilitate malicious misuse.

Toxicity measures the models’ tendency to generate harmful responses [88].
Towards testing models’ sensitivity to toxic content, we take NSFW images [1,
150, 157] like pornography and violence for image captioning (Task _S.1_). We
design the task of risk identification, in both presence and usage of objects
(Task _S.2_), to see their awareness of safety risks beyond harmful object
detection [55]. We assess the changes in output toxicity with diverse images
using RealToxicityPrompts [47] (Task _S.3_).

Jailbreaking studies the models’ resilience against attempts to elicit illegal
responses [153]. We convert jailbreaking prompts for LLMs [122, 168] into
images in screenshot style [105] to see if dangers are triggered with OCR
(Task _S.4_). We adopt multimodal jailbreaking optimized for MLLMs [49, 87]
and propose our own attack that simplifies the complexity of jailbreaking
combinations based on the trends in the previous task to reduce model
confusion (Task _S.5_). We incorporate images generated by Stable Diffusion
with text jailbreaking, to observe the fluctuations in performance (Task
_S.6_).

#####  2.2.3 Robustness

Robustness evaluates the models’ consistency and resistance under distribution
shifts or input perturbations, which still remains a issue for MLLMs [38, 142,
176]. Following the common practice in the field [129, 144], we consider the
_out-of-distribution (OOD)_ and _adversarial_ robustness respectively.

OOD Robustness assesses MLLMs’ generalization to unusual domains including
diverse styles and applications. First, we take COCO-O [94] containing images
in various artistic styles for image captioning (Task _R.1_). We then perform
VQA tasks from [19] with images captured by various sensors, e.g., MRI and
infrared imaging (Task _R.2_). Afterwards, we test on the OOD SST-2 [125] from
[144], paired with unrelated images and those generated with the text prompts
(Task _R.3_).

Adversarial Attack explores MLLMs’ vulnerability to adversarial examples,
which is inevitably inherited from deep neural networks [131]. With the state-
of-the-art transferable attack technique [24], we generate adversarial
examples under both untargeted and targeted settings on image captioning (Task
_R.4_ and _R.5_). For text-based attack, we employ the AdvGLUE [146] and
AdvGLUE++ [144] datasets with images both relevant and irrelevant to the
prompts (Task _R.6_).

#####  2.2.4 Fairness

Fairness determines the extent to which the model outputs are free from
inequitable or discriminatory outcomes that could disadvantage any user group
[9, 96, 144]. We break down this concept into _stereotype_ and _bias &
preference_ according to the types of discriminatory outputs [88, 129].

Stereotypes focus on identifying entrenched societal preconceptions [20]
perpetuated within MLLMs. We first carefully collect images of people at risk
of discrimination from diverse public sources to assess the stereotypes
contained in models’ generation (Task _F.1_). To examine the understanding and
sensitivity of MLLMs to stereotypes in practical scenarios, we synthesize
images relevant to the textual contexts with text-based tasks [99, 101, 152],
ranging from agreement on stereotypical statements (Task _F.2_),
classification of stereotypes (Task _F.3_) to stereotypical user queries (Task
_F.4_).

Bias & Preference examines the models’ tendency that either disadvantages
specific user groups or favors biased outcomes. We transform the text-based
preference choices from existing studies [129] into options represented by
images to evaluate the visual preferences embedded in MLLMs (Task _F.5_).
Then, we ask for judgment of job competency with images of people to quantify
the biases in MLLMs towards different personal attributes (e.g., gender, age)
with a Chi-square test [132], based on the annotations in [5] (Task _F.6_). We
finally examine if MLLMs are more inclined to express their preferences when
text-only questions for choices [129] are paired with various images (Task
_F.7_).

#####  2.2.5 Privacy

Privacy assesses the models’ capacity to protect personal data from
unauthorized requests [97]. It has been shown that large models are vulnerable
to data extraction [102] and prone to leak privacy in inference [127], which
is risky when deployed in privacy-sensitive applications. From the
perspectives of consciousness and behaviors [129], we evaluate privacy in
terms of _awareness_ and _leakage_.

Privacy Awareness requires the models to detect the existence of personal
information and privacy risks in their workflow [129]. Gradually increasing in
difficulties, we test MLLMs subsequently with identifying the presence of
private information in images [54, 106] (Task _P.1_) and deciding whether the
questions posed about these images involve the private information in there
(Task _P.2_), which requires reasoning beyond perception. The questions are
constructed with GPT-4V and manually labeled. Then, we pair the text-only task
of InfoFlow Expectation from [98] with different images and evaluate the
changes in models’ agreement on privacy usage (Task _P.3_).

Privacy Leakage evaluates the models’ protection of private information from
leakage in service [144]. Analogous to prompts involving privacy in red-
teaming LLMs [108], we collect a group of photos of celebrities and ask for
their personal identifiable information (PII) using these photos as visual
cues (Task _P.4_). Then, we request models to identify PII in images, which
are manually labeled from a public dataset [106] (Task _P.5_). Both are of
practical significance when models have access to private data of the public
and powerful OCR capability. We further query MLLMs for private information
contained in past text [144] and measure the leakage when paired with images
(Task _P.6_).

####  2.3 Evaluated Models

Merely compiling a leaderboard of the state-of-the-art MLLMs does not suffice
to address trustworthiness issues, as it provides few insights for future
improvements due to variations in architectures [30, 165] and training
paradigms [26, 130]. To tackle this, we strategically select models based on
several criteria. We first include 4 advanced proprietary models to highlight
trustworthiness gaps of open-source models. We then gather 6 models from the
rich LLaVA family [83] and 4 models based on MiniGPT-4 [178] and mPLUG-Owl
[164] to identify the impacts from diverse enhancements, like base LLMs [140],
improved datasets [26, 148] and reinforcement learning from human feedback
(RLHF) [130]. We also choose 7 popular MLLMs across various stages in MLLM
development to broaden the model coverage. This accumulates into a group of 21
MLLMs for thorough evaluation, as detailed in Tab. B.1. The benchmark will be
updated with more newly released models afterwards.

####  2.4 Toolbox

While existing benchmarks [19, 44, 49, 75] only provide datasets and
evaluation scripts, which lack scalability and adaptability, and greatly limit
the test of latest models and tasks, we deliberately develop a toolbox in
MultiTrust to provide a _universal and scalable_ infrastructure for evaluating
MLLM trustworthiness and facilitating future research. We integrate different
MLLMs by accommodating varying interaction formats from developers into a
unified interface, enabling standardized model evaluation. The tasks are
modularized by separating data, inference, and evaluation metrics to
facilitate tool reuse and easy updates for new tasks. This user-friendly
structure not only upholds rigorous evaluation standards but also lay a
groundwork for extension of community contributions.

Figure 2: Left: Rankings of MLLMs in each sub-aspect of MultiTrust. Right: (a)
Correlation between the overall rankings of trustworthiness and those of
general capabilities based on MMBench [90] and MME [44]. Top-8 are marked. (b)
Pearson Correlation Coefficients between sub-aspects.

###  3 Analysis on Experimental Results

We conduct extensive experiments on the 32 carefully curated tasks to fulfill
the benchmark. In this section, we present the rankings in Fig. 2 based on the
evaluation and analyze the representative experimental results under each
aspect to convey our most significant discoveries within the limited space.
Complete results and analyses are provided in appendices from Appendix C to
Appendix G.

Overall Performance. From Fig. 2, we draw a quick conclusion that proprietary
models like GPT-4V and Claude3 demonstrate consistently top performance, which
can be attributed to the efforts on alignment, safety filters, etc. From a
global perspective, there is a correlation coefficient of 0.60 between the
general capabilities and the trustworthiness of various MLLMs. As shown in
Fig. 2(a), the previous generation of MLLMs often fall short in various
aspects of trustworthiness due to their inferior multimodal perception and
reasoning capabilities, while the powerful abilities lead to better
trustworthiness to various extents. Meanwhile, the finer correlation analysis
in Fig. 2(b) shows that besides several sub-aspects like truthfulness,
toxicity, and privacy awareness, which require similar capabilities like
recognition, there is no significant correlation across different aspects,
emphasizing the necessity of comprehensive aspect division and the gaps in
trustworthiness from perfection.

Table 3: Performance of inherent capabilities in Task _T.1_/_2_ with accuracy
(%, $\uparrow$).

Task | Subtask | Gemini-Pro | InternLM-XC2 | InternVL-Chat
---|---|---|---|---
Basic (_T.1_) | Object | 80.80 | 93.20 | 88.80
Scene | 70.00 | 88.25 | 86.25
Grounding | 8.00 | 32.00 | 42.00
Advanced (_T.2_) | Commonsense | 79.29 | 73.57 | 65.71
Comparison | 54.00 | 64.00 | 55.00
Temporal | 52.50 | 47.50 | 52.50

Truthfulness. In the aspect of truthfulness, inherent deficiencies in MLLMs’
capabilities are commonly observed. Although MLLMs perform commendably on
general perception tasks like object existence judgment and scene analysis,
where most achieve an accuracy of over 80% as in Tab. 3, they show a notable
decline when shifting to more fine-grained tasks such as visual grounding
(e.g., 32% for InternLM-XC2 and even 8% for Gemini-Pro), highlighting MLLMs’
limitations in fine-grained perceptual capability [90]. Besides, disparities
also exist in MLLMs’ dependence on image and text modalities for further
cognitive inference. For instance, models generally perform better on
commonsense reasoning tasks that basically utilize knowledge learned by LLMs
but less satisfying when they need to exploit the visual modality. As shown in
Tab. 3, the latter two tasks have an obvious performance degradation compared
to commonsense reasoning. This is consistent with previous findings [25, 151].
Then, turning to external misleading factors, closed-source models generally
exhibit superior resistance to interference compared to the majority of open-
source models, which are highly susceptible to confusing or misleading images
and further cause misinformation.

Table 4: Performance in Task _S.1_/_2_/_4_.

Task | Metrics | MiniGPT-4-L2 | mPLUG-Owl2
---|---|---|---
NSFW Description | RtA (%, $\uparrow$) | 34.00 | 0.00
P.API ($\downarrow$) | 0.46 | 0.62
Risk Identification | Object (%, $\uparrow$) | 75.08 | 91.33
Risk (%, $\uparrow$) | 42.93 | 81.00
Typographic Jailbreaking | RtA (%, $\uparrow$) | 79.50 | 14.50
ASR (%, $\downarrow$) | 1.50 | 34.50

Safety. For safety tasks, open-source MLLMs generally have worse safety
awareness and guardrails compared to proprietary models. For instance, GPT-4V
and Claude3 refuse to describe 69.5% NSFW images on average, while most open-
source models reject none. The same happens for jailbreaking that these two
closed-source models refuse nearly all malicious queries, while many advanced
models are frequently attacked. It is noticeable that only by placing the
target harmful behaviors in images without any deliberately designed prompts,
many modern MLLMs are successfully jailbroken due to their attention on
images, e.g., 71% for InternLM-XC2 and 80% for LLaVA-1.5. Notably, with the
same base LLM of Llama-2 [140], MiniGPT-4-L2 and mPLUG-Owl2 show reverse
performance on typographic jailbreaking and NSFW description compared to risk
identification in multimodal contexts, as shown in Tab. 4. This indicates that
while the capabilities of perception and understanding are enhanced through
better multimodal training, the safety mechanisms originating from the well-
aligned base LLM can be catastrophically compromised.

Figure 3: ASRs (%, $\downarrow$) in Task _R.4_/_5_.

Robustness. While the results on OOD scenarios show non-drastic differences
across models, we prove that MLLMs inevitably inherit the adversarial
vulnerability from deep neural networks. In the task of image captioning on
simple objects, the accuracy of most models drops below 20% from more than 90%
under untargeted attack. As for targeted attack, more than half output the
desired objects with ratios above 50%, even for the commercial Qwen-VL-Plus,
highlighting their fragility. As shown in Fig. 3, while two advanced open-
source MLLMs have the highest attack success rates, the other two models,
CogVLM and InternVL-Chat, exhibit significantly better robustness, which have
unique visual encoders with more parameters, suppressing the transferability
of adversarial examples. The same reason may also apply to the other two
closed-source ones, which may also be equipped with input filters for
purifying noises [103].

Figure 4: RtA rate (%, $\uparrow$) in Task _F.5_.

Fairness. In terms of stereotypes, most MLLMs demonstrate a heightened
sensitivity to stereotypical user queries in real-life scenarios, maintaining
an average RtA rate of 93.79% even under the influence of relevant images.
However, when stereotypes shift from application-scenario queries to opinion-
based evaluations, the performance disparities in MLLMs not only become
apparent between proprietary and open-source models but also manifest in
stereotype themes. For example, stereotypes related to age, show significantly
higher agreement rates in MLLMs compared to other themes like gender, race,
religion, etc. This disparity in topics also exists in the evaluation of bias
and preference. As depicted in Fig. 4, the latest models, such as Claude3,
InternLM-XC2, and LLaVA-NeXT, display varying sensitivities. These models are
more sensitive to ideological topics and more lenient toward cultural or
lifestyle questions, easier to reveal their preferences and thereby influence
user decisions.

Table 5: RtA rate (%, $\uparrow$) of two MLLMs in Task _P.6_.

Model |  Only Text |  With Irrelevant |  With Relevant
---|---|---|---
LLaVA-NeXT | 51.00 | 25.67 | 1.50
MiniGPT-4-L2 | 100.00 | 66.78 | 24.00

Privacy. We show that most models possess the basic concept of privacy with an
average accuracy of 72.30% in deciding the presence of private information in
images, which is correlated to the general perception capability. However,
this awareness is severely challenged in scenarios demanding more complex
reasoning and the average performance drops notably to 55.33%, widening the
gap between closed-source and open-source models, with only GPT-4V and Gemini-
Pro achieving accuracy above 70%. For privacy leakage, we notice that by
activating the instinct for OCR, privacy extraction from images is more likely
to compromise the data protection protocols than taking images as cues for PII
query, amplifying the privacy risks of MLLMs in multimodal scenarios. The two
text-only tasks reveal a similar phenomenon that multimodality affects the LLM
behaviors of instruction following and privacy protection. As exemplified in
Tab. 5, models tend to leak PII information when paired with images, which
poses more threats when MLLMs have access to personal data, while the frozen
Llama2 can enhance the privacy protection.

###  4 Discussion

Key Findings. Overall speaking, the extensive experiments reveal that: (1)
though advanced open-source MLLMs have achieved comparable or even superior
performance in general perception and reasoning tasks compared to proprietary
models represented by GPT-4V [105] and Claude3 [8], they still carry
significant vulnerabilities and issues in terms of trustworthiness, while the
proprietary ones are more reliable, demonstrating high sensitivity to risks
and low maliciousness in responses; (2) MLLMs possess basic understandings of
the concepts in trustworthiness by directly asking them to recognize risks,
but their awareness of threats would deteriorate when the risks are more
concealed and the scenarios demand multi-step complex reasoning; (3)
Overemphasis on accomplishing general multimodal tasks such as OCR during
multimodal training can distract models from the instructions in text and make
them overlook the potential risks, failing to reject improper requests, which
render the alignment inadequate for consolidating trustworthiness.

The evaluation strategy covering multimodal risks and cross-modal impacts
draws to the conclusion that the multimodality affects the base LLMs of MLLMs
in various ways, subsequently increasing the risks in their applications. This
is reflected with phenomena that: (4) multimodal training with LLM
significantly compromises its previous trustworthy alignment, confirmed by the
comparison between MiniGPT-4-Llama2 [178] and mPLUG-Owl2 [165] which both use
well-aligned Llama2 [140] for base LLM and show more vulnerabilities when the
LLM is fine-tuned; (5) when paired with irrelevant images during multimodal
inference, the performance on text-oriented tasks becomes unstable with either
non-directional fluctuations or directional tendencies; (6) relevant visual
contexts can more directly influence the model performance, sometimes
beneficial for completing tasks in truthfulness, but more often leading to
unexpected behaviors and exacerbating their internal risks.

In light of these risks, it becomes necessary and urgent to find ways to
mitigate these threats and facilitate the trustworthiness of MLLMs. With
careful model selection and thorough analysis, we identify that: (7) improved
architectures (e.g., novel vision encoder [30], aligned base LLM [140]) and
training paradigms (e.g., SFT with data from GPT-4V [148], RLHF [130]) can
positively influence the trustworthiness of models to some extents; (8) these
advancements are not sufficient to bring all-round improvements, e.g., the
robustness to transfer attack from unique visual encoders can be nullified
when they are included into surrogate models, SFT and RLHF heavily rely on the
quality and representativeness of the training data distribution, which could
introduce bias on certain aspects. Therefore, to achieve truly trustworthy
MLLMs, it is imperative to develop more effective approaches. Here, we offer
our views on the potential directions for solution.

$\bullet$ _Drawing lessons from LLMs._ In the realm of LLMs, many methods have
been proposed to achieve better performance and alignment, including but not
limited to retrieval augmented generation (RAG) [45, 68], RLAIF with
constitutions (Constitutional AI) [15], and weak-to-strong generalization
[18]. These techniques can be extended to MLLMs with the same basic logic of
functioning.
$\bullet$ _Expanding the focus in multimodal training._ Previous work mostly
emphasizes to enhance the multimodal capabilities via multimodal training [37,
84, 149, 165]. As pointed out in this work, more issues, like the stability in
multimodal inference, the preservation of the inherent alignment in base LLMs,
and better multimodality alignment, should be prioritized.
$\bullet$ _Evaluation and evolution in dynamic environments._ It is
unrealistic to obtain a highly trustworthy model in one go, as threats in
practice cannot be fully covered during training. With the concept of agents
[136, 175], it is possible to consolidate MLLMs in dynamic environments [109],
where more challenging assessment, e.g., interactions between agents [52],
adaptive attacks [141], can take place.

Social Impacts. MultiTrust’s scrutiny of MLLM trustworthiness unveils profound
societal risks. Truthfulness issues like hallucination could propagate
dangerous misinformation, notably in applications like healthcare [158, 160].
Vulnerabilities to toxic content and jailbreaking expose the public to hazards
and illegal AI exploitation [104, 123]. MLLMs’ susceptibilities to adversarial
conditions threaten critical services like security monitoring[38]. Embedded
biases could perpetuate social inequalities in decision-making within
employment and law enforcement [42]. Data breaches and inadequate safeguards
of privacy also violate personal rights and erode trust in AI [118]. These
insights necessitate a call for the judicious use of MLLMs to prevent their
potential adverse impacts.

Limitations. (1) Absence of machine ethics: Unlike DecodingTrust [144] and
TrustLLM [129], machine ethics is excluded from MultiTrust evaluations due to
its cultural and subjective variability, which can be further appended into
the framework. (2) Limited analysis of closed-source models: The restricted
knowledge about the pretraining data, architectures, and training strategies
of proprietary models limits our in-depth analysis to attribute their success
and failure technically. (3) Malicious misuse of the dataset: Certain parts of
our dataset can potentially be exploited maliciously, causing social impacts.
We will clarify the risks and restrict the management for the released
dataset.

###  5 Conclusion

In this work, we establish MultiTrust, the first comprehensive benchmark
specifically designed to evaluate the trustworthiness of Multimodal Large
Language Models (MLLMs). Through extensive experiments across 32 tasks under
10 detailed sub-aspects on 21 advanced MLLMs, we identify significant gaps and
unexplored risks in their trustworthiness, including the susceptibilities in
novel multimodal scenarios and instabilities caused by the cross-modal
interaction. Our findings highlight the vulnerabilities and complexities posed
by the multimodal nature of these models, emphasizing the necessity for more
in-depth researches and sophisticated approaches to ensure their
trustworthiness.

### References

  * [1] Nsfw data scraper.  https://github.com/alex000kim/nsfw_data_scraper.
  * [2] LAMM: Language-Assisted Multi-Modal Instruction-Tuning Dataset, Framework, and Benchmark, volume 36, 2023.
  * [3] Pause giant ai experiments: an open letter.  Future of Life Institute.– URL: futureoflife. org/open-letter/pause-giant-ai-experiments.(Accessed 11.04. 2023), 2023.
  * [4] Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al.  Gpt-4 technical report.  arXiv preprint arXiv:2303.08774, 2023.
  * [5] Junaid Ali, Matthäus Kleindessner, Florian Wenzel, Kailash Budhathoki, Volkan Cevher, and Chris Russell.  Evaluating the fairness of discriminative foundation models in computer vision.  In Proceedings of the 2023 AAAI/ACM Conference on AI, Ethics, and Society, AIES ’23, page 809–833, New York, NY, USA, 2023. Association for Computing Machinery.
  * [6] John R Anderson.  The architecture of cognition.  Psychology Press, 2013.
  * [7] Rohan Anil, Andrew M Dai, Orhan Firat, Melvin Johnson, Dmitry Lepikhin, Alexandre Passos, Siamak Shakeri, Emanuel Taropa, Paige Bailey, Zhifeng Chen, et al.  Palm 2 technical report.  arXiv preprint arXiv:2305.10403, 2023.
  * [8] Anthropic.  The claude 3 model family: Opus, sonnet, haiku.  https://www-cdn.anthropic.com/de8ba9b01c9ab7cbabf5c33b80b7bbc618857627/Model_Card_Claude_3.pdf, 2024.
  * [9] Usman Anwar, Abulhair Saparov, Javier Rando, Daniel Paleka, Miles Turpin, Peter Hase, Ekdeep Singh Lubana, Erik Jenner, Stephen Casper, Oliver Sourbut, et al.  Foundational challenges in assuring alignment and safety of large language models.  arXiv preprint arXiv:2404.09932, 2024.
  * [10] Kyriakos D Apostolidis and George A Papakostas.  A survey on adversarial deep learning robustness in medical image analysis.  Electronics, 10(17):2132, 2021.
  * [11] Muhammad Ahsan Awais, Peter Redmond, Tomas Emmanuel Ward, and Graham Healy.  Amber: Advancing multimodal brain-computer interfaces for enhanced robustness-a dataset for naturalistic settings.  Frontiers in Neuroergonomics, 4:1216440, 2023.
  * [12] Amos Azaria and Tom Mitchell.  The internal state of an llm knows when it’s lying.  In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 967–976, 2023.
  * [13] B. Babenko, S. Belongie, and Kai Wang.  End-to-end scene text recognition.  In 2011 IEEE International Conference on Computer Vision (ICCV), pages 1457–1464, November 2011.
  * [14] Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou.  Qwen-vl: A frontier large vision-language model with versatile abilities.  arXiv preprint arXiv:2308.12966, 2023.
  * [15] Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, et al.  Constitutional ai: Harmlessness from ai feedback.  arXiv preprint arXiv:2212.08073, 2022.
  * [16] Zechen Bai, Pichao Wang, Tianjun Xiao, Tong He, Zongbo Han, Zheng Zhang, and Mike Zheng Shou.  Hallucination of multimodal large language models: A survey.  arXiv preprint arXiv:2404.18930, 2024.
  * [17] Rishabh Bhardwaj and Soujanya Poria.  Red-teaming large language models using chain of utterances for safety-alignment.  arXiv preprint arXiv:2308.09662, 2023.
  * [18] Collin Burns, Pavel Izmailov, Jan Hendrik Kirchner, Bowen Baker, Leo Gao, Leopold Aschenbrenner, Yining Chen, Adrien Ecoffet, Manas Joglekar, Jan Leike, et al.  Weak-to-strong generalization: Eliciting strong capabilities with weak supervision.  arXiv preprint arXiv:2312.09390, 2023.
  * [19] Rizhao Cai, Zirui Song, Dayan Guan, Zhenhao Chen, Xing Luo, Chenyu Yi, and Alex Kot.  Benchlmm: Benchmarking cross-style visual capability of large multimodal models.  arXiv preprint arXiv:2312.02896, 2023.
  * [20] Yang Trista Cao, Anna Sotnikova, Jieyu Zhao, Linda X Zou, Rachel Rudinger, and Hal Daume III.  Multilingual large language models leak human stereotypes across language boundaries.  arXiv preprint arXiv:2312.07141, 2023.
  * [21] Nicholas Carlini, Milad Nasr, Christopher A. Choquette-Choo, Matthew Jagielski, Irena Gao, Pang Wei W Koh, Daphne Ippolito, Florian Tramer, and Ludwig Schmidt.  Are aligned neural networks adversarially aligned?  In Advances in Neural Information Processing Systems, volume 36, pages 61478–61500, 2023.
  * [22] Patrick Chao, Alexander Robey, Edgar Dobriban, Hamed Hassani, George J Pappas, and Eric Wong.  Jailbreaking black box large language models in twenty queries.  In R0-FoMo Workshop on Robustness of Few-shot and Zero-shot Learning in Large Foundation Models in Advances in Neural Information Processing Systems, 2023.
  * [23] Dongping Chen, Ruoxi Chen, Shilin Zhang, Yinuo Liu, Yaochen Wang, Huichi Zhou, Qihui Zhang, Pan Zhou, Yao Wan, and Lichao Sun.  Mllm-as-a-judge: Assessing multimodal llm-as-a-judge with vision-language benchmark.  arXiv preprint arXiv:2402.04788, 2024.
  * [24] Huanran Chen, Yichi Zhang, Yinpeng Dong, Xiao Yang, Hang Su, and Jun Zhu.  Rethinking model ensemble in transfer-based adversarial attacks.  In The Twelfth International Conference on Learning Representations, 2024.
  * [25] Lin Chen, Jinsong Li, Xiaoyi Dong, Pan Zhang, Yuhang Zang, Zehui Chen, Haodong Duan, Jiaqi Wang, Yu Qiao, Dahua Lin, et al.  Are we on the right way for evaluating large vision-language models?  arXiv preprint arXiv:2403.20330, 2024.
  * [26] Lin Chen, Jisong Li, Xiaoyi Dong, Pan Zhang, Conghui He, Jiaqi Wang, Feng Zhao, and Dahua Lin.  Sharegpt4v: Improving large multi-modal models with better captions.  arXiv preprint arXiv:2311.12793, 2023.
  * [27] Shuo Chen, Jindong Gu, Zhen Han, Yunpu Ma, Philip Torr, and Volker Tresp.  Benchmarking robustness of adaptation methods on pre-trained vision-language models.  In Advances in Neural Information Processing Systems, volume 36, pages 51758–51777, 2023.
  * [28] Xinlei Chen, Hao Fang, Tsung-Yi Lin, Ramakrishna Vedantam, Saurabh Gupta, Piotr Dollár, and C Lawrence Zitnick.  Microsoft coco captions: Data collection and evaluation server.  arXiv preprint arXiv:1504.00325, 2015.
  * [29] Yang Chen, Ethan Mendes, Sauvik Das, Wei Xu, and Alan Ritter.  Can language models be instructed to protect personal information?  arXiv preprint arXiv:2310.02224, 2023.
  * [30] Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Zhong Muyan, Qinglong Zhang, Xizhou Zhu, Lewei Lu, et al.  Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks.  arXiv preprint arXiv:2312.14238, 2023.
  * [31] Junjie Chu, Zeyang Sha, Michael Backes, and Yang Zhang.  Conversation reconstruction attack against gpt models.  arXiv preprint arXiv:2402.02987, 2024.
  * [32] Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al.  Scaling instruction-finetuned language models.  Journal of Machine Learning Research, 25(70):1–53, 2024.
  * [33] R. Clarke.  Asimov’s laws of robotics: Implications for information technology. 2.  Computer, 27(1):57–66, 1994.
  * [34] Chenhang Cui, Yiyang Zhou, Xinyu Yang, Shirley Wu, Linjun Zhang, James Zou, and Huaxiu Yao.  Holistic analysis of hallucination in gpt-4v (ision): Bias and interference challenges.  arXiv preprint arXiv:2311.03287, 2023.
  * [35] Wenliang Dai, Junnan Li, Dongxu Li, Anthony Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale N Fung, and Steven Hoi.  Instructblip: Towards general-purpose vision-language models with instruction tuning.  In Advances in Neural Information Processing Systems, volume 36, pages 49250–49267, 2023.
  * [36] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.  Bert: Pre-training of deep bidirectional transformers for language understanding.  arXiv preprint arXiv:1810.04805, 2018.
  * [37] Xiaoyi Dong, Pan Zhang, Yuhang Zang, Yuhang Cao, Bin Wang, Linke Ouyang, Xilin Wei, Songyang Zhang, Haodong Duan, Maosong Cao, et al.  Internlm-xcomposer2: Mastering free-form text-image composition and comprehension in vision-language large model.  arXiv preprint arXiv:2401.16420, 2024.
  * [38] Yinpeng Dong, Huanran Chen, Jiawei Chen, Zhengwei Fang, Xiao Yang, Yichi Zhang, Yu Tian, Hang Su, and Jun Zhu.  How robust is google’s bard to adversarial image attacks?  In R0-FoMo Workshop on Robustness of Few-shot and Zero-shot Learning in Large Foundation Models in Advances in Neural Information Processing Systems, 2023.
  * [39] Yinpeng Dong, Qi-An Fu, Xiao Yang, Tianyu Pang, Hang Su, Zihao Xiao, and Jun Zhu.  Benchmarking adversarial robustness on image classification.  In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 321–331, June 2020.
  * [40] Yinpeng Dong, Fangzhou Liao, Tianyu Pang, Hang Su, Jun Zhu, Xiaolin Hu, and Jianguo Li.  Boosting adversarial attacks with momentum.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 9185–9193, 2018.
  * [41] Yinpeng Dong, Tianyu Pang, Hang Su, and Jun Zhu.  Evading defenses to transferable adversarial examples by translation-invariant attacks.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 4312–4321, June 2019.
  * [42] Emilio Ferrara.  Fairness and bias in artificial intelligence: A brief survey of sources, impacts, and mitigation strategies.  Sci, 6(1):3, 2023.
  * [43] Center for AI Safety.  Statement on ai risk.  https://www.safe.ai/work/statement-on-ai-risk, 2023.
  * [44] Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Jinrui Yang, Xiawu Zheng, Ke Li, Xing Sun, Yunsheng Wu, and Rongrong Ji.  Mme: A comprehensive evaluation benchmark for multimodal large language models.  arXiv preprint arXiv:2306.13394, 2023.
  * [45] Yunfan Gao, Yun Xiong, Xinyu Gao, Kangxiang Jia, Jinliu Pan, Yuxi Bi, Yi Dai, Jiawei Sun, and Haofen Wang.  Retrieval-augmented generation for large language models: A survey.  arXiv preprint arXiv:2312.10997, 2023.
  * [46] Howard E Gardner.  Frames of mind: The theory of multiple intelligences.  Basic books, 2011.
  * [47] Samuel Gehman, Suchin Gururangan, Maarten Sap, Yejin Choi, and Noah A Smith.  Realtoxicityprompts: Evaluating neural toxic degeneration in language models.  In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 3356–3369, 2020.
  * [48] Ben Goertzel.  Artificial general intelligence: concept, state of the art, and future prospects.  Journal of Artificial General Intelligence, 5(1):1–48, 2014.
  * [49] Yichen Gong, Delong Ran, Jinyuan Liu, Conglei Wang, Tianshuo Cong, Anyu Wang, Sisi Duan, and Xiaoyun Wang.  Figstep: Jailbreaking large vision-language models via typographic visual prompts.  arXiv preprint arXiv:2311.05608, 2023.
  * [50] Ian J Goodfellow, Jonathon Shlens, and Christian Szegedy.  Explaining and harnessing adversarial examples.  In The Third International Conference on Learning Representations, 2015.
  * [51] Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh.  Making the v in vqa matter: Elevating the role of image understanding in visual question answering.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), July 2017.
  * [52] Xiangming Gu, Xiaosen Zheng, Tianyu Pang, Chao Du, Qian Liu, Ye Wang, Jing Jiang, and Min Lin.  Agent smith: A single image can jailbreak one million multimodal llm agents exponentially fast.  arXiv preprint arXiv:2402.08567, 2024.
  * [53] Danna Gurari, Qing Li, Chi Lin, Yinan Zhao, Anhong Guo, Abigale Stangl, and Jeffrey P. Bigham.  Vizwiz-priv: A dataset for recognizing the presence and purpose of private visual information in images taken by blind people.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 939–948, June 2019.
  * [54] Danna Gurari, Qing Li, Abigale J. Stangl, Anhong Guo, Chi Lin, Kristen Grauman, Jiebo Luo, and Jeffrey P. Bigham.  Vizwiz grand challenge: Answering visual questions from blind people.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 3608–3617, June 2018.
  * [55] Eungyeom Ha, Heemook Kim, and Dongbin Na.  Hod: New harmful object detection benchmarks for robust surveillance.  In 2024 IEEE/CVF Winter Conference on Applications of Computer Vision Workshops (WACVW), pages 183–192, January 2024.
  * [56] Lei Huang, Weijiang Yu, Weitao Ma, Weihong Zhong, Zhangyin Feng, Haotian Wang, Qianglong Chen, Weihua Peng, Xiaocheng Feng, Bing Qin, et al.  A survey on hallucination in large language models: Principles, taxonomy, challenges, and open questions.  arXiv preprint arXiv:2311.05232, 2023.
  * [57] Wen Huang, Hongbin Liu, Minxin Guo, and Neil Zhenqiang Gong.  Visual hallucinations of multi-modal large language models.  arXiv preprint arXiv:2402.14683, 2024.
  * [58] Yao Huang, Yinpeng Dong, Shouwei Ruan, Xiao Yang, Hang Su, and Xingxing Wei.  Towards transferable targeted 3d adversarial attack in the physical world.  arXiv preprint arXiv:2312.09558, 2023.
  * [59] Gabriel Ilharco, Mitchell Wortsman, Ross Wightman, Cade Gordon, Nicholas Carlini, Rohan Taori, Achal Dave, Vaishaal Shankar, Hongseok Namkoong, John Miller, Hannaneh Hajishirzi, Ali Farhadi, and Ludwig Schmidt.  Openclip, July 2021.
  * [60] Mohit Iyyer, John Wieting, Kevin Gimpel, and Luke Zettlemoyer.  Adversarial example generation with syntactically controlled paraphrase networks.  In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 1875–1885, 2018.
  * [61] Jiaming Ji, Tianyi Qiu, Boyuan Chen, Borong Zhang, Hantao Lou, Kaile Wang, Yawen Duan, Zhonghao He, Jiayi Zhou, Zhaowei Zhang, et al.  Ai alignment: A comprehensive survey.  arXiv preprint arXiv:2310.19852, 2023.
  * [62] Ziwei Ji, Tiezheng Yu, Yan Xu, Nayeon Lee, Etsuko Ishii, and Pascale Fung.  Towards mitigating llm hallucination via self reflection.  In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 1827–1843, 2023.
  * [63] Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al.  Mistral 7b.  arXiv preprint arXiv:2310.06825, 2023.
  * [64] Di Jin, Zhijing Jin, Joey Tianyi Zhou, and Peter Szolovits.  Is bert really robust? a strong baseline for natural language attack on text classification and entailment.  In Proceedings of the AAAI conference on artificial intelligence, volume 34, pages 8018–8025, 2020.
  * [65] Wilhelm Johannsen.  Elemente der exakten Erblichkeitslehre.  Gustav Fischer, 1909.
  * [66] Laura A Kelley and Jennifer L Kelley.  Animal visual illusion and confusion: the importance of a perceptual perspective.  Behavioral Ecology, 25(3):450–463, 2014.
  * [67] Hannah Rose Kirk, Bertie Vidgen, Paul Röttger, and Scott A Hale.  The benefits, risks and bounds of personalizing the alignment of large language models to individuals.  Nature Machine Intelligence, pages 1–10, 2024.
  * [68] Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela.  Retrieval-augmented generation for knowledge-intensive nlp tasks.  In Advances in Neural Information Processing Systems, volume 33, pages 9459–9474, 2020.
  * [69] Bo Li, Yuanhan Zhang, Liangyu Chen, Jinghao Wang, Jingkang Yang, and Ziwei Liu.  Otter: A multi-modal model with in-context instruction tuning.  arXiv preprint arXiv:2305.03726, 2023.
  * [70] Bohao Li, Rui Wang, Guangzhi Wang, Yuying Ge, Yixiao Ge, and Ying Shan.  Seed-bench: Benchmarking multimodal llms with generative comprehension.  arXiv preprint arXiv:2307.16125, 2023.
  * [71] Jinfeng Li, Shouling Ji, Tianyu Du, Bo Li, and Ting Wang.  Textbugger: Generating adversarial text against real-world applications.  arXiv preprint arXiv:1812.05271, 2018.
  * [72] Junyi Li, Xiaoxue Cheng, Wayne Xin Zhao, Jian-Yun Nie, and Ji-Rong Wen.  Halueval: A large-scale hallucination evaluation benchmark for large language models.  In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6449–6464, 2023.
  * [73] Kenneth Li, Oam Patel, Fernanda Viégas, Hanspeter Pfister, and Martin Wattenberg.  Inference-time intervention: Eliciting truthful answers from a language model.  In Advances in Neural Information Processing Systems, volume 36, pages 41451–41530, 2023.
  * [74] Linyang Li, Ruotian Ma, Qipeng Guo, Xiangyang Xue, and Xipeng Qiu.  Bert-attack: Adversarial attack against bert using bert.  In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6193–6202, 2020.
  * [75] Mukai Li, Lei Li, Yuwei Yin, Masood Ahmed, Zhenguang Liu, and Qi Liu.  Red teaming visual language models.  arXiv preprint arXiv:2401.12915, 2024.
  * [76] Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Wayne Xin Zhao, and Ji-Rong Wen.  Evaluating object hallucination in large vision-language models.  In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 292–305, 2023.
  * [77] Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michihiro Yasunaga, Yian Zhang, Deepak Narayanan, Yuhuai Wu, Ananya Kumar, et al.  Holistic evaluation of language models.  Transactions on Machine Learning Research, 2023.
  * [78] Hongzhan Lin, Ziyang Luo, Bo Wang, Ruichao Yang, and Jing Ma.  Goat-bench: Safety insights to large multimodal models through meme-based social abuse.  arXiv preprint arXiv:2401.01523, 2024.
  * [79] Stephanie Lin, Jacob Hilton, and Owain Evans.  Truthfulqa: Measuring how models mimic human falsehoods.  In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 3214–3252, 2022.
  * [80] Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick.  Microsoft coco: Common objects in context.  In European Conference on Computer Vision, pages 740–755. Springer, 2014.
  * [81] Fuxiao Liu, Tianrui Guan, Zongxia Li, Lichang Chen, Yaser Yacoob, Dinesh Manocha, and Tianyi Zhou.  Hallusionbench: You see what you think? or you think what you see? an image-context reasoning benchmark challenging for gpt-4v (ision), llava-1.5, and other multi-modality models.  arXiv preprint arXiv:2310.14566, 2023.
  * [82] Hanchao Liu, Wenyuan Xue, Yifei Chen, Dapeng Chen, Xiutian Zhao, Ke Wang, Liping Hou, Rongjun Li, and Wei Peng.  A survey on hallucination in large vision-language models.  arXiv preprint arXiv:2402.00253, 2024.
  * [83] Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee.  Improved baselines with visual instruction tuning.  In Workshop on Instruction Tuning and Instruction Following in Advances in Neural Information Processing Systems, 2023.
  * [84] Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee.  Llava-next: Improved reasoning, ocr, and world knowledge (january 2024).  URL https://llava-vl. github. io/blog/2024-01-30-llava-next, 2024.
  * [85] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee.  Visual instruction tuning.  In Advances in Neural Information Processing Systems, volume 36, pages 34892–34916, 2023.
  * [86] Xiaogeng Liu, Nan Xu, Muhao Chen, and Chaowei Xiao.  AutoDAN: Generating stealthy jailbreak prompts on aligned large language models.  In The Twelfth International Conference on Learning Representations, 2024.
  * [87] Xin Liu, Yichen Zhu, Yunshi Lan, Chao Yang, and Yu Qiao.  Query-relevant images jailbreak large multi-modal models.  arXiv preprint arXiv:2311.17600, 2023.
  * [88] Yang Liu, Yuanshun Yao, Jean-Francois Ton, Xiaoying Zhang, Ruocheng Guo Hao Cheng, Yegor Klochkov, Muhammad Faaiz Taufiq, and Hang Li.  Trustworthy llms: a survey and guideline for evaluating large language models’ alignment.  arXiv preprint arXiv:2308.05374, 2023.
  * [89] Yi Liu, Gelei Deng, Zhengzi Xu, Yuekang Li, Yaowen Zheng, Ying Zhang, Lida Zhao, Tianwei Zhang, and Yang Liu.  Jailbreaking chatgpt via prompt engineering: An empirical study.  arXiv preprint arXiv:2305.13860, 2023.
  * [90] Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, et al.  Mmbench: Is your multi-modal model an all-around player?  arXiv preprint arXiv:2307.06281, 2023.
  * [91] Jiaying Lu, Jinmeng Rao, Kezhen Chen, Xiaoyuan Guo, Yawen Zhang, Baochen Sun, Carl Yang, and Jie Yang.  Evaluation and mitigation of agnosia in multimodal large language models.  arXiv preprint arXiv:2309.04041, 2023.
  * [92] Gen Luo, Yiyi Zhou, Tianhe Ren, Shengxin Chen, Xiaoshuai Sun, and Rongrong Ji.  Cheap and quick: Efficient vision-language instruction tuning for large language models.  In Advances in Neural Information Processing Systems, pages 29615–29627, 2023.
  * [93] Weidi Luo, Siyuan Ma, Xiaogeng Liu, Xiaoyu Guo, and Chaowei Xiao.  Jailbreakv-28k: A benchmark for assessing the robustness of multimodal large language models against jailbreak attacks.  arXiv preprint arXiv:2404.03027, 2024.
  * [94] Xiaofeng Mao, Yuefeng Chen, Yao Zhu, Da Chen, Hang Su, Rong Zhang, and Hui Xue.  Coco-o: A benchmark for object detectors under natural distribution shifts.  In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 6339–6350, October 2023.
  * [95] Mantas Mazeika, Long Phan, Xuwang Yin, Andy Zou, Zifan Wang, Norman Mu, Elham Sakhaee, Nathaniel Li, Steven Basart, Bo Li, et al.  Harmbench: A standardized evaluation framework for automated red teaming and robust refusal.  arXiv preprint arXiv:2402.04249, 2024.
  * [96] Brandon McKinzie, Zhe Gan, Jean-Philippe Fauconnier, Sam Dodge, Bowen Zhang, Philipp Dufter, Dhruti Shah, Xianzhi Du, Futang Peng, Floris Weers, et al.  Mm1: Methods, analysis & insights from multimodal llm pre-training.  arXiv preprint arXiv:2403.09611, 2024.
  * [97] Fatemehsadat Mireshghallah, Mohammadkazem Taram, Praneeth Vepakomma, Abhishek Singh, Ramesh Raskar, and Hadi Esmaeilzadeh.  Privacy in deep learning: A survey.  arXiv preprint arXiv:2004.12254, 2020.
  * [98] Niloofar Mireshghallah, Hyunwoo Kim, Xuhui Zhou, Yulia Tsvetkov, Maarten Sap, Reza Shokri, and Yejin Choi.  Can llms keep a secret? testing privacy implications of language models via contextual integrity theory.  arXiv preprint arXiv:2310.17884, 2023.
  * [99] Moin Nadeem, Anna Bethke, and Siva Reddy.  StereoSet: Measuring stereotypical bias in pretrained language models.  In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 5356–5371, 2021.
  * [100] Aakanksha Naik, Abhilasha Ravichander, Norman Sadeh, Carolyn Rose, and Graham Neubig.  Stress test evaluation for natural language inference.  In Proceedings of the 27th International Conference on Computational Linguistics, pages 2340–2353, 2018.
  * [101] Nikita Nangia, Clara Vania, Rasika Bhalerao, and Samuel Bowman.  Crows-pairs: A challenge dataset for measuring social biases in masked language models.  In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1953–1967, 2020.
  * [102] Milad Nasr, Nicholas Carlini, Jonathan Hayase, Matthew Jagielski, A Feder Cooper, Daphne Ippolito, Christopher A Choquette-Choo, Eric Wallace, Florian Tramèr, and Katherine Lee.  Scalable extraction of training data from (production) language models.  arXiv preprint arXiv:2311.17035, 2023.
  * [103] Weili Nie, Brandon Guo, Yujia Huang, Chaowei Xiao, Arash Vahdat, and Animashree Anandkumar.  Diffusion models for adversarial purification.  In Proceedings of the 39th International Conference on Machine Learning, volume 162, pages 16805–16827. PMLR, July 2022.
  * [104] Zhenxing Niu, Haodong Ren, Xinbo Gao, Gang Hua, and Rong Jin.  Jailbreaking attack against multimodal large language model.  arXiv preprint arXiv:2402.02309, 2024.
  * [105] OpenAI.  Gpt-4v(ision) system card.  https://cdn.openai.com/papers/GPTV_System_Card.pdf, 2023.
  * [106] Tribhuvanesh Orekondy, Bernt Schiele, and Mario Fritz.  Towards a visual privacy advisor: Understanding and predicting privacy risks in images.  In Proceedings of the IEEE International Conference on Computer Vision (ICCV), October 2017.
  * [107] Seyed Ali Osia, Ali Taheri, Ali Shahin Shamsabadi, Kleomenis Katevas, Hamed Haddadi, and Hamid R Rabiee.  Deep private-feature extraction.  IEEE Transactions on Knowledge and Data Engineering, 32(1):54–66, 2018.
  * [108] Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul F Christiano, Jan Leike, and Ryan Lowe.  Training language models to follow instructions with human feedback.  In Advances in Neural Information Processing Systems, volume 35, pages 27730–27744, 2022.
  * [109] Joon Sung Park, Joseph C. O’Brien, Carrie J. Cai, Meredith Ringel Morris, Percy Liang, and Michael S. Bernstein.  Generative agents: Interactive simulacra of human behavior.  In In the 36th Annual ACM Symposium on User Interface Software and Technology (UIST ’23), UIST ’23, New York, NY, USA, 2023. Association for Computing Machinery.
  * [110] Charles Sanders Peirce.  Reasoning and the logic of things: The Cambridge conferences lectures of 1898.  Harvard University Press, 1992.
  * [111] Xiangyu Qi, Kaixuan Huang, Ashwinee Panda, Peter Henderson, Mengdi Wang, and Prateek Mittal.  Visual adversarial examples jailbreak aligned large language models.  In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 21527–21536, 2024.
  * [112] Yusu Qian, Haotian Zhang, Yinfei Yang, and Zhe Gan.  How easy is it to fool your multimodal llms? an empirical analysis on deceptive prompts.  arXiv preprint arXiv:2402.13220, 2024.
  * [113] Huachuan Qiu, Shuai Zhang, Anqi Li, Hongliang He, and Zhenzhong Lan.  Latent jailbreak: A benchmark for evaluating text safety and output robustness of large language models.  arXiv preprint arXiv:2307.08487, 2023.
  * [114] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever.  Learning transferable visual models from natural language supervision.  In Proceedings of the 38th International Conference on Machine Learning, volume 139, pages 8748–8763. PMLR, 2021.
  * [115] Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn.  Direct preference optimization: Your language model is secretly a reward model.  In Advances in Neural Information Processing Systems, volume 36, pages 53728–53741, 2023.
  * [116] Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang.  Squad: 100,000+ questions for machine comprehension of text.  In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 2383–2392, 2016.
  * [117] Vipula Rawte, Amit Sheth, and Amitava Das.  A survey of hallucination in large foundation models.  arXiv preprint arXiv:2309.05922, 2023.
  * [118] Rajan Rayhan and Shahana Rayhan.  Ai and human rights: balancing innovation and privacy in the digital age.  DOI: 10.13140/RG. 2.2, 35394, 2023.
  * [119] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer.  High-resolution image synthesis with latent diffusion models.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 10684–10695, June 2022.
  * [120] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al.  Imagenet large scale visual recognition challenge.  International journal of computer vision, 115:211–252, 2015.
  * [121] Tianhao Shen, Renren Jin, Yufei Huang, Chuang Liu, Weilong Dong, Zishan Guo, Xinwei Wu, Yan Liu, and Deyi Xiong.  Large language model alignment: A survey.  arXiv preprint arXiv:2309.15025, 2023.
  * [122] Xinyue Shen, Zeyuan Chen, Michael Backes, Yun Shen, and Yang Zhang.  "do anything now": Characterizing and evaluating in-the-wild jailbreak prompts on large language models.  arXiv preprint arXiv:2308.03825, 2023.
  * [123] Zhelun Shi, Zhipin Wang, Hongxing Fan, Zaibin Zhang, Lijun Li, Yongting Zhang, Zhenfei Yin, Lu Sheng, Yu Qiao, and Jing Shao.  Assessment of multimodal large language models in alignment with human values.  arXiv preprint arXiv:2403.17830, 2024.
  * [124] Douglas Sloane and S Philip Morgan.  An introduction to categorical data analysis.  Annual review of sociology, 22(1):351–375, 1996.
  * [125] Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D Manning, Andrew Y Ng, and Christopher Potts.  Recursive deep models for semantic compositionality over a sentiment treebank.  In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1631–1642, 2013.
  * [126] Ziying Song, Lin Liu, Feiyang Jia, Yadan Luo, Guoxin Zhang, Lei Yang, Li Wang, and Caiyan Jia.  Robustness-aware 3d object detection in autonomous driving: A review and outlook.  arXiv preprint arXiv:2401.06542, 2024.
  * [127] Robin Staab, Mark Vero, Mislav Balunovic, and Martin Vechev.  Beyond memorization: Violating privacy via inference with large language models.  In The Twelfth International Conference on Learning Representations, 2024.
  * [128] Hao Sun, Zhexin Zhang, Jiawen Deng, Jiale Cheng, and Minlie Huang.  Safety assessment of chinese large language models.  arXiv preprint arXiv:2304.10436, 2023.
  * [129] Lichao Sun, Yue Huang, Haoran Wang, Siyuan Wu, Qihui Zhang, Chujie Gao, Yixin Huang, Wenhan Lyu, Yixuan Zhang, Xiner Li, et al.  Trustllm: Trustworthiness in large language models.  arXiv preprint arXiv:2401.05561, 2024.
  * [130] Zhiqing Sun, Sheng Shen, Shengcao Cao, Haotian Liu, Chunyuan Li, Yikang Shen, Chuang Gan, Liang-Yan Gui, Yu-Xiong Wang, Yiming Yang, et al.  Aligning large multimodal models with factually augmented rlhf.  arXiv preprint arXiv:2309.14525, 2023.
  * [131] Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus.  Intriguing properties of neural networks.  In The Second International Conference on Learning Representations, 2014.
  * [132] Ronald J. Tallarida and Rodney B. Murray.  Chi-Square Test, pages 140–142.  Springer New York, New York, NY, 1987.
  * [133] Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B Hashimoto.  Stanford alpaca: An instruction-following llama model, 2023.
  * [134] Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, et al.  Gemini: a family of highly capable multimodal models.  arXiv preprint arXiv:2312.11805, 2023.
  * [135] InternLM Team.  Internlm: A multilingual language model with progressively enhanced capabilities, 2023.
  * [136] Yu Tian, Xiao Yang, Jingyuan Zhang, Yinpeng Dong, and Hang Su.  Evil geniuses: Delving into the safety of llm-based agents.  arXiv preprint arXiv:2311.11855, 2023.
  * [137] Shengbang Tong, Zhuang Liu, Yuexiang Zhai, Yi Ma, Yann LeCun, and Saining Xie.  Eyes wide shut? exploring the visual shortcomings of multimodal llms, 2024.
  * [138] SM Tonmoy, SM Zaman, Vinija Jain, Anku Rani, Vipula Rawte, Aman Chadha, and Amitava Das.  A comprehensive survey of hallucination mitigation techniques in large language models.  arXiv preprint arXiv:2401.01313, 2024.
  * [139] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al.  Llama: Open and efficient foundation language models.  arXiv preprint arXiv:2302.13971, 2023.
  * [140] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al.  Llama 2: Open foundation and fine-tuned chat models.  arXiv preprint arXiv:2307.09288, 2023.
  * [141] Florian Tramer, Nicholas Carlini, Wieland Brendel, and Aleksander Madry.  On adaptive attacks to adversarial example defenses.  In Advances in Neural Information Processing Systems, volume 33, pages 1633–1645, 2020.
  * [142] Haoqin Tu, Chenhang Cui, Zijun Wang, Yiyang Zhou, Bingchen Zhao, Junlin Han, Wangchunshu Zhou, Huaxiu Yao, and Cihang Xie.  How many unicorns are in this image? a safety evaluation benchmark for vision llms.  arXiv preprint arXiv:2311.16101, 2023.
  * [143] Douglas N Walton.  What is reasoning? what is an argument?  The journal of Philosophy, 87(8):399–419, 1990.
  * [144] Boxin Wang, Weixin Chen, Hengzhi Pei, Chulin Xie, Mintong Kang, Chenhui Zhang, Chejian Xu, Zidi Xiong, Ritik Dutta, Rylan Schaeffer, Sang Truong, Simran Arora, Mantas Mazeika, Dan Hendrycks, Zinan Lin, Yu Cheng, Sanmi Koyejo, Dawn Song, and Bo Li.  Decodingtrust: A comprehensive assessment of trustworthiness in gpt models.  In Advances in Neural Information Processing Systems, volume 36, pages 31232–31339, 2023.
  * [145] Boxin Wang, Hengzhi Pei, Boyuan Pan, Qian Chen, Shuohang Wang, and Bo Li.  T3: Tree-autoencoder constrained adversarial text generation for targeted attack.  In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6134–6150, 2020.
  * [146] Boxin Wang, Chejian Xu, Shuohang Wang, Shuohang Wang, Zhe Gan, Yu Cheng, Jianfeng Gao, Ahmed Awadallah, and Bo Li.  Adversarial glue: A multi-task benchmark for robustness evaluation of language models.  In Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks, volume 1, 2021.
  * [147] Jindong Wang, Cuiling Lan, Chang Liu, Yidong Ouyang, Tao Qin, Wang Lu, Yiqiang Chen, Wenjun Zeng, and S Yu Philip.  Generalizing to unseen domains: A survey on domain generalization.  IEEE transactions on knowledge and data engineering, 35(8):8052–8072, 2022.
  * [148] Junke Wang, Lingchen Meng, Zejia Weng, Bo He, Zuxuan Wu, and Yu-Gang Jiang.  To see is to believe: Prompting gpt-4v for better visual instruction tuning.  arXiv preprint arXiv:2311.07574, 2023.
  * [149] Weihan Wang, Qingsong Lv, Wenmeng Yu, Wenyi Hong, Ji Qi, Yan Wang, Junhui Ji, Zhuoyi Yang, Lei Zhao, Xixuan Song, et al.  Cogvlm: Visual expert for pretrained language models.  arXiv preprint arXiv:2311.03079, 2023.
  * [150] Xinpeng Wang, Xiaoyuan Yi, Han Jiang, Shanlin Zhou, Zhihua Wei, and Xing Xie.  Tovilag: Your visual-language generative model is also an evildoer.  In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 3508–3533, 2023.
  * [151] Yiqi Wang, Wentao Chen, Xiaotian Han, Xudong Lin, Haiteng Zhao, Yongfei Liu, Bohan Zhai, Jianbo Yuan, Quanzeng You, and Hongxia Yang.  Exploring the reasoning abilities of multimodal large language models (mllms): A comprehensive survey on emerging trends in multimodal reasoning.  arXiv preprint arXiv:2401.06805, 2024.
  * [152] Yuxia Wang, Haonan Li, Xudong Han, Preslav Nakov, and Timothy Baldwin.  Do-not-answer: A dataset for evaluating safeguards in llms.  arXiv preprint arXiv:2308.13387, 2023.
  * [153] Alexander Wei, Nika Haghtalab, and Jacob Steinhardt.  Jailbroken: How does llm safety training fail?  In Advances in Neural Information Processing Systems, volume 36, pages 80079–80110, 2023.
  * [154] Xingxing Wei, Yao Huang, Yitong Sun, and Jie Yu.  Unified adversarial patch for cross-modal attacks in the physical world.  In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 4445–4454, October 2023.
  * [155] Xingxing Wei, Jie Yu, and Yao Huang.  Physically adversarial infrared patches with learnable shapes and locations.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 12334–12342, June 2023.
  * [156] Adina Williams, Nikita Nangia, and Samuel R Bowman.  A broad-coverage challenge corpus for sentence understanding through inference.  arXiv preprint arXiv:1704.05426, 2017.
  * [157] Donghyeon Won, Zachary C Steinert-Threlkeld, and Jungseock Joo.  Protest activity detection and perceived violence estimation from social media images.  In Proceedings of the 25th ACM international conference on Multimedia, pages 786–794, 2017.
  * [158] Jinge Wu, Yunsoo Kim, and Honghan Wu.  Hallucination benchmark in medical visual question answering.  arXiv preprint arXiv:2401.05827, 2024.
  * [159] Qiong Wu, Wei Yu, Yiyi Zhou, Shubin Huang, Xiaoshuai Sun, and Rongrong Ji.  Parameter and computation efficient transfer learning for vision-language pre-trained models.  In Advances in Neural Information Processing Systems, pages 41034–41050, 2023.
  * [160] Hanguang Xiao, Feizhong Zhou, Xingyue Liu, Tianqi Liu, Zhipeng Li, Xin Liu, and Xiaoxuan Huang.  A comprehensive survey of large language models and multimodal large language models in medicine.  arXiv preprint arXiv:2405.08603, 2024.
  * [161] Peng Xu, Wenqi Shao, Kaipeng Zhang, Peng Gao, Shuo Liu, Meng Lei, Fanqing Meng, Siyuan Huang, Yu Qiao, and Ping Luo.  Lvlm-ehub: A comprehensive evaluation benchmark for large vision-language models.  arXiv preprint arXiv:2306.09265, 2023.
  * [162] Jian-Bo Yang, Jun Liu, Jin Wang, How-Sing Sii, and Hong-Wei Wang.  Belief rule-base inference methodology using the evidential reasoning approach-rimer.  IEEE Transactions on systems, Man, and Cybernetics-part A: Systems and Humans, 36(2):266–285, 2006.
  * [163] Linyi Yang, Shuibai Zhang, Libo Qin, Yafu Li, Yidong Wang, Hanmeng Liu, Jindong Wang, Xing Xie, and Yue Zhang.  Glue-x: Evaluating natural language understanding models from an out-of-distribution generalization perspective.  In Findings of the Association for Computational Linguistics, pages 12731–12750, 2023.
  * [164] Qinghao Ye, Haiyang Xu, Guohai Xu, Jiabo Ye, Ming Yan, Yiyang Zhou, Junyang Wang, Anwen Hu, Pengcheng Shi, Yaya Shi, et al.  mplug-owl: Modularization empowers large language models with multimodality.  arXiv preprint arXiv:2304.14178, 2023.
  * [165] Qinghao Ye, Haiyang Xu, Jiabo Ye, Ming Yan, Haowei Liu, Qi Qian, Ji Zhang, Fei Huang, and Jingren Zhou.  mplug-owl2: Revolutionizing multi-modal large language model with modality collaboration.  arXiv preprint arXiv:2311.04257, 2023.
  * [166] Haoxuan You, Haotian Zhang, Zhe Gan, Xianzhi Du, Bowen Zhang, Zirui Wang, Liangliang Cao, Shih-Fu Chang, and Yinfei Yang.  Ferret: Refer and ground anything anywhere at any granularity, 2023.
  * [167] Peter Young, Alice Lai, Micah Hodosh, and Julia Hockenmaier.  From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions.  Transactions of the Association for Computational Linguistics, 2:67–78, 2014.
  * [168] Jiahao Yu, Xingwei Lin, and Xinyu Xing.  Gptfuzzer: Red teaming large language models with auto-generated jailbreak prompts.  arXiv preprint arXiv:2309.10253, 2023.
  * [169] Weihao Yu, Zhengyuan Yang, Linjie Li, Jianfeng Wang, Kevin Lin, Zicheng Liu, Xinchao Wang, and Lijuan Wang.  Mm-vet: Evaluating large multimodal models for integrated capabilities.  arXiv preprint arXiv:2308.02490, 2023.
  * [170] Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, et al.  Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi.  arXiv preprint arXiv:2311.16502, 2023.
  * [171] Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer.  Sigmoid loss for language image pre-training.  In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 11975–11986, October 2023.
  * [172] Pan Zhang, Xiaoyi Dong Bin Wang, Yuhang Cao, Chao Xu, Linke Ouyang, Zhiyuan Zhao, Shuangrui Ding, Songyang Zhang, Haodong Duan, Hang Yan, et al.  Internlm-xcomposer: A vision-language large model for advanced text-image comprehension and composition.  arXiv preprint arXiv:2309.15112, 2023.
  * [173] Yichi Zhang, Yinpeng Dong, Siyuan Zhang, Tianzan Min, Hang Su, and Jun Zhu.  Exploring the transferability of visual prompting for multimodal large language models.  arXiv preprint arXiv:2404.11207, 2024.
  * [174] Yue Zhang, Yafu Li, Leyang Cui, Deng Cai, Lemao Liu, Tingchen Fu, Xinting Huang, Enbo Zhao, Yu Zhang, Yulong Chen, et al.  Siren’s song in the ai ocean: a survey on hallucination in large language models.  arXiv preprint arXiv:2309.01219, 2023.
  * [175] Andrew Zhao, Daniel Huang, Quentin Xu, Matthieu Lin, Yong-Jin Liu, and Gao Huang.  Expel: Llm agents are experiential learners.  In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 19632–19642, 2024.
  * [176] Yunqing Zhao, Tianyu Pang, Chao Du, Xiao Yang, Chongxuan LI, Ngai-Man (Man) Cheung, and Min Lin.  On evaluating adversarial robustness of large vision-language models.  In Advances in Neural Information Processing Systems, volume 36, pages 54111–54138, 2023.
  * [177] Yiyang Zhou, Chenhang Cui, Jaehong Yoon, Linjun Zhang, Zhun Deng, Chelsea Finn, Mohit Bansal, and Huaxiu Yao.  Analyzing and mitigating object hallucination in large vision-language models.  In The Twelfth International Conference on Learning Representations, 2024.
  * [178] Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny.  MiniGPT-4: Enhancing vision-language understanding with advanced large language models.  In The Twelfth International Conference on Learning Representations, 2024.
  * [179] Kaijie Zhu, Jindong Wang, Jiaheng Zhou, Zichen Wang, Hao Chen, Yidong Wang, Linyi Yang, Wei Ye, Neil Zhenqiang Gong, Yue Zhang, et al.  Promptbench: Towards evaluating the robustness of large language models on adversarial prompts.  arXiv preprint arXiv:2306.04528, 2023.
  * [180] Andy Zou, Zifan Wang, J Zico Kolter, and Matt Fredrikson.  Universal and transferable adversarial attacks on aligned language models.  arXiv preprint arXiv:2307.15043, 2023.

##  Appendix

###  Appendix A Related Work

####  A.1 Multimodal Large Language Models

Building on the foundational capabilities of groundbreaking Large Language
Models (LLMs) such as GPT [4], PALM [7], Mistral [63], and LLama [139], which
excel in language understanding and reasoning, recent innovations have
integrated these models with other modalities (especially vision), leading to
the development of Multimodal Large Language Models (MLLMs). These advanced
MLLMs combine and process visual and textual data, demonstrating enhanced
versatility in addressing both traditional vision tasks [28, 51, 54, 167] and
complex multimodal challenges [44, 90, 170]. Among all MLLMs, proprietary
models consistently perform well. OpenAI’s GPT-4-Vision [105] pioneered this
space by adeptly handling both text and image content. Anthropic’s Claude 3
series [8] integrates advanced vision capabilities and multilingual support,
enhancing its application across diverse cognitive and real-time tasks.
Alibaba’s Qwen-VL-Plus [14] excels in multimodal tasks without relying on OCR
tools, showcasing superior processing power. Similarly, Google’s Gemini-Pro
[134] is designed to perform highly sophisticated reasoning tasks using text
and images, making it ideal for complex multimodal interactions. Transitioning
from these proprietary models, the open-source community has also made notable
contributions to the MLLM landscape, with pioneering works like MiniGPT-4
[178] and InstructBLIP [35]. Recently, researchers have focused on visual
instruction tuning, which enhances the alignment of visual and textual
information and boosts in-context learning capabilities. Notable examples
include LLaVA series [84], Qwen-VL-Chat [14], and CogVLM [149]. Models like
the LLaVA series [84] also leverage high-quality instruction tuning datasets
synthesized by GPT-4 [4], achieving significant progress in harmonizing visual
encoders with LLMs. Additionally, the Qwen-VL-Chat [14] scales up multimodal
pretraining, further enhancing the performance of open-source MLLMs. Beyond
these advanced techniques, another fundamental improvement involves updating
to more advanced visual encoders and LLM models, exemplified by InternVL-Chat
[30], InternLM-XComposer [135], and InternLM-XComposer2 [37]. Methods [92,
159, 173] are also proposed to enhance the model performance on diverse
downstream tasks via adaptation. Supported by these technical advancements,
the general capabilities of open-source models are gradually aligning with
those of proprietary models. However, the dimension of trustworthiness in both
open-source and proprietary MLLMs remains under-explored. Our work aims to
fill this gap by thoroughly investigating their trustworthiness.

####  A.2 Trustworthiness of LLMs

Currently, in the domain of trustworthiness-related evaluation, there have
been many studies for LLMs. The three prominent ones are DecodingTrust [144],
TrustLLM [129], and Trustworthy LLMs [88], all offering a comprehensive view
towards the trustworthiness of LLMs. DecodingTrust [144] first provides a
thorough evaluation across several perspectives like toxicity, stereotype
bias, adversarial robustness, privacy, etc., specifically focusing on models
like GPT-3.5 and GPT-4. It aims to uncover vulnerabilities to trustworthiness
threats such as the potential for generating biased or toxic outputs and
leaking private information. Unlike DecodingTrust, TrustLLM [129] and
Trustworthy LLMs [88] extend beyond GPT models to more general LLMs. While
TrustLLM introduces a broad framework that assesses the performance of LLMs by
integrating a diverse set of benchmarks and datasets, Trustworthy LLMs
concentrate more on proposing a comprehensive survey and guideline that
addresses a broad spectrum of trustworthiness concerns in LLMs. Besides such
comprehensive works, there are also several specialized benchmarks [17, 72,
95, 99, 113, 128, 152, 179] that address particular aspects like safety, bias,
robustness, etc. For instance, SafetyBench [128] serves as a benchmark to
evaluate the safety of LLMs, employing a dataset with multiple-choice
questions that span seven distinct safety concern categories. The Red-Teaming
benchmark [17], through a variety of simulated scenarios, rigorously tests
LLMs for security vulnerabilities that could lead to undesirable behaviors.
HarmBench [95] provides a standardized, large-scale evaluation framework for
automated red teaming, offering new insights and facilitating the development
of an advanced adversarial training method that significantly improves the
robustness of LLMs across diverse attacks As for the topic of stereotype and
bias, the StereoSet [99] benchmark provides an extensible codebase designed to
measure stereotypical bias in pre-trained language models. It offers tools to
replicate existing results and encourages the community to further assess and
reduce bias, inviting contributions to its leaderboard for ongoing evaluation.
The Do-Not-Answer benchmark [152] introduces a dataset specifically curated to
evaluate the safeguard mechanisms of LLMs. It comprises prompts that
responsible models are designed to avoid answering, thus testing the models’
ability to identify and refrain from engaging with inappropriate or sensitive
queries. For robustness, PromptBench [179] assesses LLMs’ robustness against
adversarial prompts, which is crucial for understanding how models perform
under potentially deceptive or harmful input conditions. GLUE-x [163]
evaluates the open-domain robustness of LLMs, which involves their performance
across a diverse range of unanticipated content, ensuring that they maintain
functionality regardless of the variability in input. Additionally, Latent
Jailbreak [113] explores both the safety and output robustness of LLMs when
confronted with malicious instructions, testing the models’ ability to resist
being manipulated into performing harmful or unintended actions. However, when
turning to the evaluation of trustworthiness in MLLMs, beyond the above
inherent vulnerabilities of LLMs, the introduction of multimodal dimensions
poses additional, complex risks. These include susceptibility to adversarial
image attacks [38, 176], the potential for toxic content in visual media
[150], and the possibility of jailbreaking through visual contexts [21, 111].
Consequently, the systematic assessment of MLLMs’ trustworthiness is not only
more challenging but also more necessary.

####  A.3 Benchmarks of MLLMs

Compared with benchmarks evaluating MLLMs in trustworthiness, benchmarks
evaluating MLLMs’ general capabilities are more common. Current MLLM
evaluation benchmarks [2, 70, 90, 161, 164, 169, 170] aim to provide
relatively holistic evaluations for the overall reasoning capabilities of
MLLMs. For instance, the LVLM-eHub [161] serves as a comprehensive evaluation
platform, rigorously testing MLLMs across six critical multimodal capabilities
including visual perception, knowledge acquisition, reasoning, and
commonsense, among others. This benchmark employs quantitative assessments
within an interactive online arena, strategically designed to elucidate both
the strengths and weaknesses of MLLMs in real-world interactive scenarios.
Concurrently, MMBench [90] not only assesses a broad spectrum of abilities,
including fine-grained abilities but also designs an innovative "Circular
Evaluation" strategy, preventing the subjectiveness introduced by human-
centered evaluation [164]. This method enhances the reliability of ovulation
results by cyclically rotating answer choices to test consistency in
generating correct responses, thereby addressing the challenges posed by
complex, free-form outputs in practical applications. Above all, though
benchmarks evaluating MLLMs’ general capabilities are prevalent, these
assessments do not necessarily reflect MLLMs’ performance in terms of
trustworthiness. Ensuring the reliability of MLLM applications also requires
rigorous evaluations of trustworthiness.

However, existing work in evaluating the trustworthiness of MLLMs, like [19,
29, 44, 49, 76, 81, 87, 142, 158], often focuses on examining one or a few
aspects of trustworthiness at a phenomenon level, such as object
hallucinations [44, 81, 158], toxicity [49, 87], jailbreaking via visual
contexts [21, 93, 111], OOD robustness [19, 142], adversarial image attacks
[38, 176], privacy leakage [29], etc. For example, POPE [76] focuses on
evaluating object hallucination in MLLMs, e.g., how often these models
generate objects in descriptions that are not present in images, followed by
more studies with visually challenging data [112, 137]. MM-SafetyBench [87] is
a framework specifically designed to evaluate the safety of MLLMs against
image-based manipulations, especially visual prompt attacks that exploit
query-relevant images for unintended or malicious model behavior.
JailBreakV-28K [93] is tailored for assessing the performance of MLLMs against
various jailbreak attacks, which explores the transferability of jailbreak
techniques from LLMs to MLLMs, highlighting critical vulnerabilities in MLLMs
when faced with both text-based and image-based attacks. BenchLMM [19]
evaluates the robustness of MLLMs against diverse visual styles. Other
researches make attempts to extend adversarial attacks in computer vision [39,
58, 154, 155] to attacking MLLMs for different objectives [21, 38, 176]. It
evaluates MLLMs’ capabilities to handle images beyond common photographic
styles, testing their performance across artistic, sensor-based, and
application-specific styles. In total, the above mainstream benchmarks all
refer to part of trustworthiness, which cannot be called “a comprehensive
benchmark”, failing to provide a global inspection of the trustworthiness in
modern MLLMs. Moreover, such benchmarks only concern threats in images but
neglect the interactions between modalities, i.e., whether the added image
modality affects the original text modality’s trustworthiness? But, it is
vital to consider this because superficial evaluations could lead to an
oversight of certain risks and a biased understanding of model
trustworthiness. Thus, in this paper, to fill the gap of comprehensive
evaluation in MLLMs’ trustworthiness and measure dual risks in both multimodal
and cross-modal dimensions, we propose our MultiTrust, the first comprehensive
and unified benchmark to evaluate the trustworthiness of MLLMs across diverse
dimensions.

###  Appendix B Common Implementations in Evaluation

In this section, we introduce the implementation details that are universally
consistent across different aspects and tasks, involving the deployment of
models, irrelevant images for studying cross-modal impacts, and templates for
keyword matching.

####  B.1 Model Deployment

As briefly introduced in Sec. 2.3, we gather 21 modern MLLMs for evaluation,
including 4 proprietary models and 17 open-source ones, as listed in Tab. B.1.
We select the LLaVA [85] family which is well known for its numerous
derivatives featuring diverse enhancements, including but not limited to
parameter scaling [83], supervised fine-tuning (SFT) with improved data [26,
148] and reinforcement learning from human feedback (RLHF) [130], in order to
identify the effects of these techniques on the trustworthiness. Two other
families of MiniGPT-4 and mPLUG-Owl with 4 models are then supplemented, which
both utilize the well aligned Llama2 [140] as their base LLMs but differ in
whether they fine-tune the LLM in multimodal training. Hopefully, this
selection strategy can help identify the effects of these techniques on the
trustworthiness in a more direct manner. Besides, we also include 4 popular
and powerful proprietary models [8, 14, 105, 134] and another 7 open-source
MLLMs [14, 30, 35, 37, 69, 149, 172] from different stages in the development,
in order to enlarge the model coverage and guarantee the representativeness of
our results.

Table B.1: Detailed information of the evaluated MLLMs in this paper. $\ast$
indicates that the base LLM is frozen during multimodal training. InternLM-
XComposer [135] and InternLM-XComposer2 [37] are abbreviated as InternLM-XC
and InternLM-XC2 in this paper.

Model | Open-Source | #Params | Vision Encoder | Base LLM | Key Features | Institution | Deployment
---|---|---|---|---|---|---|---
GPT-4-Vision111GPT-4-Turbo Preview version: gpt-4-1106-vision-preview [105] | No | - | - | - | - | OpenAI | Official API
Claude3222Claude3-Sonnet is used since ”it strikes the ideal balance between intelligence and speed”, ideal for wide applications (https://www.anthropic.com/news/claude-3-family) [8] | No | - | - | - | - | Anthropic | Official API
Gemini-Pro333Gemini-Pro-1.0 is adopted since extremely low rate limits of requests (2 RPM) for version 1.5 at the time of our experiments (https://ai.google.dev/pricing) [134] | No | - | - | - | - | Meta | Official API
Qwen-VL-Plus [14] | No | - | - | - | - | Alibaba | Official API
LLaVA-1.5-7B [83] | Yes | 7B | CLIP ViT-L | Vicuna-7B | - | UW–Madison | Locally Load
LLaVA-1.5-13B [83] | Yes | 13B | CLIP ViT-L | Vicuna-13B | LLM Scaled | UW–Madison | Locally Load
ShareGPT4V [26] | Yes | 13B | CLIP ViT-L | Vicuna-13B | GPT-4-Assisted SFT | Shanghai AI Lab | Locally Load
LVIS-Instruct4V [148] | Yes | 13B | CLIP ViT-L | Vicuna-13B | GPT-4-Assisted SFT | FDU | Locally Load
LLaVA-RLHF [130] | Yes | 13B | CLIP ViT-L | Vicuna-13B | Factuality-RLHF | UCB | Locally Load
LLaVA-NeXT [84] | Yes | 13B | CLIP ViT-L | Vicuna-13B | All-Around Improved | UW–Madison | Locally Load
MiniGPT-4-13B [178] | Yes | 14B | EVA-G | Vicuna-13B∗ | - | KAUST | Locally Load
MiniGPT-4-L2 [178] | Yes | 8B | EVA-G | Llama2-7B∗ | LLM Aligned | KAUST | Locally Load
mPLUG-Owl [164] | Yes | 7B | CLIP ViT-L | Llama-7B | - | Alibaba | Locally Load
mPLUG-Owl2 [165] | Yes | 7B | CLIP ViT-L | Llama2-7B | LLM Aligned | Alibaba | Locally Load
InstructBLIP [35] | Yes | 12B | EVA-G | Flan-T5-XXL∗ | - | Salesforce | Locally Load
Otter [69] | Yes | 7B | CLIP ViT-L | MPT-7B∗ | - | NTU | Locally Load
CogVLM [149] | Yes | 18B | EVA2-CLIP-E | Vicuna-7B | - | THU & Zhipu AI | Locally Load
Qwen-VL-Chat [14] | Yes | 10B | ViT-G/16 | Qwen-7B | - | Alibaba | Locally Load
InternVL-Chat [30] | Yes | 19B | InternViT-6B | Vicuna-13B | - | Shanghai AI Lab | Locally Load
InternLM-XC [172] | Yes | 9B | EVA-G | InternLM-7B | - | Shanghai AI Lab | Locally Load
InternLM-XC2 [37] | Yes | 9B | CLIP ViT-L | InternLM2-7B | - | Shanghai AI Lab | Locally Load

Through the unified interface of our platform, we configure all official
commercial APIs and deploy open-source models on a local server with 8 NVIDIA
H100 GPUs. To be specific, we adapt the official demo of each model to the
unified interface with minimum modifications while ensuring the flexibility to
set generation parameters. To ensure the reproducibility of results, we set
do_sample=False for open-source MLLMs and temperature=0 for proprietary MLLMs
(a fixed seed for Qwen-VL-Plus) in all tasks. The rest parameters follow the
default settings in the original implementations.

####  B.2 Irrelevant Image Pairing for Text-only Tasks

To fulfill the evaluation strategy considering the cross-modal impacts in Sec.
2.1 and perform consistent implementations across tasks, we collect a pool of
30 irrelevant images in 3 types (Fig. B.2) and sample one image from this pool
in each run. The noisy images are generated from the Gaussian distribution for
each pixel while the color blocks are crafted with Pillow
library444https://pillow.readthedocs.io/. The natural images are sampled from
ImageNet [120] and we manually check that they do not coincide with the topics
we are interested in this work. To balance the cost and reliability of the
experiments, we sample 3 images from each type for one text sample when
conducting studies into the cross-modal impacts with text-only datasets.

(a) Gaussian Noises

(b) Color Blocks

(c) Natural Images

Figure B.2: The pool of irrelevant images for evaluating the cross-modal
impacts.

####  B.3 Templates for Keyword Matching

It is a challenging problem to decide whether the model performs a certain
behavior, e.g. rejection, in generative tasks. One way is to adopt a prominent
general model like GPT-4 [4], which is expensive when the datasets scale.
Another option is to use a specially trained classifier [152]. The issue is
that they are not general enough and may fail in certain styles of answers.
Therefore, following the common practice [129, 144], it is essential to use
keyword matching to classify the model outputs into certain behaviors. In this
work, many tasks involve the decision of refusal in answers, for which we
unify the templates for keyword matching as shown in Fig. B.3. The templates
take reference from [144] and are further augmented based on the results in
pre-test. In each task using the templates, the specific configuration is
sometimes slightly modified to better adapt to the setting.

Figure B.3: Base templates for keyword matching to decide whether the response
is rejection.

###  Appendix C Evaluation Details on Truthfulness

Truthfulness is an essential prerequisite for ensuring the trustworthiness and
faithfulness of large-scale foundation models but remains challenging [12, 56,
57, 82, 117]. It has garnered great attention to evaluating and improving the
truthfulness of LLMs and MLLMs [73, 79, 91, 138, 174]. In this section, we
evaluate such traits of MLLMs, which require models to effectively understand
instructions and respond faithfully and correctly. Previous studies have
focused on various evaluation dimensions for truthfulness, such as
hallucination [76, 88, 129], sycophancy [88, 129], adversarial actuality [75],
inconsistency [88], etc. However, such division of dimensions seems to stay at
the phenomenon level. In this work, we aim to make a more reasonable division
of truthfulness in a more macro way.

Inspired by a theory of Biological Evolution [65] that phenotype depends on
both genotype and external environment, we divide truthfulness into two sub-
aspects: inherent deficiency and misguided mistakes to evaluate and analyze
MLLMs’ performance on truthfulness comprehensively. For inherent deficiency,
we aim to explore the inaccuracies under different fine-grained tasks, which
investigate the multi-dimensional capabilities of MLLMs. Such issues always
stem from limitations in MLLMs’ inherent capabilities resulting from data,
tasks or paradigms for training, as well as characteristics of model
architecture themselves. For misguided mistakes, we aim to assess how hard
samples given during inference will impact MLLMs, involving the use of
misleading visual or textual input data. Such tasks reflect MLLMs’ higher
levels of perception, thinking and reasoning abilities, basically consistent
with human beings.

####  C.1 Inherent Deficiency

To assess the inherent limitations of MLLMs, this sub-aspect systematically
evaluates their basic capabilities across diverse perspectives. Inspired by
[6]’s division of human abilities into low-level tasks, characterized as
straightforward activities that typically require minimal cognitive effort,
and high-level tasks, which involve complex, abstract thinking and problem-
solving, we first distinguish two tasks according to the degree of difficulty.
First, we examine the MLLMs’ basic perceptual abilities [11, 44, 90] like
object existence judgment, attribute recognition, scene analysis, etc.,
intending to discern coarse and fine-grained semantic information within
images. Second, on the basis of perception, we explore their advanced
cognitive and inferential capacities [44, 90] by engaging them in tasks that
require logical or commonsensical reasoning. Besides, we also investigate how
varying levels of detailed instructions [88] influence the models’ inner
performance, thereby exploring potential limitations in their ability to
interpret and respond to complex directives. Finally, we evaluate whether the
images positively related to the queries can assist factual knowledge QA tasks
[81], thus demonstrating the MLLMs’ natural capability to actively utilize
information from visual modality as an aid when dealing with such plain text
tasks.

#####  C.1.1 Basic World Understanding

Setting. In this task, we evaluate the models across five basic capabilities:
object existence judgment, attribute recognition, scene analysis, visual
grounding, and OCR, corresponding to both coarse and fine-grained classical
perception tasks from [11, 44, 75, 90, 166]. Regarding the format of the
output, for discriminative tasks such as object existence judgment, attribute
recognition, and scene analysis, the model must respond with either “Yes” or
“No”. For the visual grounding task, the model is instructed to return the
coordinates of the predicted bounding box. For the OCR task, which includes
both numbers and letters, the models must identify and output the textual
content in the images. Due to the distinct requirements of such tasks, prompt
templates are also different, which is detailed in the examples provided in
Fig. C.1. Additionally, we add the sentence “Please Directly answer [Yes] or
[No]!” for discriminative tasks, aiming to obtain better answers that are easy
to judge.

Figure C.1: Evaluation samples of MLLMs’ basic perceptual capabilities, which
correspond to five basic capabilities. We select answers from three classic
models, i.e., GPT-4-Vision, Qwen-VL-Chat, and MiniGPT-4-L2 for demonstration.
Hallucinatory answers are marked in red.

Dataset. In this task, we construct our evaluation dataset mainly by subtly
redesigning and refining existing ones. For object existence judgment, we
first randomly select 50 images from AMBER [11]. However, to ensure the
rationality of sample construction, we avoid using randomly selected negative
questions. Instead, we specially rewrite negative ones based on objects that
frequently co-occur with real targets in the image, like “ship” with “sea”,
thus being prone to errors. Additionally, we refine positive samples by adding
less prominent targets alongside the original conspicuous ones within the
images. Based on this, we finally generate two positive and three negative
image-text pairs for each image, leading to 250 pairs. For the attribute
recognition task, we also construct the dataset based on the above 50 images
from AMBER, encompassing attributes like actions, weather, count, color,
shape, state, etc. The negative pairs follow the same rationale as above to
more effectively challenge the models. Each image is associated with two
positive and two negative image pairs here, culminating in 200 image-text
pairs. For the scene analysis task, we utilize 200 image-text pairs from MME
[44] and modify their prompts to incorporate both a positive and a negative
sample per image. For visual grounding, we focus on precisely locating one
specific object in each image, rather than multiple objects without clear
constraints. Thus, we select 50 appropriate targets from Ferret [166] and
adapt both prompts and labels to align with our specific requirements. For the
OCR task, we choose 50 high-quality images each from RTVLM [75] and SVT [13],
tailored for recognizing six-digit verification codes and street view logos,
respectively. Prompts are regenerated for such testing purposes. Overall, the
dataset for this task amounts to 800 image-text pairs.

Metrics. In this task, we utilize accuracy to evaluate five capabilities with
distinct response formats, which necessitates tailored evaluation methods for
each type. First, for the former three discriminative tasks, where responses
typically include “Yes” or “No”, we can straightforwardly assess correctness
through keyword matching, allowing direct calculation of overall, positive,
and negative sample accuracies. Additionally, given the inclusion of both
positive and negative samples, we analyze potential answer tendencies in
specific sub-tasks. Discrepancies exceeding 60% between the accuracies of
positive and negative responses are highlighted, indicating a model’s biased
inclination to answer “Yes” or “No”. For visual grounding, to address
variations in models’ abilities in following instructions and response
formats, we standardize coordinates into the format [$\text{x}_{\text{min}}$,
$\text{y}_{\text{min}}$, $\text{x}_{\text{max}}$, $\text{y}_{\text{max}}$] and
normalize them between $[0,1]$ with the assistance of GPT-4 as shown in Fig.
C.2. This standardization facilitates an accurate calculation of the
Intersection Over Union (IOU) and comparison with a threshold of 0.5, where
results exceeding this threshold are deemed correct and incorrect otherwise.
For OCR, we utilize GPT-4 to assess the alignment between the predicted
content and the ground-truth, and then compute the accuracy.

Figure C.2: Prompt for reforming response into a standard form of bounding box
in visual grounding tasks, enabling unified evaluation for different MLLMs.
Notably, the coordinates returned by Qwen-VL-Plus or Qwen-VL-Chat are all
values based on the premise that the image size is [1000,1000], so the term GT
Size in the evaluation prompt is uniformly set to [1000,1000] for such two
models. Table C.1: Results (%) of Basic World Understanding. The top five
model results are marked in green, and the bottom five model results are
marked in red. If there is a tie, the results with the same value in the
ranking are all marked. $\dagger$ denotes the model has a preference for
“Yes”, and $\ddagger$ denotes “No”.

Model | Object | Attribute | Scene | Grounding | OCR
---|---|---|---|---|---
GPT-4-Vision | 86.00 | 78.00 | 82.00 | - | 46.00
Claude3 | 92.00 | 67.00 | 81.00 | 10.00 | 64.00
Gemini-Pro | 80.80 | 72.50 | 70.00 | 8.00 | 66.00
Qwen-VL-Plus | 94.80 | 84.50 | 85.75 | 70.00 | 84.00
LLaVA-1.5-7B | 92.80 | 83.00 | 85.50 | 44.00 | 33.00
LLaVA-1.5-13B | 82.00 | 79.50 | 85.75 | 40.00 | 44.00
ShareGPT4V | 76.00 | 76.00 | 87.25 | 54.00 | 46.00
LVIS-Instruct4V | 92.40 | 81.00 | 84.75 | 34.00 | 37.00
LLaVA-RLHF | 86.80 | 76.50 | 85.75 | 28.00 | 35.00
LLaVA-NeXT | 81.60 | 82.00 | 82.75 | 6.00 | 43.00
MiniGPT-4-13B | $52.80$‡ | $34.00$‡ | 56.75 | 6.00 | 18.00
MiniGPT-4-L2 | $40.80$† | $50.00$† | 71.50 | 0.00 | 28.00
mPLUG-Owl | 46.40 | 52.50 | 73.50 | 6.00 | 48.00
mPLUG-Owl2 | 91.60 | 81.00 | 84.50 | 20.00 | 45.00
InstructBLIP | 86.40 | 72.00 | 79.75 | 0.00 | 31.00
Otter | 71.60 | 63.50 | 85.75 | 0.00 | 34.00
CogVLM | 89.20 | 82.00 | 83.75 | 6.00 | 77.00
Qwen-VL-Chat | 91.60 | 80.50 | 87.50 | 72.00 | 41.00
InternVL-Chat | 88.80 | 79.50 | 86.25 | 42.00 | 43.00
InternLM-XC | 82.40 | 68.00 | 86.00 | 8.00 | 61.00
InternLM-XC2 | 93.20 | 83.00 | 88.25 | 32.00 | 58.00
Task_Average | 80.95 | 72.67 | 81.62 | 24.30 | 46.76

Results. The results for basic world understanding are shown in Tab. C.1. From
the perspective of task performance, in the first three tasks—object existence
judgment, attribute recognition, and scene analysis, most models are well-
behaved, achieving up to 80% and in some cases, exceeding 90%. Comparing the
performance of the three tasks, a slight decline exists in more fine-grained
perception tasks, such as attribute recognition. Then, shifting to the model
performance perspective, among proprietary models, Qwen-VL-Plus is overall the
best, while the other three models often only have average-level performance.
For open-source models, InternLM-XC2 performs well on all tasks, but several
early models, e.g., MiniGPT-4 series, mPLUG-Owl, have poor performance, only
reaching about 40% to 50% accuracy on tasks like object existence judgment and
attribute recognition. Even worse, we notice that MiniGPT-4 series
demonstrates a clear preference for the response, where MiniGPT-4-Llama2 tends
to answer “Yes” and MiniGPT-4-Vicuna-13B tends to answer “No”. This means that
such models do not have a good ability to perceive these elements, so their
results are not credible as a whole. Additionally, it’s interesting to find
that LLaVA-1.5-7B has the best performance among the LLaVA series in two of
the first three tasks, indicating that the increasing parameter may not bring
the improvement of model performance in such basic tasks. Since the remaining
two fine-grained tasks do not show a clear consistent trend, we will analyze
them respectively. For the visual grounding task, Qwen-VL series has
unprecedented superiority among all models with an accuracy of 70% for Qwen-
VL-Plus and 72% for Qwen-VL-Chat. The other proprietary models all perform
limitedly in this task with accuracies of no more than 10%. Among them,
GPT-4-Vision refuses to conduct grounding with a response of “I’m sorry, …”,
while other models provide coordinates with huge deviations. On the contrary,
proprietary models show a leading performance in the OCR task, among which
Qwen-VL-Plus’s accuracy even reaches 84%. It is necessary to note that
GPT-4-Vision is slightly lower because it touches the safeguard mechanism when
performing digital recognition. In open-source models, CogVLM ranks first with
an accuracy of 77%, while several models only achieve under 30% accuracy, such
as MiniGPT-4 series.

Findings. (1) The performance of all MLLMs declines on tasks with higher fine-
grained requirements, where exists substantial room for improvement. The
performance differences are also widened in such tasks, indicating that the
limitations of the inherent abilities vary across different MLLMs. (2)
Compared with the latest open-source models like InternLM-XC2 and InternVL-
chat, the proprietary models have limited advantages for such perception
tasks, especially for the visual grounding task. Only Qwen-VL-Plus as well as
the open-source model Qwen-VL-Chat have outstanding performance on the above
tasks. This could be attributed to the fact that the capacity for fine-grained
visual understanding of the Qwen series has been specially trained with
higher-resolution input size and fine-grained corpus, enabling fine-grained
dialog performance. (3) The upgrade of LLM could bring obvious improvement to
MLLMs, which could be verified by the result that the performance of the
second-generation models in the InternLM and mPLUG-Owl series significantly
outperforms the first ones. (4) The increasing size of LLM parameters does not
necessarily enhance the overall performance of truthfulness, as evidenced by
that other models in the LLaVA series with larger LLMs have no obvious
advantages over LLaVA-1.5-7B in the above basic tasks, even possessing extra
training techniques.

#####  C.1.2 Advanced Cognitive Inference

Setting. Considering the advanced cognitive and reasoning capabilities [46,
143, 151] that human beings have, we divide the evaluation in this task into
four main facets: spatial-temporal reasoning, attribute comparison,
commonsense reasoning and specialized skills. Generally, we provide the model
with an image and a corresponding question as input, which needs to be solved
by reasoning based on the perceived elements. For spatial-temporal reasoning,
the model is asked to judge the orientation relationship between two objects
or to judge the order of several frames similar to [81, 90]. For attribute
comparison, we refer to existing task design [90] and mainly focus on the
following properties of two targets: quantity, size, age, length, etc. For
commonsense reasoning, our tasks exhibit more precise differentiation compared
to those in [44]. We require the model to assess whether a causal relationship
exists between the fact described in the prompt and the corresponding image.
Additionally, the model is tasked with determining an appropriate action in
response to the scenario depicted in the image, covering two prevalent
contexts encountered in life: daily and traffic situations. For specialized
skills, we follow [44] and consider three types of skills: mathematical
problems, code and Chinese translation tasks based on OCR. All above are
discriminative tasks, and the specific examples are shown in Fig. C.3. Given
that this task tests cognitive capabilities requiring analysis, we prompt for
explanations by adding a sentence after the question: “Please answer with
[Yes] or [No] according to the picture, and give a short explanation about
your answer.”.

Figure C.3: Evaluation samples of cognitive and inferential tasks,
corresponding to four advanced capabilities. We select answers from various
models and hallucinatory answers are marked in red.

Dataset. We evaluate this task using both existing datasets and hand-made
ones. For spatial-temporal reasoning, we construct a dataset of 100 image-text
pairs. Among them, the 60 pairs are derived from MME [44] for spatial
reasoning, which include both a positive and a negative question per image,
and the remaining 40 pairs, aimed at temporal reasoning, are from
HallusionBench [81] or online images, with positive image consisting of $2\sim
4$ sub-images and negative samples being such sub-images rearranged of their
sequences. For attribute comparison, we choose 50 images from [11, 44], and
pair each image with a positive question and negative question, finally
amounting to 100 image-text pairs. For commonsense reasoning, we construct 140
image-text pairs divided into three parts: 20 manually collected images for
causal inference, 25 images from MME [44] for daily behavior, and 25 images
from SVT [13] for decision-making in traffic scenarios. Each image is paired
with two hand-made questions with positive and negative settings. Lastly, we
improve MME [44]’s data to test three specialized skills, among which the
mathematical problems are divided into two categories: geometry and algebra.
Then each of the four parts contains 40 image-text pairs. For mathematical
problems, we filter its original datasets and the involved problems are
improved by deleting too simple ones and enriching the types of mathematical
problems. As for coding and translation tasks, we directly use original MME
samples [44] and reform their prompts to meet our needs. Overall, the whole
datasets amount to 500 image-text pairs.

Metrics. Due to the consistency of responses in discriminative tasks, we also
choose accuracy as the metric and assess MLLMs’ performance on these tasks by
keyword matching (e.g.,“Yes”,“No”,“not”). Besides, same as Sec. C.1.1, we
calculate the accuracy on the whole dataset, positive samples and negative
samples respectively, and then record whether each model has an answer
tendency in specified sub-tasks with the uniform setting. As for the average
result of each subtask, we calculate it by conducting a weighted average based
on the data number within it.

Table C.2: Results (%) of Advanced Cognitive Inference. The top five model
results are marked in green, and the bottom five model results are marked in
red. If there is a tie, the results with the same value in the ranking are
marked. $\dagger$ denotes the model has a preference for “Yes”, and $\ddagger$
denotes “No”.

| Spatial-Temporal Reasoning | Commonsense Reasoning | Specialized Skills | Attribute
---|---|---|---|---
| Spatial | Temporal | Average | Daily | Traffic | Causality | Average | Math | Code | Translate | Average | Comparison
GPT-4-Vision | 70.00 | 57.50 | 65.00 | 88.00 | 90.00 | 80.00 | 86.43 | 60.00 | 85.00 | 70.00 | 71.67 | 71.00
Claude3 | 78.33 | 67.50 | 74.00 | 72.00 | 68.00 | 65.00 | 68.57 | 55.00 | 67.50 | 57.50 | 60.00 | 68.00
Gemini-Pro | 48.30‡ | 52.50 | 50.00 | 82.00 | 76.00 | 80.00 | 79.29 | 51.25 | 65.00 | 90.00 | 68.75 | 54.00
Qwen-VL-Plus | 83.33 | 72.50 | 79.00 | 84.00 | 86.00 | 75.00 | 82.14 | 70.00 | 80.00 | 100.00 | 83.33 | 65.00
LLaVA-1.5-7B | 76.67 | 52.50 | 67.00 | 70.00 | 72.00 | 55.00 | 66.43 | 50.00‡ | 55.00‡ | 50.00‡ | 51.67 | 60.00
LLaVA-v1.5-13B | 70.00 | 57.50 | 65.00 | 70.00 | 76.00 | 67.50 | 71.43 | 48.75‡ | 57.50‡ | 50.00‡ | 52.08 | 57.00
ShareGPT4V | 63.33 | 52.50 | 59.00 | 68.00 | 74.00 | 45.00 | 63.57 | 50.00 | 55.00‡ | 47.50‡ | 50.83 | 54.00
LVIS-Instruct4V | 73.33 | 55.00 | 66.00 | 76.00 | 68.00 | 47.50 | 65.00 | 50.00‡ | 55.00‡ | 50.00‡ | 51.67 | 54.00
LLaVA-RLHF | 55.00‡ | 47.50 | 52.00 | 66.00 | 64.00‡ | 62.50 | 64.29 | 43.75‡ | 40.00‡ | 50.00‡ | 44.58 | 47.00
LLavA-NeXT | 66.67 | 47.50‡ | 59.00 | 68.00 | 74.00 | 60.00 | 67.86 | 48.75‡ | 52.50‡ | 50.00‡ | 50.42 | 56.00
MiniGPT-4-13B | 51.67 | 62.50 | 56.00 | 66.00 | 70.00 | 57.50 | 65.00 | 50.00 | 52.50‡ | 37.50‡ | 46.67 | 46.00
MiniGPT-4-L2 | 50.00† | 60.00 | 54.00 | 64.00 | 76.00 | 57.50 | 66.43 | 43.75 | 50.00† | 52.50‡ | 48.75 | 50.00†
mPLUG-Owl | 50.00† | 52.50† | 51.00 | 66.00 | 76.00 | 65.00† | 69.29 | 45.00 | 50.00† | 60.00† | 51.67 | 52.00†
mPLUG-Owl2 | 56.67 | 52.50 | 55.00 | 68.00 | 72.00 | 65.00‡ | 68.57 | 45.00‡ | 45.00‡ | 55.00‡ | 48.33 | 51.00
InstructBLIP | 50.00 | 55.00‡ | 52.00 | 70.00 | 60.00‡ | 65.00 | 65.00 | 47.50‡ | 32.50 | 50.00‡ | 43.33 | 56.00
Otter | 53.33 | 50.00 | 52.00 | 60.00 | 64.00 | 55.00 | 60.00 | 37.50 | 45.00† | 50.00‡ | 44.17 | 49.00
CogVLM | 68.33 | 52.50 | 62.00 | 76.00 | 78.00 | 62.50 | 72.86 | 46.25† | 52.50† | 52.50‡ | 50.42 | 61.00
Qwen-VL-Chat | 71.67 | 52.50 | 64.00 | 76.00 | 68.00 | 55.00 | 67.14 | 43.75 | 30.00 | 82.50 | 52.08 | 61.00
InternVL-Chat | 71.67 | 52.50 | 64.00 | 62.00 | 68.00 | 67.50 | 65.71 | 47.50‡ | 50.00 | 50.00‡ | 49.17 | 55.00
InternLM-XC | 51.67 | 57.50 | 54.00 | 56.00‡ | 56.00 | 57.50 | 56.43 | 50.00‡ | 45.00 | 60.00 | 51.67 | 50.00‡
InternLM-XC2 | 70.00 | 47.50 | 61.00 | 78.00 | 76.00 | 65.00 | 73.57 | 56.25 | 65.00 | 75.00 | 65.42 | 64.00†
Task_Average | 63.33 | 55.12 | 60.05 | 70.76 | 72.00 | 62.38 | 68.81 | 49.52 | 53.81 | 59.05 | 54.13 | 56.24

Results. The results for the advanced cognitive inference task are reported in
Tab. C.2. From the perspective of task performance, we notice that causality
inference and behavior decision-making have better performance compared with
objective and absolute inference, i.e., spatial-temporal reasoning, and
attribute comparison. In the statistical process, we further find that
temporal reasoning is worse than spatial reasoning, it is acceptable because
the former is based on the semantic analysis between several frames; while the
latter only needs to analyze two targets within a single image. Besides, most
results for specialized skills are extremely backward, reflecting their lack
of learning such professional skills. Only some models designed with
multilingual support perform well in translation tasks, even achieving an
accuracy of over 80%. Turning to the perspective of model performance, the
overall performance of proprietary models is better, ranking in the top 4. In
particular, GPT-4-Vision and Qwen-VL-Plus are more stable in all aspects,
which further verifies that the proprietary models have better inherent
cognitive capabilities. For open-source models, InternLM-XC2 still has good
performance, as well as LLaVA-1.5-7B, LLaVA-1.5-13B, and CogVLM. However, the
performance of several models is unsatisfactory, especially Otter, and LLaVA-
RLHF, where some tasks show an accuracy of less than 50%.

Findings. (1) MLLMs perform better on the knowledge-based task of commonsense
reasoning than ability-based problems, e.g., spatial-temporal reasoning, tasks
of specialized skills. Obviously, the former reasoning abilities are often
acquired from LLMs directly, while ability-based tasks must rely on a full,
fine-grained understanding of images, among which spatial reasoning even needs
to understand detailed semantics and infer across multiple images. The
deficiency of such an internal ability deserves extra attention in follow-up
improvement. (2) In such advanced tasks involving cognitive and reasoning
abilities, proprietary models generally perform better than open-source
models. (3) Some models tend to respond with all “Yes” or all “No” without
engaging in substantive reasoning based on the prompt and image, especially in
tasks involving specialized skills.

#####  C.1.3 VQA under Instruction Enhancement

Setting. In this task, we hope to explore how different instruction prompts
could affect the model’s responses to the same question. Therefore, we provide
the model with an image and three progressively complex prompts as inputs. The
initial prompt, set as the default, is designed to elicit a direct and clear
response. The second prompt encourages the model to actively engage with the
image, while the third and most detailed prompt aims to foster deeper
cognitive thinking. Then, to comprehensively assess the models’ capabilities
to understand and respond to such varying complex instructions, we select two
classical sub-tasks: logical task [110] and factual task [162] for evaluation.
Specifically, we choose mathematical problems in Sec. C.1.2 for the logical
task and VQA with nonfactual text information in Sec. C.2.1 for the factual
task. Additionally, to more clearly observe the differences in model
performance, the chosen tasks in this section are also adapted to generative
tasks, where MLLMs need to answer the target question rather than merely
return “Yes” or “No”. Examples of three varying prompts under such two tasks
are illustrated in Fig. C.4.

Figure C.4: Prompt designs for the logical and factual tasks. Figure C.5:
Evaluation samples of the logical and factual tasks under varying
instructions. Hallucinatory answers are marked in red.

Dataset. Because selected tasks overlap with other parts, we directly evaluate
MLLMs based on datasets that belong to the original two parts of the tasks,
corresponding to 40 and 100 images respectively. Among them, the instruction
prompts for mathematical problems in Sec. C.1.2 are regenerated to adapt to
the generative commands here, and details of the data composition for factual
tasks can be found in the dataset introduction of Sec. C.2.1 later. The
datasets totally contain 140 data samples, each matching the three queries
above. Overall, there are 420 image-text pairs in this task, and two examples
of such two tasks are presented in Fig. C.5.

Metrics. We aim to analyze the results from two aspects. First, given our
primary goal of maximizing response efficacy through observing the influences
of varying prompts, we take the highest accuracy among three prompt variations
as the final result. This value is then compared with the performance achieved
using the original prompts. GPT-4 is used here to judge the responses’
correctness and prompts for GPT-4 are shown in Fig. C.6. Besides, to better
understand the preferences of different MLLMs for various prompts, we also
display the detailed accuracies of the three queries in Fig. C.7.

Figure C.6: Prompt to use GPT-4 for judging the response from MLLMs. Figure
C.7: Effects of different instructions on logical and factual problems. The
models’ preference for different prompts can be seen from their performance.

Results. According to Tab. C.3, it is evident that various designs of prompts
have an impact on the prediction results. First, from the perspective of task
performance, we notice that the logical task and the factual task demonstrate
inconsistent characteristics. For the logical task, a macro trend is that
proprietary models with better comprehensive capabilities could achieve
effective use of indication information in prompts, e.g., “according to the
image”, and “step by step”, to improve their accuracy, while open-source
models, many of them could not benefit from such detailed prompts,

Table C.3: Accuracy (%) of Instruction Enhancement. The top five model results
are marked in green, and the bottom five model results are marked in red. If
there is a tie, the results with the same value in the ranking are all marked.
$\uparrow$ denotes improvement.

| Logical | Factual
---|---|---
Model | Origin | Max | Origin | Max
GPT-4-Vision | 52.5 |  65 $\uparrow$ | 61 |  68 $\uparrow$