# Bi-LORA: A Vision-Language Approach for Synthetic Image Detection

Mamadou Keita, Wassim Hamidouche, Hessen Bougueffa Eutamene, Abdenour Hadid,
Abdelmalik Taleb-Ahmed Mamadou Keita, Hessen Bougueffa Eutamene and Abdelmalik
Taleb-Ahmed are with Institut d’Electronique de Microélectronique et de
Nanotechnologie (IEMN), UMR 8520, Université Polytechnique Hauts de France,
Valenciennes, France Wassim Hamidouche is with the Technology Innovation
Institute, Abu Dhabi, UAE.Abdenour Hadid is with the Sorbonne Center for
Artificial Intelligence, Sorbonne University, Abu Dhabi, UAE.

###### Abstract

Advancements in deep image synthesis techniques, such as generative
adversarial networks and diffusion models, have ushered in an era of
generating highly realistic images. While this technological progress has
captured significant interest, it has also raised concerns about the potential
difficulty in distinguishing real images from their synthetic counterparts.
This paper takes inspiration from the potent convergence capabilities between
vision and language, coupled with the zero-shot nature of vision-language
models. We introduce an innovative method called Bi-LORA that leverages VLMs,
combined with low-rank adaptation (LORA) tuning techniques, to enhance the
precision of synthetic image detection for unseen model-generated images. The
pivotal conceptual shift in our methodology revolves around reframing binary
classification as an image captioning task, leveraging the distinctive
capabilities of cutting-edge VLM, notably bootstrapping language image pre-
training (BLIP)2. Rigorous and comprehensive experiments are conducted to
validate the effectiveness of our proposed approach, particularly in detecting
unseen diffusion-generated images from unknown diffusion-based generative
models during training, showcasing robustness to noise, and demonstrating
generalization capabilities to GANs. The obtained results showcase an
impressive average accuracy of 93.41% in synthetic image detection on unseen
generation models. The code and models associated with this research can be
publicly accessed at https://github.com/Mamadou-Keita/VLM-DETECT

###### Index Terms:

Deepfake, Text-to-Image Generation, Visual Language Model, Large Language
Model, Image Captioning, Generative Adversarial Nets, Diffusion Models, Low
Rank Adaptation.

##  I Introduction

The rapid progress in generative models has ushered in a new era of image
synthesis, marked by the effortless creation of remarkably realistic images.
This technological progress, while undeniably impressive, has also given rise
to significant concerns regarding the widespread distribution of fake images.
These artificial intelligence (AI)-synthesized images, due to their unrivalled
visual fidelity, represent a substantial threat to various facets of society.
Noteworthy examples, such as the generation of realistic face images through
techniques like GANs and DMs, have reached a point where they challenge human
perception and erode trust in a number of areas, including social media,
politics, the military, and cybersecurity.

The pressing issue of discerning between authentic and AI-generated images is
underscored by the remarkable realism achieved by contemporary generative
models, surpassing human perceptual capabilities. As demonstrated in a
quantitative study by Lu et al. [1], human observers grapple with
distinguishing genuine images from their AI-generated counterparts, achieving
an accuracy rate as low as 61.3%. This challenge extends beyond academic
research, permeating real-world applications, presenting both opportunities
and emerging threats. The advent of text-to-image models allows users to
convert textual descriptions into high-quality visual representations, a
technological leap with potential malicious applications, including the spread
of misinformation and propaganda. In light of these complex challenges, the
primary goal is to prioritize the development of efficient detectors capable
of reliably identifying fake images generated by these advanced models.

In the realm of image analysis, a myriad of techniques has been developed to
detect AI-generated images. While DMs typically utilize convolutional neural
networksor vision transformers as their foundational architecture, it’s
crucial to highlight the fundamental disparities in the underlying mechanisms
driving image generation in DMs compared to their predecessors like
variational autoencoders and GANs [2]. These distinctions in generative
processes create a significant gap between detectors designed for earlier
generative models and their ability to identify AI-generated images generated
by diffusion models. To tackle this discrepancy, a seemingly direct approach
is to train a binary classifier using CNNs or transformer-based models to
distinguish diffusion-generated images from genuine ones. However, an
empirical study conducted by Ricker et al. [3] has unveiled a fundamental
weakness inherent in this approach. Despite its initial appeal, this
simplistic classification framework demonstrates a serious lack of efficiency
in generalizing to accurately discriminate new diffusion-generated images that
have never been encountered before.

Recent years have witnessed the emergence of a new generation of powerful deep
architectures in the field of computer vision. These models, often referred to
as foundational models or large VLMs, have been trained on vast image-text
datasets from the vast expanse of the internet [4]. They have made remarkable
strides in terms of performance and have assumed a pivotal role in the
contemporary computer vision research landscape. Situated at the intersection
of natural language processing (NLP) and computer vision (CV), these models
bridge the gap between textual and visual data, ushering in a groundbreaking
era of machine understanding. Prominent instances include CLIP [5], Flamingo
[6], BLIP2 [7], Flava [8], Llava [9], SimVLM [10], Visual-BERT [11],
InstructBLIP [12], and ViTGPT2111https://ankur3107.github.io/blogs/the-
illustrated-image-captioning-using-transformers/. Within the research
community, there exists a substantial body of work dedicated to vision-
language tasks that require the simultaneous processing of information from
visual and linguistic sources to address intricate inquiries. For example, in
visual question answering (VQA), an image and a corresponding question are
provided as input, and the model generates an answer to the query [13].
Conversely, in image captioning, the model takes an image as input and
produces a natural language description of that image [14].

In this paper, we introduce a pioneering approach for synthetic image
detection, harnessing state-of-the-art VLM. Traditional methods use complex
feature extraction with CNNs [15] for image classification. In contrast, our
approach, named Bi-LORA, reframes the problem as image captioning, leveraging
VLMs’ text generation to differentiate between real and synthetic (i.e., AI-
generated) images. While a similar approach [16] adopts an alternative
strategy, such as visual question answering, our focus is on generating
descriptive captions for images. Specifically, we label authentic images as
”true” and synthetic images as ”fake”, capitalizing on the text generation
capabilities of VLMs (e.g., BLIP2) combined with LORA tuning strategy for
memory efficiency and faster training. Our primary focus is on detecting
diffusion-generated images, a challenge distinct from previous generative
methods like GANs. To the best of our knowledge, this work pioneers the
concept of treating binary classification as image captioning, harnessing
cutting-edge VLMs. Instead of binary classification, we emphasize the
potential of using VLMs (e.g., BLIP2) to create informative captions
indicating class membership. In summary, our work makes the following pivotal
contributions:

  * •

Reconceptualizing binary classification as image captioning: We redefine
binary classification as an image captioning task, harnessing the capabilities
of VLMs.

  * •

Revealing the potential of VLMs in synthetic image detection: We shed light on
the vast potential of VLMs in the realm of synthetic image detection,
showcasing their robust generalization capabilities, even when faced with
previously unseen diffusion-generated images.

  * •

Empirical validation of enhanced detection: Through rigorous and comprehensive
experiments, we substantiate the effectiveness of our proposed approach,
particularly in the context of detecting diffusion-generated images,
robustness to noise and generalization to images generated by GAN.

Our proposed approach, trained only on LSUN Bedroom and latent diffusion model
(LDM) generated images, far exceeds baseline methods: ResNet50 [17], Xception
[18], data-efficient image transformers (DeiT) [19], and ViTGPT2. Our
outstanding performance and robust generalization stem from the well-aligned
vision language representation of pretrained VLMs and with fewer trainable
parameters. Experimental results showcase an impressive average accuracy $\bf
93.41\%$ in synthetic image detection.

The remainder of this paper is organized as follow. Section II provides a
brief review of related works. Section III describes the proposed VLM-based
approach for synthetic image detection task. Then, the performance of the
proposed approach is assessed and analysed in Section IV. Finally, Section V
concludes the paper.

Figure 1: Examples of real sample and synthetic images generated by various
generators in our experiments. From left to righ, Top row: Real [20], ADM
[21], LDM. Middle row: DDPM [22], IDDPM [23], PNDM [24]. Bottom row: Stable
Diffusion v1.4 [25], GLIDE [26], DALL·E 3 [27].

##  II Related Work

In this section, we delve into the realm of synthetic image generation,
providing an overview of state-of-the-art methods for both image generation
and synthetic image detection.

###  II-A Image Generation

Image generation stands at the intersection of computer vision and machine
learning, focusing on the creation of synthetic images through the use of deep
learning-based generative models. Prominent among these models are GANs and,
more recently, DMs. The primary goal of these image generation models is to
create visually realistic images that are nearly indistinguishable from real-
world images. This field has experienced rapid growth, witnessing numerous
advancements in recent years, thereby unlocking a spectrum of applications,
including data augmentation, style transfer, and semantic manipulation.

A pioneering method for generating synthetic images was introduced by
Goodfellow et al. [28]. This approach consists of a generator network that
generates images and a discriminator network trained to distinguish between
real and AI-generated images. Since then, this framework named GANs has been
successfully applied to various tasks such as face synthesis [29, 30, 31, 32,
33, 34], style transfer [35, 36, 37], and super-resolution [38, 39, 40, 41,
42]. In GAN-based image generation, significant progress has been made with
the introduction of StyleGAN [31]. This architecture exploits style-based
techniques to generate more realistic and compelling images. By incorporating
style information into the generating network, StyleGAN gives greater control
over the images generated and produces high-quality results. Several variants
and extensions of GAN architectures have been proposed to meet specific
challenges and tasks in the field of image generation, such as unpaired data
in image generation [43], allowing control over specific image characteristics
[44], and focusing on image generation without labeled data [29].

On the other hand, text-to-image generation involves processing a given text
description, known as a prompt, to generate a corresponding image. Early
endeavors in this domain [45, 46] utilized GANs for this purpose. These
studies employed the fusion of a prompt’s embedding with a latent vector to
produce an image that accurately reflects the given description. However,
despite the attention garnered by numerous works [47, 48, 49, 50, 51, 52], it
is essential to acknowledge that GANs do not consistently yield satisfactory
generation outcomes [53, 25].

More recently, a groundbreaking architecture known as DMs [54, 26, 25, 55] has
emerged, revolutionizing tasks akin to synthetic image generation.
Distinguished by their ability to progressively remove noise from a signal,
these models stand out for their capacity to generate exceptionally realistic
images. In the realm of text-to-image generation, a provided text prompt
undergoes encoding into a latent vector, guiding the transformation process
from noisy images to clear and intricate representations. Several leading
models, such as DALL-E [53], GLIDE [26], Midjourney [56], Imagen [55], and
Stable Diffusion [25], have showcased exceptional results. These models
provide unprecedented control over the image generation process, allowing
users to craft highly credible images with an impressive level of detail (see
Fig. 1).

###  II-B Detection of Synthetic Images

With the rapid advancement of generative models, the need for efficient
methods to detect synthetic images has become evident. While existing
detectors have been effective with traditional models like GANs [28], recent
progress in generative models, particularly diffusion-based architectures and
the latest GAN models, presents major challenges for current detection
techniques.

Recent investigations conducted in [57, 3] have studied the efficacy of
existing detectors when applied to novel generative models. The outcomes of
these studies indicate that current detectors encounter challenges in
generalizing their performance to diffusion-based generative architectures and
the latest GAN models. This limitation underscores the necessity for the
development of new techniques capable of adeptly detecting images generated by
these advanced models. While substantial progress has been achieved in
detecting manipulated images produced by traditional models such as GANs, the
specific realm of detecting images generated by diffusion-based models remains
relatively unexplored.

In their work [58], the authors employed the ResNet50 [59] model to discern
between real images and those generated by AI. They introduced a detection
approach that integrates both images and textual prompts, showcasing its
efficacy in improving detection accuracy. This method exhibits promising
potential for enhancing overall detection capabilities. Similarly, Coccomini
et al. [60] conducted experiments utilizing straightforward deep learning
models such as multi-layer perceptrons and common CNNs (ResNet50 [59] and
XceptionNet [18]) to differentiate between real and AI-generated images. They
also implemented a multimodal detector that leverages CLIP as a feature
extractor from both images and their corresponding captions, followed by
passing the combined features through an MLP.

Table I: Architectures and features of state-of-the-art AI-synthesized images
detection models

Authors Architecture Dataset Task Sha et al.♣ (2022) [58] CNN, CLIP \+ MLP DE-
FAKE binary classification Coccomini et al.♣ (2023) [60] CNN, CLIP \+ MLP
Diffusers binary classification Guarnera et al.♣ (2023) [61] CNN Level up the
DeepFake detection binary classification Amoroso et al.♣ (2023) [62] CNN
COCOFake binary classification Wu et al.♣ (2023) [63] CLIP \+ MLP LSUN,
Danbooru, ProGAN, SD, BigGAN, GauGAN, styleGAN, DALLE, GLIDE, Guided
Diffusion, Latent Diffusion, ImageNet, VISION, Artist, DreamBooth, Midjourney,
NightCafe, StableAI, YiJian binary classification Xi et al.♣ (2023) [64] CNN
AI-Gen Image binary classification Lorenz et al.♣ (2023) [65] CNN \+ RF
CiFAKE, ArtiFact, DIffusionDB, LAION-5B, SAC, SD-v2.1, LSUN-Bedroom binary
classification Ju et al.♣ (2023) [66] CNN LSUN, ProGAN, $DF^{3}$ binary
classification Sinitsa et al.♣ (2023) [67] rule-based method Laion-5B, SD
v-1.4, SD v-2.1, DALL·E-Mini, GLIDE [20], DALL·E-2, MidJourney, CycleGAN,
ProGANe, ProGANt, BigGAN, StyleGAN, StyleGAN2, GauGAN, StarGAN binary
classification Guo et al.♣ (2023) [68] CNN HiFi-IFDL binary classification
Cozzolino et al.♣ (2023) [69] CLIP \+ SVM ProGAN, StyleGAN2, StyleGAN3,
StyleGAN-T, GigaGAN, (Score-SDE, ADM, GLIDE, eDiff-I, Latent and Stable
Diffusion, DiT, DeepFloyd-IF, Stable Diffusion XL, DALL·E 2, DALL·E 3,
Midjourney v5, Adobe Firefly, LSUN, FFHQ, ImageNet, COCO, LAION, RAISE binary
classification Wang et al.∙ (2023) [70] CNN DiffusionForensics binary
classification Ma et al.∙ (2023) [71] statistical-based approach, CNN CIFAR10,
TinyImageNet, CelebA binary classification Chang et al.✠ (2023) [16] vision-
language model (e.g InstructBLIP) MS COCO, Flickr30k, SD2, SDXL, DeepFloyd IF,
DALLE-2, SGXL, ControlNet, SD2-Inpainting, LaMa, SD2-SuperResolution, LTE
visual question answering Bi-LORA✠ (Ours) vision-language model (e.g BLIP2,
ViTGPT2) LSUN Bed, ADM, LDM, DDPM, IDDPM, PNDM, SD v-1.4, GLIDE image
captioning

♣ Deep feature. ∙ dms unique attribute. ✠ vlms.

Authors in [70] found that DM-generated images exhibit features that are more
easily reconstructed by pre-trained DMs than natural images. In order to
identify these features, they presented the diffusion reconstruction error
(DIRE) utilizing the reconstruction error of images using denoising diffusion
implicit models (DDIM) for inversion and reconstruction. dire mainly focuses
on the initial $x_{0}$ timestep and might miss information from intermediate
steps during diffusion and reverse diffusion. In contrast, stepwise error for
diffusion-generated image detection (SeDID) [71] was introduced a novel
detection method for diffusion-generated images. This technique leverages
diffusion patterns’ unique properties, focusing on deterministic reverse and
denoising computation errors to enhance detection accuracy. By employing the
concept of $(t,\delta\text{-error})$ with noise information during each
diffusion step, SeDID method effectively differentiates real images from
generated ones. Furthermore, in the work by Guarnera et al. [61], a multi-
level method for synthetic image detection was introduced, incorporating
meticulously designed training sets. This approach empowers the detector to
acquire comprehensive features and comprehend hierarchical attributes. The
study applied this hierarchical approach to three tasks: distinguishing real
images from those generated by AI, discerning between GANs and DMs, and
recognizing AI-specific architectures.

In their work, [62] investigated the enhancement of stylistic distinction by
decoupling semantic and stylistic features in images. The study introduces a
contrastive-based disentanglement method to analyze the role of semantics from
textual descriptions and low-level perceptual cues in synthetic image
detection. However, achieving semantic-style disentanglement in practice
proves challenging due to the need for specialized training datasets.
Additionally, authors in [65] proposed multiLID, a novel method for detecting
diffusion-generated images. This approach utilizes local intrinsic
dimensionality to estimate densities in CNN feature spaces, allowing the
distinction of models based on their internal feature distribution densities.
The process involves using an untrained ResNet18 [59] to extract low-
dimensional features from synthetic images, applying multiLID to these
features, and training a random forest classifier on the multiLID scores.

In [64], a novel approach to detect AI-generated images is introduced. The
method employs a cross-attention-enhanced dual-stream network designed to
efficiently capture various features of such images, with a specific emphasis
on texture. The network architecture consists of two streams: the residual
stream, utilizing an spatial rich model (SRM) [72] residual extraction module
to capture residual information, and the content stream, focusing on low-
frequency aspects of the images.

Furthermore, [63] proposed a distinctive synthetic image detection method
named language-guided synthesis detection (LASTED). This approach leverages
language-guided contrastive learning and a unique formulation of the detection
problem to enhance the extraction of highly discriminative representations
from limited data. lasted augments training images with carefully designed
textual labels, enabling joint image-text contrastive learning for forensic
feature extraction. Unlike traditional classification-based approaches, this
method treats synthetic image detection as an identification problem, offering
a unique and effective perspective to address the challenges in synthetic
image detection.

Figure 2: Bi-LORA finetuning for synthetic image detector.

Furthermore, a two-branch global and local feature fusion method was proposed
in [66] for the detection of AI-synthesized images. This method combines
multi-scale global features and informative local features using a ResNet50
[59] backbone network, known for its efficiency and effectiveness in various
face-related tasks. The input image is processed by ResNet50, and an
attention-based multi-scale feature fusion (AMSFF) module is employed to fuse
low-level and high-level features, creating a global representation. This
global representation is then fed into a local branch where the patch
selection module (PSM) identifies key patches with significant energy and
classification relevance. Local features are extracted from these selected
patches and integrated with the global representation using an attention-based
module for final binary classification.

In contrast to conventional rule-based techniques [73, 74, 75], Sinitsa et al.
[67] introduced a novel rule-based method that achieves high detection
accuracy, even when trained with a small set of generative images (fewer than
512). This approach leverages the inductive bias of CNNs to extract
distinctive features (fingerprints) from various generators in the training
data, enabling the detection of generative images from the same model and its
fine-tuned versions.

More recently, Chang et al. [16] drew inspiration from the zero-shot
capabilities of VLMs and proposed an approach that leverages VLMs such as
InstructBLIP. The method incorporates prompt tuning techniques to enhance
deepfake detection accuracy over unseen data. The authors reframed the
deepfake detection problem as a visual question-answering task, fine-tuning
soft prompts for InstructBLIP to distinguish whether a query image is real or
fake. On a different note, Cozzolino et al. [69] introduced a lightweight
detection strategy based on contrastive language image pre-training (CLIP)
features. These features are employed to design a linear support vector
machine (SVM) classifier, and the proposed strategy is evaluated for its
performance in challenging scenarios. Finally, Table I summarizes state-of-
the-art models proposed for the synthetic image detection problem.

##  III Synthetic Image Detection with VLMs

In this section, we present our approach for addressing synthetic image
detection using large-scale VLMs. Our approach reframes the binary
classification task into an image captioning task, leveraging cutting-edge
VLMs models (e.g., BLIP2). Fig. 2 provides a visual representation of our
methodology built around the BLIP2 architecture.

###  III-A Synthetic Image Detection

Synthetic image detection is generally a binary image classification task. Its
main objective is to develop a model $\mathcal{M}$ that learns a function
$f:\mathbf{I}\rightarrow\mathbf{Y}$ from the training set
$D=\\{({I}_{i},{y}_{i})|1\leq i\leq n\\}$, where
${I}_{i}\in\mathbf{I}=\mathbb{R}^{d\times d}$ is an image, and
$y_{i}\in\mathbf{Y}=\\{0,1\\}$ represents the class labels. Here, $0$
indicates images captured from the real world, and $1$ indicates generated
images. The model’s objective is to predict the class $\hat{y}$ for an input
image ${I}$ as follows:

$\hat{y}=f_{\theta}({I}),$ (1)

where $\hat{y}$ is the predicted class label (0 or 1), and $\theta$ represents
the model parameters.

###  III-B VLM Fine-Tuning for Synthetic Image Detection

In the previous subsection, we introduced synthetic image detection from a
general perspective and will now proceed to reframe it as an image captioning
problem, harnessing the capabilities of VLMs. Instead of treating synthetic
image detection as a traditional binary classification task, we will employ
VLMs to generate descriptive captions for each image. The VLMs will learn to
produce captions that capture the essence of the image, and these captions
will serve as indicators of the image’s authenticity. Specifically, the model
will generate captions that fall into one of two categories: ”real” or ”fake.”

Let’s define this fine-tuning process mathematically. We have a VLM
$\mathcal{M}$ with parameters $\theta$, which takes an image $I$ as input and
generates a caption $\hat{C}$. We denote this process as:

$\hat{C}=\mathcal{M}_{\theta}(I).$ (2)

The caption $\hat{C}$ is a human-like label (textual description) of the
image, and it will be used to distinguish the ”real” image from the ”fake”.

To tune the model for this task, we have a dataset $D=\\{(I_{i},C_{i})|1\leq
i\leq n\\}$, where $I_{i}$ is the $i$-th image, and $C_{i}$ is the ground
truth caption whether the image is ”real” or ”fake”. The model’s objective is
to minimize a suitable loss function $\mathcal{L}$ over this dataset:

$\theta^{*}=\arg\min_{\theta}\sum_{i=1}^{n}\mathcal{L}(\mathcal{M}_{\theta}(I_{i}),C_{i}).$
(3)

Once fine-tuned, the VLM will be capable of generating captions that help
categorizing images into the desired class, providing a more detailed and
expressive approach to synthetic image detection. In the following
subsections, we will delve into the details of the fine-tuning process of our
Bi-LORA method.

###  III-C LORA Tuning Technique

Fine-tuning large pre-trained models is a computational challenge, often
involving the adjustment of millions of parameters. This traditional tuning
approach, while efficient, requires considerable resources and computational
time, creating a bottleneck for the adaptation of these models to specific
tasks. lora [76] presents an effective solution to this problem by decomposing
the update matrix during fine-tuning, Fig. 3 (left).

In the traditional fine-tuning method, we modify the weights of a large pre-
trained model to adapt it to a new task. This adaptation involves modifying
the model’s original weight matrix $W$. Changes made to $W$ during fine-tuning
are collectively represented by $\Delta W$, so the updated weights can be
expressed as $W+\Delta W$. Instead of directly modifying $W$, the LORA
approach seeks to decompose $\Delta W$. Such decomposition is a crucial step
in reducing the computational overhead involved in fine-tuning large models.

Intrinsic rank assumption suggests that significant changes in the large pre-
trained model can be captured using a lower-dimensional representation.
Fundamentally, it asserts that not all elements of $\Delta W$ are equally
important; instead, a smaller subset of these changes can effectively capture
the necessary adjustments. Thus, based on this assumption, LORA proposes to
represent $\Delta W$ as the product of two smaller matrices, $A$ and $B$, with
a lower rank. Thus, the updated weight matrix $W^{\prime}$ becomes:

$W^{\prime}=W+\Delta W=W+BA.$ (4)

Here, $W$ remains fixed (i.e. it is not updated during training). $B$ and $A$
are low-dimensional matrices, and their product $BA$ represents a low-rank
approximation of $\Delta W$.

Choosing lower-rank matrices $A$ and $B$ via the LORA technique results in a
notable decrease in trainable parameters, providing advantages in memory
efficiency, faster training, adaptability, hardware compatibility, and
scalability for large neural networks during fine-tuning.

###  III-D LORA Tuning on Bi-LORA

Figure 3: Bi-LORA fine-tuning with LORA.

BLIP2 [7] is a method that improves vision-language alignment using a querying
transformer (Q-Former) as a bridge. q-former extracts relevant visual
information from frozen image encoders and provides it to a frozen language
model. The method undergoes two pre-training phases: one for learning visual
representations for text and another for training the Q-Former to provide
interpretable visual input to the language model. Thus, the smaller matrices
are trained to learn task-specific information using supervised learning.
Figure 3 (right) provides a visual overview of fine-tuning the BLIP2 model
using the LORA technique. During fine-tuning, we keep the pretrained BLIP2
model weights ($W$) frozen and inject a pair of trainable rank decomposition
matrices ($A$ and $B$) into each self attention layer, enabling us to optimize
and adapt these layers to our new detection task without incurring excessive
computational overhead. Thus, the smaller matrices are trained to learn task-
specific information using supervised learning. This method is commonly
referred to as LORA [76], which falls under the re-parameterization category
of parameter-efficient fine-tuning (PEFT) methods. Specifically, the update of
BLIP2 pre-trained weight matrix $W$ is constrained by a low-rank decomposition
$W+\Delta W=W+BA$. Note that both $W$ and $\Delta W=BA$ are multiplied by the
same input $x$, and their respective output vectors are summed coordinate-
wise. For $h=Wx$, the modified forward pass yields:

$h=Wx+\Delta Wx=Wx+BAx.$ (5)

It is worth highlighting that, among various attention weights, we adapt only
two types: $W_{q}$ and $W_{k}$ of the large language model (LLM) decoder
module. The first consequence of this technique is that the number of
parameters to train is really small, Table II. However, we run into a second,
maybe more important consequence, which is that because we don’t alter the
weights of the pretrained model, it doesn’t change its behavior or forget
anything it has previously learned, thus we can use it for other tasks if need
be.

Table II: Number of parameters for each Bi-LORA’s module and the number updated with LoRa Modules | Trainable | Freeze
---|---|---
Query Tokens | 0 | 24,576
Image encoder | 0 | 985,952,256
Qformer | 0 | 105,137,664
Projection Layer | 0 | 1,968,640
LLM (OPT 2.7b) | 5,242,880 | 2,651,596,800
Total | 5,242,880 | 3,744,679,936
Percentage of trainable parameters: 0.14%

##  IV Experimental Results

This section discusses the empirical results of our proposed synthetic image
detection methodology. We present an overview of the experimental setup, delve
into cross-generator evaluations, and assess the model’s resilience to
adversarial conditions.

###  IV-A Experimental Setup

Dataset. To comprehensively evaluate the performance of our proposed method in
synthetic image detection, we employed an existing dataset introduced by
Ricker et al. [3]. This dataset incorporates real images sourced from the
large-scale scene understanding (LSUN) bedroom dataset [20]. We curated a
collection of images generated by five distinct diffusion models, all trained
on LSUN bedroom dataset [20]. Among these models, four subsets of generated
images (ADM [21], DDPM [22], iDDPM [23], and PNDM [24]) are generated by
unconditional diffusion models. The fifth subset (LDM [25]) is generated by
text-to-image diffusion model. Further, for generalization evaluation to text-
to-image generation, we expanded the dataset by incorporating two additional
models: stable diffusion (SD) [25] and GLIDE [26]. The text prompt used to
generate these images was ”A photo of a bedroom”.

In our experiment, all the subsets consist of 42,000 generated images along
with their corresponding real samples from the LSUN bedroom dataset [20]. Each
subset is divided into 40,000 images for training, 1,000 for validation, and
10,000 for testing purposes. It is worth noting that the real images used for
testing is consistent across all the testing subsets. Further details
regarding the dataset are provided in the GitHub codebase, in Fig. 1, we
present a selection of representative examples.

Evaluation metrics. Following the convention of previous detection methods
[71] [60] [65], we report the accuracy (ACC) and F1 score (F1-Score) in our
experiments. Accuracy (ACC) measures the proportion of correct predictions,
with higher scores indicating better performance. While F1-Score balances
precision and recall, offering a single measure of a model’s accuracy in
identifying positive instances while minimizing false results.

Baselines. To comprehensively evaluate and compare our proposed approach, we
tailored several prominent models, namely ResNet50 [17], Xception [18], DeiT
[19], and ViTGPT21 trained on our dataset, to establish baseline benchmarks.
We opted for these models due to their extensive usage and outstanding
performance in related tasks. To formulate our baseline models, we fine-tuned
these architectures by substituting their final fully-connected (FC) layers
with a novel FC layer featuring a single neuron dedicated to discerning the
authenticity of images. These models were initialized with pre-trained weights
gleaned from the ImageNet dataset [77], thereby harnessing the knowledge
encoded in their learned representations. Employing these custom-tailored
baselines enables us to assess the efficacy of our proposed approach when
compared with well-established and widely-adopted image classification
architectures.

Implementation details. In our experiments, we leveraged the PyTorch deep
learning framework on a Windows 11 Pro computer equipped with an NVIDIA RTX
A4500 GPU with 16 GB and an Intel(R) i9-12950HX CPU. Our study used baseline
models obtained from their publicly available repositories, which we fine-
tuned to align with our specific experimental setup. For Bi-LORA model tuning,
we opted for the Adam [78] optimizer with default settings, a learning rate of
5e-5, and training epoch of 20 epochs. During training, we configured the PEFT
module, in particular LORA, with the followings parameters: a rank of 16,
lora_alpha set to 32, lora_dropout at 0.05, and a batch_size of 32. Regarding
ViTGPT2 model, we followed a procedure outlined by the author, as detailed in
provided note1. To ensure a fair comparison, we retrain all baselines on our
training dataset, instead of using their published versions directly.

###  IV-B Cross-Generator Synthetic Image Detection

Table III: Results of cross-validation on different training and testing
subsets using Bi-LORA. We report ACC (%) / F1-Score (%) values.

Training Subset Testing Subset Average (in %) LDM⋆ ADM⊕ DDPM⊕ IDDPM⊕ PNDM⊕ SD⋆
GLIDE⋆ LDM 99.12 / 99.13 85.24 / 82.97 98.47 / 98.47 97.02 / 96.97 99.22 /
99.23 77.68 / 71.79 97.09 / 97.05 93.41 / 92.23 ADM 95.91 / 95.91 96.51 /
96.57 97.24 / 97.31 97.35 / 97.42 96.16 / 96.21 48.32 / 3.35 70.99 / 61.90
86.07 / 78.38 DDPM 97.94 / 97.94 90.49 / 89.76 98.74 / 98.76 98.52 / 98.53
98.64 / 98.66 50.70 / 7.19 74.22 / 66.36 87.04 / 79.60 IDDPM 97.06 / 97.03
94.16 / 93.94 98.72 / 98.74 98.78 / 98.79 98.02 / 98.03 49.58 / 2.74 62.31 /
41.63 85.52 / 75.84 PNDM 96.66 / 96.58 73.82 / 64.96 96.9 / 96.84 91.82 /
91.19 99.52 / 99.53 85.37 / 83.05 97.93 / 97.91 91.72 / 90.01 SD 50.14 / 0.58
50.00/ 0.00 50.00 / 0.00 50.00 / 0.00 51.74 / 6.73 100.00 / 100.00 99.74 /
99.73 64.52 / 29.58 GLIDE 50.72 / 2.88 50.00 / 0.00 50.03 / 0.14 50.0 / 0.02
58.38 / 28.71 93.56 / 93.12 100.00 / 100.00 64.67 / 32.12

⋆ Text-To-Image diffusion-based model. ⊕ Unconditional diffusion-based model.

Table III presents the performance metrics of the proposed Bi-LORA model. It
was trained on the training set associated with seven generative models and
evaluated on their corresponding testing sets. In the table, the diagonal
entries reflect the detector’s performance when trained and tested on images
generated by the same generator. We observe that the detector achieves an
accuracy exceeding 96.51% when both training and testing are conducted on
images generated by the same model. Significantly, the subsets SD (SD V-1.4)
and Glide demonstrate precision rates of 100%. Nevertheless, a notable decline
in performance is observed when the model is trained and tested with data
generated by different generators. For example, when Bi-LORA is trained on the
ablated diffusion model (ADM) and improved denoising diffusion probabilistic
models (IDDPM) subsets and subsequently tested on stable diffusion (SD V-1.4),
the precision diminishes to 48.32% and 49.58%, respectively. Conversely, the
training subsets of SD (SD V-1.4) and Glide exhibit poor overall
generalization. These two models yield an accuracy of approximately 50% across
all unconditional diffusion-based models and LDM, indicating their inefficacy
in distinguishing AI-generated images from real ones. This inefficiency is
further evident in the extremely low F1-Scores of these subsets, suggesting
challenges in identifying AI-generated images by unconditional diffusion-based
models and LDM. Furthermore, the experiment highlights LDM and pseudo
numerical methods for diffusion models on manifolds (PNDM) as the most
influential contributors, while SD (SD V-1.4) and Glide exhibit limitations in
performance, especially within our bedroom dataset. Moreover, it is noteworthy
that when the Bi-LORA model is trained on an unconditional diffusion-based
model, it demonstrates robust generalization to other generative models
compared to when trained on a text-to-image model, specifically SD (SD V-1.4)
and Glide.

Table IV: Results of different methods trained on LDM and evaluated on
different testing subsets. We report ACC (%) / F1-Score (%).

Method Testing Subset Average (in %) LDM⋆ ADM⊕ DDPM⊕ IDDPM⊕ PNDM⊕ SD v1.4⋆
GLIDE⋆ ResNet50 99.92 / 99.92 72.33 / 61.83 75.26 / 67.21 88.96 / 87.61 77.20
/ 70.52 75.47 / 67.57 73.10 / 63.28 80.32 / 73.99 Xception 99.96 / 99.96 52.05
/ 7.98 58.60 / 29.41 54.62 / 16.99 60.01 / 33.43 63.84 / 43.41 58.92 / 30.35
64.00 / 37.36 DeiT 99.83 / 99.83 50.40 / 2.01 50.18 / 1.17 50.14 / 1.01 56.25
/ 22.54 96.02 / 95.86 98.15 / 98.11 71.56 / 45.79 ViTGPT2 99.40 / 99.40 70.84
/ 59.21 69.60 / 56.72 84.08 / 81.20 95.40 / 95.22 99.54 / 99.55 99.27 / 99.27
88.30 / 84.37 Bi-LORA 99.12 / 99.13 85.24 / 82.97 98.47 / 98.47 97.02 / 96.97
99.22 / 99.23 77.68 / 71.79 97.09 / 97.05 93.41 / 92.23

⋆ Text-To-Image diffusion-based model. ⊕ Unconditional diffusion-based model.

Table V:  Results of cross-validation on different training and test subsets
using different models. Five models trained on seven generators are tested on
one generator, and their average accuracy is each data point in the testing
subset column. We report ACC (%) / F1-Score (%).

Method Testing Subset Average (in %) LDM⋆ ADM⊕ DDPM⊕ IDDPM⊕ PNDM⊕ SD v1.4⋆
GLIDE⋆ ResNet50 75.72 / 54.28 69.34 / 43.50 61.23 / 25.83 76.83 / 55.22 61.87
/ 27.58 77.18 / 60.40 82.17 / 68.88 72.04 / 47.96 Xception 77.16 / 58.65 69.50
/ 41.73 72.13 / 46.72 71.80 / 45.16 64.66 / 37.00 74.09 / 54.21 85.58 / 79.23
73.56 / 51.82 DeiT 71.17 / 48.49 69.52 / 41.05 57.29 / 15.98 70.94 / 42.56
62.37 / 31.07 72.79 / 55.70 86.93 / 82.63 70.14 / 45.35 ViTGPT2 77.43 / 60.86
67.92 / 44.73 74.57 / 56.54 78.17 / 61.70 84.10 / 69.84 76.80 / 60.89 89.35 /
87.38 78.33 / 63.13 Bi-LORA 83.94 / 70.01 77.17 / 61.17 84.30 / 70.04 83.35 /
68.99 85.95 / 75.30 72.17 / 51.61 86.04 / 80.65 81.85 / 68.25

⋆ Text-To-Image diffusion-based model. ⊕ Unconditional diffusion-based model.

In light of this empirical observation, discerning fake generated images from
a specific generator may seem like a straightforward task, achieved by
training a detector on a dataset containing both authentic and synthesized
images. However, it is crucial to acknowledge that this method may have
limitations when confronted with generators of unknown provenance. In
practical scenarios, the identity of the generator often remains concealed
during the training phase. In the context of our research, we aim to
rigorously evaluate the generalization prowess of our Bi-LORA, i.e., its
ability to distinguish authentic from synthetic images, regardless of the
underlying generator. To this end, we introduce the cross-generator synthetic
image detection experiment as an effective means of evaluating the detection
capability of the AI-generating detector.

From Table III, it is evident that the most favorable outcomes are achieved
through training on the LDM. Consequently, we conduct training of the baseline
models on LDM and evaluate their performance on test subsets from various
generators. The table IV also reports the average accuracy across these
diverse test subsets.

In all test subsets, the assessed models exhibited varying levels of accuracy
in detecting synthetic images. ResNet50 [59] consistently demonstrated high
accuracy, ranging from 72.33% to 88.96%, across all tested subsets.
Conversely, Xception [18] exhibited relatively lower performance, with
accuracy ranging from 52.05% to 63.84%. deit showed an intriguing pattern,
achieving an accuracy of 96.02% on the SD v1.4 subset, while its accuracy was
notably lower on other subsets, suggesting potential challenges in specific
synthetic image detection scenarios. Specifically, DeiT performed well in the
context of text-to-image generative models (i.e., LDM, SD v1.4, GLIDE),
indicating robust generalization to unseen models from the same subcategory
used during training. ViTGPT2 delivered competitive results, with accuracy
rates ranging from 69.60% to an impressive 99.54%. Notably, Bi-LORA emerged as
a standout performer, showcasing outstanding accuracy rates across all
subsets, ranging from 77.68% to 99.22%. This underscores its robustness in
detecting various types of synthetic content.

In Table V, for each model, we have trained seven versions, one for each of
the seven generators. We evaluated each version on seven generator test
subsets. We calculated the average over seven results for a given model. For
example, we respectively trained seven Bi-LORA models using seven generators
and calculated the average of their evaluation results on LDM, resulting in
the highest accuracy of 83.94%. Our experiment of detecting synthetic images
across different generators reveals valuable information about the robustness
and limitations of the models evaluated. While some models show solid
performance when trained and tested on images from the same generator,
challenges emerge in scenarios where identity of the generator is unknown
during training. Notably, Bi-LORA stands out as an outstanding performer,
demonstrating remarkable accuracy across various generator models.

###  IV-C Degraded Synthetic Image Detection

(a) Real - $0|0|0|0|0$

(b) Fake (ADM) - $|0|0|0|0$

(c) Fake (LDM) - $|0|0|0|1$

(d) Fake (DDPM) - $|0|0|0|1$

(e) Fake (IDDPM) - $|0|0|0|1$

(f) Fake (PNDM) - $|0|0|0|1$

(g) Fake (SD) - $|0|1|1|1$

(h) Fake (GLIDE) - $|0|0|0|1$

Figure 4: Each sub-figure represents a random compressed (JPEG q=65) image
from a testing set, with predicted labels provided below. The 5-digit binary
code shows results from ResNet, Xception, DeiT, ViTGPT2, and Bi-LORA models,
where ’0’ means real and ’1’ means fake. It is important to note that all
generated images are considered fake.

In the context of image transmission, images often undergo degradation
artifacts due to downscaling to low resolution, compression, and noise
interference. Detectors are expected to demonstrate robustness to these
distortions. To address this concern, we propose evaluating detectors’
performance on degraded images, which more accurately simulate practical
conditions, as outlined in Table VI. After training the detectors on the LDM
subset, we employ various methods to degrade only the test set images.
Specifically, we reduce the image resolution to 112 pixels, employing a
scaling factor of $2X$. Additionally, we apply JPEG compression with quality
settings of 65, introducing compression artifacts. To simulate blurring
effects, we apply Gaussian blur with a standard deviation of 3, denoted as
$\sigma=3$.

Consequently, the development of a well-thought-out pre-processing approach
holds great promise in addressing the challenge of detecting degraded images.
Assessing these detectors on degraded images provides valuable insights into
their performance under various challenging conditions. This evaluation
contributes to a more comprehensive understanding of the detector’s capability
to handle practical image degradation.

Table VI: Models evaluation on degraded images. LR denotes Low Resolution, q
denotes quality.

Method ResNet50 Xception DeiT ViTGPT2 Bi-LORA Testing Subset LDM LR (112)
82.56 / 81.93 61.40 / 67.12 98.42 / 98.44 93.16 / 93.6 95.65 / 95.77 JPEG
(q=65) 95.79 / 95.96 50.11 / 0.44 50.00 / 0.00 54.58 / 16.95 90.77 / 89.85
Blur ($\sigma$= 3) 51.36 / 5.57 69.56 / 73.61 58.88 / 70.77 76.65 / 80.83 85\.
36 / 87.08 ADM LR (112) 68.74 / 62.22 56.77 / 61.65 53.99 / 18.78 85.17 /
84.99 89.14 / 88.74 JPEG (q=65) 73.37 / 67.41 50.02 / 0.08 50.00 / 0.00 50.6 /
2.56 69.38 / 56.00 Blur ($\sigma$= 3) 50.38 / 1.76 63.20 / 66.24 57.20 / 69.21
68.58 / 72.37 80.53 / 82.07 DDPM LR (112) 76.98 / 74.69 58.62 / 63.89 54.36 /
19.96 88.93 / 89.20 96.16 / 96.28 JPEG (q=65) 90.5 / 90.39 50.02 / 0.06 50.04
/ 0.14 52.33 / 9.08 93.16 / 92.67 Blur ($\sigma$= 3) 51.50 / 6.08 65.82 /
69.37 56.28 / 68.33 74.24 / 78.42 85.60 / 87.34 IDDPM LR (112) 78.05 / 76.15
61.57 / 67.30 52.52 / 13.94 90.66 / 91.05 95.88 / 96.01 JPEG (q=65) 87.77 /
87.27 50.04 / 0.18 50.01 / 0.04 52.00 / 7.89 88.16 / 86.60 Blur ($\sigma$= 3)
51.10 / 4.55 67.58 / 71.41 56.74 / 68.78 73.91 / 78.08 85.37 / 87.07 PNDM LR
(112) 78.92 / 77.32 57.03 / 61.96 68.06 / 54.84 92.76 / 93.19 96.27 / 96.39
JPEG (q=65) 89.99 / 89.82 50.02 / 0.10 50.00 / 0.00 53.44 / 13.05 97.0 / 96.91
Blur ($\sigma$= 3) 51.15 / 4.78 69.4 / 73.43 58.30 / 70.24 76.28 / 80.46 85.76
/ 87.49 SD v1.4 LR (112) 72.80 / 68.66 61.68 / 67.42 93.25 / 92.96 93.18 /
93.61 88.10 / 87.49 JPEG (q=65) 62.72 / 47.53 74.40 / 65.59 95.72 / 95.53
85.06 / 82.45 86.26 / 84.10 Blur ($\sigma$= 3) 51.94 / 7.72 77.09 / 81.36
54.12 / 66.24 76.87 / 81.04 71.14 / 70.87 GLIDE LR (112) 79.36 / 77.89 66.84 /
73.00 96.88 / 96.87 93.06 / 93.49 93.98 / 94.04 JPEG (q=65) 89.09 / 88.80
50.90 / 3.52 50.00 / 0.00 57.03 / 24.79 98.39 / 98.37 Blur ($\sigma$= 3) 50.94
/ 3.95 69.57 / 73.62 59.01 / 70.89 72.31 / 76.42 83.06 / 84.76

Table VI presents the evaluation results for different models on degraded
images. ResNet50 demonstrates varying performance across different degradation
methods, notably excelling under JPEG compression, due to being trained on
data that has been compressed with settings ranging from 60 to 100.
Conversely, Bi-LORA exhibits robust performance across all degradation
methods, highlighting its effectiveness in handling challenges associated with
detecting degraded synthetic images These results underscore the significance
of evaluating detectors under challenging conditions to assess their practical
suitability in real-world scenarios and emphasize the necessity for models
that can reliably perform in the presence of image degradation.

Table VII: Evaluation of seven trained Bi-LORA models on five GAN-based testing subsets. We report ACC (%) / F1-Score (%). Training Subset | Testing Subset |  Average (in %)
---|---|---
StyleGAN | Diff-STyleGAN2 | Diff-ProjectedGAN | ProGAN | ProjectedGAN
LDM | 99.09 / 99.09 | 99.10 / 99.10 | 99.08 / 99.08 | 99.26 / 99.26 | 99.23 / 99.23 | 99.15 / 99.15
ADM | 85.92 / 84.54 | 90.93 / 90.56 | 96.56 / 96.74 | 97.30 / 97.37 | 96.52 / 96.58 | 93.47 / 93.16
DDPM | 93.80 / 93.56 | 96.24 / 96.19 | 98.31 / 98.32 | 98.80 / 98.81 | 98.43 / 98.45 | 97.12 / 97.07
IDDPM | 89.67 / 88.76 | 93.64 / 93.37 | 97.78 / 97.78 | 98.74 / 98.75 | 97.70 / 97.70 | 95.51 / 95.27
PNDM | 99.15 / 99.15 | 99.44 / 99.45 | 98.60 / 98.59 | 99.56 / 99.57 | 99.13 / 99.13 | 99.18 / 99.18
SD | 66.06 / 48.62 | 76.40 / 69.11 | 51.12 / 4.38 | 63.91 / 43.53 | 51.43 / 5.56 | 61.78 / 34.24
GLIDE | 65.36 / 47.02 | 83.45 / 80.17 | 57.42 / 25.86 | 84.42 / 81.54 | 59.81 / 32.82 | 70.09 / 53.48
Average (in %) | 85.59 / 80.11 | 91.31 / 89.71 | 85.55 / 74.39 | 91.71 / 88.40 | 86.04 / 75.64 | 88.04 / 81.65

Fig. 4 presents visual examples of randomly selected compressed (JPEG q=65)
images from testing sets, with predicted labels below each image. The visual
examples highlight the challenges posed by image degradation on detection
models. In some cases, the models exhibit accurate predictions even in the
presence of degradation, while in others, degradation introduces ambiguity,
leading to misclassifications. Once again, Bi-LORA consistently demonstrates
robust performance across various degradation methods, showcasing its
effectiveness in handling challenges related to detecting degraded synthetic
images. These visual examples complement the quantitative results presented in
Table VI, providing a more intuitive understanding of how different models
respond to degraded images. The variability in performance across models and
degradation methods underscores the importance of comprehensive evaluations
under realistic conditions to assess the practical suitability of image
detection models.

###  IV-D Generalization to GAN models

In this section, our aim is to investigate whether our Bi-LORA approach,
primarily tuned for detecting images generated by diffusion-based models, can
effectively generalize to GAN-generated images without the need for additional
fine-tuning on GAN-specific data. One of the key objectives in assessing the
generalization of our Bi-LORA approach to GAN-generated images is to explore
the transferability of learned features. Our Bi-LORA approach has acquired
knowledge about common features and artifacts present in AI-generated images,
particularly those created using diffusion-based models. We hypothesize that
some of these features may be transferable and applicable to GAN-generated
images. To conduct this analysis, we curated a dataset consisting of various
models based on GAN, including Diff-StyleGAN2 [79], Diff-ProjectedGAN [79],
ProGAN [30], ProjectedGAN [80], and StyleGAN [31]. Each of these subsets
comprises 10,000 generated images along with 10,000 real samples. Our approach
involves using this dataset to evaluate the performance of our fine-tuned Bi-
LORA approach in distinguishing images generated by GAN from real images.

In Table VII, we present the performance evaluation of seven trained Bi-LORA
models across five testing subsets based on GANs. Notably, models such as LDM,
ADM, denoising diffusion probabilistic models (DDPM), IDDPM, and PNDM
consistently exhibit high accuracy and F1-scores. This suggests that our Bi-
LORA approach trained on diffusion-based models generalize effectively to
images generated by GANs. The remarkable consistency in performance across
different GAN models indicates that Bi-LORA have successfully learned features
and patterns that are transferable between diffusion-based and GAN-based
generative models.

However, StyleGAN presents a challenge, particularly for Bi-LORA model trained
on SD and GLIDE generators, where both accuracy and F1-score are considerably
lower compared to other GAN models. This discrepancy may be attributed to the
unique architecture and characteristics of StyleGAN, requiring specific
features that were not as prominent in diffusion-based models. Bi-LORA trained
on different generators exhibits varied performance on GAN models. For
instance, PNDM consistently performs well across all GAN subsets, while SD and
GLIDE show lower performance, indicating differences in feature importance and
generalization capabilities among these models.

The findings suggest that our Bi-LORA approach trained on diffusion-based
models can generalize to GAN-generated images to a considerable extent. This
has implications for applications where detecting AI-generated content is
crucial, such as content moderation and deepfake detection.

###  IV-E Comparative Analysis of Synthetic Image Detection Models

In this section, we perform a comparative analysis of Bi-LORA against four
recent synthetic image detection models : AntifakePrompt (Chang et al., 2023)
[16], DE-FAKE (Sha et al., 2022) [58], UniversalFakeDetect (Ojha et al., 2023)
[81], CNNDetection (Wang et al., 2020) [82]. For AntifakePrompt, we use
checkpoints of detector trained on MS COCO vs. SD2 and MS COCO vs. SD2+LaMa.
For DE-FAKE, we employ the checkpoint of the hybrid detector, which considers
both the image and the corresponding prompts in inference time. For
CNNDetection and UniversalFakeDetect, we use the detectors checkpoint trained
on a dataset containing images that may have undergone Gaussian blurring and
JPEG augmentation, each with a probability of 10%.

Table VIII: Comparaison with SOTA models

Methods Training Set [HTML]CCFFCCLSUN Bed [HTML]feb3b1ADM [HTML]feb3b1DDPM
[HTML]feb3b1IDDPM [HTML]feb3b1PNDM [HTML]feb3b1LDM [HTML]feb3b1SD v1.4
[HTML]feb3b1GLIDE MS COCO vs. SD2 [HTML]CCFFCC92.86 [HTML]feb3b129.76
[HTML]feb3b157.05 [HTML]feb3b147.99 [HTML]feb3b176.79 [HTML]feb3b157.21
[HTML]feb3b15.80 [HTML]feb3b198.86 AntifakePrompt MS COCO vs. SD2+LaMa
[HTML]CCFFCC65.58 [HTML]feb3b173.18 [HTML]feb3b187.59 [HTML]feb3b182.33
[HTML]feb3b192.31 [HTML]feb3b188.56 [HTML]feb3b16.80 [HTML]feb3b199.62 DE-FAKE
MS COCO vs. SD2 [HTML]CCFFCC27.87 [HTML]feb3b142.70 [HTML]feb3b155.95
[HTML]feb3b131.69 [HTML]feb3b144.24 [HTML]feb3b153.43 [HTML]feb3b180.95
[HTML]feb3b149.90 UniversalFakeDetect LSUN vs. ProGAN [HTML]CCFFCC99.80
[HTML]feb3b106.80 [HTML]feb3b136.10 [HTML]feb3b124.70 [HTML]feb3b147.50
[HTML]feb3b134.10 [HTML]feb3b121.70 [HTML]feb3b112.00 CNNDetection LSUN vs.
ProGAN [HTML]CCFFCC99.80 [HTML]feb3b100.90 [HTML]feb3b101.67 [HTML]feb3b101.46
[HTML]feb3b101.46 [HTML]feb3b101.30 [HTML]feb3b102.62 [HTML]feb3b107.92 Bi-
LORA (Ours) LSUN Bed vs. LDM [HTML]CCFFCC98.55 [HTML]feb3b171.93
[HTML]feb3b198.39 [HTML]feb3b195.49 [HTML]feb3b199.89 [HTML]feb3b199.70
[HTML]feb3b156.81 [HTML]feb3b195.63

Table IX: General Image Datasets

Dataset #Real #Generated [HTML]CCFFCCSource of Real Image
[HTML]feb3b1Generation Method Year Synthbuster - 9,000 [HTML]CCFFCCRaise-1k
[HTML]feb3b1DALL-E2&3, Midjourney, SDMs, GLIDE, Firefly 2023 GenImage 1,
331,167 1,350,000 [HTML]CCFFCCImagNet [HTML]feb3b1SDMs, Midjourney, BigGAN
2023 CIFAKE 60,000 60,000 [HTML]CCFFCCCIFAR-10 [HTML]feb3b1SD v1.4 2023
AutoSplice 2,273 3,621 [HTML]CCFFCCVisual News [HTML]feb3b1DALL-E 2 2023
DiffusionDB 3,300,000 16,000,000 [HTML]CCFFCCDiscordChatExporter
[HTML]feb3b1SD 2023 ArtiFact 964,989 1,531,749 [HTML]CCFFCCCOCO, FFHQ, LSUN
[HTML]feb3b1SDMs, DDPM, LDM, CIPS 2023 HiFi-IFDL ~600,000 1,300,000
[HTML]CCFFCCFFHQ, COCO, LSUN [HTML]feb3b1DDPM, GLIDE, LDM, GANs 2023
DiffForensics 232,000 232,000 [HTML]CCFFCCLSUN, ImageNet [HTML]feb3b1LDM,
DDPM, VQDM, ADM 2023 CocoGlide 512 512 [HTML]CCFFCCCOCO [HTML]feb3b1GLIDE 2023
LSUNDB 420,000 510,000 [HTML]CCFFCCLSUN [HTML]feb3b1  LDM, ADM, DDPM, IDDPM,
PNDM, StyleGAN, ProGAN, Diff-ProjectedGAN, ProjectedGAN, Diff-StyleGAN2 2023
UniFake 8,000 8,000 [HTML]CCFFCCLAIION-400M [HTML]feb3b1LDM, GLIDE 2023 REGM -
116,000 [HTML]CCFFCCCelebA, LSUN [HTML]feb3b1116 publicly available GMs 2023
DMimage 200,000 200,000 [HTML]CCFFCCCOOC, LSUN [HTML]feb3b1LDM 2022 AIGCD
360,000 508,500 [HTML]CCFFCCLSUN, COCO, FFHQ [HTML]feb3b1SDMs, GANs, ADM,
DALL·E 2, GLIDE 2023 DIF 84,300 84,300 [HTML]CCFFCCLAION-5B [HTML]feb3b1SDMs,
DALL·E 2, GLIDE, GANs 2023 Fake2M - 2,300,000 [HTML]CCFFCCCC3M
[HTML]feb3b1SD-V1.5, IF, StyleGAN3 2023

Table VIII compares the performance of four state-of-the-art (SOTA) models on
the task of synthetic image detection. It shows the average accuracy of each
model on various test datasets (ADM,DDPM etc.) when trained on different real
vs. fake image set combinations (e.g., MS COCO vs. SD2+LaMa). The performance
of all models varies depending on the training data used. For example,
UniversalFakeDetect and CNNDetection perform exceptionally well on the LSUN
Bed test set, but perform poorly on other testing sets. In addition, some test
data sets seem to be more challenging than others. For instance,
AntifakePrompt, Bi-LORA, CNNDetection and UniversalFakeDetect struggle on the
SD v1.4 dataset, suggesting that these models may have difficulty detecting
certain types of synthetic images.

AntifakePrompt achieves the highest accuracy on the GLIDE and ADM test set.
However, performance drops significantly on the datasets generated by the
other models when trained on MS COCO vs. SD2. In contrast, it generalizes
quite well on the test sets when trained on the MS COCO vs. SD2+LaMa
combination.

Regarding the DE-FAKE detector, trained on images generated by SD and MS COCO,
it achieves the highest accuracy on the SD v1.4 test set, however it suffers
from drops in accuracy when applied to images generated by unseen generative
models. Although the model considers corresponding prompts, potentially
helping to detect unusual synthetic scenarios, its effectiveness is limited
because most of the training data uses natural prompts from the MS COCO
dataset. This biases the model towards identifying false images, making it
less effective against fakes generated with different prompts.

UniversalFakeDetect and CNNDetection, both trained on the LSUN vs ProGAN
dataset, show outstanding performance (near-perfect accuracy) on LSUN Bed.
Meanwhile, their performance suffers significantly on all other datasets,
highlighting a major weakness in terms of generalizability and dependence on a
particular training data configuration.

Bi-LORA achieves good accuracy on the majority of test sets (Lsun Bed, DDPM,
IDDPM, PNDM, LDM) except SD v1.4, suggesting that Bi-LORA is effective in
generalizing to different types of synthetic images.

###  IV-F Evaluation on Diverse Datasets

The performance of deep learning models in real-world applications depends not
only on their effectiveness in the domains on which they were initially
trained, but also on their ability to generalize and adapt to diverse and new
data sources. In the context of image authenticity assessment, where it is
crucial to distinguish synthetic from real-world images, evaluation on diverse
datasets becomes an essential aspect of model validation. In this section, we
dive into a comprehensive exploration of the generalizability and robustness
of our proposed approach.

Our vision language model, preliminarily fine-tuned on the LSUN bedroom
dataset, seems initially specialized in a specific task: distinguishing
synthetic bedroom scenes from real ones. While this specialization has yielded
promising results in its niche, the wider landscape of visual data extends far
beyond the confines of bedrooms, Table. IX . In real-world applications, the
need to detect image authenticity often arises in scenarios where images
encompass a myriad of content, with diverse objects, scenes, and contexts. It
is therefore imperative to evaluate the performance of our model when applied
to more general datasets.

Table X: Performance comparison of Bi-LORA trained on a general dataset vs.
AntifakePrompt. Experiments are conducted on 2 real and 14 fake datasets,
including 3 attacked ones.

Methods Training Set No. of param MS COCO Flickr SD2 SDXL IF DALLE-2
AntifakePrompt MS COCO vs. SD2 4.86K 95.37 91.00 97.83 97.27 89.73 99.57 MS
COCO vs. SD2+LaMa 4.86K 90.83 81.04 97.10 97.10 88.37 99.07 Bi-LORA MS COCO
vs. SD2 5.24M 92.10 90.33 99.40 98.30 90.60 99.23 MS COCO vs. SD2+LaMa 5.24M
80.30 72.53 99.47 99.37 95.03 99.67 Methods Training Set No. of param SGXL
ControlNet Inpainting Super Res. LaMa SD2 LTE SD2 AntifakePrompt MS COCO vs.
SD2 4.86K 99.97 91.47 39.03 85.20 99.90 99.93 MS COCO vs. SD2+LaMa 4.86K 99.93
93.27 58.53 90.70 100.00 99.97 Bi-LORA MS COCO vs. SD2 5.24M 96.80 96.87 46.47
92.30 98.33 99.73 MS COCO vs. SD2+LaMa 5.24M 98.50 98.90 80.07 94.13 99.67
100.00 Methods Training Set No. of param Deeper- Forensics Adver. Attack
Average Backdoor Data Poisoning AntifakePrompt MS COCO vs. SD2 4.86K 97.90
96.70 93.00 91.57 91.59 MS COCO vs. SD2+LaMa 4.86K 97.77 97.20 97.10 93.63
92.60 Bi-LORA MS COCO vs. SD2 5.24M 81.67 34.37 47.33 33.00 81.05 MS COCO vs.
SD2+LaMa 5.24M 87.37 88.50 89.73 74.23 91.09

To this end, we are conducting a comprehensive evaluation of our approach on
another distinct dataset, presenting different challenges and characteristics.
The data set selected for this evaluation has been chosen to represent varying
degrees of divergence from the LSUN bedroom dataset, ensuring a rigorous
examination of the versatility and adaptability of our approach.

Table X presents the performance of two leading methods, AntifakePrompt and
Bi-LORA, on a variety of data sets. These datasets encompass a variety of
domains, including image captions (MS COCO), social media (Flickr), synthetic
images from text-image generators (SD2, SDXL, IF, DALLE-2, SGXL), image
stylization (ControlNet), image inpainting (LaMa, SD2-Inpainting), super-
resolution (LTE, SD2-SuperResolution), as well as deep forensic analysis and
attack scenarios (Deeper Forensics, Adversarial (Adver.), backdoor attacks,
and data poisoning).

AntifakePrompt and Bi-LORA were trained on different sets of data and
evaluated on these diverse data sets. The results highlight their
effectiveness in the synthetic image detection task. Notably, both methods
achieve competitive performance on most data sets, demonstrating their
adaptability in real-world applications. Analyzing the table, it’s evident
that AntifakePrompt consistently achieves high accuracy across several data
sets, particularly when trained on MS COCO vs SD2+LaMa. This indicates its
robustness in handling diverse data distributions and adversarial scenarios.
On the other hand, Bi-LORA exhibits slightly lower performance in some cases,
but still maintains strong results across various data sets.

Moreover, the comparison between training on different datasets (e.g., MS COCO
vs. SD2) provides valuable insights into the impact of training data
composition on model performance. For example, training on a combination of MS
COCO and SD2+LaMa data leads to improved performance in most cases,
highlighting the importance of incorporating diverse training data to improve
model robustness.

##  V Conclusion

In this paper, we introduced Bi-LORA, a novel approach to synthetic image
detection in response to advancements in realistic image generation. We
reconceptualized binary classification as an image captioning task, leveraging
the powerful convergence between vision and language, as well as the zero-shot
nature of VLMs. Obtained results demonstrated a remarkable average accuracy of
93.41% in the detection of synthetic image, underling the relevance and
effectiveness of Bi-LORA approach to the challenges posed by images generated
by unknown generative models. Moreover, unlike previous studies that require
millions of parameters to be tuned/learned, Bi-LORA model only needs to
adjusts far fewer parameters, thus setting a better balance between training
cost and efficiency.

Future research could be conducted on the possibility of distilling the
mixture of knowledge from very large VLMs to a lightweight model, given that
our experimental results highlight the complementarity between Bi-LORA and
VitGPT2, considering that VLMs require significant computational and memory
resources. Moreover, we aim to take a step further beyond just detecting
synthetic images from real ones by attributing them to their source generator
using VLMs in a multitask single-step approach, thereby holding generators
accountable for their actions in today’s rapidly evolving technological
landscape.

To support the principle of reproducible research and to enable future
extensions, we make the code and models publicly available at
https://github.com/Mamadou-Keita/VLM-DETECT.

## Acknowledgments

This work has been funded by the project PCI2022-134990-2 (MARTINI) of the
CHISTERA IV Cofund 2021 program.

## References

  * [1] Z. Lu, D. Huang, L. Bai, X. Liu, J. Qu, and W. Ouyang, “Seeing is not always believing: A quantitative study on human perception of ai-generated images,” _arXiv preprint arXiv:2304.13023_ , 2023.
  * [2] D. P. Kingma and M. Welling, “Auto-encoding variational bayes,” _arXiv preprint arXiv:1312.6114_ , 2013.
  * [3] J. Ricker, S. Damm, T. Holz, and A. Fischer, “Towards the detection of diffusion model deepfakes,” _arXiv preprint arXiv:2210.14571_ , 2022.
  * [4] C. Schuhmann, R. Vencu, R. Beaumont, R. Kaczmarczyk, C. Mullis, A. Katta, T. Coombes, J. Jitsev, and A. Komatsuzaki, “Laion-400m: Open dataset of clip-filtered 400 million image-text pairs,” _arXiv preprint arXiv:2111.02114_ , 2021.
  * [5] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark _et al._ , “Learning transferable visual models from natural language supervision,” in _International conference on machine learning_. PMLR, 2021, pp. 8748–8763.
  * [6] J.-B. Alayrac, J. Donahue, P. Luc, A. Miech, I. Barr, Y. Hasson, K. Lenc, A. Mensch, K. Millican, M. Reynolds _et al._ , “Flamingo: a visual language model for few-shot learning,” _Advances in Neural Information Processing Systems_ , vol. 35, pp. 23 716–23 736, 2022.
  * [7] J. Li, D. Li, S. Savarese, and S. Hoi, “Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models,” _arXiv preprint arXiv:2301.12597_ , 2023.
  * [8] A. Singh, R. Hu, V. Goswami, G. Couairon, W. Galuba, M. Rohrbach, and D. Kiela, “Flava: A foundational language and vision alignment model,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2022, pp. 15 638–15 650.
  * [9] H. Liu, C. Li, Q. Wu, and Y. J. Lee, “Visual instruction tuning,” _arXiv preprint arXiv:2304.08485_ , 2023.
  * [10] Z. Wang, J. Yu, A. W. Yu, Z. Dai, Y. Tsvetkov, and Y. Cao, “Simvlm: Simple visual language model pretraining with weak supervision,” _arXiv preprint arXiv:2108.10904_ , 2021.
  * [11] L. H. Li, M. Yatskar, D. Yin, C.-J. Hsieh, and K.-W. Chang, “Visualbert: A simple and performant baseline for vision and language,” _arXiv preprint arXiv:1908.03557_ , 2019.
  * [12] W. Dai, J. Li, D. Li, A. Tiong, J. Zhao, W. Wang, B. Li, P. Fung, and S. Hoi, “Instructblip: Towards general-purpose vision-language models with instruction tuning. arxiv 2023,” _arXiv preprint arXiv:2305.06500_ , 2023.
  * [13] S. Antol, A. Agrawal, J. Lu, M. Mitchell, D. Batra, C. L. Zitnick, and D. Parikh, “Vqa: Visual question answering,” in _Proceedings of the IEEE international conference on computer vision_ , 2015, pp. 2425–2433.
  * [14] O. Vinyals, A. Toshev, S. Bengio, and D. Erhan, “Show and tell: A neural image caption generator,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2015, pp. 3156–3164.
  * [15] K. O’Shea and R. Nash, “An introduction to convolutional neural networks,” _arXiv preprint arXiv:1511.08458_ , 2015.
  * [16] Y.-M. Chang, C. Yeh, W.-C. Chiu, and N. Yu, “Antifakeprompt: Prompt-tuned vision-language models are fake image detectors,” _arXiv preprint arXiv:2310.17419_ , 2023.
  * [17] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” 2015.
  * [18] F. Chollet, “Xception: Deep learning with depthwise separable convolutions,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , July 2017.
  * [19] H. Touvron, M. Cord, M. Douze, F. Massa, A. Sablayrolles, and H. Jégou, “Training data-efficient image transformers & distillation through attention,” 2021.
  * [20] F. Yu, A. Seff, Y. Zhang, S. Song, T. Funkhouser, and J. Xiao, “Lsun: Construction of a large-scale image dataset using deep learning with humans in the loop,” _arXiv preprint arXiv:1506.03365_ , 2015.
  * [21] P. Dhariwal and A. Nichol, “Diffusion models beat gans on image synthesis,” _Advances in neural information processing systems_ , vol. 34, pp. 8780–8794, 2021.
  * [22] J. Ho, A. Jain, and P. Abbeel, “Denoising diffusion probabilistic models,” _Advances in neural information processing systems_ , vol. 33, pp. 6840–6851, 2020.
  * [23] A. Q. Nichol and P. Dhariwal, “Improved denoising diffusion probabilistic models,” in _International Conference on Machine Learning_. PMLR, 2021, pp. 8162–8171.
  * [24] L. Liu, Y. Ren, Z. Lin, and Z. Zhao, “Pseudo numerical methods for diffusion models on manifolds,” _arXiv preprint arXiv:2202.09778_ , 2022.
  * [25] R. Rombach, A. Blattmann, D. Lorenz, P. Esser, and B. Ommer, “High-resolution image synthesis with latent diffusion models,” in _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , 2022, pp. 10 684–10 695.
  * [26] A. Nichol, P. Dhariwal, A. Ramesh, P. Shyam, P. Mishkin, B. McGrew, I. Sutskever, and M. Chen, “Glide: Towards photorealistic image generation and editing with text-guided diffusion models,” _arXiv preprint arXiv:2112.10741_ , 2021.
  * [27] J. Betker, G. Goh, L. Jing, T. Brooks, J. Wang, L. Li, L. Ouyang, J. Zhuang, J. Lee, Y. Guo _et al._ , “Improving image generation with better captions,” _Computer Science. https://cdn. openai. com/papers/dall-e-3. pdf_ , 2023.
  * [28] I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio, “Generative adversarial nets,” _Advances in neural information processing systems_ , vol. 27, 2014.
  * [29] A. Radford, L. Metz, and S. Chintala, “Unsupervised representation learning with deep convolutional generative adversarial networks,” _arXiv preprint arXiv:1511.06434_ , 2015.
  * [30] T. Karras, T. Aila, S. Laine, and J. Lehtinen, “Progressive growing of gans for improved quality, stability, and variation,” _arXiv preprint arXiv:1710.10196_ , 2017.
  * [31] T. Karras, S. Laine, and T. Aila, “A style-based generator architecture for generative adversarial networks,” in _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , 2019, pp. 4401–4410.
  * [32] F. Mokhayeri, K. Kamali, and E. Granger, “Cross-domain face synthesis using a controllable gan,” in _Proceedings of the IEEE/CVF winter conference on applications of computer vision_ , 2020, pp. 252–260.
  * [33] N. Ruiz, B.-J. Theobald, A. Ranjan, A. H. Abdelaziz, and N. Apostoloff, “Morphgan: One-shot face synthesis gan for detecting recognition bias,” _arXiv preprint arXiv:2012.05225_ , 2020.
  * [34] H. Liz-López, M. Keita, A. Taleb-Ahmed, A. Hadid, J. Huertas-Tato, and D. Camacho, “Generation and detection of manipulated multimodal audiovisual content: Advances, trends and open challenges,” _Information Fusion_ , vol. 103, p. 102103, 2024.
  * [35] W. Xu, C. Long, R. Wang, and G. Wang, “Drb-gan: A dynamic resblock generative adversarial network for artistic style transfer,” in _Proceedings of the IEEE/CVF international conference on computer vision_ , 2021, pp. 6383–6392.
  * [36] Y. Chen, Y.-K. Lai, and Y.-J. Liu, “Cartoongan: Generative adversarial networks for photo cartoonization,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2018, pp. 9465–9474.
  * [37] S. Yang, Z. Wang, Z. Wang, N. Xu, J. Liu, and Z. Guo, “Controllable artistic text style transfer via shape-matching gan,” in _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 2019, pp. 4442–4451.
  * [38] X. Zhu, L. Zhang, L. Zhang, X. Liu, Y. Shen, and S. Zhao, “Gan-based image super-resolution with a novel quality loss,” _Mathematical Problems in Engineering_ , vol. 2020, pp. 1–12, 2020.
  * [39] C. Ledig, L. Theis, F. Huszár, J. Caballero, A. Cunningham, A. Acosta, A. Aitken, A. Tejani, J. Totz, Z. Wang _et al._ , “Photo-realistic single image super-resolution using a generative adversarial network,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2017, pp. 4681–4690.
  * [40] M. S. Sajjadi, B. Scholkopf, and M. Hirsch, “Enhancenet: Single image super-resolution through automated texture synthesis,” in _Proceedings of the IEEE international conference on computer vision_ , 2017, pp. 4491–4500.
  * [41] S.-J. Park, H. Son, S. Cho, K.-S. Hong, and S. Lee, “Srfeat: Single image super-resolution with feature discrimination,” in _Proceedings of the European conference on computer vision (ECCV)_ , 2018, pp. 439–455.
  * [42] X. Wang, K. Yu, S. Wu, J. Gu, Y. Liu, C. Dong, Y. Qiao, and C. Change Loy, “Esrgan: Enhanced super-resolution generative adversarial networks,” in _Proceedings of the European conference on computer vision (ECCV) workshops_ , 2018, pp. 0–0.
  * [43] J.-Y. Zhu, T. Park, P. Isola, and A. A. Efros, “Unpaired image-to-image translation using cycle-consistent adversarial networks,” in _Proceedings of the IEEE international conference on computer vision_ , 2017, pp. 2223–2232.
  * [44] M. Mirza and S. Osindero, “Conditional generative adversarial nets,” _arXiv preprint arXiv:1411.1784_ , 2014.
  * [45] S. Reed, Z. Akata, X. Yan, L. Logeswaran, B. Schiele, and H. Lee, “Generative adversarial text to image synthesis,” in _International conference on machine learning_. PMLR, 2016, pp. 1060–1069.
  * [46] H. Zhang, T. Xu, H. Li, S. Zhang, X. Wang, X. Huang, and D. N. Metaxas, “Stackgan: Text to photo-realistic image synthesis with stacked generative adversarial networks,” in _Proceedings of the IEEE international conference on computer vision_ , 2017, pp. 5907–5915.
  * [47] N. Bodla, G. Hua, and R. Chellappa, “Semi-supervised fusedgan for conditional image generation,” in _Proceedings of the European conference on computer vision (ECCV)_ , 2018, pp. 669–683.
  * [48] Q. Lao, M. Havaei, A. Pesaranghader, F. Dutil, L. D. Jorio, and T. Fevens, “Dual adversarial inference for text-to-image synthesis,” in _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 2019, pp. 7567–7576.
  * [49] W. Li, P. Zhang, L. Zhang, Q. Huang, X. He, S. Lyu, and J. Gao, “Object-driven text-to-image synthesis via adversarial training,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2019, pp. 12 174–12 182.
  * [50] D. M. Souza, J. Wehrmann, and D. D. Ruiz, “Efficient neural architecture for text-to-image synthesis,” in _2020 International Joint Conference on Neural Networks (IJCNN)_. IEEE, 2020, pp. 1–8.
  * [51] Z. Wang, Z. Quan, Z.-J. Wang, X. Hu, and Y. Chen, “Text to image synthesis with bidirectional generative adversarial network,” in _2020 IEEE International Conference on Multimedia and Expo (ICME)_. IEEE, 2020, pp. 1–6.
  * [52] H. Zhang, J. Y. Koh, J. Baldridge, H. Lee, and Y. Yang, “Cross-modal contrastive learning for text-to-image generation,” in _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , 2021, pp. 833–842.
  * [53] A. Ramesh, M. Pavlov, G. Goh, S. Gray, C. Voss, A. Radford, M. Chen, and I. Sutskever, “Zero-shot text-to-image generation,” in _International Conference on Machine Learning_. PMLR, 2021, pp. 8821–8831.
  * [54] J. Atwood and D. Towsley, “Diffusion-convolutional neural networks,” _Advances in neural information processing systems_ , vol. 29, 2016.
  * [55] C. Saharia, W. Chan, S. Saxena, L. Li, J. Whang, E. L. Denton, K. Ghasemipour, R. Gontijo Lopes, B. Karagol Ayan, T. Salimans _et al._ , “Photorealistic text-to-image diffusion models with deep language understanding,” _Advances in Neural Information Processing Systems_ , vol. 35, pp. 36 479–36 494, 2022.
  * [56] D. M. J. T. R. S. N. Sebastian, “Midjourney,” 2023. [Online]. Available: https://www.midjourney.com/home/?callbackUrl=%2Fapp%2F
  * [57] R. Corvi, D. Cozzolino, G. Zingarini, G. Poggi, K. Nagano, and L. Verdoliva, “On the detection of synthetic images generated by diffusion models,” in _ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. IEEE, 2023, pp. 1–5.
  * [58] Z. Sha, Z. Li, N. Yu, and Y. Zhang, “De-fake: Detection and attribution of fake images generated by text-to-image diffusion models,” _arXiv preprint arXiv:2210.06998_ , 2022.
  * [59] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2016, pp. 770–778.
  * [60] D. A. Coccomini, A. Esuli, F. Falchi, C. Gennaro, and G. Amato, “Detecting images generated by diffusers,” _arXiv preprint arXiv:2303.05275_ , 2023.
  * [61] L. Guarnera, O. Giudice, and S. Battiato, “Level up the deepfake detection: a method to effectively discriminate images generated by gan architectures and diffusion models,” _arXiv preprint arXiv:2303.00608_ , 2023.
  * [62] R. Amoroso, D. Morelli, M. Cornia, L. Baraldi, A. Del Bimbo, and R. Cucchiara, “Parents and children: Distinguishing multimodal deepfakes from natural images,” _arXiv preprint arXiv:2304.00500_ , 2023.
  * [63] H. Wu, J. Zhou, and S. Zhang, “Generalizable synthetic image detection via language-guided contrastive learning,” _arXiv preprint arXiv:2305.13800_ , 2023.
  * [64] Z. Xi, W. Huang, K. Wei, W. Luo, and P. Zheng, “Ai-generated image detection using a cross-attention enhanced dual-stream network,” _arXiv preprint arXiv:2306.07005_ , 2023.
  * [65] P. Lorenz, R. Durall, and J. Keuper, “Detecting images generated by deep diffusion models using their local intrinsic dimensionality,” _arXiv preprint arXiv:2307.02347_ , 2023.
  * [66] Y. Ju, S. Jia, J. Cai, H. Guan, and S. Lyu, “Glff: Global and local feature fusion for ai-synthesized image detection,” _IEEE Transactions on Multimedia_ , 2023.
  * [67] S. Sinitsa and O. Fried, “Deep image fingerprint: Accurate and low budget synthetic image detector,” _arXiv preprint arXiv:2303.10762_ , 2023.
  * [68] X. Guo, X. Liu, Z. Ren, S. Grosz, I. Masi, and X. Liu, “Hierarchical fine-grained image forgery detection and localization,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , June 2023, pp. 3155–3165.
  * [69] D. Cozzolino, G. Poggi, R. Corvi, M. Nießner, and L. Verdoliva, “Raising the bar of ai-generated image detection with clip,” _arXiv preprint arXiv:2312.00195_ , 2023.
  * [70] Z. Wang, J. Bao, W. Zhou, W. Wang, H. Hu, H. Chen, and H. Li, “Dire for diffusion-generated image detection,” _arXiv preprint arXiv:2303.09295_ , 2023.
  * [71] R. Ma, J. Duan, F. Kong, X. Shi, and K. Xu, “Exposing the fake: Effective diffusion-generated images detection,” _arXiv preprint arXiv:2307.06272_ , 2023.
  * [72] M. Goljan, J. Fridrich, and R. Cogranne, “Rich model for steganalysis of color images,” in _2014 IEEE International workshop on information forensics and security (WIFS)_. IEEE, 2014, pp. 185–190.
  * [73] A. Odena, V. Dumoulin, and C. Olah, “Deconvolution and checkerboard artifacts,” _Distill_ , vol. 1, no. 10, p. e3, 2016.
  * [74] H. Li, B. Li, S. Tan, and J. Huang, “Identification of deep network generated images using disparities in color components,” _Signal Processing_ , vol. 174, p. 107616, 2020.
  * [75] K. Chandrasegaran, N.-T. Tran, A. Binder, and N.-M. Cheung, “Discovering transferable forensic features for cnn-generated images detection,” in _European Conference on Computer Vision_. Springer, 2022, pp. 671–689.
  * [76] E. J. Hu, Y. Shen, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang, and W. Chen, “Lora: Low-rank adaptation of large language models,” _arXiv preprint arXiv:2106.09685_ , 2021.
  * [77] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei, “Imagenet: A large-scale hierarchical image database,” in _2009 IEEE conference on computer vision and pattern recognition_. Ieee, 2009, pp. 248–255.
  * [78] D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” _arXiv preprint arXiv:1412.6980_ , 2014.
  * [79] Z. Wang, H. Zheng, P. He, W. Chen, and M. Zhou, “Diffusion-gan: Training gans with diffusion,” _arXiv preprint arXiv:2206.02262_ , 2022.
  * [80] A. Sauer, K. Chitta, J. Müller, and A. Geiger, “Projected gans converge faster,” _Advances in Neural Information Processing Systems_ , vol. 34, pp. 17 480–17 492, 2021.
  * [81] U. Ojha, Y. Li, and Y. J. Lee, “Towards universal fake image detectors that generalize across generative models,” 2023.
  * [82] S.-Y. Wang, O. Wang, R. Zhang, A. Owens, and A. A. Efros, “Cnn-generated images are surprisingly easy to spot… for now,” in _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , 2020, pp. 8695–8704.

  *[VLMs]: vision-language model
  *[LORA]: low-rank adaptation
  *[VLM]: vision-language model
  *[BLIP]: bootstrapping language image pre-training
  *[GANs]: generative adversarial network
  *[AI]: artificial intelligence
  *[DMs]: diffusion model
  *[CNNs]: convolutional neural network
  *[NLP]: natural language processing
  *[CV]: computer vision
  *[VQA]: visual question answering
  *[GAN]: generative adversarial network
  *[LDM]: latent diffusion model
  *[DeiT]: data-efficient image transformers
  *[MLP]: multi-layer perceptron
  *[CNN]: convolutional neural network
  *[CLIP]: contrastive language image pre-training
  *[SD]: stable diffusion
  *[RF]: random forest
  *[SVM]: support vector machine
  *[ADM]: ablated diffusion model
  *[DDPM]: denoising diffusion probabilistic models
  *[IDDPM]: improved denoising diffusion probabilistic models
  *[PNDM]: pseudo numerical methods for diffusion models on manifolds
  *[DM]: diffusion model
  *[DIRE]: diffusion reconstruction error
  *[DDIM]: denoising diffusion implicit models
  *[SeDID]: stepwise error for diffusion-generated image detection
  *[multiLID]: multi local intrinsic dimensionality
  *[SRM]: spatial rich model
  *[LASTED]: language-guided synthesis detection
  *[AMSFF]: attention-based multi-scale feature fusion
  *[PSM]: patch selection module
  *[Q-Former]: querying transformer
  *[PEFT]: parameter-efficient fine-tuning
  *[LLM]: large language model
  *[LSUN]: large-scale scene understanding
  *[FC]: fully-connected