# TGIF: Text-Guided Inpainting Forgery Dataset
††thanks:  This work was funded in part by the Flemish government’s Department
of Culture, Youth & Media (under the COM-PRESS project), in part by IDLab
(Ghent University – imec), in part by Flanders Innovation & Entrepreneurship
(VLAIO), in part by Research Foundation – Flanders (FWO) under grant number
V419524N, and in part by the European Union under the Horizon Europe vera.ai
project (grant number 101070093).

Hannes Mareen1, Dimitrios Karageorgiou2, Glenn Van Wallendael1, Peter
Lambert1, Symeon Papadopoulos2  1 IDLab, Ghent University – imec, Ghent,
Belgium, e-mail<EMAIL_ADDRESS>
2 Information Technologies Institute, CERTH, Thessaloniki, Greece, e-mail:
<EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

Digital image manipulation has become increasingly accessible and realistic
with the advent of generative AI technologies. Recent developments allow for
text-guided inpainting, making sophisticated image edits possible with minimal
effort. This poses new challenges for digital media forensics. For example,
diffusion model-based approaches could either splice the inpainted region into
the original image, or regenerate the entire image. In the latter case,
traditional image forgery localization (IFL) methods typically fail. This
paper introduces the Text-Guided Inpainting Forgery (TGIF) dataset, a
comprehensive collection of images designed to support the training and
evaluation of image forgery localization and synthetic image detection (SID)
methods. The TGIF dataset includes approximately 75k forged images,
originating from popular open-source and commercial methods; SD2, SDXL, and
Adobe Firefly. Using this data, we benchmark several state-of-the-art IFL and
SID methods. Whereas traditional IFL methods can detect spliced images, they
fail to detect regenerated inpainted images. Moreover, traditional SID may
detect the regenerated inpainted images to be fake, but cannot localize the
inpainted area. Finally, both types of methods fail when exposed to stronger
compression, while they are less robust to modern compression algorithms, such
as WEBP. As such, this work demonstrates the inefficiency of state-of-the-art
detectors on local manipulations performed by modern generative approaches,
and aspires to help with the development of more capable IFL and SID methods.
The dataset can be downloaded at https://github.com/IDLabMedia/tgif-dataset.

###### Index Terms:

Image Forensics, Forgery Detection, Forgery Localization, Synthetic Image
Detection.

##  I Introduction

Image manipulation is becoming more accessible with recent image editing tools
based on generative AI (GenAI) achieving outstanding levels of photorealism
[1, 2]. The advent of both open-source and commercial text-guided approaches,
such as Stable Diffusion (SD) [3] and Adobe Firefly, introduced the ability to
locally manipulate an image using simple text prompts, even to non-experts.
However, as high-quality image manipulation is becoming more accessible, there
are great risks to be exploited for malicious purposes such as disinformation,
fraud, and fake evidence. Therefore, image forensic methods aim to detect and
localize manipulations [4, 5]. While inpainting has long been part of the
target manipulation types of state-of-the-art image forgery localization (IFL)
methods, it is now revived due to the proliferation of text-to-image
generative diffusion models [2]. Previously, inpainting meant the removal of
objects [6, 7], whereas text-guided inpainting models enable the addition or
adaptation of objects.

Text-guided inpainting using GenAI models works differently than traditional
inpainting: an image (crop) is given as input to a model, along with a mask of
the inpainted area and a textual prompt. Through a series of diffusion and
denoising steps, the model then regenerates the entire image, yet only visibly
changing the inpainted area [3]. As such, there are two ways of using text-
guided inpainting. First, one can splice the inpainted area into the original
image. This is most useful when the generative model does not support high
resolutions. Second, the fully regenerated image can be used as is. With new
models supporting larger resolutions, the latter case is becoming more
practical. However, localizing the inpainted region when most forensic cues
have been corrupted due to the regeneration of the whole image is a new and
hard challenge for IFL algorithms, and untouched by synthetic image detection
(SID) methods.

Training and evaluating forensic methods requires large and varied datasets.
Although several image manipulation datasets exist, text-guided inpainting is
typically not included in them. In fact, to the best of our knowledge, the
only dataset including text-guided inpainting is COCO-Glide [8], consisting of
512 manipulated images of $256\times 256$ px, created using a single method
(GLIDE [9]). The dataset lacks extensive metadata, and does not vary in terms
of splicing type and mask type. Hence, there is a need for larger text-guided
inpainting datasets that can be utilized for both training and evaluation,
consisting of high-resolution images.

This paper presents the Text-Guided Inpainting Forgery (TGIF) dataset. The
contribution of this work is two-fold. First, we introduce our TGIF dataset in
Section III, which can be used for both training and evaluation. The dataset
consists of approx. 75k fake images, with resolutions up to $1024\times 1024$
px. We utilize three inpainting models (SD2, SDXL & Adobe Firefly).
Additionally, we provide both the manipulated image where the inpainted area
is spliced in the original image, as well as the fully-regenerated image, when
possible. We provide metadata about the estimated aesthetic quality, as well
as image-text matching scores that indicate whether the generated image aligns
with the given prompt.

Second, we evaluate our TGIF test set on state-of-the-art IFL and SID methods,
in Section IV. Additionally, we study the effects of JPEG and WEBP compression
on detection performance. As such, we provide valuable insights into the text-
guided-inpainting manipulations, how to detect them in practice, and highlight
open challenges.

##  II Related Work

###  II-A Image Forgery Localization (IFL)

To detect local image manipulations, in which a region of an image was forged
whereas the rest remains pristine, a variety of methods have been proposed
[4]. In general, IFL methods either aim to detect imperceptible
inconsistencies in the image, such as the artifacts originating from
compression [10] or the noise pattern introduced by camera sensors [11]), or
they aim to detect visible ones, such as boundaries between the forged and
pristine regions [12], while others fuse several forensic cues for increased
robustness [13, 14, 15]. Furthermore, there exist a few IFL methods that
specifically target inpainting [16, 17, 18], which exploit specific changes
made by inpainting models, such as in image Laplacian [16], or which
automatically learn to extract relevant features [17, 18]. However, they have
mostly been substituted by more generic subsequent works [19, 8].

Recent well-performing methods [19, 20, 21, 12, 22, 23, 8] are mostly based on
deep learning, where they train and evaluate on datasets containing diverse
manipulation types, such as splicing, copy-move, and traditional
inpainting/removal, in order to detect a broad range of of alterations.
However, such methods have not been trained to detect fully regenerated images
using text-guided generative models.

###  II-B Synthetic Image Detection (SID)

Early popular synthetic image generation approaches were mostly based on
Generative Adversarial Networks (GANs), whereas newer ones typically rely on
diffusion models (DMs). Similarly, several detection methods were trained on
images generated by GANs [24, 25, 26, 27, 28, 29, 30], whereas newer ones
typically focus on DMs or a mix of both [31, 32, 33, 34].

In general, most methods aim to extract features or a fingerprint, and somehow
compare that to those typically observed in synthetic images. The type of
feature may have a large influence on the model’s generalization performance.
For example, some methods were trained on GANs, yet demonstrated the ability
to detect images generated by DMs [26, 29, 30]. However, state-of-the-art SID
methods were trained and evaluated on real and synthetic images, lacking
support for images that were inpainted and hence contain both fake and real
parts (albeit regenerated by DMs). As such, they lack the ability of
localizing regions inpainted by DMs.

###  II-C Image Forensics Datasets

A variety of image forensics datasets for IFL and SID exist, and have been
listed in recent overview papers [35, 36]. Most relevant to this paper,
inpainted images can be found in the NIST16 [6] and DEFACTO [7] datasets.
However, the inpainting in them is not guided by a textual prompt, but instead
aims to simply remove the selected object through splicing.

To the best of our knowledge, the only dataset that includes text-guided
inpainting is the COCO-Glide [8], which consists of 512 manipulated images
with a resolution of $256\times 256$ px, cropped from the MS-COCO dataset and
inpainted using GLIDE [9]. This dataset is small and mainly useful for
evaluation. Additionally, the relatively low resolution is already outdated,
with new models supporting resolutions up to $1024\times 1024$ px. Moreover,
it neither considers the differences between splicing the generated content
into the inpainted region of the original image and re-generating the entire
picture with the GenAI model, nor does it asses the perceptual quality of the
inpainted images. Thus, the forensics community lacks a topic-agnostic dataset
that considers local image manipulations performed by generative models, while
including localization masks for images that have been fully regenerated.

##  III TGIF Dataset

This section presents our text-guided inpainting forgery dataset, containing
both spliced and fully regenerated images; yet only visually different in the
inpainted area.

###  III-A Source of Authentic Data

We used the MS-COCO dataset [37] (_val2017_) to collect authentic images
licensed under a Creative Commons Attribution 4.0 License, along with captions
and object masks (segmentation mask and bounding box). The images contain
objects from 80 categories (such as types of animals, food, and furniture). Of
those, we excluded _knife_ and _frisbee_ because these prompts were not
allowed by Adobe Firefly.

Although the COCO metadata links to Flickr images whose size is limited to 640
pixels on their largest dimension, we edited the Flickr URLs in order to
collect images with dimensionality of up to 1024 pixels (the largest publicly
available resolution) [38].

From each of the object classes, we selected a maximum of 50 images. We
excluded images of which their 1024p Flickr URL was not accessible, as well as
those with one of their dimensions smaller than 512 pixels. Moreover, the
object bounding box resolution should be smaller than 512$\times$512 px, and
their product larger than $64^{2}$.

###  III-B Text-Guided Inpainting Generation

For each authentic image and depicted object, we generated inpainted variants,
where the authentic object was replaced by a generated one of the same class.
For example, a real cat is replaced by a generated cat.

We utilized three text-guided inpainting methods: a) the open-source SD2
(_stabilityai/stable-diffusion-2-inpainting_), which constitutes a first-
generation high-fidelity diffusion model b) the open-source SDXL
(_diffusers/stable-diffusion-xl-1.0-inpainting-0.1_), representing the second
generation and supporting images of higher resolution, and c) the commercial
Adobe Firefly (in Adobe Photoshop 25.4.0). The open-source models are
encountered in popular inpainting pipelines, such as the Inpaint-Anything
[39], while the commercial approaches are more accessible to non-experts.
These three generative approaches constitute a representative set of the
available approaches for local manipulation through generative models.

For SD2, we took a 512$\times$512 px crop around each object of interest,
whereas we took 1024p crops for SDXL, as these are their respective native
resolutions. Additionally, the dimensions needed to be a multitude of 8. In
Adobe Firefly, we did not need to take any crops, although this may happen
internally. To perform the inpainting, a mask of the object is given to select
the area to be inpainted. For this, we utilized the segmentation mask and
bounding box provided by COCO. Moreover, each inpainting operation of each
model generated three variations, which we all included in the dataset.

As a textual prompt in SD2 and SDXL, we utilized the object category followed
by a caption from the image (i.e., describing the full image). During initial
experiments, we noticed that including the caption resulted in better results,
which was also observed in related work [40, 41]. However, note that text
misalignment between the prompt and the inpainted area could still occur. For
Adobe Firefly, we only included the object category as prompt (i.e., only
describing the inpainted area), as we noticed that this was sufficient in our
initial experiments.

We randomly chose several other parameters. For each image, the number of
inference steps is a random integer between 10 and 50, and the guidance scale
is a random float between 1 and 10. The seed is randomly chosen. These
parameters are also saved as metadata.

When inpainting using generative AI, the entire input image is regenerated.
That is, although only the masked area has visible changes, also the non-
masked area is (imperceptibly) changed by the GenAI algorithm, since the
entire image undergoes diffusion. In Adobe Firefly (Photoshop), the GenAI
inpainted image is spliced back into the original image, using the provided
mask with an additional small gradient border (i.e., a few pixels larger).
This is a similar approach as provided by other inpainting pipelines, such as
the one provided in HuggingFace diffusers [42]. For consistency, we spliced
the SD2 inpainted image into the original image using the same adapted mask.
Additionally, we saved the fully regenerated output image as a 512$\times$512
px image. For SDXL, we only saved the fully regenerated output image as a
1024p image. That is because the SDXL inpainting model discolors the entire
image, which is extremely visible when splicing into the original (a known
issue [43]).

In summary, an authentic image is transformed into multiple manipulated
variants. First, two masks are used for its generation (segmentation and
bounding box). Then, three inpainting methods are used, each creating three
generated variations. For SD2, we saved both the spliced image and fully
regenerated (synthetic) image crop. For SDXL, we saved only the fully
regenerated (synthetic) image. For Adobe Firefly (Photoshop), we saved only
the spliced image. As such, we get the sub-datasets SD2-sp, PS-sp, SD2-fr, and
SDXL-fr (sp for splicing and fr for fully regenerated). A high-level diagram
of this procedure is shown in Fig. 1.

Figure 1: TGIF dataset creation. The original image is fed to SD2, SDXL, and
Adobe Firefly (Photoshop), along with a prompt and a mask (segmentation or
bounding box). For SD2, an additional splicing step is performed, which is
done automatically in Adobe Firefly, and discarded for SDXL’s output.

The dataset can be downloaded at https://github.com/IDLabMedia/tgif-dataset.
Additionally, we intend to publish the code used for generating the dataset,
after acceptance.

###  III-C Dataset Splits

We have separated the full dataset in a training, validation, and test set, in
an approximate 80/10/10 setup, based on the authentic images. As such, the
training, validation and test set contain $58560$, $8184$, and $8232$
manipulated images, originating from $2440$, $341$, and $343$ unique authentic
images.

We have taken precaution to avoid leakage. First, we split the dataset based
on the object category. That is, the same category does not appear in multiple
subsets. Second, we deleted images that would appear in multiple subsets. That
is because an image can contain multiple objects, and hence appear multiple
times in the dataset (under different categories, with different masks and
prompts).

###  III-D Aesthetic Quality and Image-Text-Matching Scores

Due to the automatic generation of a large dataset, the quality of the
inpainted objects varies significantly. Additionally, the prompt is not always
respected by the diffusion model. To objectively measure this, we calculated
aesthetic quality scores using the neural image assessment (NIMA) [44]
(_epoch-82_ & _vgg16-397923af_) and generated image quality assessment (GIQA)
[45] (_pca95-cat_ & _gmm-cat-pca95-full7_) metrics. Additionally, we
calculated image-text-matching (ITM) scores [46] (_Salesforce_blip-itm-base-
coco_) to measure whether the prompt matches the outcome. We include these
values as metadata in the dataset.

##  IV Benchmark: IFL & SID

###  IV-A Experimental Setup

We performed the evaluation on the test set, separating the spliced subsets
(SD2-sp and PS-sp) and fully regenerated subsets (SD2-fr and SDXL-fr).
Additionally, to make the detection more challenging, we evaluated the
performance on compressed variants of the real and fake images. That is, we
used the JPEG and WEBP compression standards, using quality factors of 80 and
60.

For image forgery localization we use the mean pixel-level F1 score of the
fake images as a performance metric, which is the harmonic mean of the
precision and recall. We utilize a threshold of 0.5 to decide whether pixels
are real or fake. This threshold demonstrates the calibration of the models,
which is essential when deploying the models in the wild and requiring human
interpretation. We selected several recent high-performing AI-based IFL
methods, namely the PSCC-Net [19], SPAN [20], ImageForensicsOSN [21], MVSS-
Net++ [12], Mantranet [22], CAT-Net (v2) [23], TruFor [8], and MMFusion [13].

For synthetic image detection, as a performance metric, we use the AUC score.
We opted to report the threshold-agnostic AUC rather than threshold-specific
image-level F1 score, because several SID methods reported high F1 scores
while actually demonstrating bad performance. More specifically, these methods
often classified both real and fake images as fake, which, due to the class
imbalance, significantly boosted the F1 score. Since the AUC represents the
trade-off between the false positive rate and the false negative rate for a
range of thresholds, it better reflects the detection performance, in this
case. We selected several AI-based SID methods available in the SIDBench
framework [47], namely CNNDetect [24], DIMD [31], Dire [32], FreqDetect [25],
UnivFD [26], Fusing [48], GramNet [27], LGrad [28], NPR [29], RINE [33],
DeFake [34], and PatchCraft [30]. Some methods provide multiple models,
trained on different data. We only show the best performing in this paper.
Although some of these methods were trained on GANs, some of them have
demonstrated generalization performance to DMs. Hence, also methods trained on
GANs were included in our evaluation set.

###  IV-B Image Forgery Localization Methods

Table I reports the F1 values for all evaluated IFL methods. We highlight F1
values above 0.7. This threshold was chosen arbitrary, but allows to quickly
grasp which methods demonstrate good detection performance. For the splicing
datasets SD2 and PS, we observe that only CAT-Net, TruFor, and MMFusion report
high F1 scores. Additionally, Mantranet only reports a mediocre F1 value for
the PS dataset, while it has low performance for SD2. All other methods
demonstrate low detection performance.

For the fully regenerated datasets SD2-fr and SDXL-fr, no method is able to
detect and localize these manipulations. This is as expected, since
regenerating the image removes most invisible traces such as compression
artifacts and camera noise. In fact, diffusing and regenerating an image is an
existing attack against forgery detection methods [49]. This highlights the
threat of local image manipulations using diffusion models that fully
regenerate images; these cannot be detected by traditional image forgery
localization methods. This threat is enlarged when one considers the
widespread use of GenAI models in applications such as super-resolution[50]
and image compression [51]. Although these applications can be used with
harmless intentions, they may remove traces used by traditional image forgery
localization methods.

We only analyzed the influence of compression for the best performing methods
in the uncompressed case, and only for the splicing datasets. When performing
JPEG compression, F1 scores drop significantly. Moreover, this performance
drop is amplified when performing WEBP compression. Hence, this demonstrates a
need to develop IFL methods that are robust to compression.

TABLE I: Evaluation of Image Forgery Localization Methods on our TGIF Dataset (F1 Score).  Method | Uncompressed | JPEG Q80 | JPEG Q60 | WEBP Q80 | WEBP Q60
---|---|---|---|---|---
SD2-sp | PS-sp | SD2-fr | SDXL-fr | SD2-sp | PS-sp | SD2-sp | PS-sp | SD2-sp | PS-sp | SD2-sp | PS-sp
PSCC-Net [19] | 0.15 | 0.38 | 0.05 | 0.05 | - | - | - | - | - | - | - | -
SPAN [20] | 0.00 | 0.00 | 0.00 | 0.00 | - | - | - | - | - | - | - | -
ImageForensicsOSN [21] | 0.23 | 0.36 | 0.20 | 0.18 | - | - | - | - | - | - | - | -
MVSS-Net++ [12] | 0.07 | 0.08 | 0.06 | 0.09 | - | - | - | - | - | - | - | -
Mantranet [22] | 0.15 | 0.56 | 0.03 | 0.05 | 0.08 | 0.13 | 0.03 | 0.04 | 0.08 | 0.11 | 0.09 | 0.11
CAT-Net [23] | 0.87 | 0.85 | 0.04 | 0.03 | 0.05 | 0.07 | 0.09 | 0.08 | 0.13 | 0.13 | 0.07 | 0.07
TruFor [8] | 0.83 | 0.79 | 0.19 | 0.18 | 0.32 | 0.44 | 0.25 | 0.32 | 0.24 | 0.29 | 0.20 | 0.23
MMFusion [13] | 0.75 | 0.74 | 0.15 | 0.18 | 0.35 | 0.49 | 0.22 | 0.32 | 0.22 | 0.33 | 0.19 | 0.28

###  IV-C Synthetic Image Detection Methods

TABLE II: Evaluation of Synthetic Image Detection Methods on our TGIF Dataset (AUC Score).  Method | Trained on | Uncompressed | JPEG Q80 | JPEG Q60 | WEBP Q80 | WEBP Q80
---|---|---|---|---|---|---
SD2-fr | SDXL-fr | SD2-sp | PS-sp | SD2-fr | SDXL-fr | SD2-fr | SDXL-fr | SD2-fr | SDXL-fr | SD2-fr | SDXL-fr
CNNDetect [24] | ProGAN | 0.57 | 0.61 | 0.52 | 0.51 | 0.57 | 0.62 | 0.56 | 0.61 | 0.55 | 0.60 | 0.55 | 0.60
DIMD [31] | LDM | 1.00 | 0.94 | 0.66 | 0.54 | 1.00 | 0.95 | 0.99 | 0.92 | 0.96 | 0.71 | 0.90 | 0.61
Dire [32] | ADM | 0.49 | 0.62 | 0.49 | 0.51 | 0.49 | 0.61 | 0.49 | 0.60 | 0.50 | 0.60 | 0.48 | 0.60
FreqDetect [25] | GANs | 0.50 | 0.40 | 0.53 | 0.56 | 0.72 | 0.71 | 0.71 | 0.72 | 0.66 | 0.61 | 0.62 | 0.62
UnivFD [26] | ProGAN | 0.82 | 0.8 | 0.62 | 0.63 | 0.73 | 0.78 | 0.70 | 0.73 | 0.71 | 0.74 | 0.69 | 0.73
Fusing [48] | ProGAN | 0.55 | 0.47 | 0.52 | 0.54 | 0.52 | 0.59 | 0.53 | 0.61 | 0.50 | 0.53 | 0.50 | 0.54
GramNet [27] | GANs | 0.82 | 0.55 | 0.57 | 0.63 | 0.48 | 0.56 | 0.47 | 0.54 | 0.49 | 0.54 | 0.50 | 0.53
LGrad [28] | ProGAN | 0.85 | 0.83 | 0.55 | 0.60 | 0.51 | 0.57 | 0.51 | 0.64 | 0.47 | 0.42 | 0.47 | 0.41
NPR [29] | ProGAN | 0.51 | 0.67 | 0.50 | 0.51 | 0.50 | 0.30 | 0.51 | 0.68 | 0.49 | 0.69 | 0.49 | 0.31
RINE [33] | ProGAN | 0.89 | 0.89 | 0.64 | 0.65 | 0.79 | 0.88 | 0.75 | 0.85 | 0.77 | 0.85 | 0.76 | 0.87
| LDM | 0.97 | 0.93 | 0.61 | 0.64 | 0.84 | 0.89 | 0.80 | 0.82 | 0.75 | 0.76 | 0.72 | 0.77
DeFake [34] | Diffusion | 0.65 | 0.61 | 0.55 | 0.51 | 0.54 | 0.45 | 0.54 | 0.43 | 0.54 | 0.45 | 0.53 | 0.49
PatchCraft [30] | ProGAN | 0.98 | 0.95 | 0.55 | 0.57 | 0.66 | 0.67 | 0.60 | 0.58 | 0.60 | 0.57 | 0.59 | 0.61

Table II shows the performance (AUC) results for all evaluated SID methods. We
highlight AUC values above 0.8. This threshold was chosen arbitrarily, but
allows to quickly grasp which methods demonstrate good detection performance.
For the spliced datasets SD2-sp and PS-sp, no SID method is able to detect the
synthetic spliced regions. For the fully regenerated datasets, we observe that
6 out of 12 methods are able to detect synthetic images in uncompressed form.
Surprisingly, some methods trained on GANs (such as UnivFD, LGrad and RINE)
demonstrate better performance than methods trained on DMs (such as DeFake).
Additionally, only two of those methods survive JPEG compression, and only one
survives WEBP compression. Moreover, some methods (such as DIMD) are more
robust in detecting SD2 than SDXL, whereas others (such as RINE) are more
robust in detecting SDXL than SD2. No single method is robust for both SD2 and
SDXL on all evaluated compression levels. This demonstrates that these methods
are complementary.

In summary, we demonstrated that several state-of-the-art SID methods are able
to detect fully regenerated inpainted images by SD2 and SDXL. However, they
fail to detect the spliced inpainted images (even though the spliced regions
were generated synthetically). Additionally, their performance decreases when
exposed to compression, with a notable difference between WEBP and JPEG
compression. Finally, the tested SID methods are not able to localize the
inpainted area, but rather produce a SID score for the whole image.

##  V Discussion & Conclusion

The proposed TGIF dataset represents a significant advancement in the
resources available for training and evaluating forensic methods. By including
high-resolution images and utilizing multiple inpainting methods, as well as
providing both spliced and fully regenerated versions, our TGIF dataset
addresses the limitations of existing forensic methods and datasets. As such,
it is expected to contribute to the development of more effective forensic
tools.

Our benchmark analysis shows that some of the existing IFL methods are able to
detect and localize spliced images, whereas they fail to localize the
inpainted area in fully regenerated images. In contrast, some of the existing
SID methods are able to detect fully regenerated images, yet lack the ability
to localize the synthetic inpainted area. Additionally, they cannot detect the
synthetic data when it is spliced into the image. These limitations highlight
the need for new forensic methods, perhaps combining aspects from both IFL and
SID methods.

Furthermore, we showed that forensic methods suffer from a notable decrease in
performance when exposed to compression, particularly with WEBP. This
highlights the need to include compressed images during training.

Future work could extend and incorporate the TGIF dataset for the training of
new detection methods to localize manipulations in fully regenerated images,
with attention to robustness against compression.

## References

  * [1] C. Zhang, C. Zhang, M. Zhang, and I. S. Kweon, “Text-to-image diffusion models in generative AI: A survey,” _arXiv preprint arXiv:2303.07909_ , 2023.
  * [2] F. Zhan, Y. Yu, R. Wu, J. Zhang, S. Lu, L. Liu, A. Kortylewski, C. Theobalt, and E. Xing, “Multimodal image synthesis and editing: The generative AI era,” _IEEE Trans. Pattern Analys. Machine Intell._ , vol. 45, no. 12, pp. 15 098–15 119, 2023.
  * [3] R. Rombach, A. Blattmann, D. Lorenz, P. Esser, and B. Ommer, “High-resolution image synthesis with latent diffusion models,” in _Proc. IEEE/CVF Conf. Computer Vission Pattern Recogn._ , 2022.
  * [4] F. Z. Mehrjardi, A. M. Latif, M. S. Zarchi, and R. Sheikhpour, “A survey on deep learning-based image forgery detection,” _Pattern Recognition_ , p. 109778, 2023.
  * [5] D. Tariang, R. Corvi, D. Cozzolino, G. Poggi, K. Nagano, and L. Verdoliva, “Synthetic image verification in the era of generative ai: What works and what isn’t there yet,” _arXiv preprint arXiv:2405.00196_ , 2024.
  * [6] H. Guan, M. Kozak, E. Robertson, Y. Lee, A. N. Yates, A. Delgado, D. Zhou, T. Kheyrkhah, J. Smith, and J. Fiscus, “MFC datasets: Large-scale benchmark datasets for media forensic challenge evaluation,” in _Proc. IEEE Winter App. Computer Vision Workshops (WACVW)_. IEEE, 2019, pp. 63–72.
  * [7] G. Mahfoudi, B. Tajini, F. Retraint, F. Morain-Nicolier, J. L. Dugelay, and P. Marc, “Defacto: Image and face manipulation dataset,” in _Europ. Signal Process. Conf. (EUSIPCO)_. IEEE, 2019, pp. 1–5.
  * [8] F. Guillaro, D. Cozzolino, A. Sud, N. Dufour, and L. Verdoliva, “TruFor: Leveraging all-round clues for trustworthy image forgery detection and localization,” in _Proc. IEEE/CVF Conf. Computer Vision Pattern Recogn. (CVPR)_ , 2023, pp. 20 606–20 615.
  * [9] A. Nichol, P. Dhariwal, A. Ramesh, P. Shyam, P. Mishkin, B. McGrew, I. Sutskever, and M. Chen, “GLIDE: Towards photorealistic image generation and editing with text-guided diffusion models,” _arXiv preprint arXiv:2112.10741_ , 2021.
  * [10] H. Mareen, D. Vanden Bussche, F. Guillaro, D. Cozzolino, G. Van Wallendael, P. Lambert, and L. Verdoliva, “Comprint: Image forgery detection and localization using compression fingerprints,” in _Proc. Int. Conf. Pattern Recogn. Workshops_ , 2023, pp. 281–299.
  * [11] D. Cozzolino and L. Verdoliva, “Noiseprint: A CNN-based camera model fingerprint,” _IEEE Trans Inf. Forensics Security_ , vol. 15, pp. 144–159, 2020.
  * [12] C. Dong, X. Chen, R. Hu, J. Cao, and X. Li, “MVSS-Net: Multi-view multi-scale supervised networks for image manipulation detection,” _IEEE Trans. Pattern Analysis and Machine Intel._ , vol. 45, no. 3, pp. 3539–3553, 2022.
  * [13] K. Triaridis and V. Mezaris, “Exploring multi-modal fusion for image manipulation detection and localization,” in _Int. Conf. Multimedia Model._ Springer, 2024, pp. 198–211.
  * [14] D. Karageorgiou, G. Kordopatis-Zilos, and S. Papadopoulos, “Fusion transformer with object mask guidance for image forgery analysis,” in _Proc. IEEE/CVF Conf. Computer Vission Pattern Recogn._ , 2024, pp. 4345–4355.
  * [15] H. Mareen, L. De Neve, P. Lambert, and G. Van Wallendael, “Harmonizing image forgery detection & localization: Fusion of complementary approaches,” _J. Imaging_ , vol. 10, no. 1, p. 4, 2023.
  * [16] H. Li, W. Luo, and J. Huang, “Localization of diffusion-based inpainting in digital images,” _IEEE Trans Inf. Forensics Security_ , vol. 12, no. 12, pp. 3050–3064, 2017.
  * [17] H. Wu and J. Zhou, “Iid-net: Image inpainting detection network via neural architecture search and attention,” _IEEE Trans. Circuits Systems Video Technol._ , vol. 32, no. 3, pp. 1172–1185, 2021.
  * [18] A. Li, Q. Ke, X. Ma, H. Weng, Z. Zong, F. Xue, and R. Zhang, “Noise doesn’t lie: Towards universal detection of deep inpainting,” in _Proc. Int. Conf. Artificial Intel. IJCAI_ , Z.-H. Zhou, Ed., 2021, pp. 786–792.
  * [19] X. Liu, Y. Liu, J. Chen, and X. Liu, “PSCC-Net: Progressive spatio-channel correlation network for image manipulation detection and localization,” _IEEE Trans. Circuits Systems Video Technol._ , vol. 32, no. 11, pp. 7505–7517, 2022.
  * [20] X. Hu, Z. Zhang, Z. Jiang, S. Chaudhuri, Z. Yang, and R. Nevatia, “SPAN: Spatial pyramid attention network for image manipulation localization,” in _Proc. Europ. Computer Vision Conf._ Springer, 2020, pp. 312–328.
  * [21] H. Wu, J. Zhou, J. Tian, J. Liu, and Y. Qiao, “Robust image forgery detection against transmission over online social networks,” _IEEE Trans. Inf. Forensics Security_ , vol. 17, pp. 443–456, 2022.
  * [22] Y. Wu, W. AbdAlmageed, and P. Natarajan, “ManTra-Net: Manipulation tracing network for detection and localization of image forgeries with anomalous features,” in _Proc. IEEE/CVF Conf. Computer Vision Pattern Recogn._ , 2019, pp. 9543–9552.
  * [23] M.-J. Kwon, S.-H. Nam, I.-J. Yu, H.-K. Lee, and C. Kim, “Learning JPEG compression artifacts for image manipulation detection and localization,” _Int. J. Computer Vision_ , vol. 130, no. 8, pp. 1875–1895, 2022.
  * [24] S.-Y. Wang, O. Wang, R. Zhang, A. Owens, and A. A. Efros, “Cnn-generated images are surprisingly easy to spot…for now,” in _Proc. Int. Conf. Pattern Recogn. (CVPR)_ , 2020.
  * [25] J. Frank, T. Eisenhofer, L. Schönherr, A. Fischer, D. Kolossa, and T. Holz, “Leveraging frequency analysis for deep fake image recognition,” in _Int. Conf. Machine Learning_. PMLR, 2020, pp. 3247–3258.
  * [26] U. Ojha, Y. Li, and Y. J. Lee, “Towards universal fake image detectors that generalize across generative models,” in _Proc. IEEE/CVF Conf. Computer Vision Pattern Recogn._ , 2023, pp. 24 480–24 489.
  * [27] Z. Liu, X. Qi, and P. H. Torr, “Global texture enhancement for fake face detection in the wild,” in _Proc. IEEE/CVF Conf. Computer Vision Pattern Recogn._ , 2020, pp. 8060–8069.
  * [28] C. Tan, Y. Zhao, S. Wei, G. Gu, and Y. Wei, “Learning on gradients: Generalized artifacts representation for gan-generated images detection,” in _Proc. IEEE/CVF Conf. Computer Vision Pattern Recogn._ , 2023, pp. 12 105–12 114.
  * [29] C. Tan, Y. Zhao, S. Wei, G. Gu, P. Liu, and Y. Wei, “Rethinking the up-sampling operations in cnn-based generative network for generalizable deepfake detection,” in _Proc. IEEE/CVF Conf. Computer Vision Pattern Recogn._ , 2024, pp. 28 130–28 139.
  * [30] N. Zhong, Y. Xu, Z. Qian, and X. Zhang, “PatchCraft: Exploring texture patch for efficient ai-generated image detection,” _arXiv preprint arXiv:2311.12397_ , 2023.
  * [31] R. Corvi, D. Cozzolino, G. Zingarini, G. Poggi, K. Nagano, and L. Verdoliva, “On the detection of synthetic images generated by diffusion models,” in _Proc. IEEE Int. Conf. Acoustics, Speech Signal Process. (ICASSP)_ , 2023, pp. 1–5.
  * [32] Z. Wang, J. Bao, W. Zhou, W. Wang, H. Hu, H. Chen, and H. Li, “DIRE for diffusion-generated image detection,” in _Proc. IEEE/CVF Int. Conf Computer Vision (ICCV)_ , 2023, pp. 22 445–22 455.
  * [33] C. Koutlis and S. Papadopoulos, “Leveraging representations from intermediate encoder-blocks for synthetic image detection,” _arXiv preprint arXiv:2402.19091_ , 2024.
  * [34] Z. Sha, Z. Li, N. Yu, and Y. Zhang, “DE-FAKE: Detection and attribution of fake images generated by text-to-image generation models,” in _Proc. ACM SIGSAC Conf. Computer Comm. Security_ , 2023, pp. 3418–3432.
  * [35] L. Verdoliva, “Media forensics and deepfakes: an overview,” _IEEE J Selected Topics Signal Process._ , vol. 14, no. 5, pp. 910–932, 2020.
  * [36] L. Lin, N. Gupta, Y. Zhang, H. Ren, C.-H. Liu, F. Ding, X. Wang, X. Li, L. Verdoliva, and S. Hu, “Detecting multimedia generated by large AI models: A survey,” _arXiv preprint arXiv:2402.00045_ , 2024.
  * [37] T.-Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollár, and C. L. Zitnick, “Microsoft COCO: Common objects in context,” in _Proc. Europ. Computer Vision Conf._ Springer, 2014, pp. 740–755.
  * [38] “Flickr services photo image urls,” https://www.flickr.com/services/api/misc.urls.html, accessed: 2024-07-05.
  * [39] T. Yu, R. Feng, R. Feng, J. Liu, X. Jin, W. Zeng, and Z. Chen, “Inpaint anything: Segment anything meets image inpainting,” _arXiv preprint arXiv:2304.06790_ , 2023.
  * [40] S. Xie, Z. Zhang, Z. Lin, T. Hinz, and K. Zhang, “SmartBrush: Text and shape guided object inpainting with diffusion model,” in _Proc. IEEE/CVF Conf. Computer Vission Pattern Recogn._ , 2023, pp. 22 428–22 437.
  * [41] S. Wang, C. Saharia, C. Montgomery, J. Pont-Tuset, S. Noy, S. Pellegrini, Y. Onoe, S. Laszlo, D. J. Fleet, R. Soricut _et al._ , “Imagen editor and EditBench: Advancing and evaluating text-guided image inpainting,” in _Proc. IEEE/CVF Conf. Computer Vission Pattern Recogn._ , 2023, pp. 18 359–18 369.
  * [42] H. Face, “Diffusers: State-of-the-art diffusion models for image and audio generation in pytorch and flax.” https://github.com/huggingface/diffusers, accessed: 2024-07-05.
  * [43] “Diffusers github issue - inpainting produces results that are uneven with input image,” https://github.com/huggingface/diffusers/issues/5808, accessed: 2024-07-05.
  * [44] H. Talebi and P. Milanfar, “NIMA: Neural image assessment,” _IEEE Trans. Image Process._ , vol. 27, no. 8, pp. 3998–4011, 2018.
  * [45] S. Gu, J. Bao, D. Chen, and F. Wen, “GIQA: Generated image quality assessment,” in _Proc. Europ. Computer Vision Conf. (ECCV)_. Springer, 2020, pp. 369–385.
  * [46] J. Li, D. Li, C. Xiong, and S. Hoi, “BLIP: Bootstrapping language-image pre-training for unified vision-language understanding and generation,” in _Int. Conf. Machine Learning_. PMLR, 2022, pp. 12 888–12 900.
  * [47] M. Schinas and S. Papadopoulos, “Sidbench: A python framework for reliably assessing synthetic image detection methods,” _arXiv preprint arXiv:2404.18552_ , 2024.
  * [48] Y. Ju, S. Jia, L. Ke, H. Xue, K. Nagano, and S. Lyu, “Fusing global and local features for generalized ai-synthesized image detection,” in _Proc. IEEE Int. Conf. Image Process. (ICIP)_. IEEE, 2022, pp. 3465–3469.
  * [49] M. Tailanián, M. Gardella, A. Pardo, and P. Musé, “Diffusion models meet image counter-forensics,” in _Proc. IEEE/CVF Winter Conf. App. Computer Vision (WACV)_ , 2024, pp. 3925–3935.
  * [50] H. Li, Y. Yang, M. Chang, S. Chen, H. Feng, Z. Xu, Q. Li, and Y. Chen, “SRDiff: Single image super-resolution with diffusion probabilistic models,” _Neurocomputing_ , vol. 479, pp. 47–59, 2022.
  * [51] R. Yang and S. Mandt, “Lossy image compression with conditional diffusion models,” in _Advances in Neural Information Processing Systems_ , A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, Eds., vol. 36. Curran Associates, Inc., 2023, pp. 64 971–64 995.