# On quantifying and improving realism of images generated with diffusion

Yunzhuo Chen1, Naveed Akhtar2, Nur Al Hasan Haldar3, Ajmal Mian4
The University of Western Australia, Perth, Australia
<EMAIL_ADDRESS><EMAIL_ADDRESS>
<EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

Recent advances in diffusion models have led to a quantum leap in the quality
of generative visual content. However, quantification of realism of the
content is still challenging. Existing evaluation metrics, such as Inception
Score and Fréchet inception distance, fall short on benchmarking diffusion
models due to the versatility of the generated images. Moreover, they are not
designed to quantify realism of an individual image. This restricts their
application in forensic image analysis, which is becoming increasingly
important in the emerging era of generative models. To address that, we first
propose a metric, called Image Realism Score (IRS), computed from five
statistical measures of a given image. This non-learning based metric not only
efficiently quantifies realism of the generated images, it is readily usable
as a measure to classify a given image as real or fake. We experimentally
establish the model- and data-agnostic nature of the proposed IRS by
successfully detecting fake images generated by Stable Diffusion Model (SDM),
Dalle2, Midjourney and BigGAN. We further leverage this attribute of our
metric to minimize an IRS-augmented generative loss of SDM, and demonstrate a
convenient yet considerable quality improvement of the SDM-generated content
with our modification. Our efforts have also led to Gen-100 dataset, which
provides 1,000 samples for 100 classes generated by four high-quality models.
We will release the dataset and code.

##  1 Introduction

Generative models, including Variational Autoencoders (VAEs) [17, 37], Energy-
Based Models (EBM) [29, 32], Generative Adversarial Network (GANs) [13, 53]
and normalizing flow [41] have historically attracted significant attention
from the research community, only to be eventually surpassed by diffusion
models [25, 8]. Diffusion models have recently provided a quantum leap to the
generative visual content quality [8]. Moreover, they are claimed to also
overcome challenges such as matching posterior distributions in VAEs, managing
unpredictability in GAN objectives, high computational demands of Markov Chain
Monte Carlo [21] techniques in EBMs, and network limitations of the
normalizing flows. Naturally, we can expect an even higher popularity of the
diffusion models in generative computer vision in the future.

Figure 1: We manually group images into high and low quality sets and measure
their realism scores. FID [24] and IS [48] are model/data-dependent metrics
and produce counter-intuitive results on these sets. The proposed Image
Realism Score (IRS) is sample-specific and performs intuitive and reliable
discrimination between the low and high quality samples. Unlike FID and IS,
the proposed IRS can also be used as a loss to improve the generative ability
of a model. Figure 2: Overall framework of the proposed IRS. The left side
illustrates how IRS quantifies realism in an image through pentagon area;
larger area means higher realism. IRS maximizes the area difference between
real and generated images through a special arrangement of five statistical
measures followed by a calibration process. The right side shows that IRS is
used to modify the loss function to improve realism in the generative model
outputs, taking SDM as an example.

Owing to the importance of generative modeling in vision, a number of metrics
have been proposed to evaluate the abilities of generative models [6, 24, 48,
56]. However, due to the high quality and versatility of the content generated
by diffusion models, these metrics are now falling short on providing
meaningful evaluation of the diffusion model generated content. Not to
mention, these metrics have widely known intrinsic weaknesses. For instance,
the popular Fréchet Inception Distance (FID) [24] is known to suffer from
notable bias [6]. The Inception Score (IS) [48] is also known to be often
suboptimal [4]. Moreover, both metrics are dataset- or model-dependent in the
sense that they rely on a reference dataset or a model to compute their
scores. For example, IS [48] uses an ImageNet [46] trained Inception model
[51] to provide a meaningful evaluation score. This is particularly
problematic for forensics of generative content where normally a single sample
is available and the task is to adjudge its authenticity in a model- and data-
agnostic manner.

Nowadays, we are witnessing many new generative diffusion models surfacing on
a daily basis, each with better content quality and flexibility than the
previous ones [8]. With public access to these models, this can lead to
serious societal problems if the generated content are used with a negative
intent [42]. The inability of the current evaluation metrics for generative
modeling to verify the authenticity of individual (generated) images is an
obvious shortcoming that needs to be addressed for the forensic treatment of
the content [11]. This work fills-in the gap by proposing an Image Realism
Score (IRS) metric that is more suited to benchmark the content quality of
modern diffusion models.

The proposed IRS is a ‘non-learning’ based metric which allows it to be
dataset- and model-agnostic, while computing intuitive scores according to
image quality, see Fig. 1. The proposed metric relies on well-established
concepts in image processing, including Canny Edge Density [52], GLCM Contrast
[49], GLCM Energy [49], Variance of Laplacian [23], and Mean Spectrum [20] to
determine the realism of an image. Leveraging the sample-specific nature of
the statistics used by IRS, we can delineate between natural and fake content
easily with our metric. To facilitate further efforts towards the
quantification of realism in diffusion-generated content, we also introduce a
100-category dataset in this work, called Gen-100. Each category in this
dataset contains 1,000 images generated with ChatGPT prompts using Stable
Diffusion Model (SDM)[43], Dalle2[14], Midjourney[40] and BigGAN[9]. Our
dataset is mainly used to validate the model-agnostic and data-agnostic nature
of IRS. We eventually exploit the same property of IRS to minimize the
generative loss of SDMs, which allows us to considerably improve the quality
of SDM-generated content. To summarize, this paper makes the following three
main contributions.

  * •

It introduces Image Realism Score (IRS), a first-of-its-kind non-learning
based sample specific metric to quantify realism of an image for
differentiating natural images from those generated by generative models.

  * •

It leverages IRS to benchmark realism of popular generative models and
establishes that the metric is well-suited to forensic analysis by employing
it for fake image detection. In the process, it also proposes Gen-100 dataset
that contains 1,000 images of 100 classes generated by four models.

  * •

By regulating the training loss of Stable Diffusion Model (SDM) [43] to
further minimize the proposed metric score, it demonstrates a considerable
improvement in the quality of images generated by SDM.

##  2 Related Work

In the last decade, high-quality sample generation of Generative Adversarial
Networks (GANs) [7] has enabled deep generative models to receive widespread
attention. Nevertheless, with the emergence of diffusion models [6, 24, 48,
56, 43] GANs are no longer the dominant force in this field. Diffusion models
have gained rapid popularity in recent years owing to their stability and
superior generation quality. They address some of the common challenges
associated with GANs, such as mode collapse, the overhead of adversarial
learning, and convergence failure [2]. The training strategy of diffusion
models involves systematically corrupting the training data by gradually
adding Gaussian noise, followed by learning to retrieve the original data from
the noisy version [8]. Additionally, since their training approach makes small
changes to the original data and then corrects those changes, they manage to
learn a data distribution where samples closely follow the original data,
providing a strong sense of realism to the generated samples. These strengths
of diffusion models have led to their significant achievements in the field of
image generation technologies [2, 39]. Moreover, diffusion models have been
widely applied in various domains, including image denoising [34] and repair
[18], image super-resolution [30, 5], and text-to-image generation [44, 47].

The diffusion model training process consists of two steps. First, there is a
predefined forward process that transforms the data distribution into a
Gaussian distribution. The second, the corresponding reverse process, employs
a trained neural network to simulate regular or random stepwise reversal of
the forward process. Diffusion modeling provides a more stable training target
and higher generative quality compared to VAEs, EBMs, and normalizing flows
[17, 37, 13, 53]. However, because the prior distribution is iteratively
transformed into a complex data distribution, a significant number of function
evaluations are required in the reverse process. Consequently, diffusion
models inherently suffer from a more time-consuming sampling process.
Researchers have proposed various solutions, such as introducing new forward
processes to stabilize sampling [26, 15], and recent studies have addressed
this issue by implementing dimensionality reduction [22, 27].

Although several evaluation metrics are proposed to measure the performance of
deep generative models, there is no globally accepted agreement on the best
metrics for generative models. Currently, popular metrics include the
Inception Score (IS), the Frechet Inception Distance (FID), Maximum Mean
Discrepancy (MMD)[19] and Activation maximization score[57]. Arguably, IS [48]
is the most widely used metric for evaluating generative models. It employs a
pre-trained neural network to evaluate the desired properties of generated
samples, such as high classifiability and diversity of class labels. This
metric shows a reasonable correlation with the quality and diversity of the
generated images. However, IS is also known to have multiple limitations [55].
Firstly, IS is sensitive to over-fitting. Secondly, it may favor models that
learn from diverse images. Also, operations such as mixing natural images from
completely different distributions may cheat this metric.

Another widely used evaluation metric for generative models is FID score [24].
To compute this metric, images are embedded into a feature space. Thereafter,
mean and covariance of both generated and real data embeddings are computed
and FID measures the image quality by comparing these two distribution
parameters. While FID excels in discriminability, robustness, and speed, it
assumes that the data features follow a Gaussian distribution, which is not
always the case. It is also notable that both IS and FID are model- or
dataset-specific metrics. This makes them less suitable to quantify the
quality of individual images.

##  3 Proposed Approach

Numerous works highlight the importance of texture, edges, and frequency for
detecting fake images [16, 10, 33, 45, 1, 12]. However, most fake detection
methods proposed are still ‘learned’ models. They face intrinsic limitations
of carefully handling training demands while falling short on generalizing to
unseen generative techniques [38]. In this work, we develop a non-learning
based metric called Image Realism Score (IRS) to quantify realism in generated
images, as illustrated in Fig. 2. This metric incorporates five image
statistics, and leads to a convenient detection of synthetic images generated
by the contemporary diffusion models as well as BigGAN. In this section, we
first introduce the mathematical principles behind the employed statistics and
discuss how they positively influence our metric. Subsequently, we introduce
their infusion into IRS. Later, we discuss the use of the proposed IRS for
fake content detection and improvement of the popular Stable Diffusion Model
(SDM) [43] using the same metric.

###  3.1 Image Statistical Measures

Gray-Level Co-occurrence Matrix (GLCM) [49] is used in image processing to
extract textual features of an image. One of the statistical measures that can
be derived from a GLCM is “Energy”, also known as “Uniformity” or “Angular
Second Moment”. Given a GLCM $P\in\mathbb{R}^{N\times N}$ (where $N$ is the
number of gray levels in the image), the Energy $E$ is defined as

$\text{GLCM}_{E}=\sum_{i=0}^{N-1}\sum_{j=0}^{N-1}[P(i,j)]^{2},\vspace{-4mm}$
(1)

where $P(i,j)$ represents the joint probability of occurrence of pixel pairs
with intensity values $i$ and $j$ at a specified spatial relationship. A
higher Energy (E) score suggests that there are fewer variations in intensity
in an image, indicating more uniform or repetitive patterns. In the process of
removing noise or irregularities while generating images, generative
techniques may blur some texture details. This can leave their signature in
the $\text{GLCM}_{E}$.

The second metric, GLCM Contrast, quantifies the difference or change between
adjacent pixels present in the image. Such differences or changes are
indicative of the texture contrast in the image. The Contrast $C$ is defined
as

$\text{GLCM}_{C}=\sum_{i=0}^{N-1}\sum_{j=0}^{N-1}(i-j)^{2}P(i,j),\vspace{-2mm}$
(2)

where $P(i,j)$ represents the probability that a pixel with intensity $i$ co-
occurs with a neighboring pixel of intensity $j$ in a specified spatial
relationship. A higher value of Contrast implies more variations in intensity
between a pixel and its neighboring pixels across the image. Smooth and
blurred generated textures are expected to result in lower GLCM Contrast
values for generated images.

Canny Edge Density [52] refers to the proportion of pixels in an image that
are identified as edges using the Canny edge detection technique. Given the
total number of pixels $I$ and number of edge pixels $E$ detected by the Canny
edge detector, the Canny Edge Density, $D$, can be defined as

$\text{CED}=\frac{E}{I}.\vspace{-2mm}$ (3)

Here, CED $\in[0,1]$ represents the proportion of the edge pixels in the
image. Higher CED value indicates high edge density. A visually appealing,
generated images can be expected to differ from natural images in terms of
their edge pixel distribution. Hence, Canny Edge Density is another helpful
statistic for quantifying realism in images.

Variance Blur Measure (VBM) [23] is used to estimate the sharpness or
blurriness of an image. The process involves computing the variance of an
image after applying a Laplacian filter. For an image with dimensions $M\times
N$, the VBM is measured as,

$\text{VBM}=\frac{1}{M\times
N}\sum_{i=1}^{M}\sum_{j=1}^{N}(L(i,j)-\mu)^{2},\vspace{-2mm}$ (4)

where $L(i,j)$ is the pixel value at position $(i,j)$ in the Laplacian
filtered image and $\mu$ represents the mean pixel value of the Laplacian
filtered image. Due to the differences in smoothness between real and fake
images, VBM is expected to generate different score in fake images.

The Mean Spectrum (MS) [20] is a concept used in the frequency domain analysis
of images. When an image undergoes a Fourier Transform, it produces a spectrum
with both magnitude and phase components for each frequency. The MS provides
an average measure of the spectrum magnitude. For an image with dimensions
$M\times N$ and its Fourier Transform $F(u,v)$, the Mean Spectrum is given as

$\text{MS}=\frac{1}{M\times N}\sum_{u=0}^{M-1}\sum_{v=0}^{N-1}|F(u,v)|.$ (5)

Here, $|F(u,v)|$ represents the magnitude at frequency coordinates $(u,v)$.
Image generation usually involves blending faces or objects from various
sources, often introducing subtle artifacts [54]. By leveraging the Mean
Spectrum, these inconsistencies can be brought to light.

###  3.2 Image Realism Score

Real images exhibit natural attributes, which directly result from natural
scenes. On the contrary, attributes of generated images are dependent on the
processes underlying the generative model. This can lead to unnatural image
statistics for the generated images. The key intuition behind our Image
Realism Score (IRS) is to scrutinize the primitive image attributes to
quantify the realism of the image content. We do so by combining the above-
mentioned five image statistics in our IRS. Our technique enables leveraging
the numerical differences in the values of the image statistics to distinguish
between real and fake images.

####  3.2.1 Sort Order of the Measures

The five measures we opt (refer Section 3.1) to define IRS capture largely
unrelated primitive statistics of images. To support this argument, we report
the correlations between the five measures of ten thousand random images from
the ImageNet dataset [46] in Table 1. Since the chosen measures eventually
just provide numerical values, simply combining them using basic arithmetic
operations is ineffective for our ultimate objective. Hence, we devise a
unique strategy to maximize the collective information we can extract from our
measures. The computation method of our IRS requires defining a graph using
the measures. For $n=5$ measures that have equal importance for IRS, we choose
pentagon as the base graph geometry. Each edge of the graph, which translates
to the radii of the pentagon, signifies one of the used measures, see Fig. 2.

Table 1: Correlation coefficient between five image statistical measures. CED, GLCMC, GLCME, VBM, and MS represent Canny Edge Density, GLCM Cantrast, GLCM Energy, Variance Blur Measure, and Mean Spectrum, respectively. | CED |  GLCMC |  GLCME | VBM | MS
---|---|---|---|---|---
CED | - | 0.76 | -0.28 | 0.05 | 0.21
GLCMC | 0.76 | - | -0.13 | 0.20 | 0.14
GLCME | -0.28 | -0.13 | - | 0.21 | -0.35
VBM | 0.05 | 0.20 | 0.21 | - | -0.07
MS | 0.21 | 0.14 | -0.35 | -0.07 | -

The radii can be used to construct five triangles comprising the pentagon.
Eventually, we use the areas of these triangles to compute the area of the
pentagon that defines IRS. For our geometric graph shape, a triangle has the
central angle $\theta$. The area of the triangle delineated by two radii, say
$m_{a}$ and $m_{b}$, can be computed as

$A_{\triangle_{a,b}}=\frac{m_{a}\cdot
m_{b}}{2}\cdot\sin\left(\theta\right).\vspace{-2mm}$ (6)

Denote the radii of the pentagon as $m_{1},m_{2},...,m_{5}$. Since each radii
signifies a unique measure, their order matters for the ultimate IRS value.
The order of the radii can vary. Initial combinations of two radii that are
adjacent to, say $m_{1}$ can be calculated as

$C=\frac{(n-1)!}{x!(n-x-1)!},\vspace{-2mm}$ (7)

where $x=2$ is the number adjacent radii and $n=5$ are the total radii. This
leads to $C=6$ combinations. For these combinations, the adjacent radii can be
arranged in $2$ ways. Hence, the total number of combinations is $6\times
2=12$.

Figure 3: In row 1 and row 2, the blue normalized pentagon is computed as the
average of 10,000 ImageNet (real) images. Red pentagons are formed by the
average of 1,000 images generated by the four models. In row 2, the MS, VBM,
and GLCME values are inverted for generated images. In row 3, the red and blue
pentagons are re-scaled by the same ratio so that the red pentagons become
uniform, which makes the blue pentagons nonuniform.

To find a suitable order from the 12 combinations of metrics in the pentagon,
we leverage their mutual correlations. Based on the empirical results in Table
1, we arrange the two measures with the largest correlation as adjacent radii
and continue in a descending order. The intuition here is that this systematic
arrangement can lead to larger areas, which are eventually desirable for IRS
for its discriminative abilities. To validate this intuition, we conducted
experiments on 10,000 random ImageNet images, calculating and ranking areas
for all 12 arrangements. We recorded that this arrangement resulted in the
maximum area $\sim 37\%$ times whereas the remaining 63% was distributed
between the other 11 arrangements. Finally, we express the area of a pentagon
formed by a fixed sequence of radii as

$A=\sum_{(a,b)\in S}A_{\triangle_{a,b}},\vspace{-2mm}$ (8)

where $S\in\\{(1,2),(1,5),(2,3),(3,4),(4,5)\\}$.

####  3.2.2 Calibration of the Statistical Measures

We have carefully chosen the measures to be incorporated in our metric.
However, each measure has its own numerical variability. This calls for a
calibration before combining them into IRS. In Fig. 3(row-1), we show (red)
pentagons of fake images that get formed for the four generative models used.
We also provide a (blue) reference pentagon in each sub-figure that
corresponds to the normalized values of the five measures resulting from real
images. It can be noticed that the radii GLCME, VBN and MS of the (red)
generated image pentagons are often exceeding the radii of the reference
(blue) pentagon. The reason behind their behavior is as follows.

It is known that generative techniques do not capture all subtle textural
details found in real images [3]. This results in relatively smoother images
which in turn leads to higher GLCME, as well as higher VBM. Similarly, image
generation can lead to inconsistencies in noise distribution, which get more
pronounced in the frequency domain [31]. This causes relatively higher MS
values for the generated images. The values of the other two measure are
loosely upper-bounded by their real image values. Hence, we take
multiplicative inverses of VBM, MS and GLCME. This leads to polygons that are
confined much better within the real image pentagons, see Fig. 3(row-2).
Before further processing, we normalize the (red) pentagons of fake images and
scale the (blue) pentagons of real images by the same proportion as shown in
Fig. 3(row-3). These two steps are helpful because our technique uses polygon
area comparison for fake detection is Section 3.3.

In Table 2, we report the average values of the measures ‘before’ and ‘after’
their calibration. More precisely, the ‘before’ calibration case corresponds
to row-2 of Fig. 3, which already accounts for VBM, MS and GLCME variability.
However, notice that the difference between the average areas of real and fake
images is still only 0.88. To amplify this difference, we re-scale the
measures for fake images to 1.0, and also re-scale the measures of real images
correspondingly (see Fig. 3(row-3)) which increases the difference between the
average areas to 2.30. After this re-calibration, the eventual IRS value is
computed as

$\begin{split}{\rm IRS}=\sum_{(a,b)\in S}w_{a}\cdot w_{b}\cdot
A_{\triangle_{a,b}},\vspace{-4mm}\end{split}$ (9)

where $w_{a}$ and $w_{b}$ are the weights resulting from the re-scaling of the
corresponding measures.

Table 2: Calibration of the statistical measures. Average values are reported for all. After calibration, the difference between the average IRS of real and fake images increases considerably. Metrics |  GLCMC |  GLCME | CID | VBM | MS | IRS
---|---|---|---|---|---|---
|  | Before | Calibration |  |  |
Real Images | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 2.38
Fake Images | 0.43 | 0.97 | 0.64 | 0.90 | 0.98 | 1.50
|  | After | Calibration |  |  |
Real Images | 2.31 | 1.02 | 1.57 | 1.11 | 1.02 | 4.68
Fake Images | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 2.38

###  3.3 Fake Detection

Our proposed metric can be evaluated on a single image, making it particularly
suitable for detecting fake content on a sample-by-sample basis. This type of
detection is not possible using traditional metrics such as IS and FID, which
typically rely on large datasets. For fake content detection, the IRS adopts a
thresholding technique. The last column in Table 2 shows that the average
areas of real and fake images generally vary greatly under our proposed
scheme. We capitalize on this observation, and use a threshold
$\delta:\text{IRS}<\delta\implies\text{Fake}$, where $\delta=3$ is set
empirically in our experiments. This simple yet effective approach ensures
efficient fake image detection without the need for complex calculations.

###  3.4 Gen-100: Dataset of Generated Images

Another important contribution of this paper is the creation of an new
dataset, Gen-100. In addition to being used to evaluate the effectiveness of
our method, this dataset can also be used for benchmarking future evaluation
metrics. The Gen-100 data is generated using several popular image generation
models, including SDM, BigGAN, Dalle2, and Midjourney. It consists of 100
object categories, where all categories follow CIFAR100 [28] categories. We
use the aforementioned models to generate 1,000 images for each category. The
real counterparts of the synthetic images are extracted from the same class
labels of ImageNet [46]. We use ChatGPT [36] to generate 10 prompts for each
category, which are used to further generate the text-conditioned images from
the models. This process allows us to guarantee diversity in the dataset,
while also capturing advanced abilities of the models. Due to the prevalent
trend of non-open-source diffusion-based generative models that require
payment for access, a void exists in the realm of comparative datasets
comprising images generated by different diffusion models. Our dataset fills
this gap. The dataset will be made public after acceptance.

###  3.5 Improving Image Generation

Diffusion models (DMs) [50] learn data distributions through a unique
approach. The core idea is to learn the distribution $p(x)$ by consecutively
denoising a variable that follows a normal distribution. This progressive
denoising can be conceived as backtracking a Markov Chain of fixed length $T$.
An objective function is defined to gauge the denoising process as follows

$L_{DM}=\mathbb{E}_{\epsilon(x),t,\epsilon}\left[||\epsilon-\epsilon_{\theta}(x_{t},t)||^{2}\right],$
(10)

where $t$ is a time step, uniformly chosen from the range {1, …, T}, $x_{t}$
represents the noisy version of the input at time $t$,
$\epsilon_{\theta}(x_{t},t)$ signifies the model’s denoised prediction at time
$t$ and $L_{DM}$ is the model loss. Following DMs, a Latent Diffusion Model
(LDM) [43] generates images by iterating over denoised data in a latent
representation space, and then decode the representation results into full
images. Its loss objective is defined as

$L_{LDM}=\mathbb{E}_{\epsilon(x),t,\epsilon}\left[||\epsilon-\epsilon_{\theta}(z_{t},t)||^{2}\right],$
(11)

were, $z_{t}$ is the representation in the latent space, abstracting the
input’s finer details.

The widely popular Stable Diffusion Model [43] is an LDM. To improve the
realism in its generated images, we incorporate our IRS metric into the models
training loss objective. Specifically, we minimize
$\frac{1}{\text{IRS(image)}}$ because real images have large IRS values, see
Table 2. This encourages the model to generate more realistic images by
minimizing the training loss. It is notable that we are able to compute IRS
values on per-image basis here, which allows us to easily modify the training
objective of SDM. The improved training objective of the model is defined as

$L_{IRS}=\mathbb{E}_{\epsilon(x),t,\epsilon}\left[||\epsilon-\epsilon_{\theta}(z_{t},t)||^{2}\right]+\frac{\xi}{{\rm
IRS}(d(z_{t}))},$ (12)

were, $d$ is the decoding function of the original SDM approach, and $\xi$ is
a scaling factor for our regularization.

Table 3: Average IRS values of different generative models computed on 1K images from Gen-100 dataset. The value for Real images is computed for 10K ImageNet samples. Model Name | SDM | Dalle2 | Midjourney | BigGAN | Real
---|---|---|---|---|---
IRS score | 2.29 | 1.58 | 2.03 | 1.74 | 4.68

##  4 Experiments

Benchmarking generative models:  We first benchmark the popular generative
models using our metric. In the experiment, we employ our Gen-100 dataset -
see Sec. 3.4. The average IRS values for the SDM, Dalle2, Midjourney and
BigGAN models are reported in Table 3. The table also includes the average
value of 10K real images as a reference. It can be seen that the IRS values
are generally in accordance with the known abilities of the models.
Interestingly, whereas Midjourney is popular for its high quality images, it
scores lower than SDM on our metric. This is because despite their high
quality, Midjournay images lack in ‘realism’ as compared to SDM, and IRS is
intended to quantify realism. In Fig. 4, we show representative images
generated by each model for high and low IRS values. It is easily noticeable
that images with high IRS values indeed contain details that make them appear
highly realistic. On the other hand, images with low IRS values are indeed of
cartoonic nature, lacking in realism.

Figure 4: Representative generated images for different generative models that
led to high and low IRS values. Images with higher IRS are clearly more
realistic.

Fake Detection: In Table 4, we report the results of using IRS for fake image
detection. For this experiment, we use random 500 samples for each model from
the Gen-100 dataset and following Sec. 3.3, we use $\delta=3.0$ as the
threshold value. The results are reported with the standard metrics of
accuracy, F1 score, recall and precision. It is noticeable that the results
are generally in accordance with the realism quality reported in Table 3. That
is, the model with the highest IRS score from Table 3, i.e., SDM, has the
lowest detection rate. However, Dalle2 is still able to maintain a lower
detection rate than BigGAN despite its smaller IRS value in Table 3. We find
that this is due to the larger versatility of Dalle2 images, which allows
samples to score relatively high IRS with more frequency, instead of scoring
very large IRS on a few images to achieve higher average IRS. This observation
is inline with the general understanding that diffusion based models are more
versatile than GANs.

Table 4: Fake detection results. The dataset for each model consists of 500 Gen-100 samples and 500 real images from ImageNet. Dataset Name | BigGAN | SDM | Dalle2 | Midjourney |
---|---|---|---|---|---
Accuracy | 0.85 | 0.76 | 0.81 | 0.79 |
F1 Score | 0.87 | 0.68 | 0.79 | 0.77 |
Recall | 0.95 | 0.71 | 0.77 | 0.81 |
Precision | 0.81 | 0.73 | 0.81 | 0.78 |

Improving SDM : We improved SDM with our technique following Sec. 3.5, and
compare results with the original SDM. For a fair comparison, we use the exact
same configuration for both models. As shown in Fig. 5, the IRS-augmented SDM
produces images with sharper edges and clearer separation from the background.
This makes the main subject in the image easier to identify. Secondly, the
color transition is more natural, without abrupt color blocks or obvious
artificial traces. In addition, the shadow and lighting effects presented by
the improved model are more realistic. The images generated by the IRS-
augmented SDM present finer textures, and both the subject and the background
can clearly show their unique texture features.

Figure 5: Comparison of generated images. One set is produced with the IRS-
augmented SDM, and the other generated by the original SDM. Prompts used to
generate the images are also given. Images generated by the proposed modified
loss showcase richer details, aligning with the essence of our technique.

##  5 Further Discussion

Currently, there are two popular metrics to measure the generative model
quality, namely; Inception Score (IS) [48] and Fréchet Inception Distance
(FID) [24]. However, both these metrics have their limitations [4, 11, 35].
For instance, the underlying classification model of IS adopts the Inception
V3 architecture and is trained on the ImageNet dataset. Therefore, for
unbiased evaluation using IS, generative models and classifiers are best
trained on the ImageNet dataset. Additionally, sample size can also affect IS
output. Moreover, insufficient samples may adversely affect IS results.
Similar concerns are also valid for FID. In contrast, our IRS eliminates the
need for training, mitigating the risk of overfitting. Its design premise
allows evaluation based on only a single image, avoiding sample size concerns.

In addition to the above-noted well-known shortcomings of IS and FID that have
been verified in other papers, we also test rotation invariance of these
metrics. We performed random small rotations on the dataset and evaluated
their impact on the IS and FID score. As shown in the Table 5, after rotation,
the value of IS decreases and the value of FID increases. This means that for
IS and FID, the same image can become more “fake” after rotation. In contrast,
IRS uses only the most basic information of the image itself, which is not
affected by rotations. This is an added advantage of our proposed metric.

Table 5: Image rotation test with IS and FID. Each test dataset contains 1,000 generated images from the Gen-100 dataset. Dataset source | MidJourney | BigGAN | SDM | Dalle2
---|---|---|---|---
IS before rotation | 4.84 | 3.45 | 3.70 | 3.78
IS after rotation | 4.35 | 3.15 | 3.60 | 3.34
FID before rotation | 3135.00 | 2234.86 | 2769.05 | 2550.84
FID after rotation | 3389.25 | 2700.14 | 2980.22 | 2784.39

##  6 Conclusion

In recent years, the development of diffusion models has made it more
difficult for humans to distinguish real and fake visual content. Therefore,
exploring metrics for evaluating the authenticity of generated visual content
is important. Our proposed Image Realism Score (IRS) addresses the
shortcomings of existing metrics, such as the inability to analyze the
authenticity of individual images and difficulties in achieving good results
with tests on a new dataset different from the training sets. IRS avoids the
limitations of current metrics by computing a fusion of five statistical
measures of the input image. IRS is a non-learning based metric that does not
rely on heavy computational resources. Using IRS, we also successfully
detected fake images generated by Stable Diffusion Model (SDM), Dalle2,
Midjourney, and BigGAN (GAN), establishing the model-agnostic and data-
independent nature of IRS. Furthermore, when IRS was incorporated in the loss
function of SDM, the model performance was shown to improve.

##  7 ACKNOWLEDGMENTS

This research was supported by National Intelligence and Security Discovery
Research Grants (project$\\#$ NS220100007), funded by the Department of
Defence Australia. Professor Ajmal Mian is the recipient of an Australian
Research Council Future Fellowship Award (project number FT210100268) funded
by the Australian Government.

## References

  * [1] Darius Afchar, Vincent Nozick, Junichi Yamagishi, and Isao Echizen.  Mesonet: a compact facial video forgery detection network.  In 2018 IEEE international workshop on information forensics and security (WIFS), pages 1–7. IEEE, 2018.
  * [2] Hamed Alqahtani, Manolya Kavakli-Thorne, Gulshan Kumar, and Ferozepur SBSSTC.  An analysis of evaluation metrics of gans.  In International Conference on Information Technology and Applications (ICITA), volume 7, 2019.
  * [3] Sanjeev Arora and Yi Zhang.  Do gans actually learn the distribution? an empirical study.  arXiv preprint arXiv:1706.08224, 2017.
  * [4] Shane Barratt and Rishi Sharma.  A note on the inception score.  arXiv preprint arXiv:1801.01973, 2018.
  * [5] Georgios Batzolis, Jan Stanczuk, Carola-Bibiane Schönlieb, and Christian Etmann.  Conditional image generation with score-based diffusion models.  arXiv preprint arXiv:2111.13606, 2021.
  * [6] Mikołaj Bińkowski, Danica J Sutherland, Michael Arbel, and Arthur Gretton.  Demystifying mmd gans.  arXiv preprint arXiv:1801.01401, 2018.
  * [7] Andrew Brock, Jeff Donahue, and Karen Simonyan.  Large scale gan training for high fidelity natural image synthesis.  arXiv preprint arXiv:1809.11096, 2018.
  * [8] Hanqun Cao, Cheng Tan, Zhangyang Gao, Guangyong Chen, Pheng-Ann Heng, and Stan Z Li.  A survey on generative diffusion model.  arXiv preprint arXiv:2209.02646, 2022.
  * [9] Ting-Yun Chang and Chi-Jen Lu.  Tinygan: Distilling biggan for conditional image generation.  In Proceedings of the Asian Conference on Computer Vision, 2020\.
  * [10] Yunzhuo Chen, Naveed Akhtar, Nur Al Hasan Haldar, and Ajmal Mian.  Deepfake detection with spatio-temporal consistency and attention.  In 2022 International Conference on Digital Image Computing: Techniques and Applications (DICTA), pages 1–8. IEEE, 2022.
  * [11] Min Jin Chong and David Forsyth.  Effectively unbiased fid and inception score and where to find them.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 6070–6079, 2020.
  * [12] Davide Cozzolino, Giovanni Poggi, and Luisa Verdoliva.  Recasting residual-based local descriptors as convolutional neural networks: an application to image forgery detection.  In Proceedings of the 5th ACM workshop on information hiding and multimedia security, pages 159–164, 2017.
  * [13] Antonia Creswell, Tom White, Vincent Dumoulin, Kai Arulkumaran, Biswa Sengupta, and Anil A Bharath.  Generative adversarial networks: An overview.  IEEE signal processing magazine, 35(1):53–65, 2018.
  * [14] Giannis Daras and Alexandros G Dimakis.  Discovering the hidden vocabulary of dalle-2.  arXiv preprint arXiv:2206.00169, 2022.
  * [15] Valentin De Bortoli, James Thornton, Jeremy Heng, and Arnaud Doucet.  Diffusion schrödinger bridge with applications to score-based generative modeling.  Advances in Neural Information Processing Systems, 34:17695–17709, 2021.
  * [16] Minh N Do and Martin Vetterli.  The contourlet transform: an efficient directional multiresolution image representation.  volume 14, pages 2091–2106. IEEE, 2005.
  * [17] Carl Doersch.  Tutorial on variational autoencoders.  arXiv preprint arXiv:1606.05908, 2016.
  * [18] Patrick Esser, Robin Rombach, Andreas Blattmann, and Bjorn Ommer.  Imagebart: Bidirectional context with multinomial diffusion for autoregressive image synthesis.  Advances in neural information processing systems, 34:3518–3532, 2021.
  * [19] Robert Fortet and Edith Mourier.  Convergence de la répartition empirique vers la répartition théorique.  In Annales scientifiques de l’École Normale Supérieure, volume 70, pages 267–285, 1953.
  * [20] Joel Frank, Thorsten Eisenhofer, Lea Schönherr, Asja Fischer, Dorothea Kolossa, and Thorsten Holz.  Leveraging frequency analysis for deep fake image recognition.  In International conference on machine learning, pages 3247–3258. PMLR, 2020.
  * [21] Charles J Geyer.  Practical markov chain monte carlo.  Statistical science, pages 473–483, 1992.
  * [22] Giorgio Giannone, Didrik Nielsen, and Ole Winther.  Few-shot diffusion models.  arXiv preprint arXiv:2205.15463, 2022.
  * [23] Xiaofei He, Deng Cai, and Partha Niyogi.  Laplacian score for feature selection.  Advances in neural information processing systems, 18, 2005.
  * [24] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter.  Gans trained by a two time-scale update rule converge to a local nash equilibrium.  Advances in neural information processing systems, 30, 2017.
  * [25] Jonathan Ho, Ajay Jain, and Pieter Abbeel.  Denoising diffusion probabilistic models.  NIPS, 33:6840–6851, 2020.
  * [26] Emiel Hoogeboom, Alexey A Gritsenko, Jasmijn Bastings, Ben Poole, Rianne van den Berg, and Tim Salimans.  Autoregressive diffusion models.  arXiv preprint arXiv:2110.02037, 2021.
  * [27] Emiel Hoogeboom, Didrik Nielsen, Priyank Jaini, Patrick Forré, and Max Welling.  Argmax flows and multinomial diffusion: Learning categorical distributions.  Advances in Neural Information Processing Systems, 34:12454–12465, 2021.
  * [28] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton.  Imagenet classification with deep convolutional neural networks.  Advances in neural information processing systems, 25, 2012.
  * [29] Yann LeCun, Sumit Chopra, Raia Hadsell, M Ranzato, and Fujie Huang.  A tutorial on energy-based learning.  Predicting structured data, 1(0), 2006.
  * [30] Haoying Li, Yifan Yang, Meng Chang, Shiqi Chen, Huajun Feng, Zhihai Xu, Qi Li, and Yueting Chen.  Srdiff: Single image super-resolution with diffusion probabilistic models.  Neurocomputing, 479:47–59, 2022.
  * [31] Babak Mahdian and Stanislav Saic.  Using noise inconsistencies for blind image forensics.  Image and vision computing, 27(10):1497–1503, 2009.
  * [32] Jiquan Ngiam, Zhenghao Chen, Pang W Koh, and Andrew Y Ng.  Learning deep energy models.  In Proceedings of the 28th international conference on machine learning (ICML-11), pages 1105–1112, 2011.
  * [33] Huy H Nguyen, Junichi Yamagishi, and Isao Echizen.  Capsule-forensics: Using capsule networks to detect forged images and videos.  In ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 2307–2311. IEEE, 2019.
  * [34] Alexander Quinn Nichol and Prafulla Dhariwal.  Improved denoising diffusion probabilistic models.  In International Conference on Machine Learning, pages 8162–8171. PMLR, 2021.
  * [35] Artem Obukhov and Mikhail Krasnyanskiy.  Quality assessment method for gan based on modified metrics inception score and fréchet inception distance.  In Software Engineering Perspectives in Intelligent Systems: Proceedings of 4th Computational Methods in Systems and Software 2020, Vol. 1 4, pages 102–114. Springer, 2020.
  * [36] OpenAI.  Gpt-4 technical report, 2023.
  * [37] Achraf Oussidi and Azeddine Elhassouny.  Deep generative models: Survey.  In 2018 International conference on intelligent systems and computer vision (ISCV), pages 1–8. IEEE, 2018.
  * [38] Samay Pashine, Sagar Mandiya, Praveen Gupta, and Rashid Sheikh.  Deep fake detection: Survey of facial manipulation detection solutions.  arXiv preprint arXiv:2106.12605, 2021.
  * [39] Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen.  Hierarchical text-conditional image generation with clip latents.  arXiv preprint arXiv:2204.06125, 1(2):3, 2022.
  * [40] Nicholas Rescher.  Mid-journey: An unfinished autobiography.  1983\.
  * [41] Danilo Rezende and Shakir Mohamed.  Variational inference with normalizing flows.  In International conference on machine learning, pages 1530–1538. PMLR, 2015.
  * [42] Janko Roettgers.  Porn producers offer to help hollywood take down deepfake videos.  Variety. February, 21, 2018.
  * [43] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer.  High-resolution image synthesis with latent diffusion models.  In CVPR, pages 10684–10695, 2022.
  * [44] Robin Rombach, Andreas Blattmann, and Björn Ommer.  Text-guided synthesis of artistic images with retrieval-augmented diffusion models.  arXiv preprint arXiv:2207.13038, 2022.
  * [45] Andreas Rossler, Davide Cozzolino, Luisa Verdoliva, Christian Riess, Justus Thies, and Matthias Nießner.  Faceforensics++: Learning to detect manipulated facial images.  In Proceedings of the IEEE/CVF international conference on computer vision, pages 1–11, 2019.
  * [46] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al.  Imagenet large scale visual recognition challenge.  International journal of computer vision, 115:211–252, 2015.
  * [47] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al.  Photorealistic text-to-image diffusion models with deep language understanding.  Advances in Neural Information Processing Systems, 35:36479–36494, 2022.
  * [48] Tim Salimans, Ian Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen.  Improved techniques for training gans.  Advances in neural information processing systems, 29, 2016.
  * [49] Bino Sebastian V, A Unnikrishnan, and Kannan Balakrishnan.  Gray level co-occurrence matrices: generalisation and some new features.  arXiv preprint arXiv:1205.4831, 2012.
  * [50] Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli.  Deep unsupervised learning using nonequilibrium thermodynamics.  In ICML, pages 2256–2265. PMLR, 2015.
  * [51] Christian Szegedy, Sergey Ioffe, Vincent Vanhoucke, and Alexander Alemi.  Inception-v4, inception-resnet and the impact of residual connections on learning.  In Proceedings of the AAAI conference on artificial intelligence, volume 31, 2017.
  * [52] Taqi Tahmid and Eklas Hossain.  Density based smart traffic control system using canny edge detection algorithm for congregating traffic information.  In 2017 3rd International Conference on Electrical Information and Communication Technology (EICT), pages 1–5. IEEE, 2017.
  * [53] Kunfeng Wang, Chao Gou, Yanjie Duan, Yilun Lin, Xinhu Zheng, and Fei-Yue Wang.  Generative adversarial networks: introduction and outlook.  IEEE/CAA Journal of Automatica Sinica, 4(4):588–598, 2017.
  * [54] Chaowei Xiao, Bo Li, Jun-Yan Zhu, Warren He, Mingyan Liu, and Dawn Song.  Generating adversarial examples with adversarial networks.  arXiv preprint arXiv:1801.02610, 2018.
  * [55] Xinchen Yan, Jimei Yang, Kihyuk Sohn, and Honglak Lee.  Attribute2image: Conditional image generation from visual attributes.  In Computer Vision–ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11–14, 2016, Proceedings, Part IV 14, pages 776–791. Springer, 2016.
  * [56] Sharon Zhou, Mitchell Gordon, Ranjay Krishna, Austin Narcomey, Durim Morina, and Michael S Bernstein.  Hype: human-eye perceptual evaluation of generative models.  2019\.
  * [57] Zhiming Zhou, Han Cai, Shu Rong, Yuxuan Song, Kan Ren, Weinan Zhang, Yong Yu, and Jun Wang.  Activation maximization generative adversarial nets.  arXiv preprint arXiv:1703.02000, 2017.