# Detecting Point Outliers Using Prune-based Outlier Factor (PLOF)

Kasra Babaei1, ZhiYuan Chen1,∗, Tomas Maul1

1School of Computer Science, University of Nottingham Malaysia,
∗Corresponding Author<EMAIL_ADDRESS>
Abstract Outlier detection (also known as anomaly detection or deviation
detection) is a process of detecting data points in which their patterns
deviate significantly from others. It is common to have outliers in industry
applications, which could be generated by different causes such as human
error, fraudulent activities, or system failure. Recently, density-based
methods have shown promising results, particularly among which Local Outlier
Factor (LOF) is arguably dominating. However, one of the major drawbacks of
LOF is that it is computationally expensive. Motivated by the mentioned
problem, this research presents a novel pruning-based procedure in which the
execution time of LOF is reduced while the performance is maintained. A novel
Prune-based Local Outlier Factor (PLOF) approach is proposed, in which prior
to employing LOF, outlierness of each data instance is measured. Next, based
on a threshold, data instances that require further investigation are
separated and LOF score is only computed for these points. Extensive
experiments have been conducted and results are promising. Comparison
experiments with the original LOF and two state-of-the-art variants of LOF
have shown that PLOF produces higher accuracy and precision while reducing
execution time.

Keywords Anomaly Detection, Deviation Detection, Local Outlier Factor, Outlier
Detection,

##  1 Introduction

Outlier detection (also known as anomaly detection or deviation detection) is
a process of detecting data points in which their patterns deviate
significantly from others. This definition may vary slightly based on the area
that it is used [1]. Outlier detection has been employed in various
application domains such as financial, telecommunications, medical, and
commercial industries [2].

According to [1], outliers are categorised into three major types, namely
point outliers, collective outliers, and contextual outliers. The methods used
for detecting these three types are very different from each other and often
it is impossible to employ the same method on the other two. In this paper, we
have proposed an approach for detecting point outliers and detecting
contextual or collective outliers is out of the scope of this paper.

Previous studies have proposed several methods for outlier detection, and it
is hard to find a universal categorisation for all the methods. However, [3]
categorise them into the following four groups: distribution-based, distance-
based, clustering-based, and density-based methods. While in distribution-
based methods the assumption is that outliers are generated from a
distribution that deviates from the others, in distance-based methods the
assumption is that outlier points tend to have a distance above a threshold
from a certain proportion of the others. In clustering-based methods, the data
instances are clustered and instances that do not belong to any cluster are
considered as the outlier. Lastly, a density-based method tries to compute the
density of each data instance and then compares the density with the density
of the neighbourhood. Instances that the ratio of their density and the
density of their neighbours are above a threshold are the outlier.

[4] proposed a density-based method called Local Outlier Factor (LOF) that
generates a value for each data instance that tells its outlierness. Instead
of using a threshold, top-$n$ data instances that have the highest LOF value
are considered as outliers. LOF has shown promising results, however, one of
the major drawbacks of LOF is that it is computationally very expensive, i.e.,
$O(N^{2})$ [5]. Therefore, researchers have implemented pruning-based methods
to reduce $N$, i.e., decrease computation cost. However, it has been realised
that it will be achieved only at the cost of debilitating performance of
outlier detection.

In this paper, we propose an outlier detection method based on LOF that would
decrease the execution time by bringing down the complexity. Instead of
computing LOF value for all the data instances, we perpetuate a preliminary
step in which density of each data instance is estimated and then pruning is
carried out based on the median of the density of data instances. Finally, LOF
is only computed for the remaining data instances. The second contribution is
maintaining the performance. As mentioned in Previous Studies, several
pruning-based approaches are proposed in the literature; however, most of
these works reduce execution time by sacrificing the performance. In summary,
the contributions of this paper are two-fold:

  * •

Introducing a new density estimation that reduces the execution time of LOF.

  * •

Maintaining the performance of outlier detection.

##  2 Related Studies

[4] proposed a density-based method known as LOF in which a value is given to
each data instance that shows the outlierness for that individual. Then, the
top-$n$ points with the highest LOF value are considered as outliers. Often,
the points with LOF value above $1$ are regarded as outlier [6]. A major
disadvantage of LOF is its complexity, i.e., $O(N^{2})$ where $N$ is the size
of the data [7]. The authors of [8] proposed three improvements:
$LOF^{\prime}$, $LOF^{\prime\prime}$ and $GridLOF$ to simplify the computation
of the original LOF. In $LOF^{\prime}$, merely the ratio of $MinPts{-}dist$ of
the query object and its neighbours are used to compute the LOF value. They
argued that this ratio is sufficient and it is needless to compute the
reachability distance and local reachability density. And, in
$LOF^{\prime\prime}$, they employed two $MinPts$ instead of just one to
enhance the performance of LOF. Their last enhancement, $GridLOF$, is a
pruning-based approach that removes dense areas so that computing LOF for data
instances inside the dense areas is not required anymore. However, the
drawback of $GridLOF$ is that it requires manual grid setting, which is not
always feasible.

In previous work, [6], the authors randomly divided the dataset into chunks
and then determined the nearest neighbours of each data instance only based on
the instances within the chunk that the query point belongs to. Next, using
the computed nearest neighbours they generated LRD and LOF for all the data
instances. However, this requires a precise neighbour selection and also not
all the outliers may be detected.

Another pruning approach was proposed by [9] in which a clustering algorithm
was applied in advance, and then dense clusters are pruned based on this
assumption that they contain no outliers. Next, they employed a Local
Distance-based Outlier Factor (LDOF) to the sparse and small clusters to
detect outliers. Earlier, [10] proposed a similar method that after clustering
the dataset by using $k$-means, instances that are close to the centroid of
the cluster, which they belong to, were pruned, and LDOF was used merely on
instances that were away from the centroid, i.e., outside of a predefined
radius.

[11] claimed that their approach can not only reduce the calculation rate of
LOF but also minimises the false negative rate. Their approach has two stages.
In the first stage, the data instances are clustered using $k$-medoids, which
they believe its robustness against outliers and noise is more than $k$-means.
Next, after computing a local cut-off value based on the size of the cluster,
instances that are outside of the radius are considered as potential outliers.
In the final stage, LOF only computed for potential outliers that were
obtained from the previous stage.

The authors of [12] presented a generic method for reducing the execution time
of many density-based and distance-based outlier detection algorithms. In
their method, a new density estimation called $devToMean$ was introduced that
based on its value normal data instances were pruned, then the outlier
detection method was applied only on the rest of the data. However, the
computation of $devToMean$ is expensive, therefore, they used $k$-means to
divide the dataset into small clusters, and then calculated the value of
$devToMean$ for each instance within its cluster.

##  3 Proposed Method

###  3.1 Local Outlier Factor

This algorithm was first introduced by [4]. Local outlier factor (LOF) is a
density-based algorithm that gives a score to each data point based on the
density of its local neighbourhood. This score represents the degree by which
the data point deviates from its local neighbourhood.

For any positive integer $k$, let $k$-distance be the distance between object
$p$ and its $k$ nearest objects. The $k$-distance-neighbourhood of object $p$
contains all the objects whereby their distance from $p$ are not greater than
the $k$-distance, i.e. $N_{k{\mathchar
45\relax}dist(p)}=\\{p^{\prime}{\mid}p^{\prime}{\in}D,dist(p,p^{\prime})\leq{dist_{k}(p)}\\}$.

The $k$-distance and $k$-distance-neighbourhood are used to compute the
reachability distance of an object. The reachability distance of object $p$
with respect to object $o$ is defined as:

$reach{\mathchar 45\relax}dist_{k}(p,o)=max\\{k{\mathchar
45\relax}distance(o),d(p,o)\\}$ (1)

The authors of [4] proposed a specific instantiation of $k$ for outlier
detection, which is derived from density-based algorithms. Density-based
algorithms often have two parameters. The first parameter, known as $MinPts$,
determines the minimum number of points that are needed. There is also the
second parameter that defines a volume. The two parameters are used to define
a density threshold for algorithms. However, LOF only utilises $MinPts$ and
dynamically specifies the density by employing $reach{\mathchar
45\relax}dist_{MinPts}(p,o)$, for $o{\in}N_{MinPts}(p)$. By specifying
$MinPts$, and given the $reach-dist$, the local reachability density of an
object is defined as:

${\scriptstyle
LRD_{MinPts}(p)=1/}\Bigg{(}\frac{{\sum}_{\scriptscriptstyle{o\in_{N_{MinPts}(p)}}}{\scriptscriptstyle{reach{\mathchar
45\relax}dist_{MinPts}(p,o)}}}{\scriptscriptstyle{|N_{MinPts}(p)|}}\Bigg{)}$
(2)

which is the inverse of the average reachability distance based on the
parameter $MinPts$ of object $p$. Local reachability can be $\infty$ in the
case that every reachability distance in the summation is 0, which may happen
when there are data points that have the same spatial location of object $p$,
i.e. duplicates of $p$.

Local outlier factor, which represents the deviation of each object with
respect to its local neighbourhood, is then defined as the the local
reachability density of the object $p$ divided by the local reachability
density of object $p$’s $MintPts$-nearest neighbour. It is defined as:

$LOF_{MinPts}(p)=\frac{{{\sum}_{o\in_{N_{MinPts}(p)}}{\frac{lrd_{MinPts}(o)}{lrd_{MinPts}(p)}}}}{|N_{MinPts}(p)|}$
(3)

A low LOF value indicates that the data point is an inlier, while outliers
produce greater LOF values, i.e. usually greater than 1.

###  3.2 Prune-based Local Outlier Factor (PLOF) Approach

A major drawback of the methods that were proposed previously such as [12] and
[10] is that they all need to cluster the dataset in advance, and then prune a
portion of the data instances based on a metric. This kind of approach brings
in the complexity of the clustering method into LOF. In our proposed method,
there is no need to cluster the dataset.

As explained in Section 1, to compute a score by LOF, which indicates the
outlierness of a data instance, first need to find out the reachability
distance, then local reachability distance, and finally local outlier factor.
In our proposed work, these three steps are carried out merely on data
instances that requires further investigation. This is decided based on a
pruning method. The complexity of this method is $N$, where $N$ is the number
of instances. In our pruning method, the local density of each data instance
is estimated based on the following equation:

$\delta=\frac{{|M|}^{2}}{\sum\limits_{i=0}^{n}dist(p,i)}$ (4)

where $|M|$ is the cardinality of $p$’s neighbours and $dist(p,i)$ is the
distance between $p$ and its $i$-th neighbour. As depicted in Fig. 1, we draw
a circle around point $p$ based on $k$-distance (i.e., the radius that within
that we can have at least $k$ neighbours), and then we compute $M$. To the
best of our knowledge the Equation 4 has never been proposed before. To remove
the effect of extreme values [5], the largest and the smallest values of
$\delta$ are eliminated.

$A$$g$$r$ Figure 1: Neighbourhood of A based on $k$-distance

Having a set of $\delta$ values, the median is determined and based on the
assumption that outliers should have a low $\delta$ value, all the instances
that have a $\delta$ value less than the median are pruned. This simple
median-based clustering eliminates the problem of finding a good threshold. It
is worth to mention that pruned instances are still used for computing
Equations 1, 2, and 3 for not pruned instances. Finally, the LOF value for
points that their $\delta$ value is greater than the median is assigned to 0.

1

input : $D$: a dataset with $n\times m$ dimension

output : $\lambda$: a $1d$ array

2

3initialise: $k=minimum\ neighbours$;

4 $S$ = $\delta$ values;

5 for  _each point $x_{i}$ in $D$_ do

6 $NN_{i}$ = find the $k$-th neighbour $x_{j}$ for $x_{i}$;

7 $\delta_{i}$ = compute $\delta$ for $x_{i}$ given $NN_{i}$;

8 append $\delta_{i}$ to $S$;

9

10 end for

11eliminate extreme values of $\delta$;

12 $med$ = median of $S$;

13 for  _each $x_{i}$ in $D$_ do

14 if  _$\delta_{i} <med$_ then

15 prune $x_{i}$

16 end if

17

18 end for

19for  _each $x_{i}$ in $D$_ do

20 if  _$x_{i}$ not pruned_ then

21 compute reachability-distance of $x_{i}$;

22 compute local reachability density of $x_{i}$;

23 $\rho_{i}$ = compute LOF value of $x_{i}$;

24 append $\rho_{i}$ to $\lambda$;

25

26 else

27 append $0$ to $\lambda$;

28 end if

29

30 end for

return _$\lambda$_

Algorithm 1 Efficient LOF Algorithm

A paper should have a short, straightforward title directed at general readers
in no more than 20 words.

##  4 Complexity Analysis

The complexity depends on the use of KD-tree. In case of using KD-tree, the
time complexity of finding the nearest neighbours is $O(N*\log N)$, where $N$
is the number of instances in the dataset. In case of using a Brute-Force for
getting the nearest neighbour, the complexity becomes $O(N^{2})$.

The complexity of computing $\delta$ value of each data instance is $O(N)$
while the complexity of getting median of $\delta$ values is $O(1)$. And, the
complexity of LOF is $O(N^{2})$ [5]. By pruning, the size of $N$ reduces,
i.e., ($N^{\prime}\ll N$), therefore, the complexity of computing LOF is
substantially reduced.

Table 1: The details of 6 datasets Dataset | # of features | # of outliers | # of data
---|---|---|---
Wine | 13 | 10 | 129
Lymphography | 18 | 6 | 148
Glass | 9 | 9 | 214
Ionosphere | 33 | 126 | 351
WBC | 30 | 21 | 278
Heart | 22 | 15 | 187
Breast | 9 | 239 | 683
Table 2: The execution time Dataset | PLOF | LOF | devToMean | FastLOF
---|---|---|---|---
Wine | 0.156 | 0.229 | 0.335 | 0.067
Lymphography | 0.207 | 0.396 | 0.215 | 0.079
Glass | 0.415 | 0.576 | 0.425 | 0.113
Ionosphere | 1.167 | 1.297 | 1.081 | 0.214
WBC | 1.320 | 1.719 | 1.243 | 0.238
Heart | 0.340 | 0.445 | 0.358 | 0.125
Breast | 4.336 | 5.163 | 4.335 | 0.633
Average | 1.134 | 1.403 | 1.141 | 0.209
Table 3: The accuracy Dataset | PLOF | LOF | devToMean | FastLOF
---|---|---|---|---
Wine | 0.783 | 0.946 | 0.922 | 0.682
Lymphography | 0.743 | 0.743 | 0.926 | 0.743
Glass | 0.734 | 0.659 | 0.921 | 0.715
Ionosphere | 0.877 | 0.638 | 0.661 | 0.678
WBC | 0.757 | 0.646 | 0.915 | 0.685
Heart | 0.572 | 0.519 | 0.578 | 0.540
Breast | 0.851 | 0.613 | 0.630 | 0.848
Average | 0.759 | 0.680 | 0.793 | 0.698
Table 4: The precision Dataset | PLOF | LOF | devToMean | FastLOF
---|---|---|---|---
Wine | 0.263 | 0.714 | 0.500 | 0.030
Lymphography | 0.136 | 0.136 | 0.143 | 0.056
Glass | 0.125 | 0.000 | 0.100 | 0.036
Ionosphere | 0.895 | 0.495 | 0.706 | 0.566
WBC | 0.186 | 0.000 | 0.000 | 0.062
Heart | 0.536 | 0.446 | 0.778 | 0.481
Breast | 0.701 | 0.463 | 0.150 | 0.857
Average | 0.406 | 0.322 | 0.339 | 0.298
Table 5: The AUC Dataset | PLOF | LOF | devToMean | FastLOF
---|---|---|---|---
Wine | 0.882 | 0.882 | 0.637 | 0.416
Lymphography | 0.866 | 0.866 | 0.562 | 0.547
Glass | 0.808 | 0.344 | 0.534 | 0.479
Ionosphere | 0.849 | 0.589 | 0.537 | 0.627
WBC | 0.871 | 0.342 | 0.485 | 0.520
Heart | 0.552 | 0.498 | 0.532 | 0.518
Breast | 0.885 | 0.624 | 0.487 | 0.809
Average | 0.816 | 0.592 | 0.539 | 0.559
Table 6: The recall Dataset | PLOF | LOF | devToMean | FastLOF
---|---|---|---|---
Wine | 1.0 | 1.0 | 0.300 | 0.100
Lymphography | 1.0 | 1.0 | 0.167 | 0.333
Glass | 0.889 | 0.000 | 0.111 | 0.222
Ionosphere | 0.746 | 0.413 | 0.095 | 0.444
WBC | 0.905 | 0.000 | 0.000 | 0.333
Heart | 0.357 | 0.298 | 0.083 | 0.298
Breast | 1.0 | 0.661 | 0.013 | 0.678
Average | 0.842 | 0.481 | 0.109 | 0.344

Figure 2: Showing the first two principal components of (A) Wine, (B)
Lymphography, (C) Glass, and (D) Ionosphere

##  5 Experiment Results and Analysis

The proposed method is conducted on seven real datasets obtained from UCI
Machine Learning Repository [13], which are all publicly available. In the
context of outlier detection, the datasets that are chosen in this paper are
considered as the benchmark as they have been frequently used in the
literature. The details of each dataset that is used in our experiment are
given in Table 1. It is worth to mention that because most of these datasets
are originally used in classification, some of the datasets are modified,
e.g., merging one or two major classes to form the normal class.

Various evaluation metrics have been used in the literature, however, the most
effective and widely used metric, especially for evaluating unsupervised
methods, is Receiver Operator Characteristics (ROC) curve, which basically
shows the true positive rate versus the false negative rate at different
threshold [3]. By calculating the Area Under ROC (AUC) curve, it is possible
to show the effectiveness of the algorithm by a single value. The value of AUC
can vary between 0 to 1 in which any value closer to 1 indicates better
performance while 0.5 shows a random decision making. In this paper, we have
used AUC, and also other evaluation metrics such as accuracy, precision, and
recall [14]. The experiment on each dataset is carried out five times and the
averaged outcome is reported.

We compared our approach with the original LOF algorithm [4], and also another
two similar approaches that were published recently: FastLOF [6] and devToMean
[12]. The two latter approaches are selected because their main purpose is
also to reduce the execution time of LOF. The original LOF requires merely one
parameter which is the minimum number of neighbours. But, it is possible to
also define a threshold and pick data instances that their LOF value exceeds
the threshold. In our experiment, we have used the same parameters for each
approach.

In terms of execution time and accuracy, our approach came second after
FastLOF and devToMean respectively. However, by looking at other metrics one
can deduce that [6] and [12] sacrificed detection performance, which is the
main goal, for execution time. For instance, our approach is the dominating
method based on AUC with a relatively big margin and a standard deviation of
$0.119$. Moreover, PLOF proved to be superior, on average, based on recall and
precision achieving $0.842$ and $0.406$ respectively.

By using PLOF, not only we managed to reduce execution time but also improved
detection performance.

##  6 Conclusion

Prune-based local outlier factor reduces the execution time of LOF. By
introducing a novel pre-processing step, data instances with high density are
pruned and LOF is only applied to unpruned data instances. In this way, the
complexity of LOF reduces while its performance remains unchanged. Based on
our experiment that was conducted on 8 real-life datasets, PLOF demonstrated
superior results compared with the original LOF and two more state-of-the-art
variants of LOF. In the future, we plan to apply our novel method to other
density-based methods to evaluate its performance.

## REFERENCES

  * [1] V. Chandola, A. Banerjee, and V. Kumar, “Anomaly detection: A survey,” ACM Comput. Surv., vol. 41, pp. 15:1–15:58, July 2009.
  * [2] J. Ha, S. Seok, and J.-S. Lee, “Robust outlier detection using the instability factor,” Knowledge-Based Systems, vol. 63, pp. 15–23, 2014\.
  * [3] B. Tang and H. He, “A local density-based approach for outlier detection,” Neurocomputing, vol. 241, pp. 171–180, 2017.
  * [4] M. M. Breunig, H.-P. Kriegel, R. T. Ng, and J. Sander, “Lof: Identifying density-based local outliers,” SIGMOD Rec., vol. 29, pp. 93–104, May 2000\.
  * [5] S. Su, L. Xiao, Z. Zhang, F. Gu, L. Ruan, S. Li, Z. He, Z. Huo, B. Yan, H. Wang, and S. Liu, “N2dlof: A new local density-based outlier detection approach for scattered data,” in 2017 IEEE 19th International Conference on High Performance Computing and Communications; IEEE 15th International Conference on Smart City; IEEE 3rd International Conference on Data Science and Systems (HPCC/SmartCity/DSS), pp. 458–465, 2017.
  * [6] M. Goldstein, “Fastlof: An expectation-maximization based local outlier detection algorithm,” in Proceedings of the 21st International Conference on Pattern Recognition (ICPR2012), pp. 2282–2285, Nov 2012.
  * [7] A. Sinha and P. K. Jana, “Efficient algorithms for local density based anomaly detection,” in Distributed Computing and Internet Technology (A. Negi, R. Bhatnagar, and L. Parida, eds.), (Cham), pp. 336–342, Springer International Publishing, 2018.
  * [8] A. L. M. Chiu and A. W. chee Fu, “Enhancements on local outlier detection,” in Seventh International Database Engineering and Applications Symposium, 2003. Proceedings., pp. 298–307, July 2003.
  * [9] R. Pamula, J. Deka, and S. Nandi, “Pruning based method for outlier detection,” pp. 210–213, 2012.
  * [10] R. Pamula, J. K. Deka, and S. Nandi, “An outlier detection method based on clustering,” in 2011 Second International Conference on Emerging Applications of Information Technology, pp. 253–256, Feb 2011.
  * [11] H. Rizk, S. Elgokhy, and A. Sarhan, “A hybrid outlier detection algorithm based on partitioning clustering and density measures,” in 2015 Tenth International Conference on Computer Engineering Systems (ICCES), pp. 175–181, Dec 2015.
  * [12] S. Poddar and B. K. Patra, “Reduction in execution cost of k-nearest neighbor based outlier detection method,” in Mathematics and Computing (D. Ghosh, D. Giri, R. N. Mohapatra, E. Savas, K. Sakurai, and L. P. Singh, eds.), (Singapore), pp. 53–60, Springer Singapore, 2018.
  * [13] D. Dheeru and E. Karra Taniskidou, “UCI machine learning repository,” 2017.
  * [14] G. Campos, A. Zimek, J. Sander, R. Campello, B. Micenková, E. Schubert, I. Assent, and M. Houle, “On the evaluation of unsupervised outlier detection: measures, datasets, and an empirical study,” Data Mining and Knowledge Discovery, pp. 1–37, 2015.


# RSDiff: Remote Sensing Image Generation from Text Using Diffusion Model

Ahmad Sebaq
Nile University
<EMAIL_ADDRESS>Mohamed ElHelw
Nile University
<EMAIL_ADDRESS>
###### Abstract

Satellite imagery generation and super-resolution are pivotal tasks in remote
sensing, demanding high-quality, detailed images for accurate analysis and
decision-making. In this paper, we propose an innovative and lightweight
approach that employs two-stage diffusion models to gradually generate high-
resolution Satellite images purely based on text prompts. Our innovative
pipeline comprises two interconnected diffusion models: a Low-Resolution
Generation Diffusion Model (LR-GDM) that generates low-resolution images from
text and a Super-Resolution Diffusion Model (SRDM) conditionally produced. The
LR-GDM effectively synthesizes low-resolution by (computing the correlations
of the text embedding and the image embedding in a shared latent space),
capturing the essential content and layout of the desired scenes.
Subsequently, the SRDM takes the generated low-resolution image and its
corresponding text prompts and efficiently produces the high-resolution
counterparts, infusing fine-grained spatial details and enhancing visual
fidelity. Experiments are conducted on the commonly used dataset, Remote
Sensing Image Captioning Dataset (RSICD). Our results demonstrate that our
approach outperforms existing state-of-the-art (SoTA) models in generating
satellite images with realistic geographical features, weather conditions, and
land structures while achieving remarkable super-resolution results for
increased spatial precision.

##  1 Introduction

Satellite imagery is crucial in various domains, including remote sensing,
climate monitoring, and urban planning [6]. The ability to generate high-
quality satellite images from text prompts has significant implications for
data augmentation, simulation, and enhancing the accessibility of satellite
data in resource-constrained environments [35, 38]. Traditional methods for
generating satellite images often rely on convolutional neural networks (CNNs)
[27] or generative adversarial networks (GANs) [7], but they demand large
datasets and considerable computational resources [3, 1, 39].
Diffusion models [9] offer several benefits in the realm of data generation.
Firstly, they provide a powerful framework for modeling complex distributions
in high-dimensional spaces. By iteratively applying a diffusion process, these
models can capture intricate dependencies and generate realistic samples that
align with the true underlying data distribution. This capability makes
diffusion models well-suited for tasks such as data augmentation, where
diverse and plausible samples are required to enhance the training process of
deep learning models.
In the domain of image super-resolution, diffusion models also bring
noteworthy advantages [10]. One key strength lies in their ability to generate
high-quality images at different resolutions through a sequential process. By
starting with a lower-resolution image and gradually refining it, diffusion
models excel at capturing global dependencies and preserving fine details.
This step-wise approach allows for efficient processing and facilitates the
modeling of realistic textures and structures in real-world images. Moreover,
diffusion models can be trained using large-scale datasets, enabling the
acquisition of prior knowledge that aids in producing visually appealing and
faithful super-resolved images.
In this paper, we present a pioneering approach that leverages the power of
diffusion models to generate satellite images from text prompts. We propose a
novel pipeline composed of two diffusion models: an LR-GDM and a SRDM. The LR-
GDM generates low-resolution satellite images from text prompts, capturing the
essential content and layout of the desired scenes. The SRDM then takes these
low-resolution images as input and super-resolves them, infusing fine-grained
spatial details and enhancing visual fidelity to produce high-resolution
satellite images. This dual diffusion model approach enables us to generate
realistic and visually compelling satellite imagery aligned with the specified
textual descriptions.

The main contributions of this work are as follows:

  * •

We propose a novel pipeline that combines two diffusion models, allowing us to
generate high-resolution satellite images from text prompts efficiently. The
stepwise generation process ensures better control over image synthesis and
enhanced spatial precision.

  * •

Experiments show that our diffusion pipeline performs better on satellite
image synthesis. It achieves the new Fréchet Inception Distance (FID) SoTA for
image synthesis with only $\sim 0.75$ billion parameters in total.

##  2 Related Work

###  2.1 Generative Adversarial Networks

The challenge of generating images from textual descriptions is a significant
and complex undertaking, with the objective of producing realistic visuals
based on natural language prompts. The initial investigations into text-to-
image generation predominantly center around algorithms based on GANs. The
initial groundbreaking research involves the utilization of a text-conditional
GAN [3]. In this framework, the generator is specifically engineered to
produce realistic images by leveraging extracted text properties and deceiving
the discriminator. Conversely, the discriminator’s objective is to accurately
distinguish whether the input image is authentic or fabricated. Nevertheless,
this particular approach is limited to producing images that possess a spatial
dimension of 128x128.

In order to enhance the spatial resolution of generated images, scholars have
explored the inclusion of important point locations as supplementary input for
the generator. Reed et al [21]. proposed the utilization of the generative
adversarial what-where network as a means to synthesize images with a spatial
scale of 128x128. Another notable contribution is the StackGAN framework [37],
which introduces the concept of synthesizing images using a stacked generator,
as proposed by the authors. Although the initial generator produces images of
limited quality, measuring just 64x64 in spatial scale, the subsequent
generator will utilize these images as input and enhance their synthesis to a
larger dimension of 256x256.

###  2.2 Diffusion Probabilistic Models

In addition to the utilization of GAN-based techniques, current studies have
placed emphasis on the exploration of transformer-based models. One of the
most notable works in the field is DALL-E, which can be seen as a derivative
of GPT-3 (generative pre-trained transformer-3) [2] with a parameter count of
12 billion [20]. Due to the remarkable capacity for learning exhibited by the
huge transformer model and the extensive training data consisting of 250
million text-image pairs, DALL-E demonstrates the capability to effectively
merge disparate linguistic concepts and generate visually impressive images
with a spatial resolution of 256 x 256. The publication of the second
iteration of DALL-E has occurred in the recent past[19]. DALL-E 2 exhibits
enhanced capabilities in comparison to its predecessor, DALL-E 1, since it
employs a more sophisticated artificial intelligence framework. Similar to the
CLIP (contrastive language–image pre-training) model[16], DALL-E 2 acquires
knowledge by directly assimilating the association between images and textual
descriptions. The diffusion model is capable of producing high-quality images,
both realistic and artistic in nature, with a spatial resolution of 1024x1024.
These images are generated by leveraging the provided text descriptions.

Currently, the majority of research efforts have been directed toward the
development of natural images. However, the corresponding studies within the
remote sensing community remain underdeveloped. The generation of realistic
remote sensing images from text descriptions continues to be a problem due to
the increased demands for authenticity and plausibility in application
scenarios of remote sensing jobs.

Figure 1: The RSDiff framework utilizes the T5 text encoder for the purpose of
encoding the input text into text embeddings. The conditional diffusion model
is utilized to transform the text embedding into a 128x128 picture. RSDiff
employs text-conditional super-resolution diffusion models to do image
upsampling, increasing the image resolution to 256x256 pixels.

##  3 Methodology

The predominant focus of current research in the remote sensing domain
pertaining to text-to-image generation lies in the realm of GAN-based
methodologies [3, 1, 39]. In contrast, the proposed method centers around a
diffusion-based technique. The primary component of our approach is the
sequential progression of numerous diffusion processes. The pipeline consists
of two diffusion models. Given the considerable success and exceptional output
quality of Imagen [24], we opted to adopt a similar technique in our pipeline.
The architecture of our model comprises a text encoder responsible for
transforming text into a series of embeddings. Additionally, it incorporates a
series of conditional diffusion models that progressively convert these
embeddings into images with higher resolutions.

###  3.1 Pretrained text encoders

Language models [4, 18] are trained on a corpus consisting solely of text,
which is notably bigger in size compared to paired image-text data. As a
result, these models are exposed to a vast and diverse distribution of textual
information. The size of these models is typically larger than the text
encoders seen in existing image-text models. Therefore, it is logical to
investigate both categories of text encoders for the task of text-to-image
conversion. Pretrained text encoder T5 [18, 17] is utilized in our approach as
it uses a unified framework where all tasks, both generation, and
classification, are treated as text-to-text problems. Also, T5 is hybrid as it
is trained to output one word or multiple words for one mask. This allows the
model to be flexible in learning the language structure. To ensure simplicity,
the weights of the text encoder are held constant. The process of freezing
offers numerous benefits, including the ability to do offline computation of
embeddings. This feature leads to minimal computational requirements and
memory use for training the text-to-image model. As mentioned in [24], it is
evident that increasing the size of the text encoder enhances the quality of
text-to-image generation.

###  3.2 Diffusion models and classifier-free guidance

Diffusion models [9, 29, 30] belong to a category of generative models that
transform Gaussian noise into samples derived from a learned data distribution
using an iterative denoising procedure. The models have the capability to be
conditional, such as being dependent on class labels, text, or low-resolution
photos [5, 10, 14, 19, 23, 25, 33].
The training of a diffusion model $\hat{\mathbf{x}}_{\theta}$ involves the
utilization of a denoising objective, which has the following form:

$\mathbb{E}_{\mathbf{x},\mathbf{c},\mathbf{\epsilon},t}[w_{t}||\hat{\mathbf{x}}_{\theta}(\alpha_{t}\mathbf{x}+\sigma_{t}\mathbf{\epsilon},\mathbf{c})-\mathbf{x}||^{2}_{2}]$
(1)

Where $(\mathbf{x},\mathbf{c})$ represent pairs of data and their
corresponding conditioning factors,
$t\sim\mathcal{U}([0,1]),\mathbf{\epsilon\sim\mathcal{N}(\mathbf{0},\mathbf{I})}$,
and $\alpha_{t},\sigma_{t},w_{t}$ are functions that impact the quality of the
sample. $\hat{\mathbf{x}}_{\theta}$ is trained in a manner that aims to remove
noise from $\mathbf{z}_{t}\coloneqq\alpha_{t}\mathbf{x}+\sigma_{t}\epsilon$
and produce $\mathbf{x}$, utilizing a squared error loss function that is
weighted to prioritize specific values of $t$.
The utilization of classifier guidance [5] is a method employed to enhance the
quality of samples and decrease variability in conditional diffusion models.
This is achieved by including gradients derived from a pre-trained model,
namely $p(c|z_{t})$, during the sampling process. The utilization of
classifier-free guidance [11] approach presents an alternate methodology that
circumvents the reliance on a pre-trained model. Instead, it involves the
simultaneous training of a solitary diffusion model on both conditional and
unconditional objectives. This is achieved by randomly deleting a specific
parameter, denoted as $c$ during the training process, typically with a chance
of 10%. The process of sampling is carried out with the adjusted x-prediction
technique:

$\hat{\epsilon}_{\theta}(\mathbf{z}_{t},\mathbf{c})=w\epsilon_{\theta}(\mathbf{z}_{t},\mathbf{c})+(1-w)\epsilon_{\theta}(\mathbf{z}_{t})$
(2)

Here, $\epsilon_{\theta}(\mathbf{z}_{t},\mathbf{c})$ and
$\epsilon_{\theta}(\mathbf{z}_{t})$ are conditional and unconditional
$\epsilon$-predictions, given by
$\epsilon_{\theta}\coloneqq(\mathbf{z_{t}}-\alpha_{t}\hat{\mathbf{x}}_{\theta})/\sigma_{t}$
and $w$ is the guidance weight. When the value of $w$ is set to 1, the
classifier-free guidance is disabled, but increasing the value of $w$ above 1
enhances the impact of guidance. The effectiveness of text conditioning in
RSDiff relies heavily on the utilization of classifier-free advice.

###  3.3 Cascaded diffusion models

The proposed methodology involves employing a sequential process consisting of
a foundational 128×128 model, alongside text-conditional super-resolution
diffusion models, to enhance the resolution of a generated image from 128 x
128 to 256 x 256. The utilization of cascaded diffusion models, together with
noise conditioning augmentation, has proven to be highly successful in the
gradual generation of images with a high level of fidelity [10]. In addition,
incorporating noise level conditioning into the super-resolution models
enhances the quality of the generated samples and increases the resilience of
these models in handling artifacts produced by lower-resolution models [10].
Noise conditioning augmentation is employed for both of the super-resolution
models. This aspect is deemed crucial in the production of photographs with a
high level of accuracy and detail. The low-resolution image is subjected to
corruption by augmentation, based on a given conditioning level. Subsequently,
the diffusion model is conditioned on the augmented image. During the training
phase, the amount of augmentation is selected in a random manner. However,
during the inference phase, we systematically explore various values of
augmentation in order to identify the optimal sample quality. In this study,
Gaussian noise is employed as a means of augmentation, and we implement
variance-preserving Gaussian noise augmentation that mimics the forward
process utilized in diffusion models.

###  3.4 Neural network architecture

The U-Net architecture [15] is modified for the base 128x128 text-to-image
diffusion model. The network incorporates text embeddings by utilizing a
pooled embedding vector, which is combined with the diffusion timestep
embedding in a manner similar to the class embedding conditioning technique
[5, 10]. In order to enhance our model and also keep it light, we used only 4
upsampling and downsampling stages, we incorporated cross-attention across the
text embeddings at the last 3 stages by conditioning on the complete sequence
of text embeddings. In addition, we also used Layer Normalisation in the
attention and pooling layers of text embeddings which significantly enhanced
performance. We used only 3 residual blocks for upsampling and downsampling
the activations [31].
The Efficient U-Net model [24], is employed for achieving super-resolution
from 128x128 to 256x256. We harvested its improvements to enhance memory
efficiency, reduce inference time, and improve convergence speed, but we used
the self-attention layer only in the last stage of the upsampling and
downsampling We also kept the last text cross-attention layers which we found
to be critical.
We decided to generate a 128x128 image and then super-resolve it to 256x256
instead of generating a 256x256 image in one step, several factors influenced
this approach:

  * •

Computational efficiency: Diffusion models can be computationally intensive,
especially when applied to high-resolution images. By initially generating a
lower-resolution image, we can alleviate the computational burden, resulting
in faster processing and training times. This approach enables more practical
implementation and facilitates experimentation with larger datasets.

  * •

Enhanced information flow: Diffusion models often operate in an autoregressive
manner, processing a sequence of lower-resolution images or noise samples.
Starting with a lower resolution allows for better information flow and
conditioning at each step of the diffusion process. This sequential generation
aids in accurately modeling the data distribution and capturing complex
dependencies within the image.

  * •

Capturing global dependencies: Generating a lower-resolution image initially
enables diffusion models to capture global dependencies and structures more
effectively. This step allows the model to learn the overarching patterns and
relationships present in the image. By utilizing this understanding, the
subsequent super-resolution process can ensure that the resulting high-
resolution image aligns with the overall structure of the target image.

  * •

Prioritization of details: The initial lower-resolution generation in
diffusion models allows for the prioritization of important image details and
textures during the subsequent super-resolution step. By focusing on
recovering these critical aspects, the model can produce enhanced high-
resolution images that preserve realistic and fine-grained details. This is
particularly advantageous in real-world image super-resolution tasks, where
capturing and preserving such details are essential for generating visually
pleasing and faithful results.

##  4 Experiments

###  4.1 Dataset

The Remote Sensing Image Captioning Dataset (RSICD) [13] was utilized to
assess the proposed methodology in this investigation. The RSICD dataset was
initially gathered specifically for the purpose of facilitating the remote
sensing picture captioning assignment. The dataset comprises a comprehensive
collection of 10,921 remote-sensing photos with excellent resolution. Every
image in the dataset possesses dimensions of 224x224 pixels and is accompanied
by five textual descriptions as annotations. The dataset consists of 30
distinct scene classes, namely airport, bare land, baseball field, beach,
bridge, center, church, commercial, dense residential, desert, farmland,
forest, industrial, meadow, medium residential, mountain, parking, park,
playground, pond, port, railway station, resort, river, school, sparse
residential, square, stadium, storage tanks, and viaduct. For the experiment,
the training set consists of 8734 text-image pairings from the training split,
while the test set comprises the remaining 2187 text-image pairs [36]. In the
experiment, all photos were resized to a resolution of 256x256.

###  4.2 Evaluation metrics

The evaluation metrics employed in this study are the Inception Score (IS)
[26] and the FID [8]. These metrics are utilized to assess the degree of
stylistic resemblance between the generated images and the original samples
[16, 40] in the RSICD. In this context, the function $g(z)$ represents the
image that is formed and needs to be assessed. Additionally, $q(l|x)$ refers
to the posterior probability of a specific label, denoted as $l$, which is
computed by the Inception-V3 model for a given image $x$. Then, IS can be
calculated as:

$IS=exp[\mathbb{E}_{z\sim q(z)}[D_{kl}(q(l|g(z))||q(l)]]$ (3)

The marginal class distribution is $q(l)$ and $D_{kl}(.)$ refers to the KL-
divergence, which is a measure of the difference between two probability
distributions.
The FID score is computed using the features derived from the final average
pooling layer in the Inception-V3 model. Then, IS can be calculated as:

$FID=||\mu_{r}-\mu_{g}||^{2}+Tr((\Sigma_{r}+\Sigma_{g}-2(\Sigma_{r}\Sigma_{g})^{\frac{1}{2}})$
(4)

Where $(\mu_{r},\Sigma_{r})$ and $(\mu_{g},\Sigma_{g})$ represent the mean and
covariance of the real and generated features, respectively. The function
$Tr(.)$ denotes the trace operation in linear algebra.

###  4.3 Training

A significantly lighter diffusion model is employed for image synthesis
consisting of 260 million parameters compared to the one utilized in Imagen
[24] which consists of 2 billion parameters. Additionally, a 260 million
parameter model is employed for super-resolution tasks which is also lighter
than the one used in Imagen with 600 million parameters. Both models were
trained using a batch size of 64 and 1000 training epochs. The base 128x128
model in our study utilizes the Tesla V100 SXM2 processor, which is equipped
with 32GB of memory. Additionally, the super-resolution variant employs
another Tesla V100 chip. The Adafactor optimizer [28] is employed for our
standard 128x128 model due to its significantly reduced memory requirements
compared to Adam while attaining comparable performance. The Adam optimizer
[12] is preferred over Adafactor for super-resolution models due to its
superior performance. We used a learning rate of $1e-4$ with 10,000 linear
warmup steps for both optimizers. To provide classifier-free guidance, we
engage in joint training by randomly setting the text embeddings to zero with
a probability of 10% for all three models.

Test prompt |  Attn-GAN |  DAE-GAN |  DF-GAN |  Lafite |  DALL-E |  Txt2Img-MHN (VQVAE) |  Txt2Img-MHN (VQGAN) |  RSDiff (Ours) |  Real Image
---|---|---|---|---|---|---|---|---|---
An airport with many buildings beside in it |  |  |  |  |  |  |  |  |
There is no plant on the bare land |  |  |  |  |  |  |  |  |
A baseball field is near some green trees |  |  |  |  |  |  |  |  |
In front of the sea is a vast beach |  |  |  |  |  |  |  |  |
There is a long bridge over the river |  |  |  |  |  |  |  |  |
Narrow roads were built around the farm |  |  |  |  |  |  |  |  |
Next to the playground is a house with a grey roof |  |  |  |  |  |  |  |  |
A large number of white ships were parked around the harbor |  |  |  |  |  |  |  |  |
Several buildings are around a stadium |  |  |  |  |  |  |  |  |
Many buildings are in two sides of a railway station |  |  |  |  |  |  |  |  |
Many green trees are around a building with a swimming pool |  |  |  |  |  |  |  |  |
On the ground there are several residential buildings |  |  |  |  |  |  |  |  |
Many buildings and green trees are in a dense residential area |  |  |  |  |  |  |  |  |

Table 1: Demonstration of remote sensing images produced utilizing diverse text-to-image generation techniques, all derived from the textual descriptions within the test dataset. Method | Inception Score $\uparrow$ | FID Score $\downarrow$
---|---|---
Attn-GAN | 11.71 | 95.81
DAE-GAN | 7.71 | 93.15
DF-GAN | 9.51 | 109.41
Lafite | 10.70 | 74.11
DALL-E | 2.59 | 191.93
RSDiff (Ours) | 7.22 | 66.49
Txt2Img-MHN (VQVAE) | 3.51 | 175.36
Txt2Img-MHN (VQGAN) | 5.99 | 102.44
Table 2: The quantitative results obtained on the RSICD test set. The
inception score and the FID score are reported in our study.

##  5 Results

We assess the qualitative performance of the suggested method in comparison to
7 current SoTA approaches. A concise overview of various strategies is
provided below.

  * •

Attn-GAN: Fine-grained text-to-image generation based on GAN with attention
learning and multi-stage refinement [34].

  * •

DAE-GAN: Aspect-level text-to-image synthesis with attended global refinement
and aspect-aware local refinement [22].

  * •

DF-GAN: GAN-based text-to-image generation using the deep text-image fusion
block and a target-aware discriminator [32].

  * •

Lafite: Language-free text-to-image generation with the pre-trained CLIP model
[40].

  * •

DALL-E: Zero-shot text-to-image generation using the modified GPT-3 model
[19].

  * •

Txt2Img-MHN (VQVAE): Text-to-image modern Hopfield network with image encoder
and decoder from VQVAE [36].

  * •

Txt2Img-MHN (VQGAN): Text-to-image modern Hopfield network with image encoder
and decoder from VQGAN [36].

  * •

RSDiff (Ours): The proposed text-to-image cascaded diffusion models pipeline.

The generation of remote sensing images that are both photo-realistic and
semantic-consistent is a highly demanding task, mostly because of the
intricate spatial distribution of many ground objects. Although DF-GAN and
other advanced GAN-based techniques have demonstrated impressive capabilities
in replicating the visual style of remote sensing images, they may encounter
challenges when it comes to accurately reproduce intricate shape and boundary
details of complex ground targets, such as the playground (located in the 7
row of Table 2) and the stadium (located in the 9 row of Table 2). In
contrast, the suggested RSDiff algorithm has the capability to produce more
realistic outcomes in these complex settings.
Furthermore, it has been observed that RSDiff exhibits superior ability in
acquiring the notion of amount. Consider the last three rows depicted in Table
2 as an illustrative example. The three input text descriptions provided by
the user pertain to the subject of building objects. However, the number of
buildings mentioned in these descriptions varies, ranging from a singular
building to many buildings, and finally to a large number of buildings. While
the majority of the solutions yield satisfactory outcomes in the initial
scenario, comprehending the precise meaning of terms such as ”several” and
”many” poses a greater challenge. This tendency is most evident in the outcome
of Lafite when it erroneously produces a substantial number of densely
populated residential structures when presented with textual descriptions of
”several buildings.” In contrast, the outcomes of RSDiff exhibit a higher
degree of semantic consistency with the textual descriptions provided as
input.
Assessing the quality of synthesized images typically relies on human
observers’ visual perception, which is often intuitive. However, accurately
quantifying this judgment poses challenges due to the inherent ambiguity and
complexity of the semantic information conveyed within the image. Presently,
the prevailing metrics employed for image generation encompass the Inception
Score and the FID Score [14, 20] Although these two measures are
straightforward in nature, they were initially employed to assess the efficacy
of the GAN model, with an emphasis on either the resemblance of visual styles
or the disparity in features between synthesized and authentic data.
Table 1 presents a comprehensive overview of the Inception Score and FID Score
associated with each approach employed in the present investigation. One
intriguing observation is that GAN-based techniques tend to exhibit
significantly superior performance on metrics such as the Inception Score and
FID Score compared to Transformer-based methods, a finding that aligns with
earlier research [20]. The Attn-GAN, outperforms the proposed RSDiff model,
producing an Inception Score of 11.71 compared to RSDiff’s 7.22.
Simultaneously, it’s important to recognize that RSDiff, despite its slightly
lower score, holds a notable position as a leading diffusion model. Moreover,
pivoting to a different evaluation metric, the suggested methodology shines
with remarkable performance in FID score, attaining a state-of-the-art outcome
of 66.49.

##  6 Conclusion

This paper introduces a novel text-to-image diffusion model that aims to
generate remote-sensing images that are both photo-realistic and semantically
compatible with the given text descriptions. Furthermore, we conduct a
thorough comparison and analysis of the performance of our model in relation
to previous methodologies.

The results obtained from a series of comprehensive tests conducted on the
RSICD benchmark remote sensing text-image dataset provide evidence that the
proposed method has the capability to generate remote sensing images that
exhibit a higher degree of realism compared to existing methodologies.

A primary obstacle encountered in the process of generating remote-sensing
images from textual descriptions is the scarcity of text-image pairs available
in the training dataset. Acquiring precise textual descriptions for remote
sensing data is typically challenging, although a substantial volume of
unlabeled high-resolution remote sensing photos is readily available.
Therefore, further investigation is warranted to explore the potential of
leveraging the copious semantic data present in unlabeled photos for enhancing
the efficacy of text-to-image generation. In our forthcoming research
endeavors, we shall endeavor to delve into the intricacies of this subject
matter.

## References

  * [1] Mesay Belete Bejiga, Farid Melgani, and Antonio Vascotto.  Retro-remote sensing: Generating images from ancient texts.  IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 12(3):950–960, 2019.
  * [2] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al.  Language models are few-shot learners.  Advances in neural information processing systems, 33:1877–1901, 2020.
  * [3] Chen Chen, Hongxiang Ma, Guorun Yao, Ning Lv, Hua Yang, Cong Li, and Shaohua Wan.  Remote sensing image augmentation based on text description for waterside change detection.  Remote Sensing, 13(10):1894, 2021.
  * [4] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.  Bert: Pre-training of deep bidirectional transformers for language understanding.  arXiv preprint arXiv:1810.04805, 2018.
  * [5] Prafulla Dhariwal and Alexander Nichol.  Diffusion models beat gans on image synthesis.  Advances in neural information processing systems, 34:8780–8794, 2021.
  * [6] Pedram Ghamisi, Javier Plaza, Yushi Chen, Jun Li, and Antonio J Plaza.  Advanced spectral classifiers for hyperspectral images: A review.  IEEE Geoscience and Remote Sensing Magazine, 5(1):8–32, 2017.
  * [7] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio.  Generative adversarial nets.  Advances in neural information processing systems, 27, 2014.
  * [8] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter.  Gans trained by a two time-scale update rule converge to a local nash equilibrium.  Advances in neural information processing systems, 30, 2017.
  * [9] Jonathan Ho, Ajay Jain, and Pieter Abbeel.  Denoising diffusion probabilistic models.  Advances in neural information processing systems, 33:6840–6851, 2020.
  * [10] Jonathan Ho, Chitwan Saharia, William Chan, David J Fleet, Mohammad Norouzi, and Tim Salimans.  Cascaded diffusion models for high fidelity image generation.  The Journal of Machine Learning Research, 23(1):2249–2281, 2022\.
  * [11] Jonathan Ho and Tim Salimans.  Classifier-free diffusion guidance.  arXiv preprint arXiv:2207.12598, 2022.
  * [12] Diederik P Kingma and Jimmy Ba.  Adam: A method for stochastic optimization.  arXiv preprint arXiv:1412.6980, 2014.
  * [13] Xiaoqiang Lu, Binqiang Wang, Xiangtao Zheng, and Xuelong Li.  Exploring models and data for remote sensing image caption generation.  IEEE Transactions on Geoscience and Remote Sensing, 56(4):2183–2195, 2017.
  * [14] Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, and Mark Chen.  Glide: Towards photorealistic image generation and editing with text-guided diffusion models.  arXiv preprint arXiv:2112.10741, 2021.
  * [15] Alexander Quinn Nichol and Prafulla Dhariwal.  Improved denoising diffusion probabilistic models.  In International Conference on Machine Learning, pages 8162–8171. PMLR, 2021.
  * [16] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al.  Learning transferable visual models from natural language supervision.  In International conference on machine learning, pages 8748–8763. PMLR, 2021.
  * [17] Colin Raffel, Minh-Thang Luong, Peter J Liu, Ron J Weiss, and Douglas Eck.  Online and linear-time attention by enforcing monotonic alignments.  In International conference on machine learning, pages 2837–2846. PMLR, 2017.
  * [18] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu.  Exploring the limits of transfer learning with a unified text-to-text transformer.  The Journal of Machine Learning Research, 21(1):5485–5551, 2020\.
  * [19] Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen.  Hierarchical text-conditional image generation with clip latents.  arXiv preprint arXiv:2204.06125, 1(2):3, 2022.
  * [20] Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever.  Zero-shot text-to-image generation.  In International Conference on Machine Learning, pages 8821–8831. PMLR, 2021.
  * [21] Scott E Reed, Zeynep Akata, Santosh Mohan, Samuel Tenka, Bernt Schiele, and Honglak Lee.  Learning what and where to draw.  Advances in neural information processing systems, 29, 2016.
  * [22] Shulan Ruan, Yong Zhang, Kun Zhang, Yanbo Fan, Fan Tang, Qi Liu, and Enhong Chen.  Dae-gan: Dynamic aspect-aware gan for text-to-image synthesis supplementary document.
  * [23] Chitwan Saharia, William Chan, Huiwen Chang, Chris Lee, Jonathan Ho, Tim Salimans, David Fleet, and Mohammad Norouzi.  Palette: Image-to-image diffusion models.  In ACM SIGGRAPH 2022 Conference Proceedings, pages 1–10, 2022.
  * [24] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al.  Photorealistic text-to-image diffusion models with deep language understanding.  Advances in Neural Information Processing Systems, 35:36479–36494, 2022.
  * [25] Chitwan Saharia, Jonathan Ho, William Chan, Tim Salimans, David J Fleet, and Mohammad Norouzi.  Image super-resolution via iterative refinement.  IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(4):4713–4726, 2022.
  * [26] Tim Salimans, Ian Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen.  Improved techniques for training gans.  Advances in neural information processing systems, 29, 2016.
  * [27] Pierre Sermanet, Soumith Chintala, and Yann LeCun.  Convolutional neural networks applied to house numbers digit classification.  In Proceedings of the 21st international conference on pattern recognition (ICPR2012), pages 3288–3291. IEEE, 2012.
  * [28] Noam Shazeer and Mitchell Stern.  Adafactor: Adaptive learning rates with sublinear memory cost.  In International Conference on Machine Learning, pages 4596–4604. PMLR, 2018.
  * [29] Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli.  Deep unsupervised learning using nonequilibrium thermodynamics.  In International conference on machine learning, pages 2256–2265. PMLR, 2015.
  * [30] Yang Song and Stefano Ermon.  Generative modeling by estimating gradients of the data distribution.  Advances in neural information processing systems, 32, 2019.
  * [31] Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole.  Score-based generative modeling through stochastic differential equations.  arXiv preprint arXiv:2011.13456, 2020.
  * [32] Ming Tao, Hao Tang, Fei Wu, Xiao-Yuan Jing, Bing-Kun Bao, and Changsheng Xu.  Df-gan: A simple and effective baseline for text-to-image synthesis.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16515–16525, 2022.
  * [33] Jay Whang, Mauricio Delbracio, Hossein Talebi, Chitwan Saharia, Alexandros G Dimakis, and Peyman Milanfar.  Deblurring via stochastic refinement.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16293–16303, 2022.
  * [34] Tao Xu, Pengchuan Zhang, Qiuyuan Huang, Han Zhang, Zhe Gan, Xiaolei Huang, and Xiaodong He.  Attngan: Fine-grained text to image generation with attentional generative adversarial networks.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1316–1324, 2018.
  * [35] Yonghao Xu and Pedram Ghamisi.  Universal adversarial examples in remote sensing: Methodology and benchmark.  IEEE Transactions on Geoscience and Remote Sensing, 60:1–15, 2022\.
  * [36] Yonghao Xu, Weikang Yu, Pedram Ghamisi, Michael Kopp, and Sepp Hochreiter.  Txt2img-mhn: Remote sensing image generation from text using modern hopfield networks.  arXiv preprint arXiv:2208.04441, 2022.
  * [37] Han Zhang, Tao Xu, Hongsheng Li, Shaoting Zhang, Xiaogang Wang, Xiaolei Huang, and Dimitris N Metaxas.  Stackgan: Text to photo-realistic image synthesis with stacked generative adversarial networks.  In Proceedings of the IEEE international conference on computer vision, pages 5907–5915, 2017.
  * [38] Lefei Zhang and Liangpei Zhang.  Artificial intelligence for remote sensing data analysis: A review of challenges and opportunities.  IEEE Geoscience and Remote Sensing Magazine, 10(2):270–294, 2022\.
  * [39] Rui Zhao and Zhenwei Shi.  Text-to-remote-sensing-image generation with structured generative adversarial networks.  IEEE Geoscience and Remote Sensing Letters, 19:1–5, 2021.
  * [40] Y Zhou, R Zhang, C Chen, C Li, C Tensmeyer, T Yu, J Gu, J Xu, and T Sun.  Lafite: Towards language-free training for text-to-image generation. arxiv 2021.  arXiv preprint arXiv:2111.13792.