# Multilingual Abusiveness Identification on Code-Mixed Social Media Text

Ekagra Ranjan  Microsoft
<EMAIL_ADDRESS>Naman Poddar  Indian Institute Of Technology (IIT)
Guwahati
<EMAIL_ADDRESS>
###### Abstract

Social Media platforms have been seeing adoption and growth in their usage
over time. This growth has been further accelerated with the lockdown in the
past year when people’s interaction, conversation, and expression were limited
physically. It is becoming increasingly important to keep the platform safe
from abusive content for better user experience. Much work has been done on
English social media content but text analysis on non-English social media is
relatively underexplored. Non-English social media content have the additional
challenges of code-mixing, transliteration and using different scripture in
same sentence. In this work, we propose an approach for abusiveness
identification on the multilingual Moj dataset which comprises of Indic
languages. Our approach tackles the common challenges of non-English social
media content and can be extended to other languages as well.

##  I Introduction

Social media platorms have been seeing growing usage and adoption over time.
This growth has been further accelerated with the lockdowns in the pandemic
where people had no choice but to interact and express themselves online. This
has led to increase in new users as well as daily active users. India has seen
tremendous adoption in social media usage with the Jio revolution when
Reliance Jio made internet charges affordable. Reliance Jio allowed
democratizing internet which has benefitted social media platforms operating
in India. With the rise of intenet access to Indian people, social media
platforms are seeing rise in Indic native content. It is in interest of social
media platforms to adapt their systems to understand the regional content to
better serve the users and attract more user base. One such company which has
been focusing on serving the regional content since its foundation is
Sharechat. Sharechat is an Indian company which is one of the leading social
networking space in catering to regional content. Their short video app is
called Moj which went viral in the lockdown and has been since seeing
tremendous growth in its user base and content.

Increasing social media usage has made it important to keep the platform safe
from abusive content to enable individual safety in the online world. There is
more abusive content in the online world when compared to the offline world
because Internet enables anonymity. This can be used to share abusive content
without being held responsible. Text analysis of English social media content
is a well explored domain [1], [2]. Multilingual NLP is a relative new
subfield and has been picking up pace due to globalization [3]. Most of the
multilingual transfomer models were trained on Wikipedia dataset of Common
Crawl dataset [4], [5]. This is different from training on a social media text
dataset as social media text content have vast amount of acceptable
misspellings. On top of this, non-English social media content have their own
unique set of challenges comprising of code-mixing, transliteration, mixing of
scriptures in the same sentences. Code-Mixing is the usage of different
languages in the same sentence or utterance. This mostly used in informal
conversation and social media. This phenonmenon exits in multilingual
societies where the speaker is fluent in more than one language [6].
Transliteration is a way of writing a native language in the scripture of a
different language. The pronunciation of the transliterated langugae, by
following the rules of the scripture used, is similar to the one written using
the native scripture. Most common form of transliteration is to use the Latin
scripture to express a non-English sentence. Transliteration aggravates the
misspellings and variants of the same word. This is because not all syllables
and sounds in the native language would be available in the Latin language so
the writer has to use non-standard ways to approximate the syllable.

Prior works have performed text analysis on non-English languages and have
attempted to tackle a subset of the above challenges associated with non-
English social media text [7]. [8], [9], and [10] have extended text analysis
to Arabic, Czech, and Telugu languages. In our work, we identify the common
challenges associated with non-English social media text and propose an
approach to tackle all of these challenges in multilingual setting. Our
contributions include a spell-correction algorithm that does not rely on a
manually defined dictionary but instead infers its own from the training data.
This allows our spell-correction algorithm to learn new words by retraining on
the any new corpus. This is in contrast to the traditional dictionary based
approach where the words need to be added manually. Our approach is scalable
in the age of neologism which is prevalant in context of social media [11].

The paper is organized as follows. Sec. II describes the Moj Dataset used in
this work. Sec III defines the problem statement and challenges associated
with it. Sec. IV explains our approach in detail. Sec. V provides the
experimental setup. Sec. VI lists the result of our approach and in VII we
perform ablation studies on our approach. Sec. VIII discusses the future work
and we conclude our work in Sec. IX

##  II Dataset

The dataset provided by Moj consists of multilingual comments in $10+$ low-
resource Indic languages. The comments in the dataset have been human
annotated with a label indicating whether the comment is abusive or not. The
dataset also has meta-data like the language of the comment and the number of
likes and reports on each comment and the corresponding post on which the
comment was made. The label distribution across different language is show in
Table I

TABLE I: Label distribution across languages in Moj Dataset Language | Non-Abusive | Abusive
---|---|---
Assamese | 1496 | 1284
Bengali | 11428 | 11407
Bhojpuri | 2917 | 2887
Gujarati | 4426 | 4402
Haryanvi | 4395 | 4417
Hindi | 153433 | 153747
Kannada | 6954 | 6989
Malayalam | 31749 | 9216
Marathi | 44677 | 27367
Odia | 5475 | 5499
Rajasthani | 2183 | 2185
Tamil | 34792 | 34705
Telugu | 48461 | 48551

##  III Problem Statement

The main objective of the paper is to develop a multi-lingual approach to
identify abusive comments for social media platforms. Upon exploring the Moj
dataset, we identify the following challenges associated with multi-lingual
non-English social media content:

  * •

Code-Mixing: Social media text is not written in just one language in multi-
lingual socities.

  * •

Transliteration: People often use Latin scripture to write sentences in their
native language. Pronunciation of both original native sentence and
transliterated sentence is almost similar.

  * •

Mixing of Scriptures: Some people write sentences which contain words from
different scriptures.

  * •

Misspelling and variations in social media text.

Fig. 1 shows different representation of the same Hindi sentence. Our approach
attempts to tackle such challenges in multi-lingual setting.

Figure 1:  Example of challenges described in Sec. III.

##  IV Proposed Method

###  IV-A Data Cleaning

The dataset includes multiple comments from the same post as well. We replace
the post level meta-data of each comment, i.e., post like count and post
report count, with the maximum value of respective meta-data feature among all
the comments corresponding to the same post.

We pre-process the comments in the Moj dataset with the following steps:

  * •

Unicode normalization using `unicodedata` python package

  * •

Remove special characters, e.g, $@$, $"$, $\$$, $\\#$

  * •

Remove Emoji using `demoji`111https://pypi.org/project/demoji/ python package

  * •

Replace characters which occur more than twice consecutively with the same
character, e.g., hellooo →hello

  * •

Transliterate the comments using `indic-trans` python package [12] to the
native laguage scripture to tackle the challenge of transliteration mentioned
in Sec. III. The effect of this pre-processing step is discussed in Sec. VII.

Figure 2:  Overview of the training phase of Spell Correction using Graph
Clustering: (a) Graph is constructed using words in the training corpus. Two
words have an edge in the graph if they satisfy the nearness criteria
specified in Sec. IV-B1. (b) Maximal clique is found in this graph. The
circled subgraph in the figure is the maximal clique in this graph. This
subgraph is considered to contain words that are misspellings of one another
and are representing the same concept. (c) This subgraph/cluster is removed
from the graph and is processed to obtain its Parent Word and Anchor Words as
explained in Sec. IV-B1. The whole process is again repeated using the
remaining smaller graph untill all the nodes have been processed. Please refer
to Sec. IV-B for more details.

###  IV-B Spell Correction using Graph Clustering

Our approach builds a graph of words using the training corpus of Moj dataset.
Sec. IV-B1 shows how graph clustering is used to identify the set of related
words and the correct spelling of those words. Sec. IV-B2 shows the strategy
used to identify the correct spelling. This approach works even for the wrong
words that were not seen in the training as long as the correct word was seen
in training. Sec. VII shows the utility of our approach.

####  IV-B1 Training: Correct Spelling Identification

We intend to spell-correct the abusive words and ignore words which occur
predominantly in non-abusive comments. This is done to prevent non-abusive
words from being auto-corrected to abusive words or vice-versa. Since this is
a classification problem, presence of abusive word is more important than
presence of non-abusive word. This is because any combination of abusive and
non-abusive word will be abusive in nature.

Consider words that have occured minimum $5$ times in the abusive comments in
the training corpus. Ignore words that have length less than $6$ and have
number of consonants less than $4$. Construct a graph where nodes are these
words. Connect words with an edge if they have the same first $k$ letters and
the Levenshtein Distance between them is above a threshold score $t$. In this
work, $k$ was $3$ and score $t$ was $85$ based on hyper-parameter tuning on a
validation set. Levenshtein Distance was computed using
`rapidfuzz`222https://pypi.org/project/rapidfuzz/. Algorithm 1 is a psuedo-
code of the graph construction.

Find the maximal clique from the above constructed graph of words. A maximal
clique is the largest subgraph where nodes are connected to one another by an
edge. This maximal clique represents a subgraph of words which our algorithm
thinks are variants or misspellings of one another and represent the same
thing, e.g, `bewakoof`, `bewakoofi`, `bewakouf` where `bewakoof` means idiot
in Hindi. The word in this cluster with highest occurring frequency in the
training corpus is chosen as the correct spelling for this cluster. We refer
to this word as the Parent Word of the cluster. Next, we find the Anchor Words
for this cluster. Anchor Words are those words which will represent this
cluster during Testing phase explained in Sec. IV-B2. Top five frequent words
of this cluster are chosen as the candidates for the Anchor Words. Among them,
those words that have frequency above one-fourth of the the Parent Word is
included in Anchor Words.

The above maximal clique is removed from the graph once that cluster is
processed and its Parent Words and Anchor Words have been computed. This
process of finding a cluster using maximal clique, processing the cluster, and
removing it is repeated on the remaining smaller graph obtained from the
previous rounds of clustering until all the nodes in the original graph have
been processed. Algorithm 2 shows the psuedo-code for training phase.

####  IV-B2 Testing: Auto-correction during inference

By the end of training phase, we have a set of clusters with Parent Word and
its Anchor Words that will be used during inference to spell-correct the words
in train and test corpus. Given a word $w$ during inference, we find out the
clusters whose Parent Word has the same first $k$ letters as that of word $w$.
The association $score_{wi}$ of a cluster $C_{i}$ obtained from the previous
filtering step is given as:

$score_{wi}=\max_{a\in A(C_{i})}\\{LevenshteinDistance(w,a)\\}$ (1)

where $A(C_{i})$ is the set of Anchor Words of cluster $C_{i}$. The cluster
$C_{j}$ that has the highest association $score_{wj}$ with the word $w$ is
chosen as the appropriate cluster $C^{*}$ if the score $score_{wj}$ is above a
threshold $t$. The word $w$ is replaced with the Parent Word of the cluster
$C^{*}$ if the word $w$ was able to find its appropriate cluster $C^{*}$,
e.g., `bewakoufi` will be corrected to `bewakoof`. We spell-correct all the
comments in both training and testing data before using them for
classification. Algorithm 3 shows the psuedo-code for the testing phase.

###  IV-C Multilingual Models

Given that our task involves low-resource Indic languages, we use multiple
pre-trained multilingual transformer models and finetune them on the comments
of Moj dataset. The following models were used in our work:

  * •

mBERT [4]: is a multilingual variant of BERT [13] trained on multilingual
Wikipedia data from $104$ languages.

  * •

XLM-R [5]: is trained on cross-lingual masked language modeling task on $100$
languages.

  * •

XLM-T [3]: is XLM-R trained on Twitter comments

  * •

Multilingual DistilBERT [14]: is a distilled version of mBERT

  * •

MURIL [15]: is a variant of BERT trained on monolingual, translated and
transliterated data from $17$ Indic languages

  * •

Indic-BERT [16]: is a variant of ALBERT [17] trained on IndicCorp dataset
[16].

  * •

Canine [18]: is a tokenizer and vocabulary free transformer model which
operates directly on sequences of characters. Canine-S and Canine-C are two
variants trained on different loss functions defined in [18].

We also trained a logistic regression model on an embedding created by using
Naive-Bayes over TF-IDF feautures [19]333https://www.kaggle.com/jhoward/nb-
svm-strong-linear-baseline.

###  IV-D Meta-Data

The meta-data associated with the comments, i.e, likes and report count of the
comments as well as the post on which the comment was made, was leveraged to
further improve the predictions. We trained a Random Forest classifier over
this meta-data only to provide the predictions. These meta-data specific
predictions are later combined with the predictions of the comment-specific
models as discussed later in Sec. VI.

Input : Words $W$ present in abusive sentences; Frequency count $F$ containing
number of times a word has occurred in the corpus; Length of words $L$; Number
of consonants in words $C$; Function $T(w)$ gives the first three letters of
input word $w$; Levenshtein Distance function $score(a,b)$

Output : Word Graph $G(V,E)$ where $V$ is the nodes and $E$ is the edges in
the graph $G$

1 $V\leftarrow\phi$

2 for  _$i=1...|W|$_ do

3 if  _$F_{i}\geq 5$_ then

4 if  _$L_{i}\geq 6$_ then

5 if  _$C_{i}\geq 4$_ then

6 if  _$V=\phi$_ then

7 $V_{1}\leftarrow W_{i}$

8 end if

9 else

10 for  _$j=1...|V|$_ do

11 if  _$T(W_{i})=T(V_{j})$_ then

12 if  _$score(W_{i},V_{j})\geq 85$_ then

13 $k\leftarrow|V|+1$

14 $V_{k}\leftarrow W_{i}$

15

16 $E_{j}\leftarrow E_{j}+V_{k}$

17 $E_{k}\leftarrow E_{k}+V_{j}$

18

19 end if

20

21 end if

22

23 end for

24

25 end if

26

27 end if

28

29 end if

30

31 end if

32

33 end for

Algorithm 1 Graph Construction

1

Input : Graph $G(V,E)$; Frequency count $F$ containing number of times a word
has occurred in the corpus

Intermediate : $C$ is the maximal cluster obtained in a particular step; $p$
is the parent element for a particular cluster; $a$ is the anchor elements for
a particular cluster

Output : Parent $P$ containing list of parent member for the clusters that
will be obtained; Anchor $A$ containing list of anchor members for the
clusters that will be obtained

2 $G^{\prime}\leftarrow G$

3 $A\leftarrow\phi$

4 $P\leftarrow\phi$

5

6while  _$G^{\prime}\neq\phi$_ do

7 $C(V^{c},E^{c})=MaximalClique(G^{\prime})$

8 $j=argmax(\\{F_{i}|i\in\\{1...|V^{c}|\\}\\})$

9 $p\leftarrow V^{c}_{j}$

10 $p_{freq}=F_{j}$

11 $a\leftarrow\phi$

12 for  _$k\in\\{1...|V^{c}|\\}\\}$_ do

13 if  _$F_{k} >\frac{p_{freq}}{4}$_ then

14 $a\leftarrow a+V^{c}_{k}$

15

16 end if

17 $G^{\prime}\leftarrow G^{\prime}-C$

18 $P\leftarrow P+p$

19 $A\leftarrow A+a$

20

21 end for

22

23 end while

24

Algorithm 2 Training: Correct Spelling Identification algorithm

1

Input : $w_{T}$ is the test word that needs to be corrected; Parent $P$
containing list of parent member for the clusters obtained during training;
Anchor $A$ containing list of anchor members for the clusters obtained during
training; $score(a,b)$ is the Levenshtein distance between words $a$ and $c$;
Function $T(w)$ gives the first three letters of input word $w$

Output : $w^{*}$ is the corrected word for $w_{T}$

2 $w^{*}=w_{T}$

3 $score^{*}=0$

4 for  _$i\in\\{1...|P|\\}$_ do

5 if  _$T(w_{T})=T(P_{i})$_ then

6 for  _$j\in{1...|A_{i}|}$_ do

7 $s=score(w_{T},A_{i,j})$

8 if  _$s\geq 85$_ then

9 if  _$s\geq score^{*}$_ then

10 $s=score^{*}$

11 $w^{*}=P_{i}$

12

13 end if

14

15 end if

16

17 end for

18

19 end if

20

21 end for

22

Algorithm 3 Testing: Auto-Correction duing Inference algorithm

##  V Experimental Setup

All transformer models except for Canine are finetuned using Adam optimizer
[20] with learning rate of $1e$-$5$. Canine models were finetuned using AdamW
optimizer [21] using $2e$-$5$ learning rate. All the models were trained for
$3$ epochs with binary-cross entropy as the loss function. Threshold of $0.5$
was used to get the output class from output probabilites. Huggingface library
[22] was used for training all the transformer models.

##  VI Result

TABLE II: Comparison of Models on Moj Dataset Model | Val. F1 | Test F1
---|---|---
MURIL | 0.8675 | -
Indic-BERT | 0.8475 | -
XLM-R | 0.8623 | -
XLM-T | 0.8633 | -
Multilingual DistilBERT | 0.8553 | -
mBERT | 0.8551 | -
Canine-C | 0.8646 | -
Canine-S | 0.8622 | -
Logistic Regression + NB + TF-IDF | 0.8559 | -
Random Forest (meta-data) | 0.7109 | -
Ensemble | - | 0.8933

We create a stratified validation set from $10\%$ of the training set. Table
II shows the comparison of all the models discussed in Sec. IV-C. Logistic
Regression + Naive Bayed on TF-IDF provides a good baseline even though it is
a very simple model. MURIL scores the highest F1 score on the validation set.
XLM-T performed slightly better than XLM-R as XLM-T is trained on multilingual
Twitter comment which is closer to the Moj Dataset than the Wikipedia dataset
on which XLM-R was trained on. The meta-data contains signals essential for
this task as the Random Forest obtains an F1 score of $0.71$ using meta-data
alone. Our final model is an ensemble of all the models from all the $3$
epochs in Sec. IV-C. A logistic regression model was trained on the validation
set to combine the predictions from all models for ensembling.

##  VII Ablation Analysis

TABLE III: Effect of Data Cleaning, Native Transliteration, and Spell correction using Logistic Regression Model | Val. F1
---|---
Raw Dataset | 0.8493
Cleaned Dataset | 0.8508
Cleaned + Native Transliteration Dataset | 0.8547
Cleaned + Native Transliteration + Spell-Corrected Dataset | 0.8567
Cleaned + English Transliteration Dataset | 0.8496

###  VII-A Data Cleaning

Logistic regression was used to measure the effect of our pre-processing steps
as its simple to train, test and iterate. Table. III measures the validation
F1 score on raw data and the various pre-processing steps. It shows that using
the data cleaning as specified in Sec. IV-A improves the F1 score.

###  VII-B Transliteration

[23] shows that mBERT is unable to perform good on transliterated dataset as
it was not trained on such data. Since only MURIL was originally trained on
transliterated data, other transformer models would face domain mismatch when
finetuned on Moj Dataset. To tackle transliteration and mixture of scriptures
in same sentences as discussed on Sec. III, we experimented with
transliterating the comments before feeding it to our classifiers. Table. III
shows that transliterating all the comments to the native scripture shows the
largest improvement among all the steps. We also experimented with
transliterating everything to English so that there is knowledge sharing
between languages that have common words but have different scriptures. This
would increase the number of training data for lower resource language.
However, this led to a decrease in F1 score. Transliterating eveything to
native might be working better than transliterating to English because there
could be same sounding words that could mean different things in different
language, e.g, `kutta` means ”dog” in Hindi whereas it means ”new” in Telugu.
Transliterating to English also leads to loss of syllables which are not
present in English language but are present in the native language.

###  VII-C Spell Correction

Spelling correction allows the number of training samples per concept to
increase as different variants of words are mapped to the same underlying
concept, e.g, `bakwas` and `bakvas` where `bakwas` means ”useless” in Hindi.
Increased number of training samples per concept allows the model to learn
better representation for that particular concept. Table. III indicates that
our clustering approach in Sec. IV-B boosts the F1 score.

##  VIII Future Work

Table. I shows that there is class imbalance for Malayalam language.
Dravidian-CodeMix-FIRE [24] can be used to augment Moj Dataset to overcome
this. Adversarial training [25], [26], [27] can be used to increase robustness
of the system. Emojis could be replaced with words instead of completely
removing them as done in Sec. IV-A. Incorporating homophones in the graph
construction in Sec. IV-B might bring more gains.

##  IX Conclusion

In this paper, we propose an approach to identify abusive comments from a
multilingual dataset. We leverage pre-trained multilingual transformer models
along with classical machine learning models for abusive identification. We
identify the inherent challenges associated with this task such as code-
mixing, transliteration, mixing of scriptures, and misspellings. Our approach
attempts to tackle all of these challenges to build a robust system. We
propose a spell correction algorithm using graph clustering that does not need
manual creation of a dictionary for each language. It can also correct unseen
incorrect words during inference if the correct word was observed in training.
Results shows that our approach is well-suited for non-English social media
text analysis.

## References

  * [1] T. Mullen and N. Collier, “Sentiment analysis using support vector machines with diverse information sources,” in _Proceedings of the 2004 Conference on Empirical Methods in Natural Language Processing_. Barcelona, Spain: Association for Computational Linguistics, Jul. 2004, pp. 412–418. [Online]. Available: https://aclanthology.org/W04-3253
  * [2] A. Samuels and J. Mcgonical, “Sentiment analysis on social media content,” _arXiv preprint arXiv:2007.02144_ , 2020.
  * [3] F. Barbieri, L. E. Anke, and J. Camacho-Collados, “Xlm-t: A multilingual language model toolkit for twitter,” _arXiv preprint arXiv:2104.12250_ , 2021\.
  * [4] J. Devlin, “Multilingual bert readme,” 2018. [Online]. Available: https://github.com/google-research/bert/commits/master/multilingual.md
  * [5] A. Conneau, K. Khandelwal, N. Goyal, V. Chaudhary, G. Wenzek, F. Guzmán, E. Grave, M. Ott, L. Zettlemoyer, and V. Stoyanov, “Unsupervised cross-lingual representation learning at scale,” in _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_. Association for Computational Linguistics, Jul. 2020.
  * [6] S. Santy, A. Srinivasan, and M. Choudhury, “BERTologiCoMix: How does code-mixing interact with multilingual BERT?” in _Proceedings of the Second Workshop on Domain Adaptation for NLP_. Kyiv, Ukraine: Association for Computational Linguistics, Apr. 2021\.
  * [7] P. Patwa, G. Aguilar, S. Kar, S. Pandey, S. PYKL, B. Gambäck, T. Chakraborty, T. Solorio, and A. Das, “SemEval-2020 task 9: Overview of sentiment analysis of code-mixed tweets,” in _Proceedings of the Fourteenth Workshop on Semantic Evaluation_. Barcelona (online): International Committee for Computational Linguistics, Dec. 2020, pp. 774–790. [Online]. Available: https://aclanthology.org/2020.semeval-1.100
  * [8] H. S. Ibrahim, S. M. Abdou, and M. Gheith, “Sentiment analysis for modern standard arabic and colloquial,” _arXiv preprint arXiv:1505.03105_ , 2015\.
  * [9] I. Habernal, T. Ptáček, and J. Steinberger, “Sentiment analysis in Czech social media using supervised machine learning,” in _Proceedings of the 4th Workshop on Computational Approaches to Subjectivity, Sentiment and Social Media Analysis_. Atlanta, Georgia: Association for Computational Linguistics, Jun. 2013, pp. 65–74. [Online]. Available: https://aclanthology.org/W13-1609
  * [10] S. S. Mukku, N. Choudhary, and R. Mamidi, “Enhanced sentiment classification of telugu text using ml techniques,” in _SAAIP@IJCAI_ , 2016, pp. 29–34. [Online]. Available: http://ceur-ws.org/Vol-1619/paper5.pdf
  * [11] Q. Würschinger, M. F. Elahi, D. Zhekova, and H.-J. Schmid, “Using the web and social media as corpora for monitoring the spread of neologisms. the case of ‘rapefugee’, ‘rapeugee’, and ‘rapugee’.” in _Proceedings of the 10th Web as Corpus Workshop_. Berlin: Association for Computational Linguistics, Aug. 2016, pp. 35–43. [Online]. Available: https://aclanthology.org/W16-2605
  * [12] I. A. Bhat, V. Mujadia, A. Tammewar, R. A. Bhat, and M. Shrivastava, “Iiit-h system submission for fire2014 shared task on transliterated search,” in _Proceedings of the Forum for Information Retrieval Evaluation_ , ser. FIRE ’14. New York, NY, USA: ACM, 2015, pp. 48–53. [Online]. Available: http://doi.acm.org/10.1145/2824864.2824872
  * [13] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “BERT: Pre-training of deep bidirectional transformers for language understanding,” in _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_. Minneapolis, Minnesota: Association for Computational Linguistics, Jun. 2019.
  * [14] V. Sanh, L. Debut, J. Chaumond, and T. Wolf, “Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter,” _The 5th Workshop on Energy Efficient Machine Learning and Cognitive Computing_ , 2019.
  * [15] S. Khanuja, D. Bansal, S. Mehtani, S. Khosla, A. Dey, B. Gopalan, D. K. Margam, P. Aggarwal, R. T. Nagipogu, S. Dave _et al._ , “Muril: Multilingual representations for indian languages,” _arXiv preprint arXiv:2103.10730_ , 2021.
  * [16] D. Kakwani, A. Kunchukuttan, S. Golla, N. Gokul, A. Bhattacharyya, M. M. Khapra, and P. Kumar, “inlpsuite: Monolingual corpora, evaluation benchmarks and pre-trained multilingual language models for indian languages,” in _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: Findings_ , 2020, pp. 4948–4961.
  * [17] Z. Lan, M. Chen, S. Goodman, K. Gimpel, P. Sharma, and R. Soricut, “Albert: A lite bert for self-supervised learning of language representations,” in _International Conference on Learning Representations_ , 2020. [Online]. Available: https://openreview.net/forum?id=H1eA7AEtvS
  * [18] J. H. Clark, D. Garrette, I. Turc, and J. Wieting, “Canine: Pre-training an efficient tokenization-free encoder for language representation,” _arXiv preprint arXiv:2103.06874_ , 2021.
  * [19] S. Wang and C. Manning, “Baselines and bigrams: Simple, good sentiment and topic classification,” in _Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)_. Jeju Island, Korea: Association for Computational Linguistics, Jul. 2012, pp. 90–94. [Online]. Available: https://aclanthology.org/P12-2018
  * [20] D. Kingma and J. Ba, “Adam: A method for stochastic optimization,” _International Conference on Learning Representations_ , 12 2014.
  * [21] I. Loshchilov and F. Hutter, “Decoupled weight decay regularization,” in _7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019_. OpenReview.net, 2019. [Online]. Available: https://openreview.net/forum?id=Bkg6RiCqY7
  * [22] T. Wolf, L. Debut, V. Sanh, J. Chaumond, C. Delangue, A. Moi, P. Cistac, T. Rault, R. Louf, M. Funtowicz _et al._ , “Huggingface’s transformers: State-of-the-art natural language processing,” _arXiv preprint arXiv:1910.03771_ , 2019.
  * [23] T. Pires, E. Schlinger, and D. Garrette, “How multilingual is multilingual BERT?” in _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_. Florence, Italy: Association for Computational Linguistics, Jul. 2019.
  * [24] R. Priyadharshini, B. R. Chakravarthi, S. Thavareesan, D. Chinnappa, T. Durairaj, and E. Sherly, “Overview of the dravidiancodemix 2021 shared task on sentiment detection in tamil, malayalam, and kannada,” in _Forum for Information Retrieval Evaluation_ , ser. FIRE 2021. Association for Computing Machinery, 2021.
  * [25] T. Miyato, A. M. Dai, and I. Goodfellow, “Adversarial training methods for semi-supervised text classification,” _arXiv preprint arXiv:1605.07725_ , 2016.
  * [26] P. Kumar and A. Singh, “NutCracker at WNUT-2020 task 2: Robustly identifying informative COVID-19 tweets using ensembling and adversarial training,” in _Proceedings of the Sixth Workshop on Noisy User-generated Text (W-NUT 2020)_. Online: Association for Computational Linguistics, Nov. 2020, pp. 404–408. [Online]. Available: https://aclanthology.org/2020.wnut-1.57
  * [27] H. Goyal, A. Singh, and P. Kumar, “PAW at SemEval-2021 task 2: Multilingual and cross-lingual word-in-context disambiguation : Exploring cross lingual transfer, augmentations and adversarial training,” in _Proceedings of the 15th International Workshop on Semantic Evaluation (SemEval-2021)_. Online: Association for Computational Linguistics, Aug. 2021, pp. 743–747. [Online]. Available: https://aclanthology.org/2021.semeval-1.98


# Generative AI in Vision: A Survey on Models, Metrics and Applications

Gaurav Raut
University of Maryland
<EMAIL_ADDRESS>Apoorv Singh
Carnegie Mellon Univeristy
<EMAIL_ADDRESS>
###### Abstract

Generative AI models have revolutionized various fields by enabling the
creation of realistic and diverse data samples. Among these models, diffusion
models have emerged as a powerful approach for generating high-quality images,
text, and audio. This survey paper provides a comprehensive overview of
generative AI diffusion and legacy models, focusing on their underlying
techniques, applications across different domains, and their challenges. We
delve into the theoretical foundations of diffusion models, including concepts
such as denoising diffusion probabilistic models (DDPM) and score-based
generative modeling. Furthermore, we explore the diverse applications of these
models in text-to-image, image inpainting, and image super-resolution, along
with others, showcasing their potential in creative tasks and data
augmentation. By synthesizing existing research and highlighting critical
advancements in this field, this survey aims to provide researchers and
practitioners with a comprehensive understanding of generative AI diffusion
and legacy models and inspire future innovations in this exciting area of
artificial intelligence.

##  1 Introduction

Figure 1: a) Images generated using stable diffusion[70]; b) Image super-
resolution results from SR3[72]; c) Image inpainting results from Palette[73]

Generative models have long been at the forefront of artificial intelligence,
enabling the creation of synthetic data samples with remarkable realism and
diversity. Initially introduced as a method for denoising images, diffusion
models have evolved to become a versatile framework for generating high-
quality images, text, and audio data. Over the years, they have garnered
significant attention from researchers and practitioners alike for their
ability to capture complex data distributions and produce realistic samples.

Generative models for computer vision started in 1950 through Hidden Markov
models (HMMs) and Gaussian Mixture models (GMMs). These models used hand-
designed features with limited complexities and diversity. With the advent of
deep learning, Generative Adversarial Networks (GANs) and Variational
Autoencoders (VAEs) enabled impressive image generation. However, in practice,
GANs suffered several shortcomings in their architecture [17]. The
simultaneous training of generator and discriminator models was inherently
unstable; sometimes, the generator “collapsed” and outputted lots of similar-
seeming samples. Then came diffusion models, which were inspired by physics.
Diffusion systems borrow from diffusion in non-equilibrium thermodynamics,
where the process increases the entropy or randomness of the system over time.
The recent innovation of diffusion models from OpenAI made them more practical
in everyday applications. This paper dives into a systematic review of
techniques and methodologies involved in SOTA diffusion models.

The main contribution of this work can be summarized as follows:

  * •

An overview of generative vision models to get readers up-to-speed with the
theoretical prerequisites for going through the latest trends in diffusion
models.

  * •

In-depth survey on the SOTA approaches for diffusion models, including

  * •

Highlight current research gaps and future research directions to provoke
researchers to advance this generative vision modeling field further.

##  2 Generative Models in Vision

Figure 2: An extension of generative models classification based on[25]

Generative models in vision can be mainly classified into two categories Fig.
2: models that explicitly learn the probability density function by maximizing
the likelihood or implicitly, where the model doesn’t directly target to learn
the density but does so using other strategies.
Normalizing flow and autoregressive models are dense probability estimators
that model the exact likelihood of a distribution. Although, they are limited
by the complexity of data distribution as converging the objective to achieve
exact density representation of high dimensional complex data, such as images,
can yield to computationally heavy and impractical models.
Variational autoencoders (VAE) alleviate the computation issue by allowing
approximation of the intractable density distribution. This allows for a more
efficient generative model with a trade-off of struggling against capturing
complex data distributions.
Energy-based models offer a flexible modeling objective without any
restrictions. They use an unnormalized representation of the probability
distribution, making them excellent density estimators. However, the
intractable objective makes them computationally inefficient for both training
and sampling.
GANs[26], on the other hand, don’t model the density objective directly. They
rely on using an adversarial approach, which uses a minimax game between a
discriminator and a generator to learn the density estimation explicitly.
Although they have been largely successful on a wide set of applications,
training them is difficult and suffer problems like vanishing gradient and
mode collapse.
Diffusion models use variational or score-based approaches to model the
probability density function. They work by perturbing data with continuous or
discrete noise injection on either the data directly or a latent
representation such as the latent diffusion model[70] and learning the reverse
denoising process.

###  2.1 Diffusion models

####  2.1.1 Denoising Diffusion Probabilistic Models

Denoising Diffusion Probabilistic Models, or DDPMs for short, are a class of
diffusion models that are based on slowly introducing Gaussian noise to a
training data sample $x_{0}$ over a large set of time steps ($1\rightarrow
T$), thereby obtaining a set of noise perturbed latent intermediates of the
original sample ($x_{1}$, $x_{2}$, …, $x_{T}$). This forward process is
governed by the forward diffusion kernel (FDK). At the end of $T$ time steps,
we obtain the resulting $x_{T}$ sample, which can be approximated to represent
isotropic Gaussian noise. From here, a deep neural network is tasked to learn
the FDK, and the parameters of this network are called the Reverse Diffusion
Kernel (RDK). The aim for RDK is to predict the noise introduced by the FDK at
each time step starting from $x_{T}$ and slowly remove the noise to generate
new samples that belong to the same probability distribution as our training
dataset [33, 82].
The forward process: Let $q(\mathbf{x_{0}})$ represent the given sample’s
probability distribution before the noise perturbation. We define a Markov
chain with our FDK defined as a Gaussian distribution,

$q(\mathbf{x}_{1,\ldots,T}):=\prod_{i=1}^{T}q(\mathbf{x}_{t}\mid\mathbf{x}_{t-1})$
(1)
$q(\mathbf{x}_{t}\mid\mathbf{x}_{t-1}):=\mathcal{N}(\mathbf{x}_{t};\sqrt{1-\beta_{t}}\mathbf{x}_{t-1},\beta_{t}\mathrm{I})$
(2)
$\mathbf{x}_{t}=\sqrt{1-\beta_{t}}\mathbf{x}_{t-1}+\sqrt{\beta_{t}}\epsilon$
(3)

where, Eq. 2 defines the FDK; $\epsilon\sim\mathcal{N}(0,\mathbf{I})$. The
term $\beta_{t}$ is a hyper-parameter in the diffusion process controlled by
the variance scheduler. By applying this kernel to our $q(\mathbf{x}_{0})$
repeatedly over $T-1$ time steps, we obtain $q(\mathbf{x}_{T})$ which
approximates to an isotropic Gaussian distribution given that our covariance
matrix in the FDK is isotropic. Since the process is a Markov chain, we can
seamlessly obtain any latent representation and probability distribution from
$\mathbf{x}_{0}$ by simply substituting $\alpha_{t}:=1-\beta_{t}$ and
$\overline{\alpha}_{t}:=\prod_{s=1}^{t}\alpha_{s}$.

$\displaystyle
q({\mathbf{x}_{t}\mid\mathbf{x}_{0}}):=\mathcal{N}(\mathbf{x}_{t};\sqrt{\overline{\alpha}_{t}}\mathbf{x}_{t-1},(1-\overline{\alpha}_{t}\mathbf{I})$
(4)
$\displaystyle\mathbf{x}_{t}:=\sqrt{\overline{\alpha}_{t}}\mathbf{x}_{0}+\sqrt{1-\overline{\alpha}_{t}}\epsilon$
(5)

The reverse process: Starting from approximately an isotropic Gaussian
distribution obtained at the time step $T$, the aim is to learn the RDK
$p_{\theta}(\mathbf{x}_{t-1}\mid\mathbf{x}_{t})$ which will predict the noise
injected at each time step and will generate our original sample
$q(\mathbf{x}_{0})$ back within the finite $T$ time steps. The fact that the
reverse process starts from a random isotropic Gaussian distribution serves as
a cue that a trained network can produce new samples in the probability
distribution of the training dataset while sampling.
The probability distribution in the reverse process to obtain our original
sample can be illustrated as all the possible paths are taken from
$p_{\theta}(\mathbf{x}_{T})$ to obtain $p_{\theta}(\mathbf{x}_{0})\sim
q(\mathbf{x}_{0})$ at each time step in $T\rightarrow 1$.

$p_{\theta}(\mathbf{x}_{0}):=\int p_{\theta}(\mathbf{x}_{0\ldots
T}d\mathbf{x}_{0\ldots T})$ (6)

This integral is intractable since it integrates over a complex high-
dimensional space. To solve this, the authors [33, 82] introduced a
variational lower bound (or Evidence lower bound (ELBO)) of the negative log-
likelihood similar to VAEs[42] to minimize this. Using this, we obtain the
variational lower bound loss, which is [15],

$\begin{split}L_{VLB}=&\underbrace{-\log
p_{\theta}(\mathbf{x}_{0}\mid\mathbf{x}_{1})}_{\mathbf{L}_{0}}+\underbrace{\mathbf{D}_{KL}(p(\mathbf{x}_{T}\mid\mathbf{x}_{0})\|p_{\mathbf{x}_{T}})}_{\mathbf{L}_{T}}\\\
&+\underbrace{\sum_{t>1}\mathbf{D}_{KL}(p(\mathbf{x}_{t-1}\mid\mathbf{x}_{t},\mathbf{x}_{0})\|p_{\theta}(\mathbf{x}_{t-1}\mid\mathbf{x}_{t}))}_{\mathbf{L}_{t-1}},\end{split}$
(7)

Here, we observe that the $\mathbf{L}_{T}$ doesn’t depend on any learnable
parameters from the network and hence can be omitted from the loss. The term
$\mathbf{D}_{KL}$ is called the Kullback-Leibler divergence, a non-symmetric
measure of the statistical distances between two probability distributions.
This loss function trains the network and estimates the forward process
posterior. We then define that the RDK learned by the network is a Gaussian
distribution given as
$p_{\theta}(\mathbf{x}_{t-1}\mid\mathbf{x}_{t})=\mathcal{N}(\mathbf{x}_{t-1};\mu_{\theta}(\mathbf{x}_{t},t),\sigma_{\theta}(\mathbf{x}_{t},t))$
where the network, minimizing the variation lower bound loss, will learn to
estimate the mean $\mu_{\theta}$ and the covariance $\sigma_{\theta}$ of the
forward process posterior distribution. The authors further minimize the loss
function in [33] where they set the variance $\beta$ from Eq. 3 as a constant.
The authors then re-parameterize the mean $\mu_{\theta}$ in terms of noise,
and we obtain the simplified loss function given as:

$L_{simple}:=\mathbb{E}_{\mathbf{x}_{0},\epsilon}[\|\epsilon-\epsilon_{\theta}(\mathbf{x}_{t},t)\|^{2}],$
(8)

which is equivalent to the loss introduced in the score-based models[83].

####  2.1.2 Noise Conditional Score Models

Score-based models define the dataset’s probability distribution as
$q(\mathbf{x})$. The score function for the probability function can then be
defined as the gradient of the log of this probability distribution,
$\nabla_{\mathbf{x}}\log q(\mathbf{x})$ [83]. The objective is to train a deep
neural network, parameterized by $\theta$ to approximate over the score
function of our dataset’s probability function,
$\mathbf{s}_{\theta}(\mathbf{x})\sim\nabla_{\mathbf{x}}\log q(\mathbf{x})$,
also known as score matching. Using the score function instead of the
probability distribution allows the network to work with a tractable objective
by eliminating the normalizing constant [35]. The loss function is the Fischer
divergence between the actual score and the learned score, which is derived to
form,

$\mathbb{E}_{q(\mathbf{x})}[\mathrm{tr}(\nabla_{\mathbf{x}}\mathbf{s}_{\theta}(\mathbf{x}))+\frac{1}{2}\|\mathbf{s}_{\theta}(\mathbf{x})\|^{2}]$
(9)

With a trained network, Langevin dynamics allows us to generate new samples
using only the score function[83].

$\tilde{\mathbf{x}}_{t}=\tilde{\mathbf{x}}_{t-1}+\frac{\epsilon}{2}\nabla_{\mathbf{x}}\log
q(\tilde{\mathbf{x}}_{t-1})+\sqrt{\epsilon}\mathcal{N}(0,\mathbf{I})$ (10)

Under the ideal conditions of $t\rightarrow\infty$ and $\epsilon\rightarrow
0$, we can generate exact samples coming from our dataset’s distribution
$q(\mathbf{x})$ [97]. The generation of new samples can then happen by simply
substituting the learned score function $s_{\theta}(\mathbf{x})$ in Eq. 10.
However, the authors [83] observed that this approach did not do well in
practice because the scores generated were often inaccurate in lower-density
regions. To address this, two solutions are employed to enhance the score
matching using either the denoising score matching [83] or sliced scored
matching [86] where the loss in Eq. 9 is either bypassed or approximated using
random projections.
The idea of denoising score matching is to train the model by perturbing the
dataset by inserting an isotropic Gaussian noise
$\mathcal{N}(0,\sigma_{(1,\ldots,L)}\mathbf{I})$ and
$\sigma_{1}<\ldots<\sigma_{L}$ where the prior data distribution is
approximately equal to the noise perturbed distribution as the noise is
inserted gradually over a large number of steps, $\nabla_{\mathbf{x}}\log
q_{\sigma}(\mathbf{x})\sim\nabla_{\mathbf{x}}\log q(\mathbf{x})$. This results
in the modified loss function of,

$\sum_{i=1}^{L}\lambda(i)\mathbb{E}_{q_{\sigma_{i}}(\mathbf{x})}[\|\nabla_{\mathbf{x}}\log
q_{\sigma_{i}}(\mathbf{x})-s_{\theta}(\mathbf{x},i)\|^{2}]$ (11)

where, $\lambda(i)\sim\sigma_{i}^{2}$ is a positive weighting function. For
sampling, the Langevin dynamics are updated such that the trained network will
produce samples similar to Eq. 10 for each $i=L,\ldots,1$, and the prior
output will be used as the input for the next run. This process is called the
annealed Langevin dynamics, producing a less noisy sample after every run from
$i=L,\ldots,1$.

####  2.1.3 Stochastic Differential Equations Generative Models

So far, we have seen that the diffusion models perturb data over a range of
time steps or iterations $1\rightarrow L$ [33, 83, 84]. However, in [87], the
authors generalized the previous approaches by defining a stochastic
differential equation (SDE) to perturb data with noise in a continuum. The aim
is to govern the diffusion process in both forward and reverse direction as a
representation of an SDE.

$d\mathbf{x}=\mathbf{f}(\mathbf{x},t)dt+g(t)d\mathbf{w}$ (12)

where, $\mathbf{w}$ signifies the Brownian motion, the function
$\mathbf{f}(\mathbf{x},t)$ is called as the drift coefficient and $g(t)$ is
the diffusion coefficient [87]. This is the SDE defining the forward process.
Intuitively, it can be considered an SDE that forces a random sample to
converge in areas of high probability densities. To sample, we need to solve
this SDE backward in time. The reverse process is also a SDE[1, 87] which is
given as,

$d\mathbf{x}=[\mathbf{f}(\mathbf{x},t)-g(t)^{2}\nabla_{x}\log
q_{t}(\mathbf{x})]dt+g(t)d\overline{\mathbf{w}}$ (13)

This SDE has a negative time step since the solution must be reversed
($t=T\rightarrow t=0$). To sample from this SDE, we need to learn the score as
defined in [83] as a function of time. Analogous to the noise conditional
models, where the score function depends on the noise scales
$\sigma_{L,\ldots,i,\ldots,1}$, the score function here will depend on time
$s_{\theta}(\mathbf{x},t)$, formulating the loss function as,

$\mathbb{E}_{t}\mathbb{E}_{q_{t}(\mathbf{x})}[\lambda(t)\|\nabla_{\mathbf{x}}\log
q_{t}(\mathbf{x})-s_{\theta}(\mathbf{x},t)\|^{2}]$ (14)

After training our network $s_{\theta}(\mathbf{x},t)\sim\nabla_{x}\log
q_{t}(\mathbf{x})$, we can generate a new sample by solving the reverse SDE
starting from pure noise $q_{T}(\mathbf{x})$. Many SDE solvers exist, the
simplest of which is the Euler-Maruyama method. Similar to [83], we choose an
infinitesimally small $\Delta(t)$ to solve the generalization of this SDE.

$\displaystyle\Delta(\mathbf{x})=[\mathbf{f}(\mathbf{x},t)-g^{2}(t)s_{\theta}(\mathbf{x},t)]\Delta(t)+g(t)\sqrt{|\Delta
t}|z_{t}$ (15) $\displaystyle\mathbf{x}=\mathbf{x}+\Delta\mathbf{x}$
$\displaystyle t=t+\Delta t$

Following this, the authors improvise the sampling by introducing a predictor-
corrector sampler where the idea is that for every step, the SDE solver will
predict $\mathbf{x}(t+\Delta t)$ and then, the corrector will use this as an
initial sample to refine the sample using $s_{\theta}(\mathbf{x},t+\Delta t)$
by running it through the corrector network.

###  2.2 Generative Adversarial Networks

Generative Adversarial Networks (a.k.a. GANs)[26] are a class of generative
models that implicitly learns the probability distribution $q(\mathbf{x})$ of
the dataset using an adversarial approach where two networks, the
discriminator $D$ and the Generator $G$ play a two-player min-max game. The
discriminator’s objective is to maximize the binary classification of
distinguishing real and generated images, whereas the generator’s objective is
to fool the discriminator into misclassifying the generated images.

$\displaystyle\min_{G}\max_{D}V(D,G)=$ (16)
$\displaystyle\mathbb{E}_{\mathbf{x}\sim q_{\mathbf{x}}}[\log
D(\mathbf{x})]+\mathbb{E}_{\mathbf{z}\sim
p_{\mathbf{z}}(\mathbf{z})}[\log(1-D(G(\mathbf{z})))],$

GANs are notoriously difficult to train [26, 76, 62, 2]. The objective is to
find the Nash equilibrium (also referred to as a saddle point in other
literature) of a two-player game ($D$ & $G$) where both networks try to
minimize their cost function simultaneously Eq. 16. This results in a highly
unstable training process as optimizing $D$ can lead to the deterioration of
$G$ and vice-versa. Mode collapse is another problem where the generator’s
objective function converges to a specific data distribution instead of the
whole training set, thus only generating images belonging to this small
subset. Also, when the discriminator is trained to optimality, the gradients
of $D$ approach zero. This causes the problem of vanishing gradients for the
generator, where it has no guidance into which direction to follow for
achieving optimality.
To prune these problems, various modifications on the vanilla GAN were
introduced, which either suggested architectural optimizations or loss
function optimizations [36]. As studied by [101, 14], the loss function
optimizations were divided into optimizing the discriminator’s $D$ or
generator’s $G$ loss objective. These include minimizing the f-divergences
along with the Jensen-Shannon divergence [59], weight normalization for
stabilizing $D$’s training [55], WGAN and WGAN-GP[3, 30] which changed the
objective of $D$ from binary classification to a probability output by
applying the Earth Mover (EM) or Wasserstein distance, EBGAN [113] introduces
an energy-based formulation of $D$’s objective function where the architecture
of $D$ is modified to be an auto-encoder, BEGAN [5] which uses the same auto-
encoder architecture for $D$ from EBGAN but modifies the objective to use the
Wasserstein distance instead, SAGAN[111] introduces self-attention modules on
both $D$ and $G$ to enhance feature maps and uses spectral normalization[55].

###  2.3 Variational Autoencoders

Variational Autoencoders (VAEs)[41, 42] are generative models based on
learning the latent space representation by projecting a prior $\mathbf{z}$ on
the latent vector before generating a distribution. They have an encoder-
decoder architecture similar to auto-encoders, while mathematically, they
differ a lot [21]. Instead of learning the latent vector as a discrete
representation of the dataset, VAEs learn the probability distribution of this
space. More intuitively, $q(\mathbf{x})$ is mapped to learn a multi-variate
Gaussian distribution represented by the mean $\mu_{\mathbf{z}}$ and co-
variance $\sigma_{\mathbf{z}}$ where $\mathbf{z}$ is in the latent space.
While generating new samples, we want to start with the latent representation
of $\mathbf{z}$ as an isotropic Gaussian distribution
$\mathcal{N}(\mathbf{z}|\mu,\sigma*I)$. The regularization term and the
reconstruction loss are introduced to achieve this. This regularization term
is the KL divergence between the encoder’s estimation of the latent variables
and the standard Gaussian distribution. The loss function is then formulated
as,

$\begin{split}L(\theta,\phi;\mathbf{x})=&\underbrace{-\mathbf{D}_{KL}(p_{\phi}(\mathbf{z}|\mathbf{x})||p_{\theta}(\mathbf{z}))}_{L_{KL}}+\\\
&\underbrace{\mathbb{E}_{p_{\phi}(\mathbf{z}|\mathbf{x})}[\log
p_{\theta}(\mathbf{x}|\mathbf{z})]}_{L_{reconstruction}}\end{split}$ (17)

where $\phi$ and $\theta$ represent the encoder and decoder parameters,
respectively. The only problem here is that $p_{\theta}(\mathbf{z})$ is
intractable. To make the latent variable a learnable parameter, the authors
[42] introduced a reparameterization trick to allow backpropagation. This is
done by separating the stochastic part $\epsilon$ and reconstructing the
latent vector as $\mathbf{z}=\mu+\sigma*\epsilon$. When the network has been
trained to optimality, $p_{\theta}(\mathbf{x}|\mathbf{z})\sim q(\mathbf{x})$.

###  2.4 Autoregressive models

Autoregressive models are generative models that use sequential data to
calculate the likelihood of the next value in series[46, 90]. The joint
distribution of a dataset can be given as,

$q(\mathbf{x})=\prod_{i=1}^{N}q(\mathbf{x}_{i}|\mathbf{x}_{1},\ldots,\mathbf{x}_{n<i})$
(18)

In vision, this translates to generating images by sequencing the generation
of each pixel given the prior pixels[92, 93, 77]. More formally, the
autoregressive network consists of either recurrent or convolutional layers
that jointly learn the dataset’s density distribution in a tractable manner
and, during inference, will run $N=n^{2}$ times for an image of size $n*n$ to
generate a sample. Images are not unlike audio[91] or text[27, 95] where the
data is structured and sequenced. [92] introduces a sequential approach for
image synthesis by masking the pixels on the right and below and only
considering the pixels above and on the left of the pixel we want to predict.

###  2.5 Normalizing flow models

Normalizing flow is a way of mapping a data’s complex probability distribution
$q(\mathbf{x})$ to a simple latent distribution $p(\mathbf{z})$ using a set of
invertible, bijective and continuous functions $\mathbf{z}=f(\mathbf{x})$ such
that both $f$ and $f^{-1}$ are differentiable[69, 19, 20]. When the function
$f$ is a deep neural network, the model is called a normalizing flow model.
Using the rule of change of variables, the probability density can be
explicitly given as,

$p_{\mathbf{x}}(\mathbf{x},\theta)=p_{\mathbf{z}}(f_{\theta}(\mathbf{x}))\biggr{\lvert}\det\frac{\delta
f_{\theta}(\mathbf{x})}{\delta\mathbf{x}}\biggr{\rvert}$ (19)

Since normalizing flow models estimate the exact likelihood of the
distribution, the training is done by minimizing the negative log-likelihood
given as,

$L_{NF}=-\log p_{\mathbf{x}}(\mathbf{x},\theta)=-\log
p_{\mathbf{z}}(f_{\theta}(\mathbf{x}))-\log\biggr{\lvert}\det\frac{\delta
f_{\theta}(\mathbf{x})}{\delta\mathbf{x}}\biggr{\rvert}$ (20)

When the model is trained, the latent representation $\mathbf{z}$, often
chosen as a multivariate Gaussian distribution[43, 60], can generate a sample
from the dataset’s probability distribution by simply applying the inverse
function $f_{\theta}^{-1}$ to it. Although flow models are based on modeling
the exact data distribution, they are often computationally expensive since
they depend on the calculation of jacobians, have scalability and
expressiveness issues on large and complex data distributions, and because of
the invertibility constraint in calculations, require the input and output
dimensions to be the same[8].

###  2.6 Energy Based models

In simplicity, Energy-based models (EBMs) [47, 56, 85] are generative models
that assign an energy function $E_{\theta}(\mathbf{x})$ to a dataset’s
probability distribution $q(\mathbf{x})$ and minimize the energy function for
samples from the dataset while assigning high energy to samples that don’t
belong to it.

$p_{\theta}(\mathbf{x})=\frac{e^{-E_{\theta}(\mathbf{x})}}{Z_{\theta}}$ (21)

EBMs are incredibly flexible in the domain of the type of data[23, 85, 56]
and, in vision, are an excellent choice for tasks of anomaly detection as an
optimally trained model can distinguish between the anomalies and ideal
sample[107].
Because the energy function is unnormalized,
$Z_{\theta}=\int_{\mathbf{x}}e^{-E_{\theta}(\mathbf{x})}d\mathbf{x}$ is used
in Eq. 21 for normalizing the likelihood which is often intractable. This
makes the training and sampling of EBMs difficult, and one has to rely on
computationally heavy methods such as MCMC, contrastive divergence (CD), and
score matching, which make them an impractical choice for fast inference use
cases.

##  3 Evaluation metrics for vision generative models

Evaluating generative models in vision is an active research topic where
different tasks involve specialized metrics such as DrawBench[72],
PartiPrompts[106], CLIPScore[31] for text-to-image tasks or PSNR for image
reconstruction tasks. Here, we mainly introduce metrics that measure image
fidelity and model diversity.

###  3.1 Inception Score (IS)

Inception Score (or IS) was first introduced to assess the quality of the
images generated by GAN as an automated alternative process against human
annotators[76]. Using a feature extracting network (often the Inception
model[88]), which is trained on the same dataset as the generative model, the
score measures two components of the generated samples: entropy of a single
sample over the class labels and secondly, entropy of class distribution over
a large number of samples (suggested close to 50K samples) to measure the
diversity. For a well-trained model, the entropy of a class over a single
sample should be low, and the entropy of class distribution over all generated
samples should be high. This indicates that the network can generate both
meaningful and diverse sets of images. The score is calculated as follows:

$\mathbf{IS}=\exp(\mathbb{E}_{\mathbf{x}}[\mathbf{D}_{KL}(p_{\theta}(\mathbf{y}|\mathbf{x})\|p_{\theta}(\mathbf{y})))]$
(22)

where $\mathbf{D}_{KL}$ is the KL divergence,
$p_{\theta}(\mathbf{y}|\mathbf{x})$ is the predicted class probability and
$p_{\theta}(\mathbf{y})$ is the probability of classes over all generated
images. This implies that the higher the IS score is, the better the model’s
generative capabilities.

###  3.2 Fréchet Inception Distance

The drawback of the Inception score is that it only considers the generated
samples for evaluation and disregards comparing them with the actual dataset.
Also, the IS will only compare the class probabilities as opposed to the image
feature distribution, which causes it to miss the more relevant image features
and requires a labeled dataset. The Fréchet Inception Distance (FID)[22, 32]
allows us to overcome this by comparing the extracted features from a certain
layer of the feature extractor. More specifically, it assumes that the
extracted features belong to a Gaussian distribution with a certain mean $\mu$
and co-variance $\Sigma$ and calculates the Fréchet distance (measures the
similarity between two probability distributions) between samples generated by
the model ($\mu_{g}$,$\Sigma_{g}$) and the samples from the training
($\mu_{t}$,$\Sigma_{t}$) dataset($\sim$ 50K samples). It is given as,

$\mathbf{D}_{FID}=\|\mu_{t}-\mu_{g}\|^{2}+\mathrm{tr}(\Sigma_{t}+\Sigma_{g}-2\sqrt{\Sigma_{t}\Sigma_{g}})$
(23)

Since FID is a similarity distance, the lower the FID is, the better. For
zero-shot applications [57, 65, 105], a modified version called the zero-shot
FID is used where $\mu_{t},\Sigma_{t}$ signify the target distribution based
on textual cues as opposed to the training dataset. Kernel Inception Distance
(KID) [7] is another flavor of the FID, where the distance is measured between
a polynomial representation of the inception layer’s distribution.

###  3.3 Precision and Recall

Precision and recall [75, 45] are two metrics that follow the same motivation
as IS and FID of measuring the quality and diversity of the generated samples
while overcoming issues of mode dropping. It provides a two-dimensional score
where precision measures the quality of images produced while recall measures
the diversity coverage of the generative model.

##  4 Applications in Vision

Generative models find applications in various tasks such as image denoising,
inpainting, super-resolution, text-to-video synthesis, image-to-image
translation, image search, and reverse image search. These applications can be
divided into two broad categories:

###  4.1 Unconditional generation

As the name suggests, unconditional generative models are trained to learn a
target distribution and synthesize new samples without getting conditioned by
any other input. All models described in Sec. 2 can be considered a base
unconditioned model whose only focus is on learning the target
distribution[100, 15]. Unconditional image generation models usually start
with a seed that generates a random noise vector. The model will then use this
vector to create output images that resemble training data distribution.

###  4.2 Conditional generation

On the contrary, conditional diffusion models take a prompt and some random
initial noise and iteratively remove the noise to construct an image. The
prompt guides the denoising process, and once the denoising process ends after
a predetermined number of time steps, the image representation is decoded into
an image. There are several forms of conditional models:

Figure 3: An example of a conditional generative model: Latent diffusion
model[70]

####  4.2.1 Text-to-Image Generation

Text-to-image has been the most naturally prominent use case for generative
models. Providing conditioned textual information for image generation has
improved the model’s generative capabilities[63, 108]. Over the years,
architectures reigning the task have used recurrent layers [53], GANs [67,
110, 49, 78, 39], autoregressive models [64, 18, 106] and the diffusion-based
models [28, 66, 74, 70, 10]. Below we present an overview of their functioning
and also present their metrics in Tab. 1.

Table 1: Comparison of different text-to-image models

Models |  Architecture | Best Reported
---|---|---
|  | Zero-shot FID$\downarrow$ | IS$\uparrow$ | Params | Dataset
StackGAN[110] |  GAN | - | 8.45 | - | COCO
GigaGAN[39] |  GAN | 9.09 | - | 1.0B | COCO
DALL-E[64] |  Autoregressive |  27.50 |  18 | 12B | COCO
GLIDE[58] |  Diffusion | 12.24 |  23.7 | 5B | COCO
Stable Diffusion[70] |  Latent Diffusion | 12.63 | 30.29 | 1.45B | COCO
DALL-E2[66] |  Diffusion | 10.39 | - | 5.5B | COCO
Imagen[74] |  Diffusion | 7.27 | - | 3B | COCO
Parti-20B[105] |  Autoregressive | 3.22 | - | 20B | COCO
Re-Imagen[10] |  Diffusion | 6.88 | - | 3.6B | COCO

Text conditioned GANs: Followed by cGANs [54], [67] introduced embedding
textual information to achieve text-to-image generation. The model is jointly
trained using images and text captions. During sampling, the text prompts are
converted to text encoding using an encoder and compressed using a
128-dimensional fully connected layer. This compressed encoding is
concatenated with the latent vector $\mathbf{z}$ and passed to the generator
to generate images. StackGAN [110] introduced a series of two GAN networks
where the text encodings from the encoder are mapped to a Gaussian
distribution with random noise and are then concatenated with the latent
vector of the stage-I GAN. The stage-I GAN generates a low-dimensional image,
which is further embedded in the latent vector of the stage-II GAN to generate
a high-resolution image. The image compression is done using the trained
discriminator from stage-I.
Text conditioned autoregressive models: DALL-E [64] is a two-stage
autoregressive transformer model where the first stage has a discrete VAE to
tokenize images in a 32$\times$32 grid and the second stage concatenates the
text encodings from a BP-Encoder to create text tokens. These concatenated
image-text tokens are jointly trained to maximize the ELBO [42] to obtain an
image-text distribution. Similarly, CogView [18] employs a VQ-VAE[94] for
image tokenization and SentencePiece[44] for generating text tokens and
Parti[106] uses the ViT-VQGAN[104] as the image tokenizer and a pre-trained
BERT[16] for text encodings with the best fine-tuned model achieving SOTA FID
score of 3.22 on MS-COCO dataset.
Text conditioned Diffusion models: GLIDE[58] uses an ADM model [17] and a
transformer-based text encoder to generate prompts in place of class labels
for image synthesis. DALL-E2 (a.k.a. unCLIP)[66] is a two-stage model where
the first stage uses the CLIP[63] model to generate image embedding from text
captions and the second stage uses these image embedding as a prior to
generating samples via a diffusion decoder. The authors also experimented with
using an autoregressive decoder instead of the diffusion decoder, but the
latter yielded better results. Stable Diffusion[70] is a latent diffusion
model (LDM). Instead of directly dealing with the complex pixel
representation, LDMs apply the DDPM model’s forward kernel on the latent
representation generated by the encoder Fig. 3. A series of trained denoising
U-Net are applied to denoise this corrupted latent space. The text prompts are
embedded in the denoising steps using cross-attention layers. The retrieved
latent space, after denoising, is passed through a decoder to generate the
sample. Because LDMs use diffusion on the latent representation, training, and
sampling have proven to be computationally inexpensive as compared to other
models. Imagen[74] encodes its textual prompts using a T5-XXL LLM model
similar to CLIP. It then generates low-resolution images (64$\times$64) using
a series of denoising U-Net and then up-samples these images using a series of
two super-resolution U-Net diffusion models to generate images of size
256$\times$256 and 1024$\times$1024 respectively. Re-Imagen[10] focuses on
retrieving k-nearest neighboring images from a dataset based on the text
prompts provided and uses these images as a reference to generate new samples.
DALL-E3[6] attends to improving image quality by re-captioning text prompts
into a more descriptive prompt, which has proven to generate higher-quality
images.

####  4.2.2 Image super resolution

ViT-based models have been shown to achieve SOTA results for the task of image
super-resolution[11, 109, 13, 12, 112]. Even so, the generalization
capabilities of generative models can soon catch up the leaderboard[24, 99].
SRGAN[48] first introduced a generative framework for this task using an
adversarial objective. ESRGAN[96] and RFB-ESRGAN[81] further improvise the
SRGAN implementation by employing architectural modifications such as
relativistic discriminator, dense residual block, and upgrading the perceptual
loss. GLEAN [9] introduced a novel encoder-bank-decoder approach where the
encoder’s latent vectors and multi-layer convolutional features are passed to
a StyleGAN[40] based latent bank. This generative bank combines features from
the encoder at various scales and generates new latent feature representations
passed to the decoder to generate a super-resolution image. SR3[72] and
SRDiff[50] were the first diffusion (DDPM) based SR models. SR3 uses a
conditional DDPM U-Net architecture with some adaptations in the residual
layers and conditions the denoising process using an LR image directly at each
iteration. SRDiff uses an encoder-generated embedding of the LR image and
conditions the denoising step by concatenating the embeddings at each
iteration. IDM[24] uses an implicit neural representation in addition to the
conditioned DDPM to achieve a continuous restoration over multiple resolutions
using the current iteration’s features. EDiffSR[99] uses the SDE diffusion
process where isotropic noise is conditioned with the LR image during sampling
to generate the high-resolution image.

####  4.2.3 Image anomaly detection

AnoGAN[79] and f-AnoGAN[80] are unsupervised adversarial anomaly detection
networks that were trained on healthy data and using the proposed anomaly
score along with the residual score; the model predicts anomalies on unseen
data depending on the variation in the learned latent space. DifferNet[71]
uses the normalizing flow model to map the density of healthy image features
extracted from a feature extraction network. By training on healthy data,
anomalies will have a lower likelihood and will be out of distribution in the
density space. FastFlow[103] uses a similar approach but extends the
normalizing flow into a 2D space, which allows to directly output location
results of anomalies. CFLOW-AD[29] uses an encoder feature extractor where
features from every scale are pooled to form multi-scale feature vectors,
which are passed to the specific normalizing flow decoder along with the
positional encoding for localization of anomalies. The outputs from each
decoder are aggregated to generate an anomaly map. AnoDDPM[98] approaches the
problem using a DDPM model with simplex noise for data perturbation. The
diffusion model is trained to generate healthy images. During inference, the
simplex noise perturbs the test sample for a certain number of pre-set steps,
and the denoising diffusion model then generates an anomaly-free image using
the perturbed sample as the prior. Comparing the generated and input samples
using reconstruction error, an anomaly segmentation map is generated.

####  4.2.4 Image inpainting

Image inpainting tasks include restoration, textural synthesis, and mask
filling[4]. Context encoders [61] applied an adversarial loss along with the
reconstruction loss to achieve sharp and coherent mask filling. [37] advances
the simple discriminator by introducing a mixture of local and global context
discriminators. The local discriminator has the filled mask as input, and the
global discriminator takes the whole image as input to collectively create the
discriminator’s objective. Following it, [102] uses a two-step generative
approach where the first generator (trained on reconstruction loss) generates
a coarse prediction and the second generator with a WGAN-GP[30] based
objective is the refinement generator trained on local and global adversarial
loss along with the reconstruction loss. StructurFlow[68] bifurcates the GAN
generation in two steps: the structure generator and the texture generator.
The structure generator creates a smoothened edge-preserved image, and the
texture generator fills the texture in the smooth reconstructed image. The
texture generator uses an additional input of appearance flow in the latent
space, which predicts the texture of the masked regions based on the texture
from source regions. CoModGAN[114] further generalizes the inpainting task
with both input-masked image and stochastic noise-conditioned latent vector
input to the generator, which enabled them for large region image inpainting.
Pallete[73] is a DDPM[33] based image-to-image translation model capable of
image inpainting. It fills the masked region with standard Gaussian noise and
performs the denoising training only on the masked region. RePaint[51]
salvages the pre-trained unconditional DDPM model instead of training a model
for the inpainting task. Since DDPM follows a Markov chain for data
perturbation, the masked input image’s noise-perturbed data is known for every
iteration in reverse. Using this knowledge, the denoising process is
conditioned to add the noise-perturbed image at each reverse step to predict
the masked region. Given the stochastic nature, this process can generate
multiple candidates for the inpainting task.

####  4.2.5 Other tasks

In addition to the above tasks, these models can be used for various other
generative tasks like image-to-image translation[38, 89], image
colorization[73], video generation[34], point cloud generation[52],
restoration, etc.

##  5 Future directions

Some of the unsolved but highly sought-after directions that researchers can
take are:

  * •

Exploring Time-Series Forecasting Applications: Future research could delve
deeper into leveraging diffusion models for improved forecasting accuracy and
efficiency.

  * •

Physics-Inspired Generative Models: Future research could focus on advancing
physics-inspired generative models to achieve unprecedented speed and quality
in content creation.

  * •

Ethical Considerations: Future research could involve addressing issues
related to bias, fairness, and the societal impact of generative diffusion
models.

##  6 Conclusion

This survey paper has provided a detailed examination of generative AI
diffusion models, shedding light on their techniques, applications, and
challenges. Despite their promising capabilities, generative AI diffusion
models still face significant challenges such as training stability,
scalability issues, and interpretability concerns. Addressing these challenges
will be crucial for advancing the field and unlocking the full potential of
diffusion models in generating realistic and diverse data samples. By
synthesizing current research findings and identifying key areas for future
research, this survey aims to guide researchers and practitioners toward
further advancements in generative AI diffusion models, paving the way for
innovative applications and breakthroughs in artificial intelligence.

## References

  * Anderson [1982] Brian DO Anderson.  Reverse-time diffusion equation models.  _Stochastic Processes and their Applications_ , 12(3):313–326, 1982.
  * Arjovsky and Bottou [2017] Martin Arjovsky and Léon Bottou.  Towards principled methods for training generative adversarial networks, 2017.
  * Arjovsky et al. [2017] Martin Arjovsky, Soumith Chintala, and Léon Bottou.  Wasserstein gan, 2017.
  * Bertalmio et al. [2000] Marcelo Bertalmio, Guillermo Sapiro, Vincent Caselles, and Coloma Ballester.  Image inpainting.  In _Proceedings of the 27th Annual Conference on Computer Graphics and Interactive Techniques_ , page 417–424, USA, 2000. ACM Press/Addison-Wesley Publishing Co.
  * Berthelot et al. [2017] David Berthelot, Thomas Schumm, and Luke Metz.  Began: Boundary equilibrium generative adversarial networks, 2017.
  * Betker et al. [2023] James Betker, Gabriel Goh, Li Jing, Tim Brooks, Jianfeng Wang, Linjie Li, Long Ouyang, Juntang Zhuang, Joyce Lee, Yufei Guo, et al.  Improving image generation with better captions.  _Computer Science. https://cdn. openai. com/papers/dall-e-3. pdf_ , 2(3):8, 2023.
  * Bińkowski et al. [2021] Mikołaj Bińkowski, Danica J. Sutherland, Michael Arbel, and Arthur Gretton.  Demystifying mmd gans, 2021.
  * Bond-Taylor et al. [2022] Sam Bond-Taylor, Adam Leach, Yang Long, and Chris G. Willcocks.  Deep generative modelling: A comparative review of vaes, gans, normalizing flows, energy-based and autoregressive models.  _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , 44(11):7327–7347, 2022.
  * Chan et al. [2021] Kelvin C.K. Chan, Xintao Wang, Xiangyu Xu, Jinwei Gu, and Chen Change Loy.  Glean: Generative latent bank for large-factor image super-resolution.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 14245–14254, 2021.
  * Chen et al. [2022] Wenhu Chen, Hexiang Hu, Chitwan Saharia, and William W. Cohen.  Re-imagen: Retrieval-augmented text-to-image generator, 2022.
  * Chen et al. [2023a] Xiangyu Chen, Xintao Wang, Jiantao Zhou, Yu Qiao, and Chao Dong.  Activating more pixels in image super-resolution transformer, 2023a.
  * Chen et al. [2023b] Zheng Chen, Yulun Zhang, Jinjin Gu, Linghe Kong, and Xiaokang Yang.  Recursive generalization transformer for image super-resolution, 2023b.
  * Chen et al. [2023c] Zheng Chen, Yulun Zhang, Jinjin Gu, Linghe Kong, Xiaokang Yang, and Fisher Yu.  Dual aggregation transformer for image super-resolution, 2023c.
  * Creswell et al. [2018] Antonia Creswell, Tom White, Vincent Dumoulin, Kai Arulkumaran, Biswa Sengupta, and Anil A. Bharath.  Generative adversarial networks: An overview.  _IEEE Signal Processing Magazine_ , 35(1):53–65, 2018.
  * Croitoru et al. [2023] Florinel-Alin Croitoru, Vlad Hondru, Radu Tudor Ionescu, and Mubarak Shah.  Diffusion models in vision: A survey.  _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , 45(9):10850–10869, 2023.
  * Devlin et al. [2019] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.  Bert: Pre-training of deep bidirectional transformers for language understanding, 2019.
  * Dhariwal and Nichol [2021] Prafulla Dhariwal and Alex Nichol.  Diffusion models beat gans on image synthesis, 2021.
  * Ding et al. [2021] Ming Ding, Zhuoyi Yang, Wenyi Hong, Wendi Zheng, Chang Zhou, Da Yin, Junyang Lin, Xu Zou, Zhou Shao, Hongxia Yang, and Jie Tang.  Cogview: Mastering text-to-image generation via transformers, 2021.
  * Dinh et al. [2015] Laurent Dinh, David Krueger, and Yoshua Bengio.  Nice: Non-linear independent components estimation, 2015.
  * Dinh et al. [2017] Laurent Dinh, Jascha Sohl-Dickstein, and Samy Bengio.  Density estimation using real nvp, 2017.
  * Doersch [2021] Carl Doersch.  Tutorial on variational autoencoders, 2021.
  * Dowson and Landau [1982] D.C Dowson and B.V Landau.  The fréchet distance between multivariate normal distributions.  _Journal of Multivariate Analysis_ , 12(3):450–455, 1982.
  * Du and Mordatch [2019] Yilun Du and Igor Mordatch.  Implicit generation and modeling with energy based models.  In _Advances in Neural Information Processing Systems_. Curran Associates, Inc., 2019.
  * Gao et al. [2023] Sicheng Gao, Xuhui Liu, Bohan Zeng, Sheng Xu, Yanjing Li, Xiaoyan Luo, Jianzhuang Liu, Xiantong Zhen, and Baochang Zhang.  Implicit diffusion models for continuous super-resolution.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 10021–10030, 2023.
  * Goodfellow [2017] Ian Goodfellow.  Nips 2016 tutorial: Generative adversarial networks, 2017.
  * Goodfellow et al. [2014] Ian J. Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio.  Generative adversarial networks, 2014.
  * Graves [2014] Alex Graves.  Generating sequences with recurrent neural networks, 2014.
  * Gu et al. [2022] Shuyang Gu, Dong Chen, Jianmin Bao, Fang Wen, Bo Zhang, Dongdong Chen, Lu Yuan, and Baining Guo.  Vector quantized diffusion model for text-to-image synthesis, 2022.
  * Gudovskiy et al. [2022] Denis Gudovskiy, Shun Ishizaka, and Kazuki Kozuka.  Cflow-ad: Real-time unsupervised anomaly detection with localization via conditional normalizing flows.  In _Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV)_ , pages 98–107, 2022.
  * Gulrajani et al. [2017] Ishaan Gulrajani, Faruk Ahmed, Martin Arjovsky, Vincent Dumoulin, and Aaron Courville.  Improved training of wasserstein gans, 2017.
  * Hessel et al. [2022] Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Le Bras, and Yejin Choi.  Clipscore: A reference-free evaluation metric for image captioning, 2022.
  * Heusel et al. [2018] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter.  Gans trained by a two time-scale update rule converge to a local nash equilibrium, 2018.
  * Ho et al. [2020] Jonathan Ho, Ajay Jain, and Pieter Abbeel.  Denoising diffusion probabilistic models, 2020.
  * Ho et al. [2022] Jonathan Ho, William Chan, Chitwan Saharia, Jay Whang, Ruiqi Gao, Alexey Gritsenko, Diederik P Kingma, Ben Poole, Mohammad Norouzi, David J Fleet, et al.  Imagen video: High definition video generation with diffusion models.  _arXiv preprint arXiv:2210.02303_ , 2022.
  * Hyvärinen and Dayan [2005] Aapo Hyvärinen and Peter Dayan.  Estimation of non-normalized statistical models by score matching.  _Journal of Machine Learning Research_ , 6(4), 2005.
  * Iglesias et al. [2023] Guillermo Iglesias, Edgar Talavera, and Alberto Díaz-Álvarez.  A survey on gans for computer vision: Recent research, analysis and taxonomy.  _Computer Science Review_ , 48:100553, 2023.
  * Iizuka et al. [2017] Satoshi Iizuka, Edgar Simo-Serra, and Hiroshi Ishikawa.  Globally and locally consistent image completion.  _ACM Trans. Graph._ , 36(4), 2017.
  * Isola et al. [2018] Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, and Alexei A. Efros.  Image-to-image translation with conditional adversarial networks, 2018.
  * Kang et al. [2023] Minguk Kang, Jun-Yan Zhu, Richard Zhang, Jaesik Park, Eli Shechtman, Sylvain Paris, and Taesung Park.  Scaling up gans for text-to-image synthesis, 2023.
  * Karras et al. [2019] Tero Karras, Samuli Laine, and Timo Aila.  A style-based generator architecture for generative adversarial networks, 2019.
  * Kingma and Welling [2019] Diederik P. Kingma and Max Welling.  An introduction to variational autoencoders.  _Foundations and Trends® in Machine Learning_ , 12(4):307–392, 2019.
  * Kingma and Welling [2022] Diederik P Kingma and Max Welling.  Auto-encoding variational bayes, 2022.
  * Kobyzev et al. [2021] Ivan Kobyzev, Simon J.D. Prince, and Marcus A. Brubaker.  Normalizing flows: An introduction and review of current methods.  _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , 43(11):3964–3979, 2021.
  * Kudo and Richardson [2018] Taku Kudo and John Richardson.  Sentencepiece: A simple and language independent subword tokenizer and detokenizer for neural text processing, 2018.
  * Kynkäänniemi et al. [2019] Tuomas Kynkäänniemi, Tero Karras, Samuli Laine, Jaakko Lehtinen, and Timo Aila.  Improved precision and recall metric for assessing generative models, 2019.
  * Larochelle and Murray [2011] Hugo Larochelle and Iain Murray.  The neural autoregressive distribution estimator.  In _Proceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics_ , pages 29–37, Fort Lauderdale, FL, USA, 2011. PMLR.
  * Lecun et al. [2006] Yann Lecun, Sumit Chopra, and Raia Hadsell.  _A tutorial on energy-based learning_.  2006\.
  * Ledig et al. [2017] Christian Ledig, Lucas Theis, Ferenc Huszar, Jose Caballero, Andrew Cunningham, Alejandro Acosta, Andrew Aitken, Alykhan Tejani, Johannes Totz, Zehan Wang, and Wenzhe Shi.  Photo-realistic single image super-resolution using a generative adversarial network, 2017.
  * Li et al. [2019] Bowen Li, Xiaojuan Qi, Thomas Lukasiewicz, and Philip H. S. Torr.  Controllable text-to-image generation, 2019.
  * Li et al. [2022] Haoying Li, Yifan Yang, Meng Chang, Shiqi Chen, Huajun Feng, Zhihai Xu, Qi Li, and Yueting Chen.  Srdiff: Single image super-resolution with diffusion probabilistic models.  _Neurocomputing_ , 479:47–59, 2022.
  * Lugmayr et al. [2022] Andreas Lugmayr, Martin Danelljan, Andres Romero, Fisher Yu, Radu Timofte, and Luc Van Gool.  Repaint: Inpainting using denoising diffusion probabilistic models.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 11461–11471, 2022.
  * Luo and Hu [2021] Shitong Luo and Wei Hu.  Diffusion probabilistic models for 3d point cloud generation.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 2837–2845, 2021.
  * Mansimov et al. [2016] Elman Mansimov, Emilio Parisotto, Jimmy Lei Ba, and Ruslan Salakhutdinov.  Generating images from captions with attention, 2016.
  * Mirza and Osindero [2014] Mehdi Mirza and Simon Osindero.  Conditional generative adversarial nets, 2014.
  * Miyato et al. [2018] Takeru Miyato, Toshiki Kataoka, Masanori Koyama, and Yuichi Yoshida.  Spectral normalization for generative adversarial networks, 2018.
  * Ngiam et al. [2011] Jiquan Ngiam, Zhenghao Chen, Pang Koh, and Andrew Ng.  Learning deep energy models.  pages 1105–1112, 2011.
  * Nichol et al. [2022a] Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, and Mark Chen.  Glide: Towards photorealistic image generation and editing with text-guided diffusion models, 2022a.
  * Nichol et al. [2022b] Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, and Mark Chen.  Glide: Towards photorealistic image generation and editing with text-guided diffusion models, 2022b.
  * Nowozin et al. [2016] Sebastian Nowozin, Botond Cseke, and Ryota Tomioka.  f-gan: Training generative neural samplers using variational divergence minimization, 2016.
  * Papamakarios et al. [2021] George Papamakarios, Eric Nalisnick, Danilo Jimenez Rezende, Shakir Mohamed, and Balaji Lakshminarayanan.  Normalizing flows for probabilistic modeling and inference, 2021.
  * Pathak et al. [2016] Deepak Pathak, Philipp Krahenbuhl, Jeff Donahue, Trevor Darrell, and Alexei A Efros.  Context encoders: Feature learning by inpainting.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 2536–2544, 2016.
  * Radford et al. [2016] Alec Radford, Luke Metz, and Soumith Chintala.  Unsupervised representation learning with deep convolutional generative adversarial networks, 2016.
  * Radford et al. [2021] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever.  Learning transferable visual models from natural language supervision, 2021.
  * Ramesh et al. [2021] Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever.  Zero-shot text-to-image generation, 2021.
  * Ramesh et al. [2022a] Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen.  Hierarchical text-conditional image generation with clip latents, 2022a.
  * Ramesh et al. [2022b] Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen.  Hierarchical text-conditional image generation with clip latents, 2022b.
  * Reed et al. [2016] Scott Reed, Zeynep Akata, Xinchen Yan, Lajanugen Logeswaran, Bernt Schiele, and Honglak Lee.  Generative adversarial text to image synthesis, 2016.
  * Ren et al. [2019] Yurui Ren, Xiaoming Yu, Ruonan Zhang, Thomas H Li, Shan Liu, and Ge Li.  Structureflow: Image inpainting via structure-aware appearance flow.  In _Proceedings of the IEEE/CVF international conference on computer vision_ , pages 181–190, 2019.
  * Rezende and Mohamed [2016] Danilo Jimenez Rezende and Shakir Mohamed.  Variational inference with normalizing flows, 2016.
  * Rombach et al. [2022] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer.  High-resolution image synthesis with latent diffusion models, 2022.
  * Rudolph et al. [2020] Marco Rudolph, Bastian Wandt, and Bodo Rosenhahn.  Same same but differnet: Semi-supervised defect detection with normalizing flows, 2020.
  * Saharia et al. [2021] Chitwan Saharia, Jonathan Ho, William Chan, Tim Salimans, David J. Fleet, and Mohammad Norouzi.  Image super-resolution via iterative refinement, 2021.
  * Saharia et al. [2022a] Chitwan Saharia, William Chan, Huiwen Chang, Chris A. Lee, Jonathan Ho, Tim Salimans, David J. Fleet, and Mohammad Norouzi.  Palette: Image-to-image diffusion models, 2022a.
  * Saharia et al. [2022b] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Seyed Kamyar Seyed Ghasemipour, Burcu Karagol Ayan, S. Sara Mahdavi, Rapha Gontijo Lopes, Tim Salimans, Jonathan Ho, David J Fleet, and Mohammad Norouzi.  Photorealistic text-to-image diffusion models with deep language understanding, 2022b.
  * Sajjadi et al. [2018] Mehdi S. M. Sajjadi, Olivier Bachem, Mario Lucic, Olivier Bousquet, and Sylvain Gelly.  Assessing generative models via precision and recall, 2018.
  * Salimans et al. [2016] Tim Salimans, Ian Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen.  Improved techniques for training gans, 2016.
  * Salimans et al. [2017] Tim Salimans, Andrej Karpathy, Xi Chen, and Diederik P. Kingma.  Pixelcnn++: Improving the pixelcnn with discretized logistic mixture likelihood and other modifications, 2017.
  * Sauer et al. [2023] Axel Sauer, Tero Karras, Samuli Laine, Andreas Geiger, and Timo Aila.  Stylegan-t: Unlocking the power of gans for fast large-scale text-to-image synthesis, 2023.
  * Schlegl et al. [2017] Thomas Schlegl, Philipp Seeböck, Sebastian M. Waldstein, Ursula Schmidt-Erfurth, and Georg Langs.  Unsupervised anomaly detection with generative adversarial networks to guide marker discovery, 2017.
  * Schlegl et al. [2019] Thomas Schlegl, Philipp Seeböck, Sebastian M. Waldstein, Georg Langs, and Ursula Schmidt-Erfurth.  f-anogan: Fast unsupervised anomaly detection with generative adversarial networks.  _Medical Image Analysis_ , 54:30–44, 2019.
  * Shang et al. [2020] Taizhang Shang, Qiuju Dai, Shengchen Zhu, Tong Yang, and Yandong Guo.  Perceptual extreme super resolution network with receptive field block, 2020.
  * Sohl-Dickstein et al. [2015] Jascha Sohl-Dickstein, Eric A. Weiss, Niru Maheswaranathan, and Surya Ganguli.  Deep unsupervised learning using nonequilibrium thermodynamics, 2015.
  * Song and Ermon [2020a] Yang Song and Stefano Ermon.  Generative modeling by estimating gradients of the data distribution, 2020a.
  * Song and Ermon [2020b] Yang Song and Stefano Ermon.  Improved techniques for training score-based generative models, 2020b.
  * Song and Kingma [2021] Yang Song and Diederik P. Kingma.  How to train your energy-based models, 2021.
  * Song et al. [2019] Yang Song, Sahaj Garg, Jiaxin Shi, and Stefano Ermon.  Sliced score matching: A scalable approach to density and score estimation, 2019.
  * Song et al. [2021] Yang Song, Jascha Sohl-Dickstein, Diederik P. Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole.  Score-based generative modeling through stochastic differential equations, 2021.
  * Szegedy et al. [2014] Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich.  Going deeper with convolutions, 2014.
  * Tumanyan et al. [2023] Narek Tumanyan, Michal Geyer, Shai Bagon, and Tali Dekel.  Plug-and-play diffusion features for text-driven image-to-image translation.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 1921–1930, 2023.
  * Uria et al. [2016] Benigno Uria, Marc-Alexandre Côté, Karol Gregor, Iain Murray, and Hugo Larochelle.  Neural autoregressive distribution estimation, 2016.
  * van den Oord et al. [2016a] Aaron van den Oord, Sander Dieleman, Heiga Zen, Karen Simonyan, Oriol Vinyals, Alex Graves, Nal Kalchbrenner, Andrew Senior, and Koray Kavukcuoglu.  Wavenet: A generative model for raw audio, 2016a.
  * van den Oord et al. [2016b] Aaron van den Oord, Nal Kalchbrenner, and Koray Kavukcuoglu.  Pixel recurrent neural networks, 2016b.
  * van den Oord et al. [2016c] Aaron van den Oord, Nal Kalchbrenner, Oriol Vinyals, Lasse Espeholt, Alex Graves, and Koray Kavukcuoglu.  Conditional image generation with pixelcnn decoders, 2016c.
  * van den Oord et al. [2018] Aaron van den Oord, Oriol Vinyals, and Koray Kavukcuoglu.  Neural discrete representation learning, 2018.
  * Vaswani et al. [2023] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin.  Attention is all you need, 2023.
  * Wang et al. [2018] Xintao Wang, Ke Yu, Shixiang Wu, Jinjin Gu, Yihao Liu, Chao Dong, Chen Change Loy, Yu Qiao, and Xiaoou Tang.  Esrgan: Enhanced super-resolution generative adversarial networks, 2018.
  * Welling and Teh [2011] Max Welling and Yee W Teh.  Bayesian learning via stochastic gradient langevin dynamics.  In _Proceedings of the 28th international conference on machine learning (ICML-11)_ , pages 681–688, 2011.
  * Wyatt et al. [2022] Julian Wyatt, Adam Leach, Sebastian M. Schmon, and Chris G. Willcocks.  Anoddpm: Anomaly detection with denoising diffusion probabilistic models using simplex noise.  In _2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW)_ , pages 649–655, 2022.
  * Xiao et al. [2024] Yi Xiao, Qiangqiang Yuan, Kui Jiang, Jiang He, Xianyu Jin, and Liangpei Zhang.  Ediffsr: An efficient diffusion probabilistic model for remote sensing image super-resolution.  _IEEE Transactions on Geoscience and Remote Sensing_ , 62:1–14, 2024.
  * Yang et al. [2023] Ling Yang, Zhilong Zhang, Yang Song, Shenda Hong, Runsheng Xu, Yue Zhao, Wentao Zhang, Bin Cui, and Ming-Hsuan Yang.  Diffusion models: A comprehensive survey of methods and applications, 2023.
  * Yi et al. [2019] Xin Yi, Ekta Walia, and Paul Babyn.  Generative adversarial network in medical imaging: A review.  _Medical Image Analysis_ , 58:101552, 2019.
  * Yu et al. [2018] Jiahui Yu, Zhe Lin, Jimei Yang, Xiaohui Shen, Xin Lu, and Thomas S. Huang.  Generative image inpainting with contextual attention.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2018.
  * Yu et al. [2021] Jiawei Yu, Ye Zheng, Xiang Wang, Wei Li, Yushuang Wu, Rui Zhao, and Liwei Wu.  Fastflow: Unsupervised anomaly detection and localization via 2d normalizing flows, 2021.
  * Yu et al. [2022a] Jiahui Yu, Xin Li, Jing Yu Koh, Han Zhang, Ruoming Pang, James Qin, Alexander Ku, Yuanzhong Xu, Jason Baldridge, and Yonghui Wu.  Vector-quantized image modeling with improved vqgan, 2022a.
  * Yu et al. [2022b] Jiahui Yu, Yuanzhong Xu, Jing Yu Koh, Thang Luong, Gunjan Baid, Zirui Wang, Vijay Vasudevan, Alexander Ku, Yinfei Yang, Burcu Karagol Ayan, Ben Hutchinson, Wei Han, Zarana Parekh, Xin Li, Han Zhang, Jason Baldridge, and Yonghui Wu.  Scaling autoregressive models for content-rich text-to-image generation, 2022b.
  * Yu et al. [2022c] Jiahui Yu, Yuanzhong Xu, Jing Yu Koh, Thang Luong, Gunjan Baid, Zirui Wang, Vijay Vasudevan, Alexander Ku, Yinfei Yang, Burcu Karagol Ayan, Ben Hutchinson, Wei Han, Zarana Parekh, Xin Li, Han Zhang, Jason Baldridge, and Yonghui Wu.  Scaling autoregressive models for content-rich text-to-image generation, 2022c.
  * Zhai et al. [2016] Shuangfei Zhai, Yu Cheng, Weining Lu, and Zhongfei Zhang.  Deep structured energy based models for anomaly detection.  In _International conference on machine learning_ , pages 1100–1109. PMLR, 2016.
  * Zhang et al. [2023a] Chenshuang Zhang, Chaoning Zhang, Mengchun Zhang, and In So Kweon.  Text-to-image diffusion models in generative ai: A survey, 2023a.
  * Zhang et al. [2023b] Dafeng Zhang, Feiyu Huang, Shizhuo Liu, Xiaobing Wang, and Zhezhu Jin.  Swinfir: Revisiting the swinir with fast fourier convolution and improved training for image super-resolution, 2023b.
  * Zhang et al. [2017] Han Zhang, Tao Xu, Hongsheng Li, Shaoting Zhang, Xiaogang Wang, Xiaolei Huang, and Dimitris Metaxas.  Stackgan: Text to photo-realistic image synthesis with stacked generative adversarial networks, 2017.
  * Zhang et al. [2019] Han Zhang, Ian Goodfellow, Dimitris Metaxas, and Augustus Odena.  Self-attention generative adversarial networks, 2019.
  * Zhang et al. [2024] Leheng Zhang, Yawei Li, Xingyu Zhou, Xiaorui Zhao, and Shuhang Gu.  Transcending the limit of local window: Advanced super-resolution transformer with adaptive token dictionary, 2024.
  * Zhao et al. [2017] Junbo Zhao, Michael Mathieu, and Yann LeCun.  Energy-based generative adversarial network, 2017.
  * Zhao et al. [2021] Shengyu Zhao, Jonathan Cui, Yilun Sheng, Yue Dong, Xiao Liang, Eric I Chang, and Yan Xu.  Large scale image completion via co-modulated generative adversarial networks, 2021.