# Performance of Data Augmentation Methods for Brazilian Portuguese Text
Classification

Marcellus Amadeus  Paulo Branco
Alana AI
<EMAIL_ADDRESS>

###### Abstract

Improving machine learning performance while increasing model generalization
has been a constantly pursued goal by AI researchers. Data augmentation
techniques are often used towards achieving this target, and most of its
evaluation is made using English corpora. In this work, we took advantage of
different existing data augmentation methods to analyze their performances
applied to text classification problems using Brazilian Portuguese corpora. As
a result, our analysis shows some putative improvements in using some of these
techniques; however, it also suggests further exploitation of language bias
and non-English text data scarcity.

##  1 Introduction

Text classification is a common and essential task in natural language
processing (NLP). Much work has been done in the area. State-of-the-art
results achieved high accuracy on several related tasks such as sentiment
analysis (Thongtan and Phienthrakul, 2019; Jiang et al., 2020) and topic
classification (Kesiraju et al., 2019; Meng et al., 2020). Still, high
performance often depends on the size, quality, and perhaps most important,
training data availability. Gathering data can quickly become a tedious
assignment and it is especially challenging for non-English languages that
likely have fewer resources since most current researches use English corpora.
In such a scarce scenario, data augmentation techniques are even handier to
deal with data. Data augmentation is already widely used in computer vision
(Simard et al., 2012; Szegedy et al., 2015; Krizhevsky et al., 2017) and
speech (Cui et al., 2015; Ko et al., 2015), where it boosts performance,
especially on smaller datasets.

Text data augmentation techniques use various strategies such as applying a
set of universal functions to quickly and easily introduce diversity in the
dataset (Wei and Zou, 2019), generating new data by translating sentences into
another language and back into English (Yu et al., 2018) (also referred to as
"back translation"), using predictive language models for synonym replacement
(Kobayashi, 2018), and others. Thus, implementation cost versus performance
gain varies from technique to technique. Still, all of the methods rely on at
least one kind of language resource, which may be a WordNet dictionary, a Word
Embedding model, datasets with specific formats, or another type of dependency
closely tied to a single language.

Most of these text augmentation techniques were originally developed using
English corpora. However, some recent works extend their application scenarios
using a specific technique on, whether various languages (Ciolino et al.,
2021), or even Brazilian Portuguese corpora as evaluation language (Veríssimo
and Costa, 2020; Venturott and Ciarelli, 2020). Each work uses distinct
processes and datasets and, given some limitations, data augmentation improved
their results.

In this paper, we revisit some different existing text augmentation methods,
gathering and reconstructing the necessary resources to reproduce each
technique. Then, by applying them to Brazilian Portuguese corpora, we attempt
to expand and validate these techniques in a more generic way. Using McNemar’s
statistical test to compare the classification models, we achieve a set of
different results showing that text augmentation methods are particularly
useful, although language-specific fine-tunings should be considered to ensure
significant positive gains.

##  2 Experimental Setup

Initially, we cluster the text augmentation methods into three main groups:
(1) Easy Data Augmentation (EDA): based on Wei and Zou (2019), it consists of
a collection of four functions (synonym replacement, random insertion, random
swap, and random deletion). The first two rely on a map of synonyms; in this
case, we use the PPDB Portuguese paraphrase pack (available at
http://paraphrase.org). For the other parameters of the technique, we use the
same ones in the original paper. (2) Synonym (Syn): Many text augmentation
methods are based on some kind of synonym or allonym replacement. They
primarily use language models to effectively replace words to create diversity
in the training set synthetically. Here we use the `nlpaug}
library\footnote`https://github.com/makcedward/nlpaug to produce a pipeline of
word replacement (sequential flow) that uses the PPDB Portuguese paraphrase
pack, the Fasttext Portuguese Word Embedding
model111https://fasttext.cc/docs/en/crawl-vectors.html (Grave et al., 2018),
and the Portuguese BERT model222https://huggingface.co/neuralmind/bert-large-
portuguese-cased (Souza et al., 2019). Combined, the three resources provide a
smart replacement for similar words. We generate one new sentence per sample
in the training set using this method. (3) Back translation (BT): many
translation APIs are publicly available and free (up to reasonable usage). So
for this method, we generate one sentence per sample in the training set using
AWS Amazon Translate service, chosen for convenience (Microsoft and Google
also offer that kind of API, where it is also possible to use pre-trained
models to perform the "back translation" technique).

###  2.1 Benchmark Datasets

We conduct experiments on three public available Brazilian Portuguese text
classification datasets: (1) Tweets: TweetSentBR (Brum and das Graças Volpe
Nunes, 2017) is a corpus of 10,648 Tweets manually annotated in one of the
three sentiment classes: Positive - the user meant a positive reaction or
evaluation about the main topic on the post; Negative - the user meant a
negative reaction or evaluation about the main topic on the post; Neutral -
tweets not belonging to any of the last classes, usually not making a point,
out of topic, irrelevant, confusing or containing only objective data. (2)
B2W: B2W Open Product Reviews (available at
https://github.com/b2wdigital/b2w-reviews01) consists of a binary
classification corpus of 132,373 product reviews. The labels represent the
willingness of the customer to recommend the product to someone else; and (3)
Mercadolibre: Mercado Libre Data Challenge 2019 (available at https://ml-
challenge.mercadolibre.com) is a corpus of 693,318 purchase histories where
the goal is to predict the next item bought by the user. As demonstrated by
Wei and Zou (2019), some text augmentation techniques have a more significant
impact on smaller datasets; for that reason, we randomly resample the datasets
into subsets with different sizes $N=\\{500,1000,2000,5000,10000\\}$. For each
subset $N$, we use a 75% train split rate. Also, we use different percentages
$p=\\{0\%,5\%,10\%,20\%\\}$ of the train set in the augmentation process.
Finally, we run 15 rounds of the whole experiment for each dataset, totaling
2,700 trained models (3 datasets X 3 augmentation groups X 5 subset sizes X 4
augmentation percentages X 15 rounds).

###  2.2 Text Classification Models

Due to a large number of models, we opted to use non-deep-learning classifiers
to perform the benchmark based on their ease of use and usually faster
training. Many popular algorithms for text classification are not based on
neural networks; one of the most prominent is the Support Vector Machine (SVM)
algorithm (Kowsari et al., 2019). We use the sci-kit-learn
library333https://scikit-
learn.org/stable/modules/generated/sklearn.svm.SVC.html#sklearn.svm.SVC
implementation of SVM with `C=10}, \mintinline`Pythonkernel=rbf, and
`gamma=scale} for all trained models (other parameters set to default). As the
featurizer, we use the Fasttext Portuguese Word Embedding
model\footnote`https://fasttext.cc/docs/en/crawl-vectors.html (Grave et al.,
2018) to extract the sentence vector for each sample.

###  2.3 Model Evaluation

We use the F1-score (weighted F1-score for multi-label datasets) as the
evaluation metric. The F1-score is the harmonic mean of precision and recall,
and it was applied as a filter, leaving only the best 180 models for each
experiment round and parameter combination. After the models’ filtering, we
use the F1-score of each baseline model ($p=0\%$) to compute the model gain,
_i.e._ the difference between the baseline F1-score and the F1-score of each
augmented model ($p>0\%$). Also, to determine if a model’s performance is
significantly different, we use the continuity-corrected version of McNemar’s
test. This nonparametric statistical method is used on paired nominal data and
has been used to confront NLP models (Jiang et al., 2021; Chen et al., 2021).

##  3 Results

All details regarding baseline F1-scores, F1-score gains and $p$-values can be
found in Appendices A, B and C, respectively.

###  3.1 Tweets Dataset

As a result of the Tweets dataset analysis, the highest F1-score gains were
obtained on the smallest dataset subset for EDA and Syn augmentation groups.
Figure 1 shows the F1-score average gains for each augmentation group.

Figure 1: F1-score average gains for each augmentation group on Tweets
dataset.

McNemar’s test results show no significant difference between baseline models
and augmented models ($p$-value < 0.05). The test was applied only in positive
F1-score gain models and is depicted in Figure 2.

Figure 2: -log($p$-values) for each augmentation group on Tweets dataset.
Dashed red line: alpha = 0.05.

###  3.2 B2W Dataset

As with Tweets, the analysis of the B2W dataset resulted in the highest
F1-score gain belonging to the EDA augmentation group. Also, the dataset
subsets sizes that achieved the best results were 500 and 2000. Likewise, the
same pattern was observed in the Tweets dataset. Figure 3 shows the F1-score
average gains for each augmentation group.

Figure 3: F1-score average gains for each augmentation group on the B2W
dataset.

The statistical significance analysis was applied following the same scheme as
the Tweets dataset. Figure 4 shows the positive gain models’ $p$-values.

Figure 4: -log($p$-values) for each augmentation group on B2W dataset. Dashed
red line: alpha = 0.05.

The B2W dataset analysis resulted in a significant model, as shown in Figure
4. This best model, trained with a subset size 2000 and 0.05 augmentation,
reached a $p$-value of 0.042957.

###  3.3 Mercado Libre Dataset

Finally, in the Mercado Libre dataset, the EDA augmentation group still
achieved the best F1-score gain. Figure 5 shows the F1-score average gain for
each group.

Figure 5: F1-score average gains for each augmentation group on Mercado Libre
dataset.

All positive F1-score gains p-values resulting from McNemar’s test are
depicted in Figure 6.

Figure 6: -log($p$-values) for each augmentation group on Mercado Libre.
Dashed red line: alpha = 0.05.

Although no significant text classification model was found (Figure 6), the
dataset subset size 5000 models obtained a satisfactory performance.

###  3.4 Augmentation Group Performance

Combining all results across the three augmentation groups, Figure 7 shows the
F1-score gains for each dataset subset size.

Figure 7: F1-score gains for each augmentation group, combining all datasets.

The Syn augmentation group achieved the best overall performance, as shown in
Figure 7.

##  4 Conclusions

We grouped and compared three different text augmentation techniques that have
been reported as significant additions to solving text classification tasks.
Initially developed using English corpora, we applied these techniques using
Brazilian Portuguese publicly available language resources. We trained 2,700
text classification models using three different datasets. For each dataset,
the models were generated using combinations of the following attributes:
dataset subset size, percentage of augmentation, and the group of augmentation
technique. Comparing augmented and non-augmented best models, our analysis
showed a slight upward trend in the F1 score gain, although no expressive
statistical significance was found. This result might be caused by the model
choice, _i.e._ the SVM is considered a low-sensitive method regarding the
train set size; and by the nature of the datasets, _i.e._ the colloquialism
inherent in the Tweets texts and a large number of targets in the Mercado
Libre classification task, adding noise to the trained models. Also, since all
the augmentation techniques were developed using English corpora, some
putative language dependency might occur, biasing the final results. In future
work, we are planning to adjust the model choice and expand the analysis on
the Brazilian Portuguese language. Besides, in order to increase the available
Brazilian Portuguese data volume, we are planning to curate and annotate a new
corpus.

## References

  * Brum and das Graças Volpe Nunes (2017) Henrico Bertini Brum and Maria das Graças Volpe Nunes. 2017.  Building a sentiment corpus of tweets in brazilian portuguese.
  * Chen et al. (2021) Timothy L. Chen, Max Emerling, Gunvant R. Chaudhari, Yeshwant R. Chillakuru, Youngho Seo, Thienkhai H. Vu, and Jae Ho Sohn. 2021.  Domain specific word embeddings for natural language processing in radiology.  _Journal of Biomedical Informatics_ , 113:103665.
  * Ciolino et al. (2021) Matthew Ciolino, David Noever, and Josh Kalin. 2021.  Multilingual augmenter: The model chooses.
  * Cui et al. (2015) Xiaodong Cui, Vaibhava Goel, and Brian Kingsbury. 2015.  Data augmentation for deep neural network acoustic modeling.  _IEEE/ACM Transactions on Audio, Speech, and Language Processing_ , 23(9):1469–1477.
  * Grave et al. (2018) Edouard Grave, Piotr Bojanowski, Prakhar Gupta, Armand Joulin, and Tomas Mikolov. 2018.  Learning word vectors for 157 languages.  In _Proceedings of the International Conference on Language Resources and Evaluation (LREC 2018)_.
  * Jiang et al. (2020) Haoming Jiang, Pengcheng He, Weizhu Chen, Xiaodong Liu, Jianfeng Gao, and Tuo Zhao. 2020.  SMART: Robust and efficient fine-tuning for pre-trained natural language models through principled regularized optimization.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 2177–2190, Online. Association for Computational Linguistics.
  * Jiang et al. (2021) Tao Jiang, Jian Ping Li, Amin Ul Haq, Abdus Saboor, and Amjad Ali. 2021.  A novel stacking approach for accurate detection of fake news.  _IEEE Access_ , 9:22626–22639.
  * Kesiraju et al. (2019) Santosh Kesiraju, Oldřich Plchot, Lukáš Burget, and Suryakanth V Gangashetty. 2019.  Learning document embeddings along with their uncertainties.
  * Ko et al. (2015) T. Ko, Vijayaditya Peddinti, D. Povey, and S. Khudanpur. 2015.  Audio augmentation for speech recognition.  In _INTERSPEECH_.
  * Kobayashi (2018) Sosuke Kobayashi. 2018.  Contextual augmentation: Data augmentation by words with paradigmatic relations.  In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers)_. Association for Computational Linguistics.
  * Kowsari et al. (2019) Kowsari, Jafari Meimandi, Heidarysafa, Mendu, Barnes, and Brown. 2019.  Text classification algorithms: A survey.  _Information_ , 10(4):150.
  * Krizhevsky et al. (2017) Alex Krizhevsky, Ilya Sutskever, and Geoffrey E. Hinton. 2017.  ImageNet classification with deep convolutional neural networks.  _Communications of the ACM_ , 60(6):84–90.
  * Meng et al. (2020) Yu Meng, Yunyi Zhang, Jiaxin Huang, Yu Zhang, Chao Zhang, and Jiawei Han. 2020.  Hierarchical topic mining via joint spherical tree and text embedding.
  * Simard et al. (2012) Patrice Y. Simard, Yann A. LeCun, John S. Denker, and Bernard Victorri. 2012.  Transformation invariance in pattern recognition – tangent distance and tangent propagation.  In _Lecture Notes in Computer Science_ , pages 235–269. Springer Berlin Heidelberg.
  * Souza et al. (2019) Fábio Souza, Rodrigo Nogueira, and Roberto Lotufo. 2019.  Portuguese named entity recognition using bert-crf.
  * Szegedy et al. (2015) Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. 2015.  Going deeper with convolutions.  In _2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_. IEEE.
  * Thongtan and Phienthrakul (2019) Tan Thongtan and Tanasanee Phienthrakul. 2019.  Sentiment classification using document embeddings trained with cosine similarity.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics: Student Research Workshop_ , pages 407–414, Florence, Italy. Association for Computational Linguistics.
  * Venturott and Ciarelli (2020) Lígia Iunes Venturott and Patrick Marques Ciarelli. 2020.  Data augmentation for improving hate speech detection on social networks.  In _Proceedings of the Brazilian Symposium on Multimedia and the Web_. ACM.
  * Veríssimo and Costa (2020) Vinícius Veríssimo and Rostand Costa. 2020.  Using data augmentation and neural networks to improve the emotion analysis of brazilian portuguese texts.  In _Proceedings of the Brazilian Symposium on Multimedia and the Web_. ACM.
  * Wei and Zou (2019) Jason Wei and Kai Zou. 2019.  EDA: Easy data augmentation techniques for boosting performance on text classification tasks.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_. Association for Computational Linguistics.
  * Yu et al. (2018) Adams Wei Yu, David Dohan, Quoc Le, Thang Luong, Rui Zhao, and Kai Chen. 2018.  Fast and accurate reading comprehension by combining self-attention and convolution.  In _International Conference on Learning Representations_.

##  Appendix A Detailed Baseline F1-scores

###  A.1 Tweets Dataset

Table 1 shows all baseline F1-scores for Tweets dataset.

| Augm. Type
---|---
| EDA | Syn | BT
| Augm.Perc.
Subset Size | 0.05 | 0.1 | 0.2 | 0.05 | 0.1 | 0.2 | 0.05 | 0.1 | 0.2
500 | 0.83 | 0.82 | 0.83 | 0.83 | 0.83 | 0.83 | 0.87 | 0.87 | 0.87
1000 | 0.80 | 0.79 | 0.80 | 0.80 | 0.80 | 0.80 | 0.80 | 0.80 | 0.80
2000 | 0.81 | 0.81 | 0.81 | 0.79 | 0.79 | 0.79 | 0.78 | 0.78 | 0.78
5000 | 0.79 | 0.80 | 0.80 | 0.78 | 0.78 | 0.78 | 0.79 | 0.79 | 0.79
10000 | 0.78 | 0.78 | 0.77 | 0.77 | 0.78 | 0.77 | 0.78 | 0.78 | 0.78
Table 1: Baseline F1-scores for Tweets dataset.

###  A.2 B2W Dataset

Table 2 shows all baseline F1-scores for B2W dataset.

| Augm. Type
---|---
| EDA | Syn | BT
| Augm.Perc.
Subset Size | 0.05 | 0.1 | 0.2 | 0.05 | 0.1 | 0.2 | 0.05 | 0.1 | 0.2
500 | 0.92 | 0.92 | 0.91 | 0.94 | 0.94 | 0.94 | 0.92 | 0.92 | 0.92
1000 | 0.95 | 0.95 | 0.95 | 0.94 | 0.93 | 0.94 | 0.94 | 0.93 | 0.93
2000 | 0.93 | 0.94 | 0.93 | 0.93 | 0.93 | 0.93 | 0.94 | 0.94 | 0.93
5000 | 0.93 | 0.94 | 0.93 | 0.93 | 0.93 | 0.93 | 0.94 | 0.94 | 0.93
10000 | 0.94 | 0.94 | 0.93 | 0.94 | 0.94 | 0.94 | 0.93 | 0.93 | 0.93
Table 2: Baseline F1-scores for B2W dataset.

###  A.3 Mercado Libre Dataset

Table 3 shows all baseline F1-scores for Mercado Libre dataset.

| Augm. Type
---|---
| EDA | Syn | BT
| Augm.Perc.
Subset Size | 0.05 | 0.1 | 0.2 | 0.05 | 0.1 | 0.2 | 0.05 | 0.1 | 0.2
500 | 0.62 | 0.61 | 0.63 | 0.64 | 0.62 | 0.62 | 0.65 | 0.66 | 0.66
1000 | 0.73 | 0.73 | 0.74 | 0.69 | 0.69 | 0.69 | 0.71 | 0.72 | 0.71
2000 | 0.80 | 0.80 | 0.80 | 0.79 | 0.78 | 0.79 | 0.77 | 0.78 | 0.77
5000 | 0.85 | 0.85 | 0.85 | 0.85 | 0.84 | 0.85 | 0.84 | 0.84 | 0.84
10000 | 0.88 | 0.88 | 0.88 | 0.88 | 0.88 | 0.88 | 0.89 | 0.89 | 0.89
Table 3: Baseline F1-scores for Mercado Libre dataset.

##  Appendix B Detailed F1-score Gains

###  B.1 Tweets Dataset

Figure 8 shows all F1-score gains for Tweets dataset.

Figure 8: F1-score gains for each augmentation group on Tweets dataset.

###  B.2 B2W Dataset

Figure 9 shows all F1-score gains for B2W dataset.

Figure 9: F1-score gains for each augmentation group on B2W dataset.

###  B.3 Mercado Libre Dataset

Figure 10 shows all F1-score gains for Mercado Libre dataset.

Figure 10: F1-score gains for each augmentation group on Mercado Libre
dataset.

##  Appendix C Detailed $p$-values

###  C.1 Tweets Dataset

Table 4 shows all $p$-values for Tweets dataset.

| Augm. Type
---|---
| EDA | Syn | BT
| Augm. Perc.
Subset Size | 0.05 | 0.1 | 0.2 | 0.05 | 0.1 | 0.2 | 0.05 | 0.1 | 0.2
500 | 0.30 | 0.89 | 1.00 | 1.00 | 0.67 | 0.60 | 0.70 | 0.24 | 0.58
1000 | 0.81 | 0.61 | 0.62 | 0.74 | 0.68 | 0.87 | 0.93 | 1.00 | 0.86
2000 | 0.43 | 0.43 | 0.63 | 0.60 | 0.95 | 0.95 | 0.63 | 0.48 | 0.32
5000 | 0.53 | 0.97 | 0.24 | 0.54 | 0.78 | 0.67 | 0.55 | 0.81 | 0.50
10000 | 0.61 | 0.58 | 0.52 | 0.91 | 0.81 | 0.35 | 0.98 | 0.88 | 0.45
Table 4: McNemar’s test $p$-values for Tweets dataset.

###  C.2 B2W Dataset

Table 5 shows all $p$-values for B2W dataset.

| Augm. Type
---|---
| EDA | Syn | BT
| Augm. Perc.
Subset Size | 0.05 | 0.1 | 0.2 | 0.05 | 0.1 | 0.2 | 0.05 | 0.1 | 0.2
500 | 1.00 | 0.23 | 0.21 | 1.00 | 1.00 | 0.61 | 0.54 | 1.00 | 0.70
1000 | 0.09 | 0.73 | 0.21 | 1.00 | 0.61 | 0.32 | 0.74 | 0.30 | 0.47
2000 | 0.14 | 0.89 | 0.81 | 0.04 | 0.39 | 1.00 | 1.00 | 0.55 | 0.61
5000 | 0.92 | 0.78 | 0.35 | 0.92 | 1.00 | 0.93 | 1.00 | 0.48 | 0.79
10000 | 0.56 | 0.89 | 0.12 | 0.77 | 0.10 | 0.15 | 0.60 | 0.42 | 0.79
Table 5: McNemar’s test $p$-values for B2W dataset.

###  C.3 Mercado Libre Dataset

Table 6 shows all $p$-values for Mercado Libre dataset.

| Augm. Type
---|---
| EDA | Syn | BT
| Augm. Perc.
Subset Size | 0.05 | 0.1 | 0.2 | 0.05 | 0.1 | 0.2 | 0.05 | 0.1 | 0.2
500 | 0.93 | 0.93 | 0.80 | 1.00 | 0.93 | 0.50 | 0.60 | 0.44 | 0.81
1000 | 0.79 | 1.00 | 0.36 | 1.00 | 0.95 | 0.66 | 0.69 | 0.80 | 0.95
2000 | 0.52 | 0.55 | 0.54 | 0.75 | 0.67 | 0.56 | 0.67 | 0.67 | 0.96
5000 | 0.63 | 0.62 | 1.00 | 0.72 | 0.13 | 0.93 | 0.56 | 0.44 | 0.81
10000 | 0.51 | 0.66 | 0.23 | 0.89 | 0.72 | 0.60 | 0.23 | 0.38 | 0.21
Table 6: McNemar’s test $p$-values for Mercado Libre dataset.


11institutetext: Electrical and Computer Engineering, Northwestern University,
Evanston, U.S.A. 22institutetext: Department of Mathematics Science,
University of Liverpool, Liverpool, U.K. 33institutetext: School of Advanced
Technology, Xi’an Jiaotong-Liverpool University, Suzhou, China
33email<EMAIL_ADDRESS>33email<EMAIL_ADDRESS>
33email: {Chong.zhang19<EMAIL_ADDRESS>
33email: {Xiaobo.Jin<EMAIL_ADDRESS>111* Equal contribution.
🖂 Corresponding author.

# A Simple and Effective Baseline for Attentional Generative Adversarial
Networks

Mingyu Jin 1*1* Chong Zhang 3*3* Qinkai Yu 22 Haochen Xue 33 Xiaobo Jin🖂 33 Xi
Yang🖂 33

###### Abstract

Synthesising a text-to-image model of high-quality images by guiding the
generative model through the Text description is an innovative and challenging
task. In recent years, AttnGAN based on the Attention mechanism to guide GAN
training has been proposed, SD-GAN, which adopts a self-distillation technique
to improve the performance of the generator and the quality of image
generation, and Stack-GANs, which gradually improves the details and quality
of the image by stacking multiple generators and discriminators. However, this
series of improvements to GAN all have redundancy to a certain extent, which
affects the generation performance and complexity to a certain extent. We use
the popular simple and effective idea (1) to remove redundancy structure and
improve the backbone network of AttnGAN. (2) to integrate and reconstruct
multiple losses of DAMSM (Deep Attentional Multimodal Similarity Model). Our
improvements have significantly improved the model size and training
efficiency while ensuring that the model’s performance is unchanged and
finally proposed our SEAttnGAN. Code is available at
https://github.com/jmyissb/SEAttnGAN.

###### Keywords:

Text-to-image AttnGAN Simple and effective redundancy SEAttnGAN

##  1 Introduction

Text-to-image technology, at the intersection of computer vision and natural
language processing, converts textual descriptions into visual
representations. It finds applications in art creation, virtual scene
generation, advertising design, and game development etc. [11] [7] [12] [6]
[9]. Within the field of image generation, an influential framework is the
Generative Adversarial Network (GAN). By employing adversarial learning,
wherein the generator and discriminator are trained simultaneously, the
quality of the generated images progressively improves. GAN-based text-to-
image models have rapidly evolved since the introduction of Text descriptions
as supervised signals, demonstrated by GAN-INT-CLS [23]. Notable advancements
include the Adversarial What-Where Network (GAWWN) with its scene layout
estimator [24], the Pixel Recurrent Neural Networks (PixelCNN) that model
image pixel distributions using CNNs [30], and the Stacked Generative
Adversarial Networks (StackGAN and StackGAN-v2) that enhance image quality
through multi-stage models or multiple generators and discriminators [35][36].
Furthermore, the Semantics Disentangling GAN (SD-GAN) addresses latent space
factor confusion with a self-distillation mechanism [33].

To improve the Text-to-image model performance, Attentional Generative
Adversarial Network (AttnGAN) introduces the attention mechanism and the deep
attention multimodal similarity model in the generation work, which enables
AttnGAN to capture the word-level fine-grained information in the sentence.
Thus allowing it to synthesise the advantages of fine-grained details of
different subregions of an image by focusing on related words in the natural
language description. However, the disadvantages of AttnGAN are also apparent.
The AttnGAN contains multiple Attention models and multiple groups of baseline
networks, which makes its model structure complex and has many parameters,
which may lead to high computational complexity. To solve the problems,

  * •

We propse our optimized simplified version, called Simple and effective
Attentional Generative Adversarial Networks (SEAttnGAN ). Our work improves
the structure of AttnGAN and optimises the number of its parameters, and its
performance maintains the same level as before optimisation.

  * •

Additionally, to achieve the corresponding effect even after model
simplification and parameter reduction, we introduced semantics in the model.
Our model can use the comparative loss of semantics as auxiliary information
that can guide the GAN generator to more accurately generate images that match
the text description.

To solve the large and complex problem of the AttenGAN model, our work
proposes a simplified AttenGAN that performs similarly to the original
AttenGAN with significantly reduced structure and reduced parameter count.
Even in the CUB dataset generation test, we use one-tenth of the number of
parameters of AttnGAN (our SEAttnGAN, 26.37M; AttnGAN, 230M) surpasses the
image generation quality of AttnGAN, and the time consumption is much less
than AttnGAN, which powerfully achieves our motivation.

The overall paper is organized as follows: in Section 2, we introduce previous
related work on text-to-image task. Section 3 gives a detailed introduction to
our method and baseline; in Section 4, our algorithm will be compared with
other algorithms qualitatively and quantitatively. Subsequently, we summarize
our algorithm and possible future research directions.

##  2 Related Work

GANs [8] have the remarkable capability to generate realistic images that are
indistinguishable from real ones. Then Dong et al. proposed a method to
generate realistic images using only natural language written descriptions,
where they changed the structure of GAN by training a style encoder network to
invert the generator network [4].

While current methods can generate images with novel properties, they do not
preserve the text-independent details of the original image. To address this
issue, Nam et al. propose TAGAN (Text Adaptive Generative Adversarial
Networks) to generate semantically manipulated images while preserving text-
independent details [20]. The key idea of PPGN (Plug & Play Generative
Networks)[21] is to feed random noise vectors into generative models and
combine them with optimization techniques to guide the generative process.
LAPGAN (Laplacian Generative Adversarial Network)[2] is a generative model
framework combining Laplacian pyramid decomposition and GAN, aiming to
generate high-resolution images from coarse to fine in a multi-scale
progressive manner. MirrorGAN [22] achieves text-to-image conversion by
learning fine-grained correspondences, in which reverse information
propagation during generation can promote consistency between text and images,
and in the process of reverse generation through The discriminator provides an
additional supervisory signal. In text-to-image synthesis, conditional GANs
have been widely used, where a generator will take a text description as input
and generate an image [19] that matches the description. Recent studies have
shown that tuning GANs based on different factors such as object attributes,
styles, and layouts can improve the quality and diversity of generated images.

AttnGAN is a symbolic attention-based [32] that improves the quality and
relevance of generated images by focusing on important parts of textual
descriptions. These mechanisms have been used to guide generators to notice
connections between words and phrases and use them to create corresponding
images. A multimodal GAN [1] has been developed to generate multiple images
corresponding to a single textual description, and these diverse high-quality
images can provide different interpretations for the same textual description.
VAE(Variational Autoencoder) is a type of generative model that has been
explored for text-to-image synthesis [14]. In VAE-based text-to-image
synthesis, the model learns a latent space representation of the input text
descriptions, which can be used to generate images. One of the primary
advantages of using VAE-based models is their interpretability. But, there is
still room for improvement to generate photo-realistic images that match the
input textual description. PixelCNN was a generative model generates images
pixel by pixel, where each pixel is conditioned on the previous pixels [30].
The key idea behind PixelCNN is to use a convolutional neural network (CNN) to
model the distribution of the image pixels. PixelCNN has been shown to be
effective at generating high-quality images, it has also been used for text-
to-image synthesis, where the image is generated conditioned on an input text
description.

GAN-INT-CLS is the first application study that adds textual descriptions
(i.e., sentence embedding vectors) as supervised signals to image generation
[23]. The main goal of GAWWN is to estimate the scene layout of an image while
generating it [24]. GAWWN improves the text-to-image performance by
introducing Scene Layout Estimator in the image generation process [35].
StackGAN (Stacked Generative Adversarial Networks) is used to gradually
generate high-resolution images. This paper constructs a multi-stage
generative model, which aims to solve the challenge of traditional GAN models
in generating detail-rich images. Stackgan-v2 This paper improves and extends
StackGAN by adding multiple generator and discriminator networks [36],
introducing a super-resolution network, and introducing a triangular loss
function and a classification loss function. SD-GAN mainly solves the
confounding problem of the traditional GAN model in representing factors in
the latent space and introduces the mechanism of self-disentanglement learning
to train the generator and discriminator networks [33]. Training of the
process-aided generator for "self-distillation" by augmenting the Distiller.
SD-GAN mainly solves the confounding problem of the traditional GAN model in
representing factors in the latent space[18], and introduces the mechanism of
self-disentanglement learning to train the generator network and the
discriminator network [37]. Training of the process-aided generator for "self-
distillation" by augmenting Distiller.

##  3 Method

Figure 1: Overview architecture of AttnGAN

###  3.1 Background: AttnGAN

AttnGAN [32] achieves fine-grained text-to-image generation through attention-
driven multi-stage refinement, which focuses on related words to synthesize
some details in different regions of the image.

The AttnGAN model consists of $m$ generators ($G_{0},G_{1},...,G_{m-1}$), each
generator will generate an image as follows

$\displaystyle H_{0}$ $\displaystyle=$ $\displaystyle F_{0}(z,F^{ca}(T))$ (1)
$\displaystyle h_{i}$ $\displaystyle=$ $\displaystyle
F_{i}(h_{i-1},F^{attn}(T,H_{i-1}))$ (2) $\displaystyle\hat{x}_{i}$
$\displaystyle=$ $\displaystyle G_{i}(h_{i})$ (3)

where $i=1,2,\cdots,m-1$, $z$ is a noise variable following a Gaussian
distribution, $T$ is a matrix of word vector representations, and $F^{ca}$
uses conditional augmentation to encode each word vector in $T$. The attention
network $F^{attn}$ will generate a weighted representation of the word vector
according to the relationship between the image vectors and the text vectors.

The attn network $F^{attn}$ takes word features and image features as input:
1) maps word features to image feature space; 2) generates a new
representation of word features according to the correlation between word
features and image features, which is a linear weighting of image features
combination. The specific description is as follows $\hat{T}$ = $UT$

$\alpha_{i,j}=\frac{\exp(\hat{t}_{i}^{T}h_{j})}{\sum_{k=1}^{m}\exp(\hat{t}_{i}^{T}h_{k})},\quad
c_{i}=\sum_{j=1}^{m}\alpha_{ij}h_{j},$ (4)

where $U$ is the transformation matrix, which transforms word vectors $t_{i}$
into the semantic space of image features $h_{i}$.

AttnGAN model jointly optimizes generative adversarial loss and deep attention
multimodal similarity loss. The generative adversarial loss contains
conditional distribution and unconditional distribution likelihood function
loss, where the unconditional distribution likelihood loss determines whether
an image is real or fake, and the conditional distribution likelihood loss
determines whether an image and a sentence match. DAMSM learns two neural
networks to map image sub-regions and sentence words into a common semantic
space respectively to compute fine-grained loss for image generation.

###  3.2 Our Proposed Method

In AttnGAN, multiple generators generate images of increasing scale and use an
attention mechanism for each generator. In order to reduce the complexity of
the model, we replace multiple generators with upsampling modules to output
ever-increasing feature maps, and use attention only on feature maps at one
scale, and to align with word embedding vectors, we insert multiple
downsampling modules, as shown below in Fig. 2.

In this paper, we simplified the network structure of AttnGAN. For the input
of the generator, we mix sentence features with noise. After data
augmentation, the input is first reshaped through a fully connected layer.
Then apply a series of Up Block layers to extract image features. The Up-Block
layer is composed of an upsample layer, a redisual block, and DF-Blocks to
fuse the text and image feature during the generation process. The image
features obtained are used as input and enter the attention layer together
with the text features. Finally, the image features are re-extracted by a
layer of Up-Block and converted to image features by a convolution layer. The
detail architecrure shown in Figure 2

Figure 2: Architecture of SEAttnGAN. FC: fully connected layer. UPBlock:
residual block + DFBlock. DownBlock: downsample + residual block

We simplified the existing DFblock structure and used a new DFblock structure
consisting of a two-layer linear mlp structure and an additional hidden layer
(with ReLU activation) to further extract sentence features, map sentence
features from 256 dimensional space to 64 dimensional space, and obtain new
representation features. The new structure has the characteristic of
lightweight, reducing the number of model parameters, simplifying the
calculation process, and accelerating training speed. The detail architecrure
shown in Figure3

Figure 3: (1)The architecture of the DFblock: 2 layers linear mlp, tensor from
the previous layers. (2)The architecrure of our model’s loss function

The discriminator converts images into features through a series of DownBlock.
Then the sentence vector will be replicated and concatenated with image
features. The discriminator distinguishing generated images from real samples,
the discriminator promotes the synthesize images with higher quality and text-
image semantic consistency. And the word feature is used at word level to
compute loss by measures the similarity with image-text.
The text encoder is a bi-directional Long Short-Term Memory(LSTM). and we
directly use the pre-trained model provided by AttnGAN.
The image encoder is a Convolutional Neural Network(CNN) that maps images to
semantic vectors. we directly use the Inception-v3 model pretrained on
ImageNet. To gnerate realistic images with multiple levels(i.e.,sentence level
and word level) of conditions,the loss function of our network is defined as:

$\mathcal{L}=\mathcal{L}_{G}+\gamma\mathcal{L}_{D}$ (5)

where

$\displaystyle\mathcal{L}_{G}=-E_{G(z)}\sim\ p_{g}[D(G(z)),t]$ (6)
$\displaystyle\mathcal{L}_{D}=\lambda\mathcal{L}^{s}+(1-\lambda)\mathcal{L}^{w}.$

Here $\gamma$ and $\lambda$ is hyperparameter to balance the terms of
equation. Here we defined those two loss function, respectively.

$\displaystyle\mathcal{L}^{s}=$ $\displaystyle-E_{x\sim
P_{r}}[\min(0,-1+D(x,\underline{t}))]$ (7) $\displaystyle-(1/2)E_{G(z)\sim
P_{g}}[\min(0,-1-D(G(z),\underline{t}))]$ $\displaystyle-(1/2)E_{x\sim
P_{mis}}[\min(0,-1-D(x,\underline{t}))]$ $\displaystyle+kE_{x\sim
P_{r}}\left[\left(\left\|\nabla_{x}D(x,\underline{t})\right\|+\left\|\nabla_{t}D(x,\underline{t})\right\|\right)^{p}\right]$

where $z$ is the noise vector; $\bar{e}$ is the sentence vector;
$P_{g},P_{r},P_{mis}$ represent the synthetic data distribution, real data
distribution, and mismatching data distribution, respectively.

$\displaystyle\mathcal{L}^{w}=-\sum^{M}_{i=1}log\frac{exp(\mu\cdot
sim(c_{i},t_{i}))}{\sum^{M}_{j=1}exp(\mu\cdot sim(c_{i},t_{j}))}$ (8)

where

$c_{i}=\sum^{n-1}_{j=0}\alpha_{j}x_{j},\quad\alpha_{j}=\frac{exp(\mu_{1}s_{i,j})}{\sum^{n-1}_{k=0}exp(\mu_{1}s_{i,k})}$
(9)

$s$ is a matrix product of entire image feature and word feature defined by
$t^{T}x$. $\mu$ and $\mu_{1}$ is a smoothing factor determined by experiments.
In this batch of sentences, only $t_{i}$ matches the image $x_{i}$, and treat
all other $M-1$ sentences as mismatching descriptions. $sim(c_{i},t_{i})$ is
the cosine similarity between $c_{i}$ and $t_{i}$ which equal to
$\frac{c_{i}^{T}t_{i}}{\mid\mid c_{i}\mid\mid\mid\mid t_{i}\mid\mid}$. Based
on experiments on validation set, we set the hyperparameters in this section
as: $\gamma=5$, $\lambda=0.2$, $\mu=5$, $\mu_{1}=10$. The detail architecrure
shown in figure 3

##  4 Experiment

Datasets. We selected two datasets: CUB Bird [31] and COCO [16]. The CUB
dataset contains 11,788 bird images of 200 species, each with descriptions in
10 languages. Meanwhile, the COCO dataset has 80,000 images for training and
40,000 images for testing, each with descriptions in five languages.

Experiment settings. Both our SEAttnGAN generator and discriminator use the
Adam optimizer. The generator uses the parameters $\beta_{1}=0$,
$\beta_{2}=0.9$ and the learning rate is $0.0001$ [13]; while the
discriminator uses the parameters $\beta_{1}=0.0$, $\beta_{2}=0.9$, the
learning rate is $0.0004$.

Evaluation metrics. Finally, we choose IS (Inception Score) and FID (Fréchet
Inception Distance) to evaluate the quality of the generated image. IS
measures [27] the KL divergence between the generated image category
distribution obtained by a pre-trained Inception V3 model and the real image
category to represent the authenticity of the generated image. The smaller the
KL divergence, the closer the category distribution of the generated image is
to the real image. Higher IS values indicate better diversity and authenticity
of generated images. It’s not perfect though, as it may not capture the
quality of detail and realism of the resulting image. Therefore, when
evaluating image generation models, it is usually necessary to combine other
indicators and human subjective evaluation for comprehensive consideration.

Frechet Inception Distance (FID) [10] is a metric used to measure the quality
of image generation. It is based on the statistical feature difference between
the generated image and the real image: it first uses a pre-trained image
classifier (such as the Inception V3 model) to extract the feature
representation of the real image and the generated image, and then uses the
Fréchet distance to measure the difference between the two feature
distributions. The difference between the sample mean and the covariance
matrix of. The lower the value of FID, the better the quality of the generated
image, and the higher the coincidence with the real image distribution. The
advantage of FID over other metrics is that it can capture more subtle
differences in image quality and has a high correlation with human subjective
evaluation results. However, FID also has some limitations, such as requiring
a pre-trained classifier and a large number of image samples.

###  4.1 Quantitative Comparison

In this section, we compare our new method with baseline methods and some
other methods, here we report quantitative metrics such as IS and FID on CUB
Bird[31] and COCO datasets[16]. In particular, we also compare the memory and
time consumed by these methods during inference. During model training, some
models use additional pre-training models: SD-GAN[33] uses third-party trained
COCO pre-training models[25], CPGAN[15] uses additional pre-training YOLO
v3[5], XMC-GAN[34] use additional pre-trained VGG-19[28] and Bert[3], DAE-GAN
use additional NLTK POS markers and design manually Rules and TIME use an
additional two-dimensional positional encoding.

Table 1: The results of FID, IS and NOP comparison on dataset CUB bird and COCO. Model | CUB | COCO | Cost
---|---|---|---
| IS $\uparrow$ | FID $\downarrow$ | FID $\downarrow$ | Memory $\downarrow$ | Inference Time $\downarrow$
StackGAN[35] | 3.70 | - | - | - | -
StackGAN++[36] | 3.84 | - | - | - | -
MirrorGAN[22] | 4.56 | 18.34 | 34.71 | - | -
SD-GAN[33] | 4.67 | - | - | 335M | 6.18s
DM-GAN[37] | 4.75 | 16.09 | 32.64 | 46M | 7.06s
CPGAN[15] | - | - | 55.80 | 318M | -
TIME[17] | 4.91 | 14.30 | 31.14 | 120M | -
XMC-GAN[34] | - | - | 9.30 | 166M | -
DF-GAN[29] | 4.46 | 18.23 | 41.83 | 46.79M | 3.71s
DAE-GAN[26] | 4.42 | 15.19 | 28.12 | 98M | -
AttnGAN [32] | 4.36 | 23.98 | 35.49 | 230M | 4.05s
SEAttnGAN(ours) | 4.32 | 15.03 | 34.78 | 26.37M | 0.11s

As shown in the results in Table 1, our method achieves comparable performance
to other methods on IS and FID metrics for both datasets. In particular, our
method achieves better performance compared to the baseline model AttnGAN, for
example, on CUB, our method achieves a FID value of 15.03, which is better
than AttnGAN’s 23.98. In terms of inference time, our method runs about 45
times faster than the baseline method (from 4.05 seconds to 0.11 seconds).
Considering the memory footprint of the model, our method is much lower than
other methods. Compared with the baseline method AttnGAN, it reduces the
memory footprint to about 1/9 (26.37M vs. 230M). Our SEAttnGAN did not achieve
an absolute advantage in COCO, which may be due to the difficulty of training
generative models in a short period of time due to the large variety of items
in the COCO dataset. Comparing the FID values on the two data sets of CUB and
COCO, we also found that the image generation task on COCO is more difficult
than that of CUB. DF-GAN[29] using the same simple and effective ideas also
significantly reduce model size. Still, our SEAttnGAN outperforms DF-GAN in
generating image quality (IS, FID) and generation time. Besides, compare with
the original loss in DF-GAN, our new loss saw an outstanding improvement. The
detail is showed in figure 2

Table 2: The results of FID, IS comparison on dataset CUB bird Model | CUB
---|---
| IS $\uparrow$ | FID $\downarrow$
SEAttnGAN with original loss | 2.88 | 34.71
SEAttnGAN with new loss | 4.29 | 15.64

###  4.2 Qualitative Comparison

Figure 4: The architecture of the SEAttnGAN. FC: fully connected layer.
UPBlock: upsample + residual block + DFBlock. DownBlock: downsample + residual
block

We compare the visualization of the model with three different models in Fig.
4: AttnGAN [32], DF-GAN and DM-GAN [37]. As can be seen from Fig. 4, the
results obtained from AttnGAN and DF-GAN are not as realistic as those
obtained from DM-GAN and our model. The difference between the bird and the
background is less obvious, especially in the leg and tail areas. Also, the
feathers on the bird’s back lack a certain clarity. The image generated by DM-
GAN also loses part of the image. In contrast, the images generated by our
model exhibit a striking feature: a high degree of discrimination between the
bird and the background. The pattern of the bird in our model is easy to
discern. In addition, the background in the images of our model has stronger
brightness than those generated by the other three models.

Finally, we also observe from Fig. 4 that the images in the four columns on
the left are distorted on all three models on the COCO dataset, which means
that some images are missing, duplicated, or folded. However, compared to
images generated by the other three models, the background in our model’s
images shows stronger brightness, adding to the visual appeal and overall
aesthetics of the output.

##  5 Conclusion

Based on the results above, it appears that AttnGAN utilizes two attention
models during the fine-grained image generation process, which seems
unnecessary. However, after implementing a simple and effective target
improvement, the model’s size was significantly reduced without any
significant decrease in performance indicators like FID and IS when compared
to the original AttnGAN version. Even in the CUB bird’s FID item, we beat our
improved object AttnGAN with one-tenth of the model size, which is an exciting
result. Our improvement involved eliminating one set of attention models and
adding the upsample and residual block, resulting in better image quality.
Additionally, our combination of loss functions for the three stages of
SEAttnGAN has proven to be effective.

##  6 Future work

The limitation of our model mainly reflects on the low performance of the COCO
dataset. It occasionally generates defective images, meaning it needs to run
the model several more times to generate the correct image. Future work will
focus on taking methods to overcome these shortcomings. Initially, the number
of parameters in our model will be minimised to render our model simple and
effective. We anticipate improved performance and more reliable image
generation by reducing the complexity. our attention will be directed towards
enhancing the Attention-GAN model itself. We will explore and implement novel
techniques to refine and optimise its architecture, enhancing its ability to
generate high-quality images. Eventually, the Simple and effective philosophy
can be extended to most models that are large but widely used by industry.

##  7 Authorship Statement

All authors contributed to the study’s conception and design. Mingyu Jin and
Chong Zhang completed the construction and debugging of the basic network
framework of SEAttnGAN. Simple and effective improvements to AttnGAN ideas and
improvements to the Attention model were completed by Chong Zhang. Loss
function reconstruction and SEAttnGAN Mathematical interpretation by Qinkai
Yu. Optimising the Attention Model combined with semantic information was done
by Haochen Xue. All authors wrote the manuscript, and all authors commented on
previous versions. Xiaobo Jin supervised this work and made a comprehensive
revision and reconstruction of the work. All authors read and approved the
final manuscript.

## References

  * [1] Javad Amirian, Jean-Bernard Hayet, and Julien Pettré.  Social ways: Learning multi-modal distributions of pedestrian trajectories with gans.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, pages 0–0, 2019.
  * [2] Emily L Denton, Soumith Chintala, Rob Fergus, et al.  Deep generative image models using a laplacian pyramid of adversarial networks.  Advances in neural information processing systems, 28, 2015.
  * [3] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.  Bert: Pre-training of deep bidirectional transformers for language understanding.  arXiv preprint arXiv:1810.04805, 2018.
  * [4] Hao Dong, Simiao Yu, Chao Wu, and Yike Guo.  Semantic image synthesis via adversarial learning.  In Proceedings of the IEEE international conference on computer vision, pages 5706–5714, 2017.
  * [5] Ali Farhadi and Joseph Redmon.  Yolov3: An incremental improvement.  In Computer vision and pattern recognition, volume 1804, pages 1–6. Springer Berlin/Heidelberg, Germany, 2018.
  * [6] Stanislav Frolov, Tobias Hinz, Federico Raue, Jörn Hees, and Andreas Dengel.  Adversarial text-to-image synthesis: A review.  Neural Networks, 144:187–209, 2021.
  * [7] Leon A Gatys, Alexander S Ecker, and Matthias Bethge.  A neural algorithm of artistic style.  arXiv preprint arXiv:1508.06576, 2015.
  * [8] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio.  Generative adversarial networks.  Communications of the ACM, 63(11):139–144, 2020.
  * [9] Andreas Hald, Jens Struckmann Hansen, Jeppe Kristensen, and Paolo Burelli.  Procedural content generation of puzzle games using conditional generative adversarial networks.  In Proceedings of the 15th International Conference on the Foundations of Digital Games, pages 1–9, 2020.
  * [10] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter.  Gans trained by a two time-scale update rule converge to a local nash equilibrium.  Advances in neural information processing systems, 30, 2017.
  * [11] Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, and Alexei A Efros.  Image-to-image translation with conditional adversarial networks.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1125–1134, 2017.
  * [12] Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen.  Progressive growing of gans for improved quality, stability, and variation.  arXiv preprint arXiv:1710.10196, 2017.
  * [13] Diederik P Kingma and Jimmy Ba.  Adam: A method for stochastic optimization.  arXiv preprint arXiv:1412.6980, 2014.
  * [14] Diederik P Kingma and Max Welling.  Auto-encoding variational bayes.  arXiv preprint arXiv:1312.6114, 2013.
  * [15] Jiadong Liang, Wenjie Pei, and Feng Lu.  Cpgan: Content-parsing generative adversarial networks for text-to-image synthesis.  In Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part IV 16, pages 491–508. Springer, 2020.
  * [16] Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick.  Microsoft coco: Common objects in context.  In Computer Vision–ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13, pages 740–755. Springer, 2014.
  * [17] Bingchen Liu, Kunpeng Song, Yizhe Zhu, Gerard de Melo, and Ahmed Elgammal.  TIME: text and image mutual-translation adversarial networks.  In Thirty-Fifth AAAI Conference on Artificial Intelligence, AAAI 2021, Thirty-Third Conference on Innovative Applications of Artificial Intelligence, IAAI 2021, The Eleventh Symposium on Educational Advances in Artificial Intelligence, EAAI 2021, Virtual Event, February 2-9, 2021, pages 2082–2090. AAAI Press, 2021.
  * [18] Jie Ma, Libao Zhang, and Jue Zhang.  Sd-gan: Saliency-discriminated gan for remote sensing image superresolution.  IEEE Geoscience and Remote Sensing Letters, 17(11):1973–1977, 2019\.
  * [19] Mehdi Mirza and Simon Osindero.  Conditional generative adversarial nets.  arXiv preprint arXiv:1411.1784, 2014.
  * [20] Seonghyeon Nam, Yunji Kim, and Seon Joo Kim.  Text-adaptive generative adversarial networks: manipulating images with natural language.  Advances in neural information processing systems, 31, 2018.
  * [21] Anh Nguyen, Jeff Clune, Yoshua Bengio, Alexey Dosovitskiy, and Jason Yosinski.  Plug & play generative networks: Conditional iterative generation of images in latent space.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4467–4477, 2017.
  * [22] Tingting Qiao, Jing Zhang, Duanqing Xu, and Dacheng Tao.  Mirrorgan: Learning text-to-image generation by redescription.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1505–1514, 2019.
  * [23] Scott Reed, Zeynep Akata, Xinchen Yan, Lajanugen Logeswaran, Bernt Schiele, and Honglak Lee.  Generative adversarial text to image synthesis.  In International conference on machine learning, pages 1060–1069. PMLR, 2016.
  * [24] Scott E Reed, Zeynep Akata, Santosh Mohan, Samuel Tenka, Bernt Schiele, and Honglak Lee.  Learning what and where to draw.  Advances in neural information processing systems, 29, 2016.
  * [25] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer.  High-resolution image synthesis with latent diffusion models, 2021.
  * [26] Shulan Ruan, Yong Zhang, Kun Zhang, Yanbo Fan, Fan Tang, Qi Liu, and Enhong Chen.  Dae-gan: Dynamic aspect-aware gan for text-to-image synthesis.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 13960–13969, 2021.
  * [27] Tim Salimans, Ian Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen.  Improved techniques for training gans.  Advances in neural information processing systems, 29, 2016.
  * [28] Karen Simonyan and Andrew Zisserman.  Very deep convolutional networks for large-scale image recognition.  arXiv preprint arXiv:1409.1556, 2014.
  * [29] Ming Tao, Hao Tang, Songsong Wu, Nicu Sebe, Xiao-Yuan Jing, Fei Wu, and Bingkun Bao.  Df-gan: Deep fusion generative adversarial networks for text-to-image synthesis.  arXiv preprint arXiv:2008.05865, 2020.
  * [30] Aaron Van den Oord, Nal Kalchbrenner, Lasse Espeholt, Oriol Vinyals, Alex Graves, et al.  Conditional image generation with pixelcnn decoders.  Advances in neural information processing systems, 29, 2016.
  * [31] C. Wah, S. Branson, P. Welinder, P. Perona, and S. Belongie.  Cub.  Technical Report CNS-TR-2011-001, California Institute of Technology, 2011\.
  * [32] Tao Xu, Pengchuan Zhang, Qiuyuan Huang, Han Zhang, Zhe Gan, Xiaolei Huang, and Xiaodong He.  Attngan: Fine-grained text to image generation with attentional generative adversarial networks.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1316–1324, 2018.
  * [33] Guojun Yin, Bin Liu, Lu Sheng, Nenghai Yu, Xiaogang Wang, and Jing Shao.  Semantics disentangling for text-to-image generation.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 2327–2336, 2019.
  * [34] Han Zhang, Jing Yu Koh, Jason Baldridge, Honglak Lee, and Yinfei Yang.  Cross-modal contrastive learning for text-to-image generation.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 833–842, 2021.
  * [35] Han Zhang, Tao Xu, Hongsheng Li, Shaoting Zhang, Xiaogang Wang, Xiaolei Huang, and Dimitris N Metaxas.  Stackgan: Text to photo-realistic image synthesis with stacked generative adversarial networks.  In Proceedings of the IEEE international conference on computer vision, pages 5907–5915, 2017.
  * [36] Han Zhang, Tao Xu, Hongsheng Li, Shaoting Zhang, Xiaogang Wang, Xiaolei Huang, and Dimitris N Metaxas.  Stackgan++: Realistic image synthesis with stacked generative adversarial networks.  IEEE transactions on pattern analysis and machine intelligence, 41(8):1947–1962, 2018.
  * [37] Minfeng Zhu, Pingbo Pan, Wei Chen, and Yi Yang.  Dm-gan: Dynamic memory generative adversarial networks for text-to-image synthesis.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 5802–5810, 2019.