# Synthesizing Forestry Images Conditioned on Plant Phenotype Using a
Generative Adversarial Network

Debasmita Pal and Arun Ross

###### Abstract

Plant phenology and phenotype prediction using remote sensing data is
increasingly gaining the attention of the plant science community to improve
agricultural productivity. This work aims to generate synthetic forestry
images that satisfy certain phenotypic attributes, viz. canopy greenness. We
harness a Generative Adversarial Network (GAN) to synthesize biologically
plausible and phenotypically stable forestry images conditioned on the
greenness of vegetation (a continuous attribute) over a specific region of
interest (describing a particular vegetation type in a mixed forest). The
training data is based on the automated digital camera imagery provided by the
National Ecological Observatory Network (NEON) and processed by the PhenoCam
Network. Our method helps render the appearance of forest sites specific to a
greenness value. The synthetic images are utilized to predict another
phenotypic attribute, viz., redness of plants. The Structural SIMilarity
(SSIM) index is used to assess the quality of the synthetic images. The
greenness and redness indices of the generated synthetic images are compared
against that of the original images using Root Mean Squared Percentage Error
(RMSPE) to evaluate their accuracy and integrity. The generalizability and
scalability of our proposed GAN model is determined by effectively
transforming it to generate synthetic images for other forest sites and
vegetation types.

###### keywords:

Generative Adversarial Network (GAN) , synthetic forestry imagery , plant
phenology prediction , plant phenotype , canopy greenness (GCC) , redness of
plants (RCC)

††journal: Pattern Recognition

organization=Department of Computer Science and Engineering, Michigan State
University,country=USA

A Generative Adversarial Network (GAN) architecture is proposed to generate
synthetic images satisfying a continuous attribute over a specific region of
interest

The GAN model synthesizes forestry images conditioned on canopy greenness
describing a specific vegetation type in a mixed forest

The synthetic images are also utilized to predict redness of vegetation

The generalizability and scalability of the proposed GAN model are evaluated
by extending it to different forest sites and vegetation types

##  1 Introduction

Phenology is the study of recurring and seasonal biological life cycle events
of organisms, primarily driven by complex interactions between environmental
and genetic factors [1]. This can be utilized in optimizing crop production,
better understanding ecosystem processes like carbon and hydrology cycle,
invasive species and pests management, predicting human health related
problems (e.g., seasonal allergies), etc.111https://www.usanpn.org/about/why-
phenology Flag leaf emergence, flowering of plants, insect emergence, animal
migration are examples of phenological events in nature. These phenomena are
highly sensitive to weather and climate change, specifically to temperature
and precipitation. Due to gradual change in the global climate, plant
phenology and phenotype (the observable traits and characteristics resulted
from the interactions between genotypes and
environment)222https://www.genome.gov/genetics-glossary/Phenotype prediction
occupies a prominent place in the domain of agriculture [1, 2]. It advances
the study of phenological trends and reduces the uncertainties associated with
ecosystem processes (e.g., carbon cycle) as a consequence of phenological
shifts. With recent technological advancements, this area of research is
significantly growing due to the availability of remotely sensed near-surface
plant phenological observations through satellite and digital camera imagery
in lieu of manual measurements. Consequently, image analysis and pattern
recognition have been playing an important role in precision agriculture [3,
4]. Specifically, the adoption of deep learning and computer vision techniques
in plant research has enabled scientists to learn the representation and
regularities in high volume of data in order to increase plant productivity
[5, 6].

Over the past few years, various deep generative models (viz., energy-based
models, variational autoencoders, generative adversarial networks, normalizing
flows) [7] have been proposed in the literature to model the distribution of
input training patterns and generate new samples. Among these, Generative
Adversarial Networks (GANs) have been found to be immensely successful in
generating high-quality synthetic images from an existing distribution of
sample real images. The purpose of this work is to put forward a GAN
architecture for generating realistic-looking synthetic forestry images
conditioned on certain phenotypic attributes. We use the greenness of
vegetation canopy as a condition in generating synthetic images. Canopy
greenness measurements provide information about the foliage present and its
colors.333https://phenocam.nau.edu/webcam/about/ Tracking canopy greenness is
instrumental in the comprehensive understanding of the sources and sinks of
carbon to reduce uncertainties in global carbon cycle. Further, (a) the leaf
emergence increasing the greenness impacts the hydrologic processes by
evapotranspiration; (b) the senescence in autumn, during which the leaves
color switches from green to yellow and/or red [8] influences nutrient cycling
process by adding nutrients to the forest floor; (c) the amount and the
condition of the foliage present affects the surface energy balance.

We hypothesize that the generated synthetic images conditioned on the
greenness of vegetation canopy would help render the appearance of the forest
sites specific to a given greenness value. The synthetic images could be
utilized to predict other phenotypic attributes such as redness of plants,
leaf area index (LAI), canopy cover, etc. Studies have shown that redness of
plants is a better predictor of GPP-based (Gross Primary Productivity) start
and end of growing season in some of the vegetation forest sites [9]. LAI
quantifies the canopy development and is critical in photosynthesis process.
The overarching goal of this work is to introduce a forestry image generation
model that learns the intricate patterns of phenotypic attributes. Modeling
these patterns gives us control to simulate the environment with different
plant phenotypes and weather parameters.

###  1.1 Background

Our study is based on the RGB forestry imagery along with the derived
greenness index curated by the PhenoCam
Network.444https://phenocam.sr.unh.edu/webcam/ The images are captured by
automated, near-surface, remote sensing digital camera placed on the top of
the canopy in 30-minute time intervals throughout the year [10]. Each pixel in
an RGB image is represented by a triplet of digital numbers denoting the
intensity of red, green, and blue color channels. These images are processed
by the PhenoCam Network to gather statistics about the greenness of vegetation
canopy, measured by Green Chromatic Coordinate (GCC). GCC is the relative
brightness of the green channel, normalized against the overall brightness of
red, green, and blue channels together. Additionally, the PhenoCam Network
report the redness index of plants, measured by Red Chromatic Coordinate
(RCC). It is defined as the relative brightness of the red channel, normalized
against the overall brightness of red, green, and blue channels together.
$GCC=\frac{G_{DN}}{R_{DN}+G_{DN}+B_{DN}}$ (1)
$RCC=\frac{R_{DN}}{R_{DN}+G_{DN}+B_{DN}}$ (2)

In general, the greenness and redness index are measured over a specific
Region of Interest (ROI) on the image to describe a particular vegetation type
in a mixed forest, such as Deciduous Broadleaf (DB), Evergreen Needleleaf
(EN), Grassland (GR). The PhenoCam Network defines certain ROIs for each of
the forest sites to measure the greenness and redness statistics, and each of
these ROIs is designated by an ROI ID (e.g., DB_1000, EN_1000 etc.). The first
two letters of the ROI ID indicate the vegetation type and the last four
digits serve as a unique identifier to distinguish between multiple ROIs of
same vegetation type at a given site. The GCC and RCC corresponding to the ROI
of an image are calculated by taking the mean of GCC and RCC respectively, of
the pixels in that ROI.

We intend to use Type-I PhenoCam sites because of their high quality of the
captured images. The National Ecological Observatory Network
(NEON)555https://www.neonscience.org/ is one of the participating Type-I sites
capturing the images of plant canopy across United States following the
protocols defined by the PhenoCam Network (NEON Data Product DP1.00033 [11]).
They have strategically formulated $20$ ecoclimatic “Domains” grounded on the
vegetation, landforms and ecosystem dynamics, involving $47$ terrestrial field
sites and $34$ aquatic freshwater sites. In our experiment, we consider the
terrestrial sites belonging to the NEON domain “D01-Northeast”, which
encompasses New England and north-eastern Seaboard states along with the
northern end of the Appalachian range. This domain includes the following
terrestrial sites:

  * 1.

Harvard Forest, Massachusetts, USA

    * (a)

NEON Site ID: HARV

    * (b)

PhenoCam Site ID: NEON.D01.HARV.DP1.00033

    * (c)

Latitude: 42.53691; Longitude: -72.17265

  * 2.

Bartlett Experimental Forest, New Hampshire, USA

    * (a)

NEON Site ID: BART

    * (b)

PhenoCam Site ID: NEON.D01.BART.DP1.00033

    * (c)

Latitude: 44.063889; Longitude: -71.287375

Figure 1 shows some sample mid-day images corresponding to different times in
a year for both sites along with the GCC and RCC values of two ROIs describing
the vegetation types DB and EN. It can be observed that the greenness of
plants varies throughout the year in such a way that it is low in winter,
increases sharply in spring, and gradually falls over the summer. In Fall, the
redness starts to increase.

(a) Harvard Forest

(b) Bartlett Experimental Forest

Figure 1: Sample mid-day images of NEON terrestrial sites.

###  1.2 Objective and Contribution

The objective of this paper is to utilize the images of the above-mentioned
NEON forest sites along with derived GCC values for a specific ROI to train a
generative model, which synthesizes new examples of realistic-looking forestry
images satisfying the given GCC value over the given ROI. We exploit the
concept of Conditional GAN (CGAN) [12] to generate forestry images conditioned
on the continuous attribute GCC and the ROI image rather than conditioning on
any categorical attribute. The quality of the synthetic images is evaluated
with the help of Structural SIMilarity (SSIM) index [13], and the accuracy of
GCC in the generated images is measured in terms of Root Mean Squared
Percentage Error (RMSPE) [14]. Further, the synthetic images are utilized to
predict another phenotypic attribute, viz., RCC, reported by the PhenoCam
network, which is not used to train the model. The predicted RCC values of the
synthetic images are compared with the ground-truth RCC of the test images and
the RMSPE is calculated. Experimental results indicate that the RMSPE of the
generated images is 2.1% (GCC) and 9.5% (RCC) for Harvard forest, and 2.1%
(GCC) and 8.4% (RCC) for Bartlett Experimental Forest respectively when the
GAN model is individually trained on each site.

In order to deduce the efficacy of our proposed approach of predicting other
phenotypic attribute from the synthetic images, we study the correlation
between GCC and RCC reported by the PhenoCam Network. A negative linear
correlation is observed between these two phenotypic attributes with magnitude
as approximately 0.2 for both of the forest sites, which indicates that the
redness index is not highly correlated with greenness index; therefore, the
prediction of redness index from the synthetic images generated by our GAN
model is not directly controlled by the input greenness index to the model.
The GAN model itself plays a significant role in identifying patterns and
predicting redness index. Additionally, we attempt to verify if the model
trained on one forest site can be effectively transformed to generate the
images for other forest sites using less computational resource and time,
referred to as cross-site training. The scalability of our proposed approach
is assessed by extending the model to other vegetation type of the same forest
site.

GANs have been utilized to synthesize various kinds of images in the
literature either by imposing a condition (class labels) to control the images
being generated or in an unconditional setting. Some examples of GAN-generated
images are shown in Figure 2. Most of these applications synthesize objects
having well-defined morphological structures. Researchers have also utilized
GAN for image-to-image translation (transferring images from a source domain
to a target domain), where the image itself is used as a condition [15]. These
applications include translating a wide range of images, e.g., aerial to map,
summer to winter, day to night, horse to zebra, etc. [16, 17]. Studies have
also been performed to train GANs conditioned on a continuous space using
various kinds of datasets such as circular 2D Gaussians [18, 19], engineering
designs [20], RC-49 (3D chairs with different yaw angles), UTKFace (human
faces labeled by age), Cell-200 (fluorescence microscopy images with cell
populations), steering angles [21].

(a) MNIST handwritten digits [22]

(b) ImageNet [23]

(c) CelebA-HQ face images [24]

(d) Dining rooms based on LSUN dataset [25]

(e) Plant leaves [26]

(f) Oxford-102 flower dataset [27]

Figure 2: Examples of GAN-generated images available in literature based on
various datasets.

To the best of our knowledge, our work is the first attempt to generate forest
landscapes satisfying a phenotypic attribute, which is continuous in nature,
over a certain portion of the image. Consequently, there are two auxiliary
information (GCC and ROI) to be provided as inputs to our GAN model, whereas
existing methods do not account for this specific scenario. As the overall
geometry of the forestry images for a particular site always remains the same,
our GAN model is primarily required to learn the color of the foliage, i.e.,
green-up and green-down based on the GCC value. It is always challenging to
extract meaningful phenological information from automated plant imagery due
to lighting variations, plant rotations and occlusion [4]. Here, we focus on
generating synthetic forestry imagery based on the greenness over ROI, which
are visually appealing as well as phenotypically stable.

In a nutshell, the contributions of this work are as follows:

  * 1.

Developing a GAN architecture conditioned on a continuous attribute over a
certain portion on the image.

  * 2.

Application of GAN in the domain of agriculture by generating synthetic
forestry images satisfying a given phenotypic attribute over the ROI
describing a vegetation type.

  * 3.

Synthesizing biologically plausible and phenotypically stable images to better
portray the appearance of the forest sites.

  * 4.

Prediction of other phenotypic attributes that were not used in generation
process, from the synthetic images.

In Section 2, we provide with a brief literature review on different GAN
frameworks and the application of deep learning in agriculture. Section 3
describes the proposed approach followed by the GAN architecture developed in
this work. The experiments and the results are reported in Section 4. Section
5 concludes the paper. Our code is available on
GitHub666https://github.com/DebasmitaPal1206/synthetic_forestry_image_using_GAN.

##  2 Related Work

Since our goal is to build a GAN with the purpose of applying it in the
agricultural domain, the literature review is conducted from both aspects.
First, we introduce several GAN architectures proposed in the literature.
Then, we present a summary of deep learning approaches, including GANs that
have been used in agriculture.

###  2.1 Generative Adversarial Network (GAN)

GAN was first proposed by Goodfellow et al. in 2014 [22] using multi-layer
perceptron (MLP) network with a min-max loss function to generate synthetic
images, known as Standard GAN (SGAN). Later on, researchers suggested other
loss functions like least-square (LSGAN) [25], Wasserstein distance (WGAN)
[28], Wasserstein distance with gradient penalty (WGAN-GP) [29] and hinge loss
[30] in order to improve the performance and increase the stability during GAN
training. Radford et al. came up with stable architectural guidelines for
convolution GANs leading to a class of architectures named Deep Convolutional
GAN (DCGAN) [31]. Mirza et al. developed Conditional GAN (CGAN) by
incorporating auxiliary information (class label) during training to control
the image being generated by the GAN model [12]. In [18, 21], the authors
proposed Continuous conditional Generative Adversarial Network (CcGAN) to
train GANs on continuous, scalar conditions (regression labels) by introducing
novel label input mechanisms and reformulating empirical CGAN loss function
with Hard Vicinal Estimate and Soft Vicinal Estimate using the concept of
Vicinal Risk Minimization. In [20], the authors suggested Performance
Conditioned Diverse Generative Adversarial Network (PcDGAN) to overcome
challenges of CcGAN by incorporating Determinantal Point Processes (DPP)-based
singular vicinal loss function to increase diversity in the generated images
and applied this in engineering design tasks. Zhang et al. proposed Generator
Regularized-conditional GAN (GRcGAN) by introducing a Lipschitz regularization
term in CGAN loss function to train GAN on continuous space [19]. Isola et al.
designed an image-conditional GAN framework, called as Pix2Pix for image-to-
image translation, using a set of aligned image pairs as training data [16].
Thereafter, Cycle-consistent GAN (CycleGAN) [17] was proposed adopting an
unsupervised approach for image-to-image translation using unpaired training
data with cycle-consistency loss eliminating the requirement of aligned pairs
of images as opposed to Pix2Pix. Karras et al. proposed a new training
methodology for GAN utilizing the idea of progressive neural network (ProGAN)
[24] to generate high-resolution images. In [23], Zhang et al. incorporated
long-range dependencies by introducing self-attention modules on top of
convolution layers in the GAN architecture, referred as Self Attention GAN
(SAGAN). BigGAN was implemented on top of SAGAN architecture by employing
certain techniques (such as truncation trick, orthogonal regularization),
which substantially improves the performance of class-conditional GAN [32].
Liu et al. proposed an adaptive global and local bilevel optimization model
(GL-GAN), that embedded a local bilevel optimization technique to improve the
poor quality portion on an image along with traditional global optimization
technique to optimize the whole image [33].

###  2.2 Applications in Agriculture

In [5], the authors presented a comprehensive overview on the application of
deep learning techniques in plant phenological research over the past few
years — which indicates that most of the literature studied classification
(e.g., phenological stages) and segmentation tasks (e.g., flower or fruit
segmentation, counting buds, flowers and fruits, presence of species etc.)
based on plant imagery using Convolution Neural Networks (CNNs). Lee et al.
utilized CNN to better represent the features of leaf images for the
identification of plants [34]. Cao et al. predicted leaf phenology of DB
forests in terms of leaf growing dates after start-of-growing season in a year
from PhenoCam images using CNN [35]. In [36], the authors developed a deep
learning based platform, known as Deep Plant Phenomics, to accelerate image-
based plant phenotyping.

Due to the non-availability of large amount image data required for CNN
implementation, GAN has been used for image data augmentation by synthesizing
new realistic images in order to improve machine learning performance for
various applications related to precision agriculture and plant phenotyping
(plant disease recognition, weed control, fruit detection, leaf counting, leaf
segmentation, plant seedling, plant vigor rating etc.) [37]. Further, a semi-
automated pipeline for data augmentation was proposed using GAN for
agricultural pests detection [38]. In [39], the authors applied CycleGAN
between Sentinel-1 (Synethetic Aparture Radar) and Sentinel-2 (optical) of
satellite data in order to improve crop type mapping and identification.
Miranda et al. suggested to model plant growth as an image-to-image
translation task based on conditional GAN to predict plant growth stage as a
function of its previous growing stage and diverse environmental conditions
which influences plant growth [40].

##  3 Methodologies

A typical GAN architecture consists of two models — generator and
discriminator. The generator learns the distribution of input images and
computes a differentiable function, which maps a latent vector space to the
input data space in order to generate synthetic images, whereas the
discriminator classifies between real and synthetic images. The architecture
adopts an adversarial training mechanism in which both the models are trained
simultaneously by formulating them as a competition. The discriminator tries
to improve its classification accuracy by correctly distinguishing between
real and synthetic images, while the generator generates realistic synthetic
images from random noise (e.g., spherical Gaussian) in order to deceive the
discriminator. While any differentiable network can be used to implement a
GAN, it is common to use deep neural networks for its implementation.

In order to develop our GAN model for generating synthetic forestry images
conditioned on GCC over a specific ROI, we employ the concept of CGAN [12],
i.e., feeding auxiliary information to both the generator and the
discriminator to exercise control over the images being generated. The CGAN in
[12] utilized an MLP network as the baseline architecture and a categorical
attribute as auxiliary information to generate images conditioned on class
labels. However, recent advances have revealed the power of using CNN to
synthesize new examples of images. Specially, the DCGAN [31] architecture has
become one of the most popular and successful in literature, which was
implemented in an unconditional setting. On top of the basic architectural
guidelines recommended by DCGAN, we propose a novel GAN architecture, which
provides a continuous attribute, viz., GCC, and an ROI image as auxiliary
input to the generator and the discriminator. Figure 3 refers to the outline
of our proposed approach. Here, the idea is to utilize random values within a
range of values as an auxiliary input to the generator so that the generator
is being conditioned on the continuous attribute instead of feeding only the
precise values present in the training dataset to the generator. Further, the
ROI input to the generator as well as discriminator incorporates a masking
mechanism to focus the conditioning of the given GCC over the given ROI.

Therefore, during generator training, a random noise vector, the ROI image,
and random GCC values within the range of GCC values for the given ROI of the
forest site under consideration are input to the generator in order to
generate the synthetic images satisfying the given GCC over the given ROI. The
synthetic images are then passed through the discriminator to estimate the
probability of them being real, based on which the generator loss is
calculated and the weight parameters of the generator model is updated through
back-propagation while keeping the discriminator parameters constant. The
discriminator is trained with the real images and their corresponding GCC
values and the ROI image to calculate the discriminator loss on real images.
Further, the synthetic images generated by the generator given the GCC values
present in the training dataset are used to compute the discriminator loss on
synthetic images. These two loss components are used to update the weight
parameters of the discriminator model. The pre-training steps of the inputs to
the generator and the discriminator are described later in this section.

Figure 3: Outline of our proposed approach: Generator inputs a random noise
vector, a random GCC value within the range of GCC values for a specific ROI
of the forest site under consideration, and the corresponding ROI image to
generate a synthetic image, which satisfies the given GCC over the given ROI.
Discriminator inputs the real or synthetic image and its corresponding GCC
value and the ROI image to estimate the probability of the input image being
real.

Our GAN architecture is shown in Figure 4. We utilize the following guidelines
recommended by the DCGAN architecture:

  * 1.

Removal of fully connected hidden layers on top of convolutional features.

  * 2.

Using strided convolutions for downsampling in the discriminator and
fractional-strided convolutions for upsampling in the generator instead of
pooling and scaling, respectively.

  * 3.

Applying LeakyReLU activation in all the layers of the discriminator and ReLU
in all the layers of the generator except the last layer.

  * 4.

Use of TanH activation function in the last layer of the generator.

Additionally, we integrate self-attention modules, proposed as part of SAGAN
[23] in both the generator and the discriminator to enable long-range
dependencies. We use spectral normalization in the discriminator and spectral
normalization along with batch normalization in the generator as suggested by
SAGAN in order to improve training stability. The PhenoCam RGB image is of
size $960\times 1296$. Due to the limited computational resources and longer
training time, the length and width of the PhenoCam images and ROI image are
reduced to half of their original size using bilinear interpolation and
synthetic images of dimension $480\times 648$ are generated by our GAN
architecture. After reducing the size, we validate that the calculated GCC of
the resized PhenoCam images using Equation (1) based on the ROI “DB_1000” is
same as that of the original-sized images provided by the PhenoCam website.
Therefore, the computation of the greenness as well as the prediction of the
redness index are not affected due to the resizing of our generated images.

(a) Generator

(b) Discriminator

Figure 4: Our GAN architecture: Generator uses transposed convolution layers
for upsampling, whereas discrimator uses convolution layers for downsampling.
Spectral normalization along with batch normalization is used in generator and
spectral normalization is used in discriminator to increase the stability
during training. Self-attention modules are incorporated in both the generator
and discriminator to improve the quality of the synthetic images (ablation
study is done in Section 4.1.6).

The input real images are normalized between $[-1,1]$ during training to make
them compatible with the output (synthetic images) of the generator, that uses
TanH activation function in the last layer. The GCC value is multiplied by 100
and rounded to 2 decimal digits (referred to as “adjusted” GCC in this paper)
before feeding it to the GAN model so as to increase the variance in the input
GCC values across the training images, consequently influencing the model
discrimination ability over different GCC values. Further, the ROI image given
by the Phenocam website is a black-and-white image where the black portion
denotes the region of interest (Figure 1). With the intention of focusing on
the ROI, the pixel values in the ROI image are inverted, i.e., they are set to
a value of 1 over the ROI and 0 for the rest of the image. The pre-processed
ROI image and the adjusted-GCC inputs are concatenated as additional channels
to the inputs of the generator and the discriminator. During training,
orthogonal initialization is used for the weight parameters in the convolution
and linear layers as suggested in [32]. We use the hinge loss [30] as an
adversarial loss for GAN training. The hinge loss for discriminator and
generator are defined below:

$L_{D}=-E_{(x)\sim p_{data}}[min(0,-1+D(x))]\\\ -E_{z\sim
p_{z}}[min(0,-1-D(G(z)))]$ (3) $L_{G}=-E_{z\sim p_{z}}[D(G(z))]$ (4)

where, $D(x)$: the probability that $x$ comes from the real data distribution
$p(data)$; $z$: the input to the generator; $G(z)$: the generator’s output on
the given input, $z$; $D(G(z))$: the discriminator’s estimate that the
generated synthetic image is real.

To measure the similarity between the real images and the synthetic images,
the SSIM index [13] is used. Motivated by the fact that human visual
perception is highly adapted to extract the structural information from a
scene to identify its difference from a reference, this metric was proposed to
extract the structural information from the sample and the reference image
based on three key features: luminance, contrast and structure and provide
with a score in the scale of $[-1,1]$. The higher the score, the more similar
the sample image is to the reference image. We use the built-in SSIM method of
the Scikit-Learn [41] library to calculate the SSIM index of the generated
synthetic images against the test images. Figure 5 illustrates the overall
process being used by the generator to generate the synthetic images.
Additionally, the RMSPE [14] of GCC and RCC values between the synthetic
images and test images are calculated using the following equation to validate
the accuracy and integrity of the generated images by our method.

$RMSPE=\dfrac{\sqrt{\sum_{j=1}^{N}(\frac{E_{j}}{A_{j}}*100)^{2}}}{N}$ (5)

Here, $N$: the number of samples, $A_{j}$: ground-truth value of sample $j$,
$E_{j}$: difference between the predicted and ground-truth value of sample
$j$.

Figure 5: Overall process being used by the generator: Generator model
generates the synthetic image given an ROI image and a GCC value and SSIM is
used to assess the quality of the synthetic image by comparing with the real
image.

##  4 Experiments

As mid-day images are the most significant in understanding green-up and
green-down across the year, the images captured between 10:00 a.m. and 2:00
p.m. daily ($9$ images per day) throughout the year are considered in this
work. Based on data availability, the images captured from January 2017 to
December 2020 are used for the training of our GAN model, and images from
January 2021 to December 2021 are used for testing. Upon filtering the images
based on the availability of GCC values provided by the PhenoCam Network, the
number of training and test images, respectively, are 12,149 and 3,189 for
Harvard Forest, and 12,307 and 3,227 for Bartlett Experimental Forest. For the
purpose of training the GAN model, we use Adam optimizer with a learning rate
of 0.0001 for the discriminator and 0.00005 for the generator, which are found
to be best suited for our dataset. The beta1 value of Adam optimizer is set to
0.9 and 0.5 for Harvard and Bartlett Forests respectively, and beta2 value is
set to 0.999 for both the sites. The following are the set of experiments that
were conducted as part of this work.

###  4.1 Training Individual Sites with a Specific Vegetation Type

We first individually train our GAN model on both the forest sites based on
the ROI labeled as “DB_1000” (denoting deciduous broadleaf vegetation) with
the above-mentioned parameters. Figures 6 (Harvard) and 7 (Bartlett) show (a)
examples of real images with GCC values sampled across the entire range from
the test dataset and (b) the corresponding synthetic images with their SSIM
indices. The GCC is calculated over the ROI of the synthetic images to compute
its deviation from the input GCC. Additionally, the RCC is calculated over the
ROI of the synthetic images to compare with the ground-truth RCC of the test
images.

(a) Sample images from test dataset

(b) Synthetic images after epoch $975$

Figure 6: Sample test images and synthetic images for Harvard Forest (SSIM
indicates the similarity score of synthetic image with the corresponding test
image. GCC and RCC correspond to the ROI “DB_1000” indicated on the right).

(a) Sample images from test dataset

(b) Synthetic images after epoch $825$

Figure 7: Sample test images and synthetic images for Bartlett Experimental
Forest (SSIM indicates the similarity score of synthetic image with the
corresponding test image. GCC and RCC correspond to the ROI “DB_1000”
indicated on the right side).

####  4.1.1 Assessing Quality of Synthetic Images

In order to asses the overall quality of the synthetic images, we use the SSIM
index by comparing the synthetic images with the corresponding test images
(left side of Figure 8). However, for a single GCC value, more than one
PhenoCam image may be available in the test dataset. Therefore, we recompute
the SSIM index by utilizing the maximum SSIM score obtained when comparing the
synthetic images with all the test images having the given GCC value. We refer
to this recomputed SSIM index as “adjusted” SSIM index (right side of Figure
8). Based on the adjusted-SSIM index, the generated synthetic images appear
much more similar to real images for both the forest sites. It is possible
that our model suffers from mode collapse problem [28] generating the same
image every time for a particular GCC value. To counter this, we perform an
analysis in Section 4.1.2.

(a) Harvard Forest

(b) Bartlett Experimental Forest

Figure 8: SSIM and adjusted-SSIM index of synthetic images against test images
(SSIM index indicates the score of the synthetic image after comparing it with
the corresponding test image and adjusted-SSIM index is the maximum score
obtained for the synthetic image after comparing with all the test images
corresponding to the given GCC value).

Moreover, to acquire an understanding of the SSIM across real images for a
particular GCC, we plot the histogram of SSIM scores for every pair of test
images corresponding to a single GCC value (Figure 9). For Harvard Forest,
there are 919 unique adjusted-GCC values across 3,189 test images, out of
which 523 GCC values have more than one image. Similarly, for Bartlett Forest,
873 unique adjusted-GCC values are present across 3,227 test images, out of
which 545 GCC values have more than one image. We consider that GCC value for
which the highest number of test images are available for plotting the
histogram. This analysis can then be utilized as a benchmark (a lower and an
upper bound) for the SSIM score that can be achieved for the PhenoCam images.
For Harvard Forest, most of the image pairs corresponding to the GCC value
0.3295 have SSIM index in the range [0.3, 0.4] and the lowest possible score
is 0.18. It is observed that 43.5% of synthetic images for this forest site
have an adjusted-SSIM index of more then 0.3 and 83.22% of synthetic images
have an adjusted-SSIM index of more than the lowest possible score (0.18).
Similarly, in case of Bartlett Forest, the range is [0.4, 0.5] for the real
images corresponding to the GCC value 0.3428 with the possible lowest score as
0.23 and we observe that 31% of synthetic images have an adjusted-SSIM index
of more than 0.4 along with 99.3% of synthetic images have an adjusted-SSIM
index of more than the lowest possible score (0.23). In light of these
observations, we infer that the SSIM scores obtained between the synthetic
images and real images are consistent with the SSIM scores obtained between
real images.

(a) Harvard Forest

(b) Bartlett Experimental Forest

Figure 9: SSIM index across test images corresponding to a single GCC value
(obtained by calculating SSIM for each pair of real images having the same GCC
value).

####  4.1.2 Fidelity and Diversity of Synthetic Images

In an attempt to judge the fidelity and diversity of the synthetic images
generated by our GAN model, we present some sample test images corresponding
to a single GCC value and synthetic images generated given that GCC value
(Figure 10). Though the overall structure remains same for all the synthetic
images corresponding to a particular GCC, the computed GCC and the predicted
RCC values are not exactly the same indicating the diversity of the images
being generated by the model. At the same time, we observe that the GCC of the
generated images are very much closer to the given GCC, confirming the
fidelity of the synthetic images. In order to quantify the fidelity, we
utilize the RMSPE of GCC of synthetic images as described in the next
subsection.

(a) Sample images from test dataset corresponding to GCC value $0.3295$

(b) Synthetic images generated given the GCC value $0.3295$

Figure 10: Sample test images and synthetic images for Harvard Forest
corresponding to a single GCC value over the “DB_1000” ROI indicating the
variety of generated images by our GAN model.

####  4.1.3 Evaluating Accuracy of Computed GCC and Predicted RCC

A comparative study using GCC and RCC is performed between the test images and
generated images (Figure 11). We observe that the degree of similarity between
the GCC distributions of the test images and synthetic images is higher than
the degree of similarity between the corresponding RCC distributions. For
instance, the highest and the second highest number of test images correspond
to GCC values around 0.33 and 0.41, respectively, in Harvard Forest, and the
synthetic images also exhibit similar characteristics. In case of RCC
distribution, the count of test images reaches its peak around the RCC value
of 0.38, whereas we observe that the RCC values of the synthetic images are
mostly around the values of 0.34 and 0.38. This observation establishes the
success of conditional part of our GAN architecture based on GCC. However, our
objective is also to predict other phenotypic attributes like RCC from the
synthetic images. From that perspective, it can be observed that the range of
the predicted RCC of the synthetic images covers the most frequent values of
RCC of the test images. The RMSPE of GCC and RCC values of the generated
images based on the ground-truth GCC and RCC values of test images are 2.1%
and 9.5%, respectively in case of Harvard Forest, and the RMSPE of GCC and RCC
in case of Bartlett Forest are 2.1% and 8.4%, respectively.

(a) Harvard Forest

(b) Bartlett Experimental Forest

Figure 11: GCC and RCC distribution across test images and corresponding
synthetic images over the “DB_1000” ROI.

####  4.1.4 Evaluating Efficacy of our GAN model

As already mentioned, our goal is to build a GAN model conditioned on the
continuous attribute over a specific portion of the image. As a means to
evaluate this, we choose only those sample images from the test dataset
corresponding to the GCC values which are not used to train the model (i.e.,
not present in the train dataset). For Harvard Forest, we find 97 such test
images involving 79 adjusted-GCC values — some of the original samples with
the corresponding generated images are shown in Figure 12. The RMSPE of GCC
and RCC across these 97 synthetic images are 5% and 9.8%, respectively. In
case of Bartlett Forest, there are $42$ such images in the test dataset
involving $34$ unique adjusted-GCC values, and the RMSPE of GCC and RCC across
these images are 4.6% and 6%, respectively.

(a) Sample images from test dataset

(b) Synthetic images after epoch $975$

Figure 12: Sample test images with GCC values not being used in training and
corresponding synthetic images for Harvard Forest depicting the ability of our
GAN model to generate synthetic images given the GCC value within the range
used in training (GCC and RCC correspond to the “DB_1000” ROI indicated on the
right).

####  4.1.5 Analyzing Significance of Proposed Work

A kind of blurriness is detected across PhenoCam images for certain GCC
values. This work also aims to improve the quality of appearance of the forest
sites based on a greenness value. Therefore, we take some of the blurred
sample images from the test set of Harvard Forest and generate synthetic
images using our GAN model as shown in Figure 13. It is observed that the
generated images could be utilized to better visualize the appearance of the
forest sites in these cases. Additionally, plant biologists could leverage
these synthetic images to gain a better understanding of other phenotypic
attributes.

(a) Sample images from test dataset

(b) Synthetic images after epoch $975$

Figure 13: Sample blurred test images and corresponding synthetic images for
Harvard Forest illustrating the potential of our GAN model to better portray
the appearance of the forest based on a greenness value (GCC and RCC
correspond to the ROI “DB_1000” indicated on the right).

####  4.1.6 Ablation Study on Self-Attention Modules

In order to measure the impact of self-attention modules on the performance of
our GAN architecture, we train the GAN model for the Harvard Forest after
excluding self-attention modules. Figure 14 shows the similarity scores
obtained between the test images and the corresponding generated images by the
model trained without self-attention modules. The comparison with the
similarity scores achieved with self-attention modules for Harvard Forest
shown on Figure 8 reveals that the addition of self-attention modules improves
the quality of the generated images in terms of SSIM index as well as
adjusted-SSIM index. We observe that 9% and 39.9% of the synthetic images of
Harvard Forest, respectively, obtain SSIM and adjusted-SSIM index more than
0.3 (lower bound of the benchmark described in Section 4.1.1) without self-
attention modules, whereas 21% and 43.5% of the synthetic images of the same
forest site, respectively, achieve SSIM and adjusted-SSIM score more than 0.3
with self-attention modules.

Figure 14: SSIM index of synthetic images against test images for Harvard
Forest after training without self-attention modules (Comparison with Figure 8
shows the improvement of SSIM score after adding self-attention modules in our
GAN model).

####  4.1.7 Significance of ROI in Proposed Method

Moreover, we seek to analyze the significance of providing ROI as input to the
model in order to generate synthetic images satisfying the given GCC value
over the ROI. As part of this evaluation, the idea is to feed only the GCC
values of the ROI as auxiliary information to the GAN model (both generator
and discriminator) rather than providing two auxiliary information (GCC and
ROI) as inputs to the model. Under this consideration, the method does not
incorporate the masking mechanism to condition GCC over the ROI; instead, the
input GCC values are conditioned on the whole image with an assumption that
the generator will be able to generate synthetic images given a GCC value. The
same GAN architecture (Figure 4) is trained with the Harvard Forest dataset
(GCC values correspond to ROI “DB_1000”) by excluding the ROI image as input
to the model but using the same training process described in Section 3. We
observe that the proposed GAN architecture without ROI as input is incapable
of producing desired images and fails to achieve our goal (Figure 15), thereby
conveying the importance of ROI in our proposed method.

Figure 15: Synthetic images generated after training the GAN model without ROI
information. This is for the Harvard Forest site. Notice the degraded quality
of the images which underscores the need for the ROI information.

####  4.1.8 Investigating Limitations of Proposed Method

We also examine the limitations of our proposed GAN model. It is observed that
the adjusted-SSIM scores of synthetic images generated by our method are
comparatively higher over some GCC values than others within the specified
range for both the forest sites (right side of Figure 16). These outcomes are
borne out with the number of training images available corresponding to
different GCC values (left side of Figure 16). It indicates that the
performance of our model is affected by the limited availability of the
training images over some GCC values. Consequently, the proposed method is
unable to generate synthetic images with conformable quality over the entire
specified range of greenness.

(a) Harvard Forest

(b) Bartlett Experimental Forest

Figure 16: GCC distribution across training images (left) and adjusted-SSIM
score distribution across synthetic images (right) depicting the impact on
model performance due to limited training data (Adjusted-SSIM index is the
maximum score obtained for the synthetic image after comparing it with all the
test images corresponding to the given GCC value).

Next, we view the model output corresponding to GCC values outside the
specified range used during training. Figure 17 displays examples of synthetic
images of Harvard Forest corresponding to GCC values that fall outside the
range used during training for the “DB_1000” ROI. As there are no real images
corresponding to these GCC values, we cannot assess the quality of these
synthetic images with real images using SSIM scores. Visually, it appears that
these generated images exhibit blurriness and the image quality worsens as the
GCC value moves farther away from the specified range. However, when given GCC
values lower than the specified range as input to the model, the generated
images tend to be red in color, and the computed GCC values are closer to the
lower bound of the range. When given GCC values greater than the specified
range, the model generates images that are likely to be green in color, and
the computed GCC values are close to the upper bound of the range.

Figure 17: Impact of providing GCC values outside the range specified during
training. These examples are synthetic images of Harvard Forest corresponding
to GCC values outside the specified range [0.27,0.47] for the ROI “DB_1000”.
When providing GCC values lower than the range, the generated images tend to
be red, whereas given GCC values greater than the range, the synthetic images
tend to be green indicating their proximity to the lower and upper bound of
the range, respectively.

###  4.2 Cross-site Training: Harvard Forest to Bartlett Experimental Forest

We conduct a study to verify the generalizability of our model, i.e.,
determine if the model trained on one forest site can be extended to other
sites using a small amount of data and fewer number of iterations. In this
regard, the model trained on the Harvard Forest dataset for 975 epochs
(described in Section 4.1) is considered. It is then fine-tuned with 50% of
the training data from Bartlett Forest for 100 epochs with the training
parameters derived from this data. It is observed that the model effectively
adapts to generate synthetic images for other sites. A comparative analysis is
performed between (1) the model trained from scratch for the Bartlett Forest
described in Section 4.1, and (2) the cross-site trained model. In case (1),
the RMSPE of greenness and redness index of the generated images are 4.8% and
12.6%, respectively, whereas in case (2), we obtain RMSPEs as 3% and 12.2%,
respectively. The SSIM and adjusted-SSIM score of the synthetic images when
compared against the test images are presented in Figure 18. Figure 19 shows
some sample images from the test set and the corresponding synthetic images
for both the cases.

(a) Case 1: Trained from scratch with the full Bartlett Forest training
dataset

(b) Case 2: Updated the Harvard Forest model with 50% of the Bartlett Forest
training data

Figure 18: Cross-site experiment: Comparison of SSIM index for synthetic
images against the test dataset for Bartlett Experimental Forest.

(a) Sample Images from Test Dataset

(b) Case 1: Synthetic images after epoch 100 (Trained from scratch with the
full Bartlett Forest training dataset)

(c) Case 2: Synthetic images after epoch 100 (Updated the Harvard Forest model
using 50% of the Bartlett Forest training data )

Figure 19: Cross-site experiment: Sample test images and synthetic images for
Bartlett Experimental Forest (SSIM indicates the similarity score of synthetic
image with the corresponding test image. GCC and RCC correspond to the
“DB_1000” ROI indicated on the right).

###  4.3 Scalability to Other Vegetation Type

Next, we attempt to assess the scalability of our GAN model trained on a
particular vegetation type to another vegetation type in the same forest site.
For this, the model trained on the Harvard Forest dataset with the “DB_1000”
ROI described in Section 4.1 is first examined with another ROI “EN_1000” in
the same forest site; RMSPE of 6.4% (GCC) and 7.11% (RCC) are obtained.
Thereafter, the model is further trained for 25 epochs with 25% of the
training data of Harvard Forest for the ROI “EN_1000” and we obtain RMSPEs of
2.4% (GCC) and 7.06% (RCC). Figure 20 shows some sample images from the test
dataset and the corresponding synthetic images for each of these cases. We
observe that the SSIM index and GCC are improved after the “DB_1000” model is
extended using just 25% of the training data from the new ROI “EN_1000”. This
further explains that our proposed model is fairly robust to the size and
location of the ROI.

(a) Sample images from test dataset

(b) Synthetic images after epoch 975 (Using the Harvard Forest model trained
with the “DB_1000” ROI)

(c) Synthetic images after epoch 25 (After training on top of Harvard Forest
model (“DB_1000”) using 25% of the “EN_1000” ROI training data)

Figure 20: Cross-vegetation experiment: Sample test images and synthetic
images for Harvard Forest (SSIM indicates the similarity score of synthetic
image with the corresponding test image. GCC and RCC correspond to the ROI
“EN_1000” indicated on the right).

##  5 Conclusion

In this work, we present a novel GAN architecture for synthesizing forestry
images satisfying a specific phenotypic attribute, viz., greenness index over
the ROI of an image. Experiments on the PhenoCam dataset indicated that the
synthetic images generated by our GAN model can be utilized to (a) visualize
the appearance of a forest site based on the greenness value, and (b) predict
other phenotypic attributes (e.g.,. redness index) that were not used during
image synthesis. The SSIM scores between the generated images and real images
were observed to be analogous to the SSIM scores between real images, thereby
substantiating the quality of the generated images. Further, the proposed
model is capable of producing a variety of images pertaining to a particular
GCC value. It also has the ability to generate forestry images corresponding
to GCC values not used during training but within the specific range defined
for the forest site. We also demonstrated that our GAN model trained on one
forest site can be fine-tuned to generate images for other forest sites, which
in turn establishes the generalization capability of the model. In addition,
the model is scalable to other vegetation types within the same forest site in
an efficient manner.

From a broader perspective, this work aims to advance the study on image
generation by identifying patterns in images that do not have distinct
morphological structure. Rather, our model automatically learned the
phenomenon of green-up and green-down based on the colors and textures of
images. Additionally, we applied conditioning of continuous attribute on a
certain portion of the image (ROI), which gave us control over the image
generation process. Moreover, this technique could be leveraged to visualize
forestry based on different plant phenotypes (e.g., canopy cover) in the
context of various environmental parameters (e.g., temperature,
precipitation).

However, due to the limited size of the training dataset and asymmetric
distribution of GCC values across training images, the proposed model was
unable to generate high-quality images over some GCC values. It must also be
noted that due to computational and time constraints, the size of the
generated images was set to be smaller than that of the original PhenoCam
images. Additionally, our proposed model only captures the phenotypic
relationship patterns in a forest site without considering the change in
environmental parameters over the years.

Currently, we are working to further improve the quality of the generated
forestry images by using stable diffusion models [42]. This work can also be
extended to other forest sites belonging to other NEON domains. In addition,
other phenological and phenotypical information (e.g., LAI, canopy cover)
could also be extracted from the synthetic images. Our methodology could also
potentially be enhanced by accounting for various weather parameters and
ecological factors. We believe that the work reported in this paper provides a
first step in leveraging generative AI principles from pattern recognition and
computer vision for plant phenological research.

## Acknowledgments

This work is supported by National Science Foundation (NSF Award 1940059). We
thank all other project members, especially Dr. Bryan Heidorn, Dr. David
LeBauer, Dr. Jessica Guo, Dr. Anne Thessen, Dr. Laurel Cooper and Dr. Pankaj
Jaiswal. We also thank the reviewers for their valuable suggestions.

## References

  * [1] M. Ruml, T. Vulić, Importance of phenological observations and predictions in agriculture, Journal of Agricultural Sciences 50 (2005) 217–225.  doi:10.2298/JAS0502217R.
  * [2] J. Tang, C. Körner, H. Muraoka, S. Piao, M. Shen, S. J. Thackeray, X. Yang, Emerging opportunities and challenges in phenology: A review, Ecosphere 7 (8) (2016) e01436.  doi:10.1002/ecs2.1436.
  * [3] K. K. Bauckhage, C., Data mining and pattern recognition in agriculture, Künstl Intell 27 (2013) 313–324.  doi:10.1007/s13218-013-0273-0.
  * [4] S. Das Choudhury, A. Samal, T. Awada, Leveraging image analysis for high-throughput plant phenotyping, Frontiers in Plant Science 10 (2019).  doi:10.3389/fpls.2019.00508.
  * [5] N. Katal, M. Rzanny, P. Mäder, , J. Wäldchen, Deep learning in plant phenological research: A systematic literature review, Frontiers in Plant Science 13 (2022) 805738.  doi:10.3389/fpls.2022.805738.
  * [6] K. Mochida, S. Koda, K. Inoue, T. Hirayama, S. Tanaka, R. Nishii, F. Melgani, Computer vision-based phenotyping for improvement of plant productivity: a machine learning perspective, GigaScience 8 (1) (2018).  doi:10.1093/gigascience/giy153.
  * [7] S. Bond-Taylor, A. Leach, Y. Long, C. G. Willcocks, Deep generative modelling: A comparative review of vaes, gans, normalizing flows, energy-based and autoregressive models, IEEE Transactions on Pattern Analysis and Machine Intelligence 44 (11) (2022) 7327–7347.  doi:10.1109/TPAMI.2021.3116668.
  * [8] J. Keskitalo, G. Bergquist, P. Gardeström, S. Jansson, A cellular timetable of autumn senescence, Plant Physiol. 139 (4) (2005) 1635–48.  doi:10.1104/pp.105.066845.
  * [9] Y. Liu, C. Wu, O. Sonnentag, A. R.Desai, J. Wang, Using the red chromatic coordinate to characterize the phenology of forest canopy photosynthesis, Agricultural and Forest Meteorology 285-286 (2020) 107910.  doi:10.1016/j.agrformet.2020.107910.
  * [10] A. D. Richardson, K. Hufkens, T. Milliman, D. M. Aubrecht, M. Chen, J. M. Gray, M. R. Johnston, T. F. Keenan, S. T. Klosterman, M. Kosmala, E. K. Melaas, M. A. Friedl, S. Frolking, Tracking vegetation phenology across diverse north american biomes using phenocam imagery, Scientific Data 5 (180028) (2018).  doi:10.1038/sdata.2018.28.
  * [11] National Ecological Observatory Network (NEON), Phenology images (dp1.00033.001) (2023).
URL https://data.neonscience.org/data-products/DP1.00033.001

  * [12] M. Mirza, S. Osindero, Conditional generative adversarial nets, CoRR abs/1411.1784 (2014).
  * [13] Z. Wang, A. Bovik, H. Sheikh, E. Simoncelli, Image quality assessment: from error visibility to structural similarity, IEEE Transactions on Image Processing 13 (4) (2004) 600–612.  doi:10.1109/TIP.2003.819861.
  * [14] D. Swanson, J. Tayman, T. Bryan, Mape-r: A rescaled measure of accuracy for cross-sectional subnational population forecasts, Journal of Population Research 28 (2011) 225–243.  doi:10.1007/s12546-011-9054-5.
  * [15] Y. Pang, J. Lin, T. Qin, Z. Chen, Image-to-image translation: Methods and applications, IEEE Transactions on Multimedia 24 (2022) 3859–3881.  doi:10.1109/tmm.2021.3109419.
  * [16] P. Isola, J.-Y. Zhu, T. Zhou, A. A. Efros, Image-to-image translation with conditional adversarial networks, in: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017, pp. 5967–5976.  doi:10.1109/CVPR.2017.632.
  * [17] J.-Y. Zhu, T. Park, P. Isola, A. A. Efros, Unpaired image-to-image translation using cycle-consistent adversarial networks, in: IEEE International Conference on Computer Vision (ICCV), 2017, pp. 2242–2251.  doi:10.1109/ICCV.2017.244.
  * [18] X. Ding, Y. Wang, Z. Xu, W. J. Welch, Z. J. Wang, Cc{gan}: Continuous conditional generative adversarial networks for image generation, in: International Conference on Learning Representations (ICLR), 2021.
  * [19] Y. Zhang, Y. Zheng, X. Ma, S. Teng, Z. Zheng, Mind your step: Continuous conditional GANs with generator regularization, in: NeurIPS 2022 Workshop on Synthetic Data for Empowering ML Research, 2022.
  * [20] A. Heyrani Nobari, W. Chen, F. Ahmed, Pcdgan: A continuous conditional diverse generative adversarial network for inverse design, in: Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining, KDD ’21, 2021, p. 606–616.  doi:10.1145/3447548.3467414.
  * [21] X. Ding, Y. Wang, Z. Xu, W. J. Welch, Z. J. Wang, Continuous conditional generative adversarial networks: Novel empirical losses and label input mechanisms, IEEE Transactions on Pattern Analysis and Machine Intelligence 45 (7) (2023) 8143–8158.  doi:10.1109/TPAMI.2022.3228915.
  * [22] I. J. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, Y. Bengio, Generative adversarial networks, in: Proceedings of the 27th International Conference on Neural Information Processing Systems (NIPS), Vol. 2, 2014.  doi:10.1145/3422622.
  * [23] H. Zhang, I. Goodfellow, D. Metaxas, A. Odena, Self-attention generative adversarial networks, in: International conference on machine learning, 2019, pp. 7354–7363.
  * [24] T. Karras, T. Aila, S. Laine, J. Lehtinen, Progressive growing of gans for improved quality, stability, and variation, in: International Conference on Learning Representations (ICLR), 2018.
  * [25] X. Mao, Q. Li, H. Xie, R. K. Lau, Z. Wang, S. Smolley, Least squares generative adversarial networks, in: IEEE International Conference on Computer Vision (ICCV), 2017, pp. 2813–2821.  doi:10.1109/ICCV.2017.304.
  * [26] M. Arsenovic, M. Karanovic, S. Sladojevic, A. Anderla, D. Stefanovic, Solving current limitations of deep learning based approaches for plant disease detection, Symmetry 11 (7) (2019).  doi:10.3390/sym11070939.
  * [27] A. Dash, J. C. B. Gamboa, S. Ahmed, M. Liwicki, M. Z. Afzal, Tac-gan - text conditioned auxiliary classifier generative adversarial network (2017).  doi:10.48550/ARXIV.1703.06412.
  * [28] M. Arjovsky, S. Chintala, L. Bottou, Wasserstein generative adversarial networks, in: Proceedings of the 34th International Conference on Machine Learning (ICML), Vol. 70, 2017, p. 214–223.
  * [29] I. Gulrajani, F. Ahmed, M. Arjovsky, V. Dumoulin, A. Courville, Improved training of wasserstein gans, in: Proceedings of the 31st International Conference on Neural Information Processing Systems, NIPS’17, 2017, p. 5769–5779.
  * [30] J. H. Lim, J. C. Ye, Geometric gan (2017).  doi:10.48550/ARXIV.1705.02894.
  * [31] A. Radford, L. Metz, S. Chintala, Unsupervised representation learning with deep convolutional generative adversarial networks, in: 4th International Conference on Learning Representations, (ICLR), 2016.
  * [32] A. Brock, J. Donahue, K. Simonyan, Large scale GAN training for high fidelity natural image synthesis, in: International Conference on Learning Representations (ICLR), 2019.
  * [33] Y. Liu, H. Fan, X. Yuan, J. Xiang, GL-GAN: Adaptive global and local bilevel optimization for generative adversarial network, Pattern Recognition 123 (2022) 108375.  doi:https://doi.org/10.1016/j.patcog.2021.108375.
  * [34] S. H. Lee, C. S. Chan, S. J. Mayo, P. Remagnino, How deep learning extracts and learns leaf features for plant classification, Pattern Recognition 71 (2017) 1–13.  doi:https://doi.org/10.1016/j.patcog.2017.05.015.
  * [35] M. Cao, Y. Sun, X. Jiang, Z. Li, Q. Xin, Identifying leaf phenology of deciduous broadleaf forests from phenocam images using a convolutional neural network regression method, Remote Sensing 13 (12) (2021).  doi:10.3390/rs13122331.
  * [36] J. R. Ubbens, I. Stavness, Deep plant phenomics: A deep learning platform for complex plant phenotyping tasks, Frontiers in Plant Science 8 (2017).  doi:10.3389/fpls.2017.01190.
  * [37] Y. Lu, D. Chen, E. Olaniyi, Y. Huang, Generative adversarial networks (gans) for image augmentation in agriculture: A systematic review, Computers and Electronics in Agriculture 200 (2022) 107208.  doi:10.1016/j.compag.2022.107208.
  * [38] C. Karam, M. Awad, Y. Abou Jawdah, N. Ezzeddine, A. Fardoun, Gan-based semi-automated augmentation online tool for agricultural pest detection: A case study on whiteflies, Frontiers in Plant Science 13 (2022).  doi:10.3389/fpls.2022.813050.
  * [39] E. Şener, E. Çolak, E. Erten, G. Taşkin, The added value of cycle-gan for agriculture studies, in: IEEE International Geoscience and Remote Sensing Symposium (IGARSS), 2021, pp. 7039–7042.  doi:10.1109/IGARSS47720.2021.9553876.
  * [40] M. Miranda, L. Drees, R. Roscher, Controlled multi-modal image generation for plant growth modeling, in: International Conference on Pattern Recognition (ICPR), 2022, pp. 5118–5124.  doi:10.1109/ICPR56361.2022.9956115.
  * [41] F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, E. Duchesnay, Scikit-learn: Machine learning in Python, Journal of Machine Learning Research 12 (2011) 2825–2830.
  * [42] P. Dhariwal, A. Nichol, Diffusion models beat gans on image synthesis, in: Advances in Neural Information Processing Systems (NIPS), Vol. 34, 2021, pp. 8780–8794.