# CR electrons and positrons: what we have learned in the latest three years
and future perspectives

Daniele Gaggero  Dario Grasso  Department of Physics, Pisa University,
Largo B. Pontecorvo 3, 56127 Pisa Italy
∗E-mail<EMAIL_ADDRESS>

###### Abstract

After the PAMELA finding of an increasing positron fraction above 10 GeV, the
experimental evidence of the presence of a new electron and positron spectral
component in the cosmic ray zoo has been recently confirmed by Fermi-LAT. We
show as a simple phenomenological model which assumes the presence of an
electron and positron extra component peaked at $\sim 1~{}{\rm TeV}$ allows a
consistent description of all available data sets. We then describe the most
relevant astrophysical uncertainties which still prevent to determine
$e^{\pm}$ source properties from those data and the perspectives of
forthcoming experiments.

###### keywords:

Proceedings; World Scientific Publishing.

##  1 Introdution

Recent experimental results raised a wide interest about the origin and the
propagation of the leptonic component of the cosmic radiation.

Among the most striking of those results, there is the observation performed
by the PAMELA satellite experiment that the positron to electron fraction
$e^{+}/(e^{-}+e^{+})$ rises with energy from 10 up to 100 GeV at least
(Adriani et al. 2008 [1]). This appeared in contrast with the predictions of
the standard cosmic ray scenario and could therefore be interpreted as the
smoking gun of new physics, unless a very soft electron spectrum was assumed.

The significance of this anomaly increased when the Fermi-LAT space
observatory measured the $e^{-}+e^{+}$ spectrum in the 7 GeV - 1 TeV energy
range with unprecedented accuracy and found it to be compatible with a power-
law with index $\gamma(e^{\pm})=-3.08\pm 0.05$ (Abdo et al. 2009 [2],
Ackermann et al. 2010 [3]); this slope is significantly harder than what
estimated on the basis of previous measurements: the hypothesis of a steep
spectrum was therefore excluded.

More recently, the same collaboration provided a further, and stronger,
evidence of the positron anomaly by providing direct measurement of the
absolute $e^{+}$ and $e^{-}$ spectra, and of their fraction, between 20 and
200 GeV using the Earth magnetic field. A steady rising of the positron
fraction was observed by this experiment up to that energy in agreement with
that found by PAMELA. In the same energy range, the $e^{-}$ spectrum was
fitted with a power-law with index $\gamma(e^{-})=-3.19\pm 0.07$ which is in
agreement with what recently measured by PAMELA between 1 and 625 GeV (Adriani
et al. 2011 [4]). Most importantly, Fermi-LAT measured, for the first time,
the $e^{+}$ spectrum in the 20 - 200 GeV energy interval and showed it is
fitted by a power-law with index $\gamma(e^{+})=-2.77\pm 0.14$.

We will show in the following paragraph how all those measurements rule out
the standard scenario in which the bulk of electrons reaching the Earth in the
GeV - TeV energy range are originated by Supernova Remnants (SNRs) and only a
small fraction of secondary positrons and electrons comes from the interaction
of CR nuclei with the interstellar medium (ISM). Then we will see how the
alternative scenario in which the presence of electron + positron component
peaked at $\sim 1$ TeV is invoked allows a consistent description of all the
available data sets. Finally we will discuss to which extent astrophysical and
particle physics uncertainties still affect our modeling of cosmic ray leptons
origin and propagation and how forthcoming measurements are expected to reduce
those uncertainties.

##  2 The necessity of a primary extra-component

After the release of Fermi-LAT $e^{-}+e^{+}$ spectrum, it was clearly pointed
out in several papers (see e.g. Grasso et al. 2009 [5] and Di Bernardo et al.
2011 [6]) that both Fermi-LAT and PAMELA measurements described in the
Introduction are in contrast with a standard single-component scenario in
which positrons are the secondary products of the nuclear component of cosmic
rays (CRs) interacting with the interstellar medium (ISM).

Figure 1: Fermi-LAT and PAMELA data on electrons + positrons and electrons are
compared to a double component phenomenological model. The absolute positron
spectrum is compared to a single and double component phenomenological model.
Red dotted line: $e^{+}$ in single-component scenario. Red dot-dashed line:
$e^{+}$ in double-component scenario. Blue triple dotted-dashed line, black
solid line: $e^{-}$ and $e^{-}+e^{+}$ in double-component scenario. Blue
dashed line: $e^{-}$ diffuse background in double-component scenario. The
Kolmogorov diffusion setup is adopted.

The main problems encountered by this kind of models can be summarized as
follows.

  * •

As explained many times (see e.g. Serpico 2011 [7] for a recent review), they
cannot reproduce the rising positron-to-electron ratio measured by PAMELA and
recently confirmed by Fermi-LAT;

  * •

They are unable to reproduce all the features revealed by Fermi-LAT in the CRE
spectrum, in particular the flattening observed at around 20 GeV and the
softening at $\sim 500$ GeV. In fact, if such models are normalized against
data in the 20 - 100 GeV energy range, where systematical and theoretical
uncertainties are the smallest, they clearly fail to match CRE Fermi-LAT and
PAMELA $e^{-}$ data outside that range. A different normalization results in
even worse fits.

With the release of the $e^{-}$ and $e^{+}$ separate spectra by the Fermi-LAT
collaboration the problems with the single component scenario became even
worse. In fact, the $e^{+}$ spectrum (Fig. 1) is clearly inconsistent with the
predictions of a single component scenario computed with DRAGON numerical
diffusion package (and similar results are obtained with GALPROP). Even
without considering numerical models, the simple consideration that the
reported positron spectral slope is $-2.77\pm 0.14$ reveals how these data are
incompatible with a purely secondary origin from proton spallation on
interstellar gas: the source slope should be the same as the proton spectrum,
i.e. $\simeq-2.75$ (Adriani et al. 2011 [8]) and no room is then left for the
unavoidable steepening due to energy-dependent diffusion and energy losses.

Figure 2: Fermi-LAT and PAMELA data on the positron ratio are compared to a
single and double component phenomenological model. Dot-dashed line: positron
ratio in single-component scenario. Dotted line: positron ratio in double-
component scenario due to conventional secondary positron production. Solid
line: positron ratio in double-component scenario including extra-component.
The progagation setup and modulation potential are the same of Fig. 1. The
solar modulation potential is taken $\Phi=550$ MV in all figures of this
paper.

A double component scenario is the most straightforward solution to these
problems.

The idea dates back to the pioneering work by F. Aharonian and A. Atoyan 1995
[9] and was extensively studied after the release of ATIC and PAMELA data in
2008 (see e.g. Hooper et al. 2009 [10] and Profumo 2008 [11]).

More recently, we contributed to several papers in which it was shown that a
consistent interpretation of the $e^{+}+e^{-}$ spectrum measured by Fermi-LAT
and the PAMELA positron fraction can be naturally obtained in that framework
(Grasso et al. 2009 [5], Ackermann et al. 2010 [3], Di Bernardo et al. 2011
[6]).

For example, in Fig. 1 and Fig. 2 we show that the double component model
proposed in Ackermann et al. 2010[3] reproduces the data mentioned above and
also the $e^{+}$ and $e^{-}$ separate spectra, and their ratio, recently
released by the Fermi-LAT collaboration and not yet available at the time. The
model represented in those figures assumes a propagation setup characterized
by a cylindrical diffusive halo with half-thikness of 4 kpc; a diffusion
coefficient scaling with rigidity like $\rho^{1/3}$ (corresponding to a
Kolmogorov-like diffusion within the quasi-linear approximation) and a
relatively strong reacceleration (the Alfvén velocity is $v_{A}=30~{}{\rm
kms^{-1}}$). Solar modulation is treated here as charge independent in the
force field approximation by fixing the modulation potential $\Phi$ against
proton data taken in the same solar phase. In that model, the standard $e^{-}$
primary component is tuned to fit Fermi-LAT data at low energy in the presence
of the extra-component becoming dominant at higher energies; the injection
slope for the primary electron component is set to $-2.70$ above 2 GeV, while
under that energy a slope of $-1.6$ is adopted, in accord with recent
constraints from the synchrotron spectra (see Jaffe et al. 2011 [12]). The
extra component, instead, originates from a primary source of
electron+positron pairs; it has an injection spectrum modelled in a simple way
as a power-law with index $-1.5$ plus an exponential cutoff at $1.2$ TeV; the
spatial distribution of this source is the same as the standard one and the
propagation parameters are also the same; the normalization is tuned so that
Fermi-LAT and PAMELA data at high energy are matched by the sum of standard +
extra component. Both components are computed with DRAGON (even if it was
checked that the same result can be obtained with GALPROP).

An issue remains open about the origin of the discrepancy between the
prediction of this, or similar, models and the positron fraction measured by
PAMELA below 10 GeV. In the next section we will show as that discrepancy may
be interpreted as the consequence of an incorrect choice of the propagation
setup and discuss other uncertainties which can affect the electron and
positron spectra in that low energy range.

##  3 LOW ENERGY. Impact of astrophysical uncertainties

Figure 3: Effect of changing the diffusion halo height. Solid line: h = 1 kpc;
dashed: h = 10 kpc. Figure 4: Effect of the diffusion setup. Solid line: KRA;
dashed line: KOL.

Cosmic ray electrons and positrons, either belonging to the standard or the
extra component, propagate in the Galaxy undergoing several physical
processes: diffusion, reacceleration, energy losses. Such complex motion is
effectively described by a well-known diffusion-loss equation (Berezinskii et
al. 1990 [13]). In this equation several free parameters are involved: the
height of the halo in which the propagation takes place, the normalization and
energy dependence of the diffusion coefficient (the latter parametrized by the
parameter $\delta$), the Alfvén velocity that influences the effectiveness of
reacceleration; moreover, several astrophysical inputs need to be considered:
the injection spectrum, the spatial distribution of the source term, the
interstellar radiation field, the gas distribution.

The free parameters that appear in the diffusion-loss equation are constrained
by some CR observables such as Boron-to-Carbon (B/C) or antiproton-to-proton
ratio; different diffusion setups exist in the literature, obtained through
comparison of experimental data with the prediction of semi-analytical codes
(Maurin et al. 2001 [14], Donato et al. 2004 [15]) or numerical packages such
as DRAGON or GALPROP (see e.g. Di Bernardo et al. 2010 [16] for DRAGON-related
models and Trotta et al. 2011 [17] for a GALPROP-based analysis).

The uncertainties related to the diffusion model and to the astrophysical
inputs were discussed in the latest years in several papers making use of
semi-analytic codes (e.g. Delahaye et al. 2010 [18]). In the following we will
briefly analyse the impact of these uncertainties adopting the DRAGON code.

One of the most relevant parameter is the halo height. According to the
analytical computations by Bulanov and Dogel [19], while at low energy the
electrons (or positrons) are distributed throughout all the diffusion halo, as
the energy increases the electrons occupy a smaller and smaller fraction of
the halo due to energy losses. This is relevant especially for the secondary
positron spectrum. In fact, since their injection power is determined by the
CR nuclei density in the Galactic disk, a thicker halo results in a larger
dilution of their density in the halo hence a in smaller flux on the Earth.
Numerical computations confirm the expectation of this heuristic argument as
shown in Fig. 3. From the plot it is also evident that large halo heights are
disfavoured by the data.

Even fixing the height of the diffusion halo, the choice of the diffusion
setup can also affect the low energy spectra of CR leptons. This is evident
from Fig. 4 where we compare the predictions of two different models which
both reproduce nuclear CR data:

  * •

a Kraichnan-like diffusion setup with $\delta=0.5$ and moderate reacceleration
(that was pointed out as the preferred one in a DRAGON-based maximum
likelihood analysis with focus on both B/C and antiproton high energy
data[16])

  * •

a Kolmogorov-like diffusion setup with $\delta=0.33$ and high reacceleration
(that was pointed out as the preferred one in a GALPROP-based maximum
likelihood analysis with focus on B/C data[17])

It is clear from that plot that the Kraichnan-like setups allows a better fit
of low-energy positron ratio measured by PAMELA; this consideration, together
with several other facts (high reacceleration models do not permit a good fit
of antiproton data and cannot reproduce the spectrum of the synchrotron
emission of the Galaxy), led us to conclude that models with strong
reacceleration are disfavoured.

##  4 High energy uncertainties and the nature of the extra-component

In the double component scenario discussed in Sec. 2, the positron spectrum
above $\sim 10$ GeV is dominated by the primary extra component. The nature of
its source is one of the hottest matter of debate in the CR physics.

Galactic pulsars were suggested as natural source candidates of a primary CR
positron component well before PAMELA results (Aharonian and Atoyan, 1995
[9].) More recently, it was noticed that a single, nearby, pulsar (such as
Monogem or Geminga) could explain the positrons fraction excess found by
PAMELA (Hooper et al. 2009 [10]).

In the Fermi-LAT era, we showed (Grasso et al. 2009 [5] and Di Bernardo et al.
2010 [6]) that also the $e^{+}+e^{-}$ measured by that experiment can
consistently be explained in the same terms: if one considers the observed
nearby pulsars within 2 kpc and assumes that a relevant fraction of their
rotational energy is transferred into $e^{+}+e^{-}$ pairs ($\simeq 30$%),
under reasonable assumpions on the injection spectrum and cutoff it is
possible to reproduce all existing data. In the cosmic ray channel, this
scenario has two possible testable consequences:

  * •

the detection of a CR electron anisotropy towards the most relevant sources
(in our analysis, Monogem and Geminga [10]);

  * •

the presence of some bumpiness in the $e^{-}$ and $e^{+}$ spectra in the TeV
region due to the contribution of several pulsars.

Those two signatures are somehow complementary: if a single pulsar give the
dominant contribution to the extra component a large anisotropy and a small
bumpiness should be expected; if several pulsars contribute the opposite
scenario is expected.

So far no positive detection of CRE anisotropy was reported by the Fermi-LAT
collaboration, but some stringent upper limits were published. In Di Bernardo
et al. 2010 [6] we showed that the pulsar scenario is still compatible with
these upper limits. Also, no evidence of spectral bumpiness has been found so
far in the $e^{+}+e^{-}$ spectrum.

It should be noted that several astrophysical uncertainties prevent accurate
predictions of the CRE anisotropy and of the spectral bumpiness. For example,
unknown irregularities in the local structure of the Galactic magnetic field
may distort the angular distribution of the CRE flux due to a nearby pulsar.
Furthermore, due to the stochastic nature of the $e^{-}$ emission of nearby
SNRs, the CRE standard component is expected to be subject to fluctuations
which may produce anisotropies and spectral bumpiness which may hide those due
to pulsars.

The other possible scenario to explain the origin of the extra component is
more exotic but very appealing as it invokes DM annihilihation/decay as the
origin of the $e^{\pm}$ extra component. Plenty of papers were published on
that subject after the release of PAMELA and Fermi-LAT results (see e.g. He
2009 [20] for a review). That scenario, however, present some problems. The
most important are the following ones.

  * •

It requires a heavy DM particle mass – O(TeV) – and an annihilation cross
section much higher than that predicted by standard cosmology if one assumes
that DM is a thermal relic.

  * •

Since no excess was detected for antiprotons, the annihilation/decay channels
must include only leptons (lepto-philic DM).

Although several DM models which may fulfil those conditions were developed,
another issue arises when electroweak corrections are taken into account.
Those corrections, in fact, give rise – even in a lepto-philic scenario – to
soft electroweak gauge bosons, and hence to antiprotons, at the end of their
decay chains (Ciafaloni et al. 2011 [21]). Since those exotic ${\bar{p}}$ are
produced mainly in the Galactic Center region, the flux reaching the Earth
strongly depends on the properties of CR propagation in the Galaxy. As we
discussed in Sec. 3, these properties are still subject to strong
uncertainties. It was shown in Evoli et al. 2011 [22] that, accounting for
those uncertainties, a scenario in which a heavy DM particle annihilates into
muons is still compatible with the antiproton constraints. In the same paper
it was also shown that AMS-02 is expected to constrain even more these models
since its sensitivity to antiprotons will be much higher.

##  5 Conclusions and future perspectives

In this contribution we argued as recent experimental data rule out the
standard scenario in which CR positrons are produced only by CR spallation
onto the ISM and showed as an empirical model which invokes an extra $e^{\pm}$
component fulfils all data sets. We also discussed several uncertainties which
still prevent to infer some of the properties of CR electron and positron
sources. We argued that at low energy those uncertainties are dominated by our
poor knowledge of CR propagation (which prevent an accurate determination of
the injection spectrum of the $e^{-}$ standard component) while at high energy
the effect of the stochastic nature of astrophysical sources prevails (which
makes more difficult to decide between the astrophysical and DM origin of the
extra component).

Forthcoming experiments like AMS-02 and CALET are expected to reduce
drastically the uncertainties on the propagation parameters by providing more
accurate measurements of the spectra of the nuclear components of CR. Fermi-
LAT and those experiments are also expected to provide more accurate
measurements of the CRE spectrum and anisotropy looking for features which may
give a clue of the nature of the extra component.

## References

  * [1] O. A. et al. [PAMELA collaboration], Nature 458, 607(April 2009).
  * [2] A. A. A. et al. [Fermi Collaboration], Physical Review Letters 102, p. 181101(May 2009).
  * [3] M. A. et al. [Fermi Collaboration], Physical Review D 82, p. 092004(November 2010).
  * [4] O. A. et al. [PAMELA collaboration], Physical Review Letters 106, p. 201101(May 2011).
  * [5] D. Grasso, S. Profumo, A. W. Strong, L. Baldini, R. Bellazzini, E. D. Bloom, J. Bregeon, G. di Bernardo, D. Gaggero, N. Giglietto, T. Kamae, L. Latronico, F. Longo, M. N. Mazziotta, A. A. Moiseev, A. Morselli, J. F. Ormes, M. Pesce-Rollins, M. Pohl, M. Razzano, C. Sgro, G. Spandre and T. E. Stephens, Astroparticle Physics 32, 140(September 2009).
  * [6] G. di Bernardo, C. Evoli, D. Gaggero, D. Grasso, L. Maccione and M. N. Mazziotta, Astroparticle Physics 34, 528(February 2011).
  * [7] P. D. Serpico, ArXiv e-prints (August 2011).
  * [8] O. A. et al. [PAMELA collaboration], Science 332, p. 69(April 2011).
  * [9] A. M. Atoyan, F. A. Aharonian and H. J. Völk, Physical Review D 52, 3265(September 1995).
  * [10] D. Hooper, P. Blasi and P. Dario Serpico, Journal of Cosmology and Astroparticle Physics 1, p. 25(January 2009).
  * [11] S. Profumo, ArXiv e-prints (December 2008).
  * [12] T. R. Jaffe, A. J. Banday, J. P. Leahy, S. Leach and A. W. Strong, Monthly Notices of the Royal Astronomical Society 416, 1152(September 2011).
  * [13] V. S. Berezinskii, S. V. Bulanov, V. A. Dogiel and V. S. Ptuskin, Astrophysics of cosmic rays 1990\.
  * [14] D. Maurin, F. Donato, R. Taillet and P. Salati, Astrophysical Journal 555, 585(July 2001).
  * [15] F. Donato, N. Fornengo, D. Maurin, P. Salati and R. Taillet, Physical Review D 69, p. 063501(March 2004).
  * [16] G. di Bernardo, C. Evoli, D. Gaggero, D. Grasso and L. Maccione, Astroparticle Physics 34, 274(December 2010).
  * [17] R. Trotta, G. Jóhannesson, I. V. Moskalenko, T. A. Porter, R. Ruiz de Austri and A. W. Strong, Astrophysical Journal 729, p. 106(March 2011).
  * [18] T. Delahaye, J. Lavalle, R. Lineros, F. Donato and N. Fornengo, Astronomy and Astrophysics 524, p. A51(December 2010).
  * [19] S. V. Bulanov and V. A. Dogel, Astrophysics and Space Science 29, 305(August 1974).
  * [20] X.-G. He, Modern Physics Letters A 24, 2139 (2009).
  * [21] P. Ciafaloni, D. Comelli, A. Riotto, F. Sala, A. Strumia and A. Urbano, Journal of Cosmology and Astroparticle Physics 3, p. 19(March 2011).
  * [22] C. Evoli, I. Cholis, D. Grasso, L. Maccione and P. Ullio, ArXiv e-prints (August 2011).


# MPG: A Multi-ingredient Pizza Image Generator with Conditional StyleGANs

Fangda Han
Rutgers University
Piscataway  NJ  USA
<EMAIL_ADDRESS>Guoyao Hao
Rutgers University
Piscataway  NJ  USA
<EMAIL_ADDRESS>Ricardo Guerrero
Samsung AI Center
Cambridge  UK
<EMAIL_ADDRESS>Vladimir Pavlovic
Rutgers University
Piscataway  NJ  USA
<EMAIL_ADDRESS>
###### Abstract

Multilabel conditional image generation is a challenging problem in computer
vision. In this work we propose Multi-ingredient Pizza Generator (MPG), a
conditional Generative Neural Network (GAN) framework for synthesizing
multilabel images. We design MPG based on a state-of-the-art GAN structure
called StyleGAN2, in which we develop a new conditioning technique by
enforcing intermediate feature maps to learn scalewise label information.
Because of the complex nature of the multilabel image generation problem, we
also regularize synthetic image by predicting the corresponding ingredients.
To verify the efficacy of MPG, we test it on Pizza10, which is a carefully
annotated multi-ingredient pizza image dataset. MPG can successfully generate
photo-realist pizza images with desired ingredients. The framework can be
easily extend to other multilabel image generation scenarios.

##  1 Introduction

Table 1: Glossary of terms. Term | Description
---|---
MPG | Multi-ingredient Pizza Generator
SLE | Scalewise Label Encoder
CR | Classification Regularizer
Pizza10 | Proposed multi-ingredient annotated pizza dataset

Generative Adversarial Networks [8] (GAN) are a type of generative models that
can create high-fidelity images [18, 17, 2] by alternatively optimizing its
two components: generator and discriminator. We exploit the capacity of GAN
models for solving the problem of generating food images conditioned on
multiple ingredients. Food images are naturally complex since ingredients
usually have different colors and shapes; even the same ingredient can have
different colors with varying methods of cooking (pepperoni will become darker
with longer cooking time or higher cooking temperature). Moreover, ingredients
in food images are usually mixed together, and therefore, the model has to
learn to arrange the ingredients correctly on an image.

Previous works on food image generation either use images as input [15, 29] or
generate blurry images [10]. We propose a conditional generative adversarial
network called Multi-ingredient Pizza Generator (MPG) inspired by [19]. MPG
first encodes the ingredients into different scale-specific embeddings,
injected into the synthesis network together with a “style noise” to create
images. These same ingredient embeddings are also injected into the
discriminator to guide the discrimination process.

Our main contributions are: (1) We propose a novel multilabel conditional GAN
framework that is based on StyleGAN2 [19]. This new framework incorporates a
Scalewise Label Encoder (SLE) and a Classification Regularizer (CR) that guide
the generator in the synthesis of the desired ingredients. (2) We create a new
dataset Pizza10 from pizzaGANdata [29], by relabiling using a subset of the
ingredient labels. After re-annotation, these labels are able to perform more
accurate in the multilabel classification and image retrieval tasks.

The reminder of this manuscript is organized as follows: Sec. 2 introduces the
related works including GAN, conditional GAN, food generation and multilabel
image generation. Sec. 3 presents our conditional StyleGAN framework MPG as
well as the Scalewise Label Encoder (SLE) and Classification Regularizer (CR).
Sec. 4 describes our motivation to create Pizza10 dataset and illustrates the
improvement from the dataset reframing process. Sec. 5 includes experiments
that verify the effectiveness of our framework, an ablation study, and an
assessment of generated images. Finally, in Sec. 6 we summarize our
conclusions.

##  2 Related Works

GANs [8] were proposed in 2014 and originally trained as a two-player min-max
game to estimate the true data distribution from samples. In the past few
years, GANs have evolved rapidly, improving their ability to generate high-
fidelity digital images of compact objects and natural scenes [25, 2, 17], as
well as human faces [18, 19]. As GAN training involves two neural networks
competing with each other, it is well-known to suffer from instability during
training, leading to failure to converge or mode collapse [23, 33]. Techniques
proposed to solve these problems include mini-batch normalization [33], R1
regularization [6, 31], Wasserstein loss [1, 9], Spectral Normalization [25]
adaptive augmentation [17], .

Conditional GAN [24] extends GAN by conditioning generation process with
auxiliary information to control the appearance of generated images,
controlling handwriting digit in MNIST [24], natural object or scene category
in ImageNet dataset [26], or the category of CIFAR10 dataset [17]. The work
proposed by [28] is the closest to our proposed framework. There, the authors
apply StyleGAN to the logo generation task. Our task differs substantially
from this as we seek to condition on more than a single label (ingredients);
thus, the structures and spatial relationships among labels are far more
diverse. Another line of research that uses conditional GANs, aims at
conditioning directly with an input image, with the goal of changing its
attributes [12, 3] or associated styles [14, 41]. However, it is important to
note that the task of our proposed work is more complicated, as we only have
labels as the conditioning input, lacking the input image as a strong prior.
Finally, image GANs can be conditioned directly on natural language textual
descriptions [40, 39, 22]. While, in principle, more general, these approaches
in practice work well on objects such as birds or flowers that share similar
structures within classes, but tend to fail on complex scene generation, like
those in COCO Dataset.

Food Image Generation is usually accomplished using a conditional GAN to
attain higher fidelity compared with other generative models like VAEs [21] or
Flow [20, 4, 5]. [32] concatenates the embedding from a pretrained text
encoder with noise to generated text-based food image. [7] also uses the
embedding of a pretrained text encoder but applies the more advanced StackGAN2
[40] to generate food image. [10] also applies StackGAN2 but it regularizes
the generated image to retrieve the corresponding text. [37] applies
ProgressiveGAN [16] to unsupervised food image generation on four different
datasets. However, these works typically result in blurry, low-resolution
images with missing details; in contrast, our framework can create images of
high fidelity and resolution. [15, 29] apply GANs to transform the food type
or add/remove ingredients of a source image. This improves the quality of
generated images but requires image-based conditioning; our framework uses
labels as input, a significantly weaker form of bias.

Multilabel Image Generation is a special case of text-based image synthesis.
[40] uses a stacked multi-scale generator to synthesize images based on text
embeddings, [39] extends [40] and synthesize details at different subregions
of the image by paying attentions to the relevant words in the natural
language description. [30] extends [39] by regularizing the generator and
redescribing the corresponding text from the synthetic image. While free text
is a potentially stronger signal, it is often corrupted by image-unrelated
signals, such as the writing style. On the other hand, while image labels may
appear more specific than free-text, the visual diversity of objects described
by those labels, such as food ingredients, and the changes in appearance that
can arise from interactions of those objects can make the multilabel image
generation problem an equal if not greater challenge to that of the caption-
to-image task.

##  3 Methodology

Figure 1: Generator components overview. Left: Scalewise Label Encoder (SLE),
each scale has its own layers for encoding. Middle: Mapping Network. Right:
Synthesis Network, $\oplus$ means concatenation, notice images at different
scales are conditioned with different label embedding Figure 2: Left:
Discriminator structure, which contains one branch for conditional output and
another branch for unconditional output, notice the label embedding
$\\{\operatorname{\mathbf{t}}_{i}\\}$ are reversed to match different scales.
Top right: Generator Loss, consisting discriminator loss plus Classification
Regularizer (CR) loss for fake image. Bottom right: Discriminator Loss,
consisting three discriminator losses from fake, real and wrong images. Notice
the discriminator is trained to distinguish between (txt, real img) and (txt,
wrong img) Figure 3: Left: Injection block at scale i in the generator (image
courtesy [19]). A is a learned affine transformation, ‘weight’ is the learned
weight for the deconv layer which is modulated by
$\operatorname{\mathbf{t}}_{i}\oplus\operatorname{\mathbf{w}}$ by Mod and
Demod, refer [19] for more details. Right: Injection block at scale i in the
discriminator. The same injection idea is applied to the discriminator to
inject the label embedding at each scale, here conv layer is used instead of
the deconv layer to downsample the feature map

To generate pizza images of high-fidelity, we start from StyleGAN2 [19], one
of the state-of-the-art GAN architecture. StyleGAN2 is composed of a generator
(Fig. 1) and a discriminator (Fig. 2 Left) just like the original GAN, but can
generate high-fidelity human faces and objects of up to $1024^{2}$ resolution.
There are several techniques involved to improve image quality. (1) The image
is generated progressively. Lower-resolution image is generated first and then
upsampled and combined with higher-resolution image. (2) There are two sources
of noise as the input of the generator, one controls the style and the other
controls the detail. The style noise ($z\in\mathcal{N}(0,I)$ in Fig. 1 Middle)
and the detail noise (omitted in Fig. 1 because it is irrelevant to our work)
are injected into the generator at each scale. The style noise is injected by
first passing through a mapping network (Fig. 1 Middle) and then transformed
to be the weights of the convolutional layers in the generator, therefore
achieving a strong control of the image, while the detail noise is injected
directly by adding into the output feature map of each convolutional layer as
a perturbation. (3) The discriminator is trained with mini-batch normalization
[33] to avoid mode collapse of the generator. (4) R1 regularization [31] is
applied to improve the robustness of the network.

StyleGAN2 alone has a limited way of controlling the output image. The authors
show that changing the style noise $\operatorname{\mathbf{z}}$ will result in
a shift in human face features (face outline, skin color, gender ). However,
given the style noise’s random nature, it is impossible to determine or
control specific features in the generated image. In this paper we adjust
StyleGAN2 and create Multi-ingredient Pizza Generator (MPG), making it
possible to generate pizza images with specific, controllable ingredients and
combinations thereof.

Formally, given a binary vector $\operatorname{\mathbf{x}}$ indicating the
ingredient list (Fig. 1 Left), we seek to generate a pizza image
$\operatorname{\mathbf{y}}$ that contains the desired ingredients. We propose
to use a model endowed with Scalewise Label Encoder, Classification
Regularizer to solve this problem.

###  3.1 Scalewise Label Encoder

The generator in StyleGAN2 is injected with the style noise
$\operatorname{\mathbf{z}}$ at each scale. Inspired by this, we design SLE
(Scalewise Label Encoder in Fig. 1 Left) to condition the synthesis network on
the binary vector $\operatorname{\mathbf{x}}$. SLE consists of a sequence of
sub-encoders $\\{Enc_{i}(\operatorname{\mathbf{x}})\\}$ (one for each scale
$i$) and takes $\operatorname{\mathbf{x}}$ as input to output label embedding
$\\{\operatorname{\mathbf{t}}_{i}\\}$ (also one for each scale $i$). Formally,
if the generated image $\tilde{\operatorname{\mathbf{y}}}$ is of size
$256^{2}$, then we get scalewise label embedding
$\\{\operatorname{\mathbf{t}}_{i}\\}$

$\displaystyle\\{\operatorname{\mathbf{t}}_{i}\\}=SLE(x)=\\{Enc_{i}(\operatorname{\mathbf{x}})\\},$
(1)

where $i\in\\{4,8,16,32,64,128,256\\}$. $Enc_{i}$ is the encoder at $i-th$
scale. We implement $Enc_{i}$ by multi-layer perceptron. The mapping network
$f$ is kept to provide more diversity to the output image. The label embedding
$\\{\operatorname{\mathbf{t}}_{i}\\}$ is then concatenated with the output
from the mapping network together as the input of the synthesis network $S$.
We inject $\operatorname{\mathbf{t}}_{i}\oplus\operatorname{\mathbf{w}}$ the
same way as in StyleGAN2 (Fig. 3 Left) and generate the output image
$\tilde{\operatorname{\mathbf{y}}}$

$\displaystyle\operatorname{\mathbf{w}}$
$\displaystyle=f(\operatorname{\mathbf{z}})$ (2)
$\displaystyle\tilde{\operatorname{\mathbf{y}}}$
$\displaystyle=S\left(\\{\operatorname{\mathbf{w}},\operatorname{\mathbf{t}}_{i}\\}\right).$
(3)

Eq. 1 to Eq. 3 can be further simplified as

$\displaystyle\tilde{\operatorname{\mathbf{y}}}=G(\operatorname{\mathbf{x}},\operatorname{\mathbf{z}}),$
(4)

where G is the generator composed of $\\{SLE,f,S\\}$.

SLE outputs are applied to the discriminator as well, since the discriminator
is composed of a sequence of convolutional layers that downsample the input
image from high resolution to low resolution, we reverse
$\\{\operatorname{\mathbf{t}}_{i}\\}$ and inject each
$\operatorname{\mathbf{t}}_{i}$ without concatenating
$\operatorname{\mathbf{w}}$ at each scale the same way as the generator (Fig.
3 Right). However, during experiments we notice replacing the StyleGAN2
discriminator with the conditional version directly will lead to degeneration
in image quality. Inspired by StackGAN2 [40], we extract the convolutional
layers in StyleGAN2 discriminator and add it into the discriminator as the
“unconditional branch” (blue dash box in Fig. 2 Left).

Formally, the discriminator $D$ is combined with a conditional branch $f_{c}$,
an unconditional branch $f_{uc}$, a mini-batch normalization operation
$\sigma$ and a fully-connected layer $F$. Given the label embedding
$\\{\operatorname{\mathbf{t}}_{i}\\}$ and input image
$\operatorname{\mathbf{y}}$, the outputs can be represented as

$\displaystyle s_{c}$
$\displaystyle=D\left(\\{\operatorname{\mathbf{t}}_{i}\\},\operatorname{\mathbf{y}}\right)=F\left(\sigma\left(f_{c}(\\{\operatorname{\mathbf{t}}_{i}\\},\operatorname{\mathbf{y}})\right)\right)$
(5) $\displaystyle s_{uc}$
$\displaystyle=D\left(\operatorname{\mathbf{y}}\right)=F\left(\sigma\left(f_{uc}(\operatorname{\mathbf{y}})\right)\right),$
(6)

where $s_{c}$ and $s_{uc}$ are conditional output and unconditional output
respectively, $i\in\\{256,128,64,32,16,8,4\\}$. Notice mini-batch
normalization is used to correlate samples in a mini-batch in the
discriminator to prevent model collapse [33].

Now given a pair of ingredient list and image
$(\operatorname{\mathbf{x}},\operatorname{\mathbf{y}})$, the loss function for
the generator can be defined as

$\displaystyle\max_{G}=D(\\{\operatorname{\mathbf{t}}_{i}\\},G(\operatorname{\mathbf{x}},\operatorname{\mathbf{z}}))+\lambda_{uncond}D(G(\operatorname{\mathbf{x}},\operatorname{\mathbf{z}})),$
(7)

and the loss for the discriminator is

$\displaystyle\begin{split}\min_{D}&=D(\\{\operatorname{\mathbf{t}}_{i}\\},G(\operatorname{\mathbf{x}},\operatorname{\mathbf{z}}))+\lambda_{uncond}D(G(\operatorname{\mathbf{x}},\operatorname{\mathbf{z}}))\\\
&-D(\\{\operatorname{\mathbf{t}}_{i}\\},\operatorname{\mathbf{y}})-\lambda_{uncond}D(\operatorname{\mathbf{y}}),\end{split}$
(8)

where $\\{\operatorname{\mathbf{t}}_{i}\\}=SLE(x)$,
$\operatorname{\mathbf{z}}\in\mathcal{N}(0,I)$, and $\lambda_{uncond}$ is the
weight for unconditional output of the discriminator.

###  3.2 Classification Regularizer

To help the generator create image with desired ingredients, inspired by [27,
10], we pretrain a multilabel classifier $h$ on pairs of real image and its
ingredient list $\\{(\operatorname{\mathbf{y}},\operatorname{\mathbf{x}})\\}$.
Then during training the generator, we regularize the generated image
$\tilde{\operatorname{\mathbf{y}}}$ to predict the correct ingredients
$\operatorname{\mathbf{x}}$. The Classification Regularizer (CR) can be
formalized as

$\displaystyle\tilde{\operatorname{\mathbf{x}}}=h(\tilde{\operatorname{\mathbf{y}}}),$
(9)

this regularizer can be easily combined in the generator loss Eq. 7 as shown
in top right of Fig. 2,

$\displaystyle\begin{split}\max_{G}&=D(\\{\operatorname{\mathbf{t}}_{i}\\},G(\operatorname{\mathbf{x}},\operatorname{\mathbf{z}}))+\lambda_{c}D(G(\operatorname{\mathbf{x}},\operatorname{\mathbf{z}}))\\\
&+\lambda_{clf}BCE\left(\operatorname{\mathbf{x}},h(\tilde{\operatorname{\mathbf{y}}})\right),\end{split}$
(10)

where $BCE$ is binary cross entropy loss and $\lambda_{clf}$ is the weight for
the regularizer.

Except the contents mentioned above, R1 regularization [6] is also kept in the
discriminator to penalize the gradient flowed from the image to the output,
which has been proved to be helpful in stablize GAN training process [31]. The
final discriminator loss is now

$\displaystyle\begin{split}\min_{D}&=D(\\{\operatorname{\mathbf{t}}_{i}\\},G(\operatorname{\mathbf{x}},\operatorname{\mathbf{z}}))+\lambda_{uncond}D(G(\operatorname{\mathbf{x}},\operatorname{\mathbf{z}}))\\\
&-D(\\{\operatorname{\mathbf{t}}_{i}\\},\operatorname{\mathbf{y}})-\lambda_{uncond}D(\operatorname{\mathbf{y}})\\\
&+\lambda_{r1}\left(\dfrac{\partial
D(\operatorname{\mathbf{x}})}{\partial\operatorname{\mathbf{x}}}+\dfrac{\partial
D(\operatorname{\mathbf{x}},\\{\operatorname{\mathbf{t}}_{i}\\})}{\partial\operatorname{\mathbf{x}}}\right).\end{split}$
(11)

Eq. 10 and Eq. 11 summarize all components in the proposed framework MPG. In
the following sections, we will use Pizza10 dataset to verify the efficacy of
MPG.

##  4 Pizza10 Dataset

Figure 4: Ingredients distribution comparison between pizzaGANdata and the
refined Pizza10 dataset, ‘[empty]’ represents the number of samples with no
ingredients labeled Figure 5: Illustration of improper labels in pizzaGANdata
and the refined labels in Pizza10 dataset, the difference is highlighted in
red (best viewed in color)

The performance of neural network models critically depends on the quality of
the dataset used to train the models. We begin with the pizzaGANdata [29],
which contains $9213$ pizza images annotated with 13 ingredients, as shown in
Fig. 4. We manually inspected the labeling in this dataset and found a
significant portion of it to be mislabeled; we illustrate three instances in
Fig. 5. To address this issue, using a custom-created collaborative web
platform, we relabeled the dataset and removed three ingredients (Spinach,
Pineapple, Broccoli) with fewer number of instances (Corn is kept since the
distinct color and shape makes it a recognizable ingredient even with fewer
samples than Spinach). The refined dataset, Pizza10, decreases the number of
‘empty samples’ (pizza images with no ingredients labeled) by $40\%$ and
increases the number of samples for each ingredient category as shown in Fig.
4. We verify the impact of the curated Pizza10 on two tasks:

Retrieval. Given an ingredient list as query, we want to retrieve its
associated pizza image, we solve this task by following [34]. The ingredient
list is treated as a sentence and encoded into a vector of $\mathbb{R}^{768}$,
the text encoder is two-layer-two-head transformer encoder [35] built using
HuggingFace[38], and we initialized the embedding from the pre-trained ‘bert-
base-uncased’ model on HuggingFace Model Hub. The image is also encoded into a
vector of $\mathbb{R}^{768}$, the image encoder is initialized from ResNet50
[11] with the final fully-connected layer changed to match the output
dimension ($\mathbb{R}^{768}$). The loss is triplet loss with hard-sample
mining [36]. We use $60\%$ data for training and leave $40\%$ for testing.
After training, we use the median rank (MedR) [34] to measure the performance;
the retrieval range is $280$, roughly the number of distinct ingredient lists
in both datasets. Computed as the median rank of the true positives over all
queries, a lower MedR ($\geq 1.0$) suggests better performance.

Multilabel Classification. Given a pizza image, we seek to predict its
ingredients. We initialized the classifier from ResNet50 [11] with the final
fully-connected layer changed to match the output dimension ($\mathbb{R}^{13}$
for pizzaGANdata and $\mathbb{R}^{10}$ for Pizza10). The loss is a binary
cross entropy loss. $80\%$ data are used in training while $20\%$ data are for
testing. After training, we use the mean average precision (mAP) as the
metric.

The results are shown in Fig. 6. The same model has a big jump in performance
on the refined Pizza10 dataset in both tasks. Upon close inspection, the
average precision (AP) for each ingredient increases on most ingredients
except Corn for the classifier trained on Pizza10, therefore demonstrating the
importance of careful curating.

Figure 6: Performance comparison between pizzaGANdata and Pizza10. Top left:
MedR scores on retrieval task. Top right: mAP scores on multilabel
classification task. Bottom: average precision for each ingredient in the
classification task. ‘Rnd Pick’ means the scores when we randomly pick answer.
$\downarrow$ means the lower the better, $\uparrow$ means the higher the
better Figure 7: Qualitative comparison between our model MPG and baselines

##  5 Experiments

Network Structures. For the generator, SLE is composed of a group of sub-
encoders, each sub-encoder is a fully-connected layer with ReLU activation.
The label embedding dimension $T=256$, the mapping network is the same as
StyleGAN2 except we decrease the style noise dimension $L$ to $256$. The
synthesized network is also the same as StyleGAN2 however the input at each
scale is now different for matching the scalewise label embedding
$\\{\operatorname{\mathbf{t}}_{i}\\}$. For the discriminator, the
unconditional branch, mini-batch normalization and fully-connected layer are
the same as StyleGAN2, the conditional branch is modified from the injection
block of the generator and performs the reversed operation (injection and
downsampling).

Hyper Parameters. For the generator, $\lambda_{uncond}=1.0$. For the
discriminator, $\lambda_{clf}=1.0$, $\lambda_{r1}=10$, R1 regularization is
performed every 16 mini-batches to balance the training time. The learning
rate for both the generator and discriminator is $0.002$. The mini-batch size
is $24$ (need at least 24GB GPU memory).

Configurations. The framework is built using PyTorch 1.7 and trained on four
NVIDIA K80 GPUs (each has 12 GB memory), the training takes 6 days to converge
for a duration of 2.4M pizza images. Note the number of images needed is much
smaller than the StyleGAN model trained on LSUN BEDROOMS, CATS or CARS (see
Appendix E in [18]), we believe this is because our dataset is much smaller
($9213$ images in Pizza10 vs. 3M in LSUN BEDROOMS).

Metrics. We use FID [13] (on 10K images) to assess to the quality of the
generated images, a lower FID indicates the generated image distribution is
more close to the real image distribution. We use mean average precision [42]
($mAP$) to evaluate the conditioning on the desired ingredients. The
multilabel classifier we used is trained on Pizza10 as shown in Sec. 4 and
achieves $mAP=0.6804$.

###  5.1 Comparison with Baselines

We compare with three prior works on text-based image generation. StackGAN2
[40] first extracts text feature from a pretrained retrieval model and then
forward through a stacked generator with separate discriminators working at
different scales. CookGAN [10] uses similar structure as of StackGAN2 and adds
a cycle-consistent loss which drives the generated images to retrieve its
desired ingredients. AttnGAN [39] also shares StackGAN2-like structure, it
uses pretrained text encoder to capture the importance of each word and
leverages both word-level and sentence-level feature to guide the generation
process.

Table 2: Quantitative comparison of performances between baselines and the proposed multi-ingredient Pizza Generator (MPG) Models | Image Size | FID$\downarrow$ | mAP$\uparrow$
---|---|---|---
StackGAN2 [40] | $256^{2}$ | $63.51$ | $0.3219$
CookGAN [10] | $256^{2}$ | $45.64$ | $0.2896$
AttnGAN [39] | $256^{2}$ | $74.47$ | $0.5729$
MPG | $256^{2}$ | $\mathbf{8.43}$ | $\mathbf{0.9816}$
Figure 8: Images generated from different combinations of ingredient list and
style noise. Images in the same row are generated with identical style noise.

Tab. 2 compares the performance between our model MPG and the baselines, we
can see MPG has a huge leap forward compared with previous work both in terms
of FID and mAP, which means MPG not only generates realistic images with
diversity but also improves the conditioning on the desired ingredients.

One thing needs to mention is FID has a lower bound which can be estimated
from real images, say if we randomly split the real images of Pizza10 into two
image sets (about 5K images each set) and compute the FID between the two
sets, we get FID$\approx 8.20$, which means our model still has space to
improve. Another thing to note is that mAP is much higher on the generated
images from MPG even compared with that on real images ($0.6804$), this sounds
unreasonable but we believe since our model is trained with limited data, the
images generated is good at capturing the main characteristic of each
ingredient and incapable of understanding all possible diversities, while real
images could be even harder for the classifier since they contain more
variety.

Table 3: Quantitative comparison of performances between MPG and its counterparts with missing components Models | FID$\downarrow$ | mAP$\uparrow$
---|---|---
MPG-mapping | $9.47$ | $0.9799$
MPG-SLE | $9.29$ | $0.9187$
MPG-SLE * | $10.03$ | $0.9276$
MPG-CR | $9.85$ | $0.4712$
MPG-uncond | $22.26$ | 0.9856
MPG | 8.43 | $0.9816$

Fig. 7 shows a few samples from our model MPG as well as the baselines
conditioned on certain ingredient lists. Images generated by MPG are more
appealing and contain the desired ingredients compared with baselines. Notice
how MPG can effectively adds Pepperoni, Mushroom, Fresh basil and Black olives
one after another into the generated images.

###  5.2 Ablation Study

To effectively verify the design of MPG, we bring out the following
experiments. MPG-mapping removes the mapping network $f$ and directly
concatenates scalewise label embedding $\\{\operatorname{\mathbf{t}}_{i}\\}$
with style noise $z$ as the input of the synthesis network. MPG-SLE removes
Scalewise Label Encoder and use the same label embedding for all resolutions.
MPG-SLE * also removes SLE and uses the same label embedding for all
resolutions, but the label embedding is concatenated with the style noise
$\operatorname{\mathbf{z}}$ before passing through the mapping network,
instead of concatenating with $w$ after passing the mapping network. MPG-CR
removes Classification Regularizer. MPG-uncond removes the unconditional
branch in the discriminator.

Tab. 3 shows the performances of all models, we can observe that mapping
network (MPG-mapping) and unconditional branch in discriminator (MPG-uncond)
have impact on FID. Classification Regularizer (MPG-CR) mainly affects mAP.
And SLE influences both FID and mAP.

###  5.3 Assessment of Generated Images

We provide additional analyses of images created by MPG. We first
qualitatively study the independence between text embedding and style noise,
then we assess the smoothness of the text and the style noise spaces.

Independence. To analyze the independence between text embedding and style
noise, we fix the style noise while generating images with different
ingredient lists. The result is shown in Fig. 8. Taking the camera position of
the images from Noise 1 as the anchor, Noise 2 provides a zoom-in top view,
Noise 3 translates the camera position along x-axis, and Noise 4 offers a
scaled side view of the pizzas. The likeness of view points in each row and
the consistency of ingredients in each column demonstrate the independence
between text embedding and style noise.

Smoothness. We verify the smoothness of the learned embedding space and the
style noise space by traversing [8]. In Fig. 9, we randomly select two
ingredient lists $\operatorname{\mathbf{x}}_{1}$ and
$\operatorname{\mathbf{x}}_{2}$, forward them through SLE and obtain the
corresponding text embeddings $\\{\operatorname{\mathbf{t}}_{i}\\}_{1}$ and
$\\{\operatorname{\mathbf{t}}_{i}\\}_{2}$; we then sample two style noise
$\operatorname{\mathbf{z}}_{1}$ and $\operatorname{\mathbf{z}}_{2}$. The
traversing is conducted by linear interpolation
($\\{\operatorname{\mathbf{t}}_{i}\\}_{1}\rightarrow\\{\operatorname{\mathbf{t}}_{i}\\}_{2}$
and $\operatorname{\mathbf{z}}_{1}\rightarrow\operatorname{\mathbf{z}}_{2}$).

Figure 9: Illustration of traversing through the text embedding space and the
style noise space. Images in each row are generated with the same text
embedding (interpolated between $\\{\operatorname{\mathbf{t}}_{i}\\}_{1}$ and
$\\{\operatorname{\mathbf{t}}_{i}\\}_{2}$), while images in each column are
synthesized with the same style noise (interpolated between
$\operatorname{\mathbf{z}}_{1}$ and $\operatorname{\mathbf{z}}_{2}$).

We can observe the image generated with (Corn, Pepperoni) in the top left
corner gradually changes to the images containing (Fresh basil, Black olives)
along the vertical axis, while the view angle and coloring progressively
update along the horizontal axis, from $\operatorname{\mathbf{z}}_{1}$ to
$\operatorname{\mathbf{z}}_{2}$. This demonstrates the ability of MPG to
learn, then recreate independent and smooth major factors of variation in this
complex image space.

##  6 Conclusion

We propose MPG, a Multi-ingredient Pizza Generator image synthesis framework,
supported by a new Pizza10 dataset, that can effectively learn to create
photo-realistic pizza images from combinations of specified ingredients, as
well as manipulate the view point by independently changing the style noise.
While extremely effective in visual appearance, the model remains
computationally expensive to train and with some loss of detail at fine
resolutions, the problems we aim to investigated in future research.

## References

  * [1] Martin Arjovsky, Soumith Chintala, and Léon Bottou.  Wasserstein gan.  arXiv preprint arXiv:1701.07875, 2017.
  * [2] Andrew Brock, Jeff Donahue, and Karen Simonyan.  Large scale gan training for high fidelity natural image synthesis.  arXiv preprint arXiv:1809.11096, 2018.
  * [3] Yunjey Choi, Minje Choi, Munyoung Kim, Jung-Woo Ha, Sunghun Kim, and Jaegul Choo.  Stargan: Unified generative adversarial networks for multi-domain image-to-image translation.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 8789–8797, 2018.
  * [4] Laurent Dinh, David Krueger, and Yoshua Bengio.  Nice: Non-linear independent components estimation.  arXiv preprint arXiv:1410.8516, 2014.
  * [5] Laurent Dinh, Jascha Sohl-Dickstein, and Samy Bengio.  Density estimation using real nvp.  arXiv preprint arXiv:1605.08803, 2016.
  * [6] Harris Drucker and Yann Le Cun.  Improving generalization performance using double backpropagation.  IEEE Transactions on Neural Networks, 3(6):991–997, 1992.
  * [7] Ori Bar El, Ori Licht, and Netanel Yosephian.  Gilt: Generating images from long text.  arXiv preprint arXiv:1901.02404, 2019.
  * [8] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio.  Generative adversarial nets.  In Advances in neural information processing systems, pages 2672–2680, 2014.
  * [9] Ishaan Gulrajani, Faruk Ahmed, Martin Arjovsky, Vincent Dumoulin, and Aaron C Courville.  Improved training of wasserstein gans.  In Advances in neural information processing systems, pages 5767–5777, 2017.
  * [10] Fangda Han, Ricardo Guerrero, and Vladimir Pavlovic.  Cookgan: Meal image synthesis from ingredients.  In The IEEE Winter Conference on Applications of Computer Vision, pages 1450–1458, 2020.
  * [11] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770–778, 2016.
  * [12] Zhenliang He, Wangmeng Zuo, Meina Kan, Shiguang Shan, and Xilin Chen.  Attgan: Facial attribute editing by only changing what you want.  IEEE Transactions on Image Processing, 28(11):5464–5478, 2019.
  * [13] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter.  Gans trained by a two time-scale update rule converge to a local nash equilibrium.  In Advances in neural information processing systems, pages 6626–6637, 2017.
  * [14] Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, and Alexei A Efros.  Image-to-image translation with conditional adversarial networks.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1125–1134, 2017.
  * [15] Yoshifumi Ito, Wataru Shimoda, and Keiji Yanai.  Food image generation using a large amount of food images with conditional gan: ramengan and recipegan.  In Proceedings of the Joint Workshop on Multimedia for Cooking and Eating Activities and Multimedia Assisted Dietary Management, pages 71–74, 2018.
  * [16] Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen.  Progressive growing of gans for improved quality, stability, and variation.  arXiv preprint arXiv:1710.10196, 2017.
  * [17] Tero Karras, Miika Aittala, Janne Hellsten, Samuli Laine, Jaakko Lehtinen, and Timo Aila.  Training generative adversarial networks with limited data.  Advances in Neural Information Processing Systems, 33, 2020.
  * [18] Tero Karras, Samuli Laine, and Timo Aila.  A style-based generator architecture for generative adversarial networks.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4401–4410, 2019.
  * [19] Tero Karras, Samuli Laine, Miika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila.  Analyzing and improving the image quality of stylegan.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8110–8119, 2020.
  * [20] Durk P Kingma and Prafulla Dhariwal.  Glow: Generative flow with invertible 1x1 convolutions.  In Advances in neural information processing systems, pages 10215–10224, 2018.
  * [21] Diederik P Kingma and Max Welling.  Auto-encoding variational bayes.  arXiv preprint arXiv:1312.6114, 2013.
  * [22] Bowen Li, Xiaojuan Qi, Thomas Lukasiewicz, and Philip Torr.  Controllable text-to-image generation.  In Advances in Neural Information Processing Systems, pages 2065–2075, 2019.
  * [23] Lars Mescheder, Andreas Geiger, and Sebastian Nowozin.  Which training methods for gans do actually converge?  arXiv preprint arXiv:1801.04406, 2018.
  * [24] Mehdi Mirza and Simon Osindero.  Conditional generative adversarial nets.  arXiv preprint arXiv:1411.1784, 2014.
  * [25] Takeru Miyato, Toshiki Kataoka, Masanori Koyama, and Yuichi Yoshida.  Spectral normalization for generative adversarial networks.  arXiv preprint arXiv:1802.05957, 2018.
  * [26] Takeru Miyato and Masanori Koyama.  cgans with projection discriminator.  arXiv preprint arXiv:1802.05637, 2018.
  * [27] Augustus Odena, Christopher Olah, and Jonathon Shlens.  Conditional image synthesis with auxiliary classifier gans.  In International conference on machine learning, pages 2642–2651, 2017.
  * [28] Cedric Oeldorf and Gerasimos Spanakis.  Loganv2: Conditional style-based logo generation with generative adversarial networks.  In 2019 18th IEEE International Conference On Machine Learning And Applications (ICMLA), pages 462–468. IEEE, 2019.
  * [29] Dim P. Papadopoulos, Youssef Tamaazousti, Ferda Ofli, Ingmar Weber, and Antonio Torralba.  How to make a pizza: Learning a compositional layer-based gan model.  In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2019.
  * [30] Tingting Qiao, Jing Zhang, Duanqing Xu, and Dacheng Tao.  Mirrorgan: Learning text-to-image generation by redescription.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 1505–1514, 2019.
  * [31] Andrew Slavin Ross and Finale Doshi-Velez.  Improving the adversarial robustness and interpretability of deep neural networks by regularizing their input gradients.  arXiv preprint arXiv:1711.09404, 2017.
  * [32] Mark Sabini, Zahra Abdullah, and Darrith Phan.  Gan-stronomy: Generative cooking with conditional dcgans.
  * [33] Tim Salimans, Ian Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen.  Improved techniques for training gans.  In Advances in neural information processing systems, pages 2234–2242, 2016.
  * [34] Amaia Salvador, Nicholas Hynes, Yusuf Aytar, Javier Marin, Ferda Ofli, Ingmar Weber, and Antonio Torralba.  Learning cross-modal embeddings for cooking recipes and food images.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3020–3028, 2017.
  * [35] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  In Advances in neural information processing systems, pages 5998–6008, 2017.
  * [36] Hao Wang, Doyen Sahoo, Chenghao Liu, Ee-peng Lim, and Steven CH Hoi.  Learning cross-modal embeddings with adversarial networks for cooking recipes and food images.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 11572–11581, 2019.
  * [37] Su Wang, Honghao Gao, Yonghua Zhu, Weilin Zhang, and Yihai Chen.  A food dish image generation framework based on progressive growing gans.  In International Conference on Collaborative Computing: Networking, Applications and Worksharing, pages 323–333. Springer, 2019.
  * [38] Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, et al.  Huggingface’s transformers: State-of-the-art natural language processing.  ArXiv, pages arXiv–1910, 2019.
  * [39] Tao Xu, Pengchuan Zhang, Qiuyuan Huang, Han Zhang, Zhe Gan, Xiaolei Huang, and Xiaodong He.  Attngan: Fine-grained text to image generation with attentional generative adversarial networks.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1316–1324, 2018.
  * [40] Han Zhang, Tao Xu, Hongsheng Li, Shaoting Zhang, Xiaogang Wang, Xiaolei Huang, and Dimitris N Metaxas.  Stackgan++: Realistic image synthesis with stacked generative adversarial networks.  IEEE transactions on pattern analysis and machine intelligence, 41(8):1947–1962, 2018.
  * [41] Jun-Yan Zhu, Taesung Park, Phillip Isola, and Alexei A Efros.  Unpaired image-to-image translation using cycle-consistent adversarial networks.  In Proceedings of the IEEE international conference on computer vision, pages 2223–2232, 2017.
  * [42] Mu Zhu.  Recall, precision and average precision.  Department of Statistics and Actuarial Science, University of Waterloo, Waterloo, 2:30, 2004.

##  Appendix A Video

In the video111https://youtu.be/x3XKXMd1oC8, we show two examples to
demonstrate the main features of our proposed Multi-ingredients Pizza Image
Generation (MPG) framework. The first example is an interactive web app we
built that shows the images generated from arbitrary selected ingredients and
style noise. See Fig. 10 for details about the app layout.

Figure 10: Web app for synthesis of photo-realistic pizza images using the
proposed MPG framework. Oval legend blocks explain the main elements used in
the demo video.

The second example, showcased in Fig. 11, is another interactive web app we
designed to explore the smoothness of the learned ingredient representation
space. We encourage the reviewers to watch the included video.

Figure 11: Web app for synthesis of photo-realistic pizza images using the
interpolation approach. Image in the middle is created by interpolating the
content and the style of used to create the images A and B. Oval legend blocks
explain the main elements used in the demo video.

##  Appendix B Additional Visual Comparisons

We provide additional samples from our proposed MPG and other state-of-the-art
models in Figures 12 through 18. They illustrate:

  * •

Fig. 12 \- Fig. 15: Example images generated using our MPG under different
diversity levels.

  * •

Fig. 16: Example images generated using StackGAN2 [40].

  * •

Fig. 17: Example images generated using CookGAN [10].

  * •

Fig. 18: Example images generated using AttnGAN [39].

Figure 12: Images generated using our MPG, ingredients and style noise are
randomly sampled. The ingredients are shown on top left corner in each image.
The diversity level (truncation) is set to be $1.00$. Please check [18] for
how to accomplish this using the truncation trick. Figure 13: Images generated
using our MPG, ingredients and style noise are randomly sampled. The
ingredients are shown on top left corner in each image. The diversity level
(truncation) is set to be $0.75$. Please check [18] for how to accomplish this
using the truncation trick. Figure 14: Images generated using our MPG,
ingredients and style noise are randomly sampled. The ingredients are shown on
top left corner in each image. The diversity level (truncation) is set to be
$0.50$. Please check [18] for how to accomplish this using the truncation
trick. Figure 15: Images generated using our MPG, ingredients and style noise
are randomly sampled. The ingredients are shown on top left corner in each
image. The diversity level (truncation) is set to be $0.25$. Please check [18]
for how to accomplish this using the truncation trick. Figure 16: Images
generated using StackGAN2 [40], ingredients and style noise are randomly
sampled. The ingredients are shown on top left corner in each image. Figure
17: Images generated using CookGAN [10], ingredients and style noise are
randomly sampled. The ingredients are shown on top left corner in each image.
Figure 18: Images generated using AttnGAN [39], ingredients and style noise
are randomly sampled. The ingredients are shown on top left corner in each
image.