# Ensembling Off-the-shelf Models for GAN Training

Nupur Kumari 1 Richard Zhang2 Eli Shechtman2 Jun-Yan Zhu1
1Carnegie Mellon University 2Adobe

###### Abstract

The advent of large-scale training has produced a cornucopia of powerful
visual recognition models. However, generative models, such as GANs, have
traditionally been trained from scratch in an unsupervised manner. Can the
collective “knowledge” from a large bank of pretrained vision models be
leveraged to improve GAN training? If so, with so many models to choose from,
which one(s) should be selected, and in what manner are they most effective?
We find that pretrained computer vision models can significantly improve
performance when used in an ensemble of discriminators. Notably, the
particular subset of selected models greatly affects performance. We propose
an effective selection mechanism, by probing the linear separability between
real and fake samples in pretrained model embeddings, choosing the most
accurate model, and progressively adding it to the discriminator ensemble.
Interestingly, our method can improve GAN training in both limited data and
large-scale settings. Given only 10k training samples, our FID on LSUN Cat
matches the StyleGAN2 trained on 1.6M images. On the full dataset, our method
improves FID by $1.5$ to $2\times$ on cat, church, and horse categories of
LSUN.

##  1 Introduction

Image generation inherently requires being able to capture and model complex
statistics in real-world visual phenomenon. Computer vision models, driven by
the success of supervised and self-supervised learning techniques [74, 30, 15,
17, 62], have proven effective at capturing useful representations when
trained on large-scale data [65, 88, 100]. What potential implications does
this have on generative modeling? If one day, perfect computer vision systems
could answer any question about any image, could this capability be leveraged
to improve image synthesis models?

Figure 1: Vision-aided-GAN training. The model bank $\mathcal{F}$ consists of
widely used and state-of-the-art pretrained networks. We automatically select
a subset $\\{\hat{F}\\}_{k=1}^{K}$ from $\mathcal{F}$, which can best
distinguish between real and fake distribution. Our training procedure
consists of creating an ensemble of the original discriminator ${D}$ and
discriminators $\hat{D}_{k}=\hat{C}_{k}\circ\hat{F}_{k}$ based on the feature
space of selected off-the-shelf models. $\hat{C}_{k}$ is a shallow trainable
network over the frozen pretrained features.

Surprisingly, despite the aforementioned connection between synthesis and
analysis, current state-of-the-art generative adversarial networks (GANs) [38,
98, 9, 37] are trained in an unsupervised manner without the aid of such
pretrained networks. With a plethora of useful models easily available in the
research ecosystem, this presents a missed opportunity to explore. Can the
knowledge of pretrained visual representations actually benefit GAN training?
If so, with so many models, tasks, and datasets to choose from, which models
should be used, and in what manner are they most effective?

In this work, we study the use of a “bank” of pretrained deep feature
extractors to aid in generative model training. Specifically, GANs are trained
with a discriminator, aimed at continuously learning the relevant statistics
differentiating real and generated samples, and a generator, which aims to
reduce this gap. Naïvely using such strong, pretrained networks as a
discriminator leads to the overfitting and overwhelming the generator,
especially in limited data settings. We show that freezing the pretrained
network (with a small, lightweight learned classifier on top as shown in
Figure 1) provides stable training when used with the original, learned
discriminator. In addition, ensembling multiple pretrained networks encourages
the generator to match the real distribution in different, complementary
feature spaces.

To choose which networks work best, we propose to use an automatic model
selection strategy, based on the linear separability of real and fake images
in the feature space, and progressively add supervision from a set of
available pretrained networks. In addition, we use label smoothing [68] and
differentiable augmentation [98, 37] to stabilize the model training further
and reduce overfitting.

We experiment on several datasets in both limited and large-scale sample
setting to show the effectiveness of our method. We improve the current state
of the art on FFHQ [39] and LSUN [88] datasets given 1k training samples by
2-$3\times$ on the FID metric [32]. For LSUN Cats, we match the FID of
StyleGAN2 trained on the full dataset (1.6M images) with only 10k samples, as
shown in Figure 2. In the full-scale data setting, our method improves FID for
LSUN Cats from $6.86$ to $3.98$ and for LSUN Church from $4.28$ to $1.72$. In
case of LSUN Horse dataset our method reduces FID from $4.09$ of StyleGAN2
model to $2.11$. Finally, we visualize the internal representation of our
learned models as well as training dynamics. Our code and models are available
on the project-webpage.

##  2 Related Work

Improving GAN training. Since the introduction of GANs [27], significant
advances have been induced by architectural changes [63, 39, 38], training
schemes [36, 92], as well as objective functions [5, 51, 52, 4, 22]. In
previous works, the learning objectives often aim to minimize different types
of divergences between real and fake distribution. The discriminators are
typically trained from scratch and do not use external pretrained networks. As
a result, the discriminator is prone to overfit the training set, especially
for the limited data setting [98, 37].

Use of pretrained models in image synthesis. Pretrained models have been
widely used as perceptual loss functions [21, 24, 35] to measure the distance
between an output image and a target image in deep feature space. The loss has
proven effective for conditional image synthesis tasks such as super-
resolution [44], image-to-image translation [14, 82, 58], and neural style
transfer [24]. Zhang et al. [95] show that deep features can indeed match the
human perception of image similarity better than classic metrics. Sungatullina
et al. [76] propose a perceptual discriminator to combine perceptual loss and
adversarial loss for unpaired image-to-image translation. This idea was
recently used by a concurrent work on CG2real [64]. Our work is inspired by
the idea of perceptual discriminators [76] but differs in three ways. First,
we focus on a different application of unconditional GAN training rather than
image-to-image translation. Second, instead of using a single VGG model, we
ensemble a diverse set of feature representations that complement each other.
Finally, we propose an automatic model selection method to find models useful
for a given domain. A concurrent work [70] propose to reduce overfitting of
perceptual discriminators [76] using random projection and achieve better and
faster GAN training.

Loosely related to our work, other works have used pretrained models for
clustering, encoding, and nearest neighbor search during their model training.
Logo-GAN [67] uses deep features to get synthetic clustering labels for
conditional GAN training. InclusiveGAN [89] improves the recall of generated
samples by enforcing each real image to be close to a generated image in deep
feature space. Shocher et al. [72] uses an encoder-decoder based generative
model with pretrained encoder for image-to-image translation tasks. Pretrained
features have also been used to condition the generator in GANs [50, 12].
Different from the above work, our method empowers the discriminator with
pretrained models and requires no changes to the backbone generator.

Use of pretrained models in image editing. Pretrained models have also been
used in image editing once the generative model has been trained. Notable
examples include image projection with a perceptual distance [102, 1], text-
driven image editing with CLIP [60], finding editable directions using
attribute classifier models [71], and extracting semantic editing regions with
pretrained segmentation networks [103]. In our work, we focus on using the
rich knowledge of computer vision models to improve model training.

Transfer learning. Large-scale supervised and self-supervised models have
learned useful feature representations [42, 31, 15, 86, 10, 62] that can
transfer well to unseen tasks, datasets, and domains [20, 91, 87, 66, 80, 90,
57, 33, 41]. In generative modeling, recent works propose transferring the
weights of pretrained generators and discriminators from a source domain
(e.g., faces) to a new domain (e.g., portraits of one person) [84, 55, 83, 53,
96, 56, 47, 28]. Together with differentiable data augmentation techniques
[37, 99, 79, 98], they have shown faster convergence speed and better sampling
quality for limited-data settings. Different from them, we transfer the
knowledge of learned feature representations of computer vision models. This
enables us to leverage the knowledge from a diverse set of sources at scale.

##  3 Method

Figure 2: Performance on LSUN Cat and LSUN Church with varying training set
sizes. We compare with the current state-of-the-art methods on different sizes
of training samples and full-dataset. Our method outperforms with significant
margins, especially in limited sample setting. For LSUN Cat we achieve similar
FID as StyleGAN2 [40] trained on full-dataset using only $0.7\%$ of the
dataset.

Generative Adversarial Networks (GANs) aim to approximate the distribution of
real samples from a finite training set $\bm{x}\sim\mathbb{P}_{\mathcal{X}}$.
The generator network ${G}$, maps latent vectors
$\bm{z}\sim\mathbb{P}(\bm{z})$ (e.g., a normal distribution) to samples
${G}(\bm{z})\sim\mathbb{P}_{\theta}$. The discriminator network ${D}$ is
trained adversarially to distinguish between the continuously changing
generated distribution $\mathbb{P}_{\theta}$ and target real distribution
$\mathbb{P}_{\mathcal{X}}$. GANs perform the minimax optimization
$\min_{G}\max_{{D}}V(D,G)$, where

$\begin{gathered}V(D,G)=\mathbb{E}_{\bm{x}\sim\mathbb{P}_{\mathcal{X}}}[\log{D}(\bm{x})+\mathbb{E}_{\bm{z}\sim\mathbb{P}(\bm{z})}[\log(1-{D}({G}(\bm{z})))].\end{gathered}$
(1)

Ideally, the discriminator should measure the gap between
$\mathbb{P}_{\mathcal{X}}$ and $\mathbb{P}_{\theta}$ and guide the generator
towards $\mathbb{P}_{\mathcal{X}}$. However, in practice, large capacity
discriminators can easily overfit on a given training set, especially in the
limited-data regime [98, 37]. Unfortunately, as shown in Figure 3, even when
we adopt the latest differentiable data augmentation [37] to reduce
overfitting, the discriminator still tends to overfit, failing to perform well
on a validation set. In addition, the discriminator can potentially focus on
artifacts that are indiscernible to humans but obvious for machines [81].

To address the above issues, we propose ensembling a diverse set of deep
feature representations as our discriminator. This new source of supervision
can benefit us in two manners. First, training a shallow classifier over
pretrained features is a common way to adapt deep networks to a small-scale
dataset, while reducing overfitting [26, 16]. As shown in Figure 3, our method
reduces the discriminator overfitting significantly. Second, recent studies
[91, 6] have shown that deep networks can capture meaningful visual concepts
from low-level visual cues (edges and textures), to high-level concepts
(objects and object parts). A discriminator built on these features may better
match human perception [95].

###  3.1 Formulation

Given a set of pretrained feature extractors
$\mathcal{F}=\\{F_{n}\\}_{n=1}^{N}$, which learns to tackle different vision
tasks, we train corresponding discriminators $\\{D_{n}\\}_{n=1}^{N}$. We add
small classifier heads $\\{C_{n}\\}_{n=1}^{N}$ to measure the gap between
$\mathbb{P}_{\mathcal{X}}$ and $\mathbb{P}_{\theta}$ in the pretrained models’
feature spaces. During discriminator training, the feature extractor $F_{n}$
is frozen, and only the classifier head is updated. The generator ${G}$ is
updated with the gradients from ${D}$ and the discriminators $\\{D_{n}\\}$,
based on pretrained feature extractors. In this manner, we propose to leverage
pretrained models in an adversarial fashion for GAN training, which we refer
to as Vision-aided-Adversarial training:

$\displaystyle\min_{G}\;\max_{{D},\\{C_{n}\\}_{n=1}^{N}}V(D,G)+\overbrace{\sum_{n=1}^{N}V(D_{n},G)}^{\text{vision-
aided-adversarial loss}},$ (2) $\displaystyle\textnormal{where
}D_{n}=C_{n}\circ F_{n}.$

Here, $C_{n}$ is a small trainable head over the pretrained features. The
above training objective involves the sum of discriminator losses based on all
available pretrained models $\\{F_{n}\\}$. Solving for this at each training
iteration would be computationally and memory-intensive. Using all pretrained
models would force a significant reduction in batch size to fit all models
into memory, potentially hurting performance [9]. To bypass the computational
bottleneck, we automatically select a small subset of $K$ models, where $K<N$:

$\displaystyle\min_{G}\;\max_{{D},\\{\hat{C}_{k}\\}_{k=1}^{K}}V(D,G)+\sum_{k=1}^{K}V(\hat{D}_{k},G),$
(3)

where $\hat{D}_{k}=\hat{C}_{k}\circ\hat{F}_{k}$ denotes the discriminator
corresponding to $k^{\text{th}}$ selected model, and $k\in\\{1,\dots,K\\}$.

Figure 3: Training and validation accuracy w.r.t. training iterations for our
pretrained model (DINO [10]) based discriminator vs. baseline StyleGAN2-ADA
discriminator on FFHQ 1k dataset. Our discriminator based on off-the-shelf
model features has higher accuracy on validation real images and thus shows
better generalization. In our method, we add our training loss at the 2M
iteration.  Algorithm 1 GAN training with _Vision-aided-Adversarial_ loss.

1:${G}$, ${D}$ trained with standard GAN loss for baseline number of
iterations. Off-the-shelf model bank $\mathcal{F}=\\{F_{n}\\}_{n=1}^{N}$.
Training data $\\{\bm{x}_{i}\\}$.

2:$K$: maximum number of pretrained models to use. $T$: training interval for
adding next pretrained model.

3:Selected model set $\mathcal{\hat{F}}=$Ø

4:for $k=1$ to $K$ do

5: Select best model $\hat{F}_{k}\in\mathcal{F}$ using Eqn. 4

6: $\mathcal{\hat{F}}=\mathcal{\hat{F}}\cup\\{\hat{F}_{k}\\}$

7: $\hat{D}_{k}=\hat{C}_{k}\circ\hat{F}_{k}$ $\triangleright$ $\hat{C}_{k}$ is
a shallow trainable network

8: $\mathcal{F}=\mathcal{F}\setminus\hat{F}_{k}$

9: for $t=1$ to $T$ do

10: Sample $\bm{x}\sim\\{\bm{x}_{i}\\}$

11: Sample $\bm{z}\sim\mathbb{P}(\bm{z})$

12: Update ${D},\hat{D}_{j}$ $\forall j=1,...,k$ using Eqn. 3

13: Sample $\bm{z}\sim\mathbb{P}(\bm{z})$

14: Update ${G}$ using Eqn. 3

15: end for

16:end for

17:${G}$ with best training set FID

###  3.2 Model Selection

We choose the models whose off-the-shelf feature spaces best distinguish
samples from real and fake distributions. Given the pretrained model’s
features of real and fake images, the strongest adversary from the set of
models is $\hat{F}_{k}$, where

$\displaystyle
k=\arg\max_{n}\big{\\{}\max_{C^{\prime}_{n}}V(D^{\prime}_{n},G)\big{\\}},$ (4)
$\displaystyle\text{where }D^{\prime}_{n}=C^{\prime}_{n}\circ F_{n}.$

Here $F_{n}$ is frozen, and $C^{\prime}_{n}$ is a linear trainable head over
the pretrained features. In the case of limited real samples available and for
computational efficiency, we use linear probing to measure the separability of
real and fake images in the feature space of $F_{n}$.

We split real training samples $\\{\bm{x}_{i}\\}$ and generated images
$\\{{G}(\bm{z}_{i})\\}$ into training and validation sets. For each pretrained
model $F_{n}$, we train a logistic linear discriminator head to classify
whether a sample comes from $\mathbb{P}_{\mathcal{X}}$ or
$\mathbb{P}_{\theta}$ and measure $V(D^{\prime}_{n},G)$ on the validation
split. The above term measures the negative binary cross-entropy loss and
returns the model with the lowest error. A low validation error correlates
with higher accuracy of the linear probe, indicating that the features are
useful for distinguishing real from generated samples and using these features
will provide more useful feedback to the generator. We empirically validate
this on GAN training with 1k training samples of FFHQ and LSUN Cat datasets.
Figure 4 shows that the GANs trained with the pretrained model $F_{n}$ with
higher linear probe accuracy in general achieve better FID metrics.

To incorporate feedback from multiple off-the-shelf models, we explore two
variants of model selection and ensembling strategies – (1) K-fixed model
selection strategy chooses the K best off-the-shelf models at the start of
training and trains until convergence and (2) K-progressive model selection
strategy iteratively selects and adds the best, unused off-the-shelf model
after a fixed number of iterations.

K-progressive model selection.  We find including multiple models in a
progressive manner has lower computational complexity compared to the K-fixed
strategy. This also helps in the selection of pretrained models, which
captures different aspects of the data distribution. For example, the first
two models selected through the progressive strategy are usually a pair of
self-supervised and supervised models. For these reasons, we primarily perform
all of our experiments using the progressive strategy. We also show a
comparison between the two strategies in Section 4.4.

Figure 4: Model selection using linear probing of pretrained features. The
plot shows correlation of FID with the accuracy of a logistic linear model
trained for real vs fake classification over the features of various off-the-
shelf models. Top dotted line indicates the FID of baseline StyleGAN2-ADA
generator from which we finetune with our proposed vision-aided-adversarial
loss. Similar analysis for LSUN Cat is shown in Figure 12 in the appendix.

Discussion. The idea of linear separability as a metric has been previously
used for evaluating GAN via classifier two-sample tests [49, 97]. We adopt
this in our work to evaluate the usefulness of available off-the-shelf
discriminators, rather than evaluating generators. “Linear probing” is also a
common technique for measuring the effectiveness of intermediate features
spaces in both self-supervised [94, 29, 15] and supervised [3] contexts, and
model selection has been explored in previous works to predict expert models
for transfer learning [61, 54, 23]. We explore this in context of generative
modeling and propose a progressive addition of next best model to create an
ensemble [11] of discriminators.

Dataset | StyleGAN2 | DiffAugment | ADA | Ours (w/ ADA) | Ours (w/ DiffAugment)
---|---|---|---|---|---
+$1^{\text{st}}$ D | +$2^{\text{nd}}$ D | +$3^{\text{rd}}$ D | +$1^{\text{st}}$ D | +$2^{\text{nd}}$ D | +$3^{\text{rd}}$ D
FFHQ | 1k | 62.16 | 27.20 | 19.57 | 11.43 | 10.39 | 10.58 | 12.33 | 13.39 | 12.76
2k | 42.62 | 16.63 | 16.06 | 10.17 | 8.73 | 8.18 | 10.01 | 9.24 | 10.99
10k | 16.07 | 8.15 | 8.38 | 6.90 | 6.39 | 5.90 | 6.94 | 6.26 | 6.43
LSUN Cat | 1k | 185.75 | 43.32 | 41.14 | 15.49 | 12.90 | 12.19 | 13.52 | 12.52 | 11.01
2k | 68.03 | 25.70 | 23.32 | 13.44 | 13.35 | 11.51 | 12.20 | 11.79 | 11.33
10k | 18.59 | 12.56 | 13.25 | 8.37 | 7.13 | 6.86 | 8.19 | 7.90 | 7.79
LSUN Church | 1k | - | 19.38 | 19.66 | 11.39 | 9.78 | 9.56 | 10.15 | 9.87 | 9.94
2k | - | 13.46 | 11.17 | 5.25 | 5.06 | 5.26 | 6.09 | 6.37 | 5.56
10k | - | 6.69 | 6.12 | 4.80 | 4.82 | 4.47 | 3.42 | 3.41 | 3.25

Table 1: FFHQ and LSUN results with varying training samples from 1k to 10k.
FID$\downarrow$ is measured with complete dataset as reference distribution.
We select the best snapshot according to training set FID, and report mean of
3 FID evaluations with full reference distribution. Our method works with both
ADA and DiffAugment strategy for augmenting images input to the
discriminators.

###  3.3 Training Algorithm

As shown in Algorithm 1, our final algorithm consists of first training a GAN
with standard adversarial loss [40, 27]. Given this baseline generator, we
search for the best off-the-shelf models using linear classifier test and
introduce our proposed loss objective during training. In K-progressive
strategy, we add the new pretrained model based discriminator after a fixed
number of iterations proportional to the number of available real training
samples. The next pretrained models is added to the snapshot with the best
training set FID. During training, we perform data augmentation through
horizontal flipping and use differentiable augmentation techniques [37, 98].
If the linear probe accuracy corresponding to the selected model is above
$90\%$, we use one-sided label smoothing [68] as a regularization. We also
observe that only using off-the-shelf models as the discriminator leads to
divergence. Thus, the benefit is brought by ensembling the original
discriminator and the newly added off-the-shelf models. We show results with
the use of three pretrained models. We observe minimal benefit with the
progressive addition of next model if the linear probe accuracy is low and
worse than the models already in the selected set.

##  4 Experiments

Here we conduct extensive experiments on multiple datasets of different
resolutions with the StyleGAN2 architecture. We show results on FFHQ [39],
LSUN Cat, and LSUN Church datasets [88] while varying training sample size
from 1k to 10k, as well as with the full dataset. For real-world limited
sample datasets, we perform experiments on the cat, dog, and wild categories
of AFHQ [18] dataset at 512 resolution and MetFaces [37] at 1024 resolution.

Baseline and metrics. We compare our method with current state-of-the-art
methods for limited dataset GAN training, StyleGAN2-ADA [37] and DiffAugment
[98]. We compute the commonly used Fréchet Inception Distance (FID) metric
[32] using the clean-fid library [59] to evaluate models. We also report KID
[8] and precision and recall [43] for all experiments in Appendix B.

Off-the-shelf models. We include eight large-scale self-supervised and
supervised networks. Specifically, we perform experiments with CLIP [62],
VGG-16 [74] trained for ImageNet [19] classification, and self-supervised
models, DINO [10] and MoBY [86]. We also include face parsing [45] and face
normals prediction networks [2]. Finally, we have Swin-Transformer [48] based
segmentation model based on trained on ADE-20K [101] and object detection
model trained on MS-COCO [46]. Full details of all models is given in Appendix
C.

Figure 5: LSUN Cat, FFHQ, and LSUN Church paired sample comparison in 1k
training dataset setting. For each dataset, the top row shows the baseline
StyleGAN2-ADA samples, and the bottom row shows the samples by Our method for
the same randomly sample latent code. As our method finetunes from a
StyleGAN2-ADA model, we observe improved image quality with our method for the
same latent code.

Pretrained model-based discriminator’s architecture. For discriminator
$\hat{D}_{k}$ based on pretrained model features, we extract spatial features
from the last layer and use a small Conv-LeakyReLU-Linear-LeakyReLU-Linear
architecture for binary classification. In the case of big transformer
networks, such as CLIP and DINO, we explore a multi-scale architecture that
works better. For all experiments, we use three pretrained models selected by
the model selection strategy during training. Details about the architecture,
model training, memory requirements, and hyperparameters are provided in
Appendix C.

Figure 6: Linear probe accuracy of off-the-shelf models during our
K-progressive ensemble training on FFHQ 1k. For the StyleGAN2-ADA, ViT (DINO)
model has the highest accuracy and is selected first, then ViT (CLIP) and then
Swin-T (MoBY). As we include more discriminators based on off-the-shelf
models, linear probe accuracy corresponding to the pretrained models
decreases. Similar trend for all our experiments are shown in the Appendix C.

###  4.1 FFHQ and LSUN datasets

Table 1 shows the results of our method when the training sample is varied
from 1k to 10k for FFHQ, LSUN Cat, and LSUN Church datasets. The considerable
gain in FID for all settings shows the effectiveness of our method in the
limited data scenario. To qualitatively analyze the difference between our
method and StyleGAN2-ADA, we show randomly generated samples from both models
given the same latent code in Figure 5. Our method improves the quality of the
worst samples, especially for FFHQ and LSUN Cat (also see Figure 13 in
Appendix A). Figure 6 shows the accuracy of linear probe over the pretrained
models’s features as we progressively add the next discriminator during
training.

To analyze the overfitting behavior of discriminators, we evaluate its
training and validation accuracy across iterations. Compared to the baseline
StyleGAN2-ADA discriminator, our discriminator based on pretrained features
shows better generalization on the validation set specifically for limited-
data regime as shown in Figure 3 for FFHQ 1k setting.

Full-dataset training. In the full-dataset setting, we finetune the trained
StyleGAN2 config-F [40] generator with our method. Table 2 shows the results
using a maximum of three ensembled discriminators. We report both FID and
Perceptual Path Length (PPL) [39] (W space) metric. On LSUN Cat, our method
improves FID from $6.86$ to $3.98$, on LSUN Church from $4.28$ to $1.72$, and
on LSUN Horse from $4.09$ to $2.11$. For FFHQ dataet, our method improves the
PPL metric from $144.62$ to $127.58$ and has similar performance on FID
metric. Perceptual path length has been shown to correlate with image quality
and indicates a smooth mapping in generator latent space [40].

Dataset | Resolution | StyleGAN2(F) | Ours (w/ ADA)
---|---|---|---
FID $\downarrow$ | PPL $\downarrow$ | FID $\downarrow$ | PPL $\downarrow$
FFHQ | 1024 $\times$ 1024 | 2.98 | 144.62 | 3.01 | 127.58
LSUN Cat | 256 $\times$ 256 | 6.86 | 437.13 | 3.98 | 420.15
LSUN Church | 256 $\times$ 256 | 4.28 | 343.02 | 1.72 | 388.94
LSUN Horse | 256 $\times$ 256 | 4.09 | 337.98 | 2.11 | 307.12

Table 2: Results on full-dataset setting. we improve the FID metric on LSUN
categories by a significant margin. On the FFHQ dataset, our method improves
the PPL metric.  Figure 7: GAN Dissection visualization of improved units. We
analyze StyleGAN2 and our model trained on LSUN Church using GAN Dissection
[7] and show here qualitative examples of units with improved IoU to a
semantic category. The total number of detected units also increases from 254
to 297 for our model.

GAN Dissection analysis on LSUN Church. How does the generator change with the
use of off-the-shelf models as discriminators? To analyze this, we use the
existing technique of GAN Dissection [7, 6], which calculates the correlation
between convolutional feature maps of the generator and scene parts obtained
through a semantic segmentation network [85]. Specifically, we select the
convolutional layer with $32$ resolution in the generator trained with our
method and StyleGAN2 on the full LSUN Church dataset. The total number of
interpretable units [7] increases from $254$ to $297$ by our method,
suggesting that our model may learn a richer representation of semantic
concepts. Figure 7 shows the complete statistics of detected units
corresponding to each semantic category and some of the example images of
improved units by our method. We observe an overall increase in the number of
detected units as well as units corresponding to new semantic categories.

Dataset | Transfer | StyleGAN2 | StyleGAN2-ADA | Ours (w/ ADA)
---|---|---|---|---
FID $\downarrow$ | KID $\downarrow$ | Recall $\uparrow$ | FID $\downarrow$ | KID $\downarrow$ | Recall $\uparrow$ | FID $\downarrow$ | KID $\downarrow$ | Recall $\uparrow$
AFHQ Dog | ✗ | 22.35 | 10.05 | 0.20 | 7.60 | 1.29 | 0.47 | 4.73 | 0.39 | 0.60
✓ | 9.28 | 3.13 | 0.42 | 7.52 | 1.22 | 0.43 | 4.81 | 0.37 | 0.61
AFHQ Cat | ✗ | 5.16 | 1.72 | 0.26 | 3.29 | 0.72 | 0.41 | 2.53 | 0.47 | 0.52
✓ | 3.48 | 1.07 | 0.47 | 3.02 | 0.38 | 0.45 | 2.69 | 0.62 | 0.50
AFHQ Wild | ✗ | 3.62 | 0.84 | 0.15 | 3.00 | 0.44 | 0.14 | 2.36 | 0.38 | 0.29
✓ | 2.11 | 0.17 | 0.35 | 2.72 | 0.17 | 0.29 | 2.18 | 0.28 | 0.38
MetFaces | ✓ | 57.26 | 2.50 | 0.34 | 17.56 | 1.55 | 0.22 | 15.44 | 1.03 | 0.30

Table 3: Results on AFHQ and MetFaces. Our method, in general, results in
lower FID and higher Recall. In transfer setup we finetune from a FFHQ trained
model of similar resolution with $D$ updated according to freezeD technique
[53] similar to [37]. We select the snapshot with the best FID and show an
average of three evaluations. KID is shown in $\times 10^{3}$ units following
[37].

Human preference study. We perform a human preference study on Amazon
Mechanical Turk (AMT) to verify that our results agree with the human judgment
regarding the improved sample quality. We compare StyleGAN2-ADA and our method
trained on $1$k samples of LSUN Cat, LSUN Church, and FFHQ datasets. Since we
finetune StyleGAN2-ADA with our method, the same latent code corresponds to
similar images for the two models, as also shown in Figure 5. For randomly
sampled latent codes, we show the two images generated by our method and
StyleGAN2-ADA for six seconds to the test subject and ask the subject to
select the more realistic image. We perform this study for $50$ test subjects
per dataset, and each subject is shown a total of $55$ images. On the FFHQ
dataset, human preference for our method is $53.8\%\pm 1.3$. For the LSUN
Church dataset, our method is preferred over StyleGAN2-ADA with $60.5\%\pm
1.7$, and for the LSUN Cat dataset $63.5\%\pm 1.6$. These results correlate
with the improved FID metric. Example images from our study are shown in
Figure 14 in Appendix A.

Figure 8: Qualitative comparison of our method with StyleGAN2-ADA on AFHQ.
Left: randomly generated samples for both methods. Right: For both our model
and StyleGAN2-ADA, we independently generate 5k samples and find the worst-
case samples compared to real image distribution. We first fit a Gaussian
model using the Inception [77] feature space of real images. We then calculate
the log-likelihood of each sample given this Gaussian prior and show the
images with minimum log-likelihood (maximum Mahalanobis distance). We show
more samples in Figure 18 and Figure 19 in Appendix A.  Figure 9:
FID$\downarrow$ w.r.t. training time comparison between StyleGAN2-ADA and our
method (with one pretrained model) when applied from the start for FFHQ, LSUN
Cat 1k, and LSUN Cat full-dataset setting. There is a warm-up of $0.5$M
images, and then our loss is added. Our method results in similar FID at more
than twice the speedup. We show training time in hours measured on one RTX
3090.

###  4.2 AFHQ and MetFaces

To further evaluate our method on real-world limited sample datasets, we
perform experiments on MetFaces (1336 images) and AFHQ dog, cat, wild
categories with $\sim$ 5k images per category. We compare with the current
state-of-the-art StyleGAN2-ADA under two settings, (1) Fine-tuning
StyleGAN2-ADA model with our loss (2) Fine-tuning from a StyleGAN2 model
trained on FFHQ dataset of same resolution (transfer setup) using FreezeD
[53]. The second setting evaluates the transfer learning capability when fine-
tuned from a generator trained on a different domain. Table 3 shows the
comparison of our method with StyleGAN2 and StyleGAN2-ADA on multiple metrics.
We outperform or perform on-par compared to the existing methods in most of
the cases. Figure 8 shows the qualitative comparison between our method and
StyleGAN2-ADA.

Method | Bridge | AnimalFace Cat | AnimalFace Dog
---|---|---|---
FID $\downarrow$ | KID$\downarrow$ | FID $\downarrow$ | KID $\downarrow$ | FID$\downarrow$ | KID$\downarrow$
| DiffAugment | 54.50 | 15.68 | 43.87 | 7.56 | 60.50 | 20.13
| ADA | - | - | 38.01 | 5.61 | 52.59 | 14.32
Ours | +$1^{\text{st}}$ D | 44.18 | 9.27 | 30.62 | 1.15 | 34.23 | 2.01
+$2^{\text{nd}}$ D | 33.89 | 2.35 | 28.01 | 0.37 | 33.03 | 1.37
+$3^{\text{rd}}$ D | 34.35 | 2.96 | 27.35 | 0.34 | 32.56 | 1.67

Table 4: Low-shot generation results on 100-shot Bridge dataset [98],
AnimalFace cat and dog [73] categories. Our method significantly improves FID
and KID compared to existing state-of-the-art methods. KID is shown in $\times
10^{3}$ units.

###  4.3 Low-shot Generation

To test our method to the limit of low-shot samples, we evaluate our method
when only 100-400 samples are available. We train StyleGAN2-ADA with our
method on AnimalFace cat (169 images) and dog (389 images) [73], and 100-shot
Bridge-of-Sighs [98]. For differentiable augmentation we use ADA except for
the 100-shot dataset we find that the DiffAugment [98] works better than ADA
[37], and therefore employ that. Our method leads to considerable improvement
over existing methods on both FID and KID metrics as shown in Table 4. We show
latent space interpolations in Figure 21 in Appendix A.

###  4.4 Ablation Study

Finetuning vs. training from scratch. In the above experiments, we finetuned a
well-trained StyleGAN2-ADA model with our loss. We show here that our method
works similarly well when trained from scratch in the few-shot setting. Figure
9 shows the plot of FID with training time for both StyleGAN2-ADA and our
method with a single pretrained discriminator on FFHQ and LSUN Cat trained
with 1k samples, and LSUN Cat full-dataset setting. Our method results in
better FID and converges more than $2\times$ faster. During training from
scratch, we train with the standard adversarial loss for the first $0.5$M
images and then introduce the discriminator selected by the model selection
strategy. Table 5 shows the FID of our method when trained from scratch with
three off-the-shelf models.

K-progressive vs. K-fixed model selection. We compare the K-progressive and
K-fixed model selection strategies in this section. Figure 10 shows the
comparison for FFHQ 1k and LSUN Cat 1k trained for the same number of
iterations with two models from our model bank. We observe that training with
two fixed pretrained models from the start results in a similar or slightly
worse FID at the cost of extra training time compared to the progressive
addition.

Figure 10: K-progressive vs K-fixed comparison on FFHQ and LSUN Cat 1k
setting. Progressive addition of the next best model is computationally
efficient and results in similar FID at lower run time. We show training time
in hours measured on one RTX 3090.

Our model selection vs. random selection. We showed earlier in Figure 4 that
FID correlates with model selection ranking for training with our loss using a
single pre-trained model. To show the effectiveness of model selection in
K-progressive strategy, we compare it with randomly selecting models during
progressive addition. In FFHQ 1k, our method selects DINO, CLIP, and Swin-T
(MoBY) during training. Random selection consists of VGG-16, Swin-T (MoBY),
and U-Net (Face Parsing) networks. For LSUN Cat 1k setting, CLIP, DINO, and
Swin-T (Segmentation) networks are selected by our method during training. In
random selection VGG-16, CLIP, and Swin-T (Segmentation) networks are
selected. The results are as shown in Table 5. We observe that random
selection of pretrained models from the model bank already provides benefit in
FID, but with our model selection, it can be improved further.

Ours |  FFHQ 1k |  LSUN Cat 1k
---|---|---
+$1^{\text{st}}$ D | +$2^{\text{nd}}$ D | +$3^{\text{rd}}$ D | +$1^{\text{st}}$ D | +$2^{\text{nd}}$ D | +$3^{\text{rd}}$ D
Finetune | 11.43 | 10.39 | 10.58 | 15.49 | 12.90 | 12.19
From scratch | 13.57 | 12.26 | 10.60 | 15.13 | 12.72 | 12.24
Random selection | 15.48 | 12.54 | 11.92 | 19.02 | 15.12 | 14.28

Table 5: FID$\downarrow$ metric for models trained with different settings in
our method. $1^{\text{st}}$ Row: when fine-tuned from a StyleGAN2-ADA model.
$2^{nd}$ Row: when our loss is used for training from scratch. $3^{rd}$ Row:
comparison of our model selection based approach with random model selection.
Figure 11: Ablation of augmentation and label smoothing on FFHQ and LSUN Cat
with 1k training samples and LSUN Cat full-dataset setting. We show the plot
of FID w.r.t training iterations when ADA [37] augmentation and label
smoothing [68] are individually removed from our training. We observe that
without having any differentiable augmentation training quickly collapses in
limited sample setting. Even for full-dataset, having differentiable
augmentation on the input of pretrained model based discriminator results in
better FID. Label smoothing has a reasonable effect in case of LSUN Cat 1k and
is marginally helpful for FFHQ 1k. We also change the augmentation technique
to DiffAugment [98] for both original and vision model based discriminator and
observe that it performs comparable to ADA [37].

Method |  | FFHQ
---
1k
| LSUN Cat
---
1k
| LSUN Cat
---
1.6M
StyleGAN2-ADA | 19.57 | 41.14 | 6.86
Ours (w/ ViT (CLIP)) | 11.43 | 15.49 | 4.61
Ours w/ finetune ViT (CLIP) | ✗ | ✗ | ✗
Ours w/ ViT random weights | 19.10 | 33.77 | 6.35
Ours w/ multi-discriminator | 17.59 | 37.01 | ✗
Longer StyleGAN2-ADA | 19.07 | 39.36 | 6.52

Table 6: Additional ablation studies evaluated on FID$\downarrow$ metric.
Having two discriminators during training (frozen with random weights or
trainable) or standard adversarial training for more iterations leads to only
marginal benefits in FID. Thus the improvement is through an ensemble of
original and pretrained feature based discriminators. ✗ means FID increased to
twice the baseline, and therefore, we stop the training run.

Role of data augmentation and label smoothing. Here, we investigate the role
of differentiable augmentation [37, 99, 79, 98] which is one of the important
factors that enable the effective use of pretrained features. Label smoothing
[68] further improves the training dynamics, especially in a limited sample
setting. We ablate each of these component and show its contribution in Figure
11 on FFHQ and LSUN Cat dataset in 1k sample setting, and LSUN Cat full-
dataset setting. Figure 11 shows that replacing ADA [37] augmentation strategy
with DiffAugment [98] in our method also performs comparably. Moreover, in the
limited sample setting, without data augmentation, model collapses very early
in training, and FID quickly diverges. The role of label smoothing is more
prominent for LSUN Cat 1k.

Additional ablation study. Here we further analyze the importance of our
design choice. All the experiments are done on LSUN Cat and FFHQ. We compare
our method with the following settings: (1) Fine-tuning ViT (CLIP) network as
well in our vision-aided adversarial loss; (2) Randomly initializing the
feature extractor network ViT (CLIP); (3) Training with two discriminators,
where the $2^{\text{nd}}$ discriminator is of same architecture as StyleGAN2
original discriminator; (4) Training the StyleGAN2-ADA model longer for the
same number of iterations as ours with standard adversarial loss. The results
are as shown in Table 6. We observe that the baseline methods provide marginal
improvement, whereas our method offers significant improvement over
StyleGAN2-ADA, as measured by FID.

##  5 Limitations and Discussion

In this work, we propose to use available off-the-shelf models to help in the
unconditional GAN training. Our method significantly improves the quality of
generated images, especially in the limited-data setting. While the use of
multiple pre-trained models as discriminators improves the generator, it has a
few limitations. First, this increases memory requirement for training.
Exploring the use of efficient computer vision models [78, 69] will
potentially make our method more accessible. Second, our model selection
strategy is not ideal in the low-shot settings when only a dozen samples are
available. We observe increased variance in the linear probe accuracy with
sample size $\sim 100$ which can lead to ineffective model selection. We plan
to adopt few-shot learning [75, 25] methods for these settings in future.

Nonetheless, as more and more self-supervised and supervised computer vision
models are readily available, they should be used to good advantage for
generative modeling. This paper serves as a small step towards improving
generative modeling by transferring the knowledge from large-scale
representation learning.

Acknowledgment. We thank Muyang Li, Sheng-Yu Wang, Chonghyuk (Andrew) Song for
proofreading the draft. We are also grateful to Alexei A. Efros, Sheng-Yu
Wang, Taesung Park, and William Peebles for helpful comments and discussion.
The work is partly supported by Adobe Inc., Kwai Inc, Sony Corporation, and
Naver Corporation.

## References

  * [1] Rameen Abdal, Yipeng Qin, and Peter Wonka.  Image2stylegan: How to embed images into the stylegan latent space?  In ICCV, 2019.
  * [2] Victoria Fernández Abrevaya, Adnane Boukhayma, Philip HS Torr, and Edmond Boyer.  Cross-modal deep face normals with deactivable skip connections.  In CVPR, 2020.
  * [3] Guillaume Alain and Yoshua Bengio.  Understanding intermediate layers using linear classifier probes.  arXiv preprint arXiv:1610.01644, 2016.
  * [4] Isabela Albuquerque, João Monteiro, Thang Doan, Breandan Considine, Tiago Falk, and Ioannis Mitliagkas.  Multi-objective training of generative adversarial networks with multiple discriminators.  In ICML, 2019.
  * [5] Martin Arjovsky, Soumith Chintala, and Léon Bottou.  Wasserstein Generative Adversarial Networks.  In ICML, 2017.
  * [6] David Bau, Jun-Yan Zhu, Hendrik Strobelt, Agata Lapedriza, Bolei Zhou, and Antonio Torralba.  Understanding the role of individual units in a deep neural network.  Proceedings of the National Academy of Sciences, 2020.
  * [7] David Bau, Jun-Yan Zhu, Hendrik Strobelt, Bolei Zhou, Joshua B Tenenbaum, William T Freeman, and Antonio Torralba.  Gan dissection: Visualizing and understanding generative adversarial networks.  In ICLR, 2019.
  * [8] Mikołaj Bińkowski, Danica J Sutherland, Michael Arbel, and Arthur Gretton.  Demystifying mmd gans.  In ICLR, 2018.
  * [9] Andrew Brock, Jeff Donahue, and Karen Simonyan.  Large scale gan training for high fidelity natural image synthesis.  In ICLR, 2019.
  * [10] Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin.  Emerging properties in self-supervised vision transformers.  In ICCV, 2021.
  * [11] Rich Caruana, Alexandru Niculescu-Mizil, Geoff Crew, and Alex Ksikes.  Ensemble selection from libraries of models.  In ICML, 2004.
  * [12] Arantxa Casanova, Marlène Careil, Jakob Verbeek, Michal Drozdzal, and Adriana Romero.  Instance-conditioned gan.  In NeurIPS, 2021.
  * [13] Lucy Chai, David Bau, Ser-Nam Lim, and Phillip Isola.  What makes fake images detectable? understanding properties that generalize.  In European Conference on Computer Vision, pages 103–120. Springer, 2020.
  * [14] Qifeng Chen and Vladlen Koltun.  Photographic image synthesis with cascaded refinement networks.  In ICCV, 2017.
  * [15] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton.  A simple framework for contrastive learning of visual representations.  In ICML, 2020.
  * [16] Wei-Yu Chen, Yen-Cheng Liu, Zsolt Kira, Yu-Chiang Frank Wang, and Jia-Bin Huang.  A closer look at few-shot classification.  In ICLR, 2019.
  * [17] Xinlei Chen, Haoqi Fan, Ross Girshick, and Kaiming He.  Improved baselines with momentum contrastive learning.  arXiv preprint arXiv:2003.04297, 2020.
  * [18] Yunjey Choi, Youngjung Uh, Jaejun Yoo, and Jung-Woo Ha.  Stargan v2: Diverse image synthesis for multiple domains.  In CVPR, 2020.
  * [19] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei.  Imagenet: A large-scale hierarchical image database.  In CVPR, 2009.
  * [20] Jeff Donahue, Yangqing Jia, Oriol Vinyals, Judy Hoffman, Ning Zhang, Eric Tzeng, and Trevor Darrell.  Decaf: A deep convolutional activation feature for generic visual recognition.  In ICML, 2014.
  * [21] Alexey Dosovitskiy and Thomas Brox.  Generating images with perceptual similarity metrics based on deep networks.  In NeurIPS, 2016.
  * [22] Ishan Durugkar, Ian Gemp, and Sridhar Mahadevan.  Generative multi-adversarial networks.  In ICLR, 2017.
  * [23] Kshitij Dwivedi and Gemma Roig.  Representation similarity analysis for efficient task taxonomy & transfer learning.  In CVPR, 2019.
  * [24] Leon A Gatys, Alexander S Ecker, and Matthias Bethge.  Image style transfer using convolutional neural networks.  In CVPR, 2016.
  * [25] Spyros Gidaris, Andrei Bursuc, Nikos Komodakis, Patrick Pérez, and Matthieu Cord.  Boosting few-shot visual learning with self-supervision.  In ICCV, 2019.
  * [26] Ross Girshick, Jeff Donahue, Trevor Darrell, and Jitendra Malik.  Rich feature hierarchies for accurate object detection and semantic segmentation.  In CVPR, 2014.
  * [27] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio.  Generative adversarial networks.  Communications of the ACM, 63(11):139–144, 2020.
  * [28] Zheng Gu, Wenbin Li, Jing Huo, Lei Wang, and Yang Gao.  Lofgan: Fusing local representations for few-shot image generation.  In ICCV, 2021.
  * [29] Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick.  Momentum contrast for unsupervised visual representation learning.  In CVPR, 2020.
  * [30] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In CVPR, 2016.
  * [31] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In CVPR, 2016.
  * [32] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter.  Gans trained by a two time-scale update rule converge to a local nash equilibrium.  In NeurIPS, 2017.
  * [33] Minyoung Huh, Pulkit Agrawal, and Alexei A Efros.  What makes imagenet good for transfer learning?  arXiv preprint arXiv:1608.08614, 2016.
  * [34] Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, and Alexei A Efros.  Image-to-image translation with conditional adversarial networks.  In CVPR, 2017.
  * [35] Justin Johnson, Alexandre Alahi, and Li Fei-Fei.  Perceptual losses for real-time style transfer and super-resolution.  In ECCV, 2016.
  * [36] Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen.  Progressive growing of gans for improved quality, stability, and variation.  In ICLR, 2018.
  * [37] Tero Karras, Miika Aittala, Janne Hellsten, Samuli Laine, Jaakko Lehtinen, and Timo Aila.  Training generative adversarial networks with limited data.  In NeurIPS, 2020.
  * [38] Tero Karras, Miika Aittala, Samuli Laine, Erik Härkönen, Janne Hellsten, Jaakko Lehtinen, and Timo Aila.  Alias-free generative adversarial networks.  In NeurIPS, 2021.
  * [39] Tero Karras, Samuli Laine, and Timo Aila.  A style-based generator architecture for generative adversarial networks.  In CVPR, 2019.
  * [40] Tero Karras, Samuli Laine, Miika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila.  Analyzing and improving the image quality of stylegan.  In CVPR, 2020.
  * [41] Simon Kornblith, Jonathon Shlens, and Quoc V Le.  Do better imagenet models transfer better?  In CVPR, 2019.
  * [42] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton.  Imagenet classification with deep convolutional neural networks.  In NeurIPS, 2012.
  * [43] Tuomas Kynkäänniemi, Tero Karras, Samuli Laine, Jaakko Lehtinen, and Timo Aila.  Improved precision and recall metric for assessing generative models.  In NeurIPS, 2019.
  * [44] Christian Ledig, Lucas Theis, Ferenc Huszár, Jose Caballero, Andrew Cunningham, Alejandro Acosta, Andrew Aitken, Alykhan Tejani, Johannes Totz, Zehan Wang, et al.  Photo-realistic single image super-resolution using a generative adversarial network.  In CVPR, 2017.
  * [45] Cheng-Han Lee, Ziwei Liu, Lingyun Wu, and Ping Luo.  Maskgan: Towards diverse and interactive facial image manipulation.  In CVPR, 2020.
  * [46] Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick.  Microsoft coco: Common objects in context.  In ECCV, 2014.
  * [47] Bingchen Liu, Yizhe Zhu, Kunpeng Song, and Ahmed Elgammal.  Towards faster and stabilized gan training for high-fidelity few-shot image synthesis.  In ICLR, 2020.
  * [48] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo.  Swin transformer: Hierarchical vision transformer using shifted windows.  In ICCV, 2021.
  * [49] David Lopez-Paz and Maxime Oquab.  Revisiting classifier two-sample tests.  In ICLR, 2017.
  * [50] Puneet Mangla, Nupur Kumari, Mayank Singh, Balaji Krishnamurthy, and Vineeth N Balasubramanian.  Data instance prior (disp) in generative adversarial networks.  arXiv preprint arXiv:2012.04256, 2020.
  * [51] Xudong Mao, Qing Li, Haoran Xie, Raymond YK Lau, Zhen Wang, and Stephen Paul Smolley.  Least squares generative adversarial networks.  In CVPR, 2017.
  * [52] Lars Mescheder, Andreas Geiger, and Sebastian Nowozin.  Which training methods for gans do actually converge?  In ICML, 2018.
  * [53] Sangwoo Mo, Minsu Cho, and Jinwoo Shin.  Freeze the discriminator: a simple baseline for fine-tuning gans.  In CVPRW, 2020.
  * [54] Basil Mustafa, Carlos Riquelme, Joan Puigcerver, André Susano Pinto, Daniel Keysers, and Neil Houlsby.  Deep ensembles for low-data transfer learning.  arXiv preprint arXiv:2010.06866, 2020.
  * [55] Atsuhiro Noguchi and Tatsuya Harada.  Image generation from small datasets via batch statistics adaptation.  In ICCV, 2019.
  * [56] Utkarsh Ojha, Yijun Li, Jingwan Lu, Alexei A Efros, Yong Jae Lee, Eli Shechtman, and Richard Zhang.  Few-shot image generation via cross-domain correspondence.  In CVPR, 2021.
  * [57] Maxime Oquab, Leon Bottou, Ivan Laptev, and Josef Sivic.  Learning and transferring mid-level image representations using convolutional neural networks.  In CVPR, 2014.
  * [58] Taesung Park, Ming-Yu Liu, Ting-Chun Wang, and Jun-Yan Zhu.  Semantic image synthesis with spatially-adaptive normalization.  In CVPR, 2019.
  * [59] Gaurav Parmar, Richard Zhang, and Jun-Yan Zhu.  On buggy resizing libraries and surprising subtleties in fid calculation.  arXiv preprint arXiv:2104.11222, 2021.
  * [60] Or Patashnik, Zongze Wu, Eli Shechtman, Daniel Cohen-Or, and Dani Lischinski.  Styleclip: Text-driven manipulation of stylegan imagery.  In ICCV, 2021.
  * [61] Joan Puigcerver, Carlos Riquelme, Basil Mustafa, Cedric Renggli, André Susano Pinto, Sylvain Gelly, Daniel Keysers, and Neil Houlsby.  Scalable transfer learning with expert models.  In ICLR, 2021.
  * [62] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al.  Learning transferable visual models from natural language supervision.  In ICML, 2021.
  * [63] Alec Radford, Luke Metz, and Soumith Chintala.  Unsupervised representation learning with deep convolutional generative adversarial networks.  In ICLR, 2016.
  * [64] Stephan R Richter, Hassan Abu AlHaija, and Vladlen Koltun.  Enhancing photorealism enhancement.  arXiv preprint arXiv:2105.04619, 2021.
  * [65] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al.  Imagenet large scale visual recognition challenge.  IJCV, 115(3):211–252, 2015.
  * [66] Kate Saenko, Brian Kulis, Mario Fritz, and Trevor Darrell.  Adapting visual category models to new domains.  In ECCV, 2010.
  * [67] Alexander Sage, Eirikur Agustsson, Radu Timofte, and Luc Van Gool.  Logo synthesis and manipulation with clustered generative adversarial networks.  In CVPR, 2018.
  * [68] Tim Salimans, Ian Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen.  Improved techniques for training gans.  In NeurIPS, 2016.
  * [69] Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chieh Chen.  Mobilenetv2: Inverted residuals and linear bottlenecks.  In CVPR, 2018.
  * [70] Axel Sauer, Kashyap Chitta, Jens Müller, and Andreas Geiger.  Projected gans converge faster.  In NeurIPS, 2021.
  * [71] Yujun Shen, Ceyuan Yang, Xiaoou Tang, and Bolei Zhou.  Interfacegan: Interpreting the disentangled face representation learned by gans.  IEEE TPAMI, 2020.
  * [72] Assaf Shocher, Yossi Gandelsman, Inbar Mosseri, Michal Yarom, Michal Irani, William T Freeman, and Tali Dekel.  Semantic pyramid for image generation.  In CVPR, 2020.
  * [73] Zhangzhang Si and Song-Chun Zhu.  Learning hybrid image templates (hit) by information projection.  PAMI, 34(7):1354–1367, 2011.
  * [74] Karen Simonyan and Andrew Zisserman.  Very deep convolutional networks for large-scale image recognition.  In ICLR, 2015.
  * [75] Jake Snell, Kevin Swersky, and Richard S Zemel.  Prototypical networks for few-shot learning.  In NeurIPS, 2017.
  * [76] Diana Sungatullina, Egor Zakharov, Dmitry Ulyanov, and Victor Lempitsky.  Image manipulation with perceptual discriminators.  In ECCV, 2018.
  * [77] Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jon Shlens, and Zbigniew Wojna.  Rethinking the inception architecture for computer vision.  In CVPR, 2016.
  * [78] Mingxing Tan and Quoc Le.  Efficientnet: Rethinking model scaling for convolutional neural networks.  In ICML, 2019.
  * [79] Ngoc-Trung Tran, Viet-Hung Tran, Ngoc-Bao Nguyen, Trung-Kien Nguyen, and Ngai-Man Cheung.  Towards good practices for data augmentation in gan training.  arXiv preprint arXiv:2006.05338, 2, 2020.
  * [80] Eric Tzeng, Judy Hoffman, Kate Saenko, and Trevor Darrell.  Adversarial discriminative domain adaptation.  In CVPR, 2017.
  * [81] Sheng-Yu Wang, Oliver Wang, Richard Zhang, Andrew Owens, and Alexei A Efros.  Cnn-generated images are surprisingly easy to spot… for now.  In CVPR, 2020.
  * [82] Ting-Chun Wang, Ming-Yu Liu, Jun-Yan Zhu, Andrew Tao, Jan Kautz, and Bryan Catanzaro.  High-resolution image synthesis and semantic manipulation with conditional gans.  In CVPR, 2018.
  * [83] Yaxing Wang, Abel Gonzalez-Garcia, David Berga, Luis Herranz, Fahad Shahbaz Khan, and Joost van de Weijer.  Minegan: effective knowledge transfer from gans to target domains with few images.  In CVPR, 2020.
  * [84] Yaxing Wang, Chenshen Wu, Luis Herranz, Joost van de Weijer, Abel Gonzalez-Garcia, and Bogdan Raducanu.  Transferring gans: generating images from limited data.  In ECCV, 2018.
  * [85] Tete Xiao, Yingcheng Liu, Bolei Zhou, Yuning Jiang, and Jian Sun.  Unified perceptual parsing for scene understanding.  In ECCV, pages 418–434, 2018.
  * [86] Zhenda Xie, Yutong Lin, Zhuliang Yao, Zheng Zhang, Qi Dai, Yue Cao, and Han Hu.  Self-supervised learning with swin transformers.  arXiv preprint arXiv:2105.04553, 2021.
  * [87] Jason Yosinski, Jeff Clune, Yoshua Bengio, and Hod Lipson.  How transferable are features in deep neural networks?  In NeurIPS, 2014.
  * [88] Fisher Yu, Ari Seff, Yinda Zhang, Shuran Song, Thomas Funkhouser, and Jianxiong Xiao.  Lsun: Construction of a large-scale image dataset using deep learning with humans in the loop.  arXiv preprint arXiv:1506.03365, 2015.
  * [89] Ning Yu, Ke Li, Peng Zhou, Jitendra Malik, Larry Davis, and Mario Fritz.  Inclusive gan: Improving data and minority coverage in generative models.  In ECCV, 2020.
  * [90] Amir R Zamir, Alexander Sax, William Shen, Leonidas J Guibas, Jitendra Malik, and Silvio Savarese.  Taskonomy: Disentangling task transfer learning.  In CVPR, 2018.
  * [91] Matthew D Zeiler and Rob Fergus.  Visualizing and understanding convolutional networks.  In ECCV, 2014.
  * [92] Han Zhang, Tao Xu, Hongsheng Li, Shaoting Zhang, Xiaogang Wang, Xiaolei Huang, and Dimitris N Metaxas.  Stackgan: Text to photo-realistic image synthesis with stacked generative adversarial networks.  In ICCV, 2017.
  * [93] Richard Zhang.  Making convolutional networks shift-invariant again.  In ICML, 2019.
  * [94] Richard Zhang, Phillip Isola, and Alexei A Efros.  Split-brain autoencoders: Unsupervised learning by cross-channel prediction.  In CVPR, 2017.
  * [95] Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang.  The unreasonable effectiveness of deep features as a perceptual metric.  In CVPR, 2018.
  * [96] Miaoyun Zhao, Yulai Cong, and Lawrence Carin.  On leveraging pretrained gans for generation with limited data.  In ICML, 2020.
  * [97] Shengyu Zhao, Jonathan Cui, Yilun Sheng, Yue Dong, Xiao Liang, Eric I Chang, and Yan Xu.  Large scale image completion via co-modulated generative adversarial networks.  In ICLR, 2021.
  * [98] Shengyu Zhao, Zhijian Liu, Ji Lin, Jun-Yan Zhu, and Song Han.  Differentiable augmentation for data-efficient gan training.  In NeurIPS, 2020.
  * [99] Zhengli Zhao, Zizhao Zhang, Ting Chen, Sameer Singh, and Han Zhang.  Image augmentations for gan training.  arXiv preprint arXiv:2006.02595, 2020.
  * [100] Bolei Zhou, Agata Lapedriza, Aditya Khosla, Aude Oliva, and Antonio Torralba.  Places: A 10 million image database for scene recognition.  IEEE TPAMI, 40(6):1452–1464, 2017.
  * [101] Bolei Zhou, Hang Zhao, Xavier Puig, Tete Xiao, Sanja Fidler, Adela Barriuso, and Antonio Torralba.  Semantic understanding of scenes through the ade20k dataset.  International Journal of Computer Vision, 127(3):302–321, 2019\.
  * [102] Jun-Yan Zhu, Philipp Krähenbühl, Eli Shechtman, and Alexei A Efros.  Generative visual manipulation on the natural image manifold.  In ECCV, 2016.
  * [103] Peihao Zhu, Rameen Abdal, John Femiani, and Peter Wonka.  Barbershop: Gan-based image compositing using segmentation masks, 2021\.

Appendix

We show more visualizations and quantitative results to show the efficacy of
our method. Namely, Section A shows qualitative comparisons between our method
and existing state-of-the-art methods, StyleGAN2-ADA [37] and DiffAugment
[98]. In Section B, we show more evaluation of our model on other metrics.
Section C details our training hyperparameters, discriminator architectures,
and selected models in each experiment. In Section D, we discuss the societal
impact of our work.

##  A Qualitative Image Analysis

In Figure 13, we show more randomly generated images by StyleGAN2-ADA and our
method with the same latent code for FFHQ, LSUN Cat, and LSUN Church 1k
training sample setting similar to Figure 5 of the main paper. In many cases,
our method improves the visual quality of samples compared to the baseline.

For the human preference study conducted on the 1k sample setting, Figure 14
shows the sample images for both the cases where users preferred our generated
images vs. StyleGAN2-ADA generated images. Figure 15 shows randomly generated
images by our method, StyleGAN2-ADA, and DiffAugment for varying sample
settings of FFHQ, LSUN Cat, and LSUN Church.

For AFHQ cat, dog, and wild categories, Figure 18 and Figure 19 show the
qualitative comparison between StyleGAN2-ADA and our method similar to Figure
8. Figure 19 and Figure 20 show similar comparison for FFHQ, LSUN Cat, and
LSUN Church 1k training sample setting. Figure 21 shows the latent
interpolation of models trained by our method in the low-shot setting with
only $100-400$ real samples. The smooth interpolation shows that the model is
probably not overfitting on the few real samples.

Figure 12: Model selection using linear probing of pretrained features on LSUN
Cat 1k training sample setting. We plot the linear probe accuracy of
pretrained models in our model bank with the FID of generator trained with
that model. The graph shows correlation between the two. Thus, selecting
models with higher linear probe accuracy in general results in better
generative model with respect to FID metric.

##  B Evaluation metrics

We measure FID using clean-fid library [59] with $50$k generated samples and
complete training dataset as the reference distribution in all our experiments
similar to StyleGAN2-ADA [37] except LSUN Church. For LSUN Church we use 50k
randomly sampled real images as reference distribution following the
evaluation setting of StyleGAN2 [40]. For low-shot 100-300 sample setting we
compute FID and KID with 5k generated samples and full available real dataset
as the reference following DiffAugment [98]. In addition to FID metric
reported in the main paper, we report in Table 8 and Table 9, precision and
recall metrics [43] for FFHQ, LSUN Cat, and LSUN Church experiments with
varying training sample size. We observe that our method improves the recall
metric in all cases and has similar or better precision, particularly in
limited sample settings.

Discriminator Architecture | Dataset
---|---
|  LSUN Cat 1k |  FFHQ 1k
Single-scale (MLP) | 17.21 | 13.31
Multi-scale | 16.16 | 12.55

Table 7: FID$\downarrow$ on Single-scale vs Multi-scale discriminator head for
ViT(CLIP). We observe slightly better FID with a multi-scale discriminator
head compared to a 2-layer MLP head on final classification token feature,
without any significant increase in training time. Therefore, we select the
multi-scale discriminator head for all our experiments on both CLIP and DINO
with ViT-B architecture.

##  C Training and Hyperparameter details

#### Off-the-shelf models and discriminator head architecture.

We provide network details of off-the-shelf models we used in our experiments
in Table 10. For extracting features, we resize both real and fake images to
the resolution that the pretrained network was trained on. For the trainable
discriminator head, we use a Conv-LeakyReLU-Linear-LeakyReLU-Linear
architecture over the spatial features after $2\times$ downsampling for all
pre-trained models except CLIP and DINO. In the case of CLIP and DINO with
ViT-B architecture, we observed that a multi-scale architecture leads to
slightly better results (comparison shown in Table 7). We extract the spatial
features at $4$ and $8$ layers and the final classifier token feature. For
each spatial feature, we use a Conv-LeakyReLU-Conv with downsampling to
predict a $3\times 3$ real vs fake logits similar to PatchGAN [34] and the
loss is averaged over the $3\times 3$ spatial grid. On the classifier token,
we use Linear-LeakyReLU-Linear discriminator head for a global real vs. fake
prediction. The final loss is the sum of losses at the three scales. Extracted
feature size for each model and exact architecture of the trainable
discriminator head are detailed in Table 10.

Figure 13: LSUN Cat, FFHQ, and LSUN Church paired sample comparison in 1k
training dataset setting. For each dataset, the top row shows random samples
of the baseline StyleGAN2-ADA, and the bottom row shows the samples by our
method for the same latent code. As our method finetunes from a StyleGAN2-ADA
model, on average we observe improved image quality with our method for the
same latent code.

Figure 14: Example images shown to users in the human preference study between
our method and current state-of-the-art StyleGAN2-ADA on LSUN Cat, FFHQ, and
LSUN Church 1k training sample setting. Left: example instances where images
generated by our method is preferred by users. Right: where images generated
by StyleGAN2-ADA is preferred. For each dataset, top and bottom row show the
two images generated by StyleGAN2-ADA and our method from the same random
latent code and shown to the user. For LSUN Cat, FFHQ, and LSUN Church our
method is preferred with $63.5\%$, $53.8\%$, and $60.5\%$ as mentioned in the
main paper.

Dataset | StyleGAN2 | DiffAugment | ADA | Ours (w/ ADA)
---|---|---|---|---
+$1^{\text{st}}$ D | +$2^{\text{nd}}$ D | +$3^{\text{rd}}$ D
|  | P $\uparrow$ | R $\uparrow$ | P $\uparrow$ | R $\uparrow$ | P $\uparrow$ | R $\uparrow$ | P $\uparrow$ | R $\uparrow$ | P $\uparrow$ | R $\uparrow$ | P $\uparrow$ | R $\uparrow$
FFHQ | 1k | 0.580 | 0.000 | 0.681 | 0.034 | 0.675 | 0.088 | 0.719 | 0.139 | 0.740 | 0.139 | 0.694 | 0.173
2k | 0.586 | 0.025 | 0.709 | 0.099 | 0.676 | 0.137 | 0.701 | 0.242 | 0.719 | 0.251 | 0.719 | 0.251
10k | 0.669 | 0.191 | 0.704 | 0.256 | 0.700 | 0.255 | 0.683 | 0.334 | 0.687 | 0.342 | 0.697 | 0.351
LSUN Cat | 1k | 0.290 | 0.000 | 0.539 | 0.015 | 0.468 | 0.012 | 0.653 | 0.033 | 0.627 | 0.053 | 0.624 | 0.058
2k | 0.527 | 0.001 | 0.607 | 0.032 | 0.617 | 0.066 | 0.667 | 0.084 | .645 | 0.105 | 0.652 | 0.120
10k | 0.632 | 0.099 | 0.628 | 0.188 | 0.598 | 0.111 | 0.639 | 0.152 | 0.615 | 0.181 | 0.599 | 0.203
LSUN Church | 1k | - | - | 0.593 | 0.015 | 0.554 | 0.038 | 0.652 | 0.047 | 0.609 | 0.065 | 0.645 | 0.063
2k |  |  | 0.613 | 0.042 | 0.604 | 0.075 | 0.637 | 0.100 | 0.626 | 0.107 | 0.649 | 0.114
10k | - | - | 0.567 | 0.256 | 0.617 | 0.108 | 0.662 | 0.132 | 0.645 | 0.112 | 0.643 | 0.133

Table 8: Precision (P) and Recall (R) metrics for experiments on FFHQ and LSUN
datasets with varying training samples from 1k to 10k. Complete training
dataset is used as the reference distribution for calculating the above
metrics and average of 3 evaluation runs is reported. Our method results in
higher recall and precision in all settings. In addition, we observe that as
we add pretrained model based discriminators, recall increases at the cost of
slight decrease in precision.

Dataset | Resolution | StyleGAN2 (F) | Ours (w/ ADA)
---|---|---|---
|  |  | +$1^{\text{st}}$ D | +$2^{\text{nd}}$ D | +$3^{\text{rd}}$ D
|  | FID $\downarrow$ | P $\uparrow$ | R $\uparrow$ | FID $\downarrow$ | P$\uparrow$ | R $\uparrow$ | FID $\downarrow$ | P$\uparrow$ | R $\uparrow$ | FID $\downarrow$ | P$\uparrow$ | R $\uparrow$
FFHQ | 1024 $\times$ 1024 | 2.98 | 0.684 | 0.495 | 3.11 | 0.656 | 0.519 | 3.01 | 0.678 | 0.499 | 3.09 | 0.665 | 0.507
LSUN Cat | 256 $\times$ 256 | 6.86 | 0.606 | 0.318 | 4.55 | 0.626 | 0.341 | 4.19 | 0.608 | 0.355 | 3.98 | 0.598 | 0.381
LSUN Church | 256 $\times$ 256 | 4.28 | 0.594 | 0.391 | 2.05 | 0.596 | 0.447 | 1.81 | 0.603 | 0.451 | 1.72 | 0.612 | 0.451
LSUN Horse | 256 $\times$ 256 | 4.09 | 0.609 | 0.357 | 2.79 | 0.636 | 0.369 | 2.38 | 0.614 | 0.406 | 2.11 | 0.611 | 0.416

Table 9: FID, Preicison (P), and Recall (R) metrics on full-dataset setting.
Our method results in improved recall for most cases and has similar precision
compared to StyleGAN2. A higher recall is usually preferred as with truncation
precision can be recovered [40].

Vision task | Network | Params | Extracted feature size |  $D_{i}$ Architecture
---|---|---|---|---
ImageNet [19] classifier | VGG-16 [93] | 138M | $512\times 7\times 7$ | $\begin{Bmatrix}\text{2}\times\text{avg. downsampling}\\\ \text{Conv3x3: ch}\rightarrow\text{256}\\\ \text{LeakyReLU(0.2)}\\\ \text{Linear: 256}\times\text{h}\times\text{w}\rightarrow\text{256}\\\ \text{LeakyReLU(0.2)}\\\ \text{Linear: 256}\rightarrow\text{1}\end{Bmatrix}$
MoBY [86] | tiny Swin-T | 29M | $768\times 7\times 7$
Face parsing [45] | U-Net | 1.9M | $256\times 8\times 8$
Face normals [2] | U-Net + ResNet | 35M | $512\times 8\times 8$
Segmentation [48] | tiny Swin-T | 29M | $768\times 8\times 8$
Object detection [48] | tiny Swin-T | 29M | $768\times 8\times 8$
CLIP [62] | ViT-B32 | 86M | $\begin{matrix}768\times 7\times 7\\\ 768\times 7\times 7\\\ 512\end{matrix}$ |  $2\times$ $\begin{Bmatrix}\text{Conv3x3: ch}\rightarrow\text{256}\\\ \text{LeakyReLU(0.2)}\\\ \text{2}\times\text{avg. downsample}\\\ \text{Conv3x3: 256}\rightarrow\text{1}\end{Bmatrix}$, $\begin{Bmatrix}\text{Linear: 512}\rightarrow\text{256}\\\ \text{LeakyReLU(0.2)}\\\ \text{Linear: 256}\rightarrow\text{1}\end{Bmatrix}$
DINO [10] | ViT-B16 | 85M | $\begin{matrix}768\times 14\times 14\\\ 768\times 14\times 14\\\ 768\end{matrix}$ |  $2\times$ $\begin{Bmatrix}\text{2}\times\text{avg. downsample}\\\ \text{Conv3x3: ch}\rightarrow\text{128}\\\ \text{LeakyReLU(0.2)}\\\ \text{2}\times\text{avg. downsample}\\\ \text{Conv3x3: 128}\rightarrow\text{1}\end{Bmatrix}$, $\begin{Bmatrix}\text{Linear: 768}\rightarrow\text{128}\\\ \text{LeakyReLU(0.2)}\\\ \text{Linear: 128}\rightarrow\text{1}\end{Bmatrix}$

Table 10: Off-the-shelf Model Bank. We select state-of-the-art feature
extractors and task specific networks to use as an ensemble of off-the-shelf
discriminators during GAN training. We keep the discriminator head
architecture small and fairly similar across different models. In the multi-
scale architecture of CLIP and DINO, we extract the spatial features from 4
and 8 layers and final classification token feature.

#### Linear accuracy analysis of all experiments

Figure 22, Figure 23, and Figure 24 shows the linear probe accuracy of the
pretrained models and the selected model based on that. We calculate linear
probe accuracy on the average of 3 runs (variance is always less than $1.\%$
except in $100$-$400$ low-sample setting where it increases to $\sim 5-8\%$ ).
For the limited sample setting, we use the complete set of real training
samples and the same amount of generated samples. For the full-dataset
setting, we randomly sample a subset of 10k real and generated samples during
linear probe accuracy calculation. We observe diminished variance in the
linear classifier validation accuracy with the increase in sample size. The
computational cost of calculating linear probe accuracy for a model varies
with the sample size and dataset resolution but is always in the order of
$5-10$ minutes, including the time for fake image generation for training
linear classifier as measured on one RTX 3090.

Figure 15: Randomly generated samples by DiffAugment [98], StyleGAN2-ADA [37]
and Our method on 2k and 10k sample setting of FFHQ and LSUN Cat.  Figure 16:
Randomly generated samples by DiffAugment [98], StyleGAN2-ADA [37] and Our
method on LSUN Church 2k and 10k training sample setting.  Figure 17:
Uncurated samples generated by StyleGAN2 [40] and Our method trained on full-
dataset of FFHQ, LSUN Cat, and LSUN Church.  Figure 18: Qualitative comparison
of our method with StyleGAN2-ADA on AFHQ Dog and AFHQ Cat. Left: randomly
generated samples for both methods. Right: Worst FID samples. For both our
model and StyleGAN2-ADA, we independently generate 5k samples and find the
worst-case samples compared to real image distribution. We first fit a
Gaussian model using the Inception [77] feature space of real images. We then
calculate the log-likelihood of each sample given this Gaussian prior and show
the images with minimum log-likelihood. We call these samples Worst FID
samples. Our method shows better worst case image quality on average compared
to StyleGAN2-ADA.  Figure 19: Qualitative comparison of our method with
StyleGAN2-ADA on AFHQ Wild and FFHQ 1k Left: randomly generated samples for
both methods. Right: Worst FID samples as explained in Figure 18. Our method
to a large extent prevents generation of rotated images with extreme artifacts
in case of FFHQ 1k training sample setting.  Figure 20: Qualitative comparison
of our method with StyleGAN2-ADA on LSUN Cat and LSUN Church 1k Left: randomly
generated samples for both methods. Right: Worst FID samples as explained in
Figure 18. Our method does not show any significant improvement in case of
LSUN Cat but is relatively better than StyleGAN2-ADA on LSUN Church on worst
case samples.  Figure 21: Latent interpolation results of models trained with
our method on AnimalFace Cat, Dog [73] and 100-shot Bridge-of-Sighs [98] low
sample datasets. The smooth interpolation suggests that there is little
overfitting in the trained generator.  Figure 22: Linear probe accuracy of
off-the-shelf models during our K-progressive ensemble training on FFHQ and
LSUN Church for different training sample setting. The selected model at each
stage is annotated at the top of the bar-plot. As we include more
discriminators based on off-the-shelf models, linear probe accuracy
corresponding to the pretrained models decreases.  Figure 23: Linear probe
accuracy of off-the-shelf models during our K-progressive ensemble training on
LSUN Cat for different training sample setting and on MetFaces, and AFHQ
categories. The selected model at each stage is annotated at the top of the
bar-plot. As we include more discriminators based on off-the-shelf models,
linear probe accuracy corresponding to the pretrained models decreases.
Figure 24: Linear probe accuracy of off-the-shelf models during our
K-progressive ensemble training on AnimalFace Cat, Dog [73] and 100-shot
Bridge-of-Sighs [98] low-shot datasets and AFHQ categories (transfer from FFHQ
trained generator). In case of transfer from FFHQ, linear probe accuracy is
$100\%$ at the start as human faces and AFHQ categories have a significant
domain gap. The selected model at each stage is annotated at the top of the
bar-plot. As we include more discriminators based on off-the-shelf models,
linear probe accuracy corresponding to the pretrained models decreases.

#### Training hyperparameters and memory requirement.

We keep similar architecture and training hyperparameters as StyleGAN2-ADA
[37] i.e. gamma regularization is $1$, learning rate $0.002$, path length
regularization $2$ for experiments with datasets of $256$ resolution. For
datasets with $512$ resolution, gamma of $0.5$ and learning rate $0.0025$ is
used similar to StyleGAN2-ADA. The ADA target value for original discriminator
in StyleGAN2-ADA is $0.6$. For discriminators based on pre-trained models,
target probability for ADA technique is $0.3$ for all experiments except full-
dataset setting on FFHQ and LSUN categories where its $0.1$. For augmentation,
we used cutout + ADA (bgc) [37] for augmenting real and fake images input to
the pretrained model based discriminator unless DiffAugment is mentioned. In
case of full-dataset experiment where we finetune StyleGAN2 model, original
discriminator has non-augmented real and fake images as input. For experiments
on FFHQ, LSUN Cat, and LSUN Church with varying sample size, number of feature
maps at shallow layers is halved [37]. All experiments are done with a batch
size of $16$ (mini-batch std $4$) on a single RTX 3090 GPUs for $256$
resolution and $4$ GPUs for $512$, $1024$ resolution datasets. In case of LSUN
Horse, we used a batch size of 64.

In all experiments, we train with an ensemble of maximum three pre-trained
model based discriminators. Number of training iterations after which we add
the second model is $1$ million ($1$M) in low-shot generation settings, $4$M
for 1k training sample setting and $8$M for the rest. With the second and
third pretrained model, we train for $1$M training iterations on $<1$k
training sample setting and $2$M otherwise. The GPU memory requirement of our
method is maximum when using VGG-16 model at $\sim 2.5$GB. Next, CLIP and DINO
model with ViT-B architecture have a memory requirement of $\sim 2$GB.
Pretrained models based on tiny Swin-T architecture and face normals and
parsing model lead to $<1$GB of overhead in GPU memory during training.
Compared to the StyleGAN2(F) model architecture training on $256$ resolution
images which required $\sim 10.5$GB of GPU memory on a single RTX 3090, the
overhead in memory with a single pretrained model is $10-25\%$ approximately.
The maximum overhead in memory is when CLIP, DINO and Swin-T based models are
selected by model selection strategy for the three pretrained model based
discriminators. This results in $\sim 4.5$GB of additional memory requirement
as measured on one RTX 3090 while training with our default batch-size of $16$
on $256$ resolution dataset. In future, we hope to explore the use of
efficient [78, 69] computer vision models to reduce the increased memory
requirement of our method.

##  D Societal Impact

Our proposed method is towards improving the image quality of GANs,
specifically in limited sample setting. This can help users in novel content
creation where usually only few relevant samples are available for
inspiration. Also, the faster convergence of our method when used in training
from scratch makes it accessible to a broad set of people as the model can be
trained at a lower computational cost. This can lead to negative societal
impact as well through the creation of fake data and disinformation. One of
the possible solutions to mitigate this can be to ensure reliable detection of
fake generated data [81, 13].