O45 O1em m #3

# A Survey on Generative Modeling with Limited Data, Few Shots, and Zero Shot

Milad Abdollahzadeh  0000-0003-4011-4670 ,  Touba Malekzadeh
milad˙abdollahzadeh, touba˙malekzadeh @sutd.edu.sg  0000-0002-0348-9935 ,
Christopher T.H. Teo<EMAIL_ADDRESS>0009-0006-3801-2658 ,
Keshigeyan Chandrasegaran  0000-0002-6965-5377 ,  Guimeng Liu
0009-0005-3609-1412 and  Ngai-Man Cheung  keshigeyan, guimeng˙liu,
ngaiman˙cheung @sutd.edu.sg  0000-0003-0135-3791
Singapore University of Technology and Design (SUTD)8 Somapah
RdSingapore487372

(2023)

###### Abstract.

In machine learning, generative modeling aims to learn to generate new data
statistically similar to the training data distribution. In this paper, we
survey learning generative models under limited data, few shots and zero shot,
referred to as Generative Modeling under Data Constraint (GM-DC). This is an
important topic when data acquisition is challenging, e.g. healthcare
applications. We discuss background, challenges, and propose two taxonomies:
one on GM-DC tasks and another on GM-DC approaches. Importantly, we study
interactions between different GM-DC tasks and approaches. Furthermore, we
highlight research gaps, research trends, and potential avenues for future
exploration. Project website: https://gmdc-survey.github.io.

Generative Modeling, Generative AI, Generative Adversarial Networks, Diffusion
Models, Variational Auto-Encoder, Few-Shot Learning, Zero-Shot Learning,
Transfer Learning, Data Augmentation.

††copyright: acmcopyright††journalyear: 2023††doi: na††ccs: Computing
methodologies Neural networks††ccs: Computing methodologies Computer
vision11footnotetext: This research is supported by the National Research
Foundation, Singapore under its AI Singapore Programmes (AISG Award No.:
AISG2-RP-2021-021; AISG Award No.: AISG-100E2018-005).

##  1\. Introduction

Generative modeling is a field of machine learning that focuses on learning
the underlying distribution of the training samples, enabling the generation
of new samples that exhibit similar statistical properties to the training
data. Generative modeling has profound impacts in various fields including
computer vision (Ramesh et al., 2022; Karras et al., 2020b; Brock et al.,
2019), natural language processing (Yu et al., 2017; Gulrajani et al., 2017;
Van Den Oord et al., 2017) and data engineering (Antoniou et al., 2017; Karras
et al., 2020a; Tran et al., 2021). Over the years, significant advancements
have been made in generative modeling. Innovative approaches such as
Generative Adversarial Networks (GANs) (Goodfellow et al., 2014; Karras et
al., 2019; Brock et al., 2019; Arjovsky et al., 2017; Choi et al., 2020; Park
et al., 2019; Zhu et al., 2017), Variational Autoencoders (VAEs) (Kingma and
Welling, 2014; Vahdat and Kautz, 2020; Van Den Oord et al., 2017), and
Diffusion Models (DMs) (Rombach et al., 2022; Song et al., 2020; Dhariwal and
Nichol, 2021; Nichol and Dhariwal, 2021) have played a pivotal role in
enhancing the quality and diversity of generated samples. The advancements in
generative modeling have fueled the recent disruption in generative AI,
unlocking new possibilities in various applications such as image synthesis
(Reuters, 2023; Clarke, 2022), text generation (Hern, 2023; Hopkin, 2023),
music composition (Easton and Jacob, 2023; Xu, 2022), genomics (Nguyen et al.,
2023b), and more (Sargent et al., 2023; Koh et al., 2023). The ability to
generate realistic and diverse samples has opened doors to creative
applications and novel solutions (Roose, 2023; Review, 2023).

Figure 1. Research Landscape of GM-DC. The figure shows the interaction
between GM-DC tasks and approaches (main and sub categories), and GM-DC
methods. Tasks are defined in our proposed taxonomy in Tab. 2, and approaches
in our proposed taxonomy in Tab. LABEL:tab:approaches. An interactive version
of this diagram is available at our project website. Best viewed in color and
with zoom.

Research on generative modeling has been mainly focusing on setups with
sizeable training datasets. StyleGAN (Karras et al., 2019) learns to generate
realistic and diverse face images using Flickr-Faces-HQ (FFHQ), a high-quality
dataset of 70k human face images collected from the photo-sharing website
Flickr. The more recent text-to-image generative model is trained on millions
of image-text pairs, e.g. Latent Diffusion Model (Rombach et al., 2022) is
trained on LAION-400M with 400 million samples (Schuhmann et al., 2021).
However, in many domains (e.g. , medical), the collection of data samples is
challenging and expensive.

In this paper, we survey Generative Modeling under Data Constraint (GM-DC).
This research area is important for many domains/ applications where
challenges in data collection exist. We conduct a thorough literature review
on learning generative models under limited data, few shots, and zero shot.
Our survey is the first to provide a comprehensive overview and detailed
analysis of all types of generative models, tasks, and approaches studied in
GM-DC, offering an accessible guide on the research landscape (Fig. 1). We
cover the essential backgrounds, provide detailed analysis of unique
challenges of GM-DC, discuss current trends, and present the latest
advancements in GM-DC.

Our Contributions: i) Trends, technical evolution, and statistics of GM-DC
(Fig. 3; Fig. 4; Sec. 5.1); ii) New insights on GM-DC challenges (Sec. 3.2);
iii) Two novel and detailed taxonomies, one on GM-DC tasks (Sec. 3.1) and
another on GM-DC approaches (Sec. 4); iv) A novel Sankey diagram to visualize
the research landscape and relationship between GM-DC tasks, approaches, and
methods (Fig. 1); v) An organized summary of individual GM-DC works (Sec. 4);
vi) Discussion of future directions (Sec. 5.2). We further provide a project
website with an interactive diagram to visualize GM-DC landscape. Our survey
aims to be an accessible guide to provide fresh perspectives on the current
research landscape, organized pointers to comprehensive literature, and
insightful trends on the latest advances of GM-DC.

Figure 2.  Overall publications statistics in GM-DC. GM-DC Publications
(Left): GM-DC publication trends indicate rising interest in this area. We
remark that the previous survey (Li et al., 2022c) only covers $\sim$27% of
publications discussed in our survey. Publication Venues (Right): The
distribution of publications in major machine learning and computer vision
venues, other venues, and arXiv. Best viewed in color.  Figure 3.  Analysis of
publications in GM-DC. Data Constraints: Different types of data constraints
studied in GM-DC. See Sec. 2 for more details on setups. Models: Different
types of models are studied including Generative Adversarial Network (GAN),
Diffusion Model (DM), and Variational Auto-Encoder (VAE). Tasks: Different GM-
DC tasks that are studied; See Sec. 3.1, and Tab. 2 for details on task
definitions in our proposed task taxonomy. Approaches: Different approaches
that are applied for addressing GM-DC; More details on our proposed taxonomy
of approaches can be found in Sec. 4 and Tab. LABEL:tab:approaches. Best
viewed in color.

Survey on GM-DC is inadequate, and our work aims to fill this gap. We found
only one survey in arXiv on the early work of GM-DC focusing on some aspects
of GM-DC (Li et al., 2022c). This previous survey has focused on a subset of
GM-DC papers, studying only works with GANs as the generative model and a
subset of technical tasks/ approaches. Our survey differentiates itself from
(Li et al., 2022c) in: i) Scope - Our survey is the first to cover all types
of generative models and all GM-DC tasks and approaches (Fig. 3); ii) Scale -
Our study includes 113 papers and covers broad GM-DC works, while previous
survey (Li et al., 2022c) covers only $\approx$27% of works discussed in our
survey (Fig. 2); iii) Timeliness - Our survey collects and surveys the most
up-to-date papers in GM-DC; iv) Detailedness - Our paper includes detailed
visualizations (Sankey diagram, charts) and tables to highlight interactions
and important attributes of GM-DC literature; v) Technical evolution analysis
- Our paper analyzes the evolution of GM-DC tasks and approaches, providing
new perspectives on recent advances; vi) Horizon analysis - Our paper
discusses distinctive obstacles encountered in GM-DC and identifies avenues
for future research.

The rest of the paper is organized as follows. In Sec. 2 we provide the
necessary background. In Sec. 3, we discuss GM-DC tasks and unique challenges.
In Sec. 4, we analyze GM-DC approaches and methods. In Sec. 5, we discuss open
research problems and future directions. Sec. 6 concludes the survey.

##  2\. Background

In this section, we first define ‘domain’ and ‘generative modeling’, then we
discuss common approaches of generative modeling and data constraints studied
in GM-DC.

Domain. In this survey, a domain consists of two components: i) a sample space
$\mathcal{X}$, and ii) a marginal probability distribution $P_{data}$, which
models the probability of samples from $\mathcal{X}$ (Pan and Yang, 2009).
This is written as $\mathcal{D}=\\{\mathcal{X},P_{data}\\}$, and $x\sim
P_{data}\in\mathcal{X}$ denoting a sample in this space. An example of a
domain is the domain of image of human faces:
$\mathcal{D}^{h}=\\{\mathcal{X},P_{data}^{h}\\}$. Here $\mathcal{X}$ is the
sample space of images, and $P_{data}^{h}$ is the probability distribution of
human faces.

Generative Modeling. Given a set of training sample $x$ of a domain
$\mathcal{D}=\\{\mathcal{X},P_{data}\\}$, i.e., with an underlying probability
distribution $P_{data}$, generative modeling aims to learn to capture
$P_{data}$ —sometimes also denoted as $P(x)$ in literature. The result of
generative modeling is a generative model $G$ encoding a probability
distribution $P_{model}$. The learning objective is to have $P_{model}$
similar to $P_{data}$ statistically. After the training, $G$ can generate
samples following $P_{model}$. For example, generative modeling with a
training set of human face images aims to learn to capture $P_{data}^{h}$,
thereby the resulting $G^{h}$ can generate human face images that are
statistically similar to samples from $P_{data}^{h}$. We also refer to the
domain of training samples as target domain.

Conditional vs Unconditional Sample Generation. After learning the underlying
distribution of data $P_{data}$, the generative model can generate new samples
by sampling from the learned distribution $P_{model}$. Typically, generation
starts with sampling a random vector $z$ —also called latent code— as input.
Then, this input is passed into the generative model $G$ to transform the
latent code into a new sample $G(z)\sim P_{model}$. Ideally, a good generator
is able to capture the characteristics, quality and diversity of the training
dataset, i.e. , $P_{model}$ is similar to $P_{data}$ statistically. If an
additional condition $c$ (like a class label or attribute) is used alongside
with the latent code to steer the sample generation towards $c$, the sample
generation is called conditional generation: $G(z,c)$.

Figure 4. Illustration of the timeline when a GM-DC task/approach was
introduced based on our proposed taxonomies: task taxonomy (details in Sec.
3.1, and Tab. 2), and approach taxonomy (details in Sec. 4, and Tab.
LABEL:tab:approaches). Best viewed in color.

###  2.1. Approaches for Generative Modeling

Earlier works on generative models study Gaussian Mixture Models (Reynolds et
al., 2009), Hidden Markov Models (Phung et al., 2005), Latent Dirichlet
Allocation (Chauhan and Shah, 2021) and Boltzmann Machines (Ackley et al.,
1985). With the introduction of deep neural networks, recent works study
powerful generative models, particularly those for image generation, which
most GM-DC works focus on.

Variational Auto Encoders (VAE) (Pouyanfar et al., 2018). VAE is a variant of
Auto-Encoder (AE) (Zhai et al., 2018), where both consist of the encoder and
decoder networks. AE focuses on dimensional reduction. The encoder in AE
learns to map an input $x$ into a latent (compressed) representation,
$z=E(x)$. Then, the decoder aims to reconstruct the image from that latent
representation, $\hat{x}=D(z)$. Model parameters are optimized with the
following reconstruction loss:

(1) $\mathcal{L}_{rec}=||x-D(z)||_{2}$

AEs are notorious for latent space irregularity making them improper for
sample generation (Kingma et al., 2019). VAE aims to address this problem by
enforcing $E$ to return a normal distribution over latent space. Assuming a
distribution $z\sim\mathcal{N}(\mu,\sigma^{2})$ for latent space, this is done
by adding the KL-divergence term to the loss function:

(2)
$\mathcal{L}=||x-D(z)||_{2}+KL(\mathcal{N}(\mu,\sigma^{2}),\mathcal{N}(0,I))$

Due to the challenges of direct maximization of the likelihood in pixel space,
Vector-Quantized VAE (VQ-VAE) proposes tokenization where a codebook
$\mathbf{e}_{k}$, $k\in 1,\dots,K$ is used to quantize the embeddings $E(x)$
into visual tokens (indices), acting like a lookup table. In addition, a
latent prior of the visual tokens is predicted (usually using a transformer),
and the decoder is modified to map the visual tokens into the image space.

Table 1. List of common datasets used in GM-DC works. Number of samples (#
Samples) refers to the sample size of the entire dataset. In GM-DC
experiments, usually, only a subset of the dataset is used. We remark that  ❍
/  ● denotes the absence/presence of the dataset under the data constraint
settings: LD: Limited-Data, FS: Few-Shot and ZS: Zero-Shot, and Labels
indicate if training labels are available (but not necessarily used).

Dataset | Description | # Samples | Resolution | LD | FS | ZS | Labels
---|---|---|---|---|---|---|---
Flickr-Faces-HQ (FFHQ) (Karras et al., 2019) |  Images with human faces, containing variation in terms of age, ethnicity, and image background. | 70K | 1024$\times$1024 |  ● |  ● |  ● |  ❍
Large-scale Scene Understanding (LSUN) (Yu et al., 2015) |  Images with large-scale scene containing 10 scene and 20 object categories. | 3M | 256$\times$256 |  ● |  ● |  ❍ |  ●
MetFace (Karras et al., 2020a) |  Images depicting paintings, drawings, and statues of human faces | 1336 | 1024$\times$1024 |  ● |  ● |  ❍ |  ❍
BreCaHAD (Aksac et al., 2019) |  Images of breast cancer histopathology. | 162 | 1360$\times$1024 |  ● |  ❍ |  ❍ |  ❍
Animal FacesHQ (AFHQ) (Choi et al., 2020) |  Images of animal faces in the domains of cat, dog, and wildlife. | 15K | 512$\times$512 |  ● |  ● |  ❍ |  ❍
CIFAR-10 (Krizhevsky et al., 2009) |  Images including objects and animals. | 60K | 32$\times$32 |  ● |  ❍ |  ❍ |  ●
CIFAR-100 (Krizhevsky et al., 2009) |  A dataset similar to CIFAR-10, but with 100 classes | 60K | 32$\times$32 |  ● |  ❍ |  ❍ |  ●
100-shot Obama/ Gumpy Cat/Panda (Zhao et al., 2020b) |  Colored images of Obama/Gumpy Cat/Panda | 100 | 256$\times$256 |  ● |  ❍ |  ❍ |  ❍
Sketches (Wang and Tang, 2008) |  Face sketches in frontal pose, normal lighting, and neutral expressions | 606 | 256$\times$256 |  ❍ |  ● |  ❍ |  ❍
Sunglasses (Ojha et al., 2021) |  Images of human faces wearing sunglasses. | 2700 | 256$\times$256 |  ❍ |  ● |  ❍ |  ❍
Babies (Ojha et al., 2021) |  Images of baby faces. | 2500 | 256$\times$256 |  ❍ |  ● |  ❍ |  ❍
Artistic-Faces (Yaniv et al., 2019) |  Images containing 160 artistic portraits of 16 different artists. | 160 | 256$\times$256 |  ❍ |  ● |  ❍ |  ❍
Haunted houses (Yu et al., 2015) |  Images of haunted houses | 1K | 256$\times$256 |  ❍ |  ● |  ❍ |  ❍
Wrecked cars (Yu et al., 2015) |  Images of wrecked cars | 1K | 256$\times$256 |  ❍ |  ● |  ❍ |  ❍

Table 2. Our proposed taxonomy for tasks in GM-DC. For each task, we extract
their key characteristics. [Attributes] C: Conditional generation, P: Pre-
trained generator given, I: Images (as input), TP: Text-Prompt (as input), X:
X(Cross)-domain adaptation; [Data Constraint] LD: Limited-Data, FS: Few-Shot,
ZS: Zero-Shot.  ❍ /  ● denotes the absence/presence, respectively. Best viewed
in color.

Task | Attributes | Data Constraint |  Task Illustration
---|---|---|---
| C | P | I | TP |  X | LD | FS | ZS |
uGM-1 |  ❍ |  ❍ |  ● |  ❍ |  ❍ |  ● |  ❍ |  ❍ |
Description: Given $K$ samples from a domain $\mathcal{D}$, learn to generate
diverse and high-quality samples from $\mathcal{D}$
Example: ADA (Karras et al., 2020a) learns a StyleGAN2 using 1k images from
AFHQ-Dog
uGM-2 |  ❍ |  ● |  ● |  ❍ |  ● |  ● |  ● |  ❍ |
Description: Given a pre-trained generator on a source domain
$\mathcal{D}_{s}$ and $K$ samples from a target domain $\mathcal{D}_{t}$,
learn to generate diverse and high-quality samples from $\mathcal{D}_{t}$
Example: CDC (Ojha et al., 2021) adapts a pre-trained GAN on FFHQ to Sketches
using 10 samples
uGM-3 |  ❍ |  ● |  ❍ |  ● |  ● |  ❍ |  ❍ |  ● |
Description: Given a pre-trained generator on a source domain
$\mathcal{D}_{s}$ and a text prompt describing a target domain
$\mathcal{D}_{t}$, learn to generate diverse and high-quality samples from
$\mathcal{D}_{t}$
Example: NADA (Gal et al., 2022b) adapts pre-trained GAN on FFHQ to the
painting domain using ‘Fernando Botero Painting’ as input
cGM-1 |  ● |  ❍ |  ● |  ❍ |  ❍ |  ● |  ❍ |  ❍ |
Description: Given $K$ samples with class labels from a domain $\mathcal{D}$,
learn to generate diverse and high-quality samples conditioning on the class
labels from $\mathcal{D}$
Example: CbC (Shahbazi et al., 2022) trains conditional generator on 20
classes of ImageNet Carnivores using 100 images per class
cGM-2 |  ● |  ● |  ● |  ❍ |  ❍ |  ❍ |  ● |  ❍ |
Description: Given a pre-trained generator on the seen classes $C_{seen}$ of a
domain $\mathcal{D}$ and $K$ samples with class labels from unseen classes
$C_{unseen}$ of $\mathcal{D}$, learn to generate diverse and high-quality
samples conditioning on the class labels for $C_{unseen}$ from $\mathcal{D}$
Example: LoftGAN (Gu et al., 2021) learns from 85 classes of Flowers to
generate images for an unseen class with only 3 samples
cGM-3 |  ● |  ● |  ● |  ❍ |  ● |  ● |  ● |  ❍ |
Description: Given a pre-trained generator on a source domain
$\mathcal{D}_{s}$ and $K$ samples with class labels from a target domain
$\mathcal{D}_{t}$ , learn to generate diverse and high-quality samples
conditioning on the class labels from $\mathcal{D}_{t}$
Example: VPT (Sohn et al., 2023) adapts a pre-trained conditional generator on
ImageNet to Places365 with 500 images per class
IGM |  ❍ |  ❍ |  ● |  ❍ |  ❍ |  ❍ |  ● |  ❍ |
Description: Given $K$ samples (usually $K=1$) and assuming rich internal
distribution for patches within these samples, learn to generate diverse and
high-quality samples with the same internal patch distribution
Example: SinDDM (Kulikov et al., 2023) trains a generator using a single image
of Marina Bay Sands, and generates variants of it
SGM |  ❍ |  ● |  ● |  ● |  ❍ |  ❍ |  ● |  ❍ |
Description: Given a pre-trained generator, $K$ samples of a particular
subject, and a text prompt, learn to generate diverse and high-quality samples
containing the same subject
Example: DreamBooth (Ruiz et al., 2023) trains a generator using 4 images of a
particular backpack and adapts it with a text-prompt to be in the ‘grand
canyon’

Generative Adversarial Models (GAN) (Saxena and Cao, 2021; Jabbar et al.,
2021). GAN applies an adversarial approach to learn the distribution of data
$P_{data}$. It consists of a generator $G$ and a discriminator $D$ playing a
min-max game. Specifically, given the latent code $z$, the $G$ learns to
generate the images $G(z)$, $z\sim P_{z}$, where $P_{z}$ is usually a Gaussian
distribution. Then, $D$ learns to distinguish the real images $x\sim P_{data}$
from the generated ones $G(z)\sim P_{model}$. The $D$ and $G$ are optimized by
respectively maximizing and minimizing the following value function:

(3) $\mathcal{V}(D,G)=\mathbb{E}_{x\sim p_{data}}[\log D(x)]+\mathbb{E}_{z\sim
p_{z}}[\log(1-D(G(z)))]$

Flow-based Models (Ho et al., 2019). The flow-based model includes a series of
invertible yet differentiable functions $f$, between latent distribution
$P_{z}$, and data distribution $P_{data}$. The following log-likelihood
function is maximized to train $f(.|\theta)$:

(4) $\max_{\theta}\textstyle\sum_{i=1}^{K}\log
P_{z}(f(x^{(i)}|\theta))+\log|\det Df(x^{(i)}|\theta)|$

For ease of discussion, we simplify the model as a single flow and denote the
training samples with $\\{x^{(i)}\\}^{K}_{i=1}$, and the Jacobian of $f(x)$ as
$Df(x)$. We remark that, unlike VAEs that estimate the lower bounds of the
log-likelihood, flow-based models evaluate the exact log-likelihood in their
loss function.

Diffusion Models (DM) (Kingma et al., 2021). DM leverages the concept of the
diffusion process from stochastic calculus and consists of forward diffusion
and reverse diffusion processes. In the forward diffusion process, based on
the foundations of Markov chains, the noise $\epsilon\sim\mathcal{N}(0,I)$ is
iteratively added to data samples until it approaches an isotropic Gaussian
distribution. Then, in the backward process, the DM learns to denoise the
noisy vector $x_{T}$ and reconstruct the data samples $x_{0}$. This is done by
learning the noise estimation model $\epsilon_{\theta}$ with minimizing the
following loss function (Ho et al., 2020):

(5)
$\mathcal{L}=\mathbb{E}_{t,x_{0},\epsilon}[||\epsilon-\epsilon_{\theta}(\sqrt{\bar{\alpha}_{t}}x_{0}+\sqrt{1-\bar{\alpha}_{t}}\epsilon,t)||_{2}]$

Then, during the generation process, DM first samples a noise
$x_{T}\sim\mathcal{N}(0,I)$, and utilizes the learned noise function
$\epsilon_{\theta}$ to iteratively apply the following denoising process (Ho
et al., 2020):

(6)
$x_{t-1}=\frac{1}{\sqrt{\alpha_{t}}}(x_{t}-\frac{1-\alpha_{t}}{\sqrt{1-\bar{\alpha}}}\epsilon_{\theta}(x_{t},t))+\sqrt{\beta_{t}}\epsilon,\quad
t\in[0,T]$

Here, $x_{t}$ is the generated sample at step $T-t$, $\beta_{t}$ is variance
scheduler, $\alpha_{t}=1-\beta_{t}$ and
$\bar{\alpha_{t}}=\prod^{t}_{s=1}\alpha_{s}$.

Remark. We remark that among discussed models, only GANs, DMs, and VAEs are
adopted in the context of GM-DC.

###  2.2. Data Constraints and Commonly Used Datasets

In GM-DC, three data constraints have been considered in most works: (i)
_Limited data (LD)_ , when 50 to 5,000 training samples are given; (ii) _Few-
Shot (FS)_ , when 1 to 50 training samples are given; (iii) _Zero-Shot (ZS)_ ,
when no training samples are given. Training under these data constraints
often results in various problems e.g. over-fitting. We remark that these
ranges are the typically used values as there are no fixed definitions in the
literature. Tab. 1 lists the most common datasets used in GM-DC with related
details.

##  3\. Generative Modeling under Data Constraint: Task Taxonomy, Challenges

In this section, first, we present our proposed taxonomy on different GM-DC
tasks (Sec. 3.1) highlighting their relationships and differences based on
their attributes, e.g. unconditional or conditional generation. Then, we
present the unique challenges of GM-DC (Sec. 3.2), including new insights such
as domain proximity, and incompatible knowledge transfer. Later, in Sec. 4, we
present our proposed taxonomy on approaches for GM-DC, with a detailed review
of individual work organized under our proposed taxonomy.

###  3.1. Generative Modeling under Data Constraint: A Taxonomy on Tasks

The goal of GM-DC is to learn to generate diverse and high-quality samples
given only a small number of training samples. A number of GM-DC setups have
been studied in different works (Fig. 4). In this section, we propose a GM-DC
task taxonomy to categorize setups in different works. Tab. 2 tabulates our
GM-DC task taxonomy.

  1. (1)

Unconditional generative modeling under data constraint (uGM-1).

###### Definition 0 (uGM-1).

Given $K$ samples from domain $\mathcal{D}$, learn to generate diverse and
high-quality samples from $\mathcal{D}$.

Without leveraging other side information, existing work has studied uGM-1
under limited samples ranging from 100 to several thousands. uGM-1 is an
important task, especially for a domain that is distant from common domains,
e.g. medical images which are distant from common personal photos in terms of
content and characteristics. In such scenarios, leveraging from common domains
would not provide any advantage.

  2. (2)

Unconditional generative modeling under data constraint with pre-trained
generator and cross-domain adaptation (uGM-2).

###### Definition 0 (uGM-2).

Given a pre-trained generator on a source domain $\mathcal{D}_{s}$ (with
numerous and diverse samples) and $K$ samples from a target domain
$\mathcal{D}_{t}$, learn to generate diverse and high-quality samples from
$\mathcal{D}_{t}$.

uGM-2 is similar to uGM-1, except that a pre-trained generator on another
source domain $\mathcal{D}_{s}$ is additionally given. uGM-2 is a major task
in GM-DC and has been studied in many works. In most works, close proximity in
semantic between $\mathcal{D}_{s}$ and $\mathcal{D}_{t}$ is assumed, e.g.
$\mathcal{D}_{s}$ is photos of human faces, $\mathcal{D}_{t}$ is sketches of
human faces. For uGM-2, transfer learning has been a popular approach to
tackle this task driving GM-DC into the few-shot regime, e.g. only 10 samples
from $\mathcal{D}_{t}$ are given (Li et al., 2020) (See Sec. 4 for the
taxonomy of GM-DC approaches). Recent work has started to look into the
challenging setup when $\mathcal{D}_{s}$ and $\mathcal{D}_{t}$ are more
semantically apart (Zhao et al., 2022a), e.g. $\mathcal{D}_{s}$ is photos of
human faces, $\mathcal{D}_{t}$ is photos of cat faces. See Sec. 3.2 for
further discussion on domain proximity in GM-DC.

  3. (3)

Unconditional generative modeling under data constraint with pre-trained
generator and cross-domain adaptation, using text prompt (uGM-3).

###### Definition 0 (uGM-3).

Given a pre-trained generator on a source domain $\mathcal{D}_{s}$ (with
numerous and diverse samples) and a text prompt describing a target domain
$\mathcal{D}_{t}$, learn to generate diverse and high-quality samples from
$\mathcal{D}_{t}$.

uGM-3 is similar to uGM-2, except that a text prompt is provided to describe
$\mathcal{D}_{t}$ instead of samples from $\mathcal{D}_{t}$. Particularly,
this task requires generating samples from $\mathcal{D}_{t}$ without seeing
any sample from that domain, i.e. zero-shot domain adaptation. Important work
to tackle this task leverages recent large vision-language models to provide
textual direction to guide the adaptation of the pre-trained generator to
$\mathcal{D}_{t}$ (Gal et al., 2022b).

  4. (4)

Conditional generative modeling under data constraint (cGM-1).

###### Definition 0 (cGM-1).

Given $K$ samples with class labels from a domain $\mathcal{D}$, learn to
generate diverse and high-quality samples conditioning on the class labels
from $\mathcal{D}$.

cGM-1 is similar to uGM-1 but focuses on conditional generation, i.e. inputs
to the generator include a random latent vector and a class label. Conditional
generative models such as BigGAN (Brock et al., 2019) could achieve high-
quality image generation when they are trained on large-scale datasets e.g.
ImageNet. However, under limited data, it is challenging to achieve diverse
and high-quality conditional sample generation. As a natural extension of
uGM-1, data augmentation has been studied for cGM-1 among other approaches,
see Sec. 4.

  5. (5)

Conditional generative modeling under data constraint with pre-trained
generator (cGM-2).

###### Definition 0 (cGM-2).

Given a pre-trained generator on the seen classes $C_{seen}$ of a domain
$\mathcal{D}$, and $K$ samples with class labels from unseen classes
$C_{unseen}$ of $\mathcal{D}$, learn to generate diverse and high-quality
samples conditioning on the class labels for $C_{unseen}$ from $\mathcal{D}$.

cGM-2 is similar to cGM-1, except that a pre-trained generator on the seen
classes $C_{seen}$ is additionally given. Note that in cGM-2, $C_{seen}$ and
$C_{unseen}$ contain disjoint classes, but both of them are from the same
domain $\mathcal{D}$. For example, (Shahbazi et al., 2021) studies the setup
when CIFAR100 (Krizhevsky et al., 2009) is partitioned into 80 seen classes
for the pre-trained generator and 20 unseen classes as the target, with 100
samples per unseen class given for training. Meta-learning and transfer
learning (regularizer-based fine-tuning, etc.) have been effective approaches
for cGM-2, see Sec. 4.

  6. (6)

Conditional Generative Modeling under data constraint with pre-trained
generator and cross-domain adaptation (cGM-3).

###### Definition 0 (cGM-3).

Given a pre-trained generator on a source domain $\mathcal{D}_{s}$ (with
numerous and diverse samples) and $K$ samples with class labels from a target
domain $\mathcal{D}_{t}$, learn to generate diverse and high-quality samples
conditioning on the class labels from $\mathcal{D}_{t}$.

Table 3.  Our proposed taxonomy for approaches in GM-DC. For each approach,
the addressed GM-DC tasks (see Tab. 2 for task definitions) and the data
constraints are indicated. A detailed list of methods under each sub-category
is also tabulated (some methods are under multiple categories).  ❍ /  ●
denotes the absence/presence of the tasks commonly addressed by each approach,
and the data constraints usually considered: LD: Limited-Data, FS: Few-Shot
and ZS: Zero-Shot.  Transfer Learning (Sec. 4.1)
---
Description: |  Improve GM-DC on target domain by knowledge of a generator pre-trained on source domain (with numerous and diverse samples).
Task: |  uGM-1  ❍ uGM-2  ● uGM-3  ● cGM-1  ❍ cGM-2  ● cGM-3  ● IGM  ❍ SGM  ● Data constraint: LD  ● FS  ● ZS  ●
1) Regularizer-based Fine-Tuning: Explore regularizers to preserve source
generators’ knowledge.
_Methods:_ TGAN(Wang et al., 2018), BSA(Noguchi and Harada, 2019), FreezeD(Mo
et al., 2020), EWC(Li et al., 2020), CDC(Ojha et al., 2021),
cGANTransfer(Shahbazi et al., 2021), W3(Grigoryev et al., 2022), C3(Lee et
al., 2021), DCL(Zhao et al., 2022b), RSSA(Xiao et al., 2022), fairTL(Teo et
al., 2023), GenOS(Zhang et al., 2022a) , SVD(Robb et al., 2020), D3-TGAN(Wu et
al., 2023), JoJoGAN(Chong and Forsyth, 2022), KDFSIG(Hou et al., 2022a),
CtlGAN(Wang et al., 2022b), ICGAN(Casanova et al., 2021), MaskD(Zhu et al.,
2022b), F3(Kato et al., 2023), ICGAN (Casanova et al., 2021), DDPM-PA (Zhu et
al., 2022c), DWSC (Hou et al., 2022b), CSR (Gou et al., 2023), ProSC (Moon et
al., 2023)
2) Latent Space: Explore latent space of source generator to identify suitable
knowledge for adaptation.
_Methods:_ MineGAN(Wang et al., 2020), MineGAN++(Wang et al., 2021),
LCL(Mondal et al., 2023), WeditGAN(Duan et al., 2023), GenDA(Yang et al.,
2021b), SiSTA (Thopalli et al., 2023), MultiDiffusion (Bar-Tal et al., 2023)
3) Modulation: Leverage trainable modulation weights on top of frozen weights
of the source generator.
_Methods:_ AdaFMGAN(Zhao et al., 2020a), GAN-Memory (Cong et al., 2020), CAM-
GAN(Varshney et al., 2021), AdAM(Zhao et al., 2022a), DynaGAN(Kim et al.,
2022), HyperDomainNet(Alanov et al., 2022)
4) Natural Language-guided: Use the feedback of vision-language models to
adapt the source generator with text prompts.
_Methods:_ StyleGAN-NADA(Gal et al., 2022b), MTG(Zhu et al., 2022a),
HyperDomainNet(Alanov et al., 2022), DiFa(Zhang et al., 2022b), OneCLIP(Kwon
and Ye, 2023), IPL(Guo et al., 2023), SINE(Zhang et al., 2023a),
DreamBooth(Ruiz et al., 2023), MCC(Kumari et al., 2023), Textual-Inversion(Gal
et al., 2022a), SpecialistDiffusion(Lu et al., 2023), BLIP-Diffusion(Li et
al., 2023b)
5) Adaptation-Aware: Preserve the source generator’s knowledge that is
important to the adaptation task.
_Methods:_ AdAM(Zhao et al., 2022a), RICK(Zhao et al., 2023a)
6) Prompt Tuning: Freeze the source generator and add/ generate visual prompts
to guide generation for the target domain.
_Methods:_ VPT (Sohn et al., 2023)
Data Augmentation (Sec. 4.2)
Description: |  Improve GM-DC by increasing coverage of the data distribution by applying various transformations on the given samples.
Task: |  uGM-1  ● uGM-2  ❍ uGM-3  ❍ cGM-1  ❍ cGM-2  ❍ cGM-3  ❍ IGM  ❍ SGM  ❍ Data constraint: LD  ● FS  ❍ ZS  ❍
1) Image-Level Augmentation: Apply data transformations on image space.
_Methods:_ ADA (Karras et al., 2020a), DiffAugment(Zhao et al., 2020b),
IAG(Zhao et al., 2020c), DiffusionGAN(Wang et al., 2023b), bCR(Zhao et al.,
2021), CR-GAN(Zhang et al., 2020), APA (Jiang et al., 2021),
PatchDiffusion(Wang et al., 2023a)
2) Feature-Level Augmentation: Apply data transformations on the feature
space.
_Methods:_ AdvAug(Chen et al., 2021), AFI(Dai et al., 2022)
3) Transformation-Driven Design: Leverage the information of individual
transformations to design an efficient learning mechanism.
_Methods:_ DAG(Tran et al., 2021), SSGAN-LA(Hou et al., 2021)
Network Architectures (Sec. 4.3)
Description: |  Design specific architecture for the generator to improve its learning under data constraints.
Task: |  uGM-1  ● uGM-2  ❍ uGM-3  ❍ cGM-1  ● cGM-2  ❍ cGM-3  ❍ IGM  ❍ SGM  ❍ Data constraint: LD  ● FS  ❍ ZS  ❍
1) Feature Enhancement: Design additional modules/ layers to enhance/ retain
the feature maps of the generator for better generative modeling.
_Methods:_ FastGAN(Liu et al., 2021), MoCA(Li et al., 2022a), DFSGAN(Yang et
al., 2023a), SCHA-VAE (Giannone and Winther, 2022)
2) Ensemble Large Pre-trained Vision Models: Improve architecture by
integrating pre-trained vision models to enable more accurate GM-DC.
_Methods:_ Vision-aided GAN(Kumari et al., 2022), ProjectedGAN (Sauer et al.,
2021)
3) Dynamic Network Architecture: Improve generative learning with limited data
by evolving the generator architecture during training.
_Methods:_ CbC(Shahbazi et al., 2022), DynamicD(Yang et al., 2022a),
AdvAug(Chen et al., 2021), Re-GAN(Saxena et al., 2023), AutoInfoGAN (Shi et
al., 2023)
Multi-Task Objectives (Sec. 4.4)
Description: |  Introduce additional task(s) to extract generalizable representations that are useful for all tasks, to reduce overfitting under data constraints.
Task: |  uGM-1  ● uGM-2  ● uGM-3  ❍ cGM-1  ● cGM-2  ❍ cGM-3  ❍ IGM  ❍ SGM  ❍ Data constraint: LD  ● FS  ● ZS  ❍
1) Regularizer: Add an additional task objective as a regularizer to prevent
an undesirable behaviour during training generative model.
_Methods:_ LeCam(Tseng et al., 2021), DigGAN(Fang et al., 2022), MDL(Kong et
al., 2022), RegLA(Hou, 2023)
2) Contrastive Learning: Introduce a pretext task to enhance the learning
process of the generative model.
_Methods:_ InsGen(Yang et al., 2021a), FakeCLR(Li et al., 2022b), DCL(Zhao et
al., 2022b), C3(Lee et al., 2021), ctlGAN(Wang et al., 2022b), IAG(Zhao et
al., 2020c), CML-GAN(Phaphuangwittayakul et al., 2022)
3) Masking: Mask a part of the image/ information to increase the task
hardness and prevent learning the trivial solutions.
_Methods:_ MaskedGAN(Huang et al., 2022), MaskD(Zhu et al., 2022b), DMD (Zhang
et al., 2023b)
4) Knowledge Distillation: Add a task objective that enforces the generator to
follow a strong teacher.
_Methods:_ KD-DLGAN(Cui et al., 2023), KDFSIG(Hou et al., 2022a)
5) Prototype Learning: Emphasize learning prototypes for samples/ concepts
within the distribution as an additional task objective.
_Methods:_ ProtoGAN(Yang et al., 2023b), MoCA(Li et al., 2022a)
6) Other Multi-Task Objectives: Apply other types of multi-task objectives
including co-training, patch-level learning, and diffusion.
_Methods:_ GenCo(Cui et al., 2022), PatchDiffusion(Wang et al., 2023a),
AnyRes-GAN(Chai et al., 2022) , DiffusionGAN(Wang et al., 2023b), D2C(Sinha et
al., 2021), AdaptiveIMLE (Aghabozorgi et al., 2023), FSDM (Giannone et al.,
2022)
Exploiting Frequency Components (Sec. 4.5)
Description: |  Exploit frequency components to improve learning the generative model by reducing frequency bias.
Task: |  uGM-1  ● uGM-2  ❍ uGM-3  ❍ cGM-1  ❍ cGM-2  ● cGM-3  ❍ IGM  ❍ SGM  ❍ Data constraint: LD  ● FS  ● ZS  ❍
_Methods:_ FreGAN(Yang et al., 2022c), WaveGAN(Yang et al., 2022b),
MaskedGAN(Huang et al., 2022), Gen-co(Cui et al., 2022)
Meta-Learning (Sec. 4.6)
Description: |  Learn meta-knowledge from seen classes to improve generator learning for unseen classes.
Task: |  uGM-1  ❍ uGM-2  ❍ uGM-3  ❍ cGM-1  ❍ cGM-2  ● cGM-3  ❍ IGM  ❍ SGM  ❍ Data constraint: LD  ❍ FS  ● ZS  ❍
1) Optimization: Learn initialization weights from the seen classes as meta-
knowledge to enable quick adaptation to unseen classes.
_Methods:_ GMN(Bartunov and Vetrov, 2018), FIGR(Clouâtre and Demers, 2019),
Dawson(Liang et al., 2020), FAML(Phaphuangwittayakul et al., 2021), CML-
GAN(Phaphuangwittayakul et al., 2022)
2) Transformation: Learn sample transformations from the seen classes as meta-
knowledge and use them for sample generation for unseen classes.
_Methods:_ DAGAN(Antoniou et al., 2017), DeltaGAN(Hong et al., 2022a),
Disco(Hong et al., 2022b), AGE(Ding et al., 2022), SAGE(Ding et al., 2023),
HAE(Li et al., 2022d), LSO (Zheng et al., 2023)
3) Fusion: Learn to fuse the samples of the seen classes as meta-knowledge,
and apply learned meta-knowledge to generation for unseen classes.
_Methods:_ MatchingGAN(Hong et al., 2020a), F2GAN(Hong et al., 2020b),
LofGAN(Gu et al., 2021), WaveGAN(Yang et al., 2022b), AMMGAN(Li et al., 2023d)
Modeling Internal Patch Distribution (Sec. 4.7)
Description: |  Learn the internal patch distribution within one image to generate diverse samples with the same visual content (patch distribution).
Task: |  uGM-1  ❍ uGM-2  ❍ uGM-3  ❍ cGM-1  ❍ cGM-2  ❍ cGM-3  ❍ IGM  ● SGM  ❍ Data constraint: LD  ❍ FS  ● ZS  ❍
1) Progressive Training: Train a generative model progressively to learn the
patch distribution at different scales/ noise levels.
_Methods:_ SinDiffusion(Wang et al., 2022a), SinDDM(Kulikov et al., 2023),
Deff-GAN(Kumar and Sivakumar, 2023), BlendGAN(Kligvasser et al., 2022),
SinGAN(Shaham et al., 2019), ConSinGAN(Hinz et al., 2021)
2) Non-progressive Training: Train a generative model on the same scale/ noise
but with changes to the model’s architecture.
_Methods:_ SinFusion(Nikankin et al., 2023), One-Shot GAN(Sushko et al., 2021)

  1. cGM-3 is similar to uGM-2 as cross-domain adaptation is required in both tasks, but cGM-3 focuses on conditional generation while uGM-2 focuses on unconditional generation. Furthermore, cGM-3 is similar to cGM-2, but seen classes and unseen classes are from different domains in cGM-3. For example, (Shahbazi et al., 2021) has studied the setup when a pre-trained generator on ImageNet is adapted to generate samples for several classes from Places365 (Zhou et al., 2017). Transfer learning is one of the effective approaches for cGM-3, see Sec. 4.

  2. (7)

Internal patch distribution Generative Modeling (IGM).

###### Definition 0 (IGM).

Given $K$ samples and assuming rich internal distribution for patches within
these samples, learn to generate diverse and high-quality samples with the
same internal patch distribution.

IGM aims to capture the internal distribution of patches within the samples.
With the model capturing the samples’ patch statistics, it is then possible to
generate high quality, diverse samples with the same content as the given
training samples. In most works, $K=1$, and IGM focuses on images (Shaham et
al., 2019), learning to generate new images with significant variability while
maintaining both the global structure and fine textures of the training image.

  3. (8)

Subject-driven Generative Modeling (SGM).

###### Definition 0 (SGM).

Given $K$ samples of a particular subject and a text prompt, learn to generate
diverse and high-quality samples containing the same subject.

SGM is a recent GM-DC task introduced in (Ruiz et al., 2023). Given a few
images (3-5 in most cases) of a subject and leveraging a large text-to-image
generative model, (Ruiz et al., 2023) learns to generate diverse images of the
subject in different contexts with the guidance of text prompts. The goals
are: i) to achieve natural interactions between the subject and diverse new
contexts, and ii) to maintain high fidelity to the key visual features of the
subject. In (Ruiz et al., 2023), a natural language-guided transfer learning
approach and a new prior preservation loss have been proposed to achieve SGM.

###  3.2. Generative Modeling under Data Constraint: Challenges

####  3.2.1. Challenges for Training Generative Models under Data Constraint

Data constraints typically introduce additional challenges and amplify
existing ones when training generative models. Here, we delve into the
challenges of training GM-DC. These limitations include pervasive issues of
overfitting and frequency bias which are commonly observed across various
approaches. Additionally, knowledge transfer between domains brings forth
specific problems including the proximity between source and target domains
and the transfer of incompatible source knowledge. As shown in Fig. 3, around
$39\%$ of works directly rely on knowledge transfer as a mainstream method to
tackle GM-DC, and more than $20\%$ of works propose methods based on other
approaches that are compatible with transfer learning.

Overfitting to Training Data. In machine learning, overfitting is a common
issue when powerful models start to memorize the training data instead of
learning the generalizable semantics (Santos and Papa, 2022). In generative
modeling, the overfitting problem exacerbates under data constraints due to
the high capacity of current generative models (Noguchi and Harada, 2019; Liu
et al., 2021; Karras et al., 2020a). When limited training data is available,
generative models may simply remember the training data (Li et al., 2020; Ojha
et al., 2021) and learn to generate the exact training samples (Zhao et al.,
2022a) instead of capturing the data distribution. Furthermore, under data
constraints, generative modeling is more prone to mode collapse (Tran et al.,
2021), i.e., the generators learn only a limited set of modes and fail to
capture other modes of the data distribution, resulting in limited diversity
in generated samples (Yu et al., 2022; Nguyen et al., 2023a).

Frequency Biases. Generative models are notorious for their spectral bias
(Rahaman et al., 2019; Khayatkhoei and Elgammal, 2022), i.e. tendency to
prioritize fitting low-frequency components while disregarding high-frequency
components within a data distribution (Durall et al., 2020; Tancik et al.,
2020; Chandrasegaran et al., 2021). The exclusion of these high-frequency
components which encode intricate image details (Gonzales and Wintz, 1987) can
significantly impact the quality of generated samples, i.e. , accurate
modeling of high-frequency details is critical in various fields including
medical imaging (X-rays, CT-scans, MRIs), satellite/ aerial imaging,
astrophotography, and art restoration. This issue becomes more severe under
limited data (Yang et al., 2022c, b) as even advanced network structures
tailored for such scenarios (Liu et al., 2021) struggle to maintain the
desired level of details in generated samples.

Figure 5.  Source-target domain proximity visualization indicates that
distant/ remote target domains have not been explored in GM-DC setups and are
very challenging. We use FFHQ (Karras et al., 2019) as the source domain. We
show source-target domain proximity qualitatively by visualizing Inception-v3
(Left) (Szegedy et al., 2016), LPIPS (Middle) (Zhang et al., 2018) and
DreamSim (Right) (Fu et al., 2023) features. For feature visualization, we use
t-SNE (van der Maaten and Hinton, 2008) and show centroids ($\bigtriangleup$)
for all domains. We clearly show using feature visualizations that additional
setups – Flowers (Nilsback and Zisserman, 2008) and Church (Yu et al., 2015) –
represent target domains that are remote from the source domain (FFHQ)
compared to target domains used in the literature. This indicates that the
exploration of distant/ remote target domains under GM-DC setups has not been
pursued and poses notable challenges (Fig. 6). Best viewed in color.

Modeling Distant/ Remote Target Domains under GM-DC Setups. Substantial number
of GM-DC tasks rely on the transfer learning principle (uGM-2, uGM-3, cGM-2,
cGM-3, SGM), which aims to enhance the generative capabilities for a target
domain by leveraging the knowledge of a generator pre-trained on a large and
diverse source domain (See Fig. 1). A significant amount of research has been
focused on target domains that are semantically/ perpetually similar to the
source domain, e.g. , learn to generate Baby faces using a pre-trained
generator trained on Human faces. In particular, when dealing with GM-DC
setups involving significant domain shifts between the source and target
domains (Human Faces$\rightarrow$Animal Faces), many proposed methods fail to
outperform a basic fine-tuning approach (Zhao et al., 2022a). This is due to
these methods prioritizing knowledge preservation from the source domain/
task, overlooking the adaptation step to the target domain (Zhao et al.,
2022a). Recently, adaptation-aware algorithms have characterized
source$\rightarrow$target domain proximity (Zhao et al., 2022a) and addressed
GM-DC setups with pronounced domain shifts between the source and target
domains (Human Faces$\rightarrow$Animal Faces) (Zhao et al., 2022a, 2023a). To
understand the concept of distant/ remote target domains, we additionally
introduce two remote target domains that further exhibit a considerable degree
of domain shifts: i) Human Faces (FFHQ) (Karras et al., 2019)$\rightarrow$
Flowers (Nilsback and Zisserman, 2008), ii) Human Faces (FFHQ) (Karras et al.,
2019)$\rightarrow$Church (Yu et al., 2015). Domain proximity visualization is
shown in Fig. 5. In particular, we conducted a GM-DC experiment (uGM-2) to
adapt a pre-trained Human face (FFHQ) generator to Flowers under 10-shot setup
using AdAM (Zhao et al., 2022a), obtaining a FID value of 124.46. Adaptation
results are shown in Fig. 6. As one can observe, multiple instances of low
quality synthesis are observed in AdAM (Zhao et al., 2022a). In summary, we
remark that modeling distant/ remote target domains remains an important and
challenging area for GM-DC.

Identifying and Removing Incompatible Knowledge Transfer. Another challenge
with leveraging source domain’s knowledge for GM-DC tasks is incompatible
knowledge transfer, which is discovered recently (Zhao et al., 2023a). In
particular, many methods may transfer knowledge that is incompatible with the
target domain, e.g. hat from source domain FFHQ to target domain flowers,
significantly degrading the realisticness of the generated samples. In Fig. 6,
we show multiple examples of incompatible knowledge transfer using AdAM for
10-shot flower adaptation. Although some recent effort has been invested in
identifying and proactively truncating incompatible knowledge transfer (Zhao
et al., 2023a) in Human Faces $\rightarrow$ Animal Faces adaptation setups, it
is worth noting that identifying and removing incompatible knowledge remains a
critical and demanding area in GM-DC.

Figure 6.  Knowledge transfer under distant/ remote target domain (Human face
$\rightarrow$ Flowers) suffers from low synthesis quality and incompatible
knowledge transfer.  We show 10-shot adaptation results for FFHQ $\rightarrow$
Flowers using AdAM (Zhao et al., 2022a). The FID for the 10-shot adaptation
using AdAM is 124.46. We highlight multiple instances of low synthesis quality
and incompatible knowledge transfer (i.e. glasses, hat from FFHQ to flowers),
showing that GM-DC modeling of remote target domains poses significant
challenges. Best viewed in color.

####  3.2.2. Challenges on Selecting Samples for GM-DC

Although considerable research effort has been invested in developing
algorithms for GM-DC, the task of sample selection for GM-DC remains a
challenging and relatively unexplored area. It is essential that the samples
selected for GM-DC should represent the target domain. In particular, we
observe significant variation in performance with different selection of
target samples as the training datasets in GM-DC. We perform a 10-shot data-
centric GM-DC experiment using AdAM (Zhao et al., 2022a) to emphasize the
importance of sample selection in GM-DC. Following (Zhao et al., 2022a,
2023a), we use AFHQ-Cat dataset (Choi et al., 2020) and select 3 random sets
of 10-shot cat data for GM-DC. Data and 10-shot adaptation FID results are
shown in Fig. 7. We obtain FID values of 90.0, 71.6 and 49.9 for Sets 1, 2 and
3 respectively (iteration=2500). This study provides evidence that sample
selection plays a vital role in determining the capabilities of GM-DC.
Specifically, due to cost/ privacy concerns, the role of sample selection is
critical in applications including biomedical imaging, satellite/ aerial
imaging and remote sensing. In summary, sample selection for GM-DC holds
significant importance and remains an area with limited investigation thus
far.

Figure 7.  Sample selection for GM-DC remains challenging and relatively
unexplored. We use AdAM (Zhao et al., 2022a) to adapt a pre-trained StyleGAN2
on FFHQ to AFHQ-Cat dataset (Choi et al., 2020) using 3 random sets of 10-shot
data (Right). We report FID results during training (Left) for these sets of
data. Following (Zhao et al., 2022a) FID is measured between 5000 generated
samples and the entire AFHQ-Cat dataset consisting of 5115 samples. We use
clean-FID library (Parmar et al., 2022), obtaining FID values of 90.0, 71.6
and 49.9 for Sets 1, 2, and 3 respectively at iteration=2500. As indicated by
FID trends, the generative capabilities of GM-DCs are drastically influenced
by sample selection. Best viewed in color.

####  3.2.3. Challenges in Evaluating Generative Models under Data Constraint

The assessment of generative modeling capabilities presents lots of
challenges, encompassing both objective and subjective evaluation
(Kynkäänniemi et al., 2023). These issues are aggravated under low-data
regimes resulting in the evaluation of GM-DC to be challenging and an active
topic of research. In contemporary GM-DC literature, sample quality and
diversity are used as the main attributes for evaluating generation
capability. A summary of prominent metrics for GM-DC is included in Tab. 4.

Existing GM-DC evaluation metrics present multiple challenges: i) Statistical
measures including FID, KID, IS, FIDCLIP lose their significance when dealing
with setups where there is an extreme scarcity (Few-shots) or complete absence
(Zero-shot) of target domain data. For example, when the reference
distribution contains only 10 real images, the mean and trace components of
FID are not statistically significant. ii) Although human judgment/ user
feedback is used for the subjective evaluation of GM-DC, the absence of a
unified framework/ protocol for such evaluation strategy results in inadequacy
when comparing the generative capabilities of different GM-DC models. iii) The
over-reliance on objective GM-DC measures on deep features extracted from pre-
trained networks remains challenging and relatively unexplored. For example,
FID, KID, and IS use features extracted from an Inception model trained on
ImageNet-1K (Deng et al., 2009); LPIPS, and Intra-LPIPS, use features
extracted from models trained on BAPPS (Zhang et al., 2018) dataset. Although
these pre-trained models effectively function as general-purpose feature
extractors, their ability to capture properties/ attributes of out-of-domain
data to objectively quantify the capabilities of GM-DC requires more
investigation, e.g. medical images. In summary, the area of evaluation
measures for GM-DC cannot be overstated, as it remains critical and
challenging.

Table 4. List of common metrics used for evaluating GM-DC works. LD: Limited-
Data, FS: Few-Shot, ZS: Zero-Shot.  ❍ /  ● denotes the absence/presence,
respectively.

Metrics FID (Heusel et al., 2017)/ FIDCLIP (Kynkäänniemi et al., 2023) KID
(Bińkowski et al., 2018) IS (Salimans et al., 2016) Intra-LPIPS (Ojha et al.,
2021) SIFID (Shaham et al., 2019) Image/ Text Similarity (Gal et al., 2022a)
User Feedback LD ● ● ● ❍ ❍ ❍ ● FS ● ● ● ● ● ● ● ZS ❍ ❍ ❍ ● ● ● ●

##  4\. Comprehensive Review

In this section, first, we will present our proposed taxonomy of approaches
for GM-DC which systematically categorizes and organizes GM-DC methods under
seven approaches (Tab. LABEL:tab:approaches) based on the principal ideas of
these methods. Then, we will discuss individual GM-DC methods organized under
our proposed taxonomy.

Our Proposed Taxonomy of Approaches for GM-DC categorizes GM-DC methods into
seven groups:

  1. (1)

Transfer Learning: In GM-DC, transfer learning (TL) aims to improve the
learning of the generator for the target domain using the knowledge of a
generator pre-trained on a source domain (with numerous and diverse samples).
For example, some methods under this category use the knowledge of a StyleGAN2
pre-trained on the large FFHQ (Karras et al., 2019) to improve the learning of
generation for face paintings by an artist, given only a few images of the
artist’s paintings (Ojha et al., 2021; Zhao et al., 2022b; Xiao et al., 2022).
Major challenges for TL-based GM-DC are to identity, select and preserve
suitable knowledge of the source generator for the target generator. Along
this line, there are six subcategories: i) Regularization-based Fine-tuning,
explores regularizers to preserve suitable source generator’s knowledge to
improve learning target generator; ii) Latent Space, explores transformation/
manipulation of the source generator latent space; iii) Modulation, freezes
and transfers weights of the source generator to the target generator and adds
trainable modulation weights on top of frozen weights to increase the
adaptation capability to the target domain; iv) Natural Language-guided, uses
natural language prompt and supervision signal from language-vision models to
adapt source generator to target domain; v) Adaptation-Aware, identifies and
preserves the source generator’s knowledge that is important to the adaptation
task; vi) Prompt Tuning, is an emerging idea that freezes the weights of the
source generator and learns to generate visual prompts (tokens) to guide
generation for the target domain.

  2. (2)

Data Augmentation: Augmentation aims to improve GM-DC by increasing coverage
of the data distribution with applying various transformations
$\\{T_{k}\\}_{k=1}^{K}$ to available data. For example, within this category,
some works augment the available limited data to train an unconditional
StyleGAN2 (Karras et al., 2020b) using the 100-shot Obama dataset or train a
conditional BigGAN (Brock et al., 2019) with only 10% of the CIFAR-100
dataset. A major challenge of these approaches is augmentation leakage, where
the generator learns the augmented distribution, e.g. , generating rotated/
noisy samples. There are three representative categories: i) Image-Level
Augmentation, applies the transformations on the image space; ii) Feature-
Level Augmentation, applies the transformations on the feature space; iii)
Transformation-Driven Design, leverages the information on each individual
transformation $T_{k}$ for an efficient learning mechanism.

  3. (3)

Network Architectures: These approaches design specific architectures for the
generators to improve their learning under data constraints. Some works in
this category design shallow/ sparse generators to prevent overfitting to
training data due to over-parameterization. The primary challenge is that when
endeavoring to design a new architecture, the process of discovering the
optimal hyperparameters can be laborious. There are three major types of
architectural designs for GM-DC: i) Feature Enhancement, introduces additional
modules to enhance/ retain the knowledge within feature maps; ii) Ensemble
Large Pre-trained Vision Models, leverages large pre-trained vision models to
aid more accurate generative modeling, iii) Dynamic Network Architecture,
evolves the architecture of the generative model during training to compensate
for data constraints.

  4. (4)

Multi-Task Objectives: These approaches modify the learning objective of the
generative model by introducing additional task(s) to extract generalizable
representations and reduce overfitting under data constraints. As an example,
some works define a pretext task based on contrastive learning (He et al.,
2020) to pull the positive samples and push away negative ones in addition to
the original generative learning task to prevent overfitting under limited
available data. The efficient integration of the new objective with the
generative learning objective could be challenging under data constraints.
These works can be categorized into several kinds of approaches: i)
Regularizer, adds an additional learning objective as a regularizer to prevent
an undesirable behavior during training generative model under data
constraint. Note that this category is different from regularizer-based fine-
tuning as the latter aims to preserve source knowledge, but the former is for
training without a source generator; ii) Contrastive Learning, adds the
learning objective related to a pretext task to enhance the learning process
of the generative model using additional supervision signal from solving this
pretext task; iii) Masking, introduces alternative learning objective by
masking a part of the image/ information to improve generative modeling by
increasing the task hardness and preventing learning the trivial solutions;
iv) Knowledge Distillation, introduces an additional learning objective that
enforces the generator to follow a strong teacher; v) Prototype Learning,
emphasizes learning prototypes for samples/ concepts within the distribution
as an additional objective; vi) Other Multi-Task Objectives, include co-
training, patch-level learning, and using diffusion to enhance generation.

  5. (5)

Exploiting Frequency Components: Deep generative models exhibit frequency bias
tending to ignore high-frequency signals as they are hard to generate (Schwarz
et al., 2021). Data constraints can exacerbate this problem (Yang et al.,
2022c). The approaches in this category aim to improve frequency awareness of
the generative models by leveraging frequency components during training. For
instance, certain approaches employ Haar Wavelet transform to extract high-
frequency components from the samples. These frequency components are then fed
into various layers using skip connections, aiming to alleviate the challenges
associated with generating high-frequency details. Despite its effectiveness,
utilizing frequency components for GM-DC has not been thoroughly investigated.
The performance can be enhanced by incorporating more advanced techniques for
extracting frequency components.

  6. (6)

Meta-Learning: These approaches create sample generation tasks with data
constraints for the seen classes, and learn the meta-knowledge —knowledge that
is shared between all tasks— across these tasks during meta-training. This
meta-knowledge is then used in improving generative modeling for the unseen
classes with data constraints. For instance, some studies, as meta-knowledge,
learn to fuse the samples from the seen categories $C_{seen}$ of the Flowers
dataset (Nilsback and Zisserman, 2008) for sample generation. This meta-
knowledge enables the model to generate new samples from unseen classes
$C_{unseen}$ within the same dataset ($C_{seen}\cap C_{unseen}=\emptyset$) by
fusing only 3 samples from each class. Note that as these works employ
episodic learning within a generative framework, the training stability can be
impacted. Approaches within this line can be classified into three categories:
i) Optimization, initializes the generative model with weights learned on the
seen classes as meta-knowledge, to enable quick adaptation to unseen classes
with limited steps of the optimization; ii) Transformation, learns cross-
category transformations from the samples of the seen classes as meta-
knowledge and applies them to available samples of the unseen classes to
generate new samples; iii) Fusion, learns to fuse the samples of the seen
classes as meta-knowledge, and applies learned meta-knowledge to sample
generation by fusing samples of the unseen classes.

  7. (7)

Modeling Internal Patch Distribution: These approaches aim to learn the
internal patch distribution within one image (in some cases a few images), and
then generate diverse samples that carry the same visual content (patch
distribution) with an arbitrary size, and aspect ratio. As an example, some
works train a Diffusion Model using a single image of the “Marina Bay Sands”,
and after training, the Diffusion Model can generate similar images, but
include additional towers topped by the similar “Sands Skypark”. However, a
major limitation of these methods lies in the fact that for every single
image, usually a separate generative model is trained from scratch, neglecting
the potential for efficient training through knowledge transfer in this
context. Approaches proposed along this line can be categorized into two major
groups: i) Progressive Training, progressively trains a generative model to
learn the patch-distribution at different scales or noise levels; ii) Non-
Progressive Training, learns a generative model at a single scale by
implementing additional sampling techniques or new model architectures.

In what follows we delve into detailed descriptions of the approaches within
each category.

###  4.1. Transfer Learning

Transfer Learning (TL) is a major approach for GM-DC. Given a source generator
$G_{s}$ (for GANs both $G_{s}$ and $D_{s}$) pre-trained on a large and diverse
source domain $\mathcal{D}_{s}$, these approaches aim to learn an adapted
generator to the target domain $G_{t}$ by initializing the weights to that of
the source generator.

####  4.1.1. Regularizer-based Fine-Tuning

TGAN (Wang et al., 2018) is the first systematic study to evaluate transfer
learning in GANs. TGAN shows that transfer learning reduces the convergence
time and improves generative modeling under limited data. The knowledge
transfer is performed by using the source GAN for initializing the weights of
the target GAN, followed by fine-tuning the weights on target data. TGAN (Wang
et al., 2018) demonstrates that: i) transferring $D$ is more important than
$G$, while transferring both $G$ and $D$ gives the best results; ii) transfer
learning performance degrades by increasing the distance between source and
target domains or decreasing the number of samples from target domain; iii) to
select a pre-trained GAN for a target domain, in addition to a smaller
distance, more dense source domains are preferable. As an example, for the
Flower (Nilsback and Zisserman, 2008) target domain, surprisingly, a GAN pre-
trained on semantically unrelated LSUN Bedrooms (Yu et al., 2015) is shown to
be among the best sources (Wang et al., 2018).

W3 (Grigoryev et al., 2022) revisits the transfer learning in GANs with a
modern structure —StyleGAN2-ADA (Karras et al., 2020b, a) instead of WGAN-GP
(Gulrajani et al., 2017) used in TGAN. Results in (Grigoryev et al., 2022)
suggest that for SOTA GANs, it is beneficial to transfer the knowledge from
sparse and diverse sources (pre-trained StyleGAN2 on ImageNet) rather than
dense and less diverse ones. One major limitation of TGAN is that under
limited data simply fine-tuning the whole generator destroys a considerable
portion of the general knowledge obtained on the source domain, and results in
overfitting. Almost all of the following works aim to address this by
different approaches to preserve the knowledge of the source generator.

BSA (Noguchi and Harada, 2019) limits scale and shift parameter updates during
fine-tuning for batch normalization (BN) layers. FreezeD (Mo et al., 2020),
hypothesizes that as $D$ performs the discriminative task during training a
GAN, based on common knowledge in discriminative learning (Yosinski et al.,
2014), its early layers encode general knowledge which is shared between
source and target domains. Therefore, this general knowledge is preserved
during adaptation by freezing early layers of $D$. cGANTransfer (Shahbazi et
al., 2021) assumes that the pre-trained $G$ is conditioned on class labels
using BN parameters, i.e. each class has its own BN parameters (Brock et al.,
2019). Then, explicit knowledge propagation from seen classes to unseen
classes is enforced by defining the BN parameters of the unseen classes to be
the weighted average of the BN parameters of seen classes. SVD (Robb et al.,
2020) applies singular value decomposition (Van Loan, 1976), and only updates
the singular values that are related to changing entanglement between
different attributes within data to constrain the parameter update.

EWC (Li et al., 2020) aims to preserve the diversity of the source GAN during
adapting to a target domain with only a few samples, e.g. , 10-shot. The
importance of each parameter in source GAN is measured by Fisher Information
(FI), and the change on each parameter during adaption is penalized based on
its importance, e.g. , change over important parameters is penalized more. CDC
(Ojha et al., 2021) aims to keep the diversity of the generated samples using
a cross-domain correspondence loss. Specifically, first, a batch of $N+1$
latent codes are sampled for image generation: $\\{G(z_{0}),...,G(z_{N})\\}$.
Then, using $G(z_{0})$ as a reference, the similarity of generated samples to
reference is measured for the generator before and after adaptation, resulting
in two $N-way$ probability vectors. The diversity is preserved by adding the
KL divergence between these two probability vectors to the standard loss as a
regularizer. MaskD (Zhu et al., 2022b) applies random masks to extracted
features of $D$, on top of CDC (Ojha et al., 2021), to prevent overfitting.
DDPM-PA (Zhu et al., 2022c) uses a pairwise adaptation method similar to CDC
for adapting diffusion models to the new domain. RSSA (Xiao et al., 2022)
extends the cross-domain consistency idea of the CDC (Ojha et al., 2021) to a
more constrained form by preserving the structural similarity of the samples
before and after adaptation. ProSC (Moon et al., 2023) extends RRSA by
performing a progressive adaptation to the target domain in $N$ iterations
instead of a single adaptation to reduce the gap between pair of domains. CSR
(Gou et al., 2023) uses a similar idea to CDC but applies semantic loss
directly to the spatial space, i.e. , generated images with $G_{s}$ and
$G_{t}$.

$C^{3}$ (Lee et al., 2021), DCL (Zhao et al., 2022b), and CtlGAN (Wang et al.,
2022b) aim to preserve the diversity by applying contrastive learning.
Assuming $G_{s}(z_{i})$ as an anchor point, the generated sample for the same
latent code with the adapted generator ($G_{t}(z_{i})$) is considered a
positive pair, and the generated samples with the adapted generator for other
latent code values ($G_{t}(z_{j})$, $i\neq j$) are considered as negative
pairs. Additionally, DCL applies similar contrastive learning to the $D$.

JoJoGAN (Chong and Forsyth, 2022) addresses one-shot image generation using
the style space of StyleGAN2. First, GAN inversion is used to find the
corresponding style code of the reference image. Then, style mixing is used to
generate a set of style codes, and generated images with these styles are used
for GAN adaptation. GenOS (Zhang et al., 2022a) includes entity transfer with
some related entity mask using an auxiliary network. D3-TGAN (Wu et al., 2023)
first inverts each target sample into the latent code space of the source GAN.
Then, the maximum mean discrepancy between the features of the source $G$ for
inverted code and features of the adapted GAN for a random latent code is used
as a regularizer.

FairTL (Teo et al., 2023) adopts transfer learning in GANs to train a fair
generative model w.r.t. a sensitive attribute (SA) using a limited fair
dataset. To model complex distributions like ImageNet, IC-GAN (Casanova et
al., 2021) learns data distribution as a mixture of conditional distributions.
This enables IC-GAN to generate images from unseen distributions, by just
changing the conditioning instances on the target samples. KDFSIG (Hou et al.,
2022a) exploits the knowledge distillation idea by treating the source model
as a teacher and the target model as a student. F3 (Kato et al., 2023)
proposes a faster method to generate face images with features of a specific
group. First, a GAN Inversion of target images is applied and then PCA is
leveraged as a feature extraction strategy to render features of target group.
DWSC (Hou et al., 2022b) proposes the dynamic weighted semantic correspondence
between the source and target generator during adaption to preserve the
diversity.

####  4.1.2. Latent Space

MineGAN (Wang et al., 2020) trains a miner network $M$ during adaptation, to
map the latent space $z$ of the source GAN to another space $u=M(z)$ more
appropriate for the target domain. MineGAN++(Wang et al., 2021) extends
MineGAN by only updating important parameters. GenDA (Yang et al., 2021b)
proposes a lightweight attribute adaptor in the form of scaling and shifting
latent codes to adapt the latent space of the source GAN to the target GAN.
LCL (Mondal et al., 2023) freezes $G$ and learns a network to map the latent
codes from the $\mathcal{Z}$ space to the extended intermediate space
$\mathcal{W}^{+}$ of a pre-trained StyleGAN2 during adapting GAN. WeditGAN
(Duan et al., 2023) proposes to learn a constant offset parameter ($\Delta w$)
for the target domain in the intermediate latent space of StyleGAN2 to
relocate source latent codes to the target domain. After fine-tuning the
generator to a target domain, SiSTA (Thopalli et al., 2023) perturbs latent
representations of the fine-tuned generator that falls below a threshold,
either by replacing them with zero or reverting them back to the pre-trained
generator’s weights. MultiDiffusion (Bar-Tal et al., 2023) freezes the whole
parameters of the source diffusion model and optimizes the latent code as a
post-processing method to generate the desired output based on a conditioned
input.

####  4.1.3. Modulation

In signal processing literature, modulation varies some key attributes of a
signal to add the desired information to it (Oppenheim et al., 1997).
Similarly, in deep neural networks, modulation is used to add some desired
information to a base network by adding modulation parameters to the
parameters/ features of the base network. AdaFMGAN (Zhao et al., 2020a) shows
that layers closer to the sample (earlier layers in $D$, and later layers in
$G$) encoder general knowledge. This general knowledge is conceptually shared
between source and target domains and aimed to be preserved by Adaptive Filter
Modulation which trains a scale and shift parameter for each $k\times k$
kernel. GAN-Memory (Cong et al., 2020), and CAM-GAN (Varshney et al., 2021)
use similar modulation ideas to modulate a pre-trained GAN for generative
continual learning. AdAM (Zhao et al., 2022a) uses kernel modulation
(Abdollahzadeh et al., 2021) for few-shot generative modeling by aiming to
preserve the important wights of a pre-trained GAN during a few-shot
adaptation to a target domain. HyperDomainNet (Alanov et al., 2022) adds an
additional modulation to StyleGAN2 (Karras et al., 2020b) for adapting to a
new domain, while optimizing only $6K$ parameters.

####  4.1.4. Adaptation-Aware

Adaptation-aware transfer learning approaches propose that different parts of
the knowledge encoded on a pre-trained generative model could be important
based on the target domain. AdAM (Zhao et al., 2022a) proposes a probing step
before the main adaptation, where the importance of each kernel for adapting a
source GAN to the target domain using a few samples is measured using FI.
Then, during the main adaptation, the important kernels are preserved using
modulation, and the other kernels are simply fine-tuned. RICK (Zhao et al.,
2023a) shows that incompatible knowledge from a source domain to a target
domain is related to the kernels with the least importance to this adaptation,
and this knowledge can not be removed by simple fine-tuning. Therefore, RICK
proposes a dynamic probing schedule during adaptation where it gradually
prunes the kernels with the least importance.

####  4.1.5. Natural Language-Guided

Vision-Language models like CLIP (Radford et al., 2021) are usually trained on
large-scale image-text pairs and learn to encapsulate the generic information
by combining image and text modalities. This generic information is shown to
be helpful in various downstream tasks including zero-shot and few-shot image
generation. StyleGAN-NADA (NADA) (Gal et al., 2022b) is the pioneering work on
utilizing CLIP for zero-shot image generation. NADA (Gal et al., 2022b) uses a
text prompt $T_{t}$ –which describes the target domain– as input and uses
feedback to adapt a pre-trained StyleGAN2 to the target domain. Specifically,
assuming a text prompt $T_{s}$ that describes the source domain (e.g. ,
”Photo” for a StyleGAN2 pre-trained on the FFHQ), and a given $T_{t}$ (e.g. ,
”Fernando Botero Painting”), CLIP’s text encoder $E_{T}$ is used to find the
update direction in the embedding space: $\Delta T=E_{T}(T_{t})-E_{T}(T_{s})$.
Similarly, the direction of the update/ change for the images can be computed
using generated images with source and target generators: $\Delta
I=E_{I}(G_{t}(z))-E_{I}(G_{s}(z))$, where $E_{I}$ denotes CLIP’s image
encoder. Since the image and corresponding texts are aligned in the CLIP
space, NADA (Gal et al., 2022b) proposes to update the generator’s parameters
in a way to match $\Delta I$ and $\Delta T$ leading to the directional loss
$\mathcal{L}_{directional}$:

(7) $\mathcal{L}_{directional}=1-\frac{\Delta I(G_{s}(z),G_{t}(z)).\Delta
T(T_{s},T_{t})}{|\Delta I(G_{s}(z),G_{t}(z))||\Delta T(T_{s},T_{t})|}$

This idea can be easily extended to one-shot image generation, by replacing
$\Delta T$ with the direction obtained by target image $I_{t}$ and a batch of
generated images by the source generator: $\Delta
I^{\prime}=E_{I}(I_{t})-\mathbb{E}_{i}\\{E_{I}(G_{s}(z_{i}))\\}$, where
$\mathbb{E}_{i}\\{E_{I}(G_{s}(z_{i}))\\}$ denotes the mean of the CLIP
embedding for a batch of generated images. MTG (Zhu et al., 2022a) extends the
idea for one-shot image generation by replacing the mean embedding with the
projection of the target image on the source generator denoted as $I^{*}_{s}$.
Specifically, MTG uses GAN inversion to get the corresponding $z^{ref}$ for
$I_{t}$, and uses it to generate the projected image:
$I^{*}_{s}=G_{s}(z^{ref})$. HyperDomainNet improves the performance of the
NADA and MTG by freezing the weights of the source generator and training
modulation weights for the synthesis part inside the generator. DiFa (Zhang et
al., 2022b) adds an attentive style loss to directional loss of NADA (Gal et
al., 2022b) as a local-level adaptation which aligns the intermediate tokens
of the generated image with source and pre-trained GAN. OneCLIP (Kwon and Ye,
2023) exploits the CLIP embedding for three major modules in one-shot
learning: i) inverting sample into latent space, ii) preserving the diversity
of the GAN during adaptation, and iii) a patch-wise contrastive learning
approach for preserving local consistency.

IPL (Guo et al., 2023) mentions that NADA (Gal et al., 2022b) uses a fixed
update direction for a target domain due to manual text prompts being used for
describing source and target domains. To address this, (Guo et al., 2023)
learns a specific prompt for each generated image, e.g. , ”Elder glass man
photo” $\rightarrow$ ”Elder glass man Fernando Botero Painting”. A mapper
function $F$ is used to learn prompts for the generated images $G_{s}(z_{i})$:

(8) $F(z_{i})=[V]_{1}^{i}[V]_{2}^{i}...[V]_{m}^{i}$

where $[V]_{j}^{i}$ represents the $j^{th}$ prompt vector. Then, the domain
embedding $[Y_{s}]$ is added to this prompt to represent both the prompt and
the domain: $M_{s}^{i}=F(z_{i})[Y_{s}]$. The mapper function is trained by
contrastive loss in the CLIP space. After training $F$, the same adaptation
process as NADA (Gal et al., 2022b) can be used but using
$T_{s}(z_{i})=[V]_{1}^{i}[V]_{2}^{i}...[V]_{m}^{i}[Y_{s}]$.

DreamBooth (Ruiz et al., 2023) addresses subject-driven sample generation by
fine-tuning a text-to-image diffusion model e.g. Imagen (Saharia et al.,
2022), DALL-E2 (Ramesh et al., 2022). Input images are paired with a text
prompt that contains a unique identifier and the subject class (e.g., “A [V]
dog”), and the pair is used to fine-tune the model. They further propose a
class-specific prior preservation regularizer to encourage diversity and to
mitigate language drift, i.e., the model progressively loses syntactic and
semantic knowledge during fine-tuning. Textual-Inversion (Gal et al., 2022a)
optimizes a word vector for the new subject given a few images and uses that
word vector for the subject-driven generation. MCC (Kumari et al., 2023)
extends DreamBooth by the capability of adding multiple subjects and improves
convergence and performance by restricting fine-tuning to a subset of cross-
attention layer parameters in DM. BLIP-Diffusion (Li et al., 2023b) uses
BLIP-2 (Li et al., 2023c) multimodal encoder to extract a more text-aligned
representation for each subject, and then a subject representation learning
step is performed to enable DM to leverage such representation for fast and
high-fidelity subject-driven sample generation. SINE (Zhang et al., 2023a)
uses a similar fine-tuning of text-to-image diffusion model with a unique
identifier, but at the patch level. In addition, the mixing of the latent code
is also used to edit the subject and put it in a new context.
SpecialistDiffusion (Lu et al., 2023) addresses adapting the text-to-image
models to an unseen style given a few samples from that style using augmenting
both text and image, and sparse diffusion for computation efficiency.

####  4.1.6. Prompt Tuning

VQ-VAEs (Sec. 2.1) can be broadly categorized into two types from the
perspective of predicting the latent prior of visual tokens. AutoRegressive
(AR) approaches like DALL$\cdot$E (Ramesh et al., 2021) and VQ-GAN (Esser et
al., 2021), learn an AR predictor that follows a raster scan order and
predicts the visual tokens from left to right, line-by-line. Non-
AutoRegressive (NAR) approaches like DALL$\cdot$E2 (Ramesh et al., 2022),
MaskGIT (Chang et al., 2022), Latent Diffusion (Rombach et al., 2022), or
Imagen (Saharia et al., 2022) usually resort to masking techniques (Devlin et
al., 2019) to predict the visual tokens in a series of refinement or denoising
steps. VPT (Sohn et al., 2023) is the first work that adopts the prompt tuning
idea for image generation with generative knowledge transfer. It uses a VQ-VAE
framework where a MaskGIT(Chang et al., 2022)/ VQ-GAN(Esser et al., 2021) on
the ImageNet dataset (as an example of NAR/ AR approach) is used as a pre-
trained network. Then, during adaptation, all the parameters of the VQ
Encoder, VQ decoder, and transformer are frozen, and a generator is learned to
minimize the adaptation loss by generating and appending a set of visual
tokens to the predicted prior. These visual tokens guide the generation
process for the target domain by helping the transformer to predict proper
tokens to the VQ decoder.

###  4.2. Data Augmentation

Data augmentation increases the quantity and diversity of the training data
which is shown to be beneficial for GM-DC. However, if it is not deployed
correctly, augmentations can leak into the generator resulting in generating
samples with similar augmentations, e.g. noisy or rotated images, which is
undesirable.

####  4.2.1. Image-Level Augmentation

CR-GAN (Zhang et al., 2020) and bCR (Zhao et al., 2021) apply various
transformations on images and enforce the output of the generator to be the
same for original and transformed images. Even though not developed for GM-DC,
experimental results in (Karras et al., 2020a) show that CR-GAN and bCR are
beneficial for limited data scenarios. ADA proposes applying the
transformations to real and fake images but with a probability $p<1$. The
central design in ADA (Karras et al., 2020a) is that the strength of the
augmentation ($p$) is being adapted based on the training dynamics.
Specifically, the portion of the real images that get positive output from the
discriminator, i.e. , $r=\mathbb{E}[Sign(D)]$, is used as an indicator of the
discriminator overfitting ($r=0$ no overfitting, and $r=1$ complete
overfitting). Then, during training, $p$ is adjusted to keep $r$ low.
DiffAugment (Zhao et al., 2020b), and IAG (Zhao et al., 2020c) use a similar
idea to ADA, but without the adaptive component ($p=1$). APA (Jiang et al.,
2021) uses the same adaptive augmentation mechanism in ADA, but instead of
using transformations like rotation, it randomly labels generated images as
pseudo-real ones to prevent an overconfident discriminator.

DiffusionGAN (Wang et al., 2023b) applies the gradual diffusion process on
real and generated images during training GAN. Training starts with real and
generated images, and each diffusion step is applied after a certain number of
training epochs making the bi-classification task harder for the discriminator
to prevent its overfitting. PatchDiffusion (Wang et al., 2023a) augments the
data during training diffusion models by sampling patches with random
locations and random sizes alongside the full image and conditioning the
denoising score function (Karras et al., 2022) on the patch size and the
location information.

####  4.2.2. Feature-Level Augmentation

AdvAug (Chen et al., 2021) computes the adversarial perturbation $\delta$ for
the feature maps of the discriminator and generator using the projected
gradient descent (Madry et al., 2018). Denoting the discriminator as
$D=D_{2}\circ D_{1}$, the adversarial augmentation is applied on the
intermediate feature maps ($D_{1}$) of both real and generated images,
resulting in $D_{1}(x)+{\delta}$, and $D_{1}(G(z))+{\delta}$. The adversarial
loss is then added to the loss function of $D$ during GAN training to maximize
the score of the perturbed real image and minimize the score of the perturbed
generated image:

(9)
$\mathcal{L}_{D}^{adv}\coloneqq\max_{\|\delta\|_{\infty}<\epsilon}\mathbb{E}_{x\sim
p_{data}}[f_{D}(-D_{2}(D_{1}(x)+\delta))]+\max_{\|\hat{\delta}\|_{\infty}<\epsilon}\mathbb{E}_{z\sim
p_{z}}[f_{D}(D_{2}(D_{1}(G(z))+\hat{\delta}))]$

As AdvAug is performed on the feature level, it is shown to be complementary
to image-level augmentations like ADA (Karras et al., 2020a) and DiffAug (Zhao
et al., 2020b). AFI (Dai et al., 2022) observes a flattening effect in
discriminators with multiple output neurons, and takes advantage of this
observation by proposing feature interpolation as implicit data augmentation.

####  4.2.3. Transformation-Driven Design

DAG (Tran et al., 2021) uses a separate discriminator $D_{k}$ for
discriminating real and fake images that are augmented by transformation
$T_{k}$. A weight-sharing mechanism between all discriminators is used to
prevent overfitting. Additionally, DAG provides a theoretical ground for
training convergence under augmentation. As mentioned in (Goodfellow et al.,
2014), for an optimal discriminator $D^{*}$, optimizing $G$ is equivalent to
minimizing the Jensen-Shannon (JS) divergence between the real data
distribution $P_{data}$ and generated data distribution $P_{model}$, i.e. ,
$JS(P_{data},P_{model})$. Denoting $P_{data}^{T}$, and $P_{model}^{T}$ as the
distribution of the real and generated data under augmentation $T$, (Tran et
al., 2021) shows that JS divergence between two distributions is invariant
under differentiable and invertible transformations:

(10) $JS(P_{data},P_{g})=JS(P_{data}^{T},P_{g}^{T})$

This means that as long as the augmentation is differentiable and invertible,
training convergence is guaranteed. SSGAN-LA (Hou et al., 2021) extends DAG by
merging all discriminators to a single discriminator and augmenting the label
space of the discriminator, i.e. , asking $D$ to detect the type of
augmentation in addition to conventional real/ fake detection.

###  4.3. Network Architectures

####  4.3.1. Feature Enhancement

FastGAN (Liu et al., 2021) proposes a light-weight GAN structure —shallower
$G$ and $D$ compared to SOTA GANs like StyleGAN2— to decrease the risk of
overfitting. Inspired by skip connections (He et al., 2016), and squeeze-and-
excitation module (Hu et al., 2018), FastGAN fuses features with different
resolutions in $G$ through proposed skip-layer excitation modules. An
additional reconstruction task is defined for $D$. MoCA (Li et al., 2022a)
learns some prototypes for each semantic concept within a domain, e.g. ,
railroad, or sky in a photo of a train. Then, by attending to these prototypes
during image generation, some residual feature maps are produced to improve
image generation. DFSGAN (Yang et al., 2023a) proposes to preserve the content
and layout information in intermediate layers of $G$ by extracting channel-
wise and pixel-wise information and using them to scale corresponding feature
maps. SCHA-VAE (Giannone and Winther, 2022) extend latent variable models for
sets to a fully hierarchical approach and propose Set-Context-Hierarchical-
Aggregation VAE for few-shot generation.

####  4.3.2. Ensemble Pre-trained Vision Models

ProjectedGAN (Sauer et al., 2021) proposes to project real and generated
images into the feature space of a pre-trained vision model to enhance $D$’s
performance in discriminating real and fake images by adding two modules.
First, the output from multiple layers is used with separate discriminators.
Then a random projection is used to dilute the features and encourage the
discriminator to focus on a subset of the features. Vision-aided GAN (Kumari
et al., 2022) uses an ensemble of the original discriminator $D$ and
additional discriminators $\\{\hat{D}_{n}\\}_{n=1}^{N}$ to perform the
classification task. The additional discriminators
$\\{\hat{D}_{n}\\}_{n=1}^{N}$ have a set of pre-trained feature extractors
$\mathcal{F}=\\{F_{n}\\}_{n=1}^{N}$ (extracted form pre-trained vision models)
with a small trainable head $C_{n}$ added on top: $\hat{D}_{n}=F_{n}\circ
C_{n}$.

####  4.3.3. Dynamic Network Architecture

CbC (Shahbazi et al., 2022) shows that under data constraints, where an
unconditional GAN can generate satisfactory performance, training the
conditional GANs (cGANs) result in mode collapse. To mitigate this issue, CbC
(Shahbazi et al., 2022) starts training from an unconditional GAN and slowly
transitions to a cGAN using a transition function $0\leq\lambda_{t}\leq 1$.
Considering the conditioning variable as $c$, this transition is implemented
in $G$ as $G(z,c,\lambda_{t})=G(S(z)+\lambda_{t}.E(c))$, with $S$ and $E$ as
neural networks that transform the latent code and the conditioning variable.
DynamicD (Yang et al., 2022a) dynamically reduces the capacity of $D$ by
randomly sampling a subset of channels of $D$ during each training iteration
to prevent overfitting. Inspired by the lottery ticket hypothesis (Frankle and
Carbin, 2019), AdvAug (Chen et al., 2021) and Re-GAN (Saxena et al., 2023)
have shown that a much sparse subnetwork of the original generator can be
useful for GM-DC. AutoInfoGAN (Shi et al., 2023) applies a reinforcement
learning-based neural architecture search to find the best network
architecture for the generator.

###  4.4.  Multi-Task Objectives

####  4.4.1. Regularizer

LeCam (Tseng et al., 2021) uses two moving average values to track $D$’s
prediction for real and generated images, denoted by $\alpha_{R}$ and
$\alpha_{F}$, respectively. Then the distance between the $D$’s prediction for
real (fake) images and $\alpha_{F}$ ($\alpha_{R}$) is decreased by adding a
regularizer to prevent overfitting. Analysis in (Tseng et al., 2021) shows
that this regularizer enforces WGAN (Arjovsky et al., 2017)/ BigGAN(Brock et
al., 2019) to minimize the LeCam-divergence which is beneficial for GM-DC.
Reg-LA (Hou, 2023) uses a similar idea to regularize the label-augmented GANs
discussed in Sec. 4.2. DIG (Fang et al., 2022) shows that the discriminator
gradient gap between real and generated images increases when training GANs
with limited data, and adds this gap as a regularizer to prevent this
behavior. MDL (Kong et al., 2022) addresses the pre-training free few-shot
image generation by adding a regularizer that aims to keep the similarities
between the latent codes in $\mathcal{Z}$ space and corresponding generated
images in image space.

####  4.4.2. Contrastive Learning

InsGen (Yang et al., 2021a) uses contrastive learning to improve learning $D$
by introducing a pretext task. The pretext task is defined as instance
discrimination, meaning that each sample should be mapped to a separate class.
This is done by constructing the query and key from the same sample as
positive pair, and all remaining images as negative pair. FakeCLR (Li et al.,
2022b) analyze three different contrastive learning strategies, namely
instance-real, instance-fake, and instance-perturbation. It is shown that
instance-perturbation contributes the most improvement in quality and can
effectively alleviate the issue of latent space discontinuity. Note that
contrastive learning is also combined with other approaches as discussed
before (C3 (Lee et al., 2021), DCL (Zhao et al., 2022b), CtlGAN (Wang et al.,
2022b), CML-GAN (Phaphuangwittayakul et al., 2022), and IAG (Zhao et al.,
2020c)).

####  4.4.3. Masking

MaskedGAN (Huang et al., 2022) utilizes a masking idea for training GANs under
limited data by masking both spatial and spectral information. For spatial
masking, they use a patch-based mask to enable random masking of all spatial
parts. For spectral masking, they mask each frequency channel (extracted by
Fourier transform) based on the amount of information, i.e. , channels with
more information are more probable to be masked. MaskD (Zhu et al., 2022b)
randomly masking feature maps extracted by $D$ for a few-shot setup. DMD
(Zhang et al., 2023b) detects that the discriminator slows down learning and
applies random masking to its features adaptively to balance its learning pace
with the generator.

####  4.4.4. Knowledge Distillation

KD-DLGAN (Cui et al., 2023) proposes a knowledge distillation (KD) (Hinton et
al., 2015; Chandrasegaran et al., 2022b) approach by leveraging CLIP (Radford
et al., 2021) as the teacher model to distill text-image knowledge to the
discriminator. They propose two designs: aggregated generative knowledge
designs a harder learning task, and correlated generative knowledge
distillation improves the generation diversity by distilling and preserving
the diverse image-text correlation from CLIP. As discussed before, KDFSIG (Hou
et al., 2022a) also uses KD in the context of transfer learning for few-shot
image generation.

####  4.4.5. Prototype Learning

Inspired by the recent success of learning prototypes in few-shot
classification, ProtoGAN (Yang et al., 2023b), aims to improve the fidelity
and diversity of the FastGAN under limited data(Snell et al., 2017). ProtoGAN
has two main modules: prototype alignment for increasing the fidelity of the
generated images, and diversity loss to improve the generation diversity. MoCA
(Li et al., 2022a) also learns prototypes but for different semantic concepts
through an attend and replace mechanism on the extracted feature maps of $G$.

####  4.4.6. Other Multi-Task Objectives

Gen-Co (Cui et al., 2022) uses multiple discriminators to extract diverse and
complementary information from samples. This co-training has two major
modules: weight-discrepancy co-training which trains separate $D$s with
different weights, and data-discrepancy co-training which in addition to
training separate $D$s also uses different information as inputs, i.e. ,
spatial or frequency information. AdaptiveIMLE (Aghabozorgi et al., 2023)
proposes an adaptive version of implicit maximum likelihood estimation (Li and
Malik, 2018) to improve the mode coverage by assigning different boundary
radii for each sample. PathcDiffusion (Wang et al., 2023a) and AnyResGAN (Chai
et al., 2022) show the effectiveness of Patch-Level learning of the
generators. Diffusion-GAN (Wang et al., 2023b) leverages the diffusion process
to improve training GANs by gradually increasing the task hardness for $D$.
D2C (Sinha et al., 2021) uses a DM to improve the sampling process of VAEs by
denoising the latent codes and feeding VAE with a clean latent code for sample
generation. FSDM (Giannone et al., 2022) uses an attentive conditioning
mechanism and aggregates image patch information using a vision transformer
for image generation for unseen classes.

###  4.5. Exploiting Frequency Components

Approaches in this category aim to improve frequency awareness to improve GM-
DC. FreGAN (Yang et al., 2022c) extracts high-frequency information ($HF$) of
images (related to details in images) using Haar Wavelet transform (Porwik and
Lisowska, 2004) and uses three different modules to emphasize learning high-
frequency information: high-frequency discriminator uses $HF$ as an additional
signal to perform real/fake classification, frequency skip connection feeds
the $HF$ information of each feature map to the next one in $G$ to prevent
frequency loss, and a frequency alignment loss is used to make sure $G$ and
$D$ are learning frequency information in the same pace. WaveGAN (Yang et al.,
2022b) uses similar idea, but in a different setup to address task cGM-2. Gen-
Co extracts some frequency information of the image and feeds it to a separate
$D$ in addition to using original real and fake images. MaskedGAN (Huang et
al., 2022) masks out some frequency bands of the input during training to
enforce the generative model to focus more on under-represented frequency
bands.

###  4.6. Meta-Learning

Meta-learning shifts the learning paradigm from data level to task level to
capture across-task knowledge as meta-knowledge, and then adapt this meta-
knowledge to improve the learning process of unseen tasks in the future. An
abundant of recent works adopt meta-learning to tackle few-shot classification
(Finn et al., 2017; Snell et al., 2017; Vinyals et al., 2016; Sung et al.,
2018; Abdollahzadeh et al., 2021) and few-shot semantic segmentation (Wang et
al., 2019). These works usually follow the episodic learning setup which
matches the way that model is trained and tested. Considering task
distribution $P_{\mathcal{T}}$, a set of training tasks are constructed from
seen classes $\mathcal{T}^{train}=\\{\mathcal{T}^{train}_{i}\\}$, where
$\mathcal{T}^{train}_{i}$ denotes $i^{th}$ training (meta-training) task. The
model is trained on the meta-training tasks and later tested on the meta-test
tasks $\mathcal{T}^{test}=\\{\mathcal{T}^{test}_{j}\\}$ constructed from
unseen classes. Usually meta-training and meta-testing tasks follow the same
distribution $P_{\mathcal{T}}$. Similarly, the approaches in this category use
meta-learning to address image generation: train a generative model on a set
of few-shot image generation tasks constructed from seen classes of a domain,
then, test it on the few-shot image generation tasks from unseen classes of
the same domain.

####  4.6.1. Optimization

Optimization-based meta-learning algorithms are used in these approaches for
learning meta-knowledge. Generative Matching Network (GMN) proposes a similar
attention mechanism used in Matching Networks (Vinyals et al., 2016) for few-
shot image generation with variational inference. FIGR (Clouâtre and Demers,
2019) meta-trains a GAN using Reptile (Nichol et al., 2018). Training has an
inner loop that adapts the GAN weights based on a few-shot image generation
task and an outer loop that updates the meta-knowledge using Reptile. Dawson
(Liang et al., 2020) modifies the inner loop training to directly get the
gradients for the generator from evaluation data. FAML (Phaphuangwittayakul et
al., 2021) uses a similar idea to FIGR, but instead of using the standard GAN
structure, it uses an encoder-decoder architecture for the generator. CML-GAN
(Phaphuangwittayakul et al., 2022) extends FAML (Phaphuangwittayakul et al.,
2021) by leveraging contrastive learning to learn quality representations.

####  4.6.2. Fusion

MatchingGAN (Hong et al., 2020a) learns to generate new images for a category
by fusing the available images of that category. A set of encoders are used to
estimate the similarity between the embedding of the latent code and input
images. Then, these similarities are used as interpolation coefficients by an
auto-encoder to extract the embeddings of the training images and fuse them
for generating new images. F2GAN (Hong et al., 2020b) uses random coefficients
for general information, and attention module for details. The attention
module takes the weighted average of the real image features and the
corresponding features from the decoder to produce the image details. LoFGAN
(Gu et al., 2021) focuses on local features in the fusion process. Given a
batch of images, one sample is selected as a base while the remaining are
utilized as a reference set. This set acts as a feature bank for the fusing
process. WaveGAN (Yang et al., 2022b) adds frequency awareness to LofGAN by
extracting and feeding the frequency components of feature maps to later
layers of the generator. AMMGAN (Li et al., 2023d) utilizes an adaptive fusion
mechanism for learning pixel-wise metric coefficients during the fusion.

####  4.6.3. Transformation

DAGAN (Antoniou et al., 2017) leverages the task of the learning augmentation
manifold in the learning process of the GAN. This is modeled as some
transformations on the input, and these transformations are applied to the new
sample from the unseen classes for sample generation. DeltaGAN (Hong et al.,
2022a) learns the difference between images (delta) in the feature space, and
then uses this delta concept for diverse sample generation. Disco (Hong et
al., 2022b) learns a dictionary based on seen images to encode input images
into visual tokens. These tokens are then fed into the decoder with the style
embedding of seen images to generate images from unseen classes. AGE (Ding et
al., 2022) uses GAN inversion to invert the samples of a category to
$\mathcal{W}^{+}$ space of StyleGAN2 (Karras et al., 2020b). The mean latent
code for all samples of a category is used as a prototype and all differences
are considered as general attributes. These attributes are then used to
diversify sample generation for unseen classes. SAGE (Ding et al., 2023)
addresses the class inconsistency in AGE by taking all given samples from
unseen classes into account during inference. HAE (Li et al., 2022d) uses a
similar idea to AGE (Ding et al., 2022), but in the Hyperbolic space instead
of using Euclidian distance which allows more semantic diversity control. LSO
(Zheng et al., 2023) finds a prototype for each class similar to AGE (Ding et
al., 2022). Then it adjusts GAN to produce similar images to target samples
using latent samples from the neighborhood of the prototype, followed by
updating the prototype in latent space using the adapted GAN.

###  4.7. Modeling Internal Patch Distribution

####  4.7.1. Progressive Training

SinGAN (Shaham et al., 2019) is the pioneering work that makes use of the
internal distribution of the patches within an image to train a generative
model. It trains a pyramid of generators $\\{G_{0},\dots,G_{N}\\}$ against a
pyramid of real images $\\{x_{0},\dots,x_{N}\\}$, where $x_{n}$ is a
downsampled version of input image $x$ by a factor of $r^{n}$. The generator
at scale $n$ uses random noise $z_{n}$ and upsampled version of the generated
image from the lower resolution $\tilde{x}_{n+1}$ as input:
$\tilde{x}_{n}=G_{n}(z_{n},(\tilde{x}_{n+1})\uparrow^{r_{n}})$. Similarly, a
pyramid of discriminators is used where $D_{n}$ compares the $\tilde{x}_{n}$
and $x_{n}$ in patch-level for real-fake classification. SinDDM (Kulikov et
al., 2023) applies the same idea but uses diffusion models with a fully
convolutional lightweight denoiser. ConSinGAN (Hinz et al., 2021) stacks the
new layers for a bigger scale on top of the previous layers used for a smaller
scale instead of using separate generators for each scale. BlendGAN
(Kligvasser et al., 2022) and DEff-GAN (Kumar and Sivakumar, 2023) extend
previous approaches for learning the internal distribution for $k$ images,
thereby allowing for the potential mixing of different image semantics and
improving diversity. SinDiffusion (Wang et al., 2022a) addresses artifacts in
SinGAN due to progressive resolution growth by applying progressive denoising
using a diffusion model architecture.

####  4.7.2. Non-Progressive Training

One-Shot GAN (Sushko et al., 2021) uses a standard generator (single-scale),
but multiple paths for the discriminator to enforce learning objects’
appearance and how to combine them. Within the discriminator the low-level
loss is defined on low-level features and two different losses are defined to
learn the content and the layout in image patches. SinFusion (Nikankin et al.,
2023) explores learning the internal patch distribution from both a single
image and video. SinFusion extends on DPPM (Ho et al., 2020) and reduces the
size of the receptive fields by first removing attention layers, then adopting
ConvNext (Liu et al., 2022) blocks in the U-Net (Ronneberger et al., 2015)
architecture. To reconstruct videos, a series of images are fed into a series
of 3 identical models. The first model predicts the next frame; the second
model denoises and removes small artifacts from the generated images; the last
model interpolates between the different frames.

##  5\. Discussion

Here, we present an analysis of the literature and discuss the research gap
and future directions in GM-DC.

###  5.1. Analysis of the Research Landscape

In this work, we propose a taxonomy of eight different tasks for GM-DC ( Fig.
1, Tab. 2) based on the problem setups of GM-DC publications. Our
investigation of the literature focusing on each task (Fig. 3) reveals that a
significant portion of the works (up to 80%) concentrate on unconditional
generation, either through training from scratch or adapting from a pre-
trained model. Additionally, zero-shot unconditional generation is beginning
to attract more attention. Similarly, adaptation for in-domain classes has
garnered considerable interest for conditional generation. Meanwhile,
conditional generation for out-of-domain classes via adaptation has not been
explored adequately. Furthermore, subject-driven generation, which enables
more control over content generation, is an emerging task. We anticipate
increasing interest on this task as recent text-to-image generative models
become more accessible.

We further present a taxonomy of approaches for GM-DC (Fig. 1, Tab.
LABEL:tab:approaches) as our another contribution. Our study reveals that
transfer learning is a predominant solution for GM-DC, capable of tackling a
large number of tasks (specifically, 5 out of 8 tasks, as indicated in Tab.
LABEL:tab:approaches and Fig. 1), while effectively handling all data
constraints including limited data, few-shot, and zero-shot. Moreover,
$\approx$39% of the studies propose new methods based on transfer learning
(Fig. 3). More than 20% of the studies propose methods based on other
approaches that are compatible to transfer learning, e.g. data augmentation.
These methods could be used with transfer learning-based methods to improve
performance. The primary challenges in transfer learning are selection and
preservation of source knowledge useful for generating high-quality and
diverse target domain samples. Adaptation-aware approach (Zhao et al., 2022a,
2023a) could be a sound direction in this aspect where they consider both
source and target domains (the adaptation process) for knowledge preservation.
Language-guided approaches (Gal et al., 2022b; Zhu et al., 2022a; Alanov et
al., 2022; Guo et al., 2023) are gaining increasing attention due to their
ability to facilitate zero-shot generation through appropriate application of
vision-language models during the transfer learning phase. Visual prompt
tuning (Sohn et al., 2023) is a recent method, which guides the generation of
target domain samples by generating visual tokens.

Data augmentation (Karras et al., 2020a; Zhao et al., 2020b; Tran et al.,
2021) remains a potent technique in GM-DC where it boosts performance under
limited data by increasing coverage of the data distribution through various
transformations. Multi-task objectives (Yang et al., 2021a; Tseng et al.,
2021; Huang et al., 2022) which incorporate additional learning objectives are
usually complementary to data augmentation. Various network architecture
designs (Liu et al., 2021; Li et al., 2022a) that aim to prevent overfitting
or preserve the feature maps are also shown to be significantly effective for
GM-DC. Given that generative models tend to exhibit biases in capturing
frequency components, enhancing the frequency awareness in these models is an
emerging direction for GM-DC (Yang et al., 2022c). Meta-learning (Clouâtre and
Demers, 2019) enables generative models to learn inter-task knowledge from
seen classes, and then handle new generation tasks from unseen classes usually
without fine-tuning (Gu et al., 2021; Hong et al., 2022a). Internal patch-
distribution modeling (Shaham et al., 2019; Nikankin et al., 2023) effectively
trains a generative model from scratch using a single reference image (scene)
to produce novel scene compositions.

Regarding the types of generating models, our study shows that around 86% of
the GM-DC works focus on GANs (Fig. 3). This preference can be attributed to
the extensive research in GANs. Recently, there has been a growing interest in
DMs (12%) and VAEs (3%), particularly VQ-VAE, driven by the success of DMs
(Ramesh et al., 2022; Saharia et al., 2022) and transformer-based token
prediction methods in generative modeling (Chang et al., 2022; Esser et al.,
2021). We anticipate increasing attention directed toward DMs and VQ-VAEs.
Furthermore, our survey reveals an interesting trend: around 64% of the works
focus on addressing the challenging task of few-shot learning, while 33%
concentrate on limited data scenarios. While only 3% of works address zero-
shot learning, we expect growing interest due to recent advancements in
vision-language models (Kwon and Ye, 2023; Li et al., 2023a).

###  5.2. Research Gap and Future Directions

####  5.2.1. Harnessing the power of foundation models

As previously discussed, transfer learning is a prominent and highly effective
solution for GM-DC. Nevertheless, the majority of existing literature uses
pre-trained StyleGAN2 (FFHQ) or BigGAN (ImageNet) networks as source models. A
potential future direction for GM-DC is to explore the capabilities of
foundation models (Bommasani et al., 2021), i.e. large models trained using
massive amounts of data. In particular, recent text-image generation models
including DALL$\cdot$E-2 (Ramesh et al., 2022) ($\approx$3.5B parameters),
Imagen (Saharia et al., 2022) ($\approx$4.6B parameters) and Stable Diffusion
(Rombach et al., 2022) ($\approx$890M parameters) encode knowledge regarding a
wide range of concepts for high-quality, diverse image generation. Leveraging
such foundation models for GM-DC is relatively under-explored.

####  5.2.2. Grounding zero-shot image generative capabilities

Recent studies have demonstrated the feasibility of zero-shot image generation
for well-known concepts, e.g. “Tolkien Elf” (Gal et al., 2022b). However,
grounding zero-shot image generation models to generate evolving/ new semantic
concepts remains a relatively unexplored and challenging area. For instance,
how to generate an image depicting “The coronation of Charles III and Camilla
as King and Queen of the United Kingdom,” an event that occurred in May 2023,
that related images may not be captured by existing models. This requires
strategies that allow continual learning, semantic concept editing, and the
incorporation of temporal contexts.

####  5.2.3.  Knowledge transfer for distant/ remote target domains

Knowledge transfer has received significant attention in GM-DC research. Many
works concentrate on utilizing pre-trained knowledge of a source domain to
enhance learning in the target domain, as evident from the statistics in Fig.
1 and Fig. 3. However, we remark that exploring knowledge transfer for
modeling target domains which are distant/ remote from the source domains
still remains largely unexplored. This problem is challenging, as demonstrated
in our experiment to transfer knowledge from Human Faces $\rightarrow$ Flowers
(Fig. 6), which clearly demonstrates the complexity of the task. We urge more
investigation in knowledge transfer for modeling distant/ remote target
domains in GM-DC research.

####  5.2.4. Exploring different types of generative models for GM-DC

Our analysis reveals that around 85% of GM-DC works focus on GANs and there is
less attention to other types of generative models. Meanwhile, recent
generative models such as diffusion models have made a lot of progress,
achieving comparable performance to GANs in terms of quality and diversity of
generated samples (Dhariwal and Nichol, 2021). We remark that recent
generative models are fundamentally different from GANs, e.g. multiple
iterations are required to generate samples in diffusion models, suggesting
that current GM-DC methods developed for GANs could be sub-optimal for other
types of generative models.

####  5.2.5. Holistic evaluation of GM-DC

Evaluation of GM-DC presents multiple challenges including difficulties in
estimating real data statistics under low-data regimes, lack of unified
framework for human evaluation of GM-DC samples, and heavy reliance on
particular (pre-trained) feature extractors to quantify the capabilities of
GM-DC. In particular, developing holistic evaluation frameworks integrating
both objective measurements and subjective judgements tailored for different
tasks is essential for understanding GM-DC capabilities. Advancing holistic
evaluation is important for GM-DC methods to be applied in a variety of real-
world scenarios.

####  5.2.6. Data-centric approaches for GM-DC

We remark that data-centric approaches (Whang et al., 2023) for advancing GM-
DC have been relatively overlooked in the literature. Majority of GM-DC
methods focus on advancing training procedures based on a given set of
training samples, but little attention has been put on how GM-DC performance
may be affected by characteristics of the given training samples.
Particularly, for GM-DC problems, where a domain is described using limited
training samples, the characteristics of the samples can have noticeable
impact on performance of GM-DC methods, as hinted in our analysis (see Fig.
7). We suggest greater emphasis on data collection, curation and pre-
processing for GM-DC advancement.

###  5.3. Beyond Image Generation

Existing GM-DC works focus on image generation primarily. There are a few
recent works to study other data types. (Zhu et al., 2023) studies 3D shape
generation under few-shot target data (10-shot) utilizing pre-trained 3D
generative models and optimization adaptation to retain the probability
distributions of pairwise adapted samples. CLIP-Sculptor (Sanghi et al., 2023)
leverages CLIP guidance for zero-shot shape generation. (Wang et al., 2023c)
studies few-shot font generation which aims to transfer the source domain
style to the target domain. In particular, they introduce a content fusion
module and a projected character loss to improve the quality of skeleton
transfer for few-shot font generation. (Careil et al., 2023) explores the
problem of few-shot semantic image generation where the objective is to
generate realistic images based on semantic segmentation maps. Their approach
employs transfer learning on both GANs and DMs for few-shot semantic image
synthesis.

##  6\. Conclusion

Generative Modeling under Data Constraints (GM-DC) is a burgeoning research
area. This survey delves into this field by meticulously examining research
papers in this area, encompassing different types of generative models
including VAEs, GANs, and Diffusion Models. Drawing from this analysis, we
identify several challenges encountered in GM-DC, including those related to
training, data selection, and model evaluation. Moreover, we propose two
taxonomies to categorize works related to GM-DC: a task taxonomy that
identifies the variety of generation tasks, and an approach taxonomy that
categorizes the extensive list of solutions for these tasks. We present a
Sankey diagram to illuminate the interactions between different GM-DC tasks,
approaches, and methods. Additionally, we present an organized review of
existing GM-DC works and discuss research gaps and future research directions.
Our aspiration is that this survey not only could offer valuable insights to
researchers but also help spark further advancements in GM-DC.

Ethics Statement. Generative models could be mis-used to disseminate mis- and
disinformation due to their ability to generate realistic content. In
particular, advanced generative models could be mis-used by malicious users to
fabricate deepfake images, portraying individuals engaging in actions they
never actually performed. Advances in GM-DC could exacerbate the situation as
it becomes possible to generate realistic content with less data. We advocate
for ethical and responsible usage of GM-DC methods and studying of mitigation
techniques (Mirsky and Lee, 2021; Chandrasegaran et al., 2021, 2022a; Zhao et
al., 2023b; Wen et al., 2023).

## References

  * (1)
  * Abdollahzadeh et al. (2021) Milad Abdollahzadeh, Touba Malekzadeh, and Ngai-Man Man Cheung. 2021\.  Revisit multimodal meta-learning through the lens of multi-task learning. In _Advances in Neural Information Processing Systems_.
  * Ackley et al. (1985) David H Ackley, Geoffrey E Hinton, and Terrence J Sejnowski. 1985\.  A learning algorithm for Boltzmann machines.  _Cognitive science_ 9, 1 (1985), 147–169.
  * Aghabozorgi et al. (2023) Mehran Aghabozorgi, Shichong Peng, and Ke Li. 2023\.  Adaptive IMLE for few-shot pretraining-free generative modelling. In _Proceedings of the International Conference on Machine Learning_.
  * Aksac et al. (2019) Alper Aksac, Douglas J Demetrick, Tansel Ozyer, and Reda Alhajj. 2019.  BreCaHAD: A dataset for breast cancer histopathological annotation and diagnosis.  _BMC research notes_ 12, 1 (2019), 1–3.
  * Alanov et al. (2022) Aibek Alanov, Vadim Titov, and Dmitry P Vetrov. 2022\.  HyperDomainNet: Universal domain adaptation for generative adversarial networks. In _Advances in Neural Information Processing Systems_.
  * Antoniou et al. (2017) Antreas Antoniou, Amos Storkey, and Harrison Edwards. 2017\.  Data augmentation generative adversarial networks.  _arXiv preprint arXiv:1711.04340_ (2017).
  * Arjovsky et al. (2017) Martin Arjovsky, Soumith Chintala, and Léon Bottou. 2017\.  Wasserstein generative adversarial networks. In _Proceedings of the International Conference on Machine Learning_.
  * Bar-Tal et al. (2023) Omer Bar-Tal, Lior Yariv, Yaron Lipman, and Tali Dekel. 2023\.  Multidiffusion: Fusing diffusion paths for controlled image generation. In _Proceedings of the International Conference on Machine Learning_.
  * Bartunov and Vetrov (2018) Sergey Bartunov and Dmitry Vetrov. 2018.  Few-shot generative modeling with generative matching networks. In _International Conference on Artificial Intelligence and Statistics_.
  * Bińkowski et al. (2018) Mikołaj Bińkowski, Dougal J. Sutherland, Michael Arbel, and Arthur Gretton. 2018.  Demystifying MMD GANs. In _International Conference on Learning Representations_.
  * Bommasani et al. (2021) Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, et al. 2021\.  On the opportunities and risks of foundation models.  _arXiv preprint arXiv:2108.07258_ (2021).
  * Brock et al. (2019) Andrew Brock, Jeff Donahue, and Karen Simonyan. 2019\.  Large scale GAN training for high fidelity natural image synthesis. In _International Conference on Learning Representations_.
  * Careil et al. (2023) Marlène Careil, Jakob Verbeek, and Stéphane Lathuilière. 2023\.  Few-shot semantic image synthesis with class affinity transfer. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_.
  * Casanova et al. (2021) Arantxa Casanova, Marlene Careil, Jakob Verbeek, Michal Drozdzal, and Adriana Romero Soriano. 2021\.  Instance-conditioned GAN. In _Advances in Neural Information Processing Systems_.
  * Chai et al. (2022) Lucy Chai, Michael Gharbi, Eli Shechtman, Phillip Isola, and Richard Zhang. 2022.  Any-resolution training for high-resolution image synthesis. In _Proceedings of the European Conference on Computer Vision_.
  * Chandrasegaran et al. (2022a) Keshigeyan Chandrasegaran, Ngoc-Trung Tran, Alexander Binder, and Ngai-Man Cheung. 2022a.  Discovering Transferable Forensic Features for CNN-generated Images Detection. In _Proceedings of the European Conference on Computer Vision (ECCV)_.
  * Chandrasegaran et al. (2021) Keshigeyan Chandrasegaran, Ngoc-Trung Tran, and Ngai-Man Cheung. 2021\.  A closer look at Fourier spectrum discrepancies for CNN-generated images detection. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_.
  * Chandrasegaran et al. (2022b) K. Chandrasegaran, N. T. Tran, Y. Zhao, and N. M. Cheung. 2022b.  Revisiting label smoothing and knowledge distillation compatibility: What was missing?. In _Proceedings of the International Conference on Machine Learning_.
  * Chang et al. (2022) Huiwen Chang, Han Zhang, Lu Jiang, Ce Liu, and William T Freeman. 2022.  Maskgit: Masked generative image transformer. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_.
  * Chauhan and Shah (2021) Uttam Chauhan and Apurva Shah. 2021.  Topic modeling using latent Dirichlet allocation: A survey.  _ACM Computing Surveys (CSUR)_ 54, 7 (2021), 1–35.
  * Chen et al. (2021) Tianlong Chen, Yu Cheng, Zhe Gan, Jingjing Liu, and Zhangyang Wang. 2021.  Data-efficient GAN training beyond (just) augmentations: A lottery ticket perspective. In _Advances in Neural Information Processing Systems_.
  * Choi et al. (2020) Yunjey Choi, Youngjung Uh, Jaejun Yoo, and Jung-Woo Ha. 2020\.  StarGAN v2: Diverse image synthesis for multiple domains. In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_.
  * Chong and Forsyth (2022) Min Jin Chong and David Forsyth. 2022.  JoJoGAN: One shot face stylization. In _Proceedings of the European Conference on Computer Vision_. Springer.
  * Clarke (2022) Laurie Clarke. 2022\.  When AI can make art – what does it mean for creativity?  (2022).  https://www.theguardian.com/technology/2022/nov/12/when-ai-can-make-art-what-does-it-mean-for-creativity-dall-e-midjourney
  * Clouâtre and Demers (2019) Louis Clouâtre and Marc Demers. 2019.  FIGR: Few-shot Image Generation with Reptile.  _arXiv preprint arXiv:1901.02199_ (2019).
  * Cong et al. (2020) Yulai Cong, Miaoyun Zhao, Jianqiao Li, Sijia Wang, and Lawrence Carin. 2020.  GAN memory with no forgetting. In _Advances in Neural Information Processing Systems_.
  * Cui et al. (2022) Kaiwen Cui, Jiaxing Huang, Zhipeng Luo, Gongjie Zhang, Fangneng Zhan, and Shijian Lu. 2022\.  GenCo: Generative co-training for generative adversarial networks with limited data. In _Proceedings of the AAAI Conference on Artificial Intelligence_.
  * Cui et al. (2023) Kaiwen Cui, Yingchen Yu, Fangneng Zhan, Shengcai Liao, Shijian Lu, and Eric Xing. 2023\.  KD-DLGAN: Data limited image generation via knowledge distillation. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_.
  * Dai et al. (2022) Mengyu Dai, Haibin Hang, and Xiaoyang Guo. 2022.  Adaptive feature interpolation for low-shot image generation. In _Proceedings of the European Conference on Computer Vision_.
  * Deng et al. (2009) Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. 2009\.  Imagenet: A large-scale hierarchical image database. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  BERT: Pre-training of deep bidirectional transformers for language understanding. In _Proceedings of North American Chapter of the Association for Computational Linguistics_.
  * Dhariwal and Nichol (2021) Prafulla Dhariwal and Alexander Nichol. 2021.  Diffusion models beat GANs on image synthesis. In _Advances in Neural Information Processing Systems_.
  * Ding et al. (2023) Guanqi Ding, Xinzhe Han, Shuhui Wang, Xin Jin, Dandan Tu, and Qingming Huang. 2023\.  Stable attribute group editing for reliable few-shot image generation.  _arXiv preprint arXiv:2302.00179_ (2023).
  * Ding et al. (2022) Guanqi Ding, Xinzhe Han, Shuhui Wang, Shuzhe Wu, Xin Jin, Dandan Tu, and Qingming Huang. 2022.  Attribute group editing for reliable few-shot image generation. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_.
  * Duan et al. (2023) Yuxuan Duan, Li Niu, Yan Hong, and Liqing Zhang. 2023\.  WeditGAN: Few-shot image generation via latent space relocation.  _arXiv preprint arXiv:2305.06671_ (2023).
  * Durall et al. (2020) Ricard Durall, Margret Keuper, and Janis Keuper. 2020\.  Watch your up-convolution: CNN based generative deep neural networks are failing to reproduce spectral distributions. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_.
  * Easton and Jacob (2023) Joe Easton and Sarah Jacob. 2023.  New Beatles song shows AI opportunity for music, Citigroup says.  (2023).  https://www.bloomberg.com/news/articles/2023-06-15/new-beatles-song-shows-ai-opportunity-for-music-citigroup-says?leadSource=uverify%20wall
  * Esser et al. (2021) Patrick Esser, Robin Rombach, and Bjorn Ommer. 2021\.  Taming transformers for high-resolution image synthesis. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_.
  * Fang et al. (2022) Tiantian Fang, Ruoyu Sun, and Alex Schwing. 2022.  DigGAN: Discriminator gradIent Gap regularization for GAN training with limited data. In _Advances in Neural Information Processing Systems_.
  * Finn et al. (2017) Chelsea Finn, Pieter Abbeel, and Sergey Levine. 2017\.  Model-agnostic meta-learning for fast adaptation of deep network. In _Proceedings of the International Conference on Machine Learning_.
  * Frankle and Carbin (2019) Jonathan Frankle and Michael Carbin. 2019.  The lottery ticket hypothesis: Finding sparse, trainable neural networks. In _International Conference on Learning Representations_.
  * Fu et al. (2023) Stephanie Fu, Netanel Tamir, Shobhita Sundaram, Lucy Chai, Richard Zhang, Tali Dekel, and Phillip Isola. 2023.  DreamSim: Learning New Dimensions of Human Visual Similarity using Synthetic Data.  _arXiv:2306.09344_ (2023).
  * Gal et al. (2022a) Rinon Gal, Yuval Alaluf, Yuval Atzmon, Or Patashnik, Amit H Bermano, Gal Chechik, and Daniel Cohen-Or. 2022a.  An image is worth one word: Personalizing text-to-image generation using textual inversion.  _arXiv preprint arXiv:2208.01618_ (2022).
  * Gal et al. (2022b) Rinon Gal, Or Patashnik, Haggai Maron, Amit H Bermano, Gal Chechik, and Daniel Cohen-Or. 2022b.  StyleGAN-NADA: CLIP-guided domain adaptation of image generators.  _ACM Transactions on Graphics_ 41, 4 (2022), 1–13.
  * Giannone et al. (2022) Giorgio Giannone, Didrik Nielsen, and Ole Winther. 2022\.  Few-shot diffusion models.  _arXiv preprint arXiv:2205.15463_ (2022).
  * Giannone and Winther (2022) Giorgio Giannone and Ole Winther. 2022.  SCHA-VAE: Hierarchical context aggregation for few-Shot generation. In _Proceedings of the International Conference on Machine Learning_.
  * Gonzales and Wintz (1987) Rafael C Gonzales and Paul Wintz. 1987.  _Digital image processing_.  Addison-Wesley Longman Publishing Co., Inc.
  * Goodfellow et al. (2014) Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. 2014\.  Generative adversarial nets. In _Advances in Neural Information Processing Systems_.
  * Gou et al. (2023) Yao Gou, Min Li, Yilong Lv, Yusen Zhang, Yuhang Xing, and Yujie He. 2023\.  Rethinking cross-domain semantic relation for few-shot image generation.  _Applied Intelligence_ (2023), 1–14.
  * Grigoryev et al. (2022) Timofey Grigoryev, Andrey Voynov, and Artem Babenko. 2022\.  When, why, and which pretrained GANs are useful?. In _International Conference on Learning Representations_.
  * Gu et al. (2021) Zheng Gu, Wenbin Li, Jing Huo, Lei Wang, and Yang Gao. 2021.  LoFGAN: Fusing local representations for few-shot image generation. In _Proceedings of the IEEE/CVF International Conference on Computer Vision_.
  * Gulrajani et al. (2017) Ishaan Gulrajani, Faruk Ahmed, Martin Arjovsky, Vincent Dumoulin, and Aaron C Courville. 2017\.  Improved training of Wasserstein GANs. In _Advances in Neural Information Processing Systems_.
  * Guo et al. (2023) Jiayi Guo, Chaofei Wang, You Wu, Eric Zhang, Kai Wang, Xingqian Xu, Humphrey Shi, Gao Huang, and Shiji Song. 2023\.  Zero-shot generative model adaptation via image-specific prompt learning. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_.
  * He et al. (2020) Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. 2020.  Momentum contrast for unsupervised visual representation learning. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_.
  * He et al. (2016) Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016\.  Deep residual learning for image recognition. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_.
  * Hern (2023) Alex Hern. 2023\.  The AI tools that will write our emails, attend our meetings – and change our lives .  (2023).  https://www.theguardian.com/technology/2023/mar/21/the-ai-tools-that-will-write-our-emails-attend-our-meetings-and-change-our-lives
  * Heusel et al. (2017) Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. 2017\.  GANs trained by a two time-scale update rule converge to a local nash equilibrium. In _Advances in Neural Information Processing Systems_.
  * Hinton et al. (2015) Geoffrey Hinton, Oriol Vinyals, and Jeffrey Dean. 2015\.  Distilling the knowledge in a neural network. In _NeurIPS Deep Learning and Representation Learning Workshop_.
  * Hinz et al. (2021) Tobias Hinz, Matthew Fisher, Oliver Wang, and Stefan Wermter. 2021.  Improved techniques for training single-image GANs. In _Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision_.