# Real-Fake: Effective Training Data Synthesis Through Distribution Matching

Jianhao Yuan$\dagger\diamond$ Jie Zhang$\ddagger$ Shuyang Sun$\dagger$ Philip
Torr$\dagger$ Bo Zhao$\diamond$
$\dagger$University of Oxford $\ddagger$ ETH Zurich $\diamond$ Beijing Academy
of Artificial Intelligence

###### Abstract

Synthetic training data has gained prominence in numerous learning tasks and
scenarios, offering advantages such as dataset augmentation, generalization
evaluation, and privacy preservation. Despite these benefits, the efficiency
of synthetic data generated by current methodologies remains inferior when
training advanced deep models exclusively, limiting its practical utility. To
address this challenge, we analyze the principles underlying training data
synthesis for supervised learning and elucidate a principled theoretical
framework from the distribution-matching perspective that explicates the
mechanisms governing synthesis efficacy. Through extensive experiments, we
demonstrate the effectiveness of our synthetic data across diverse image
classification tasks, both as a replacement for and augmentation to real
datasets, while also benefits challenging tasks such as out-of-distribution
generalization and privacy preservation111Code is released at
https://github.com/BAAI-DCAI/Training-Data-Synthesis. Corresponding to Bo Zhao
$\langle$<EMAIL_ADDRESS>$\rangle$..

##  1 Introduction

Large-scale annotated datasets (Deng et al., 2009; Lin et al., 2014) are
essential in deep learning for image classification, providing the
comprehensive information that models need to effectively discover patterns,
learn representations, and generate accurate predictions. However, manually
collecting such datasets is time-consuming and labor-intensive, and may cause
privacy concerns. Given these challenges, training data synthesis offers a
promising alternative to traditional data collection for augmenting or
replacing original real datasets.

Among numerous image synthesis strategies, deep generative models have gained
significant attention, primarily due to their capacity to produce high-
fidelity images. Early studies (Besnier et al., 2020; Li et al., 2022; Zhang
et al., 2021; Zhao & Bilen, 2022) utilize Generative Adversarial Networks
(GANs) to synthesize annotated training data for image classification and
segmentation. Recently, more works have focused on synthesizing training data
with the powerful diffusion models for self-supervised pre-training (Tian et
al., 2023), transfer learning (He et al., 2022), domain generalization (Yuan
et al., 2022; Bansal & Grover, 2023), and supervised image classification
(Azizi et al., 2023; Sarıyıldız et al., 2023; Lei et al., 2023). However,
despite the extensive research, a notable performance gap persists when
comparing the performances of models trained on synthetic data to those
trained on real data. A primary reason for this discrepancy is the
misalignment between the synthetic and real data distributions, even the
diffusion model is trained on web-scale datasets, as illustrated in Fig. 1(a).
While previous works have attempted to address this issue through heuristic-
driven approaches such as prompt engineering (Sarıyıldız et al., 2023; Lei et
al., 2023) and the expensive inversion approaches (Zhao & Bilen, 2022; Zhou et
al., 2023), these solutions are neither sufficient nor efficient. Furthermore,
there is an absence of theoretical frameworks that can adequately explain and
analyze the efficacy of synthetic training data in a principled way.

(a) Distribution Visualization

(b) Baseline Comparison

(c) Scale-up Performance

Figure 1: Left: Visualization of the synthetic and real ImageNet data
distribution using the first two principal components of features extracted by
the CLIP image encoder. Our synthetic data better aligns with real data
distribution than the baseline (vanilla Stable Diffusion). Middle: Our
synthetic data achieves better performance compared with the baseline, and can
effectively augment real data across all datasets. Right: Scaling up synthetic
training data can improve the image classification performances in both in-
distribution and out-of-distribution (OOD) tasks, even outperform training
with real data in OOD tasks.

To further enhance the quality and utility of synthetic training data produced
by deep generative models, we present a theoretical framework for training
data synthesis from a distribution-matching perspective. Specifically,
starting from the first principle of supervised learning (Sutskever, 2023;
Cunningham et al., 2008), we recast the training data synthesis as a
distribution matching problem. This emphasizes two primary principles of
synthetic data: (1) The distribution discrepancy between target and synthetic
data, and (2) The cardinality of the training set.

Building on this foundation, we employ the state-of-the-art text-to-image
diffusion model, Stable Diffusion (Rombach et al., 2022), to undertake a
careful analysis and refinement of the training objectives, condition
generation, and prior initialization to achieve a better alignment between
synthetic and target data distributions. We empirically validate our
theoretical framework and synthesis method across diverse benchmarks, covering
various scenarios: (1) training exclusively with synthetic data, (2)
augmenting the real training data, and (3) evaluating the scaling law between
synthetic data and performance. In particular, for ImageNet1k classification
using ResNet50 (He et al., 2016), training solely with synthetic data
equivalent to 1 $\times$ the original real data size yielded a 70.9% Top1
classification accuracy, which increases to 76.0% when using 10 $\times$
synthetic data. Additionally, we explore (1) out-of-distribution (OOD)
generalization and (2) privacy preservation when learning with synthetic data,
and present promising results. Beyond advancing the state of the art, our
findings offer insights into potential strategies for refining the training
data synthesis pipeline. Our primary contributions are as follows:

  * •

Introducing a principled distribution matching framework for training data
synthesis, emphasizing two foundational aspects that drive the effectiveness
of synthetic data.

  * •

Employing the state-of-the-art text-to-image diffusion model, with a
comprehensive analysis and refinement of its components, to better align
synthetic and target data distributions.

  * •

Advancing the state-of-the-art in training data synthesis for image
classification tasks, while demonstrating the advantages in OOD generalization
and privacy preservation.

##  2 Background

###  2.1 Training Data Synthesis

Synthesizing informative training samples remains a compelling yet challenging
area of research. One line of work focuses on synthesis through pre-trained
deep generative models. Early attempts (Zhang et al., 2021; Li et al., 2022;
Zhao & Bilen, 2022) explore Generative Adversarial Networks (GANs) (Goodfellow
et al., 2014; Brock et al., 2018) for informative and annotated training
sample synthesis. Recently, diffusion models have gained more attention. He et
al. (2022); Tian et al. (2023) employ synthetic data from text-conditioned
diffusion models for self-supervised pre-training and few/zero-shot learning,
highlighting the transfer learning capacity of synthetic training data. Also,
Yuan et al. (2022); Bansal & Grover (2023); Vendrow et al. (2023) demonstrate
the utility of synthetic data in out-of-distribution settings by augmenting
and diversifying the training dataset. Additionally, Sarıyıldız et al. (2023);
Lei et al. (2023); Azizi et al. (2023) utilize Stable Diffusion (Rombach et
al., 2022) and Imagen (Saharia et al., 2022) with prompt engineering to
synthesize class-conditioned samples, showcasing the potential of using
synthetic data solely for training classification models.

Despite the promising results in various tasks, a noticeable performance gap
remains between models trained on real and synthetic datasets. This difference
can be attributed to the misalignment between the distributions of the
synthetic training data and the target downstream data. Existing research
(Sarıyıldız et al., 2023; Lei et al., 2023) employs prompt engineering to
bridge the domain gap, which is insufficient. Zhou et al. (2023) implement
diffusion inversion to obtain synthetic images close to real ones, which is
expensive and unscalable to large datasets. To understand the mechanisms
underlying the efficacy of synthetic data, we aim to find the theoretical
principle and further tackle the challenges associated with suboptimal
training data synthetic.

Another line of work focuses on synthesizing informative training samples by
distilling the large real dataset into smaller synthetic one, i.e., dataset
distillation. These methods optimize synthetic images (pixels) through the
minimization of meta-loss (Wang et al., 2018), gradient matching loss (Zhao et
al., 2021), trajectory matching loss (Cazenavette et al., 2022) and
distribution matching loss (Zhao & Bilen, 2023; Zhao et al., 2023). In
particular, distribution matching approaches have gained prominence due to
their model-agnostic nature. However, the intricate and expensive optimization
processes of dataset distillation often pose challenges in scalability.
Despite these challenges, the principled approaches of dataset distillation
have proven effective and inspired our work.

###  2.2 Diffusion Probabilistic Models

Diffusion Probabilistic Models (Sohl-Dickstein et al., 2015; Ho et al., 2020;
Nichol & Dhariwal, 2021) are latent variable models that learn a data
distribution $q(\bm{x}_{0})$ by reversing a gradual noising process. They
define a forward diffusion Markov process
$q\left(\bm{x}_{1:T}\mid\bm{x}_{0}\right)=\prod_{t=1}^{T}q\left(\bm{x}_{t}\mid\bm{x}_{t-1}\right)$
that uses a handcrafted Gaussian transition kernel,
$q\left(\bm{x}_{t}\mid\bm{x}_{t-1}\right)=\mathcal{N}\left(\bm{x}_{t};\sqrt{1-\beta_{t}}\bm{x}_{t-1},\beta_{t}\mathbf{I}\right)$,
with a noise schedule $\beta_{t}\in(0,1)$, to transform the data distribution
into a known prior Gaussian distribution. Then, a reverse Markovian process
$p_{\bm{\theta}}\left(\bm{x}_{0:T}\right)=p\left(\bm{x}_{T}\right)\prod_{t=1}^{T}p_{\bm{\theta}}\left(\bm{x}_{t-1}\mid\bm{x}_{t}\right)$
is learned to gradually remove noise added in the forward process, with
Gaussian transitions parameterized by a neural network:
$p_{\bm{\theta}}\left(\bm{x}_{t-1}\mid\bm{x}_{t}\right)=\mathcal{N}\left(\bm{x}_{t-1};\bm{\mu}_{\bm{\theta}}\left(\bm{x}_{t},t\right),\bm{\Sigma}_{\bm{\theta}}\left(\bm{x}_{t},t\right)\right)$.
The training objective of the diffusion model is to minimize the KL-divergence
between the joint distributions of the forward and backward processes. This
can be further simplified to optimizing the standard variational bound on the
negative log-likelihood of the model (Yang et al., 2022) (detailed proof can
be found in Apx. A).

$\operatorname{KL}\left[q\left(\bm{x}_{0},\bm{x}_{1},\cdots,\bm{x}_{T}\right)\|p_{\bm{\theta}}\left(\bm{x}_{0},\bm{x}_{1},\cdots,\bm{x}_{T}\right)\right]\geq\mathbb{E}\left[-\log
p_{\bm{\theta}}\left(\bm{x}_{0}\right)\right]$ (1)

In practice, by marginalizing over the intermediate sampling steps, an
analytic form of the forward process can be obtained:
$q\left(\bm{x}_{t}\mid\bm{x}_{0}\right)=\mathcal{N}\left(\bm{x}_{t};\sqrt{\bar{\alpha}_{t}}\bm{x}_{0},(1-\bar{\alpha}_{t})\mathbf{I}\right)$,
where $\alpha_{t}:=1-\beta_{t}$ and
$\bar{\alpha}_{t}:=\prod_{s=0}^{t}\alpha_{s}$. By further setting the variance
in the reverse process to a non-learnable constant $\sigma_{t}$ and choosing a
specific parameterization of $\epsilon_{\bm{\theta}}$ to predict the added
noise $\epsilon$, the overall training objective becomes:

$\mathbb{E}_{\bm{x},\bm{\epsilon}\sim\mathcal{N}(0,1),t}\left[\frac{\beta_{t}^{2}}{2\sigma_{t}^{2}\alpha_{t}(1-\bar{\alpha}_{t})}\left\|\bm{\epsilon}-\bm{\epsilon}_{\bm{\theta}}\left(\sqrt{\bar{\alpha}_{t}}\bm{x}_{0}+\sqrt{1-\bar{\alpha}_{t}}\bm{\epsilon},t\right)\right\|^{2}\right],$
(2)

where
$\bm{x}_{t}=\sqrt{\bar{\alpha}_{t}}\bm{x}_{0}+\sqrt{1-\bar{\alpha}_{t}}\bm{\epsilon}$
is derived from the reparameterization of the marginal distribution
$q\left(\bm{x}_{t}\mid\bm{x}_{0}\right)$. This objective (Ho et al., 2020)
aligns with the denoising score matching loss of Score-based Generative Models
(SGM) (Song & Ermon, 2019) when appropriately reweighted.

Leveraging the theoretical foundation, in this study, we focus on the state-
of-the-art Latent Diffusion Model (LDM), Stable Diffusion (Rombach et al.,
2022), to provide an alternative interpretation of its training objectives and
generative sampling process from a distribution matching perspective for
training data synthesis.

##  3 Training Data Synthesis: A Distribution Matching Perspective

The goal of training data synthesis is to generate synthetic data from the
target distribution $D:=q(x,y)$ of data $x$ and annotation $y$. In supervised
training (Sutskever, 2023; Cunningham et al., 2008), the difference between
training and testing error over the whole target distribution $D$ is bounded
by the inverse of the square root of the sampled training set cardinality
$|S|$222The probability that the error is lower than a bound, which explicitly
depends on the cardinality of the training set, is guaranteed to be higher
than $1-\delta$, where $\delta$ is small.. Please refer to Apx. B for the
detailed proof.

$\operatorname{Pr}_{S\sim
D^{|S|}}\left[\operatorname{Test}_{D}(f)-\operatorname{Train}_{S}(f)\leq\sqrt{\frac{\log|\mathcal{F}|+\log
1/\delta}{|S|}}\forall f\in\mathcal{F}\right]>1-\delta$ (3)

For training data synthesis given a fixed model space $\mathcal{F}$ (i.e.
fixed model structure with finite parameters), a salient takeaway from Eq. 3
is the identification of two pivotal factors: (1) Training and testing data
distribution discrepancy (2) Cardinality of training set. This formalizes the
first principle of training data synthesis: With infinite training samples
from target distribution, the testing error will converge to the minimized
training error.

However, direct sampling from data distribution can be intractable, we instead
learn a generative model $p_{\bm{\theta}}$, parameterized with $\bm{\theta}$,
that can synthesize data following the same distribution, i.e.,
$p_{\bm{\theta}}(\bm{x},y)=q(\bm{x},y)$ (Bishop, 2006). This effectively
converts the problem of informative training data synthesis into a
distribution matching problem. We can further reframe such distribution
matching problems as
$q(y|\bm{x})q(\bm{x})=p_{\bm{\theta}}(y|\bm{x})p_{\bm{\theta}}(\bm{x})$ using
Bayes rule. This allows us to separate the problem of matching a joint data-
annotation distribution as two sub-problems: (1) data distribution matching
$q(\bm{x})=p_{\bm{\theta}}(\bm{x})$; (2) conditioned class likelihood matching
$q(y|\bm{x})=p_{\bm{\theta}}(y|\bm{x})$. Under the classification protocol,
the former ensures in-distribution data synthesis, and the latter ensures a
robust decision boundary between classes. Overall, the objective of training
data synthesis for supervised learning can be framed as the following
optimization:

$\boxed{S^{*}=\operatorname*{arg\,min}_{S\sim
p_{\bm{\theta}}(\bm{x},y)}\left(D(q(\bm{x}),p_{\bm{\theta}}(\bm{x}))+D(q(y|\bm{x}),p_{\mathbf{\bm{\theta}}}(y|\bm{x}))-\lambda|S|\right)}$
(4)

where $S^{*}$ denotes the optimal synthetic data sampled from the learned
distribution $S\sim p_{\bm{\theta}}(\bm{x},y)$. $D(\cdot,\cdot)$ is a distance
measure between two distributions. The regularization term $\lambda|S|$ with
$\lambda\in\mathbb{R}^{+}$ encourages a larger training set. Fortunately, both
distribution matching problems can be solved with deep generative models. In
particular, with the diffusion model, the data distribution is learned through
a denoising process, and the class likelihood can be modeled through
classifier or classifier-free guidance (Dhariwal & Nichol, 2021; Ho &
Salimans, 2022).

Based on this theoretical framework, we perform an analysis of each component
of the diffusion model in the context of distribution matching and propose
potential improvements. Specifically, we introduce a distribution matching-
centric synthesis framework tailored for training data synthesis, including
three aspects of (1) feature distribution alignment (2) conditioned visual
guidance (3) latent prior initialization.

###  3.1 Distribution Matching with Maximum Mean Discrepancy

Firstly, we quantify the distribution discrepancy between target and synthetic
data. Although the objective of KL-divergence minimization in the diffusion
model has implicitly provided an upper bound for the data distribution
matching (Yang et al., 2022), it is observed to be loose (Kingma et al., 2021)
due to the gap between the bound and the negative log-likelihood Eq. 1. More
importantly, the discrete nature of empirical distribution (i.e. data samples)
in the context of training data synthesis further makes it a suboptimal
measure. Instead, we measure this discrepancy with the Maximum Mean
Discrepancy (MMD) (Gretton et al., 2008), which alleviates the problem of
potentially biased discrete distribution as a sample test and is widely and
successfully applied in dataset distillation Zhao & Bilen (2023) because of
its non-parametric nature and simplicity. At the end, we find a mathematical
equivalence between the training objective of diffusion model and the
minimization of the MMD upper bound under certain assumptions, which allows us
to further relax the variational bound and better align the data distribution
in the feature space.

Consider a (real) target dataset
$\mathcal{R}=\left\\{\left(k_{1},y_{1}\right),\ldots,\left(k_{|\mathcal{R}|},y_{|\mathcal{R}|}\right)\right\\}$
and a synthetic dataset
$\mathcal{S}=\left\\{\left(s_{1},y_{1}\right),\ldots,\left(s_{|\mathcal{S}|},y_{|\mathcal{S}|}\right)\right\\}$.
Our objective is to minimize the MMD between target data distribution $q(x)$
and the synthetic data distribution $p_{\bm{\theta}}(x)$ represented by some
feature extractor $\psi\in\mathcal{F}$.

$\operatorname{MMD}[\mathcal{F},p,q]=\sup_{\left\|\psi_{\vartheta}\right\|_{\mathcal{H}}\leq
1}\left(\mathbb{E}_{q}\left[\psi(\mathcal{R})\right]-\mathbb{E}_{p}\left[\psi(\mathcal{S})\right]\right),$
(5)

where $\psi$ represents a function residing within a unit ball in the
universal Reproducing Kernel Hilbert Space $\mathcal{H}$ (RKHS) (Hilbert,
1904). By following the reproductive properties of RKHS and empirically
estimating the expectations for all distributions, we can simplify Eq. 5 to
Eq. 6:

$\operatorname{MMD}^{2}[\mathcal{F},p,q]=||\frac{1}{|\mathcal{T}|}\sum_{i=1}^{|\mathcal{T}|}\psi_{\vartheta}\left(k_{i}\right)-\frac{1}{|\mathcal{S}|}\sum_{j=1}^{|\mathcal{S}|}\psi_{\vartheta}\left(\bm{s}_{j}\right)||_{\mathcal{H}}^{2}.$
(6)

Please refer to Apx. C for the detailed proof.

In Latent Diffusion Model, $\psi$ can be conceptualized as the Variational
Autoencoder (VAE) with the latent embedding serving as the feature map for MMD
computation. The distributions of target and synthetic data can be effectively
approximated by the initial noise-free latent embedding $\bm{x}_{0}$ and the
predicted denoised latent embedding
$\bm{x}_{\bm{\theta}}\left(\bm{x}_{t},t\right)$. Assuming that MMD is computed
within the same batch, where $|\mathcal{R}|=|\mathcal{S}|=|\mathcal{N}|$, our
objective can be further refined as
$||\frac{1}{|\mathcal{N}|}\sum_{i=1}^{|\mathcal{N}|}(\bm{x}-\bm{x}_{\bm{\theta}}\left(\bm{x}_{t},t\right))||_{\mathcal{H}}^{2}$.
Following Ho et al. (2020), choosing the parameterization with
$\bm{\epsilon}_{\bm{\theta}}$ as a predictor of added noise $\bm{\epsilon}$
from $x_{t}$ in denoising, allows us to frame our distribution matching
objective (DM) as Eq. 7:

$L_{DM}:=||\frac{1}{|\mathcal{N}|}\sum_{i=1}^{|\mathcal{N}|}\left(\bm{\epsilon}_{0}-\bm{\epsilon}_{\bm{\theta}}\left(\bm{x}_{t},t\right)\right)||_{\mathcal{H}}^{2}\leq\frac{1}{|\mathcal{N}|}\sum_{i=1}^{|\mathcal{N}|}||\left(\bm{\epsilon}_{0}-\bm{\epsilon}_{\bm{\theta}}\left(\bm{x}_{t},t\right)\right)||_{\mathcal{H}}^{2}.$
(7)

Since the mean and norm operations are both convex functions, by applying
Jensen’s inequality, we obtain an upper bound on the original distribution
matching objective under equal weighting to each time step. Note that, this
expression resembles the diffusion model training loss Eq. 2. This indicates
when training the diffusion model, we implicitly optimize an upper bound of
the MMD between synthetic and real data distribution. Moreover, this allows us
to directly optimize the objective in Eq. 7 to mitigate the losseness in the
variational bound. We use this objective to augment the original diffusion
model loss during finetuning, which could ensure a more aligned feature
distribution under the MMD measure.

###  3.2 Conditioned Generation via Text-Vision Guidance

Beyond the general feature-level distribution matching, i.e.,
$q(\bm{x})=p_{\bm{\theta}}(\bm{x})$, a pivotal aspect of training data
synthesis is to ensure a congruent conditional class distribution with well-
defined decision boundaries, i.e., $q(y|\bm{x})=p_{\bm{\theta}}(y|\bm{x})$.
Classifier(-free) guidance (Dhariwal & Nichol, 2021; Ho & Salimans, 2022)
plays a crucial role in the conditioned sampling process in the diffusion
model. Under SGM framework, to match the conditioned class likelihood, we can
equivalently match the score function of each distribution, i.e.
$\nabla_{\bm{x}_{t}}\log q(y|\bm{x})=\nabla_{\bm{x}_{t}}\log
p_{\bm{\theta}}(y|\bm{x})$, which is estimated through the noise prediction
$\epsilon_{\bm{\theta}}\left(\bm{x}_{t}\right)$ by reformulating the
conditional score function as Eq. 8:

$\nabla_{\bm{x}_{t}}\log
p_{\bm{\theta}}\left(y\mid\bm{x}_{t}\right)=\nabla_{\bm{x}_{t}}\log
p_{\bm{\theta}}\left(\bm{x}_{t}\mid y\right)-\nabla_{\bm{x}_{t}}\log
p_{\bm{\theta}}\left(\bm{x}_{t}\right)=\frac{1}{\sqrt{1-\bar{\alpha}_{t}}}\left(\bm{\epsilon}_{\bm{\theta}}\left(\bm{x}_{t},y\right)-\bm{\epsilon}_{\bm{\theta}}\left(\bm{x}_{t}\right)\right)$
(8)

Most works of training data synthesis align the conditioned distribution via
text-prompt engineering with class-level description (He et al., 2022),
instance-level description (Lei et al., 2023), and lexical definition
(Sarıyıldız et al., 2023). Following (Lei et al., 2023), we incorporate the
class name with the BLIP2 (Li et al., 2023) caption of each instance as the
text prompt. Moreover, while text condition offers certain adaptability, it
ignores intrinsic visual information, including both low-level ones, e.g.,
exposure and saturation, and high-level ones, e.g. co-occurrence of objects
and scenes. To address this, we adopt a more direct prompting strategy by
conditioning on image features. In particular, we extract image features
encoded with the CLIP (Radford et al., 2021) image encoder, compute the mean
of image embeddings for randomly sampled images of a class, and estimate the
intra-class feature distribution, i.e. the mean feature. This is then
concatenated with the text embeddings for jointly finetuning the diffusion
model using LoRA (Hu et al., 2021), thus injecting the extra conditional
control into the cross-attention layer of the denoising UNet (Ronneberger et
al., 2015). The resultant multi-modal condition (embedding) takes form in
"photo of [classname], [Image Caption], [Intra-class Visual Guidance]".

###  3.3 Latent Prior Initialization

Another key aspect in the sampling process of Latent Diffusion Model is the
latent prior $p_{\mathbf{\bm{\theta}}}\left(\bm{x}_{T}\right)$ which acts as
the informative guide for the reverse diffusion process, governed by Langevin
dynamics (Song & Ermon, 2019; Parisi, 1981), as shown in Eq. 9:

$\bm{x}_{t-1}=\bm{x}_{t}+\sqrt{2\alpha_{t}}\mathbf{z}_{t}-\alpha_{t}\nabla_{\bm{x}_{t}}\log
p_{\bm{\theta}}(\bm{x}_{t}|\bm{x}_{T}),\mathbf{z}_{t}\sim\mathcal{N}\left(\mathbf{0},\mathbf{I}\right),$
(9)

where $p{{}_{\bm{\theta}}}(\bm{x}_{t}|\bm{x}_{T})$ represents the conditional
distribution of $\bm{x}_{t}$ given the latent prior $\bm{x}_{T}$, and the
corresponding score function $\nabla_{\bm{x}_{t}}\log
p_{\bm{\theta}}(\bm{x}_{t}|\bm{x}_{T})$ guides the Langevin dynamics towards
regions of higher probability in the data distribution, ensuring the synthetic
samples align closely with the target distribution. While diffusion models
often employ a Gaussian distribution as the initial prior $\bm{x}_{T}\sim
q\left(\bm{x}_{T}\right):=\mathcal{N}\left(\bm{x}_{T};\mathbf{0},\mathbf{I}\right)$,
recent studies using latent inversion (Zhou et al., 2023; Lian et al., 2023)
and learning-based prior rendering (Liao et al., 2023) have highlighted the
advantages of informative non-Gaussian latent priors with better sampling
speed and synthesis quality. However, obtaining such an informative prior can
be expensive due to intensive computation or need for external architecture.
Instead, following (Meng et al., 2021), we leverage the VAE encoder to obtain
the latent code of specific real samples, which is extremely cheap. This also
provides an informative latent prior initialization closely aligned with the
target distribution, resulting in better synthetic samples.

##  4 Experiments

#### Settings.

We empirically evaluate the proposed distribution-matching framework and
assess the utility of synthetic training data. We explore the application of
synthetic data in various supervised image classification scenarios: (1)
Replacing the real training set (Sec. 4.1), (2) Augmenting the real training
set (Sec. 4.2), and (3) Evaluating the scaling law of synthetic training data
(Sec. 4.3). We aim to validate two main factors identified in Eq. 4: better
alignment between target and synthetic data (in scenarios 1 and 2) and the
advantages of larger training set cardinality (in scenario 3). Then, we
further explore the (4) Out-of-distribution generalization (Sec. 4.4) and (5)
Privacy-preservation (Sec. 4.5) of synthetic training data. For all
experiments, we finetune Stable Diffusion 1.5 (SDv1.5) (Rombach et al., 2022)
with LoRA.

#### Datasets.

We conduct benchmark experiments with ResNet50 (He et al., 2016) across three
ImageNet datasets: ImageNette (IN-10) (Howard, 2019), ImageNet100 (IN-100)
(Tian et al., 2020), and ImageNet1K (IN-1K) (Deng et al., 2009). Beyond these,
we also experiment with several fine-grained image classification datasets,
CUB (Wah et al., 2011), Cars (Krause et al., 2013), PET (Parkhi et al., 2012),
and satellite images, EuroSAT (Helber et al., 2018).

More details of dataset specifications (Apx. D), data synthesis (Apx. E),
model training (Apx. F), and experiment setting (Apx. G) are provided in the
Appendix.

###  4.1 Image Classification With Synthetic Data Only

We begin by assessing the informativeness of synthetic training data as a
replacement for real training data on image classification tasks. To replace
the real training set, we synthesize training samples with the same number as
the real training set in each dataset.

As shown in Tab. 1, in comparison to real data, our synthetic data reduces the
performance gap to less than 3% for small-scale dataset IN-10. In the context
of the more challenging large-scale dataset, IN-1K, the performance
differential remains under 10%. Regarding training data synthesis methods, our
method outperforms all state-of-the-art techniques across all benchmarks. It
is crucial to highlight that our synthetic data exhibits improvements of 16.8%
and 28.0% in IN-1K top-1 accuracy, compared to CiP (Lei et al., 2023) and
FakeIt (Sarıyıldız et al., 2023) respectively, given the same generative model
backbone.

Table 1:  Synthetic Image Classification Performance Top-1 accuracies of ResNet50 reported on seven datasets. The upper part indicates training with synthetic data only, and the lower part indicates joint training with combined real and synthetic data.  | Model | IN-10 | IN-100 | IN-1K | CUB | Cars | PET | EuroSAT
---|---|---|---|---|---|---|---|---
Training without real data |  |  |  |  |  |  |  |
BigGAN (Brock et al., 2018) | BigGAN | - | - | $42.7$ |  |  |  |
VQ-VAE-2 (Razavi et al., 2019) | VQ-VAE | - | - | $54.8$ |  |  |  |
CDM (Ho et al., 2022) | CDM | - | - | $63.0$ |  |  |  |
FakeIt (Sarıyıldız et al., 2023) | SDv1.5 | - | - | $42.9$ | $33.7$ | $47.1$ | $75.9$ | $94.0$
Imagen (Azizi et al., 2023) | Imagen | - | - | $69.2$ |  |  |  |
CiP (Lei et al., 2023) | SDv1.5 | $79.4$ | $62.4$ | $54.1$ |  |  |  |
OURS | SDv1.5 | $90.5$ | $80.0$ | $70.9$ | $64.3$ | $81.8$ | $89.2$ | $94.6$
$\Delta$ with the previous state-of-the-art |  | +$10.1$ | +$17.6$ | +$1.7$ | +$30.6$ | +$34.7$ | +$13.3$ | +$0.6$
Training with real data |  |  |  |  |  |  |  |
Baseline real data only | - | $93.0$ | $86.3$ | $79.6$ | $81.5$ | $89.5$ | $93.2$ | $97.6$
OURS + real data | SDv1.5 | $95.1$ | $88.2$ | $79.9$ | $83.5$ | $90.3$ | $94.0$ | $98.9$
$\Delta$ with the real data |  | +$2.1$ | +$1.9$ | +$0.3$ | +$2.0$ | +$0.8$ | +$0.8$ | +$1.3$

In fine-grained classification tasks, our method demonstrates a more
significant improvement compared to the state-of-the-art method (FakeIt),
which can be attributed to the need for a more aligned decision boundary.

#### Ablation Study.

We next perform a more comprehensive ablation study to evaluate the efficacy
of our proposed enhancements: distribution matching objective (Sec. 3.1),
conditioned visual guidance (Sec. 3.2), and latent prior initialization (Sec.
3.3). Due to computational cost, we conducted the ablation study on the IN-10
and IN-100 datasets. As illustrated in Tab. 2, every proposed module
contributes to remarkable performance improvement. The combination of three
modules achieves the best results which outperform the baseline by 10.7% and
14.0% on IN-10 and IN-100 respectively.

Table 2: Ablation study on proposed improvements with IN-10 and IN-100 Top-1 accuracy. Latent Prior | Visual Guidance | Distribution Matching | Finetune | ImageNette | ImageNet100
---|---|---|---|---|---
Sec. 3.3 | Sec. 3.2 | Sec. 3.1 |  |  |
|  |  |  | 79.8 | 66.0
|  |  | ✓ | 80.8 | 73.3
| ✓ |  | ✓ | 82.3 | 74.0
|  | ✓ | ✓ | 81.2 | 73.8
| ✓ | ✓ | ✓ | 82.9 | 75.1
✓ |  |  |  | 88.0 | 76.3
✓ | ✓ |  | ✓ | 89.5 | 79.3
✓ |  | ✓ | ✓ | 88.9 | 79.8
✓ | ✓ | ✓ | ✓ | 90.5 | 80.0

###  4.2 Augmenting Real Data with Synthetic Data

We study whether the synthetic data can serve as dataset augmentation. We
compare the models trained with only real data and those trained with both
real and synthetic data. As shown in lower part of Tab. 1, we observe
improvements across all benchmarks when combining the synthetic and real data.
Especially, our synthetic data boosts the performances by 2.1% and 1.9% on
IN-10 and IN-100 datasets respectively. This validates that our synthetic data
align well with the real data distribution.

###  4.3 Scaling Up Synthetic Training Data

(a) Imagenette

(b) Imagenet100

Figure 2: Effect of Scaling Up Synthetic Dataset Top-1 image classification performance with synthetic data only (indicated by blue solid curve) increases with synthetic dataset size, eventually outperforming real data (indicated by red dash line). The horizontal axis represents the amount of synthetic data used as multiples of the original real dataset size. Table 3: Scaling-up synthetic ImageNet-1K. Synthetic vs. real data size |  Real data? | Top-1 | Top-5
---|---|---|---
$\times 1$ (1.3M) |  | $70.9$ | $89.9$
$\times 2$ (2.6M) |  | $72.9$ | $91.1$
$\times 5$ (6.4M) |  | $74.5$ | $92.1$
$\times 10$ (13M) |  | $76.0$ | $93.1$
$\times 1$ (1.3M) | ✓ | $79.6$ | $94.6$

Besides data distribution alignment, training set cardinality is another key
factor influencing the utility of synthetic data identified in Eq. 4.
Fortunately, it is easy to synthesize more data for deep generative model. In
the experiment, we scale up the synthetic dataset and train the image
classifiers using increasing amounts of synthetic data, ranging from 1$\times$
to 10$\times$ the size of the original real dataset. As shown in Fig. 2,
solely scaling up the synthetic dataset enables the image classification
performance to surpass that of models trained on real data, even without
trained on it. For the small-scale dataset IN-10, the threshold scale of
synthetic data to outperform the real data is only 2.5$\times$ the size of the
real data. While, the challenge is greater for larger-scale dataset IN-100. To
surpass the real data efficacy, synthetic data with 6.5$\times$ size is needed
on IN-100. As shown in Tab. 3, we also scale up synthetic training images for
the large dataset IN-1K, which has 1.3M real samples. The results show that
the trend of increasing image classification accuracy along with the
increasing synthetic data, still holds, where we achieve $72.82\%$ and
$76.00\%$ with 2 $\times$ and 10 $\times$ ImageNet-1K size, respectively. By
scaling up synthetic data, we reduce the gap between real and synthetic data
down to $3.6\%$ in top-1 accuracy and more encouraging $1.5\%$ in top-5
accuracy.

###  4.4 Generalization to Out-of-Distribution Data

We also investigate the Out-of-distribution (OOD) generalization performance
on four OOD variants of ImageNet: (1) ImageNet-v2 (IN-v2) (Recht et al., 2019)
(2) ImageNet-Sketch (IN-Sketch) (Wang et al., 2019) (3) ImageNet-R (IN-R)
(Hendrycks et al., 2021a) (4) ImageNet-A (IN-A) (Hendrycks et al., 2021b). We
test the ResNet50 (He et al., 2016) trained with in-distribution real data
(i.e. ImageNet-1K training set) or synthetic data (i.e. the generative model
is only tuned on ImageNet-1K training set like above experiments) on the OOD
test sets.

Table 4: OOD Generalization Performance. $\times k$ indicates training solely
with the synthetic dataset of scale $k$ times that of real dataset. Top-1 and
5 classification accuracies are reported.

|  ImageNet-v2 (Recht et al., 2019) |  ImageNet-Sketch (Wang et al., 2019) |  ImageNet-R (Hendrycks et al., 2021a) |  ImageNet-A (Hendrycks et al., 2021b)
---|---|---|---|---
Top-1 | Top-5 | Top-1 | Top-5 | Top-1 | Top-5 | Top-1 | Top-5
Training without real data |  |  |  |  |  |  |  |
FakeIt Sarıyıldız et al. (2023) | $43.0$ | $70.3$ | $16.6$ | $35.2$ | $26.3$ | $45.3$ | $3.6$ | $15.1$
CiP Lei et al. (2023) | $53.8$ | $80.5$ | $18.5$ | $35.5$ | $33.6$ | $51.1$ | $5.2$ | $21.7$
OURS | $67.2$ | $88.0$ | $21.9$ | $38.2$ | $35.4$ | $51.9$ | $4.5$ | $23.5$
OURS $\times 2$ | $69.5$ | $89.2$ | $25.2$ | $43.0$ | $36.1$ | $52.5$ | $6.8$ | $29.2$
OURS $\times 5$ | $71.1$ | $90.6$ | $27.8$ | $46.2$ | $39.7$ | $56.3$ | $9.4$ | $34.9$
Training with real data |  |  |  |  |  |  |  |
Baseline real data only | $74.7$ | $92.2$ | $28.1$ | $45.8$ | $39.4$ | $54.1$ | $8.1$ | $34.7$
OURS + real data | $75.7$ | $92.7$ | $29.0$ | $46.8$ | $40.5$ | $55.9$ | $9.0$ | $36.4$

As illustrated in Tab. 4, we observe that when training with $1\times$
synthetic data only, our method achieves the best generalization performance
across three out of four benchmarks, outperforming previous synthesis
strategies. When jointly training with real data, our synthetic data further
boosts the OOD generalization performance of real data. More importantly, when
we scale up the synthetic data, its OOD generalization performance exceeds
that of real data, e.g. on ImageNet-R and ImageNet-A, even before achieving
comparable performance on the in-distribution test set. This further
highlights the promising utility of synthetic training data in enhancing OOD
generalization.

###  4.5 Privacy Analysis

Synthetic training data is a promising solution to privacy-preserving
learning. In this subsection, we examine the privacy-preserving nature of our
synthetic data from the perspectives of both privacy attacks and visual
similarity. More experiment details are provided in Sec. G.4.

#### Membership Inference Attack.

We implement the membership inference attack that enables an adversary to
query a trained model and determine whether a specific example is part of the
model’s training set. Specifically, we follow the state-of-the-art Likelihood
Ratio Attack (LiRA) Carlini et al. (2022) and report the MIA results for two
training approaches: the privacy training dataset and synthetic data, in the
low false-positive rate regime. To account for training duration, we conduct
10 sampling iterations. Concretely, we initially divide the privacy data in
IN-10 into two halves: member data and non-member data. We only employ LoRA to
finetune the diffusion model on the member data, then generate synthetic data
of equal size. Subsequently, we train a ResNet50 on synthetic data. As
depicted in Fig. 3, regardless of online attack, offline attack, or fixed
variance scenarios, the model trained on synthetic data demonstrates superior
defense against MIA.

Figure 3: Membership Inference Attack (MIA) Performance with LiRA.  LiRA
achieves a TPR of 0.001% at a low FPR of 0.1% when applied to synthetic data,
while the results for private data is 0.01%, which indicates training with
synthetic data is much more privacy-preserving.

#### Visual Similarity.

We follow the previous works (Zhang et al., 2023; Somepalli et al., 2023) and
employ the Self-Supervised Content Duplication (SSCD) (Pizzi et al., 2022)
method for content plagiarism detection. SSCD is a self-supervised approach
based on SimCLR (Chen et al., 2020), utilizing InfoNCE loss (Oord et al.,
2018), entropy regularization of latent space representation, and various data
augmentation techniques. We use the ResNeXt101 model trained on the DISC
dataset (Douze et al., 2021) For a given query image and all reference images,
we infer through the detection model to obtain a 1024-dimensional feature
vector representation for each image then normalize each vector using L2
normalization.

Figure 4: Visualization on synthetic and retrieved real data with SSCD. The
synthesized data does not exhibit evident copying or memorization.

By computing the inner product between the query feature and each reference
feature, we obtain the similarity between them. Subsequently, we select the
the top five reference features with the highest similarity to the query
feature, in order to retrieve the corresponding images. Then, we rank the
similarity and present the top 3 images with their corresponding real data in
Fig. 4. From the figure, it can be observed that there are no apparent issues
of copy or memorization Carlini et al. (2023). At least visually, these
synthesized images should provide sufficient privacy protection.

##  5 Conclusion

In this work, we propose a principled theoretical framework for training data
synthesis from a distribution-matching perspective. Based on this, we
empirically push the limit of synthetic training data by advancing the state-
of-the-art performances over diverse image classification benchmarks. We also
demonstrate promising benefits of improving the OOD generalization and privacy
preservation performances by training models with our synthetic data.

## References

  * Aronszajn (1950) Nachman Aronszajn.  Theory of reproducing kernels.  _Transactions of the American mathematical society_ , 68(3):337–404, 1950.
  * Azizi et al. (2023) Shekoofeh Azizi, Simon Kornblith, Chitwan Saharia, Mohammad Norouzi, and David J Fleet.  Synthetic data from diffusion models improves imagenet classification.  _arXiv preprint arXiv:2304.08466_ , 2023.
  * Bansal & Grover (2023) Hritik Bansal and Aditya Grover.  Leaving reality to imagination: Robust classification via generated datasets.  _arXiv preprint arXiv:2302.02503_ , 2023.
  * Besnier et al. (2020) Victor Besnier, Himalaya Jain, Andrei Bursuc, Matthieu Cord, and Patrick Pérez.  This dataset does not exist: training models from generated images.  In _ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , pp. 1–5. IEEE, 2020.
  * Bishop (2006) Christopher M. Bishop.  _Pattern Recognition and Machine Learning_.  Springer, New York, NY, 2006.
  * Brock et al. (2018) Andrew Brock, Jeff Donahue, and Karen Simonyan.  Large scale gan training for high fidelity natural image synthesis.  _arXiv preprint arXiv:1809.11096_ , 2018.
  * Carlini et al. (2022) Nicholas Carlini, Steve Chien, Milad Nasr, Shuang Song, Andreas Terzis, and Florian Tramer.  Membership inference attacks from first principles.  In _2022 IEEE Symposium on Security and Privacy (SP)_ , pp. 1897–1914. IEEE, 2022.
  * Carlini et al. (2023) Nicolas Carlini, Jamie Hayes, Milad Nasr, Matthew Jagielski, Vikash Sehwag, Florian Tramer, Borja Balle, Daphne Ippolito, and Eric Wallace.  Extracting training data from diffusion models.  In _32nd USENIX Security Symposium (USENIX Security 23)_ , pp. 5253–5270, 2023.
  * Cazenavette et al. (2022) George Cazenavette, Tongzhou Wang, Antonio Torralba, Alexei A. Efros, and Jun-Yan Zhu.  Dataset distillation by matching training trajectories.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2022.
  * Chen et al. (2020) Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton.  A simple framework for contrastive learning of visual representations.  In _International conference on machine learning_ , pp. 1597–1607. PMLR, 2020.
  * Cunningham et al. (2008) Pádraig Cunningham, Matthieu Cord, and Sarah Jane Delany.  _Supervised Learning_ , pp. 21–49.  Springer Berlin Heidelberg, Berlin, Heidelberg, 2008.  ISBN 978-3-540-75171-7.  doi: 10.1007/978-3-540-75171-7˙2.
  * Deng et al. (2009) Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei.  Imagenet: A large-scale hierarchical image database.  In _2009 IEEE Conference on Computer Vision and Pattern Recognition_ , pp. 248–255, 2009.  doi: 10.1109/CVPR.2009.5206848.
  * Dhariwal & Nichol (2021) Prafulla Dhariwal and Alexander Nichol.  Diffusion models beat gans on image synthesis.  _Advances in neural information processing systems_ , 34:8780–8794, 2021.
  * Douze et al. (2021) Matthijs Douze, Giorgos Tolias, Ed Pizzi, Zoë Papakipos, Lowik Chanussot, Filip Radenovic, Tomas Jenicek, Maxim Maximov, Laura Leal-Taixé, Ismail Elezi, et al.  The 2021 image similarity dataset and challenge.  _arXiv preprint arXiv:2106.09672_ , 2021.
  * Goodfellow et al. (2014) Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio.  Generative adversarial nets.  _Advances in neural information processing systems_ , 27, 2014.
  * Gretton et al. (2008) Arthur Gretton, Karsten Borgwardt, Malte J Rasch, Bernhard Scholkopf, and Alexander J Smola.  A kernel method for the two-sample problem.  _arXiv preprint arXiv:0805.2368_ , 2008.
  * He et al. (2016) Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pp. 770–778, 2016.
  * He et al. (2022) Ruifei He, Shuyang Sun, Xin Yu, Chuhui Xue, Wenqing Zhang, Philip Torr, Song Bai, and Xiaojuan Qi.  Is synthetic data from generative models ready for image recognition?  _arXiv preprint arXiv:2210.07574_ , 2022.
  * Helber et al. (2018) Patrick Helber, Benjamin Bischke, Andreas Dengel, and Damian Borth.  Introducing eurosat: A novel dataset and deep learning benchmark for land use and land cover classification.  In _IGARSS 2018-2018 IEEE International Geoscience and Remote Sensing Symposium_ , pp. 204–207. IEEE, 2018.
  * Hendrycks et al. (2021a) Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang, Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, Dawn Song, Jacob Steinhardt, and Justin Gilmer.  The many faces of robustness: A critical analysis of out-of-distribution generalization.  _ICCV_ , 2021a.
  * Hendrycks et al. (2021b) Dan Hendrycks, Kevin Zhao, Steven Basart, Jacob Steinhardt, and Dawn Song.  Natural adversarial examples.  _CVPR_ , 2021b.
  * Hilbert (1904) David Von Hilbert.  Grundzüge einer allgemeinen theorie der linearen integralgleichungen, von david hilbert.  1904\.
  * Ho & Salimans (2022) Jonathan Ho and Tim Salimans.  Classifier-free diffusion guidance.  _arXiv preprint arXiv:2207.12598_ , 2022.
  * Ho et al. (2020) Jonathan Ho, Ajay Jain, and Pieter Abbeel.  Denoising diffusion probabilistic models.  _Advances in neural information processing systems_ , 33:6840–6851, 2020.
  * Ho et al. (2022) Jonathan Ho, Chitwan Saharia, William Chan, David J Fleet, Mohammad Norouzi, and Tim Salimans.  Cascaded diffusion models for high fidelity image generation.  _The Journal of Machine Learning Research_ , 23(1):2249–2281, 2022.
  * Hoeffding (1994) Wassily Hoeffding.  Probability inequalities for sums of bounded random variables.  _The collected works of Wassily Hoeffding_ , pp. 409–426, 1994\.
  * Howard (2019) Jeremy Howard.  Imagewang, 2019.  URL https://github.com/fastai/imagenette/.
  * Hu et al. (2021) Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen.  Lora: Low-rank adaptation of large language models.  _arXiv preprint arXiv:2106.09685_ , 2021.
  * Kingma et al. (2021) Diederik Kingma, Tim Salimans, Ben Poole, and Jonathan Ho.  Variational diffusion models.  _Advances in neural information processing systems_ , 34:21696–21707, 2021.
  * Krause et al. (2013) Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei.  3d object representations for fine-grained categorization.  In _Proceedings of the IEEE International Conference on Computer Vision (ICCV) Workshops_ , June 2013.
  * Lei et al. (2023) Shiye Lei, Hao Chen, Sen Zhang, Bo Zhao, and Dacheng Tao.  Image captions are natural prompts for text-to-image models.  _arXiv preprint arXiv:2307.08526_ , 2023.
  * Li et al. (2022) Daiqing Li, Huan Ling, Seung Wook Kim, Karsten Kreis, Sanja Fidler, and Antonio Torralba.  Bigdatasetgan: Synthesizing imagenet with pixel-wise annotations.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pp. 21330–21340, 2022.
  * Li et al. (2023) Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi.  Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models.  _arXiv preprint arXiv:2301.12597_ , 2023.
  * Lian et al. (2023) Long Lian, Boyi Li, Adam Yala, and Trevor Darrell.  Llm-grounded diffusion: Enhancing prompt understanding of text-to-image diffusion models with large language models.  _arXiv preprint arXiv:2305.13655_ , 2023.
  * Liao et al. (2023) Ting-Hsuan Liao, Songwei Ge, Yiran Xu, Yao-Chih Lee, Badour AlBahar, and Jia-Bin Huang.  Text-driven visual synthesis with latent diffusion prior.  _arXiv preprint arXiv:2302.08510_ , 2023.
  * Lin et al. (2014) Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick.  Microsoft coco: Common objects in context.  In _Computer Vision–ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13_ , pp. 740–755. Springer, 2014.
  * Meng et al. (2021) Chenlin Meng, Yutong He, Yang Song, Jiaming Song, Jiajun Wu, Jun-Yan Zhu, and Stefano Ermon.  Sdedit: Guided image synthesis and editing with stochastic differential equations.  _arXiv preprint arXiv:2108.01073_ , 2021.
  * Nichol & Dhariwal (2021) Alexander Quinn Nichol and Prafulla Dhariwal.  Improved denoising diffusion probabilistic models.  In _International Conference on Machine Learning_ , pp. 8162–8171. PMLR, 2021.
  * Oord et al. (2018) Aaron van den Oord, Yazhe Li, and Oriol Vinyals.  Representation learning with contrastive predictive coding.  _arXiv preprint arXiv:1807.03748_ , 2018.
  * Parisi (1981) G. Parisi.  Correlation functions and computer simulations.  _Nuclear Physics B_ , 180(3):378–384, 1981.  ISSN 0550-3213.  doi: https://doi.org/10.1016/0550-3213(81)90056-0.
  * Parkhi et al. (2012) Omkar M. Parkhi, Andrea Vedaldi, Andrew Zisserman, and C. V. Jawahar.  Cats and dogs.  In _IEEE Conference on Computer Vision and Pattern Recognition_ , 2012\.
  * Pizzi et al. (2022) Ed Pizzi, Sreya Dutta Roy, Sugosh Nagavara Ravindra, Priya Goyal, and Matthijs Douze.  A self-supervised descriptor for image copy detection.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pp. 14532–14542, 2022.
  * Radford et al. (2021) Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al.  Learning transferable visual models from natural language supervision.  In _International conference on machine learning_ , pp. 8748–8763. PMLR, 2021.
  * Razavi et al. (2019) Ali Razavi, Aaron Van den Oord, and Oriol Vinyals.  Generating diverse high-fidelity images with vq-vae-2.  _Advances in neural information processing systems_ , 32, 2019.
  * Recht et al. (2019) Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar.  Do imagenet classifiers generalize to imagenet?  In _International conference on machine learning_ , pp. 5389–5400. PMLR, 2019.
  * Rombach et al. (2022) Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer.  High-resolution image synthesis with latent diffusion models.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pp. 10684–10695, 2022.
  * Ronneberger et al. (2015) Olaf Ronneberger, Philipp Fischer, and Thomas Brox.  U-net: Convolutional networks for biomedical image segmentation.  In _Medical Image Computing and Computer-Assisted Intervention–MICCAI 2015: 18th International Conference, Munich, Germany, October 5-9, 2015, Proceedings, Part III 18_ , pp. 234–241. Springer, 2015.
  * Saharia et al. (2022) Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al.  Photorealistic text-to-image diffusion models with deep language understanding.  _Advances in Neural Information Processing Systems_ , 35:36479–36494, 2022.
  * Sarıyıldız et al. (2023) Mert Bülent Sarıyıldız, Karteek Alahari, Diane Larlus, and Yannis Kalantidis.  Fake it till you make it: Learning transferable representations from synthetic imagenet clones.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pp. 8011–8021, June 2023.
  * Sohl-Dickstein et al. (2015) Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli.  Deep unsupervised learning using nonequilibrium thermodynamics.  In _International conference on machine learning_ , pp. 2256–2265. PMLR, 2015.
  * Somepalli et al. (2023) Gowthami Somepalli, Vasu Singla, Micah Goldblum, Jonas Geiping, and Tom Goldstein.  Diffusion art or digital forgery? investigating data replication in diffusion models.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pp. 6048–6058, 2023.
  * Song & Ermon (2019) Yang Song and Stefano Ermon.  Generative modeling by estimating gradients of the data distribution.  _Advances in neural information processing systems_ , 32, 2019.
  * Sutskever (2023) Ilya Sutskever.  An observation on generalization, 2023.
  * Tian et al. (2020) Yonglong Tian, Dilip Krishnan, and Phillip Isola.  Contrastive multiview coding.  In _Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XI 16_ , pp. 776–794. Springer, 2020.
  * Tian et al. (2023) Yonglong Tian, Lijie Fan, Phillip Isola, Huiwen Chang, and Dilip Krishnan.  Stablerep: Synthetic images from text-to-image models make strong visual representation learners.  _arXiv preprint arXiv:2306.00984_ , 2023.
  * Vendrow et al. (2023) Joshua Vendrow, Saachi Jain, Logan Engstrom, and Aleksander Madry.  Dataset interfaces: Diagnosing model failures using controllable counterfactual generation.  _arXiv preprint arXiv:2302.07865_ , 2023.
  * Wah et al. (2011) C. Wah, S. Branson, P. Welinder, P. Perona, and S. Belongie.  Caltech-ucsd birds-200-2011 (cub-200-2011).  Technical Report CNS-TR-2011-001, California Institute of Technology, 2011\.
  * Wang et al. (2019) Haohan Wang, Songwei Ge, Zachary Lipton, and Eric P Xing.  Learning robust global representations by penalizing local predictive power.  In _Advances in Neural Information Processing Systems_ , pp. 10506–10518, 2019.
  * Wang et al. (2018) Tongzhou Wang, Jun-Yan Zhu, Antonio Torralba, and Alexei A Efros.  Dataset distillation.  _arXiv preprint arXiv:1811.10959_ , 2018.
  * Wightman et al. (2021) Ross Wightman, Hugo Touvron, and Hervé Jégou.  Resnet strikes back: An improved training procedure in timm.  _arXiv preprint arXiv:2110.00476_ , 2021.
  * Yang et al. (2022) Ling Yang, Zhilong Zhang, Yang Song, Shenda Hong, Runsheng Xu, Yue Zhao, Yingxia Shao, Wentao Zhang, Bin Cui, and Ming-Hsuan Yang.  Diffusion models: A comprehensive survey of methods and applications.  _arXiv preprint arXiv:2209.00796_ , 2022.
  * Yuan et al. (2022) Jianhao Yuan, Francesco Pinto, Adam Davies, Aarushi Gupta, and Philip Torr.  Not just pretty pictures: Text-to-image generators enable interpretable interventions for robust representations.  _arXiv preprint arXiv:2212.11237_ , 2022.
  * Zhang et al. (2023) Jie Zhang, Xiaohua Qi, and Bo Zhao.  Federated generative learning with foundation models.  _arXiv preprint arXiv:2306.16064_ , 2023.
  * Zhang et al. (2021) Yuxuan Zhang, Huan Ling, Jun Gao, Kangxue Yin, Jean-Francois Lafleche, Adela Barriuso, Antonio Torralba, and Sanja Fidler.  Datasetgan: Efficient labeled data factory with minimal human effort.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pp. 10145–10155, 2021.
  * Zhao & Bilen (2022) Bo Zhao and Hakan Bilen.  Synthesizing informative training samples with GAN.  _NeurIPS 2022 Workshops SyntheticData4ML_ , 2022.
  * Zhao & Bilen (2023) Bo Zhao and Hakan Bilen.  Dataset condensation with distribution matching.  _IEEE/CVF Winter Conference on Applications of Computer Vision (WACV)_ , 2023.
  * Zhao et al. (2021) Bo Zhao, Konda Reddy Mopuri, and Hakan Bilen.  Dataset condensation with gradient matching.  In _International Conference on Learning Representations_ , 2021.  URL https://openreview.net/forum?id=mSAKhLYLSsl.
  * Zhao et al. (2023) Ganlong Zhao, Guanbin Li, Yipeng Qin, and Yizhou Yu.  Improved distribution matching for dataset condensation.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pp. 7856–7865, 2023.
  * Zhou et al. (2023) Yongchao Zhou, Hshmat Sahak, and Jimmy Ba.  Training on thin air: Improve image classification with generated data.  _arXiv preprint arXiv:2305.15316_ , 2023.

##  Appendix A Minimization of KL-Divergence Objective in Diffusion
Probabilistic Model

Consider the training objective of the diffusion model, minimization of KL-
divergence between the joint distributions of forward and backward process $q$
and $p_{\bm{\theta}}$ over all time steps:

$\operatorname{KL}\left(q\left(\bm{x}_{0},\bm{x}_{1},\cdots,\bm{x}_{T}\right)\|p_{\bm{\theta}}\left(\bm{x}_{0},\bm{x}_{1},\cdots,\bm{x}_{T}\right)\right)$
(10)

By definition of KL-divergence and the property of Markovian transition, we
can further decompose the expression in time (Yang et al., 2022):

$\operatorname{KL}\left(q\|p_{\bm{\theta}}\right)=\mathbb{E}_{q}\left[-\log
p\left(\bm{x}_{T}\right)-\sum_{t=1}^{T}\log\frac{p_{\bm{\theta}}\left(\bm{x}_{t-1}\mid\bm{x}_{t}\right)}{q\left(\bm{x}_{t}\mid\bm{x}_{t-1}\right)}\right]+\mathrm{const}$
(11)

Using Jensen’s inequality, we can derive:

$\operatorname{KL}\left(q\left(\bm{x}_{0},\bm{x}_{1},\cdots,\bm{x}_{T}\right)\|p_{\bm{\theta}}\left(\bm{x}_{0},\bm{x}_{1},\cdots,\bm{x}_{T}\right)\right)\geq-\log
p_{\bm{\theta}}\left(\bm{x}_{0}\right)+\mathrm{const}$ (12)

The objective in training is to maximize the variational lower bound (VLB) of
the log-likelihood of the data $\bm{x}_{0}$, which is equivalent to minimizing
the negative VLB as in Eq. 1.

##  Appendix B Training Data Synthesis for Supervised Learning

Consider the following formulation of supervised learning (Sutskever, 2023;
Cunningham et al., 2008):

$\operatorname{Pr}_{S\sim
D^{|S|}}\left[\operatorname{Test}_{D}(f)-\operatorname{Train}_{S}(f)\leq\sqrt{\frac{\log|\mathcal{F}|+\log
1/\delta}{|S|}}\forall f\in\mathcal{F}\right]>1-\delta$ (13)

where $D$ denotes the distribution of the whole target dataset and $S$ is a
training subset of $D$. $\mathcal{F}$ represents all potential functional
space, which can be represented by a neural network with a fixed structure,
where each unique combination of parameter values corresponds to a unique $f$.
The training and test errors are defined as
$\operatorname{Train}_{S}(f)=\frac{1}{|S|}\sum_{\left(X_{i},Y_{i}\right)\in
S}L\left(f\left(X_{i}\right),Y_{i}\right)$ and
$\operatorname{Test}_{D}(f)=\frac{1}{|D|}\sum_{\left(X_{i},Y_{i}\right)\in
D}L\left(f\left(X_{i}\right),Y_{i}\right)$, respectively. Mathematically, it
is evident that when the training loss is minimal and the training set
cardinality significantly exceeds the model parameters, the test loss will
also be notably low.

We now prove this proposition. Consider the formulation of test error:

$\displaystyle\operatorname{Pr}_{S\sim
D^{|S|}}\left[\operatorname{Test}_{D}(f)-\operatorname{Train}_{S}(f)\geq
t\text{ for some }f\in\mathcal{F}\right]$
$\displaystyle\leq\sum_{f\in\mathcal{F}}\operatorname{Pr}_{S\sim
D^{|S|}}\left[\operatorname{Test}_{D}(f)-\operatorname{Train}_{S}(f)\geq
t\right]$ (14) $\displaystyle=\sum_{f\in\mathcal{F}}\operatorname{Pr}_{\sim
D^{|S|}}\left[\frac{1}{|S|}\sum\left(L_{i}-\mathbb{E}[L]\right)\geq t\right]$
$\displaystyle\leq|\mathcal{F}|\exp\left(-2|S|t^{2}\right)$

The first step is a straightforward extension of the original formulation. The
second step employs Hoeffding’s inequality (Hoeffding, 1994). Given a series
of independent random variables $X_{1},X_{2},\ldots,X_{n}$ bounded by
$a_{i}\leq X_{i}\leq b_{i}$, Hoeffding’s inequality states:

$\operatorname{Pr}\left[\sum_{i=1}^{n}X_{i}-\mathbb{E}\left[\sum_{i=1}^{n}X_{i}\right]\geq
t\right]\leq\exp\left(-\frac{2t^{2}}{\sum_{i=1}^{n}(b_{i}-a_{i})^{2}}\right)$
(15)

It bounds the probability that the sum of independent bounded random variables
deviates from its expected value beyond a set limit. Consider each
$L_{i}-\mathbb{E}[L]$ as a bounded random variable, capturing the deviation of
a specific data point’s loss from its expected value. By assuming a bounded
loss function $L_{i}$, which is valid as we can project any unbounded loss
function monotonically to a bounded range of $[0,1]$ through sigmoid function,
we obtain the upper bound on the probability of this disparity surpassing
threshold $t$, culminating in the expression
$|\mathcal{F}|\exp\left(-2|S|t^{2}\right)$. This implies the convergence of
test error to zero when the size of the training dataset goes to infinite:

$\lim_{|S|->\infty}\operatorname{Pr}\left[\sup_{f\in\mathcal{F}}\left(\operatorname{Test}_{D}(f)-\operatorname{Train}_{S}(f)\right)>\varepsilon\right]=0$
(16)

which eventually validates our statement on the implication of training data
synthesis.

##  Appendix C Maximum Mean Discrepancy (MMD) in RKHS

The Maximum Mean Discrepancy (MMD) in a Reproducing Kernel Hilbert Space
(RKHS) (Aronszajn, 1950) $\mathcal{H}$ provides a metric to measure the
difference between two probability distributions.

$\displaystyle\operatorname{MMD}[\mathcal{F},p,q]$
$\displaystyle=\sup_{\left\|\psi_{\vartheta}\right\|_{\mathcal{H}}\leq
1}\left(\mathbb{E}_{q}\left[\psi_{\vartheta}(\mathcal{T})\right]-\mathbb{E}_{p}\left[\psi_{\vartheta}(\mathcal{S})\right]\right)$
(17) $\displaystyle\operatorname{MMD}[\mathcal{F},p,q]$
$\displaystyle=\sup_{\left\|\psi_{\vartheta}\right\|_{\mathcal{H}}\leq
1}\left\langle\mu_{\mathcal{T}}-\mu_{\mathcal{S}},\psi_{\vartheta}\right\rangle_{\mathcal{H}}$
(18) $\displaystyle\operatorname{MMD}^{2}[\mathcal{F},p,q]$
$\displaystyle=\left[\sup_{\left\|\psi_{\vartheta}\right\|_{\mathcal{H}}\leq
1}\left\langle\mu_{\mathcal{T}}-\mu_{\mathcal{S}},\psi_{\vartheta}\right\rangle_{\mathcal{H}}\right]^{2}$
(19) $\displaystyle\operatorname{MMD}^{2}[\mathcal{F},p,q]$
$\displaystyle=\left\|\mu_{\mathcal{T}}-\mu_{\mathcal{S}}\right\|_{\mathcal{H}}^{2}$
(20)

In Eq. 17, the MMD between distributions $p$ and $q$ is formulated as the
supremum of the difference of expectations under these distributions,
constrained by the norm of the function $\psi_{\vartheta}$ in the RKHS. This
difference can be further expanded using the inner product in the RKHS,
leading to an expression in terms of the mean embeddings of the distributions,
as shown in Eq. 18. Squaring the MMD gives us a more interpretable metric,
which is the squared distance between the mean embeddings of the two
distributions in the RKHS, as depicted in Eq. 19 and Eq. 20. This leads to the
derivation from Eq. 5 to Eq. 6 in Sec. 3.1.

##  Appendix D Dataset Details

We adopt seven datasets in total across different settings: ImageNette (IN-10)
(Howard, 2019), ImageNet100 (IN-100) (Tian et al., 2020), and ImageNet1K
(IN-1K) (Deng et al., 2009), CUB (Wah et al., 2011), Cars (Krause et al.,
2013), PET (Parkhi et al., 2012), EuroSAT (Helber et al., 2018). The detailed
dataset statistics are specified in Tab. 5.

Table 5: Detailed Dataset Statistics.

| IN-10 | IN-100 | IN-1K | CUB | Cars | PETS | EuroSAT
---|---|---|---|---|---|---|---
Dataset Size | 14197 | 141971 | 1419712 | 11788 | 16,185 | 7349 | 27000
Training Data Size | 12811 | 128116 | 1281167 | 5994 | 8144 | 3680 | 21600
Test Data Size | 1000 | 10000 | 100000 | 5794 | 8041 | 3669 | 5400
No. Classes | 10 | 100 | 1000 | 200 | 196 | 37 | 7
Domain | Natural Image | Natural Image | Natural Image | Fine-grain Bird | Fine-grain Cars | Pet Breed | Satellite Image

##  Appendix E Training Data Synthesis Details

We use Stable Diffusion v1.5 (Rombach et al., 2022) across all benchmarks.
Unless specified (i.e. we specifically scale up synthetic data in Sec. 4.3),
we generate an equal amount of synthetic data samples as in the corresponding
real dataset for both replacing and augmenting real training data. The stable
diffusion generation parameters are specified in Tab. 6. We use text prompt
mentioned in Sec. 3.2, of the form "photo of [classname], [Image Caption],
[Intra-class Visual Guidance]" for ImageNet dataset (IN-10, IN-100, IN-1K);
for the rest four datasets, we do not use the image caption generated with
BLIP2 (Li et al., 2023). Also, all generations use a uniform set of negative
prompts "distorted, unrealistic, blurry, out of frame".

Table 6: Hyperparameters used in training data synthesis.

Model | Sampling steps | Scheduler | Guidance scale | Img_strength | Image size
---|---|---|---|---|---
Stable Diffusion v1.5 | 30 | UniPC | 2.0 | 0.75 | $512\times 512$

For fine-tuning the Stable Diffusion, we use Low-Rank Adaptation (LoRA) (Hu et
al., 2021) with the MMD augmented objective Sec. 3.1 and inject the visual
guidance Sec. 3.2. The fine-tuning hyperparameters used are specified in Tab.
7.

Table 7: Hyperparameters used in LoRA fine-tuning

Rank | Epoch | Leanring Rate | Batch Size | SNR-$\gamma$ | Image size
---|---|---|---|---|---
Stable Diffusion v1.5 | 300 | 1e-4 | 8 $\times$ 8 | 0.75 | $512\times 512$

We use the loss function mentioned in Sec. 3.1, where we augment the MMD
distribution matching loss with the simple diffusion model training objective
as in Eq. 21, we we choose $\gamma=0.05$.

$L_{overall}=L_{DM}+L_{simple}=\frac{1}{|\mathcal{N}|}\sum_{i=1}^{|\mathcal{N}|}||\left(\bm{\epsilon}_{0}-\bm{\epsilon_{\bm{\theta}}\left(\bm{x}_{t},t\right)}\right)||_{\mathcal{H}}^{2}+\gamma||\frac{1}{|\mathcal{N}|}\sum_{i=1}^{|\mathcal{N}|}\left(\bm{\epsilon}_{0}-\bm{\epsilon_{\bm{\theta}}\left(\bm{x}_{t},t\right)}\right)||_{\mathcal{H}}^{2}$
(21)

##  Appendix F Downstream Image Classification Training Details

We adopt ResNet-50 (He et al., 2016) through all benchmarks in the image
classification task. For all three ImageNet datasets, we train the classifier
model from scratch. Follow the training recipe of (Lei et al., 2023) and
(Wightman et al., 2021), the training hyperparameters are specified in Tab. 8

Table 8: Training Hyperparameters of Different Dataset.

| IN-10 | IN-100 | IN-1K | CUB | CARS | PETS | EuroSAT
---|---|---|---|---|---|---|---
Train Res $\rightarrow$ Test Res | 224 $\rightarrow$ 224 | 224 $\rightarrow$ 224 | 224 $\rightarrow$ 224 | 448 $\rightarrow$ 448 | 448 $\rightarrow$ 448 | 224 $\rightarrow$ 224 | 448 $\rightarrow$ 448
Epochs | 200 | 200 | 300 | 200 | 200 | 200 | 200
Batch size | 128 $\times$ 8 | 128 $\times$ 8 | 512 $\times$ 8 | 128 $\times$ 8 | 128 $\times$ 8 | 128 $\times$ 8 | 128 $\times$ 8
Optimizer | SGD | SGD | LAMB | SGD | SGD | SGD | SGD
LR | 0.1 | 0.1 | 5e-3 | 0.1 | 0.1 | 0.1 | 0.1
LR decay | multistep | multistep | cosine | multistep | multistep | multistep | multistep
decay rate | 0.2 | 0.2 | 0.2 | 0.2 | 0.2 | 0.2 | 0.2
decay epochs | 50/100/150 | 50/100/150 | - | 50/100/150 | 50/100/150 | 50/100/150 | 50/100/150
Weight decay | 5e-4 | 5e-4 | 0.02 | 5e-4 | 5e-4 | 5e-4 | 5e-4
Warmup epochs | - | - | 5 | - | - | - | -
Label smoothing | - | - | 0.1 | - | - | - | -
Dropout | x | x | x | x | x | x | x
Stoch. Depth | x | x | 0.05 | x | x | x | x
H. flip | - | - | ✓ | - | - | - | -
Rand Augment | x | x | 7 / 0.5 | x | x | x | x
Mixup alpha | x | x | 0.2 | x | x | x | x
Cutmix alpha | x | x | 1.0 | x | x | x | x
CE loss $\rightarrow$ BCE loss | x | x | ✓ | x | x | x | x
Mixed precision | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓

##  Appendix G Experiment Setting Details and More Results

###  G.1 Image Classification With Synthetic Data Only

We compare with the baselines of previous works on synthetic ImageNet data for
supervised image classification. All baselines synthesize equal amounts of
data as real datasets. The accuracies reported are from the original baseline
papers, except for FakeIt (Sarıyıldız et al., 2023) in CUB (Wah et al., 2011),
Cars (Krause et al., 2013), PET (Parkhi et al., 2012), and EuroSAT (Helber et
al., 2018). The number reported is based on our implementation of the method
with the base prompt "a photo/image of [classname]".

Note that there is a difference in the generation resolution among different
methods. In particular BigGAN-deep (Brock et al., 2018), VQ-VAE-2 (Razavi et
al., 2019), and CDM (Ho et al., 2022), as reported in (Azizi et al., 2023),
use image resolution of $256\times 256$. CiP (Lei et al., 2023), FakeIt
(Sarıyıldız et al., 2023), and our method uses image resolution of $512\times
512$. Imagen (Azizi et al., 2023) uses image resolution of $1024\times 1024$.
Even though we resize all samples in downstream training to $224\times 224$,
the generation resolution still has influences as reported in (Azizi et al.,
2023) potentially due to the generation details and quality. Thus, comparing
our method and non-Stable Diffusion baselines can potentially be an unfair
comparison. However, our main focus in training data synthesis is in
comparison with real data and the comparison between ours and other Stable
Diffusion-based method show our framework leads to improved synthetic data
informativeness to close the gap between real and synthetic data.

###  G.2 Augmenting Real Data With Synthetic Data

For CUB (Wah et al., 2011), Cars (Krause et al., 2013), and PET (Parkhi et
al., 2012) datasets, we use a pre-trained ResNet-50 due to the scarcity of
data to train a decent ResNet-50 classifier. For EuroSAT (Helber et al.,
2018), ImageNette (IN-10) (Howard, 2019), ImageNet100 (IN-100) (Tian et al.,
2020), and ImageNet1K (IN-1K) (Deng et al., 2009) we train the model from
scratch. We augment the real data with an equal amount of synthetic data,
which effectively raises the amount of training by two.

###  G.3 Generalization To Out-of-Distribution Data

We train the image classifier on synthetic in-distribution synthetic and real
ImageNet-1K and then test on four Out-of-Distribution test sets: (1)
ImageNet-v2 (IN-v2) (Recht et al., 2019), which is a different ImageNet test
set collected on purpose to eliminate the effect of adaptive overfitting for
original ImageNet test set; ImageNet-Sketch (IN-Sketch) (Wang et al., 2019),
which contains sketch of ImageNet classes with black and white scheme only;
ImageNet-R (IN-R) (Hendrycks et al., 2021a), which contains renditions of
ImageNet classes such as art, cartoon, graphics, etc.; and ImageNet-A (IN-A)
(Hendrycks et al., 2021b), which contain natural adversarial examples for
ResNet models trained with ImageNet-1K training set. In particular, ImageNet-A
and ImageNet-R only contain 200 classes as a subset of ImageNet-1K, thus we
only consider the output logits of the image classifier over these sub-
classes.

###  G.4 Privacy Analysis

#### Visual Appearance

For the visual appearance experiment, we use the ResNeXt101 model trained on
the DISC dataset (Douze et al., 2021) provided by the official
https://github.com/facebookresearch/sscd-copy-detection/tree/main as our
detection model.

##  Appendix H Further Discussion

###  H.1 Significant of Class-likelihood Matching in Fine-grained
Classification

In Sec. 4.1, we observe a more significant advantage of our synthetic data
over fine-grain classification tasks (i.e. CUB (Wah et al., 2011), Cars
(Krause et al., 2013)) than natural image classification (i.e ImageNet (Deng
et al., 2009)) in comparison to baseline. This is due to the requirement of
more detailed decision boundaries among classes. This highlights the
importance of the class-likelihood alignment
$q(y|\bm{x})=p_{\bm{\theta}}(y|\bm{x})$. This also indicates solely increasing
in the capacity of the generative model in synthesizing high-quality images is
insufficient to handle hard image classification without explicit alignment
between the joint distribution between image and annotation.

###  H.2 Better Out-of-Distribution Generalization Performance

In the assessment of OOD generalization performance Sec. 4.4, we find that
with scaling-up ImageNet-1K performance, although the in-distribution
performance of training with synthetic data is still inferior to real data,
the OOD generalization performance already outperforms the real counterpart at
an earlier breaking point (i.e. in ImageNet-R (IN-R) (Hendrycks et al., 2021a)
and ImageNet-A (IN-A) (Hendrycks et al., 2021b)). While we synthesize training
data to reproduce the target data distribution, the superior OOD
generalization for models trained on synthetic data indicates a greater
potential for synthetic data in strengthing the OOD generalization
performance.