# DC-BENCH: Dataset Condensation Benchmark

Justin Cui1, Ruochen Wang1, Si Si2, Cho-Jui Hsieh1
1Department of Computer Science, UCLA, 2Google Research
{justincui<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>

###### Abstract

Dataset Condensation is a newly emerging technique aiming at learning a tiny
dataset that captures the rich information encoded in the original dataset. As
the size of datasets contemporary machine learning models rely on becomes
increasingly large, condensation methods become a prominent direction for
accelerating network training and reducing data storage. Despite numerous
methods have been proposed in this rapidly growing field, evaluating and
comparing different condensation methods is non-trivial and still remains an
open issue. The quality of condensed dataset are often shadowed by many
critical contributing factors to the end performance, such as data
augmentation and model architectures. The lack of a systematic way to evaluate
and compare condensation methods not only hinders our understanding of
existing techniques, but also discourages practical usage of the synthesized
datasets. This work provides the first large-scale standardized benchmark on
Dataset Condensation. It consists of a suite of evaluations to comprehensively
reflect the generability and effectiveness of condensation methods through the
lens of their generated dataset. Leveraging this benchmark, we conduct a
large-scale study of current condensation methods, and report many insightful
findings that open up new possibilities for future development. The benchmark
library, including evaluators, baseline methods, and generated datasets, is
open-sourced to facilitate future research and application.

##  1 Introduction

Dataset plays a central role in the performance of machine learning models.
With advanced data collection and labeling tools, it becomes easier than ever
to construct large scale datasets. The rapidly growing size of contemporary
datases not only posts challenges to data storage and preprocessing, but also
makes it increasingly expensive to train machine learning models and design
new methods, such as architecture, hyperparameter, and loss function [3, 15,
48]. As a result, data condensation emerges as a promising direction that aims
at compressing the original large scale dataset into a small subset of
information-rich examples.

Existing condensation methods can be divided into two categories: 1). Data-
Selection methods, where the condensed dataset comprises of representative
examples selected directly from the full set. Example methods include random
selection, Herding [8, 35, 4, 1], and K-Center [45, 36]. 2). Data-Synthesis
methods, where the condensed dataset comprises of a set of synthesized
samples, learned to matching some statistics to the original dataset or
maximizing some utility. In particular, [44] proposed a dataset distillation
algorithm to learn the synthesized dataset via bi-level optimization, showing
outstanding performance than existing data selection methods under lower
compression ratio111compression ratio = compressed dataset size / full dataset
size. After that, many data condensation methods have been proposed to
construct synthesized datasets based on various objectives, such as matching
gradients [51, 49], embeddings [50], model parameters [5], and kernel ridge
regression [31, 32]. It has been reported that all these data-synthesis
methods significantly outperform classical data-selection methods.

Despite recent efforts in improving dataset condensation, evaluating and
comparing different methods is non-trivial and still remains an open issue to
date. Previous works [51, 50, 49, 5] mainly evaluate condensation methods by
training a randomly initialized model on the condensed dataset and report its
test accuracy. During this process, many factors could come into play on the
resulting performance, such as data augmentation and architecture. These
factors are orthogonal to the condensation algorithms, but might significantly
alter their performance. However, there does not exist a unified evaluation
protocol among prior works that aligns them. Moreover, the relative
performance of condensation methods on real-world downstream applications are
also rarely discussed. The lack of a systematic way of evaluation prevents us
from establishing fair and throughout comparisons of condensation methods,
thereby hinders our understanding of existing algorithms and discourages their
practical applications.

This work aims to provide the first benchmark to systematically evaluate data
condensation methods through the lens of the condensed dataset. We start by
asking the following question: What attributes should a high-quality condensed
dataset possess? We identify three main criteria: 1). Models trained on the
condensed dataset should achieve good performance across different training
protocols, such as different data augmentations and architectures. 2). The
condensed dataset should achieve higher performance above naive baseline
(random subset of full dataset) across various compression ratios. 3). The
condensed dataset should be able to benefit downstream tasks, such as
accelerating Neural Architecture Search (NAS). Inspired by these criteria, we
propose to measure the strength of condensation algorithms from the following
four aspects: 1). Performance under different augmentation 2). Transferability
to different architectures 3). Performance under different compression ratio
4). Performance on NAS task. These evaluations constitute the core of our
benchmark library.

Leveraging the proposed benchmark, we conduct a large-scale comprehensive
empirical analysis of State-of-the-art condensation methods. The collection of
methods in our library covers four representative data-synthesis methods: Data
Condensation with Gradient Matching (DC) [51], Differentiable Siamese
Augmentation (DSA) [49], Distribution Matching (DM) [51], and Trajectory
Matching (TM) [5]. We further include two data-selection methods in our
comparisons: random selection and our proposed KMeans-emb - a simple algorithm
with strong performance. The experimental results reveal insightful findings
on the behavior of condensed dataset, such as:

  * •

Among all existing methods in our comparison, TM demonstrates better
performance on the proposed evaluation protocols, followed by DSA which often
performs the second best, showing promising progressions in the field.
However, TM is not scalable to larger datasets and compression ratio.

  * •

Adding data augmentation to the evaluation of condensed dataset alone can
significantly boost the performance of all methods.

  * •

Condensation methods are most effective under extremely low compression
ratios. As the ratio increases (e.g., CIFAR-10 with 400 images per class), all
condensation methods perform similarly to the random selection baseline.

  * •

Despite promising results on training a single model, current condensation
methods perform poorly on large-scale real-world tasks, such as Neural
Architecture Search and training deep networks beyond the pre-specified
architecture. All existing methods encounter similar transferability to the
simple KMeans-emb baseline on CIFAR-10 and CIFAR-100.

  * •

Using better data-selection methods (e.g., our KMeans-emb) as initialization
can drastically improve the convergence speed and accuracy of condensation
methods.

We hope our work could shed lights on deeper understanding of dataset
condensation algorithms, designing more comprehensive evaluation, and
stimulating future research in advancing the state-of-the-art methods.

##  2 Dataset Condensation Benchmark

In this section, we will layout the details of our benchmark for evaluating
different condensation methods. Section 2.1 introduces our proposed evaluation
protocols. Section 2.2 explains popular DC algorithms which are included in
our library. In terms of evaluation datasets, we mainly consider three
standard image datasets - CIFAR-10, CIFAR-100, and TinyImageNet - in our
evaluation. These datasets are widely adopted in prior works on dataset
condensation. Note that we do not include those smaller and simpler datasets
such as MNIST [24] or FASHION-MNIST [46], because the performance of different
methods are quite similar on these datasets. CIFAR-10 [21] and CIFAR-100 [21]
both have 50K training images and 10K testing images from 10 and 100 classes.
TinyImageNet [23, 12] is a subset of the large-scale ImageNet dataset with 200
classes. The training split contains 100K images, and both the validation and
test set include 10K images.

###  2.1 Evaluation protocol

In this subsection, we will introduce how we measure the performance of a DC
algorithm. More specifically, we will evaluate DC algorithms for four aspects:
data augmentation, compression ratio, transferability, and performance on the
downstream task of Neural Architecture Search.

####  2.1.1 Performance under different data augmentations

To evaluate the quality of a condensed dataset, a straightforward way is to
train a randomly initialized model on it and evaluate the performance (test
accuracy) of the model. However, even for the same architecture and condensed
dataset, the resulting model can perform very differently under different
training protocols, such as the choices of data augmentation and optimizers.
Since the condensed datasets are usually very small, we observe that data
augmentation methods can significantly impact the test accuracy. Therefore, we
investigate the performance of existing condensation methods under four
different data augmentation strategies, as listed below. On the other hand,
the choice of optimizer (e.g., SGD versus Adam [20]) usually has minor impact
to the performance according to our experiments (see the appendix for
details), so we do not include it in the benchmark.

ImagenetAug is a simple manually designed augmentation containing random crop
[22], random horizontal flip and color jitters [22]. It is one of the most
commonly used strategy for training ImageNet models [27, 18].

Differentiable Siamese Augmentation (DSA) is a set of augmentation policies
designed for improving the data efficiency of Generative Adversarial Networks
(GAN) [16, 34, 40, 52]. The set includes six operations: crop, cutout [13],
flip, scale, rotate, and color jitters. At each iteration, one policy is
sampled and applied to the input image. DSA is later extended to training
synthesized images for dataset condensation [49].

AutoAugment [10] is a strong and widely adopted augmentation strategy
discovered using AutoML. The searched polices contains a total 16 data
augmentations from the popular PIL image library, plus two additional
operations: cutout and sample pairing [19]. At each iteration, a pair of
augmentations are sampled from the whole policy set and applied onto the
search. It achieves state-of-the-art accuracy on CIFAR-10, CIFAR-100, SVHN
[30] and ImageNet (without additional data).

RandAugment [11] is another popular method of automated policy search, with a
different set of discovered policies than AutoAugment. In practice, both
RandAugment and AutoAugment are commonly used for training image
classification models. Therefore, we also include RandAugment in our benchmark
for a more comprehensive evaluation.

####  2.1.2 Compression ratios

A critical dimension for evaluating dataset condensation methods is their
performance under different compression ratios, measured by the number of
synthetic Images allocated Per Class (IPC). IPC is typically set by the user
of the dataset, according to practical requirements such as storage budget.
Since the right amount of data for various scenarios might be different (e.g.
larger models often require much more data to train), we expect the user to
potentially select a wide range of IPCs in practice. As a result, it is
crucial to evaluate condensation methods under various compression ratios and
identify their effective ranges. However, previous works mainly adopt three
IPCs in their evaluation: 1, 10, 50, corresponding to 0.02%, 0.2%, and 1% of
the training split in CIFAR-10 [51, 49, 50, 5]; This range is far from
comprehensive. In contrast, we evaluate existing condensation methods for up
to 1000 IPCs, covering a much wider range of compression ratios. The results
under this setting reveal many new insights into the behavior of dataset
condensation.

####  2.1.3 Transferability across architectures

Another dimension of evaluation is on how dataset condensation methods perform
across different model architectures. Concretely, a neural network is required
to extract statistics from the original and synthetic dataset, and the
synthetic dataset is optimized by aligning the extracted information.
Therefore, we expect the condensed dataset to perform equally well when it is
used to train different architectures. Several previous works [51, 49, 50, 5]
provide evaluations under this transfer setting; however the evaluations are
mainly with on one dataset or under one IPC. Therefore the resulting
conclusion might not generalize to different datasets or IPCs.

To have a deeper understanding for the transferability of the condensation
methods, we propose a comprehensive protocol that evaluates the performance of
condensed datasets under four model architectures, three datasets (CIFAR-10,
CIFAR-100, and TinyImageNet), and three IPCs (1, 10, 50). The architectures of
choice are as follows:

MLP: We first consider a simple MLP architecture. The network includes 3 fully
connected layers and the width is set to 128. The number of trainable
parameters is around 411K.

ConvNet [22, 37, 38]: This is the standard architecture used for both training
and evaluating synthetic dataset in previous condensation works. The default
network contains three 3x3 convolution layers, each followed by 2x2 average
pooling and instance normalization. The hidden embedding size is set to 128.
There are around 320K trainable parameters. For TinyImageNet, we increase the
number of layers to 4 for improved performance, as suggested in previous work
[50, 5].

ResNet18, ResNet152 [17]: They are commonly used ResNet architecture with 4/50
residual blocks respectively. Each block contains 2 convolution layers
followed by ReLU activation and instance normalization (IN) [41]. There are
11M/60M trainable parameters in ResNet18/ResNet152.

####  2.1.4 Neural Architecture Search

One of the most promising application of data condensation methods is
accelerating Neural Architecture Search (NAS) [25, 43, 47, 26, 7, 33, 42, 6,
18]. The goal of NAS is to automatically search for a top architectures from a
vast search space. As a result, the search process typically requiring
training and evaluating hundreds or thousands of candidate architectures to
obtain their relative performance, which consumes a lot of computation
resources [53]. Since the condensed dataset are much smaller than the original
dataset, it can potentially be used to accelerate candidate training for NAS
algorithms [51]. In the ideal case, the condensed dataset, when deployed to
training architectures, can accurately reflect the relative strength of
architectures. Concretely, the ranking order (Spearman correlation) of models
trained on the condensed dataset should match those trained on the original
dataset. To effectively evaluate the condensed dataset, we propose to adopt
NAS-Bench-201 [14], a large-scale benchmark database consisting of the ground-
truth performance of 15,625 architectures of relatively large size (17
layers). This is different from previous works [51] that only experimented
with toy search space made of 720 simple ConvNet architectures.

###  2.2 Dataset Condensation methods

In this section, we introduce the condensation methods included in the current
iteration of our benchmark. Our collection of methods are open-sourced and
representative of recent progress, including four Data-Synthesis methods and
two Data-Selection methods.

####  2.2.1 Data-Synthesis methods

DC - Dataset Condensation with Gradient Matching [51] It proposes to infer the
synthetic dataset by matching the optimization trajectory of a model trained
on synthetic dataset to that on the original dataset. The optimization
trajectory is defined as the gradient direction along SGD steps.

DSA - Dataset Condensation with Differentiable Siamese Augmentation [49] DSA
proposes to apply Differentiable Siamese Augmentation [52] while learning
synthetic image, resulting in more informative synthetic images. The added
augmentations are also carried to the post evaluation of condensed dataset.

DM - Dataset Condensation with Distribution Matching [50] Unlike DC and DSA,
DM learns condensed dataset by directly matching the output features between
real and synthetic samples. The features are acquired from the last layer of a
ConvNet model with randomized weights, which corresponds to data distribution
in a randomly projected embedding space.

TM - Dataset Condensation with Trajectory matching [5] is a recently proposed
condensation method that builds a condensed dataset to match the parameter
trajectory of full data set training.

####  2.2.2 Data-Selection methods

Data-selection methods aim at cleverly picking a small subset of
representative examples from the training dataset. Since the selection is
restricted to only existing data, these methods are often considered to be
inferior to Data-Synthesis methods. In our comparison, we include two methods
along this line: random selection baseline and a novel KMeans-emb algorithm
that outperforms previous selection methods discussed in DC [51]. Note that
there are some other recently proposed data selection methods showing
promising results when integrated with model training [29, 39, 9], but their
setting is slightly different and they are focusing on higher compression
rates, so we do not include in the current paper.

Random selection One naive method of condensation is to randomly pick data
from the original dataset, which serves as the baseline for all condensation
methods. The performance of random selection is expected to increase steadily
as IPC increases, and reach full accuracy when the size of the subset equals
the full dataset.

Table 1: Kmeans test accuracy comparison using raw image feature and embedding
feature generated by ConvNet on CIFAR10/100 and TinyImageNet. For CIFAR10/100,
a 3-layer ConvNet is used. For TinyImageNet, a 4-layer ConveNet is used.

Dataset | IPC | Image Feature
---|---|---
raw input | embedding
CIFAR-10 | 1 | 21.86 | 25.16
10 | 32.21 | 41.49
50 | 47.00 | 56.00
CIFAR-100 | 1 | 6.74 | 10.89
10 | 20.08 | 25.04
50 | 35.15 | 38.64
TinyImageNet | 1 | 1.98 | 3.03
10 | 6.88 | 11.38
50 | 17.85 | 22.02

KMeans-emb Prior condensation works have examined a variety of selection-based
methods as the baseline, such as Herding, K-Center, and Forgetting [39]. But
surprisingly, these methods often fail to outperform even the random selection
baseline. For instance, K-Center, which pick data based on the centers of
KMean [28] algorithm, reports 16.4% (absolute difference) lower test accuracy
on CIFAR-10 and 50 IPCs than random selection in the original DC paper [51].
After careful investigation, we find that the reason is that these selection
based method operates on raw images. However, data similarity in raw input
space does not necessarily transfer to the embedding space where the actual
decision boundaries are drawn. Based on this insight, we propose to directly
use the distances of data points in the embedding space of a pretrained
ConvNet for selection. The resulting algorithm, which we term KMeans-emb,
leads to a substantial performance gain over both random selection and prior
selection methods (Table 1, 2). In addition, KMeans-emb can even match the
performance of some condensation methods in some cases (Table 2) while being
much faster to run. We refer the reader to Algorithm 2 in the appendix for a
detail description of KMeans-emb.

####  2.2.3 Combining Data-Selection with Data-Synthesis methods

All Data-Synthesis methods require initializing the synthetic data. This posts
a natural way of combining Data-Selection with Data-Synthesis methods - to
initialize condensed dataset with the results from Data-selection. We
experiment with three initialization strategies: using 1). random selection
baseline, 2). KMeans-emb, and 3). Gaussian noise. In contrast to prior finding
that observes similar results among different initialization methods, we show
that (Section 3.6) initialization plays an important role in accelerating the
convergence of Data-Synthesis algorithms.

##  3 Empirical studies

###  3.1 Experimental setup

Follow previous works, we use ConvNet (Section 2.1.3) to generate the
condensed dataset for all our experiments. We follow the default
hyperparameters and training configurations of the considered methods, with
two exceptions: 1). For IPCs above 50, we manually set and tune the number of
iterations for outer and inner optimization for DC and DSA, as they are
undefined for large IPCs in the original papers. 2). TM sometimes applies ZCA
whitening as a preprocessing step during both synthetic image training and
evaluation, and reported mixed results [5]. Since this is an orthogonal trick
that can be applied to any condensation algorithms, we disable it for fair
comparison with other methods. After the condensed dataset is generated, we
train 5 randomly initialized network on it for 1000 epochs using SGD
optimizer. Due to space limit, we only highlight and discuss the cases that
lead to most insightful findings in this section. We refer to readers to the
full set of results in the appendix.

###  3.2 Data augmentation

Table 2: Test accuracy for random selection, Kmeans-emb, DC, DSA, DM, and TM
under different augmentation settings on CIFAR-10, CIFAR-100 and TinyImageNet.
They are first evaluated without any augmentation (n/a in the table) and the
performance is reported. Then they are evaluated using ImagenetAug, DSA,
AutoAugment and RandAugment, the average performance (avg) and the best
performance (best) are reported.

Dataset | IPC | Random | Kmeans-emb | DC | DSA | DM | TM | Whole Dataset
---|---|---|---|---|---|---|---|---
|  | n/a | avg | best | n/a | avg | best | n/a | avg | best | n/a | avg | best | n/a | avg | best | n/a | avg | best |
| 1 | 15.06 | 14.71 | 15.4 | 23.34 | 22.89 | 25.16 | 28.08 | 26.24 | 29.34 | 27.75 | 26.26 | 27.76 | 26.15 | 24.25 | 26.45 | _39.30_ | 31.89 | 44.19 |
CIFAR10 | 10 | 25.67 | 28.03 | 31.00 | 36.43 | 38.04 | 41.49 | 44.43 | 47.11 | 50.99 | 43.54 | 47.58 | 52.96 | 42.45 | 45.70 | 47.64 | _53.49_ | 56.38 | 63.66 | 85.95
| 50 | 44.59 | 47.60 | 50.55 | 48.71 | 53.06 | 56.0 | 53.29 | 54.54 | 56.81 | 54.25 | 56.88 | 60.28 | 56.54 | 60.04 | 61.99 | _62.24_ | 65.90 | 70.28 |
| 1 | 4.28 | 4.77 | 5.30 | 8.59 | 9.47 | 10.89 | 12.55 | 12.76 | 13.66 | 13.03 | 12.72 | 13.73 | 10.79 | 8.83 | 11.20 | _16.69_ | 13.84 | 22.3 |
CIFAR100 | 10 | 14.53 | 16.78 | 18.64 | 20.73 | 23.19 | 25.04 | 25.36 | 26.94 | 28.42 | 27.12 | 29.49 | 32.23 | 25.40 | 27.64 | 29.23 | _31.76_ | 33.14 | 38.18 | 56.69
| 50 | 29.50 | 32.79 | 34.66 | 33.61 | 36.59 | 38.64 | 29.74 | 27.17 | 30.56 | 38.58 | 40.58 | 43.13 | 37.70 | 40.57 | 42.32 | _43.04_ | 42.86 | 46.32 |
| 1 | 1.42 | 1.49 | 1.65 | 2.68 | 2.65 | 3.03 | 5.26 | 4.57 | 5.27 | 5.48 | 5.12 | 5.67 | 3.73 | 3.58 | 3.82 | _5.88_ | 6.19 | 8.27 |
TinyImageNet | 10 | 4.70 | 6.00 | 6.88 | 7.83 | 9.90 | 11.38 | 11.20 | 11.20 | 12.83 | 12.43 | 14.38 | 16.43 | 12.06 | 12.69 | 13.51 | _13.60_ | 17.32 | 20.11 | 39.83
| 50 | 13.98 | 16.86 | 18.62 | 16.72 | 20.90 | 22.02 | 11.19 | 10.89 | 12.66 | _21.41_ | 22.69 | 25.31 | 20.93 | 21.57 | 22.76 | 20.12 | 25.82 | 28.16 |

  * •

_Numbers_ in italic highlights the best accuracy without augmentation. Numbers
with underline shows the best average accuracy under different augmentations.
Numbers in bold represents the highest accuracy with the best augmentation.

(a) CIFAR10 IPC 10

(b) CIFAR100 IPC 10

(c) TinyImageNet IPC 10

Figure 1: Test accuracy for different methods with different augmentations.

We start by evaluating dataset condensation methods using our benchmark with
five augmentation strategies: DSA, AutoAugment, RandAugment, ImagenetAug, and
no augmentation. Note that the change only affects the evaluation of the
condensed datasets: If a method applies data augmentation during synthetic
dataset training (DSA, DM, and TM), we keep them enabled. The results are
visualized in Figure 1. We observe that applying the right data augmentation
during evaluation significantly improves the performance of model trained on
the condensed dataset. For example, on CIFAR-10 and IPC=10, the test accuracy
of DC, DSA, DM and TM increases by 6.56%, 9.4%, 5.2% and 10.2% respectively.

One interesting comparison is between DC and DSA. Their only difference is
that DSA applies augmentation during both synthetic dataset training and
evaluation, whereas DC does not use any augmentation [51, 49]. In our
experiment, we found that the performance of DC is largely underestimated, due
to misalignment in the evaluation protocol. After applying the best
augmentation, DC’s performance increases by up to 6%. On the other hand, when
data augmentation is disabled during evaluation, the performance of condensed
dataset trained by DSA drops to a similar level of DC in many cases.

Although in some cases users may stick with the best augmentation associated
with the condensed dataset, there exists cases that user wish to use different
augmentations depending on the task at hand. Therefore, we propose to evaluate
each method under: 1). no augmentation 2). best augmentation and 3). the
average test accuracy under all augmentations. The numerical results are
summarized in Table 2. We observe that: 1). DSA augmentation overall is the
best strategies in our experiments. 2). Some methods exhibit much higher
variance across different augmentation. For example, although DSA and TM
achieves top 2 performance, they have a much wider accuracy range than DM.
This indicates that DSA and TM might potentially overfit to the augmentation
used during synthetic dataset learning.

Further, to establish fair comparison, we also reevaluate Data-Selection
methods with augmentation enabled. This has been overlooked in many previous
works where the baseline data-selection methods they compared against are all
evaluated and reported without augmentation. As shown in Table 2, similar to
Data-Synthesis methods, Data-Selection method also benefits significantly from
augmentation. For instance, both random selection and our KMeans-emb achieves
over 5% absolute gain in test accuracy on all dataset under IPC 50 (except for
TinyImageNet which is close: 4.64%). Most noticeably, under 50 IPCs, KMeans-
emb even outperforms DC on CIFAR-100 and TinyImageNet, and on-par with DC on
CIFAR-10. The impressive performance of our simple KMeans-emb showcases that
the potential of selection-based methods is drastically underestimated.

###  3.3 Different compression ratios

As previous discussed, prior works mainly evaluate condensed dataset of size
up to 50 IPCs (1% compression ratio). However, this is a rather extreme case,
and in most practical applications users may be willing to increase the
compression ratio to get a more informative subset. To analyze the performance
of condensation algorithms under larger compression ratios, we rerun the
selected methods with IPC up to 1,000. For fair comparison, DSA augmentation
is enabled during evaluation for all base methods, and the synthetic dataset
is initialized from random real images.

Figure 2: Performance comparison with different compression ratios on CIFAR-10
using Random selection, Kmeans-emb, DC, DSA and DM.

Ideally, we expect the performance trajectory of condensation methods to
approach the oracle accuracy (brown horizontal line) on the full dataset much
faster than the random selection. However, this is not the case in practice.
As shown in Figure 2, the performance gain of Data-Synthesis methods over
selection based methods is only obvious for IPCs less than 200. As IPC
increases further, the margin shrinks considerably and all methods perform
similarly to the random selection baseline. Considering the fact that the
synthetic datasets are initialized from random selection in the first place,
it reveals that all current condensed methods fail to effectively explore the
increased capacity brought by the extra IPCs. Moreover, although DM [50]
outperforms DSA [49] with IPC 50, it’s not able to consistently outperform DSA
[49] with larger IPCs which contradicts with the claim made in [50]. We do not
include TM in the plot as it does not scale well to IPCs beyond 50 on
CIFAR-10, in terms of both memory and run-time. The scalability becomes worse
on larger dataset with more classes and higher resolutions (See Appendix for
details). For ease of reference, we also report numerical results in Appendix.

###  3.4 Transferability

Table 3: Testing accuracy of different methods on ConvNet versus transferred
to other architectures. The "Transfer" column records the average results on
MLP, ResNet18, and ResNet152. All methods are evaluated with 10 IPCs. Results
under other IPCs can be found in the Appendix.

| CIFAR-10 | CIFAR-100 | TinyImageNet
---|---|---|---
| ConvNet | Transfer | ConvNet | Transfer | ConvNet | Transfer
Random | 31.00 | 23.48 | 18.64 | 14.49 | 6.88 | 2.49
Kmeans-emb | 41.19 | 30.78 | 25.04 | 14.41 | 11.38 | 3.91
DC | 50.99 | 31.51 | 28.42 | 11.86 | 12.83 | 3.60
DSA | 52.96 | 30.90 | 32.23 | 14.49 | 16.34 | 5.27
DM | 47.64 | 29.42 | 29.23 | 12.44 | 13.51 | 2.96
TM | 63.66 | 30.87 | 38.18 | 15.62 | 20.11 | 5.50

We evaluated the transferability of different condensation methods on the 3
datasets with IPC 1, 10 and 50 using the 4 architectures described previously.
As showing in Figure 3, the performance of all synthetic methods drop when
transferring to other architectures. Moreover, the relative ranking of
different condensed dataset might not necessarily be preserved when
transferred to different architectures. For instance, on CIFAR-100, while DC
outperforms KMeans-emb on ConvNet, it falls behind on ResNet architecture.
Another example is on CIFAR-10, where we see that the dataset learned by the
top-notch method, TM, transfers poorly to MLP networks. In addition, we
observe that our KMeans-emb achieves the better transferability than several
data-synthesis methods (Table 3).

One particular observation we want to point out is that none of the
condensation methods we tested perform well when transferred to large models
like ResNet152. We conjecture that it is because the larger ResNet152 might
require more data to optimize than models of much smaller size, such as
ConvNet. This is evidenced by the fact that ResNet152 performs worse than any
other architecture on random selection with small IPCs. To further verify it,
we trained ResNet152 with 300 randomly selected images per class and obtain an
accuracy of 45.59%; When the IPC is increased to 1,000, ResNet152 achieves
70.29% accuracy, gradually gaining back its full potential. This result serves
as an extra piece of evidence that a good condensed method should operate
robustly under different compression ratios. Please refer to appendix for the
numerical results.

(a) CIFAR10 IPC 10

(b) CIFAR100 IPC 10

(c) TinyImageNet IPC 10

Figure 3: Condensed dataset performance evaluated using different networks.

###  3.5 Neural Architecture Search (NAS)

One promising application of DC is on Neural Architecture Search as shown in
[51]. To evaluate this task, we randomly sample 100 networks from NAS-
Bench-201 [14], which contains ground-truth performance of 15,625 networks.
All models are trained on CIFAR-10 for 50 epochs under 5 random seeds, and
ranked according to their average accuracy on a held-out validation set of 10k
images. We reduce the number of repeated blocks from 15 to 3 during the search
phase, as we found that original networks perform poorly when trained on
condensed dataset due to their size. This is a common practice in NAS [27]. We
measure the performance on task NAS with two metrics: 1). Correlation between
the ranking of models trained on condensed dataset and original dataset 2).
The ground-truth performance of the best architecture trained on the condensed
dataset (Top 1). We argue that ranking correlation is more important than Top
1 accuracy, as it measures how well the condensed dataset can reflect the
relative strength of various architectures [2].

Table 4: Spearman’s rank correlation using NAS-Bench-201. The state-of-the-art
performance on the test set is 94.36%. The rank correlation of original
dataset is lower than 1.0 because we use a small architecture and perform
ranking based on validation set.

| Random | Kmeans-emb | DC | DSA | DM | TM | Original Dataset
---|---|---|---|---|---|---|---
Correlation | -0.06 | 0.11 | -0.19 | -0.37 | -0.37 | -0.09 | 0.7487
Top 1 (%) | 91.9 | 91.78 | 86.44 | 73.54 | 92.16 | 92.91 | 93.5

Although prior work on utilizing condensed dataset for NAS reports promising
results (0.79 Spearman correlation using DC [51]), they mainly consider toy
search spaces made of 720 ConvNet architectures. With larger scale NAS
benchmark with modern architectures, we have a different observation. As shown
in Table 4, we found that there is little or even negative correlation between
the performance on condensed and full dataset. All methods produce negative
correlation except for our proposed Kmeans-emb. This shows that condensed
dataset fails to preserve the true strength of the underlying model. Moreover,
the best architecture discovered from DC and DSA’s condensed dataset performs
poorly on the full dataset. Our result indicates that, despite the performance
gain brought by recent condensation methods on training a single specific
model, it remains challenging to truly utilize the condensed dataset to guide
model designs.

###  3.6 Combining Data-Selection methods with Data-Synthesis methods

Initialization of synthetic dataset is a relatively unexplored territory.
Prior Data-Synthesis methods typically initialize the dataset from either
Gaussian noise or randomly selected images. Since advanced selection methods
such as KMeans-emb outperforms random selection by a large margin, a natural
question is whether condensation methods would benefit from images selected
from KMeans-emb. This also serves as a direct way of combining data selection
methods with data synthesis methods. We test our hypothesis by using images
from KMeans-emb to initialize condensation methods. As shown in Figure 4,
KMeans-emb initialization not only converges faster, but also achieves
improved end performance in some cases. On average, we observe that KMeans-emb
only requires about 30% of the computation budget to reach the same level of
performance as random selection; The end performance is also 1.3% higher as
well. The saving is desirable, especially considering the fact that the
condensed dataset usually takes a long time to train (e.g. 15h for DSA under
50 IPCs on CIFAR-10). Due to space limit, we only show the curves of DM in the
main text; the plots of other methods can be found in the appendix.

(a) DM IPC 1

(b) DM IPC 10

(c) DM IPC 50

Figure 4: Test accuracy comparison among initializing synthetic images with
Random selection, Kmeans-emb and Gaussian noise on CIFAR-10 using DM.

##  4 DC-BENCH Library

Algorithm 1 PyTorch integration with DC-BENCH

⬇

1from evaluator.evaluator import CrossArchEvaluator

2from evaluator.evaluator_utils import EvaluatorUtils

3from dataloader import CondensedDataLoader

4from nas import NASEvaluator

5

6# Set evaluation parameters.

7ipc=50, augmentations=’autoaug,randaug,imagenetaug,dsa,’,

8init=’kmeans-emb’, dc_method=’TM’, dataset=’TinyImagenet’,

9models=’ConvNet,ResNet18,MLP,ResNet152’, data_path=’.’

10# Load training and test dataset.

11dst_train = CondensedDataLoader.load(’path_to_data’, dc_method)

12dst_test = EvaluatorUtils.get_testset(dataset)

13# Start evaluation with different models and different augmentations.

14per_arch_performance = CrossArchEvaluator.evaluate(dst_train, dst_test, ipc,
augmentations, init, models)

15# Set Neural Architecture Search parameters.

16num_network=1000, repeat_block=5, eval_method=’spearman’

17# Start Neural Architecture Search.

18rank_correlation = NASEvaluator.evaluate(data_path, num_network,
repeat_block, eval_method)

We design and implement an evaluation library that incorporates all the
aforementioned protocols. An example usage of the library API is shown in 1.
To facilitate the research on dataset condensation, we set up a website (see
the appendix for details) with four leaderboards to record the performance of
existing condensation methods and new submissions. The entire benchmark,
including the evaluation library, condensed datasets, and scripts to reproduce
the results in this paper, can be found at
(https://github.com/justincui03/dc_benchmark).

##  5 Outlook

Conclusion This paper introduces the first large-scale benchmark on dataset
condensation methods. Leveraging the proposed benchmark, we conduct the first
comprehensive empirical analysis of existing condensation algorithms. Our
study reveals several scenarios where current method can be improved, leading
to the following potential research directions: 1). (automated) designing
better augmentation that suits the synthetic data, 2). improving the
transferability of condensed dataset to other architectures 3). condensation
methods for NAS 4). developing condensed methods that perform well for a wide
range of compression ratios. and 5) effective ways to combine Data-Selection
with Data-Synthesis methods. We hope the proposed benchmark could guide users
to choose and evaluate various DC methods and facilitate future developments
of advanced data condensation methods. Limitation and outlook While the
current iteration of our benchmark is comprehensive at the moment, it could
become limited in scope as the field advances. In the future, we plan to
expand the scope of our DC-Bench to include more architectures, datasets, and
downstream tasks. As current methods primarily focus on image classification,
we will also incorporate tasks from other modality, such as text, graph, and
audio.

## Ethics Statements

The condensed dataset used in this paper are all generated from the following
standard non-private dataset: CIFAR-10, CIFAR-100, and TinyImageNet. The
library itself does not include any sensitive information or components.
Therefore, we are not aware of any ethical concern of the benchmark. However,
the end users should be aware of potential data leakage through condensed
dataset, when they try to apply any condensation methods included in our
benchmark to their tasks at hand.

## References

  * [1] Eden Belouadah and Adrian Popescu  “Scail: Classifier weights scaling for class incremental learning”  In _Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision_ , 2020, pp. 1266–1275
  * [2] Gabriel Bender et al.  “Understanding and Simplifying One-Shot Architecture Search”  In _Proceedings of the 35th International Conference on Machine Learning_ 80, Proceedings of Machine Learning Research  Stockholmsmässan, Stockholm Sweden: PMLR, 2018, pp. 550–559  URL: http://proceedings.mlr.press/v80/bender18a.html
  * [3] James Bergstra and Yoshua Bengio  “Random search for hyper-parameter optimization.”  In _Journal of machine learning research_ 13.2, 2012
  * [4] Francisco M Castro et al.  “End-to-end incremental learning”  In _Proceedings of the European conference on computer vision (ECCV)_ , 2018, pp. 233–248
  * [5] George Cazenavette et al.  “Dataset distillation by matching training trajectories”  In _arXiv preprint arXiv:2203.11932_ , 2022
  * [6] Xiangning Chen et al.  “DrNAS: Dirichlet Neural Architecture Search”  In _International Conference on Learning Representations_ , 2021  URL: https://openreview.net/forum?id=9FWas6YbmB3
  * [7] Xin Chen, Lingxi Xie, Jun Wu and Qi Tian  “Progressive differentiable architecture search: Bridging the depth gap between search and evaluation”  In _Proceedings of the IEEE International Conference on Computer Vision_ , 2019, pp. 1294–1303
  * [8] Yutian Chen, Max Welling and Alex Smola  “Super-samples from kernel herding”  In _Proceedings of the Twenty-Sixth Conference on Uncertainty in Artificial Intelligence_ , 2010, pp. 109–116
  * [9] Cody Coleman et al.  “Selection via proxy: Efficient data selection for deep learning”  In _arXiv preprint arXiv:1906.11829_ , 2019
  * [10] Ekin D Cubuk et al.  “Autoaugment: Learning augmentation policies from data”  In _arXiv preprint arXiv:1805.09501_ , 2018
  * [11] Ekin D Cubuk, Barret Zoph, Jonathon Shlens and Quoc V Le  “Randaugment: Practical automated data augmentation with a reduced search space”  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops_ , 2020, pp. 702–703
  * [12] Jia Deng et al.  “Imagenet: A large-scale hierarchical image database”  In _2009 IEEE conference on computer vision and pattern recognition_ , 2009, pp. 248–255  Ieee
  * [13] Terrance DeVries and Graham W Taylor  “Improved regularization of convolutional neural networks with cutout”  In _arXiv preprint arXiv:1708.04552_ , 2017
  * [14] Xuanyi Dong and Yi Yang  “Nas-bench-201: Extending the scope of reproducible neural architecture search”  In _arXiv preprint arXiv:2001.00326_ , 2020
  * [15] Thomas Elsken, Jan Hendrik Metzen and Frank Hutter  “Neural architecture search: A survey”  In _The Journal of Machine Learning Research_ 20.1 JMLR. org, 2019, pp. 1997–2017
  * [16] Ian Goodfellow, Yoshua Bengio, Aaron Courville and Yoshua Bengio  “Deep learning”  MIT Press, 2016
  * [17] Kaiming He, Xiangyu Zhang, Shaoqing Ren and Jian Sun  “Deep residual learning for image recognition”  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2016, pp. 770–778
  * [18] Shoukang Hu et al.  “Generalizing Few-Shot NAS with Gradient Matching”  In _International Conference on Learning Representations_ , 2021
  * [19] Hiroshi Inoue  “Data augmentation by pairing samples for images classification”  In _arXiv preprint arXiv:1801.02929_ , 2018
  * [20] Diederik P Kingma and Jimmy Ba  “Adam: A method for stochastic optimization”  In _arXiv preprint arXiv:1412.6980_ , 2014
  * [21] Alex Krizhevsky and Geoffrey Hinton  “Learning multiple layers of features from tiny images”  Citeseer, 2009
  * [22] Alex Krizhevsky, Ilya Sutskever and Geoffrey E Hinton  “Imagenet classification with deep convolutional neural networks”  In _Advances in neural information processing systems_ 25, 2012
  * [23] Ya Le and Xuan Yang  “Tiny imagenet visual recognition challenge”
  * [24] Yann LeCun, Léon Bottou, Yoshua Bengio and Patrick Haffner  “Gradient-based learning applied to document recognition”  In _Proceedings of the IEEE_ 86.11 Ieee, 1998, pp. 2278–2324
  * [25] Liam Li and Ameet Talwalkar  “Random Search and Reproducibility for Neural Architecture Search”, 2019  arXiv:1902.07638 [cs.LG]
  * [26] Chenxi Liu et al.  “Progressive Neural Architecture Search”  In _Lecture Notes in Computer Science_ , 2018, pp. 19–35
  * [27] Hanxiao Liu, Karen Simonyan and Yiming Yang  “DARTS: Differentiable Architecture Search”  In _International Conference on Learning Representations_ , 2019
  * [28] Stuart Lloyd  “Least squares quantization in PCM”  In _IEEE transactions on information theory_ 28.2 IEEE, 1982, pp. 129–137
  * [29] Baharan Mirzasoleiman, Jeff Bilmes and Jure Leskovec  “Coresets for data-efficient training of machine learning models”  In _International Conference on Machine Learning_ , 2020, pp. 6950–6960  PMLR
  * [30] Yuval Netzer et al.  “Reading digits in natural images with unsupervised feature learning”, 2011
  * [31] Timothy Nguyen, Zhourong Chen and Jaehoon Lee  “Dataset Meta-Learning from Kernel Ridge-Regression”  In _International Conference on Learning Representations_ , 2020
  * [32] Timothy Nguyen, Roman Novak, Lechao Xiao and Jaehoon Lee  “Dataset Distillation with Infinitely Wide Convolutional Networks”  In _Advances in Neural Information Processing Systems_ , 2021
  * [33] Hieu Pham et al.  “Efficient Neural Architecture Search via Parameters Sharing”  In _Proceedings of the 35th International Conference on Machine Learning_ 80, Proceedings of Machine Learning Research  Stockholmsmässan, Stockholm Sweden: PMLR, 2018, pp. 4095–4104
  * [34] Alec Radford, Luke Metz and Soumith Chintala  “Unsupervised representation learning with deep convolutional generative adversarial networks”  In _arXiv preprint arXiv:1511.06434_ , 2015
  * [35] Sylvestre-Alvise Rebuffi, Alexander Kolesnikov, Georg Sperl and Christoph H Lampert  “icarl: Incremental classifier and representation learning”  In _Proceedings of the IEEE conference on Computer Vision and Pattern Recognition_ , 2017, pp. 2001–2010
  * [36] Ozan Sener and Silvio Savarese  “Active learning for convolutional neural networks: A core-set approach”  In _arXiv preprint arXiv:1708.00489_ , 2017
  * [37] Karen Simonyan and Andrew Zisserman  “Very deep convolutional networks for large-scale image recognition”  In _arXiv preprint arXiv:1409.1556_ , 2014
  * [38] Christian Szegedy et al.  “Going deeper with convolutions”  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2015, pp. 1–9
  * [39] Mariya Toneva et al.  “An empirical study of example forgetting during deep neural network learning”  In _arXiv preprint arXiv:1812.05159_ , 2018
  * [40] Ngoc-Trung Tran et al.  “Towards good practices for data augmentation in gan training”  In _arXiv preprint arXiv:2006.05338_ 2, 2020, pp. 3
  * [41] Dmitry Ulyanov, Andrea Vedaldi and Victor Lempitsky  “Improved texture networks: Maximizing quality and diversity in feed-forward stylization and texture synthesis”  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2017, pp. 6924–6932
  * [42] Ruochen Wang et al.  “RANK-NOSH: Efficient Predictor-Based Architecture Search via Non-Uniform Successive Halving”  In _ICCV_ , 2021
  * [43] Ruochen Wang et al.  “Rethinking Architecture Selection in Differentiable NAS”  In _International Conference on Learning Representation_ , 2021
  * [44] Tongzhou Wang, Jun-Yan Zhu, Antonio Torralba and Alexei A Efros  “Dataset distillation”  In _arXiv preprint arXiv:1811.10959_ , 2018
  * [45] Gert W Wolf  “Facility location: concepts, models, algorithms and case studies. Series: Contributions to Management Science”  In _International Journal of Geographical Information Science_ 25.2 Taylor & Francis, Inc. Bristol, PA, USA, 2011, pp. 331–333
  * [46] Han Xiao, Kashif Rasul and Roland Vollgraf  “Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms”  In _arXiv preprint arXiv:1708.07747_ , 2017
  * [47] Quanming Yao, Ju Xu, Wei-Wei Tu and Zhanxing Zhu  “Efficient Neural Architecture Search via Proximal Iterations”  In _AAAI_ , 2020
  * [48] Chris Ying et al.  “Nas-bench-101: Towards reproducible neural architecture search”  In _International Conference on Machine Learning_ , 2019, pp. 7105–7114  PMLR
  * [49] Bo Zhao and Hakan Bilen  “Dataset condensation with differentiable siamese augmentation”  In _International Conference on Machine Learning_ , 2021, pp. 12674–12685  PMLR
  * [50] Bo Zhao and Hakan Bilen  “Dataset Condensation with Distribution Matching”  In _arXiv preprint arXiv:2110.04181_ , 2021
  * [51] Bo Zhao, Konda Reddy Mopuri and Hakan Bilen  “Dataset Condensation with Gradient Matching”  In _International Conference on Learning Representations_ , 2020
  * [52] Shengyu Zhao et al.  “Differentiable augmentation for data-efficient gan training”  In _Advances in Neural Information Processing Systems_ 33, 2020, pp. 7559–7570
  * [53] Barret Zoph and Quoc V. Le  “Neural Architecture Search with Reinforcement Learning”  In _International Conference on Learning Representations (ICLR)_ , 2017  URL: https://arxiv.org/abs/1611.01578

## Checklist

  1. 1.

For all authors…

    1. (a)

Do the main claims made in the abstract and introduction accurately reflect
the paper’s contributions and scope? [Yes]

    2. (b)

Did you describe the limitations of your work? [Yes]

    3. (c)

Did you discuss any potential negative societal impacts of your work? [Yes]

    4. (d)

Have you read the ethics review guidelines and ensured that your paper
conforms to them? [Yes]

  2. 2.

If you are including theoretical results…

    1. (a)

Did you state the full set of assumptions of all theoretical results? [N/A]

    2. (b)

Did you include complete proofs of all theoretical results? [N/A]

  3. 3.

If you ran experiments…

    1. (a)

Did you include the code, data, and instructions needed to reproduce the main
experimental results (either in the supplemental material or as a URL)? [Yes]

    2. (b)

Did you specify all the training details (e.g., data splits, hyperparameters,
how they were chosen)? [Yes]

    3. (c)

Did you report error bars (e.g., with respect to the random seed after running
experiments multiple times)? [Yes] We report the full data in appendix

    4. (d)

Did you include the total amount of compute and the type of resources used
(e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] We report
these data in appendix section

  4. 4.

If you are using existing assets (e.g., code, data, models) or
curating/releasing new assets…

    1. (a)

If your work uses existing assets, did you cite the creators? [Yes]

    2. (b)

Did you mention the license of the assets? [Yes] We report the license in
appendix

    3. (c)

Did you include any new assets either in the supplemental material or as a
URL? [Yes]

    4. (d)

Did you discuss whether and how consent was obtained from people whose data
you’re using/curating? [Yes]

    5. (e)

Did you discuss whether the data you are using/curating contains personally
identifiable information or offensive content? [Yes]

  5. 5.

If you used crowdsourcing or conducted research with human subjects…

    1. (a)

Did you include the full text of instructions given to participants and
screenshots, if applicable? [N/A]

    2. (b)

Did you describe any potential participant risks, with links to Institutional
Review Board (IRB) approvals, if applicable? [N/A]

    3. (c)

Did you include the estimated hourly wage paid to participants and the total
amount spent on participant compensation? [N/A]

##  Appendix A Appendix

###  A.1 Hyperparameters

For all synthesis based methods, we use the default parameters given by the
authors in their original paper. E.g. For DC and DSA, we run it for 1000
iterations. The only change we make in order to have them scale up to IPCs
larger than 50 is setting inner loop and outer loop to both 10 which are not
given by the original authors. For DM, we run the condensation method for
20,000 iterations which is the same as the author. For TM, we use the
condensed dataset provided by the author whose exact settings can be found in
[5]. For Kmeans-emb, we use the Kmeans implementation inside scikit-learn
which is a publicly available library with maximum 300 iterations. In order to
generate the embedding features for the image, we create a random ConvNet
model and train it with 1 epoch using the whole dataset. For CIFAR10/100, we
use a 3 layer ConvNet, for TinyImageNet, we use a 4 layer ConvNet as suggested
by [50].

###  A.2 Data augmentation

In Table 2, we report the average and best performance of different methods
under different augmentation settings. Here we show the complete numerical
results in Table 5

Table 5: Complete test accuracy for Random selection, Kmeans-emb, DC, DSA, DM,
TM under different augmentation settings on CIFAR-10, CIFAR-100 and
TinyImageNet.

Dataset | Method | IPC | Augmentation
---|---|---|---
n/a | imagenet_aug | randaug | autoaug | DSA
CIFAR-10 | Random | 1 | 15.06 $\pm$ 0.60 | 15.07 $\pm$ 0.27 | 14.58 $\pm$ 0.54 | 13.78 $\pm$ 0.69 | 15.4 $\pm$ 0.28
10 | 25.67 $\pm$ 0.36 | 29.22 $\pm$ 0.33 | 26.59 $\pm$ 0.58 | 25.32 $\pm$ 0.67 | 31.00 $\pm$ 0.48
50 | 44.59 $\pm$ 0.32 | 50.00 $\pm$ 0.48 | 47.36 $\pm$ 0.52 | 42.47 $\pm$ 0.57 | 50.55 $\pm$ 0.32
Kmeans-emb | 1 | 23.34 $\pm$ 0.90 | 25.92 $\pm$ 1.12 | 21.99 $\pm$ 0.53 | 18.5 $\pm$ 0.62 | 25.16 $\pm$ 0.45
10 | 36.43 $\pm$ 0.63 | 39.46 $\pm$ 0.59 | 38.18 $\pm$ 1.52 | 33.01 $\pm$ 0.69 | 41.49 $\pm$ 0.73
50 | 48.71 $\pm$ 0.33 | 55.53 $\pm$ 0.51 | 53.09 $\pm$ 0.41 | 47.62 $\pm$ 0.53 | 56.00 $\pm$ 0.29
DC | 1 | 28.08 $\pm$ 0.80 | 25.70 $\pm$ 0.82 | 27.39 $\pm$ 0.81 | 22.51 $\pm$ 0.91 | 29.34 $\pm$ 0.37
10 | 44.43 $\pm$ 0.85 | 46.67 $\pm$ 0.43 | 47.65 $\pm$ 0.36 | 43.13 $\pm$ 0.80 | 50.99 $\pm$ 0.62
50 | 53.29 $\pm$ 0.92 | 55.04 $\pm$ 0.20 | 54.82 $\pm$ 0.31 | 51.49 $\pm$ 0.42 | 56.81 $\pm$ 0.44
DSA | 1 | 27.75 $\pm$ 0.54 | 25.81 $\pm$ 0.66 | 26.71 $\pm$ 0.57 | 24.75 $\pm$ 0.98 | 27.76 $\pm$ 0.47
10 | 43.54 $\pm$ 0.37 | 47.84 $\pm$ 0.41 | 47.78 $\pm$ 0.57 | 41.75 $\pm$ 0.81 | 52.96 $\pm$ 0.41
50 | 54.25 $\pm$ 0.57 | 59.11 $\pm$ 0.57 | 56.15 $\pm$ 0.71 | 51.98 $\pm$ 0.51 | 60.28 $\pm$ 0.37
DM | 1 | 26.15 $\pm$ 0.80 | 24.54 $\pm$ 0.67 | 24.96 $\pm$ 0.89 | 21.06 $\pm$ 2.03 | 26.45 $\pm$ 0.39
10 | 42.45 $\pm$ 0.43 | 45.67 $\pm$ 0.55 | 44.95 $\pm$ 0.53 | 44.55 $\pm$ 0.97 | 47.64 $\pm$ 0.55
50 | 56.54 $\pm$ 0.41 | 60.42 $\pm$ 0.42 | 60.17 $\pm$ 0.43 | 57.56 $\pm$ 0.42 | 61.99 $\pm$ 0.33
TM | 1 | 39.30 $\pm$ 1.14 | 28.23 $\pm$ 2.09 | 31.57 $\pm$ 1.41 | 23.57 $\pm$ 1.45 | 44.19 $\pm$ 1.18
10 | 53.49 $\pm$ 0.74 | 56.86 $\pm$ 0.57 | 57.87 $\pm$ 0.93 | 47.11 $\pm$ 0.98 | 63.66 $\pm$ 0.38
50 | 62.24 $\pm$ 0.52 | 65.67 $\pm$ 0.75 | 66.38 $\pm$ 0.44 | 61.25 $\pm$ 0.76 | 70.28 $\pm$ 0.61
CIFAR-100 | Random | 1 | 4.28 $\pm$ 0.20 | 4.67 $\pm$ 0.15 | 4.60 $\pm$ 0.17 | 4.49 $\pm$ 0.13 | 5.30 $\pm$ 0.23
10 | 14.53 $\pm$ 0.27 | 17.1 $\pm$ 0.36 | 16.44 $\pm$ 0.30 | 14.93 $\pm$ 0.24 | 18.64 $\pm$ 0.25
50 | 29.50 $\pm$ 0.26 | 34.24 $\pm$ 0.23 | 31.94 $\pm$ 0.26 | 30.31 $\pm$ 0.31 | 34.66 $\pm$ 0.41
Kmeans-emb | 1 | 8.59 $\pm$ 0.27 | 9.25 $\pm$ 0.19 | 9.14 $\pm$ 0.23 | 8.60 $\pm$ 0.32 | 10.89 $\pm$ 0.17
10 | 20.73 $\pm$ 0.22 | 23.56 $\pm$ 0.19 | 23.48 $\pm$ 0.19 | 20.67 $\pm$ 0.23 | 25.04 $\pm$ 0.30
50 | 33.61 $\pm$ 0.41 | 37.73 $\pm$ 0.34 | 36.19 $\pm$ 0.32 | 33.79 $\pm$ 0.37 | 38.64 $\pm$ 0.43
DC | 1 | 12.55 $\pm$ 0.37 | 11.46 $\pm$ 0.28 | 12.98 $\pm$ 0.42 | 12.93 $\pm$ 0.24 | 13.66 $\pm$ 0.29
10 | 25.36 $\pm$ 0.28 | 25.51 $\pm$ 0.36 | 27.96 $\pm$ 0.30 | 25.87 $\pm$ 0.53 | 28.42 $\pm$ 0.29
50 | 29.74 $\pm$ 0.34 | 24.72 $\pm$ 0.34 | 27.54 $\pm$ 0.42 | 25.87 $\pm$ 0.18 | 30.56 $\pm$ 0.56
DSA | 1 | 13.03 $\pm$ 0.14 | 11.50 $\pm$ 0.15 | 12.61 $\pm$ 0.33 | 13.03 $\pm$ 0.33 | 13.73 $\pm$ 0.45
10 | 27.12 $\pm$ 0.27 | 28.38 $\pm$ 0.32 | 29.78 $\pm$ 0.29 | 27.58 $\pm$ 0.25 | 32.23 $\pm$ 0.35
50 | 38.58 $\pm$ 0.28 | 40.26 $\pm$ 0.34 | 40.81 $\pm$ 0.27 | 38.11 $\pm$ 0.54 | 43.13 $\pm$ 0.33
DM | 1 | 10.79 $\pm$ 0.31 | 7.52 $\pm$ 0.22 | 8.75 $\pm$ 0.30 | 7.85 $\pm$ 0.44 | 11.20 $\pm$ 0.27
10 | 25.40 $\pm$ 0.22 | 26.5 $\pm$ 0.13 | 28.15 $\pm$ 0.32 | 26.66 $\pm$ 0.25 | 29.23 $\pm$ 0.26
50 | 37.7 $\pm$ 0.26 | 40.45 $\pm$ 0.32 | 40.69 $\pm$ 0.35 | 38.81 $\pm$ 0.31 | 42.32 $\pm$ 0.37
TM | 1 | 16.69 $\pm$ 0.64 | 11.55 $\pm$ 0.33 | 10.87 $\pm$ 0.34 | 10.65 $\pm$ 0.35 | 22.3 $\pm$ 0.55
10 | 31.76 $\pm$ 0.54 | 30.46 $\pm$ 0.37 | 34.37 $\pm$ 0.36 | 29.53 $\pm$ 0.41 | 38.18 $\pm$ 0.42
50 | 43.04 $\pm$ 0.48 | 41.55 $\pm$ 0.26 | 44.73 $\pm$ 0.33 | 38.84 $\pm$ 0.26 | 46.32 $\pm$ 0.26
TinyImageNet | Random | 1 | 1.42 $\pm$ 0.08 | 1.45 $\pm$ 0.05 | 1.50 $\pm$ 0.08 | 1.37 $\pm$ 0.08 | 1.65 $\pm$ 0.11
10 | 4.70 $\pm$ 0.18 | 6.15 $\pm$ 0.11 | 5.66 $\pm$ 0.16 | 5.27 $\pm$ 0.19 | 6.88 $\pm$ 0.25
50 | 13.98 $\pm$ 0.28 | 17.39 $\pm$ 0.21 | 16.44 $\pm$ 0.25 | 15.00 $\pm$ 0.32 | 18.62 $\pm$ 0.22
Kmeans-emb | 1 | 2.68 $\pm$ 0.20 | 2.68 $\pm$ 0.18 | 2.53 $\pm$ 0.11 | 2.34 $\pm$ 0.15 | 3.03 $\pm$ 0.12
10 | 7.83 $\pm$ 0.35 | 10.17 $\pm$ 0.20 | 9.40 $\pm$ 0.22 | 8.63 $\pm$ 0.18 | 11.38 $\pm$ 0.26
50 | 16.72 $\pm$ 0.41 | 20.47 $\pm$ 0.23 | 19.79 $\pm$ 0.46 | 17.75 $\pm$ 0.18 | 22.02 $\pm$ 0.40
DC | 1 | 5.26 $\pm$ 0.19 | 4.02 $\pm$ 0.13 | 4.80 $\pm$ 0.09 | 4.18 $\pm$ 0.21 | 5.27 $\pm$ 0.10
10 | 11.12 $\pm$ 0.28 | 9.95 $\pm$ 0.24 | 12.16 $\pm$ 0.20 | 9.86 $\pm$ 0.22 | 12.83 $\pm$ 0.14
50 | 11.19 $\pm$ 0.27 | 9.26 $\pm$ 0.37 | 12.303 $\pm$ 0.19 | 9.33 $\pm$ 0.13 | 12.66 $\pm$ 0.36
DSA | 1 | 5.48 $\pm$ 0.14 | 4.19 $\pm$ 0.10 | 5.23 $\pm$ 0.32 | 5.38 $\pm$ 0.11 | 5.67 $\pm$ 0.14
10 | 12.43 $\pm$ 0.29 | 12.53 $\pm$ 0.24 | 14.46 $\pm$ 0.23 | 14.18 $\pm$ 0.14 | 16.34 $\pm$ 0.21
50 | 21.41 $\pm$ 0.25 | 22.47 $\pm$ 0.37 | 22.98 $\pm$ 0.35 | 20.00 $\pm$ 0.29 | 25.31 $\pm$ 0.22
DM | 1 | 3.73 $\pm$ 0.22 | 3.15 $\pm$ 0.19 | 3.65 $\pm$ 0.21 | 3.70 $\pm$ 0.16 | 3.82 $\pm$ 0.21
10 | 12.06 $\pm$ 0.43 | 12.43 $\pm$ 0.31 | 13.04 $\pm$ 0.18 | 11.76 $\pm$ 0.17 | 13.51 $\pm$ 0.31
50 | 20.93 $\pm$ 0.32 | 22.19 $\pm$ 0.34 | 22.03 $\pm$ 0.22 | 19.31 $\pm$ 0.33 | 22.76 $\pm$ 0.28
TM | 1 | 5.88 $\pm$ 0.41 | 5.13 $\pm$ 0.30 | 5.92 $\pm$ 0.16 | 5.44 $\pm$ 0.23 | 8.27 $\pm$ 0.36
10 | 13.6 $\pm$ 0.47 | 16.89 $\pm$ 0.15 | 17.31 $\pm$ 0.31 | 14.97 $\pm$ 0.38 | 20.11 $\pm$ 0.16
50 | 20.12 $\pm$ 0.30 | 25.33 $\pm$ 0.31 | 26.49 $\pm$ 0.32 | 23.30 $\pm$ 0.18 | 28.16 $\pm$ 0.45

  * •

Whole training set performances are: 85.95 $\pm$ 0.09 on CIFAR-10, 56.69 $\pm$
0.18 on CIFAR-100 and 39.83 $\pm$ 0.41 on TinyImageNet with DSA augmentation.

###  A.3 Transferability

Besides Figure 3 shown previously which includes the case when IPC equals 10,
here we show Figure 5 that contains IPC 1 and 50 for CIFAR-10, CIFAR-100,
TinyImageNet. We also include all the numerical results in Table 6 for
reader’s references.

(a) CIFAR10 IPC 1

(b) CIFAR100 IPC 1

(c) TinyImageNet IPC 1

(d) CIFAR10 IPC 50

(e) CIFAR100 IPC 50

(f) TinyImageNet IPC 50

Figure 5: Condensation method transferability for IPC 1 and 50 Table 6:
Transferability of different methods using 4 different networks with IPC 1, 10
and 50. All results are evaluated with DSA augmentation.

Dataset | Method | IPC | Network
---|---|---|---
ConvNet | MLP | ResNet18 | ResNet152
CIFAR-10 | Random | 1 | 15.40 $\pm$ 0.28 | 14.37 $\pm$ 0.38 | 16.56 $\pm$ 0.46 | 12.15 $\pm$ 1.80
10 | 31.00 $\pm$ 0.48 | 25.08 $\pm$ 0.27 | 29.52 $\pm$ 0.87 | 15.84 $\pm$ 0.91
50 | 50.55 $\pm$ 0.32 | 35.21 $\pm$ 0.44 | 47.26 $\pm$ 0.27 | 23.36 $\pm$ 2.31
Kmeans-emb | 1 | 25.16 $\pm$ 0.45 | 24.01 $\pm$ 0.32 | 25.99 $\pm$ 0.57 | 14.64 $\pm$ 1.30
10 | 41.49 $\pm$ 0.73 | 32.92 $\pm$ 0.38 | 40.08 $\pm$ 0.88 | 19.35 $\pm$ 0.71
50 | 56.00 $\pm$ 0.29 | 40.61 $\pm$ 0.34 | 52.69 $\pm$ 0.70 | 27.84 $\pm$ 1.07
DC | 1 | 29.34 $\pm$ 0.37 | 29.02 $\pm$ 0.52 | 27.43 $\pm$ 0.71 | 15.31 $\pm$ 0.36
10 | 50.99 $\pm$ 0.62 | 34.06 $\pm$ 0.40 | 43.96 $\pm$ 1.37 | 16.51 $\pm$ 0.89
50 | 56.81 $\pm$ 0.44 | 31.63 $\pm$ 0.55 | 45.94 $\pm$ 1.41 | 17.98 $\pm$ 1.06
DSA | 1 | 27.76 $\pm$ 0.47 | 25.04 $\pm$ 0.77 | 25.59 $\pm$ 0.56 | 15.12 $\pm$ 0.65
10 | 52.96 $\pm$ 0.41 | 34.49 $\pm$ 0.47 | 42.11 $\pm$ 0.56 | 16.10 $\pm$ 1.03
50 | 60.28 $\pm$ 0.37 | 41.01 $\pm$ 0.36 | 49.52 $\pm$ 0.72 | 19.65 $\pm$ 1.16
DM | 1 | 26.45 $\pm$ 0.39 | 10.02 $\pm$ 0.55 | 20.64 $\pm$ 0.47 | 14.09 $\pm$ 0.58
10 | 47.64 $\pm$ 0.55 | 34.44 $\pm$ 0.30 | 38.21 $\pm$ 1.05 | 15.60 $\pm$ 1.51
50 | 61.99 $\pm$ 0.33 | 40.49 $\pm$ 0.38 | 52.76 $\pm$ 0.44 | 21.67 $\pm$ 1.34
TM | 1 | 44.19 $\pm$ 1.18 | 10.40 $\pm$ 0.48 | 34.17 $\pm$ 1.41 | 13.40 $\pm$ 0.86
10 | 63.66 $\pm$ 0.38 | 30.77 $\pm$ 0.60 | 45.22 $\pm$ 1.37 | 16.61 $\pm$ 1.37
50 | 70.28 $\pm$ 0.61 | 38.45 $\pm$ 0.27 | 59.96 $\pm$ 0.72 | 20.90 $\pm$ 1.60
CIFAR-100 | Random | 1 | 5.30 $\pm$ 0.23 | 4.27 $\pm$ 0.09 | 4.36 $\pm$ 0.15 | 1.73 $\pm$ 0.12
10 | 18.64 $\pm$ 0.25 | 10.20 $\pm$ 0.18 | 15.77 $\pm$ 0.24 | 5.19 $\pm$ 0.46
50 | 34.66 $\pm$ 0.41 | 16.80 $\pm$ 0.31 | 30.23 $\pm$ 0.61 | 18.55 $\pm$ 1.29
Kmeans-emb | 1 | 10.89 $\pm$ 0.17 | 7.96 $\pm$ 0.17 | 8.75 $\pm$ 0.43 | 2.22 $\pm$ 0.19
10 | 25.04 $\pm$ 0.30 | 13.92 $\pm$ 0.20 | 22.18 $\pm$ 0.59 | 7.14 $\pm$ 0.79
50 | 38.64 $\pm$ 0.43 | 19.32 $\pm$ 0.36 | 34.00 $\pm$ 0.51 | 21.25 $\pm$ 1.46
DC | 1 | 13.66 $\pm$ 0.29 | 9.78 $\pm$ 0.27 | 9.71 $\pm$ 0.46 | 2.67 $\pm$ 0.16
10 | 28.42 $\pm$ 0.29 | 12.36 $\pm$ 0.20 | 17.94 $\pm$ 0.59 | 5.28 $\pm$ 1.05
50 | 30.56 $\pm$ 0.56 | 13.29 $\pm$ 0.30 | 17.64 $\pm$ 0.31 | 11.36 $\pm$ 0.95
DSA | 1 | 13.73 $\pm$ 0.45 | 10.56 $\pm$ 0.22 | 9.95 $\pm$ 0.55 | 2.95 $\pm$ 0.44
10 | 32.23 $\pm$ 0.35 | 16.17 $\pm$ 0.26 | 21.86 $\pm$ 0.43 | 5.45 $\pm$ 1.04
50 | 43.13 $\pm$ 0.33 | 21.42 $\pm$ 0.31 | 34.34 $\pm$ 0.44 | 20.79 $\pm$ 1.76
DM | 1 | 11.20 $\pm$ 0.27 | 8.17 $\pm$ 0.21 | 5.36 $\pm$ 0.31 | 2.11 $\pm$ 0.13
10 | 29.23 $\pm$ 0.26 | 14.68 $\pm$ 0.18 | 18.72 $\pm$ 0.49 | 3.91 $\pm$ 0.73
50 | 42.32 $\pm$ 0.37 | 20.14 $\pm$ 0.24 | 33.34 $\pm$ 0.40 | 17.29 $\pm$ 2.41
TM | 1 | 22.3 $\pm$ 0.55 | 8.69 $\pm$ 0.33 | 13.32 $\pm$ 1.29 | 2.54 $\pm$ 0.11
10 | 38.18 $\pm$ 0.42 | 14.35 $\pm$ 0.24 | 26.78 $\pm$ 0.58 | 5.74 $\pm$ 0.60
50 | 46.32 $\pm$ 0.26 | 21.92 $\pm$ 0.27 | 41.08 $\pm$ 0.29 | 27.87 $\pm$ 1.99
TinyImageNet | Random | 1 | 1.65 $\pm$ 0.11 | 1.37 $\pm$ 0.08 | 1.27 $\pm$ 0.08 | 0.63 $\pm$ 0.08
10 | 6.88 $\pm$ 0.25 | 3.12 $\pm$ 0.13 | 3.34 $\pm$ 0.16 | 1.01 $\pm$ 0.15
50 | 18.62 $\pm$ 0.22 | 5.28 $\pm$ 0.2 | 10.35 $\pm$ 0.33 | 2.90 $\pm$ 0.40
Kmeans-emb | 1 | 3.03 $\pm$ 0.12 | 2.53 $\pm$ 0.13 | 2.29 $\pm$ 0.10 | 0.82 $\pm$ 0.10
10 | 11.38 $\pm$ 0.26 | 4.73 $\pm$ 0.08 | 5.46 $\pm$ 0.24 | 1.55 $\pm$ 0.21
50 | 22.02 $\pm$ 0.40 | 5.99 $\pm$ 0.17 | 13.51 $\pm$ 0.34 | 3.91 $\pm$ 0.49
DC | 1 | 5.27 $\pm$ 0.10 | 2.67 $\pm$ 0.17 | 3.17 $\pm$ 0.21 | 0.90 $\pm$ 0.14
10 | 12.83 $\pm$ 0.14 | 4.12 $\pm$ 0.11 | 5.44 $\pm$ 0.21 | 1.24 $\pm$ 0.18
50 | 12.66 $\pm$ 0.36 | 3.81 $\pm$ 0.17 | 7.05 $\pm$ 0.21 | 2.39 $\pm$ 0.21
DSA | 1 | 5.67 $\pm$ 0.14 | 3.90 $\pm$ 0.16 | 3.20 $\pm$ 0.13 | 0.84 $\pm$ 0.12
10 | 16.34 $\pm$ 0.21 | 6.31 $\pm$ 0.21 | 7.60 $\pm$ 0.36 | 1.90 $\pm$ 0.21
50 | 25.31 $\pm$ 0.22 | 6.72 $\pm$ 0.20 | 13.36 $\pm$ 0.40 | 3.78 $\pm$ 0.56
DM | 1 | 3.82 $\pm$ 0.21 | 3.11 $\pm$ 0.10 | 1.79 $\pm$ 0.17 | 0.89 $\pm$ 0.07
10 | 13.51 $\pm$ 0.31 | 4.24 $\pm$ 0.13 | 3.57 $\pm$ 0.20 | 1.06 $\pm$ 0.19
50 | 22.76 $\pm$ 0.28 | 5.74 $\pm$ 0.27 | 11.07 $\pm$ 0.39 | 3.33 $\pm$ 0.46
TM | 1 | 8.27 $\pm$ 0.36 | 3.33 $\pm$ 0.13 | 4.45 $\pm$ 0.23 | 0.95 $\pm$ 0.09
10 | 20.11 $\pm$ 0.16 | 4.59 $\pm$ 0.20 | 10.16 $\pm$ 0.43 | 1.74 $\pm$ 0.21
50 | 28.16 $\pm$ 0.45 | 6.04 $\pm$ 0.20 | 20.65 $\pm$ 0.44 | 4.10 $\pm$ 0.52

###  A.4 Kmeans-emb

In Section 2.2.2, we propose Kmeans-emb that achieves competitive performances
compared to all previous selection based method. We show the flow of it in
Algorithm 2

Algorithm 2 Embedding based Kmeans Coreset Selection

Randomly initialize a ConvNet model M and train for 1 single epoch

Compute the embedding features for each image using model M

for i in 1 .. N do

Select all the images belonging to class i and assign them to $S_{i}$

Randomly select P images from $\textbf{S}_{i}$ and assign to $\textbf{C}_{i}$
as the centers for KMmeans.

while $j\leq\textbf{K}$ do

update $\textbf{C}_{i}$ based on the $L_{2}$ distance in the embedding space.

end while

Based on the center $\textbf{C}_{i}$, get the nearest N images using $L_{2}$
distance in the embedding space

end for

Return {$\textbf{C}_{i}$ | i = 1..N}

###  A.5 Combining selection based methods with synthesis based methods

As shown in Figure 4 for DM, if we combine synthesis based method with better
selection method, it will not only converge faster but also achieve better
performance. Similar results from DC and DSA can be seen from Figure 6 on
dataset CIFAR-10. We are not able to get the performance of TM under the
computation resource limit we set.

(a) DC IPC 1

(b) DC IPC 10

(c) DC IPC 50

(d) DSA IPC 1

(e) DSA IPC 10

(f) DSA IPC 50

Figure 6: Test accuracy comparison between initializing synthetic images with
Random selection, Kmeans-emb and Gaussian noise on CIFAR-10

###  A.6 Different compression ratios

Previously, we show the condensation method performances in Figure 2 with IPC
up to 1000, here we show the numeric results in Table 7 for easier references.

Table 7: Test accuracy of condensation methods under different IPCs. All
numbers are recorded on ConvNet and CIFAR-10 dataset with DSA augmentation.

IPC | Method
---|---
Random | Kmeans-emb | DC | DSA | DM
1 | 15.40 $\pm$ 0.28 | 25.16 $\pm$ 0.45 | 29.34 $\pm$ 0.37 | 27.76 $\pm$ 0.47 | 26.45 $\pm$ 0.39
10 | 31.00 $\pm$ 0.48 | 41.49 $\pm$ 0.73 | 50.99 $\pm$ 0.62 | 52.96 $\pm$ 0.41 | 47.64 $\pm$ 0.55
50 | 50.55 $\pm$ 0.32 | 56.00 $\pm$ 0.29 | 56.81 $\pm$ 0.44 | 60.28 $\pm$ 0.37 | 61.99 $\pm$ 0.33
100 | 57.89 $\pm$ 0.57 | 62.18 $\pm$ 0.18 | 65.70 $\pm$ 0.44 | 66.18 $\pm$ 0.30 | 65.12 $\pm$ 0.40
200 | 64.70 $\pm$ 0.44 | 67.25 $\pm$ 0.48 | 68.41 $\pm$ 0.45 | 69.49 $\pm$ 0.13 | 69.15 $\pm$ 0.17
300 | 68.52 $\pm$ 0.29 | 70.12 $\pm$ 0.09 | 69.42 $\pm$ 0.45 | 71.46 $\pm$ 0.27 | 69.36 $\pm$ 0.35
400 | 70.28 $\pm$ 0.39 | 71.88 $\pm$ 0.34 | 70.86 $\pm$ 0.42 | 72.22 $\pm$ 0.48 | 72.61 $\pm$ 0.54
500 | 73.19 $\pm$ 0.29 | 74.31 $\pm$ 0.33 | 72.05 $\pm$ 0.35 | 73.62 $\pm$ 0.28 | 75.09 $\pm$ 0.26
600 | 74.00 $\pm$ 0.29 | 75.98 $\pm$ 0.19 | 72.84 $\pm$ 0.40 | 74.99 $\pm$ 0.21 | 76.07 $\pm$ 0.20
700 | 75.29 $\pm$ 0.25 | 76.74 $\pm$ 0.34 | 73.70 $\pm$ 0.31 | 76.07 $\pm$ 0.26 | 76.78 $\pm$ 0.27
800 | 75.52 $\pm$ 0.10 | 76.94 $\pm$ 0.21 | 74.80 $\pm$ 0.29 | 76.76 $\pm$ 0.27 | 77.41 $\pm$ 0.19
900 | 77.44 $\pm$ 0.45 | 78.21 $\pm$ 0.20 | 75.26 $\pm$ 0.34 | 77.77 $\pm$ 0.44 | 78.33 $\pm$ 0.41
1000 | 78.38 $\pm$ 0.20 | 79.47 $\pm$ 0.29 | 76.62 $\pm$ 0.32 | 78.68 $\pm$ 0.25 | 78.83 $\pm$ 0.05

###  A.7 Impact of different optimizers

In Table 8, we show the results of using SGD optimizer versus Adam optimizer
with all IPCs(1, 10, 50) on CIFAR-10. All performances are evaluated with DSA
augmentation. We get similar performances with the 2 different optimizers.

Table 8: Test accuracy of different condensation methods with SGD and Adam
optimizer on CIFAR-10. All results are evaluated with DSA augmentation.

Optimizer | IPC | Random | Kmeans-emb | DC | DSA | DM | TM
---|---|---|---|---|---|---|---
SGD | 1 | 15.40 $\pm$ 0.28 | 25.16 $\pm$ 0.45 | 29.34 $\pm$ 0.37 | 27.76 $\pm$ 0.47 | 26.45 $\pm$ 0.39 | 44.19 $\pm$ 1.18
10 | 31.00 $\pm$ 0.48 | 41.49 $\pm$ 0.73 | 50.99 $\pm$ 0.62 | 52.96 $\pm$ 0.41 | 47.64 $\pm$ 0.55 | 63.66 $\pm$ 0.38
50 | 50.55 $\pm$ 0.32 | 56.00 $\pm$ 0.29 | 56.81 $\pm$ 0.44 | 60.28 $\pm$ 0.37 | 61.99 $\pm$ 0.33 | 70.28 $\pm$ 0.61
Adam | 1 | 15.39 $\pm$ 0.38 | 25.40 $\pm$ 1.21 | 28.76 $\pm$ 0.71 | 27.83 $\pm$ 1.27 | 26.22 $\pm$ 0.40 | 46.62 $\pm$ 1.39
10 | 32.71 $\pm$ 0.92 | 43.26 $\pm$ 0.41 | 52.01 $\pm$ 0.62 | 51.17 $\pm$ 0.91 | 48.64 $\pm$ 1.02 | 64.94 $\pm$ 0.61
50 | 49.75 $\pm$ 1.10 | 54.82 $\pm$ 1.23 | 54.75 $\pm$ 1.52 | 58.73 $\pm$ 0.96 | 61.13 $\pm$ 0.92 | 71.82 $\pm$ 0.52

###  A.8 Computation resources

In order to perform a fair comparison, we run all the experiments on the same
virtual machine with 1 NVIDIA A100 GPU with 40GB GPU memory on Google Cloud.
If some method is not able to run on the machine within GPU memory limit, we
report it as out-of-memory(OOM).

###  A.9 Training time

Here we show the training time of each condensation methods in Table 9. For
Kmeans-emb, we iterate 300 times to find the cluster center. The run time is
computed by dividing the total time by 300. For synthesis based methods, we
run each method 100 epochs and report the mean and standard deviation of the
data.

Table 9: Training time of different condensation methods. All results for
synthesis based methods are acquired by running 100 iterations and all results
for Kmeans-emb are acquired by performing 300 iterations on CIFAR-10 with IPC
1, 10, 50. The average result and standard deviation per iteration are
reported for each IPC

Method | run time(sec) | GPU memory(MB)
---|---|---
1 | 10 | 50 | 1 | 10 | 50
Kmeans-emb | 0.0012 $\pm$ 0.00 | 0.015 $\pm$ 0.00 | 0.05 $\pm$ 0.00 | 3575 | 3575 | 3575
DC | 0.16 $\pm$ 0.01 | 3.31 $\pm$ 0.02 | 15.74 $\pm$ 0.10 | 3515 | 3621 | 4527
DSA | 0.22 $\pm$ 0.02 | 4.47 $\pm$ 0.12 | 20.13 $\pm$ 0.58 | 3513 | 3639 | 4539
DM | 0.08 $\pm$ 0.02 | 0.08 $\pm$ 0.02 | 0.08 $\pm$ 0.02 | 3323 | 3455 | 3605
TM | 0.36 $\pm$ 0.23 | 0.40 $\pm$ 0.20 | OOM | 2711 | 8049 | OOM

  * •

Note: different methods need different iterations to converge, this table
shows the run time and memory needed per iteration which reflect the
bottleneck of the algorithm.

###  A.10 Example real images and synthetic images

Here we show the selected real images generated by Random selection, Kmeans-
emb and synthetic images generated by DC, DSA, DM and TM in Figure 7 for
reader’s references.

(a) Randomly selected images

(b) Kmeans-emb selected images

(c) Synthetic images by DC

(d) Synthetic images by DSA

(e) Synthetic images by DM

(f) Synthetic images by TM

Figure 7: Real images vs synthetic images

###  A.11 Source code and leaderboard

We open source our benchmark and evaluation library at
https://github.com/justincui03/dc_benchmark as a contribution to the research
community. At the same time, we build a leaderboard
https://justincui03.github.io/dcbench/ to track the most up-to-date progress
in the field. All these will be maintained regularly and we will keep updating
the benchmark to reflect the newest changes.

###  A.12 Assets license

Code License:Our codebase is open sourced under the MIT license.
Dataset License: The datasets(CIFAR10/100, TinyImageNet) used in this paper
are not part of our assets, if readers are going to use the datasets, please
follow the instructions below

  * •

CIFAR10/100 Please refer to requirements for usage for these 2 datasets.

  * •

TinyImageNet Please refer to term of access for using TinyImageNet.