# Extrapolating from a Single Image to a Thousand Classes using Distillation

Yuki M. Asano11footnotemark: 1
Video & Image Sense Lab
University of Amsterdam
Aaqib Saeed11footnotemark: 1
Dept. of Computer Science and Technology
University of Cambridge

###### Abstract

What can neural networks learn about the visual world from a single image?
While it obviously cannot contain the multitudes of all existing objects,
scenes and lighting conditions – within the space of all $256^{3\cdot 224\cdot
224}$ possible $224$-sized square images, it might still provide a strong
prior for natural images. To analyze this hypothesis, we develop a simple
framework for training neural networks from scratch using a single image and
knowledge distillation from a supervised pretrained teacher. With this, we
find the answer to the above question to be: ‘surprisingly, a lot’. In
quantitative terms, we find accuracies of $94\%$/$74\%$ on CIFAR-10/100, $66$%
on ImageNet, and by extending this method to video and audio, $51\%$ on
Kinetics-400 and $84$% on SpeechCommands. In extensive analyses we disentangle
the effect of augmentations, choice of data and network architectures and also
discover “panda neurons” in networks that have never even seen one. This work
demonstrates that one image can be used to extrapolate to thousands of object
classes and motivates research on the fundamental interplay of augmentations
and images.222https://single-image-distill.github.io/

∗Equal contribution<EMAIL_ADDRESS><EMAIL_ADDRESS>
Figure 1: Extrapolating from one image. Strongly augmented patches from a
single image are used to train a student (S) to distinguish semantic classes,
such as those in ImageNet. The student neural network is initialized randomly
and learns from a pretrained teacher (T) via KL-divergence. Although almost
none of target categories are present in the image, we find student
performances of $>\\!66\%$ for classifying ImageNet’s 1000 classes. In this
paper, we develop this single datum learning framework and investigate it
across datasets and domains.

###  1 Introduction

Deep learning has both relied and improved significantly with the increase in
dataset sizes. In turn, there are many works that show the benefits of dataset
scale in terms of data points and modalities used. Within computer vision,
these models trained on ever larger datasets, such as Instagram-1B [44] or
JFT-3B [20], have been shown to successfully distinguish between semantic
categories at high accuracies. In stark contrast to this, there is little
research on understanding neural networks trained on very small datasets. Why
would this be of any interest? While smaller dataset sizes allow for better
understanding and control of what the model is being trained with, we are most
interested in its ability to provide insights into fundamental aspects of
learning: For example, it is an open question as to what exactly is required
for arriving at semantic visual representations from random weights, and also
of how well neural networks can extrapolate from their training distribution.

While for visual models it has been established that few or no real images are
required for arriving at basic features, like edges and color-contrasts [1,
32, 10, 51], we go far beyond these and instead ask what the minimal data
requirements are for neural networks to learn semantic categories, such as
those of ImageNet. This approach is also motivated by studies that investigate
the early visual development in infants, which have shown how little visual
diversity babies are exposed to in the first few months whilst developing
generalizeable visual systems [53, 3]. In this paper, we study this question
in its purest form, by going to one extreme and analyzing whether neural
networks can learn to extrapolate from a single datum.

However, addressing this question naïvely runs into the difficulties of i)
current deep learning methods, such as SGD or BatchNorm being tailored to
large datasets and not working with a single datum and ii) extrapolating to
semantic categories requiring information about the space of natural images
beyond the single datum. In this paper, we address these issues by developing
a simple framework that recombines augmentations and knowledge distillation.

First, augmentations can be used to generate a large number of variations from
a single image. This can effectively address issue i) and allow for evaluating
the research question on standard architectures and datasets. This use of data
augmentations to generate variety is drastically different to their usual use-
case in which transformations are generated to implicitly encode desirable
invariances during training.

Second, to tackle the difficulty of providing information about the “rest of
the world” in the single datum setting, we opt to use the outputs of a
supervisedly trained model in a knowledge distillation (KD) fashion. While KD
[31] is originally proposed for improving small models’ performance by
leveraging what larger models have learned, we re-purpose this as a simple way
to provide a supervisory signal about semantic classes into the training
process.

We combine the above two ideas and provide both student and teacher only with
augmented versions of a single datum, and train the student to match the
teacher’s imagined class-predictions of classes – almost all of which are not
contained in the single datum, see Extrapolating from a Single Image to a
Thousand Classes using Distillation. While practical applications do result
from our method – for example we provide results on single image dataset based
model compression in the Appendix – our goal in this paper is analyzing the
fundamental question of how well neural networks trained from a single datum
can extrapolate to semantic classes, like those of CIFAR, SpeechCommands,
ImageNet or even Kinetics.

What we find is that despite the fact that the resulting model has only seen a
single datum plus augmentations, surprisingly high quantitative performances
are achieved: _e.g_. $74$% on CIFAR-100, $84\%$ on SpeechCommands and $66$%
top-1, single-crop accuracy on ImageNet-12. We further make the novel
observations that our method benefits from high-capacity student and low-
capacity teacher models, and that the source datum’s characteristics matter –
random noise or less dense images yield much lower performances than dense
pictures like the one shown in Figure Extrapolating from a Single Image to a
Thousand Classes using Distillation.

In summary, in this paper we make these four main contributions:

  1. 1.

A minimal framework for training neural networks with a single datum from
scratch using distillation.

  2. 2.

Extensive ablations of the proposed method, such as the dependency on the
source image, the choice of augmentations and network architectures.

  3. 3.

Large scale empirical evidence of neural networks’ ability to extrapolate on
$>\\!12$ vision and audio datasets.

  4. 4.

Qualitative insights on what and how neural networks trained with a single
image learn.

###  2 Related Work

The work presented builds on top of insights from the topics of knowledge
distillation and single- and no-image training of visual representations and
yields insights into neural networks’ ability to extrapolate.

###### Distillation.

In knowledge distillation (KD), the goal is to train a typically lower
capacity student model from a pretrained teacher model in order to surpass the
performance of solely training with a label-supervised objective.
Specifically, KD has been explored extensively to train more performant and,
or compressed student models from the soft-target predictions of teacher
models [2, 31, 24, 22, 18]. Similarly, other approaches have also been
developed to improve transfer from teacher to student, including sharing
intermediate layers’ features [60], spatial attention transfer [83],
similarity preservation between activations of the networks [70], and
contrastive distillation [69]. KD has also been shown to be an effective
approach for learning from noisy labels [38].

More recently, [6] conducted a comprehensive empirical investigation to
identify important design choices for successful distillation from large-scale
vision models. In particular, they show that long training schedules, paired
with consistent augmentations (including MixUp) for both student and teacher,
result in better performances.

###### Without the training data.

Distilling knowledge without access to the original training dataset was
originally proposed in $2017$ [41] and first experiments of leaving out single
MNIST classes from the training data were shown in [31]. Yet, this paradigm is
gaining importance as many recent advancements have been made possible due to
extremely large proprietary datasets that are kept private. This has lead to
either the sole release of trained models [56, 23, 44] or even more restricted
access to only the model outputs via APIs [9]. While the original work [41],
still required activation statistics from the training dataset of the network,
more recent works do not require this “meta-data”. These approaches are
typically GAN based [14, 80, 45], for example generating datasets of synthetic
images that maximally activate neurons in the final layer of teacher.
Similarly, in [81], features of pretrained networks are inverted, a large
dataset generated and then used for distillation with high accuracies.

Related to this, there are works which conduct “dataset” distillation, where
the objective is to distill large-scale datasets into much smaller ones, such
that models trained on it reach similar levels of performance as on the
original data. These methods generate synthetic images [74, 57, 40, 86, 11],
labels [8], or both [46], but due to their meta-learning nature have only been
applied to small-scale datasets, such as MNIST or CIFAR. Finally, the ability
to copy models based on API outputs while having access to a dataset with only
a partial overlap to the original one has been explored in [52, 73], though
[52] relies on pretrained models.

In contrast to GAN- and inversion-based methods, as well as to dataset
distillation, our approach does not require the knowledge of the weights and
architecture of the teacher model, and instead works with black-box
“API”-style teacher models and much smaller “datasets” of just a single datum
plus augmentations. Compared to model copying, we do not assume access to any
meaningfully similar training dataset, nor start with a pretrained model.
Compared to both, this paper follows a radically different goal: We are
instead interested in analysing the potential of extrapolating from a single
image to the manifold of natural images, for which we choose KD as a well-
suited and simple tool.

###### Neural network extrapolation.

Work on extrapolation behavior mostly focuses on MLPs [25, 5, 30] and GNNs
[79]. For vision models, only model generalization behavior is analysed,
_e.g_., in [58, 88, 61], however this confounds training data similarity with
re-usability of learned features. In this work, we instead investigate the
extrapolation performance of vision models in isolation – to the best of our
knowledge, for the very first time.

###### Single and no-image training of visual representations.

Several prior works propose to learn representations with a single training
example, such as a single image, while focusing on specific tasks, such as
super resolution or inpainting [71, 63], generative modelling [62] and
tracking [59]. In self-supervised learning, a single image was shown to be
enough for pretraining low-level visual representations when combined with
strong data augmentations [1]. Related to this, [32] pretrained deep visual
models with images of fractals and [4] with uninitialized GAN outputs. While
these lines of work provide a compelling argument and motivation for this
paper by demonstrating the potential of augmentations and limited data, our
work is both complementary and goes beyond unsupervised pretraining by showing
that KD allows us to learn semantic categories, such as those of ImageNet,
from one image.

###  3 Method

We believe that simplicity is the key for demonstrating and analysing the
question of how far a single image can take us. Any bells-and-whistles will
surely improve performance, but would distract from the science. Inspired by
recent work of Asano _et al_. [1], which “patchifies” a single-image using
augmentations, and the knowledge distillation method presented by Beyer _et
al_. [6], our technical contribution lies in unifying them to develop a
single-image distillation framework.

###### i. Dataset generation.

In [1], a single “source” image is augmented many times to generate a static
dataset of fixed size. This is done by applying the following augmentations in
sequence: cropping, rotation and shearing, and color jittering. For this, we
follow their official
implementation333https://github.com/yukimasano/single_img_pretraining, and do
not change any hyperparameters. We do however analyze the choice of source
images more thoroughly by additionally including a random noise, and a Hubble-
telescope image to our analysis. In addition, we also conduct experiments on
audio classification. For generating a dataset of augmented audio-clips, we
apply the set of audio-augmentations from [7], which consist of operations,
such as random volume increasing, background noise addition and pitch shifting
to yield log-Mel spectrograms from raw waveforms (see Appendix for complete
details).

###### ii. Knowledge distillation.

The knowledge distillation objective is proposed in [31] to transfer the
knowledge of a pretrained teacher to a lower capacity student model. In this
case, the optimization objective for the student network is a weighted
combination of dual losses: a standard supervised cross-entropy loss and a
“distribution-matching” objective that aims to mimic the teacher’s output.
However, in our case there are no class-labels for the patches generated from
a single image, so we solely use the second objective formulated as a
Kullback–Leibler (KL) divergence between the student output $p^{s}$ and the
teacher’s output $p^{t}$:

$\displaystyle\mathcal{L}_{\text{KL}}=\sum_{c~{}\in~{}\mathcal{C}}-p^{t}_{c}\log
p^{s}_{c}+p^{t}_{c}\log p^{t}_{c}$ (1)

where $c$ are the teachers’ classes and the outputs of both student and
teacher are temperature-flattened probabilities, $p=\texttt{softmax}(l/\tau)$,
that are generated from logits $l$ and a temperature $\tau$.

For training, we follow [6] in employing a function matching strategy, where
the teacher and student models are fed consistently augmented instances, that
include heavy augmentations, such as MixUp [85] or CutMix [82]. However, in
contrast to [6], we neither have access to TPUs nor can train $10$K epochs on
ImageNet-sized datasets. While both of these would likely improve the
quantitative results, we believe that this handicap is actually blessing in
disguise: This means that the results we show in this paper are not specific
to heavy-compute, or extremely large batch-sizes, but instead are fundamental.

###  4 Experiments

####  4.1 Implementation details

###### Source data.

For the source data, we utilize the single images of [1], except for one
image, which we replace by a similar one as we could not retrieve its licence.
These images are of sizes up to 2,560x1,920 (the “City” image of Extrapolating
from a Single Image to a Thousand Classes using Distillation. In addition, we
collect one satellite image of an airport for testing performance in a
different remote-sensing domain. For the audio experiments, we use two short
audio-clips from Youtube, a $5$mins BBC newsclip, as well as a $5$mins clip
showing $11$ Germanic languages. Images and audio visualizations, sources and
licences are provided in Appendix A.

###### Tasks.

For simplicity, we focus on classification tasks here and provide results on a
sample application of single-image based model compression in Section B.10.
For our ablations and small-scale experiments we focus on CIFAR-10/100 [35]
and also on the different domain of satellite image classification with the
EuroSAT dataset [28]. For larger-scale experiments using $224\\!\times\\!224$
sized images we evaluate our method on datasets with varying number of classes
(see Table 8): STL-10 [16], Pets [55], ImageNet-100 [69], Flowers [48],
Places365 [87] and finally ImageNet LSVRC12 (IN-1k) [19]. For our audio
experiments in Table 7, we use MUSAN [65], Voxforge [43], SpeechCommands [75],
and LibriSpeech [54] datasets for varying complexity and categories of sounds
to be recognized. For our experiments on learning video models from a single
image we use the Kinetics-400 [13] and UCF-101 [66] datasets.

###### Optimization.

For each of these datasets, we first train a teacher network in a usual
supervised manner, that is then used as the teacher for the distillation. For
the distillation to the student model, we use a temperature $\tau=8$ motivated
by findings in [6]. We keep this temperature fixed throughout the whole of the
paper due to limited compute. For optimization, we use AdamW [42]. Further
implementation details are provided in Appendix C.

####  4.2 Ablation

Distillation dataset |  | Accuracy
---|---|---
Name | # Images | # Pixels | Size (MB) |  | C10 | C100
CIFAR-10 | 50,000 | 51M | 145 |  | 95.26 | 76.29
CIFAR-100 | 50,000 | 51M | 145 |  | 94.51 | 78.06
CIFAR-10 | 10,000 | 10M | 29 |  | 94.58 | 72.95
Ours | 1 | 2.8M | 0.27 |  | 94.14 | 73.80
Table 1: Distilling dataset. 1 image + augmentations $\approx$ almost 50K in-
domain CIFAR-10/100 images. Here, both teacher and student are WideR40-4
networks.

###### 1-image vs full training set.

We first examine the capability to extrapolate from one image to small-scale
datasets, such as CIFAR-10 and CIFAR-100. In Table 1, we compare various
datasets for distilling a teacher model into a student. We find that while
distillation using the source dataset always works best ($95.26\%$ and
$78.06\%$) on CIFAR-10/100, using a single image can yield models which almost
reach this upper bar ($94.1\%$ and $73.8\%$). Moreover, we find that one image
distillation even outperforms using $10$K images of CIFAR-10 when teaching
CIFAR-100 classes even though these two datasets are remarkably similar. To
better understand why using a single image works, we next disentangle the
various components used in the training procedure: (a) the source image, (b)
the generated image dataset size and (c) the augmentations used during
distillation, corresponding to Tables 2(a), 2(b) and 2(c).

Distillation dataset | Accuracy
---|---
Name | # Images | # Pixels | C10 | C100
Image “Noise” | 1 | 4.9M | 69.30 | 19.50
Image “Universe” | 1 | 4.8M | 88.18 | 39.68
Image “Bridge” | 1 | 1.1M | 92.24 | 57.87
Image “City” | 1 | 4.9M | 93.13 | 64.85
Image “Animals” | 1 | 2.8M | 93.28 | 66.12
(a) Distilling image. Content matters.

Training | Accuracy
---|---
Dataset size | Epochs | C10 | C100
500 | 100,000 | 92.31 | 58.95
5K | 10,000 | 93.01 | 63.84
50K | 1,000 | 93.28 | 66.12
500K | 100 | 93.52 | 67.05
(b) Source dataset size. Larger initial pool better.

Augmentations | Accuracy
---|---
Hflip. | RCrop. | MixUp | CutMix | C10 | C100
✓ | ✓ |  |  | 89.34 | 55.05
✓ |  | ✓ |  | 91.03 | 58.24
✓ |  |  | ✓ | 92.86 | 64.26
✓ | ✓ | ✓ |  | 92.41 | 63.50
✓ | ✓ |  | ✓ | 93.32 | 68.69
(c) Varying Augmentation. More helps.

Table 2: Single image distillation. We analyze key components in our
experimental setup. We report top-1 accuracies for CIFAR-10/100. For the
teacher we use WideR40-4 and for the student a WideR16-4 and train for 1K
epochs.

###### 1-image training parameters.

From Table 2(a), we find that the choice of source image content is crucial:
Random noise or sparser images perform significantly worse compared to the
denser “City” and “Animals” pictures. This is in contrast to self-supervised
pretraining analyzed in [1], suggesting that the underlying mechanisms are
different. Next, from Table 2(b), we find that adding more variation to the
initial set of crops improves performance in a progressive manner, where the
first $10$x in the number of initial crops adds $4.9$% in CIFAR-100
performance and the last one only adds 0.9%. While out of scope for this
paper, this suggest that there is still room to optimize the initial set of
crops used to generate this dataset. Finally in Table 2(c), we ablate the
augmentation we use during knowledge distillation. Besides observing the
general trend of “more augmentations are better”, we find that CutMix performs
better than MixUp on our single-image distillation task. This might be because
in our case the model is tasked with learning how to extrapolate towards real
datapoints, while MixUp is derived and therefore might be more useful for
interpolating between (real) data samples [85, 6].

In the Table 3, we also show that even if the student can learn even with much
degraded learning signals. For example even if it receives only the top-5
predictions or solely the argmax prediction (_i.e_. a hard label) of the
teacher without any confidence value, the student is still able to extrapolate
at a significant level ($>91\%/60\%$ for CIFAR-10/100). This also suggests
that copying models solely from API outputs is possible, similar to [52]. We
also evaluated L1 and L2 distillation functions (see Appendix Table 14 for
details) and found that these perform slightly worse. This echoes the finding
of [69] that standard KD loss of Eq. 1 is actually a strong baseline hence use
this in the rest of our experimental analysis.

| Full output | Top-5 | Argmax
---|---|---|---
C10 Accuracy | 93.32 | 92.98 | 91.89
C100 Accuracy | 68.69 | 64.72 | 60.75
Table 3: Teacher signal. Even with only top-5 predictions (class & probability) or hard distillation, performance only slightly degrades. Same setup as in Table 2 CIFAR-10 | Fractals [32] | StyleGAN [4] | ZeroSKD [45] | 1-Image (Ours)
---|---|---|---|---
92.61 | 33.26 | 83.42 | 86.60 | 89.27
Table 4: Comparison against other datasets on CIFAR-10. Here the teacher model
is WideR40-2 and student is WideR40-1 for comparability to [45].

###### 1-image vs other datasets.

In Table 4, we find that there is something unique about using a single image,
as our method outperforms several synthetic datasets, as well as the GAN-based
approach of [45]. This is despite the fact that these synthetic datasets
contain $\sim\\!50$K images. We provide insights on the characteristics of
this 1-image dataset in Section 4.3.3. Next, using the insights gained in this
section, we scale the experiments towards other network architectures, dataset
domains and dataset sizes.

####  4.3 Scalability

#####  4.3.1 Architectures

|  |  |  | Distillation
---|---|---|---|---
| Teacher${}_{\mathrm{\qquad Acc.}}$ | Student${}_{\mathrm{\quad\,\,\,\,Acc.}}$ |  | Full | Ours
CIFAR-$10$ |  VGG-19${}_{\,\,\quad\,\,93.28}$ |  VGG-16${}_{\,\,\,\quad 92.42}$ |  | 92.84 | 92.14
ResNet-56 93.77 |  ResNet-20 92.52 |  | 92.29 | 90.70
ResNet-110 93.55 |  ResNet-56 93.77 |  | 92.09 | 90.77
WideR40-4 95.42 |  WideR16-4 95.20 |  | 95.00 | 93.32
WideR40-4 95.42 |  WideR40-4 95.42 |  | 94.36 | 94.14
WideR16-4 95.20 |  WideR40-4 95.42 |  | 94.30 | 94.02
CIFAR-$100$ |  VGG-19${}_{\,\,\quad\,\,70.79}$ |  VGG-16${}_{\,\,\,\quad 73.26}$ |  | 71.19 | 58.66
ResNet-56 70.99 |  ResNet-20 65.74 |  | 67.04 | 52.43
ResNet-110 71.92 |  ResNet-56 70.99 |  | 70.43 | 58.13
WideR40-4 78.14 |  WideR16-4 75.56 |  | 76.26 | 68.69
WideR40-4 78.14 |  WideR40-4 78.14 |  | 75.54 | 73.80
WideR16-4 78.14 |  WideR40-4 75.56 |  | 76.29 | 74.08
Table 5: Distilling various architectures on CIFAR-10. We compare student
accuracy when distilling with full training set vs our 1-image dataset.

In Table 5, we experiment with different common architectures on CIFAR-10 and
CIFAR-100. On CIFAR-10, we find that almost all architectures perform
similarly well, except for the case of ResNet-56 to ResNet-20 which could be
attributed to the lower capacity of the student model. We find that the
distillation performance of WideR40-4 to WideR40-4 on CIFAR-10 is very close
to distilling from original source data; achieving accuracy of $94.14\%$ and
only around one-percentage point lower than supervised training. This analysis
shows that our method generalizes to other architectures hints that larger
capacity student models might be important for high performances.

#####  4.3.2 Varying data domain

Distillation dataset | Accuracy
---|---
Name | # Images | # Pixels | C100 | EuroSAT
EuroSAT | 21,600 | 88M | 9.36 | 97.85
CIFAR-100 | 50,000 | 51M | 76.26 | 52.24
Image “Animals” | 1 | 2.8M | 68.69 | 95.04
Image “Satellite-pic” | 1 | 15.7M | 47.44 | 93.46
50% “Animals’ + 50% “Satellite-pic” | 2 | 19.5M | 66.07 | 95.37
Table 6: Distilling dataset. Combining single images works across domains. The
teacher is a WideR40-4 and the student a WideR16-4 network.

We investigate how the proposed single-image distillation performs on varying
the data domain. To this end, we focus on satellite imagery classification
following the same experimental protocol as before. In Table 6, we first
report distillation performance when using EuroSAT as the source data for
CIFAR-100 and vice versa. The performance on the latter drops significantly to
$9.3\%$ when input domain changes as these datasets are largely unrelated and
EuroSAT is less diverse. Next, we perform distillation with a single-image
datasets generated from an ‘Animal’ and ‘Satellite-picture’ images. To our
surprise, we notice significantly high accuracy of $95\%$ even when using
completely unrelated data, while combining the data from varying sources
equally slightly improves the distillation performance on EuroSAT with only
two-percentage point lower accuracy on CIFAR-100. From this evaluation, we can
conclude that the our distillation framework works equally well on different
data domains, and being strictly in-domain is not necessary as long as the
image is diverse.

###### Extension to audio

|  |  | Distillation
---|---|---|---
Dataset | Categories (#Classes) | Teacher | Full | $\text{Ours}_{A}$ | $\text{Ours}_{B}$
MUSAN [65] | Generic sounds (3) | 98.76 | 96.78 | 89.85 | 96.28
Voxforge [43] | Languages (6) | 91.13 | 89.04 | 72.85 | 78.47
Sp. Commands [75] | Keywords (12) | 95.15 | 94.86 | 82.90 | 84.19
LibriSpeech [54] | Speakers (100) | 99.89 | 99.54 | 78.67 | 84.12
Table 7: Distilling audio representations. $1$ audio clip $+$ augmentations
provide stronger supervisory signal for the student to distill teacher’s
knowledge.

So far, our proposed approach has shown its success in distilling useful
knowledge for images. To further test the generalizability of our approach on
other modalities, we conduct experiments on several audio recognition tasks of
varying difficulty. We perform distillation via $50$K randomly generated short
audio clips from two (_i.e_. $\text{Ours}_{A}$ and $\text{Ours}_{B}$), $5$mins
YouTube videos (see Appendix A for more details). In Table 7, we compare the
results against the performance of the teacher model and distilling directly
using source dataset. We find that even for audio, distilling with merely a
single audio clip’s data provides enough supervisory signal to train the
student model to reach above $80\%$ accuracy in the majority of the cases. In
particular, we see significant improvement in distillation performance when a
single audio has a wide variety of sounds to boost accuracy on challenging
Voxforge dataset from $72.8\%$ to $78.4\%$. Our results on audio recognition
tasks demonstrate the modality agnostic nature of our approach and further
highlight the capability to perform knowledge distillation in the absence of
large amount of data.

#####  4.3.3 Large-scale datasets and models

Dataset | Categories (#Classes) | Teacher | 1-image + Distill
---|---|---|---
STL-10 [16] | Generic objects (10) | 96.3 | 93.9
Pets [55] | Cat & dog species (37) | 91.4 | 76.8
ImageNet-100 [68] | Objects (100) | 89.6 | 84.4
Flowers [48] | Flower types (102) | 87.9 | 81.5
Places [87] | Scenes (365) | 54.0 | 50.3
ImageNet [19] | Objects (1000) | 69.5 | 66.2
Table 8: Larger scale datasets. Teacher is ResNet-18 and student is ResNet-50.
Teacher models were finetuned from IN-1k pretrained models, students are
trained from scratch.

From this section on, we scale our experiments to larger models, and large-
scale vision datasets. We show our results in Table 8. We find that, overall,
using the setting in Table 8, a single image is not enough to recover the full
performance on these more difficult datasets. This might be to a large degree
because of the fine-grained nature of these datasets, _e.g_. ImageNet includes
more than $120$ kinds of Extrapolating from a single image to these very
similar categories might require a more diverse source image. Nevertheless, we
find a surprisingly high accuracy of $66\%$ on ImageNet’s validation set, even
though this dataset comprises $1000$ classes, and the student only having seen
heavily augmented crops of a single image. In Table 9, we conduct further
analyses on distilling ImageNet models.

Table 9: Analysis of IN-1k model distillations. We vary distillation dataset and teacher and student configurations. ‘-’ refers to experiments not run as their compute cost would outweigh the insights gained. ‘⋆’ refers to a model trained with many augmentations including RandAugment from [76]. ‘†’ refers to models trained with half the batch size due to memory constraints  |  | Setting | Epochs
---|---|---|---
| Images | Teacher |  | Student | 10 | 20 | 30 | 50 | 200
(a) | 10 | R50 | $\rightarrow$ | R50 | 7.6 | 13.3 | - | - | -
(b) | 10 patchified | R50 | $\rightarrow$ | R50 | 17.2 | 27.3 | - | - | -
(c) | 100 | R50 | $\rightarrow$ | R50 | 40.5 | 52.3 | 57.8 | - | -
(d) | 100 patchified | R50 | $\rightarrow$ | R50 | 23.6 | 36.1 | 42.8 | - | -
(e) | 1000 | R50 | $\rightarrow$ | R50 | 39.8 | 51.8 | 58.4 | 61.9 | -
(f) | 1000 patchified | R50 | $\rightarrow$ | R50 | 27.1 | 39.4 | 45.2 | 50.9 | -
(g) | City Image | R101 | $\rightarrow^{\dagger}$ | R101 | 22.4 | 31.4 | - | - | -
(h) | City Image | WR50-2 | $\rightarrow^{\dagger}$ | WR50-2 | 29.1 | 39.9 | - | - | -
(i) | City Image |  R50⋆ | $\rightarrow$ | R50 | 4.6 | 6.6 | - | - | -
(j) | City Image | R50 | $\rightarrow$ | R50 | 18.0 | 34.0 | 39.8 | 45.6 | 55.5
(k) | City Image | R50 | $\rightarrow^{\dagger}$ | WR50-2 | 27.3 | 34.2 | 44.2 | 50.5 | 62.2
(l) | Noise Image | R18 | $\rightarrow$ | R50 | 0.7 | 1.5 | 3.4 | - | -
(m) | Bridge Image | R18 | $\rightarrow$ | R50 | 21.2 | 34.8 | 40.0 | - | -
(n) | City Image | R18 | $\rightarrow$ | R50 | 34.5 | 47.0 | 52.2 | 56.8 | 66.2

###### Patchification vs more data.

At the top part of the Table 9, we analyse the effect of using original vs
patchified random images from the training set of ImageNet. We find that while
for $10$ and $100$ source images, patchification improves performance, this is
not the case when a larger number of images is used. This indicates that
increased diversity is especially a crucial component for the small data
regime, while capturing larger parts of objects becomes important only beyond
this stage. We also make the surprising observation that patches from a single
high-quality image (“City”) obtain better performances than patches generated
from $1000$ ImageNet training images. To understand why this might be the
case, we note that the high coherency of training patches from a single image
strikingly resembles what babies see during their early visual development,
_e.g_., looking at only few toys and people but from many angles [3, 53]. It
is hypothesized that this unique combination of coherence and variability is
ideal for learning to recognize objects [53].

###### Teacher-student architectures.

In the bottom half of Table 9 (rows g-n), we vary the different teacher and
student architectures and find surprising results: First, in contrast to
normal KD, we find that the teacher’s performance is not directly related to
the final student performance. A case in point is the model R$50^{\star}$ (row
i), which yields the worst student performances, likely because the teacher
was trained to ignore heavy RandAugment [17] augmentations. Second, we find
that the choice of source image is even more important for this more difficult
task (see rows l-n), as switching from the “City” even to the “Bridge” one
decreases performance considerably and the Noise image does not train at all.
Third, we find that the best settings are those in which the student’s
capacity is higher than that of the teacher (rows k-n). In particular, we can
improve from $55.5\%$ top-1 accuracy to $66.2\%$ when switching to a weaker
ResNet-18 teacher. We also find that this a R18$\rightarrow$R50 setup (row n)
obtains a performance on ImageNet-12, which – though it uses a different
architecture and a teacher – surpasses the ground-breaking AlexNet [36]
performance of $56.5\%$ by almost $10\%$.

1-img Model | IN-1k | Places | IN-1k (1%) | PVOC | Places | COCO
---|---|---|---|---|---|---
imgs / class |  $\sim$1K |  $\sim$10K |  $\sim$13 | 4 | 16 | 4 | 16 | NA
MoCo-v2 | 28.5 | 28.8 | 14.4 | 17.2 | 26.6 | 4.5 | 9.7 | 35.6
Distill | 68.8 | 47.2 | 64.1 | 58.9 | 73.5 | 21.0 | 31.3 | 35.4
Table 10: Representation learning performance. We compare our distillation
approach (R50$\rightarrow$R50) against self-supervised pretraining via MoCo-v2
on the same data. We report: linear eval. accuracy on IN-1k and Places;
accuracy when finetuning and using $1$% of ImageNet with labels; SVM
classification mAP on PVOC07 and accuracy on Places with varying numbers of
images per class; mAP for COCO-detection with FPN and 1x-schedule.

###### Representation learning experiments.

Given that we can train a ResNet with only a single image and outputs of a
pretrained teacher model, it is natural to ask how well this model serves as a
general visual encoder. For this, we conduct a number of standard
representation learning evaluations in Table 10 and also compare against
running a common self-supervised learning method, MoCo-v2 [26], on the same
dataset (see Appendix C for training and evaluation details). We find that
freezing the backbone and only training a linear layer on top of it, can
achieve performances of $68.7$% and $47.2$% for ImageNet and Places,
respectively, vastly outperforming the self-supervised variant. A similar
trend is observed for data-efficient classification.

###### Scaling to video.

We conduct further experiments on video action recognition tasks. As
distillation data, we generate simple pseudo-videos of 12 frames that show a
linear interpolation of two crop locations at taken at the beginning and end
for the City image. We generate 200K of these videos and apply AugMix [29] and
CutMix [82] during the distillation for additional augmentations and use X3D
video architectures [21] as teachers and students, see Section B.1 for further
details. In Table 11, we find that our approach indeed generalizes well to
distilling video models with performances of 75% and 51.8% on UCF-101 [66] and
K400 [13]–despite the fact that the source data is not even a real video.

Table 11: Scaling to video. Student models are trained from scratch. Top-1 accuracy is shown with 10-temporal center-cropped clips per video. Details in Section B.1 Dataset | Categories (#Classes) | Teacher | 1-image + Distill
---|---|---|---
UCF-101 [66] | Human actions (101) | 90.4 | 75.2
K400 [13] | Human actions (400) | 67.8 | 51.8

####  4.4 Analysis

To better understand how our method is achieving these performances, we next
analyse the learned models.

###### Output confidence scores.

In Fig. 2, we visualize the distribution of the confidence scores of the
predicted classes for student and teacher models for the training and
evaluation data. We observe that the student indeed learns to mimic the
teacher’s predictions well for the patch training data. However, during
evaluation on CIFAR images, the student model exhibits a much broader
distribution of values, some with extremely high confidence scores. This might
indicate that the student has learned to identify many discriminative features
of object classes from the training data, which are all triggered when shown
real examples, leading to much higher confidence values.

###### Feature space analysis.

Figure 2: Output confidence scores. Temperature scaled softmax scores of the
predicted classes for $5$K patches and the $5$K CIFAR-10 validation set
images. Figure 3: Feature space visualization. $t$-SNE plot of jointly
embedding random patches ($\cdot$) and CIFAR-$10$ test set ($\times$)
instances with our model distilled to classify semantic classes. We also show
example images from patches and actual validation set images (enclosed in a
box). Note how most training images are contained within a small region while
the network needs to extrapolate for the real images which occupy the outer
regions in this plot. Best viewed zoomed-in.

Next, we analyse the network’s embedding of training patches when compared to
validation-set inputs. For this, we fit a $t$-SNE [72] using the features of
$5$K training patches and $5$K test set images in Fig. 3. We find that, as
expected, the training patches mapped to individual CIFAR classes do not
resemble real counter parts. Moreover, we observe that all the training
patches are embedded close to each other in the center, while CIFAR images are
clustered towards the outside. This shows that the network is indeed learning
features that are well suited for effective extrapolation.

###### Neuron visualizations.

Figure 4: Visualizing neurons via activation maximisation of final layer
neurons. We compare an ImageNet supervisedly trained model against our model
which has been trained with single image distillation. Best viewed zoomed-in.
We provide further examples in Section B.6.

We next analyze a single-image distilled model trained to predict IN-$1$k
classes. In Fig. 4, we visualize four final-layer neurons using the Lucid [49]
library. When we compare neurons of the standard ImageNet supervised and our
distilled model, we find that the neurons activate for very similar looking
inputs. The clear neuron visualizations for “panda” or “lifeboat” (and more
are provided in the Appendix B.6) are especially surprising since this network
was trained using only patches from the City Image, and has never seen any of
these objects during its learning phase.

###### Per-class evaluations.

Figure 5: Per-class validation accuracy vs training time.

Figure 6: Per-class student performance vs frequency of the class being the
top-1 prediction of the teacher.

We now turn to models trained to predict IN-$1$k classes and analyze per-class
performances against training time. In Fig. 6, we find that the validation
performance of IN-1k classes is very diverse: some are identified with
$>\\!80\%$ accuracy after just a few epochs of training, such as the class
“Black Swan”, while others require longer training and some are almost
impossible to learn, even after $200$ epochs. In Fig. 6, we plot the
validation performance against the frequency of how often the class appears as
a top-1 prediction of the teacher during $1$ epoch of training and find that
it is unrelated, showing the student is profiting heavily from the knowledge
contained in the soft-predictions, echoing findings from [22, 31].

###  5 Discussion

###### Limitations.

In this paper, we were mainly concerned with showing that extrapolating from a
single image works, empirically. While we do conduct several quantitative and
qualitative analyses, we do not provide a theoretical framework but instead
believe that theory will follow experimental evidence. Due to the limited
nature of our compute resources, we have not exhaustively analyzed the choice
of initial image or patchification augmentations, instead relying on insights
from previous work.

###### Potential negative societal impact.

While our research question is of fundamental nature, one possible negative
impact could be that the method is used to steal models and thus intellectual
property from API providers – although in practice the performance varies
widely especially for large-scale datasets (see Table 8).

###### Conclusion and outlook.

In this work, we have analyzed whether it is possible to train neural networks
to extrapolate to unseen semantic classes. Our quantitative and qualitative
results demonstrate that our novel single-image knowledge distillation
framework can indeed enable training networks from scratch to achieve high
accuracies on several architectures, datasets and domains. While this paper
has set out to answer the question posed in the abstract, in so doing it has
raised several further research questions, such as the dependency of the
source image and the target semantic classes; how networks combine features
for extrapolation; and the role and informational content of augmentations;
all of which we hope inspires further research.

#### Acknowledgements

Y.M.A is thankful for MLRA funding from AWS. We thank T. Blankevoort, A.F.
Biten, S. Albanie for useful comments on a draft of this paper.

### References

  * [1] Yuki M. Asano, Christian Rupprecht, and Andrea Vedaldi.  A critical analysis of self-supervision, or what we can learn from a single image.  In ICLR, 2020.
  * [2] Jimmy Ba and Rich Caruana.  Do deep nets really need to be deep?  In NeurIPS, volume 27, 2014.
  * [3] Sven Bambach, David Crandall, Linda Smith, and Chen Yu.  Toddler-inspired visual object learning.  In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett, editors, NeurIPS, volume 31, 2018.
  * [4] Manel Baradad, Jonas Wulff, Tongzhou Wang, Phillip Isola, and Antonio Torralba.  Learning to see by looking at noise.  arXiv preprit arXiv:2106.05963, 2021.
  * [5] E. Barnard and L.F.A. Wessels.  Extrapolation and interpolation in neural network classifiers.  IEEE Control Systems Magazine, 12(5):50–53, 1992.
  * [6] Lucas Beyer, Xiaohua Zhai, Amélie Royer, Larisa Markeeva, Rohan Anil, and Alexander Kolesnikov.  Knowledge distillation: A good teacher is patient and consistent.  arXiv preprint arXiv:2106.05237, 2021.
  * [7] Joanna Bitton and Zoe Papakipos.  Augly: A data augmentations library for audio, image, text, and video.  https://github.com/facebookresearch/AugLy, 2021.
  * [8] Ondrej Bohdal, Yongxin Yang, and Timothy Hospedales.  Flexible dataset distillation: Learn labels instead of images.  arXiv preprint arXiv:2006.08572, 2020.
  * [9] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei.  Language models are few-shot learners.  In NeurIPS, volume 33, pages 1877–1901, 2020.
  * [10] Joan Bruna and Stéphane Mallat.  Invariant scattering convolution networks.  PAMI, 35(8):1872–1886, 2013.
  * [11] Yaohui Cai, Zhewei Yao, Zhen Dong, Amir Gholami, Michael W Mahoney, and Kurt Keutzer.  Zeroq: A novel zero shot quantization framework.  In CVPR, pages 13169–13178, 2020.
  * [12] Mathilde Caron, Piotr Bojanowski, Julien Mairal, and Armand Joulin.  Unsupervised pre-training of image features on non-curated data.  In ICCV, 2019.
  * [13] Joao Carreira and Andrew Zisserman.  Quo vadis, action recognition? a new model and the kinetics dataset.  In CVPR, pages 6299–6308, 2017.
  * [14] Hanting Chen, Yunhe Wang, Chang Xu, Zhaohui Yang, Chuanjian Liu, Boxin Shi, Chunjing Xu, Chao Xu, and Qi Tian.  Dafl: Data-free learning of student networks.  In ICCV, 2019.
  * [15] Xinlei Chen, Haoqi Fan, Ross Girshick, and Kaiming He.  Improved baselines with momentum contrastive learning.  arXiv preprint arXiv:2003.04297, 2020.
  * [16] Adam Coates, Andrew Ng, and Honglak Lee.  An analysis of single-layer networks in unsupervised feature learning.  In AISTATS, volume 15 of Proceedings of Machine Learning Research, pages 215–223, Fort Lauderdale, FL, USA, 11–13 Apr 2011. PMLR.
  * [17] Ekin Dogus Cubuk, Barret Zoph, Jon Shlens, and Quoc Le.  Randaugment: Practical automated data augmentation with a reduced search space.  NeurIPS, 33, 2020.
  * [18] Wojciech M Czarnecki, Simon Osindero, Max Jaderberg, Grzegorz Swirszcz, and Razvan Pascanu.  Sobolev training for neural networks.  In NeurIPS, 2017.
  * [19] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei.  Imagenet: A large-scale hierarchical image database.  In CVPR, pages 248–255. Ieee, 2009.
  * [20] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby.  An image is worth 16x16 words: Transformers for image recognition at scale.  ICLR, 2021.
  * [21] Christoph Feichtenhofer.  X3d: Expanding architectures for efficient video recognition.  In CVPR, pages 203–213, 2020.
  * [22] Tommaso Furlanello, Zachary Lipton, Michael Tschannen, Laurent Itti, and Anima Anandkumar.  Born again neural networks.  In ICML, pages 1607–1616. PMLR, 2018.
  * [23] Deepti Ghadiyaram, Du Tran, and Dhruv Mahajan.  Large-scale weakly-supervised pre-training for video action recognition.  In CVPR, pages 12046–12055, 2019.
  * [24] Jianping Gou, Baosheng Yu, Stephen J Maybank, and Dacheng Tao.  Knowledge distillation: A survey.  IJCV, 129(6):1789–1819, 2021.
  * [25] P.J. Haley and D. Soloway.  Extrapolation limitations of multilayer feedforward neural networks.  In IJCNN International Joint Conference on Neural Networks, volume 4, pages 25–30 vol.4, 1992.
  * [26] Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick.  Momentum contrast for unsupervised visual representation learning.  In CVPR, pages 9729–9738, 2020.
  * [27] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In CVPR, 2016.
  * [28] Patrick Helber, Benjamin Bischke, Andreas Dengel, and Damian Borth.  Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification.  IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 2019.
  * [29] Dan Hendrycks, Norman Mu, Ekin Dogus Cubuk, Barret Zoph, Justin Gilmer, and Balaji Lakshminarayanan.  Augmix: A simple data processing method to improve robustness and uncertainty.  In ICLR, 2019.
  * [30] P Hettiarachchi, MJ Hall, and AW Minns.  The extrapolation of artificial neural networks for the modelling of rainfall—runoff relationships.  Journal of Hydroinformatics, 7(4):291–296, 2005.
  * [31] Geoffrey Hinton, Oriol Vinyals, and Jeff Dean.  Distilling the knowledge in a neural network.  arXiv preprint arXiv:1503.02531, 2015.
  * [32] Hirokatsu Kataoka, Kazushige Okayasu, Asato Matsumoto, Eisuke Yamagata, Ryosuke Yamada, Nakamasa Inoue, Akio Nakamura, and Yutaka Satoh.  Pre-training without natural images.  ACCV, 2020.
  * [33] Diederik P Kingma and Jimmy Ba.  Adam: A method for stochastic optimization.  ICLR, 2015.
  * [34] Simon Kornblith, Mohammad Norouzi, Honglak Lee, and Geoffrey Hinton.  Similarity of neural network representations revisited.  In ICML, pages 3519–3529. PMLR, 2019.
  * [35] Alex Krizhevsky, Geoffrey Hinton, et al.  Learning multiple layers of features from tiny images.  2009\.
  * [36] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton.  Imagenet classification with deep convolutional neural networks.  In NeurIPS, volume 25, 2012.
  * [37] Junnan Li, Pan Zhou, Caiming Xiong, and Steven C.H. Hoi.  Prototypical contrastive learning of unsupervised representations.  ICLR, 2021.
  * [38] Yuncheng Li, Jianchao Yang, Yale Song, Liangliang Cao, Jiebo Luo, and Li-Jia Li.  Learning from noisy labels with distillation.  In ICCV, pages 1910–1918, 2017.
  * [39] Tsung-Yi Lin, Piotr Dollár, Ross Girshick, Kaiming He, Bharath Hariharan, and Serge Belongie.  Feature pyramid networks for object detection.  In CVPR, pages 2117–2125, 2017.
  * [40] Pengpeng Liu, Irwin King, Michael R Lyu, and Jia Xu.  Ddflow: Learning optical flow with unlabeled data distillation.  In AAAI, pages 8770–8777, 2019.
  * [41] Raphael Gontijo Lopes, Stefano Fenu, and Thad Starner.  Data-free knowledge distillation for deep neural networks.  arXiv preprint arXiv:1710.07535, 2017.
  * [42] Ilya Loshchilov and Frank Hutter.  Decoupled weight decay regularization.  In ICLR, 2018.
  * [43] Ken MacLean.  Voxforge.  Ken MacLean.[Online]. Available: http://www. voxforge. org/home.[Acedido em 2012], 2018.
  * [44] Dhruv Mahajan, Ross Girshick, Vignesh Ramanathan, Kaiming He, Manohar Paluri, Yixuan Li, Ashwin Bharambe, and Laurens Van Der Maaten.  Exploring the limits of weakly supervised pretraining.  In ECCV, pages 181–196, 2018.
  * [45] Paul Micaelli and Amos J Storkey.  Zero-shot knowledge transfer via adversarial belief matching.  In NeurIPS, pages 9551–9561, 2019.
  * [46] Timothy Nguyen, Roman Novak, Lechao Xiao, and Jaehoon Lee.  Dataset distillation with infinitely wide convolutional networks.  arXiv preprint arXiv:2107.13034, 2021.
  * [47] Thao Nguyen, Maithra Raghu, and Simon Kornblith.  Do wide and deep networks learn the same things? uncovering how neural network representations vary with width and depth.  In ICLR, 2020.
  * [48] Maria-Elena Nilsback and Andrew Zisserman.  A visual vocabulary for flower classification.  In CVPR, volume 2, pages 1447–1454, 2006.
  * [49] Chris Olah, Alexander Mordvintsev, and Ludwig Schubert.  Feature visualization.  Distill, 2(11):e7, 2017.
  * [50] Aude Oliva and Antonio Torralba.  Modeling the shape of the scene: A holistic representation of the spatial envelope.  ICCV, 42(3):145–175, 2001.
  * [51] Bruno A Olshausen and David J Field.  Emergence of simple-cell receptive field properties by learning a sparse code for natural images.  Nature, 381(6583):607–609, 1996.
  * [52] Tribhuvanesh Orekondy, Bernt Schiele, and Mario Fritz.  Knockoff nets: Stealing functionality of black-box models.  In CVPR, 2019.
  * [53] Emin Orhan, Vaibhav Gupta, and Brenden M Lake.  Self-supervised learning through the eyes of a child.  NeurIPS, 33, 2020.
  * [54] Vassil Panayotov, Guoguo Chen, Daniel Povey, and Sanjeev Khudanpur.  Librispeech: an asr corpus based on public domain audio books.  In ICASSP, pages 5206–5210. IEEE, 2015.
  * [55] Omkar M. Parkhi, Andrea Vedaldi, Andrew Zisserman, and C. V. Jawahar.  Cats and dogs.  In CVPR, 2012.
  * [56] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al.  Learning transferable visual models from natural language supervision.  arXiv preprint arXiv:2103.00020, 2021.
  * [57] Ilija Radosavovic, Piotr Dollár, Ross Girshick, Georgia Gkioxari, and Kaiming He.  Data distillation: Towards omni-supervised learning.  In CVPR, pages 4119–4128, 2018.
  * [58] Colorado J Reed, Xiangyu Yue, Ani Nrusimha, Sayna Ebrahimi, Vivek Vijaykumar, Richard Mao, Bo Li, Shanghang Zhang, Devin Guillory, Sean Metzger, et al.  Self-supervised pretraining improves self-supervised pretraining.  arXiv preprint arXiv:2103.12718, 2021.
  * [59] Andres Rodriguez, Vishnu Naresh Boddeti, BVK Vijaya Kumar, and Abhijit Mahalanobis.  Maximum margin correlation filter: A new approach for localization and classification.  IEEE Transactions on Image Processing, 22(2):631–643, 2012.
  * [60] Adriana Romero, Nicolas Ballas, Samira Ebrahimi Kahou, Antoine Chassang, Carlo Gatta, and Yoshua Bengio.  Fitnets: Hints for thin deep nets.  arXiv preprint arXiv:1412.6550, 2014.
  * [61] Mert Bulent Sariyildiz, Yannis Kalantidis, Diane Larlus, and Karteek Alahari.  Concept generalization in visual representation learning.  In ICCV, pages 9629–9639, 2021.
  * [62] Tamar Rott Shaham, Tali Dekel, and Tomer Michaeli.  Singan: Learning a generative model from a single natural image.  In ICCV, pages 4570–4580, 2019.
  * [63] Assaf Shocher, Nadav Cohen, and Michal Irani.  “zero-shot” super-resolution using deep internal learning.  In CVPR, pages 3118–3126, 2018.
  * [64] Karen Simonyan and Andrew Zisserman.  Very deep convolutional networks for large-scale image recognition.  arXiv preprint arXiv:1409.1556, 2014.
  * [65] David Snyder, Guoguo Chen, and Daniel Povey.  Musan: A music, speech, and noise corpus.  arXiv preprint arXiv:1510.08484, 2015.
  * [66] Khurram Soomro, Amir Roshan Zamir, and Mubarak Shah.  Ucf101: A dataset of 101 human actions classes from videos in the wild.  arXiv preprint arXiv:1212.0402, 2012.
  * [67] Marco Tagliasacchi, Beat Gfeller, Félix de Chaumont Quitry, and Dominik Roblek.  Self-supervised audio representation learning for mobile devices.  arXiv preprint arXiv:1905.11796, 2019.
  * [68] Yonglong Tian, Dilip Krishnan, and Phillip Isola.  Contrastive multiview coding.  In ECCV, pages 776–794. Springer, 2020.
  * [69] Yonglong Tian, Dilip Krishnan, and Phillip Isola.  Contrastive representation distillation.  ICLR, 2020.
  * [70] Frederick Tung and Greg Mori.  Similarity-preserving knowledge distillation.  In ICCV, pages 1365–1374, 2019.
  * [71] Dmitry Ulyanov, Andrea Vedaldi, and Victor Lempitsky.  Deep image prior.  In CVPR, pages 9446–9454, 2018.
  * [72] Laurens Van der Maaten and Geoffrey Hinton.  Visualizing data using t-sne.  Journal of machine learning research, 9(11), 2008.
  * [73] Dongdong Wang, Yandong Li, Liqiang Wang, and Boqing Gong.  Neural networks are more productive teachers than human raters: Active mixup for data-efficient knowledge distillation from a blackbox model.  In CVPR, pages 1498–1507, 2020.
  * [74] Tongzhou Wang, Jun-Yan Zhu, Antonio Torralba, and Alexei A Efros.  Dataset distillation.  arXiv preprint arXiv:1811.10959, 2018.
  * [75] Pete Warden.  Speech commands: A dataset for limited-vocabulary speech recognition.  arXiv preprint arXiv:1804.03209, 2018.
  * [76] Ross Wightman, Hugo Touvron, and Hervé Jégou.  Resnet strikes back: An improved training procedure in timm.  arXiv preprint arXiv:2110.00476, 2021.
  * [77] Yuxin Wu and Kaiming He.  Group normalization.  In ECCV, pages 3–19, 2018.
  * [78] Yuxin Wu, Alexander Kirillov, Francisco Massa, Wan-Yen Lo, and Ross Girshick.  Detectron2.  https://github.com/facebookresearch/detectron2, 2019.
  * [79] Keyulu Xu, Mozhi Zhang, Jingling Li, Simon Shaolei Du, Ken-Ichi Kawarabayashi, and Stefanie Jegelka.  How neural networks extrapolate: From feedforward to graph neural networks.  In ICLR, 2020.
  * [80] Jingwen Ye, Yixin Ji, Xinchao Wang, Xin Gao, and Mingli Song.  Data-free knowledge amalgamation via group-stack dual-gan.  In CVPR, pages 12516–12525, 2020.
  * [81] Hongxu Yin, Pavlo Molchanov, Jose M Alvarez, Zhizhong Li, Arun Mallya, Derek Hoiem, Niraj K Jha, and Jan Kautz.  Dreaming to distill: Data-free knowledge transfer via deepinversion.  In CVPR, pages 8715–8724, 2020.
  * [82] Sangdoo Yun, Dongyoon Han, Seong Joon Oh, Sanghyuk Chun, Junsuk Choe, and Youngjoon Yoo.  Cutmix: Regularization strategy to train strong classifiers with localizable features.  In ICCV, pages 6023–6032, 2019.
  * [83] Sergey Zagoruyko and Nikos Komodakis.  Paying more attention to attention: Improving the performance of convolutional neural networks via attention transfer.  arXiv preprint arXiv:1612.03928, 2016.
  * [84] Sergey Zagoruyko and Nikos Komodakis.  Wide residual networks.  In BMVC, 2016.
  * [85] Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz.  mixup: Beyond empirical risk minimization.  In ICLR, 2018.
  * [86] Bo Zhao and Hakan Bilen.  Dataset condensation with differentiable siamese augmentation.  In ICML, 2021.
  * [87] Bolei Zhou, Agata Lapedriza, Aditya Khosla, Aude Oliva, and Antonio Torralba.  Places: A 10 million image database for scene recognition.  PAMI, 2017.
  * [88] Kaiyang Zhou, Ziwei Liu, Yu Qiao, Tao Xiang, and Chen Change Loy.  Domain generalization: A survey.  arXiv preprint arXiv:2103.02503, 2021.

##  Extrapolating from a Single Image to a Thousand Classes using Distillation
Appendix

###  Appendix A Additional Visualizations

####  A.1 Input images

(a) The “Noise” Image. From uniform noise [0,255]. Size: 2,560x1,920, PNG:
16.3MB.

(b) The “Universe” Image. Size: 2,300x2,100, JPEG: 7.2MB.

(c) The “Bridge” Image. Size: 1,280x853, JPEG: 288KB.

(d) The “City” Image. Size: 2,560x1,920, JPEG: 1.9MB.

(e) The “Animals” Image. Size: 1,300x600, JPEG: 267KB.

Figure 7: Single Images analysed. Here we show the images analysed in Table 1.

###### Image sources and licences.

From top to bottom, the images in Fig. 7 have the following licences and
sources: (a): CC-BY (we created it),
(b)444https://commons.wikimedia.org/wiki/File:Hubble_ultra_deep_field_high_rez_edit1.jpg
public domain (NASA) (c)555https://commons.wikimedia.org/wiki/File:GG-ftpoint-
bridge-2.jpg CC BY-SA 3.0 (by David Ball),
(d)666https://pixabay.com/photos/japan-ueno-japanese-street-sign-217883/
pixabay licence (personal and commercial use allowed),
(e)777https://www.teahub.io/viewwp/wJmboJ_jungle-animal-wallpaper-
wallpapersafari-jungle-animal/ personal use licence.

Images (b), (d) and (e) are identical to the ones in the repo of [1], while
(a) we have recreated on our own and (c) is a replacement for a similar one
whose licence we could not retrieve.

###### Audio source.

For the experiment on audio representation distillation, we use a $5.5$mins
English news-clip taken from BBC about how can Europe tackle climate
change888https://www.youtube.com/watch?v=nZgD4iPapVo and a $5$mins clip about
$11$ Germanic languages999https://www.youtube.com/watch?v=iq2_gTETBXM.

####  A.2 Training patches and predictions

Figure 8: ImageNet training data. Here we visualize the augmented training
data used for training along with the temperature-scaled Top-5 predictions of
the ImageNet pretrained ResNet-50 teacher.

In Fig. 8, we show several training patches as they are being used during
training along-side the teacher-network’s temperature-adjusted predictions.
Only very few training samples can be interpreted (_e.g_. $3$rd row: “nail” or
“slide rule”), while for most other patches, the teacher is being very
“creative”.

Figure 9: CIFAR-10 training data. We visualize temperature scaled predictions
of WideResNet-40-4 teacher on random patches generated from the Animal image
(see Figure 7).

In Fig. 9, we show a similar plot for the training patches used in CIFAR-10
training. Here we show the whole teaching signal, and highlight the top-$5$
predictions of the teacher for visualization.

Figure 10: Audio spectrograms. We visualize spectrograms generated from a one
second segment of the augmented audio. We generate $50$K audio clips from a
$5$ minutes audio recording by randomly taking a cropped audio segment of $2$
seconds and augmenting via one of the transformation functions mentioned in
Section C.1. During model training, we use $1$ second segment and apply MixUp
augmentation that varies across batch.

Finally, in Fig. 10, we show some some augmented training samples for the
audio classification experiment. This figure shows how various spectrograms
can be generated from a single clip by utilizing many augmentations.

(a) Source image.

(b) Patches.

Figure 11: Satellite image patches. We visualize random patches of size
$32\times 32$ generated from a single satellite image of Yogyakarta Airport,
Indonesia. The image is of size $3846\times 4087$ and $17$MB large as a JPEG.

###  Appendix B Additional Experiments

####  B.1 Video action recognition

We conduct further experiments on video action recognition tasks. For this we
use the common UCF-101 [66] dataset, more specifically the first split. As
distillation data, we generate simple source videos of $12$ frames that show a
linear interpolation between two crops of the City image. We generate 200K of
these videos and apply AugMix [29] and CutMix [82] during the distillation for
additional augmentations. For the architectures we use the recent, state-of-
the-art X3D architectures [21] as they obtain strong performances and allow
for flexible scaling of network architectures. We use a X3D-XS model as the
teacher, which is trained with inputs of $160\times 160$ and $4$ frames and a
temporal subsampling factor of $12$. For the student model we scale this
X3D-XS model in terms of width and depth by increasing the depth factor and
width factor from $2.2$ to $3$ and $4$ respectively, yielding a network with
$18.1$M parameters compared to $3.2$M. For UCF-101 we finetune the teacher
model starting from supervised Kinetics-400 [13] pretraining, which achieves a
$90.4$% performance. We use a batch size of $128$ per GPU on two GPUs, a
learning rate of $10^{-3}$, a temperature of $5$, cosine learning rate
schedule with a warmup of $5$ epochs, with no weight decay and no dropout. As
this experiment takes even longer than the image ones, we have not conducted
systematic ablations on these parameters. Instead, we merely picked ones which
looked promising after a few epochs but believe that even this small
experimental evidence is enough to investigate to what extent these 1-image
“fake-videos” can be used for learning to extrapolate.

The results are given in Table 11. We observe a performance of over $72$% on
UCF-101 using just a single fake-video as the training data along with the
pretrained teacher. While this number is far below the state of the art or the
teacher’s performance, it for example outperforms CLIP’s $69.8$% zero-shot
performance with a ViT-B16 model and shows that the student is able to
extrapolate to the action classes of UCF from just one datum.

####  B.2 COCO object detection and segmentation

In Section B.2 we show the result of running detection and segmentation full-
finetuning using FPN [39] starting from various initializations. We vary the
initial learning rate for all models and find that increasing it yields
benefits to the random baseline and our distillation based one. Despite the
increased learning rate we find that even under the ‘2x’ schedule, _i.e_.
double the number of training iterations, the performance increase is large.
We follow MoCo-v2’s [15] in implementing this experiment using detectron2
[78], for details see Section C.6.

| LR | Bounding-box |  | Segmentation
---|---|---|---|---
Init. |  | AP |  AP${}_{\text{50}}$ |  AP${}_{\text{75}}$ |  | AP |  AP${}_{\text{50}}$ |  AP${}_{\text{75}}$
Random | 1x | 26.4 | 44.0 | 27.8 |  | 29.3 | 46.9 | 30.8
Random | 4x | 32.8 | 51.3 | 35.9 |  | 30.1 | 48.5 | 32.4
Supv. IN-1k | 1x | 38.9 | 59.6 | 42.7 |  | 35.4 | 56.5 | 38.1
Supv. IN-1k | 4x | 36.7 | 56.6 | 39.6 |  | 33.6 | 53.6 | 36.3
MoCo-v2 (1-img) | 1x | 35.6 | 54.9 | 38.9 |  | 32.4 | 52.1 | 34.6
MoCo-v2 (1-img) | 4x | 34.3 | 53.3 | 37.1 |  | 31.5 | 50.6 | 33.6
Distill (1-img) | 1x | 28.7 | 48.8 | 30.1 |  | 27.2 | 45.8 | 28.5
Distill (1-img) | 4x | 35.3 | 55.5 | 38.3 |  | 32.8 | 52.4 | 35.4
Distill (1-img)† | 4x | 38.3 | 58.4 | 41.9 |  | 35.1 | 55.6 | 37.8
Table 12: Object detection & segmentation on COCO using R50-FPN, ‘1x’
schedule, except for the row with †, which is trained with double the number
of iterations (‘2x’), we vary initial learning rate (LR) by 1x or 4x.

####  B.3 Large scale experiment with noise

| Setting | Epochs
---|---|---
Images | Teacher |  | Student | 10 | 20 | 30 | 50
Noise Image | R18 | $\rightarrow$ | R50 | 0.1 | 0.1 | 0.1 | 0.1
Table 13: Top-1 accuracy in % when distilling an ImageNet ResNet-18 using the
‘Noise’ image.

In Table 13, we show for completeness the results of distilling a ImageNet
ResNet-18 using the ‘Noise’ image. As the accuracy constantly stays below 1%,
we can safely conclude that the choice of source image strongly matters for
the task of single-image distillation.

####  B.4 Varying Distillation Loss Functions

In Table 14 we compare the standard KD loss against L1 and L2 losses (using
scaled logits as in the KD loss) for training the student. We find that while
for CIFAR-10 the choice of the distillation method does not impact the
performance, on the more difficult ImageNet dataset, there is a small
difference of around $-0.4\%$ and $-0.5\%$ for L1 and L2 losses as compared to
the KD loss at epoch 30. This shows firstly that our method is not dependend
on a specific loss for distilling knowledge from a teacher to a student.
Secondly, it shows that the standard KD loss is actually a strong baseline, as
was also reported in [69].

Table 14: Distillation Loss Function Comparison. We compare the distillation performance with different loss functions along with teacher and student supervised training. Results are shown for CIFAR-10 with a WideRNet40-4$\rightarrow$WiderRNet16-4 setting and for ImageNet with R-18$\rightarrow$R-50. For ImageNet we show the performance progression after 10/20/30 epochs  Data | Teacher | Student | L1 | L2 | KD
---|---|---|---|---|---
C10 | 95.4 | 95.2 | 93.3 | 93.4 | 93.4
IN-1k | 69.8 | 76.1 | 36.2/45.1/51.8 | 25.1/45.9/51.7 | 34.5/47.0/52.2

####  B.5 Standard deviations in performance

In Table 15, we report the performance of five independent runs on CIFAR-10
dataset to further highlight the robustness of our experimental results. We
notice consistent accuracy scores across different training runs with minor
changes of $\approx\pm 0.2$.

Table 15: Model training with different random seeds. We compare the performance of five independent runs on CIFAR-10 dataset. We use WRN40-4 as teacher and WRN16-4 as student. “Full” shows the accuracy distilling using CIFAR-10 data, and “Ours” represents distillation using random patches generated from “Animals” image. Teacher | Student | Full | Ours
---|---|---|---
95.38 $\pm$ 0.09 |  94.76 $\pm$ 0.22 |  93.26 $\pm$ 0.17 |  93.38 $\pm$ 0.07

####  B.6 Neuron activation maximizations

Figure 12: Visualizing neurons via activation maximisation of final layer
neurons. We compare an ImageNet supervisedly trained model against our model
which has been trained with single image distillation.

In Fig. 12, we show further class-neuron activation maximization
visualizations for the R50$\rightarrow$R50 distillation experiment and compare
them to the corresponding ones of the supervised teacher model. Similar to the
examples in the main paper, we find that the neurons visualizations are
remarkably similar.

Figure 13: Visualizing intermediate neurons via activation maximisation. We
visualize five randomly picked neurons for each layer and show their maximally
responding input images.

In Fig. 13, we show further visualizations of the trained student model from
the R50$\rightarrow$R50 distillation experiment. We show five randomly chosen
neurons for every layer in the ResNet.

####  B.7 t-SNE of feature space

Figure 14: t-SNE embeddings of CIFAR-10. We visualize t-SNE embeddings of a
WideR16-4 model distilled from WideR40-4 with random patches generated from an
‘Animal’ image.

We visualize representations of distilled student model on CIFAR-10 validation
set in order to highlight semantic relevance of the features in Fig. 14. We
extract the feature from penultimate layer of WideR16-4 model for computing
$2$-D t-SNE embeddings. The visualization clearly highlights that distilled
features contain substantial amount of semantic information to correctly
differentiate between different object categories.

####  B.8 Centered Kernel Alignment comparison

(a) WideR$16-4$ via CIFAR-10 supervised training vs 1-image distilled.

(b) R50 via ImageNet supervised training vs 1-image distilled.

Figure 15: Representation similarity with CKA [34]. We visualize the
similarity between supervised representations and those distilled using random
patches of $1-$image. As in [47], we compute CKA for all the layers in a model
using CIFAR-10 testset. The layers in the same block group shows high
similarity though both models are trained in different ways, indicating that
distilled features are highly similar to those learned in a supervised way.

In Fig. 15, we visualize similarity between different layers of the models
trained in a general supervised manner and with distillation using patches of
$1$-image for CIFAR-10 and ImageNet datasets. We use centered kernel alignment
[34] method to identify the resemblance of features learned with these
different training approaches. We notice high similarity in representations,
which reinforce our empirical evaluation that the student models indeed learn
useful features required for differentiating between semantic categories.

####  B.9 Single-image training data vs toddler data

(a) Our 1-image dataset.

(b) Toddler data. Figure from [3].

Figure 16: Pair-wise distances of L$2$-normalized GIST features for $10$K
images from our $1$-image training dataset used for our ImageNet experiments

In Fig. 16(a), we compute the distances of GIST features [50] of $10$K
training images of the “City” image at resolution $256\times 256$ in Fig.
16(a). Following [3], we L2 normalize these GIST features before computing
pair-wise distances and plotting the histogram of the values. To make the
comparison of our training data with the visual inputs of toddlers more
concrete, we compare low-level GIST [50] features of our 1-image dataset
against those computed from visual inputs of a toddler in [3]. As we do not
have access to the dataset of [3], we copy their Figure 3b for reference in
Fig. 16(b). While the GIST distances for ImageNet are very different with a
mean around $0.75$ [3], we find that our single image dataset’s distribution
in Fig. 16(a) closely resembles that of the visual inputs of a toddler. While
this is one possible explanation for why this type of data might work well for
developing visual representations, further research is still required.

Dataset | Model | Standard |  | Quantization
---
(source)
| Ours - Quantization
---
(single-image)
| Pruning
---
(source)
| Ours - Pruning
---
(single-image)
CIFAR-10 | ResNet56 | 93.77 | 93.64 | 93.45 | 93.38 | 93.18
VGG11 | 91.57 | 91.22 | 90.97 | 91.14 | 90.86
VGG19 | 93.28 | 92.94 | 93.00 | 92.84 | 92.96
WideRNet16-4 | 94.81 | 94.40 | 94.59 | 94.76 | 94.43
WideRNet40-4 | 95.42 | 95.09 | 94.90 | 95.29 | 94.82
CIFAR-100 | ResNet56 | 70.99 | 70.89 | 69.85 | 70.74 | 70.42
VGG11 | 69.65 | 69.92 | 68.86 | 69.77 | 69.13
VGG19 | 70.79 | 70.60 | 70.21 | 70.89 | 70.56
WideRNet16-4 | 75.81 | 75.45 | 74.71 | 75.68 | 75.51
WideRNet40-4 | 78.14 | 78.27 | 77.60 | 77.94 | 77.78
Table 16: Self-distillation with a single-image for efficient deep models. We
perform model compression via self-distillation using source data and $50$k
random patches generated from ‘Animals’ image for $50$% sparsity (in case of
pruning) and $8$-bits quantization without any noticeable loss in performance.

####  B.10 Data-free Pruning and Quantization of Pre-trained Models

Neural network compression has been studied extensively in the literature to
produce light-weight models with the objective of improving computational
efficiency. In addition to knowledge distillation, network pruning and
quantization are other well-known approaches. In the former, the goal is to
remove or sparsify (zeroing-out) the network’s weights based on a specific
criteria, such as subset of weights with the lowest magnitude. The later is
concerned with reducing precision from $32$-bits floating numbers to $8$-bits
and even single bit as binary neural networks. In general, from practical
standpoint post-training compression is largely employed in conjunction with
fine-tuning to avoid accuracy drop.

Here, we study utilize our proposed single-image distillation framework for
post-training model compression without using any real data. Specifically, we
ask the question, whether a pre-trained model can be compressed when no in-
domain data is available? To this end, we utilize Tensorflow Model
Optimization Toolkit101010https://www.tensorflow.org/model_optimization to
prune and quantize various pre-trained model on CIFAR-10 and CIFAR-100
datasets. We use self-distillation, where a pre-trained model acts as teacher
and student clone of it is compressed during fine-tuning phase. In Table 16,
we report accuracy scores for the pruned and quantized models in comparison
with the standard models and those compressed using real in-domain data. In
all the cases, we can notice that there is negligible loss in accuracy, i.e.,
$<=$ $1\%$, while using random patches.

Sparsity | ResNet56 | VGG11 | WideRNet16-4
---|---|---|---
| C10 | C100 | C10 | C100 | C10 | C100
0% | 93.77 | 70.99 | 91.57 | 69.65 | 94.81 | 75.81
25% | 93.55 | 70.85 | 91.03 | 69.37 | 94.55 | 75.40
50% | 93.18 | 70.74 | 90.86 | 69.13 | 94.43 | 75.51
75% | 92.68 | 67.87 | 90.54 | 67.81 | 94.10 | 74.05
85% | 91.50 | 61.22 | 89.12 | 65.23 | 93.05 | 71.46
Table 17: Varying Sparsity. Pruning results achieved on CIFAR-10/100 for
different sparsity rates using self-distillation with random patches generated
from the ‘Animal’ image. Figure 17: Representation similarity with CKA of a
pruned and standard WideR16-4 models trained on CIFAR-100.

Further, in Table 17, we vary sparsity ratio and can observe that a model can
be pruned up to $75\%$ sparsity with three-points drop in accuracy but with
$25\%$ pruning ratio the loss is merely noticeable. In Figure 17, we analyze
representational similarity of pruned WideRNet16-4 model with the original
trained on CIFAR-100, the diagonal entries highlight that even after $50\%$
sparsity the representations are very similar. To summarize, with model
compression use case, we barely scratched the surface of what is possible with
our single-image distillation framework and we hope it will inspire further
studies and applications in other domains.

###  Appendix C Implementation Details

All code will be released open-source and is attached as supplementary
information to this submission.

####  C.1 Initial patch generation

We do not tune the individual patch generation and instead adapt the procedure
directly from [1]111111https://github.com/yukimasano/single_img_pretraining.
The augmentations applied to the input image of size $H\times W$ to arrive at
individual patches of size $P\times P$ is (in order) as follows:

1) RC(size=0.5*min(H,W))

2) RRC(size=(1.42*P), scale=(2e-3, 1))

3) RandomAffine(degrees=30, shear=30)

4) RandomVFlip(p=0.5)

5) RandomHFlip(p=0.5)

6) CenterCrop(size=(P,P))

7) ColorJitter(0.4,0.4,0.4,0.1, p=0.5)

All transformations are standard operations in PyTorch: RC stands for
RandomCrop, _i.e_. taking a random crop in the image with a specific size; RRC
for RandomResizedCrop, _i.e_. taking a random sized crop withing the size
specified in the scale tuple (relative to the input); RandomAffine for random
affine transformations (rotation and shear); RandomVFlip and RandomHFlip for
random flipping operations in the vertical and horizontal direction, and
CenterCrop crops the image in the center to a square image of size $P\times
P$. Finally, ColorJitter computes photometric jittering, where the parameters
for the strengths are given in the order of brightness, contrast, saturation
and hue and applied with a certain probability.

Similarly, for audio clips generation given a single audio, we use
augmentation operations from [7] with default settings. Specifically, to
create a single example we apply the following procedure: we randomly crop a
segment of $2$-seconds, and use randomly sample an augmentation function to
create transformed instances and save them in mono format. In our work, we use
these augmentations, all with their default settings:

1) add-background-noise

2) change-volume

3) clicks

4) clip

5) harmonic

6) high-pass-filter

7) low-pass-filter

8) normalize

9) peaking-equalizer

10) percussive

11) pitch-shift

12) reverb

13) speed

14) time-stretch

####  C.2 Computing log-Mel spectrograms

The log-Mel spectrograms are generated on-the-fly during training from a
randomly selected $1$-second crop of an audio waveform as the model’s input.
We compute it by applying a short-time Fourier transform with a window size of
$25$ms and a hop size equal to $10$ms to extract $64$ Mel-spaced frequency
bins for each window. During evaluation, we average over the predictions of
non-overlapping segments of an entire audio clip.

####  C.3 Audio neural network architecture

Our audio convolutional neural network is inspired by [67] and it consists of
four blocks. We perform separate convolutions in each block with a kernel size
of $4$. One on the temporal and another on the frequency dimension, we
concatenate their outputs afterward to perform a joint $1\times 1$
convolution. It allows model to capture fine-grained features from each
dimension and discover high-level features from shared output. We apply L$2$
regularization with a rate of $0.0001$ in each convolution layer and also use
group normalization [77]. Between the blocks, we utilize max-pooling to reduce
the time-frequency dimensions by a factor of $2$ and use a spatial dropout
rate with a rate of $0.2$ to avoid over-fitting. We apply ReLU as a non-linear
activation function and feature maps in the convolutions blocks are
$24,32,64$, and $128$. Finally, we aggregate the feature with a global max
pooling layer which are fed into a fully-connected layer with number of units
equivalent to the number of classes.

####  C.4 Training

#####  C.4.1 Small-scale experiments

For experiments on CIFAR-$10$, CIFAR-$100$, and other smaller datasets, we use
Tensorflow for running experiments on a single T$4$ GPU with a batch size of
$512$ using an Adam [33] optimizer with a fixed learning rate of $0.001$. We
use standard augmentations including, random left right flip, and random crop.
The supervised models also uses cutout augmentations with a cutout size of
$16\times 16$. For Mix-up, we sample $\alpha$ uniformly at random between zero
and one. In Cut-Mix, we use a fixed value of $0.25$ for $\alpha$ and $\beta$.
With the following setup, each of the single image distillation experiment of
$1$K epochs took around $2-3$ days. The supervised and standard (using source
data) distillation models are trained for $100$ epochs (per epoch $2000$
steps) with a batch size of $128$ using SGD for optimization. For VGG [64] and
ResNet [27] models, we use a learning rate schedule of $0.01,0.1,0.01,0.001$
decayed at following steps $400,32000,48000,64000$ with momentum of $0.9$. For
WideResNet [84], we use a learning rate schedule of $0.1,0.02,0.004,0.0008$
that is decayed at following steps $24000,48000,64000,80000$ with Nesterov
enabled.

In audio experiments, we use an Adam [33] optimizer with a fixed learning rate
of $0.001$ and use batch size of $128$ and $512$ for standard models and
single-clip distillation, respectively. Furthermore, we also utilize Mix-Up
augmentation during our knowledge distillation experiments.

#####  C.4.2 Large-scale experiments

We use PyTorch’s DistributedDataParallel engine for running experiments on $2$
A$6000$ GPUs in parallel with batch-sizes of $512$ each. For optimization we
use AdamW [42] with a learning rate of $0.01$ and a weight-decay of $10^{-4}$.
These values were determined by eyeballing the results from [6] to find a
setting that might generalize across datasets, as we do not have enough
compute to run hyper-parameter sweeps. With this setup, each $200$ epoch
experiment took around $5$ days. We have also found that halving the batch-
size performs equally well, and is more amenable to lower-memory GPUs. For the
distillation experiments, we use Cut-Mix [82] with its default parameters of
$\alpha=\beta=1.0$.

####  C.5 GIST features comparison

In Fig. 16(a), we compute the distances of GIST features [50] of $10$K
training images of the “City” image at resolution $256\times 256$ in Fig.
16(a). Following [3], we L2 normalize these GIST features before computing
pair-wise distances and plotting the histogram of the values.

####  C.6 Representation learning experiments

###### Pretraining.

We pretrain using the official code of MoCo-v2 [26] in the standard $200$
epoch setting, while applying their default “v2” augmentations during
training.

###### Evaluation.

We follow standard linear evaluation procedure from the MoCo-v2 repo. which
uses a batch size of $256$, learning rate of $30$ which is multiplied by $0.1$
at epochs $60$ and $80$ for a total of $90$ epochs. For the data-efficient
full-fine tuning, we utilize the implementation from SwAV [12], which trains
for $20$ epochs using a cosine decay learning rate schedule and a batch size
of $256$. For the data-efficient SVM classification experiments, we follow the
implementation of [37] and report average results from $5$ trials and keep the
SVM’s cost parameter fixed at a value of $0.5$. For the COCO object detection
experiments we use the detectron2 repo [78] and the 1x schedule using a FPN
[39] as in [26]. We additionally vary the learning rates from 1x and 4x as the
models are trained with very different losses compared to the supervised
variant for which the learning schedule is made. The results are shown in the
last row of Table 12 and show that our models benefit from a higher initial
learning rate, which also does not have any negative effect on training longer
_e.g_. with the 2x schedule, while the MoCo pretrained model works best with
the 1x learning rate.

###  Appendix D Version history

This paper is version 3. Compared to version 1 of this paper we have:

  * •

Extended video experiments (200 epochs and 400 epochs) and added new
Kinetics-400 experiments.

  * •

Added pruning and quantization experiments.

  * •

Provided more detailed references to previous work.

  * •

Added COCO downstream evaluation experiments.

  * •

Added large scale experiment with the Noise image.

Compared to version 2 of this paper we have:

  * •

Discovered a bug in our large-scale experiments that was adapting BatchNorm
statistics of the teacher with the student’s training data. We have fixed this
such that these statistics remain frozen. CIFAR and audio results were not
affected from this, as this was a PyTorch (Lightning) specific issue. Fixing
this has lead to strong performance gains (_e.g_. R18$\rightarrow$R50 results
increased from $59.1\%$ to $66.2\%$). We have therefore rerun our experiments
of Tables 8-14 and recomputed the Figures 5-7,13,14,16b. All results
(qualitative and quantitative) in this paper therefore now use the (better)
updated version.