# CMKD: CNN/Transformer-Based Cross-Model Knowledge Distillation for Audio
Classification

Yuan Gong, Sameer Khurana, Andrew Rouditchenko, and James Glass Preprint. Code
will be released after the review process.The authors are with the Computer
Science and Artificial Intelligence Laboratory, Massachusetts Institute of
Technology, Cambridge, MA 02139 USA (e-mail: {yuangong, skhurana, roudi,
glass}@mit.edu).

###### Abstract

Audio classification is an active research area with a wide range of
applications. Over the past decade, convolutional neural networks (CNNs) have
been the de-facto standard building block for end-to-end audio classification
models. Recently, neural networks based solely on self-attention mechanisms
such as the Audio Spectrogram Transformer (AST) have been shown to outperform
CNNs. In this paper, we find an intriguing interaction between the two very
different models - _CNN and AST models are good teachers for each other_. When
we use either of them as the teacher and train the other model as the student
via knowledge distillation (KD), the performance of the student model
noticeably improves, and in many cases, is better than the teacher model. In
our experiments with this CNN/Transformer Cross-Model Knowledge Distillation
(CMKD) method we achieve new state-of-the-art performance on FSD50K, AudioSet,
and ESC-50.

###### Index Terms:

Audio Classification, Convolutional Neural Networks, Transformer, Knowledge
Distillation

##  1 Introduction

Audio classification aims to identify sound events that occur in a given audio
recording and enables a variety of artificial intelligence-based systems to
disambiguate sounds and understand the acoustic environment. Historically,
hand-crafted features and hidden Markov models (HMMs) were used for audio
classification [1, 2, 3]. With the rise of neural networks in the past decade,
Convolutional Neural Networks (CNNs) [4] have became the de-facto standard
building block for _end-to-end_ audio classification models, which aim to
learn a direct mapping from audio waveforms or spectrograms to corresponding
labels [5, 6, 7, 8, 9, 10]. More recently, neural networks based purely on
self-attention, such as the Audio Spectrogram Transformer (AST) [11, 12, 13],
have been shown to further outperform deep learning models constructed with
convolutional neural networks on various audio classification tasks, thus
extending the success of Transformers from natural language processing [14,
15] and computer vision [16, 17] to the audio domain.

CNN and Transformer models both have their advantages. For example, CNN models
have several built-in inductive biases such as spatial locality and
translation equivariance, making them well suited to spectrogram-based end-to-
end audio classification. Transformer models do not have such built-in
inductive biases and learn in a more data-driven manner, making them more
flexible. As a consequence, the representations learned by CNN and Transformer
models are noticeably different [18]. On the other hand, while Transformer
models perform better, they are less computationally efficient than CNN models
on long audio input due to their $O(n^{2})$ complexity.

In this paper, we show an intriguing interaction between the two very
different models - _CNN and AST models are good teachers for each other_. When
we use one model as the teacher and train another model as the student via
knowledge distillation (KD), the performance of the student model noticeably
improves, and in most cases, is better than the teacher model. We refer to
this knowledge distillation framework between a CNN and a Transformer model as
Cross-Model Knowledge Distillation (CMKD). The success of cross-model
knowledge distillation is not trivial because 1) cross-model knowledge
distillation works bi-directionally in both CNN$\rightarrow$Transformer and
Transformer$\rightarrow$CNN directions. Usually in KD, the teacher needs to be
stronger than the student, but for cross-model knowledge distillation, a weak
teacher can still improve a student’s performance. 2) For both directions, the
student outperforms the teacher after knowledge distillation, even when the
teacher is originally stronger. 3) KD between two models of the same class
leads to a much smaller or no performance improvement. Consequently, with the
proposed cross-model knowledge distillation, a simple EfficientNet KD-CNN
model with mean pooling outperforms the much larger AST model on FSD50K and
ESC50 dataset. Conversely, the KD-AST model achieves new state-of-the-art
performance on FSD50K, AudioSet, and ESC50 datasets.

The contribution of this paper is threefold: First, to the best of our
knowledge, we are the first to explore bi-directional knowledge distillation
between CNN and Transformer models; previous efforts [17, 19] only study the
CNN$\rightarrow$Transformer direction and are performed in the visual domain.
Second, we conduct extensive experiments on standard audio classification
datasets and find the optimal knowledge distillation setting. In addition, we
conduct a series of probing tests and ablation studies to explore the working
mechanism of cross-model knowledge distillation. Third, due to the proposed
cross-model KD framework, the small and efficient CNN models match or
outperform previous state-of-the-art models; the AST models achieve even
better performance and achieve new state-of-the-art results on FSD50K,
AudioSet, and ESC50. We also evaluate CMKD on various CNN, CNN-attention, and
Transformer models and find it improves the performance of all these models,
showing CMKD is a generic method - all types of models can benefit from CMKD.

Figure 1: Illustration of the CNN and AST model architecture and the cross-
model knowledge disitllation process. For both AST$\rightarrow$CNN and
CNN$\rightarrow$AST, we always use a trained teacher model and freeze it
during the student model training. The training objective is minimizing the
ground truth loss $Loss_{g}$ and the distillation loss $Loss_{d}$.

##  2 Cross-Model Knowledge Distillation

This section discusses the details of the proposed cross-model knowledge
distillation between a CNN model and an AST model. Specifically, we introduce
the architecture of CNN and AST models used in this work in Sections 2.1 and
2.2, respectively. We intend to use CNN and AST models that achieve the best
performance on audio classification [20, 11] to see if CMKD can further
improve their performance and achieve a new state-of-the-art. We then discuss
the main difference between these two classes of models in Section 2.3.
Finally, we show the knowledge distillation setting and notation in Section
2.4.

###  2.1 Convolution Neural Networks

In this work, we use the CNN model without attention module proposed in [20].
To the best of our knowledge, it is the best CNN model on the audio
classification task. As illustrated in the left side of Figure 1, the input
audio waveform of $t$ seconds is first converted into a sequence of
128-dimensional log Mel filterbank (fbank) features computed with a 25ms
Hanning window every 10ms. This results in a $128\times 100t$ spectrogram as
input to the CNN model. The spectrogram is then input to a standard CNN model
such as EffcientNet [21] and DenseNet [22]. The output of the penultimate
layer is a tensor of size $(\lceil 128/c\rceil,\lceil 100t/c\rceil,d)$ in
frequency, time, and embedding dimension, respectively, where $c$ is the
feature downsampling factor of the CNN model. We apply a time and frequency
mean pooling to produce the $d$ dimensional spectrogram-level representation.
Finally, a linear layer with a sigmoid (for multi-label classification) or
softmax (for single-label classification) activation maps the audio
spectrogram representation to labels for classification. We mainly use
EfficientNet-B2 [21] as the CNN model in our experiments because it has been
shown to be the best CNN model on audio classification [20]. To study the
impact of model size and the generalization of the proposed cross-model
knowledge distillation strategy, we also include Efficient-B0, Efficient-B6,
and DenseNet-121 in our experiments. The statistics of the different models
are presented in Table I.

TABLE I: Statistics of the CNN and AST models used in this work. Model |  | # Layers
---
/Blocks
| # Attention
---
Heads
| Representation
---
Dimension $d$
| # Params
---
(M)
CNN Models
EfficientNet-B0 | 237 | - | 1,280 | 4
EfficientNet-B2 | 339 | - | 1,408 | 8
EfficientNet-B6 | 666 | - | 2,304 | 41
DenseNet-121 | 121 | - | 1,024 | 7
AST Models
AST-Tiny | 12 | 3 | 192 | 6
AST-Small | 12 | 6 | 384 | 23
AST-Base | 12 | 12 | 768 | 88

###  2.2 Audio Spectrogram Transformers

In this work, we use the original AST model proposed in [11] that has the best
performance on the audio classification task. As illustrated on the right side
of Figure 1, the input audio waveform of $t$ seconds is converted to a
$128\times 100t$ spectrogram in the same way as the CNN model. We then split
the spectrogram into a sequence of $N$ $16\times 16$ patches with an overlap
of 6 in both time and frequency dimension, where
$N=12\lceil{(100t-16)/10}\rceil$ is the number of patches and the effective
input sequence length for the Transformer. We flatten each $16\times 16$ patch
to a 1D patch embedding of size $d$ using a linear projection layer. We refer
to this linear projection layer as the patch embedding layer. Since the
Transformer architecture does not capture input order information and the
patch sequence is also not in temporal order, we add a trainable positional
embedding (also of size $d$) to each patch embedding to allow the model to
capture the spatial structure of the 2D audio spectrogram. We append a [CLS]
token at the beginning of the sequence. The resulting sequence is then input
to a standard Transformer encoder [14]. The Transformer encoder’s output of
the [CLS] token serves as the audio spectrogram representation. A linear layer
with sigmoid activation (for multi-label classification) or softmax (for
single-label classification) maps the audio spectrogram representation to
labels for classification. To study the impact of model size, we use three AST
models of different sizes in our experiments: AST-Tiny, AST-Small, and AST-
Base. The statistics of the models are shown in Table I.

For both CNN and AST models (including No-KD baseline models), we always use
ImageNet pretraining, a common technique to improve audio classification [20,
11, 23, 24, 25, 6]. Specifically, we initialize the CNN and Transformer with
ImageNet-pretrained weights from public model checkpoints
111https://pytorch.org/vision/stable/models.html,222https://github.com/rwightman/pytorch-
image-models and then conduct audio classification training. Since the input
to the vision models is a 3-channel image while the audio model input is a
single-channel spectrogram, we average the weights corresponding to each of
the three input channels of the vision model checkpoints and use them as the
weights of the first layer of the audio models. This is equivalent to
expanding a single-channel spectrogram to 3-channels with the same content,
but is computationally more efficient. Note that for all AST models (including
the No-KD baselines), we initialize the model with the same vision model
checkpoint [17] as the original AST [11]. This vision model checkpoint is
trained with KD during the ImageNet training stage [17] and has been shown to
lead to better performance for audio classification than checkpoints trained
without KD. This setting allows us to clearly identify the improvement due to
KD in the audio domain training stage and make a fair comparison with the
original AST [11].

###  2.3 Difference between CNN and AST

As described in the previous two subsections, CNN and AST models have
completely different design philosophies. CNN models have several built-in
inductive biases that make them well suited to the spectrogram-based end-to-
end audio classification task such as translation equivariance (without
pooling) or translation invariance (with pooling) and locality. CNN models
typically have a hierarchical architecture with an increasingly large
receptive field in deeper layers. In contrast, AST lacks such spectrogram-
specific built-in inductive biases and needs to learn in a more data-driven
manner. However, this also allows AST models to be more flexible, e.g., the
receptive field of an AST is learnable and the very first layer can capture
long-range global context while the locality makes the first few layers of
CNNs struggle to relate temporal- and frequency- distant features. As a
consequence, CNNs typically perform better with small datasets while
Transformer models start showing their superiority with more training data
[26, 27, 17, 16]. Due to the different model architectures, the learned
representations of CNNs and Transformers have striking differences [18].
Efficiency-wise, AST has quadratic complexity with respect to the input size
due to the global attention design while the CNN model has linear complexity
and thus is more efficient for long audio samples.

###  2.4 Knowledge Distillation

In this work, we follow the original knowledge distillation setup [28] with
_consistent teaching_ [29] and find this simple setup works better than the
more complex attention distillation strategy [17] for the audio classification
task. Specifically, we first train the teacher model, and then during the
student model training, we feed the input audio spectrogram with the exact
same augmentations (e.g., SpecAugment [30], noise, time shift, and mixup [31])
to the teacher and student models (consistent teaching). We use the following
loss for the student model training:

$Loss=\lambda
Loss_{g}(\psi(Z_{s}),y)+(1-\lambda)Loss_{d}(\psi(Z_{s}),\psi(Z_{t}/\tau))$ (1)

where $\lambda$ is the balancing coefficient; $Loss_{g}$ and $Loss_{d}$ are
the ground truth and distillation losses, respectively; $\psi$ is the
activation function; $Z_{s}$ and $Z_{t}$ are the logits of the student and
teacher model, respectively; and $\tau$ is the temperature. The teacher model
is frozen during the student model training.

By default, we use the Kullback–Leibler divergence as $Loss_{d}$. For cross-
model KD, the teacher and student may have different logit distribution
softness, thus we only apply $\tau$ on the teacher logits to explicitly
control the difference. For simplicity, we fix $\lambda=0.5$ and do not scale
it with $\tau$. We use the cross-entropy (CE) loss as $Loss_{g}$ and the
softmax activation function for single-label classification tasks such as
ESC-50; the binary cross-entropy (BCE) loss as $Loss_{g}$ and the sigmoid
activation function for multi-label classification tasks such as FSD50K and
AudioSet. We study the performance impact of KD settings in Section 4.2. In
this paper, we use teacher$\rightarrow$student to denote the direction of
knowledge distillation, e.g., EfficientNet-B0$\rightarrow$AST-Base means we
first train an EfficientNet-B0 model, and then use it as the teacher to train
an AST-Base model as the student.

TABLE II: Training settings of CNN and AST models on FSD50K, AudioSet, and ESC-50. Baseline and KD models are trained with same settings. Setting | FSD50K | AudioSet | ESC-50
---|---|---|---
| CNN | AST | CNN | AST | CNN | AST
Initial LR | 5e-4 | 5e-5 | 1e-4 | 1e-5 | 1e-4 | 1e-5
Batch Size | 24 | 12 | 120 | 12 | 48
Epoch | 50 | 30 | 5 | 25
Loss | Binary Cross Entropy | Cross Entropy
Mixup Ratio | 0.5 | 0
| SpecAugment
---
(Frequency/Time)
48/192 | 24/96
Class Balancing | Yes | No
Label Smoothing | 0.1
Random Noise |  $U(0,0.05)$ on spectrogram
Random Time Shift |  $\pm$10 frames
TABLE III: MAP of the student model of various (teacher,student) pair on the FSD50K evaluation set. $\dagger$ denotes that student model performance improves with KD; $\ast$ denotes that student model outperforms teacher model. | Student
---
Model
| Model Size
---
(# Params)
| Teacher Model
---
(Teacher Model Performance Shown in Grey)
|  | No KD | EffNet-B0 | EffNet-B2 | EffNet-B6 | DenseNet-121 | AST-Tiny | AST-Small | AST-Base
|  | N/A | 55.7 | 56.2 | 54.7 | 53.9 | 56.3 | 56.6 | 57.6
EffNet-B0 | 4M | 55.7 |  55.9†∗ |  56.0† |  55.4∗ |  56.1†∗ |  57.4†∗ |  56.9†∗ |  56.6†
EffNet-B2 | 8M | 56.2 |  56.2∗ |  56.3†∗ |  55.4∗ |  56.8†∗ |  58.2†∗ |  57.4†∗ |  57.3†
EffNet-B6 | 41M | 54.7 |  55.6† |  56.1† |  54.0 |  55.1†∗ |  58.0†∗ |  57.7†∗ |  57.8†∗
DenseNet-121 | 7M | 53.9 |  53.1 |  54.6† |  54.0† |  52.7 |  56.9†∗ |  56.7†∗ |  57.0†
AST-Tiny | 6M | 56.3 |  58.6†∗ |  58.5†∗ |  58.2†∗ |  59.1†∗ |  57.2†∗ |  56.5† |  56.8†
AST-Small | 23M | 56.6 |  60.5†∗ |  60.1†∗ |  59.3†∗ |  60.3†∗ |  59.4† |  57.7†∗ |  57.8†∗
AST-Base | 88M | 57.6 |  61.7†∗ |  61.5†∗ |  61.5†∗ |  61.5†∗ |  60.9†∗ |  59.6†∗ |  59.6†∗

##  3 Experiments Settings

###  3.1 Datasets

We evaluate the proposed method on three widely-used audio classification
datasets: FSD50K [32], AudioSet [33], and ESC-50 [34]. The FSD50K dataset [32]
is a collection of sound event audio clips with 200 classes, which contains
37,134 audio clips for training, 4,170 audio clips for validation, and 10,231
audio clips for evaluation. The audio clips are of variable length from 0.3 to
30s with an average of 7.6s. In our experiments, we sample audio at 16kHz and
trim all clips to 10s. We use the FSD50K dataset for the majority of our
experiments in this paper (Section 4) for three reasons. First, FSD50K allows
more rigorous experiments since it has an official training, validation, and
evaluation split. In this paper, we train the model with the training set,
tune all hyper-parameters and select the model based on the validation set,
and report the mean average precision (mAP) on the evaluation set. Second,
FSD50K is a publicly available dataset, so results on FSD50K are easy to
reproduce. Third, the FSD50K dataset is of moderate size (50K samples)
compared with AudioSet (2M samples) and ESC-50 (2K samples), which allows us
to conduct extensive experiments with our computational resources. 333In
contrast, the AudioSet only has official training and evaluation split, and
contains considerable label noise [35, 36, 37]. In addition, AudioSet is
composed of YouTube videos that are not freely distributable. Downloading
AudioSet from YouTube is laborious.

AudioSet [33] is a collection of over 2 million 10-second audio clips excised
from YouTube videos and labeled with the sounds that the clip contains from a
set of 527 labels. The balanced training, full training, and evaluation set
contain 22k, 2M, and 20k samples, respectively. We use AudioSet to study the
generalization ability of the proposed method on larger datasets. The ESC-50
[34] dataset consists of 2,000 5-second environmental audio recordings
organized into 50 classes. We use ESC-50 to study the transferability of
models trained with the proposed knowledge distillation method.

###  3.2 Training Settings

The hyper-parameters we use to train the models are shown in Table II. In
order to make fair comparisons, we train models with and without KD with
exactly the same settings. Specifically, we use the training recipes that are
optimized for the original CNN and AST without KD in [20, 11] including the
initial learning rate, learning rate scheduler, batch size, class balancing,
mixup training [31], and SpecAugment [30]. This ensures the baseline models
without KD are properly trained. The only changes we made from [20, 11] is to
add label smoothing, random noise augmentation, and random time shift
augmentation to the training pipeline. These training techniques may have
similar effects to cross-model knowledge distillation, e.g., label smoothing
may have a similar effect to the teacher’s soft label, random time shift
augmentation may have a similar effect to transferring the inductive bias of
translation invariance in CNN$\rightarrow$AST. We apply these training
techniques to both baseline and KD models to exclude the improvement that can
be achieved by such simple techniques, and thus more clearly identify the
improvement from the cross-model knowledge distillation. In practice, we find
these techniques slightly improve the performance of the baseline and KD
models. Therefore, our baseline model performance is slightly better than that
reported in [20, 11]. We also intentionally minimize the training setting
differences (e.g., mixup and augmentations) between the CNN and AST models to
ensure performance differences are due to the architecture, rather than the
training settings.

##  4 FSD50K Experiments

###  4.1 Which model is a good teacher?

We first explore the optimal teacher model for each student model.
Specifically, we use the KD setting mentioned in Section 2 and test a set of
(teacher, student) model pairs. Specifically, we consider the following
models: EfficientNet B0, B2, B6 [21] and DenseNet-121 [22] (CNN models); and
AST Tiny, Small, Base [11] (Transformer models). The reason for selecting
these models are: 1) EfficientNet-B2 and AST-Base are the best CNN and
Transformer models, and we want to see if KD can further improve their
performance; 2) AST-Base is larger than EfficientNet-B2 in terms of the number
of parameters. We include models with various sizes to clearly identify
whether the improvement of KD is due to different model sizes or different
model architecture; 3) We also include one different CNN architecture
(DenseNet) to check if the cross-model KD framework generalizes to different
CNN architectures.

For each (teacher, student) pair, we select the temperature
$\tau\in\\{1.0,2.5\\}$ based on the performance on the validation set, all
other knowledge distillation settings are identical for all (teacher, student)
pairs. The mean average precisions (mAPs) of the student model of various
(teacher,student) pairs on the FSD50K evaluation set are shown in Table III.
Key observations are as follows:

1\. CNNs and ASTs are good teachers for each other. While knowledge
distillation improves the student model performance in almost all settings
(denoted by $\dagger$ in Table III), we find that models always prefer a
_different_ teacher. Specifically, all CNN models (including DenseNet) achieve
the best performance when ASTs are the teacher; all AST models achieve the
best performance when CNNs are the teacher. Note that even though DenseNet and
EfficientNet models also have different architectures, knowledge distillation
between these two models leads to limited improvement. This demonstrates that
the inherent differences between CNNs and ASTs is crucial for the success of
knowledge distillation.

2\. For both directions, the student model matches or outperforms its teacher.
Usually, in knowledge distillation, the student model gets closer to, but is
still weaker than, its teacher model. However, we find that for both
AST$\rightarrow$CNN and CNN$\rightarrow$AST, the student model performs better
than its teacher in most cases (denoted by $\ast$ in Table III), e.g., the
best KD-CNN and KD-AST model achieves 58.2 and 61.7 mAP while their teacher
model achieves only 56.3 and 55.7, respectively. This demonstrates that the
student models are not simply copying the teacher model, but are able to
combine the strength of the teacher with its own strength.

Figure 2: Performance correlation between teacher and student model when the
model architectures are fixed. We save the checkpoint teacher models at epoch
0, 5, 10, 20, 30, 40, and 50 during teacher model training, then train student
models with these checkpoint teacher models until convergence. We observe that
for a fixed teacher model architecture, a stronger teacher trained with more
epochs generally leads to a better student. Nevertheless, the student models
outperform the no-KD baselines (the dashed lines) even when their teacher
models are trained with only 5 epochs with poor performance. The examples we
show in this figure are the optimal (teacher,student) pairs, i.e.,
AST$\rightarrow$CNN: AST-Tiny$\rightarrow$EfficientNet-B2;
CNN$\rightarrow$AST: EfficientNet-B0$\rightarrow$AST-Base.

3\. The strongest teacher is not the best teacher. Normally in knowledge
distillation, the student model performance correlates with the teacher
performance. However, we find for both AST$\rightarrow$CNN and
CNN$\rightarrow$AST, the best teacher is not the strongest teacher. Instead,
both CNN and AST perform better with a smaller (and weaker) teacher. We
further verify that this is not because a stronger teacher requires a higher
temperature, e.g., for AST-Base (strongest teacher)
$\rightarrow$EfficientNet-B2, using a temperature of 2.5 (used in Table III)
leads to an mAP of 57.3 while further increasing the temperature to 3.0 leads
to a worse mAP of 57.1. For the same student model EfficientNet-B2, the
smaller teacher AST-Tiny leads to a better mAP of 58.2. We conjecture it is
because smaller models have similar properties (e.g., inductive bias) with
larger models of the same type but are less overfit, and thus are better
teachers.

However, as shown in Figure 2, when the teacher model architecture is fixed,
we find a stronger teacher trained with sufficient epochs generally leads to
better student performance than a weaker teacher that is not adequately
trained. Nevertheless, the student models still outperform the no-KD baselines
even when their teacher models are trained with only 5 epochs with poor
performance. This further confirms that the student is not simply copying the
teacher’s behavior as copying the behavior of a teacher weaker than the
student itself can only cause a performance drop, but our experiments show
that cross-model KD still leads to a performance improvement in this case.

4\. Self-KD leads to smaller or no improvement. We observe that performance
improvement can also sometimes be achieved by using the same teacher and
student architecture (the main diagonal of Table III), particularly for AST
models. This is as expected because 1) the soft labels helps optimization of
the student model, and 2) since we use strong data augmentation (e.g.,
SpecAugment, mixup, random noise, and time shift) during training, knowledge
distillation with the same model architecture can have a similar effect to
self-training [38]. Nevertheless, the improvement led by self-knowledge
distillation is much smaller than cross-model knowledge distillation, which
demonstrates the importance of using a different teacher for knowledge
distillation.

5\. Iterative knowledge distillation does not further improve model
performance. Since cross-model knowledge distillation works bi-directionally,
the performance of both KD-CNN and KD-AST are better than no-KD baselines.
Will using KD models instead of no-KD models as teachers lead to even better
students? The answer is no, as shown in Table IV. While KD models (with their
optimal teachers) are stronger, they are worse teachers than the no-KD models.
This again proves our finding that a stronger teacher is not necessarily a
better teacher as the student is not simply copying the teacher’s behavior.

To summarize, in this section, we find both CNN and AST models prefer a
_small_ and _different_ teacher. Cross-model knowledge distillation is beyond
simple behavior copying but more about transferring and combining the strength
of the teacher and student models as the performance of the teacher is
surprisingly not important.

TABLE IV: Performance of iterative knowledge disitllation. Though the KD models are stronger than no-KD models, they are worse teachers. Therefore, iterative knowledge distillation cannot further improve the performance. | Student
---
Model
| Teacher
---
Model
| Teacher
---
Performance
| Student
---
Performance
No KD
EfficientNet-B2 | - | - | 56.2
AST-Base | - | - | 57.6
KD Iteration 1
EfficientNet-B2 | AST-Tiny w/o KD | 56.3 | 58.2
AST-Base | EfficientNet-B0 w/o KD | 55.7 | 61.7
KD Iteration 2
EfficientNet-B2 | AST-Tiny w/ KD | 58.6 | 58.0
AST-Base | EfficientNet-B0 w/ KD | 57.4 | 61.4

###  4.2 Impact of Knowledge Distillation Settings

TABLE V: Student model performance on the FSD50K evaluation set due to various
knowledge distillation settings. We conduct this experiment with the optimal
teacher student pairs, i.e., AST$\rightarrow$CNN: AST-
Tiny$\rightarrow$EfficientNet-B2;
CNN$\rightarrow$AST: EfficientNet-B0$\rightarrow$AST-Base. KD Setting | AST$\rightarrow$CNN | CNN$\rightarrow$AST
---|---|---
No KD Baseline | 56.2 | 57.6
KD Temperature $\tau$
$\tau$ = 0.5 | 56.7 | 60.9
$\tau$ = 1.0 | 57.2 | 61.7
$\tau$ = 2.5 | 58.2 | 61.2
KD Balancing Coefficient $\lambda$
$\lambda$ = 0 (No ground truth label) | 56.4 | 59.7
$\lambda$ = 0.25 | 58.0 | 60.9
$\lambda$ = 0.5 (Default) | 58.2 | 61.7
$\lambda$ = 0.75 | 57.2 | 61.3
$\lambda$ = 1.0 (No KD) | 56.2 | 57.6
Teacher Type
Conventional Teacher | 58.2 | 60.4
Consistent Teacher (Default) | 58.2 | 61.7
Separate Distillation Head
Yes | 56.9 | 61.4
No (Default) | 58.2 | 61.7

We next study the performance impact of the knowledge distillation setting.
Specifically, we focus on the optimal (teacher,student) pair found in the
previous section, i.e., AST-Tiny$\rightarrow$EfficientNet-B2 and
EfficientNet-B0$\rightarrow$AST-Base, and set the knowledge distillation
setting mentioned in Section 2.4 (marked as default in Table V) as the base
setting, and then change one factor at a time to observe the performance
change. We show the results in Table V.

KD Temperature $\tau$: In Section 4.1, we vary the temperature $\tau$ from
{1.0, 2.5}. We find a higher temperature of 2.5 is better for
AST$\rightarrow$CNN and a lower temperature of 1.0 is better for all other
(teacher,student) pairs. Using an even lower temperature (or “hard” knowledge
distillation [17] used in the vision domain) cannot lead to good performance.
This is due to audio classification usually being a multi-label classification
task (each audio clip usually has more than one label), which is different
from the single-label image classification task.

KD Balancing Coefficient $\lambda$: As shown in Table V, balancing the ground
truth label and teacher’s soft prediction with a factor of $\lambda=0.5$ leads
to the optimal performance for both AST$\rightarrow$CNN and
CNN$\rightarrow$AST. Merely using the soft prediction of the teacher model can
still improve the model, but with a much smaller performance gain.

Consistent teacher makes a difference: We compare the student model
performance with a consistent teacher [29] and a conventional teacher [28].
The consistent teacher generates predictions based on the same augmented input
with the student, while the conventional teacher generates prediction based on
the original, unaugmented input. We find that in the CNN$\rightarrow$AST
direction, a consistent teacher leads to a noticeable 1.3 mAP improvement
while for AST$\rightarrow$CNN, it does not hurt performance. Therefore,
consistent teaching should be used as a default setting.

Separate Knowledge Distillation Head: In the vision domain, knowledge
distillation with a separate distillation head [17] was shown to have better
performance than conventional knowledge distillation [28]. We compare these
two settings for the audio classification task. Specifically, for a KD model
with a separate distillation head, for CNN$\rightarrow$AST, we follow [17] to
append a distillation token [DIS] to the AST token sequence and use a separate
linear head to map the output of [DIS] token to the label space. During
knowledge distillation, the output of the original [CLS] token is used to
generate the loss versus the ground truth label $Loss_{g}$ while the output of
the new [DIS] token is used to generate the loss versus the teacher prediction
$Loss_{d}$. Similarly, for AST$\rightarrow$CNN, we use two independent linear
heads to map the representation of the penultimate layer to the label space,
we then use the output of one head to generate $Loss_{g}$ and another to
generate $Loss_{d}$. As shown in Table V, while the more complex separate
distillation head solution still outperforms the baseline model, the
performance gains are smaller for both AST$\rightarrow$CNN and
CNN$\rightarrow$AST for audio classification, which is different from the
finding in the vision domain. We, therefore, stick to the conventional
knowledge distillation [28] and do not use a separate distillation head.

To summarize, throughout Table V, we find that knowledge distillation with any
setting improves model performance. But the optimal setting is quite different
from that of the vision domain [17]. In general, multi-label audio
classification knowledge distillation requires an appropriate temperature
larger than 1, a balance between the ground truth label and teacher model
prediction, and consistent teaching [29]. It is sufficient to just follow
conventional knowledge distillation to generate both $Loss_{g}$ and $Loss_{d}$
using the original student model prediction without adding a separate
distillation head.

(a)  (b)

Figure 3: Left: Learning curve of the CNN model (EfficientNet-B2) and AST model (AST-Base) trained with and without KD on FSD50K. $\star$ denotes the epoch that the model achieves the best performance on the validation set. Each experiment is run three times, and the standard deviation is shown in the shade. The learning rate is cut in half when the model performance on the validation set does not improve for 2 epochs so the three runs of each experiment have slightly different learning rate schedulers and the performance variance is relatively large around the epochs of the learning rate change. Right: Training and evaluation mAP of AST and CNN models with and without KD. KD models have a higher evaluation mAP but lower training mAP, indicating they are less overfit. TABLE VI: Prediction similarity in Pearson correlation coefficient (upper part) and model representation similarity in singular vector canonical correlation analysis [39] (lower part) of various model pairs. Similarity value between KD model with corresponding teacher is underlined. |  | EfficientNet-B0
---
w/o KD
| EfficientNet-B2
---
w/o KD
| EfficientNet-B2
---
w/ KD
| AST-Tiny
---
w/o KD
| AST-Base
---
w/o KD
| AST-Base
---
w/ KD
Prediction Correlation
EfficientNet-B2 w/o KD | 0.861 | 1.000 | 0.870 | 0.804 | 0.791 | 0.865
EfficientNet-B2 w/ KD | 0.862 | 0.870 | 1.000 | 0.833 | 0.810 | 0.882
AST-Base w/o KD | 0.786 | 0.791 | 0.810 | 0.844 | 1.000 | 0.880
AST-Base w/ KD | 0.880 | 0.865 | 0.882 | 0.880 | 0.880 | 1.000
SVCCA
EfficientNet-B2 w/o KD | 0.720 | 1.000 | 0.734 | 0.705 | 0.651 | 0.706
EfficientNet-B2 w/ KD | 0.724 | 0.734 | 1.000 | 0.718 | 0.695 | 0.721
AST-Base w/o KD | 0.651 | 0.651 | 0.695 | 0.779 | 1.000 | 0.784
AST-Base w/ KD | 0.746 | 0.706 | 0.721 | 0.807 | 0.784 | 1.000

###  4.3 How Does CMKD Improve Performance?

In the previous two sections, we see that knowledge distillation between CNN
and AST models noticeably boosts model performance. It is natural to ask why
it works so well. In this section, we try to answer this question with a
series of probing tests and analysis. We focus on the optimal
(teacher,student) pair found in Section 4.1, i.e., AST-
Tiny$\rightarrow$EfficientNet-B2 for AST$\rightarrow$CNN and
EfficientNet-B0$\rightarrow$AST-Base for CNN$\rightarrow$AST, respectively.
Key findings are as follows:

Knowledge distillation makes the model learn faster and be less prone to
overfitting. We compare the learning curve of the CNN model (EfficientNet-B2)
and the AST model (AST-Base) trained with and without knowledge in Figure 3
(left). We find that for both CNN and AST models, knowledge distillation helps
the models learn faster and be less prone to overfitting. The KD models
outperform the no-KD models from the first epoch, and consistently improves
until the end of training while the performance of no-KD models saturate much
earlier. For example, at the epoch that the no-KD AST achieves the best
performance (the $21^{st}$ epoch), KD AST already outperforms no-KD AST by 2.9
mAP with its faster learning; from the $21^{st}$ epoch to the last epoch, KD
AST further enlarges its advantage over no-KD AST from 2.9 mAP to 4.1 mAP. In
Figure 3 (right), we show the training and evaluation mAP of AST and CNN
models with and without KD. Both KD-AST and KD-CNN models have a higher
evaluation mAP but lower training mAP, confirming they are less overfit.

Knowledge distillation narrows the gap between two classes of models. Next, we
analyze the similarity between CNN and AST models before and after cross-model
knowledge distillation. Specifically, we show the prediction similarity in
Pearson correlation coefficient (PCC) and model representation similarity in
singular vector canonical correlation analysis (SVCCA) [39]. Since FSD50K is a
multi-label classification task, we average the PCC of the predictions of each
evaluation sample of the two models being compared as the prediction
similarity value. SVCCA is a technique to measure the similarity of two
representations. It has nice properties such as affine transform invariance
and supports the comparison of two representations of two different
dimensions. We use SVCCA with the singular vector that explains 30% variance
of each representation as the model representation similarity.

As shown in Table VI, we find that cross-model distillation not only makes the
student model more similar to its teacher, but also to models of the same
class with its teacher. For example, for EfficientNet-B0$\rightarrow$AST-Base,
after knowledge distillation, not only the prediction similarity score between
AST-Base (student) and Efficient-B0 (teacher) increases from 0.786 to 0.880,
but also the similarity score between AST-Base (student) and EfficientNet-B2
(another CNN model with different architecture and independent training
process) increases from 0.791 to 0.865. This phenomenon is consistent for both
CNN and AST models, and for both prediction and model representation
similarity, indicating that cross-model knowledge distillation not only makes
the specific teacher and student model instances similar, but actually narrows
the gap between two classes of models of CNN and AST.

Figure 4: Model performance drop (%) on input with various type and level of
input contamination during inference. Upper left: random frequency masking,
upper right: randome time masking; lower left: random time shift; lower right:
random noise. Each experiment is run three times, and the stand deviation is
shown in the shade. Figure 5: Comparison of the robustness of AST models
trained with self-KD and CMKD to input time shifting. We show the mean and
standard deviation of the performance drop of AST models trained with various
CNN teachers (EfficientNet-B0, B2, B6, and DenseNet 121) in red (CMKD), and
those of AST models trained with various AST models (AST-Tiny, Small, and
Base) in blue (self-KD). AST models trained with CMKD are more time shift-
invariant than those trained with self-KD, indicating CMKD indeed transfers
the translation invariance property from CNN to AST.

Knowledge distillation improves model robustness. As mentioned in Section 2.3,
the major properties of CNN models are _translation invariance_ and _locality_
while one major property of AST is _non-locality_ as it looks at all of the
context of earlier layers. Therefore, CNN models are more robust to input
shifting but are more prone to local feature contamination while AST models
are the opposite [40]. We verify this point by evaluating the models on a
contaminated FSD50K evaluation set. As shown in Figure 4, when knowledge
distillation is not applied, the CNN model is indeed less impacted by random
time-shifting (input shifting) while the AST model is less impacted by random
frequency masking and random noise (local feature contamination).

Interestingly, we find cross-model knowledge distillation not only improves
models’ mean average precision on clean input, but also enhances their
robustness to contaminated input, e.g., KD-AST inherits its CNN teacher’s
advantage of translation invariance and is significantly more robust to random
time shifts. Similarly, the CNN model inherits its AST teacher’s property and
is more robust to frequency masking and random noise. It is worth noting that
random time-shifting and noise are used as training data augmentation for all
models, which already encourages the models to be robust. Therefore, the
robustness improvement led by cross-model KD might be underestimated.

Are such robustness improvements led by the knowledge transferred from the
teacher, or are they a product of the knowledge distillation process itself
(e.g., the effect of the soft label and consistent teaching)? To answer this
question, we further compare the robustness to random time-shifting of AST
models trained with cross-model KD and self-KD. For self-KD AST, we evaluate
the AST model trained with AST-Tiny, AST-Small, and AST-Base teachers. For
cross-model KD, we evaluate AST model trained with EfficientNet-B0,
EfficientNet-B2, EfficientNet-B6, and DenseNet-121 teachers. As shown in
Figure 5, both cross-model KD and self-KD lead to a robustness improvement,
but cross-model KD is noticeably more effective, demonstrating both the
knowledge transferred from the teacher and the KD process contributes to
robustness improvements. Our finding is consistent with previous work [41]
that showed knowledge distillation can transfer model properties such as
inductive bias and allow models to get advantages from each other, which is
one major reason for the success of cross-model knowledge distillation.

Knowledge distillation changes the attention map of AST. Finally, we observe
an interesting phenomenon that KD noticeably changes the attention map of the
AST model. As discussed in Section 2.3, the AST model’s receptive field is
learnable and can be arbitrarily large while the receptive field of the CNN
model is manually set and fixed. In Figure 6, we visualize the AST model’s
mean attention distance of each layer in the frequency and time dimension. The
attention distance is the distance between a given query token and all other
tokens averaged by the attention weight between that query token and each
other token, which is analogous to the receptive field. As expected, we
observe that the higher layers of AST have a larger mean attention distance,
indicating AST can learn hierarchical features from the data. Interestingly,
we find KD changes the attention behavior of AST. Particularly, the attention
distance of the second layer almost doubled, and the $7^{th}$-$10^{th}$ layers
slightly decrease. This means the KD-AST model captures more global
information at lower layers, which potentially is one key reason for the
performance improvement.

Figure 6: AST model’s mean attention distance of each layer in frequency
(upper) and time (lower) dimension before and after KD. KD noticeably changes
the attention behavior of the $2^{nd}$ layer and $7-10^{nd}$ layers.

###  4.4 Multiple-Teacher Knowledge Distillation

In Sections 4.1-4.3, we use a _single_ teacher model during knowledge
distillation. In this section, we further explore whether using _multiple_
teachers [19] improves performance. Specifically, we consider two types of
multi-teacher knowledge distillation: _1) Multi-Loss KD_ : the student model
is trained with multiple distillation losses, each corresponds to a teacher
model. Each distillation loss is balanced with the coefficient of
$(1-\lambda)/n$, where $n$ is the number of teachers. _2) Ensemble KD_ : The
predictions of the teachers are averaged and a single distillation loss is
applied. For both CNN and AST students, we test a set of different
combinations of teachers. We mainly consider teacher committees that consist
of models of a different class with the student model based on our finding in
Section 4.1. We also try using both AST and CNN as the teachers.

As shown in Table VII, multi-teacher knowledge distillation further improves
the performance of the AST model. While any teacher combination and multi-
teacher KD setting leads to better performance than single-teacher KD, we
again observe the importance of diversity in the teacher models. When the
teacher committee consists of different model architectures (EfficientNet and
DenseNet) and the teacher models’ type is different from the student model’s
type (CNN and AST), KD leads to the best performance. Multi-teacher KD,
however, does not work for AST$\rightarrow$CNN.

TABLE VII: Results with multi-teacher knowledge distillation. We consider various combinations of teachers and two types of multi-teacher knowledge distillation: _1) Multi-Loss KD_ : the student model is trained with multiple distillation losses, each corresponds to a teacher model and is balanced with the coefficient of $(1-\lambda)/n$, where $n$ is the number of teachers. _2) Ensemble KD_ : The predictions of the teachers are averaged and a single distillation loss is applied. Student Model | Teacher Model | Multi-Loss | Ensemble
---|---|---|---
EfficientNet-B2 |  AST-Tiny | 58.2
EfficientNet-B2 |  |  AST-Tiny
---
\+ AST-Small
58.0 | 57.9
EfficientNet-B2 |  |  AST-Tiny
---
\+ AST-Small
\+ AST-Base
57.4 | 58.2
EfficientNet-B2 |  |  AST-Tiny
---
\+ EfficientNet-B0
57.7 | 58.1
AST-Base |  EfficientNet-B0 | 61.7
AST-Base |  |  EfficientNet-B0
---
\+ EfficientNet-B2
61.8 | 62.3
AST-Base |  |  EfficientNet-B0
---
\+ DenseNet-121
62.3 | 62.2
AST-Base |  |  EfficientNet-B0
---
\+ AST-Tiny
62.1 | 61.9

###  4.5 KD with CNN-Attention Hybrid Model

In previous sections, we focus on cross-model KD between _pure_ CNN and
Transformer models. Recently, a few _CNN-attention hybrid models_ have been
proposed that combine CNN and attention models by integrating their model
components [42, 20, 43, 44, 45]. How does cross-model KD work with such CNN-
Attention hybrid models? In this section, we consider a CNN-Attention hybrid
model proposed in [20]. The hybrid model uses EfficientNet as the feature
extractor and adds a 4-headed self-attention attention module on top of the
EfficientNet. Without knowledge distillation, the hybrid model performs
slightly better than the pure CNN model on FSD50K. We apply the same default
KD setting and study knowledge distillation of Hybrid$\rightarrow$CNN and
Hybrid$\rightarrow$AST (hybrid model as teacher) as well as
CNN$\rightarrow$Hybrid and AST$\rightarrow$Hybrid (hybrid model as student).
For the hybrid model, we consider a small model hybrid-B0 and a large model
hybrid-B2 with EfficientNet-B0 and EffcientNet-B2 feature extractors,
respectively.

TABLE VIII: Results of CNN-attention hybrid model knowledge distillation on FSD50K. CNN-Attention hybrid model is good teacher of both pure CNN and AST models. Conversely, the performance of CNN-Attention hybrid model can also be noticeably improved when trained with KD with both pure CNN and AST models. | KD Setting
---
(Teacher $\rightarrow$ Student)
| Teacher
---
Performance
| Student Performance
---
(No KD$\rightarrow$KD)
Hybrid-B0 $\rightarrow$ EfficientNet-B2 | 56.3 | 56.2$\rightarrow$57.9
Hybrid-B0 $\rightarrow$ AST-Base | 56.3 | 57.6$\rightarrow$61.9
Hybrid-B2 $\rightarrow$ EfficientNet-B2 | 56.2 | 56.2$\rightarrow$57.4
Hybrid-B2 $\rightarrow$ AST-Base | 56.2 | 57.6$\rightarrow$61.7
EfficientNet-B0 $\rightarrow$ Hybrid-B2 | 55.7 | 56.2$\rightarrow$57.9
AST-Tiny $\rightarrow$ Hybrid-B2 | 56.3 | 56.2$\rightarrow$57.9

As shown in Table VIII, we find the CNN-attention hybrid model is a good
teacher of both pure CNN and AST models. Specifically, Hybrid$\rightarrow$AST
leads to a mAP of 61.9, which is slightly better than CNN$\rightarrow$AST
(61.7 mAP); Hybrid$\rightarrow$CNN also achieves similar performance with
AST$\rightarrow$CNN. Interestingly, we find the small hybrid model is also a
better teacher than the large model, which is consistent with our finding in
Section 4.1. Conversely, the performance of the CNN-attention hybrid model can
also be noticeably improved when trained with either a pure CNN or an AST
teacher. Nevertheless, its performance is worse than KD-AST and KD-CNN.

To conclude, CNN-attention hybrid models are in the middle of pure CNN models
and pure attention models. It is a good teacher of both CNN and AST, and
conversely, its performance also boosts when trained with a CNN or AST
teacher. Nevertheless, despite the more complex architecture, the hybrid model
does not outperform either KD-CNN or KD-AST, demonstrating the advantage in
simplicity and effectiveness of the proposed CNN-AST cross-model knowledge
distillation.

###  4.6 Comparison with State-of-the-art

Finally, we compare the best KD models with state-of-the-art models. We
summarize the performance improvement on the FSD50K evaluation set led by KD
in Table IX. As with previous work [20, 11], we find the best CNN model is
EffcientNet-B2 and the best AST model is AST-Base on FSD50K. For
EfficientNet-B2, we find its optimal teacher is AST-Tiny; for AST-Base, we
find its optimal KD setting is using multiple CNN teachers (EfficientNet-B0
and DenseNet-121). In previous sections, we report the evaluation set mAP of
the model checkpoint that achieves the highest mAP on the validation set. In
addition to that, here we also report the performance of the model with weight
averaging (WA) [50] and ensemble models. For weight averaging, we average all
weights of the model checkpoints of multiple epochs, the start averaging epoch
is selected using the validation set and the end averaging epoch is the last
epoch of training (the $50^{th}$ epoch). Weight averaging does not increase
the computation overhead of training and inference. For ensemble, we average
the predictions of three models trained with the same setting but different
random seeds. As with [20], we find weight averaging and ensembles both
improve the performance of all models.

After applying weight averaging, the KD-CNN and KD-AST model achieves
58.5$\pm$0.2 and 62.9$\pm$0.1 mAP respectively, which both significantly
outperform all existing models (summarized in Table X). After ensembling, the
KD-CNN and KD-AST model achieves an even better performance of 60.7 and 63.4
mAP.

TABLE IX: Comparison of the mAP of CNN and AST models trained with and without KD in weight averaging (WA) and ensemble settings on FSD50K. Student Model | KD |  | Best Model
---
on Validation Set
w/ WA | Ensemble
EfficientNet-B2 | No | 56.2$\pm$0.5 | 57.6$\pm$0.3 | 59.4
EfficientNet-B2 | Yes | 58.2$\pm$0.1 | 58.5$\pm$0.2 | 60.7
AST-Base | No | 57.6$\pm$0.4 | 60.1$\pm$0.2 | 61.1
AST-Base | Yes | 62.3$\pm$0.1 | 62.9$\pm$0.1 | 63.4
TABLE X: Comparison with previous methods on FSD50K. Since previous methods do not use ensemble, we compare with them using our best _single_ model for fairness. Model | Mean Average Precision (mAP)
---|---
FSD50K Baseline [32] | 43.4
Wav2CLIP [46] | 43.1
YAMNet [47] | 50.4
Audio Transformers [48] | 53.7
Shift-Invariant CNN [49] | 54.1
PSLA [20] | 56.7
AST [11] | 57.6
KD-CNN | 58.5$\pm$0.2
KD-AST | 62.9$\pm$0.1

##  5 AudioSet and ESC-50 Experiments

Throughout Section 4, we focus on FSD50K dataset due to the reasons mentioned
in Section 3.1. In order to study the generalization of the proposed KD
strategy, in this section, we further evaluate on AudioSet and ESC-50.

###  5.1 AudioSet Experiments

AudioSet [33] is a collection of over 2 million 10-second audio clips excised
from YouTube videos and labeled with the sounds that the clip contains from a
set of 527 labels. The balanced training, full training, and evaluation set
contain 22k, 2M, and 20k samples, respectively. To study the impact of
training data size, we conduct experiments on both the balanced and full
training set and evaluate on the AudioSet evaluation set. As the goal is to
study the generalization of the proposed method, we do not search the KD
hyperparameters with AudioSet but just re-use the optimal KD setting found
with the FSD50K dataset in Table V. We find small models such as
EfficientNet-B0 and AST-Tiny do not perform well with AudioSet as the
dimension of the penultimate layer representation is close to or smaller than
the label dimension of AudioSet (527). Therefore, we focus on larger models
EfficentNet-B2 and AST-Base. As with FSD50K, we report the performance in mAP
of 1) model of the last training epoch, 2) model with weight averaging (WA),
and 3) ensemble three models trained with same setting but different random
seeds. We show the results in Table XI. The key findings are as follows:

TABLE XI: Performance in mAP of CNN and AST models on the AudioSet evaluation set before and after knowledge distillation. $\diamond$ denotes the model is trained with extra 5 epochs (10 epochs in total). Student Model | KD | Last Epoch | w/ WA | Ensemble
---|---|---|---|---
Balanced AudioSet Training Set (20K)
EfficientNet-B2 | No | 31.3$\pm$0.4 | 31.5$\pm$0.5 | 33.7
EfficientNet-B2 | Yes | 34.4$\pm$0.2 | 34.5$\pm$0.2 | 35.6
AST-Base | No | 33.8$\pm$0.1 | 35.3$\pm$0.1 | 36.6
AST-Base | Yes | 36.2$\pm$0.2 | 36.4$\pm$0.2 | 37.6
Full AudioSet Training Set (2M)
EfficientNet-B2 | No | 44.1$\pm$0.1 | 44.4$\pm$0.1 | 46.2
EfficientNet-B2 | Yes | 45.3$\pm$0.1 | 45.4$\pm$0.1 | 46.2
AST-Base | No | 45.4$\pm$0.1 | 46.1$\pm$0.1 | 47.8
AST-Base | Yes | 46.8$\pm$0.1 | 46.8$\pm$0.1 | 48.1
AST-Base⋄ | Yes | 47.1$\pm$0.0 | 47.1$\pm$0.0 | 48.2
TABLE XII: Comparison with previous and concurrent methods on AudioSet. Since not all previous methods use ensemble and ensemble settings are usually different, we compare with them using our best single model for fairness. $\dagger$ denotes the experiment uses 32kHz audio, which can lead to about 0.4 mAP improvement [51] compared with using 16kHz audio. $\star$ denotes concurrent efforts (_arxiv_ preprint). We use a more commonly used and computationally efficient sampling rate of 16kHz, but still, our KD-AST outperforms or matches all previous methods while our KD-CNN outperforms all previous CNN and CNN-attention models. Model | Type | mAP
---|---|---
AudioSet Baseline [33] | CNN | 31.4
TALNet [52] | CNN-Attention | 36.2
DeepRes [43] | CNN-Attention | 39.2
PANNs [51] † | CNN | 43.9
PSLA [20] | CNN-Attention | 44.4
AST [11] | Transformer | 45.9
HTS-AT [13]†,⋆ | Transformer | 47.1
PaSST [12]†,⋆ | Transformer | 47.1
KD-CNN | CNN | 45.4$\pm$0.1
KD-AST | Transformer | 47.1$\pm$0.0
TABLE XIII: Accuracy of CNN and AST models on ESC-50. $a\rightarrow b$ denotes that the model achieves an accuracy of $a$ and $b$ without and with KD, respectively; $\uparrow$ denotes KD improves the performance. |  |  | ESC Fine-Tuning KD Teacher Model
---|---|---|---
Student Model |  Student Model AudioSet Pretrain |  KD in Student AudioSet Pretrain | No Pretrain | AudioSet Pretrain w/o KD | AudioSet Pretrain w/ KD
Teacher (AST-Base) Performance | 89.23 | 96.07 | 96.45
EfficientNet-B2 | No | - | 88.95$\rightarrow$89.58 $\uparrow$ | 88.95$\rightarrow$89.80 $\uparrow$ | 88.95$\rightarrow$89.87 $\uparrow$
EfficientNet-B2 | Yes | No | 96.25$\rightarrow$96.37 $\uparrow$ | 96.25$\rightarrow$96.47 $\uparrow$ | 96.25$\rightarrow$96.45 $\uparrow$
EfficientNet-B2 | Yes | Yes | 96.32$\rightarrow$96.42 $\uparrow$ | 96.32$\rightarrow$96.48 $\uparrow$ | 96.32$\rightarrow$96.50 $\uparrow$
Teacher (EfficientNet-B2) Performance | 88.95 | 96.25 | 96.32
AST-Base | No | - | 89.23$\rightarrow$89.72 $\uparrow$ | 89.23$\rightarrow$90.27 $\uparrow$ | 89.23$\rightarrow$90.07 $\uparrow$
AST-Base | Yes | No | 96.07$\rightarrow$96.08 $\uparrow$ | 96.07$\rightarrow$96.22 $\uparrow$ | 96.07$\rightarrow$96.02 $\downarrow$
AST-Base | Yes | Yes | 96.45$\rightarrow$96.57 $\uparrow$ | 96.45$\rightarrow$96.67 $\uparrow$ | 96.45$\rightarrow$96.80 $\uparrow$

CMKD works out-of-the-box. Though we did not search the KD setting with
AudioSet, KD works well on AudioSet for both CNN$\rightarrow$AST and
AST$\rightarrow$CNN, demonstrating the proposed cross-model knowledge
generalizes for audio classification tasks. In addition, training KD models
with more epochs can lead to further performance improvement, which is
consistent with our finding that KD models are less prone to overfitting. For
example, if we train the AST-Base model with 5 more epochs (10 epochs in
total), it achieves an mAP of 47.1, which is 0.3 mAP better than the model
trained with 5 epochs. Training no-KD models with more epochs does not lead to
such performance improvement.

KD leads to larger improvement on smaller datasets. In all settings, KD is
more effective when the model is trained with the smaller balanced training
set. This is as expected because AST and CNN models get closer with more
training data and cross-model knowledge distillation thus plays a smaller
role. However, even with the massive 2M training data, KD still leads to
noticeable performance improvement.

The advantage of KD narrows after weight averaging and ensemble. This finding
is consistent in FSD50K and AudioSet experiments. We conjecture this is
because knowledge distillation limits the _diversity_ of model checkpoints of
different training epochs and trained with different random seeds as the soft
labels from the trained teacher model reduce the student model optimization
oscillations.

Comparison with State-of-the-art As a consequence of cross-model knowledge
distillation, as shown in Table XII, our KD-AST model achieves a new state-of-
the-art of 47.1$\pm$0.0 mAP on AudioSet, which outperforms the previous state-
of-the-art of 45.9 mAP [11]. Our KD-CNN model also outperforms all previous
CNN and CNN-attention models. Our KD-AST matches the performance of two
concurrent444[13, 12] are arxiv preprints released after we started this work.
efforts [13, 12] that aim to improve the AST architecture and training
efficiency. Note that both [13, 12] use a sampling rate of 32kHz, which is
higher than the more commonly used and computationally efficient 16kHz
sampling rate used in our experiments. In [51], it is shown that using a 32kHz
sampling rate can lead to an improvement of about 0.4 mAP. In other words, our
KD-AST matches [13, 12] with lower resolution input. Also, CMKD is orthogonal
to model architecture improvement, as we showed in Section 4.1 and 4.5, CMKD
can improve all models we evaluated, thus it can be potentially combined with
[13, 12] to further improve model performance.

###  5.2 ESC-50 Experiments

Finally, we conduct experiments on ESC-50 to study KD on a tiny dataset and
transfer learning. The ESC-50 [34] dataset consists of 2,000 5-second
environmental audio recordings organized into 50 classes. We follow the
standard 5-fold cross-validation to evaluate our model, repeat each experiment
three times, and report the mean accuracy. Note for KD experiments, the
teacher model is trained with the same data folds as the student model during
cross-validation.

For both teacher and student models, we consider three settings: 1) model
without audio domain pretraining; 2) model pretrained on full AudioSet, but
during AudioSet training, KD is not applied; and 3) model pretrained on full
AudioSet and CMKD is applied during pretraining. We train or fine-tune the
models on ESC-50 with and without KD and report the performance change in
Table XIII. In other words, we study the impact of KD in both the pretraining
and fine-tuning stages. As with the AudioSet experiments, we re-use the
optimal FSD50K KD setting in Table V but increase the temperature to {5.0,
6.0} as the model performance is already high and the prediction of the
teacher needs to be further softened. Key findings are as follows:

KD leads to small but consistent improvement: In almost all settings, KD
consistently enhances the accuracy of the model. But compared with FSD50K and
AudioSet, we find the improvement on ESC-50 is smaller. This is potentially
due to the accuracy of the model being already high on ESC-50, thus the
distinction between the teacher prediction and ground truth label is small.

The advantage of KD in pretraining transfers. As discussed in Section 5.1,
models trained with KD have a performance advantage over models trained
without KD. When we fine-tuned the AudioSet pretrained model on ESC-50, we
find such an advantage still exists. Specifically, no matter if KD is applied
in the fine-tuning stage, the model trained with KD in the pretraining stage
always performs better.

Nevertheless, for teacher models, KD in the pretraining stage does not impact
its student’s performance. This is consistent with our finding in Section 4.1
that the performance of the teacher is not important. In fact, even for
teacher models with and without AudioSet pretraining that have a great
performance gap, their student models only have a small performance
difference.

The state-of-the-art model [11] achieves 95.6$\pm$0.4% on ESC-50. Our KD-AST
and KD-CNN achieves 96.8$\pm$0.1% and 96.5$\pm$0.3%, respectively, both
outperform [11]. Two concurrent efforts [12, 13] on improving AST architecture
report 96.8% and 97.0% on ESC-50. Nevertheless, their results are based on
32kHz audio while our experiment is based on 16kHz audio. Using a higher
sampling rate is known to improve the model performance for audio
classification [51], but it could also increase the storage and computation
cost.

##  6 Related Work

CNN and Transformer Convolution neural network [4, 54, 21, 55] is a class of
neural network model that typically has a hierarchical architecture composed
of convolution layers, pooling layers, and fully connected layers. In the past
decade, convolution neural networks have been the de-facto model for vision
and audio tasks. Transformers [14] is another class of neural network that is
solely based on the self-attention mechanism. It was originally proposed for
natural language processing tasks [15, 56], but recently, the Vision
Transformer [16] and Audio Spectrogram Transformer [11] have been shown to
outperform deep learning models constructed with convolutional neural networks
(CNNs) on vision and audio tasks, respectively, thus extending the success of
Transformers to the vision and audio domain.

Besides pure CNN and pure attention based models, there is another line of
research on combining CNN and attention models by integrating convolution
operations into attention models, or conversely, introducing attention
mechanism into CNN models [20, 57, 58, 59, 60, 61, 62, 63, 64, 45, 44]. Unlike
these efforts on CNN-attention hybrid models, we show the strength of CNN and
Transformer models can be combined more efficiently by simple knowledge
distillation without model architecture modification. Both our KD-CNN and KD-
AST model outperform the hybrid model for the audio classification tasks. We
also show the proposed cross-model knowledge distillation can also be applied
to CNN-attention hybrid models.

Audio Classification [65] is a task of identifying sound events that occur in
a given audio recording. In recent years, audio classification research has
moved from small _single-label_ datasets such as ESC-50 [34] to much larger
_multi-label_ datasets such as FSD50K [32] and AudioSet [33]. With the support
of large and realistic datasets, many new neural network models have been
proposed including CNN models (e.g., [33, 51, 52]), CNN-attention models
(e.g., [42, 20, 43, 44, 45]), and pure attention based models [11, 13, 12]. In
particular, the pure-attention-based Audio Spectrogram Transformer (AST) and
its variants [11, 13, 12] were shown to outperform CNN and CNN-attention based
models and achieved state-of-the-art on multiple audio classification
datasets. Unlike these previous efforts, we do not propose a new model
architecture but instead, focus on knowledge distillation between these
existing models. We show that noticeable performance improvement can be
obtained for existing models of all categories (CNN, CNN-attention, and pure
attention based models) by simply letting them learn from each other.

Knowledge Distillation was first proposed in [28] with the goal of compressing
the machine learning model. It refers to the training paradigm in which a
_student_ model uses predictions from a _teacher_ model as soft labels instead
of the hard ground truth label as the training target. In the conventional
setting, the student model is _smaller_ and typically _weaker_ than the
teacher model. Specifically for the audio classification task, knowledge
distillation is mainly used for model compression purpose [66, 53, 67]. Unlike
these previous efforts, our main goal is to improve the model performance with
knowledge distillation. We show that a _small_ (and weaker) teacher is in fact
a good teacher in cross-model knowledge distillation.

Recent studies find knowledge distillation can also be used to transfer the
inductive bias of neural networks [41, 17, 19, 68]. In particular, it was
found that Transformer models can be trained more effectively with CNN
teachers for vision tasks [17, 19]. Unlike these previous efforts, in this
paper, we focus on the audio domain, and study the _bidirectional_ knowledge
distillation between CNN and Transformer models rather than just the
CNN$\rightarrow$Transformer direction. We find that for both directions, the
student model noticeably improves, and in many cases, outperforms the teacher
model. With extensive experiments, we discover the optimal knowledge
distillation settings for audio classification, which are different from those
for the vision tasks.

##  7 Conclusion

In this paper, we show an intriguing interaction between CNN and Transformer
models and propose cross-model knowledge distillation for audio
classification. When we use any one of CNN and AST models as the teacher and
train another model as the student via knowledge distillation, the student
model noticeably improves and outperforms the teacher. With extensive
experiments, we find a small model that has a different architecture with the
student model is the optimal teacher. For CNN$\rightarrow$AST, using multiple
teachers with different architectures can further improve the model
performance. Surprisingly, the performance of the teacher model is not
important because cross-model knowledge distillation is beyond simple behavior
copying. We observe that knowledge distillation makes the model learn faster
and be less prone to overfitting as well as improving the model’s robustness.
Compared with CNN-attention hybrid models, cross-model knowledge distillation
is more effective and does not require any model architecture change. As a
consequence, our KD-CNN model with only 8M parameters outperforms the original
AST with 88M parameters on FSD50K; our KD-AST model performs even better and
achieves new state-of-the-art on FSD50K, AudioSet, and ESC50.

## References

  * [1] J. P. Woodard, “Modeling and classification of natural sounds by product code hidden markov models,” _IEEE Transactions on Signal Processing_ , vol. 40, no. 7, pp. 1833–1835, 1992.
  * [2] R. S. Goldhor, “Recognition of environmental sounds,” in _IEEE International Conference on Acoustics, Speech, and Signal Processing_ , 1993.
  * [3] S. Chachada and C.-C. J. Kuo, “Environmental sound recognition: A survey,” _APSIPA Transactions on Signal and Information Processing_ , vol. 3, 2014\.
  * [4] Y. LeCun, Y. Bengio _et al._ , “Convolutional networks for images, speech, and time series,” _The handbook of brain theory and neural networks_ , vol. 3361, no. 10, 1995.
  * [5] S. Hershey, S. Chaudhuri, D. P. Ellis, J. F. Gemmeke, A. Jansen, R. C. Moore, M. Plakal, D. Platt, R. A. Saurous, B. Seybold _et al._ , “Cnn architectures for large-scale audio classification,” in _IEEE International Conference on Acoustics, Speech, and Signal Processing_ , 2017.
  * [6] K. Palanisamy, D. Singhania, and A. Yao, “Rethinking cnn models for audio classification,” _arXiv preprint arXiv:2007.11154_ , 2020.
  * [7] J. Salamon and J. P. Bello, “Deep convolutional neural networks and data augmentation for environmental sound classification,” _IEEE Signal processing letters_ , vol. 24, no. 3, pp. 279–283, 2017.
  * [8] K. J. Piczak, “Environmental sound classification with convolutional neural networks,” in _IEEE International Workshop on Machine Learning for Signal Processing_ , 2015.
  * [9] Y. Tokozume and T. Harada, “Learning environmental sounds with end-to-end convolutional neural network,” in _IEEE International Conference on Acoustics, Speech, and Signal Processing_ , 2017.
  * [10] M. Huzaifah, “Comparison of time-frequency representations for environmental sound classification using convolutional neural networks,” _arXiv preprint arXiv:1706.07156_ , 2017.
  * [11] Y. Gong, Y.-A. Chung, and J. Glass, “Ast: Audio spectrogram transformer,” in _Interspeech_ , 2021.
  * [12] K. Koutini, J. Schlüter, H. Eghbal-zadeh, and G. Widmer, “Efficient training of audio transformers with patchout,” _arXiv preprint arXiv:2110.05069_ , 2021.
  * [13] K. Chen, X. Du, B. Zhu, Z. Ma, T. Berg-Kirkpatrick, and S. Dubnov, “Hts-at: A hierarchical token-semantic audio transformer for sound classification and detection,” _arXiv preprint arXiv:2202.00874_ , 2022.
  * [14] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin, “Attention is all you need,” in _Advances in Neural Information Processing Systems_ , 2017.
  * [15] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “BERT: Pre-training of deep bidirectional transformers for language understanding,” in _Conference of the North American Chapter of the Association for Computational Linguistics_ , 2019.
  * [16] A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, J. Uszkoreit, and N. Houlsby, “An image is worth 16x16 words: Transformers for image recognition at scale,” in _International Conference on Learning Representations_ , 2021.
  * [17] H. Touvron, M. Cord, M. Douze, F. Massa, A. Sablayrolles, and H. Jégou, “Training data-efficient image transformers & distillation through attention,” in _International Conference on Machine Learning_ , 2021.
  * [18] M. Raghu, T. Unterthiner, S. Kornblith, C. Zhang, and A. Dosovitskiy, “Do vision transformers see like convolutional neural networks?” _Advances in Neural Information Processing Systems_ , 2021.
  * [19] S. Ren, Z. Gao, T. Hua, Z. Xue, Y. Tian, S. He, and H. Zhao, “Co-advise: Cross inductive bias distillation,” _arXiv preprint arXiv:2106.12378_ , 2021.
  * [20] Y. Gong, Y.-A. Chung, and J. Glass, “Psla: Improving audio tagging with pretraining, sampling, labeling, and aggregation,” _IEEE/ACM Transactions on Audio, Speech, and Language Processing_ , 2021.
  * [21] M. Tan and Q. Le, “Efficientnet: Rethinking model scaling for convolutional neural networks,” in _International Conference on Machine Learning_ , 2019\.
  * [22] G. Huang, Z. Liu, L. Van Der Maaten, and K. Q. Weinberger, “Densely connected convolutional networks,” in _IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2017.
  * [23] G. Gwardys and D. M. Grzywczak, “Deep image features in music information retrieval,” _International Journal of Electronics and Telecommunications_ , vol. 60, no. 4, pp. 321–326, 2014.
  * [24] A. Guzhov, F. Raue, J. Hees, and A. Dengel, “Esresnet: Environmental sound classification based on visual domain models,” in _International Conference on Pattern Recognition_ , 2021.
  * [25] S. Adapa, “Urban sound tagging using convolutional neural networks,” in _DCASE Workshop_ , 2019.
  * [26] Z. Liu, H. Mao, C.-Y. Wu, C. Feichtenhofer, T. Darrell, and S. Xie, “A convnet for the 2020s,” _arXiv preprint arXiv:2201.03545_ , 2022.
  * [27] A. Trockman and J. Z. Kolter, “Patches are all you need?” _arXiv preprint arXiv:2201.09792_ , 2022.
  * [28] G. Hinton, O. Vinyals, and J. Dean, “Distilling the knowledge in a neural network,” in _NIPS Deep Learning and Representation Learning Workshop_ , 2015\.
  * [29] L. Beyer, X. Zhai, A. Royer, L. Markeeva, R. Anil, and A. Kolesnikov, “Knowledge distillation: A good teacher is patient and consistent,” _arXiv preprint arXiv:2106.05237_ , 2021.
  * [30] D. S. Park, W. Chan, Y. Zhang, C.-C. Chiu, B. Zoph, E. D. Cubuk, and Q. V. Le, “SpecAugment: A simple data augmentation method for automatic speech recognition,” _Interspeech_ , 2019.
  * [31] H. Zhang, M. Cisse, Y. N. Dauphin, and D. Lopez-Paz, “mixup: Beyond empirical risk minimization,” in _International Conference on Learning Representations_ , 2018.
  * [32] E. Fonseca, X. Favory, J. Pons, F. Font, and X. Serra, “Fsd50k: an open dataset of human-labeled sound events,” _arXiv preprint arXiv:2010.00475_ , 2020.
  * [33] J. F. Gemmeke, D. P. Ellis, D. Freedman, A. Jansen, W. Lawrence, R. C. Moore, M. Plakal, and M. Ritter, “Audio set: An ontology and human-labeled dataset for audio events,” in _IEEE International Conference on Acoustics, Speech, and Signal Processing_ , 2017.
  * [34] K. J. Piczak, “Esc: Dataset for environmental sound classification,” in _ACM International Conference on Multimedia_ , 2015.
  * [35] A. Shah, A. Kumar, A. G. Hauptmann, and B. Raj, “A closer look at weak label learning for audio events,” _arXiv preprint arXiv:1804.09288_ , 2018.
  * [36] M. Meire, L. Vuegen, and P. Karsmakers, “The impact of missing labels and overlapping sound events on multi-label multi-instance learning for sound event classification,” in _DCASE Workshop_ , 2019.
  * [37] E. Fonseca, S. Hershey, M. Plakal, D. P. Ellis, A. Jansen, and R. C. Moore, “Addressing missing labels in large-scale sound event recognition using a teacher-student framework with loss masking,” _IEEE Signal Processing Letters_ , vol. 27, pp. 1235–1239, 2020.
  * [38] Q. Xie, M.-T. Luong, E. Hovy, and Q. V. Le, “Self-training with noisy student improves imagenet classification,” in _IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2020.
  * [39] M. Raghu, J. Gilmer, J. Yosinski, and J. Sohl-Dickstein, “Svcca: Singular vector canonical correlation analysis for deep learning dynamics and interpretability,” _Advances in Neural Information Processing Systems_ , 2017\.
  * [40] M. M. Naseer, K. Ranasinghe, S. H. Khan, M. Hayat, F. Shahbaz Khan, and M.-H. Yang, “Intriguing properties of vision transformers,” _Advances in Neural Information Processing Systems_ , 2021.
  * [41] S. Abnar, M. Dehghani, and W. Zuidema, “Transferring inductive biases through knowledge distillation,” _arXiv preprint arXiv:2006.00555_ , 2020.
  * [42] S. Srivastava, Y. Wang, A. Tjandra, A. Kumar, C. Liu, K. Singh, and Y. Saraf, “Conformer-based self-supervised learning for non-speech audio tasks,” _arXiv preprint arXiv:2110.07313_ , 2021.
  * [43] L. Ford, H. Tang, F. Grondin, and J. R. Glass, “A deep residual network for large-scale acoustic scene analysis.” in _Interspeech_ , 2019.
  * [44] K. Miyazaki, T. Komatsu, T. Hayashi, S. Watanabe, T. Toda, and K. Takeda, “Convolution augmented transformer for semi-supervised sound event detection,” in _DCASE Workshop_ , 2020.
  * [45] Q. Kong, Y. Xu, W. Wang, and M. D. Plumbley, “Sound event detection of weakly labelled data with cnn-transformer and automatic threshold optimization,” _IEEE/ACM Transactions on Audio, Speech, and Language Processing_ , vol. 28, pp. 2450–2460, 2020.
  * [46] H.-H. Wu, P. Seetharaman, K. Kumar, and J. P. Bello, “Wav2clip: Learning robust audio representations from clip,” _arXiv preprint arXiv:2110.11499_ , 2021.
  * [47] “Sound classification with yamnet,” https://www.tensorflow.org/hub/tutorials/yamnet.
  * [48] P. Verma and J. Berger, “Audio transformers: Transformer architectures for large scale audio understanding. adieu convolutions,” _arXiv preprint arXiv:2105.00335_ , 2021.
  * [49] E. Fonseca, A. Ferraro, and X. Serra, “Improving sound event classification by increasing shift invariance in convolutional neural networks,” _arXiv preprint arXiv:2107.00623_ , 2021.
  * [50] P. Izmailov, D. Podoprikhin, T. Garipov, D. Vetrov, and A. G. Wilson, “Averaging weights leads to wider optima and better generalization,” in _Conference on Uncertainty in Artificial Intelligence_ , 2018.
  * [51] Q. Kong, Y. Cao, T. Iqbal, Y. Wang, W. Wang, and M. D. Plumbley, “Panns: Large-scale pretrained audio neural networks for audio pattern recognition,” _IEEE/ACM Transactions on Audio, Speech, and Language Processing_ , vol. 28, pp. 2880–2894, 2020.
  * [52] Y. Wang, J. Li, and F. Metze, “A comparison of five multiple instance learning pooling functions for sound event detection with weak labeling,” in _IEEE International Conference on Acoustics, Speech, and Signal Processing_ , 2019.
  * [53] K. Koutini, H. Eghbal-zadeh, and G. Widmer, “Iterative knowledge distillation in r-cnns for weakly-labeled semi-supervised sound event detection,” in _DCASE Workshop_ , 2018.
  * [54] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in _IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2016.
  * [55] A. G. Howard, M. Zhu, B. Chen, D. Kalenichenko, W. Wang, T. Weyand, M. Andreetto, and H. Adam, “Mobilenets: Efficient convolutional neural networks for mobile vision applications,” _arXiv preprint arXiv:1704.04861_ , 2017.
  * [56] Y. Liu, M. Ott, N. Goyal, J. Du, M. Joshi, D. Chen, O. Levy, M. Lewis, L. Zettlemoyer, and V. Stoyanov, “Roberta: A robustly optimized bert pretraining approach,” _arXiv preprint arXiv:1907.11692_ , 2019.
  * [57] A. Gulati, J. Qin, C.-C. Chiu, N. Parmar, Y. Zhang, J. Yu, W. Han, S. Wang, Z. Zhang, Y. Wu _et al._ , “Conformer: Convolution-augmented transformer for speech recognition,” _arXiv preprint arXiv:2005.08100_ , 2020.
  * [58] Y. Li, K. Zhang, J. Cao, R. Timofte, and L. Van Gool, “Localvit: Bringing locality to vision transformers,” _arXiv preprint arXiv:2104.05707_ , 2021\.
  * [59] Y. Chen, X. Dai, D. Chen, M. Liu, X. Dong, L. Yuan, and Z. Liu, “Mobile-former: Bridging mobilenet and transformer,” _arXiv preprint arXiv:2108.05895_ , 2021.
  * [60] J. Guo, K. Han, H. Wu, C. Xu, Y. Tang, C. Xu, and Y. Wang, “Cmt: Convolutional neural networks meet vision transformers,” _arXiv preprint arXiv:2107.06263_ , 2021.
  * [61] T. Xiao, P. Dollar, M. Singh, E. Mintun, T. Darrell, and R. Girshick, “Early convolutions help transformers see better,” _Advances in Neural Information Processing Systems_ , 2021.
  * [62] K. Yuan, S. Guo, Z. Liu, A. Zhou, F. Yu, and W. Wu, “Incorporating convolution designs into visual transformers,” in _IEEE/CVF International Conference on Computer Vision_ , 2021.
  * [63] H. Yan, Z. Li, W. Li, C. Wang, M. Wu, and C. Zhang, “Contnet: Why not use convolution and transformer at the same time?” _arXiv preprint arXiv:2104.13497_ , 2021.
  * [64] Z. Dai, H. Liu, Q. Le, and M. Tan, “Coatnet: Marrying convolution and attention for all data sizes,” _Advances in Neural Information Processing Systems_ , 2021.
  * [65] A. Mesaros, T. Heittola, E. Benetos, P. Foster, M. Lagrange, T. Virtanen, and M. D. Plumbley, “Detection and classification of acoustic scenes and events: Outcome of the dcase 2016 challenge,” _IEEE/ACM Transactions on Audio, Speech, and Language Processing_ , vol. 26, no. 2, pp. 379–393, 2017.
  * [66] J.-W. Jung, H.-S. Heo, H.-J. Shim, and H.-J. Yu, “Knowledge distillation in acoustic scene classification,” _IEEE Access_ , vol. 8, pp. 166 870–166 879, 2020.
  * [67] K. Choi, M. Kersner, J. Morton, and B. Chang, “Temporal knowledge distillation for on-device audio classification,” _arXiv preprint arXiv:2110.14131_ , 2021\.
  * [68] A. Kuncoro, C. Dyer, L. Rimell, S. Clark, and P. Blunsom, “Scalable syntax-aware language models using knowledge distillation,” _arXiv preprint arXiv:1906.06438_ , 2019.

|  Yuan Gong (Member, IEEE) received the B.S. degree in biomedical engineering
from Fudan University, Shanghai, China, and the Ph.D. degree in computer
science from the University of Notre Dame, IN, USA, in 2015 and 2020,
respectively. He is a Postdoctoral Researcher with the MIT Computer Science
and Artificial Intelligence Laboratory. Currently, his primary research
interests include automatic speech recognition, speech based healthcare
system, and acoustic events detection. He won the 2017 AVEC depression
detection challenge and one of his papers was nominated for the Best Student
Paper Award in Interspeech 2019.
---|---
|  Sameer Khurana is a Ph.D. candidate in Electrical Engineering and Computer
Science at MIT. He works with Dr. James Glass in MIT’s Computer Science and
Artificial Intelligence Lab on speech recognition, translation, and self-
supervised speech processing. His primary research interest is to build
automatic speech recognition and translation models that could learn from a
few human-labeled examples. In addition, his work aims to expand current
automatic speech recognition and translation technology to low-resource
languages.
---|---
|  Andrew Rouditchenko received the S.B. degree in electrical engineering and
computer science from MIT in 2019 and the M.Eng degree in electrical
engineering and computer science from MIT in 2021. He is a Ph.D. student
working with Dr. Jim Glass in the MIT Spoken Language Systems Group.
---|---
|  James Glass (Fellow, IEEE) is a Senior Research Scientist with MIT where he
Leads the Spoken Language Systems Group with the Computer Science and
Artificial Intelligence Laboratory. He is also a member of the Harvard
University Program in Speech and Hearing Bioscience and Technology. Since
obtaining the S.M. and Ph.D. degrees with MIT in electrical engineering and
computer science, his research interests include automatic speech recognition,
unsupervised speech processing, and spoken language understanding. He is a
Fellow of the International Speech Communication Association, and is currently
an Associate Editor for the IEEE Transactions on Pattern Analysis and Machine
Intelligence.
---|---