In this section, we present our results with some discussions.

##  0.1 Comparison with baseline

Figure LABEL:fig:results_magnatagatune shows the results on the MagnaTagATune
dataset. Comparing results with or without embedding compression, we can
observe that in all cases the embedding compression module can improve the
performance, except for the FitNet + PaSST combination where the result stays
the same. Comparing the improvement between supervised embeddings and
unsupervised embeddings, we notice that embedding compression leads to a
larger improvement when it is applied to unsupervised embeddings. Moreover, we
can see that without embedding compression, the EAsT method tends to
deteriorate the performance compared with the baseline, but adding the
embedding compression enables the student model to outperform the baseline.

These observations are consistent with our assumption that the irrelevant
knowledge in the teacher embeddings negatively impacts the knowledge transfer.
The extent of irrelevance depends on the domain shift from the source task to
the target task, which is greater in the case of unsupervised embeddings than
supervised ones.

While the transformation of the teacher embeddings shows some parallels to
fine-tuning the teacher model, no parameters of the teacher model are changed.
Therefore, the feature distortion is minimized and the risk of overfitting
reduced. In addition, a considerable amount of computational resources can be
saved during the training.

##  0.2 Generalizability

| None | PaSST | MULE
---|---|---|---
SC-CNN | .863 | – | –
EAsTFitNet | – | .904 | .890
EAsTFitNet \+ Linear | – | .904 | .887
EAsTDC | – | .892 | .874
EAsTDC \+ Linear | – | .901 | .887
Table 1: Generalizability test. All models are trained on MagnaTagATune and
tested on the overlapping labels in OpenMIC. SC-CNN trained from scratch on
MagnaTagATune serves as the baseline model.

The results of the generalizability test are listed in Table 1. Note that
these models are not trained on the target dataset.

Comparing the EAsT methods with the baseline SC-CNN, we can see that the
models trained with the guidance of embeddings show improved performance. This
is true for the models with or without embedding compression. These results
suggest that adding the knowledge of embeddings during training improves the
generalizability of student models. The teacher models, trained on large-scale
datasets and thus having better generalizability, can transfer this general
knowledge to the students.

In the case of DC, embedding compression tends to slightly improve the
results, and in the case of FitNet, the performance stays the same or has a
slight decay. However, the differences are relatively small compared to the
improvement over the baseline, which means that while adding the embedding
compression module might lead to a bias toward the target task, it does not
have a significant negative impact on the student model’s generalizability.

##  0.3 Complexity

Table 2 lists the number of parameters and the rough inference speed
measurements of the models we use.

Model | Parameters (M) | Iteration / s
---|---|---
PaSST | 86.1 | 18.7
PANNs | 79.7 | 70.6
MULE | 62.4 | 53.5
DisCogs | 4.0 | 101.0
HCNN | 3.6 | 164.2
SC-CNN | 9.1 | 196.4
Table 2: Comparison of the model complexity.

We can see that the student models we use have much fewer parameters and are
faster to run compared to the teacher embedding models (except for DisCogs,
which is based on EfficientNet and has fewer parameters but also leads to
suboptimal performance).

##  0.4 Results on other datasets

We report the performance in terms of classification accuracy on the ESC50
dataset and the FMA-small dataset in Table 3. The experimental setup is the
same as in Sect. 0.1, i.e., the student models have been trained on the target
dataset.

ESC50 | None | PaSST | MULE
---|---|---|---
SC-CNN | .732 | – | –
EAsTFitNet | – | .754 | .749
EAsTFitNet \+ Linear | – | .747 | .758
EAsTDC | – | .753 | .728
EAsTDC \+ Linear | – | .753 | .746
FMA-small | None | PaSST | MULE
SC-CNN | .516 | – | –
EAsTFitNet | – | .512 | .526
EAsTFitNet \+ Linear | – | .527 | .527
EAsTDC | – | .505 | .494
EAsTDC \+ Linear | – | .545 | .531
Table 3: Results on other datasets.

On the ESC50 dataset, we observe a similar trend as in Fig.
LABEL:fig:results_magnatagatune that when using the unsupervised MULE
embedding, embedding compression can improve the performance. However, in the
case of supervised PaSST embedding, adding the embedding compression module
shows no benefits, as the performance either deteriorates or stays the same. A
possible reason is that the dissimilarity between the source task (AED) that
PaSST is trained on and the target task (sound event classification) is
smaller than that between AED and music auto-tagging, therefore the embedding
compression is not able to reduce much irrelevant information. On the FMA-
small dataset, we find that the embedding compression module can improve the
results with both embeddings, which supports our assumption that embedding
compression is more effective especially when there is a greater dissimilarity
between the source task and the target task.


§ BACKGROUND INFORMATION

Missing 'biblatex' package
The bibliography requires the 'biblatex' package.

titleOn the geometry of positive cones in finitely generated
journaltitleJournal of the London Mathematical Society

titleRegular left-orders on groups
journaltitleto appear in Journal of Combinatorial

Springer Science
Business Media LLC

titleBranch rings, thinned rings, tree enveloping rings
journaltitleIsrael Journal of Mathematics

EMS Press

booktitleHandbook of Automata Theory, Volume II
titleRational subsets of groups
Berlin, GER

Computational complexity, automata theory, Thue systems
titleOn the complexity of some extended word problems defined by
cancellation rules
journaltitleInformation Processing Letters

titleQuantifying residual finiteness
journaltitleJournal of Algebra

Coxeter groups; automatic groups; positive roots; root systems;
generating sets
titleA finiteness property and an automatic structure for
Coxeter groups.
journaltitleMathematische Annalen

titleGeometry of the word problem for 3-manifold groups
journaltitleJournal of Algebra

titleFiniteness properties of groups
journaltitleJournal of Pure and Applied Algebra

titleThe combinatorial structure of cocompact discrete hyperbolic
journaltitleGeometriae Dedicata

titleRational sets in commutative monoids
journaltitleJournal of Algebra

A. K. Peters, Ltd.

titleWord Processing in Groups

titleSpinal groups: semidirect product decompositions and
Hausdorff dimension
journaltitleJournal of Group Theory

titleOn the subexponential growth of groups acting on rooted
journaltitleGroups, Geometry, and Dynamics

Annals of Mathematics

titleRational Subgroups of Biautomatic Groups
journaltitleAnnals of Mathematics

Princeton University Press

booktitleCombinatorial Group Theory and Topology
titleGroups with a rational cross-section

IOP Publishing

titleDegrees of growth of finitely generated groups, and the
theory of invariant means
journaltitleMathematics of the USSR-Izvestiya

IOP Publishing

titleAn example of a finitely presented amenable group not
belonging to the class EG
journaltitleSbornik: Mathematics

booktitleNew Horizons in pro-$p$ Groups
titleJust Infinite Branch Groups
Boston, MA

booktitleEssays in Group Theory
titleHyperbolic Groups
New York, NY

Cambridge University Press

titleThe Dehn function and a regular set of normal forms for
R. Thompson's group $F$
journaltitleJournal of the Australian Mathematical Society

titleSur les groupes hyperboliques d'après Mikhael Gromov
Boston, MA

titleAlgorithms and Geometry for Graph Products of Groups
journaltitleJournal of Algebra

titleNo positive cone in a free product is regular
journaltitleInt. J. Algebra Computation

titleThe first cohomology of a group with permutation module
journaltitleArchiv der Mathematik

titleThe relaxation normal form of braids is regular
journaltitleInternational Journal of Algebra and Computation

Group, Formal language, Decision problem, Rational subset
titleOn the rational subset problem for groups
journaltitleJournal of Algebra

IOP Publishing

titleA finitely presented solvable group with unsolvable word
journaltitleMathematics of the USSR-Izvestiya

titleA system of defining relations for a Grigorchuk group
journaltitleMathematical notes of the Academy of Sciences of the

issn0003486X, 19398980
titlePalindromic subshifts and simple periodic groups of
intermediate growth
journaltitleAnnals of Mathematics

titleSome Remarks on Infinite Groups
journaltitleJournal of the London Mathematical Society

titleAutomatic structures, rational growth, and geometrically
finite hyperbolic groups
journaltitleInventiones mathematicae

titleInfinite Convergent String-rewriting Systems and
Cross-sections for Finitely Presented Monoids
journaltitleJournal of Symbolic Computation

titleFinite Automata and Their Decision Problems
journaltitleIBM Journal of Research and Development

Walter de Gruyter GmbH

titleHausdorff dimension of some groups acting on the binary tree
journaltitleJournal of Group Theory

titleFormal language convexity in left-orderable groups
journaltitleInternatinal Journal of Algebra Computation

titleGrowth functions of some classes of languages