# NLP-CIC @ DIACR-Ita: POS and Neighbor Based Distributional Models for
Lexical Semantic Change in Diachronic Italian Corpora††thanks:  “Copyright ©
2020 for this paper by its authors. Use permitted under Creative Commons
License Attribution 4.0 International (CC BY 4.0).”

Jason Angel
CIC, Instituto Politécnico Nacional
Mexico City, Mexico
<EMAIL_ADDRESS>&Carlos A. Rodriguez-Diaz
CIC, Instituto Politécnico Nacional
Mexico City, Mexico
<EMAIL_ADDRESS>Alexander Gelbukh
CIC, Instituto Politécnico Nacional
Mexico City, Mexico
www.gelbukh.com &Sergio Jimenez
Instituto Caro y Cuervo
Bogota, Colombia
<EMAIL_ADDRESS>
###### Abstract

We present our systems and findings on unsupervised lexical semantic change
for the Italian language in the DIACR-Ita shared-task at EVALITA 2020. The
task is to determine whether a target word has evolved its meaning with time,
only relying on raw-text from two time-specific datasets. We propose two
models representing the target words across the periods to predict the
changing words using threshold and voting schemes. Our first model solely
relies on part-of-speech usage and an ensemble of distance measures. The
second model uses word embedding representation to extract the neighbor’s
relative distances across spaces and propose “the average of absolute
differences” to estimate lexical semantic change. Our models achieved
competent results, ranking third in the DIACR-Ita competition. Furthermore, we
experiment with the $k\\_neighbor$ parameter of our second model to compare
the impact of using “the average of absolute differences” versus the cosine
distance used in [Hamilton et al. (2016].

##  1 Introduction

Lexical semantic change has recently gained interest in the intersection of
natural language processing and historical linguistics111see
https://languagechange.org/, therefore several datasets have been proposed for
different languages [Schlechtweg et al. (2020a]. This work take place in the
context of DIACR-Ita [Basile et al. (2020a] at EVALITA 2020 [Basile et al.
(2020b], which sets the task for the Italian language in a fully unsupervised
fashion. From DIACR-Ita we received 18 target words222’egemonizzare’,
’lucciola’, ’campanello’, ’trasferibile’, ’brama’, ’polisportiva’, ’palmare’,
’processare’, ’pilotato’, ’cappuccio’, ’pacchetto’, ’ape’, ’unico’,
’discriminatorio’, ’rampante’, ’campionato’, ’tac’, ’piovra’, and two time-
specific and preprocessed Italian corpora, namely T0 and T1, which include
part-of-speech tagging and lemmatization information.

We present two perspectives to approach the problem, regarding how we
represent target words and estimate the lexical-semantic change across
datasets. (1) uses the POS distribution of target words as representation, and
employee an ensemble of distance measures for the estimation. (2) uses the
target words neighbor similarities as representation and one (of two proposed)
similarity measure for estimation.

The following three sections describe the previous works, modeling, and
results we obtained using these approaches. Following that, section 5
(Discussion) focuses on examine the second approach to illustrate the impact
of the $k$ parameter in similarity measures and the discriminatory performance
of our embedding-based model.

##  2 Related works

Previous works have employed similar approaches to address the unsupervised
lexical-semantic-change task, mostly for the English language [Schlechtweg et
al. (2020a, Asgari et al. (2020, Schlechtweg et al. (2020b]. Our first
approach follows the idea of “syntactic models” [Kulkarni et al. (2015], which
supposes that some semantic changes could imply a new syntactic functionality,
such as acquiring a new part-of-speech category, as ?) exemplify: the word
“apple” increased his use as a proper name in the ’80s.

On the other hand, our second approach follows the idea of “embedding-based
models” [Kulkarni et al. (2015, Hamilton et al. (2016, Shoemark et al. (2019],
which compares word vector representations from each period using an aligned
space, which can be computed either globally (for the full model) or locally
(only for a target words). A common strategy for local aligning is to perform
a new transformation representing the target words (the same from different
spaces) through neighborhood structures, under the assumption that independent
training of embedding algorithms on comparable corpora will still produce
similar neighborhood structures [Kulkarni et al. (2015].

Our second approach align the space locally using the nearest neighbors of
target words as shared feature.

##  3 Methodology

In this section we provide a detailed description of our systems, each of them
composed of two stages, the model and the voting scheme.

###  3.1 Models

We represented the target words as vectors for each time of period using two
perspectives that originate our submitted systems: the POS-model and the
embedding-model. The word representations are comparable across spaces, and
serve to estimate the lexical semantic change through similarity and distance
measures, from which we finally predict the changing words using thresholds
and voting schemes.

POS-model: we simply analyzes the Part-Of-Speech distribution as the relative
frequency over the datasets taking the top 4 most common POS-tags, namely ADJ,
NOUN, PROPN and VERB. The produced four-dimensional vector pairs are then used
to assess the lexical semantic change of each target word from the perspective
of their Euclidean, Manhattan and Cosine distances333we noticed that at this
point ?) uses Jenssen-Shannon divergence measure.

Embedding-model: We lowercase and concatenate each word form with its
corresponding POS to build embedding models for each dataset $T$, namely $T0$
and $T1$. Specifically, we used Word2Vec models[Mikolov et al. (2013] with the
CBOW version from gensim444https://radimrehurek.com/gensim/models/word2vec.htm
with the following parameters: size of 256, window of 5, min_count of 3. Then
we take the common vocabulary of both $V_{c}=V_{(TO)}\cap V_{(T1)}$, and use
it to constraint the set of top $k$ nearest neighbors of the target word only
from $T0$555Unlike ?) that takes the top-$k$ neighbors from each model and
union them ($N_{k}=N_{k}^{T0}\cup N_{k}^{T1}$)., i.e.,
$N_{k}=\\{n_{1},n_{2}...n_{k}\\},n_{k}\in V_{c}$, to build the representation
of the target word for each space based on its neighbor proximity, i.e.
$\vec{W}^{T}=[cos\\_sim(\vec{w},\vec{n_{k}})|n_{k}\in T]$, and estimate the
lexical semantic change using the following two formulas666 ?) only uses
cosine distance.:

$\text{{avg.abs.diff}}=Avg(|\vec{W}^{T0}-\vec{W}^{T1}|)$ (1)
$\text{{cosine\\_similarity}}=cos\\_sim(\vec{W}^{T0},\vec{W}^{T1})$ (2)

The average of absolute point-wise differences (avg.abs.diff for short) works
under the assumption that the neighbors a non-changing word preserves their
relative distance each other across diachronic representations. Therefore, the
value of this measure increases according to the lexical semantic change a
target word underwent. In our submission we used $k=10$.

###  3.2 Threshold and voting schemes

Given that DIACR-Ita is an unsupervised task we experiment with different
threshold and voting schemes to aggregate the measure ranks and determine
which target words have underwent a lexical semantic change. As a result, we
propose three voting schemes from which we derive our results.

System1: Upper-third of distance ranks (used for POS model): we sorted the
target words in descending order and rank their positions according to the
Euclidean, Manhattan and Cosine distances. We then sum all these ranks and
sort in descending order again. Finally we label the first upper-third part of
this list as changing words.

System2: Half intersection (used for the embedding model): We sort the target
words in descending and ascending order for the lineal-difference scores (1)
and the cosine-similarity (2) respectively. Then we take the top 50% of each
group, and intersect them to obtained the words that we predicted as changing
words.

System3: Union of Upper-third and Half intersection: This is just the union of
results from System1 and System2.

Figure 1:  Analysis of estimation ranges of lexical semantic change by
neighbor-based distributional models using several measures, and two
aggregation methods: only from T0 (at left) and the union of T0 and T1 (at
right).

##  4 Results

Table 1 summarize the results we obtained during the competition. One can see
that the system3 which combine system 2 and 3 also combine its false positive
results while removing the False negative ones. We officially ranked third
place with the System1, which in spite of exhibit equal results than System3,
is much simpler. We also made error analysis over the system 1 for the case of
“polisportiva” at Table 2, the results show that there is a large difference
in the POS usage of “polisportiva” across the time periods, NOUN and PROPN
seems to invert their distribution usage. We also made the
code777https://github.com/ajason08/evalita2020_diacrita publicly available for
the systems reproduction.

S (#) | Acc. |  False positive |  False negative
---|---|---|---
1 | 0.88 |  polisportiva |  rampante
2 | 0.83 |  egemonizzare |  lucciola, ape
3 | 0.88 |  polisportiva, egemonizzare |  –
Table 1:  Submission results using Accuracy  Corpus | ADJ | NOUN | PROPN | VERB
---|---|---|---|---
T0 | 0.04 | 0.18 | 0.76 | 0.02
T1 | 0.02 | 0.61 | 0.34 | 0.02
Table 2: POS usage of “polisportiva” over the time periods

##  5 Discussion: Post-evaluation analysis

In this section we employee the gold-standard labels of the target words to
analyze at Figure 1 the capabilities of our neighbor-based embedding-model
using several settings. To this end, we divide the Figure 1 into vertical and
horizontal views. The vertical view defines 3 groups (from top to bottom),
that serves to compare the three proposed measures to estimate the lexical
semantic change, namely the average of absolute differences, cosine similarity
and cosine distance. At the same time, the horizontal view serves to compare
the strategy of only use T0 (at left), versus the union of T0 and T1 (at
right), to define the top nearest neighbors $N_{k}$.

Next, each of the charts shows an analysis of the model for the given measure
across the $k$ parameter. The area charts represent by color regions the
ranges that discriminate the lexical semantic change of target words:
“changing words” (orange region) and “non-changing words” (purple region). The
yellow region in the middle marks the intersection of these ranges, thus,
words falling into the yellow region are difficult to estimate, according to
the used measure. We also identified the threshold that best discriminate
changing and non-changing target words, and draw a dashed line at that point.
On the other hand, the line charts throw light on all the possible performance
that the model could obtain by changing the $k$ parameter while using the best
possible discriminator threshold.

These results suggest that the “average of absolute difference” is the best
proposed measure because it obtains a better performance for a larger number
of $k$ values as displayed in the line charts. Moreover, the “average of
absolute difference” offers a larger range for possible discriminator
thresholds (as shown in the area charts), and it is tolerant to the $N_{k}$
election, since it remains almost unchanged while using either the union of T0
and T1, or only T0. One can also note that the area charts for the cosine
similarity versus cosine distance mirror each other, as expected, and their
performance is the same when using $N_{k}$ only from T0 (at left), but
slightly differ when using $N_{k}$ as the union of T0 and T1 (at right).

##  6 Conclusion

We tackle the problem of unsupervised lexical semantic change on two time-
specific datasets for 18 target words in Italian language. Our two approaches
focus on the representation of target words across the provided diachronic
datasets, they use part-of-speech usage and nearest neighbors respectively,
and a number of measures between these representation to estimate the lexical
semantic change. Then, this estimation serves to decide which target words
underwent a change by the use of proposed threshold and voting schemes.
Afterward, in the last part of this work, we analyzed the nearest neighbor
model through the impact of deciding the $k$ parameter and the similarity
measure that estimates the lexical semantic change. Our results for the DIACR-
Ita datasets suggest that the estimations of “the average of absolute
differences” measures have a better performance for a larger number of $k$
values than the cosine similarity and the cosine distance used in ?).

As for future work, we plan to investigate different mechanism for deciding
the threshold, and explore other diachronic datasets for other languages such
as English, German and Spanish. We also believe that further experiments on a
larger number of target words will benefit the reliability of models to judge
the lexical semantic change in an unsupervised fashion.

## Acknowledgments

The authors thank CONACYT for the computer resources provided through the
INAOE Supercomputing Laboratory’s Deep Learning Platform for Language
Technologies.

## References

  * [Asgari et al. (2020]  Ehsaneddin Asgari, Christoph Ringlstetter, and Hinrich Schütze.  2020\.  Unsupervised embedding-based detection of lexical semantic changes.  arXiv preprint arXiv:2005.07979.
  * [Basile et al. (2020a]  Pierpaolo Basile, Annalina Caputo, Tommaso Caselli, Pierluigi Cassotti, and Rossella Varvara.  2020a.  DIACR-Ita @ EVALITA2020: Overview of the EVALITA2020 Diachronic Lexical Semantics (DIACR-Ita) Task.  In Valerio Basile, Danilo Croce, Maria Di Maro, and Lucia C. Passaro, editors, Proceedings of the 7th evaluation campaign of Natural Language Processing and Speech tools for Italian (EVALITA 2020), Online. CEUR.org.
  * [Basile et al. (2020b]  Valerio Basile, Danilo Croce, Maria Di Maro, and Lucia C. Passaro.  2020b.  Evalita 2020: Overview of the 7th evaluation campaign of natural language processing and speech tools for italian.  In Valerio Basile, Danilo Croce, Maria Di Maro, and Lucia C. Passaro, editors, Proceedings of Seventh Evaluation Campaign of Natural Language Processing and Speech Tools for Italian. Final Workshop (EVALITA 2020), Online. CEUR.org.
  * [Hamilton et al. (2016]  William L. Hamilton, Jure Leskovec, and Dan Jurafsky.  2016\.  Cultural shift or linguistic drift? comparing two computational measures of semantic change.  In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 2116–2121, Austin, Texas, November. Association for Computational Linguistics.
  * [Kulkarni et al. (2015]  Vivek Kulkarni, Rami Al-Rfou, Bryan Perozzi, and Steven Skiena.  2015\.  Statistically significant detection of linguistic change.  In WWW ’15: Proceedings of the 24th International Conference on World Wide Web), page 625–635, Florence, Italy, May. International World Wide Web Conferences Steering Committee, Republic and Canton of Geneva, Switzerland.
  * [Mikolov et al. (2013]  Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean.  2013\.  Distributed representations of words and phrases and their compositionality.  In Advances in neural information processing systems, pages 3111–3119.
  * [Schlechtweg et al. (2020a]  Dominik Schlechtweg, Barbara McGillivray, Simon Hengchen, Haim Dubossarsky, and Nina Tahmasebi.  2020a.  Semeval-2020 task 1: Unsupervised lexical semantic change detection.
  * [Schlechtweg et al. (2020b]  Dominik Schlechtweg, Barbara McGillivray, Simon Hengchen, Haim Dubossarsky, and Nina Tahmasebi.  2020b.  Semeval-2020 task 1: Unsupervised lexical semantic change detection.  arXiv preprint arXiv:2007.11464.
  * [Shoemark et al. (2019]  Philippa Shoemark, Farhana Ferdousi Liza, Dong Nguyen, Scott Hale, and Barbara McGillivray.  2019\.  Room to Glo: A systematic comparison of semantic change detection approaches with word embeddings.  In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 66–76, Hong Kong, China, November. Association for Computational Linguistics.


# Knowledge Distillation from A Stronger Teacher

Tao Huang1,2 Shan You1 Fei Wang3 Chen Qian1 Chang Xu2
1SenseTime Research
2School of Computer Science, Faculty of Engineering, The University of Sydney
3University of Science and Technology of China
Correspondence to: Shan You<EMAIL_ADDRESS>
###### Abstract

Unlike existing knowledge distillation methods focus on the baseline settings,
where the teacher models and training strategies are not that strong and
competing as state-of-the-art approaches, this paper presents a method dubbed
DIST to distill better from a stronger teacher. We empirically find that the
discrepancy of predictions between the student and a stronger teacher may tend
to be fairly severer. As a result, the exact match of predictions in KL
divergence would disturb the training and make existing methods perform
poorly. In this paper, we show that simply preserving the relations between
the predictions of teacher and student would suffice, and propose a
correlation-based loss to capture the intrinsic inter-class relations from the
teacher explicitly. Besides, considering that different instances have
different semantic similarities to each class, we also extend this relational
match to the intra-class level. Our method is simple yet practical, and
extensive experiments demonstrate that it adapts well to various
architectures, model sizes and training strategies, and can achieve state-of-
the-art performance consistently on image classification, object detection,
and semantic segmentation tasks. Code is available at:
https://github.com/hunto/DIST_KD.

##  1 Introduction

The advent of automatic feature engineering fuels deep neural networks to
achieve remarkable success in a plethora of computer vision tasks, such as
image classification [38, 17, 53, 48, 19], object detection [23, 2], and
semantic segmentation [54, 5]. In the path of pursuing better performance,
current deep learning models generally grow deeper and wider [13, 45].
However, such heavy models are clumsy to deploy in practice due to the
limitations of computational and memory resources. For an efficient model with
competitive performance to those larger models, knowledge distillation (KD)
[16] has been proposed to boost the performance of the efficient model
(student) by distilling the knowledge of a larger model (teacher) during
training.

The essence of knowledge distillation relies on how to formulate and transfer
the knowledge from teacher to student. The most intuitive yet effective
approach is to match the probabilistic prediction (response) scores between
the teacher and student via Kullback–Leibler (KL) divergence [16]. In this
way, the student can be guided with more informative signals during training,
and is thus expected to have more promising performance than that being
trained stand-alone. Besides this vanilla prediction match, other works [14,
34, 41, 11] also investigate the knowledge within intermediate representations
to further boost the distillation performance, but this usually induces
additional training cost as a consequence. For example, OFD [14] proposes to
distill the information via multiple intermediate layers, but requires
additional convolutions for feature alignments; CRD [41] introduces a
contrastive loss to transfer pair-wise relationships, but it needs to hold a
memory bank for all 128-d features of ImageNet images, and produces additional
260M FLOPs of computation cost.

(a) Stronger model sizes (b) Stronger strategies

Figure 1: Comparisons of KD and our proposed DIST on ImageNet with different
teachers. (a) The ResNet-18 students are trained using baseline strategy with
different model sizes of the teacher. (b) The ResNet-18 students are trained
using different strategies with ResNet-50 teachers.

Recently, a few studies [8, 29, 39] have been performed to address the poor
learning issue of the student network when the student and teacher model sizes
significantly differ. For example, TAKD [29] proposes to reduce the
discrepancy of teacher and student by resorting to an additional teaching
assistant of moderate model size; DGKD [39] further improves TAKD by densely
gathering all the assistant models to guide the student. However, increasing
the model size is only one of the popular approaches to have a stronger
teacher. There lacks a thorough analysis on the training strategies to derive
a stronger teacher and their effect on KD. Most importantly, a generic enough
solution is preferred to address the difficulty of KD brought by stronger
teachers, rather than struggling to deal with different types of stronger
teachers (with larger model size or stronger training strategy) individually.

To understand what makes a stronger teacher and their effect on KD, we
systematically study the prevalent strategies for designing and training deep
neural networks, and show that:

  * •

Beyond scaling up the model size, a stronger teacher can also be derived
through advanced training strategies, _e.g._ , label smoothing and data
augmentation [51]. However, given a stronger teacher, the student’s
performance on the vanilla KD could be dropped, even worse than training from
scratch without KD, as shown in Figure 1.

  * •

The discrepancy between teacher and student tends to get fairly larger when we
switch their training strategy to a stronger one (see Figure 2). In this case,
an exact recovery of predictions via KL divergence could be challenging and
lead to the failure of vanilla KD.

  * •

Preserving the relation of predictions between teacher and student is
sufficient and effective. When transferring the knowledge from teacher to
student, what we really care about is preserving the preference (relative
ranks of predictions) by the teacher, instead of recovering the absolute
values accurately. Correlation between teacher and student predictions could
be favored to relax the exact match of KL divergence and distill the intrinsic
relations.

In this paper, we thus leverage the Pearson correlation coefficient [33] as a
new match manner to replace the KL divergence. In addition, besides the inter-
class relations in prediction vector (see Figure 3), with the intuition that
different instances have different spectrum of similarities with respect to
each class, we also propose to distill the intra-class relations for further
boosting the performance as Figure 3. Concretely, for each class, we gather
its corresponding predicted probabilities of all instances in a batch, then
transfer this relation from teacher to student. Our proposed method (dubbed
DIST) is super simple, efficient, and practical, which can be implemented with
only several lines of code (see Appendix A.1) and has almost the same training
cost as the vanilla KD. As a result, the student can be liberated from the
burden of matching the exact output of a strong teacher, but only be guided
appropriately to distill those truly informative relations.

Extensive experiments are conducted on benchmark datasets to verify our
effectiveness on various tasks, including image classification, object
detection, and semantic segmentation. Experimental results show that our DIST
significantly outperforms vanilla KD and those sophisticatedly-designed state-
of-the-art KD methods. For example, with the same baseline settings on
ImageNet, our DIST achieves the highest 72.07% accuracy on ResNet-18. With the
stronger strategy, our method obtains 82.3% accuracy on the recent transformer
Swin-T [27], improving KD by 1%.

(a) KL div. ($\tau=1$) (b) KL div. ($\tau=4$)

Figure 2: Discrepancy between the predictions of models trained standalone
with different strategies on ImageNet validation set. R18B1 represents
ResNet-18 trained with strategy B1 for instance. Details of training
strategies B1 and B2 refer to Table 1.

##  2 Revisiting Prediction Match of KD

In vanilla knowledge distillation [16], the knowledge is transferred from a
pre-trained teacher model to a student model by minimizing the discrepancy
between the prediction scores of the teacher and student models.

Formally, with the logits $\bm{Z}^{(\mathrm{s})}\in\mathbb{R}^{B\times C}$ and
$\bm{Z}^{(\mathrm{t})}\in\mathbb{R}^{B\times C}$ of student and teacher
networks, where $B$ and $C$ denote batch size and the number of classes,
respectively, the vanilla KD loss [16] is represented as

$\displaystyle\begin{split}\mathcal{L}_{\mathrm{KD}}&:=\frac{\tau^{2}}{B}\sum_{i=1}^{B}\mbox{KL}(\bm{Y}^{(\mathrm{t})}_{i,:},\bm{Y}^{(\mathrm{s})}_{i,:})=\frac{\tau^{2}}{B}\sum_{i=1}^{B}\sum_{j=1}^{C}Y^{(\mathrm{t})}_{i,j}\mbox{log}\left(\frac{Y^{(\mathrm{t})}_{i,j}}{Y^{(\mathrm{s})}_{i,j}}\right),\end{split}$
(1)

where KL refers to Kullback–Leibler divergence with

$\displaystyle\bm{Y}^{(\mathrm{s})}_{i,:}=softmax(\bm{Z}^{(\mathrm{s})}_{i,:}/\tau),\quad\bm{Y}^{(\mathrm{t})}_{i,:}=softmax(\bm{Z}^{(\mathrm{t})}_{i,:}/\tau),$
(2)

being the probabilistic prediction vectors, and $\tau$ is the temperature
factor to control the softness of logits.

In addition to the teacher’s soft targets in Eq.(1), KD [16] stated that it is
beneficial to train the student together with ground-truth labels, and the
overall training loss is composed of the original classification loss
$\mathcal{L}_{\mathrm{cls}}$ and KD loss $\mathcal{L}_{\mathrm{KD}}$, _i.e._ ,

$\mathcal{L}_{\mathrm{tr}}=\alpha\mathcal{L}_{\mathrm{cls}}+\beta\mathcal{L}_{\mathrm{KD}},$
(3)

where $\mathcal{L}_{\mathrm{cls}}$ is usually the cross-entropy loss between
the predictions of student network and ground-truth labels, $\alpha$ and
$\beta$ are factors for balancing the losses.

###  2.1 Catastrophic discrepancy with a stronger teacher

As illustrated in Section 1, the effect of a teacher on KD has not been
sufficiently investigated, especially when the performance of pre-trained
teacher grows stronger, such as with larger model size or being trained with
more advanced and competing strategies, _e.g._ , label smoothing, mix-up [51],
auto augmentations [9], etc. With this regard, as Figure 2, we train ResNet-18
and ResNet-50 standalone with strategy B1 and strategy B2111Training with B2
obtains higher accuracy compared to B1, _e.g._ , 73.4% (B2) vs. 69.8% (B1) on
ResNet-18., and obtain 4 trained models (R18B1, R18B2, R50B1, and R50B2 with
accuracies 69.76%, 73.4%, 76.13%, and 78.5%, respectively), then compare their
discrepancy using KL divergence ($\tau=1$ and $\tau=4$) on the predicted
probabilities $\bm{Y}$. We have the following observations:

  * •

The outputs of ResNet-18 do not change much with the stronger strategy
compared to ResNet-50. This implies that the representational capacity limits
the student’s performance, and it tends to be fairly challenging for the
student to exactly match the teacher’s outputs as their discrepancy becomes
larger.

  * •

When the teacher and student models are trained with a stronger strategy, the
discrepancy between teacher and student would be larger. This indicates that
when we adopt KD with a stronger training strategy, the misalignment between
KD loss and classification loss would be severer, thus disturbing the
student’s training.

As a result, the exact match (_i.e._ , the loss reaches the minimal if and
only if the teacher and student outputs are exactly identical) with KL
divergence seems way too overambitious and demanding since the discrepancy
between student and teacher can be considerably huge. Since the exact match
can be detrimental with a stronger teacher, our intuition is to develop a
relaxed manner for matching the predictions between the teacher and student.

Figure 3: Difference between our DIST and existing KD methods. Conventional KD
matches the outputs of student ($s\in\mathbb{R}^{5}$) to teacher
($t\in\mathbb{R}^{5}$) point-wisely; instance relation methods operate on the
feature level and measure the internal correlations (corr.) between instances
in student and teacher separately, then transfer the teacher’s correlations to
student. Our DIST proposes to maintain the inter-class and intra-class
relations between student and teacher. Inter-class relation: correlation
between the predicted probabilistic distributions on each instance of teacher
and student. Intra-class relation: correlation of the probabilities of all the
instances on each class.

##  3 DIST: Distillation from A Stronger Teacher

###  3.1 Relaxed match with relations

The prediction scores indicate the teacher’s confidence (or preference) over
all classes. For a relaxed match of predictions between the teacher and
student, we are motivated to consider what we really care about for the
teacher’s output. Instead of the exact probabilistic values, actually, during
inference, we are only concerned about their relations, _i.e._ , relative
ranks of predictions of teacher.

In this way, for some metric $d(\cdot,\cdot)$ with
$\mathbb{R}^{C}\times\mathbb{R}^{C}\rightarrow\mathbb{R}^{+}$, the exact match
can be formulated that $d({\bm{a}},{\bm{b}})=0$ if ${\bm{a}}={\bm{b}}$ for any
two prediction vector as $\bm{Y}^{(\mathrm{s})}_{i,:}$ and
$\bm{Y}^{(\mathrm{t})}_{i,:}$ in the KL divergence of Eq.(1). Then as a
relaxed match, we can introduce additional mappings $\phi(\cdot)$ and
$\psi(\cdot)$ with $\mathbb{R}^{C}\rightarrow\mathbb{R}^{C}$ such that

$d(\phi({\bm{a}}),\psi({\bm{b}}))=d({\bm{a}},{\bm{b}}),\forall{\bm{a}},{\bm{b}}$
(4)

Therefore, $d({\bm{a}},{\bm{b}})=0$ does not necessarily require ${\bm{a}}$
and ${\bm{b}}$ should be exactly the same. Nevertheless, since we care about
the relation within ${\bm{a}}$ or ${\bm{b}}$, the mappings $\phi$ and $\psi$
should be isotone and do not affect the semantic information and inference
result of the prediction vector.

With this regard, a simple yet effective choice for the isotone mapping is the
positive linear transformation, namely,

$d(m_{1}{\bm{a}}+n_{1},m_{2}{\bm{b}}+n_{2})=d({\bm{a}},{\bm{b}}),$ (5)

where $m_{1}$, $m_{2}$, $n_{1}$, and $n_{2}$ are constants with $m_{1}\times
m_{2}>0$. As a result, this match could be invariant under separate changes in
scale and shift for the predictions. Actually, to satisfy the property Eq.(5),
we can thus adopt the widely-used Pearson’s distance as the metric, _i.e._ ,

$d_{\mathrm{p}}({\bm{u}},{\bm{v}}):=1-\rho_{\mathrm{p}}({\bm{u}},{\bm{v}}).$
(6)

$\rho_{\mathrm{p}}({\bm{u}},{\bm{v}})$ is the Pearson correlation coefficient
between two random variables ${\bm{u}}$ and ${\bm{v}}$,

$\displaystyle\begin{split}\rho_{\mathrm{p}}({\bm{u}},{\bm{v}})&:=\frac{\mbox{Cov}({\bm{u}},{\bm{v}})}{\mbox{Std}({\bm{u}})\mbox{Std}({\bm{v}})}=\frac{\sum_{i=1}^{C}(u_{i}-\bar{u})(v_{i}-\bar{v})}{\sqrt{\sum_{i=1}^{C}(u_{i}-\bar{u})^{2}\sum_{i=1}^{C}(v_{i}-\bar{v})^{2}}}\\\
\end{split}$ (7)

where $\mbox{Cov}({\bm{u}},{\bm{v}})$ is the covariance of ${\bm{u}}$ and
${\bm{v}}$, $\bar{u}$ and $\mbox{Std}({\bm{u}})$ denote the mean and standard
derivation of ${\bm{u}}$, respectively.

In this way, we can define the relation as correlation. More specifically, and
the original exact match in vanilla KD [16] can thus be relaxed and replaced
by maximizing the linear correlation to preserve the relation of teacher and
student on the probabilistic distribution of each instance, which we call
inter-class relation. Formally, for each pair of prediction vector
$\bm{Y}^{(\mathrm{s})}_{i,:}$ and $\bm{Y}^{(\mathrm{t})}_{i,:}$, the inter-
relation loss can be formulated as

$\mathcal{L}_{\mathrm{inter}}:=\frac{1}{B}\sum_{i=1}^{B}d_{\mathrm{p}}(\bm{Y}^{(\mathrm{s})}_{i,:},\bm{Y}^{(\mathrm{t})}_{i,:}).$
(8)

Some isotone mappings or metrics can also be used to relax the match as
Eq.(4), such as cosine similarity investigated empirically in Section 4.5;
other more advanced and delicate choices could be left as future work.

###  3.2 Better distillation with intra-relations

Besides the inter-class relation, where we transfer the relation of multiple
classes in each instance, the prediction scores of multiple instances in each
class are also informative and useful. This scores indicate the similarities
of multiple instances to one class. For instance, suppose we have three images
containing “cat”, “dog”, and “plane”, respectively, and they have three
prediction scores on the ‘cat’ class, denoted as $e$, $f$, and $g$. Generally,
the picture “cat” should have the largest score to the “cat” class, while the
“plane” should have the smallest score since it is inanimate. This relation of
“$e>f>g$” could also be transferred to the student. Besides, even for the
images from the same class, the intrinsic intra-class variance of the semantic
similarities is actually also informative. It indicates the prior from the
teacher that which one is more reliable to cast in this class.

Therefore, we also encourage to distill this intra-relation for better
performance. Actually, define prediction matrix $\bm{Y}^{(\mathrm{s})}$ and
$\bm{Y}^{(\mathrm{t})}$ with each row as $\bm{Y}^{(\mathrm{s})}_{i,:}$ and
$\bm{Y}^{(\mathrm{t})}_{i,:}$, then the above inter-relation is to maximize
the correlation row-wisely (see Figure 3). In contrast, for intra-relation,
the corresponding loss is thus to maximize the correlation column-wisely,
_i.e._ ,

$\mathcal{L}_{\mathrm{intra}}:=\frac{1}{C}\sum_{j=1}^{C}d_{\mathrm{p}}(\bm{Y}^{(\mathrm{s})}_{:,j},\bm{Y}^{(\mathrm{t})}_{:,j}).$
(9)

As a result, the overall training loss $\mathcal{L}_{\mathrm{tr}}$ can be
composed of the classification loss, inter-class KD loss, and intra-class KD
loss, _i.e._ ,

$\mathcal{L}_{\mathrm{tr}}=\alpha\mathcal{L}_{\mathrm{cls}}+\beta\mathcal{L}_{\mathrm{inter}}+\gamma\mathcal{L}_{\mathrm{intra}},$
(10)

where $\alpha$, $\beta$, and $\gamma$ are factors for balancing the losses. In
this way, via the relation loss, we have endowed the student with freedom more
or less to match the teacher network’s output adaptively, thus boosting the
distillation performance to a great extent.

Table 1: Training strategies on image classification tasks. BS: batch size; LR: learning rate; WD: weight decay; LS: label smoothing; EMA: model exponential moving average; RA: RandAugment [9]; RE: random erasing; CJ: color jitter. Strategy | Dataset | Epochs |  Total BS |  Initial LR | Optimizer | WD | LS | EMA | LR scheduler | Data augmentation
---|---|---|---|---|---|---|---|---|---|---
A1 | CIFAR-100 | 240 | 64 | 0.05 | SGD | $5\times 10^{-4}$ | - | - |  $\times 0.1$ at 150,180,210 epochs | crop + flip
B1 | ImageNet | 100 | 256 | 0.1 | SGD | $1\times 10^{-4}$ | - | - |  $\times 0.1$ every 30 epochs | crop + flip
B2 | ImageNet | 450 | 768 | 0.048 | RMSProp | $1\times 10^{-5}$ | 0.1 | 0.9999 |  $\times 0.97$ every 2.4 epochs |  {B1} + RA + RE
B3 | ImageNet | 300 | 1024 | 5e-4 | AdamW | $5\times 10^{-2}$ | 0.1 | - | cosine |  {B2} + CJ + Mixup + CutMix

##  4 Experiments

###  4.1 Experimental settings

Training strategies. The training strategies of image classification task are
summarized in Table 1. CIFAR-100. For fair comparisons, we use the same
training strategies (referred to A1 in Table 1) and pretrained models
following CRD [41]. ImageNet. B1: for comparisons with previous KD methods, we
train our baselines with the same simple training strategy as CRD [41]. B2: to
validate the effectiveness of KD methods on modern training strategies, we
follow EfficientNet [40] and design a training strategy B2, which can
significantly improve the performance compared to B1. B3: the strategy B3 is
used for training Swin-Transformers [27], and contains even more stronger data
augmentations and regularization.

Loss weights. On CIFAR-100 and ImageNet, we set $\alpha=1$, $\beta=2$, and
$\gamma=2$ in Eq.(10). On object detection and semantic segmentation, these
three factors are all equal to $1$. For KD [16], we set $\alpha=0.9$,
$\beta=1$ in Eq.(3), and use a default temperature $\tau=4$. Specifically,
instead of using $\tau=1$ on ImageNet, we choose a larger temperature $\tau=4$
on CIFAR-100, as it is easy to get overfit and the learned probabilistic
distribution is sharp on CIFAR-100.

###  4.2 Image Classification

Table 2: Evaluation results of baseline settings on ImageNet. We use ResNet-34 and ResNet-50 released by Torchvision [28] as our teacher networks, and follow the standard training strategy (B1). Student (teacher) | Teacher | Student |  KD [16] |  OFD [14] |  CRD [41] |  SRRL [47] |  Review [7] | DIST
---|---|---|---|---|---|---|---|---
ResNet-18 (ResNet-34) | Top-1 | 73.31 | 69.76 | 70.66 | 71.08 | 71.17 | 71.73 | 71.61 | 72.07
Top-5 | 91.42 | 89.08 | 89.88 | 90.07 | 90.13 | 90.60 | 90.51 | 90.42
MobileNet (ResNet-50) | Top-1 | 76.16 | 70.13 | 70.68 | 71.25 | 71.37 | 72.49 | 72.56 | 73.24
Top-5 | 92.86 | 89.49 | 90.30 | 90.34 | 90.41 | 90.92 | 91.00 | 91.12

Baseline results on ImageNet. We first compare our method with prior works
using the baseline settings. As shown in Table 2, our DIST significantly
outperforms prior KD methods. Note that our method is only conducted on the
outputs of models, and has a similar computational cost as KD [16].
Nevertheless, it even achieves better performance compared to those
sophisticatedly-designed methods. For example, CRD [41] needs to preserve a
memory bank for all 128-d features of ImageNet images, and produces additional
260M FLOPs of computation cost; SRRL [47] and Review [7] require additional
convolutions for feature alignments. The implementation of DIST can be found
in Appendix A.1, which is quite simple compared to these methods.

Distillation from stronger teacher models. As the stronger teachers come from
larger model sizes and stronger strategies, we here first conduct experiments
to compare our DIST with the vanilla KD on different scales (model sizes) of
ResNets with baseline strategy B1. As shown in Table 3, when the teacher goes
larger, the ResNet-18 students perform even worse than that with a medium-
sized ResNet-50 teacher. Nevertheless, our DIST shows an upward trend with
larger teachers, and the improvements compared to KD also become more
significant, indicating that our DIST tackles better on the large discrepancy
between the student and larger teacher.

Table 3: Performance of ResNet-18 and ResNet-34 on ImageNet with different sizes of teachers. Student | Teacher | Top-1 ACC (%)
---|---|---
student | teacher | KD | DIST
ResNet-18 | ResNet-34 | 69.76 | 73.31 | 71.21 | 72.07 (+0.86)
ResNet-50 | 76.13 | 71.35 | 72.12 (+0.77)
ResNet-101 | 77.37 | 71.09 | 72.08 (+0.99)
ResNet-152 | 78.31 | 71.12 | 72.24 (+1.12)
ResNet-34 | ResNet-50 | 73.31 | 76.13 | 74.73 | 75.06 (+0.33)
ResNet-101 | 77.37 | 74.89 | 75.36 (+0.47)
ResNet-152 | 78.31 | 74.87 | 75.42 (+0.55)
Table 4: Performance of students trained with strong strategies on ImageNet. The Swin-T is trained with strategy B3 in Table 1, others are trained with B2. $\dagger$: trained by [44]. $\ddagger$: Pretrained on ImageNet-22K. Teacher |  Student | Top-1 ACC (%)
---|---|---
teacher | student |  KD [16] |  RKD [30] |  SRRL [47] | DIST
ResNet-50† | ResNet-18 | 80.1 | 73.4 | 72.6 | 72.9 | 71.2 | 74.5
ResNet-34 | 76.8 | 77.2 | 76.6 | 76.7 | 77.8
MobileNetV2 | 73.6 | 71.7 | 73.1 | 69.2 | 74.4
EfficientNet-B0 | 78.0 | 77.4 | 77.5 | 77.3 | 78.6
Swin-L‡ | ResNet-50 | 86.3 | 78.5 | 80.0 | 78.9 | 78.6 | 80.2
Swin-T | 81.3 | 81.5 | 81.2 | 81.5 | 82.3

Distillation from stronger training strategies. Recently, the performance of
models on ImageNet has been significantly improved by the sophisticated
training strategies and strong data augmentations (_e.g._ , TIMM [44] achieves
80.4% accuracy on ResNet-50 while the baseline strategy B1 only obtains
76.1%). However, most of the KD methods still conduct experiments with simple
training settings. It is seldomly investigated whether the KD methods are
suitable to the advanced strategies. In this way, we conduct experiments with
advanced training strategies and compare our method with vanilla KD, instance
relation-based RKD [30], and SRRL [47].

We first train traditional CNNs with strong strategies, and also use a strong
ResNet-50 with 80.1% accuracy trained by [44] as the teacher. As results shown
in Table 4, on both similar architectures (ResNet-18, ResNet-34) and
dissimilar architectures (MobileNetV2, EfficientNet-B0), our DIST can achieve
the best performance. Note that RKD and SRRL can perform worse than training
from scratch, especially when the students are small (ResNet-18 and MobileNet)
or the architectures of teacher and student are fairly different (ResNet-50
and Swin-L), this might be because they focus on the intermediate features,
which can be more challenging for the student to recover teacher’s features
compared to predictions.

Furthermore, we experiment on the recent state-of-the-art Swin-Transformer
[27]. The results show that our DIST gains improvements on even more stronger
models and strategies. For example, with Swin-L teacher, our method improves
ResNet-50 and Swin-T by 1.7% and 1.0%, respectively.

CIFAR-100. The results on CIFAR-100 dataset in Table 5 show that, by
distilling on the predicted logits, our method even outperforms those
sophisticatedly-designed feature distillation methods.

Table 5: Evaluation results on CIFAR-100 dataset. The upper and lower models denote teacher and student, respectively.  Method | Same architecture style | Different architecture style
---|---|---
WRN-40-2 WRN-40-1 |  ResNet-56 ResNet-20 |  ResNet-32x4 ResNet-8x4 |  ResNet-50 MobileNetV2 |  ResNet-32x4 ShuffleNetV1 |  ResNet-32x4 ShuffleNetV2
Teacher | 75.61 | 72.34 | 79.42 | 79.34 | 79.42 | 79.42
Student | 71.98 | 69.06 | 72.50 | 64.6 | 70.5 | 71.82
Feature-based methods
FitNet [35] |  72.24$\pm$0.24 |  69.21$\pm$0.36 |  73.50$\pm$0.28 |  63.16$\pm$0.47 |  73.59$\pm$0.15 |  73.54$\pm$0.22
VID [1] |  73.30$\pm$0.13 |  70.38$\pm$0.14 |  73.09$\pm$0.21 |  67.57$\pm$0.28 |  73.38$\pm$0.09 |  73.40$\pm$0.17
RKD [30] |  72.22$\pm$0.20 |  69.61$\pm$0.06 |  71.90$\pm$0.11 |  64.43$\pm$0.42 |  72.28$\pm$0.39 |  73.21$\pm$0.28
PKT [31] |  73.45$\pm$0.19 |  70.34$\pm$0.04 |  73.64$\pm$0.18 |  66.52$\pm$0.33 |  74.10$\pm$0.25 |  74.69$\pm$0.34
CRD [41] |  74.14$\pm$0.22 |  71.16$\pm$0.17 |  75.51$\pm$0.18 |  69.11$\pm$0.28 |  75.11$\pm$0.32 |  75.65$\pm$0.10
Logits-based methods
KD [16] |  73.54$\pm$0.20 |  70.66$\pm$0.24 |  73.33$\pm$0.25 |  67.35$\pm$0.32 |  74.07$\pm$0.19 |  74.45$\pm$0.27
DIST | 74.73$\pm$0.24 | 71.75$\pm$0.30 | 76.31$\pm$0.19 | 68.66$\pm$0.23 | 76.34$\pm$0.18 | 77.35$\pm$0.25

###  4.3 Object Detection

Table 6: Results on COCO validation set. T: teacher; S: student. *: We implement KD using $\tau=1$ and other settings are the same as DIST. Method | AP |  AP50 |  AP75 |  APS |  APM |  APL
---|---|---|---|---|---|---
Two-stage detectors
T: Cascade Mask RCNN-X101 | 45.6 | 64.1 | 49.7 | 26.2 | 49.6 | 60.0
S: Faster RCNN-R50 | 38.4 | 59.0 | 42.0 | 21.5 | 42.1 | 50.3
KD [16]∗ | 39.7 | 61.2 | 43.0 | 23.2 | 43.3 | 51.7
FKD [52] | 41.5 | 62.2 | 45.1 | 23.5 | 45.0 | 55.3
CWD [37] | 41.7 | 62.0 | 45.5 | 23.3 | 45.5 | 55.5
DIST | 40.4 | 61.7 | 43.8 | 23.9 | 44.6 | 52.6
DIST + mimic | 41.8 | 62.4 | 45.6 | 23.4 | 46.1 | 55.0
One-stage detectors
T: RetinaNet-X101 | 41.0 | 60.9 | 44.0 | 23.9 | 45.2 | 54.0
S: RetinaNet-R50 | 37.4 | 56.7 | 39.6 | 20.0 | 40.7 | 49.7
KD [16]∗ | 37.2 | 56.5 | 39.3 | 20.4 | 40.4 | 49.5
FKD [52] | 39.6 | 58.8 | 42.1 | 22.7 | 43.3 | 52.5
CWD [37] | 40.8 | 60.4 | 43.4 | 22.7 | 44.5 | 55.3
DIST | 39.8 | 59.5 | 42.5 | 22.0 | 43.7 | 53.0
DIST + mimic | 40.1 | 59.4 | 43.0 | 23.2 | 44.0 | 53.6

Table 7: Results on Cityscapes val dataset. All models are pretrained on ImageNet. Method | mIoU (%)
---|---
T: DeepLabV3-R101 | 78.07
S: DeepLabV3-R18 | 74.21
SKD [26] | 75.42
IFVD [43] | 75.59
CWD [37] | 75.55
CIRKD [46] | 76.38
DIST | 77.10
S: PSPNet-R18 | 72.55
SKD [26] | 73.29
IFVD [43] | 73.71
CWD [37] | 74.36
CIRKD [46] | 74.73
DIST | 76.31

We further investigate the effectiveness of DIST on downstream tasks. We
conduct experiments on MS COCO object detection dataset [25], and simply
leverage our DIST as an additional supervision on the final predictions of
classes. Following [52, 37], we use the same standard training strategies and
utilize Cascade Mask R-CNN [2] with ResNeXt-101 backbone as the teacher for
two-stage student of Faster R-CNN [23] with ResNet-50 backbone; while for one-
stage RetinaNet [24] with ResNet-50 backbone, the RetinaNet with ResNeXt-101
backbone is utilized as the teacher.

As shown in Table 7, our DIST achieves competitive results on COCO validation
set. For comparisons, we train the vanilla KD under the same settings as our
DIST, the results show that our DIST significantly outperforms vanilla KD by
simply replacing the loss functions. Moreover, by combining DIST with mimic,
which minimizes the mean square error between FPN features of teacher and
student, we can even outperform the state-of-the-art KD methods designed for
object detection.

###  4.4 Semantic Segmentation

We also perform experiments on semantic segmentation, a challenging dense
prediction task. Following [43, 37, 46], we train DeepLabV3 [6] and PSPNet
[54] with ResNet-18 backbone on Cityscapes dataset, and adopt our DIST on the
predictions of classification head using a teacher with ResNet-101 backbone of
DeepLabV3. As the results summarized in Table 7, with only the supervision of
class predictions, our DIST can significantly outperform existing knowledge
distillation methods on semantic segmentation task. For example, our DIST
outperforms recent state-of-the-art method CIRKD [46] by 1.58% on PSPNet-R18.
This demonstrate our effectiveness on relation modeling.

###  4.5 Ablation studies

Effects of inter-class and intra-class correlations. This paper proposes two
types of relations: inter-class and intra-class relations. To validate the
effectiveness of each relation, we conduct experiments to train students with
these relations separately. The results on Table 8 verify that, both inter-
class and intra-class relations can outperform the vanilla KD; also, the
performance could be further boosted by combining them together.

Table 8: Ablation of inter-class and intra-class relations on ImageNet. The student and teacher models are ResNet-18 and ResNet-34, respectively. Method | Inter | Intra | ACC (%)
---|---|---|---
KD | - | - | 71.21
DIST (KL div.) |  |  | 70.61
DIST (KL div.) |  |  | 71.62
DIST |  |  | 71.63
DIST |  |  | 71.55
DIST |  |  | 72.07

Effect of intra-class relation in vanilla KD. To investigate the effectiveness
of intra-class relation in vanilla KD, we adopt experiments to train our DIST
using KL divergence as the relation metric, denoted as DIST (KL
div.)222Specifically, the vanilla KD is the same as DIST (KL div.) with inter-
class relation only.. As the results summarized in Table 8, adding intra-class
relation in the vanilla KD can also improve the performance (from 71.21% to
71.62%). However, when the student is trained with intra-class relation only,
the improvement of using KL divergence is less significant than using Pearson
correlation (70.61% vs. 71.55%), since the means and variances of intra-class
distributions could be varied.

Effect of training students with KD loss only. Training student with only the
KD loss can better reflect the distillation ability and the information
richness of supervision signals. As results in Table 9 show that, when the
student is trained with only the KD loss, our DIST significantly outperforms
the vanilla KD. Without using the ground-truth labels, it can even outperform
the standalone training accuracy, which indicates the effectiveness of our
DIST in distilling those truly-beneficial relations.

Table 9: Comparisons of training KD with or without the classification loss on ImageNet. The student and teacher models are ResNet-18 and ResNet-34, respectively. The original accuracy of ResNet-18 without KD is 69.76%. Method | w/ cls. loss | w/o cls. loss
---|---|---
KD | 71.21 | 68.12
DIST | 72.07 | 70.65

More ablation studies can be found in Section A.3.

##  5 Conclusion

This paper presents a new knowledge distillation (KD) method named DIST to
implement better distillation from a stronger teacher. We empirically study
the catastrophic discrepancy problem between the student and a stronger
teacher, and propose a relation-based loss to relax the exact match of KL
divergence in a linear sense. Our method DIST is simple yet effective in
handling strong teachers. Extensive experiments show our superiority in
various benchmark tasks. For example, DIST even outperforms state-of-the-art
KD methods designed specifically for object detection and semantic
segmentation.

## Acknowledgements

This work was supported in part by the Australian Research Council under
Project DP210101859 and the University of Sydney Research Accelerator (SOAR)
Prize.

## References

  * [1] S. Ahn, S. X. Hu, A. Damianou, N. D. Lawrence, and Z. Dai.  Variational information distillation for knowledge transfer.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9163–9171, 2019.
  * [2] Z. Cai and N. Vasconcelos.  Cascade r-cnn: High quality object detection and instance segmentation.  IEEE Transactions on Pattern Analysis and Machine Intelligence, page 1–1, 2019.
  * [3] K. Chandrasegaran, N.-T. Tran, Y. ZHAO, and N. man Cheung.  To smooth or not to smooth? on compatibility between label smoothing and knowledge distillation, 2022.
  * [4] H. Chen, Y. Wang, C. Xu, C. Xu, and D. Tao.  Learning student networks via feature embedding.  IEEE Transactions on Neural Networks and Learning Systems, 32(1):25–35, 2020.
  * [5] L.-C. Chen, G. Papandreou, F. Schroff, and H. Adam.  Rethinking atrous convolution for semantic image segmentation.  arXiv preprint arXiv:1706.05587, 2017.
  * [6] L.-C. Chen, Y. Zhu, G. Papandreou, F. Schroff, and H. Adam.  Encoder-decoder with atrous separable convolution for semantic image segmentation.  In Proceedings of the European conference on computer vision (ECCV), pages 801–818, 2018.
  * [7] P. Chen, S. Liu, H. Zhao, and J. Jia.  Distilling knowledge via knowledge review.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5008–5017, 2021.
  * [8] J. H. Cho and B. Hariharan.  On the efficacy of knowledge distillation.  In Proceedings of the IEEE/CVF international conference on computer vision, pages 4794–4802, 2019.
  * [9] E. D. Cubuk, B. Zoph, J. Shlens, and Q. V. Le.  Randaugment: Practical automated data augmentation with a reduced search space.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, pages 702–703, 2020.
  * [10] Y. Dodge.  The concise encyclopedia of statistics.  Springer Science & Business Media, 2008.
  * [11] S. Du, S. You, X. Li, J. Wu, F. Wang, C. Qian, and C. Zhang.  Agree to disagree: Adaptive ensemble knowledge distillation in gradient space.  advances in neural information processing systems, 33:12345–12355, 2020.
  * [12] J. Gou, B. Yu, S. J. Maybank, and D. Tao.  Knowledge distillation: A survey.  International Journal of Computer Vision, 129(6):1789–1819, 2021\.
  * [13] K. He, X. Zhang, S. Ren, and J. Sun.  Deep residual learning for image recognition.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770–778, 2016.
  * [14] B. Heo, J. Kim, S. Yun, H. Park, N. Kwak, and J. Y. Choi.  A comprehensive overhaul of feature distillation.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1921–1930, 2019.
  * [15] B. Heo, M. Lee, S. Yun, and J. Y. Choi.  Knowledge transfer via distillation of activation boundaries formed by hidden neurons.  In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 3779–3787, 2019.
  * [16] G. Hinton, O. Vinyals, and J. Dean.  Distilling the knowledge in a neural network.  arXiv preprint arXiv:1503.02531, 2015.
  * [17] A. G. Howard, M. Zhu, B. Chen, D. Kalenichenko, W. Wang, T. Weyand, M. Andreetto, and H. Adam.  Mobilenets: Efficient convolutional neural networks for mobile vision applications.  arXiv preprint arXiv:1704.04861, 2017.
  * [18] T. Huang, Z. Li, H. Lu, Y. Shan, S. Yang, Y. Feng, F. Wang, S. You, and C. Xu.  Relational surrogate loss learning.  In International Conference on Learning Representations, 2022.
  * [19] T. Huang, S. You, B. Zhang, Y. Du, F. Wang, C. Qian, and C. Xu.  Dyrep: Bootstrapping training with dynamic re-parameterization.  arXiv preprint arXiv:2203.12868, 2022.
  * [20] M. G. Kendall.  A new measure of rank correlation.  Biometrika, 30(1/2):81–93, 1938.
  * [21] J. Kim, S. Park, and N. Kwak.  Paraphrasing complex network: Network compression via factor transfer.  arXiv preprint arXiv:1802.04977, 2018.
  * [22] S. Kong, T. Guo, S. You, and C. Xu.  Learning student networks with few data.  In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 4469–4476, 2020.
  * [23] T.-Y. Lin, P. Dollár, R. Girshick, K. He, B. Hariharan, and S. Belongie.  Feature pyramid networks for object detection.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2117–2125, 2017.
  * [24] T.-Y. Lin, P. Goyal, R. Girshick, K. He, and P. Dollár.  Focal loss for dense object detection.  In Proceedings of the IEEE international conference on computer vision, pages 2980–2988, 2017.
  * [25] T.-Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollár, and C. L. Zitnick.  Microsoft coco: Common objects in context.  In European conference on computer vision, pages 740–755. Springer, 2014.
  * [26] Y. Liu, C. Shu, J. Wang, and C. Shen.  Structured knowledge distillation for dense prediction.  IEEE transactions on pattern analysis and machine intelligence, 2020\.
  * [27] Z. Liu, Y. Lin, Y. Cao, H. Hu, Y. Wei, Z. Zhang, S. Lin, and B. Guo.  Swin transformer: Hierarchical vision transformer using shifted windows.  arXiv preprint arXiv:2103.14030, 2021.
  * [28] S. Marcel and Y. Rodriguez.  Torchvision the machine-vision package of torch.  In Proceedings of the 18th ACM international conference on Multimedia, pages 1485–1488, 2010.
  * [29] S. I. Mirzadeh, M. Farajtabar, A. Li, N. Levine, A. Matsukawa, and H. Ghasemzadeh.  Improved knowledge distillation via teacher assistant.  In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 5191–5198, 2020.
  * [30] W. Park, D. Kim, Y. Lu, and M. Cho.  Relational knowledge distillation.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3967–3976, 2019.
  * [31] N. Passalis, M. Tzelepi, and A. Tefas.  Probabilistic knowledge transfer for lightweight deep representation learning.  IEEE Transactions on Neural Networks and Learning Systems, 32(5):2030–2039, 2020.
  * [32] A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga, et al.  Pytorch: An imperative style, high-performance deep learning library.  Advances in neural information processing systems, 32:8026–8037, 2019.
  * [33] K. Pearson.  Vii. mathematical contributions to the theory of evolution.—iii. regression, heredity, and panmixia.  Philosophical Transactions of the Royal Society of London. Series A, containing papers of a mathematical or physical character, (187):253–318, 1896.
  * [34] B. Peng, X. Jin, J. Liu, D. Li, Y. Wu, Y. Liu, S. Zhou, and Z. Zhang.  Correlation congruence for knowledge distillation.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 5007–5016, 2019.
  * [35] A. Romero, N. Ballas, S. E. Kahou, A. Chassang, C. Gatta, and Y. Bengio.  Fitnets: Hints for thin deep nets.  arXiv preprint arXiv:1412.6550, 2014.
  * [36] Z. Shen, Z. Liu, D. Xu, Z. Chen, K.-T. Cheng, and M. Savvides.  Is label smoothing truly incompatible with knowledge distillation: An empirical study.  In International Conference on Learning Representations, 2020.
  * [37] C. Shu, Y. Liu, J. Gao, Z. Yan, and C. Shen.  Channel-wise knowledge distillation for dense prediction.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 5311–5320, 2021.
  * [38] K. Simonyan and A. Zisserman.  Very deep convolutional networks for large-scale image recognition.  arXiv preprint arXiv:1409.1556, 2014.
  * [39] W. Son, J. Na, J. Choi, and W. Hwang.  Densely guided knowledge distillation using multiple teacher assistants.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 9395–9404, 2021.
  * [40] M. Tan and Q. Le.  Efficientnet: Rethinking model scaling for convolutional neural networks.  In International Conference on Machine Learning, pages 6105–6114. PMLR, 2019.
  * [41] Y. Tian, D. Krishnan, and P. Isola.  Contrastive representation distillation.  In International Conference on Learning Representations, 2019.
  * [42] F. Tung and G. Mori.  Similarity-preserving knowledge distillation.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1365–1374, 2019.
  * [43] Y. Wang, W. Zhou, T. Jiang, X. Bai, and Y. Xu.  Intra-class feature variation distillation for semantic segmentation.  In European Conference on Computer Vision, pages 346–362. Springer, 2020.
  * [44] R. Wightman, H. Touvron, and H. Jégou.  Resnet strikes back: An improved training procedure in timm.  arXiv preprint arXiv:2110.00476, 2021.
  * [45] S. Xie, R. Girshick, P. Dollár, Z. Tu, and K. He.  Aggregated residual transformations for deep neural networks.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1492–1500, 2017.
  * [46] C. Yang, H. Zhou, Z. An, X. Jiang, Y. Xu, and Q. Zhang.  Cross-image relational knowledge distillation for semantic segmentation.  arXiv preprint arXiv:2204.06986, 2022.
  * [47] J. Yang, B. Martinez, A. Bulat, and G. Tzimiropoulos.  Knowledge distillation via softmax regression representation learning.  In International Conference on Learning Representations, 2020.
  * [48] S. You, T. Huang, M. Yang, F. Wang, C. Qian, and C. Zhang.  Greedynas: Towards fast one-shot nas with greedy supernet.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1999–2008, 2020.
  * [49] S. You, C. Xu, C. Xu, and D. Tao.  Learning from multiple teacher networks.  In Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pages 1285–1294, 2017.
  * [50] S. Zagoruyko and N. Komodakis.  Paying more attention to attention: Improving the performance of convolutional neural networks via attention transfer.  arXiv preprint arXiv:1612.03928, 2016.
  * [51] H. Zhang, M. Cisse, Y. N. Dauphin, and D. Lopez-Paz.  mixup: Beyond empirical risk minimization.  In International Conference on Learning Representations, 2018.
  * [52] L. Zhang and K. Ma.  Improve object detection with feature-based knowledge distillation: Towards accurate and efficient detectors.  In International Conference on Learning Representations, 2020.
  * [53] X. Zhang, X. Zhou, M. Lin, and J. Sun.  Shufflenet: An extremely efficient convolutional neural network for mobile devices.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 6848–6856, 2018.
  * [54] H. Zhao, J. Shi, X. Qi, X. Wang, and J. Jia.  Pyramid scene parsing network.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2881–2890, 2017.

## Checklist

  1. 1.

For all authors…

    1. (a)

Do the main claims made in the abstract and introduction accurately reflect
the paper’s contributions and scope? [Yes]

    2. (b)

Did you describe the limitations of your work? [Yes] See Appendix.

    3. (c)

Did you discuss any potential negative societal impacts of your work? [Yes]
See Appendix.

    4. (d)

Have you read the ethics review guidelines and ensured that your paper
conforms to them? [Yes]

  2. 2.

If you are including theoretical results…

    1. (a)

Did you state the full set of assumptions of all theoretical results? [N/A]

    2. (b)

Did you include complete proofs of all theoretical results? [N/A]

  3. 3.

If you ran experiments…

    1. (a)

Did you include the code, data, and instructions needed to reproduce the main
experimental results (either in the supplemental material or as a URL)? [Yes]
Training details are provided in the paper. Training code and logs are
released at GitHub: https://github.com/hunto/DIST_KD.

    2. (b)

Did you specify all the training details (e.g., data splits, hyperparameters,
how they were chosen)? [Yes]

    3. (c)

Did you report error bars (e.g., with respect to the random seed after running
experiments multiple times)? [Yes] Standard deviations on CIFAR-100 are
reported.

    4. (d)

Did you include the total amount of compute and the type of resources used
(e.g., type of GPUs, internal cluster, or cloud provider)? [Yes]

  4. 4.

If you are using existing assets (e.g., code, data, models) or
curating/releasing new assets…

    1. (a)

If your work uses existing assets, did you cite the creators? [Yes]

    2. (b)

Did you mention the license of the assets? [Yes]

    3. (c)

Did you include any new assets either in the supplemental material or as a
URL? [Yes] Code and training logs are included.

    4. (d)

Did you discuss whether and how consent was obtained from people whose data
you’re using/curating? [N/A]

    5. (e)

Did you discuss whether the data you are using/curating contains personally
identifiable information or offensive content? [N/A]

  5. 5.

If you used crowdsourcing or conducted research with human subjects…

    1. (a)

Did you include the full text of instructions given to participants and
screenshots, if applicable? [N/A]

    2. (b)

Did you describe any potential participant risks, with links to Institutional
Review Board (IRB) approvals, if applicable? [N/A]

    3. (c)

Did you include the estimated hourly wage paid to participants and the total
amount spent on participant compensation? [N/A]

##  Appendix A Appendix

###  A.1 Implementation of DIST

This section presents the implementation code of DIST, as shown in Figure 4.
With only the output logits of student and teacher models, our DIST leverages
quite simple inputs and computations, and is easy to implement.

Figure 4: The PyTorch [32] implementation of DIST.

###  A.2 Related work

There exist some KD methods [34, 30, 4] which transfer the relations between
instances from teacher to student (see Figure 3 for illustration). The purpose
of these methods is to learn the similarity relationships between instances
from teacher, _e.g._ , the semantic spaces of instances with the same class
should be close, while the instances of different classes should have larger
distances. To distill this instance relation, CCKD [34] proposes to measure
the correlation between each instance pair-wisely, then minimizes the
difference of correlation scores between teacher and student; RKD [30]
formulates the instance relation via the distance and angle between each pair;
4 introduces a locality preserving loss to embed the teacher’s instance
relation into student.

Nevertheless, in this paper, we focus on the probabilistic prediction instead
of intermediate features, and leverage the relation on a new perspective to
relax and replace the KL divergence in almost all KD methods, which help us to
achieve better performance especially when the student is trained with a
stronger teacher. Besides, different from previous works which measure
relations between instances of teacher and student individually, we propose
inter-class relation and intra-class relation simultaneously to better distill
from a teacher.

###  A.3 More ablation studies

Correlations between teacher and student. To validate the effectiveness of our
correlation-based loss, we measure the correlations between teacher and
student models, where the student models are trained by plain classification
loss, KD, and our DIST. We choose commonly used Pearson correlation
coefficient, Spearman’s [10] and Kendall’s Tau [20] rank correlation
coefficients as the metrics of correlation. As summarized in Figure 5, our
DIST obtains higher inter-class and intra-class correlations compared to
baselines.

(a) Inter-class correlation (b) Intra-class correlation

Figure 5: Correlations between ResNet-18 student and ResNet-34 teacher. We
train the methods on ImageNet with B1 strategy.

Using cosine similarity in DIST. In our method, the relation matching can be
any function with the same form of Eq.(4). We simply adopt a commonly used
Pearson correlation as our relation metric in DIST. Here we conduct
experiments to investigate the efficacy of our method with cosine similarity.

Table 10: Ablation of cosine similarity and Pearson correlation in DIST. We train the student ResNet-18 and teacher ResNet-34 on ImageNet with or without label smoothing (LS). Method | w/o LS | w/ LS
---|---|---
Teacher | 73.31 | 73.78
KD ($\tau=4$) | 71.21 | 70.71
KD ($\tau=1$) | 71.49 | 71.37
DIST (cosine) | 71.79 | 71.63
DIST (Pearson) | 72.07 | 72.18

Both cosine similarity and Pearson correlation coefficient can evaluate the
relations between teacher and student. Compared to the scale invariance in
cosine similarity, the Pearson correlation has an additional shift-invariance
by centering the vectors first (see Eq.(7)), and it could be more robust to
the distribution changes. We conduct experiments to compare these two metrics
in our DIST and train the models with or without label smoothing. Since recent
studies [36, 3] state that KD with high temperatures is incompatible with
label smoothing, we also train the models with KD ($\tau=1$). As shown in
Table 10, adopting DIST with Pearson correlation achieves higher accuracies
compared to KD and DIST with cosine similarity, especially when the teacher
and student are trained with label smoothing (the predicted probabilistic
distributions would be shifted by it). As a result, the scale-and-shift-
invariant Pearson correlation may be a better metric for measuring relations
in DIST.

###  A.4 Comparisons of training speed

We compare the training speed of our DIST with vanilla KD [16], RKD [30], CRD
[41], and SRRL [47], as summarized in Table 11. Our DIST has almost the same
highest training speed as the vanilla KD, outperforming other feature-based KD
methods.

Table 11: Average training speed (batches / second) of training ResNet-18 student with ResNet-34 teacher on ImageNet using strategy B1. The speed is tested based on our implementations on 8 NVIDIA V100 GPUs. KD | RKD | SRRL | CRD | DIST
---|---|---|---|---
[16] | [30] | [47] | [41] |
14.28 | 11.11 | 12.98 | 8.33 | 14.19

###  A.5 Landscapes of matching functions

As discussed in our main text, the matching functions such as KL divergence
and MSE are used to match the outputs between student and teacher in KD. In
DIST, we propose to relax the match in KD with relational matching functions,
_i.e._ , Pearson distance. Here we visualize the landscapes of KL divergence,
MSE, cosine distance, and Pearson distance to better show our effectiveness on
relaxing the constraints. As shown in Figure 6 and Figure 7, we randomly
generate a 2-dimensional vector as the output logits of student and teacher,
then we adjust the first element in the teacher and student logits (namely,
$Z_{1}^{(s)}$ and $Z_{1}^{(t)}$) independently and fix the remaining elements,
to draw the loss or gradient landscapes w.r.t. $Z_{1}^{(s)}$ and
$Z_{1}^{(t)}$. Specifically, when $Z_{1}^{(s)}=Z_{1}^{(t)}$, the logits of
student and teacher are identical, and the loss values of these functions
should be zero. When we change the values of logits, the curves shows that how
the losses are changed by it, _i.e._ , the sensitivity of loss to the
distribution shifts. The landscapes of gradient in Figure 7 show that, the PCC
in our DIST has sharper gradients curves in non-optimal regions and has the
largest area on the region of optima. This indicates that our DIST can get
optimized quickly with wider ranges of $Z^{(t)}_{1}$ and $Z^{(s)}_{1}$, thus
it would have smaller conflict to the task loss.

Figure 6: Visualization of loss landscapes. (a) MSE. (b) KL divergence. (c)
Cosine distance. (d) Pearson distance. Figure 7: Visualization of gradient
landscapes.

###  A.6 Discussion

Limitations. DIST can improve multi-class classification consistently in
various tasks such as classification, object detection, and semantic
segmentation. However, it would be less-effective on binary classification
task, as the task only contains two classes and the information in inter-class
relation is limited.

Societal impacts. Investigating the efficacy of the proposed method would
consume considerable computing resources. These efforts can contribute to
increased carbon emissions, which could raise environmental concerns. However,
the proposed knowledge distillation method can improve the performance of
light-weight compact models, where replacing the heavy models with light
models in production could save more energy consumption, and it is necessary
to validate the efficacy of DIST adequately.