# Feature Structure Distillation for BERT Transferring

Hee-Jun Jung${}^{\href https://orcid.org/0000-0002-9805-8192}$, Doyeon
Kim${}^{\href https://orcid.org/0000-0001-7728-3414}$, Seung-Hoon Na${}^{\href
https://orcid.org/0000-0002-4372-7125}$, Kangil Kim${}^{\href
https://orcid.org/0000-0003-3220-6401}$ H.-J. jung, D. Kim, and K. Kim are
with the AI Graduate School, Gwangju Institute of Science and Technology
(GIST), Gwangju 61005, Republic of Korea (e-mail: $\\{$jungheejun93,
<EMAIL_ADDRESS>$\\{$kikim01$\\}$@gist.ac.kr).N.-H. Na, is with
the Department of Computer Science and Engineering, Jeonbook National
University, Jeonju 54896, Republic of Korea (e-mail:
nash@jbnu.ac.kr).(Corresponding author: Kangil Kim.)

###### Abstract

Knowledge distillation is an approach to transfer information on
representations from a teacher to a student by reducing their difference. A
challenge of this approach is to reduce the flexibility of the student’s
representations inducing inaccurate learning of the teacher’s knowledge. To
resolve it in BERT transferring, we investigate distillation of structures of
representations specified to three types: intra-feature, local inter-feature,
global inter-feature structures. To transfer them, we introduce feature
structure distillation methods based on the Centered Kernel Alignment, which
assigns a consistent value to similar features structures and reveals more
informative relations. In particular, a memory-augmented transfer method with
clustering is implemented for the global structures. In the experiments on the
nine tasks for language understanding of the GLUE dataset, the proposed
methods effectively transfer the three types of structures and improve
performance compared to state-of-the-art distillation methods. Indeed, the
code for the methods is available in https://github.com/maroo-sky/FSD.

###### Index Terms:

Knowledge distillation, Centered Kernel Alignment, feature structure, BERT,
natural language processing.

##  I Introduction

In current deep learning models, knowledge distillation (KD) is a common
approach to transfer information of features of a larger model to a smaller
student model. This approach reduces the difference in prediction confidence
between two models. Confidence is usually represented as a probability vector.
Moreover, distillation can be applied to various vector distributions to
transfer more feature information. For example, distribution on an
intermediate layer [1, 2] or a final fully connected layer [3] have been
directly compared. To transfer more rich information, pairwise relations
between features [4, 5, 6] have also been used. A problem with the direct
fitting of a vector is its huge flexibility in geometric space, even in the
same setting of neural networks. This flexibility may cause ambiguity to the
guide for a student to learn the teacher’s knowledge.

Transferring more rich information on representations’ connectivity is a
possible solution. Centered Kernel Alignment (CKA) [7] is a suitable metric
for this approach as it assigns a similarity value to feature structure.
Furthermore, its score is more consistent on potentially similar
representations trained on different architectures and layers [8]. This
property is expected to help distillation focus on more informative feature
distribution. Implementations of this approach have been reported in a few
recent computer vision tasks, but widely used BERT model in natural language
processing is not sufficiently studied yet.

In this work, we propose feature structure distillation (FSD) methods to adapt
CKA to KD between BERT models. The proposed methods transfer rich information
categorized into three types of structures on the feature representations:
intra-feature, local inter-feature, and global inter-feature structures. A
separate distillation loss is introduced for each structure defined on the
feature distribution generated from the penultimate layer. To obtain the
global inter-feature structures over the full batch of training samples, we
newly add a memory architecture that is induced via clustering.

We present experiments on the General Language Understanding Evaluation (GLUE)
[9] benchmark with BERT distilled by FSD methods. In the results, FSD methods
show possibility that these methods outperform other state-of-the-art KD
methods and even teacher models in some tasks. Far from many previous works
[3, 10, 2, 1, 4, 5, 6, 11] which mainly focus on model performance, we provide
the results of the restoration rate of the teacher’s prediction and the
similarity change of geometric structures for deeper understanding of the
structure distillation.

Our key contributions are summarized as follows:

  * •

We adapt CKA to KD for more informative transfer of structures in BERT.

  * •

We categorize three feature structures (intra-feature, local inter-feature,
and global inter-feature structure).

  * •

We propose their distillation methods, especially memory-augmentation with
clustering for global structures.

  * •

We empirically analyze restoration rate, patterns of transferring feature
structures, and task-specific properties.

  * •

We validate practical usefulness over a wide range of language understanding
tasks (GLUE benchmark).

Terms and Notations
---
feature | a representation vector
relation | a pair-wise relation of two features
feature structure | a set of relations
feature distribution | a set of features
$\displaystyle a$ | A scalar (integer or real)
$\displaystyle{\bm{a}}$ | a vector
$\displaystyle{\bm{A}}$ | a matrix
$\displaystyle{\bm{I}}_{n}$ | identity matrix with $n$ rows and $n$ columns
$\displaystyle{\bm{J}}_{n}$ | all-ones matrix with $n$ rows and $n$ columns
$\displaystyle\mathbb{R}$ | the set of real numbers
$\displaystyle\mathbb{R}^{m\times n}$ |  $m$ by $n$ shape of matrix
$\displaystyle\mathbb{R}^{m\times n\times k}$ |  $m$ by $n$ by $k$ shape of 3rd-order tensor
$\displaystyle\\{0,1,\dots,n\\}$ | the set of all integers between $0$ and $n$
$\displaystyle D_{\mathrm{KL}}(P\|Q)$ | Kullback-Leibler divergence of P and Q
$\displaystyle||{\bm{x}}||_{2}$ | L2 norm of ${\bm{x}}$

##  II Related Work

###  II-A Analysis of Similarity of Representation

The similarity between representations of deep networks has been measured by
various methods. Canonical Correlation Analysis (CCA) [12] estimates the
association between two variables and identifies a linear relationship with
weight to maximize correlation. CCA is sensitive to perturbation when the
condition number of representations is large [13]. To reduce the sensitivity
of perturbation, Singular Value Canonical correlation Analysis (SVCCA) [14]
applies singular value decomposition to use more important principal
components, and Projection Weighted CCA (PWCCA) [15] assigns higher weights to
more important canonical correlations. These methods aimed to assign the same
relation of flexibly located representations in different models, but the
consistency of their methods is insufficient [8]. CKA is an alternative for
enhancing the invariance to orthogonal transformation and isotropic scaling,
which is expected to enhance the consistency [8] The metric improved the
performance of alignment-based algorithms [7], measuring the similarity
between kernels or kernel matrices. Furthermore, CKA outperforms CCA, SVCCA,
and PWCCA on the test of identifying corresponding layers [8].

###  II-B Knowledge Distillation for BERT

KD [16] is a method to transfer dark knowledge of a large teacher model to a
smaller student model while preserving the training accuracy, and this method
is applied to DistilBERT [17]. An extension of KD is to directly reduce
distance between representations. For example, TinyBERT [3] delivers word
embedding, self-attention head, and representations on selected intermediate
layers. MobileBERT [10] moves representations on all layers to a student of
the same number of layers, and MiniLM [2] uses relations between the values in
the self-attention and the attention distribution that is computed from the
scaled dot products of the queries and keys [18]. DistilBERT and TinyBERT
models perform distillation on the pre-training and fine-tuning stages but
MobileBERT and MiniLM models operate distillation only on the pre-training
stage. To reduce the interference of other factors in the analysis, we conduct
distillation on the fine-tuning stage as patient knowledge distillation (PKD)
[1]. The method implements teacher representations from multiple intermediate
layers normalized to the student layers in downstream tasks, enabling transfer
between neural networks of different numbers of layers. These methods penalize
the difference between teacher and student features, then force them closer in
the same vector space. However, CKA allows of using different vector spaces
and dimensions.

###  II-C Transferring Rich Information

Transferring rich information as differences or relation has been introduced
in a few vision tasks. Correlation Congruence for Knowledge Distillation
(CCKD) [4] transfers a correlation matrix between representations generated
from kernels, Relational Knowledge Distillation (RKD) [5] evaluates the
difference in Euclidean distance or cosine similarity, and Local Correlation
Consistency for Knowledge Distillation (LKD) [6] additionally uses the
difference of angles, each of which is determined by three representations.
Indeed, Contextual Knowledge Distillation CKD [11] proposed layer transforming
relation as well as word relation-based contextual knowledge distillation with
same manner of RKD to evaluate the difference of sturcture. It extends
transferring teacher structure from word level (whthin same layer) to layer
level (over layers). Differently, our methods cover a wider range of feature
structures as global structures and effective intra-feature structures are
specifically designed for transformers. To measure the difference between
structures, we adopt CKA, which effectively maintains the implicit relations
between representations [8]. CKA has been adopted for KD in convolutional
networks [19] and has shown successful performance improvement, but the
extension to global structure and a deeper analysis in the transformer
networks has rarely been discussed.

(a)

(b)

Figure 1: Overview of the proposed loss functions of distillation methods for
three feature structures: $\mathcal{L}_{I}$ and $\mathcal{L}_{L}$ in Figure 1a
uses CKA between teacher and student representations on the penultimate layer.
In Figure 1b, the first stage is an unsupervised post-training and the second
stage is a memory-augmented transfer method. In the first stage,
$\displaystyle{\bm{M}}^{T}$ is first trained to memorize the centroids of
representations on the penultimate layer of a teacher model. Then, its stored
centroids are transferred to a student model by $\mathcal{L}_{G}$ in the
second stage.

##  III Method

In this section, we clarify the KD settings for presenting the FSD methods.

###  III-A Base Knowledge Distillation

We set two compatible base settings for KD to transfer a feature distribution
introduced in previous works [16, 1]. In the former setting, given $N$
training samples for training a student model $S$, a fine-tuned teacher model
$T$ transfers a feature distribution on the final layer by training $S$ with
the similarity loss as

$\mathcal{L}_{KLD}=\sum_{i=1}^{N}\displaystyle
D_{\mathrm{KL}}(\frac{\emph{f}_{T}(\displaystyle{\bm{x}}_{i})}{\displaystyle\tau}\|\frac{\emph{f}_{S}(\displaystyle{\bm{x}}_{i})}{\displaystyle\tau}),$
(1)

where $\emph{f}(\cdot)$ is a neural network to generate a probability vector
through softmax function and layers. An input sample for the network is
composed of vector $\displaystyle{\bm{x}}_{i}$ and its ground-truth
$\displaystyle{\bm{y}}_{i}$, where $\displaystyle i$ is the sample index in
the training data. The temperature $\displaystyle\tau$ is to control
relaxation. To distill teacher knowledge, the model is trained by a task-
specific cross-entropy loss $\mathcal{L}_{CE}$ with $\mathcal{L}_{KLD}$ as

$\mathcal{L}_{VKD}=\displaystyle\alpha\mathcal{L}_{CE}+(1-\displaystyle\alpha)\mathcal{L}_{KLD},$
(2)

where $\alpha$ is the interpolation rate between two loss functions, which
needs to be tuned empirically. This approach to transfer a feature with the
label is called vanilla knowledge distillation (VKD) in this paper. The latter
setting follows the Eq. 2, but the similarity loss is replaced with the
distance between hidden vectors generated from the several intermediate layers
and penultimate layers $f^{\prime}(\cdot)$ of $T$ and $S$, which is the method
of PKD [1]. As the setting of both methods, we initialize the parameters of
$S$ from the pre-trained $T$ and then perform distillation on the fine-tuning
stage.

In this paper, the proposed FSD methods are used as distillation loss
functions in training the student model as

$\mathcal{L}=\mathcal{L}_{VKD}+\displaystyle\beta\mathcal{L}_{\kappa},$ (3)

where $\displaystyle\beta$ is a hyper-parameter to control the interpolation
rate between VKD loss and the proposed distillation loss. Notation $\kappa$
indicates a selected type among the five structure distillation losses, i.e.,
$\\{I\textrm{(intra-feature)},$ $L\textrm{(local inter-feature)},$
$G\textrm{(global inter-feature)},~{}IL\textrm{(intra-local inter-feature)},$
$ILG\textrm{(intra-local-global inter-feature)}\\}$ explain in the following
subsections.

###  III-B Feature Structure Transfer Method

####  III-B1 Overview

In this paper, we propose FSD methods to transfer more rich information on
representations by comparing feature structures relations rather than
relation. Feature structure is split into three groups with respect to their
locality: intra-feature, local inter-feature, and global inter-feature
structures. Fig. 1 shows the distinction of the structures.

####  III-B2 Similarity Between Feature Structures

We adapt CKA for evaluating similarity between feature structures in order to
use its robustness to the flexibility of feature distribution and consequently
to reduce ambiguity of distillation.

$\emph{CKA}(\displaystyle{\bm{E}}^{1},\displaystyle{\bm{E}}^{2})=\frac{\emph{HSIC}(\displaystyle{\bm{K}},\displaystyle{\bm{L}})}{\sqrt{\emph{HSIC}(\displaystyle{\bm{K}},\displaystyle{\bm{K}})\emph{HSIC}(\displaystyle{\bm{L}},\displaystyle{\bm{L}})}},$
(4)

where $\displaystyle{\bm{E}}^{1}$ and $\displaystyle{\bm{E}}^{2}$ are
features; the function HSIC is the Hilbert-Schmidt Independence Criterion for
determining Independence of two sets of variables [20];
$\displaystyle{\bm{K}}={\displaystyle{\bm{E}}^{1}}^{\textsc{t}}\displaystyle{\bm{E}}^{1}$,
$\displaystyle{\bm{L}}={\displaystyle{\bm{E}}^{2}}^{\textsc{t}}\displaystyle{\bm{E}}^{2}$.
The function HSIC is the Hilbert-Schmidt Independence Criterion [20] defined
as

$\emph{HSIC}(\displaystyle{\bm{K}},\displaystyle{\bm{L}})=\frac{1}{(\displaystyle
N-1)^{2}}tr(\displaystyle{\bm{K}}\displaystyle{\bm{C}}\displaystyle{\bm{L}}\displaystyle{\bm{C}}),$
(5)

where $tr$ is a trace in a matrix, $\displaystyle{\bm{C}}$ is a centering
matrix $\displaystyle{\bm{C}}_{\displaystyle
n}=\displaystyle{\bm{I}}_{\displaystyle n}-\frac{1}{\displaystyle
n}\displaystyle{\bm{J}}_{\displaystyle n}$.

In each proposed method, we use different $\displaystyle{\bm{E}}^{1}$ and
$\displaystyle{\bm{E}}^{2}$, but they are all based on the hidden vectors
generated from the penultimate layer of teacher, and student, notated as
$\displaystyle{\bm{H}}^{T}$ for the teacher and $\displaystyle{\bm{H}}^{S}$
for the student in the shape of $\mathbb{R}^{|B|\times|W||D|}$. The constants
$|\displaystyle B|$, $|\displaystyle W|$, and $|\displaystyle D|$ are the
number of samples in a mini-batch, the maximum sequence length, and the hidden
state dimension, respectively.

####  III-B3 Intra-Feature Structure Distillation (FSDI)

intra-feature structure implies the set of difference values between segments
of the hidden vector from the penultimate layer from a single input sample. In
the transformer networks, the unit for segmentation is a token. To obtain the
difference of token-level structures between the teacher and student model, we
split the hidden vector
$\displaystyle{\bm{H}}^{S}_{i}\in\displaystyle\mathbb{R}^{|W||D|}$ into token-
level feature vectors
$\displaystyle{\bm{H}}^{S}_{i,j}\in\displaystyle\mathbb{R}^{|D|}$ for the
$j$th token from $i$th input sample and this is equally applied to the teacher
model. Then, we reshape teacher and student penultimate layer representation
$\displaystyle{\bm{H}}^{T}$ and $\displaystyle{\bm{H}}^{S}$ into
$\displaystyle{\bm{H}}^{T}_{re}$ and $\displaystyle{\bm{H}}^{S}_{re}$ in shape
of $\mathbb{R}^{|B|\times|W|\times|D|}$. The loss function implying the
difference is defined as

$\mathcal{L}_{I}=-\frac{1}{|B|}\sum_{i=1}^{|B|}\log\emph{CKA}|(\displaystyle{\bm{H}}^{S}_{re},\displaystyle{\bm{H}}^{T}_{re})|.$
(6)

####  III-B4 Local Inter-Feature Structure Distillation (FSDL)

Local inter-feature structure implies the set of difference between hidden
vectors generated at the penultimate layer from the samples of a mini-batch.
Compared to the intra-feature structure method, it deals with the structure
between samples rather than internal units from a single sample. The
distillation loss adapting CKA for comparing the local inter-feature
structures is

$\displaystyle\mathcal{L}_{L}$ $\displaystyle=$
$\displaystyle-\log{|\emph{CKA}(\displaystyle{\bm{H}}^{S},\displaystyle{\bm{H}}^{T})|}.$
(7)

####  III-B5 Global Inter-Feature Structure Distillation via Memory
Augmentation (FSDG)

Figure 2: The difference of covered relations between a local structure and a
global structure is shown. The blue blocks of the right-side matrix are
relations of the global structure in the full batch. The red blocks are found
in mini-batches (local structure). Variously colored and shaped markers
indicate features and the matrix shows relations over the features.

The local approach is easy to implement but transfers only the partial inter-
feature structures because it evaluates the structures only for samples in the
same mini-batch. To transfer all inter-feature structures, a pairwise
difference for two samples have to be evaluated for $O(n^{2})$ relations, but
the local method considers $O(\frac{n}{b}b^{2})=O(n)$ where $n$ is the total
number of samples in the full batch, and $b$ is a given constant number of
samples in a mini-batch. As shown in Fig. 2, relations in mini-batches could
not cover all relations in full batches. Thus, the coverage of FSDL rapidly
decreases by the input samples sizes. This decrease implies that the
transferred feature structure is easily insufficient for a large dataset.
Moreover, the large scale makes transferring all structures computationally
inefficient and infeasible in parallel programming with limited hardware.

To overcome this problem, we propose an approach to effectively transfer the
inter-feature structures between samples in different mini-batches, called
global inter-feature structures. The key idea is to store centroids of full
batch into a memory and then transfer the structures between centroids in a
teacher and a student. As locating features of the student to its centroids,
the method transfers the global inter-feature structures built by samples near
the involved centroids. This method has two sequential stages: 1) unsupervised
post-training and 2) memory-augmented transfer method. In the first stage, we
use the fine-tuned teacher to generate representations of all samples from the
penultimate layer in the full batch. After randomly initialized element
vectors in memory, each vector is updated to minimize the Euclidean distance
to its corresponding k-nearest neighbors, thereby performing k-means
clustering. The second stage is the memory-augmented transfer method to reduce
the difference of inter-feature structures over the centroids in the teacher
memory $\displaystyle{\bm{M}}^{T}$ and student memory
$\displaystyle{\bm{M}}^{S}\in\displaystyle\mathbb{R}^{(|C|\times|W||D|)}$,
where the number of elements of the memory $|\displaystyle
C|\in\\{100,~{}300\\}$. The difference $\textbf{F}_{\mathbf{MM}}$ is defined
as following equation:

$\begin{split}\textbf{F}_{\mathbf{MM}}=-\log|\emph{CKA}(\displaystyle{\bm{M}}^{T},\displaystyle{\bm{M}}^{S})|.\end{split}$
(8)

The transferred structures on the centroids are propagated to features of the
student by learning the teacher’s feature-to-centroid difference. For
transferring centroid difference, we define $\textbf{F}_{\mathbf{Mh}}$ as

$\textbf{F}_{\mathbf{Mh}}=\frac{\sum_{i,j}||\psi(\displaystyle{\bm{H}}^{T}_{i},\displaystyle{\bm{M}}^{T}_{j})-\psi(\displaystyle{\bm{H}}^{S}_{i},\displaystyle{\bm{M}}^{S}_{j})||_{2}^{2}}{|B||C|},$
(9)

where $\psi$ is a function for evaluating distance.
$\textbf{F}^{E}_{\mathbf{Mh}}$ and $\textbf{F}^{C}_{\mathbf{Mh}}$ specify
$\psi$ to Euclidean distance and cosine similarity, respectively. The final
distillation loss for transferring the global inter-feature structure is
defined as follows:

$\mathcal{L}_{G}=\gamma_{m}\textbf{F}_{\mathbf{Mh}}^{E}+(1-\gamma_{m})\textbf{F}_{\mathbf{Mh}}^{C}+\textbf{F}_{\mathbf{MM}},$
(10)

where $\gamma_{m}$ is a hyper-parameter for interpolation.

We used Euclidean and cosine similarity as [5, 11] instead of CKA for
$\textbf{F}_{\mathbf{Mh}}$ to allow this method to set a flexible memory size
because CKA can not be directly applied when the batch size of ${\bm{H}}$ is
different to the number of centroids in the memory.

####  III-B6 Integrated Methods (FSDIL & FSDILG)

The intra-feature, local inter-feature, and global inter-feature structures
represent different types of structures. Thus, their integration is a natural
extension for transferring more rich feature structure. The integration is
simply implemented as the sum of all three loss functions as:

$\mathcal{L}_{ILG}=\gamma_{i}\mathcal{L}_{I}+\gamma_{l}\mathcal{L}_{L}+\gamma_{g}\mathcal{L}_{G},$
(11)

where the hyper-parameter $\gamma_{i}$, $\gamma_{l}$, and $\gamma_{g}$ are
interpolation rates; $\gamma_{g}$ is zero with $\mathcal{L}_{IL}$.

##  IV Experiments

We empirically analyze FSD methods applied to BERT for well-known language
understanding tasks. Beyond usual performance and computational efficiency
evaluation to evaluate practical impact, we focus on understanding of how
teacher knowledge is effectively transferred, because its impact to
performance is promising as shown in [21] and the final performance is
interfered by other effects of distillation as generalization [22, 23]. The
analysis has four parts for 1) performance, 2) restoration of teacher’s
prediction, 3) transfer patterns of structures on traditional distance
metrics, and 4) task-specific patterns on CKA similarity. The performance is
for evaluating practical impact of our methods in comparison with state-of-
the-art. The other three parts are to evaluate the effectiveness of
transferring teacher knowledge on feature distributions in various
perspectives.

###  IV-A Datasets

The General Language Understanding Evaluation (GLUE)
111https://gluebenchmark.com/tasks benchmark is presented in Table I

corpus |  $|$train$|$ |  $|$dev$|$ | metrics
---|---|---|---
single-sentence tasks
CoLA | 8.5k | 1k | Matthews Corr
SST-2 | 67k | 872 | Accuracy
similarity and paraphrase tasks
QQP | 364k | 40k | Accuracy/F1
MRPC | 3.7k | 408 | Accuracy/F1
STS-B | 7k | 1.5k | Pearson Corr
Spearman Corr
inference tasks
MNLI | 393k | 20k | Accuracy
RTE | 2.5k | 276 | Accuracy
QNLI | 105k | 5.5k | Accuracy
WNLI | 634 | 71 | Accuracy
TABLE I: Evaluation Metrics and the number of dataset of GLUE benchmark.
—train— and —dev— are the size of training and development dataset, and Corr
is a correlation.

GLUE the General Language Understanding Evaluation [9] consists of nine
English sentence-understanding tasks. Single-sentence tasks include the Corpus
of Linguistic Acceptability (CoLA) [24], Stanford Sentiment Treebank [25]. In
the similarity and paraphrase tasks, Microsoft Research Paraphrase Corpus
(MRPC) [26], Quora Question Pairs (QQP)222https://data.quora.com/First-Quora-
Dataset-Release- Question-Pairs, and Semantic Textual Similarity Benchmark
(STS-B) [27] are included. In the last, inference tasks include the Multi-
Genre Natural Language Inference Corpus (MNLI) [28], Stanford Question
Answering Dataset (QNLI) [29], Recognizing Textual Entailment (RTE) [30, 31,
32], and Winograd Schema Challenge (WNLI) [33].

###  IV-B Distillation Settings

####  IV-B1 Environment Setup

We conduct knowledge distillation with GLUE on a single RTX-2080-Ti and
RTX-8000 GPU with 32 batches, 128 max sequence length, and 768 dimension.
WNLI, MPRC, and SST-2 are implemented on single RTX-2080-Ti, and RTE, STS-B,
CoLA, and QNLI are implemented on a single RTX-8000 GPU. FSDG and FSDILG of
QQP and MNLI are implemented on a single RTX-8000 because of the memory size,
and other methods of QQP and MNLI are operated on a single RTX-2080-Ti GPU.
Each task performance is slightly different depend on GPU device and number of
device.

####  IV-B2 BERT-base Preparation

We set a 12-layer transformer encoder with 768 hidden nodes and 12 attention
heads as a teacher model. We conduct fine-tuning with the uncased version of
pre-trained BERT-base333https://s3.amazonaws.com/models.huggingface.co/
bert/bertbase-uncased-pytorch model.bin on nine GLUE tasks independently. The
maximum sequence length is 128 which is referred in [1]. The number of train
epochs is 3. The training batch size is 32. The learning rate is 2e-5 except
for STS-B and WNLI tasks, which are set in 5e-5 to slightly improve teacher
performance. We note that fine-tuning of BERT-base can be more improved by
adding other methods irrelevant to knowledge distillation. In this paper, our
primary goal is not to solve language understanding tasks by whatever means
possible, but to prove the impact of more accurate transferring of teacher’s
knowledge in a practical environment. Therefore, fair comparative group is the
state-of-the-art transferring methods rather than the state-of-the-art
language understanding model.

####  IV-B3 Baseline Method Settings

We reproduce VKD, PKD, MiniLM, and RKD but MiniLM [2] performs KD on the pre-
training stage. For consistency with VKD, PKD, and FSD, we apply MiniLM method
on the fine-tuning stage. Previous work [1] uses 6-layers of BERT model
(BERT6) as a student and we implement distillation experiments with the same
student architecture. We utilize parameters from 1st to 6th layer of pre-
trained BERTBASE to initialize BERT6. Fine-tuning for VKD, we conduct each
task with $\alpha$ from $\\{0.2,0.5,0.7\\}$, temperature $\tau$ from
$\\{5,10,20\\}$, and learning rate from $\\{$1e-5, 2e-5, 5e-5$\\}$ to search
for the best model. Additionally, we set angle and distance loss hyper-
parameters introduced in To reproduce RKD [5], we set hyper-parameters for its
angle and distance loss from $\\{200.0,2000.0,20000.0\\}$, and
$\\{100.0,1000.0,10000.0\\}$.

###  IV-C FSD Settings

####  IV-C1 Common FSD methods setting

We fix the best cases of $\alpha,\tau$ and epochs on each downstream task as
the result of grid search to reduce the cost of tuning hyper-parameters in
FSD. We conduct additional hyper-parameters $\beta$ set 1 in FSDIL and FSDILG
methods to reduce hyper-parameter space and set $\beta$ from $\\{3,4,5,6\\}$
except STS-B, which is set from $\\{3,4,5,6,10\\}$ and learning rate from
$\\{$3e-5, 4e-5, 5e-5, 6e-5$\\}$. Besides, by applying FSDIL method, we fix
the best case of $\beta$ on methods for the intra-feature and local inter-
feature structure methods and set $\gamma$ from $\\{0.1,0.2,\cdots,0.9\\}$.

####  IV-C2 FSDG settings

Implementing unsupervised post-training for the global inter-feature
structures, we set different memory sizes depending on a given dataset size.
In a large task such as QQP and MNLI, 300 memory entries ($|C|=300$) are used
to store centroids. The other smaller tasks used 100 entries ($|C|=100$). The
number of epochs for clustering is set by 3 for a teacher memory, which shows
sufficient convergence in preliminary tests. After training the teacher
memory, the structures in the memory are transferred to a randomly initialized
student memory by distillation loss. Hyper-parameters for the distillation are
separately set for each downstream task by greedy and grid search in terms of
performance.

First, we set $\gamma_{m}$ and $\beta$ from $\\{$0, 1e-7, 1e-6, 1e-5, 1e-4,
1e-3,1e-2, 0.1, 1.0$\\}$ find the best case, then set again $\gamma_{m}$ from
$\\{$6e-7, 7e-7, 8e-7, 9e-7, 2e-6, 3e-6, 4e-6, 5e-6$\\}$ in the WNLI, RTE, and
MNLI, and set from $\\{$6e-5, 7e-5, 8e-5, 9e-5 2e-4, 3e-4, 4e-4, 5e-4$\\}$ in
the SST-2, and from $\\{$6e-3, 7e-3, 8e-3, 9e-3 2e-2, 3e-2, 4e-2, 5e-2$\\}$ in
the QQP and set from $\\{$0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9$\\}$ in
the STS-B, CoLA, MRPC, and QNLI. Also, we set $\beta$ again from $\\{$6e-8,
7e-8, 8e-8, 9e-8, 2e-7, 3e-7, 4e-7, 5e-7$\\}$ in the STS-B, SST-2, and QQP,
and set $\\{$6e-7, 7e-7, 8e-7, 9e-7, 2e-6, 3e-6, 4e-6, 5e-6$\\}$ in the CoLA,
MRPC, QNLI, and MNLI, and set from $\\{$0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7,
0.8, 0.9$\\}$ in the WNLI, and RTE. The $\alpha,\tau$ and epoch are fixed as
the best case in VKD to reduce hyper-parameter optimization cost.

| method | WNLI | RTE | STS-B | CoLA | MRPC
---|---|---|---|---|---|---
teacher | HF∗ | 56.34 | 67.15 | 93.95/83.70 | 49.23 | 89.47/85.29
BERT-base | 56.34 | 66.79 | 88.78/88.48 | 55.47 | 86.00/81.10
baseline | VKD | 54.65($\pm$1.18) | 65.40($\pm$1.56) | 88.16($\pm$0.17)/87.83($\pm$0.16) | 41.46($\pm$0.89) | 86.24($\pm$0.52)/80.75($\pm$0.65)
PKD | 54.37($\pm$2.36) | 63.90($\pm$0.79) | 88.45($\pm$0.10)/88.08($\pm$0.06) | 41.87($\pm$1.13) | 86.37($\pm$0.60)/80.98($\pm$0.81)
MiniLM${\dagger}$ | 48.45($\pm$6.19) | 61.49($\pm$0.59) | 87.82($\pm$0.08)/87.49($\pm$0.09) | 33.34($\pm$0.83) | 84.57($\pm$1.36)/78.13($\pm$2.02)
RKD | 55.21($\pm$3.21) | 65.22($\pm$0.90) | 88.43($\pm$0.17)/88.12($\pm$0.14) | 43.07($\pm$1.49) | 86.87($\pm$0.47)/81.84($\pm$0.35)
proposed | FSDI | 55.49($\pm$1.26) | 66.13($\pm$1.28) | 88.65($\pm$0.18)/88.31($\pm$0.18) | 42.10($\pm$0.95) | 86.95($\pm$0.25)/81.87($\pm$ 0.31)
FSDL | 54.93($\pm$2.23) | 66.31($\pm$0.29) | 88.66($\pm$0.14)/88.26($\pm$0.12) | 39.52($\pm$1.34) | 87.02($\pm$0.50)/82.01($\pm$0.63)
FSDG | 53.52($\pm$ 3.59) | 66.49($\pm$1.11) | 88.50($\pm$0.12)/88.14($\pm$0.16) | 42.34($\pm$1.91) | 86.02($\pm$0.42)/80.43($\pm$0.46)
FSDIL | 55.49($\pm$1.59) | 66.55($\pm$1.20) | 88.68($\pm$0.10)/88.33($\pm$0.09) | 41.91($\pm$1.42) | 87.03($\pm$0.30)/82.13($\pm$0.37)
FSDILG | 55.49($\pm$1.61) | 66.61($\pm$1.01) | 88.69($\pm$0.10)/88.33($\pm$0.09) | 43.03($\pm$1.43) | 87.10($\pm$0.24)/82.20($\pm$0.33)
TABLE II: Results on development (dev.) datasets in the small tasks of the GLUE benchmark with mean and standard deviation. We evaluate model with six different random seeds. The numbers imply accuracy in WNLI and RTE, Pearson/Spearman correlation in STS-B, Matthew’s correlation in CoLA, and F1/accuracy in MRPC task. A bold number is the best result among distillation methods in each task. ∗ denotes that the results are taken from the huggingface BERTBERT. ${\dagger}$ denotes reproduced model for consistency with VKD, PKD, and FSD methods. | method | SST-2 | QNLI | QQP |  MNLI/MNLI-mm
---|---|---|---|---|---
teacher | HF∗ | 91.97 | 87.46 | 88.40/88.31 | 90.61/81.08
BERT-base | 92.09 | 91.60 | 91.07/88.06 | 84.70/84.65
baseline | VKD | 90.92($\pm$0.62) | 88.46($\pm$0.47) | 91.03($\pm$0.08)/87.96($\pm$0.10) | 82.20($\pm$0.19)/82.85($\pm$0.12)
PKD | 90.77($\pm$0.41) | 88.57($\pm$0.17) | 91.00($\pm$0.11)/87.92($\pm$0.14) | 82.27($\pm$0.10)/82.57($\pm$0.21)
MiniLM${\dagger}$ | 90.23($\pm$0.39) | 89.48($\pm$0.19) | 90.53($\pm$0.02)/87.21($\pm$0.02) | 82.23($\pm$0.09)/82.58($\pm$0.13)
RKD | 91.02($\pm$0.19) | 88.91($\pm$0.31) |  91.21($\pm$0.06)/ 88.13($\pm$0.08) | 82.39($\pm$0.19)/82.92($\pm$0.13)
proposed | FSDI | 90.63($\pm$0.69) | 88.84($\pm$0.27) | 91.16($\pm$0.06)/88.09($\pm$0.08) | 82.34($\pm$0.17)/82.85($\pm$0.24)
FSDL | 90.52($\pm$0.43) | 88.47($\pm$0.35) | 91.19($\pm$0.06)/88.15($\pm$0.10) | 82.35($\pm$0.26)/82.61($\pm$0.17)
FSDG | 90.75($\pm$0.75) | 88.51($\pm$0.35) | 91.01($\pm$0.04)/87.94($\pm$0.07) | 82.05($\pm$0.16)/82.74($\pm$0.23)
FSDIL | 90.88($\pm$0.29) | 88.84($\pm$0.28) | 91.13($\pm$0.06)/88.05($\pm$0.10) |  82.52($\pm$0.13)/82.93($\pm$0.27)
FSDILG | 91.04($\pm$0.28) | 88.97($\pm$0.25) | 91.19($\pm$0.05)/88.14($\pm$0.08) | 82.42($\pm$0.13)/83.00($\pm$0.20)
TABLE III: Results on development (dev.) set in the large tasks of the GLUE
benchmark with mean and standard deviation. We evaluate model with six
different random seeds. The numbers are measured by accuracy in SST-2, QNLI,
and MNLIs, and accuracy/F1 in QQP task. A bold number is defined by the same
manner in Table II.

####  IV-C3 FSD Integration Settings

We fix $\gamma_{m}$ and set $\gamma_{g}$ from $\\{$1e-7, 1e-6, 1e-5, 1e-4,
1e-3,1e-2, 0.1, 1.0$\\}$. Also, we set $\gamma_{i}$ from $\\{$0.1, 1.0,
$\beta$ of FSD${}_{I}\\}$ and set $\gamma_{l}$ from $\\{$0.1, 1.0, $\beta$ of
FSD${}_{L}\\}$.

##  V Results and Discussions

###  V-A Performance

Table II presents the results on the GLUE dev. sets in small-sized tasks whose
samples are less than 10,000. Table III shows the other larger-sized tasks.
The FSD methods show significantly higher performance in the seven WNLI, RTE,
MRPC, SST-2, STS-B, QQP, and MNLI (match and mismatch) tasks than the baseline
methods, but CoLA, and QNLI show lower performance than the baseline.

In addition, the applied FSDILG method represents the highest performance
among proposed methods in the nine GLUE tasks. Compared to the teacher model,
the proposed methods show slightly higher performance than BERTBASE(T), by
$1.0\%$ on the MRPC task.

In the results, the FSD methods effectively and stably improve the test
accuracy of the benchmark. Among the FSD methods, FSDILG generally improves
performance in all GLUE tasks by transferring all types of structure (intra,
local, and global). Furthermore, the methods can generalize the student model
to show better performance than the teacher models in particular tasks.

###  V-B Restoration Rate of Teacher Prediction

(a)

(b)

(c)

(d)

(e)

(f)

(g)

(h)

(i)

Figure 3: Restoration rate of teachers’ predictions in students on the GLUE
tasks. MNLI used the matched subset.

Fig. 3 shows the restoration rates of teacher prediction. As all tasks are
binary classification except STS-B, we plot precision, recall, and F1-score
for each method in each task, over teacher prediction result as the ground-
truth. FSD methods generally show higher restoration rates than baseline, and
FSDILG results are highest in most of the cases. In detailed observations,
FSDG recall is equal and higher than FSDL in the WNLI, RTE, CoLA, and SST-2.

In KD, correctly recovering a teacher’s prediction affects the quality of
transferring. In the restoration rate results analysis, the higher restoration
rate implies student imitates teacher results accurately, therefore the
proposed methods are effective to emulate teachers. Moreover, FSDILG shows
better mimicry of teacher results than the other proposed methods. As shown in
the WNLI, RTE, CoLA, and SST-2 results, the equal and the higher recall in
FSDG represent that FSDG effectively generalizes the students in relatively
small size tasks.

###  V-C Patterns of Transferring Structures

WNLI
---
|  |  |
RTE | STS-B
|  |  |  |  |  |  |
CoLA | MRPC
|  |  |  |  |  |  |
SST-2 | QNLI
|  |  |  |  |  |  |
QQP | MNLI
|  |  |  |  |  |  |
TABLE IV: The results of relation difference. Each task shows the results of
using different four difference metrics: $RD^{E}_{intra}$, second is
$RD^{E}_{inter}$, third is $RD^{C}_{intra}$, and $RD^{C}_{inter}$. Fisrt row,
WNLI, results are representative. The right side number of y-axis is the rank
of $FSD_{ILG}$. Average Rank of Relation Difference Table
---
| method | WNLI | RTE | STS-B | CoLA | MRPC | SST-2 | QNLI | QQP | MNLI | Avg
baseline | VKD | 4.75 | 4.25 | 5.00 | 4.25 | 5.00 | 5.50 | 4.00 | 4.50 | 4.00 | 4.58
PKD | 4.00 | 4.00 | 4.00 | 4.00 | 4.00 | 5.00 | 4.25 | 4.25 | 4.25 | 4.19
RKD | 4.50 | 4.75 | 4.50 | 4.00 | 4.00 | 2.75 | 3.25 | 4.00 | 3.75 | 3.94
proposed | $\textbf{FSD}_{I}$ | 2.75 | 2.75 | 2.75 | 4.00 | 3.25 | 4.75 | 4.00 | 4.00 | 3.50 | 3.53
$\textbf{FSD}_{L}$ | 4.75 | 4.75 | 4.00 | 4.25 | 4.25 | 1.50 | 4.00 | 4.00 | 4.50 | 3.94
$\textbf{FSD}_{G}$ | 5.00 | 4.50 | 5.25 | 5.75 | 4.20 | 5.50 | 4.75 | 3.25 | 4.00 | 4.69
$\textbf{FSD}_{ILG}$ | 2.50 | 3.50 | 2.50 | 1.75 | 3.25 | 3.00 | 4.00 | 4.00 | 4.00 | 3.17
TABLE V: Each value is the average of ranks of four relation difference
results of each task. The lower is the better and the final difference values
after training are used for the ranking. (Avg: the average of the average
ranks over all tasks).

To analyze the patterns of transferred structures, we evaluate relation
difference ($RD$) for inter-feature structure ($RD_{inter}$) as

$RD_{inter}=\sum_{i,j}\frac{|\psi(\displaystyle{\bm{H}}^{T}_{i},\displaystyle{\bm{H}}^{T}_{j})-\psi(\displaystyle{\bm{H}}^{S}_{i},\displaystyle{\bm{H}}^{S}_{j})|}{|B|^{2}},\\\
$ (12)

where $i,j\in\\{1,2,\cdots,|B|\\}$, and for intra-feature structure
($RD_{intra}$) as

$RD_{intra}=\sum_{i,j,k}\frac{|\psi(\displaystyle{\bm{H}}^{T}_{i,j},\displaystyle{\bm{H}}^{T}_{i,k})-\psi(\displaystyle{\bm{H}}^{S}_{i,j},\displaystyle{\bm{H}}^{S}_{i,k})|}{|W|^{2}\cdot|B|},\\\
$ (13)

where $i\in\\{$1,2, $\cdots$, $|B|\\}$ and $j,k\in\\{$1,2, $\cdots$, $|W|\\}$.
This metric implies the average difference of the relation unit building
intra-feature structures and inter-feature structures between the teacher and
students. $RD^{E}$ and $RD^{C}$ are defined by the same manner of
$\textbf{F}^{E}_{\mathbf{Mh}}$ and $\textbf{F}^{C}_{\mathbf{Mh}}$ to specify
$\psi$. Then, we evaluate the rank of the last iteration $RD$ values over each
method on each task. Baseline models and primitive FSD methods are tested for
clear analysis of the impact of structure types.

$RD$ values in training are illustrated in Table IV, and their $RD$ ranks are
shown on the Table V. The lower $RD$, the better rank close to one. In the
WNLI graph, most proposed methods were more effective to reduce $RD^{E}$ than
the baselines, while $RD^{C}$ shows inconsistent superiority. The pattern is
similarly observed in the other GLUE tasks. Differently to the pattern, FSDG
is better to transfer $RD^{C}$ structures than the other FSD methods. In Table
V, the proposed FSD methods show best ranks on most tasks except QNLI. FSDILG
shows the best average rank on the GLUE task, where FSDILG is the first rank
in the WNLI, RTE, STS-B, CoLA, and MRPC and second-best rank in the rest of
GLUE tasks.

The $RD$ results of WNLI task show that the proposed methods preserve
structures on $RD^{E}$ but unstably transfer the structures on $RD^{C}$
because of the CKA property to preserve Euclidean distance and dot product.
Most FSD methods effectively transfer structures except the global case, which
is an expected result because $RD$ metrics are oriented to evaluate local
relations in a same batch. It also implies that the transferred global
structures are not covered by local structures in mini-batches.

The higher ranks of FSD methods in most tasks imply that CKA similarity is
effective to preserve structures on $RD$ metrics. RKD, the best among
baseline, is still significantly worse than FSDILG even if it spends larger
computational cost for evaluating $O(n^{3})$ relations than $O(n^{2})$ of FSD
methods. In sum, FSDILG using CKA similarity effectively transfers teachers’
potential structures defined by various difference metrics with relatively
good computational efficiency.

###  V-D Task-Specific Structural Similarity Between Teacher and Student

We evaluate CKA similarity heat maps of teacher by teacher (T-T), teacher by
no-Distillation-Student (T-noDS), and teacher by student with methods for each
task to investigate structural properties on CKA perspective. Fig. 4 shows CkA
heat maps and Table VI shows the average of similarities on the diagonal lines
and the average over all tasks. CKA similarity evaluation is conducted on
mini-batches. We split a test set to build a mini-batch pool for each task.
Then, teacher and another model separately select their mini-batch and
generates corresponding feature for evaluating CKA similarity. This value is
shown in a pixel and we repeated it for all mini-batch pairs. The size of
mini-batches differs by tasks because of different test set size. We randomly
sampled test inputs for the mini-batch pool preparation in QNLI, QQP, and MNLI
tasks because of their large size of test data. In the other tasks, we used
all test samples for building the pool.

Fig. 4, the lightness of each diagonal line implies the accuracy of
transferring teacher knowledge captured by CKA. Its maximum value is 1.0
obtained in any T-T cases. Overall diagonal lines are lighter in WNLI, RTE,
STS-B, MRPC, QNLI, and QQP than CoLA, SST-2, and MNLI, which shows the task-
specific difference between teacher and student knowledge. More accurate
numerical comparison in Table VI shows that FSDILG show significantly better
average diagonal values over all tasks than other methods except FSDL and T-T
cases.

In the comparison of heat map patterns, the FSDI, FSDL, and FSDILG are similar
to the T-T heat map, called teacher group. In contrast, VKD, PKD, (baselines)
and FSDG are more closer to noDS, called noDS group.

In the results, the different range of heat maps and patterns show the
implicit difference of structures between tasks. Depending on the complexity
of transferring the structures and conflict with students’ structures, the
performance is heavily affected. The two cases, higher CKA similarity in FSDL
and T-T, are natural to have higher values because T-T uses the same structure
and FSDL uses the same evaluation metric for the heat map and training. The
clear distinction of heat map patterns between the teacher and noDS groups
implies that the transferred structures largely differ by their types. The
distinctive patterns in FSDs imply that the knowledge transferred by them is
all different, which explains why the integration shows the best performance.

Figure 4: CKA heat maps of GLUE benchmark. CoLA data set result is
representative on GLUE benchmark, x-axis, y-axis are indices of mini-batches
on the same data set. Each pixel shows the CKA similarity between teacher’s
and other model’s representations generated from their mini-batches. In the
diagonal lines, the same samples are used for the generation. Right side bar
shows the normalized range of CKA similarities observed in each task. CKA heat
map diagonals
---
| method | WNLI | RTE | STS-B | CoLA | MRPC | SST-2 | QNLI | QQP | MNLI | Avg
No KD | T-T | 1.000 | 1.000 | 1.000 | 1.000 | 1.000 | 1.000 | 1.000 | 1.000 | 1.000 | 1.000
T-nDS | 0.998 | 0.985 | 0.954 | 0.803 | 0.981 | 0.794 | 0.972 | 0.909 | 0.947 | 0.927
baseline | T-VKD | 0.997 | 0.984 | 0.955 | 0.811 | 0.979 | 0.910 | 0.965 | 0.889 | 0.919 | 0.934
T-PKD | 0.997 | 0.980 | 0.956 | 0.813 | 0.979 | 0.900 | 0.961 | 0.903 | 0.944 | 0.937
proposed | T-$\textbf{FSD}_{I}$ | 0.999 | 0.991 | 0.962 | 0.769 | 0.985 | 0.774 | 0.972 | 0.934 | 0.941 | 0.925
T-$\textbf{FSD}_{L}$ | 0.999 | 0.995 | 0.985 | 0.847 | 0.990 | 0.952 | 0.983 | 0.969 | 0.971 | 0.966
T-$\textbf{FSD}_{G}$ | 0.998 | 0.974 | 0.936 | 0.798 | 0.975 | 0.900 | 0.935 | 0.897 | 0.735 | 0.905
T-$\textbf{FSD}_{ILG}$ | 0.999 | 0.990 | 0.981 | 0.835 | 0.990 | 0.942 | 0.985 | 0.958 | 0.942 | 0.958
TABLE VI: Each value is the average of diagonal values for each task and each
teacher-student pair.

###  V-E Additional Discussion

In this subsection, we discuss about additional observation from the above
analysis results.

####  V-E1 Impact of CKA to model performance

As shown in table V, the average of FSDILG $RD$ rank is the highest, but $RD$
structure graph pattern shown in Table IV is not consistent in all GLUE task.
On the other hand, CKA heat map diagonals in Table VI is consistent regardless
of tasks. As referred on the Results section, CKA heat map could be a clue to
explain the best case of method (FSDILG). Therefore, CKA analysis is more
related metric than Euclidean distance and cosine similarity to explain KD
model performance.

####  V-E2 Stability of Global Structure for Transferring

method | MRPC | SST-2
---|---|---
FSDL | 0.66/0.44 | 0.68
FSDG | 0.47/0.38 | 0.62
TABLE VII: Each value represents the standard deviation of model performance
of 18 runs composed 6 runs for three mini-batch sizes in $\\{$8, 16, 32$\\}$.

We estimate of standard deviation in different mini-batch cases to show that
utilized memory method (FSDG) is less affected by mini-batch size. We select
MRPC and SST-2 tasks because each task is in small and large dataset of the
GLUE and both tasks are stably improved model performance. As shown in Table
VII, two tasks standard deviation is the lowest in FSDG. These results imply
that regardless of mini-batch size, FSDG method preserves teacher feature
structure information in memory and stably maintains model performance.

##  VI Conclusion

In this paper, we addressed transferring rich information of feature
representations via knowledge distillation in BERT. To represent the features,
we proposed three levels of feature structures defined by CKA similarity:
intra-feature, local inter-feature, and global inter-feature structures. To
transfer them, we implemented feature structure distillation methods
separately for the structures, especially for the global structures using
memory-augmented transferring with clustering. We could find that transferring
all the structures induces more similar student representations to its teacher
and consistently improves performance in the GLUE language understanding
tasks. This work can be extended to more downstream applications using pre-
trained BERT for more fine transferring.

## Acknowledgment

This work was supported by the National Research Foundation of Korea (NRF)
grant funded by the Korea government (MSIT) (2022R1A2C2012054).

## References

  * [1] S. Sun, Y. Cheng, Z. Gan, and J. Liu, “Patient knowledge distillation for BERT model compression,” in _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , Nov. 2019, pp. 4323–4332.
  * [2] W. Wang, F. Wei, L. Dong, H. Bao, N. Yang, and M. Zhou, “Minilm: Deep self-attention distillation for task-agnostic compression of pre-trained transformers,” in _Advances in Neural Information Processing Systems_ , vol. 33. Curran Associates, Inc., 2020, pp. 5776–5788.
  * [3] X. Jiao, Y. Yin, L. Shang, X. Jiang, X. Chen, L. Li, F. Wang, and Q. Liu, “TinyBERT: Distilling BERT for natural language understanding,” in _Findings of the Association for Computational Linguistics: EMNLP 2020_ , Nov. 2020, pp. 4163–4174.
  * [4] B. Peng, X. Jin, D. Li, S. Zhou, Y. Wu, J. Liu, Z. Zhang, and Y. Liu, “Correlation congruence for knowledge distillation,” in _2019 IEEE/CVF International Conference on Computer Vision (ICCV)_ , 2019, pp. 5006–5015.
  * [5] W. Park, D. Kim, Y. Lu, and M. Cho, “Relational knowledge distillation,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , June 2019.
  * [6] X. Li, J. Wu, H. Fang, Y. Liao, F. Wang, and C. Qian, “Local correlation consistency for knowledge distillation,” in _Computer Vision – ECCV 2020_ , A. Vedaldi, H. Bischof, T. Brox, and J.-M. Frahm, Eds. Cham: Springer International Publishing, 2020, pp. 18–33.
  * [7] C. Cortes, M. Mohri, and A. Rostamizadeh, “Algorithms for learning kernels based on centered alignment,” _J. Mach. Learn. Res._ , vol. 13, no. 1, p. 795–828, mar 2012.
  * [8] S. Kornblith, M. Norouzi, H. Lee, and G. Hinton, “Similarity of neural network representations revisited,” in _Proceedings of the 36th International Conference on Machine Learning_ , ser. Proceedings of Machine Learning Research, K. Chaudhuri and R. Salakhutdinov, Eds., vol. 97. PMLR, 09–15 Jun 2019, pp. 3519–3529.
  * [9] A. Wang, A. Singh, J. Michael, F. Hill, O. Levy, and S. R. Bowman, “GLUE: A multi-task benchmark and analysis platform for natural language understanding,” in _International Conference on Learning Representations_ , 2019. [Online]. Available: https://openreview.net/forum?id=rJ4km2R5t7
  * [10] Z. Sun, H. Yu, X. Song, R. Liu, Y. Yang, and D. Zhou, “MobileBERT: a compact task-agnostic BERT for resource-limited devices,” in _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , Jul. 2020, pp. 2158–2170.
  * [11] G. Park, G. Kim, and E. Yang, “Distilling linguistic context for language model compression,” in _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , Nov. 2021, pp. 364–378.
  * [12] H. Hotelling, _Relations Between Two Sets of Variates_. New York, NY: Springer New York, 1992, pp. 162–190.
  * [13] G. H. Golub and H. Zha, “The canonical correlations of matrix pairs and their numerical computation,” in _Linear Algebra for Signal Processing_. New York, NY: Springer New York, 1995, pp. 27–49.
  * [14] M. Raghu, J. Gilmer, J. Yosinski, and J. Sohl-Dickstein, “Svcca: Singular vector canonical correlation analysis for deep learning dynamics and interpretability,” in _Advances in Neural Information Processing Systems_ , vol. 30. Curran Associates, Inc., 2017, pp. 6076–6085.
  * [15] A. Morcos, M. Raghu, and S. Bengio, “Insights on representational similarity in neural networks with canonical correlation,” in _Advances in Neural Information Processing Systems_ , vol. 31. Curran Associates, Inc., 2018, pp. 5727–5736.
  * [16] G. Hinton, O. Vinyals, and J. Dean, “Distilling the knowledge in a neural network,” 2015, cite arxiv:1503.02531Comment: NIPS 2014 Deep Learning Workshop. [Online]. Available: http://arxiv.org/abs/1503.02531
  * [17] V. Sanh, L. Debut, J. Chaumond, and T. Wolf, “Distilbert, a distilled version of BERT: smaller, faster, cheaper and lighter,” _CoRR_ , vol. abs/1910.01108, 2019. [Online]. Available: http://arxiv.org/abs/1910.01108
  * [18] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. u. Kaiser, and I. Polosukhin, “Attention is all you need,” in _Advances in Neural Information Processing Systems_ , vol. 30. Curran Associates, Inc., 2017, pp. 5998–6008.
  * [19] J. Wu, S. Yu, W. Chen, K. Ma, R. Fu, H. Liu, X. Di, and Y. Zheng, “Leveraging undiagnosed data for glaucoma classification with teacher-student learning,” in _Medical Image Computing and Computer Assisted Intervention – MICCAI 2020_. Cham: Springer International Publishing, 2020, pp. 731–740.
  * [20] A. Gretton, O. Bousquet, A. Smola, and B. Schölkopf, “Measuring statistical dependence with hilbert-schmidt norms,” in _Algorithmic Learning Theory_. Berlin, Heidelberg: Springer Berlin Heidelberg, 2005, pp. 63–77.
  * [21] C. Xu, W. Zhou, T. Ge, K. Xu, J. McAuley, and F. Wei, “Beyond preserved accuracy: Evaluating loyalty and robustness of BERT compression,” in _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_. Online and Punta Cana, Dominican Republic: Association for Computational Linguistics, Nov. 2021, pp. 10 653–10 659.
  * [22] L. Yuan, F. E. H. Tay, G. Li, T. Wang, and J. Feng, “Revisiting knowledge distillation via label smoothing regularization,” _2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pp. 3902–3910, 2020.
  * [23] H. Jung, K. Kim, H. Kim, and J.-H. Shin, “Learning from matured dumb teacher for fine generalization,” 2021. [Online]. Available: https://arxiv.org/abs/2108.05776
  * [24] A. Warstadt, A. Singh, and S. R. Bowman, “Neural network acceptability judgments,” _Transactions of the Association for Computational Linguistics_ , vol. 7, pp. 625–641, 2019.
  * [25] R. Socher, A. Perelygin, J. Wu, J. Chuang, C. D. Manning, A. Ng, and C. Potts, “Recursive deep models for semantic compositionality over a sentiment treebank,” in _Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing_. Seattle, Washington, USA: Association for Computational Linguistics, Oct. 2013, pp. 1631–1642.
  * [26] W. B. Dolan and C. Brockett, “Automatically constructing a corpus of sentential paraphrases,” in _Proceedings of the Third International Workshop on Paraphrasing (IWP2005)_ , 2005. [Online]. Available: https://aclanthology.org/I05-5002
  * [27] D. Cer, M. Diab, E. Agirre, I. Lopez-Gazpio, and L. Specia, “SemEval-2017 task 1: Semantic textual similarity multilingual and crosslingual focused evaluation,” in _Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017)_. Vancouver, Canada: Association for Computational Linguistics, Aug. 2017, pp. 1–14.
  * [28] A. Williams, N. Nangia, and S. Bowman, “A broad-coverage challenge corpus for sentence understanding through inference,” in _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)_. New Orleans, Louisiana: Association for Computational Linguistics, Jun. 2018, pp. 1112–1122.
  * [29] P. Rajpurkar, J. Zhang, K. Lopyrev, and P. Liang, “SQuAD: 100,000+ questions for machine comprehension of text,” in _Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing_. Austin, Texas: Association for Computational Linguistics, Nov. 2016, pp. 2383–2392.
  * [30] I. Dagan, O. Glickman, and B. Magnini, “The pascal recognising textual entailment challenge,” in _Machine Learning Challenges. Evaluating Predictive Uncertainty, Visual Object Classification, and Recognising Tectual Entailment_. Berlin, Heidelberg: Springer Berlin Heidelberg, 2006, pp. 177–190.
  * [31] D. Giampiccolo, B. Magnini, I. Dagan, and B. Dolan, _The Third PASCAL Recognizing Textual Entailment Challenge_. USA: Association for Computational Linguistics, 2007, p. 1–9.
  * [32] L. Bentivogli, B. Magnini, I. Dagan, H. T. Dang, and D. Giampiccolo, “The fifth PASCAL recognizing textual entailment challenge,” in _Proceedings of the Second Text Analysis Conference, TAC 2009, Gaithersburg, Maryland, USA, November 16-17, 2009_. NIST, 2009. [Online]. Available: https://tac.nist.gov/publications/2009/additional.papers/RTE5_overview.proceedings.pdf
  * [33] H. J. Levesque, E. Davis, and L. Morgenstern, “The winograd schema challenge,” in _Proceedings of the Thirteenth International Conference on Principles of Knowledge Representation and Reasoning_ , ser. KR’12. AAAI Press, 2012, p. 552–561.

|  Hee-Jun Jung received the B.S. from the Kyung Hee University (KHU), South
Korea, in 2020. He is currently in Ph.D. degree with the Gwangju Institute of
Science and Technology (GIST) in South Korea. His research interests include
natural language processing, knowledge distillation, variational auto-encoder,
and disentanglement learning.
---|---
|  Doyeon Kim received the B.S. from the Han Yang University (HYU), South
Korea, in 2020. She is currently a M.S. degree student with the Gwangju
Institute of Science and Technology (GIST) in South Korea. Her research
interests include Vector Quantizated Transformer, and natural language
processing.
---|---
|  Seung-Hoon Na received Ph.D. degrees in Computer Science and Engineering
from POSTECH in 2008. He is currently an associate professor in Dept. of
Computer Science at Chonbuk National University, South Korea. His research
interests include natural language processing, information retrieval, and
machine learning.
---|---
|  Kangil Kim received Ph.D. degree from Seoul National University, Seoul,
Korea, in 2012. He worked as a senior researcher in ETRI, Daejon from 2013 to
2016, as an assistant professor at Konkuk university from 2016 to 2019. He is
currently an assistant professor in Gwangju Institute of Science and
Technology. His research interests include artificial intelligence,
evolutionary computation, machine learning, and natural language processing.
---|---