# Fixing the Teacher-Student Knowledge Discrepancy in Distillation

Jiangfan Han1∗, Mengya Gao2, Yujie Wang2, Quanquan Li2, Hongsheng Li1,
Xiaogang Wang1
1CUHK-SenseTime Joint Laboratory, The Chinese University of Hong Kong
2SenseTime Research
<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>
{gaomengya, wangyujie<EMAIL_ADDRESS>Equal contribution.

###### Abstract

Training a small student network with the guidance of a larger teacher network
is an effective way to promote the performance of the student. Despite the
different types, the guided knowledge used to distill is always kept unchanged
for different teacher and student pairs in previous knowledge distillation
methods. However, we find that teacher and student models with different
networks or trained from different initialization could have distinct feature
representations among different channels. (_e.g_. the high activated channel
for different categories). We name this incongruous representation of channels
as teacher-student knowledge discrepancy in the distillation process. Ignoring
the knowledge discrepancy problem of teacher and student models will make the
learning of student from teacher more difficult. To solve this problem, in
this paper, we propose a novel student-dependent distillation method,
knowledge consistent distillation, which makes teacher’s knowledge more
consistent with the student and provides the best suitable knowledge to
different student networks for distillation. Extensive experiments on
different datasets (CIFAR100, ImageNet, COCO) and tasks (image classification,
object detection) reveal the widely existing knowledge discrepancy problem
between teachers and students and demonstrate the effectiveness of our
proposed method. Our method is very flexible that can be easily combined with
other state-of-the-art approaches.

##  1 Introduction

Recent years have witnessed the great success of applying convolutional neural
networks (CNN) in various tasks and the remarkable performance can be owing to
the deeper and wider design of network structures [22, 7, 11]. Whereas, it is
hard to deploy such heavy networks in practice where computational resource
and memory space is limited. Knowledge Distillation [10] has been proposed to
decrease the model size by training a small and shallow network (student)
under the supervision of a larger and deeper network (teacher). Benefit by
knowledge distillation, knowledge in a heavy teacher network can be compressed
into a lightweight student network and the performance of the student can be
improved.

Figure 1:  Visualization of average feature map activations of boy category on
CIFAR-100. S1 and S2 are the students with the same network structure ResNet20
but with different initialization. T is the teacher network ResNet56.
Transformed T represents the teacher’s transformed feature maps by our method.
The feature maps are extracted at the last residual block of the network. Each
row of a subfigure illustrates an expansion of an $8\times 8$ channel and
there are 64 channels in total. The figures in the second row show the
difference between the pair of average activations. Through our proposed
method, the differences in activations are reduced, as shown in Transformed T
- S1. Best viewed in color.

Recently many works have been developed to transfer the knowledge from the
teacher to the student in different ways [24, 23, 19]. Despite the different
types of knowledge or the different ways to transfer knowledge, almost all of
these works adopt the same type of knowledge between different teacher and
student models.

However, the knowledge or the learned representations for the teacher and
student can be distinct, as a result of different network architectures.
Recent works [4, 6] also show that initialization has a strong relationship
with the final representation of the trained network. Therefore, with the
different random initialization, even models with the same network structure
could have distinct feature representations. Although most of the networks can
achieve consistent performances, when analyzing the activation of intermediate
layers of teacher and student models to the same sample, we find the
activation responses of the same channel between different networks could have
large variations, which we name such phenomenon as teacher-student knowledge
discrepancy. Visualization of knowledge discrepancy is shown in Figure 1. We
gather the training data of one category and calculate the activation maps for
students and teachers respectively. It is obvious that the activation map
between T and S has a large difference. What’s more, even with the same model,
_e.g_. S1 and S2, different initialization also makes them show different
activation responses. T-S1, Transformed T-S1, S1-S2 shows the difference
between activation maps.

Existing knowledge distillation methods ignore the discrepancy and directly
train the student with the supervision of the teacher network. We claim that
such discrepancy will impede the learning of the student. To describe the
discrepancy quantitatively, we propose consistency measurements between
teacher and student models. Based on the consistency measurements, a channel-
based feature transformation is explored and deployed on the teacher feature.
The transformation can alleviate the discrepancy problem between the teacher
and student dramatically and provide a suitable feature that is easier for the
student to learn and obtain better performance. As shown in Figure 1, the
feature of teacher model after transformation is more consistent with the
student model. Besides, our approach is independent of other knowledge
distillation methods. We can integrate our method into other methods and
achieve further performance improvement.

The contributions of this work can be summarized as threefold.

  * •

We demonstrate the existence of knowledge discrepancy among different teacher
and student models in the knowledge distillation process.

  * •

We propose a new method to solve the discrepancy problem and analyze the
effectiveness of the proposed feature transformation operation.

  * •

Extensive experiments on different benchmarks and visual tasks demonstrate the
advantage and generalization ability of our method. Moreover, it can be easily
integrated into most of the existing state-of-the-art knowledge distillation
methods to further improve the performance.

##  2 Related Work

Knowledge distillation is firstly introduced in [2] and then brought back to
popularity by [10]. The rationale behind is to use a student model ($S$) to
learn from a teacher model ($T$) without sacrificing much accuracy compared to
the teacher. Challenges lie in two aspects, which are (1) how to extract the
knowledge from $T$, and (2) how to transfer the knowledge to $S$.

Existing methods have designed various types of knowledge to improve their
performance. Methods in [1] treated the hard label predicted by $T$ as the
underlying knowledge, with the assumption that the well-trained $T$ has
already eliminated some label errors contained in the groundtruth data. Hinton
_et al_. [10] argued that the soft label produced by $T$, _i.e_. the
classification probabilities, can provide richer information. Some work [21,
26, 25] extracted the knowledge from $T$ by processing the hidden feature map.
AT [26] averaged the feature map across channel dimension to obtain spatial
attention map, while work in [25] defined inter-layer flow by computing the
inner product of two feature maps, and Lee  _et al_. [14] improved this idea
with singular value decomposition (SVD). Jin  _et al_. [12] claimed that the
representation of a converged heavy model is not easy to learn. As a result,
an approach called route constrained optimization (RCO) was proposed, from the
perspective of curriculum learning.

Furthermore, some methods focus on instance feature space transformation other
than teaching the feature distribution of teacher to student. To address this
issue, Yim  _et al_. [25] proposed the Flow of Solution Procedure (FSP) which
transfers the learning procedure of the teacher to the student instead of
mimicking the feature of intermediate layers. Liu  _et al_. [18] combined
feature space transformation and instance relationship which uses input images
from a training batch to calculate an instance graph and makes the student
learn this graph from the teacher.

Different from all the methods above, our method focuses on how to make the
knowledge distillation easier via transforming their feature maps given
different pairs of teacher and student. The proposed method can be combined
with most of the knowledge distillation methods and improve their performance
further.

##  3 Method

###  3.1 Preliminary

Knowledge distillation is a widely used training strategy which uses a well-
trained teacher network giving extra guidance to the student network. Let
$\mathcal{H(\theta)}$ be the network with parameter $\theta$, then
$\mathcal{H}(\theta^{T})$,$\mathcal{H}(\theta^{S})$ is the teacher and student
network with corresponding parameters $\theta^{T}$ and $\theta^{S}$. Note that
the architecture of the teacher and student can be different. For input
samples $\mathbf{X}$ with groundtruth label $\mathbf{Y}$, we denote the
activation feature map of a certain layer $k$ as
$F_{k}=\mathcal{H}_{k}(\theta;\mathbf{X})\in\mathbb{R}^{b\times c\times
h\times w}$, where $b$ is the batch size of the input samples, $c$ is the
number of channels, $h$ and $w$ are the height and width of the feature map.
Then the feature map of teacher and student network should be
$F_{k}^{T}=\mathcal{H}_{k}(\theta^{T};\mathbf{X})\in\mathbb{R}^{b\times
c_{T}\times h_{T}\times w_{T}}$ and
$F_{k}^{S}=\mathcal{H}_{k}(\theta^{S};\mathbf{X})\in\mathbb{R}^{b\times
c_{S}\times h_{S}\times w_{S}}$ respectively. Generally speaking, the size of
the feature maps between different models are not the same. But in the case of
distillation, we usually use feature map with the same size, _i.e_. we have
$c_{T}=c_{S},h_{T}=h_{S},w_{T}=w_{S}$ and do not need to distinguish them.

For the traditional knowledge distillation methods, the training methods can
be written as follows:

$\displaystyle\mathcal{L}=\mathcal{L}_{cls}(\mathcal{H}(\theta^{S};\mathbf{X}),\mathbf{Y})+\mathcal{L}_{dis}$
(1)
$\displaystyle\mathcal{L}_{dis}=\sum_{(k,k^{\prime})\in\Omega}{\alpha_{(k,k^{\prime})}\mathcal{L}_{d}(F_{k}^{T},F_{k^{\prime}}^{S})}$
(2)

Where $\mathcal{L}_{cls}$ is the cross-entropy loss between the prediction of
student network $\mathcal{H}(\theta^{S};\mathbf{X})$ and groundtruth
$\mathbf{Y}$. $\mathcal{L}_{dis}$ is the distillation loss measuring the
difference of knowledge obtained by the teacher and learned by the student,
and such distillation operation can be executed in different layers. Each loss
term $\mathcal{L}_{d}$ can be Euclidean distance, Kullback–Leibler divergence,
or other measuring metrics. $\Omega$ is the set of all teacher/student feature
pairs used for knowledge distillation. $\alpha_{(k,k^{\prime})}$ is the weight
of the loss between the feature pair $F_{k}^{T}$ and $F_{k^{\prime}}^{S}$.

###  3.2 Teacher-Student Knowledge Discrepancy

As we stated above, teacher-student knowledge discrepancy is the phenomenon
that the activation responses of the same channel between the teacher and the
student could have large variations. To describe such a phenomenon
quantitatively, we propose consistency measurement as following:

For the feature of teacher $F_{k}^{T}$ and student $F_{k^{\prime}}^{S}$, a
global average pooling along the height and weight dimension $\mathcal{A}$ is
executed for those feature maps. Then for teacher
$\mathcal{A}(F^{T}_{k})\in\mathbb{R}^{b\times c}$ and for student
$\mathcal{A}(F^{S}_{k^{\prime}})\in\mathbb{R}^{b\times c}$. We will discuss
the consistency measurement based on these features.

$L_{p}$ consistency. A natural idea of defining the consistency is using the
$L_{p}$ norm. In this work, we define $L_{p}$ consistency, which is measured
by the inverse of $L_{p}$ norm. The consistency matrix
$M(F^{T}_{k},F^{S}_{k^{\prime}})\in\mathbb{R}^{c\times c}$ is given by

$\displaystyle
M_{ij}=\frac{1}{||\mathcal{A}(F^{T}_{k})[:,i]-\mathcal{A}(F^{S}_{k^{\prime}})[:,j]||_{p}}$
(3)

where $\mathcal{A}(F^{T}_{k})[:,i]\in\mathbb{R}^{b}$ represent the $i$th
channel of the activation feature.

The consistency matrix $M(F^{T}_{k},F^{S}_{k^{\prime}})$ measures the
consistent score for each channel pair. If $M_{ij}$ gets large value, which
means the $ith$ channel of teacher and $jth$ channel of student get similar
activation to the same sample, _i.e_. they are consistent. In the real case,
we consider $p=1,2$.

Correlation consistency. Pearson correlation coefficient, simplified as
“correlation” can also be used as the consistency measurement. The consistency
matrix $M$ under the correlation measurement is calculated by

$\displaystyle
M_{ij}=\frac{Cov(\mathcal{A}(F^{T}_{k})[:,i],\mathcal{A}(F^{S}_{k^{\prime}})[:,j])}{\sigma_{\mathcal{A}(F^{T}_{k})[:,i]}\sigma_{\mathcal{A}(F^{S}_{k^{\prime}})[:,j]}}$
(4)

Based on the definition of these two measurements, $L_{p}$ measurement will
obtain a high consistency score only if two channels activate almost the same,
while correlation measurement provides a looser constraint that as long as two
channels have a consistent activation trend, the consistency score will be
high.

There are also many other ways to define the measurement such as cosine
distance and KL divergence. Their definition is quite straightforward and we
will not state them in detail.

For all of the consistency measurements, larger $M_{ij}$ means the $ith$
channel of teacher and $jth$ channel are more consistent. For most of the
traditional distillation methods, they distill the knowledge straightly
following the default channel order from the teacher to the student. We define
the consistency score $\Gamma$ as:

$\displaystyle\Gamma=Tr(M(F_{k}^{T},F^{S}_{k^{\prime}}))$ (5)

Here $Tr$ is the trace of matrix $M$. The consistency score $\Gamma$ can
describe the consistency of corresponding channels of the feature maps
entirely. A larger $\Gamma$ value means they are more consistent, _i.e_. less
discrepant.

In the general case, we calculate the consistency between the trained teacher
and student. Since the structure and the initialization of the teacher and
student are different, their discrepancy is obvious and the consistency score
$\Gamma$ is quite small in such cases. We believe that such an obvious
discrepancy is harmful for the student to learn knowledge from the teacher. So
we propose channel-based transformation for teacher features. The transformed
feature will be more consistent with the student feature and promote the
student learning procedure.

###  3.3 Channel-based Feature Transformation

In this part, we will introduce the method to find the transformation
$\mathcal{T}_{\theta^{S}_{0}}$. The subscript $\theta^{S}_{0}$ means the
transformation is related to the structure and the initialization of the
student. The transformation will be deployed on the teacher feature to reduce
its discrepancy with student model. Since the transformation is defined in the
channel dimension, we name this method a channel-based transformation. To
evaluate the consistency of the transformed teacher feature
$\mathcal{T}_{\theta^{S}_{0},(k,k^{\prime})}(F_{k}^{T})$ and the student
feature $F^{S}_{k^{\prime}}$. The consistency score $\Gamma$ is been modified
as:

$\displaystyle\Gamma=Tr(M(\mathcal{T}_{\theta^{S}_{0},(k,k^{\prime})}(F_{k}^{T}),F^{S}_{k^{\prime}}))$
(6)

The traditional method can be viewed as a special case when
$\mathcal{T}_{\theta^{S}_{0}}$ is an identity transformation. Then we aim to
find $\mathcal{T}_{\theta^{S}_{0}}$ that maximizes $\Gamma$ under some
constraints. We will introduce the methods in the following part.

Greedy. First, we propose a straightforward solution: For each student
channel, a teacher channel that has the largest consistency score to it will
be reordered to match it. Assume for the $i$th channel of student, the $j$th
channel of teacher obtains largest consistency score between them, _i.e_.
$j=\arg\max M[:,i]$, then the transformation is defined as

$\displaystyle\mathcal{T}_{\theta^{S}_{0},(k,k^{\prime})}(F_{k}^{T})[:,i]=F_{k}^{T}[:,j]$
(7)

The greedy transformation is a 1 to N matching scenario. It can get the
maximum $\Gamma$ score when each teacher channel can supervise many student
channels. We call such transformation strategy as “greedy matching”,
simplified as “greedy”.

Bipartite. Although greedy transformation can get the maximum $\Gamma$ score.
However, a problem exists in such a greedy method. Much information contained
in different channels of the teacher will be discarded in such a 1 to N
matching strategy. To avoid such information loss, we propose a 1 to 1
matching to ensure we can maintain all information from the teacher.

For the 1 to 1 matching, we want to find a transformation that contains not
repeating teacher channels while maximizing $\Gamma$. The problem can be
formulated as an optimal $\mathcal{T}^{*}$, where

$\displaystyle\mathcal{T}^{*}=\ $ $\displaystyle\arg\max_{\mathcal{T}}\
Tr(M(\mathcal{T}_{\theta^{S}_{0},(k,k^{\prime})}(F_{k}^{T}),F^{S}_{k^{\prime}}))$
$\displaystyle s.t.\ $
$\displaystyle\mathcal{T}_{\theta^{S}_{0},k}(F_{k}^{T})[:,i]=F_{k}^{T}[:,j],$
(8) $\displaystyle j_{m}\neq j_{n}\ \text{for}\ \forall\ m\neq n$

If we regard the channels in teacher and student activation maps as elements
in different sets and the consistency score as the weights between different
elements, the solution to Equation 8 is actually the maximum matching of a
weighted bipartite graph. We call such transformation as “bipartite matching”,
simplified as “bipartite”. We can use a well-developed algorithm like
Kuhn–Munkres algorithm to solve the problem. Compared with the greedy
strategy, bipartite maintains all information in the teacher to supervise the
learning of the student.

Learning-based. Theoretically, the transformation finds by greedy and
bipartite are special cases for a general linear transformation matrix, which
can be represented by a Fully-Connected layer. The FC layer can also learn N
to 1 matching. What’s more, a complex transformation may be learned by multi-
layer networks with non-linearity. Thus, we design two kinds of sub-networks
to learn the transformation. Sub-net-FC only contains a single Fully-Connected
layer trying to learn the linear transformation of the original teacher’s
feature. Sub-net-Res includes one residual block, which is used to learn non-
linear transformation.

In such a scenario, Teacher’s features are first fed into the sub-network, and
output features $\mathcal{T}_{\theta^{S}_{0}}(F_{k}^{T})$ will be used to
calculate the consistency score with corresponding students features. The
consistency score will act as the loss function to optimize the parameters of
sub-nets. We call such strategy “learning-based”

###  3.4 Knowledge Consistent Distillation

Once getting the transformation $\mathcal{T}_{\theta^{S}_{0}}$, we can get a
transformed teacher which is more consistent with the student.

Then we re-initialize the student network by the same initial value
$\theta^{S}_{0}$ and re-train the network from scratch with distillation. In
the current stage, the distillation loss in Equation 1 will be replaced by:

$\displaystyle\mathcal{L}_{condis}=\sum_{(k,k^{\prime})\in\Omega}{\alpha_{(k,k^{\prime})}\mathcal{L}_{d}(\mathcal{T}_{\theta^{S}_{0},(k,k^{\prime})}(F_{k}^{T}),F_{k^{\prime}}^{S})}$
(9)

Where $(k,k^{\prime})$ in $\mathcal{T}_{\theta^{S}_{0},(k,k^{\prime})}$
indicates the transformation for certain $(k,k^{\prime})$ pair. Then the
overall loss function can be written as:

$\displaystyle\mathcal{L}=\mathcal{L}_{cls}(\mathcal{H}(\theta^{S};\mathbf{X}),\mathbf{Y})+\mathcal{L}_{condis}$
(10)

where $\mathcal{L}_{condis}$ is shown in Equation 9. The whole training
procedure is shown in Algorithm 1

Algorithm 1 Knowledge Consistent Distillation

1: Given dataset $(\mathbf{X},\mathbf{Y})$, pretrained teacher
$\mathcal{H}(\theta^{T})$, student $\mathcal{H}(\theta^{S})$, and the initial
value of student $\theta^{S}_{0}$.

2: Initialize student $\theta^{S}\leftarrow\theta^{S}_{0}$

3: Train the student on $(\mathbf{X},\mathbf{Y})$ using $\mathcal{L}_{cls}$
and get trained student $\mathcal{H}(\theta^{S}_{1})$.

4: Calculate consistency matrix $M$ between $\mathcal{H}(\theta^{S}_{1})$ and
$\mathcal{H}(\theta^{T})$ on the dataset $(\mathbf{X},\mathbf{Y})$

5: Get transformation $\mathcal{T}_{\theta^{S}_{0}}$ by one of the method
proposed in Sec.3.3

6: Reinitialize student by the same initial value
$\theta^{S}\leftarrow\theta^{S}_{0}$

7: Train the student by Equation 10

##  4 Experiments

###  4.1 Implementation Details

CIFAR-100 [13] is commonly used to validate the performance of knowledge
distillation methods, which composed of 50K training images and 10K testing
images from 100 classes. Networks of ResNet [7] family are used on this
dataset. We set the weight decay to 5e-4 and use SGD with momentum 0.9. The
models are trained with 2 GPUs with batch size 64 on each of them. We set the
initial learning rate to be 0.1 and divide the learning rate by 10 at 32K, 48K
iterations, terminating at 64K iterations.

ImageNet [3] is a large-scale dataset which contains over 1M training images
and 50K test images collecting from 1000 categories. We adopt ResNet and
MobileNet V2 to validate the effectiveness. For ResNet, we set the initial
learning rate to 0.2 and drop it by 0.1 at 20K, 40K, 60K iterations and
terminate at 65K iterations. Weight decay is set to 1e-4 and batch size is
2048 with 32 GPUs. As for MobileNetV2, the learning rate starts at 0.1 and
drops by 0.1 at 45K, 75K, and 100K iterations with 130K iterations in total.

COCO [17] is a very challenging object detection benchmark that has 80 object
categories. The union of 80K train images and a 35K subset of validation
images are used as the training set and 5K subset of validation images
(minival) is used as the evaluation set following [15, 16]. We use standard
Average Precision(AP) for evaluation and adopt FPN [15] as the detection
method. The $2\times$ setting released by
Detectron111https://github.com/facebookresearch/Detectron is used.

We compare our proposed method with a number of state-of-the-art knowledge
distillation methods including KD [10], FitNets [21], AT [26], AB [9], RKD
[20] and RCO [12]. For KD, we set $T=4$ and $\lambda=16,9$ for CIFAR-100 and
ImageNet, following [10]. For FitNets, AT, AB, RKD, and RCO, the loss weight
is set to $10^{2},10^{3},10,10,10^{2}$, following [26][12][9].

To further demonstrate the generality of our approach, we also apply it
together with FitNets, RKD, AB, and RCO. For FitNets, AB and RCO, they are
operated on the original feature maps so we just have to apply our
transformation before calculating their distillation losses. For RKD, its
distillation loss is calculated among instance distances into a batch, we add
another loss calculated with $L_{2}$ distances on transformed feature maps to
combine it with our approach.

On COCO, the loss weight we used for $L_{2}$ loss is set to be $10^{-4}$ to
balance with task loss. And following [8], we apply distillation methods
before ReLU at the end of the last layer block.

###  4.2 Ablation Studies

Effectiveness of Different Transformation Strategies. In Section 3.3, we
propose bipartite, greedy, and learning-based strategy for transforming
features from teacher to student. A randomly generated transformation is also
applied to show how much these transformation strategies contribute to the
final performance. Table 1 shows the comparison results of different
strategies. Bipartite achieves the best performance, significantly surpasses
the baseline and random transformation.

Sub-net-FC and Sub-net-Res are learning-based strategies described in Section
3.3. Intuitively, learning a transformation will achieve better results than
the bipartite or greedy solution. However, results show that the learned
combinations of teacher’s features as guidance to the student will bring a
large performance drop even worse than a random transformation result. This is
because the classification ability of the learned features is not guaranteed.
Even classification supervision is added to the learning strategy, it’s still
hard to balance the trade-off between classification ability and
transformation. As a result, bipartite and greedy strategy are better
solutions than the learning-based strategy and random strategies.

Effectiveness of Different Consistency Measurements. In Section 3.2, $L_{p}$,
correlation, KL divergence, and cosine consistency have been mentioned to
measure consistency between two feature maps. Table 2 shows the results of
using these different measurements. Correlation consistency measurement shows
better performance than other consistency measurement selections.

Transformation Strategy | Top-1
---|---
Baseline | 67.52
Random Transformation | 67.57
Bipartite | 68.35
Greedy | 67.83
Sub-net-FC | 65.47
Sub-net-Res | 65.26
Table 1: Ablation on different transformation strategies using correlation metric on ImageNet with MobileNetV2-0.5$\times$ as the student and ResNet50 as the teacher. The baseline is standard training with distillation but without transformation. Consistency Metric | Top-1
---|---
$L_{1}$ | 68.14
$L_{2}$ | 68.29
Cosine | 67.94
Correlation | 68.35
KL Divergence | 68.33
Table 2: Ablation on different consistency metrics using bipartite
transformation on ImageNet with MobileNetV2-0.5$\times$ as the student and
ResNet50 as the teacher.

Influence of Initialization. As shown in Algorithm 1, we should use the same
parameters $\theta_{0}^{S}$ to initialize the student while calculating
transformation $\mathcal{T}_{\theta^{S}_{0}}$ and distilling student model. To
measure the importance of keeping initialization the same, we retrain the
student model with an achieved transformation matrix but different
initialization. The student model is randomly reinitialized three times,
obtaining S1, S2, S3 with initial parameters $\theta_{1}^{S}$,
$\theta_{2}^{S}$, $\theta_{3}^{S}$. Then, train these three student models
will be trained under the guidance of the teacher using the same transform
$\mathcal{T}_{\theta^{S}_{0}}$.

As shown in Table 3 (a), students with different initial parameters will
achieve much worse performance than that with the same initial parameters
$\theta_{0}^{S}$, which means that useful information is lost with randomly
initialization and the effectiveness of transformation is not guaranteed under
this situation. Similarly, experiments on teacher models with different
initialization can be seen in Table 3 (b).

Results show that there is a correspondence between the achieved
$\mathcal{T}_{\theta^{S}_{0}}$ and the initialization of student as well as
teacher model. They prove that the initial weights will decide or at least
have an apparent effect on the final feature distribution. Analysis in [4]
also shows the importance of initialization which guesses that initial weights
are closely related to their final values after training.

Consistency Measurement on Random Initialized Models. In Section 3.2, the
consistency is calculated on trained models. We can also calculate the
consistency on random initialized models directly. Experiments showed in Table
4 use initialized student/trained teacher, trained student/trained teacher,
initialized student/initialized teacher, trained student/initialized teacher
four pairs to calculate the consistency respectively. Results show that with
initialized student and teacher, the transformation matrix does not work well,
which implies that initial weights can not replace their final trained values.

(a) |  | S | S1 | S2 | S3
---|---|---|---|---|---
Init. Params. | $\theta_{0}^{S}$ | $\theta_{1}^{S}$ | $\theta_{2}^{S}$ | $\theta_{3}^{S}$
Top-1 acc | 68.35 | 67.55 | 67.66 | 67.49
(b) |  | T | T1 | T2 | T3
Init. Params. | $\theta_{0}^{T}$ | $\theta_{1}^{T}$ | $\theta_{2}^{T}$ | $\theta_{3}^{T}$
Top-1 acc | 68.35 | 67.43 | 67.62 | 67.58
Table 3: Ablation on relation between initialization and transformation. The effect of randomly initialized student (MobileNetV2-0.5$\times$) and teacher (ResNet-50) is shown in (a) and (b) respectively. | $\bm{T_{0}}$ | $\bm{T}$
---|---|---
$\bm{S_{0}}$ | 67.47 | 67.6
$\bm{S}$ | 67.56 | 68.35
Table 4: Ablation on effect of initialization in calculating transformation
matrix $\mathcal{T}_{\theta^{S}_{0}}$. $S_{0}$ and $T_{0}$ refer to
MobileNetV2-0.5$\times$ and ResNet50 with initial weights. S and T denote
well-trained models on ImageNet.

Comparison of Static and Dynamic Transformation. As described in Section 3.4
and Algorithm 1, we seek for transformation $\mathcal{T}_{\theta^{S}_{0}}$ to
alleviate discrepancy between student and teacher and
$\mathcal{T}_{\theta^{S}_{0}}$ is fixed during the whole distillation process.
Since the training process is dynamic, we apply experiments of dynamic
transformation to show whether it is better to update the
$\mathcal{T}_{\theta^{S}_{0}}$ as the training goes on. Feature of the student
will be recorded in the distillation process, and
$\mathcal{T}_{\theta^{S}_{0}}$ is updated every 5 epoch with the help of
recorded features. However, experiment results in Table 5 show that a dynamic
transformation performs (68.31%) nearly the same as a static one (68.35%).

According to the hypothesis in [5] and ablation in the previous part, the
initialization has a large influence on the final network. Since we use the
same initialization in the student learning phase, the training process will
tend to a specific result which can be represented with a static
transformation. From this perspective, a static transformation has the ability
to match the features well. As a result, it’s unnecessary to use a dynamic
transformation.

Training Strategy | Baseline | Static | Dynamic
---|---|---|---
Top-1 | 67.52 | 68.35 | 68.31
Table 5: Ablation on different training strategy using correlation metric and
bipartite transformation on ImageNet with MobileNetV2-0.5$\times$ as student
and ResNet-50 as teacher.

Number of Transformations. Note that the proposed method seeks one
transformation that maps teacher features close to student features and
benefits its learning. It is also possible to calculate multiple
transformation matrices according to different input data, _i.e_. use
different transformations for different categories. In this section, we
conduct experiments to compare one transformation with multiple
transformations, according to the data category.

Taking ImageNet as an example, there are 1000 categories. We divide the data
into $K$ equal-sized partitions according to its class label. For each part,
we can obtain a transformation $\mathcal{T}_{\theta^{S}_{0}}^{k}$. For
instance, when $K=100$, the first part contains data with class label 1-10,
the second contains 11-20, etc. The extreme situation is $K=1000$ with one
class data in each part which means we calculate a transformation for each
class. Compared with the original $\mathcal{T}_{\theta^{S}_{0}}$, the
difference is that only part of the data is used when calculating each
transformation $\mathcal{T}_{\theta^{S}_{0}}^{k}$. Then for applying
transformation at the student training phase, for each input image, we first
figure out the part number $k$ it belongs to and find out its corresponding
$\mathcal{T}_{\theta^{S}_{0}}^{k}$. After that,
$\mathcal{T}_{\theta^{S}_{0}}^{k}$ is used to transform the teacher feature
and the transformed teacher feature will be employed to the knowledge
distillation method.

Table 6 exhibits the results. From the table, we can see that using more
transformations leads to a slight improvement in performance, and our method
already performs well compared with the extreme $K=1000$ situation.

K | 0 | 1 | 10 | 100 | 1000
---|---|---|---|---|---
Top-1 | 67.52 | 68.35 | 68.32 | 68.34 | 68.39
Table 6: Ablation on the influence of using different numbers of
transformation. $K=0$ means no transforming performed which is the common
knowledge distillation with $L_{2}$ loss. Our proposed method uses one
transformation matrix which is $K=1$.

###  4.3 Comparison with State-of-the-art Methods

We employ our transformation on several datasets of different vision tasks to
validate its advantages and generalization ability. The implementation of
knowledge distillation methods are from their source code except FitNets[21],
and we reproduced FitNets for comparison.

Method Student | ResNet20-0.5$\times$ | ResNet20
---|---|---
Baseline | 59.41 | 67.96
KD [10] | 60.25 | 68.88
FitNets [21] | 60.63 | 69.09
AT [26] | 60.58 | 69.42
RKD [20] | 61.43 | 69.87
AB [9] | 61.36 | 69.53
RCO [12] | 60.87 | 69.26
Ours | 61.38 | 69.69
Ours + FitNets | 61.59 | 69.77
Ours + AB | 61.87 | 70.03
Ours + RKD | 61.92 | 69.86
Ours + RCO | 61.47 | 70.13
Table 7: Comparison with other knowledge distillation methods of image
classification task on CIFAR-100. The teacher is ResNet-56 with 71.21%
accuracy. Baseline represents the individually trained students’ performance.

CIFAR-100. A ResNet56 is adopted as the teacher network while a ResNet20 and a
ResNet20-0.5$\times$ as student networks. The ResNet20-0.5$\times$ is obtained
by reducing the number of channels of the ResNet20 by half. Except for that
Hinton  _et al_. [10] uses output predictions to calculate distillation loss,
other methods use the output feature maps of the third residual block to
calculate mimicking loss. We show the top-1 accuracy of each method and our
proposed results in Table 7. From the results, we can find that our method
significantly improves performance. To be specific, with the ResNet56 as the
teacher and the ResNet20-0.5$\times$ as the student, combined with RKD, our
transformation achieves top-1 accuracy of 61.92%, improves 0.49% compared with
original RKD performance. For the ResNet20 as the student, the best
performance achieves 70.13% applied with RCO, which improves 2.17% compared
with the baseline.

ImageNet. We perform experiments on ImageNet to validate the existence and
effectiveness of feature transformation on large scale classification dataset.
Furthermore, we also adopt different architectures of teacher and student to
validate them. A ResNet50 is chosen to be the teacher while a ResNet18 and a
MobileNetV2-0.5$\times$ are used to be students separately. We compare the
top-1 accuracy with other distillation method and show the results in Table 8.
The results show that knowledge discrepancy also exists in large scale
classification, and equipped with our method can also further improve the
performances of state-of-the-art methods [12][9][20].

Method Student | ResNet18 | MBV2-0.5$\times$
---|---|---
Baseline | 70.29 | 64.27
KD [10] | 70.76 | 66.75
FitNets [21] | 70.73 | 67.58
AT [26] | 70.82 | 67.66
RKD [20] | 71.07 | 68.24
AB [9] | 71.26 | 68.49
RCO [12] | 71.04 | 68.21
Ours | 71.41 | 68.35
Ours + FitNets | 71.43 | 68.36
Ours + AB | 71.52 | 68.62
Ours + RKD | 71.27 | 68.93
Ours + RCO | 71.58 | 68.73
Table 8: Comparison with other knowledge distillation methods of image
classification task on ImageNet. The teacher is a ResNet50 with top-1 accuracy
75.49%. Baseline represents the individually trained students’ performance.

###  4.4 Extended Experiments on Object Detection

COCO. To further validate the generalization ability of the proposed method on
different tasks, we conducted experiments on the challenging COCO object
detection benchmark. Experiments are performed on FPN [15] with backbone
ResNet18 and ResNet50 as students separately. The teacher is an FPN with
backbone ResNet152. Knowledge distillation is applied to the output of the
last residual feature map of the backbone network and $L_{2}$ loss is used as
the distillation method. Performance has been shown in Table 9. All the
results state that the knowledge discrepancy also exists in distillation of
object detection tasks. Our method shows a consistent performance improvement
on the detection task. In the case of ResNet18, the results increases from
33.6 to 36.8 on AP which makes ResNet18 achieve comparable performance with
ResNet50. Experiments on a larger student ResNet50 also shows a significant
improvement from 37.7 to 40.8, outperforming the baseline by 3.1 points on AP.

Backbone | Setting | AP
---|---|---
ResNet-18 | Student | 33.6
ResNet-50 | Student | 37.7
ResNet-152 | Teacher | 41.6
ResNet-18 | $L_{2}$ | 36.3
ResNet-18 | Ours | 36.8
ResNet-50 | $L_{2}$ | 40.4
ResNet-50 | Ours | 40.8
Table 9: Object detection results on COCO benchmark with FPN. Results are
described in Average Precision (AP). Last four rows use ResNet-152 as the
teacher

##  5 Analysis

###  5.1 Effects on Increasing Channel Overlap

In this section, we apply quantitative analysis on channel activations to show
the discrepancy between T and S.

Settings. ImageNet has 1000 classes and we use the training data for each
class to calculate averaging activation maps respectively. For each class, we
sort channels by the value of averaged activations. Then, we gather the top-k
sets of channel indexes for each class. After that, for a pair of student and
teacher models, the overlapping ratio for each class is calculated by the
intersection of top-k sets. After that, the overall overlapping ratio is
obtained by averaging overlapping ratios across all classes. Table 10 shows
the result of channel overlap percentage over all channels with top-10 to
top-100 highest activation channels respectively.

Analysis. The first row in Table 10 indicates that the activation responses of
student and teacher models have large variations that the highly activated
channels share different indexes in the whole feature map. The result is
consistent with what we demonstrate in Section 3.2.

We measure the overall overlapping ratio between two differently initialized
students and their activation maps also show a large difference as shown in
the second row of Table 10, which implies that the initialization will have an
important effect on the distribution of the activation area of models even
they share the same network structure.

Our method aims to reduce this discrepancy of activation responses. As
depicted in the third row of Table 10, with S and transformed T, the
discrepancy is narrowed down and it will be much easier for S to learn
knowledge from T.

Models | Top-10 | Top-20 | Top-50 | Top-100
---|---|---|---|---
$\bm{S}$ | $\bm{T}$ | 18% | 24% | 31% | 34%
$\bm{S_{1}}$ | $\bm{S_{2}}$ | 32% | 35% | 44% | 48%
$\bm{S}$ | $\bm{\mathcal{T}(T)}$ | 74% | 78% | 89% | 93%
Table 10: Analysis on whether channels with high activation in different
models share order overlap. S and T are MobileNetV2-0.5$\times$ and ResNet50
trained on ImageNet. $S_{1}$ and $S_{2}$ are trained student with different
initialization. $\mathcal{T}(T)$ refers to transformed T with the proposed
method. All the results are averaged over all the 1000 classes activation
maps.

###  5.2 Effects on Reducing Discrepancy

Method | KL divergence |  $L_{2}$ distance
---|---|---
Baseline | 0.89 | 277.19
Baseline + Transformation | 0.86 | 189.93
FitNets [26] | 0.76 | 53.74
Ours + FitNets | 0.71 | 45.72
RKD [20] | 0.82 | 52.19
Ours + RKD | 0.64 | 39.77
AB [9] | 0.73 | 47.82
Ours + AB | 0.58 | 41.84
RCO [12] | 0.69 | 46.62
Ours + RCO | 0.65 | 40.03
Table 11: Analysis on feature consistency between teacher and student. Results
are averaged over all test images on ImageNet. Baseline is calculated between
T and trained S.

To see how well the knowledge distillation results improved by our proposed
method, we measure the consistency of the teacher’s and student’s output. On
the evaluation set of ImageNet, we compare the KL divergence and $L_{2}$
distance between the teacher and student feature map. Results are shown in
Table 11. With a transformation from teacher to student, the distance is
reduced before training. All the knowledge distillation methods show a
reduction in KL divergence and $L_{2}$ distance, which implies that the
student has learned knowledge from the teacher and becomes closer to the
teacher. Specifically, our proposed method shows a considerable high
consistency compared to those without the transformation operation. In other
words, our proposed feature transformation promotes the original distillation
methods obtaining more consistent features.

##  6 Conclusion

In this paper, we claim that students should be taught according to their
fitness to the teacher while applying knowledge distillation methods. The
proposed method provides an easier way for the student to learn from a
transformed teacher feature. Different from existing methods, the internal
discrepancy between student and teacher is considered and reduced. We have
proved the promising results of the proposed method on different datasets and
conducted plenty of ablation studies to verify the correctness of our
intuition as well as the effect of the method.

## References

  * [1] Jimmy Ba and Rich Caruana.  Do deep nets really need to be deep?  In NeurIPS, 2014.
  * [2] Cristian Buciluǎ, Rich Caruana, and Alexandru Niculescu-Mizil.  Model compression.  In SIGKDD, 2006.
  * [3] Jia Deng, Wei Dong, Richard Socher, Li Jia Li, and Fei Fei Li.  Imagenet: a large-scale hierarchical image database.  In 2009 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR 2009), 20-25 June 2009, Miami, Florida, USA, 2009.
  * [4] Jonathan Frankle and Michael Carbin.  The lottery ticket hypothesis: Finding sparse, trainable neural networks.  ICLR, 2018.
  * [5] Jonathan Frankle, Gintare Karolina Dziugaite, Daniel M Roy, and Michael Carbin.  The lottery ticket hypothesis at scale.  arXiv preprint arXiv:1903.01611, 2019.
  * [6] Adam Gaier and David Ha.  Weight agnostic neural networks.  In Advances in Neural Information Processing Systems, pages 5364–5378, 2019.
  * [7] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In CVPR, 2016.
  * [8] Byeongho Heo, Jeesoo Kim, Sangdoo Yun, Hyojin Park, Nojun Kwak, and Jin Young Choi.  A comprehensive overhaul of feature distillation.  In International Conference on Computer Vision (ICCV), 2019.
  * [9] Byeongho Heo, Minsik Lee, Sangdoo Yun, and Jin Young Choi.  Knowledge transfer via distillation of activation boundaries formed by hidden neurons.  In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 3779–3787, 2019.
  * [10] Geoffrey Hinton, Oriol Vinyals, and Jeff Dean.  Distilling the knowledge in a neural network.  In NeurIPS Workshop, 2014.
  * [11] Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger.  Densely connected convolutional networks.  In CVPR, 2017.
  * [12] Xiao Jin, Baoyun Peng, Yichao Wu, Yu Liu, Jiaheng Liu, Ding Liang, Junjie Yan, and Xiaolin Hu.  Knowledge distillation via route constrained optimization.  arXiv preprint arXiv:1904.09149, 2019.
  * [13] Alex Krizhevsky.  Learning multiple layers of features from tiny images.  2009\.
  * [14] Seung Hyun Lee, Dae Ha Kim, and Byung Cheol Song.  Self-supervised knowledge distillation using singular value decomposition.  In ECCV, 2018.
  * [15] Tsung Yi Lin, Piotr Dollar, Ross Girshick, Kaiming He, Bharath Hariharan, and Serge Belongie.  Feature pyramid networks for object detection.  In CVPR, 2017.
  * [16] Tsung-Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár.  Focal loss for dense object detection.  In ICCV, 2017.
  * [17] Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick.  Microsoft coco: Common objects in context.  In ECCV, 2014.
  * [18] Yufan Liu, Jiajiong Cao, Bing Li, and Chunfeng Yuan.  Knowledge distillation via instance relationship graph.  In CVPR, 06 2019.
  * [19] Yifan Liu, Ke Chen, Chris Liu, Zengchang Qin, Zhenbo Luo, and Jingdong Wang.  Structured knowledge distillation for semantic segmentation.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 2604–2613, 2019.
  * [20] Wonpyo Park, Dongju Kim, Yan Lu, and Minsu Cho.  Relational knowledge distillation.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 3967–3976, 2019.
  * [21] Adriana Romero, Nicolas Ballas, Samira Ebrahimi Kahou, Antoine Chassang, Carlo Gatta, and Yoshua Bengio.  Fitnets: Hints for thin deep nets.  In ICLR, 2015.
  * [22] Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jon Shlens, and Zbigniew Wojna.  Rethinking the inception architecture for computer vision.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2818–2826, 2016.
  * [23] Frederick Tung and Greg Mori.  Similarity-preserving knowledge distillation.  In Proceedings of the IEEE International Conference on Computer Vision, pages 1365–1374, 2019.
  * [24] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  In Advances in neural information processing systems, pages 5998–6008, 2017.
  * [25] Junho Yim, Donggyu Joo, Jihoon Bae, and Junmo Kim.  A gift from knowledge distillation: Fast optimization, network minimization and transfer learning.  In CVPR, 2017.
  * [26] Sergey Zagoruyko and Nikos Komodakis.  Paying more attention to attention: Improving the performance of convolutional neural networks via attention transfer.  In ICLR, 2017.


That is, we pick the $\sigma_{0}(1)$-th row of $N_{ij}$ as the first row of
the new matrix, the $\sigma_{0}(2)$-th row as the second row of the new
matrix, and so on until the $\sigma_{0}(N_{\rm g})$-th old row as the $N_{\rm
g}$-th row of the new matrix. This yields a new matrix $N^{\prime}_{lj}$, with
rows permuted from $N_{ij}$. Here we use “$l$” as the row index for the new
matrix; we keep “$j$” as the column index because the columns are preserved
after the re-indexing. We emphasize that the re-indexing just gives each true
group a “new name”, instead of changing the classification results. By
construction, we have

$N^{\prime}_{lj}=\left.N_{ij}\right|_{i=\sigma_{0}(l)}\text{ and
}N_{ij}=\left.N^{\prime}_{lj}\right|_{l=\sigma_{0}^{-1}(i)}.$

Based on this, we can easily derive that

$N^{\prime}_{\sigma^{\prime}}=N_{\sigma}$ (4)

where $\sigma^{\prime}\equiv\sigma_{0}^{-1}\cdot\sigma$ (for example,
$\sigma^{\prime}_{0}\equiv\sigma_{0}^{-1}\cdot\sigma_{0}=I$). These relations
are valid for all indices $l$ and mappings $\sigma$. Therefore, the accuracy
becomes

${\rm accuracy}=\frac{N^{\prime}_{\sigma^{\prime}_{0}}}{\sum_{l=1}^{N_{\rm
g}}\sum_{j=1}^{N_{\rm g}}N^{\prime}_{lj}}.$ (5)

Let us consider a series of mappings $\\{\sigma^{\prime}_{k}\\}$ defined as

$\sigma^{\prime}_{k}(j)\equiv\left\\{\begin{array}[]{ll}j+k,&\text{if}\
j+k\leq N_{\rm g}\\\ j+k-N_{\rm g},&\text{if}\ j+k>N_{\rm
g}\end{array}\right.$

That is, $\sigma^{\prime}_{k}(j)$ shifts the index $j$ by $k$ with a periodic
boundary condition. Note that $\sigma^{\prime}_{0}$ is naturally included in
this series as the case of $k=0$. We can prove that

$\displaystyle\sum_{k=0}^{N_{\rm g}-1}N^{\prime}_{\sigma^{\prime}_{k}}$
$\displaystyle\equiv\sum_{k=0}^{N_{\rm g}-1}\sum_{j=1}^{N_{\rm
g}}\left.N^{\prime}_{lj}\right|_{l=\sigma^{\prime}_{k}(j)}=\sum_{j=1}^{N_{\rm
g}}\sum_{k=0}^{N_{\rm
g}-1}\left.N^{\prime}_{lj}\right|_{l=\sigma^{\prime}_{k}(j)}$
$\displaystyle=\sum_{j=1}^{N_{\rm g}}\left(\sum_{k=0}^{N_{\rm
g}-j}\left.N^{\prime}_{lj}\right|_{l=j+k}+\sum_{k=N_{\rm g}-j+1}^{N_{\rm
g}-1}\left.N^{\prime}_{lj}\right|_{l=j+k-N_{\rm g}}\right)=\sum_{j=1}^{N_{\rm
g}}\left(\sum_{l=j}^{N_{\rm
g}}N^{\prime}_{lj}+\sum_{l=1}^{j-1}N^{\prime}_{lj}\right)$
$\displaystyle=\sum_{j=1}^{N_{\rm g}}\sum_{l=1}^{N_{\rm
g}}N^{\prime}_{lj}=\sum_{l=1}^{N_{\rm g}}\sum_{j=1}^{N_{\rm
g}}N^{\prime}_{lj}.$

The last equality of the equation equals the denominator of Eq. (5), i.e., the
total number of data points. Using Eq. (4) and the definition of $\sigma_{0}$,
we obtain $N^{\prime}_{\sigma^{\prime}_{k}}\leq
N^{\prime}_{\sigma^{\prime}_{0}}$ for all $k$. Plugging this inequality and
the equation above into Eq. (5), we get

${\rm accuracy}=\frac{N^{\prime}_{\sigma^{\prime}_{0}}}{\sum_{k=0}^{N_{\rm
g}-1}N^{\prime}_{\sigma^{\prime}_{k}}}\geq\frac{N^{\prime}_{\sigma^{\prime}_{0}}}{\sum_{k=0}^{N_{\rm
g}-1}N^{\prime}_{\sigma^{\prime}_{0}}}=\frac{1}{N_{\rm g}}.$ (6)

So far we have proved that the accuracy is always greater than or equal to
$1/N_{\rm g}$. The equality can be reached if all
$N^{\prime}_{S^{\prime}_{k}}$ are the same, corresponding to completely random
classification. Thus, any classification with accuracy close to this value
should be regarded as ineffective.

##  Appendix B Maximum achievable accuracy

Although the classification accuracy by definition (Appendix A) can reach 100%
in ideal cases, achieving a perfect accuracy may not be feasible due to the
intrinsic mixing of distributions in the multi-dimensional data space. To
address this, we conduct a cross-validation examination utilizing supervised
classification methods to determine a more realistic and statistically
significant upper bound of accuracy. We take the in-situ vs. ex-situ case as a
case study to derive the maximum achievable accuracy.

To estimate the maximum achievable accuracy, we employ a cross-validation
technique similar to the approach in §4.2. This process splits the data into a
training set and a validation set. We train a supervised classifier on the
training set and evaluate the classification accuracy on the validation set.
We allocate 80% of the data to the training set and the remaining 20% to the
validation set. This ensures that the training set is large enough for the
classifier to learn adequate information. We repeat the training and
validation process 500 times to fully sample the accuracy distribution on the
validation sets.

In Fig. 13, we plot the accuracy distribution on validation sets for our four
sample galaxies. We obtain the distribution using three different supervised
classification methods detailed in §3.3. The distribution depends weakly on
the choices of method, indicating that the limiting factor is indeed the
intrinsic mixing of the data, not the specific classification technique.
Hence, these accuracy values represent the maximum achievable accuracy.

For the four galaxies in our study, the maximum achievable accuracy ranges
from 87% to 95%, with a median around 90%. Any categorization using the same
10 input variables is unlikely to significantly exceed these values without
overfitting the data. Our best accuracy values in §3 are very close to these
upper bounds, suggesting that we are approaching the theoretical maximum with
the 10 variables. We also note a galaxy-to-galaxy variation, where m12i
exhibits a maximum achievable accuracy greater than 93%, while that for 519311
is around 87%. This agrees with our findings in §3, where 519311 often shows
lower accuracy. This indicates that the extent of phase mixing within the 10D
property space differs among galaxies.