# Exploring Graph-based Knowledge: Multi-Level Feature Distillation via
Channels Relational Graph

Zhiwei Wang1,3 Jun Huang1 Longhua Ma2 Chengyu Wu2 Hongyu Ma
1Ningbo Innovation Center, Zhejiang University, China 2NingboTech University
<EMAIL_ADDRESS>
Correspondence to: Longhua Ma<EMAIL_ADDRESS>
###### Abstract

In visual tasks, large teacher models capture essential features and deep
information, enhancing performance. However, distilling this information into
smaller student models often leads to performance loss due to structural
differences and capacity limitations. To tackle this, we propose a
distillation framework based on graph knowledge, including a multi-level
feature alignment strategy and an attention-guided mechanism to provide a
targeted learning trajectory for the student model. We emphasize spectral
embedding (SE) as a key technique in our distillation process, which merges
the student’s feature space with the relational knowledge and structural
complexities similar to the teacher network. This method captures the
teacher’s understanding in a graph-based representation, enabling the student
model to more accurately mimic the complex structural dependencies present in
the teacher model. Compared to methods that focus only on specific
distillation areas, our strategy not only considers key features within the
teacher model but also endeavors to capture the relationships and interactions
among feature sets, encoding these complex pieces of information into a graph
structure to understand and utilize the dynamic relationships among these
pieces of information from a global perspective. Experiments show that our
method outperforms previous feature distillation methods on the CIFAR-100, MS-
COCO, and Pascal VOC datasets, proving its efficiency and applicability.

##  1 Introduction

In the field of deep learning, the impressive success of large neural networks
has come at the cost of increased computational complexity, which poses
significant challenges for deployment in resource-constrained environments.
While these heavyweight models, often referred to as teacher networks, set
state-of-the-art benchmarks on various tasks, their practical applicability is
limited by their demanding requirements for memory, processing power, and
energy. Knowledge Distillation (KD) [16] emerges as a promising solution to
address this dichotomy by transferring the knowledge from a cumbersome model
to a more compact and efficient student networks.

The quintessence of knowledge distillation lies in its ability to encapsulate
the representational power of a larger model into a smaller one without
incurring a substantial loss in performance. Pioneered by Hinton et al. [16],
the process involves training a smaller model to mimic the behavior of the
pre-trained larger model by softening the output logits [19], thus leveraging
the rich information embedded in the output distributions of the teacher
network.

Recent advances in KD techniques have extended beyond the mere replication of
output distributions. Contemporary works explore the distillation of
intermediate representations [1, 49, 31], attention mechanisms [50, 20], and
even the inculcation of adversarial robustness from teacher to student [33,
55, 37]. The underlying hypothesis is that the intermediate layers of a neural
network embody a wealth of information about the data manifold that, when
transferred effectively, can endow the student with nuanced understanding akin
to its teacher.

Large and complex teacher models often capture a wealth of features and deep
information, crucial for enhancing task performance [7]. However, attempting
to directly distill this rich information into smaller capacity student models
often results in suboptimal performance reproduction due to the student
model’s capacity limitations. The student model may struggle to process the
complex information present in the teacher model, leading to ineffective
distillation. Moreover, simply mimicking all features of the teacher model
overlooks the differences in relationships and structures between the
information, particularly when there is a significant gap between the teacher
and student models [12]. Recently, researchers have adopted a strategy of
carefully selecting distillation areas, aiming to identify and choose key
areas that positively impact student learning [3, 6, 43, 23, 38, 13]. This
approach is based on the notion that not all information in the teacher model
is equally important for the student model’s learning. By focusing on features
and areas most beneficial for the student model’s performance improvement,
more efficient and targeted knowledge transfer can be achieved. Although this
selective distillation strategy has seen some success, the process of
determining which areas are "key" can be influenced by subjective judgment,
leading to different outcomes based on different strategies and criteria. The
effectiveness of this method may be limited by the lack of a clear and
consistent standard. Over-focusing on local features while ignoring the
interdependencies and global structural information among features limits the
student model’s comprehensive understanding of the teacher model’s deep
knowledge structure. The student model may perform poorly when faced with new
or changing data.

To overcome these challenges, we present a novel distillation framework that
delves into the sophisticated feature hierarchies and attention mechanisms
naturally acquired by deep convolutional networks. Central to our approach is
a multi-level feature alignment strategy that harmonizes the representational
layers of the student model with those of the teacher model, moving past
traditional feature mimicry. Additionally, we introduce an attention-guided
mechanism that highlights the teacher’s critical focus areas, enabling a more
targeted and efficient learning trajectory for the student model.Inspired by
the success of spectral embedding in the field of clustering [32], we
particularly emphasize the introduction of spectral embedding as a cornerstone
of our distillation methodology. This advanced technique underpins a nuanced
channel-wise distillation process. Recognizing that the channels of the
teacher network do not exist in isolation but are interdependent in expressing
input data, together forming a complex network of feature interactions [28],
spectral embedding infuses the student model’s feature space with the
relational knowledge and structural complexities akin to those in the teacher
network. Through this method, we aim to enrich the student model with a deeper
level of distilled insights, capturing the essence of the teacher’s
understanding in a graph-based representation. We hope that the use of
spectral embedding and graph-based knowledge encapsulation can enrich the
learning experience for students and provide a novel perspective to
conventional distillation practices. Unlike approaches that target limited
distillation regions, our methodology not only identifies crucial features in
the teacher model but also aims to grasp the intricate interplay between these
features. By representing this sophisticated data as a graph, we can
holistically comprehend and leverage the evolving interconnections between
various elements, offering a comprehensive view of the knowledge to be
distilled.

We conducted comprehensive experiments across a wide spectrum of benchmark
datasets, encompassing CIFAR-100 [21] for image classification, MS-COCO [27]
and Pascal VOC [8] for object detection. Detailed experimental details can be
found in Section III. This broad experimental setup allowed us to compare
various teacher-student architectures and adaptation layers across multiple
visual tasks.

In summary, our contributions in this paper are threefold:

  * •

We pioneer the modeling of inter-channel correlations within the teacher
network as a graphical structure. This methodology lays the foundation for our
Spectral Embedding Knowledge Distillation (SEKD) technique.

  * •

Our method employs spectral embedding to encapsulate the knowledge of the
teacher network, and through an attention-guided mechanism, focuses on a
holistic understanding across channel-level, relational, and structural
dimensions, ensuring the student network inherits a comprehensive
representation of the teacher’s knowledge.

  * •

We provide comprehensive empirical evidence that our approach outperforms
existing distillation methods, achieving state-of-the-art performance on
multiple benchmarks.

##  2 Method

Figure 1: The overall framework of the Multi-Level Feature Distillation. The
process of generating embedding vectors in groups is shown in Figure 2. We
align the channel relational graphs of the teacher and student across multiple
levels: vertices, edges, and spectral embeddings.

The overall framework of our method is illustrated in Fig. 1. In this section,
we will first construct the channels and their relationships of the feature
maps into a graph called $\boldsymbol{CRG}$. Then the vertex and edge loss
with attention mechanism of $\boldsymbol{CRG}$ will be formulated to
constitute the $\boldsymbol{CRG}$ loss. Subsequently, the spectral embedding
process and the loss will be described in detail. Finally, we combine the
multi-level feature loss to supervise the distillation.

###  2.1 Channels Relational Graph

We use $\boldsymbol{F}\in R^{C\times W\times H}$ to denote the feature maps
from one layer of the backbone, where $C$, $W$, $H$ denotes its channel
number, width and height. Consider each channel as the vertex and the relation
between the channels as the edge. Then the Channels Relation Graph
($\boldsymbol{CRG}$) can be formulated as:

$\boldsymbol{CRG}=(\mathcal{V},\mathcal{E})=(\\{\boldsymbol{F}_{k,:,:}\\}^{C}_{k=1},\boldsymbol{A})$
(1)

where $\mathcal{V}$ is the vertex set of $\boldsymbol{CRG}$ composed of
various channels of feature maps, $\mathcal{E}$ is the edge set of
$\boldsymbol{CRG}$ represented by a weighted adjacency matrix
$\boldsymbol{A}\in R^{C\times C}$, the subscript $k,:,:$ of
$\boldsymbol{F}_{k,:,:}$ indicates the $k$-th channel of the feature maps in
the layer. Each element of the adjacency matrix is obtained by calculating
cosine similarity using two flattened feature maps as vectors, which can be
formulated as:

$\boldsymbol{A}_{ij}=cos\angle
vec(\boldsymbol{F}_{i,:,:})vec(\boldsymbol{F}_{j,:,:})=\frac{<vec(\boldsymbol{F}_{i,:,:}),vec(\boldsymbol{F}_{j,:,:})>}{\parallel
vec(\boldsymbol{F}_{i,:,:})\parallel\parallel
vec(\boldsymbol{F}_{j,:,:})\parallel},i,j=1,2,3,...,C,$ (2)

where the operator $vec(\cdot)$ vectorize a channel feature map. The adjacency
matrix normalizes the pairwise relationship between channels using cosine
similarity.

###  2.2 Loss for CRG

Figure 2: Illustration of generating embedding vectors group. We consider the
Laplacian matrix as an algebraic representation of the graph. These embedding
vectors provide a structural mapping and topological characteristics of the
graph in a low-dimensional subspace.

The $\boldsymbol{CRG}$ Distillation loss function $\mathcal{L}_{CRG}$ consists
of two parts: the Vertex loss $L_{V}$ and the Edge loss $L_{E}$. Instead of
having the student directly mimic the teacher, we utilize the attention mask
matrix to guide the distillation. Previous studies have shown that spatial and
channel attention masks from teacher models can guide students to focus on
learning important foreground information. By overlaying the absolute value of
feature maps from teacher model and using softmax function to convert, the
attention mask matrices can be formulated as:

$\boldsymbol{M}^{s}=H\cdot W\cdot
softmax(\frac{1}{C}\sum_{k=1}^{C}\left|\boldsymbol{F}_{k,:,:}\right|),\>\boldsymbol{M}^{c}=C\cdot
softmax(\frac{1}{HW}\sum_{i=1}^{W}\sum_{j=1}^{H}\left|\boldsymbol{F}_{:,i,j}\right|)$
(3) $\boldsymbol{M}^{r}=softmax(\left|A\right|)$ (4)

where $\boldsymbol{F}_{k,:,:}$ and $\boldsymbol{F}_{:,i,j}$ denotes the $k$-th
slice channel and the $i$-th row $j$-th column sequence of $\boldsymbol{F}$
respectively. Note that the superscripts "s", "c" and "r" represent "spatial",
"channel" and "relation" respectively. The spatial attention mask matrix
$\boldsymbol{M}^{s}$ implies the important region of the feature map while the
channel attention mask matrix $\boldsymbol{M}^{c}$ implies the important
channel in a group of channels. Similarly, the relational attention mask
matrix indicates the relationships between the most noteworthy feature maps.
$\mathcal{L}_{V}$ is utilized to enforce the student to absorb the feature
maps knowledge by a $L_{2}$ norm loss under the decoration of
$\boldsymbol{M}^{s}$ and $\boldsymbol{M}^{c}$, which can be formulated as:

$\mathcal{L}_{V}=\frac{1}{CHW}\sum_{k=1}^{C}\sum_{i=1}^{H}\sum_{j=1}^{W}(\boldsymbol{F}_{k,i,j}^{T}-\boldsymbol{F}_{k,i,j}^{S})^{2}\cdot\boldsymbol{M}^{s}_{i,j}\cdot\boldsymbol{M}^{c}_{k}$
(5)

where the superscripts T and S is utilized to discriminate the teacher model
and student model. Likewise, $\mathcal{L}_{E}$ is utilized to enforce the
student to absorb the channel relation knowledge by a $L_{2}$ norm loss under
the decoration of $M^{r}$, which can be formulated as:

$\mathcal{L}_{E}=\frac{1}{C^{2}}\sum_{i=1}^{W}\sum_{j=1}^{H}(\boldsymbol{A}^{T}_{i,j}-\boldsymbol{A}^{S}_{i,j})^{2}\cdot\boldsymbol{M}^{r}_{i,j}$
(6)

Algorithm 1 Multi-Level Feature Distillation

1:$F^{S},F^{T}$: feature maps of student and teacher from layers 1 to L

2:$L_{Total}\leftarrow 0$ $\triangleright$ Initialize the total loss

3:for $i\leftarrow 1$ to $L$ do

4: $F^{S,v}_{i}\leftarrow F^{S}[i].view(F^{S}[i].size(0),-1)$ $\triangleright$
Vectorize and Normalize student features

5: $F^{T,v}_{i}\leftarrow F^{T}[i].view(F^{T}[i].size(0),-1)$ $\triangleright$
Vectorize and Normalize teacher features

6: $A^{S}\leftarrow F^{S,v}_{i}@F^{S,v}_{i}.T$ $\triangleright$ Adjacency
matrix for student

7: $A^{T}\leftarrow F^{T,v}_{i}@F^{T,v}_{i}.T$ $\triangleright$ Adjacency
matrix for teacher

8: $L_{V}\leftarrow$ ComputeVertexLoss($A^{S},A^{T}$) $\triangleright$ Vertex
loss based on feature similarity

9: $L_{E}\leftarrow$ ComputeEdgeLoss($A^{S},A^{T}$) $\triangleright$ Edge loss
from graph theory perspective

10: $D^{S}\leftarrow torch.diag(A^{S}.sum(1))$

11: $D^{T}\leftarrow torch.diag(A^{T}.sum(1))$

12: $L^{sym,S}\leftarrow
torch.eye(D^{S}.size(0))-D_{S}^{-0.5}<EMAIL_ADDRESS>$\triangleright$
Laplacian for student

13: $L^{sym,T}\leftarrow
torch.eye(D^{T}.size(0))-D_{T}^{-0.5}<EMAIL_ADDRESS>$\triangleright$
Laplacian for teacher

14: $E^{S},\\_\leftarrow torch.linalg.eigh(L^{sym,S})$

15: $E^{T},\\_\leftarrow torch.linalg.eigh(L^{sym,T})$

16: $L_{S}\leftarrow$ ComputeSpectralLoss($E^{S},E^{T}$) $\triangleright$
Spectral embedding loss from eigenvalues

17: $L_{M}\leftarrow L_{V}+L_{E}+L_{S}$ $\triangleright$ Combine losses for
multiple distillation levels

18: $L_{Total}\leftarrow L_{Total}+L_{M}$ $\triangleright$ Accumulate total
loss across all layers

19:end for

20:$L_{Total}\leftarrow L_{Total}+L_{Original}$ $\triangleright$ Include any
original task-specific loss

###  2.3 Loss for Spectral Embedding

Spectral Embedding is an effective method to extract the features of a group
of data points by reducing the dimension and retaining the key information,
which is often used in clustering algorithms. In our method, we apply the
spectral embedding to capture the geometric and manifold structure of the
$\boldsymbol{CRG}$. In order to perform spectral embedding processing on
$\boldsymbol{CRG}$, we need to calculate the weighted degree matrix
$\boldsymbol{D}$ and the the symmetric normalized Laplacian matrix $L^{sym}$,
which can be formulated as:

$\boldsymbol{D}_{i,i}=\sum_{i=1}^{C}\boldsymbol{A}_{i,j},L^{sym}=I-\boldsymbol{D}^{-1/2}\boldsymbol{AD}^{-1/2}$
(7)

where the weighted degree matrix $\boldsymbol{D}$ is a diagonal matrix and the
elements on the diagonal $\boldsymbol{D}_{i,i}$ is calculated by summing the
$i$-th column of the adjacency matrix $\boldsymbol{A}$. Note that $I$ is an
identity matrix. Due to $L^{sym}$ being a symmetric matrix, we can perform
eigenvalue decomposition on it as
$L^{sym}=\boldsymbol{U}\Lambda\boldsymbol{U}^{T}$. Where $\Lambda$ is a
diagonal matrix composed of the $C$ eigenvalues of a diagonal element
$L^{sym}$ and $\boldsymbol{U}$ is an eigenvector matrix composed of linearly
independent eigenvectors corresponding to $C$ eigenvalues. By selecting the
eigenvectors corresponding to the top $N$ largest eigenvalues, we can form a
spectral embedding vectors group matrix $\boldsymbol{E}$. The hyperparameter
sensitivity experiment for $N$ is detailed in the Appendix C.3. We provide a
detailed explanation of the process of generating spectral embeddings group in
Fig. 2. Now, we can define the spectral embedding loss function with a $L_{2}$
norm loss of $\boldsymbol{E}$ to enforce the student to imitate the teacher
model in a reduced dimensional space, which can be formulated as:

$\mathcal{L}_{S}=\frac{1}{CN}\sum_{i=1}^{C}\sum_{j=1}^{N}(\boldsymbol{E}^{T}_{i,j}-\boldsymbol{E}^{S}_{i,j})^{2}$
(8)

###  2.4 Multi-Level Feature Loss And Overall Loss

Now we have designed the Vertex Loss, the Edge loss and the Spectral Embedding
loss, and by introducing three hyper-parameters to balance them the Multi-
Level Feature Loss can be combined as:

$\mathcal{L}_{M}=\alpha\cdot\mathcal{L}_{V}+\beta\cdot\mathcal{L}_{E}+\gamma\cdot\mathcal{L}_{S},\>\mathcal{L}_{Total}={L}_{M}+{L}_{Origin}$
(9)

Wherein, ${L}_{Origin}$ represents the loss function between the prediction
results and the Ground Truth. ${L}_{M}$ denotes the Multi-Level Feature Loss.
${L}_{Total}$ denotes the global loss function. Our source of knowledge comes
from the feature representations of the intermediate layers. Therefore, it can
be generalized to various detectors, various teacher-student architectures,
and other visual tasks.

Now that we have introduced the calculation methods for all loss functions, we
can select several important layers in the teacher and student networks for
distillation. The main flow of the Multi-Level Feature Distillation is
summarized in Algorithm 1. For the sensitivity experiments pertaining to the
hyperparameters $\alpha$, $\beta$ and $\gamma$, please refer to Section 3.2.

Table 1: We benchmark our approach by comparing it with other state-of-the-art
(SOTA) methods across various detectors. In this comparison, the teacher
network employed is ResNet101, while the student network is ResNet50. The
evaluation is conducted on the MS-COCO val2017 dataset. † indicates that the
method was reproduced by us, while the remaining results are sourced from the
related paper.

Method | AP | AP50 | AP75 | APS | APM | APL
---|---|---|---|---|---|---
Faster RCNN [9]-FPN [25]
Teacher | 42.0 | 62.5 | 45.9 | 25.2 | 45.6 | 54.6
Student | 37.4 | 58.4 | 40.7 | 21.8 | 41.0 | 47.8
†KD [16] | 38.3 | 59.4 | 41.7 | 22.2 | 41.9 | 50.4
†FitNet [1] | 38.8 | 59.6 | 41.8 | 22.3 | 42.2 | 50.7
FGFI [43] | 39.4 | 60.3 | 43.0 | 22.9 | 43.6 | 52.8
ReviewKD [4] | 40.4 | 61.0 | 44.0 | 23.6 | 43.8 | 52.9
MKD [19] | 40.2 | 61.6 | 44.5 | 23.2 | 44.5 | 53.4
Ours | 41.9 | 61.8 | 45.5 | 24.1 | 44.6 | 54.0
RetinaNet [26]
Teacher | 38.9 | 58.0 | 41.5 | 21.0 | 42.8 | 52.4
Student | 37.4 | 56.7 | 39.6 | 20.0 | 40.7 | 49.7
†FitNet [1] | 37.4 | 57.1 | 40.0 | 20.8 | 40.8 | 50.9
FGFI [43] | 39.0 | 58.8 | 41.9 | 21.7 | 42.9 | 52.2
FRS [53] | 39.3 | 58.8 | 42.0 | 21.5 | 43.3 | 52.6
GI [6] | 39.1 | 59.0 | 42.3 | 22.8 | 43.1 | 52.3
†FGD [48] | 39.6 | 59.4 | 42.7 | 22.9 | 43.7 | 53.6
Ours | 39.9 | 59.8 | 43.2 | 23.6 | 44.1 | 54.0

Method | AP | AP50 | AP75 | APS | APM | APL
---|---|---|---|---|---|---
FCOS [40]
Teacher | 40.8 | 60.0 | 44.0 | 24.2 | 44.3 | 52.4
Student | 38.5 | 57.7 | 41.0 | 21.9 | 42.8 | 48.6
†FitNet [1] | 39.9 | 58.6 | 43.1 | 23.1 | 43.4 | 52.2
GI [6] | 42.0 | 60.4 | 45.5 | 25.6 | 45.8 | 54.2
†FGD [48] | 42.1 | 60.6 | 45.9 | 27.0 | 46.0 | 54.6
FRS [53] | 40.9 | 60.3 | 43.6 | 25.7 | 45.2 | 51.2
MasKD [17] | 42.6 | 61.2 | 46.3 | 26.5 | 46.9 | 54.2
Ours | 43.1 | 61.4 | 46.5 | 27.3 | 47.9 | 55.6
GFL [24]
Teacher | 45.0 | 63.7 | 48.9 | 27.2 | 48.8 | 54.5
Student | 40.1 | 58.2 | 43.1 | 23.3 | 44.4 | 52.5
†FitNet [1] | 40.7 | 58.6 | 44.0 | 23.7 | 44.4 | 53.2
GI [6] | 41.5 | 59.6 | 45.2 | 24.3 | 45.7 | 53.6
†FGD [48] | 41.6 | 59.3 | 44.8 | 24.5 | 45.7 | 53.7
PGD [42] | 41.8 | 59.8 | 45.2 | 25.0 | 46.4 | 53.4
LD [52] | 42.1 | 60.3 | 45.6 | 24.5 | 46.2 | 54.8
Ours | 42.7 | 60.9 | 46.2 | 25.0 | 46.3 | 55.1

##  3 Experiments

Details of the experimental implementation can be found in Appendix B.

###  3.1 Main Results

Comparison to State-of-The-Arts on Ms-COCO. In spectral graph theory, spectral
embedding captures the global structure and relationships of data by encoding
implicit relationships between features, including group clustering and
connectivity. Our approach integrates multi-level joint distillation of graph-
based vertices, edges, and spectral embedding, aiming to enable the student
model to transcend limitations of model capacity and structural differences,
thus fully understanding and accurately replicating the behavior of the
teacher model. In our research on object detection tasks on the MS-COCO
dataset, our method demonstrates consistent performance across various
detection architectures. As shown in Table 1, our training strategy
outperforms the simple feature mimicking of FitNet [1] by 7.9% AP in Faster
RCNN [9]-FPN [25], and exceeds the focus distillation strategies of FGFI [43],
FRS [53], GI [6], and FGD [48] by 0.9, 0.6, 0.8, and 0.3 AP respectively in
RetinaNet [26]. By focusing on the structural relationships between foreground
and background areas, our method surpasses the high-precision localization
strategy LD [52] by 1.4 $\text{AP}_{75}$ in GFL [24]. MasKD [17] employs
masking to extract pixel-level relational embeddings from feature maps, yet
its performance on FCOS [40] still falls 0.5 AP short of our method.

Experimental results on Pascal VOC are summarized in the Appendix C.1

Various student backbones. As shown in Table 2, we opted for the compact one-
stage detectors GFL [24] as the embeddable frameworks, utilizing ResNet101 as
the teacher backbone network. We reported the competitive advantage of our
method in enhancing the performance of lightweight student networks such as
ResNet34, ResNet18, and MobileNetv2, in comparison to CM [46]. Notably, our
proposed technique consistently improved the student networks’ AP by
approximately 4%. Under heterogeneous teacher-student settings (ResNet101 to
MobileNetv2), the improvements in $\text{AP}_{S}$, $\text{AP}_{M}$, and
$\text{AP}_{L}$ relative to CM [46] are around 6%, 5%, and 2%, respectively.

Table 2: Our algorithm has been evaluated under various teacher-student architectural configurations, including heterogeneous MobileNetv2 and homogenous ResNet34 and ResNet18 architectures. The teacher model utilized throughout the assessments is ResNet101, with the MS-COCO val2017 dataset serving as the testing benchmark. Backbone | Method | AP | AP50 | AP75 | APS | APM | APL
---|---|---|---|---|---|---|---
| Student | 34.5 | 51.3 | 37.1 | 18.9 | 37.6 | 45.4
MobileNetV2 | CM [46] | 35.7 | 52.3 | 38.8 | 19.5 | 38.9 | 47.1
| Ours | 37.4 | 55.9 | 40.1 | 22.4 | 40.4 | 48.5
| Student | 35.5 | 52.7 | 37.9 | 18.9 | 38.6 | 46.8
ResNet18 | CM [46] | 37.6 | 54.6 | 40.6 | 20.3 | 41.0 | 49.7
| Ours | 38.4 | 57.1 | 42.3 | 22.9 | 41.6 | 51.2
| Student | 39.1 | 56.7 | 42.3 | 21.5 | 43.1 | 51.9
ResNet34 | CM [46] | 40.8 | 58.4 | 44.3 | 22.8 | 44.9 | 53.8
| Ours | 41.3 | 59.8 | 45.6 | 24.2 | 45.1 | 54.6
| Student | 40.1 | 58.2 | 43.1 | 23.3 | 44.4 | 52.5
ResNet50 | CM [46] | 42.0 | 59.7 | 45.7 | 24.2 | 46.3 | 54.4
| Ours | 42.7 | 60.9 | 46.2 | 25.0 | 46.3 | 55.1

Generalization to classification task. Our method demonstrates minimal
coupling with object detection tasks, facilitating its seamless extension to
other visual tasks like image classification. We assess our method’s
competitive advantage relative to other approaches in both homogeneous and
heterogeneous (Table 3) environments. Notably, despite significant disparities
in the topological structures of intermediate layer outputs between teacher
and student networks in heterogeneous environments, our method still empowers
students to surpass teachers in certain scenarios. For instance, the teacher
ResNet-110x2 lags behind the distilled student ShuffleNetV2 by 1.14 in top-1
accuracy. Furthermore, employing the method we introduce with WRN-40-2 results
in a top-1 accuracy surpassing that of the teacher ResNet-32x4 by 0.99. As
illustrated in Fig. 3, we have randomly selected five categories from the
CIFAR-100 dataset and visualized the classification features using t-SNE, with
each color representing a distinct category. The teacher network is comprised
of ResNet110x2, in contrast to the student network which utilizes
ShuffleNetV2.

Table 3: Performance in classification task on the CIFAR-100 dataset. Top-1
accuracy is adopted as the evaluation metric.

Method | Homogeneous architecture | Heterogeneous architecture
---|---|---
Teacher | WRN-40-2 | ResNet-32x4 | VGG13 | ResNet-110x2 | WRN-40-2 | ResNet-32x4
76.31 | 79.42 | 74.64 | 78.18 | 76.31 | 79.42
Student | WRN-40-1 | ResNet-8x4 | VGG-8 | ShuffleNetV2 | MobileNetV2 | WRN-40-2
71.92 | 73.09 | 70.46 | 72.60 | 65.43 | 76.35
KD [16] | 74.12 | 74.42 | 72.98 | 76.05 | 69.07 | 77.70
FitNet [1] | 74.17 | 74.32 | 71.02 | 76.40 | 68.64 | 77.69
AT [50] | 74.67 | 75.07 | 71.90 | 76.84 | 68.62 | 78.45
SP [41] | 73.90 | 74.29 | 73.12 | 76.60 | 68.73 | 78.34
CRD [39] | 74.80 | 75.59 | 73.94 | 76.67 | 70.28 | 78.15
SimKD [2] | 75.56 | 78.08 | 73.76 | 78.25 | 70.71 | 79.29
Ours | 75.99 | 78.89 | 74.57 | 79.32 | 71.18 | 80.41

Figure 3: We visualized the classification effects of classifiers using our
method and the SimKD [2] method through t-SNE dimensionality reduction. We
also displayed the classification effects of the teacher, student, and
untrained models. Multi-level joint alignment based on inter-channel
relationships indirectly improves the classification performance between
sample points by enhancing the representation of internal channel
relationships in the network. We attribute this enhancement to spectral
embedding technology, which boosts the student model’s understanding of
complex data structures, thereby increasing intra-class compactness and inter-
class separability in classification.

###  3.2 Analyses

Effects of the adaptation layer. The role of the projection layer is to align
the outputs of the teacher and student at the same semantic level, aiming to
reduce additional noise introduced during feature tensor alignment. We
compared two types of linear projection: point convolution and 3x3
convolution. The hyperplane projection of the activation layer output and the
two types of transfer layers embedded in the network structure, including the
regression-based transfer proposed in FitNet [1] and the attention-based
transfer in AT [50], were also evaluated. Since the parameters of the
convolution layers are randomly assigned, we compared the average AP obtained
from 5 experiments. We use Faster RCNN [9] with FPN [25] as the detector for
benchmark testing. The results are shown in Table 4. We found that the noise
suppression effect of the 3x3 convolution in our proposed method is
significantly better than other types of projection layers.

Table 4: The table examines the influence of different adaptation layers on
the performance of a student model ResNet50, with ResNet101 as the teacher
model, on the MS-COCO val2017 dataset. Performance metrics are analyzed in
relation to various transformer types and their positions within the network
architecture. † describing the strategy used by the original method.

Detector | Method | AP | AP50 | AP75 | Transformer | Feature position
---|---|---|---|---|---|---
Faster RCNN [9]-FPN [25] | Teacher | 42.0 | 62.5 | 45.9 | - | -
Student | 37.4 | 58.4 | 40.7 | - | -
FitNet | 38.9 | 59.4 | 41.7 | 1$\times$1 conv | Mid layer
FitNet | 38.5 | 59.7 | 41.8 | 3$\times$3 conv | Mid layer
FitNet [1] | 38.8 | 59.6 | 41.8 |  †regressor | Mid layer
AB [15] | 39.1 | 59.9 | 42.0 | Pre-ReLu | End of group
AT [50] | 39.6 | 60.2 | 42.1 | Attention | End of group
Ours | 41.5 | 61.3 | 45.4 | 1$\times$1 conv | Mid layer
Ours | 41.9 | 61.8 | 45.5 | 3$\times$3 conv | Mid layer
Ours | 41.6 | 61.1 | 45.0 | Regressor | Mid layer
Ours | 41.3 | 61.7 | 44.9 | Pre-ReLu | End of group
Ours | 41.7 | 61.5 | 45.3 | Attention | End of group

Figure 4: RetinaNet R-50 AP score for various box sizes. S denotes the types
of errors made by the student. D represents the types of errors after applying
our method.

Error types. As shown in Fig. 4, we reanalyzed the mechanism of the proposed
mothod from the perspective of error components. The horizontal axis of the
bar chart represents the errors disregarding the corresponding categories,
while the vertical axis represents AP. C75 and C50 represent different
Intersection over Union (IoU) thresholds, Loc corresponds to $\text{AP}_{10}$,
Sim and Oth respectively indicate confusions with similar and other
categories, and BG denotes background false positives. Our method enhances the
discriminative capabilities of the student, explicitly evidenced by the
significant reduction in the proportion of classification errors (Sim and Oth)
in False Positives.

Combination with response-based distillation Method. We investigated the
integration of response-based distillation methods into the proposed
framework. As shown in Table 5, incorporating $\text{KD}_{cls}$ resulted in a
gain of 1.2 $\text{AP}_{S}$. Interestingly, contrary to intuition, the
effectiveness of incorporating category regression distillation
$\text{KD}_{cls}$ was significantly better than that of incorporating bounding
box prediction regression $\text{KD}_{loc}$. We also observed that the fused
method’s AP was lower than that before fusion, suggesting conflicting learning
objectives between intermediate and output layers leading to a decrease in
distillation performance.

Table 5: Our method is combined with a response-based distillation approaches.
“cls” indicates category prediction, and “loc” corresponds to bounding box
prediction. The teacher network is ResNet101, and the student network is
ResNet50. The dataset employed for evaluation is MS-COCO.

Detector | Method | AP | AP50 | AP75 | APS | APM | APL
---|---|---|---|---|---|---|---
Faster RCNN [9]-FPN [25] | Teacher | 42.0 | 62.5 | 45.9 | 25.2 | 45.6 | 54.6
Student | 37.4 | 58.4 | 40.7 | 21.8 | 41.0 | 47.8
Ours | 41.9 | 61.8 | 45.5 | 24.1 | 44.6 | 54.0
$+$ KDcls | 41.8 | 61.4 | 46.2 | 25.3 | 44.1 | 52.6
$+$ KDloc | 41.4 | 61.1 | 45.3 | 23.8 | 44.3 | 53.3
$+$ KDcls $+$ KDloc | 41.8 | 61.9 | 45.2 | 24.3 | 44.5 | 53.7

Hyperparameters sensitivity. As shown in Fig. 5, we assess the sensitivity of
three hyperparameters using GFL [24] detector. The model shows insensitivity
to adjustments in $\alpha$ but shows a notable change in AP with adjustments
to $\gamma$, performing optimally under a relatively balanced hyperparameter
setting. We set $\alpha=\beta=\gamma=1$ to ensure the repeatability of the
experiments.

Table 6: Employing the Faster RCNN [9]-FPN [25] detector within a ResNet101-50 teacher-student architecture on COCO. LV denotes vertex loss, LE edge loss, and LS spectral embedding loss. We incorporate attention masks by default. Loss | Result
---|---
LV | LE | LS | AP | AP50 | AP75
✗ | ✗ | ✗ | 37.4 | 58.4 | 40.7
✓ | ✗ | ✗ | 38.6 | 59.8 | 41.9
✗ | ✓ | ✗ | 37.8 | 58.9 | 43.2
✗ | ✗ | ✓ | 41.5 | 61.6 | 45.1
✗ | ✓ | ✓ | 41.6 | 61.4 | 45.3
✓ | ✓ | ✗ | 39.7 | 59.2 | 41.4
✓ | ✗ | ✓ | 40.3 | 61.1 | 44.9
✓ | ✓ | ✓ | 41.9 | 61.8 | 45.5

Ablation study. We investigated the contributions of various components in our
proposed method to the distillation performance, including vertex loss, edge
loss, and spectral embedding loss. As shown in Table 6, we discovered that the
combination of edges and SE (Spectral Embedding) vertices performs better than
distilling vertices alone, indicating that various features exhibit chaotic
relationships. The combination of vertices and edges is less effective than
the combination of vertices and SE, suggesting that the student model may
struggle to establish connections between the spatial topology of local
features and the overall topology through surface mimicry alone. Employing all
four components simultaneously yields better results than other variants,
demonstrating that these components are orthogonal to each other and each is
indispensable. We conducted ablation studies on two types of attention masks,
with experimental results in the Appendix C.2.

(a) $\alpha$ for vertex loss.

(b) $\beta$ for edge loss.

(c) $\gamma$ for SE loss.

Figure 5:  For each hyperparameter, we observed its impact trends on three
different student networks.

##  4 Conclusion

In summary, We innovatively apply spectral embedding (SE) technology to
precisely extract structural knowledge from channels relational graph. By
using attention masks to guide the vertices and edges of the relational graph,
and combining this with spectral embedding, we achieve multi-level fine
alignment. Our framework not only surpasses traditional feature distillation
methods in efficiency but also achieves unparalleled performance across
benchmark datasets.

## References

  * [1] Romero Adriana, Ballas Nicolas, K Samira Ebrahimi, Chassang Antoine, Gatta Carlo, and Bengio Yoshua.  Fitnets: Hints for thin deep nets.  Proc. ICLR, 2(3):1, 2015.
  * [2] Defang Chen, Jian-Ping Mei, Hailin Zhang, Can Wang, Yan Feng, and Chun Chen.  Knowledge distillation with the reused teacher classifier.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 11933–11942, 2022.
  * [3] Guobin Chen, Wongun Choi, Xiang Yu, Tony Han, and Manmohan Chandraker.  Learning efficient object detection models with knowledge distillation.  Advances in neural information processing systems, 30, 2017.
  * [4] Pengguang Chen, Shu Liu, Hengshuang Zhao, and Jiaya Jia.  Distilling knowledge via knowledge review.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5008–5017, 2021.
  * [5] Yixin Chen, Pengguang Chen, Shu Liu, Liwei Wang, and Jiaya Jia.  Deep structured instance graph for distilling object detectors.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 4359–4368, 2021.
  * [6] Xing Dai, Zeren Jiang, Zhao Wu, Yiping Bao, Zhicheng Wang, Si Liu, and Erjin Zhou.  General instance distillation for object detection.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 7842–7851, 2021.
  * [7] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al.  An image is worth 16x16 words: Transformers for image recognition at scale.  arXiv preprint arXiv:2010.11929, 2020.
  * [8] Mark Everingham, Luc Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman.  The pascal visual object classes (voc) challenge.  International journal of computer vision, 88:303–338, 2010.
  * [9] RCNN Faster.  Towards real-time object detection with region proposal networks.  Advances in neural information processing systems, 9199(10.5555):2969239–2969250, 2015.
  * [10] Ross Girshick.  Fast r-cnn in proceedings of the ieee international conference on computer vision (pp. 1440–1448).  Piscataway, NJ: IEEE.[Google Scholar], 2, 2015.
  * [11] Ross Girshick, Jeff Donahue, Trevor Darrell, and Jitendra Malik.  Rich feature hierarchies for accurate object detection and semantic segmentation.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 580–587, 2014.
  * [12] Jianping Gou, Baosheng Yu, Stephen J Maybank, and Dacheng Tao.  Knowledge distillation: A survey.  International Journal of Computer Vision, 129(6):1789–1819, 2021.
  * [13] Jianyuan Guo, Kai Han, Yunhe Wang, Han Wu, Xinghao Chen, Chunjing Xu, and Chang Xu.  Distilling object detectors via decoupled features.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2154–2164, 2021.
  * [14] Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick.  Mask r-cnn.  In Proceedings of the IEEE international conference on computer vision, pages 2961–2969, 2017.
  * [15] Byeongho Heo, Minsik Lee, Sangdoo Yun, and Jin Young Choi.  Knowledge transfer via distillation of activation boundaries formed by hidden neurons.  In Proceedings of the AAAI conference on artificial intelligence, volume 33, pages 3779–3787, 2019.
  * [16] Geoffrey Hinton, Oriol Vinyals, and Jeff Dean.  Distilling the knowledge in a neural network.  arXiv preprint arXiv:1503.02531, 2015.
  * [17] Tao Huang, Yuan Zhang, Shan You, Fei Wang, Chen Qian, Jian Cao, and Chang Xu.  Masked distillation with receptive tokens.  arXiv preprint arXiv:2205.14589, 2022.
  * [18] Xiaoqi Jiao, Yichun Yin, Lifeng Shang, Xin Jiang, Xiao Chen, Linlin Li, Fang Wang, and Qun Liu.  Tinybert: Distilling bert for natural language understanding.  arXiv preprint arXiv:1909.10351, 2019.
  * [19] Ying Jin, Jiaqi Wang, and Dahua Lin.  Multi-level logit distillation.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 24276–24285, 2023.
  * [20] Jangho Kim, Minsung Hyun, Inseop Chung, and Nojun Kwak.  Feature fusion for online mutual knowledge distillation.  In 2020 25th International Conference on Pattern Recognition (ICPR), pages 4619–4625. IEEE, 2021.
  * [21] Alex Krizhevsky, Geoffrey Hinton, et al.  Learning multiple layers of features from tiny images.  2009\.
  * [22] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton.  Imagenet classification with deep convolutional neural networks.  Advances in neural information processing systems, 25, 2012.
  * [23] Quanquan Li, Shengying Jin, and Junjie Yan.  Mimicking very efficient network for object detection.  In Proceedings of the ieee conference on computer vision and pattern recognition, pages 6356–6364, 2017.
  * [24] Xiang Li, Wenhai Wang, Lijun Wu, Shuo Chen, Xiaolin Hu, Jun Li, Jinhui Tang, and Jian Yang.  Generalized focal loss: Learning qualified and distributed bounding boxes for dense object detection.  Advances in Neural Information Processing Systems, 33:21002–21012, 2020.
  * [25] Tsung-Yi Lin, Piotr Dollár, Ross Girshick, Kaiming He, Bharath Hariharan, and Serge Belongie.  Feature pyramid networks for object detection.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2117–2125, 2017.
  * [26] Tsung-Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár.  Focal loss for dense object detection.  In Proceedings of the IEEE international conference on computer vision, pages 2980–2988, 2017.
  * [27] Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick.  Microsoft coco: Common objects in context.  In Computer Vision–ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13, pages 740–755. Springer, 2014.
  * [28] Li Liu, Qingle Huang, Sihao Lin, Hongwei Xie, Bing Wang, Xiaojun Chang, and Xiaodan Liang.  Exploring inter-channel correlation for diversity-preserved knowledge distillation.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 8271–8280, 2021.
  * [29] Yang Liu, Peng Sun, Nickolas Wergeles, and Yi Shang.  A survey and performance evaluation of deep learning methods for small object detection.  Expert Systems with Applications, 172:114602, 2021.
  * [30] Yifan Liu, Ke Chen, Chris Liu, Zengchang Qin, Zhenbo Luo, and Jingdong Wang.  Structured knowledge distillation for semantic segmentation.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 2604–2613, 2019.
  * [31] Yufan Liu, Jiajiong Cao, Bing Li, Chunfeng Yuan, Weiming Hu, Yangxi Li, and Yunqiang Duan.  Knowledge distillation via instance relationship graph.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7096–7104, 2019.
  * [32] Bin Luo, Richard C Wilson, and Edwin R Hancock.  Spectral embedding of graphs.  Pattern recognition, 36(10):2213–2230, 2003.
  * [33] Javier Maroto, Guillermo Ortiz-Jiménez, and Pascal Frossard.  On the benefits of knowledge distillation for adversarial robustness.  arXiv preprint arXiv:2203.07159, 2022.
  * [34] Hideki Oki, Motoshi Abe, Jyunichi Miyao, and Takio Kurita.  Triplet loss for knowledge distillation.  In 2020 International Joint Conference on Neural Networks (IJCNN), pages 1–7. IEEE, 2020.
  * [35] Jiangmiao Pang, Kai Chen, Jianping Shi, Huajun Feng, Wanli Ouyang, and Dahua Lin.  Libra r-cnn: Towards balanced learning for object detection.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 821–830, 2019.
  * [36] Dian Qin, Jia-Jun Bu, Zhe Liu, Xin Shen, Sheng Zhou, Jing-Jun Gu, Zhi-Hua Wang, Lei Wu, and Hui-Fen Dai.  Efficient medical image segmentation based on knowledge distillation.  IEEE Transactions on Medical Imaging, 40(12):3820–3831, 2021.
  * [37] Rulin Shao, Jinfeng Yi, Pin-Yu Chen, and Cho-Jui Hsieh.  How and when adversarial robustness transfers in knowledge distillation?  arXiv preprint arXiv:2110.12072, 2021.
  * [38] Ruoyu Sun, Fuhui Tang, Xiaopeng Zhang, Hongkai Xiong, and Qi Tian.  Distilling object detectors with task adaptive regularization.  arXiv preprint arXiv:2006.13108, 2020.
  * [39] Yonglong Tian, Dilip Krishnan, and Phillip Isola.  Contrastive representation distillation.  arXiv preprint arXiv:1910.10699, 2019.
  * [40] Zhi Tian, Xiangxiang Chu, Xiaoming Wang, Xiaolin Wei, and Chunhua Shen.  Fully convolutional one-stage 3d object detection on lidar range images.  Advances in Neural Information Processing Systems, 35:34899–34911, 2022.
  * [41] Frederick Tung and Greg Mori.  Similarity-preserving knowledge distillation.  In Proceedings of the IEEE/CVF international conference on computer vision, pages 1365–1374, 2019.
  * [42] Luting Wang, Xiaojie Li, Yue Liao, Zeren Jiang, Jianlong Wu, Fei Wang, Chen Qian, and Si Liu.  Head: Hetero-assists distillation for heterogeneous object detectors.  In European Conference on Computer Vision, pages 314–331. Springer, 2022.
  * [43] Tao Wang, Li Yuan, Xiaopeng Zhang, and Jiashi Feng.  Distilling object detectors with fine-grained feature imitation.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4933–4942, 2019.
  * [44] Tiancheng Wen, Shenqi Lai, and Xueming Qian.  Preparing lessons: Improve knowledge distillation with better supervision.  Neurocomputing, 454:25–33, 2021.
  * [45] Pingfan Wu, Jiayu Zhang, Han Sun, and Ningzhong Liu.  Regional filtering distillation for object detection.  Machine Vision and Applications, 35(2):24, 2024.
  * [46] Aijia Yang, Sihao Lin, Chung-Hsing Yeh, Minglei Shu, Yi Yang, and Xiaojun Chang.  Context matters: Distilling knowledge graph for enhanced object detection.  IEEE Transactions on Multimedia, 2023.
  * [47] Chuanguang Yang, Helong Zhou, Zhulin An, Xue Jiang, Yongjun Xu, and Qian Zhang.  Cross-image relational knowledge distillation for semantic segmentation.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12319–12328, 2022.
  * [48] Zhendong Yang, Zhe Li, Xiaohu Jiang, Yuan Gong, Zehuan Yuan, Danpei Zhao, and Chun Yuan.  Focal and global knowledge distillation for detectors.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4643–4652, 2022.
  * [49] Junho Yim, Donggyu Joo, Jihoon Bae, and Junmo Kim.  A gift from knowledge distillation: Fast optimization, network minimization and transfer learning.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4133–4141, 2017.
  * [50] Sergey Zagoruyko and Nikos Komodakis.  Paying more attention to attention: Improving the performance of convolutional neural networks via attention transfer.  arXiv preprint arXiv:1612.03928, 2016.
  * [51] Linfeng Zhang and Kaisheng Ma.  Structured knowledge distillation for accurate and efficient object detection.  IEEE Transactions on Pattern Analysis and Machine Intelligence, 2023.
  * [52] Zhaohui Zheng, Rongguang Ye, Qibin Hou, Dongwei Ren, Ping Wang, Wangmeng Zuo, and Ming-Ming Cheng.  Localization distillation for object detection.  IEEE Transactions on Pattern Analysis and Machine Intelligence, 2023.
  * [53] Du Zhixing, Rui Zhang, Ming Chang, Shaoli Liu, Tianshi Chen, Yunji Chen, et al.  Distilling object detectors with feature richness.  Advances in Neural Information Processing Systems, 34:5213–5224, 2021.
  * [54] Sheng Zhou, Yucheng Wang, Defang Chen, Jiawei Chen, Xin Wang, Can Wang, and Jiajun Bu.  Distilling holistic knowledge with graph neural networks.  In Proceedings of the IEEE/CVF international conference on computer vision, pages 10387–10396, 2021.
  * [55] Bojia Zi, Shihao Zhao, Xingjun Ma, and Yu-Gang Jiang.  Revisiting adversarial robustness distillation: Robust soft labels make student better.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 16443–16452, 2021.

##  Appendix A Related Work

###  A.1 Knowledge Distillation

Knowledge distillation is a model compression technique that has been applied
across various tasks and domains, including image classification [16],object
detection [51, 43, 3, 45], natural language processing [18], semantic
segmentation [36, 30], and more. In recent years, it has also been explored
for enhancing model generalization capabilities, reducing model reliance on
data biases, and improving model performance on specific tasks, such as
learning with limited labeled data. Hinton [16]and others [39, 34, 44] posited
that the logit outputs of a teacher model, relative to one-hot labels, contain
rich interpretive information. Minimizing the KL divergence difference between
student and teacher outputs can help the student approximate the teacher’s
decision boundary. As research progressed, researchers discovered that the
rich feature representations in the intermediate layers of the teacher model
contain additional information useful for the learning task. FitNet [1]
proposed treating all features equally in distillation. DeFeat [13] introduced
hyperparameters to weight foreground and background features to overcome the
adverse effects of uneven positive and negative regions. To match the semantic
levels between teacher and student feature layers, AT [50] used attention maps
as carriers of knowledge transfer and compared the distillation performance of
activation-based and gradient-based attention maps. Guobin Chen [3] were
pioneers in introducing knowledge distillation techniques to object detection
tasks. Relational distillation goes further than feature distillation by
exploring interactions and dependencies between instances. The intrinsic
structure of the data, as a more abstract and general form of knowledge, aids
the student in mimicking the teacher’s generalization capabilities. SP [41]
aligned pair-wise similarities between teacher and student features, encoding
relational knowledge into an adjacency matrix refined into the student model.
Sheng Zhou et al. unified the representation of individual knowledge in
Euclidean metric spaces and sample correlations as a graph network based on
vertices and edges. Total mutual information was used as an additional
supervisory signal to guide the training of the student model. Yufan Liu et
al. [31], by constructing instance relation graphs to simulate the knowledge
in network layers, proposed the IRG transformation to smooth the transition
between feature layers, showing strong robustness to different network
architectures.

FSP [49] addresses the pairwise similarity between feature layers. Li Liu and
others [28] explored the contribution of homogeneity and diversity among
channels to visual tasks, proposing the ICKD method, which is based on the
correlation between feature channels. Pixel-level relational distillation, as
exemplified by CIRKD, is primarily used for segmentation tasks [47].

###  A.2 Object detection

Compared to classification, detection provides an understanding of the image’s
foreground and background [29].The landscape of object detection has been
predominantly shaped by deep learning techniques, leveraging Convolutional
Neural Networks (CNNs) to achieve remarkable feats [22]. These methods are
broadly categorized into two distinct paradigms: two-stage and one-stage
detectors.Two-stage Detectors: Pioneered by the R-CNN family, including Fast
R-CNN and Faster RCNN [11, 10, 9, 14, 35], this paradigm initially generates
region proposals to highlight areas of interest and subsequently performs
classification and bounding box regression on each proposal. One-stage anchor-
based detectors, such as RetinaNet [26], predict class scores and bounding
boxes directly from the image, utilizing predefined anchor boxes to
accommodate a variety of scales and aspect ratios. Meanwhile, anchor-free
detectors like FCOS [40] eliminate the necessity for anchor boxes by directly
predicting object key points or centers. FPN [25] constructs a multi-level
feature pyramid, effectively integrating spatial information across different
scales through a top-down pathway and lateral connections, thereby achieving
precise detection of objects of varying sizes.

###  A.3 Knowledge Distillation Based on Graphs

By constructing relational graphs, we provide an intuitive understanding of
abstract relational knowledge distillation. Vertices on the graph represent
feature information, while edges depict the interactions and dependencies
between pieces of knowledge. IRG [31] models the relationships between
instances at each layer as a graph. Yang (CM) [46] utilizes a transformer to
perform message passing and aggregation operations on the graph, achieving a
comprehensive representation of vertices that captures both the local
neighborhood structure and the global graph structure. Chen et al. [5] modeled
the relationships between proposal-level instance features within a Region
Proposal Network as a structured graph, conveying both local and global
feature information. Holistic KD [54] posits that individual knowledge and
relational knowledge offer mutually orthogonal feature representations.
Integrating both into a unified graph-based embedding, this encoded knowledge
representation is then transferred to the student network. In our method, we
do not introduce any additional auxiliary modules; instead, by extracting the
spectral embedding of the relational graph, we facilitate the propagation and
aggregation of representations from vertices and their highly similar
neighboring regions.

##  Appendix B Experiments Setting

MS-COCO is a widely used large-scale object detection dataset that includes
120K training images and 5K validation images across 80 object categories and
91 scene categories.

Pascal VOC dataset encompasses 20 object categories, including people,
animals, vehicles, and indoor objects, among others.

CIFAR-100 dataset, designed for classification tasks, comprises 60,000 color
images with a resolution of 32x32 pixels. It includes 50,000 training images
and 10,000 validation images. The images are categorized into 100 classes,
which are further grouped into 20 superclasses.

We test the performance of our method on various datasets to demonstrate its
generalization ability.

We conducted a comprehensive performance comparison between our proposed
method and other state-of-the-art (SOTA) methods on the COCO dataset.
Specifically, we detailed the performance of our method with various
mainstream detectors, including two-stage detectors such as Faster RCNN [9]
with FPN [25], one-stage anchor-based detectors like RetinaNet [26], and one-
stage anchor-free detectors such as FCOS [40], as well as other detectors
including GFL [24]. The evaluation metrics we established on the COCO dataset
include AP, $\text{AP}_{50}$, $\text{AP}_{75}$, $\text{AP}_{S}$,
$\text{AP}_{M}$, and $\text{AP}_{L}$. We also designed experiments involving
both homogeneous and heterogeneous teacher-student network architectures, such
as T-ResNet101 with S-ResNet50 (homogeneous) and T-ResNet101 with
S-MobileNetV2 (heterogeneous). To demonstrate the versatility of our method,
we evaluated its performance on the VOC2007 testing set, which consists of
4952 images, across three typical detection architectures: Faster RCNN [9],
RetinaNet [26], and FCOS [40]. Specifically, we assessed the AP,
$\text{AP}_{50}$, and $\text{AP}_{75}$ metrics for each architecture. For
classification tasks, we utilize top-1 accuracy as the primary criterion to
assess model performance. Similarly, we have conducted experiments to evaluate
the classification accuracy of our method across a variety of network
architectures, including ResNet, Wide Residual Networks (WRN), VGG,
ShuffleNetV2, and MobileNetV2.

We employed the SGD optimizer for updating the model parameters, with momentum
configured at 0.9, an initial learning rate set to 0.01, batch size determined
at 16, and weight decay fixed at 0.01. All training utilizes AMP (Automatic
Mixed Precision) optimization to accelerate the training process, and for the
teacher networks across all datasets, we employ a 2x schedule training plan.
The training was executed on eight Tesla 32G V100 GPUs on the AutoDL cloud
platform. We have summarized the computational costs of our method when
applied to various backbones within the GFL [24] framework, as detailed in
Table 7. Our approach utilizes the hyperparameters $\alpha$, $\beta$ and
$\gamma$ to achieve a balance between graph loss and spectral embedding loss.
Without prior specification, we treat all supervision signals with equal
importance. Due to the constraints of paper length, not all results could be
fully displayed.

Table 7: Training costs Backbone | Training hours | Schedule | FLOP | #Param.
---|---|---|---|---
ResNet101 | 14.3h | 2x | 297.6G | 51.1M
MobileNetV2 | 10.7h | 1x | 130.2G | 10.4M
ResNet18 | 8.3h | 1x | 162.8G | 19.1M
ResNet34 | 9.1h | 1x | 202.7G | 29.0M
ResNet50 | 10.5h | 1x | 214.7G | 32.1M

##  Appendix C More Experiments

###  C.1 Pascal VOC

As demonstrated in Table 8, our experiments on the VOC2007 test set revealed
the performance of our method across three typical detection architectures
(Faster RCNN [9], RetinaNet [26], and FCOS [40]). We maintained ResNet101 as
the guiding teacher for the student ResNet50. Our method brings notable AP
gains, with student models generally outperforming teacher models. For
example, in the Faster RCNN [9] setting, the distilled student model achieves
a lead of 1.9 AP over the teacher network. In the case of lightweight one-
stage detectors, our method particularly enhances localization accuracy. For
instance, the student model of RetinaNet [26] exhibits an 8.7% improvement in
$\text{AP}_{75}$, surpassing GI [6] by 4.6%. We visualized the attention
distribution of RetinaNet [26] in the form of a heatmap. As shown in Figure 7,
we have compared the results of SE, GI [6], students, and teachers side by
side. Our approach is most closely aligned with that of the teacher.

Table 8: Performance on VOC. FGFI is only applicable to two-stage and anchor-
based scenarios.

Method | Faster RCNN [9] Res101-50 | RetinaNet [26] Res101-50 | FCOS [40] Res101-50
---|---|---|---
AP | AP50 | AP75 |  AP | AP50 | AP75 |  AP | AP50 | AP75
Teacher |  56.3 | 82.7 | 62.6 |  58.2 | 82.0 | 63.0 |  58.4 | 81.6 | 64.3
Student |  54.2 | 82.1 | 59.9 |  56.1 | 80.9 | 60.7 |  56.1 | 80.0 | 61.6
FitNet [1] |  55.0 | 82.2 | 61.2 |  56.4 | 81.7 | 61.7 |  57.0 | 80.3 | 62.1
FGFI [43] |  55.3 | 82.1 | 61.1 |  55.6 | 81.4 | 60.5 |  — | — | —
GI [6] |  56.5 | 82.6 | 61.6 |  57.9 | 82.0 | 63.2 |  58.4 | 81.3 | 62.9
Ours |  58.2 | 83.6 | 63.9 |  59.9 | 83.1 | 65.8 |  59.5 | 83.2 | 65.6

###  C.2 Attention Mechanism Ablation Study

Table 9: Conducting ablation experiments on attention mechanisms with the Faster RCNN-FPN detector in a ResNet101-50 teacher-student architecture on the COCO dataset. Attention | Result
---|---
Vertex | Edge | AP | AP50 | AP75
✗ | ✗ | 40.9 | 60.7 | 43.8
✓ | ✗ | 41.4 | 61.6 | 45.1
✗ | ✓ | 41.2 | 60.9 | 44.7
✓ | ✓ | 41.9 | 61.8 | 45.5

As shown in the Table 9. When only vertex attention is enabled, the model sees
an improvement across all three metrics: AP increases to 41.4, AP50 rises to
61.6, and AP75 grows to 45.1. With just the edge attention activated, there’s
a slight bump in AP to 41.2, AP50 remains competitive at 60.9, and there’s a
more noticeable boost in AP75, reaching 44.7. However, when both attention
mechanisms are employed, the model achieves the best outcomes on all metrics:
AP further advances to 41.9, AP50 climbs to 61.8, and AP75 ascends to 45.5.
The integration of both vertex and edge attention mechanisms contributes to
the most substantial performance gains, underscoring their complementary roles
in enhancing the model’s capability to discern and process spatial
relationships within the data.

###  C.3 Sensitivity Analysis of the Hyperparameter N

In our methodology, the chosen detector is GFL, and the teacher network
employed is ResNet101. In spectral embedding, the number of eigenvectors of
the Laplacian matrix can reach its dimensionality, corresponding to the number
of vertices in the graph. This is closely related to the spectral properties
of the Laplacian, encoding the graph’s topology. As illustrated in Figure 7,
the selection of eigenvectors (denoted as N/C, where N is the number of
selected eigenvectors and C is the number of channels) affects the performance
metric AP%, which stands for Average Precision Percentage. The graph
demonstrates how increasing the proportion of utilized eigenvectors relative
to the number of channels enhances the discriminative power of the resulting
spectral features.

Figure 6: The attention distribution of different types of detectors.

Figure 7: Sensitivity experiment curve of the hyperparameter N

##  Appendix D Limitations

One inherent limitation of our framework pertains to the subjective nature of
feature and channel selection for distillation. Our method necessitates
identifying specific features and channels that are deemed crucial for
effectively transferring knowledge from the teacher to the student model. This
selection process is inherently influenced by the criteria and strategies we
employ, which introduces a degree of subjectivity.