# Aligning in a Compact Space: Contrastive Knowledge Distillation between
Heterogeneous Architectures

Hongjun Wu1, Li Xiao1, Xingkuo Zhang1, Yining Miao1
1School of Artificial Intelligence, Beijing University of Posts and
Telecommunications.
{hongjun.wu, andrewxiao, dd_zxk<EMAIL_ADDRESS>Corresponding author.

###### Abstract

Knowledge distillation is commonly employed to compress neural networks,
reducing the inference costs and memory footprint. In the scenario of
homogenous architecture, feature-based methods have been widely validated for
their effectiveness. However, in scenarios where the teacher and student
models are of heterogeneous architectures, the inherent differences in feature
representation significantly degrade the performance of these methods. Recent
studies have highlighted that low-frequency components constitute the majority
of image features. Motivated by this, we propose a Low-Frequency Components-
based Contrastive Knowledge Distillation (LFCC) framework that significantly
enhances the performance of feature-based distillation between heterogeneous
architectures. Specifically, we designe a set of multi-scale low-pass filters
to extract the low-frequency components of intermediate features from both the
teacher and student models, aligning them in a compact space to overcome
architectural disparities. Moreover, leveraging the intrinsic pairing
characteristic of the teacher-student framework, we design an innovative
sample-level contrastive learning framework that adeptly restructures the
constraints of within-sample feature similarity and between-sample feature
divergence into a contrastive learning task. This strategy enables the student
model to capitalize on intra-sample feature congruence while simultaneously
enhancing the discrimination of features among disparate samples.
Consequently, our LFCC framework accurately captures the commonalities in
feature representation across heterogeneous architectures. Extensive
evaluations and empirical analyses across three architectures (CNNs,
Transformers, and MLPs) demonstrate that LFCC achieves superior performance on
the challenging benchmarks of ImageNet-1K and CIFAR-100. All codes will be
publicly available.

##  1 Introduction

Knowledge distillation has emerged as an extensively utilized strategy for
model compression. It typically employs a teacher-student architecture, where
the student model is guided to mimic the teacher’s predictions[1, 2, 3, 4] or
feature representations[5, 6, 7, 8], thereby enhancing the performance of
compact models. Current methods predominantly focus on scenarios where both
the teacher and student models share the same architectural lineage, often
involving pairing a larger-scale teacher model with a smaller-scale student,
exemplified by transitions such as ResNet18 to ResNet34[9] or ViT-S to
ViT-G[10].

However, the performance ceilings of models vary due to the differing
scalability of the architectures. This limitation narrows the applicability of
knowledge distillation homogeneous architectures. For instance, it is
challenging for a student model from the MobileNet[11] family to find a
teacher model within its own lineage that can match the capabilities of
ViT-G[10]. Knowledge distillation in heterogeneous architectures addresses
this limitation by expanding the pool of potential teacher-student pairings.
While some research has ventured into exploring teacher-student combinations
in heterogeneous architectures[12, 13, 14, 4, 15], the field is still in its
early stages and holds substantial potential for further exploration and
advancement.

Existing knowledge distillation methods primarily employ logit[1, 2, 3, 4] or
intermediate feature maps[5, 6, 7, 8] as the medium for knowledge transfer.
The logit space serves as a task-specific shared domain, applicable to all
knowledge distillation tasks. In contrast to logit-based approaches, feature-
based methods transmit knowledge via aligning intermediate feature
representations point-to-point. While these methods often excel in knowledge
distillation with homogeneous architectures due to the intrinsic similarity of
feature representations, they face serious setback in heterogeneous
settings[4]. The substantial differences in meta-architecture, input format,
and spatial representation between heterogeneous models preclude the use of
simple similarity metrics for aligning stage-wise feature representations.
Moreover, mimicing the local details in feature representations may lead to
detrimental guidance. Therefore, extracting effective common knowledge from
the potential feature space of heterogeneous models remains a pressing issue.

Inspired by the effectiveness of low-frequency components in feature
representation[16, 17], we propose a Low-Frequency Components-based
Contrastive Knowledge Distillation (LFCC) framework for knowledge distillation
in heterogeneous architectures. The overarching strategy involves extracting
and condensing the low-frequency components of image features into an aligned
compact space. Specifically, we employ a multi-scale low-pass filter, and its
learnable derivative, to extract and compress the low-frequency components of
both teacher and student features into a compact categorical representation.
This approach mitigates the substantial representational disparity between
heterogeneous models, preventing detrimental guidance from spatial noise.
Harnessing the intrinsic pairing of the teacher-student framework, we design a
novel sample-level contrastive learning framework that transforms the feature-
based knowledge distillation into a sample-level contrastive learning. This
framework treats the representations of the student and teacher for the same
sample as positive pairs and contrasts them with other samples in the same
batch as negative pairs. By mapping positive pairs closer together and
negative pairs further apart, the student model not only benefits from the
intra-sample feature similarity but also extracts hidden knowledge from inter-
sample feature disparity. To entirely evaluate the proposed LFCC framework, we
conduct extensive experiments across 15 model combinations involving three
heterogeneous architectures. Our method surpasses the prior state-of-the-art
OFA-KD[4] approach, achieving up to a 0.83% and 4.19% improvement on
ImageNet-1K and CIFAR-100, respectively. Moreover, extensive and thorough
ablation experiments validate the effectiveness of the LFCC.

##  2 Related works

###  2.1 Knowledge distillation

The recent methods for knowledge distillation can be roughly categorized into
two aspects, i.e., logits-based methods and feature-based methods.

Logits-based knowledge distillation[18, 19, 20] enhances a smaller student
model by transferring soft targets from a larger teacher model. Hinton et
al.[1] introduce a method to soften logits in the softmax function, guiding
the students to acquire knowledge from the teacher’s prediction. Zhao et
al.[2] decomposes the logit-based distillation loss into the target domain and
the non-target domain, so as to learn the dark knowledge in the non-target
domain. Recognizing the challenge for students to learn from stronger
teachers, Huang et al.[3] suggest a relation-based loss to mitigate the strict
match requirement of KL divergence. Sun et al.[21] highlight issues with a
shared temperature in KD and propose logit normalization to allow students to
focus more on the teacher’s logits internal relations. Hao et al.[4] present a
KD framework for heterogeneous architectures, employing projectors to align
student features with teacher outputs.

Feature-based knowledge distillation[22, 23, 24] involves transferring
knowledge within the feature space. FitNet[5] introduces a method that
utilizes features from intermediate layers of a teacher model to guide the
learning process of the student model. Unlike conventional point-to-point
knowledge transfer methods, RKD[7] proposes learning the relations between
output feature maps. CRD[8] employs a contrastive learning approach to
maximize the lower bound of mutual information between teacher and student.
FSPNet[25] contends that the variations in features between each layer
represent learnable knowledge. CC[6], also based on relational knowledge
distillation, proposes a generalized kernel method based on Taylor series
expansion to better capture relationships between features. CAT-KD[26] propose
a highly interpretable knowledge distillation method based on class activation
map. Yang et al.[27] believes that directly mimicking the teacher is not
necessary to enhance the expressive power of student features. Instead, they
propose MGD which initially masks student features randomly and then employs a
simple block to generate complete teacher features from the masked features.
FGD[28] enhances the effectiveness of knowledge distillation in object
detection by employing focal distillation and global distillation.

###  2.2 Computer vision models

To comprehensively evaluate the generalization of our approach, we select
teacher-student combinations from three meta-architectures, including CNNs,
Transformers, and MLPs.

Convolutional Neural Networks (CNNs) are adopted at image processing due to
their inherent inductive biases, translational invariance, and the sliding
window mechanism. Researchers have introduced several enhancements to CNNs,
including residual connections[9], depthwise separable convolutions[11],
systematic model scaling[29], deep neural architecture search[30], and the use
of large convolutional kernels[31]. These innovations have reinforced CNNs’
pivotal role in the field of image processing, attributable to their superior
performance and robust generalization capabilities.

Transformers, initially designed for Natural Language Processing (NLP)
tasks[32], have increasingly been integrated into computer vision. The
introduction of Vision Transformers (ViT)[10] has catalyzed a significant
convergence between language and vision tasks, revolutionizing traditional
network architecture design. ViT processes images by segmenting them into a
series of patches for sequential input, outperforming classic CNN models.
Subsequent advancements, such as window attention[33] and pyramidal
architectures[34], have further expanded the capability of Transformer family.
Additionally, Transformers have been extensively applied in downstream vision
tasks, such as object detection[35], semantic segmentation[36] and image
generation[37].

Multi-Layer Perceptrons (MLPs), integral to the success of Transformers, have
garnered increased research attention. Given that MLPs are simple nonlinear
mappings that are less sensitive to local information, strategies are
essential to adapt them for image processing. These strategies include token
or channel mixing[38], affine transformations[39], gating mechanisms[40],
structural reparameterization[41], and spatial splitting[42].

##  3 Method

###  3.1 Recap of knowledge distillation

Knowledge distillation aims to transfer knowledge from a large-size teacher
model T to a small-size student model S. This process often uses logits and
features as carriers for knowledge transfer.

#### Logits-based Distillation.

Logits carry category distribution information that directly impacts
classification probabilities. Logit-based approach has proven to be an
effective strategy to circumvent the discrepancies in the spatial distribution
of features among heterogeneous models, as recent research has demonstrated.
Specifically, the output category distribution of the student model can be
made to approximate that of the teacher model, thereby improving the
performance of the student model.

$\mathcal{L}=\mathcal{L}_{\text{CE}}(\mathbf{p}^{s},y)+\lambda\mathcal{L}_{\text{KLD}}(\mathbf{p}^{s},\mathbf{p}^{t}),$
(1)

where $p^{s}$ and $p^{t}$ are the predicted logits of student model and
teacher model, respectively. $\mathcal{L}_{\text{KLD}}$ is the Kullback-
Leibler divergence loss function. $y$ is the one-hot groundtruth labels.
$\lambda$ is the hyperparameter adjusts the weight of soft label $p^{t}$.

#### Feature-based Distillation.

By minimizing the Euclidean distance between features, feature distillation
offers a finer-grained supervision for the student model through the
introduction of sample-level comparisons. Formally, the objective of feature
distillation approaches can be written as:

$\mathcal{L}=\mathcal{L}_{\text{CE}}(\mathbf{p}^{s},y)+\lambda_{\text{KLD}}\mathcal{L}_{\text{KLD}}(\mathbf{p}^{s},\mathbf{p}^{t})+\lambda_{\text{MSE}}\sum_{i=1}||\mathrm{F}^{t}_{i}-\phi(\mathrm{F}^{s}_{i})||^{2},$
(2)

where $F^{s}$ and $F^{t}$ are the features extracted from the i-th layer of
student model and teacher model. $\phi(\cdot)$ is the projector that transform
student feature to match the dimension of teacher feature.
$\lambda_{\text{KLD}}$ and $\lambda_{\text{MSE}}$ balance the logit loss and
feature loss.

Existing feature distillation methods simply align feature dimensions through
convolution or MLP modules, which is effective when both the teacher and
student models belong to the same meta-architecture. However, models with
different meta-architectures exhibit significant differences in terms of model
inputs, inductive biases, and spatial distributions, among others.
Consequently, enforcing complete feature-level alignment between the student
and teacher models may not be the optimal approach.

Figure 1: The pipeline of our LFCC framework. The overall architecture of the
LFCC framework is located in the top half, where we align the features output
from each stage via low-pass filters and contrastive learning. The detailed
process of the alignment module is located in the lower part.

###  3.2 Low-Frequency Components-based Contrastive Knowledge Distillation

We propose a novel feature-based distillation framework, named Low-Frequency
Components-based Contrastive Knowledge Distillation (LFCC). Figure 1 provides
an overview of the proposed LFCC.

#### Multi-scale Low-Pass Filters.

In light of the substantial discrepancies in feature representation between
heterogeneous teacher-student models, we propose to align features from a
frequency domain perspective. Low-frequency components account for the
majority of the content in an image and convey most of the semantic
information[16], while high-frequency information often contains noise.
Consequently, we employ low-pass filters to extract the low-frequency
components from image features.

We devise two low-pass filters tailored for the teacher and student branches,
each with differing levels of learnability. For the teacher branch, we adopte
a conventional average pooling as the low-pass filter and craft multi-scale
filters by adjusting different kernel sizes and strides across multiple groups
to accommodate varying cutoff frequencies. For the $m$-th group, we have

$\text{msLF}_{m}(f_{m})=\Lambda(\Theta_{k\times k}(f_{m}))$ (3)

where $\Theta_{k\times k}$ denotes the average pooling function with the
output size of $k\times k$. $\Lambda(\cdot)$ is the bilinear interpolation to
align the size of each group.

Meanwhile, for the student branch, we engineer a learnable low-pass filter
(LLF) composed of a multi-scale low-pass filter, a convolutional downsampling
module, and a depth-wise separable convolution (DSConv). The learnable low-
pass filter can be represent as

$\text{LLF}(f)=fc(\Phi_{3\times 3}([\Gamma_{s\times s}(f)),\text{msLF}(f)])$
(4)

where $\Phi_{3\times 3}$ is a depth-wise separable convolution with kernel
size $3\times 3$. $\Gamma_{s\times s}$ represents a convolution layer with
kernel size $s\times s$ and stride $s\times s$. $[\cdot]$ denotes concatenate
operation. $fc$ aligns the number of channels by linear projection. $f$
denotes the intermediate feature.

#### Aligning features in a compact space.

The models with heterogeneous architectures exhibit significant differences in
feature representation, which can lead to the introduction of detrimental
spatial noise through direct feature alignment, thereby degrading the
performance of the student model. We observe that mainstream image
classification models typically map the results of global average pooling or a
preset cls embedding into the logit space during the final stage, indicating
that the cls embedding encapsulates sufficient semantic information to
describe the image. In our framework, we use global feature representation to
measure the differences between the features of the two branches and achieve
feature alignment within a more compact space, thus avoiding the noise
introduced from the local details of the representation and reducing the
difficulty of feature alignment in heterogeneous settings.

#### Sample-level constrastive learning.

Traditional feature distillation methods focus on guiding the student model to
learn the teacher model’s feature representation via positive sample
comparison, overlooking discrimination between positive and negative samples.
The teacher-student framework, where sample pairs are formed by the outputs of
the teacher and student models, renders contrastive learning suitable for
knowledge distillation. We introduces sample-level contrastive learning in the
feature space, enabling fine-grained learning by comparing the features at
each stage of the teacher and student models.

For feature distillation, contrastive learning can be formulated as the
distance between the features of teacher and student:

$\mathcal{L}_{\text{CFD}}=\mathbb{E}_{\bm{x_{i}}\sim\mathcal{X}}\left[\text{dist}(t(x_{i}),s(x_{i}))-\mathbb{E}_{\bm{x_{j}}\sim\mathcal{X}}\text{dist}(t(x_{i}),s(x_{j}))\right]$
(5)

where $\mathcal{X}$ is the distribution of samples. $t(\cdot)$ and $s(\cdot)$
represent the teacher model and student model. Hence, teacher features
$t(x_{i})$ and student features $s(x_{i})$ constitute a positive pair, while
$t(x_{i})$ and $s(x_{j})$ form a negative pair. We set the distance metric as
$\mathcal{D}=-dist(a,b)$ to represent the similarity between $a$ and $b$. The
formula 5 can be rewritten as:

$\displaystyle\mathcal{L}_{\text{CFD}}=$
$\displaystyle\mathbb{E}_{\bm{x_{i}}\sim\mathcal{X}}\left[-(\mathcal{D}(t(x_{i}),s(x_{i}))-\mathbb{E}_{\bm{x_{j}}\sim\mathcal{X}}\mathcal{D}(t(x_{i}),s(x_{j}))\right]$
(6) $\displaystyle\simeq$
$\displaystyle\mathbb{E}_{\bm{x_{i}}\sim\mathcal{X}}-log\left[\frac{\exp(\mathcal{D}(t(x_{i}),s(x_{i})))}{\mathbb{E}_{\bm{x_{j}}\sim\mathcal{X}}\exp(\mathcal{D}(t(x_{i}),s(x_{j})))}\right]$

To facilitate computation, we confine the sample space for contrastive
learning to a single batch $B$. The goal is to learn with $t(x_{i})$ as the
anchor, distinguishing the positive sample $s(x_{i}),i=1,2,\cdots,B$ from
other student features $s(x_{j}),j=1,2,\cdots,i-1,i+1,\cdots,B$ within the
batch. Here we use cosine similarity as a distance metric, and at the $l$-th
stage, the loss function can be expressed as:

$\mathcal{L}_{\text{CFD}}(l)=-\frac{1}{B}\sum^{B}_{i=1}\mathbb{E}_{\bm{x_{i}}\sim\mathcal{X}}log\left[\frac{\exp(t_{l}(x_{i})\cdot
s_{l}(x_{i})/\tau)}{\sum_{j,j\neq
i}\mathbb{E}_{\bm{x_{j}}\sim\mathcal{X}}\exp((t_{l}(x_{i})\cdot
s_{l}(x_{j})/\tau))}\right],$ (7)

Table 1: Comparison to state of the art KD methods with heterogeneous architectures on ImageNet-1K. Underlined: the second best result. Bold: the best result. $*$: results achieved by combining with FitNet[5]. Teacher | Student | From Scratch | Logits-based | Feature-based
---|---|---|---|---
T. | S. | KD | DKD | DIST | OFA | FitNet | CC | RKD | CRD | LFCC
_CNN-based students_ |  |  |  |  |  |  |  |  |  |  |
DeiT-T | ResNet-18 | 72.17 | 69.75 | 70.22 | 69.39 | 70.64 | 71.34 | 70.44 | 69.77 | 69.47 | 69.25 | 71.40
Swin-T | ResNet-18 | 81.38 | 69.75 | 71.14 | 71.10 | 70.91 | 71.85 | 71.18 | 70.07 | 68.89 | 69.09 | 72.51
Mixer-B/16 | ResNet-18 | 76.62 | 69.75 | 70.89 | 69.89 | 70.66 | 71.38 | 70.78 | 70.05 | 69.46 | 68.40 | 71.39
DeiT-T | MobileNetV2 | 72.17 | 68.87 | 70.87 | 70.14 | 71.08 | 71.39 | 70.95 | 70.69 | 69.72 | 69.60 | 71.73
Swin-T | MobileNetV2 | 81.38 | 68.87 | 72.05 | 71.71 | 71.76 | 72.32 | 71.75 | 70.69 | 67.52 | 69.58 | 72.84
Mixer-B/16 | MobileNetV2 | 76.62 | 68.87 | 71.92 | 70.93 | 71.74 | 72.12 | 71.59 | 70.79 | 69.86 | 68.89 | 72.05
_Transformer-based students_ |  |  |  |  |  |  |  |  |  |  |
ResNet-50 | DeiT-T | 80.38 | 72.17 | 75.10 |  75.60∗ |  75.13∗ | 76.55∗ | 75.84 | 72.56 | 72.06 | 68.53 | 76.04
ConvNeXt-T | DeiT-T | 82.05 | 72.17 | 74.00 | 73.95 | 74.07 | 74.41 | 70.45 | 73.12 | 71.47 | 69.18 | 75.24
Mixer-B/16 | DeiT-T | 76.62 | 72.17 | 74.16 | 72.82 | 74.22 | 74.46 | 74.38 | 72.82 | 72.24 | 68.23 | 74.57
ResNet-50 | Swin-N | 80.38 | 75.53 | 77.58 |  78.23∗ |  77.95∗ |  78.64∗ | 78.33 | 76.05 | 75.90 | 73.90 | 78.71
ConvNeXt-T | Swin-N | 82.05 | 75.53 | 77.15 | 77.00 | 77.25 | 77.50 | 74.81 | 75.79 | 75.48 | 74.15 | 77.85
Mixer-B/16 | Swin-N | 76.62 | 75.53 | 76.26 | 75.03 | 76.54 | 76.63 | 76.17 | 75.81 | 75.52 | 73.38 | 76.91
_MLP-based students_ |  |  |  |  |  |  |  |  |  |  |
ResNet-50 | ResMLP-S12 | 80.38 | 76.65 | 77.41 |  78.23∗ |  77.71∗ |  78.53∗ | 78.13 | 76.21 | 75.45 | 73.23 | 78.64
ConvNeXt-T | ResMLP-S12 | 82.05 | 76.65 | 76.84 | 77.23 | 77.24 | 77.53 | 74.69 | 75.79 | 75.28 | 73.57 | 77.59
Swin-T | ResMLP-S12 | 81.38 | 76.65 | 76.67 | 76.99 | 77.25 | 77.31 | 76.48 | 76.15 | 75.10 | 73.40 | 77.65

where $\tau$ denotes temperature parameter that modulates the discrimination
against negative samples. We empirically set $\tau$ to 0.1 in all experiments.
$t_{l}(\cdot)$ and $s_{l}(\cdot)$ represent the feature at the $l$-th stage.
The overall Low-pass Contrastive Feature Distillation objective for the
student extends the loss in eq.1 with $\mathcal{L}_{\text{CFD}}$ provides
sample-level supervision.

$\mathcal{L}=\mathcal{L}_{\text{CE}}(\mathbf{p}^{s},y)+\lambda_{\text{KLD}}\mathcal{L}_{\text{KLD}}(\mathbf{p}^{s},\mathbf{p}^{t})+\lambda_{\text{CFD}}\sum_{l}\mathcal{L}_{\text{CFD}}(l).$
(8)

##  4 Experiment

###  4.1 Dataset and Implementation

We conduct comprehensive evaluation of the proposed LFCC method. In this
section, we briefly introduce the dataset and Implementation, and the detailed
setup is provided in the supplementary material.

#### Dataset

We evaluate the heterogeneous distillation method on ImageNet-1K[43] and
CIFAR-100[44]. The ImageNet-1K dataset consists of 1000 categories. The
training set has about 128 million images and the validation set contains 50k
images. The CIFAR-100 dataset contains 100 classes with 600 images per class.
The training and validation sets contain 50K and 10K images.

#### Model

We quantitatively evaluate our LFCC method on two datasets and compare it with
both logit-based and hint-based approaches. The former includes vanilla KD[1],
DKD[2], DIST[3], OFA-KD[4], and the latter contains FitNet[5], CC[6], RKD[7],
and CRD[8]. Additionally, we select models from three architectures: ResNet[9]
and MobileNet V2[45] from the CNN family, DeiT[46], ViT[10], and Swin
Transformer[33] from the Transformer family, and MLP-Mixer[38] and ResMLP[39]
from the MLP family. The Swin Nano and Swin Pico[33, 4] are smaller variants
of the Swin Tiny.

For feature-based distillation, alignment of feature representations between
teacher and student models at each stage is essential. For consistency in
comparison, we adopt the configuration from OFA-KD. In pyramid-structured
models like ResNet[9], we position the feature alignment module at the
conclusion of each stage. Conversely, for models with uniform layer
structures, such as ViT[10], we insert the module at the quartiles of the
overall layer count.

#### Implementation

Images from both datasets are resized to 224x224 for training. Different
optimizers are tailored for each type of student model based on empirical
observations: the CNN family employs the SGD optimizer, while the Transformer
and MLP families use the AdamW optimizer. Students from CNN family are trained
for 100 epochs on the ImageNet-1K dataset, while other families are trained
for 300 epochs. For CIFAR-100, all students are trained for 300 epochs. We
report the top-1 accuracy average on 3 runs in all experiments.

###  4.2 Performance on ImageNet-1K

The comparative experiments on ImageNet-1K involve student models from three
architectures, 15 heterogeneous teacher-student combinations, and 9 knowledge
distillation methods including the proposed LFCC method. The results are
reported in Table 1. Compare to previous distillation methods, our LFCC
achieves state-of-the-art performance in most teacher-student combinations.
Particularly, when the student model follows a CNN architecture, our approach
outperforms other distillation methods in five combinations. When training
students based on Transformer or MLP architectures, LFCC demonstrates
comprehensive advantages, achieving performance gains ranging from 0.06% to
0.83% over the state-of-the-art method OFA-KD[4]. Notably, methods mark with *
in the table combine logit-based methods with FitNet[5], leveraging the
advantages of ResNet in shallow detail extraction. In contrast, our approach
achieves superior performance without introducing additional techniques,
surpassing the combination of the two methods.

Table 2: Comparison to state of the art KD methods with heterogeneous architectures on CIFAR-100. Underlined: the second best result. Bold: the best result.  Teacher | Student | From Scratch | Logits-based | Feature-based
---|---|---|---|---
T. | S. | KD | DKD | DIST | OFA | FitNet | CC | RKD | CRD | LFCC
_CNN-based students_ |  |  |  |  |  |  |  |  |  |  |
Swin-T | ResNet-18 | 89.26 | 74.01 | 78.74 | 80.26 | 77.75 | 80.54 | 78.87 | 74.19 | 74.11 | 77.63 | 83.56
ViT-S | ResNet-18 | 92.04 | 74.01 | 77.26 | 78.10 | 76.49 | 80.15 | 77.71 | 74.26 | 73.75 | 76.42 | 81.45
Mixer-B/16 | ResNet-18 | 87.29 | 74.01 | 77.79 | 78.67 | 76.36 | 79.39 | 77.15 | 74.26 | 73.75 | 76.42 | 81.96
Swin-T | MobileNetV2 | 89.26 | 73.68 | 74.68 | 71.07 | 72.89 | 80.98 | 74.28 | 71.19 | 69.00 | 79.80 | 84.23
ViT-S | MobileNetV2 | 92.04 | 73.68 | 72.77 | 69.80 | 72.54 | 78.45 | 73.54 | 70.67 | 68.46 | 78.14 | 82.06
Mixer-B/16 | MobileNetV2 | 87.29 | 73.68 | 73.33 | 70.20 | 73.26 | 78.78 | 73.78 | 70.73 | 68.95 | 78.15 | 82.49
_Transformer-based students_ |  |  |  |  |  |  |  |  |  |  |
ConvNeXt-T | DeiT-T | 88.41 | 68.00 | 72.99 | 74.60 | 73.55 | 75.76 | 60.78 | 68.01 | 69.79 | 65.94 | 77.95
Mixer-B/16 | DeiT-T | 87.29 | 68.00 | 71.36 | 73.44 | 71.67 | 73.90 | 71.05 | 68.13 | 69.89 | 65.35 | 78.09
ConvNeXt-T | Swin-P | 88.41 | 72.63 | 76.44 | 76.80 | 76.41 | 78.32 | 24.06 | 72.63 | 71.73 | 67.09 | 79.21
Mixer-B/16 | Swin-P | 87.29 | 72.63 | 75.93 | 76.39 | 75.85 | 78.93 | 75.20 | 73.32 | 70.82 | 67.03 | 78.23
_MLP-based students_ |  |  |  |  |  |  |  |  |  |  |
ConvNeXt-T | ResMLP-S12 | 88.41 | 66.56 | 72.25 | 73.22 | 71.93 | 81.22 | 45.47 | 67.70 | 65.82 | 63.35 | 79.08
Swin-T | ResMLP-S12 | 89.26 | 66.56 | 71.89 | 72.82 | 11.05 | 80.63 | 63.12 | 68.37 | 64.66 | 61.72 | 78.88

###  4.3 Performance on CIFAR-100

The experiments on the CIFAR-100 dataset are divided into three groups based
on the family to which the student models belong, involving 12 heterogeneous
teacher-student combinations. The proposed LFCC method is compared with 8
other state-of-the-art knowledge distillation methods. The results are
presented in Table 2.

LFCC significantly outperforms existing heterogeneous distillation methods
when CNN-based model or DeiT-T is used as a student. Meanwhile, when MLP-based
model is employed as a student, LFCC tends to yield suboptimal results.
CIFAR-100, being a relatively small-scale dataset, logit-based methods
typically outperform most feature-based methods, especially when the student
model is belong to Transformer or MLP families. This indicates substantial
disparities in feature representations among the models with heterogeneous
architectures. Previous feature-based methods attempt to mimic the teacher’s
feature representations at local details, which may be detrimental, leading to
overfitting on small datasets. LFCC extracts low-frequency components of
features and aligns them in a compact space between teacher and student,
uncovering commonalities between the models with heterogeneous architectures
and significantly enhancing the performance and robustness of feature-based
methods.

By comparing Table 1 and Table 2, we observe that the performance of
distillation methods is influenced by the architectures of teacher and student
models as well as the scale of the dataset. Despite our method being solely
based on CNN architecture, it still achieves state-of-the-art results in most
scenarios. This indicates that by extracting low-frequency components and
compressing the feature alignment space, LFCC identifies commonalities in
feature representations among models of various architectures. This exciting
finding overturns the current situation where feature-based methods are deemed
unsuitable for heterogeneous knowledge distillation.

Figure 2: Ablation studies on ImageNet-1K. (a) Effect of proposed components.
We incrementally incorporate the proposed components into the vanilla teacher-
student framework to validate their efficacy. $\lambda_{\text{CFD}}$ is set as
1.0 in all experiments. (b) We perform a comprehensive assessment of the
impact of $\lambda_{\text{CFD}}$ on two distinct teacher-student
configurations.

###  4.4 Ablation study

#### Effect of the proposed components.

We systematically incorporate the proposed components into the vanilla
teacher-student (ResNet18 vs DeiT-tiny) framework to comprehensively assess
their efficacy. The findings are depicted in Figure 2(a). The results strongly
validate the effectiveness of our methodology, with each component
contributing to a marked improvement in the student model’s performance. It is
noteworthy that within our learnable low-pass filters, both the multi-scale
low-pass filtering and convolutional downsample pathways positively impact the
distillation process, highlighting their complementary functions.

Table 3: Effect of number and position of aligned module on ImageNet-1K. Stage | T: DeiT-T | T: ResNet50
---|---|---
1 | 2 | 3 | 4 | S: ResNet18 | S: ResMLP-S12
- | - | - | - | 70.59 | 77.32
✓ | - | - | - | 70.72 | 77.41
- | ✓ | - | - | 70.69 | 77.49
- | - | ✓ | - | 70.91 | 77.67
- | - | - | ✓ | 71.21 | 77.94
- | - | ✓ | ✓ | 71.33 | 78.32
- | ✓ | ✓ | ✓ | 71.36 | 78.54
✓ | ✓ | ✓ | ✓ | 71.40 | 78.64

#### Weight of contrastive feature loss.

Within our LFCC framework, a diverse array of loss functions is employed,
mandating the identification of an optimal equilibrium among these components.
Considering the extensive search for combinations of weights to be
impractical, we fix the weights for cross-entropy and Kullback-Leibler
divergence empirically, focusing solely on tuning $\lambda_{\text{CFD}}$.
Experiments are executed on two distinct teacher-student pairs: (ResNet18 vs
DeiT-tiny) and (ResMLP-S12 vs ResNet50). The results, as illustrated in Figure
2(b), indicate that the LFCC framework achieves peak performance at
$\lambda_{\text{CFD}}=4$. Consequently, $\lambda_{\text{CFD}}=4$ is adopted as
the default setting for subsequent experiments.

#### Position and number of aligned modules.

Our LFCC inserts auxiliary feature alignment modules into intermediate layers
to capture the intercommunity between teacher and student. In our experiments,
we investigate the optimal positioning and quantity of alignment modules. Both
the student and teacher models are divided into four stages, as represented by
corresponding numbers in Table 3. We observe that while alignment modules at
each stage contribute to the training of the student model, inserting an
alignment module into the final stage yields the most significant benefits,
particularly for ResNet18. Experimental results indicate that incorporating
alignment modules in all stages provides the most pronounced enhancement to
the student’s performance.

Table 4: KD methods with homogeneous architectures on ImageNet-1K. _T_ : ResNet34, _S_ : ResNet18. | T. | S. |  KD [1] |  OFD [47] |  Rev [48] |  CRD [8] |  DKD [2] |  DIST [3] |  OFA [4] | LFCC
---|---|---|---|---|---|---|---|---|---|---
Acc | 73.31 | 69.75 | 70.66 | 70.81 | 71.61 | 71.17 | 71.70 | 72.07 | 72.10 | 72.26
Figure 3: CKA-assessed similarity heatmap of intermediate features, where
RMLP-S12 denotes ResMLP-S12 and MbNet represents MobileNet V2. Each row of the
figure representing the feature similarity for a given model in comparison to
the teacher model across three conditions: the teacher model itself, a student
model trained from scratch, and a student model trained via LFCC.

#### Distillation in homogeneous architectures.

We further validate the performance of our proposed Low-Frequency Components-
based Contrastive Knowledge Distillation (LFCC) on the benchmark for
homogenous architecture distillation, which is widely recognized. The results,
as presented in Table 4, indicate that LFCC surpasses other baseline
distillation approaches when applied to the teacher-student pair (ResNet18 vs
ResNet34), thereby convincingly highlighting its robust capability and
generalizability.

#### Visualization of the feature alignment.

Centered Kernel Alignment (CKA)[49, 50] serves as a metric for assessing
feature similarity, enabling cross-architectural comparisons by accommodating
inputs of different dimensions. We qualitatively evaluate the efficacy of the
proposed LFCC in heterogeneous architectural feature alignment via CKA. The
comparison is conducted on four teacher-student pairs, assessing their
intermediate features both before and after guidance from a heterogeneous
teacher. The results on the CIFAR-100 dataset are depicted in Figure 3. Within
the LFCC framework, convergence of the intermediate features between the
student and teacher models is observed, signifying the efficacy of our method
in establishing feature alignment across heterogeneous architectures and
consequently improving the student model’s performance.

##  5 Conclusion

To mitigate the diminished performance of feature-based distillation methods
across heterogeneous architectures, we propose an innovative framework, named
Low-Frequency Components-based Contrastive Knowledge Distillation (LFCC) .
This framework is engineered to discover shared low-frequency components
within the intermediate features of teacher and student models, aligning them
within a condensed space. Consequently, LFCC neutralizes the intrinsic
disparities in feature representation inherent to heterogeneous models,
facilitating the precise conveyance of knowledge encapsulated in the teacher’s
intermediate features. Furthermore, capitalizing on the innate pairing within
the teacher-student construct, we construct batches of teacher and student
features into pairs of positive and negative samples, thereby engaging in
sample-level contrastive learning. The efficacy of our approach is
corroborated by its robust performance on the formidable benchmarks of
ImageNet-1K and CIFAR-100.

## References

  * [1] Geoffrey Hinton, Oriol Vinyals, and Jeff Dean.  Distilling the knowledge in a neural network.  arXiv preprint arXiv:1503.02531, 2015.
  * [2] Borui Zhao, Quan Cui, Renjie Song, Yiyu Qiu, and Jiajun Liang.  Decoupled knowledge distillation.  In Proceedings of the IEEE/CVF Conference on computer vision and pattern recognition, pages 11953–11962, 2022.
  * [3] Tao Huang, Shan You, Fei Wang, Chen Qian, and Chang Xu.  Knowledge distillation from a stronger teacher.  Advances in Neural Information Processing Systems, 35:33716–33727, 2022.
  * [4] Zhiwei Hao, Jianyuan Guo, Kai Han, Yehui Tang, Han Hu, Yunhe Wang, and Chang Xu.  One-for-all: Bridge the gap between heterogeneous architectures in knowledge distillation.  In Advances in Neural Information Processing Systems, 2023.
  * [5] Adriana Romero, Nicolas Ballas, Samira Ebrahimi Kahou, Antoine Chassang, Carlo Gatta, and Yoshua Bengio.  Fitnets: Hints for thin deep nets.  In International conference on machine learning. PMLR, 2015.
  * [6] Baoyun Peng, Xiao Jin, Jiaheng Liu, Dongsheng Li, Yichao Wu, Yu Liu, Shunfeng Zhou, and Zhaoning Zhang.  Correlation congruence for knowledge distillation.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 5007–5016, 2019.
  * [7] Wonpyo Park, Dongju Kim, Yan Lu, and Minsu Cho.  Relational knowledge distillation.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 3967–3976, 2019.
  * [8] Yonglong Tian, Dilip Krishnan, and Phillip Isola.  Contrastive representation distillation.  In International Conference on Learning Representations, 2019.
  * [9] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770–778, 2016.
  * [10] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al.  An image is worth 16x16 words: Transformers for image recognition at scale.  In International Conference on Learning Representations, 2020.
  * [11] Andrew G Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, and Hartwig Adam.  Mobilenets: Efficient convolutional neural networks for mobile vision applications.  arXiv preprint arXiv:1704.04861, 2017.
  * [12] Sihui Luo, Xinchao Wang, Gongfan Fang, Yao Hu, Dapeng Tao, and Mingli Song.  Knowledge amalgamation from heterogeneous networks by common feature learning.  In Proceedings of the 28th International Joint Conference on Artificial Intelligence, pages 3087–3093, 2019.
  * [13] Chengchao Shen, Mengqi Xue, Xinchao Wang, Jie Song, Li Sun, and Mingli Song.  Customizing student networks from heterogeneous teachers via adaptive knowledge amalgamation.  In 2019 IEEE/CVF International Conference on Computer Vision (ICCV), Oct 2019.
  * [14] Nikolaos Passalis, Maria Tzelepi, and Anastasios Tefas.  Heterogeneous knowledge distillation using information flow modeling.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2339–2348, 2020.
  * [15] Zhimeng Zheng, Tao Huang, Gongsheng Li, and Zuyi Wang.  Promoting cnns with cross-architecture knowledge distillation for efficient monocular depth estimation.  arXiv preprint arXiv:2404.16386, 2024.
  * [16] Zhenyu Wang, Hao Luo, Pichao Wang, Feng Ding, Fan Wang, and Hao Li.  Vtc-lfc: Vision transformer compression with low-frequency components.  Advances in Neural Information Processing Systems, 35:13974–13988, 2022.
  * [17] Bo Dong, Pichao Wang, and Fan Wang.  Head-free lightweight semantic segmentation with linear transformer.  In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pages 516–524, 2023.
  * [18] Gang Li, Xiang Li, Yujie Wang, Shanshan Zhang, Yichao Wu, and Ding Liang.  Knowledge distillation for object detection via rank mimicking and prediction-guided feature imitation.  In Proceedings of the AAAI conference on artificial intelligence, volume 36, pages 1306–1313, 2022.
  * [19] Rongzhi Zhang, Jiaming Shen, Tianqi Liu, Jialu Liu, Michael Bendersky, Marc Najork, and Chao Zhang.  Do not blindly imitate the teacher: Using perturbed loss for knowledge distillation.  arXiv preprint arXiv:2305.05010, 2023.
  * [20] Seyed Iman Mirzadeh, Mehrdad Farajtabar, Ang Li, Nir Levine, Akihiro Matsukawa, and Hassan Ghasemzadeh.  Improved knowledge distillation via teacher assistant.  In Proceedings of the AAAI conference on artificial intelligence, volume 34, pages 5191–5198, 2020.
  * [21] Shangquan Sun, Wenqi Ren, Jingzhi Li, Rui Wang, and Xiaochun Cao.  Logit standardization in knowledge distillation.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024.
  * [22] Junqiang Huang and Zichao Guo.  Pixel-wise contrastive distillation.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 16359–16369, 2023.
  * [23] Chaoning Zhang, Dongshen Han, Yu Qiao, Jung Uk Kim, Sung-Ho Bae, Seungkyu Lee, and Choong Seon Hong.  Faster segment anything: Towards lightweight sam for mobile applications.  arXiv preprint arXiv:2306.14289, 2023.
  * [24] Sihao Lin, Hongwei Xie, Bing Wang, Kaicheng Yu, Xiaojun Chang, Xiaodan Liang, and Gang Wang.  Knowledge distillation via the target-aware transformer.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10915–10924, 2022.
  * [25] Junho Yim, Donggyu Joo, Jihoon Bae, and Junmo Kim.  A gift from knowledge distillation: Fast optimization, network minimization and transfer learning.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4133–4141, 2017.
  * [26] Ziyao Guo, Haonan Yan, Hui Li, and Xiaodong Lin.  Class attention transfer based knowledge distillation.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11868–11877, 2023.
  * [27] Zhendong Yang, Zhe Li, Mingqi Shao, Dachuan Shi, Zehuan Yuan, and Chun Yuan.  Masked generative distillation.  In European Conference on Computer Vision, pages 53–69. Springer, 2022.
  * [28] Zhendong Yang, Zhe Li, Xiaohu Jiang, Yuan Gong, Zehuan Yuan, Danpei Zhao, and Chun Yuan.  Focal and global knowledge distillation for detectors.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4643–4652, 2022.
  * [29] Mingxing Tan and Quoc Le.  Efficientnet: Rethinking model scaling for convolutional neural networks.  In International conference on machine learning, pages 6105–6114. PMLR, 2019.
  * [30] Ilija Radosavovic, Raj Prateek Kosaraju, Ross Girshick, Kaiming He, and Piotr Dollár.  Designing network design spaces.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10428–10436, 2020.
  * [31] Zhuang Liu, Hanzi Mao, Chao-Yuan Wu, Christoph Feichtenhofer, Trevor Darrell, and Saining Xie.  A convnet for the 2020s.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 11976–11986, 2022.
  * [32] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  Advances in neural information processing systems, 30, 2017.
  * [33] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo.  Swin transformer: Hierarchical vision transformer using shifted windows.  In Proceedings of the IEEE/CVF international conference on computer vision, pages 10012–10022, 2021.
  * [34] Xiaosong Zhang, Yunjie Tian, Lingxi Xie, Wei Huang, Qi Dai, Qixiang Ye, and Qi Tian.  Hivit: A simpler and more efficient design of hierarchical vision transformer.  In International Conference on Learning Representations, 2023.
  * [35] Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko.  End-to-end object detection with transformers.  In European conference on computer vision, pages 213–229. Springer, 2020.
  * [36] Enze Xie, Wenhai Wang, Zhiding Yu, Anima Anandkumar, Jose M Alvarez, and Ping Luo.  Segformer: Simple and efficient design for semantic segmentation with transformers.  In Neural Information Processing Systems (NeurIPS), 2021.
  * [37] William Peebles and Saining Xie.  Scalable diffusion models with transformers.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 4195–4205, 2023.
  * [38] Ilya O Tolstikhin, Neil Houlsby, Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai, Thomas Unterthiner, Jessica Yung, Andreas Steiner, Daniel Keysers, Jakob Uszkoreit, et al.  Mlp-mixer: An all-mlp architecture for vision.  Advances in neural information processing systems, 34:24261–24272, 2021.
  * [39] Hugo Touvron, Piotr Bojanowski, Mathilde Caron, Matthieu Cord, Alaaeldin El-Nouby, Edouard Grave, Gautier Izacard, Armand Joulin, Gabriel Synnaeve, Jakob Verbeek, et al.  Resmlp: Feedforward networks for image classification with data-efficient training.  IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(4):5314–5321, 2022.
  * [40] Hanxiao Liu, Zihang Dai, David So, and Quoc V Le.  Pay attention to mlps.  Advances in neural information processing systems, 34:9204–9215, 2021.
  * [41] Xiaohan Ding, Honghao Chen, Xiangyu Zhang, Jungong Han, and Guiguang Ding.  Repmlpnet: Hierarchical vision mlp with re-parameterized locality.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 578–587, 2022.
  * [42] Qibin Hou, Zihang Jiang, Li Yuan, Ming-Ming Cheng, Shuicheng Yan, and Jiashi Feng.  Vision permutator: A permutable mlp-like architecture for visual recognition.  IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022\.
  * [43] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei.  Imagenet: A large-scale hierarchical image database.  In 2009 IEEE conference on computer vision and pattern recognition, pages 248–255. Ieee, 2009.
  * [44] Alex Krizhevsky, Geoffrey Hinton, et al.  Learning multiple layers of features from tiny images.  2009\.
  * [45] Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chieh Chen.  Mobilenetv2: Inverted residuals and linear bottlenecks.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4510–4520, 2018.
  * [46] Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou.  Training data-efficient image transformers & distillation through attention.  In International conference on machine learning, pages 10347–10357. PMLR, 2021.
  * [47] Byeongho Heo, Jeesoo Kim, Sangdoo Yun, Hyojin Park, Nojun Kwak, and Jin Young Choi.  A comprehensive overhaul of feature distillation.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1921–1930, 2019.
  * [48] Pengguang Chen, Shu Liu, Hengshuang Zhao, and Jiaya Jia.  Distilling knowledge via knowledge review.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5008–5017, 2021.
  * [49] Corinna Cortes, Mehryar Mohri, and Afshin Rostamizadeh.  Algorithms for learning kernels based on centered alignment.  The Journal of Machine Learning Research, 13(1):795–828, 2012.
  * [50] Simon Kornblith, Mohammad Norouzi, Honglak Lee, and Geoffrey Hinton.  Similarity of neural network representations revisited.  In International conference on machine learning, pages 3519–3529. PMLR, 2019.


# PowerYOLO: Mixed Precision Model for Hardware Efficient Object Detection
with Event Data ††thanks: The work presented in this paper was supported by:
the program ”Excellence initiative –- research university” for the AGH
University of Krakow. We gratefully acknowledge Polish high-performance
computing infrastructure PLGrid (HPC Centers: ACK Cyfronet AGH) for providing
computer facilities and support within computational grant no. PLG/2023/01619

Dominika Przewlocka-Rus1, Tomasz Kryjak2 Senior Member IEEE and Marek Gorgon3
Senior Member IEEE Embedded Vision Systems Group, AGH University of Krakow
Krakow, Poland
Email<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

The performance of object detection systems in automotive solutions must be as
high as possible, with minimal response time and, due to the often battery-
powered operation, low energy consumption. When designing such solutions, we
therefore face challenges typical for embedded vision systems: the problem of
fitting algorithms of high memory and computational complexity into small low-
power devices. In this paper we propose PowerYOLO – a mixed precision
solution, which targets three essential elements of such application. First,
we propose a system based on a Dynamic Vision Sensor (DVS), a novel sensor,
that offers low power requirements and operates well in conditions with
variable illumination. It is these features that may make event cameras a
preferential choice over frame cameras in some applications. Second, to ensure
high accuracy and low memory and computational complexity, we propose to use
4-bit width Powers-of-Two (PoT) quantisation for convolution weights of the
YOLO detector, with all other parameters quantised linearly. Finally, we
embrace from PoT scheme and replace multiplication with bit-shifting to
increase the efficiency of hardware acceleration of such solution, with a
special convolution-batch normalisation fusion scheme. The use of specific
sensor with PoT quantisation and special batch normalisation fusion leads to a
unique system with almost 8x reduction in memory complexity and vast
computational simplifications, with relation to a standard approach. This
efficient system achieves high accuracy of mAP $0.301$ on the GEN1 DVS
dataset, marking the new state-of-the-art for such compressed model.

###### Index Terms:

embedded vision systems, dynamic vision sensor, hardware-aware algorithm
desing, logarithmic quantization, mixed precision, pedestrian detection,
power-of-two quantization, vehicle detection,

##  I Introduction

Vision systems based on machine learning algorithms are already becoming a
standard in many areas of our lives: in applications and devices that provide
entertainment (intelligent photo and video processing, augmented/virtual
reality), in surveillance and security systems and autonomous vehicles, or
advanced driver assistance systems.

Naturally, in most applications we consider devices with limited energy budget
– mobile and battery-powered – which means that the use of power-hungry
algorithms based on neural networks requires careful design of such solutions,
generally with some reduction in memory-computational complexity relative to
standard approaches.

Such a reduction generally means changing the precision of computations from
32-bit floating-point numbers to 16-bit floating-point, 8-bit fixed-point or
even, in special cases, 6-bit and 4-bit fixed-point numbers, but above all to
8-bit or even lower-width integers, including binary and ternary. With the
right hardware platform, such simplifications can lead to increased
performance (GPU, CPU), often also allowing a reduction in energy requirements
with the right design of custom processor (FPGA, ASIC). In addition, they
allow to address to problem of memory bandwidth gap and reduce the size of the
model. It is a standard practice to use quantisation to 8-bit integers, which
only slightly affects the effectiveness of the neural network, especially if
appropriate quantisation aware training (QAT) is used.

The use of lower precision while retaining the accuracy of the full precision
model may require specific quantisation schemes, such as the logarithmic
quantisation benchmarked earlier for the classification task in [1], [2], [3].

Pedestrian and vehicle detection are tasks present in a wide range of
applications: from driver assistance systems and autonomous vehicles, to
security and surveillance systems. All those applications have similar
requirements – high performance, real-time data processing, and rather low
energy budget. Therefore, the design of such a solution must be approached
comprehensively: by choosing the right algorithm, hardware platform and even
sensor. A standard digital camera is a natural choice, while an event camera
(dynamic vision sensor, neuromorphic camera) [4] is recently becoming an
interesting alternative.

In their operation, event cameras mimic the characteristics of the human
visual system, noticing only changes in brightness per pixel (and therefore
events per pixel). As a result, they record and transmit data with lower
latency than standard cameras and also have a higher dynamic range, making
them work well in a wide variety of lighting conditions, including unevenly
and poorly illuminated scenes. The latter can be particularly important in
changing road conditions.

Figure 1: Overview of the proposed system: event camera data is gathered over
a predetermined time window (10 ms), and represented in the form of a pseudo-
image (event frame). Pedestrians are detected using a mixed precision YOLO
network, with convolution layer weights quantised logarithmically to PoT
values, and the rest of parameters and activations quantised linearly.

In this paper we propose to use logarithmic quantisation for a task complex
and commonly considered in the context of low-power devices – pedestrian and
vehicle detection. In addition to designing an algorithm with low memory and
computational complexity, it was also decided to use an event camera which
characteristics are perfectly suited to the real-time and high accuracy
requirements of embedded perception systems. A schematic of the system is
presented in Fig. 1: the data received from the event camera is represented by
a pseudo-image (i.e. event frame), which is then processed by the PowerYOLO
mixed-precision network. Due to the PoT weights, a hardware implementation of
the PowerYOLO network can introduce significant energy gains, resulting from
the simplification of multiplication operations and therefore reducing the
number of required electronic components needed to implement the corresponding
accelerator. Thus, the main contribution of this work can be summarised as
follows:

  1. 1.

we propose a highly efficient INT8/LOG4 mixed precision PowerYOLO network for
pedestrian and vehicle detection for GEN1 event dataset. Our model stands out
for having the best ratio of efficiency to memory-computation complexity
achieving mAP 0.301.

  2. 2.

we propose a method for convolution and batch normalisation layers fusion in a
way that takes into account the special form of the neural network weights
when quantised to powers of two.

The reminder of this paper is organised as follows: Section II presents basic
information about event cameras. Section III discusses previous work related
to the optimisation of neural network architectures and an overview of object
detection methods based on event data stream. Section IV presents the
PowerYOLO network, which uses a mixed integer and power-of-two quantisation
scheme. Its evaluation is then provided is Section V. The article concludes
with a brief summary and suggestions for further research.

##  II Event cameras

Event cameras record changes in light intensity individually (asynchronously)
for each pixel. A single event is represented by a vector $e=\\{t,x,y,p\\}$,
where $t$ is the occurrence time of the event (timestamp), $x$ and $y$ are the
spatial coordinates of the recorded pixel, and $p$, called polarisation, takes
the value $-1$ or $1$ (sometimes also $0$ and $1$) depending on the direction
of brightness change. Therefore, data acquisition only takes place when the
brightness of the pixel changes, which enables high-resolution with low-
latency data flow. In addition, this results in a significant reduction in
blur in the event of fast movement of objects or the camera. Moreover, the
high dynamic range resolution allows events to be recorded in both bright and
dark regions. This is especially important for outdoor solutions, where
lighting is highly variable (e.g. in automotive: tunnel crossings, under
overpasses, etc.). From the perspective of designing energy-efficient
solutions, these sensors seem particularly interesting: due to their
asynchronous and sparse operation (recording only changes in brightness), they
are on average more energy-efficient than typical digital cameras (frame
cameras).

However, the format of event data (spatio-temporal sparse cloud) makes it
impossible to use explicitly standard digital image processing algorithms,
including those based on neural networks, for their analysis. Therefore four
possibilities are considered: designing a solution dedicated to event data
(e.g. spiking neural networks – SNN) [5], processing a point cloud e.g. using
graph neural networks (which can be challenging due to the different density
of such a cloud in successive time segments) [6], projecting (aggregating)
events onto a plane and creating so-called pseudo-images [7], or image
reconstruction from such a cloud (which is computationally very complex) [8].

The use of pseudo-images best cooperates with the typical neural networks
developed for images registered with frame cameras. A number of such
approaches have been proposed in the literature - they differ in the way they
take into account the time dimension, in particular the change in signal over
time. In this work, we use the event frame method, which transforms events
from a specific time slice (in our case 10ms) $\tau$ according to
$f(u,t)=P_{e}(u)$ for $t-\sigma_{e}(u)\in(0;\tau)$ and $f(u,t)=0$ otherwise,
where $\sigma_{e}$ denotes the time of the event $u(x,y)$ and $P_{e}$ its
polarity. This choice is dictated by two factors: firstly, the chosen object
detection algorithm is dedicated to classical images, so the input format must
be similar to them; secondly, due to the necessary limitations of memory and
computational complexity for low-power devices, the method of converting the
event cloud to a pseudo-image should not be too complex. Other pseudo-image
representations worth mentioning are: simple binary frame [9], exponentially
decaying time surface [10] and event frequency [11].

In this research we used the Gen1 event dataset consisting of over 39 hours of
recordings of urban, highway, suburbs and countryside scenes with a Prophesee
Gen1 sensor attached to a car. The collection was manually tagged based on
greyscale images recorded simultaneously with the events, ultimately
highlighting nearly $230000$ of cars and nearly $28000$ of pedestrians. For
the generation of pseudo-images, we used the tools provided by the authors of
the database [7].

##  III Related Work

In this section we present previous work on two topics closely related to our
research.

###  III-A Reducing the memory and computational complexity of neural networks

Standard methods for reducing the memory-computation complexity of neural
networks include quantisation, pruning, and appropriate transformations in the
form of layer fusion that do not change the inference result. The first two
methods lead to a reduction in the number of bits needed for parameter storage
and the number of parameters, respectively. In its simplest form, pruning,
i.e. zeroing the connections or neurons with low impact on the final result,
leads to a sparse network in which the number of arithmetic operations
performed can be considerably reduced. However, this introduces the overhead
of having to remember which weights or neurons have been zeroed (i.e. which
calculations should be omitted). An alternative is to train sparse networks,
i.e. networks in which the structure of the removed connections is
predetermined. Since neural networks are generally redundant, appropriately
performed pruning, or sparse network training, allows to achieve the
performance of a dense network.

For the same reason, a reduction in computational precision (quantisation) may
also not result in a significant drop in performance relative to a floating-
point network, while at the same time reducing memory complexity by several
times. A certain standard available in many tools, both software tools (e.g.
tensorflow [12], OpenVino [13], brevitas [14]) but also in hardware
implementations (e.g. FINN [15], VitisAI [16], nncf [17]) is linear
quantisation, in particular to 8-bit integer values.

This method reduces the model by a factor of four (in terms of number of
bytes) and significantly simplifies the implementation of arithmetic
operations, which is particularly important for low-power hardware
implementations. Linear quantisation to lower bit-widths can significantly
affect the accuracy of the solution, therefore logarithmic quantisation
methods, first proposed in [18], are an interesting alternative. This scheme
allows to preserve the accuracy of the full-precision network for parameters
4-bit and smaller (after appropriate training of the quantised network) [1],
[2]. In addition, this approach allows to replace the multiplication with a
bit-shift, which, using the appropriate computing platform (GPU, eGPU, and
especially reconfigurable platforms like FPGAs), results in reduced latency
and reduced power consumption [2], [3], [19]. Obviously, the most radical form
of reduction in computational precision is binary quantisation, however it is
not without an impact on the accuracy of the network [20], causing losses of
several tens of per cent.

Extremely low-precision networks, or rather new quantisation methods, are most
often benchmarked for classification tasks, so it is difficult to say
unequivocally whether the impact of different precision reduction methods is
same for each task solved by a neural network, in particular for object
detection. However, it seems that due to the higher complexity and the need
for higher precision information at the output, quantisation to very low bit-
widths for detection task may have a greater impact on performance: especially
if we consider the coordinates of the bounding box, rather than a class
probability score. In principle, the latter should simply be higher for true
class objects than for others, rather than with some specific value. Several
experiments with very low-precision quantisation of YOLO networks have been
described in the literature.

In [21], the authors note that quantisation of YOLO networks is challenging
due to the problem of oscillation of weights between quantisation thresholds
when learning low-precision networks, even at final epochs. The use of two
additional methods EMA (Exponentially Moving Average) and QC (Quantisation
Correction) have been proposed to minimise these oscillations ,in combination
with quantised training. In the Quantisation Aware Training (QAT) version, EMA
works similarly as in full precision network training, storing a history of
both the weights and the scaling factors of the weights and activations. With
the introduction of EMA into QAT, the training is ﬁnitely smoother. In
addition, Quantisation Correction has been introduced after the training
process, to minimise oscillations errors by linearly transforming the
activations based on some pre-trained parameters. The YOLOv5s network trained
with such 4-bit quantisation on the COCO dataset achieves a mAP of $0.34$,
with a baseline (full-precision network) mAP of $0.374$ – thus a decrease of
about $11\%$ (for more results for different architectures, we refer to the
article). In [22], the authors proposed a Post-Training Quantisation method
that achieved a mAP of $0.196$ for YOLOv5s and a COCO dataset for 4-bit
weights.

It is worth noting that both of these methods propose the use of a linear
quantisation scheme with additional mechanisms to prevent significant drops in
efficiency, as achieved in [21]. In this paper, we propose a mixed precision
solution, with the network weights quantised logarithmically to powers-of-two,
and the rest of the activation and parameters quantised linearly.

Such a method allows a significant reduction in computational complexity due,
firstly, to the possibility of changing the multiplication operation to a bit
shift and, secondly, performing the remaining computations as integer
operations.

The gains from converting multiplication operations to bit shifting have been
described in several papers. In [1], the authors implemented GPU kernels for
PoT-weighted computing and showed gains over standard filtering with
multiplication, shortening the inference time for ResNet18 by $25\%$. In [23]
the authors show gains in energy and logic resources used for 5-bit
logarithmic quantisation relative to standard 16-bit multiplier for UMC 55nm
Low Power Process. The paper [24] presents dynamic power consumption for a
hardware implementation of matrix multiplication for uniformly and PoT
quantised values. At 3 bits, one can assume a decrease in power requirements
of about $20\%$, at 5 bits – $35\%$, in favour of PoT weights. As was
previously shown in [19], the hardware convolution layer accelerator on the
Zynq UltraScale+ MPSoC ZCU104 development board with a SoC FPGA chip, with
4-bit PoT weights can be at least $1.4x$ more power efficient than the
linearly quantised version, and the difference in power efficiency increases
at higher chip clock frequencies.

In this paper, we propose a mixed-precision detector that optimises all
parameters of a neural network: to our knowledge, all previous work on
logarithmic quantisation has referred to bit-width reduction mainly of weights
and, less often, also of activation, neglecting the issue of the batch
normalisation layer. Although this layer is not a necessary part of building
neural network architectures, it is a certain standard in most classical
solutions. It is therefore important to propose a method that is usable for
virtually any ready-made full-precision model.

###  III-B Object detection based on event data

Work on pedestrian and vehicle detection on the GEN1 event dataset [25] can be
found in the literature. The dataset itself was created from more than 39
hours of recordings using the GEN1 event sensor from the Prophesee company,
collecting data at a resolution of 304x240.

In [26] authors proposed general methods for converting models trained on
image-like event representations to models that operate asynchronously. Event
data are represented by a Sparse Recursive Representation, which can be
updated as new events arrive in a sparse manner (i.e. only where a change has
occurred). This form of input allows the use of so-called sparse convolutions,
recalculating only those activations which value may have changed. Using such
a model, a mAP of $0.129$ was obtained for the GEN1 set and event histogram
representation ($0.145$ using a standard convolutional neural network).

The paper [27] proposes a directly trained spiking network – EMS-YOLO. The
first convolution layer is trained by converting the input to spikes, the
subsequent layers consist of EMS-Modules, which are fully spiking residual
blocks with LIF (Leaky Integrate and Fire) activation function. The input of
the network is an image-based 2D representation of events within a specific
time window. On the GEN1 set, mAP of $0.267$ was achieved, for inference over
4 time steps, with a Firing Rate of $21.15\%$ neurons, and a network with
$6.20M$ parameters.

A spiking feature pyramid network (SpikeFPN), consisting of an encoder
backbone spiking neural network, a spiking feature pyramid building network
and a spiking multi-head-detection module, was presented in [5]. Event data
are represented using Stacking Based on Time (SBT), which, with standard
compression of the event stream into a frame, allows some temporal information
to be retained. A mAP of $0.223$ was achieved for the GEN1 dataset.

The Group Event Transformer architecture is proposed in [28], which is
dedicated to process event data directly and thus without the need to convert
events to a 2D representation, which results in the loss of some timing or
polarity information. Events are embedded into Group Tokens based on timing
and polarity information, and then processed by a transformer-based
architecture using self-attention mechanisms. On the GEN1 dataset, the
proposed model achieves mAP of $0.406$, and with additional memory mechanisms
to support the challenges of capturing only moving objects by the event
camera, $0.484$.

Similarly, in [29] the authors also propose to analyse events directly, in the
form of a structured (in terms of spatial dimensions) point cloud. At each
time step, the model based on the Vision Transformer architecture processes a
new data stream together with previous states from recursive layers. The
recursive layers help to solve the problem of slow-moving objects not captured
by the event camera at each time step. The described model achieves mAP of
$0.472$ on the GEN1 dataset, for the largest of the proposed architectures
(for the 4x smaller model mAP $0.441$).

A recursive architecture consisting of standard convolution layers and LSTM
blocks is presented in [7]. The model directly processes the event data stream
as a tensor with fixed spatial dimensions. The proposed method achieves mAP of
$0.4$ for GEN1 dataset, with $24.1M$ parameters. In [30] the authors proposed
to use the YOLO (You Only Look Once) model, together with an adaptive
conversion of the events to Hyper Histogram form, which allows the
preservation of both temporal and polarity information. In addition, a
corresponding modification of one of the key data augmentation methods for
YOLO detectors in the form of Shadow Mosaic was also proposed. The model
trained in such a way achieves mAP of $0.47$, while the analogous solution
without Hyper Histogram and Shadow Mosaic $0.394$ (treated by the authors as a
baseline). In [31] the authors proposed another way of representing event data
- Temporal Active Focus (TAF) - along with a small Agile Event Detector (AED)
neural network model with a corresponding event encoder extracting semantic
information from temporal to flat vector input. The architecture of the
detector itself is based on Darknet21 and head like for YOLOX. Ultimately, the
model achieves mAP of $0.454$, with $14.8M$ of parameters and a baseline
(YOLOX architecture and Event Volume representation) of $0.35$.

Table I summarises the discussed previous work on object detection on the GEN1
dataset. It is worth noting that these solutions are classical in the sense of
computational precision, i.e. they operate on 32- or 16-bit floating point
numbers, and mostly require GPU acceleration. For solutions based on spiking
neural networks ([26], [27], [5]), the firing rate can be as low as $20\%$,
which of course translates into lower computational complexity, but these
solutions have much lower accuracy (and the memory complexity remains
constant). Independently, due to their size, efficient use of the mentioned
models in low-power devices directly is virtually impossible.

TABLE I: Overview of existing object detectors for the GEN1 dataset, distinguishing mAP and number of model parameters. Method |  mAP |  Model size
---|---|---
ASCNN [26] |  $0.129$ |  $133$M
EMS-ResNet [27] |  $0.267$ |  $6.2$M
SpikeFPN [5] |  $0.223$ |  $11$M
GET [28] |  $0.479$ |  $21.9$M
RVT-S [29] |  $0.465$ |  $9.9$M
RVT-T [29] |  $0.441$ |  $4.4$M
Gray-RetinaNet [7] |  $0.44$ |  $32.8$M
YOLOv5l [30] |  $0.47$ |  $46.5$M
AED [31] |  $0.454$ |  $14.8$M

For this article, the last two methods proposed by [30] and [31] are
particularly relevant: the former because of the used YOLO architecture, and
the latter because of the small size of the network. To the authors’ best
knowledge, these models determine the current SoTA (in the sense of highest
detection accuracy) on the GEN1 dataset, and it is to these models that we
will compare our proposed mixed precision network. The analysis shows that,
besides the choice of architecture (spiking network, classical deep detectors
or transformers), the representation of the input data itself
(multidimensional tensor or 2D image), but also its augmentation, is
important.

In addition, object detection on event frames faces the problem of continuity
of information in case of objects moving at variable speeds (in particular low
or even static). This is an issue not encountered in such form for algorithms
designed for standard images. However, this paper does not aim to make further
improvements in these areas, but to propose a method with the lowest possible
memory-computational complexity, while maintaining the highest performance.
Reducing memory-computational complexity promotes efficient information
processing in embedded devices, particularly those with the potential for
significant parallelisation of computation: it can provide low latency while
requiring little energy.

##  IV PowerYOLO Network

One of the most popular algorithms for object detection are networks based on
the YOLO (You Only Look Once) architecture [32]. However, due to their memory
and computational complexity, proper optimisation is required for efficient
inference of such architectures on low-power devices. In this section, we
first discuss the complexity of the YOLO network and then present the proposed
mixed-precision quantisation – power-of-two and linear.

###  IV-A YOLO computational and memory complexity

The neural networks from the YOLO “family” consist of three main blocks:
backbone for image feature extraction, neck for feature pyramid extraction (by
combining features from different levels of the backbone) and head as the
final stage determining bounding boxes, class labels, probabilities and
objectness scores. The architecture of the YOLO network has evolved over the
years, and the latest version widely accepted by the community as SoTA is
YOLOv8, available open source at [32]. Depending on the number of backbone
layers, the network comes in several sizes (by convention: n, s, m, l, x), and
for the purpose of this paper the commonly used (due to its good ratio of
memory and computational complexity to achieved accuracy) YOLOv8s version was
chosen.

The entire network has 7.2 M parameters (and thus, in the classical approach,
32- or 16-bit floating-point values), of which almost $79\%$ are the weights
of the convolution layers. Linear quantisation of the weights and activations
to 8-bit numbers generally preserves the accuracy of the full-precision
network, while reducing the memory complexity by 4x (relative to 32-bit
numbers) and increasing the throughput of the mathematical operations by 16x
[33]. If a further reduction in the precision of the convolution layer weights
to 4-bit values is made, a very compact architecture both in terms of memory
and computation is obtained. In such a case, according to [33], the throughput
of mathematical operations on the GPU increases up to x32. Implementing such a
mixed-precision network (with 4-bit weights and 8-bit activations) in both
eGPUs and dedicated processors therefore allows for significant improvements
in inference time, while reducing memory complexity by almost x8. Using a
logarithmic quantisation scheme of weights to powers of two, we will
additionally enable the conversion of multiply-accumulate (MAC – Multiply
ACcumulate) operations to shift-accumulate (BAC – Bitshift ACcumulate)
operations, ultimately introducing a radical simplification in neural network
computation.

###  IV-B Mixed Precision Quantization

In this work, we propose to use mixed quantisation, with convolution layer
weights quantised logarithmically to 4-bit powers of two (so each weight can
be represented by $2^{n}$ where $n$ is integer) and activations quantised
linearly to 8-bit values. Bias is customarily quantised to a 32-bit number to
avoid overflow during the accumulation of convolution results. Linear
quantisation of 8-bit activations allows the network to maintain high
performance, without the need to re-train the quantised network, but only by
calibrating the values of the scaling factors and activation offset on some
representative set of input data (customarily a subset of the training data).

Using PoT weights quantisation, which concentrates more quantisation intervals
around zero thus mimics the shape of the distribution of weight values in the
convolution layer, we can achieve better accuracy results with lower than
standard 8-bit bit widths.

PoT quantisation is used in the so-called quatisation aware training (QAT).
The weights of the quantised network are initialised with the trained full-
precision network and then the quantised network is trained (fine-tuned) for a
relatively small number of epochs. Due to the discontinuity of the
quantisation function, the gradient method cannot be used explicitly. The
forward pass is done using quantised weights, and during backpropagation the
gradients are computed using full precision equivalents (after the weights are
updated, the network is re-quantised).

Another method of reducing memory and computational complexity in neural
network inference is the fusion of convolution layers with batch normalisation
layers according to Eq.(1):

$\small
y_{bn}=\frac{x_{conv}-\mu}{\sqrt{\sigma^{2}-\epsilon}}\gamma+\beta=\frac{\gamma}{\sqrt{\sigma^{2}-\epsilon}}x_{conv}-\mu\frac{\gamma}{\sqrt{\sigma^{2}-\epsilon}}+\beta$
(1)

where $x_{conv}$ is the output of the convolution layer. Thus, the weights $w$
and the bias $b$ of the filters for inference are modified accordingly:
$w_{fused}=\frac{\gamma}{\sqrt{\sigma^{2}-\epsilon}}*w$ and
$b_{fused}=b-\mu\frac{\gamma}{\sqrt{\sigma^{2}-\epsilon}}+\beta$.

Naturally, such a modification does not affect the accuracy of the neural
network, but it reduces the required number of multiplication and addition
operations, and the number of parameters. Introducing exactly such a fusion to
a power-of-two network would result in the weights no longer being in the form
of powers of two, and therefore losing an important property of the described
method. In order to reduce the number of operations, we propose a small yet
powerful modification. First, since the bias is quantised linearly, it can be
successfully modified according to the standard fusion scheme (Eq. (1)). Next,
we propose to combine the multiplier usually associated with weights not with
weights, but with the scaling factor of the quantisation operation. In this
way, we reduce as much as possible all the additional computations introduced
by batch normalisation layer, while only slightly increasing the memory
complexity relative to the model after standard fusion - instead of one
scaling factor for the whole layer, we get scaling factors for each output
feature map separately, and the batch normalisation completes during re-
quantisation between layers.

Ultimately, we obtain a mixed-precision model, efficient in terms of memory
and computational complexity, for which the calculations in the standard
convolution layer – batch normalisation block can be written with the
simplified equation (2), where $\phi=\sqrt{\sigma^{2}-\epsilon}$:

$\small
y_{q}=\frac{\gamma}{\phi}\frac{s_{x}s_{w}}{s_{y}}(\sum_{i=0}^{k}\sum_{j=0}^{k}x_{q}^{ij}>>w_{pot}^{ij}+B_{q})$
(2)

where $s_{w},s_{x},s_{y}$ are the scaling factors for quantising the weights,
block inputs and block outputs, $k$ is the dimensions of the convolution
filter, $x_{q}$is the activation from the previous layer quantised to INT8,
$B_{q}$ is the modified bias $B=\frac{\gamma(b-\mu)}{\phi}+\beta$ quantised to
INT16, and $w_{pot}$ are weights in the INT4 format, in powers of two.

For re-quantisation between successive layers of the neural network during
inference, for PoT quantisation the factor
$\frac{\gamma}{\phi}\frac{s_{x}s_{w}}{s_{y}}$ is in a form of vector. In
comparison, in case of linear quantisation with standard convolution and batch
normalisation fusion, it’s a scalar (with value $\frac{s_{x}s_{w}}{s_{y}}$).
For simplicity, the indices indicating the output feature map are omitted in
the equation.

Although the granularity of quantisation can be chosen in many ways, in this
paper we chose the layer-wise quantisation. This decision is based on the
following facts: firstly, the shape of the distribution of weights in the
convolution layer of the network is logarithmic; secondly, other schemes are
characterised by higher computational complexity. For example in the case of
the channel-wise scheme, quantisation is performed for each filter separately,
which significantly increases the training time. For the network in question,
experiments indicated as much as a 25 times increase in the duration of one
epoch. However, due to the proposed convolution and batch normalisation fusion
scheme, channel-wise quantisation would not introduce additional computation
during inference, and the impact on accuracy remains an open topic.
Nonetheless, as shown in section V, the layer-wise scheme allows for very high
accuracy.

##  V Results

In this section, we present the results of the proposed PowerYOLO network for
the task of object detection on event camera GEN1 dataset. We present
comparison both with other solutions for the same task, and with other
networks belonging to the TinyML group. In addition, we show the applicability
of the obtained results for building a hardware accelerator.

###  V-A Mixed Precision YOLO

Neural network training was performed based on YOLO8s code available in [32],
suitably modified to perform quantised training. The full precision network
was trained for 100 epochs with 4 A100 NVIDIA GPUs, using the SGD method, with
a momentum of $0.9$ and an initial learning rate of $0.01$ reduced linearly to
$0.0001$. The weights of the full precision network were then used to
initialise the quantised model. The quantised network was trained for 20
epochs, using single A100 NVIDIA GPU, also using SGD and with a low initial
learning rate value of $0.0001$. The value of the learning rate was reduced at
epochs 5, 8 and 15 according to $lr=\gamma lr$, where $\gamma=0.1$. The
Exponentially Moving Average (EMA) model was used during training – the final
model parameters were a weighted average of the parameters from each training
iteration (this average is quantised logarithmically, and the EMA model is
quantised after each update). In the conducted experiments, it was found that
disabling EMA in the quantised learning process results in a decrease in the
performance of the network in terms of mAP by several percents. Table II shows
the training results of the Yolov8s network on the Gen1 dataset. The mAP50 and
mAP50-95 are averaged over the two object classes specified: vehicles and
pedestrians. In general, the metric is defined as area under the precision-
recall curve and is widely used for evaluation of detection algorithms.
Quantisation to 8-bit integer values after custom QAT PoT training was
performed using the OpenVino library (Fig. 2 demonstrates the detector
operation).

Figure 2: Example results of object detection using PowerYOLO mixed precision
network (bottom) compared to labels (top)

.

TABLE II: Object detection performance for networks: full-precision (baseline), with uniform (INT*) and PoT quantisation (LOG*).  Architecture |  Quantisation |  mAP50 |  mAP50-95
---|---|---|---
YOLO8s |  Baseline |  0.586 |  0.340
|  INT8 W/A |  0.582 |  0.330
|  INT4W |  0.523 |  0.288
|  LOG4W |  0.566 |  0.312
|  LOG4W/INT8A |  0.554 |  0.301

A decrease in detection performance is evident for both networks quantised to
INT8 and LOG4, with the former being approximately $1\%$ (mAP50), $3\%$
(mAP50-95) and the latter $3.5\%$ (mAP50), $8.3\%$ (mAP50-95) and $5.5\%$
(mAP50), $11.5\%$ (mAP50-95) depending on whether other parameters and
activations are also quantised. Using the same QAT approach, we also trained a
model with 4-bit width uniformly quantised weights. The gap to baseline model
is almost 2 times higher than in case of LOG4 model, which shows the
superiority of logarithmic quantisation. However, it’s worth noting that there
are efforts in the literature to minimise the gap between full precision and
low-bitwidth uniformly quantised models by introducing additional mechanisms,
like in [34].

The comparison with SoTA has to be done in two steps, as to the authors’
knowledge there are no other solutions that process event data using YOLO
networks with very low computational precision. Thus, it is first necessary to
refer to other object detectors developed on the Gen1 dataset. A comparison
with the full-precision SoTA for the Gen1 dataset is shown in Table III. To
enable the best possible comparison, we have trained the YOLOv8l network in
addition to the YOLOv8s network. The solution proposed in [30] uses the
YOLOv5l network with a slightly different representation of the event data.
Without the additional mechanisms proposed by the authors, and therefore with
the difference only due to the way the data are processed before and after the
neural network, and the way the event data are represented, the difference is
small, less than $2\%$ for the mAP50-95 index in favour of [30]. On the other
hand, considering a model with Hyper Histogram event representation and with
data augmentation adapted to the event representation, the difference is more
than $20\%$.

TABLE III: Comparison with SoTA full precision for Gen1 dataset against mAP and model size.  Solution |  Event representation |  Architecture |  mAP50-95 |  No Parameters
---|---|---|---|---
Ours |  Event frame |  YOLOv8s |  0.347 |  11.2M
|  |  YOLOv8l |  0.375 |  43.7M
Baseline [30] |  ITS |  YOLOv5l |  0.382 |  46.5M
AEC [30] |  HH + SM |  |  0.470 |

Secondly, low-precision YOLO networks should also be analysed. Table IV
compares our solution with other 4-bit detectors in which all convolution
layer weights except the first and last are quantised. It is worth noting that
in [22] the solution is based on Post Training Quantisation, which, as a rule,
with such low precision computatnois, does not allow to maintain satisfactory
performance. In [34], the authors also quantised the activations, so a direct
comparison to our model in any version is not possible – even comparing to the
mixed-precision version, for which in our case the difference to the floating-
point version is about $11.5\%$, it is hard to say unequivocally which model
performs better. In the case of the mixed-precision version, we additionally
quantise the bias and the batch normalisation parameters, which has, as the
results presented in Table II indicate, a major impact.

Nevertheless, our solution is comprehensive, i.e. it reduces computation on
floating-point numbers to a minimum (actually limiting it only to re-
quantisation operations, which is unavoidable with any quantisation) and
targets all elements (all layers) of the information flow in the neural
network. For this reason, we believe that, in terms of ultimate efficiency
with respect to memory-computational complexity, our solution is at least as
good as [34], especially given the potential for reducing computational
complexity introduced by using quantisation for weights with powers of two (as
we show in section LABEL:subsec:hw_design), and thus we set a new SoTA in this
category.

TABLE IV: Comparison of different SoTAs of low-precision YOLO detectors. It is worth noting that the other methods listed in the table are based on processing classical images and only ours treats event data.  Solution |  Architecture |  Dataset |  Method |  Bitwidth |  mAP50-95 gap
---|---|---|---|---|---
Ours |  Yolov8s |  Gen1 |  PoT QAT, layerwise |  4bits W |  8.3%
EMA+QC [34] |  Yolov5s |  COCO |  Uni QAT, layerwise |  4bits W&A |  10.4%
Q-Yolo [22] |  Yolov5s |  COCO |  Uni PTQ, channelwise |  4bits W |  52.4%

Moreover, as was already established in [3] and [19], the hardware
implementation of neurons and layers based on BAC (Bitshift Accumulate) units
leads to significant area and power reductions. For a single processing
element with weights of bit width 4 and 8-bit activations, using PoT weights
allows to increase the power efficiency by a factor of 2. Using this PE for a
complete convolution layer leads to $1.6$ increase of power efficiency and
allows for higher frequency operation, as shown for ZCU 104 platform.

##  VI Conclusion

In this work we presented an extremely efficient, mixed precision PowerYOLO
detector achieving mAP of $0.301$ for the GEN1 event dataset. By using both
logarithmic quantisation of weights and linear quantisation of activations and
other parameters, we eliminated most of floating point operations in the
inference flow, allowed replacing multiplication with efficient bit-shifting,
and reduced the model size almost 8x while still proposing an accurate
detector. In order to close the gap between floating point solutions, for
future work we consider extending the training flow with shadow mosaic
augmentation (proper for event data) and using other event representation
techniques, possibly allowing to deal with slowly moving and static objects.
Our initial thesis stated that for embedded applications one should address
the algorithm, hardware and sensor (data) at once, and with our solution we
show that comprehensive analysis of all those elements leads to truly tiny,
yet powerful, machine learning solution.

## References

  * [1] Elhoushi, M., Chen, Z., Shafiq, F., Tian, Y. H., and Li, J. Y. (2021). DeepShift: Towards Multiplication-Less Neural Networks. 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pp. 2359-2368.
  * [2] Li, Y., Dong, X., and Wang, W. (2019). Additive Powers-of-Two Quantization: A Non-uniform Discretization for Neural Networks. CoRR, volume abs/1909.13144, 2019. Available at: http://arxiv.org/abs/1909.13144.
  * [3] Przewlocka-Rus, D., Sarwar, S. S., Sumbul, H. E., Li, Y., and De Salvo, B. (2022). Power-of-Two Quantization for Low Bitwidth and Hardware Compliant Neural Networks. 2022\. arXiv preprint https://arxiv.org/abs/2203.05025 [cs.LG].
  * [4] Lichtsteiner, P., Posch, C. & Delbruck, T., A 128× 128 120 dB 15 $\mu$s Latency Asynchronous Temporal Contrast Vision Sensor in IEEE Journal of Solid-State Circuits, vol. 43, no. 2, pp. 566-576, Feb. 2008, doi: 10.1109/JSSC.2007.914337.
  * [5] Zhang, H., Li, Y., Leng, L., Che, K., Liu, Q., Guo, Q., Liao, J., & Cheng, R. (2023). Automotive Object Detection via Learning Sparse Events by Spiking Neurons. https://doi.org/10.48550/arXiv.2307.12900
  * [6] Gehrig, D., & Scaramuzza, D. (2022). Pushing the limits of asynchronous graph-based object detection with event cameras. arXiv preprint https://arxiv.org/abs/2211.12324 [cs.CV]
  * [7] Perot, E., de Tournemire, P., Nitti, D., Masci, J., & Sironi, A. (2020). Learning to detect objects with a 1 megapixel event camera. NIPS’20: Proceedings of the 34th International Conference on Neural Information Processing Systems, pages 16639–16652.
  * [8] Rebecq, H., Ranftl, R., Koltun, V., and Scaramuzza, D., High Speed and High Dynamic Range Video with an Event Camera in IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 43, no. 6, pp. 1964-1980, 1 June 2021, doi: 10.1109/TPAMI.2019.2963386.
  * [9] Liu, M. et al. (2017). Block-matching optical flow for dynamic vision sensors: Algorithm and FPGA implementation. 2017 IEEE ISCAS, 1–4. https://doi.org/10.1109/ISCAS.2017.8050295.
  * [10] Afshar, S. et al. Event-Based Feature Extraction Using Adaptive Selection Thresholds. Sensors 2020, 20, 1600. https://doi.org/10.3390/s20061600.
  * [11] Chen, N. Y. (2018). Pseudo-Labels for Supervised Learning on Dynamic Vision Sensor Data, Applied to Object Detection Under Ego-Motion. 2018 IEEE/CVF Conf. CVPR Workshops, 757–75709. https://doi.org/10.1109/CVPRW.2018.00107.
  * [12] Abadi, M., et al. (2015). TensorFlow: Large-scale machine learning on heterogeneous systems. Software available from tensorflow.org. https://www.tensorflow.org/
  * [13] OpenVINO toolkit. Available at: https://github.com/openvinotoolkit/openvino.
  * [14] Pappalardo, A., (2023). Xilinx/brevitas. Zenodo. https://doi.org/10.5281/zenodo.3333552
  * [15] Umuroglu, Y., Fraser, N.J., Gambardella, G., Blott, M., Leong, P., Jahre, M., Vissers, K. (2017) Finn: A framework for fast, scalable binarized neural network inference. In: Proceedings of the 2017 ACM/SIGDA International Symposium on Field-Programmable Gate Arrays. pp. 65–74. FPGA ’17, ACM. https://doi.org/10.48550/arXiv.1612.07119
  * [16] Xilinx, VitisAI deveop environment, https://www.xilinx.com/products/design-tools/vitis/vitis-ai.html
  * [17] Kozlov, A., Lazarevich, I., Shamporov, V., Lyalyushkin, N. & Gorbachev, Y., (2020). Neural network compression framework for fast model inference. arXiv preprint arXiv:2002.08679
  * [18] Miyashita, D., Lee, E. H., and Murmann, B. (2016). Convolutional Neural Networks using Logarithmic Data Representation. CoRR, volume abs/1603.01025, 2016. Available at: http://arxiv.org/abs/1603.01025
  * [19] Przewlocka-Rus, D., and Kryjak, T. (2023). Energy Efficient Hardware Acceleration of Neural Networks with Power-of-Two Quantisation. In: Chmielewski, L.J., Orłowski, A. (eds) Computer Vision and Graphics. ICCVG 2022. Lecture Notes in Networks and Systems, vol 598. Springer, Cham. https://doi.org/10.1007/978-3-031-22025-8_1 ]
  * [20] Wang, Z., Wu, Z., Lu. J, and Zhou, J. (2020). BiDet: An Efficient Binarized Object Detector. CoRR, volume abs/2003.03961. Available at: https://arxiv.org/abs/2003.03961.
  * [21] Gupta, K., and Asthana, A. (2023). Reducing the Side-Effects of Oscillations in Training of Quantized YOLO Networks. arXiv:2311.05109.
  * [22] Wang, M. et al. (2023). Q-YOLO: Efficient Inference for Real-Time Object Detection. In: Lu, H., Blumenstein, M., Cho, SB., Liu, CL., Yagi, Y., Kamiya, T. (eds) Pattern Recognition. ACPR 2023. Lecture Notes in Computer Science, vol 14408. Springer, Cham. https://doi.org/10.1007/978-3-031-47665-5_25
  * [23] Xu, J., Huan, Y., Jin, Y. et al. (2020). Base-Reconfigurable Segmented Logarithmic Quantization and Hardware Design for Deep Neural Networks. Springer, Journal of Signal Processing Systems, vol. 92, 2020. https://doi.org/10.1007/s11265-020-01557-8
  * [24] Lee, E. H.,iyashita, D., Chai, E.,Murmann, B. and Wong, S. S. (2017). LogNet: Energy-efficient neural networks using logarithmic computation. In: 2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). DOI: 10.1109/ICASSP.2017.7953288.
  * [25] de Tournemire, P., Nitti, D., Perot, E., Migliore, D., & Sironi, A. (2020). A Large Scale Event-based Detection Dataset for Automotive. arXiv: 2001.08499.
  * [26] Messikommer, N., Gehrig, D., Loquercio, A., & Scaramuzza, D. (2020). Event-based Asynchronous Sparse Convolutional Networks. European Conference on Computer Vision (ECCV).
  * [27] Su, Q., Chou, Y., Hu, Y., Li, J., Mei, S., Zhang, Z., & Li, G. (2023). Deep Directly-Trained Spiking Neural Networks for Object Detection. Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 6555-6565
  * [28] Peng, Y., Zhang, Y., Xiong, Z., Sun, X., Wu, F. (2023). GET: Group Event Transformer for Event-Based Vision. International Conference on Computer Vision (ICCV) 2023.
  * [29] Gehrig, M., & Scaramuzza, D. (2023). Recurrent Vision Transformers for Object Detection with Event Cameras. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) 2023.
  * [30] Peng, Y., Zhang, Y., Xiao, P., Sun, X., & Wu, F. (2023). Better and Faster: Adaptive Event Conversion for Event-Based Object Detection. Proceedings of the AAAI Conference on Artificial Intelligence, 37(2), 2056-2064. https://doi.org/10.1609/aaai.v37i2.25298
  * [31] B. Liu, C. Xu, W. Yang, H. Yu and L. Yu, ”Motion Robust High-Speed Light-Weighted Object Detection With Event Camera,” in IEEE Transactions on Instrumentation and Measurement, vol. 72, pp. 1-13, 2023, Art no. 5013113, doi: 10.1109/TIM.2023.3269780.
  * [32] Ultralytics. Ultralytics YOLOv8. Available at: https://github.com/ultralytics/ultralytics. Accessed: 3rd February, 2024
  * [33] Wu, H., Judd, P., Zhang, X., Isaev, M., & Micikevicius, P. (2020). Integer Quantization for Deep Learning Inference: Principles and Empirical Evaluation. ArXiv, abs/2004.09602.
  * [34] Gupta, K., & Asthana, A. (2023). Reducing the Side-Effects of Oscillations in Training of Quantized YOLO Networks. IEEE/CVF Winter Conference on Applications of Computer Vision (WACV) 2024