# Enhancing Low-resolution Face Recognition with Feature Similarity Knowledge
Distillation

Sungho Shin
School of Integrated Technology (SIT)
Gwangju Institute of Science and Technology (GIST)
<EMAIL_ADDRESS>
&Yeonguk Yu
School of Integrated Technology (SIT)
Gwangju Institute of Science and Technology (GIST)
<EMAIL_ADDRESS>
&Kyoobin Lee
School of Integrated Technology (SIT)
Gwangju Institute of Science and Technology (GIST)
<EMAIL_ADDRESS>
###### Abstract

In this study, we introduce a feature knowledge distillation framework to
improve low-resolution (LR) face recognition performance using knowledge
obtained from high-resolution (HR) images. The proposed framework transfers
informative features from an HR-trained network to an LR-trained network by
reducing the distance between them. A cosine similarity measure was employed
as a distance metric to effectively align the HR and LR features. This
approach differs from conventional knowledge distillation frameworks, which
use the $L_{p}$ distance metrics and offer the advantage of converging well
when reducing the distance between features of different resolutions. Our
framework achieved a 3% improvement over the previous state-of-the-art method
on the AgeDB-30 benchmark Moschoglou et al. (2017) without bells and whistles,
while maintaining a strong performance on HR images. The effectiveness of
cosine similarity as a distance metric was validated through statistical
analysis, making our approach a promising solution for real-world applications
in which LR images are frequently encountered. The code and pretrained models
are publicly available on https://github.com/gist-ailab/feature-similarity-KD.

_Keywords_ Low-resolution Face Recognition $\cdot$ Feature Similarity $\cdot$
Knowledge Distillation

##  1 Introduction

Deep learning has been widely utilized for face recognition tasks because of
its superior performance Deng et al. (2019, 2021); Huang et al. (2020); Li et
al. (2021). A recent study reported accuracy rates of 99% on the MegaFace
benchmark, which involves identifying a given image among one million
distractors Kemelmacher-Shlizerman et al. (2016). However, in real-world
situations, face images may be captured under a variety of challenging
conditions, such as low-resolution (LR), low-illumination, and small size,
which can significantly degrade the performance of deep-learning models. These
conditions are not well represented in existing face recognition benchmarks,
such as CASIA Yi et al. (2014) and MegaFace Kemelmacher-Shlizerman et al.
(2016), which comprise high-resolution (HR) images with a resolution of at
least 112 pixels in width and height. LR images, on the other hand, may
contain as few as 64 pixels (8 $\times$ 8) in extreme cases as depicted in
Figure 1.

Figure 1: Example images of high-resolution (HR) and low-resolution (LR)
samples of faces and vehicle registration plates. While various approaches
have been proposed for face identity and digit recognition in HR images, as
shown in (a), many face and registration plate images are frequently captured
in low-resolution circumstances, as depicted in (b) and (c).

The resolution of face images can affect the accuracy of face recognition
models. When the resolution is degraded, the models may not capture sufficient
detail for accurate recognition, owing to the loss of spatial information.
When the high-accuracy margin network (ArcFace Deng et al. (2019)) was tested
on LR datasets (e.g., IJB-B Whitelam et al. (2017), IJB-C Maze et al. (2018),
and TinyFace Cheng et al. (2018)), a significant degradation was observed
(accuracies below 70%). Kim et al. noted that significant resolution
degradation can make images unidentifiable, causing models to rely on other
features, such as hairstyle and clothing, which can bias model training Kim et
al. (2022). Recent studies have addressed this by imposing different weights
based on the image quality Chang et al. (2020); Kim et al. (2022); Liu et al.
(2021); Tran et al. (2017). For example, AdaFace demonstrated that the feature
norm is positively correlated with the image quality and designed an adaptive
margin based on the feature norm to emphasize high-quality hard samples and
de-emphasize low-quality ones Kim et al. (2022). These approaches aim to
select informative images from a large number of mixed-resolution face images,
but do not focus on extracting informative features from LR images.

Other studies Kim et al. (2021); Massoli et al. (2020); Shin et al. (2022)
employed the knowledge distillation (KD) framework to extract informative
features from LR images based on guidance from a teacher network trained on HR
images. These approaches encourage the intermediate features produced by the
face recognition network at lower resolutions to resemble those produced by a
teacher network trained using HR images. For example, QualNet-LM Kim et al.
(2021) uses a face recognition network with a decoder structure trained on HR
images as the teacher network. To extract informative features from LR images,
they added a reconstruction loss to regulate the student network’s LR features
such that they could be decoded into HR images when passed through the decoder
of the teacher network. A-SKD Shin et al. (2022) uses KD to transfer well-
constructed attention maps from the HR network to the attention maps of the LR
network, achieving state-of-the-art (SOTA) results on various LR face
recognition benchmarks. However, these methods are limited to specific
frameworks (e.g., decoder, and attention modules) to align the features of the
teacher and student networks at different resolutions.

In this study, we present a simple approach for knowledge transfer between an
HR and LR face recognition network using feature similarity knowledge
distillation (F-SKD). We aim to align the features produced by both networks
and transfer the knowledge obtained from the HR network to the LR network
without additional modules. Through experiments and statistical analyses, we
found that cosine similarity is key to reducing the distance between features
of different resolutions, outperforming traditional KD frameworks that use
$L_{p}$ distance metrics. Our approach outperformed previous SOTA results on
LR face recognition tasks by 3% for AgeDB-30 Moschoglou et al. (2017) and
1.55% for TinyFace Cheng et al. (2018) without bells and whistles. Even in the
LR digit classification task, our approach outperformed the previous methods
by 1.73% on the SVHN benchmark Netzer et al. (2011). F-SKD is a promising
solution for LR face recognition applications in the real-world, owing to its
efficiency and ease of implementation.

##  2 Related Works

Face Recognition. Face recognition networks must differentiate between a large
number of face images belonging to different identities in open-set
environments Deng et al. (2019); Liu et al. (2017); Wang et al. (2018). The
traditional softmax loss function lacks a margin between the decision
boundaries of different classes, which can result in incorrect classifications
owing to the small perturbations. To address this issue, several margin-based
face recognition losses have been proposed to increase the similarity of
intra-class samples and dissimilarity of inter-class samples Deng et al.
(2019); Liu et al. (2017); Wang et al. (2018). SphereFace Liu et al. (2017)
introduced the angular-softmax (A-softmax) loss, which multiplies the margin
($m$) with the angle ($\theta$) between the feature vector ($\mathbf{x}$) and
weight vector ($\mathbf{W}_{y}$) of the target class ($y$). However, the
margin of A-softmax loss is angle-dependent and disappears when the angle
becomes zero. CosFace Wang et al. (2018) avoids this problem by subtracting
$m$ from $cos(\theta)$. ArcFace Deng et al. (2019) proposed adding $m$
directly to $\theta$ resulting in $cos(\theta+m)$, which provides a constant
linear angular margin throughout the interval, whereas CosFace has only a
nonlinear angular margin. These angular margin-based loss functions have
become the standard for face recognition owing to their superior performance
and simplicity of implementation.

Low-resolution Face Recognition. LR face recognition studies have focused on
leveraging the knowledge obtained from HR images to improve LR face
recognition performance using KD frameworks Kim et al. (2021); Massoli et al.
(2020); Shin et al. (2022). A teacher network trained on HR images can extract
more informative features for recognition, which can then be transferred to a
student network trained on LR images. Massoli et al. demonstrated that
reducing the feature distance between HR and LR networks significantly
enhances LR recognition performance Massoli et al. (2020). Kim et al. proposed
the QualNet-LM framework that employs a face recognition network with a
decoder structure trained on HR images as the teacher network Kim et al.
(2021). To effectively extract information from the LR images, a
reconstruction loss was incorporated to regulate the LR features of the
student network, enabling them to be decoded into HR images when passed
through the decoder of the teacher network. Shin et al. suggested utilizing KD
to transfer well-designed attention maps from an HR network to the attention
maps of an LR network, obtaining SOTA results on various LR face recognition
benchmarks Shin et al. (2022). However, existing studies are limited to
specific frameworks, such as decoders and attention modules, to align features
of varying resolutions. This study investigates an efficient KD approach that
can align different resolution features without being restricted to any
specific frameworks, achieving superior performance.

##  3 Proposed Approach

This section first introduces the CosFace angular margin face recognition
network Wang et al. (2018), which was utilized in the previous KD method
QualNet-LM (Sec.3.1). We then describe the F-SKD approach, which increases the
similarity between the teacher and student network features (Sec.3.2).
Finally, we present the t-test Student (1908) and Pearson’s correlation
analysis Pearson (1896) to evaluate the effectiveness of the proposed cosine
similarity-based distance metric (Sec.3.3).

###  3.1 Recap: CosFace

For the recognition task, softmax loss is widely utilized to separate features
from different classes by maximizing the posterior probability of the labels.
Given an input feature vector of $i$th sample
$\mathbf{x}_{i}\in\mathbb{R}^{d}$ with its corresponding label $y_{i}$, the
softmax loss can be formulated as

$L_{softmax}=-\frac{1}{N}\sum_{i=1}^{N}log\frac{e^{\mathbf{W}^{T}_{y_{i}}\mathbf{x}_{i}}}{\sum_{j=1}^{n}e^{\mathbf{W}^{T}_{j}\mathbf{x}_{i}}},$
(1)

$N$ and $n$ are the batch size and number of classes, respectively,
$\mathbf{W}_{j}\in\mathbb{R}^{d}$ denotes the $j$th column of the last fully
connected layer’s weight $\mathbf{W}\in\mathbb{R}^{d\times n}$. For
simplicity, we set the bias terms to zero.

For effective feature learning, previous studies Liu et al. (2017); Wang et
al. (2018) have fixed $\lvert\lvert\mathbf{W}_{j}\rvert\rvert_{2}=1$ and
$\lvert\lvert\mathbf{x}_{i}\rvert\rvert_{2}=s$ to ensure that the posterior
probability relies only on the cosine angle between the feature and fully
connected layer’s weights. We can then rewrite the term
$\mathbf{W}^{T}_{j}\mathbf{x}_{i}=\lvert\lvert\mathbf{W}_{j}\rvert\rvert_{2}\lvert\lvert\mathbf{x}_{i}\rvert\rvert_{2}cos(\theta_{j})=scos(\theta_{j})$,
where $\theta$ denotes the angle between $\mathbf{x}_{i}$ and
$\mathbf{W}_{j}$. Finally, CosFace Wang et al. (2018) introduced the cosine
margin ($m$) to strengthen the discrimination between the cosine angles of
different classes: $cos(\theta_{i})-m>cos(\theta_{j})$ and
$cos(\theta_{j})-m>cos(\theta_{i})$, where $i$ and $j$ denote different
classes. CosFace Wang et al. (2018) loss can be formulated as:

$L_{cosface}=-\frac{1}{N}\sum_{i=1}^{N}log\frac{e^{s(cos(\theta_{y_{i}})-m)}}{e^{s(cos(\theta_{y_{i}})-m)}+\sum_{j\neq
y_{i}}e^{scos(\theta_{j})}}.$ (2)

###  3.2 Feature Similarity Knowledge Distillation

To extract informative features from LR images, we utilized a KD framework
that uses paired HR-LR samples as input (Figure 2). This framework aims to
reduce the feature difference between the corresponding locations in the
paired samples, thereby enabling the LR network to extract richer features at
a level comparable to that of the HR network. The $L_{p}$ distance function is
employed for the KD loss in general object classification tasks Ahn et al.
(2019); Park et al. (2019); Romero et al. (2014); Yim et al. (2017). However,
in this study, we proposed the use of cosine similarity-based distance
function.

Our loss function was designed to minimize the angle between the feature
vectors of the teacher and student networks. Previous studies have shown that
feature norm is positively correlated with image quality, and an adaptive
margin function has been introduced to emphasize samples with high-quality
features based on their norm. Following this, we transfer only the direction
component of the feature vector (the normalized feature vector) from the
teacher network, instead of trying to match the norm component of the student
network’s features to those of the teacher’s. This approach enables the LR
network to learn where to focus among the feature elements for extracting
richer information guided by the HR network while avoiding bias towards other
characteristics such as clothing and hairstyle.

Let $\mathbf{f}_{i}=\mathcal{H}_{i}(\mathbf{I})$ be the intermediate feature
outputs from the ith block of the CNN for the given image $\mathbf{I}$. The
cosine similarity-based KD loss can be described as follows:

$\begin{split}L_{distill}&=\frac{1}{L}\sum_{i=1}^{L}(1-\langle\mathbf{f}_{T,i},\mathbf{f}_{S,i}\rangle)\\\
&=\frac{1}{L}\sum_{i=1}^{L}(1-\frac{\mathbf{f}_{T,i}}{\lVert\mathbf{f}_{T,i}\rVert_{2}}\cdot\frac{\mathbf{f}_{S,i}}{\lVert\mathbf{f}_{S,i}\rVert_{2}}),\end{split}$
(3)

where $\langle\cdot,\cdot\rangle$ indicates the cosine similarity operation,
$L$ denotes the number of blocks employed for the distillation, and
$\mathbf{f}_{T,i}$ and $\mathbf{f}_{S,i}$ denote the ith CNN layer’s features
from the teacher and student networks, respectively.

Figure 2: Proposed F-SKD framework where the HR network guides the LR network
to extract informative features, such as eyes, nose, and lips by improving the
similarity between their respective features. The figure shows F-SKD only for
the first block.

The total loss for the LR face recognition network is the sum of the face
recognition loss (2), CosFace is utilized in this study, and the cosine
similarity-based KD loss (3) weighted by the factor ($\lambda_{distill}$).

$L_{total}=L_{cosface}+L_{distill}*\lambda_{distill}.$ (4)

###  3.3 Statistical Analysis

In this study, we propose using similarity loss (3) to distill only the
directional components of the teacher network’s features in the student
network. To assess the effectiveness of this approach, we used a t-test to
determine if the norm of the student network’s features is statistically the
same as that of the teacher network after distillation and performed Pearson’s
correlation analysis to determine if the direction of the student network’s
features aligns with that of the teacher network after distillation.

t-test. t-test Student (1908) is a statistical hypothesis test that tests the
null hypothesis that the mean difference between two groups of samples is
equal to zero, against the alternative hypothesis that the mean difference is
not zero. This is based on the assumption that data are normally distributed.
To validate the null hypothesis, $\mu_{x}-\mu_{y}=0$, where $\mu$ is the mean
statistic of each group, the t-test uses the t-statistic, which can be
calculated as follows:

$t=\frac{\mu_{x}-\mu_{y}}{\sqrt{(s_{x}^{2}+s_{y}^{2})/n}},$ (5)

where $s_{x}$ and $s_{y}$ are the standard deviations of the two groups and
$n$ is the number of observations in each group.

The calculated t-value is then compared to the critical value ($c$), which is
determined based on the significance level ($\alpha$). If the calculated
t-value is greater than the critical value, then we reject the null
hypothesis. The critical value $c$ can be determined by the condition
$\int_{c}^{\infty}f(t)dt=\alpha$, where $f(t)$ represents the probability
density function of the Student’s t-distribution. In the study, $\alpha=0.01$.

Pearson’s Correlation Analysis. The Pearson’s correlation analysis Pearson
(1896) measures the linear relationship between two continuous variables and
their strengths. The correlation coefficient was calculated using the
following equation:

$r=\frac{\sum_{i=1}^{n}(x_{i}-\mu_{x})(y_{i}-\mu_{y})}{\sqrt{\sum_{i=1}^{n}(x_{i}-\mu_{x})^{2}}\sqrt{\sum_{i=1}^{n}(y_{i}-\mu_{y})^{2}}},$
(6)

where $x_{i}$ and $y_{i}$ are the values of the two variables for the $i$th
observation, $\mu_{x}$ and $\mu_{y}$ are the means of the two variables, and
$n$ is the number of observations.

The range of the coefficient is between -1 and 1. If the coefficient is 1, it
means there is a perfect positive linear relationship between the variables.
Conversely, if the coefficient is -1, it indicates a perfect negative linear
relationship between the variables. On the other hand, if the coefficient is
0, it suggests that there is no linear relationship between the variables.

##  4 Experiments

###  4.1 Task

Face recognition is the process of identifying individuals in facial images.
There are two main types of face recognition methods: face verification and
face identification. In face verification, the system compares two facial
images to confirm whether they belong to the same person using a 1:1
comparison. In face identification, the system attempts to identify the person
in a probe image by comparing it to a larger set of images, known as the
gallery set, through a 1:N comparison. Digit classification, in contrast, is
the task of recognizing numbers in images containing a single digit. The
system classifies these images as 0–9.

###  4.2 Datasets

Face Recognition. We used the CASIA Yi et al. (2014) dataset, which contains
0.5 million face images and 10K identities, to train our face recognition
model. To create HR-LR paired face images, we downsampled the resolution of
the images from CASIA by reducing their size and then enlarged them to their
original size using bilinear interpolation. For the evaluation, we used the
AgeDB-30 Moschoglou et al. (2017) and TinyFace Cheng et al. (2018) datasets.
AgeDB-30 had 16,516 images with 570 subjects for the 1:1 verification task,
and TinyFace had 169,403 LR face images with 5,139 subjects for the 1:N
identification task. TinyFace is a well-known LR face recognition benchmark
captured in real-world settings.

Digit Classification. We used the SVHN Netzer et al. (2011) dataset, which has
character-level digit images (0 – 9) obtained from house numbers in Google
Street View images. It contained 73,257 and 26,032 digit images for training
and evaluation, respectively.

###  4.3 Settings

Evaluation Protocol. We downsampled images from CASIA with ratios of
2$\times$, 4$\times$, and 8$\times$ in accordance with recent LR face
recognition studies Kim et al. (2021); Shin et al. (2022). Here, 1$\times$
denotes the HR images. Two benchmark settings were employed for the
evaluation: (1) a single-resolution, where a student network is trained to
recognize images from one resolution, and (2) multiple-resolutions, where a
student network is trained to recognize images from multiple-resolutions.
Because AgeDB-30 has a similar resolution to CASIA, networks trained on
downsampled CASIA images were validated on AgeDB-30 downsampled images with
matching ratios and validated in both single- and multiple-resolution
settings. In contrast, TinyFace, a real-world LR benchmark, comprises face
images with different resolutions; therefore, networks trained on multiple-
resolution settings were validated using TinyFace. For digit classification,
we compared the classification accuracy on 4$\times$ downsampled SVHN images
by following a single-resolution evaluation setting.

Backbone. Our method employs the KD framework, which uses a teacher-student
network. To ensure fair comparison with previous studies, we used the same
backbone structures as those used in previous studies. We utilized the
iResNet50 Deng et al. (2019) backbone for both the teacher and student
networks compared with FitNet Romero et al. (2014) and RKD Park et al. (2019).
When compared with QualNet-LM Kim et al. (2021), we used iResNet50 for the
student network and iResNet50 combined with a decoder module (iRevNet-300
Jacobsen et al. (2018)) for the teacher network because QualNet-LM requires a
decoder module for the teacher network. Lastly, when compared with A-SKD Shin
et al. (2022), we used iResNet50 with CBAM modules for both the teacher and
student networks, as A-SKD requires attention modules to extract and transfer
the attention maps. For digit classification, we replaced iResNet50 with a
ResNet50 He et al. (2016) backbone, which is commonly used in classification
tasks.

Implementation Details. In this study, we followed the common practices
proposed by Deng et al. (2019) for preprocessing the face recognition dataset.
This process included using the MTCNN Zhang et al. (2016) face detector to
locate and crop the face region and then resizing the cropped image to 112
$\times$ 112 pixels using bilinear interpolation. To ensure a fair comparison,
all methods were re-implemented using the CosFace Wang et al. (2018) margin
and iResNet50 backbone by following the settings of the previous SOTA method
(QualNet-LM Kim et al. (2021)). The learning rate was set to 0.1 initially and
decreased by a factor of 10 at 18K, 28K, 36K, and 44K iterations. The SGD
optimizer was used with a batch size of 256. The training process finished
after 47K iterations. For digit classification, the learning rate was set to
0.01 initially and decreased by a factor of 10 at 30, 60, and 80 epochs. The
SGD optimizer was used with a batch size of 64. The training process was
completed after 90 epochs. For the hyper-parameter search
($\lambda_{distill}$), the CFP-FP Sengupta et al. (2016) face dataset was used
as an external validation set and a random search was conducted. Through the
search, $\lambda_{distill}=5$ for the face recognition task. We also employed
the same weight factor for the digit classification task and it generally
achieved superior performance.

##  5 Results and Discussion

###  5.1 Low-resolution Face Recognition

Evaluation on AgeDB-30. After training a face recognition network using
downsampled images from the CASIA dataset, we evaluated the performance of the
network using downsampled images from the AgeDB-30 dataset. First, we compared
our approach to previous studies by following a single-resolution setting
(Table 1). The F-SKD outperformed the previous SOTA methods in all cases
(2$\times$, 4$\times$, and 8$\times$). For the most extreme case (8$\times$),
F-SKD achieved 3.0% and 0.8% improvements compared with QualNet-LM Kim et al.
(2021) and A-SKD Shin et al. (2022), respectively. Second, when evaluated
using multiple-resolution settings, F-SKD outperformed the other methods on
average (Table 2). Although A-SKD slightly outperformed F-SKD at 8$\times$
downsampled images (+0.001%), it showed significant degradation in HR
recognition performance compared to the teacher network (-2.1%). In contrast,
our approach maintained superior HR recognition performance while
significantly improving LR recognition performance.

Table 1: Single-resolution evaluation results of proposed distillation
approaches on AgeDB-30. (a) Verification accuracy of FitNet, RKD, QualNet-LM,
and F-SKD using iResNet50 as the student network. (b) Verification accuracy of
A-SKD and F-SKD using iResNet50+CBAM as the student network. Base refers to a
network that has not undergone any KD methods. The term Dec denotes the
decoder structure utilized in the QualNet-LM. The highest scores among the
comparisons are shown in bold.

Resolution | Type | Teacher | ACC (%)
---|---|---|---
(a) |  |  |
112 $\times$ 112 | Base | iResNet50 | 92.63
iResNet50+Dec | 93.03
56 $\times$ 56 | Base | - | 91.35
FitNet Romero et al. (2014) | iResNet50 | 91.57
RKD Park et al. (2019) | 92.05
F-SKD (Ours) | 92.52
QualNet-LM Kim et al. (2021) | iResNet50+Dec | 91.37
F-SKD (Ours) | 92.10
28 $\times$ 28 | Base | - | 85.68
FitNet Romero et al. (2014) | iResNet50 | 86.05
RKD Park et al. (2019) | 86.02
F-SKD (Ours) | 87.42
QualNet-LM Kim et al. (2021) | iResNet50+Dec | 86.45
F-SKD (Ours) | 87.68
14 $\times$ 14 | Base | - | 73.67
FitNet Romero et al. (2014) | iResNet50 | 74.73
RKD Park et al. (2019) | 74.30
F-SKD (Ours) | 75.97
QualNet-LM Kim et al. (2021) | iResNet50+Dec | 74.33
F-SKD (Ours) | 76.55
(b) |  |  |
112 $\times$ 112 | Base | iResNet50+CBAM | 93.12
56 $\times$ 56 | Base | - | 90.78
A-SKD Shin et al. (2022) | iResNet50+CBAM | 91.62
F-SKD (Ours) | 91.70
28 $\times$ 28 | Base | - | 85.90
A-SKD Shin et al. (2022) | iResNet50+CBAM | 86.35
F-SKD (Ours) | 87.53
14 $\times$ 14 | Base | - | 73.62
A-SKD Shin et al. (2022) | iResNet50+CBAM | 74.78
F-SKD (Ours) | 75.35

Table 2: Multiple-resolution evaluation results of proposed distillation
approaches on AgeDB-30. (a) Verification accuracy of QualNet-LM and F-SKD on
four different resolutions (112$\times$112, 56$\times$56, 28$\times$28, and
14$\times$14) using iResNet50 as the student network. (b) Verification results
of A-SKD and F-SKD using iResNet50+CBAM as the student network.

Resolution | Type | Teacher | 112 $\times$ 112 (%) | 64 $\times$ 64 (%) | 28 $\times$ 28 (%) | 14 $\times$ 14 (%) | Average (%)
---|---|---|---|---|---|---|---
(a) |  |  |  |  |  |  |
112 $\times$ 112 | Base | iResNet50+Dec | 93.03 | - | - | - | -
All | Base | - | 90.27 | 89.63 | 85.58 | 74.58 | 85.02
QualNet-LM Kim et al. (2021) | iResNet50+Dec | 92.77 | 91.85 | 87.35 | 75.57 | 86.89
F-SKD (Ours) | 92.90 | 91.88 | 87.35 | 76.02 | 87.04
(b) |  |  |  |  |  |  |
112 $\times$ 112 | Base | iResNet50+CBAM | 93.12 | - | - | - | -
All | Base | - | 90.57 | 89.95 | 85.60 | 74.32 | 85.11
A-SKD Shin et al. (2022) | iResNet50+CBAM | 91.12 | 90.35 | 86.03 | 75.32 | 85.71
F-SKD (Ours) | 92.58 | 91.50 | 86.57 | 75.23 | 86.47

Evaluation on TinyFace. TinyFace Cheng et al. (2018) is the most widely used
LR benchmark obtained from real-world scenarios. Because the images from
TinyFace have diverse resolutions, we validated our approach using multiple-
resolution settings (Table 3). Similar to the AgeDB-30 results, F-SKD achieved
1.55% and 3.10% increased identification accuracy compared with QualNet-LM and
A-SKD, respectively. From the results, we demonstrated that increasing the
similarity between the HR and LR network features through the teacher-student
framework is a simple but effective solution for efficiently incorporating
multi-scale features into a single network.

Figure 3: Visualization of self-attention maps from the first block of HR and
LR networks using different distillation approaches. After applying our
approach, LR network focused on fine-grained features such as eyebrows and
lips, similar to the HR network, as indicated by the red arrow. The colors red
and blue represent high and low attention, respectively. Facial images used in
the visualization are sourced from the AgeDB-30. Table 3: Results of proposed
distillation approaches on the real-world LR face identification benchmark,
TinyFace.

Type | Student | Teacher | ACC (%)
---|---|---|---
Base | iResNet50 | - | 57.30
QualNet-LM Kim et al. (2021) | iResNet50+Dec | 58.56
F-SKD (Ours) | 59.47
Base | iResNet50+CBAM | - | 54.94
A-SKD Shin et al. (2022) | iResNet50+CBAM | 55.42
F-SKD (Ours) | 57.14

Visualization of Self-attention Map. The features of the first block are
visualized in Figure 3 through a self-attention map Zagoruyko and Komodakis
(2016). To generate this map, channel-wise average pooling was applied to the
features and resulting values were normalized to a range between 0 and 1. This
visualization highlights the regions that are most relevant for face
recognition in both the HR and LR networks. An HR network focuses on facial
features, such as the eyes, nose, and lips, which are crucial for successful
recognition Kumar and Chellappa (2020). However, the LR network fails to
capture these details owing to the loss of spatial information, leading to a
degraded performance.

Unlike previous distillation methods, such as FitNet Romero et al. (2014) and
QualNet-LM Kim et al. (2021), our approach enhances the LR network’s self-
attention maps to match the HR network’s attention patterns. Remarkably, our
method extracts fine-grained details, such as wrinkle-like structures from
limited visual information. This ability to focus on highly detailed facial
features is a significant improvement over previous methods and a key factor
in achieving SOTA performance.

###  5.2 Effectiveness of Similarity Distillation

To incorporate multi-scale features into a single network, we need to define
the target object for KD and its distance measures. FitNet Romero et al.
(2014) distills all features from the teacher network into student features
via $L_{2}$ distance measures. However, it did not exhibit a significant
improvement in LR recognition performance. Instead, recent studies (QualNet-LM
and A-SKD) have employed an additional module (decoder or attention) to
extract the teacher network’s knowledge and transfer it to the student network
more concisely. Although they achieved significant improvements in LR face
recognition performance on various benchmarks, they could not transfer the
entire knowledge obtained from the teacher network’s raw features.

Types of Feature Knowledge Distillation. From this perspective, we analyzed
various approaches to feature KD and identified the optimal distance measures
for aggregating features of different resolutions. We divided the feature
vector ($\mathbf{f}$) into two components: norm
($\lvert\lvert\mathbf{f}\rvert\rvert_{p}$) and direction
($\frac{\mathbf{f}}{\lvert\lvert\mathbf{f}\rvert\rvert_{p}}$). Based on this,
we defined three types of feature KD:

  * •

FitNet Romero et al. (2014): This approach distills the entire feature using
the distance measure
$\lvert\lvert\mathbf{f}_{T}-\mathbf{f}_{S}\rvert\rvert_{p}$.

  * •

Norm-KD: This approach only distills the norm components using the distance
measure
$\lvert\lvert(\lvert\lvert\mathbf{f}_{T}\rvert\rvert_{p}-\lvert\lvert\mathbf{f}_{S}\rvert\rvert_{p})\rvert\rvert_{p}$.

  * •

F-SKD: This approach only distills the direction components via similarity
loss
$(1-\frac{\mathbf{f}_{T}}{\lvert\lvert\mathbf{f}_{T}\rvert\rvert_{p}}\cdot\frac{\mathbf{f}_{S}}{\lvert\lvert\mathbf{f}_{S}\rvert\rvert_{p}})$.

In each of the above approaches, $\mathbf{f}_{T}$ and $\mathbf{f}_{S}$
represent the features of the teacher and student networks, respectively.
$p=2$ following the FitNet.

We evaluated the three feature KD approaches on the 4$\times$ downsampled
AgeDB-30 dataset, as shown in Table 4. Our results demonstrated that F-SKD
outperformed the other feature-distillation approaches. This indicates that
distilling only the direction components through similarity loss is effective
for extracting informative features and guiding the LR network to focus on
important regions. In contrast, including norm component distillation between
the HR and LR networks led to performance degradation.

Table 4: Evaluation results on the 4$\times$ downsampled AgeDB-30, varying
with distillation components (norm and direction).

Type | Component | ACC (%)
---|---|---
Norm ($\lvert\lvert\mathbf{f}\rvert\rvert_{p}$) | Direction ($\frac{\mathbf{f}}{\lvert\lvert\mathbf{f}\rvert\rvert_{p}}$)
Base | - | - | 85.68
FitNet Romero et al. (2014) |  |  | 86.05
Norm-KD |  | - | 85.18
F-SKD (Ours) | - |  | 87.42

Statistical Analysis. In theory, if FitNet Romero et al. (2014) successfully
enables the LR network to extract the same features as the HR network, the LR
network should achieve the same accuracy as the HR network. However, based on
the results of t-test, we demonstrated that the average feature norm of the LR
network remains statistically different from that of the HR network (p-value
$<0.01$), even after applying FitNet, as shown in Figure 4. Additionally, the
average feature norm of the LR network decreased across all blocks when FitNet
was applied, indicating a potential conflict between the norm component
constraints and the task loss associated with face recognition.

Figure 4: Box plot showing the feature norm for different distillation
methods, along with the $t$-test results for the null hypothesis that the
features of HR and LR networks are equal. The notation B{$i$} represents the
$i$th ResNet block. A p-value of less than 0.01 (*** in the plot) rejects the
null hypothesis, indicating that the feature norm is not statistically equal.
t-test was performed using the 4$\times$ downsampled AgeDB-30.

We conducted a Pearson’s correlation analysis for the baseline, FitNet, and
our proposed F-SKD approaches to investigate the directional component of the
features after distillation, as shown in Figure 5. Our analysis revealed that
the baseline network did not correlate with the HR network features,
indicating that important features for HR recognition were not captured in the
LR network. This lack of correlation is a known cause of performance
degradation when resolution decreases Kim et al. (2022). In contrast, FitNet
yielded a strong positive correlation in the first block’s features, but the
correlation became weaker in the later blocks. Although there was a slight
improvement in LR recognition performance owing to the correlation in the
initial blocks, the latter blocks, which extract high-level semantic features,
did not resemble the HR network’s features. In contrast, our proposed approach
achieved a strong positive correlation ($r>0.6$) between the HR and LR network
features for all the blocks. This indicates that our F-SKD approach converged
and effectively guides the LR network to focus on informative regions for all
the blocks, similar to those of the HR network.

Figure 5: Pixel-level Pearson’s correlation between HR and LR network features
using various distillation methods, where B{$i$} represents the $i$th ResNet
block. Pearson’s correlation coefficient ($r$) measures the strength and
direction of the linear relationship between the variables. Pearson’s
correlation was measured using the evaluation set of AgeDB-30, and the plot
shows data points obtained from 1000 randomly selected images.

###  5.3 Low-resolution Digit Classification

We extended our approach to the LR digit classification task, which is useful
for digit recognition on vehicle registration plates. An evaluation of SVHN
Netzer et al. (2011) shows that our F-SKD method outperforms previous
distillation methods with significant margins, as shown in Table 5 (+1.73%
compared to RKD, +0.87% compared to QualNet-LM, and 1.90% compared to A-SKD).
Whereas previous SOTA methods for LR face recognition (QualNet-LM and A-SKD)
achieved lower accuracy than conventional KD approaches (FitNet and RKD), our
method outperformed all other approaches. This demonstrates that improving the
similarity between different resolution features is an effective solution for
distilling knowledge from HR to LR networks, not only in face recognition, but
also in other general vision tasks.

Table 5: Single-resolution evaluation results of proposed distillation
approaches on SVHN. (a) Classification accuracy of FitNet, RKD, QualNet-LM,
and F-SKD using iResNet50 as the student network. (b) Classification accuracy
of A-SKD and F-SKD using iResNet50+CBAM as the student network.

Resolution | Type | Teacher | ACC (%)
---|---|---|---
(a) |  |  |
32 $\times$ 32 | Base | iResNet50 | 93.97
iResNet50+Dec | 93.81
8 $\times$ 8 | Base | - | 84.43
FitNet Romero et al. (2014) | iResNet50 | 85.33
RKD Park et al. (2019) | 85.36
F-SKD (Ours) | 86.84
QualNet-LM Kim et al. (2021) | iResNet50+Dec | 85.04
F-SKD (Ours) | 85.78
(b) |  |  |
32 $\times$ 32 | Base | iResNet50+CBAM | 93.80
8 $\times$ 8 | Base | - | 84.25
A-SKD Shin et al. (2022) | iResNet50+CBAM | 84.94
F-SKD (Ours) | 86.55

##  6 Conclusion

Our study demonstrates that F-SKD is an effective approach for transferring
knowledge from an HR network to an LR network. F-SKD distills the directional
component of the features by reducing the cosine similarity-based distillation
loss, making it the most effective solution for aggregating features of
different resolutions. Our experiments show that F-SKD achieves SOTA
performance without additional complexity. Statistical analysis tests
validated its effectiveness in making LR network features similar to those of
the HR network across all blocks. Moreover, F-SKD’s efficiency and ease of
implementation make it a promising solution for LR recognition applications
beyond face recognition in real-world scenarios. In conclusion, our findings
demonstrate the potential of F-SKD as a simple yet effective approach to
knowledge transfer in LR recognition tasks.

Potential Societal Impacts. The use of deep learning in real-world
applications, particularly in face recognition, has significant societal
implications. To address the potential privacy concerns associated with facial
recognition, we carefully designed our experiments to exclude problematic
datasets. Specifically, we excluded MegaFace Kemelmacher-Shlizerman et al.
(2016) and MS1MV Liu et al. (2015), which have been withdrawn by their authors
owing to ethical concerns, as well as IJB-C Maze et al. (2018), which includes
YouTube data in violation of Terms of Service. By prioritizing the
minimization of negative societal impacts in our research, we aim to
contribute to the development of face recognition technology that can be
applied in a responsible and ethical manner. Our study represents a small but
important step toward the responsible development and deployment of facial
recognition technology, and we hope to inspire further ethical considerations
in the field.

##  7 Acknowledgements

This research was financially supported by the Institute of Civil Military
Technology Cooperation funded by the Defense Acquisition Program
Administration and Ministry of Trade, Industry and Energy of Korean government
under grant No. 22-CM-GU-08 as well as by a grant from the Institute of
Information and Communications Technology Planning and Evaluation (IITP)
funded by the Korean government (MSIT) (No. 2020-0-00857, Development of cloud
robot intelligence augmentation, sharing and framework technology to integrate
and enhance the intelligence of multiple robots).

## References

  * Moschoglou et al. [2017] Stylianos Moschoglou, Athanasios Papaioannou, Christos Sagonas, Jiankang Deng, Irene Kotsia, and Stefanos Zafeiriou.  AgeDB: The First Manually Collected, In-the-Wild Age Database.  pages 1997–2005, 2017.  doi:10.1109/CVPRW.2017.250.
  * Deng et al. [2019] Jiankang Deng, Jia Guo, Niannan Xue, and Stefanos Zafeiriou.  Arcface: Additive angular margin loss for deep face recognition.  In _2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 4685–4694, 2019.  doi:10.1109/CVPR.2019.00482.
  * Deng et al. [2021] Jiankang Deng, Jia Guo, Jing Yang, Alexandros Lattas, and Stefanos Zafeiriou.  Variational prototype learning for deep face recognition.  In _2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 11901–11910, 2021.  doi:10.1109/CVPR46437.2021.01173.
  * Huang et al. [2020] Y. Huang, Y. Wang, Y. Tai, X. Liu, P. Shen, S. Li, J. Li, and F. Huang.  Curricularface: Adaptive curriculum learning loss for deep face recognition.  In _2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 5900–5909, Los Alamitos, CA, USA, jun 2020. IEEE Computer Society.  doi:10.1109/CVPR42600.2020.00594.
  * Li et al. [2021] Shen Li, Jianqing Xu, Xiaqing Xu, Pengcheng Shen, Shaoxin Li, and Bryan Hooi.  Spherical confidence learning for face recognition.  In _2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 15624–15632, 2021.  doi:10.1109/CVPR46437.2021.01537.
  * Kemelmacher-Shlizerman et al. [2016] I. Kemelmacher-Shlizerman, S. M. Seitz, D. Miller, and E. Brossard.  The megaface benchmark: 1 million faces for recognition at scale.  In _2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 4873–4882, 2016.  doi:10.1109/CVPR.2016.527.
  * Yi et al. [2014] Dong Yi, Zhen Lei, Shengcai Liao, and Stan Z. Li.  Learning Face Representation from Scratch.  nov 2014.  URL http://arxiv.org/abs/1411.7923.
  * Whitelam et al. [2017] Cameron Whitelam, Emma Taborsky, Austin Blanton, Brianna Maze, Jocelyn Adams, Tim Miller, Nathan Kalka, Anil K. Jain, James A. Duncan, Kristen Allen, Jordan Cheney, and Patrick Grother.  Iarpa janus benchmark-b face dataset.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) Workshops_ , July 2017.
  * Maze et al. [2018] Brianna Maze, Jocelyn Adams, James A. Duncan, Nathan Kalka, Tim Miller, Charles Otto, Anil K. Jain, W. Tyler Niggel, Janet Anderson, Jordan Cheney, and Patrick Grother.  Iarpa janus benchmark - c: Face dataset and protocol.  In _2018 International Conference on Biometrics (ICB)_ , pages 158–165, 2018.  doi:10.1109/ICB2018.2018.00033.
  * Cheng et al. [2018] Zhiyi Cheng, Xiatian Zhu, and S. Gong.  Low-resolution face recognition.  In _ACCV_ , 2018.
  * Kim et al. [2022] Minchul Kim, Anil K. Jain, and Xiaoming Liu.  Adaface: Quality adaptive margin for face recognition.  _2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 18729–18738, 2022.
  * Chang et al. [2020] J. Chang, Z. Lan, C. Cheng, and Y. Wei.  Data uncertainty learning in face recognition.  In _2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 5709–5718, Los Alamitos, CA, USA, jun 2020. IEEE Computer Society.  doi:10.1109/CVPR42600.2020.00575.
  * Liu et al. [2021] Jiaheng Liu, Yudong Wu, Yichao Wu, Chuming Li, Xiaolin Hu, Ding Liang, and Mengyu Wang.  Dam: Discrepancy alignment metric for face recognition.  In _2021 IEEE/CVF International Conference on Computer Vision (ICCV)_ , pages 3794–3803, 2021.  doi:10.1109/ICCV48922.2021.00379.
  * Tran et al. [2017] Luan Tran, Xi Yin, and Xiaoming Liu.  Disentangled representation learning gan for pose-invariant face recognition.  In _2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 1283–1292, 2017.  doi:10.1109/CVPR.2017.141.
  * Kim et al. [2021] Insoo Kim, Seungju Han, Ji-won Baek, Seong-Jin Park, Jae-Joon Han, and Jinwoo Shin.  Quality-agnostic image recognition via invertible decoder.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 12257–12266, June 2021.
  * Massoli et al. [2020] Fabio Valerio Massoli, Giuseppe Amato, and Fabrizio Falchi.  Cross-resolution learning for Face Recognition.  _Image and Vision Computing_ , 99:103927, jul 2020.  ISSN 02628856.  doi:10.1016/j.imavis.2020.103927.
  * Shin et al. [2022] Sungho Shin, Joosoon Lee, Junseok Lee, Yeonguk Yu, and Kyoobin Lee.  Teaching where to look: Attention similarity knowledge distillation for low resolution face recognition.  In Shai Avidan, Gabriel Brostow, Moustapha Cissé, Giovanni Maria Farinella, and Tal Hassner, editors, _Computer Vision – ECCV 2022_ , pages 631–647, Cham, 2022. Springer Nature Switzerland.  ISBN 978-3-031-19775-8.
  * Netzer et al. [2011] Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Ng.  Reading digits in natural images with unsupervised feature learning.  _NIPS_ , 01 2011.
  * Liu et al. [2017] W. Liu, Y. Wen, Z. Yu, M. Li, B. Raj, and L. Song.  Sphereface: Deep hypersphere embedding for face recognition.  In _2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 6738–6746, 2017.
  * Wang et al. [2018] Hao Wang, Yitong Wang, Zheng Zhou, Xing Ji, Dihong Gong, Jingchao Zhou, Zhifeng Li, and Wei Liu.  Cosface: Large margin cosine loss for deep face recognition.  In _2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 5265–5274, 2018.  doi:10.1109/CVPR.2018.00552.
  * Student [1908] Student.  The probable error of a mean.  _Biometrika_ , 6(1):1–25, 1908.  ISSN 00063444.
  * Pearson [1896] Karl Pearson.  Mathematical contributions to the theory of evolution. iii. regression, heredity, and panmixia.  _Philosophical Transactions of the Royal Society of London. Series A, Containing Papers of a Mathematical or Physical Character_ , 187:253–318, 1896.  ISSN 02643952.
  * Ahn et al. [2019] Sungsoo Ahn, Shell Xu Hu, Andreas Damianou, Neil D. Lawrence, and Zhenwen Dai.  Variational information distillation for knowledge transfer.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , June 2019.
  * Park et al. [2019] Wonpyo Park, Dongju Kim, Yan Lu, and Minsu Cho.  Relational knowledge distillation.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , June 2019.
  * Romero et al. [2014] Adriana Romero, Nicolas Ballas, Samira Ebrahimi Kahou, Antoine Chassang, Carlo Gatta, and Yoshua Bengio.  Fitnets: Hints for thin deep nets.  _CoRR_ , abs/1412.6550, 2014.
  * Yim et al. [2017] Junho Yim, Donggyu Joo, Jihoon Bae, and Junmo Kim.  A gift from knowledge distillation: Fast optimization, network minimization and transfer learning.  In _2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 7130–7138, 2017.  doi:10.1109/CVPR.2017.754.
  * Jacobsen et al. [2018] Jörn-Henrik Jacobsen, Arnold W.M. Smeulders, and Edouard Oyallon.  i-revnet: Deep invertible networks.  In _International Conference on Learning Representations_ , 2018.  URL https://openreview.net/forum?id=HJsjkMb0Z.
  * He et al. [2016] K. He, X. Zhang, S. Ren, and J. Sun.  Deep residual learning for image recognition.  In _2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 770–778, Los Alamitos, CA, USA, jun 2016. IEEE Computer Society.  doi:10.1109/CVPR.2016.90.
  * Zhang et al. [2016] Kaipeng Zhang, Zhanpeng Zhang, Zhifeng Li, and Yu Qiao.  Joint Face Detection and Alignment using Multi-task Cascaded Convolutional Networks.  _IEEE Signal Processing Letters_ , 23(10):1499–1503, apr 2016.  doi:10.1109/LSP.2016.2603342.  URL http://arxiv.org/abs/1604.02878http://dx.doi.org/10.1109/LSP.2016.2603342.
  * Sengupta et al. [2016] Soumyadip Sengupta, Jun-Cheng Chen, Carlos Castillo, Vishal M. Patel, Rama Chellappa, and David W. Jacobs.  Frontal to profile face verification in the wild.  In _2016 IEEE Winter Conference on Applications of Computer Vision (WACV)_ , pages 1–9, 2016.  doi:10.1109/WACV.2016.7477558.
  * Zagoruyko and Komodakis [2016] Sergey Zagoruyko and Nikos Komodakis.  Paying More Attention to Attention: Improving the Performance of Convolutional Neural Networks via Attention Transfer.  _5th International Conference on Learning Representations, ICLR 2017 - Conference Track Proceedings_ , dec 2016.
  * Kumar and Chellappa [2020] Amit Kumar and Ramalingam Chellappa.  S2ld: Semi-supervised landmark detection in low resolution images and impact on face verification.  _2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW)_ , pages 3275–3283, 2020.
  * Liu et al. [2015] Z. Liu, P. Luo, X. Wang, and X. Tang.  Deep learning face attributes in the wild.  In _2015 IEEE International Conference on Computer Vision (ICCV)_ , pages 3730–3738, Los Alamitos, CA, USA, dec 2015. IEEE Computer Society.