# Facial Landmark Points Detection Using Knowledge Distillation-Based Neural
Networks

Ali Pourramezan Fard<EMAIL_ADDRESS> Mohammad H. Mahoor
<EMAIL_ADDRESS> Department of Electrical and Computer Engineering,
University of Denver, 2155 E Wesley Ave, Denver 80208, USA

###### Abstract

Facial landmark detection is a vital step for numerous facial image analysis
applications. Although some deep learning-based methods have achieved good
performances in this task, they are often not suitable for running on mobile
devices. Such methods rely on networks with many parameters, which makes the
training and inference time-consuming. Training lightweight neural networks
such as MobileNets are often challenging, and the models might have low
accuracy. Inspired by knowledge distillation (KD), this paper presents a novel
loss function to train a lightweight Student network (e.g., MobileNetV2) for
facial landmark detection. We use two Teacher networks, a Tolerant-Teacher and
a Tough-Teacher in conjunction with the Student network. The Tolerant-Teacher
is trained using Soft-landmarks created by active shape models, while the
Tough-Teacher is trained using the ground truth (aka Hard-landmarks) landmark
points. To utilize the facial landmark points predicted by the Teacher
networks, we define an Assistive Loss (ALoss) for each Teacher network.
Moreover, we define a loss function called KD-Loss that utilizes the facial
landmark points predicted by the two pre-trained Teacher networks
(EfficientNet-b3) to guide the lightweight Student network towards predicting
the Hard-landmarks. Our experimental results on three challenging facial
datasets show that the proposed architecture will result in a better-trained
Student network that can extract facial landmark points with high accuracy.

##  1 Introduction

Facial image alignment based on landmark points is a crucial step in many
facial image analysis applications including face recognition Lu and Tang
(2015); Soltanpour et al. (2017), face verification Sun et al. (2014, 2013),
face frontalization Hassner et al. (2015), pose estimation Vicente et al.
(2015), and facial expression recognition Sun et al. (2014); Zhao et al.
(2003). The goal is to detect and localize the coordinates of predefined
landmark points on human faces and use them for face alignment. In the past
two decades, great progress has been made toward improving facial landmark
detection algorithms’ accuracy. However, most of the previous research does
not focus on designing and/or training lightweight networks that can run on
mobile devices with limited computational power.

While facial landmark points detection is still considered a challenging task
for faces with large pose variations and occlusion Dong et al. (2018a); Wu et
al. (2018), recent methods have designed heavy models with a large number of
parameters, which making them unsuitable for real-time applications. Moreover,
with the growth of Internet-of-Things (IoT), robotics, and mobile devices, it
is vital to balance accuracy and model efficiency (i.e., computational time).
Recently, deep learning-based methods have caught the attention of people in
tackling this problem too. Among many lightweight neural network models,
MobileNetV2 Sandler et al. (2018) is proven to be a good trade-off between
accuracy and speed. However, because of the small number of network
parameters, the face alignment task’s accuracy using MobileNetV2 Sandler et
al. (2018) might not be enough, especially when applied to faces with extreme
poses or occlusions.

Tan and Le Tan and Le (2019) have recently proposed EfficientNet, a family of
eight different networks designed to put a trade-off between the accuracy and
model size. The designer of EfficientNet found a strong connection between the
accuracy of a network and its depth, width, and resolution. Consequently, the
proposed EfficientNet family is designed to be efficient. In other words,
EfficientNet family are designed to achieve good accuracy while they are
relatively small means having a fewer number of network parameters and fast
means having a smaller number of floating points operation (FLOPs).

Recently, knowledge distillation (KD) was utilized in image classification
Hinton et al. (2015); Romero et al. (2014), object detection Li et al. (2017),
and semantic segmentation Xie et al. (2018). Initially, the idea was to train
a lightweight network with acceptable accuracy by transferring features and
knowledge generated by an ensemble network into the single smaller network
Buciluǎ et al. (2006). Later, Hinton et al. Hinton et al. (2015) introduced
the term knowledge distillation as a technique to create a small model, called
Student network, learned to generate the results that are created by a more
cumbersome model, called Teacher network.

Inspired by the concept of KD, we propose a novel loss function called KD-Loss
to improve face alignment accuracy. Specifically, we propose a KD-based
architecture using two different Teacher networks – EfficientNet-B3 Tan and Le
(2019) – to guide the lightweight Student-Network, which is MobileNetV2
Sandler et al. (2018), to better cope with the facial landmark detection task.
Using the facial landmarks predicted by each of the Teacher networks, we
introduce two ALoss functions. Being assisted by the ALoss, the KD-Loss
considers the geometrical relation between the facial landmarks predicted by
the Student and the two Teacher networks to improve the accuracy of the
MobileNetV2 Sandler et al. (2018). In other words, we proposed to use two
independent sets of facial landmark points which are predicted by our Teacher
network to guide the lightweight Student network towards better localization
of the landmark points.

We train our method in two phases. In the first phase, we create Soft-
landmarks inspired by ASM Cootes et al. (1998). Soft-landmarks are more
similar to the Mean-landmark compared to the Hard-landmarks, which are the
original facial landmarks. Hence, as a rule of thumb, it is easier for a
lightweight model to predict the distribution of these Soft-landmarks compared
to the original ground truth. We use this attribute to create a Teacher-
Student architecture to improve the accuracy of the Student network. More
clearly, in the first phase, we train one Teacher network using the Hard-
landmarks and call it Tough-Teacher, and another Teacher network using the
Soft-landmarks as the ground truth landmark points and call it Tolerant-
Teacher. Then, in the second phase, we use our proposed KD-Loss to transfuse
the information gathered by both Teacher networks into the Student model
during the training phase. Fig. 5 depicts a general architecture of our
proposed training architecture. We tested our proposed method on the
challenging 300W Sagonas et al. (2013), WFLW Wu et al. (2018), and COFW
Burgos-Artizzu et al. (2013) datasets. The results of our experiments show
that the accuracy of facial landmark points detection using MobileNetV2
trained using our KD-Loss approach is more accurate than the original
MobileNetV2 Sandler et al. (2018). The results are also comparable with state-
of-the-art methods, while the network size is significantly smaller than most
of the previously proposed networks.

Fig. 1: The process of creating Mean-landmark, and Soft-landmarks with
different accuracy using Hard-landmarks.

The contributions of our approach are summarized as follows. First, to the
best of our knowledge, this is the first time the concept of KD is applied to
a coordinate-based regression facial landmark detection model. Second, we
proposed two different Teacher networks for guiding the Student network toward
the ground truth landmark point. Third, different from the popular loss
functions, the magnitude of our proposed ALoss can be either a positive or a
negative number. Finally, using ALoss, we propose KD-Loss, which uses the
geometrical relation between the Student and Teacher networks to improve
accuracy in facial landmark detection.

The remaining of this paper is organized as follows. Sec. 2 reviews the
related work in facial landmark points detection. Sec. 3 explains the details
of the proposed method and the training process. Then, the evaluation of the
method as well as the experimental results are provided in Sec. 4. Finally,
Sec. 5 concludes with discussion on the proposed method and future research
directions.

##  2 Related Work

The facial landmark detection task dates back to over twenty years ago, when
classical methods (aka template-based methods) were introduced. Active Shape
Model (ASM) Cootes et al. (2000) and Active Appearance Model (AAM) Cootes et
al. (1998); Martins et al. (2013) are among the first methods for facial
landmark detection. Based on these methods, Principal Components Analysis
(PCA) is applied to simplify the problem and learn parametric features of
faces to model facial landmarks variations. The model is iteratively fit into
new instances. To match a 3D deformable face model to 2D images, Martins et
al. (2013) proposed a 2.5D AAM that combines a 3D metric Point Distribution
Model (PDM) a 2D appearance model. The Constrained Local Model (CLM) proposed
by Cristinacce and Cootes Cristinacce and Cootes (2006) and its various
extensions including Asthana et al. (2013); Baltrušaitis et al. (2012);
Saragih et al. (2011); Wang et al. (2008), are among the most promising
methods for face alignment. CLM models the face shapes with Procrustes
analysis and principal component analysis. However, CLM methods are sensitive
to occlusion as well as illumination when detecting landmarks in unseen
datasets. To reduce the effect of outliers, Robust Cascade Pose Regression
(RCPR) Burgos-Artizzu et al. (2013) was introduced to detect occlusions
explicitly while using robust shape-indexed features. Another computationally
lightweight method was Local Binary Features (LBF) Ren et al. (2014), which
uses the locality principle to learn a set of highly discriminative local
binary features from each facial landmarks independently. More over, Kazemi
Kazemi and Sullivan (2014) proposed a gradient boosting framework using
ensemble of regression trees for computationally-efficient face alignment. In
their proposed framework, they first extract a sparse subset of intensity
values form the input image, and then using cascade of regression trees to
localize the facial landmark points.

Coordinate-based regression models predict the facial landmark coordinates
vector from the input image directly. Mnemonic Descent Method (MDM) Trigeorgis
et al. (2016) has utilized a recurrent convolutional network to detect facial
landmarks. Feng et al. Feng et al. (2018) introduced Wingloss, a new loss
function that is capable of overcoming the widely used L2 loss in conjunction
with a strong data augmentation method as well as a pose-based data balancing
(PDB). To ease the parts variations and regresses the coordinates of different
parts, Two-Stage Re-initialization Deep Regression MODEL (TSR) Lv et al.
(2017) splits face into several parts. Zhang and Hu (2018) proposed Exemplar-
based Cascaded Stacked Auto-Encoder Network (ECSAN) for face alignment, which
is utilized to handle partial occlusion in the image. To cope with self-
occlusions and large face rotations, Valle et al. (2019) proposed a face
alignment algorithm based on a coarse-to-fine cascade of ensembles of
regression trees, which is initialized by robustly fitting a 3D face model to
the probability maps produced by a pre-trained convolutional neural network
(CNN). Guo Guo et al. (2019) proposed a framework for practical face
alignment, which estimates the rotation information during the train phase and
use such information to better cope with the challenging faces with extreme
pose, lighting and occlusion. Feng Feng et al. (2020) proposed RWing loss, a
piece-wise loss that amplifies the impact of the samples with small-medium
errors, while rectifying the loss function for very small errors. More
recently, Fard Fard et al. (2021) proposed a ASMNet, a lightweight multi-task
network for jointly detecting facial landmark points as well as the estimation
of face pose.

Fig. 2: Examples of Soft-landmarks generated using $\tilde{m}=90\%$ of all the
Eigenvectors, as well as Hard-landmarks. Fig. 3: Distribution of Hard-
landmarks and the Soft-landmarks created on WFLW Wu et al. (2018) training
set. (a) shows the distribution of all facial landmark, while for better
visualization, (b) displays the landmark points belonging to the face
boundary.

In heatmap-based regression models, first, the likelihood heatmaps for each
facial landmark are created, and then the network is trained to generate those
heatmaps for each input image. A two-part network proposed by Yang Yang et al.
(2017), including a supervised transformation to normalize faces and a stacked
hourglass network Newell et al. (2016), is designed to predict heatmaps. In
another work JMFA Deng et al. (2019) leveraged stacked hourglass network for
multi-view face alignment, and it achieved state-of-the-art accuracy and
demonstrated more accurate than the best three entries of the last Menpo
Challenge Zafeiriou et al. (2017). LAB Wu et al. (2018) proposed by Wu first
expressed that the facial boundary line contains valuable information. Hence,
they utilized boundary lines as the geometric structure of a face to help
facial landmark detection. In another work, for a better initialization to
Ensemble of Regression Trees (ERT) regressor, Valle Valle et al. (2018)
proposed a simple CNN to generate heatmaps of landmark locations.
Additionally, Sun et al. (2019) introduced HRNet, a high-resolution network
that is applicable in many Computer Vision tasks such as facial landmark
detection and achieves a reasonable accuracy. Iranmanesh et al. (2020)
proposed an approach that provides a robust facial landmark detection
algorithm that handles shape variations in facial landmark detection while the
aggregating set of manipulated images to capture robust landmark
representation. In another work, Xiong et al. (2020) proposed the Gaussian
heatmap vectors instead of the widely used heatmap for facial landmark points
detection. More recently, to deal with the more challenging faces, Mahpod et
al. (2021) proposed a two-paired cascade subnetwork to generate heatmap and
accordingly the coordinates of the facial landmark points.

Although heatmap regression models are more accurate than coordinate
regression models, we follow the latter models since such models are
significantly smaller in terms of both memory usage and the number of FLOPs,
hence more suitable for mobile and embedded applications.

In addition, most of the previous work has proposed and/or utilized heavy
networks with a large number of parameters and arithmetic operations.
Consequently, such models are not applicable when utilized by embedded and
mobile devices. In contrast, we propose a KD-based architecture and our novel
KD-Loss to train lightweight models (e.g., MobileNetV2 having about 2.2
million parameters) that have significantly fewer parameters and arithmetic
operations, while its accuracy is comparable with previous work.

Fig. 4: We train the Tough-Teacher, and the Tolerant-Teacher networks
independently using the Hard-landmarks and the Soft-landmarks respectively
utilizing the L2 loss.

##  3 Proposed Model

In this section, we first explain the process of creating the Soft-landmarks
inspired from ASM. The Soft-landmarks are utilized for training the Tolerant-
Teacher network. Then, we illustrate our proposed Student-Teacher
architecture. After that, we explain our proposed KD-Loss function using the
the proposed assistive loss functions.

###  3.1 Soft and Hard Landmarks

Inspired by ASM, we model a face shape object $\mathbf{f}$, which is a vector
containing the coordinates of landmark points for each face, using Eq. 1:

$\mathbf{f}_{(k\times 1)}\approx\mathbf{\overline{f}}_{k\times
1}+\mathbf{V}_{k\times m}\mathbf{b}_{m\times 1}$ (1)

where $k$ is the number of the landmark points, $\mathbf{\overline{f}}$, the
Mean-landmark, is the point-wise mean of all facial landmarks in the training
set, and $\mathbf{V}=\\{v_{1},v_{2},...,v_{m}\\}$ is a set containing m
Eigenvectors of the covariance matrix of all facial landmarks. $\mathbf{b}$ is
also a $m-$dimensional vector given by Eq. 2:

$\mathbf{b}_{m\times 1}=\mathbf{V}_{m\times k}^{\intercal}[\mathbf{f}_{k\times
1}-\mathbf{\overline{f}}_{k\times 1}]$ (2)

To ensure that the generated face is similar to the original face,
$\mathbf{\tilde{b}}$ is defined by placing a restriction over $\mathbf{b}$
vector and limiting each of its elements to be between $-3\sqrt{\lambda_{i}}$
and $+3\sqrt{\lambda_{i}}$ Cootes et al. (2000), where $\lambda_{i}$ is the
statistical variance of the $i^{th}$ parameter of $b$. In Eq. 3, the new face
shape $\mathbf{f\text{-}new}$ is created after applying this constraint:

$\mathbf{f\text{-}new}_{k\times 1}=\mathbf{\overline{f}}_{k\times
1}+\mathbf{V}_{k\times m}\mathbf{\tilde{b}}_{m\times 1}$ (3)

Then we define the parameter $\tilde{m}$ as the proportion of the total number
of Eigenvectors we use to generate the Soft-landmarks. In other words,
$\tilde{m}$ define the similarity between the generated Soft-landmarks and the
Hard-landmarks. Consequently, the smaller the parameter $\tilde{m}$, the fewer
Eigenvectors we use to create $\mathbf{f\text{-}new}$ and thus, the generated
$\mathbf{f\text{-}new}$ becomes more similar to the Mean-landmark. This effect
is shown in Fig. 2 by providing some examples of Soft-landmarks.

Moreover, Fig. 3 shows the distribution of the Hard-landmarks alongside Soft-
landmarks created on WFLW Wu et al. (2018) data set. While Fig. 3-(a) shows
all the facial landmarks, in order to display the variations more clearly, in
Fig. 3-(b) we only visualize the landmark points belonging to the face
boundary. As Fig. 3 shows, there are less variations in Soft-landmarks, and
therefore it is easier for a deep neural network to learn such distributions.
In this paper, we choose the parameter $\tilde{m}$ as 90% of all the
Eigenvectors and accordingly generate the Soft-landmarks using Eqs. 1, 2, 3
(examples are shown in Fig. 1).

Fig. 5: Proposed KD-based architecture for training the Student network. KD-
Loss uses the knowledge of the previously trained Teacher networks by
utilizing the assistive loss functions ALossTou and ALossTol, to improve the
performance the face alignment task.

###  3.2 Proposed Architecture

Our proposed model consists of three main parts, two Teacher networks and a
Student network. Being one of the best in the category of lightweight
networks, we use MobileNetV2 Sandler et al. (2018) as the Student network.
Furthermore, we choose EfficientNet-B3 Tan and Le (2019) as our Teacher
network. In our proposed architecture, we have two Teacher networks: Tough-
Teacher, which is trained using Hard-landmarks, and Tolerant-Teacher trained
using Soft-landmarks. Since the variation of Soft-landmarks is smaller
compared to the Hard-landmarks (see Sec.3.1), it is easier for deep neural
networks to learn the face alignment task over the Soft-landmarks. However,
the accuracy of the Soft-landmarks is lower in comparison to the original
Hard-landmarks. We introduce the KD-Loss which uses the advantages of both
Teachers to guide the Student network to learn the facial alignment task
better. Tolerant-Teacher, trained on the Soft-landmarks has lower accuracy,
but is easier to predict, while Tough-Teacher trained on the Hard-landmarks
has higher accuracy, but is harder to predict.

Moreover, our proposed method consists of two phases. In the first phase, we
independently train both Teacher networks using a standard L2 loss. In the
second phase, we train the Student network using KD-Loss. More specifically,
KD-Loss uses the landmark points generated by Tough and Tolerant Teachers to
guide the Student towards learning the face alignment task more precisely.

###  3.3 Assistive Loss

We define ALoss to make the advantages of the geometrical knowledge of the
Teacher networks. In other words, ALoss uses the facial landmark points
predicted by each Teacher network to guide the Student network towards the
ground truth. After training both the Tough-Teacher and Tolerant-Teacher on
the training set independently, we have two different kinds of soft targets:
the Accurate-landmarks predicted by the Tough-Teacher as well as the Smooth-
landmarks predicted by the Tolerant-Teacher. We define $P_{Gt}$ as an
arbitrary facial landmark point from Hard-landmarks set, $P_{Ac}$, and
$P_{Sm}$ the corresponding landmark points from the Accurate-landmarks and
Smooth-landmarks sets respectively. Likewise, $P_{Pr}$ is the corresponding
predicted points by the Student-Network. The idea behind the ALoss is to use
the facial landmark points predicted using the Teacher networks as either a
Positive or negative assistant. Positive assistant means that the assistive
loss function penalize the network to generate a landmark point which is close
to the corresponding point predicted by the Teacher network, while the
Negative assistant means the assistive loss function penalize the Student
network to predict a landmark point which is far from the corresponding
landmark point predicted by the Teacher network.

As an example, in Fig. 6-A, in order to minimize the distance between $P_{Pr}$
and $P_{Gt}$, we use the ALossTou as a Positive assistant, which means the
ALossTou penalize the network to reduces the distance between $P_{Ac}$ and
$P_{Pr}$. In contrary, the ALossTol is a Negative assistant, which means the
it penalize the network to predict $P_{Pr}$ to be far from $P_{Sm}$. More
clearly, besides penalizing the Student network to learn the distribution of
the Hard-landmarks, we guide it to learn both the distribution of the
Accurate-landmarks as well as the Smooth-landmarks which are easier for a
lightweight model.

Moreover, to simplify and make the assistive loss function symmetric with
respect to the coordinate of $P_{Gt}$, we need both $P_{Te}$ (a facial
landmark point predicted using either of the Teacher networks) and $P_{Pr}$ to
be in one side of $P_{Gt}$. Therefore, we adapt the coordinate of $P_{Te}$
using Eq. 4:

$P_{Te}=P_{Gt}+sign(P_{Pr}-P_{Gt})~{}|P_{Te}-P_{Gt}|$ (4)

As Fig. 6-B shows, $P_{Ac}$ (or $P_{Sm}$) is not between $P_{Pr}$ and
$P_{Gt}$, we use its symmetric point $P^{\prime}_{Ac}$ calculated using Eq.4
to make the assistive loss a symmetric function with respect to $P_{Gt}$.

Fig. 6: Examples of using the assistive loss functions as a Positive and
Negative assistant. A: ALossTou performs as a Positive assistant, penalizing
the Student network to predict $P_{Pr}$ to be close to $P_{Ac}$, while
ALossTol is a Negative assistant, penalizing the Student network to predict
$P_{Pr}$ such that it is far from $P_{Sm}$. B: First, since $P_{Ac}$ is not
between $P_{Pr}$ and $P_{Gt}$, we implicitly define $P^{\prime}_{AC}$, and
then consider both ALossTou and ALossTol as a Positive assistant.

Then, we define parameter $\beta_{Te}$ with respect to the coordinate of both
$P_{Gt}$ and $P_{Te}$ using Eq. 5:

$\beta_{Te}=P_{Gt}+\sigma_{Te}~{}sign(P_{Pr}-P_{Gt})~{}|P_{Te}-P_{Gt}|$ (5)

$\beta_{Te}$ defines a threshold for the ALoss. As such, if the distance
between $P_{Pr}$ and $P_{Gt}$ is smaller than the distance between
$\beta_{Te}$ and $P_{Gt}$, then $P_{Pr}$ is considered as accurate enough, and
the ALoss gradually reduces the magnitude of the assistive loss. As shown in
Eq. 5, we define $\beta_{Te}$ to be a portion ($\sigma_{Te}$) of the distance
between $P_{Gt}$ and $P_{Te}$, where smaller values of $\sigma_{Te}$ penalize
the Student network more. However, it forces the network to put too much
effort toward improving the accuracy of landmark points which can reduce the
accuracy in general. We choose $\sigma_{Te}$ as $0.4$, so the distance between
$\beta_{Te}$ and $P_{Gt}$ is $40\%$ of the distance between $P_{Gt}$ and
$P_{Te}$. Then the ALoss consider the prediction as accurate enough.

In order to define the ALoss, we define an assistant weight function
$\omega_{Te}$, which enables the ALoss to adjust its magnitude according to
the coordinates of the $P_{Gt}$, $P_{Te}$ and $P_{Pr}$. We define
$\omega_{Te}$ using Eq. 6:

$\omega_{Te}(p)=\left\\{\begin{matrix}[l]&1&\forall~{}p\in~{}\mathcal{R}_{P}\\\
&-0.5&\forall~{}p\in~{}\mathcal{R}_{N}\\\
&\frac{-0.5}{\beta_{Te}-P_{Gt}}&\forall~{}p\in~{}\mathcal{R}_{L}\\\
\end{matrix}\right.$ (6)

We define $\mathcal{R}_{P}$ as a region in which ALoss acts as a Positive
assistant. We define this region using the relation between the location of
the $P_{Gt}$, $P_{Pr}$, and $P_{Te}$ as follow in Eq.7:

$\mathcal{R}_{P}:|P_{Pr}-P_{Gt}|-|P_{Te}-P_{Gt}|\geq 0$ (7)

which means for any predicted $P_{Pr}$, if the distance between $P_{Pr}$ and
$P_{Gt}$ is greater than the distance between the $P_{Te}$ and $P_{Gt}$, ALoss
acts as a Positive assistant and penalize the Student network to predict
$P_{Pr}$ to be close to $P_{Te}$. Similarly, we define $\mathcal{R}_{N}$ as a
region in which ALoss acts as a Negative assistant, and penalize the Student
network to predict $P_{Pr}$ to be far from $P_{Te}$. We define this region as
follow in Eq.8:

$\mathcal{R}_{N}:\left\\{\begin{matrix}[l]&|P_{Te}-P_{Gt}|-|P_{Pr}-P_{Gt}|\geq
0\\\ &~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}\&\\\
&|P_{Pr}-P_{Gt}|-|B_{Te}-P_{Gt}|\geq 0\\\ \end{matrix}\right.$ (8)

Then, we define $\mathcal{R}_{L}$ as the Low Influence region, meaning we
consider the prediction of the landmark point $P_{Pr}$ as accurate enough if
it is located in this region. Although ALoss acts as a Negative assistant in
this region, the magnitude of the ALoss decreases as $P_{Pr}$ get closer to
$P_{Gt}$. we define $\mathcal{R}_{L}$ as follows in Eq.9:

$\mathcal{R}_{L}:|B_{Te}-P_{Gt}|-|P_{Pr}-P_{Gt}|\geq 0$ (9)

In other words, we consider a predicted $P_{Pr}$ as accurate enough if it is
located between the ground truth point, $P_{Gt}$, and $B_{Te}$ (see Eq.5).

Fig. 7:  $\mathbf{\omega}_{Te}$ and ALoss for $P_{Gt}=0$, and $P_{Te}=0.4$.

Next, we define $ALoss_{Te}$ as the assistive loss with respect to $P_{Te}$
using Eq. 10:

$ALoss_{Te}=\omega_{Te}|P_{Te}-P_{Pr}|$ (10)

We define Both $\omega_{Te}$ and consequently $ALoss_{Te}$ according to three
different regions(see Fig. 7, and Eqs. 6 and 10): a) the Positive Assistant
region, $\mathcal{R}_{P}$ where the value of $\omega_{Te}$ is $1$, a positive
number, and hence, $ALoss_{Te}$ is also a positive assistive loss, meaning
that the closer $P_{Pr}$ is to $P_{Te}$, the smaller the value of the
$ALoss_{Te}$. b) the Negative Assistant region, $\mathcal{R}_{N}$, where
$P_{Pr}\in[\beta_{Te},P_{Te})$. In this region, $\omega_{Te}$ is define as
$-0.5$, which is a negative number. Consequently, $ALoss_{Te}$ is defined as a
negative assistive loss, meaning that the further the $P_{Pr}$ is from the
$P_{Te}$, the smaller the value of $ALoss_{Te}$ becomes. In other words, in
this region we design $ALoss_{Te}$ to train the Student network to use the
coordinates of $P_{Te}$ and tries to increase the distance between $P_{Pr}$
and $P_{Te}$. c) the Low Influence region, $\mathcal{R}_{L}$, where
$P_{Pr}\in[P_{Gt},\beta_{Te})$. As shown in Eq. 6, in this region
$\omega_{Te}$ is a linear function with negative slope where its minimum
value, $-0.5$, is at $\beta_{Te}$ and its maximum value, $0$, is at $P_{Gt}$.
If the predicted point $P_{Pr}$ is closer to $P_{Gt}$ than $\beta_{Te}$, we
consider the prediction as good enough, and we design $\omega_{Te}$ to
gradually reduce the loss magnitude. Furthermore, $\omega_{Te}$ is designed
such that $ALoss_{Te}$ be continuous and its value is $0$ when $P_{Pr}$ is
equal to $P_{Gt}$. In practice, the facial landmark points used for training
the models are zero-centered normalized, which means for any arbitrary facial
landmark point $P_{x,y}$, both x and y coordinates are in $[-0.5,0.5]$ range.

As Eq. 10 shows, $ALoss_{Te}$ is a piece-wise continuous function. In
addition, it is a linear function in $\mathcal{R}_{P}$, and $\mathcal{R}_{N}$
regions, and a quadratic function in $\mathcal{R}_{L}$ region. Fig. 7 shows
$\omega_{Te}$ and $ALoss_{Te}$ functions, for $P_{Gt}=0$ and $P_{Te}=0.4$.

Fig. 8:  LossMain for $P_{Gt}=0$. Fig. 9: Two examples of Proposed KD-Loss.
The larger the distance between $P_{Gt}$ and $P_{Ac}$ (or $P_{Sm}$), the
greater the absolute value of the corresponding $ALoss_{Tou}$ (or
$ALoss_{Tol}$) in $\mathcal{R}_{N}$. As a consequence, the effect of the
negative assistive loss is more in the left figure, compared to the right
figure.

###  3.4 Proposed KD-Loss

Hinton Hinton et al. (2015) used the term Knowledge Distillation, as they
created soft targets as the class probabilities generated by the Teacher
model, which replaced the soft max layer of the network with a logit layer. In
contrast, our novel KD-Loss and the proposed a Teacher-Student architecture
are different than the standard KD concept in 2 ways: first, while KD is
mostly used for classification tasks, to the best of our knowledge, this is
the first time KD is used in a coordinate regression task. Second, in the
original KD concept, the Student network tries to mimic the output results of
the Teacher network while in our proposed KD-Loss, we provide two different
assistant loss functions, $ALoss_{Tou}$ and $ALoss_{Tol}$ corresponding to the
facial landmark points predicted by the Tough and the Tolerant Teachers
respectively. Accordingly, opposite to the original KD idea, where the Student
network tries to mimic the results predicted by the Teacher network, our
Student-Network, guided by KD-Loss, uses the coordinates of facial landmark
points predicted by the Teacher networks to better predict the ground truth
points. Third, we utilize two different Teacher networks, Tough-Teacher and
call its predicted facial landmark points as Accurate-landmarks, as well as
the Tolerant-Teacher and call its corresponding predicted facial landmark
points as Smooth-landmarks. Comparing with the ground truth set, called as the
Hard-landmarks, Accurate-landmarks are more accurate than the Smooth-
landmarks, while having more complex distribution.

In order to define the KD-Loss, we first need to define the formal equation
for the both assistive loss functions. We define $P_{GT}[i,n]$ as the ground
truth coordinate of the $i^{th}$ landmark point of the $n^{th}$ image in the
training set. Likewise, we define $P_{Pr}[i,n]$, $P_{Ac}[i,n]$, and
$P_{Sm}[i,n]$ as the corresponding landmark points from the predicted,
accurate, and smooth landmark points set respectively. In addition, for each
facial landmark point $P_{GT}[i,n]$, we define $\omega_{Tou}[i,n]$ and
$\omega_{Tol}[i,n]$ as the assistant weight functions corresponding to the
Tough, and Tolerant Teachers, respectively (see Eq. 6).

Then, we define $ALoss_{Tou}[i,n]$ and $ALoss_{Tol}[i,n]$, the assistive loss
functions corresponding to the Tough, and Tolerant Teachers, respectively
using Eq. 10. Accordingly, we define the assistant loss functions for all
facial landmark points in the training set using Eq. 11:

$\begin{split}ALoss_{Tou}=\frac{1}{N~{}k}\sum_{n=1}^{N}\sum_{i=1}^{k}ALoss_{Tou}[i,n]\\\
ALoss_{Tol}=\frac{1}{N~{}k}\sum_{n=1}^{N}\sum_{i=1}^{k}ALoss_{Tol}[i,n]\end{split}$
(11)

where $k$ is the number of the facial landmark points in a face, and N is the
number of all samples in the training set. Moreover, we define $Loss_{Main}$
in Equations. 12, 13:

$\Delta[i,n]=|P_{Gt}[i,n]-P_{Pr}[i,n]|$ (12)

$Loss_{Main}=\left\\{\begin{matrix}[l]&\frac{1}{N~{}k}\sum_{n=1}^{N}\sum_{i=1}^{k}\Delta[i,n]&\text{If:
~{}}\Delta[i,n]\leq 0.5\\\
&\frac{1}{N~{}k}\sum_{n=1}^{N}\sum_{i=1}^{k}\Delta^{2}[i,n]+C&\text{otherwise}\end{matrix}\right.$
(13)

where $C=0.25$ is defined to smoothly connect the two pieces together. As
shown in Fig. 8, for any facial landmark point $P_{Gt}[i,n]$, we define
$Loss_{Main}$ as a continuous piece-wise function with respect to the
parameter $\Delta$. Accordingly, for any facial landmark points if
$\Delta[i,n]$ is greater than $0.5$, we define $Loss_{Main}$ as L2 loss. On
the contrary, for $\Delta[i,n]$ smaller than $0.5$, we define $Loss_{Main}$ as
L1 loss. To make the most advantages of both L2 and L1 loss, we define our
proposed $Loss_{Main}$ as a combination of both loss functions.

L2 loss ($y=x^{2}$) penalize the model more for the large. Since its
derivative ($y^{\prime}=2x$) is a linear function of the errors, the larger
the error, the larger the magnitude of the derivative (the influence of the
loss function). For small errors (errors that are smaller than 1), its
influence becomes very low, leading the network to focus more on large errors,
while neglecting small errors. On the contrary, the influence of L1 loss
($y=x$) is 1 ($y=1$), meaning L1 is not sensitive to the large errors.
Accordingly, the magnitude and the influence of the errors in L1 loss is
larger compared to L2 loss for small errors. Given this, we define
$Loss_{Main}$ with the parameter $\Delta=0.5$ (in Sec.3.3, we discussed that
coordinates of all facial landmarks are in [-0.5, 0.5]) to both be sensitive
to small and large errors.

Then, we define KD-Loss as a linear combination of $ALoss_{Tou}$,
$ALoss_{Tou}$ and $Loss_{Main}$ in Eq. 14, and also depict it in Fig. 9:

$\begin{split}\text{KD-Loss}=\phi\times
Loss_{Main}+ALoss_{Tou}+ALoss_{Tou}\end{split}$ (14)

As we have two assistant loss functions, we the parameter $\phi$ to be 2 to
equalize the effect of $Loss_{Main}$ with the sum of $ALoss_{Tou}$ and
$ALoss_{Tou}$. Moreover, since we have trained both Tough and Tolerant
Teachers separately before, the coordinates of the points predicted by them is
in $[-0.5,+05]$. Besides, According to Equations. 6 and 10, the minimum value
for ALoss will be at its corresponding $\beta$. By defining the parameter
$\sigma=0.4$ (see Eq. 5), we ensure that the proposed KD-Loss will never be
negative in its domain of declaration.

According to Eq. 10 (also is shown in Fig 9), the further the points predicted
by the Teacher from its corresponding ground truth point, the higher the
negative value of the corresponding assistant loss. In other words, the
distance between the points predicted by Teacher networks from their
corresponding ground truth can imply how hard is the distribution of that
point. Accordingly, when the distribution of the ground truth points is hard,
KD-Loss tries to guide the network by increasing the effect of the assistant
loss functions in order to put more attention on corresponding Smooth or
Accurate faces rather than Hard-landmarks.

While for the L2 loss, the curvature of the loss function corresponds to the
difference between the ground truth points and the corresponding predicted
points, the curvature of KD-Loss is defined according to $\beta_{Tou}$ and
$\beta_{Tol}$ (see Eq. 5). For each point $P_{Gt}[i,n]$, we define the
parameters $\beta_{Tou}[i,n]$ and $\beta_{Tol}[i,n]$ with respect to
$P_{Ac}[i,n]$ and $P_{Sm}[i,n]$ respectively. The relation between the
$\beta_{Tou}[i,n]$ and $\beta_{Tol}[i,n]$, and the corresponding points
predicted by Teacher networks, $P_{Ac}[i,n]$ and $P_{Sm}[i,n]$, adjusts the
curvature of the corresponding $KD-Loss[i,n]$. This means that, the further
the distance between the points predicted by Teacher networks from their
corresponding ground truth, the further the corresponding
$\beta^{n}\\_Tou_{i}$ and $\beta^{n}\\_Tol_{i}$, and thus the wider the Low
Influence region (see Sec. 3.3), and the smoother the curvature of $KD-
Loss[i,n]$. It is also shown in Fig. 9, that the curvature of the KD-Loss
relates to the both $\beta_{Tou}$ and $\beta_{Tol}$.

##  4 Experimental Results

In this section, we first explain the training phase and the datasets that we
used in evaluating our proposed model. We then describe the test phase as well
as the implementation details and the evaluation metrics. Lastly, we present
the results of facial landmark points detection using our proposed KD-Loss
method.

###  4.1 Datasets

We conducted the training and evaluation of our models on three popular and
challenging datasets: 300W Sagonas et al. (2013), COFW Burgos-Artizzu et al.
(2013), and WFLW Wu et al. (2018).

300W: Following the protocol described in Ren et al. (2014), we train our
networks using all 3148 68-points manually annotated faces. We also perform
the testing on the common subset (554 images), the challenging subset (135
images), and the full set, which is the sum of the challenging subset and the
common subset with 689 images consisting of 2000 images from the training
subset of HELEN Le et al. (2012) dataset, 337 images from the full set of AFW
Zhu and Ramanan (2012) dataset, and 811 images from the training subset of
LFPW Belhumeur et al. (2013) dataset with a 68-point annotation. Images in
HELEN, LFPW, and AFW datasets are collected in the wild environment; as a
result, expression and large pose variations, as well as partial occlusions,
may exist. For the test phase, the images are divided into two subsets, such
that the common subset contains 554 images from LFPW Belhumeur et al. (2013)
and HELEN Le et al. (2012), and the challenging subset contains 135 images
from IBUG Sagonas et al. (2013). The sum of the challenging subset and the
common subset is considered as the full subset, containing 689 images.

COFW: This dataset has 1345 facial images as training and 507 facial images as
the testing set. The dataset provides us with facial images having large pose
variations plus heavy occlusions. Each image in the COFW Burgos-Artizzu et al.
(2013) dataset has 29 manually annotated landmarks.

Table 1: NME (in %) and failure rate of 29-point landmarks detection on COFW
Burgos-Artizzu et al. (2013) dataset.

Method | NME | FR
---|---|---
SFPD Wu et al. (2017) | 6.40 | -
DAC-CSR Feng et al. (2017) | 6.03 | 4.73
CNN6 (Wing + PDB) Feng et al. (2018) | 5.44 | 3.75
ResNet50 (Wing + PDB) Feng et al. (2018) | 5.07 | 3.16
LAB Wu et al. (2018) | 3.92 | 0.39
ODN Zhu et al. (2019) | 5.30 | -
HRNetV2 Sun et al. (2019) | 3.45 | 0.19
ResNet50-FFLD Yan et al. (2020) | 5.32 | -
GV(HRNet) Xiong et al. (2020) | 3.37 | 0.39
mnv2 | 5.04 | 3.74
mnv2KD | 4.11 | 2.36
efn | 3.81 | 1.97

WFLW: This is another widely used facial dataset, which contains 7500 images
for training, and 2500 images for testing and recently has been proposed based
on WIDER FACE Yang et al. (2016). Each image in this dataset contains 98
manually annotated landmarks. This dataset consists of 6 subsets, including
326 large pose images, 314 expression images, 698 illumination images, 206
make-up images, 736 occlusion images, and 773 blurred images. Consequently, it
is possible to validate the robustness of the proposed model against each
different condition.

###  4.2 Evaluation Metrics

We evaluate our proposed KD-based architecture using normalized mean error
(NME), failure rate (FR), and the area-under-the-curve (AUC) Yang et al.
(2015). For the NME, we use “inter-ocular” distance (the distance between the
outer-eye-corners) as the normalizing factor followed by MDM Trigeorgis et al.
(2016) andSagonas et al. (2013). On COFW Burgos-Artizzu et al. (2013), and
WFLW Wu et al. (2018), we calculate the FR, which is the proportion of failed
detected faces for a maximum error of 0.1. We report the AUC for WFLW Wu et
al. (2018) as well.

Table 2: NME (in %) of 68-point landmarks detection on 300W Sagonas et al.
(2013).

Method | Normalized Mean Error
---|---
Common | Challenging | Fullset
DAN Kowalski et al. (2017) | 3.19 | 5.24 | 3.59
DSRN Miao et al. (2018) | 4.12 | 9.68 | 5.21
RCN Honari et al. (2016) | 4.67 | 8.44 | 5.41
CPM Dong et al. (2018b) | 3.39 | 8.14 | 4.36
PCD-CNN Kumar and Chellappa (2018) | 3.67 | 7.62 | 4.44
ODN Zhu et al. (2019) | 3.56 | 6.67 | 4.17
SAN Dong et al. (2018a) | 3.34 | 6.60 | 3.98
LAB Wu et al. (2018) | 2.98 | 5.19 | 3.49
DCFE Valle et al. (2018) | 2.76 | 5.22 | 3.24
PFLD 1X Guo et al. (2019) | 3.01 | 5.08 | 3.40
LRefNets Su et al. (2019) | 2.71 | 4.78 | 3.12
HRNetV Sun et al. (2019) | 2.87 | 5.15 | 3.32
AWing Wang et al. (2019) | 2.72 | 4.52 | 3.07
3DDE Valle et al. (2019) | 2.69 | 4.92 | 3.13
ResNet50-FFLD Yan et al. (2020) | 3.06 | 5.44 | 3.50
GEAN Iranmanesh et al. (2020) | 2.68 | 4.71 | 3.05
GV(HRNet) Xiong et al. (2020) | 2.62 | 4.51 | 2.99
GV(HRNet) Xiong et al. (2020) | 2.62 | 4.51 | 2.99
CCNN Mahpod et al. (2021) | 3.23 | 3.99 | 3.44
mnv2 | 3.93 | 6.84 | 4.50
mnv2KD | 3.56 | 6.13 | 4.06
efn | 3.34 | 5.80 | 3.82

###  4.3 Implementation Details

For each training image, we first crop and extract the faces. For 300W Sagonas
et al. (2013), no bounding boxes are provided, and for COFW Burgos-Artizzu et
al. (2013) and WFLW Wu et al. (2018), we figure out that the provided bounding
boxes are not accurate enough. Accordingly, we generate the new bounding boxes
based on the ground truth facial landmark points for each image. We then
expand the bounding boxes randomly up 10%. The next step is to resize each
facial image to $224\times 224$ pixels. To improve our models’ accuracy and
robustness, we augment each facial image multiple times in terms of
brightness, contrast, and color modification as well as adding Gaussian
noises. Moreover, we randomly rotate each image by ±45 degree and flip
horizontally with the probability of 50%. In the first stage, we train the
both Tough and Tolerant Teacher networks independently for about 250 epochs
with a batch size of 40 using L2 loss. Then, we train the Student network for
about 250 epochs with a batch size of 70, using the proposed KD-Loss. We use
Adam optimizer Kingma and Ba (2014) with a learning rate $10^{-3}$,
$\beta_{1}=0.9$, $\beta_{2}=0.999$, and $decay=10^{-6}$. We use TensorFlow
library to implement our codes and run them on a NVidia 1080Ti GPU. The code
is publicly available online on GitHub.

Table 3: Comparison of the NME (in %) of lightweight models in landmarks
localization on 300W Sagonas et al. (2013) dataset.

Method | NME
---|---
Common | Challenging | Fullset
| inter-ocular normalization
res_loss Ning et al. (2020) | - | - | 4.93
ASMNet Fard et al. (2021) | 4.82 | 8.2 | 5.50
MobileNet+ASMLoss Fard et al. (2021) | 3.88 | 7.35 | 4.59
mnv2KD | 3.56 | 6.13 | 4.06
| inter-pupil normalization
LBF Ren et al. (2014) | 4.95 | 11.98 | 6.32
LBF fast Ren et al. (2014) | 5.38 | 15.50 | 7.37
CFSS Zhu et al. (2015) | 4.73 | 9.98 | 5.76
3DDFA Zhu et al. (2016) | 6.15 | 10.59 | 7.01
DOF Wu et al. (2021) | 4.86 | 9.13 | 5.55
MuSiCa68 Shapira et al. (2021) | 4.63 | 8.16 | 5.32
G&LSR$\omega$ Shao et al. (2021) | 4.52 | 7.82 | 5.17
mnv2KD | 4.97 | 8.90 | 5.66

Table 4: NME (in %), FR (in %), and AUC of 98-point landmarks detection on
WFLW Wu et al. (2018) dataset.

Metric | Method | Test set | Pose | Expression | Illumination | Make-Up | Occlusion | Blur
---|---|---|---|---|---|---|---|---
NME |  |  DVLN Wu and Yang (2017)
---
LAB Wu et al. (2018)
ResNet50(Wing+PDB) Feng et al. (2018)
3DDE Valle et al. (2019)
GV(HRNet) Xiong et al. (2020)
| 6.08
---
5.27
5.11
4.68
4.33
| 11.54
---
10.24
8.75
8.62
7.41
| 6.78
---
5.51
5.36
5.21
4.51
| 5.73
---
5.23
4.93
4.65
4.24
| 5.98
---
5.15
5.41
4.60
4.18
| 7.33
---
6.79
6.37
5.77
5.19
| 6.88
---
6.32
5.81
5.41
4.93
| mnv2
---
mnv2KD
efn
| 9.07
---
8.57
7.86
| 16.06
---
15.06
14.15
| 9.16
---
8.81
7.97
| 8.72
---
8.15
7.60
| 9.11
---
8.75
8.43
| 10.46
---
9.92
9.28
| 9.88
---
9.40
8.69
FR |  |  DVLN Wu and Yang (2017)
---
LAB Wu et al. (2018)
ResNet50(Wing+PDB) Feng et al. (2018)
3DDE Valle et al. (2019)
GV(HRNet) Xiong et al. (2020)
| 10.84
---
7.56
6.00
5.04
3.52
| 46.93
---
28.83
22.70
22.39
16.26
| 11.15
---
6.37
4.78
5.41
2.55
| 7.31
---
6.73
4.30
3.86
3.30
| 11.65
---
7.77
7.77
6.79
3.40
| 16.30
---
13.72
12.50
9.37
6.79
| 13.71
---
10.74
7.76
6.72
5.05
| mnv2
---
mnv2KD
efn
| 27.12
---
24.08
19.68
| 86.50
---
81.59
70.55
| 27.70
---
27.38
19.42
| 23.35
---
19.91
16.04
| 25.24
---
22.33
23.30
| 36.27
---
33.42
29.21
| 33.63
---
29.10
23.80
AUC |  |  DVLN Wu and Yang (2017)
---
LAB Wu et al. (2018)
ResNet50(Wing+PDB) Feng et al. (2018)
3DDE Valle et al. (2019)
GV(HRNet) Xiong et al. (2020)
| 0.4551
---
0.5323
0.5504
0.5544
0.5775
| 0.1474
---
0.2345
0.3100
0.2640
0.3166
| 0.3889
---
0.4951
0.4959
0.5175
0.5636
| 0.4743
---
0.5433
0.5408
0.5602
0.5863
| 0.4494
---
0.5394
0.5582
0.5536
0.5881
| 0.3794
---
0.4490
0.4885
0.4692
0.5035
| 0.3973
---
0.4630
0.4918
0.4957
0.5242
| mnv2
---
mnv2KD
efn
| 0.3758
---
0.4134
0.4755
| 0.0321
---
0.0530
0.0905
| 0.3200
---
0.3570
0.4366
| 0.4015
---
0.4377
0.4944
| 0.3651
---
0.3940
0.4493
| 0.3092
---
0.3390
0.3842
| 0.3166
---
0.3521
0.4067

###  4.4 Comparison with Existing Approaches

In order to assess the performance of our proposed KD architecture, we
conducted three different experiments. We report the performance of our Tough-
Teacher, which is EfficientNet-B3 Tan and Le (2019), named efn, our Student-
Network, which is MobileNetV2 Sandler et al. (2018), named as mnv2KD, as well
as MobileNetV2 Sandler et al. (2018), named mnv2.

####  4.4.1 Evaluation on COFW

Table 1 shows the state-of-the-art results as well as our Teacher and Student
networks. As shown, Student-Network, called mnv2KD, achieves 4.11% NME with
2.36% FR, while the these metrics are 5.04% and 3.74% for MobileNetV2 Sandler
et al. (2018) respectively. The table shows that training our proposed
Student-Teacher architecture results in significantly better performance in
comparison to the base-network, MobileNetV2 Sandler et al. (2018).

####  4.4.2 Evaluation on 300W

Table 2 shows a comparison between mnv2KD, mnv2 and the state-of-the-art
methods on 300W Sagonas et al. (2013) dataset. While the calculated NME for
the Teacher network, and MobileNetV2 Sandler et al. (2018) over the
Challenging set are 5.80% and 6.84% respectively, mnv2KD achieves 6.13% by far
outperforms MobileNetV2 Sandler et al. (2018). For the Common set, the NME for
Student network is 3.56%, which is better than mnv2 NME, 3.93%. The reduction
in NME for the Challenging subset, which is about 0.71 %, is much higher than
the Common subset (about 0.37%), showing that the proposed KDLoss function
performs much better on the challenging faces. The calculated NME for the Full
subset are 4.06% as well 4.50% for Student network and mnv2 respectively,
indicating about 0.44% reduction. The results show that the proposed KDLoss
performs a vital role in better training of the Student-Model leading to
better performance.

Fig. 10: CED curve, FR (in %), and AUC being generated using Teacher network,
Student-Network, and MobileNetV2 Sandler et al. (2018) on COFW Burgos-Artizzu
et al. (2013), 300W Sagonas et al. (2013), and WFLW Wu et al. (2018).

####  4.4.3 Evaluation on WFLW

In Table 4 we compare the performance of the recently proposed methods as well
as our Student-Network, mnv2KD, Teacher network and MobileNetV2 Sandler et al.
(2018) on WFLW Wu et al. (2018) and its 6 subsets. Although the performance of
mnv2KD does not outperform the state-of-the-art methods, the performance of
mnv2KD is better than mnv2 in terms of NME (from 9.07% reduced to 8.57%), FR
(from 27.12% reduced to 24.08%), and AUC (from 0.3758 increased to 0.4134).
Similar to evaluation on 300W Sagonas et al. (2013) dataset, mnv2KD performs
much better over the pose set, faced 1.0%(from 16.06% to 15.06%), 4.91% (from
86.50% to 81.59%) reduction on NME, and FR respectively, as well as 2.09%
increase (from 0.0321 to 0.0530) on AUC. In addition, we discovered that
MobileNetV2 Sandler et al. (2018) is not able to perform very well over this
dataset, as the number of its parameters (about 2.4M) might not be enough for
predicting 98 different pair of landmark points. However, the qualitative
study in Sec. 4.8 shows that mnv2KD has an acceptable qualitative performance.

Fig. 11: Face alignment using mnv2KD and mnv2 on COFW Burgos-Artizzu et al.
(2013) dataset. For each landmark point if the error rate with respect to the
normalization factor, is more than $0.1$, it is considered as a failure and we
printed it red, and otherwise it is green.

###  4.5 Cumulative Error Distribution Curve Comparison

Cumulative Error Distribution (CED) curve is the cumulative distribution
function of the normalized mean error(NME). Although NME can be considered as
a good evaluation metric, it is very sensitive to the outliers. As such, when
the average error over the testing set is low except for some outlier samples,
NME can dramatically be aggravated.

Hence, we visualize the CED curve with the failure rate threshold as 0.1 for
Teacher network, Student network and the base-network, MobileNetV2 Sandler et
al. (2018). Fig. 10 shows the CED curves for COFW Burgos-Artizzu et al.
(2013), the Full subset of 300W Sagonas et al. (2013), and the Full subset of
WFLW Wu et al. (2018). We also show FR (in %) and AUC to better compare
performance of our proposed KD-Loss as well as the KD-based architecture. As
shown in Fig. 10, the performance of Student network is much better than the
MobileNetV2 Sandler et al. (2018) and a bit less accurate than Teacher network
on COFW Burgos-Artizzu et al. (2013) dataset . Similarly, on 300W Sagonas et
al. (2013), Student network performs better than MobileNetV2 Sandler et al.
(2018), but its performance is not as good as the Teacher network. On WFLW Wu
et al. (2018), although the performance of Student network is better than
MobileNetV2 Sandler et al. (2018), Teacher network performs much better than
both former networks.

According to Fig. 10, our proposed Student network performs much better
comparing to the base-network on all datasets. However, as the the CED curves
which are depicted in Fig. 10 show, the best accuracy improvement accrues on
COFW Burgos-Artizzu et al. (2013) (the CED Student network curve is much
closer to the Teacher network curves). We conclude that since the number of
parameters in our base-network, MobileNetV2 Sandler et al. (2018) (about 2.44
million), is much smaller than that of Teacher network (about 12 million), its
accuracy has a heavy reliance on the number of facial landmark points that
exists in a dataset. Consequently, the best improvement in the results that
Student network achieved comparing to the Teacher network is on COFW Burgos-
Artizzu et al. (2013), containing only 29 landmark points, and the least
improvement on WFLW Wu et al. (2018), with 98 landmark points.

###  4.6 Model Size and Computational Cost Study

To evaluate the model size and computational complexity, we report the number
of network parameters and the FLOPs (over the resolution of 224 $\times$ 224 )
in Table 5. According to Table 5, our Student network, mnv2KD(the architecture
is same as MobileNetV2 Sandler et al. (2018)) has only $2.4M$ parameters as
well as $0.6G$ FLOPs indicating that the models is very efficient, while the
its accuracy is comparable with the state-of-the-art methods. Only ASMNet Fard
et al. (2021) has smaller number of parameter (1.43M), and FLOPs (0.51G)
compared to the mnv2KD, while its accuracy is much less than our Student
network.

Model size is another important factor in the context of lightweight CNNs. In
Table 6 we compare the size of our proposed mnv2KD model with the recently
proposed lightweight models for face alignment. According to Table 6, the size
of mnv2KD is only 3.2MB, which is the smallest among other proposed models.
The model size has a heavy reliance on the time of loading the graph of a
model to the memory (either GPU or RAM). Accordingly, our mnv2KD can be
considered to be more efficient (in terms of the model size) than the other
methods reported in Table 6.

Fig. 12: Face alignment using mnv2KD and mnv2 on the 300W Sagonas et al.
(2013). For each landmark point if the error rate with respect to the
normalization factor, is more than 0.1, it is considered as a failure and we
printed it red, and otherwise it is green. Fig. 13: Comparison of the model
response time and the fps of 3 lightweight models.

As Fig. 13 shows, we compare the model response time as well as the fps of our
mnv2KD (the Student network), efn (the Teacher network), and ASMNet. For the
first experiment shown in Fig. 13 (the left figure), we used an Intel i7-6850K
CPU and repeated each experiments 100 times and report the average response
time. As Fig. 13 shows, on CPU, the response time of mnv2KD is 3.95ms, while
the response time is 9.44ms and 2.38ms for the Teacher network and ASMNet Fard
et al. (2021), respectively. On CPU, the speed of mnv2KD is very close to the
ASMNet Fard et al. (2021) (which is the smallest in terms of the number of the
parameters and FLOPs among all the state-of-the-art models in the context of
face alignment), but its accuracy is much better. Moreover, mnv2KD can reaches
to 253 fps on CPU indicating the efficiency of the model. For the second
experiment, we used a 1080ti GPU. As Fig. 13 shows, our mnv2KD model can reach
to 417 fps indicating how fast the model can perform for detecting the
landmark points in a facial image.

In addition, our proposed models do not have any post-processing costs, while
for the heatmap-based models, converting heatmaps to points should be
considered. By utilizing our proposed Tough and Tolerant Teachers and KD-Loss,
we improve the accuracy of MobileNetV2 Sandler et al. (2018), known as one of
the best lightweight models, and consequently, we create a balance between
efficiency and accuracy.

Fig. 14: Face alignment using mnv2KD and mnv2 on WFLW Wu et al. (2018)
dataset. For each landmark point if the error rate with respect to the
normalization factor, is more than $0.1$, it is considered as a failure and we
printed it red, and otherwise it is green.

###  4.7 Ablation Study

In order to figure out the effectiveness of using Tough-Teacher as well as
Tolerant-Teacher, we train two different model on 300W Sagonas et al. (2013)
dataset, mnv2Tou using only Accurate-landmarks, as well as mnv2Tol using only
Smooth-landmarks. Then we modify the proposed KD-Loss function such that the
former only defined using $ALoss_{Tou}$ and the latter using $ALoss_{Tol}$. As
it is shown in Table. 7, although using the modified KD-Loss results in better
performance compared to the original mnv2 being trained using L2 loss, mnv2Tou
performs worse than mnv2Tol. It indicates that since the distribution of
Accurate-landmarks is much harder for the lightweight MobileNetV2 Sandler et
al. (2018) to be learned in comparison to the distribution of Smooth-
landmarks, the modified KD-Loss generated using only Accurate-landmarks
performs not much better when compared to the original L2 loss. The better
performance of mnv2Tol highlights the effect of different regions that are
defined using the assistant loss functions. In KD-Loss, since the distribution
of the Smooth-landmarks is much easier than distribution of the Accurate-
landmarks, the Negative Assistant region defined by $ALoss_{Tol}$ is able to
guide the Student network towards the Accurate-landmarks. Accordingly, when we
define the KD-Loss using only $ALoss_{Tou}$ the performance of the
corresponding mnv2Tou is not as good as the performance of mnv2Tol.

Table 5:  Comparison of different methods in terms of number of parameters and
Flops.

Method | Backbone | #Params (M) | FLOPs (B)
---|---|---|---
DVLN Wu and Yang (2017) | VGG-16 | 132.0 | 14.4
SAN Dong et al. (2018a) | ResNet-152 | 57.4 | 10.7
LAB Wu et al. (2018) | Hourglass | 25.1 | 19.1
ResNet50 (Wing + PDB) Feng et al. (2018) | ResNet-50 | 25 | 3.8
HRNetV Sun et al. (2019) | HRNetV2-W18 | 9.3 | 4.3
ASMNet Fard et al. (2021) | reduced MobileNetV2 | 1.43 | 0.51
EfficientNet-B3 Tan and Le (2019) | - | 12 | 1.8
mnv2KD | MobileNetV2 | 2.4 | 0.6

Table 6: Comparison of the model size of the lightweight models for face
alignment.

Method |  efn |  mnv2KD |  ASMNet Fard et al. (2021) |  MuSiCa68 Shapira et al. (2021) |  G&LSR$\omega$Shao et al. (2021)
---|---|---|---|---|---
Model Size (MB) | 7.7 | 3.2 | 3.6 | 9.1 | 5.9

Besides, although mnv2Tol performs better than mnv2Tou, it cannot outperform
the performance of mnv2KD since Smooth-landmarks is not accurate enough to be
used as the only Teacher. Using both Teachers simultaneity with the
conjunction of the proposed $ALoss_{Tou}$ and $ALoss_{Tol}$ achieves the best
accuracy, as Tolerant-Teacher guides the Student network towards learning the
Accurate-landmarks and Tough-Teacher it the network towards learning the
original ground truths.

We compare our KD-Loss with the other widely-used loss functions for face
alignment in Table 8. In Table 8, we also present the effect of different loss
functions by comparing the NME of the face alignment task using MobileNetV2
Sandler et al. (2018) as the network on 300W Sagonas et al. (2013) dataset. On
all the 3 subsets, the L2 loss performs the least accurate face alignment,
while KD-Loss achieves the best accuracy. In addition, the NME generated by
the L1 Loss and the Smooth L1 loss are very similar. On the Challenging subset
of 300W Sagonas et al. (2013) dataset, the NME reduces from 6.84% for L2 loss
to 6.33% for Smooth L1 loss (0.51% reduction), and then to 6.13% for KD-Loss
(0.2% reduction compared to Smooth L1). For the Common subset of 300W Sagonas
et al. (2013) dataset, the NME of the MobileNetV2 Sandler et al. (2018)
trained using the L1 loss is 3.93%, which reduces to 3.66% using the Smooth L1
and then reduces to 6.13% for training the model using KD-Loss. Hence, KD-Loss
performers more accurately compared to other widely-used loss functions.

Table 7: Comparing the performance of mnv2KD, mnv2Tou, mnV2Tol, and mnV2 with respect to NME (in %), FR (in %), and AUC on 300W Sagonas et al. (2013). | mnv2 |  mnv2Tou |  mnv2Tol | mnv2KD
---|---|---|---|---
NME | Challenging | 6.84 | 6.59 | 6.41 | 6.13
Common | 3.93 | 3.85 | 3.73 | 3.56
Full | 4.50 | 4.39 | 4.25 | 4.06
FR | Challenging | 7.40 | 5.92 | 4.44 | 3.70
Common | 0.18 | 0.18 | 0.18 | 0.18
Full | 1.59 | 1.30 | 1.01 | 0.87
AUC | Challenging | 0.5425 | 0.5509 | 0.5736 | 0.6029
Common | 0.8102 | 0.8146 | 0.8230 | 0.8356
Full | 0.7578 | 0.7630 | 0.7742 | 0.7900
Table 8: NME (in %) of different loss functions using MobileNetV2 Sandler et
al. (2018) for 68-point landmarks detection on 300W Sagonas et al. (2013).

Loss Function | Common | Challenging | Fullset
---|---|---|---
L2 Loss | 3.93 | 6.84 | 4.50
L1 Loss | 3.67 | 6.38 | 4.20
smooth L1 Rashid et al. (2017) | 3.66 | 6.33 | 4.18
KD Loss | 3.56 | 6.13 | 4.06

###  4.8 Qualitative Results

To better shows the quality of our proposed KD-based architecture and loss
function, we provide some examples of facial landmark detection using our
Student network, and the base-network, MobileNetV2 Sandler et al. (2018). The
result shows that Student network outperforms the base-network. Fig. 11 shows
some example of face alignment using on COFW Burgos-Artizzu et al. (2013)
dataset. In addition, Fig. 14 shows examples of facial landmark detection on
WFLW Wu et al. (2018) dataset. Finally, in Fig. 12 we display examples of face
alignment on 300W Sagonas et al. (2013) dataset.

##  5 Conclusion and Future Work

This paper proposed a novel architecture inspired by the KD concept for the
facial landmark detection task. Using the Active Shape Model Cootes et al.
(2000), we defined Mean-landmark, Soft-landmarks as well as Hard-landmarks
terms, and, used them to train our proposed Tolerant as well as Tough Teacher
networks, which are EfficientNet-B3 Tan and Le (2019). In addition, we used
MobileNetV2 Sandler et al. (2018) as our Student-Network. The main novelty and
idea of our paper are to design KD-Loss as well as a Teacher-Student
architecture utilizing a Tough and a Tolerant Teacher to help a lightweight
Student network to learn the facial landmark points better. Moreover, we
proposed KD-Loss, an adaptive point-wise loss function that uses the
corresponding landmark points to direct the network toward the ground truth.
The results of evaluating our proposed Student-Teacher architecture on widely
used 300W Sagonas et al. (2013), COFW Burgos-Artizzu et al. (2013), and WFLW
Wu et al. (2018) datasets show that the accuracy of the Student network is
significantly better than the original MobileNetV2 Sandler et al. (2018),
specifically when it comes to faces with an extreme pose. Overall, the
accuracy of Student network is comparable to state-of-the-art methods in
facial landmark points detection. The proposed architecture can potentially be
used in other similar computer vision tasks such as body-joint tracking. We
will investigate using this method for such applications as a future research
direction.

##  6 Acknowledgements

We wish to show our appreciation to Dr. Julia Madsen for her great help on
editing this paper.

## References

  * Asthana et al. (2013) Asthana, A., Zafeiriou, S., Cheng, S., Pantic, M., 2013\.  Robust discriminative response map fitting with constrained local models, in: Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3444–3451.
  * Baltrušaitis et al. (2012) Baltrušaitis, T., Robinson, P., Morency, L.P., 2012.  3d constrained local model for rigid and non-rigid facial tracking, in: 2012 IEEE conference on computer vision and pattern recognition, IEEE. pp. 2610–2617.
  * Belhumeur et al. (2013) Belhumeur, P.N., Jacobs, D.W., Kriegman, D.J., Kumar, N., 2013\.  Localizing parts of faces using a consensus of exemplars.  IEEE transactions on pattern analysis and machine intelligence 35, 2930–2940.
  * Buciluǎ et al. (2006) Buciluǎ, C., Caruana, R., Niculescu-Mizil, A., 2006.  Model compression, in: Proceedings of the 12th ACM SIGKDD international conference on Knowledge discovery and data mining, ACM. pp. 535–541.
  * Burgos-Artizzu et al. (2013) Burgos-Artizzu, X.P., Perona, P., Dollár, P., 2013.  Robust face landmark estimation under occlusion, in: Proceedings of the IEEE International Conference on Computer Vision, pp. 1513–1520.
  * Cootes et al. (2000) Cootes, T., Baldock, E., Graham, J., 2000.  An introduction to active shape models.  Image processing and analysis , 223–248.
  * Cootes et al. (1998) Cootes, T.F., Edwards, G.J., Taylor, C.J., 1998.  Active appearance models, in: European conference on computer vision, Springer. pp. 484–498.
  * Cristinacce and Cootes (2006) Cristinacce, D., Cootes, T.F., 2006\.  Feature detection and tracking with constrained local models., in: Bmvc, Citeseer. p. 3.
  * Deng et al. (2019) Deng, J., Trigeorgis, G., Zhou, Y., Zafeiriou, S., 2019\.  Joint multi-view face alignment in the wild.  IEEE Transactions on Image Processing 28, 3636–3648.
  * Dong et al. (2018a) Dong, X., Yan, Y., Ouyang, W., Yang, Y., 2018a.  Style aggregated network for facial landmark detection, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 379–388.
  * Dong et al. (2018b) Dong, X., Yu, S.I., Weng, X., Wei, S.E., Yang, Y., Sheikh, Y., 2018b.  Supervision-by-registration: An unsupervised approach to improve the precision of facial landmark detectors, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 360–368.
  * Fard et al. (2021) Fard, A.P., Abdollahi, H., Mahoor, M., 2021.  Asmnet: A lightweight deep neural network for face alignment and pose estimation, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1521–1530.
  * Feng et al. (2018) Feng, Z.H., Kittler, J., Awais, M., Huber, P., Wu, X.J., 2018.  Wing loss for robust facial landmark localisation with convolutional neural networks, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2235–2245.
  * Feng et al. (2020) Feng, Z.H., Kittler, J., Awais, M., Wu, X.J., 2020\.  Rectified wing loss for efficient and robust facial landmark localisation with convolutional neural networks.  International Journal of Computer Vision 128, 2126–2145.
  * Feng et al. (2017) Feng, Z.H., Kittler, J., Christmas, W., Huber, P., Wu, X.J., 2017.  Dynamic attention-controlled cascaded shape regression exploiting training data augmentation and fuzzy-set sample weighting, in: Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2481–2490.
  * Guo et al. (2019) Guo, X., Li, S., Yu, J., Zhang, J., Ma, J., Ma, L., Liu, W., Ling, H., 2019\.  Pfld: A practical facial landmark detector.  arXiv preprint arXiv:1902.10859 .
  * Hassner et al. (2015) Hassner, T., Harel, S., Paz, E., Enbar, R., 2015.  Effective face frontalization in unconstrained images, in: Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4295–4304.
  * Hinton et al. (2015) Hinton, G., Vinyals, O., Dean, J., 2015.  Distilling the knowledge in a neural network.  arXiv preprint arXiv:1503.02531 .
  * Honari et al. (2016) Honari, S., Yosinski, J., Vincent, P., Pal, C., 2016\.  Recombinator networks: Learning coarse-to-fine feature aggregation, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 5743–5752.
  * Iranmanesh et al. (2020) Iranmanesh, S.M., Dabouei, A., Soleymani, S., Kazemi, H., Nasrabadi, N., 2020.  Robust facial landmark detection via aggregation on geometrically manipulated faces, in: The IEEE Winter Conference on Applications of Computer Vision, pp. 330–340.
  * Kazemi and Sullivan (2014) Kazemi, V., Sullivan, J., 2014\.  One millisecond face alignment with an ensemble of regression trees, in: Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1867–1874.
  * Kingma and Ba (2014) Kingma, D.P., Ba, J., 2014.  Adam: A method for stochastic optimization.  arXiv preprint arXiv:1412.6980 .
  * Kowalski et al. (2017) Kowalski, M., Naruniec, J., Trzcinski, T., 2017.  Deep alignment network: A convolutional neural network for robust face alignment, in: Proceedings of the IEEE conference on computer vision and pattern recognition workshops, pp. 88–97.
  * Kumar and Chellappa (2018) Kumar, A., Chellappa, R., 2018\.  Disentangling 3d pose in a dendritic cnn for unconstrained 2d face alignment, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 430–439.
  * Le et al. (2012) Le, V., Brandt, J., Lin, Z., Bourdev, L., Huang, T.S., 2012\.  Interactive facial feature localization, in: European conference on computer vision, Springer. pp. 679–692.
  * Li et al. (2017) Li, Q., Jin, S., Yan, J., 2017\.  Mimicking very efficient network for object detection, in: Proceedings of the ieee conference on computer vision and pattern recognition, pp. 6356–6364.
  * Lu and Tang (2015) Lu, C., Tang, X., 2015.  Surpassing human-level face verification performance on lfw with gaussianface, in: Twenty-ninth AAAI conference on artificial intelligence.
  * Lv et al. (2017) Lv, J., Shao, X., Xing, J., Cheng, C., Zhou, X., 2017\.  A deep regression architecture with two-stage re-initialization for high performance facial landmark detection, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3317–3326.
  * Mahpod et al. (2021) Mahpod, S., Das, R., Maiorana, E., Keller, Y., Campisi, P., 2021.  Facial landmarks localization using cascaded neural networks.  Computer Vision and Image Understanding , 103171\.
  * Martins et al. (2013) Martins, P., Caseiro, R., Batista, J., 2013.  Generative face alignment through 2.5 d active appearance models.  Computer Vision and Image Understanding 117, 250–268.
  * Miao et al. (2018) Miao, X., Zhen, X., Liu, X., Deng, C., Athitsos, V., Huang, H., 2018.  Direct shape regression networks for end-to-end face alignment, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 5040–5049.
  * Newell et al. (2016) Newell, A., Yang, K., Deng, J., 2016.  Stacked hourglass networks for human pose estimation, in: European conference on computer vision, Springer. pp. 483–499.
  * Ning et al. (2020) Ning, X., Duan, P., Li, W., Shi, Y., Li, S., 2020\.  A cpu real-time face alignment for mobile platform.  IEEE Access 8, 8834–8843.
  * Rashid et al. (2017) Rashid, M., Gu, X., Jae Lee, Y., 2017.  Interspecies knowledge transfer for facial keypoint detection, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 6894–6903.
  * Ren et al. (2014) Ren, S., Cao, X., Wei, Y., Sun, J., 2014.  Face alignment at 3000 fps via regressing local binary features, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1685–1692.
  * Romero et al. (2014) Romero, A., Ballas, N., Kahou, S.E., Chassang, A., Gatta, C., Bengio, Y., 2014\.  Fitnets: Hints for thin deep nets.  arXiv preprint arXiv:1412.6550 .
  * Sagonas et al. (2013) Sagonas, C., Tzimiropoulos, G., Zafeiriou, S., Pantic, M., 2013\.  300 faces in-the-wild challenge: The first facial landmark localization challenge, in: Proceedings of the IEEE International Conference on Computer Vision Workshops, pp. 397–403.
  * Sandler et al. (2018) Sandler, M., Howard, A., Zhu, M., Zhmoginov, A., Chen, L.C., 2018.  Mobilenetv2: Inverted residuals and linear bottlenecks, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4510–4520.
  * Saragih et al. (2011) Saragih, J.M., Lucey, S., Cohn, J.F., 2011.  Deformable model fitting by regularized landmark mean-shift.  International journal of computer vision 91, 200–215.
  * Shao et al. (2021) Shao, X., Xing, J., Lyu, J., Zhou, X., Shi, Y., Maybank, S.J., 2021.  Robust face alignment via deep progressive reinitialization and adaptive error-driven learning.  IEEE Transactions on Pattern Analysis and Machine Intelligence .
  * Shapira et al. (2021) Shapira, G., Levy, N., Goldin, I., Jevnisek, R.J., 2021\.  Knowing when to quit: Selective cascaded regression with patch attention for real-time face alignment.  arXiv preprint arXiv:2108.00377 .
  * Soltanpour et al. (2017) Soltanpour, S., Boufama, B., Wu, Q.J., 2017.  A survey of local feature methods for 3d face recognition.  Pattern Recognition 72, 391–406.
  * Su et al. (2019) Su, J., Wang, Z., Liao, C., Ling, H., 2019.  Efficient and accurate face alignment by global regression and cascaded local refinement, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pp. 0–0.
  * Sun et al. (2019) Sun, K., Zhao, Y., Jiang, B., Cheng, T., Xiao, B., Liu, D., Mu, Y., Wang, X., Liu, W., Wang, J., 2019\.  High-resolution representations for labeling pixels and regions.  arXiv preprint arXiv:1904.04514 .
  * Sun et al. (2013) Sun, Y., Wang, X., Tang, X., 2013.  Hybrid deep learning for face verification, in: Proceedings of the IEEE international conference on computer vision, pp. 1489–1496.
  * Sun et al. (2014) Sun, Y., Wang, X., Tang, X., 2014.  Deep learning face representation from predicting 10,000 classes, in: Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1891–1898.
  * Tan and Le (2019) Tan, M., Le, Q.V., 2019.  Efficientnet: Rethinking model scaling for convolutional neural networks.  arXiv preprint arXiv:1905.11946 .
  * Trigeorgis et al. (2016) Trigeorgis, G., Snape, P., Nicolaou, M.A., Antonakos, E., Zafeiriou, S., 2016.  Mnemonic descent method: A recurrent process applied for end-to-end face alignment, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4177–4187.
  * Valle et al. (2018) Valle, R., Buenaposada, J.M., Valdés, A., Baumela, L., 2018\.  A deeply-initialized coarse-to-fine ensemble of regression trees for face alignment, in: Proceedings of the European Conference on Computer Vision (ECCV), pp. 585–601.
  * Valle et al. (2019) Valle, R., Buenaposada, J.M., Valdés, A., Baumela, L., 2019\.  Face alignment using a 3d deeply-initialized ensemble of regression trees.  Computer Vision and Image Understanding 189, 102846.
  * Vicente et al. (2015) Vicente, F., Huang, Z., Xiong, X., De la Torre, F., Zhang, W., Levi, D., 2015\.  Driver gaze tracking and eyes off the road detection system.  IEEE Transactions on Intelligent Transportation Systems 16, 2014–2027.
  * Wang et al. (2019) Wang, X., Bo, L., Fuxin, L., 2019.  Adaptive wing loss for robust face alignment via heatmap regression, in: Proceedings of the IEEE International Conference on Computer Vision, pp. 6971–6981.
  * Wang et al. (2008) Wang, Y., Lucey, S., Cohn, J.F., 2008.  Enforcing convexity for improved alignment with constrained local models, in: 2008 IEEE Conference on Computer Vision and Pattern Recognition, IEEE. pp. 1–8.
  * Wu et al. (2021) Wu, B.F., Chen, B.R., Hsu, C.F., 2021.  Design of a facial landmark detection system using a dynamic optical flow approach.  IEEE Access 9, 68737–68745.
  * Wu et al. (2018) Wu, W., Qian, C., Yang, S., Wang, Q., Cai, Y., Zhou, Q., 2018.  Look at boundary: A boundary-aware face alignment algorithm, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2129–2138.
  * Wu and Yang (2017) Wu, W., Yang, S., 2017.  Leveraging intra and inter-dataset variations for robust face alignment, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pp. 150–159.
  * Wu et al. (2017) Wu, Y., Gou, C., Ji, Q., 2017\.  Simultaneous facial landmark detection, pose and deformation estimation under facial occlusion, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3471–3480.
  * Xie et al. (2018) Xie, J., Shuai, B., Hu, J.F., Lin, J., Zheng, W.S., 2018\.  Improving fast segmentation with teacher-student learning.  arXiv preprint arXiv:1810.08476 .
  * Xiong et al. (2020) Xiong, Y., Zhou, Z., Dou, Y., Su, Z., 2020.  Gaussian vector: An efficient solution for facial landmark detection, in: Proceedings of the Asian Conference on Computer Vision.
  * Yan et al. (2020) Yan, Y., Duffner, S., Phutane, P., Berthelier, A., Naturel, X., Blanc, C., Garcia, C., Chateau, T., 2020\.  Fine-grained facial landmark detection exploiting intermediate feature representations.  Computer Vision and Image Understanding 200, 103036.
  * Yang et al. (2015) Yang, H., Jia, X., Loy, C.C., Robinson, P., 2015.  An empirical study of recent face alignment methods.  arXiv preprint arXiv:1511.05049 .
  * Yang et al. (2017) Yang, J., Liu, Q., Zhang, K., 2017.  Stacked hourglass network for robust facial landmark localisation, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pp. 79–87.
  * Yang et al. (2016) Yang, S., Luo, P., Loy, C.C., Tang, X., 2016.  Wider face: A face detection benchmark, in: Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 5525–5533.
  * Zafeiriou et al. (2017) Zafeiriou, S., Trigeorgis, G., Chrysos, G., Deng, J., Shen, J., 2017.  The menpo facial landmark localisation challenge: A step towards the solution, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pp. 170–179.
  * Zhang and Hu (2018) Zhang, J., Hu, H., 2018.  Exemplar-based cascaded stacked auto-encoder networks for robust face alignment.  Computer Vision and Image Understanding 171, 95–103.
  * Zhao et al. (2003) Zhao, W., Chellappa, R., Phillips, P.J., Rosenfeld, A., 2003\.  Face recognition: A literature survey.  ACM computing surveys (CSUR) 35, 399–458.
  * Zhu et al. (2019) Zhu, M., Shi, D., Zheng, M., Sadiq, M., 2019.  Robust facial landmark detection via occlusion-adaptive deep networks, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3486–3496.
  * Zhu et al. (2015) Zhu, S., Li, C., Change Loy, C., Tang, X., 2015\.  Face alignment by coarse-to-fine shape searching, in: Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4998–5006.
  * Zhu et al. (2016) Zhu, X., Lei, Z., Liu, X., Shi, H., Li, S.Z., 2016\.  Face alignment across large poses: A 3d solution, in: Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 146–155.
  * Zhu and Ramanan (2012) Zhu, X., Ramanan, D., 2012.  Face detection, pose estimation, and landmark localization in the wild, in: 2012 IEEE conference on computer vision and pattern recognition, IEEE. pp. 2879–2886.