Andrés Prados-Torreblancaa.prados@upm.es1,2 José M.
<EMAIL_ADDRESS>Luis<EMAIL_ADDRESS>ETSII
Universidad Rey Juan Carlos
Móstoles, Spain Departamento de Inteligencia Artificial.
Universidad Politécnica de Madrid,
Boadilla del Monte, Spain Shape preserving facial landmarks

# Shape Preserving Facial Landmarks with Graph Attention Networks

###### Abstract

Top-performing landmark estimation algorithms are based on exploiting the
excellent ability of large convolutional neural networks (CNNs) to represent
local appearance. However, it is well known that they can only learn weak
spatial relationships. To address this problem, we propose a model based on
the combination of a CNN with a cascade of Graph Attention Network regressors.
To this end, we introduce an encoding that jointly represents the appearance
and location of facial landmarks and an attention mechanism to weigh the
information according to its reliability. This is combined with a multi-task
approach to initialize the location of graph nodes and a coarse-to-fine
landmark description scheme. Our experiments confirm that the proposed model
learns a global representation of the structure of the face, achieving top
performance in popular benchmarks on head pose and landmark estimation. The
improvement provided by our model is most significant in situations involving
large changes in the local appearance of landmarks. The code is publicly
available at https://github.com/andresprados/SPIGA

##  1 Introduction

Landmarks (or keypoints) are a widely used representation to address high-
level vision tasks such as image retrieval [Moskvyak et al.(2021)Moskvyak,
Maire, Dayoub, and Baktashmotlagh], facial expression recognition [Sun et
al.(2019)Sun, Li, Huan, Liu, and Han], face reenactment [Zhang et
al.(2020)Zhang, Zeng, Wang, Pan, Liu, Liu, Ding, and Fan], etc. The
performance of computer vision algorithms on the final task depends, to a
great extent, on the accuracy and robustness of this intermediate
representation. Thus, although many algorithms with excellent performance have
recently emerged, research is still very intense in this area.

Top facial landmark estimation methods may be broadly grouped into coordinate
and heatmap regression approaches. Coordinate regression approaches directly
estimate the landmark position by projecting the representation estimated by a
CNN encoder onto a set of 2D coordinates [Feng et al.(2018)Feng, Kittler,
Awais, Huber, and Wu, Kowalski et al.(2017)Kowalski, Naruniec, and Trzcinski,
Feng et al.(2020)Feng, Kittler, Awais, and Wu, Trigeorgis et
al.(2016)Trigeorgis, Snape, Nicolaou, Antonakos, and Zafeiriou, Lin et
al.(2021)Lin, Zhu, Wang, Liao, Qian, Lu, and Zhou]. They are the most
efficient since they only require an encoder architecture to compute the
facial representation. The heatmap regression approach is based on appending
multiple encoder-decoder modules to estimate a 2D data structure modeling the
landmark position likelihood, the heatmap [Honari et al.(2016)Honari,
Yosinski, Vincent, and Pal, Wu et al.(2018)Wu, Qian, Yang, Wang, Cai, and
Zhou, Wang et al.(2019)Wang, Bo, and Fuxin, Huang et al.(2020)Huang, Deng,
Shen, Zhang, and Ye, Kumar et al.(2020)Kumar, Marks, Mou, Wang, Jones,
Cherian, Koike-Akino, Liu, and Feng, Huang et al.(2021)Huang, Yang, Li, Kim,
and Wei]. The landmark coordinates are typically estimated at the maximum of
each heatmap. This architecture provides an increase in accuracy at the
expense of a considerable boost in computational and memory requirements. A
fundamental limitation of both approaches is their degradation when there is
ambiguity or noise contaminating the local landmark appearance. This typically
happens at the presence of occlusions, heavy make-up, blur and extreme
illuminations or poses. This is because of the known fact that CNNs cannot
learn simple spatial relationships [Santoro et al.(2017)Santoro, Raposo,
Barrett, Malinowski, Pascanu, Battaglia, and Lillicrap] and, in the case of
facial landmarks, are unable to learn a global representation of the face
structure. However, a human face is a highly structured object with a
prominent landmark configuration. Therefore, an effective way of representing
the local appearance of each landmark and its geometric relationship to the
other landmarks is needed.

This problem has been partially addressed in the literature with a local
attention module combining landmarks with facial boundaries [Wu et
al.(2018)Wu, Qian, Yang, Wang, Cai, and Zhou, Huang et al.(2020)Huang, Deng,
Shen, Zhang, and Ye, Huang et al.(2021)Huang, Yang, Li, Kim, and Wei]. This is
a solution that learns short-distance geometrical relationships. An
alternative solution combines the advantages of a CNN description with
traditional Ensemble of Regression Trees (ERT) [Valle et al.(2018)Valle,
Buenaposada, Valdés, and Baumela, Valle et al.(2019)Valle, Buenaposada,
Valdés, and Baumela]. Although this solution is able to learn long-distance
geometrical dependencies, it is not fully satisfactory because of the limited
learning capabilities of ERTs and the impossibility of end-to-end training.
Other approaches use a Graph Convolutional Network (GCN) to learn the facial
geometrical structure [Li et al.(2020)Li, Lu, Zheng, Liao, Lin, Luo, Cheng,
Xiao, Lu, Kuo, and Miao, Lin et al.(2021)Lin, Zhu, Wang, Liao, Qian, Lu, and
Zhou]. This is achieved by combining the landmark local description, extracted
from the CNN representation, with geometrical information represented by the
relative landmark locations. However, poor initialization and the lack of an
advanced attention mechanism reduce the performance of these models. More
recent approaches use transformers [Li et al.(2022)Li, Guo, Rhee, Han, and
Han, Xia et al.(2022)Xia, Qu, Huang, Zhang, Wang, and Xu] in a cascade shape
regressor, obtaining very good results due to the built-in attention
mechanisms.

In this paper, we present the SPIGA (_Shape Preserving wIth GAts_) model for
the estimation of human face landmarks. We follow the traditional regressor
cascade approach [Cao et al.(2014)Cao, Wei, Wen, and Sun] and present an
algorithm that combines a multi-stage heatmap backbone with a cascade of Graph
Attention Network (GAT) regressors [Velickovic et al.(2018)Velickovic,
Cucurull, Casanova, Romero, Lio, and Bengio]. The backbone provides a top-
performing facial appearance representation. The cascaded GAT regressor is
endowed with a positional encoding and attention mechanism that learn the
geometrical relationship among landmarks. Another element of our proposal that
improves the convergence of the GAT cascade is a coarse-to-fine feature
extraction procedure and a good initialization. To do this, we train our
backbone with a multi-task approach that also estimates the head pose, using
its projection to establish the initial landmark locations. We evaluate the
performance of our proposal in 300W, COFW-68, MERL-RAV and WFLW datasets. It
achieves top performance on both head pose and face landmarks estimation. The
improvement is most significant in situations involving large appearance
changes, such as occlusions, heavy make-up, blur and extreme illuminations. We
make the following contributions: 1) A GAT cascade with an attention mechanism
to weigh the information provided by each landmark according to its
reliability; 2) A positional encoding to jointly represent relative landmark
locations and local appearance; 3) A multi-task approach to initialize the
location of graph nodes; 4) A coarse-to-fine landmark description scheme.

##  2 Shape Regressor Model

We propose a coarse-to-fine cascade of landmark regressors [Dollar et
al.(2010)Dollar, Welinder, and Perona, Cao et al.(2014)Cao, Wei, Wen, and Sun]
that iteratively refines the landmarks coordinates while preserving the face
shape. Our approach involves three critical components: 1) the initialization,
2) the features used for regression, and 3) the regressors that estimate the
face shape deformation at each step of the cascade.

In our proposal, we use a multi-task CNN backbone to provide both, the
initialization and the local appearance representation. We set the initial
shape of the face, ${\mathchoice{\mbox{\boldmath$\displaystyle\bf
x$}}{\mbox{\boldmath$\textstyle\bf x$}}{\mbox{\boldmath$\scriptstyle\bf
x$}}{\mbox{\boldmath$\scriptscriptstyle\bf x$}}}_{0}\in\mathbb{R}^{L\times
2}$, by projecting $L$ landmarks from a generic 3D rigid face mesh oriented
using the head pose backbone prediction. At each cascade step $t$, a GAT-based
[Velickovic et al.(2018)Velickovic, Cucurull, Casanova, Romero, Lio, and
Bengio] regressor computes a displacement vector,
$\Delta{\mathchoice{\mbox{\boldmath$\displaystyle\bf
x$}}{\mbox{\boldmath$\textstyle\bf x$}}{\mbox{\boldmath$\scriptstyle\bf
x$}}{\mbox{\boldmath$\scriptscriptstyle\bf x$}}}_{t}$, to update the landmarks
location, ${\mathchoice{\mbox{\boldmath$\displaystyle\bf
x$}}{\mbox{\boldmath$\textstyle\bf x$}}{\mbox{\boldmath$\scriptstyle\bf
x$}}{\mbox{\boldmath$\scriptscriptstyle\bf
x$}}}_{t}={\mathchoice{\mbox{\boldmath$\displaystyle\bf
x$}}{\mbox{\boldmath$\textstyle\bf x$}}{\mbox{\boldmath$\scriptstyle\bf
x$}}{\mbox{\boldmath$\scriptscriptstyle\bf
x$}}}_{t-1}+\Delta{\mathchoice{\mbox{\boldmath$\displaystyle\bf
x$}}{\mbox{\boldmath$\textstyle\bf x$}}{\mbox{\boldmath$\scriptstyle\bf
x$}}{\mbox{\boldmath$\scriptscriptstyle\bf x$}}}_{t}$. After $K$ steps, the
final face shape is ${\mathchoice{\mbox{\boldmath$\displaystyle\bf
x$}}{\mbox{\boldmath$\textstyle\bf x$}}{\mbox{\boldmath$\scriptstyle\bf
x$}}{\mbox{\boldmath$\scriptscriptstyle\bf
x$}}}_{K}={\mathchoice{\mbox{\boldmath$\displaystyle\bf
x$}}{\mbox{\boldmath$\textstyle\bf x$}}{\mbox{\boldmath$\scriptstyle\bf
x$}}{\mbox{\boldmath$\scriptscriptstyle\bf
x$}}}_{0}+\sum_{t=1}^{K}\Delta{\mathchoice{\mbox{\boldmath$\displaystyle\bf
x$}}{\mbox{\boldmath$\textstyle\bf x$}}{\mbox{\boldmath$\scriptstyle\bf
x$}}{\mbox{\boldmath$\scriptscriptstyle\bf x$}}}_{t}$. We denote the 2D
location of $l$-th landmark at step $t$ as
${\mathchoice{\mbox{\boldmath$\displaystyle\bf
x$}}{\mbox{\boldmath$\textstyle\bf x$}}{\mbox{\boldmath$\scriptstyle\bf
x$}}{\mbox{\boldmath$\scriptscriptstyle\bf x$}}}_{t}^{l}\in\mathbb{R}^{2}$. In
Fig. 1 we show the regressor with a two-step cascade configuration.

Figure 1: Regressor architecture with a two-step cascade.

###  2.1 Initialization by Head Pose Estimation

Our multi-task backbone, termed _Multi Task Network_ (_MTN_), is a cascade of
$M$ encoder-decoder Hourglass (HG) modules. Each HG module in MTN is composed
of a shared encoder with two task branches: 1) a 3D head pose estimation
branch and 2) a landmark estimation decoder to the end of which we attach the
next HG module. Defining and balancing the depth of the three components is a
critical factor to boost the head pose estimation accuracy. We supervise the
$h$-th module pose head by comparing its estimation,
${\mathchoice{\mbox{\boldmath$\displaystyle\bf
p$}}{\mbox{\boldmath$\textstyle\bf p$}}{\mbox{\boldmath$\scriptstyle\bf
p$}}{\mbox{\boldmath$\scriptscriptstyle\bf p$}}}\in\mathbb{R}^{6}$, with the
ground truth, $\tilde{{\mathchoice{\mbox{\boldmath$\displaystyle\bf
p$}}{\mbox{\boldmath$\textstyle\bf p$}}{\mbox{\boldmath$\scriptstyle\bf
p$}}{\mbox{\boldmath$\scriptscriptstyle\bf p$}}}}$, using the L2 loss,
$\mathcal{L}_{{\mathchoice{\mbox{\boldmath$\displaystyle\bf
p$}}{\mbox{\boldmath$\textstyle\bf p$}}{\mbox{\boldmath$\scriptstyle\bf
p$}}{\mbox{\boldmath$\scriptscriptstyle\bf
p$}}}}^{h}({\mathchoice{\mbox{\boldmath$\displaystyle\bf
p$}}{\mbox{\boldmath$\textstyle\bf p$}}{\mbox{\boldmath$\scriptstyle\bf
p$}}{\mbox{\boldmath$\scriptscriptstyle\bf
p$}}},\tilde{{\mathchoice{\mbox{\boldmath$\displaystyle\bf
p$}}{\mbox{\boldmath$\textstyle\bf p$}}{\mbox{\boldmath$\scriptstyle\bf
p$}}{\mbox{\boldmath$\scriptscriptstyle\bf
p$}}}})=||\tilde{{\mathchoice{\mbox{\boldmath$\displaystyle\bf
p$}}{\mbox{\boldmath$\textstyle\bf p$}}{\mbox{\boldmath$\scriptstyle\bf
p$}}{\mbox{\boldmath$\scriptscriptstyle\bf
p$}}}}-{\mathchoice{\mbox{\boldmath$\displaystyle\bf
p$}}{\mbox{\boldmath$\textstyle\bf p$}}{\mbox{\boldmath$\scriptstyle\bf
p$}}{\mbox{\boldmath$\scriptscriptstyle\bf p$}}}||^{2}$. Our annotations for
pose, $\tilde{{\mathchoice{\mbox{\boldmath$\displaystyle\bf
p$}}{\mbox{\boldmath$\textstyle\bf p$}}{\mbox{\boldmath$\scriptstyle\bf
p$}}{\mbox{\boldmath$\scriptscriptstyle\bf p$}}}}$, are obtained from the
ground truth landmarks using a rigid head model (see Fig. 1). In the landmarks
task we optimize a coordinate smooth L1 loss ($\mathcal{L}_{coord}$) enhanced
by a local attention mechanism ($\mathcal{L}_{att}$) on the heatmaps, like
[Wang et al.(2019)Wang, Bo, and Fuxin, Huang et al.(2021)Huang, Yang, Li, Kim,
and Wei]. The final landmark loss is defined as
$\mathcal{L}_{lnd}=\sum_{h=1}^{M}2^{h-1}(\lambda_{c}\mathcal{L}_{coord}^{h}+\lambda_{att}\mathcal{L}_{att}^{h})$,
where $\lambda$’s are scalars empirically optimized. For further details,
please see the supplementary material.

To obtain a top-performing head pose estimation model (see Table 1) we pre-
train the network only with the landmark task, $\mathcal{L}_{lnd}$, and fine-
tune with both tasks, landmarks and pose, like [Valle et al.(2021)Valle,
Buenaposada, and Baumela]. For multi-task fine-tuning we use the loss
$\mathcal{L}_{mt}=\mathcal{L}_{lnd}+\lambda_{{\mathchoice{\mbox{\boldmath$\displaystyle\bf
p$}}{\mbox{\boldmath$\textstyle\bf p$}}{\mbox{\boldmath$\scriptstyle\bf
p$}}{\mbox{\boldmath$\scriptscriptstyle\bf
p$}}}}\sum_{h=1}^{M}2^{h-1}\mathcal{L}_{{\mathchoice{\mbox{\boldmath$\displaystyle\bf
p$}}{\mbox{\boldmath$\textstyle\bf p$}}{\mbox{\boldmath$\scriptstyle\bf
p$}}{\mbox{\boldmath$\scriptscriptstyle\bf p$}}}}^{h}$, where
$\lambda_{{\mathchoice{\mbox{\boldmath$\displaystyle\bf
p$}}{\mbox{\boldmath$\textstyle\bf p$}}{\mbox{\boldmath$\scriptstyle\bf
p$}}{\mbox{\boldmath$\scriptscriptstyle\bf p$}}}}$ is a hyperparameter.
Although we use intermediate supervision at every HG module, the prediction of
$\textstyle\bf p$ to estimate ${\mathchoice{\mbox{\boldmath$\displaystyle\bf
x$}}{\mbox{\boldmath$\textstyle\bf x$}}{\mbox{\boldmath$\scriptstyle\bf
x$}}{\mbox{\boldmath$\scriptscriptstyle\bf x$}}}_{0}$, as well as the visual
features, are extracted from the last module. Let
${\mathchoice{\mbox{\boldmath$\displaystyle\bf
X$}}{\mbox{\boldmath$\textstyle\bf X$}}{\mbox{\boldmath$\scriptstyle\bf
X$}}{\mbox{\boldmath$\scriptscriptstyle\bf X$}}}\in\mathbb{R}^{L\times 3}$ be
the 3D coordinates on the 3D head model that correspond to the $L$ 2D
landmarks. If the pose estimated by the backbone is given by $\textstyle\bf
p$, then the _initial shape_ , ${\mathchoice{\mbox{\boldmath$\displaystyle\bf
x$}}{\mbox{\boldmath$\textstyle\bf x$}}{\mbox{\boldmath$\scriptstyle\bf
x$}}{\mbox{\boldmath$\scriptscriptstyle\bf x$}}}_{0}$, is computed by
projecting the 3D model, ${\mathchoice{\mbox{\boldmath$\displaystyle\bf
x$}}{\mbox{\boldmath$\textstyle\bf x$}}{\mbox{\boldmath$\scriptstyle\bf
x$}}{\mbox{\boldmath$\scriptscriptstyle\bf
x$}}}_{0}=\pi({\mathchoice{\mbox{\boldmath$\displaystyle\bf
X$}}{\mbox{\boldmath$\textstyle\bf X$}}{\mbox{\boldmath$\scriptstyle\bf
X$}}{\mbox{\boldmath$\scriptscriptstyle\bf
X$}}};{\mathchoice{\mbox{\boldmath$\displaystyle\bf
p$}}{\mbox{\boldmath$\textstyle\bf p$}}{\mbox{\boldmath$\scriptstyle\bf
p$}}{\mbox{\boldmath$\scriptscriptstyle\bf p$}}})$, where $\pi(\cdot)$ is the
3D$\rightarrow$2D projection function.

###  2.2 Geometric and Visual Feature Extraction

For each step in the cascaded regressor, the input features are a combination
of local appearance at each landmark (i.e. visual features) and global
representation of the facial structure (i.e. geometric features). How visual
and positional information is extracted and combined has a direct impact on
the performance of the regressor (see Table 5).

Figure 2: Appearance and shape feature extraction for the $t$-th step
regressor.

Let $\textstyle\tt F$ be the output feature map of the last stacked HG module
in the MTN. We extract local appearance information from a square window,
$\mathcal{W}_{t}$, of size $w_{t}\times w_{t}$, centered at each landmark
location, ${\mathchoice{\mbox{\boldmath$\displaystyle\bf
x$}}{\mbox{\boldmath$\textstyle\bf x$}}{\mbox{\boldmath$\scriptstyle\bf
x$}}{\mbox{\boldmath$\scriptscriptstyle\bf x$}}}_{t-1}^{l}$, in $\textstyle\tt
F$. We use a fixed affine transform with a grid generator and sampler
[Jaderberg et al.(2015)Jaderberg, Simonyan, Zisserman, and Kavukcuoglu] to
crop and re-sample $\mathcal{W}_{t}$ at a fixed size, regardless of $w_{t}$.
Then, using convolutional layers, we extract the visual features,
${\mathchoice{\mbox{\boldmath$\displaystyle\bf
v$}}{\mbox{\boldmath$\textstyle\bf v$}}{\mbox{\boldmath$\scriptstyle\bf
v$}}{\mbox{\boldmath$\scriptscriptstyle\bf v$}}}_{t}^{l}$, corresponding to
the $l$-th landmark at step $t$. We iteratively reduce $w_{t}$ at each step
$t$, in a coarse-to-fine approach.

Positional information is crucial to maintain the shape of the face when local
appearance alone is not sufficient (e.g. in presence of occlusions, blur,
make-up, etc.). Relative distances between landmarks provide enhanced
geometrical features compared to their absolute locations since they
explicitly represent the facial shape. This relative positional information
can be defined from displacement vectors between landmarks [Li et al.(2020)Li,
Lu, Zheng, Liao, Lin, Luo, Cheng, Xiao, Lu, Kuo, and Miao]. Let
${\mathchoice{\mbox{\boldmath$\displaystyle\bf
q$}}{\mbox{\boldmath$\textstyle\bf q$}}{\mbox{\boldmath$\scriptstyle\bf
q$}}{\mbox{\boldmath$\scriptscriptstyle\bf
q$}}}_{t}^{l}=\\{{\mathchoice{\mbox{\boldmath$\displaystyle\bf
x$}}{\mbox{\boldmath$\textstyle\bf x$}}{\mbox{\boldmath$\scriptstyle\bf
x$}}{\mbox{\boldmath$\scriptscriptstyle\bf
x$}}}_{t-1}^{l}-{\mathchoice{\mbox{\boldmath$\displaystyle\bf
x$}}{\mbox{\boldmath$\textstyle\bf x$}}{\mbox{\boldmath$\scriptstyle\bf
x$}}{\mbox{\boldmath$\scriptscriptstyle\bf x$}}}_{t-1}^{i}\\}_{i\neq
l}\in\mathbb{R}^{2\times(L-1)}$ be the displacement vector corresponding to
$l$-th landmark in the $t$-th step. In contrast to [Li et al.(2020)Li, Lu,
Zheng, Liao, Lin, Luo, Cheng, Xiao, Lu, Kuo, and Miao], we learn a high
dimensional embedding from ${\mathchoice{\mbox{\boldmath$\displaystyle\bf
q$}}{\mbox{\boldmath$\textstyle\bf q$}}{\mbox{\boldmath$\scriptstyle\bf
q$}}{\mbox{\boldmath$\scriptscriptstyle\bf q$}}}_{t}^{l}$ using a Multi layer
Perceptron (MLP), ${\mathchoice{\mbox{\boldmath$\displaystyle\bf
r$}}{\mbox{\boldmath$\textstyle\bf r$}}{\mbox{\boldmath$\scriptstyle\bf
r$}}{\mbox{\boldmath$\scriptscriptstyle\bf
r$}}}_{t}^{l}=\Phi_{t}({\mathchoice{\mbox{\boldmath$\displaystyle\bf
q$}}{\mbox{\boldmath$\textstyle\bf q$}}{\mbox{\boldmath$\scriptstyle\bf
q$}}{\mbox{\boldmath$\scriptscriptstyle\bf q$}}}_{t}^{l})$, that facilitates
the aggregation of the visual local appearance and the facial shape
information. In the experiments, we show that this way of encoding relative
positional information in $\textstyle\bf r$ improves the shape-preserving
ability of the network (see section 3.4).

Let ${\mathchoice{\mbox{\boldmath$\displaystyle\bf
f$}}{\mbox{\boldmath$\textstyle\bf f$}}{\mbox{\boldmath$\scriptstyle\bf
f$}}{\mbox{\boldmath$\scriptscriptstyle\bf f$}}}_{t}^{l}$ be the feature
vector used to compute $\Delta{\mathchoice{\mbox{\boldmath$\displaystyle\bf
x$}}{\mbox{\boldmath$\textstyle\bf x$}}{\mbox{\boldmath$\scriptstyle\bf
x$}}{\mbox{\boldmath$\scriptscriptstyle\bf x$}}}_{t}^{l}$. At each step $t$ of
the cascade (see Fig. 2), and for each landmark $l$, we add the visual
features extracted from the backbone network,
${\mathchoice{\mbox{\boldmath$\displaystyle\bf
v$}}{\mbox{\boldmath$\textstyle\bf v$}}{\mbox{\boldmath$\scriptstyle\bf
v$}}{\mbox{\boldmath$\scriptscriptstyle\bf v$}}}_{t}^{l}$, with the relative
positional features, ${\mathchoice{\mbox{\boldmath$\displaystyle\bf
r$}}{\mbox{\boldmath$\textstyle\bf r$}}{\mbox{\boldmath$\scriptstyle\bf
r$}}{\mbox{\boldmath$\scriptscriptstyle\bf r$}}}_{t}^{l}$, computed from the
current shape, ${\mathchoice{\mbox{\boldmath$\displaystyle\bf
x$}}{\mbox{\boldmath$\textstyle\bf x$}}{\mbox{\boldmath$\scriptstyle\bf
x$}}{\mbox{\boldmath$\scriptscriptstyle\bf x$}}}_{t-1}$, to produce the
encoded features, ${\mathchoice{\mbox{\boldmath$\displaystyle\bf
f$}}{\mbox{\boldmath$\textstyle\bf f$}}{\mbox{\boldmath$\scriptstyle\bf
f$}}{\mbox{\boldmath$\scriptscriptstyle\bf
f$}}}_{t}^{l}={\mathchoice{\mbox{\boldmath$\displaystyle\bf
v$}}{\mbox{\boldmath$\textstyle\bf v$}}{\mbox{\boldmath$\scriptstyle\bf
v$}}{\mbox{\boldmath$\scriptscriptstyle\bf
v$}}}_{t}^{l}+{\mathchoice{\mbox{\boldmath$\displaystyle\bf
r$}}{\mbox{\boldmath$\textstyle\bf r$}}{\mbox{\boldmath$\scriptstyle\bf
r$}}{\mbox{\boldmath$\scriptscriptstyle\bf r$}}}_{t}^{l}$.

###  2.3 Cascade Shape Regressor Using GATs

The step regressor architecture (Fig. 2) is composed of stacked GAT layers
inspired by the ones in the Attentional Graph Neural Net [Sarlin et
al.(2020)Sarlin, DeTone, Malisiewicz, and Rabinovich]. We consider the facial
shape as a single densely connected graph where nodes are the landmark
locations, ${\mathchoice{\mbox{\boldmath$\displaystyle\bf
x$}}{\mbox{\boldmath$\textstyle\bf x$}}{\mbox{\boldmath$\scriptstyle\bf
x$}}{\mbox{\boldmath$\scriptscriptstyle\bf x$}}}_{t}$. To weigh the shared
information across nodes, we compute a dynamic adjacency matrix per GAT layer
$s$, $\mathcal{A}_{t}^{s}$. We learn these matrices as an attention from a
given landmark to every other in the graph.

The input to the first GAT layer at step $t$ are the encoded features,
$\\{{\mathchoice{\mbox{\boldmath$\displaystyle\bf
f$}}{\mbox{\boldmath$\textstyle\bf f$}}{\mbox{\boldmath$\scriptstyle\bf
f$}}{\mbox{\boldmath$\scriptscriptstyle\bf f$}}}_{t}^{i}\\}_{i=1}^{L}$. Let
${\mathchoice{\mbox{\boldmath$\displaystyle\bf
f$}}{\mbox{\boldmath$\textstyle\bf f$}}{\mbox{\boldmath$\scriptstyle\bf
f$}}{\mbox{\boldmath$\scriptscriptstyle\bf f$}}}_{t}^{i,s-1}$ be the features
of the $i$-th landmark produced by the ($s$-1)-th GAT layer, that are also the
input to $s$-th layer (${\mathchoice{\mbox{\boldmath$\displaystyle\bf
f$}}{\mbox{\boldmath$\textstyle\bf f$}}{\mbox{\boldmath$\scriptstyle\bf
f$}}{\mbox{\boldmath$\scriptscriptstyle\bf
f$}}}_{t}^{i,0}\equiv{\mathchoice{\mbox{\boldmath$\displaystyle\bf
f$}}{\mbox{\boldmath$\textstyle\bf f$}}{\mbox{\boldmath$\scriptstyle\bf
f$}}{\mbox{\boldmath$\scriptscriptstyle\bf f$}}}_{t}^{i}$). From now on, we
drop the step-index $t$ to simplify the notation. The updated feature vector
after the $s$-th layer is defined as
${\mathchoice{\mbox{\boldmath$\displaystyle\bf
f$}}{\mbox{\boldmath$\textstyle\bf f$}}{\mbox{\boldmath$\scriptstyle\bf
f$}}{\mbox{\boldmath$\scriptscriptstyle\bf
f$}}}^{i,s}={\mathchoice{\mbox{\boldmath$\displaystyle\bf
f$}}{\mbox{\boldmath$\textstyle\bf f$}}{\mbox{\boldmath$\scriptstyle\bf
f$}}{\mbox{\boldmath$\scriptscriptstyle\bf
f$}}}^{i,s-1}+MLP([{\mathchoice{\mbox{\boldmath$\displaystyle\bf
f$}}{\mbox{\boldmath$\textstyle\bf f$}}{\mbox{\boldmath$\scriptstyle\bf
f$}}{\mbox{\boldmath$\scriptscriptstyle\bf
f$}}}^{i,s-1}||{\mathchoice{\mbox{\boldmath$\displaystyle\bf
m$}}{\mbox{\boldmath$\textstyle\bf m$}}{\mbox{\boldmath$\scriptstyle\bf
m$}}{\mbox{\boldmath$\scriptscriptstyle\bf m$}}}^{i,s}])$ where
$[\cdot||\cdot]$ is the concatenation operator,
${\mathchoice{\mbox{\boldmath$\displaystyle\bf
m$}}{\mbox{\boldmath$\textstyle\bf m$}}{\mbox{\boldmath$\scriptstyle\bf
m$}}{\mbox{\boldmath$\scriptscriptstyle\bf m$}}}^{i,s}$ is the information
aggregated, or message, of the nodes neighboring $i$. Focusing on the message
generation procedure, a query vector
${\mathchoice{\mbox{\boldmath$\displaystyle\bf
h$}}{\mbox{\boldmath$\textstyle\bf h$}}{\mbox{\boldmath$\scriptstyle\bf
h$}}{\mbox{\boldmath$\scriptscriptstyle\bf h$}}}_{q}^{i,s}$, is assigned to
landmark $i$ and key ${\mathchoice{\mbox{\boldmath$\displaystyle\bf
h$}}{\mbox{\boldmath$\textstyle\bf h$}}{\mbox{\boldmath$\scriptstyle\bf
h$}}{\mbox{\boldmath$\scriptscriptstyle\bf h$}}}_{k}^{j,s}$, and value vectors
${\mathchoice{\mbox{\boldmath$\displaystyle\bf
h$}}{\mbox{\boldmath$\textstyle\bf h$}}{\mbox{\boldmath$\scriptstyle\bf
h$}}{\mbox{\boldmath$\scriptscriptstyle\bf h$}}}_{v}^{j,s}$, to every other
landmark $j$. The attention weight of landmark $i$ to landmark $j$ is the
SoftMax over the key-query similarities $\alpha_{ij}=$
SoftMax${}_{j}({\mathchoice{\mbox{\boldmath$\displaystyle\bf
h$}}{\mbox{\boldmath$\textstyle\bf h$}}{\mbox{\boldmath$\scriptstyle\bf
h$}}{\mbox{\boldmath$\scriptscriptstyle\bf
h$}}}_{q}^{i,s}\cdot{\mathchoice{\mbox{\boldmath$\displaystyle\bf
h$}}{\mbox{\boldmath$\textstyle\bf h$}}{\mbox{\boldmath$\scriptstyle\bf
h$}}{\mbox{\boldmath$\scriptscriptstyle\bf h$}}}_{k}^{j,s})$, being
$\alpha_{ij}$ the elements of the adjacency matrix $\mathcal{A}^{s}_{t}$ and
the transmitted message ${\mathchoice{\mbox{\boldmath$\displaystyle\bf
m$}}{\mbox{\boldmath$\textstyle\bf m$}}{\mbox{\boldmath$\scriptstyle\bf
m$}}{\mbox{\boldmath$\scriptscriptstyle\bf m$}}}^{i,s}$ the weighted average
of the value vectors: ${\mathchoice{\mbox{\boldmath$\displaystyle\bf
m$}}{\mbox{\boldmath$\textstyle\bf m$}}{\mbox{\boldmath$\scriptstyle\bf
m$}}{\mbox{\boldmath$\scriptscriptstyle\bf m$}}}^{i,s}=\sum_{i\neq
j}\alpha_{ij}{\mathchoice{\mbox{\boldmath$\displaystyle\bf
h$}}{\mbox{\boldmath$\textstyle\bf h$}}{\mbox{\boldmath$\scriptstyle\bf
h$}}{\mbox{\boldmath$\scriptscriptstyle\bf h$}}}_{v}^{j,s}$, where
${\mathchoice{\mbox{\boldmath$\displaystyle\bf
h$}}{\mbox{\boldmath$\textstyle\bf h$}}{\mbox{\boldmath$\scriptstyle\bf
h$}}{\mbox{\boldmath$\scriptscriptstyle\bf
h$}}}_{q}^{i,s}={\mathchoice{\mbox{\boldmath$\displaystyle\tt
W$}}{\mbox{\boldmath$\textstyle\tt W$}}{\mbox{\boldmath$\scriptstyle\tt
W$}}{\mbox{\boldmath$\scriptscriptstyle\tt
W$}}}_{1}^{s}{\mathchoice{\mbox{\boldmath$\displaystyle\bf
f$}}{\mbox{\boldmath$\textstyle\bf f$}}{\mbox{\boldmath$\scriptstyle\bf
f$}}{\mbox{\boldmath$\scriptscriptstyle\bf
f$}}}^{i,s}+{\mathchoice{\mbox{\boldmath$\displaystyle\bf
b$}}{\mbox{\boldmath$\textstyle\bf b$}}{\mbox{\boldmath$\scriptstyle\bf
b$}}{\mbox{\boldmath$\scriptscriptstyle\bf b$}}}_{1}^{s}$,
${\mathchoice{\mbox{\boldmath$\displaystyle\bf
h$}}{\mbox{\boldmath$\textstyle\bf h$}}{\mbox{\boldmath$\scriptstyle\bf
h$}}{\mbox{\boldmath$\scriptscriptstyle\bf
h$}}}_{k}^{j,s}={\mathchoice{\mbox{\boldmath$\displaystyle\tt
W$}}{\mbox{\boldmath$\textstyle\tt W$}}{\mbox{\boldmath$\scriptstyle\tt
W$}}{\mbox{\boldmath$\scriptscriptstyle\tt
W$}}}_{2}^{s}{\mathchoice{\mbox{\boldmath$\displaystyle\bf
f$}}{\mbox{\boldmath$\textstyle\bf f$}}{\mbox{\boldmath$\scriptstyle\bf
f$}}{\mbox{\boldmath$\scriptscriptstyle\bf
f$}}}^{j,s}+{\mathchoice{\mbox{\boldmath$\displaystyle\bf
b$}}{\mbox{\boldmath$\textstyle\bf b$}}{\mbox{\boldmath$\scriptstyle\bf
b$}}{\mbox{\boldmath$\scriptscriptstyle\bf b$}}}_{2}^{s}$ and
${\mathchoice{\mbox{\boldmath$\displaystyle\bf
h$}}{\mbox{\boldmath$\textstyle\bf h$}}{\mbox{\boldmath$\scriptstyle\bf
h$}}{\mbox{\boldmath$\scriptscriptstyle\bf
h$}}}_{v}^{j,s}={\mathchoice{\mbox{\boldmath$\displaystyle\tt
W$}}{\mbox{\boldmath$\textstyle\tt W$}}{\mbox{\boldmath$\scriptstyle\tt
W$}}{\mbox{\boldmath$\scriptscriptstyle\tt
W$}}}_{3}^{s}{\mathchoice{\mbox{\boldmath$\displaystyle\bf
f$}}{\mbox{\boldmath$\textstyle\bf f$}}{\mbox{\boldmath$\scriptstyle\bf
f$}}{\mbox{\boldmath$\scriptscriptstyle\bf
f$}}}^{j,s}+{\mathchoice{\mbox{\boldmath$\displaystyle\bf
b$}}{\mbox{\boldmath$\textstyle\bf b$}}{\mbox{\boldmath$\scriptstyle\bf
b$}}{\mbox{\boldmath$\scriptscriptstyle\bf b$}}}_{3}^{s}$. Matrices
${\mathchoice{\mbox{\boldmath$\displaystyle\tt
W$}}{\mbox{\boldmath$\textstyle\tt W$}}{\mbox{\boldmath$\scriptstyle\tt
W$}}{\mbox{\boldmath$\scriptscriptstyle\tt W$}}}_{i}$ and bias vectors
${\mathchoice{\mbox{\boldmath$\displaystyle\bf
b$}}{\mbox{\boldmath$\textstyle\bf b$}}{\mbox{\boldmath$\scriptstyle\bf
b$}}{\mbox{\boldmath$\scriptscriptstyle\bf b$}}}_{i}$ are learned.

Finally, the last GAT layer output
${\mathchoice{\mbox{\boldmath$\displaystyle\bf
f$}}{\mbox{\boldmath$\textstyle\bf f$}}{\mbox{\boldmath$\scriptstyle\bf
f$}}{\mbox{\boldmath$\scriptscriptstyle\bf f$}}}_{t}^{i,4}$ is processed by a
decoder, an MLP, to obtain the corresponding displacement,
$\Delta{\mathchoice{\mbox{\boldmath$\displaystyle\bf
x$}}{\mbox{\boldmath$\textstyle\bf x$}}{\mbox{\boldmath$\scriptstyle\bf
x$}}{\mbox{\boldmath$\scriptscriptstyle\bf x$}}}_{t}^{i}$. We constraint the
values in $\Delta{\mathchoice{\mbox{\boldmath$\displaystyle\bf
x$}}{\mbox{\boldmath$\textstyle\bf x$}}{\mbox{\boldmath$\scriptstyle\bf
x$}}{\mbox{\boldmath$\scriptscriptstyle\bf x$}}}_{t}^{i}$, applying an ArcTan
activation and scaling the result, to be in the interval $[-w_{t}/2,w_{t}/2]$.
In practice, this constraint makes the single-step regressor search problem
simpler, boosting training convergence. Given a trained MTN backbone, we train
the cascade with the
$\mathcal{L}_{CR}=\sum_{t=1}^{K}L1_{smooth}[\tilde{{\mathchoice{\mbox{\boldmath$\displaystyle\bf
x$}}{\mbox{\boldmath$\textstyle\bf x$}}{\mbox{\boldmath$\scriptstyle\bf
x$}}{\mbox{\boldmath$\scriptscriptstyle\bf
x$}}}}-({\mathchoice{\mbox{\boldmath$\displaystyle\bf
x$}}{\mbox{\boldmath$\textstyle\bf x$}}{\mbox{\boldmath$\scriptstyle\bf
x$}}{\mbox{\boldmath$\scriptscriptstyle\bf
x$}}}_{t-1}+\Delta{\mathchoice{\mbox{\boldmath$\displaystyle\bf
x$}}{\mbox{\boldmath$\textstyle\bf x$}}{\mbox{\boldmath$\scriptstyle\bf
x$}}{\mbox{\boldmath$\scriptscriptstyle\bf x$}}}_{t})]$ loss, where
$\tilde{{\mathchoice{\mbox{\boldmath$\displaystyle\bf
x$}}{\mbox{\boldmath$\textstyle\bf x$}}{\mbox{\boldmath$\scriptstyle\bf
x$}}{\mbox{\boldmath$\scriptscriptstyle\bf x$}}}}$ are the ground truth
landmark coordinates.

##  3 Experiments

To train and evaluate our method, we conduct different experiments in four
complementary datasets which have been acquired in-the-wild and bear different
levels of difficulty:

300W [Sagonas et al.(2016)Sagonas, Tzimiropoulos, Zafeiriou, and Pantic]
provides 68 manually annotated landmarks. We employ the 300W private
extension, which uses 3837 images as training set and adds 600 test images
divided into indoor and outdoor subgroups.

COFW-68 is a re-annotated version of COFW [Burgos-Artizzu et al.(2013)Burgos-
Artizzu, Perona, and Dollar] with 68 landmarks. It is conceived for testing
landmark detectors with occlusions in a cross-dataset approach. The testing
set in COFW-68 is made of 507 images. The annotations include the landmark
positions and the visibility labels for the same 68 points as in 300W.

WFLW [Wu et al.(2018)Wu, Qian, Yang, Wang, Cai, and Zhou] is composed of
challenging in-the-wild images and provides 98 manually annotated landmarks.
The dataset has 7500 training and 2500 testing faces. It is divided into 6
subgroups: pose, expression, illumination, make-up, occlusion and blur.

MERL-RAV [Kumar et al.(2020)Kumar, Marks, Mou, Wang, Jones, Cherian, Koike-
Akino, Liu, and Feng] is a re-annotated version of 19,000 AFLW images with 68
landmarks, like 300W. It provides 15,449 training and 3,865 test faces divided
into 3 orientation subsets: frontal, half-profile and profile. This recent
dataset includes externally occluded visibility and self-occluded labels.

###  3.1 Evaluation Metrics

In order to quantify the head pose estimation error, we use the Mean Absolute
Error (MAE) metric,
$MAE=\frac{1}{N}\sum\limits_{i=1}^{N}|\tilde{p}_{i}-p_{i}|$, where $N$ is the
number of testing images, $\tilde{p}_{i}$ is the ground truth and $p_{i}$
represents a single predicted pose parameter.

Focusing on the landmark estimation task, Normalized Mean Error (NME) is the
standard metric,
$NME=\frac{100}{N}\sum\limits_{i=1}^{N}\sum\limits_{l=1}^{L}\frac{||\tilde{{\mathchoice{\mbox{\boldmath$\displaystyle\bf
x$}}{\mbox{\boldmath$\textstyle\bf x$}}{\mbox{\boldmath$\scriptstyle\bf
x$}}{\mbox{\boldmath$\scriptscriptstyle\bf
x$}}}}_{l}^{i}-{\mathchoice{\mbox{\boldmath$\displaystyle\bf
x$}}{\mbox{\boldmath$\textstyle\bf x$}}{\mbox{\boldmath$\scriptstyle\bf
x$}}{\mbox{\boldmath$\scriptscriptstyle\bf x$}}}_{l}^{i}||_{2}}{d_{i}}$. Where
$\tilde{{\mathchoice{\mbox{\boldmath$\displaystyle\bf
x$}}{\mbox{\boldmath$\textstyle\bf x$}}{\mbox{\boldmath$\scriptstyle\bf
x$}}{\mbox{\boldmath$\scriptscriptstyle\bf x$}}}}_{l}^{i}$ and
${\mathchoice{\mbox{\boldmath$\displaystyle\bf
x$}}{\mbox{\boldmath$\textstyle\bf x$}}{\mbox{\boldmath$\scriptstyle\bf
x$}}{\mbox{\boldmath$\scriptscriptstyle\bf x$}}}_{l}^{i}$ denote,
respectively, the ground-truth and predicted coordinates of the $i$-th
landmark and $d_{i}$ is a normalization value which varies depending on the
dataset: inter-ocular (int-ocul), distance between outer eye corners; inter-
pupils, distance between pupil/eye centers; and box, computed as the geometric
mean of the landmarks ground truth bounding box
($d=\sqrt{w_{bbox}*h_{bbox}}$).

We also use Failure Rate (FR) and Area Under the Curve (AUC). FR evaluates the
robustness of algorithms in terms of NME, indicating the percentage of images
with an NME above a given threshold. AUC is calculated by computing the area
under the Cumulative Error Distribution (CED) curve from 0 to the FR
threshold. We introduce the Normalized mean Percentile Error 90 ($NPE_{90}$)
which represents the NME for the image at the 90% of the dataset, sorted by
NME. This metric is particularly convenient for small data subsets where the
FR is not representative.

In all our tables results ranked  first,  second and  third are shown
respectively in blue, green and red colors.

###  3.2 3D Pose Estimation Results

First, we evaluate the MTN performance in 3D pose estimation. In Table 1, we
compare our pose estimation in 300W and WFLW with previous works in the
literature. Our model shows a significant improvement. We reduce the mean MAE
of the previous top performer, MNN [Valle et al.(2021)Valle, Buenaposada, and
Baumela], by 17% and 27% respectively in 300W and WFLW. The main reason behind
this improvement is a better network architecture, stacked HGs vs. a single
encoder-decoder in [Valle et al.(2021)Valle, Buenaposada, and Baumela] and the
use of an attention mechanism. Having such a precise head pose estimation is a
critical factor in our proposal, since the cascade shape regressor
initialization relies on this prediction.

| 300W | WFLW | MERL-RAV
---|---|---|---
|  Angular error $(^{\circ})(\downarrow)$ |  Angular error $(^{\circ})(\downarrow)$ |  Angular error $(^{\circ})(\downarrow)$
Method | yaw | pitch | roll | mean | yaw | pitch | roll | mean | yaw | pitch | roll | mean
Yang [Yang et al.(2015)Yang, Mou, Zhang, Patras, Gunes, and Robinson] | 4.2 | 5.1 | 2.4 | 3.9 | - | - | - | - | - | - | - | -
JFA [Xu and Kakadiaris(2017)] | 2.5 | 3.0 | 2.6 | 2.7 | - | - | - | - | - | - | - | -
ASMNet [Fard et al.(2021)Fard, Abdollahi, and Mahoor] | 1.62 | 1.80 | 1.24 | 1.55 | 2.97 | 2.93 | 2.21 | 2.70 | - | - | - | -
MNN [Valle et al.(2021)Valle, Buenaposada, and Baumela] | - | - | - | 1.56 | - | - | - | 2.08 | - | - | - | -
SPIGA (Ours) | 1.41 | 1.70 | 0.77 | 1.29 | 1.78 | 1.86 | 0.93 | 1.52 | 3.23 | 2.24 | 1.71 | 2.39
Table 1: Head pose MAE, in degrees, for 300W public, WFLW and MERL-RAV
datasets.

###  3.3 Landmark Detection Results

WFLW is the most popular benchmark to evaluate the performance of facial
landmark detection. Recent methods that adopt this dataset use the bounding
boxes provided by HRnet [Wang et al.(2021)Wang, Sun, Cheng, Jiang, Deng, Zhao,
Liu, Mu, Tan, Wang, Liu, and Xiao], that were obtained from the ground truth
landmark annotations. By doing so, they achieve better performance (see Table
2, AWing results improve from 4.36 to 4.21 NME). In Table 2, we clearly
distinguish the bounding boxes used in the evaluation. Another important
aspect to perform a fair comparison is the use of additional training data. In
our discussion we do not consider methods that train with images or
annotations other than those provided by WFLW.

In Table 2, we show that our model outperforms current state-of-the-art (SOTA)
in most of the WFLW subsets, as well as in the full set metrics. When it is
compared with other GraphNets-based methods, our approach is 4% and 32% better
in terms of NME and FR than SLD [Li et al.(2020)Li, Lu, Zheng, Liao, Lin, Luo,
Cheng, Xiao, Lu, Kuo, and Miao], and 7% and 23% better than SDFL [Lin et
al.(2021)Lin, Zhu, Wang, Liao, Qian, Lu, and Zhou]. These results show that
our relative positional encoding and the per layer graph attention mechanism
have a strong impact on the performance of GraphNets. Further, our proposal is
also more accurate than recent approaches based on transformers, when these
models are trained only with WFLW data, DTLD-s [Li et al.(2022)Li, Guo, Rhee,
Han, and Han] and SPLT [Xia et al.(2022)Xia, Qu, Huang, Zhang, Wang, and Xu],
both with 4.14 NME in the full set. If we analyze the performance on some of
the subsets, our method is 35%, 25%, 23% and 39% better than the previous
SOTA, ADNet [Huang et al.(2021)Huang, Yang, Li, Kim, and Wei], in the
illumination, make-up, occlusion and blur subsets. This proves the importance
of learning a global representation of the facial structure, that CNNs alone
do not provide. Additionally, the low FR across the different subsets and
better AUC values reaffirm that our model achieves a balanced trade-off
between robustness and precision, taking advantage of the complementary
benefits from the CNN and GAT architectures.

Metric | Method | Testset | Pose | Expression | Illumination | Make-up | Occlusion | Blur
---|---|---|---|---|---|---|---|---
| Bounding boxes from WFLW benchmark
$NME_{int-ocul}$ (%)($\downarrow$) |  3DDE [Valle et al.(2019)Valle, Buenaposada, Valdés, and Baumela] | 4.68 | 8.62 | 5.21 | 4.65 | 4.60 | 5.77 | 5.41
DeCaFA [Dapogny et al.(2019)Dapogny, Cord, and Bailly] | 4.62 | 8.11 | 4.65 | 4.41 | 4.63 | 5.74 | 5.38
AVS+SAN [Qian et al.(2019)Qian, Sun, Wu, Qian, and Jia] | 4.39 | 8.42 | 4.68 | 4.24 | 4.37 | 5.60 | 4.86
AWing [Wang et al.(2019)Wang, Bo, and Fuxin] | 4.36 | 7.38 | 4.58 | 4.32 | 4.27 | 5.19 | 4.96
Bounding boxes from GT landmarks (HRnet [Wang et al.(2021)Wang, Sun, Cheng,
Jiang, Deng, Zhao, Liu, Mu, Tan, Wang, Liu, and Xiao] annotations)
GlomFace [Zhu et al.(2022)Zhu, Wan, Xie, Li, and Gu] | 4.81 | 8.17 | - | - | - | 5.14 | -
LUVLI [Kumar et al.(2020)Kumar, Marks, Mou, Wang, Jones, Cherian, Koike-Akino, Liu, and Feng] | 4.37 | 7.56 | 4.77 | 4.30 | 4.33 | 5.29 | 4.94
SDFL [Lin et al.(2021)Lin, Zhu, Wang, Liao, Qian, Lu, and Zhou] | 4.35 | 7.42 | 4.63 | 4.29 | 4.22 | 5.19 | 5.08
AWing [Wang et al.(2019)Wang, Bo, and Fuxin] | 4.21 | 7.21 | 4.46 | 4.23 | 4.02 | 4.99 | 4.82
SLD [Li et al.(2020)Li, Lu, Zheng, Liao, Lin, Luo, Cheng, Xiao, Lu, Kuo, and Miao] | 4.21 | 7.36 | 4.49 | 4.12 | 4.05 | 4.98 | 4.82
HIHc111Use RetinaFace detections. [Lan et al.(2021)Lan, Hu, and Cheng] | 4.18 | 7.20 | 4.19 | 4.45 | 3.97 | 5.00 | 4.81
ADNet [Huang et al.(2021)Huang, Yang, Li, Kim, and Wei] | 4.14 | 6.96 | 4.38 | 4.09 | 4.05 | 5.06 | 4.79
DTLD-s [Li et al.(2022)Li, Guo, Rhee, Han, and Han] | 4.14 | - | - | - | - | - | -
SPLT [Xia et al.(2022)Xia, Qu, Huang, Zhang, Wang, and Xu] | 4.14 | 6.96 | 4.45 | 4.05 | 4.00 | 5.06 | 4.79
SPIGA (Ours) | 4.06 | 7.14 | 4.46 | 4.00 | 3.81 | 4.95 | 4.65
$FR_{10}$ (%)($\downarrow$) |  GlomFace [Zhu et al.(2022)Zhu, Wan, Xie, Li, and Gu] | 3.77 | 17.48 | - | - | - | 6.73 | -
DTLD-s [Li et al.(2022)Li, Guo, Rhee, Han, and Han] | 3.44 | - | - | - | - | - | -
LUVLI [Kumar et al.(2020)Kumar, Marks, Mou, Wang, Jones, Cherian, Koike-Akino, Liu, and Feng] | 3.12 | 15.95 | 3.18 | 2.15 | 3.40 | 6.39 | 3.23
SDFL [Lin et al.(2021)Lin, Zhu, Wang, Liao, Qian, Lu, and Zhou] | 2.72 | 12.88 | 1.59 | 2.58 | 2.43 | 5.71 | 3.62
AWing [Wang et al.(2019)Wang, Bo, and Fuxin] | 2.04 | 9.20 | 1.27 | 2.01 | 0.97 | 4.21 | 2.72
SLD [Li et al.(2020)Li, Lu, Zheng, Liao, Lin, Luo, Cheng, Xiao, Lu, Kuo, and Miao] | 3.04 | 15.95 | 2.86 | 2.72 | 1.46 | 5.29 | 4.01
HIHc1 [Lan et al.(2021)Lan, Hu, and Cheng] | 2.96 | 15.03 | 1.59 | 2.58 | 1.46 | 6.11 | 3.49
|  ADNet [Huang et al.(2021)Huang, Yang, Li, Kim, and Wei] | 2.72 | 12.72 | 2.15 | 2.44 | 1.94 | 5.79 | 3.54
|  SPLT [Xia et al.(2022)Xia, Qu, Huang, Zhang, Wang, and Xu] | 2.76 | 12.27 | 2.23 | 1.86 | 3.40 | 5.98 | 3.88
| SPIGA (ours) | 2.08 | 11.66 | 2.23 | 1.58 | 1.46 | 4.48 | 2.20
$AUC_{10}$ (%)($\uparrow$) |  AWing [Wang et al.(2019)Wang, Bo, and Fuxin] | 58.95 | 33.37 | 57.18 | 59.58 | 60.17 | 52.75 | 53.93
SLD [Li et al.(2020)Li, Lu, Zheng, Liao, Lin, Luo, Cheng, Xiao, Lu, Kuo, and Miao] | 58.93 | 31.50 | 56.63 | 59.53 | 60.38 | 52.35 | 53.29
HIHc1 [Lan et al.(2021)Lan, Hu, and Cheng] | 59.70 | 34.20 | 59.00 | 60.60 | 60.40 | 52.70 | 54.90
ADNet [Huang et al.(2021)Huang, Yang, Li, Kim, and Wei] | 60.22 | 34.41 | 52.34 | 58.05 | 60.07 | 52.95 | 54.80
|  SPLT [Xia et al.(2022)Xia, Qu, Huang, Zhang, Wang, and Xu] | 59.50 | 34.80 | 57.40 | 60.10 | 60.50 | 51.50 | 53.50
| SPIGA (Ours) | 60.56 | 35.31 | 57.97 | 61.31 | 62.24 | 53.31 | 55.31
Table 2: Evaluation of landmark detection on WFLW.

On the other hand, results of subsets where our approach is not competitive
also bear some relevant insights. First, further research is needed in the
expression subset, where our performance is not as good as the rest. This is
due to the fact that the 3D facial model used to initialize the cascade is
rigid (see Fig. 3). Second, seemingly, in the pose subset, we are not the top
performers. However, as we can see in Fig. 3, faces with extreme poses are not
well annotated and self-occlusions are not marked. So, the evaluation on this
subset of WFLW is questionable.

Figure 3: WFLW results on expressions (first 2 cols.) and pose examples (last
4 cols.). Shown in blue the ground truth and in green estimated landmarks.

MERL-RAV is one of the newest datasets, created to evaluate 2D facial
alignment in-the-wild. It improves landmark annotations at half-profile and
profile images by labeling the self-occlusion of landmarks. Hence, this
dataset allows to correctly measure the performance of landmark detectors on
samples with extreme poses. As we can see in Table 3, in terms of NMEbox, our
model is 6% better than LUVLI’s [Kumar et al.(2020)Kumar, Marks, Mou, Wang,
Jones, Cherian, Koike-Akino, Liu, and Feng] baseline, performing the best in
all pose subsets.

|  NMEbox(%)($\downarrow$) |  AUC${}^{7}_{box}$(%)($\uparrow$)
---|---|---
Method | All | Frontal | Half-Prof. | Profile | All | Frontal | Half-Prof. | Profile
DU-Net | 1.99 | 1.89 | 2.50 | 1.92 | 71.80 | 73.25 | 64.78 | 72.79
LUVLI [Kumar et al.(2020)Kumar, Marks, Mou, Wang, Jones, Cherian, Koike-Akino, Liu, and Feng] | 1.61 | 1.74 | 1.79 | 1.25 | 77.08 | 75.33 | 74.69 | 82.10
SPIGA (Ours) | 1.51 | 1.62 | 1.68 | 1.19 | 78.47 | 76.96 | 75.64 | 83.00
Table 3: Evaluation of landmark detection on MERL-RAV.

Finally, to verify the generalization and performance against occlusions, we
conduct a cross-dataset experiment training with the 300W public split and
testing with COFW-68 and 300W private. Results are summarized in Table 4. They
prove the importance of the graph attention mechanism, which dynamically
weighs landmark relationships according to the local image appearance and
relative position, versus a learned static relationship approach, such as SLD
[Li et al.(2020)Li, Lu, Zheng, Liao, Lin, Luo, Cheng, Xiao, Lu, Kuo, and
Miao], ($NME_{int-ocul}$ of 3.93 vs 4.22 in COFW-68). Further, SPIGA trained
on the 300W public dataset beats LUVLI [Kumar et al.(2020)Kumar, Marks, Mou,
Wang, Jones, Cherian, Koike-Akino, Liu, and Feng] ($NME_{box}$ of 2.52 vs 2.75
in COFW-68) with a backbone that has half the number of HG modules. It also
obtains comparable results to a recent transformer-based method trained from
scratch, DTLD-s [Li et al.(2022)Li, Guo, Rhee, Han, and Han]. It is marginally
better than DTLD-s in 300W private and worse in COFW-68. These results prove
that a general architecture using GATs can complement and enhance CNN-based
models, reaching better results in situations where ambiguity or noise is
contaminating the local landmark appearance, where preserving structural
landmarks consistency contributes to the final solution.

|  $NME_{box}$ (%)($\downarrow$) |  $AUC_{box}^{7}$ (%)($\uparrow$) |  $NME_{int-ocul}$ (%)($\downarrow$)
---|---|---|---
| 300W priv. | COFW-68 | 300W priv. | COFW-68 | COFW-68
HRNetV2-W18 [Wang et al.(2021)Wang, Sun, Cheng, Jiang, Deng, Zhao, Liu, Mu, Tan, Wang, Liu, and Xiao] | - | - | - | - | 5.06
HG$\times$1+SAAT [Zhu et al.(2021)Zhu, Li, Li, and Dai] | - | - | - | - | 4.61
LUVLI(8) [Kumar et al.(2020)Kumar, Marks, Mou, Wang, Jones, Cherian, Koike-Akino, Liu, and Feng] | 2.24 | 2.75 | 68.3 | 60.8 | -
GlomFace [Zhu et al.(2022)Zhu, Wan, Xie, Li, and Gu] | - |  2.69 222Result comes from a personal communication with authors of [Zhu et al.(2022)Zhu, Wan, Xie, Li, and Gu], 2.09 mistakenly in the paper. | - | - | 4.21
SLD [Li et al.(2020)Li, Lu, Zheng, Liao, Lin, Luo, Cheng, Xiao, Lu, Kuo, and Miao] | - | - | - | - | 4.22
SDFL [Lin et al.(2021)Lin, Zhu, Wang, Liao, Qian, Lu, and Zhou] | - | - | - | - | 4.18
SPLT [Xia et al.(2022)Xia, Qu, Huang, Zhang, Wang, and Xu] | - | - | - | - | 4.10
DTLD-s [Li et al.(2022)Li, Guo, Rhee, Han, and Han] | 2.05 | 2.47 | 70.9 | 65.0 | -
SPIGA(4) (ours) | 2.03 | 2.52 | 71.0 | 64.1 | 3.93
Table 4: Landmark detection results on 300W private and COFW-68. In (·) we
show the number of HG modules.

###  3.4 Ablation Study

We conduct our ablation study on WFLW to understand how SPIGA components
impact specific subset metrics. Table 5 shows that the addition of the cascade
shape regressor outperforms the bare MTN backbone (using SoftArgMax). Our new
relative positional encoding is better than stacking the vector
${\mathchoice{\mbox{\boldmath$\displaystyle\bf
q$}}{\mbox{\boldmath$\textstyle\bf q$}}{\mbox{\boldmath$\scriptstyle\bf
q$}}{\mbox{\boldmath$\scriptscriptstyle\bf q$}}}_{t}^{l}$ with the visual
features, and much better than using no positional information. The estimation
of an attention per layer with the GAT improves with respect to use of a
common attention matrix (GCN). An extended view of the effect of the learned
adjacency matrix is shown in Fig. 4. Occlusion images show how the attention
mechanism relies on visible landmarks regardless of the layer. The regressor
"looks" at distant and unoccluded landmarks at the first GAT layer and then at
closer ones in the last layers. The contribution of the proposed coarse-to-
fine scheme w.r.t. a constant size window ($w=8$) or a single pixel window
($w=1$) is also clear in Table 5. The improvement provided by SPIGA can be
seen across all metrics. However, it is more prominent with the hard cases, as
demonstrated by the results for the subsets Makeup, Occlusion, and Blur, and
the $NPE_{90}$ of the full set.

Figure 4: Left pupil attention mechanism at first and last layer, respectively, of the first regressor step. Changed from SPIGA model: | Full | Make-up | Occlusion | Blur
---|---|---|---|---
Changed |  From $\rightarrow$ To | $NME$ | $NPE_{90}$ | $NME$ | $NPE_{90}$ | $NME$ | $NPE_{90}$ | $NME$ | $NPE_{90}$
Shape model |  SPIGA $\rightarrow$ MTN backbone | 4.13 | 6.93 | 4.06 | 7.43 | 5.10 | 8.58 | 4.81 | 7.70
Positional encoding |  SPIGA $\rightarrow$ w/o pos. encod. | 4.17 | 7.07 | 4.01 | 6.71 | 5.03 | 8.33 | 4.72 | 7.52
SPIGA $\rightarrow$ stacking | 4.09 | 6.87 | 3.83 | 6.47 | 4.97 | 8.15 | 4.68 | 7.37
Attention |  GAT $\rightarrow$ GCN | 4.08 | 6.79 | 3.84 | 6.54 | 4.98 | 8.05 | 4.68 | 7.37
Coarse-to-Fine |  $w=16,8,4$ $\rightarrow$ $w=1,1,1$ | 4.12 | 6.95 | 3.88 | 6.76 | 4.99 | 8.19 | 4.71 | 7.44
$w=16,8,4$ $\rightarrow$ $w=8,8,8$ | 4.08 | 6.84 | 3.82 | 6.53 | 4.98 | 8.13 | 4.67 | 7.43
- | Best SPIGA model | 4.06 | 6.76 | 3.81 | 6.32 | 4.95 | 8.09 | 4.65 | 7.31
Table 5:  Contribution of the SPIGA components to the $NME_{int-
ocul}$($\downarrow$) and $NPE_{90}$($\downarrow$) in WFLW.

In each row of Table 6, we display respectively the performance of three SPIGA
models configured with one, two and three steps cascade. In each column, we
show the NME obtained at each step. The final NME is reduced gradually as we
increase the number of steps. Further, shorter cascades tend to have a better
NME at the first step (4.17 vs 4.22). However, given also the larger FR they
achieve (2.60 vs 2.44), we can conclude that longer cascades focus their first
steps on improving their robustness.

| Step 1 | Step 2 | Step 3
---|---|---|---
Method | $NME_{int-ocul}$ | $AUC_{10}$ | $FR_{10}$ | $NME_{int-ocul}$ | $AUC_{10}$ | $FR_{10}$ | $NME_{int-ocul}$ | $AUC_{10}$ | $FR_{10}$
| $(\downarrow)$ | $(\uparrow)$ | $(\downarrow)$ | $(\downarrow)$ | $(\uparrow)$ | $(\downarrow)$ | $(\downarrow)$ | $(\uparrow)$ | $(\downarrow)$
SPIGA(1) | 4.17 | 59.53 | 2.60 | - | - | - | - | - | -
SPIGA(2) | 4.17 | 59.55 | 2.44 | 4.07 | 60.45 | 2.20 | - | - | -
SPIGA(3) | 4.22 | 59.10 | 2.44 | 4.08 | 60.41 | 2.12 | 4.06 | 60.56 | 2.08
Table 6: SPIGA results for cascades with different number of steps, shown in
().

In Fig. 5 we show the initialization and the landmark locations estimated at
each step of the regressor cascade. When the face displays a neutral
expression (top row), the initialization is reasonably good and the model
converges to a solution within one regression step. Since SPIGA initializes
landmarks with a 3D model featuring a neutral expression, when the face
displays any other configuration, the initialization is much worse (lower
row). However, even in this situation, the model is able to estimate the
correct landmark locations in three regression steps.

Figure 5: Estimated landmark locations: from 2D projection of the rigid 3D
model (left) to the final result after the 3 regressor steps (right).

##  4 Conclusions

We presented SPIGA, a face landmark regressor that combines a CNN with a
cascade of Graph Attention Networks (GATs). The CNN provides the local
appearance representation. The GAT regressor is endowed with a positional
encoding and attention mechanism that learn the geometrical relationship among
landmarks and encourage the model to produce plausible face shapes. It
establishes a new SOTA in the WLFW, COFW-68 and MERL-RAV datasets. In our
experimentation we verify that the positional encoding is the component that
contributes most to the final result and the first steps of the cascade focus
on improving the robustness. In addition, at each step, the regressor "looks"
at distant and reliable landmarks in the first GAT layer and progressively
focuses its attention on closer landmarks in the following ones. These
insights from our ablation analysis confirm that SPIGA is learning a global
representation and explains why its improvement is most significative in
challenging situations involving occlusions, heavy make-up, blur and extreme
illumination.

## Acknowledgements

The following funding is gratefully acknowledged. Andrés Prados was funded by
the Comunidad de Madrid, Ayudantes de Investigación grant
PEJ-2019-AI/TIC-15032. José M. Buenaposada is funded by the Comunidad de
Madrid project RoboCity2030-DIH-CM (S2018 /NMT-4331).

## References

  * [Burgos-Artizzu et al.(2013)Burgos-Artizzu, Perona, and Dollar] Xavier P. Burgos-Artizzu, Pietro Perona, and Piotr Dollar.  Robust face landmark estimation under occlusion.  In _ICCV_ , pages 1513–1520, 2013.
  * [Cao et al.(2014)Cao, Wei, Wen, and Sun] Xudong Cao, Yichen Wei, Fang Wen, and Jian Sun.  Face alignment by explicit shape regression.  _IJCV_ , 107(2):177–190, 2014.
  * [Dapogny et al.(2019)Dapogny, Cord, and Bailly] Arnaud Dapogny, Matthieu Cord, and Kevin Bailly.  Decafa: Deep convolutional cascade for face alignment in the wild.  In _ICCV_ , pages 6892–6900. IEEE, 2019.
  * [Dollar et al.(2010)Dollar, Welinder, and Perona] Piotr Dollar, Peter Welinder, and Pietro Perona.  Cascaded pose regression.  In _CVPR_ , pages 1078–1085, 2010.
  * [Fard et al.(2021)Fard, Abdollahi, and Mahoor] Ali Pourramezan Fard, Hojjat Abdollahi, and Mohammad H. Mahoor.  Asmnet: A lightweight deep neural network for face alignment and pose estimation.  In _CVPRW_ , pages 1521–1530. CVF/IEEE, 2021.
  * [Feng et al.(2020)Feng, Kittler, Awais, and Wu] ZH. Feng, J. Kittler, M. Awais, and Xiao-Jun Wu.  Rectified wing loss for efficient and robust facial landmark localisation with convolutional neural networks.  _IJCV_ , 128:2126–2145, 2020.
  * [Feng et al.(2018)Feng, Kittler, Awais, Huber, and Wu] Zhen-Hua Feng, Josef Kittler, Muhammad Awais, Patrik Huber, and Xiao-Jun Wu.  Wing loss for robust facial landmark localisation with convolutional neural networks.  In _CVPR_ , pages 2235–2245, 2018.
  * [Honari et al.(2016)Honari, Yosinski, Vincent, and Pal] Sina Honari, Jason Yosinski, Pascal Vincent, and Christopher J. Pal.  Recombinator networks: Learning coarse-to-fine feature aggregation.  In _CVPR_ , pages 5743–5752, 2016.
  * [Huang et al.(2020)Huang, Deng, Shen, Zhang, and Ye] Xiehe Huang, Weihong Deng, Haifeng Shen, Xiubao Zhang, and Jieping Ye.  Propagationnet: Propagate points to curve to learn structure information.  In _CVPR_ , June 2020.
  * [Huang et al.(2021)Huang, Yang, Li, Kim, and Wei] Yangyu Huang, Hao Yang, Chong Li, Jongyoo Kim, and Fangyun Wei.  Adnet: Leveraging error-bias towards normal direction in face alignment.  In _ICCV_ , pages 3080–3090, October 2021.
  * [Jaderberg et al.(2015)Jaderberg, Simonyan, Zisserman, and Kavukcuoglu] Max Jaderberg, Karen Simonyan, Andrew Zisserman, and Koray Kavukcuoglu.  Spatial transformer networks.  In Corinna Cortes, Neil D. Lawrence, Daniel D. Lee, Masashi Sugiyama, and Roman Garnett, editors, _NeurIPS_ , pages 2017–2025, 2015.
  * [Kowalski et al.(2017)Kowalski, Naruniec, and Trzcinski] Marek Kowalski, Jacek Naruniec, and Tomasz Trzcinski.  Deep alignment network: A convolutional neural network for robust face alignment.  In _CVPRW_ , pages 2034–2043, 2017.
  * [Kumar et al.(2020)Kumar, Marks, Mou, Wang, Jones, Cherian, Koike-Akino, Liu, and Feng] Abhinav Kumar, Tim K. Marks, Wenxuan Mou, Ye Wang, Michael Jones, Anoop Cherian, Toshiaki Koike-Akino, Xiaoming Liu, and Chen Feng.  Luvli face alignment: Estimating landmarks’ location, uncertainty, and visibility likelihood.  In _CVPR_ , pages 8233–8243, 2020.
  * [Lan et al.(2021)Lan, Hu, and Cheng] Xing Lan, Qinghao Hu, and Jian Cheng.  Revisting quantization error in face alignment.  In _ICCVW_ , pages 1521–1530, October 2021.
  * [Li et al.(2022)Li, Guo, Rhee, Han, and Han] Hui Li, Zidong Guo, Seon-Min Rhee, Seungju Han, and Jae-Joon Han.  Towards accurate facial landmark detection via cascaded transformers.  In _Proceedings of the IEEE/CVF CVPR_ , pages 4176–4185, June 2022\.
  * [Li et al.(2020)Li, Lu, Zheng, Liao, Lin, Luo, Cheng, Xiao, Lu, Kuo, and Miao] Weijian Li, Yuhang Lu, Kang Zheng, Haofu Liao, Chihung Lin, Jiebo Luo, Chi-Tung Cheng, Jing Xiao, Le Lu, Chang-Fu Kuo, and Shun Miao.  Structured landmark detection via topology-adapting deep graph learning.  In Andrea Vedaldi, Horst Bischof, Thomas Brox, and Jan-Michael Frahm, editors, _ECCV_ , pages 266–283. Springer International Publishing, 2020\.
  * [Lin et al.(2021)Lin, Zhu, Wang, Liao, Qian, Lu, and Zhou] Chunze Lin, Beier Zhu, Quan Wang, Renjie Liao, Chen Qian, Jiwen Lu, and Jie Zhou.  Structure-coherent deep feature learning for robust face alignment.  _IEEE TIP_ , 30:5313–5326, 2021.
  * [Moskvyak et al.(2021)Moskvyak, Maire, Dayoub, and Baktashmotlagh] Olga Moskvyak, Frederic Maire, Feras Dayoub, and Mahsa Baktashmotlagh.  Keypoint-aligned embeddings for image retrieval and re-identification.  In _WACV_ , pages 676–685, January 2021.
  * [Qian et al.(2019)Qian, Sun, Wu, Qian, and Jia] Shengju Qian, Keqiang Sun, Wayne Wu, Chen Qian, and Jiaya Jia.  Aggregation via separation: Boosting facial landmark detector with semi-supervised style translation.  In _ICCV_ , October 2019.
  * [Sagonas et al.(2016)Sagonas, Tzimiropoulos, Zafeiriou, and Pantic] Christos Sagonas, Georgios Tzimiropoulos, Stefanos Zafeiriou, and Maja Pantic.  300 faces in-the-wild challenge: database and results.  _IVC_ , 47:3–18, 2016.
  * [Santoro et al.(2017)Santoro, Raposo, Barrett, Malinowski, Pascanu, Battaglia, and Lillicrap] A. Santoro, D. Raposo, D. G Barrett, M. Malinowski, R. Pascanu, P. Battaglia, and T. Lillicrap.  A simple neural network module for relational reasoning.  In _NeurIPS_ , 2017.
  * [Sarlin et al.(2020)Sarlin, DeTone, Malisiewicz, and Rabinovich] Paul-Edouard Sarlin, Daniel DeTone, Tomasz Malisiewicz, and Andrew Rabinovich.  Superglue: Learning feature matching with graph neural networks.  In _CVPR_ , June 2020.
  * [Sun et al.(2019)Sun, Li, Huan, Liu, and Han] Ning Sun, Qi Li, Ruizhi Huan, Jixin Liu, and Guang Han.  Deep spatial-temporal feature fusion for facial expression recognition in static images.  _PRL_ , 119:49–61, 2019.
  * [Trigeorgis et al.(2016)Trigeorgis, Snape, Nicolaou, Antonakos, and Zafeiriou] George Trigeorgis, Patrick Snape, Mihalis A. Nicolaou, Epameinondas Antonakos, and Stefanos Zafeiriou.  Mnemonic descent method: A recurrent process applied for end-to-end face alignment.  In _CVPR_ , pages 4177–4187, 2016.
  * [Valle et al.(2018)Valle, Buenaposada, Valdés, and Baumela] Roberto Valle, José M. Buenaposada, Antonio Valdés, and Luis Baumela.  A deeply-initialized coarse-to-fine ensemble of regression trees for face alignment.  In _ECCV_ , pages 609–624, 2018.
  * [Valle et al.(2019)Valle, Buenaposada, Valdés, and Baumela] Roberto Valle, José M. Buenaposada, Antonio Valdés, and Luis Baumela.  Face alignment using a 3D deeply-initialized ensemble of regression trees.  _CVIU_ , 189:102846, 2019.
  * [Valle et al.(2021)Valle, Buenaposada, and Baumela] Roberto Valle, José M. Buenaposada, and Luis Baumela.  Multi-task head pose estimation in-the-wild.  _IEEE TPAMI_ , 43(8):2874–2881, 2021.
  * [Velickovic et al.(2018)Velickovic, Cucurull, Casanova, Romero, Lio, and Bengio] Petar Velickovic, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Lio, and Yoshua Bengio.  Graph attention networks.  In _ICLR_ , 2018.
  * [Wang et al.(2021)Wang, Sun, Cheng, Jiang, Deng, Zhao, Liu, Mu, Tan, Wang, Liu, and Xiao] Jingdong Wang, Ke Sun, Tianheng Cheng, Borui Jiang, Chaorui Deng, Yang Zhao, Dong Liu, Yadong Mu, Mingkui Tan, Xinggang Wang, Wenyu Liu, and Bin Xiao.  Deep high-resolution representation learning for visual recognition.  _IEEE TPAMI_ , 43(10):3349–3364, 2021.
  * [Wang et al.(2019)Wang, Bo, and Fuxin] Xinyao Wang, Liefeng Bo, and Li Fuxin.  Adaptive wing loss for robust face alignment via heatmap regression.  In _ICCV_ , October 2019.
  * [Wu et al.(2018)Wu, Qian, Yang, Wang, Cai, and Zhou] Wayne Wu, Chen Qian, Shuo Yang, Quan Wang, Yici Cai, and Qiang Zhou.  Look at boundary: A boundary-aware face alignment algorithm.  In _CVPR_ , pages 2129–2138, 2018.
  * [Xia et al.(2022)Xia, Qu, Huang, Zhang, Wang, and Xu] Jiahao Xia, Weiwei Qu, Wenjian Huang, Jianguo Zhang, Xi Wang, and Min Xu.  Sparse local patch transformer for robust face alignment and landmarks inherent relation learning.  In _Proceedings of the IEEE/CVF CVPR_ , pages 4052–4061, June 2022\.
  * [Xu and Kakadiaris(2017)] Xiang Xu and Ioannis A. Kakadiaris.  Joint head pose estimation and face alignment framework using global and local CNN features.  In _IEEE Int. Conf. on Automatic Face and Gesture Recognition_ , pages 642–649. IEEE Computer Society, 2017.
  * [Yang et al.(2015)Yang, Mou, Zhang, Patras, Gunes, and Robinson] Heng Yang, Wenxuan Mou, Yichi Zhang, Ioannis Patras, Hatice Gunes, and Peter Robinson.  Face alignment assisted by head pose estimation.  In _BMVC_ , pages 130.1–130.13, 2015.
  * [Zhang et al.(2020)Zhang, Zeng, Wang, Pan, Liu, Liu, Ding, and Fan] Jiangning Zhang, Xianfang Zeng, Mengmeng Wang, Yusu Pan, Liang Liu, Yong Liu, Yu Ding, and Changjie Fan.  Freenet: Multi-identity face reenactment.  In _CVPR_ , pages 5325–5334, 2020.
  * [Zhu et al.(2021)Zhu, Li, Li, and Dai] Congcong Zhu, Xiaoqiang Li, Jide Li, and Songmin Dai.  Improving robustness of facial landmark detection by defending against adversarial attacks.  In _ICCV_ , pages 11751–11760, October 2021.
  * [Zhu et al.(2022)Zhu, Wan, Xie, Li, and Gu] Congcong Zhu, Xintong Wan, Shaorong Xie, Xiaoqiang Li, and Yinzheng Gu.  Occlusion-robust face alignment using a viewpoint-invariant hierarchical network architecture.  In _Proceedings of the IEEE/CVF CVPR_ , pages 11112–11121, June 2022\.


# Self adaptive global-local feature enhancement for Radiology Report
Generation

###### Abstract

Automated radiology report generation aims at automatically generating a
detailed description of medical images, which can greatly alleviate the
workload of radiologists and provide better medical services to remote areas.
Most existing works pay attention to the holistic impression of medical
images, failing to utilize important anatomy information. However, in actual
clinical practice, radiologists usually locate important anatomical
structures, and then look for signs of abnormalities in certain structures and
reason the underlying disease. In this paper, we propose a novel framework
AGFNet to dynamically fuse the global and anatomy region feature to generate
multi-grained radiology report. Firstly, we extract important anatomy region
features and global features of input Chest X-ray (CXR). Then, with the region
features and the global features as input, our proposed self-adaptive fusion
gate module could dynamically fuse multi-granularity information. Finally, the
captioning generator generates the radiology reports through multi-granularity
features. Experiment results illustrate that our model achieved the state-of-
the-art performance on two benchmark datasets including the IU X-Ray and
MIMIC-CXR. Further analyses also prove that our model is able to leverage the
multi-grained information from radiology images and texts so as to help
generate more accurate reports.

Index Terms—  Radiology Report Generation, Attention Mechanism, Image
Captioning

##  1 Introduction

Radiology reports consist of abundant abnormal observations about the medical
images are important to the clinical diagnosis. In practice, writing a
comprehensive and accurate radiology report is very time-consuming and
difficult [1]. Fully automated report generation can effectively assist
physicians in writing imaging reports, improving the accuracy of disease
detection while reducing their workload.

Recently, a lot of substantial progress has been made towards research on
automated radiology report generation models. Most frameworks adopt the
encoder-decoder architecture [1, 2, 3], e.g., a VIT [4] image encoder and then
a report decoder based on transformer decoder [5]. Nevertheless, physicians
typically write radiology reports through multi-granularity information,
instead of only focused on holistic impression. For instance, some sentences
”There is minimal patchy atelectasis or early infiltrate in left upper lung
zone” reflected some statements have a strong correlation with the
anatomies(region features). However, some sentences like ”No pneumothorax, or
pleural effusion.”, describing CXR from a global perspective is more likely to
be inspired by global features.

To explore the intrinsic relationship between different radiology reports,
anatomies and holistic impression, we propose a new Anatomy-enhanced framework
(AGFNet), which mimics the human visual system and thinking process. Inspired
by AnaXNet [6], our proposed AGFNet consists of three modules including a
feature extractor, an adaptive fusion module, and captioning generator. For
the feature extractor module, it consists of region and bilateral
global(frontal and lateral) feature extractor. Then, an adaptive fusion of
weight-based merging mechanisms is introduced to dynamically fuse multi-
granularity feature which match the pathological changes degree of the
patient. Finally, the captioning generator fed with multi-granularity features
generates the radiology report. In summary, our approach creates disease-
oriented visual features that can represent abnormal regions of the input
image. Meanwhile, during model training, the self-adaptive gate can
effectively learn the multi-grain relationships between images of different
pathological changes and corresponding radiology reports. The main
contributions of this paper are summarized as follows:

  * •

We propose a novel radiology report generation framework that integrates both
global and local anatomical visual features

  * •

We develop a novel self-adaptive fusion gate module to dynamically fuse the
global and region features.

  * •

In-depth experiment and analysis demonstrate that our proposed AGFNet model
outperforms previous baselines and state-of-the-art models.

##  2 Related works

Radiology report generation is most similar to image captioning. Some studies
use Reinforcement learning obtained a good results [7, 8]. Image retrieval and
knowledge embedding also achieved great success in this domain[9, 10]. The
encoder-decoder architecture, generally used in image captioning, is the most
successful architecture in radiology report generation[1, 2, 11, 3]. Most
methods based on encoder-decoder leverage the architecture like vision
transformer[4] as the image encoder[2, 11, 3], which split an input image as a
series of image patches. Then, decoders[12, 5]generate the target sequence
based on the all patch features. All these methods lead to the serious
destruction of region information, which result in the model failed to learn
the characteristics of region pathological changes and utilize the strong
relationship between different anatomies.

##  3 Methods

###  3.1 Model overview

Our model consists of three components: The global and region feature
extractor, the self-adaptive fusion gate can dynamically fuse global and
regional features according to the severity of the pathological changes, and
the captioning generator. Our proposed method is shown Fig. 1.

###  3.2 Global and Region feature extractor

The goal of the regional feature extractor is to accurately locate the most
prominent anatomy structures and obtain their robust features differentiated
by pathological changes. These features have fine-grained differences owing to
the severity level of the pathological changes. Consequently, we pretrained
ResNet101 Backbone Faster-RCNN [13] to locate the anatomies and apply the
RoIPooling [13] to extract anatomy regions of interest (ROIs) which frequently
appear in radiology reports, i.e., cardiac silhouette, left lower lung, left
mid lung zone, trachea, mediastinum, etc are extracted, denoted as

$V_{region}=\left\\{v_{cardiac},v_{trachea},v_{mediastinum},...\right\\}\in{\mathbf{R}^{N\times
d}}$

where N is the number of anatomies, each with d dimensionality. For global
features, the global feature of frontal view is generated by the backbone of
the region feature generator, the following is an fully connected layers layer
to flatten the feature map. The feature of lateral view is acquired by the
other ResNet101 backbone. Such a design is to reduce the computational
overhead. Briefly, the global feature is formulated as

$V_{global}=\left\\{V_{frontal},V_{lateral}\right\\}\in{\mathbf{R}^{2\times
d}}$

As shown in Table 1, the region feature extractor demonstrate an outstanding
result. We define right lung (RL), right apical zone (RAZ), right upper lung
zone (RULZ), right mid lung zone (RMLZ), right lower lung zone (RLLZ), right
costophrenic angle (RCA), left lung (LL), left apical zone (LAZ), left upper
lung zone (LULZ), left mid lung zone (LMLZ), left lower lung zone (LLLZ), left
costophrenic angle (LCA), mediastinum (Med), upper mediastinum (UMed), cardiac
silhouette (CS) and trachea (Trach).mAP is applied to evaluate the performence
of region feature detector.

Fig. 1: Illustration of the AFGNet, the feature extractor acquires local and
global features of the CXR. Then, the self-adaptive fusion gate generates
multi-granularity features related to the degree of pathological changes, and
finally generates diverse radiological reports through the captioning
generator. Table 1: Detection result of region feature extractor Detection
Results
---
Anatomy |<EMAIL_ADDRESS>|  Anatomy |<EMAIL_ADDRESS>
RL |  0.950 |  RAZ |  0.900
RAZ |  0.912 |  RMZ |  0.725
RLLZ |  0.905 |  LULZ |  0.950
LAZ |  0.887 |  LF |  0.776
LMLZ |  0.898 |  LLLZ |  0.852
Med |  0.773 |  UMed |  0.886
CS |  0.920 |  TR |  0.975

###  3.3 Self-Adaptive Fusion Gate

In clinical practice, there is a consensus that the radiologist writes the
radiology report in different granularity according to different degree of
pathological changes. This situation can be demonstrated in the dataset such
as IU-X-ray or MIMIC, e,g, most description in some normal cases is like “No
pneumothorax, or pleural effusion”, which is more likely concluded from a
global impression of the whole image. However, some cases with pathological
changes are often described in more detail, including many fine-grained
descriptions of abnormalities occurring in the associated anatomies, are
highly correlated with features of several pathological regions. Consequently,
in the radiologist decision process, the region feature and global feature
dynamical aggregated based on image morphology. Inspired by attention
mechanism[5], we propose the self-adaptive Fusion gate to dynamically fuse the
multi-granularity features. The self-adaptive fusion gate adopted region
features, front view features, lateral view features which can be indicated as
$V_{r},V_{f},V_{l}$. We use the global vector $V_{view}\in\mathbf{R}^{1\times
d}$ which is a element of $V_{global}=\left\\{V_{f},V_{l}\right\\}$ represents
the pathological manifestation as the $Q$ and region feature
$V_{r}\in\mathbf{R}^{N\times d}$ as a set of key-value pairs $K,V$, aims at
creating shortcuts between every region feature and the global feature. We
adopted multi-head attention $MHA$ as the main structure. The output is a
weighted sum of the values, where the weight assigned to each value is
determined by the dot-product of the gloabl feature with all region features:

$MHA(Q,K,V)=softmax(\frac{QK^{T}}{\sqrt{d_{k}}})V$

For the purpose of dynamically merging the biliteral view selected region
features, the weight coefficients of the two parts are considered to be added.
Therefore, the fused region feature $V_{\bar{r}}$ is acquired by:

$V_{\bar{r}}=MHA(V_{f},V_{r},V_{r})+MHA(V_{l},V_{r},V_{r})$

Finally, the self-adaptive fusion gate concatenate the fused region features
and global features:

$V_{MG}=[V_{\bar{r}};V_{f};V_{l}]$

The fusion gate is trained to learn how to dynamically control the fusion
degree of region features, and global features from the front view and lateral
view. For normal cases, global features was more reserved and the region
features are given less weight as the model need to generate more descriptive
sentence about the holistic health status. For a diseased case with
pathological changes occur in different anatomy regions, the self-adaptive
fusion gate endow higher coefficients to region features which contain visual
information more relevant to important pythology. Finally, we adopted the
transformer encoder [5] to learn the relationship between different anatomy
regions.

###  3.4 Captioning Generator

As R2Gen[2] adopted a hidden memory matrix in captioning generator to restore
the abnormal pattern information demonstrates its excellent performance, we
adopted the R2Gen decoder as our captioning generator, aiming at correcting
and updating the abundant information according to the different lesions
occurring in different anatomies.

Table 2: Results from the IU-Xray and MIMIC-CXR datasets. The best values are highlighted in bold. Dataset | Method | BL-1 | BL-2 | BL-3 | BL-4 | RG-L | MTOR
---|---|---|---|---|---|---|---
IU-Xray |  $R2Gen$ [2] | 0.470 | 0.304 | 0.219 | 0.165 | 0.371 | 0.187
$KERP$ [14] | 0.482 | 0.325 | 0.226 | 0.162 | 0.339 | -
$CMN$ [11] | 0.473 | 0.305 | 0.217 | 0.162 | 0.378 | 0.186
$AlignTransformer$ [3] | 0.484 | 0.313 | 0.225 | 0.168 | 0.379 | 0.204
$M2TR$ [15] | 0.486 | 0.317 | 0.232 | 0.170 | 0.390 | 0.192
$Ours$ | 0.505 | 0.345 | 0.243 | 0.176 | 0.396 | 0.205
MIMIC |  $R2Gen$ [2] | 0.353 | 0.218 | 0.145 | 0.103 | 0.277 | 0.128
-CXR |  $PPKED$ [16] | 0.360 | 0.224 | 0.149 | 0.106 | 0.284 | 0.149
$CMN$ [11] | 0.334 | 0.217 | 0.140 | 0.097 | 0.281 | 0.142
$AlignTransformer$ [3] | 0.378 | 0.235 | 0.156 | 0.112 | 0.283 | 0.158
$M2TR$ [15] | 0.378 | 0.232 | 0.154 | 0.107 | 0.283 | 0.145
$(Ours)$ | 0.363 | 0.235 | 0.164 | 0.118 | 0.301 | 0.136

##  4 EXPERIMENTS

In this section, we describe the foundation for our research, including
benchmark datasets, experimental settings, and evaluation metrics. Through
rigorous ablation experiments, we have verified the effectiveness of each
component.

###  4.1 Datasets, Metrics and Implementation details

We conduct experiments on MIMIC-CXR [17] and IU-XRay [18], which are the most
widely used in radiology report generation. For a fair comparison with
existing models, we use the standard evaluation toolkit [19] to compute widely
used metrics, namely BLEU, ROUGE-L and METEOR measure how well the generated
reports match the ground truth reports. We used 14 anatomy structures that
frequently appear in radiology reports as the region features
$V_{region}\in{\mathbf{R}^{14\times d}}$. The evaluation matrix of region
feature extractor is mean of average precision when IOU threthold is 0.5
(mAP@0.5), which is commonly used to analyze the performance of object
detection. The Adam optimizer is adopted and a batch size of 128. The feature
dimension d=2048, with learning rate of 2e-4 training for 100 epochs. The data
used to pre-train the region feature extractor is Chest Genome [20] . The
details of MIMIC-CXR and IU-XRay are shown in Table 3.

Dataset | IU X-Ray | MIMIC-CXR
---|---|---
Train | Val | Test | Train | Val | Test
Image | 5.2K | 0.7K | 1.5K | 369.0K | 3.0K | 5.2K
Report | 2.8K | 0.4K | 0.8K | 222.8K | 1.8K | 3.3K
Patient | 2.8K | 0.4K | 0.8K | 64.6K | 0.5K | 0.3K
Table 3: The statistics of training, validation and test sets of two benchmark
datasets, including the numbers of images, reports and patients.

###  4.2 Compare with Previous Work

Based on the IU-Xray and MIMIC-CXR datasets, we compare the our proposed
AGFnet with previous methods in this section. Table 2 demonstrates that our
approach significantly outperforms M2TR[15], the previous best SOTA method. In
IU-Xray dataset, our methods surpasses the previous sota work by 1.9%, 3.2%,
0.9%, 1.3% in BLUE-1, BLUE-2, BLUE-3, MTOR. In the MIMIC-CXR dataset, Our
method obtains and achieves a greater degree of optimization than other
methods in terms of BLUE-3, blue-4, and RG-L.The main results demonstrate the
effectiveness of our approach and show the advantages of our approach in
generating long report descriptions. The experiment shows potential of our
AGFnet in learning and generating more sophisticated radiology report for
cases with many pathological changes.

###  4.3 Ablation Study

To investigate the contribution of each component to our approach, we conduct
a rigorous ablation study on IU-Xray. It is obviously that the performance is
greatly enhanced by two components we developed, the global and region feature
extractor and the self-adaptive fusion gate. The removal of any of them will
cause significant performance deterioration. As shown in Table 4, when the
region feature extractor is removed, the BLUE-1 and the BLUE-2 scores
decreased by %3.5 and 4.1% respectively, and all other assessment metrics
declined with vary degrees. This proves that region feature extractor can
provide richer multi-granularity visual information for captioning generator.
Similarly, with the self-adaptive gate , the BLUE-1 and BLU-2 increase by 3.1%
and 2.0%. In summary, our proposed components improve on all metrics, which
demonstrate the prominent effectiveness of our approach.

Table 4: The experimental results of ablation studies, The best values are highlighted in bold. IU-Xray | BL-1 | BL-2 | BL-3 | BL-4 | RG-L | MTOR
---|---|---|---|---|---|---
Ours | 0.505 | 0.345 | 0.243 | 0.176 | 0.396 | 0.205
w/o region | 0.470 | 0.304 | 0.219 | 0.165 | 0.371 | 0.187
w/o global | 0.483 | 0.311 | 0.c224 | 0.171 | 0.387 | 0.194
w/o SAGate | 0.463 | 0.325 | 0.240 | 0.176 | 0.389 | 0.202

##  5 Conclusion

In our work, we propose a novel AGFNet to mine region-level semantic
information by extracting regional anatomy features. At the same time, the
self-adaptive fusion gate we developed can dynamically fusion the multi-
granularity features according to the degree of pathological changes. Based on
the MIMIC-CXR and IU-Xray datasets, experiments demonstrate that our technique
not only produces reports with correct aberrant descriptions but also achieves
state-of-the-art outcomes across all measures and surpasses the traditional
models. The analysis further demnonstrates the effectiveness of our approach
in generating detailed radiology report.

##  6 Acknowledgement

This study was funded by the National Key Research and Development Program of
China (2019YFB1404804), National Natural Science Foundation of China (grants
61906105, and 62272055), Young Elite Scientists Sponsorship Program by CAST
(2021QNRC001), Major Key Project of PCL (PCL2021A15).

## References

  * [1] Baoyu Jing, Pengtao Xie, and Eric Xing,  “On the automatic generation of medical imaging reports,”  arXiv preprint arXiv:1711.08195, 2017.
  * [2] Zhihong Chen, Yan Song, Tsung-Hui Chang, and Xiang Wan,  “Generating radiology reports via memory-driven transformer,”  in Proceedings of the Conference on Empirical Methods in Natural Language Processing 2020, 2020, pp. 1439–1449.
  * [3] Di You, Fenglin Liu, Shen Ge, Xiaoxia Xie, Jing Zhang, and Xian Wu,  “Aligntransformer: Hierarchical alignment of visual regions and disease tags for medical report generation,”  in International Conference on Medical Image Computing and Computer-Assisted Intervention. Springer, 2021, pp. 72–82.
  * [4] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al.,  “An image is worth 16x16 words: Transformers for image recognition at scale,”  arXiv preprint arXiv:2010.11929, 2020.
  * [5] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin,  “Attention is all you need,”  Advances in neural information processing systems, vol. 30, 2017\.
  * [6] Nkechinyere N Agu, Joy T Wu, Hanqing Chao, Ismini Lourentzou, Arjun Sharma, Mehdi Moradi, Pingkun Yan, and James Hendler,  “Anaxnet: Anatomy aware multi-label finding classification in chest x-ray,”  arXiv e-prints, pp. arXiv–2105, 2021.
  * [7] Han Qin and Yan Song,  “Reinforced cross-modal alignment for radiology report generation,”  in Findings of the Association for Computational Linguistics: ACL 2022, 2022, pp. 448–458.
  * [8] Yasuhide Miura, Yuhao Zhang, Emily Bao Tsai, Curtis P Langlotz, and Dan Jurafsky,  “Improving factual completeness and consistency of image-to-text radiology report generation,”  arXiv preprint arXiv:2010.10042, 2020.
  * [9] Shuxin Yang, Xian Wu, Shen Ge, Shaohua Kevin Zhou, and Li Xiao,  “Knowledge matters: Radiology report generation with general and specific knowledge,”  2022\.
  * [10] Sixing Yan,  “Memory-aligned knowledge graph for clinically accurate radiology image report generation,”  in Proceedings of the 21st Workshop on Biomedical Language Processing, 2022, pp. 116–122.
  * [11] Zhihong Chen, Yaling Shen, Yan Song, and Xiang Wan,  “Cross-modal memory networks for radiology report generation,”  arXiv preprint arXiv:2204.13258, 2022.
  * [12] Sepp Hochreiter and Jürgen Schmidhuber,  “Long short-term memory,”  Neural computation, vol. 9, no. 8, pp. 1735–1780, 1997.
  * [13] Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun,  “Faster r-cnn: Towards real-time object detection with region proposal networks,”  Advances in neural information processing systems, vol. 28, 2015\.
  * [14] Christy Y. Li, Xiaodan Liang, Zhiting Hu, and Eric P. Xing,  “Knowledge-driven encode, retrieve, paraphrase for medical image report generation,”  national conference on artificial intelligence, 2019.
  * [15] Farhad Nooralahzadeh, Nicolas Perez Gonzalez, Thomas Frauenfelder, Koji Fujimoto, and Michael Krauthammer,  “Progressive transformer-based generation of radiology reports,”  empirical methods in natural language processing, 2021.
  * [16] Fenglin Liu, Xian Wu, Shen Ge, Wei Fan, and Yuexian Zou,  “Exploring and distilling posterior and prior knowledge for radiology report generation,”  computer vision and pattern recognition, 2021.
  * [17] Alistair E. W. Johnson, Tom J. Pollard, Seth A. Berkowitz, Nathaniel R. Greenbaum, Matthew P. Lungren, Chih-Ying Deng, Roger G. Mark, and Steven Horng,  “Mimic-cxr: A large publicly available database of labeled chest radiographs,”  arXiv: Computer Vision and Pattern Recognition, 2019.
  * [18] Dina Demner-Fushman, Marc D Kohli, Marc B Rosenman, Sonya E Shooshan, Laritza Rodriguez, Sameer Antani, George R Thoma, and Clement J McDonald,  “Preparing a collection of radiology examinations for distribution and retrieval,”  Journal of the American Medical Informatics Association, vol. 23, no. 2, pp. 304–310, 2016.
  * [19] Xinlei Chen, Hao Fang, Tsung-Yi Lin, Ramakrishna Vedantam, Saurabh Gupta, Piotr Dollár, and C Lawrence Zitnick,  “Microsoft coco captions: Data collection and evaluation server,”  arXiv preprint arXiv:1504.00325, 2015.
  * [20] Joy T Wu, Nkechinyere N Agu, Ismini Lourentzou, Arjun Sharma, Joseph A Paguio, Jasper S Yao, Edward C Dee, William Mitchell, Satyananda Kashyap, Andrea Giovannini, et al.,  “Chest imagenome dataset for clinical reasoning,”  arXiv preprint arXiv:2108.00316, 2021.