# Occlusion-Aware Deep Convolutional Neural Network via Homogeneous Tanh-
transforms for Face Parsing

Jianhua Qiu<EMAIL_ADDRESS>Weihua Liu⋆<EMAIL_ADDRESS>Chaochao Lin
<EMAIL_ADDRESS>Jiaojiao Li<EMAIL_ADDRESS>Haoping Yu<EMAIL_ADDRESS>Said Boumaraf<EMAIL_ADDRESS>
###### Abstract

Face parsing infers a pixel-wise label map for each semantic facial component.
Previous methods generally work well for uncovered faces, however, they
overlook facial occlusion and ignore some contextual areas outside a single
face, especially when facial occlusion has become a common situation during
the COVID-19 epidemic. Inspired by the lighting phenomena in everyday life,
where illumination from four distinct lamps provides a more uniform
distribution than a single central light source, we propose a novel
homogeneous tanh-transform for image preprocessing, which is made up of four
tanh-transforms. These transforms fuse the central vision and the peripheral
vision together. Our proposed method addresses the dilemma of face parsing
under occlusion and compresses more information from the surrounding context.
Based on homogeneous tanh-transforms, we propose an occlusion-aware
convolutional neural network for occluded face parsing. It combines
information in both Tanh-polar space and Tanh-Cartesian space, capable of
enhancing receptive fields. Furthermore, we introduce an occlusion-aware loss
to focus on the boundaries of occluded regions. The network is simple,
flexible, and can be trained end-to-end. To facilitate future research of
occluded face parsing, we also contribute a new cleaned face parsing dataset.
This dataset is manually purified from several academic or industrial
datasets, including CelebAMask-HQ, Short-video Face Parsing, and the Helen
dataset, and will be made public. Experiments demonstrate that our method
surpasses state-of-the-art methods in face parsing under occlusion.

###### keywords:

face parsing , face occlusion , convolutional neural networks

###### PACS:

0000 , 1111

###### MSC:

0000 , 1111

††journal: Image and Vision Computing

[inst1]organization=Hunan University.,addressline=No.116 Lushan South Road,
city=Changsha, postcode=410082, state=Hunan, country=China

[inst2]organization=AthenaEyesCO., LTD.,addressline=Building 14, Zhongdian
Software Park, No. 39 Jianshan Road, city=Changsha, postcode=410205,
state=Hunan, country=China

[inst3]organization=Johns Hopkins University,addressline=3400 N Charles
Street, city=Baltimore, postcode=21218, state=Maryland, country=United States

[inst4]organization=Khalifa University of Science and
Technology,postcode=127788, state=Abu Dhabi, country=United Arab Emirates
††footnotetext: * Corresponding author: Weihua Liu

##  1 Introduction

Face parsing aims to predict labels for each pixel in input face images based
on pixel semantics, and the parsed results are widely used for other
applications, such as face analysis (Selfa and Qazi, 2005), face editing (De
Clerck and Proffit, 2015), face swapping (Bitouk et al., 2008) and face
recognition (Zhao et al., 2003). While parsing a face is very similar to
generic scene segmentation (Hariharan et al., 2014; Noh et al., 2015), they
are not the same because faces have similar contexts and translation
invariant. With the rapid development of convolutional neural networks (CNN)
(LeCun et al., 1989), a plethora of methods for basic face parsing tasks has
been widely developed due to the position-independent characteristic. (Yin et
al., 2021).

Previous face parsing methods have encountered challenges in effectively
addressing occlusions. Based on their modeling approaches, classical face
parsing methods are typically categorized into two main types: methods that
consider the entire face as a holistic entity, and methods that focus on
modeling individual facial components separately. The former examines an
entire face (Liu et al., 2015a), whereas the latter models each facial
component individually. (Liu et al., 2017; Luo et al., 2012). Lin et al.
(2019) ’s experiment shows that the former methods can label the background
and hair better, but their performance is worse than latter methods and they
mostly process the face in Cartesian coordinates system. The recent emergence
of non-Cartesian coordinate systems provides an alternative to the constraints
of classical methods. Liu et al. (2017) and Lin et al. (2019) perform the face
parsing tasks in a new coordinate system and achieve significant improvement.
However, when key face components, such as the nose and mouth, are occluded,
the performance of these methods suffers. Because of the COVID-19 pandemic,
the number of people wearing masks to cover their faces has increased
significantly (Wang et al., 2020), making occluded face parsing difficult.
Sunglasses, eye masks, and fake noses are also examples of facial occlusions.
In addition, sunglasses, eye masks, and fake noses are examples of facial
occlusions. As a result, when analyzing occluded faces, classical methods are
prone to misidentify occlusions as face components with similar positions,
resulting in unsatisfactory results.

The main cause of the aforementioned issues is summarized, which is that
previous methods frequently failed to pay attention to the edges of the face.
As a result, our method attempts to focus on the four corners of a face,
drawing inspiration from image illumination theory. More facial details can be
captured with the illumination theory when divide a face image into four parts
and perform a kind of transform into a non-Cartesian coordinate system,
because this is equivalent to dispersing a center point light source into an
area light source. The four slices of face details can be enhanced separately
by this process, and occlusions have a limited effect on all slices. This
process can improve the four slices of face details individually, and
occlusions have a limited effect on all slices. Based on this characteristic,
we propose a novel homogeneous Tanh-transforms, called four-point transform,
which transforms the image to the four Tanh-polar coordinate systems using the
four corner points of the face as the origin respectively. Tanh-polar
representation differs from the original Cartesian representation and it
contains the contextual information of the face image. Then, the transformed
face is fed into an occlusion-aware convolutional neural network. A new
building block, called Four-point block (FPB) and a special occlusion-aware
loss are designed to better extract occluded features. When compared to
existing face parsing method, the proposed architecture improves the
robustness of face parsing under occlusion and can be trained in an end-to-end
fashion. In order to improve the anti-occlusion capability of our model and to
validate the superiority of the method in facial occlusion, we present
Sheltered Face Parsing Dataset which contains 52,097 in-the-wild images. The
annotation categories are reorgnized for these images to include 11 face
regions and 3 occlusions. Extensive experiments show that these approaches
have strong anti-occlusion capabilities, as well as excellent performance for
both intra-dataset and cross-dataset evaluation.

Figure 1: Examples from our Sheltered Face Parsing Dataset and the color-coded
labels predicted by our proposed method, which performs precise parsing of
each facial component when some objects cover the face.

Fig. 1 shows the benefit of the proposed method on a few samples from the test
set. Our method is insensitive to various forms of occlusions and more
importantly, arrive at much stable predictions that hard to attain with
ordinary coordinate.

Overall, our contributions are summarized as follows:

  * 1.

A novel four-point transform neural network (FTNet) is proposed to address the
problem of face parsing under occlusion which takes full advantage of the
contextual information of the face image and report experimental results in
face parsing dataset. To better elucidate the motivation and principles of our
method, we used lighting scenarios to illustrate, akin to illumination theory.
As far as we know, this serves as an intriguing analogy.

  * 2.

The homogeneous Tanh-transforms, called four-point transform, is introduced,
which is used to warp faces into the Tanh-polar coordinate system by four
corner points of expanded bounding box. In Tanh-polar coordinate systems with
four points as the origin respectively, the key information of the facial
components is enhanced which can improve the generalization capability of
occluded face parsing.

  * 3.

Leveraging the proposed homogeneous Tanh-transforms, the structure of basic
convolutional neural network with Four-point Block (FPB) is designed for
occluded faces. Using our coordinate transform as constraints, a new loss
function named occlusion-aware loss is created, which enhances occlusion
segmentation. Our overall approach is simple and can be trained end-to-end.

  * 4.

As an additional contribution, the Sheltered Face Parsing Dataset is
presented, which is a novel in-the-wild sheltered face parsing dataset with
over 54 thousand images and we intend to make public to aid future research.

The remainder of the paper is organized as follows. Section 2 discusses
related work, Section 3 describes our proposed method, Section 4 reports the
experimental result and Section 5 concludes our contribution along with future
work.

##  2 Related Work

Face parsing research has yielded impressive results, thanks in large part to
advances in deep learning. This section provides a brief overview of related
work on face parsing, image polar transform, and occlusion handling.

###  2.1 Face Parsing

Most existing approaches for face parsing can be categorized into two groups:
global-based and local-based.

Global-based methods analyze the entire image and directly predict per-pixel
semantic labels. Earlier works concentrated on developing a model to represent
the spatial relationships of entire images (Warrell and Prince, 2009; Smith et
al., 2013). With the rapid development of CNN, face parsing models tend to use
network designs with multiple loss functions (Liu et al., 2015a). Zhou et al.
(2017) created a unified framework by combining FCN, super-pixel information,
and the CRF model. Nonetheless, Lin et al. (2019) ’s experiment showed that
these global-based approaches can only provide a limited accuracy due to the
lack of focus on facial components. Local-based methods train the model for
each facial component separately in order to predict the mask for each part
individually. Early works on local-based methods predicted all facial
components separately, then combined them based on spatial correlations (Luo
et al., 2012). Liu et al. (2017) improved accuracy using a two-stage approach
that included a shallow CNN and a spatially variant recurrent neural network.
ICNN (Zhou et al., 2015) incorporates two stages of face parsing, first
locating the facial component and then labelling the pixels of the identified
facial parts. Local-based methods can improve facial component accuracy, but
their accuracy in predicting hair and background is limited.

###  2.2 Image Polar Transform

Most existing computer vision models are built on images in a Cartesian
coordinate system. These models are very sensitive to many complex affine
transformations such as rotation and scaling due to the large number of
irreversible interpolations required. Therefore, many image processing
algorithms that convert the original image to a representation in another
coordinate system have been investigated. Among them, polar coordinate is a
well-known sampling method in the field of image processing.

Salehinejad et al. (2018) used a sampling method based on radial
transformation to augment training data, which transform each input image into
multiple images in polar coordinate systems, thus facilitate the training of
models from limited source data. Kim et al. (2018) proposed a user-guided
point as the origin of the polar coordinate system. The transformed image is
then segmented using a CNN. Jiang and Mei (2019) created a novel polar
coordinate CNN (PC-CNN) for rotation invariant feature learning, which
transformed the rotation invariance of training samples into translation
invariance through polar coordinate transform. Fu et al. (2018) introduced a
polar transform to represent the original image in a polar coordinate system,
which further improved the segmentation performance of the proposed
architecture. Furthermore, Lin et al. (2019) used the RoI Tanh-Warping to
alter the image’s coordinate system, which transformed the face in the image’s
center. The performance was then improved by Lin et al. (2021) by proposing a
Tanh-polar representation and introducing a well-designed CNN backbone.
However, unlike our proposed method, these methods ignore the impact of
occluded scenes on the face parsing task, rendering them ineffective in the
presence of occlusion.

###  2.3 Occlusion Handling

Occlusion handling strategies have been extensively studied. Gao et al. (2011)
proposed a segmentation-aware model with binary variables to handle
occlusions, which denote the visibility of objects in each bounding box cell.
Ghiasi et al. (2014) used deformable models with local templates to handle
occlusion in human pose estimation. Chen et al. (2015) used an energy
minimization framework to address the occlusion problem by incorporating top-
down class-specific inference and shape prediction as well as bottom-up
segments. Zheng et al. (2020) proposed an effective feature extraction
approach to capture the local and contextual information for occluded face
recognition. Ke et al. (2021) addressed occlusion through explicit modelling
of occlusion patterns in shape and appearance. Albalas et al. (2022) developed
a model to detect occluded or masked faces based on fused spatial-based
convolutional graphs of key facial features. However, they only focus on the
occlusions in other natural scenes, rather than the more difficult occlusions
on the face.

##  3 Methodology

This paper introduces a four-point transform network (FTNet) with homogeneous
Tanh-transforms for occluded face parsing. The homogeneous Tanh-transforms
perform four Tanh-polar transforms based on four points of an image, so we
also call it four-point transform, which will be explained in section 3.1.
Given a face image of Cartesian coordinate and a bounding box of the target
face, four-point transform warps the entire image into four Tanh-polar
coordinate systems and stitches them together. The proposed four-point
transform focuses on the non-occluded area of the target face to extract more
robust spatial features under occlusion.

Figure 2: The overview of proposed FTNet structure. Given an input image and
bounding box (in green), the four-point transform is applied to convert the
RoI (in red) of input image to a new representation. The warped origin of
input image (in white light) is defined as the corner of bounding box. The
proposed occlusion-aware CNN is mainly composed of multiple Four-point Block
(FPB) detailed in 3.2 to extract feature from the warped image. An occlusion-
aware loss is designed to guide the warped parsing label prediction. Finally,
the warped label is restored to the regular coordinate system.

The whole pipeline of our method is shown in Fig. 2. Our framework consists of
four major components: 1) the four-point transform, which warps input face
image of Cartesian coordinate into occlusion-aware Tanh-polar coordinate; 2)
the backbone feature extraction module, which extracts informative features
from the warped face for subsequent operations; 3) the parsing and restoring
modules, which predict the segmentation scores through up-samplings on the
feature map and restore the result into Cartesian coordinate; 4) the
occlusion-aware loss, which introduces a penalty coefficient for the false
prediction of occlusions. Next, each part will be introduced in detail.

###  3.1 Homogeneous Tanh-transforms and Restore

Previous face parsing works typically used face alignment techniques, which
resulted in peripheral information loss (Liu et al., 2015b). This is due to
large variations in the target face’s position, rotation, and size, and the
network’s input typically ignores the importance of the uncovered facial
components. Traditional FCN can handle a large portion of face parsing, but
its accuracy is limited, particularly when occlusion is present, due to a lack
of focusing on each non-occluded individual part. To solve this problem, we
propose homogeneous Tanh-transforms, called four-point transform, which warps
the target face into Tanh-polar coordinate through four focusing points and
makes precise parsing. Our warping approach transforms the entire image to the
multiple homogeneous Tanh-polar representation, which expands the ratio of the
region of interest (RoI) and compresses the background information, which lead
to less data loss (Lin et al., 2019, 2021).

Figure 3: The homogeneous Tanh-transforms pipeline, namely, four-point
transform. (1) Input image, bounding box and the RoI area (2) Four warping
regions according to four corners of bounding box (3) An example of the
homogeneous Tanh-transforms in one warping area (4) The image in Tanh-
Cartesian coordinate system and warping output in Tanh-polar coordinate
system. Note that semi-transparent area which is out of RoI area do not
participate in the calculation.

Fig. 3 illustrates the process of the homogeneous Tanh-transforms. Given an
image $I$, as shown in Fig. 3, the RoI rectangle
$r=(x_{1}^{\prime},x_{2}^{\prime},x_{3}^{\prime},x_{4}^{\prime})$ is presented
by multiplying the size of the bounding box $b=(x_{1},x_{2},x_{3},x_{4})$ by a
variable zoom factor $l$ and all subsequent calculations are performed in the
RoI region. Following the concept of warping, our method chooses four points
as the original warping points, which chosen from the corner of bounding box
$b$. These points divide the entire image into four warping regions for four-
point transform. According to the bounding box $b=(x_{1},x_{2},x_{3},x_{4})$ ,
calculate the width and height of $r$: $w=x_{2}-x_{1},h=y_{2}-y_{1}$. Then,
the image that warped by four-point transform can be defined as:

$I^{{}^{\prime}}=FT(I,r)=P(W(I_{i},I_{j},w,h)),\forall I_{i}\in I_{x},\forall
I_{j}\in I_{y}$

where $I^{\prime}$ is the warped image, $FT$ is the function of four-point
Transform, $P$ is a bilinear interpolation function which can resample
discrete coordinates of an image and project them onto determined coordinates
system linearly by using a typical interpolation method (Jaderberg et al.,
2015; Tai et al., 2019), and $W$ is the mapping function that projects every
pixel to the Tanh-polar coordinate.

In general, the center of bounding box is considered as the original point.
However, to determine the mapping function of four-point transform
$W(i,j,w,h)$ for each pixel $(i,j)$, it first need to transform the pixel
position to make sure the original points are separately located at the four
corners of bounding box. For an example, in the warping area 1 which
highlighted in red in Fig. 3, the relative position $(i^{\prime},j^{\prime})$
of (i,j) is defined as:

$i^{{}^{\prime}}=i-\frac{i}{\left|i\right|}\frac{w}{2},j^{{}^{\prime}}=j-\frac{j}{\left|j\right|}\frac{h}{2}$

As Fig. 3 shows, the vector $(x,y)$ is parallel to the vector
$(i^{\prime},j^{\prime})$, and $(x,y)$ is described by the canonical equation
of the ellipse $\frac{x^{2}}{a^{2}}+\frac{y^{2}}{b^{2}}=1$, whose major axes
$a=\frac{w}{2\sqrt{\pi}}$ and minor axes $b=\frac{w}{2\sqrt{\pi}}$ are related
to the height/width of the bounding box. Therefore, $x$ and $y$ satisfy the
following equation:

$x^{2}=\frac{(\frac{w}{2\sqrt{\pi}})^{2}\cdot((\frac{h}{2\sqrt{\pi}})^{2}-y^{2})}{(\frac{h}{2\sqrt{\pi}})^{2}}=\frac{w^{2}}{h^{2}}(\frac{h^{2}}{4\pi}-y^{2})$

Then, the Tanh-polar coordinate system by mapping function is defined as:

$W(i,j,w,h)=(\theta,\rho)=(\rm{tanh}(\frac{j^{\prime}}{i^{\prime}}),\rm{tanh}(\frac{\sqrt{i^{\prime
2}+j^{\prime 2}}}{\sqrt{a^{2}+b^{2}})})$

where $\theta$, $\rho$ are the polar angle and diameter, respectively.

For the other three warping regions, the homogeneous Tanh-transforms are the
same as described above. The only difference is the relative position
$(i^{\prime},j^{\prime})$ when different corners are used as the origin of the
coordinate system. Four transforms on the four warping regions constitute our
four-point transform method. As shown in Fig. 3, the image $I$ is warped into
four parts after this mapping algorithm and then merged together. It maps each
original Tanh-Cartesian coordinate system pixel to four Tanh-polar coordinate
systems and then interpolates unmapped pixels. Each part focuses on one corner
of the bounding box, and its area takes up the majority of the image. Though
our homogeneous Tanh-transforms, fixed-size images with higher proportions of
RoI regions can be produced. The transformed image contains all information
inside the bounding box, and some losses occur outside of the bounding box,
while enhancing the compressed information of non-RoI regions for use in
feature extraction modules.

For restore, it first recovers four split labels from four parts, then
calculates and combines them based on a selected basis. However, because some
duplicated warping areas exist in more than one part, as shown in Fig. 3, it
is necessary to perform the following step to restore the transformation,
which restore four individual sections that correspond to four parts in

$I=FT^{-1}(I^{{}^{\prime}},r)=P(W_{combine}^{-1}(W^{-1}(I_{i}^{{}^{\prime}},I_{j}^{{}^{\prime}},w,h))),\forall
I_{i}^{{}^{\prime}}\in I_{x}^{{}^{\prime}},\forall I_{j}^{{}^{\prime}}\in
I_{y}^{{}^{\prime}}$

where $W_{combine}^{-1}$ combines these four sections, then the selected basis
calculates and combines overlapped areas. In our system, the mean function is
chosen as our base calculation.

Figure 4: Comparison of original image and restored image. Observing that
outside the RoI area, some information losses.

Fig. 4 provides the comparison between the original image $I$ and the restored
image $W^{(}-1)W(I)$ . As the figure illustrates, we can observe that
$W^{(}-1)W(I)$ restores most of the information in the RoI area, and other
areas are basically recovered. Although some losses happen in the areas
outside of RoI, the losses are limited since most important labels are inside
the RoI area for face parsing.

###  3.2 Four-point Block

Given the warped face image $W(I,r)$, which contains the compressed
information of whole image through the four-point transform, the backbone
feature extraction modules are deployed to capture implicit features under
Tanh-polar coordinate. However, although the information of the rotation
equivariance can be preserved, four-point transform may face a risk of losing
the translation equivariance. Therefore, we upgrade the original residual
block of feature extraction to a new building block, called Four-point Block.
It is able to extract features more efficiently in a Tanh-polar coordinate.

Fig. 2 illustrates the branch structure of the FPB in detail. The four-point
branch in Tanh-polar coordinate system learns the rotation equivariant
representation. The Tanh-Cartesian branch, which transforms feature maps into
Tanh-Cartesian coordinate systems, learns the translation equivariant
representation, so as to achieve the fundamental purpose of complementary
learning. The Tanh-Cartesian coordinate system can be defined by

$TC(\vec{v}):=\rm{tanh}(\frac{i^{\prime}}{\sqrt{x^{2}+y^{2}}})\rm{tanh}(\frac{j^{\prime}}{\sqrt{x^{2}+y^{2}}})$

where $TC(i,j)$ is the function of Tanh-Cartesian transform.

The input to FPB is a four-point representation $X_{fp}$ of shape $(h,w,c)$.
The residual path uses a stack of $1\times 1$, $3\times 3$ and $1\times 1$
convolutions following the bottleneck design (He et al., 2016). The first
$1\times 1$ conv layer is used to reduce the channel dimension. Its output
feature maps are divided into two branches as described above. In Tanh-
Cartesian branch, the feature maps are split back to four four-point warping
regions and transformed to the Tanh-Cartesian space, respectively. In Tanh-
cartesian coordinate system, four $1\times 1$ conv layers are used to compute
these four-point feature maps, which then are transformed to the Tanh-polar
space through four-point transform and merged together. In four-point branch,
a $3\times 3$ conv layer is used to compute feature maps in Tanh-polar space.
Then, the feature maps from these two branches are multiplied in Tanh-polar
coordinate system. The last $1\times 1$ conv layer restores the channel
dimension so the residual representation can be added to the input $X_{fp}$.

###  3.3 FTNet Architecture

Through homogeneous Tanh-transforms, namely, four-point transform, the input
image of the Tanh-polar coordinate system is obtained which compresses more
RoI key information and surrounding semantics. For a higher quality feature
extraction module fusing the advantage of Tanh-Cartesian and Tanh-polar
representation, we propose FPB for composing our new backbone network, four-
point transform network (FTNet). FPBs are stacked according to the structure
of ResNet to build FTNet for feature extraction. Thanks to the grouped conv
$1\times 1$ and the conv $3\times 3$ with halved channels, the overall number
of parameters are less than the ResNet50 backbone. To predict the masks, we
use the FCN-8’s (Long et al., 2015) decoder. More advanced decoders
necessitate dedicated hyperparameter tuning, which can significantly affect
performance. To map the feature maps to pixel-wise prediction logits, the
decoder employs two conv $3\times 3$ layers and a bilinear up-sampling layer.
Each component of the transformed image is invertible in theory.

###  3.4 Occlusion-aware Loss

Based on homogeneous tanh-transforms, we design an occlusion robust network as
described above, namely FTNet. The focus of the loss function used in the
current face parsing field is mainly on the classification of pixels, while
ignoring the adverse effects of masks, glasses and other occlusions.
Therefore, to train our FTNet, an especial occlusion-aware loss is introduced
to further address the face occlusion problem in face parsing. Furthermore, in
order to better focus on occlusion semantic information, different weights are
assigned to occluded pixels and non-occluded pixels in contrast to treating
all pixel equally as the shape of the parsed face will be destroyed by
occlusion. So, the loss weight of the pixels in the occluded face region are
increased which added as the constraints of the loss function as follows,

$L_{FTNet}=l\cdot
L_{ce}+(1-l)L_{dice}+\alpha\frac{1}{N}\sum_{i}P(g_{i}^{f})||g_{i}^{f}-s_{i}^{f}||_{2}^{2}$

where

$L_{ce}=\frac{1}{N}\sum_{i}\sum_{c}g_{i}^{c}\log{s_{i}^{c}}$
$L_{dice}=1-\frac{2\sum_{i}\sum_{c}g_{i}^{c}s_{i}^{c}}{\sum_{i}\sum_{c}{g_{i}^{c}}^{2}+\sum_{i}\sum_{c}{s_{i}^{c}}^{2}}$

Here, $N$ is the total training samples of a batch, $\alpha$ is the
hyperparameters of constraints, $g_{i}^{c}$ denotes the ground truth of class
$c$ in $i$-th sample, $s_{i}^{c}$ denotes the predicted probability of class
$c$ in $i$-th sample, $f$ denotes the face skin class, $P(g_{i}^{f})$ is the
occlusion value of $g_{i}^{f}$ calculated from the mean of itself and the
surrounding 8 pixels ($P(g_{i}^{f})$ is high when $g_{i}^{f}$ is occluded) and
the weight $l$ is determined by the average point to curve Euclidean distance
among points around curve of predicted segmentation to the ground truth.

##  4 Experimental Evaluation

###  4.1 Implementation Details

Dataset. We perform experiments on a new dataset called Sheltered Face Parsing
Dataset (SFPD). SFPD has 52097 images, collected from CelebAMask-HQ (Lee et
al., 2020), the Short-video Face Parsing (denoted as SVFP) (maadaa.ai, 2021),
and the HELEN dataset (Le et al., 2012) which contains facial occlusion
challenges. Table 1 compares the number of different datasets and illustrates
that SFPD contains in-the-wild images and more occluded images which is of
higher difficulty to face parsing. The collected images are carefully screened
for no watermarks or blurring, and manually reannotated to eliminate label
discrepancies. Furthermore, SFPD is more concerned with facial occlusions
which presented by elaborate pixel-wise annotation, including 9 general face
part categories and 4 additional face occlusion categories (i.e., hand
occlusion(hand_occ), eyeglasses(e_glasses), sunglasses and mouth
mask(m_mask)).The images are divided into 42097 samples for training, 5000 for
validation and 5000 for testing, respectively.

Benchmark | images | In-the-wild | occluded images
---|---|---|---
CelebAMask-HQ | 27176 | $\times$ | 828
SVFP | 19445 | ✓ | 974
Helen | 2000 | ✓ | 60
Ours | 42097 | ✓ | 1381
Table 1: The comparison of training set of face parsing datasets.

Training. Our method is implemented on PyTorch and trained on two Tesla V100S
GPUs. For data preparation, we apply data augmentation techniques for all the
training samples to avoid over-fitting, e.g., random rotation and scale
augmentation. The rotation angle is randomly selected from
$(-30^{\circ},30^{\circ})$, while the scale factor is randomly selected from
$(0.75,1.25)$. During training, our four-point transformed images are warped
from original images and resized to $512\times 512$. For our proposed
occlusion-aware loss, the hyperparameters $\alpha$ is set to 2. For
optimization, we adopt the Stochastic Gradient Descent (SGD) and we use the
poly learning rate schedule (Chen et al., 2017),
$lr^{*}=(1-\frac{iter}{max\\_iter})^{p}$, $p=0.9$, in which the initial $lr$
is $0.01$. The $max\\_iter$ is $epochs\times batchsize$, where $batchsize=8$
and $epochs=50$.

Evaluation. All the testing procedures are carried out on a single Tesla V100S
GPU. For each test sample, the size of the predicted labels is set to
$512\times 512$ and scaled to original size for visualization. For evaluation,
we report F1 score (F1) for all categories and their mean. The predicted masks
are evaluated on the original image scale. The F1 can be represented by the
formula as follow:

$F=\frac{2\rm{Precision}\cdot\rm{Recall}}{\rm{Precision}+\rm{Recall}}$

###  4.2 Comparisons

The thorough comparisons are performed between our model and 6 state-of-the-
arts (i.e., Masi et al. (2020), Lin et al. (2021), Huang et al. (2021), Kim et
al. (2022), Li et al. (2023), Han and Yoon (2023)) on our dataset in Tab.2.
Lin et al. (2021) is related to our work which also adopt image transform and
Li et al. (2023) propose a de-occlusion module for parsing occluded faces in a
semi-supervised fashion, while other methods (Masi et al., 2020; Huang et al.,
2021; Kim et al., 2022; Han and Yoon, 2023) are currently performing well in
face parsing.The majority of the methods referenced are open-source. If not
available as open-source, we implemented the models ourselves, with
appropriate adjustments based on the literature’s hyperparameters.These
methods focus on the related to the shape of facial components to improve the
face parsing performance, in detail, including consistent based learning
mechanism (Masi et al., 2020), and attention based feature extraction (Huang
et al., 2021; Kim et al., 2022; Han and Yoon, 2023). Results are measured by
F1 which is commonly used by existing face parsing literature. We present the
comparison results for both non-occluded and occluded images from the SFPD
dataset in Tab.2 and Tab.3 separately. Each column shows the F1 percentage
corresponding to a specific face label, including our reannotation and the
overall F1-score as well.

Methods | skin | l_brow | r_brow | l_eye | r_eye
---|---|---|---|---|---
Masi et al. (2020) | 91.4 | 89.16 | 85.21 | 92.07 | 91.39
Lin et al. (2021) | 94.22 | 90.70 | 90.10 | 95.59 | 94.84
Huang et al. (2021) | 90.87 | 89.23 | 85.13 | 90.14 | 90.72
Kim et al. (2022) | 92.89 | 91.20 | 91.05 | 91.18 | 95.77
Li et al. (2023) | 91.73 | 90.24 | 90.32 | 89.78 | 89.94
Han and Yoon (2023) | 91.25 | 92.82 | 92.46 | 88.79 | 89.17
FTNet(Ours) | 92.04 | 94.31 | 93.26 | 92.71 | 91.50
Methods | nose | u_lip | l_lip | inner_mouth | overall
Masi et al. (2020) | 86.97 | 83.98 | 89.81 | 91.63 | 89.07
Lin et al. (2021) | 90.25 | 90.96 | 84.28 | 85.12 | 90.67
Huang et al. (2021) | 88.48 | 83.39 | 89.47 | 91.07 | 88.72
Kim et al. (2022) | 86.62 | 90.06 | 83.74 | 96.14 | 90.96
Li et al. (2023) | 90.64 | 89.37 | 89.52 | 92.17 | 90.41
Han and Yoon (2023) | 89.82 | 90.23 | 90.11 | 89.87 | 90.50
FTNet(Ours) | 93.97 | 92.56 | 92.60 | 92.22 | 92.80
Table 2: Comparison with state-of-the-art methods on non-occluded images from the SFPD dataset. F1 scores are represented in percentage. Methods | h_occ | e_glasses | sunglasses | m_mask | overall
---|---|---|---|---|---
Masi et al. (2020) | 89.18 | 88.09 | 84.37 | 90.15 | 87.95
Lin et al. (2021) | 91.14 | 89.75 | 91.16 | 84.25 | 89.08
Huang et al. (2021) | 89.46 | 88.46 | 84.96 | 90.22 | 88.28
Kim et al. (2022) | 90.88 | 90.59 | 91.68 | 85.45 | 89.65
Li et al. (2023) | 90.87 | 89.46 | 91.29 | 90.81 | 90.61
Han and Yoon (2023) | 89.46 | 91.71 | 89.78 | 90.43 | 90.35
FTNet(Ours) | 90.06 | 93.41 | 93.47 | 92.75 | 92.42
Table 3: Comparison with state-of-the-art methods on occluded images from the
SFPD dataset. (h_occ:hand_occlusion, e_glasses:eye_glasses and
m_mask:mouth_mask)

As Tab.2 shows, our FTNet achieves better performance on non-occluded image
dataset (average F1 1.96 % better than the second best method Kim et al.
(2022), and 2.13 % better than the third best method Lin et al. (2021)). As
shown in Tab.2, Lin et al. (2021) outperforms our model in skin parsing.
However, as shown in Tab.3, it falls far short of our model in mouth mask
occlusion (by 8.5%). This is because Lin et al. (2021) perform the center as
the origin of the transform. As in the illumination theory, the radiation
range of the surface light source is better than that of the point light
source, so that the performance of their model will greatly reduce when the
center is occluded, such as face mask occlusion. However, under such
occlusion, our method can utilize the homogeneous tanh-transforms of the four
corners to alleviate the adverse effect of occlusion on face parsing.

In terms of the occlusions, Tab.3 also demonstrates that our proposed FTNet
(overall 92.42% ) exhibits superior robustness in occlusion scenarios,
outperforming Li et al. (2023) (overall 90.61%, the second-placed method) by a
margin of 1.81%. This further demonstrate the effectiveness of our proposed
method on occlusions.

The quantitative analysis of these methods is illustrated in Fig. 5. The
testing results on SFPD datasets (non-occluded images and occluded images)
show that our method outperforms all state-of-the-art methods. Such
performance gains are particularly impressive considering that improvement on
our dataset is very challenging.

Figure 5: Qualitative comparisons with state-of-the-art methods.

###  4.3 Ablation Study

To demonstrate how each component in our FTNet contributes to the performance,
a series of ablation experiments are conducted on SFPD using F1 metric. As the
proposed homogeneous Tanh-transforms, i.e., four-point transform, provides a
more general and powerful form of image representation, we first quantity the
effectiveness of each essential component of FTNet (Section 4.3.1), and then
perform in-depth analyses of homogeneous tanh-transforms (Section 4.3.2). This
would better verify our main points and coincide with the experiments in prior
conference papers. The training and evaluation followed the same protocol as
in Section 4.1.

####  4.3.1 Ablation Study for Each Component of FTNet

Table 5 shows the evaluation of our FTNet compared to ablated version without
certain key components, including homogeneous Tanh-transform, four-point
block, occlusion-aware loss. $V_{1}$ denotes the ablated version without
homogeneous tanh-transforms, $V_{2}$ denotes the ablated version without four-
point block, $V_{3}$ denotes the ablated version without occlusion-aware loss
and $V_{0}$ denotes the full FTNet. All the variants are retrained
independently with their specific architectures. we report their average
performance on face parts and occlusions (i.e., sunglasses, eyeglasses, face
mask) respectively.

Network | F1 for face parts | F1 for occlusions
---|---|---
full FTNet $V_{0}$ (homogeneous tanh-transforms \+ four-point block \+ occlusion-aware loss) | 92.76 | 93.21
ablated FTNet $V_{1}$ (four-point block \+ occlusion-aware loss) | 85.46 | 83.43
ablated FTNet $V_{2}$ (homogeneous tanh-transforms \+ occlusion-aware loss) | 86.71 | 88.31
ablated FTNet $V_{3}$ (homogeneous tanh-transforms \+ four-point block) | 85.28 | 85.16
Table 4: Result on Different Transform. F1 scores are represented in
percentage.

_Homogeneous tanh-transforms._ Consider the first rows in Table 4, we can find
that the homogeneous tanh-transforms provides substantial performance gain.
Note that the ablated network uses the original image as the input directly
instead of four-point transformed image.This demonstrates the benefit of four-
point transform and Tanh-polar representation in occluded face parsing.

_Four-point block._ Instead of original block in backbone, four-point block
can also bring a performance gain (85.28-92.76 for face parts, 85.16-93.21 for
occlusion). This suggests that feature fusion of Tanh-Cartesian and Tanh-polar
coordinate systems enables comprehensive understanding of facial occlusion
semantics.

_Occlusion-aware loss._ Comparing the performance of our full FTNet and the
ablated version without occlusion-aware loss, we can conclude that the
occlusion-aware loss boosts performance, as the occluded face margin can be
con-strained. This also provides a new glimpse into the learning objective and
constraint for occlusion handling.

####  4.3.2 In-depth Analyses for Homogeneous Tanh-transforms

Transform | face parts | sheltered areas
---|---|---
None | 80.75 | 73.79
Resize | 74.56 | 70.39
Tanh-Cartesian transform | 82.66 | 79.03
Tanh-Polar transform | 85.75 | 81.37
Four point transform (ours) | 92.76 | 93.21
Table 5: Result on Different Transform. F1 scores are represented in
percentage.

In above section, we investigate the necessity of comprehensive exploring
homogeneous tanh-transforms and the other component of FTNet. Next,we discuss
the effectiveness of the homogeneous Tanh-transforms representation.
Concretely, we studied three related transforms, as list in Tab. 5: 1) Resize,
2) Tanh-Cartesian transform by (Lin et al., 2019), 3) Tanh-Polar transform by
(Lin et al., 2021), 4) No transform. All transforms are applied to the dataset
before extracting features to constitute the variant model. All five variant
models use same backbone and other training parameters. We also evaluate these
five models on masked images, then check the dice loss, our loss and the F1
score for face parts or occlusions. We can draw a main conclusion that our
homogeneous tanh-transforms, i.e., four-point trans-form, helps with semantic
learning of occluded faces as our full model is significantly better than the
one without four-point transform which further proves the generality of our
method.

###  4.4 Qualitative Results

Figure 6: Qualitative comparisons in cropped images between previous methods
RoI Tanh-warping (Lin et al., 2019), RoI Tanh-polar (Lin et al., 2021) and
ours on the SFPD. The proposed four-point transform addressed the typical
occlusion, including sunglasses, eyeglasses, hand and face mask. Figure 7:
Qualitative comparisons in wild scenes between previous methods RoI Tanh-
warping (Lin et al., 2019), RoI Tanh-polar (Lin et al., 2021) and ours. Figure
8: Qualitative results on other challenging images with complicate occlusions.

Finally, we compare the proposed FTNet with two most related competitors
(i.e., RoI Tanh-warping (Lin et al., 2019) and RoI Tanh-polar (Lin et al.,
2021)) on SFPD. Some qualitative comparison results on the cropped images are
depicted in Fig.6. We can see that our approaches output more precise parsing
results than other competitors under the occlusions, including sunglasses,
eyeglasses, hand and face mask. Neither RoI Tanh-warping (Lin et al., 2019)
nor RoI Tanh-polar (Lin et al., 2021) can precisely parse face parts under all
occlusion conditions. The performance of these two methods can be accepted
under sunglasses and eyeglasses (2nd column), because this kind of occlusion
differ greatly from the skin color and can be easily distinguished in most
cases. However, when the occlusion prone to skin color confusion occurs, such
as face masks and hands, these methods are prone to parsing errors and far
inferior to our proposed method. In addition, with its better leveraging of
illumination theory, our four-point transform eliminates the interference from
the occlusion and gets more robust results.

The qualitative comparisons on the complete scene image are shown in Fig. 7
and Fig. 8 gives some other challenging case, where our FTNet still correctly
recognizes the confusing parts of the person under occlusion. The qualitative
results show that the significant improvement brought by our FTNet in parsing
occluded faces.

###  4.5 Failure Case Analysis

Figure 9: Visualizations of typical failure cases on SFPD test set, including
(a) low-quality images, (b) low-light images.

To give a deeper insight into our method, we present two representative
failure cases in Fig. 9. As seen our proposed model face difficulties with low
quality images or low-light occluded scenes. In the future, we will therefore
focus on addressing these issues.

##  5 Conclusion

In this paper, we propose a novel occlusion-aware network combined with
homogeneous tanh-transforms for occluded face parsing, namely, four-point
trans-form neural network FTNet. We use four-point transform to warp the image
to four Tanh-polar coordinate systems, which provides a high anti-occultation
ability. Our occlusion-aware network utilizes receptive fields of different
shapes for warped images, while compressing more information of surrounding
context. Moreover, an occlusion-aware loss also enhances the robustness of the
model to occluded face parsing. Ablation studies show the effectiveness of
four-point transform and our proposed method. The superior performance show
that we have solved the dilemma of face parsing on occluded faces in most wild
scenes and our method is highly capable on face parsing tasks with face
coverings. In the future work, we would like to focus on the addressing
difficult case, including low-quality and low-light images, as well as apply
our method into other occluded image segmentation tasks, such as human parsing
and object segmentation.

## References

  * Albalas et al. (2022) Albalas, F., Alzu’bi, A., Alguzo, A., Al-Hadhrami, T., Othman, A., 2022.  Learning discriminant spatial features with deep graph-based convolutions for occluded face detection.  IEEE Access 10, 35162–35171.
  * Bitouk et al. (2008) Bitouk, D., Kumar, N., Dhillon, S., Belhumeur, P., Nayar, S.K., 2008.  Face swapping: automatically replacing faces in photographs, in: ACM SIGGRAPH 2008 papers, pp. 1–8.
  * Chen et al. (2017) Chen, L.C., Papandreou, G., Kokkinos, I., Murphy, K., Yuille, A.L., 2017.  Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs.  IEEE transactions on pattern analysis and machine intelligence 40, 834–848.
  * Chen et al. (2015) Chen, Y.T., Liu, X., Yang, M.H., 2015.  Multi-instance object segmentation with occlusion handling, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3470–3478.
  * De Clerck and Proffit (2015) De Clerck, H.J., Proffit, W.R., 2015\.  Growth modification of the face: A current perspective with emphasis on class iii treatment.  American Journal of Orthodontics and Dentofacial Orthopedics 148, 37–46.
  * Fu et al. (2018) Fu, H., Cheng, J., Xu, Y., Wong, D.W.K., Liu, J., Cao, X., 2018.  Joint optic disc and cup segmentation based on multi-label deep network and polar transformation.  IEEE transactions on medical imaging 37, 1597–1605.
  * Gao et al. (2011) Gao, T., Packer, B., Koller, D., 2011.  A segmentation-aware object detection model with occlusion handling, in: CVPR 2011, IEEE Computer Society. pp. 1361–1368.
  * Ghiasi et al. (2014) Ghiasi, G., Yang, Y., Ramanan, D., Fowlkes, C.C., 2014\.  Parsing occluded people, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2401–2408.
  * Han and Yoon (2023) Han, S., Yoon, H., 2023.  Scanet: Real-time face parsing using spatial and channel attention, in: 2023 20th International Conference on Ubiquitous Robots (UR), IEEE. pp. 13–18.
  * Hariharan et al. (2014) Hariharan, B., Arbeláez, P., Girshick, R., Malik, J., 2014\.  Simultaneous detection and segmentation, in: European conference on computer vision, Springer. pp. 297–312.
  * He et al. (2016) He, K., Zhang, X., Ren, S., Sun, J., 2016.  Deep residual learning for image recognition, in: Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770–778.
  * Huang et al. (2021) Huang, P., Han, J., Zhang, D., Xu, M., 2021.  Clrnet: Component-level refinement network for deep face parsing.  IEEE Transactions on Neural Networks and Learning Systems .
  * Jaderberg et al. (2015) Jaderberg, M., Simonyan, K., Zisserman, A., et al., 2015.  Spatial transformer networks.  Advances in neural information processing systems 28\.
  * Jiang and Mei (2019) Jiang, R., Mei, S., 2019.  Polar coordinate convolutional neural network: From rotation-invariance to translation-invariance, in: 2019 IEEE International Conference on Image Processing (ICIP), IEEE. pp. 355–359.
  * Ke et al. (2021) Ke, L., Tai, Y.W., Tang, C.K., 2021.  Deep occlusion-aware instance segmentation with overlapping bilayers, in: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 4019–4028.
  * Kim et al. (2018) Kim, B.S., Sun, J.Y., Kim, S.W., Kang, M.C., Ko, S.J., 2018\.  Cnn-based ugs method using cartesian-to-polar coordinate transformation.  Electronics Letters 54, 1321–1322.
  * Kim et al. (2022) Kim, H., Kim, H., Cho, S., Hwang, E., 2022.  An end-to-end face parsing model using channel and spatial attentions.  Measurement 191, 110807\.
  * Le et al. (2012) Le, V., Brandt, J., Lin, Z., Bourdev, L., Huang, T.S., 2012\.  Interactive facial feature localization, in: European conference on computer vision, Springer. pp. 679–692.
  * LeCun et al. (1989) LeCun, Y., Boser, B., Denker, J.S., Henderson, D., Howard, R.E., Hubbard, W., Jackel, L.D., 1989.  Backpropagation applied to handwritten zip code recognition.  Neural computation 1, 541–551.
  * Lee et al. (2020) Lee, C.H., Liu, Z., Wu, L., Luo, P., 2020.  Maskgan: Towards diverse and interactive facial image manipulation, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5549–5558.
  * Li et al. (2023) Li, L., Zhang, T., Kang, Z., Jiang, X., 2023.  Mask-fpan: semi-supervised face parsing in the wild with de-occlusion and uv gan.  Computers & Graphics 116, 185–193.
  * Lin et al. (2019) Lin, J., Yang, H., Chen, D., Zeng, M., Wen, F., Yuan, L., 2019.  Face parsing with roi tanh-warping, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5654–5663.
  * Lin et al. (2021) Lin, Y., Shen, J., Wang, Y., Pantic, M., 2021.  Roi tanh-polar transformer network for face parsing in the wild.  Image and Vision Computing 112, 104190\.
  * Liu et al. (2017) Liu, S., Shi, J., Liang, J., Yang, M.H., 2017.  Face parsing via recurrent propagation.  arXiv preprint arXiv:1708.01936 .
  * Liu et al. (2015a) Liu, S., Yang, J., Huang, C., Yang, M.H., 2015a.  Multi-objective convolutional learning for face labeling, in: Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3451–3459.
  * Liu et al. (2015b) Liu, Z., Luo, P., Wang, X., Tang, X., 2015b.  Deep learning face attributes in the wild, in: Proceedings of the IEEE international conference on computer vision, pp. 3730–3738.
  * Long et al. (2015) Long, J., Shelhamer, E., Darrell, T., 2015.  Fully convolutional networks for semantic segmentation, in: Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3431–3440.
  * Luo et al. (2012) Luo, P., Wang, X., Tang, X., 2012.  Hierarchical face parsing via deep learning, in: 2012 IEEE Conference on Computer Vision and Pattern Recognition, IEEE. pp. 2480–2487.
  * maadaa.ai (2021) maadaa.ai, 2021.  Short-video face parsing challenge cvpr 2021.  https://maadaa.ai/cvpr2021-short-video-face/-parsing-challenge/.  Accessed June, 2021.
  * Masi et al. (2020) Masi, I., Mathai, J., AbdAlmageed, W., 2020.  Towards learning structure via consensus for face segmentation and parsing, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5508–5518.
  * Noh et al. (2015) Noh, H., Hong, S., Han, B., 2015.  Learning deconvolution network for semantic segmentation, in: Proceedings of the IEEE international conference on computer vision, pp. 1520–1528.
  * Salehinejad et al. (2018) Salehinejad, H., Valaee, S., Dowdell, T., Barfett, J., 2018\.  Image augmentation using radial transform for training deep neural networks, in: 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), IEEE. pp. 3016–3020.
  * Selfa and Qazi (2005) Selfa, T., Qazi, J., 2005.  Place, taste, or face-to-face? understanding producer–consumer networks in “local” food systems in washington state.  Agriculture and human values 22, 451–464.
  * Smith et al. (2013) Smith, B.M., Zhang, L., Brandt, J., Lin, Z., Yang, J., 2013.  Exemplar-based face parsing, in: Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3484–3491.
  * Tai et al. (2019) Tai, K.S., Bailis, P., Valiant, G., 2019.  Equivariant transformer networks, in: International Conference on Machine Learning, PMLR. pp. 6086–6095.
  * Wang et al. (2020) Wang, C., Chudzicka-Czupała, A., Grabowski, D., Pan, R., Adamus, K., Wan, X., Hetnał, M., Tan, Y., Olszewska-Guizzo, A., Xu, L., et al., 2020\.  The association between physical and mental health and face mask use during the covid-19 pandemic: a comparison of two countries with different views and practices.  Frontiers in psychiatry 11, 569981\.
  * Warrell and Prince (2009) Warrell, J., Prince, S.J., 2009\.  Labelfaces: Parsing facial features by multiclass labeling with an epitome prior, in: 2009 16th IEEE international conference on image processing (ICIP), IEEE. pp. 2481–2484.
  * Yin et al. (2021) Yin, Z., Yiu, V., Hu, X., Tang, L., 2021.  End-to-end face parsing via interlinked convolutional neural networks.  Cognitive Neurodynamics 15, 169–179.
  * Zhao et al. (2003) Zhao, W., Chellappa, R., Phillips, P.J., Rosenfeld, A., 2003\.  Face recognition: A literature survey.  ACM computing surveys (CSUR) 35, 399–458.
  * Zheng et al. (2020) Zheng, W., Gou, C., Wang, F.Y., 2020.  A novel approach inspired by optic nerve characteristics for few-shot occluded face recognition.  Neurocomputing 376, 25–41.
  * Zhou et al. (2017) Zhou, L., Liu, Z., He, X., 2017.  Face parsing via a fully-convolutional continuous crf neural network.  arXiv preprint arXiv:1708.03736 .
  * Zhou et al. (2015) Zhou, Y., Hu, X., Zhang, B., 2015.  Interlinked convolutional neural networks for face parsing, in: International symposium on neural networks, Springer. pp. 222–231.