# jointVIP: Prioritizing variables in observational study design with joint
variable importance plot in R

Lauren D. Liao
Division of Biostatistics
University of California, Berkeley
<EMAIL_ADDRESS>
Samuel D. Pimentel
Department of Statistics
University of California, Berkeley
<EMAIL_ADDRESS>

###### Abstract

Credible causal effect estimation requires treated subjects and controls to be
otherwise similar. In observational settings, such as analysis of electronic
health records, this is not guaranteed. Investigators must balance background
variables so they are similar in treated and control groups. Common approaches
include matching (grouping individuals into small homogeneous sets) or
weighting (upweighting or downweighting individuals) to create similar
profiles. However, creating identical distributions may be impossible if many
variables are measured, and not all variables are of equal importance to the
outcome. The joint variable importance plot (jointVIP) package to guides
decisions about which variables to prioritize for adjustment by quantifying
and visualizing each variable’s relationship to both treatment and outcome.

_K_ eywords R $\cdot$ observational study $\cdot$ study design $\cdot$
visualization $\cdot$ causal inference

##  1 Statement of need

Consider an observational study to measure the effect of a binary treatment
variable (treated/control) on an outcome, in which additional covariates
(background variables) are measured. A covariate may be associated with
outcomes, and it may also differ in distribution between treated and controls;
if the covariate is associated with treatment and outcome, the covariate in
question is a confounder. Ignored confounders introduce bias into treatment
effect estimates. For instance, when testing a blood pressure drug, if older
patients both take the drug more and have worse initial blood pressure, a
simple difference in mean blood pressure between treated and control subjects
will understate the drug’s benefits. Confounding can be addressed by matching,
under which blood pressure is compared only within pairs of patients with
similar ages, or by weighting, in which older control subjects receive larger
weights than younger control subjects when averaging blood pressure. When many
potential confounders are measured, however, neither matching nor weighting
can perfectly adjust for all differences, and researchers must select which
variables to focus on balancing.

Current practice for selecting variables for adjustment focuses primarily on
understanding the treatment relationship, via tools such as balance tables and
the Love plot [1, 2, 3, 4, 5]. A key metric is the standardized mean
difference (SMD), or the difference in treated and control means over a
covariate measure in standard deviations. Researchers commonly try to adjust
so that all SMD values are moderately small, or focus on adjustments for
variables with the largest initial SMD. However, these approaches neglect
important information about the relationship of each covariate with the
outcome variable, which substantially influences the degree of bias incurred
by ignoring it.

To improve observational study design, we propose the joint variable
importance plot (jointVIP) [6], implemented in the jointVIP package. The
jointVIP represents both treatment and outcome relationships for each variable
in a single image: each variable’s SMD is plotted against an outcome
correlation measure (computed in a pilot control sample to avoid bias from
multiple use of outcome data). Bias curves based on the omitted variable bias
(OVB) framework also are plotted to improve variable comparison. The jointVIP
provides valuable insight into variable importance and can be used to specify
key parameters in existing matching and weighting methods.

##  2 Development

The jointVIP package was created in the R programming language [7]. The
package uses the S3 object system and leverages system generic functions,
print(), summary(), and plot(). Plotting the jointVIP object outputs a plot of
the ggplot2 class. An interactive R Shiny application, available online at
https://ldliao.shinyapps.io/jointVIP/, showcases the package.

##  3 Usage

The jointVIP package is available from the Comprehensive R Archive Network
CRAN and GitHub. To create an object of the jointVIP class, the user needs to
supply two datasets and specify the treatment, outcome, and background
variable names. Two processed datasets, “pilot” and “analysis” samples, are in
the form of data.frames. The analysis sample contains both treated and control
groups. The pilot sample contains only control individuals, and they are
excluded from the subsequent analysis stage. The treatment variable must be
binary: 0 specified for the control group and 1 specified for the treated
group. Background variables are measured before both treatment and outcome.
The outcome of interest can be either binary or continuous.

⬇

# installation using CRAN:

# install.packages("jointVIP")

# installation of development version using GitHub:

# remotes::install_github(’ldliao/jointVIP’)

library(jointVIP)

We demonstrate the utility of this package to investigate the effect of a job
training program on earnings [8, 9, 10]. The treatment is whether the
individual is selected for the job training program. The outcomeis earnings in
1978. Covariates are age, education, race/ethnicity, and previous earnings in
1974 and 1975. After preprocessing both dataset and log-transforming the
earnings, we use the create_jointVIP() function to create a jointVIP object
stored as _new_jointVIP_.

⬇

# first define and get pilot_df and analysis_df

# they should both be data.frame objects

treatment <- "treat"

outcome <- "log_re78"

covariates <- c("age", "educ", "black",

"hisp", "marr", "nodegree",

"log_re74", "log_re75")

new_jointVIP = create_jointVIP(treatment = treatment,

outcome = outcome,

covariates = covariates,

pilot_df = pilot_df,

analysis_df = analysis_df)

The plot() function displays a jointVIP (Figure 1). The x-axis describes
treatment imbalance in SMD (computed with a denominator based on the pilot
sample as in [6]). The y-axis describes outcome correlations in the pilot
sample. The summary() function outputs the maximum absolute bias and the
number of variables required for adjustment above the absolute bias tolerance,
_bias_tol_. The _bias_tol_ parameter can be used in the print() function to
see which variables are above the desired tolerance. Additional tuning
parameters can be specified in these functions, for details and examples, see
the additional options vignette.

⬇

plot(new_jointVIP)

Figure 1: Joint variable importance plot example

⬇

summary(new_jointVIP,

smd = "OVB-based",

use_abs = TRUE,

bias_tol = 0.01)

# > Max absolute bias is 0.113

# > 2 variables are above the desired 0.01 absolute bias tolerance

# > 8 variables can be plotted

print(new_jointVIP,

smd = "OVB-based",

use_abs = TRUE,

bias_tol = 0.01)

# > bias

# > log_re75 0.113

# > log_re74 0.045

To interpret our working example, the most important variables are the
previous earning variables in 1975 and 1974, log_re75 and log_re74 variables,
respectively. Using the traditional visualization method, the Love plot, would
only identify variables based on the SMD. The same information can be
interpreted from the x-axis of the jointVIP. For example, the Love plot would
indicate variables, nodegree and hisp, to be more important for adjustment
than log_re74. In comparison, those variables, nodegree and hisp, show low
bias using the jointVIP.

After adjusting for variables, for example, using optimal matching [11, 5] to
select pairs for analysis, a post-adjustment dataset, _post_analysis_df_ , can
be used to create a post adjustment object of class post_jointVIP. The
create_post_jointVIP() function can be used to visualize and summarize the
post adjustment results, as seen in Figure 2. The functions: summary(),
print(), and plot() all can take in the post_jointVIP object and provide
comparison between original and post adjusted jointVIPs.

⬇

post_optmatch_jointVIP <- create_post_jointVIP(new_jointVIP,

post_analysis_df = optmatch_df)

plot(post_optmatch_jointVIP,

plot_title = "Post-match␣jointVIP␣using␣optimal␣matching")

Figure 2: Post match example showing balanced sample based on new mean
differences.

⬇

summary(post_optmatch_jointVIP)

# > Max absolute bias is 0.113

# > 2 variables are above the desired 0.01 absolute bias tolerance

# > 8 variables can be plotted

# >

# > Max absolute post-bias is 0.005

# > Post-measure has 0 variable(s) above the desired 0.005 absolute bias
tolerance

print(post_optmatch_jointVIP)

# > bias post_bias

# > log_re75 0.113 0.005

# > log_re74 0.045 0.003

##  4 Discussion

We have developed user-friendly software to prioritize variables for
adjustment in observational studies. This package can help identify important
variables related to both treatment and outcome. One limitation is that each
background variable is individually evaluated for bias. Thus, conditional
relationships, interactions, or higher moments of variables need to be
carefully considered or preprocessed by the user.

## Acknowledgments

SDP is supported by Hellman Family Fellowship and by the National Science
Foundation (grant 2142146). LDL is supported by National Science Foundation
Graduate Research Fellowship (grant DGE 2146752).

## References

  * [1] Ali Ahmed, Ahsan Husain, Thomas E Love, Giovanni Gambassi, Louis J Dell’Italia, Gary S Francis, Mihai Gheorghiade, Richard M Allman, Sreelatha Meleth, and Robert C Bourge.  Heart failure, chronic diuretic use, and increase in mortality and hospitalization: an observational study using propensity score methods.  European heart journal, 27(12):1431–1439, 2006.
  * [2] Noah Greifer and Elizabeth A Stuart.  Choosing the estimand when matching or weighting in observational studies.  arXiv preprint arXiv:2106.10577, 2021.
  * [3] Ben B Hansen and Jake Bowers.  Covariate balance in simple, stratified and clustered comparative studies.  Statistical Science, pages 219–236, 2008.
  * [4] Paul R Rosenbaum and Donald B Rubin.  Constructing a control group using multivariate matched sampling methods that incorporate the propensity score.  The American Statistician, 39(1):33–38, 1985.
  * [5] Elizabeth A Stuart, Gary King, Kosuke Imai, and Daniel Ho.  Matchit: nonparametric preprocessing for parametric causal inference.  Journal of statistical software, 2011.
  * [6] Lauren D Liao, Yeyi Zhu, Amanda L Ngo, Rana F Chehab, and Samuel D Pimentel.  Using joint variable importance plots to prioritize variables in assessing the impact of glyburide on adverse birth outcomes.  arXiv preprint arXiv:2301.09754, 2023.
  * [7] R Core Team.  R: A Language and Environment for Statistical Computing.  R Foundation for Statistical Computing, Vienna, Austria, 2020.
  * [8] Nick Huntington-Klein and Malcolm Barrett.  causaldata: Example Data Sets for Causal Inference Textbooks, 2021\.  R package version 0.1.3.
  * [9] Rajeev H Dehejia and Sadek Wahba.  Causal effects in nonexperimental studies: Reevaluating the evaluation of training programs.  Journal of the American statistical Association, 94(448):1053–1062, 1999.
  * [10] Robert J LaLonde.  Evaluating the econometric evaluations of training programs with experimental data.  The American economic review, pages 604–620, 1986.
  * [11] Ben B. Hansen and Stephanie Olsen Klopfer.  Optimal full matching and related designs via network flows.  Journal of Computational and Graphical Statistics, 15(3):609–627, 2006.


# Vacuum stability of the scalar potential in the compact 341 model

Meriem Djouala<EMAIL_ADDRESS>Noureddine Mebarki  Laboratoire de
physique mathématique et subatomique, university of Frères Mentouri-
Constantine 1-Algeria.

###### Abstract

By applying the concepts of copositivity and using the gauge orbit spaces on
the scalar potential, we derive analytic necessary and sufficient conditions
which guarantee the boundedness of the scalar potential in all the directions
in the field space at the tree level in the context of the compact 341 model.

###### keywords:

Beyond the Standard Model , the comapct 341 model , vacuum stability

contents=  Submitted to the Proceedings of the African Conference on
Fundamental and Applied Physics Second Edition, ACP2021, March 7–11, 2022 —
Virtual Event

, scale=1,placement=top,opacity=1,color=black,position=3.25in,1.2in

##  1 Introduction

The Standard Model (SM) failed to address the answer of many outstanding
questions such as neutrinos oscillations, dark matter,etc.; thus, going beyond
it becomes mandatory. The SM can be extended in several different ways, by
adding new fermions fields [1, 2], augmenting the scalar sector to more than
Higgs field [3, 4] and enlarging the local gauge group [5]. Among the theories
beyond the Standard Model, we are interested in the second extension of the
electroweak gauge group of SM, the model based on the gauge group
$SU(3)_{C}\otimes SU(4)_{L}\otimes U(1)_{X}$ (341 model just for short) where
$C$ refers the color charge, $L$ is the left handed chirality while $X$ is a
new quantum number associated to the group $U(1)_{X}$. This model predicts the
existence of new particles included the exotic leptons and quarks, new gauge
and scalar bosons.
To specify the allowed regions of the scalar parameters one needs to derive
the theoretical constraints besides the experiment limits. The vacuum
stability condition determines the first set of the theoretical constraints;
in our work, we derive the necessary conditions that ensure the boundedness of
the scalar potential from below in any direction in the field space in the
compact 341 model [6] by using the copositivity criteria and the gauge orbit
space.
This paper is organized as follows. In Section 2, we present the fermion
content of the compact 341 model. In Section 3, we discuss the vacuum
stability conditions using the copositivity criteria and the gauge orbit
space. Finally, in Section, 4 we draw our conclusions.

##  2 The compact 341 model

In the compact 341 model, the electric charges of new particles are defined
through the following electric charge operator

$Q=\frac{1}{2}\bigg{(}\lambda_{3}-\frac{1}{\sqrt{3}}\lambda_{8}-\frac{4}{\sqrt{6}}\lambda_{15}\bigg{)}+X,$
(1)

where $\lambda_{m=3,8,15}$ are the diagonal matrices of the group $SU(4)$. The
fermion content in this model is

$\psi_{aL}\equiv\left(\begin{array}[]{ccc}\nu_{a}\\\ l_{a}\\\ \nu_{a}^{c}\\\
l_{a}^{c}\\\
\end{array}\right)_{(1,4,0)},Q_{1L}\equiv\left(\begin{array}[]{ccc}u_{1}\\\
d_{1}\\\ U_{1}\\\ J_{1}\\\
\end{array}\right)_{(3,4,\frac{2}{3})},Q_{iL}\equiv\left(\begin{array}[]{ccc}d_{i}\\\
u_{i}\\\ D_{i}\\\ J_{i}\\\ \end{array}\right)_{(1,\bar{4},-\frac{1}{3})}$ (2)

where $a\equiv e,\mu,\tau$. $Q_{1L}$ is the first left handed quark generation
and $Q_{iL(i=2,3)}$ represent the second and third families of left handed
quarks.
To cancel the gauge anomalies in the compact 341 model, the quark generations
should belong to different representations, one of the quark families $Q_{1L}$
with the three left handed lepton generations $\psi_{aL}$ arrange in the
fundamental representation whereas the other two families $Q_{iL(i=2,3)}$ are
in the conjugate fundamental representation (or vice versa).
Notice that the right-handed quarks transform as singlets under
$SU(4)_{L}\otimes U(1)_{X}$, while, the right handed leptons belong to the
left handed quadruplets.
The most general scalar potential in the compact 341 model is given by

$\displaystyle V(\eta,\rho,\chi)$ $\displaystyle=$
$\displaystyle\mu_{\eta}^{2}\eta^{{\dagger}}\eta+\mu_{\rho}^{2}\rho^{{\dagger}}\rho+\mu_{\chi}^{2}\chi^{{\dagger}}\chi+\lambda_{1}(\eta^{{\dagger}}\eta)^{2}+\lambda_{2}(\rho^{{\dagger}}\rho)^{2}+\lambda_{3}(\chi^{{\dagger}}\chi)^{2}$
(3) $\displaystyle+$
$\displaystyle\lambda_{4}(\eta^{{\dagger}}\eta)(\rho^{{\dagger}}\rho)+\lambda_{5}(\eta^{{\dagger}}\eta)(\chi^{{\dagger}}\chi)+\lambda_{6}(\rho^{{\dagger}}\rho)(\chi^{{\dagger}}\chi)+\lambda_{7}(\rho^{{\dagger}}\eta)(\eta^{{\dagger}}\rho)$
$\displaystyle+$
$\displaystyle\lambda_{8}(\chi^{{\dagger}}\eta)(\eta^{{\dagger}}\chi)+\lambda_{9}(\rho^{{\dagger}}\chi)(\chi^{{\dagger}}\rho),$

where $\mu^{2}_{\mu,\rho,\chi}$ are the mass dimension parameters,
$\lambda_{j}(j=1...9)$ are dimensionless coupling constants and $\eta$, $\rho$
and $\chi$ are the scalar fields that are given by

$\rho\equiv\left(\begin{array}[]{ccc}\rho^{+}_{1}\\\ \rho^{0}\\\
\rho^{+}_{2}\\\ \rho^{++}\\\
\end{array}\right)_{(1,4,1)},\chi\equiv\left(\begin{array}[]{ccc}\chi^{-}_{1}\\\
\chi^{--}\\\ \chi^{-}_{2}\\\ \chi^{0}\\\
\end{array}\right)_{(3,4,-1)},\eta\equiv\left(\begin{array}[]{ccc}\eta^{0}_{1}\\\
\eta_{1}^{-}\\\ \eta^{0}_{2}\\\ \eta^{+}_{2}\end{array}\right)_{(1,4,0)}$ (4)

where the VEVs structure is represented as

$\langle\rho\rangle=\frac{1}{\sqrt{2}}\left(\begin{array}[]{ccc}0\\\
v_{\rho}\\\ 0\\\ 0\\\
\end{array}\right),\langle\chi\rangle=\frac{1}{\sqrt{2}}\left(\begin{array}[]{ccc}0\\\
0\\\ 0\\\ v_{\chi}\\\
\end{array}\right),\langle\eta\rangle=\frac{1}{\sqrt{2}}\left(\begin{array}[]{ccc}0\\\
0\\\ v_{\eta}\\\ 0\end{array}\right)$ (5)

The VEV $v_{\chi}$ is responsible for the first step in breaking the 341
symmetry to 331, whereas $v_{\eta}$ breaks the 331 symmetry to 321 (the SM),
and $v_{\rho}$ breaks the SM to $U(1)_{EM}$. Thus one can write
$v_{\chi}>v_{\eta}>v_{\rho}$.
The conditions for the minimum of the scalar potential (3) are

$\displaystyle\mu_{\eta}^{2}+\lambda_{1}\upsilon_{\eta}^{2}+\frac{1}{2}\lambda_{4}\upsilon_{\rho}^{2}+\frac{1}{2}\lambda_{5}\upsilon_{\chi}^{2}=0,$
(6)
$\displaystyle\mu_{\rho}^{2}+\lambda_{2}\upsilon_{\rho}^{2}+\frac{1}{2}\lambda_{4}\upsilon_{\eta}^{2}+\frac{1}{2}\lambda_{6}\upsilon_{\chi}^{2}=0,$
(7)
$\displaystyle\mu_{\chi}^{2}+\lambda_{3}\upsilon_{\chi}^{2}+\frac{1}{2}\lambda_{5}\upsilon_{\eta}^{2}+\frac{1}{2}\lambda_{6}\upsilon_{\rho}^{2}=0.$
(8)

##  3 The vacuum stability

To ensure the boundedness of the scalar potential in the compact 341 model
from below in any direction in the field space we derive analytic necessary
and sufficient conditions of the vacuum stability by using the concepts of
copositivity and gauge orbit spaces.
In the limit of large field values, it is enough to work with the quartic
terms. We ignore other terms with dimension $d<4$ since they are negligible in
comparison with the quartic couplings of the scalar potential, thus
$V^{4}(\eta,\rho,\chi)$ takes the following form

$\displaystyle V^{4}(\eta,\rho,\chi)=$
$\displaystyle\lambda_{1}(\eta^{{\dagger}}\eta)^{2}+\lambda_{2}(\rho^{{\dagger}}\rho)^{2}+\lambda_{3}(\chi^{{\dagger}}\chi)^{2}+\lambda_{4}(\eta^{{\dagger}}\eta)(\rho^{{\dagger}}\rho)+\lambda_{5}(\eta^{{\dagger}}\eta)(\chi^{{\dagger}}\chi)$
(9)
$\displaystyle+\lambda_{6}(\rho^{{\dagger}}\rho)(\chi^{{\dagger}}\chi)+\lambda_{7}(\rho^{{\dagger}}\eta)(\eta^{{\dagger}}\rho)+\lambda_{8}(\chi^{{\dagger}}\eta)(\eta^{{\dagger}}\chi)+\lambda_{9}(\rho^{{\dagger}}\chi)(\chi^{{\dagger}}\rho),$

We have three scalar field directions; thus, we define a parametrization as
follows [6]

$\displaystyle r^{2}$ $\displaystyle\equiv$
$\displaystyle\eta^{\dagger}\eta+\rho^{\dagger}\rho+\chi^{\dagger}\chi,$
$\displaystyle\eta^{\dagger}\eta$ $\displaystyle\equiv$ $\displaystyle
r^{2}\cos^{2}\theta\sin^{2}\phi,$ $\displaystyle\rho^{\dagger}\rho$
$\displaystyle\equiv$ $\displaystyle r^{2}\sin^{2}\theta\sin^{2}\phi,$
$\displaystyle\chi^{\dagger}\chi$ $\displaystyle\equiv$ $\displaystyle
r^{2}\cos^{2}\phi,$ $\displaystyle\frac{\eta^{\dagger}\rho}{|\eta||\rho|}$
$\displaystyle\equiv$ $\displaystyle\xi_{1}e^{i\psi_{1}},$
$\displaystyle\frac{\eta^{\dagger}\chi}{|\eta||\chi|}$ $\displaystyle\equiv$
$\displaystyle\xi_{2}e^{i\psi_{2}},$
$\displaystyle\frac{\rho^{\dagger}\chi}{|\rho||\chi|}$ $\displaystyle\equiv$
$\displaystyle\xi_{3}e^{i\psi_{3}},$ (10)

where the radius $r$ scans the domain [0,$\infty$[, $\theta\in$ [0,2$\pi$],
$\phi\in$[0,$\frac{\pi}{2}$] and $\xi_{i=(1,2,3)}\in$ [0,1] [6].
Inserting this parametrization in Eq.(9), we get expressions that have the
following form [6]

$\displaystyle f(\chi)=a\chi^{2}+b(1-\chi)^{2}+c\chi(1-\chi),$ (11)

The conditions for copositivity are given below [6]:

$\displaystyle a>0,~{}~{}b>0~{}~{}c+2\sqrt{ab}>0.$ (12)

Notice that the quartic part of the scalar potential $(\ref{eq:mimzy})$ is
bounded from below only if it satisfies the copositivity conditions (12).
Thus, by applying this criteria to our results, we find [6]:

$\displaystyle\lambda_{1}$ $\displaystyle>$ $\displaystyle
0,~{}~{}\lambda_{2}>0~{}~{}\lambda_{3}>0,$ $\displaystyle\lambda_{4}$
$\displaystyle+$ $\displaystyle 2\sqrt{\lambda_{1}\lambda_{2}}>0.$
$\displaystyle\lambda_{4}$ $\displaystyle+$
$\displaystyle\lambda_{7}+2\sqrt{\lambda_{1}\lambda_{2}}>0.$ (13)

The conditions (13) ensure the stability of the scalar potential in the
compact 341 model, together with the positivity of the Hessian matrix and the
perturbative unitarity conditions we get the set of the theoretical
constraints on the scalar couplings [6].

##  4 Conclusions

throughout this paper, we have studied the vacuum stability constraint in the
compact 341 model which determines the necessary and sufficient conditions for
the boundedness of the scalar potential from below in any direction in the
field space by using the copositivity criteria and gauge orbit space. Together
with the other theoretical constraints we can determine the allowed and
compatible regions of the scalar parameters space [6].

## Acknowledgments

We would like to thank Garv Chauhan for fruitful discussions. We are very
grateful to the Algerian ministry of higher education and scientific research
and DGRSDT for the financial support.

## References

## References

  * [1] M. Djouala ,N. Mebarki and H. Aissaoui, A new anomaly-free flipped 341 model, Int.J.Mod.Phys.A Vol. 36 (2021) 2150101. https://doi.org/10.1142/S0217751X21501013.
  * [2] M.Djouala and N.Mebarki, A flipped $SU(3)_{C}\otimes SU(4)_{L}\otimes U(1)_{X}$ model, IOPscience vol. 1766, (2021) 012016. https://doi.org/10.1088/1742-6596/1766/1/012016.
  * [3] G. Chauhan, Vacuum Stability and Symmetry Breaking in Left-Right Symmetric Model, JHEP vol. 12 (2019) 137.  https://doi.org/10.48550/arXiv.1907.07153.
  * [4] J. Chakrabortty, P. Konar and T. Mondal, Copositive criteria and boundedness of the scalar potential, Phys. Rev. D vol. 89 (2014) 095008.  https://doi.org/10.48550/arXiv.1311.5666.
  * [5] A. G. Dias, P. R. D. Pinheiro, C. A. de S. Pires and P. S. Rodrigues da Silva, A compact 341 model at TeV scale, Ann. Phys. vol. 349 (2014) 232.  https://doi.org/10.48550/arXiv.1309.6644.
  * [6] M.Djouala and N.Mebarki, Theoretical constraints on scalar parameters in the compact 341 model, Int.J.Mod.Phys.A vol.36 (2021) 2150093.  https://doi.org/10.48550/arXiv.2002.08758.


(eccv) Package eccv Warning: Package ‘hyperref’ is loaded with option
‘pagebackref’, which is *not* recommended for camera-ready version

11institutetext: SberDevices

# EasyPortrait – Face Parsing and Portrait Segmentation Dataset

Karina Kvanchiani  Elizaveta Petrova  0009-0005-8454-7701  Karen Efremyan
0009-0007-1606-6716 Alexander Sautin  Alexander Kapitanov  0009-0009-6359-2019

###### Abstract

Recently, video conferencing apps have become functional by accomplishing such
computer vision-based features as real-time background removal and face
beautification. Limited variability in existing portrait segmentation and face
parsing datasets, including head poses, ethnicity, scenes, and occlusions
specific to video conferencing, motivated us to create a new dataset,
EasyPortrait, for these tasks simultaneously. It contains 40,000 primarily
indoor photos repeating video meeting scenarios with 13,705 unique users and
fine-grained segmentation masks separated into 9 classes. Inappropriate
annotation masks from other datasets caused a revision of annotator
guidelines, resulting in EasyPortrait’s ability to process cases, such as
teeth whitening and skin smoothing. The pipeline for data mining and high-
quality mask annotation via crowdsourcing is also proposed in this paper. In
the ablation study experiments, we proved the importance of data quantity and
diversity in head poses in our dataset for the effective learning of the
model. The cross-dataset evaluation experiments confirmed the best domain
generalization ability among portrait segmentation datasets. Moreover, we
demonstrate the simplicity of training segmentation models on EasyPortrait
without extra training tricks. The proposed dataset and trained models are
publicly available111https://github.com/hukenovs/easyportrait.

###### Keywords:

Face Parsing Portrait Segmentation Dataset Creation

##  1 Introduction

Figure 1: The face parsing and portrait segmentation annotation examples from
the EasyPortrait dataset.

Video conferencing apps have quickly gained popularity in corporations for
online meetings [28] and in our habitual life to keep in touch with relatives
from a distance [4]. As a result, these apps have been enhanced with various
beneficial features, including face beautification, background blur, and noise
reduction [2]. Such extensions can improve user experience, allowing them to
change the background or smooth their skin [5].

Our work primarily aimed to integrate the described features into the video
conferencing app. The app should ensure a real-time CPU-based experience on
the user’s device and produce a highly accurate response. Besides, our system
needs to be robust to the amount of context in images, backgrounds, persons in
the frame, their poses, attributes (e.g. race and age), and accessories
(headphones, hats, etc.). Finally, it is preferable to improve the users’
experience with such functions as teeth whitening and accurate background
changing in the case of transparent glasses lenses and uneven hair.

All these requirements impose restrictions on the solution and training
dataset. We decided to exploit deep learning models solving portrait
segmentation and face parsing tasks for our purpose. The system must function
in real-time without any delays and produce fine-grained segmentation masks.
The suitable data is required to be 1) heterogeneous in subjects, their head
turns, subject-to-camera distances, scenes, and specific for videoconferencing
domain subjects’ accessories like eyeglasses and headphones; 2) annotated with
main face parsing classes (“skin”, “brows”, “eyes”, and “lips”) and an extra
class “teeth”; 3) accurately annotated for both tasks simultaneously. The last
can allow training only one model for all possible use cases and save limited
resources for model inference.

Existing datasets are unsuitable for our production purposes due to the
limitations described in Sec. 2. It motivated us to create the EasyPortrait
dataset – our first contribution. The second is a pipeline for gathering and
labeling images with fine-grained segmentation masks utilizing two
crowdsourcing platforms, Yandex.Toloka222https://toloka.yandex.ru and ABC
Elementary333https://elementary.activebc.ru. It allowed us to collect 40,000
pairs of images and segmentation masks from 13,705 individuals in domain-
suitable scenes with different head poses and various specific
videoconferencing app accessories. We considered the importance of ethnic
diversity to solve problems based on persons and their facial attributes and
collected images from users of various countries and continents. Images were
annotated manually by 9 classes according to specially designed rules, which
allowed us to cover all described cases.

We checked that such data characteristics as the number of samples and their
diversity in head poses have a positive impact on the model’s robustness and
effectiveness (Sec. 4). The generalization ability of the training data was
also assessed through cross-dataset evaluation experiments (Sec. 5).

##  2 Related Work

In this section, we discuss existing portrait segmentation and face parsing
datasets separately. We will consider them from two points of view: 1) the
applicability to the videoconferencing domain and 2) data creation techniques
for each segmentation task and their consequences.

###  2.1 Portrait Segmentation and Matting Datasets

The portrait segmentation task involves labeling every pixel in an image as
either “person” or “background”. Since matting annotations can be reduced to
binary ones, image matting datasets will also be considered. As
videoconferencing apps always take portraits for input and not a person
entirely, only datasets with waist-deep people are reviewed. Therefore, other
popular image person segmentation and image matting datasets, e.g., P3M-10k
[14], PPR10K [15], PhotoMatte13K/85 [16], Persons Labeled [3], are not
described in this paper. We selected EG1800 [29], AiSeg [1], FVS [11], Face
Synthetics [31], and HumanSeg14K [7] as the most widespread and predominantly
containing photos of people from the waist up. Tab. 1 provides the numerical
analysis of reviewed datasets.

Table 1: Comparison of portrait segmentation and face parsing datasets.
Because of the specifics of the tasks, we indicated the task name for the
first ones and the number of classes for the second. Several datasets include
images of different sizes, so the standard label was provided in the
resolution column. 96% of images in the EasyPortrait are FullHD; see more
information in Sec. 3.2. We also included notes about the annotation method,
which can be important regarding label quality. For more transparency, 20
classes in the FaceOcc dataset contain 19 classes from CelebAMask-HQ.

Dataset | Samples | Task / Classes | Resolution | Annotation Method
---|---|---|---|---
EG1800, 2016 [29] | 1,800 | segmentation | 600 × 800 | Photoshop
FVS, 2018 [11] | 3,600 | segmentation | 640 × 360 | chroma-key
AISeg, 2018 [1] | 34,427 | matting | 600 × 800 | automatically
PP-HumanSeg14K, 2021 [7] | 14,117 | segmentation | 1280 × 720 | manually
The Face Synthetics, 2021 [31] | 100,000 | segmentation | 512 × 512 | automatically
Helen, 2012 [12] | 2,330 | 11 | 400 × 400 | automatically
LFW-PL, 2013 [10] | 2,927 | 3 | 250 × 250 | automatically & refined
CelebAMask-HQ, 2019 [13] | 30,000 | 19 | 512 × 512 | manually & checked & refined
LaPa, 2020 [19] | 22,176 | 11 | LR | automatically & refined
iBugMask, 2021 [18] | 22,866 | 11 | HR | manually
The Face Synthetics, 2021 [31] | 100,000 | 19 | 512 × 512 | automatically
FaceOcc, 2022 [33] | 30,000 | 20 | 512 × 512 | manually
EasyPortrait, 2023 | 40,000 | 9 | FullHD | manually & checked

Content. Chosen datasets can be divided into three groups in terms of image
source: 1) downloaded from websites, 2) collected manually, and 3) generated.
The last two allow the data authors to directly determine content on their
own. Images of EG1800 and AiSeg were collected from services like Flickr,
which made their scenes multi-domain. The Face Synthetics dataset was
completely generated, entailing primarily blurred backgrounds, and thus far
from in the wild. The manually collected FVS [11] and PP-HumanSeg14 [7]
portrait segmentation datasets are single-domain with a bias towards
videoconferencing. The FVS dataset provides composite images from 10
conference-style green-screen videos and virtual backgrounds. As a result, FVS
samples suffer from the remaining green screen around a person in the frame.
The PP-HumanSeg14 dataset includes 23 different conference backgrounds and
samples of participants performing actions such as waving hands and drinking
water. The provided samples contain one or more labeled people with faces
blurred for privacy.

Annotation. Since segmentation mask annotation is one of the most challenging
problems in the computer vision field, data authors prefer automatic methods.
All reviewed datasets except PP-HumanSeg14K were annotated automatically or
using Photoshop (see Fig. 5 in the supplementary material for visual
examples). Such methods frequently produce coarse masks that prevent accurate
high-frequency parts segmentation (e.g., hair) – one of the main hardships of
background removal in video conferencing.

###  2.2 Face Parsing Datasets

The face parsing task aims to assign pixel-level semantic labels for facial
images. Generally, face parsing refers to classifying image pixels, such as
brows, eyes, nose, lips, mouth, and skin. We considered several widespread
face parsing datasets for our purposes (Tab. 1 and Fig. 6 in the supplementary
material).

Content. The main limitation of existing face parsing datasets’ content is low
diversity in head poses and the absence of specific occlusions. Such
limitation is obtained by utilizing other datasets or websites like Flickr, as
Helen’s [12] authors performed. They searched for “portrait” in various
languages to avoid cultural bias, and manually filtered out low-quality and
false positive images. The CelebAMask-HQ dataset [13] mainly contains front-
facing images of celebrities from Celeba [20] with centered heads. Besides,
the faces are usually occupy a significant part of the image, thus background
information is mainly discarded. The LaPa dataset [19] was designed based on
images from the 300W-LP [35] and Megaface [30] datasets. Received faces were
aligned and mostly cropped with limited background information preserved. This
image collection method was also utilized to create the iBugMask dataset [18]
containing samples from Helen’s training set. The iBugMask authors focused on
head pose diversity and augmented images with a synthetic rotation algorithm
from 3DFFA [9], which led to huge artifacts. The Face Synthetics dataset was
specifically diversified during generation by various head poses, human
identities, clothes, and such occlusions as eyeglasses and face masks.

Annotation. Since the face parsing annotation process is more challenging than
portrait segmentation, the data is frequently marked manually or automatically
with further refining. Also, annotations of existing datasets were received by
inappropriate annotation rules, which prevented their usage due to our data
limitations. First, almost all reviewed datasets relate beard to skin class
and nostrils to nose class. Second, some contain glasses as skin and others –
annotate transparent glasses as non-transparent ones. Such factors made the
skin enhancement task impossible due to further artifacts. Finally, none of
them contain separate annotations for teeth. There are other difficulties and
features related to concrete ones:

  * •

LFW-PL [10] is limited to only 3 classes (background, face, hair), which
unsuitable for solving our specific problems.

  * •

Helen’s [12] 2,330 facial images were annotated by facial part landmarks
utilizing Amazon Mechanical Truck, and then masks were generated
automatically. The LaPa’s [19] authors pointed out Helen’s annotation errors.

  * •

CelebAMask-HQ [13] ignored occlusions on its own, however, the authors [33]
solved this problem with the dataset extension – FaceOcc. It contains images
from CelebAMask-HQ and is annotated with only one class – occlusions
(eyeglasses, tongue, makeup, and others). Regardless, FaceOcc includes a beard
to the skin class. We are considered FaceOcc as CelebAMask-HQ with FaceOcc.

  * •

The iBugMask [18] contains many persons with annotated masks for only one of
them.

The mentioned datasets are inappropriate for our task due to the described and
other problems such as low-resolution images, limited subject quantity,
privacy concerns, and poor annotation quality. In addition to general
shortcomings, other datasets lack video-conferencing domain-specific
characteristics like task-specific occlusions and situations, various head
poses, and domain context scenes. Inspired by the above and the necessity for
an appropriate dataset for video conference apps due to its widespread use, we
created a new dataset, EasyPortrait, with face parsing and portrait
segmentation annotations simultaneously. We intentionally diversified the
EasyPortrait by head poses, subjects, scenes, subjects’ attributes such as
ethnicity, and their occlusions (glasses, beards, piercing, etc.). It was
annotated by all required classes for our applications, with specific rules
for the skin class and occlusions especially (Tab. 4 in the supplementary
material).

##  3 EasyPortrait Dataset

We have created a high-resolution image dataset, EasyPortrait, which contains
portrait segmentation and face parsing annotations for each of 40,000 samples.
This part provides our dataset creation pipeline overview, the dataset
characteristics, and its splitting.

###  3.1 Image Collection & Labeling

Two crowdsourcing platforms, Yandex Toloka (for low-price labor) and ABC
Elementary (for high-quality workers), were chosen for all stages of dataset
creation. Our pipeline consists of two main stages: (1) the image collection
stage, which is followed by validation completely realized on Yandex Toloka,
and (2) the mask creation stage, which was accomplished on both platforms. At
each stage, the responses of low-skilled workers were subjected to our quality
control methods. A more detailed description is provided below.

Image Collection. The crowd workers’ task was to take a selfie or a photo of
themselves in front of the computer. As we aimed to design a diverse dataset
in terms of occlusions, races, and head turns and make it suitable to solve
teeth whitening problems, one of the further conditions periodically
supplemented the task:

  * •

Occlusions. The sent photo should contain one of such characteristics as hands
in front of the face, glasses, the tongue out, headphones, or hats.

  * •

Head turns. The head on the sent photo should be turned in any direction at
various angles.

  * •

Teeth whitening. Random workers were asked to send photos with open mouths.

  * •

Ethnicity. We recognized the significance of having a diverse dataset of
facial images and ensured participation of individuals from various countries.

Note that all workers have signed a document on the consent to the photo
publication before starting the tasks and have been notified of how the data
will be used (see Fig. 7 in the supplementary material).

Image Collection Quality Check. We foresaw the possible dishonesty of the
crowdworkers and checked all images for duplicates by image hash comparison.
The image collection quality check also includes image validation, where
images are reviewed for compliance with conditions such as the distinctness of
the face, the head being entirely in the frame, and the clarity of the frame.
Validation stage was available to crowdworkers only after training and
examination tasks. Each image was checked from 3 to 5 times by different users
for at least 80% confidence.

Figure 2: Example of data collection pipeline. The image was annotated with
individual pair classes by 5 crowd workers. The masks are averaged with the
expert-verified one and merged to obtain the final segmentation mask.

Image Labeling. The annotation of portrait segmentation usually has several
ambiguous instances, such as occlusions in front of the person, hand-held
items, hats, headphones, hair, and others. Face parsing masks are also unclear
because of occlusions in front of the face, including tongue, hair,
eyeglasses, beard, etc. The annotation rules directly affect the final
segmentation masks and the model trained on them. Rules for annotating each
class and processing occlusions for workers are given in Tab. 4 in the
supplementary material.

The labeling stage was divided into parts to simplify the annotation process
for the workers. All images received after the collection stage were gradually
sent to the annotation of individual pairs of classes: person and background,
skin and occlusions (which include such things as eyeglasses, beard, tongue
out, and others), eyes and brows, lips and teeth. The crowdsourcing platform’s
interface asks the worker to draw the polygons for the proposed instruction
pair. After labeling, we separated the overall masks of eyes and brows into
left and right ones using heuristics. Fig. 2 visualizes the mask creation
stages.

Image Labeling Quality Check. We required all workers to complete training
tasks for each pair to enhance mask quality. We analyzed crowd workers’
quality in training by automatically comparing masks from expert workers under
our control.

Figure 3: Image resolution, brightness, subjects, and class separability
analysis. a) image resolution distribution: samples overlap with equal
transparency and density reveals quantity; b), c), d) image distribution by
subjects in train, validation, and test sets, respectively (each bar
represents the count of images recorded by a particular subject group); e)
subjects’ countries distribution; f) subjects’ devices: only smartphones,
personal computers, and tablets were used while recording; g) brightness
distribution; h) mask area distribution.

We requested ABC Elementary’s qualified workers to label each image with the
subsequent verification by the platform’s experts. Due to distrust in the
platform’s experts, whom we did not control, and the lack of qualified
annotators to provide overlap only through them, we incorporated low-skilled
annotators into the pipeline with an overlap of 5. Thus, each pair of each
main image was annotated by 5 crowd workers for subsequent averaging and
getting the best result. Segmentation masks were created from polygons for
each annotation. We aggregated all 5 markups to one segmentation mask (see
bottom of the Fig. 2), checked IoU (Intersection over Union), and compared it
to a unique threshold, selected for each pair manually444The thresholds were
chosen by comparing crowd workers’ masks with corresponding experts’ masks
from training tasks. Based on this comparison, we determined a threshold for
each pair to ensure a qualitative visual result.. Then, we averaged the
received aggregated mask with verified one with weights of 0.3 and 0.7,
respectively.

Mask Merging. Whole masks were received by the alternate overlay of masks in
the following order: person, face skin, left brow, right brow, left eye, right
eye, lips, and teeth.

In addition, the decision to release the dataset to the public and ethical
reasons prompted us to add the filtration stage to the end of the pipeline –
checking for children under 18, naked people, religious signs, and watermarks.

###  3.2 Dataset Characteristics

The mean and standard deviation of images in EasyPortrait are
$[0.562,0.521,0.497]$ and $[0.236,0.236,0.232]$, respectively.

Classes. EasyPortrait is annotated with 9 classes, including “background”,
“person”, “face skin”, “left brow”, “right brow”, “left eye”, “right eye”,
“lips”, and “teeth”. We extracted all occlusions, such as glasses, hair,
hands, etc., from the skin. The beard is extracted from the skin if it is
clearly defined (refer to Fig. 10 in the supplementary material for details).
However, such parts of a person as headphones, car belts, and others are
included in the person class to facilitate background removal and streamline
the data annotation process. Nevertheless, EasyPortrait is in the process of
annotation with new classes, including mouth, hair, headphones, glasses,
earrings, nose, hat, neck, and beard.

Diversity. The proposed dataset contains images with such scenes as an office,
living room, kitchen, bedroom, outdoors, car, cafe, etc. Samples in the
dataset display various clothes, hats, headphones, and medical masks (see Fig.
9 in the supplementary material for examples). They are also diverse in
lighting conditions, subjects’ age, gender, and poses. Almost all images
contain only one person, which is especially common at meetings through video
conferencing services. We also collected images from regions such as Africa,
Asia, India, and Europe, giving us approximate region and ethnic diversity
(see Fig. 3e). Furthermore, some individuals in the pictures display emotions,
such as smiling, expressing anger, sticking their tongue out, being surprise,
and others.

Images Resolution. Most images, namely 38,353, in EasyPortrait are FullHD: the
maximum side is 1,920, and the minimum side is in the range of 835 to 1,920
(Fig. 3a). The minimal resolution is 720 × 720. Fig. 3h shows the dataset’s
samples’ mask area distribution.

Dataset Quality. We analyzed the number of points per image for each class
since images were labeled by polygons. On average, each EasyPortrait image has
655 points, from which it can be concluded that the annotation is of high
quality. In comparison, Helen [12] was annotated only with 194 points per
image and LaPa [19] with 106. Fig. 8 in the supplementary material provided
visual mask details. An essential part of the images was annotated with the
utmost carefulness.

###  3.3 Dataset Splitting

All annotated images were divided into training, validation, and testing sets,
with 30,000, 4,000, and 6,000 samples, respectively. Training images were
received from 4,398 unique users, while validation and testing images were
collected from 3,468 and 6,000 unique users, accordingly (see Fig. 3b-d). Note
that the test set has the maximum amount of unique users, as we aimed to make
it the most subject-diverse. It is also worth mentioning that subjects from
all three sets are not intersecting, eliminating any possibility of data
leakage. In addition, we added the anonymized user ID hash to the annotation
file. It can be used for manual dataset splitting into train, test, and
validation subsets.

##  4 Ablation Study

An ablation study was conducted to evaluate the influence of the dataset’s
primary characteristics. We examined the requirement for data volume and
variability in head poses in our dataset by modifying these attributes,
training the models, and then comparing the results with those achieved using
the original dataset. In the ablation study, we utilized BiSeNetv2 [34], FPN
[17], FCN [21], and Segformer-B0 [32] for portrait segmentation and face
parsing tasks. Validation and test sets remain unchanged in all ablation
experiments.

Figure 4: The impact visualization of such dataset characteristics as a)
sample amount, b) head pose diversity (both for face parsing), and c) sample
amount for portrait segmentation task. Solid lines correspond to models
trained and tested on the EasyPortrait dataset, whereas the dotted line is the
model pretrained on the EasyPortrait and tested on other datasets (see the
legend for details). We evaluated all the datasets discussed in Sec. 5.2;
however, for datasets without significant metric changes, we did not create
visualizations. Note that all the plots have different scales.

Quantitative Necessity. To evaluate the impact of data quantity, we trained
selected models using varying training set sizes: 30,000 (original), 20,000,
10,000, and 5,000 images. The deterministic slice was used for a train set
expansion, i.e., images in the $n[i]$ set are included in the $n[i+1]$ set.
The ablation study results are provided in Fig. 4. The quantitative necessity
experiments revealed an increase in metrics as the size of the training set
expanded. Both portrait segmentation and face parsing metrics show an increase
with the expansion of the training set size, however the portrait segmentation
improvement is less prominent than the face parsing results.

Pose-Diversity Necessity. We also assess the importance of the head pose by
varying pose diversity in 10,000 training images. We obtained the head pose
coefficients (yaw and pitch) for each image in the dataset using 3DDFA network
[9]. For both of these coefficients, we chose three coefficient windows from
homogeneous to heterogeneous pose distribution: [-7.5; 7.5], [-15; 15] [-200;
200]. Reducing head pose heterogeneity results in declining face parsing
metrics (Fig. 4). Variations in head rotations do not significantly impact
portrait segmentation metrics; therefore, we did not include a plot for these
experiments.

Cross-Dataset Ablation study. We also conducted an additional set of
experiments: we trained the FPN model on the EasyPortrait dataset with changes
in data diversity and then evaluated the model on other face parsing and
portrait segmentation test sets, mentioned in Sec. 5.2.

Alterations in data quantity and variations in head pose diversity have
minimal impact on portrait segmentation results, while face parsing an
increase in data diversity positively influences the model’s metrics (Fig. 4).
On average, head pose diversity tends to have a more significant impact on
results across other datasets than data quantity changes.

##  5 Experiments

The main goal of extensive base experiments is to demonstrate that the dataset
has the ability to train models, achieving concurrent results without the need
to simulate facial occlusion or pose variations (as in [19] and [18],
respectively). For this reason, we chose various models for our base
experiments. We evaluate the models’ quality via the mean Intersection-over-
Union (mIoU) metric [22].

Table 2: Evaluation results on the EasyPortrait. Column “mIoU” is divided into
two subcolumns: face parsing and portrait segmentation tasks. For face
parsing, we present mIoU metrics for each class separately, while for portrait
segmentation, we provide only the overall mIoU score. We additionally trained
FPN and Segformer-B0 on 224 × 224, 512 × 512, and 1024 × 1024 resolutions to
demonstrate the overall increasing tendency amongst both convolutional and
transformer models depending on increasing resolution.

Model | Input Size | Model Size (MB) | FPS | mIoU
---|---|---|---|---
Face Parsing | PS
skin | l-eye | r-eye | l-brow | r-brow | lips | teeth | overall | overall
BiSeNetv2 [34] | 384 | 56.5 | 91.47 | 90.75 | 71.94 | 72.57 | 67.67 | 67.53 | 80.87 | 63.09 | 76.72 | 97.95
SegFormer-B0 [32] | 14.19 | 72.45 | 92.05 | 78.55 | 79.26 | 72.5 | 72.21 | 83.53 | 73.52 | 81.38 | 98.61
FCN + MobileNetv2 [21] | 31.17 | 66.07 | 90.49 | 69.95 | 70.63 | 66.29 | 66.09 | 79.23 | 59.84 | 75.23 | 98.19
FPN + ResNet50 [17] | 108.91 | 58.1 | 92.28 | 79.48 | 80.08 | 72.64 | 72.47 | 84.15 | 74.09 | 81.83 | 98.64
DeepLabv3 [6] | 260.02 | 25.65 | 91.77 | 73.78 | 74.63 | 69.61 | 69.74 | 83.42 | 70.53 | 79.11 | 98.63
Fast SCNN [27] | 6.13 | 93.89 | 88.58 | 58.42 | 58.7 | 58.68 | 58.87 | 73.16 | 44.86 | 67.56 | 97.64
DANet [8] | 190.29 | 42.43 | 91.8 | 74.01 | 74.93 | 70.01 | 69.75 | 83.7 | 70.8 | 79.3 | 98.63
EHANet [24] | 44.81 | 132.78 | 89.68 | 68.87 | 69.26 | 63.6 | 63.82 | 73.98 | 52.05 | 72.56 | -
SINet [25] | 0.13 | 134.18 | - | - | - | - | - | - | - | - | 93.32
ExtremeC3Net [26] |  | 0.15 | 71.75 | - | - | - | - | - | - | - | - | 96.54
SegFormer-B0 | 224 | 14.9 | 74.84 | 90.19 | 68.59 | 70.46 | 65.79 | 65.72 | 77.94 | 60.66 | 74.83 | 98.17
FPN + ResNet50 | 108.91 | 61.56 | 90.6 | 69.67 | 71.88 | 65.84 | 65.64 | 78.94 | 62.95 | 75.6 | 98.31
SegFormer-B0 | 512 | 14.9 | 65.88 | 92.5 | 81.03 | 81.18 | 74.31 | 74.08 | 84.87 | 78.14 | 83.19 | 98.66
FPN + ResNet50 | 108.91 | 53.14 | 92.55 | 81.55 | 81.47 | 74.33 | 74.38 | 85.27 | 77.77 | 83.33 | 98.64
SegFormer-B0 | 1024 | 14.9 | 62.9 | 93.13 | 84.2 | 83.97 | 76.41 | 76.12 | 86.88 | 83.2 | 85.42 | 98.74
FPN + ResNet50 | 108.91 | 52.34 | 92.94 | 84.55 | 84.24 | 76.11 | 76.11 | 86.93 | 82.62 | 85.37 | 98.54

###  5.1 Base Experiments

Separation on Two Tracks. We split our experiments into two tracks – portrait
segmentation and face parsing – to transparently compare EasyPortrait with
other datasets separately. This division is also necessary to avoid ambiguity
and ensure the obtained metrics are representative of both tasks. The portrait
segmentation is based on two EasyPortrait classes (“background” and “person”),
whereas the face parsing masks include eight classes (“background”, “skin”,
“left brow”, “right brow”, “left eye”, “right eye”, “lips” and “teeth”). For
portrait segmentation, we defined all classes of EasyPortrait except the
background as a person, while for face parsing, we designated the person class
as the background. The model configuration and training process are identical
for both tasks, except for the number of classes in the decoder model’s head.

Models. We prioritized lightweight architectures for easy integration into
videoconferencing apps, enabling real-time use. As general segmentation
architectures, we selected BiSeNetv2 [34], DeepLabv3 [6], FPN [17], FCN [21],
DANet [8], and Fast SCNN [27] models. We utilized Segformer-B0 [32] to assess
the performance of the transformer model on the proposed dataset. Besides the
aforementioned widespread segmentation architectures, we also experimented
with models specifically designed for portrait segmentation and face parsing.
For this purpose, we chose the SINet [25] and ExtremeC3Net [26] for the first
one and EHANet [24] model for the second.

We trained each of these networks for 100 epochs with batch size 32. AdamW
[23] was used as an optimizer and learning rate with the initial value of
0.0002. The learning rate changes according to the polynomial learning rate
scheduler with factor 1.0 by default.

Augmentations and Images Resolution. Images and segmentation masks were
resized to the maximum side of 384 with aspect ratio preservation and
symmetrically padded to square. We used bilinear interpolation for image
resizing, while nearest neighbor interpolation was applied to masks to
maintain consistency among classes. At last, photometric distortion was used
with a brightness delta of 16, a contrast in the range [0.5, 1.0], saturation
in the range [0.5, 1.0], and a hue delta of 5.

The results of our experiments are presented in the Tab. 2. All the models
trained on our dataset achieve high metrics, with the FPN model outperforming
others in face parsing and portrait segmentation tasks.

###  5.2 Cross-Dataset Evaluation

We conduct cross-dataset evaluation to compare our dataset with existing ones
in 2 domains – face parsing and portrait segmentation.

Experiments Configuration. We train the FPN model for two segmentation tasks
on each dataset. All datasets’ samples were exposed to resizing to fixed 384 ×
384 shape and base augmentations pipeline described in Sec. 5.1. The training
process and model configuration are the same as the base experiments for both
tasks.

Portrait Segmentation. Besides our dataset, the model was trained and tested
on HumanSeg14K [7], Face Synthetics [31], and FVS [11] portrait segmentation
datasets. We couldn’t include the EG-1800 [29] and the AiSeg [1] datasets due
to a lack of images on the public shared sources and inappropriate samples,
respectively.

Some preprocessing steps were applied to each of the datasets:

  * •

We led the EasyPortrait’s class “person” to a consistent appearance by
labeling others classes (without “background”) as “person” class.

  * •

FVS [11] is announced as a segmentation dataset; however, the provided masks
are not binary, so we binarize them. We found out that pixel values are mostly
scattered near 0 or 255; therefore, the average threshold of 127 was chosen to
separate the “person” and “background” classes. The original dataset has been
split into 1,326 training and 935 testing samples. We randomly sampled 200
images from the training set to design the validation.

  * •

HumanSeg14K [7] dataset was divided into the training, validation, and test
parts with 8,770, 2,431, and 2,482 samples, respectively.

  * •

Similar to EasyPortrait’s preprocessing, we prepared the Face Synthetics [7]
dataset to portrait segmentation masks. We randomly picked 75,000 training,
15,000 testing, and 10,000 validation samples.

Face Parsing. As far as the EasyPortrait skin class was annotated by the
unique rules and most face parsing datasets are not annotated with the teeth
class, we selected only 6 classes for cross-data evaluation: “background”,
“left brow”, “right brow”, “left eye”, “right eye”, “lips”. We adopted the
original annotations of face parsing datasets to the target ones:

  * •

Such EasyPortrait classes as “teeth”, “person” and “skin” classes were mapped
to the “background”.

  * •

Since lips of the CelebAMask-HQ dataset are divided into two classes: “lower
lip” and “upper lip”, we combined them into one “lips” class. The remaining
classes are considered as background. All datasets below were preprocessed in
the same way as CelebAMask-HQ [13]. We divided the CelebAMask-HQ dataset into
22,500 training, 3,000 validation, and 4,500 test samples.

  * •

The LaPa [19] dataset was originally split into 18,167 training, 2,000
validation, and 2,000 test samples.

  * •

Originally, images from iBugMask [18] were split into 21,866 training and
1,000 testing examples. The validation set was randomly sampled from the
training set and contains 1,866 images. Note that iBugMask [18] contains
images with a bounding box for the face in the provided mask. To avoid parsing
other faces, we crop them as described in the original paper.

  * •

The Face Synthetics dataset was distributed into 75,000 training, 10,000
validation, and 15,000 test samples by us.

Table 3: Cross-dataset evaluation results. Each cell value contains mean
(among classes) IoU metrics for the corresponding training and testing sets
pair. Train (test) average mIoU represents the overall mean IoU value on the
listed testing (training) sets. There is a direct relationship between the
high train average mIoU metric and the dataset’s generalization ability to
other distributions. The low test average mIoU metric value reflects the
dataset’s complexity, as a model pre-trained on a different training set
struggles to achieve a high metric. This metric indicates the dataset’s
suitability as a benchmark for face parsing or portrait segmentation tasks. We
highlighted the best metric in each column to emphasize the dataset’s ability
to generalize to other distributions. The best metric in all columns except
the last one was chosen, excluding diagonal values. (upper table) The FPN
model, trained on the EasyPortrait dataset for portrait segmentation, achieves
state-of-the-art results and surpasses FVS results even on their own test set.
(lower table) Despite the limited videoconferencing domain, we achieved
concurrent results in the face parsing task.

Portrait Segmentation
---
|  | Tested |
| Dataset | EasyPortrait (ours) | FVS | HumanSeg14K | Face Synthetics | Train avg. mIoU
Trained | EasyPortrait (ours) | 98.64 | 97.86 | 93.18 | 97.76 | 96.86
FVS [11] | 79.05 | 96.24 | 90.6 | 80.36 | 86.56
HumanSeg14K [7] | 76.01 | 96.23 | 97.53 | 71.66 | 85.35
Face Synthetics [31] | 84.99 | 57.14 | 57.87 | 99.44 | 74.86
| Test avg. mIoU | 84.67 | 86.87 | 84.8 | 87.31 |

Face Parsing
---
|  | Tested |
| Dataset | EasyPortrait (ours) | CelebAMask-HQ | iBugMask | Face Synthetics | LaPa | Train avg. mIoU
Trained | EasyPortrait (ours) | 81.51 | 76.01 | 39.0 | 51.2 | 61.03 | 61.75
CelebAMask-HQ [13] | 66.17 | 83.41 | 54.74 | 46.6 | 60.62 | 62.31
iBugMask [18] | 61.58 | 79.1 | 64.59 | 44.42 | 66.3 | 63.19
Face Synthetics [31] | 55.55 | 40.67 | 18.84 | 83.12 | 42.63 | 48.16
LaPa [19] | 68.56 | 73.92 | 47.66 | 48.05 | 79.02 | 63.44
| Test avg. mIoU | 66.67 | 70.62 | 44.97 | 54.68 | 61.92 |

Results. The cross-dataset evaluation results in Tab. 3 demonstrate that the
EasyPortrait has the best generalization capability regarding mIoU metrics on
each portrait segmentation test set. Due to the reduced list of classes, the
quantitative assessment provides limited insights into the dataset’s
applicability for the face parsing task. Besides, our domain is slightly
limited555Images in other datasets are more heterogeneous in context,
displaying multiple people and different activities. EasyPortrait consistently
shows a single person in front of a computer or phone., which prevented
achieving more optimistic results.

##  6 Conclusion

In this paper, we proposed a large-scale image dataset for portrait
segmentation and face parsing tasks. It consists of 40,000 photos of ordinary
people predominantly indoors, and each image is provided with a 9-classes
high-quality semantic mask. Our dataset can be used in several beautification
and segmentation tasks, such as background removal, face skin enhancement, and
teeth whitening, which can improve user experience in video conferencing apps.
We provide extensive experiments on different models and cross-dataset
comparisons for both described tasks. We also conducted an ablation study
revealing the importance of such dataset characteristics as data quantity and
head pose diversity for training a precise and robust model. For future work,
we plan to add several occlusions to the annotation and improve the dataset by
including additional classes, such as mouth, hair, headphones, glasses,
earrings, nose, hat, neck, and beard.

## References

  * [1] AISeg. https://github.com/aisegmentcn/matting_human_datasets (2019)
  * [2] 10 video conferencing trends enhanced by face ar technology. https://www.banuba.com/blog/10-video-conferencing-trends-enhanced-by-face-ar-technology (2020)
  * [3] Persons Labeled. https://ecosystem.supervise.ly/projects/persons (2020)
  * [4] How video tech can bring us closer than ever. https://inlnk.ru/Vo8ydV (2023)
  * [5] There’s a filter on zoom you can use to look better on video calls while working from home. https://www.businessinsider.com/zoom-video-conferencing-filter-touch-up-work-from-home-coronavirus-2020-3?international=true&r=US&IR=T (2023)
  * [6] Chen, L.C., Papandreou, G., Schroff, F., Adam, H.: Rethinking atrous convolution for semantic image segmentation (2017)
  * [7] Chu, L., Liu, Y., Wu, Z., Tang, S., Chen, G., Hao, Y., Peng, J., Yu, Z., Chen, Z., Lai, B., Xiong, H.: Pp-humanseg: Connectivity-aware portrait segmentation with a large-scale teleconferencing video dataset. 2022 IEEE/CVF Winter Conference on Applications of Computer Vision Workshops (WACVW) pp. 202–209 (2021)
  * [8] Fu, J., Liu, J., Tian, H., Li, Y., Bao, Y., Fang, Z., Lu, H.: Dual attention network for scene segmentation (2019)
  * [9] Guo, J., Zhu, X., Lei, Z.: 3ddfa. https://github.com/cleardusk/3DDFA (2018)
  * [10] Kae, A., Sohn, K., Lee, H., Learned-Miller, E.G.: Augmenting crfs with boltzmann machine shape priors for image labeling. 2013 IEEE Conference on Computer Vision and Pattern Recognition pp. 2019–2026 (2013)
  * [11] Kuang, Z., Tie, X.: Flow-based video segmentation for human head and shoulders. ArXiv abs/2104.09752 (2021)
  * [12] Le, V., Brandt, J., Lin, Z.L., Bourdev, L.D., Huang, T.S.: Interactive facial feature localization. In: European Conference on Computer Vision (2012)
  * [13] Lee, C.H., Liu, Z., Wu, L., Luo, P.: Maskgan: Towards diverse and interactive facial image manipulation. 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) pp. 5548–5557 (2019)
  * [14] Li, J., Ma, S., Zhang, J., Tao, D.: Privacy-preserving portrait matting. Proceedings of the 29th ACM International Conference on Multimedia (2021)
  * [15] Liang, J., Zeng, H., Cui, M., Xie, X., Zhang, L.: Ppr10k: A large-scale portrait photo retouching dataset with human-region mask and group-level consistency. 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) pp. 653–661 (2021)
  * [16] Lin, S., Ryabtsev, A., Sengupta, S., Curless, B., Seitz, S.M., Kemelmacher-Shlizerman, I.: Real-time high-resolution background matting. 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) pp. 8758–8767 (2020)
  * [17] Lin, T.Y., Dollár, P., Girshick, R., He, K., Hariharan, B., Belongie, S.: Feature pyramid networks for object detection (2017)
  * [18] Lin, Y., Shen, J., Wang, Y., Pantic, M.: Roi tanh-polar transformer network for face parsing in the wild. Image Vis. Comput. 112, 104190 (2021)
  * [19] Liu, Y., Shi, H., Shen, H., Si, Y., Wang, X., Mei, T.: A new dataset and boundary-attention semantic segmentation for face parsing. In: AAAI Conference on Artificial Intelligence (2020)
  * [20] Liu, Z., Luo, P., Wang, X., Tang, X.: Deep learning face attributes in the wild (December 2015)
  * [21] Long, J., Shelhamer, E., Darrell, T.: Fully convolutional networks for semantic segmentation. 2015 Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) 3431-3440 (2014)
  * [22] Long, J., Shelhamer, E., Darrell, T.: Fully convolutional networks for semantic segmentation (2015)
  * [23] Loshchilov, I., Hutter, F.: Decoupled weight decay regularization. 2019 The International Conference on Learning Representations (ICLR) abs/1711.05101 (2017)
  * [24] Luo, L., Xue, D., Feng, X.: Ehanet: An effective hierarchical aggregation network for face parsing. Applied Sciences 10(9), 3135 (2020)
  * [25] Park, H., Sjösund, L.L., Monet, N., Yoo, Y., Kwak, N.: Sinet: Extreme lightweight portrait segmentation networks with spatial squeeze modules and information blocking decoder. arXiv preprint arXiv:1911.09099 (2019)
  * [26] Park, H., Sjösund, L.L., Yoo, Y., Kwak, N.: Extremec3net: Extreme lightweight portrait segmentation networks using advanced c3-modules. arXiv preprint arXiv:1908.03093 (2019)
  * [27] Poudel, R.P.K., Liwicki, S., Cipolla, R.: Fast-scnn: Fast semantic segmentation network (2019)
  * [28] Sander, E.L.J.: Coronavirus could spark a revolution in working from home: Are we ready? The conversation (2020)
  * [29] Shen, X., Hertzmann, A., Jia, J., Paris, S., Price, B.L., Shechtman, E., Sachs, I.: Automatic portrait segmentation for image stylization. Computer Graphics Forum 35 (2016)
  * [30] Taherkhani, F., Nasrabadi, N.M., Dawson, J.: A deep face identification network enhanced by facial attributes prediction (2018)
  * [31] Wood, E., Baltruvsaitis, T., Hewitt, C., Dziadzio, S., Johnson, M., Estellers, V., Cashman, T.J., Shotton, J.: Fake it till you make it: face analysis in the wild using synthetic data alone. 2021 IEEE/CVF International Conference on Computer Vision (ICCV) pp. 3661–3671 (2021)
  * [32] Xie, E., Wang, W., Yu, Z., Anandkumar, A., Álvarez, J.M., Luo, P.: Segformer: Simple and efficient design for semantic segmentation with transformers. In: Neural Information Processing Systems (2021)
  * [33] Yin, X., Chen, L.L.: Faceocc: A diverse, high-quality face occlusion dataset for human face extraction. ArXiv abs/2201.08425 (2022)
  * [34] Yu, C., Wang, J., Peng, C., Gao, C., Yu, G., Sang, N.: Bisenet: Bilateral segmentation network for real-time semantic segmentation. In: European Conference on Computer Vision (2018)
  * [35] Zhu, X., Liu, X., Lei, Z., Li, S.Z.: Face alignment in full pose range: A 3d total solution. IEEE transactions on pattern analysis and machine intelligence (2017)

##  Appendix 0.A Supplementary material

Figure 5: Visual comparison of existing portrait segmentation datasets. One
can notice high-frequency details (e.g. hair) in segmentation masks in samples
from our dataset. The AiSeg [1] dataset is not considered since it provides
the extracted foreground images without corresponding annotation mask. Figure
6: Visual comparison of existing face parsing datasets. Only Face Synthetics
[31] and EasyPortrait datasets can be used to solve background removal and
face enhancement problems. None of them except EasyPortrait can be used in the
teeth whitening task. We don’t include LFW-PL [10] and FaceOcc [33] datasets
in the visualization due to the lack of classes and the need for
preprocessing, respectively. Figure 7: Example of instruction for the task
with notification of further use of data for training neural networks. Figure
8: Examples of facial details in the EasyPortrait dataset. Samples contain
many different facial expressions, with open and closed mouth, with or without
teeth shown. Note that obstacles in front of the face (like glasses and a
cigarette) are annotated as a class “person". Figure 9: Samples from
EasyPortrait dataset. Note that images have different orientation and height-
width ratio. Some of the pictures are overexposed, while other are
underexposed. Figure 10: Visualization of the beard annotation rules. (up) The
beard is included in the skin if it is a separate hair or barely noticeable.
(bottom) The beard is excluded from the skin if it is clear. Figure 11: Head
turns distributions for several face parsing and portrait segmentation
datasets, including EasyPortrait. Yaw and pitch coefficients were obtained via
3DDFA network [9]. Table 4: EasyPortrait annotators rules. Visual examples are
provided in Fig. 8.

Class |  Rules
---|---
Person |  – headphones and things in front of the person are defined as a person’s class
– individual hairs and all empty areas closed by a person are not included in
the person class
Eyebrows |  stand out along a strict border, excluding individual hairs
Eyes |  distinguished by whites, excluding eyelids and eyelashes
Skin |  – the skin class should affect only skin without hair, eyes, and other face attributes
– the boundaries of the skin of the face or person should be highlighted
logically on overexposed or darkened photos
– the rare bristle also considered skin
– ears, second chin, and nostrils are not included in the skin class
Teeth |  teeth and everything else in the open mouth stand out separately, the latter as an occlusion
Occlusions |  – makeups and piercing are defined as occlusions
– the part of eyeglasses, which cover skin, should be annotated as occlusion,
including sunglasses and glare on clear glasses
– beard with a strict border are considered occlusion
– the tongue out of the mouth should be annotated as occlusion