11institutetext: Korea Institute of Sciene and Technology, Korea
22institutetext: Korea National University of Science and Technology, Korea
33institutetext: Yonsei-KIST Convergence Research Institute, Yonsei University
33email: {kim1102, mcsagong, gpnam, jhcho<EMAIL_ADDRESS>
# VIGFace: Virtual Identity Generation Model for Face Image Synthesis

Minsoo Kim∗ 1122 0000-0001-6813-7335 Min-Cheol Sagong∗ 11 0000-0001-9191-4147
Gi Pyo Nam 1122 0000-0002-3383-7806 Junghyun Cho 112233 0000-0003-1913-8037
Ig-Jae Kim 112233 0000-0002-2741-7047

###### Abstract

Deep learning-based face recognition continues to face challenges due to its
reliance on huge datasets obtained from web crawling, which can be costly to
gather and raise significant real-world privacy concerns. To address this
issue, we propose VIGFace, a novel framework capable of generating synthetic
facial images. Initially, we train the face recognition model using a real
face dataset and create a feature space for both real and virtual IDs where
virtual prototypes are orthogonal to other prototypes. Subsequently, we
generate synthetic images by using the diffusion model based on the feature
space. Our proposed framework provides two significant benefits. Firstly, it
allows for creating virtual facial images without concerns about portrait
rights, guaranteeing that the generated virtual face images are clearly
differentiated from existing individuals. Secondly, it serves as an effective
augmentation method by incorporating real existing images. Further experiments
demonstrate the efficacy of our framework, achieving state-of-the-art results
from both perspectives without any external data.

###### Keywords:

Face recognition, Diffusion model, Image generation, Data augmentation,
Synthetic data

**footnotetext: These authors contributed equally to this work

##  1 Introduction

Deep learning-based Face Recognition (FR) models have significantly improved
their performance due to recent advancements in network architectures [46, 48,
16, 25, 17, 20] and enhancements in loss functions [44, 43, 54, 30, 10, 27].
The latest FR models utilize the softmax-variant loss for training to reduce
intra-class variance and increase inter-class separability in the embedding
space. This necessitates a very large dataset with numerous distinct
individuals, significant variations within each individual for intra-class
variance, and precise labels of subject identities (IDs). However, the
datasets are typically collected through web-crawling and then refined using
automatic techniques that employ FR logits [9, 58]. While this approach is
successful in eliminating mislabeled data, it still struggles with persistent
issues of small intra-class variance. Moreover, in the face recognition field,
unlike other ordinary image classifications, there are practical issues such
as portrait rights, making it even more difficult to collect training data.
For instance, datasets such as those referenced in [55, 15, 58] consist of
images of celebrities crawled from the internet without consent. Additionally,
datasets mentioned in [26, 37] include facial images of the general populace,
including children. After facing strong criticism, public access to these
datasets has been revoked [52].

Synthetic datasets have been employed to overcome limitations caused by a
scarcity of real datasets [51, 24, 8] and biases present in the available real
datasets [6, 19]. In the realm of face recognition, artificial faces show
promise in addressing the aforementioned issues related to real face datasets.
Generated faces have low risk from label noise due to conditional generation.
Bias problems such as long-tailed distributions, which lead to class
imbalances, can be mitigated by data augmentation. Importantly, there are no
privacy concerns with facial images of virtual identities if the method is
effective. Therefore, when creating artificial datasets, it is essential to
consider: 1) ensuring the generated data mirrors real data distribution, 2)
enhancing ability to generate new subjects in the target domain by taking the
full source domain into account for augmentation, and 3) maintaining ID
consistency for each subject.

Previous attempts to create artificial face datasets have addressed some of
the three aspects individually, but to the best of our knowledge, none have
simultaneously taken into account all three aspects[2, 39, 28]. SynFace [39]
introduces a high-quality synthetic face image dataset, closely resembling
real face images, using DiscoFaceGAN [12]. Nonetheless, DiscoFaceGAN is only
capable of generating a limited number of unique subjects, fewer than 500
[28]. On the other hand, DigiFace [2] utilizes 3D parametric modeling to
create synthetic face images of various subjects. However, it faces
difficulties in accurately replicating the quality and style distribution
observed in real face images. DCFace [28] proposes a diffusion-based method
for generating data that can produce a large number of novel subjects,
maintain the style of real datasets, and ensure label consistency. However,
DCFace primarily focuses on creating synthetic data rather than increasing the
capabilities of data augmentation.

We present a novel approach for generating face data that enhances both intra-
class variance and inter-class diversity. The main concept of our paper is
incorporating virtual prototypes into the FR model. The diffusion model is
trained to preserve the identity of the individuals when generating face
images based on the FR embedding. Since virtual prototypes are orthogonal to
other prototypes, virtual subjects are guaranteed to be distinguished from the
original data. Fig. 1 shows toy example visualizing the embeddings of real and
virtual subjects to demonstrate the effectiveness of the suggested method. The
virtual embedding can be distinct from the real individual clusters, whereas
the images generated from virtual prototypes form unique clusters. The
visualization demonstrates that our framework generates distinctive virtual
human faces with high ID consistency. The proposed method effectively
addresses privacy concerns by creating datasets of non-existent individuals
and achieving state-of-the-art performance compared to models trained with
previous virtual person generation methods. Additionally, the FR model, which
was trained using combination of real and synthetic images together, achieves
better performance than the model trained using only real images. This shows
that the proposed method is also superior from the perspective of a data-
augmentation method. Our contribution can be summarized as follows:

  * $\bullet$

Proposing VIGFace, a solution capable of generating virtual face images that
addresses three important criteria for face recognition: realistic appearance,
novel subjects, and consistent ID characteristics.

  * $\bullet$

Showing that the synthetic data generated by our proposed method can enhance
intra-class and inter-class variance by achieving SOTA performance in face
recognition.

  * $\bullet$

Releasing the first virtual-only face dataset that can fully substitute the
real dataset, aiding in alleviating privacy concerns.

Figure 1: T-distributed Stochastic Neighbor Embedding (T-SNE) [33] plot of
embeddings from real and synthesized images. The filled and lined stars
represent the real and virtual prototypes, while circles and crosses indicate
the position for embeddings of real and synthesized images, respectively.

##  2 Related work

###  2.1 Face Recognition Models

Current SOTA FR methods, such as [30, 53, 10, 23, 34, 11, 4, 27], are designed
based on the softmax loss but particularly utilize angular/cosine distance
instead of Euclidean distance. The goal of these methods is to maximizing the
similarity between the embeddings and the ground-truth (GT) prototype while
minimizing the similarity between the embedding and prototypes of other
classes. As a result, all embeddings in the same class, including prototypes,
converge while maintaining their maximum distance from other classes.
Therefore, if the embedding dimension is large enough, the clusters of each
class will be nearly orthogonal [10]. Naturally, a dataset that is larger and
contains a greater various of images allows for the training of better-
performing FR models. Therefore, over time, larger and more diverse datasets
[21, 15, 1, 58] have been collected and published in the academic field.
However, issues such as portrait rights and the high cost of building large-
scale data still remain unresolved.

###  2.2 Synthetic Face Image Generation

Synthetic training datasets have demonstrated advantages in areas such as face
recognition [2, 39] and anti-spoofing[31, 49]. They can address class
imbalance issues related to informed consent and racial biases that often
noted in conventional large-scale face datasets [10, 39, 2]. Moreover, they
endorse the importance of ethical considerations, like portrait rights.
Despite their benefits, the use of synthetic datasets lacks widespread
acceptance, causing reduced recognition accuracy when employed as the sole
training data.

SynFace [39] proposed the method using DiscoFaceGAN[12] to synthesize the
virtual face. DigiFace-1M [2] suggested 3D model-based face rendering with
image augmentations to create virtual face data and showed that the FR model
could be trained using only virtual face images. DCFace [28] proposed a
diffusion-based face generator combining subject appearance (ID) and external
factor (style) conditions. Although the above images have succeeded in
reducing the dependence on images crawled from the web, there is still a
performance gap between the FR model trained on real training datasets and the
FR model trained on synthetic training datasets. We proposed a data
augmentation method that can generate the virtual ID with high intra-class
variance and large number of unique individuals, thereby aiding in FR model
training.

##  3 Preliminary

###  3.1 Diffusion Model

Diffusion model gradually denoises random Gaussian noise to generate real
data. The diffusion model reverses the forward noise-injection process, also
called diffusion process, which is formulated as a Markov chain:

$q(x_{1:T}|x_{0})=\prod_{t=1}^{T}q(x_{t}|x_{t-1}),$ (1)

where $x_{0}$ is real data,
$q(x_{t}|x_{t-1})=\mathcal{N}(x_{t};\sqrt{1-\beta_{t}}x_{t-1},\beta_{t}I)$,
and $\beta_{t}$ is the differentiable noise schedule. To reverse this process,
also called denoising process, the diffusion model is learned to denoise
$x_{t}\sim q(x_{t}|x_{0})$ into an estimate
$\hat{x}_{\theta}(x_{t},t)~{}\approx~{}x_{0}$ for all $t$ that are uniform
between 1 and $T$. Formally, this is achieved by training $\hat{x}_{\theta}$
using squared error loss:

$\min_{\theta}\mathbb{E}_{\epsilon,t}\|\hat{x}_{\theta}(x_{t},t,c)-x_{0}\|^{2}_{2},$
(2)

where $c$ is an optional conditioning label or its continuous embedding
vector. Recent work shows that $c$ can be from various modalities such as a
low-resolution image or text embedding [41, 42].

##  4 Methods

Our framework comprises two stages. Firstly, we train the FR model using the
real face dataset and design the feature space for both real and virtual IDs.
Subsequently, synthetic images are generated using the diffusion model based
on the feature space of a pre-trained FR model. This section provides a
detailed explanation of each component of the proposed framework.

###  4.1 Stage 1: FR Model Training

Figure 2:  Pipeline for stage 1: suggesting face recognition model training.
Conventional FR training includes prototypes for real individuals only,
denoted as $W_{R}=[w_{r}^{1},w_{r}^{2},...,w_{r}^{n}]$. We add $k$ prototypes
for virtual IDs, denoted as $W_{V}=[w_{v}^{1},w_{v}^{2},...,w_{v}^{k}]$.

The proposed framework begins with training the FR model using real face
images. The overall pipeline of this stage is illustrated in Fig. 2. This
stage serves two purposes: 1) Training the FR model to extract ID embeddings
from face images, which are necessary for training the diffusion model in the
second stage, and 2) Simultaneously assigning the positions of both the real
ID and the virtual ID on the feature space. We choose ArcFace [10] to train
the FR model in this stage. The ArcFace loss used to train the FR backbone
$f_{\mathrm{FR}}$ and the prototype $W=[w_{1},w_{2},...,w_{n}]$ can be
described as follows:

$L_{\mathrm{arc}}=-\log{\frac{e^{s\,\cos(\theta_{y_{i}}+m)}}{e^{s\,\cos(\theta_{y_{i}}+m)}+\sum_{j=1,j\neq
y_{i}}^{n}e^{s\,\cos\theta_{j}}}},\vspace{-0.5pt}$ (3)
$\cos\theta_{j}=\frac{{w_{j}}^{T}f_{\mathrm{FR}}(x_{i})}{\|w_{j}\|\|f_{\mathrm{FR}}(x_{i})\|},$
(4)

where $n$ denotes the total number of real IDs, while $m$ and $s$ represent
the margin and scale hyperparameters, respectively. With conventional FR
training methods, only prototypes for real individuals, that can be denoted as
$W_{R}=[w_{r}^{1},w_{r}^{2},...,w_{r}^{n}]$, are necessary for training.
However, in our work, we include additional $k$ prototypes for virtual IDs,
denoted as $W_{V}=[w_{v}^{1},w_{v}^{2},...,w_{v}^{k}]$, which are used to
generate facial images of non-existent individuals. As a result, the prototype
is defined as a linear transformation matrix $W\in\mathbb{R}^{(n+k)\times D}$,
where $D$ refers to the dimension of the embedding features.

However, due to the absence of a face image for virtual IDs, their prototypes
cannot be updated to maximize distance from each other with $L_{arc}$.
Consequently, all virtual prototypes converge to a single point in the feature
space when trained with the original $L_{arc}$ as shown in Fig. 3(a). To
address this problem, we propose to use virtual feature embedding
$f^{\prime}_{\mathrm{FR}}(x_{j})$ to update the virtual prototypes
$w_{v}^{j}$. The virtual embedding $f^{\prime}_{\mathrm{FR}}(x_{j})$
corresponding to virtual person ID: $j$ was generated as follows:

$f^{\prime}_{\mathrm{FR}}(x_{j})=w_{v}^{j}+\mathcal{N}(0,\,1)\cdot\sigma,$ (5)
$\sigma^{2}=\frac{1}{b}\sum_{i=1}^{b}(f_{\mathrm{FR}}(x_{i})-w_{r}^{i})^{2},$
(6)

where $b$ refers the mini-batch size. As can be seen from the equations, the
virtual embedding $f^{\prime}_{\mathrm{FR}}(x_{j})$ follows a distribution in
which the standard deviation matches that of the real embeddings in a mini-
batch. This virtual embedding enables the virtual prototype $w_{v}^{j}$ to be
updated while maintaining minimum similarities from other prototypes, as
depicted in Fig. 3(a). Since batch configuration impacts on the calculated
standard deviation, we utilized Exponential Moving Average (EMA) to reduce
this influence. Corrected standard deviation $\sigma$ for current iteration
$t$ was computed as follows:

$\sigma=\sigma_{(t)}\cdot\alpha+\sigma_{(t-1)}\cdot(1-\alpha)$ (7)

The hyperparameter $\alpha$ is set to 0.9. Each iteration, we generated
virtual embeddings of the same size as the mini-batch $b$ for training. Then,
Eq. 3 is calculated using both virtual and real embeddings together. As shown
in Fig. 3(b), we confirm that the similarity distribution between virtual
prototypes closely resembles the similarity distribution between real
prototypes. Note that gradients for virtual embedding does not effect on the
backbone, as shown in Fig. 2.

(a) Cosine similarity values between prototypes.

# ID | Condition | Mean (Std)
---|---|---
10K(Real) + 60K(Virtual) | R-R | 0.009 (0.140)
R-V | 0.000 (0.044)
V-V | 0.010 (0.109)

(b) Means and standard deviations of cosine similarity values.

Figure 3: The change of min-max normalized cosine similarity matrix of
prototypes from our method. In the table, R and V represent the prototypes for
real and virtual ID, respectively.

###  4.2 Stage 2: Face Synthesis with Diffusion Model

The next step is synthesizing face images using the diffusion model. To obtain
the training dataset for the diffusion model, we utilize the FR model, which
entails collecting pairs of images $x_{0}$ and their corresponding embeddings
$f_{\mathrm{FR}}(x_{0})$. The input to our diffusion model includes the
timestep $t$, the embedding vector $f_{\mathrm{FR}}(x_{0})$, and the noisy
image $x_{t}$. In line with the approach proposed in the previous method [40],
our model predicts clean images $x_{0}$ rather than noise $\epsilon$ injected
into $x_{t}$. We adopt the U-ViT architecture [3] using the design approach of
the visual transformer (ViT) [13]. We modify the U-ViT model by incorporating
adaptive layer normalization (AdaLN), which shares similarities with the
conditioning mechanism used in generative models [14, 35, 38]. Formally,
$\mathrm{AdaLN}(h,c)=f_{s}(c)\mathrm{LayerNorm}(h)+f_{b}(c)$, where $h$ is the
hidden states associated with the transformer block, and $f_{s},f_{b}$ are
linear projections for condition embedding. In order to achieve both the
generation of facial images and the synchronization of their feature space on
the FR model, our diffusion model incorporates a constraint which aims to
minimize the feature distance between the original image and the generated
image as follows:

$\min_{\theta}\mathbb{E}_{\epsilon,t}\|f_{\mathrm{FR}}(\hat{x}_{\theta}(x_{t},t,f_{\mathrm{FR}}(x_{0})))-f_{\mathrm{FR}}(x_{0})\|^{2}_{2},$
(8)

where $f_{\mathrm{FR}}(x_{0})$ indicates the FR model embedding for input
image $x_{0}$. We adopt classifier-free guidance [18] by randomly assigning
zero values to condition embeddings $f_{\mathrm{FR}}(x_{0})$, _i.e_. the FR
model embedding, 10% of the time. Sampling is performed as follows:

$\tilde{x}_{\theta}(x_{t},t,f_{\mathrm{FR}}(x_{0}))=w\cdot
x_{\theta}(x_{t},t,f_{\mathrm{FR}}(x_{0}))+(1-w)\cdot x_{\theta}(x_{t},t),$
(9)

where $x_{\theta}(x_{t},t,c)$ and $x_{\theta}(x_{t},t)$ are conditional and
unconditional $x_{0}$-prediction, respectively and $w$ is guidance weight. The
pipeline of diffusion model is illustrated in the Appendix.

##  5 Experiments

We analyzed our framework from two perspectives. Firstly, by training the FR
model exclusively with facial images generated from virtual IDs, we
demonstrated the capability of our model to serve as a viable alternative to
real face datasets, addressing concerns such as label noise or privacy.
Secondly, we train the FR model using both real images and generated face
images from virtual IDs, showcasing the potential of our model as a data
augmentation framework for face recognition.

###  5.1 Implementation Details

####  5.1.1 For stage 1:

All experiments share the same implementation details for stage 1, with the
only difference being the use of the virtual embedding
$f^{\prime}_{\mathrm{FR}}(x_{j})$ during training. Modified ResNet-50 [10] and
ArcFace [10] were employed for training the FR backbone. CASIA-WebFace [21]
and MS1M-V2 [10] datasets were used for training. Following the approach in
[30], the faces were aligned resulting in an image size of $112\times 112$.
After the alignment, the pixel values of the training dataset were normalized,
setting the mean and standard deviation at 0.5 each. The size of the mini-
batch was set to 512, and Stochastic Gradient Descent (SGD) was used as the
optimizer. The weight decay was set at 5e-4 and the momentum at 0.9. The
initial learning rate was set at 0.1. We divided our training protocol into
two, depending on the training dataset. When training the model with CASIA-
WebFace dataset, the learning rate decayed by 0.1 at the 24th, 30th, and 36th
epochs, and training terminated after the 40th epoch. The hyperparameters for
the margin $m$ and scale factor $s$ in Eq. 3 were set to 0.5 and 30,
respectively. When training the model with the MS1M-V2 dataset, we decayed the
learning rate by 0.1 at the 10th, 18th, and 22nd epochs and terminated the
training after the 25th epoch. The hyperparameters for the margin $m$ and
scale factor $s$ in Eq. 3 were set to 0.5 and 64, respectively, when training
with the MS1M-V2 dataset.

####  5.1.2 For stage 2:

To generate synthetic face images, we explored the best setting for the
diffusion model. To facilitate comparison, we utilized the widely adopted
ViT-B model for all experiments. Since the traditional FR model typically
employs a resolution of $112\times 112$ for face images, we set the window
size of the ViT patch extractor to 4. We followed the publicly released
implementation of DDIM [47] with the cosine noise scheduler [7]. The diffusion
model was trained for 5M iterations with a batch size of 512 using AdamW
Optimizer [29, 32] with a learning rate of 1e-4. For sampling, we used
classifier-free guidance implementation [18] in 100 timesteps.

Figure 4: Generated images of real ID with different classifier-free guidance
weights. Higher guidance weights enforce stronger conditioning of input labels
but slightly similar samples. However, a high guidance weight results in over-
enhanced images.

###  5.2 Virtual Identity Generation

Since diffusion models focus on optimizing the visual aspect with input
conditions, they tend to produce images that are high in consistency but lack
of variance. One simple way to achieve a balance between consistency and
variance is by adjusting the classifier-free guidance weight scale. Higher
guidance enforces stronger conditioning of input labels but slightly similar
samples, as shown in Fig. 4. However, it is evident that a high guidance
weight results in over-enhanced images. To effectively generate constant but
diverse face images, we evaluate the similarity between the synthesized images
and compare their distributions with real data on different guidance scales.
In Fig. 5, we illustrate the similarity distributions of our generated images
and real images as predicted by pre-trained ResNet-18 models trained with
ArcFace[10] and Glint-360K[1]. As shown in the figure, a higher weight causes
the similarity distribution to shift towards higher average. In other words, a
higher guidance weight results in more constant images. To determine the
optimal scale for replicating the real data distribution, we assess the
Intersection over Union (IoU) between the distributions of real and synthetic
data. Note that the similarity distributions of real ID and virtual ID exhibit
distinct patterns on the guidance scale, as shown in Tab. 1. We observe that
the proposed model simulates the actual data distribution for the real ID when
the scale is w = 0.5, but the synthesis of the virtual ID performs best as the
scale becomes w = 2.5.

(a) Cosine similarity distribution changes on different guidance scales in the
genuine class.

(b) Cosine similarity distribution changes on different guidance scales in the
imposter class.

Figure 5: Data distributions of the real and our proposed generated data.

CFG scale |  0.5 |  1.0 |  1.5 |  2.0 |  2.5 |  3.0
---|---|---|---|---|---|---
Genuine |  0.88 |  0.90 |  0.83 |  0.81 |  0.81 |  0.81
Imposter |  0.93 |  0.89 |  0.87 |  0.83 |  0.76 |  0.76
Average |  0.90 |  0.89 |  0.85 |  0.82 |  0.78 |  0.73

(a) IoU of the similarity distributions between real and synthesized images
for real ID.

CFG scale |  0.5 |  1.0 |  1.5 |  2.0 |  2.5 |  3.0
---|---|---|---|---|---|---
Genuine |  0.28 |  0.40 |  0.51 |  0.60 |  0.65 |  0.68
Imposter |  0.81 |  0.77 |  0.79 |  0.77 |  0.77 |  0.73
Average |  0.54 |  0.78 |  0.65 |  0.68 |  0.71 |  0.70

(b) IoU of the similarity distributions between real and synthesized images
for virtual ID.

Table 1: Impact of classifier-free guidance scale on similarity among the
genuine (positive pair) and imposter (negative pair), respectively. We
evaluate the IoU between the distributions of synthesized and real images on
different guidance scales.

We construct a toy example visualizing the embeddings of three real persons
and five generated virtual persons to demonstrate the efficacy of the
synthesized images in Fig. 1. As seen on the left figure, the virtual
embedding optimization obviously provides separable virtual prototypes from
real individual clusters. The right figure illustrates that our synthesized
images enforce high intra-class variance. Synthesized images of real
individuals filling the gap in the cluster provide variance to the subjects.
Additionally, the cluster of generated images from virtual ID prototypes,
while staying separate from other subjects, demonstrates the effectiveness of
our framework for generating unique virtual human faces in high consistency.

Figure 6: Comparison of the generated virtual ID images with the conventional
methods [39, 2, 28] and our methods that are trained on CASIA-WebFace and
MS1M-V2. Each row shows samples from the same ID.

We compare the synthetic face images of virtual ID with the conventional
methods in Fig. 6. Each row is sampled from the same individual. Since
DigiFace [2] synthesizes virtual human faces using 3D digital modeling, it
fails to match the distribution of real face images in terms of appearance as
depicted in the figure. Although SynFace [39] produces facial images of high
quality and consistency, it has limitations in separability, leading to the
generation of a restricted set of unique individuals, fewer than 500 [28]. On
the other hand, DCFace [28] demonstrates different scenarios for the same
person, but struggles to maintain uniformity. Generated facial images using
VIGFace model demonstrates remarkable uniformity in generating consistent
virtual individuals, while also incorporating variations such as hair styles,
glasses, makeup, and age. Note that the important characteristic of a training
dataset to achieve a high-performance FR model is not the high quality of the
images themselves, but rather the high consistency and variance of the
generated facial images belonging to one person.

Fig. 7 illustrates the synthesized results of real ID trained on MS1M-V2. Note
that we report subjects which only contains few number (only 2) of images in
the original dataset. Since the lack of variance in the class introduces
considerable bias into the training process, increasing the variance within
the class is crucial. The results demonstrate that the proposed approach is
capable of producing diverse conditions even when working with a small number
of images without any help from external data.

Figure 7: Generated real ID images of our MS1M-V2 trained model. We note that
we report subjects with only two images in the original dataset. This shows
that our method can generate diverse conditions even with a limited number of
images.

####  5.2.1 Property of generated Dataset

To investigate the properties of the generated images, we measured the 1)
intra-class consistency, 2) intra-class variance, and 3) inter-class
seperability of generated images. The intra-class consistency indicates how
consistent the samples are in adhering to the label condition. Consequently,
the consistency of class $k$ ($\mathrm{CST}_{k}$) was measured as follows:

$\mathrm{CST}_{k}=\frac{1}{N^{2}}\sum_{j=1}^{N}\sum_{i=1}^{N}\frac{f(x_{i})f(x_{j})}{\|f(x_{i})\|\|f(x_{j})\|},$
(10)

where $N$ is the number of images in a class $k$. Higher intra-class
consistency means that the similarities between samples under the same subject
are high.

Figure 8: The normalized intra-class consistency ($\mathrm{CST}$) versus
inter-class separability ($\mathrm{SEP}$) graph of various methods. The sizes
of circles indicate the intra-class diversities ($\mathrm{DIV}$).

The intra-class diversity measures how various the conditions of samples are
under the same label condition. Various conditions, such as pose, low light,
and occlusions, make face recognition more challenging. Motivated by this
observation, we calculated the intra-class diversity of a class $k$
($\mathrm{DIV}_{k}$) using the variance of Face Image Quality Assessment
(FIQA) scores as follows:

$\mathrm{DIV}_{k}=\frac{1}{N}\sum_{i=1}^{N}(\operatorname{FIQA}(x_{i})-\mkern
1.5mu\overline{\mkern-1.5mu\operatorname{FIQA}(x_{i})\mkern-1.5mu}\mkern
1.5mu)^{2},$ (11) $\mkern
1.5mu\overline{\mkern-1.5mu\operatorname{FIQA}(x_{i})\mkern-1.5mu}\mkern
1.5mu=\frac{1}{N}\sum_{i=1}^{N}\operatorname{FIQA}(x_{i}),$ (12)

where $\operatorname{FIQA}(x_{i})$ indicates the SER-FIQ score [50] of the
generated image $x_{i}$. We also measured the inter-class separability to
assess the integrity of the dataset; in other words, to ensure that all
subjects in the dataset are unique. Inspired by [5], the inter-class
separability for a class $k$ ($\mathrm{SEP}_{k}$) is measured based on the
class center angular similarity (CCS) and the nearest negative class center
angular similarity (NNCCS) as follows:

$\mathrm{SEP}_{k}=\frac{1}{N}\sum_{i=1}^{N}\frac{\operatorname{CCS}_{x_{i}}}{\operatorname{NNCCS}_{x_{i}}},$
(13) $\operatorname{CCS}_{x_{i}}=\frac{f(x_{i})\mkern
1.5mu\overline{\mkern-1.5muf(x_{i})\mkern-1.5mu}\mkern
1.5mu}{\|f(x_{i})\|\|\mkern
1.5mu\overline{\mkern-1.5muf(x_{i})\mkern-1.5mu}\mkern
1.5mu\|},\operatorname{NNCCS}_{x_{i}}=\max_{j\in\left\\{{1,...,C}\right\\},j\neq
i}\frac{f(x_{i})\mkern 1.5mu\overline{\mkern-1.5muf(x_{j})\mkern-1.5mu}\mkern
1.5mu}{\|f(x_{i})\|\|\mkern
1.5mu\overline{\mkern-1.5muf(x_{j})\mkern-1.5mu}\mkern 1.5mu\|},$ (14)

where $\mkern 1.5mu\overline{\mkern-1.5muf(x_{i})\mkern-1.5mu}\mkern 1.5mu$
represents the class center obtained by averaging the embedding vectors of the
generated image $x_{i}$ under the same class. As shown in the formula, a high
$\mathrm{SEP}_{k}$ indicates that the generated images are closer to the
center than to the nearest negative class center. The scores,
$\mathrm{CST}_{k}$, $\mathrm{DIV}_{k}$, and $\mathrm{SEP}_{k}$, are derived
from a pretrained ArcFace model that was trained using the Glint-360K dataset.
Fig. 8 plots the average values of the properties for all classes. VIGFace
shows balanced scores in terms of intra-class consistency, intra-class
variance, and inter-class separability compared to other methods. DCFace, the
SOTA model that also utilizes a diffusion model, exhibits similar
separability, but lower consistency compared to VIGFace. DigiFace achieves the
highest consistency, due to its utilization of 3D rendering methods, but its
image style may differ significantly from real datasets, leading to lower
accuracy in real-world applications. SynFace shows similar consistency as ours
but has the lowest separability due to its mix-up generation method.

###  5.3 Evaluation

Training Dataset |  # of Images (classes $\times$ variations) | Accuracy on Benchmark Dataset |  Average Accuracy |  GR (%)
---|---|---|---|---
LFW |  CFP-FP |  CPLFW |  AgeDB |  CALFW |  |
CASIA-Webface (Real) | 0.49M ($\approx$10.5K$\times 47$) |  99.35 |  95.97 |  84.12 |  93.65 |  90.78 |  92.77 |  -
SynFace | 0.5M (10K$\times 50$) |  84.10 |  70.06 |  59.42 |  63.00 |  63.93 |  68.10 |  36.2
DigiFace | 1.2M (10K$\times 72$ \+ 100K$\times 5$) |  85.22 |  75.33 |  60.35 |  61.72 |  61.80 |  68.88 |  34.7
DCFace | 1.2M (20K$\times 50$ \+ 40K$\times 5$) |  98.42 |  84.53 |  76.67 |  89.78 |  88.95 |  87.67 |  5.8
VIGFace(S) | 0.5M (10K$\times 50$) |  96.60 |  86.66 |  75.03 |  82.50 |  83.42 |  84.84 |  9.3
VIGFace(B)-LT | 1.2M (20K$\times 50$ \+ 40K$\times 5$) |  98.30 |  90.60 |  79.88 |  88.63 |  87.50 |  88.98 |  4.3
VIGFace(B) | 1.2M (60K$\times 20$) |  98.45 |  92.07 |  80.15 |  89.82 |  88.43 |  89.78 |  3.3
VIGFace(L) | 2.4M (120K$\times 20$) |  98.65 |  93.47 |  83.68 |  90.55 |  90.10 |  91.29 |  1.6
VIGFace(H) | 4.2M (60K$\times 50$ \+ 60K$\times 20$) |  99.13 |  91.87 |  84.83 |  94.63 |  93.38 |  92.77 |  0.0

Table 2: FR benchmark results trained with various synthesized datasets. The
1:1 verification accuracies(%) of the famous five benchmarks are reported. We
specify our method according to the image and subject sizes as small(S),
base(B), large(L), and huge(H). All datasets are generated based on CASIA-
Webface, except for (H), which is based on MS1M-V2.

####  5.3.1 VIGFace as a Virtual Dataset

We compared the performance of the FR model trained with generated facial
images from virtual IDs using VIGFace with conventional methods: SynFace [39],
DigiFace [2], and DCFace [28]. For the experiment, we set the number of
virtual IDs to $60K$ and $120K$. To ensure a valid comparison, we establish
image counts of 0.5M and 1.2M, representing the scale of the real CASIA-
WebFace dataset and previous synthetic approaches, respectively. In Tab. 2, we
present the 1:1 verification accuracy (%) on five benchmarks [22, 45, 56, 36,
57]. The last column is calculated using the average performance of five
benchmarks to represent the FR performance gap from the FR model trained with
the real dataset, as suggested in [28]. Gap-to-Real (GR) is calculated as
$\mathrm{GR}=(\mathrm{Average}_{\text{real}}-\mathrm{Average}_{\text{syn}})/\mathrm{Average}_{\text{syn}}$.
GRs are calculated using the performance of CASIA-WebFace. From the table, it
is evident that DCFace achieves a superior performance among conventional
methods. However, VIGFace(B) shows comparable or slightly better performance
than the previous state-of-the-art method on the LFW, AgeDB, and CALFW
datasets, while demonstrating superior improvement on the CFP-FP and CPLFW
datasets. This indicates that our methods can generate virtual IDs with more
diverse pose variations while maintaining ID consistency. When the FR model
was trained using VIGFace(B), it achieved a GR of $3.3\%$, and a large set
VIGFace(L) achieved a GR of only $1.6\%$. This proves that the model trained
with VIGFace can achieve FR performance close to that of FR models trained
with a real dataset. Note that the FR model trained with VIGFace(H), _i.e_.
generated using MS1M-V2, achieves the same verification accuracy as the model
trained with the real CASIA-WebFace. This indicates that our virtual dataset
can completely replace the CASIA-WebFace dataset to train the FR model without
any privacy issues.

####  5.3.2 VIGFace for Data Augmentation

To demonstrate the efficacy of VIGFace as an augmentation framework, we
evaluate the accuracy of the FR model trained with both real and virtual ID
face images. Tab. 3 shows the performance change of the trained FR model
utilizing various combinations of datasets. As depicted in the table, the FR
model that was trained with a combination of real data and virtual images
produced by VIGFace exhibits enhanced performance across all benchmarks in
comparison to the model trained solely on real data. This contrasts with the
performance decline of the model using conventional synthetic datasets as an
augmentation method. The results of our experiment indicate that the proposed
method can accurately mirror the real facial data to blend and make synergy
with them. Consequently, VIGFace not only effectively solves the privacy
issue, but also can be used with real data for FR training to help achieve
better generalization and high performance, which conventional methods cannot.

Training Dataset |  # of Images | Accuracy on Benchmark Dataset |  Average Accuracy
---|---|---|---
LFW |  CFP-FP |  CPLFW |  AgeDB |  CALFW
Real | 0.49M |  99.35 |  95.97 |  84.12 |  93.65 |  90.78 | 92.77
$+$SynFace | 0.5M |  99.38 |  96.03 |  84.15 |  93.38 |  90.68 | 92.72 ($-$0.05)
$+$DCFace | 1.2M |  99.38 |  95.90 |  83.63 |  93.53 |  90.58 | 92.60 ($-$0.17)
$+$VIGFace | 1.2M |  99.50 |  96.93 |  86.20 |  94.78 |  92.05 | 93.89 ($+$1.12)

Table 3: FR benchmark results of FR models that trained using both of real and
synthetic datasets. The 1:1 verification accuracies (%) are reported for the
famous five benchmarks. We utilize VIGFace(B) in this experiment.

##  6 Conclusions

This paper presents a simple yet effective method for creating a synthetic
dataset that guarantees the unique virtual identities. Synthesized facial data
can serve as a solution for various challenges faced by traditional real
datasets, such as high expenses, inaccuracies and biases in labeling, and
concerns regarding privacy. To this end, we propose the Virtual Identity
Generation framework and demonstrate that it can generate realistic and
diverse facial images of virtual individuals, significantly narrowing the
performance gap with FR models trained on real data. Furthermore, the model
exhibits superior performance when trained on a combination of data from
VIGFace and existing real data compared to models trained solely on real data.
This confirms that our proposed method has potential as an effective
augmentation technique. We will publicly distribute the virtual face dataset
created by VIGFace and believe that this new virtual data will contribute to
resolving the privacy issues inherent in face recognition training dataset.

## References

  * [1] An, X., Zhu, X., Gao, Y., Xiao, Y., Zhao, Y., Feng, Z., Wu, L., Qin, B., Zhang, M., Zhang, D., et al.: Partial fc: Training 10 million identities on a single machine. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 1445–1449 (2021)
  * [2] Bae, G., de La Gorce, M., Baltrušaitis, T., Hewitt, C., Chen, D., Valentin, J., Cipolla, R., Shen, J.: Digiface-1m: 1 million digital face images for face recognition. In: Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision. pp. 3526–3535 (2023)
  * [3] Bao, F., Nie, S., Xue, K., Cao, Y., Li, C., Su, H., Zhu, J.: All are worth words: A vit backbone for diffusion models. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 22669–22679 (2023)
  * [4] Boutros, F., Damer, N., Kirchbuchner, F., Kuijper, A.: Elasticface: Elastic margin loss for deep face recognition. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 1578–1587 (2022)
  * [5] Boutros, F., Fang, M., Klemt, M., Fu, B., Damer, N.: Cr-fiqa: Face image quality assessment by learning sample relative classifiability. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). pp. 5836–5845 (June 2023)
  * [6] van Breugel, B., Kyono, T., Berrevoets, J., van der Schaar, M.: Decaf: Generating fair synthetic data using causally-aware generative networks. Advances in Neural Information Processing Systems 34, 22221–22233 (2021)
  * [7] Chen, T.: On the importance of noise scheduling for diffusion models. arXiv preprint arXiv:2301.10972 (2023)
  * [8] Cho, H., Park, H., Kim, I.J., Cho, J.: Data augmentation of backscatter x-ray images for deep learning-based automatic cargo inspection. Sensors (2021)
  * [9] Deng, J., Guo, J., Liu, T., Gong, M., Zafeiriou, S.: Sub-center arcface: Boosting face recognition by large-scale noisy web faces. In: Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XI 16. pp. 741–757. Springer (2020)
  * [10] Deng, J., Guo, J., Xue, N., Zafeiriou, S.: Arcface: Additive angular margin loss for deep face recognition. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 4690–4699 (2019)
  * [11] Deng, J., Guo, J., Yang, J., Lattas, A., Zafeiriou, S.: Variational prototype learning for deep face recognition. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 11906–11915 (2021)
  * [12] Deng, Y., Yang, J., Chen, D., Wen, F., Tong, X.: Disentangled and controllable face image generation via 3d imitative-contrastive learning. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 5154–5163 (2020)
  * [13] Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., et al.: An image is worth 16x16 words: Transformers for image recognition at scale. arxiv 2020. arXiv preprint arXiv:2010.11929 (2010)
  * [14] Dumoulin, V., Shlens, J., Kudlur, M.: A learned representation for artistic style. In: International Conference on Learning Representations (2016)
  * [15] Guo, Y., Zhang, L., Hu, Y., He, X., Gao, J.: Ms-celeb-1m: A dataset and benchmark for large-scale face recognition. In: Computer Vision–ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part III 14. pp. 87–102. Springer (2016)
  * [16] He, K., Zhang, X., Ren, S., Sun, J.: Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In: Proceedings of the IEEE international conference on computer vision. pp. 1026–1034 (2015)
  * [17] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 770–778 (2016)
  * [18] Ho, J., Salimans, T.: Classifier-free diffusion guidance. In: NeurIPS 2021 Workshop on Deep Generative Models and Downstream Applications (2021)
  * [19] Hong, J.H., Kim, H., Kim, M., Nam, G.P., Cho, J., Ko, H.S., Kim, I.J.: A 3d model-based approach for fitting masks to faces in the wild. In: 2021 IEEE International Conference on Image Processing (ICIP) (2021)
  * [20] Hu, J., Shen, L., Sun, G.: Squeeze-and-excitation networks. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 7132–7141 (2018)
  * [21] Huang, G., Mattar, M., Lee, H., Learned-Miller, E.: Learning to align from scratch. Advances in neural information processing systems 25 (2012)
  * [22] Huang, G.B., Mattar, M., Berg, T., Learned-Miller, E.: Labeled faces in the wild: A database forstudying face recognition in unconstrained environments. In: Workshop on faces in’Real-Life’Images: detection, alignment, and recognition (2008)
  * [23] Huang, Y., Wang, Y., Tai, Y., Liu, X., Shen, P., Li, S., Li, J., Huang, F.: Curricularface: adaptive curriculum learning loss for deep face recognition. In: proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 5901–5910 (2020)
  * [24] Hwang, H., Jang, C., Park, G., Cho, J., Kim, I.J.: Eldersim: A synthetic data generation platform for human action recognition in eldercare applications. IEEE Access (2023)
  * [25] Ioffe, S., Szegedy, C.: Batch normalization: Accelerating deep network training by reducing internal covariate shift. In: International conference on machine learning. pp. 448–456. pmlr (2015)
  * [26] Kemelmacher-Shlizerman, I., Seitz, S.M., Miller, D., Brossard, E.: The megaface benchmark: 1 million faces for recognition at scale. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 4873–4882 (2016)
  * [27] Kim, M., Jain, A.K., Liu, X.: Adaface: Quality adaptive margin for face recognition. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 18750–18759 (2022)
  * [28] Kim, M., Liu, F., Jain, A., Liu, X.: Dcface: Synthetic face generation with dual condition diffusion model. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 12715–12725 (2023)
  * [29] Kinga, D., Adam, J.B., et al.: A method for stochastic optimization. In: International conference on learning representations (ICLR). vol. 5, p. 6. San Diego, California; (2015)
  * [30] Liu, W., Wen, Y., Yu, Z., Li, M., Raj, B., Song, L.: Sphereface: Deep hypersphere embedding for face recognition. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 212–220 (2017)
  * [31] Liu, Y., Liu, X.: Spoof trace disentanglement for generic face anti-spoofing. IEEE Transactions on Pattern Analysis and Machine Intelligence 45(3), 3813–3830 (2022)
  * [32] Loshchilov, I., Hutter, F.: Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101 (2017)
  * [33] Van der Maaten, L., Hinton, G.: Visualizing data using t-sne. Journal of machine learning research 9(11) (2008)
  * [34] Meng, Q., Zhao, S., Huang, Z., Zhou, F.: Magface: A universal representation for face recognition and quality assessment. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 14225–14234 (2021)
  * [35] Miyato, T., Koyama, M.: cgans with projection discriminator. In: International Conference on Learning Representations (2018)
  * [36] Moschoglou, S., Papaioannou, A., Sagonas, C., Deng, J., Kotsia, I., Zafeiriou, S.: Agedb: the first manually collected, in-the-wild age database. In: proceedings of the IEEE conference on computer vision and pattern recognition workshops. pp. 51–59 (2017)
  * [37] Nech, A., Kemelmacher-Shlizerman, I.: Level playing field for million scale face recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. pp. 7044–7053 (2017)
  * [38] Preechakul, K., Chatthee, N., Wizadwongsa, S., Suwajanakorn, S.: Diffusion autoencoders: Toward a meaningful and decodable representation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 10619–10629 (2022)
  * [39] Qiu, H., Yu, B., Gong, D., Li, Z., Liu, W., Tao, D.: Synface: Face recognition with synthetic data. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 10880–10890 (2021)
  * [40] Ramesh, A., Dhariwal, P., Nichol, A., Chu, C., Chen, M.: Hierarchical text-conditional image generation with clip latents. arXiv preprint arXiv:2204.06125 1(2), 3 (2022)
  * [41] Rombach, R., Blattmann, A., Lorenz, D., Esser, P., Ommer, B.: High-resolution image synthesis with latent diffusion models. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 10684–10695 (2022)
  * [42] Saharia, C., Chan, W., Saxena, S., Li, L., Whang, J., Denton, E.L., Ghasemipour, K., Gontijo Lopes, R., Karagol Ayan, B., Salimans, T., et al.: Photorealistic text-to-image diffusion models with deep language understanding. Advances in Neural Information Processing Systems 35, 36479–36494 (2022)
  * [43] Sankaranarayanan, S., Alavi, A., Castillo, C.D., Chellappa, R.: Triplet probabilistic embedding for face verification and clustering. In: 2016 IEEE 8th international conference on biometrics theory, applications and systems (BTAS). pp. 1–8. IEEE (2016)
  * [44] Schroff, F., Kalenichenko, D., Philbin, J.: Facenet: A unified embedding for face recognition and clustering. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 815–823 (2015)
  * [45] Sengupta, S., Chen, J.C., Castillo, C., Patel, V.M., Chellappa, R., Jacobs, D.W.: Frontal to profile face verification in the wild. In: 2016 IEEE winter conference on applications of computer vision (WACV). pp. 1–9. IEEE (2016)
  * [46] Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556 (2014)
  * [47] Song, J., Meng, C., Ermon, S.: Denoising diffusion implicit models. arXiv preprint arXiv:2010.02502 (2020)
  * [48] Srivastava, N., Hinton, G., Krizhevsky, A., Sutskever, I., Salakhutdinov, R.: Dropout: a simple way to prevent neural networks from overfitting. The journal of machine learning research 15(1), 1929–1958 (2014)
  * [49] Sun, Y., Liu, Y., Liu, X., Li, Y., Chu, W.S.: Rethinking domain generalization for face anti-spoofing: Separability and alignment. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 24563–24574 (2023)
  * [50] Terhörst, P., Kolf, J.N., Damer, N., Kirchbuchner, F., Kuijper, A.: Ser-fiq: Unsupervised estimation of face image quality based on stochastic embedding robustness. In: CVPR. vol. 1, p. 2 (2020)
  * [51] Tremblay, J., Prakash, A., Acuna, D., Brophy, M., Jampani, V., Anil, C., To, T., Cameracci, E., Boochoon, S., Birchfield, S.: Training deep networks with synthetic data: Bridging the reality gap by domain randomization. In: Proceedings of the IEEE conference on computer vision and pattern recognition workshops. pp. 969–977 (2018)
  * [52] Van Noorden, R.: The ethical questions that haunt facial-recognition research. Nature 587(7834), 354–359 (2020)
  * [53] Wang, H., Wang, Y., Zhou, Z., Ji, X., Gong, D., Zhou, J., Li, Z., Liu, W.: Cosface: Large margin cosine loss for deep face recognition. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 5265–5274 (2018)
  * [54] Wen, Y., Zhang, K., Li, Z., Qiao, Y.: A discriminative feature learning approach for deep face recognition. In: Computer Vision–ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11–14, 2016, Proceedings, Part VII 14. pp. 499–515. Springer (2016)
  * [55] Yi, D., Lei, Z., Liao, S., Li, S.Z.: Learning face representation from scratch. arXiv preprint arXiv:1411.7923 (2014)
  * [56] Zheng, T., Deng, W.: Cross-pose lfw: A database for studying cross-pose face recognition in unconstrained environments. Beijing University of Posts and Telecommunications, Tech. Rep 5(7) (2018)
  * [57] Zheng, T., Deng, W., Hu, J.: Cross-age lfw: A database for studying cross-age face recognition in unconstrained environments. arXiv preprint arXiv:1708.08197 (2017)
  * [58] Zhu, Z., Huang, G., Deng, J., Ye, Y., Huang, J., Chen, X., Zhu, J., Yang, T., Lu, J., Du, D., et al.: Webface260m: A benchmark unveiling the power of million-scale deep face recognition. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 10492–10502 (2021)