# Generative One-Shot Face Recognition

Zhengming Ding, Yandong Guo, Lei Zhang, , and Yun Fu Z. M. Ding is with the
Department of Computer, Information and Technology Indiana University-Purdue
University Indianapolis, 420 University Blvd Indianapolis, IN 46202.
E-mail<EMAIL_ADDRESS>Yandong Guo is with XPeng Motors,
Email<EMAIL_ADDRESS>Lei Zhang is with Microsoft Research,
Email<EMAIL_ADDRESS>Y. Fu is with the Department of Electrical and
Computer Engineering and the College of Computer and Information Science,
Northeastern University, Boston, MA, 02115 USA.
E-mail<EMAIL_ADDRESS>
###### Abstract

One-shot face recognition measures the ability to identify persons with only
seeing them at one glance, and is a hallmark of human visual intelligence. It
is challenging for conventional machine learning approaches to mimic this way,
since limited data are hard to effectively represent the data variance. The
goal of one-shot face recognition is to learn a large-scale face recognizer,
which is capable to fight off the data imbalance challenge. In this paper, we
propose a novel generative adversarial one-shot face recognizer, attempting to
synthesize meaningful data for one-shot classes by adapting the data variances
from other normal classes. Specifically, we target at building a more
effective general face classifier for both normal persons and one-shot
persons. Technically, we design a new loss function by formulating knowledge
transfer generator and a general classifier into a unified framework. Such a
two-player minimax optimization can guide the generation of more effective
data, which effectively promote the underrepresented classes in the learned
model and lead to a remarkable improvement in face recognition performance. We
evaluate our proposed model on the MS-Celeb-1M one-shot learning benchmark
task, where we could recognize $94.98\%$ of the test images at the precision
of $99\%$ for the one-shot classes, keeping an overall Top1 accuracy at
$99.80\%$ for the normal classes. To the best of our knowledge, this is the
best performance among all the published methods using this benchmark task
with the same setup, including all the participants in the recent MS-Celeb-1M
challenge at ICCV 2017111http://www.msceleb.org/challenge2/2017.

###### Index Terms:

One-Shot Learning, Generative Adversarial Nets, Large-Scale Face Recognition

##  1 Introduction

One-shot face recognition is to recognize persons with only seeing them once
(Figure 1). This problem exists in many real applications. For example, in the
scenario of large-scale celebrity recognition, it naturally happens that some
celebrities only have one or very limited number of images available. Another
example is in the law enforcement scenario: it is usually the case that only
one image of the personal ID is available for the target person. The challenge
of one-shot face recognition lies in two parts.

_First of all_ , a representation model is needed to transfer the face image
into a discriminative feature domain. Although recent years have witnessed
great progresses in deep learning for visual recognition, computer vision
systems still lack the capability of learning visual concepts from just one or
a very few examples [1]. A typical solution is to leverage many images from a
different group of people (we call them base set and name the persons with
limited number of training images low-shot set), and train a representation
model using the images from the base set to extract face features for the
images in the low-shot set. Recently, there have been many research efforts
focusing on training representation models with good generalization capability
[2, 3, 4, 5], where face representation model is trained and tested across
different groups of persons. However, improving the generalization and
capability of face representation model is still an open problem which has
attracted substantial effort in the area. When the distributions of the face
dataset-A and face dataset-B are very different, the representation model
trained on dataset-A may not be discriminative enough on dataset-B. For
example, if the data used to train a representation model do not include
sufficient number of images for persons with a certain type of skin color, the
trained model usually suffer from lower accuracy for those persons.

Figure 1: Illustration of One-Shot Challenge, where the one-shot image in the
leftmost column is used for training and the rest images (in the right panel)
are the corresponding images for testing (partially selected from the test
set). With only one image for each person, the challenge is how to recognize
all these test images from hundreds of thousands of other testing images. More
detailed results are presented in the experimental results section.  Figure 2:
Illustration of one-shot face recognition problem with two phases. Stage 1:
Representation learning phase seeks general face visual knowledge through
training effective feature extractor using the base set. Stage 2: One-shot GAN
learning phase builds a general classifier to recognize persons in both the
base set and the one-shot set based on the deep features.

_Secondly_ , the challenge of one-shot face recognition comes from estimating
the partition for a given person in the feature space. A representation model
transfers the face images of the same person into a cluster of dots in the
feature space. To recognize all the faces for a given person, we need to
estimate the shape, size, and location of the partition for this person in the
feature space. However, with only one image (corresponding to one dot in the
feature space), it is not easy to accurately and reliably estimate the
distribution of the faces of the person to be recognized, which makes it
challenging to estimate the boundary of the partition for this person in the
feature space. A straightforward yet a bit over-simplified way to estimate the
partition for the one-shot person in the feature space is to assume each
person claims a hyper-sphere with equal size in the feature space, and then
use the $k$-nearest neighborhood ($k$-NN) classifier with a certain threshold
to recognize persons each having only one image available. However, many
recent works [6, 7, 8] demonstrate that directly using $k$-NN is often a sub-
optimal solution compared with other methods which can learn the partition
boundary in more informative way.

In order to better study the one-shot face recognition problem, there have
been some benchmark tasks designed. One typical example is MegaFace in [9]. In
this benchmarks task, one face image for a given celebrity is provided to
search for another face image for this celebrity from up-to one million
distractor images from regular persons. Since this task requires participants
to extract feature vectors for all the face images and use 1-NN method to
search for the image with the smallest distance, this task mainly focuses on
evaluating the first challenge of the one-shot face recognition: to learn a
discriminative face representation model. Recently, there has been another
benchmark task called MS-Celeb-1M: low-shot challenge [6] proposed. This task
focuses on one-shot learning in the large-scale face recognition scenario. The
task is to study, with these training images only, how to develop an algorithm
to recognize the persons in both data sets. The main focus of this task is the
recognition accuracy for persons in the low-shot set as it shows the one-shot
learning capability of a vision system, while also checking the recognition
accuracy for those in the base set to ensure no harm to their performance.

Most recently, one-shot learning has attracted great attentions, attempting to
make progress towards imparting this human ability to modern recognition
systems [10, 11, 12, 13, 6, 14, 15, 16]. Generally, there are two ways to
improve the low-shot face recognition performance. The first strategy is to
enhance the generalization and discriminative capability of representation
model. Examples include range loss [17], fisher face [18], center invariant
loss [19], marginal loss [20], sphere face [21], etc. The second strategy is
to improve the estimation of partitions in the feature space, which mainly
includes two lines, i.e., data augmentation and classifier adaptation. Along
the first line, some human-designed data generation strategies are adopted to
synthesize fake data for the low-shot/one-shot classes to boost the
classification ability. Edwards et. al proposed handling the one-shot
classification task by learning dataset statistics using the amortized
inference of a variational auto-encoder [22]. Hariharan et al. designed a
method for hallucinating additional examples for the data-starved novel
classes [10]. Mehrotra et al. presented an additional generator network based
on the Generative Adversarial Networks where the discriminator is the proposed
residual pairwise network [12]. For the second line, the idea is to adapt the
base classifier to the novel classifier through multi-layer transformations or
some specific loss to boost the classifier space of one-shot classes. Guo et
al. proposed a novel supervision loss named as Underrepresented-classes
Promotion (UP) loss term, which aligns the norms of the weight vectors of the
one-shot classes to those of the normal classes [6], which directly boosts the
one-shot classifier parameters without considering the difference of class
variance for different one-shot classes. However, human-designed generation
rules cannot learn the data distribution well to synthesize data to
effectively improve the recognition.

In this work, we address one-shot face recognition includes the following two
phases (Figure 2). The first phase is named as representation learning. In
this phase, we build face representation model using all the training images
from the base set. The second phase is called as one-shot learning. In this
phase, we train a multi-class classifier to recognize the persons in both base
set and one-shot set based on the representation model learned in phase one.
We design a generative one-shot learning model to improve the recognition
performance for the persons in the one-shot set. The core idea of our generate
model is to synthesize effective auxiliary data for the one-shot classes, and
thus we can span the feature space for the one-shot classes to facilitate the
one-shot face recognition. To our best knowledge, this is one of the first
works to explore the generative model in a general classifier learning for
one-shot challenge. Specifically, we mainly focus on the one-shot learning
stage, where we attempt to train a more powerful classifier for both base and
novel classes. Finally, we also provide several interesting and challenging
future directions for one-shot face recognition. The main contributions of our
paper are listed in three folds as follows:

  * •

We jointly incorporate generative adversarial networks in training a general
classifier for both base and novel classes. In detail, the generator attempts
to synthesize more effective fake data for the one-shot classes to enrich the
data space of one-shot classes, while the discriminator is built to guide the
face data generation to mimic the data variation of base classes and adapt to
generate novel classes.

  * •

We design generative adversarial networks in the feature domain which we first
obtain by training a deep ConvNet model on the base classes 222We train
generative model on feature domain instead of image domain for the following
reasons. Typically, image synthesis is a more challenging task than image
classification/recognition. Especially for the current generative models, it
is still an open problem to generate meaningful faces with high quality in
many cases [23, 24]. This might be the reason that we don’t find any existing
one-shot learning work using generative models to synthesize face images.
Furthermore, we joint our generative model and the deep architecture into a
unified model to seek more general feature extractor.b. More specifically, we
build the conditional generative adversarial networks with an auxiliary
classifier to augment more effective features and enhance the general
classifier learning for one-shot classes.

  * •

We evaluate our proposed model on a large-scale one-shot face dataset, and
achieve significant improvement in the one-shot classification with coverage
rate $94.98\%$ at the precision of $99\%$. Meanwhile, our model can still
achieve very appealing performance as Top1 accuracy of $99.80\%$ for the base
classes.

The rest sections of this paper are organized as follows. In Section 2, we
first review the progress of one-shot face recognition nowadays, later present
a brief discussion of the related works and highlight the difference between
zero-shot learning, transfer learning and one-shot learning. Then we propose
two-stage one-shot learning including representation learning and one-shot
face recognizer through generative learning in Section 3. Experimental
evaluations on large-scale face datasets are reported in Section 4, which is
followed by the conclusion in Section 5.

##  2 Related Work

In this section, we mainly review the current status of one-shot learning in
the literature.

In the general image recognition domain, the recent low-shot learning work
[10, 15] also attracts a lot of attentions. Their benchmark task is very
similar to one-shot face recognition but in the general image recognition
domain: the authors split the ImageNet data333http://www.image-net.org/ into
the base and low-shot (called novel in [10]) classes, and the target is to
recognize images from both the base and low-shot classes. Their solution is
quite different from ours since the domain is quite different. We will not
review their solution here due to the space constraint, but list results from
their solution as one of the comparisons in the experiments section.

Overall, one-shot learning is still an open problem. A natural source of
information comes from additional data via “data manufacturing” [25] in
various ways. In a broad sense, learning novel classes is addressed by
exploiting and transferring knowledge gained from familiar classes. This is to
imitate the human ability of adapting previously acquired experience when
recognizing novel classes. In the following, we will revisit different
categories of one-shot learning methods, including the most popular “general
feature learning”, classifier learning & adaption, and data augmentation.

###  2.1 Generalized Feature learning

Cross entropy with Softmax has demonstrated good performance in supervising
the face feature extraction model training. In order to further improve the
performance of representation learning, many methods have been proposed to add
extra loss terms or slightly modify the cross entropy loss (used together with
Softmax for multinomial logistic regression learning) to regularize the
representation learning in order to improve the feature discrimination and
generalization capability.

Among all these works, we consider the center loss [26] as one of the most
representative one (a similar idea published in [27] during the same time). In
[26], face features from the same class are encouraged to be close to their
corresponding class center (actually, approximation of the class center,
usually dynamically updated). By adding this loss term to the standard
Softmax, the authors obtain a better face feature representation model [26].
There are many other alternative methods, including the range loss in [17],
fisher face in [18], marginal loss in [20], sphere face in [21], etc. Each of
these methods has its own uniqueness and advantages under certain setup. Guo
et al. designed a different kind of loss term adding to the cross entropy loss
of the Softmax to improve the feature extraction performance [6]. Compared
with center loss in [26] or sphere face in [21] (these two are the most
similar ones), Guo et al. demonstrated that their proposed method has better
performance from the perspective of theoretical discussion and experimental
verification. Unfortunately, it is not very practical to compare all these
cost function design methods fairly and thoroughly, since these cost functions
were implemented with different networks structures, and trained on different
datasets. Sometimes parameter adjustment is critically required when the
training data is switched.

###  2.2 $k$-Nearest Neighbor & Softmax Classifier

After a good face feature extractor is obtained, the template-based method,
e.g., $k$-nearest neighborhood ($k$-NN) classifier, is widely used for face
identification these days. The advantages of $k$-NN is clear: no classifier
training is needed, and $k$-NN does not suffer much from imbalanced data, etc.
However, experiments in [8, 28, 6, 7] demonstrate that the accuracy of $k$-NN
with the large-scale face identification setup is usually lower than Softmax
Classifier, when the same feature extractor is used. Moreover, if we use all
the face images for every person in the gallery, the complexity is usually too
high for large scale recognition, and the gallery dataset needs to be very
clean to ensure high precision. If we do not keep all the images per person,
how to construct representer for each class is still an open problem.

As described above, Softmax classifier demonstrates overall higher accuracy
compared with $k$-NN in many previous publications. This is mainly because in
Softmax classifier, the weight vectors for each of the classes is estimated
using discriminant information from all the classes, while in the $k$-NN
setup, the query image only needs to be close enough to one local class to be
recognized. Moreover, after feature extraction, with Softmax classifier, the
computational complexity of estimating the persons’ identity is linear to the
number of persons, not the number of images in the gallery. However, the
standard Softmax classifier suffers from the imbalanced training data and has
poor performance with the low-shot classes even these classes are oversampled,
though the overall accuracy is higher than $k$-NN. Recently, some works
develop hybrid solutions by combining Softmax classifier and $k$-NN [28, 7]
and achieve promising results. In these work, when Softmax classifier does not
have high confidence (threshold tuning is needed), $k$-NN is used.

We solve this problem from a different perspective. Different from the hybrid
solution, our solution only has one Softmax classifier as the classifier so
that no threshold is needed to switch between classifiers. We boost the
performance of Softmax classifier by involving the generated data.

###  2.3 Classifier Adaptation

Another type of knowledge transfer focuses on modeling (hyper-)parameters that
are shared across domains, typically in the context of generative statistical
modeling [29, 30, 31]. Li et al. operated in a variational Bayesian framework
by incorporating previously learned classes into the prior and combining with
the likelihood to yield a new class posterior distribution [29, 32]. Gaussian
processes and hierarchical Bayesian models are also employed to allow
transferring in a non-parametric Bayesian way. Specifically, hierarchical
Bayesian program learning utilizes the principles of compositionality and
causality to build a probabilistic generative model of visual objects [33, 1].
In addition, adaptive SVM and its variants present SVM-based model adaptation
by combining classifiers learned on related categories [11, 34]. Wang et al.
assumed there exists a generic, category agnostic transformation from small-
sample models to the underlying large-sample models [35], and thus they
explored a novel learning to learn approach that leverages the knowledge
gained when learning models in large sample sets to facilitate recognizing
novel categories from few samples. Despite many notable successes, it is still
unclear what kind of underlying structures are shared across a wide variety of
categories and are useful for transfer.

###  2.4 Data Augmentation

Data augmentation is a straightforward way to boost the one-shot class
performance, since it could compensate the shortage of one-shot class samples
by synthesizing more data. However, it is the key to generate meaningful data
with enough data variance for one-shot classes. Along this line, Hariharan et
al. presented a way of “hallucinating” additional examples for one-shot (low-
shot) classes by transferring modes of variation from the base classes [10].
Their experiments demonstrated those additional examples improve the one-shot
top-5 accuracy on low-shot classes while also maintaining accuracy on the base
classes. Note that Hariharan et al. adopted some human-designed rules to
augment the data space for low-shot classes, which is very complexed and not
easily spread in real-world applications [10].

Most recently, generative models [36, 23, 37] are exploited to synthesize more
training data for one-shot classes by automatically capturing the data
variance from base classes. Specifically, Rezende et al. developed a class of
sequential generative models by combining the representational power of deep
learning with the inferential power of Bayesian reasoning, which is among the
state-of-the art in density estimation and image generation [38]. Choe et al.
adapted a generator to increase the size of training dataset, which includes a
base set, a widely available dataset, and a novel set, a given limited
dataset, while adopting transfer learning as a backend [39]. Mehrotra et al.
proposed a deep residual network with an additional generator network that
allows the efficient computation of this more expressive pairwise similarity
objective [12]. Our proposed generative model also belongs to this category.
The key difference from previous work is we exploit data generation in feature
domain instead of image domain by jointly seeking a general classifier.
Moreover, we involve the class center and class variance to synthesize more
efficient data. Moreover, this work is our previous conference extension [40].
In this journal extension, we review more recent progress on one-shot
learning. Then, we improve our representation learning model with WFB loss
(Section 3.2), in term of the feature discrimination and generalization
capability. The newly proposed WFB loss encourages the face features belong to
the same class to have similar direction as their associated classification
weight vectors. Moreover, we further enhance our generative model by
incorporating the data variance of base classes. Our generator can better
mimic the human cognitive by transferring their previous knowledge. Here the
previous knowledge denotes the data variance from base classes.

##  3 The Proposed Algorithm

The objective of one-shot face recognition is to measure the recognition
ability of a model across classes with one training sample. Specifically, the
model is trained on labeled training data with two sets without identity
overlap, i.e., base set (i.e., normal classes) $\\{X_{b},Y_{b}\\}$ with
$c_{b}$ classes and one-shot set $\\{X_{n},Y_{n}\\}$ with $c_{n}$ classes. The
goal is to build a general $c$-class recognizer $(c=c_{b}+c_{n})$. We address
one-shot face recognition includes the following two phases (Figure 2). The
first phase is named as representation learning. In this phase, we build face
representation model using all the training images from the base set. The
second phase is called as one-shot learning. In this phase, we train a multi-
class classifier to recognize the persons in both base set and one-shot set
based on the representation model learned in phase one. We design a generative
one-shot learning model to improve the recognition performance for the persons
in the one-shot set.

###  3.1 Motivation

Figure 3: Illustration of generative model to synthesize more samples for one-
shot classes then update decision boundary bias. (a) in the beginning, we have
one training sample for one-shot class while many samples for base classes,
thus the classifier would be dominated by the bias. (b) we explore our
generate model to synthesize more samples for one-shot class. (c) with the
augmentation of feature space for one-shot classes, the classifier also be
updated with its one-shot classifier space enlarged.

One-shot face recognition is challenging due to limited samples during model
training, while general deep frameworks treat base and one-shot class equally,
which leads to biased updates of the recognition model. Thus, it is essential
to generate more effective data to improve the ability of the general
classifier. Traditional data augmentation strategies [10] only adopt human
designed rules to generate more data for the one-shot classes. Hence, the
enhancement to the classifier is limited. Another challenge is that it usually
hurts the base classification when we try to improve the classification
ability for one-shot classes. That is, the learned classifier is impractical
in real-world applications when dealing with a general face recognition
problem. Hence, it is essential to balance these two sets.

Moreover, generative models are very popular due to its promising ability to
synthesize effective data automatically, which are similar to the real data
with a guidance from a discriminator. While for one-shot face recognition, it
is essential to generate effective data with large variations for the one-shot
classes in order to span their classifier space. Generally, data from the base
classes have large within-class variations, and thus, it is helpful to adapt
the variations of base classes to the one-shot classes during data generation.
To generate more meaningful data for one-shot classes, we jointly seek a
general classifier with the input of real data and fake data. Such a joint
learning framework could benefit generating meaningful data and improving the
classification ability, specially updating the decision boundary of the
classifier shown in Figure 3, where more meaningful data are augmented to
enlarge the feature space then updating the classifier boundary.

###  3.2 Representation Learning

To learn more effective feature representation, we design our face
representation model with supervised learning framework considering persons’
identities as class labels. Specifically, we propose the loss function as
follows:

$\mathcal{L}=\mathcal{L}_{s}+\lambda\mathcal{L}_{a}\,,$ (1)

where $\mathcal{L}_{s}$ is the standard cross-entropy loss used for the
Softmax layer, while $\mathcal{L}_{a}$ is the newly proposed loss used to
improve the feature discrimination and generalization capability. $\lambda$ is
the trade-off parameter between two loss functions. More specifically, we
recap the first term, cross-entropy $\mathcal{L}_{s}$ as follows:

$\displaystyle\mathcal{L}_{s}$
$\displaystyle=-\sum\limits_{i}\sum\limits_{k}t_{k,i}\log
p_{k}(\emph{x}_{i}),$ (2)

where $t_{k,i}\in\\{0,1\\}$ is the ground truth label indicating whether the
$i$-th image belongs to the $k$-th class, and the term $p_{k}(\emph{x}_{i})$
is the estimated probability that the image $\emph{x}_{i}$ belongs to the
$k$-th class, defined as,

$p_{k}(\emph{x}_{n})=\dfrac{\exp\big{(}\mathbf{{w}}_{k}^{\top}{\boldsymbol{\phi}(\emph{x}_{i})}\big{)}}{\sum\limits_{i}\exp\big{(}\mathbf{w}_{k}^{\top}{\boldsymbol{\phi}(\emph{x}_{i})}\big{)}}\,,$
(3)

where $\mathbf{w}_{k}$ is the weight vector for the $k$-th class, and
$\boldsymbol{\phi}(\cdot)$ denotes the feature extractor for image
$\emph{x}_{n}$. Note that in all of our experiments, we always set the bias
term $b_{k}=0$. We choose the standard residual network with 34 layers
(ResNet-34) [41] as our feature extractor $\boldsymbol{\phi}(\cdot)$ using the
last pooling layer as the face representation. ResNet-34 is used due to its
good trade-off between prediction accuracy and model complexity, yet our
method is general enough to be extended to deeper network structures for even
better performance. We have conducted comprehensive experiments and found that
removing the bias term from the standard Softmax layer in deep convolutional
neural network does not affect the performance. However, it is worth noting
that this leads to a much better understanding of the geometry property of the
classification space.

The second term $\mathcal{L}_{a}$ in the cost function (Eq. (1)) is defined as

$\displaystyle\mathbf{w}^{\prime}_{k}$ $\displaystyle\leftarrow\mathbf{w}_{k}$
(4) $\displaystyle\mathcal{L}_{a}$ $\displaystyle=-\sum_{k}\sum_{i\in
C_{k}}\frac{\mathbf{w}_{k}^{{}^{\prime}\top}\boldsymbol{\phi}(\emph{x}_{i})}{\|\mathbf{w}^{\prime}\|_{2}\|\boldsymbol{\phi}(\emph{x}_{i})\|_{2}}\,,$
(5)

where we set the parameter vector $\mathbf{w}^{\prime}_{k}$ to be equal to the
weight vector $\mathbf{w}_{k}$. This loss term encourages the face features
belong to the same class to have similar direction as their associated
classification weight vector $\mathbf{w}_{k}^{\top}$. We name this loss term
as Weights-guided Feature vector Bundling (WFB). Calculating the derivative
with respect to $\boldsymbol{\phi}(\emph{x}_{i})$, we have

$\dfrac{\partial\mathcal{L}_{a}}{\partial\boldsymbol{\phi}(\emph{x}_{i})}=\frac{1}{\|\boldsymbol{\phi}(\emph{x}_{i})\|_{2}}\left(\frac{\mathbf{w}_{k}^{{}^{\prime}\top}}{\|\mathbf{w}^{\prime}_{k}\|_{2}}-\frac{\boldsymbol{\phi}(\emph{x}_{i})^{\top}\cos\theta_{i,k}}{\|\boldsymbol{\phi}(\emph{x}_{i})\|_{2}}\right)\,,$
(6)

where $\theta_{i,k}$ is the angle between $\mathbf{w}^{\prime}_{k}$ and
$\boldsymbol{\phi}(x_{i})$. Note that $\mathbf{w}^{\prime}_{k}$ in this term
is the parameter copied from $\mathbf{w}_{k}$, so there is no derivative to
$\mathbf{w}^{\prime}_{k}$. For experiment ablation purpose, we also tried to
back propagate the derivative of $\mathbf{w}_{k}$, but did not observe better
results.

To further understand Eq. (1), without loss of generality, we discuss the
decision hyperplane between any two adjacent classes. Note we set all the bias
terms $b_{k}$ and $b_{j}$ to $0$ throughout the paper. With this setup, we
apply Eq. (3) to both the $k$-th class and the $j$-th class to determine the
decision hyperplane between the two classes (note we do not have bias terms
throughout our paper):

$\frac{p_{j}(x)}{p_{k}(x)}=\frac{\exp(\mathbf{{w}}_{j}^{\top}{\boldsymbol{\phi}(\emph{x})})}{\exp(\mathbf{{w}}_{k}^{\top}{\boldsymbol{\phi}(\emph{x})})}=\exp[(\mathbf{{w}}_{j}-\mathbf{{w}}_{k})^{\top}\boldsymbol{\phi}(\emph{x})]$
(7)

(a) $\|\mathbf{w}_{k}\|_{2}=\|\mathbf{w}_{j}\|_{2}$ (b)
$\|\mathbf{w}_{k}\|_{2}<\|\mathbf{w}_{j}\|_{2}$

Figure 4: Relationship between the norm of $\mathbf{w}_{k}$ and the volume
size of the partition for the $k$-th class. The dash line represents the
hyper-plane (perpendicular to $\mathbf{w}_{j}-\mathbf{w}_{k}$) which separates
the two adjacent classes. As shown, when the norm of $\mathbf{w}_{k}$
decreases, the $k$-th class tends to possess a smaller volume size in the
feature space.

As shown in Figure 4, the hyperplane to separate two adjacent classes $k$ and
$j$ is perpendicular to the vector $\mathbf{w}_{j}-\mathbf{w}_{k}$. When the
norm of $\mathbf{w}_{k}$ gets decreased, this hyperplane is pushed towards the
$k$-th class, and the volume for the $k$-th class also gets decreased. As this
property holds for any two classes, we can clearly see the connection of the
norm of a weight vector and the volume size of its corresponding partition
space in the feature space.

Discussion: There have been a lot of effort in integrating extra terms with
cross-entropy loss to improve the feature generalization ability. The most
similar version is the center loss in [26], also known as the dense loss in
[27] published during the same time. In center loss, the extra term is defined
as

$\mathcal{L}_{c}=-\sum_{k}\sum_{i\in
C_{k}}\|\mathbf{c}_{k}-\boldsymbol{\phi}(\emph{x}_{i})\|_{2}^{2}\,,$ (8)

where $\mathbf{c}_{k}$ is defined as the class center (might be dynamically
updated as the approximation of the true class center due to implementation
cost).

Our method is different from center loss from two perspectives. First,
minimizing the cost function (Eq. (8)) may lead to two consequences. While it
helps reduce the discrepancy between $\boldsymbol{\phi}(\emph{x}_{i})$ and its
associated center $\mathbf{c}_{k}$, it also reduces the norms of
$\boldsymbol{\phi}(\emph{x}_{i})$ and $\mathbf{c}_{k}$. The second consequence
is usually not good as it may hurt the classification performance. We did
observe in our experiment that over training with center loss would lead to
features with too small norms and worse performance compared with not using
center loss (also reported in [26]). On the contrary, our loss term only
considers the angular between $\boldsymbol{\phi}(\emph{x}_{i})$ and
$\mathbf{w}^{\prime}_{k}$, and will not affect the norm of the feature. In our
experiment section, we demonstrate that our method is not sensitive to the
parameter tuning.

Second, please note that we use the weight vector in Softmax $\mathbf{w}_{k}$
to represent the classification center, while in (8), the variable
$\mathbf{c}_{k}$ is the class center. The major difference is that
$\mathbf{w}_{k}$ is updated (naturally happens during minimizing
$\partial\mathcal{L}_{c}$) using not only the information from the $k$-th
class, but also the information from the other classes. In contrast,
$\mathbf{c}_{k}$ is updated only using the information from the $k$-th class
(calculated separately). More specifically, according to the derivative of the
cross-entropy loss in Eq. (2),

$\dfrac{\partial\mathcal{L}_{s}}{\partial\mathbf{w}_{k}}=\sum_{i}(p_{k}(\emph{x}_{i})-t_{k,i}){\boldsymbol{\phi}(\emph{x}_{i})}\,,$
(9)

the direction of $\mathbf{w}_{k}$ is close to the direction of the face
features in the $k$-th class, and being pushed far away from the directions of
the face features not in the $k$-th class.

###  3.3 Generative One-Shot Learning

Conventional generative adversarial network (GAN) models attempt to synthesize
fake data in the image space [36]. Many kinds of improved variants incorporate
class labels or latent information to obtain class conditional samples [23,
42, 43]. To seek consistent distribution between real and synthesized data is
a common problem to all the GAN-based approaches. In our proposed generative
one-shot learning, we target at augmenting one-shot classes in the feature
space. A general believe is that it is relatively easier to make the two
distributions more consistent in the feature space rather than in the image
space.

Given random noise $\emph{z}\in\mathbb{R}^{d_{z}}$, real feature
$\boldsymbol{\phi}(\emph{x})\in\mathbb{R}^{d_{x}}$ with its one-hot label
$\emph{y}\in\mathbb{R}^{d_{y}}$. In the generator, the prior input noise
$p_{z}(\emph{z})$, and one-hot label y are combined in joint hidden
representation, and the adversarial training framework allows for considerable
flexibility in how this hidden representation is composed. In the
discriminator, $\boldsymbol{\phi}(\emph{x})$ and y are presented as inputs and
to a discriminative function, where $\boldsymbol{\phi}(\cdot)$ denotes the
feature extractor for sample x. The objective function of a two-player minimax
game is as follows:

$\begin{array}[]{c}\mathcal{L}^{f}_{d}=\mathbb{E}[\mathrm{log}(1-D(G(\emph{z}|\emph{y})))]\\\
\\\
\mathcal{L}^{r}_{d}=\mathbb{E}[\mathrm{log}(D(\boldsymbol{\phi}(\emph{x})))]\end{array}$
(10)

where the generator aims to make the generated features similar to real
features, attempting to minimize $\mathcal{L}^{f}_{d}$; the discriminator aims
to differentiate the real and fake features by maximizing
$\mathcal{L}^{r}_{d}+\mathcal{L}^{f}_{d}$.

Figure 5: Illustration of generative one-shot face recognizer, where $z$ is
the random noise vector, y is the one-hot label,
$\emph{x}_{r}=\boldsymbol{\phi}(\emph{x})$ is the real feature, while
$\emph{x}_{f}$ is the generated fake feature. $G(\cdot)$ is the generator with
the input of random noise z, original real feature $\emph{x}_{r}$, and one-hot
label y. The output of generator with normalization $N(\cdot)$ will achieve
the fake feature $\emph{x}_{f}$. $D(\cdot)$ is the discriminator which aims to
differentiate the real and fake features, while $C(\cdot)$ is the a general
multi-class classifier.

####  3.3.1 Knowledge Transferable Generator

As we know, the above conditional generative model does not involve the
variance of base classes, which may not effectively adapt the knowledge from
base classes to one-shot classes. We assume that base and one-shot classes
share the same true distribution. While instances in the base class well-
sample the true distribution, the one-shot instances are under sampling. By
transferring the intra-class variance of a base class to a one-shot class, the
feature distribution within the one-shot class can be enriched to be similar
to base classes. The task then is to model the intra-class variance. By
assuming the variance to be a multi-variate Gaussian, Cao et al. proposed a
joint Bayesian model that takes the facial feature as the identity feature
plus the intra-class variance [44]. From the observation of researchers, the
identity feature is well-approximated by the feature center. Thus, the facial
feature is represented as:
$\boldsymbol{\phi}(\emph{x}_{i})=\mathbf{c}_{i}+v_{i}$, where $c_{i}$ is
$i$-th class feature center, $\boldsymbol{\phi}(\emph{x}_{i})$ is one
arbitrary sample in the class while $v_{i}$ represents the variance from
arbitrary feature to the class center.

Different from Cao’s assumption for the variance [44], we completely rely on
the data variance from the base classes. That follows the human cognitive
process that we could adapt the previous knowledge (data variance from base
classes) to learn new things (one-shot classes). Here we are not to model the
variance across all the classes but rather to model the one-shot class
variance. Given the observed feature distribution in base classes, we seek a
parameterization to transfer the variance to the one-shot classes.
Theoretically, any complete space decomposition method is valid here. In this
paper we incorporate such knowledge transfer into the generator $G(\cdot)$ to
simulate for effective features for one-shot classes.

Specifically, we design the generator $G(\cdot)$ with the input of random
noise $\emph{z}\in\mathbb{R}^{d_{z}}$, intra-classes variance of base classes
$V\in\mathbb{R}^{d\times{d}}$ (later we will discuss how to obtain this
matrix), and one-hot label $\emph{y}\in\mathbb{R}^{c}$.

$\begin{array}[]{rl}G(\emph{z}|V,\emph{y})&=f_{1}(W_{g}\begin{bmatrix}\emph{z}\\\
\emph{y}\end{bmatrix}+Vv_{i})\\\
&=f_{1}([W_{z},W_{y}]\begin{bmatrix}\emph{z}\\\
\emph{y}\end{bmatrix}+Vv_{i})\\\
&=f_{1}(W_{z}\emph{z}+W_{y}\emph{y}+Vv_{i}),\end{array}$ (11)

where $W_{g}=[W_{z},W_{y}]$ while $W_{z}\in\mathbb{R}^{d\times{d_{z}}}$ and
$W_{y}\in\mathbb{R}^{d\times{c}}$. $v_{i}\in\mathbb{R}^{d}$ is a coefficient
vector to find the most related intra-class variances from the base class
($V$). To go deeper, we consider the variance of ons-shot classes could be
represented by its nearby base classes. And $f_{1}(\cdot)$ is the element-wise
activation function, e.g., ReLU function or Sigmoid function.

Remark: For the generator $G(\cdot)$, we attempt to synthesize meaningful data
to augment the one-shot classes. The goal is to span the feature space of one-
shot classes around its center features. Generally, we can calculate the mean
feature of base classes as their centers, while the one-shot features for
novel classes are usually not the centers and may be far away from their
centers. From Eq. (11), we notice there are three parts used to generate the
fake features. There two parts $W_{z}\emph{z}+W_{y}\emph{y}$ are from the
conventional conditional generator [23], which aims to seek two projections,
one is for random noise and the other for conditional one-hot label. The third
part $Vv_{i}$ follows a dictionary-based reconstruction format, that is, we
hope to select the most relevant data variances from the base classes to
synthesize one-shot feature. That is, we consider there are $d$ bases for
class variance which can be well captured from base classes. Thus, the class
variance for one-shot classes can be represented the combination of the bases
in specific coefficients. On one hand, this term can adapt the knowledge of
class variance from base classes to one-shot classes to synthesize meaningful
data for one-shot classes. On the other hand, this term is able to better
estimate the one-shot class centers since we only have one training sample in
advanced so that we cannot obtain good one-shot class centers in the
beginning. For simplicity, we adopt Principal Component Analysis (PCA) to
parametrize the intra-class variance of regular classes, i.e., we first build
a matrix
$\mathcal{V}=[\boldsymbol{\phi}(\emph{x}_{j}^{i})-c_{i}]_{j,i}\in\mathbb{R}^{d\times{n_{b}}}$
($d\ll n_{b}$), then we select $d$ eigen-vectors with large eigen-values to
represent the dictionary $V$.

However, our reconstruction term $Vv_{i}$ makes the sample dependent when
optimizing $v_{i}$. Thus, we explore a projective dictionary to approximate
$v_{i}$ with $W_{v}\boldsymbol{\phi}(\emph{x}_{i})$. In this way, we only need
to optimize a shared projective dictionary $W_{v}\in\mathbb{R}^{d\times{d}}$
for all classes. Hence, the generator can be reformulated as:

$\begin{array}[]{c}G(\emph{z}|V,\boldsymbol{\phi}(\emph{x}),\emph{y})=f_{1}\Big{(}W_{z}\emph{z}+W_{y}\emph{y}+VW_{v}\boldsymbol{\phi}(\emph{x})\Big{)},\end{array}$
(12)

Moreover, we hope $W_{y}\emph{y}$ could keep the class center information,
while $W_{z}\emph{z}$ to compensate the residual information. Therefore, we
initialize $W_{y}$ with the class-center features, and then we would get its
class center by multiplying $W_{y}$ and its one-hot label y. Specifically, the
base part is initialized with the mean features of $c_{b}$ classes, while
novel part is initialized with the available one-shot features. Note that the
one-shot class centers are not the real centers, so that we hope the intra-
class variance part could facilitate to optimize the $W_{y}$, aseptically the
novel part. For random noise part, we initialize $W_{z}$ and z randomly. In
this way, we can capture the data variation within base classes and adapt to
generate more meaningful data for novel classes. Another thing is that the
scale of $W_{y}\emph{y}$ is the same as that of x, and thus, the scale would
be improved if we add a random part $W_{z}\emph{z}$. Therefore, we add a
normalization process to make the fake feature with the same scale to the real
feature. Specifically, we define our normalization process as

$N(G(\emph{z}|V,\boldsymbol{\phi}(\emph{x}),\emph{y}))=\dfrac{\alpha
G(\emph{z}|V,\boldsymbol{\phi}(\emph{x}),\emph{y})}{\|G(\emph{z}|V,\boldsymbol{\phi}(\emph{x}),\emph{y})\|_{2}},$
(13)

where $\alpha$ is the mean norm of real feature across all samples.

####  3.3.2 Joint Discriminator & Classifier

The generator $G(\cdot)$ attempts to augment more training data for one-shot
classes, which is guided by the discriminator $D(\cdot)$. The goal of
discriminator manages to differentiate the fake features and real features. In
this way, two players could compete with each other to synthesize more
effective data. Since we build the generator in the feature domain, we design
one-layer fully-connected network to build the discriminator as follows:

$D(\emph{x})=f_{2}(W_{d}\bar{\emph{x}}),$ (14)

where $W_{d}\in\mathbb{R}^{1\times{d}}$ and $f_{2}(\cdot)$ projects
$\bar{\emph{x}}$ to a scale between 0 and 1. $\bar{\emph{x}}$ can be the real
feature or generated feature. For real feature, the output of
$D(\bar{\emph{x}})$ tends to be 1, otherwise 0.

Simultaneously, we target at building a general classifier $C(\cdot)$ across
$c$ classes for both base classes and one-shot classes based on the real
feature and synthesized feature. Specifically, we adopt the standard Softmax
classifier with loss function $\mathcal{L}_{s}$ defined as (Eq. (2)).

To sum up, we propose our generative one-shot learning model by incorporating
generator, discriminator and classifier together into a unified framework.
Specifically, there are two players, i.e., $D(\cdot)+C(\cdot)$ and $G(\cdot)$.
For $D(\cdot)+C(\cdot)$, we train to maximize
$-\mathcal{L}_{s}+\mathcal{L}^{r}_{d}-\mathcal{L}^{f}_{d}$, while $G(\cdot)$
is trained to maximize $-\mathcal{L}_{s}+\mathcal{L}^{f}_{d}$.

Remark: For $C(\cdot)$, $W_{c}$ are the classifier parameters for both the
base and novel classes. We initialize the classifier parameters trained on the
base and novel dataset with the ResNet-34 deep features [41] (See detail in
experiments). As known to all, a deep model training on base classes with many
samples per class can achieve very promising results for base classes [6].
That is, the classifier parameters are good enough for base classes
recognition. The goal of one-shot learning is to improve the classifier
parameters for one-shot classes. Hence, we hope the base classifier parameters
to be similar to the pre-training one. We develop a square loss regularizer to
constrain the base classifier not far away from its initialized one. In this
way, not only can we update the classifier parameters for novel classes to
enhance the classification ability, but also relax the classifier space for
base classes, triggering the expansion of novel classifier space.

Implementation: Training with one-shot classes usually results in a biased
classifier. Our algorithm aims to correct this classifier bias by transferring
variances from base classes to one-shot classes. For the generative one-shot
learning model, we adopt the deep features from (ResNet-34) with loss (Eq.
(1)) as the input and set the learning rate as $10^{-4}$ with the optimizer as
Adam optimizer. We adopt leaky-relu and sigmoid activation functions for
$G(\cdot)$ and $D(\cdot)$, respectively. Since GANs can be solved as a minimax
optimization problem, we first constrain the generator to optimize the
discriminator, then fix the discriminator to update the generator. Thus, we
iteratively update two neural networks until the model converges. This
violation will help to correct the classifier bias and reshape the decision
boundary (Figure 3).

##  4 Experimental Results

In this section, we first introduce the one-shot face data as well as its
feature representation process. Then we provide the one-shot evaluations with
other comparisons to verify the effectiveness of our proposed model. Finally,
we go deep and show some phenomena of our model.

We first train a general face representation model with the training images in
the base set, and then train a multi-class classification model with the
training images in both the base and novel sets. We list the experimental
results in details in the following subsections.

###  4.1 One-Shot Face Dataset

The face dataset444http://www.msceleb.org/challenge2/2017 used here is sampled
from MS-Celeb-1M dataset [45]. In total, this dataset contains 21K people with
1.2M images, which is considerably larger than other publicly available
datasets except for the MS-Celeb-1M dataset. To evaluate the one-shot
challenge, we divide the dataset into base set ($20$K) parts, i.e., base set
(20K people) and novel set (1K people). Since we want to build a general
21K-class classifier for both the base and novel classes, we hope our
optimized classifier achieve promising performance on both sets, otherwise it
is meaningless in the real-world applications.

In the base set, there are 20K persons, each of which having 50-100 images for
training and 5 for test. In the novel set, there are 1000 persons, each with
one image for training and 10 for test. The experimental results in this paper
were obtained with 100K test images for the base set and 20K test images for
the novel set. We focus on the recognition performance in the novel set, while
monitoring the recognition performance in the base set to ensure that the
performance improvement in the novel set does not harm the performance in the
base set.

To recognize the test images for the persons in the novel set is a challenging
task. The one training image per person was randomly preselected, and the
selected image set includes images of low resolution, profile faces, and faces
with occlusions. The training images in the novel set show a large range of
variations in gender, race, ethnicity, age, camera quality (or evening
drawings), lighting, focus, pose, expressions, and many other parameters.

###  4.2 Face Representation Learning

Learning good feature is the foundation of one-shot face recognition task. In
order to evaluate the discrimination and generalization capability of our face
representation model, we leverage the LFW [46, 47] verification task, which is
to verify whether a given face pair (in total 6000) belongs to the same person
or not.

We train our face representation model (Eq. (1)) using the images in our base
set (already published to facilitate the research in the area, excluding
people in LFW by design) with ResNet-34 [41] with input faces’ resolution as
224$\times$224\. Specifically, we seek a 20K-class classifier using all the
training images of the 20K persons in the base set. There are about 50-100
images per person in the base set. The wrong labels in the base set are very
limited (less than 1% based on manual check). We crop and align face areas to
generate the training data555http://www.msceleb.org/download/lowshot. Our face
representation model is learned from predicting the 20K classes. We have tried
different network structures and adopted the standard residual network with 34
layers [41] due to its good trade-off between prediction accuracy and model
complexity. Features extracted from the last pooling layer are adopted as the
face representation (512 dimensions).

The verification accuracy with different models are listed in Table I. As
shown, for the loss function, we investigated the standard cross entropy,
cross entropy plus our WFB-loss term in Eq. (1), the center loss in [26], and
the sphere face loss in [21]. For the WFB-loss, we set $\lambda$ in Eq. (1) as
$0.1$. For the center loss, we tried different sets of parameters and found
the best performance could be achieved when the balancing coefficient was
$0.005$, as reported in the table. For the sphere face [21], we noticed this
paper very recently and only tried limited sets of parameters (there are four
parameters to be adjusted together). The parameters reported in the paper can
not make the network converge on our dataset. The only parameter set we found
to make the network converge leads to worse results, compared with the
standard cross-entropy loss. Due to time constraint, for the other methods, we
only report the results for some of them referring the numbers stated in the
published corresponding papers. Please note that these methods use different
datasets and different networks structures.

As shown in Table I, we obtain the face representation model with the cutting-
edge performance with the help of our WFB-loss term in Eq. (1). We regard our
model good enough to let us start to investigate the one-shot learning phase.
We also tried different values of $\lambda$ in Eq. (1) and found our method is
not sensitive to the choose of $\lambda$, shown in the Figure 6. Larger
$\lambda$ means stronger regularizer applied. Note $\lambda=0$ corresponds to
no WFB-loss applied. From the results, we found $\lambda=0.1$ could generate
better performance.

TABLE I:  LFW verification results obtained with models trained with our published base set. All the models use ResNet-34 [41] as the feature extractor. For the sphere face, please refer to our paper for explanation (fail to converge). Methods | Network | Accuracy
---|---|---
Cross entropy only | 1 | $98.88\%$
Center face [26] | 1 | $99.06\%$
Sphere face [21] | 1 | $-.--\%$
Cross entropy + WFB in Eq. (1) (ours) | 1 | $\mathbf{99.28}\%$
|  |
TABLE II:  For reference, LFW verification results (partially) reported in peer-reviewed publications. Different datasets and network structures were used. Methods | Dataset | Network | Accuracy
---|---|---|---
JB [44] | Public | – | $96.33\%$
Human | – | – | $97.53\%$
DeepFace[5] | Public | 1 | $97.27\%$
DeepID2,3 [48, 49] | Public | 200 | $99.53\%$
FaceNet [4] | Private | 1 | $99.63\%$
Center face [26] | Private | 1 | $99.28\%$
Center face [26] | Public | 1 | $99.05\%$
Sphere face [21] | Public | 1 | $99.42\%$
Our WFB in Eq. (1) | Public | 1 | 99.73%
|  |  |
Figure 6: LFW verification results obtained with different $\lambda$ for our
WFB in Eq. (4), where x-axis denotes the values of $\lambda$.

###  4.3 One-shot Face Recognition

In phase two, we train a $21,000$-class classifier to recognize the persons in
both the base set and the one-shot set. Since there is only one image per
person for training in the one-shot set, we repeat each sample in the one-shot
set for $100$ times through all the experiments in this section. In order to
test the performance, we apply this classifier with $120,000$ test images
consists of images from the base or one-shot set. We focus on the recognition
performance in the novel set while monitoring the recognition performance in
the base set to ensure that the performance improvement in the novel set does
not harm the performance in the base set.

To recognize the test images for the persons in the novel set is a challenging
task. The one training image per person was randomly preselected, and the
selected image set includes images of low resolution, profile faces, and faces
with occlusions. We provide more examples in the supplementary materials due
to space constraint. The training images in the novel set show a large range
of variations in gender, race, ethnicity, age, camera quality (or evening
drawings), lighting, focus, pose, expressions, and many other parameters.
Moreover, we applied de-duplication algorithms to ensure that the training
image is visually different from the test images, and the test images can
cover many different looks for a given person.

TABLE III: Coverage at Precisions = 99% and 99.9% on the one-shot set, where our generative model significantly improves the coverage at precision 99% and 99.9%. Method | C@P=99% |<EMAIL_ADDRESS>
---|---|---
Fixed-Feature | 25.65% | 0.89%
SGM [10] | 27.23% | 4.24%
Update Feature | 26.09% | 0.97%
Direct Train | 15.25% | 0.84%
Shrink Norm [6] | 32.58% | 2.11%
Equal Norm [6] | 32.56% | 5.18%
Up Term [6] | 77.48% | 47.53%
Hybrid [7] | $92.64\%$ | N/A
Doppelganger [50] | $73.86\%$ | N/A
Generation-based [39] | $61.21\%$ | N/A
Ours | 94.98% | 83.94%
|  |
Figure 7: Precision-Coverage curves of five methods on the one-shot set, where
our model achieves a very appealing coverage@precision=99%.

We compare with the following algorithms:

  * •

Fixed-Feature: updates the feature extractor and only train the Softmax
classifier with the feature extractor provided by phase one.

  * •

Updated Feature: fine-tunes the feature extractor simultaneously when we train
the Softmax classifier in phase two. The feature updating does not change the
recognizer’s performance too much.

  * •

SGM [10]: is known as squared gradient magnitude loss, is obtained by updating
the feature extractor during phase one using the feature shrinking method.

  * •

Shrink norm [6]: adopts $L_{2}$-norm to shrink classifier parameters, which is
one typical strategy to handle insufficient data problem efficiency.

  * •

Equal norm [6]: is a weight regularizer, which constrains the classifier
parameters of both novel and base classes to the same value.

  * •

UP Term [6]: is a weight regularizer, which only enforces the classifier
parameters of the novel classes to the same value.

All the methods are based on a 21K-class classifier (trained with different
methods). Note that we boost all the samples in the novel set for 100 times
for all the methods, since the largest number of samples per person in the
base set is about 100.

Figure 8: Top1 accuracy (%) of base set and novel set with different
iterations, where we notice that our model could significantly improve the
Top1 accuracy for the novel classes while keeping a very promising Top1
accuracy for the base classes. Figure 9: Face retrieval results, where row (a)
denotes the three challenges one-shot training faces, i.e., occlusion, sketch,
low-resolution. Row (b) represents the test images, while the the bottom three
rows show the recognized results of three models, i.e., (c) $k$-NN, (d)
Softmax, and (e) Our generative model.  Figure 10: Norm of the classifier
weight vector $w$ for each class in $W_{c}$. The x-axis is the class index.
The rightmost 1000 classes on the x-axis correspond to the persons in the
novel set. As shown in the figure, with more iterations from (a) to (f),
$\|w\|_{2}$ for the novel set tends to have similar values as that of the base
set (Green bounding box denotes the weights for one-shot classes). This
promotion introduces significant performance improvement.

The experimental results of our method and the alternative methods are listed
in Table III. We adopt coverage rate at precision $99\%$ and $99.9\%$ as our
evaluation metrics since this is the major requirement for a real recognizer
[6]. As shown in the table, our method significantly improves the recall at
precision $99\%$ and $99.9\%$ and achieves the best performance among all the
methods. Unless numbers reported by other papers (Hybrid, Doppelganger, and
Generation-based), the face feature extractor was trained with cross entropy
loss.

Compared with the Fixed-Feature, SGM method obtains around 2% improvements in
recall when precision is $99\%$, while 4% improvements when precision
requirement is $99.9\%$. The gain for face recognition by feature shrinking in
[10] is not as significant as that for general image. The reason might be that
the face feature is already a good representation for faces and the
representation learning is not a major bottleneck. Note that we did not apply
the feature hallucinating method as proposed in [10] for fair comparison and
to highlight the contribution of model learning, rather than data
augmentation. To couple the feature hallucinating method (may need to be
modified for face) is a good direction for the next step.

Our model significantly improves the one-shot classification, preserving base
classification at a very promising performance. Specifically, as shown in
Table III, our generative model improves the coverage@precision=$99\%$ and
<EMAIL_ADDRESS>significantly. Moreover, we notice that our model
can achieve the state-of-the-art performance without any external data by
comparing the competitors in the low-shot challenge
666http://www.msceleb.org/leaderboard/c2. This verifies our generative model
is able to synthesize very effective features to alleviate the one-shot
classification.

The coverage at precision $99\%$ on the base set obtained by using any
classifier-based methods in Table III is $100\%$. The Top1 accuracy on the
base set obtained by any of these classifier-based methods is
$99.80\pm{0.02}$%. Thus, we do not report them separately in the table. That
verifies that our generative model could synthesize meaningful one-shot
samples to boost classifier space for one-shot classes.

###  4.4 Face Retrieval Results

We select three typical one-shot training cases (Figure 9), i.e., low-
solution, sketch, occlusion, to quantitatively show the performance of
different models. We compare with $k$-nearest neighbor classifier ($k$-NN)
($k=1$), Softmax, and our one-shot generative model. All the models are input
with the pre-trained deep ResNet-34 features with our newly designed loss (Eq.
(1)).

From the results (Figure 9), we observe that our model can well handle these
three challenging cases and recognize these persons correctly. $k$-NN cannot
correctly recognize the testing images of these three persons, which results
from that the testing images are quite different from the one-shot training
image. Softmax can retrieve some correct ones, which shows more promising
results than $k$-NN. Hence, we consider $k$-NN is not suitable in one-shot
face classification in large-scale dataset. Our model can significantly handle
those three challenging cases, which results from the generation of effective
data in facilitating the classifier learning.

###  4.5 Property Analysis

First of all, we evaluate the Top1 accuracy of the base set and novel set with
the model optimization. From the results 8, we observe that the Top1 accuracy
of one-shot set is significantly improved from $77.01\%$ to $96.82\%$. This
shows that our model enhances the classification for one-shot classes by
spanning the feature space. We further notice that the classification accuracy
for the base set is hurt somehow, but very slightly. That demonstrates our
generative model can learn a good general classifier, which is much practical
in real-world scenarios.

Secondly, we present more information for the classifier to deeply understand
why our model can improve the one-shot classification. Specifically, we have a
$c$-class classifier, with each class weight vector $w$. Thus, we evaluate the
norm of each class weight vector to see the variations of these information.
From the results (Figure 10), we notice that from (a) to (f) with more
iterations’ optimization, the norms of the novel classes are triggered to
similar distribution as the base classes (f). Actually, (a) shows the results
of the initialized parameters obtained from Softmax trained on ResNet-34 deep
features. That is the reason we consider why our model significantly improves
the one-shot classification, since we boost the mean of novel classes to be
similar to base classes. Such phenomenon is also obtained in [6], where the
assume the norm of classifier weight is related to the classifier space.

##  5 Conclusions

In this paper, we proposed a generative framework for one-shot face
recognition, where we attempted to synthesize more effective augmented data
for one-shot classes by borrowing the data variation of base set.
Specifically, generative learning was jointly incorporated in the general
classifier training for both the base and novel classes. Thus, more effective
fake data were generated for the one-shot classes to enrich the data space of
one-shot classes. Furthermore, a discriminator was designed to guide the face
data generation to mimic the data variation of base classes and adapt to
generate novel classes. Experiments on a large-scale one-shot face benchmark
showed that our model could significantly improve the performance of one-shot
classification, while keeping the promising classification ability for the
base set. Studying how to generate data in image space will lead to another
topic – a trivial face image synthesis work cannot lead to a convincing
conclusion.

## References

  * [1] B. M. Lake, R. Salakhutdinov, and J. B. Tenenbaum, “Human-level concept learning through probabilistic program induction,” _Science_ , vol. 350, no. 6266, pp. 1332–1338, 2015.
  * [2] Y. Taigman, M. Yang, M. Ranzato, and L. Wolf, “Deepface: Closing the gap to human-level performance in face verification,” in _Proc. of IEEE Conf. on Computer Vision and Pattern Recognition_ , June 2014.
  * [3] ——, “Web-scale training for face identification.” in _Proc. of IEEE Conf. on Computer Vision and Pattern Recognition_. IEEE, 2015, pp. 2746–2754.
  * [4] F. Schroff, D. Kalenichenko, and J. Philbin, “Facenet: A unified embedding for face recognition and clustering,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2015, pp. 815–823.
  * [5] O. M. Parkhi, A. Vedaldi, and A. Zisserman, “Deep face recognition,” in _Proceedings of the British Machine Vision Conference (BMVC)_ , 2015.
  * [6] Y. Guo and L. Zhang, “One-shot face recognition by promoting underrepresented classes,” _arXiv preprint arXiv:1707.05574_ , 2017.
  * [7] Y. Wu, H. Liu, and Y. Fu, “Low-shot face recognition with hybrid classifiers,” in _The IEEE International Conference on Computer Vision (ICCV)_ , Oct 2017.
  * [8] Y. Wu, J. Li, Y. Kong, and Y. Fu, “Deep convolutional neural network with independent softmax for large scale face recognition,” in _Proceedings of the 2016 ACM on Multimedia Conference_. ACM, 2016, pp. 1063–1067.
  * [9] I. Kemelmacher-Shlizerman, S. Seitz, D. Miller, and E. Brossard, “The megaface benchmark: 1 million faces for recognition at scale,” _ArXiv e-prints_ , 2015.
  * [10] B. Hariharan and R. Girshick, “Low-shot visual object recognition,” _arXiv preprint arXiv:1606.02819_ , 2016.
  * [11] Y.-X. Wang and M. Hebert, “Learning from small sample sets by combining unsupervised meta-training with cnns,” in _Advances in Neural Information Processing Systems_ , 2016, pp. 244–252.
  * [12] A. Mehrotra and A. Dukkipati, “Generative adversarial residual pairwise networks for one shot learning,” _arXiv preprint arXiv:1703.08033_ , 2017\.
  * [13] S. Motiian, Q. Jones, S. Iranmanesh, and G. Doretto, “Few-shot adversarial domain adaptation,” in _Advances in Neural Information Processing Systems_ , 2017, pp. 6670–6680.
  * [14] S. Gidaris and N. Komodakis, “Dynamic few-shot visual learning without forgetting,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2018, pp. 4367–4375.
  * [15] Y.-X. Wang, R. Girshick, M. Hebert, and B. Hariharan, “Low-shot learning from imaginary data,” pp. 7278–7286, 2018.
  * [16] S. Bartunov and D. Vetrov, “Few-shot generative modelling with generative matching networks,” in _International Conference on Artificial Intelligence and Statistics_ , 2018, pp. 670–678.
  * [17] X. Zhang, Z. Fang, Y. Wen, Z. Li, and Y. Qiao, “Range loss for deep face recognition with long-tail,” in _Proc. of Int’l Conf. on Computer Vision_ , 2017.
  * [18] H. Hanselmann, S. Yan, and H. Ney, “Deep fisher faces,” 2017.
  * [19] Y. Wu, H. Liu, J. Li, and Y. Fu, “Deep face recognition with center invariant loss,” in _ACM Multimedia Workshop_ , 2017.
  * [20] J. Deng, J. Deng, Y. Zhou, and S. Zafeiriou, “Marginal loss for deep face recognition,” in _Proc. of IEEE Conf. on Computer Vision and Pattern Recognition_ , 2017.
  * [21] W. Liu, Y. Wen, Z. Yu, M. Li, B. Raj, and L. Song, “Sphereface: Deep hypersphere embedding for face recognition,” in _Proc. of IEEE Conf. on Computer Vision and Pattern Recognition_ , vol. abs/1704.08063, 2017.
  * [22] H. Edwards and A. Storkey, “Towards a neural statistician,” _arXiv preprint arXiv:1606.02185_ , 2016.
  * [23] M. Mirza and S. Osindero, “Conditional generative adversarial nets,” _arXiv preprint arXiv:1411.1784_ , 2014.
  * [24] X. Chen, Y. Duan, R. Houthooft, J. Schulman, I. Sutskever, and P. Abbeel, “Infogan: Interpretable representation learning by information maximizing generative adversarial nets,” in _Advances in Neural Information Processing Systems_ , 2016, pp. 2172–2180.
  * [25] E. Bart and S. Ullman, “Cross-generalization: Learning novel classes from a single example by feature replacement,” in _IEEE Conference on Computer Vision and Pattern Recognition_ , vol. 1. IEEE, 2005, pp. 672–679.
  * [26] Y. Wen, K. Zhang, Z. Li, and Y. Qiao, “A discriminative feature learning approach for deep face recognition,” in _European Conference on Computer Vision_. Springer, 2016, pp. 499–515.
  * [27] L. Pemula, “Low-shot visual recognition,” 2016, master Thesis at Virginia Polytechnic Institute and State University.
  * [28] Y. Xu, Y. Cheng, J. Zhao, Z. Wang, L. Xiong, K. Jayashree, H. Tamura, T. Kagaya, S. Shen, S. Pranata, J. Feng, and J. Xing, “High performance large scale face recognition with multi-cognition softmax and feature retrieval,” in _Proc. of Int’l Conf. on Computer Vision Workshop (ICCV-W)_ , Oct 2017.
  * [29] L. Fei-Fei, F. Rob, and P. Pietro, “A bayesian approach to unsupervised one-shot learning of object categories,” in _IEEE International Conference on Computer Vision_. IEEE, 2003, pp. 1134–1141.
  * [30] S.-I. Lee, V. Chatalbashev, D. Vickrey, and D. Koller, “Learning a meta-level prior for feature relevance from multiple related tasks,” in _Proceedings of the 24th international conference on Machine learning_. ACM, 2007, pp. 489–496.
  * [31] E. Rodner and J. Denzler, “One-shot learning of object categories using dependent gaussian processes,” in _Joint Pattern Recognition Symposium_. Springer, 2010, pp. 232–241.
  * [32] L. Fei-Fei, R. Fergus, and P. Perona, “One-shot learning of object categories,” _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , vol. 28, no. 4, pp. 594–611, 2006.
  * [33] B. M. Lake, R. R. Salakhutdinov, and J. Tenenbaum, “One-shot learning by inverting a compositional causal process,” in _Advances in neural information processing systems_ , 2013, pp. 2526–2534.
  * [34] T. Tommasi, F. Orabona, and B. Caputo, “Learning categories from few examples with multi model knowledge transfer,” _IEEE transactions on pattern analysis and machine intelligence_ , vol. 36, no. 5, pp. 928–941, 2014.
  * [35] Y.-X. Wang and M. Hebert, “Learning to learn: Model regression networks for easy small sample learning,” in _European Conference on Computer Vision_. Springer, 2016, pp. 616–634.
  * [36] I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio, “Generative adversarial nets,” in _Advances in neural information processing systems_ , 2014, pp. 2672–2680.
  * [37] Z. Dai, Z. Yang, F. Yang, W. W. Cohen, and R. R. Salakhutdinov, “Good semi-supervised learning that requires a bad gan,” in _Advances in Neural Information Processing Systems_ , 2017, pp. 6510–6520.
  * [38] D. Rezende, I. Danihelka, K. Gregor, D. Wierstra _et al._ , “One-shot generalization in deep generative models,” in _International Conference on Machine Learning_ , 2016, pp. 1521–1529.
  * [39] J. Choe, S. Park, K. Kim, J. H. Park, D. Kim, and H. Shim, “Face generation for low-shot learning using generative adversarial networks,” in _Proc. of Int’l Conf. on Computer Vision Workshop (ICCV-W)_ , 2017.
  * [40] Z. Ding, Y. Guo, L. Zhang, and Y. Fu, “One-shot face recognition via generative learning,” in _IEEE International Conference on Automatic Face and Gesture Recognition_ , vol. 1. IEEE, 2018, pp. 1–6.
  * [41] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in _Proc. of IEEE Conf. on Computer Vision and Pattern Recognition_ , 2016, pp. 770–778.
  * [42] A. v. d. Oord, N. Kalchbrenner, O. Vinyals, L. Espeholt, A. Graves, and K. Kavukcuoglu, “Conditional image generation with pixelcnn decoders,” _arXiv preprint arXiv:1606.05328_ , 2016.
  * [43] S. Reed, Z. Akata, X. Yan, L. Logeswaran, B. Schiele, and H. Lee, “Generative adversarial text to image synthesis,” _arXiv preprint arXiv:1605.05396_ , 2016.
  * [44] X. Cao, D. Wipf, F. Wen, G. Duan, and J. Sun, “A practical transfer learning algorithm for face verification,” in _IEEE International Conference on Computer Vision_. IEEE, 2013, pp. 3208–3215.
  * [45] Y. Guo, L. Zhang, Y. Hu, X. He, and J. Gao, “Ms-celeb-1m: A dataset and benchmark for large-scale face recognition,” in _European Conference on Computer Vision_. Springer, 2016, pp. 87–102.
  * [46] G. B. Huang, M. Ramesh, T. Berg, and E. Learned-Miller, “Labeled faces in the wild: A database for studying face recognition in unconstrained environments,” University of Massachusetts, Amherst, Tech. Rep. 07-49, October 2007.
  * [47] G. B. Huang and E. Learned-Miller, “Labeled faces in the wild: Updates and new reporting procedures,” University of Massachusetts, Amherst, Tech. Rep. UM-CS-2014-003, May 2014.
  * [48] Y. Sun, Y. Chen, X. Wang, and X. Tang, “Deep learning face representation by joint identification-verification,” in _Advances in Neural Information Processing Systems_ , ser. NIPS’14. MIT Press, 2014, pp. 1988–1996.
  * [49] Y. Sun, X. Wang, and X. Tang, “DeepID3: Face recognition with very deep neural networks,” _arXiv preprint arXiv:1502.00873_ , 2014.
  * [50] E. Smirnov, A. Melnikov, S. Novoselov, E. Luckyanets, and G. Lavrentyeva, “Doppelganger mining for face representation learning,” in _Proc. of Int’l Conf. on Computer Vision Workshop (ICCV-W)_ , 2017.