11institutetext:  Institute of AI for Health, Helmholtz Munich – German
Research Center for Environmental Health, Neuherberg, Germany 22institutetext:
Technical University of Munich, Munich, Germany 33institutetext: Computer
Aided Medical Procedures, Technical University of Munich, Munich, Germany
44institutetext: Red Blood Cell Research Group, Institute of Veterinary
Physiology, Vetsuisse Faculty and the Zurich Center for Integrative Human
Physiology, University of Zurich, Zurich, Switzerland 55institutetext: Clinic
for Diagnostic and Interventional Radiology, University Hospital Bonn, Germany
66institutetext: Helmholtz AI, Helmholtz Munich – German Research Center for
Environmental Health, Neuherberg, Germany 77institutetext: Faculty of
Informatics, Technical University Munich, Munich, Germany

# Anomaly-aware multiple instance learning for rare anemia disorder
classification

Salome Kazeminia 1122 Ario Sadafi 1133 Asya Makhro 44 Anna Bogdanova 44 Shadi
Albarqouni 556677 Carsten Marr ${}^{(\textrm{{\char 0\relax}})}$ 11

###### Abstract

Deep learning-based classification of rare anemia disorders is challenged by
the lack of training data and instance-level annotations. Multiple Instance
Learning (MIL) has shown to be an effective solution, yet it suffers from low
accuracy and limited explainability. Although the inclusion of attention
mechanisms has addressed these issues, their effectiveness highly depends on
the amount and diversity of cells in the training samples. Consequently, the
poor machine learning performance on rare anemia disorder classification from
blood samples remains unresolved. In this paper, we propose an interpretable
pooling method for MIL to address these limitations. By benefiting from
instance-level information of negative bags (i.e., homogeneous benign cells
from healthy individuals), our approach increases the contribution of
anomalous instances. We show that our strategy outperforms standard MIL
classification algorithms and provides a meaningful explanation behind its
decisions. Moreover, it can denote anomalous instances of rare blood diseases
that are not seen during the training phase.

###### Keywords:

Multiple instance learning Anomaly pooling Rare anemia disorder Red blood
cells

##  1 Introduction

The appearance of human red blood cells changes directly with their volume.
Typically red blood cells have a discoid concave shape, and as their volume
increases or decrease in physiological conditions, they shrivel into star-
shaped cells or swell into spherical shapes [9]. In hereditary hemolytic
anemias (HHAs), a cell’s membrane is not formed properly and cannot
accommodate volume changes, resulting in the formation of anomalous
morphologies. Membrane instability and reduced deformability are other factors
in a number of HHAs, such as hereditary spherocytosis, sickle cell disease,
and thalassemia, where improper membrane formation leads to irregularly formed
cells.

Tools for close monitoring of blood samples in HHA patients is crucial for
diagnosis [8], disease progression [4], and severity estimation [17]. However,
since most cells in a patient’s sample have no evident morphological feature
relevant to the HHA, identifying hallmark cells is a needle-in-the-hay-stack
search. Furthermore, presence of a few anomalous cells in a sample does not
necessarily link to an underlying condition making the diagnosis even more
challenging.

In addition, proper single cell annotation for supervised model training is a
cumbersome, expensive task and introduces intra-expert variablity. Thus, the
automated analysis of blood samples is a perfect problem for multiple instance
learning-based approaches. Multiple Instance Learning (MIL) is a weakly
supervised learning method, where only bag-level labels are used [3]. Notably,
a bag with a negative label does not contain any positive instance, while bags
with a positive label may contain negative instances. In other words, the
class of a bag depends on the existence of one or more class-related instance.
Thus, the learning signal, back-propagated to the instance-level feature
extractor may be very weak and ineffective. To address this problem, MIL-based
deep classifiers require a large number of training data to perform
promisingly [3]. Providing such data is particularly difficult in the case of
rare disease. A possible solution is the deployment of intelligent pooling
methods to magnify the learning signal back-propagated from informative
instances [14]. In basic MIL classifiers, when the bag contains a small number
of class-related instances, a typical max or average pooling cannot to provide
such magnification [14].

Another requirement for health AI applications is explainability. MIL
classifiers should thus be able to specify HHA-related structures of instances
anomalies. Recently, attention-based pooling mechanisms showed a good
performance in tackling both learning signal magnification and explainability
challenges [10, 14, 16]. However, attention mechanisms are prone to fail in
scenarios where only a few training samples are available, and consequently,
the algorithm can not identify relevant instances in the bags correctly. In
HHA classification, such a problem leads to noisy attention scores where lots
of disorder-relevant (positive) cells receive low attention, and the
distribution of attention on non-disorder (negative) cells would be non-
uniform. Here, a strategy accounting for the diversity of disorder-relevant
cells and data imbalance is required.

This paper introduces an anomaly-aware pooling method, which is able to
address limitations of the attention mechanism with an efficient amount of
training data. While instance-level information for positive bags is not
available, we exploit negative bag labels that apply to all instances of the
bag and estimate the distribution of negative instances using Bayesian
Gaussian mixture models. By measuring the distance of instances to this
distribution, we define an anomaly score that is used for pooling in addition
to learned attention scores. Our method’s performance is explainable by
specifying the contribution of instances in the final bag classification.
Also, our proposed anomaly score can specify anomalous instances that are rare
and haven’t been seen during the training. We make our code publicly available
at https://github.com/marrlab/Anomaly-aware-MIL .

##  2 Method

Our MIL classifier $f_{\theta}$ consists of an instance detector, a deep
instance feature extractor $h_{\psi}$, and an attention-anomaly pooling
mechanism that assigns a contribution weight to each instance. The input of
our classifier is a bag of instances $\\{I_{1},...,I_{N}\\}\in B$; the output
is a class $c_{i}$ from possible classes defined in $C$. In our target
application, bags are red blood cells from brieghtfield microscopic images of
blood samples, each containing between 12 to 45 cells (See Fig. 1).

Figure 1: Overview of our proposed anomaly-aware MIL classifier. A Mask R-CNN
outputs detected instance bounding boxes and basic instance features. Then, an
encoder maps those features to a latent representation. Dashed path: A GMM
distribution is fitted to instances from negative bags $B_{neg}$, and its
parameters are saved. Solid path: Attention scores $a_{n\in N}$ and anomaly
scores $d_{n\in N}$ are estimated and combined through a convolution layer to
generate pooling weights $p_{n\in N}$. Finally, a fully connected layer
predicts the bag label.

###  2.1 Instance-level feature extraction

####  2.1.1 Preprocessing.

Since instance-level information plays a crucial role in bag classification
and interpretation, we need to detect them and follow their footprint in the
final classification result. For instance detection, we prefer two-stage
object detection methods due to the higher accuracy of their performance in
comparison with one-stage methods [5]. As a preprocessing step, any two-stage
object detector can be used. We employ Mask R-CNN [6] with ResNet [7] backbone
since it provides an accuracy above $91\%$ in detecting instances in
microscopic images [5, 1]. In our setting, the instance detector takes the bag
$B$ and outputs a bounding box and a feature map for each instance, regardless
of their class. We use these feature maps as instances
$I_{n}\in\\{I_{1},...,I_{N}\\}=B$. Mask R-CNN extracts features of each
instance independently, regardless of its size, scale, and location. This
simple step is extremely beneficial in our MIL training as it avoids
challenges stemming from non-uniform information distributions of instances,
which is widely researched in the literature [12, 18, 2].

####  2.1.2 Instance representation.

Using a 4-layer convolutional encoder
$h_{\psi}:{\mathbb{R}}^{m}\rightarrow{\mathbb{R}}^{k}$, we map each instance
$I_{n}$ to a point $z_{n}$ in the $k$-dimensional representation space $Z$:

$z_{n}=h_{\psi}(I_{n}),\quad n\in N$ (1)

###  2.2 Anomaly-aware pooling

####  2.2.1 Negative distribution estimation.

We estimate the distribution of negative instances in our latent space and
detect anomalies by measuring their distance to this distribution. As a prior
assumption, we consider a Gaussian Mixture Model (GMM) probability
distribution on negative instance representations. Due to Bayes’ rule, the
posterior probability distribution would be a GMM as well. Therefore we fit a
GMM model $g_{\Gamma}$ on negative instance representations. To estimate
$\Gamma$, we use an Expectation-Maximization (EM) algorithm that miximizes the
likelihood $p(\Gamma|Z_{neg})$ given $Z_{neg}:\\{z_{j}|I_{j}\in B_{neg}\\}$

$p(\Gamma|Z_{neg})=\sum_{i}\tilde{\gamma_{i}}\mathcal{N}(\tilde{\mu_{i}},\tilde{\Sigma_{i}}),$
(2)

where $\tilde{\gamma}$, $\tilde{\mu_{i}}$, and $\tilde{\Sigma_{i}}$ are
estimated weight, mean, and covariance matrix of a Gaussian distribution,
characterizing the $i^{th}$ GMM component, respectively. When the algorithm
converges, we save the model’s parameters as a reference for anomaly scoring.

####  2.2.2 Anomaly scoring.

For anomaly scoring we measure the Mahalanobis distance between instances and
the $g_{\Gamma}$

$d_{n}=\sqrt{(z_{n}-\mu)^{T}\Sigma^{-1}(z_{n}-\mu)},$ (3)

where $\mu$ and $\Sigma$ are mean and covariance of the $g_{\gamma}$
distribution. We have chosen Mahalanobis distance over other distances since
it considers the distribution’s covariance in its distance measurement.

####  2.2.3 Attention.

The attention mechanism is a method to extract meaningful instances
contributing to the final decision of the model. Specially, in MIL it showed a
good performance in both classification and interpretation of deep networks
[10, 16, 13, 14, 19]. Our deep attention network $v_{\omega}$ estimates an
attention score $a_{n}$ for each instance $I_{n}$ as

$a_{n}=v_{\omega}(z_{n}).$ (4)

####  2.2.4 Pooling.

Our pooling mechanism combines attention $a_{n}$ and anomaly $d_{n}$ scores to
estimate a contribution weight for each instance and conclude the latent
representation of the bag:

$z_{B}=\sum_{n=0}^{N}(W_{D_{n}}d_{n}+W_{A_{n}}a_{z_{n}})z_{n}.$ (5)

Here $W_{D}$ and $W_{A}$ are respectively learned parameters, to specify the
intervention share of attention and anomaly scores. Finally, a fully connected
layer $f_{\theta}$ maps the $k$-dimensional representation $z_{B}$ to a class
$c_{i}$ (see Fig. 1). Both anomaly and attention scores of am instance are
independently calculated and fed to a 1×1 convolution to estimate the final
pooling weight of the corresponding instance. This makes our pooling method
permutation invariant.

###  2.3 Optimization

The functionality of $v_{\omega}$ and fitting $g_{\Gamma}$ on negative
instances highly depends on the encoded instances by $h_{\psi}$. To boost the
training procedure, we start with a single instance classifier (SIC)
$s_{\zeta}$ using noisy labels for learning [16]. In other words, we assign
the bag label to all of its instances and calculate a cross-entropy loss
($\mathrm{CE_{SIC}}$) to optimize the model in the first epochs. Then, by
increasing the epoch number $e$, we gradually decrease SIC’s impact on the
training and conversely increase the MIL cross-entropy loss
($\mathrm{CE_{MIL}}$) contribution. Therefore, the final loss function,
considering a dynamic coefficients $\beta(e)$, is defined as:

$L(\theta,\zeta,\omega,\psi)=(1-\beta(e))\mathrm{CE_{MIL}}+\beta(e)\mathrm{CE_{SIC}},$
(6)

##  3 Experiment

We applied our classifier on microscopic images of blood samples from HHA
patients. In particular, we evaluated three aspects of our method’s
functionality: 1) classification, 2) explainability, and 3) rare anomaly
recognition.

####  3.0.1 Dataset.

Patients previously diagnosed with hereditary spherocytosis, xerocytosis,
thalassemia, and sickle cell disease were enrolled in the CoMMiTMenT study.
The study protocols were approved by the Medical Ethical Research Board of the
University Medical Center Utrecht, the Netherlands, under reference code
15/426M and by the Ethical Committee of Clinical Investigations of Hospital
Clinic, Spain (IDIBAPS) under reference code 2013/8436. Further blood samples
of SCD patients were produced for MemSID (NCT02615847) clinical trial. The
study protocol for it was approved by the Ethics committee of Canton Zurich
(KEK-ZH 2015-0297) and the regulatory authority. Our dataset contains 3630
microscopic images taken from 71 patients/healthy individuals at different
times and under variant treatments. The distribution of data in different
classes is as follows: Sickle Cell Disease (SCD, 13 patients, 170 image sets),
Thalassemia (Thal, 3 patients, 25 image sets), Hereditary Xerocytosis (Xero, 9
patients, 56 image sets), and Hereditary Spherocytosis (HS, 13 patients, 89
image sets). Also, we have a healthy control group (33 individuals, 181 image
sets), used as negative bags.

####  3.0.2 Training.

For training we considered $3-$fold cross-validation and 100 epochs. We have
two training sets: i) one set containing only control samples to estimate GMM,
and ii) an other set containing a mixture of samples from all classes to train
our network. We considered one GMM component with full covariance and iterated
an EM algorithm up to 100 times. Mixed data are fed to the MIL classifier and
every $5$ epoch, control set is fed to the model to calibrate GMM parameters
with updates. For optimization, we use the Adam optimizer [15] with the
learning rate of $5\times 10^{-5}$ and weight decay of $10^{5}$. We chose the
dynamic SIC loss coefficient $\beta(e)$ as $0.95^{e}$.

###  3.1 Classification

Our metrics for classification evaluation are average accuracy, F1 score, and
area under the precision recall curve (AUC). Table 1 shows our results
(Anomaly + Att + SIC) in comparison with other strategies proposed by Ilse et
al. [10] (Att) and Sadafi et al. [16] (Att $+$ SIC). In addition, as an
ablation study, we show the performance of our proposed anomaly scoring
without SIC loss (Anomaly + Att) and attention mechanism (Anomaly). The class-
wise performance of our proposed method compared to other methods is shown in
Fig. 2.

Table 1: Average classification performance of our proposed method compared to four baselines. The numbers show the mean value along with the standard deviation of metrics measured in 5 runs. Method | Accuracy | F1-score | AU RoC
---|---|---|---
Att [10] | 0.752$\pm$0.009 | 0.741$\pm$0.010 | 0.826$\pm$0.009
Att + SIC [16] | 0.770$\pm$0.008 | 0.757$\pm$0.009 | 0.808$\pm$0.013
Anomaly | 0.630$\pm$0.029 | 0.604$\pm$0.026 | 0.648$\pm$0.025
Anomaly + Att | 0.760$\pm$0.019 | 0.743$\pm$0.021 | 0.833$\pm$0.006
Anomaly + Att + SIC | 0.787$\pm$0.015 | 0.770$\pm$0.016 | 0.854$\pm$0.008
Figure 2: Average and standard deviation of the class-wise area under
precision recall curve in five runs. In all classes, our novel method
outperforms others.

###  3.2 Explainability

Our method’s anomaly scoring mechanism is interpretable at the instance level.
In Fig. 3, for each class, we visualized a sample image, in which anomaly and
attention scores are depicted by the color of the cell’s bounding box. The
anomaly mechanism can distinguish class-related cells better than the
attention mechanism, which is particularly striking for the SCD case, where
irregular elliptoid cells are missed by the attention mechanism (Fig. 3). In
Fig. 4 we show the distribution of scores assigned to cells in the test data.
The frequency of both anomaly and attention scores are focused around zero,
which is compatible with a majority of healthy cells (see also Fig. 4). On the
other hand, for HHA, a small distribution of high scores notifies the
anomalous cells. These cells show HHA properties.

Anomaly

Attention

Control

SCD

Thal

Xero

HS

Figure 3: Interpretation of bag classification at the instance level based on
anomaly (top) and attention (bottom) scores. Compared with attention
mechanism, disorder-relevant cells are more accurately scored by the anomaly
mechanism.  Figure 4: Distribution of anomaly scores compared to attention
scores in every class. Our anomaly scoring mechanism can distinguish a higher
fraction of disorder-relevant cells.

###  3.3 Anomaly recognition

When our model estimates the distribution of control cells, high anomaly
scores should be able to identify anomalous cells, whether the training set
contains them or not. This property of our method is useful for detecting
anomalies in rare or unseen classes during training. To examine our method in
this context, we trained the model with the two HHA classes SCD and
Thalassemia, separately. Then, we tested the model with unseen class samples
and measured their anomaly and attention scores. Fig. 5 shows the results,
where the small distribution of anomalous cells is clearly indicating anomaly
awareness of the model. For both experiments, the distribution of scores in
the control class is also shown.

Figure 5: Distribution of anomaly scores compared to attention scores for SCD
and Thal classes when their samples were absent during the training. Our
method still can assign high anomaly scores to abnormal cells.

###  3.4 CO2 Emission Related to Experiments

Experiments were conducted using our in-house infrastructure with a carbon
efficiency of 0.432 kgCO2eq/kWh. A cumulative 15 hours of computation was
performed on the Tesla V100-SXM2-32GB (TDP of 300W) hardware. Total emissions
are estimated to be 1.94 kg CO2eq. Estimations were conducted using the
MachineLearning Impact calculator presented in [11].

##  4 Conclusion

We introduce anomaly-aware MIL, a simple method for enhancing classification
of hereditary hemolytic anemias. Evaluations show that our method outperforms
other MIL classifiers and provides an instance-level explanation for bag-level
classification results. Moreover, our novel scoring mechanism identifies
anomalies in rare disease, even if they have not been seen in training sets.
An interesting future work would be to upgrade the anomaly scoring to deep-
learning-based out-of-distribution detection method. Another interesting
future work would be uncertainty estimation on anomalous instances, which
might lead to a better understanding of models coverage in representation
space.

####  4.0.1 Acknowledgements

The Helmholtz Association supports the present contribution under the joint
research school “Munich School for Data Science - MUDS”. C.M. has received
funding from the European Research Council (ERC) under the European Union’s
Horizon 2020 research and innovation programme (Grant agreement No. 866411).
CoMMiTMenT study was funded by the the European Seventh Framework Program
under grant agreement number 602121 (CoMMiTMenT) and from the European Union’s
Horizon 2020 Research and Innovation Programme. MemSID (NCT02615847) clinical
trial was funded by the Foundation for Clinical Research Hematology for
supporting the clinical trail at the Division of Hematology, University
Hospital Zurich, and, partially, by the following foundations: Baugarten
Zürich Genossenschaft und Stiftung, the Ernst Goehner Stiftung, the René und
Susanna Braginsky Stiftung, the Stiftung Symphasis and the Botnar Foundation.”
Further funding for analysis of the obtained data was obtained European
Union’s Horizon 2020 Research and Innovation Programme under grant agreement
number 675115-RELEVANCE-H2020-MSCA-ITN-2015/H2020-MSCA-ITN-2015.

## References

  * [1] Bessis, M.: Corpuscles: Atlas of Red Blood Cell Shape. Springer Science & Business Media (2012)
  * [2] Bi, Q., Yu, S., Ji, W., Bian, C., Gong, L., Liu, H., Ma, K., Zheng, Y.: Local-global dual perception based deep multiple instance learning for retinal disease classification. In: International Conference on Medical Image Computing and Computer-Assisted Intervention. pp. 55–64. Springer (2021)
  * [3] Campanella, G., Hanna, M.G., Geneslaw, L., Miraflor, A., Werneck Krauss Silva, V., Busam, K.J., Brogi, E., Reuter, V.E., Klimstra, D.S., Fuchs, T.J.: Clinical-grade computational pathology using weakly supervised deep learning on whole slide images. Nature medicine 25(8), 1301–1309 (2019)
  * [4] Fermo, E., Vercellati, C., Bianchi, P.: Screening tools for hereditary hemolytic anemia: new concepts and strategies. Expert review of hematology 14(3), 281–292 (2021)
  * [5] Fujita, S., Han, X.H.: Cell detection and segmentation in microscopy images with improved mask r-cnn. In: Proceedings of the Asian Conference on Computer Vision (2020)
  * [6] He, K., Gkioxari, G., Dollár, P., Girshick, R.: Mask r-cnn. In: Proceedings of the IEEE international conference on computer vision. pp. 2961–2969 (2017)
  * [7] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 770–778 (2016)
  * [8] Huisjes, R., van Solinge, W., Levin, M., van Wijk, R., Riedl, J.: Digital microscopy as a screening tool for the diagnosis of hereditary hemolytic anemia. International Journal of Laboratory Hematology 40(2), 159–168 (2018)
  * [9] Huisjes, R., Makhro, A., Llaudet-Planas, E., Hertz, L., Petkova-Kirova, P., Verhagen, L.P., Pignatelli, S., Rab, M.A., Schiffelers, R.M., Seiler, E., et al.: Density, heterogeneity and deformability of red cells as markers of clinical severity in hereditary spherocytosis. haematologica 105(2), 338 (2020)
  * [10] Ilse, M., Tomczak, J., Welling, M.: Attention-based deep multiple instance learning. In: International conference on machine learning. pp. 2127–2136. PMLR (2018)
  * [11] Lacoste, A., Luccioni, A., Schmidt, V., Dandres, T.: Quantifying the carbon emissions of machine learning. arXiv preprint arXiv:1910.09700 (2019)
  * [12] Li, S., Liu, Y., Sui, X., Chen, C., Tjio, G., Ting, D.S.W., Goh, R.S.M.: Multi-instance multi-scale cnn for medical image classification. In: International Conference on Medical Image Computing and Computer-Assisted Intervention. pp. 531–539. Springer (2019)
  * [13] Lu, M.Y., Chen, T.Y., Williamson, D.F., Zhao, M., Shady, M., Lipkova, J., Mahmood, F.: Ai-based pathology predicts origins for cancers of unknown primary. Nature 594(7861), 106–110 (2021)
  * [14] Lu, M.Y., Williamson, D.F., Chen, T.Y., Chen, R.J., Barbieri, M., Mahmood, F.: Data-efficient and weakly supervised computational pathology on whole-slide images. Nature biomedical engineering 5(6), 555–570 (2021)
  * [15] Reddi, S.J., Kale, S., Kumar, S.: On the convergence of adam and beyond. arXiv preprint arXiv:1904.09237 (2019)
  * [16] Sadafi, A., Makhro, A., Bogdanova, A., Navab, N., Peng, T., Albarqouni, S., Marr, C.: Attention based multiple instance learning for classification of blood cell disorders. In: International Conference on Medical Image Computing and Computer-Assisted Intervention. pp. 246–256. Springer (2020)
  * [17] Sadafi, A., Makhro, A., Livshits, L., Navab, N., Bogdanova, A., Albarqouni, S., Marr, C.: Sickle cell disease severity prediction from percoll gradient images using graph convolutional networks. In: Domain Adaptation and Representation Transfer, and Affordable Healthcare and AI for Resource Diverse Global Health, pp. 216–225. Springer (2021)
  * [18] Shi, X., Xing, F., Xie, Y., Zhang, Z., Cui, L., Yang, L.: Loss-based attention for deep multiple instance learning. In: Proceedings of the AAAI Conference on Artificial Intelligence. vol. 34, pp. 5742–5749 (2020)
  * [19] Wu, Y., Schmidt, A., Hernández-Sánchez, E., Molina, R., Katsaggelos, A.K.: Combining attention-based multiple instance learning and gaussian processes for ct hemorrhage detection. In: International Conference on Medical Image Computing and Computer-Assisted Intervention. pp. 582–591. Springer (2021)


# ID-to-3D: Expressive ID-guided 3D Heads
via Score Distillation Sampling

Francesca Babiloni, Alexandros Lattas, Jiankang Deng, Stefanos Zafeiriou
Imperial College London, UK
https://idto3d.github.io

###### Abstract

We propose ID-to-3D, a method to generate identity- and text-guided 3D human
heads with disentangled expressions, starting from even a single casually
captured in-the-wild image of a subject. The foundation of our approach is
anchored in compositionality, alongside the use of task-specific 2D diffusion
models as priors for optimization. First, we extend a foundational model with
a lightweight expression-aware and ID-aware architecture, and create 2D priors
for geometry and texture generation, via fine-tuning only 0.2% of its
available training parameters. Then, we jointly leverage a neural parametric
representation for the expressions of each subject and a multi-stage
generation of highly detailed geometry and albedo texture. This combination of
strong face identity embeddings and our neural representation enables accurate
reconstruction of not only facial features but also accessories and hair and
can be meshed to provide render-ready assets for gaming and telepresence. Our
results achieve an unprecedented level of identity-consistent and high-quality
texture and geometry generation, generalizing to a “world” of unseen 3D
identities, without relying on large 3D captured datasets of human assets.

Figure 1:  ID-to-3D leverages identity conditioning and score distillation
sampling on large diffusion models, achieving high-quality 3D human asset
generation from “in-the-wild” images, without training on large scanned
datasets. From left to right: a) renderings, b) input images, c) normals.

##  1 Introduction

The remarkable ability of humans to discern facial characteristics and
emotional cues in others makes the development of high-quality 3D head avatars
a challenging yet foundational task for a diverse array of emerging
applications, including digital telepresence, game character generation, and
the creation of virtual and augmented reality experiences. However, the
acquisition of such 3D human assets remains a daunting task, that requires
either manual work typically performed by graphic artists, or expensive and
laborious scanning. High-quality 3D facial scanning, achieved initially by
hardware with controlled polarized illumination [15, 50, 22], has been
simplified tremendously using simpler devices, color-space methods, and
inverse rendering [24, 38, 3]. Nevertheless, the requirements for expert
knowledge, hardware, and computational time do not allow wide adoption or mass
usage.

Consequently, statistical modeling and the rise of deep learning investigated
techniques to reconstruct 3D face assets from casually captured images of a
subject in arbitrary poses, lighting conditions, and occlusions (referred to
as “in-the-wild”). 3D Morphable Models (3DMMs) and Generative Adversarial
Networks (GANs) can be used to model facial geometry [4, 18, 55, 23], while
GANs and diffusion models have achieved state-of-the-art modeling and
reconstruction of facial appearance [39, 21, 92, 60].

However, a common denominator in all the above is the requirement for vast
datasets of scanned facial shapes and appearances, in order to avoid limited
generalization, ethnicity under-representation, and oversmoothed geometries,
requiring up to $10,000$ scans for 3DMMs training [6] or facial appearance
modeling [21]. Moreover, the requirement for aligned data in 3D, and also in
UV texture maps, limits the utilized area to the facial region, and introduces
registration errors and additional costs.

Figure 2: (Left) Overall pipeline. ID-to-3D generates expressive 3D head
avatars via ArcFace $y_{\text{id}}$ and textual $y_{\text{text}}$
conditioning. It uses as prior geometry-oriented $\phi_{g}$ and albedo
oriented $\phi_{a}$ pretrained models. Training) The training phase uses SDS
to optimize 3D geometry $\psi_{g}$, texture $\psi_{a}$, and a set of
expressions latent codes $\mathbf{k}_{\textbf{exp}}$. It also leverages random
lighting $\mathbf{l}$ and random expression conditioning $y_{\text{exp}}$.
Inference) At deployment time, ID-to-3D extracts high-quality identity-aware
expressive 3D meshes. (Right) ID-consistent expressive 3D heads generated by
our method. ID-to-3D creates 3D assets that support relighting, ID-consistent
editing, and physical simulation.

Our proposed method bypasses these significant issues by utilizing large 2D
generative models, pre-trained on vast and easy-to-acquire 2D images, and
using only a small set of rasterized 3D data for finetuning, while achieving
state-of-the-art 3D head generation.

More recently, the use of Score Distillation Sampling (SDS) [63] and large-
scale diffusion models explored the automatic generation of 3D content thanks
to text [85, 10, 44, 43, 30, 92, 36, 25] or image prompts [66, 91, 64, 79,
80]. Despite promising results, leveraging 2D priors to generate realistic 3D
head avatars remains challenging: (1) Large text-to-image foundational models
are usually trained to generate realistic RGB images, where geometry, texture,
and lighting cannot be individually segregated. This narrow “rendering
knowledge" in the 2D guidance compromises the 3D fidelity, texture quality,
and consistency of the generated assets, leading to unrealistic geometries and
distorted textures with recognizable artifacts such as Janus problems,
incorrect proportions, oversaturated albedo, and mismatching texture and
geometry details. (2) It is difficult to precisely control facial attributes
solely using typical prompting methods [13, 59, 88, 84]. Textual prompts lack
the granularity to single out the specificity of a subject’s identity and
facial expression that might be lengthy and complex to convey in natural
language. Moreover, methods that leverage image prompts lack the capability to
capture features that represent the facial identities of a subject
independently of pose, expression, or contextual scene information (i.e., ID
embeddings). The lack of reliable control over identity and facial expression
prevents the personalization of 3D head avatars, resulting in a limited range
of possible output expressions or custom attributes without incurring identity
drifting.

In this work, we present ID-to-3D, a new method to generate expressive and
identity-consistent 3D human heads using a text prompt and a small set of 1-5
casually captured, in-the-wild, images of a subject. ID-to-3D creates a
variety of separated yet ID-consistent expressions in a single optimization,
leveraging as priors compositionality and task-specific 2D diffusion models.
To ensure consistency, the identity of each subject is encoded via facial
embedding features and, to encourage expressivity, emotions are disentangled
via a novel ID-specific neural parametric representation. The generation of
each 3D asset leverages score distillation sampling and a two-stage pipeline
to create shape details, texture features, and materials. During optimization,
the guidance is given by a foundational model extended into an a) identity-
aware, b) task-aware and c) expression-aware variant, working as 2D prior for
either geometry or texture generation. Each stage guidance is trained only
once for all the potential identities, with a lightweight fine-tuning strategy
changing only $0.2\%$ of all the available training parameters.

Overall, we present the following contributions: (1) ArcFace-Conditioned 3D
Head Asset Generation: we introduce the first method for arcface-conditioned
generation of 3D head assets using SDS. (2) Novel ID-Conditioned Expressive
Model: our model creates an identity-conditioned expressive representation for
each subject, enabling the generation of up to 13 unique and ID-consistent
expressions captured by latent codes and associated with a set of 3D assets
with separate geometric, albedo, and material information. (3) Novel Text-
to-2D-Normals and Text-to-2D-Albedo Models: we present a novel approach to
creating ID-conditioned and expression-conditioned text-to-image models
capable of generating realistically plausible normals and albedo images from a
small set of 3D assets. Extensive experiments confirm that our method
outperforms text-based and image-based SDS baselines by producing relight-able
3D head assets with unprecedented geometric details, superior texture quality,
and able to exhibit a wide variety of expressions. Fig. 1 displays 3D heads
generated with ID-to-3D.

##  2 Related Work

3D Human Generation and Reconstruction. Human modelling and reconstruction in
the 3D domain is typically based on 3D Morphable Models (3DMMs) [4, 6, 42,
78], which model variations in human shape and appearance using PCA. The
advent of deep learning enabled very expressive reconstructions [14, 68], but
also extended this line of research beyond linear spaces. Neural Parametric
Head models [23] explored the use of signed distance function and deformation
field to generate expressive geometries. Generative Adversarial Networks have
also been proposed to generate [41, 20] and reconstruct faces from “in-the-
wild” images [21, 49, 39]. Diffusion models showed impressive results in
modeling skin textures [92, 60] especially when paired with large-scale, high-
quality datasets of real scans.

Text-to-3D Human Generation. The creation of 3D human assets via text
conditioning has seen significant progress, building on the foundations laid
by advances in text-to-2D generation [65, 69, 72, 67]. Initial attempts [86,
28, 53, 73, 54, 31, 11] utilized the CLIP language model to optimize implicit
or explicit 3D representations. The seminal work of DreamFusion [63]
introduced the Score Distillation Sampling (SDS) loss, which uses a pre-
trained 2D diffusion as prior for 3D generation. This work led to a revolution
in text-to-3D generation [52, 56, 26, 85, 44, 47, 51, 46, 12] and text-to-3D
human generation [9, 76, 44, 52, 25, 45, 48]. DreamAvatar [8], TADA [43] and
Headevolver [82] build upon the use of a template [61, 42, 5] to create 3D
human avatars with controllable shapes and poses. Fantasia3D [10] separated
geometry and texture training, employing DMTET [77] and PBR texture [58] for
the 3D representation. HumanNorm [30] introduced the idea of training a normal
diffusion model to guide high-quality geometry generation. Despite promising
results, these models face challenges in generating highly detailed and
expressive 3D heads, due to the inherent limitations of natural language
conditioning.

Personalized Generation with Diffusion Models. Controllable generation is
crucial to develop widely applicable generative models. Work in this domain
includes the costumization of GANs [74, 32, 34, 35, 33] and research dedicated
to steer the generation of large-scale foundation models with additional
control signals [19, 70, 29, 93, 88, 71]. The use of ID embeddings as an
alternative to text prompts showed promising results in 2D generation [62, 87,
84, 59], but remains underexplored in the creation of 3D human avatars with
SDS. Related research supplements traditional text prompts in SDS pipelines
with image-based prompts. DreamBooth3D [66] and Avatarbooth [91] proposed to
train an ad hoc model to use as a guide in the generation of 3D objects or
avatars. Magic123 [64] incorporate 3D and 2D priors in SDS generation.
DreamCraft3D [79] proposed a hierarchical 3D content pipeline to generate
textured 3D meshes from a single unposed image. Despite encouraging results,
the creation of high-quality head assets with these methods remains
challenging, due to the difficulty of extracting appropriate facial features
using only naive image prompting.

##  3 ID-to-3D

We propose a novel method for identity-driven human head generation, which
utilizes a pre-trained 2D model to distill expressive head avatars with high
geometric details and high-fidelity textures, avoiding the need for large-
scale training on 3D datasets. As illustrated in Figure 2, starting from a
subject’s identity embeddings, our method trains a set of latent expression
representations using a two-stage SDS pipeline. After convergence, the learned
3D representation can be used to create ID-driven expressive heads, that are
ready to be used in common rendering engines.

###  3.1 3D Head Optimization Objective

A Score Distillation Sampling generation pipeline optimizes a 3D
representation $\theta$ using a pre-trained 2D diffusion model $\phi$ as
guidance. The pipeline optimization objective is to align the distribution of
3D asset renderings with the target distribution
$p(\mathbf{x}_{0}|y_{\text{text}})$, created by the 2D diffusion model
conditioned on an input text $y_{\text{text}}$. Given the distribution of
renderings under various camera conditions
$q^{\theta}(\mathbf{x}_{0}|y_{\text{text}})=\int
q^{\theta}(\mathbf{x}_{0}|y_{\text{text}},\mathbf{c})p(\mathbf{c})d\mathbf{c}$,
the optimization objective reads:

$\min_{\theta}D_{KL}(q^{\theta}(\mathbf{x}_{0}|y_{\text{text}})\ \|\
p(\mathbf{x}_{0}|y_{\text{text}})).$ (1)

The target distribution $p(\mathbf{x}_{0}|y_{\text{text}})$ is typically
estimated by a foundational text-to-image diffusion model that approximates
the distribution of natural RGB images [69] (i.e., trained over large and
uncurated datasets such as LAION-5B [75]). Despite its indisputable success in
creating a variety of assets, using the above objective and guidance to
generate detailed 3D heads remains a complicated task. First, this target
distribution might drift significantly from the distribution of natural heads
rendered in realistic light and camera conditions. Second, a general one-shot
guidance model does not have explicit ways to differentiate texture and
geometric characteristics, which makes the creation of light-independent
texture and high-quality geometry extremely challenging. Third, the naive use
of text prompts limits the control over the generated head assets, since
textual prompts cannot easily or exhaustively capture facial and expression
features. In our pipeline’s generation, we decompose the Obj. 1 into two
smaller and more controllable objectives:

$\begin{split}\min_{\theta_{g},\theta_{a}}\underbrace{D_{KL}(q^{\theta_{g}}(\mathbf{z}_{0}^{n}|\mathbf{c},y_{\text{text}},y_{\text{exp}},y_{\text{id}})\
\|\
p(\mathbf{z}_{0}^{n}|\mathbf{c},y_{\text{text}},y_{\text{exp}},y_{\text{id}}))}_{geometry\
generation\ objective}\\\
+\underbrace{D_{KL}(q^{\theta_{a}}(\mathbf{z}_{0}^{a}|\mathbf{c},\mathbf{l},y_{\text{text}},y_{\text{exp}},y_{\text{id}})\
\|\
p(\mathbf{z}_{0}^{a}|\mathbf{c},y_{\text{text}},y_{\text{exp}},y_{\text{id}}))}_{texture\
generation\ objective}.\end{split}$ (2)

In the above equation, $\theta_{g}$ represents the parameterization of the 3D
geometry, $\mathbf{z}_{0}^{n}$ denotes normal maps, $\theta_{a}$ denotes the
parameterization of the 3D textures and $\mathbf{z}_{0}^{a}$ albedo textures.
Conditioning is introduced in the form of textual prompt $y_{\text{text}}$,
identity condition $y_{\text{id}}$, and expression condition $y_{\text{exp}}$.
The letter $\mathbf{l}$ denotes the lighting condition of the rendered image.
The target distributions for geometry and texture generation refer to the
ideal distribution of head-specific normal and texture maps, which are in
practice estimated via geometry-oriented and albedo-oriented models guided by
face-specific conditioning.

###  3.2 2D Guidance

To initiate the 3D head reconstruction, we start from the development of 2D
priors capable of accurately separating texture and geometric details while,
at the same time, consistently representing the facial characteristic of a
subject under various expressions, conveying different emotional states. The
difficulty of this task lies in its nuanced nature, exacerbated by the lack of
large-scale 3D human scan datasets, which makes the capture of detailed face
features and the generalization to new identities particularly difficult when
training from scratch or even when naively fine-tuning from a large-scale
model. To solve this challenge, we propose to explicitly model geometry and
appearance domains, identity conditioning $y_{\text{id}}$ and expression
conditioning $y_{\text{exp}}$, achieving a modular separation of otherwise
entangled-together information. To overcome the need for a large-scale
dataset, we leverage a small dataset of human heads with different expressions
(NHPM) [23], a pre-trained stable diffusion model (SD) [69], and a selective
fine-tuning strategy that affects only a minimal number of parameters, needed
to accommodate these new conditionings. In practice, we use rasterized normals
as a 2D proxy for geometric information and rasterized albedo as a
representation of appearance information. We treat the shift from the natural
image distribution towards normal maps and albedo textures as “style-transfer”
tasks, aiming to leave the content of the SD features unchanged while
modifying their self-similarity information. We use Low-Rank adaptation
matrices (LoRA) [29] to adjust Query $\mathbf{Q}$, Key $\mathbf{K}$ and Value
$\mathbf{V}$ features of the self-attention to work in the adjusted normal and
albedo domains. The normal-adapted self-attention equation becomes the
following:

$\mathbf{Z}^{n}_{\mathbf{{SA}}}=\text{Att}(\mathbf{Q}^{n},\mathbf{K}^{n},\mathbf{V}^{n}),\mathbf{Q}^{n}=\mathbf{XW}_{Q}+\mathbf{XW}_{Q}^{n},\mathbf{K}^{n}=\mathbf{XW}_{K}+\mathbf{XW}_{K}^{n},\mathbf{V}^{n}=\mathbf{XW}_{V}+\mathbf{XW}_{V}^{n}$
(3)

while the albedo-adapted self-attention can be read by changing the
superscript $n$ to $a$. As identity representation we select the identity
embeddings $\mathbf{y_{\text{id}}}$, from a state-of-the-art face recognition
network [16], a compact vector of facial features extracted from “in-the-wild”
images of a subject. As expression conditioning, we use
$\mathbf{y_{\text{exp}}}$ CLIP embeddings [65] for the textual descriptor of
the 23 FACS coded expressions proposed in FaceWareHouse [7]. To ensure control
over the generated head and face representation during deployment, we treat
the integration of identity and expression information in the baseline
architecture as “multimodal conditioning”, by including their contribution in
the SD cross-attention layers via IP-Adapter [88] strategy:

$\small\mathbf{Z}^{n}_{\mathbf{{CA}}}=\text{Att}(\mathbf{Q},\mathbf{K^{text}},\mathbf{V^{text}})+\lambda_{id}\cdot\text{Att}(\mathbf{Q},\mathbf{K^{id}},\mathbf{V^{id}})+\lambda_{exp}\cdot\text{Att}(\mathbf{Q},\mathbf{K^{exp}},\mathbf{V^{exp}})$
(4)

In the above equation, $\lambda_{id}$ and $\lambda_{exp}$ control the
contribution of identity and expression conditioning. The
${\scriptstyle\mathbf{Q}=\mathbf{XW}_{Q}}$ term represents the Query extracted
from SD features,
${\scriptstyle\mathbf{K^{exp}}=\mathbf{y_{\text{exp}}}\mathbf{W}_{K}^{\text{exp}}}$
and
${\scriptstyle\mathbf{V^{exp}}=\mathbf{y_{\text{exp}}}\mathbf{W}_{V}^{\text{exp}}}$
the Key and Values extracted from the expression embedding, and
${\scriptstyle\mathbf{K^{id}}=\mathbf{y_{\text{id}}}\mathbf{W}_{K}^{\text{id}}}$
and
${\scriptstyle\mathbf{V^{id}}=\mathbf{y_{\text{id}}}\mathbf{W}_{V}^{\text{id}}}$
the Key and Values extracted from the identity embedding. We train only our
additional parameters, leaving the rest of the model frozen, targeting $0.2\%$
of the overall trainable parameters. We separately optimize a 2D prior for
geometry $\phi_{g}$ and textures $\phi_{a}$, using image-conditioning pairs
created from the renders of the NHPM dataset under various camera poses
$\mathbf{c}$. The training objective for the geometric 2D prior follows the
same training objective as a traditional SD model [69]:

$L_{\text{simple}}=\mathbb{E}_{\mathbf{z}_{0}^{n},\boldsymbol{\epsilon},\boldsymbol{c},t,\mathbf{y_{\text{text}}},\mathbf{y_{\text{id}}},\mathbf{y_{\text{exp}}}}\|\boldsymbol{\epsilon}-\boldsymbol{\epsilon}_{\phi_{g}}\big{(}\mathbf{z}_{t}^{n},t,\boldsymbol{c},\mathbf{y_{\text{text}}},\mathbf{y_{\text{id}}},\mathbf{y_{\text{exp}}}\big{)}\|^{2}.$
(5)

while the analogous training objective for the 2D texture prior can be derived
by changing both superscript $n$ and subscript $g$ to $a$. Note that in
Equation 5, $\mathbf{y_{\text{exp}}}$ indicates CLIP embeddings for the
textual descriptor of the renders (i.e., camera view and subject’s attributes)
and does not convey face-identity. $\mathbf{z}_{t}^{n}$ indicates
$\mathbf{z}_{0}^{n}$ noised at timestep $t$. Examples of generated prompt-to-
images can be seen in Figure 2 as well as additional materials.

###  3.3 Geometry Generation

We represent an identity-specific geometry as a neural parametric head model
composed of a deep marching tetrahedra (DMTET [77]) representation,
additionally coupled with a set of identity-dependent facial expression latent
codes. Our lightweight geometric representation produces highly detailed
geometry and a broad range of expressions without notable identity drift, as
shown by our experiments. Compared to explicit template-based approaches, it
has the flexibility to dynamically modulate the local resolution of the mesh
to capture high-frequency geometrical details. In other words, it adjusts the
mesh resolution of specific regions of the face to adapt to the given subject
and expression.

The DMTET geometry representation uses a deformable tetrahedral grid $\Gamma$
and a network $\Psi_{g}$ to generate a 3D asset [77]. We extend DMTET to learn
multiple expressions at the same time. We model each facial expression with a
learnable latent code $\mathbf{k}^{n}_{\text{exp}}\in\mathbb{R}^{d_{exp}}$ and
design the network $\Psi_{g}(\Gamma,\mathbf{k}^{n}_{\text{exp}})$ as a
Transformer [81], parameterized by $\psi_{g}$ learnable parameters, that
processes both the deformable grid and the expression information. During the
training phase, we randomly select one of the potential expressions, estimate
the signed distance function (SDF) for the underlying head, and use a
differentiable marching tetrahedra layer to convert the implicit
representation into the explicit surface mesh for that ID and expression,
compelling the $\Psi_{g}$ model to learn a diverse set of expressions that are
consistent with the identity at hand. We supervise optimization through an SDS
loss [63], computed using the rasterized geometry model $\phi_{g}$ as 2D
prior. We optimize the 3D representation $\theta_{g}$ as follows:

$\begin{gathered}\nabla\mathcal{L}_{SDS}(\theta_{g})=\mathbb{E}_{\mathbf{c},t,\epsilon}\left[\omega(t)(\mathbf{\epsilon}_{\phi_{g}}(\mathbf{z}_{t}^{n},\mathbf{y_{id}},\mathbf{y_{exp}},\mathbf{y_{text}},t)-\epsilon)\frac{\partial
g(\theta_{g},\mathbf{c})}{\partial\theta_{g}}\right];\theta_{g}=\left[\mathbf{k}^{n}_{\text{exp}},\psi_{g}\right].\end{gathered}$
(6)

In the above equation, $g(\theta_{g},\mathbf{c})$ represents the normals of
the rendered image, created using the differentiable render $g$ and the camera
pose $\mathbf{c}$, and $\mathbf{k}^{n}_{\text{exp}}$ is the randomly sampled
expression code. Lastly, we combine the SDS loss with a Laplacian regularizer,
to encourage smooth surfaces.

Figure 3: Qualitative results for text-to-3D (*) and image-to-3D methods.
Methods are evaluated under the same text prompt and rendering conditions.
DreamCraft3D is reported as DC3D. Geometry is displayed via normal maps in
camera coordinates. Using only a small set of $5$ images as conditioning, ID-
to-3D achieves high geometric quality and realistic textures.

###  3.4 Texture Generation

Given a trained geometry model $\theta_{g}$, we model the texture appearance
$\theta_{a}$. To ensure ID-aligned expression generation, we follow an
analogous parameterization for the expressions in the texture domain. We
represent an identity-specific appearance as a neural parametric head model
composed of a pseudo-albedo prediction network $\Psi_{a}$, coupled with a set
of ID-dependent facial expression latent codes. We instantiate each expression
as a learnable latent code $\mathbf{k}^{a}_{\text{exp}}\in R^{d_{exp}}$ and
model $\Psi_{a}(\mathbf{k}^{a}_{\text{exp}})$ as a Transformer trained to
predict the spatially-varying reflectance in a UV-map representation [89],
namely the albedo, roughness and specularity, for each ID and expression-
specific texture. We use an off-the-shelf physically-based renderer [37].
Reflectance disentanglement is an ill-posed problem, and in the absence of
prior data, we use camera and illumination randomization as a regularization
constraint [17, 40]. On each iteration, we sample random environment
illumination maps, augmented with random Y-axis rotations. During training, we
randomize the sampling of the latent expression and deploy the albedo model
$\phi_{a}$ SDS loss to optimize $\theta_{a}$:

$\begin{gathered}\nabla\mathcal{L}_{SDS}(\theta_{a})=\mathbb{E}_{\mathbf{c},t,\epsilon}\left[\omega(t)(\mathbf{\epsilon}_{\phi_{a}}(\mathbf{z}_{t}^{a},\mathbf{y_{id}},\mathbf{y_{exp}},\mathbf{y_{text}},t)-\epsilon)\frac{\partial
g(\theta_{a},\mathbf{c},\mathbf{l})}{\partial\theta_{a}}\right];\theta_{a}=\left[\mathbf{k}^{a}_{\text{exp}},\psi_{a}\right].\end{gathered}$
(7)

where $g(\theta_{a},\mathbf{c},\mathbf{l})$ represents the pseudo-albedo maps
created using a differentiable render $g$ and a sample camera pose and
lighting condition.

##  4 Experiments

We assess the efficacy of ID-to-3D as a specialized method for ID-driven
expressive human face generation in different scenarios and report comparative
analysis against state-of-the-art text-to-3D and image-to-3D generation
pipelines. Further analysis, implementation details and discussion can be
found in additional material.

###  4.1 Identity Generation

We benchmark ID-to-3D against several state-of-the-art methods in the domain
of SDS-based 3D face asset generation. Specifically, we consider Fantasia3D
[10], three recent and similar methods specialized in text-to-human avatar
generation (i.e., Human-Norm [30], TADA [43], DreamFace [92]) and two methods
that leverage both text and images to create 3D assets (i.e., Magic123 [64],
DreamCraft3D [79]). In order to compare together text-to-3D and image-to-3D
methods, we select a test benchmark of 40 celebrity names, suggested by
ChatGPT, covering actors, sports, and media personalities. We automatically
download $25$ images for each identity from BingImages [1]: $5$ images to use
as input and $20$ to use as references for our comparisons. For all methods,
we use the same textual prompt “a DSLR face portrait of…” and the same input
images.

Qualitative Comparisons. Results of existing methods are reported in Figure 3
under the same lighting and rendering conditions. The 3D assets created by ID-
to-3D show realistic texture, sharp fine-grained details, and ID fidelity,
capturing facial characteristics of the input identity without relying heavily
on often ambiguous text prompts or naively lifting 2D images in the 3D domain.

Quantitative Comparisons: Identity Similarity Distribution. We quantitatively
assess the ID fidelity of the 3D assets using the CosFace similarity metric
[83, 39]. We center and align the 3D objects and collect renders in a wide
range of camera positions (i.e. elevation [$-15^{\circ},+15^{\circ}$] and
rotation [$-65^{\circ},+65^{\circ}$]). We measure the cosine similarity
between the ID features of each render and a set of $20$ in-the-wild images of
the same identity, used as reference. We report the distribution of the
identity similarity for each method in Figure 4 (Left). Note that the variance
of the distribution correlates with the ID consistency of the 3D object across
viewpoints. Despite being able to generate realistic skin textures, DreamFace
cannot create hair or eyes by design, resulting in the lowest average
similarity score. DreamCraft3D and Magic123 leverage the input image to
achieve a photorealistic front-facing camera render, but struggle to create 3D
consistent heads, resulting in distributions associated with the highest
variance. As clearly visible from Figure 4, ID-to-3D is capable of creating
realistic and consistent 3D heads, reporting the lowest variance and the
highest similarity score.

Figure 4:  (Left) Identity Similarity Distribution between “in-the-wild”
images and renderings of 3D heads. (Right) Comparative Preference Survey on
texture quality (outside) and geometry quality (in). We report $\%$ of
preferences.

Texture $\downarrow$ Geometry$\downarrow$ Fantasia3D* [10] 252 159 DreamFace*
[92] 188 145 TADA* [43] 215 118 HumanNorm* [30] 164 98 Magic123 [64] 162 180
DreamCraft3D [79] 205 165 Ours 153 85

Table 1: Frechet Inception Distance measures the geometric and texture quality
for the generated 3D heads. “*” indicates text-to-3D SDS pipelines.

Quantitative Comparisons: FID and User-Study. The evaluation of the generated
3D geometries and textures is performed using the Frechet Inception Distance
(FID) metric [27]. For texture quality, the FID is calculated between the
renderings and the images from Stable Diffusion V1.5 [69]. For geometry
quality, FID is determined by comparing normal maps with those extracted from
the NHPM test set [23]. To further substantiate our analysis, we conduct two
user preference surveys, comparing our method with the $4$ strongest baselines
for the generation of texture and geometry. As shown in Table 1 and Figure 4
(Right), FID metrics and user evaluations report aligned results. Our model
achieves the lowest FID scores and the highest user preference in both
geometry and texture generation, showcasing together the overall stronger
performance of ID-to-3Das a human-specific geometry and texture generator.

###  4.2 Expressive ID-conditioned Generation

ID-to-3D is specifically designed to create complex, uncommon, and subtle
expressions with a level of details not previously achievable using existing
SDS methods. In Figure 5 we showcase the unique ability of our method to
generate a wide range of expressions that remain identifiable and yet
identity-consistent.

Figure 5: ID-to-3D expression diversity. Renderings and normal maps in camera
coordinates are taken for $3$ identities: Will Smith, Anya Taylor Joy, and
Kanye West. Our method achieves fine-grained geometry carving and high-quality
texture generation, realistically reproducing various skin tones.

Quantitative Analysis: Identity Similarity Distribution Across Expressions. We
test the ID consistency across views and expressions by reporting the cosine
similarity between the ID features of a reference render (i.e., neutral
expression and front-facing camera) and the ID features captured across the
remaining expressions and view points. Figure 6 (Right) shows the distribution
of ID similarity computed for a set of 10 subjects. ID-to-3D consistently
produces 3D assets with high ID similarity and low variance.

Figure 6:  Expressions analysis. ID-to-3D creates a variety of expressions
with robust ID consistency. (Left) Visualizations  of different expressions
for $2$ identities (i.e. Bill Gates, Alexander Skarsgard). (Middle) Expression
diversity. t-SNE plot visualizing the ID embeddings computed considering
different camera poses, expressions, and subjects. Different identities and
expressions are clustered separately. (Right) Identity similarity
distributions between neutral-pose and remaining expressions. Rendered with
[2].

Quantitative Analysis: Expression Variety Visualization. A core characteristic
of ID-to-3D is its robust ability to produce a wide range of unique and vivid
expressions. In this section, we provide a visualization of this expression
diversity. We select a subset of 13 expressions and 10 subjects, extracting
for each 3D head a set of renders in a range of 9 camera poses. Then we
extract the ID features for each render and project them into 2D points using
t-SNE. As visible in Figure 6 (Middle), plotting these points clearly shows
the heterogeneity of the generated expressions and identities. Firstly,
renders associated with the same identity are grouped in distinct clusters.
Furthermore, within each cluster, we observe how there is a local variation of
ID features as they adapt their response to different expressions.

Figure 7: Identity-Consistent Editing. (Left) De-aged 3D heads  generated
using different identity conditioning and the textual prompt: “…as a cute
baby”. Normal maps are displayed in world coordinates next to photorealistic
renderings.  (Right) Geometry and texture editing  with text prompts. ID-to-3D
edits appearance and geometric features in an ID-consistent manner.

###  4.3 ID-conditioned Text-based Customization

The unique characteristics of our model allow for the customization of 3D
objects without identity or expression drift. Figure 7 and 2 (Right) showcase
how a variety of alterations can be imposed on the object’s geometry, given
the appropriate text conditioning while maintaining the subject’s general
physiognomy. In particular, ID-to-3D displays aligned geometry and appearance
even after editing, preserving the ability to convey different expressions
(e.g. from "eyes Closed" to "row lowerer", from "neutral" to "squeeze") even
after facial hair textures have been altered via textual prompt.

##  5 Conclusion and Ethical Considerations

#### Limitations.

Despite setting a new state-of-the-art, we acknowledge ID-to-3D
limitations:(1) the generalization capacity is limited by the used face
embedding network [16] and diffusion model [69, 88], which could introduce
inaccurate representation biases; (2) the lack of specific optimization for
physically bounded textures and geometries might occasionally produce
unnatural facial characteristics, hence we refer to albedo as pseudo-albedo,
even though we produce render-ready assets; (3) the computational resources
needed for PBR rendering hinder potential applications in video-driven
problems.

#### Societal Impact.

Technological advancements in automatic 3D human generation have various
beneficial applications, but also raise important ethical considerations about
representation and potential misuses. We advocate for responsible research and
take the following steps to mitigate unauthentic reconstructions: (1) we
restrict training data to the facial region only. (2) we advocate for the
replacement of text-based prompts with larger ID embeddings that are sometimes
lacking [90] but entail significantly less bias than methods trained only on
celebrity datasets and text [65].

#### Conclusion.

In this work, we present ID-to-3D , a novel method for expressive 3D head
asset generation from one or more face images. Our method deploys a novel
human parametric expression model in tandem with specialized geometry and
albedo guidance, not only to create intricately detailed head avatars with
realistic textures but also to achieve strikingly ID-consistent results across
a wide range of expressions, setting a new benchmark in comparison to existing
SDS techniques. Without having to rely on 3D captured datasets that are
expensive to collect and typically biased, and without being constrained on a
specific geometry template, our method can be employed by a broad range of
subjects, with different features such as skin tone and hairstyle.

## References

  * [1] https://www.bing.com/images/feed.
  * [2] Marmoset toolbag.  https://marmoset.co/toolbag/.  Accessed: 2022-05-20.
  * [3] Dejan Azinović, Olivier Maury, Christophe Hery, Matthias Nießner, and Justus Thies.  High-res facial appearance capture from polarized smartphone images.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16836–16846, 2023.
  * [4] Volker Blanz and Thomas Vetter.  A morphable model for the synthesis of 3d faces.  In SIGGRAPH ’99, 1999.
  * [5] Federica Bogo, Angjoo Kanazawa, Christoph Lassner, Peter Gehler, Javier Romero, and Michael J Black.  Keep it smpl: Automatic estimation of 3d human pose and shape from a single image.  In Computer Vision–ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part V 14, pages 561–578. Springer, 2016.
  * [6] James Booth, Anastasios Roussos, Stefanos Zafeiriou, Allan Ponniah, and David Dunaway.  A 3d morphable model learnt from 10,000 faces.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 5543–5552, 2016.
  * [7] Chen Cao, Yanlin Weng, Shun Zhou, Yiying Tong, and Kun Zhou.  Facewarehouse: A 3d facial expression database for visual computing.  IEEE Transactions on Visualization and Computer Graphics, 20(3):413–425, 2013.
  * [8] Yukang Cao, Yan-Pei Cao, Kai Han, Ying Shan, and Kwan-Yee K Wong.  Dreamavatar: Text-and-shape guided 3d human avatar generation via diffusion models.  arXiv preprint arXiv:2304.00916, 2023.
  * [9] Yukang Cao, Yan-Pei Cao, Kai Han, Ying Shan, and Kwan-Yee K Wong.  Guide3d: Create 3d avatars from text and image guidance.  arXiv preprint arXiv:2308.09705, 2023.
  * [10] Rui Chen, Yongwei Chen, Ningxin Jiao, and Kui Jia.  Fantasia3d: Disentangling geometry and appearance for high-quality text-to-3d content creation.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 22246–22256, 2023.
  * [11] Yongwei Chen, Rui Chen, Jiabao Lei, Yabin Zhang, and Kui Jia.  Tango: Text-driven photorealistic and robust 3d stylization via lighting decomposition.  Advances in Neural Information Processing Systems, 35:30923–30936, 2022.
  * [12] Yen-Chi Cheng, Hsin-Ying Lee, Sergey Tulyakov, Alexander G Schwing, and Liang-Yan Gui.  Sdfusion: Multimodal 3d shape completion, reconstruction, and generation.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4456–4465, 2023.
  * [13] Siying Cui, Jiankang Deng, Jia Guo, Xiang An, Yongle Zhao, Xinyu Wei, and Ziyong Feng.  Idadapter: Learning mixed features for tuning-free personalization of text-to-image models.  arXiv preprint arXiv:2403.13535, 2024.
  * [14] Radek Daněček, Michael J Black, and Timo Bolkart.  Emoca: Emotion driven monocular face capture and animation.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 20311–20322, 2022.
  * [15] Paul Debevec, Tim Hawkins, Chris Tchou, Haarm-Pieter Duiker, Westley Sarokin, and Mark Sagar.  Acquiring the reflectance field of a human face.  In Proceedings of the 27th annual conference on Computer graphics and interactive techniques, pages 145–156, 2000.
  * [16] Jiankang Deng, Jia Guo, Niannan Xue, and Stefanos Zafeiriou.  Arcface: Additive angular margin loss for deep face recognition.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 4690–4699, 2019.
  * [17] Valentin Deschaintre, Miika Aittala, Fredo Durand, George Drettakis, and Adrien Bousseau.  Single-image svbrdf capture with a rendering-aware deep network.  ACM Transactions on Graphics (ToG), 37(4):1–15, 2018.
  * [18] Bernhard Egger, William AP Smith, Ayush Tewari, Stefanie Wuhrer, Michael Zollhoefer, Thabo Beeler, Florian Bernard, Timo Bolkart, Adam Kortylewski, Sami Romdhani, et al.  3d morphable face models—past, present, and future.  ACM Transactions on Graphics (ToG), 39(5):1–38, 2020.
  * [19] Rinon Gal, Yuval Alaluf, Yuval Atzmon, Or Patashnik, Amit H Bermano, Gal Chechik, and Daniel Cohen-Or.  An image is worth one word: Personalizing text-to-image generation using textual inversion.  arXiv preprint arXiv:2208.01618, 2022.
  * [20] Baris Gecer, Alexandros Lattas, Stylianos Ploumpis, Jiankang Deng, Athanasios Papaioannou, Stylianos Moschoglou, and Stefanos Zafeiriou.  Synthesizing coupled 3d face modalities by trunk-branch generative adversarial networks.  In European conference on computer vision, pages 415–433. Springer, 2020.
  * [21] Baris Gecer, Stylianos Ploumpis, Irene Kotsia, and Stefanos Zafeiriou.  Ganfit: Generative adversarial network fitting for high fidelity 3d face reconstruction.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 1155–1164, 2019.
  * [22] Abhijeet Ghosh, Graham Fyffe, Borom Tunwattanapong, Jay Busch, Xueming Yu, and Paul Debevec.  Multiview face capture using polarized spherical gradient illumination.  In Proceedings of the 2011 SIGGRAPH Asia Conference, pages 1–10, 2011.
  * [23] Simon Giebenhain, Tobias Kirschstein, Markos Georgopoulos, Martin Rünz, Lourdes Agapito, and Matthias Nießner.  Learning neural parametric head models.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 21003–21012, 2023.
  * [24] Paulo Gotardo, Jérémy Riviere, Derek Bradley, Abhijeet Ghosh, and Thabo Beeler.  Practical dynamic facial appearance modeling and acquisition.  ACM Transactions on Graphics (ToG), 37(6):1–13, 2018.
  * [25] Xiao Han, Yukang Cao, Kai Han, Xiatian Zhu, Jiankang Deng, Yi-Zhe Song, Tao Xiang, and Kwan-Yee K Wong.  Headsculpt: Crafting 3d head avatars with text.  Advances in Neural Information Processing Systems, 36, 2024.
  * [26] Ayaan Haque, Matthew Tancik, Alexei A Efros, Aleksander Holynski, and Angjoo Kanazawa.  Instruct-nerf2nerf: Editing 3d scenes with instructions.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 19740–19750, 2023.
  * [27] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter.  Gans trained by a two time-scale update rule converge to a local nash equilibrium.  Advances in neural information processing systems, 30, 2017.
  * [28] Fangzhou Hong, Mingyuan Zhang, Liang Pan, Zhongang Cai, Lei Yang, and Ziwei Liu.  Avatarclip: Zero-shot text-driven generation and animation of 3d avatars.  arXiv preprint arXiv:2205.08535, 2022.
  * [29] Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen.  Lora: Low-rank adaptation of large language models.  arXiv preprint arXiv:2106.09685, 2021.
  * [30] Xin Huang, Ruizhi Shao, Qi Zhang, Hongwen Zhang, Ying Feng, Yebin Liu, and Qing Wang.  Humannorm: Learning normal diffusion model for high-quality and realistic 3d human generation.  arXiv preprint arXiv:2310.01406, 2023.
  * [31] Ajay Jain, Ben Mildenhall, Jonathan T Barron, Pieter Abbeel, and Ben Poole.  Zero-shot text-guided object generation with dream fields.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 867–876, 2022.
  * [32] Minguk Kang, Jun-Yan Zhu, Richard Zhang, Jaesik Park, Eli Shechtman, Sylvain Paris, and Taesung Park.  Scaling up gans for text-to-image synthesis.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10124–10134, 2023.
  * [33] Tero Karras, Miika Aittala, Samuli Laine, Erik Härkönen, Janne Hellsten, Jaakko Lehtinen, and Timo Aila.  Alias-free generative adversarial networks.  Advances in neural information processing systems, 34:852–863, 2021.
  * [34] Tero Karras, Samuli Laine, and Timo Aila.  A style-based generator architecture for generative adversarial networks.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 4401–4410, 2019.
  * [35] Tero Karras, Samuli Laine, Miika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila.  Analyzing and improving the image quality of stylegan.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 8110–8119, 2020.
  * [36] Nikos Kolotouros, Thiemo Alldieck, Andrei Zanfir, Eduard Bazavan, Mihai Fieraru, and Cristian Sminchisescu.  Dreamhuman: Animatable 3d avatars from text.  Advances in Neural Information Processing Systems, 36, 2024.
  * [37] Samuli Laine, Janne Hellsten, Tero Karras, Yeongho Seol, Jaakko Lehtinen, and Timo Aila.  Modular primitives for high-performance differentiable rendering.  ACM Transactions on Graphics (ToG), 39(6):1–14, 2020.
  * [38] Alexandros Lattas, Yiming Lin, Jayanth Kannan, Ekin Ozturk, Luca Filipi, Giuseppe Claudio Guarnera, Gaurav Chawla, and Abhijeet Ghosh.  Practical and scalable desktop-based high-quality facial capture.  In European Conference on Computer Vision, pages 522–537. Springer, 2022.
  * [39] Alexandros Lattas, Stylianos Moschoglou, Stylianos Ploumpis, Baris Gecer, Jiankang Deng, and Stefanos Zafeiriou.  Fitme: Deep photorealistic 3d morphable model avatars.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8629–8640, 2023.
  * [40] Alexandros Lattas, Stylianos Moschoglou, Stylianos Ploumpis, Baris Gecer, Abhijeet Ghosh, and Stefanos Zafeiriou.  Avatarme++: Facial shape and brdf inference with photorealistic rendering-aware gans.  IEEE Transactions on Pattern Analysis and Machine Intelligence, 44(12):9269–9284, 2021.
  * [41] Ruilong Li, Karl Bladin, Yajie Zhao, Chinmay Chinara, Owen Ingraham, Pengda Xiang, Xinglei Ren, Pratusha Prasad, Bipin Kishore, Jun Xing, et al.  Learning formation of physically-based face attributes.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 3410–3419, 2020.
  * [42] Tianye Li, Timo Bolkart, Michael J Black, Hao Li, and Javier Romero.  Learning a model of facial shape and expression from 4d scans.  ACM Trans. Graph., 36(6):194–1, 2017.
  * [43] Tingting Liao, Hongwei Yi, Yuliang Xiu, Jiaxaing Tang, Yangyi Huang, Justus Thies, and Michael J Black.  Tada! text to animatable digital avatars.  arXiv preprint arXiv:2308.10899, 2023.
  * [44] Chen-Hsuan Lin, Jun Gao, Luming Tang, Towaki Takikawa, Xiaohui Zeng, Xun Huang, Karsten Kreis, Sanja Fidler, Ming-Yu Liu, and Tsung-Yi Lin.  Magic3d: High-resolution text-to-3d content creation.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 300–309, 2023.
  * [45] Hongyu Liu, Xuan Wang, Ziyu Wan, Yujun Shen, Yibing Song, Jing Liao, and Qifeng Chen.  Headartist: Text-conditioned 3d head generation with self score distillation.  arXiv preprint arXiv:2312.07539, 2023.
  * [46] Minghua Liu, Chao Xu, Haian Jin, Linghao Chen, Mukund Varma T, Zexiang Xu, and Hao Su.  One-2-3-45: Any single image to 3d mesh in 45 seconds without per-shape optimization.  Advances in Neural Information Processing Systems, 36, 2024.
  * [47] Ruoshi Liu, Rundi Wu, Basile Van Hoorick, Pavel Tokmakov, Sergey Zakharov, and Carl Vondrick.  Zero-1-to-3: Zero-shot one image to 3d object.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 9298–9309, 2023.
  * [48] Xian Liu, Xiaohang Zhan, Jiaxiang Tang, Ying Shan, Gang Zeng, Dahua Lin, Xihui Liu, and Ziwei Liu.  Humangaussian: Text-driven 3d human generation with gaussian splatting.  arXiv preprint arXiv:2311.17061, 2023.
  * [49] Huiwen Luo, Koki Nagano, Han-Wei Kung, Qingguo Xu, Zejian Wang, Lingyu Wei, Liwen Hu, and Hao Li.  Normalized avatar synthesis using stylegan and perceptual refinement.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11662–11672, 2021.
  * [50] Wan-Chun Ma, Tim Hawkins, Pieter Peers, Charles-Felix Chabert, Malte Weiss, Paul E Debevec, et al.  Rapid acquisition of specular and diffuse normal maps from polarized spherical gradient illumination.  Rendering Techniques, 9(10):2, 2007.
  * [51] Luke Melas-Kyriazi, Iro Laina, Christian Rupprecht, and Andrea Vedaldi.  Realfusion: 360deg reconstruction of any object from a single image.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 8446–8455, 2023.
  * [52] Gal Metzer, Elad Richardson, Or Patashnik, Raja Giryes, and Daniel Cohen-Or.  Latent-nerf for shape-guided generation of 3d shapes and textures.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12663–12673, 2023.
  * [53] Oscar Michel, Roi Bar-On, Richard Liu, Sagie Benaim, and Rana Hanocka.  Text2mesh: Text-driven neural stylization for meshes.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13492–13502, 2022.
  * [54] Nasir Mohammad Khalid, Tianhao Xie, Eugene Belilovsky, and Tiberiu Popa.  Clip-mesh: Generating textured meshes from text using pretrained image-text models.  In SIGGRAPH Asia 2022 conference papers, pages 1–8, 2022.
  * [55] Stylianos Moschoglou, Stylianos Ploumpis, Mihalis A Nicolaou, Athanasios Papaioannou, and Stefanos Zafeiriou.  3dfacegan: Adversarial nets for 3d face representation, generation, and translation.  International Journal of Computer Vision, 128:2534–2551, 2020.
  * [56] Norman Müller, Yawar Siddiqui, Lorenzo Porzi, Samuel Rota Bulo, Peter Kontschieder, and Matthias Nießner.  Diffrf: Rendering-guided 3d radiance field diffusion.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4328–4338, 2023.
  * [57] Thomas Müller, Alex Evans, Christoph Schied, and Alexander Keller.  Instant neural graphics primitives with a multiresolution hash encoding.  ACM transactions on graphics (TOG), 41(4):1–15, 2022.
  * [58] Jacob Munkberg, Jon Hasselgren, Tianchang Shen, Jun Gao, Wenzheng Chen, Alex Evans, Thomas Müller, and Sanja Fidler.  Extracting triangular 3d models, materials, and lighting from images.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8280–8290, 2022.
  * [59] Foivos Paraperas Papantoniou, Alexandros Lattas, Stylianos Moschoglou, Jiankang Deng, Bernhard Kainz, and Stefanos Zafeiriou.  Arc2face: A foundation model of human faces.  arXiv preprint arXiv:2403.11641, 2024.
  * [60] Foivos Paraperas Papantoniou, Alexandros Lattas, Stylianos Moschoglou, and Stefanos Zafeiriou.  Relightify: Relightable 3d faces from a single image via diffusion models.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 8806–8817, 2023.
  * [61] Georgios Pavlakos, Vasileios Choutas, Nima Ghorbani, Timo Bolkart, Ahmed AA Osman, Dimitrios Tzionas, and Michael J Black.  Expressive body capture: 3d hands, face, and body from a single image.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10975–10985, 2019.
  * [62] Xu Peng, Junwei Zhu, Boyuan Jiang, Ying Tai, Donghao Luo, Jiangning Zhang, Wei Lin, Taisong Jin, Chengjie Wang, and Rongrong Ji.  Portraitbooth: A versatile portrait model for fast identity-preserved personalization.  arXiv preprint arXiv:2312.06354, 2023.
  * [63] Ben Poole, Ajay Jain, Jonathan T Barron, and Ben Mildenhall.  Dreamfusion: Text-to-3d using 2d diffusion.  In ICLR, 2023.
  * [64] Guocheng Qian, Jinjie Mai, Abdullah Hamdi, Jian Ren, Aliaksandr Siarohin, Bing Li, Hsin-Ying Lee, Ivan Skorokhodov, Peter Wonka, Sergey Tulyakov, et al.  Magic123: One image to high-quality 3d object generation using both 2d and 3d diffusion priors.  arXiv preprint arXiv:2306.17843, 2023.
  * [65] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al.  Learning transferable visual models from natural language supervision.  In International conference on machine learning, pages 8748–8763. PMLR, 2021.
  * [66] Amit Raj, Srinivas Kaza, Ben Poole, Michael Niemeyer, Nataniel Ruiz, Ben Mildenhall, Shiran Zada, Kfir Aberman, Michael Rubinstein, Jonathan Barron, et al.  Dreambooth3d: Subject-driven text-to-3d generation.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 2349–2359, 2023.
  * [67] Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen.  Hierarchical text-conditional image generation with clip latents.  arXiv preprint arXiv:2204.06125, 1(2):3, 2022.
  * [68] George Retsinas, Panagiotis P Filntisis, Radek Danecek, Victoria F Abrevaya, Anastasios Roussos, Timo Bolkart, and Petros Maragos.  3d facial expressions through analysis-by-neural-synthesis.  arXiv preprint arXiv:2404.04104, 2024.
  * [69] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer.  High-resolution image synthesis with latent diffusion models.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10684–10695, 2022.
  * [70] Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfir Aberman.  Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 22500–22510, 2023.
  * [71] Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Wei Wei, Tingbo Hou, Yael Pritch, Neal Wadhwa, Michael Rubinstein, and Kfir Aberman.  Hyperdreambooth: Hypernetworks for fast personalization of text-to-image models.  arXiv preprint arXiv:2307.06949, 2023.
  * [72] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al.  Photorealistic text-to-image diffusion models with deep language understanding.  Advances in neural information processing systems, 35:36479–36494, 2022.
  * [73] Aditya Sanghi, Hang Chu, Joseph G. Lambourne, Ye Wang, Chin-Yi Cheng, Marco Fumero, and Kamal Rahimi Malekshan.  Clip-forge: Towards zero-shot text-to-shape generation.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 18603–18613, June 2022.
  * [74] Axel Sauer, Tero Karras, Samuli Laine, Andreas Geiger, and Timo Aila.  Stylegan-t: Unlocking the power of gans for fast large-scale text-to-image synthesis.  In International conference on machine learning, pages 30105–30118. PMLR, 2023.
  * [75] Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al.  Laion-5b: An open large-scale dataset for training next generation image-text models.  Advances in Neural Information Processing Systems, 35:25278–25294, 2022.
  * [76] Junyoung Seo, Wooseok Jang, Min-Seop Kwak, Hyeonsu Kim, Jaehoon Ko, Junho Kim, Jin-Hwa Kim, Jiyoung Lee, and Seungryong Kim.  Let 2d diffusion model know 3d-consistency for robust text-to-3d generation.  arXiv preprint arXiv:2303.07937, 2023.
  * [77] Tianchang Shen, Jun Gao, Kangxue Yin, Ming-Yu Liu, and Sanja Fidler.  Deep marching tetrahedra: a hybrid representation for high-resolution 3d shape synthesis.  Advances in Neural Information Processing Systems, 34:6087–6101, 2021.
  * [78] William AP Smith, Alassane Seck, Hannah Dee, Bernard Tiddeman, Joshua B Tenenbaum, and Bernhard Egger.  A morphable face albedo model.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5011–5020, 2020.
  * [79] Jingxiang Sun, Bo Zhang, Ruizhi Shao, Lizhen Wang, Wen Liu, Zhenda Xie, and Yebin Liu.  Dreamcraft3d: Hierarchical 3d generation with bootstrapped diffusion prior.  arXiv preprint arXiv:2310.16818, 2023.
  * [80] Junshu Tang, Tengfei Wang, Bo Zhang, Ting Zhang, Ran Yi, Lizhuang Ma, and Dong Chen.  Make-it-3d: High-fidelity 3d creation from a single image with diffusion prior.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 22819–22829, 2023.
  * [81] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  Advances in neural information processing systems, 30, 2017.
  * [82] Duotun Wang, Hengyu Meng, Zeyu Cai, Zhijing Shao, Qianxi Liu, Lin Wang, Mingming Fan, Ying Shan, Xiaohang Zhan, and Zeyu Wang.  Headevolver: Text to head avatars via locally learnable mesh deformation.  arXiv preprint arXiv:2403.09326, 2024.
  * [83] Hao Wang, Yitong Wang, Zheng Zhou, Xing Ji, Dihong Gong, Jingchao Zhou, Zhifeng Li, and Wei Liu.  Cosface: Large margin cosine loss for deep face recognition.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2018.
  * [84] Qixun Wang, Xu Bai, Haofan Wang, Zekui Qin, and Anthony Chen.  Instantid: Zero-shot identity-preserving generation in seconds.  arXiv preprint arXiv:2401.07519, 2024.
  * [85] Zhengyi Wang, Cheng Lu, Yikai Wang, Fan Bao, Chongxuan Li, Hang Su, and Jun Zhu.  Prolificdreamer: High-fidelity and diverse text-to-3d generation with variational score distillation.  In Advances in Neural Information Processing Systems, volume 36, 2023.
  * [86] Jiale Xu, Xintao Wang, Weihao Cheng, Yan-Pei Cao, Ying Shan, Xiaohu Qie, and Shenghua Gao.  Dream3d: Zero-shot text-to-3d synthesis using 3d shape prior and text-to-image diffusion models.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 20908–20918, 2023.
  * [87] Yuxuan Yan, Chi Zhang, Rui Wang, Yichao Zhou, Gege Zhang, Pei Cheng, Gang Yu, and Bin Fu.  Facestudio: Put your face everywhere in seconds.  arXiv preprint arXiv:2312.02663, 2023.
  * [88] Hu Ye, Jun Zhang, Sibo Liu, Xiao Han, and Wei Yang.  Ip-adapter: Text compatible image prompt adapter for text-to-image diffusion models.  arXiv preprint arXiv:2308.06721, 2023.
  * [89] Jonathan Young.  xatlas.  https://github.com/jpcy/xatlas.  Accessed: 2022-05-20.
  * [90] Seyma Yucer, Furkan Tektas, Noura Al Moubayed, and Toby P Breckon.  Measuring hidden bias within face recognition via racial phenotypes.  In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 995–1004, 2022.
  * [91] Yifei Zeng, Yuanxun Lu, Xinya Ji, Yao Yao, Hao Zhu, and Xun Cao.  Avatarbooth: High-quality and customizable 3d human avatar generation.  arXiv preprint arXiv:2306.09864, 2023.
  * [92] Longwen Zhang, Qiwei Qiu, Hongyang Lin, Qixuan Zhang, Cheng Shi, Wei Yang, Ye Shi, Sibei Yang, Lan Xu, and Jingyi Yu.  Dreamface: Progressive generation of animatable 3d faces under text guidance.  arXiv preprint arXiv:2304.03117, 2023.
  * [93] Lvmin Zhang, Anyi Rao, and Maneesh Agrawala.  Adding conditional control to text-to-image diffusion models.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 3836–3847, 2023.

##  Appendix A Additional Results

We provide additional results for our methods. Figure 8 displays neutral
expression for additional subjects, showcasing realistic, high-detail, ID-
conditioned 3D assets that can be relit under various conditions. Figure 11
reports additional comparisons with text-to-3D SDS pipelines specialized in
the generation of human avatars via SDS, while Figure 12 displays additional
comparisons with methods that leverage text and images to create 3D assets.
All methods are presented using the same rendering conditions. As apparent
from the figures, the comparisons are consistent with the results presented in
the main manuscript. ID-to-3D achieves the highest degree of geometric and
texture quality. Figure 9 present ID-conditioned examples for AI-generated
identities created starting from the test cases of the NHPM dataset,
demonstrating results consistent with the findings presented in the main
manuscript.

|  |  |
---|---|---|---
|  |  |
|  |  |
|  |  |

Figure 8: 3D assets created by ID-to-3D can be photorealistically rendered in
common 3D engines.

Visualization in Video Format. Taking in input only unconstrained pictures of
a subject, ID-to-3D produces high-fidelity shapes and textures that can be
photorealistically relighted in an arbitrary environment and illumination.
Moreover, ID-to-3D establishes a new State-of-the-Art in the generation of 3D
consistent human heads. Relighting videos for ID-to-3D’s 3D heads assets that
are shown in this paper, can be explored using our project page, together with
video comparisons of existing SDS methods under fixed rendering conditions.

Impact of Randomized Lighting We provide an ablation study on the impact of
randomized lighting during texture training. As apparent in Figure 10 the use
of this augmentation, paired with an albedo-oriented 2D guidance, allows for
the generation of crisp details and vibrant colors, not achievable with
standard training.

---

Figure 9: ID-consistent texture and geometry generation. The input images for
ArcFace conditioning (Left) have been created using Stable-Diffusion. Normal
maps (Center) are displayed next to renderings in studio lighting (Right). ID-
to-3D creates ID-consistent geometry and textures with precise alignment.

Figure 10: Randomized lighting (Left) creates sharper textures and realist
colors. Results generated by training with fix lighting (Right) struggle to
create crisp details.

Impact of Different Identity Conditionings Our method uses identity embedding
to guide the geometry and texture generation of facial details, and is
therefore bounded by the ability of these representations to convey in a
concise and distinct manner the unique features of each face identity. Figure
13 compares the results of our final pipeline with that using less robust
identity conditioning. We compare a 3D asset obtained using the identity
embeddings derived from five images, with two different 3D heads created using
a single in-the-wild picture of the same subject. Our pipeline is capable of
creating 3D consistent heads in all the considered cases. However, The ArcFace
features collected from a small poll of images provide better identity
conditioning, with fewer texture artifacts and closer ID similarity with the
reference identity.

Figure 11: Comparisons with text-to-3D generation methods. ID-to-3D creates
realistic 3D heads when compared with human-centric SDS methods based on text
prompts.

Figure 12: Evaluating text+image-to-3D generation techniques. ID-to-3D
consistently produces high-quality 3D heads across various viewpoints,
outperforming state-of-the-art SDS methods utilizing text prompts and images.
Figure 13: Impact of Different ID-embeddings. All the 3D heads are generated
using a Neutral expression and rendered under the same lighting conditions.
The first row displays the images used as input. Geometry is displayed as
normal maps in camera coordinates.

##  Appendix B Implementation Details

#### Implementation Details: 2D guidance

We develop the geometry-oriented and albedo-oriented 2D guidance models by
finetuning a large-scale diffusion model on a small dataset of real 3D scans.
We follow [30] and use normal maps in camera coordinates as a 2D proxy for
geometric information. We select albedo maps as texture information. We
modified a large-scale Stable Diffusion model pre-trained to perform
photorealistic 2D face portrait generation [88], and finetuned the geometry
and albedo model in two stages to minimize identity drifting. First, we
modified the architecture with LoRA layers to perform a style transfer task.
Then, we modified and trained the same architecture to accommodate expression
conditioning. Both fine-tuning is done for $200k$ iterations with early
stopping with a lr of $1e-4$ on $8$ Nvidia-V100 GPUS. We use $16$ as the
hidden LoRA size for self-attention layers. We use LoRA for cross-attention
layers with a hidden size $32$.

Figure 14: Camera aware 2D generation enables the generation of samples
aligned with different camera poses. We display images for back views (Left),
side views (Center) and front views (Right).

Figure 15: Generation of id and expression conditioned images. The geometry-
oriented model (Right), creates high quality 2D normal maps that can be used
as proxy for geometric information. The albedo-oriented 2D model (Left)
generate images with consistent lighting that can be used as guidance for
high-quality texture generation.

For training, we use the recently released neural head parameter data set
[23], comprising $255$ subjects and $23$ facial expressions. We selected $25$
subject to test the drifting ID during training. For the geometry-oriented 2D
guidance model, we created a training set of $250k$ samples that extract 2D
normal maps in camera coordinates from random subjects in random expressions
and camera poses. For the texture-oriented 2D guidance model, we extracted a
dataset of $250k$ samples from the texture of the objects. We extracted text
descriptions for each render and supplemented the training with a text prompt
containing camera pose, gender, ethnicity, and age group for each subject
(e.g., ’A front view normal map face portrait of a young Caucasian female on a
white background’). Figure 14 and 15 display images generated by our 2D
guidance models for a range of identities, poses, and expressions. As visible,
after convergence, our models are able to convincingly separate geometric and
texture information for a given subject, realistically portrait side and back
views, and reliably convey a wide range of id-consistent expressions.

#### Implementation Details: geometry and texture generation

Given an ArcFace identity, geometry generation is achieved through a two-stage
pipeline. First, we follow [10] and initialize the 3D geometry with a FLAME
head template by regression fitting. Then, we train the geometry model using
randomly selected camera poses and expression conditioning. We used a neural
head expression model as a hybrid 3D representation for geometry. We used
HashGrid [57] encoding and a grid with $256$ resolution. We used a $4$ layers
transformer and jointly trained $3$, $6$, or $13$ expressions each associated
with its unique latent code of dimensions $32$. Our model can train a neutral
identity representation in $6000$ iterations, which takes approximately $30$
minutes on v100 GPUS. Our model can jointly train a larger set of $13$
expressions in approximately $4$ hours. We use a learning rate of $1e-4$. We
used a random sampling strategy and considered time steps in the range [$200$,
$700$] for the first half of the training. We use an annealing strategy with
time steps in the range [$200$, $50$] in the second half of the training. We
use a score distillation sampling loss with a guidance weight $10$ and a
Laplacian smoothing loss with a weight $5000$.

Given a learned geometry model, the texture generation is achieved via a
three-stage pipeline conditioned on the same ArcFace identity. We train the
texture model using randomly selected camera poses and expression
conditioning. We use random lighting during training, using a list of $60$ HDR
maps and a set of random augmentations. We trained a $3$ layers transformer
and jointly trained $3$, $6$, or $13$ expressions each associated with its
unique latent code of dimensions $32$. Our model can train a neutral identity
representation in $2000$ iterations, taking approximately $15$ minutes on v100
GPUS. Our model can jointly train a larger set of $13$ expressions in
approximately $10000$ iterations. As first step, the model is optimized to
learn only the diffuse term of the textures. This portion of the training uses
$80\%$ of all available iterations, a learning rate of $0.01$, and time steps
in the range [$50$, $900$]. The second step then proceeds to jointly optimize
the roughness and metallic term of the texture together with the diffuse term,
using the same learning rate but a different timesteps range [$50$, $500$].
Lastly, the pipeline uses an optional and quick refinement stage with the goal
of introducing high-frequency details. The last $20$ iterations use a large
photorealistic text-to-image model as guidance and deploy timesteps in the
range [$50$, $100$]. In all stages, we use a score distillation sampling loss
with a weight $10$.

#### Implementation Details: User study

In the main paper, we evaluate the 3D heads generated by ID-to-3D in terms of
perceptual geometric quality and texture quality through a user study. We
follow [30] and compare with four state-of-the-art methods on 30 prompts. We
collect the preferences of 50 participants using anonymous online survey
forms. To fairly compare among methods, we created 3D assets using comparable
input (i.e. same text and image prompt), and gathered as visualization a GIF
accumulating 360 renderings under the same lighting and camera conditions. To
assess geometric quality, we remove the contribution of texture by visualizing
normal maps. Each volunteer was asked to: 1) Choose which 3D head among the 5
options represents the most appealing, detailed, and ready-to-use 3D object.
2) Choose which 3D head among the 5 options represents the most appealing,
detailed, and ready-to-use 3D geometry. As visible in the main manuscript, ID-
to-3Daccumulates higher preferences for both metrics, showing superior
performance compared to both human-specific text-based methods and image-to-3D
baselines.

#### Results Rendering

For the rendering of results shown in this paper, as well as in the
supplemental video files, an off-the-shelf commercial rendering engine has
been used [2], highlighting the easy of integration of our results to the
tools of the industry. In that manner, any similar standard rendering engine
could also be used.