11institutetext:  Samsung AI Center – Moscow 22institutetext: Skolkovo
Institute of Science and Technology 33institutetext: Yandex Armenia
https://samsunglabs.github.io/rome/

# Realistic One-shot Mesh-based Head Avatars

Taras Khakhulin 1122 Vanessa Sklyarova 1122
Victor Lempitsky 33 Egor Zakharov 1122

###### Abstract

We present a system for realistic one-shot mesh-based human head avatars
creation, ROME for short. Using a _single_ photograph, our model estimates a
person-specific head mesh and the associated neural texture, which encodes
both local photometric and geometric details. The resulting avatars are rigged
and can be rendered using a neural network, which is trained alongside the
mesh and texture estimators on a dataset of in-the-wild videos. In the
experiments, we observe that our system performs competitively both in terms
of head geometry recovery and the quality of renders, especially for the
cross-person reenactment.

##  1 Introduction

Personalized human avatars are becoming a key technology across several
application domains, such as telepresence, virtual worlds, and online
commerce. In many practical cases, it is sufficient to personalize only a part
of the avatar’s body, while the remaining areas can then be picked from a pre-
defined library of assets or omitted from the interface. Towards this end,
many applications require personalization at the head level, i.e., the
creation of person-specific head models, thus making it an important and
viable intermediate step between personalizing only the face and the entire
body. Alone, face personalization is often insufficient, while the full-body
reconstruction remains a complicated task and leads to the reduced quality of
the models or requires cumbersome data collection.

Acquiring human avatars from a single photograph (in a “one-shot” setting)
offers the highest convenience for the end-user. However, their creation
process is particularly challenging and requires strong priors on human
geometry and appearance. To this end, parametric models are long known to
offer good personalization solutions [1] and were recently adapted to one-shot
performance [2, 3, 4]. Such models can be learned from a relatively small
dataset of 3D scans and represent geometry and appearance via textured meshes,
making them compatible with many computer graphics applications and pipelines.
However, they cannot be trivially expanded to the whole head region due to the
large geometric variability of the non-facial parts such as hair and neck.

Our proposed system addresses this issue and allows parametric face models to
represent the non-facial parts. In order to handle the increased geometric and
photometric variability, we train our method on a large dataset of in-the-wild
videos [5] and use neural networks to parameterize both the geometry and the
appearance. For the appearance modeling, we follow the deferred neural
rendering [6] paradigm and employ a combination of neural textures and
rendering networks. In addition, a neural rendering framework [7] is used to
enable end-to-end training and achieve high visual realism of the resulting
head models. After training, the geometric and appearance networks can be
conditioned on the information extracted from a single photograph, enabling
one-shot realistic avatar generation.

To the best of our knowledge, our system is the first that is capable of
creating realistic personalized human head models in a rigged mesh format from
a single photograph. This distinguishes our model from a growing class of
approaches that a) recover neural head avatars without explicit geometry [8,
9, 10, 11], b) can personalize the face region but not the whole head [12, 13,
1, 2], and c) from commercial systems that create non-photorealistic mesh
avatars from a single image [14, 15]. Alongside our main model, we also
discuss its simplified version based on a linear blendshapes basis and show
how to train it using the same video dataset. Below, we refer to the avatars
generated by our system as ROME avatars (Realistic One-shot Mesh-based
avatars).

|  |  |  |  |  |  |
---|---|---|---|---|---|---|---
|  |  |  |  |  |  |
Source | Driver | Mesh | Render | Source |  Driver | Mesh | Render

Figure 1: Our system creates realistic mesh-based avatars from a single source
photo. These avatars are rigged, i.e., they can be driven by the animation
parameters from a different driving frame. At the same time, our obtained
meshes and renderes achieve a high degree of personalization in both
appearance and geometry and are trained in an end-to-end fashion on a dataset
of in-the-wild videos without any additional 3D supervision.

##  2 Related work

Parametric models of human faces. Over the recent decades, 3D face
reconstruction methods have been actively employed to tackle the problems of
face tracking and alignment [16, 3], face recognition [17, 18], and generative
modelling [12, 13, 19, 20, 21, 22]. In all these scenarios, statistical mesh-
based models, aka parametric models [1], remain one of the widely used tools
[23, 24]. State-of-the-art parametric models for human heads consist of rigged
meshes [25] which support a diverse range of animations via disentangled shape
and expression blendshapes and rigid motions for the jaw, neck, and eyeballs.
However, they only provide reconstructions for the face, ears, neck, and
forehead regions, limiting the range of applications. Including full head
reconstruction (i.e., hair and shoulders) into these parametric models is
possible, but existing approaches require significantly more training data to
be gathered in the form of 3D scans. Instead, in our work, we propose to
leverage existing large-scale datasets [5] of in-the-wild videos via the
learning-by-synthesis paradigm without any additional 3D annotations.

Neural 3D human head models. While parametric models provide sufficient
reconstruction quality for many downstream applications, they are not able to
depict the fine appearance details that are needed for photorealistic
modelling. In recent years, the problem of representing complex geometry and
appearance of humans started being addressed using high-capacity deep neural
networks. Some of these works use strong human-specific priors [20, 2, 26,
27], while others fit high-capacity networks to data without the use of such
priors [28, 29, 19, 22, 21, 30, 31]. The latter methods additionally differ by
the type of data structure used to represent the geometry, namely, mesh-based
[2, 22, 21, 32], point-based [28, 33], and implicit models [19, 29, 20, 26,
31, 27, 34]. Additionally, recently there have emerged the hybrid models [35,
36] where authors integrate face priors from parametric models with implicit
representations to learn geometry and rendering for the specific person from
the video.

However, mesh-based models arguably represent the most convenient class of
methods for downstream applications. They provide better rendering quality and
better temporal stability than point-based neural rendering. Also, unlike
methods based on implicit geometry, mesh-based methods preserve topology and
rigging capabilities and are much faster during fitting and rendering.
However, current mesh-based methods either severely limit the range of
deformations [2], making it infeasible to learn more complex geometry like
hair and clothed shoulders, operate in the multi-shot setting [32] or require
3D scans as training data [22, 21]. Our proposed method is also mesh-based,
but we allow the prediction of complex deformations without 3D supervision and
using a single image, lifting the limitations of the previous and concurrent
works.

One-shot neural head models. Advances in neural networks also led to the
development of methods that directly predict images using large ConvNets
operating in the 2D image domain, with effectively no underlying 3D geometry
[9, 10, 11] or very coarse 3D geometry [8]. These methods achieve state-of-
the-art realism [8], use in-the-wild images or videos with no 3D annotations
for training, and can create avatars from a single image. However, the lack of
an explicit geometric model makes these models incompatible with many real-
world applications and limits the span of camera poses that these methods can
handle.

Figure 2:  Overview of our approach and the detailed scheme of the  head mesh
reconstruction. Given the source photo, we first estimate a _neural texture_
that encodes local geometric and photometric details of visible and occluded
parts. We then use a pre-trained system [2] for face reconstruction to
estimate an initial mesh with a reconstructed facial part. We call this step
face and 3D pose estimation. During head mesh reconstruction (bottom), using
the estimated neural texture and the initial mesh, we predict the offsets for
the mesh vertices, which do not correspond to a face. The offsets are
predicted with a combination of a convolutional network $E_{\text{geom}}$ and
a perceptron network $G_{\text{geom}}$. We then render the personalized head
mesh using the camera parameters, estimated by a pre-trained regressor [2]
while superimposing the predicted neural texture. Finally, the rendering
network $E_{\text{img}}$ estimates the RGB image and the mask from the render.

Neural mesh rendering. Recently, approaches that combine explicit data
structures (point clouds or meshes) with neural image generation have emerged.
These methods gained popularity thanks to the effectiveness of the pioneering
Deferred Neural Rendering system [6], as well as recent advances in
differentiable mesh rendering [7, 37, 38]. Neural mesh rendering uses 2D
convolutional networks to model complex photometric properties of surfaces. It
achieves high realism of renders with fine details present even when they are
missing in the underlying geometric model. In this work, we adapt these
advances to human head modelling while training using a large dataset of in-
the-wild videos.

##  3 Method

Our goal is to build a system that jointly learns to produce photorealistic
renders of human heads and estimate their 3D meshes using only a _single
image_ without any 3D supervision.

To achieve that, we use a large-scale dataset [5] of in-the-wild videos with
talking speakers. All frames in each video are assumed to depict the same
person in the same environment (defined by lighting, hairstyle, and person’s
clothing). At each training step, we sample two random frames $\mathbf{x}_{s}$
and $\mathbf{x}_{t}$ from a random training video. Our goal is to reconstruct
and render the target image $\hat{\mathbf{x}}_{t}$ given a) the personal
details and the face shape extracted from the source image $\mathbf{x}_{s}$,
as well as b) the head pose, the facial expression, and the camera pose
estimated from the target image $\mathbf{x}_{t}$. The final reconstruction
loss is backpropagated and used to update the parameters of the model’s
components.

After training, we can create a personalized head model by estimating all
parameters from a single image. This model can then be _animated_ using face
tracking parameters extracted from any talking head sequence and rendered from
a range of viewpoints similar to those present in the training dataset (Figure
1).

###  3.1 Model overview

In our model, we jointly train multiple neural networks that perform rendering
and mesh reconstruction. The training pipeline proceeds as follows (Figure 2):

Neural texture estimation. The source image $\mathbf{x}_{s}$ is encoded into a
neural texture $\mathbf{T}_{s}$, which describes both person-specific
appearance and geometry. The encoding is done by a convolutional neural
network $E_{\text{tex}}$.

Face and 3D pose estimation. In parallel, we apply a pre-trained DECA system
[2] for face reconstruction to both the source and the target image, which
estimates facial shape, expression, and head pose. Internally, it uses the
FLAME parametric head model [25], which includes mesh topology, texture
mapping, and blendshapes. We use the face shape from the source image
$\mathbf{x}_{s}$ as well as the facial expression and the camera pose from the
target image $\mathbf{x}_{t}$ for further processing.

Head mesh reconstruction. The vertices of the DECA mesh with personalized face
region and generic non-facial parts are rendered into an $xyz$-coordinate
texture using the predefined texture mapping. The $xyz$-texture and the neural
texture $\mathbf{T}_{s}$ are concatenated and processed with the U-Net network
[39] $E_{\text{geom}}$ into a new texture map $\mathbf{Z}_{t}$, called latent
geometry map. The 3D displacements for each mesh vertex are then decoded
independently by the multi-layer perceptron $G_{\text{geom}}$ that predicts a
3D offset $\Delta\hat{\mathbf{v}}$ for each vertex. This step reconstructs the
personalized model for non-face parts of the head mesh. The obtained
reconstructions are compatible with the topology/connectivity of the FLAME
mesh [25].

Deferred neural rendering. The personalized head mesh is rendered using the
pose estimated by DECA for the target image and with the superimposed neural
texture. The rendered neural texture and the rasterized surface normals are
concatenated and processed by the decoding (rendering) U-Net network
$E_{\text{img}}$ to predict the rendered image $\hat{\mathbf{x}}_{t}$ and the
segmentation mask $\hat{\mathbf{s}}_{t}$. During training, the difference
between the predictions and the ground truths is used to update all components
of our system.

Below we discuss our system and its training process in more detail. We also
describe a training procedure for a simplified version of our model, which
represents head geometry using a linear basis of blendshapes.

###  3.2 Parametric face modeling

Our method uses a predefined head mesh with the corresponding topology,
texture coordinates $w$, and rigging parameters, which remain fixed for all
avatars. More specifically, we use FLAME [25] head model that has $N$ base
vertices $\mathbf{v}_{\text{base}}~{}\in~{}\mathbb{R}^{3N}$, and two sets of
$K$ and $L$ basis vectors (blendshapes) that encode shape
$\mathcal{B}~{}\in~{}\mathbb{R}^{3N\times K}$ and expression
$\mathcal{D}~{}\in~{}\mathbb{R}^{3N\times L}$. The reconstruction process is
carried out in two stages. First, the basis vectors are blended using the
person- and expression-specific vectors of linear coefficients $\phi$ and
$\psi$. Then, the linear blend skinning [25] function $\mathcal{W}$ is
applied, parameterized by the angles $\theta$, which rotates the predefined
groups of vertices around linearly estimated joints. The final reconstruction
in world coordinates can be expressed as follows:

$\mathbf{v}(\phi,\psi,\theta)=\mathcal{W}\Big{(}\mathbf{v}_{\text{base}}+\mathcal{B}\phi+\mathcal{D}\psi,\
\theta\Big{)}.$

In previous works [12], a similar set of parameters for the 3DMM [1]
parametric model was obtained via photometric optimization. More recently,
learning-based methods [2, 3] capable of feed-forward estimation started to
emerge. In our work, given an input image, we use a pre-trained feed-forward
DECA system [2] to estimate $\phi,\psi$, $\theta$, and the camera parameters.

During training, we apply DECA to both source image $\mathbf{x}_{s}$ and the
target image $\mathbf{x}_{t}$. The face shape parameters $\phi_{s}$ from the
source image $\mathbf{x}_{s}$ alongside the expression $\psi_{t}$, head pose
$\theta_{t}$ and camera parameters from the target image $\mathbf{x}_{t}$ are
then used to reconstruct the initial FLAME vertices
$\mathbf{v}_{t}=\mathbf{v}(\phi_{s},\psi_{t},\theta_{t})$, as well as camera
transform $\mathcal{P}_{t}$.

###  3.3 Head mesh reconstruction

The FLAME vertices $\mathbf{v}_{t}$ estimated by DECA provide good
reconstructions for the face region but lack any person-specific details in
the remaining parts of the head (hair and shoulders). To alleviate that, we
predict person-specific mesh offsets for non-facial regions while preserving
the face shape predicted by DECA. We additionally exclude ear regions since
their geometry in the initial mesh is too complex to be learned from in-the-
wild video datasets.

These mesh offsets are estimated in two steps. First, we encode both the
$xyz$-coordinate texture and the neural texture $\mathbf{T}_{s}$ into the
latent geometry texture map $\mathbf{Z}_{t}$ via a U-Net network
$E_{\text{geom}}$. It allows the produced latent map to contain both positions
of the initial vertices $\mathbf{v}_{t}$ and their semantics, provided by the
neural texture.

From $\mathbf{Z}_{t}$ we obtain the vectors $\mathbf{z}_{t}^{w}$ by bilinear
interpolation at the fixed texture coordinates $w$. The vectors
$\mathbf{z}_{t}^{w}$ and their coordinates $w$ are then concatenated and
passed through a multi-layer perceptron $G_{\text{geom}}$ to predict the
coefficients $\hat{\mathbf{m}}_{t}\in\mathbb{R}^{3N\times 3}$ independently
for each vertex in the mesh. These coefficients are multiplied elementwise by
the normals $\mathbf{n}_{t}$, calculated for each vertex in $\mathbf{v}_{t}$,
to obtain the displacements:
$\Delta\hat{\mathbf{v}}_{t}=\hat{\mathbf{m}}\odot\mathbf{n}_{t}$. These
displacements are then zeroed out for face and ear regions, and the final
reconstruction in world coordinates is obtained as follows:
$\hat{\mathbf{v}}_{t}=\mathbf{v}_{t}+\Delta\hat{\mathbf{v}}_{t}$.

###  3.4 Deferred neural rendering

We render the reconstructed head vertices $\hat{\mathbf{v}}_{t}$ using the
topology and texture coordinates $w$ from the FLAME model with the
superimposed neural texture $\mathbf{T}_{s}$. For that, we use a
differentiable mesh renderer $\mathcal{R}$ [7] with the camera transform
$\mathcal{P}_{t}$ estimated by DECA for the target image $\mathbf{x}_{t}$.

The resulting rasterization, which includes both the neural texture and the
surface normals, is processed by the rendering network $E_{\text{img}}$ to
obtain the predicted image $\hat{\mathbf{x}}_{t}$ and the segmentation mask
$\hat{\mathbf{s}}_{t}$. $E_{\text{img}}$ consists of two U-Nets that
separately decode an image and a mask. The result of the deferred neural
rendering is the reconstruction of the target image $\hat{\mathbf{x}}_{t}$ and
its mask $\hat{\mathbf{s}}_{t}$, which are compared to the ground-truth image
$\mathbf{x}_{t}$ and mask $\mathbf{s}_{t}$ respectively.

###  3.5 Training objectives

In our approach, we learn the geometry of hair and shoulders, which are not
reconstructed by the pre-trained DECA estimator, without any ground-truth 3D
supervision during training. For that we utilize two types of objectives:
segmentation-based geometric losses $\mathcal{L}_{\text{geom}}$ and
photometric losses $\mathcal{L}_{\text{photo}}$.

We found that explicitly assigning subsets of mesh vertices to the neck and
the hair regions helps a lot with the quality of final deformations. It allows
us to introduce a topological prior for the predicted offsets, which is
enforced by.

To evaluate the geometric losses, we calculate two separate occupancy masks
using a soft rasterization operation [37]. First,
$\hat{\mathbf{o}}_{t}^{\text{hair}}$ is calculated with detached neck
vertices, so that the gradient flows through that mask only to the offsets
corresponding to the hair vertices, and then $\hat{\mathbf{o}}_{t}$ is
calculated with detached hair vertices. We match the hair occupancy mask to
the ground-truth mask $\mathbf{s}^{\text{hair}}_{t}$ (which covers the hair,
face, and ears), and the estimated occupancy mask to the whole segmentation
mask $\mathbf{s}_{t}$:
$\mathcal{L}_{\text{occ}}=\lambda_{\text{hair}}\big{\|}\hat{\mathbf{o}}_{t}^{\text{hair}}-\mathbf{s}_{t}^{\text{hair}}\big{\|}_{2}^{2}+\lambda_{\text{o}}\big{\|}\hat{\mathbf{o}}_{t}-\mathbf{s}_{t}\big{\|}_{2}^{2}.$

We also use an auxiliary Chamfer loss to ensure that the predicted mesh
vertices cover the head more uniformly. Specifically, we match the 2D
coordinates of the mesh vertices projected into the target image to the head
segmentation mask. We denote the subset of predicted mesh vertices, visible in
the target image, as
$\hat{\mathbf{p}}_{t}=\mathcal{P}^{\prime}_{t}(\hat{\mathbf{v}}_{t})$, and the
number of these vertices as $N_{t}$, so that
$\hat{\mathbf{p}}_{t}\in\mathbf{R}^{N_{t}\times 2}$. Notice that operator
$\mathcal{P}^{\prime}_{t}$ here not only does the camera transformation but
also discards the $z$ coordinate of the projected mesh vertices. To compute
the loss, we then sample $N_{t}$ 2D points from the segmentation mask
$\mathbf{s}_{t}$ and estimate the Chamfer distance between the sampled set of
points $\mathbf{p}_{t}$ and the vertex projections:

$\begin{split}\mathcal{L}_{\text{chm}}=&\frac{1}{2N_{t}}\sum_{\hat{p}_{t}\in\hat{\mathbf{p}}_{t}}\Big{\|}\hat{p}_{t}-\arg\min_{p\in\mathbf{p}_{t}}\big{\|}p-\hat{p}_{t}\big{\|}\Big{\|}+\\\
&\frac{1}{2N_{t}}\sum_{p_{t}\in\mathbf{p}_{t}}\Big{\|}p_{t}-\arg\min_{\hat{p}\in\hat{\mathbf{p}}_{t}}\big{\|}\hat{p}-p_{t}\big{\|}\Big{\|}.\end{split}$

Lastly, we regularize the learned geometry using the Laplacian penalty [40].
Initially, we found that regularizing offsets $\Delta\hat{\mathbf{v}}$ worked
better than regularizing full coordinates $\hat{\mathbf{v}}$ and stuck to that
approach for all experiments. Our version of the Laplacian loss can be written
as:

$\mathcal{L}_{\text{lap}}=\dfrac{1}{V}\sum_{i=1}^{V}\Big{\|}\Delta\hat{\mathbf{v}}_{i}-\dfrac{1}{\mathcal{N}(i)}\sum\limits_{j\in\mathcal{N}(i)}\Delta\hat{\mathbf{v}}_{j}\Big{\|}_{1},$

where $\mathcal{N}(i)$ denotes a set indices for vertices adjacent to the
$i$-th vertex in the mesh.

We also use photometric optimization that matches the predicted and the ground
truth images. This allows us to obtain photorealistic renders and aid in
learning proper geometric reconstructions. We utilize perceptual loss
$\mathcal{L}_{\text{per}}$ [41], the face recognition loss
$\mathcal{L}_{\text{idt}}$ [42] and adversarial loss
$\mathcal{L}_{\text{adv}}$ [43, 44]. We also use the Dice loss
$\mathcal{L}_{\text{seg}}$ [45] to match the predicted segmentation masks.

The final objective is weighted sum of the geometric and the photometric
losses described above.

###  3.6 Linear deformation model

In addition to the full non-linear model introduced above, we consider a
simplified parametric model with a linear basis of offsets
$\Delta\hat{\mathbf{v}}$. While this model is similar to parametric models
[25, 46], we still do not use 3D scans for training and instead obtain our
linear model by “distilling” the non-linear model. Additionally, we train a
feed-forward estimator that predicts the linear coefficients from the input
image.

The motivation for training this additional model is to show that the
deformations learned by our method can be approximated using a system with a
significantly lower capacity. Such a simple regression model can be easier to
apply for inference on low-performance devices.

To train the linear model, we first obtain the basis of offsets
$\mathcal{F}\in\mathbb{R}^{3N\times K}$, which is similar to the blendshapes
used in the FLAME parametric model. This basis is obtained by applying a low-
rank PCA [47] to the matrix of offsets
$\Delta\hat{\mathbf{V}}\in\mathbb{R}^{3N\times M}$, calculated using $M$
images from the dataset. Following [25], we discard most of the basis vectors
and only keep $K$ components corresponding to maximal singular values. The
approximated vertex offsets $\tilde{\mathbf{v}}$ for each image can then be
estimated as following $\tilde{\mathbf{v}}=\mathcal{F}\eta$, where $\eta$ is
obtained by applying the pseudo-inverse of a basis matrix $\mathcal{F}$ to the
corresponding offsets $\Delta\hat{\mathbf{v}}$:
$\eta=\big{(}\mathcal{F}^{T}\mathcal{F}\big{)}^{-1}\mathcal{F}^{T}\Delta\hat{\mathbf{v}}$

We then train the regression network by estimating a vector of basis
coefficients $\eta_{t}$, given an image $\mathbf{x}_{t}$. For that, we
minimize the mean squared error (MSE) loss
$\|\hat{\eta}_{t}-\eta_{t}\|_{2}^{2}$ between the estimated coefficients and
the ground truth, as well as the segmentation loss $\mathcal{L}_{\text{occ}}$
and a Chamfer distance between predicted and ground truth meshes.

|  |  |  |  |  |
---|---|---|---|---|---|---
|  |  |  |  |  |
|  |  |  |  |  |
Source | H3D-Net | ROME | GT | H3D-Net | ROME |
Figure 3: Qualitative comparison of the representative cases from the H3DS
dataset. While neither of the two methods achieves perfect results, arguably,
ROME achieves more realistic renders and better matches the head geometry than
H3D-Net in the single-shot mode. Furthermore, an important advantage of ROME
is that the resulting avatars are ready for animation and are obtained in a
feed-forward manner without the lengthy fine-tuning process employed by
H3D-Net.

##  4 Experiments

We train our models on the VoxCeleb2 [5] dataset of videos. This large-scale
dataset contains an order of $10^{5}$ videos of $10^{3}$ different speakers.
It is widely used [48, 8, 11] to train talking head models. However, the main
drawback of this dataset is the mixed quality of videos and the heavy bias
towards frontal poses.

To address these well-known limitations, we process this dataset using an off-
the-shelf image quality analysis model [49] and a 3D face-alignment network
[50]. We then filter out the data which has poor quality and non-diverse head
rotations. Our final training dataset has $\approx{}15000$ sequences. We note
that filtering/pruning does not fully solve the problem of head rotation bias,
and our method still works best in frontal views. For more details, please
refer to the supplementary materials.

We also use the H3DS [20] dataset of photos with associated 3D scans to
evaluate the quality of head reconstructions.

###  4.1 Implementation details

In the experiments, unless noted otherwise, we train all architectures jointly
and end-to-end. We use the following weights: $\lambda_{\text{hair}}=10$,
$\lambda_{\text{per}}=1$, $\lambda_{\text{idt}}=0.1$,
$\lambda_{\text{adv}}=0.1$, $\lambda_{\text{seg}}=10$, and enable the neck and
the 2D Chamfer loss $\lambda_{\text{chm}}=0.01$) and
$\lambda_{\text{lap}}=10$. We ablate all geometry losses and method parts
below.

We train our models at $256\times 256$ resolution using ADAM [51] with the
fixed learning rate of $10^{-4}$, $\beta_{1}=0$, $\beta_{2}=0.999$, and a
batch size of $32$. For more details, please refer to the supplementary
materials.

###  4.2 Evaluation

Table 1: Evaluation results on the H3DS dataset in the one-shot scenario for
our models, H3D-Net, and DECA. We compute Chamfer distance (lower is better)
across all available scans, reconstructed from three different viewpoints.
Both of the ROME variants significantly exceed H3D-Net in the one-shot
reconstruction quality.

Method | DECA | H3D-Net | ROME | LinearROME
---|---|---|---|---
Chamfer Distance | $15.0$ | $15.1$ | $12.6$ | $12.5$

3D reconstruction. We evaluate our head reconstruction quality using a novel
H3DS dataset [20]. We compare against the state-of-the-art head reconstruction
method H3D-Net [20], which uses signed distance functions to represent the
geometry. While providing great reconstruction quality in the sparse-view
scenario, their approach has several limitations. For example, H3D-Net
requires a dataset of full head scans to learn the prior on head shapes.
Additionally, its results do not have fixed topology or rigging and their
method requires fine-tuning per scene, while our method works in a feed-
forward way.

We carry out the comparison with H3D-Net in a single-view scenario, which is
native for our method but is beyond the capabilities stated by the authors in
the original publication [20]. However, to the best of our knowledge, H3D-Net
is the closest system to ours in single-view reconstruction capabilities (out
of all systems with either their code or results available). Additionaly, we
tried to compare our system with PIFuHD [52], which unfortunately failed to
work with heads images without body (see supplementary).

We show qualitative comparison in Figure 3. We evaluate our method and H3D-Net
both for frontal- and side-view reconstruction. We note the significant
overfitting of H3D-Net to the visible hair geometry, while our model provides
reconstructions more robust to the change of viewpoint.

In total, we compared our models on all scans available in the test set of the
H3DS dataset, and each scan was reconstructed from three different viewpoints.
We provide the measured mean Chamfer distance both for our method and
baselines across all scans in Tab. 1.

|  |  |  |  |
---|---|---|---|---|---
|  |  |  |  |
|  |  |  |  |
Source | Driver | FOMM | Bi-Layer | FLAMETex | ROME
Figure 4: Comparison of renders on a VoxCeleb2 dataset. The task is to reenact
the source image with the expression and pose of the driver image. Here, we
picked diverse examples in terms of pose variation to highlight the
differences in performance of compared methods. We observe that for the large
head pose rotations, purely neural-based methods (FOMM, Bi-Layer) struggle to
maintain consistent quality. In contrast, our rendering method produces images
that are more robust to pose changes. Admittedly, for small pose changes,
neural-based methods exhibit a smaller identity gap than ROME (bottom row) and
overall outperform our method in terms of rendering quality. As a reference,
we also include a non-neural FLAMETex rendering method, which is employed in
state-of-the-art one-shot face reconstruction systems [2] but is not able to
personalize the avatar at the head level.

Rendering. We evaluate the quality of our renders on the hold-out subset
VoxCeleb2 dataset. We use a cross-driving comparison scenario for qualitative
comparison to highlight the animation capabilities of our method, and self-
driving scenario for quantitative comparison.

First, we compare with a FLAMETex [25] rendering system, which works
explicitly with mesh rendering. From the source image, FLAMETex estimates the
albedo via a basis of RGB textures, and then combines it with predicted scene-
specific shading. In contrast, our method predicts a rendered image directly
and avoids the complexity of explicit albedo-shading decomposition.

We then compare with publicly available geometry-free rendering methods, which
were trained on the same dataset. For that, we use the First-Order Motion
Model (FOMM) [9], the Bi-Layer Avatar Model [11] and recently proposed Thin-
Plate-Spline-Motion-Mode (TPSMM) [53]. Both these systems bypass explicit 3D
geometry estimation and rely only on learning the scene structure via the
parameters of generative ConvNets. Other methods [8, 48], which internally
utilize some 3D structures, like camera rotations, were out of the scope of
our comparison due to the unavailability of pre-trained models.

Table 2: Here we present the quantitative results on the VoxCeleb2-HQ dataset in the self-reenactment and cross-reenactment modes. Our ROME system performs on par with FOMM and TPSMM in self-reenactment, notably outperforming them in the most perceptually-plausible LPIPS metrics. On the contrary, in the cross-driving scenario, when the task is complex for pure neural-based systems, our method obtains better results. | self-reenactment |  | cross-reenactment
---|---|---|---
Method | LPIPS$\downarrow$ | SSIM$\uparrow$ | PSNR$\uparrow$ |  | FID$\downarrow$ | CSIM$\uparrow$ | IQA$\uparrow$
FOMM | $0.09$ | $0.87$ | $25.8$ |  | $52.95$ | $0.53$ | $55.9$
Bi-Layer | $0.08$ | $0.83$ | $23.7$ |  | $51.4$ | $0.56$ | $50.48$
TPSMM | $0.09$ | $0.85$ | $26.1$ |  | $49.27$ | $0.57$ | $59.5$
ROME | $0.08$ | $0.86$ | $26.2$ |  | $45.32$ | $0.62$ | $66.3$

We present the qualitative comparison in Figure 4, and a quantitative
comparison across a randomly sampled hold-out VoxCeleb2 subset in Table 2. We
restrict the comparison to the face and hair region as the shoulder pose is
not controlled by our method (driven by DECA parameters), which is admittedly
a limitation of our system. We thus mask the results according to the face and
hair mask estimated from the ground truth image.

Generally, we observe that over the entire test set, the quality of ROME
avatars in the self-reenactment mode is similar to FOMM and better than the
Bi-layer model. For the cross-reenactment scenario, our model is clearly
better than both alternatives according to three metrics, that help to asses
unsupervised quality of the images in three aspects: realism, identity
preservation and blind quality of the image. The huge gap for IQA [49] and FID
[54] is also noticeable in the qualitative comparison, especially for strong
pose change (see CSIM [10] column in Tab. 2). The PSNR and SSIM metrics
penalize slight misalignments between the sharp ground truth and our
renderings much stronger than the blurriness in FOMM reconstructions. The
advatage of ROME avatar is noticable even for self-driving case according to
LPIPS. We provide a more extensive qualitative evaluation in the supplementary
materials.

###  4.3 Linear basis experiments

|  |  |  |  |  |
---|---|---|---|---|---|---
|  |  |  |  |  |
Input | Full | w/o $\Delta\hat{\mathbf{v}}$ | w/o $\vec{n}$ | w/o $\mathcal{L}_{\text{occ}}$ | w/o $\mathcal{L}_{\text{lap}}$ | w/o $\mathcal{L}_{\text{chm}}$
Figure 5: Ablation study. We qualitatively evaluate the individual components
of our _full_ model. w/o $\Delta\hat{\mathbf{v}}$: without the per-vertex
displacements, we obtain a significantly worse render quality. w/o $\vec{n}$:
when we apply per-vertex deformations instead of per-vertex displacements
(i.e., deformations alongside the normals), we obtain noisy reconstructions in
neck area and worse renders. w/o $\mathcal{L}_{\text{occ}}$: without
silhouette-based losses, our model fails to learn proper reconstructions. w/o
$\mathcal{L}_{\text{lap}}$: Laplacian regularization smooths the
reconstructions. w/o $\mathcal{L}_{\text{chm}}$: chamfer loss allows us to
constrain the displaced vertices to lie inside the scene boundaries, which
positively affects the smoothness of the visible part of the reconstruction.

As discussed above, we distill our ROME head reconstruction model into a
linear parametric model. To do that, we set the number of basis vectors to
$50$ for the hair and $10$ for the neck offsets and run low-rank Principle
Component Analysis (PCA) to estimate them. The number of components is chosen
to obtain a low enough approximation error. Interestingly, the offsets learned
by our model can be compressed by almost two orders of magnitude in terms of
degrees of freedom without any practical loss in quality (Figure 66(a)), which
suggests that the capacity of the offset generator is underused in our model.
We combine estimated basis vectors with the original basis of the FLAME.

After that, we train feed-forward encoders that directly predict the
coefficients of the two basis from the source image. The prediction is
performed in two stages. First, face expression, pose and camera parameters
are predicted with a MobileNetV2 [55] encoder. Then a slower ResNet-50 encoder
[56] is used to predict hair, neck and shape coefficients. The choice of
architectures are motivated by the fact that in many practical scenarios only
the first encoder needs to be invoked frequently (per-frame), while the second
can run at much lower rate or even only at the model creation time.

###  4.4 Ablation study

We demonstrate results of ablation study at Figure 5. As expected, predicting
more accurate geometry affect the renders (first row). Also, we verify the
necessity of all terms of geometry loss. We observe significant improvement in
quality of renders with additional geometry (see Supp 3), which leads us to an
optimistic conclusion that our learned coarse mesh may be integrated into
other neural rendering systems [8] to improve their quality. Additionally, we
observe that geometry losses allows to correctly model coarse details on the
hair without noise and reconstruct the hair without sticking with neck.
Similar artifacts are removed by adding shifts along the normals.

|  |
---|---|---
Image | ROME | Distilled
(a) Linear model

|  |
---|---|---
Long hair | Ear cover | Failed renders
(b) Limitations

Figure 6: Linear model results and the examples of limitations. On the left,
we show how reconstructions learned by our method, ROME, could be distilled
using a linear parametric model. We are able to compress the learned offsets
into a small basis, reducing the degrees of freedom by two orders of
magnitude. We can then distill these offsets using a much faster regression
network with a small gap in terms of the reconstruction quality. On the right,
we highlight the main limitations of our method, which include the failure
related to long hair modelling, caused by an incorrect topological prior, no
coverage of ears and unrealistic renders under a significant change of scales.

###  4.5 Limitations

In some cases, the proposed system often produces somewhat oversmoothed
geometry without person-specific attributes. Most of those attributes simply
cannot be presented with the limited set of vertices. Our preliminary
experiments with subdivision during training did not help to alleviate this
problem. It may be interesting to study this problem in a few-shot scenario
(given different views) or to use images and proxy geometry in a higher
resolution. Another frequent geometry artifact is the roughness of the
clothing approximation. As we know, modeling any clothes without 3D data is an
extremely difficult task and the fact that our models is not able to do so
from videos is not surprising. A principled solution to this problem will
likely require learning from datasets that cover these areas (e.g., circular
3D scans of human heads) [57].

Our current model is trained at roughly fixed scale, though explicit geometry
modeling allows it to generalize to adjacent scale reasonably well. Still,
strong changes of scale lead to poor performance (Figure 66(b)). More examples
are provided in the supplementary materials. Addressing this issue via mip-
mapping and multi-scale GAN training techniques remains future work.

Lastly, our model can have artifacts with long hair (Figure 66(b), left) or
ears (Figure 66(b), middle). Handling such cases gracefully are likely to
require a departure from the predefined FLAME mesh connectivity to new person-
specific mesh topology. Handling such issues using a limited set of pre-
designed hair meshes is an interesting direction for future research.

##  5 Summary

We have presented ROME avatars: a system for creating realistic one-shot mesh-
based human head models that can be animated and are compatible with FLAME
head models. We compare our model with representative state-of-the-art models
from different classes, and show that it is highly competitive both in terms
of geometry estimation and the quality of rendering.

Crucially, our system can learn to model head geometry without direct
supervision in the form of 3D scans. Despite that, we have observed it to
achieve state-of-the-art results in head geometry recovery from a single
photograph. At the same time, it also performs better than previous one-shot
neural rendering approaches in the cross- and self-driving scenario. We have
thus verified that the resulting geometry could be used to improve the
rendering quality.

As neural rendering becomes more widespread within graphics systems, ROME
avatars and similar systems can become directly applicable, while their one-
shot capability and the simplicity of rigging derived from DECA and FLAME
could become especially important in practical applications.

## Acknowledgements

We sincerely thank Eduard Ramon for providing us the one-shot H3D-Net
reconstructions. We also thank Arsenii Ashukha for their comments and
suggestions regarding the text contents and clarity, as well as Julia Churkina
for helping us with proof-reading. The computational resources for this work
were mainly provided by Samsung ML Platform.

## References

  * [1] Volker Blanz and Thomas Vetter.  A morphable model for the synthesis of 3d faces.  In SIGGRAPH ’99, 1999.
  * [2] Yao Feng, Haiwen Feng, Michael J. Black, and Timo Bolkart.  Learning an animatable detailed 3d face model from in-the-wild images.  ACM Transactions on Graphics (TOG), 40:1 – 13, 2020.
  * [3] Jianzhu Guo, Xiangyu Zhu, Yang Yang, Fan Yang, Zhen Lei, and Stan Z Li.  Towards fast, accurate and stable 3d dense face alignment.  In Proceedings of the European Conference on Computer Vision (ECCV), 2020.
  * [4] Soubhik Sanyal, Timo Bolkart, Haiwen Feng, and Michael J. Black.  Learning to regress 3d face shape and expression from an image without 3d supervision.  2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 7755–7764, 2019.
  * [5] Joon Son Chung, Arsha Nagrani, and Andrew Zisserman.  Voxceleb2: Deep speaker recognition.  In INTERSPEECH, 2018.
  * [6] Justus Thies, Michael Zollhöfer, and Matthias Nießner.  Deferred neural rendering: Image synthesis using neural textures.  arXiv: Computer Vision and Pattern Recognition, 2019.
  * [7] Nikhila Ravi, Jeremy Reizenstein, David Novotny, Taylor Gordon, Wan-Yen Lo, Justin Johnson, and Georgia Gkioxari.  Accelerating 3d deep learning with pytorch3d.  arXiv:2007.08501, 2020.
  * [8] Ting-Chun Wang, Arun Mallya, and Ming-Yu Liu.  One-shot free-view neural talking-head synthesis for video conferencing.  2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 10034–10044, 2021.
  * [9] Aliaksandr Siarohin, Stéphane Lathuilière, S. Tulyakov, Elisa Ricci, and N. Sebe.  First order motion model for image animation.  Advances in Neural Information Processing Systems (NeurIPS), 2019\.
  * [10] Egor Zakharov, Aliaksandra Shysheya, Egor Burkov, and Victor S. Lempitsky.  Few-shot adversarial learning of realistic neural talking head models.  In 2019 IEEE/CVF International Conference on Computer Vision (ICCV), 2019.
  * [11] Egor Zakharov, Aleksei Ivakhnenko, Aliaksandra Shysheya, and Victor S. Lempitsky.  Fast bi-layer neural synthesis of one-shot realistic head avatars.  In ECCV, 2020.
  * [12] J. Thies, M. Zollhöfer, M. Stamminger, C. Theobalt, and M. Nießner.  Face2face: Real-time face capture and reenactment of rgb videos.  In Proc. Computer Vision and Pattern Recognition (CVPR), IEEE, 2016\.
  * [13] Hyeongwoo Kim, Pablo Garrido, Ayush Tewari, Weipeng Xu, Justus Thies, Matthias Nießner, Patrick Pérez, Christian Richardt, Michael Zollöfer, and Christian Theobalt.  Deep video portraits.  ACM Transactions on Graphics (TOG), 37(4):163, 2018.
  * [14] AvatarSDK.  https://avatarsdk.com/.
  * [15] Pinscreen.  https://www.pinscreen.com/.
  * [16] Tal Hassner, Shai Harel, Eran Paz, and Roee Enbar.  Effective face frontalization in unconstrained images.  2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 4295–4304, 2015.
  * [17] Volker Blanz, Sami Romdhani, and Thomas Vetter.  Face identification across different poses and illuminations with a 3d morphable model.  Proceedings of Fifth IEEE International Conference on Automatic Face Gesture Recognition, pages 202–207, 2002.
  * [18] A. Tran, Tal Hassner, Iacopo Masi, and Gérard G. Medioni.  Regressing robust and discriminative 3d morphable models with a very deep neural network.  2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 1493–1502, 2017.
  * [19] Ben Mildenhall, Pratul P. Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, and Ren Ng.  Nerf: Representing scenes as neural radiance fields for view synthesis.  In ECCV, 2020.
  * [20] Eduard Ramon, Gil Triginer, Janna Escur, Albert Pumarola, Jaime Garcia Giraldez, Xavier Giró i Nieto, and Francesc Moreno-Noguer.  H3d-net: Few-shot high-fidelity 3d head reconstruction.  Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021.
  * [21] Stephen Lombardi, Jason M. Saragih, Tomas Simon, and Yaser Sheikh.  Deep appearance models for face rendering.  ACM Transactions on Graphics (TOG), 37:1 – 13, 2018.
  * [22] Stephen Lombardi, Tomas Simon, Jason M. Saragih, Gabriel Schwartz, Andreas M. Lehrmann, and Yaser Sheikh.  Neural volumes.  ACM Transactions on Graphics (TOG), 38:1 – 14, 2019.
  * [23] Bernhard Egger, W. Smith, Ayush Tewari, Stefanie Wuhrer, Michael Zollhöfer, Thabo Beeler, Florian Bernard, Timo Bolkart, Adam Kortylewski, Sami Romdhani, Christian Theobalt, Volker Blanz, and Thomas Vetter.  3d morphable face models—past, present, and future.  ACM Transactions on Graphics (TOG), 39:1 – 38, 2020.
  * [24] Stylianos Ploumpis, Evangelos Ververas, Eimear O’ Sullivan, Stylianos Moschoglou, Haoyang Wang, Nick E. Pears, W. Smith, Baris Gecer, and Stefanos Zafeiriou.  Towards a complete 3d morphable model of the human head.  In IEEE Transactions on Pattern Analysis and Machine Intelligence, 2021.
  * [25] Tianye Li, Timo Bolkart, Michael J. Black, Hao Li, and Javier Romero.  Learning a model of facial shape and expression from 4d scans.  ACM Transactions on Graphics (TOG), 36:1 – 17, 2017.
  * [26] Shunsuke Saito, Tomas Simon, Jason M. Saragih, and Hanbyul Joo.  Pifuhd: Multi-level pixel-aligned implicit function for high-resolution 3d human digitization.  2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 81–90, 2020.
  * [27] Stephen Lombardi, Tomas Simon, Gabriel Schwartz, Michael Zollhoefer, Yaser Sheikh, and Jason M. Saragih.  Mixture of volumetric primitives for efficient neural rendering.  ACM Transactions on Graphics (TOG), 40:1 – 13, 2021.
  * [28] Qianli Ma, Shunsuke Saito, Jinlong Yang, Siyu Tang, and Michael J. Black.  Scale: Modeling clothed humans with a surface codec of articulated local elements.  In CVPR, 2021.
  * [29] Jeong Joon Park, Peter R. Florence, Julian Straub, Richard A. Newcombe, and S. Lovegrove.  Deepsdf: Learning continuous signed distance functions for shape representation.  2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 165–174, 2019.
  * [30] Petr Kellnhofer, Lars Jebe, Andrew Jones, Ryan P. Spicer, Kari Pulli, and Gordon Wetzstein.  Neural lumigraph rendering.  2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2021.
  * [31] Michael Oechsle, Songyou Peng, and Andreas Geiger.  Unisurf: Unifying neural implicit surfaces and radiance fields for multi-view reconstruction.  International Conference on Computer Vision (ICCV), 2021.
  * [32] Philip-William Grassal, Malte Prinzler, Titus Leistner, Carsten Rother, Matthias Nießner, and Justus Thies.  Neural head avatars from monocular rgb videos.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
  * [33] Ilya Zakharkin, Kirill Mazur, Artur Grigoriev, and Victor S. Lempitsky.  Point-based modeling of human clothing.  2021 IEEE/CVF International Conference on Computer Vision (ICCV), 2021.
  * [34] Tarun Yenamandra, Ayush Tewari, Florian Bernard, Hans-Peter Seidel, Mohamed A. Elgharib, Daniel Cremers, and Christian Theobalt.  i3dmm: Deep implicit 3d morphable model of human heads.  2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 12798–12808, 2021.
  * [35] Yufeng Zheng, Victoria Fernández Abrevaya, Marcel C. Bühler, Xu Chen, Michael J. Black, and Otmar Hilliges.  I m avatar: Implicit morphable head avatars from videos.  In 2022 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
  * [36] Guy Gafni, Justus Thies, Michael Zollhöfer, and Matthias Nießner.  Dynamic neural radiance fields for monocular 4d facial avatar reconstruction.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2021.
  * [37] Shichen Liu, Tianye Li, Weikai Chen, and Hao Li.  Soft rasterizer: A differentiable renderer for image-based 3d reasoning.  2019 IEEE/CVF International Conference on Computer Vision (ICCV), pages 7707–7716, 2019.
  * [38] Samuli Laine, Janne Hellsten, Tero Karras, Yeongho Seol, Jaakko Lehtinen, and Timo Aila.  Modular primitives for high-performance differentiable rendering.  ACM Transactions on Graphics, 39(6), 2020.
  * [39] Olaf Ronneberger, Philipp Fischer, and Thomas Brox.  U-net: Convolutional networks for biomedical image segmentation.  In MICCAI, 2015.
  * [40] Olga Sorkine-Hornung.  Laplacian mesh processing.  In Eurographics, 2005.
  * [41] Justin Johnson, Alexandre Alahi, and Li Fei-Fei.  Perceptual losses for real-time style transfer and super-resolution.  In ECCV, 2016.
  * [42] Qiong Cao, Li Shen, Weidi Xie, Omkar M. Parkhi, and Andrew Zisserman.  Vggface2: A dataset for recognising faces across pose and age.  2018 13th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2018), pages 67–74, 2018.
  * [43] Ian J. Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron C. Courville, and Yoshua Bengio.  Generative adversarial nets.  In NIPS, 2014.
  * [44] Ting-Chun Wang, Ming-Yu Liu, Jun-Yan Zhu, Andrew Tao, Jan Kautz, and Bryan Catanzaro.  High-resolution image synthesis and semantic manipulation with conditional gans.  2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8798–8807, 2018.
  * [45] Fausto Milletari, Nassir Navab, and Seyed-Ahmad Ahmadi.  V-net: Fully convolutional neural networks for volumetric medical image segmentation.  2016 Fourth International Conference on 3D Vision (3DV), pages 565–571, 2016.
  * [46] Silvia Zuffi, Angjoo Kanazawa, Tanya Berger-Wolf, and Michael J. Black.  Three-d safari: Learning to estimate zebra pose, shape, and texture from images ”in the wild”, 2019.
  * [47] Nathan Halko, Per-Gunnar Martinsson, and Joel A. Tropp.  Finding structure with randomness: Probabilistic algorithms for constructing approximate matrix decompositions.  SIAM Rev., 53:217–288, 2011.
  * [48] Michail Christos Doukas, Stefanos Zafeiriou, and Viktoriia Sharmanska.  Headgan: Video-and-audio-driven talking head synthesis.  2021\.
  * [49] Shaolin Su, Qingsen Yan, Yu Zhu, Cui cui Zhang, Xin Ge, Jinqiu Sun, and Yanning Zhang.  Blindly assess image quality in the wild guided by a self-adaptive hyper network.  2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2020.
  * [50] Adrian Bulat and Georgios Tzimiropoulos.  How far are we from solving the 2d & 3d face alignment problem? (and a dataset of 230,000 3d facial landmarks).  In International Conference on Computer Vision, 2017.
  * [51] Diederik P. Kingma and Jimmy Ba.  Adam: A method for stochastic optimization.  International Conference for Learning Representations, 2015.
  * [52] Shunsuke Saito, Tomas Simon, Jason Saragih, and Hanbyul Joo.  Pifuhd: Multi-level pixel-aligned implicit function for high-resolution 3d human digitization.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, June 2020.
  * [53] Jian Zhao and Hui Zhang.  Thin-plate spline motion model for image animation.  2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
  * [54] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter.  Gans trained by a two time-scale update rule converge to a local nash equilibrium.  In Advances in Neural Information Processing Systems, 2017.
  * [55] Mark Sandler, Andrew G. Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chieh Chen.  Mobilenetv2: Inverted residuals and linear bottlenecks.  2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2018.
  * [56] Kaiming He, X. Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 770–778, 2016.
  * [57] Thiemo Alldieck, Marcus Magnor, Bharat Lal Bhatnagar, Christian Theobalt, and Gerard Pons-Moll.  Learning to reconstruct people in clothing from a single rgb camera.  In 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2019.
  * [58] Karen Simonyan and Andrew Zisserman.  Very deep convolutional networks for large-scale image recognition.  International Conference on Learning Representations, 2015.
  * [59] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael S. Bernstein, Alexander C. Berg, and Li Fei-Fei.  Imagenet large scale visual recognition challenge.  International Journal of Computer Vision, 2015.
  * [60] Kevin Cortacero, Tobias Fischer, and Y. Demiris.  Rt-bene: A dataset and baselines for real-time blink estimation in natural environments.  2019 IEEE/CVF International Conference on Computer Vision Workshop (ICCVW), pages 1159–1168, 2019.
  * [61] Takeru Miyato, Toshiki Kataoka, Masanori Koyama, and Yuichi Yoshida.  Spectral normalization for generative adversarial networks.  ArXiv, abs/1802.05957, 2018.
  * [62] Ting-Chun Wang, Ming-Yu Liu, Andrew Tao, Guilin Liu, Jan Kautz, and Bryan Catanzaro.  Few-shot video-to-video synthesis.  Advances in Neural Information Processing Systems (NeurIPS), 2019\.
  * [63] Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, and Alexei A. Efros.  Image-to-image translation with conditional adversarial networks.  2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 5967–5976, 2017.
  * [64] Yuxin Wu and Kaiming He.  Group normalization.  In ECCV, 2018.
  * [65] Siyuan Qiao, Huiyu Wang, Chenxi Liu, Wei Shen, and Alan Loddon Yuille.  Weight standardization.  ArXiv, abs/1903.10520, 2019.
  * [66] Dmitry Ulyanov, Andrea Vedaldi, and Victor S. Lempitsky.  Instance normalization: The missing ingredient for fast stylization.  ArXiv, abs/1607.08022, 2016.
  * [67] Kaiming He, X. Zhang, Shaoqing Ren, and Jian Sun.  Identity mappings in deep residual networks.  ECCV, 2016.
  * [68] Tero Karras, Samuli Laine, and Timo Aila.  A style-based generator architecture for generative adversarial networks.  2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 4396–4405, 2019.
  * [69] Yao Feng, Vasileios Choutas, Timo Bolkart, Dimitrios Tzionas, and Michael Black.  Collaborative regression of expressive bodies using moderation.  In International Conference on 3D Vision (3DV), 2021.

##  Appendix 0.A Supplementary material

###  0.A.1 Implementation details

#### Photometric training objectives.

During training, we use the photometric loss $\mathcal{L}_{\text{photo}}$ to
aid in learning the geometry, as well as to train the rendering. Our
photometric loss is the combination of the perceptual, the identity, the
adversarial, and the segmentation losses:

$\mathcal{L}_{\text{photo}}=\lambda_{\text{per}}\mathcal{L}_{\text{per}}+\lambda_{\text{idt}}\mathcal{L}_{\text{idt}}+\lambda_{\text{adv}}\mathcal{L}_{\text{adv}}+\lambda_{\text{seg}}\mathcal{L}_{\text{seg}}.$
(1)

For the perceptual loss $\mathcal{L}_{\text{per}}$, we use a weighted
combination of distances between features of predicted and target images.
These features are taken from two pre-trained convolutional neural networks.
We use features from a VGG19 [58] network pre-trained on ImageNet [59] to
match the general content and features from a VGG16-based gaze detection
network [60] to match the gaze direction. This loss, therefore, can be
expressed as follows:

$\mathcal{L}_{\text{per}}=\mathcal{L}_{\text{vgg}}+\mathcal{L}_{\text{gaze}}.$
(2)

In both of these losses, we measure the L1 distance between conv1_1, conv2_1,
conv3_1, conv4_1, and conv5_1 features after ReLU activations. We then sum
these distances with the following weights:
$\frac{1}{32},\frac{1}{16},\frac{1}{8},\frac{1}{4},$ and $1$ to obtain the
loss value.

In the gaze detection network, we independently process both eyes and average
the losses corresponding to each one of them. Before feature extraction, we
additionally perform alignment of both eyes using the keypoints extracted from
the image via a differentiable interpolation. For more details, please refer
to [60].

The face identity loss $\mathcal{L}_{\text{idt}}$ is the cosine distance
between the embeddings of a pre-trained face recognition network [42],
evaluated for the predicted and the target images. We use this loss to better
preserve the person’s identity in the renders. We calculate the cosine
distance as minus the cosine similarity.

To calculate the adversarial loss $\mathcal{L}_{\text{adv}}$ we jointly train
a discriminator network alongside our reconstruction and rendering networks.
We use a weighted combination of the hinge-loss discriminator loss as well as
the feature-matching loss [44]. We additionally apply spectral normalization
[61] to the discriminator network. The configuration of the adversarial
training closely follows the related works on image-to-image translation [62].
In particular, we use the PatchGAN [63] architecture for the discriminator.

Lastly, we use Dice loss to match the predicted segmentation
$\hat{\mathbf{s}}_{t}$ to the ground-truth $\mathbf{s}_{t}$:

$\mathcal{L}_{\text{seg}}=1-2\frac{\hat{\mathbf{s}}_{t}\cdot\mathbf{s}_{t}}{\|\hat{\mathbf{s}}_{t}\|^{2}_{2}+\|\mathbf{s}_{t}\|^{2}_{2}},$
(3)

where $\cdot$ denotes a scalar product. The segmentation loss is calculated
using each batch element separately and then averaged across the batch.

#### Architectures of the neural networks.

We use group normalization [64] paired with weight standardization [65] in all
networks to facilitate training with smaller batch sizes. Empirically we found
this combination to perform better than standard instance normalization [66].

In the autoencoder $E_{\text{tex}}$, which encodes the input image
$\mathbf{x}_{s}$ into the neural texture $\mathbf{T}_{s}$, we use pre-
activation residual blocks [67]. We set the number of channels in the neural
texture $\mathbf{T}_{s}$ to eight (we observed that values between $8$ and
$16$ result in similar performance). We additionally align the input image
using the transformation similar to the one used in the FFHQ dataset [68]. We
only modify the zoom-out factor to $1.25$ so that the aligned image contains
more hair and upper-body regions.

For the networks $E_{\text{img}}$ and $E_{\text{geom}}$ we use a standard
U-Net architecture. The network $E_{\text{img}}$ is U-Net, which predict an
output image and the segmentation mask. Besides the neural texture, we
additionally condition these two networks on the rendered mesh normals.
Specifically, we process each input dimension of the normal vectors using
$\\{\sin(kx)\\}_{k=1}^{K}$ and $\\{\cos(kx)\\}_{k=1}^{K}$ functions. In our
experiments, we set $K=6$. We then concatenate the resulting encodings to the
neural texture. We use max-pooling layers for downsampling and nearest
neighbors upsampling in the network, which decodes an image, and average
pooling with bilinear upsampling in the network, which decodes segmentations.
Additionally, the segmentation U-Net network has two times fewer channels than
the image network.

The architecture of $E_{\text{geom}}$ is the same as the image encoding U-Net,
albeit with a different number of input and output channels. We set the
dimensionality of a latent geometry map $\mathbf{Z}_{t}$, which is an output
of $E_{\text{geom}}$, to 32. Additionally, we encode the $xyz$-texture using
harmonic functions via the same process described before. We then concatenate
the obtained embeddings to the initial $xyz$-texture. The resulting feature
map has $3+3\cdot 6\cdot 2=39$ channels. In total, $E_{\text{geom}}$ has
$8+39=47$ input channels, since we also concatenate the embeddings of the
$xyz$-texture to the neural texture.

Finally, $G_{\text{geom}}$ consists of an MLP network which we apply
separately for each vertex to predict its offsets. To obtain its inputs, we
first resample the latent geometry map $\mathbf{Z}_{t}$ using an irregular
grid specified by the texture coordinates $w$. The obtained features for each
vertex are denoted as $\mathbf{z}_{t}^{w}$. These features are concatenated to
the harmonic embeddings of $w$ in the same way as the $xyz$-texture.
Specifically, each vector $w$ has two dimensions, therefore we encode it into
$2+2\cdot 6\cdot 2=26$ features, and then concatenate with $32$ channels of
$\mathbf{z}_{t}^{w}$ to obtain a vector with the dimensionality of $58$ — the
number of $G_{\text{geom}}$ input dimensions.

#### Training details.

We use the ADAM optimizer to train all networks in our model. We set the
learning rate to $1\cdot 10^{-4}$ for all the networks except for the
discriminator. For it, we set the learning rate to $4\cdot 10^{-4}$. We also
set $\beta_{1}=0$, and $\beta_{2}=0.999$. We train using eight P40 NVIDIA GPUs
to facilitate the batch size of 32. We observe that decreasing the batch size
leads to visible degradation of both rendering and reconstructions, but we did
no ablations to measure this effect.

###  0.A.2 Evaluation

We present additional results for the 3D reconstruction in both self-driving
(reconstruction using frames from the same video), and cross-driving
(reconstruction from a photo of one person and animation from a video of a
different person) scenarios. In Figure 8, we present more self-driving results
on the H3DS dataset, as well as a side-by-side comparison with H3D-Net. In
Figure 9, we present cross-driving evaluation results using the hold-out
samples from the VoxCeleb2 dataset, as well as paintings, the latter allowing
us to evaluate the susceptibility of our approach to domain shifts.

|  |
---|---|---
| PIFuHD
---
Full body
| Ours
---
Head
| PIFuHD
---
Head
(a) Comparison with PIFuHD.

|  |
---|---|---
|  |
| Source
---
Image
| ROME
---
w/ DECA
| ROME(L)
---
w/ PIXIE
(b) ROME integrated with PIXIE.

Additionally, we show the results in Fig. 7(a). PIFuHD can only recover head
geometry from the full-body images, which prevents us from doing a comparison
with this method using existing head reconstruction and reenactment
benchmarks. While having more detailed reconstructions, PIFuHD requires
training with 3D supervision and does not have animation capabilities.
Currently, these limitations can be lifted only at the cost of lengthy multi-
shot training per each avatar (IMAvatars), or by sacrificing some detalization
of reconstructions and retaining both real-time and one-shot capabilities,
which is done in our method.

To demonstrate the the ease of integration with existing SMPL-X based models
we predict the vertices corresponded to hair using distilled version of the
ROME with PIXIE [69]. The resulted mesh contains the hair from ROME basis and
shoulders from SMPL-X.

We provide an extended cross-driving qualitative comparison with neural-based
rendering methods in Figure 10. Then, we provide an additional self-driving
qualitative comparison in Figure 11. Most of the results are consistent with
the metrics obtained in the main text.

Method | LPIPS$\downarrow$ | SSIM$\uparrow$ | PSNR$\uparrow$
---|---|---|---
w/o $\Delta\hat{\mathbf{v}}$ | $0.10$ | $0.81$ | $23.1$
ROME | $0.08$ | $0.86$ | $25.8$
Table 3: Quantitative ablation on a hold-out set of the VoxCeleb2 dataset. We
observe that the deferred neural rendering trained without offsets achieves
lower image quality for face and hair regions, than a full ROME system.

We evaluate the effect that trained offsets have on the quality of rendering.
To do that, we remove the head reconstruction step from the training pipeline
and only train deferred neural rendering system using base FLAME meshes. The
quantitative results are in Table 3. Notice how the quality degrades for the
model with no offsets in the hair and shoulders areas and to the overall worse
quantitatively measured performance. This leads us to the conclusion that our
system for coarse mesh estimation can aid other neural rendering systems
produce better quality reconstructions.

###  0.A.3 Linear model

We evaluate visual quality in Figure 13. We note that the value of MSE that is
achieved by our regressor leads to visually similar reconstructions. Our mesh
reconstruction model can achieve up to 10 times speed-up without any perceived
degradation in reconstruction quality.

Additionally, we evaluate the semantic manipulation capabilities of the linear
model in a similar way to the face parametric models. Specifically, we pick
individual basis vectors and see how varying their coefficient changes the
reconstructed mesh. The results can be seen in Figure 12. We observe a certain
semantic disentanglement for the first hair and neck basis vectors. This
disentanglement allows us to perform mesh and image editing tasks, like the
editing of hairstyle, which we show in Figure 14. Here, we obtain this
modified reconstruction by simply varying one of the predicted coefficients
for the linear basis.

|  |  |  |  |
---|---|---|---|---|---
|  |  |  |  |
|  |  |  |  |
|  |  |  |  |
|  |  |  |  |
|  |  |  |  |
|  |  |  |  |
|  |  |  |  |
|  |  |  |  |
Source | H3D-Net | ROME | Target | H3D-Net | ROME
Figure 8: Extended qualitative comparison on the H3DS dataset. We compare 3D reconstructions and renders obtained using a single source image. |  |  |  |  |  |  |
---|---|---|---|---|---|---|---
|  |  |  |  |  |  |
|  |  |  |  |  |  |
|  |  |  |  |  |  |
|  |  |  |  |  |  |
|  |  |  |  |  |  |
|  |  |  |  |  |  |
|  |  |  |  |  |  |
Source | Driver | ROME |  Source | Driver | ROME
Figure 9: Additional examples of mesh-based avatars creating using a single source image, and animated using the camera pose and the expression parameters estimated from the driver image. First five rows contains results for samples out of the train distribution. |  |  |  |
---|---|---|---|---
|  |  |  |
|  |  |  |
|  |  |  |
|  |  |  |
|  |  |  |
|  |  |  |
Source | Driver | FOMM | Bi-Layer | ROME
Figure 10: Additional comparison of renders on a VoxCeleb2 dataset. The task is to reenact the source image with the expression and pose of the driver image. This comparison is done in a cross-driving scenario, which we also use for quantitative comparison. |  |  |  |
---|---|---|---|---
|  |  |  |
|  |  |  |
|  |  |  |
|  |  |  |
|  |  |  |
|  |  |  |
|  |  |  |
Source | Driver | FOMM | Bi-Layer | ROME
Figure 11: Additional comparison of renders on a VoxCeleb2 dataset. The task is to reenact the source image with the expression and pose of the driver image. This comparison is done in a self-driving scenario, which we also use for quantitative comparison. 1 |  |  |  |  |  |
---|---|---|---|---|---|---
2 |  |  |  |  |  |
4 |  |  |  |  |  |
52 |  |  |  |  |  |
$k$ | $\eta^{k}_{(1)}$ | $\eta^{k}_{(\lfloor 0.1M\rfloor)}$ | $\eta^{k}_{(\lfloor 0.25M\rfloor)}$ | $\eta^{k}_{(\lfloor 0.75M\rfloor)}$ | $\eta^{k}_{(\lfloor 0.9M\rfloor)}$ | $\eta^{k}_{(M)}$
Figure 12: We show how the estimated meshes can be semantically manipulated by varying the individual components of the PCA basis. We assume that $M$ meshes were initially predicted by the ROME system. Then, we estimate $M$ coefficients $\eta_{m}\in\mathbb{R}^{K}$, each is used to reconstruct $m$-th mesh via the PCA basis. In our experiments, the set of 50 basis vectors are used to reconstruct the hair, and the set of 10 basis vectors are used to reconstruct the neck and the shoulders. We denote each component of $\eta_{m}$ as $\eta_{m}^{k}$. Then, we use the mean vector $\eta=\frac{1}{M}\eta_{m}$ to reconstruct the base mesh, and modify its $k$-th component to the $p$-th order statistic $\eta_{(p)}^{k}$ over the dataset $\\{\eta_{m}^{k}\\}_{m=1}^{M}$, where $p\in\\{1,\dots,M\\}$. We show the resulting reconstructions above. Each row corresponds to a component which is modified, and each column corresponds to its new value (minimum, 10%, 25%, 75%, 90%, maximum). The first three lines correspond to the hair components, and the last line – to the neck component. |  |  |  |  |  |  |
---|---|---|---|---|---|---|---
|  |  |  |  |  |  |
|  |  |  |  |  |  |
Image | ROME | PCA | Distilled |  Image | ROME | PCA | Distilled
Figure 13: We provide more examples to qualitatively evaluate the performance of the distilled linear model. |  |  |  |  |
---|---|---|---|---|---
|  |  |  |  |
Image | Original | Edit 1 | Edit 2 | Edit 4 | Edit 5
Figure 14: Examples of hair style manipulation using only individual
components of a PCA basis.


#  Anomalously strong polarization dependence of the intensity of
magnetically-induced atomic transitions

Armen Sargsyan  Institute for Physical Research – National Academy of Sciences
of Armenia, 0204 Ashtarak-2, Armenia  Emmanuel Klinger  Institut FEMTO-ST –
UMR 6174 CNRS, SupMicroTech-ENSMM, Université de Franche-Comté, 25030
Besançon, France  Corresponding author<EMAIL_ADDRESS>Arevik
Amiryan  Institute for Physical Research – National Academy of Sciences of
Armenia, 0204 Ashtarak-2, Armenia  David Sarkisyan  Institute for Physical
Research – National Academy of Sciences of Armenia, 0204 Ashtarak-2, Armenia

###### Abstract

The impact of the optical field polarization on the spectrum of magnetically-
induced transitions, a class of transitions forbidden at zero magnetic field,
is studied with a weak-probe sub-Doppler technique. The high spectral
resolution of the technique combined with the simplicity in interpreting the
observed spectra, allows to follow the behavior of individual transitions as a
function of the magnetic field amplitude. We observe only one intense
transition (out of $2F_{g}+1$) in the case of linear ($\pi$) polarization in
the $F_{g}\rightarrow F_{g}+2$ manifolds of 85Rb, 87Rb and 133Cs for fields
above a few hundreds of gauss. We show that this behavior is in agreement with
a model based on the diagonalization of the Zeeman Hamiltonian matrix. With
the rapid development of micro-machined vapor-cell-based sensors these results
will be of use to magnetometers operating above Earth field, wide-range laser
frequency stabilization systems and atomic Faraday filters.

††journal: opticajournal

Optical and magneto-optical processes formed in hot atomic vapors of alkali
metal atoms are at the core of many devices and applications, such as atomic
optical clocks and gyroscopes [1], optical magnetometers [2], frequency
markers of atomic transitions [3], realization of narrow optical resonances
based on coherent processes [4], sensitive sensors of electric and microwave
fields based on Rydberg atoms [5], etc. Therefore, the identification of new
intense atomic transitions of alkali metals is of great interest.

The Zeeman effect occurs when atoms are in the presence of an external
magnetic field, leading to a significant modification of the intensity of
atomic transition [6, 7]. One interesting consequence of the couplings between
magnetic sub-levels induced by the Zeeman effect is the possibility to observe
transitions with an apparent $\Delta F=\pm 2$ selection rule at non-zero
field. These transitions, referred to as magnetically-induced (MI)
transitions, form a large class of one hundred transitions with interesting
and important features [8, 9, 10]. Interest in these transitions is primarily
driven by the fact that, in wide ranges of magnetic fields, their intensities
can significantly exceed the intensities of “usual” (i.e. $\Delta F=0,\pm 1$),
widely used atomic transitions. In addition, their frequency shifts in strong
magnetic fields reach 20 to 30 GHz, enabling the exploration of new frequency
ranges for the frequency stabilization of lasers operating at significantly
shifted frequencies compared to unperturbed atomic transitions. These
transition can also be used to form coherent processes such as
electromagnetic-induced transparency in large magnetic fields [11, 12]. A
thorough understanding of the evolution of alkali transitions in a magnetic
field is also important for magnetometry above Earth field [13, 14].

Despite possessing interesting features, magnetically-induced transitions are
often disregarded because traditional weak-probe spectroscopy lacks the
resolution to distinguish individual transitions due to the Doppler effect.
Conversely, sub-Doppler spectra acquired through nonlinear techniques, such as
saturated absorption, is cumbersome to interpret because of the large number
of transitions [15]. However, with nanocells (NCs), where the vapor is
confined between two almost-parallel windows separated by a distance ranging
from a few nm to several $\mu$m, one can realize weak-probe sub-Doppler
spectroscopy [16]. By selecting the proper cell thickness, this can be used to
realize sub-Doppler spectroscopy of molecular lines [17]. Beyond spectroscopy,
NCs serve as a crucial tool in fundamental physics. They are, for instance,
used to study atom-surface interactions [18, 19, 20], cooperative effects
[21], and realize single-photon generation at room temperature [22]. New
advancements in manufacturing techniques [23] are making nanocells (NCs)
increasingly accessible.

In this article, we study the evolution of MI transitions of Rb and Cs vapors,
confined in a NC, in the presence of a magnetic field. To improve the
resolution of the measurement, second derivative spectroscopy [24, 25] is
used. We show that the spectrum of $F_{g}\rightarrow F_{g}+2$ MI transitions
excited under linearly-polarized ($\pi$) optical field contains only one
intense transition out of $2F_{g}+1$. In contrast, the spectrum of
$F_{g}\rightarrow F_{g}+2$ MI transitions excited under circularly-polarized
($\sigma^{+}$) optical field contains $2F_{g}+1$ intense transitions, that is
3 transitions for 87Rb, 5 transitions for 85Rb, and 7 transitions for 133Cs.

In the presence of a magnetic field, magnetic sublevels are split and mixed,
leading to changes in transition probabilities and frequencies. These changes
can be calculated by diagonalizing the Hamiltonian matrix [6], which accounts
for the hyperfine atomic structure, and the Zeeman Hamiltonian, that is

$H=H_{\text{hfs}}+\frac{\mu_{B}}{\hbar}(g_{S}\,S_{z}+g_{L}\,L_{z}+g_{I}\,I_{z})B_{z}\,,$
(1)

where $\mu_{B}$ is the Bohr magneton [26], $g_{S,L,I}$ are the Landé factors
[27] and $S_{z}$, $L_{z}$, $I_{z}$ the projection of quantum numbers on the
$z$-axis (quantization axis). The eigenvalues obtained after the
diagonalization of the matrix provide access to the resonance frequencies. The
transition intensities are proportional to the squared dipole moment, given by

$|\langle
e||d||g\rangle|^{2}\propto\Gamma_{\text{N}}\,a^{2}[\psi(F_{e}^{\prime},m_{F_{e}});\psi(F_{g}^{\prime},m_{F_{g}});q]\,,$
(2)

where $\Gamma_{\text{N}}$ the natural linewidth of the transition,
$a[\psi(F_{e}^{\prime},m_{F_{e}});\psi(F_{g}^{\prime},m_{F_{g}});q]$ are the
transfer coefficients modified by the field [6], calculated using the
eigenvectors. Here, $q=0,\pm 1$ is associated with the polarization of the
incident laser field. To excite $\pi$ ($q=0$) transitions, the polarization of
the electric field of the laser should be collinear with the applied magnetic
field axis.

Figure 1: Dependence of the intensity of (a) 133Cs, (b) 85Rb and (c) 87Rb $\
F_{g}\rightarrow F_{g}+2$ MI transitions versus magnetic field when excited by
linear $\pi$-polarization. In large magnetic fields, atoms are left with only
one MI transition (out of $2F_{g}+1$). In each plot, the inset shows the
diagram of transitions with the corresponding labels.

Figure 1 shows the calculated evolution of $F_{g}\rightarrow F_{g}+2$ MI
transitions of the $D_{2}$ line of 133Cs (a), 85Rb (b) and 87Rb (c) under
$\pi$ optical excitation. A rapid increase of the MI transition intensities
can be seen as a function of the magnetic field for all three atoms. The
characteristic magnetic field111$B_{0}$ is 0.7 kG, 2.4 kG and 1.7 kG for 85Rb,
87Rb and 133Cs, respectively. $B_{0}=A_{\text{hfs}}/\mu_{B}$, where
$A_{\text{hfs}}$ is the magnetic dipole constant of the ground state, is a
good indicator for the dynamics of transitions. For each atom, one transition
is seen to have a much larger increase in intensity, reaching a maximum at
about $B\sim 0.5B_{0}$. With a further increase of the magnetic field, the
intensity smoothly decreases to almost zero. Note that a field $B\gg B_{0}$
marks the onset of the hyperfine Paschen-Back regime [28]. These transitions
are labeled ③, ⑤, ⑦, for 87Rb, 85Rb and 133Cs, respectively, see the insets in
Fig. 1. Conversely, the intensities of all other transitions quickly tend to
zero for fields $B>0.2B_{0}$. Thus, in magnetic field above $0.2B_{0}$, only
one $\Delta F=+2$ transition remains in the spectrum of alkali D2 line
observed under $\pi$-polarization. Qualitatively, this behavior can be
explained as follows: the excitation by linear $\pi$-radiation can be
represented as an excitation by a coherent superposition of $\sigma^{+}$ and
$\sigma^{-}$ polarized light. When excited by $\sigma^{-}$-polarization, only
one MI transition remains in the spectrum [28]. The excitation by
$\pi$-polarized radiation is then dominated by the $\sigma^{-}$ part and only
one intense MI transition can be observed. Note that our calculations also
show that $\Delta F=-2$ atomic transitions formed in strong $B$-fields with
$\pi$-polarization are much weaker in intensity than for $\Delta F=+2$
transitions. For this reason, we focus only on the latter.

Figure 2: Sketch of the experimental setup. ECDL – extended cavity diode
laser, FI – Faraday isolator, PBS – polarizing beam splitter, L – lens, PM –
permanent magnet, NC filled with Rb or Cs in the oven, PD – photodiodes. The
PMs are oriented such that applied magnetic field B is collinear with the
electric field of the laser radiation E (B$\parallel$E) in order to excite
$\pi$ transitions. The reference channel comprises a Rb (or Cs) nanocell from
which the transmission spectrum of the D2 line is recorded at zero magnetic
field.

The sketch of the experimental setup is shown in Fig. 2. A linearly-polarized
laser beam, emitted by a cw extended cavity tunable diode laser (ECDL) [29] is
used. When the experiments is carried out with Rb atomic vapor, an ECDL with
$\lambda=780$ nm is tuned in the vicinity of the Rb $D_{2}$ line. In the case
of Cs atomic vapor, another ECDL with $\lambda=852$ nm, is tuned in the
vicinity of the Cs $D_{2}$ line. In both cases, the spectral width of the
probe laser radiation is $\gamma_{l}<1$ MHz. In order to avoid feedback in the
cavity, a Faraday insulator is placed immediately after the laser. A
polarizing beam splitter (PBS) is used to purify the initial linear optical
polarization and direct part of the light to an auxiliary channel serving as a
frequency reference. The laser fields transmitted trough NCs in both the
measurements and reference channels are detected by a photodiode and the
signal fed to a Tektronix TDS 2014B digital four-channel oscilloscope.

To excite $\pi$ transitions, the applied magnetic field should be collinear
with the polarization of the incident laser field, i.e
$\mathbb{B}\parallel\mathbb{E}$. To apply fields above 0.5 kG, a set of two
strong permanent magnets mounted on a translation stage was used. The magnetic
field strength applied to the atomic vapor is varied via a simple longitudinal
displacement of the PM set [30], calibrated with a Teslameter HT201
magnetometer. Because, in this configuration, the field is applied along the
long side of the nanocell with PMs, a high gradient is also introduced and not
mitigated as in the case of $\sigma^{\pm}$ excitations [13]. This results in
an additional broadening of the transitions. To form the narrowest resonance
possible, the spectroscopy is performed at a vapor column thickness of
$\ell=\lambda/2$ [16]. The laser beam diameter is reduced down to $\sim 1\;$mm
to only illuminate a region where $\ell$ is homogeneous. The acquired spectra
are then post-processed with second derivative (SD). The resulting spectrum
$T^{\prime\prime}(\nu)$ typically shows transitions with spectral linewidth of
$\sim 20-30$ MHz, which is narrower by a factor of $\sim$20 compared with the
Doppler broadening. This allows to separate and study atomic transitions
individually. Note that the SD method correctly shows the relative intensities
of atomic transitions and their frequency positions, it can therefore be used
for quantitative measurements [25].

Figure 3 shows the experimental (black dots) SD transmission spectrum obtained
using a NC filled with natural-abundance Rb atomic vapor at a column thickness
$\ell=390$ nm and a field $B=800\;$G, upon excitation by a $\pi$-polarized
radiation. It was recorded with a laser power of $20\;\mu$W, and a NC
temperature of $120^{\circ}$C. The laser frequency is scanned across the 85Rb
$F=2\rightarrow 3^{\prime},4^{\prime}$ and 87Rb $F=1\rightarrow 2^{\prime}$,
$3^{\prime}$ manifolds of the D2 line. The red solid line corresponds to the
theoretical SD spectrum, calculated by second order numerical differentiation
of the transmission spectrum. The theoretical transmission spectrum is
obtained by summing the contribution of all possible transitions $j$, each of
which is treated as an independent two-level system:

$T(\nu)\propto\frac{1}{|Q|^{2}}\cdot\sum_{j}\text{Im}\left[\chi_{j}(\nu,\ell,B)\right],$
(3)

where $Q=1-r^{2}\exp(2ik\ell)$, $r$ is the (field) reflection coefficient of
the cell windows. The two-level system lineshape $\chi_{j}(\nu,\ell,B)$,
derived in Ref. [31], is a function of the cell thickness $\ell$ and the laser
frequency $\nu$; it reads

$\begin{split}\chi_{j}(\nu,\ell,B)=-4(1-r\exp[ik\ell])^{2}&\cdot\frac{\sin^{2}(k\ell/2)}{Q}\\\
&\cdot\frac{\mathcal{N}}{ku\sqrt{\pi}}\cdot\frac{iA_{j}(B)}{\Gamma/4\pi-i\Delta_{j}(B)},\end{split}$
(4)

where $\mathcal{N}$ is the vapor density,
$u(\Theta)=\sqrt{2k_{B}\Theta/m_{a}}$ is the thermal velocity at a temperature
$\Theta$ for atoms of mass $m_{a}$. The transition parameters
$\Delta_{j}(B)=\nu-\nu_{j}(B)$ and $A_{j}(B)$ are fed from the results
obtained after diagonalization of the Hamiltonian matrix, see (1). The
homogeneous broadening $\Gamma$, including contributions from natural
linewidth, collisional broadening, etc., is left as a free parameter in our
simulations. As seen in Fig. 3, only one MI transition can be observed for
each atom: ⑤ for 85Rb and ③ for 87Rb. In similar conditions, $2F_{g}+1$
transitions have been observed in the case of $\sigma^{+}$-polarized
excitation [32]. Such a strong difference in the number of MI transitions
excited when using $\sigma^{+}$ and $\pi$-polarizations is a manifestation of
an anomalously strong polarization dependence of the intensity of transitions.
Other nearby transitions belong to the 85Rb $F=2\rightarrow 3^{\prime}$ and
87Rb $F=1\rightarrow 2^{\prime}$ manifolds. A good agreement between the
experiment and the theory can be observed. Nevertheless, some discrepancies
can be seen which are likely due to laser instability during the measurements.

Figure 3: Second derivative transmission spectrum recorded in the vicinity of
the 85Rb $F=2\rightarrow 3^{\prime},4^{\prime}$ and 87Rb $F=1\rightarrow
2^{\prime},3^{\prime}$ manifolds. The experimental spectrum is obtained with a
$\pi$-polarized excitation at $B=800$ G and $\ell=390$ nm. The red solid line
represents the calculated spectrum. The weighted center of the 85Rb D2 line
was chosen as the zero laser frequency. In this spectrum, only one MI
transition remains at this field: transition ⑤ (out of a possible five, for
85Rb), transition ③ (out of a possible three for, 87Rb).  Figure 4: Second
derivative transmission spectrum recorded in the vicinity of the
$F=3\rightarrow 4^{\prime},5^{\prime}$ manifolds of the 133Cs D2 line. The
experimental spectra are obtained with a $\pi$-polarized excitation, with
$\ell=426$ nm and $B=2.90$ kG (a), $1.95$ kG (b), $1.30$ kG (c). The red solid
lines represent the respective calculated spectra. The weighted center of the
133Cs D2 line was chosen as the zero laser frequency. In this spectrum, only
one MI transition (out of a possible seven) remains at these fields:
transition ⑦. The theory describes very well the frequency positions and
amplitudes of the peaks.

Figure 4 shows SD transmission spectra upon excitation by a linear
$\pi$-polarized radiation, obtained from a Cs-filled NC at $\ell=426$ nm and
$B=2.90\;$kG (a), $1.95\;$kG (b) and $1.30\;$kG (c). For the experimental
spectra (black dots), the laser is scanned in the vicinity of the D2 line
$F=3\rightarrow 3^{\prime},4^{\prime},5^{\prime}$ transitions with a power of
30 $\mu$W. The NC temperature is set to 100∘C. As the theory predicts, there
remains only one MI transition at these magnetic fields, the transition
labeled ⑦, out of the possible 7. The other visible transitions belong to the
$F=3\rightarrow 3^{\prime},4^{\prime}$ of the Cs D2 line manifold. Note that
seven MI transitions have been observed in the case of $\sigma^{+}$-polarized
excitation in similar conditions [9]. As the magnetic field is increased above
1 kG, the intensity of the MI transition decreases. According to Fig. 1, the
ratio of amplitude of the MI transition ⑦ at 1.30 kG to that at 2.90 kG should
be about 1.8. However, on the spectra in Fig. 4, the observed ratio is about
2.7. This discrepancy is due to the increased transition broadening because of
the magnetic field gradient from the PMs. The calculated spectra show a very
good agreement with the experimental ones. To match, the broadening
$\Gamma/2\pi$ in (4) had to be increased from 150 to 210 MHz. Note that the SD
transmission spectrum is about 2.5 times narrower than the transmission
spectrum given by (3).

In conclusion, we have studied the evolution of $F_{g}\rightarrow F_{g}+2$ MI
transitions of Rb and Cs D2 lines, in the presence of a magnetic field upon
linear $\pi$ excitation. In the case of $\pi$ excitation, the applied magnetic
field is collinear with the laser polarization. In this configuration, the
vapor is much more sensitive to the gradient produced by permanent magnets,
which adversely leads the transition linewidth. This was partly overcome using
weak-probe second derivative spectroscopy [16, 25], allowing individual atomic
transitions to be resolved. For magnetic fields $B>0.5B_{0}$, we predict and
observe that only one MI transition (out of possible 3 for 87Rb, 5 for 85Rb
and 7 for 133Cs) remains in the spectrum. This is in contrast with the
evolution of MI transitions in the case of circular $\sigma^{+}$ polarized
excitation [32, 9]. Our calculations also show that such a feature exists for
the MI transitions of all alkali atoms D2 lines such as Na, K, Li, etc.
Furthermore a similar feature is expected for
$n^{2}S_{1/2}\rightarrow(n+1)^{2}P_{3/2}$ alkali lines, where $n=3,4,5$ is the
principal quantum number for Na, K, Rb respectively, which amounts to about
100 MI transitions [33]. With the rapid development of micro-machined vapor-
cell-based sensors these results are important for magnetometers operating
above Earth field, wide-range laser frequency stabilization systems and atomic
Faraday filters.

## Funding

The work was supported by the Science Committee of RA, in the frame of the
research project No 21T-1C005.

## Acknowledgments

We thank Ara Tonoyan for fruitfull discussions.

## Disclosures

The authors declare no conflicts of interest.

## Data availability statement

The data that support the findings of this study are available from the
corresponding author upon reasonable request.

## References

  * [1] J. Kitching, Applied Physics Reviews 5, 031302 (2018).
  * [2] A. Fabricant, I. Novikova, and G. Bison, New Journal of Physics 25, 025001 (2023).
  * [3] D. Pizzey, J. D. Briscoe, F. D. Logue, _et al._ , New Journal of Physics 24, 125001 (2022).
  * [4] R. Finkelstein, S. Bali, O. Firstenberg, and I. Novikova, New Journal of Physics 25, 035001 (2023).
  * [5] L. A. Downes, L. Torralbo-Campo, and K. J. Weatherill, New Journal of Physics 25, 035002 (2023).
  * [6] P. Tremblay, A. Michaud, M. Levesque, _et al._ , Phys. Rev. A 42, 2766 (1990).
  * [7] E. B. Alexandrov, M. P. Chaika, and G. I. Khvostenko, _Interference of atomic states_ , vol. 7 (Springer, 1993).
  * [8] A. Sargsyan, A. Amiryan, E. Klinger, and D. Sarkisyan, J. of Phy. B: At., Mol. and Opt. Phys. 53, 185002 (2020).
  * [9] A. Sargsyan, A. Amiryan, A. Tonoyan, _et al._ , Physics Letters A 390, 127114 (2021).
  * [10] A. Sargsyan, E. Klinger, A. Tonoyan, and D. Sarkisyan, Optik 303, 171757 (2024).
  * [11] A. Sargsyan, A. Tonoyan, A. Papoyan, and D. Sarkisyan, Optics letters 44, 1391 (2019).
  * [12] A. Sargsyan, A. Amiryan, A. Tonoyan, _et al._ , Physics Letters A 434, 128043 (2022).
  * [13] E. Klinger, H. Azizbekyan, A. Sargsyan, _et al._ , Appl. Opt. 59, 2231 (2020).
  * [14] H. Stærkind, K. Jensen, J. H. Müller, _et al._ , PRX Quantum 5, 020320 (2024).
  * [15] S. Scotto, D. Ciampini, C. Rizzo, and E. Arimondo, Phys. Rev. A 92, 063810 (2015).
  * [16] G. Dutier, A. Yarovitski, S. Saltiel, _et al._ , Europhysics Letters 63, 35 (2003).
  * [17] G. G. Arellano, J. C. de Aquino Carvalho, H. Mouhanna, _et al._ , Nature Communications 15, 1862 (2024).
  * [18] A. Laliotis, B.-S. Lu, M. Ducloy, and D. Wilkowski, AVS Quantum Science 3, 043501 (2021).
  * [19] A. Sargsyan, R. Momier, C. Leroy, and D. Sarkisyan, Physics Letters A 483, 129069 (2023).
  * [20] B. Dutta, J. C. d. A. Carvalho, G. Garcia-Arellano, _et al._ , Phys. Rev. Res. 6, L022035 (2024).
  * [21] T. Peyrot, Y. R. P. Sortais, J.-J. Greffet, _et al._ , Phys. Rev. Lett. 122, 113401 (2019).
  * [22] F. Ripka, H. Kübler, R. Löw, and T. Pfau, Science 362, 446 (2018).
  * [23] T. Peyrot, C. Beurthe, S. Coumar, _et al._ , Opt. Lett. 44, 1940 (2019).
  * [24] W. Demtröder, NASA STI/Recon Technical Report A 82, 12273 (1981).
  * [25] A. Sargsyan, A. Amiryan, Y. Pashayan-Leroy, _et al._ , Opt. Lett. 44, 5533 (2019).
  * [26] B. Olsen, B. Patton, Y.-Y. Jau, and W. Happer, Physical Review A 84, 063410 (2011).
  * [27] H. Stærkind, K. Jensen, J. H. Müller, _et al._ , Physical Review X 13, 021036 (2023).
  * [28] L. Weller, K. S. Kleinbach, M. A. Zentile, _et al._ , Journal of Physics B: Atomic, Molecular and Optical Physics 45, 215005 (2012).
  * [29] V. Vassiliev, S. Zibrov, and V. Velichansky, Review of scientific instruments 77, 013102 (2006).
  * [30] A. Sargsyan, G. Hakhumyan, C. Leroy, _et al._ , Optics letters 37, 1379 (2012).
  * [31] G. Dutier, S. Saltiel, D. Bloch, and M. Ducloy, J. Opt. Soc. Am. B 20, 793 (2003).
  * [32] E. Klinger, A. Sargsyan, A. Tonoyan, _et al._ , The European Physical Journal D 71, 216 (2017).
  * [33] A. Tonoyan, A. Sargsyan, R. Momier, _et al._ , Optical Memory and Neural Networks 32, S343–S348 (2023).

Bib-main