Figure 1. We present NFR, an end-to-end deep-learning approach for automatic
rigging and retargeting of 3D models of human faces in the wild. Top: Given an
unrigged facial mesh with an unknown expression and identity in an arbitrary
triangulation (yellow), NFR can transfer the expression to unrigged facial
meshes with arbitrary triangulations (cyan). Bottom: NFR provides an
interpretable latent space for user-friendly editing of the retargeted meshes.

# Neural Face Rigging for Animating and Retargeting Facial Meshes in the Wild

Dafei Qin<EMAIL_ADDRESS>The University of Hong KongHong Kong ,  Jun
Saito<EMAIL_ADDRESS>Adobe ResearchSeattleUSA ,  Noam Aigerman
<EMAIL_ADDRESS>Adobe ResearchSan FranciscoUSA ,  Thibault Groueix
<EMAIL_ADDRESS>Adobe ResearchSan FranciscoUSA and  Taku Komura
<EMAIL_ADDRESS>The University of Hong KongHong Kong

(2023)

###### Abstract.

We propose an end-to-end deep-learning approach for automatic rigging and
retargeting of 3D models of human faces in the wild. Our approach, called
Neural Face Rigging (NFR), holds three key properties: (i) NFR’s expression
space maintains human-interpretable editing parameters for artistic controls;
(ii) NFR is readily applicable to arbitrary facial meshes with different
connectivity and expressions; (iii) NFR can encode and produce fine-grained
details of complex expressions performed by arbitrary subjects. To the best of
our knowledge, NFR is the first approach to provide realistic and controllable
deformations of in-the-wild facial meshes, without the manual creation of
blendshapes or correspondence. We design a deformation autoencoder and train
it through a multi-dataset training scheme, which benefits from the unique
advantages of two data sources: a linear 3DMM with interpretable control
parameters as in FACS and 4D captures of real faces with fine-grained details.
Through various experiments, we show NFR’s ability to automatically produce
realistic and accurate facial deformations across a wide range of existing
datasets and noisy facial scans in-the-wild, while providing artist-
controlled, editable parameters.

Facial Modeling, Facial Animation, Data-Driven Animation, Retargeting

††journalyear: 2023††copyright: licensedothergov††conference: Special Interest
Group on Computer Graphics and Interactive Techniques Conference Conference
Proceedings; August 6–10, 2023; Los Angeles, CA, USA††booktitle: Special
Interest Group on Computer Graphics and Interactive Techniques Conference
Conference Proceedings (SIGGRAPH ’23 Conference Proceedings), August 6–10,
2023, Los Angeles, CA, USA††price: 15.00††doi: 10.1145/3588432.3591556††isbn:
979-8-4007-0159-7/23/08††ccs: Computing methodologies Animation††ccs:
Computing methodologies Machine learning††ccs: Computing methodologies Mesh
models

##  1\. Introduction

This paper is concerned with leveraging deep learning for automatic rigging
and retargeting of 3D meshes of human faces in the wild, supporting both raw
noisy scans of real faces, as well as clean, artist-authored meshes, while
exposing interpretable expression parameters which an artist can use to
intuitively bring any facial mesh to a desired expression.

Faces are a critical component of any human-centric tasks, with facial
movements being of significant interest in many fields of computer science
including graphics, computer vision, and HCI. Many computational
representations of 3D human faces have been proposed, with early formulations
such as _Facial Action Coding System_ (FACS) (Ekman and Friesen, 1978) and
PCA-based learned linear _3D Morphable Models_ (3DMM) (Blanz and Vetter,
1999). However, these rigged models can only operate on faces with known
geometry and triangulation (where a consistent correspondence is provided),
and cannot be automatically extended to novel face models with arbitrary
triangulations.

Thus, the goal of this paper is to provide a method to understand and control
real-world facial data with the following properties:

  * •

The deformation can be controlled via _interpretable_ parameters, enabling
users to intuitively control expressions.

  * •

The method should be applicable to arbitrary human faces of unknown subjects
and expressions. It should generalize to arbitrary triangulations, and be
robust to both noise and missing areas on the 3D face.

  * •

The method should produce highly-accurate expressions. It should accurately
encode and decode the fine-grained details of nonlinear deformations on real
human faces.

We address these challenges by developing an autoencoder for facial
expressions. Specifically, we design an encoder to transform a given face mesh
into a latent code representing its expression. We then employ recent
advancements in deep learning of 3D deformations (Aigerman et al., 2022) to
construct a decoder that takes an expression code and alters the face to the
desired expression. To ensure the capture of all facial features and
expression nuances, we create an encoder that combines image encoding of input
face renderings with a state-of-the-art mesh encoder (Sharp et al., 2022). Our
encoder features two branches—one for input expression and one for facial
identity—and incorporates a training setup that separates identity from
expression.

Lastly, we aim to learn the _interpretable_ latent space, i.e., that each
entry in the code can be interpreted as a parameter corresponding to a
specific, meaningful localized activation on the human face. This goal faces a
significant challenge posed by the lack of data: existing datasets comprise
real human scans which are not directly coupled with expression parameters,
hence we need to devise an architecture and a training scheme that will
disentangle interpretable parameters from human scans with unknown
expressions.

Specifically, we use two different complementary datasets to achieve our goal:
1) _Multiface_ (Wuu et al., 2022) which is rich in natural and identity-
specific deformations, but lacks interpretable expressions; 2) _ICT FaceKit_
(Li et al., 2020), which has interpretable parameters as in FACS, however, the
expressions are synthetic and less realistic. We then use these two datasets
in a training scheme of our expression autoencoder. We train the system such
that for the ICT dataset the latent code matches the ICT deformations
parameters, while for the _Multiface_ dataset, the rich and expressive details
of the faces are reconstructed. This joint training empirically leads to a
latent space that directly matches the FACS parameters, while further
representing the rich deformations in the _Multiface_ dataset. Our framework
thus couples the interpretable FACS parameters with the capability to produce
realistic deformations as in the _Multiface_ dataset.

We show through experiments that our framework is able to perform various
tasks, such as deformation transfer from unknown faces to other faces, as well
as user editing of facial expressions. We show our network carries its
abilities across various models, from artist-authored artistic meshes up to
noisy, partial face scans in the wild. Our system significantly improves the
efficiency of the human facial animation pipeline, by-passing heavy data pre-
processing such as facial alignment, remeshing, and manual rigging.

##  2\. Related Works

In this section, we review techniques about facial deformation models and
deformation transfer.

###  2.1. Facial Deformation Models

#### Anatomy-inspired models

Facial Action Coding System (FACS) (Ekman and Friesen, 1978) defines facial
movements as the combination of muscle activations, or _Action Units_ (AUs).
Variants of FACS have been adopted in graphics and animation for their
intuitive artistic controls, typically implemented with blendshape deformers
(Lewis et al., 2014). Such FACS models can be viewed as 3D Morphable Models
(3DMM) (Blanz and Vetter, 1999) with hand-crafted basis using the domain
knowledge of the human anatomy. This in turn means FACS-based 3DMMs require
manual sculpting of many shapes. The inverse rig problem (a.k.a. retargeting
in the context of facial animation) which solves the optimal 3DMM parameters
fitting to target shapes is also not trivial for conventional FACS-based
models (Lewis and Anjyo, 2010; Seol et al., 2011; Cetinaslan and Orvalho,
2020a, b).

Several recent studies learn compact and sparse neural representations from
FACS models. Bailey et al. (2020) replaces film-quality animation rigs with a
learned deep model. Vesdapunt et al. (2020) propose a person-specific joint-
based neural skinning model with highly compact and sparse latent space. Choi
et al. (2022) design an intuitive interface for controlling character
expressions through curves drawn on the face along facial muscle structures.
Although these models preserve intuitive deformation controls, they are
subject-specific, i.e., building these models for a new character requires
starting from scratch. Our goal is to maintain the intuition of FACS but learn
subject-specific rigging and retargeting from data with minimal manual labor.

#### Data-driven models

To avoid the ad-hoc manual sculpting of 3DMM basis, learning linear 3DMMs from
face scans is a popular choice (Blanz and Vetter, 1999; Li et al., 2017;
Paysan et al., 2009; Huber et al., 2016; Choe et al., 2001; Choe and Ko, 2006;
Wu et al., 2016; Tewari et al., 2017; Brunton et al., 2014). The readers are
referred to (Egger et al., 2020) for a thorough survey of 3DMM related
methods. Limited by the linear nature, these models struggle to produce highly
variant and complex facial expressions. Additionally, the PCA bases fail to
provide interpretable and sparse controllers.

Mesh-based neural networks are proposed to learn more expressive facial
representations than linear 3DMMs. Monti et al. (2017) propose a unified
framework to operate on 3D mesh local geometries. Ranjan et al. (2018) apply
spectral convolutions on 3D meshes and uses hierarchical sampling to capture
local and global features. Verma et al. (2018) enable the model to dynamically
learn the correspondence between the data during training. Gong et al. (2019)
propose spiral convolution to retrieve information on triangle mesh
neighborhood. Bouritsas et al. (2019) follow a similar approach to directly
processing vertex offsets and achieves better results on face representation
than 3DMM. Song et al. (2020) propose to learn the facial model in the
differential subspace. Zhou et al. (2020) generalize the 3D mesh autoencoder
to train on tetrahedra and non-manifold meshes. Because these methods depend
on specific mesh templates, they fail to apply to meshes with different
representations.

Several recent works target triangulation agnosticism. Chandran et al. (2022b)
utilize a transformer architecture and positional encoding to align input
meshes to a canonical space. While supporting different triangulations,
correspondence is required if users want to apply their model to unseen facial
meshes. Yang et al. (2022) propose a physically-based implicit model to
control soft bodies like human faces, which supports different mesh
resolutions of the same identity.

In summary, meshes from one face dataset/model cannot be repurposed for other
datasets/models without going through intensive processing steps using non-
rigid registration guided by manual landmarks to take the dense correspondence
(Li et al., 2020). Our method removes this burden by training on multiple
datasets with different representations, enabling application to diverse
facial meshes.

Figure 2. Overview. Given an unrigged facial mesh with an unknown expression
and identity in an arbitrary triangulation (yellow) and another unrigged
neutral facial mesh with the target identity in an arbitrary triangulation
(cyan), NFR extracts an expression code $z_{e}$ from the first mesh, and an
identity code $z_{i}$ in the second mesh, and transfer the expression to the
target identity with non-linear deformations as in real human faces. A part of
$z_{e}$, denoted as $z_{FACS}$, follows the interpretable rigging parameters,
allowing NFR to behave as an artist-friendly auto-rigger and retargeter.

###  2.2. Deformation Transfer

Deformation Transfer is a retargeting technique that directly works on meshes.
Sumner and Popović (2004) transfer the animation by mapping the deformation
gradients from the source to the target. Li et al. (2010) propose to build a
target blendshape model by combining existing rigging prior and a few target
example expressions. These methods require dense correspondence of the source-
target pair.

#### Neural Deformation Transfer

The investigation of object deformation through neural networks (Gao et al.,
2018) have been extensively explored. A comprehensive overview can be found in
the survey by (Roberts et al., 2021). Tan et al. (2018) establish a mesh VAE
for learning deformation spaces of a specifically given mesh. Gao et al.
(2018) establish automatic deformation transfer between two mesh datasets,
which does not require explicit correspondence between the pair of meshes.
However, these methods assume the input mesh structures are consistent: Mesh
VAEs cannot aggregate information from multiple datasets where the
connectivities of the meshes are different. Retraining is required for
transferring the deformation between new mesh pairs.

Certain facial models (Jiang et al., 2019; Chandran et al., 2020, 2022a)
disentangle identity and expression spaces, simplifying deformation transfer
within the identity space. Still, these models do not accommodate mesh
templates divergent from the training set, precluding transfer to or from
custom meshes. Moser et al. (2021) suggest transferring animations between
rendered videos and 3D characters via an image-to-image model, but this
approach is character-specific and sacrifices deformation interpretability
with PCA bases. Notably, none of these methods addresses the primary objective
of this paper: integrating the neural deformation space with interpretable
parameters to facilitate fine-grained human control over deformation while
accommodating in-the-wild facial meshes.

##  3\. Method

In this section, we first give an overview of the NFR framework. Then we
explain how to leverage existing face datasets and 3DMMs to train a network
that is both interpretable and generalizable to meshes with various shapes and
triangulations.

###  3.1. Architecture

In essence, the architecture of NFR can be described as that of an autoencoder
(Fig. 2). Specifically, given a neutral identity mesh $M_{i}$ to be deformed
to an expression, and an expression mesh $M_{e}$ representing the desired
expressions, the expression encoder first maps $M_{e}$’s deformation into a
FACS-like latent expression code $z_{e}$. Similarly, the identity encoder maps
$M_{i}$ into an identity code $z_{i}$. Based on $z_{e}$ and $z_{i}$, the
decoder deforms $M_{i}$ to $M_{e}^{*}$ to approximate the expression mesh as
best as possible $M_{e}^{*}\approx M_{e}$. The trained deformation decoder
thus acts as a high-fidelity facial rig applicable to any facial mesh, with
human-friendly controls (its FACS-like latent space). The end-to-end pipeline
allows NFR to automatically transfer facial expressions to different
identities while maintaining interpretability. To accomplish this, we leverage
the recent advances in triangulation-agnostic neural geometry learning.
Namely, we use a combination of image CNN’s applied to renderings, along with
_DiffusionNet_ (DN) (Sharp et al., 2022) to build encoders to extract 3D shape
features, and use _Neural Jacobian Fields_ (NJF) (Aigerman et al., 2022) as a
decoder to produce deformations of the facial meshes.

#### Identity Encoder

We first feed a front-view rendering of $M_{i}$ through $CNN$, a plain 2D CNN,
to receive a code $c_{i}\in\mathbb{R}^{128}$. This $c_{i}$ is then fed into
$DN_{i}$, a DiffusionNet-based encoder. DiffusionNet is applied on Per-vertex
features, which is the local shape features of $M_{i}$, namely the
concatenation of the vertex coordinates $v_{i}$ and vertex normals $n_{i}$.
Intuitively, combining the 2D CNN and the Per-vertex features enables $DN_{i}$
to capture subtle facial features. $DN_{i}$ receives the neutral identity as
inputs, and outputs an identity latent code $z_{i}\in\mathbb{R}^{100}$, which
is the weighted means of the per-vertex output.

#### Expression Encoder

Similarly, for the expression encoder, the rendering of $M_{e}$ is fed into
$CNN$ to produce a code $c_{e}\in\mathbb{R}^{128}$, which is concantenated
with the Per-vertex features, $(v_{e},n_{e})$ to be fed to a DiffusionNet-
based encoder $DN_{e}$. In contrast to $DN_{i}$, $DN_{e}$ receives a face mesh
with an expression, and outputs a predicted expression code
$z_{e}\in\mathbb{R}^{128}$ whose elements are trained to be control parameters
that are interpretable. During training, the first 53 dimensions of $z_{e}$
are regularized to mimic the ARKit-compatible FACS. We denote them as
$z_{FACS}$. The remaining part of $z_{e}$, denoted as $z_{ext}$, is learned
from the data. This explicit decoupling is critical for NFR to learn realistic
expressions from real scans while maintaining maximum interpretability for
editing.

#### Decoder

We adopt NJF as the decoder, where the core is a multi-layer perception (MLP).
The MLP takes Per-triangle features, i.e. triangle centers $f_{i}^{c}$ and
normals $f_{i}^{n}$ of the identity mesh $M_{i}$, the expression code $z_{e}$,
the identity code $z_{i}$ and the shape code of the identity mesh $c_{i}$ as
the input, and outputs the deformation Jacobian $g^{*}\in\mathbb{R}^{3\times
3}$ for each triangle.

###  3.2. Datasets

The crux of our approach lies in the simultaneous training of both actual
facial expressions from scanned faces and computer-generated expressions
created through the manipulation of parametric controls on rigged meshes. As a
result of this strategy, our neural method is capable of generating highly
realistic expressions for these parameterized controls. Hence, we leverage two
types of data: synthetic data and real scans.

#### Synthetic data.

_ICT FaceKit (ICT)_ (Li et al., 2020) is a linear 3DMM for facial expressions.
It has 153 control parameters: a 100-dimensional PCA-based identity space, and
a 53-dimensional FACS-inspired hand-crafted expression rig. This FACS model is
compatible with the Apple ARKit blendshape model
111https://developer.apple.com/documentation/arkit/arfaceanchor/blendshapelocation
which allows us to collect data with plausible AU activations with ARKit face
tracking. These AUs have pre-defined semantics, e.g, the first parameter,
which is called ’browInnerUp_L’, corresponds to lifting the left inner
eyebrow. We adopt a template of ICT that has 3,694 vertices and 7,007
triangles after the mesh standardization (Sec. 3.3).

We generate two synthetic datasets with _ICT FaceKit_

  * •

_ICT-Random-AU_. We sample eight random identities and 5,819 random AUs per
identity. _ICT-Random-AU_ thus contains 41,322 samples. We select the first
six identities as the training set, one as validation, and one as testing.

  * •

_ICT-Real-AU_. We sample 22 random identities from ICT and 1,630 frames from
ARKit face tracking of our own performances to capture plausible AU
activations. We perform an 8:1:1 split on the frames. _ICT-Real-AU_ thus
contains 32,600 training frames, 3260 validation frames, and 3260 test frames.

#### Real Scans.

_Multiface_ (Wuu et al., 2022) is a large facial dataset with fine-grained
details from 4D scans. It contains 13 different identities. Each subject is
captured by a multi-camera setup at 30 FPS while reading a script designed to
cover over a hundred facial expressions. We use the same train/test split as
in (Wuu et al., 2022) while omitting expressions that are too close to the
neutral face from the training set. The training split contains 28,991 meshes
and the testing split contains 13,610 meshes. Each mesh has 5,385 vertices and
10,581 faces after applying the mesh standardization.

###  3.3. Data Augmentation and Standardization

#### Augmentation.

In-the-wild facial meshes vary in many ways: some have no necks, some have no
back of the head, and some have bad geometry regions and holes. We conduct two
types of data augmentation to adjust to these scenarios. (i) randomly shift
the input off the center and scale independently along the $x,y,$ and $z$
axis, and apply masks that remove parts of the face. (ii) randomly cut holes
on the input mesh to provide geometrical variance. We will show in Sec.4.7
this data augmentation is crucial for generalizing to in-the-wild inputs.

#### Standardization

Facial datasets have different ways of modeling the internal structures of
human faces, most notably eye sockets and the oral cavity. E.g. _ICT_ has
realistic eye sockets, whereas _Multiface_ covers up eye openings with meshes.
Therefore, we cut out the eye and mouth internals from mesh templates to
standardize.

###  3.4. Multi-dataset training

We introduce a multi-dataset training scheme to aggregate information from the
synthetic data and real scans. We first train on data generated from _ICT
FaceKit_ to initialize our model to imitate a linear 3DMM, then we further
train on a mix of real scans from the _Multiface_ dataset and synthetic data.

#### Decoder Training

Throughout the training, we use the same loss on the decoder, which contains a
vertex $L_{2}$ loss: $L_{v}=||v_{e}-v_{e^{*}}||^{2}$, a Jacobian loss
$L_{g}=||g-g^{*}||^{2}$ and a normal vector loss
$L_{n}=||n_{e}-n_{e}^{*}||^{2}$. Here $g$ is the ground truth deformation
Jacobian from $M_{i}$ to $M_{e}$. $v_{e}^{*}$, $n_{e}^{*},g^{*}$ are the
vertex positions, vertex normals, and deformation Jacobians of the output
deformed mesh $M_{e}^{*}$. Given the weights of three loss terms
$\lambda_{v}$, $\lambda_{g}$ and $\lambda_{n}$, the total loss of the decoder
is defined as:

(1) $L_{dec}=\lambda_{v}L_{v}+\lambda_{g}L_{g}+\lambda_{n}L_{n}.$

#### Training on ICT FaceKit.

We first train the model with the two _ICT_ datasets and explicitly supervise
$z_{FACS}$ with an $L_{2}$ loss to reproduce the same expression code as
_ICT_. The remaining latent expression code, $z_{ext}$, is forced to be zero.
We warm up the decoder by feeding the ground truth $z_{e}$ to the MLP. The
model is thus initialized to imitate a linear 3DMM. We define the output
expression code as $z_{e}^{*}=[z_{FACS}^{*},z_{ext}^{*}]$. The encoder loss is
defined as:

(2) $L_{enc}=||z_{FACS}-z_{FACS}^{*}||^{2}+||z_{ext}^{*}||^{2}.$

The total loss is a weighted sum of $L_{enc}$ and $L_{dec}$:
$L=\lambda_{e}L_{enc}+L_{dec}$.

#### Training on Multiface and ICT FaceKit

Second, we train with _Multiface_ to strengthen the network’s ability to
represent fine-grained expressions that out-performs linear 3DMM. The
challenge is that _Multiface_ does not have ground truth latent expression
parameters. To maintain an interpretable latent space, we rely on the fact
that NFR is correctly initialized to form a FACS-like latent space. We keep a
part of the batch from _ICT-Real-AU_ with direct latent supervision to
maintain this interpretability. _ICT-Random-AU_ is not used here since this
randomly generated dataset may contain unrealistic expressions. On _Multiface_
we simply regularize the latent space if the parameters go out of the range
$[0,1]$ to follow the convention of FACS AUs:

(3) $L_{r}(x)=\left\\{\begin{aligned} -x&,&x<0\\\ 0&,&0\leq x\leq 1\\\
x-1&.&x>1\end{aligned}\right.$

The loss of the encoder at this stage becomes:

(4) $L_{enc}=\left\\{\begin{aligned}
||z_{FACS}-z_{FACS}^{*}||^{2}+||z_{ext}^{*}||^{2}&,&\emph{ICT-Real-AU}\\\
L_{r}(z_{e}^{*})&.&\emph{Multiface}\end{aligned}\right.$

After these two stages of training, our model can retarget realistic
expressions to in-the-wild meshes and preserves an interpretable latent
expression space for manipulation.

##  4\. Experiments

In this section, a series of experiments are conducted to showcase the
capabilities of NFR. Sec. 4.1 evaluates the encoder’s effectiveness by
comparing inverse-rigging outcomes for _ICT-Real-AU_ against _Seol_ (Seol et
al., 2011). Sec. 4.2 benchmarks the expression quality of NFR against
template-specific mesh reconstruction methods. Sec. 4.3 confirms the
triangulation-agnostic property through inverse rigging experiments on _ICT-
Real-AU_ with a different mesh template. Sec. 4.4 demonstrates NFR’s
practicality by retargeting expressions to in-the-wild meshes of varying
connectivity and resolutions. Sec. 4.5 further illustrates the model’s
practicality, plotting the activations of each position in $z_{FACS}$ to
reveal a semantically related latent expression space. The ease of use is
demonstrated through two sequences of editing processes applied to in-the-wild
expressions. Sec. 4.6 qualitatively exhibits that sampling in the learned
expression space produces plausible expressions, surpassing a basic 3DMM.
Lastly, Sec. 4.7 presents various ablation studies to substantiate the key
design choices of the model and training scheme. The readers are referred to
the supplementary video for the details.

We use the per-vertex Euclidean distance as the error metric and additionally
report the 90% percentile value to emphasize the bad cases.

###  4.1. Inverse Rigging

_Inverse rigging_ is a task to find optimal rigging parameters fitting the
resulting deformation to a given geometry. Our encoder behaves as a
triangulation-agnostic inverse rig predictor extracting FACS AUs from a facial
mesh with unknown, entangled AU activations.

Table. 1 compares the inverse rigging results of our method against _Seol_
(Seol et al., 2011), an iterative optimization-based method that relies on the
ground truth rigging model. Here, _Ours (ICT rig)_ represents the deformation
from ICT’s linear 3DMM driven by our inverse-rigged parameters. _Ours (NFR)_
represents the deformation output of our decoder. In this complex setting
where many AUs are activated simultaneously, our method outperforms _Seol_ by
a large margin, where the performance of _Ours (ICT rig)_ validates the
effectiveness of the expression encoder, and that of _Ours (NFR)_ indicates
that the rigging space of our model is close to the ICT rig.

In Fig. 3 we apply $z_{FACS}$ of _Multiface_ on the ICT rig. Though limited by
the expression ability of the linear rig, the expressions on the ICT rig are
still semantically close to that of _Multiface_. This validates the inverse
rigging ability of NFR on in-the-wild expressions. Note that while _Ours
(NFR)_ performs slightly worse on the synthetic dataset ICT-Real-AU than _Ours
(ICT rig)_ , NFR’s major advantage is to work on in-the-wild data with
arbitrary triangulation and high-frequency details.

Table 1.  Inverse Rigging on _ICT-Real-AU_. Our encoder, evaluated both via ICT rig and NFR, outperform _Seol_ (Seol et al., 2011) by a large margin. Method |  Mean (mm) |  Median (mm) |  90% (mm)
---|---|---|---
Seol (Seol et al., 2011) |  0.688 $\pm$ 0.979 | 0.369 | 1.640
Ours (ICT rig) | 0.378 $\pm$ 0.467 | 0.225 | 0.860
Ours (NFR) |  0.443 $\pm$ 0.454 | 0.307 | 0.920
Figure 3. Inverse Rigging. Top: The input _Multiface_ expressions. Bottom:
Applying the latent $z_{e,FACS}$ to the ICT rig. All deformations on the
eyebrows, eyes, and mouth are solved correctly.

###  4.2. Expression Quality

In this section, we assess the ability of our model to generate precise
geometric details. The expression encoder is employed to map target
deformations into an interpretable expression space, subsequently
reconstructing the input deformation through the decoder. Our model is
compared to three template-specific competitors: COMA (Ranjan et al., 2018),
N3DMM (Bouritsas et al., 2019), and SpiralNet++ (Gong et al., 2019). All
models are trained on the same _Multiface_ training split, with our model also
receiving training on synthetic _ICT_ datasets. NFR is designed to maintain an
interpretable latent space and handle meshes in a triangulation-agnostic
manner, whereas competitor models are constrained by a single mesh template,
limiting their capacity to learn from diverse datasets and resulting in less
interpretability. latent spaces. A baseline method, _NFR (Multiface)_ ,
trained solely on the _Multiface_ dataset, demonstrates the advantages of
multi-dataset training.

Table. 2 reveals that NFR significantly outperforms its competitors, achieving
a 32.3% reduction in the 90% percentile value compared to the best-performing
competitor. Training on multiple datasets establishes a sparse and semantic
latent space and enhances expression reconstruction. In Fig. 4, we visualize
the per-vertex Euclidean error on _Multiface_ samples and provide a magnified
comparison in Fig. 5. NFR exhibits fewer coarse surface artifacts than other
baselines and more effectively preserves the shape around facial features such
as the eyes, nose, and mouth.

Figure 4. Qualitative reconstruction results on Multiface. We color the meshes by their per-vertex Euclidean error. We compare COMA, SpiralNet++, N3DMM, and NFR. The color map clearly shows that NFR outperforms the competing approaches. Figure 5. Zoomed-in comparison on Multiface. NFR not only gets rid of the coarse surface artifacts of the other baselines but also preserves the shape better around the eyes, nose, and mouth. Left is the ground truth. Table 2.  Quantitative reconstruction results on Multiface. NFR outperforms COMA, Neural3DMM, and SpiralNet++ on all metrics by a large margin. See Fig. 4 and Fig. 5 for a qualitative comparison.  Method |  Mean (mm) |  Median (mm) |  90% (mm)
---|---|---|---
COMA |  2.324 $\pm$ 1.724 | 1.858 | 4.380
Neural3DMM |  1.254 $\pm$ 1.092 | 0.951 | 2.406
SpiralNet++ |  1.256 $\pm$ 1.105 | 0.944 | 2.438
NFR (_Multiface_) |  1.005 $\pm$ 0.824 | 0.772 | 1.895
NFR | 0.879 $\pm$ 0.727 | 0.678 | 1.651

###  4.3. Triangulation Invariance

Our model is composed of multiple triangulation-agnostic components: The $CNN$
takes fixed-size rendered images as inputs; $DN_{e}$ and $DN_{i}$ are agnostic
to the mesh templates (Sharp et al., 2022); the MLP has shared parameters
across all the input triangles. Thus, our model naturally supports meshes with
different resolutions and connectivity. To quantitatively evaluate this
property, we apply our model on _ICT-Real-AU_ with a high-resolution mesh
template and different identities. The template has 10,089 vertices and 19,758
triangles. Table. 3 quantitatively validate that NFR has minimal performance
degradation on different triangulations, despite the fact that it is only
trained on the original mesh template with less than half of vertices and
triangles. Sec. 4.4 shows qualitatively that our model can transfer expression
to in-the-wild meshes, even raw scans with a significant amount of noise.

Table 3.  Triangulation Invariance. We compare the inverse-rigging performance of NFR on the trained mesh template (Original) and a retriangulated version (High-resolution) of _ICT-Real-AU_. NFR performs equally well on both datasets, even though it is not trained with the retriangulated template.  Method | Templates |  Mean (mm) |  Median (mm) |  90% (mm)
---|---|---|---|---
Ours (ICT rig) | Original |  0.378 $\pm$ 0.467 | 0.225 | 0.860
High-resolution |  0.343 $\pm$ 0.416 | 0.212 | 0.774
Ours (NFR) | Original |  0.443 $\pm$ 0.454 | 0.307 | 0.920
High-resolution |  0.444 $\pm$ 0.436 | 0.312 | 0.930

###  4.4. Retargeting in the Wild

Given the target mesh of an arbitrary subject with unknown expressions,
_retargeting_ is a task to interpret the target facial expression and transfer
it to another subject. Note that this task is more challenging than a typical
deformation transfer task where the neutral target mesh is known.

In Fig. 6, we test the generalizability of our model’s retargeting with
various identities, expressions, triangulations, and mesh quality from
different face models and datasets. The retargeted results to various neutral
face meshes successfully capture our high-level perception of the target
expressions with natural deformations, though the ground-truth AUs of the
target expressions are unknown. We retarget an ARKit face tracking sequence
and a _Multiface_ speaking sequence to these in-the-wild meshes. Please refer
to the supplementary video for details.

###  4.5. Human-Friendly Editing

NFR behaves as a user-friendly rig to edit the expressions with the latent
code activation intensities. Fig. 7 shows the series of such controllable
manual edits, starting off from the latent code inferred by the inverse
rigging, then increasing/decreasing the latent activations corresponding to
FACS AUs. We show all the 53 FACS-like control codes of our latent space in
Fig. 8. Note that each activation follows the semantics of the pre-defined
FACS.

###  4.6. Non-Linear Deformations

Along with the interpretable latent code learned from _ICT FaceKit_ , our
model also captures natural, non-linear deformations from _Multiface_. We show
this in Fig. 9 where many AUs are activated simultaneously. Linear 3DMMs
struggle with such complex activations as they simply add more offsets on top
of offsets, resulting in implausible deformations. A usual workaround is to
activate manually-sculpted corrective shapes of activation combinations to
introduce non-linearity (Lewis et al., 2014). Our model maintains natural,
non-linear deformations without such ad-hoc sculpting.

###  4.7. Ablations

We perform a series of ablation studies to demonstrate the effectiveness of
our key designs.

#### Multi-dataset Training Scheme

The training scheme is the core of our method for realistic expression
generation and interpretable editing. Fig. 7 demonstrates the effectiveness of
our user-friendly editing. In contrast, in Fig. 10 we show the activation of
the first five individual AUs, without training on the ICT datasets. The
deformed vertices are highly entangled. Without the direct supervision of ICT
AUs, making adjustments by tweaking the expression codes is impractical. We
then show quantitatively in Table. 4 that our model has poor quality on the
reconstructed expressions without training on the real world _Multiface_ data.

Table 4.  Ablation showing the importance of real data (_Multiface_). We evaluate on _Multiface_ test set with and without training on _Multiface_ real scans. Method |  Mean (mm) |  Median (mm) |  90% (mm)
---|---|---|---
With _Multiface_ | 0.879 $\pm$ 0.727 | 0.678 | 1.651
Without _Multiface_ |  1.470 $\pm$ 1.319 | 1.088 | 2.860

#### Data Augmentation

We show in Fig. 11 NFR with/without data augmentation to retarget meshes from
two different datasets: _FaceWarehouse_ (Cao et al., 2013) and
_Triplegangers_. Without training on the augmented datasets in Sec 3.3, the
model fails to apply the correct deformations to these in-the-wild meshes.

#### Extended latent space

We expand the latent expression space to be 128-dimensional, where the first
53-dimensional vector, denoted as $z_{FACS}$ is supervised by the ICT
expression codes. The remaining $z_{ext}$ is learned from data. Fig. 12
compares individual activations of $z_{FACS}$ between NFR and a similar model
without $z_{ext}$. Both models supervise $z_{FACS}$ to follow the ICT rig.
Since _Multiface_ contains deformation patterns that are not covered by the
ICT rig space, the model without $z_{ext}$ adapts some AUs, e.g.
‘eyeLookUp_L’, to capture these new patterns (e.g. neck deformation). With the
additional expression space of $z_{ext}$, NFR can capture the _Multiface_
deformations and maintain maximum interpretability.

#### Network Structures

The $CNN$ serves to generalize NFR to the translation and scale introduced in
data augmentation for better in-the-wild applicability. $\text{DN}_{e}$ and
$\text{DN}_{i}$ are crucial for solving the correct expression codes and
transferring them to in-the-wild meshes. Table. 5 compares a model without the
CNN features and another one that replaces the two DNs by PointNets (PN) (Qi
et al., 2017). The performance drops significantly.

Table 5.  Ablation study on the network structures. Without the CNN features, the model struggles to deal with the random shift and scale introduced in data augmentation. When replacing the DN layers with PN, the model fails to solve for the correct expression codes. Dataset | Method |  Mean (mm) |  Median (mm) |  90% (mm)
---|---|---|---|---
ICT-Real-AU | Ours (w/o CNN) |  1.034 $\pm$ 0.985 | 0.748 | 2.053
Ours (PN) |  0.710 $\pm$ 0.756 | 0.483 | 1.502
Ours (DN) |  0.443 $\pm$ 0.454 | 0.307 | 0.920
Multiface | Ours (w/o CNN) |  1.559 $\pm$ 1.278 | 1.184 | 3.011
Ours (PN) |  1.408 $\pm$ 1.207 | 1.050 | 2.758
Ours (DN) |  0.879 $\pm$ 0.727 | 0.678 | 1.651

##  5\. Conclusions

We have presented _Neural Face Rigging_ , a novel learning approach to
instantly rig and retarget 3D facial meshes in the wild with any reasonable
shape variations and triangulation. The key technical contribution is the
multi-stage training combining the advantages of different face datasets to
learn interpretable and editable latent code over high-fidelity facial
deformations. While our model is unique in its generalization to mesh
triangulation, it learns better facial deformation than other methods that
require fixed triangulation.

#### Limitations and Future Work

Although our model provides a triangulation-agnostic facial rigging and
retargeting pipeline, users still need a standardization step by removing
internal structures around the eyes and mouth. Segmenting the facial regions
could automate the process.

We chose to learn an ARKit-compatible FACS rig for its popularity and
accessibility. In theory, NFR could learn any other arbitrary rig
parameterizations such as MetaHuman, though this is not tested because of the
limited access to such data.

With an interpretable, controllable latent space and triangulation invariance,
our model can serve as a backbone for various facial animation tasks such as
talking face generation. Including the appearance model for photo-realistic
expression generation is also worth exploring.

Human faces are sensitive subjects. We must take precautions deploying our
model, with in-depth studies on the bias to different human attributes, e.g.
age, gender, and ethnicity. We hope this work can contribute to the community
by minimizing such bias with its ability to combine multiple face datasets and
perform instant rigging on in-the-wild meshes.

###### Acknowledgements.

This research is supported by Innovation and Technology Commission
(Ref:ITS/319/21FP) and Research Grant Council (Ref: 17210222), Hong Kong.

## References

  * (1)
  * Aigerman et al. (2022) Noam Aigerman, Kunal Gupta, Vladimir G. Kim, Siddhartha Chaudhuri, Jun Saito, and Thibault Groueix. 2022.  Neural Jacobian Fields: Learning Intrinsic Mappings of Arbitrary Meshes.  _ACM Trans. Graph._ 41, 4, Article 109 (jul 2022), 17 pages.  https://doi.org/10.1145/3528223.3530141
  * Bailey et al. (2020) Stephen W Bailey, Dalton Omens, Paul Dilorenzo, and James F O’Brien. 2020.  Fast and deep facial deformations.  _ACM Transactions on Graphics (TOG)_ 39, 4 (2020), 94–1.
  * Blanz and Vetter (1999) Volker Blanz and Thomas Vetter. 1999.  A morphable model for the synthesis of 3D faces. In _Proceedings of the 26th annual conference on Computer graphics and interactive techniques_. 187–194.
  * Bouritsas et al. (2019) Giorgos Bouritsas, Sergiy Bokhnyak, Stylianos Ploumpis, Michael Bronstein, and Stefanos Zafeiriou. 2019\.  Neural 3d morphable models: Spiral convolutional networks for 3d shape representation learning and generation. In _Proceedings of the IEEE/CVF International Conference on Computer Vision_. 7213–7222.
  * Brunton et al. (2014) Alan Brunton, Timo Bolkart, and Stefanie Wuhrer. 2014\.  Multilinear wavelets: A statistical shape space for human faces. In _European Conference on Computer Vision_. Springer, 297–312.
  * Cao et al. (2013) Chen Cao, Yanlin Weng, Shun Zhou, Yiying Tong, and Kun Zhou. 2013.  Facewarehouse: A 3d facial expression database for visual computing.  _IEEE Transactions on Visualization and Computer Graphics_ 20, 3 (2013), 413–425.
  * Cetinaslan and Orvalho (2020a) Ozan Cetinaslan and Verónica Orvalho. 2020a.  Sketching Manipulators for Localized Blendshape Editing.  _Graphical Models_ 108 (2020), 101059.
  * Cetinaslan and Orvalho (2020b) Ozan Cetinaslan and Verónica Orvalho. 2020b.  Stabilized blendshape editing using localized Jacobian transpose descent.  _Graphical Models_ 112 (2020), 101091.
  * Chandran et al. (2020) Prashanth Chandran, Derek Bradley, Markus Gross, and Thabo Beeler. 2020.  Semantic deep face models. In _2020 International Conference on 3D Vision (3DV)_. IEEE, 345–354.
  * Chandran et al. (2022a) Prashanth Chandran, Gaspard Zoss, Markus Gross, Paulo Gotardo, and Derek Bradley. 2022a.  Facial Animation with Disentangled Identity and Motion using Transformers.  _ACM/Eurographics Symposium on Computer Animation_ (2022).
  * Chandran et al. (2022b) Prashanth Chandran, Gaspard Zoss, Markus Gross, Paulo Gotardo, and Derek Bradley. 2022b.  Shape Transformers: Topology-Independent 3D Shape Models Using Transformers. In _Computer Graphics Forum_ , Vol. 41. Wiley Online Library, 195–207.
  * Choe and Ko (2006) Byoungwon Choe and Hyeong-Seok Ko. 2006.  Analysis and synthesis of facial expressions with hand-generated muscle actuation basis.  In _ACM SIGGRAPH 2006 Courses_. 21–es.
  * Choe et al. (2001) Byoungwon Choe, Hanook Lee, and Hyeong-Seok Ko. 2001\.  Performance-driven muscle-based facial animation.  _The Journal of Visualization and Computer Animation_ 12, 2 (2001), 67–79.
  * Choi et al. (2022) Byungkuk Choi, Haekwang Eom, Benjamin Mouscadet, Stephen Cullingford, Kurt Ma, Stefanie Gassel, Suzi Kim, Andrew Moffat, Millicent Maier, Marco Revelant, Joe Letteri, and Karan Singh. 2022\.  Animatomy: An Animator-Centric, Anatomically Inspired System for 3D Facial Modeling, Animation and Transfer. In _SIGGRAPH Asia 2022 Conference Papers_ (Daegu, Republic of Korea) _(SA ’22)_. Association for Computing Machinery, New York, NY, USA, Article 16, 9 pages.  https://doi.org/10.1145/3550469.3555398
  * Egger et al. (2020) Bernhard Egger, William A. P. Smith, Ayush Tewari, Stefanie Wuhrer, Michael Zollhoefer, Thabo Beeler, Florian Bernard, Timo Bolkart, Adam Kortylewski, Sami Romdhani, Christian Theobalt, Volker Blanz, and Thomas Vetter. 2020.  3D Morphable Face Models - Past, Present and Future.  _ACM Transactions on Graphics_ 39, 5 (August 2020).  https://doi.org/10.1145/3395208
  * Ekman and Friesen (1978) Paul Ekman and Wallace V. Friesen. 1978.  Facial action coding system: a technique for the measurement of facial movement. In _Consulting Psychologists Press_.
  * Gao et al. (2018) Lin Gao, Jie Yang, Yi-Ling Qiao, Yu-Kun Lai, Paul L Rosin, Weiwei Xu, and Shihong Xia. 2018.  Automatic unpaired shape deformation transfer.  _ACM Transactions on Graphics (TOG)_ 37, 6 (2018), 1–15.
  * Gong et al. (2019) Shunwang Gong, Lei Chen, Michael Bronstein, and Stefanos Zafeiriou. 2019.  Spiralnet++: A fast and highly efficient mesh convolution operator. In _Proceedings of the IEEE/CVF International Conference on Computer Vision Workshops_. 0–0.
  * Huber et al. (2016) P Huber, G Hu, R Tena, P Mortazavian, P Koppen, WJ Christmas, M Ratsch, and J Kittler. 2016.  A Multiresolution 3D Morphable Face Model and Fitting Framework.
  * Jiang et al. (2019) Zi-Hang Jiang, Qianyi Wu, Keyu Chen, and Juyong Zhang. 2019\.  Disentangled representation learning for 3d face shape. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_. 11957–11966.
  * Lewis et al. (2014) John P Lewis, Ken Anjyo, Taehyun Rhee, Mengjie Zhang, Frederic H Pighin, and Zhigang Deng. 2014\.  Practice and theory of blendshape facial models.  _Eurographics (State of the Art Reports)_ 1, 8 (2014), 2\.
  * Lewis and Anjyo (2010) John P Lewis and Ken-ichi Anjyo. 2010.  Direct manipulation blendshapes.  _IEEE Computer Graphics and Applications_ 30, 4 (2010), 42–50.
  * Li et al. (2010) Hao Li, Thibaut Weise, and Mark Pauly. 2010.  Example-based facial rigging.  _Acm transactions on graphics (tog)_ 29, 4 (2010), 1–6.
  * Li et al. (2020) Ruilong Li, Karl Bladin, Yajie Zhao, Chinmay Chinara, Owen Ingraham, Pengda Xiang, Xinglei Ren, Pratusha Prasad, Bipin Kishore, Jun Xing, et al. 2020\.  Learning formation of physically-based face attributes. In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_. 3410–3419.
  * Li et al. (2017) Tianye Li, Timo Bolkart, Michael J Black, Hao Li, and Javier Romero. 2017.  Learning a model of facial shape and expression from 4D scans.  _ACM Trans. Graph._ 36, 6 (2017), 194–1.
  * Monti et al. (2017) Federico Monti, Davide Boscaini, Jonathan Masci, Emanuele Rodola, Jan Svoboda, and Michael M Bronstein. 2017.  Geometric deep learning on graphs and manifolds using mixture model cnns. In _Proceedings of the IEEE conference on computer vision and pattern recognition_. 5115–5124.
  * Moser et al. (2021) Lucio Moser, Chinyu Chien, Mark Williams, Jose Serra, Darren Hendler, and Doug Roble. 2021\.  Semi-supervised video-driven facial animation transfer for production.  _ACM Transactions on Graphics (TOG)_ 40, 6 (2021), 1–18.
  * Paysan et al. (2009) Pascal Paysan, Reinhard Knothe, Brian Amberg, Sami Romdhani, and Thomas Vetter. 2009.  A 3D Face Model for Pose and Illumination Invariant Face Recognition. In _2009 Sixth IEEE International Conference on Advanced Video and Signal Based Surveillance_. 296–301.  https://doi.org/10.1109/AVSS.2009.58
  * Qi et al. (2017) Charles R Qi, Hao Su, Kaichun Mo, and Leonidas J Guibas. 2017\.  Pointnet: Deep learning on point sets for 3d classification and segmentation. In _Proceedings of the IEEE conference on computer vision and pattern recognition_. 652–660.
  * Ranjan et al. (2018) Anurag Ranjan, Timo Bolkart, Soubhik Sanyal, and Michael J Black. 2018.  Generating 3D faces using convolutional mesh autoencoders. In _Proceedings of the European conference on computer vision (ECCV)_. 704–720.
  * Ravi et al. (2020) Nikhila Ravi, Jeremy Reizenstein, David Novotny, Taylor Gordon, Wan-Yen Lo, Justin Johnson, and Georgia Gkioxari. 2020.  Accelerating 3D Deep Learning with PyTorch3D.  _arXiv:2007.08501_ (2020).
  * Roberts et al. (2021) Richard A. Roberts, Rafael Kuffner dos Anjos, Akinobu Maejima, and Ken Anjyo. 2021\.  Deformation transfer survey.  _Computers Graphics_ (2021).  https://doi.org/10.1016/j.cag.2020.10.004
  * Seol et al. (2011) Yeongho Seol, Jaewoo Seo, Paul Hyunjin Kim, John P Lewis, and Junyong Noh. 2011.  Artist friendly facial animation retargeting.  _ACM Transactions on Graphics (TOG)_ 30, 6 (2011), 1–10.
  * Sharp et al. (2022) Nicholas Sharp, Souhaib Attaiki, Keenan Crane, and Maks Ovsjanikov. 2022.  Diffusionnet: Discretization agnostic learning on surfaces.  _ACM Transactions on Graphics (TOG)_ 41, 3 (2022), 1–16.
  * Song et al. (2020) Steven L. Song, Weiqi Shi, and Michael Reed. 2020.  Accurate Face Rig Approximation with Deep Differential Subspace Reconstruction.  _ACM Trans. Graph._ 39, 4, Article 34 (aug 2020), 12 pages.  https://doi.org/10.1145/3386569.3392491
  * Sumner and Popović (2004) Robert W Sumner and Jovan Popović. 2004.  Deformation transfer for triangle meshes.  _ACM Transactions on graphics (TOG)_ 23, 3 (2004), 399–405.
  * Tan et al. (2018) Qingyang Tan, Lin Gao, Yu-Kun Lai, and Shihong Xia. 2018\.  Variational Autoencoders for Deforming 3D Mesh Models. In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_.
  * Tewari et al. (2017) Ayush Tewari, Michael Zollhofer, Hyeongwoo Kim, Pablo Garrido, Florian Bernard, Patrick Perez, and Christian Theobalt. 2017.  Mofa: Model-based deep convolutional face autoencoder for unsupervised monocular reconstruction. In _Proceedings of the IEEE International Conference on Computer Vision Workshops_. 1274–1283.
  * Verma et al. (2018) Nitika Verma, Edmond Boyer, and Jakob Verbeek. 2018\.  Feastnet: Feature-steered graph convolutions for 3d shape analysis. In _Proceedings of the IEEE conference on computer vision and pattern recognition_. 2598–2606.
  * Vesdapunt et al. (2020) Noranart Vesdapunt, Mitch Rundle, HsiangTao Wu, and Baoyuan Wang. 2020.  JNR: Joint-based neural rig representation for compact 3D face modeling. In _Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XVIII 16_. Springer, 389–405.
  * Wu et al. (2016) Chenglei Wu, Derek Bradley, Markus Gross, and Thabo Beeler. 2016.  An anatomically-constrained local deformation model for monocular face capture.  _ACM transactions on graphics (TOG)_ 35, 4 (2016), 1–12.
  * Wuu et al. (2022) Cheng-hsin Wuu, Ningyuan Zheng, Scott Ardisson, Rohan Bali, Danielle Belko, Eric Brockmeyer, Lucas Evans, Timothy Godisart, Hyowon Ha, Alexander Hypes, Taylor Koska, Steven Krenn, Stephen Lombardi, Xiaomin Luo, Kevyn McPhail, Laura Millerschoen, Michal Perdoch, Mark Pitts, Alexander Richard, Jason Saragih, Junko Saragih, Takaaki Shiratori, Tomas Simon, Matt Stewart, Autumn Trimble, Xinshuo Weng, David Whitewolf, Chenglei Wu, Shoou-I Yu, and Yaser Sheikh. 2022\.  Multiface: A Dataset for Neural Face Rendering. In _arXiv_.  https://doi.org/10.48550/ARXIV.2207.11243
  * Yang et al. (2022) Lingchen Yang, Byungsoo Kim, Gaspard Zoss, Baran Gözcü, Markus Gross, and Barbara Solenthaler. 2022\.  Implicit Neural Representation for Physics-Driven Actuated Soft Bodies.  _ACM Trans. Graph._ 41, 4, Article 122 (jul 2022), 10 pages.  https://doi.org/10.1145/3528223.3530156
  * Zhou et al. (2020) Yi Zhou, Chenglei Wu, Zimo Li, Chen Cao, Yuting Ye, Jason Saragih, Hao Li, and Yaser Sheikh. 2020\.  Fully convolutional mesh autoencoder using efficient spatially varying kernels.  _Advances in Neural Information Processing Systems_ 33 (2020), 9251–9262.

Figure 6.  In-the-wild retargeting. Yellow: target expression of non-rigged
face meshes with complex expressions composed of unknown AUs. Cyan: retargeted
results. NFR is robust to variations in identity, expression, triangulation,
and mesh quality. Figure 7.  Human-friendly editing. Yellow: non-rigged face
mesh. Gray: Meshes after editing AUs. Vertices are colored by the Euclidean
distance before and after each modification. Figure 8. Disentangled latent
code visualization. First 53 activations of our latent codes on a _MetaHuman_
mesh. Figure 9. Linear (top) vs. non-linear (bottom) models. Simultaneous AU
activation in linear models such as _ICT FaceKit_ ends up with unnatural
deformations. The non-linearity in NFR helps ensure rational deformations.
Figure 10. Ablation showing the importance of synthetic data (_ICT FaceKit_).
We show the effects of the first five AUs on _Multiface_ without training on
_ICT FaceKit_. The deformations are highly entangled and impractical for human
editing. Figure 11. Qualitative ablation of our data augmentation. Without
(top) vs. with (bottom) data augmentation. (a), (b) from _FaceWarehouse_ ,
(c), (d) from _Triplegangers_. Our data augmentation is critical to NFR
working on these in-the-wild meshes.

(a) Expression eyeLookIn_L lowers the left eye lid.

(b) The same deformation heatmap without $z_{ext}$.

(c) Expression eyeLookUp_L raises the left eye lid.

(d) The same deformation heatmap without $z_{ext}$.

Figure 12. Effects of $z_{ext}$. Here we plot the deformation heatmaps of two
expression codes. Left: with the extended latent space $z_{ext}$, $z_{FACS}$
successfully captures the disentangled deformation with correct semantics.
Right: Without $z_{ext},z_{FACS}$ gets contaminated with entangled deformation
in _Multiface_.

##  Appendix A Implementation Details

We render the front views of $S$ and $T$ by Pytorch3D (Ravi et al., 2020) with
a fixed point light and a gray Lambert shading. The model is not sensitive to
these settings as long as the whole face is shown in the image.

The rendered 256$\times$256 RGBD images feed into $CNN_{ST}$ which has four 2D
convolution layers followed by a fully-connected layer to output $c_{S}$ and
$c_{T}$. $\text{DN}_{e}$ and $\text{DN}_{i}$ have four and two diffusion
layers, respectively. $MLP_{dec}$ contains eight linear layers of 256 hidden
dimensions, with ReLU activations in between.

We train the model with $\lambda_{v}=10$, $\lambda_{g}=1$, $\lambda_{n}=1$ and
$\lambda_{e}=0.1$. During the first training stage, we warm up the MLP by
providing the ground truth $z_{FACS}$ in the first 100 epochs. And then train
another 200 epochs with $z^{*}_{FACS}$ from the expression encoder. In the
second stage, we train on both _ICT-Real-AU_ and _Multiface_ until
convergence. We set the initial learning rate as 1e-4 and decrease it by a
factor of 0.75 for every 100 epochs.

##  Appendix B Data Augmentation

Here we visualize the output meshes after data augmentation.

(a)

(b)

(c)

(d)

Figure 13. Data augmentation scheme. (a) and (c) show shift, scale, and
applying a front-facial mask. (b)and (d) show holes cutting. (a), (b) are
identities from the _ICT_ and (c), (d) are from _Multiface_.

##  Appendix C Standardization

We give two examples of the data standardization treatment. Fig. 14
illustrates the process for an in-the-wild raw scan, while Fig. 15 shows the
pipeline for an artist-created mesh.

Figure 14. Left: Raw Scan. Right: Raw Scan after standardization. For an in-
the-wild mesh with connected eyes, nose, and mouth, we need to cut those areas
to have the correct global solving of the deformation transfer process. Figure
15. Left: The original MetaHuman mesh. Right: The same mesh after
standardization. We remove the inner mouth socket to make it consistent with
other meshes during training.


††thanks: Now at the University of Sussex, Department of Physics and
Astronomy, Brighton††thanks: Corresponding author<EMAIL_ADDRESS>Now at
the University of California, Berkeley

# Total Neutron Cross-section Measurement on CH with a Novel 3D-projection
Scintillator Detector

A. Agarwal  University of Pennsylvania, Department of Physics and Astronomy,
Philadelphia, Pennsylvania, U.S.A.  H. Budd  University of Rochester,
Department of Physics and Astronomy, Rochester, New York, U.S.A.  J. Capó
Institut de Fisica d’Altes Energies (IFAE), The Barcelona Institute of Science
and Technology, Campus UAB, Bellaterra (Barcelona) Spain  P. Chong  University
of Pennsylvania, Department of Physics and Astronomy, Philadelphia,
Pennsylvania, U.S.A.  G. Christodoulou  CERN European Organization for Nuclear
Research, CH-1211 Genève 23, Switzerland  M. Danilov  Lebedev Physical
Institute of the Russian Academy of Sciences, Moscow, Russia  A. Dergacheva
Institute for Nuclear Research of the Russian Academy of Sciences, Moscow,
Russia  A. De Roeck  CERN European Organization for Nuclear Research, CH-1211
Genève 23, Switzerland  N. Dokania  Department of Physics and Astronomy, Stony
Brook University, Stony Brook, New York, U.S.A.  D. Douqa  University of
Geneva, Section de Physique, DPNC, Geneva, Switzerland  K. Dugas  Louisiana
State University, Department of Physics and Astronomy, Baton Rouge, Louisiana,
U.S.A.  S. Fedotov  Institute for Nuclear Research of the Russian Academy of
Sciences, Moscow, Russia  S. Gwon  Chung-Ang University, Seoul, South Korea
R. Howell  University of Rochester, Department of Physics and Astronomy,
Rochester, New York, U.S.A.  K. Iwamoto  University of Tokyo, Department of
Physics, Tokyo, Japan  C. Jesús-Valls  Institut de Fisica d’Altes Energies
(IFAE), The Barcelona Institute of Science and Technology, Campus UAB,
Bellaterra (Barcelona) Spain  C.K. Jung  Department of Physics and Astronomy,
Stony Brook University, Stony Brook, New York, U.S.A.  S. P. Kasetti
Louisiana State University, Department of Physics and Astronomy, Baton Rouge,
Louisiana, U.S.A.  M. Khabibullin  Institute for Nuclear Research of the
Russian Academy of Sciences, Moscow, Russia  A. Khotjantsev  Institute for
Nuclear Research of the Russian Academy of Sciences, Moscow, Russia  T. Kikawa
Kyoto University, Department of Physics, Kyoto, Japan  U. Kose  CERN European
Organization for Nuclear Research, CH-1211 Genève 23, Switzerland  Y. Kudenko
Institute for Nuclear Research of the Russian Academy of Sciences, Moscow,
Russia  Moscow Institute of Engineering and Physics (MEPhl), Moscow, Russia
Moscow Institute of Physics and Technology (MIPT), Moscow, Russia  S.
Kuribayashi  Kyoto University, Department of Physics, Kyoto, Japan  T. Kutter
Louisiana State University, Department of Physics and Astronomy, Baton Rouge,
Louisiana, U.S.A.  D. Last  University of Pennsylvania, Department of Physics
and Astronomy, Philadelphia, Pennsylvania, U.S.A.  L. S. Lin  University of
Pennsylvania, Department of Physics and Astronomy, Philadelphia, Pennsylvania,
U.S.A.  S. Lin  Louisiana State University, Department of Physics and
Astronomy, Baton Rouge, Louisiana, U.S.A.  T. Lux  Institut de Fisica d’Altes
Energies (IFAE), The Barcelona Institute of Science and Technology, Campus
UAB, Bellaterra (Barcelona) Spain  S. Manly  University of Rochester,
Department of Physics and Astronomy, Rochester, New York, U.S.A.  D. A.
Martinez Caicedo  South Dakota School of Mines and Technology, Rapid City,
South Dakota, U.S.A.  S. Martynenko  Department of Physics and Astronomy,
Stony Brook University, Stony Brook, New York, U.S.A.  T. Matsubara  High
Energy Accelerator Research Organization (KEK), Tsukuba, Ibaraki, Japan  C.
Mauger  University of Pennsylvania, Department of Physics and Astronomy,
Philadelphia, Pennsylvania, U.S.A.  K. McFarland  University of Rochester,
Department of Physics and Astronomy, Rochester, New York, U.S.A.  C. McGrew
Department of Physics and Astronomy, Stony Brook University, Stony Brook, New
York, U.S.A.  A. Mefodiev  Institute for Nuclear Research of the Russian
Academy of Sciences, Moscow, Russia  O. Mineev  Institute for Nuclear Research
of the Russian Academy of Sciences, Moscow, Russia  T. Nakadaira  High Energy
Accelerator Research Organization (KEK), Tsukuba, Ibaraki, Japan  E. Noah
University of Geneva, Section de Physique, DPNC, Geneva, Switzerland  A.
Olivier  University of Rochester, Department of Physics and Astronomy,
Rochester, New York, U.S.A.  V. Paolone  University of Pittsburgh, Department
of Physics and Astronomy, Pittsburgh, Pennsylvania, U.S.A.  S. Palestini  CERN
European Organization for Nuclear Research, CH-1211 Genève 23, Switzerland  A.
Paul-Torres  Louisiana State University, Department of Physics and Astronomy,
Baton Rouge, Louisiana, U.S.A.  R. Pellegrino  University of Pennsylvania,
Department of Physics and Astronomy, Philadelphia, Pennsylvania, U.S.A.  M.A.
Ramírez  University of Pennsylvania, Department of Physics and Astronomy,
Philadelphia, Pennsylvania, U.S.A.  C. Riccio  Department of Physics and
Astronomy, Stony Brook University, Stony Brook, New York, U.S.A.  J. Rodriguez
Rondon  South Dakota School of Mines and Technology, Rapid City, South Dakota,
U.S.A.  F. Sanchez  University of Geneva, Section de Physique, DPNC, Geneva,
Switzerland  D. Sgalaberna  ETH Zurich, Institute for Particle Physics and
Astrophysics, Zurich, Switzerland  W. Shorrock  Imperial College London,
Department of Physics, London, United Kingdom  A. Sitraka  South Dakota School
of Mines and Technology, Rapid City, South Dakota, U.S.A.  K. Siyeon  Chung-
Ang University, Seoul, South Korea  N. Skrobova  Lebedev Physical Institute of
the Russian Academy of Sciences, Moscow, Russia  S. Suvorov  Institute for
Nuclear Research of the Russian Academy of Sciences, Moscow, Russia  A. Teklu
Department of Physics and Astronomy, Stony Brook University, Stony Brook, New
York, U.S.A.  M. Tzanov  Louisiana State University, Department of Physics and
Astronomy, Baton Rouge, Louisiana, U.S.A.  Y. Uchida  Imperial College London,
Department of Physics, London, United Kingdom  C. Wret  University of
Rochester, Department of Physics and Astronomy, Rochester, New York, U.S.A.
G. Yang  Department of Physics and Astronomy, Stony Brook University, Stony
Brook, New York, U.S.A.  N. Yershov  Institute for Nuclear Research of the
Russian Academy of Sciences, Moscow, Russia  M. Yokoyama  University of Tokyo,
Department of Physics, Tokyo, Japan  P. Zilberman  Department of Physics and
Astronomy, Stony Brook University, Stony Brook, New York, U.S.A.

###### Abstract

In order to extract neutrino oscillation parameters, long-baseline neutrino
oscillation experiments rely on detailed models of neutrino interactions with
nuclei. These models constitute an important source of systematic uncertainty,
partially because detectors to date have been blind to final state neutrons.
Three-dimensional projection scintillator trackers comprise components of the
near detectors of the next generation long-baseline neutrino experiments. Due
to the good timing resolution and fine granularity, this technology is capable
of measuring neutron kinetic energy in neutrino interactions on an event-by-
event basis and will provide valuable data for refining neutrino interaction
models and ways to reconstruct neutrino energy. Two prototypes have been
exposed to the neutron beamline at Los Alamos National Laboratory (LANL) in
both 2019 and 2020, with neutron energies between 0 and 800 MeV. In order to
demonstrate the capability of neutron detection, the total neutron-
scintillator cross section as a function of neutron energy is measured and
compared to external measurements. The measured total neutron cross section in
scintillator between 98 and 688 MeV is 0.36 $\pm$ 0.05 barn.

##  I Introduction

The goal of the current and future long-baseline (LBL) neutrino oscillation
experiments is to perform precise measurements of neutrino oscillations,
determine the neutrino mass hierarchy and $\theta_{23}$ octant, and test if
neutrinos violate CP symmetry. A key element in the measurement sensitivity is
the precision with which the neutrino energy of each event can be determined.

These experiments reconstruct the energy of the neutrino or anti-neutrino
based on measurements of the resultant visible particles from the neutrino
interaction. Neutrons produced in the interactions may carry a significant
fraction of the energy and have heretofore been hard to detect and measure.
Consequently, neutrons present a significant challenge, but also a major
opportunity for improvement in the reliability and precision of the neutrino
energy reconstruction.

The near detectors of LBL experiments must make high-precision measurements of
neutrino interactions for a palatable cost. Since weak interaction cross
sections are small, the occupancy of neutrino detectors is low compared with
detectors measuring charged-particle beams. A novel approach is taken for the
upgrade of the T2K near detector [1], employing solid scintillator cubes as
the neutrino target. The optically isolated cubes are one centimeter per side
and are arranged in a three-dimensional array. The cube surfaces were etched,
forming a reflective 50-80 $\mu$m thick polystyrene micropore deposit [2].
Each cube has three orthogonal holes with a diameter of 1.5 mm. Optical fibers
with a diameter of 1 mm pass through the cubes in the $x$, $y$, and $z$
directions (three fibers per cube through these holes). Each fiber passes
through a full row or column of cubes. The conceptual design of the detector
is shown in Fig. 1.

Figure 1: The concept of the 3D-projection scintillator tracker. Figure is
taken from [1].

When a charged particle passes through a cube, light generated by the
scintillator is collected by the wavelength-shifting (WLS) optical fibers [3].
Photon sensors at the end of each fiber detect this light [4]. Using timing
and geometry, the outgoing particles from a neutrino interaction can be
reconstructed in three dimensions. This approach yields a significant
improvement in reconstruction capability for particle trajectories transverse
to the neutrino beam over previous designs which employ planes of scintillator
bars [5]. The approach further enables the measurement of the kinetic energy
of outgoing neutrons by time-of-flight (ToF) techniques from energy deposits
in the primary medium of the detector. The importance of neutron kinematic
detection in the T2K upgrade has been discussed [1] [6] [7].

Our team utilized the Los Alamos National Laboratory (LANL) Weapons Neutron
Research (WNR) facility [8] to measure the detailed neutron response of two
prototypes of the near detector, known as SuperFGD, that will be deployed in
the T2K experiment in Japan. We operated two prototype detectors in the
beamline for two one-week periods in both December 2019 and December 2020.
Here, we describe a measurement of the total neutron cross section on
polystyrene using one of the prototype detectors. This measurement represents
an improvement in the precision of this cross section for neutron energies
between 500 and 688 MeV.

The paper is arranged as follows. In section II, we discuss the setup of the
experiment including the beamline and the detector. In section III, we
describe the the detector calibration and in section IV, we introduce the
methodology for the total cross-section measurement. In section V, we explain
the event reconstruction and selection, followed by the systematic uncertainty
consideration in section VI. In the last section, we talk about the
measurement result with some discussion.

##  II Experimental setup

WNR provides a spallation-produced neutron beam with kinetic energies between
0 and 800 MeV. The primary proton beam is composed of sub-nanosecond wide
proton bunches separated by 1.8 $\mu$s [9]. Each proton bunch produces photons
and neutrons along with other hadrons that are swept away by a magnetic field.
The photons and neutrons pass through an aperture in the shielding and
traverse the flight path (90 m). The ToF of neutrons is determined by
measuring their arrival times relative to the initial flux of photons (gamma
flash). The highest energy neutrons arrive soon after the gamma flash while
the lowest energy neutrons come much later, with some arriving after the gamma
flash from the subsequent proton bunch (wrap-around neutrons). By positioning
our detector at 90 m from the beam target, the farthest location from the
tungsten target in the facility, we enhance the energy resolution for the
highest energy neutrons while suffering wrap-around for neutrons below 13 MeV.
In order to shape the neutron beam profile, a 0.4 cm radius collimator was
located 1 m upstream of the detector.

The data analyzed here are from our deployment of a 24$\times$8$\times$48 cm3
prototype detector consisting of 9216 1 cm scintillator cubes, exposed to the
neutron beam [10]. This is the same detector deployed in a charged particle
beam at CERN [11]. The detector was oriented such that the neutron beam
($z$-direction) was parallel to the longest dimension. In the transverse
plane, the horizontal ($x$-direction) dimension was 24 cubes wide and the
vertical ($y$-direction) dimension was 8 cubes tall. Each energy deposit
signal in a cube was collected by three WLS fibers and mapped in three
orthogonal views: $XY$ (beam view), $XZ$ (top view) and $YZ$ (side view). The
detector used three types of Hamamatsu MPPCs, S13360-1325CS, S13081-050CS and
S12571-025C, installed in three different regions of the top view of the
detector. The arrangement can be found in Fig. 5 of Ref. [11]. The beam view
and side view are only equipped with Hamamtsu S13360-1325CS MPPCs. The MPPCs
signals are read out by customized front-end boards (FEBs) [12]. Fig. 2 shows
event time distribution with respect to $T_{0}$. The $T_{0}$ is the time
proton bunch hits the tungsten target in the beamline. The distribution in
Fig. 2 has been obtained using only a subset of the data. The gamma peak width
for a single channel, which is 1.4 ns, provides a validation of the timing
resolution. A hit is a single fiber channel readout above a threshold. The
base threshold for the hit is 40 PE in order to remove the cross-talk. An
event includes all hits in a 1.8 $\mu$s time window. The gamma peak appears
first in time, followed by the neutron peak. By selecting events in the latter
and measuring their time relative to the former, we can determine the energy
of the neutron. Reconstructed neutron energy is required to be $>13$ MeV to
get rid of wrap-around neutrons.

Figure 2: Detected event time distribution with respect to $T_{0}$. The first
peak is the gamma flash after the protons hit the tungsten target and the
following delayed peak is the neutron. The FEB sampling rate is every 2.5 ns.
The binning size is the same as the sampling time size.

Fig. 3 shows the hit distributions of a single neutron interaction candidate
with a ToF-determined kinetic energy of 173 MeV in the YZ view. The deposited
energy is reconstructed to be 112 MeV. For this event the deposited energy is
lower than the ToF energy likely because of the binding energy and the
invisible particles such as secondary neutrons leaving the detector.

Figure 3: Single neutron interaction candidate with 173 MeV TOF-measured
kinetic energy on the YZ view. The reconstructed deposited energy is about 112
MeV. Note the Y and Z may have different sizes for one centimeter.

In 2018, the prototype detector was exposed to a charged particle beam at
CERN, where light yield for the minimum ionizing particle (MIP) for one fiber
readout was found to be 52.5, 51.6, 42.1 photoelectrons (PE) on average with 8
cm fibers for MPPC type I, II, and III, respectively. The time resolution for
a single fiber readout was found to be 1.1 ns [11].

##  III Calibration

For the gain of a single channel, defined as a fiber readout with a single
MPPC and processed through the FEB, the calibration was carried out with a
pulsed LED. The gain of each channel was measured in the absence of the beam.
The gain is extracted from the PE peaks distribution measured in the presence
of LED signals for each MPPC channel using a custom LED system [13, 11].

In addition to the LED calibration, cosmic muons were used to construct a
light yield uniformity map for all the detector channels. More than 22,000
through-going muons were selected. The direction of each muon was fitted, thus
the light yield per travel length was obtained for the channels that the muon
passed through. The average light yield with all muons is used for each
channel. Combining the gain and light yield calibrations, a uniform response
for all channels was obtained. For the majority of the channels, the light
yield measured with cosmic muons is consistent with the expected light yield
from the 2018 charged particle beam test. We apply two additional corrections:
time walk, the dependence of measured time on the deposited energy, and light
attenuation in the fiber (lower than 10$\%$), measured during the CERN beam
test [11].

##  IV Methodology for the total cross-section measurement

The total neutron cross-section measurement was estimated utilizing the so-
called extinction technique [14]. In the presence of neutron-nucleus
interactions, the signal event rate decreases exponentially along the
$z$-coordinate:

$N\left(z\right)=N_{0}e^{-T\sigma_{\mathrm{tot}}z},$ (1)

where $N_{0}$ and $N\left(z\right)$ are the neutron event rates at the
reconstructed $z$ positions in the first layer, and in a more downstream layer
in the detector. $T$111$T=\left(\rho_{\mathrm{CH}}\times
N_{\mathrm{Avogadro}}\right)/m_{\mathrm{CH}}$ $=\,4.623\times
10^{22}\,\mathrm{nucleons/cm^{3}}$ and $\sigma_{\mathrm{tot.}}$ are the
nuclear density and the neutron total cross section, respectively. Neutron
interactions in the detector cause an event rate depletion from which we can
extract the neutron total cross section as a function of its kinetic energy.
By fitting the event rate distribution along $z$ using an exponential function
of the form $N_{0}e^{-\lambda z}$ (in accordance with Eq. 1), the exponential
coefficient $\lambda$, referred to as the extinction coefficient, can be
determined. The total cross section is extracted by fitting the event rate
distribution along $z$ for each bin of neutron energy. The beam center was
measured for each layer to ensure the detector was orthogonal to the beam.

The signal is defined as single-track events because it is easier to identify
the vertex and removes the potential issues of pile-up events and light noise.
For each layer the cross-section ratio of single-track and multiple-track
topologies is constant. Thus, the single-track event rate depletion along $z$
gives the total cross section for each energy range.

In this letter, the measured neutron total cross section on the plastic
scintillator (CH), is reported from 98 to 688 MeV. The energy binning was
optimized taking into account the energy resolution. The region with low
kinetic energy neutron candidates ($<$ 98 MeV) does not result in long enough
clusters to form tracks (a few centimeters) and the uncertainty due to
’invisible’ scattering222Invisible scattering includes elastic scattering as
well as any interactions that do not produce visible tracks above the
threshold. in this region is large. The region above 688 MeV is statistically
limited.

##  V MC simulation

A realistic geometry has been generated to simulate the detector, experimental
hall and the beamline. The finer detector structure such as each cube, cube
hole, WLS fiber and the MPPC is implemented in the geometry. Two collimators
upstream the detector at 20 m and 89 m have been included in the geometry as
well.

The Geant4 simulation is used for neutron interaction and particle propagation
in the detector, to provide some of the systematic uncertainty evaluation and
model comparison. The Bertini model is chosen as our default model [15]. As an
alternative, the INCLXX model is used throughout the analysis to cross-check
against any model dependence introduced by the choice of the default model
[16]. For both the systematic uncertainty evaluation and the total cross
section comparison, these two models show very consistent results, thus for
the remaining of the paper, we present the results with Bertini model. Our
cross-section measurement has no simulation or model dependency.

A comparison between the reconstructed neutron energy distribution in MC and
data is shown in Fig. 4. The simulated events were generated following the
measured neutron flux[17]. The measured neutron energy in data and simulation
are consistent within the MC statistical error. It is worth to note that the
scintillation light yield non-linearity is not included in the current
simulation. Constant light yields extracted from the data are used for the MC
simulation. In future studies, this effect will be taken into account.

Figure 4: Neutron energy spectra obtained with data (blue line) and MC (black
line). The error bars represent the MC statistical uncertainty, while the
statistical uncertainty associated with data is included but too small to be
visible. The measured flux (red dashed line) is included as reference.

##  VI Event reconstruction and selection

The goal of the event reconstruction for this analysis is to aggregate
information on hits recorded by the prototype and convert them into
information from which we select neutron interactions producing one
reconstructed track in the event. Only events occuring after the gamma flash
are selected, for this reason we select hits collected in a time window
between $-815$ and 850 ns with respect to T0. To reject hits produced by MPPC
noise and optical cross talks in adjacent cubes, we set a base threshold of 20
PE per hit and we require a minimum number of three hits in an event. After
having sorted the hits in time, a time clustering algorithm is used: if
consecutive hits are greater than 17.5 ns apart they belong to different
clusters. This time clustering value has been optimized such that the
probability of overlapping neutrons in one single event is below 0.1$\%$.

After that, only events with one time cluster are selected to avoid pile-up
with other neutron interactions. At this point, voxels, defined as
reconstructed cubes, are built combining the three views of the detector and a
density-based spatial clustering algorithm (DBScan) is employed to select
voxels close in space [18]. A cluster is defined if there is at least one
voxel and if the distance between voxels does not exceed 1.8 cm, which ensures
there are not holes between voxels. Once the spatial clusters are built, we
select events with only one spatial cluster. The particle traveling in the
beginning and end voxels may not go through the full cube. In order to remove
the z-dependency on the selection, layer-dependent PE cuts on the first and
last voxels are applied for each track. The PE cut values are determined by
minimizing the track length distribution difference among all layers.

For each cluster a matrix is defined that encodes the distance between the
center of the cluster and each voxel:

$M_{ij}=\sum^{N}_{ij}\frac{(\vec{v}-\vec{c})_{i}(\vec{v}-\vec{c})_{j}}{N},$
(2)

where $N$ is the total number of voxels in the cluster, $\vec{v}$ is the
displacement vector associated with a voxel in the cluster and $\vec{c}$ is
the center of the cluster. A Principal Component Analysis (PCA) of this matrix
is performed to find the three principal eigenvectors and eigenvalues. A
variable defined as $L=(\lambda_{1}-\lambda_{2})/\lambda_{1}$, where
$\lambda_{1}$ and $\lambda_{2}$ are the first two eigenvalues of the PCA
components, quantifies the development of the cluster along the principal
vector. Requiring $L>$ 0.7 rejects clusters inconsistent with a single final
state particle origin. Two other variables are employed to select straight
tracks. The first is the maximum distance between a voxel and the line defined
by the largest principal vector. The second is the largest projected distance
between two voxels on the second principle eigenvector. They are required to
be below 1.2 cm and 1.4 cm, respectively.

Only events having between three and eight voxels are accepted to reduce the
dependency on the detector acceptance. For such events, the first voxel along
the $z$-coordinate is taken as the vertex. In addition, only events with a
vertex within a rectangular parallelepiped built around the beam center of
1.5$\times$1.5$\times$40 cm3 are selected.

The first $z$ layer of the prototype is rejected since it is contaminated by
the hits produced by particles from the interactions in the upstream material
and collimator. The last nine layers are rejected as a result of the cut on
the number of voxels mentioned above. The timing associated with the selected
vertex is used to compute the energy of the neutron using ToF.

There is almost no background in the final sample. The overlapping events are
below 0.1$\%$ due to the low event rate and they are rejected by the time and
space clustering. Multiple interactions are rejected by the single cluster
selection. The remaining multiple interactions with the first interaction
invisible are included in the invisible scattering uncertainty described in
the next section. The background from the neutron interaction in the
collimator upstream of the detector and cosmic muons have been rejected by
cuts on the fiducial volume. The final sample purity is above 99$\%$.

##  VII Systematic uncertainties

The detector has a geometrical and electronics non-uniformity which generates
a detection uncertainty. The fiber, fiber hole, MPPC, insulating Tyvek layer
and cubes have some variation in their alignment. For our neutron experiment,
the beam profile is rather narrow and our measurements may be sensitive to
such variations in the detector. Additionally, we employ three different types
of MPPCs for the prototype and they are not uniformly deployed on the
detector.

To evaluate the uncertainty associated with these variations, we compare our
results to a “no-cut” sample. The no-cut sample only requires a hit to have
more than 20 PE and any number of reconstructed voxels. All other topological
cuts existing in the single-track selection are removed. For each energy
range, the event rate along $z$ in the “no-cut” sample was normalized to that
in the single-track sample. Then the event rate fractional difference at each
layer between the two samples is taken as the detector systematic uncertainty.
Fig. 5 shows the resulting systematic uncertainty by calculating the residual
of the two samples as a function of the neutron kinetic energy and $z$. The
“no-cut” sample is normalized to the single-track signal sample. To understand
the cause of the detector systematic uncertainty, three dedicated studies were
completed. First, we compared the event rate along Z with the same rate in the
data taken with the detector rotated by 180 degrees around Y (vertical). The
ratio between the two is consistent with the fractional uncertainty shown in
Fig. 5. This ruled out that the reconstruction is the major reason for such
uncertainty. Second, we used only two views, the beam and side view, which are
equipped with the same MPPC type. Also in this case we compared the event rate
we measured with the non- and rotated detector finding a good agreement with
the estimated fractional uncertainty. This test indicates that the difference
between MPPC-type is not the major reason as well. Then, we performed a MC
simulation introducing a misalignment of $\pm$0.5 millimeter for groups of 10
cubes. Such misalignment can introduce variations similar to the fractional
systematic uncertainty shown in Fig. 5. Therefore, we concluded that the
dominant reason for the detector systematic uncertainty is likely to be the
cube misalignment due to the fact that the cubes were assembled without a
solid support on the bottom, but they were supported only by the fibers.

Figure 5: The fractional systematic uncertainty due to the detector non-
uniformity and reconstruction inefficiency. See text for detail.

The extinction method requires the reconstruction of the first interaction
vertex. Intrinsic contamination arises when neutrons scatter, leaving a
deposited energy below the detector threshold. In this case, the primary
vertex is missed and a subsequent scattering may be mis-reconstructed as the
vertex. Generally, the secondary vertex is downstream of the true primary
vertex.

This invisible scattering constitutes a source of systematic uncertainty. In
order to estimate the uncertainty, we studied both the Bertini and INCLXX
models in Geant4 version 10.3, and the conclusions are consistent with each
other. One of the consequences of invisible scattering is that it generates
transverse spread of the neutron beam. We conservatively assumed all of the
transverse spread comes from invisible scattering, tuned the simulations to
match the spread seen in data, and calculated the change in the reconstructed
neutron cross section.

Fig. 6 shows the width of the vertex transverse spread as a function of $z$
with different fractions of invisible scattering in the MC simulation
(indicated as ”weight” in the legend). Varying such fractions change the
strength of invisible scattering. For example, simulation with weight 0
correspond to absence of invisible scattering including elastic scattering. If
an event had one invisible scattering before leaving visible signal, the event
was assigned a weight of 0.6. If twice, it was assigned a weight of 0.36. The
weight of 0.6 shows the best match with data.

The impact of the invisible scattering is as large as 10% below 100 MeV but
limited to a few percent above 100 MeV. The difference between the cross
section results with and without the invisible scattering is taken as a
systematic uncertainty.

Fig. 6 shows the transverse spread as a function of $z$ integrated over the
entire neutron energy range. In general, the spread increases as function of
the depth in the detector and is more pronounced for low energy than high
energy neutrons.

Figure 6: Width of the vertex transverse spread as a function of Z with
different invisible scattering strength. The weight reported in the legend is
the factor applied to the original invisible scattering strength in the model.
For example, simulation with weight 0 corresponds to no invisible scattering
including elastic scattering. MC simulation has been normalized to data.

The limited dimensions of the prototype have an impact on the single-track
event selection efficiency. For the single-track selection, the downstream
part of the detector has higher efficiency than the upstream part. The
downstream part reduces the room for multiple-track events to develop. A data-
driven correction is considered in order to avoid any possible bias. We
compare the number of events selected at a given $z$-layer with different
total lengths of fiducial volume (e.g. $z$-layer is 2 and use the detector
through $z$-layer 48, then $z$-layer is 2 and use the detector through
$z$-layer 47, and so on). The uncertainty on the acceptance correction was
computed by setting the starting layer to 2 through 8 and taking the largest
acceptance correction difference between any two fixed layers. The resulting
uncertainty is under 2% for the whole energy range.

The finite timing resolution results in an uncertainty on the neutron ToF and
consequently its reconstructed kinetic energy. The overall uncertainty was
computed combining the resolution for a single fiber mentioned above and the
uncertainty on T0, and it was found to be 1.37 ns.

Contribution to the uncertainty on the cross section by the timing uncertainty
is estimated varying the ToF values thousands of times according to a Gaussian
distribution centered at the actual ToF and a width corresponding to the
timing resolution. The cross section is extracted for every variation and the
spread of the resulting distribution is taken as the uncertainty. In addition,
in order to evaluate the light yield variation, for each channel, the light
yield fluctuation estimated by the cosmic muon track fitting is propagated
through the reconstruction with a simulated neutron interaction sample,
providing the uncertainty on the vertex location induced by light yield
fluctuations.

Finally, we considered uncertainty due to neutrons interacting in the
collimators located upstream of the detector. If neutrons lose energy in such
interactions, they can arrive in the detector with an energy lower than that
reconstructed by ToF. We studied this effect with simulated data and found it
to be negligible. Generally, neutrons which lose a large amount of energy in
the collimators are scattered transversely and are not included in our data
sample. Neutrons that scatter in the collimator and interact in the detector
lose less than 1 MeV. This uncertainty is propagated to the cross-section
uncertainty by varying the energy distribution according with the spread
induced by the neutron interactions inside the collimators.

Contribution by the uncertainties that do not change the neutron energy are
computed as a function of $z$-layer and neutron energy (e.g. Fig. 5). They are
propagated in the following way: the number of events in each $z$-layer and
energy bin is varied according to a Gaussian with mean the number of events in
that particular bin and width the estimated uncertainty. The neutron cross
section is extracted from every variation and the width of the distribution is
taken as uncertainty. As stated before, the major cause of the systematic
uncertainty is the cube misalignment. The correlation of the cube misalignment
at each layer is not assessable. This measurement relies on this assumption
and the future experiments should consider dedicated ways to measure the
uncertainty induced by the cube misalignment. For example, systematic cube
alignment variations can be conducted to understand the impact of the
misalignment.

Additionally, we performed the following checks to confirm the robustness of
the analysis:

  * •

The cross-section was extracted from a MC sample obtained employing the same
reconstruction and selection as for data. The results were consistent with the
input total cross section within the uncertainties we have evaluated.

  * •

Data were divided into individual calendar days and comparison of vertex
distributions between each other showed consistency.

  * •

Different fitting ranges were employed to understand the local structure of
the $z$ distribution

  * •

A constant term was added into the exponential function to to include any
potential constant background such as noise or external background across $z$.
It results in a consistent cross-section measurement as the one obtained using
the nominal exponential function.

  * •

The total cross section was re-evaluated by refitting the exponential function
after subtracting the invisible scattering predicted by the tuned simulation
for each $z$ layer from data. The resulting total cross section is consistent
with the original result.

##  VIII Result and discussion

With the exponential fit to the vertex distribution along z for each energy
range, the total neutron-CH cross section is obtained as a function of the
neutron kinetic energy. All the systematic uncertainties discussed above are
included varied separately and then summed in quadrature. The total neutron
cross section on hydrocarbon as a function of neutron kinetic energy is shown
in Fig. 7. For neutron energies below 200 MeV, we obtain a slightly higher
measured value than the Bertini model in Geant4.

Figure 7: The total neutron-CH cross section as a function of neutron kinetic
energy. The black vertical bars represent the total uncertainty and the red
the statistical uncertainty. The Geant4 Bertini model is shown in blue.

In the energy region between 200 to 688 MeV, our measurement shows good
agreement with the model. The overall data and the model agree within the
uncertainties. It should be noted that for each energy bin, the measurement is
effectively independent and systematic uncertainties are considered
uncorrelated across energy bins. For the region below 150 MeV, the
$\chi^{2}$/d.o.f. is 15.2/11 and for that below 200 MeV, the $\chi^{2}$/d.o.f.
is 16.1/18. The energy-integrated (98-688 MeV) cross section is 0.36 $\pm$
0.05 barn with a $\chi^{2}/d.o.f.$ of 22.03/38, corresponding to 38 layers in
$z$. The comparison between the true and reconstructed cross sections in the
MC has been done in order to understand the bias introduced by the
reconstruction. The difference among them is well within the error bars. The
total uncertainty is broken down into each contribution by various sources in
Fig. 8. The total uncertainty is dominated by the contribution from the
detection systematic uncertainty.

Figure 8: Breakdown of the total uncertainty. The systematic uncertainty
includes contributions from invisible scattering, light yield, detection and
reconstruction, time resolution, collimator interaction and geometric
acceptance correction.

Our result on CH from 98 to about 600 MeV is consistent within the error bar
with existing neutron-Carbon total cross-section measurements[19, 20, 21, 22],
indicating a minor contribution from the neutron-Hydrogen interaction. Our
measurement provides a new result on the total neutron-CH cross section across
a broad energy range that is important for the LBL neutrino oscillation
experiments.

##  IX Data availability

Data will be made available on request.

###### Acknowledgements.

This work was performed, in part, at the Los Alamos Neutron Science Center
(LANSCE), a NNSA User Facility operated for the U.S. Department of Energy
(DOE) by Los Alamos National Laboratory (Contract 89233218CNA000001). We thank
Keegan Kelly for technical support throughout the experiment and analysis.
This work was supported by JSPS KAKENHI Grant Numbers JP26247034, JP16H06288,
and JP20H00149. This work was supported in part by the RSF grant
No.19-12-00325 and by the MHES (Russia) grant ”Neutrino and astroparticle
physics” No. 075-15-2020-778. We acknowledge funding from the Spanish
Ministerio de Economía y Competitividad (SEIDI-MINECO) under Grants No.
PID2019-107564GB-I00. IFAE is partially funded by the CERCA program of the
Generalitat de Catalunya. We ackowledge the Swiss National Foundation through
grant No. 200021 85012. We further acknowledge the U.S.-Japan Science and
Technology Cooperation Program in High Energy Physics, the support of the US
Department of Energy, Office of High Energy Physics and the support from the
University of Pennsylvania.

## References

  * [1] T2K Collaboration.  T2K ND280 Upgrade - Technical Design Report.  e-Print: 1901.03750 [physics.ins-det].
  * [2] Yu. G. Kudenko, L. S. Littenberg, V. A. Mayatsky, O. V. Mineev, and N. V. Ershov.  Extruded plastic counters with WLS fiber readout.  Nucl. Instrum. Meth. A, 469:340–346, 2001.
  * [3] Kuraray website.  http://kuraraypsf.jp/psf/ws.html.  Accessed: 2022-03-31.
  * [4] Hamamatsu website.  https://www.hamamatsu.com/jp/en/product/optical-sensors/mppc.html.  Accessed: 2022-03-31.
  * [5] P. A. Amaudruz et al.  The T2K Fine-Grained Detectors.  Nucl. Instrum. Meth. A, 696:1–31, 2012.
  * [6] L. Munteanu et al.  New method for an improved antineutrino energy reconstruction with charged-current interactions in next-generation detectors.  Phys. Rev. D, 101:092003, 2020.
  * [7] S. Dolan et al.  Sensitivity of the upgraded T2K Near Detector to constrain neutrino and antineutrino interactions with no mesons in the final state by exploiting nucleon-lepton correlations.  Phys. Rev. D, 105:032010, 2022.
  * [8] Michael S. Moore.  WNR, a Neutron Time-of-Flight Facility at the Los Alamos Scientific Laboratory, pages 327–333.  Springer US, Boston, MA, 1974.
  * [9] Suzanne F. Nowicki, Stephen A. Wender, and Michael Mocko.  The los alamos neutron science center spallation neutron sources.  Physics Procedia, 90:374–380, 2017.  Conference on the Application of Accelerators in Research and Industry, CAARI 2016, 30 October – 4 November 2016, Ft. Worth, TX, USA.
  * [10] A. Blondel et al.  A fully active fine grained detector with three readout views.  JINST, 13(02):P02006, 2018.
  * [11] A. Blondel et al.  The SuperFGD Prototype Charged Particle Beam Tests.  JINST, 15(12):P12003, 2020.
  * [12] M. Antonova et al.  Baby MIND: A magnetized segmented neutrino detector for the WAGASCI experiment.  JINST 12(2017)07.C07028.
  * [13] Georgi Mitev Mitev, Ludmil Todorov Tsankov, Mityo Georgiev Mitev, and Etam Noah Messomo.  Light pulse generator for multi-element scintillation detectors testing.  In 2016 XXV International Scientific Conference Electronics (ET), pages 1–4, 2016.
  * [14] B. Bhandari et al.  First Measurement of the Total Neutron Cross Section on Argon Between 100 and 800 MeV.  Phys. Rev. Lett., 123(4):042502, 2019.
  * [15] D.H. Wright and M.H. Kelsey.  The geant4 bertini cascade.  Nuclear Instruments and Methods in Physics Research Section A: Accelerators, Spectrometers, Detectors and Associated Equipment, 804:175–188, 2015.
  * [16] A. Boudard, J. Cugnon, S. Leray, and C. Volant.  Intranuclear cascade model for a comprehensive description of spallation reaction data.  Phys. Rev. C, 66:044615, Oct 2002.
  * [17] Provided by beamline experts of Los Alamos National Laboratory in private communications.
  * [18] Martin Ester, Hans-Peter Kriegel, Jörg Sander, and Xiaowei Xu.  A density-based algorithm for discovering clusters in large spatial databases with noise.  pages 226–231. AAAI Press, 1996.
  * [19] W. Schimmerling, T. J. Devlin, W. W. Johnson, K. G. Vosburgh, and R. E. Mischke.  Neutron-nucleus total and inelastic cross-sections - 900 to 2600 mev/c.  Phys. Rev. C, 7:248–262, 1973.
  * [20] P J Dimbylow.  Neutron cross-sections and kerma values for carbon, nitrogen and oxygen from 20 to 50 MeV.  Physics in Medicine and Biology, 25(4):637–649, jul 1980.
  * [21] R. W. Finlay, W. P. Abfalterer, G. Fink, E. Montei, T. Adami, P. W. Lisowski, G. L. Morgan, and R. C. Haight.  Neutron total cross-sections at intermediate-energies.  Phys. Rev. C, 47:237–247, 1993.
  * [22] W. P. Abfalterer, F. B. Bateman, F. S. Dietrich, R. W. Finlay, R. C. Haight, and G. L. Morgan.  Measurement of neutron total cross-sections up to 560-MeV.  Phys. Rev. C, 63:044608, 2001.