(eccv) Package eccv Warning: Package ‘hyperref’ is loaded with option
‘pagebackref’, which is *not* recommended for camera-ready version

11institutetext: Texas A&M University, Texas, USA. 11email: {lyq, wenping,
<EMAIL_ADDRESS>22institutetext: The University of British Columbia,
Vancouver, Canada. 22email<EMAIL_ADDRESS>33institutetext: Qingdao
University, Shandong, China. 33email<EMAIL_ADDRESS>
# Skull-to-Face: Anatomy-Guided 3D Facial Reconstruction and Editing

Yongqing Liang 11 Congyi Zhang 22 Junli Zhao 33 Wenping Wang 11 Xin Li∗ 11

###### Abstract

Deducing the 3D face from a skull is an essential but challenging task in
forensic science and archaeology. Existing methods for automated facial
reconstruction yield inaccurate results, suffering from the non-determinative
nature of the problem that a skull with a sparse set of tissue depth cannot
fully determine the skinned face. Additionally, their texture-less results
require further post-processing stages to achieve a photo-realistic
appearance. This paper proposes an end-to-end 3D face reconstruction and
exploration tool, providing textured 3D faces for reference. With the help of
state-of-the-art text-to-image diffusion models and image-based facial
reconstruction techniques, we generate an initial reference 3D face, whose
biological profile aligns with the given skull. We then adapt these initial
faces to meet the statistical expectations of extruded anatomical landmarks on
the skull through an optimization process. The joint statistical distribution
of tissue depths is learned on a small set of anatomical landmarks on the
skull. To support further adjustment, we propose an efficient face adaptation
tool to assist users in tuning tissue depths, either globally or at local
regions, while observing plausible visual feedback. Experiments conducted on a
real skull-face dataset demonstrated the effectiveness of our proposed
pipeline in terms of reconstruction accuracy, diversity, and stability. Our
project page is https://xmlyqing00.github.io/skull-to-face-page/.

###### Keywords:

Skull-to-Face Reconstruction Human Face Tissue Depth Modeling Human Face
Adaptation

##  1 Introduction

3D facial reconstruction from a skull is an effective approach used by
forensic investigators to identify skeletal remains when other information is
unavailable. It has been successfully applied in forensic cases [52] and
anthropological studies [1]. Given a 3D skull scan, the objective is to derive
a 3D face that aligns with both the geometric features and the biological
profile of the skull. The anatomic setting of the reconstruction process is
illustrated in Fig. 1: A set of predefined _skull landmarks_ [11] is
distributed on the query _skull_. The _tissue depths_ (blue sticks) on
landmarks are estimated using anatomical statistic data. The endpoints of
these tissue depths, _i.e_., the _facial landmarks_ (green spheres), depict a
coarse geometry shape of the 3D face. A plausible 3D face geometry should be
created, satisfying the constraints of these facial landmarks.

Figure 1: Given an input skull scan, based on skull landmarks and tissue depth
(blue sticks), facial landmarks (green spheres) can be obtained. The 3D face
is inferred under the constraints of facial landmarks.

Traditional facial reconstruction in forensic applications is performed
manually using plasticine or clay [53, 25]. Recent approaches [5, 35, 55]
employ a digital environment to replace these manual clay modeling methods.
Both physical and digital methods for facial reconstruction require years of
anatomy and artistic modeling experience, and the results are often unreliable
due to the subjective factors [18, 56]. Besides the geometry, a vivid facial
texture may be inferred from (_e.g_., ancestry, age of) the skull. Such
realistic facial textures can be desirable in anthropological and forensic
applications, but their creation often requires further post-processing steps
and significant extra effort.

In the past two decades, researchers have been studying computer-aided skull-
to-face reconstruction algorithms to produce 3D faces. They access the problem
from various perspectives and multidisciplinary backgrounds, from data-driven
[10, 39, 8, 12, 16, 56] to physical simulation [24, 26, 36, 11], providing
solid research baselines. Nevertheless, (1) all these face generation methods
overlooked multi-modal characteristics beyond the geometry of the skull, such
as gender, age, and ancestry. (2) Without modeling the distribution of tissue
depth, these methods addressed the problem by solving a deterministic mapping
from one skull to one face with statistically average tissue depths [43, 44,
48, 49], making it hard for further tuning in real scenarios.

Our proposed work builds upon two main observations. (1) Firstly, the
biological profile, encompassing ancestry, age, gender, and stature of the
body, can be estimated from skulls [47]. Different population groups exhibit
distinct facial priors, and we argue that incorporating these is crucial for
accurate facial reconstruction. We generate an initial guess of the 3D face
with these attributes, whereas existing methods often overlook this key
information. (2) Secondly, the combinations of tissue depths in real human
faces follow intrinsic distributions that have not been well-studied. We
introduce an additional control based on the distribution of tissue depth over
facial reconstruction, providing an intuitive and effective tool for 3D face
manipulation and exploration.

Our approach includes an offline tissue depth distribution training stage and
two online inference stages: semantic face model initialization, and anatomy-
guided face adaptation. We also develop landmark-based face editing as an
optional adjustment stage. (1) During the preprocessing stage, tissue depth
distribution can be analyzed using available datasets of scanned skull and
face pairs, or obtained from statistical data in forensic anthropology
literature [43, 44, 48, 49]. (2) When processing a query skull, the key
biological profile of the skull is used to generate a 2D realistic facial
image, which is then reconstructed into a textured 3D face. Then, facial
landmarks are generated, guided by the pre-trained tissue depth distribution
and the geometry of the query skull. Finally, the face geometry is adjusted to
align with these facial landmarks, resulting in the final skull-conformed
face. (3) As an optional step, we provide a tool that allows users to
intuitively adjust the face geometry by tuning a one-dimensional feature in
the distribution space of tissue depths.

The main contributions of this work are as follows: (1) We present a first
approach that incorporates the biological profile into skull-to-face
reconstruction. (2) By analyzing the distribution of tissue depth, we devise
an additional control mechanism over the automatic reconstruction workflow,
offering an intuitive and effective way to tune the face geometry. (3)
Compared to existing methods, our method achieves state-of-the-art geometric
accuracy qualitatively and quantitatively. We will release the codes and pre-
trained models, making it the first publicly accessible approach in this
field.

##  2 Related Work

###  2.1 Facial Reconstruction from Skull

##### Data-driven facial reconstruction.

Statistics-based methods for facial reconstruction employ high-dimensional
vectors, typically representing faces through their 3D coordinates or feature
points. These methods, using techniques such as PCA, learn the mapping between
corresponding skulls and faces from training datasets [10, 39, 8, 12, 57, 33,
16, 22]. A new face is then generated as a linear combination of principal
components. Other approaches [33, 14, 6, 39, 50, 30, 40] involve regression
algorithms to predict faces from skulls. Zhang _et al_. [56] project 3D skulls
to 2D images and leverage image-to-image style transferring networks [20, 58]
for facial reconstruction. However, these data-driven methods require a
significant number of joint skull-face scans, which are costly and difficult
to acquire. The limited available datasets often result in overfitted networks
and dataset bias.

##### Deformation-based facial reconstruction.

Another category of reconstruction algorithms employs 3D modeling techniques
to digitally mimic the manual reconstruction process. A prevalent strategy
involves deforming a template face surface [11, 36] or tissue volume [51, 26,
24] to fit with the subject’s skull. This deformation is controlled by
imposing penalties on certain geometric smoothness energy to regularize the
transformation while adhering to the positional constraints of facial
landmarks.

###  2.2 Digital Face Synthesis

##### Constrained face editing.

Constrained face editing or re-synthesis techniques involve generating new
faces that are both realistic and adhere to specific constraints. Many recent
2D-based methods [2, 27, 38, 13, 46, 45, 19] accomplish this by manipulating
semantic codes in the latent space to generate new 2D face images. However,
these 2D methods lack underlying 3D face geometry, making it difficult to
enforce fine geometry constraints during face editing.

##### Large face generation models.

Recent years have witnessed significant advancements in large face generation
models, including text-to-image generation models (_e.g_. Stable Diffusion XL
[4] and DALL-E2 [41]), and 2D image to 3D face generation models (_e.g_.,
GaGAN [28], CoMA [42], DECA [15], EG3D [9], and PanoHead [3]). A recent study,
Rodin [54], proposed a text-to-3D-face pipeline, but the synthesized faces are
somewhat cartoon-like and unrealistic. NeRF-based methods such as EG3D and
PanoHead are less effective in editing 3D geometry (to meet specific geometric
constraints).

##  3 Method

Figure 2: An overview of the proposed Skull-to-Face pipeline, which includes
three modules. The Face Generation module synthesizes an initial 3D face in
consideration of the biological profile of the skull such as age, gender, and
ancestry. The Tissue Depth Distribution (TDD) can suggest a valid combination
of facial landmarks as geometry constraints. The Anatomy-guided Face
adaptation module optimizes the initial face as the facial reconstruction
according to the landmark constraints.

###  3.1 Pipeline Overview

Our skull-to-face generation pipeline has three components, as is shown in
Fig.2. In the _Tissue Depth Distribution_ (TDD) modeling, we analyze the
tissue depth distribution in the training set and use the distribution to
suggest a valid combination of tissue depths for the skull (Sec. 3.2). Then,
when reconstructing a face from a query skull, we have two stages. In the
_Initial Face Generation_ stage, we utilize key biological profiles about the
human body, _i.e_., gender, age, and ancestry to generate a 2D candidate
portrait using Stable Diffusion XL [4] and lift it to a 3D face using the
FLAME model [15] (Sec. 3.3). Finally, with the facial landmarks computed from
the tissue depth model as the geometry constraints, we propose an _Anatomy-
guided Face Adaptation_ method to modify the 3D face according to facial
landmark constraints. The details of the reconstructed face are preserved from
the 2D candidate portrait image (Sec. 3.4).

###  3.2 Distribution of Tissue Depth

Tissue depths define the displacement between facial landmarks and their
corresponding skull landmarks. The combination of tissue depths controls the
shape of the reconstructed face, which is critical for the fidelity of facial
reconstruction and directly influences the reconstruction error. Instead of
directly learning the mapping from skull to face as other works [22, 56] did,
we analyzed and explored the intrinsic pattern of the tissue depth
distribution. We then used the distribution pattern to guide facial
reconstruction. This also provides the benefits for users to further adjust
tissue depths to explore various plausible faces on skulls.

On each skull $J$, a set of landmarks
$\mathcal{P}^{s}=\\{\mathbf{p}^{s}_{i}\in\mathbb{R}^{2},i=1,\ldots,n\\}$ are
defined and can be extracted. In this work, we follow [11] to extract $n=78$
landmarks. Each such skull landmark $\mathbf{p}^{s}_{i}$ is extended outward
along the normal directions $\hat{\mathbf{n}}^{s}_{i}$ with a tissue depth
length $d_{i}$. The extended landmarks are referred to as facial landmarks
$\mathcal{P}^{f}=\\{\mathbf{p}^{f}_{i}|\mathbf{p}^{f}_{i}=\mathbf{p}^{s}_{i}+d_{i}\hat{\mathbf{n}}^{s}_{i}\\}$,
which provide a sparse constraint of the face shape. $\mathcal{D}=\\{d_{i}\\}$
represents the tissue depths of a person, and
$\mathcal{X}=\\{\mathcal{D}_{m}\\}\in\mathbb{R}^{n}$ represents the collection
of tissue depths of $m$ people in the training dataset.

(a) Three representative values of the first principal component are
visualized overlaid on the tissue depth distribution.

(b) Adjusting the values of the first principal component provides easy
control of tissue depths, and the face shapes change from thin to fat.

Figure 3: The first principal component can meaningfully provide control on
the thickness of tissue depths. (a) shows the tissue depth distribution $C$ in
the first principal component. Vertical lines represent three representative
values, signifying three different combinations of tissue depths, resulting in
three types of face shapes: thin, normal, and fat, as illustrated in (b).

####  3.2.1 TDD-Global Model

We analyze the tissue depth distribution using the training set of the skull-
face dataset, which comprises $m$ samples, each with $n=78$ landmarks. It is
important to note that the number of training samples $m$ in the publicly
accessible skull-face dataset is limited ($m\leq 100$ in our experiments).
Consequently, employing a categorized or neural-based model to fit the
landmark distribution may be inappropriate. We therefore designed a simple yet
effective way to model and control the tissue depths.

We first used principle component analysis (PCA) to project the joint
distribution of tissue depths from an $n-$dimension space to a low-dimensional
space. We observed that the first principal component predominantly governs
the tissue depth distribution, accounting for $51.9\%$ of the variance. This
finding significantly reduces the degree of freedom for the combination of 78
tissue depths. We projected joint tissue depths to the first component axis,
denoted as $C$. The distribution of $C$ is visualized in Fig. 3(a).
Furthermore, the effect of adjusting the values of the first principal
component is illustrated In Fig. 3(b), where the directions of the tissues are
depicted as blue sticks and the tissue depths in the training set are green
spheres. A more comprehensive discussion of PCA components is included in the
_Supplementary_.

When we monotonously sample values from the distribution $C$, the tissue
depths change from short to long, and the corresponding face shape changes
from thinner to rounder. To better visualize the relationship between face
shape and the distribution $C$, we select mean values from the top, middle,
and bottom $33\%$ of $C$ as representative combinations of tissue depths, and
visualize them in Fig. 3(b). Note that while some related morphable face
models also provide correlations between PCA components and face fullness [7],
our model is the first model explicitly guided by anatomy, connecting facial
skin to the underlying skull.

The above model enables us to modify face shapes based on the joint
distributions of all the tissue depths on a skull. And we denote it as the
TDD-Global model. Specifically, (a) we can use $C=0$ to sample the average
tissue depths when we would like to generate average faces with moderate
facial thickness. (b) we also can vary values in the first principal component
to sample multiple plausible tissue depth combinations, thereby exploring
different face shapes. The effectiveness of TDD-Global is validated in face
interpolation and face shape modification experiments (see Sec. 4.3.1).

####  3.2.2 TDD-Regional Model

We integrate the above TDD-Global in our pipeline for automatic facial
reconstruction. In addition, experts and investigators often require a manual
tool to support local fine-tuning. The TDD-Global model alone does not allow
for precise modification of specific facial landmarks without impacting other
regions. On the other hand, modifying single tissue depth alone may result in
a non-realistic face shape, because the tissue depths in a local region also
adhere to a joint distribution. This feature is essential for detailed face
editing. By enabling localized modification, the editing capability can be
significantly enhanced.

To enable local face shape adjustments, we modify the TDD-Global model to
account for the joint distribution of tissue depths in local regions. We
follow a common facial region partitioning [34, 37] (forehead, cheeks, chin,
middle region, and mouth) to divide facial landmarks into distinct groups
$\mathcal{Y}_{l}\subset\mathcal{X}$, $k=1,\cdots,5$, as is visualized in Fig.
7(b). Then, by employing PCA to model the tissue depth distribution in each
group individually, tissue depths can be adjusted locally within each
subgroup.

According to the facial anatomy study [34, 37], facial musculature and
neurovascular structures are located throughout faces and can be divided into
several regions, each responding to different movements and expressions. Our
TDD-Regional model can work effectively because it supports fine-tuning local
face regions by adjusting regional tissue depths in accordance with relevant
anatomical structures. We verified our TDD-Regional model by extensive
experiments, as shown in Sec. 4.3.2 (_e.g_., Fig. 7) and the _Supplementary_
video/document.

###  3.3 Initial Face Generation

When reconstructing a 3D face from a skull, in addition to geometric
constraints, the biological profile derived from the skull can also aid in the
reconstruction process. The biological attributes often include age, gender,
and ancestry, which determine the basic characteristics of the generated face.
In this first stage, we generate initial faces following the characteristics
of the population group specified by these attributes.

We adopt Stable Diffusion XL [4] to generate high-resolution 2D portrait
images from text prompts. We design four main dimensions of facial attributes:
gender, ancestry, age, and face shape to generate various faces covering
different populations. We also add additional prompts to ensure the diffusion
model generates appropriate front faces.

We lift the synthesized portrait images to 3D faces by using DECA [15]. It
encodes the image to latent space, consisting of shape, expression, pose, and
albedo codes, then decodes the latent code to a 3D face with the FLAME
representation. We transfer the texture and geometry details from the image to
the 3D mesh. The generated faces conform to the biological profile and will be
adapted to the final face by the geometric constraints of facial landmarks.

###  3.4 Anatomy-guided Face Adaptation

Given an initial 3D face model $F_{in}$, and a set of _facial landmark
constraints_ , $\mathcal{P}^{f}=\\{\mathbf{p}_{i}^{f}\\}$ recommended by our
TDD model, we design an anatomy-guided face adaptation model to modify the
initial face, ensuring the revised face $F_{out}$ aligns with all facial
landmarks. Since our face models $\\{F\\}$ are generated by DECA, they have
corresponding latent codes $\\{f\\}$, and can be decoded into FLAME models
$\\{G_{F}\\}$; thus, all generated faces share consistent mesh connectivity,
$F=G_{F}(f)$. We manually labeled the corresponding $n$ vertex indices on one
face as _facial landmarks_ , denoted as
$\mathcal{Q}^{f}=\\{\mathbf{q}^{f}_{i},i=1,\cdots,n\\}$. These vertex indices
remain consistent across all the generated faces.

Firstly, we estimate a transformation $H$ to make a coarse registration
between $\mathcal{Q}^{f}_{in}$ of initial face $F_{in}$ and $\mathcal{P}^{f}$.
We solve a Procrustes optimization to estimate the scaling, rotation, and
translation parameters of $H$. With this linear transformation, we transform
$\mathcal{P}^{f}$ to $\mathcal{\tilde{P}}^{f}$ as the roughly aligned landmark
constraints. Secondly, we optimize the latent code $f$ by minimizing the
objective function $\mathcal{L}$,

$\displaystyle f_{out}$
$\displaystyle=\arg\underset{f}{\min}\leavevmode\nobreak\ \leavevmode\nobreak\
\mathcal{L}(f),$ (1) $\displaystyle\mathcal{L}$
$\displaystyle=\alpha_{0}\mathcal{L}_{lmk}+\alpha_{1}\mathcal{L}_{proj}+\alpha_{2}\mathcal{L}_{sym}.$
(2)

We design three losses in the objective function and will detail each of them
below.

##### Landmark loss.

The deviation between the aligned landmark constraints
$\mathcal{\tilde{P}}^{f}$ and the landmarks from the generated face should be
minimized:

$\mathcal{L}_{lmk}=\|\mathcal{Q}^{f}-\mathcal{\tilde{P}}^{f}\|_{2}^{2}.$ (3)

##### Projection loss.

Since the corresponding landmarks on face $\mathcal{Q}^{f}$ are manually
labeled, they are inevitably mismatched to the target facial landmarks
$\mathcal{\tilde{P}}^{f}$. In this sense, $\mathcal{L}_{lmk}$ can be regarded
as a regularization term. To improve the registration accuracy from the face
geometry to the target landmarks, we project the target facial landmark
constraints $\mathcal{\tilde{P}}^{f}$ to the reconstructed face as
$\mathcal{\tilde{P}}^{pf}$. Then we minimize the Euclidean distance between
$\mathcal{\tilde{P}}^{f}$ and $\mathcal{\tilde{P}}^{pf}$ as a data term to
constrain the face shape,

$\mathcal{L}_{proj}=\sum_{p\in\mathcal{\tilde{P}}^{pf}}\underset{v\in
F}{\min}\|v-p\|_{2}^{2},$ (4)

where $v\in F$ is vertex of the reconstructed face $F$.

##### Symmetry loss.

Facial symmetry plays an important role in authentic face synthesis that
people prefer symmetry faces [17, 23]. Hence, we propose a symmetry loss term
to encourage the symmetry of the synthesized face. We first split the facial
landmarks $\mathcal{Q}^{f}$ into three groups,
$\\{\mathcal{Q}^{f}_{L},\mathcal{Q}^{f}_{M},\mathcal{Q}^{f}_{R}\\}$,
representing on the left, mid, right sides. The mid-plane $\Pi$ is
perpendicular to the line through the eyes, which crosses the tip of the nose
and the chin, which is computed by minimizing the least square error of
$\mathcal{Q}^{f}_{M}$.

For any landmark on the left side $\mathbf{q}^{f}_{i}\in\mathcal{Q}^{f}_{L}$,
we denote $\mathbf{q}^{pf}_{i}\in\mathcal{Q}^{f}_{R}$ as the corresponding
landmark on the right side. We also denote the reflected point across the
plane $\Pi$ as $\mathbf{q}^{rf}_{i}$. The symmetry loss function is defined as
the Euclidean distance between the actual midpoint $\mathbf{q}^{M}$ and the
expected midpoint $\mathbf{q}^{M^{\prime}}$,

$\displaystyle\mathbf{q}^{M}=\big{(}\mathbf{q}^{f}_{i}+$
$\displaystyle\mathbf{q}^{pf}_{i}\big{)}/2,\leavevmode\nobreak\
\leavevmode\nobreak\ \leavevmode\nobreak\
\mathbf{q}^{M^{\prime}}=(\mathbf{q}^{f}_{i}+\mathbf{q}^{rf}_{i})/2,$ (5)
$\displaystyle\mathcal{L}_{sym}$
$\displaystyle=\sum_{\mathbf{q}^{f}_{i}\in\mathcal{Q}^{f}_{L}}\|\mathbf{q}^{M}-\mathbf{q}^{M^{\prime}}\|_{2}^{2}.$
(6)

When the face latent code $f$ is optimized, it outputs the geometry of the 3D
face $F_{out}$ that conforms to the landmarks and skull constraints. We
transfer the appearance texture and geometry details previously extracted from
the 2D portrait to $F_{out}$ as our facial reconstruction.

Figure 4: Examples of the synthesized faces by different attributes.

##  4 Experiment

###  4.1 Dataset and Error Metric

We adopted a CT-scan dataset, named _Skull100_ , comprising 100 pairs of 3D
skulls and faces. These pairs were reconstructed (using [31]) from whole-head
CT scans. Due to the privacy issue, this dataset doesn’t collect face images
or appearance data. Details of dataset collection are explained in the
_supplementary_. Skull100 is used as the ground truth to evaluate faces
reconstructed from their corresponding skulls. We divide the dataset into two
subsets: 50 pairs for training and 50 for testing. This large test set size is
chosen to ensure the statistical significance of our evaluation results. We
conducted the K-Fold cross-validation to evaluate the distribution bias (Sec.
4.4.2).

We adopted the normalized mean error metric $E$ [29] to evaluate the geometric
deviation of the reconstructed face $F$ and the ground truth $G$, where $F$
and $G$ are represented as triangular meshes. For each vertex $v$ in the
ground truth face $G$, we find the closest vertex $u$ from the reconstructed
face $F$ and sum the Euclidean distance as the reconstruction error,

$E=\frac{1}{d|G|}\sum_{v\in G}\leavevmode\nobreak\ \underset{u\in
F}{\min}\leavevmode\nobreak\ \|v-u\|_{2},$ (7)

where $d$ is the distance between the two ears of the ground truth face, and
$|G|$ is the total number of vertices in the ground truth face. Metrics _Mean_
, _Max_ , and _Std_ are the average, maximum, and standard deviation of
reconstruction errors $E$ of all test skulls.

###  4.2 Initial Face Generation

Since facial reconstruction should align with the biological profile of the
skull, we design the attributes in the text prompts and leverage the Stable
Diffusion XL to generate 2D portraits. We show the examples of the generated
2D face in Fig. 4. The complete text prompt and portrait attributes are
reported in _Supplementary_. We will release this synthesized face dataset
with attributes.

###  4.3 Evaluation of Tissue Depth Distribution

####  4.3.1 TDD-Global Model

The TDD-Global model allows users to use only one parameter $C$ to sample
various valid combinations of tissue depths. The distribution and parameter
choice of PCA decomposition are analyzed in the Supp.

In Fig. 5, we enumerated the TDD-Global model to sample different combinations
of tissue depths from short to long. The 3D faces fit these tissue depths well
and the face shape becomes fatter from left to right. Existing facial
reconstruction methods [22, 56] only forcibly memorize the distribution of
skull and face. Instead, our TDD-Global models and discovers the intrinsic
pattern of tissue depths and human face shapes. It allows researchers to
efficiently try different tissue depths and examine a variety of facial
reconstruction possibilities.

Figure 5: TDD-Global model can explore faces with various shapes. From left to
right, we iteratively sampled values of distribution $C$, and the
corresponding tissue depths increased from short to long. The adapted faces
that fit the tissue depth become fatter. Figure 6: Validation of the face
shape modification by TDD-Global model celebrities Christian Bale, Adele
Adkins, and Ling Jia. We started from a 2D image (a) and used DECA to create
its initial 3D face (b). We estimated proxy skull landmarks and adjusted the
tissue depths upon them to tune the face shape to multiple roundness level
(c). Face pose in (d) was intentionally adjusted for better visual comparisons
with (e). (e) show the picture of the same person in a different face shape.

In the second experiment, we modified the face shapes of celebrities to show
the ability of 3D face adjustment by the tissue depth model. Fig. 6 shows the
results of three celebrities: Christian Bale, Adele Adkins, and Ling Jia. The
initial 3D face of a celebrity was generated by the DECA model. We then used
tissue depths of the pretrained TDD-Global model to estimate proxy skull
landmarks (Sec. 3.2). We adjust $C$ to sample various valid combinations of
tissue depths (_e.g_. from thin to fat) and then compute the target facial
landmarks. Finally, the proposed anatomy-guided face adaptation is applied to
adapt the initial 3D face to the new facial landmarks. The modified face is
compared with the picture of the same person in a different face shape.

Figure 7:  Adjusting local face shapes using the TDD-regional model. Tissue
depths increase on the nose (from left to right) and cheek (from top to
bottom), separately. These modified tissue depths fine-tune facial landmarks,
thereby adjusting the face shapes.

####  4.3.2 TDD-Regional Model

We decoupled facial landmarks into distinct regions and applied separate PCA
models to model the tissue depth distribution in each region, resulting in the
_TDD-Regional_ Model. This model offers explicit control for manipulating face
shapes locally. As illustrated in Fig. 7, variations in the landmarks of the
nose and cheek regions demonstrate the model’s capability. The nose size
visibly increases from left to right, while the face shape appears fuller from
top to bottom. We refer to the _supplementary video_ for the animation of face
fine-tuning.

Figure 8: Validating the influence of initial faces on the final results. (a)
are the query skulls and facial landmarks. (b) is the synthesized portrait
image. (c) are the initial 3D faces with texture and the reconstruction
errors. (d) are our facial reconstructions, which and fit the landmark
constraints well. The final reconstruction errors are similar: 0.899% (top)
and 0.925% (bottom). They are both close to the GT face (e).  Figure 9:
Qualitative comparisons with existing methods. (a) is the input skull. (b-e)
show the results of HF-GGR [22], Pix2Pix [20], CycleGAN [58], and CGAN-CFG
[56]. (f) show our reconstructed face with texture and geometry. (g) is the
CT-scanned GT.  Figure 10: Facial reconstruction demo on historical figure:
Robert the Bruce, was King of Scots from 1306 to his death in 1329. (a) is the
3D scan of the skull. (b) show the facial reconstruction by craniofacial
experts Wilkinson  _et al_. [55]. (c) is our synthesized portrait. We manually
labeled the skull landmarks and used the pretrained TDD-Global to estimate the
facial landmarks. (d) show the 3D reconstructed face that fits the facial
landmarks. We adjusted the head pose for better visual comparison with
reference (e).

###  4.4 Evaluation of Skull-to-Face Reconstruction

####  4.4.1 Influence of Initial Faces on Final Results

We chose the average tissue depths to compute the extended landmarks as facial
constraints. Fig. 8 shows that with face refinement, our face adaptation model
can effectively modify the face shape to align with the facial landmarks. With
facial landmark constraints, the two initial faces, after adaptation, converge
to a similar face geometry. Their geometries still exhibit subtle differences
inherited from the initial faces (see error maps in (d)).

We conducted a quantitative experiment where we randomly synthesized four
portraits for each skull and measured the final reconstruction errors. The
reconstruction errors $E$ are similar for different initials: 1.756, 1.673,
1.724, and 1.755. For each skull, the overall face geometries amongst the
final faces (from different initial faces) are similar. Subtle differences are
noticed which are inherited from the initial faces (_e.g_. dimples, wrinkles,
and lip thick).

We also conducted a user study to assess the convergence quality of the final
facial reconstruction results from different initial faces. For each question,
four initial faces are randomly synthesized. Three of them are used to
reconstruct faces from Skull A, and the other one is used to reconstruct a
face from Skull B. Users are required to identify the face geometries amongst
the reconstructed faces and select the unique face that is reconstructed from
Skull B. The detailed explanation and results of the user study are included
in the _Supplementary_. The high average correctness of this user study
indicates that our model can adapt different initial faces and converge to a
face shape space that is related to the query skull, which is distinct from
the reconstructed faces that are from the other skulls.

Table 1: Quantitative comparisons of facial reconstructions with existing
methods. The distance between two ears of the groundtruth skin is normalized
to 1.

Method | Mean (%)$\downarrow$ | Max (%)$\downarrow$ | Std. (%)$\downarrow$
---|---|---|---
HF-GGR [22] | 2.198 | 4.755 | 0.841
Pix2Pix [21] | 2.651 | 4.859 | 0.928
CycleGAN [59] | 2.954 | 4.833 | 0.859
CGAN-CFG [56] | 4.002 | 7.402 | 1.192
Ours (avg. tissue) | 1.732 | 1.775 | 0.039
Ours (best tissue) | 1.526 | 1.571 | 0.035

Table 2: Reconstruction errors before vs after adaptation and loss functions.

Ablation Study | Mean Recon. | Mean Recon.
---|---|---
(%) | (millimeter)
1\. Before adaptation | 2.631 | 5.262
(from DECA)
2\. w/ $\mathcal{L}_{proj}$ | 2.273 | 4.547
3\. w/ $\mathcal{L}_{lmk}$ | 1.836 | 3.672
4\. w/ $\mathcal{L}_{lmk}+\mathcal{L}_{proj}$ | 1.758 | 3.576
5\. After adaptation | 1.732 | 3.464
($\mathcal{L}_{lmk}+\mathcal{L}_{proj}+\mathcal{L}_{sym}$)

####  4.4.2 Comparisons with SOTA Facial Reconstruction Methods

Quantitative side-by-side comparisons are challenging due to the high cost and
privacy concerns associated with joint skull-face CT scans, compounded by the
absence of public benchmark data. Additionally, the source code for existing
methods is not publicly accessible. Despite these limitations, we endeavored
to conduct a fair comparison. CGAN-CFG [56], a state-of-the-art DNN-based
facial reconstruction method, was tested using our training and testing
dataset, with reconstruction results provided by the original authors. We also
re-implemented CycleGAN [58] and Pix2Pix [20] following the guidelines in
[56]. We chose HF-GGR [22] as the representative traditional PCA method and
also re-implemented their model.

Tab. 4.4.1 shows the quantitative comparisons of facial reconstructions. Our
method used synthesized portraits as initial faces. _Ours (avg. tissue)_ used
the average tissue depths from TDD-Global to compute the facial landmarks and
then adapted the initial faces to the final results. _Ours (best tissue)_
utilized three types of tissue depths thin, normal, and fat from TDD-Global as
shown in Fig. 3. We selected the smallest reconstruction result as the final
face. Our method outperforms existing methods in all metrics, largely due to
the effective adaptation of our loss function to geometric constraints. The CT
scan resolution is 0.25mm/pixel. The average of the reconstruction error is
3.464 millimeters.

Fig. 9 show qualitative comparisons. CGAN-CFG [56] tends to overfit the
training set, leading to worse results on the test set. Similarly, Pix2Pix
[20] and CycleGAN [58] often produce unrecognizable faces as they rely on the
availability of large training datasets. HF-GGR [22] is a PCA-based model and
typically generates average faces. More results are in the _Supplementary_.

To evaluate the distribution bias of training/test set, we used 5-fold cross-
validation to split the dataset into training (80%) and testing (20%). The
average reconstruction errors on the five testing sets are: 1.761%, 1.454%,
1.828%, 1.575%, and 1.665%. The results are consistent with Table 4.4.1.

####  4.4.3 Validation on Real-world Facial Reconstruction

We downloaded the 3D-scanned skull from the Anatomical Museums at the
University of Edinburgh and used the proposed pipeline to reconstruct the
faces for it. We used the text prompt to generate the initial face according
to the historical background. Fig. 10 shows the comparisons between our
results and the bust of Robert the Bruce at the National Wallace Monument.
Wilkinson  _et al_. [55] manually sculpted and painted the facial
reconstruction. Although manual facial reconstruction could be subjective to
different researchers [55], the reconstruction of Wilkinson and ours have
similar chin, jawline, eyes, and face proportions, which are aligned to the
latest depiction [55].

###  4.5 Ablation Study on Face Adaptation

We conducted an ablation study to evaluate the impact of various loss
functions via three settings: (1) using only $\mathcal{L}_{lmk}$, (2) using
$\mathcal{L}_{lmk}$ \+ $\mathcal{L}_{proj}$, and (3) our complete loss. Tab. 2
reports the average reconstruction errors. We computed the errors in the
physical metric (millimeter) by the average face width (20cm). The projection
loss $\mathcal{L}_{proj}$ aids in reducing the distance between the
reconstructed face and landmark constraints. The symmetric loss
$\mathcal{L}_{sym}$ enhances the face realism by promoting symmetry. The
second and third rows show the face adaptation results of the same skull,
using two different initial faces.

##  5 Conclusions

We proposed a novel anatomy-guided 3D face reconstruction approach that can
reconstruct faces from a given skull using both biological profiles of the
skull and the statistical tissue depth information defined on physiological
landmarks. Our pipeline includes three main designs, the biological profile-
based 3D face initialization, the facial landmark prediction based on a pre-
computed tissue depth distribution, and the facial geometry adaptation to the
landmarks. These designs enable the effective generation of multiple realistic
faces with varying facial features, skin textures, and adiposity levels. Our
results demonstrate that the proposed pipeline can effectively generate faces
that closely conform to the given skull. This work provides a new tool for law
enforcement agencies to generate and explore possible faces from skeletal
remains, which greatly facilitates forensic identification when other
identifying information is unavailable.

##### Limitations.

A limitation of our current approach is the constrained complexity of the
tissue depth model due to the limited size of the available skull-face
dataset. The high cost and prohibitive nature of collecting a large dataset
have restricted our ability to train more sophisticated models. We will
explore potential synthetic data generation techniques to enhance the dataset
size and diversity, thereby enabling the exploration of more complex models
for facial reconstruction.

## References

  * [1] Aidonis, A., Achillas, C., Tzetzis, D., Athanassiou, A., Karkazi, E., Darlas, A., Papageorgopoulou, C.: Digital 3d facial approximation of the petralona skull. methodological issues and applications. Journal of Archaeological Science: Reports 51, 104206 (2023)
  * [2] Alaluf, Y., Patashnik, O., Cohen-Or, D.: Only a matter of style: Age transformation using a style-based regression model. ACM Trans. Graph. 40(4) (2021), https://doi.org/10.1145/3450626.3459805
  * [3] An, S., Xu, H., Shi, Y., Song, G., Ogras, U.Y., Luo, L.: Panohead: Geometry-aware 3d full-head synthesis in 360deg. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). pp. 20950–20959 (June 2023)
  * [4] Anonymous: SDXL: Improving latent diffusion models for high-resolution image synthesis. In: Submitted to The Twelfth International Conference on Learning Representations (2023), https://openreview.net/forum?id=di52zR8xgf, under review
  * [5] Baldasso, R.P., Moraes, C., Gallardo, E., Stumvoll, M.B., Crespo, K.C., Strapasson, R.A.P., de Oliveira, R.N.: 3d forensic facial approximation: Implementation protocol in a forensic activity. Journal of Forensic Sciences 66(1), 383–388 (2021)
  * [6] Berar, M., Tilotta, F.M., Glaunes, J.A., Rozenholc, Y.: Craniofacial reconstruction as a prediction problem using a latent root regression model. Forensic science international 210(1-3), 228–236 (2011)
  * [7] Blanz, V., Vetter, T.: A morphable model for the synthesis of 3d faces. In: Proceedings of the 26th Annual Conference on Computer Graphics and Interactive Techniques. p. 187–194. SIGGRAPH ’99, ACM Press/Addison-Wesley Publishing Co., USA (1999). https://doi.org/10.1145/311535.311556, https://doi.org/10.1145/311535.311556
  * [8] Cao, C., Weng, Y., Zhou, S., Tong, Y., Zhou, K.: Facewarehouse: A 3d facial expression database for visual computing. TVCG 20(3), 413–425 (2014)
  * [9] Chan, E.R., Lin, C.Z., Chan, M.A., Nagano, K., Pan, B., Mello, S.D., Gallo, O., Guibas, L., Tremblay, J., Khamis, S., Karras, T., Wetzstein, G.: Efficient geometry-aware 3D generative adversarial networks. In: CVPR (2022)
  * [10] Claes, P., Vandermeulen, D., Greef, S.D., Willems, G., Suetens, P.: Craniofacial reconstruction using a combined statistical model of face shape and soft tissue depths: Methodology and validation. Forensic Science International 159(supplement), 147–158 (2006)
  * [11] Deng, Q., Zhou, M., Shui, W., Wu, Z., Ji, Y., Bai, R.: A novel skull registration based on global and local deformations for craniofacial reconstruction. Forensic science international 208(1), 95 – 102 (2011)
  * [12] Deng, Q., Zhou, M., Wu, Z., Shui, W., Ji, Y., Wang, X., Liu, C.Y.J., Huang, Y., Jiang, H.: A regional method for craniofacial reconstruction based on coordinate adjustments and a new fusion strategy. Forensic Science International 259, 19–31 (2016). https://doi.org/https://doi.org/10.1016/j.forsciint.2015.10.033, https://www.sciencedirect.com/science/article/pii/S0379073815004673
  * [13] Dhariwal, P., Nichol, A.: Diffusion models beat gans on image synthesis. Advances in Neural Information Processing Systems 34, 8780–8794 (2021)
  * [14] Duan, F., Huang, D., Tian, Y., Lu, K., Wu, Z., Zhou, M.: 3d face reconstruction from skull by regression modeling in shape parameter spaces. Neurocomputing 151, 674–682 (2015)
  * [15] Feng, Y., Feng, H., Black, M.J., Bolkart, T.: Learning an animatable detailed 3d face model from in-the-wild images. ACM Transactions on Graphics (ToG) 40(4), 1–13 (2021)
  * [16] Gietzen, T., Brylka, R., Achenbach, J., Zum Hebel, K., Schömer, E., Botsch, M., Schwanecke, U., Schulze, R.: A method for automatic forensic facial reconstruction based on dense statistics of soft tissue thickness. PloS one 14(1), e0210257 (2019)
  * [17] Grammer, K., Thornhill, R.: Human (homo sapiens) facial attractiveness and sexual selection: the role of symmetry and averageness. Journal of comparative psychology 108(3), 233 (1994)
  * [18] Gupta, S., Gupta, V., Vij, H., Vij, R., Tyagi, N.: Forensic facial reconstruction: The final frontier. Journal of clinical and diagnostic research: JCDR 9(9), ZE26 (2015)
  * [19] Hong, F.T., Zhang, L., Shen, L., Xu, D.: Depth-aware generative adversarial network for talking head video generation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 3397–3406 (2022)
  * [20] Isola, P., Zhu, J.Y., Zhou, T., Efros, A.A.: Image-to-image translation with conditional adversarial networks. In: Computer Vision and Pattern Recognition (CVPR), 2017 IEEE Conference on (2017)
  * [21] Isola, P., Zhu, J.Y., Zhou, T., Efros, A.A.: Image-to-image translation with conditional adversarial networks. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 1125–1134 (2017)
  * [22] Jia, B., Zhao, J., Xin, S., Duan, F., Pan, Z., Wu, Z., Li, J., Zhou, M.: Craniofacial reconstruction based on heat flow geodesic grid regression (hf-ggr) model. Computers & Graphics 97, 258–267 (2021)
  * [23] Jones, B.C., Little, A.C., Penton-Voak, I.S., Tiddeman, B.P., Burt, D.M., Perrett, D.I.: Facial symmetry and judgements of apparent health: Support for a “good genes” explanation of the attractiveness–symmetry relationship. Evolution and human behavior 22(6), 417–429 (2001)
  * [24] Jones, M.: Facial reconstruction using volumetric data. In: Proceedings of the Vision Modeling and Visualization Conference 2001. pp. 135–150. VMV ’01 (2001)
  * [25] Joukal, M., Frišhons, J.: A facial reconstruction and identification technique for seriously devastating head wounds. Forensic Science International 252, 82–86 (2015)
  * [26] Kähler, K., Haber, J., Seidel, H.: Reanimating the dead: reconstruction of expressive faces from skull data. TOG 22(3), 554–561 (2003)
  * [27] Karras, T., Aittala, M., Laine, S., Härkönen, E., Hellsten, J., Lehtinen, J., Aila, T.: Alias-free generative adversarial networks. In: Proc. NeurIPS (2021)
  * [28] Kossaifi, J., Tran, L., Panagakis, Y., Pantic, M.: Gagan: Geometry-aware generative adversarial networks. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 878–887 (2018)
  * [29] Kumar, A., Marks, T.K., Mou, W., Feng, C., Liu, X.: Uglli face alignment: Estimating uncertainty with gaussian log-likelihood loss. In: Proceedings of the IEEE/CVF International Conference on Computer Vision Workshops. pp. 0–0 (2019)
  * [30] Liu, C., Li, X.: Superimposition-guided facial reconstruction from skull. arXiv preprint arXiv:1810.00107 (2018)
  * [31] Lorense, W.E.: A high resolution 3d surface construction algorithm. Proc Siggraph (1987)
  * [32] Loshchilov, I., Hutter, F.: Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101 (2017)
  * [33] Madsen, D., Lüthi, M., Schneider, A., Vetter, T.: Probabilistic joint face-skull modelling for facial reconstruction. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 5295–5303 (2018)
  * [34] Marur, T., Tuna, Y., Demirci, S.: Facial anatomy. Clinics in dermatology 32(1), 14–23 (2014)
  * [35] Miranda, G.E., Wilkinson, C., Roughley, M., Beaini, T.L., Melani, R.F.H.: Assessment of accuracy and recognition of three-dimensional computerized forensic craniofacial reconstruction. PloS one 13(5), e0196770 (2018)
  * [36] Muller, J., Mang, A., Buzug, T.: A template-deformation method for facial reproduction. In: ISPA. pp. 359–364 (2005)
  * [37] Nguyen, J.D., Duong, H.: Anatomy, head and neck, face. In: StatPearls [Internet]. StatPearls Publishing (2022)
  * [38] Patashnik, O., Wu, Z., Shechtman, E., Cohen-Or, D., Lischinski, D.: Styleclip: Text-driven manipulation of stylegan imagery. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 2085–2094 (2021)
  * [39] Paysan, P., Lüthi, M., Albrecht, T., Lerch, A., Amberg, B., Santini, F., Vetter, T.: Face reconstruction from skull shapes and physical attributes. In: Proceedings of the 31st DAGM Symposium on Pattern Recognition. pp. 232–241. Springer-Verlag (2009)
  * [40] Qiu, Z., Li, Y., He, D., Zhang, Q., Zhang, L., Zhang, Y., Wang, J., Xu, L., Wang, X., Zhang, Y., et al.: Sculptor: Skeleton-consistent face creation using a learned parametric generator. ACM Transactions on Graphics (TOG) 41(6), 1–17 (2022)
  * [41] Ramesh, A., Dhariwal, P., Nichol, A., Chu, C., Chen, M.: Hierarchical text-conditional image generation with clip latents. arXiv preprint arXiv:2204.06125 1(2), 3 (2022)
  * [42] Ranjan, A., Bolkart, T., Sanyal, S., Black, M.J.: Generating 3d faces using convolutional mesh autoencoders. In: Proceedings of the European conference on computer vision (ECCV). pp. 704–720 (2018)
  * [43] Rhine, J.S., Campbell, H.R.: Thickness of facial tissues in american blacks. Journal of Forensic Sciences 25(4), 847–858 (1980)
  * [44] Rhine, J., Moore, C.: Tables of facial tissue thickness of american caucasoids in forensic anthropology. maxwell museum technical series, 1984; 1. TAYLOR, K. T.: Forensic Art and Illustrations. CRC Press, New York (2001)
  * [45] Rombach, R., Blattmann, A., Lorenz, D., Esser, P., Ommer, B.: High-resolution image synthesis with latent diffusion models. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 10684–10695 (2022)
  * [46] Sehwag, V., Hazirbas, C., Gordo, A., Ozgenel, F., Canton, C.: Generating high fidelity data from low-density regions using diffusion models. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 11492–11501 (2022)
  * [47] Spradley, M.K.: Metric methods for the biological profile in forensic anthropology: sex, ancestry, and stature. Academic forensic pathology 6(3), 391–399 (2016)
  * [48] Stephan, C.N.: Accuracies of facial soft tissue depth means for estimating ground truth skin surfaces in forensic craniofacial identification. International journal of legal medicine 129, 877–888 (2015)
  * [49] Stephan, C.N., Guyomarc’h, P.: Facial soft tissue depth measurement: Validation of the 75-shormax. Journal of forensic sciences 61(5), 1327–1330 (2016)
  * [50] Tilotta, F.M., Glaunès, J.A., Richard, F.J., Rozenholc, Y.: A local technique based on vectorized surfaces for craniofacial reconstruction. Forensic science international 200(1-3), 50–59 (2010)
  * [51] Valsecchi, A., Damas, S., Cordón, O.: A robust and efficient method for skull-face overlay in computerized craniofacial superimposition. IEEE Transactions on Information Forensics and Security 13(8), 1960–1974 (2018)
  * [52] Vandermeulen, D., Claes, P., De Greef, S., Willems, G., Clement, J., Suetens, P.: Automated facial reconstruction. Craniofacial identification 203 (2012)
  * [53] Verzé, L., et al.: History of facial reconstruction. Acta Biomed 80(1), 5–12 (2009)
  * [54] Wang, T., Zhang, B., Zhang, T., Gu, S., Bao, J., Baltrusaitis, T., Shen, J., Chen, D., Wen, F., Chen, Q., et al.: Rodin: A generative model for sculpting 3d digital avatars using diffusion. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 4563–4573 (2023)
  * [55] Wilkinson, C.M., Roughley, M., Moffat, R.D., Monckton, D.G., MacGregor, M.: In search of robert bruce, part i: craniofacial analysis of the skull excavated at dunfermline in 1819. Journal of Archaeological science: reports 24, 556–564 (2019)
  * [56] Zhang, N., Zhao, J., Duan, F., Pan, Z., Wu, Z., Zhou, M., Gu, X.: An end-to-end conditional generative adversarial network based on depth map for 3d craniofacial reconstruction. In: Proceedings of the 30th ACM International Conference on Multimedia. pp. 759–768 (2022)
  * [57] Zhao, J., Duan, F., Pan, Z., Wu, Z., Li, J., Deng, Q., Li, X., Zhou, M.: Craniofacial similarity analysis through sparse principal component analysis. Plos one 12(6), e0179671 (2017)
  * [58] Zhu, J.Y., Park, T., Isola, P., Efros, A.A.: Unpaired image-to-image translation using cycle-consistent adversarial networks. In: Computer Vision (ICCV), 2017 IEEE International Conference on (2017)
  * [59] Zhu, J.Y., Park, T., Isola, P., Efros, A.A.: Unpaired image-to-image translation using cycle-consistent adversarial networks. In: Proceedings of the IEEE international conference on computer vision. pp. 2223–2232 (2017)

In addition to this supplementary material, we refer the reader to the
supplementary videos for a more comprehensive explanation and additional
results.

  1. 1.

demo1-pipeline.mp4 shows the end-to-end pipeline of the facial reconstruction
from a skull.

  2. 2.

demo2-landmark.mp4 shows animations of 3D facial editing, globally and in
local regions.

  3. 3.

demo3-comparisons.mp4 shows the comparison between our method and existing
methods.

  4. 4.

demo4-king-face.mp4 shows a real-world facial reconstruction example, which
uses the skull of Robert the Bruce, King of Scots from 1306 to 1329.

##  Appendix 0.A Implementation Details

###  0.A.1 Text Prompt

We aim to synthesize 2D portraits that are depicted from the frontal view
without occlusion. Hence, we designed the following prompt template to
synthesize 2D portraits as initial faces,

> _prompt_ : A photo of a human face, facing the camera, full head visible,
> sharp focus, fine art portrait photography, with {Age}, {Ancestry},
> {Gender}, and {Face Shape}.

The attributes $\\{\leavevmode\nobreak\ \\}$ are replaced by the values in
Table 3. We also used the negative prompt to ensure no accessories such as
hats and glasses occluding the face,

> _negative prompt_ : Hat, glasses, black and white, cartoon, anime.

To reconstruct the face of Robert the Bruce, King of Scotts, from the scanned
skull. We used the following text prompt to generate the initial faces from
his historical background,

> _prompt_ : A photo of a king’s face. A Scottish male with a mustache, long
> hair, crown, and royal attire. Looking at the camera. Realistic portrait.

Table 3: Attributes used for initial face generation to match the biological
profile of the skull.

Attributes | Values
---|---
Age | 10 to 20, 20 to 30, 30 to 50, 50 to 70
Ancestry | African, Asian, European, Latino
Gender | Female, Male
Face Shape | Normal, Fat, Thin

###  0.A.2 Network Weights and Hyper-parameters

We used PCA to fit the tissue depths of the training set. To provide local
control, we further divided the face into five regions and used PCA to fit
tissue depths in each region. We kept the principal component with the largest
eigenvalue for both global control and local control. We used the pre-trained
models and fixed their weights in our pipeline (Stable Diffusion XL, DECA
encoder, and FLAME decoder).

In Eqn. 2, we set $\alpha_{0}=5$, $\alpha_{1}=1$, $\alpha_{2}=1$. We use AdamW
[32] to optimize the face latent code $f$ with an initial learning rate of
$10^{-2}$. The learning rate is multiplied by $0.2$ for every $200$
iterations, with $1000$ iterations in total.

###  0.A.3 Dataset Processing

The whole-head CT scans were obtained by using a clinical multi-slice CT
scanner system which provides CT images with a resolution of $512\times 512$
(pixel size $0.5\times 0.5mm$, inter-slice thickness $0.75mm$). After
extracting meshes from the CT data, the heads and their underlying skulls are
normalized into a unified Frankfurt coordinate system [11] to eliminate the
inconsistency in posture and scale.

###  0.A.4 Pipeline Running Time

We calculated the overall running time on a single NVIDIA 4090 GPU. The 2D
portrait is synthesized by a pre-trained Stable Diffusion XL model (7s). The
initial 3D face is reconstructed from the synthesized 2D image by a pre-
trained DECA model (2s). Then we adapt the face to match the landmark
constraints and use a back-propagation to optimize the face latent code (5s).
The overall time consumption is around 14 seconds for facial reconstruction
from a skull.

Figure 11:  PCA decomposition of tissue depths in the training set, with the
1st component dominating the distribution with $51.9\%$ of the variance.

Figure 12: Ablation study on loss functions. Projection loss
$\mathcal{L}_{proj}$ helps reduce face deviation from landmark constraints;
symmetric loss $\mathcal{L}_{sym}$ enhances face realism.

##  Appendix 0.B Analysis on Tissue Depth Distribution

We used PCA to decompose the joint distribution of tissue depths from the
training set of the skull-skin dataset. We observed that the first component
dominates the variances of PCA results, as Fig. 11 shown.

##  Appendix 0.C More Analysis on Face Adaptation Module

###  0.C.1 Qualitative Results of Loss Functions

Fig. 12 shows the reconstruction results using different loss functions. The
proposed projection loss $\mathcal{L}_{proj}$ and symmetric loss
$\mathcal{L}_{sym}$ help enhance the face realism. A quantitative ablation
study is reported in the main text.

Figure 13: Example questions in the user study. Faces A, B, and C were
reconstructed from one skull, while Face D was reconstructed from the other
skull. Faces are shuffled in the user study. Figure 14: User study of
convergence quality of final facial reconstruction from different initial
faces. We collected 500 valid responses from 50 valid users. The result shows
that our model can adapt different initial faces and converge to a face shape
space that is related to the query skull, which is distinct from the
reconstructed faces that are from the other skulls.

###  0.C.2 User Study: Convergence Quality of Final Facial Reconstruction from
different initial faces

Fig. 13 shows two examples of user study questions. For each question, we
randomly synthesized four faces from four different portraits and used the
proposed Anatomy-guided Face Adaptation module to adapt to the query skull.
Faces A, B, and C are final reconstructions of one skull. Face D is the final
reconstruction of the other skull. Faces are shuffled in the user study. Users
are asked to compare the reconstructed faces and select the unique one
reconstructed from the other skull.

We collected 500 valid responses from 50 valid users. For each question, we
count 1 score so the overall score is 10 for 10 questions. Fig. 14 shows the
user study results, where the average correctness is $91.2\%$. The result of
the user study demonstrates that (1) our model can adapt different initial
faces and converge to face shapes that are closely related to the query skull,
resulting in distinct reconstructions for different skulls. (2) The
reconstructed faces from one skull are not identical, they still preserved
subtle geometric details inherited from the initial faces.

|
---|---
(a) Changes of Tissue Depths | (b) Faces Adapted to the Facial Landmarks
Figure 15:  Adjusting local face regions by TDD-Regional. Tissue depths increase on forehead (from left to right) cheek (from top to bottom). The resulting faces closely align with the corresponding landmarks. Rectangles highlight the affected regions.  |
---|---
(a) Changes of Tissue Depths | (b) Faces Adapted to the Facial Landmarks
Figure 16:  Adjusting local face regions by TDD-Regional. Tissue depths
increase on the mouth (from left to right) and cheek (from top to bottom). The
resulting faces closely align with the corresponding landmarks. Rectangles
highlight the affected regions.

##  Appendix 0.D Detailed Results of Regional Control

We project the tissue depths in each face region to the first component of the
PCA decomposition. Then users can intuitively and conveniently control the
face shape by tuning one parameter. We show several examples below to
demonstrate how users can intuitively and conveniently control the face
shapes. For each example, we randomly select two different regions to
visualize the results when tuning multiple regions at the same time.

##### Forehead and cheek control.

Fig. 15 shows the adjustment of the forehead and cheek regions by using
different combinations of tissue depths. Our face adaption module can
consistently fit the face to the landmark constraints well.

##### Mouth and cheek control.

We further show the adjustment of the mouth and cheek regions by using
different combinations of tissue depths in Fig. 16.

Figure 17: Qualitative comparisons with existing methods. (a) is the input
skull. (b-e) show the results of HF-GGR [22], Pix2Pix [20], CycleGAN [58], and
CGAN-CFG [56]. (f) show our reconstructed face with texture and geometry. (g)
is the CT scanned groundtruth.

##  Appendix 0.E Extensive Comparisons with Existing Methods

Fig. 17 shows extensive comparisons with the existing skull-to-face
reconstruction methods. HF-GGR [22] is a method that tends to produce average
faces that lack details. CGAN-CFG [56] directly memorizes the relationships
between skull and face in the training set, leading to difficulties in
generalization to the test set. Pix2Pix [20] and CycleGAN [58] focus on
learning image-to-image translation for skull and face images, often resulting
in unrecognizable faces due to their requirement of larger training datasets.


# Reslicing Ultrasound Images for Data Augmentation
and Vessel Reconstruction

Cecilia G. Morales, Jason Yao, Tejas Rane, Robert Edman, Howie Choset, Artur
Dubrawski
*This work was supported by U.S. Department of Defense contracts W81XWH-19-C0083, W81XWH-19-C0101, and W81XWH-19-C-0020Authors are with the Robotics Institute, Carnegie Mellon University, Pittsburgh, PA, USA {cgmorale, jlyao, tejasr, redman, choset<EMAIL_ADDRESS>
###### Abstract

Robot-guided catheter insertion has the potential to deliver urgent medical
care in situations where medical personnel are unavailable. However, this
technique requires accurate and reliable segmentation of anatomical landmarks
in the body. For the ultrasound imaging modality, obtaining large amounts of
training data for a segmentation model is time-consuming and expensive. This
paper introduces RESUS (RESlicing of UltraSound Images), a weak supervision
data augmentation technique for ultrasound images based on slicing
reconstructed 3D volumes from tracked 2D images. This technique allows us to
generate views which cannot be easily obtained in vivo due to physical
constraints of ultrasound imaging, and use these augmented ultrasound images
to train a semantic segmentation model. We demonstrate that RESUS achieves
statistically significant improvement over training with non-augmented images
and highlight qualitative improvements through vessel reconstruction.

##  I INTRODUCTION

Adequate vascular access in trauma patients prior to decompensation of
physiological processes is essential for a patient’s survival. Vascular access
is of paramount importance to provide a route for anesthesia or other
medications, resuscitative fluids, intravenous contrast for diagnostic
procedures, and it may be a useful adjunct for measuring physiological
parameters during primary resuscitation [1, 2]. An example where the provision
of fluids is contingent on obtaining suitable vascular access to the patient’s
venous system is hemorrhaging, the leading cause of death in severely injured
trauma patients: it accounts for 31% of deaths in the first hour after injury
[1]. Clinical management involves achieving hemostasis by replacing lost
intravascular volume with fluids and blood, and treating coagulopathy [3].
Other treatments use the insertion of a vascular catheter for Resuscitative
Endovascular Balloon Occlusion of the Aorta (REBOA) via the femoral artery to
prevent significant loss of blood or extracorporeal membrane oxygenation
(ECMO).

For large volume resuscitation, the insertion of a central venous line is
potentially lifesaving since it enables rapid administration of high volumes
of isotonic fluids and medications that would be caustic to peripheral veins
[3]. Taking into account the rapid deterioration in the health of a patient,
it is vital that a patient is treated within a short period of time in order
to save the patient’s life. However, in situations where the patient is in a
remote location such as a soldier on the battlefield, or in massive casualty
scenarios, access to appropriate treatment resources can be limited.

While catheterization is a standard procedure in clinical practice, robot-
guided catheter insertion could enable necessary medical care to the patient
within a short period of time where medical personnel are not yet readily
available. Given the important role that femoral blood vessels have in the
resuscitation of a patient, it is important to identify them inside the leg.
To be able to determine the proper robot needle insertion location
autonomously, a suitable medical imaging modality is necessary.

Medical imaging involves non-invasive procedures aimed to visualize anatomical
features within the body. Some of the most common imaging techniques in
clinical practice are X-ray, computer tomography (CT), ultrasound (US),
magnetic resonance (MR), and positron emission tomography (PET). This work
utilizes ultrasound as it allows real-time imaging, easy portability, no need
for intravenous contrast agents, no ionizing radiation, and low-costs.

Ultrasound imaging, also called sonography, uses sound waves to capture
internal images of the body. An instrument called a transducer emits high-
frequency sound, inaudible to human ears, and records the echoes as the sound
waves bounce back to determine the size, shape, and consistency of soft
tissues and organs [4].

Figure 1: Our Mechanical Ultrasound Scanning System, which consists of a
linear array ultrasound transducer attached to the end-effector of a 6-DoF
Universal Robots UR3e serial manipulator.

By attaching an ultrasound transducer to the robot’s end-effector and
utilizing the method described in [5], we can scan the femoral region to
identify the exact location of the vessels. Using a robotic ultrasound system
ensures accuracy, stability, maneuverability, and tracking the location of the
transducer during ultrasound image acquisition.

Nonetheless, automatically segmenting anatomical landmarks in the body such as
arteries or veins is a challenging task. One must consider the noise and
artifacts in images, inter-patient and machine variability, as well as natural
and injury-inflicted anatomical differences. Acquiring comprehensive sets of
training data for all these scenarios would be time-consuming, difficult and
expensive, since only professionally trained personnel are able to adjudicate
them.

In this work, we introduce RESUS (RESlicing of UltraSound Images), a weak
supervision data augmentation technique for ultrasound images. Specifically,
the method is designed to reconstruct 3D volumes made up of tracked 2D
ultrasound images. The volume can then be resliced to reveal 2D images in
planes that would not be accessible due to the physical restrictions of the
scanning process, i.e., a plane parallel to the surface of the skin. These new
reconstructed ultrasound images are then used to train a semantic segmentation
U-Net network [6] such that it is able to generalize to different anatomical
variations and slight altercations, artifacts, and varying image resolutions.

We collect ultrasound images with a robotic arm through a series of
experiments on a medical imaging phantom as well as real-world data. We
compare our method with the predictions of a U-Net trained on non-augmented
images, a U-Net pretrained on ImageNet, and a U-Net trained on random
augmentation images. RESUS achieves statistically significant, respectively
8%, 5%, and 23% increases in Intersection over Union (IoU) score, also known
as Jaccard Index, in segmenting images.

The main contributions of our work are: 1) Developing a novel method that
augments ultrasound images to train on new imaging perspectives which are
otherwise inaccessible due to the physical constraints of the system; 2)
Creating a novel volume reconstruction technique to adapt to the data
collected; 3) Evaluating the effectiveness of our method both qualitatively
and quantitatively in increasing the IoU score; 4) Achieving better vessel
reconstructions affordably using weak supervision on real-world data; 5)
Increasing the availability of data and easing the labelling task.

##  II BACKGROUND AND RELATED WORK

Figure 2: Flow diagram of the proposed method. We collect 2D ultrasound images
from a linear array ultrasound transducer attached to the end-effector of the
6-DoF Universal Robots UR3e serial manipulator. A 3D volume is reconstructed
from these images, and then resliced to generate frames, some of which which
would have been inaccessible during scanning. These frames are then added to
the training data to obtain the segmentation model, which identifies the
vessels. The segmented vessels inform vessel reconstruction needed to find the
optimal needle insertion location.

###  II-A Ultrasound Systems

The three main types of ultrasound (US) are the 3D ultrasound system, and the
mechanical and freehand scanning system using a 2D ultrasound probe. In the 3D
ultrasound system, data is directly acquired by a series of dedicated 3D
probes with an oscillating mechanism that sweeps a predefined region of
interest [7]. Although, it allows a whole volume of data to be acquired at a
time, it is expensive and not commonly available. Furthermore, the size of the
acquired volume is limited by the dimensions of the transducer which is
incapable of scanning large volume organs. The freehand scanning method with a
position tracker is comparatively cheaper and more flexible, but the frames
acquired typically are irregularly spaced and highly sparse.

Our approach uses the mechanical scanning system with a conventional linear
array transducer. During ultrasound image acquisition, the transducer is
maneuvered under the computer control to collect regularly spaced ultrasound
images, and the pose of the scanning head is recorded synchronously with the
image [7]. Although, such systems are relatively costly, they allow for more
reliable scanning and tracking, and the limited scanning range is enough to
perform a simple linear scan along the leg of a patient.

###  II-B Volume Reconstruction Strategies

Volume reconstruction starts by obtaining the ultrasound images from a linear
array probe and their corresponding collection poses. The two main
reconstruction methods are pixel based and voxel based.

Pixel nearest neighbor (PNN) is the most common pixel based reconstruction
method. It proceeds traveling across each pixel of the acquired 2D ultrasound
images, and the nearest voxel in the 3D reconstructed volume is filled with
that pixel value [8]. Although PNN results in loss of important information
from the 2D images, it can still be favorable for 3D reconstruction because of
its simplicity and relatively low computation costs [9].

Voxel based methods are used to reconstruct the 3D volume by traveling across
each voxel in the volume grid and gathering the pixel values from input 2D
ultrasound frames, and computing the voxel values [9]. The most common ways of
computing the voxel values include voxel-nearest neighbor (VNN) and distance-
weighted (DW). The VNN selects the nearest pixel value from a set of 2D frames
to be put on the voxel, whereas the DW takes a weighted mean of local
neighborhood pixels by the inverse distances between the pixel and the voxel
[9]. The VNN is able to preserve the original texture patterns from the 2D
frames, but it tends to generate large reconstruction artifacts and preserve
the speckle noise from corrupted frames. The DW is able to suppress the
speckle noise, but it smooths out the 3D reconstructed volume, causing loss of
important information.

Since our training data is dense and reconstruction speed is not a high
priority, we use the squared-distance-weighted (SDW) reconstruction algorithm
from Huang et al. [10], an extension of the DW method that aims to reduce the
smoothing effect in the reconstructed volume.

###  II-C Semantic Segmentation using Deep Neural Networks

U-Net [11] is one of the deep learning techniques most widely used for
analyzing medical images to identify organs and lesions. Manual segmentation
of images implies a heavy workload for doctors, and it can introduce bias if
it involves their subjective opinions. U-Net is designed primarily for image
segmentation [12]. The basic structure of the U-Net architecture consists of
two paths: the encoder, which provides classification information, and the
decoder, which helps in localizing the classification information. This
structure meets the requirements of medical image segmentation with its fast
training speeds, and ability to yield reliable models using small amounts of
training data. Various strategies are applied to modify the U-Net structure,
to address different segmentation problems, and it has improved in the areas
of application range, feature enhancement, training speed optimization,
training accuracy, feature fusion, small sample training set, and
generalization improvement [11].

Another approach adopted by [13] used the You Only Look Once (YOLO) v3-Tiny
network to detect and compute bounding boxes on arteries, veins, and
bifurcations. The network was first trained on the ImageNet [14] database,
which consists of millions of natural images. Then, to adapt the network to
ultrasound images, transfer learning was performed by retraining the network
using a database of annotated ultrasound images of the porcine femoral region.
Furthermore, affine transformations and random intensities were used as data
augmentation techniques to improve the generalization performance.

###  II-D Augmentations

When working with deep neural networks for semantic segmentation, data
augmentation is a relatively simple and commonly used method for generalizing
across unseen domains and to account for invariances. Many data augmentation
methods for medical images involve simple image transformations, such as
flips, skips, skews and blurs. However these augmentations do not seem to have
much of an impact in the context of ultrasound images.

A novel data augmentation technique for ultrasound images was developed by
[15], where a 3D U-Net was trained adaptively to generalize over different
anatomical variations. The augmentation module, an agent feed forward neural
network, is responsible for generating the synthetic images for further
training [15]. This method was able to enhance the generalizability of the 3D
U-Net, but at the cost of high memory usage and long computation time,
therefore not being feasible for real-time field applications, or large
amounts of data. Another limitation of this method was that it could miss
image deformations in areas of the image not covered by the segmentation
ground-truth labels.

Our method improves the aforementioned approaches by reslicing a 3D
reconstructed volume to create augmented data which is more intuitive and
interpretable than performing random transformations for data augmentation or
applying transfer learning when the domain images are highly distinct. This
allows us to re-generate unseen views in the form of raw images, such as a
longitudinal slice where all the raw images were transverse images, and create
slight variations between vessels that could be seen in a new patient. By
generating a variety of reconstructed views we increase the volume and
comprehensiveness of training data for the segmentation network, as the single
reconstructed volume can generate many different altercations of the vessels
and resulting images.

##  III METHOD

In our approach, a robot with an ultrasound probe in its end-effector scans
the femoral region of the leg and reconstructs a volume that is then sliced to
reveal 2D images that would have not been accessible for the ultrasound due to
the physical restrictions of the scanning process. This produces a diverse set
of views of the vessel that help generalize to different patients. Those
images are used to train a vessel segmentation model. Once the vessels are
segmented, they can be reconstructed and used to find appropriate needle
insertion sites.

###  III-A Dataset

Our method was first tested on a medical imaging phantom, CAE Blue Phantom
anthropomorphic gel model. Images were scanned from both the right side and
left side of the phantom. Once the pipeline was ready, we transitioned to
experiments in a surgical environment at the University of Pittsburgh Medical
Center (UPMC) with live pigs under anesthesia. The experiments involving live
animals have been conducted in accordance with the Institutional Animal Care
and Use Committee (IACUC) protocol approved by the cognizant authority. We
collected ultrasound images of the femoral arteries and veins from eight
different pigs, 500 to 2070 images each, using the 6-DoF Universal Robot UR3e
serial manipulator to move the probe used for scanning. The probe used was the
Fukuda Denshi portable point-of-care scanner (POCUS) with 5-12MHz linear
transducer with a maximum depth of 5cm and 10cm, respectively. A total of
14,207 images were collected and labelled by expert clinicians using the
Computer Vision Annotating Tool (CVAT) [16]. Robot Operating System (ROS) [17]
was used to record the poses of the robot and concurrent ultrasound images.

###  III-B Volume Definition

We first define the voxel volume where the reconstruction will take place.
From the scanned images, we pick a start and end frame to exclude less useful
data such as stationary images and extraneous movement. From the resulting
subset of image frames, all the pixels in the images are treated as points in
a point cloud and transformed to their position in space according to the
transforms from ROS.

We then assume that the resulting point cloud is shaped like a parallelepiped
and choose one of its corners to be the new origin of the voxel volume and
vectors to its three adjacent vertices become the new basis vectors. We
perform a change of coordinates on all the points in the point cloud to the
new volume coordinates, resulting in all of the pixels that would contribute
to the reconstruction ending up in the unit cube. We then scale each dimension
up by the number of voxels desired in the final volume. We assume the point
cloud is shaped like a parallelepiped since scanning vessels in the leg need a
relatively linear motion without many complex movements.

The main challenge for this volume definition method is finding the relevant
vertices of the parallelepiped. We first fit planes to each side of the
parallelepiped using a modified RANSAC [18] algorithm on the subset of the
point cloud corresponding to each face, which helps reduce the effects of
extraneous movements on the overall volume definition. The algorithm
repeatedly samples 3 points randomly from the point cloud and defines a plane
through them. It then computes the distance of that plane to all the points in
the point cloud and counts the number of inlier points within a certain
distance threshold. The plane that has the greatest number of inliers after a
set number of iterations is chosen as the best-fit plane.

Figure 3: Point clouds created from transformations of the top (left), left
(center), and all sides (right) of the $3\times 3$ image. The planes are
fitted with RANSAC on each side separately.

To reduce the number of points in the point cloud, we treat each image as
having dimensions of $3\times 3$ pixels and scale them accordingly such that
the pixels from the corners of the smaller image end up in the same
coordinates as the corners of the original image. The point clouds for the 4
side faces of the volume are created by transforming 3 pixels from each image
corresponding to the 4 sides of the image (see Figure 4). For the 2 base
faces, the normal vector is first found by running RANSAC on the point cloud
created by only performing the rotation transform on each image and omitting
the translation. This plane represents the overall orientation of all the
image frames, and is then translated to the coordinates of the start and end
frames to close out the volume.

From these 6 planes, we calculate the relevant vertices by finding the
intersections between the planes. The origin is chosen to be the intersection
point closest to the top-left corner of the start frame. The vertex along the
vector representing the new $z$ axis is the point closest to the top-left
corner of the end frame, and the $x$ and $y$ vectors are chosen according to
the right-hand rule.

###  III-C Volume Reconstruction

Our reconstruction strategy uses the SDW algorithm from Huang et al. [10],
which assigns the intensity for each voxel in the volume by considering all
the image pixels that fall into the spherical region within radius $r$ of the
voxel. A voxel at coordinate $\vec{V}_{C}$ has intensity

$I(\vec{V}_{C})=\frac{\sum_{k=0}^{n}W_{k}I(\vec{V}_{P}^{k})}{\sum_{k=0}^{n}W_{k}}$
(1)

where $I{(\vec{V}_{P}^{k})}$ is the intensity of the $k$th pixel out of $n$
falling within the spherical region at coordinate $\vec{V}_{P}^{k}$. $W_{k}$
is the weight of the $k$th pixel, defined by

$W_{k}=\frac{1}{(d_{k}+\alpha)^{2}}$ (2)

where $d_{k}$ is the distance between $\vec{V}_{C}$ and $\vec{V}_{P}^{k}$ and
$\alpha$ is a positive parameter to adjust the reconstruction.

After the initial reconstruction using image pixel data, holes in the
reconstructed volume are filled using the same algorithm, just with nearby
voxels instead of image pixels. The hole filling involves multiple passes,
with $r$ being increased after each pass up to a limit.

Implementation wise, we keep track of the numerators and denominators for each
voxel separately and loop over each image frame, adding every pixel’s
contribution to nearby voxels. For each frame, the pixel vectors in the image
are transformed to its coordinates in the volume by first applying the
corresponding ROS transform to get coordinates in terms of the robot base, and
then a second transform from the robot base coordinates to the volume
coordinates as defined in the previous section. Specifically, for each 2D
pixel vector $\vec{P}$, we have:

$\vec{V}_{P}=\begin{bmatrix}n_{x}&0&0\\\ 0&n_{y}&0\\\
0&0&n_{z}\end{bmatrix}X^{-1}\begin{bmatrix}R&\vec{b}-\vec{u}\end{bmatrix}\begin{bmatrix}\vec{P}\\\
0\\\ 1\end{bmatrix}$ (3)

where $R$ is the $3\times 3$ rotation matrix and $\vec{b}$ is the $3\times 1$
translation vector from the ultrasound image frame to the robot base frame,
$\vec{u}$ is the new origin, $X$ is the $3\times 3$ matrix with columns
consisting of the new basis vectors, and $n_{x},\,n_{y},\,n_{z}$ are the
number of voxels for each dimension of the volume. We augment the initial
pixel vector to be able to perform matrix multiplication and vector addition
with a single multiplication. Then, each pixel’s distance and intensity to all
voxels within a radius $r$ is calculated, and its contribution to each voxel’s
overall summation is added.

Once all the images have been processed, hole locations are determined by the
voxels that have 0 for its denominator. These are temporarily set to 1, and
each voxel’s intensity is calculated by division. During the hole-filling
stage, the algorithm is slightly modified to exclude contributions from voxels
that are holes themselves.

All the raw ultrasound images were converted to gray scale for reconstruction,
and each label class (e.g. vein, artery, and background) was reconstructed
separately, with a pixel having a value of 1 if it was part of a label and 0
otherwise. The final label volume was created by assigning the label with the
maximum value for each voxel.

###  III-D Volume Slicing

From the reconstructed volumes, we now extract the augmented ultrasound images
and labels. To achieve this, the volumes are rotated around the $x$ and $y$
axes and fit within a new output volume using spline interpolation. The output
volume is then expanded to include all the data from the original volume, with
missing data filled with zeros.

Then, vertical slices are taken along the $z$ axis of the rotated output
volume to generate the augmented images. These vertical slices of the rotated
volume correspond to oblique slices of the original volume. Each augmented
image slice is first checked to ensure that the percentage of pixels
corresponding to the original volume (i.e., nonzero pixels) meets a certain
threshold to ensure the augmented image contains enough useful information. If
threshold is met, the maximum amount of zero pixels are cropped from each side
of the image, and then the image is resized to $256\times 256$ pixel grid. For
our results, the volumes were rotated $\pm 30^{\circ}$ around both axes with
$10^{\circ}$ steps. Vertical slices were taken once every 5 voxels, and our
threshold was 40% for the proportion of nonzero pixels needed in the augmented
image.

Figure 4: Raw ultrasound image and label (left) and sliced augmented image and
label (right)

###  III-E Training

Our goal is to develop semantic segmentation of the femoral artery and vein in
ultrasound images for the purpose of 3D reconstruction of the vessels for the
robot to find an adequate needle insertion point. To that end, the deep
learning network architecture that we use is a U-Net with an image
classification model, ResNet34 [19], backbone in PyTorch [20] as the encoder
using the Segmentation Models library [21]. ResNet34 was chosen due to its
relatively high training speed and low memory demands [22]. ResNet34 consists
of 34 layers with (3x3) convolutional filters using same padding, max-pooling
layers and fully-connected layers, ending with a sigmoid activation function
used to ascertain the output class. We train our network until the validation
Dice loss converges with batch size 8 on lower resolution (256 × 256) images.
We use the Adam optimizer[23] with a learning rate of 0.0001. The network was
trained on NVIDIA RTX A6000 GPUs.

The model is evaluated using leave-one-subject-out crossvalidation protocol,
in which each pig’s data is used once as a test set while the remaining data
forms the training set. Augmented images from RESUS are only used for
training. To avoid over-fitting, images resliced from the test pig are removed
from the training set.

###  III-F Vessel Reconstruction

Using the method described in III-C, we can input the predictions to
reconstruct the vessels at the exact location they were scanned, thus creating
an internal map of the vessels. To differentiate between different types of
vessels (artery vs. vein), we use the knowledge of body’s anatomy.

##  IV EXPERIMENTS

We first test our method using the medical phantom, by creating a 3D volume
and reslicing it. We consider two different models, one trained on unaugmented
ultrasound images in transverse view, and the other trained on the unaugmented
images along with the resliced images from the reconstructed volumes. We
compare the models with two different test sets, the first consisting of the
unaugmented images only, and the second consisting of both the unaugmented and
the resliced images representing different variations in real world data, such
as longitudinal vessels. We evaluate the results by reconstructing the vessels
and comparing them qualitatively to the ground truth, as well as
quantitatively with the IoU segmentation score.

Then we transition to real world data, by training and testing on ultrasound
data collected from live pigs. All the methods were tested using the
ultrasound images gathered for each pig at the time of scanning. We compare
our RESUS augmentation method with models trained on images collected from
different subjects to assess the inter-subject generalization capabilities of
the considered models. We also test models that were pretrained on ImageNet
[14], and others that used common augmentation techniques for medical images
such as horizontal and vertical flip, Gaussian noise addition, sharpening of
the images, random brightness, and random contrast. Using the model
predictions, we reconstruct the vessels and compare them visually with their
ground truth. To corroborate our results, we use the IoU score metric to
evaluate accuracy of segmentation.

##  V RESULTS AND ANALYSIS

Figure 5: A) Pig 1 Vessel Reconstruction; B) Pig 4 Vessel Reconstruction; C)
Phantom Vessel Reconstruction

To evaluate the efficacy of our method we use qualitative and quantitative
approaches. We observe that our method consistently surpasses the rest of the
models at vessel reconstruction and removal of artifacts. As it can be
observed in Figure 5C, RESUS reconstruction is visually closest to the ground
truth. The U-Net prediction has artifacts that could be confused by a robot
for a vessel, which could be life threatening in a real-life scenario.
Quantitatively, the proposed method yields a 20% improvement in IoU score for
the phantom vessels.

Evaluated on in-vivo animal data, the IoU score is on average 8% higher than
for the method using only unaugmented images and higher than attainable with
other medical image segmentation techniques such as transfer learning with
ImageNet (5%) and random augmentations (23%). Paired $t$-test performed to
assess significance of the observed improvements, yields p-values of $0.034$,
$0.049$ and $0.00046$, respectively. Thus, we can confidently conclude that
the proposed approach prevalently improves accuracy of image segmentation.
Table II summarizes quantitative results.

While the IoU scores might seem low, one of the limitations of this work
includes many flaws in the labels which lead to lower than attainable success
rates of the models. Another limitation was that for several pigs, the
ultrasound was stationary, while in others it was zigzagging making it
difficult to create a volume, thus for the eight pigs which we used to train
on, we only had five volumes of small sections of the data. Lastly, the
experiments schedule was sparse to allow for technical developments of the
prime purpose of the animal study, some of which mildly affected the collected
ultrasound data. Even with the aforementioned obstacles, by comparing all
qualitative results as shown in Figure 5(A&B) with the different methods, we
can observe that the RESUS segmentation is again the closest to the ground
truth. By including data obtained from our method to augment the training set,
our vessel reconstruction results qualitatively better represent the shape and
bifurcations of the vessels, and the resulting segmentations are more accurate
than attainable with alternative approaches under consideration. We observe
that these improvements can increase the likelihood of successful needle
insertions for various kinds of treatments in trauma care in the field.

TABLE I: IoU score of Segmentations tested on the Phantom | Train US | Train US and RESUS
---|---|---
Test US | 0.536 | 0.731
Test US and RESUS | 0.186 | 0.779
TABLE II: IoU Scores of Segmentations Tested on the pigs | U-Net | ImageNet | Aug | RESUS
---|---|---|---|---
Pig 1 ($n=850$) | 0.3685 | 0.591 | 0.383 | 0.610
Pig 2 ($n=951$) | 0.587 | 0.607 | 0.537 | 0.603
Pig 3 ($n=1307$) | 0.631 | 0.691 | 0.324 | 0.693
Pig 4 ($n=900$) | 0.513 | 0.590 | 0.350 | 0.665
Pig 5 ($n=900$) | 0.665 | 0.408 | 0.319 | 0.610
Pig 6 ($n=500$) | 0.7483 | 0.785 | 0.536 | 0.7529
Pig 7 ($n=2070$) | 0.422 | 0.509 | 0.262 | 0.571
Pig 8 ($n=1200$) | 0.574 | 0.531 | 0.555 | 0.601
AVERAGE | 0.557 | 0.592 | 0.408 | 0.638

##  VI CONCLUSION

This paper introduces RESUS, an augmentation method for image segmentation,
specific to imaging modalities consisting of multiple 2D slices of a 3D
volume. Examples of such modalities include ultrasound data and CT scans. We
have demonstrated RESUS on in-vivo animal data, showing performance
improvements over models trained on unaugmented data alone, models trained on
ImageNet, and models trained using commonly used medical imaging augmentation
techniques.

The goal of RESUS is to support using ultrasound imaging as a sensor for
robotic control. Currently, intelligent use of ultrasound imaging is limited
by the required large volumes of training data and the cost of harvesting
labels from domain experts. RESUS directly addresses these challenges by
providing affordable image augmentations, specific to ultrasound images that
are slices of 3D volumes. It yields high quality ultrasound image
segmentation, which when used for robotic control, can automate important
tasks in critical and trauma care.

Currently, we use the anatomical knowledge of the body [24] to differentiate
between the femoral artery and vein. Next, we will leverage models of
deformation to distinguish between veins and arteries automatically, and
integrate our tool with the needle insertion controller. We also observe that
even if the IoU score of our model was not always the highest, our method
showed the qualitatively best vessel reconstruction consistently. Thus, the
IoU score may not be the best evaluation metric for the segmentation of our
images, and we will look for a better one. Finally, the current robotic setup
is not yet suitable for the application in field emergency care: form factor,
weight, power needs, sterility, safety and FDA certification, all need to be
addressed to facilitate real-world deployment.

## ACKNOWLEDGMENTS

We would like to thank Nico Zevallos, Dr. Michael R. Pinsky, and Dr. Hernando
Gomez for gathering the data for our experiments, and Mononito Goswami for
thoughtful suggestions on this manuscript.

## References

  * [1] K. Verhoeff, R. Saybel, P. Mathura, B. Tsang, V. Fawcett, and S. Widder, “Ensuring adequate vascular access in patients with major trauma: a quality improvement initiative,” _BMJ Open Quality_ , vol. 7, no. 1, p. e000090, Jan 2018. [Online]. Available: https://qir.bmj.com/lookup/doi/10.1136/bmjoq-2017-000090
  * [2] C. Buckenmaier and P. F. Mahoney, _Vascular Access and Infusion Devices for Combat Anesthesia_. Office of the Surgeon General, United States Army, 2015, p. 63–73.
  * [3] B. B. Castro D, Martin Lee LAM, _Femoral Vein Central Venous Access_. ”https://www.ncbi.nlm.nih.gov/books/NBK459255/”: StatPearls [Internet], 2022.
  * [4] RMZarate. (2017) What is ultrasound? [Online]. Available: https://www.cypressdiagnosticimaging.com/what-is-ultrasound
  * [5] R. Goel, F. Abhimanyu, K. Patel, J. Galeotti, and H. Choset, “Autonomous ultrasound scanning using bayesian optimization and hybrid force control,” in _2022 International Conference on Robotics and Automation (ICRA)_. IEEE Press, 2022, p. 8396–8402. [Online]. Available: https://doi.org/10.1109/ICRA46639.2022.9812410
  * [6] O. Ronneberger, P. Fischer, and T. Brox, “U-net: Convolutional networks for biomedical image segmentation,” in _Medical Image Computing and Computer-Assisted Intervention – MICCAI 2015_ , N. Navab, J. Hornegger, W. M. Wells, and A. F. Frangi, Eds. Cham: Springer International Publishing, 2015, pp. 234–241.
  * [7] Q. Huang and Z. Zeng, “A review on real-time 3D ultrasound imaging technology,” _Biomed Res. Int._ , vol. 2017, p. 6027029, Mar. 2017.
  * [8] X. Chen, T. Wen, X. Li, W. Qin, D. Lan, W. Pan, and J. Gu, “Reconstruction of freehand 3D ultrasound based on kernel regression,” _Biomed Eng Online_ , vol. 13, p. 124, Aug 2014.
  * [9] F. Mohamed and C. V. Siang, “A survey on 3d ultrasound reconstruction techniques,” in _Artificial Intelligence_ , M. A. Aceves-Fernandez, Ed. Rijeka: IntechOpen, 2019, ch. 4. [Online]. Available: https://doi.org/10.5772/intechopen.81628
  * [10] Q. Huang, Y. Zheng, M. Lu, and Z. Chi, “Development of a portable 3d ultrasound imaging system for musculoskeletal tissues,” _Ultrasonics_ , vol. 43, no. 3, pp. 153–163, 2005. [Online]. Available: https://www.sciencedirect.com/science/article/pii/S0041624X04002185
  * [11] X. X. Yin, L. Sun, Y. Fu, R. Lu, and Y. Zhang, “U-Net-Based Medical Image Segmentation,” _J Healthc Eng_ , vol. 2022, p. 4189781, 2022.
  * [12] N. Siddique, S. Paheding, C. P. Elkin, and V. Devabhaktuni, “U-net and its variants for medical image segmentation: A review of theory and applications,” _IEEE Access_ , vol. 9, p. 82031–82057, 2021. [Online]. Available: http://dx.doi.org/10.1109/ACCESS.2021.3086020
  * [13] L. J. Brattain, T. T. Pierce, L. A. Gjesteby, M. R. Johnson, N. D. DeLosa, J. S. Werblin, J. F. Gupta, A. Ozturk, X. Wang, Q. Li, B. A. Telfer, and A. E. Samir, “AI-Enabled, Ultrasound-Guided Handheld Robotic Device for Femoral Vascular Access,” _Biosensors (Basel)_ , vol. 11, no. 12, Dec 2021.
  * [14] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei, “Imagenet: A large-scale hierarchical image database,” in _2009 IEEE Conference on Computer Vision and Pattern Recognition_ , 2009, pp. 248–255.
  * [15] E. Chen, H. Choset, and J. Galeotti, “Uncertainty-based adaptive data augmentation for ultrasound imaging anatomical variations,” in _2021 IEEE 18th International Symposium on Biomedical Imaging (ISBI)_ , 2021, pp. 438–442.
  * [16] B. Sekachev, N. Manovich, M. Zhiltsov, A. Zhavoronkov, D. Kalinin, B. Hoff, TOsmanov, D. Kruchinin, A. Zankevich, DmitriySidnev, M. Markelov, Johannes222, M. Chenuet, a andre, telenachos, A. Melnikov, J. Kim, L. Ilouz, N. Glazov, Priya4607, R. Tehrani, S. Jeong, V. Skubriev, S. Yonekura, vugia truong, zliang7, lizhming, and T. Truong, “opencv/cvat: v1.1.0,” Aug. 2020. [Online]. Available: https://doi.org/10.5281/zenodo.4009388
  * [17] Stanford Artificial Intelligence Laboratory et al., “Robotic operating system.” [Online]. Available: https://www.ros.org
  * [18] M. A. Fischler and R. C. Bolles, “Random sample consensus: A paradigm for model fitting with applications to image analysis and automated cartography,” _Commun. ACM_ , vol. 24, no. 6, p. 381–395, jun 1981. [Online]. Available: https://doi.org/10.1145/358669.358692
  * [19] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in _2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2016, pp. 770–778.
  * [20] A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga, A. Desmaison, A. Kopf, E. Yang, Z. DeVito, M. Raison, A. Tejani, S. Chilamkurthy, B. Steiner, L. Fang, J. Bai, and S. Chintala, “Pytorch: An imperative style, high-performance deep learning library,” in _Advances in Neural Information Processing Systems 32_. Curran Associates, Inc., 2019, pp. 8024–8035. [Online]. Available: http://papers.neurips.cc/paper/9015-pytorch-an-imperative-style-high-performance-deep-learning-library.pdf
  * [21] P. Iakubovskii, “Segmentation models pytorch,” 2019.
  * [22] J. Howard _et al._ , “fastai,” https://github.com/fastai/fastai, 2018\.
  * [23] D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” in _3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings_ , Y. Bengio and Y. LeCun, Eds., 2015. [Online]. Available: http://arxiv.org/abs/1412.6980
  * [24] E. J. Chun, “Ultrasonographic evaluation of complications related to transfemoral arterial procedures,” _Ultrasonography_ , vol. 37, no. 2, pp. 164–173, Apr. 2018.