# AudioEar: Single-View Ear Reconstruction for Personalized Spatial Audio

Xiaoyang Huang1, Yanjun Wang1, Yang Liu, Bingbing Ni1,
Wenjun Zhang1, Jinxian Liu1, Teng Li2∗ Corresponding Author: Teng Li, Bingbing
Ni

###### Abstract

Spatial audio, which focuses on immersive 3D sound rendering, is widely
applied in the acoustic industry. One of the key problems of current spatial
audio rendering methods is the lack of personalization based on different
anatomies of individuals, which is essential to produce accurate sound source
positions. In this work, we address this problem from an interdisciplinary
perspective. The rendering of spatial audio is strongly correlated with the 3D
shape of human bodies, particularly ears. To this end, we propose to achieve
personalized spatial audio by reconstructing 3D human ears with single-view
images. First, to benchmark the ear reconstruction task, we introduce
AudioEar3D, a high-quality 3D ear dataset consisting of 112 point cloud ear
scans with RGB images. To self-supervisedly train a reconstruction model, we
further collect a 2D ear dataset composed of 2,000 images, each one with
manual annotation of occlusion and 55 landmarks, named AudioEar2D. To our
knowledge, both datasets have the largest scale and best quality of their
kinds for public use. Further, we propose AudioEarM, a reconstruction method
guided by a depth estimation network that is trained on synthetic data, with
two loss functions tailored for ear data. Lastly, to fill the gap between the
vision and acoustics community, we develop a pipeline to integrate the
reconstructed ear mesh with an off-the-shelf 3D human body and simulate a
personalized Head-Related Transfer Function (HRTF), which is the core of
spatial audio rendering. Code and data are publicly
available111https://github.com/seanywang0408/AudioEar.

## Introduction

Spatial audio is widely applied in virtual reality, gaming, and movie
production (Begault and Trejo), for distinguishing sound source positions and
generating immersive 3D sound. Without it, people would lose the spatial sense
of sound. The rendering of spatial audio depends on the Head-Related Transfer
Function (HRTF) (Elliott, Jung, and Cheer; Blauert, Allen, and Press). HRTF is
adopted in mid-to-high-end audio equipment, such as stereos, headphones, Hi-
Fi, and so on. HRTF varies from one person to another since it depends on the
3D structure of human bodies including ears, head, and torso. While a
personalized HRTF could be approximated by acoustic simulation (Pierce) given
a complete 3D human body structure, it is laborious to model human bodies
using 3D scanners. Single-view reconstruction is an alternative solution. It
is well studied to reconstruct personalized human heads and torsos within
single-view images. However, without the modeling of ear structure, which is
the central organ in the human hearing system, the simulated HRTF is still
biased. In this work, we propose to obtain personalized HRTF by reconstructing
the 3D mesh (Huang et al.) of human ears within a single-view image.
222Strictly speaking, the ear structure include outer ear, middle ear, and
inner ear. Our work only focuses on the visible structure (pinna) in the outer
ear. Yet we still use the more understandable name ear in the paper, following
prior works (Jin et al.).

To benchmark the task of ear reconstruction, we collect a high-quality 3D ear
dataset with an advanced structured-light 3D scanner, named AudioEar3D.
Compared to prior evaluation protocol that uses 2D ear landmark re-projection
error as the metric, a 3D benchmark is more precise. AudioEar3D includes $112$
ear point cloud scans with RGB images from $56$ individuals, each scan with
$100,000$ to $250,000$ points, which capture high-resolution shape
characteristics of ears. To the best of our knowledge, this is the largest and
most accurate 3D ear dataset that is publicly available, as analyzed in
Section Ear Datasets in 3D and 2D. We believe that except for spatial audio,
this dataset could also contribute to other applications, including digital
humans (Kappel et al.), morphology (Krishan, Kanchan, and Thakur) and medical
surgery research (Varman et al.).

Prior reconstruction methods for other parts of human body train on large-
scale 2D datasets self-supervisedly (Zhang et al.; Chen et al.). However,
existing 2D ear datasets either lack semantic annotations or suffer from a
small scale. This situation greatly limits the extension of CV studies on
human ears. To this end, we build a large-scale 2D ear images dataset,
including $2,000$ high-resolution ear images. They are selected from the FFHQ
human face dataset, each of which accompanies manual annotations of occlusion
and $55$ ear landmarks. The landmarks annotations greatly enrich the semantics
of the ear images, enabling their usage in extensive applications.

Based on the self-supervised reconstruction pipeline, we proposed AudioEarM, a
depth-guided reconstruction method tailored for ears. Due to the lack of
public ear texture, which hinders self-supervised training, we first build an
ear texture space from DECA (Feng et al.) for the UHM Ear (Ploumpis et al.),
the ear shape model we use in our work. We excavate UV coordinate
correspondence between UHM Ear vertices and the texture of DECA via K-Nearest-
Neighbors searching and distance-based weighting. Besides, motivated by the
characteristics of ear data, we design two loss functions: (i) contour loss
which better accommodates the annotation error than landmark loss; (ii)
similarity loss which encourages the network to predict distinct shapes for
different samples. Moreover, we introduce synthetic data to improve
reconstruction quality. Instead of directly combining synthetic data with real
data and using 3D supervision to train the model, we use the synthetic data to
train a monocular depth model to extract depth feature from ear images. Then
we inject the depth information into the main network to guide the whole
reconstruction process.

Lastly, to fill the gap between ear reconstruction and acoustic applications,
we develop a pipeline to integrate the reconstructed ear into a 3D mesh human
torso for acoustic simulation. We propose approximated Delaunay triangulation
on the 3D vertices to ensure the simulation validity of the integrated body
mesh. We obtain the personalized HRTF with the reconstructed ear via
simulation, and further demonstrate the effectiveness of the reconstruction
model over the baseline by comparison on HRTF.

## Related Work

### Spatial Audio

Spatial audio, or spatial hearing, is a long-standing subject in acoustics
(Culling and Akeroyd; Vorländer). It focuses on how humans localize the
position of acoustic sources and how to reproduce the spatial sense with a
sound system. Humans locate the sound sources by biaural and monaural cues.
Biaural cues include interaural time difference (ITD) and interaural level
difference (ILD), which means that the arriving sound signals are filtered by
the diffraction and reflection of the human body, including ear (pinna), head,
and torso, leading to a difference of time, and intensity of arrival. Monaural
cue is the spectral distortion of sounds. Further, the 3D shape of the human
body varies between individuals, resulting in different binaural and monaural
cues between different people. To model different spatial hearing
characteristics between people, the Head-Related Transfer Function (HRTF) (Li
and Peissig; Xie, Zhong, and He) is proposed and widely used in spatial sound
rendering. HRTF describes the Sound Pressure Level (SPL, dB) of a sound source
in each direction for a specific individual. It is used to convert an
arbitrary sound to a specific position as if the sound is originated from
there. However, obtaining an accurate personalized HRTF is laborious since the
measurement requires expensive equipment and special acoustic laboratories.
Current implementations tend to deploy an average HRTF or choose one from an
HRTF database (Guo et al.). Besides physical measurement, HRTF could also be
numerically simulated (Conrad; Jensen) based on Boundary Element Method (BEM).
Meshram et al. proposed to simulate a personalized HRTF by reconstructing the
3D human body using multi-view stereo (MVS). However, the reconstructed body
is heavily blurred due to the imperfection of the reconstruction method,
yielding large HRTF error.

### Ear Datasets in 3D and 2D

Table 1: AudioEar3D and its counterparts. Quality is assessed based on the
precision of acquisition equipments.

3D Ear Dataset | Scale | with Image | Quality | Accessibility
---|---|---|---|---
UND-J2 | $1,800$ | ✓ | $\star$ | ✓
York3DEar | $500$ | ✗ | $\star$ | ✓
SYMARE-1 | $20$ | ✗ | $\star\star\star$ | ✓
SYMARE-2 | $102$ | ✗ | $\star\star\star$ | ✗
Ploumpis et al. | $234$ | ✗ | $\star\star\star$ | ✗
AudioEar3D | $112$ | ✓ | $\star\star\star~{}\star$ | ✓

Table 2: Comparisons of AudioEar2D and its counterparts. Lmks. denotes
landmark annotation. Occ. means annotations of whether the ear is partially
occluded by hair or earrings.

2D Ear Dataset | Scale | Source | Annotations | Usage
---|---|---|---|---
Lmks. | Occ.
UND-E | $464$ | Limited | ✗ | ✗ | Biometrics
AMI | $700$ | Limited | ✗ | ✗ | Biometrics
IIT Delhi Ear | $754$ | Limited | ✗ | ✗ | Biometrics
WPUTEDB | $3,348$ | Limited | ✗ | ✗ | Biometrics
UBEAR | $4,410$ | Limited | ✗ | ✗ | Biometrics
IBug-B | $2,058$ | In-the-wild | ✗ | ✗ | Biometrics
AWE | $9,500$ | In-the-wild | ✗ | ✗ | Biometrics
EarVN | $28,412$ | In-the-wild | ✗ | ✗ | Biometrics
IBug-A | $605$ | In-the-wild | ✓ | ✗ | Reconstruction
AudioEar2D | $2,000$ | In-the-wild | ✓ | ✓ | Reconstruction

Previous CV studies on human ears are mostly confined to biometric application
(O’Sullivan and Zafeiriou), and do not prevail as other parts of human body,
such as faces, hands and skeletons. This leads to the lack of a rich ear
dataset in 3D and 2D, as stated in Table 2, 2.

Existing 3D ear datasets suffer from either non-accessibility, small scale, or
low quality. Yan and Bowyer collect $1,800$ ear depth maps in a resolution of
$640\times 480$ using a depth sensor. Since the depth maps are single-view,
they do not represent complete ear shapes. Dai, Pears, and Smith publish
York3DEar, which is composed of $500$ deformed 3D ear meshes. These meshes are
not collected with instruments, but are estimated by a data-augmenting
technique based on a 2D ear dataset, which introduces unforeseeable error. Jin
et al. propose the SYMARE ear database consisting of the measured HRIR and the
upper torso, head and ear mesh model collected by magnetic resonance imaging
(MRI) from $61$ individuals. However, only $10$ (SYMARE-1) individuals’ data
are accessible, while the rest (SYMARE-2) are kept private. Ploumpis et al.
collect $121$ ear models from $64$ adults and $133$ ears models from children
via CT scans. However, these data are not publicly available. Besides, most of
the above 3D ear datasets lack the corresponding ear images (except for
UND-J2), making it unfeasible to perform single-view reconstruction tasks upon
them. Moreover, most of these data are collected by MRI, whose measurement
error is generally about $1$ millimeter (Nowogrodzki). This measurement error
is not negligible for such an elaborately-structured organ, thereby
introducing additional error in the simulation of HRTF.

Existing 2D ear datasets include UND-E (Chang et al.), AMI (Gonzalez, Alvarez,
and Mazorra), IIT Delhi Ear (Kumar and Wu), WPUTEDB (Frejlichowski and
Tyszkiewicz), UBEAR (Raposo et al.), IBug(-A/B) (Zhou and Zaferiou), AWE
(Emeršič et al.) and EarVN (Hoang), most of which are for human identity
recognition. The summarization of these datasets is shown in Table 2.

### 3D Morphable Models Reconstruction

A 3D Morphable Model (3DMM) is a parametric model that encodes the shape and
texture of 3D meshes into a latent space. The most studied structure in 3DMM
reconstruction are human faces (Ploumpis et al.; Ploumpis et al.), hands (Chen
et al.; Wang et al.; Zhang et al.) and bodies (Choi, Moon, and Lee; Corona et
al.). To achieve plausible performance, the 3DMM reconstruction algorithms
require a large set of 2D images for self-supervised training, usually
accompanied by semantic annotations, such as landmarks or poses. They use a
feature extractor and a multi-layer perceptron (MLP) to regress the latent
code of shape and texture. The latent codes are fed into the differentiable
3DMM models to obtain colored 3D meshes. Then the meshes are projected to
images with a differentiable renderer (Kato, Ushiku, and Harada; Huang et
al.). The photometric loss on images and the landmark re-projection loss are
minimized to train the network. The popular face reconstruction algorithm DECA
(Feng et al.) proposes to conduct robust detail reconstruction by regressing a
UV displacement map. Sun, Pears, and Dai propose to reconstruct 3D ear meshes
with the YEM model (Dai, Pears, and Smith). They derived a color model from
images and minimized the photometric and landmark error to regress shape
latent codes.

## Method

### Data collection

We collect two ear datasets for personalized spatial audio. One is AudioEar3D,
a 3D ear scan dataset, for benchmarking the ear reconstruction task. The other
is AudioEar2D, for the training of ear reconstruction models.

#### AudioEar3D

Among various 3D scanning equipments, we choose a structured-light 3D scanner
to collect our 3D ear data, for the following reasons: (i) Compared to
previous 3D ear scanning methods, such as MRI and CT, a structured-light
scanner is much more convenient and accessible (it can be integrated into a
small portable device), which makes it easy to cover a larger population; (ii)
The operation and post-processing are simpler; (iii) The resolution of
structured-light scanners is higher compared to others, under the same price.
Specifically, we use MantisVision® F6-SR 333Product introduction:
https://mantis-vision.com/handheld-3d-scanners/, a portable high-resolution 3d
scanner, which is designed to be able to scan detailed organ models in medical
applications. Its plane resolution and depth resolution are $0.1$ mm and $0.4$
mm respectively (compared to a resolution of $1$ mm in most MRI scanners),
with a frame rate of 8 FPS.

To alleviate the self-occlusion issue brought by the complex ear structure and
obtain a clear and complete 3D ear scan, we scan each individual with the
above device for about $90$ seconds from different directions. We remove the
other parts of the body and only keep the ear part. This yields approximately
$100,000$ to $250,000$ points for each ear scan, which captures high-
resolution shape characteristics of ears. Besides, we take frontal pictures of
the ears from a distance similar to the scanner. At present we have obtained
data of $112$ ears from $56$ subjects. We plan to further extend the scale to
cover a larger population. We show several samples in Figure 1. The out-coming
dataset is completely anonymized for privacy issues.

Figure 1: Visualization of the two collected datasets. AudioEar3D (top): We
show five samples with their three-view rendered point clouds and RGB images.
Compared to existing 3D ear dataset, AudioEar3D is high-quality, large-scale,
and also publicly available. AudioEar2D (bottom): We illustrated several
samples with their occlusion annotations and the 55 landmarks. The four colors
of the landmarks indicate four main contours of the ears. We flip the left
ears to the right for better visualization.

#### AudioEar2D

We obtain our 2D ear dataset from a high-quality public human face dataset,
Flickr-Faces-HQ Dataset (FFHQ) (Karras, Laine, and Aila), which consists of
$70,000$ face images in the resolution of $1,024\times 1,024$ that are various
in terms of pose, age, and ethnicity. To find images that contain clear ears,
we first train an ear detection model on the IBug Collection-B dataset (Zhou
and Zaferiou), which contains the bounding box of ears, from a pre-trained
Yolov4 (Bochkovskiy, Wang, and Liao). Then we adopt the trained detection
model on the FFHQ dataset to roughly sift out high-quality ear images
according to detection confidence and bounding box area. Besides, we leverage
WHENet (Zhou and Gregson), a pre-trained head pose estimation model, to
further remove those images that have bad view angles. We cut the remaining
images into squares based on the bounding box. We preserve the original
resolution of the cut ear image to avoid information loss induced by
interpolation resizing. Thanks to the bounding box area filtering, the image
resolution is mostly above $300\times 300$, as illustrated in Figure 1 bottom
(a). Next, we manually filter the remaining images and annotate the $55$
landmarks, following a landmark protocol in IBug Ear (Zhou and Zaferiou).
Moreover, we annotate whether the ear is clean or is occluded by hair or by
earrings, which could act as noise in some applications. The distribution of
occlusion is shown in Figure 1 bottom (b). We obtain $2,000$ high-resolution
ear images, each with $55$ annotated landmarks in the end.

### AudioEarM: Depth-Guided Ear Reconstruction

We develop our method upon a common pipeline of parametric shape
reconstruction, which regresses the latent code of the 3D shape, texture,
camera, and lighting via self-supervised training. We first adapt this
pipeline to ears with several effective modifications, which are motivated by
the specific characteristic of ears. Besides, to further boost the
performance, we propose to leverage synthetic ear images to train a depth
estimation model and fuse the depth information into the reconstruction model
via multi-scale feature alignment. We name our method AudioEarM.

#### Texture Acquisition

We use the UHM Ear (Ploumpis et al.), the best-quality ear 3DMM to transform
the shape latent code into a 3D mesh. The UHM Ear reduces the N vertices
($2800$) of an ear mesh to a lower dimension ($236$) using PCA. Let
$\overline{\mathbf{S}}\in\mathbb{R}^{3N}$ denotes the mean shape,
$\vec{v}\in\mathbb{R}^{236}$ and $\mathcal{U}\in\mathbb{R}^{3N\times 236}$
denote the eigenvalues and eigenvectors of the 3DMM. The shape variations are
modeled by eigenvalue-weighted linear blendshapes:
$B_{\mathcal{U}}(\vec{\beta};\mathcal{U})=\sum_{n=1}^{236}v_{n}\beta_{n}\mathcal{U}_{n}$,
where $\vec{\beta}\in\mathbb{R}^{236}$ is the predicted shape latent code. The
final ear mesh is obtained by:

$M(\vec{\beta})=\sum_{n=1}^{236}v_{n}\beta_{n}\mathcal{U}_{n}+\overline{\mathbf{S}}$
(1)

The UHM Ear does not provide texture, which is indispensable to minimize the
photometric loss in a self-supervised reconstruction pipeline. We extract the
ear texture space from DECA (Feng et al.) to enable colored rendering. In
DECA, given a texture latent code
$\vec{\theta}\in\mathbb{R}^{|\vec{\theta}|}$, a texture map
$T(\boldsymbol{\theta})\in\mathbb{R}^{h\times w\times 3}$ ($h\times w$ is the
resolution) could be calculated. Each vertex in the DECA mesh corresponds to a
UV coordinate ${p}_{deca}$ in the texture map. However, the vertices of the
UHM ear do not have correspondence with DECA. To this end, we first visually
align the mean UHM ear with the ear in DECA as close as possible. For each
vertex $v_{uhm}\in\mathbb{R}^{3}$ in UHM model, we then search its K-Nearest-
Neighbors $v_{deca}^{i}\in\mathbb{R}^{3},i=1,2\dots,k$ in DECA vertices (k is
$3$ in our implementation). We assign the KNN-distance-weighted UV coordinates
to the UHM vertices based on Euclidean distance between $v_{uhm}$ and
$v_{deca}^{i}$. Formally, the UV coordinates of $v_{uhm}$ are computed by:

${p}_{uhm}=\sum_{i=1}^{k}\frac{D(v_{uhm},v_{deca}^{i})}{\sum_{j=1}^{k}D(v_{uhm},v_{deca}^{j})}{p}_{deca}^{i}$
(2)

##### Contour Loss.

Contour loss is concerned with the four contours formed by the 2D landmarks.
(The four contours are indicated by four different landmark colors in
AudioEar2D in Figure 1.) As illustrated in Figure 1 bottom (c), the landmark
annotation have bias along the contour, while the contour is well represented
by the annotations. The reason is that the contour is more visually salient
for human eyes. Hence measuring the error of contour prediction in training is
superior to landmark error. Practically, we connect the landmarks one by one
to form four polylines and uniformly re-sample dense points on these
polylines. We measure the chamfer distance of the re-sampled points
$\mathcal{P}$ between the ground truth and prediction as contour loss:

$L_{contour}=\frac{1}{4}\sum_{i=1}^{4}ChamferDistance(\mathcal{P}_{gt}^{i},\mathcal{P}_{pred}^{i})$
(3)

where the superscript $i$ is the index of the four contours.

Figure 2: The architecture of our depth-guided reconstruction model,
AudioEarM. A depth estimation model, which is trained on a synthetic dataset,
guides the main reconstruction network with multi-scale feature alignment.

##### Similarity Loss.

In our early experiments, we find that the predicted shape latent codes are
too similar across all instances. This similarity is undesirable since the
shape should be distinct across different individuals. We encourage the
network to predict distinct shapes by measuring the mean cosine similarity
between the shape latent codes in a batch and penalizing high similarity:

$L_{sim}=\frac{1}{bs\times(bs-1)}\sum_{i,j=0,i\neq
j}^{bs-1}\frac{\beta_{i}\beta_{j}}{|\beta_{i}||\beta_{j}|}$ (4)

where $bs$ denotes the batch size and $\beta_{i},\beta_{j}$ denote the $i$th,
$j$th sample in a batch.

Besides, we prevent extreme rotation angles and scale parameters in camera
with $L1$-penalization, since the angle and scale of ears in images are
generally within a certain scope. We combine these three loss functions with
the commonly used losses: landmark, photometric loss, mesh smooth loss and
latent code regularization. The overall loss is the weighted sum of the losses
above. See appendix for a complete formulation of all loss functions.

#### Depth-Guided Reconstruction

Zhang et al. demonstrates that introducing synthetic data with 3D supervision
into trainset could improve the quality of single-view hand reconstruction.
Yet we empirically find that directly migrating this approach to our task
brings negative effect (see ablation study). Alternatively, we leverage the
power of pre-training (Yang et al.), that is to use synthetic data to pre-
train a monocular depth estimation (MDE) model. Then we leverage the depth
feature extracted by the MDE model to facilitate the whole reconstruction
process, as shown in Figure 2.

To generate a synthetic ear dataset with 3D supervision, we randomly sample
shape and texture latent codes from a Gaussian distribution, and render
corresponding RGB images and depth maps. We generate $10,000$ samples in this
way. Then we train an MDE model on this synthetic dataset, using the depth map
as direct supervision. The MDE model (Laina et al.) consists of an encoder and
a decoder (Figure 2). The ResNet-34 (He et al.) encoder extracts high-level
feature from images, while the decoder predicts a depth map from the feature.
The extracted feature is upsampled by the decoder layer by layer until the
original resolution is restored. We aim to leverage the depth information
obtained by decoder to guide the reconstruction process. To this end, we
concatenate the multi-scale upsampled feature with the feature before each
layer in the main network. We insert an additional convolution layer each time
after concatenation to transform the increased channel numbers into original
ones, so as to match the channel of the next layer. Since the last-layer
resolution of the MDE model is twice as big as the first-layer resolution in
the main network, we conduct max-pooling on the last-layer MDE feature before
concatenation. In this way, we explicitly fuse the depth information into the
main network to guide the reconstruction process. Ablation study shows that
this method brings notable improvement.

Figure 3: The proposed integration and simulation pipeline. We conduct
approximated Delaunay triangulation to combine the reconstructed ear with the
body and simulate the personalized HRTF in all azimuth angles ($361$ angles)
on the horizontal plane.

### Acoustic Simulation

Spatial audio rendering depends on the HRTF, which is defined as the Sound
Pressure Level (SPL) measured at the eardrum (or the ear canal entrance in
practice). We simulate a personalized HRTF with COMSOLTM, a Multiphysics
software that uses the Boundary Element Method (BEM) for simulation. However,
the simulation requires a complete human upper body, not just an ear. To this
end, we develop a pipeline to integrate our reconstructed ears into an off-
the-shelf human body 3D model (Braren and Fels; Braren and Fels). The pipeline
is illustrated in Figure 3.

We manually remove the original ears of the given 3D body and place the
reconstructed ear to the consequent hole. To combine the ear and body mesh
together, we propose a method named approximated Delaunay triangulation.
Delaunay triangulation is a triangulation method for 2D point sets, which
maximize the minimal angle of all the angles of the triangles in the
triangulation process, such that sliver triangles are avoided. Sliver
triangles are inappropriate for BEM simulation. Since both the edges of the
ear and body hole are not on a 2D plane (but close to a plane), we project the
edges to a plane which is found by the averaged normals of the mesh faces
around the edges. In this way, we can conduct approximated Delaunay
triangulation on the vertices of the two edges and fix the crack between ears
and heads. Since the geometry of the edges is not greatly changed by the
projections, the sliver triangles could still be avoided as much as possible.
We leverage the Detri2 program 444http://www.wias-
berlin.de/people/si/detri2.html for Delaunay triangulation. At last, we smooth
the generated mesh and use it for simulation. We simulate the HRTF in all
azimuth angles ($361$ angles in total) on the horizontal plane for three
frequencies (f = 1033.6 Hz, 2067.5 Hz, and 3962.1 Hz), following a general
protocol. Note that the simulation pipeline involves manual efforts and is not
part of the evaluation protocol of ear reconstruction, but for application
use. We evaluate reconstruction performance on AudioEar3D.

## Experiments

### Ear Reconstruction Benchmark

#### Evaluation

We leverage the full AudioEar3D dataset to evaluate the performance of ear
reconstruction models. We compute the distance from each point in the ground-
truth ear scan to the predicted ear mesh. (The distance from a point to a mesh
is defined as the distance from the point to the closest triangular face in
the mesh.) We average the distance across all points in the scan as the
evaluation metric, named scan2mesh (S2M). However, since the scale and
position are not aligned between the scans and the predicted meshes, we first
register the mesh with the scan in a gradient-based method. We iteratively
optimize scale and position parameters for registration. To reduce the
evaluation time and avoid convergence in a local minimum, we design a three-
stage registration scheme. First, we minimize four manually chosen key points
to obtain a coarse registration. Next, we randomly sample points on the
predicted mesh surfaces and minimize the chamfer distance between the sampled
points and the scans. Last, we directly minimize the S2M distance between the
scan and mesh. Since the dense points clouds in AudioEar3D would cause intense
computation, we randomly down-sample the scan to $1,000$ points for the S2M
and only compute the S2M distance on the whole scan in the last iteration. We
use Adam (Kingma and Ba) with an initial learning rate of $0.45$ for
optimization. Each stage lasts for $166$ iterations. The learning rate is
multiplied by $0.1$ when entering the next stage. After numerous tests, we
empirically find that this setting can make most registrations converge to a
global minimum. We conduct the registration process on all the $112$ samples
in AudioEar3D, which takes about half an hour, and average the S2M across all
samples as the final evaluation metric. The left ear scans are transformed to
the right ones by reflection in the sagittal plane.

#### Setting

To compare, we send the average ear of UHM into the registration process, as a
baseline without any personalization. We also compare our method with a prior
ear reconstruction method named HERA (Sun, Pears, and Dai). Since HERA did not
publish their code and ear model, we implement it using our ear model.
Besides, we would like to compare AudioEarM with the SOTA face reconstruction
algorithm, DECA (Feng et al.). Since a detailed UV displacement map in DECA is
not available for ears, we only implement the coarse branch in DECA, denoted
as DECA-coarse. We describe the training configurations in the appendix.

#### Results

Figure 4: We visualize the S2M error map of our method and two baselines on
testset (AudioEar3D). Our method has lower error compared to baselines.

As listed in Table 3 #6-9, our method surpasses the average ear baseline, HERA
and DECA-coarse, with a final S2M distance of $1.28$ mm. The average ear
yields an S2M of $1.78$ mm, while HERA and DECA-coarse yields an S2M of $1.70$
and $1.46$ mm. We visualize the S2M error map of our method and two baselines
on testset (AudioEar3D). Our method has lower error compared to baselines.

Table 3: Comparisons of reconstruction methods. {#1 + texture} means texture is added on top of Method#1. Similar meanings for other $+$,$-$ notations. # | Model | S2M $\downarrow$
---|---|---
1 | Naive | 1.73mm
2 | #1 $+$ texture | 1.65mm
3 | #2 $+$ similarity loss | 1.63mm
4 | #2 $+$ contour loss | 1.52mm
5 | #2 $+$ contour & similarity | 1.47mm
6 |  Ours (#5 $+$ depth-guided) | 1.28mm
7 | Avg ear | 1.78mm
8 | HERA | 1.70mm
9 | DECA-coarse | 1.46mm

### Comparisons of Reconstruction Models

We examine the impact of each part in our method (Table 3 #1-6). All the
models are trained with the same setting described in Section Ear
Reconstruction Benchmark. We first implement a baseline model (Method#1) in
which a common self-supervised reconstruction pipeline is adopted to our data
without modifications. A single skin color is assigned to the texture instead
of our texture space. Method#1 yields an S2M of $1.73$ mm, which has little
improvement over the average ear. Then we replace the single color with our
texture space (Method#2), which reduces the S2M error to $1.65$ mm. On top of
it, we add the similarity loss and contour loss into the training process
(Method#3-5). Adding the two losses together yields a result of $1.47$ mm. The
effect of contour loss is more notable than the similarity loss. Finally, we
use the depth estimation model to perform depth-guide reconstruction, and
obtain an S2M of $1.28$ mm, which is our best result. The reason behind the
superiority is that predicting a regular depth map is less ill-posed than
predicting a shape latent code. Besides, depth could inform the object’s
geometry. Similar motivation is enlightening in pseudo-lidar approaches for 3D
object detection (Qian et al.).

Table 4: Results of HRTF simulation measured in SPL error. The first and second row in each sample denote Avg and Pred results respectively. We see that better reconstruction results bring more realistic HRTF. The predicted ears in the first 5 samples have lower S2M and also lower SPL error than average, vice versa andfor the last 2 samples. Sample | S2M $\downarrow$ | SPL Error $\downarrow$ (dB$\times$ 10)
---|---|---
(mm) | f=1kHz | f=2kHz | f=4kHz
033-right | 1.49 | 1.12$\pm$0.75 | 3.59$\pm$5.97 | 4.13$\pm$5.44
0.94 | 0.33$\pm$0.19 | 0.59$\pm$0.89 | 2.32$\pm$2.84
021-left | 1.66 | 0.58$\pm$0.27 | 1.20$\pm$2.19 | 4.25$\pm$3.43
0.88 | 0.30$\pm$0.12 | 1.30$\pm$1.47 | 2.39$\pm$1.85
036-left | 2.23 | 1.23$\pm$1.17 | 3.52$\pm$9.16 | 6.97$\pm$11.97
1.85 | 1.13$\pm$0.89 | 2.69$\pm$4.58 | 4.75$\pm$5.89
008-left | 1.23 | 1.69$\pm$1.31 | 4.50$\pm$6.71 | 5.29$\pm$5.40
0.74 | 1.35$\pm$0.69 | 3.532$\pm$5.97 | 6.75$\pm$6.49
048-left | 1.35 | 0.40$\pm$0.30 | 1.27$\pm$1.64 | 5.00$\pm$4.76
0.94 | 0.23$\pm$0.14 | 1.18$\pm$1.23 | 3.31$\pm$2.88
040-right | 1.27 | 1.00$\pm$0.66 | 2.83$\pm$5.65 | 5.54$\pm$10.23
1.58 | 1.17$\pm$0.81 | 3.57$\pm$6.54 | 5.99$\pm$10.43
029-left | 1.33 | 2.11$\pm$0.98 | 5.95$\pm$9.76 | 8.06$\pm$9.47
1.46 | 1.12$\pm$0.76 | 3.44$\pm$5.98 | 10.09$\pm$10.74

## HRTF Simulation

#### Evaluation.

In this part, we conduct the HRTF simulation using our predicted ears. We
compare the HRTF of the predicted ears against the ground-truth ears. The HRTF
of the ground-truth ears are simulated using the ear meshes registered from
the raw ear scans in AudioEar3D. Specifically, we send the mean UHM Ear into
the same registration process described in Section Ear Reconstruction
Benchmark, except that the optimized parameters in the second and third stages
include the shape latent code of UHM besides the original ones. The
registration yields an average S2M of $0.11$ mm among all scans, which is
fairly low. The low S2M error indicates the optimized meshes are rather close
to the true ear shape. We measure the mean SPL error in absolute value across
all angles between ground-truth and predicted ones. As comparisons, we replace
the predicted ears with the mean UHM Ear (denoted as Avg in Table 4). The SPL
error of the mean UHM Ear represents the spatial audio effect without
personalization. Larger errors indicate worse spatial audio. Since the
simulation experiment is labour-intense and time-consuming, we only show the
results of several samples for demonstration.

#### Results.

Table 4 compares the results between Avg and Pred of each sample. The first
and second row in each sample denote Avg and Pred results respectively. We see
that better reconstruction results (lower S2M) bring more realistic HRTF
(lower SPL error). For those samples whose reconstructed S2M distances are
lower than the mean UHM Ear (the first $5$ samples), both the mean value and
variation of SPL error are also lower, meaning that they obtain a closer HRTF
to the ground truth than the mean UHM Ear. For the last two samples, the S2M
of predicted ears is higher and the SPL error is somewhat larger, indicating
that the obtained HRTF is highly related to the reconstruction results. It is
necessary to develop advanced ear reconstruction algorithms for more realistic
spatial sound effects in the coming VR trend. For a better understanding on
the SPL error, we illustrate it with polar coordinates in the appendix.

## Conclusion

This work considers 3D ear reconstruction from single-view images for
personalized spatial audio. For this purpose, we collect a 3D ear dataset for
benchmarking and a 2D dataset for the training of ear reconstruction models.
Besides, we propose a reconstruction method guided by a depth estimation
network that is trained on synthetic data, with two loss functions tailored
for ear data. Lastly, we develop a pipeline to integrate the reconstructed ear
mesh with a human body for acoustic simulation to obtain personalized spatial
audio.

## Acknowledgement

This work was supported by National Science Foundation of China (U20B2072,
61976137). This work was also partially supported by Grant YG2021ZD18 from
Shanghai Jiao Tong University Medical Engineering Cross Research.

## References

  * Begault and Trejo (2000) Begault, D. R.; and Trejo, L. J. 2000.  3-D sound for virtual reality and multimedia.  _No. NASA/TM-2000-209606_.
  * Blauert, Allen, and Press (1997) Blauert, J.; Allen, J.; and Press, M. 1997.  _Spatial Hearing: The Psychophysics of Human Sound Localization_.  MIT Press.  ISBN 9780262024136.
  * Bochkovskiy, Wang, and Liao (2020) Bochkovskiy, A.; Wang, C.-Y.; and Liao, H.-Y. M. 2020.  Yolov4: Optimal speed and accuracy of object detection.  _arXiv preprint arXiv:2004.10934_.
  * Braren and Fels (2020a) Braren, H.; and Fels, J. 2020a.  A High-Resolution Individual 3D Adult Head and Torso Model for HRTF Simulation and Validation: 3D Data.  _Medical Acoustics Group, Institute of Technical Acoustics, RWTH Aachen_.
  * Braren and Fels (2020b) Braren, H.; and Fels, J. 2020b.  A High-Resolution Individual 3D Adult Head and Torso Model for HRTF Simulation and Validation: HRTF Measurement.  _Medical Acoustics Group, Institute of Technical Acoustics, RWTH Aachen_.
  * Chang et al. (2003) Chang, K.; Bowyer, K. W.; Sarkar, S.; and Victor, B. 2003.  Comparison and combination of ear and face images in appearance-based biometrics.  _IEEE Transactions on pattern analysis and machine intelligence_ , 25(9): 1160–1165.
  * Chen et al. (2019) Chen, W.; Ling, H.; Gao, J.; Smith, E.; Lehtinen, J.; Jacobson, A.; and Fidler, S. 2019.  Learning to predict 3d objects with an interpolation-based differentiable renderer.  _Advances in Neural Information Processing Systems_ , 32: 9609–9619.
  * Chen et al. (2021) Chen, Y.; Tu, Z.; Kang, D.; Bao, L.; Zhang, Y.; Zhe, X.; Chen, R.; and Yuan, J. 2021\.  Model-based 3D hand reconstruction via self-supervised learning.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 10451–10460.
  * Choi, Moon, and Lee (2020) Choi, H.; Moon, G.; and Lee, K. M. 2020.  Pose2Mesh: Graph convolutional network for 3D human pose and mesh recovery from a 2D human pose.  In _European Conference on Computer Vision_ , 769–787. Springer.
  * Conrad (2018) Conrad, Z. 2018.  Hats Off to the Boundary Element Method.  _IEEE Spectrum Multiphysics Simulation_ , 30.
  * Corona et al. (2021) Corona, E.; Pumarola, A.; Alenya, G.; Pons-Moll, G.; and Moreno-Noguer, F. 2021\.  SMPLicit: Topology-aware generative model for clothed people.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 11875–11885.
  * Culling and Akeroyd (2012) Culling, J.; and Akeroyd, M. 2012.  Spatial hearing.  _Oxford Handbook of Auditory Science Hearing_.
  * Dai, Pears, and Smith (2018) Dai, H.; Pears, N.; and Smith, W. 2018.  A data-augmented 3D morphable model of the ear.  In _2018 13th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2018)_, 404–408. IEEE.
  * Desbrun et al. (1999) Desbrun, M.; Meyer, M.; Schröder, P.; and Barr, A. H. 1999.  Implicit fairing of irregular meshes using diffusion and curvature flow.  In _Proceedings of the 26th annual conference on Computer graphics and interactive techniques_ , 317–324.
  * Elliott, Jung, and Cheer (2018) Elliott, S. J.; Jung, W.; and Cheer, J. 2018.  Head tracking extends local active control of broadband sound to higher frequencies.  _Scientific reports_ , 8(1): 1–7.
  * Emeršič et al. (2017) Emeršič, Ž.; Štepec, D.; Štruc, V.; Peer, P.; George, A.; Ahmad, A.; Omar, E.; Boult, T. E.; Safdaii, R.; Zhou, Y.; et al. 2017\.  The unconstrained ear recognition challenge.  In _2017 IEEE international joint conference on biometrics (IJCB)_ , 715–724. IEEE.
  * Feng et al. (2021) Feng, Y.; Feng, H.; Black, M. J.; and Bolkart, T. 2021.  Learning an animatable detailed 3D face model from in-the-wild images.  _ACM Transactions on Graphics (TOG)_ , 40(4): 1–13.
  * Frejlichowski and Tyszkiewicz (2010) Frejlichowski, D.; and Tyszkiewicz, N. 2010.  The west pomeranian university of technology ear database–a tool for testing biometric algorithms.  In _International Conference Image Analysis and Recognition_ , 227–234. Springer.
  * (19) Gonzalez, E.; Alvarez, L.; and Mazorra, L. ????  AMI Ear Database.  https://ctim.ulpgc.es/research˙works/ami˙ear˙database/.
  * Guo et al. (2020) Guo, X.; Bai, Y.; Liu, Q.; Xiong, D.; Wang, Y.; and Du, J. 2020.  3D Head Anthropometry for Head-Related Transfer Function of Chinese Pilots.  In _International Conference on Man-Machine-Environment System Engineering_ , 171–178. Springer.
  * He et al. (2016) He, K.; Zhang, X.; Ren, S.; and Sun, J. 2016.  Deep residual learning for image recognition.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 770–778.
  * Hoang (2019) Hoang, V. T. 2019.  EarVN1. 0: A new large-scale ear images dataset in the wild.  _Data in brief_ , 27.
  * Huang et al. (2022a) Huang, X.; Yang, J.; Wang, Y.; Chen, Z.; Li, L.; Li, T.; Ni, B.; and Zhang, W. 2022a.  Representation-Agnostic Shape Fields.  In _International Conference on Learning Representations_.
  * Huang et al. (2022b) Huang, X.; Zhang, Y.; Ni, B.; Li, T.; Chen, K.; and Zhang, W. 2022b.  Boosting Point Clouds Rendering via Radiance Mapping.  _arXiv preprint arXiv:2210.15107_.
  * Jensen (2009) Jensen, M. H. 2009.  Improving the Performance of Hearing Aids Using Acoustic Simulations.  _COMSOL Conference_.
  * Jin et al. (2013) Jin, C. T.; Guillon, P.; Epain, N.; Zolfaghari, R.; Van Schaik, A.; Tew, A. I.; Hetherington, C.; and Thorpe, J. 2013.  Creating the Sydney York morphological and acoustic recordings of ears database.  _IEEE Transactions on Multimedia_ , 16(1): 37–46.
  * Kappel et al. (2021) Kappel, M.; Golyanik, V.; Elgharib, M.; Henningson, J.-O.; Seidel, H.-P.; Castillo, S.; Theobalt, C.; and Magnor, M. 2021.  High-Fidelity Neural Human Motion Transfer from Monocular Video.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 1541–1550.
  * Karras, Laine, and Aila (2019) Karras, T.; Laine, S.; and Aila, T. 2019.  A style-based generator architecture for generative adversarial networks.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 4401–4410.
  * Kato, Ushiku, and Harada (2018) Kato, H.; Ushiku, Y.; and Harada, T. 2018.  Neural 3d mesh renderer.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 3907–3916.
  * Kingma and Ba (2014) Kingma, D. P.; and Ba, J. 2014.  Adam: A method for stochastic optimization.  In _ICLR_.
  * Krishan, Kanchan, and Thakur (2019) Krishan, K.; Kanchan, T.; and Thakur, S. 2019.  A study of morphological variations of the human ear for its applications in personal identification.  _Egyptian Journal of Forensic Sciences_ , 9(1): 1–11.
  * Kumar and Wu (2012) Kumar, A.; and Wu, C. 2012.  Automated human identification using ear imaging.  _Pattern Recognition_ , 45(3): 956–968.
  * Laina et al. (2016) Laina, I.; Rupprecht, C.; Belagiannis, V.; Tombari, F.; and Navab, N. 2016.  Deeper depth prediction with fully convolutional residual networks.  In _2016 Fourth international conference on 3D vision (3DV)_ , 239–248. IEEE.
  * Li and Peissig (2020) Li, S.; and Peissig, J. 2020.  Measurement of head-related transfer functions: A review.  _Applied Sciences_ , 10(14): 5014.
  * Meshram et al. (2014) Meshram, A.; Mehra, R.; Yang, H.; Dunn, E.; Franm, J.-M.; and Manocha, D. 2014.  P-HRTF: Efficient personalized HRTF computation for high-fidelity spatial sound.  In _2014 IEEE International Symposium on Mixed and Augmented Reality (ISMAR)_ , 53–61. IEEE.
  * Nealen et al. (2006) Nealen, A.; Igarashi, T.; Sorkine, O.; and Alexa, M. 2006.  Laplacian mesh optimization.  In _Proceedings of the 4th international conference on Computer graphics and interactive techniques in Australasia and Southeast Asia_ , 381–389.
  * Nowogrodzki (2018) Nowogrodzki, A. 2018.  The world’s strongest MRI machines are pushing human imaging to new limits.  _Nature News Feature_ , 563(7729): 24–26.
  * O’Sullivan and Zafeiriou (2020) O’Sullivan, E.; and Zafeiriou, S. 2020.  3D Landmark Localization in Point Clouds for the Human Ear.  In _2020 15th IEEE International Conference on Automatic Face and Gesture Recognition (FG 2020)_ , 402–406. IEEE.
  * Pierce (2019) Pierce, A. D. 2019.  _Acoustics: an introduction to its physical principles and applications_.  Springer.
  * Ploumpis et al. (2020) Ploumpis, S.; Ververas, E.; O’Sullivan, E.; Moschoglou, S.; Wang, H.; Pears, N.; Smith, W.; Gecer, B.; and Zafeiriou, S. P. 2020.  Towards a complete 3D morphable model of the human head.  _IEEE transactions on pattern analysis and machine intelligence_.
  * Ploumpis et al. (2019) Ploumpis, S.; Wang, H.; Pears, N.; Smith, W. A.; and Zafeiriou, S. 2019.  Combining 3D Morphable Models: A Large scale Face-and-Head Model.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 10934–10943.
  * Qian et al. (2020) Qian, R.; Garg, D.; Wang, Y.; You, Y.; Belongie, S.; Hariharan, B.; Campbell, M.; Weinberger, K. Q.; and Chao, W.-L. 2020.  End-to-end pseudo-lidar for image-based 3d object detection.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 5881–5890.
  * Raposo et al. (2011) Raposo, R.; Hoyle, E.; Peixinho, A.; and Proença, H. 2011.  Ubear: A dataset of ear images captured on-the-move in uncontrolled conditions.  In _2011 IEEE workshop on computational intelligence in biometrics and identity management (CIBIM)_ , 84–90. IEEE.
  * Sun, Pears, and Dai (2021) Sun, H.; Pears, N.; and Dai, H. 2021.  A Human Ear Reconstruction Autoencoder.  In _2021 16th International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications (VISIGRAPP 2021)_ , 136–145.
  * Varman et al. (2021) Varman, R. M.; Schrader, K.; Fort, C.; Daniel, H.; Al Mekdash, M. H.; and Demke, J. 2021.  Ability to Identify External Ear Deformities and Normal External Ear Anatomy Based on Year and Specialty of Medical Training.  _The American Journal of Cosmetic Surgery_ , 0748806821993681.
  * Vorländer (2020) Vorländer, M. 2020.  _Auralization_.  Springer.
  * Wang et al. (2020) Wang, J.; Mueller, F.; Bernard, F.; Sorli, S.; Sotnychenko, O.; Qian, N.; Otaduy, M. A.; Casas, D.; and Theobalt, C. 2020.  Rgb2hands: real-time tracking of 3d hand interactions from monocular rgb video.  _ACM Transactions on Graphics (TOG)_ , 39(6): 1–16.
  * Xie, Zhong, and He (2015) Xie, B.; Zhong, X.; and He, N. 2015.  Typical data and cluster analysis on head-related transfer functions from Chinese subjects.  _Applied Acoustics_ , 94: 1–13.
  * Yan and Bowyer (2007) Yan, P.; and Bowyer, K. W. 2007.  Biometric recognition using 3D ear shape.  _IEEE Transactions on pattern analysis and machine intelligence_ , 29(8): 1297–1308.
  * Yang et al. (2021) Yang, J.; Huang, X.; He, Y.; Xu, J.; Yang, C.; Xu, G.; and Ni, B. 2021.  Reinventing 2d convolutions for 3d images.  _IEEE Journal of Biomedical and Health Informatics_ , 25(8): 3009–3018.
  * Zhang et al. (2020a) Zhang, F.; Bazarevsky, V.; Vakunov, A.; Tkachenka, A.; Sung, G.; Chang, C.-L.; and Grundmann, M. 2020a.  Mediapipe hands: On-device real-time hand tracking.  _Fourth Workshop on Computer Vision for AR/VR_.
  * Zhang et al. (2020b) Zhang, Y.; Chen, W.; Ling, H.; Gao, J.; Zhang, Y.; Torralba, A.; and Fidler, S. 2020b.  Image GANs meet Differentiable Rendering for Inverse Graphics and Interpretable 3D Neural Rendering.  In _International Conference on Learning Representations_.
  * Zhou and Gregson (2020) Zhou, Y.; and Gregson, J. 2020.  WHENet: Real-time Fine-Grained Estimation for Wide Range Head Pose.
  * Zhou and Zaferiou (2017) Zhou, Y.; and Zaferiou, S. 2017.  Deformable models of ears in-the-wild for alignment and recognition.  In _2017 12th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2017)_, 626–633. IEEE.

##  Appendix A Appendix A: Details of Loss Functions

#### Contour Loss

We split the $55$ landmarks $K_{i}\in\mathbb{R}^{2},i=1,2\dots,55$ into $4$
groups according to their distribution on the ears. The groups are indicated
in Figure 2 (AudioEar2D) by four different landmark colors. Then we connect
the landmarks within one group one by one and get $4$ polylines. Then we
uniformly sample $N$ points on each polylines, denoted as
$\mathcal{P}^{1},\mathcal{P}^{2},\mathcal{P}^{3},\mathcal{P}^{4}\in\mathbb{R}^{N\times
2}$. We execute these steps on both the annotated landmarks and the projected
landmarks from the predicted meshes, and obtain four pairs of sampled points
$(\mathcal{P}_{proj}^{1},\mathcal{P}_{gt}^{1}),(\mathcal{P}_{proj}^{2},\mathcal{P}_{gt}^{2}),(\mathcal{P}_{proj}^{3},\mathcal{P}_{gt}^{3}),(\mathcal{P}_{proj}^{4},\mathcal{P}_{gt}^{4})$.
The contour loss is the average of the chamfer distance between
$\mathcal{P}_{proj}$ and $\mathcal{P}_{gt}$ in each pair:

$L_{contour}=\frac{1}{4}\sum_{i=1}^{4}ChamferDistance(\mathcal{P}_{proj}^{i},\mathcal{P}_{gt}^{i})$
(5)

where the superscript $i$ is the index of the four contours.

#### Camera Loss

We first define the range loss, which penalize a variable with
$L2$-regularization when it is out of a specified range $[x_{min},x_{max}]$:

$\begin{split}L_{range}(x,x_{min},x_{max})=\begin{cases}(x_{min}-x)^{2}&\text{
if }x<x_{min}\\\ (x-x_{max})^{2}&\text{ if }x>x_{max}\\\ 0&\text{ otherwise
}\end{cases}\end{split}$ (6)

Based on the range loss, we constrain the scale $s$ and the rotation angle
along y axis, $r_{y}$. The camera loss is formulated as:

$L_{cam}=L_{range}(s,0.5,4)+L_{range}(r_{y},-1.5,-0.5)$

#### Landmark Loss

We use the normalized landmark loss proposed by Zhou and Zaferiou. The
landmarks on the predicted 3D ear mesh
$K_{pred}^{i}\in\mathbb{R}^{3},i=1,2\dots,55$ are projected to the image plane
by an orthogonal projection function $\mathbf{C}$. The landmark loss is
calculated between the projected landmarks and the ground-truth 2D landmarks
$K_{gt}^{i}\in\mathbb{R}^{2},i=1,2\dots,55$ on the image plane. The landmark
loss is formulated as:

$L_{lmk}=\frac{\sum_{i=1}^{55}\left\|K_{gt}^{i}-\mathbf{C}(K_{pred}^{i})\right\|_{2}}{55\times
D_{gt}}$ (7)

where $D_{gt}$ is the diagonal length of the ground-truth landmarks’ bounding
box.

#### Photometric Loss

The photometric loss computes the pixel error between the input image $I$ and
the rendered image $I_{r}$. To ignore the background of the input image, we
use the 2D landmarks to form an ear mask $M_{I}$. Specifically, we connect the
outer landmarks to form a closed polygon and fill the inner part of the
polygon with value $1$, the outer part with value $0$. The loss is computed as

$L_{p}=\|M_{I}\cdot I-I_{r}\|_{2}$ (8)

#### Mesh Smooth Loss

We add laplacian smoothing (Desbrun et al.; Nealen et al.) on the predicted
ear meshes to avoid sharp deformation. We adopt uniform laplacian for
simplicity. For each vertex $v_{i}$, we denote its neighbor vertices as
$\mathcal{N}_{v_{i}}$ and the number of neighbor vertices as
$N=|\mathcal{N}_{v_{i}}|$. The laplacian of $v_{i}$ is defined as:

$\mathbf{Laplacian}(v_{i})=\sum_{j=1}^{N}w_{ij}\left(\mathcal{N}_{v_{i}}^{j}-v_{i}\right)$
(9)

For uniform laplacian, $w_{ij}=1/|\mathcal{N}_{v_{i}}|=1/N$. Therefore
Equation 9 could be simplified as:

$\mathbf{Laplacian}(v_{i})=\frac{1}{N}\sum_{j=1}^{N}\mathcal{N}_{v_{i}}^{j}-v_{i}$
(10)

The term $\frac{1}{N}\sum_{j=1}^{N}\mathcal{N}_{v_{i}}^{j}$ indicates the
centroid of the neighbor vertices. The final mesh smoothing loss is the
average on the laplacian of all vertices.

#### Regularization

We add $L1$-regularization on the shape latent code $\beta\in\mathbb{R}^{236}$
and texture latent code $\theta\in\mathbb{R}^{50}$ to avoid them from being
too large. The regularization is formulated as:

$L_{reg}=\frac{1}{236}\sum_{i=1}^{236}\left|\beta_{i}\right|+\frac{1}{50}\sum_{i=1}^{50}\left|\theta_{i}\right|$
(11)

The overall loss is the weighted sum of the losses above:

$\begin{split}L_{total}=\lambda_{contour}L_{contour}+\lambda_{sim}L_{sim}+\lambda_{cam}L_{cam}\\\
+\lambda_{lmk}L_{lmk}+\lambda_{photo}L_{photo}+\lambda_{smooth}L_{smooth}+\lambda_{reg}L_{reg}.\end{split}$
(12)

where $\lambda_{contour}=100$, $\lambda_{sim}=1$, $\lambda_{cam}=100$,
$\lambda_{lmk}=10$, $\lambda_{photo}=100$, $\lambda_{smooth}=10$, and
$\lambda_{reg}=0.005$ in our experiments. The regularization is weighted by a
small value since large regularization would lead to underfitting of the
model. The landmark loss is weighted by 10 instead of 100 since it is less
precise than the contour loss. The mesh smoothing could also be regarded as a
regularization on meshes so it is also less weighted than other losses.

##  Appendix B Appendix B: Training Configurations

For the depth estimation model, we use $8,000$ synthetic data for training and
$2,000$ for validation. We train the model for $100$ epochs using Adam
optimizer with an initial learning rate of $0.001$, which is declined by $0.2$
in the $50$th epoch. For the reconstruction model, we randomly split
AudioEar2D into training and validation set by $9$:$1$. All left ears are
flipped to the right following prior works, so that the model only process
one-sided ears. We train AudioEarM for $100$ epochs in two stages. For the
first $20$ epochs, we fix the shape and texture latent code so that the model
outputs mean shape and texture, and minimize only landmark loss and
regularization loss to train the camera and lighting. We empirically find this
benefits a reasonable shape reconstruction. For the rest $80$ epochs, we
optimize all the parameters by minimizing the weighted sum of all losses. We
use Adam optimizer with an initial learning rate of $0.001$ and multiply the
learning rate by $0.2$ at the $50$th, $75$th and $90$th epoch.

##  Appendix C Appendix C: Impact of Trainset

Table 5: The impact of trainset. #2-3 analyze the occlusion of AudioEar2D. #4-5 compare AudioEar2D and IBug-A. #6 uses synthetic data to train the reconstruction model with explicit latent code supervision instead of using it in the depth-guided way. # | Trainset | Scale | S2M
---|---|---|---
1 | AudioEar2D | 2,000 | 1.28mm
2 | #1 $-$ earrings | 1,682 | 1.36mm
3 | #1 $-$ hair & earrings | 849 | 1.44mm
4 | IBug-A | 605 | 1.48mm
5 | #1 $+$ IBug-A | 2,605 | 1.45mm
6 | #1 $+$ synthetic | 12,000 | 1.63mm

We examine the impact of trainset by training AudioEarM on different
trainsets. First, we remove the partly occluded ears in AudioEar2D from the
trainset, to examine the impact of occlusions (Table 5 #2-3). We observe that
as we remove more occluded images, the performance gets worse. It seems that
partial occlusion does not affect the performance as much as a smaller
trainset does. Then we compare AudioEar2D with IBug-A (Zhou and Zaferiou), a
prior ear dataset with landmarks. IBug-A contains $605$ images compared to
$2,000$ in AudioEar2D (Table 5 #4-5). Training AudioEarM only on IBug-A yields
a result of $1.48$ mm ($1.28$ mm on AudioEar2D). It demonstrates that a
smaller scale of trainset does lead to performance degradation, which proves
the value of AudioEar2D. We also combine AudioEar2D with IBug-A together for
cross-domain training and find that the performance is worse than training on
AudioEar2D alone, although the trainset scale is larger. We reckon it is
caused by a domain gap between the two datasets. Specifiaclly, the domain gap
between IBug-A and our AudioEar2D could be in two aspects. One is that IBug-A
includes grayscale images, while AudioEar2D is entirely RGB. The other is
about the landmark annotation standard. Since some of the landmarks lack
explicit definition, our annotation standard might be different from theirs,
which might leads to a domain gap. We reckon the first factor is more critical
to the degraded performance since our contour loss could alleviate the second
one. Lastly, instead of using the synthetic data to train a depth estimation
model, we combine it with AudioEar2D together as the trainset and use the
known latent code of the synthetic data as explicit supervision for the
training of reconstruction model. This practice is similar to the MediaPipe
Hands (Zhang et al.). It yields a result of $1.63$ mm, which is rather poor.
In this respect, the depth estimation model in our method might be playing the
role to bridge the gap between the synthetic and realistic data.

##  Appendix D Appendix D: SPL Error

For a better understanding on the SPL error, we illustrate it with polar
coordinates in Figure 5. HRTF is simulated in all azimuth angles ($361$
angles) on the horizontal plane. Then the SPL error is drawn in a polar
coordinate system with azimuth angles as angular coordinates and the error in
each angle as radial coordinate. The final SPL error is averaged among all
angles.

Figure 5: Illustrations of the SPL error in polar coordinates under three
simulation frequencies. Polar angles denotes the simulated azimuths, while
radius denotes the magnitude of error. The mean SPL error is calculated among
all azumith angles.

##  Appendix E Appendix E: Limitations

Although the reconstruction from single-view images in this work has obtained
a relatively good result compared to the average ear, there is still room for
further improvements in order to obtain a better spatial audio effect. In
future work, we will consider a multi-view reconstruction setting. More views
could reveal more 3D structures of the ears and lead to better reconstruction.

The HRTF is not only conditional on the ear structure, but also on the head
and body structure. Besides, the position and orientation of the reconstructed
ears with respect to the head are not automatically estimated. Therefore the
simulated HRTF is not the real HRTF of the individual. In future work, we are
going to jointly reconstruct the human body and estimate the relative position
of ears at the same time to obtain a more realistic HRTF.


# Unitarity Problem in Gribov-Zwanziger Theory

Thitipat Sainapha<EMAIL_ADDRESS>Department of Physics,
Graduate School of Science and Engineering, Chiba University, Chiba 263-8522,
Japan  Kei-Ichi Kondo<EMAIL_ADDRESS>Department of Physics,
Graduate School of Science, Chiba University, Chiba 263-8522, Japan

###### Abstract

We show that the unitarity problem in the Gribov-Zwanziger theory can be
solved by the famous quartet mechanism. The construction of the new quartet in
such a theory suggests the introduction of a new form of ghost charge and
projection operator.

††preprint: APS/123-QED

##  I Introduction

Unitarity is considered to be one of the important properties of the quantum
field theories since it will be not acceptable at all if there is such a case
that the considering quantum theory does contain an ill-defined probability.
Historically, the quantum Yang-Mills theory constructed through the Faddeev-
Popov quantization procedure [1] risks violating the unitarity condition. The
particular reason is that the quantization method conditionally introduces
anti commuting fields with zero spin, known as ghost and antighost into the
theory which violate the so-called spin-statistics theorem. It was shown
mathematically that the violation of the spin-statistics theorem will lead to
the violation of the unitarity accordingly [2]. However, the work of Kugo and
Ojima showed that the existence of the Becchi-Rouet-Stora-Tyutin (BRST)
symmetry in the quantum Yang-Mills theory will smoothly rule out the
unphysical degrees of freedom like ghost from the physical Hilbert space.
Their method is known as the Kugo-Ojima quartet mechanism [3, 4].
Unfortunately, the discussion of the BRST quartet mechanism excludes the
possibility that the Faddeev-Popov method fails to quantize the theory.
Theoretically speaking, to quantize the Yang-Mills theory, we have to impose
the gauge fixing condition to eliminate the gauge degrees of freedom to make
the quantization procedure unambiguous. However, the gauge orbit turns out to
cross the gauge fixing hypersurface more than once implying that the gauge
redundant degrees of freedom still remain after fixing the gauge. This problem
is known famously as the Gribov ambiguity [5]. To remedy the problem, we have
to force the gauge orbit to be bounded inside the so-called Gribov region. In
level of action, it is equivalent to include the extra term into the action of
the quantum Yang-Mills theory, the resulting action is called the Gribov-
Zwanziger action [6]. Since the resulting action is non-local, to solve that,
we have to add some more sets of auxiliary fields to localize the theory. This
attempt causes more serious results since the added fields essentially be the
spin-1 anti-commuting fields which again lead to the question about the
unitarity of the Gribov-Zwanziger theory.
The paper will be organized as follows: In section II, we will explain how the
occurrence of the BRST symmetry in the usual quantum Yang-Mills theory will
form the BRST quartet which influences the statement that rules out the
unphysical degrees of freedom from the theory. In section III, we will take
the Gribov ambiguity into account and show that the quartet mechanism works
well in the Gribov-Zwanziger model.

##  II BRST quartet in usual quantum Yang-Mills Theory

Let us start by recalling the occurrence and the impact of the BRST quartet in
the usual quantum Yang-Mills theory before we move on to discuss the Gribov-
Zwanziger case. The action of the quantum Yang-Mills theory, constructed based
on the Faddeev-Popov procedure in the Euclidean space, takes the following
form

$\begin{split}S_{qYM}&=\int
d^{D}x\;\mathcal{L}_{qYM},\;\;\;\mathcal{L}_{qYM}=\mathcal{L}_{YM}+\mathcal{L}_{gf}\\\
\mathcal{L}_{YM}&=\frac{1}{4}F^{a}_{\mu\nu}F^{a}_{\mu\nu}\\\
\mathcal{L}_{gf}&=B\cdot\partial_{\mu}A_{\mu}+\bar{c}\cdot\partial_{\mu}\mathcal{D}_{\mu}c,\end{split}$
(1)

where this action is constrained by the Landau gauge condition
$\partial_{\mu}A^{a}_{\mu}=0$ to eliminate the gauge redundant degrees of
freedom before completing the quantization process. Generically, the theory is
no longer invariant under the usual gauge transformation but still invariant
under the new symmetry, known as the BRST symmetry, which is the reminiscence
of the usual one. According to Noether’s theorem, the BRST symmetry
transformation is technically generated by the BRST charge $Q_{B}$ of the form

$Q_{B}=\int
d^{d}x\;\left(\pi_{\mu}^{a}(\mathcal{D}_{\mu}c)^{a}-\frac{g}{2}f^{abc}\pi^{a}_{c}c^{b}c^{c}+\pi_{\bar{c}}^{a}B^{a}\right),$
(2)

where the canonical momenta $\pi_{\mu}^{a}$, $\pi_{c}^{a}$, and
$\pi^{a}_{\bar{c}}$ conjugate respectively to the Yang-Mills field
$A^{a}_{\mu}$, ghost $c^{a}$, and antighost $\bar{c}^{a}$ are defined as
follows

$\begin{split}\pi_{\mu}^{a}&\equiv\frac{\partial\mathcal{L}_{qYM}}{\partial(\partial_{0}A^{a}_{\mu})}=F^{a}_{0\mu},\\\
\pi_{c}^{a}&\equiv\frac{\partial\mathcal{L}_{qYM}}{\partial(\partial_{0}c^{a})}=-\partial_{0}\bar{c}^{a},\\\
\pi_{\bar{c}}^{a}&\equiv\frac{\partial\mathcal{L}_{qYM}}{\partial(\partial_{0}\bar{c}^{a})}=-(\mathcal{D}_{0}c)^{a}.\end{split}$
(3)

Let us note that the Hermiticity of ghost $c^{a}$ and antighost $\bar{c}^{a}$
are assigned to be $(c^{a})^{\dagger}=c^{a}$ and
$(\bar{c}^{a})^{\dagger}=-\bar{c}^{a}$ which is the same as the Hermiticity
assignment in [3, 4]. The particular reason to do so is to guarantee that
$Q_{B}$ is Hermitian which will be useful later.
One other thing we have to notice is that the BRST charge carries one ghost
number counted by the ghost number operator

$Q_{c}=\int d^{d}x\;(\bar{c}^{a}\pi_{\bar{c}}^{a}-\pi_{c}^{a}c^{a}).$ (4)

The fact that the BRST charge does carry exactly one ghost number is expressed
mathematically by the following BRST algebra

$[iQ_{c},Q_{B}]=Q_{B}.$ (5)

This behavior is theoretically meaningful as we will clarify in what follows.
Recall that there are four main unphysical modes in the quantum Yang-Mills
theory which are the longitudinal mode of gluon $A_{\mu}$, Nakanishi-Lautrup
field $B$, ghost $c$, and antighost $\bar{c}$. With the simple calculation, we
can easily obtain the commutation (graded) relations between these four fields
with the BRST operator

$\begin{split}[iQ_{B},A_{\mu}^{a}]&=(\mathcal{D}_{\mu}c)^{a},\\\
\\{iQ_{B},c^{a}\\}&=-\frac{g}{2}f^{abc}c^{b}c^{c},\\\
\\{iQ_{B},\bar{c}^{a}\\}&=B^{a},\\\ [iQ_{B},B^{a}]&=0.\end{split}$ (6)

The above graded commutation relation truly guarantees that the BRST charge is
generically the symmetry generator of the BRST transformation.
From this step, the main objective of the following steps is to ensure that
these unphysical modes are not asymptotically observable. To do so, we recall
that these fields admit the massless pole structure, so these fields can be
written asymptotically as the following expressions

$\begin{split}A_{\mu}&\rightarrow\partial_{\mu}\chi+...\\\
B&\rightarrow\beta+...\\\
\mathcal{D}_{\mu}c&\rightarrow\partial_{\mu}\gamma+...\\\
\bar{c}&\rightarrow\bar{\gamma}+...\end{split}$ (7)

As a consequence, the graded commutation relation (5) changes accordingly, it
casts into

$\begin{split}[Q_{B},\chi^{a}]&=\gamma^{a},\\\ \\{Q_{B},\gamma^{a}\\}&=0,\\\
\\{Q_{B},\bar{\gamma}^{a}\\}&=\beta^{a},\\\ [Q_{B},\beta^{a}]&=0.\end{split}$
(8)

Through the canonical quantization, the Fourier modes or the creation operator
$(\chi_{k}^{a})^{\dagger}$, $(\beta_{k}^{a})^{\dagger}$,
$(\gamma^{a}_{k})^{\dagger}$, and $(\bar{\gamma}^{a}_{k})^{\dagger}$ of the
asymptotic fields $\chi^{a}$, $\beta^{a}$, $\gamma^{a}$, and
$\bar{\gamma}^{a}$ satisfy the following commutation relations

$\begin{split}[Q_{B},(\chi^{a}_{k})^{\dagger}]&=-(\gamma^{a}_{k})^{\dagger},\\\
\\{Q_{B},(\gamma^{a}_{k})^{\dagger}\\}&=0,\\\
\\{Q_{B},(\bar{\gamma}^{a}_{k})^{\dagger}\\}&=(\beta^{a}_{k})^{\dagger},\\\
[Q_{B},(\beta^{a}_{k})^{\dagger}]&=0.\end{split}$ (9)

These creation operators for asymptotic fields can be legitimately used to
construct the states quantum mechanically. Interestingly, these states form
the vector space with the BRST complex. As a result, due to the existence of
the BRST algebra which is actually the $\mathbb{Z}$-graded Poisson
superalgebra, we are mathematically allowed to split the Hilbert space of the
considering theory into the direct sum among the vector spaces of each degree
characterized by the ghost number. This is equivalent to constructing the
projection operator onto the state in the Hilbert space with $n$ ghost numbers
[4] as the following expression

$\begin{split}P^{(n)}=&\frac{1}{n}(-\beta^{\dagger}_{k}P^{(n-1)}\chi_{k}-\chi^{\dagger}_{k}P^{(n-1)}\beta_{k}\\\
&+\bar{\gamma}^{\dagger}_{k}P^{(n-1)}\gamma_{k}+\gamma^{\dagger}_{k}P^{(n-1)}\bar{\gamma}_{k}).\end{split}$
(10)

Note that the choice of the projection operator’s expression is not totally
unique, one might add the term like
$-\omega_{kl}\beta^{\dagger}_{k}P^{(n-1)}\beta_{l}$ where
$\omega\equiv[\chi_{k},\chi^{\dagger}_{l}]$ without changing the nature of the
projection operator itself.
Significantly, this projection operator can be realized as the exact operator
under the BRST transformation. Namely, we can simply write

$P^{(n)}=\\{Q_{B},R^{(n)}\\},$ (11)

where
$R^{(n)}=-\frac{1}{n}(\gamma^{\dagger}_{k}P^{(n-1)}\chi_{k}+\chi^{\dagger}_{k}P^{(n-1)}\gamma_{k})$.
This relation shows that the inner product between the state projected to
$P^{(n)}$ and a state with zero ghost number yields zero, implying that the
states with non-zero ghost numbers are unphysical by default. This statement
really ensures that the physical Hilbert space will not include the particles
that violate the spin-statistics theorem; hence, unitarity.

##  III New Definition of Ghost Charge and Quartet in Gribov-Zwanziger Theory

Let’s consider the case in which the usual treatment of the quantization of
the Yang-Mills theory fails. If we choose the Landau gauge fixing condition
which is the covariant gauge constraint, the Gribov ambiguity will occur
consequently. The way to partially solve this problem is to introduce the
boundary that cuts off the integral over the gauge space, i.e. the Gribov
horizon. The occurrence of the Gribov horizon in the level of action
introduces the new action accordingly. Therefore, we have the Gribov-Zwanziger
action of the following expression [7]

$\begin{split}S_{GZ}=&S_{YM}+\int
d^{D}x\;(B^{h}\cdot\partial_{\mu}A_{\mu}+\bar{c}\cdot\partial_{\mu}\mathcal{D}_{\mu}c)\\\
&+\int
d^{D}x\;(\bar{\varphi}_{\mu}^{ac}M^{ab}\varphi^{bc}_{\mu}-\bar{\omega}^{ac}_{\mu}M^{ab}\omega_{\mu}^{bc}\\\
&+\gamma^{1/2}_{G}gf^{abc}A^{h,a}_{\mu}(\varphi^{bc}_{\mu}+\bar{\varphi}^{bc}_{\mu}),\end{split}$
(12)

where a set of the auxiliary fields
$\\{\varphi,\bar{\varphi},\omega,\bar{\omega}\\}$ has been added to the theory
to make the Gribov-Zwanziger action localized and $M^{ab}$ is a Faddeev-Popov
operator defined to be $M^{ab}\equiv-\partial_{\mu}\mathcal{D}^{ab}_{\mu}$.
These four fields transform as vectors under the Poincare transformation but
the latter two fields, i.e. $\omega$, and $\bar{\omega}$ are spin-one anti-
commuting fields, hence, ghost. The existence of these fields would destroy
the unitarity of the model since we expect that the same procedure can still
be used. First, we have to construct the new nilpotent Noether charge which
leaves the localized Gribov-Zwanziger action (7) intact [8, 7]. In this
particular case, we have

$\tilde{Q}=Q_{B}+Q_{\gamma}=Q_{B}+\int
d^{d}x\;(\pi^{ab}_{\varphi\;\mu}\omega^{ab}_{\mu}+\pi^{ab}_{\bar{\omega}\;\mu}\bar{\varphi}^{ab}_{\mu}),$
(13)

where the canonical momenta $\pi_{\varphi}$, $\pi_{\bar{\varphi}}$,
$\pi_{\omega}$, and $\pi_{\bar{\omega}}$ associated to auxiliary fields
$\varphi$, $\bar{\varphi}$, $\omega$, and $\bar{\omega}$, respectively, are
defined below

$\begin{split}\pi^{ab}_{\varphi\;\mu}&\equiv\frac{\partial\mathcal{L}_{GZ}}{\partial(\partial_{0}\varphi_{\mu}^{ab})}=\partial_{0}\bar{\varphi}^{ab}_{\mu},\\\
\pi^{ab}_{\bar{\varphi}\;\mu}&\equiv\frac{\partial\mathcal{L}_{GZ}}{\partial(\partial_{0}{\bar{\varphi}}_{\mu}^{ab})}=\mathcal{D}_{0}{\varphi}^{ab}_{\mu},\\\
\pi^{ab}_{\omega\;\mu}&\equiv\frac{\partial\mathcal{L}_{GZ}}{\partial(\partial_{0}\omega_{\mu}^{ab})}=-\partial_{0}\bar{\omega}^{ab}_{\mu},\\\
\pi^{ab}_{\bar{\omega}\;\mu}&\equiv\frac{\partial\mathcal{L}_{GZ}}{\partial(\partial_{0}\bar{\omega}_{\mu}^{ab})}=-\mathcal{D}_{0}\bar{\omega}^{ab}_{\mu}.\end{split}$
(14)

We can easily check that the new charge $\tilde{Q}$ of BRST type satisfies the
following set of the graded commutation relations

$\begin{split}[\tilde{Q},\varphi^{ab}_{\mu}]&=\omega^{ab}_{\mu},\\\
\\{\tilde{Q},\omega^{ab}_{\mu}\\}&=0,\\\
\\{\tilde{Q},\bar{\omega}^{ab}_{\mu}\\}&=\bar{\varphi}^{ab}_{\mu},\\\
[\tilde{Q},\bar{\varphi}^{ab}_{\mu}]&=0.\end{split}$ (15)

However, we have to remark that this charge no longer commutes with the ghost
charge in the same as the usual $Q_{B}$ does. Explicitly speaking, it is
simple enough to see that

$[iQ_{c},\tilde{Q}]=Q_{B}\not=\tilde{Q}.$ (16)

This is a straightforward result we expect to have because the usual BRST
charge has nothing related to the new auxiliary fields. To maintain the same
structure of the quartet mechanism as those in the Faddeev-Popov quantum Yang-
Mills theory, we necessarily introduce the new definition of the ghost number
operator as follows

$Q_{G}=Q_{c}+\int
d^{d}x\;(\bar{\omega}^{ab}_{\mu}\pi^{ab}_{\bar{\omega}\;\mu}-\pi_{\omega\;\mu}^{ab}\omega^{ab}_{\mu}).$
(17)

In this situation, we can easily check that the new ghost number really counts
the auxiliary field $\omega$ as a new ghost field and $\bar{\omega}$ as a new
antighost field. Namely, we obtain naturally

$\begin{split}[iQ_{G},\omega^{ab}_{\mu}(x)]=&-i\int
d^{d}y\;[\pi_{\omega\;\nu}^{cd}(y)\omega^{cd}_{\nu}(y),\omega^{ab}_{\mu}(x)]\\\
=&-i\int
d^{d}y\;(\pi_{\omega\;\nu}^{cd}(y)\omega^{cd}_{\nu}(y)\omega^{ab}_{\mu}(x)\\\
&-\omega^{ab}_{\mu}(x)\pi_{\omega\;\nu}^{cd}(y)\omega^{cd}_{\nu}(y))\\\
=&\;i\int
d^{d}y\;(\pi_{\omega\;\nu}^{cd}(y)\omega^{ab}_{\mu}(x)\omega^{cd}_{\nu}(y)\\\
&+\omega^{ab}_{\mu}(x)\pi_{\omega\;\nu}^{cd}(y)\omega^{cd}_{\nu}(y))\\\
=&\;i\int
d^{d}y\;(\\{\pi_{\omega\;\nu}^{cd}(y),\omega^{ab}_{\mu}(x)\\}\omega^{cd}_{\nu}(y))\\\
=&\;i\int
d^{d}y\;(-i\delta^{d}(x-y)\delta^{ac}\delta^{bd}\delta_{\mu\nu})\omega^{cd}_{\nu}(y)\\\
=&\;\omega^{ab}_{\mu}(x),\end{split}$ (18)

and

$[iQ_{G},\bar{\omega}^{ab}_{\mu}(x)]=-\bar{\omega}^{ab}_{\mu}(x).$ (19)

Similarly, we can obtain the commutation relation between the new ghost number
operator and the canonical momenta, we obtain

$[iQ_{G},\pi_{\omega\;\mu}^{ab}(x)]=-\pi_{\omega\;\mu}^{ab}(x)$ (20)

and

$[iQ_{G},\pi^{ab}_{\bar{\omega}\;\mu}(x)]=\pi^{ab}_{\bar{\omega}\;\mu}(x),$
(21)

respectively. As a result, we end up with the commutation relation we expect
to have

$\begin{split}[iQ_{G},\tilde{Q}]&=Q_{B}+\int
d^{d}x[iQ_{G},\pi^{ab}_{\varphi\;\mu}\omega^{ab}_{\mu}+\pi^{ab}_{\bar{\omega}\;\mu}\bar{\varphi}^{ab}_{\mu}]\\\
&=Q_{B}+\int d^{d}x\;\pi^{ab}_{\varphi\;\mu}[iQ_{G},\omega^{ab}_{\mu}]+\int
d^{d}x\;[iQ_{G},\pi^{ab}_{\bar{\omega}\;\mu}]\bar{\varphi}^{ab}_{\mu}\\\
&=Q_{B}+\int
d^{d}x\;(\pi^{ab}_{\varphi\;\mu}\omega^{ab}_{\mu}+\pi^{ab}_{\bar{\omega}\;\mu}\bar{\varphi}^{ab}_{\mu})\\\
&=\tilde{Q}.\end{split}$ (22)

This commutation relation ensures that the new BRST charge truly carries
exactly one ghost number implying that the new BRST charge will behave as the
ladder operator among the states of different ghost numbers. Here repeating
the same procedure by introducing the asymptotic form of the new fields

$\begin{split}\omega_{\mu}&\rightarrow\partial_{\mu}\Omega+...\\\
\bar{\omega}_{\mu}&\rightarrow\partial_{\mu}\bar{\Omega}+...\\\
\varphi_{\mu}&\rightarrow\partial_{\mu}\phi+...\\\
\bar{\varphi}_{\mu}&\rightarrow\partial_{\mu}\bar{\phi}+...\end{split}$ (23)

The superalgebra between the Fourier modes of these asymptotic fields and
$\tilde{Q}$ can be written concisely as follows

$\begin{split}[\tilde{Q},\phi^{ab}_{k}]&=\Omega^{ab}_{k},\\\
\\{\tilde{Q},\Omega^{ab}_{k}\\}&=0,\\\
\\{\tilde{Q},\bar{\Omega}^{ab}_{k}\\}&=\bar{\phi}^{ab}_{k},\\\
[\tilde{Q},\bar{\phi}^{ab}_{k}]&=0,\end{split}$ (24)

and

$\begin{split}[\tilde{Q},(\phi^{ab}_{k})^{\dagger}]&=-(\Omega^{ab}_{k})^{\dagger},\\\
\\{\tilde{Q},(\Omega^{ab}_{k})^{\dagger}\\}&=0,\\\
\\{\tilde{Q},(\bar{\Omega}^{ab}_{k})^{\dagger}\\}&=(\bar{\phi}^{ab}_{k})^{\dagger},\\\
[\tilde{Q},(\bar{\phi}^{ab}_{k})^{\dagger}]&=0.\end{split}$ (25)

This graded commutation relation implies that we can construct the new set of
quartet states legitimately. Once again, the theory mathematically admits the
$\tilde{Q}$ Poisson superalgebra, we can consequently construct the projection
operator onto the certain state of $n$ ghost numbers as

$\begin{split}\tilde{P}^{(n)}&=P^{(n)}+P^{(n)}_{\gamma}\\\ &\equiv
P^{(n)}+\frac{1}{n}(\phi^{\dagger}_{k}P^{(n-1)}_{\gamma}\mathcal{O}_{1}+\bar{\phi}^{\dagger}_{k}P^{(n-1)}_{\gamma}\mathcal{O}_{2}\\\
&\;\;+\Omega^{\dagger}_{k}P^{(n-1)}_{\gamma}\mathcal{O}_{3}+\bar{\Omega}^{\dagger}_{k}P^{(n-1)}_{\gamma}\mathcal{O}_{4}),\end{split}$
(26)

where $P^{(n)}$ is the same projection operator as that defined in the
previous section and the $P^{(n)}_{\gamma}$ is the projection operator
contributed by the influence of the Gribov horizon. Since $P^{(n)}_{\gamma}$
is defined to be the projection operator, it deserves to be one implying that
the double action of the projector has to be the same, i.e.
$(P^{(n)}_{\gamma})^{2}=P^{(n)}_{\gamma}$. Thus, we can deduce that
$\mathcal{O}_{1}$ and $\mathcal{O}_{2}$ are commuting fields while
$\mathcal{O}_{3}$ and $\mathcal{O}_{4}$ are anti-commuting fields.
Nevertheless, more information follows from the fact the Hilbert space
equipped with the Poisson superalgebra will be mathematically split into the
direct sum of the Hilbert space of each ghost degree, we have

$|\psi\big{>}=P^{(0)}|\psi\big{>}+\sum_{n=1}^{\infty}P^{(n)}|\psi\big{>}.$
(27)

If we impose the subsidiary condition $\tilde{Q}|\psi\big{>}=0$, the
projection operator has to commute with $\tilde{Q}$ inevitably. By
substituting the relation (26) into the commutation relation

$[\tilde{Q},P^{(n)}_{\gamma}]=0,$ (28)

it yields the following equation

$\begin{split}0=&-\Omega^{\dagger}_{k}P^{(n-1)}_{\gamma}\mathcal{O}_{1}+\phi^{\dagger}_{k}P^{(n-1)}_{\gamma}\delta\mathcal{O}_{1}+\bar{\phi}^{\dagger}_{k}P^{(n-1)}_{\gamma}\delta\mathcal{O}_{2}\\\
&-\Omega^{\dagger}_{k}P^{(n-1)}_{\gamma}\delta\mathcal{O}_{3}+\bar{\phi}^{\dagger}_{k}P^{(n-1)}_{\gamma}\mathcal{O}_{4}-\bar{\Omega}^{\dagger}_{k}P^{(n-1)}_{\gamma}\delta\mathcal{O}_{4}\end{split}$
(29)

Combining with the fact that the first two operators are commuting fields and
the latter two are anti-commuting fields, the possible solution to this
equation is as follows

$\mathcal{O}_{1}=-\bar{\phi}_{k},\;\mathcal{O}_{2}=-\phi_{k},\;\mathcal{O}_{3}=\bar{\Omega}_{k},\;\mathcal{O}_{4}=\Omega_{k},$
(30)

where the sign of $\mathcal{O}_{1}$ and $\mathcal{O}_{3}$ are not determined
directly by the equation (29) but it is chosen to force the projection
operator to be Hermitian.
We can show that the resulting projection operator of degree $n$ is exact
under the action of the new BRST charge $\tilde{Q}$. Namely, we have

$P^{(n)}_{\gamma}=\\{\tilde{Q},R^{(n)}_{\gamma}\\},$ (31)

where

$R^{(n)}_{\gamma}=-\frac{1}{n}(\phi^{\dagger}_{k}P^{(n-1)}_{\gamma}\bar{\Omega}_{k}+\bar{\Omega}^{\dagger}_{k}P^{(n-1)}_{\gamma}\phi_{k}).$
(32)

The theoretical meaning of this behavior is as follows: Firstly, if we define
the projected state of the ghost number (degree) $n\geq 1$ as
$\tilde{P}^{(n)}|\psi\big{>}\equiv(P^{(n)}+P^{(n)}_{\gamma})|\psi\big{>}\equiv|\psi_{0}\big{>}\in\mathcal{V}_{0}$.
The inner product between this state with the state in the total Hilbert space
$|\psi\big{>}\in\mathcal{V}$, as long as the subsidiary condition is implied,
turns out to be zero identically

$\big{<}\psi|\psi_{0}\big{>}=\big{<}\psi|P^{(n)}|\psi\big{>}=\big{<}\psi|\\{\tilde{Q},R^{(n)}\\}|\psi\big{>}=0.$
(33)

This implies that the state $|\psi\big{>}$ and $|\psi+\psi_{0}\big{>}$ share
the same physical interpretation. Therefore, the physical Hilbert space
$\mathcal{H}_{phys}$ is a quotient vector space constructed as the total
Hilbert space $\mathcal{V}$ mod the unphysical Hilbert space
$\mathcal{V}_{0}$, i.e.

$\mathcal{H}_{phys}\equiv\mathcal{V}/\mathcal{V}_{0}.$ (34)

This fantastic result clearly guarantees that the unitarity property of the
local Gribov-Zwanziger theory containing the ghost fields is theoretically
safe.

##  IV Conclusion and Remark

We showed that the quartet mechanism works very well in the Gribov-Zwanziger
theory by introducing the new definition of ghost charge which follows as a
consequence of introducing the new spin-1 anti-commuting auxiliary fields. The
interesting question is whether this result hold in the refined Gribov-
Zwanziger case or not since, naively speaking, an additional dimension two
condensation in the refined Gribov-Zwanziger action might cause a trouble to
the new BRST symmetry. This question is important since the refined Gribov-
Zwanziger model so far produces several accurate lattice results. Fortunately
enough, it has been shown that the generalization of the BRST symmetry can be
done properly in the refined case [9]. Therefore, there is no problem at all
to proceed the same analysis in that model.

## References

  * Faddeev and Popov [1967] L. Faddeev and V. Popov, Feynman diagrams for the yang-mills field, Physics Letters B 25, 29 (1967).
  * und A S Wightman [1964] R. F. S. und A S Wightman,  _PCT, Spin Statistics, And All That_ (W A Benjamin Inc., New York, Amsterdam, 1964).
  * Kugo and Ojima [1978] T. Kugo and I. Ojima, Manifestly covariant canonical formulation of the yang-mills field theories. i: – general formalism –, Progress of Theoretical Physics 60, 1869 (1978).
  * Kugo and Ojima [1979] T. Kugo and I. Ojima, Local covariant operator formalism of non-abelian gauge theories and quark confinement problem, Progress of Theoretical Physics Supplement 66, 1 (1979).
  * Gribov [1978] V. Gribov, Quantization of non-abelian gauge theories, Nuclear Physics B 139, 1 (1978).
  * Zwanziger [1989] D. Zwanziger, Local and renormalizable action from the gribov horizon, Nuclear Physics B 323, 513 (1989).
  * Capri  _et al._ [2015] M. A. L. Capri, D. Fiorentini, M. S. Guimaraes, B. W. Mintz, L. F. Palhares, S. P. Sorella, D. Dudal, I. F. Justo, A. D. Pereira, and R. F. Sobreiro, Exact nilpotent nonperturbative brst symmetry for the gribov-zwanziger action in the linear covariant gauge, Phys. Rev. D 92, 045039 (2015).
  * Kondo [2009] K.-I. Kondo, The nilpotent ”brst” symmetry for the gribov-zwanziger theory (2009).
  * Sorella [2009] S. P. Sorella, Gribov horizon and brst symmetry: A few remarks, Phys. Rev. D 80, 025013 (2009), arXiv:0905.1010 [hep-th] .