11institutetext: Haozhe Xie
11email<EMAIL_ADDRESS>

Hongxun Yao
11email<EMAIL_ADDRESS>

Shengping Zhang
11email<EMAIL_ADDRESS>

Shangchen Zhou
11email<EMAIL_ADDRESS>

Wenxiu Sun
11email<EMAIL_ADDRESS>

1 Harbin Institute of Technology, China
2 SenseTime Research, China
3 Nanyang Technological University, Singapore
4 Peng Cheng Laboratory, China

# Pix2Vox++: Multi-scale Context-aware 3D Object Reconstruction from Single
and Multiple Images

Haozhe Xie1,2 Hongxun Yao1 Shengping Zhang1,4
Shangchen Zhou3 Wenxiu Sun2

(Received: 24 December 2019 / Revised: 28 March 2020 / Accepted: 12 June 2020)

###### Abstract

Recovering the 3D shape of an object from single or multiple images with deep
neural networks has been attracting increasing attention in the past few
years. Mainstream works (e.g. 3D-R2N2) use recurrent neural networks (RNNs) to
sequentially fuse feature maps of input images. However, RNN-based approaches
are unable to produce consistent reconstruction results when given the same
input images with different orders. Moreover, RNNs may forget important
features from early input images due to long-term memory loss. To address
these issues, we propose a novel framework for single-view and multi-view 3D
object reconstruction, named Pix2Vox++. By using a well-designed encoder-
decoder, it generates a coarse 3D volume from each input image. A multi-scale
context-aware fusion module is then introduced to adaptively select high-
quality reconstructions for different parts from all coarse 3D volumes to
obtain a fused 3D volume. To further correct the wrongly recovered parts in
the fused 3D volume, a refiner is adopted to generate the final output.
Experimental results on the ShapeNet, Pix3D, and Things3D benchmarks show that
Pix2Vox++ performs favorably against state-of-the-art methods in terms of both
accuracy and efficiency.

###### Keywords:

3D object reconstruction Multi-scale Context-aware Convolutional neural
network

††journal: International Journal of Computer Vision

##  1 Introduction

Figure 1: Overview of the proposed Pix2Vox++. The network recovers the 3D
shape of an object from arbitrary (uncalibrated) single or multiple images.
The reconstruction result can be refined when more input images are available.
Note that the weights of the encoder and decoder are shared among all views.

Inferring the complete and precise 3D shape of an object is essential in
robotics, 3D modeling and animation, object recognition, and medical
diagnosis. Traditional methods, such as Structure from Motion (SfM) (Özyeşil
et al., 2017) and Simultaneous Localization and Mapping (SLAM) (Fuentes-
Pacheco et al., 2015), match features across images captured from slightly
different views, and then use the triangulation principle to recover 3D
coordinates of the image pixels. Although these methods can produce 3D
reconstruction with satisfactory quality, they typically capture multiple
images of the same object using well-calibrated cameras, which is not
practical or feasible in some situations (Yang et al., 2019).

Recently, several deep learning-based approaches, including 3D-R2N2 (Choy et
al., 2016), LSM (Kar et al., 2017), DeepMVS (Huang et al., 2018), RayNet
(Paschalidou et al., 2018), and AttSets (Yang et al., 2020), recover the 3D
shape of an object from one or more RGB images without complex camera
calibration and show promising results. Both 3D-R2N2 (Choy et al., 2016) and
LSM (Kar et al., 2017) formulate multi-view 3D reconstruction as a sequence
learning problem and fuse multiple feature maps extracted by a shared encoder
from input images using recurrent neural networks (RNNs). The feature maps are
incrementally refined when more views of the object are available. However,
RNN-based methods suffer from three limitations. First, RNNs are unable to
consistently estimate the 3D shape of an object due to permutation variance
(Vinyals et al., 2016) when given the same images with different orders.
Second, important features of early input images may be forgotten (Pascanu et
al., 2013) due to long-term memory loss in RNNs. Third, RNN-based methods are
time-consuming since input images are processed sequentially without
parallelization (Hwang and Sung, 2015).

To overcome the shortcomings of the RNN-based methods, DeepMVS (Huang et al.,
2018) employs max pooling to aggregate deep features across a set of unordered
images for multi-view stereo reconstruction. RayNet (Paschalidou et al., 2018)
applies average pooling to aggregate the deep features extracted from the same
voxel to recover the 3D structure. Both max and average pooling eliminate the
above limitations of RNNs, but they capture only max or mean values without
learning to attentively preserve useful information. Very recent AttSets (Yang
et al., 2020) uses an attentional aggregation module to automatically predict
a weight matrix as attention scores for input features. However, aggregating
features before the decoder is challenging for images with complex backgrounds
and may cause problems in reconstructing objects in real-world scenarios.

To address the issues mentioned above, we propose Pix2Vox++, a novel framework
for single-view and multi-view 3D reconstruction. It contains four modules:
encoder, multi-scale context-aware fusion, decoder, and refiner. As shown in
Figure 1, the encoder and decoder generate coarse 3D volumes from multiple
input images in parallel, which eliminate the effect of orders of input images
and accelerate computation. The multi-scale context-aware fusion module then
selects high-quality reconstructions from all coarse 3D volumes and generates
a fused 3D volume, which exploits information from all input images without
long-term memory loss. Finally, the refiner further corrects the wrongly
recovered parts of the fused 3D volumes to obtain a refined reconstruction.

The contributions can be summarized as follows:

  * •

We present a unified framework for both single-view and multi-view 3D object
reconstruction, namely Pix2Vox++. It is composed of a well-designed encoder,
decoder, and refiner, which shows strong abilities to handle 3D reconstruction
in both synthetic and real-world images.

  * •

We propose a multi-scale context-aware fusion module to adaptively select
high-quality reconstructions for each part from different coarse 3D volumes in
parallel to produce a fused reconstruction of the whole object.

  * •

We construct a large-scale dataset, named Things3D, containing 1.68M images of
280K objects collected from over 39K indoor scenarios. To the best of our
knowledge, it is the first large-scale dataset for multi-view 3D object
reconstruction from naturalistic images.

  * •

Experimental results on the ShapeNet, Pix3D, and Things3D datasets demonstrate
that the proposed approaches outperform state-of-the-art methods in terms of
both accuracy and efficiency.

A preliminary version of this work has been published in ICCV 2019 (Xie et
al., 2019). We make several extensions in this work compared to the
preliminary version. First, we replace VGG (Simonyan and Zisserman, 2015) with
ResNet (He et al., 2016) as the new backbone network. The improved method
contains 25% fewer parameters and is 5% faster during inference compared to
the one in the preliminary version. In addition, there is a 1.5% increase in
Intersection-over-Union (IoU) on ShapeNet. Second, we propose the multi-scale
context-aware fusion to aggregate multi-scale features from multiple coarse 3D
volumes. Compared to the context-aware fusion in the preliminary version, it
brings about a 1% increase in IoU for multi-view reconstruction at $128^{3}$
resolution. Third, we add several layers in the decoder to generate 3D volumes
with higher resolutions of $64^{3}$ and $128^{3}$ which preserve better
details of 3D objects. Finally, we propose a large-scale naturalistic dataset
for multi-view 3D reconstruction, which provides 708 times more 3D models than
Pix3D (Sun et al., 2018). Codes and pretrained models are publicly available
at https://gitlab.com/hzxie/Pix2Vox.

##  2 Related Work

Figure 2: The network architectures of Pix2Vox++/F (top) and Pix2Vox++/A
(bottom) for low-resolution reconstruction. EDLoss and RLoss are defined in
Equation 3. To reduce the model size, the refiner is removed in Pix2Vox++/F.

In this section, we review 3D object reconstruction methods closely related to
this work. Comprehensive reviews of 3D object reconstruction approaches can be
found in Han et al. (2019).

Single-view 3D Reconstruction. Predicting the complete 3D shape of an object
from a single image is a long-standing and extremely challenging task. Many
attempts have been made to address this issue, such as Shape from X (Barron
and Malik, 2015), where X may represent silhouettes (Dibra et al., 2017),
shading (Richter and Roth, 2015), or texture (Witkin, 1981). However, these
methods are rarely applied to real-world scenarios, as they all require strong
presumptions and abundant expertise in natural images (Zhang et al., 2019).
With the success of generative adversarial networks (GANs) (Goodfellow et al.,
2014) and variational autoencoders (VAEs) (Kingma and Welling, 2014), 3D-VAE-
GAN (Wu et al., 2016) uses GAN and VAE to generate 3D reconstructions by
taking a single-view image as input. MarrNet (Wu et al., 2017) reconstructs 3D
objects by estimating depth, surface normals, and silhouettes of 2D images.
ShapeHD (Wu et al., 2018) extends MarrNet by incorporating a shape naturalness
network to improve reconstruction results. Kato and Harada (Kato and Harada,
2019) adopt a discriminator to ensure that reconstructed shapes are reasonable
from any viewpoint. OGN (Tatarchenko et al., 2017) uses octree to represent
high-resolution 3D volumes with a limited memory budget. Matryoshka Networks
(Richter and Roth, 2018) recursively decomposes a 3D shape into nested shape
layers, which outperforms octree-based reconstruction methods. Recently,
several representations for 3D models, including point cloud (Fan et al.,
2017), mesh (Wang et al., 2018), and signed distance field (Xu et al., 2019),
have been adopted in 3D object reconstruction to reduce memory requirements
for high-resolution 3D volumetric grids. PSG (Fan et al., 2017) firstly
recovers a point cloud from a single image with deep neural networks.
Pixel2Mesh (Wang et al., 2018) is the first to reconstruct the 3D shape in a
triangular mesh from a single image. DISN (Xu et al., 2019) predicts the
underlying signed distance field given a single input image. With available
fine-grained 3D part annotations (Mo et al., 2019b), several methods
(Paschalidou et al., 2020; Mo et al., 2019a; Zhu et al., 2018) perform 3D
reconstruction by compositing 3D parts in a hierarchical manner. However,
single-view 3D object reconstruction is an ill-posed and inherently ambiguous
problem since partial observation of an object can theoretically be associated
with an infinite number of possible 3D models.

Multi-view 3D Reconstruction. Traditionally, 3D dense reconstruction in SfM
and SLAM requires a collection of RGB images (Harltey and Zisserman, 2006).
The 3D structure of an object is recovered by dense feature extraction and
matching, or by minimizing reprojection errors (Cadena et al., 2016). However,
the matching process becomes extremely difficult when multiple viewpoints are
separated by a large margin. Furthermore, scanning all surfaces of an object
before reconstruction is sometimes impossible, leading to incomplete 3D shapes
with occluded or hollowed-out areas (Yang et al., 2019). Recently, deep neural
networks have been designed to learn the 3D shape from multiple RGB images.
Both 3D-R2N2 (Choy et al., 2016) and LSM (Kar et al., 2017) are RNN-based,
resulting in the networks being permutation variant and inefficient for
aggregating features from long sequence images. DeepMVS (Huang et al., 2018)
and RayNet (Paschalidou et al., 2018) employ max and average pooling to
aggregate deep features. Recent AttSets (Yang et al., 2020) uses an
attentional aggregation module to effectively aggregate deep features.
However, these methods capture only partial information, ignoring many deep
features, which may lead to low-quality reconstruction. In addition to
volumetric representations, recent works also reconstruct 3D objects in the
form of point clouds and meshes. Lin et al. (2018) use 2D convolutional
operations to predict a dense point cloud from multiple viewpoints and jointly
apply geometric reasoning with 2D projection optimization. Pixel2Mesh++ (Wen
et al., 2019) recovers 3D mesh by leveraging cross-view information with a
graph convolutional network. Lin et al. (2019) reconstruct 3D objects from
aligned videos by optimizing object meshes for multi-view photometric
consistency while constraining mesh deformations with a shape prior. These
methods require extrinsic camera parameters or aligned images as input.
However, it is not always feasible to obtain extrinsic camera parameters,
especially from those scenarios that viewpoints are separated by a large
margin.

##  3 The Proposed Method: Pix2Vox++

###  3.1 Overview

The proposed Pix2Vox++ aims to reconstruct the 3D shape of an object from
either single or multiple RGB images. The 3D shape of an object is represented
by a 3D voxel grid, where $0$ and $1$ denote an empty cell and an occupied
cell, respectively. The key components of Pix2Vox++ are illustrated in Figure
1. First, the encoder produces feature maps from input images. Second, the
decoder takes each feature map as input and correspondingly generates a coarse
3D volume. Third, single or multiple 3D volumes are forwarded to the multi-
scale context-aware fusion module that adaptively selects high-quality
reconstructions for different parts from all coarse 3D volumes in parallel and
generates a fused 3D volume. Finally, the refiner further corrects the wrongly
recovered parts of the fused 3D volume to produce the final reconstruction
result.

###  3.2 Network Architecture

To achieve a good balance between accuracy and model size, we implement two
versions of the proposed framework: Pix2Vox++/F and Pix2Vox++/A, as shown in
Figure 2. The former involves much fewer parameters and lower computational
complexity. The latter has more parameters, which can reconstruct more
accurate 3D shapes but has higher computational complexity.

####  3.2.1 Encoder

The encoder aims to compute a set of features for the decoder to recover the
3D shape of the object. The first three convolutional blocks of ResNet (He et
al., 2016) are used to obtain a $512\times 28^{2}$ feature map from a
$224\times 224\times 3$ image. We adopt ResNet-18 and ResNet-50 for
Pix2Vox++/F and Pix2Vox++/A, respectively. ResNet is followed by three sets of
2D convolutional layers, batch normalization layers, and ReLU layers to embed
semantic information into feature maps. The kernel sizes of the three
convolutional layers are $3^{2}$, with padding of $1$. There is a max pooling
layer with a kernel size of $2^{2}$ after the second and third ReLU layers. In
Pix2Vox++/F, the output channels in the convolutional layer are numbered
$128$, $64$ and $64$, respectively. In Pix2Vox++/A, the output channels of the
three convolutional layers are $512$, $256$ and $256$, respectively. The
feature maps produced by Pix2Vox++/F and Pix2Vox++/A are of sizes $64\times
7^{2}$ and $256\times 7^{2}$, respectively.

####  3.2.2 Decoder

The decoder is responsible for transforming information of 2D feature maps
into 3D volumes.

Low-resolution Reconstruction. There are five 3D transposed convolutional
layers in both Pix2Vox++/F and Pix2Vox++/A. Specifically, the first four
transposed convolutional layers are of kernel sizes $4^{3}$, with strides of
$2$ and paddings of $1$. There is an additional transposed convolutional layer
with a bank of $1^{3}$ filter. Each transposed convolutional layer is followed
by a batch normalization layer and a ReLU activation except for the last layer
followed by a sigmoid function. In Pix2Vox++/F, the output channels of
transposed convolutional layers are numbered $128$, $64$, $32$, $8$ and $1$,
respectively. In Pix2Vox++/A, the output channel numbers of the five
transposed convolutional layers are $512$, $128$, $32$, $8$ and $1$,
respectively. The decoder outputs a $32^{3}$ voxelized shape in the object’s
canonical view.

High-resolution Reconstruction. To generate 3D volumes at $64^{3}$ resolution,
there are six transposed convolutional layers in the decoders of Pix2Vox++/F
and Pix2Vox++/A. Each convolutional layer is with a batch normalization and a
ReLU activation except for the last layer followed by a sigmoid function. In
Pix2Vox++/F, the output channels of the six transposed convolutional layers
are numbered $128$, $64$, $32$, $16$, $8$ and $1$, respectively. In
Pix2Vox++/A, the output channel numbers of transposed convolutional layers are
$512$, $128$, $32$, $16$, $8$ and $1$, respectively. To generate 3D volumes at
$128^{3}$ resolution, there are seven transposed convolutional layers in the
decoders of Pix2Vox++/F and Pix2Vox++/A. For Pix2Vox++/F, the output channels
of the seven transposed convolutional layers are numbered $128$, $64$, $32$,
$32$, $32$, $8$ and $1$, respectively. For Pix2Vox++/A, the output channel
numbers of the seven transposed convolutional layers are $512$, $128$, $32$,
$32$, $32$, $8$ and $1$, respectively.

####  3.2.3 Multi-scale Context-aware Fusion

Different parts of an object can be seen from different viewpoints. The
reconstruction qualities of visible parts are much higher than those of
invisible parts. Inspired by this observation, we propose a multi-scale
context-aware fusion module to adaptively select high-quality reconstruction
for each part from different coarse 3D volumes. The selected reconstructions
are fused to produce a 3D volume of the entire object. As shown in Figure 3,
the multi-scale context-aware fusion generates higher scores for high-quality
reconstructions, which can eliminate the effect of the missing of the wrongly
recovered parts.

As shown in Figure 4, given coarse 3D volumes and the corresponding context,
the multi-scale context-aware fusion module generates a score map for each
coarse volume and then fuses them into one volume by weighted summation of all
coarse volumes according to their score maps. The spatial information of
voxels is preserved in the multi-scale context-aware fusion module, and thus
Pix2Vox++ can use multi-view information to recover the structure of an object
better.

Figure 3: Visualization of the score maps in the multi-scale context-aware
fusion module.

Figure 4: An overview of the multi-scale context-aware fusion module. It aims
to select high-quality reconstructions for each part to construct the final
results. The objects in the bounding box describe the procedure score
calculation for a coarse volume $v^{c}_{1}$. The other scores are calculated
according to the same procedure. Note that the weights of the context scoring
network are shared among different views.

Deeper convolutional layers have larger receptive fields, which help to
explore contextual information in 3D volumes. However, the features in deeper
convolutional layers may lose details of the object. To address this problem,
we concatenate multiple feature maps with different scales to preserve details
in shallower convolutional layers, which is important for recovering details
in high-resolution 3D volumes. Specifically, the multi-scale context-aware
fusion generates the context $c_{r}$ of the $r$-th coarse volume $v_{r}^{c}$
by concatenating the output of the last two layers in the decoder. The context
scoring network then generates a score $m_{r}$ for the context of the $r$-th
coarse voxel. The context scoring network consists of five sets of 3D
convolutional layers, each with a kernel size of $3^{3}$ and padding of $1$,
followed by a batch normalization and a leaky ReLU activation. The numbers of
output channels of the convolutional layers are $9$ except for the last layer
whose number of output channels is $1$. The feature maps generated by the
first four convolutional layers are concatenated and forwarded to the fifth
convolutional layer. The learned score $m_{r}$ for context $c_{r}$ is
normalized across all learned scores. We choose softmax as the normalization
function. Therefore, the score $s_{r}^{(i,j,k)}$ at the position $(i,j,k)$ for
the $r$-th voxel can be calculated as

$s_{r}^{(i,j,k)}=\frac{\exp\left(m_{r}^{(i,j,k)}\right)}{\sum_{p=1}^{n}\exp\left(m_{p}^{(i,j,k)}\right)}$
(1)

where $n$ represents the number of views. Finally, the fused voxel $v^{f}$ is
produced by summing up the product of coarse voxels and the corresponding
scores altogether.

$v^{f}=\sum_{r=1}^{n}s_{r}v_{r}^{c}$ (2)

####  3.2.4 Refiner

The refiner can be seen as a residual network, which aims to correct the
wrongly recovered parts of a 3D volume. It follows the concept of a 3D
encoder-decoder with U-net connections (Ronneberger et al., 2015) that
preserves the local structure in the fused volume.

Low-resolution Reconstruction. To generate 3D volumes at $32^{3}$ resolution,
the encoder has three 3D convolutional layers, each with a bank of $4^{3}$
filters with padding of $2$, followed by a batch normalization layer, a leaky
ReLU activation and a max pooling layer with kernel size of $2^{3}$. The
output channels of convolutional layers are numbered $32$, $64$ and $128$,
respectively. The encoder is finally followed by two fully connected layers
with dimensions of $2048$ and $8192$. The decoder consists of three transposed
convolutional layers, each with a bank of $4^{3}$ filters with padding of $2$
and stride of $1$. Except for the last transposed convolutional layer which is
followed by a sigmoid function, other layers are followed by a batch
normalization layer, and a ReLU activation. The numbers of output channels of
the transposed convolutional layers are $64$, $32$ and $1$, respectively.

High-resolution Reconstruction. To generate 3D volumes at $64^{3}$ resolution,
we add a 3D convolutional layer before the first convolutional layer and a 3D
transposed convolutional layer before the last layer. Both layers are with
output channels of $16$, kernel sizes of $4^{3}$, paddings of $2$, followed by
batch normalization layers and leaky ReLU activations. There is a max pooling
layer with a kernel size of $2^{3}$ after the added 3D convolutional layer. To
generate 3D volumes at $128^{3}$ resolution, there are five 3D convolutional
layers in the encoder, whose output channels are $8$, $16$, $32$, $64$ and
$128$, respectively. In the decoder, there are five 3D transposed
convolutional layers with kernel sizes of $4^{3}$, paddings of $2$, and stride
of $1$. The output channels of the five layers are $64$, $32$, $16$, $8$ and
$1$, respectively.

###  3.3 Loss Function

The loss function of the network is defined as the mean value of the voxel-
wise binary cross entropies between the reconstructed object and the ground
truth. More formally, it can be defined as

$\ell=\frac{1}{N}\sum_{i=1}^{N}\left[gt_{i}\log(p_{i})+(1-gt_{i})\log(1-p_{i})\right]$
(3)

where $N$ denotes the number of voxels in the ground truth. $p_{i}$ and
$gt_{i}$ represent the predicted occupancy and the corresponding ground truth.
The smaller the $\ell$ value is, the closer the prediction is to the ground
truth.

##  4 The Proposed Dataset: Things3D

There are several datasets available for 3D object reconstruction, including
ShapeNet (Wu et al., 2015) and Pix3D (Sun et al., 2018). ShapeNet is a large
dataset for 3D models, but does not contain naturalistic background images.
Pix3D has real images, but it only contains 395 3D models and 10,069 images,
which is not enough to train networks (Tatarchenko et al., 2019).

To generate a large-scale dataset for 3D object reconstruction with
naturalistic backgrounds, Su et al. (2015) and Lin et al. (2019) randomly warp
and crop spherical images from the SUN database (Xiao et al., 2010) and SUN360
database (Xiao et al., 2012) to sample background images, respectively.
Consequently, multi-view images are obtained by compositing foreground and
background images together at the corresponding camera poses. In contrast, we
generate more realistic images from diverse virtual scenes with Blender, an
open-source 3D creation suite, where we can easily control camera pose and
location, as well as lighting conditions.

We present a large-scale naturalistic dataset for 3D object reconstruction,
named Things3D, which contains 1.68M images of 280K objects (with 21K unique
objects) collected from over 39K SUNCG (Song et al., 2017) indoor scenarios.
Sample images and corresponding CAD models are shown in Figure 5. To increase
the diversity of 3D objects, we use 3D models in the ShapeNet dataset instead
of the ones in the SUNCG dataset. In particular, each 3D model in SUNCG scenes
is replaced with one that is randomly selected from the same category in the
ShapeNet dataset. In addition, the replaced 3D model is of equal or smaller
size than the original. We randomly sample 24 viewing spheres for each object
with yaw $\in[0,360)$, pitch $=30$, roll $=0$ degrees. The distance between
the camera and the object is set to 10 unit length. The camera has a focal
length of 96 mm. The power of light is uniformly sampled from $[500,2000]$,
and the specular of light is randomly sampled from $[0.75,3]$. Both the camera
and the light track to the rendered object. Specially, we ignore rendered
images if more than 12.5% of the object is occluded. The images are of
resolution $256\times 256$. In addition to rendered images, information about
the canonical orientation and ground truth for objects is provided as well.
The data generation process lasts 32 days and runs on 15 servers with 4 Intel
Xeon E5-2682<EMAIL_ADDRESS>CPUs and 256 GB RAM. The dataset is available at
https://gateway.haozhexie.com/?fileName=Things3D.

Figure 5: Sample images and the corresponding CAD models in Things3D, where
each 3D model is rendered from a diverse set of naturalistic scenes.

##  5 Experiments

Table 1: Comparison of single-view 3D object reconstruction on ShapeNet at
$32^{3}$ resolution. We report the mean IoU per category. The best number for
each category is highlighted in bold.

Category | 3D-R2N2 | OGN | Matryoshka | AtlasNet | Pixel2Mesh | OccNet | IM-Net | AttSets | Pix2Vox++/F | Pix2Vox++/A
---|---|---|---|---|---|---|---|---|---|---
airplane | 0.513 | 0.587 | 0.647 | 0.493 | 0.508 | 0.532 | 0.702 | 0.594 | 0.607 | 0.674
bench | 0.421 | 0.481 | 0.577 | 0.431 | 0.379 | 0.597 | 0.564 | 0.552 | 0.544 | 0.608
cabinet | 0.716 | 0.729 | 0.776 | 0.257 | 0.732 | 0.674 | 0.680 | 0.783 | 0.782 | 0.799
car | 0.798 | 0.828 | 0.850 | 0.282 | 0.670 | 0.671 | 0.756 | 0.844 | 0.841 | 0.858
chair | 0.466 | 0.483 | 0.547 | 0.328 | 0.484 | 0.583 | 0.644 | 0.559 | 0.548 | 0.581
display | 0.468 | 0.502 | 0.532 | 0.457 | 0.582 | 0.651 | 0.585 | 0.565 | 0.529 | 0.548
lamp | 0.381 | 0.398 | 0.408 | 0.261 | 0.399 | 0.474 | 0.433 | 0.445 | 0.448 | 0.457
speaker | 0.662 | 0.637 | 0.701 | 0.296 | 0.672 | 0.655 | 0.683 | 0.721 | 0.721 | 0.721
rifle | 0.544 | 0.593 | 0.616 | 0.573 | 0.468 | 0.656 | 0.723 | 0.601 | 0.594 | 0.617
sofa | 0.628 | 0.646 | 0.681 | 0.354 | 0.622 | 0.669 | 0.694 | 0.703 | 0.696 | 0.725
table | 0.513 | 0.536 | 0.573 | 0.301 | 0.536 | 0.659 | 0.621 | 0.590 | 0.609 | 0.620
telephone | 0.661 | 0.702 | 0.756 | 0.543 | 0.762 | 0.794 | 0.762 | 0.743 | 0.782 | 0.809
watercraft | 0.513 | 0.632 | 0.591 | 0.355 | 0.471 | 0.579 | 0.607 | 0.601 | 0.583 | 0.603
Overall | 0.560 | 0.596 | 0.635 | 0.352 | 0.552 | 0.626 | 0.659 | 0.642 | 0.645 | 0.670

Table 2: Comparison of single-view 3D object reconstruction on ShapeNet. We
report the mean F-Score@1% per category. For voxel reconstruction methods, the
points are sampled from triangular meshes generated by the marching cube
algorithm. The best number for each category is highlighted in bold.

Category | 3D-R2N2 | OGN | Matryoshka | AtlasNet | Pixel2Mesh | OccNet | IM-Net | AttSets | Pix2Vox++/F | Pix2Vox++/A
---|---|---|---|---|---|---|---|---|---|---
airplane | 0.412 | 0.487 | 0.446 | 0.415 | 0.376 | 0.494 | 0.598 | 0.489 | 0.493 | 0.583
bench | 0.345 | 0.364 | 0.424 | 0.439 | 0.313 | 0.318 | 0.361 | 0.406 | 0.399 | 0.478
cabinet | 0.327 | 0.316 | 0.381 | 0.350 | 0.450 | 0.449 | 0.345 | 0.367 | 0.363 | 0.408
car | 0.481 | 0.514 | 0.481 | 0.319 | 0.486 | 0.315 | 0.304 | 0.497 | 0.523 | 0.564
chair | 0.238 | 0.226 | 0.302 | 0.406 | 0.386 | 0.365 | 0.442 | 0.334 | 0.262 | 0.309
display | 0.227 | 0.215 | 0.400 | 0.451 | 0.319 | 0.468 | 0.466 | 0.310 | 0.253 | 0.296
lamp | 0.267 | 0.249 | 0.276 | 0.217 | 0.219 | 0.361 | 0.371 | 0.315 | 0.287 | 0.315
speaker | 0.231 | 0.225 | 0.279 | 0.199 | 0.190 | 0.249 | 0.200 | 0.211 | 0.256 | 0.152
rifle | 0.521 | 0.541 | 0.514 | 0.405 | 0.340 | 0.219 | 0.407 | 0.524 | 0.553 | 0.574
sofa | 0.274 | 0.290 | 0.326 | 0.337 | 0.343 | 0.324 | 0.354 | 0.334 | 0.320 | 0.377
table | 0.340 | 0.352 | 0.374 | 0.373 | 0.502 | 0.549 | 0.461 | 0.419 | 0.385 | 0.406
telephone | 0.504 | 0.528 | 0.598 | 0.545 | 0.485 | 0.273 | 0.423 | 0.469 | 0.588 | 0.633
watercraft | 0.305 | 0.328 | 0.360 | 0.296 | 0.266 | 0.347 | 0.369 | 0.315 | 0.346 | 0.390
Overall | 0.351 | 0.368 | 0.391 | 0.362 | 0.398 | 0.393 | 0.405 | 0.395 | 0.394 | 0.436

Figure 6: Example of single-view 3D object reconstruction on ShapeNet. For
voxel reconstruction methods, the output 3D volumes are at $32^{3}$
resolution.

In this section, we present extensive experimental evaluations of Pix2Vox++ on
the ShapeNet (Wu et al., 2015), Pix3D (Sun et al., 2018), and Things3D
datasets. We first describe the datasets and evaluation protocols. Next, we
demonstrate the implementation details of the proposed methods. Finally, we
report experimental evaluations of the proposed methods against state-of-the-
art methods.

###  5.1 Datasets

ShapeNet. The ShapeNet dataset (Wu et al., 2015) is a collection of 3D CAD
models organized according to the WordNet taxonomy. We use a subset of the
ShapeNet dataset consisting of 44K models and 13 major categories following
Choy et al. (2016). More specifically, we use renderings provided by 3D-R2N2
which contains 24 random views of size $137\times 137$ for each 3D model. We
also apply a uniform colored background to the image during training and
testing.

Pix3D. The Pix3D dataset (Sun et al., 2018) provides perfectly aligned real-
world images and CAD models. The dataset contains 395 3D models of nine object
classes. Each model is associated with a set of real images, capturing the
exact object in diverse environments. Following Sun et al. (2018), we use
2,894 untruncated and unoccluded images from the chair category for testing in
the following experiments.

Things3D. The Things3D dataset proposed in this paper contains 1.68M images of
280K objects collected from over 39K SUNCG (Song et al., 2017) indoor
scenarios. See Section 4 for more details.

###  5.2 Metrics

To evaluate the reconstruction quality of the proposed methods, we binarize
probabilities at a fixed threshold of 0.3 and use intersection over union
(IoU) as a similarity measure between prediction and ground truth. More
formally,

${\rm IoU}=\frac{\sum_{i,j,k}{\rm I}(\hat{p}_{(i,j,k)}>t){\rm
I}(\hat{p}_{(i,j,k)})}{\sum_{i,j,k}{\rm I}\left[{\rm
I}(\hat{p}_{(i,j,k)}>t)+{\rm I}(p_{(i,j,k)})\right]}$ (4)

where $\hat{p}_{(i,j,k)}$ and $p_{(i,j,k)}$ represent the predicted occupancy
probability and ground truth at $(i,j,k)$, respectively. ${\rm I}(\cdot)$ is
an indicator function and $t$ denotes a voxelization threshold. Higher IoU
values indicate better reconstruction results.

Following Tatarchenko et al. (2019), we also take F-Score as an extra metric
to evaluate the performance of 3D reconstruction results, which can be defined
as

$\textnormal{F-Score}(d)=\frac{2P(d)R(d)}{P(d)+R(d)}$ (5)

where $P(d)$ and $R(d)$ denote the precision and recall for a distance
threshold $d$, respectively. They can be computed as

$P(d)=\frac{1}{n_{\mathcal{R}}}\sum_{r\in\mathcal{R}}\left[\min_{g\in\mathcal{G}}||g-r||<d\right]$
(6)

$R(d)=\frac{1}{n_{\mathcal{G}}}\sum_{g\in\mathcal{G}}\left[\min_{r\in\mathcal{R}}||g-r||<d\right]$
(7)

where $\mathcal{R}$ and $\mathcal{G}$ denote the reconstructed and ground
truth point clouds, respectively. $n_{\mathcal{R}}$ and $n_{\mathcal{G}}$ are
the numbers of points in $\mathcal{R}$ and $\mathcal{G}$, respectively. For
voxel reconstruction methods, we first apply the marching cubes algorithm
(Lorensen and Cline, 1987) to generate the object surface. We then sample
8,192 points from the object surface to compute F-Score between prediction and
ground truth. For mesh and signed distance field reconstruction methods, we
also sample 8,192 points from the object surface to compute F-Score. Higher
F-Score values indicate better reconstructions.

###  5.3 Implementation Details

We train the proposed methods with batch size $64$ using $224\times 224$ RGB
images as input. The output voxelized reconstruction is $32^{3}$ in size. We
implement our networks in PyTorch (Paszke et al., 2019) and train both
Pix2Vox++/F and Pix2Vox++/A using an Adam optimizer (Kingma and Ba, 2015) with
a $\beta_{1}$ of $0.9$ and a $\beta_{2}$ of $0.999$. The initial learning rate
is set to $0.001$ and decayed by 2 after 150 epochs. First, we train networks
except for the multi-scale context-aware fusion using single-view images for
250 epochs. We then train the networks using multi-view images for 100 epochs.

###  5.4 Evaluation on the ShapeNet Dataset

Table 3: Comparison of multi-view 3D object reconstruction on ShapeNet at $32^{3}$ resolution. We report the mean IoU and F-Score@1% for all categories. Methods (IoU) |  1 view |  2 views |  3 views |  4 views |  5 views |  8 views | 12 views | 16 views | 20 views
---|---|---|---|---|---|---|---|---|---
Metric: IoU
3D-R2N2 |  0.560 |  0.603 |  0.617 |  0.625 |  0.634 |  0.635 | 0.636 | 0.636 | 0.636
AttSets |  0.642 |  0.662 |  0.670 |  0.675 |  0.677 |  0.685 | 0.688 | 0.692 | 0.693
Pix2Vox++/F |  0.645 |  0.669 |  0.678 |  0.682 |  0.685 |  0.690 | 0.692 | 0.693 | 0.694
Pix2Vox++/A |  0.670 |  0.695 |  0.704 |  0.708 |  0.711 |  0.715 | 0.717 | 0.718 | 0.719
Metric: F-Score@1%
3D-R2N2 |  0.351 |  0.368 |  0.372 |  0.378 |  0.382 |  0.383 | 0.382 | 0.382 | 0.383
AttSets |  0.395 |  0.418 |  0.426 |  0.430 |  0.432 |  0.444 | 0.445 | 0.447 | 0.448
Pix2Vox++/F |  0.394 |  0.422 |  0.432 |  0.437 |  0.440 |  0.446 | 0.449 | 0.450 | 0.451
Pix2Vox++/A |  0.436 |  0.452 |  0.455 |  0.457 |  0.458 |  0.459 | 0.460 | 0.461 | 0.462

####  5.4.1 Single-view 3D Object Reconstruction

To evaluate the performance of the proposed methods in handling clean
background images, we compare our methods against several state-of-the-art
methods, including 3D-R2N2 (Choy et al., 2016), OGN (Tatarchenko et al.,
2017), Matryoshaka (Richter and Roth, 2018), AtlasNet (Groueix et al., 2018),
Pixel2Mesh (Wang et al., 2018), OccNet (Mescheder et al., 2019), IM-Net (Chen
and Zhang, 2019), and AttSets (Yang et al., 2020). Tables 1 and 2 show the IoU
and F-Score@1% of all methods on the ShapeNet test set. Both Pix2Vox++/F and
Pix2Vox++/A outperform all competitive methods in terms of both IoU and
F-Score@1%. Figure 6 provides qualitative results for single-view
reconstruction results showing that Pix2Vox++/F and Pix2Vox++/A generate more
visually compelling 3D shapes than other methods.

####  5.4.2 Multi-view 3D Object Reconstruction

Figure 7: Example of multi-view 3D object reconstruction on ShapeNet at
$32^{3}$ resolution.

To evaluate the performance of reconstructing 3D objects from multi-view
images, we compare the proposed methods with 3D-R2N2 (Choy et al., 2016) and
AttSets (Yang et al., 2020). As shown in Table 3, the proposed Pix2Vox++/F and
Pix2Vox++/A consistently outperform 3D-R2N2 and AttSets in all numbers of
views. Figure 7 shows several examples reconstructed from three input images.
Both Pix2Vox++/F and Pix2Vox++/A are able to recover better details than other
methods. Pix2Vox++/A performs better in 3D object reconstruction by comparing
with Pix2Vox++/F.

To provide a detailed analysis of the multi-scale context-aware fusion, we
visualize score maps of corresponding coarse volumes when reconstructing the
3D shape of a table and a chair, as shown in Figure 3. The chair seat on the
right is of low quality, and the score of the corresponding part is lower than
that in the other coarse volumes. The fused 3D volume is obtained by combining
selected high-quality reconstruction parts, where bad reconstructions can be
effectively eliminated by our scoring scheme.

####  5.4.3 Higher-Resolution 3D Object Reconstruction

Table 4: Comparison of single-view and multi-view 3D object reconstruction on
ShapeNet-Cars at $64^{3}$ and $128^{3}$ resolutions. We report the mean IoU
and F-Score@1% of all models.

Methods | IoU | F-Score@1%
---|---|---
1 view |  2 views |  4 views |  8 views |  1 view |  2 views |  4 views |  8 views
Resolution: $64^{3}$ |  |  |  |  |  |  |  |
OGN |  0.771 |  N/A |  N/A |  N/A |  0.361 |  N/A |  N/A |  N/A
Matryoshka |  0.784 |  N/A |  N/A |  N/A |  0.380 |  N/A |  N/A |  N/A
Pix2Vox++/F |  0.793 |  0.807 |  0.811 |  0.815 |  0.401 |  0.429 |  0.439 |  0.453
Pix2Vox++/A |  0.803 |  0.813 |  0.815 |  0.819 |  0.418 |  0.448 |  0.450 |  0.457
Resolution: $128^{3}$ |  |  |  |  |  |  |  |
OGN |  0.782 |  N/A |  N/A |  N/A |  0.390 |  N/A |  N/A |  N/A
Matryoshka |  0.794 |  N/A |  N/A |  N/A |  0.426 |  N/A |  N/A |  N/A
Pix2Vox++/F |  0.817 |  0.832 |  0.838 |  0.840 |  0.459 |  0.502 |  0.520 |  0.528
Pix2Vox++/A |  0.826 |  0.837 |  0.841 |  0.843 |  0.475 |  0.509 |  0.521 |  0.539

Figure 8: Example of single-view 3D object reconstruction on ShapeNet-Cars at
$128^{3}$ resolution.

Low-resolution 3D volumes are naturally limited to the low level of details
they can represent. To evaluate the performance of Pix2Vox++ in high-
resolution 3D reconstruction, we compare it to Matryoshka Networks (Richter
and Roth, 2018) and OGN (Tatarchenko et al., 2017). We follow the experimental
setup of OGN and predict 3D volumes of ShapeNet-Cars at $64^{3}$ and $128^{3}$
resolutions given a single RGB image. We then upsample the predicted 3D
volumes to $256^{3}$ resolution and compute IoU with ground truth shapes. To
make a fair comparison, we use the same dataset split and ground truth shapes
generated by OGN. We report quantitative results in Table 4 and qualitative
results at $128^{3}$ resolution in Figure 8. Experimental results on single-
view reconstruction show that both Pix2Vox++/F and Pix2Vox++/A outperform
Matryoshka Networks and OGN at $64^{3}$ resolution. At $128^{3}$ resolution,
Pix2Vox++/A outforms Matryoshka Networks and OGN. Pix2Vox++/F is comparable to
both competitive methods. As shown in Figure 8, our methods recover better
details than compared methods. Furthermore, we provide results for high-
resolution reconstruction from multi-view images. Experimental results show
that Pix2Vox++/A outperforms Pix2Vox++/F in all numbers of views.

###  5.5 Evaluation on the Pix3D Dataset

To evaluate the performance of the proposed methods on real-world images, we
evaluate our methods for single-view reconstruction on the Pix3D dataset. We
train Pix2Vox++/F and Pix2Vox++/A on ShapeNet-Chairs and Things3D-Chairs and
test both networks on the chair category of the Pix3D dataset. As shown in
Table 5, our networks trained on Things3D-Chairs have better results than
those trained on ShapeNet-Chairs.

Following Pix3D (Sun et al., 2018), we use Render for CNN (Su et al., 2015) to
generate 60 images for each chair in the ShapeNet dataset by adding random
backgrounds sampled from the SUN database (Xiao et al., 2010), i.e. ShapeNet-
Chairs-RfC. We also render a new dataset, called Things-3D-Chairs-RfC, by
rendering chairs in the naturalist scenes where the camera poses are sampled
from a distribution estimated on the Pix3D dataset. Table 5 illustrates that
our networks trained on Things3D-Chairs-RfC have better results in
reconstructing 3D objects in Pix3D than those trained on ShapeNet-Chairs-RfC.
The two above experiments show that the networks trained on datasets generated
by rendering naturalist scenes archive better results than those trained on
datasets with random backgrounds. Furthermore, the proposed Pix2Vox++/A
trained on Things3D-Chairs-RfC archives the best results on the Pix3D dataset,
suggesting that the Thing3D dataset helps the networks generalize better to
real-world datasets.

Table 5: Comparison of single-view 3D object reconstruction on Pix3D at $32^{3}$ resolution. We report the mean IoU and F-Score@1% of the chair category. The best number is highlighted in bold. Method |  IoU |  F-Score@1%
---|---|---
Training on ShapeNet-Chairs
Pix2Vox++/F |  0.179 |  0.012
Pix2Vox++/A |  0.204 |  0.018
Training on Things3D-Chairs
Pix2Vox++/F |  0.256 |  0.028
Pix2Vox++/A |  0.269 |  0.036
Training on ShapeNet-Chairs-RfC
Pix3D |  0.282 |  0.041
Pix2Vox++/F |  0.276 |  0.042
Pix2Vox++/A |  0.292 |  0.068
Training on Things3D-Chairs-RfC
Pix2Vox++/F |  0.297 |  0.072
Pix2Vox++/A |  0.324 |  0.084

Figure 9: Example of multi-view 3D object reconstruction on Things3D at
$32^{3}$ resolution.

Figure 10: Example of single-view 3D object reconstruction on Pix3D at
$32^{3}$ resolution trained on ShapeNet-Chairs-RfC.

###  5.6 Evaluation on the Things3D Dataset

Figure 11: Example of single-view 3D object reconstruction on Things3D. For
voxel reconstruction methods, the output 3D volumes are at $32^{3}$
resolution. Table 6: Comparison of single-view 3D object reconstruction on
Things3D at $32^{3}$ resolution. We report the mean IoU per category. The best
number for each category is highlighted in bold.

Category | 3D-R2N2 | OGN | Matryoshka | AtlasNet | Pixel2Mesh | OccNet | IM-Net | AttSets | Pix2Vox++/F | Pix2Vox++/A
---|---|---|---|---|---|---|---|---|---|---
chair | 0.327 | 0.212 | 0.399 | 0.251 | 0.372 | 0.432 | 0.462 | 0.403 | 0.435 | 0.442
display | 0.240 | 0.153 | 0.332 | 0.259 | 0.311 | 0.328 | 0.324 | 0.301 | 0.324 | 0.349
lamp | 0.257 | 0.189 | 0.323 | 0.196 | 0.306 | 0.361 | 0.328 | 0.334 | 0.350 | 0.362
piano | 0.072 | 0.060 | 0.234 | 0.064 | 0.087 | 0.168 | 0.156 | 0.194 | 0.190 | 0.244
sofa | 0.457 | 0.450 | 0.548 | 0.284 | 0.490 | 0.525 | 0.550 | 0.554 | 0.560 | 0.569
table | 0.159 | 0.116 | 0.305 | 0.137 | 0.247 | 0.317 | 0.297 | 0.306 | 0.305 | 0.320
Overall | 0.313 | 0.244 | 0.395 | 0.228 | 0.360 | 0.414 | 0.419 | 0.400 | 0.419 | 0.430

Table 7: Comparison of single-view 3D object reconstruction on Things3D. We
report the mean F-Score@1% per category and the average F-Score@1% for all
categories. The best number for each category is highlighted in bold.

Category | 3D-R2N2 | OGN | Matryoshka | AtlasNet | Pixel2Mesh | OccNet | IM-Net | AttSets | Pix2Vox++/F | Pix2Vox++/A
---|---|---|---|---|---|---|---|---|---|---
chair | 0.166 | 0.096 | 0.231 | 0.268 | 0.248 | 0.272 | 0.253 | 0.244 | 0.240 | 0.273
display | 0.136 | 0.126 | 0.164 | 0.124 | 0.128 | 0.266 | 0.277 | 0.172 | 0.150 | 0.163
lamp | 0.177 | 0.098 | 0.208 | 0.166 | 0.170 | 0.272 | 0.248 | 0.229 | 0.249 | 0.275
piano | 0.012 | 0.006 | 0.127 | 0.069 | 0.057 | 0.036 | 0.095 | 0.099 | 0.108 | 0.136
sofa | 0.189 | 0.214 | 0.257 | 0.268 | 0.261 | 0.264 | 0.259 | 0.253 | 0.252 | 0.270
table | 0.108 | 0.086 | 0.182 | 0.182 | 0.177 | 0.201 | 0.185 | 0.172 | 0.198 | 0.200
Overall | 0.165 | 0.118 | 0.223 | 0.226 | 0.217 | 0.259 | 0.244 | 0.231 | 0.238 | 0.263

Table 8: Comparison of multi-view 3D object reconstruction on Things3D at $32^{3}$ resolution. We report the mean IoU and F-Score@1% for all categories. Methods |  1 view |  2 views |  3 views |  4 views |  5 views |  6 views |  7 views |  8 views
---|---|---|---|---|---|---|---|---
Metric: IoU
3D-R2N2 |  0.307 |  0.316 |  0.322 |  0.325 |  0.329 |  0.331 |  0.332 |  0.334
AttSets |  0.402 |  0.415 |  0.422 |  0.427 |  0.429 |  0.431 |  0.433 |  0.434
Pix2Vox++/F |  0.417 |  0.433 |  0.442 |  0.447 |  0.451 |  0.454 |  0.456 |  0.458
Pix2Vox++/A |  0.428 |  0.444 |  0.452 |  0.456 |  0.460 |  0.462 |  0.465 |  0.467
Metric: F-Score@1%
3D-R2N2 |  0.142 |  0.148 |  0.151 |  0.153 |  0.156 |  0.157 |  0.158 |  0.159
AttSets |  0.228 |  0.240 |  0.245 |  0.247 |  0.248 |  0.249 |  0.250 |  0.250
Pix2Vox++/F |  0.230 |  0.240 |  0.243 |  0.246 |  0.248 |  0.249 |  0.250 |  0.251
Pix2Vox++/A |  0.260 |  0.271 |  0.274 |  0.275 |  0.276 |  0.277 |  0.278 |  0.279
Table 9: The numbers of parameters, inference time, and the corresponding IoUs of Pix2Vox++/A with different backbone models on ShapeNet. Pretrained Models |  # Parameters (M) |  Inference Time (ms) | IoU
---|---|---|---
|  |  w/o Pretrained |  w/ Pretrained
VGG16 |  97.78 |  11.14 |  0.659 |  0.661
VGG19 |  98.37 |  11.19 |  0.658 |  0.660
ResNet50 |  96.31 |  10.64 |  0.669 |  0.670
DenseNet101 |  102.85 |  16.78 |  0.668 |  0.669
DenseNet169 |  109.02 |  18.26 |  0.668 |  0.669

####  5.6.1 Single-view 3D Object Reconstruction

To evaluate the performance of the proposed methods in dealing with
naturalistic images, we compare our methods to several state-of-the-art
methods on the Things3D test set. We fine-tune all competitive methods on
Things3D training and crop the input images as required by each method. To
make a fair comparison, all methods are fed with the same input images during
testing. The IoU and F-Score@1% are reported in Tables 6 and 7, respectively.
Both Pix2Vox++/F and Pix2Vox++/A outperform all competitive methods. Figure 11
shows the qualitative results, which indicate that Pix2Vox++/A has the best
ability to recover the 3D shapes from a single natural scene image.

####  5.6.2 Multi-view 3D Object Reconstruction

We also compare the proposed methods with 3D-R2N2 (Choy et al., 2016) and
AttSets (Yang et al., 2020) in reconstructing objects in natural scenes from
multi-view images. As mentioned in Section 4, different objects have different
occlusions in different scenes, which leads to different numbers of views for
these objects. To use the same test set in different numbers of views, we only
use test samples with no less than eight rendering images in this experiment.
To make a fair comparison, we feed all methods with the same images and crop
the images as required by each method during testing. Table 8 shows the multi-
view reconstruction results on the Things3D test set. The proposed Pix2Vox++/F
and Pix2Vox++/A consistently outperform 3D-R2N2 and AttSets in all numbers of
views. As shown in Figure 9, Pix2Vox++/F and Pix2Vox++/A perform better at
reconstructing the 3D shape of an object from multiple natural images.

##  6 Analysis and Discussion

###  6.1 Effectiveness of Different Backbone Models

To provide a detailed analysis of different backbone models, we replace
ResNet50 in Pix2Vox++/A with other backbone models, including VGG (Simonyan
and Zisserman, 2015) and DenseNet (Huang et al., 2017). We only use partial
convolutional layers in backbone models that produce feature maps of size
$512\times 28\times 28$ to guarantee that the rest of the network architecture
of the encoder is the same. We report the IoU on ShapeNet in Table 9. Encoders
with the pretrained models perform slightly better than those without
pretrained models. Compared to VGG and DenseNet, the encoder with ResNet50 has
the best performance in terms of both accuracy and efficiency.

###  6.2 Effectiveness of the Refiner

$1$$2$$3$$4$$5$$6$$7$$8$$9$$10$$11$$12$$13$$14$$15$$16$$17$$18$$19$$20$0.660.670.680.690.700.710.72The
Number of ViewsIntersection over Union (IoU)Pix2Vox++/APix2Vox++/A without
Refiner

Figure 12: Effectiveness of the refiner and the number of views on the
evaluation IoU.

Pix2Vox++/A uses a refiner to further correct the wrongly recovered parts in
the fused 3D volume, which has an IoU of $0.670$ for single-view
reconstruction on ShapeNet. In contrast, IoU decreases to $0.658$ without a
refiner. As shown in Figure 12, removing the refiner causes considerable
degeneration in reconstruction accuracy.

###  6.3 Effectiveness of the Camera Parameters

Pix2Vox++ recovers the 3D shape of an object without knowing the camera
parameters. It aligns multi-view features with the supervision of ground truth
3D volumes with canonical orientation. In contrast, LSM (Kar et al., 2017)
aligns multi-view features with the unprojection operation, which requires
camera parameters as input. Table 10 shows multi-view reconstruction results
on ShapeNet compared to LSM. Experimental results show that LSM significantly
outperforms Pix2Vox++/F and Pix2Vox++/A with more than one view, indicating
that precise camera parameters help align features of multi-view images
better.

To further demonstrate the superior ability of the multi-scale context-aware
fusion in multi-view stereo (MVS) systems, we replace the recurrent fusion in
LSM with the multi-scale context-aware fusion to fuse features extracted from
multiple input images, denoted by LSM-Ctx-Fusion. As shown in Table 10, LSM-
Ctx-Fusion outperforms LSM in all numbers of views.

Table 10: Comparison of multi-view 3D object reconstruction on ShapeNet at $32^{3}$ resolution. We report the mean IoU for all categories. Note that both LSM and LSM-Ctx-Fusion take camera parameters as an additional input. The ShapeNet dataset is provided by LSM (Kar et al., 2017). Methods | 1 view | 2 views | 4 views | 8 views
---|---|---|---|---
LSM | 0.615 | 0.721 | 0.782 | 0.816
Pix2Vox++/F | 0.614 | 0.647 | 0.653 | 0.662
Pix2Vox++/A | 0.636 | 0.668 | 0.685 | 0.693
LSM-Ctx-Fusion | 0.639 | 0.739 | 0.806 | 0.838
Table 11: Comparison of multi-view 3D object reconstruction on ShapeNet-Cars at $128^{3}$ resolution. We report the mean IoU for all categories. The marker † and ‡ denote the multi-scale context-aware fusion is replaced with the average pooling fusion and context-aware fusion, respectively. Methods | 1 view | 2 views | 4 views | 8 views
---|---|---|---|---
Pix2Vox++/F † | 0.803 | 0.804 | 0.805 | 0.806
Pix2Vox++/F ‡ | 0.803 | 0.784 | 0.778 | 0.768
Pix2Vox++/F | 0.803 | 0.813 | 0.815 | 0.819
Pix2Vox++/A † | 0.826 | 0.828 | 0.829 | 0.829
Pix2Vox++/A ‡ | 0.826 | 0.813 | 0.808 | 0.801
Pix2Vox++/A | 0.826 | 0.837 | 0.841 | 0.843

Figure 13: Example of multi-view 3D object reconstruction on ShapeNet-Cars at
$128^{3}$ resolution. “Ctx Fusion” and “MCtx Fusion” denote the context-aware
fusion and the multi-scale context-aware fusion, respectively.

###  6.4 Comparison with Other Fusion Methods

To quantitatively evaluate multi-scale context-aware fusion, we replace the
multi-scale context-aware fusion in Pix2Vox++/A with the average fusion,
context-aware fusion, 3D convolutional LSTM, and attentional aggregation,
respectively.

Average Pooling Fusion. In the average pooling fusion, the voxel at $(i,j,k)$
among different coarse volumes are averaged. Specifically, the value of the
fused voxel $v^{f}$ can be calculated as

$v^{f}_{(i,j,k)}=\frac{1}{n}\sum_{r=1}^{n}v_{(i,j,k)}^{r}$ (8)

As shown in Table 13, replacing the multi-scale context-aware fusion with the
average fusion in Pix2Vox++/F and Pix2Vox++/A causes degeneration in
reconstruction results.

Context-aware Fusion. We also compare the multi-scale context-aware fusion
with the context-aware fusion in the preliminary version (Xie et al., 2019),
denoted by Pix2Vox++/F‡ and Pix2Vox++/A‡, respectively. As shown in Table 13,
IoU has an approximately 1% increase in the multi-scale context-aware fusion
compared to the context-aware fusion. Accurately reconstructing a 3D volume at
high resolution is challenging due to increasing voxels reflecting object
details. Both the average fusion and context-aware fusion may not fully
exploit the features of multi-view images. In contrast, the proposed multi-
scale context-aware fusion preserves object details by concatenating feature
maps with different scales. As shown in Table 11, the multi-scale context-
aware fusion outperforms the average fusion and the context-aware fusion in
reconstructing high-resolution 3D volumes. As illustrated in Figure 13, the
multi-scale context-aware fusion recovers better details than the context-
aware fusion.

Table 12: Comparison of single-view 3D object reconstruction on the ShapeNetCore dataset. We report the mean IoU for all categories in both object-centered and viewer-centered coordinates. Note that “Unseen” denotes no instances from the categories are seen during training. Methods | Object-centered | Viewer-centered
---|---|---
Seen |  Unseen |  Seen |  Unseen
OGN |  0.593 |  0.154 |  0.404 |  0.267
Matryoshka |  0.634 |  0.187 |  0.427 |  0.299
Pix2Vox++/F |  0.632 |  0.216 |  0.449 |  0.324
Pix2Vox++/A |  0.688 |  0.241 |  0.485 |  0.386
Table 13: Comparison of multi-view 3D object reconstruction on ShapeNet at $32^{3}$ resolution. We report the mean IoU per category and the average IoU for all categories. The marker † and ‡ denote the multi-scale context-aware fusion is replaced with the average pooling fusion and context-aware fusion, respectively. Methods |  1 view |  2 views |  3 views |  4 views |  5 views |  8 views | 12 views | 16 views | 20 views
---|---|---|---|---|---|---|---|---|---
Pix2Vox++/F † |  0.645 |  0.655 |  0.664 |  0.668 |  0.670 |  0.672 | 0.673 | 0.674 | 0.675
Pix2Vox++/F ‡ |  0.645 |  0.663 |  0.673 |  0.676 |  0.680 |  0.683 | 0.686 | 0.687 | 0.688
Pix2Vox++/F |  0.645 |  0.669 |  0.678 |  0.682 |  0.685 |  0.690 | 0.692 | 0.693 | 0.694
Pix2Vox++/A † |  0.670 |  0.680 |  0.690 |  0.695 |  0.699 |  0.703 | 0.704 | 0.705 | 0.706
Pix2Vox++/A ‡ |  0.670 |  0.690 |  0.699 |  0.702 |  0.706 |  0.710 | 0.712 | 0.713 | 0.714
Pix2Vox++/A-R2N2 |  0.663 |  0.672 |  0.680 |  0.684 |  0.686 |  0.688 | 0.689 | 0.689 | 0.690
Pix2Vox++/A-AttSets |  0.638 |  0.675 |  0.689 |  0.696 |  0.701 |  0.707 | 0.710 | 0.713 | 0.713
Pix2Vox++/A |  0.670 |  0.695 |  0.704 |  0.708 |  0.711 |  0.715 | 0.717 | 0.718 | 0.719
Table 14: The numbers of parameters, memory footprint, and inference time on the ShapeNet dataset. Note that the memory is measured in backward computation of single-view reconstruction with a batch size of 1. The voxel reconstruction methods, including 3D-R2N2, AttSets, Pix2Vox++/F, and Pix2Vox++/A, output 3D volumes at $32^{3}$ resolution. Methods | AtlasNet | Pixel2Mesh | OccNet | IM-Net | 3D-R2N2 | AttSets | Pix2Vox++/F | Pix2Vox++/A
---|---|---|---|---|---|---|---|---
#Parameters (M) | 45.06 | 21.36 | 13.43 | 55.45 | 35.97 | 17.71 | 4.83 | 96.31
Memory (MB) | 1293 | 1289 | 955 | 3935 | 1407 | 3911 | 647 | 2411
Inference Time (ms) |
1 view | 38.47 | 60.78 | 1261 | 10886 | 78.86 | 26.32 | 9.93 | 10.64
2 views | N/A | N/A | N/A | N/A | 112.27 | 47.62 | 13.55 | 17.51
4 views | N/A | N/A | N/A | N/A | 116.68 | 52.63 | 23.72 | 29.88
8 views | N/A | N/A | N/A | N/A | 122.04 | 58.83 | 39.02 | 56.52

3D Convolutional LSTM. To further compare with RNN-based fusion, we remove the
multi-scale context-aware fusion from Pix2Vox++/A and add a 3D convolutional
LSTM (Choy et al., 2016) after the encoder. To fit the 3D convolutional LSTM
input, we add an additional fully connected layer with a dimension of $1024$
before it. The resulting method is named Pix2Vox++/A-R2N2. As shown in Table
13, both Pix2Vox++/A and Pix2Vox++/A† consistently outperform Pix2Vox++/A-R2N2
in all numbers of views.

Attentional Aggregation. To demonstrate the superior reconstruction ability
over the attentional aggregation (Yang et al., 2020), we remove the multi-
scale context-aware fusion from Pix2Vox++/A and add an attentional aggregation
module after the encoder, denoted by Pix2Vox++/A-AttSets. Experimental results
in Table 13 show that Pix2Vox++/A outperforms Pix2Vox++/A-AttSets in all
numbers of views.

###  6.5 Viewer-centered vs. Object-centered Coordinates

As mentioned in Section 6.3, Pix2Vox++ relies on the object-centered
coordinates to align multi-view features. However, object-centered coordinates
encourage the network to memorize observed meshes, which may lead to poor
generalization abilities (Shin et al., 2018).

To evaluate the generalization capability of the proposed methods, we compare
the performance of reconstructing 3D objects from “seen” and “unseen”
categories in both viewer-centered and object-centered coordinates. For
object-centered prediction, different views of the same object should produce
the same 3D shape. In viewer-centered coordinates, the reconstructed 3D object
should be oriented according to the input viewpoint, so different views of the
same object correspond to different 3D shapes.

In this experiment, we use Blender to render objects in 57 categories of
ShapeNetCore (Chang et al., 2015) from 24 random views for each object. When
reconstructing 3D objects from “unseen” categories, all pretrained models have
never “seen” either the objects in these categories or the labels of the
objects before. More specifically, all methods are pretrained on the 13 major
categories of ShapeNet and tested on the remaining 44 categories of
ShapeNetCore with the same input images. As shown in Table 12, 3D shapes
produced by object-centered models outperform those produced by viewer-
centered models for objects from “seen” categories, suggesting that
reconstructing viewer-centered 3D shapes of objects is more challenging. For
“unseen” categories, viewer-centered models perform better than object-
centered models, suggesting that viewer-centered reconstruction improves the
generalization ability to reconstruct 3D shapes from “unseen” categories.
Compared to OGN (Tatarchenko et al., 2017) and Matryoshka Networks (Richter
and Roth, 2018), Pix2Vox++/F and Pix2Vox++/A perform better at reconstructing
3D shapes in both object-centered and viewer-centered coordinates.

###  6.6 Space and Time Complexity

To test the space and time complexity of our methods, we compare them with
several state-of-the-art methods in terms of number of parameters, memory
usage, and inference time. Table 14 presents the comparison results in single-
view and multi-view reconstruction, where the voxel reconstruction methods are
at $32^{3}$ resolution. Table 15 provides a comparison of the single-view
reconstruction results at $128^{3}$ resolution.

Running times are obtained on the same PC with a single NVIDIA GTX 1080 Ti
GPU. For more precise timing, we exclude reading and writing time when
evaluating inference time. For multi-view reconstruction, both Pix2Vox++/F and
Pix2Vox++/A outperform 3D-R2N2 and AttSets in inference time and training
time. Both Pix2Vox++/F and Pix2Vox++/A are approximately seven times faster in
forward inference than 3D-R2N2 for single-view reconstruction at $32^{3}$
resolution. Although the proposed methods outperform OGN (Tatarchenko et al.,
2017) and Matryoshka Networks (Richter and Roth, 2018) in reconstructing high-
resolution 3D volumes, memory requirements scale dramatically with the
resolution of 3D volumes because our methods do not use efficient data
representations.

Table 15: The numbers of parameters, memory footprint, and inference time at $128^{3}$ resolution on the ShapeNet dataset. P2V/F and P2V/A denote Pix2Vox++/F and Pix2Vox++/A for $128^{3}$ resolution. “Inf. Time” stands for “Inference Time” for single-view reconstruction with a batch size of 1. Note that the memory is measured in backward computation of single-view reconstruction with a batch size of 1. Methods | OGN | Matryoshka | P2V/F | P2V/A
---|---|---|---|---
#Params (M) | 12.46 | 45.66 | 5.02 | 96.57
Memory (MB) | 861 | 1593 | 2227 | 3997
Inf. Time (ms) | 52.87 | 11.92 | 37.10 | 51.96

##  7 Conclusion

In this paper, we propose a unified framework for both single-view and multi-
view 3D reconstruction, named Pix2Vox++. Compared with existing methods that
directly fuse the features from multi-view images, the proposed framework
fuses the 3D volumes reconstructed from input images, which better preserves
multi-view spatial constraints. In addition, we construct the first large-
scale naturalistic dataset for multi-view 3D object reconstruction, named
Things3D, containing 1.68M images of 280K objects collected from over 39K
indoor scenes. Quantitative and qualitative evaluation for both single-view
and multi-view reconstruction on the ShapeNet, Pix3D, and Things3D benchmarks
shows that the proposed methods perform favorably against state-of-the-art
methods. The proposed methods are also computationally efficient, about seven
times faster than 3D-R2N2 in terms of inference time in single-view
reconstruction.

###### Acknowledgements.

This work was supported in part by the National Natural Science Foundation of
China under Pro-ject (Nos. 61772158, 61702136 and 61872112), in part by
National Key Research and Development Program of China (Nos. 2018YFC0806802
and 2018YFC0832105), and in part by Self-Planned Task (No. SKLRS202002D) from
the State Key Laboratory of Robotics and System (HIT). We would like to thank
anonymous reviewers for their valuable feedback during this research.

## References

  * Barron and Malik (2015) Barron JT, Malik J (2015) Shape, illumination, and reflectance from shading. TPAMI 37(8):1670–1687
  * Cadena et al. (2016) Cadena C, Carlone L, Carrillo H, Latif Y, Scaramuzza D, Neira J, Reid ID, Leonard JJ (2016) Past, present, and future of simultaneous localization and mapping: Toward the robust-perception age. IEEE Transactions on Robotics 32(6):1309–1332
  * Chang et al. (2015) Chang AX, Funkhouser TA, Guibas LJ, Hanrahan P, Huang Q, Li Z, Savarese S, Savva M, Song S, Su H, Xiao J, Yi L, Yu F (2015) ShapeNet: An information-rich 3D model repository. arXiv 1512.03012
  * Chen and Zhang (2019) Chen Z, Zhang H (2019) Learning implicit fields for generative shape modeling. In: CVPR
  * Choy et al. (2016) Choy CB, Xu D, Gwak J, Chen K, Savarese S (2016) 3D-R2N2: A unified approach for single and multi-view 3D object reconstruction. In: ECCV
  * Dibra et al. (2017) Dibra E, Jain H, Öztireli AC, Ziegler R, Gross MH (2017) Human shape from silhouettes using generative HKS descriptors and cross-modal neural networks. In: CVPR
  * Fan et al. (2017) Fan H, Su H, Guibas LJ (2017) A point set generation network for 3D object reconstruction from a single image. In: CVPR
  * Fuentes-Pacheco et al. (2015) Fuentes-Pacheco J, Ascencio JR, Rendón-Mancha JM (2015) Visual simultaneous localization and mapping: a survey. Artif Intell Rev 43(1):55–81
  * Goodfellow et al. (2014) Goodfellow IJ, Pouget-Abadie J, Mirza M, Xu B, Warde-Farley D, Ozair S, Courville AC, Bengio Y (2014) Generative adversarial nets. In: NIPS
  * Groueix et al. (2018) Groueix T, Fisher M, Kim VG, Russell BC, Aubry M (2018) A papier-mâché approach to learning 3D surface generation. In: CVPR
  * Han et al. (2019) Han X, Laga H, Bennamoun M (2019) Image-based 3D object reconstruction: State-of-the-art and trends in the deep learning era. TPAMI DOI 10.1109/TPAMI.2019.2954885
  * Harltey and Zisserman (2006) Harltey A, Zisserman A (2006) Multiple view geometry in computer vision (2. ed.). Cambridge University Press
  * He et al. (2016) He K, Zhang X, Ren S, Sun J (2016) Deep residual learning for image recognition. In: CVPR
  * Huang et al. (2017) Huang G, Liu Z, van der Maaten L, Weinberger KQ (2017) Densely connected convolutional networks. In: CVPR
  * Huang et al. (2018) Huang P, Matzen K, Kopf J, Ahuja N, Huang J (2018) Deepmvs: Learning multi-view stereopsis. In: CVPR
  * Hwang and Sung (2015) Hwang K, Sung W (2015) Single stream parallelization of generalized LSTM-like RNNs on a GPU. In: ICASSP
  * Kar et al. (2017) Kar A, Häne C, Malik J (2017) Learning a multi-view stereo machine. In: NIPS
  * Kato and Harada (2019) Kato H, Harada T (2019) Learning view priors for single-view 3D reconstruction. In: CVPR
  * Kingma and Ba (2015) Kingma DP, Ba J (2015) Adam: A method for stochastic optimization. In: ICLR
  * Kingma and Welling (2014) Kingma DP, Welling M (2014) Auto-encoding variational bayes. In: ICLR
  * Lin et al. (2018) Lin C, Kong C, Lucey S (2018) Learning efficient point cloud generation for dense 3D object reconstruction. In: AAAI
  * Lin et al. (2019) Lin C, Wang O, Russell BC, Shechtman E, Kim VG, Fisher M, Lucey S (2019) Photometric mesh optimization for video-aligned 3D object reconstruction. In: CVPR
  * Lorensen and Cline (1987) Lorensen WE, Cline HE (1987) Marching cubes: A high resolution 3D surface construction algorithm. In: SIGGRAPH
  * Mescheder et al. (2019) Mescheder LM, Oechsle M, Niemeyer M, Nowozin S, Geiger A (2019) Occupancy networks: Learning 3D reconstruction in function space. In: CVPR
  * Mo et al. (2019a) Mo K, Guerrero P, Yi L, Su H, Wonka P, Mitra NJ, Guibas LJ (2019a) StructureNet: hierarchical graph networks for 3D shape generation. ACM Transactions on Graphics 38(6):242:1–242:19
  * Mo et al. (2019b) Mo K, Zhu S, Chang AX, Yi L, Tripathi S, Guibas LJ, Su H (2019b) PartNet: A large-scale benchmark for fine-grained and hierarchical part-level 3D object understanding. In: CVPR
  * Özyeşil et al. (2017) Özyeşil O, Voroninski V, Basri R, Singer A (2017) A survey of structure from motion. Acta Numerica 26:305–364
  * Pascanu et al. (2013) Pascanu R, Mikolov T, Bengio Y (2013) On the difficulty of training recurrent neural networks. In: ICML
  * Paschalidou et al. (2018) Paschalidou D, Ulusoy AO, Schmitt C, Gool LV, Geiger A (2018) Raynet: Learning volumetric 3D reconstruction with ray potentials. In: CVPR
  * Paschalidou et al. (2020) Paschalidou D, Gool LV, Geiger A (2020) Learning unsupervised hierarchical part decomposition of 3D objects from a single RGB image. In: CVPR
  * Paszke et al. (2019) Paszke A, Gross S, Massa F, Lerer A, Bradbury J, Chanan G, Killeen T, Lin Z, Gimelshein N, Antiga L, Desmaison A, Kopf A, Yang E, DeVito Z, Raison M, Tejani A, Chilamkurthy S, Steiner B, Fang L, Bai J, Chintala Sa (2019) PyTorch: An imperative style, high-performance deep learning library. In: NeurIPS
  * Richter and Roth (2015) Richter SR, Roth S (2015) Discriminative shape from shading in uncalibrated illumination. In: CVPR
  * Richter and Roth (2018) Richter SR, Roth S (2018) Matryoshka networks: Predicting 3D geometry via nested shape layers. In: CVPR
  * Ronneberger et al. (2015) Ronneberger O, Fischer P, Brox T (2015) U-net: Convolutional networks for biomedical image segmentation. In: MICCAI
  * Shin et al. (2018) Shin D, Fowlkes CC, Hoiem D (2018) Pixels, voxels, and views: A study of shape representations for single view 3D object shape prediction. In: CVPR
  * Simonyan and Zisserman (2015) Simonyan K, Zisserman A (2015) Very deep convolutional networks for large-scale image recognition. In: ICLR
  * Song et al. (2017) Song S, Yu F, Zeng A, Chang AX, Savva M, Funkhouser TA (2017) Semantic scene completion from a single depth image. In: CVPR
  * Su et al. (2015) Su H, Qi CR, Li Y, Guibas LJ (2015) Render for CNN: viewpoint estimation in images using cnns trained with rendered 3D model views. In: ICCV
  * Sun et al. (2018) Sun X, Wu J, Zhang X, Zhang Z, Zhang C, Xue T, Tenenbaum JB, Freeman WT (2018) Pix3D: Dataset and methods for single-image 3D shape modeling. In: CVPR
  * Tatarchenko et al. (2017) Tatarchenko M, Dosovitskiy A, Brox T (2017) Octree generating networks: Efficient convolutional architectures for high-resolution 3D outputs. In: ICCV
  * Tatarchenko et al. (2019) Tatarchenko M, Richter SR, Ranftl R, Li Z, Koltun V, Brox T (2019) What do single-view 3D reconstruction networks learn? In: CVPR
  * Vinyals et al. (2016) Vinyals O, Bengio S, Kudlur M (2016) Order matters: Sequence to sequence for sets. In: ICLR
  * Wang et al. (2018) Wang N, Zhang Y, Li Z, Fu Y, Liu W, Jiang Y (2018) Pixel2Mesh: Generating 3D mesh models from single RGB images. In: ECCV
  * Wen et al. (2019) Wen C, Zhang Y, Li Z, Fu Y (2019) Pixel2Mesh++: Multi-view 3D mesh generation via deformation. In: ICCV
  * Witkin (1981) Witkin AP (1981) Recovering surface shape and orientation from texture. Artif Intell 17(1-3):17–45
  * Wu et al. (2016) Wu J, Zhang C, Xue T, Freeman B, Tenenbaum J (2016) Learning a probabilistic latent space of object shapes via 3D generative-adversarial modeling. In: NIPS
  * Wu et al. (2017) Wu J, Wang Y, Xue T, Sun X, Freeman B, Tenenbaum J (2017) MarrNet: 3D shape reconstruction via 2.5D sketches. In: NIPS
  * Wu et al. (2018) Wu J, Zhang C, Zhang X, Zhang Z, Freeman WT, Tenenbaum JB (2018) Learning shape priors for single-view 3D completion and reconstruction. In: ECCV
  * Wu et al. (2015) Wu Z, Song S, Khosla A, Yu F, Zhang L, Tang X, Xiao J (2015) 3D ShapeNets: A deep representation for volumetric shapes. In: CVPR
  * Xiao et al. (2010) Xiao J, Hays J, Ehinger KA, Oliva A, Torralba A (2010) SUN database: Large-scale scene recognition from abbey to zoo. In: CVPR
  * Xiao et al. (2012) Xiao J, Ehinger KA, Oliva A, Torralba A (2012) Recognizing scene viewpoint using panoramic place representation. In: CVPR
  * Xie et al. (2019) Xie H, Yao H, Sun X, Zhou S, Zhang S (2019) Pix2Vox: Context-aware 3D reconstruction from single and multi-view images. In: ICCV
  * Xu et al. (2019) Xu Q, Wang W, Ceylan D, Mech R, Neumann U (2019) DISN: deep implicit surface network for high-quality single-view 3D reconstruction. In: NeurIPS
  * Yang et al. (2019) Yang B, Rosa S, Markham A, Trigoni N, Wen H (2019) Dense 3D object reconstruction from a single depth view. TPAMI 41(12):2820–2834
  * Yang et al. (2020) Yang B, Wang S, Markham A, Trigoni N (2020) Attentional aggregation of deep feature sets for multi-view 3D reconstruction. IJCV 128(1):53–73
  * Zhang et al. (2019) Zhang Y, Liu Z, Liu T, Peng B, Li X (2019) RealPoint3D: An efficient generation network for 3D object reconstruction from a single image. IEEE Access 7:57,539–57,549
  * Zhu et al. (2018) Zhu C, Xu K, Chaudhuri S, Yi R, Zhang H (2018) SCORES: shape composition with recursive substructure priors. ACM Transactions on Graphics 37(6):211:1–211:14


# Rare and Exotic Higgs decays at ATLAS and CMS

Pallabi Das on behalf of the ATLAS and CMS Collaborations 111Copyright 2024
CERN for the benefit of the ATLAS and CMS Collaborations. Reproduction of this
article or parts of it is allowed as specified in the CC-BY-4.0 license
Department of Physics, Princeton University
Princeton, New Jersey 08544, USA

###### Abstract

After the Higgs boson discovery in 2012, the experiments at the LHC are
continuing to study this particle and look for physics beyond the standard
model. Some of the Higgs boson properties, such as the mass, has been measured
with sub-percent level accuracy. Yet the present integrated luminosity is
still a limiting factor for measuring the Higgs boson self-coupling or the
first generation Yukawa couplings. The current constraints on the Higgs boson
couplings would still allow for a sizeable branching fraction into undetected
final states, which motivates the direct searches for rare and exotic decay
modes. This presentation discusses several new results from these searches
utilizing advanced online selection methods or analysis techniques with the
entire Run 2 data.

##  1 Introduction

The ATLAS [1] and CMS [2] experiments discovered the Higgs boson in Run 1 of
the LHC [3, 4, 5]. While testing the validity of the Standard Model (SM), the
experiments also search for beyond SM (BSM) physics. However, with no evidence
of new particles, understanding the nature of the Higgs boson is now of
paramount importance. Several theories suggest that new physics particles
couple preferentially to the Higgs boson. Even as the experiments are limited
by integrated luminosity to probe several SM Higgs boson couplings, the
available data includes potentially unexplored phase space. Direct searches of
rare and exotic Higgs boson decays are designed to fully exploit the LHC
collision data to examine the possibility of BSM interactions.

##  2 Rare Higgs boson decays to a meson and a photon

The existence of new physics may alter some very rare decay widths of the
Higgs boson, e.g., processes that are suppressed in the SM, producing flavour
changing neutral currents (FCNCs) or lepton flavour violation (LFV). Exclusive
Higgs boson decays to a photon and a quarkonium state are also sensitive to
Yukawa couplings of the charm and lighter quarks, the latter being out of
reach with the present LHC dataset. Three new results are discussed in this
presentation which reconstruct meson-photon final states resulting from Higgs
boson decays. For all three searches, upper limits on the respective Higgs
boson rare decay branching fraction are obtained by a fit of the meson-photon
system invariant mass distribution.

The ATLAS experiment performed a search for $\mathrm{H}\to D^{*}\gamma$ which
has an extremely low expected branching fraction of $\sim 10^{-27}$ due to
FCNC [6]. Dedicated online selection (trigger) is used to select events
containing $D^{0}\to K^{-}\pi^{+}$ signature and offline requirements are
placed on reconstructing the displaced decay vertex of the $D^{0}$ meson. The
first ever 95% CL upper limit on $\mathcal{B}(\mathrm{H}\to D^{*}\gamma)$ is
obtained with observed values below $10^{-3}$. The analysis also constrains
$\mathcal{B}(\mathrm{Z}\to K^{s0}\gamma)$ and $\mathcal{B}(\mathrm{Z}\to
D^{0}\gamma)$ values. The invariant mass distribution for the data and
background model in the signal region of this analysis are shown in Fig. 1
(left).

The CMS search for $\mathrm{H}\to\mathrm{J}/\psi\gamma$ or
$\psi^{\prime}\gamma$ is sensitive to charm Yukawa coupling [7]. The 95% CL
allowed range of the coupling modifier ratio $\kappa_{c}/\kappa_{\gamma}$ is
within (-157, 199). The upper limits on Z boson branching fractions are also
obtained. The photon-meson invariant mass spectrum is shown in Fig. 1
(middle). Observed $\mathcal{B}(\mathrm{H}\to J/\psi\gamma)$ and
$\mathcal{B}(\mathrm{H}\to\psi^{\prime}\gamma)$ values are constrained below
$9.9\times 10^{-4}$ and $2.6\times 10^{-4}$, respectively, at 95% CL.

Very recently, the CMS experiment has probed the rare final states of
$\mathrm{H}\to\rho^{0}\gamma$ or $\phi\gamma$ or $K^{*0}\gamma$ [8].
Specifically, $\mathcal{B}(\mathrm{H}\to K^{*0}\gamma)$ is expected to be
$\sim 10^{-19}$ due to flavor violating couplings of the s and d quarks. While
full Run 2 luminosity is analyzed, partial data was selected with a dedicated
trigger requiring a single photon and meson event signature. This analysis
provides the most stringent upper limits to date on these final states with
observed values $\mathcal{B}(\mathrm{H}\to\rho^{0}\gamma)<3.74\times 10^{-4}$,
$\mathcal{B}(\mathrm{H}\to\phi\gamma)<2.97\times 10^{-4}$ and
$\mathcal{B}(\mathrm{H}\to\rho^{0}\gamma)<1.71\times 10^{-4}$ at 95% CL. The
Kaon-photon invariant mass spectrum in one of the analysis categories is shown
in Fig. 1 (right).

---
Figure 1: Invariant mass spectra in the signal regions searching for (left)
$\mathrm{H}\to D^{*}\gamma$ [6], (middle) $\mathrm{H}\to\mathrm{J}/\psi\gamma$
or $\psi^{\prime}\gamma$ [7], and (right) $\mathrm{H}\to K^{*0}\gamma$ [8].

##  3 Higgs boson decays to dark photons

An example of dark matter particles coupling preferentially to the Higgs
boson, described in the FRVZ model [9, 10], is examined by the ATLAS
experiment [11]. A search for Higgs boson decays to a pair of dark photons
($\gamma_{d}$) is performed considering $\gamma_{d}$ masses between 0.1–30
GeV. The lifetime of $\gamma_{d}$ depends on the kinematic mixing parameter,
$\epsilon$, with SM photon. A specially designed trigger targets long-lived
$\gamma_{d}$ signature and vector-boson-fusion Higgs production mode. The
displaced collimated shower of SM particles is reconstructed in the
calorimeter and the muon detectors. Figure 2 shows the 90% CL observed limits
on $\mathcal{B}(\mathrm{H}\to\gamma_{d}\gamma_{d})$ as a function of the
$\gamma_{d}$ proper decay length and $\epsilon$. Values above 10% are excluded
for a mass of 10 GeV and mean proper decay length between 173–1296 mm
corresponding to small values of $\epsilon$ ($<10^{-5}$).

---
Figure 2: 90% CL observed limits on
$\mathcal{B}(\mathrm{H}\to\gamma_{d}\gamma_{d})$ considering both prompt and
long-lived $\gamma_{d}$ [11].

##  4 Higgs boson decays to axion-like particles

Axion-like particles or ALPs are gauge-singlet pseudoscalars predicted in
extended SM theories to address the strong CP problem. ALP decays to diphoton
can be identified at the LHC experiments and provide complimentary
measurements to cosmological searches. The ATLAS experiment has recently
studied the on-shell Higgs boson decay modes $\mathrm{H}\to\mathrm{aa}$ and
$\mathrm{H}\to\mathrm{Za}$, with $\mathrm{a}\to\gamma\gamma$, using the
$4\gamma$ and diphoton invariant mass distributions, respectively. The photon
signature may be merged for low mass of the ALPs, reconstructed as a single
object, or fully resolved.

In the search for $\mathrm{H}\to\mathrm{aa}\to 4\gamma$, a mass range of
$0.1<m_{\mathrm{a}}<60$ GeV is considered for both prompt and long-lived
signatures [12]. This analysis provides the most stringent upper limits till
date on long-lived ALPs with observed branching fractions $<$ 2–6$\times
10^{-5}$ for $m_{\mathrm{a}}>10$ GeV and $<10^{-5}$–3$\times 10^{-2}$ for
$m_{\mathrm{a}}<10$ GeV at 95% CL. Upper limits on prompt signatures are
illustrated in Fig. 3 (left). Only promptly decaying ALPs are considered in
the search for $\mathrm{H}\to\mathrm{Za}\to\ell\ell\gamma\gamma$ [13].
$\mathcal{B}(\mathrm{H}\to\mathrm{Za})$ values above 0.08–2$\times 10^{-2}$
are excluded at 95% CL within a mass range $0.1<m_{\mathrm{a}}<33$ GeV. The
upper limits on the effective couplings of the ALP to photons,
$|C_{\gamma\gamma}|/\Lambda$, are shown in Fig. 3 (right).

---
Figure 3: 95% CL upper limits on (left)
$\mathcal{B}(\mathrm{H}\to\mathrm{aa}\to 4\gamma)$ [12] and (right) effective
ALP coupling to photons [13] as a function of the ALP mass.

##  5 Higgs boson decays to a pair of pseudoscalars

Several other beyond SM theories predict Higgs boson exotic decays to a pair
of pseudoscalars. They occur naturally in two-Higgs-doublet models extended
with a singlet (2HDM+S), which is a special case of next-to-minimal
supersymmetric model (NMSSM). Other possibilities include supersymmetric
(SUSY) models with hidden sectors (dark-SUSY), vector-portal models etc. Many
possible final states from $\mathrm{H}\to\mathrm{aa}$ decays are
experimentally probed by the CMS and ATLAS experiments. In this presentation
two recent CMS searches with the full Run 2 data set are reported.

In the first search with both pseudosclars decaying to a pair of b-quarks, the
vector-boson-associated production of the Higgs boson is considered which can
be selected using leptonic trigger [14]. The fully hadronic final state is
difficult to reconstruct, and the analysis searches for a signal in different
categories based on the number of jets and further b-tagging requirements. A
boosted decision tree (BDT) is used to discriminate the signal from background
and the score is used as the discriminating variable for signal extraction. An
example of the BDT score distribution is shown in Fig. 4 (left). Assuming
$\mathcal{B}(\mathrm{H}\to\mathrm{aa}\to 4\mathrm{b})$ to be 100%, the
analysis excludes $m_{\mathrm{a}}$ values between 21–60 GeV at 95% CL.

The second search probes the $\mathrm{H}\to\mathrm{aa}\to 4\mu$ final state
considering $0.21<m_{\mathrm{a}}<60$ GeV and proper decay length between 0–100
mm [15]. The results are interpreted in terms of vector-portal models, ALPs,
NMSSM and dark-SUSY scenarios. A long-lived muon trigger is used for selecting
events, but this trigger was only active in part of the Run 2 data set. Since
both muon systems can be well reconstructed, signal selection is based on the
agreement between the dimuon mass and the mass of the pseudoscalar. The 95% CL
observed upper limit on $\sigma(\mathrm{H}\to
2\mathrm{a+X})\times\mathcal{B}^{2}(\mathrm{a}\to 2\mu)$ is within the range
0.049–0.247 fb. The model independent upper limits are shown in Fig. 4
(right).

---
Figure 4: (Left) BDT score distribution in one of the signal regions of
$\mathcal{B}(\mathrm{H}\to\mathrm{aa}\to 4\mathrm{b})$ analysis [14] and
(right) 95% CL upper limits on $\sigma(\mathrm{H}\to
2\mathrm{a+X})\times\mathcal{B}^{2}(\mathrm{a}\to 2\mu)$ as a function of the
pseudoscalar mass [15].

##  6 Summary

The 125 GeV Higgs boson serves as a probe for dark matter candidates and other
exotic particles from extended Higgs sectors. On the other hand, exclusive
Higgs boson decays to a final state with quarkonia can probe anomalous Yukawa
couplings to charm and lighter quarks. While the searches for rare and exotic
Higgs boson decays are dominated by statistical uncertainties, they are
sensitive to the Higgs boson interaction with new physics. No excess over SM
is yet observed, but the LHC experiments are working on improving search
strategies and exploring all available phase space. Going forward, Run 3 of
LHC will increase BSM signal sensitivity by adding new phase space in terms of
data volume and specially designed triggers.

## References

## References

  * [1] ATLAS Collaboration, JINST 3, S08003 (2008)
  * [2] CMS Collaboration, JINST 3, S08004 (2008)
  * [3] ATLAS Collaboration, Phys. Lett. B 716, 1 (2012)
  * [4] CMS Collaboration, Phys. Lett. B 716, 30 (2012)
  * [5] CMS Collaboration,  JHEP 06, 081 (2013)
  * [6] ATLAS Collaboration, arXiv:2402.18731, submitted to Phys. Lett. B
  * [7] CMS Collaboration, CMS-PAS-SMP-22-012
  * [8] CMS Collaboration, CMS-PAS-HIG-23-005
  * [9] A. Falkowski, J. T. Ruderman, T. Volansky and J. Zupan, JHEP 05, 077 (2010)
  * [10] A. Falkowski, J. T. Ruderman, T. Volansky and J. Zupan, Phys. Rev. Lett. 105, 241801 (2010)
  * [11] ATLAS Collaboration, arXiv:2311.18298, submitted to Eur. Phys. J. C
  * [12] ATLAS Collaboration, arXiv:2312.03306, submitted to Eur. Phys. J. C
  * [13] ATLAS Collaboration, Phys. Lett. B 848, 138536 (2024)
  * [14] CMS Collaboration, arxiv:2403.10341, submitted to JHEP
  * [15] CMS Collaboration, CMS-PAS-HIG-21-004