Pixel2Mesh++: 3D Mesh Generation and Refinement from Multi-View Images

Chao Wen^⋆,
Yinda Zhang^⋆,
Chenjie Cao,
Zhuwen Li,
Xiangyang Xue,
Yanwei Fu^†

^⋆ indicates equal contributions, ^† refers to the corresponding author.

Wenchao,  Chenjie Cao, Xiangyang Xue, and Yanwei Fu are with Fudan University, China.  E-mail<EMAIL_ADDRESS>{xyxue<EMAIL_ADDRESS>Yinda Zhang is with Google. Email<EMAIL_ADDRESS>Zhuwen Li is with Nuro, Inc., Mountain View, California.
Email<EMAIL_ADDRESS>Yanwei Fu is with School of Data Science, Fudan University.  Email<EMAIL_ADDRESS>
We study the problem of shape generation in 3D mesh representation from a small number of color images with or without camera poses.
While many previous works learn to hallucinate the shape directly from priors, we adopt to further improve the shape quality by leveraging cross-view information with a graph convolution network.
Instead of building a direct mapping function from images to 3D shape, our model learns to predict series of deformations to improve a coarse shape iteratively.
Inspired by traditional multiple view geometry methods, our network samples nearby area around the initial mesh's vertex locations and reasons an optimal deformation using perceptual feature statistics built from multiple input images.
Extensive experiments show that our model produces accurate 3D shapes that are not only visually plausible from the input perspectives, but also well aligned to arbitrary viewpoints.
With the help of physically driven architecture, our model also exhibits generalization capability across different semantic categories, and the number of input images. Model analysis experiments show that our model is robust to the quality of the initial mesh and the error of camera pose, and can be combined with a differentiable renderer for test-time optimization.

3D shape generation, Multi-view, Graph convolutional neural network, Mesh reconstruction.

§ INTRODUCTION

The topic of shape generation has become increasingly popular over the past decade.
With the astonishing capability of deep learning, lots of works have been demonstrated to successfully generate the 3D shape from merely a single color image.
However, due to limited visual cues from only one viewpoint, single image based approaches usually produce rough geometry in the occluded area and do not perform well when generalized to domains other than training, cross semantic categories.

Multi-View Shape Generation. From multiple input images, we produce shapes aligning well to input (c and d) and arbitrary random (e) camera viewpoint.
Single view based approach, Pixel2Mesh (P2M) [1], usually generates shape looking good from the input viewpoint (c) but significantly worse from others.
Naive extension with multiple views (MVP2M, Sec. <ref>) does not effectively improve the quality.

Adding a few more multi-view images (3-5) of the object is an effective way to provide the shape generation system with more information about the 3D shape.
On one hand, multi-view images provide more visual appearance information, and thus grant the system with more chance to build the connection between 3D shape and image priors.
On the other hand, it is well-known that traditional multi-view geometry methods [2] accurately infer 3D shape from correspondences across views, which is analytically well defined and less vulnerable to the generalization problem.

Even though typical multiview methods suffer from many problems, such as large baselines and textureless regions, and are likely to degenerate with very limited input images (less than 5), the way they leverage cross-view information might be implicitly encoded and learned by a deep model.
While well-motivated, there are very few works in the literature exploiting in this direction. And a naïve multi-view extension that fuses output 3D shapes or intermediate image features of single image based model does not work well as shown in Fig. <ref>.

In this work, we propose a deep learning model to generate the object shape from multiple color images with or without known camera poses.
Especially, we focus on endowing the deep model with the capacity of improving shapes using cross-view information.
The key component in our method is a new network architecture, named Multi-View Deformation Network (MDN), which works in conjunction with the Graph Convolutional Network (GCN) architecture proposed in Pixel2Mesh [1] to generate accurate 3D geometry shape in the desirable mesh representation.
In Pixel2Mesh, a GCN is trained to deform an initial shape to the target using features from a single image, which often produces plausible shapes but lack of accuracy (Fig. <ref> P2M).
We inherit this characteristic of “generation via deformation” and further deform the mesh in MDN using features carefully pooled from multiple images.
Instead of learning to hallucinate via shape priors like in Pixel2Mesh, MDN reasons shapes according to correlations across different views through a physically driven architecture inspired by classic multi-view geometry methods.
In particular, MDN proposes hypothesis deformations for each vertex and move it to the optimal location that best explains features pooled from multiple views.
By imitating correspondences search rather than learning priors, MDN generalizes well in various aspects, such as cross semantic category, number of input views, and the mesh initialization.

Besides the above-mentioned advantages, MDN is in addition featured with several good properties.
First, it can be trained end-to-end. Note that it is non-trivial since MDN searches deformation from hypotheses, which requires a non-differentiable argmax/min.
Inspired by [3], we apply a differentiable 3D soft argmax, which takes a weighted sum of the sampled hypotheses as the vertex deformation. Consequently, this component becomes differentiable and the whole system can be trained end-to-end.
Second, it works with varying number of input views in a single forward pass. This requires the feature dimension to be invariant with the number of inputs, which is typically broken when aggregating features from multiple images (when using concatenation).

To address this issue, we concatenate the statistics (mean, max, and standard deviation) of the pooled feature, which achieves invariance against the order of the input images as well.
We find this statistics feature encoding explicitly provides the network cross-view information, and encourages it to automatically utilize image evidence when more are available.
In addition, it can cope with the varying initial mesh topology. Thanks to the strategy of separating the generation of 3D shapes based on image priors and the shape refinement, MDN can refine the initial mesh created from various other sources, Pixel2Mesh [1], or via implicit representation method such as the Deep Implicit Surface Network (DISN) [4].

We found that MDN can still further improve the accuracy of the 3D shape even when the initial mesh has stronger expressive ability.
Moreover, we found that our method is robust to noisy camera pose, and can still work predicted camera pose if it is not known beforehand.

In this case, we estimate camera parameters which bring an object from the world coordinates to the image coordinates through a camera pose estimation network. This relaxation on the input made MDN more convenient to use in real scenario where camera pose may not be immediately accessible.

Last but not least, the nature of “generation via deformation” allows an iterative refinement.
In particular, the model output can be taken as the input, and the quality of the 3D shape is gradually improved throughout iterations.
With these desiring features, our model achieves the state-of-the-art performance on ShapeNet for shape generation from multiple images under standard evaluation metrics.

To summarize, we propose a GCN framework that produces 3D shape in mesh representation from a small number of observations of the object in different viewpoints.
The core component is a physically driven architecture that searches optimal deformation to improve a coarse mesh using perceptual feature statistics built from multiple images, which produces accurate 3D shape and generalizes well across different semantic categories, numbers of input images, and the quality of coarse meshes. Our method can produce 3D shapes even when the initial topology is varied and with or without knowing the ground truth camera position.

An early version of this work has been published in [5]. Compared with [5], we extend our method to work without known camera pose, with various initialization methods, more experiments and model analysis, as well as other technical improvements such as differentiable renderer.
We show that our method can further deform  the initial coarse meshes from those objects of arbitrary topology.

§ RELATED WORK

§.§ 3D Shape Representations

Since 3D CNN is readily applicable to 3D volumes, the volume representation has been well-exploited for 3D shape analysis and generation [6, 7]. With the debut of PointNet [8], the point cloud representation has been adopted in many works [9, 10]. Most recently, the mesh representation [11, 1] has become competitive due to its compactness and nice surface properties. Some other representations have been proposed, such as geometry images [12], depth images [13, 14], classification boundaries [15, 16, 17, 18, 19], signed distance function [20], etc., and most of them require post-processing to get the final 3D shape. Consequently, the shape accuracy may vary and the inference take extra time.

§.§ Single view shape generation

Classic single view shape reasoning can be traced back to shape from shading [21, 22], texture [23], and de-focus [24], which only reason the visible parts of objects.
With deep learning, many works leverage the data prior to hallucinate the invisible parts, and directly produce shape in 3D volume [6, 25, 26, 27, 28, 29, 30], point cloud [9], mesh models [11, 31, 32, 33, 34], or as an assembling of shape primitive [35, 36, 37, 38, 39, 40].

Alternatively, 3D shape can be also generated by deforming an initialization, which is more related to our work.

Tulsiani [41] and Kanazawa [42] learn a category-specific 3D deformable model and reasons the shape deformations in different images.

Wang [1] learn to deform an initial ellipsoid to the desired shape in a coarse to fine fashion.

Combining deformation and assembly, Huang [43] and Su [44] retrieve shape components from a large dataset and deform the assembled shape to fit the observed image.
Kuryenkov [45] learns free-form deformations to refine shape.
Even though with impressive success, most single view deep models adopt an encoder-decoder framework, and it is arguable if they perform shape generation or shape retrieval [46].
Groueix  [31] also learn to generate 3D shapes by deforming explicit surfaces. Their method dubbed Atlasnet, uses the global latent vector and MLP parameters to represent the patches deformation.
In contrast, our work models the deformation by passing message in GCN and gathers local perceptual feature for each vertex of GCN.
It is feasible for Atlasnet to model the shape with holes by using multiple patches, while it is non-trivial to handle patch overlap and get the continuous surface meshes.
Our architecture, however, allows our network to generate and refine plausible smooth shapes when proper coarse shape initialization is used.

§.§ Multi-view shape generation

Recovering 3D geometry from multiple views has been well studied.
Traditional multi-view stereo (MVS) [2] relies on correspondences built via photo-consistency and thus it is vulnerable to large baselines, occlusions, and texture-less regions.
Most recently, deep learning based MVS models have drawn attention, and most of these approaches [47, 48, 49, 50] rely on a cost volume built from depth hypotheses or plane sweeps.
Donne  [51] refine a set of input depth maps in the image domain.
However, these approaches usually generate depth maps, and it is non-trivial to fuse a full 3D shape from them.
On the other hand, direct multi-view shape generation uses fewer input views with large baselines, which is more challenging and has been less addressed.
Choy [6] propose a unified framework for single and multi-view object generation reading images sequentially. Kar [52] learn a multi-view stereo machine via recurrent feature fusion. Gwak [53] learns shapes from multi-view silhouettes by ray-tracing pooling and further constrains the ill-posed problem using GAN.
Our approach belongs to this category but is fundamentally different from the existing methods. Rather than sequentially feeding in images, our method learns a GCN to deform the mesh using features pooled from all input images at once.

§.§ Differentiable Rendering

Differentiable rendering retains derivatives during the forward process of image synthesis [54]. Therefore, many differentiable rendering enhanced neural networks can directly optimize their neural fields [55] through image pixel colors, i.e., neural rendering.

According to the representation, this type of work can be categorized into neural radiance fields [56], implicit differentiable renderer [57, 58] and differentiable rasterizer [59]. Neural radiance fields (NeRF) and its successors [56, 60, 61, 62, 63, 64, 65] focus on representing the scene as 5D neural radiance fields. The success of these approaches mostly requires per-scene optimization, which is more suitable for novel-view synthesis tasks than 3D shape generation.
Among these, Yu  [62] propose pixelNeRF that shows the potential of cross-category and sparse input view rendering. However it is difficult to extract a high-quality surface. Implicit differentiable renderer approaches [57, 58] can generate 3D shapes via 2D supervision and volumetric rendering. Liu  [59] propose a soft version rasterizer and learn to deform template mesh using multiple silhouette and color images.
Despite that these approaches could produce compelling results having dense input views, they still have performance degradation when very few views are available.
Our method can not only handle this this situation, but also can absorb differentiable renderer as a tool to optimize 3D shapes during inference.

§.§ Camera Pose Estimation

As a sub-problem in 3D shape recovery, many works are trying to tackle the problem of camera pose estimation.
Kendall  [66] proposed PoseNet, which represents the pose as the camera position and camera rotation, and obtains the value of the pose by normalizing the output quaternion.

Since two quaternions with opposite signs represent the same rotation under boundary condition, Wu  [67] use Euler angles to represent rotation.

Peng  [68] regard camera pose estimation as a key point regression problem, and use voting strategy to densely estimate the key point offset.
Recently, Zhou  [69] show that the discontinuity of the rotation representation is the main reason for the difficulty in pose estimation, and proposed a continuous 6D rotation representation.
Xu  [4] continue to use the 6D rotation representation, and convert the pose numerical regression to the matching problem of the point cloud, which enhances the generalization performance.
We followed the 6D rotation representation, and further represented the translation as the scalar offset of the object along the optical axis in the camera coordinate.

§ METHOD

§.§ System Overview

Our model receives multiple color images of an object captured from different viewpoints (with or without known camera poses) and produces a 3D mesh model in the world coordinate. This procedure is conceptually visualized in Fig. <ref>.
Our approach consists of two parts: 3D shape prediction and camera pose estimation .
The 3D shape prediction framework adopts the strategy of coarse-to-fine (Fig. <ref>), in which a plausible but rough shape is generated first, and details are added later.
Realizing that existing 3D shape generators usually produce reasonable shape even from a single image, we simply use Pixel2Mesh [1] or DISN [4] trained from multiple views to produce the coarse shape, which is taken as input to our Multi-View Deformation Network (MDN) for further improvement.
In MDN, each vertex first samples a set of deformation hypotheses from its surrounding area (Fig. <ref> (a)).
Each hypothesis then pools cross-view perceptual feature from early layers of a perceptual network , where the feature resolution is high and contains more low-level geometry information (Fig. <ref> (b)).
These features are gathered by projection with the help of the ground truth or predicted camera parameters.
The cross-view perceptual features are further leveraged by the network to reason the best deformation to move the mesh vertex.
It is worth noting that our MDN can be applied iteratively for multiple times to gradually improve shapes.
In cases where we cannot obtain the camera parameters in advance, we estimate them through a neural network.
The camera pose estimation network predicts the compact 7D camera pose representation with respect to the canonical pose (Fig. <ref>).

Overview of our framework. Our method takes a small number of color images with ground truth or predicted camera poses as inputs and generates 3D meshes. The pretrained VGG-16 is used to extract perceptual features from input images. Details about the coarse shape generation and the MDN are shown in Fig. <ref>.

Shape Generation Pipeline. Our 3D shape generation system consists of a 2D CNN extracting image features and a GCN deforming an ellipsoid to target shape. A coarse shape is generated from multi-view version Pixel2Mesh [1] or DISN [4] and refined iteratively in Multi-View Deformation Network. To leverage cross-view information, our network pools perceptual features from multiple input images for hypothesis locations in the area around each vertex and predicts the optimal deformation.

§.§ Multi-View Deformation Network

In this section, we introduce Multi-View Deformation Network, which is the core of our system to enable the network exploiting cross-view information for shape generation.
We assume that we know the extrinsics of the camera. (We will come back to address this issue in the next section.)
It first generates deformation hypotheses for each vertex and learns to reason an optimum using feature pooled from inputs.
Our model is essentially a GCN, and can be jointly trained with other GCN based models like Pixel2Mesh.
We refer reader to [70, 71] for details about GCN, and Pixel2Mesh [1] for graph residual block which will be used in our model.

§.§.§ Deformation Hypothesis Sampling

The first step is to propose deformation hypotheses for each vertex.
This is equivalent as sampling a set of target locations in 3D space where the vertex can be possibly moved to.
To uniformly explore the nearby area, we sample from a level-1 icosahedron centered on the vertex with a scale of 0.02, which results in 42 hypothesis positions (Fig. <ref> (a), left).
We then build a local graph with edges on the icosahedron surface and additional edges between the hypotheses to the vertex in the center, which forms a graph with $43$ nodes and $120+42=162$ edges.
Such the local graph is built for all mesh vertices, and then fed into a GCN to predict vertex movements (Fig. <ref> (a), right).

Deformation Hypothesis and Perceptual Feature Pooling. (a) Deformation Hypothesis Sampling. We sample 42 deformation hypotheses from a level-1 icosahedron and build a GCN among hypotheses and the vertex. (b) Cross-View Perceptual Feature Pooling. The 3D vertex coordinates are projected to multiple 2D image planes using camera intrinsics and extrinsics. Perceptual features are pooled using bilinear interpolation, and feature statistics are kept on each hypothesis.

§.§.§ Cross-View Perceptual Feature Pooling

The second step is to assign each node (in the local GCN) features from the multiple input color images.
Inspired by Pixel2Mesh, we use the prevalent VGG-16 architecture to extract perceptual features.
Since the camera poses are either as known inputs or can be predicted by our camera pose estimation network, each vertex and hypothesis can find their projections in all input color image planes using camera intrinsics and extrinsics and pool features from four neighboring feature grids using bilinear interpolation (Fig. <ref> (b)).
Different from Pixel2Mesh where high level features from later layers of the VGG (i.e. `conv3_3', `conv4_3’, and `conv5_3’) are pooled to better learn shape priors, MDN pools features from early layers (i.e. `conv1_2', `conv2_2’, and `conv3_3’), which are in high spatial resolution and considered maintaining more detailed information.

To combine multiple features, concatenation has been widely used as a loss-less way;  however, it ends up with total dimension changing with respect to the number of input images.
Statistics feature has been proposed for multi-view shape recognition [72] to tackle this problem.
Inspired by this work, we concatenate some statistics ($mean$, $max$, and $std$) of the features pooled from all views for each vertex, which makes our network naturally adaptive to variable input views and behave invariant to different input orders.
This also encourages the network to learn from cross-view feature correlations rather than each individual feature vector.
In addition to image features, we also concatenate the 3-dimensional vertex coordinate into the feature vector.
In total, we compute for each vertex and hypothesis a 339 dimension feature vector.

Deformation Reasoning. The goal is to reason a good deformation from the hypotheses and pooled features. We first estimate a weight (green circle) for each hypothesis using a GCN. The weights are normalized  by a softmax layer (yellow circle), and the output deformation is the weighted sum of all the deformation hypotheses.

§.§.§ Deformation Reasoning

The next step is to reason an optimal deformation for each vertex from the hypotheses using pooled cross-view perceptual features.
Note that picking the best hypothesis of all needs an argmax operation, which requires stochastic optimization and usually is not optimal.
Instead, we design a differentiable network component to produce desirable deformation through soft-argmax of the 3D deformation hypotheses, which is illustrated in Fig. <ref>.
Specifically, we first feed the cross-view perceptual feature $P$ into a scoring network, consisting of 6 graph residual convolution layers [1] activated by ReLU, to predict a scalar weight $c_i$ for each hypothesis.
All the weights are then fed into a softmax layer and normalized to scores $s_i$, with $\sum_{i=1}^{43} s_i=1$.
The vertex location is then updated as the weighted sum of all the hypotheses,
i.e. $v=\sum_{i=1}^{43} s_i*h_i$, where $h_i$ is the location of each deformation hypothesis including the vertex itself.
This deformation reasoning unit runs on all local GCN built upon every vertex with shared weights, as we expect all the vertices leveraging multi-view feature in a similar fashion.

§.§.§ Differentiable Renderer

Lastly, as an option, our method can be combined with differentiable renderer to achieve instance optimization during inference.
We use common mesh rasterizer renderer pipeline [59, 73] as an optional component.
When the silhouettes of images can be obtained from the input object, the 3D shape of the object can be further optimized by matching the silhouette of the predicted shapes with the multi-view input silhouette images through differentiable renderer.

Network Architecture of Camera Pose Estimation Network. Our camera pose estimation network predicts transformation from canonical view to camera view.
The rotation is represented as a continuous 6D vector [69] and the translation is defined as 1D distance between object center and camera.

The predicted camera poses and the ground truth value are applied to the canonical view point cloud respectively to calculate the loss.

§.§ Camera Pose Estimation Network

In this section, we introduce camera pose estimation network.
In case the poses of the camera cannot be obtained, we use another convolution network to estimate them.

Given an input image without camera pose, our goal is to estimate the corresponding camera extrinsics. The models in the ShapeNet Core dataset [74] are aligned for each category. Therefore, the world space is aligned with respect to this canonical coordinate system. As discussed in [75], regressing camera pose directly often fails, Zhou  [69] proposed a continuous rotation representation. We inherit this representation and regard the camera translation as the offset of the object along the direction of the camera optical axis, so that the camera extrinsics can be represented as 6D rotation and 1D translation. As illustrated in Fig. <ref>, we use a simple convolution network to regress the rotation and translation. The network first extract image feature through convolution layers, then using two branches to predict 6D rotation and 1D translation
separately; and each branch  consists of multiple fully connected layers.

§.§ Loss

3D Shape Generation.
We train our model fully supervised using ground truth 3D CAD models.
Our loss function includes all terms from Pixel2Mesh, but extends the Chamfer distance loss to a re-sampled version.
Chamfer distance measures “distance” between two point clouds, which can be problematic when points are not uniformly distributed on the surface.
We propose to randomly re-sample the predicted mesh when calculating Chamfer loss using the re-parameterization trick proposed in [76, 77].
Specifically, given a triangle defined by 3 vertices $\left\{ v _ { 1 } , v _ { 2 } , v _ { 3 } \right\} \in \mathbb { R } ^ { 3 }$, a uniform sampling can be achieved by:

s = \left( 1 - \sqrt { r _ { 1 } } \right) v _ { 1 } + \left( 1 - r _ { 2 } \right) \sqrt { r _ { 1 } } v _ { 2 } + \sqrt { r _ { 1 } } r _ { 2 } v _ { 1 },

where $s$ is a point inside the triangle, and $r _ { 1 } , r _ { 2 } \sim U [ 0,1 ]$.
Knowing this, when calculating the loss, we uniformly sample our generated mesh for 4000 points, with the number of points per triangle proportional to its area.
We find this is empirically sufficient to produce a uniform sampling on our output mesh with 2466 vertices, and calculating Chamfer loss on the re-sampled point cloud, containing 6466 in total, helps to remove artifacts in the results.
The overall learning objective is as follows,

$$\mathcal{L} = \lambda_1 \mathcal{L}_{chamfer} + \lambda_2 \mathcal{L}_{normal} + \lambda_3  \mathcal{L}_{edge} + \lambda_4 \mathcal{L}_{lap}, $$

where $\mathcal{L}_{chamfer}$, $\mathcal{L}_{normal}$, $\mathcal{L}_{edge}$ and $\mathcal{L}_{lap}$ denotes  re-sampled chamfer loss, normal loss, edge length regularization and Laplacian regularization. Loss weights $\lambda_1=1$, $\lambda_2=1.6\times10^{-4}$, $\lambda_3=0.1$ and $\lambda_4=0.5$ are the hyperparameters which help to balance the losses.
Moreover, we use an extra $L_2$ silhouette loss by comparing the projected silhouette between predicted shape and ground truth when using the differentiable renderer during inference.

Camera Pose Estimation. Following Xu  [4], we calculate $L_2$ distance between the ground truth point cloud in the camera view and the point cloud transformed from the canonical view by the predicted camera parameters. Specifically, the loss function is as follows:

\begin{gather*}
\mathcal{L}_{cam} = \left\|\mathcal{T}^{\prime}(p) - \mathcal{T}(p) \right\|_{2}^{2} ,\\
\mathcal{T}^{\prime}(p) = \mathbf{R}^{\prime} p + \mathbf{t}^{\prime} ,\\
\mathcal{T}(p) = \mathbf{R} p + \mathbf{t} ,
\end{gather*}

where $p$ denotes point cloud in canonical view, $\mathcal{T}^{\prime}(p)$ and $\mathcal{T}(p)$ denotes the predict and ground truth transformation respectively, $\mathbf{R}$ denotes rotation and $\mathbf{t}$ denotes translation, and the symbol with superscript “ $^{\prime}$ ” indicates the estimated value. Both $\mathcal{T}$ and $\mathcal{T}^{\prime}$ transform point cloud from canonical coordinates  to camera coordinates.

§.§ Variants of Pixel2mesh++

As in Fig. <ref>, we introduce two variants of Pixel2mesh++ in term of different coarse-shape generation subnetwork.

Naïve Methods for multi-view shape generation. By directly extending previous works (Choy [6] and Kar [52] )  to multiple input images, there are several näive ways of shape generation from multi-view images, as compared in Tab. <ref>. Essentially, these methods produce the coarse meshes, which can be utilized as the initial coarse mesh module in our Pixel2mesh++. Specifically,

(1) P2M-M: we directly run single-view Pixel2Mesh on each of the input image and fuse multiple results [78, 79].

(2) MVP2M: we extend Pixel2Mesh to access multiple images in a single network forward pass by having it pools multi-view features from all the inputs. This can be achieved by replacing the perceptual feature pooling layers with our multi-view version as introduced in Sec. <ref>. In particular, perceptual features are pooled from layer `conv3_3', `conv4_3', and `conv5_3' from the VGG-16 network, and feature statistics (Sec. <ref>) are calculated and concatenated, which ends up with a 1280 dimension feature vector.
In practice, we also tried to pool geometry related features from early convolution layers (i.e. `conv1_2', `conv2_2', and `conv3_3'), but found it doesn't work as well as the case with semantic feature pooled from later layers.
Using a template with a fixed topology will limit the expression ability of the shape.

(3) MVDISN:  in order to show the generalization of our method in arbitrary topology, we build another implicit representation baseline based on DISN. We equip DISN with our perceptual feature pooling layers and extend it to a multi-view version (Tab. <ref>, MVDISN). The output signed distance function of MVDISN is converted to mesh using marching cubes algorithm.

Our Variants. We thus define two variants of our Pixel2mesh++ via using different initial coarse meshes, which are produced by MVP2M and MVDISN.
Particularly, we have the variants as follows,

(1) Ours-P: As in Fig. <ref>, we take MVP2M as the initial coarse shape, and combine with our MDN. This variant needs ellipsoid mesh as input for coarse shape generation. We train the model in an end-to-end manner.

(2) Ours-D: Since using a template with a fixed topology will limit the expression ability of the shape. We combine the implicit methods MVDISN with stronger representation ability as initialization of the coarse shape. The variant of Ours-D is also learned end-to-end in our experiments. Note that MVDISN can also produce the coarse meshes of objects of arbitrary topology, and our method can still efficiently improve the quality of object meshes.

§ EXPERIMENTS

In this section, we perform extensive evaluation of our model for multi-view shape generation.
We compare to state-of-the-art methods, as well as conduct controlled experiments with respect to () various aspects, cross category generalization, quantity of inputs, etc. All the experiments assume camera poses are known without specification.

§.§ Experimental setup

§.§.§ Dataset

We adopt the dataset provided by Choy [6] as it is widely used by many existing 3D shape generation works. The dataset is created using a subset of ShapeNet[74] containing 50k 3D CAD models from 13 categories. Each model is rendered from 24 randomly chosen camera viewpoints, and the camera intrinsic and extrinsic parameters are given. For fair comparison, we use the same training/testing split as in Choy [6] for all our experiments. The ground truth models are transformed to camera coordinate based on the camera parameters from the Choy [6], and scaled by a factor of 0.57 to aligned with rendering images following Wang [1].

§.§.§ Evaluation Metric

3D Shape Generation.
As for 3D shape, we use standard evaluation metrics for 3D shape generation. Following Fan [9], we calculate Chamfer Distance(CD) between 2048 points uniformly sampled from the ground truth and our prediction to measure the surface accuracy. We also use F-score following Wang [1] to measure the completeness and precision of generated shapes, where $\tau$=1.0$\times10^{-4}$. For CD, the smaller is better. For F-score, the larger is better. We also evaluate volumetric IoU. We transform the ground truth and predictions back to canonical view to calculate IoU.
We obtain unbiased estimates of the volumetric IoU by randomly sampling 100k points following Mescheder  [15].

Camera Pose Estimation.
We use four types of standard metrics to evaluate our camera pose estimation method: 2D reprojection error $d_{2D}$, mean distance $d_{3D}$ [4], 2D reprojection accuracy Acc$_{2D}$ and average 3D distance of model points (ADD) accuracy metric ADD$_{3D}$ [68].
The reprojection error $d_{2D}$ is compared with the image size $137\times 137$ pixels. Different from the threshold selected for general 6 DOF estimation tasks [68], we use more strict criteria. Specifically, we set the threshold to 2 for the Acc$_{2D}$ metric and the threshold to $5\%$ for the ADD$_{3D}$ metric.

§.§.§ Implementation Details

3D Shape Generation.
For initialization, we use Pixel2Mesh [1] to generate a coarse shape with 2466 vertices, or use DISN [4] to generate a coarse shape with varying topology.
To improve the quality of initial mesh, we equip these methods with our cross-view perceptual feature pooling layer, which allows it to extract features from multiple views.
We denote these multi-view version shape generation methods as MVP2M and MVDISN.
The network is implemented in Pytorch and optimized using Adam with weight decay as $5\times 10^{-6}$ and mini-batch size as 1.
The model use MVP2M as coarse shape generation method is trained for 40 epochs with learning rate $1\times 10^{-5}$ over 72 hours, and the model use MVDISN is trained for 100 epochs with learning rate $3 \times 10^{-4}$ over 96 hours. The MDN network is trained for another 20 epoch over 24 hours with the learning rate as $1\times 10^{-6}$.
The overall network needs to learn the statistical information between the multi-view features, and the MDN module needs plausible initialization from the coarse shape generation stage to effectively refine the 3D shape. Therefore, the convergence speed of our method is slower than single view deep models and requires longer training time.
During training, we randomly pick three images for a mesh as input.
All models are trained on a NVIDIA Titan Xp GPU.
During test, it takes 0.32s to produce a mesh.

Camera Pose Estimation.
The camera parameter estimation network is trained for 70 epochs, over 48 hours. The learning rate is set to $1\times 10^{-4}$. All 24 views of a mesh and the ground truth mesh points near surface are used as training data. During inference, we fed a single image to the network to obtain the corresponding camera pose.

Comparison to Multi-view Shape Generation Methods. We show F-score on each semantic category. Our model significantly outperforms previous methods, i.e. 3DR2N2 [6] and LSM [52], and competitive baselines derived from Pixel2Mesh [1]. The notation $\dagger$ indicates the methods which does not require camera extrinsics.
to X[1.5,l]X[1.5,c]X[c]X[1.5,c]X[1.5,c]X[1.5,c]X[1.5c]X[1.5c]X[1.5,c]X[1.2,c]X[1.5,c]X[1.5,c]X[1.5,c]X[1.5c]X[1.5c]
7cF-score($\tau$) $\uparrow$
7cF-score($2\tau$) $\uparrow$

(lr)2-8 (lr)9-15
3DR2N2$^{\dagger}$     LSM     P2M-M     MVP2M     Ours-P     MVDISN     Ours-D     3DR2N2$^{\dagger}$     LSM     P2M-M     MVP2M     Ours-P     MVDISN     Ours-D

Couch     45.47     43.02     53.70     52.53     57.53     68.61     71.66
59.97     55.49     72.04     72.17     75.74     80.85     82.84

Cabinet     54.08     50.80     63.55     58.04     61.46     51.12     54.11
64.42     60.72     79.93     77.28     79.46     67.71     69.72

Bench     44.56     49.33     61.14     60.20     67.70     74.22     80.14
62.47     65.92     75.66     75.89     81.49     85.79     89.23

Chair     37.62     48.55     55.89     54.04     61.78     70.73     76.77
54.26     64.95     72.36     72.04     77.99     83.00     86.72

Monitor     36.33     43.65     54.50     53.36     58.95     68.42     73.41
48.65     56.33     70.51     70.34     75.03     82.97     85.89

Firearm     55.72     56.14     74.85     78.52     85.31     91.37     94.27
76.79     73.89     84.82     88.67     92.64     96.43     97.75

Speaker     41.48     45.21     51.61     49.21     53.75     56.95     61.58
52.29     56.65     68.53     67.66     71.20     70.30     74.02

Lamp      32.25     45.58     51.00     50.46     63.40     65.89     74.39
49.38     64.76     64.72     65.63     75.62     77.04     82.66

Cellphone     58.09     60.11     70.88     71.88     75.79     85.14     87.65
69.66     71.39     84.09     85.24     87.72     92.58     94.31

Plane     47.81     55.60     72.36     76.67     84.17     81.37     85.72
70.49     76.39     82.74     87.18     91.76     89.92     92.53

Table     48.78     48.61     67.89     66.35     70.52     67.59     71.78
62.67     62.22     81.04     80.12     83.15     80.00     82.70

Car       59.86     51.91     67.29     65.18     67.64     67.57     69.96
78.31     68.20     84.39     82.38     83.62     79.48     80.82

Watercraft     40.72     47.96     57.72     59.21     65.98     71.10     76.62
63.59     66.95     72.96     75.63     80.41     82.80     86.26

Mean     46.37     49.73     61.72     61.20     67.23     70.78     75.24
62.53     64.91     76.45     76.94     81.22     82.22     85.04

Comparison to Multi-view Shape Generation Methods. We show the Chamfer Distance on each semantic category. Our method achieves the best performance overall. The notation $\dagger$ indicates the methods which does not require camera extrinsics.
to 0.7X[1.5,l]X[1.5,c]X[c]X[1.5,c]X[1.5,c]X[1,c]X[1.5,c]X[2,c]
7cChamfer Distance(CD) $\downarrow$

3DR2N2$^{\dagger}$     LSM     P2M-M     MVP2M     Ours-P     MVDISN     Ours-D

Couch       0.806     0.730     0.496     0.440     0.383     0.396     0.362

Cabinet     0.613     0.634     0.359     0.381     0.354     0.765     0.737

Bench       1.362     0.572     0.594     0.462     0.368     0.343     0.285

Chair       1.534     0.495     0.561     0.494     0.402     0.409     0.346

Monitor     1.465     0.592     0.654     0.585     0.496     0.332     0.285

Firearm     0.432     0.385     0.428     0.250     0.184     0.165     0.137

Speaker     1.443     0.767     0.697     0.666     0.604     0.746     0.663

Lamp        6.780     1.768     1.184     0.891     0.681     0.820     0.722

Cellphone     1.161     0.362     0.360     0.373     0.318     0.179     0.152

Plane       0.854     0.496     0.457     0.251     0.183     0.271     0.226

Table       1.243     0.994     0.441     0.463     0.406     0.587     0.532

Car         0.358     0.326     0.264     0.260     0.243     0.362     0.344

Watercraft     0.869     0.509     0.627     0.413     0.337     0.332     0.279

Mean     1.455     0.664     0.548     0.456     0.381     0.439     0.390

Comparison to Baselines and Our Variants. We show the volumetric IoU on each semantic category.
to X[1.5,l]X[1.5,c]X[1,c]X[1.5,c]X[1.5,c]
4cIoU $\uparrow$

MVP2M     Ours-P     MVDISN     Ours-D

Couch       0.546     0.569     0.671     0.688

Cabinet     0.536     0.550     0.398     0.397

Bench       0.211     0.224     0.418     0.399

Chair       0.343     0.362     0.515     0.509

Monitor     0.383     0.408     0.519     0.502

Firearm     0.371     0.426     0.621     0.539

Speaker     0.480     0.496     0.525     0.528

Lamp        0.218     0.240     0.321     0.310

Cellphone     0.613     0.644     0.679     0.628

Plane       0.424     0.484     0.556     0.533

Table       0.306     0.314     0.453     0.442

Car         0.513     0.523     0.602     0.596

Watercraft     0.391     0.429     0.565     0.556

Mean     0.411     0.436     0.527     0.508

§.§ Comparison to Multi-view Shape Generation

Tab. <ref> shows quantitative comparison in F-score.
As can be seen, our baselines already outperform other methods, which show the advantage of mesh representation in finding surface details.
Moreover, directly equipping Pixel2Mesh with multi-view features does not improve too much (even slightly worse than the average of multiple runs of single-view Pixel2Mesh), which shows dedicate architecture is required to efficiently learn from multi-view features.
In contrast, our Multi-View Deformation Network significantly further improves the results from the MVP2M baseline (i.e. our coarse shape initialization).
It is worth noting that the MVDISN method has more accurate initial shapes and is a stronger baseline. Therefore, using this baseline directly is better than using MVP2M. However, by combining it with our MDN method, the performance will also be significantly improved.

Tab. <ref> shows quantitative comparison in Chamfer Distance. Our method is not only better than other multi-view shape generation methods, but also has a consistent performance improvement for the baseline algorithm we constructed.

We also compare quantitative results for volumetric IoU. The calculation of IoU may be affected by self-intersecting faces and non-watertight surfaces, since it is complicated to determine whether a point is inside self-intersecting faces.
As shown in Tab. <ref>, when the initial coarse shape generated from the ellipsoid, Ours-P has an improvement compared to MVP2M. However, when the topology changes, Ours-D has a slightly lower IoU than MVDISN due to the complicated procedures to determine the inside points.

More qualitative results are shown in Fig. <ref>.
We show results from different methods aligned with one input view (left) and a random view (right).
As can be seen, Choy [6] (3D-R2N2) and Kar [52] (LSM) produce 3D volume, which lose thin structures and surface details.
Pixel2Mesh (P2M) produces mesh models but shows obvious artifacts when visualized in viewpoint other than the input.
In comparison, our results contain better surface details and more accurate geometry learned from multiple views.

§.§ Generalization Capability

Our MDN is inspired by multi-view geometry methods, where 3D location is reasoned via cross-view information.
In this section, we investigate the generalization capability of MDN in different aspects to improve the initialization mesh. For all the experiments in this section, we fix the coarse stage and train/test MDN under different settings.

2*Category     3cF-score($\tau$) $\uparrow$2-4 3-4 4-4
Corase     Except     Alllamp     50.82     61.78(+10.9)     62.55(+11.7)cabinet     56.85     65.73(+8.88)     65.72(+8.99)cellphone     66.07     73.17(+7.10)     74.36(+8.29)chair     54.19     60.68(+6.49)     62.05(+7.86)monitor     53.41     59.47(+6.06)     60.01(+6.60)speaker     48.90     54.65(+5.75)     54.88(+5.98)table     65.95     71.39(+5.44)     71.89(+5.94)bench     60.37     65.67(+5.30)     66.24(+5.87)couch     53.17     56.93(+3.76)     57.56(+4.39)plane     75.16     76.28(+1.12)     76.79(+1.63)firearm     79.66     80.33(+0.67)     80.73(+1.07)watercraft     61.85     62.06(+0.21)     62.99(+1.14)car     67.26     67.40(+0.14)     68.44(+1.18)

(a) Absolute and relative improvements
of training except one category     (b) Relative improvements of training
with one category

Quantitative Results of Cross-Category Generalization. (a) Improved F-score($\tau$) and absolute F-score($\tau$) of MDN trained on 12 out of 13 categories and tested on the one left. (b) MDN trained on 1 category and tested on the others. Each block represents the improved F-score($\tau$) of MDN trained on horizontal category and tested on vertical category.

Qualitative Results of Cross-Category Generalization.
`Coarse' means results from MVP2M. `Except' indicates that results are generated from the model that has not been trained with the same category. `All' indicates that the model has been trained with data from all categories.

Robustness to Initialization. Our model is robust to added noise, shift, and input mesh from other sources.

§.§.§ Semantic Category

We first verify how our network generalizes across semantic categories.
We fix the initial MVP2M and train MDN with 12 out 13 categories and test on the one left out. The improved and the absolute F-scores($\tau$) upon the initialization from Pixel2Mesh are both shown in Fig. <ref> (a).
As can be seen, the performance is only slightly lower when the testing category is removed from the training set compared to the model trained using all categories.
To make it more challenging, we further train MDN on only one category and test on other categories.
Surprisingly, MDN still generalizes well among most categories as shown in Fig. <ref> (b).
Besides, we provide some qualitative results of the cross-category generalization in Fig. <ref>. These results show that our MDN enjoys good generalization for the unseen categories.

Strong generalizing categories (chair, table, lamp) tend to have relatively complex geometry, thus the model has better chance to learn from cross-view information.
On the other hand, categories with super simple geometry (speaker, cellphone) do not help to improve other categories, even not for themselves.
On the whole, MDN shows good generalization capability across semantic categories.

§.§.§ Number of Views

We then test how MDN performs the number of input views.
In Tab. <ref>, we see that MDN consistently performs better when more input views are available, even though the number of view is fixed as 3 for efficiency during the training.
This indicates that features from multiple views are well encoded in the statistics, and MDN is able to exploit additional information when seeing more images.
For reference, we train five MDNs with the input view number fixed at 2 to 5 respectively.
As shown in Tab. <ref> “Resp.”, the 3-view MDN performs very close to models trained with more views (4 and 5), which shows the model learns efficiently from fewer number of views during the training.
The 3-view MDN also outperform models trained with less views (2), which indicates additional information provided during the training can be effectively activated during the test even when observation is limited.

Overall, MDN generalizes well to different number of inputs.

Performance Input View Numbers. `Resp.' means that MDN is trained with the same views as the testing. MDN performs consistently better when more view is given, even trained using only 3 views.

#train     #test     2     3     4     5

3*3     F-score($\tau$) $\uparrow$     64.48     66.44     67.66     68.29

F-score($2 \tau$) $\uparrow$     78.74     80.33     81.36     81.97

CD $\downarrow$     0.515     0.484     0.468     0.459

3*Resp.     F-score($\tau$) $\uparrow$    64.11     66.44     68.54     68.82

F-score($2 \tau$) $\uparrow$     78.34     80.33     81.56     81.99

CD $\downarrow$     0.527     0.484     0.467     0.452

§.§ Model Robustness Analysis

In this section, we analyze the robustness of the model in many aspects.
We analyze the model's ability to deal with initialization noise, cameras position error, and instance level optimization during test with differentiable renderer when silhouette mask are available.
Through these analyses, we explore why multiple views help for 3D reconstruction.

§.§.§ Initialization

Firstly, we test if the model overfits to the input initialization, i.e. the MVP2M.
To this end, we add translation and random noise to the rough shape from MVP2M.
We also take as input the mesh converted from 3DR2N2 using marching cubes algorithm [79].
As shown in Fig. <ref>, MDN successfully removes the noise, aligns the input with ground truth, and adds significant geometry details.
This experiments show that MDN is tolerant to input variance.

§.§.§ Camera Pose

We compare the performance of our method on camera pose estimation with Insafutdinov  [75] and Xu  [4]. Given a point cloud in world space, we transform this point cloud using ground truth camera parameters and predicted parameters to camera view respectively. We first calculate the the mean distance in 3D space which is called $d_{3D}$. We then project the point cloud onto the input image plane using known intrinsics to compute the 2D reprojection error $d_{2D}$. We also evaluate
Acc$_{2D}$ and ADD$_{3D}$ metrics.
With the help of continuous rotation and compact 1D translation representation, our method outperforms Insafutdinov  [75] and Xu  [4] in terms of $d_{2D}$, Acc$_{2D}$ and ADD$_{3D}$. And our method achieves comparable results on $d_{3D}$.

Quantitative Camera Pose Estimation Comparison. We compare our method with Zhou  [69] and DISN [4].

Methods      $d_{2D}$ $\downarrow$     $d_{3D}$ $\downarrow$     Acc$_{2D}$ $\uparrow$     ADD$_{3D}$ $\uparrow$

Zhou  [69]     4.86     0.073     -     -

DISN [4]     2.95     0.047     73.34     87.31

Ours     2.39     0.049     74.32     93.32

Moreover, we test whether MDN is able to produce plausible 3D shape when camera pose has error. Since gathering image features require reasonable camera pose, using random camera pose is not feasible. Instead, we use camera pose estimation network to obtain reasonable camera extrinsics.
As shown in Tab. <ref> (w/ pred cam), even using predicted camera parameters, our MDN can also refine the initial mesh shape and improve performance compared to the baseline algorithms.
Benefiting from more visual cues provided by multi-view images, the above experimental results show that MDN has a certain degree of robustness to camera pose errors.

Robustness of Different Methods. pc, gtc, and s are short for predicted camera pose, ground-truth camera pose, and sihouette.  CD:Chamfer Distance.
to X[1.5,l]X[2,c]X[1.5,c]X[1.5,c]X[1.5,c]X[2,c]X[1.5,c]
2*Methods     2*w/pc     2*w/gtc     2*w/s     2cF-score $\uparrow$     2*CD$\downarrow$5-6
$\tau$     2$\tau$     4*MVP2M     $\surd$               59.93     75.85     0.482         $\surd$          61.20     76.96     0.456    $\surd$          $\surd$     60.67     76.41     0.472         $\surd$     $\surd$     62.80     78.01     0.4374*Ours-P     $\surd$               63.22     78.52     0.428         $\surd$          67.32     81.22     0.381    $\surd$          $\surd$     63.56     78.89     0.420         $\surd$     $\surd$     68.45     82.19     0.3604*MVDISN     $\surd$               67.86     80.09     0.553         $\surd$          70.78     82.22     0.439    $\surd$          $\surd$     68.39     80.38     0.545         $\surd$     $\surd$     71.78     82.71     0.4294*Ours-D     $\surd$               70.07     81.79     0.518         $\surd$          75.24     85.04     0.390    $\surd$          $\surd$     70.26     81.95     0.511         $\surd$     $\surd$     75.74     85.33     0.383

§.§.§ Differentiable Renderer

We test the effect of our method combined with the differentiable renderer. The differentiable renderer can perform test-time optimization for 3D shape during inference.

In this section, we use the differentiable renderer during inference. The 3D shapes can be further optimized by matching the silhouette of the predicted shapes with the multi-view input images.
We use 2D silhouette loss for the optimization, so that the generated 3D model has a better alignment at the views corresponding to the input images.
Optimization at the image level helps to restore the unique geometric details of the 3D shape. Moreover, the optimization process can make the shape and the input image align better.
Since the previous method based on the differentiable renderer either requires a large number of multi-view silhouettes as supervision [59], or processes a single color image [80], it is not suitable for our task.
On the basis of Liu  [59], we only use the part of its differentiable renderer. We use three input images with corresponding silhouettes and network output 3D mesh models for optimization.

As shown in Tab. <ref> (w/ silhouette), through the optimization leveraging the differentiable renderer, the performance of our method can be further improved. The qualitative results are shown in the Fig. <ref>. It can be seen that using the differentiable renderer can further optimize the predicted 3D shape, especially geometric details such as larger corners.

Qualitative Examples with or without differentiable renderer.

Qualitative and Quantitative Results Compared with DVR [57] and IDR [58]. Note that our method only takes 3-view inputs. DVR and IDR are evaluated with the same or more input views.

§.§.§ Comparison of Neural Rendering Approaches

We also compare our method with representative neural rendering methods, i.e., Differentiable Volumetric Rendering (DVR) [57] and Implicit Differentiable Rendering (IDR) [58] in Fig. <ref> qualitatively. Quantitative results of them are also shown in Fig. <ref>.

Both DVR and IDR are trained with the instance-level, i.e., a separate model needs to be trained for each individual instance. Such online optimization is powerful in reconstruction, but suffers from poor efficiency to test a new coming instance. Because the computation with expensive inference time optimization is required.
We train these methods according to the official multi-view reconstruction setting.

The initial shape of our method is MVDISN. Since our method only takes 3 images for each inference, we also compare the results of DVR and IDR with 3-view input images. Although rendering-based methods can achieve good results in Fig. <ref>(b)(c) with 24-view inputs, it costs several hours to train an instance-level neural rendering model for each sample, which is inefficient. Besides, if we reduce the input views to three, the performances of DVR and IDR are degraded obviously as shown in Fig <ref>(d)(e). And our method can get comparable results with 3-view input images more efficiently.

§.§.§ Cross-dataset Generalization

We evaluated the cross-data generalization ability of our method. We train models on ShapeNet and test them on the ABC dataset [81] directly without any finetuning for the generalization analysis. Qualitative and quantitative results are shown in Fig. <ref>.

Thanks to feature learning from multiple views, our coarse shape generation from MVDISN has certain cross-data generalization capabilities. And our MDN can further refine the 3D shape in both qualitative and quantitative results, especially when the coarse shape has geometry noise. Besides, the single-view based OccNet [15] may fails to achieve reliable results in ABC dataset.
Nevertheless, generalization across datasets is a very challenging task. When the geometric shape of the new dataset is complex or the coarse shape generation stage failed, the performance of our approach will degrade (the third example in Fig. <ref>).

Qualitative and Quantitative Results on ABC dataset [81]. We show the cross-dataset generalization ability. From left to right, input images, results from OccNet, coarse results from MVDISN, results from Our-D, and the ground truth. Our MDN can still improve results from the cross-dataset.

§.§ Ablation Study

In this section, we verify the qualitative and quantitative improvements from statistic feature pooling, re-sampled Chamfer distance, and iterative refinement.

§.§.§ Statistical Feature

We first check the importance of using feature statistics. We train MDN with the ordinary concatenation. This maintains all the features loss-less to potentially produce better geometry, but does not support variable number of inputs any more.
Surprisingly, our model with feature statistics (Tab. <ref>, “Full Model”) still outperforms the one with concatenation (Tab. <ref>, “-Feat Stat”).
This is probably because that our feature statistics is invariant to the input order, such that the network learns more efficiently during the training. It also explicitly encodes cross-view feature correlations, which can be directly leveraged by the network.

Qualitative Ablation Study. We show meshes from the MDN with statistics feature or re-sampling loss disabled.

Quantitative Ablation Study. We show the metrics of the MDN with statistics feature or re-sampling loss disabled.

Metrics     F-score($\tau$) $\uparrow$     F-score($2\tau$) $\uparrow$     CD $\downarrow$

-Feat Stat     65.26     79.13     0.511

-Re-sample Loss     66.26     80.04     0.496

Full Model     67.32     81.22     0.381

Qualitative Evaluation. From top to bottom, we show in each row: two camera views, results of 3DR2N2, LSM, multi-view Pixel2Mesh, ours, and the ground truth. Our predicts maintain good details and align well with different camera views.
Please see supplementary materials for more results.

§.§.§ Re-sampled Chamfer Distance

We then investigate the impact of the re-sampled Chamfer loss.
We train our model using the traditional Chamfer loss only on mesh vertices as defined in Pixel2Mesh, and all metrics drop consistently (Tab. <ref>, “-Re-sample Loss”).
Intuitively, our re-sampling loss is especially helpful for places with sparse vertices and irregular faces, such as the elongated lamp neck as shown in Fig. <ref>, 3rd column.
It also prevents big mistakes from happening on a single vertex, the spike on bench, where our loss penalizes a lot of sampled points on wrong faces caused by the vertex but the standard Chamfer loss only penalizes one point.

Performance with Different Iterations.  The performance keeps improving with more iterations and saturate at three.

§.§.§ Number of Iteration

Figure <ref> shows that the performance of our model keeps improving with more iterations, and is roughly saturated at three.
Therefore we choose to run three iterations during the inference even though marginal improvements can be further obtained from more iterations.

Qualitative Examples with Different Iterations.

Qualitative results of real-world images from the online products dataset and Internet.

Comparison of different coarse initial shape method.

§ CONCLUSION

We propose a graph convolutional framework to produce 3D mesh model from multiple images.
Our model learns to exploit cross-view information and generates vertex deformation iteratively to improve the mesh produced from the direct prediction methods, Pixel2Mesh and its multi-view extension.
Inspired by multi-view geometry methods, our model searches in the nearby area around each vertex for an optimal place to relocate it.

Compared to previous works, our model achieves the state-of-the-art performance, produces shapes containing accurate surface details rather than merely visually plausible from input views, and shows good generalization capability in many aspects.
For future work, combining with efficient shape retrieval for initialization, integrating with multi-view stereo models for explicit photometric consistency, and extending to scene scales are some of the practical directions to explore.
On a high level, how to integrating the similar idea in emerging new representations, such as part based model with shape basis and learned function [20] are interesting for further study.

§ ACKNOWLEDGMENTS

§ ACKNOWLEDGMENT

This work was supported in part by the National Natural Science Foundation of China Grant (62076067),  SMSTM Project (2021SHZDZX0103),  Shanghai Municipal Science and Technology Major Project (No.2018SHZDZX01), ZJ Lab, and Shanghai Center for Brain Science and Brain-Inspired Technology. Dr. Yanwei Fu is also with and Shanghai Key Lab of Intelligent Information Processing, and Fudan ISTBI—ZJNU Algorithm Centre for Brain-inspired Intelligence, Zhejiang Normal University, Jinhua, China.

[1]
N. Wang, Y. Zhang, Z. Li, Y. Fu, W. Liu, and Y.-G. Jiang, “Pixel2mesh:
Generating 3d mesh models from single rgb images,” in Proceedings of
the European Conference on Computer Vision (ECCV), 2018.

[2]
A. Harltey and A. Zisserman, Multiple view geometry in computer vision
(2. ed.).1em plus 0.5em minus 0.4emCambridge University
Press, 2006.

[3]
A. Kendall, H. Martirosyan, S. Dasgupta, and P. Henry, “End-to-end learning of
geometry and context for deep stereo regression,” in Proceedings of
the IEEE International Conference on Computer Vision (ICCV), 2017, pp.

[4]
Q. Xu, W. Wang, D. Ceylan, R. Mech, and U. Neumann, “Disn: Deep implicit
surface network for high-quality single-view 3d reconstruction,” in
Advances in Neural Information Processing Systems (NeurIPS), 2019, pp.

[5]
C. Wen, Y. Zhang, Z. Li, and Y. Fu, “Pixel2mesh++: Multi-view 3d mesh
generation via deformation,” in Proceedings of the IEEE International
Conference on Computer Vision (ICCV), 2019, pp. 1042–1051.

[6]
C. B. Choy, D. Xu, J. Gwak, K. Chen, and S. Savarese, “3d-r2n2: A unified
approach for single and multi-view 3d object reconstruction,” in
Proceedings of the European Conference on Computer Vision (ECCV),

[7]
P.-S. Wang, Y. Liu, Y.-X. Guo, C.-Y. Sun, and X. Tong, “O-cnn: Octree-based
convolutional neural networks for 3d shape analysis,” ACM Transactions
on Graphics (TOG), vol. 36, no. 4, p. 72, 2017.

[8]
C. R. Qi, H. Su, K. Mo, and L. J. Guibas, “Pointnet: Deep learning on point
sets for 3d classification and segmentation,” in Proceedings of the
IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017.

[9]
H. Fan, H. Su, and L. J. Guibas, “A point set generation network for 3d object
reconstruction from a single image,” in Proceedings of the IEEE
Conference on Computer Vision and Pattern Recognition (CVPR), 2017.

[10]
C. R. Qi, W. Liu, C. Wu, H. Su, and L. J. Guibas, “Frustum pointnets for 3d
object detection from RGB-D data,” in Proceedings of the IEEE
Conference on Computer Vision and Pattern Recognition (CVPR), 2018.

[11]
H. Kato, Y. Ushiku, and T. Harada, “Neural 3d mesh renderer,” in
Proceedings of the IEEE Conference on Computer Vision and Pattern
Recognition (CVPR), 2018.

[12]
A. Sinha, A. Unmesh, Q. Huang, and K. Ramani, “Surfnet: Generating 3d shape
surfaces using deep residual networks,” in Proceedings of the IEEE
Conference on Computer Vision and Pattern Recognition (CVPR), 2017.

[13]
M. Tatarchenko, A. Dosovitskiy, and T. Brox, “Multi-view 3d models from single
images with a convolutional network,” in Proceedings of the European
Conference on Computer Vision (ECCV), 2016.

[14]
S. R. Richter and S. Roth, “Matryoshka networks: Predicting 3d geometry via
nested shape layers,” in Proceedings of the IEEE Conference on
Computer Vision and Pattern Recognition (CVPR), 2018.

[15]
L. Mescheder, M. Oechsle, M. Niemeyer, S. Nowozin, and A. Geiger, “Occupancy
networks: Learning 3d reconstruction in function space,” in
Proceedings of the IEEE Conference on Computer Vision and Pattern
Recognition (CVPR), 2019, pp. 4460–4470.

[16]
Z. Chen and H. Zhang, “Learning implicit fields for generative shape
modeling,” in Proceedings of the IEEE Conference on Computer Vision
and Pattern Recognition (CVPR), 2019, pp. 5939–5948.

[17]
Y. Liao, S. Donne, and A. Geiger, “Deep marching cubes: Learning explicit
surface representations,” in Proceedings of the IEEE Conference on
Computer Vision and Pattern Recognition (CVPR), 2018, pp. 2916–1925.

[18]
M. Atzmon, N. Haim, L. Yariv, O. Israelov, H. Maron, and Y. Lipman,
“Controlling neural level sets,” in Advances in Neural Information
Processing Systems (NeurIPS), 2019, pp. 2032–2041.

[19]
M. Atzmon and Y. Lipman, “Sal: Sign agnostic learning of shapes from raw
data,” in Proceedings of the IEEE Conference on Computer Vision and
Pattern Recognition (CVPR), 2020, pp. 2565–2574.

[20]
J. J. Park, P. Florence, J. Straub, R. Newcombe, and S. Lovegrove, “Deepsdf:
Learning continuous signed distance functions for shape representation,” in
Proceedings of the IEEE Conference on Computer Vision and Pattern
Recognition (CVPR), June 2019.

[21]
J. Durou, M. Falcone, and M. Sagona, “Numerical methods for
shape-from-shading: A new survey with benchmarks,” Computer Vision
and Image Understanding, vol. 109, no. 1, pp. 22–43, 2008.

[22]
R. Zhang, P. Tsai, J. E. Cryer, and M. Shah, “Shape from shading: A
survey,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 21, no. 8,
pp. 690–706, 1999.

[23]
C. Marinos and A. Blake, “Shape from texture: the homogeneity hypothesis,” in
Proceedings of the IEEE International Conference on Computer Vision
(ICCV), 1990, pp. 350–353.

[24]
P. Favaro and S. Soatto, “A geometric approach to shape from defocus,”
IEEE Trans. Pattern Anal. Mach. Intell., vol. 27, no. 3, pp.
406–417, 2005.

[25]
R. Girdhar, D. F. Fouhey, M. Rodriguez, and A. Gupta, “Learning a predictable
and generative vector representation for objects,” in Proceedings of
the European Conference on Computer Vision (ECCV), 2016.

[26]
J. Wu, C. Zhang, T. Xue, B. Freeman, and J. Tenenbaum, “Learning a
probabilistic latent space of object shapes via 3d generative-adversarial
modeling,” in Advances in Neural Information Processing Systems
(Neurips), 2016.

[27]
C. Hane, S. Tulsiani, and J. Malik, “Hierarchical surface prediction for 3d
object reconstruction,” in Proceedings of the International Conference
on 3D Vision (3DV), 2017.

[28]
G. Riegler, A. O. Ulusoy, and A. Geiger, “Octnet: Learning deep 3d
representations at high resolutions,” in Proceedings of the IEEE
Conference on Computer Vision and Pattern Recognition (CVPR), 2017.

[29]
M. Tatarchenko, A. Dosovitskiy, and T. Brox, “Octree generating networks:
Efficient convolutional architectures for high-resolution 3d outputs,” in
Proceedings of the IEEE International Conference on Computer Vision
(ICCV), 2017.

[30]
A. Johnston, R. Garg, G. Carneiro, and I. D. Reid, “Scaling cnns for high
resolution volumetric reconstruction from a single image,” in
Proceedings of the IEEE International Conference on Computer Vision
(ICCV), 2017.

[31]
T. Groueix, M. Fisher, V. G. Kim, B. Russell, and M. Aubry, “AtlasNet: A
Papier-Mâché Approach to Learning 3D Surface Generation,” in
Proceedings of the IEEE Conference on Computer Vision and Pattern
Recognition (CVPR), 2018, pp. 216–224.

[32]
S. Goel, A. Kanazawa, and J. Malik, “Shape and viewpoint without keypoints,”
in Proceedings of the European Conference on Computer Vision (ECCV),
2020, pp. 88–104.

[33]
Y. Ye, S. Tulsiani, and A. Gupta, “Shelf-supervised mesh prediction in the
wild,” in Proceedings of the IEEE Conference on Computer Vision and
Pattern Recognition (CVPR), 2021, pp. 8843–8852.

[34]
K. Gupta and M. Chandraker, “Neural mesh flow: 3d manifold mesh generation via
diffeomorphic flows,” in Advances in Neural Information Processing
Systems (NeurIPS), 2020, pp. 1747–1758.

[35]
S. Tulsiani, H. Su, L. J. Guibas, A. A. Efros, and J. Malik, “Learning shape
abstractions by assembling volumetric primitives,” in Proceedings of
the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017.

[36]
C. Niu, J. Li, and K. Xu, “Im2struct: Recovering 3d shape structure from a
single RGB image,” in Proceedings of the IEEE Conference on Computer
Vision and Pattern Recognition (CVPR), 2018.

[37]
D. Paschalidou, A. O. Ulusoy, and A. Geiger, “Superquadrics revisited:
Learning 3d shape parsing beyond cuboids,” in Proceedings of the IEEE
Conference on Computer Vision and Pattern Recognition (CVPR), 2019, pp.
10 344–10 353.

[38]
K. Genova, F. Cole, D. Vlasic, A. Sarna, W. T. Freeman, and T. Funkhouser,
“Learning shape templates with structured implicit functions,” in
Proceedings of the IEEE/CVF International Conference on Computer
Vision, 2019, pp. 7154–7164.

[39]
B. Deng, K. Genova, S. Yazdani, S. Bouaziz, G. Hinton, and A. Tagliasacchi,
“Cvxnet: Learnable convex decomposition,” in Proceedings of the IEEE
Conference on Computer Vision and Pattern Recognition (CVPR), 2020, pp.

[40]
D. Paschalidou, A. Katharopoulos, A. Geiger, and S. Fidler, “Neural parts:
Learning expressive 3d shape abstractions with invertible neural networks,”
in Proceedings of the IEEE Conference on Computer Vision and Pattern
Recognition (CVPR), 2021, pp. 3204–3215.

[41]
S. Tulsiani, A. Kar, J. Carreira, and J. Malik, “Learning category-specific
deformable 3d models for object reconstruction,” IEEE Trans. Pattern
Anal. Mach. Intell., vol. 39, no. 4, pp. 719–731, 2017.

[42]
A. Kanazawa, S. Tulsiani, A. A. Efros, and J. Malik, “Learning
category-specific mesh reconstruction from image collections,” in
Proceedings of the European Conference on Computer Vision (ECCV),

[43]
Q. Huang, H. Wang, and V. Koltun, “Single-view reconstruction via joint
analysis of image and shape collections,” ACM Trans. Graph.,
vol. 34, no. 4, pp. 87:1–87:10, 2015.

[44]
H. Su, Q. Huang, N. J. Mitra, Y. Li, and L. J. Guibas, “Estimating image depth
using shape collections,” ACM Trans. Graph., vol. 33, no. 4, pp.
37:1–37:11, 2014.

[45]
A. Kurenkov, J. Ji, A. Garg, V. Mehta, J. Gwak, C. Choy, and S. Savarese,
“Deformnet: Free-form deformation network for 3d shape reconstruction from a
single image,” in WACV, 2018, pp. 858–866.

[46]
M. Tatarchenko, S. Richter, R. Ranftl, Z. Li, V. Koltun, and T. Brox, “What do
single-view 3d reconstruction networks learn?” in Proceedings of the
IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2019.

[47]
Y. Yao, Z. Luo, S. Li, T. Fang, and L. Quan, “Mvsnet: Depth inference for
unstructured multi-view stereo,” in Proceedings of the European
Conference on Computer Vision (ECCV), 2018.

[48]
P.-H. Huang, K. Matzen, J. Kopf, N. Ahuja, and J.-B. Huang, “Deepmvs: Learning
multi-view stereopsis,” in Proceedings of the IEEE Conference on
Computer Vision and Pattern Recognition (CVPR), 2018.

[49]
S. Im, H.-G. Jeon, S. Lin, and I. S. Kweon, “Dpsnet: End-to-end deep plane
sweep stereo,” in International Conference on Learning Representations
(ICLR), 2018.

[50]
Y. Zhang, S. Khamis, C. Rhemann, J. Valentin, A. Kowdle, V. Tankovich,
M. Schoenberg, S. Izadi, T. Funkhouser, and S. Fanello, “Activestereonet:
End-to-end self-supervised learning for active stereo systems,” in
Proceedings of the European Conference on Computer Vision (ECCV),
2018, pp. 784–801.

[51]
S. Donne and A. Geiger, “Learning non-volumetric depth fusion using successive
reprojections,” in Proceedings of the IEEE Conference on Computer
Vision and Pattern Recognition (CVPR), 2019, pp. 7634–7643.

[52]
A. Kar, C. Häne, and J. Malik, “Learning a multi-view stereo machine,” in
Advances in Neural Information Processing Systems (Neurips), 2017, pp.

[53]
J. Gwak, C. B. Choy, M. Chandraker, A. Garg, and S. Savarese, “Weakly
supervised 3d reconstruction with adversarial constraint,” in
Proceedings of the International Conference on 3D Vision (3DV), 2017.

[54]
M. M. Loper and M. J. Black, “Opendr: An approximate differentiable
renderer,” in European Conference on Computer Vision.1em plus
0.5em minus 0.4emSpringer, 2014, pp. 154–169.

[55]
Y. Xie, T. Takikawa, S. Saito, O. Litany, S. Yan, N. Khan, F. Tombari,
J. Tompkin, V. Sitzmann, and S. Sridhar, “Neural fields in visual computing
and beyond,” arXiv preprint arXiv: Arxiv-2111.11426, 2021.

[56]
B. Mildenhall, P. P. Srinivasan, M. Tancik, J. T. Barron, R. Ramamoorthi, and
R. Ng, “Nerf: Representing scenes as neural radiance fields for view
synthesis,” in Proceedings of the European Conference on Computer
Vision (ECCV), 2020, pp. 405–421.

[57]
M. Niemeyer, L. Mescheder, M. Oechsle, and A. Geiger, “Differentiable
volumetric rendering: Learning implicit 3d representations without 3d
supervision,” in Proceedings of the IEEE Conference on Computer Vision
and Pattern Recognition (CVPR), 2020, pp. 3504–3515.

[58]
L. Yariv, Y. Kasten, D. Moran, M. Galun, M. Atzmon, B. Ronen, and Y. Lipman,
“Multiview neural surface reconstruction by disentangling geometry and
appearance,” vol. 33, pp. 2492–2502, 2020.

[59]
S. Liu, T. Li, W. Chen, and H. Li, “A general differentiable mesh renderer for
image-based 3d reasoning,” IEEE Transactions on Pattern Analysis and
Machine Intelligence, 2020.

[60]
L. Liu, J. Gu, K. Zaw Lin, T.-S. Chua, and C. Theobalt, “Neural sparse voxel
fields,” in Advances in Neural Information Processing Systems
(Neurips), 2020, pp. 15 651–15 663.

[61]
J. T. Barron, B. Mildenhall, M. Tancik, P. Hedman, R. Martin-Brualla, and P. P.
Srinivasan, “Mip-nerf: A multiscale representation for anti-aliasing neural
radiance fields,” in Proceedings of the IEEE International Conference
on Computer Vision (ICCV), 2021, pp. 5855–5864.

[62]
A. Yu, V. Ye, M. Tancik, and A. Kanazawa, “pixelnerf: Neural radiance fields
from one or few images,” in Proceedings of the IEEE Conference on
Computer Vision and Pattern Recognition (CVPR), 2021, pp. 4578–4587.

[63]
W. Song, C. Shi, Z. Xiao, Z. Duan, Y. Xu, M. Zhang, and J. Tang, “Autoint:
Automatic feature interaction learning via self-attentive neural networks,”
in Proceedings of the 28th ACM International Conference on Information
and Knowledge Management, 2019, pp. 1161–1170.

[64]
A. Trevithick and B. Yang, “Grf: Learning a general radiance field for 3d
scene representation and rendering,” in Proceedings of the IEEE
International Conference on Computer Vision (ICCV), 2021, pp.
15 182–15 192.

[65]
P. Henzler, J. Reizenstein, P. Labatut, R. Shapovalov, T. Ritschel, A. Vedaldi,
and D. Novotny, “Unsupervised learning of 3d object categories from videos
in the wild,” in Proceedings of the IEEE Conference on Computer Vision
and Pattern Recognition (CVPR), 2021, pp. 4700–4709.

[66]
A. Kendall, M. Grimes, and R. Cipolla, “Posenet: A convolutional network for
real-time 6-dof camera relocalization,” in Proceedings of the IEEE
International Conference on Computer Vision (ICCV), 2015, pp. 2938–2946.

[67]
J. Wu, L. Ma, and X. Hu, “Delving deeper into convolutional neural networks
for camera relocalization,” in Proceedings of the IEEE International
Conference On Robotics And Automation (ICRA), 2017, pp. 5644–5651.

[68]
S. Peng, Y. Liu, Q. Huang, X. Zhou, and H. Bao, “Pvnet: Pixel-wise voting
network for 6dof pose estimation,” in Proceedings of the IEEE
Conference on Computer Vision and Pattern Recognition (CVPR), 2019, pp.

[69]
Y. Zhou, C. Barnes, J. Lu, J. Yang, and H. Li, “On the continuity of rotation
representations in neural networks,” in Proceedings of the IEEE
Conference on Computer Vision and Pattern Recognition (CVPR), 2019, pp.

[70]
M. M. Bronstein, J. Bruna, Y. LeCun, A. Szlam, and P. Vandergheynst,
“Geometric deep learning: Going beyond euclidean data,” IEEE Signal
Process. Mag., vol. 34, no. 4, pp. 18–42, 2017.

[71]
T. N. Kipf and M. Welling, “Semi-supervised classification with graph
convolutional networks,” in International Conference on Learning
Representations (ICLR), 2016.

[72]
H. Su, S. Maji, E. Kalogerakis, and E. G. Learned-Miller, “Multi-view
convolutional neural networks for 3d shape recognition,” in
Proceedings of the IEEE International Conference on Computer Vision
(ICCV), 2015.

[73]
N. Ravi, J. Reizenstein, D. Novotny, T. Gordon, W.-Y. Lo, J. Johnson, and
G. Gkioxari, “Accelerating 3d deep learning with pytorch3d,”
arXiv:2007.08501, 2020.

[74]
A. X. Chang, T. Funkhouser, L. Guibas, P. Hanrahan, Q. Huang, Z. Li,
S. Savarese, M. Savva, S. Song, H. Su et al., “Shapenet: An
information-rich 3d model repository,” arXiv preprint
arXiv:1512.03012, 2015.

[75]
E. Insafutdinov and A. Dosovitskiy, “Unsupervised learning of shape and pose
with differentiable point clouds,” in Advances in Neural Information
Processing Systems (NeurIPS), 2018, pp. 2807–2817.

[76]
L. Ladicky, O. Saurer, S. Jeong, F. Maninchedda, and M. Pollefeys, “From point
clouds to mesh using regression,” in Proceedings of the IEEE
International Conference on Computer Vision (ICCV), 2017, pp. 3893–3902.

[77]
E. Smith, S. Fujimoto, A. Romero, and D. Meger, “Geometrics: Exploiting
geometric structure for graph-encoded objects,” in International
Conference on Machine Learning (ICML), 2019, pp. 5866–5876.

[78]
B. Curless and M. Levoy, “A volumetric method for building complex models from
range images,” 1996.

[79]
W. E. Lorensen and H. E. Cline, “Marching cubes: A high resolution 3d
surface construction algorithm,” in SIGGRAPH, 1987.

[80]
W. Chen, H. Ling, J. Gao, E. Smith, J. Lehtinen, A. Jacobson, and S. Fidler,
“Learning to predict 3d objects with an interpolation-based differentiable
renderer,” Advances in Neural Information Processing Systems
(NeurIPS), pp. 9609–9619, 2019.

[81]
S. Koch, A. Matveev, Z. Jiang, F. Williams, A. Artemov, E. Burnaev, M. Alexa,
D. Zorin, and D. Panozzo, “Abc: A big cad model dataset for geometric deep
learning,” in Proceedings of the IEEE Conference on Computer Vision
and Pattern Recognition (CVPR), 2019, pp. 9601–9611.

[]Chao Wen received the BE degree in computer science from University of Electronic Science and Technology of China, in 2018, the Master degree in computer science from the Academy of Engineering and Technology, Fudan University. His research is focused on 3D shape reconstruction and human image synthesis.

[]Yinda Zhang is a Senior Research Scientist at Google. His research interests lie at the intersection of computer vision, computer graphics, and machine learning. Recently, he focuses on empowering 3D vision and perception via machine learning, including dense depth estimation, 3D shape analysis, 3D scene understanding, and neural rendering. He received his Ph.D. in Computer Science from Princeton University, advised by Professor Thomas Funkhouser. Before that, he received a Bachelor degree from Dept. Automation in Tsinghua University, and a Master degree from Dept. ECE in National University of Singapore co-supervised by Prof. Ping Tan and Prof. Shuicheng Yan.

[]Chenjie Cao received the M.S. degree in Computer Science from East China University of Science and Technology, in 2019. He is currently pursuing the Ph.D. degree in Statistics from Fudan University. His research interests include machine learning, deep learning, 3D shape reconstruction and computer vision.

[]Zhuwen Li received the BE degree in computer
science from Tianjin University, in 2008, the master’s
degree in computer science from Zhejiang
University, in 2011, and the PhD degree from the
Department of Electrical and Computer Engineering,
National University of Singapore, in 2014.
Currently, he is a research scientist at Nuro, Inc.
Recently, he is working on perception in autonomous
driving, specifically in 3D structure recovery,
motion analysis, point cloud analysis, and
object detection.

[]Xiangyang Xue received the BS, MS, and PhD
degrees in communication engineering from
Xidian University, Xian, China, in 1989, 1992,
and 1995, respectively. He is currently a professor
of computer science with Fudan University,
Shanghai, China. His research interests include
multimedia information processing and machine

[]Yanwei Fu received the MEng degree from the Department of Computer Science and Technology, Nanjing University, China, in 2011, and the PhD degree from the Queen Mary University of London, in 2014. He held a post-doctoral position at Disney Research, Pittsburgh, PA, from 2015 to 2016. He is currently a tenure-track professor with Fudan University.
He was appointed as the Professor of Special Appointment (Eastern Scholar) at Shanghai Institutions of Higher Learning.
His work has led to many awards, including the IEEE ICME 2019 best paper.
He published more than 100 journal/conference papers including IEEE TPAMI, TMM, ECCV, and CVPR. His research interests are one-shot learning, and learning based 3D reconstruction.