# ReFu: Refine and Fuse the Unobserved View for Detail-Preserving Single-Image
3D Human Reconstruction

Gyumin Shim  ,  Minsoo Lee  Korea Advanced Institute of Science and
TechnologyDaejeonSouth Korea shimgyumin<EMAIL_ADDRESS>and  Jaegul
Choo  Korea Advanced Institute of Science and TechnologyDaejeonSouth Korea
<EMAIL_ADDRESS>
(2022)

###### Abstract.

Single-image 3D human reconstruction aims to reconstruct the 3D textured
surface of the human body given a single image. While implicit function-based
methods recently achieved reasonable reconstruction performance, they still
bear limitations showing degraded quality in both surface geometry and texture
from an unobserved view. In response, to generate a realistic textured
surface, we propose ReFu, a coarse-to-fine approach that refines the projected
backside view image and fuses the refined image to predict the final human
body. To suppress the diffused occupancy that causes noise in projection
images and reconstructed meshes, we propose to train occupancy probability by
simultaneously utilizing 2D and 3D supervisions with occupancy-based volume
rendering. We also introduce a refinement architecture that generates detail-
preserving backside-view images with front-to-back warping. Extensive
experiments demonstrate that our method achieves state-of-the-art performance
in 3D human reconstruction from a single image, showing enhanced geometry and
texture quality from an unobserved view.

3D human reconstruction, implicit function, volume rendering, conditional
generative adversarial nets

††copyright: acmlicensed††journalyear: 2022††conference: Proceedings of the
30th ACM International Conference on Multimedia; October 10–14, 2022; Lisboa,
Portugal††booktitle: Proceedings of the 30th ACM International Conference on
Multimedia (MM ’22), October 10–14, 2022, Lisboa, Portugal††price:
15.00††isbn: 978-1-4503-9203-7/22/10††doi: 10.1145/3503161.3547971††ccs:
Computing methodologies Shape representations††ccs: Computing methodologies
Appearance and texture representations Figure 1. Qualitative results of our
method in terms of surface geometry and texture. Given the source image (first
column), our method results in realistic textured meshes with high-frequency
details. In contrast, PaMIR shows over-smoothed outputs from an unobserved
view.

##  1\. Introduction

3D human reconstruction has received substantial attention due to its
importance and practicality in VR/AR content and metaverse domains. Image-
based 3D human reconstruction aims to predict the textured human mesh given a
single or multiple images. However, recovering a realistic 3D human from a
single image is an ill-posed problem due to depth ambiguity and occlusions.
Also, the high variance of human pose and clothes worsens the problem.

Recently, numerous deep-learning-based methods (Natsume et al., 2019; Saito et
al., 2019; Zheng et al., 2021b, 2019; Varol et al., 2018; Alldieck et al.,
2019; Jackson et al., 2018; Pavlakos et al., 2019; Zhu et al., 2019b) have
been proposed to reconstruct a 3D human body from a single image. In the early
stages, leveraging the pre-defined human template mesh, e.g. SMPL (Loper et
al., 2015), several methods (Pavlakos et al., 2019; Alldieck et al., 2019; Zhu
et al., 2019b) learn to deform the template mesh conditioned on a given source
image. Recently, among non-parametric representations such as voxels(Varol et
al., 2018; Jackson et al., 2018), silhouettes(Natsume et al., 2019), and depth
maps (Gabeur et al., 2019), methods based on implicit representations have
achieved a paradigm shift in 3D human reconstruction tasks, showing
substantial improvement for representing the textured surface of clothed
humans. Implicit-function-based methods (Saito et al., 2019; Saito et al.,
2020) conditioned on the pixel-aligned feature of a source image learn to
estimate occupancy fields and corresponding textures. PaMIR (Zheng et al.,
2021b) further improves reconstruction accuracy by combining the volumetric
feature from SMPL with condition variables.

However, from a view not observed in the given source image, the quality of
the surface geometry and the texture are significantly degraded with an over-
smoothed mesh surface and a lack of high-frequency texture details. We argue
that this limitation is due to the per-point loss function, e.g. the L1 Loss
and the mean squared error which encourage the point-wise averages of
plausible solutions causing over-smoothed results (Isola et al., 2017; Ledig
et al., 2017). To generate a realistic textured mesh from any view, we propose
a staged method where we predict a coarse human body (initial stage), refine
the coarse image projected from the unobserved view (refinement stage), and
fuse the refined image with the input condition to reconstruct the final human
body (fusion stage).

To obtain a projected coarse image from a 3D implicit function, we adopt a
volume rendering process, i.e. ray-tracing techniques for compositing colored
density, that is proven to be effective in synthesizing high-quality images.
However, as shown in Fig. 2, the volume-rendered images of PaMIR (Zheng et
al., 2021b), the state-of-the-art 3D human reconstruction method, are noisy
and blurry. This undesirable noise stems from the diffused occupancy, i.e.
non-zero occupancy value outside the object surface, that blocks the object
surface. Also, the diffused occupancy can cause floating artifacts in the
reconstructed meshes. To suppress the diffused occupancy around the object, we
propose to train the occupancy probability by simultaneously utilizing 2D and
3D supervisions with the occupancy-based volume rendering. To achieve this, we
interpret alpha values in the cumulative rendering process (Mildenhall et al.,
2020) as occupancy probabilities that are trained with 3D supervisions. Since
the higher transmittance value is computed at the diffused occupancy area, the
volume rendering loss is efficient in lowering the entropy of the occupancy
values. Thanks to the volume-rendering-combined 3D supervised learning, we can
acquire noiseless occupancy fields and a clear projected image from any target
view.

---
Figure 2. Volume-rendered images and reconstructed meshes of PaMIR (Zheng et
al., 2021b) and ours. While PaMIR presents artifacts in the volume-rendered
images and the meshes, our initial stage shows the clear projection images and
the reconstructed meshes.

Leveraging a clear projection image, we introduce architecture that generates
a realistic image from an unobserved view, i.e. backside, in a coarse-to-fine
manner. To refine the coarse image with high-frequency details, we transfer
the source local feature to the target image using the front-to-back warping
function. The function can be obtained by volume rendering the 3D query points
with the value of the source grid coordinates. The rationale behind this
warping function is that the corresponding coordinates between the front and
backside view share semantically similar visual features such as the texture
of clothes and skin color. By transferring the local feature of the front view
to the backside and injecting it with spatial modulation (Albahar et al.,
2021; Park et al., 2019b), we extend the StyleGAN generator (Karras et al.,
2020) that takes a coarse image as input and generates a realistically refined
image. Using the refined backside-view images as an additional input, our
final model can predict a perceptually enhanced 3D human mesh. We demonstrate
that the quality of the mesh and texture outperforms the previous state-of-
the-art methods.

In summary, our contributions are as follows:

  * •

We propose to train occupancy probabilities by leveraging both 2D and 3D
supervisions with the occupancy-based volume rendering.

  * •

We also propose the refinement architecture that can generate high-frequency
details in the backside-view image by warping local features from the source
to the target view.

  * •

We demonstrate that the quality of single-image 3D human reconstruction is
improved for both geometry and texture by fusing the refined backside-view
image.

##  2\. Related Work

###  2.1. 3D Human Reconstruction

Previous studies focus on reconstructing 3D human meshes using video sequences
(Alldieck et al., 2018a, b) or multi-view RGB-D data (Yu et al., 2018; Zheng
et al., 2018). While these approaches show reasonable 3D reconstruction
qualities by addressing the depth ambiguity of shapes, they do not work well
against only single or sparse input images. To reconstruct human meshes from a
single image, several methods (Loper et al., 2015; Pavlakos et al., 2019)
utilize parametric body models that provide a strong prior knowledge of the
human body. To represent fine details of the surface of the parametric human
body, other studies (Alldieck et al., 2019; Lazova et al., 2019; Lahner et
al., 2018) attempted to estimate displacements in a pre-defined UV space.
Methods (Varol et al., 2018; Jackson et al., 2018) that combine a voxel
representation with the parametric body model further improve the
reconstruction quality. However, such parametric-model-based methods still
show severe limitations on predicting surfaces that have large deformations
such as loose clothes or hairs. Also, the high memory cost of voxel limits the
resolution, which leads to the degradation of the fine details of mesh
surfaces.

Compared to explicit representations such as voxel grids (Maturana and
Scherer, 2015), point clouds (Qi et al., 2017; Lin et al., 2018), and meshes
(Groueix et al., 1802; Wang et al., 2018), implicit representations (Park et
al., 2019a; Mildenhall et al., 2020) define the 3D surface with continuous
probability fields. As implicit representations have been widely utilized for
3D object representation due to their memory efficiency and continuity, they
are also widely utilized in 3D human reconstruction (Saito et al., 2019; Saito
et al., 2020; Zheng et al., 2021b; He et al., 2020). Recently, PIFu (Saito et
al., 2019) introduced a pixel-aligned implicit function that represents the
continuous occupancy and color value for each 3D query point conditioned on
the corresponding 2D image feature. PIFu-HD (Saito et al., 2020) and Geo-PIFu
(He et al., 2020) further improved the reconstruction quality level of detail
by formulating coarse-to-fine approaches. PaMIR (Zheng et al., 2021b) combines
a parametric human body model with the implicit function to improve the
performance under the severe occlusions or large pose variations. Although
these recent methods show a reasonable reconstruction quality of surface
geometry and texture, there are still limitations in representing fine details
from the backside view. To overcome the degraded quality from unobserved
views, we propose ReFu that refines and fuses the backside view to enhance the
final output result.

###  2.2. Condition-guided Human Image Synthesis

Condition-guided human image synthesis has attracted a substantial amount of
attention due to its wide range of applications, e.g. motion transfer, human
reposing, and virtual try-on. This task aims to translate a source human image
to a target image in which the shape and appearance are changed according to
given conditions, e.g. viewpoint, input pose, and target clothes. Inspired by
spatial transformer networks (Jaderberg et al., 2015), recent works predict a
spatial transformation and transfer the source image pixel/feature to a target
image space to preserve the high-frequency details. The warping function is
predicted by conditions such as skeleton pose (Zhu et al., 2019a; Ren et al.,
2020), UV parameterization pose (Sarkar et al., 2021; Albahar et al., 2021;
Yoon et al., 2021), and human parsing map (Yu et al., 2019; Choi et al., 2021;
Xu and Loy, 2021). A state-of-the-art human reposing method (Albahar et al.,
2021) predicts the warping function from the UV parameterization pose (Güler
et al., 2018) and successfully renders a photo-realistic human image
leveraging the StyleGAN (Karras et al., 2019) architecture with the warped
source feature.

Although we adopt the condition-guided StyleGAN architecture for refinement,
our method differs from the previous work (Albahar et al., 2021) in two
aspects. First, our method is conditioned on the coarse backside-view image
obtained in advance by projecting our 3D human implicit function that contains
the pose information and the approximate texture. Second, we compute the
front-to-back warping function in a 3D-aware manner by projecting source grid
coordinates to the backside view. In this way, the coarse backside-view image
is refined into the photo-realistic image that is utilized for generating the
detail-preserving 3D human body.

---
Figure 3. Overview of our approach. (a) denotes the initial stage that
predicts the coarse textured surface. (b) indicates the refinement stage that
takes the projected backside-view image as an input and generates a realistic
image with warped source features. Fusing the refined target image with the
source image, our fusion network predicts the final human body using the
identical architecture of the initial stage.

##  3\. Proposed Method

An overview of our proposed method is illustrated in Figure 3. We propose a
three-stage approach that consists of 1) the initial stage, 2) the refinement
stage, and 3) the fusion stage. Given a single image I as input, our initial
stage predicts the coarse textured surface. In the next stage, we refine the
image projected from the backside view where the initial geometry and texture
lack reality. Lastly, the fusion network reconstructs the final textured mesh
by incorporating the refined backside-view image into the input conditions.

###  3.1. Preliminary : SMPL

Following previous studies (Zheng et al., 2019, 2021b), the parametric human
body model, SMPL (Loper et al., 2015), is utilized to extract volumetric
features from an input image. SMPL is a function M($\cdot$) that maps pose
$\theta$ and shape $\beta$ to a mesh of 6890 vertices. Given the template
model ${\mathbf{T}}$, we obtain a posed shape instance as

(1) $\begin{gathered}M(\beta,\theta)=W(T(\beta,\theta),J(\beta),\theta,{\bf
W}),\\\ T(\beta,\theta)={\bf T}+B_{s}(\beta)+B_{p}(\theta),\end{gathered}$

where $W$ denotes a linear blend-skinning function with the weight $\bf W$,
$B_{p}(\theta)$ and $B_{s}(\beta)$ are pose-dependent and shape-dependent
deformations from the T-shape template model T, respectively, and J($\cdot$)
denotes the skeleton joints. As proven in previous work (Zheng et al., 2019,
2021b), the SMPL model fit to the input image provides a strong prior with
respect to the pose and shape of a human body. Among recent studies
(Kolotouros et al., 2019b; Kocabas et al., 2020; Kolotouros et al., 2019a)
that estimate SMPL parameters from a single image, we use GCMR (Kolotouros et
al., 2019a), the state-of-the-art method, for our experiments.

###  3.2. Occupancy and Texture Fields

The textured surface can be represented with two components: occupancy and
texture. Inspired by the previous studies (Mescheder et al., 2019; Park et
al., 2019a), we define a deep implicit function that predicts an occupancy
probability and texture at a given 3D query point as

(2) ${G}_{\theta}(p,C(p))=(\alpha_{p},{\mathbf{c}}_{p},{\gamma}_{p}),$

where $\alpha_{p}$ and ${\mathbf{c}}_{p}$ denote the occupancy probability and
texture, respectively, at the corresponding 3D coordinate $p$. Output values
are predicted from the network $G_{\theta}$ given a condition variable $C(p)$.
$\gamma$ is used for blending the predicted RGB value with the one sampled
from the source image. The final color is defined as
${\mathbf{c}}^{\prime}_{p}=\gamma\cdot
S(\mathbf{I},\pi(p))+(1-\gamma)\cdot{\mathbf{c}}_{p}$, where $\pi(p)$ denotes
the 2D projection coordinate of $p$ on the source image $\mathbf{I}$ and
$S(\cdot)$ indicates the sampling function to sample the value at a query
location using the bilinear interpolation. Given occupancy fields that
represent the continuous inside/outside probabilities, the surface can be
defined as the level set of an occupancy probability by extracting the iso-
surface, i.e. $\alpha=0.5$.

####  3.2.1. Pixel/Voxel-aligned Representation

To infer the 3D textured surface from a single image using an implicit
function, first, the pixel-aligned feature (Saito et al., 2019) is utilized as
the condition variable. Furthermore, to strengthen the reconstruction
performance under severe occlusions or large pose variations, the voxel-
aligned feature is also integrated with the condition variables (Zheng et al.,
2021b). The volumetric feature can be encoded from the predefined parametric
human body model, i.e. SMPL (Loper et al., 2015), using a 3D CNN network. The
condition variable is defined as

(3) ${C(p)}=(S({\mathbf{F}}_{I},\pi(p)),S({\mathbf{F}}_{V},p))^{T},$

where ${\mathbf{F}}_{I}=E_{I}({\mathbf{I}})$ indicates the 2D image feature
map extracted from the 2D CNN encoder $E_{I}$,
${\mathbf{F}}_{V}=E_{V}({\mathbf{V}}_{O})$ represents the volumetric feature
encoded from the 3D CNN network $E_{V}$, and ${\mathbf{V}}_{O}$ is the
occupancy volume generated by voxelizing the SMPL model predicted from the
input image.

To train our model, we sample 3D points around the ground-truth mesh. Our
model learns to predict the occupancy probabilities and texture by minimizing
the per-point reconstruction loss with 3D supervision. The reconstruction loss
is defined as

(4)
$\mathcal{L}_{R}=\frac{1}{n_{p}^{o}}\sum_{i=1}^{n_{p}^{o}}|{\bf\alpha}_{p_{i}}-{\bf\alpha}^{*}_{p_{i}}|^{2}~{}+~{}\frac{1}{n_{p}^{c}}\sum_{i=1}^{n_{p}^{c}}|{\mathbf{c}}_{p_{i}}-{\mathbf{c}}^{*}_{p_{i}}|,$

where $n_{p}^{o}$ and $n_{p}^{c}$ are the number of sampled points for
occupancy and color, respectively. $p_{i}$ is the i-th sampled 3D point, and
$\alpha^{*}$ and ${\mathbf{c}}^{*}$ are the ground-truth occupancy and RGB
value of $p_{i}$, respectively. To train the color prediction, query points
are only sampled near the surface, while query points are sampled around the
entire 3D space for learning occupancy probabilities. Similarly, the color
reconstruction loss is applied to the $\gamma$-composited final color,
${\mathbf{c}}^{\prime}$.

####  3.2.2. Volume Rendering Loss

However, as shown in Fig. 2, we discovered that the network trained only with
3D supervision produces the diffused occupancy that results in undesirable
artifacts in the projection image and the reconstructed mesh. To avoid such
issues, we propose combining the 2D volume rendering loss (Mildenhall et al.,
2020) with the 3D reconstruction loss by interpreting alpha values in the
volume rendering process as occupancy probabilities. We found that the alpha
values follow a similar tendency with occupancy probabilities that are close
to zero for points outside the surface while approaching 1 for points inside
the surface. Similar interpretation with respect to alpha values is attempted
in a 3D-aware image generation method (Xu et al., 2021) to define a 3D surface
when only 2D supervisions are provided. Meanwhile, we consider the alpha
values as the occupancy probabilities that are trained with 3D supervisions,
which helps in additionally utilizing 2D supervisions, i.e. projected images
from ground-truth meshes. Thus, we formulate the occupancy-based volume
rendering as

(5)
$\mathbf{c}_{r}=\sum_{j=1}^{N}\alpha_{j}\prod_{k=1}^{j-1}\left(1-\alpha_{k}\right)\mathbf{c}_{j},$

where ${\mathbf{r}}$ is the ray marched from the randomly selected target view
and $N$ and $j$ are the number and the index of sampled points along a ray,
respectively. The volume rendering loss is defined as

(6)
$\mathcal{L}_{\text{vol}}=\frac{1}{n_{r}}\sum_{i=1}^{n_{r}}|{\mathbf{c}}_{r}-{\mathbf{c}}^{*}_{r}|,$

where $n_{r}$ is the number of rays, and ${\mathbf{c}}^{*}$ is the ground
truth RGB value in the target-view image. The same volume rendering loss is
applied to the $\gamma$-composited final color value, ${\mathbf{c}}^{\prime}$.
We adopt the hierarchical sampling strategy proposed in NeRF (Mildenhall et
al., 2020) that densely samples points based on the normalized weights of
colors along the points on the ray.

###  3.3. Refinement and Fusion

Since the geometry and the texture of the initially predicted 3D surface are
far short of reality from the backside view, we propose the refine-and-fuse
approach that 1) refines the coarse backside-view image by warping the source
feature and 2) fuses the refined image with the input condition to generate
the final textured surface.

####  3.3.1. Refinement Stage

After the initial stage, we can render the coarse image
${\mathbf{I}}^{\prime}_{trg}$ from the backside view and the warping function
${\mathbf{f}}$ from the source image to the target image. The warping function
is computed by volume-rendering the 3D query points $p$ with the value of
source grid coordinates $\pi(p)$ using the initially learned occupancy values.
This warping function maps the coordinates of the source view to those of the
target view and transfers the semantically similar visual features, e.g. the
texture of clothes and skin color, from the front to the back.

Figure 3(b) shows the overview of our refinement network. We use the StyleGAN2
architecture (Karras et al., 2020) with the spatial modulation (Park et al.,
2019b; Albahar et al., 2021) that takes the coarse target-view image
${\mathbf{I}}^{\prime}_{trg}$ as the input and the warped source image feature
as the modulation condition. The source image encoder encodes the source image
${\mathbf{I}}$ to multi-scale features ${\mathbf{F}}^{src}_{i}$ with several
residual blocks. Then the source features are warped to the target view with
the warping function $\mathbf{f}$ for spatial alignment. As illustrated in
Fig. 4, the warped source feature ${\mathbf{F}}^{trg}_{i}$ is passed to two
1$\times$1 convolution layers that output affine parameters of the modulation
layer, i.e. scale ${\boldsymbol{\alpha}}_{i}$ and shift
${\boldsymbol{\beta}}_{i}$, to perform the spatial modulation. Unlike the
standard modulation of StyleGAN (Karras et al., 2019), our network utilizes
the spatial modulation for the intermediate feature of the i-th style block
${\mathbf{F}}_{i}$ to preserve the fine details of a source image. The
modulated feature $\hat{\mathbf{F}}_{i}$ is passed to a 3$\times$3 convolution
layer of the style block that outputs $\hat{\mathbf{F}}^{out}_{i}$ and is then
standardized to have zero mean and unit standard deviation. The final output
of the style block ${\mathbf{F}}_{i+1}$ is computed by adding the bias and the
noise (Karras et al., 2020) to the standardized feature
$\bar{\mathbf{F}}^{out}_{i}$. The entire formulation of the style block with
feature warping is written as

(7)
$\begin{gathered}{\mathbf{F}}^{trg}_{i}=\text{warp}({\mathbf{F}}^{src}_{i},{\mathbf{f}}),\quad{\boldsymbol{\alpha}}_{i},{\boldsymbol{\beta}}_{i}=E_{\text{affine}}({\mathbf{F}}^{trg}_{i}),\\\
\hat{\mathbf{F}}_{i}={\boldsymbol{\alpha}}_{i}\otimes\
{\mathbf{F}}_{i}+{\boldsymbol{\beta}}_{i},\quad\bar{\mathbf{F}}^{out}_{i}=\frac{\hat{\mathbf{F}}^{out}_{i}-\mu(\hat{\mathbf{F}}^{out}_{i})}{\sigma(\hat{\mathbf{F}}^{out}_{i})},{\mathbf{F}}_{i+1}=\bar{\mathbf{F}}^{out}_{i}+{\mathbf{b}}_{i}+{\mathbf{n}}_{i}\end{gathered}$

where $\mu(\hat{\mathbf{F}}^{out}_{i})$ and
$\sigma(\hat{\mathbf{F}}^{out}_{i})$ denote the mean and standard deviation of
$\hat{\mathbf{F}}^{out}_{i}$, and ${\mathbf{b}}_{i}$ and ${\mathbf{n}}_{i}$
are the bias and the noise (Karras et al., 2020), respectively.

---
Figure 4. Style block with feature warping. To spatially modulate the
intermediate feature of the i-th style block $\mathbf{F}_{i}$, we warp the
source feature with the warping function ${\mathbf{f}}$. After being modulated
with $\boldsymbol{\alpha}$ and $\boldsymbol{\beta}$, the feature is
standardized. The bias $\mathbf{b}_{i}$ and the noise $\mathbf{n}_{i}$ are
then added to the output. The upsampling layer is added before the 3$\times$3
convolution layer in every two style blocks.

Our refinement network is trained by optimizing the following objectives: 1)
the $\ell_{1}$ loss, 2) the perceptual loss, and 3) the adversarial loss. We
minimize the $\ell_{1}$ loss between the foreground human regions of the
refined target-view image $\hat{\mathbf{I}}_{trg}$ and those of the ground-
truth target-view image ${\mathbf{I}}_{trg}$. The reconstruction loss is
formulated as

(8)
$\mathcal{L}_{\ell_{1}}(\psi)=\left\|\hat{\mathbf{I}}_{trg}\cdot{\mathbf{M}}_{trg}-{\mathbf{I}}_{trg}\cdot{\mathbf{M}}_{trg}\right\|_{1},$

where ${\mathbf{M}}_{trg}$ is the human foreground mask, and
$\hat{\mathbf{I}}_{trg}$ denotes the refined target-view image. Also, we apply
the perceptual loss to our training objective by minimizing the weighted sum
of the $\ell_{1}$ loss between the pre-trained VGG features of the refined
target-view image $\hat{\mathbf{I}}_{trg}$ and those of the ground-truth
target-view image ${\mathbf{I}}_{trg}$ as

(9)
$\mathcal{L}_{\text{vgg}}(\psi)=\sum_{i=1}^{5}w_{i}\cdot\left\|VGG_{l_{i}}\left(\hat{\mathbf{I}}_{trg}\cdot{\mathbf{M}}_{trg}\right)-VGG_{l_{i}}\left({\mathbf{I}}_{trg}\cdot{\mathbf{M}}_{trg}\right)\right\|_{1},$

where $w$ is [$\frac{1}{32},\frac{1}{16},\frac{1}{8},\frac{1}{4},1.0$] and $l$
is [1,6,11,20,29].

Lastly, we adopt a conditional GAN (Isola et al., 2017) by training a
discriminator to match the real and fake images conditioned on the coarse
target-view image. The non-saturating GAN loss (Goodfellow et al., 2014) is
used with R1 regularization (Mescheder et al., 2018). The entire formulation
of the conditional GAN loss is written as

(10)
$\begin{split}\vspace{-0.3cm}\mathcal{L}_{\text{adv}}(\psi,\phi)=\mathbf{E}[g(D_{\phi}(G_{\psi}({\mathbf{I}},{\mathbf{I}}^{\prime}_{trg},{\mathbf{f}}),{\mathbf{I}}^{\prime}_{trg}))]\\\
+\mathbf{E}[g(-D_{\phi}({\mathbf{I}},{\mathbf{I}}^{\prime}_{trg}))+\lambda\|\nabla
D_{\phi}({\mathbf{I}},{\mathbf{I}}^{\prime}_{trg})\|^{2}],\vspace{-0.3cm}\end{split}$

where $g(x)=-log(1+exp(-x))$, $G_{\psi}$ denotes our refinement generator, and
$D_{\phi}$ indicates the discriminator. $\lambda$ is the hyper-parameter that
balances the gradient penalty effect.

Our full training objective functions for the refinement network $G_{\psi}$
are written as

(11)
$\displaystyle\mathcal{L}_{\text{total}}=\mathcal{L}_{\text{adv}}+\lambda_{\text{vgg}}\mathcal{L}_{\text{vgg}}+\lambda_{\ell_{1}}\mathcal{L}_{\ell_{1}},$

where $\lambda_{vgg}$ and $\lambda_{\ell_{1}}$ are hyperparameters determining
the importance of each loss.

####  3.3.2. Fusion Stage

To generate the final output utilizing the refined backside-view image, we
train the fusion network that takes a pair of front and backside-view images
as an input. Instead of the output of our refinement network, i.e. the refined
image from the coarse backside-view image, the ground-truth backside-view
image is used for training. Unlike the initial stage, the pixel-aligned
feature is additionally extracted from the backside-view image and is then
concatenated to the pixel-aligned feature from the source image. The condition
variable is defined as

(12)
${C(p)}=(S({\mathbf{F}}_{I},\pi(p)),S({\mathbf{F}}_{{\mathbf{I}}_{trg}},\pi(p)),S({\mathbf{F}}_{V},p))^{T}.$

The $\gamma$ values are also estimated to independently sample the color
values from the given source image and the backside-view image. The final
$\gamma$-composited color value is calculated as

(13) $\displaystyle{\bf c}^{\prime}_{p}=\gamma_{1}\cdot
S(\mathbf{I},\pi(p))+\gamma_{2}\cdot
S(\mathbf{I}_{trg},\pi(p))+\gamma_{3}\cdot{\bf c}_{p},$

where $\gamma$ values are normalized with the softmax function. Except for the
input condition and the $\gamma$ compositing, the training process, including
the network architecture and the loss function, is identical to the initial
stage.

##  4\. Experiments

###  4.1. Experimental Setup

####  4.1.1. Dataset

For training and evaluating our method, we collected 526 samples from the
THUman2.0 (Zheng et al., 2021a) dataset and 498 samples from the Twindom
dataset that include high-quality textured human scans. 50 samples from the
THUman2.0 and Twindom datasets were used for evaluation and the remaining
samples were used for training (476 samples from THUman2.0 and 448 samples
from Twindom). 1,396 real-world full-body images collected from DeepFashion
(Liu et al., 2016) were also utilized for evaluations.

To preprocess the training data, we followed PaMIR (Zheng et al., 2021b) by
rendering the training scans from 360 viewpoints at a resolution of
512$\times$512\. MuVS (Huang et al., 2017) was applied to the rendered multi-
view images for preparing the ground-truth SMPL model. For evaluation, the
evaluation set of 3D scans from 4 views spanning every 90 degrees in the
horizontal axis were rendered.

####  4.1.2. Training/Evaluation Setup

For the initial stage, the network is trained with a learning rate of $2\times
10^{-4}$ using the ADAM optimizer with $\beta_{1}=0.9$ and $\beta_{2}=0.999$.
The network is trained for 200,000 iterations with a batch size of 3, and
5,000 points are sampled both for occupancy and color reconstruction. For
optimizing volume rendering loss, the camera locations are randomly sampled
among 360 degrees around the object. 1,000 rays are cast for each sampled
camera location and 48 samples are used per ray adopting hierarchical
sampling. Instead of using the predicted SMPL models (Zheng et al., 2021b), we
use the ground-truth ones during training since we empirically found that
using the predicted SMPL model causes unstable training due to large pose
variations of THUman2.0. The pre-trained GCMR is used and body reference SMPL
optimization (Zheng et al., 2021b) is performed for more accurate SMPL input
in the evaluation. The fusion stage follows the same experimental setup as the
initial stage, except for a batch size of 2.

For the refinement network, different hyper-parameter values are set for the
generator and the discriminator. The network is trained with a batch size of
1, a learning rate of $2\times 10^{-3}\times ratio$, and the ADAM optimizer
with $\beta_{1}=0$ and $\beta_{2}=0.99^{ratio}$. The $ratio$ is set to
$\frac{4}{5}$ for the generator and $\frac{16}{17}$ for the discriminator. We
train the initial stage first and then train the refinement stage using the
output of the initial stage.

###  4.2. Experimental Results

Here, we compare our method with the state-of-the-art methods and demonstrate
that our method produces more realistic 3D human reconstruction from a single
image. We select PIFu (Saito et al., 2019) and PaMIR (Zheng et al., 2021b) as
our baselines which are single-image 3D human reconstruction methods based on
an implicit function. Note that we omit the comparisons where textures cannot
be estimated and open-source codes are not available. The output meshes are
extracted using the Marching Cube algorithm at an occupancy threshold of 0.5
and textured by predicting the colors of the extracted vertices. All
comparison methods are retrained on our training datasets.

####  4.2.1. Qualitative Comparisons

Qualitative comparisons are shown in Figure 5. As shown in the figure, our
method shows the best visual quality from any view in terms of surface
geometry and texture. PIFu (Saito et al., 2019) has difficulty in
reconstructing human bodies in challenging poses and suffers from depth
ambiguity (see row 6) since it does not utilize any prior knowledge of human
shape as a condition. While PaMIR (Zheng et al., 2021b) succeeds in
reconstructing a full-body model under challenging poses, it still suffers
from floating artifacts (row 3). Both compared methods represent an over-
smoothed textured surface, especially in the unseen regions, for all examples.
However, as noticeable in the first row of Figure 5, our method shows the
backside view with high-frequency details such as wrinkles in clothes and
hair. Our method is also capable of preserving patterns of clothes by warping
the source features (row 5) and generating the face when the backside of the
human body is captured for the source input image (row 6). Thanks to our
refine-and-fuse approach, our method exhibits the realistic quality of both
surface geometry and texture from the unobserved view.

---
Figure 5. Qualitative comparison of output meshes on the THUman2.0, Twindom,
and DeepFashion datasets. Given the source image in the first column, meshes
without texture are presented in rows 1-3, and textured meshes are presented
in rows 4-7. The output meshes are captured from the front, side, and back
views for each method.

####  4.2.2. Quantitative Comparisons

We quantitatively evaluated our method by measuring the geometry
reconstruction accuracy and texture quality. To measure the geometry
reconstruction, we followed the previous studies (Saito et al., 2019; Zheng et
al., 2021b), by measuring the point-to-surface (P2S) error and the Chamfer
distance between the predicted mesh and the ground-truth one. The quantitative
results are presented in Table 1. Although our method shows outstanding
qualitative results (see Fig. 5), it shows marginal improvement compared to
the baselines because these distance-based metrics cannot measure the
perceptual quality.

Table 1. Quantitative comparison of surface geometry reconstruction on the THUman2.0 and Twindom datasets. | THUman2.0 | Twindom
---|---|---
Method | P2S↓ (cm) | Chamfer↓ (cm) | P2S↓ (cm) | Chamfer↓ (cm)
PIFu (Saito et al., 2019) | 1.997 | 1.807 | 1.199 | 1.198
PaMIR (Zheng et al., 2021b) | 1.346 | 1.417 | 0.949 | 0.950
Ours (Initial) | 1.358 | 1.459 | 0.941 | 0.953
Ours | 1.339 | 1.440 | 0.923 | 0.940

To further evaluate the texture quality of our approach, we measured the peak
signal-to-noise ratio (PSNR) and the learned perceptual image patch similarity
(LPIPS). PSNR measures the distortion and LPIPS measures the perceptual
similarity between images. We also introduce the ReID distance to evaluate the
perceptual similarity between the source image and the predicted backside-view
image. Using the pre-trained encoder for the person re-identification task
(Zhou et al., 2019), the cosine distance is computed between the features
extracted from the source image and the generated backside-view image. To
evaluate the pure texture quality without geometric errors, the textured mesh
is generated with the ground-truth vertices by predicting their vertex colors.
Note that the ground-truth SMPL is used for the input of both PaMIR and our
method. We then render the backside color image of the output textured mesh by
projecting to the reversed viewpoints of the source views without a light
condition. The quantitative results for texture are shown in Table 2. As shown
in the table, ReFu achieves the best LPIPS and ReID distance by a large
margin, which implies that our method predicts realistic and source-consistent
texture.

Table 2. Quantitative comparison of texture quality on the THUman2.0, Twindom, and DeepFashion datasets. The texture quality is presented by rendering the predicted vertex colors of the ground-truth mesh vertices. Since the ground-truth mesh vertices of the DeepFashion dataset do not exist, the colors of the predicted mesh vertices are rendered. | THUman2.0 | Twindom | DeepFashion
---|---|---|---
Method | PSNR↑ | LPIPS↓ | ReID dist↓ | PSNR↑ | LPIPS↓ | ReID dist↓ | ReID dist↓
PIFu (Saito et al., 2019) | 25.13 | 0.095 | 0.199 | 24.60 | 0.089 | 0.194 | 0.269
PaMIR (Zheng et al., 2021b) | 26.62 | 0.078 | 0.163 | 24.18 | 0.081 | 0.176 | 0.271
Ours (Initial) | 26.46 | 0.079 | 0.167 | 24.68 | 0.079 | 0.171 | 0.271
Ours | 25.97 | 0.074 | 0.139 | 25.77 | 0.077 | 0.148 | 0.227

###  4.3. Ablation Study

####  4.3.1. Refine-and-Fuse Approach

Here, we quantitatively and qualitatively demonstrate the effectiveness of our
refine-and-fuse methodology. The quantitative comparison between the initial
and the final outputs in terms of surface geometry and texture are shown in
Table 1 and Table 2, respectively. Our refine-and-fuse approach improves the
results of the initial stage in P2S and the Chamfer distance in Table 1, which
indicates our methodology enhances the geometry reconstruction quality by
alleviating the ambiguity of the backside shape. Also, our final outputs
present a perceptually improved texture quality showing a lower LPIPS and ReID
distance than the ones of the initial stage in Table 2.

The qualitative differences between the outputs of the initial and the fusion
stage are presented in Figure 6. Our fusion network successfully generates
realistic details such as faces and cloth wrinkles, while the initial outputs
show an over-smoothed surface on the backside. Also, the undesired artifacts
copied from the source image are removed after refinement (row 2).

---
Figure 6. Ablation study of our refine-and-fuse approach. The images of
textured meshes captured from the front and backside views are presented.

####  4.3.2. Spatial Modulation with Source Feature Warping

We analyze the effectiveness of the source feature and the spatial modulation
with the front-to-back warping function in the refinement stage. Our
refinement network is compared with two variants: (a) spatial modulation with
the coarse target feature and (b) AdaIN (Huang and Belongie, 2017) modulation
with the source feature. For variant (b), the condition of AdaIN is defined as
the style vector that is the average-pooled output of the source image
encoder. As shown in Figure 7, variant (a) fails to generate realistic texture
since the high-frequency information is not provided from any real images.
Variant (b) cannot preserve the local texture details in the source image,
such as patterns in clothes, as the condition of modulation loses the spatial
information of the source feature. Thanks to our spatial modulation with the
source feature warping, our full refinement network effectively generates
realistic images by preserving the local details in the source image.

Table 3. Ablation study of the source feature and the spatial modulation with feature warping. Note that the texture quality on the output images of the refinement stage are measured. | THUman2.0 | Twindom | DeepFashion
---|---|---|---
Method | PSNR↑ | LPIPS↓ | ReID dist↓ | PSNR↑ | LPIPS↓ | ReID dist↓ | ReID dist↓
(a) | 25.23 | 0.065 | 0.168 | 24.78 | 0.065 | 0.157 | 0.208
(b) | 25.75 | 0.059 | 0.158 | 25.48 | 0.058 | 0.148 | 0.199
Ours | 26.06 | 0.053 | 0.143 | 25.85 | 0.052 | 0.126 | 0.167
---
Figure 7. Ablation study on the source feature and spatial modulation with
warping in the refinement network. Given the source image and the coarse
target-view image in columns 1-2, we compare the refined images with those of
the variants. Detailed experimental settings for the variants are found in
Sec. 4.3.2.

##  5\. Limitations and Conclusions

Although the parametric body model provides strong prior knowledge of a human
body, a performance bottleneck may occur based on the predicted SMPL accuracy.
The large errors of the SMPL prediction on challenging poses often lead to
failure cases, the examples of which are presented in the supplementary
material. Improving the performance of the SMPL prediction model may boost the
output quality of our model. Also, by utilizing multi-modal data such as
silhouettes, segmentations, and 2D/3D keypoints from pre-trained networks, we
expect more accurate inference-stage SMPL optimization compared to body
reference optimization (Zheng et al., 2021b).

Our model achieves successful results; however, its training and inference
process is complicated due to the staged methodology. In another trial, we
attempted single-stage learning where the volume-rendered image is refined
while training the initial occupancy and texture. However, the volume
rendering process of a high-resolution image disrupts the training process,
consuming substantial computational time and memory. We plan to address this
issue in our future work by considering more memory-efficient 3D
representations or volume-rendering techniques.

In this study, we propose ReFu that refines the projected backside-view image
and fuses the refined image for predicting the final human body in a coarse-
to-fine manner to improve over-smoothed surface geometry and blurry texture
from an unobserved view. To alleviate the artifacts in the projected images
and the reconstructed meshes, we propose training the occupancy probability by
concurrently leveraging 2D and 3D supervisions with the occupancy-based volume
rendering. In addition, we designed a refinement network that generates
backside-view images with fine details using the front-to-back warping
function. The proposed method achieves state-of-the-art performance in 3D
human reconstruction from a single image in terms of surface geometry and
texture quality.

###### Acknowledgements.

This work was supported by FNS HOLDINGS and the Institute of Information &
communications Technology Planning & Evaluation (IITP) grant funded by the
Korean government(MSIT) (No. 2019-0-00075, Artificial Intelligence Graduate
School Program(KAIST), No. 2021-0-01778, Development of human image synthesis
and discrimination technology below the perceptual threshold, No.2021-0-02068,
Artificial Intelligence Innovation Hub).

## References

  * (1)
  * Albahar et al. (2021) Badour Albahar, Jingwan Lu, Jimei Yang, Zhixin Shu, Eli Shechtman, and Jia-Bin Huang. 2021\.  Pose with Style: Detail-preserving pose-guided image synthesis with conditional stylegan.  _ACM Trans. on Graph. (ToG)_ 40, 6 (2021), 1–11.
  * Alldieck et al. (2018a) Thiemo Alldieck, Marcus Magnor, Weipeng Xu, Christian Theobalt, and Gerard Pons-Moll. 2018a.  Detailed human avatars from monocular video. In _International Conference on 3D Vision (3DV)_. IEEE, 98–109.
  * Alldieck et al. (2018b) Thiemo Alldieck, Marcus Magnor, Weipeng Xu, Christian Theobalt, and Gerard Pons-Moll. 2018b.  Video based reconstruction of 3d people models. In _Proc. of Computer Vision and Pattern Recognition (CVPR)_. 8387–8397.
  * Alldieck et al. (2019) Thiemo Alldieck, Gerard Pons-Moll, Christian Theobalt, and Marcus Magnor. 2019.  Tex2shape: Detailed full human body geometry from a single image. In _Proc. of Int’l Conf. on Computer Vision (ICCV)_. 2293–2303.
  * Choi et al. (2021) Seunghwan Choi, Sunghyun Park, Minsoo Lee, and Jaegul Choo. 2021\.  Viton-hd: High-resolution virtual try-on via misalignment-aware normalization. In _Proc. of Computer Vision and Pattern Recognition (CVPR)_. 14131–14140.
  * Gabeur et al. (2019) Valentin Gabeur, Jean-Sébastien Franco, Xavier Martin, Cordelia Schmid, and Gregory Rogez. 2019\.  Moulding humans: Non-parametric 3d human shape estimation from single images. In _Proc. of Int’l Conf. on Computer Vision (ICCV)_. 2232–2241.
  * Goodfellow et al. (2014) Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. 2014\.  Generative adversarial nets.  _Proc. of Advances in Neural Information Processing Systems_ 27 (2014).
  * Groueix et al. (1802) T Groueix, M Fisher, VG Kim, BC Russell, and M Aubry. 1802.  AtlasNet: A Papier-Mâché Approach to Learning 3D Surface Generation. arXiv 2018.  _arXiv preprint arXiv:1802.05384_ (1802).
  * Güler et al. (2018) Rıza Alp Güler, Natalia Neverova, and Iasonas Kokkinos. 2018.  Densepose: Dense human pose estimation in the wild. In _Proc. of Computer Vision and Pattern Recognition (CVPR)_. 7297–7306.
  * He et al. (2020) Tong He, John Collomosse, Hailin Jin, and Stefano Soatto. 2020\.  Geo-pifu: Geometry and pixel aligned implicit functions for single-view human reconstruction.  _Proc. of Advances in Neural Information Processing Systems_ 33 (2020), 9276–9287.
  * Huang and Belongie (2017) Xun Huang and Serge Belongie. 2017.  Arbitrary style transfer in real-time with adaptive instance normalization. In _Proc. of Int’l Conf. on Computer Vision (ICCV)_. 1501–1510.
  * Huang et al. (2017) Yinghao Huang, Federica Bogo, Christoph Lassner, Angjoo Kanazawa, Peter V Gehler, Javier Romero, Ijaz Akhter, and Michael J Black. 2017\.  Towards accurate marker-less human shape and pose estimation over time. In _International Conference on 3D Vision (3DV)_. IEEE, 421–430.
  * Isola et al. (2017) Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, and Alexei A Efros. 2017.  Image-to-image translation with conditional adversarial networks. In _Proc. of Computer Vision and Pattern Recognition (CVPR)_. 1125–1134.
  * Jackson et al. (2018) Aaron S Jackson, Chris Manafas, and Georgios Tzimiropoulos. 2018\.  3d human body reconstruction from a single image via volumetric regression. In _Proceedings of the European Conference on Computer Vision (ECCV) Workshops_. 0–0.
  * Jaderberg et al. (2015) Max Jaderberg, Karen Simonyan, Andrew Zisserman, et al. 2015\.  Spatial transformer networks.  _Proc. of Advances in Neural Information Processing Systems_ 28 (2015).
  * Karras et al. (2019) Tero Karras, Samuli Laine, and Timo Aila. 2019.  A style-based generator architecture for generative adversarial networks. In _Proc. of Computer Vision and Pattern Recognition (CVPR)_. 4401–4410.
  * Karras et al. (2020) Tero Karras, Samuli Laine, Miika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. 2020\.  Analyzing and improving the image quality of stylegan. In _Proc. of Computer Vision and Pattern Recognition (CVPR)_. 8110–8119.
  * Kocabas et al. (2020) Muhammed Kocabas, Nikos Athanasiou, and Michael J Black. 2020\.  Vibe: Video inference for human body pose and shape estimation. In _Proc. of Computer Vision and Pattern Recognition (CVPR)_. 5253–5263.
  * Kolotouros et al. (2019b) Nikos Kolotouros, Georgios Pavlakos, Michael J Black, and Kostas Daniilidis. 2019b.  Learning to reconstruct 3D human pose and shape via model-fitting in the loop. In _Proc. of Int’l Conf. on Computer Vision (ICCV)_. 2252–2261.
  * Kolotouros et al. (2019a) Nikos Kolotouros, Georgios Pavlakos, and Kostas Daniilidis. 2019a.  Convolutional mesh regression for single-image human shape reconstruction. In _Proc. of Computer Vision and Pattern Recognition (CVPR)_. 4501–4510.
  * Lahner et al. (2018) Zorah Lahner, Daniel Cremers, and Tony Tung. 2018\.  Deepwrinkles: Accurate and realistic clothing modeling. In _Proc. of European Conf. on Computer Vision (ECCV)_. 667–684.
  * Lazova et al. (2019) Verica Lazova, Eldar Insafutdinov, and Gerard Pons-Moll. 2019\.  360-degree textures of people in clothing from a single image. In _International Conference on 3D Vision (3DV)_. IEEE, 643–653.
  * Ledig et al. (2017) Christian Ledig, Lucas Theis, Ferenc Huszár, Jose Caballero, Andrew Cunningham, Alejandro Acosta, Andrew Aitken, Alykhan Tejani, Johannes Totz, Zehan Wang, et al. 2017\.  Photo-realistic single image super-resolution using a generative adversarial network. In _Proc. of Computer Vision and Pattern Recognition (CVPR)_. 4681–4690.
  * Lin et al. (2018) Chen-Hsuan Lin, Chen Kong, and Simon Lucey. 2018.  Learning efficient point cloud generation for dense 3d object reconstruction. In _proceedings of the AAAI Conference on Artificial Intelligence_ , Vol. 32.
  * Lin et al. (2017) Tsung-Yi Lin, Piotr Dollár, Ross Girshick, Kaiming He, Bharath Hariharan, and Serge Belongie. 2017\.  Feature pyramid networks for object detection. In _Proc. of Computer Vision and Pattern Recognition (CVPR)_. 2117–2125.
  * Liu et al. (2016) Ziwei Liu, Ping Luo, Shi Qiu, Xiaogang Wang, and Xiaoou Tang. 2016.  Deepfashion: Powering robust clothes recognition and retrieval with rich annotations. In _Proc. of Computer Vision and Pattern Recognition (CVPR)_. 1096–1104.
  * Loper et al. (2015) Matthew Loper, Naureen Mahmood, Javier Romero, Gerard Pons-Moll, and Michael J Black. 2015\.  SMPL: A skinned multi-person linear model.  _ACM Trans. on Graph. (ToG)_ 34, 6 (2015), 1–16.
  * Maturana and Scherer (2015) Daniel Maturana and Sebastian Scherer. 2015.  Voxnet: A 3d convolutional neural network for real-time object recognition. In _IEEE/RSJ Int’l Conf. on Intelligent Robots and Systems (IROS)_. IEEE, 922–928.
  * Mescheder et al. (2018) Lars Mescheder, Andreas Geiger, and Sebastian Nowozin. 2018\.  Which training methods for GANs do actually converge?. In _International conference on machine learning_. PMLR, 3481–3490.
  * Mescheder et al. (2019) Lars Mescheder, Michael Oechsle, Michael Niemeyer, Sebastian Nowozin, and Andreas Geiger. 2019\.  Occupancy networks: Learning 3d reconstruction in function space. In _Proc. of Computer Vision and Pattern Recognition (CVPR)_. 4460–4470.
  * Mildenhall et al. (2020) Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng. 2020\.  Nerf: Representing scenes as neural radiance fields for view synthesis. In _Proc. of European Conf. on Computer Vision (ECCV)_. Springer, 405–421.
  * Natsume et al. (2019) Ryota Natsume, Shunsuke Saito, Zeng Huang, Weikai Chen, Chongyang Ma, Hao Li, and Shigeo Morishima. 2019.  Siclope: Silhouette-based clothed people. In _Proc. of Computer Vision and Pattern Recognition (CVPR)_. 4480–4490.
  * Newell et al. (2016) Alejandro Newell, Kaiyu Yang, and Jia Deng. 2016.  Stacked hourglass networks for human pose estimation. In _Proc. of European Conf. on Computer Vision (ECCV)_. Springer, 483–499.
  * Park et al. (2019a) Jeong Joon Park, Peter Florence, Julian Straub, Richard Newcombe, and Steven Lovegrove. 2019a.  Deepsdf: Learning continuous signed distance functions for shape representation. In _Proc. of Computer Vision and Pattern Recognition (CVPR)_. 165–174.
  * Park et al. (2019b) Taesung Park, Ming-Yu Liu, Ting-Chun Wang, and Jun-Yan Zhu. 2019b.  Semantic image synthesis with spatially-adaptive normalization. In _Proc. of Computer Vision and Pattern Recognition (CVPR)_. 2337–2346.
  * Pavlakos et al. (2019) Georgios Pavlakos, Vasileios Choutas, Nima Ghorbani, Timo Bolkart, Ahmed AA Osman, Dimitrios Tzionas, and Michael J Black. 2019.  Expressive body capture: 3d hands, face, and body from a single image. In _Proc. of Computer Vision and Pattern Recognition (CVPR)_. 10975–10985.
  * Qi et al. (2017) Charles R Qi, Hao Su, Kaichun Mo, and Leonidas J Guibas. 2017\.  Pointnet: Deep learning on point sets for 3d classification and segmentation. In _Proc. of Computer Vision and Pattern Recognition (CVPR)_. 652–660.
  * Ren et al. (2020) Yurui Ren, Xiaoming Yu, Junming Chen, Thomas H Li, and Ge Li. 2020.  Deep image spatial transformation for person image generation. In _Proc. of Computer Vision and Pattern Recognition (CVPR)_. 7690–7699.
  * Saito et al. (2019) Shunsuke Saito, Zeng Huang, Ryota Natsume, Shigeo Morishima, Angjoo Kanazawa, and Hao Li. 2019\.  Pifu: Pixel-aligned implicit function for high-resolution clothed human digitization. In _Proc. of Int’l Conf. on Computer Vision (ICCV)_. 2304–2314.
  * Saito et al. (2020) Shunsuke Saito, Tomas Simon, Jason Saragih, and Hanbyul Joo. 2020.  Pifuhd: Multi-level pixel-aligned implicit function for high-resolution 3d human digitization. In _Proc. of Computer Vision and Pattern Recognition (CVPR)_. 84–93.
  * Sarkar et al. (2021) Kripasindhu Sarkar, Vladislav Golyanik, Lingjie Liu, and Christian Theobalt. 2021.  Style and pose control for image synthesis of humans from a single monocular view.  _arXiv preprint arXiv:2102.11263_ (2021).
  * Varol et al. (2018) Gul Varol, Duygu Ceylan, Bryan Russell, Jimei Yang, Ersin Yumer, Ivan Laptev, and Cordelia Schmid. 2018.  Bodynet: Volumetric inference of 3d human body shapes. In _Proc. of European Conf. on Computer Vision (ECCV)_. 20–36.
  * Wang et al. (2018) Nanyang Wang, Yinda Zhang, Zhuwen Li, Yanwei Fu, Wei Liu, and Yu-Gang Jiang. 2018\.  Pixel2mesh: Generating 3d mesh models from single rgb images. In _Proc. of European Conf. on Computer Vision (ECCV)_. 52–67.
  * Xu and Loy (2021) Xiangyu Xu and Chen Change Loy. 2021.  3D human texture estimation from a single image with transformers. In _Proc. of Int’l Conf. on Computer Vision (ICCV)_. 13849–13858.
  * Xu et al. (2021) Xudong Xu, Xingang Pan, Dahua Lin, and Bo Dai. 2021\.  Generative Occupancy Fields for 3D Surface-Aware Image Synthesis.  _Proc. of Advances in Neural Information Processing Systems_ 34 (2021).
  * Yoon et al. (2021) Jae Shin Yoon, Lingjie Liu, Vladislav Golyanik, Kripasindhu Sarkar, Hyun Soo Park, and Christian Theobalt. 2021.  Pose-guided human animation from a single image in the wild. In _Proc. of Computer Vision and Pattern Recognition (CVPR)_. 15039–15048.
  * Yu et al. (2019) Ruiyun Yu, Xiaoqi Wang, and Xiaohui Xie. 2019.  Vtnfp: An image-based virtual try-on network with body and clothing feature preservation. In _Proc. of Int’l Conf. on Computer Vision (ICCV)_. 10511–10520.
  * Yu et al. (2018) Tao Yu, Zerong Zheng, Kaiwen Guo, Jianhui Zhao, Qionghai Dai, Hao Li, Gerard Pons-Moll, and Yebin Liu. 2018\.  Doublefusion: Real-time capture of human performances with inner body shapes from a single depth sensor. In _Proc. of Computer Vision and Pattern Recognition (CVPR)_. 7287–7296.
  * Zheng et al. (2021a) Yang Zheng, Ruizhi Shao, Yuxiang Zhang, Tao Yu, Zerong Zheng, Qionghai Dai, and Yebin Liu. 2021a.  Deepmulticap: Performance capture of multiple characters using sparse multiview cameras. In _Proc. of Int’l Conf. on Computer Vision (ICCV)_. 6239–6249.
  * Zheng et al. (2018) Zerong Zheng, Tao Yu, Hao Li, Kaiwen Guo, Qionghai Dai, Lu Fang, and Yebin Liu. 2018.  Hybridfusion: Real-time performance capture using a single depth sensor and sparse imus. In _Proc. of European Conf. on Computer Vision (ECCV)_. 384–400.
  * Zheng et al. (2021b) Zerong Zheng, Tao Yu, Yebin Liu, and Qionghai Dai. 2021b.  Pamir: Parametric model-conditioned implicit representation for image-based human reconstruction.  _IEEE Trans. Pattern Anal. Mach. Intell. (TPAMI)_ (2021).
  * Zheng et al. (2019) Zerong Zheng, Tao Yu, Yixuan Wei, Qionghai Dai, and Yebin Liu. 2019.  Deephuman: 3d human reconstruction from a single image. In _Proc. of Int’l Conf. on Computer Vision (ICCV)_. 7739–7749.
  * Zhou et al. (2019) Kaiyang Zhou, Yongxin Yang, Andrea Cavallaro, and Tao Xiang. 2019.  Omni-scale feature learning for person re-identification. In _Proc. of Int’l Conf. on Computer Vision (ICCV)_. 3702–3712.
  * Zhu et al. (2019b) Hao Zhu, Xinxin Zuo, Sen Wang, Xun Cao, and Ruigang Yang. 2019b.  Detailed human shape estimation from a single image by hierarchical mesh deformation. In _Proc. of Computer Vision and Pattern Recognition (CVPR)_. 4491–4500.
  * Zhu et al. (2019a) Zhen Zhu, Tengteng Huang, Baoguang Shi, Miao Yu, Bofei Wang, and Xiang Bai. 2019a.  Progressive pose attention transfer for person image generation. In _Proc. of Computer Vision and Pattern Recognition (CVPR)_. 2347–2356.

##  Appendix A Supplementary: Overview

Here, we present implementation details and additional experiments which
support the validity of our proposed method. We provide details of our network
architectures in Sec. B. We also show additional experiments including the
$\gamma$ visualization and target view expansion in Sec. C. Lastly, we present
the failure cases and the additional results of our method in Sec. D.

##  Appendix B Network Architecture Detail

###  B.1. Occupancy and Texture Network

The occupancy and texture network,i.e. the networks of the initial stage and
the fusion stage, consist of feature extractors and a Multi-Layer Perceptrons
(MLP) network. For the image feature extractor, we follow PIFu (Saito et al.,
2019) by adapting the stacked hourglass network (Newell et al., 2016) that
outputs a 256-channel feature map with a resolution of 128$\times$128\. We
also adopt the same 3D volume encoder in PaMIR (Zheng et al., 2021b) that
outputs a 32-channel volume feature with a resolution of
32$\times$32$\times$32\. As shown in Fig. 3 in the main paper, after
extracting features from the input image and the SMPL model, the MLP-based
network predicts the occupancy probability, color, and $\gamma$ conditioned on
these extracted features. We concatenate the query coordinate to the input
features after applying the positional encoding (Mildenhall et al., 2020). The
numbers of intermediate channels are (1024,512,256,128), and the input feature
is concatenated to each intermediate channel. The final output has 5 channels
for the network in the initial stage and 7 channels for the fusion network to
predict the RGB, occupancy, and $\gamma$ values.

###  B.2. Refinement Network

As illustrated in Fig. H, we present the fine details of the refinement
network. We utilize the StyleGAN2 architecture (Karras et al., 2019) with a
spatial modulation (Park et al., 2019b; Albahar et al., 2021) that takes the
coarse target-view image ${\mathbf{I}}^{\prime}_{trg}$ as an input and the
warped source image feature as a modulation condition. The source image
${\mathbf{I}}$ is encoded to multi-scale features ${\mathbf{F}}^{src}_{i}$ by
the source image encoder that consists of several residual blocks. For
simplification, the main paper explains that the warped source feature
${\mathbf{F}}^{trg}_{i}$, i.e. the result of warping the source feature
${\mathbf{F}}^{src}_{i}$ with the warping function $\mathbf{f}$, is passed to
the style block. In the implementation, the multi-scale warped source features
${\mathbf{F}}^{trg}_{i}$ are processed with a feature pyramid network (Lin et
al., 2017). As illustrated in Fig. H, the warped source features are
concatenated to the resized foreground mask, and then passed to several
convolution and up-sampling layers. The outputs of the feature pyramid network
are utilized as the modulation conditions of the style blocks. The ground-
truth foreground mask of a target image is used in the training, while the
mask is rendered in the inference stage by volume-rendering the occupancy
values from the target view and thresholding them with the value of 0.5.

We designed a discriminator that distinguishes real and fake images
conditioned on the coarse target-view image by adopting the architecture of
StyleGAN2 (Karras et al., 2020). The architecture details are presented in
Table D.

---
Figure H. The detailed network architecture of our refinement network. The
multi-scale outputs of the source image encoder are warped to the target image
space with the warping function. Utilizing a feature pyramid network (Lin et
al., 2017), the warped source features are fed into the style blocks as the
spatial-modulation conditions.  Table D. The configuration of discriminator
$D_{\phi}$.

Type | Activation | Input Shape | Output Shape
---|---|---|---
Input Image and Condition | - | 6$\times$512$\times$512 | 6$\times$512$\times$512
Conv (1$\times$1) | LeakyReLU (0.2) | 6$\times$512$\times$512 | 64$\times$512$\times$512
Conv (3$\times$3) | LeakyReLU (0.2) | 64$\times$512$\times$512 | 64$\times$512$\times$512
Conv (3$\times$3, Stride 2) | LeakyReLU (0.2) | 64$\times$512$\times$512 | 128$\times$256$\times$256
Residual Connection | - | 128$\times$256$\times$256 | 128$\times$256$\times$256
Conv (3$\times$3) | LeakyReLU (0.2) | 128$\times$256$\times$256 | 128$\times$256$\times$256
Conv (3$\times$3, Stride 2) | LeakyReLU (0.2) | 128$\times$256$\times$256 | 256$\times$128$\times$128
Residual Connection | - | 256$\times$128$\times$128 | 256$\times$128$\times$128
Conv (3$\times$3) | LeakyReLU (0.2) | 256$\times$128$\times$128 | 256$\times$128$\times$128
Conv (3$\times$3, Stride 2) | LeakyReLU (0.2) | 256$\times$128$\times$128 | 512$\times$64$\times$64
Residual Connection | - | 512$\times$64$\times$64 | 512$\times$64$\times$64
Conv (3$\times$3) | LeakyReLU (0.2) | 512$\times$64$\times$64 | 512$\times$64$\times$64
Conv (3$\times$3, Stride 2) | LeakyReLU (0.2) | 512$\times$64$\times$64 | 512$\times$32$\times$32
Residual Connection | - | 512$\times$32$\times$32 | 512$\times$32$\times$32
Conv (3$\times$3) | LeakyReLU (0.2) | 512$\times$32$\times$32 | 512$\times$32$\times$32
Conv (3$\times$3, Stride 2) | LeakyReLU (0.2) | 512$\times$32$\times$32 | 512$\times$16$\times$16
Residual Connection | - | 512$\times$16$\times$16 | 512$\times$16$\times$16
Conv (3$\times$3) | LeakyReLU (0.2) | 512$\times$16$\times$16 | 512$\times$16$\times$16
Conv (3$\times$3, Stride 2) | LeakyReLU (0.2) | 512$\times$16$\times$16 | 512$\times$8$\times$8
Residual Connection | - | 512$\times$8$\times$8 | 512$\times$8$\times$8
Conv (3$\times$3) | LeakyReLU (0.2) | 512$\times$8$\times$8 | 512$\times$8$\times$8
Conv (3$\times$3, Stride 2) | LeakyReLU (0.2) | 512$\times$8$\times$8 | 512$\times$4$\times$4
Residual Connection | - | 512$\times$4$\times$4 | 512$\times$4$\times$4
Minibatch stddev | - | 512$\times$4$\times$4 | 513$\times$4$\times$4
Conv2d (3$\times$3) | LeakyReLU (0.2) | 513$\times$4$\times$4 | 512$\times$4$\times$4
Conv2d (3$\times$3) | LeakyReLU (0.2) | 513$\times$4$\times$4 | 512$\times$4$\times$4
Reshape | - | 512$\times$4$\times$4 | 8192
Linear | LeakyReLU (0.2) | 8192 | 512
Linear | - | 512 | 1

---
Figure I. The $\gamma$ visualization. Given the source input image and the
refined target-view image in the first and second columns, the $\gamma$ values
predicted from the networks are visualized. For each example,the results of
the initial stage are presented in the first row and the results of the fusion
stage in the second row. We visualize the values of $\gamma$ for the
prediction, frontside, and backside with red, green, and blue colors,
respectively.

##  Appendix C Additional Experiments

In this section, we show additional experiments including the $\gamma$
visualization and the target view expansion.

###  C.1. $\gamma$ Visualization

We visualize the $\gamma$ value that is used to blend the predicted color
value with the ones sampled from the source image and the backside-view image.
As shown in Fig. I, given a source image in the first column, the generated
backside-view image is shown in the second column, and the meshes captured
from the front, side, and back views are presented in the following columns.
For each example, the results of the initial stage are presented in the first
row and the results of the fusion stage in the second row. As described in the
main paper, the final $\gamma$-composited color value is calculated as Eq.
(13). We visualize the $\gamma_{1}$ in green, $\gamma_{2}$ in blue, and
$\gamma_{3}$ in red. Note that $\gamma_{2}$ is 0 and $\gamma_{3}$ is
$1-\gamma_{1}$ in the initial stage. We confirm that the $\gamma$ is
appropriately predicted, as the $\gamma_{1}$ and $\gamma_{2}$ are close to 1
at the locations visible from the source view and target view, respectively.

###  C.2. Target View Expansion

The proposed model is conditioned on two views including the source and the
refined backside-view image. In this section, we demonstrate that our method
is expandable to a variant with more target views. We conducted the experiment
of a 4-view variant, i.e. the model that utilizes a source-view image and
three additional target-view images including the back-, left-, and right-side
views. An identical refine-and-fuse approach was applied to these additional
target-view images. As shown in Fig. J, we report the qualitative results of
our 4-view variant. Compared to the original version that generates the
backside-view image only, the 4-view variant directly alleviates the ambiguity
of the side view by additionally generating side-view images. However,
utilizing more than four views harms the computation efficiency without a
significant improvement in the perceptual quality. This is because the total
area of observable region barely increases when using more than 4 views. We
empirically found that using 2-4 views improves the perceptual quality without
degrading the computation efficiency; thus, the 2-view version, i.e. the
source image with the refined backside-view image, is utilized as our default
setting.

---
Figure J. Qualitative results of the 4-view variant. Given a source image in
the first column, refined images from the right, back, and left views are
presented in the next column, and the textured meshes captured from the front,
side, and back view are presented.
---
Figure K. Failure cases. We present the reconstructed meshes from our method
on challenging poses. (a) denotes the outputs when utilizing the predicted
SMPL, and (b) denotes those with the ground-truth SMPL.

##  Appendix D Additional Results

###  D.1. Failure cases

This section presents the failure cases of our proposed method. As described
in Sec. 5, despite the fact that the parametric body model provides strong
prior knowledge of the human body, a performance bottleneck occurs based on
the accuracy of the predicted SMPL. In practice, SMPL prediction methods
(Kolotouros et al., 2019a, b) show a large error of accuracy for cases of
extreme poses, such as sitting, since such poses are rarely seen in their
training dataset. The imprecise SMPL prediction inevitably degrades the
performance of our method resulting in failure cases, as shown in Fig. K. Our
method fails to reconstruct accurate meshes for sitting poses since the input
SMPL is not aligned to the source image(Fig. K(a)). However, our method shows
an improved quality of the mesh when the accurate SMPL input is used (Fig.
K(b)).

###  D.2. Additional Examples

Here, we include additional visual results to show the mesh quality of our
method. Additional mesh results with and without texture on the THUman2.0,
Twindom, and DeepFashion datasets are presented in Fig. M. The meshes are
captured from the front, side, and back views. Given the source image in the
first column, our method successfully reconstructs the textured meshes. Note
that our method generates high-frequency details such as hair, cloth wrinkles,
and faces, even from an unobserved view. Also, the patterns of clothes are
generated in a source-consistent manner due to our spatial modulation with
feature warping.

---
Figure L. Qualitative results of our proposed method. Given the source image
in the first column, meshes without texture are presented in columns 1-3, and
meshes with texture are presented in columns 4-6. We capture the output meshes
from the front, side, and back views (continued.).
---
Figure M. Qualitative results of our proposed method. Given the source image
in the first column, meshes without texture are presented in columns 1-3, and
meshes with texture are presented in columns 4-6. We capture the output meshes
from the front, side, and back views.