# Human 3Diffusion: Realistic Avatar Creation
via Explicit 3D Consistent Diffusion Models

Yuxuan Xue1,2 Xianghui Xie1,2,3 Riccardo Marin1,2 Gerard Pons-Moll1,2,3
1University of Tübingen 2 Tübingen AI Center
3Max Planck Institute for Informatics, Saarland Informatics Campus
https://yuxuan-xue.com/human-3diffusion/

###### Abstract

Creating realistic avatars from a single RGB image is an attractive yet
challenging problem. Due to its ill-posed nature, recent works leverage
powerful prior from 2D diffusion models pretrained on large datasets. Although
2D diffusion models demonstrate strong generalization capability, they cannot
provide multi-view shape priors with guaranteed 3D consistency. We propose
Human 3Diffusion: Realistic Avatar Creation via Explicit 3D Consistent
Diffusion. Our key insight is that 2D multi-view diffusion and 3D
reconstruction models provide complementary information for each other, and by
coupling them in a tight manner, we can fully leverage the potential of both
models. We introduce a novel image-conditioned generative 3D Gaussian Splats
reconstruction model that leverages the priors from 2D multi-view diffusion
models, and provides an explicit 3D representation, which further guides the
2D reverse sampling process to have better 3D consistency. Experiments show
that our proposed framework outperforms state-of-the-art methods and enables
the creation of realistic avatars from a single RGB image, achieving high-
fidelity in both geometry and appearance. Extensive ablations also validate
the efficacy of our design, (1) multi-view 2D priors conditioning in
generative 3D reconstruction and (2) consistency refinement of sampling
trajectory via the explicit 3D representation. Our code and models will be
released here.

---
Figure 1: Given a single image of a person (top), our method Human 3Diffusion
creates 3D Gaussian Splats of realistic avatars with high-fidelity geometry
and texture.

###  1 Introduction

Realistic human avatar creation is crucial for various applications such as
AR/VR, as well as the movie and gaming industry. Methods for creating a 3D
avatar from a single RGB image are especially important to scale up avatar
creation and make it more consumer-friendly compared to traditional studio-
based capture methods. This task is, however, very challenging due to the vast
diversity of human bodies and poses, further complicated by the wide variety
of clothing and accessories. These challenges are exacerbated by the lack of
large-scale 3D human data and ambiguities inherent in a monocular 2D view
setting.

Recent image-to-3D approaches can be categorized into reconstruction-based and
multi-view diffusion-based methods. Reconstruction-based approaches directly
predict a 3D representation that can be rendered from any viewpoint. Due to
the explicit 3D representation, these methods produce an arbitrary number of
consistent viewpoint renderings. They obtain the 3D reconstruction either
based on common template [21, 85, 86, 106] which utilize the SMPL [44] body
model as the shape prior, or a flexible implicit function to represent loose
clothing [5, 54, 55]. These methods, either template-based [21, 85, 86, 106]
or template-free [5, 24, 54, 55, 72, 108], are typically deterministic which
produce blurry textures and geometry in the occluded regions. More
importantly, they are trained on small-scale datasets due to the limited
amount of high-quality 3D data, which further restricts their ability to
generalize to diverse shapes and textures.

Multi-view diffusion methods [40, 58, 75] distill the inherent 3D structure
present in 2D diffusion models [53]. Typically, they fine-tune a large-scale
2D foundation model [23, 62] on a large 3D dataset of objects [12, 80, 97], to
produce a _fixed_ number of viewpoints. However, since these models diffuse
images purely in 2D without explicit 3D constraints or representation, the
resulting multi-views often lack 3D consistency [52, 39], which restricts
downstream applications [65].

To address these challenges, we propose 3Diffusion: realistic avatar creation
via 3D consistent Diffusion models. We design our method based on two key
insights: 1) 2D multi-view diffusion models provide large-scale shape priors
that can help 3D reconstruction; 2) A reconstructed 3D representation ensures
3D consistency across multi-views in 2D diffusion. Specifically, we propose a
novel diffusion method, which bridges 3D Gaussian Splatting (3D-GS) [34]
generation with a 2D multi-view diffusion model. At every iteration, multi-
view images are denoised and reconstructed to 3D-GS to be re-rendered to
continue the diffusion process. This 3D lifting during iterative sampling
ensures the 3D consistency of the 2D diffusion model while leveraging a large-
scale foundation model trained on billions of images. Our framework elegantly
combines reconstruction methods with multi-view diffusion models. In summary,
our contributions are:

  * •

We propose a novel image-conditioned 3D-GS generation model for 3D
reconstruction that bridges large-scale priors from 2D multi-view diffusion
models and the efficient and explicit 3D-GS representation.

  * •

A sophisticated diffusion process that incorporates reconstructed 3D-GS to
improve the 3D consistency of 2D diffusion models by refining the reverse
sampling trajectory.

  * •

Our proposed formulation enables us to jointly train 2D diffusion and our 3D
model on $\sim 6000$ high-quality human scans and our method shows superior
performance and generalization capability than prior works. Our code and
pretrained models will be publicly released on our project page.

###  2 Related Work

##### Image to 3D.

Creating realistic human avatar from consumer grade sensors [31, 94, 71, 90,
91, 92] is essential for downstream tasks such as human behaviour
understanding [8, 49, 82, 84, 83] and gaming application [36, 42, 18, 103,
104]. Researchers have explored avatar creation from monocular RGB [29, 78],
Depth [15, 91] video or single image [54, 55, 57, 85, 86]. Avatar from single
image is particularly interesting and existing methods can be roughly
categorized as template-based [21, 85, 86, 106] and template-free [54, 55, 57,
93]. Despite the impressive performance, template-based approaches rely on the
naked body model [44, 48] and fail to reason extremely loose clothing, while
template-free methods produce blurry back side textures. Instead of SMPL shape
prior, our method is template-free and leverages strong 2D image priors to
create high-quality avatars.
Orthogonal to humans, object reconstruction methods typically adopt template-
free paradigms and early works [6, 69, 79, 81, 105] focus mainly on geometry.
With the advance of 2D diffusion models [53] and efficient 3D representation
[11], recent works can reconstruct 3D objects with detailed textures [24, 39,
43, 58, 65, 72, 87, 88, 108]. One popular paradigm is first using strong 2D
models [40, 59, 75] to produce multi-view images and then train another model
to reconstruct 3D from multi-view images [39, 38, 43, 65, 89]. In practice,
their performance is limited by the accuracy of the multi-view images
generated by 2D diffusion modes. Our method tightly couples 2D and 3D models
and yields better performance by guiding 2D sampling with 3D reconstruction.

##### Shape Prior from 2D Diffusion Model.

Being trained on billions of images [56], 2D image diffusion models [53] have
been shown to have 3D awareness and some works tried to use score distillation
sampling [51, 77] to distil 3D knowledge of 2D models [37, 46, 107]. Other
works propose to further enhance the 3D reasoning ability by fine-tuning the
model on large-scale datasets [12, 80, 97] to generate multi-view images [32,
35, 40, 41, 58, 59, 66, 74, 75]. Dense self-attention [73, 75], depth-aware
attention [26] or epipolar attention [28, 64] are introduced to enhance the 3D
consistency of multi-views. However, these methods do not have explicit 3D
while our method incorporates explicit 3D consistency into the reverse
sampling process and obtains better results.

###  3 Preliminaries

##### Denoising Diffusion Probabilistic Models.

DDPM [23] is a generative model which learns a data distribution by
iteratively adding (forward process) and removing (reverse process) the noise.
Formally, the forward process iteratively adds noise to a sample
${\mathbf{x}}_{0}$ drawn from a distribution $p_{\text{data}}(\mathbf{x})$:

${\mathbf{x}}_{t}\sim\mathcal{N}({\mathbf{x}}_{t};\sqrt{\alpha_{t}}{\mathbf{x}}_{t-1},(1-\alpha_{t})\mathbf{I}):=\sqrt{\bar{\alpha}_{t}}{\mathbf{x}}_{0}+\sqrt{1-\bar{\alpha}_{t}}\mathbf{\epsilon},\text{
where }\mathbf{\epsilon}\sim\mathcal{N}(0,\mathbf{I}),$ (1)

where $\alpha_{t},\bar{\alpha}_{t}$ schedules the amount of noise added at
each step $t$ [23]. To sample data from the learned distribution, the reverse
process starts from ${\mathbf{x}}_{T}\sim\mathcal{N}(0,\mathbf{I})$ and
iteratively denoises it until $t=0$:

${\mathbf{x}}_{t-1}\sim\mathcal{N}({\mathbf{x}}_{t-1};\mathbf{\mu}_{\theta}({\mathbf{x}}_{t},t),\tilde{\beta}_{t-1}\mathbf{I}),\text{
where
}\tilde{\beta}_{t-1}=\frac{1-\bar{\alpha}_{t-1}}{1-\bar{\alpha}_{t}}(1-\alpha_{t})$
(2)

A network parametrized by $\theta$ is trained to estimate the posterior mean
$\mathbf{\mu}_{\theta}$ at each step $t$. One can also model conditional
distribution with DDPM by adding the condition to the network input [14, 22].

##### 2D Multi-View Diffusion Models.

Many recent works [40, 41, 43, 58, 66, 75] propose to leverage strong 2D image
diffusion prior [53] pre-trained on billions images [56] to generate multi-
view images from a single image. Among them, ImageDream [75] demonstrated a
superior generalization capability to unseen objects [65]. Given a single
condition image $\mathbf{x}^{\text{c}}$ and an optional text description $y$,
ImageDream generate 4 orthogonal target views $\mathbf{x}^{\text{tgt}}$ with a
model $\mathbf{\epsilon}_{\theta}$, which is trained to estimate the noise
added at each step $t$. With the estimated noise $\mathbf{\epsilon}_{\theta}$,
one can compute the "clear" target views $\tilde{\mathbf{x}}^{\text{tgt}}_{0}$
with close-form solution in Eq. 1:

$\displaystyle\tilde{\mathbf{x}}_{0}^{\text{tgt}}$
$\displaystyle=\frac{1}{\sqrt{\bar{\alpha}_{t}}}(\mathbf{x}^{\text{tgt}}_{t}-\sqrt{1-\bar{\alpha}_{t}}\bm{\epsilon}_{\theta}(\mathbf{x}^{\text{tgt}}_{t},\mathbf{x}^{\text{c}},y,t)).$
(3)

This _one-step_ estimation of $\tilde{\mathbf{x}}_{0}^{\text{tgt}}$ can be
noisy, especially when $t$ is large and $\mathbf{x}_{t}^{\text{tgt}}$ is
extremely noisy. Thus, the iterative sampling of $\mathbf{x}_{t}^{\text{tgt}}$
is required until $t=0$. To sample next step $\mathbf{x}_{t-1}^{\text{tgt}}$,
standard DDPM [23] computes the posterior mean $\mu_{\theta}$ from current
$\mathbf{x}_{t}^{\text{tgt}}$ and estimated
$\tilde{\mathbf{x}}_{0}^{\text{tgt}}$ at step $t$ with:

$\mu_{\theta}({\mathbf{x}}_{t}^{\text{tgt}},t):=\mu_{t-1}({\mathbf{x}}_{t}^{\text{tgt}},\tilde{{\mathbf{x}}}_{0}^{\text{tgt}})=\frac{\sqrt{\alpha_{t}}\left(1-\bar{\alpha}_{t-1}\right)}{1-\bar{\alpha}_{t}}{\mathbf{x}}^{\text{tgt}}_{t}+\frac{\sqrt{\bar{\alpha}_{t-1}}\beta_{t}}{1-\bar{\alpha}_{t}}\tilde{{\mathbf{x}}}_{0}^{\text{tgt}},\text{
where }\beta_{t}=1-\alpha_{t}.$ (4)

Afterwards, $\mathbf{x}_{t-1}^{\text{tgt}}$ can be sampled from Gaussian
distribution with mean $\mu_{t-1}$ and variance
$\tilde{\beta}_{t-1}\mathbf{I}$ (Eq. 2) and used as the input for the next
iteration. The reverse sampling is repeated until $t=0$ where 4 clear target
views are generated.

Although multi-view diffusion models [41, 58, 75] generate multiple views
together, the 3D consistency across these views is not guaranteed due to the
lack of an explicit 3D representation. Thus, we propose a novel 3D consistent
diffusion model, which ensures the multi-view consistency at each step of the
reverse process by diffusing 2D images using reconstructed 3D Gaussian Splats
[34].

###  4 3Diffusion

Figure 2: Method Overview. Given a single RGB image (A), we sample a realistic
3D avatar represented as 3D Gaussian Splats (D). At each reverse step, our 3D
generation model $g_{\phi}$ leverages 2D multi-view diffusion prior from
$\epsilon_{\theta}$ which provides a strong shape prior but is not 3D
consistent (B, cf. Sec. 4.1). We then refine the 2D reverse sampling
trajectory with generated 3D renderings that are guaranteed to be 3D
consistent (C, cf. Sec. 4.2). Our tight coupling ensures 3D consistency at
each sampling step and obtains a high-quality 3D avatar (D).

##### Overview.

Given a single RGB image, we aim to create a realistic 3D avatar consistent
with the input. We adopt an image-conditioned 3D generation paradigm due to
inherent ambiguities in the monocular view. We introduce a novel 3D Gaussian
Splatting (3D-GS [34]) generative model that combines shape priors from 2D
multi-view diffusion models with the explicit 3D-GS representation. This
allows us to jointly train our 3D generative model and a 2D multi-view
diffusion model end-to-end and improves the 3D consistency of 2D multi-view
generation at inference time.

In this section, we first introduce our novel generative 3D-GS reconstruction
model in Sec. 4.1. We then describe how we leverage the 3D reconstruction to
generate 3D consistent multi-view results by refining the reverse sampling
trajectory (Sec. 4.2). An overview of our method can be found in Fig. 2.

####  4.1 Generative 3D-GS Reconstruction with Diffusion Priors

Given a context image $\mathbf{x}^{c}$, we use a conditional diffusion model
to learn and sample from a plausible 3D distribution. Previous works
demonstrated that 3D generation can be done implicitly via diffusing rendered
images of a differentiable 3D represetation [7, 33, 68] such as NeRF [47, 95].
In this work, we introduce a novel generative model for 3D Gaussian Splatings
[34], which diffuses rendered images of 3D-GS and enables sampling of 3D-GS at
inference time. Single image to 3D generation is however very challenging, we
hence propose to leverage 2D multi-view diffusion models in a tightly coupled
manner which allows us to train it end-to-end with our novel 3D generative
model.

##### Generative 3D-GS Reconstruction.

In this work, we propose a 3D-GS generative model $g_{\phi}$, which is
conditioned on input context image $\mathbf{x}^{c}$ to perform reconstruction
of 3D Gaussian Splats $\mathcal{G}$. Diffusing directly in the space of
$\mathcal{G}$ parameters requires pre-computing Gaussian Splats from scans,
which is exorbitant. Instead, we diffuse the multi-view renderings of
$\mathcal{G}$ using a differentiable rendering function renderer.
We denote $\mathbf{x}^{\text{tgt}}_{0}$ as the ground truth images at target
views to be diffused and $\mathbf{x}^{\text{novel}}_{0}$ as the additional
novel views for supervision. At training time, we uniformly sample a timestep
$t\sim\mathcal{U}(0,T)$ and add noise to $\mathbf{x}^{\text{tgt}}_{0}$ using
Eq. 1 to obtain noisy target views $\mathbf{x}^{\text{tgt}}_{t}$. Our
generative model $g_{\phi}$ takes $\mathbf{x}^{\text{tgt}}_{t}$, diffusion
timestep $t$, and the conditional image $\mathbf{x}^{c}$ as input, and
estimates 3D Gaussians $\hat{\mathcal{G}}$:

$\hat{\mathcal{G}}=g_{\phi}(\mathbf{x}^{\text{tgt}}_{t},t,\mathbf{x}^{c}),\text{
where
}\mathbf{x}^{\text{tgt}}_{t}=\sqrt{\bar{\alpha}_{t}}\mathbf{x}_{0}^{\text{tgt}}+\sqrt{1-\bar{\alpha}_{t}}\epsilon,\text{
and }\epsilon\sim\mathcal{N}(0,\mathbf{I})$ (5)

We adopt an asymmetric U-Net Transformer proposed by [65] for $g_{\phi}$ to
directly predict 3D-GS parameters from per-pixel features of the last U-Net
layer. To supervise the generative model $g_{\phi}$, we use a differentiable
rendering function
${\color[rgb]{0.6484375,0.36328125,0.44921875}\definecolor[named]{pgfstrokecolor}{rgb}{0.6484375,0.36328125,0.44921875}\texttt{renderer}}:\\{\mathcal{G},\pi^{\text{p}}\\}\mapsto\mathbf{x}^{\text{p}}$
to render images at target views $\pi^{\text{tgt}}$ and additional novel views
$\pi^{\text{novel}}$. Denoting
$\mathbf{x}_{0}:=\\{{\mathbf{x}}^{\text{tgt}}_{0},{\mathbf{x}}^{\text{novel}}_{0}\\}$
as ground truth and
$\hat{\mathbf{x}}_{0}:=\\{\hat{\mathbf{x}}^{\text{tgt}}_{0},\hat{\mathbf{x}}^{\text{novel}}_{0}\\}$
as rendered images, we compute the loss on images and generated 3D-GS:

$\begin{split}\mathcal{L}_{gs}=\lambda_{1}\cdot\mathcal{L}_{\text{MSE}}\big{(}\mathbf{x}_{0},\hat{\mathbf{x}}_{0}\big{)}+\lambda_{2}\cdot\mathcal{L}_{\text{Percep}}\big{(}\mathbf{x}_{0},\hat{\mathbf{x}}_{0}\big{)}+\lambda_{3}\cdot\mathcal{L}_{\text{reg}}(g_{\phi}(\mathbf{x}^{\text{tgt}}_{t},t,\mathbf{x}^{c})),\\\
\text{ where
}\hat{\mathbf{x}}_{0}:=\\{\hat{\mathbf{x}}^{\text{tgt}}_{0},\hat{\mathbf{x}}^{\text{novel}}_{0}\\}={\color[rgb]{0.6484375,0.36328125,0.44921875}\definecolor[named]{pgfstrokecolor}{rgb}{0.6484375,0.36328125,0.44921875}\texttt{renderer}}(g_{\phi}(\mathbf{x}^{\text{tgt}}_{t},t,\mathbf{x}^{c}),\\{\pi^{\text{tgt}},\pi^{\text{novel}}\\}),\end{split}$
(6)

here $\mathcal{L}_{\text{MSE}}$ denotes the Mean Square Error (MSE) and
$\mathcal{L}_{\text{Percep}}$ is the perceptual loss based on VGG-19 [60]. We
also apply $\mathcal{L}_{\text{reg}}$, a geometry regularizer [27, 98] to
stabilize the generation of $\hat{\mathcal{G}}$.

With this, we can train a generative model that diffuses 3D-GS _implicitly_ by
diffusing 2D images $\mathbf{x}^{\text{tgt}}_{t}$. At inference time, we can
generate 3D-GS given the input image by denoising 2D multi-views sampled from
Gaussian distribution. We initialize $\mathbf{x}^{\text{tgt}}_{T}$ from
$\mathcal{N}(0,\mathbf{I})$, and iteratively denoise the rendered images of
predicted $\hat{\mathcal{G}}$ from our model $g_{\phi}$. At each reverse step,
our model $g_{\phi}$ estimates a clean state $\hat{\mathcal{G}}$ and render
target images $\hat{\mathbf{x}}^{\text{tgt}}_{0}$. We then calculate target
images $\mathbf{x}^{\text{tgt}}_{t-1}$ for the next step via Eq. 4 and repeat
the process until $t=0$. For more details, please refer to Sec. A.3

Our generative 3D-GS reconstruction model archives superior performance on in-
distribution human reconstruction yet generalizes poorly to unseen categories
such as general objects (Sec. 5.3 Fig. 5). Our key insight for better
generalization is leveraging strong priors from pretrained 2D multi-view
diffusion models for 3D-GS generation.

##### 3D-GS Generation with 2D Multi-view Diffusion.

Pretrained 2D multi-view diffusion models (MVD) [41, 59, 75] have seen
billions of real images [56] and millions of 3D data [12], which provide
strong prior information and can generalize to unseen objects [65, 87]. Here,
we propose a simple yet elegant idea for incorporating this multi-view prior
into our generative 3D-GS model $g_{\phi}$. We can also leverage generated
3D-GS to guide 2D MVD sampling process which we discuss in Sec. 4.2.
Our key observation is that both 2D MVD and our proposed 3D-GS generative
model are diffusion-based and share the same sampling state
$\mathbf{x}_{t}^{\text{tgt}}$ at timestep $t$. Thus, they are _synchronized_.
This enables us to couple and facilitate information exchange between 2D MVD
$\bm{\epsilon}_{\theta}$ and 3D-GS generative model $g_{\phi}$ at the same
diffusion timestep $t$. To inject the 2D diffusion priors into 3D generation,
we first compute _one-step_ estimation of
$\tilde{\mathbf{x}}^{\text{tgt}}_{0}$ (Eq. 3) using 2D MVD
$\epsilon_{\theta}$, and condition our 3D-GS generative mode $g_{\phi}$
additionally on it. Formally, our 3D-GS generative model enhanced with 2D
multi-view diffusion priors is written as:

$\displaystyle\hat{\mathcal{G}}=g_{\phi}(\mathbf{x}^{\text{tgt}}_{t},t,\mathbf{x}^{c},\tilde{\mathbf{x}}^{\text{tgt}}_{0}),\text{
where
}\tilde{\mathbf{x}}_{0}^{\text{tgt}}=\frac{1}{\sqrt{\bar{\alpha}_{t}}}(\mathbf{x}^{\text{tgt}}_{t}-\sqrt{1-\bar{\alpha}_{t}}\bm{\epsilon}_{\theta}(\mathbf{x}^{\text{tgt}}_{t},\mathbf{x}^{\text{c}},y,t))$
(7)

The visualization of $\tilde{\mathbf{x}}^{\text{tgt}}_{0}$ along the whole
sampling trajectory in Fig. 7 shows that the pretrained 2D diffusion model
$\bm{\epsilon}_{\theta}$ can already provide useful multi-view shape prior
even in large timestep $t=1000$. This is further validated in our experiments
where the additional 2D diffusion prior $\tilde{\mathbf{x}}^{\text{tgt}}_{0}$
leads to better avatar reconstruction (Tab. 4) as well as more robust
generalization to general objects (Fig. 5). By utilizing the timewise
iterative manner of 2D and 3D diffusion models, we can not only leverage 2D
priors for 3D-GS generation but also train both models jointly end to end,
which we discuss next.

##### Joint Training with 2D Model.

We adopt pretrained ImageDream [75] as our 2D multi-view diffusion model
$\epsilon_{\theta}$ and jointly train it with our 3D-GS generative model
$g_{\phi}$. We observe that our joint training is important for coherent 3D
generation, as opposed to prior works that frozen pretrained 2D multi-view
models [65, 72]. We summarize our training algorithm in Algorithm 1. We
combine the loss of 2D diffusion and our 3D-GS generation loss
$\mathcal{L}_{gs}$( Eq. 6):

$\begin{split}\mathcal{L}_{total}=\mathcal{L}_{\text{MSE}}(\bm{\epsilon},\bm{\epsilon}_{\theta})+\mathcal{L}_{gs}\end{split}$
(8)

Once trained, one can sample a plausible 3D-GS avatar $\mathcal{G}$
conditioned on the input image from the learned 3D distributions. However, we
observe that the multi-view diffusion model $\bm{\epsilon}_{\theta}$ can still
output inconsistent multi-views along the sampling trajectory (see Fig. 2). On
the other hand, our 3D generator produces explicit 3D-GS which can be rendered
as 3D consistent multi-views. Our second key idea is to use the 3D consistent
renderings to guide 2D sampling process for more 3D consistent mulit-view
generation. We discuss this in Sec. 4.2.

####  4.2 Guide 2D Multi-view Sampling with Reconstructed 3D-GS

Algorithm 1 Training

1:Dataset of posed multi-view images ${\mathbf{x}}^{\text{tgt}}_{0}$,
$\pi^{\text{tgt}}$, ${\mathbf{x}}^{\text{novel}}_{0}$, $\pi^{\text{novel}}$, a
context image ${\mathbf{x}}^{\text{c}}$, text description $y$

2:Optimized 2D multi-view diffusion model $\epsilon_{\theta}$ and 3D-GS
generative model $g_{\phi}$

3:repeat

4:
$\\{{\mathbf{x}}^{\text{tgt}}_{0},{\mathbf{x}}^{\text{novel}}_{0},{\mathbf{x}}^{\text{c}},y\\}\sim
q(\\{{\mathbf{x}}^{\text{tgt}}_{0},{\mathbf{x}}^{\text{novel}}_{0},{\mathbf{x}}^{\text{c}},y\\})$

5: $t\sim\mathrm{Uniform}(\\{1,\dotsc,T\\})$;
$\bm{\epsilon}\sim\mathcal{N}(\mathbf{0},\mathbf{I})$

6:
$\mathbf{x}^{\text{tgt}}_{t}=\sqrt{\bar{\alpha}_{t}}{\mathbf{x}}_{0}^{\text{tgt}}+\sqrt{1-\bar{\alpha}_{t}}\bm{\epsilon}$

7:
$\tilde{\mathbf{x}}^{\text{tgt}}_{0}=\frac{1}{\sqrt{\bar{\alpha}_{t}}}(\mathbf{x}^{\text{tgt}}_{t}-\sqrt{1-\bar{\alpha}_{t}}\bm{\epsilon}_{\theta}(\mathbf{x}^{\text{tgt}}_{t},\mathbf{x}^{\text{c}},y,t))$

8:
$\hat{\mathcal{G}}=g_{\phi}\left(\mathbf{x}_{t}^{\text{tgt}},t,\mathbf{x}^{\text{c}},\tilde{\mathbf{x}}_{0}^{\text{tgt}}\right)$
// Enhance conditional 3D generation with 2D diffusion prior
$\tilde{\mathbf{x}}_{0}^{\text{tgt}}$ from $\mathbf{\epsilon}_{\theta}$

9:
$\\{\hat{\mathbf{x}}_{0}^{\text{tgt}},\hat{\mathbf{x}}_{0}^{\text{novel}}\\}={\color[rgb]{0.6484375,0.36328125,0.44921875}\definecolor[named]{pgfstrokecolor}{rgb}{0.6484375,0.36328125,0.44921875}\texttt{renderer}}\left(\hat{\mathcal{G}},\\{\pi^{\text{tgt}},\pi^{\text{novel}}\\}\right)$

10: Compute loss $\mathcal{L}_{total}$ ( Eq. 8)

11: Gradient step to update $\mathbf{\epsilon}_{\theta},g_{\phi}$

12:until converged

Algorithm 2 3D Consistent Sampling

1:A context image ${\mathbf{x}}^{c}$ and text $y$; Converged 2D diffusion
model $\epsilon_{\theta}$ and 3D generative model $g_{\phi}$

2:A 3D Gaussian Avatar $\mathcal{G}$ of the 2D image ${\mathbf{x}}^{c}$

3:${\mathbf{x}}_{T}^{\text{tgt}}\sim\mathcal{N}(\mathbf{0},\mathbf{I})$

4:for $t=T,\dotsc,1$ do

5:
$\tilde{\mathbf{x}}_{0}^{\text{tgt}}=\frac{1}{\sqrt{\bar{\alpha}_{t}}}(\mathbf{x}^{\text{tgt}}_{t}-\sqrt{1-\bar{\alpha}_{t}}\bm{\epsilon}_{\theta}(\mathbf{x}^{\text{tgt}}_{t},\mathbf{x}^{\text{c}},y,t))$

6:
$\hat{\mathcal{G}}=g_{\phi}\left(\mathbf{x}_{t}^{\text{tgt}},t,\mathbf{x}^{\text{c}},\tilde{\mathbf{x}}_{0}^{\text{tgt}}\right)$

7:
$\hat{\mathbf{x}}_{0}^{\text{tgt}}={\color[rgb]{0.6484375,0.36328125,0.44921875}\definecolor[named]{pgfstrokecolor}{rgb}{0.6484375,0.36328125,0.44921875}\texttt{renderer}}\left(\hat{\mathcal{G}},\pi^{\text{tgt}}\right)$

8:
$\mu_{t-1}({\mathbf{x}}_{t}^{\text{tgt}},\hat{{\mathbf{x}}}_{0}^{\text{tgt}})=\frac{\sqrt{\alpha_{t}}\left(1-\bar{\alpha}_{\text{t-1}}\right)}{1-\bar{\alpha}_{t}}{\mathbf{x}}^{\text{tgt}}_{t}+\frac{\sqrt{\bar{\alpha}_{\text{t-1}}}\beta_{t}}{1-\bar{\alpha}_{t}}\hat{{\mathbf{x}}}_{0}^{\text{tgt}}$
// Guide 2D sampling with 3D consistent renderings

9:
${\mathbf{x}}^{\text{tgt}}_{t-1}\sim\mathcal{N}\left(\mathbf{x}^{\text{tgt}}_{t-1};\tilde{\bm{\mu}}_{t}\left({\mathbf{x}}^{\text{tgt}}_{t},\hat{\mathbf{x}}^{\text{tgt}}_{0}\right),\tilde{\beta}_{t-1}\mathbf{I})\right)$

10:end for

11:return
$\mathcal{G}=g_{\phi}\left(\mathbf{x}_{0}^{\text{tgt}},\tilde{\mathbf{x}}_{0}^{\text{tgt}},\mathbf{x}^{\text{c}},t=0\right)$

With the shared and _synchronized_ sampling state
$\mathbf{x}_{t}^{\text{tgt}}$ of 2D multi-view diffusion model
$\bm{\epsilon}_{\theta}$ and 3D-GS reconstruction model $g_{\phi}$, we couple
both models at arbitrary $t$ during training. Similarly, they are also
connected by both using estimated clean multi-views
$\mathbf{x}_{0}^{\text{tgt}}$ at sampling time. To leverage the full potential
of both models, we carefully design a joint sampling process that utilizes the
reconstructed 3D-GS $\hat{\mathcal{G}}$ at each timestep $t$ to guide 2D
multi-view sampling, which is summarized in Algorithm 2.
We observe that the key difference between the clean multi-views estimated
$\mathbf{x}_{0}^{\text{tgt}}$ from 2D diffusion model and our 3D-GS generation
lies in 3D consistency: 2D MVD computes multi-view
$\tilde{\mathbf{x}}_{0}^{\text{tgt}}$ from 2D network prediction which can be
3D inconsistent while our $\hat{\mathbf{x}}_{0}^{\text{tgt}}$ are rendered
from explicit 3D-GS representation which are guaranteed to be 3D consistent.
Our idea is to guide the 2D multi-view reverse sampling process with our 3D
consistent renderings $\hat{\mathbf{x}}_{0}^{\text{tgt}}$ such that the 2D
sampling trajectory is more 3D consistent. Specifically, we leverage 3D
consistent multi-view renderings $\hat{\mathbf{x}}_{0}^{\text{tgt}}$ to refine
the posterior mean $\mu_{\theta}({\mathbf{x}}_{t}^{\text{tgt}},t)$ at each
reverse step:

$\displaystyle\begin{split}\text{Original:
}\mu_{\theta}({\mathbf{x}}_{t}^{\text{tgt}},t):=\mu_{t-1}({\mathbf{x}}_{t}^{\text{tgt}},\tilde{{\mathbf{x}}}_{0}^{\text{tgt}})\quad\rightarrow\quad\text{Ours:
}\mu_{\theta}({\mathbf{x}}_{t}^{\text{tgt}},t):=\mu_{t-1}({\mathbf{x}}_{t}^{\text{tgt}},\hat{{\mathbf{x}}}_{0}^{\text{tgt}}),\\\
\text{ where
}\hat{{\mathbf{x}}}_{0}^{\text{tgt}}={\color[rgb]{0.6484375,0.36328125,0.44921875}\definecolor[named]{pgfstrokecolor}{rgb}{0.6484375,0.36328125,0.44921875}\texttt{renderer}}(\hat{\mathcal{G}},\pi^{\text{tgt}}),\text{
and
}\mu_{t-1}({\mathbf{x}}_{t}^{\text{tgt}},\hat{{\mathbf{x}}}_{0}^{\text{tgt}})=\frac{\sqrt{\alpha_{t}}\left(1-\bar{\alpha}_{t-1}\right)}{1-\bar{\alpha}_{t}}{\mathbf{x}}^{\text{tgt}}_{t}+\frac{\sqrt{\bar{\alpha}_{t-1}}\beta_{t}}{1-\bar{\alpha}_{t}}\hat{{\mathbf{x}}}_{0}^{\text{tgt}}\end{split}$
(9)

With this refinement, we guarantee the 3D consistency at each reverse step $t$
and avoid 3D inconsistency accumulation in original multi-view sampling [75].
In Fig. 7, we visualize the evolution of originally generated multi-views
$\tilde{\mathbf{x}}^{\text{tgt}}_{0}$ and multi-views rendering
$\hat{\mathbf{x}}^{\text{tgt}}_{0}$ from generated 3D-GS $\hat{\mathcal{G}}$
along the whole reverse sampling process. It intuitively shows how effective
the sampling trajectory refinement is. We perform extensive ablation in Sec.
5.3 showing the importance of the consistent refinement for sampling
trajectory.

###  5 Experiments

In this section, we first compare against baseline methods for human
reconstruction in Sec. 5.2 and then ablate our design choices in Sec. 5.3.

####  5.1 Experimental Setup

Datasets. We train our model on a combined 3D human dataset [1, 3, 4, 2, 19,
25, 63, 96] compromising $\sim 6000$ high quality scans. Please refer to Sec.
D.1. We evaluate qualitatively and quantitatively on Sizer [70] and more
challenging IIIT [30] dataset due to extremely loose clothing of traditional
custom suits. Please refer to Sec. D.2 for examples in evaluation dataset.

Implementation Details. We trained our model on 8 NVIDIA A100 GPUs over
approximately 5 days. Each GPU was configured with a batch size 2 and gradient
accumulations of 16 steps to achieve an effective batch size of 256. For more
training details regarding hyperparameters, diffusion schedulers, etc., please
refer to Sec. A.1.

Evaluation Metrics. We evaluate the geometry quality using Chamfer Distance
(CD), F-score [67] (w/ threshold of $0.01m$), and Normal Consistency (NC)
between the extracted mesh (Sec. A.4) and the groundtruth scan. Appearance
quality is assessed by rendering the reconstructed avatar from 32 novel views
with uniform azimuth and 0 elevation angle. The metrics for appearance
reported include multi-scale Structure Similarity (SSIM) [76], Learned
Perceptual Image Patch Similarity (LPIPS) [102], and Peak Signal to Noise
Ratio (PSNR) between rendered and ground-truth views. Moreover, we report the
Fréchet inception distance (FID) [20] between synthesized views and ground
truth renderings, which reflects the quality and realism of the unseen
regions.

####  5.2 Realistic Avatar from Image

Figure 3: Qualitative comparison with baselines on SIZER [70] and IIIT [30]
dataset. Template-based methods (SiTH [21] and SIFU [106]) cannot reconstruct
loose clothing coherently and template free methods (PIFu [54] and TripoSR
[72]) tend to generate blurry texture in unseen regions due to their
deterministic formulation. LGM [65] and InstantMesh [87] cannot correct 3D
inconsistency from 2D multi-views hence the texture is also blurry. In
contract, our method is able to reconstruct avatars with realistic textures
and plausible 3D geometry in both seen and unseen region.

We compare our approach against prior methods for image-to-avatar
reconstruction. including template-based [21, 54, 106] and template-free [54]
human reconstruction methods, as well as general image-to-3D methods [65, 72].
To further assess performance, we also fine-tuned the state-of-the-art object
reconstruction method LGM [65] and its deployed multi-view diffusion model
[75] on our training data, denoted as $\text{LGM}_{\text{human}}$.
Quantitative evaluations reported in Tab. 1 demonstrate that our proposed
method excels in reconstructing realistic avatars with accurate geometry (CD,
NC, F-score) and realistic texture (SSIM, LPIPS, PSNR, FID) from a single RGB
image.

We present qualitative comparison examples in Fig. 3 and Sec. B.1,
highlighting the strengths and weaknesses of competing methods. Template-based
methods such as SiTH [21] and SIFU [106] struggle to accurately reconstruct
the geometry of loose clothing (as shown in row 4) due to their reliance on
the naked SMPL body shape. In contrast, template-free methods like PIFu [54]
and TripoSR [72] offer greater flexibility and better performance on loose
clothing. However, they are not generative models and their deterministic
formulations lead to blurry textures in unseen regions, as they tend to
produce average textures rather than distinct details. Similar to our
approach, LGM [65] and InstantMesh [87] utilize 2D diffusion models to
generate multi-view images and perform sparse-view 3D reconstruction.
Nonetheless, their separation of 2D and 3D models cannot correct the 3D
inconsistencies that may arise from the 2D models. Even further fine-tuning of
LGM on human scans (Fig. 4) does not adequately address these challenges due
to the complex and sensitive nature of human geometry and textures. In
contrast, our conditional generative formulation and inherent 3D consistency
by tightly coupling 2D-3D models allow us to obtain accurate reconstruction in
front view and realistic generation in unseen regions. We also show the
generative power of our method in Sec. C.5: by sampling with different seed,
we obtain diverse yet plausible reconstruction.

Please also refer to Fig. 6, Appendix C and our project page for additional
reconstruction results on challenging subjects not previously observed,
encompassing a diverse range of appearances such as loose skirts and custom
suits, as well as accessories like bags and gloves.

Method |  $\text{CD}_{\text{(cm)}}$ $\downarrow$ | F-score $\uparrow$ | $\text{NC}\uparrow$ | SSIM $\uparrow$ | LPIPS $\downarrow$ | PSNR $\uparrow$ | FID $\downarrow$
---|---|---|---|---|---|---|---
TripoSR [72] | $2.59$ | $0.360$ | $0.771$ | $0.743$ | $0.095$ | $20.05$ | $26.13$
InstantMesh [87] | $2.47$ | $0.338$ | $0.787$ | $0.900$ | $0.084$ | $20.54$ | $26.38$
LGM [65] | $3.29$ | $0.275$ | $0.562$ | $0.892$ | $0.088$ | $20.11$ | $24.21$
PIFu [54] | $2.83$ | $0.333$ | $0.769$ | $0.907$ | $0.078$ | $20.66$ | $32.73$
SiTH [21] | $3.92$ | $0.250$ | $0.735$ | $0.907$ | $0.074$ | $20.88$ | $24.76$
SIFU [106] | $3.34$ | $0.235$ | $0.739$ | $0.896$ | $0.085$ | $20.39$ | $42.63$
$\text{LGM}_{\text{human}}$ | $2.08$ | $0.300$ | $0.661$ | $0.904$ | $0.075$ | $20.61$ | $15.01$
_Ours_ | 1.35 | 0.550 | 0.798 | 0.918 | 0.060 | 21.49 | 9.57
Table 1: Quantitative evaluation on SIZER [70] and IIIT [30] dataset. Our
method reconstructs more accurate geometry (CD, F-score, NC) and realistic
textures (SSIM, LPIPS, PSNR, FID).

####  5.3 Ablation Studies

Figure 4: 3D reconstruction conditioned on different multi-view priors. Without our 3D-consistent sampling, the 2D diffusion model cannot generate 3D consistent multi-views (MVD, $\text{MVD}_{\text{ft}}$), leading to artifacts like floating 3D Gaussians splats. Our method obtains more consistent multi-views hence better 3D-GS and rendering.  Method |  LPIPS $\downarrow$ |  SSIM $\uparrow$ |  PSNR $\uparrow$
---|---|---|---
MVD | $0.078$ | $0.911$ | $22.32$
$\text{MVD}_{\text{ft}}$ | $0.061$ | $0.926$ | $24.14$
_Ours_ | 0.048 | 0.934 | 24.69
Table 2: Evaluating trajectory refinement for 2D multi-view diffusion. Our
proposed refinement improves multi-view image quality.

Importance of Trajectory Refinement. One of our key ideas is leveraging our
explicit 3D model to refine the 2D multi-view reverse sampling trajectory,
ensuring 3D consistency in Multi-View Diffusion (MVD) generation (see Sec. 4.2
and Eq. 9). To evaluate this, we compare the multi-view images generated by
pretrained MVD, fine-tuned MVD on our data ($\text{MVD}_{\text{ft}}$) and MVD
with our 3D consistent sampling (ours), as shown in Tab. 2. The results
demonstrate that our proposed method effectively enhances the quality of
generated multi-view images by leveraging the explicit 3D model to refine
sampling trajectory. Additionally. we analyze the 3D reconstruction results
with the multi-view images generated by these models in Fig. 4. MVD and
$\text{MVD}_{\text{ft}}$ produce inconsistent multi-view images, which
typically lead to floating Gaussian and hence blurry boundaries. In contrast,
our method can generate more consistent multi-views, result in better 3D
Gaussians Splats and sharper renderings.

Method |  $\text{CD}_{\text{(cm)}}$$\downarrow$ |  F-score$\uparrow$ |  $\text{NC}\uparrow$ |  LPIPS$\downarrow$ |  SSIM$\uparrow$ |  PSNR$\uparrow$
---|---|---|---|---|---|---
Our w/o Traj. Ref. |  $1.57$ |  $0.498$ |  $0.794$ |  $0.064$ |  $0.908$ |  $21.09$
_Ours_ |  1.35 |  0.550 |  0.798 |  0.060 |  0.918 |  21.49
Table 3: Evaluating trajectory refinement for final 3D reconstruction. Our
sampling trajectory refinement ensures multi-view consistency and hence yields
better 3D results.

We further quantitatively evaluate the impact of our proposed sampling
trajectory refinement on final 3D reconstruction in Tab. 3. We compare the
reconstruction results of methods with and without our trajectory refinement
while using the same 2D MVD and 3D reconstruction models. It can be clearly
seen that our trajectory refinement improves the quality of 3D reconstruction.

##### Importance of 2D Multi-view Prior.

Another key idea of our work is the use of multi-view priors
$\tilde{{\mathbf{x}}}_{0}^{\text{tgt}}$ from 2D diffusion model pretrained on
massive data [12, 53, 56] to enhance our 3D generative model. This additional
prior information is pivotal for ensuring accurate reconstruction of both in-
distribution human dataset and generalizing to out-of-distribution objects.

Method |  FID $\downarrow$
---|---
Ours w/o $\tilde{\mathbf{x}}^{\text{tgt}}_{0}$ | $12.75$
_Our full model_ | $\mathbf{9.57}$
Table 4: 2D multi-view priors $\tilde{\mathbf{x}}^{\text{tgt}}_{0}$ improve
human reconstruction quality.

We evaluate the performance of our 3D model $g_{\phi}$ by comparing generation
results with and without the 2D diffusion prior
$\tilde{\mathbf{x}}_{0}^{\text{tgt}}$ (refer to Eq. 7 and Eq. 5). Notably,
without the 2D multi-view conditioning, the alignment of the generated 3D
model in the front view is not guaranteed due to the relative camera pose
settings in our 3D generative model $g_{\phi}$. Therefore, we evaluate the
overall quality solely through the Fréchet Inception Distance (FID).

For avatars reconstruction, our powerful 3D reconstruction model can already
achieve state-of-the-art performance. Moreover, our full model with multi-view
prior $\tilde{{\mathbf{x}}}_{0}^{\text{tgt}}$ generates avatars with higher
quality as demonstrated in Tab. 4. We further evaluate it on the GSO [16]
dataset which consists of unseen general objects to our model. The
improvements are even more pronounced in this setting, highlighting the
challenges of generating coherent 3D structures from a single 2D image,
particularly with unseen objects. For additional examples, please see Fig. 15
in Supp..

Method | FID $\downarrow$
---|---
w/o $\tilde{\mathbf{x}}^{\text{tgt}}_{0}$ | $107.14$
_Ours_ | $\mathbf{83.89}$

Figure 5: 2D multi-view priors $\tilde{{\mathbf{x}}}_{0}^{\text{tgt}}$
enhances generalization to general objects in GSO [16] dataset. Figure 6:
Visualization of reconstructed mesh and synthesized novel view of generated
3D-GS on subjects from (A) Sizer [70], (B) IIIT [30], (C) Twindom [4], (D) UBC
Fashion [99], (E) GSO [16] and (F) online image "the Rock". More results are
presented in Appendix C and our project page.

###  6 Limitations and Future Work

Currently, our method is constrained by the $256\times 256$ resolution of the
multi-view diffusion model, which restricts the sharpness of texture details
(see Appendix E). Upgrading to more powerful high-resolution ($512\times 512$)
multi-view diffusion models [17, 66] could potentially resolve these issues.
Moreover, our approach may struggle in reconstructing subjects with
challenging poses, as we further discussed in Appendix E. Synthesizing
training data with challenging poses [9, 84] could be a potential solution.

Our method is a general framework for image-to-3D reconstruction, which is
applicable to various objects and compositional shapes like human-object
interactions. We leave these to future works.

###  7 Conclusion

In this paper, we introduce Human 3Diffusion, a 3D consistent diffusion model
for creating realistic avatars from single RGB images. Our key ideas are two
folds: 1) Leveraging strong multi-view priors from pretrained 2D diffusion
models to generate 3D Gaussian Splats, and 2) Using the reconstructed explicit
3D Gaussian Splats to refine the sampling trajectory of the 2D diffusion model
which enhances 3D consistency. We carefully designed a diffusion process that
synergistically combines the strengths of both 2D and 3D models. Our
experiments show that our approach outperforms all previous reconstruction
works in both appearance and geometry. We also extensively ablate our method
which proves the effectiveness of our proposed ideas. Our code and pretrained
models will be released on our Project Page to foster future research.

##### Acknowledgements

We appreciate G.Tiwari, Y.He, Y. Xiu, Z.Liu, Z.Qiu, S.Li and others for their
feedback to improve the work. This work is made possible by funding from the
Carl Zeiss Foundation. This work is also funded by the Deutsche
Forschungsgemeinschaft (DFG, German Research Foundation) - 409792180
(EmmyNoether Programme, project: Real Virtual Humans) and the German Federal
Ministry of Education and Research (BMBF): Tübingen AI Center, FKZ:
01IS18039A. The authors thank the International Max Planck Research School for
Intelligent Systems (IMPRS-IS) for supporting Y.Xue. G. Pons-Moll is a member
of the Machine Learning Cluster of Excellence, EXC number 2064/1 – Project
number 390727645. R. Marin has been supported by innovation program under the
Marie Skłodowska-Curie grant agreement No 101109330.

### References

  * axy [2023] Axyz, Nov 2023.  URL https://secure.axyz-design.com.
  * ren [2023] Renderpeople, Nov 2023.  URL https://renderpeople.com/.
  * tre [2023] Treedy, Nov 2023.  URL https://treedys.com/.
  * twi [2023] Twindom, Nov 2023.  URL https://web.twindom.com/.
  * Alldieck et al. [2022] Thiemo Alldieck, Mihai Zanfir, and Cristian Sminchisescu.  Photorealistic Monocular 3D Reconstruction of Humans Wearing Clothing.  In _2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 1496–1505, New Orleans, LA, USA, June 2022. IEEE.  ISBN 978-1-66546-946-3.  doi: 10.1109/CVPR52688.2022.00156.  URL https://ieeexplore.ieee.org/document/9878998/.
  * Alwala et al. [2022] Kalyan Vasudev Alwala, Abhinav Gupta, and Shubham Tulsiani.  Pre-train, self-train, distill: A simple recipe for supersizing 3d reconstruction.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 3773–3782, June 2022.
  * Anciukevicius et al. [2023] Titas Anciukevicius, Zexiang Xu, Matthew Fisher, Paul Henderson, Hakan Bilen, Niloy J. Mitra, and Paul Guerrero.  Renderdiffusion: Image diffusion for 3d reconstruction, inpainting and generation.  In _IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2023, Vancouver, BC, Canada, June 17-24, 2023_ , pages 12608–12618. IEEE, 2023.  doi: 10.1109/CVPR52729.2023.01213.  URL https://doi.org/10.1109/CVPR52729.2023.01213.
  * Bhatnagar et al. [2022] Bharat Lal Bhatnagar, Xianghui Xie, Ilya Petrov, Cristian Sminchisescu, Christian Theobalt, and Gerard Pons-Moll.  Behave: Dataset and method for tracking human object interactions.  In _IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_. IEEE, jun 2022.
  * Black et al. [2023] Michael J. Black, Priyanka Patel, Joachim Tesch, and Jinlong Yang.  BEDLAM: A synthetic dataset of bodies exhibiting detailed lifelike animated motion.  In _Proceedings IEEE/CVF Conf. on Computer Vision and Pattern Recognition (CVPR)_ , pages 8726–8737, June 2023.
  * Cao et al. [2022] Xu Cao, Hiroaki Santo, Boxin Shi, Fumio Okura, and Yasuyuki Matsushita.  Bilateral normal integration.  2022\.
  * Chan et al. [2022] Eric R. Chan, Connor Z. Lin, Matthew A. Chan, Koki Nagano, Boxiao Pan, Shalini De Mello, Orazio Gallo, Leonidas J. Guibas, Jonathan Tremblay, Sameh Khamis, Tero Karras, and Gordon Wetzstein.  Efficient geometry-aware 3d generative adversarial networks.  In _IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2022, New Orleans, LA, USA, June 18-24, 2022_ , pages 16102–16112. IEEE, 2022.  doi: 10.1109/CVPR52688.2022.01565.  URL https://doi.org/10.1109/CVPR52688.2022.01565.
  * Deitke et al. [2023] Matt Deitke, Dustin Schwenk, Jordi Salvador, Luca Weihs, Oscar Michel, Eli VanderBilt, Ludwig Schmidt, Kiana Ehsani, Aniruddha Kembhavi, and Ali Farhadi.  Objaverse: A universe of annotated 3d objects.  In _IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2023, Vancouver, BC, Canada, June 17-24, 2023_ , pages 13142–13153. IEEE, 2023.  doi: 10.1109/CVPR52729.2023.01263.  URL https://doi.org/10.1109/CVPR52729.2023.01263.
  * Denninger et al. [2023] Maximilian Denninger, Dominik Winkelbauer, Martin Sundermeyer, Wout Boerdijk, Markus Knauer, Klaus H. Strobl, Matthias Humt, and Rudolph Triebel.  Blenderproc2: A procedural pipeline for photorealistic rendering.  _Journal of Open Source Software_ , 8(82):4901, 2023.  doi: 10.21105/joss.04901.  URL https://doi.org/10.21105/joss.04901.
  * Dhariwal and Nichol [2021] Prafulla Dhariwal and Alexander Quinn Nichol.  Diffusion models beat GANs on image synthesis.  In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan, editors, _Advances in Neural Information Processing Systems_ , 2021.  URL https://openreview.net/forum?id=AAWuCvzaVt.
  * Dong et al. [2022] Zijian Dong, Chen Guo, Jie Song, Xu Chen, Andreas Geiger, and Otmar Hilliges.  Pina: Learning a personalized implicit neural avatar from a single rgb-d video sequence.  _arXiv_ , 2022.
  * Downs et al. [2022] Laura Downs, Anthony Francis, Nate Koenig, Brandon Kinman, Ryan Hickman, Krista Reymann, Thomas Barlow McHugh, and Vincent Vanhoucke.  Google scanned objects: A high-quality dataset of 3d scanned household items.  In _2022 International Conference on Robotics and Automation, ICRA 2022, Philadelphia, PA, USA, May 23-27, 2022_ , pages 2553–2560. IEEE, 2022.  doi: 10.1109/ICRA46639.2022.9811809.  URL https://doi.org/10.1109/ICRA46639.2022.9811809.
  * Gao* et al. [2024] Ruiqi Gao*, Aleksander Holynski*, Philipp Henzler, Arthur Brussee, Ricardo Martin-Brualla, Pratul P. Srinivasan, Jonathan T. Barron, and Ben Poole*.  Cat3d: Create anything in 3d with multi-view diffusion models.  _arXiv_ , 2024.
  * Guzov et al. [2021] Vladimir Guzov, Aymen Mir, Torsten Sattler, and Gerard Pons-Moll.  Human poseitioning system (HPS): 3d human pose estimation and self-localization in large scenes from body-mounted sensors.  _CoRR_ , abs/2103.17265, 2021.  URL https://arxiv.org/abs/2103.17265.
  * Han et al. [2023] Sang-Hun Han, Min-Gyu Park, Ju Hong Yoon, Ju-Mi Kang, Young-Jae Park, and Hae-Gon Jeon.  High-fidelity 3d human digitization from single 2k resolution images.  In _IEEE Conference on Computer Vision and Pattern Recognition (CVPR2023)_ , June 2023.
  * Heusel et al. [2017] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter.  Gans trained by a two time-scale update rule converge to a local nash equilibrium.  In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Garnett, editors, _Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA_ , pages 6626–6637, 2017.  URL https://proceedings.neurips.cc/paper/2017/hash/8a1d694707eb0fefe65871369074926d-Abstract.html.
  * Ho et al. [2023] Hsuan-I Ho, Jie Song, and Otmar Hilliges.  Sith: Single-view textured human reconstruction with image-conditioned diffusion.  _CoRR_ , abs/2311.15855, 2023.  doi: 10.48550/ARXIV.2311.15855.  URL https://doi.org/10.48550/arXiv.2311.15855.
  * Ho and Salimans [2021] Jonathan Ho and Tim Salimans.  Classifier-free diffusion guidance.  In _NeurIPS 2021 Workshop on Deep Generative Models and Downstream Applications_ , 2021.  URL https://openreview.net/forum?id=qw8AKxfYbI.
  * Ho et al. [2020] Jonathan Ho, Ajay Jain, and Pieter Abbeel.  Denoising diffusion probabilistic models.  In Hugo Larochelle, Marc’Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tien Lin, editors, _Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual_ , 2020.  URL https://proceedings.neurips.cc/paper/2020/hash/4c5bcfec8584af0d967f1ab10179ca4b-Abstract.html.
  * Hong et al. [2023] Yicong Hong, Kai Zhang, Jiuxiang Gu, Sai Bi, Yang Zhou, Difan Liu, Feng Liu, Kalyan Sunkavalli, Trung Bui, and Hao Tan.  LRM: large reconstruction model for single image to 3d.  _CoRR_ , abs/2311.04400, 2023.  doi: 10.48550/ARXIV.2311.04400.  URL https://doi.org/10.48550/arXiv.2311.04400.
  * Hsuan-I et al. [2023] Ho Hsuan-I, Xue Lixin, Song Jie, and Hilliges Otmar.  Learning locally editable virtual humans.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2023.
  * Hu et al. [2024] Hanzhe Hu, Zhizhuo Zhou, Varun Jampani, and Shubham Tulsiani.  MVD-Fusion: Single-view 3D via Depth-consistent Multi-view Generation, April 2024.
  * Huang et al. [2024a] Binbin Huang, Zehao Yu, Anpei Chen, Andreas Geiger, and Shenghua Gao.  2d gaussian splatting for geometrically accurate radiance fields.  _CoRR_ , abs/2403.17888, 2024a.  doi: 10.48550/ARXIV.2403.17888.  URL https://doi.org/10.48550/arXiv.2403.17888.
  * Huang et al. [2024b] Zehuan Huang, Hao Wen, Junting Dong, Yaohui Wang, Yangguang Li, Xinyuan Chen, Yan-Pei Cao, Ding Liang, Yu Qiao, Bo Dai, and Lu Sheng.  EpiDiff: Enhancing Multi-View Synthesis via Localized Epipolar-Constrained Diffusion, April 2024b.
  * Jiang et al. [2022] Tianjian Jiang, Xu Chen, Jie Song, and Otmar Hilliges.  Instantavatar: Learning avatars from monocular video in 60 seconds.  _arXiv_ , 2022.
  * Jinka et al. [2023] Sai Sagar Jinka, Astitva Srivastava, Chandradeep Pokhariya, Avinash Sharma, and P. J. Narayanan.  SHARP: shape-aware reconstruction of people in loose clothing.  _Int. J. Comput. Vis._ , 131(4):918–937, 2023.  doi: 10.1007/S11263-022-01736-Z.  URL https://doi.org/10.1007/s11263-022-01736-z.
  * Kabadayi et al. [2024] Berna Kabadayi, Wojciech Zielonka, Bharat Lal Bhatnagar, Gerard Pons-Moll, and Justus Thies.  Gan-avatar: Controllable personalized gan-based human head avatar.  In _International Conference on 3D Vision (3DV)_ , March 2024.
  * Kant et al. [2024] Yash Kant, Ziyi Wu, Michael Vasilkovsky, Guocheng Qian, Jian Ren, Riza Alp Güler, Bernard Ghanem, Sergey Tulyakov, Igor Gilitschenski, and Aliaksandr Siarohin.  SPAD : Spatially aware multiview diffusers.  _CoRR_ , abs/2402.05235, 2024.  doi: 10.48550/ARXIV.2402.05235.  URL https://doi.org/10.48550/arXiv.2402.05235.
  * Karnewar et al. [2023] Animesh Karnewar, Andrea Vedaldi, David Novotny, and Niloy Mitra.  Holodiffusion: Training a 3D diffusion model using 2D images.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , 2023.
  * Kerbl et al. [2023] Bernhard Kerbl, Georgios Kopanas, Thomas Leimkühler, and George Drettakis.  3d gaussian splatting for real-time radiance field rendering.  _ACM Trans. Graph._ , 42(4):139:1–139:14, 2023.  doi: 10.1145/3592433.  URL https://doi.org/10.1145/3592433.
  * Kong et al. [2024] Xin Kong, Shikun Liu, Xiaoyang Lyu, Marwan Taher, Xiaojuan Qi, and Andrew J. Davison.  Eschernet: A generative model for scalable view synthesis.  _CoRR_ , abs/2402.03908, 2024.  doi: 10.48550/ARXIV.2402.03908.  URL https://doi.org/10.48550/arXiv.2402.03908.
  * Li et al. [2024] Yifei Li, Hsiao-yu Chen, Egor Larionov, Nikolaos Sarafianos, Wojciech Matusik, and Tuur Stuyck.  DiffAvatar: Simulation-ready garment optimization with differentiable simulation.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , June 2024.
  * Liao et al. [2023] Tingting Liao, Hongwei Yi, Yuliang Xiu, Jiaxiang Tang, Yangyi Huang, Justus Thies, and Michael J. Black.  Tada! text to animatable digital avatars.  _CoRR_ , abs/2308.10899, 2023.  doi: 10.48550/ARXIV.2308.10899.  URL https://doi.org/10.48550/arXiv.2308.10899.
  * Liu et al. [2023a] Minghua Liu, Ruoxi Shi, Linghao Chen, Zhuoyang Zhang, Chao Xu, Xinyue Wei, Hansheng Chen, Chong Zeng, Jiayuan Gu, and Hao Su.  One-2-3-45++: Fast single image to 3d objects with consistent multi-view generation and 3d diffusion.  _CoRR_ , abs/2311.07885, 2023a.  doi: 10.48550/ARXIV.2311.07885.  URL https://doi.org/10.48550/arXiv.2311.07885.
  * Liu et al. [2023b] Minghua Liu, Chao Xu, Haian Jin, Linghao Chen, Mukund Varma T, Zexiang Xu, and Hao Su.  One-2-3-45: Any single image to 3d mesh in 45 seconds without per-shape optimization.  In Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine, editors, _Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 - 16, 2023_ , 2023b.  URL http://papers.nips.cc/paper_files/paper/2023/hash/4683beb6bab325650db13afd05d1a14a-Abstract-Conference.html.
  * Liu et al. [2023c] Ruoshi Liu, Rundi Wu, Basile Van Hoorick, Pavel Tokmakov, Sergey Zakharov, and Carl Vondrick.  Zero-1-to-3: Zero-shot one image to 3d object.  In _IEEE/CVF International Conference on Computer Vision, ICCV 2023, Paris, France, October 1-6, 2023_ , pages 9264–9275. IEEE, 2023c.  doi: 10.1109/ICCV51070.2023.00853.  URL https://doi.org/10.1109/ICCV51070.2023.00853.
  * Liu et al. [2023d] Yuan Liu, Cheng Lin, Zijiao Zeng, Xiaoxiao Long, Lingjie Liu, Taku Komura, and Wenping Wang.  Syncdreamer: Generating multiview-consistent images from a single-view image.  _CoRR_ , abs/2309.03453, 2023d.  doi: 10.48550/ARXIV.2309.03453.  URL https://doi.org/10.48550/arXiv.2309.03453.
  * Liu et al. [2023e] Zhen Liu, Yao Feng, Yuliang Xiu, Weiyang Liu, Liam Paull, Michael J. Black, and Bernhard Schölkopf.  Ghost on the shell: An expressive representation of general 3d shapes.  _arXiv preprint arXiv:2310.15168_ , 2023e.
  * Long et al. [2023] Xiaoxiao Long, Yuan-Chen Guo, Cheng Lin, Yuan Liu, Zhiyang Dou, Lingjie Liu, Yuexin Ma, Song-Hai Zhang, Marc Habermann, Christian Theobalt, and Wenping Wang.  Wonder3d: Single image to 3d using cross-domain diffusion.  _CoRR_ , abs/2310.15008, 2023.  doi: 10.48550/ARXIV.2310.15008.  URL https://doi.org/10.48550/arXiv.2310.15008.
  * Loper et al. [2015] Matthew Loper, Naureen Mahmood, Javier Romero, Gerard Pons-Moll, and Michael J. Black.  SMPL: a skinned multi-person linear model.  _ACM Trans. Graph._ , 34(6):248:1–248:16, 2015.  doi: 10.1145/2816795.2818013.  URL https://doi.org/10.1145/2816795.2818013.
  * Ma et al. [2019] Qianli Ma, Siyu Tang, Sergi Pujades, Gerard Pons-Moll, Anurag Ranjan, and Michael J. Black.  Dressing 3d humans using a conditional mesh-vae-gan.  _CoRR_ , abs/1907.13615, 2019.  URL http://arxiv.org/abs/1907.13615.
  * Melas-Kyriazi et al. [2023] Luke Melas-Kyriazi, Christian Rupprecht, Iro Laina, and Andrea Vedaldi.  Realfusion: 360° reconstruction of any object from a single image.  In _Arxiv_ , 2023.
  * Mildenhall et al. [2022] Ben Mildenhall, Pratul P. Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, and Ren Ng.  Nerf: representing scenes as neural radiance fields for view synthesis.  _Commun. ACM_ , 65(1):99–106, 2022.  doi: 10.1145/3503250.  URL https://doi.org/10.1145/3503250.
  * Pavlakos et al. [2019] Georgios Pavlakos, Vasileios Choutas, Nima Ghorbani, Timo Bolkart, Ahmed A. A. Osman, Dimitrios Tzionas, and Michael J. Black.  Expressive body capture: 3d hands, face, and body from a single image.  _CoRR_ , abs/1904.05866, 2019.  URL http://arxiv.org/abs/1904.05866.
  * Petrov et al. [2023] Ilya A. Petrov, Riccardo Marin, Julian Chibane, and Gerard Pons-Moll.  Object pop-up: Can we infer 3d objects and their poses from human interactions alone?  In _IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2023, Vancouver, BC, Canada, June 17-24, 2023_ , pages 4726–4736. IEEE, 2023.  doi: 10.1109/CVPR52729.2023.00458.  URL https://doi.org/10.1109/CVPR52729.2023.00458.
  * Pons-Moll et al. [2017] Gerard Pons-Moll, Sergi Pujades, Sonny Hu, and Michael J. Black.  Clothcap: seamless 4d clothing capture and retargeting.  _ACM Trans. Graph._ , 36(4):73:1–73:15, 2017.  doi: 10.1145/3072959.3073711.  URL https://doi.org/10.1145/3072959.3073711.
  * Poole et al. [2022] Ben Poole, Ajay Jain, Jonathan T. Barron, and Ben Mildenhall.  Dreamfusion: Text-to-3d using 2d diffusion.  _arXiv_ , 2022.
  * Qian et al. [2023] Guocheng Qian, Jinjie Mai, Abdullah Hamdi, Jian Ren, Aliaksandr Siarohin, Bing Li, Hsin-Ying Lee, Ivan Skorokhodov, Peter Wonka, Sergey Tulyakov, and Bernard Ghanem.  Magic123: One image to high-quality 3d object generation using both 2d and 3d diffusion priors.  _CoRR_ , abs/2306.17843, 2023.  doi: 10.48550/ARXIV.2306.17843.  URL https://doi.org/10.48550/arXiv.2306.17843.
  * Rombach et al. [2022] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer.  High-resolution image synthesis with latent diffusion models.  In _IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2022, New Orleans, LA, USA, June 18-24, 2022_ , pages 10674–10685. IEEE, 2022.  doi: 10.1109/CVPR52688.2022.01042.  URL https://doi.org/10.1109/CVPR52688.2022.01042.
  * Saito et al. [2019] Shunsuke Saito, Zeng Huang, Ryota Natsume, Shigeo Morishima, Hao Li, and Angjoo Kanazawa.  Pifu: Pixel-aligned implicit function for high-resolution clothed human digitization.  In _2019 IEEE/CVF International Conference on Computer Vision, ICCV 2019, Seoul, Korea (South), October 27 - November 2, 2019_ , pages 2304–2314. IEEE, 2019.  doi: 10.1109/ICCV.2019.00239.  URL https://doi.org/10.1109/ICCV.2019.00239.
  * Saito et al. [2020] Shunsuke Saito, Tomas Simon, Jason M. Saragih, and Hanbyul Joo.  Pifuhd: Multi-level pixel-aligned implicit function for high-resolution 3d human digitization.  In _2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2020, Seattle, WA, USA, June 13-19, 2020_ , pages 81–90. Computer Vision Foundation / IEEE, 2020.  doi: 10.1109/CVPR42600.2020.00016.  URL https://openaccess.thecvf.com/content_CVPR_2020/html/Saito_PIFuHD_Multi-Level_Pixel-Aligned_Implicit_Function_for_High-Resolution_3D_Human_Digitization_CVPR_2020_paper.html.
  * Schuhmann et al. [2022] Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, Patrick Schramowski, Srivatsa Kundurthy, Katherine Crowson, Ludwig Schmidt, Robert Kaczmarczyk, and Jenia Jitsev.  LAION-5B: an open large-scale dataset for training next generation image-text models.  In Sanmi Koyejo, S. Mohamed, A. Agarwal, Danielle Belgrave, K. Cho, and A. Oh, editors, _Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Systems 2022, NeurIPS 2022, New Orleans, LA, USA, November 28 - December 9, 2022_ , 2022.  URL http://papers.nips.cc/paper_files/paper/2022/hash/a1859debfb3b59d094f3504d5ebb6c25-Abstract-Datasets_and_Benchmarks.html.
  * Sengupta et al. [2024] Akash Sengupta, Thiemo Alldieck, Nikos Kolotouros, Enric Corona, Andrei Zanfir, and Cristian Sminchisescu.  DiffHuman: Probabilistic Photorealistic 3D Reconstruction of Humans, March 2024.  URL http://arxiv.org/abs/2404.00485.  arXiv:2404.00485 [cs].
  * Shi et al. [2023a] Ruoxi Shi, Hansheng Chen, Zhuoyang Zhang, Minghua Liu, Chao Xu, Xinyue Wei, Linghao Chen, Chong Zeng, and Hao Su.  Zero123++: a single image to consistent multi-view diffusion base model.  _CoRR_ , abs/2310.15110, 2023a.  doi: 10.48550/ARXIV.2310.15110.  URL https://doi.org/10.48550/arXiv.2310.15110.
  * Shi et al. [2023b] Yichun Shi, Peng Wang, Jianglong Ye, Mai Long, Kejie Li, and Xiao Yang.  Mvdream: Multi-view diffusion for 3d generation.  _CoRR_ , abs/2308.16512, 2023b.  doi: 10.48550/ARXIV.2308.16512.  URL https://doi.org/10.48550/arXiv.2308.16512.
  * Simonyan and Zisserman [2015] Karen Simonyan and Andrew Zisserman.  Very deep convolutional networks for large-scale image recognition.  In Yoshua Bengio and Yann LeCun, editors, _3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings_ , 2015.  URL http://arxiv.org/abs/1409.1556.
  * Song et al. [2021a] Jiaming Song, Chenlin Meng, and Stefano Ermon.  Denoising diffusion implicit models.  In _9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021_. OpenReview.net, 2021a.  URL https://openreview.net/forum?id=St1giarCHLP.
  * Song et al. [2021b] Yang Song, Jascha Sohl-Dickstein, Diederik P. Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole.  Score-Based Generative Modeling through Stochastic Differential Equations, February 2021b.  URL http://arxiv.org/abs/2011.13456.  arXiv:2011.13456 [cs, stat].
  * Su et al. [2023] Zhaoqi Su, Tao Yu, Yangang Wang, and Yebin Liu.  Deepcloth: Neural garment representation for shape and style editing.  _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , 45(2):1581–1593, 2023.  doi: 10.1109/TPAMI.2022.3168569.
  * Suhail et al. [2022] Mohammed Suhail, Carlos Esteves, Leonid Sigal, and Ameesh Makadia.  Generalizable patch-based neural rendering.  In Shai Avidan, Gabriel J. Brostow, Moustapha Cissé, Giovanni Maria Farinella, and Tal Hassner, editors, _Computer Vision - ECCV 2022 - 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XXXII_ , volume 13692 of _Lecture Notes in Computer Science_ , pages 156–174. Springer, 2022.  doi: 10.1007/978-3-031-19824-3\\_10.  URL https://doi.org/10.1007/978-3-031-19824-3_10.
  * Tang et al. [2024a] Jiaxiang Tang, Zhaoxi Chen, Xiaokang Chen, Tengfei Wang, Gang Zeng, and Ziwei Liu.  LGM: large multi-view gaussian model for high-resolution 3d content creation.  _CoRR_ , abs/2402.05054, 2024a.  doi: 10.48550/ARXIV.2402.05054.  URL https://doi.org/10.48550/arXiv.2402.05054.
  * Tang et al. [2024b] Shitao Tang, Jiacheng Chen, Dilin Wang, Chengzhou Tang, Fuyang Zhang, Yuchen Fan, Vikas Chandra, Yasutaka Furukawa, and Rakesh Ranjan.  Mvdiffusion++: A dense high-resolution multi-view diffusion model for single or sparse-view 3d object reconstruction.  _CoRR_ , abs/2402.12712, 2024b.  doi: 10.48550/ARXIV.2402.12712.  URL https://doi.org/10.48550/arXiv.2402.12712.
  * Tatarchenko et al. [2019] Maxim Tatarchenko, Stephan R. Richter, René Ranftl, Zhuwen Li, Vladlen Koltun, and Thomas Brox.  What do single-view 3d reconstruction networks learn?  In _IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2019, Long Beach, CA, USA, June 16-20, 2019_ , pages 3405–3414. Computer Vision Foundation / IEEE, 2019.  doi: 10.1109/CVPR.2019.00352.  URL http://openaccess.thecvf.com/content_CVPR_2019/html/Tatarchenko_What_Do_Single-View_3D_Reconstruction_Networks_Learn_CVPR_2019_paper.html.
  * Tewari et al. [2023] Ayush Tewari, Tianwei Yin, George Cazenavette, Semon Rezchikov, Josh Tenenbaum, Frédo Durand, Bill Freeman, and Vincent Sitzmann.  Diffusion with forward models: Solving stochastic inverse problems without direct supervision.  In Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine, editors, _Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 - 16, 2023_ , 2023.  URL http://papers.nips.cc/paper_files/paper/2023/hash/28e4ee96c94e31b2d040b4521d2b299e-Abstract-Conference.html.
  * Thai et al. [2020] Anh Thai, Stefan Stojanov, Vijay Upadhya, and James M. Rehg.  3d reconstruction of novel object shapes from single images, 2020.
  * Tiwari et al. [2020] Garvita Tiwari, Bharat Lal Bhatnagar, Tony Tung, and Gerard Pons-Moll.  SIZER: A dataset and model for parsing 3d clothing and learning size sensitive 3d clothing.  In Andrea Vedaldi, Horst Bischof, Thomas Brox, and Jan-Michael Frahm, editors, _Computer Vision - ECCV 2020 - 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part III_ , volume 12348 of _Lecture Notes in Computer Science_ , pages 1–18. Springer, 2020.  doi: 10.1007/978-3-030-58580-8\\_1.  URL https://doi.org/10.1007/978-3-030-58580-8_1.
  * Tiwari et al. [2021] Garvita Tiwari, Nikolaos Sarafianos, Tony Tung, and Gerard Pons-Moll.  Neural-gif: Neural generalized implicit functions for animating people in clothing.  In _2021 IEEE/CVF International Conference on Computer Vision, ICCV 2021, Montreal, QC, Canada, October 10-17, 2021_ , pages 11688–11698. IEEE, 2021.  doi: 10.1109/ICCV48922.2021.01150.  URL https://doi.org/10.1109/ICCV48922.2021.01150.
  * Tochilkin et al. [2024] Dmitry Tochilkin, David Pankratz, ZeXiang Liu, Zixuan Huang, Adam Letts, Yangguang Li, Ding Liang, Christian Laforte, Varun Jampani, and Yan-Pei Cao.  Triposr: Fast 3d object reconstruction from a single image.  _CoRR_ , abs/2403.02151, 2024.  doi: 10.48550/ARXIV.2403.02151.  URL https://doi.org/10.48550/arXiv.2403.02151.
  * Vaswani et al. [2017] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Garnett, editors, _Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA_ , pages 5998–6008, 2017.  URL https://proceedings.neurips.cc/paper/2017/hash/3f5ee243547dee91fbd053c1c4a845aa-Abstract.html.
  * Voleti et al. [2024] Vikram Voleti, Chun-Han Yao, Mark Boss, Adam Letts, David Pankratz, Dmitry Tochilkin, Christian Laforte, Robin Rombach, and Varun Jampani.  SV3D: Novel Multi-view Synthesis and 3D Generation from a Single Image using Latent Video Diffusion, March 2024.
  * Wang and Shi [2023] Peng Wang and Yichun Shi.  Imagedream: Image-prompt multi-view diffusion for 3d generation.  _CoRR_ , abs/2312.02201, 2023.  doi: 10.48550/ARXIV.2312.02201.  URL https://doi.org/10.48550/arXiv.2312.02201.
  * Wang et al. [2003] Z. Wang, E.P. Simoncelli, and A.C. Bovik.  Multiscale structural similarity for image quality assessment.  In _The Thrity-Seventh Asilomar Conference on Signals, Systems & Computers, 2003_, volume 2, pages 1398–1402 Vol.2, 2003.  doi: 10.1109/ACSSC.2003.1292216.
  * Wang et al. [2023] Zhengyi Wang, Cheng Lu, Yikai Wang, Fan Bao, Chongxuan Li, Hang Su, and Jun Zhu.  Prolificdreamer: High-fidelity and diverse text-to-3d generation with variational score distillation.  _arXiv preprint arXiv:2305.16213_ , 2023.
  * Weng et al. [2022] Chung-Yi Weng, Brian Curless, Pratul P. Srinivasan, Jonathan T. Barron, and Ira Kemelmacher-Shlizerman.  HumanNeRF: Free-viewpoint rendering of moving people from monocular video.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 16210–16220, June 2022.
  * Wu et al. [2018] Jiajun Wu, Chengkai Zhang, Xiuming Zhang, Zhoutong Zhang, William T Freeman, and Joshua B Tenenbaum.  Learning 3D Shape Priors for Shape Completion and Reconstruction.  In _European Conference on Computer Vision (ECCV)_ , 2018.
  * Wu et al. [2023] Tong Wu, Jiarui Zhang, Xiao Fu, Yuxin Wang, Jiawei Ren, Liang Pan, Wayne Wu, Lei Yang, Jiaqi Wang, Chen Qian, Dahua Lin, and Ziwei Liu.  Omniobject3d: Large-vocabulary 3d object dataset for realistic perception, reconstruction and generation.  In _IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2023, Vancouver, BC, Canada, June 17-24, 2023_ , pages 803–814. IEEE, 2023.  doi: 10.1109/CVPR52729.2023.00084.  URL https://doi.org/10.1109/CVPR52729.2023.00084.
  * Xian et al. [2022] Yongqin Xian, Julian Chibane, Bharat Lal Bhatnagar, Bernt Schiele, Zeynep Akata, and Gerard Pons-Moll.  Any-shot gin: Generalizing implicit networks for reconstructing novel classes.  In _2022 International Conference on 3D Vision (3DV)_. IEEE, 2022.
  * Xie et al. [2022] Xianghui Xie, Bharat Lal Bhatnagar, and Gerard Pons-Moll.  CHORE: contact, human and object reconstruction from a single RGB image.  _CoRR_ , abs/2204.02445, 2022.  doi: 10.48550/ARXIV.2204.02445.  URL https://doi.org/10.48550/arXiv.2204.02445.
  * Xie et al. [2023] Xianghui Xie, Bharat Lal Bhatnagar, and Gerard Pons-Moll.  Visibility aware human-object interaction tracking from single RGB camera.  In _IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2023, Vancouver, BC, Canada, June 17-24, 2023_ , pages 4757–4768. IEEE, 2023.  doi: 10.1109/CVPR52729.2023.00461.  URL https://doi.org/10.1109/CVPR52729.2023.00461.
  * Xie et al. [2024] Xianghui Xie, Bharat Lal Bhatnagar, Jan Eric Lenssen, and Gerard Pons-Moll.  Template free reconstruction of human-object interaction with procedural interaction generation.  In _IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , June 2024.
  * Xiu et al. [2022] Yuliang Xiu, Jinlong Yang, Dimitrios Tzionas, and Michael J. Black.  ICON: implicit clothed humans obtained from normals.  In _IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2022, New Orleans, LA, USA, June 18-24, 2022_ , pages 13286–13296. IEEE, 2022.  doi: 10.1109/CVPR52688.2022.01294.  URL https://doi.org/10.1109/CVPR52688.2022.01294.
  * Xiu et al. [2023] Yuliang Xiu, Jinlong Yang, Xu Cao, Dimitrios Tzionas, and Michael J. Black.  ECON: explicit clothed humans optimized via normal integration.  In _IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2023, Vancouver, BC, Canada, June 17-24, 2023_ , pages 512–523. IEEE, 2023.  doi: 10.1109/CVPR52729.2023.00057.  URL https://doi.org/10.1109/CVPR52729.2023.00057.
  * Xu et al. [2024a] Jiale Xu, Weihao Cheng, Yiming Gao, Xintao Wang, Shenghua Gao, and Ying Shan.  Instantmesh: Efficient 3d mesh generation from a single image with sparse-view large reconstruction models.  _arXiv preprint arXiv:2404.07191_ , 2024a.
  * Xu et al. [2023] Yinghao Xu, Hao Tan, Fujun Luan, Sai Bi, Peng Wang, Jiahao Li, Zifan Shi, Kalyan Sunkavalli, Gordon Wetzstein, Zexiang Xu, and Kai Zhang.  DMV3D: denoising multi-view diffusion using 3d large reconstruction model.  _CoRR_ , abs/2311.09217, 2023.  doi: 10.48550/ARXIV.2311.09217.  URL https://doi.org/10.48550/arXiv.2311.09217.
  * Xu et al. [2024b] Yinghao Xu, Zifan Shi, Wang Yifan, Sida Peng, Ceyuan Yang, Yujun Shen, and Wetzstein Gordon.  Grm: Large gaussian reconstruction model for efficient 3d reconstruction and generation.  _arxiv: 2403.14621_ , 2024b.
  * Xue et al. [2022] Yuxuan Xue, Haolong Li, Stefan Leutenegger, and Joerg Stueckler.  Event-based non-rigid reconstruction from contours.  In _33rd British Machine Vision Conference 2022, BMVC 2022, London, UK, November 21-24, 2022_ , page 78. BMVA Press, 2022.  URL https://bmvc2022.mpi-inf.mpg.de/78/.
  * Xue et al. [2023] Yuxuan Xue, Bharat Lal Bhatnagar, Riccardo Marin, Nikolaos Sarafianos, Yuanlu Xu, Gerard Pons-Moll, and Tony Tung.  NSF: neural surface fields for human modeling from monocular depth.  In _IEEE/CVF International Conference on Computer Vision, ICCV 2023, Paris, France, October 1-6, 2023_ , pages 15004–15014. IEEE, 2023.  doi: 10.1109/ICCV51070.2023.01382.  URL https://doi.org/10.1109/ICCV51070.2023.01382.
  * Xue et al. [2024] Yuxuan Xue, Haolong Li, Stefan Leutenegger, and Jörg Stückler.  Event-based non-rigid reconstruction of low-rank parametrized deformations from contours.  In _International Journal of Computer Vision (IJCV)_. Springer Science and Business Media LLC, February 2024.  doi: 10.1007/s11263-024-02011-z.  URL http://dx.doi.org/10.1007/s11263-024-02011-z.
  * Yang et al. [2023] Xueting Yang, Yihao Luo, Yuliang Xiu, Wei Wang, Hao Xu, and Zhaoxin Fan.  D-IF: Uncertainty-aware Human Digitization via Implicit Distribution Field.  In _2023 IEEE/CVF International Conference on Computer Vision (ICCV)_ , pages 9088–9098, Paris, France, October 2023. IEEE.  ISBN 9798350307184.  doi: 10.1109/ICCV51070.2023.00837.  URL https://ieeexplore.ieee.org/document/10377954/.
  * Youwang et al. [2024] Kim Youwang, Tae-Hyun Oh, and Gerard Pons-Moll.  Paint-it: Text-to-texture synthesis via deep convolutional texture map optimization and physically-based rendering.  In _IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2024.
  * Yu et al. [2021a] Alex Yu, Vickie Ye, Matthew Tancik, and Angjoo Kanazawa.  pixelnerf: Neural radiance fields from one or few images.  In _IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2021, virtual, June 19-25, 2021_ , pages 4578–4587. Computer Vision Foundation / IEEE, 2021a.  doi: 10.1109/CVPR46437.2021.00455.  URL https://openaccess.thecvf.com/content/CVPR2021/html/Yu_pixelNeRF_Neural_Radiance_Fields_From_One_or_Few_Images_CVPR_2021_paper.html.
  * Yu et al. [2021b] Tao Yu, Zerong Zheng, Kaiwen Guo, Pengpeng Liu, Qionghai Dai, and Yebin Liu.  Function4d: Real-time human volumetric capture from very sparse consumer rgbd sensors.  In _IEEE Conference on Computer Vision and Pattern Recognition (CVPR2021)_ , June 2021b.
  * Yu et al. [2023] Xianggang Yu, Mutian Xu, Yidan Zhang, Haolin Liu, Chongjie Ye, Yushuang Wu, Zizheng Yan, Chenming Zhu, Zhangyang Xiong, Tianyou Liang, Guanying Chen, Shuguang Cui, and Xiaoguang Han.  Mvimgnet: A large-scale dataset of multi-view images.  In _IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2023, Vancouver, BC, Canada, June 17-24, 2023_ , pages 9150–9161. IEEE, 2023.  doi: 10.1109/CVPR52729.2023.00883.  URL https://doi.org/10.1109/CVPR52729.2023.00883.
  * Yu et al. [2024] Zehao Yu, Torsten Sattler, and Andreas Geiger.  Gaussian opacity fields: Efficient high-quality compact surface reconstruction in unbounded scenes.  _arXiv:2404.10772_ , 2024.
  * Zablotskaia et al. [2019] Polina Zablotskaia, Aliaksandr Siarohin, Bo Zhao, and Leonid Sigal.  Dwnet: Dense warp-based network for pose-guided human video generation.  In _30th British Machine Vision Conference 2019, BMVC 2019, Cardiff, UK, September 9-12, 2019_ , page 51. BMVA Press, 2019.  URL https://bmvc2019.org/wp-content/uploads/papers/1039-paper.pdf.
  * Zeng et al. [2017] Andy Zeng, Shuran Song, Matthias Nießner, Matthew Fisher, Jianxiong Xiao, and Thomas Funkhouser.  3dmatch: Learning local geometric descriptors from rgb-d reconstructions.  In _CVPR_ , 2017.
  * Zhang et al. [2017] Chao Zhang, Sergi Pujades, Michael J. Black, and Gerard Pons-Moll.  Detailed, accurate, human shape estimation from clothed 3d scan sequences.  In _2017 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017, Honolulu, HI, USA, July 21-26, 2017_ , pages 5484–5493. IEEE Computer Society, 2017.  doi: 10.1109/CVPR.2017.582.  URL https://doi.org/10.1109/CVPR.2017.582.
  * Zhang et al. [2018a] Richard Zhang, Phillip Isola, Alexei A. Efros, Eli Shechtman, and Oliver Wang.  The unreasonable effectiveness of deep features as a perceptual metric.  In _2018 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2018, Salt Lake City, UT, USA, June 18-22, 2018_ , pages 586–595. Computer Vision Foundation / IEEE Computer Society, 2018a.  doi: 10.1109/CVPR.2018.00068.  URL http://openaccess.thecvf.com/content_cvpr_2018/html/Zhang_The_Unreasonable_Effectiveness_CVPR_2018_paper.html.
  * Zhang et al. [2022] Xiaohan Zhang, Bharat Lal Bhatnagar, Sebastian Starke, Vladimir Guzov, and Gerard Pons-Moll.  COUCH: towards controllable human-chair interactions.  In Shai Avidan, Gabriel J. Brostow, Moustapha Cissé, Giovanni Maria Farinella, and Tal Hassner, editors, _Computer Vision - ECCV 2022 - 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part V_ , volume 13665 of _Lecture Notes in Computer Science_ , pages 518–535. Springer, 2022.  doi: 10.1007/978-3-031-20065-6\\_30.  URL https://doi.org/10.1007/978-3-031-20065-6_30.
  * Zhang et al. [2024] Xiaohan Zhang, Bharat Lal Bhatnagar, Sebastian Starke, Ilya Petrov, Vladimir Guzov, Helisa Dhamo, Eduardo Pérez-Pellitero, and Gerard Pons-Moll.  FORCE: dataset and method for intuitive physics guided human-object interaction.  _CoRR_ , abs/2403.11237, 2024.  doi: 10.48550/ARXIV.2403.11237.  URL https://doi.org/10.48550/arXiv.2403.11237.
  * Zhang et al. [2018b] Xiuming Zhang, Zhoutong Zhang, Chengkai Zhang, Joshua B Tenenbaum, William T Freeman, and Jiajun Wu.  Learning to Reconstruct Shapes From Unseen Classes.  In _Advances in Neural Information Processing Systems (NeurIPS)_ , 2018b.
  * Zhang et al. [2023] Zechuan Zhang, Zongxin Yang, and Yi Yang.  SIFU: side-view conditioned implicit function for real-world usable clothed human reconstruction.  _CoRR_ , abs/2312.06704, 2023.  doi: 10.48550/ARXIV.2312.06704.  URL https://doi.org/10.48550/arXiv.2312.06704.
  * Zhou and Tulsiani [2023] Zhizhuo Zhou and Shubham Tulsiani.  Sparsefusion: Distilling view-conditioned diffusion for 3d reconstruction.  In _CVPR_ , 2023.
  * Zou et al. [2023] Zi-Xin Zou, Zhipeng Yu, Yuan-Chen Guo, Yangguang Li, Ding Liang, Yan-Pei Cao, and Song-Hai Zhang.  Triplane meets gaussian splatting: Fast and generalizable single-view 3d reconstruction with transformers.  _CoRR_ , abs/2312.09147, 2023.  doi: 10.48550/ARXIV.2312.09147.  URL https://doi.org/10.48550/arXiv.2312.09147.

##  Appendix

###  Appendix A Implementation Details

####  A.1 Training Details

As described in Sec. 5.1, we use an effective batch size of 256. Each batch
involved sampling 4 orthogonal images with zero elevation angle as target
views $\mathbf{x}^{\text{tgt}}_{0}$, and 12 additional images as novel views
$\mathbf{x}^{\text{novel}}_{0}$ to supervise the 3D generative model Eq. 8.
The hyperparameters for training Eq. 8 were set as follows: $\lambda_{1}=1.0$,
$\lambda_{2}=1.0$, and $\lambda_{3}=100.0$.

During training, we employed the standard DDPM scheduler [23] to construct
noisy target images $\mathbf{x}_{t}^{\text{tgt}}$. The maximum diffusion step
$T$ is set to $1000$. At inference time, we use DDIM scheduler [61] to perform
faster reverse sampling. The reverse steps is set to 50 in following
experiments. The text prompt $y$ used in our multi-view diffusion model(Eq. 3)
is set to "Photorealistic 3D human" for both training and inference across all
subjects.

####  A.2 Joint Framework

Implementation. Our 2D multi-view diffusion model $\bm{\epsilon}_{\theta}$ is
a latent diffusion model [75]. Thus, we use the frozen VAE in [53] to obtain
input $\mathbf{x}_{t}^{\text{tgt}}$ in image space for the 3D generative model
$g_{\phi}$ and encode refined $\tilde{\mathbf{x}}_{t}^{\text{tgt}}$ back to
latent space for $\bm{\epsilon}_{\theta}$. We extract triangle mesh from
predicted Gaussian splats using Gaussian Opacity field [98] and TSDF [100].
Please refer to Sec. A.4 for more details.

####  A.3 Generative 3D-GS Reconstruction Model

In this section, we provide details about our 3D generative model $g_{\phi}$
in Eq. 5 and Eq. 7 as well as the
${\color[rgb]{0.6484375,0.36328125,0.44921875}\definecolor[named]{pgfstrokecolor}{rgb}{0.6484375,0.36328125,0.44921875}\texttt{renderer}}(\circ)$.
Following [7, 33, 68], we learn the 3D generative model by adding and removing
noise on the rendered 2D images from a 3D representation. A pseudo algorithm
of the training and sampling process of our 3D generative model can be found
in Algorithm 3.

Since we integrate both function into the reverse sampling process iteratively
(eq. 2), we expect them be efficient and fast to execute. Tewari et al. [68]
base their model on pixelNeRF [95], which is a generalizable NeRF [47]
conditioned on a context view image. We adopt 3D Gaussian Splats [34] as our
3D state representation $\mathcal{G}$ due to its efficiency and simplicity.
Our
${\color[rgb]{0.6484375,0.36328125,0.44921875}\definecolor[named]{pgfstrokecolor}{rgb}{0.6484375,0.36328125,0.44921875}\texttt{renderer}}(\circ)$
is the differentiable rasteraizer accelerated and implemented in CUDA, which
achieves around $2700$ times faster rendering than volume-rendering-based
${\color[rgb]{0.6484375,0.36328125,0.44921875}\definecolor[named]{pgfstrokecolor}{rgb}{0.6484375,0.36328125,0.44921875}\texttt{renderer}}(\circ)$
in [47, 68, 95].

For sampling the 3D State $\mathbf{S}$ from $\mathbf{x}_{t}^{\text{tgt}}$,
$\tilde{\mathbf{x}}_{0}^{\text{tgt}}$, $\mathbf{x}^{\text{c}}$, and $t$ (eq.
7), we adopt the time-conditioned UNet-Transformer architecture [53] due to
the efficiency of convolutional layers and the scalability of transformers.
For enabling the awareness of camera poses in the encoding process, we
concatenate the Plücker Camera Ray Embedding
$\\{\mathbf{o}_{i}\times\mathbf{d}_{i},\mathbf{d}_{i}\\}$ [65, 88] with the
image $\mathbf{x}_{t}^{\text{tgt}}$ and $\tilde{\mathbf{x}}_{0}^{\text{tgt}}$.
To enhance the control ability of context view in the 3D generation process,
we additionally concatenate the clear context view ${\mathbf{x}}^{\text{c}}$
with target images $\mathbf{x}_{t}^{\text{tgt}}$ follow [75]. This operation
enables 3D dense self-attention process between the input multi-view target
images and the clear context view image, provides pixel-level local
conditional signal. Since the camera pose of context view $\mathbf{x}^{c}$ is
unknown, we use the $0$-vector as its embedding.

Figure 7: Visualization intermediate sampling steps from a Gaussian Noise
($t=1000$) to the last denoising step ($t=0$). From top to bottom: current
state $\mathbf{x}^{\text{tgt}}_{t}$, estimated clear view by 2D diffusion
models $\tilde{\mathbf{x}}^{\text{tgt}}_{0}$, and corrected clear view by
generated 3D Gaussian Splatting $\hat{\mathbf{x}}^{\text{tgt}}_{0}$. Our 2D
diffusion model $\bm{\epsilon}_{\phi}(\circ)$ already provides strong multi-
view prior at an early stage with large $t$. Our 3D reconstruction model
$\mathbf{g}_{\phi}(\circ)$ can correct the inconsistency in
$\tilde{\mathbf{x}}^{\text{tgt}}_{0}$ illustrated in red circle.

Algorithm 3 Learn 3D distribution

1:Dataset of posed multi-view images ${\mathbf{x}}^{\text{tgt}}_{0}$,
$\pi^{\text{tgt}}$, ${\mathbf{x}}^{\text{novel}}_{0}$, $\pi^{\text{novel}}$, a
context image ${\mathbf{x}}^{\text{c}}$

2:Optimized 3D State diffusion network $g_{\phi}$

3:repeat

4:
$\\{{\mathbf{x}}^{\text{tgt}}_{0},{\mathbf{x}}^{\text{novel}}_{0},{\mathbf{x}}^{\text{c}},y\\}\sim
q(\\{{\mathbf{x}}^{\text{tgt}}_{0},{\mathbf{x}}^{\text{novel}}_{0},{\mathbf{x}}^{\text{c}},y\\})$

5: $t\sim\mathrm{Uniform}(\\{1,\dotsc,T\\})$;
$\bm{\epsilon}\sim\mathcal{N}(\mathbf{0},\mathbf{I})$

6:
$\mathbf{x}^{\text{tgt}}_{t}=\sqrt{\bar{\alpha}_{t}}{\mathbf{x}}_{0}^{\text{tgt}}+\sqrt{1-\bar{\alpha}_{t}}\bm{\epsilon}$

7:
$\hat{\mathcal{G}}=g_{\phi}\left(\mathbf{x}^{\text{c}},\mathbf{x}_{t}^{\text{tgt}},t\right)$

8:
$\\{\hat{\mathbf{x}}_{0}^{\text{tgt}},\hat{\mathbf{x}}_{0}^{\text{novel}}\\}={\color[rgb]{0.6484375,0.36328125,0.44921875}\definecolor[named]{pgfstrokecolor}{rgb}{0.6484375,0.36328125,0.44921875}\texttt{renderer}}\left(\hat{\mathcal{G}},\\{\pi^{\text{tgt}},\pi^{\text{novel}}\\}\right)$

9: Compute loss $\mathcal{L}_{gs}$ ( Eq. 6)

10: Gradient step to update $g_{\phi}$

11:until converged

Algorithm 4 Sample from 3D distribution

1:A context image ${\mathbf{x}}^{c}$; Converged 3D diffusion model $g_{\phi}$

2:A 3D Gaussian Avatar $\mathcal{G}$ of the 2D image ${\mathbf{x}}^{c}$

3:${\mathbf{x}}_{T}^{\text{tgt}}\sim\mathcal{N}(\mathbf{0},\mathbf{I})$

4:for $t=T,\dotsc,1$ do

5:
$\hat{\mathcal{G}}=g_{\phi}\left(\mathbf{x}^{\text{c}},\mathbf{x}_{t}^{\text{tgt}},t\right)$

6:
$\hat{\mathbf{x}}_{0}^{\text{tgt}}={\color[rgb]{0.6484375,0.36328125,0.44921875}\definecolor[named]{pgfstrokecolor}{rgb}{0.6484375,0.36328125,0.44921875}\texttt{renderer}}\left(\hat{\mathcal{G}},\pi^{\text{tgt}}\right)$

7:
$\mu_{t-1}({\mathbf{x}}_{t}^{\text{tgt}},\hat{{\mathbf{x}}}_{0}^{\text{tgt}})=\frac{\sqrt{\alpha_{t}}\left(1-\bar{\alpha}_{\text{t-1}}\right)}{1-\bar{\alpha}_{t}}{\mathbf{x}}^{\text{tgt}}_{t}+\frac{\sqrt{\bar{\alpha}_{\text{t-1}}}\beta_{t}}{1-\bar{\alpha}_{t}}\hat{{\mathbf{x}}}_{0}^{\text{tgt}}$

8:
${\mathbf{x}}^{\text{tgt}}_{t-1}\sim\mathcal{N}\left(\mathbf{x}^{\text{tgt}}_{t-1};\tilde{\bm{\mu}}_{t}\left({\mathbf{x}}^{\text{tgt}}_{t},\hat{\mathbf{x}}^{\text{tgt}}_{0}\right),\tilde{\beta}_{t-1}\mathbf{I})\right)$

9:end for

10:return
$\mathcal{G}=g_{\phi}\left(\mathbf{x}_{0}^{\text{tgt}},\mathbf{x}^{\text{c}},t=0\right)$

####  A.4 Textured Mesh Extraction

Gaussian Opacity Fields [98] enables extraction of triangle meshes from an
existing 3D Gaussian Splatting. However, because the location of 3D-Gaussian
Splats is not necessary to be on the real surface, we observe that the
extracted meshes as well as the rendered depth maps are noisy. Since our
method generate realistic RGB images, we use PiFU-HD [55] to estimate the
normals and use Bilateral Normal Integration (BiNI) [10] to refine the noisy
rendered depth with the estimated normal. As we only want the estimated normal
to denoise the rendered depth map instead of modifying geometry, we set up the
hyperparameter in BiNI with $\lambda=1\times 10^{4}$. Such a large number
ensures that the normal map is not used to modify the geometry but just
regularize the depth map.

Assuming we have a generated 3D-Gaussian Splats $\mathcal{G}$ from
$g_{\phi}(\circ)$ and $n$ camera views ${\pi^{1},\pi^{2},...\pi^{n}}$, we
obtain $n$ paris of posed RGB-D images by _Gaussian Splatting_ , _Normal
Estimation_ , and _Bilateral Normal Integration_. Finally, we perform
volumetric TSDF fusion [100] to obtain high quality textured mesh from $n$
pairs RGB-D images. Given generated 3D-GS $\mathcal{G}$, we set up 36 views to
obtain the refined RGB-D image pairs. The rendering view of each camera i can
be calculated as:

$\displaystyle\text{elevation}_{i}$
$\displaystyle=-\frac{1}{4}\pi+\frac{1}{4}\pi*\frac{i}{36},$ (10)
$\displaystyle\text{azimuth}_{i}$ $\displaystyle=0+3\pi*\frac{i}{36}.$ (11)

###  Appendix B Comparison

####  B.1 Qualitative Comparison

---
Figure 8: Qualitative comparison on Sizer [70] and IIIT [30].

---
Figure 9: Qualitative comparison on IIIT [30].

###  Appendix C More Qualitative Results

In this section, we show more qualitative results on in-the-wild data, UBC
fashion dataset [99], GSO dataset [16], and human-object interaction data
[84].

####  C.1 In-the-wild Data

Figure 10: Qualitative results on unseen data during training. Input image is
in left column. Our method successfully reconstructs different degree of loose
clothing. Figure 11: Qualitative results on more unseen data during training.
Input image is in left column. Our method successfully reconstructs different
types of clothing, including casual, sport, suits, custom, etc., in both
appearance and geometry. Figure 12: Qualitative results on more unseen data
during training. Input image is in left column. Our method successfully
reconstructs clothing and interacting objects (racket and bag here) in both
appearance and geometry. Figure 13: Qualitative results on more unseen data
during training. Input image is in left column. Our method successfully
reconstructs rarely seen suits and objects, in both appearance and geometry.

####  C.2 UBC Fashion Dataset

In this section, we show qualitative result of our model on UBC fashion [99]
dataset. The input images are the first frame extracted from each video in the
dataset.

Figure 14: Qualitative results on UBC fashion [99] dataset. Results
demonstrate that our model generalizes well to real world images in both
geometry and appearance.

####  C.3 Google Scan Objects (GSO)

Figure 15: Ablation study: benefit of 2D multi-view prior
$\tilde{\mathbf{x}}_{0}^{\text{tgt}}$ in 3D generation. The 2D prior from 2D
diffusion model is essntial for generalization on general objects dataset GSO
[16].

####  C.4 Human Object Interaction

Figure 16: Qualitative results of Human-Object Interaction reconstruction on
online stock images. Results show that our model is able to generalize to
casual human-object-interactions.  Figure 17: Qualitative results of Human-
Object Interaction reconstruction on ProciGen [84] dataset. Results
demonstrate that our model can reconstruct some simple Human-Object-
Interaction images with large objects.

####  C.5 Generative Power in Reconstruction

Our model learns a conditional distribution of the 3D representation given 2D
context image. Thus, by sampling from the distribution with different seed, we
obtain diverse yet plausible 3D representation. As illustrated in Fig. 18, the
appearance of the occluded region (back side of subject) is different with
different sampling in hair styple, texture, and cloth wrinkles.

The generative power of our approach is the key to generate clear self-
occluded regions, which is impossible by non-generative reconstruction methods
[54, 55, 72, 106]. As shown in Fig. 3 and Fig. 8, non-generative approaches
tend to generate blurry self-occluded results because they cannot sample from
distribution but only regress to a mean value of the training datasets.

Figure 18: Our model learns 3D distribution. By different sampling from the
learned distribution, we obtain diverse yet plausible 3D representations. The
generative power is a key to generate clear self-occluded region, which is
impossible in non-generative reconstruction approaches [54, 55, 72, 106].

###  Appendix D Dataset Overview

To ensure robust performance and generalization, we train our model on a
combined dataset comprising 3520 scans from publicly available datasets [19,
25, 63, 96] and 2320 scans from commercial 3D human datasets [1, 3, 4, 2].
These datasets encompass a diverse range of body shapes, genders, ages,
clothing, accessories, and interacting objects. For further details and
examples of our training datasets, please refer to section D.1. All 3D scans
are rendered into RGB-A images using BlenderProc [13] along a spiral path as
described in Eq. 13.
For evaluation, we note that the commonly-used CAPE dataset [45, 50, 101] in
previous works [21, 85, 86, 106] often contains artifacts in scans, such as
holes, and not all 3D scans are fully publicly available. To effectively and
fairly evaluate performance, we propose using Sizer [70] and IIIT-Human [30]
datasets, from which we randomly sample 150 scans each for evaluation. While
Sizer [70] provides scans with normal human appearance similar to our training
datasets, IIIT-Human [30] can be considered as out-of-distribution (o.o.d.)
evaluation dataset due to its inclusion of unseen clothing types, such as
traditional Indian suits. For more additional examples and analysis, please
see Fig. 3 and Sec. D.2.

####  D.1 Training Dataset

##### Datasets

To prevent the overfitting of our large neural network, namely the 2D multi-
view diffusion models $\bm{\epsilon}_{\theta}(\circ)$ and 3D generative models
$\mathbf{g}_{\phi}(\circ)$, we train on as much data as we can. Unlike general
objects community which has massive dataset such as Objaverse (800K) and
Objaverse-XL (10M) [12]) OmniObject3D (6K) [80], MVImageNet (87K) [97], we
don’t have a single 3D human dataset available at such a scale. To collect
data as much as possible, we collect both following public datasets and
commercial human scans.

We collect several publicly available datasets inluding 2k2k (2K) [19],
CustomHuman (640) [25], Thuman2.0 (520) [96], and Thuman3.0 (360) [63]). Among
them, CustomHuman, Thuman2.0, and Thuman3.0 have more repeating subjects with
different poses, which have less diverse subject appearance compared to 2k2k.
It is worth mentioning that 2k2k [19] is a high quality dataset which contains
human with diverse clothing (such as skirt) and accessories (such as cap, hat,
scarf).

We also utilize in total 2320 high quality commercial scans from AXYZ [1],
Treedy [3], Twindom [4], and RenderPeople [2]. All of these scans are with
casual clothing and without interaction with objects.

##### Rendering

For each scan, we render 100 views following a spiral trajectory with each
view $i$:

$\displaystyle\text{elevation}_{i}$
$\displaystyle=-\frac{1}{4}\pi+\frac{7}{8}\pi*\frac{i}{100},$ (12)
$\displaystyle\text{azimuth}_{i}$ $\displaystyle=0+5\pi*\frac{i}{100}.$ (13)

Additionally, we render 32 views uniformly around z-axis with each view $j$:

$\displaystyle\text{elevation}_{j}$ $\displaystyle=0,$ (14)
$\displaystyle\text{azimuth}_{j}$ $\displaystyle=0+\pi*\frac{j}{32}.$ (15)

To protect the privacy of subjects in the training dataset, we only use the
frontal view (with $\text{azimuth}_{j}\in[-\frac{\pi}{2},\frac{\pi}{2}]$) as
the input context view during training. Thus, we expect the model will not
learn faces of subjects when it takes the back view as input.

####  D.2 Evaluation Dataset

For quantitative evaluation, we use Sizer [70] and IIIT 3D human dataset [30].
In this section, we start with introducing the two used evaluation datasets,
and explain why we omit the commonly used CAPE dataset [45, 50, 101] in our
experiments. Finally, we provide a summary of the evaluation datasets.

##### Sizer

Sizer [70] is a high quality 3D human scan dataset which contains 100
different subjects wearing casual clothing items in various sizes. We randomly
sample 150 scans from Sizer [70] as one of our evaluation dataset.

##### IIIT

IIIT 3D Humans [30] is a high quality dataset from IIIT Hyderabad in India.
Different from the casual clothing setup in Sizer [70], IIIT dataset mainly
focuses on subjects wearing traditional India custom suits, including
ethnicity, diverse color pattern and extremely loose clothing (Fig. 21). It
brings the huge variety of the subject appearance which can be considered as
o.o.d. evaluation set to our model and baselines. We randomly sample 150 scans
from IIIT dataset [30] for evaluation.

##### Why not CAPE

Unlike previous methods [21, 85, 86, 106] which evaluate on CAPE dataset [45,
50, 101], we omitted CAPE in our evaluation mainly because of the limitation
in appearance variety, geometrical artifacts as well as the publicly
unavailability. CAPE only contains simple clothing such as T-shirts and jeans,
but no garments of loose clothing. As illustrated in Fig. 22, CAPE contains
several artifacts, such as holes on the head, missing hands, and wrong mesh
geometry. Moreover, CAPE doesn’t have most original scans publicly available,
but only the SMPL+D fitting. Due to this reason, we cannot render the CAPE
scan to RGB images at desired camera view to evaluate the appearance
performance such as PSNR, SSIM, and LPIPS.

##### Summary

We observe that the high quality datasets Sizer [70] and IIIT 3D Human [30]
are unexplored for the community of 3D avatars reconstruction. In fact, Sizer
[70] contains casual clothing which is suitable to evaluate performance, and
IIIT [30] contains challenging texture and loose clothing which is suitable to
evaluate robustness. All high quality scans in [30, 70] have no severe
artifacts and are fully publicly available, which are the benefits unprovided
in CAPE dataset [45, 50, 101]. By evaluating on these datasets [30, 70] and
release our randomly sampled subjects which are used in our experiments, we
hope the 3D avatars community can discover and benefit from them.

Figure 19: Example scans in training datasets [1, 2, 3, 4, 19, 25, 63, 96].
Figure 20: Example scans in Sizer [70] dataset. Sizer contains human in casual
clothing. Figure 21: Example scans in IIIT [30] dataset. IIIT contains
subjects with diverse color pattern and loose garments, which rarely appear in
training datasets. Figure 22: Example artifacts in CAPE [45, 50, 101] dataset.
Images shown here are rendered by ICON [85] due to the inaccessibility of
original scans.

###  Appendix E Failure Cases

Limited by low resolution ($256\times 256$) of our multi-view diffusion model
[75], our model can often fail in reconstructing fine details such as text on
the cloth as illustrated in Fig. 23. One potential solution is to switch to a
recent powerful high-resolution multi-view diffusion models [17, 66].

Figure 23: Failure Case: our model cannot reconstruct the numbers on the
cloth.

In addition, we observe that our model can fail when reconstructing human
extremely challenging poses. As shown in Fig. 24, our model cannot infer head
geometry and appearance accurately due to the challenging pose in input image.

Figure 24: Failure Case: our model fails in infer appearance of human with
challenging pose.

###  Appendix F Broader Impacts

Our work shows generality across different ethnicities and humans, providing a
useful tool for a fair representation of different cultures. Having a robust
method to synthesize realistic 3D geometry from a single RGB image may be used
in surveillance and inappropriate content generation.