# Weak limit of $W^{1,2}$ homeomorphisms in $\mathbb{R}^{3}$ can have any
degree

Ondřej Bouchala  Czech Technical University in Prague, Faculty of Information
Technology, Thákurova 9, 160 00 Prague 6, Czech Republic
<EMAIL_ADDRESS>,  Stanislav Hencl  Department of Mathematical
Analysis, Charles University, Sokolovská 83, 186 00 Prague 8, Czech Republic
<EMAIL_ADDRESS>and  Zheng Zhu  School of Mathematical Sciences
Beihang University
Changping District Shahe Higher Education Park South Third Street No. 9
Beijing 102206
P. R. China<EMAIL_ADDRESS>
(Date: September 2023)

###### Abstract.

In this paper for every $k\in\mathbb{Z}$ we construct a sequence of weakly
converging homeomorphisms $h_{m}\colon B(0,10)\to\mathbb{R}^{3}$,
$h_{m}\rightharpoonup h$ in $W^{1,2}(B(0,10))$, such that $h_{m}(x)=x$ on
$\partial B(0,10)$ and for every
$r\in\left(\tfrac{5}{16},\tfrac{7}{16}\right)$ the degree of $h$ with respect
to the ball $B(0,r)$ is equal to $k$ on a set of positive measure.

###### Key words and phrases:

limits of Sobolev homeomorphisms, topological degree

The first two authors were supported by the grant GAČR P201/21-01976S. The
third author was supported by the NSFC grant no. 12301111. This research was
done when Z. Zhu was visiting the Department of Mathematical Analysis, Faculty
of Mathematics and Physics, Charles University. He wishes to thank Charles
University for its hospitality.

##  1\. Introduction

In this paper, we study classes of mappings that appear naturally in the
modeling of deformations in Continuum Mechanics models. Let
$\Omega\subseteq\mathbb{R}^{3}$ be a domain, i.e. a non-empty connected open
set, and let $h\colon\Omega\to\mathbb{R}^{3}$ be a mapping. Following the
pioneering papers of Ball [1] and Ciarlet and Nečas [7] we ask if our mapping
is in some sense injective, as the physical “non-interpenetration of the
matter” indicates that a deformation should be one-to-one.

One of the crucial tools in the study of the injectivity of such a mapping (if
it is continuous) is the use of topological degree. Let $B\subseteq\Omega$ be
a ball, let $h$ be continuous and $y\notin h(\partial B)$. Informally speaking
the topological degree $\deg(h,B,y)$ is the number of preimages of $y$ under
$h$ in $B$ taking the orientation into account (see Preliminaries for more
about the degree). For example the topological degree of a homeomorphism is
always $0$ or $1$ (if $y\notin h(B)$ or $y\in h(B)$) or always $-1$ or $0$ (if
$h$ is reversing the orientation), see e.g. [14] or Proposition 2.1.

The topological degree of a continuous mapping depends only on the boundary
mapping $h|_{\partial B}$, i.e. we can extend it in any continuous way inside
and we get the same value of $\deg(h,B,y)$. Given $h\in W^{1,p}$, $p>2$, we
know that $h|_{\partial B(c,r)}\in W^{1,p}$ for almost all $r$, and hence $h$
has a continuous representative there (as $p>2$ and $2$ is the dimension of
the sphere) and we can thus define $\deg(h,B(x,r),y)$. This idea was used by
Müller and Spector [18] (see also e.g. [2, 13, 12, 19, 20, 21, 22]) to define
and study the class of mappings in $W^{1,p}(\Omega,\mathbb{R}^{3}),\ p>2$,
that satisfy the so called $\mathrm{(INV)}$ condition. Informally speaking,
the $\mathrm{(INV)}$ condition means that the ball $B(c,r)$ is mapped inside
the image of the sphere $h(S(c,r))$ and the complement
$\Omega\setminus\overline{B(c,r)}$ is mapped outside $h(S(c,r))$ (see [18] for
the formal definition). From [18] we know that mappings in this class with
$J_{h}>0$ a.e. have many desirable properties: they are one-to-one a.e., map
disjoint balls into essentially disjoint balls, $\deg(h,B,\cdot)\in\\{0,1\\}$
for a.e. ball $B$, this class is weakly closed and so on.

In all results in the previous paragraph, the authors assume that $h\in
W^{1,p}(\Omega,\mathbb{R}^{3})$ for some $p>2$. However, in some real models
for $n=3$, one often works with integrands containing the classical Dirichlet
term $|Df|^{2}$ and thus this assumption is too strong. Therefore, for $n=3$,
Conti and De Lellis [8] introduced the concept of $\mathrm{(INV)}$ condition
also for $W^{1,2}\cap L^{\infty}$ (see also [4], [3], [10] and [11] for some
recent work). Note that mappings in $W^{1,2}|_{\partial B(c,r)}$ do not need
to be continuous, but we can still define some notion of degree there using
the ideas of Brezis and Nirenberg [6], i.e. we can still define what “inside
$h(\partial B(c,r))$” is.

Mappings satisfying the $\mathrm{(INV)}$ condition in $W^{1,2}\cap L^{\infty}$
(as defined in [8]) have still all desirable properties but unfortunately this
class is not weakly closed and therefore not suitable for the approach of
Calculus of Variations. Conti and De Lellis [8] constructed a sequence of bi-
Lipschitz mappings $u_{m}$ in $\mathbb{R}^{3}$ (which can be easily extended
onto $B(0,10)$ with $u_{m}(x)=x$ on $\partial B(0,10)$) such that their weak
limit $u$ does not satisfy the $\mathrm{(INV)}$ condition as the degree of the
limit is $-1$ on a set of positive measure. This somehow means that the weak
limit of $W^{1,2}\cap L^{\infty}$ sense-preserving homeomorphisms can in some
sense “revert its orientation” or map something that was inside the ball
outside of the image of its sphere.

The main aim of this paper is to study if the weak limit of $W^{1,2}\cap
L^{\infty}$ homeomorphisms can have even more pathological behavior, i.e. if
the degree could be any $k\in\mathbb{Z}$. Our example shows that this is
indeed possible, i.e. the weak limit of $W^{1,2}\cap L^{\infty}$
homeomorphisms can somehow “fold itself $k$-times”.

######  Theorem 1.1.

Let $k\in\mathbb{Z}$. There exists a sequence of $W^{1,2}$-homeomorphisms
$\\{h_{m}\\}$ from $\overline{B(0,10)}\subseteq\mathbb{R}^{3}$ onto
$\overline{B(0,10)}\subseteq\mathbb{R}^{3}$ such that:

  1. (i)

$h_{m}$ weakly converges to $h$ in $W^{1,2}(B(0,10),\mathbb{R}^{3})$.

  2. (ii)

$h\big{|}_{\partial B(0,10)}$ and $h_{m}\big{|}_{\partial B(0,10)}$ are
identity for every $m\in\mathbb{N}$.

  3. (iii)

There exists a set $A\subseteq B(0,10)$ with positive measure such that for
every $y\in A$ we have

$\operatorname{Deg}\left(h,\ B(0,r),\ y\right)=k$

for every $r\in\left(\frac{5}{16},\frac{7}{16}\right)$.

Such an example of a weak limit of homeomorphisms cannot be constructed in
$W^{1,p}$, $p>2$: homeomorphisms clearly satisfy the $\mathrm{(INV)}$
condition and so do their weak limits in $W^{1,p}$, $p>2$, (see [18, Lemma
3.3]) and mapping in the $\mathrm{(INV)}$ class with $J_{h}>0$ a.e. have
$\deg(h,B,\cdot)\in\\{0,1\\}$ for a.e. ball $B$ (see [18, Lemma 3.5]). Also
(under some minor conditions) such an example of a strong limit of
homeomorphisms cannot be constructed in $W^{1,2}$, since the strong limit of
$W^{1,2}$ \- homeomorphism also satisfies the $\mathrm{(INV)}$ condition,
please see [10, Theorem 3.1(b)].

To give the reader some idea about the construction of mapping of Conti and De
Lellis we show in Fig. 1 (everything is radially symmetric around the
$x$-axis) how the image of single problematic sphere $\partial B(0,r)$ by
$u_{m}$ and limiting $u$ in the limit looks like.

$\partial B(0,r)$$C_{m,r}$$u_{m}$$\deg(u_{m},B(0,r),y)=0$$u_{m}(\partial
B(0,r))$$m\to\infty$$\deg(u,B(0,r),y)=-1$$u(\partial B(0,r))$ Figure 1.
Construction of Conti and De Lellis - everything is radially symmetric.
Vectors denote the orientation of the sphere and its image.

The image of a spherical cap $C_{m,r}\subseteq\partial B(0,r)$ (of size
roughly $1/m$) by $u_{m}$ is the outer “dashed” bubble (of size roughly $1$)
and this bubble disappears in the limit as
$\operatorname{diam}(C_{m,r})\overset{m\to\infty}{\to}0$. The limiting mapping
has a degree equal to $-1$ inside the inner bubble as the orientation of the
bubble is reversed with respect to the original ball. The derivative of
$h_{m}$ on $C_{m,r}$ is comparable to

$\int_{C_{m,r}}|Dh_{m}|^{2}\approx\mathcal{H}^{2}(C_{m,r})\left|\frac{1}{\tfrac{1}{m}}\right|^{2}\approx\frac{1}{m^{2}}|m|^{2}=1$

and thus it remains uniformly bounded in $m$ even when we integrate it over
$r$ and thus $f_{m}$ have weak limit in $W^{1,2}$. Of course, one needs to
extend it as a homeomorphism on all neighboring spheres, make it identity on
$\partial B(0,10)$, estimate derivatives in all directions but the essential
idea is in the figure. To obtain degree $k=2$ in our example, we need to
achieve something like the following behavior for the limit mapping $h$, see
Fig. 2 (again everything is rotated around the $x$-axis), which is more
complicated. We need more than one spherical cap with strange behavior and our
$h_{m}$ are not radially symmetric.

$\deg(h,B(0,r),y)=2$$\deg(h,B(0,r),y)=1$$\deg(h,B(0,r),y)=0$$h(\partial
B(0,r))$$h(\partial B(0,r))$ Figure 2. Our construction of limit mappings -
everything is radially symmetric.

Let us also note that the class of weak limits of Sobolev homeomorphisms was
recently characterized in the planar case by Iwaniec and Onninen [15, 16] and
De Philippis and Pratelli [9]. For some other kind of pathological examples of
limits of $W^{1,2}\cap L^{\infty}$ homeomorphisms where the limit fails to be
one to one a.e., we refer the reader to Bouchala, Hencl and Molchanova [5].
For other interesting examples of non-injective mappings see [17].

##  2\. Preliminaries

We use $B(x,r)$ for a ball in $\mathbb{R}^{3}$ with the center
$x\in\mathbb{R}^{3}$ and with the radius $r>0$. By $\operatorname{int}(A)$ we
denote the interior of the set $A$. Throughout the paper, $C$ will be a
generic positive constant, which may even be different in a single string of
estimates.

###  2.1. Degree for continuous mappings

Let $\Omega\subseteq\mathbb{R}^{3}$ be a bounded open set. Given a continuous
map $f\colon\overline{\Omega}\to\mathbb{R}^{3}$ and
$y\in\mathbb{R}^{3}\setminus f(\partial\Omega)$, we can define the topological
degree as

$\deg(f,\ \Omega,\ y)=\sum_{x\in\Omega\cap
f^{-1}(y)}\operatorname{sgn}(J_{f}(x))$

if $f$ is smooth in $\Omega$ and $J_{f}(x)\neq 0$ for each $x\in\Omega\cap
f^{-1}(y)$. By uniform approximation, this definition can be extended to an
arbitrary continuous mapping $f\colon\overline{\Omega}\to\mathbb{R}^{3}$. Note
that the degree depends only on values of $f$ on $\partial\Omega$.

If $f\colon\overline{\Omega}\to\mathbb{R}^{n}$ is a homeomorphism, then either
$\deg(f,\Omega,y)=1$ for all $y\in f(\Omega)$ ($f$ is sense preserving), or
$\deg(f,\Omega,y)=-1$ for all $y\in f(\Omega)$ ($f$ is sense reversing). If,
in addition, $f\in W^{1,n-1}(\Omega,\mathbb{R}^{n})$, then this topological
orientation corresponds to the sign of the Jacobian. More precisely, we have

######  Proposition 2.1 ([14]).

Let $f\in W^{1,n-1}(\Omega,\mathbb{R}^{n})$ be a homeomorphism on
$\overline{\Omega}$ with $J_{f}>0$ a.e. Then

$\deg(f,\ \Omega,\ y)=1,\qquad y\in f(\Omega).$

###  2.2. Degree for $W^{1,2}\cap L^{\infty}$ mappings

Let $B$ be a ball, $f\in W^{1,2}(\partial
B,\mathbb{R}^{3})\cap\mathcal{C}(\partial B,\mathbb{R}^{3})$, $|f(\partial
B)|=0$, and $\mathbf{u}\in\mathcal{C}^{1}(\mathbb{R}^{3},\mathbb{R}^{3})$,
then (see [18, Proposition 2.1])

$\int_{\mathbb{R}^{3}}\deg(f,B,y)\operatorname{div}\mathbf{u}(y)\,\mathrm{d}y=\int_{\partial
B}(\mathbf{u}\circ
f)\cdot(\Lambda_{2}D_{\tau}f)\nu\,\mathrm{d}\mathcal{H}^{2},$ (2.1)

where $D_{\tau}f$ denotes the tangential gradient and $\Lambda_{2}D_{\tau}f$
is the restriction of $\operatorname{cof}Df$ to the corresponding subspace
(see [10] for details).

Following [8] (see also [6]) we need a more general version of the degree
which works for mappings in $W^{1,2}\cap L^{\infty}$ that are not necessarily
continuous.

######  Definition 2.2.

Let $B\subseteq\mathbb{R}^{3}$ be a ball and let $f\in W^{1,2}(\partial
B,\mathbb{R}^{3})\cap L^{\infty}(\partial B,\mathbb{R}^{3})$. Then we define
$\operatorname{Deg}(f,B,\cdot)$ as the distribution satisfying

$\int_{\mathbb{R}^{3}}\operatorname{Deg}(f,B,y)\psi(y)\,\mathrm{d}y=\int_{\partial
B}(\mathbf{u}\circ
f)\cdot(\Lambda_{2}D_{\tau}f)\nu\,\mathrm{d}\mathcal{H}^{2}$ (2.2)

for every test function $\psi\in\mathcal{C}_{c}^{\infty}(\mathbb{R}^{3})$ and
every $\mathcal{C}^{\infty}$ vector field $\mathbf{u}$ on $\mathbb{R}^{3}$
satisfying $\operatorname{div}\mathbf{u}=\psi$.

As in [8] (see also [10]) it can be verified that the right-hand side does not
depend on the way $\psi$ is expressed as $\operatorname{div}\mathbf{u}$ and
that the distribution $\operatorname{Deg}(f,B,\cdot)$ can be represented as a
$BV$ function.

######  Remark 2.3.

Let $B$ be a ball and $f\in W^{1,2}(\partial
B,\mathbb{R}^{3})\cap\mathcal{C}(\overline{B},\mathbb{R}^{3})$. If
$|f(\partial B)|=0$, then $\operatorname{Deg}(f,B,y)=\deg(f,B,y)$ for a.e.
$y\in\mathbb{R}^{3}$.

###  2.3. Matrix of derivatives in different coordinates

Let $(x_{1},r,\varphi)$ denote the usual cylindrical coordinates in
$\mathbb{R}^{3}$ and let $a\colon\mathbb{R}^{3}\to\mathbb{R}^{3}$ be a mapping
from cylindrical to cylindrical coordinates, i.e.

$a(x_{1},r,\varphi)=(a^{x_{1}}(x_{1},r,\varphi),a^{r}(x_{1},r,\varphi),a^{\varphi}(x_{1},r,\varphi))$

It is well-known that the matrix of derivatives of $a$ in this coordinate
system is

$Da(x_{1},r,\varphi)=\begin{pmatrix}\frac{\partial a^{x_{1}}}{\partial
x_{1}}&\frac{\partial a^{x_{1}}}{\partial r}&\frac{1}{r}{\cdot}\frac{\partial
a^{x_{1}}}{\partial\varphi}\\\\[5.69054pt] \frac{\partial a^{r}}{\partial
x_{1}}&\frac{\partial a^{r}}{\partial r}&\frac{1}{r}{\cdot}\frac{\partial
a^{r}}{\partial\varphi}\\\\[5.69054pt] \raise
0.0pt\hbox{\scalebox{0.75}{\lower 0.0pt\hbox{$a^{r}$}}}{\cdot}\frac{\partial
a^{\varphi}}{\partial x_{1}}&\raise 0.0pt\hbox{\scalebox{0.75}{\lower
0.0pt\hbox{$a^{r}$}}}{\cdot}\frac{\partial a^{\varphi}}{\partial
r}&\frac{a^{r}}{r}{\cdot}\frac{\partial
a^{\varphi}}{\partial\varphi}\end{pmatrix}.$ (2.3)

Let $(r,\theta,\varphi)$ denote the usual spherical coordinates in
$\mathbb{R}^{3}$, i.e. $\varphi\in(0,2\pi)$ and ${\theta\in(0,\pi)}$. Let
$b\colon\mathbb{R}^{3}\to\mathbb{R}^{3}$ be a mapping from cylindrical to
spherical coordinates, i.e.

$b(x_{1},r,\varphi)=(b^{r}(x_{1},r,\varphi),b^{\theta}(x_{1},r,\varphi),b^{\varphi}(x_{1},r,\varphi)).$

It is well-known that the matrix of derivatives of $a$ in this coordinate
system is

$Db(x_{1},r,\varphi)=\begin{pmatrix}\frac{\partial b^{r}}{\partial
x_{1}}&\frac{\partial b^{r}}{\partial r}&\frac{1}{r}{\cdot}\frac{\partial
b^{r}}{\partial\varphi}\\\\[5.69054pt] \raise
0.0pt\hbox{\scalebox{0.75}{\lower 0.0pt\hbox{$b^{r}$}}}{\cdot}\frac{\partial
b^{\theta}}{\partial x_{1}}&\raise 0.0pt\hbox{\scalebox{0.75}{\lower
0.0pt\hbox{$b^{r}$}}}{\cdot}\frac{\partial b^{\theta}}{\partial
r}&\frac{b^{r}}{r}{\cdot}\frac{\partial
b^{\theta}}{\partial\varphi}\\\\[5.69054pt] \raise
0.0pt\hbox{\scalebox{0.75}{\lower 0.0pt\hbox{$b^{r}\sin
b^{\theta}$}}}{\cdot}\frac{\partial b^{\varphi}}{\partial x_{1}}&\raise
0.0pt\hbox{\scalebox{0.75}{\lower 0.0pt\hbox{$b^{r}\sin
b^{\theta}$}}}{\cdot}\frac{\partial b^{\varphi}}{\partial r}&\frac{b^{r}\sin
b^{\theta}}{r}{\cdot}\frac{\partial
b^{\varphi}}{\partial\varphi}\end{pmatrix}.$ (2.4)

##  3\. Proof of main theorem for $k=2$

###  3.1. Overview

In this subsection we roughly explain the construction of the mappings $h$ and
$h_{m}$ from the Theorem 1.1. To create degree two somewhere we need to go
around that area twice (imagine the planar case). To achieve that for the
limit function $h$, we will define $h_{m}$ to go around three times, twice in
the positive and once in the negative direction (or orientation). Using the
idea and the mapping $u_{m}$ from [8, Section 6] we can create such loops (or
bubbles). And by preparing the set before applying the mapping $u_{m}$ we can
control which bubbles will disappear in the limit.

The mappings $h_{m}$ will be the composition of three homeomorphisms,

$h_{m}(x):=u_{m}\circ l_{m}\circ b(x).$

The Figure 3 roughly explains what these three homeomorphisms do.

$b$$l_{m}$$u_{m}$ Figure 3. Overview of the construction.

The mapping $b$ is bi-Lipschitz and it does not depend on $m$. It moves the
dashed part of the circle/sphere (that will not disappear) and the dotted part
(which will disappear) close together so that the image of the dashed bubble
will be inside the image of the dotted one at the end.

The mappings $l_{m}$ are Lipschitz, and they squash the dashed part (of size
comparable to $1$) to something smaller (of size comparable to
$\tfrac{1}{m}$). In this way, the dashed arc does not disappear in the limit
(as it was big at the start) even after applying the Conti – De Lellis mapping
$u_{m}$. After this careful preparation, we can take the mapping $u_{m}$
exactly as in the paper [8]. Of course, it would be possible to use also a
mapping from [10, Theorem 1.2].

###  3.2. Definition of $b$

The mapping $b$ is an identity on the set $K$, where

$K:=\left\\{(x_{1},x_{2},x_{3})\in B(0,10):\frac{2}{8}<x_{1}<\frac{4}{8},\
0\leq\sqrt{x_{2}^{2}+x_{3}^{2}}<\frac{1}{8}\right\\}.\\\ $

We define for every $m\in\mathbb{N}$ (see Fig. 4)

$C_{m}:=\left\\{(x_{1},x_{2},x_{3})\in B(0,10):\frac{2}{8}<x_{2}<\frac{4}{8},\
0\leq\sqrt{x_{1}^{2}+x_{3}^{2}}<\frac{1}{8m}\right\\}.$

On cylinder ${C_{1}}$ the mapping $b$ is defined by

$b(x_{1},x_{2},x_{3})=\left(-x_{2}+\tfrac{9}{8},x_{1},x_{3}\right).$

That means that on $C_{1}$ the mapping $b$ is just translation and rotation,
see Figure 4. For every $m\in\mathbb{N}$ it maps $C_{m}$ onto the set

$b({C_{m}})=\left\\{(x_{1},x_{2},x_{3})\in
B(0,10):\frac{5}{8}<x_{1}<\frac{7}{8},\
0\leq\sqrt{x_{2}^{2}+x_{3}^{2}}<\frac{1}{8m}\right\\}.$

$x_{2}$$x_{1}$$\tfrac{2}{8}$$\tfrac{4}{8}$$\tfrac{1}{8}$$\tfrac{2}{8}$$\tfrac{4}{8}$$-\tfrac{1}{8}$$C_{1}$$C_{1}$$K$$K$$b$$x_{2}$$x_{1}$$\tfrac{2}{8}$$\tfrac{4}{8}$$\tfrac{5}{8}$$\tfrac{7}{8}$$\tfrac{1}{8}$$-\tfrac{1}{8}$$b(K)$$b(K)$$b(C_{1})$$b(C_{1})$
Figure 4. The mapping $b$ in the plane $\\{x_{1},x_{2}\\}$.

The Jacobian matrix of $b$ for $x\in C_{1}$ is

$Db(x)=\begin{pmatrix}0&-1&0\\\ 1&0&0\\\ 0&0&1\end{pmatrix}.$

It is not difficult to see that we can extend $b$ to be a bi-Lipschitz
homeomorphism which maps $\overline{B(0,10)}$ onto $\overline{B(0,10)}$ and
which is the identity on the boundary, for example by defining it by hand. We
do not give the exact formula here because it would be too lengthy and
confusing.

Later we add additional requirements about the behavior on spheres near
$\partial B(0,\tfrac{3}{8})$ so that their images do not cross some
"forbidden" regions but this can be clearly satisfied as well.

###  3.3. Definition of $l_{m}$

First, we define a thick cylinder (containing the sets $K=b(K)$ and $b(C_{m})$
from earlier) by setting

$D_{0}:=\left\\{(x_{1},x_{2},x_{3})\in
B(0,10):\tfrac{1}{8}<x_{1}<\tfrac{7}{8},\
0\leq\sqrt{x_{2}^{2}+x_{3}^{2}}<\tfrac{2}{8}\right\\}.$

The mapping $l_{m}$ is the identity outside of this cylinder. For every
$m\in\mathbb{N}$, we define a slim cylinder by setting

$D_{m}:=\left\\{(x_{1},x_{2},x_{3})\in
B(0,10):\tfrac{1}{8}<x_{1}<\tfrac{7}{8},\
0\leq\sqrt{x_{2}^{2}+x_{3}^{2}}<\frac{1}{8m}\right\\},$

and we define two cones $T_{m}^{L}$ and $T_{m}^{R}$ by setting (see Fig. 5)

$\displaystyle T_{m}^{L}$ $\displaystyle:=\left\\{(x_{1},x_{2},x_{3})\in
B(0,10):\tfrac{1}{8}\leq x_{1}\leq\tfrac{2}{8},\
0\leq\sqrt{x_{2}^{2}+x_{3}^{2}}<\left(1-\tfrac{1}{m}\right)x_{1}+\tfrac{2-m}{8m}\right\\},$
$\displaystyle T_{m}^{R}$ $\displaystyle:=\left\\{(x_{1},x_{2},x_{3})\in
B(0,10):\tfrac{4}{8}\leq x_{1}<\tfrac{5}{8},\
0\leq\sqrt{x_{2}^{2}+x_{3}^{2}}<\left(\tfrac{1}{m}-1\right)x_{1}-\tfrac{5m-4}{8m}\right\\}.$

Then, we define a “bonbon-like” domain $B_{m}$ by setting

$B_{m}:=T_{m}^{L}\cup b(K)\cup T_{m}^{R}\cup b(C_{m}).$

$-\tfrac{1}{8m}$$x_{2}$$x_{1}$$\tfrac{2}{8}$$\tfrac{7}{8}$$\tfrac{1}{8}$$T^{L}_{m}$$T^{L}_{m}$$b(K)=K$$b(K)=K$$T^{R}_{m}$$T^{R}_{m}$$b(C_{m})$$b(C_{m})$$\tfrac{4}{8}$$\tfrac{5}{8}$$\tfrac{7}{8}$$\tfrac{1}{8}$
Figure 5. The set $B_{m}$ in the plane $\\{x_{1},x_{2}\\}$.

The mapping $l_{m}$ squeezes $B_{m}$ into the slim cylinder $D_{m}$ and
stretches the set $D_{0}\setminus B_{m}$ onto $D_{0}\setminus D_{m}$.

To be precise, we define the homeomorphism $l_{m}$ inside $D_{0}$ using the
cylindrical coordinates $(x_{1},r,\varphi)$, that is
$(x_{1},x_{2},x_{3})=(x_{1},r\cos(\varphi),r\sin(\varphi))$:

The definition of $l_{m}$ for $x=(x_{1},r,\varphi)\in B_{m}$ is

$l_{m}(x):=\begin{cases}\left(x_{1},\ \frac{r}{(8m-8)x_{1}+2-m},\
\varphi\right),&x\in T_{m}^{L},\\\\[5.69046pt] \left(x_{1},\ \tfrac{r}{m},\
\varphi\right),&x\in b(K),\\\\[5.69046pt] \left(x_{1},\
\frac{r}{(8-8m)x_{1}+5m-4},\ \varphi\right),&x\in T_{m}^{R},\\\\[5.69046pt]
x,&x\in b(C_{m}).\end{cases}$

On the set $D_{0}\setminus{B_{m}}$ we define $l_{m}$ to be linear, with
respect to the radius $r$ (and mapping $x_{1}$ to $x_{1}$ and $\varphi$ to
$\varphi$), such that it maps the set $D_{0}\setminus B_{m}$ onto the
“annulus” $D_{0}\setminus D_{m}$ . It is not difficult to see that $l_{m}$ is
Lipschitz on $D_{0}\setminus B_{m}$.

Next, we compute the matrix of derivatives of $l_{m}$. Obviously, this matrix
is the identity on $C_{m}$. The matrix of derivatives of $l_{m}$ on
$T_{m}^{L}$, with respect to the cylindrical coordinates $(x_{1},r,\varphi)$,
is (see (2.3))

$Dl_{m}(x)=\begin{pmatrix}1&0&0\\\
\frac{(8-8m)r}{\left((8m-8)x_{1}+2-m\right)^{2}}&\frac{1}{(8m-8)x_{1}+2-m}&0\\\
0&0&\tfrac{1}{(8m-8)x_{1}+2-m}\end{pmatrix}.$ (3.1)

The matrix of derivatives of $l_{m}$ on $b(K)$ with respect to the cylindrical
coordinates is

$Dl_{m}(x)=\begin{pmatrix}1&0&0\\\ 0&\tfrac{1}{m}&0\\\
0&0&\tfrac{1}{m}\end{pmatrix}.$ (3.2)

And on $T_{m}^{R}$ the matrix of derivatives w.r.t. the cylindrical
coordinates is

$Dl_{m}(x)=\begin{pmatrix}1&0&0\\\
\frac{(8m-8)r}{\left((8-8m)x_{1}+5m-4\right)^{2}}&\frac{1}{(8-8m)x_{1}+5m-4}&0\\\
0&0&\tfrac{1}{(8-8m)x_{1}+5m-4}\end{pmatrix}.$ (3.3)

###  3.4. Definition of $u_{m}$

The mapping $u_{m}$ is the same as in [8, Section 6] with
$\varepsilon=\tfrac{1}{8m}$. They define it piecewise in several regions. For
convenience, we include their picture here as Fig. 6. The mapping is axially
symmetric with respect to $x_{1}$-axis so Fig. 6 is rotated around it and the
regions a, a’, b, c, d, e, e’ on the left are mapped by the mapping to the
corresponding regions on the right. In the limit the regions a’, c, e’
disappear and the region a which was outside of e is mapped inside the image
of the boundary of e.

aecdbfa’e’$u_{m}$baa’e’edfc Figure 6. The areas as in [8, Figure 2.].

For us it is important how it works in the region that contains our bonbons
$B_{m}$, that is in the region c,

$\texttt{c}:=\left\\{(x_{1},r,\varphi)\in\mathbb{R}^{3}:0\leq x_{1}\leq 1,\
0\leq r\leq\tfrac{1}{8m}\right\\}.$

The mapping $u_{m}$ in the region c is defined from cylindrical to spherical
coordinates as

$\displaystyle u_{m}(x_{1},r,\varphi)$
$\displaystyle:=\left(u_{m}^{r}(x_{1},r,\varphi),u_{m}^{\theta}(x_{1},r,\varphi),\varphi\right),$
(3.4) $\displaystyle u_{m}^{r}(x_{1},r,\varphi)$
$\displaystyle:=\left(1+\frac{1}{8m}\right)\cos\left(2\arctan(8m\cdot
r)\right)+\frac{2}{8m}+\frac{x_{1}}{64m^{2}},$ $\displaystyle
u_{m}^{\theta}(x_{1},r,\varphi)$ $\displaystyle:=2\arctan(8m\cdot r).$

Obviously $l_{m}(B_{m})$ is a subset of c. Since $u_{m}^{r}$ does not depend
on $\varphi$ and $u_{m}^{\theta}$ does not depend on $x_{1}$ or $\varphi$, the
matrix of derivatives of $u_{m}$ at the point $(x_{1},r,\varphi)\in\texttt{c}$
with respect to the cylindrical coordinates is (see (2.4))

$Du_{m}(x_{1},r,\varphi)=\begin{pmatrix}\frac{\partial u_{m}^{r}}{\partial
x_{1}}&\frac{\partial u_{m}^{r}}{\partial r}&0\\\
0&u_{m}^{r}{\cdot}\frac{\partial u_{m}^{\theta}}{\partial r}&0\\\
0&0&\frac{u_{m}^{r}\sin(u_{m}^{\theta})}{r}\end{pmatrix}.$ (3.5)

It is easy to check that

$\displaystyle\frac{\partial u_{m}^{r}}{\partial x_{1}}$
$\displaystyle=\frac{1}{64m^{2}},$ $\displaystyle\frac{\partial
u_{m}^{r}}{\partial r}$
$\displaystyle=-\left(1+\frac{1}{8m}\right)\sin\left(2\arctan(8m\cdot
r)\right)\frac{16m}{1+64m^{2}r^{2}},$ $\displaystyle u_{m}^{r}\frac{\partial
u_{m}^{\theta}}{\partial r}$
$\displaystyle=\left(\left(1+\frac{1}{8m}\right)\cos(2\arctan(8m\cdot
r))+\frac{1}{4m}+\frac{x_{1}}{64m^{2}}\right)\frac{16m}{1+64m^{2}r^{2}},$
$\displaystyle\frac{u_{m}^{r}\sin(u_{m}^{\theta})}{r}$
$\displaystyle=\left(\left(1+\frac{1}{8m}\right)\cos(2\arctan(8m\cdot
r))+\frac{1}{4m}+\frac{x_{1}}{64m^{2}}\right)\frac{\sin(2\arctan(8m\cdot
r))}{r}.$

For $x=(x_{1},r,\theta)\in\texttt{c}$ there is a positive constant $C$
independent on $m$ such that

$\begin{array}[]{ccc}\phantom{m\cdot}0\leq\left|\dfrac{\partial
u_{m}^{r}(x)}{\partial x_{1}}\right|\leq\frac{C}{m^{2}}\phantom{\cdot
m}&&\phantom{m\cdot}0\leq\left|\dfrac{\partial u_{m}^{r}(x)}{\partial
r}\right|\leq C\cdot m,\\\\[14.22636pt]
\phantom{m\cdot}0\leq\left|u_{m}^{r}(x)\dfrac{\partial
u_{m}^{\theta}(x)}{\partial r}\right|\leq C\cdot
m,&&\phantom{m\cdot}0\leq\left|\dfrac{u_{m}^{r}(x)\sin(u_{m}^{\theta}(x))}{r}\right|\leq
C\cdot m.\end{array}$ (3.6)

In their paper [8] they do not need the mappings $u_{m}$ to be identity on the
boundary. But it is not difficult to observe that their mappings are bi-
Lipschitz and “well-behaved” on $\partial B(0,3)$, so we can extend them by
hand to be uniformly bi-Lipschitz on $B(0,10)\setminus B(0,3)$ and identity on
$\partial B(0,10)$. So we get a sequence of homeomorphisms with

$\sup_{m\in\mathbb{N}}\int_{B(0,10)}\left|Du_{m}(y)\right|^{2}\,\mathrm{d}y<\infty.$
(3.7)

As we have already mentioned we need a small additional requirement about the
behavior of our first map $b$ on the set $A:=B(0,\tfrac{7}{16})\setminus
B(0,\tfrac{5}{16})$. We need $A$ to be mapped to the regions b, d and f from
the Figure 6 on the left (i.e. they do not enter $a$, $a^{\prime}$, $e$ and
$e^{\prime}$), except for the intersection of $A$ with $b(K)$ and $b(C_{1})$,
which can be mapped into the set c from the same figure. This could be easily
achieved as seen in Fig. 7.

$x_{1}$$\tfrac{1}{8}$$\tfrac{2}{8}$$\tfrac{4}{8}$$C_{1}$$C_{1}$$K$$K$$\tfrac{2}{8}$$x_{2}$$-\tfrac{3}{8}$$b$$x_{2}$$x_{1}$$\tfrac{2}{8}$$\tfrac{4}{8}$$\tfrac{5}{8}$$\tfrac{7}{8}$$\tfrac{1}{8}$$-\tfrac{1}{8}$$-\tfrac{3}{8}$ae
Figure 7. The mapping $b$ near the sphere $\partial B(0,\tfrac{3}{8})$ in the
plane $\\{x_{1},x_{2}\\}$.

We shall also need that $l_{m}(D_{0}\setminus B_{m})$ is the subset of d (see
Fig. 7) and that the derivative of $u_{m}$ is bounded by a constant
(independent on $m$) as $u_{m}$ are uniformly Lipchitz there (see [8,
construction of d \- pages 544-545]).

###  3.5. Computation of the $\mathbf{W^{1,2}}$ norm.

Now, we set

$h_{m}(x):=u_{m}\circ l_{m}\circ b(x).$

Since all three mappings are homeomorphisms it is easy to see that $h_{m}$ is
homeomorphism as well. Analogously it is easy to see that $h_{m}(x)=x$ for
$x\in\partial B(0,10)$.

Since $b$ is bi-Lipschitz, in order to show

$\sup_{m\in\mathbb{N}}\int_{B(0,10)}\left|Dh_{m}(x)\right|^{2}\,\mathrm{d}x<\infty,$

it suffices to show

$\sup_{m\in\mathbb{N}}\int_{B(0,10)}\left|D(u_{m}\circ
l_{m})(x)\right|^{2}\,\mathrm{d}x<\infty.$

Since $l_{m}$ is identity on $B(0,10)\setminus D_{0}$, from (3.7), we obtain

$\sup_{m\in\mathbb{N}}\int_{B(0,10)\setminus D_{0}}\left|D(u_{m}\circ
l_{m})(x)\right|^{2}\,\mathrm{d}x<\infty.$ (3.8)

As mentioned at the end of the previous subsection $u_{m}$ are uniformly
Lipschitz on $l_{m}(D_{0}\setminus B_{m})$. Hence, for every $x\in
D_{0}\setminus\overline{B}_{m}$, we have

$|D(u_{m}\circ l_{m})(x)|\leq|Du_{m}(l_{m}(x))|\cdot|Dl_{m}(x)|\leq C.$

Therefore

$\sup_{m\in\mathbb{N}}\int_{D_{0}\setminus\overline{B}_{m}}\left|D(u_{m}\circ
l_{m})(x)\right|^{2}\,\mathrm{d}x<\infty.$ (3.9)

It remains to consider the derivative on $B_{m}$. For almost every $x\in
B_{m}$ the chain rule (with respect to the correct system of coordinates)
gives

$D(u_{m}\circ l_{m})(x)=Du_{m}(l_{m}(x))\cdot Dl_{m}(x).$ (3.10)

Since $l_{m}$ is always identity on $C_{m}$, by (3.7), we have

$\sup_{m}\int_{C_{m}}|D(u_{m}\circ l_{m})(x)|^{2}\,\mathrm{d}x<\infty.$ (3.11)

By (3.2), (3.5) and (3.10) we know that for every $x\in b(K)$

$D(u_{m}\circ l_{m})(x)=\begin{pmatrix}\frac{\partial u_{m}^{r}}{\partial
x_{1}}&\frac{1}{m}{\cdot}\frac{\partial u_{m}^{r}}{\partial r}&0\\\
0&\frac{u_{m}^{r}}{m}{\cdot}\frac{\partial u_{m}^{\theta}}{\partial r}&0\\\
0&0&\frac{u_{m}^{r}\sin(u_{m}^{\theta})}{mr}\end{pmatrix}$

and (3.6) now imply $\left|D(u_{m}\circ l_{m})(x)\right|\leq C.$ Hence we have

$\sup_{m\in\mathbb{N}}\int_{b(K)}\left|D(u_{m}\circ
l_{m})(x)\right|^{2}\,\mathrm{d}x<\infty.$ (3.12)

Let

$\diamondsuit:=(8m-8)x_{1}+2-m.$

By (3.1), (3.5) and (3.10), for every $x\in\operatorname{int}(T_{m}^{L})$ it
holds that

$D(u_{m}\circ l_{m})(x)=\begin{pmatrix}\frac{\partial u_{m}^{r}}{\partial
x_{1}}+\frac{(8-8m)r}{\diamondsuit^{2}}{\cdot}\frac{\partial
u_{m}^{r}}{\partial r}&\tfrac{1}{\diamondsuit}{\cdot}\frac{\partial
u_{m}^{r}}{\partial r}&0\\\\[5.69054pt] \frac{(8-8m)r\cdot
u_{m}^{r}}{\diamondsuit^{2}}{\cdot}\frac{\partial u_{m}^{\theta}}{\partial
r}&\frac{u_{m}^{r}}{\diamondsuit}{\cdot}\frac{\partial
u_{m}^{\theta}}{\partial r}&0\\\\[5.69054pt]
0&0&\frac{u_{m}^{r}\sin(u_{m}^{\theta})}{r\cdot\diamondsuit}\end{pmatrix}.$
(3.13)

By definition of $T_{m}^{L}$ we know that for
$x\in\operatorname{int}(T_{m}^{L})$ we have

$0\leq
r<\left(1-\frac{1}{m}\right)x_{1}+\frac{2-m}{8m}=\frac{\diamondsuit}{8m},$
(3.14)

so

$\left|\frac{(8-8m)r}{\diamondsuit}\right|\leq C$

for a constant $C$ independent on $x_{1}$, $r$ and $m$. By (3.13) and (3.6),
for every $x\in\operatorname{int}(T_{m}^{L})$, we have

$\left|D(u_{m}\circ l_{m})(x)\right|\leq\frac{C\cdot m}{\diamondsuit}.$

Hence the Fubini theorem implies (see also (3.14))

$\sup_{m\in\mathbb{N}}\int_{\operatorname{int}(T_{m}^{L})}\left|D(u_{m}\circ
l_{m})(x)\right|^{2}\,\mathrm{d}x\leq
C\int_{\frac{1}{8}}^{\frac{2}{8}}\left(\frac{\diamondsuit}{8m}\right)^{2}\frac{m^{2}}{\diamondsuit^{2}}\,\mathrm{d}x_{1}<\infty.$
(3.15)

Since $T_{m}^{R}$ is essentially the same as $T_{m}^{L}$, similar computation
gives

$\sup_{m\in\mathbb{N}}\int_{\operatorname{int}(T_{m}^{R})}\left|D(u_{m}\circ
l_{m})(x)\right|^{2}\,\mathrm{d}x<\infty.$ (3.16)

The boundaries of $D_{0}$, $C_{m}$, $b(K)$, $T_{m}^{L}$ and $T_{m}^{R}$ have
zero measure, so after summing the inequalities (3.8), (3.9), (3.11), (3.12),
(3.15) and (3.16) we obtain the desired inequality

$\sup_{m\in\mathbb{N}}\int_{B(0,10)}|D(u_{m}\circ
l_{m})(x)|^{2}\,\mathrm{d}x<\infty.$

It implies that $\\{h_{m}\\}$ is a bounded and (possibly after taking a
subsequence) weakly convergent sequence in the space $W^{1,2}(B(0,10),\
B(0,10))$, $h_{m}\rightharpoonup h$, where $h$ is the pointwise limit of
$h_{m}$.

###  3.6.  Degree satisfies $\mathbf{\operatorname{Deg}(h,B(0,r),y)=2}$

We claim that for every point $y\in
B\big{(}(\tfrac{1}{2},0,0),\tfrac{1}{2}\big{)}$ and for every radius
${r\in(\tfrac{5}{16},\tfrac{7}{16})}$ we have

$\operatorname{Deg}\left(h,B(0,r),y\right)=2.$

Indeed, for every $r\in(\tfrac{5}{16},\tfrac{7}{16})$ it holds that $h\in
W^{1,2}(\partial B(0,r))\cap L^{\infty}(\partial B(0,r))$. Fix such an $r$.
The mappings $h_{m}$ map the sphere $\partial B(0,r)$ onto three bubbles, see
Fig. 3. In the limit, the filled and the dashed bubbles become topological
spheres with the same orientation as the original sphere $B(0,r)$, and the
dotted bubble disappears, see Fig. 2. Therefore it is not difficult to see
that the degree of $h$ is $2$ inside the smaller topological sphere.

##  4\. For other degrees

In this section, we explain an idea of how to construct a sequence of bounded
and weakly convergent homeomorphisms $\\{h_{m}\\}\subseteq
W^{1,2}(B(0,10),B(0,10))$ which are identity on the boundary $\partial
B(0,10)$, such that the weak limit has degree $k\in\mathbb{Z}$ on a subset of
positive measure. For $k=0,-1,1$, the original construction by Conti and De
Lellis in [8] already gives the desired result. For $k=2$ please see our
construction above.

For degrees $|k|\geq 2$ the construction is similar to the case $k=2$. We need
to modify our mapping $b$ and $l_{m}$ from Fig. 7. Instead of three, we need
to create the appropriate number of bubbles to achieve the desired degree $k$.
To do that we use a bi-Lipschitz mapping $b$ that maps the sphere onto a
sufficiently wiggly shape as in Fig. 8. And by choosing which arrows to shrink
using $l_{m}$ we can decide which bubbles will disappear in the limit. Arrows
that are shrunk before applying the “bubble making function $u_{m}$” will not
disappear and can change the final degree, the arrows that do not change size
by $l_{m}$ will disappear in the limit and so will not affect the final
degree.

$b$$l_{m}$“$u_{m}$”$b$$l_{m}$“$u_{m}$” Figure 8. The construction of arbitrary
degree. On the left-hand side for degree $3$, on the right-hand side for
degree -2. The dotted lines will disappear in the limit, the dashed will
remain.

## References

  * [1] J. Ball.  Global invertibility of Sobolev functions and the interpenetration of matter.  Proc. Roy. Soc. Edinburgh Sect. A, 88(3–4):315–328, 1981.
  * [2] M. Barchiesi, D. Henao, and C. Mora-Corral.  Local invertibility in Sobolev spaces with applications to nematic elastomers and magnetoelasticity.  Arch. Ration. Mech. Anal., 224:743–816, 2017.
  * [3] M. Barchiesi, D. Henao, C. Mora-Corral, and R. Rodiac.  On the lack of compactness problem in the axisymmetric neo-Hookean model.  Arch. Ration. Mech. Anal., 224:743–816, 2017.
  * [4] M. Barchiesi, D. Henao, C. Mora-Corral, and R. Rodiac.  Harmonic dipoles and the relaxation of the neo-Hookean energy in 3D elasticity.  Arch. Rational Mech. Anal, 247(70, 46 pp), 2023.
  * [5] O. Bouchala, S. Hencl, and A. Molchanova.  Injectivity almost everywhere for weak limits of Sobolev homeomorphisms.  J. Funct. Anal., 279(108658), 2020.
  * [6] H. Brezis and L. Nirenberg.  Degree theory and BMO. I. Compact manifolds without boundaries.  Selecta Math., 1:197–263, 1995.
  * [7] P. G. Ciarlet and J. Nečas.  Injectivity and self-contact in nonlinear elasticity.  Arch. Ration. Mech. Anal., 97:171–188, 1987.
  * [8] S. Conti and C. De Lellis.  Some remarks on the theory of elasticity for compressible Neohookean materials.  Ann. Sc. Norm. Super. Pisa Cl. Sci., 2:521–549, 2003.
  * [9] G. De Philippis and A. Pratelli.  The closure of planar diffeomorphisms in Sobolev spaces.  Ann. Inst. H. Poincare Anal. Non Lineaire, 37:181–224, 2020.
  * [10] A. Doležalová, S. Hencl, and J. Malý.  Weak Limit of Homeomorphisms in $W^{1,n-1}$ and (INV) Condition.  Arch. Ration. Mech. Anal., 247(5):80, 2023.
  * [11] A. Doležalová, S. Hencl, and A. Molchanova.  Weak limit of homeomorphisms in $W^{1,n-1}$: invertibility and lower semicontinuity of energy.  arXiv 2212.06452, 2022.
  * [12] D. Henao and C. Mora-Corral.  Fracture surfaces and the regularity of inverses for BV deformations.  Arch. Rational Mech. Anal., 201:575–629, 2011.
  * [13] D. Henao and C. Mora-Corral.  Lusin’s condition and the distributional determinant for deformations with finite energy.  Adv. Calc. Var., 5:355–409, 2012.
  * [14] S. Hencl and J. Malý.  Jacobians of Sobolev homeomorphisms.  Calc. Var. Partial Differential Equations, 38(1-2):233–242, 2010\.
  * [15] T. Iwaniec and J. Onninen.  Monotone Sobolev Mappings of Planar Domains and Surfaces.  Arch. Ration. Mech. Anal., 219:159–181, 2016.
  * [16] T. Iwaniec and J. Onninen.  Limits of Sobolev homeomorphisms.  J. Eur. Math. Soc., 19:473–505, 2017.
  * [17] J. Kauhanen, P. Koskela, and J. Malý.  Mappings of finite distortion: discreteness and openness.  Arch. Ration. Mech. Anal., 160(2):135–151, 2001.
  * [18] S. Müller and S. J. Spector.  An existence theory for nonlinear elasticity that allows for cavitation.  Arch. Rational Mech. Anal., 131(1):1–66, 1995.
  * [19] S. Müller, S. J. Spector, and Q. Tang.  Invertibility and a topological property of Sobolev maps.  SIAM J. Math. Anal., 27:959–976, 1996.
  * [20] D. Swanson and W. P. Ziemer.  A topological aspect of Sobolev mappings.  Calc. Var. Partial Differential Equations, 14(1):69–84, 2002.
  * [21] D. Swanson and W. P. Ziemer.  The image of a weakly differentiable mapping.  SIAM J. Math. Anal., 35(5):1099–1109, 2004.
  * [22] Q. Tang.  Almost-everywhere injectivity in nonlinear elasticity.  Proc. Roy. Soc. Edinburgh Sect. A, 109:79–95, 1988.


# Generalizable Human Gaussians from Single-View Image

Jinnan Chen1 Chen Li1  Jianfeng Zhang2 Hanlin Chen1 Buzhen Huang1
Gim Hee Lee1
1 School of Computing, National University of Singapore
2 Bytedance
https://github.com/jinnan-chen/HGM

###### Abstract

In this work, we tackle the task of learning generalizable 3D human Gaussians
from a single image. The main challenge for this task is to recover detailed
geometry and appearance, especially for the unobserved regions. To this end,
we propose single-view generalizable Human Gaussian model (HGM), a diffusion-
guided framework for 3D human modeling from a single image. We design a
diffusion-based coarse-to-fine pipeline, where the diffusion model is adapted
to refine novel-view images rendered from a coarse human Gaussian model. The
refined images are then used together with the input image to learn a refined
human Gaussian model. Although effective in hallucinating the unobserved
views, the approach may generate unrealistic human pose and shapes due to the
lack of supervision. We circumvent this problem by further encoding the
geometric priors from SMPL model. Specifically, we propagate geometric
features from SMPL volume to the predicted Gaussians via sparse convolution
and attention mechanism. We validate our approach on publicly available
datasets and demonstrate that it significantly surpasses state-of-the-art
methods in terms of PSNR and SSIM. Additionally, our method exhibits strong
generalization for in-the-wild images.

##  1 Introduction

Automatic 3D human reconstruction from single image is crucial in augmented
and virtual reality (AR/VR), game industry, filmmaking, etc. Previous works
either rely on strong 3D supervision such as groundtruth signed distance value
pifu19 ; pifuhd20 ; gta23 ; icon22 ; econ23 ; sifu24 ; sith24 , or extra input
icon22 ; econ23 ; sifu24 ; sith24  such as normal and depth. However, most of
them focus on surface geometry rather than novel view synthesis quality. With
the development of neural radiance fields nerf , novel view rendering quality
has been greatly improved with multi-view posed images. SHERF sherf23  is
proposed to predict human radiance fields from a single image. However, the
back and side views are always blurry and lack details without additional
prior. Furthermore, it needs large amounts of query points sampled for volume
rendering, which hinders practical real-time application in the industries. 3D
contents generation has been sigificantly boosted with the help of pre-trained
diffusion models. Some methods optimize underlying geometry from scratch by
introducing score distillation sampling during the optimization. Although
effective, these methods still suffer from long time optimization and over-
saturation problems. Recent 3D Gaussians generation methods lgm24 ; grm24
combine multi-view diffusion models 0123 ; imagedream23 ; mvdream23 ;
instant3d ; dmv3d23  with generalizable multi-view Gaussians prediction models
to generate 3D Gaussians with high quality and efficiency. We aim to extend
this on human reconstruction. However, directly employ such methods to human
images with complex texture and poses may give unsatisfying results due to: 1)
Original input image are down-sampled and then regenerated with multi-view
diffusion models, leading to severe quality degradation of the front view
image. 2) Inconsistency between training and inference. Specifically, the
input multi-view images are ground-truth images with accurate camera poses and
multi-view consistency during training. On the other hand, the generated
multi-view images from image can be inconsistent and the camera poses can
deviate from the training settings during inference. To address the above-
mentioned problems, we introduce a novel Human Gaussians Model (HGM) with
Gaussians splatting, which support fast and high quality rendering from single
view input in 180 FPS, and generalize well to in-the-wild cartoon images as
shown in Fig 1. We do not use multi-view diffusions on the consideration of
the inconsistency problem.

Instead, we propose a diffusion based coarse-to-fine framework, where the
diffusion model is adapted to refine back-view images rendered from a coarse
human Gaussian model. We inject human prior to the Gaussian prediction process
for generalization ability in terms of different poses. Specifically, our
model consists of two branches: 1) The first branch is a UNet to directly
predict the Gaussians from the input image, as inspired by image splatter
splatter24 . 2) The second branch predicts Gaussians based on the
predicted/fitted SMPL vertices with projected image features due to the depth
ambiguity and complexity of the human pose. Although the SMPL predicted
Gaussians are enhanced with human pose and shape prior, the generated
Gaussians are still blurry when trained with limited data. To distill the
human prior from the SMPL predicted Gaussians and to overcome the difficulty
to register two branches of Gaussians, we do a soft combination of the two
gaussains by diffusing the SMPL Gaussians features to the bounding box and
resample the diffused SMPL volume feature with the mean of the UNet predicted
Gaussians. As a result, the SMPL predicted Gaussians correct the wrong depth
and pose of the UNet predicted Gaussians without losing its high-frequency
details. Subsequently, we combine the feature from two branches and input them
into a transformer to predict the final Gaussians. Given the loss of details
of the back-view by directly predicting the Gaussians from a single view, we
further apply pre-trained diffusion models to refine the back view. We then
input the original front view and refined back view images to our two-view
model with additional cross-view self-attentions to get the final output.
Meshes can be extracted from densely rendered depth map with Gaussians
renderer and TSDF fusion. Our model can be trained with only posed multi-view
images without any 3D supervision and generalize well to in the wild images,
even for human images in different domains such as cartoon characters.

In summary, our contributions are:

  * •

To the best of our knowledge, we are first to propose a generalizable feed-
forward human Gaussian reconstruction framework from single image.

  * •

We design a method to inject human prior into a generalizable Gaussian model
to better infer human geometry and texture while keeping multi-view
consistency and high fidelity.

  * •

We conduct extensive experiments to demonstrate the effectiveness of our
method in both novel view synthesis and 3D reconstruction.

Figure 1: Our reconstructed Gaussians, refined back-view, extracted textured
mesh and novel view synthesis from single view images, including in the wild
images and OOD images in the last row.

##  2 Related works

Single-view Human Reconstruction. PIFu pifu19 , PIFuHD pifuhd20 , PaMIR
pamir20 , and GTA gta23  are capable of inferring full textures from a single
image. Techniques such as PHORHUM phorhum22  and S3F s3f23  go further by
separating albedo and global illumination. However, these methods lack
information from other views or prior knowledge, such as diffusion models,
often resulting in unsatisfactory textures. TeCH tech24  utilizes diffusion-
based models for visualizing unseen areas, producing realistic results.
However, it requires time-intensive per-subject optimization and depends on
accurate SMPL-X. The emergence of Neural Radiance Fields (NeRF) has led to
methods sherf23 ; elicit23 ; mps22 ; nhp21  using videos or multi-view images
to optimize NeRF for human form capture. Recent advancements like SHERF
sherf23  and ELICIT elicit23  aim to generate human NeRFs from single images.
While NeRF-based approaches are effective in creating high-quality images from
various perspectives, they often struggle with detailed 3D mesh generation
from single images and require extensive optimization time. More recently,
SiTH sith24  proposes to combine a back-view hallucination model with an SDF-
based mesh reconstruction model. Similarly, SiFU sifu24  employs a text-to-
image diffusion-based prior to generate realistic and consistent textures for
invisible views. However, these methods require 3D annotations such as the SDF
of the meshes and texture maps as strong supervision, which are not always
available. Compared to these methods, our approach can be trained solely on
multi-view images and achieves better performance and generalize well to in-
the-wild images.

Human Gaussians. 3D Gaussians gs23  and differentiable splatting splatter24
have gained broad popularity due to their efficiency in reconstructing high-
fidelity 3D scenes from posed images using only a moderate number of 3D
Gaussians. This representation has been quickly adopted for various
applications, including image or text-conditioned 3D generation and avatar
reconstruction. Among these methods, Gauhuamn and HUGS gauhuman ; hugs  are
first to propose optimizing human guassians from monocular human videos.
However, they are not applicable to single static human images. GPS-
Gaussiangpsgaussian  propose a generalizable multi-view huaman Gaussian model
with high quality rendering, however, it need dense views 16 or 8, which can
not be directly applied to single view human images. In contrast, our human
model is able to generate Gaussians from single view images with strong
generalization.

Generalizable Gaussians with Multi-view Diffusion. The Large Reconstruction
Model (LRM) lrm24  scales up both the model and the dataset to predict a
neural radiance field (NeRF) from single-view images. Although LRM is
primarily a reconstruction model, it can be combined with Diffusion Models
(DMs) to achieve text-to-3D and image-to-3D generation as demonstrated by
extensions such as Zero1230123 , Image Dreamimagedream23  Instant3D instant3d
and DMV3D dmv3d23 . Our method also builds on a strong reconstruction model
and uses pretrained 2D DMs to provide input images missing information in a
feed-forward manner. Some concurrent works, such as LGM lgm24 , AGG agg24 ,
and Splatter Image splatter24 , also utilize 3D Gaussians in a feed-forward
model. LGM lgm24  combines novel view generation diffusion models with
generalizable Gaussians in a feed-forward manner, while GRM grm24  replaces
the U-Net architecture with a pure-transformer one and scales up to large
resolution. However, these methods face two main challenges when using pre-
trained diffusion models. Firstly, the generated input view image becomes
blurry compared to the original input, which affects the subsequent
generalizable Gaussian model. Secondly, diffusion models can introduce multi-
view inconsistency, especially for human images with different poses, making
direct adaptation unfeasible. We solve these problems by only using DMs as the
refine tools without damage the input image quality or introducing multi-view
inconsistency. We adopt highly efficient 3D Gaussiansgs23  for representing
and rendering. This design alleviates the computation and memory bottleneck
posed by NeRF and volumetric rendering, enabling us to generate high-quality
3D avatars within seconds.

##  3 Our Method

###  3.1 Preliminaries

3D Gaussian Splatting (3DGS). Introduced by gs23 , 3D Gaussian splatting
represents 3D assets or scenes using a collection of 3D Gaussians. Each
Gaussian is characterized by its center $x\in\mathbb{R}^{3}$, scaling factor
$s\in\mathbb{R}^{3}$, rotation $r\in\mathbb{R}^{3}$, opacity
$\alpha\in\mathbb{R}$, and color feature $c\in\mathbb{R}^{c}$. View-dependent
effects can be modeled with spherical harmonics. A 3D scene can thus be
represented explicitly by a set of Gaussians $G=\\{G_{i}\\}$, where
$G_{i}=\\{{x_{i},s_{i},r_{i},\alpha_{i},c_{i}}\\}$ represents the attributes
for the $i$-th Gaussian. Compared with NeRF nerf , 3DGS performs fast
rendering by first projecting Gaussians onto the image plane as 2D Gaussians
and performing alpha-blending for each pixel in a front-to-back depth order.
Building on this, Image Splatter splatter24  proposes predicting Gaussians
from a single image through image-to-image translation. Specifically, each
pixel is converted to a Gaussian with corresponding attributes, supervised by
multi-view images. The Gaussians center are represented by unprojected pixels
with predicted depth. Our model builds on this representation by directly
predicting XYZ coordinates from the image instead of depth.

###  3.2 Overview

Fig. 2 shows an overview of our framework. Given single input human image $I$,
our aim is to predict the corresponding human Gaussians $G$, which can be
further rendered for novel view synthesis and mesh extraction. As shown in the
upper part of Fig. 2, our proposed method consists of three parts: 1) Coarse
Gaussians prediction with human prior by HGM (cf. Sec. 3.3). We utilize human
shape and pose prior from SMPL as guidance to provide additional feature for
the coarse Gaussian prediction. 2) Back-view refinement with diffusion models
(cf. Sec. 3.4). Gaussians predicted by the front-view are always blurry and
lack details, we add this part to enrich the back-view details. 3) Two-view
reconstruction (cf. Sec. 3.5). We put the images from both views our fusion
reconstruction module (a slight modification of HGM with cross-attention) to
get the final refined Gaussians.

Figure 2: Our framework and HGM model. (Top) Our framework consists of three
steps: 1) Coarse Gaussians prediction with human prior by HGM. 2) Back view
refinement with diffusion models. 3) Two view reconstruction to get the
refined $G_{refine}$. (Bottom) Our HGM model consists of two branches of
Gaussians prediction by $\operatorname{UNet}$ and $\operatorname{Tr}_{smpl}$.
Features $f_{smpl}$ and $f_{u}$ sampled by the Gaussian centers from the SMPL
$G_{smpl}$ and image $G_{u}$ branches are then fused to get the final ouput.

###  3.3 Coarse Gaussians Prediction with Human Prior by HGM

The lower part of Fig. 2 shows our proposed Human Gaussian Model (HGM). The
direct prediction of Gaussians from image pixels with UNet splatter24 ; lgm24
lacks human shape and pose prior, thus leading to unsatisfactory results. We
therefore introduce a second branch that utilizes SMPL to enforce human shape
and pose prior into the Gaussian predictions. Specifically, for UNet branch,
the collection of the RGB value and ray embedding for each pixel are
concatenated into a 9-channel feature map as the input
$F_{I}=\\{c_{i},o_{i}\times d_{i},d_{i}\mid i=1,2,...,N\\}$. Our HGM model
predicts Gaussians from the U-Net as:

$\displaystyle G_{u}=\operatorname{UNet}(F_{I}),$ (1)

For SMPL branch, we first densify the SMPL vertices by adding the mean of each
edge and the center of each face in the SMPL mesh to get a dense feature
volume. We then extract images features $F^{\prime}_{I}$ with another shallow
UNet for $G_{smpl}$ prediction. We project $F^{\prime}_{I}$ from $K$ input
views to each vertices as $F_{q}^{k}$, where $q=1,2,...,N_{smpl},k=1,...,K$.
$K=1$ for coarse gaussian prediction and $K=2$ for two-view reconstruction. To
merge features from different views, we follow gpnerf  to use a view-attention
block to aggregate them: $F_{smpl}=\\{F_{q}\mid q=1,2,...,N_{smpl}\\},$ where
$F_{q}=\operatorname{Att}(F_{q}^{1},F_{q}^{2},...,F_{q}^{K})$. The merged
features are subsequently used to predict the coordinates residuals for the
densified SMPL vertices and other Gaussian features with:

$\displaystyle G_{smpl}=\operatorname{Tr}_{smpl}(F_{smpl}),$ (2)

where $G_{smpl}^{i}=\\{{\Delta x_{i},s_{i},r_{i},\alpha_{i},c_{i}}\\}$.
$G_{smpl}$ ensures the coverage of finer details such as clothes and hair in
the reconstruction. We observed that while $G_{u}$ provides better texture
details, it lacks the human shape and pose priors that $G_{smpl}$ can offer.
To address this issue, we apply the SparseConvNet spconv18  (SPConv $\Psi$) to
propagate SMPL features to the whole bounding box, and we denote this feature
as the SMPL volume feature: $S^{vol}=\Psi(S_{smpl})$, where $S_{smpl}$ are
features sampled by the center $C_{smpl}$ of $G_{smpl}$. Note that $S_{smpl}$
and $F_{smpl}$ are sampled from $F^{\prime}_{I}$. The volume feature
reconstructed from the SMPL vertex feature provides geometric cues of the
target human body. The centers of the Gaussians from $G_{u}$ are then used to
sample the propagated SMPL volume features, denoted as
$f_{smpl}=S^{vol}(C_{u})$, where $C_{u}$ are the centers of $G_{u}$. The
sampled SMPL features are then concatenated with the intermediate features
from the UNet $f_{u}$ for each Gaussian. This concatenated feature is fed into
another transformer $\operatorname{Tr}_{mix}$ to obtain the mixed Gaussians:

$\displaystyle G_{mix}=\operatorname{Tr}_{mix}([f_{u},f_{smpl}]).$ (3)

Both $\operatorname{Tr}_{smpl}$ and $\operatorname{Tr}_{mix}$ contain multiple
self-attention blocks among the Gaussians to ensure that each Gaussian is
aware of the other.

Figure 3: Our back-view refinement results.

###  3.4 Back-view Refinement

Back-view hallucination poses a significant challenge in single-view human
reconstruction. As shown in Fig. 3, directly using diffusion models to
generate a back-view image can result in incorrect perspective projection with
the front-view image sith24 . The reason is that their diffusion is
conditioned on the back-view mask, and thus can only be applied for orthogonal
projection when directly flipped with the front-view image. However, the back-
view mask is not available during our inference stage since our prediction is
based on perspective projection. To address this issue, we adopt a generate-
then-refine strategy that leverages both front and back views, and the
diffusion prior to produce a perspective-fitted and realistic back-view image
that is suitable for the subsequent two-view fusion stage. Specifically, we
use the ControlNet control23  tile model, which enhances realistic details
based on our coarse results instead of merely performing super-resolution.
Specifically,

$\displaystyle I_{B}^{r}=\operatorname{CN}(y,I_{B}^{c}),$ (4)

where $y$ is the prompt set as ‘Best quality’, $I_{B}^{c}$ and $I_{B}^{r}$ are
back-view image before and after refinement, the controlnet condition and
input image are the same coarse back-view image, the sampling strength as 0.5
and sampling steps as 30. The sampling process takes around 4 seconds. In Fig.
3, we showcase our generated and refined back-view results, comparing them
with the back-view hallucination diffusion network in SiTH sifu24 . Our
results maintain high resolution and generate details such as the hair for the
first woman and the wrinkles in the clothes, whereas SiTH sifu24  produces
less detailed results that are not perspectively fitted to the input image.

Figure 4: Novel view synthesis comparison with SHERF on HuMMan dataset. The
details are highlighted in the red boxes.

###  3.5 Two-view Reconstruction

We combine the refined perspective-fitted back-view image $I_{B}^{r}$ with the
front-view image $I$ and input them into the fusion reconstruction model to
get:

$\displaystyle{{G_{refine}=\operatorname{FUS}(I,I_{B}^{r}).}}$ (5)

Specifically, our fusion model retains the design of the coarse Gaussians
prediction HGM model, but modifies the original single-view Gaussian self-
attention to cross-view attention. The cross-attention allows the Gaussians
predicted from both views to be aware of each other. Additionally for the SMPL
branch, we get the projected features from both views and perform a view
attention to learn to select features from the correct view.

###  3.6 Losses and Implementation

Our complete objective function includes the L2 color loss,
$\mathcal{L}_{rgb}$, the VGG-based LPIPS perceptual loss,
$\mathcal{L}_{lpips}$ perceptual , and the L2 background mask loss with ground
truth masks, $\mathcal{L}_{bg}$. Each of these losses has corresponding
weights that are treated as hyperparameters:

$\displaystyle\mathcal{L}=\lambda_{rgb}\mathcal{L}_{rgb}+\lambda_{lpips}\mathcal{L}_{lpips}+\lambda_{bg}\mathcal{L}_{bg},$
(6)

where $\lambda_{rgb}=\lambda_{lpips}=\lambda_{bg}$=1.0. We separately train
$\operatorname{UNet}$ and $\operatorname{Tr}_{smpl}$, and jointly train with
$\operatorname{Tr}_{mix}$ on 4 NVIDIA RTX A6000 with batch size of 4 for 20
hours. Our input image size is 512×512, and the number of Gaussians for each
view is 256×256, totaling 65,536 Gaussians per view. For
$\operatorname{UNet}$, we use lgm24  pre-trained model as initialization. For
SMPL estimation, we use SMPLERsmpler . Network structures are in the
supplementary materials.

##  4 Experiments

We conduct experiments on the publicly available 3D human datasets THuman2.0
thuman21  and HuMMan ntuhuman22 . Our method is compared with state-of-the-art
(SOTA) methods in both novel view synthesis and 3D mesh reconstruction.

Table 1: Novel view synthesis comparison with SOTA HumanNeRF methods on HuMMan.  Method | PSNR $\uparrow$ | SSIM $\uparrow$ | LPIPS $\downarrow$
---|---|---|---
NHP nhp21  | 18.99 | 0.84 | 0.18
MPS-NERF mps22  | 17.44 | 0.82 | 0.19
SHERF sherf23  | 20.83 | 0.89 | 0.12
Ours | 23.86 | 0.95 | 0.062

###  4.1 Novel View Synthesis

For novel view synthesis, we compare our method with state-of-the-art
HumanNeRF methods: SHERF sherf23 , MPS-NeRF mps22 , and NHP nhp21  on the
HuMMan dataset ntuhuman22 . Additionally, we compare our method with mesh
reconstruction methods PIFu pifu19 , S3F s3f23 , HumanSGD humansgd , and SiTH
sith24 , as well as diffusion-guided reconstruction methods Magic123 magic123
and TEXture texture  on the THuman2.0 dataset. We follow SHERF sherf23  and
SIFU sifu24  to evaluate our method on HuMMan dataset and THuman2.0 dataset.

Table 2: Rendering speed comparison with NHP, MPS-NeRF, SHERF. “FPS” represents inference frames per second, the higher the better. Method | NHP nhp21  | MPS-NeRF mps22  | SHERF sherf23  | Ours
---|---|---|---|---
FPS | 0.15 | 0.60 | 1.33 | 382

As shown in Table 1, our method significantly surpasses the state-of-the-art
single-view HumanNeRF sherf23  by 14.5% in PSNR on the HuMMan dataset. We also
present qualitative comparisons with the SOTA method SHERF and SIFU for both
categories, respectively, in Fig. 4 and Fig. 5. While SHERF predicts blurry
results and loses fidelity, our method preserves high-frequency details and
generates realistic back-views such as wrinkles and hair that are well-fitted
to the front views. This shows the effectiveness of our proposed framework in
generating high quality unseen back-view while preserving the details of the
front-view. SIFU predicts incorrect poses due to inaccurate SMPL estimation,
and also loses texture details. In contrast, our method is more robust to SMPL
estimation errors since we only use the SMPL-diffused feature volume to
provide additional features implicitly. We also compare the rendering
efficiency with NeRF-based methods in Table 2. Our method is around
$300\times$ faster due to the Gaussians rendering. For cross-dataset
evaluation, we train on THuman 2.0 and test on HuMMan dataset, and we include
the rendered 360-degree video in the supplementary materials.

Figure 5: Novel view synthesis comparison with SIFU on THuman2.0 dataset. The details are highlighted in the red boxes. Table 3: Novel view synthesis comparison with SOTA mesh reconstruction methods on THuman2.0.  Method |  PSNR $\uparrow$ |  SSIM $\uparrow$ |  LPIPS $\downarrow$
---|---|---|---
PIFu pifu19  |  18.0934 |  0.9117 |  0.1372
Impersonator++imperson19  |  16.4791 |  0.9012 |  0.1468
TEXturetexture  |  16.7869 |  0.8740 |  0.1435
Magic123magic123  |  14.5013 |  0.8768 |  0.1880
S3F s3f23  |  14.1212 |  0.8840 |  0.1868
HumanSGDhumansgd  |  17.3651 |  0.8946 |  0.1300
SIFUsifu24  |  22.1024 |  0.9236 |  0.0794
Ours w GT SMPL |  23.7415 |  0.9560 |  0.0511
Ours w/o GT SMPL |  22.5243 |  0.9356 |  0.0594
Table 4:  3D reconstruction comparison with SOTA. Only our method trained without 3D supervision. Method |  Chamfer $\downarrow$ |  P2S $\downarrow$ |  Normal $\downarrow$
---|---|---|---
PiFUpifu19  |  1.5991 |  1.4333 |  0.0843
PaMIRpamir20  |  1.2152 |  1.0582 |  0.0730
ICONicon22  |  0.9491 |  0.9846 |  0.0621
ECONecon23  |  1.2585 |  1.4184 |  0.0612
D-IFdif  |  1.1696 |  1.2900 |  0.0936
GTAgta23  |  0.7329 |  0.7297 |  0.0492
SiFUsifu24  |  0.5961 |  0.6058 |  0.0407
Ours |  0.8532 |  0.9323 |  0.0586
Table 5: Ablation study for each component on THuman2.0 dataset . Components |  PSNR($\uparrow$) |  SSIM($\uparrow$) |  LPIPS($\downarrow$)
---|---|---|---
w/o SMPL guide |  21.85 |  0.9303 |  0.619
w/o Two view refine |  22.95 |  0.9322 |  0.656
w/o Diffusion |  23.31 |  0.9352 |  0.639
Full model |  23.74 |  0.9560 |  0.511

###  4.2 3D reconstruction

We also extract the 3D mesh by densely rendering the depth map with Gaussian
render and using TSDFfusion to extract the surface. We compare our results
with SOTA human surface reconstruction methods PIFu pifu19 , Impersonator++
imperson19 , ICON icon22 , ECON econ23 , D-IF dif , GTA gta23 , SiFU sifu24 .
Following sifu24 , we use Chamfer Distance (CD), Point-to-Surface Distance
(P2S), and L2 normal map error as evaluation metrics. Note that our method do
not use the 3D ground truth for supervision, but can also achieve satisfying
results compared with all those fully supervised methods. We show the 3D
reconstructed results with some in the wild images in Fig6, we can see our our
model has strong generalization ability even though without post-optimization
like icon22 ; econ23 ; sifu24 .

Figure 6: 3D reconstruction visualization for some in the wild images, the
first row are input images, the second row are texure-less meshes and textured
meshes

###  4.3 Ablation Studies

We conduct ablation studies to evaluate the effectiveness of our SMPL-guided
Gaussian prediction model, coarse-to-fine refinement strategy, and diffusion
model. As shown in Table 5, both PSNR and SSIM decrease when any component is
removed. This shows the importance of SMPL guidance. The predicted Gaussians
lack human shape and pose priors without SMPL guidance, resulting in unnatural
shapes and poses. The visual comparisons of SMPL guidance are shown in the
supplementary materials. The diffusion-based refinement is crucial for
improving the quality of back-view images as shown in Fig. 3. Additionally,
the Gaussians tend to concentrate more on the front view without the two-view
refinement strategy, leading to poorer rendering of the back part. The best
performance is achieved with all three components.

##  5 Conclusion

In this paper, we introduce a generalizable single-view human Gaussian
reconstruction framework. By incorporating human priors and diffusion priors
into the Gaussian prediction process, our method effectively handles invisible
parts and varying poses. We validate the proposed method on several benchmarks
and demonstrate that it achieves state-of-the-art performance in novel view
synthesis and also show strong generalization ability for in the wild images.

#### Limitations

Currently, our mesh extraction process is not end-to-end and the extracted
meshes lacks surface regularization. As a result, the extracted meshes exhibit
more noise compared to state-of-the-art single-view human mesh reconstruction
methods sith24 ; sifu24 . How to regularize Gaussians on the human surface is
our future research.

#### Broader Impact

This technology can revolutionize the creation of digital content in movies,
video games, and virtual reality. It allows for the rapid and cost-effective
generation of detailed 3D human models from a single image, enhancing the
realism and diversity of virtual characters and scenes.

## References

  * [1] Badour AlBahar, Shunsuke Saito, Hung-Yu Tseng, Changil Kim, Johannes Kopf, and Jia-Bin Huang.  Single-image 3d human digitization with shape-guided diffusion.  In SIGGRAPH Asia, 2023.
  * [2] Thiemo Alldieck, Mihai Zanfir, and Cristian Sminchisescu.  Photorealistic monocular 3d reconstruction of humans wearing clothing.  In CVPR, 2022.
  * [3] Anonymous.  Instant3d: Fast text-to-3d with sparse-view generation and large reconstruction model.  Under Review, 2023.
  * [4] Zhongang Cai, Daxuan Ren, Ailing Zeng, Zhengyu Lin, Tao Yu, Wenjia Wang, Xiangyu Fan, Yang Gao, Yifan Yu, Liang Pan, Fangzhou Hong, Mingyuan Zhang, Chen Change Loy, Lei Yang, and Ziwei Liu.  HuMMan: Multi-modal 4d human dataset for versatile sensing and modeling.  In ECCV, 2022.
  * [5] Mingfei Chen, Jianfeng Zhang, Xiangyu Xu, Lijuan Liu, Yujun Cai, Jiashi Feng, and Shuicheng Yan.  Geometry-guided progressive nerf for generalizable and efficient neural human rendering.  In ECCV, 2022.
  * [6] Enric Corona, Mihai Zanfir, Thiemo Alldieck, Eduard Gabriel Bazavan, Andrei Zanfir, and Cristian Sminchisescu.  Structured 3d features for reconstructing relightable and animatable avatars.  In CVPR, 2023.
  * [7] Xiangjun Gao, Jiaolong Yang, Jongyoo Kim, Sida Peng, Zicheng Liu, and Xin Tong.  Mps-nerf: Generalizable 3d human rendering from multiview images.  In PAMI, 2022.
  * [8] Benjamin Graham, Martin Engelcke, and Laurens van der Maaten.  3d semantic segmentation with submanifold sparse convolutional networks.  In CVPR, 2018.
  * [9] Hsuan-I Ho, Jie Song, and Otmar Hilliges.  Sith: Single-view textured human reconstruction with image-conditioned diffusion.  In CVPR, 2024.
  * [10] Yicong Hong, Kai Zhang, Jiuxiang Gu, Sai Bi, Yang Zhou, Difan Liu, Feng Liu, Kalyan Sunkavalli, Trung Bui, and Hao Tan.  Lrm: Large reconstruction model for single image to 3d, 2024.
  * [11] Shoukang Hu, Fangzhou Hong, Liang Pan, Haiyi Mei, Lei Yang, and Ziwei Liu.  Sherf: Generalizable human nerf from a single image.  In ICCV, 2023.
  * [12] Shoukang Hu and Ziwei Liu.  Gauhuman: Articulated gaussian splatting from monocular human videos.  In CVPR, 2024.
  * [13] Yangyi Huang, Hongwei Yi, Weiyang Liu, Haofan Wang, Boxi Wu, Wenxiao Wang, Binbin Lin, Debing Zhang, and Deng Cai.  One-shot implicit animatable avatars with model-based priors.  In ICCV, 2023.
  * [14] Yangyi Huang, Hongwei Yi, Yuliang Xiu, Tingting Liao, Jiaxiang Tang, Deng Cai, and Justus Thies.  TeCH: Text-guided Reconstruction of Lifelike Clothed Humans.  In 3DV, 2024.
  * [15] Bernhard Kerbl, Georgios Kopanas, Thomas Leimkühler, and George Drettakis.  3d gaussian splatting for real-time radiance field rendering.  ACM Transactions on Graphics, 42(4), July 2023.
  * [16] Muhammed Kocabas, Rick Chang, James Gabriel, Oncel Tuzel, and Anurag Ranjan.  Hugs: Human gaussian splats, 2023.
  * [17] Youngjoong Kwon, Dahun Kim, Duygu Ceylan, and Henry Fuchs.  Neural human performer: Learning generalizable radiance fields for human performance rendering.  NIPS, 2021.
  * [18] Ruoshi Liu, Rundi Wu, Basile Van Hoorick, Pavel Tokmakov, Sergey Zakharov, and Carl Vondrick.  Zero-1-to-3: Zero-shot one image to 3d object.  In ICCV, 2023.
  * [19] Wen Liu, Zhixin Piao, Min Jie, Wenhan Luo, Lin Ma, and Shenghua Gao.  Liquid warping gan: A unified framework for human motion imitation, appearance transfer and novel view synthesis.  In ICCV, 2019.
  * [20] Ben Mildenhall, Pratul P. Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, and Ren Ng.  Nerf: Representing scenes as neural radiance fields for view synthesis.  In ECCV, 2020.
  * [21] Guocheng Qian, Jinjie Mai, Abdullah Hamdi, Jian Ren, Aliaksandr Siarohin, Bing Li, Hsin-Ying Lee, Ivan Skorokhodov, Peter Wonka, Sergey Tulyakov, and Bernard Ghanem.  Magic123: One image to high-quality 3d object generation using both 2d and 3d diffusion priors.  In ICLR, 2024.
  * [22] Elad Richardson, Gal Metzer, Yuval Alaluf, Raja Giryes, and Daniel Cohen-Or.  Texture: Text-guided texturing of 3d shapes.  In SIGGRAPH, 2023.
  * [23] Shunsuke Saito, , Zeng Huang, Ryota Natsume, Shigeo Morishima, Angjoo Kanazawa, and Hao Li.  Pifu: Pixel-aligned implicit function for high-resolution clothed human digitization.  In ICCV, 2019.
  * [24] Shunsuke Saito, Tomas Simon, Jason Saragih, and Hanbyul Joo.  Pifuhd: Multi-level pixel-aligned implicit function for high-resolution 3d human digitization.  In CVPR, 2020.
  * [25] Yichun Shi, Peng Wang, Jianglong Ye, Long Mai, Kejie Li, and Xiao Yang.  Mvdream: Multi-view diffusion for 3d generation.  arXiv:2308.16512, 2023.
  * [26] Stanislaw Szymanowicz, Christian Rupprecht, and Andrea Vedaldi.  Splatter image: Ultra-fast single-view 3d reconstruction.  In CVPR, 2024.
  * [27] Jiaxiang Tang, Zhaoxi Chen, Xiaokang Chen, Tengfei Wang, Gang Zeng, and Ziwei Liu.  Lgm: Large multi-view gaussian model for high-resolution 3d content creation.  arXiv preprint arXiv:2402.05054, 2024.
  * [28] Peng Wang and Yichun Shi.  Imagedream: Image-prompt multi-view diffusion for 3d generation, 2023.
  * [29] Yuliang Xiu, Jinlong Yang, Xu Cao, Dimitrios Tzionas, and Michael J. Black.  ECON: Explicit Clothed humans Optimized via Normal integration.  In CVPR, 2023.
  * [30] Yuliang Xiu, Jinlong Yang, Dimitrios Tzionas, and Michael J. Black.  ICON: Implicit Clothed humans Obtained from Normals.  In CVPR, 2022.
  * [31] Dejia Xu, Ye Yuan, Morteza Mardani, Sifei Liu, Jiaming Song, Zhangyang Wang, and Arash Vahdat.  Agg: Amortized generative 3d gaussians for single image to 3d.  arXiv preprint 2401.04099, 2024.
  * [32] Xiangyu Xu, Lijuan Liu, and Shuicheng Yan.  Smpler: Taming transformers for monocular 3d human shape and pose estimation.  PAMI, 2024.
  * [33] Yinghao Xu, Hao Tan, Fujun Luan, Sai Bi, Peng Wang, Jiahao Li, Zifan Shi, Kalyan Sunkavalli, Gordon Wetzstein, Zexiang Xu, and Kai Zhang.  Dmv3d: Denoising multi-view diffusion using 3d large reconstruction model, 2023.
  * [34] Xueting Yang, Yihao Luo, Yuliang Xiu, Wei Wang, Hao Xu, and Zhaoxin Fan.  D-if: Uncertainty-aware human digitization via implicit distribution field.  In ICCV, 2023.
  * [35] Xu Yinghao, Shi Zifan, Yifan Wang, Chen Hansheng, Yang Ceyuan, Peng Sida, Shen Yujun, and Wetzstein Gordon.  Grm: Large gaussian reconstruction model for efficient 3d reconstruction and generation, 2024.
  * [36] Tao Yu, Zerong Zheng, Kaiwen Guo, Pengpeng Liu, Qionghai Dai, and Yebin Liu.  Function4d: Real-time human volumetric capture from very sparse consumer rgbd sensors.  In CVPR, 2021.
  * [37] Lvmin Zhang, Anyi Rao, and Maneesh Agrawala.  Adding conditional control to text-to-image diffusion models.  In ICCV, 2023.
  * [38] Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang.  The unreasonable effectiveness of deep features as a perceptual metric.  In CVPR, 2018.
  * [39] Zechuan Zhang, Li Sun, Zongxin Yang, Ling Chen, and Yi Yang.  Global-correlated 3d-decoupling transformer for clothed avatar reconstruction.  In NIPS, 2023.
  * [40] Zechuan Zhang, Zongxin Yang, and Yi Yang.  Sifu: Side-view conditioned implicit function for real-world usable clothed human reconstruction.  In CVPR, 2024.
  * [41] Shunyuan Zheng, Boyao Zhou, Ruizhi Shao, Boning Liu, Shengping Zhang, Liqiang Nie, and Yebin Liu.  Gps-gaussian: Generalizable pixel-wise 3d gaussian splatting for real-time human novel view synthesis.  In CVPR, 2024.
  * [42] Zerong Zheng, Tao Yu, Yebin Liu, and Qionghai Dai.  Pamir: Parametric model-conditioned implicit representation for image-based human reconstruction.  In PAMI, 2021.

##  Appendix A Appendix / Supplementary Materials

#### Experimental Environment.

We conduct all the experiments on NVIDIA RTX A6000 GPU. The experimental
environmentis PyTorch 2.2.1 and CUDA 12.2.

#### Network structures.

Our UNet model consists of 6 down blocks, 1 middle block, and 5 up blocks,
with an input image size of 512×512 and an output Gaussian feature map size of
256×256. We use 2 input views, resulting in a total of 256×256×4 = 131,072
output Gaussians. Each block contains several residual layers and an optional
down-sample or up-sample layer. For the last 3 down blocks, the middle block,
and the first 3 up blocks, we insert cross-view self-attention layers after
the residual layers. The final feature maps are processed by a 1×1 convolution
layer to produce 14-channel pixel-wise Gaussian features. We adopt SiLU
activation and group normalization for the UNet. Our $\operatorname{Tr}{smpl}$
and $\operatorname{Tr}{mix}$ share the same structure, consisting of multiple
self-attention blocks. Specifically, they each have 5 up blocks and 5 down
blocks. The down-sample channels are [64, 128, 256, 512, 1024], and the up-
sample channels are [1024, 512, 256, 128, 64]. The input dimensions for
$\operatorname{Tr}{smpl}$ and $\operatorname{Tr}{mix}$ are 35 and 256,
respectively. The output dimension for both is 14, matching the dimension of
the Gaussian features. For the attention blocks, we use a memory-efficient
attention implementation.

#### Ablation visualization for SMPL guidance.

We visualize the rendered images using only UNet without SMPL guidance and
those produced by our full model, as shown in Fig. 7. Without SMPL guidance,
the side view images exhibit significant artifacts, such as misaligned arms
and unnatural shapes of clothes and heads, highlighted in the red boxes. This
demonstrates the effectiveness of our SMPL-guided Gaussian prediction design.

Figure 7: Ablation studies in terms of SMPL guidance. Details are highlighted
in the red boxes.