# Technical Report on: Anchoring Sagittal Plane Templates in a Spatial
Quadruped

Timothy Greco and Daniel E. Koditschek *This work was supported by AFRL grant
FA865015D1845 (subcontract 669737-1) and ONR grant #N00014-16-1-2817, a
Vannevar Bush Fellowship held by the second author, sponsored by the Basic
Research Office of the Assistant Secretary of Defense for Research and
Engineering.The authors are with the GRASP Lab, University of Pennsylvania,
Philadelphia, PA 19014. {tmgreco, kod} @ seas.upenn.edu

###### Abstract

This technical report provides a more thorough treatment of the proofs and
derivations in [1]. The description of the anchoring controller is reproduced
here without abridgement, and additional appendices provide a clearer account
of the implementation details.

##  I Anchoring Controller

When two paired legs are in contact with the ground, define the stance frame
as the right-handed inertial coordinate frame with the $z-$axis pointing up
and the $y-$axis pointing from the right toe to the left toe, with the origin
at the midpoint between the toes. Define the body frame as the right-handed
coordinate frame with the $z-$axis normal to the robot’s dorsal plane and the
$y-$axis pointing from the robot’s right hip to its left, with the origin at
the robot’s center of mass. The term pose will denote the transformation in
$SE(3)$ that maps the stance frame to the body frame. Define the set
$\mathcal{S}\subset SE(3)$ of _sagittal poses_ as
$\mathcal{S}:=\\{(p_{x},0,p_{z})\\}\times\mathcal{P}$, where $\mathcal{P}$
denotes the pitches — i.e., the set of rotations around the stance frame that
fix the y-axis — and define the _sagittal velocities_ ,
$\mathcal{V}\subset\mathbb{R}^{6}$, as
$\mathcal{V}=\\{(\dot{p}_{x},\,0,\,\dot{p}_{z},\,0,\,\alpha,\,0)\\}$. This
anchoring controller must make $\mathcal{S}\times\mathcal{V}$ attracting and
invariant under the resulting closed loop dynamics.

Assume that the robot behaves as a single rigid body with massless legs, and
that two feet are in contact with the ground, either the front feet or the
hind feet. Suppose further that the robot’s legs do not slip or reach
kinematic singularity. In this configuration, the robot can directly actuate
five of the torso’s six degrees of freedom. It cannot directly produce a
torque about the line between the toes, i.e. in the pitch direction, so this
rotational degree of freedom must remain coupled to the $x$ and $z$
translational degrees of freedom. Templates defined in the sagittal plane
expect and account for this coupling between $x$, $z$, and pitch, so this
anchoring controller reserves those degrees of actuation for the template
controller. Since the remaining degrees of freedom are fully actuated, the
anchoring can use one controller to stabilize the lateral position while a
second controller stabilizes the orientation.

Let the lateral translation controller take the familiar potential-dissipative
form (proportional-derivative, for these translational components of the
pose),

$u_{y}=-K_{p}p_{y}-K_{d}\dot{p_{y}},$ (1)

where $u_{y}$ is a lateral force applied on the robot at the midpoint of the
toes and $p_{y}$ is the body position in the $y-$direction of the stance
frame. If the robot can maintain contact with the ground and if gravity has a
negligible effect on the robot’s lateral movement, this controller must
stabilize the robot’s position to the sagittal plane. These assumptions match
the observed behavior documented in [1]. Thus, it remains to introduce a
controller stabilizing the pitches, $\mathcal{P}$, to achieve the desired
anchor to the sagittal plane, $\mathcal{S}$.

The orientation of the body can also be stabilized by a potential-dissipative
controller [2] of the form

$\tau=-\nabla\Phi(R)-K_{D}\omega,$ (2)

where the rotation matrix $R$ represents the angular component of the pose,
$\omega\in\mathbb{R}^{3}$ is its angular velocity, and the controller terms
are parametrized as

$K_{D}=\begin{bmatrix}\kappa_{1}&0&0\\\ 0&0&0\\\
0&0&\kappa_{2}\end{bmatrix}\quad\text{ and }\quad\Phi(R)=y^{T}Ry$ (3)

for $y=[0,\ 1,\ 0]^{T}$ and $\kappa_{1},\kappa_{2}>0$. The following
subsections formally demonstrate that the controller defined in (2) makes
$T\mathcal{P}\subset SO(3)$ attracting an invariant. Because $\mathcal{P}$ is
a Lie Group and therefore parallelizable [3, Cor. 8.39], we find it convenient
to express it as the product $T\mathcal{P}\approx\mathcal{P}\times\mathcal{W}$
where $\mathcal{W}\approx\mathbb{R}$ is a one-dimensional manifold embedded in
the space of angular velocities, here expressed as
$\mathrm{span}(y)\subset\mathbb{R}^{3}$.

This demonstration is organized as follows. In Section I-A, calculating the
gradient of $\Phi$ confirms that $\mathcal{P}$ lies in its critical set, so
$\mathcal{P}\times\mathcal{W}$ is an equilibrium of the closed-loop dynamics
(10) arising from (2) and thus it is invariant. Section I-B continues with the
calculation of the Hessian of $\Phi$ at the components of its critical set.
Section I-C uses total energy as a Lasalle function to show that the critical
set of $\Phi$ is globally attracting. Section I-D concludes with an
examination of the local stability of equilibrium points in each component of
the critical set, applying the results from Section I-B to conclude that only
point equilibria in $\mathcal{P}\times\mathcal{W}$ are locally attracting,
while point equilibria elsewhere in the critical set are unstable. The results
of Sections I-C and I-D establish that $\mathcal{P}\times\mathcal{W}$ is
“almost globally” attracting in the sense of [4], [5] when the only invariant
set generated by the template dynamics in the sagittal plane consists of local
equilibrium states. Some of the sagittal template dynamics we use in this work
have this property, whereby the combination of global (I-C) and local (I-D)
results suffice to guarantee that only a zero measure set of initial
conditions generate trajectories that fail to converge to the desired sagittal
plane behavior in $\mathcal{P}\times\mathcal{W}$. We conjecture (but do not
prove in this paper) that this controller ensures almost global convergence to
$\mathcal{P}\times\mathcal{W}$ regardless of the properties of the template
dynamics. If this conjecture is sound, as our empirical results suggest, then
it formally guarantees global efficacy of this anchoring controller in
parallel composition with any desired sagittal-plane behavior (including, for
example, the steady state bounding template that introduces a non-equilibrium
attracting invariant set, i.e. a hybrid limit cycle) on any quadruped with
twelve or more actuated degrees of freedom.

###  I-A The Gradient of $\Phi$

Consider the gradient $\nabla_{R}\Phi$ with respect to
$\omega\in\mathbb{R}^{3}$. Applying equation (3.1) from [6],

$\nabla_{R}\Phi(\omega)=\frac{1}{2}\mathbf{Tr}\big{(}(yy^{T}-R^{T}yy^{T}R^{T})RJ(\omega)\big{)},$
(4)

where $J:\mathbb{R}^{3}\to\mathfrak{so}(3)$ is the skew map such that

$J(a)b=a\times b.$

After some massaging,

$\displaystyle\nabla_{R}\Phi(\omega)$
$\displaystyle=\frac{1}{2}\mathbf{Tr}\big{(}(yy^{T}-R^{T}yy^{T}R^{T})RJ(\omega)\big{)}$
$\displaystyle=\frac{1}{2}\mathbf{Tr}\big{(}yy^{T}RJ(\omega)-R^{T}yy^{T}J(\omega)\big{)}$
$\displaystyle=\frac{1}{2}\mathbf{Tr}\big{(}y^{T}RJ(\omega)y-y^{T}J(\omega)R^{T}y\big{)}$
$\displaystyle=\frac{1}{2}\big{(}y^{T}R(\omega\times y)-y^{T}(\omega\times
R^{T}y)\big{)}$ $\displaystyle=\frac{1}{2}\big{(}\omega^{T}(y\times
R^{T}y)-\omega^{T}(R^{T}y\times y)\big{)}$ $\displaystyle=\omega^{T}(y\times
R^{T}y)$

yields an expression of $\nabla_{R}\Phi\in\mathbb{R}^{3}$ as

$\nabla_{R}\Phi=y\times R^{T}y.$ (5)

This formulation makes it easy to determine the critical regions of $\Phi$,
since $\nabla_{R}\Phi=0$ if and only if $R^{T}y=\lambda y$ for
$\lambda\in\\{-1,1\\}$. Now $R^{T}y=y$ if and only if $R\in\mathcal{P}$;
hence, it is useful to define the corresponding antipodal set as
$\mathcal{Q}:=\\{R\in SO(3)\ |\ R^{T}y=-y\\}$. These two disjoint sets
comprise the critical set of $\Phi$. Since $\mathcal{P}$ is in the critical
set of $\Phi$ and $\mathcal{W}$ is in the null space of $K_{D}$, the
controller defined in (2) exerts zero input on $\mathcal{P}\times\mathcal{W}$,
so $\mathcal{P}\times\mathcal{W}$ is invariant under this controller. Fig. 1
shows what configurations in $\mathcal{P}$ and $\mathcal{Q}$ actually look
like for the robot.

Figure 1: Visualizing the critical set of $\Phi$. A displays a pose is in
$\mathcal{P}$, and B displays a pose in $\mathcal{Q}$. This latter
configuration is kinematically feasible but difficult to achieve. In each
picture, the stance frame is labeled ‘s’ and the body frame ‘b’.

###  I-B The Hessian of $\Phi$

Lemma 3.2 from [6] gives an expression for the Hessian of $\Phi$ at any
critical point as a symmetric bilinear form on $\mathfrak{so}(n)$. Applying
this result to (3) yields

$h_{\Phi}(U,V)=-\mathbf{Tr}(yy^{T}RUV)$ (6)

where $U,V\in\mathfrak{so}(3)$. To determine the relative maxima and minima of
$\Phi$, evaluate the Hessian for some $U=V$. Since $\mathfrak{so}(3)$ is
isomorphic to $\mathbb{R}^{3}$, there is some $u$ such that $J(u)=U$. This
isomorphism enables the representation of the Hessian as a quadratic form on
$\mathbb{R}^{3}$, simplifying its evaluation.

$\displaystyle h_{\Phi}(u,u)$ $\displaystyle=-\mathbf{Tr}(yy^{T}RJ(u)J(u)$
$\displaystyle=-y^{T}RJ(u)J(u)y$ $\displaystyle=(R^{T}y)\cdot\big{(}u\times
J(u)y\big{)}$ $\displaystyle=-\big{(}J(u)y\big{)}\cdot\big{(}R^{T}y\times
u\big{)}$ $\displaystyle=\big{(}J(y)u\big{)}\cdot J(R^{T}y)u$
$\displaystyle=u^{T}J(y)^{T}R^{T}J(y)Ru$
$\displaystyle=u^{T}J(y)^{T}J(R^{T}y)u$

The resulting Hessian matrix is

$H_{\Phi}(R):=J(y)^{T}J(R^{T}y).$ (7)

If $R$ is in the critical set, $H_{\Phi}(R)=\lambda J(y)^{T}J(y)$. Evaluating
this equation for $R\in\mathcal{P}$,

$H_{\Phi}(R)=\begin{bmatrix}1&0&0\\\ 0&0&0\\\
0&0&1\end{bmatrix}=:H_{\mathcal{P}}.$ (8)

Similarly, for $R\in\mathcal{Q}$,

$H_{\Phi}(R)=\begin{bmatrix}-1&0&0\\\ 0&0&0\\\
0&0&-1\end{bmatrix}=:H_{\mathcal{Q}}.$ (9)

Thus $\mathcal{P}$ is a minimum and $\mathcal{Q}$ a maximum of $\Phi$ on
$SO(3)$, and since $SO(3)$ has no boundary, these extrema are global. The
kernel of Hessian matrix for each of these sets is in the direction of the
vector $y$, which spans the tangent spaces of both $\mathcal{P}$ and
$\mathcal{Q}$. Thus $\Phi$ is a Morse-Bott function on $SO(3)$ [7].

###  I-C $\Phi$ as a potential

Consider the following dynamical system on $TSO(3)$:

$\displaystyle\dot{R}$ $\displaystyle=RJ(\omega)^{T}$
$\displaystyle\dot{\omega}$ $\displaystyle=M^{-1}(\tau-\omega\times M\omega),$
(10)

where $\tau$ is defined as in (2). This system describes the effect of the
orientation controller on the robot in the absence of any other torques.
Define the stance-frame moment of inertia $M(R)=R^{T}I_{B}R$, where $I_{B}$
denotes the diagonal matrix that expresses the robot’s inertia tensor in the
body frame. Consider the total energy function $\eta:\ TSO(3)\to\mathbb{R}$
defined as

$\eta(R,\omega)=\Phi(R)+\frac{1}{2}\omega^{T}M(R)\omega,$ (11)

where $\omega$ is the robot’s angular velocity in the stance frame. This
function is a Lasalle function for the system defined in (10) [8], [9]. To see
this, first note that $\eta(R,\omega)\geq 0$, since $\Phi(R)\geq 0$ and $M$ is
symmetric positive definite. Examining $\dot{\eta}(R,\omega)$,

$\dot{\eta}(R,\omega)=\omega^{T}(y\times
R^{T}y)+\omega^{T}M\dot{\omega}+\frac{1}{2}\omega^{T}\dot{M}\omega.$ (12)

Since $M=R^{T}I_{B}R$,

$\omega^{T}\dot{M}\omega=2\omega^{T}\dot{R^{T}}I_{B}R\omega=2\omega^{T}(\omega\times
M\omega)=0.$ (13)

Substituting (2), (13), and (10) into (12),

$\displaystyle\dot{\eta}(R,\omega)$ $\displaystyle=\omega^{T}(y\times R^{T}y)$
$\displaystyle\quad+\omega^{T}MM^{-1}(-\nabla\Phi(R)-K_{D}\omega-\omega\times
M\omega)$ $\displaystyle=\omega^{T}(y\times
R^{T}y)-\omega^{T}\nabla\Phi(R)-\omega^{T}K_{D}\omega$
$\displaystyle\quad-\omega^{T}(\omega\times M\omega)$
$\displaystyle=\omega^{T}(y\times
R^{T}y)-\omega^{T}\nabla\Phi(R)-\omega^{T}K_{D}\omega.$

Applying (5) yields

$\dot{\eta}(R,\omega)=-\omega^{T}K_{D}\omega,$ (14)

and since $K_{D}$ is positive semidefinite, $\dot{\eta}\leq 0$. Note that
$\dot{\eta}=0$ if and only if $\omega\in\mathrm{span}(y)$ (i.e.,
$\omega=\alpha y$ for some $\alpha\in\mathbb{R}$) in the stance frame.

By Lasalle’s invariance principle, the largest invariant subset of
$\eta^{-1}[0]$ is attracting and contains all forward limit points of this
dynamical system; it remains to be shown whether this subset contains
$\mathcal{P}\times\mathcal{W}$. Let $\mathcal{L}\subset\eta^{-1}[0]$ be this
largest invariant subset. For any $(R,\omega)\in\mathcal{L}$, the angular
acceleration must be in the $y$ direction to remain in that set under the flow
of 10. If $\omega=\alpha y$ for some $\alpha\in\mathbb{R}$, then

$\displaystyle\dot{\alpha}y$ $\displaystyle=\dot{\omega}$
$\displaystyle=M^{-1}\\!\big{(}\\!-\\!\nabla\Phi(R)-K_{D}(\alpha y)-(\alpha
y)\times M(\alpha y)\big{)}$ $\displaystyle\dot{\alpha}My$
$\displaystyle=-(y\times R^{T}y)-\alpha^{2}y\times My.$ (15)

Since $M$ is symmetric positive definite, $My$ must have a nonzero component
in the $y$ direction. However, neither of the terms on the RHS of (15) can
have any nonzero component in the $y$ direction, so (15) can only hold if
$\dot{\alpha}=0$, from which it follows that

$0=-(y\times R^{T}y)-\alpha^{2}y\times My.$ (16)

To completely characterize the rotational trajectories
$\big{(}R(t),\,\omega(t)\big{)}$ of the flow generated by (10) that satisfy
(16), it is helpful to impose the consequent necessary requirement that time
variation of (16) also evaluate to the constant 0. Accordingly, taking the
derivative of (16) with respect to time yields

$\displaystyle 0$ $\displaystyle=-y\times R(\alpha y\times
y)-2\alpha\dot{\alpha}y\times My-\alpha^{2}\big{(}y\times(\alpha y\times My)$
$\displaystyle\quad\;\\!+y\times M(\alpha y\times y)\big{)}$ $\displaystyle 0$
$\displaystyle=-\alpha^{3}\big{(}y\times(y\times My)\big{)}.$ (17)

Since $y$ must be perpendicular to $y\times My$ if the latter is nonzero, (17)
can only be satisfied if $\alpha=0$ or $y\times My=0$. In either case, (16)
simplifies to

$0=-y\times R^{T}y=-\nabla\Phi(R).\\\ $ (18)

Thus $(R,\omega)\in\mathcal{L}$ if and only if
$R\in\mathcal{P}\cup\mathcal{Q}$ and $\omega\in\mathrm{span}(y)$. Now the
embeddings in $\mathbb{R}^{3}$ of the tangent spaces of both $\mathcal{P}$ and
$\mathcal{Q}$ are both represented by $\mathrm{span}(y)$ in the stance frame,
so
$\mathcal{L}=(\mathcal{P}\times\mathcal{W})\cup(\mathcal{Q}\times\mathcal{U})$,
where $T\mathcal{Q}\approx\mathcal{Q}\times\mathcal{U}$. By Lasalle’s
invariance principle [8], [9],
$(\mathcal{P}\times\mathcal{W})\cup(\mathcal{Q}\times\mathcal{U})$ is
attracting and contains all forward limit points.

###  I-D Local Stability of Equilibria

Suppose that, in addition to the anchorning controller (2), the robot is
subject to some template dynamics on $T\mathcal{P}$; further suppose that the
template is “pitch-steady” as described in [10], so there will be some
$p_{0}\in\mathcal{P}$ that is locally attracting within $T\mathcal{P}$. The
template dynamics around $p_{0}$ in $T\mathcal{P}$ can be approximated by a
second-order linear system

$\begin{bmatrix}\dot{p}\\\ \ddot{p}\end{bmatrix}=\begin{bmatrix}0&1\\\
-\frac{\gamma}{\mu}&-\frac{\beta}{\mu}\end{bmatrix}\begin{bmatrix}p\\\
\dot{p}\end{bmatrix},$ (19)

where $\beta$, $\gamma$, and $\mu$ are positive constants. Since
$T\mathcal{P}$ is invariant under (2), the anchoring will not alter this
behavior. Accordingly, the linearized dynamics about $p_{0}$ in $TSO(3)$ are

$\begin{bmatrix}\dot{\theta}\\\
\dot{\omega}\end{bmatrix}\\!=\\!\begin{bmatrix}0&I\\\
-M^{-1}(H_{\mathcal{P}}+{K})&-M^{-1}(K_{D}+{B})\end{bmatrix}\\!\\!\begin{bmatrix}\theta\\\
\omega\end{bmatrix},$ (20)

where $K=\mathrm{diag}([0,\gamma,0])$ and $B=\mathrm{diag}([0,\beta,0])$.
Since $K_{D}+B$ and $M$ are symmetric positive definite, and
$H_{\mathcal{P}}+K$ is positive definite, (20) is asymptotically stable [2,
Lemma 3.5]. Thus $p_{0}$ is an attractor.

Conversely, consider the action of the same template dynamics when the robot’s
pose and velocity lie in $T\mathcal{Q}$. There will be some corresponding
equilibrium point $q_{0}\in\mathcal{Q}$ which admits the same linearized
dynamics as in (19). In this case, the linearized dynamics about $q_{0}$ on
$TSO(3)$ are

$\begin{bmatrix}\dot{\theta}\\\
\dot{\omega}\end{bmatrix}\\!=\\!\begin{bmatrix}0&I\\\
-M^{-1}(H_{\mathcal{Q}}+{K})&-M^{-1}(K_{D}+{B})\end{bmatrix}\\!\\!\begin{bmatrix}\theta\\\
\omega\end{bmatrix}.$ (21)

In this system, $H_{\mathcal{Q}}+{K}$ has both positive and negative
eigenvalues so $q_{0}$ is a saddle [2, Lemma 3.5].

These local stability results clarify the results of the previous section. The
equilibrium point of any pitch-stable template is stable in $\mathcal{P}$ but
unstable in $\mathcal{Q}$, suggesting that of the two disjoint sets that
compose a global attractor on $SO(3)$, only $\mathcal{P}$ is locally
attracting. While the extension of this result to limit cycles or abitrary
trajectories in pitch remains unproven, the empirical results of [1] support
the plausibility of such a conjecture.

##  II Further Implementation Details

Assuming a single rigid body model of the robot’s torso, let $(x,y,z)$ be the
translational degrees of freedom in the stance frame and let
$(\theta,\phi,\psi)$ be the roll, pitch, and yaw. The template controllers
specify forces $u_{x}$ and $u_{z}$ to be exerted at a virtual toe, and thus
control the robot’s motion in the $x$, $z$, and $\phi$ directions. The
anchoring controllers specify a lateral force $u_{y}$ and a torque $\tau$, and
aim to stabilize the robot’s motion in $y$, $\theta$, and $\psi$. This
appendix explains how the implementation maps from these controller outputs to
toe forces, which are represented in sum and difference coordinates. Here
$p=[x,y,z]^{T}$ is the vector from the COM to the midpoint between the toes
and $q$ is the vector from the midpoint between the toes to the left toe. In
the stance frame, $q$ points directly in the $y$ direction. Assume that the
robot has two feet are in contact with the ground, either the front feet or
the hind feet, and that each leg can exert an arbitrary force in
$\mathbb{R}^{3}$ on the body without slipping or lifting off the ground. Let
$f_{l}$ and $f_{r}$ be the forces at the left and right toes respectively, and
let $x_{l}$ and $x_{r}$ be their locations with respect to the center of mass.
Ignoring gravity and coriolis terms, Newton’s second law yields

$\displaystyle m\ddot{p}$ $\displaystyle=f_{l}+f_{r}$ $\displaystyle
M\dot{\omega}$ $\displaystyle=x_{l}\times f_{l}+x_{r}\times f_{r},$

where $m$ is the mass and $M$ is the moment of inertia matrix as represented
in the stance frame. Translating this into sum and difference coordinates will
make it easier to separate the forces acting on the center of mass from the
torques. Let $s=f_{l}+f_{r}$ and $d=f_{l}-f_{r}$. It follows from the
definition of the stance frame that $p=\frac{1}{2}(x_{l}+x_{r})$, and that
$q=\frac{1}{2}(x_{l}-x_{r})$ points in the direction of the $y$ axis.
Inverting these transformations yields $f_{l}=\frac{1}{2}(s+d)$ and
$f{{}_{r}}=\frac{1}{2}(s-d)$. Similarly, $x_{l}=p+q$ and $x_{r}=p-q$. Thus

$\displaystyle m\ddot{p}=$ $\displaystyle s$ $\displaystyle M\dot{\omega}=$
$\displaystyle(p+q)\times\frac{1}{2}(s+d)+(p-q)\times\frac{1}{2}(s-d)$
$\displaystyle=$ $\displaystyle\frac{1}{2}\Big{(}p\times s+q\times s+p\times
d+q\times d+p\times s$ $\displaystyle{-}\>q\times s-p\times d+q\times
d\Big{)}$ $\displaystyle=$ $\displaystyle\frac{1}{2}\Big{(}2\big{(}p\times
s\big{)}+2\big{(}q\times d)\Big{)}$ $\displaystyle=$ $\displaystyle p\times
s+q\times d.$

Assume that the robot’s orientation is close enough to the sagittal plane that
the matrix $M$ is approximately diagonal. Since $q$ is in the $y$ direction,
$(q\times d)$ has no $y$ component and thus the only acceleration in the pitch
direction will be from $p\times s$. Similarly, the $y$ component $s_{y}$ will
not affect the pitch dynamics. Thus set $s_{x}=u_{x}$ and $s_{z}=u_{z}$ to
execute the template dynamics in the sagittal-plane degrees of freedom $x,z,$
and $\phi$.

Similarly, $s_{y}$ is the only input that exerts a force the $y$ direction, so
set $s_{y}=u_{y}$. While $\theta$ and $\psi$ are affected by both $d$ and $s$,
in practice the torque induced by $p\times s$ is almost entirely in the pitch
direction.

Let $p=r_{p}+e_{y}$, where $r_{p}$ lies in the $xz$ plane and $e_{y}$ points
in the $y$ direction. Similarly, let $s_{xz}$ be the sagittal plane component
of $s$ and let $s_{y}$ be the $y$-component. Then

$\displaystyle p\times s$ $\displaystyle=r_{p}\times s+e_{y}\times s$
$\displaystyle=r_{p}\times s_{xz}+e_{y}\times s_{xz}+r_{p}\times
s_{y}+e_{y}\times s_{y}$

The vectors $e_{y}$ and $s_{y}$ are parallel, so their cross product is zero.
The terms $e_{y}\times s_{xz}+r_{p}\times s_{y}$ are both perpendicular to the
$y$ axis, so they can be directly cancelled out by the $q\times d$ term. In
practice, $e_{y}$ and $s_{y}$ are small enough that the effect of this term
can be neglected, admitting the approximation

$s\times p\approx r_{p}\times s_{xz}$

as only effecting the pitch. This leaves the $q\times d$ as the only input
that controls the other rotational degrees of freedom. Thus the torque exerted
by the anchoring controller is

$\tau=\begin{bmatrix}\tau_{\theta}\\\ 0\\\ \tau_{\psi}\end{bmatrix}=q\times
d,$

so $d=J(q)^{\dagger}\tau$ yields a value for $d$ that will generate the
desired body torques, where $J(q)$ is a skew-symmetric matrix and
$J(q)^{\dagger}$ denotes its pseudoinverse. The $y$ component of $d$ is set to
zero to eliminate internal forces between the toes.

##  III Assumptions

  1. 1.

The dynamics are that of a single rigid body; the legs have negligible mass.
Furthermore, the legs are assumed to not collide with each other and to not
reach singularity.

  2. 2.

The robot has two feet in contact with the ground, either the forelegs or the
hind legs. When all four legs are in contact with the ground, there may not be
consensus between the two halves on which plane is the sagittal plane, causing
internal forces and hindering the robot from converging to either. However,
these effects were not noticeable during double-stance periods, such as those
occurring during the pronk or box jump.

  3. 3.

Each of the two toes in contact with the ground is fixed in the world frame
and can exert an arbitrary force in $\mathbb{R}^{3}$ on the robot’s body.

     * •

This assumption is violated when the robot’s toes slip. For the most part, the
only times when slipping occurs are when the toes bounce off the ground during
the steady-state gaits or when the robot is recovering from large disturbances
in yaw.

     * •

The assumption of arbitrary force in $\mathbb{R}^{3}$ guarantees that the
robot can exert whatever force is necessary for the anchoring controller.
Blindly following this assumption could cause the robot to pull one of its
toes off the ground to respond to a disturbance in roll. The code does take
this into account, and will scale down the input from the anchoring controller
to prevent it.

  4. 4.

Gravity has a negligible effect outside of the sagittal plane, i.e. the ground
is flat and the center of mass is close to the sagittal plane (so there is
little action of gravity in the roll and $y$ directions).

  5. 5.

The stability analysis assumes that all of these assumptions are true for an
infinite amount of time, so will be violated by cases when, for example, the
trajectory of the robot makes it kinematically unable to maintain contact with
the ground (such as very high lateral velocity).

  6. 6.

As written above, $s\times p$ has negligible components outside of the pitch
direction. This assumption is not necessary, but it simplifies the
implementation.

## References

  * [1] T. Greco and D. E. Koditschek, “Anchoring sagittal plane templates in a spatial quadruped.” Submitted to 2023 International Conference on Robotics and Automation (ICRA), 2022.
  * [2] D. E. Koditschek, “The application of total energy as a lyapunov function for mechanical control systems,” Contemporary Mathematics, February 1989.
  * [3] J. M. Lee, “Introduction to smooth manifolds, volume 218 of graduate texts in mathematics,” SpringerVerlag, New York, 2003.
  * [4] D. E. Koditschek, “Application of a new lyapunov function to global adaptive attitude tracking,” in Proceedings of the IEEE Conference on Decision and Control, pp. 63–68, 1988.
  * [5] D. E. Koditschek, “Some applications of natural motion control,” Journal of Dynamic Systems, Measurement, and Control, vol. 113, p. 552–557, Dec 1991\.
  * [6] H. Bozma, W. Gillam, and F. Öztürk, “Morse-bott functions on orthogonal groups,” Topology and its Applications, vol. 265, p. 106807, 2019.
  * [7] R. Bott, “Nondegenerate critical manifolds,” Annals of Mathematics, vol. 60, no. 2, pp. 248–261, 1954.
  * [8] J. LaSalle, “Some extensions of liapunov’s second method,” IRE Transactions on Circuit Theory, vol. 7, no. 4, pp. 520–527, 1960.
  * [9] H. Khalil, Nonlinear Systems.  Pearson Education, Prentice Hall, 2002.
  * [10] A. De, T. T. Topping, J. D. Caporale, and D. E. Koditschek, “Mode-reactive template-based control in planar legged robots,” IEEE Access, vol. 10, pp. 16010–16027, 2022.


# ART3D: 3D Gaussian Splatting for Text-Guided Artistic Scenes Generation

Pengzhi Li1, Chengshuai Tang1, Qinxuan Huang2, Zhiheng Li1†
1Tsinghua Shenzhen International Graduate School
2Tsinghua-Berkeley Shenzhen Institute

###### Abstract

In this paper, we explore the existing challenges in 3D artistic scene
generation by introducing ART3D, a novel framework that combines diffusion
models and 3D Gaussian splatting techniques. Our method effectively bridges
the gap between artistic and realistic images through an innovative image
semantic transfer algorithm. By leveraging depth information and an initial
artistic image, we generate a point cloud map, addressing domain differences.
Additionally, we propose a depth consistency module to enhance 3D scene
consistency. Finally, the 3D scene serves as initial points for optimizing
Gaussian splats. Experimental results demonstrate ART3D’s superior performance
in both content and structural consistency metrics when compared to existing
methods. ART3D significantly advances the field of AI in art creation by
providing an innovative solution for generating high-quality 3D artistic
scenes.

Figure 1: We introduce ART3D, a novel framework for generating 3D artistic
scenes. We can render 3D images with consistent views from predefined camera
trajectories by inputting text descriptions. ART3D harnesses the powerful
generation capabilities of the diffusion model and state-of-the-art 3D
Gaussian splatting, fostering advancements in AI-driven art creation.

††footnotetext:  †Corresponding author.

##  1 Introduction

With the progress of Artificial Intelligence Generated Content (AIGC) and
advancements in 3D vision technology, AI-driven art creation [11, 6, 30] has
become a hot topic for designers and artists. The latest generative models
[37, 9] in the realm of 2D art are capable of producing high-quality images
based on artistic prompts, making it easier for users to create 2D art and
promoting the growth of AI art. However, despite success in the 2D field, 3D
art creation still faces significant challenges.

The current approaches face a series of challenges when attempting to apply
similar generative models to create 3D models from textual descriptions. While
some emerging methods [33, 35, 52] strive to address the issues from text to
3D models, significant difficulties arise due to the lack of available 3D art
training data [4, 1], especially data tailored to artistic creations rather
than real-world scenes. These methods encounter substantial challenges when
dealing with domain differences between real-world scenes and artistic works.
Furthermore, some approaches [43, 22, 10] adopt a two-step process of
generating images first and then converting them to 3D. However, the 3D
information required by these models is often based on training with real-
world datasets, making accurate information prediction for artistic inputs a
difficult task. All these challenges contribute to the fact that research on
3D technology in artistic creation is still in its early exploratory stages.

In this paper, we aim to explore a novel research question: how to generate
high-quality 3D scenes with artistic styles based on text or reference RGB
images. As shown in Figure 1, leveraging the powerful generative capabilities
of Stable Diffusion models [37], we introduce an innovative 3D Gaussian
splatting [12] technique to produce artistic 3D scenes from user-provided
textual input by updating a point cloud map.

Specifically, we propose an image semantic transfer algorithm to bridge the
gap between artistic and realistic images. This algorithm extracts feature
maps with the same semantic layout obtained through UNet from user-provided
text or reference images. Subsequently, we use text prompts to generate new
realistic images, ensuring they adhere to the same semantic layout. While
realistic images play a role in the intermediate layers, once we predict the
depth information of realistic images, we can transfer this information to
their corresponding artistic image. Using the depth map and the initial
artistic image to generate an initial point cloud, we successfully address the
domain gap between artistic and realistic images.

Next, we generate a reasonable intrinsic matrix and camera pose, using the
camera pose to reproject the initial point cloud onto a new camera plane.
Employing the inpainting technique of the Stable Diffusion model [37], we
complete the hollow areas of the novel view image, ultimately obtaining a
complete image. By iteratively performing point cloud generation and
projection, we finally obtain an initial point for the scene point cloud
optimized using 3D Gaussian splatting [12]. In this process, we introduce a
depth consistency module to enhance consistency between multiple views.
Aligning depth domains, this module ensures seamless integration of the newly
generated point cloud into the entire system, improving consistency in multi-
view generation. Ultimately, we render an impressive 3D scene based on a
continuous representation of the point cloud map using 3D Gaussian splatting.

Our experiments demonstrate that our approach excels in both style consistency
and continuity metrics for artistic images compared to existing methods,
exhibiting greater visual appeal. We further include the ablation studies
involving different core components. Finally, we extend our method to various
application scenarios, such as text-driven fusion of different artistic
styles. This paper provides an innovative approach to high-quality artistic 3D
scene generation based on textual descriptions or reference images, making
significant contributions to the intersection of art and technology. Our
contributions can be summarized as follows:

  * •

We introduce ART3D, which achieves high-quality 3D artistic scene generation
through diffusion models and 3D Gaussian splatting techniques.

  * •

Our method compensates for the domain gap between artistic and realistic
images through an image semantic transfer algorithm, and the introduction of a
depth consistency module improves the overall consistency of global scene
generation.

  * •

We innovatively address the generation of high-quality 3D artistic scenes from
text or reference images, making a significant contribution to the development
of the interdisciplinary field of AI in art creation.

Figure 2: Pipeline of ART3D. We introduce the process of our method. First, we
input textual descriptions or reference images and use semantic transfer
algorithms to obtain accurate depth 3D information. Then, construct a point
cloud and enhance multi-view consistency through consistency/alignment
algorithms. Finally, we render high-quality 3D artistic scenes using 3D
Gaussian splatting technology.

##  2 Related Works

In this section, we provide a brief review of the literature related to
diffusion models, text-guided 3D generation and 3D scene synthesis.

Diffusion Models. In recent years, large-scale models have become a hot topic
in the field of AI. Among them, diffusion models [8, 37] are widely recognized
as powerful tools in the areas of complex data modeling and generation. Their
remarkable and stable capabilities in complex data modeling have led to
successful applications in various domains, including image restoration [23],
image translation [38], image /video generation [13, 19], super-resolution
[40], and image editing [28, 18, 20]. P2P [7] and PnP [46] further explore the
crucial role of attention mechanisms and features in the process of image
generation, proposing the use of cross-channel attention or self-attention
mechanisms for global image editing tasks. Recent research indicates that
these techniques have successfully extended from the 2D domain to the 3D
domain, as demonstrated in [24, 33]. Our approach maximizes the advantages of
diffusion models in both 2D and 3D domains, leveraging their powerful
restoration and generation capabilities, as well as feature mechanisms. This
has led to successful applications in the field of AI artistic creation.

Text-to-3D. With the emergence of pre-trained text-conditioned image
generation diffusion models, the trend of extending these 2D models to achieve
text-to-3D generation has become increasingly popular. Some methods employ
supervised training of text-to-3D models using 3D data [4, 1, 53]; however,
due to the lack of large-scale aligned text and 3D datasets, this direction
remains challenging. To achieve open-vocabulary 3D generation, pioneering work
by DreamFusion [33] has paved the way, where some approaches propose the
application of text-to-image diffusion models [37]to 3D generation [49, 52,
15, 3, 45]. Subsequently, numerous extensive improvements have been made to
DreamFusion [21, 35, 44, 49]. ProlificDreamer [49] introduces Variational
Score Distillation (VSD) and generates high-fidelity texture results. Magic3D
[21] adopts a coarse-to-fine strategy and utilizes DMTET [42] as a 3D
representation to achieve texture refinement through SDS loss. Recent methods
[21, 27, 29, 47] combine large-scale text-to-image diffusion models [39, 37]
with neural radiance fields [31] to generate 3D objects without the need for
training. However, most of these methods are tailored to real images or common
stylistic works. Approaches for text-driven 3D scene generation, particularly
for diverse artistic styles, remain largely unexplored.

3D Scene Synthesis. The typical methods for presenting 3D scenes include
explicit approaches like point clouds, meshes, and voxels. Recently, there has
been a focus on implicit methods, such as signed distance functions [32, 32]
and neural radiance fields [31], for expressing 3D scenes. The 3D Gaussian
splatting technique [12], by cleverly combining Gaussian splats, spherical
harmonics, and opacity, achieves a fast and high-quality reconstruction of
complete and boundaryless 3D scenes. Its features align well with our approach
to freely generate scenes, so we opt to use this rendering method in this
paper.

##  3 Method

In this section, we provide a detailed overview of ART3D, which consists of
four key components, as illustrated in Figure 2. Firstly, we leverage the
attention mechanism of Stable Diffusion models [37] and design an image
semantic transfer algorithm to enhance the accuracy of obtaining depth
information from artistic images. In Section 3.2, we establish a point cloud
map that transforms depth information $D_{i}$ into a point cloud, and through
camera reprojection, generates new images $I_{i+1}$. Subsequently, we apply
the operations from the first part to the inpainting image $\hat{I}_{i+1}$ to
obtain depth map $D_{i+1}$. In Section 3.3, we introduce a depth consistency
module to improve consistency between multiple views and seamlessly integrate
the new point cloud into the existing point cloud map. Finally, employing the
3D Gaussian splatting technique, we successfully render high-quality 3D
artistic scenes and novel views.

###  3.1 Image Semantic Transfer

The pre-trained stable diffusion model [37] can generate an artistic image
with a corresponding style based on input prompts such as mosaic, Van Gogh,
etc. However, simply appending keywords of realistic styles at the end of the
prompts makes it nearly impractical to generate images that align with
semantic information due to the stochastic nature of the diffusion model.
Currently, conditional generation methods like ControlNet [51], which leverage
depth and edge information as conditional inputs, struggle when applied to
artistic images, as these artistic images inherently possess fuzzy
characteristics. Therefore, adopting such an approach is still challenging.

To address this issue and generate real images with a semantic layout similar
to artistic images, similar to [26], we utilize the internal features of the
Stable Diffusion model to align the semantic information of the two images.
The intermediate layers of the UNet in the Stable Diffusion model consist of
residual blocks, self-attention layers, and cross-attention layers. The self-
attention layer of the $i$-th block at time $t$ computes features as follows:

${\boldsymbol{f}}_{t}={\boldsymbol{A}}^{i}_{t}{\boldsymbol{V}}_{t}^{i},{\boldsymbol{A}}_{t}^{i}=\text{Softmax}(\dfrac{{\boldsymbol{Q}}^{i}_{t}{\boldsymbol{K}}_{t}^{i^{T}}}{\sqrt{d}}),$
(1)

where
${\boldsymbol{Q}}^{i}_{t},{\boldsymbol{K}}^{i}_{t},{\boldsymbol{V}}^{i}_{t}$
represent the query, key, and value.

Similar to PnP [46], we note that the self-attention maps
${\boldsymbol{A}}_{t}$ at each time step $t$ during the denoising process in
Stable Diffusion control the spatial structure of the resulting image.
Therefore, while generating the artistic image ${\boldsymbol{x}}$, we retain
the intermediate self-attention maps ${\boldsymbol{A}}_{t}$ for all time steps
$t$. Injecting the output residual block features ${\boldsymbol{f}}_{t}$ is
employed to enhance structural alignment. Specifically, we store the features
from the 4-th layer in the output blocks. Then, we use the self-attention maps
during the denoising process of Stable Diffusion to control the layout of the
generated image. We inject the output residual block features
${\boldsymbol{f}}_{t}$ and self-attention maps ${\boldsymbol{A}}_{t}$ into the
UNet module. This process can be represented as follows:

$\displaystyle\begin{aligned}
{\boldsymbol{x}}_{t-1},{\boldsymbol{A}}_{t},{\boldsymbol{f}}_{t}&={\boldsymbol{\epsilon_{\theta}}}({\boldsymbol{x}}_{t},{\boldsymbol{c}},t),\end{aligned}\
$ (2) $\displaystyle\begin{aligned}
\hat{{\boldsymbol{x}}}_{t-1}&={\boldsymbol{\hat{\epsilon}_{\theta}}}(\hat{{\boldsymbol{x}}}_{t},\hat{{\boldsymbol{c}}},t;{\boldsymbol{A}}_{t},{\boldsymbol{f}}_{t}),\end{aligned}$
(3)

where ${\boldsymbol{\epsilon_{\theta}}}$ is a standard denoising UNet.
${\boldsymbol{x}}_{t}$ and $\hat{{\boldsymbol{x}}}_{t}$ are noisy images at
time step $t$ corresponding to the artistic and realistic images,
respectively. ${\boldsymbol{\hat{\epsilon}_{\theta}}}$ is the modified UNet,
which takes injected features as input.

During the generation process, the alignment of semantic features between
artistic and realistic images is guaranteed by leveraging the stored features
${\boldsymbol{f}}_{t}$ and self-attention maps ${\boldsymbol{A}}_{t}$.

###  3.2 Point Cloud Map

Our approach achieves view-consistent 3D scene generation by updating a point
cloud map. Initially, we set up the camera’s intrinsic and extrinsic
parameters as $T$. Next, utilizing a monocular scale depth estimation model,
ZoeDepth [2], we acquire a depth map $D_{i}$ and lift pixels of $I_{i}$ to 3D
space based on depth information, thus creating an initial point cloud
$P_{i}$. Along the camera’s trajectory, we project the point cloud $P_{i}$ and
then obtain a new image $I_{i+1}$ by reprojecting it back onto the camera
plane. This process can be represented as follows:

$P_{i}={\mathbf{\phi}}(I_{i},D_{i},T),I_{i+1}={\mathbf{\psi}}(P_{i},T),$ (4)

where ${\mathbf{\phi}}$ and ${\mathbf{\psi}}$ denote the projection from 2D to
3D and the reprojection from 3D to 2D.

Subsequently, we utilize a Stable Diffusion model [37] to inpainting the
missing areas in $I_{i+1}$. We then repeat the aforementioned process
${\mathbf{\phi}}$ and ${\mathbf{\psi}}$, obtaining scene images from every
viewpoint along the camera’s trajectory. Due to the independence among these
3D point clouds and the discontinuity in the depth estimation model, the scale
information among these point clouds varies. Therefore, in Section 3.3, we
introduce an efficient depth consistency algorithm to standardize the initial
depth scale information obtained from the depth estimation model, ensuring the
maintenance of a scale-consistent point cloud map.

Next, it is essential to align the projected point clouds corresponding to
each pose to ensure the generation of a continuous 3D artistic style scene.
The 3D point cloud $P_{i}$, projected from the camera pose of the previous
time step, results in the image $I_{i+1}$. After inpainting, a new image
$\hat{I}_{i+1}$ (representing the point cloud to be aligned) is obtained from
$I_{i+1}$. In $\hat{I}_{i+1}$, a significant portion of content overlaps with
$I_{i}$, and we denote this overlapping region as the $M$. Consequently, the
3D point clouds $P_{i+1}$ and $P_{i}$, projected from the current pose of
$\hat{I}_{i+1}$, also share many redundant points. The point alignment can be
represented as follows:

$\min_{s}\|M\times({\mathbf{\phi}}(I_{i},S\times D_{i},T)-P_{i+1})\|_{1}.$ (5)

We obtain the aligned coarse global factor $S$ by optimizing the two point
cloud regions using $M$. Due to our earlier depth consistency module, the
aligned point cloud maintains global and local consistency.

Figure 3: We demonstrate the effectiveness of our depth consistency module.
Frames from (a) to (b) represent consecutive frames, while (c) and (d)
illustrate depth value slices evolving over time at the red line in (a) and
(b). A smoother x-t slice indicates more consistent depth. Our approach
significantly enhances depth consistency.

###  3.3 Depth Consistency Module

According to the analysis in Section 3.2, achieving a consistent 3D point
cloud scene necessitates relying on consistent depth information. Due to the
independence of predictions from depth estimation networks, the depth
consistency algorithm faces two challenges: the global distribution of depth
ranges for different depths is inconsistent, making it impractical to simply
employ a global scaling factor $S$ for standardization. Additionally, methods
[25, 16, 17] that involve the re-optimization of depth information based on
camera poses require RGB domain image information, making their application
challenging in the artistic domain.

To address these challenges, we propose DCM, which eliminates the dependence
on RGB domain image information, thereby enhancing depth consistency across
different views of the same 3D scene. Specifically, DCM takes the depth
$D_{i}$ from the previous view and the initial depth $D_{i+1}$ from the
current view as input, producing the updated depth $D_{i+1}$ as output. This
can be represented as follows:

${D}_{i+1}={D}_{i}+DCM\left({D}_{i+1},{D}_{i}\right).$ (6)

Based on this formula, DCM attempts to learn a depth residual to update $D$
and minimize the inconsistency between $D_{i}$ and $D_{i+1}$. To ensure
computational efficiency, we employ shallow convolutional layers and residual
blocks to implement DCM. Its architecture comprises an encoder and a decoder,
connected via skip connections.

While the structure of DCM is concise, we encounter a scarcity of real multi-
view datasets containing ground truth depth from different perspectives,
posing challenges to DCM’s training. Considering that our scenario only
involves static scenes without dynamic objects, we train the model using the
virtual multi-frame dataset IRS [48]. This dataset includes highly accurate
depth information for static scenes. Specifically, we select a set of 7 frames
as training data and the initial depth map is resized and cropped to a
resolution of 384. To improve the depth accuracy, we also employ a depth
domain loss similar to MiDaS [36], which constrains the same depth range.
Additionally, we design a depth consistency loss as follows:

$D_{i}\Leftarrow D_{i}+DCM(D_{i+1},D_{i}),$ (7)
$\mathcal{L}_{C}=\sum_{i=1}^{7}\left\|M_{s}\left(D_{i+1}-\hat{D}_{i}\right)\right\|_{1},$
(8)

where $M_{i}=\exp(-\alpha|O_{i+1}-\hat{O}_{i}|^{2})$ represents the occlusion
weight between $D_{i+1}$ and $\hat{D}_{i}$, and $\alpha$ is set to $50$.
$O_{i}$ and $O_{i+1}$ are RGB frames. $\hat{D}_{i}$ is derived by warping
$D_{i}$ to $D_{i+1}$ based on the backward optical flow $F_{i+1\Rightarrow
i}$, which is computed using GMFlow [50]. Furthermore, $\hat{O}{i}$ is
obtained by warping $O_{i}$ to $O_{i+1}$ according to $F_{i+1\Rightarrow i}$.
This involves utilizing the optical flow to transform visualized images,
ensuring alignment between $O_{i}$ and $O_{i+1}$. It is important to note that
$D_{i}$ is warped by the optical flow $F_{i+1\Rightarrow i}$.

As shown in Figure 3, we demonstrate how our approach effectively enhances
depth consistency, where (c) and (d) represent x-t slices of scene motion,
with more smooth indicating better depth consistency.

Figure 4: Visualization results of our method. (a) represents the inputs,
where we can use only text or a combination of a reference image and text as
input. (b) is a novel view image rendered from the generated 3D artistic
scene, showing stylistic consistency. (c) demonstrates 3D artistic scenes
generated by our method through a predefined camera trajectory. Our approach
can accurately and high-quality generate structurally consistent and diverse
3D artistic scenes.

###  3.4 3D Gaussian Splatting for Rendering

Following the point cloud alignment stage, we obtain a complete point cloud
map, which serves as the initial point cloud for the Structure from Motion
(SfM) required in training the 3D Gaussian splatting [12] model. Each Gaussian
splats point is initialized with these point cloud values and optimized for
volume and position using the ground truth from the projection images as
supervision. In contrast to traditional SfM methods [41], which may exhibit
shortcomings in the domain of artistic images, our approach not only yields
more accurate initial point clouds but also accelerates the convergence of the
network, facilitating deeper learning of more nuanced features. Additionally,
considering that regions generated by the Stable Diffusion Inpainting model
[37] may contain inaccurate information, we intentionally ignore these areas
when computing the loss function. Given that points in the point cloud are
represented by Gaussian distributions, these regions will naturally be filled
during the training process.

Figure 5: Qualitative comparison with 3D scene generation methods.
LucidDreamer [5] and Text2Room [10] perform poorly on artistic images due to
the gap between the artistic and realistic domains. As highlighted in the red
circle, they struggle to obtain accurate 3D information, leading to structural
errors caused by depth accuracy or alignment. These issues result in the
generation of unsatisfactory 3D scenes. In contrast, our method excels in the
artistic domain.

##  4 Experiments

In this section, we present the details of our experiments, the results of
comparisons, and qualitative results of ART3D.

###  4.1 Experiment Setup

#### Implementation Details

For our text-to-image model, we adopt the Stable Diffusion [37] model and its
inpainting version, fine-tuned on the image inpainting task with an additional
mask input. Concurrently, we employ the ZoeDepth [2] model as our monocular
depth estimator, representing the SOTA in scale-depth estimation models.
Subsequently, we provide a detailed overview of the training parameters for
our DCM below.

#### Details of DCM

The DCM adopts an encoder-decoder architecture, where the encoder consists of
two downsampling strided convolutional layers, followed by five residual
blocks. In the decoding process, two transposed convolutional layers are
employed, incorporating essential skip connections from the encoder to the
decoder. Instance Normalization is consistently applied, with the exception of
the last layer. To confine the output within the range of -1 to 1 after
decoding, a Tanh layer is employed. The model is implemented using PyTorch and
is trained with the Adam solver [14] for 20,000 iterations, maintaining a
steady learning rate of 1e-4. Throughout the training phase, a batch size of 4
is utilized, and the training data undergoes random cropping to dimensions of
384×384.

#### Evaluation Metrics

We need to ensure that the rendered images maintain consistency with the
reference images’ styles. To achieve this, we employ CLIP-I [34] metrics to
calculate the similarity of image features. The second metric needs to
evaluate the consistency between the rendered image and textual descriptions.
Additionally, we utilize the CLIP-T metric to calculate the cosine similarity
between text prompts and CLIP embeddings. Furthermore, user studies are
conducted to evaluate the feasibility of our approach comprehensively.

Method | 2D Metrics | User Study
---|---|---
CLIP-I $\rightarrow$ | CLIP-T $\rightarrow$ | SC $\rightarrow$ | CC $\rightarrow$
Text2room [10] | 53.44 | 21.32 | 2.78 | 2.53
LucidDreamer [5] | 64.43 | 25.87 | 3.65 | 4.11
Ours w/o IST | 62.86 | 24.32 | 3.57 | 3.98
Ours w/o PCM | 60.36 | 23.34 | 3.72 | 3.78
Ours w/o DCM | 63.32 | 24.59 | 3.46 | 3.89
Ours | 68.15 | 26.81 | 4.01 | 4.32

Table 1:  Quantitative comparison. We report image metrics and user study
results, including ClIP-I and CLIP-T Score. We also use two metrics for user
study: Structural Consistency Score and Content Consistency. Our method
creates 3D artistic scenes with the highest quality. Figure 6: The depth map
directly estimated from the artistic image (a) lacks numerous details. In (b),
our image semantic transfer algorithm can generate a more accurate depth map.
This addresses the challenge of obtaining precise 3D information to generate
3D artistic scenes. Figure 7: We demonstrated the effectiveness of our depth
consistency module. In (a), we optimize only the global depth scale factor
$S$, leading to discontinuities and structural distortions within the white
region. After processing with our depth consistency algorithm, the generated
3D scenes exhibit improved consistency between different views.

###  4.2  Qualitative Results

As shown in Figure 4, we illustrate the results generated by our method when
provide with multiple sets of artistic textual descriptions. Our ART3D
successfully produces consistent 3D artistic scenes, as depicted in (d).
Leveraging the generative capabilities of the diffusion model, our approach
achieves stylistic coherence in scene generation based on textual prompts.
Additionally, as shown in (b), we can render images from various perspectives
by incorporating given camera poses, presenting a novel approach for novel
view image synthesis tasks.

To further assess the superiority of our method in text-driven 3D artistic
scene generation, we conduct a comparison with another method utilizing the
diffusion model for 3D scene generation [5, 10] (Figure 5). In contrast to
approaches applied in real-world domains or those closely simulating real-
world texture, our method produces more continuous multi-view images and more
consistent and plausible 3D scenes. The structural consistency in our
generated 3D scenes surpasses that of other methods, which may exhibit errors
in spatial positioning due to depth alignment issues, as indicated within the
red circles in Figure 5. Specifically, LucidDreamer [5] exhibits some depth
alignment problems, while Text2Room [10] shows reduced robustness for outdoor
scenes, leading to potential inaccuracies in structural information.

###  4.3 Quantitative Results

As shown in Table 1, we present the averaged quantitative results for multiple
scenes. We generate ten distinct styles of 3D artistic scenes using textual
descriptions and render 50 new scene images from each scenario for calculating
image quality metrics. CLIP-I [34] and CLIP-T scores are employed to assess
the similarity between generated images, reference images, and textual
descriptions. Simultaneously, we conduct a user study, selecting 25 sets of
images for 20 users to rate (0 to 5 represents worst to best score).
Specifically, we focus on structural consistency metrics(SC), such as the
presence of incoherent elements in the scenes, such as holes, and content
consistency metrics (CC), assessing the alignment of images with textual
descriptions. Across all these metrics, our approach consistently achieves the
highest scores, highlighting its ability to generate structurally consistent
high-quality 3D artistic scenes accurately.

##  5 Ablation Studies

The key components of our method include the image semantic transfer algorithm
(Section 3.1), point cloud map (Section 3.2), and depth consistency (Section
3.3). We conduct an extensive ablation study to validate the effectiveness of
each component.

#### Effects of Image Semantic Transfer

As we know, most depth estimation models are typically trained on real
datasets, posing challenges in accurately estimating depth information for
artistic images. However, accurate depth information is crucial for generating
point clouds, ensuring the fidelity of the generated scenes. Our designed
image semantic transfer algorithm can generate realistic scene images with the
same semantic layout as corresponding artistic style images. As shown in
Figure 6, directly predicted depth map (a) struggles to capture scene
information accurately, while our method can generate a depth map (b) with
rich details, leading to more accurate point clouds and avoiding issues like
object misplacement. Our approach effectively bridges the gap between artistic
and real-world image applications and provides new inspiration and insights
for AI artistic creation.

#### Effects of Point Cloud Map

We evaluate the point cloud generation strategy we employed for 3D Gaussian
Splatting initialization. We compare it with point clouds obtained using
COLMAP [41]. For the task of artistic scene generation, our point cloud
exhibits exceptionally high quality, providing a substantial number of high-
quality points for 3D Gaussian Splatting initialization and significantly
accelerating the reconstruction speed. As shown in Table 2, we evaluate our
generated artistic results using image quality metrics, demonstrating the
effectiveness of our method in enhancing the rendering quality of artistic
scenes.

#### Effects of Depth Consistency Module

Due to the fact that the 3D information required for point cloud generation is
predicted from a monocular depth estimation model, inconsistencies in scale
and depth range exist in the predictions of monocular depth across consecutive
frames. Our proposed monocular depth consistency algorithm ensures the
generation of a coherent 3D scene, thereby avoiding issues such as
discontinuities, holes, gaps, and distortions in the final 3D representation.
This algorithm learns residual information between depths from two different
viewpoints, effectively unifying the depth range and scale. In Figure 7 (a),
we present visual results where a simple global scaling factor for depth
alignment leads to noticeable discontinuities and distortions in the 3D
structure within the white boxes.

Metric | Method | 1000 iters | 4000 iters | 7000 iters
---|---|---|---|---
PSNR $\uparrow$ | COLMAP | 21.125 | 22.287 | 23.138
| Ours | 23.032 | 23.613 | 24.041
SSIM $\uparrow$ | COLMAP | 0.810 | 0.834 | 0.843
| Ours | 0.838 | 0.847 | 0.863
LPIPS $\downarrow$ | COLMAP | 0.258 | 0.237 | 0.224
| Ours | 0.229 | 0.221 | 0.214

Table 2:  Quantitative comparison. We compare our method with the 3D artistic
scene reconstruction results obtained using COLMAP. Underline shows that we
achieve competitive performance with few iterations. Our approach optimizes 3D
Gaussian splats more efficiently and exhibits superior reconstruction metrics.

##  6 Conclusion

In conclusion, ART3D represents an advancement in AI-driven 3D art creation.
By effectively addressing challenges in domain gaps and global scene
consistency, our approach, utilizing diffusion models and 3D Gaussian
splatting, excels in generating high-quality 3D artistic scenes from textual
descriptions. Beyond quantitative metrics, ART3D significantly contributes to
the intersection of AI and art by providing a novel solution for creating
visually appealing 3D scenes.

## References

  * Bautista et al. [2022] Miguel Angel Bautista, Pengsheng Guo, Samira Abnar, Walter Talbott, Alexander Toshev, Zhuoyuan Chen, Laurent Dinh, Shuangfei Zhai, Hanlin Goh, Daniel Ulbricht, et al.  Gaudi: A neural architect for immersive 3d scene generation.  _Advances in Neural Information Processing Systems_ , 35:25102–25116, 2022.
  * Bhat et al. [2023] Shariq Farooq Bhat, Reiner Birkl, Diana Wofk, Peter Wonka, and Matthias Müller.  Zoedepth: Zero-shot transfer by combining relative and metric depth.  _arXiv preprint arXiv:2302.12288_ , 2023.
  * Chan et al. [2023] Eric R Chan, Koki Nagano, Matthew A Chan, Alexander W Bergman, Jeong Joon Park, Axel Levy, Miika Aittala, Shalini De Mello, Tero Karras, and Gordon Wetzstein.  Generative novel view synthesis with 3d-aware diffusion models.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 4217–4229, 2023.
  * Chen et al. [2019] Kevin Chen, Christopher B Choy, Manolis Savva, Angel X Chang, Thomas Funkhouser, and Silvio Savarese.  Text2shape: Generating shapes from natural language by learning joint embeddings.  In _Computer Vision–ACCV 2018: 14th Asian Conference on Computer Vision, Perth, Australia, December 2–6, 2018, Revised Selected Papers, Part III 14_ , pages 100–116. Springer, 2019.
  * Chung et al. [2023] Jaeyoung Chung, Suyoung Lee, Hyeongjin Nam, Jaerin Lee, and Kyoung Mu Lee.  Luciddreamer: Domain-free generation of 3d gaussian splatting scenes.  _arXiv preprint arXiv:2311.13384_ , 2023.
  * Guo et al. [2023] Chao Guo, Yue Lu, Yong Dou, and Fei-Yue Wang.  Can chatgpt boost artistic creation: The need of imaginative intelligence for parallel art.  _IEEE/CAA Journal of Automatica Sinica_ , 10(4):835–838, 2023.
  * Hertz et al. [2022] Amir Hertz, Ron Mokady, Jay Tenenbaum, Kfir Aberman, Yael Pritch, and Daniel Cohen-Or.  Prompt-to-prompt image editing with cross attention control.  _arXiv preprint arXiv:2208.01626_ , 2022.
  * Ho et al. [2020] Jonathan Ho, Ajay Jain, and Pieter Abbeel.  Denoising diffusion probabilistic models.  _Advances in Neural Information Processing Systems_ , 33:6840–6851, 2020.
  * Ho et al. [2022] Jonathan Ho, William Chan, Chitwan Saharia, Jay Whang, Ruiqi Gao, Alexey Gritsenko, Diederik P Kingma, Ben Poole, Mohammad Norouzi, David J Fleet, et al.  Imagen video: High definition video generation with diffusion models.  _arXiv preprint arXiv:2210.02303_ , 2022.
  * Höllein et al. [2023] Lukas Höllein, Ang Cao, Andrew Owens, Justin Johnson, and Matthias Nießner.  Text2room: Extracting textured 3d meshes from 2d text-to-image models.  _arXiv preprint arXiv:2303.11989_ , 2023.
  * Jiang et al. [2023] Harry H Jiang, Lauren Brown, Jessica Cheng, Mehtab Khan, Abhishek Gupta, Deja Workman, Alex Hanna, Johnathan Flowers, and Timnit Gebru.  Ai art and its impact on artists.  In _Proceedings of the 2023 AAAI/ACM Conference on AI, Ethics, and Society_ , pages 363–374, 2023.
  * Kerbl et al. [2023] Bernhard Kerbl, Georgios Kopanas, Thomas Leimkühler, and George Drettakis.  3d gaussian splatting for real-time radiance field rendering.  _ACM Transactions on Graphics_ , 42(4), 2023.
  * Khachatryan et al. [2023] Levon Khachatryan, Andranik Movsisyan, Vahram Tadevosyan, Roberto Henschel, Zhangyang Wang, Shant Navasardyan, and Humphrey Shi.  Text2video-zero: Text-to-image diffusion models are zero-shot video generators.  _arXiv preprint arXiv:2303.13439_ , 2023.
  * Kingma and Ba [2014] Diederik P Kingma and Jimmy Ba.  Adam: A method for stochastic optimization.  _arXiv preprint arXiv:1412.6980_ , 2014.
  * Li and Li [2024] Pengzhi Li and Baijuan Li.  Generating daylight-driven architectural design via diffusion models.  _arXiv preprint arXiv:2404.13353_ , 2024.
  * Li and Li [2023] Pengzhi Li and Zhiheng Li.  Efficient temporal denoising for improved depth map applications.  In _Proc. Int. Conf. Learn. Representations, Tiny papers_ , 2023.
  * Li et al. [2023a] Pengzhi Li, Yikang Ding, Linge Li, Jingwei Guan, and Zhiheng Li.  Towards practical consistent video depth estimation.  In _Proceedings of the 2023 ACM International Conference on Multimedia Retrieval_ , pages 388–397, 2023a.
  * Li et al. [2023b] Pengzhi Li, Qinxuan Huang, Yikang Ding, and Zhiheng Li.  Layerdiffusion: Layered controlled image editing with diffusion models.  In _SIGGRAPH Asia 2023 Technical Communications_ , pages 1–4. 2023b.
  * Li et al. [2023c] Pengzhi Li, Baijuan Li, and Zhiheng Li.  Sketch-to-Architecture: Generative AI-aided Architectural Design.  In _Pacific Graphics Short Papers and Posters_. The Eurographics Association, 2023c.
  * Li et al. [2024] Pengzhi Li, Qiang Nie, Ying Chen, Xi Jiang, Kai Wu, Yuhuan Lin, Yong Liu, Jinlong Peng, Chengjie Wang, and Feng Zheng.  Tuning-free image customization with image and text guidance.  _arXiv preprint arXiv:2403.12658_ , 2024.
  * Lin et al. [2023] Chen-Hsuan Lin, Jun Gao, Luming Tang, Towaki Takikawa, Xiaohui Zeng, Xun Huang, Karsten Kreis, Sanja Fidler, Ming-Yu Liu, and Tsung-Yi Lin.  Magic3d: High-resolution text-to-3d content creation.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 300–309, 2023.
  * Liu et al. [2021] Andrew Liu, Richard Tucker, Varun Jampani, Ameesh Makadia, Noah Snavely, and Angjoo Kanazawa.  Infinite nature: Perpetual view generation of natural scenes from a single image.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 14458–14467, 2021.
  * Lugmayr et al. [2022] Andreas Lugmayr, Martin Danelljan, Andres Romero, Fisher Yu, Radu Timofte, and Luc Van Gool.  Repaint: Inpainting using denoising diffusion probabilistic models.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 11461–11471, 2022.
  * Luo and Hu [2021] Shitong Luo and Wei Hu.  Diffusion probabilistic models for 3d point cloud generation.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 2837–2845, 2021.
  * Luo et al. [2020] Xuan Luo, Jia-Bin Huang, Richard Szeliski, Kevin Matzen, and Johannes Kopf.  Consistent video depth estimation.  _ACM Transactions on Graphics (ToG)_ , 39(4):71–1, 2020.
  * Mahapatra et al. [2023] Aniruddha Mahapatra, Aliaksandr Siarohin, Hsin-Ying Lee, Sergey Tulyakov, and Jun-Yan Zhu.  Text-guided synthesis of eulerian cinemagraphs.  _ACM Transactions on Graphics (TOG)_ , 42(6):1–13, 2023.
  * Melas-Kyriazi et al. [2023] Luke Melas-Kyriazi, Iro Laina, Christian Rupprecht, and Andrea Vedaldi.  Realfusion: 360deg reconstruction of any object from a single image.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 8446–8455, 2023.
  * Meng et al. [2021] Chenlin Meng, Yang Song, Jiaming Song, Jiajun Wu, Jun-Yan Zhu, and Stefano Ermon.  Sdedit: Image synthesis and editing with stochastic differential equations.  _arXiv preprint arXiv:2108.01073_ , 2021.
  * Metzer et al. [2023] Gal Metzer, Elad Richardson, Or Patashnik, Raja Giryes, and Daniel Cohen-Or.  Latent-nerf for shape-guided generation of 3d shapes and textures.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 12663–12673, 2023.
  * Miah et al. [2023] Jonayet Miah, Duc M Cao, Md Abu Sayed, and Md Sabbirul Haque.  Generative ai model for artistic style transfer using convolutional neural networks.  _arXiv preprint arXiv:2310.18237_ , 2023.
  * Mildenhall et al. [2021] Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng.  Nerf: Representing scenes as neural radiance fields for view synthesis.  _Communications of the ACM_ , 65(1):99–106, 2021.
  * Park et al. [2019] Jeong Joon Park, Peter Florence, Julian Straub, Richard Newcombe, and Steven Lovegrove.  Deepsdf: Learning continuous signed distance functions for shape representation.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pages 165–174, 2019.
  * Poole et al. [2022] Ben Poole, Ajay Jain, Jonathan T Barron, and Ben Mildenhall.  Dreamfusion: Text-to-3d using 2d diffusion.  _arXiv preprint arXiv:2209.14988_ , 2022.
  * Radford et al. [2021] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al.  Learning transferable visual models from natural language supervision.  In _International conference on machine learning_ , pages 8748–8763. PMLR, 2021.
  * Raj et al. [2023] Amit Raj, Srinivas Kaza, Ben Poole, Michael Niemeyer, Nataniel Ruiz, Ben Mildenhall, Shiran Zada, Kfir Aberman, Michael Rubinstein, Jonathan Barron, et al.  Dreambooth3d: Subject-driven text-to-3d generation.  _arXiv preprint arXiv:2303.13508_ , 2023.
  * Ranftl et al. [2020] René Ranftl, Katrin Lasinger, David Hafner, Konrad Schindler, and Vladlen Koltun.  Towards robust monocular depth estimation: Mixing datasets for zero-shot cross-dataset transfer.  _IEEE transactions on pattern analysis and machine intelligence_ , 44(3):1623–1637, 2020.
  * Rombach et al. [2022] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer.  High-resolution image synthesis with latent diffusion models.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pages 10684–10695, 2022.
  * Saharia et al. [2022a] Chitwan Saharia, William Chan, Huiwen Chang, Chris Lee, Jonathan Ho, Tim Salimans, David Fleet, and Mohammad Norouzi.  Palette: Image-to-image diffusion models.  In _ACM SIGGRAPH 2022 Conference Proceedings_ , pages 1–10, 2022a.
  * Saharia et al. [2022b] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al.  Photorealistic text-to-image diffusion models with deep language understanding.  _Advances in Neural Information Processing Systems_ , 35:36479–36494, 2022b.
  * Saharia et al. [2022c] Chitwan Saharia, Jonathan Ho, William Chan, Tim Salimans, David J Fleet, and Mohammad Norouzi.  Image super-resolution via iterative refinement.  _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , 45(4):4713–4726, 2022c.
  * Schonberger and Frahm [2016] Johannes L Schonberger and Jan-Michael Frahm.  Structure-from-motion revisited.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 4104–4113, 2016.
  * Shen et al. [2021] Tianchang Shen, Jun Gao, Kangxue Yin, Ming-Yu Liu, and Sanja Fidler.  Deep marching tetrahedra: a hybrid representation for high-resolution 3d shape synthesis.  _Advances in Neural Information Processing Systems_ , 34:6087–6101, 2021.
  * Shih et al. [2020] Meng-Li Shih, Shih-Yang Su, Johannes Kopf, and Jia-Bin Huang.  3d photography using context-aware layered depth inpainting.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 8028–8038, 2020.
  * Tang et al. [2023a] Jiaxiang Tang, Jiawei Ren, Hang Zhou, Ziwei Liu, and Gang Zeng.  Dreamgaussian: Generative gaussian splatting for efficient 3d content creation.  _arXiv preprint arXiv:2309.16653_ , 2023a.
  * Tang et al. [2023b] Junshu Tang, Tengfei Wang, Bo Zhang, Ting Zhang, Ran Yi, Lizhuang Ma, and Dong Chen.  Make-it-3d: High-fidelity 3d creation from a single image with diffusion prior.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 22819–22829, 2023b.
  * Tumanyan et al. [2023] Narek Tumanyan, Michal Geyer, Shai Bagon, and Tali Dekel.  Plug-and-play diffusion features for text-driven image-to-image translation.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 1921–1930, 2023.
  * Wang et al. [2023a] Haochen Wang, Xiaodan Du, Jiahao Li, Raymond A Yeh, and Greg Shakhnarovich.  Score jacobian chaining: Lifting pretrained 2d diffusion models for 3d generation.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 12619–12629, 2023a.
  * Wang et al. [2021] Qiang Wang, Shizhen Zheng, Qingsong Yan, Fei Deng, Kaiyong Zhao, and Xiaowen Chu.  Irs: A large naturalistic indoor robotics stereo dataset to train deep models for disparity and surface normal estimation.  In _2021 IEEE International Conference on Multimedia and Expo (ICME)_ , pages 1–6. IEEE, 2021.
  * Wang et al. [2023b] Zhengyi Wang, Cheng Lu, Yikai Wang, Fan Bao, Chongxuan Li, Hang Su, and Jun Zhu.  Prolificdreamer: High-fidelity and diverse text-to-3d generation with variational score distillation.  _arXiv preprint arXiv:2305.16213_ , 2023b.
  * Xu et al. [2022] Haofei Xu, Jing Zhang, Jianfei Cai, Hamid Rezatofighi, and Dacheng Tao.  Gmflow: Learning optical flow via global matching.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pages 8121–8130, 2022.
  * Zhang et al. [2023] Lvmin Zhang, Anyi Rao, and Maneesh Agrawala.  Adding conditional control to text-to-image diffusion models.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 3836–3847, 2023.
  * Zhao et al. [2023] Minda Zhao, Chaoyi Zhao, Xinyue Liang, Lincheng Li, Zeng Zhao, Zhipeng Hu, Changjie Fan, and Xin Yu.  Efficientdreamer: High-fidelity and robust 3d creation via orthogonal-view diffusion prior.  _arXiv preprint arXiv:2308.13223_ , 2023.
  * Zhou et al. [2021] Linqi Zhou, Yilun Du, and Jiajun Wu.  3d shape generation and completion through point-voxel diffusion.  In _Proceedings of the IEEE/CVF international conference on computer vision_ , pages 5826–5835, 2021.