# Learning to write with the fluid rope trick

Gaurav Chaudhary  School of Engineering and Applied Sciences, Harvard
University, Cambridge, MA 02138.  Department of Mechanical Engineering,
Massachusetts Institute of Technology, Cambridge, MA 02139  Stephanie Christ
School of Engineering and Applied Sciences, Harvard University, Cambridge, MA
02138.  A John Hart  Department of Mechanical Engineering, Massachusetts
Institute of Technology, Cambridge, MA 02139  L Mahadevan
<EMAIL_ADDRESS>School of Engineering and Applied Sciences, Department
of Physics, Department of Organismic and Evolutionary Biology, Harvard
University, Cambridge, MA 02138.

###### Abstract

The range and speed of direct ink writing, the workhorse of 3d and 4d
printing, is limited by the practice of liquid extrusion from a nozzle just
above the surface to prevent instabilities to cause deviations from the
required print path. But what if could harness the “fluid rope trick”, whence
a thin stream of viscous fluid falling from a height spontaneously folds or
coils, to write specified patterns on a substrate? Using Deep Reinforcement
Learning we control the motion of the extruding nozzle and thence the fluid
patterns that are deposited on the surface. The learner (nozzle) repeatedly
interacts with the environment (a viscous filament simulator), and improves
its strategy using the results of this experience. We demonstrate the results
in an experimental setting where the learned motion control instructions are
used to drive a viscous jet to accomplish complex tasks such as cursive
writing and Pollockian paintings.

##  I Introduction

The rapid evolution of three-dimensional (3D) printing technology has enabled
new manufacturing capabilities[1], and typically involves layer-by-layer
deposition of material through a computer-controlled nozzle. While motion
control of the nozzle and development of materials with suitable rheological
properties for printing have sped up the adoption of 3D printing, from a
physical perspective, the ultimate limits on print quality are imposed by the
fluid dynamics of the printed material. Such effects are seen practically in
non-uniformly extruded/deposited material and instabilities such as folding
and coiling of fluid jets [2]. A simple protocol to prevent these defects
necessitates limiting the nozzle trajectory to exactly mimic the target print
pattern from a very small height offset.

While this approach increases the accuracy of layer-by-layer direct writing of
3D objects, the result is that printing is typically a very slow process.
Moreover, it is challenging to adapt these methods to complex topographies of
the substrate on which the deposited material is laid down, or to precisely
control sharp turns without distorting the extruded filament. Here, we explore
the possibility that 3D printing could be sped up by operating with a nozzle
that is elevated above the print surface, and harnessing the dynamic
instability of a falling fluid jet [3, 4, 5, 6, 7, 8] to enable rapid accurate
printing without requiring the nozzle to exactly mimic the target pattern.
This technique relies on finding printing solutions, i.e. nozzle paths in
space-time that take advantage of folding and coiling instabilities rather
than avoiding them. Thus, given a final target pattern or shape in a plane, we
ask how to optimize the nozzle trajectory so that a continuous stream of
material can be printed without having to precisely translate the nozzle along
the print path, or bringing the nozzle close to the substrate. Proof that this
can be done, and very well, is seen in the striking art of Jackson Pollock,
who created paintings by dripping and pouring paint on a canvas from a height
while moving his hand [9]. Can a machine be trained to learn this technique?

Our approach taps into the recent success of a class of machine learning
algorithms known as reinforcement learning methods [10] that are able to
harness the expressive power of neural nets (NN) to explore and exploit a
large state and action space to find solutions to difficult tasks. Our
starting point is a physical simulator that characterizes the necessary
physics of thin threads of viscous fluid extruded from a height and captures
the interactions between agent (nozzle head) and environment (print surface).
Then, by building a framework of learning through repeated interactions with
the environment, we show that it is possible to find a nozzle trajectory that
can print a target pattern.

##  II The dynamics of viscous filaments

Figure 1: Representation of the fluid thread and the associated material
frame. Figure 2: Printing with the liquid-rope instability. (A) The state of a
printing nozzle is represented by its translational coordinates $(X,Y,Z)$
w.r.t. a stationary frame. The azimuthal angle $\varphi$ and polar angle
$\theta$ further specify the ink extrusion direction (solid red arrow).
Simulation of viscous thread ($\eta=5000$ cP) coiling as the print head is
translated vertically at a constant speed. The top view shows coils of
increasing radii as the nozzle is raised from $z=0.5$ to $z=12.5$ cm.
Translating the nozzle at a constant speed of $0.5$ cm/s creates periodic
coils on the surface as shown for a fixed set of parameters. (B) In the
controlled setting, the agent (nozzle) interacts with the environment to
improve its action-selection policy $\pi$. It receives a state $s_{t}$ and
numerical reward $r_{t}$ from the environment, based on which it updates its
policy and returns the next action $a_{t}$. By taking action $a_{t}$, the
environment transitions to the next state $s_{t+1}$. It measures its
desirability through the reward $r_{t+1}$ and sends the information, along
with the new state, back to the agent.

The simulator used to model the liquid-rope instability of a viscous fluid
follows earlier work on characterizing the coiling of liquid jets[11], which
we summarize briefly. A thin stream of viscous thread is represented using a
discrete Lagrangian description that accounts for the narrow aspect ratio
(length/radius) of the viscous thread; then the centerline representation of
the coordinates provides a reduced order description for the fluid threads.
The centerline $x(S,t)\in{R}^{3}$ and the material frame
$[d_{1}(S,t),\,d_{2}(S,t),\,d_{3}(S,t)]$ is used to capture the time evolution
of viscous thread as shown in Fig. 1. The material frame is adapted to
centerline by requiring $d_{3}$ to align with the tangent to the centerline,
and $d_{1},d_{2}$ span the plane normal to centerline’s tangent.

The dynamical equations of the thread per unit length of the thread are given
by the balance of linear and angular momentum for the position $x(S,t)$ and
the spin component of the angular velocity at every cross-section
$v(S,t)$[11]:

$\displaystyle\rho\,A_{0}\,\ddot{x}(S,t)$ $\displaystyle=F_{v}(S,t)+f(S,t)$
(1) $\displaystyle l\,J\,\dot{v}(S,t)$ $\displaystyle=M_{v}(S,t)+m(S,t)$ (2)

Here, $F_{v}$ and $M_{v}$ are internal resultant viscous forces and moments in
the thread arising from the rates of stretching, bending and twisting of the
thread, while $f$ and $m$ are the external body force and body moment density,
respectively. The expressions for the resultant force and moments in the fluid
thread are expressed using the kinematics of the centerline representation in
terms of the local area $A_{0}$ and moments of inertia $J$ at every cross-
section $S$, as well as the viscosity of the fluid (assuming it be modeled as
a simple Newtonian liquid); we refer the reader to [11], [7] for the detailed
expressions of the individual terms in the above equation and their respective
discretizations, which form the basis for this work 00footnotetext: ‡
Codebase: https://gitlab.com/stchrist/ViscousThreads.

A critical dynamical variable underlying the generation of complex patterns
[12] is the ability to move the nozzle; we assume that it can be moved freely
in a plane using horizontal movements and also allow it to rotate relative to
two axes, as shown in Fig. 2 A, with the angles $\varphi$ and $\theta$
describing the rotational degrees of freedom (DoF). Assuming that the flow
rate ($Q$) and nozzle extrusion diameter $a$ are constant, controlling the
writing/painting pattern on the substrate requires that the agent (nozzle)
have time-varying translational and rotational velocities, likely to be
bounded within a range of practical relevance. Vertical motion of the nozzle
changes the nature of the pattern deposited; at low heights, the jet is
rectilinear, then starts to coil as the height is increased past a first
threshold, and then the jet coils rapidly as inertial effects start to
dominate at even larger falling heights. In Fig. 2 A (bottom), we show a
coiling pattern deposited on the surface ($z=0$); the coil radius increases as
the height of the nozzle is increased. When the agent is translated linearly
along the $y-$ axis with a constant velocity, following a short transient, we
see that a coiling pattern is deposited on the substrate $z=0$ similar to what
has been observed experimentally [12]. These examples suggest that to write a
desired pattern with such an agent, we have to actively control the various
DoFs of the nozzle. But how can we learn the form of the agent’s actions?

##  III A reinforcement learning framework

Reinforcement learning (RL) [10] provides one possible answer by allowing the
action of an agent to repeatedly interact with the environment and converge
towards an optimal policy without having any prior knowledge of the underlying
model. An RL problem is defined in terms of states, actions and rewards. The
state $s\in S$ is a quantitative description of the environment at the current
time, with $S$ being the set of all possible states. In every state, a set of
actions $A(s)$ are available to the agent. By taking an action $a\in A(s)$,
the agent transitions from one state to another and receives a numerical
scalar reward signal $r$ from the environment. The reward is a measure of how
desirable it is to take action $a$ in state $s$. Overall, the goal of the
learner (or agent) is to find an action policy $\pi(a|s)$ which maximizes the
cumulative reward over the entire learning episode by exploring different ways
of interacting with the environment; this schematic is summarized in Fig. 2 B.

For the printing problem, the current state is defined by the following
continuous parameters: the arc length $S$ of the pattern that has been printed
so far, the current position of the nozzle $(x,y,z,\varphi,\theta)$ with
$z\in[h_{\text{min}},h_{\text{max}}]$, and the last action, corresponding to
the motion of the nozzle $(v_{x},v_{y},v_{z},v_{\varphi},v_{\theta})$. If the
height of the nozzle $z$ is outside of its prescribed range, the state is
considered invalid. Just as the states, the actions (DoFs) that describe the
motion of the nozzle are continuous. The agent has up to five degrees of
freedom: translation velocities ($v_{x},v_{y},v_{z}$) and rotational velocity
($v_{\varphi},v_{\theta}$). Furthermore, we assume that the range of all
actions is bounded within the interval
$\left[v_{\text{min}},v_{\text{max}}\right]$. For the position and motion of
the nozzle, only the DoF that are part of the action are considered, as the
other parameters stay constant. Thus, the dimension of the state is between 3
(one DoF) and 11 (five DoF). Theoretically, it would be sufficient to use the
arc length as the only only state variable, however, we find that the rewards
converge very slowly in that case.

More specifically, we use an off-policy actor-critic named V-RACER [13] as our
RL framework, although many other methods are likely to also work. A summary
of this follows: the algorithm trains a neural network to approximate a
continuous path $\pi^{w}(a|s)$ for the nozzle (continuous policy
approximation). The policy network is randomly initialized and then
iteratively updated through repeated attempts to reach the target following
the policy gradient theorem. We employ ”Remember and Forget Experience Replay”
to reuse past experiences over multiple iterations to update the policy in a
stable and data-efficient manner, with hidden network layers with 128 LSTM
(long short-term memory) units each. This representational capacity of the
network was found to be sufficient for the cases considered in this work. The
actuation period (temporal resolution) of our simulations was generally fixed
to 0.01 s; varying the number of units and the number of layers did not change
the final optimized solutions.

Since the goal of the learning to write is equivalent to depositing a 2D
pattern of the same shape as a target input pattern, the reward/penalty is
chosen so as to reflect the mismatch between the target and the printed
pattern. The target pattern is given as an ordered list of 2D coordinates. We
implemented the reward by comparing the position of the printed vertices with
vertices of the given target shape. Denoting the total arc length of the
pattern that has been created so far by $S_{i}$ – where $i$ is the current
learning step, the error is computed by integrating it from the last step to
the current step:

$r=-\int_{S_{i-1}}^{S_{i}}\lvert\tilde{\mathbf{x}}_{target}(S)-\tilde{\mathbf{x}}(S)\rvert\,dS$
(3)

Here we have chosen the error at a certain arc length in terms of the absolute
difference between the target position and the actual position, linearly
interpolated from their two respective neighbors. In addition to the reward
based on the pattern, we give a strong negative reward for invalid states,
i.e. for invalid heights $z$. Using this reward function, the maximum – and
optimal – cumulative reward is 0, as this would indicate that the created
pattern does not deviate from the target pattern at any point.

To create regular initial conditions for learning, the simulation is first
stabilized by using a constant velocity
$(v_{x},v_{y},v_{z},v_{\varphi},v_{\theta})=(v_{0},0,0,0,0)$ for a fixed
number of time steps until the pattern follows a simple straight line. From
there, the agent is allowed to start learning the appropriate set of actions
to replicate the target pattern. In practice, in addition to the simulation
parameters (nozzle flow/radius and fluid properties) and RL parameters
(learning rate, discount factor, size of NN, activation function), we also
specify the maximum number of steps per episode step $n_{s}$, i.e. the number
of actions that the agent can take before the episode is over.

##  IV Using reinforcement to learn writing

Figure 3: Using reinforcement learning to control the liquid coiling
instability. (A-i) The agent takes actions, $a_{t}=\\{v_{x},v_{y}\\}$ while
extruding liquid from a fixed height of 7 cm and learns to draw a straight
line over multiple iterations, seeking to maximize the total reward. The red
and green curves correspond of the maximum limit of 1 cm/s and 5 cm/s on the
velocity components. (A-ii) Maximum reward depends on the limits on
translation velocity, and the height from which the liquid is extruded. A
better control can be learned when extruding from a smaller height for the
same limits of velocity components. (A-iii) A density plot of the actions
taken at various episodes. To create a straight line, the agent learns to
suppress coiling by moving solely in the x-direction. (B) A more complex task
of drawing a varying amplitude wave using a larger set of actions,
$a_{t}=\\{v_{x},v_{y},v_{z}\\}$. Unlike the straight line in (A), the agent
harnesses the liquid coils to draw the curved pattern. (C) The performance of
the agent also depends on the relative scale of liquid coils and target
curvature. A comparison of the local curvature ($|\kappa|$) of the drawn
pattern at different scales show the challenge in learning when the scale of
target curvature is significantly smaller than the nature curvature of liquid
coils. (D-i) The constrained set of actions limits the agent’s performance in
drawing larger curvature. (D-ii) The fluid material properties, such as the
kinematic viscosity $\eta$, also limit the accuracy of the learned pattern.

To validate that the RL agent takes optimal actions, we first ask if we can
reproduce the simplified scenario of printing a straight line from a nozzle.
As shown earlier in Fig. 2 A, for an arbitrary unidirectional velocity, an
uncontrolled agent will produce a pattern consisting of overlaid coils. If the
nozzle is allowed to vary its planar velocity ($v_{x}$ and $v_{y}$), we see
that over time associated with an increase in the number of episodes, the
agent converges to create a straight line as shown in Fig. 3 A i. Consistent
with this, the cumulative reward plateaus with increasing episode number (Fig.
3 A ii); the value of the plateau depends on the height of fluid extrusion and
bounds on the nozzle actions. To draw a straight line, the agent speed must
counter the effective velocity at which the coils are naturally laid on the
surface, i.e. the optimal solution must satisfy the relations
$v_{x}\geq\sim\Omega R$, and $v_{y}\sim 0$. The action density plots in Fig. 3
A iii show that the agent does indeed converge to this state.

To truly learn to write, the agent must learn to negotiate curves with complex
curvature profiles. Therefore, we need a target pattern with a wide range of
curvatures and rate of variation of curvature; an exponentially decaying
sinusoidal wave $y=10\,e^{-1000\,x}\,\sin{2000\pi x}$ serves both purposes.
The agent is now allowed an additional action $v_{z}\neq 0$ in addition to the
ability to vary $v_{x},v)y$, and in Fig. 3 B we show the learned patterns as a
function of episode number, and see that after about 5 000 episodes, the
correct shape is learned, and the visible accuracy of the task does not
improve much beyond that. A closer look at the evolution of the learned
pattern shows that the agent first learns to match the pattern on larger
length scales, followed by further improvements to closely match the features
with sharp changes in curvature, i.e. on smaller length scales. This is
consistent with the cumulative reward first showing a sharp increase followed
by a very weak increase corresponding to the plateau-like regime in Fig. 3 A
ii) associated with later episodes.

An optimal strategy for the agent would aim to utilize or avoid the coiling
instability depending on the nature of the curvature profiles in the target
pattern. Thus, for curvatures comparable to the coiling radius, it makes sense
to use the natural coiling instability, and otherwise to avoid it by moving
quickly in the plane. However, when the (position and velocity) actions of the
nozzle are bounded, writing patterns with curvatures that are significantly
larger than the coil radius can be a challenge. To challenge the agent with
such patterns, we change the absolute scale of the target pattern, thus
changing the curvature profiles overall, and ask how well the RL agent, with
the same bounds on action, learns to draw patterns at different scales. In
Fig. 3 C we compare the absolute curvature of the target pattern with that of
the learned patterns as a function of scale, using the same exponential form
used in Fig. 3 B. For each case, the overall scale of the target pattern is
halved or doubled, keeping all other parameters fixed. We see that the
magnitude of curvature of the learned patterns is generally different from the
target curves at the locations of extreme curvature; the case with smallest
scales shows significant deviations from the target, implying that decreasing
the scale of the target pattern (hence increasing the curvature), results in a
reduction of the agent’s ability to print the target, consistent with
intuition.

The substrate patterns associated with the jet coiling instability are
affected by both the limits on the dynamics of the nozzle as well as the
properties of the fluid. To explore their respective roles, in Fig. 3 D ii we
show the effect of limiting the agent’s actions to $v_{x},v_{y}\neq
0,v_{z}=0$, and compare it to the case when $v_{z}\neq 0$ (Fig. 3 B); we see
that extruding fluid from a constant height introduces an unwanted coiling
response in the learned pattern, especially near the regions of large
curvature in the target pattern, demonstrating the influence of limiting the
action space in the learning problem. Fig. 3 D ii shows the effect of changing
the magnitude of the viscous forces by simply changing the kinematic viscosity
($\eta$) of the fluid; reducing the viscosity leads to an increase in the
coiling frequency in a predictable way and we see that after a similar number
of learning episodes, the learned pattern is not quite as accurate. These
tests of deploying RL point to a simple but important lesson; choosing the
right range of actions is critical for good performance in any task, and
becomes particularly clear in the context of interacting with and learning
using physical systems.

Figure 4: Using RL to reproduce cursive writing. The agent, with actions
$a_{t}=\\{v_{x},v_{y}\\}$, can harness liquid coiling to write cursive text.
(A) The predicted deposited patterns from fixed heights as shown. In each case
the z-component of velocity is set to zero. (B) A comparison of local
direction (local orientation w.r.t the horizontal $\theta$) of translation of
the nozzle and the printed pattern show that non-trivial actions are performed
to control the target pattern, and the agent does not just follow the
trajectory of target pattern. (C) The magnitude squared coherence between the
printed patterns and the target pattern is shown. A higher coherence is
observed at lower spatial frequencies. (D) The curvature of the cursive text
plotted as a function of arc length shows misaligned curvature peaks for
patterns deposited from different heights. The error in $\kappa$ vs $S$ curves
can be decomposed into vertical and a horizontal components (in E) using
elastic functional data analysis. (F) Changing the height of extrusion allows
for directly controlling the total printing time (inset). A speed-error trade-
off results as a consequence. Within a given range of actions, a lower error,
defined as $\varepsilon=(\int(1-\kappa(S)/\kappa_{0}(S))^{p}dS)^{1/p}$, is
achieved when nozzle is held closer to the surface as a result of low-coiling
frequency and smaller liquid coil radii. Here we define non-dimensional
printing speed $v^{*}=(S_{total}/t_{p})/\sqrt{gh}$

Following the calibration of the ability of the RL agent, a moving nozzle, to
learn to write out patterns suggests that it is possible to print any
continuous planar curve, given a sufficiently large and bounded action space.
To further explore the ability of RL in demonstrably difficult tasks, we now
demonstrate that RL can be used to guide a nozzle to mimic cursive writing or
paint like Pollock. We first prescribe the target pattern: a continuous curve
associated with the word “Cambridge” along with an action space
$(v_{x},v_{y})$, but no vertical velocity, i.e. $v_{z}=0$. In Fig. 4 A, we
show how the agent learns to write from different fixed heights in the range
$z\in[2,10]$ cm, after the same number of training episodes. While the overall
patterns look similar, a closer look shows noticeable differences, especially
in the regions of the curve with large curvature e.g. letters “r”, “d” and
“g”. Since the coil radius and frequency depend on the nozzle height, the
actions taken depend on this parameter. In regions of the pattern that are
almost straight (or weakly curved), the agent avoids the coiling instability
by moving rapidly; similarly, it uses the coiling instability to achieve
higher curvature in regions where the target pattern demands it.

To understand the discrepancies between the target and learned patterns, in
Fig. 4 B we show the local direction of motion of the agent and local
direction of the pattern that gets laid on the surface. It is clear that there
are two types of errors: at some places, the local orientation of curves
appears phase-shifted, while in other regions the amplitudes of the curves are
poorly correlated.

To quantify the accuracy of learning the print path, we take two approaches
that use local measures to inspire global metrics, noting that since we work
with planar curves, upto rigid motions (that we do not worry about), the
target and learned patterns are completely characterized by their scalar
curvature as a function of the arc-length. For one measure of the error on the
scale of the whole pattern, we define a coherence metric in the spatial
Fourier domain as follows :

$C_{\kappa_{0}\kappa}(f)=\frac{|P_{\kappa_{0}\kappa}(f)|^{2}}{(P_{\kappa_{0}\kappa_{0}}(f)\,P_{\kappa\kappa}(f))}$
(4)

where $P_{\kappa\kappa}(f)$ and $P_{\kappa_{0}\kappa_{0}}(f)$ are the power
spectral densities of the curvatures $\kappa_{0}(S)$ (target) and $\kappa(S)$
(learned), respectively, and $P_{\kappa_{0}\kappa}(f)$ is the cross power
spectral density between $\kappa_{0}(S)$ and $\kappa(S)$. [14], we note that
$C_{\kappa_{0}\kappa}(f)\in[0,1]$. Plotting the coherence in Fig. 4 C, we note
that $\kappa_{0}(S)$ and $\kappa(S)$ are strongly associated at very low
spatial frequencies (or relatively larger scales) while at higher spatial
frequencies the association is relatively weak. This is consistent with the
observations from other learning experiments as well: the agent is can capture
the large scale features of the curves either by moving in straighter or
slightly curved path or printing coils, but when the target curvature scales
are larger than the natural coiling scales, a mismatch between the target and
learned curvature may occur.

To compare the difference between the target and learned curvature along the
trajectory raises a familiar problem of registration - how does one align
points along the learned path with points along the target pattern, given that
the speed of the nozzle is not necessarily a constant? To solve this problem,
we use methods from functional and shape data analysis [15] whereby we
simultaneously solve the problem of registration and determination of error by
insisting on reparametrization invariance of the error metric. A natural
solution that presents itself is to align the curvature data for various cases
with the curvature of the target curve using the Fisher-Rao metric and the
square-root velocity function (SRVF) representation of curvature data [15],
thus separating out errors in the phase and the amplitude of the curvature.
Fig. 4 E shows the aligned curvature of different cases shows in Fig. 4 D and
their respective phase shift. Defining the error as the $l-p$ norm of the
curvature mismatch,

$\varepsilon=\left[\int\left(1-\frac{\kappa(S)}{\kappa_{0}(S)}\right)^{p}dS\right]^{1/p}$
(5)

Figure 5: Using RL to mimic Pollock’s paintings. Liquid coiling was
extensively utilized by famous American painter Jackson Pollock in his drip
paintings (as shown in (A) [16]). (B) With the available set of action
$(v_{x},v_{y},v_{z})$, the agent can learn to draw parts of Jackson Pollock’s,
_Figure, 1948_ , indicating that Pollock’s drip painting may owe its
complexity to the liquid coiling instability.

Based on the error defined above, we quantify agent’s capability to print from
various heights above the surface. Printing with different nozzle heights is
associated with a trade-off. Small nozzle heights lead to very accurate
learned paths but ones that are very slow, since they do not exploit the
instability, while larger nozzle heights lead to faster printing albeit with
poorer accuracy. In the inset of Fig. 4 F we quantify show that a higher
extrusion height results in a higher coiling frequency and hence faster
overall printing. In Fig. 4 F we show the trade-off by comparing the overall
curves printed from different heights; faster printing is also less accurate.
We note that there is an additional effect, since the filament diameter at the
substrate changes with the height of the nozzle, but here we ignore this and
only consider matching the target pattern with the centerline of the fluid
jet.

Figure 6: Testing the predicted RL strategies using physical experiments to
write. (A) An experimental print showing a pattern with a first region printed
with RL-derived and a second region without control where the nozzle is moved
at 0.5 cm/s. (inset) The pattern printed in simulation. (B) RL controlled
cursive handwriting using silicone oil, overlaid with the numerically
simulated solution. (C) An experimentally printed self-similar structure
(solid line), and the numerically learned pattern (dashed line). The mismatch
between the experiment and numerical result grows as the arc length grows
starting from the bottom left corner. (Inset) The experimental pattern
deposited with a noisy set of actions shows the sensitivity of the final
pattern to the RL actions. (D) A comparison of error in the curvature along
the entire arc length of the pattern in (C) is shown. Here, $\kappa_{c}(S)$
and $\kappa_{e}(S)$ are the curvatures of computed and experimentally printed
pattern, respectively, from the learned set of actions.(E) In a practical
setting, the RL agent can be deployed to automate processes such as printing
chocolate syrup on edible surfaces. (insets) a natural coiling of chocolate
syrup extruded from a stationary nozzle held at a height of 6.6 cm above the
surface and the top view of the final print.

Finally we ask if the agent can learn to not just write, but sketch, paint or
draw. As an exemplar, we chose part of a painting by Jackson Pollock (_Figure_
, 1948 [16])), famous for his ”painting at a distance” style, wherein he
allowed paint to drip and drizzle from a brush or a rod held far above the
canvas, thus exploiting both fluid coiling and jetting instabilities [9] as
shown in Fig. 5 A. Could an agent learn the complex movements of the artist
given the final result? Restricting ourselves to a limited part of the
painting which we could easily track as a continuous curve as shown in Fig. 5
B inset, we show that by allowing the action space for the RL agent to be non-
zero $v_{x},v_{y},v_{z}$ does in fact reproduce a fragment of the Pollockian
painting as shown in Fig. 5 B.

##  V Physical experiments testing how to learn writing

The ultimate test of the RL protocols described above is to use physical
experiments based on the learned strategies. To do so, we dispense a viscous
fluid, silicone oil ($\eta=500$ cP), through a nozzle under a controlled flow
rate, and deposit it along a digitally defined path on a smooth sheet of paper
glued on a flat, stationary platform. We use the RL control that was learned
in the numerical setting as the digital path along which the printer head
translates. Since a fixed time step, $\Delta t=0.01$ s was used in the
simulations, we find this digital path by integrating the action set
(velocities) at every step. The diameter of the nozzle and flow rate are same
as the numerical experiments. In all the experiments the oil is first extruded
until the coiling becomes steady, to ensure similar initial conditions as that
of the numerical simulations.

We start by showing the difference between the RL protocol and uncontrolled
flow, in Fig. 6 A (see S.I. video 1) for an experimental print shown in the
inset. Using the learned RL control, the agent is able to smoothly print the
pattern whereas when the agent moves at a constant velocity, fluid coiling
readily occurs, thus producing a mimic of the given target pattern. To test
the ability to write cursively, in Figure 6 B (see S.I. video 2) we show an
example parallel to the simulation results in Fig. 4 A using the action set
$v_{x},v_{y},v_{z}$, with RL controls based on allowable states for nozzle
heights $z\in[4-10]$ cm. An overlay of the numerically learned pattern on the
experimentally printed pattern shows that the regions of highest errors are
generally the regions of maximum curvature, most likely due to velocity
mismatch between the RL actions and the experimental implementation. In
particular when the nozzle makes sharp changes in direction over a short time
duration (typically nearly high curvature regions), the executed
(experimental) velocity actions are expected to be different from the desired
actions. As a consequence, the overall mismatch (error) between the numerical
and the experimental pattern builds up as a function of arc length of the
pattern. We emphasize this using a self-similar target pattern of a Peano
curve in Fig. 6 C (see S.I. video 3) using actions $v_{x},v_{y}$. We observe
that the experimental print and the simulated print show close agreement in
the initial part of arc length, but as the printing progresses, the apparent
experimental error accumulate over the arc length. A comparison between the
curvatures of computed pattern and the target pattern (computational error),
and experimentally printed pattern and computed pattern is shown in Fig. 6 D.
In both the cases the error is computed between the aligned curvature vectors
using the methodology described in the previous section. It is evident that
along the entire arc length, the error in the curvature of the computed
pattern deviates consistently from the target curvature. The error between the
computed and the experimentally printed pattern has even larger magnitude due
the additional experimental errors. In the inset of Fig. 6 C we show the
sensitivity of the resulting pattern on the error in optimal set of actions.
Here, we add a random noise to the actions ($<5\%$ of the action value), and
use the noisy actions to the print the pattern. The resulting pattern deviates
significantly, highlighting the non-trivial nature of the action-reward
landscape due to the unsteady and nonlinear effects in the physical problem.

As a test of the RL control approach developed for Newtonian fluids extruded
on smooth surfaces, we ask how well the strategy does when printing a thick
chocolate syrup on a textured wafer, inspired by a tasty application of our
approach. Chocolate syrup is a non-Newtonian fluid with a strain-rate
dependent viscosity, bu here we assume that it has a constant shear viscosity
($\sim 100$ cP based on ball drop experiments). We find that this
approximation results in a reasonable agreement between the printed shape and
the target pattern as shown in Figure 6 D (see S.I. video 4). We note that
printing from a height naturally handles rough surfaces, unlike the
traditional direct-ink write where an irregular surface will result in
irregularities in the deposited material.

##  VI Conclusions

A natural next step that follows understanding and predicting fluid
instabilities is to control them. Inspired by 3-d and 4-d printing
technologies that rely on the movement of a nozzle that dispenses complex
fluids onto a substrate from just above, we ask if we can harness the folding
and coiling instabilities that arise as soon as the jet falls from a
sufficient height above a surface. We answer this in the affirmative by
combining a physics-based simulation engine and a variant of reinforcement
learning to control the fluid coiling instability and learn to ”print at a
distance.” By varying the action space, material properties and geometric
scales that govern the dynamcis of viscous coiling, we quantified the
performance of the RL agent for a variety of problems, and showed that it is
possible to learn to write cursively and mimic Pollockian paintings. Deploying
the learned policy in physical experiments, demonstrated that our approach can
create complex physical patterns leveraging a natural fluid instability. We
envision such an approach can be further extended to more challenging
scenarios such as printing on non-planar surfaces and using robotic
manipulators with greater dexterity and improved motion control.

## Acknowledgements

The computations in this paper were run on the FASRC Cannon cluster supported
by the FAS Division of Science Research Computing Group at Harvard University.
We thank Crystal Owens and the Hatsopoulos Microfluids Laboratory at MIT for
help and support with the experiments, and the NSF Harvard MRSEC DMR-2011754,
the Simons Foundation and the Henri Seydoux fund for partial financial
support.

## References

  * Sydney Gladman  _et al._ [2016] A. Sydney Gladman, E. A. Matsumoto, R. G. Nuzzo, L. Mahadevan, and J. A. Lewis, Biomimetic 4d printing, Nature materials 15, 413 (2016).
  * Yuk and Zhao [2018] H. Yuk and X. Zhao, A new 3d printing strategy by harnessing deformation, instability, and fracture of viscoelastic inks, Advanced Materials 30, 1704028 (2018).
  * Barnes and Woodcock [1958] G. Barnes and R. Woodcock, Liquid rope-coil effect, American Journal of Physics 26, 205 (1958).
  * Mahadevan  _et al._ [1998] L. Mahadevan, W. S. Ryu, and A. D. Samuel, Fluid ‘rope trick’investigated, Nature 392, 140 (1998).
  * Mahadevan  _et al._ [2000] L. Mahadevan, W. S. Ryu, and A. D. Samuel, Erratum: Fluid ’rope trick’ investigated (Nature (1998) 140 (392)), Nature 403, 502 (2000).
  * Kim  _et al._ [2010] H.-Y. Kim, M. Lee, K. J. Park, S. Kim, and L. Mahadevan, Nanopottery: coiling of electrospun polymer nanofibers, Nano letters 10, 2138 (2010).
  * Ribe [2017] N. M. Ribe, Liquid rope coiling: a synoptic view, Journal of Fluid Mechanics 812, 9745 (2017).
  * Chakrabarti  _et al._ [2021] A. Chakrabarti, S. Al-Mosleh, and L. Mahadevan, Instabilities and patterns in a submerged jelling jet, Soft Matter 17, 9745 (2021).
  * Herczynski  _et al._ [2011] A. Herczynski, C. Cernuschi, and L. Mahadevan, Painting with drops, jets, and sheets, Physics Today (2011).
  * Sutton and Barto [2017] R. S. Sutton and A. G. Barto,  _Reinforcement Learning: An Introduction (2nd Edition, Draft)_, 2nd ed. (MIT Press, Cambridge, MA, 2017) p. 455.
  * Audoly  _et al._ [2013] B. Audoly, N. Clauvelin, P.-T. Brun, M. Bergou, E. Grinspun, and M. Wardetzky, A discrete geometric approach for simulating the dynamics of thin viscous threads, Journal of Computational Physics 253, 18 (2013).
  * Morris  _et al._ [2008] S. W. Morris, J. H. Dawes, N. M. Ribe, and J. R. Lister, Meandering instability of a viscous thread, Physical Review E 77, 066218 (2008).
  * Novati and Koumoutsakos [2019] G. Novati and P. Koumoutsakos, Remember and forget for experience replay, International Conference on Machine Learning 97, 4851 (2019).
  * Kay [1988] S. M. Kay, Modern spectral estimation (englewood cliffs) (1988).
  * Srivastava and Klassen [2016] A. Srivastava and E. P. Klassen,  _Functional and shape data analysis_ (Springer, 2016).
  * Pollock [1970] J. Pollock, Figure (1970).


(eccv) Package eccv Warning: Package ‘hyperref’ is loaded with option
‘pagebackref’, which is *not* recommended for camera-ready version

11institutetext:  Nanjing University 22institutetext: Institution of
Automation, Chinese Academy of Science 33institutetext: Fudan University

# STAG4D: Spatial-Temporal Anchored Generative 4D Gaussians

Yifei Zeng 11 Yanqin Jiang 22 Siyu Zhu 33 Yuanxun Lu 11 Youtian Lin 11 Hao Zhu
11 Weiming Hu 22 Xun Cao 11 Yao Yao 11

###### Abstract

Recent progress in pre-trained diffusion models and 3D generation have spurred
interest in 4D content creation. However, achieving high-fidelity 4D
generation with spatial-temporal consistency remains a challenge. In this
work, we propose STAG4D, a novel framework that combines pre-trained diffusion
models with dynamic 3D Gaussian splatting for high-fidelity 4D generation.
Drawing inspiration from 3D generation techniques, we utilize a multi-view
diffusion model to initialize multi-view images anchoring on the input video
frames, where the video can be either real-world captured or generated by a
video diffusion model. To ensure the temporal consistency of the multi-view
sequence initialization, we introduce a simple yet effective fusion strategy
to leverage the first frame as a temporal anchor in the self-attention
computation. With the almost consistent multi-view sequences , we then apply
the score distillation sampling to optimize the 4D Gaussian point cloud. The
4D Gaussian spatting is specially crafted for the generation task, where an
adaptive densification strategy is proposed to mitigate the unstable Gaussian
gradient for robust optimization. Notably, the proposed pipeline does not
require any pre-training or fine-tuning of diffusion networks, offering a more
accessible and practical solution for the 4D generation task. Extensive
experiments demonstrate that our method outperforms prior 4D generation works
in rendering quality, spatial-temporal consistency, and generation robustness,
setting a new state-of-the-art for 4D generation from diverse inputs,
including text, image, and video. Project Page:
https://nju-3dv.github.io/projects/STAG4D ††*Equal Contribution

###### Keywords:

4D Generation 3D Gaussian Splatting Diffusion Model

##  1 Introduction

Recent advancements in large-scale pre-trained diffusion models have shown
remarkable progress in producing high-quality and diverse visual content,
including images, videos, and 3D assets [23, 25, 6, 32, 40, 14, 28, 17]. The
progress naturally extends to the realm of dynamic 3D generation [29, 7, 42,
1, 13, 37, 41, 33]. This endeavor has gained prominence in computer vision and
generative AI research, as high-quality 4D content generation is key to a
broad range of applications such as autonomous driving simulation, game and
film industries, digital Avatar creation, and spatial video production.
However, previous 4D generation methods face challenges including blurry
rendering, spatial-temporal inconsistency, and slow generation speed.
Generating high-quality 4D content efficiently and practically remains a
significant challenge.

Figure 1: Visualization of the generated 4D asserts. Our approach can generate
diverse 4D content from various inputs, including text, image, and video.

Given the promising strides in 3D generation from pre-trained diffusion
models, there has been a concerted focus on generalized dynamic 3D generation
from text or uncalibrated monocular video [29, 13, 2, 42, 7, 33]. MAV3D [29]
is the pioneering work of text-to-4D generation which creates 4D scenes from
textual descriptions through Score Distillation Sampling (SDS) from a video
diffusion model. Alternatively, Consistent4D [7] takes a video as input, and
focuses on the task of video-to-4D generation. The pipeline is more flexible
in that it can take advantage of high-quality input videos, dividing the
difficult problem of realistic 4D content creation into a practical pipeline
of text-to-video and video-to-4D generation. However, current video-to-4D
generation methods still struggle to generate high-fidelity 4D asserts with
spatial and temporal rendering consistency. Meanwhile, the commonly used
Neural Radiance Field(NeRF) representation would suffer from an over-saturated
appearance and a long optimization time.

In this paper, we address two important factors in 4D content creation: a
suitable 4D representation and spatial-temporal consistency imposed during
pseudo-projection generation (i.e., multi-view video generation). Our approach
employs a 4D Gaussian splatting approach specially tailored for the generation
task. In the generation process, we first generate the multi-view images at
all timestamps of the 4D scene. To enforce the 4D consistency of the multi-
view video initialization, we design a direct fusion approach for attention
computation in both spatial and temporal domains. This approach implicitly
addresses texture degradation and geometric misalignment, thereby avoiding the
need for adding an explicit multi-view or temporal consistency loss during the
Score Distillation Sampling(SDS) [23] optimization. Then, the reference
photometric loss and the SDS loss conditioned on the initial multi-view video
are applied for 4D Gaussian point cloud optimization. To tackle the
instability in optimizing the 4D Gaussian points, we introduce an adaptive
densification strategy that is informed by the Gaussian gradient distribution.
This approach yields high-quality and robust 4D scene generation from a
monocular video input.

We have conducted comprehensive experiments to demonstrate the effectiveness
of each component of the proposed method. Our approach achieves a 2x faster
generation speed compared with the previous video-to-4D approaches (e.g., [7])
and a significantly better generation quality than prior state-of-the-art
methods. It is also noteworthy that the 4D content generated by our method can
be rendered in real-time, opening up a wide range of possibilities for
practical applications. In summary, our primary contributions are as follows:

  * •

We introduce a holistic 4D generation pipeline that streamlines the generation
process into sequential stages: video generation, multi-view video
initialization, and 4D optimization using multi-view video conditioned SDS.

  * •

We harness the dynamic 3D Gaussian representation for 4D generation,
complemented by an adaptive densification strategy. This combination enables
highly precise and efficient 4D generation from monocular video inputs.

  * •

We present a novel training-free attention fusion module to effectively
integrate temporal anchor frames into the multi-view diffusion process,
significantly enhancing the 4D consistency of the generated multi-view videos.

  * •

Our method outperforms previous approaches in optimization efficiency,
rendering quality, and 4D consistency, establishing a new benchmark for 4D
generation across various input types, such as text, images, and videos.

##  2 Related work

###  2.1 3D Generation

Dreamfusion [23] first proposes the SDS loss [23, 31] for NeRF optimization,
which stands for the most prevalent technique for nowadays text-to-3D
approaches. To mitigate the multi-view Janus problem, Zero123 [14] and
SycnDreammer [15] fine-tune 2D diffusion models to grant the image generator
with the ability of viewpoint control, while later works [28, 27, 16, 18]
explicitly generate fixed multi-view images in one diffusion pass. Building
upon the success of multi-view diffusion methods, we integrate a multi-view
generation module into the 4D generation task, with meticulous handling of 4D
spatial-temporal consistency for enhanced 4D generation quality.

Except for equipping diffusion models with multi-view or depth awareness,
another direction of progress focuses on 3D representations. Following the
idea of differentiable optimization, Magic3D [11] applies a two-stage
generation pipeline and changes the 3D representation to instant-NGP [21] and
DMTet [26], achieving faster runtime and better generation quality. Direct2.5
[18] applies an explicit mesh representation and uses differentiable
rasterization for fast mesh optimization. Recently, with the newly developed
3DGS, DreamGaussian [30] demonstrates the ability to generate 3D objects
within several minutes by substituting NeRF with the 3D Gaussian
representation, showing the potential of using an explicit point-based
representation in 3D generation tasks. In this work, we introduce a novel 4D
Gaussian representation and a specially tailored optimization scheme for the
4D generation task.

###  2.2 4D Reconstruction

Dynamic 3D reconstruction is yet another heated research topic in the field of
computer vision and graphics. By extending the static NeRF [20] framework to
dynamic scenes, dynamic NeRFs [10, 35, 5, 4] have demonstrated remarkable
progress on dynamic 3D reconstruction. However, limited by the implicit neural
representation and the complex nature of dynamic 3D information, dynamic NeRFs
still suffer from slow optimization speed and low reconstruction quality. With
the development of 3DGS [8], researchers have quickly extended the 3D Gaussian
to dynamic scene representation, which achieves faster training and rendering
speeds compared to D-NeRFs based on implicit neural representations. Dynamic
3D Gaussian [19] applies the per-frame 3DGS optimization for 4D scene
reconstruction. Some studies [34, 36] attempt to amalgamate the explicit
point-based 3DGS with an implicit neural field for dynamic information
modeling, while Gaussian-Flow [12] introduces an explicit per-point motion
model to represent a 4D scene without using implicit neural networks. It is
noteworthy that it is non-trivial to extend the Gaussian reconstruction to 4D
reconstruction as each reconstruction pipeline requires heuristic tuning of
hyper-parameters at each stage. In this paper, we target combining the 4DGS
for the challenging task of 4D generation.

###  2.3 4D Generation

Recent developments in 3D content generation and video diffusion technologies
have triggered researchers’ interest in exploring 4D content generation from
various input conditionings. MAV3D [29] presents the early attempt at text-
to-4D generation. By utilizing score distillation sampling derived from video
diffusion models, MAV3D optimizes a dynamic NeRF based on textual prompts. In
a parallel vein, Consistent4D [7] has introduced the task of video-to-4D. This
method capitalizes on the pre-trained knowledge from image diffusion models to
optimize dynamic NeRFs via SDS optimization, showcasing the potential of high-
quality 4D content generation from pre-trained 2D diffusion models.
Concurrently with our work, 4DGen [37] has introduced a framework for
generating dynamic 3D models, which employs spatial-temporal pseudo labels on
keyframes within a multi-view diffusion model. However, the overall quality of
its generation can be further improved, as its rendering fidelity is limited
by the implicit neural representation and inadequate spatial-temporal
information exchange exists in the diffusion generation process. In contrast,
our method applies an efficient 4D Gaussian representation with spatial-
temporal attention from reference- and the first-frame anchors, achieving
substantially better generation results than previous works.

##  3 Method

Figure 2: Overall pipeline. Given a video input, we apply a multi-view
diffusion model to produce coherent multi-view sequences, which serve as
spatial and temporal anchors. Next, we train a deformable 3D Gaussian using
multi-view SDS loss and reference loss. For a text-to-4D generation, our
pipeline can be naturally extended to accept the text input by integrating
with an off-the-shelf text-to-video module.

The framework of the proposed approach, as depicted in Figure 2, outlines the
generation and optimization process of a dynamic scene from a given video
input. Also, our approach can be easily extended to a text-to-4D generation
pipeline by utilizing an off-the-shelf video generation module. Section 3.1
presents the 4D representation used in the proposed method, along with an
adaptive densification strategy that adjusts the densification threshold based
on the relative motion gradient of the scene points. Subsequently, Section 3.2
provides a brief overview of the multi-view diffusion model used in the
pipeline and introduces a novel spatial-temporal attention fusion mechanism
designed to produce the almost consistent multi-view image sequences. Finally,
as detailed in Section 3.3, the generated multi-view sequences are leveraged
with spatial and temporal anchor frames for consistent 4D generation with the
SDS optimization.

###  3.1 4D Representation

####  3.1.1 4D Gaussian Splatting

The concept of 3D Gaussian Splatting was initially introduced in the work of
[8], where an explicit point-based representation is used to model the 3D
scene. Specifically, the 3D Gaussian point cloud, denoted as $\mathcal{S}$ for
a static scene, is characterized by the tuple:

$\mathcal{S}=[\mathcal{X},s,r,\sigma,\zeta],$ (1)

where $\mathcal{X}=(x,y,z)$ denotes the positions of the 3D Gaussian points,
and $s$, $r$, $\sigma$ and $\zeta$ represent the scale, rotation, opacity, and
spherical harmonics (SH) coefficients of the radiance, respectively. In 3D
reconstruction, the 3D Gaussian point cloud will be optimized through a point-
based differentiable volume rendering.

This representation can be naturally extended to 4D scenes by presenting the
continuous scene dynamics as a 3D motion field. Inspired by [34], we present a
4D scene as a 3D Gaussian point cloud with a hex-plane-based deformation
field, denoted as $\mathcal{F}(\mathcal{S},t)$. The 3D Gaussian point cloud at
time $t$ can be expressed as:

$\mathcal{F}(\mathcal{S},t)=[\mathcal{X}_{t},s_{t},r_{t},\sigma,\zeta],$ (2)

where $\mathcal{X}_{t}=(x_{t},y_{t},z_{t})$, $s_{t}$ and $r_{t}$ represent the
updated information of Gaussian position, scale, and rotation at time $t$.

####  3.1.2 Adaptive Densification

The standard 3D Gaussian Splatting technique typically employs a point cloud
densification control strategy to dynamically adjust the number of Gaussians
and their density within a unit volume. This adaptive approach allows for the
transition from an initial sparse Gaussian set to a denser configuration to
better represent the 3D scene. The 4D Gaussian Splatting method introduced in
the work [34] utilizes a densification strategy similar to that presented in
vanilla 3DGS [8], which involves the use of a fixed densification threshold
based on the view-space position gradient. However, while the fixed gradient
threshold strategy demonstrates efficacy in the context of reconstruction,
particularly when multiple views afford robust and redundant coverage of the
target scene, it performs sub-optimally in the generative setting. This
limitation arises from the constraints imposed by the input single image or
monocular videos, leading to significant uncertainty in the spatial and scale
dimensions for each training object. This will lead to different optimal
thresholds for different cases, as illustrated in the ablation study.

To address the issue, we propose an adaptive threshold approach, wherein only
candidates with relatively large gradients are selected for densification. Our
approach is based on the statistical analysis of the accumulated gradient of
each point, which follows a log distribution of similar shapes throughout the
training process. We apply an adaptive threshold that filters out Gaussians
with a relatively small gradient and only densifies those with a large
gradient. The adaptive threshold is set to select a fixed percentage (top
$\lambda\%$) of points with the highest gradient in each densification
operation. This ensures that the threshold adapts to the distribution of the
gradient and maintains a stable relative position. Experimental results
demonstrate that the proposed simple strategy can significantly enhance the
quality and robustness of the 4D generation.

###  3.2 Temporal and Multi-view Consistent Diffusion

####  3.2.1 Multi-view Consistent Diffusion

Our pipeline adopts Score Distillation Sampling(SDS) [23] from image diffusion
models for the optimization of dynamic Gaussian. Specifically. Image-to-image
diffusion model Zero123 [14] is utilized and the SDS loss gradient could be
formulated as follows:

$\displaystyle\nabla_{\theta}\mathcal{L}_{SDS}(\phi,\mathbf{x})$
$\displaystyle=\mathbb{E}_{t,\epsilon}\left[\omega(t)(\hat{\epsilon}_{\theta}(\mathbf{z}_{t};\mathbf{I}_{in},\mathbf{R},\mathbf{T},t)-\epsilon)\frac{\partial\mathbf{x}}{\partial\theta}\right],$
(3) $\displaystyle\hat{z}$
$\displaystyle=z_{t}-\sigma_{t}\hat{\epsilon}_{t}(z;\mathbf{I}_{in},\mathbf{R},\mathbf{T}).$

where $\theta$ represents the parameters of the 3D representation,
$\mathbf{x}$ the rendered image at the current view, $t$ the timestamp in the
diffusion process, $\epsilon$ the ground truth noise, $\hat{\epsilon}$ the
predicted noise from the noisy image $\mathbf{z_{t}}$ conditioned on an
initial input $\mathbf{I}_{in}$, and the relative camera pose between input
view and target view $(\mathbf{R},\mathbf{T})$. Zero123 can generate the
target view image at any relative camera location but only one target view at
a time, thus good at optimizing the 3D object from all views whilst bad at
generating spatially consistent images from multiple target views. In
contrast, Zero123++ [27] leverages reference attention [38] to model the
relationship of the images from multiple target views as well as the input
view, resulting in multi-view consistency output yet at the cost of fixed
target camera locations. We combine the advantages of both models by taking
the multi-view consistent output of Zero123++ as the input images of Zero123
when calculating SDS loss.

####  3.2.2 Temporally Consistent Diffusion

Following the view generation philosophy of multi-view diffusion based 3D
generation, a practical solution to 4D content creation is to generate the
multi-view videos of the 4D scene, and then apply the multi-view video
conditioned SDS loss to optimize the scene. However, it is rather difficult to
generate temporally consistent multi-view videos by the separate per-frame
generation. Inspired by zero-shot video generation method [9], we propose a
training-free temporal attention module to enable Zero123++ [27] with
temporal-awareness.

In our design, we obtain the attention information during denoising the
multiview images of the first frame. Then we apply the recorded attention to
the later denoising process. Specifically, during the denoising process of the
latent code, the self-attention layer computes three essential
components—queries ($Q$), keys ($K$), and values ($V$), which is formulated
as:

$Self\text{-}Attn(Q,K,V)=Softmax(\frac{QK^{T}}{\sqrt{c}})V.$ (4)

We first compute the key $K_{0}$ and value $V_{0}$ for the initial frame,
$T_{0}$. Subsequently, for each subsequent frame $T_{t}$, where
$t\in\\{1,...,N\\}$, we perform a mixing operation on the key $K_{t}$ and
value $V_{t}$ with the initial key $K_{0}$ and value $V_{0}$ derived from the
first frame $T_{0}$. This process can be formally represented as follows:

$\begin{cases}K_{t}=\gamma\mathbf{K}_{0}+(1-\gamma)\mathbf{K}_{t}\\\
V_{t}=\gamma\mathbf{V}_{0}+(1-\gamma)\mathbf{V}_{t},\end{cases}$ (5)

where the parameter $\gamma$ serves as a weighting factor that governs the
influence of the initial frame, as illustrated in the ablation part.
Additionally, we employ reference attention following the approach outlined in
[27, 39], to extract local conditioning from the input image. In this
reference attention mechanism, both the key and the value are obtained through
the concatenation of features from the reference image and the noisy input
image. Our design of temporal attention module is simple yet effective with
superior temporal consistency, image quality and 3D consistency to vanilla
cross-frame attention proposed in [9], as demonstrated in experiment section.

Figure 3: Illustration of the proposed spatial and temporal attention fusion
during multi-view sequence inference.

###  3.3 Training Objectives

Following the generation of multi-view sequences from a monocular reference
video, we can acquire 6 anchor views $\\{I^{i}_{t}\\}_{i\in\\{1...6\\}}$ and a
reference view $I^{ref}_{t}$ at each timestep $t$. In our optimization
process, we employ multi-view score distillation sampling (SDS) utilizing the
generated images $\\{I^{i}_{t}\\}_{i=1...6}$ in conjunction with the reference
images $I^{ref}_{t}$. The multi-view score distillation loss function
$\mathcal{L}_{MVSDS}$ can be defined as:

$\displaystyle\mathcal{L}_{MVSDS}$
$\displaystyle=\lambda_{1}\mathcal{L}_{SDS}^{i}+\lambda_{2}\mathcal{L}_{SDS}^{ref}$
(6)
$\displaystyle=\lambda_{1}\mathcal{L}_{SDS}(\phi,I^{i}_{t})+\lambda_{2}\mathcal{L}_{SDS}(\phi,I^{ref}_{t}),$

where $\lambda_{1}$ and $\lambda_{2}$ are two weighting factors, the index $i$
is determined based on the proximity of the rendering viewpoint to the
viewpoint of the generated images. This selection process, which we refer to
as multi-view score distillation sampling, involves choosing the nearest
reference image to the rendered camera view for calculating the SDS loss.

Following the approach [30], we utilize the reference image to compute both
the reconstruction loss $\mathcal{L}_{rec}$ and the foreground mask loss
$\mathcal{L}_{mask}$. Thus, the final optimization objective is:

$\begin{split}\mathcal{L}=&\mathcal{L}_{MVSDS}+\lambda_{3}\mathcal{L}_{rec}+\lambda_{4}\mathcal{L}_{mask},\end{split}$
(7)

where $\lambda_{3}$ and $\lambda_{4}$ are the weighting parameters . During
training, we first use $\mathcal{L}$ to supervise a fixed frame to get static
canonical 3D Gaussian. Then we use all anchors and reference images to train
the dynamic 4D Gaussian point cloud.

###  3.4 Text-to-4D Extension

Our method is designed to be readily adaptable to text and image inputs,
offering novel capabilities such as directly generating video sequences from
textual descriptions or static images. We initiate this process using a 2D
diffusion model SDXL [22], which generates the image from textual input. This
2D output is then transformed into a video sequence via a video diffusion
model, such as SVD [3], adding temporal coherence and motion to the static
image. Finally, the aforementioned pipeline further lifts the video sequence
to a 4D scene, introducing an extra spatial dimension that creates immersive
experiences beyond conventional 2D or 3D media. The generation pipeline with
the given text, image, and video inputs is visualized in Fig. 2.

##  4 Experiment

###  4.1 Experiment Setup

Dataset For the video-to-4D task, we utilize the dataset provided by
Consistent4D [7] for quantitative evaluation, which comprises multi-view
videos depicting 7 dynamic objects. Additionally, for qualitative evaluation,
we curate a set of challenging videos from online sources to assess the
robustness and generalization capabilities of each method. For text/image-
to-4D tasks, we follow the data settings in 4Dfy [2] and DreamGaussian4D [24]
to create the corresponding results. Notably, we use the same input image and
input video with DreamGaussian4D for a fair comparison.

Figure 4: Qualitative comparison on video-to-4D generation. Our method
achieves better faithfulness to the input video with higher quality on the
overall reconstruction.

Evaluation metrics We employ CLIP, LPIPS, and FVD for Video-to-4D evaluation
as in the previous approach [7]. Specifically, CLIP and LPIPS serve as image-
level metrics, assessing the semantic similarity between rendered images and
ground truths; FVD is a video-level metric commonly used in video generation
tasks, considering not only single-frame quality but also temporal coherence;
we also consider the FID-VID metric to measure the temporal consistency. As
4D-Gen[37] does not support a 30-frame setting we report its FVD under a
16-frame reconstruction, namely FVD-16 as an additional metrics. For text-
to-4D generation, we follow previous works and provide a user study for
quantitative comparisons between different methods. We also provide a user
study for video-to-4D and both studies are detailed in the supplementary
material.

Baselines For video-to-4D generation, we compare our method with two
baselines: Consistent4D [7] and the concurrent work 4DGen [37]. We utilize the
official code released by the authors to generate comparison results. For
text&image-to-4D generation, we make comparisons with two state-of-art models
that are open-sourced: 4Dfy [2] and DreamGaussian4D [24]. Their results are
generated using codes from their official GitHub repository.

###  4.2 Implementation Details

In the training of the video-to-4D generative model, we adopt a two-stage
approach. Initially, the model is initialized in canonical space and trained
for 1000 steps. Subsequently, the deformation fields are learned over 7000
additional steps to accurately capture the dynamic scene. For the deformation
process, we employ multi-layer perceptrons (MLPs) with 64 hidden layers and 32
hidden features per layer. The initial learning rate for the deformation MLPs
is set to $1.6\times 10^{-4}$ and is decayed to $1.6\times 10^{-6}$ by the end
of the training. In the context of adaptive densification, we opt to densify
the top $2.5\%$ of points with the most accumulated gradient. In the
generative phase, we ultilize the spatial-temporal consistent videos for the
multiview score distillation sampling optimization. Empirically, we set
$\lambda$ to 0.5 for our temporally consistent diffusion. Regarding the loss
functions, we maintain the SDS loss weight at 1 and adjust the weights of
other losses accordingly. Specifically, the reconstruction loss is assigned to
$4\times 10^{4}$, the mask loss is weighted at $1\times 10^{4}$. The training
process requires approximately 1 hour on an RTX 3090 GPU, and the rendering
process can be performed at 150 FPS in real time. For Text&Image-to-4D, we use
SDXL [22] for image generation, and apply SVD [3] to the generated image to
create a corresponding video. Then we use the same setting as Video-to-4D to
transfer the video into 4D content.

Figure 5: Qualitative comparison on text-to-4D generation. Our method achieves
the best results on the text alignment and visual quality.

Figure 6: Ablation on spatial and temporal attention. We evaluate the effect
of spatial and temporal attention on the reconstruction results.

###  4.3 Comparison with State-of-the-art Approaches

In this section, we conduct a comprehensive comparison with the aforementioned
baselines using synthetic and in-the-wild data. The superior performance
demonstrated in both quantitative and qualitative evaluations serves to
substantiate the effectiveness and robustness of our proposed method.

Quantitative Results on Video-to-4D The experimental results presented in
Table 1 offer a comparative analysis of our method against state-of-the-art
techniques, namely Consistent4D and 4DGen, across several metrics. Our method
outperforms Consistent4D and 4DGen in CLIP, showing better semantic
consistency with the target content. It also achieves the lowest LPIPS score,
meaning it generates more realistic images than the others. For the video
quality and smoothness, our method beats 4DGen in FID-VID and FVD-16, and
Consistent4D in FVD, implying that our videos are closer to real videos and
have less temporal artifacts. Overall, the numbers confirm that our method is
superior in generating semantically aligned and perceptually convincing videos
with high fidelity and coherence.

Qualitative Comparison on Video-to-4D In our investigation of the video-to-4D
task, we have employed various methodologies to conduct qualitative
comparisons. Figure 4 shows the rendered outcomes at distinct temporal
intervals and perspectives. A critical observation of our findings reveals
that Gaussian splatting is inadequate in producing credible and temporally
consistent images during per-frame reconstruction. Furthermore, we have
observed that Consistent4D often produces over-saturated and unrealistic
patterns, which may stem from the over-restrictive geometry representation in
their cascade Dynerf. Additionally, our analysis indicates that 4DGen
struggles to generate reasonable motion for the Gaussians, leading to its
points lacking deformation over time. Moreover, 4DGen fails to generate
detailed textures for the surface, which appear to be over-smoothed in some
cases. Conversely, our methodology demonstrates substantial enhancements in
comparison to existing strategies, particularly in aspects of reconstruction
fidelity and stability. Notably, our technique facilitates seamless color
transitions, as exemplified in the Fish case, and renders a convincing low-
poly effect, as demonstrated in the Trump case, underscoring our method’s
adaptability to diverse dynamic environments.

Qualitative Comparison on Text-to-4D We also evaluate our method on the
Text&Image-to-4D task, which aims to generate 4D content from text and image
inputs. Figure 5 shows the 4D generation results of our method and two
baselines, 4Dfy [2] and DreamGaussian4D [24], under different views. As can be
seen, 4Dfy [2] produces coarse results and fails to capture the complex
semantics of the text input, e.g. in the Cat case. DreamGaussian4D [24]
generates meshes with fine-grained details in the front view, but suffers from
severe artifacts and distortions in the side view or back view. In contrast,
our method synthesizes Gaussians with smooth and consistent geometry and
realistic texture across different views, thanks to our attention mechanism
that leverages spatial and temporal information. Our method demonstrates its
superiority and versatility on the Text&Image-to-4D task, which can be seen as
a direct application of our Video-to-4D framework.

Table 1: Comparison with state-of-the-art methods.

CLIP $\uparrow$ LPIPS $\downarrow$ FID-VID $\downarrow$ FVD $\downarrow$
FVD-16 $\downarrow$ Consistent4D 0.877 0.134 / 1133.93 / 4DGen 0.894 0.130
71.99 / 1005.72 Ours 0.909 0.126 52.58 992.21 952.41

###  4.4 Ablation Study

The experimental analysis presented in Table 3 provides a comprehensive
evaluation of the impact of various components on the performance of a
diffusion model with a 4D representation. The components under consideration
include baseline diffusion, spatial anchor, spatial-temporal anchor, and
adaptive densification. In the first configuration of Table 3, we regard the
model utilizing the diffusion process [14] with adaptive densification as the
baseline.

Table 2: Ablation study on effectiveness of each novel component.

Diffusion Model 4D Gaussian Evaluation Metrics Baseline Spat. Temp. Adaptive
Dens. CLIP $\uparrow$ LPIPS $\downarrow$ FID-VID $\downarrow$ FVD $\downarrow$
✓ ✓ 0.895 0.135 77.88 1369.65 ✓ ✓ ✓ 0.899 0.130 67.04 1198.23 ✓ ✓ ✓ 0.890
0.136 90.30 1453.53 ✓ ✓ ✓ ✓ 0.909 0.126 52.58 992.21

Table 3: Ablation study on different attention mechanisms.

Evaluation Metrics CLIP $\uparrow$ LPIPS $\downarrow$ FID-VID $\downarrow$ FVD
$\downarrow$ Spat. only 0.899 0.130 67.04 1198.23 Cross-Frame 0.901 0.129
63.32 1053.25 Spat.-Temp. 0.909 0.126 52.58 992.21

Spatial and Temporal Anchors Table 3 shows that adding the spatial anchor to
the baseline model leads to a marginal improvement in all metrics, especially
FID-VID and FVD. The final model, which uses both spatial-temporal anchor and
adaptive densification, improves all metrics significantly. It has the best
CLIP, LPIPS, FID-VID, and FVD scores, meaning it generates the most
semantically consistent, realistic, and smooth videos.

Figure 6 shows the importance of our spatial-temporal anchor for improving the
4D content quality. The baseline 4D Gaussian method, labeled as “w./o.
spatial,” fails to deform the point clouds properly, leading to unreasonable
shapes without spatial-temporal constraints. A model that uses only the
spatial anchor, “w./o. temporal,” struggles to deform solid shapes, as seen in
the rabbit’s leg and the frog’s back. Our method, which combines both spatial
and temporal anchors, overcomes these issues and produces visually consistent
and accurate 4D content.

Adaptive Densification We present the statistical analysis of the adaptive
densification module in the third and final configurations of Table 3. The
results show that the adaptive densification module improves the overall
quality of the generative model, especially in terms of the FID-VID and FVD
metrics. We also compare the proposed adaptive densification with a fixed
threshold approach in Figure 7(b). We observe that a fixed threshold can lead
to either under-densification or over-densification, depending on the case.
For example, in the Bird case, the fixed threshold is too low for
densification, which results in excessive Gaussian points. On the other hand,
in the Duck case, the fixed threshold is too high for densification, which
causes the foot to be blurry due to insufficient Gaussian points. Our method
overcomes this limitation by adapting the gradient threshold to different
cases, thus producing robust and stable 4D Gaussian results.

(a) Ablation on Multiview Diffusion Models We evaluate the effect of different
attention mechanism on Zero123++. Our attention mechanism can achieve both
temporal coherence and geometric fidelity given a monocular input video.

(b) Ablation on Adaptive Densification We evaluate the effect of using
different densify mechanism during reconstruction. Our adaptive threshold
provides reasonable densification guidance for each case.

Different Attention Mechanisms In our study, we evaluated three distinct
attention mechanisms within the context of images generated by our diffusion
model: spatial attention, cross-frame attention, and spatial-temporal
attention. The visual outcomes of these varied approaches are illustrated in
Figure 7(a). We employed a parameter $\lambda$ to modulate the extent of
spatial and temporal attention exerted, as delineated in Equation 5. Our
findings indicate that exclusive reliance on spatial attention does not
guarantee consistency across sequential frames. While cross-frame attention
enhances frame-to-frame coherence, it fails to achieve a satisfactory
alignment with reference images. Conversely, the integration of spatial-
temporal attention ensures consistency across both dimensions. We
quantitatively assessed the quality of images generated under different
attention regimes using four established metrics: CLIP, LPIPS, FID-VID, and
FVD. The comparative results are tabulated in Table 3. It was observed that
images generated with spatial-temporal attention outperformed others across
all evaluation metrics. This underscores the efficacy of our proposed
attention mechanism in augmenting the diffusion model for enhanced 4D content
generation.

###  4.5 Limitations

While our method has shown promising results, it is important to acknowledge
several limitations. Specifically, our approach is constrained in handling
complicated and fast motion due to the inherent limitations of current 4D
representation in terms of Gaussian Splatting, which can lead to suboptimal
outcomes. Additionally, inherent video limitations such as image blurriness
can restrict the effectiveness of diffusion, consequently impacting the
subsequent optimization of 4D Gaussian Splatting. Furthermore, domain transfer
to videos lacking well-segmented foreground or containing multiple foreground
objects presents a non-trivial challenge that warrants consideration. These
limitations are important to address in future research and development
efforts.

##  5 Conclusion

The paper presents a novel approach for dynamic 3D content generation from
monocular videos, addressing the challenges of 4D representation and spatial-
temporal consistency. By leveraging specially tailored 4D Gaussian splatting
and a novel information fusion module, the proposed method achieves high-
quality and robust 4D scene generation. Comprehensive experiments demonstrate
the method’s effectiveness, showcasing an obvious faster generation speed and
significant improvements in rendering quality and temporal consistency
compared to prior state-of-the-art methods. Overall, the proposed method sets
a new benchmark for training speed, rendering quality, and 4D consistency in
dynamic 3D content generation from monocular videos, opening up possibilities
for real-world applications.

## References

  * [1] Bahmani, S., Skorokhodov, I., Rong, V., Wetzstein, G., Guibas, L., Wonka, P., Tulyakov, S., Park, J.J., Tagliasacchi, A., Lindell, D.B.: 4d-fy: Text-to-4d generation using hybrid score distillation sampling. arXiv preprint arXiv:2311.17984 (2023)
  * [2] Bahmani, S., Skorokhodov, I., Rong, V., Wetzstein, G., Guibas, L., Wonka, P., Tulyakov, S., Park, J.J., Tagliasacchi, A., Lindell, D.B.: 4d-fy: Text-to-4d generation using hybrid score distillation sampling (2023)
  * [3] Blattmann, A., Dockhorn, T., Kulal, S., Mendelevitch, D., Kilian, M., Lorenz, D., Levi, Y., English, Z., Voleti, V., Letts, A., Jampani, V., Rombach, R.: Stable video diffusion: Scaling latent video diffusion models to large datasets (2023)
  * [4] Du, Y., Zhang, Y., Yu, H.X., Tenenbaum, J.B., Wu, J.: Neural radiance flow for 4d view synthesis and video processing. In: 2021 IEEE/CVF International Conference on Computer Vision (ICCV). pp. 14304–14314. IEEE Computer Society (2021)
  * [5] Gao, C., Saraf, A., Kopf, J., Huang, J.B.: Dynamic view synthesis from dynamic monocular video. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 5712–5721 (2021)
  * [6] Ho, J., Chan, W., Saharia, C., Whang, J., Gao, R., Gritsenko, A., Kingma, D.P., Poole, B., Norouzi, M., Fleet, D.J., et al.: Imagen video: High definition video generation with diffusion models. arXiv preprint arXiv:2210.02303 (2022)
  * [7] Jiang, Y., Zhang, L., Gao, J., Hu, W., Yao, Y.: Consistent4d: Consistent 360 $\\{$$\backslash$deg$\\}$ dynamic object generation from monocular video. arXiv preprint arXiv:2311.02848 (2023)
  * [8] Kerbl, B., Kopanas, G., Leimkühler, T., Drettakis, G.: 3d gaussian splatting for real-time radiance field rendering. ACM Transactions on Graphics (ToG) 42(4), 1–14 (2023)
  * [9] Khachatryan, L., Movsisyan, A., Tadevosyan, V., Henschel, R., Wang, Z., Navasardyan, S., Shi, H.: Text2video-zero: Text-to-image diffusion models are zero-shot video generators. arXiv preprint arXiv:2303.13439 (2023)
  * [10] Li, Z., Niklaus, S., Snavely, N., Wang, O.: Neural scene flow fields for space-time view synthesis of dynamic scenes. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (2021)
  * [11] Lin, C.H., Gao, J., Tang, L., Takikawa, T., Zeng, X., Huang, X., Kreis, K., Fidler, S., Liu, M.Y., Lin, T.Y.: Magic3d: High-resolution text-to-3d content creation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 300–309 (2023)
  * [12] Lin, Y., Dai, Z., Zhu, S., Yao, Y.: Gaussian-flow: 4d reconstruction with dynamic 3d gaussian particle. arXiv:2312.03431 (2023)
  * [13] Ling, H., Kim, S.W., Torralba, A., Fidler, S., Kreis, K.: Align your gaussians: Text-to-4d with dynamic 3d gaussians and composed diffusion models. arXiv preprint arXiv:2312.13763 (2023)
  * [14] Liu, R., Wu, R., Van Hoorick, B., Tokmakov, P., Zakharov, S., Vondrick, C.: Zero-1-to-3: Zero-shot one image to 3d object. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 9298–9309 (2023)
  * [15] Liu, Y., Lin, C., Zeng, Z., Long, X., Liu, L., Komura, T., Wang, W.: Syncdreamer: Generating multiview-consistent images from a single-view image. arXiv preprint arXiv:2309.03453 (2023)
  * [16] Long, X., Guo, Y.C., Lin, C., Liu, Y., Dou, Z., Liu, L., Ma, Y., Zhang, S.H., Habermann, M., Theobalt, C., et al.: Wonder3d: Single image to 3d using cross-domain diffusion. arXiv preprint arXiv:2310.15008 (2023)
  * [17] Lu, Y., Zhang, J., Li, S., Fang, T., McKinnon, D., Tsin, Y., Quan, L., Cao, X., Yao, Y.: Direct2. 5: Diverse text-to-3d generation via multi-view 2.5 d diffusion. arXiv preprint arXiv:2311.15980 (2023)
  * [18] Lu, Y., Zhang, J., Li, S., Fang, T., McKinnon, D., Tsin, Y., Quan, L., Cao, X., Yao, Y.: Direct2.5: Diverse text-to-3d generation via multi-view 2.5d diffusion (2023)
  * [19] Luiten, J., Kopanas, G., Leibe, B., Ramanan, D.: Dynamic 3d gaussians: Tracking by persistent dynamic view synthesis. arXiv preprint arXiv:2308.09713 (2023)
  * [20] Mildenhall, B., Srinivasan, P.P., Tancik, M., Barron, J.T., Ramamoorthi, R., Ng, R.: Nerf: Representing scenes as neural radiance fields for view synthesis. In: ECCV (2020)
  * [21] Müller, T., Evans, A., Schied, C., Keller, A.: Instant neural graphics primitives with a multiresolution hash encoding. ACM Trans. Graph. 41(4), 102:1–102:15 (Jul 2022). https://doi.org/10.1145/3528223.3530127, https://doi.org/10.1145/3528223.3530127
  * [22] Podell, D., English, Z., Lacey, K., Blattmann, A., Dockhorn, T., Müller, J., Penna, J., Rombach, R.: Sdxl: Improving latent diffusion models for high-resolution image synthesis (2023)
  * [23] Poole, B., Jain, A., Barron, J.T., Mildenhall, B.: Dreamfusion: Text-to-3d using 2d diffusion. arXiv preprint arXiv:2209.14988 (2022)
  * [24] Ren, J., Pan, L., Tang, J., Zhang, C., Cao, A., Zeng, G., Liu, Z.: Dreamgaussian4d: Generative 4d gaussian splatting. arXiv preprint arXiv:2312.17142 (2023)
  * [25] Rombach, R., Blattmann, A., Lorenz, D., Esser, P., Ommer, B.: High-resolution image synthesis with latent diffusion models. In: Proceedings of the IEEE/CVF International Conference on Computer Vision (2022)
  * [26] Shen, T., Gao, J., Yin, K., Liu, M.Y., Fidler, S.: Deep marching tetrahedra: a hybrid representation for high-resolution 3d shape synthesis. In: Advances in Neural Information Processing Systems (NeurIPS) (2021)
  * [27] Shi, R., Chen, H., Zhang, Z., Liu, M., Xu, C., Wei, X., Chen, L., Zeng, C., Su, H.: Zero123++: a single image to consistent multi-view diffusion base model. arXiv preprint arXiv:2310.15110 (2023)
  * [28] Shi, Y., Wang, P., Ye, J., Mai, L., Li, K., Yang, X.: Mvdream: Multi-view diffusion for 3d generation. arXiv:2308.16512 (2023)
  * [29] Singer, U., Sheynin, S., Polyak, A., Ashual, O., Makarov, I., Kokkinos, F., Goyal, N., Vedaldi, A., Parikh, D., Johnson, J., et al.: Text-to-4d dynamic scene generation. arXiv preprint arXiv:2301.11280 (2023)
  * [30] Tang, J., Ren, J., Zhou, H., Liu, Z., Zeng, G.: Dreamgaussian: Generative gaussian splatting for efficient 3d content creation. arXiv preprint arXiv:2309.16653 (2023)
  * [31] Wang, H., Du, X., Li, J., Yeh, R.A., Shakhnarovich, G.: Score jacobian chaining: Lifting pretrained 2d diffusion models for 3d generation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 12619–12629 (2023)
  * [32] Wang, J., Yuan, H., Chen, D., Zhang, Y., Wang, X., Zhang, S.: Modelscope text-to-video technical report. arXiv preprint arXiv:2308.06571 (2023)
  * [33] Wang, X., Wang, Y., Ye, J., Wang, Z., Sun, F., Liu, P., Wang, L., Sun, K., Wang, X., He, B.: Animatabledreamer: Text-guided non-rigid 3d model generation and reconstruction with canonical score distillation. arXiv preprint arXiv:2312.03795 (2023)
  * [34] Wu, G., Yi, T., Fang, J., Xie, L., Zhang, X., Wei, W., Liu, W., Tian, Q., Wang, X.: 4d gaussian splatting for real-time dynamic scene rendering. arXiv preprint arXiv:2310.08528 (2023)
  * [35] Xian, W., Huang, J.B., Kopf, J., Kim, C.: Space-time neural irradiance fields for free-viewpoint video. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 9421–9431 (2021)
  * [36] Yang, Z., Gao, X., Zhou, W., Jiao, S., Zhang, Y., Jin, X.: Deformable 3d gaussians for high-fidelity monocular dynamic scene reconstruction. arXiv preprint arXiv:2309.13101 (2023)
  * [37] Yin, Y., Xu, D., Wang, Z., Zhao, Y., Wei, Y.: 4dgen: Grounded 4d content generation with spatial-temporal consistency. arXiv preprint arXiv:2312.17225 (2023)
  * [38] Zhang, L.: Reference-only control., https://github.com/Mikubill/sd-webui-controlnet/discussions/1236,2023
  * [39] Zhang, L.: Reference-only control (2023), https://github.com/Mikubill/sd-webui-controlnet/discussions/1236
  * [40] Zhang, S., Wang, J., Zhang, Y., Zhao, K., Yuan, H., Qing, Z., Wang, X., Zhao, D., Zhou, J.: I2vgen-xl: High-quality image-to-video synthesis via cascaded diffusion models (2023)
  * [41] Zhao, Y., Yan, Z., Xie, E., Hong, L., Li, Z., Lee, G.H.: Animate124: Animating one image to 4d dynamic scene. arXiv preprint arXiv:2311.14603 (2023)
  * [42] Zheng, Y., Li, X., Nagano, K., Liu, S., Hilliges, O., De Mello, S.: A unified approach for text-and image-guided 4d scene generation. arXiv preprint arXiv:2311.16854 (2023)

Supplementary Materials

##  6 Graident Distribution Analysis for Adaptive Densification

Figure 8: Gradient Distribution Analysis for Adaptive Densification

In Fig 8, we present a visual examination of the gradient distribution for the
Duck and Bird scenarios across various training epochs. This analysis
elucidates the rationale behind the ablation study of our adaptive
densification approach. The graphical data indicates that the gradient
distributions for different scenarios converge to disparate values, signifying
a substantial numerical disparity when plotted on a logarithmic scale. For the
Duck scenario, a suitable gradient threshold is approximately 40 ($e^{3.7}$),
whereas for the Bird scenario, it is around 150 ($e^{5}$). Moreover, the
gradient distribution maintains a consistent shape throughout the training
phase for each case. Consequently, we introduce the adaptive densification
mechanism that dynamically modulates the threshold in response to the specific
scenario, which is to densify the Gaussian points that exhibit relatively
higher gradient accumulation. This strategy is anticipated to enhance the
robustness and quality of the generative performance.

##  7 User Study

###  7.1 Video-to-4D

In the conducted user study focusing on Video-to-4D content generation, we
juxtaposed our methodology against Consistent4D[7] and 4DGen[37] across 14
test instances. Participants were solicited to appraise the methods according
to visual quality (Vis.), temporal consistency (Cons.), and alignment with the
input videos (Align.). As illustrated in Table 4, 4DGen did not receive any
endorsements in the evaluated categories. In contrast, Consistent4D was
acknowledged with 28.6% for Vis. and Cons., and 35.7% for Align.
Predominantly, our method was favored, securing 71.4% of the votes for Vis.
and Cons., and 64.3% for Align. These outcomes attest to the superior efficacy
of our method in synthesizing high-fidelity and temporally coherent Video-
to-4D content.

| Vis. | Cons. | Align.
---|---|---|---
Consistent4D | 28.6% | 28.6% | 35.7%
4DGen | 0% | 0% | 0%
Ours | 71.4% |  71.4% |  64.3%
Table 4: User study on the best-performing Video-to-4D generation methods.

###  7.2 Text/Video-to-4D

A user study was conducted to evaluate the efficacy of Text&Video-to-4D
content generation. For this purpose, 4Dfy[2] and DreamGaussian4D[24] were
selected as comparative benchmarks. The study encompassed 14 test scenarios,
and 30 evaluators were recruited to assess the methods based on visual quality
(Vis.), temporal consistency (Cons.), and congruence with the input text
(Align.). As delineated in Table 5, our approach garnered the highest ratings
across all metrics. These findings underscore our method’s preeminence and
adaptability in the Text&Image-to-4D content generation domain.

| Vis. | Cons. | Align.
---|---|---|---
4Dfy | 2.04 | 2.15 | 2.26
DG4D | 2.91 | 2.92 | 3.26
Ours | 4.02 | 4.41 | 4.15
Table 5: User study for Text&Image-to-4D generation methods.

##  8 Attention Mechanism for Normal Map

Figure 9: Normal Map Illutstration

Building upon our attention mechanism, we have integrated our design with the
newly introduced normal map model by Zero123++[27]. This adaptation enables
the generation of coherent multiview images with corresponding normal maps, as
depicted in Figure 9. The successful application of our method to normal maps
exemplifies the flexibility and applicability of our attention framework. This
enhancement not only broadens the scope of our model but also demonstrates its
potential for diverse 4D content generation scenarios.

##  9 More Results for Multiview Generation

Figure 10: More Results for Multiview Generation (Text input)  Figure 11: More
Results for Multiview Generation (Video input)

We show other results of our spatial and temporal consistent multiview images
in Figure 10 & Figure 11. These results demonstrate the robustness and
effectiveness of our method.

##  10 More Results for 4D Generation

We show more results of our 4D Generation in Figure 12. Our method could
handle various objects and motions, which applies widely to different cases.

##  11 Discussion on the Potential Negative Impact

The generation process of our method involves various pre-trained diffusion
models, which still have a controversy on the copyright of the generated
result. This will remain an ethical issue until the relevant laws are matured.

Figure 12: More Results for 4D Generation