# Phase Ordering in Binary Mixtures of Active Nematic Fluids

Saraswat Bhattacharyya<EMAIL_ADDRESS>Julia M.
Yeomans  Rudolf Peierls Centre for Theoretical Physics, Parks Road, University
of Oxford, OX1 3PU. United Kingdom.

###### Abstract

We use a continuum, two-fluid approach to study a mixture of two active
nematic fluids. Even in the absence of thermodynamically-driven ordering, for
mixtures of different activities we observe turbulent microphase separation,
where domains form and disintegrate chaotically in an active turbulent
background. This is a weak effect if there is no elastic nematic alignment
between the two fluid components, but is greatly enhanced in the presence of
an elastic alignment or substrate friction. We interpret the results in terms
of relative flows between the two species which result from active anchoring
at concentration gradients. Our results may have relevance in interpreting
epithelial cell sorting and the dynamics of multi-species bacterial colonies.

††preprint: APS/123-QED

##  I Introduction

Phase separation is a ubiquitous phenomenon found across a wide variety of
biological systems. Inside cells, membrane-less organelles such as stress
granules and nucleoli phase separate from their surroundings to allow
different chemical environments for biochemical reactions [1, 2]. Different
cell types sort themselves into distinct regions in confluent layers [3, 4]
during growth and morphogenesis [5, 6]. Bacterial colonies also undergo
segregation, with species of different phenotypes clustering together [7].

A large body of research has looked at biological phase separation through the
lens of equilibrium thermodynamics, attributing the ordering to the
minimization of free energy. A notable thermodynamic model for cell sorting is
the differential adhesion hypothesis (DAH) [8, 9], which proposes that cells
preferentially adhere to other cells of the same type because of differences
in surface tension between like and unlike cells. Other thermodynamic
approaches include considerations of line tension [10, 3] and surface
contraction [11, 3] of cells. However, biological matter is inherently out of
thermodynamic equilibrium - which opens the possibility of phase ordering
mechanisms that are outside the realm of free energy minimization principles
[12].

An important class of non-equilibrium systems is active matter, which deals
with the collective behaviour of self-motile particles. Motility-induced phase
separation (MIPS) [13, 14] is an example of active phase separation, where
self-propelled particles can become trapped in regions of high density thus
forming a dense phase and a dilute phase. Many other novel mechanisms of
ordering in active systems have been reported in the literature, including
aligning torques [15], bond formation and breaking [16], control of boundary
conditions [17], and the hydrodynamic interactions between dumbbell-shaped
swimmers [18]. Continuum models of scalar active matter have been used to
study phase separation in active Brownian particles [19], self-propelled
particles [20, 21], poroelastic materials [22] and cellular aggregates [23].
These show steady states that range from bulk phase separation to bubbles,
droplets, elongated filaments and active foams. Recent work on active phase
field models [24, 25] and a vertex model [26] on mixed cell layers have also
shown phase separation.

Active nematics [27, 28, 29] comprise rod-like particles with orientational
order, which pump energy into their surroundings by generating dipolar
stresses along their long axes. These models have been successfully used to
describe the motility of Madin-Darby Canine Kidney (MDCK) cells [30],
spontaneous flow in confined cell channels [31], active turbulence in
microtubule-kinesin mixtures [32], and topological defects in growing
bacterial colonies [33]. Recently, Assante et. al. [34] showed that coupling
concentration and nematic ordering can lead to spontaneous microphase
separation in inhomogeneous active nematics, and we used a continuum theory to
study active phase separation, driven by flows, in a mixture of an active
nematic and a passive isotropic fluid [35]. In this paper, we extend this work
to discuss mixtures of two nematics, with different activities, coupled by
viscous drag. This is motivated by recent experiments which demonstrate cell
sorting in mixtures of extensile and contractile cells [24, 36].

The paper is organized as follows: In Sec. 2, we extend the active two-fluid
model introduced in Ref. [35] to describe interacting active nematics. In Sec.
3, we discuss the angle between the orientations of the two active species
when they are coupled only by viscous drag. In Sec. 4, we move on to phase
separation in active-active mixtures. We review the mechanism discussed in
Ref. [35], and show how this applies when both species are active, and how the
phase ordering depends on the elastic coupling between nematogens. In Sec. 5,
we discuss how changing concentration fractions and friction affect phase
separation in an extensile-contractile mixture. Finally, in Sec. 6, we
conclude with a summary of our results.

##  II Model

We study a mixture of two active fluids [37, 38, 22, 35]. Each fluid component
has a local density $\rho^{i}$, velocity field $u_{\alpha}^{i}$, and chemical
potential $\mu^{i}$, where Latin superscripts $i=1,2$ index the different
fluids, and the Greek subscripts $\alpha=1,2$ denote the spatial directions.
Summation convention is used for the Greek indices but will be specified
explicitly for the Latin indices when applicable.

Each component fluid obeys the mass continuity equation

$\partial_{t}\rho^{i}+\nabla_{\alpha}\rho^{i}u_{\alpha}^{i}=0$ (1)

and the momentum balance equation

$\begin{split}\partial_{t}\rho^{i}u_{\alpha}^{i}+\nabla_{\beta}\rho^{i}u_{\alpha}^{i}u_{\beta}^{i}=-\rho^{i}\nabla_{\alpha}\mu^{i}+F_{\alpha}^{visc,i}+F_{\alpha}^{body,i}\\\
-f\phi^{i}u_{\alpha}^{i}+\gamma\phi(1-\phi)(u_{\alpha}^{3-i}-u_{\alpha}^{i}),\end{split}$
(2)

where

$\begin{split}\rho^{c}&=\rho^{1}+\rho^{2}\,,\\\
\phi^{1}&=\phi=\rho^{1}/\rho^{c}\,,\\\
\phi^{2}&=1-\phi=\rho^{2}/\rho^{c}\,\end{split}$ (3)

are the total density, concentration fraction of component 1, and
concentration fraction of component 2 respectively. In Eq. (2), the left-hand
side denotes the convective derivative of the fluid momentum density
($\rho^{i}u_{\alpha}^{i}$), while the right-hand side describes the force
acting on the fluid per unit volume. The forces are modelled by a
thermodynamic force ($-\rho^{i}\nabla_{\alpha}\mu^{i}$), a viscous drag
between the component fluids
$\gamma\phi(1-\phi)(u_{\alpha}^{3-i}-u_{\alpha}^{i})$, an internal viscous
dissipation for each fluid $F_{\alpha}^{visc,i}$, a substrate friction term
$-f\phi^{i}u_{\alpha}^{i}$, and a body force $F_{\alpha}^{body,i}$ which
models the local forces generated by nematic stresses.

We formulate our equations to treat this system as an incompressible fluid
with compressible components. In order to do so, we define new velocity fields

$\begin{split}u_{\alpha}^{c}&=\phi u_{\alpha}^{1}+(1-\phi)u_{\alpha}^{2}\,,\\\
\delta u_{\alpha}&=u^{1}_{\alpha}-u^{2}_{\alpha},\end{split}$ (4)

which are the centre of mass velocity of the total fluid and relative flow
between the fluids respectively. We reserve the superscript $c$ to refer to
the combined (centre of mass) fluid. Moving forward, we will assume that the
relative flow is much smaller than the combined velocity of the fluid i.e.
$|\delta\mathbf{u}|\ll|\mathbf{u}^{c}|$ .

Adding Eqs. (1), (2) for each component, and neglecting terms of order
$(\delta\mathbf{u})^{2}$, gives the equations of motion for the combined fluid
[39, 40]:

$\partial_{t}\rho^{c}+\nabla_{\alpha}\rho^{c}u_{\alpha}^{c}=0,$ (5)
$\begin{split}\partial_{t}\rho^{c}u_{\alpha}^{c}+\nabla_{\beta}\rho^{c}u_{\alpha}^{c}u_{\beta}^{c}&=\sum_{i=1}^{2}\big{[}-\rho^{i}\nabla_{\alpha}\mu^{i}+\\\
&F_{\alpha}^{visc,i}-f\phi^{i}u_{\alpha}^{i}+F_{\alpha}^{body,i}\big{]}.\end{split}$
(6)

Notice that the viscous drag between the components drops out of the equation
for the combined fluid. The combined fluid conserves mass density and momentum
and acts as a typical incompressible fluid.

We now discuss the terms on the right-hand side of the momentum balance
equation (6) in turn. The first term is the thermodynamic force, which follows
from a Ginzburg-Landau free energy functional of the form [39, 40]

$\mathcal{F}_{LG}=\int
d^{2}\mathbf{r}\,\big{(}\psi(\rho^{1},\rho^{2})+\frac{1}{2}\kappa||\mathbf{\nabla}\phi||^{2}\big{)}$
(7)

where

$\psi=\frac{1}{3}\rho^{c}\ln\rho^{c}+\rho^{c}\\{a\,(\phi-\frac{1}{2})^{2}+b\,(\phi-\frac{1}{2})^{4}\\}.$
(8)

The first term of (8) promotes incompressibility of the combined fluid, with
an isothermal equation of state. The second term is a Landau free energy which
drives the system to a uniformly mixed configuration if $a\geq 0,b\geq 0$. The
chemical potentials for each fluid are defined as
$\mu^{i}=\partial\mathcal{F}_{LG}/\partial\rho^{i}$. We can write [39, 40] the
chemical potential term for the combined fluid
$\sum_{i=1}^{2}-\rho^{i}\nabla_{\alpha}\mu^{i}$ as the divergence of a stress
tensor $-\nabla_{\beta}\sigma^{thermo}_{\alpha\beta}$, where

$\sigma^{thermo}_{\alpha\beta}=p\delta_{\alpha\beta}+\kappa(\nabla_{\alpha}\nabla_{\beta}\phi-\frac{1}{2}||\mathbf{\nabla}\phi||^{2}\delta_{\alpha\beta}),$
(9)
$-p=\psi-\frac{\partial\psi}{\partial\rho^{1}}\rho^{1}-\frac{\partial\psi}{\partial\rho^{2}}\rho^{2}.$
(10)

$p=\rho^{c}/3$ is an isotropic pressure consistent with an isothermal equation
of state. The term in $\kappa$ is an anisotropic stress resulting from the
surface tension between the two component fluids.

The second term on the right-hand side of Eq. (6) is the usual viscous stress,
defined by

$F_{\alpha}^{visc,i}=\nabla_{\beta}\,\eta^{i}(\nabla_{\alpha}u^{i}_{\beta}+\nabla_{\beta}u^{i}_{\alpha}-\delta_{\alpha\beta}\nabla_{\gamma}u^{i}_{\gamma}).$
(11)

The third term describes the friction between each component fluid and the
substrate. The final term in Eq. (6) is the body force acting locally on the
fluid at each point which arises from passive and active nematic stresses
$F_{\alpha}^{body,i}\equiv F_{\alpha}^{Q,i}$. We next discuss the dynamics of
each active nematic species and the form of $F_{\alpha}^{Q,i}$.

Each fluid component $i$ is a nematic liquid crystal [27, 28, 29] described by
a symmetric traceless tensor [41]

$Q^{i}_{\alpha\beta}=S^{i}_{nem}(2n^{i}_{\alpha}n^{i}_{\beta}-\delta_{\alpha\beta})$
(12)

in 2D where $n^{i}_{\alpha}$ is a headless vector denoting the orientation of
the local nematic order, called the director field, and $S^{i}_{nem}$ is the
magnitude of the nematic order. The order parameter relaxes towards the
minimum of the Landau-de Gennes free energy density [29, 41]

$\mathcal{F}^{LdG,i}=C^{i}(\,(S^{i}_{nem})^{2}-Q^{i}_{\alpha\beta}Q^{i}_{\alpha\beta})^{2}+\frac{K^{i}}{2}\nabla_{\gamma}Q^{i}_{\alpha\beta}\nabla_{\gamma}Q^{i}_{\alpha\beta}$
(13)

defined so that the minimum free energy corresponds to a state with an order
parameter of magnitude $S_{nem}^{i}$, with the director uniformly aligned in
space.

We couple the orientation fields of the two species explicitly by adding an
extra aligning or anti-aligning interaction in the free energy of the form

$\mathcal{F}^{Q_{1},Q_{2}}=-L_{12}\phi(1-\phi)\,Q_{\alpha\beta}^{1}\,Q_{\alpha\beta}^{2}.$
(14)

When both species are mixed together, this term aligns the director fields for
$L_{12}>0$, and anti-aligns them for $L_{12}<0$. From Eqs. (7), (13) and (14)
the total free energy of the system is given by

$\mathcal{F}=\mathcal{F}_{LG}+\mathcal{F}^{LdG,1}+\mathcal{F}^{LdG,2}+\mathcal{F}^{Q_{1},Q_{2}}.$
(15)

We now describe the dynamics of the orientation field. We expect the director
field to not only be advected by the fluid but also to be rotated according to
the vorticity tensor
$\Omega^{i}_{\alpha\beta}=(\partial_{\alpha}u^{i}_{\beta}-\partial_{\beta}u^{i}_{\alpha})/2$.
Additionally, the director may tend to orient along the strain rate tensor
$E^{i}_{\alpha\beta}=(\partial_{\alpha}u^{i}_{\beta}+\partial_{\beta}u^{i}_{\alpha})/2$
of the fluid. The response to gradients of the flow is modelled by the co-
rotation term [41, 29]

$\displaystyle S_{\alpha\beta}^{i}$ $\displaystyle=(\lambda
E^{i}_{\alpha\chi}+\Omega^{i}_{\alpha\chi})(Q^{i}_{\chi\beta}+\delta_{\chi\beta}/2)+$
(16)
$\displaystyle\qquad\qquad(Q^{i}_{\alpha\chi}+\delta_{\alpha\chi}/2)(\lambda
E^{i}_{\chi\beta}-\Omega^{i}_{\chi\beta})$
$\displaystyle\qquad\qquad-2\lambda^{i}(Q^{i}_{\alpha\beta}+\delta_{\alpha\beta}/2)Q^{i}_{\chi\gamma}\nabla_{\chi}u^{i}_{\gamma}.$

where $\lambda^{i}$ (known as the flow-tumbling or flow-aligning parameter)
models the extent to which the director field of species $i$ explicitly
orients with the strain axis of the fluid (the direction of the positive
eigenvalue of $E^{i}_{\alpha\beta}$).

We also expect that the system will evolve towards the minimum of the free
energy. This is modelled by the molecular field [29, 27]

$H_{\alpha\beta}^{i}=-\frac{\partial\mathcal{F}}{\partial
Q^{i}_{\alpha\beta}}+\frac{\delta_{\alpha\beta}}{2}\bigg{(}\frac{\partial\mathcal{F}}{\partial
Q^{i}_{\chi\gamma}}\bigg{)}\delta_{\chi\gamma}.$ (17)

Combining Eqs. (16) and (17), the dynamics of the nematic tensor associated
with fluid $i$ is [42]

$\partial_{t}Q^{i}_{\alpha\beta}+u^{i}_{\chi}\nabla_{\chi}Q^{i}_{\alpha\beta}-S^{i}_{\alpha\beta}=\Gamma
H^{i}_{\alpha\beta}.$ (18)

The nematic field itself generates stresses, which drive flows in the fluid.
This is modelled by a body force

$F^{Q,i}_{\alpha}=\nabla_{\beta}\cdot\Pi^{i}_{\alpha\beta}$ (19)

where the stress tensor is a sum of the elastic and active stresses [29, 28]

$\displaystyle\Pi^{i}_{\alpha\beta}$
$\displaystyle=\Pi_{\alpha\beta}^{i,el}+\Pi_{\alpha\beta}^{i,act},$ (20)
$\displaystyle\Pi^{i,el}_{\alpha\beta}$
$\displaystyle=2\lambda(Q^{i}_{\alpha\beta}+\delta_{\alpha\beta}/2)\,Q^{i}_{\chi\gamma}H^{i}_{\chi\gamma}-\lambda
H^{i}_{\alpha\chi}(Q^{i}_{\chi\beta}+\delta_{\chi\beta}/2)$
$\displaystyle\qquad-\lambda(Q^{i}_{\alpha\chi}+\delta_{\alpha\chi}/2)H^{i}_{\chi\beta}-\nabla_{\alpha}Q^{i}_{\chi\gamma}\frac{\partial\mathcal{F}^{LdG}}{\partial\nabla_{\beta}Q^{i}_{\chi\gamma}}$
$\displaystyle\qquad+Q^{i}_{\alpha\chi}H^{i}_{\chi\beta}-H^{i}_{\alpha\chi}Q^{i}_{\chi\beta},$
(21) $\displaystyle\Pi^{i,act}_{\alpha\beta}$
$\displaystyle=-\zeta_{i}\phi^{i}Q^{i}_{\alpha\beta}.$ (22)

Now, we return to the equation of motion for the first compressible fluid
component. Using Eqs. (1) and (4), the momentum equation (2) for the first
component can be written as

$\begin{split}\rho^{1}[\partial_{t}u_{\alpha}^{1}+u_{\beta}^{1}\nabla_{\beta}u_{\alpha}^{1}]=-\phi\nabla_{\beta}\sigma_{\alpha\beta}+F_{\alpha}^{visc,1}+F_{\alpha}^{body,1}\\\
-f\phi
u_{\alpha}^{1}+[\gamma\phi(u_{\alpha}^{c}-u_{\alpha}^{1})+\phi(1-\phi)\nabla_{\alpha}\delta\mu]\end{split}$
(23)

where $\delta\mu=-\delta\mathcal{F}/\delta\phi$ [39, 40]. Defining an internal
force density acting between the fluids,

$G_{\alpha}=\gamma(u_{\alpha}^{c}-u_{\alpha}^{1})+(1-\phi)\nabla_{\alpha}\delta\mu,$
(24)

Eq. (23) can then be rewritten as

$\rho[\partial_{t}u_{\alpha}^{1}+u_{\beta}^{1}\nabla_{\beta}u_{\alpha}^{1}]=-\nabla_{\beta}\sigma_{\alpha\beta}+G_{\alpha}+\frac{F_{\alpha}^{visc,1}+F_{\alpha}^{body,1}}{\phi}-fu_{\alpha}^{1}.$
(25)

The equation of the second compressible fluid component follows by symmetry.

We numerically solve the combined fluid, described by Eqs. (5)-(6), using the
Lattice-Boltzmann (LB) method [43]. We simultaneously calculate the evolution
of the first component fluid, described by Eqs. (1) and (23), using a LB
method modified to account for compressibility [39, 40]. The dynamics of the
nematic fields and stresses (Eqs. (18)-(22)) are solved using a finite
difference approach [44].

We used the parameters $\rho^{C}=40$, $\gamma=4$, $\kappa=5$, $a=0.0001$,
$b=0.0001$, $\eta^{1}=\eta^{2}=10/3$, $S_{nem}^{1}=S_{nem}^{2}=1$,
$C^{1}=C^{2}=K^{1}=K^{2}=0.1$, $L_{12}=0.1$, $\lambda=0$, $f=0$, and
$\Gamma^{1}=\Gamma^{2}=0.1$, unless otherwise specified. $\zeta_{1},\zeta_{2}$
were varied in the range $[-0.1,0.1]$. We simulated the equations for 50,000
time steps on a 200 x 200 grid with periodic boundary conditions. The initial
condition was chosen to be $\rho^{1}=\rho^{2}=20$, with the initial director
and velocity field configurations obtained by simulating 500 LB time-steps
from a randomly initialized director configuration without active forcing.

##  III Active flows introduce director alignment between nematic fluid
components

Figure 1: Alignment between nematic director fields. (a) Probability density
function shows that for $\lambda=0$, contractile nematics tend to align
perpendicular to the extensional strain axis (blue), while extensile nematics
align parallel (red) ($\zeta_{1}=-0.10,\zeta_{2}=0.067$). (b) Contour plot
showing nematic alignment between the different components for different
activities for $\lambda=0$. The colourbar shows
$\langle\cos(2(\theta_{1}-\theta_{2}))\rangle$ which is +1 (yellow) for
parallel and -1 (blue) for perpendicular. Circles denote individual
simulations, while the contour plot shows the interpolated values. (c)
Probability density function shows that for $\lambda=1.6$, contractile
nematics have a bimodal distribution preferentially aligning either parallel
and perpendicular to the extensional strain axis (blue), while extensile
nematics tend to align parallel (red) ($\zeta_{1}=-0.10,\zeta_{2}=0.05$). (d)
Cross-section of nematic alignment while varying $\zeta_{1}$, with
$\zeta_{2}=-0.10$. The $\lambda=0$ cross-section is marked by the black dotted
line in the contour plot. Changing to $\lambda=1.6$ changes the nematic
alignment significantly in both contractile-contractile ($\zeta_{1}<0$) and
contractile-extensile ($\zeta_{1}>0$) mixtures.

In this section, we argue that there is alignment (or anti-alignment) between
the two components of the nematic mixture due to the director fields aligning
with the rate of strain tensor in the combined fluid. Throughout this section,
we set $L_{12}=0$ and $a=0.2$ to study flow-induced director-director coupling
in the absence of imposed elastic alignment or phase separation. We first
consider the flow tumbling parameter $\lambda=0$, and then discuss the effects
of a non-zero $\lambda$.

###  III.1 Zero flow-tumbling parameter ($\lambda=0$)

Assuming that the momentum balance in each fluid (Eq. (2)) is dominated by
viscous and active stresses then, for $\lambda=0$, Eqs. (11) and (22) give a
force balance

$\begin{split}\nabla_{\beta}\cdot\eta^{i}E^{c}_{\alpha\beta}=\zeta_{i}\nabla_{\beta}\cdot
Q^{i}_{\alpha\beta}\qquad\\\ \end{split}$ (26)

for each of the fluid components. Here, we have assumed that both component
fluids have approximately the same strain rate $E_{\alpha\beta}^{c}$.

If $\zeta_{i}>0$ (extensile), $Q_{\alpha\beta}$ has the same sign as
$E_{\alpha\beta}$, and the nematic aligns along the stretching direction of
the combined fluid ($\theta_{director}\parallel\theta_{strain}$). If
$\zeta_{i}<0$ (contractile), $Q_{\alpha\beta}$ and $E_{\alpha\beta}$ have
opposite signs, and the nematic aligns along the compression direction of the
combined fluid ($\theta_{director}\perp\theta_{strain}$). This is shown in
Fig. 1(a). The spread in the distribution is due to elasticity in the nematic
field of each component and the effect of the passive backflow terms in the
Navier Stokes equations which are neglected in Eq. (26).

When two active fluids are strongly coupled through viscous drag, the
alignment of each nematic with the combined fluid velocity gradient induces an
effective alignment between the two director fields. Both species orient in
the same (perpendicular) direction if they have the same (opposite) sign of
activity. The overall degree of alignment can be characterised by calculating
the average value of $\cos(2(\theta_{1}-\theta_{2}))$, where $\theta_{i}$ is
the angle of the $i$-th director field. This quantity is +1 for parallel
alignment and -1 for perpendicular alignment as shown in Fig. 1(b).

###  III.2 Effect of the flow-tumbling parameter $\lambda$

We now consider the effects of having a large flow-tumbling parameter
$\lambda$. The passive elastic terms proportional to $\lambda$ in the stress
(Eq.(21)) tend to align the nematic director field with the extensional strain
axis, regardless of active flows, for both extensile and contractile nematics.
For an extensile fluid this merely slightly enhances the alignment along the
strain axis caused by the activity. However, the angle between contractile
nematogens and the strain axis forms a bimodal distribution with some regions
aligning parallel to the strain, due to elastic flows, and others
perpendicular to the strain, due to active flows, with the relative fraction
of each alignment dependent on $\lambda$. (Fig. 1(c)).

This in turn can affect the effective alignment between the director fields of
the two components. For extensile-extensile mixtures, there is just a small
decrease in $\langle\theta_{1}-\theta_{2}\rangle$. For an extensile-
contractile mixture, some contractile directors align with the extensional
strain axis, leading to an increase in alignment between the two species. For
a contractile-contractile mixture, both director fields are frustrated, and
$\langle\theta_{1}-\theta_{2}\rangle$ depends on the fraction of director
fields which align parallel or perpendicular to the strain. In Fig. 1(d), we
plot the average nematic alignment
$\langle\cos(2(\theta_{1}-\theta_{2}))\rangle$ along a cross-section varying
$\zeta_{1}$ keeping $\zeta_{2}=-0.10$ fixed, for both $\lambda=0$ and
$\lambda=1.6$.

##  IV Phase Separation in Active-Active Mixtures

In previous work, we showed that an active nematic fluid mixed with a passive
isotropic fluid spontaneously orders to form microphase-separated domains
[35], even in the absence of any terms in the free energy favouring phase
separation. The domains form and disintegrate chaotically in an active
turbulent background, a state we term ‘turbulent microphase separation’.

In this section, we first review the phase-separation mechanism and then apply
it to the case of two active nematic fluids of different activities. Secondly,
we determine how the strength of phase separation depends on the activities of
each species. Finally, we look at how the results change when the nematic
species are constrained to align with each other through elastic interactions.

###  IV.1 Mechanism

Figure 2: Mechanism of Phase Separation: (a) Interface normal $\mathbf{m}$,
tangent $\mathbf{l}$, and nematic director $\mathbf{n}$ at an interface.
Schematic representation of the mechanism of phase separation for (b) a
contractile component 1 (blue) and (c) an extensile component 2 (yellow). Red
arrows show the forces at the interface created by each nematogen, and black
arrows show the net direction of the active force. (d)–(f) simulation data
demonstrating the mechanism for $\zeta_{1}=-0.10,\zeta_{2}=0.067$, and
$L_{12}=0$: (d) Contractile species align homeotropically at interfaces. (e)
Extensile species align parallel to the interface. (f) The relative flow
between the fluid components tends to orient normal to the interface, in the
direction of increasing concentration gradient. (g) Relative flows between the
fluids are stronger at the interface. The trend lines show the mean, combining
data from 50 measurements taken in intervals of 1000 timesteps. The error bars
indicate the standard deviation of the mean. The shaded region shows the
standard deviation in the spread of the measured quantities.

Active phase separation begins when the two component species generate a flow
fluctuation which locally moves them in different directions, setting up a
small concentration gradient in the fluid. This difference in concentration
leads to an imbalance in active stress across the interface, which drives
active flows normal and tangential to the interface in each fluid.

Consider how each active fluid component $i$ behaves at an interface. This
fluid generates tangential and normal active forces per unit volume given by
$\mathbf{F}^{\footnotesize{\mbox{tangential},i}}=2\zeta_{i}|\nabla(S_{nem}^{i}\phi^{i})|\,(\mathbf{m\cdot
n})(\mathbf{l\cdot n})\,\mathbf{l}$ and
$\mathbf{F}^{\footnotesize{\mbox{normal},i}}=-\zeta_{i}|\nabla(S_{nem}^{i}\phi^{i})|\,(2(\mathbf{m\cdot
n})^{2}-1)\,\mathbf{m}$ respectively [35, 44]. Here $\mathbf{m}$ and
$\mathbf{l}$ are unit vectors normal (pointing away from the more active
region) and tangential to the interface respectively, and $\mathbf{n}$ is a
unit vector along the nematic director as shown in Fig. 2(a).

The tangential flows acting on the fluid component tend to orient its director
parallel to the interface for extensile nematics, and perpendicular to the
interface for contractile nematics [44]. Due to this active anchoring the
active stresses generated by each fluid component, and acting on that
component, tend to point normal to the interface and towards the region of
higher concentration, as shown by the black arrows in Fig. 2(b) for a
contractile nematic, and Fig. 2(c) for an extensile one. Since both species
are active, each fluid generates its own active stresses and flows. If the sum
of the active forces is stronger than the passive restoring forces, this
creates relative flows between the two components, magnifying the
concentration difference across the interface further, leading to phase
separation.

Figure 3: Phase separation in active-active mixtures: (a) Snapshot of the
concentration field for a turbulent microphase separated state without imposed
alignment, $L_{12}=0$ (corresponding to the point marked by $\clubsuit$ in
panel (b)). The colourbar shows local concentration $\phi$. (b) Magnitude of
the phase separation, $\Delta$, for different activities. Circles denote
individual simulations, while the contour plot shows the interpolated values.
(c) Snapshot of the concentration field for a turbulent microphase separated
state with imposed alignment, $L_{12}=0.1$ (corresponding to the point marked
by $\spadesuit$ in panel (d)). (d) Magnitude of the phase separation,
$\Delta$, for different activities. Circles denote individual simulations,
while the contour plot shows the interpolated values. Note the increase in the
magnitude of the ordering when there is elastic alignment between the director
fields of the fluid components. Variation of (e) the magnitude of the
ordering, $\Delta$, and (f) the relative flow magnitude, $|\delta\mathbf{u}|$,
with $\zeta_{1}$ for $\zeta_{2}=0.067$ (i.e. along the red dotted lines in (b)
and (d)), comparing $L_{12}=0$ and $L_{12}\neq 0$.

We numerically verify this mechanism by looking at the active anchoring and
flow alignment at concentration gradients. We define the director angle of
species $i$ as $\theta_{i}$, the orientation of the relative flow velocity
$\delta\mathbf{u}$ as $\theta_{v}$, and the direction of the gradient of
$\phi$ as $\theta_{\phi}$. $\theta_{\phi}$ is normal to the interface and
points towards increasing $\phi$. We quantify anchoring by measuring
$\langle\cos 2(\theta_{i}-\theta_{\phi})\rangle$, the angle between the
director field and concentration gradient, which ranges from $+1$ for
homeotropic anchoring to $-$1 for planar anchoring. At interfaces, the
contractile fluid tends to align normal to the interface (Fig. 2(d)) while the
extensile fluid prefers to align tangentially (Fig. 2(e)). We also check the
orientation of the relative flow $\delta\mathbf{u}$ with respect to the normal
to the interface by measuring $\langle\cos(\theta_{v}-\theta_{\phi})\rangle$,
which is +1 for relative flows pointing towards higher $\phi$, and -1 for
relative flows towards lower $\phi$. Figure 2(f) shows that the net flow
between the fluid components tends to orient normal to the interface, in the
direction of increasing concentration gradient. Finally Fig. 2(g) confirms
that the magnitude of the relative flow $\langle|\delta\mathbf{u}|\rangle$ is
stronger at interfaces. (See SM Fig. A1 for a similar figure confirming the
same mechanism for imposed alignment between the nematogens, $L_{12}\neq 0$.)

###  IV.2 No imposed alignment between directors

In this subsection, we look at phase separation when there is no imposed
elastic alignment between the nematic director fields of the two components
($L_{12}=0$). Viscous drag between the fluids aligns (anti-aligns) the
directors if the two species have the same (opposite) sign of activity. In all
cases, we observe chaotic turbulent microphase separation, with phases forming
and dissociating rapidly, similar to Ref. [35]. A snapshot of the
concentration field is shown in Fig. 3(a) (see also Movie 1 in the SM).

We characterize the magnitude of phase separation by calculating $\Delta$, the
standard deviation of the concentration field, which quantifies the variation
from the uniformly mixed state. A contour plot showing how $\Delta$ depends on
the activities $\zeta_{1}$ and $\zeta_{2}$ is shown in Fig. 3(b). The
magnitude of phase separation depends on the difference in magnitude of the
activities, but not on their signs.

This is because, although changing the sign of activity changes the relative
alignment of the directors, the flow fields in both fluids remain in the same
direction. Thus the highest phase separation is observed when an active
species is mixed with a passive one, corresponding to the maximum relative
flows between the two fluids.

Figure 4: Changing Concentration Fraction: (a) Concentration field for a 30-70
mixture which phase separates into circular droplets
($\zeta_{1}=0.02,\zeta_{2}=-0.005,$ and $\bar{\phi}=0.3$). The colourbar shows
local concentration $\phi$. (b) Normalised magnitude of phase separation,
$\Delta/\Delta_{max}$, as a function of concentration fraction, $\bar{\phi}$,
for three different activities $\zeta_{2}$, while holding $\zeta_{1}=0.05$
fixed. (Inset) $\Delta/\Delta_{max}$ plotted as a function of effective
activity $\zeta_{eff}=\zeta_{1}\bar{\phi}+\zeta_{2}(1-\bar{\phi})$ is highest
at low effective activity.

###  IV.3 Imposing director alignment

We next impose $L_{12}\neq 0$, so that the director fields of the two nematic
components are strongly aligned. A contour plot showing the magnitude of phase
separation on varying the activities $\zeta_{1}$ and $\zeta_{2}$ is shown in
Fig. 3(d).

If both species have the same sign of activity, the imposed alignment merely
reinforces the flow-induced alignment discussed in Sec. III. The resultant
phases look very similar to Fig. 3(a), but the phase ordering is slightly
lower because the active flows are better aligned.

However, the magnitude of the phase separation increases very significantly if
the fluid components have opposite signs of activity. The relative flow
between the two components is much higher in the extensile-contractile case
because each fluid generates active forces in opposite directions at a
concentration gradient. The strongest phase separation, corresponding to the
largest spread of concentration $\max(\phi)-\min(\phi)\approx 1$, is observed
when a highly extensile fluid is mixed with a highly contractile one. The
fluids form elongated droplet networks with large differences in concentration
between the different regions, as shown in Fig. 3(c) (see also Movie 2 in the
SM).

To directly compare the flow-alignment and imposed alignment cases the
variation of the magnitudes of phase separation, $\Delta$, and the relative
flow between the fluid components, $\delta\mathbf{u}$, with $\zeta_{1}$ for a
fixed $\zeta_{2}$ are plotted in Fig. 3(e) and (f).

##  V Varying other parameters in the system

###  V.1 Concentration Fractions

We consider the effect of changing $\bar{\phi}$, the average of $\phi$
(concentration fraction of species 1), to study how the phase separation is
affected if the system is not a 50-50 mixture of each species. On reducing
$\bar{\phi}$, the elongated droplet network of Fig. 3(a) is replaced by small
droplets of fluid 1 in a background of fluid 2 as shown in Fig. 4(a) (and
Movie 3 in the SM). At lower activities or higher surface tensions, the
droplets are rounder and the flows are less turbulent.

For a given pair of activities $\zeta_{1}$ and $\zeta_{2}$, different
concentration fractions $\bar{\phi}$ give different degrees of phase
separation $\Delta$. The maximum possible $\Delta$ for a perfectly phase-
separated system with infinitely sharp interfaces is
$\Delta_{max}=\sqrt{\bar{\phi}(1-\bar{\phi})}$. Since this is dependent on
$\bar{\phi}$, we quantify the magnitude of phase separation at different
$\bar{\phi}$ using the normalized $\Delta/\Delta_{max}$.

A plot of $\Delta/\Delta_{max}$ for three different values of $\zeta_{2}$ is
shown in Fig. 4(b) ($\zeta_{1}=0.05$). We note that stronger phase separation
is achieved when the effective activity
$\zeta_{eff}=\zeta_{1}\bar{\phi}+\zeta_{2}(1-\bar{\phi})$ is small (See inset,
Fig. 4(b)) as this suppresses the active turbulence which leads to droplets
breaking up. This implies that the highest phase separation is observed when
the more active component has a smaller concentration fraction.

###  V.2 Friction

Figure 5: Changing Substrate Friction: (a) Concentration field for high
substrate friction ($\zeta_{1}=0.05,\zeta_{2}=-0.05,f=4$), showing large,
well-separated domains. The colourbar shows local concentration $\phi$. (b)
Magnitude of phase ordering, $\Delta$, increases with higher substrate
friction. (c) Size of phase-separated regions, $L_{\phi}$, increases
significantly with higher substrate friction.

Finally, we look at the effect of adding substrate friction $f$. On increasing
friction, the active flows are weaker and less turbulent. The elongated
droplet networks tend to aggregate to form more strongly phase-separated
regions that are larger in size. A snapshot of the concentration profile is
shown in Fig. 5(a) (see also Movie 4 in the SM). Fig. 5(b) shows the increase
in the magnitude of phase separation upon increasing the substrate friction.
The typical lengthscales of the phase-separated domains, $L_{\phi}$, also
increase with higher substrate friction (Fig. 5(c)).

At first glance, it might appear surprising that high substrate friction
facilitates phase separation driven by active flows. However, although the
overall dynamics are slower, the active turbulent flows are smaller in
magnitude, and dissociation by active instabilities is very rare. Moreover,
the force balance condition changes and the velocity is now directly
proportional to the applied force $\mathbf{v}^{i}\approx\mathbf{F}^{i}/f$,
instead of $\mathbf{\nabla}^{2}v^{i}\approx\mathbf{F}^{i}/\eta$. As a result,
although the velocity field is weaker, it is better aligned at the interface,
as shown in Fig. A2 in the SM.

##  VI Discussion

To summarise, we have extended a two-fluid model to study mixtures where both
species are active nematics. We argued that even in the absence of any
externally imposed elastic ordering between components, active nematics
coupled by viscous drag tend to align or anti-align depending on the relative
signs of the activities and the value of the flow-tumbling parameter.

We observed turbulent microphase separation in mixtures of two active nematic
fluids, each with a different activity. Imposing an elastic director alignment
between the two active species plays a major role in determining the magnitude
of the phase separation. In the absence of imposed alignment, there is weak
segregation which is most pronounced for a mixture of a highly active and a
passive component. However, in the presence of imposed alignment, coexisting
domains comprised almost entirely of one component can be achieved with an
extensile–contractile fluid mixture.

The degree and morphology of phase separation are also affected by other
parameters in the system. As expected, varying the concentration fraction can
change the morphology from elongated droplet networks to isolated circular
drops. Perhaps more surprisingly, increasing substrate friction leads to
stronger phase separation and a considerable increase in the size of the
phase-separated domains.

Our results add to a growing body of work describing the possibility of phase
separation driven by activity, here focusing on active flows in fluid
mixtures. Cell ordering and sorting in embryogenesis, and compartmentalisation
into different cell types, are important biological processes where activity
may play a role [24, 36, 25, 26, 12]. Our work may also be relevant in
understanding organization and movement in multi-species bacterial colonies
[45, 33], or the intracellular liquid-liquid phase separation [46, 1] which
results in membrane-less organelles.

With biological examples in mind, in future work, it will be interesting to
consider nematic mixtures above the nematic-isotropic transition temperature,
compare phase ordering in the vertex model or multiphase-field description of
cells, and study the interplay between confinement, wetting and phase ordering
in active materials.

###### Acknowledgements.

We thank A. Mietke, J. Rozman, J. N. Graham, and I. Hadjifrangiskou for
valuable discussions, and J. Rozman for helpful comments on the manuscript. SB
acknowledges support from the Rhodes Trust and the Crewe Graduate Scholarship.
JMY acknowledges support from the UK EPSRC (Award EP/W023849/1).

## References

  * Hyman  _et al._ [2014] A. A. Hyman, C. A. Weber, and F. Jülicher, Annual Review of Cell and Developmental Biology 30, 39 (2014).
  * Banani  _et al._ [2017] S. F. Banani, H. O. Lee, A. A. Hyman, and M. K. Rosen, Nature Reviews Molecular Cell Biology 18, 285 (2017).
  * Krens and Heisenberg [2011] S. F. G. Krens and C.-P. Heisenberg, Current Topics in Developmental Biology 95, 189 (2011).
  * Suzuki  _et al._ [2021] S. Suzuki, I. Omori, R. Kuraishi, and H. Kaneko, Development, Growth & Differentiation 63, 343 (2021).
  * Sharrock and Sanson [2020] T. E. Sharrock and B. Sanson, Seminars in Cell & Developmental Biology 107, 147 (2020).
  * Hogan [1999] B. L. Hogan, Cell 96, 225 (1999).
  * Nadell  _et al._ [2016] C. D. Nadell, K. Drescher, and K. R. Foster, Nature Reviews Microbiology 14, 589 (2016).
  * Steinberg [1975] M. S. Steinberg, Journal of Theoretical Biology 55, 431 (1975).
  * Foty and Steinberg [2005] R. Foty and M. Steinberg, Developmental Biology 278, 255 (2005).
  * Brodland [2002] G. W. Brodland, Journal of Biomechanical Engineering 124, 188 (2002).
  * Harris [1976] A. K. Harris, Journal of Theoretical Biology 61, 267 (1976).
  * Heine  _et al._ [2021] P. Heine, J. Lippoldt, G. A. Reddy, P. Katira, and J. A. Käs, New Journal of Physics 23, 043034 (2021).
  * Cates and Tailleur [2015] M. E. Cates and J. Tailleur, Annual Review of Condensed Matter Physics 6, 219 (2015).
  * Gonnella  _et al._ [2015] G. Gonnella, D. Marenduzzo, A. Suma, and A. Tiribocchi, Comptes Rendus Physique 16, 316 (2015).
  * Zhang  _et al._ [2021] J. Zhang, R. Alert, J. Yan, N. S. Wingreen, and S. Granick, Nature Physics 17, 961 (2021).
  * Gokhale  _et al._ [2022] S. Gokhale, J. Li, A. Solon, J. Gore, and N. Fakhri, Physical Review E 105, 054605 (2022).
  * Thutupalli  _et al._ [2018] S. Thutupalli, D. Geyer, R. Singh, R. Adhikari, and H. A. Stone, Proceedings of the National Academy of Sciences of the United States of America 115, 5403 (2018).
  * Furukawa  _et al._ [2014] A. Furukawa, D. Marenduzzo, and M. E. Cates, Physical Review E 90, 022303 (2014).
  * Stenhammar  _et al._ [2013] J. Stenhammar, A. Tiribocchi, R. J. Allen, D. Marenduzzo, and M. E. Cates, Physical Review Letters 111, 145702 (2013).
  * Wittkowski  _et al._ [2014] R. Wittkowski, A. Tiribocchi, J. Stenhammar, R. J. Allen, D. Marenduzzo, and M. E. Cates, Nature Communications 5, 4351 (2014).
  * Tiribocchi  _et al._ [2015] A. Tiribocchi, R. Wittkowski, D. Marenduzzo, and M. E. Cates, Physical Review Letters 115, 188302 (2015).
  * Weber  _et al._ [2018] C. A. Weber, C. H. Rycroft, and L. Mahadevan, Physical Review Letters 120, 248003 (2018).
  * Kuan  _et al._ [2021] H.-S. Kuan, W. Pönisch, F. Jülicher, and V. Zaburdaev, Physical Review Letters 126, 018102 (2021).
  * Balasubramaniam  _et al._ [2022] L. Balasubramaniam, R.-M. Mège, and B. Ladoux, Current Opinion in Genetics & Development 73, 101897 (2022).
  * Graham  _et al._ [2024] J. N. Graham, G. Zhang, and J. M. Yeomans, Soft Matter (2024).
  * Krajnc [2020] M. Krajnc, Soft Matter 16, 3209 (2020).
  * Aditi Simha and Ramaswamy [2002] R. Aditi Simha and S. Ramaswamy, Physical Review Letters 89, 058101 (2002).
  * Marchetti  _et al._ [2013] M. C. Marchetti, J. F. Joanny, S. Ramaswamy, T. B. Liverpool, J. Prost, M. Rao, and R. A. Simha, Reviews of Modern Physics 85, 1143 (2013).
  * Doostmohammadi  _et al._ [2018] A. Doostmohammadi, J. Ignés-Mullol, J. M. Yeomans, and F. Sagués, Nature Communications 9, 3246 (2018).
  * Saw  _et al._ [2017] T. B. Saw, A. Doostmohammadi, V. Nier, L. Kocgozlu, S. Thampi, Y. Toyama, P. Marcq, C. T. Lim, J. M. Yeomans, and B. Ladoux, Nature 544, 212 (2017).
  * Duclos  _et al._ [2018] G. Duclos, C. Blanch-Mercader, V. Yashunsky, G. Salbreux, J. F. Joanny, J. Prost, and P. Silberzan, Nature Physics 14, 728 (2018).
  * Opathalage  _et al._ [2019] A. Opathalage, M. M. Norton, M. P. N. Juniper, B. Langeslay, S. A. Aghvami, S. Fraden, and Z. Dogic, Proceedings of the National Academy of Sciences of the United States of America 116, 4788 (2019).
  * Dell’Arciprete  _et al._ [2018] D. Dell’Arciprete, M. L. Blow, A. T. Brown, F. D. C. Farrell, J. S. Lintuvuori, A. F. McVey, D. Marenduzzo, and W. C. K. Poon, Nature Communications 9, 4190 (2018).
  * Assante  _et al._ [2023] R. Assante, D. Corbett, D. Marenduzzo, and A. Morozov, Soft Matter 19, 189 (2023).
  * Bhattacharyya and Yeomans [2023] S. Bhattacharyya and J. M. Yeomans, Physical Review Letters 130 (2023).
  * Skamrahl  _et al._ [2023] M. Skamrahl, J. Schünemann, M. Mukenhirn, H. Pang, J. Gottwald, M. Jipp, M. Ferle, A. Rübeling, T. A. Oswald, A. Honigmann, and A. Janshoff, Proceedings of the National Academy of Sciences of the United States of America 120, e2213186120 (2023).
  * Levy [1999] S. Levy,  _Two-Phase Flow in Complex Systems_, A Wiley-Interscience publication (Wiley, 1999).
  * Joanny  _et al._ [2007] J. F. Joanny, F. Jülicher, K. Kruse, and J. Prost, New Journal of Physics 9, 422 (2007).
  * Malevanets and M. Yeomans [1999] A. Malevanets and J. M. Yeomans, Faraday Discuss. 112, 237 (1999).
  * Malevanets and M. Yeomans [2000] A. Malevanets and J. M. Yeomans, Computer Physics Communications 127, 105 (2000).
  * de Gennes and Prost [1993] P. de Gennes and J. Prost,  _The Physics of Liquid Crystals_, International Series of Monographs on Physics (Clarendon Press, 1993).
  * Beris and Edwards [1994] A. N. Beris and B. J. Edwards,  _Thermodynamics of Flowing Systems with Internal Microstructure_, Oxford Engineering Science Series (Oxford University Press, New York, 1994).
  * Krueger  _et al._ [2016] T. Krueger, H. Kusumaatmaja, A. Kuzmin, O. Shardt, G. Silva, and E. Viggen,  _The Lattice Boltzmann Method: Principles and Practice_ , Graduate Texts in Physics (Springer, 2016).
  * Blow  _et al._ [2014] M. L. Blow, S. P. Thampi, and J. M. Yeomans, Physical Review Letters 113, 248303 (2014).
  * Hallatschek  _et al._ [2023] O. Hallatschek, S. S. Datta, K. Drescher, J. Dunkel, J. Elgeti, B. Waclaw, and N. S. Wingreen, Nature Reviews Physics 5, 407 (2023).
  * Agudo-Canalejo and Golestanian [2019] J. Agudo-Canalejo and R. Golestanian, Physical Review Letters 123, 018101 (2019).


# Consistent4D: Consistent 360° Dynamic Object Generation from Monocular Video

Yanqin Jiang1, Li Zhang3, Jin Gao1, Weimin Hu1, Yao Yao2 ✉
1CASIA, 2Nanjin University, 3Fudan University
<EMAIL_ADDRESS><EMAIL_ADDRESS>
{jin.gao<EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

In this paper, we present Consistent4D, a novel approach for generating 4D
dynamic objects from uncalibrated monocular videos. Uniquely, we cast the
360-degree dynamic object reconstruction as a 4D generation problem,
eliminating the need for tedious multi-view data collection and camera
calibration. This is achieved by leveraging the object-level 3D-aware image
diffusion model as the primary supervision signal for training Dynamic Neural
Radiance Fields (DyNeRF). Specifically, we propose a Cascade DyNeRF to
facilitate stable convergence and temporal continuity under the supervision
signal which is discrete along the time axis. To achieve spatial and temporal
consistency, we further introduce an Interpolation-driven Consistency Loss. It
is optimized by minimizing the discrepancy between rendered frames from DyNeRF
and interpolated frames from a pre-trained video interpolation model.
Extensive experiments show that our Consistent4D can perform competitively to
prior art alternatives, opening up new possibilities for 4D dynamic object
generation from monocular videos, whilst also demonstrating advantage for
conventional text-to-3D generation tasks. Our project page is
https://consistent4d.github.io/.

Figure 1: Video-to-4D results achieved by our method. We show the renderings
of 2 objects at 2 viewpoints and 3 timestamps.

##  1 Introduction

Perceiving dynamic 3D information from visual observations is one of the
fundamental yet challenging problems in computer vision, which is the key to a
broad range of downstream applications e.g., virtual content creation,
autonomous driving simulation, and medical image analysis. However, due to the
high complexity nature of dynamic 3D signals, it is rather difficult to
recover such information from a single monocular video observation. As a
result, existing dynamic object reconstruction approaches usually take
synchronized multi-view videos as inputs (Li et al., 2022b; a; Shao et al.,
2023), or rely on training data containing effective multi-view cues (e.g.,
teleporting cameras or quasi-static scenes (Li et al., 2021; Pumarola et al.,
2021; Park et al., 2021b; a)). However, current reconstruction approaches
often fail in reconstructing regions that were not observed in input sequences
(Gao et al., 2022a). Moreover, multi-view data capturing requires synchronized
camera rigs and meticulous calibrations, which inevitably limit the methods to
potential real-world applications.

On the other hand, given only a video clip of a dynamic object, humans are
capable of depicting the appearance, geometry, and movement of the object.
This is achieved by prior knowledge of visual appearance accumulated through
human life. In contrast to the multi-view video setting, our approach favors
such simple and practical input settings of a static monocular video. The
static monocular video offers several advantages: ease of collection for
handheld cameras, minimizing the risk of motion blur due to camera movement,
and obviating the need for camera parameter estimation. As a static monocular
video does not provide effective multi-view information for reconstruction, we
instead opt for the generation approaches which demonstrates diminished
reliance on multi-view information.

In this work, we present a novel video-to-4D generation approach termed
Consistent4D. In this approach, we represent dynamic objects through a
specially designed Cascade DyNeRF and leverage a pre-trained 2D diffusion
model to regulate the DyNeRF optimization, inspired by recent advancements in
text-to-3D (Poole et al., 2023; Wang et al., 2022; Chen et al., 2023; Wang et
al., 2023) and image-to-3D (Deng et al., 2022; Tang et al., 2023; Melas-
Kyriazi et al., 2023) techniques. The challenge is to achieve both spatial and
temporal consistency. To tackle this challenge, we introduce an Interpolation-
driven Consistency Loss (ICL), which leverages a pre-trained video
interpolation model to provide spatiotemporally coherent supervision signals.
Notably, the ICL loss not only enhances consistency in 4D generation but also
mitigates multi-face issues in 3D generation. Furthermore, we train a
lightweight video enhancer to enhance the video generated from dynamic NeRF as
a post-processing step.

We have extensively evaluated our approach on both synthetic videos rendered
from animated 3D model and in-the-wild videos collected from the Internet. To
summarize, the contribution of this work includes:

  * •

We propose a video-to-4D framework for dynamic object generation from a
statically captured monocular video. A specially designed Cascade DyNeRF is
applied to represent the object, optimized through the Score Distillation
Sampling (SDS) loss by a pre-trained 2D diffusion model. Moreover, for the
comprehensiveness of the framework, we train a video enhancer to improve the
rendering of 4D object as a post-processing step.

  * •

To address the challenge of maintaining temporal and spatial consistency in 4D
generation task, we introduce a novel Interpolation-driven Consistency Loss
(ICL). The proposed ICL loss can significantly improve consistency, e.g.,
multi-face problem, in both video-to-4D and text-to-3D generation tasks.

  * •

We extensively evaluate our method on both synthetic and in-the-wild videos
collected from the Internet, showing promising results for the new task of
video-to-4D generation.

##  2 Related work

#### 3D Generation

3D generation aims to generate 3D content conditionally or unconditionally.
Early works mainly make use of GAN, i.e., generate category-specific 3D
objects or scenes from random noise after learning the category prior (Schwarz
et al., 2020; Chan et al., 2021; Gao et al., 2022b). Recently, general-purpose
3D generation has been enabled by text-to-image diffusion model pre-trained on
Internet-scale data, and it also becomes more controllable, e.g., controlled
by text prompt or single image. The pioneer work of text-to-3D is DreamFusion
(Poole et al., 2023), which proposes Score Distillation Sampling (SDS) loss to
leverage the image diffusion model for neural radiance field (NeRF) training.
The following works (Lin et al., 2023; Chen et al., 2023; Wang et al., 2023)
further enhance the visual quality of the generated object by using mesh
representation, Variational Score Distillation, etc. However, the challenging
problem, multi-face Janus problem, has not been addressed in the above works.
Image is another popular condition for 3D generation. Different from 3D
reconstruction, which focuses on the reconstruction of visible regions from
multi-view images, 3D generation usually has only a single image and relies
much on the image diffusion model to generate invisible regions of the object
(Melas-Kyriazi et al., 2023; Tang et al., 2023; Liu et al., 2023). So many
works simply translate the image to words using Lora and then exploit text-
to-3D methods (Melas-Kyriazi et al., 2023; Tang et al., 2023; Seo et al.,
2023). One exception is Zero123 (Liu et al., 2023), which trains a 3D-aware
image-to-image model using multi-view data and could generate a novel view of
the object in the input image directly. Benefiting from multi-view data
training, the multi-face Janus problem has been alleviated to some extent in
Zero123.

#### 4D Reconstruction

4D reconstruction, aka dynamic scene reconstruction, is a challenging task.
Some early works focus on object-level reconstruction and adopt parametric
shape models (Loper et al., 2023; Vo et al., 2020) as representation. In
recent years, dynamic neural radiance field become popular, and convenient
dynamic scene reconstruction is enabled. These works can be classified into
two categories: a deformed scene is directly modeled as a NeRF in canonical
space with a time-dependent deformation (Pumarola et al., 2021; Park et al.,
2021a; b; Wu et al., 2022b; Tretschk et al., 2021) or time-varying NeRF in the
world space (Gao et al., 2021; Li et al., 2021; Xian et al., 2021; Fridovich-
Keil et al., 2023; Cao & Johnson, 2023). Some of them require multi-view
synchronized data to reconstruct dynamic scenes, however, data collection and
calibration is not convenient (Li et al., 2022b; Shao et al., 2023). So,
reconstruction from monocular videos gain attention. However, those monocular
methods either require teleporting camera or quaic-static scenes (Pumarola et
al., 2021; Park et al., 2021a; b), which are not representative of daily life
scenarios (Gao et al., 2022a).

#### 4D Generation

4D generation extends 3D generation to space+time domain and thus is more
challenging. Early works are mainly category-specific and adopt parametric
shape models (Zuffi et al., 2017; 2018; Vo et al., 2020; Kocabas et al., 2020)
as representation. They usually take images or videos as conditions and need
category-specific 3D templates or per-category training from a collection of
images or videos (Ren et al., 2021; Wu et al., 2021; Yang et al., 2022; Wu et
al., 2022a). Recently, one zero-shot category-agnostic work, text-to-4D
(Singer et al., 2023), achieves general-purpose dynamic scene generation from
text prompt. It follows DreamFusion (Poole et al., 2023) and extends it to the
time domain by proposing a three-stage training framework. However, the
quality of generated scenes is limited due to low-quality video diffusion
models.

##  3 Preliminaries

###  3.1 Score Distillation Sampling for image-to-3D

Score Distillation Sampling (SDS) is first proposed in DreamFusion (Poole et
al., 2023) for text-to-3D tasks. It enables the use of a 2D text-to-image
diffusion model as a prior for optimization of a NeRF. We denote the NeRF
parameters as $\theta$, text-to-image diffusion model as $\phi$, text prompt
as $\rho$, the rendering image and the reimage as $\mathbf{x}$ and
$\mathbf{z}$, the SDS loss is defined as:

$\nabla_{\theta}\mathcal{L}_{SDS}\left(\phi,\mathbf{x}\right)=\mathbb{E}_{\tau,\epsilon}\left[\omega(t)\left(\hat{\epsilon}_{\theta}\left(\mathbf{z}_{t};\rho,\tau\right)-\epsilon\right)\frac{\partial{\mathbf{x}}}{\partial{\theta}}\right],$
(1)

where $\tau$ is timestamps in diffusion process, $\epsilon$ denotes noise, and
$\omega$ is a weighted function. Intuitively, this loss perturbs $\mathbf{x}$
with a random amount of noise corresponding to the timestep $\tau$, and
estimates an update direction that follows the score function of the diffusion
model to move to a higher density region.

Besides text-to-3D, SDS is also widely used in image-to-3D tasks. Zero123 (Liu
et al., 2023) is one prominent representative. It proposes a viewpoint-
conditioned image-to-image translation diffusion model fine-tuned from Stable
Diffusion (Rombach et al., 2022), and exploits this 3D-aware image diffusion
model to optimize a NeRF using SDS loss. This image diffusion model takes one
image, denoted by $\mathbf{I}_{in}$, and relative camera extrinsic between
target view and input view, denoted by $\left(\mathbf{R},\mathbf{T}\right)$,
as the input, and outputs the target view image. Compared with the original
text-to-image diffusion model, text prompt in Equation 1 is not required in
this model cause the authors translate the input image and the relative camera
extrinsic to text embeddings. Then Equation 1 could be re-written as:

$\nabla_{\theta}\mathcal{L}_{SDS}\left(\phi,\mathbf{x}\right)=\mathbb{E}_{\tau,\epsilon}\left[\omega(t)\left(\hat{\epsilon}_{\theta}\left(\mathbf{z}_{t};\mathbf{I}_{in},\mathbf{R},\mathbf{T},\tau\right)-\epsilon\right)\frac{\partial{\mathbf{x}}}{\partial{\theta}}\right],$
(2)

###  3.2 K-planes

K-planes (Fridovich-Keil et al., 2023) is a simple and effective dynamic NeRF
method which factorizes a dynamic 3D volume into six feature planes (i.e.,
hex-plane), denoted as $P=\\{P_{o}\\}$, where $o\in\\{xy,yz,xz,xt,yt,zt\\}$.
The first three planes correspond to spatial dimensions, while the last three
planes capture spatiotemporal variations. Each of the planes is structured as
a $M\times M\times F$ tensor in the memory, where $M$ represents the size of
the plane and $F$ is the feature size that encodes scene density and color
information. Let $t$ denote the timestamp of a video clip, given a point
$p=(x,y,z,t)$ in the 4D space, we normalize the coordinate to the range
$[0,M)$ and subsequently project it onto the six planes using the equation
$f(p)_{o}=P_{o}(\iota_{o}(p))$, where $\iota_{o}$ is the projection from a
space point $p$ to a pixel on the $o$’th plane. The plane feature $f(p)_{o}$
is extracted via bilinear interpolation. The six plane features are combined
using the Hadamard product (i.e., element-wise multiplication), to produce a
final feature vector as follows:

$f(p)=\prod_{o\in\\{xy,yz,xz,xt,yt,zt\\}}f(p)_{o},$ (3)

Then, the color and density of $p$ is calculated as $c(p)=c(f(p))$ and
$d(p)=d(f(p))$, where $c$ and $d$ denotes mlps for color and density,
respectively.

##  4 Method

Figure 2: Schematic illustration of Consistent4D. The framework consists of
the optimization of a Cascade DyNeRF and the training of a post video
enhancer. The Cascade DyNeRF, which adopts a residual learning strategy, is
supervised by SDS loss from an image-to-image diffusion model. Particularly, a
novel Interpolation-driven Consistency Loss is proposed to compensate for the
spatiotemporal inconsistency brought by the image diffusion model. For post-
processing, we train a lightweight cross-frame video enhancer using GAN to
further improve the quality of the video rendered from DyNeRF.

In this work, we target to generate a 360∘ dynamic object from a statically
captured monocular video. To achieve this goal, we develop a framework
consisting of a DyNeRF and a video enhancer, supervised by the pre-trained 2D
diffusion model in Zero123 and a GAN, respectively. As shown in Figure 2, we
first train a specially designed cascade DyNeRF using SDS loss and image
reconstruction loss. To guarantee spatial and temporal consistency, we propose
a novel Interpolation-driven Consistency Loss as the extra regularization for
the DyNeRF. For post-processing, we apply GAN to train a lightweight video
enhancer, inspired by pix2pix (Isola et al., 2017).To summarize, taking an
uncalibrated monocular video as the input, we obtain a DyNeRF from which we
can render 360∘ view of the dynamic object, and the rendered results can be
further enhanced by the video enhancer.

In the following sections, we will first introduce our design of the Cascade
DyNeRF, and then illustrate the Interpolation-driven Consistency loss. At
last, we will detail the video enhancer.

###  4.1 Cascade DyNeRF

Existing DyNeRF methods mainly assume the supervision signals are temporally
coherent, however, this assumption does not hold in our task since our main
supervision signal is from an image diffusion model. In order to minimize the
impact of temporal discontinuity in the supervision signals, we are prone to
4D representations which naturally guarantee a certain level of temporal
continuity. Therefore, we build our DyNeRF based on K-planes (Fridovich-Keil
et al., 2023) which explots temporal interpolation, an operator naturally
inclined to temporal smoothing. Empirically, reducing the time resolution of
spatiotemporal planes helps enhance temporal consistency, however, this
results in over-smoothed renderings where finer details are lost. In contrast,
increasing the time resolution leads to crisp renderings, but the continuity
of images within the same time series diminishes. To achieve both temporal
continuity and high image quality, we adjust the multi-scale technique in
K-planes and introduce Cascade DyNeRF.

Let us denote the scale index by $s$. In K-planes, multi-scale features are
exploited by concatenation along feature dimension, then the color and density
could be calculated as:

$c(p)=c(\mathrm{concat}(\\{f(p)^{s})\\}_{s=1}^{S}),\;d(p)=d(\mathrm{concat}(\\{f(p)^{s})\\}_{s=1}^{S}),$
(4)

where $S$ is the number of scales. In our setting, the experiment results show
simple concatenation is hard to balance between image quality and temporal
consistency. So we propose to leverage the cascade architecture, which is
commonly used in object detection algorithms (Cai & Vasconcelos, 2018; Carion
et al., 2020), to first output a coarse yet temporally coherent dynamic object
from low-resolution planes and then let the high-resolution planes learn the
residual between the coarse and fine results. That is, the color and density
of the object after scale $s$ is:

$c(p)^{s}=\sum_{k=1}^{s}c(f(p)^{k}),\;d(p)^{s}=\sum_{k=1}^{s}d(f(p)^{k}),$ (5)

where k indicates the scale index. Note that losses are applied to the
rendering results of each scale to guarantee that planes with higher
resolution learn the residual between results from previous scales and the
target object. In this way, we can improve temporal consistency without
sacrificing much object quality. But Cascade DyNeRF alone is not enough for
spatiotemporal consistency, so we resort to extra regularization, please see
in the next section.

###  4.2 Interpolation-driven Consistency Loss

Video generation methods usually train an inter-frame interpolation module to
enhance the temporal consistency between keyframes (Ho et al., 2022; Zhou et
al., 2022; Blattmann et al., 2023). Inspired by this, we exploit a pre-trained
light-weighted video interpolation model and propose Interpolation-driven
Consistency Loss to enhance the spatiotemporal consistency of the 4D
generation.

The interpolation model adopted in this work is RIFE (Huang et al., 2022),
which takes a pair of consecutive images as well as the interpolation ratio
$\gamma~{}(0<\gamma<1)$ as the input, and outputs the interpolated image. In
our case, we first render a batch of images that are either spatially
continuous or temporally continuous, denoted by $\\{\mathbf{x}\\}_{j=1}^{J}$,
where $J$ is the number of images in a batch. Let us denote the video
interpolation model as $\psi$, the interpolated image as $\hat{\mathbf{x}}$,
then we calculate the Interpolation-driven Consistency Loss as:

$\displaystyle\hat{\mathbf{x}}_{j}$
$\displaystyle=\psi(\mathbf{x}_{1},\mathbf{x}_{J},\gamma_{j}),$ (6)
$\displaystyle\mathcal{L}_{ICL}$
$\displaystyle=\sum_{j=2}^{J-1}\lVert\mathbf{x}_{j}-\hat{\mathbf{x}}_{j}\rVert_{2},$

where $\gamma_{j}=\frac{j-1}{J-1}$, and $2\leq j\leq J-1$.

This simple yet effective loss enhances the continuity between frames thus
improving the spatiotemporal consistency in dynamic object generation by a
large margin. Moreover, we find the spatial version of this loss alleviates
the multi-face problem in 3D generation tasks as well. Please refer to the
experiment sections to see quantitative and qualitative results. The
Interpolation-driven Consistency Loss and some other regularization losses are
added with SDS loss in Equation 2, details of which can be found in the
experiment section.

###  4.3 Cross-frame Video Enhancer

Sometimes image sequence rendered from the optimized DyNeRF suffers from
artifacts, such as blurry edges, small floaters, and insufficient smoothness,
especially when the object motion is abrupt or complex. To further improve the
quality of rendered videos, we design a lightweight video enhancer and
optimize it via GAN, following pix2pix (Isola et al., 2017). The real images
are obtained with image-to-image technique (Meng et al., 2021) using a super-
resolution diffusion model, and the fake images are the rendered ones.

To better exploit video information, We add cross-frame attention to the UNet
architecture in pix2pix, i.e., each frame will query information from two
adjacent frames. We believe this could enable better consistency and image
quality. Denote the feature map before and after cross-frame-attention as
$\mathbf{F}$ and $\mathbf{F}_{j}^{\prime}$, we have:

$\displaystyle F_{j}^{\prime}$
$\displaystyle=\mathrm{Attention}(\mathcal{Q}_{j},\mathcal{K}_{j},\mathcal{V}_{j}),$
(7) $\displaystyle\mathcal{Q}_{j}=\mathrm{flatten}(F_{j}),$
$\displaystyle\mathcal{K}_{j}=\mathcal{V}_{j}=\mathrm{flatten}(\mathrm{concat}(F_{j-1},F_{j+1}),$

where $\mathcal{Q}$, $\mathcal{K}$ and $\mathcal{V}$ denotes query, key, and
value in attention mechanism, and $\mathrm{concat}$ denotes the concatenation
along the width dimension.

Loss for the generator and discriminator are the same as pix2pix.

##  5 Experiment

We have conducted extensive experiments to evaluate the proposed Consistent4D
generator using both synthetic data and in-the-wild data. The experimental
setup, comparison with dynamic NeRF baselines, and ablations are provided in
the following sections.

###  5.1 Implementation Details

Data Preparation For each input video, we initially segment the foreground
object utilizing SAM (Kirillov et al., 2023) and subsequently sample 32 frames
uniformly. The majority of the input videos span approximately 2 seconds, with
some variations extending to around 1 second or exceeding 5 seconds. For the
ablation study of video sampling, please refer to the appendix A.3.

Training During SDS and interpolation consistency optimization, we utilize
zero123-xl trained by Deitke et al. (2023) as the diffusion model for SDS
loss. For Cascade DyNeRF, we set $s=2$, i.e., we have coarse-level and fine-
level DyNeRFs.The spatial and temporal resolution of Cascade DyNeRF are
configured to 50 and 8 for coarse-level, and 100 and 16 for fine-level,
respectively. We first train DyNeRF with batch size 4 and resolution 64 for
5000 iterations. Then we decrease the batch size to 1 and increase the
resolution to 256 for the next 5000 iteration training. ICL is employed in the
initial 5000 iterations with a probability of 25%, and we sample consecutive
temporal frames at intervals of one frame and sample consecutive spatial
frames at angular intervals of 5∘-15∘ in azimuth. SDS loss weight is set as
0.01 and reconstruction loss weight is set as 500. In addition to SDS and ICL,
we also apply foreground mask loss, normal orientation loss, and 3D smoothness
loss. The learning rate is set as 0.1 and the optimizer is Adam. In the post-
video enhancing stage, we train the video enhancer with a modified Unet
architecture. The learning rate is set as 0.002, the batch size is 16, and the
training epoch is 200. The main optimization stage and the video enhancing
stage cost about 2.5 hours and 15 minutes on a single V100 GPU. For details,
please refer to the appendix A.4.

| Pistol | Guppie | Crocodile | Monster | Skull | Trump | Aurorus | Average
---|---|---|---|---|---|---|---|---
| LPIPS | CLIP | LPIPS | CLIP | LPIPS | CLIP | LPIPS | CLIP | LPIPS | CLIP | LPIPS | CLIP | LPIPS | CLIP | LPIPS | CLIP
D-NeRF | 0.52 | 0.66 | 0.32 | 0.76 | 0.54 | 0.61 | 0.52 | 0.79 | 0.53 | 0.72 | 0.55 | 0.60 | 0.56 | 0.66 | 0.51 | 0.68
K-planes | 0.40 | 0.74 | 0.29 | 0.75 | 0.19 | 0.75 | 0.47 | 0.73 | 0.41 | 0.72 | 051 | 0.66 | 0.37 | 0.67 | 0.38 | 0.72
ours | 0.10 | 0.90 | 0.12 | 0.90 | 0.12 | 0.82 | 0.18 | 0.90 | 0.17 | 0.88 | 0.23 | 0.85 | 0.17 | 0.85 | 0.16 | 0.87

Table 1: Video-to-4D quantitative comparison. Figure 3: Comparison with
dynamic NeRF methods. We render each dynamic object from a novel view at two
timestamps.

###  5.2 Comparisons with Other Methods

To date, few methods have been developed for 4D generation utilizing video
obtained from a static camera, so we only manage to compare our method with
D-NeRF (Pumarola et al., 2021) and K-planes (Fridovich-Keil et al., 2023).

Quantitative Results To quantitatively evaluate the proposed video-4D
generation method, we select and download seven animated models, namely
Pistol, Guppie, Crocodie, Monster, Skull, Trump, Aurorus, from Sketchfab (ske,
2023) and render the multi-view videos by ourselves, as shown in Figure 3 and
appendix A.2. We render one input view for scene generation and 4 testing
views for our evaluation. The per-frame LPIPS (Zhang et al., 2018) score and
the CLIP (Radford et al., 2021) similarity are computed between testing and
rendered videos. We report the scores averaged over the four testing views in
Table. 1. Note that the commonly used PSNR and SSIM scores were not applied in
our scenario as the pixel- and patch-wise similarities are too sensitive to
the reconstruction difference, which does not align with the generation
quality usually perceived by humans. As shown in Table 1, our dynamic 3D
generation produces the best quantitative results over the other two methods
on both the LPIPS and CLIP scores, which well aligns with the qualitative
comparisons shown in Figure 3.

Qualitative Results The outcomes of our method and those of dyNeRFs are
illustrated in Figure 3. It is observable that both D-NeRF and HyperNeRF
methods struggle to achieve satisfactory results in novel views, owing to the
absence of multi-view information in the training data. In contrast,
leveraging the strengths of the generation model, our method proficiently
generates a 360∘ representation of the dynamic object. For additional results,
please refer to the appendix A.1.

###  5.3 Ablations

We perform ablation studies for every component within our framework. For
clarity, the video enhancer is excluded when conducting ablations for SDS and
interpolation consistency optimization.

Figure 4: Ablation of Cascade DyNeRF.

(a) Video-to-4D. For each dynamic object, we render it from a novel view for
two timestamps with textureless rendering for each timestamp. For clarity, we
describe the input videos as follows (from left to right): blue jay pecking,
T-rex roaring, corgi smiling, seagull turning around.

(b) Text-to-3D. For each 3D model, we render it from two views with a
textureless rendering for each view and remove the background to focus on the
actual 3D shape.

Figure 5: Ablation of Interpolation-driven Consistency Loss.

| w/o ICL | w/ ICL
---|---|---
preference rate(%) | 24.5 | 75.5
(a) Video-to-4D.

| w/o ICL | w/ ICL
---|---|---
success rate(%) | 19.3 | 28.6
(b) Text-to-3D.

(c) Details of Text-to-3D user study. Data is sorted.

Table 2: User study of Interpolation-driven Consistency Loss.

Cascade DyNeRF In Figure 4, we conduct an ablation study for Cascade DyNeRF.
Specifically, we substitute Cascade DyNeRF with the original K-planes
architecture, maintaining all other settings unchanged. In the absence of the
cascade architecture, the training proves to be unstable, occasionally
yielding incomplete or blurry objects, as demonstrated by the first and second
objects in Figure 4. In some cases, while the model manages to generate a
complete object, the moving parts of the object lack clarity, exemplified by
the leg and beak of the bird. Conversely, the proposed Cascade DyNeRF exhibits
stable training, leading to relatively satisfactory generation results.

Interpolation-driven Consistency Loss The introduction of Interpolation-driven
Consistency Loss (ICL) stands as a significant contribution of our work.
Therefore, we conduct extensive experiments to investigate both its advantages
and potential limitations. Figure 5(a) illustrates the ablation of both
spatial and temporal Interpolation-driven Consistency Loss (ICL) in the video-
to-4D task. Without ICL, the objects generated exhibit spatial and temporal
inconsistency, as evidenced by the multi-face/foot issue in the blue jay and
T-rex, and the unnatural pose of the seagull. Additionally, color
discrepancies, such as the black backside of the corgi, are also noticeable.
Employing either spatial or temporal ICL mitigates the multi-face issue, and
notably, the use of spatial ICL also alleviates the color defect problem.
Utilizing both spatial and temporal ICL concurrently yields superior results.
We further perform a user study, depicted in Figure 2(a), which includes
results w/ and w/o ICL for 20 objects. For efficiency in evaluation, cases in
which both methods fail are filtered out in this study. 20 users participate
in this evaluation, and the results unveiled a preference for results w/ ICL
in 75% of the cases.

We further explore whether ICL could alleviate multi-face problems for text-
to-3D tasks. We compared the success rate of DreamFusion implemented w/ and
w/o the proposed ICL loss. For the sake of fairness and rigor, we collect all
prompts related to animals from the official DreamFusion project page,
totaling 230. 20 users are asked to participate in this non-cherry-pick user
study, where we establish three criteria for a successful generation:
alignment with the text prompt, absence of multi-face issues, and clarity in
geometry and texture. We visualize the statistics in Table 2(b) and Table 2.
The results show that although users have different understandings of
successful generation, results w/ ICL always outperform results w/o it. For a
comprehensive understanding, qualitative comparisons are presented in Figure
5(b), which indicates the proposed technique effectively alleviates the multi-
face Janus problem and thus promotes the success rate. Implementation details
about text-to-3D can be found in the appendix A.4, and we also analyze the
failure cases and limitations of ICL in A.5.

Cross-frame Video Enhancer In Figure 11 (see in appendix), we show the
proposed cross-frame video enhancer could improve uneven color distribution
and smooth out the rough edges, as shown in almost all figures, and remove
some floaters, as indicated by the cat in the red and green box.

##  6 Conclusion

We introduce a novel video-to-4D framework, named Consistent4D, aimed at
generating 360∘ 4D objects from uncalibrated monocular videos captured by a
stationary camera. Specifically, we first optimize a Cascade DyNeRF which is
specially designed to facilitate stable training under the discrete
supervisory signals from an image-to-image diffusion model. More crucially, we
introduce an Interpolation-driven Consistency Loss to enhance spatial and
temporal consistency, the main challenge in this task. For comprehensiveness,
we train a lightweight video enhancer to rectify scattered color discrepancies
and eliminate minor floating artifacts, as a post-processing step. Extensive
experiments conducted on both synthetic and in-the-wild data demonstrate the
effectiveness of our method.

## References

  * dee (2023) Deepfloyd.  https://huggingface.co/DeepFloyd, 2023.
  * ske (2023) Sketchfab.  https://sketchfab.com/3d-models, 2023.
  * Blattmann et al. (2023) Andreas Blattmann, Robin Rombach, Huan Ling, Tim Dockhorn, Seung Wook Kim, Sanja Fidler, and Karsten Kreis.  Align your latents: High-resolution video synthesis with latent diffusion models.  In _CVPR_ , 2023.
  * Cai & Vasconcelos (2018) Zhaowei Cai and Nuno Vasconcelos.  Cascade r-cnn: Delving into high quality object detection.  In _CVPR_ , 2018.
  * Cao & Johnson (2023) Ang Cao and Justin Johnson.  Hexplane: A fast representation for dynamic scenes.  In _arXiv preprint_ , 2023.
  * Carion et al. (2020) Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko.  End-to-end object detection with transformers.  In _ECCV_ , 2020.
  * Chan et al. (2021) Eric R Chan, Marco Monteiro, Petr Kellnhofer, Jiajun Wu, and Gordon Wetzstein.  pi-gan: Periodic implicit generative adversarial networks for 3d-aware image synthesis.  In _CVPR_ , 2021.
  * Chen et al. (2023) Rui Chen, Yongwei Chen, Ningxin Jiao, and Kui Jia.  Fantasia3d: Disentangling geometry and appearance for high-quality text-to-3d content creation.  In _arXiv preprint_ , 2023.
  * Deitke et al. (2023) Matt Deitke, Ruoshi Liu, Matthew Wallingford, Huong Ngo, Oscar Michel, Aditya Kusupati, Alan Fan, Christian Laforte, Vikram Voleti, Samir Yitzhak Gadre, et al.  Objaverse-xl: A universe of 10m+ 3d objects.  In _arXiv preprint_ , 2023.
  * Deng et al. (2022) Congyue Deng, Chiyu Jiang, Charles R Qi, Xinchen Yan, Yin Zhou, Leonidas Guibas, Dragomir Anguelov, et al.  Nerdi: Single-view nerf synthesis with language-guided diffusion as general image priors.  In _arXiv preprint_ , 2022.
  * Fridovich-Keil et al. (2023) Sara Fridovich-Keil, Giacomo Meanti, Frederik Warburg, Benjamin Recht, and Angjoo Kanazawa.  K-planes: Explicit radiance fields in space, time, and appearance.  In _arXiv preprint_ , 2023.
  * Gao et al. (2021) Chen Gao, Ayush Saraf, Johannes Kopf, and Jia-Bin Huang.  Dynamic view synthesis from dynamic monocular video.  In _ICCV_ , 2021.
  * Gao et al. (2022a) Hang Gao, Ruilong Li, Shubham Tulsiani, Bryan Russell, and Angjoo Kanazawa.  Monocular dynamic view synthesis: A reality check.  In _NeurIPS_ , 2022a.
  * Gao et al. (2022b) Jun Gao, Tianchang Shen, Zian Wang, Wenzheng Chen, Kangxue Yin, Daiqing Li, Or Litany, Zan Gojcic, and Sanja Fidler.  Get3d: A generative model of high quality 3d textured shapes learned from images.  In _NeurIPS_ , 2022b.
  * Guo et al. (2023) Yuan-Chen Guo, Ying-Tian Liu, Ruizhi Shao, Christian Laforte, Vikram Voleti, Guan Luo, Chia-Hao Chen, Zi-Xin Zou, Chen Wang, Yan-Pei Cao, and Song-Hai Zhang.  threestudio: A unified framework for 3d content generation.  https://github.com/threestudio-project/threestudio, 2023.
  * Ho et al. (2022) Jonathan Ho, William Chan, Chitwan Saharia, Jay Whang, Ruiqi Gao, Alexey Gritsenko, Diederik P Kingma, Ben Poole, Mohammad Norouzi, David J Fleet, et al.  Imagen video: High definition video generation with diffusion models.  In _arXiv preprint_ , 2022.
  * Huang et al. (2022) Zhewei Huang, Tianyuan Zhang, Wen Heng, Boxin Shi, and Shuchang Zhou.  Real-time intermediate flow estimation for video frame interpolation.  In _ECCV_ , 2022.
  * Isola et al. (2017) Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, and Alexei A Efros.  Image-to-image translation with conditional adversarial networks.  In _CVPR_ , 2017.
  * Kirillov et al. (2023) Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan-Yen Lo, et al.  Segment anything.  In _arXiv preprint_ , 2023.
  * Kocabas et al. (2020) Muhammed Kocabas, Nikos Athanasiou, and Michael J Black.  Vibe: Video inference for human body pose and shape estimation.  In _CVPR_ , 2020.
  * Li et al. (2022a) Ruilong Li, Julian Tanke, Minh Vo, Michael Zollhöfer, Jürgen Gall, Angjoo Kanazawa, and Christoph Lassner.  Tava: Template-free animatable volumetric actors.  In _ECCV_ , 2022a.
  * Li et al. (2022b) Tianye Li, Mira Slavcheva, Michael Zollhoefer, Simon Green, Christoph Lassner, Changil Kim, Tanner Schmidt, Steven Lovegrove, Michael Goesele, Richard Newcombe, et al.  Neural 3d video synthesis from multi-view video.  In _CVPR_ , 2022b.
  * Li et al. (2021) Zhengqi Li, Simon Niklaus, Noah Snavely, and Oliver Wang.  Neural scene flow fields for space-time view synthesis of dynamic scenes.  In _CVPR_ , 2021.
  * Lin et al. (2023) Chen-Hsuan Lin, Jun Gao, Luming Tang, Towaki Takikawa, Xiaohui Zeng, Xun Huang, Karsten Kreis, Sanja Fidler, Ming-Yu Liu, and Tsung-Yi Lin.  Magic3d: High-resolution text-to-3d content creation.  In _CVPR_ , 2023.
  * Liu et al. (2023) Ruoshi Liu, Rundi Wu, Basile Van Hoorick, Pavel Tokmakov, Sergey Zakharov, and Carl Vondrick.  Zero-1-to-3: Zero-shot one image to 3d object.  In _arXiv preprint_ , 2023.
  * Loper et al. (2023) Matthew Loper, Naureen Mahmood, Javier Romero, Gerard Pons-Moll, and Michael J Black.  Smpl: A skinned multi-person linear model.  In _Seminal Graphics Papers: Pushing the Boundaries, Volume 2_ , 2023.
  * Melas-Kyriazi et al. (2023) Luke Melas-Kyriazi, Christian Rupprecht, Iro Laina, and Andrea Vedaldi.  Realfusion: 360 $\\{$$\backslash$deg$\\}$ reconstruction of any object from a single image.  In _arXiv preprint_ , 2023.
  * Meng et al. (2021) Chenlin Meng, Yutong He, Yang Song, Jiaming Song, Jiajun Wu, Jun-Yan Zhu, and Stefano Ermon.  Sdedit: Guided image synthesis and editing with stochastic differential equations.  In _ICLR_ , 2021.
  * Park et al. (2021a) Keunhong Park, Utkarsh Sinha, Jonathan T Barron, Sofien Bouaziz, Dan B Goldman, Steven M Seitz, and Ricardo Martin-Brualla.  Nerfies: Deformable neural radiance fields.  In _CVPR_ , 2021a.
  * Park et al. (2021b) Keunhong Park, Utkarsh Sinha, Peter Hedman, Jonathan T Barron, Sofien Bouaziz, Dan B Goldman, Ricardo Martin-Brualla, and Steven M Seitz.  Hypernerf: A higher-dimensional representation for topologically varying neural radiance fields.  In _arXiv preprint_ , 2021b.
  * Poole et al. (2023) Ben Poole, Ajay Jain, Jonathan T Barron, and Ben Mildenhall.  Dreamfusion: Text-to-3d using 2d diffusion.  In _ICLR_ , 2023.
  * Pumarola et al. (2021) Albert Pumarola, Enric Corona, Gerard Pons-Moll, and Francesc Moreno-Noguer.  D-nerf: Neural radiance fields for dynamic scenes.  In _CVPR_ , 2021.
  * Radford et al. (2021) Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al.  Learning transferable visual models from natural language supervision.  In _ICML_ , 2021.
  * Ren et al. (2021) Zhongzheng Ren, Xiaoming Zhao, and Alex Schwing.  Class-agnostic reconstruction of dynamic objects from videos.  In _NeurIPS_ , 2021.
  * Rombach et al. (2022) Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer.  High-resolution image synthesis with latent diffusion models.  In _CVPR_ , 2022.
  * Schwarz et al. (2020) Katja Schwarz, Yiyi Liao, Michael Niemeyer, and Andreas Geiger.  Graf: Generative radiance fields for 3d-aware image synthesis.  In _NeurIPS_ , 2020.
  * Seo et al. (2023) Junyoung Seo, Wooseok Jang, Min-Seop Kwak, Jaehoon Ko, Hyeonsu Kim, Junho Kim, Jin-Hwa Kim, Jiyoung Lee, and Seungryong Kim.  Let 2d diffusion model know 3d-consistency for robust text-to-3d generation.  In _arXiv preprint_ , 2023.
  * Shao et al. (2023) Ruizhi Shao, Zerong Zheng, Hanzhang Tu, Boning Liu, Hongwen Zhang, and Yebin Liu.  Tensor4d: Efficient neural 4d decomposition for high-fidelity dynamic reconstruction and rendering.  In _CVPR_ , 2023.
  * Singer et al. (2023) Uriel Singer, Shelly Sheynin, Adam Polyak, Oron Ashual, Iurii Makarov, Filippos Kokkinos, Naman Goyal, Andrea Vedaldi, Devi Parikh, Justin Johnson, et al.  Text-to-4d dynamic scene generation.  In _arXiv preprint_ , 2023.
  * Tang et al. (2023) Junshu Tang, Tengfei Wang, Bo Zhang, Ting Zhang, Ran Yi, Lizhuang Ma, and Dong Chen.  Make-it-3d: High-fidelity 3d creation from a single image with diffusion prior.  In _arXiv preprint_ , 2023.
  * Tretschk et al. (2021) Edgar Tretschk, Ayush Tewari, Vladislav Golyanik, Michael Zollhöfer, Christoph Lassner, and Christian Theobalt.  Non-rigid neural radiance fields: Reconstruction and novel view synthesis of a dynamic scene from monocular video.  In _ICCV_ , 2021.
  * Verbin et al. (2022) Dor Verbin, Peter Hedman, Ben Mildenhall, Todd Zickler, Jonathan T Barron, and Pratul P Srinivasan.  Ref-nerf: Structured view-dependent appearance for neural radiance fields.  In _CVPR_ , 2022.
  * Vo et al. (2020) Minh Vo, Yaser Sheikh, and Srinivasa G Narasimhan.  Spatiotemporal bundle adjustment for dynamic 3d human reconstruction in the wild.  In _IEEE TPAMI_ , 2020.
  * Wang et al. (2022) Haochen Wang, Xiaodan Du, Jiahao Li, Raymond A Yeh, and Greg Shakhnarovich.  Score jacobian chaining: Lifting pretrained 2d diffusion models for 3d generation.  In _arXiv preprint_ , 2022.
  * Wang et al. (2023) Zhengyi Wang, Cheng Lu, Yikai Wang, Fan Bao, Chongxuan Li, Hang Su, and Jun Zhu.  Prolificdreamer: High-fidelity and diverse text-to-3d generation with variational score distillation.  In _arXiv preprint_ , 2023.
  * Wu et al. (2021) Shangzhe Wu, Tomas Jakab, Christian Rupprecht, and Andrea Vedaldi.  Dove: Learning deformable 3d objects by watching videos.  In _arXiv preprint_ , 2021.
  * Wu et al. (2022a) Shangzhe Wu, Ruining Li, Tomas Jakab, Christian Rupprecht, and Andrea Vedaldi.  Magicpony: Learning articulated 3d animals in the wild.  In _arXiv preprint_ , 2022a.
  * Wu et al. (2022b) Tianhao Wu, Fangcheng Zhong, Andrea Tagliasacchi, Forrester Cole, and Cengiz Oztireli.  D $\mathaccent 1799{e}$ 2 nerf: Self-supervised decoupling of dynamic and static objects from a monocular video.  In _arXiv preprint_ , 2022b.
  * Xian et al. (2021) Wenqi Xian, Jia-Bin Huang, Johannes Kopf, and Changil Kim.  Space-time neural irradiance fields for free-viewpoint video.  In _CVPR_ , 2021.
  * Yang et al. (2022) Gengshan Yang, Minh Vo, Natalia Neverova, Deva Ramanan, Andrea Vedaldi, and Hanbyul Joo.  Banmo: Building animatable 3d neural models from many casual videos.  In _CVPR_ , 2022.
  * Zhang et al. (2018) Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang.  The unreasonable effectiveness of deep features as a perceptual metric.  In _cvpr_ , 2018.
  * Zhou et al. (2022) Daquan Zhou, Weimin Wang, Hanshu Yan, Weiwei Lv, Yizhe Zhu, and Jiashi Feng.  Magicvideo: Efficient video generation with latent diffusion models.  In _arXiv preprint_ , 2022.
  * Zuffi et al. (2017) Silvia Zuffi, Angjoo Kanazawa, David W Jacobs, and Michael J Black.  3d menagerie: Modeling the 3d shape and pose of animals.  In _CVPR_ , 2017.
  * Zuffi et al. (2018) Silvia Zuffi, Angjoo Kanazawa, and Michael J. Black.  Lions and tigers and bears: Capturing non-rigid, 3d, articulated shape from images.  In _CVPR_ , 2018.

##  Appendix A Appendix

###  A.1 Additional Visualization Results

Figure 6: Visualization results of our method. All four input videos are in-
the-wild videos. The novel views presented are 22.5∘ and 112.5∘ away from the
input view, respectively. The results of our methods include RGB, normal map
and depth map (from left to right).

In Figure 6, we present the result of our method on four in-the-wild videos.
For clarity, we describe the input videos as follows: robot dancing, squirrel
feeding, toy-spiderman dancing, toy-rabbit deforming. Due to limited space,
the reviewers are strongly recommended to watch the video in the attached
files to see various visualization results.

###  A.2 Data Used in Video-to-4D Quantitative Evaluation

Figure 7: Data and comparison results for video-to-4D quantitative evaluation.

Sin three dynamic objects are shown in Figure 7, we only visualize the rest
four here, as shown in Figure 7. The observation is similar to the results in
the main paper.

We demonstrate the effectiveness of video enhancer in Figure 11. The analysis
can be found in the main paper.

###  A.3 The Number of Frames

Figure 8: Ablation of video frame sampling. Videos w/ sampling contain 32
frames. Videos w/o sampling contain 72 and 39 frames for Aurorus and minions,
respectively. The results of our methods include RGB and normal map (from left
to right).

For simplicity, we sample each input video to 32 frames in all experiments.
However, we find input videos without sampling sometimes give slightly better
results, as shown in Figure 8.

###  A.4 Implementation Details

Loss Function in Video-to-4D Besides SDS loss $\mathcal{L}_{SDS}$,
Interpolation-driven consistency loss $\mathcal{L}_{ICL}$, we also apply
reconstruction loss $\mathcal{L}_{rec}$ and mask loss $\mathcal{L}_{m}$ for
the input view. 3D normal smooth loss $\mathcal{L}_{n}$ and orientation loss
$\mathcal{L}_{ori}$ (Verbin et al., 2022) are also applied. Therefore, the
final loss is calculated as
$\mathcal{L}=\lambda_{1}\mathcal{L}_{SDS}+\lambda_{2}\mathcal{L}_{ICL}+\lambda_{3}\mathcal{L}_{rec}+\lambda_{4}\mathcal{L}_{m}+\lambda_{5}\mathcal{L}_{n}+\lambda_{6}\mathcal{L}_{ori}$,
where $\lambda_{1}=0.1$, $\lambda_{2}=2500$, $\lambda_{3}=500$,
$\lambda_{4}=50$, $\lambda_{5}=2.0$, and $\lambda_{6}$ is initially 1 and
increased to 20 linearly until 5000 iterations. Note that the reconstruction
loss and SDS loss are applied alternatively.

Video Enhancer For video enhancer architecture, we follow pix2pix (Isola et
al., 2017) except for that we modify the unet256 architecture to a light-
weighted version, with only three up/down layers and one cross-frame attention
layer. The feature dimensions for the unet layers are set as 64, 128, and 256.
Besides, we inject a cross-attention layer in the inner layer of the unet to
enable the current frame to query information from adjacent frames. For real
images, we use DeepFloyd-IF stage II (dee, 2023), which is a diffusion model
for super-resolution tasks. The input image, i.e., the rendered image, is
resized to $64\times 64$ and the output resolution is $256\times 256$. The
prompt needed by the diffusion model is manually set, i.e., we use the ”a $*$”
as the prompt, in which $*$ is the category of the dynamic object. For
example, the prompts for dynamic objects in Figure 11 are a bird, a cat, a
minions. The prompt cloud also be obtained from image or video caption models,
or large language models.

Text-to-3D Details We choose Threestudio built by (Guo et al., 2023) as the
codebase since it is the best public implementation we could find. DeepFloy-IF
(dee, 2023) is employed as the diffusion model, and all default tricks in
Threestudio are utilized. The hyper-parameters for results w/ and w/o ICL,
such as batch size and learning rate, are kept consistent between the
implementations w/ and w/o ICL, except for those related to ICL. We train the
model for 5000 iterations, the first 1000 iterations with batch size 8 and
resolution 64, and the rest 4000 with batch size 2 and resolution 256. The
learning rate is 0.01 and the optimizer is Adam, the same as the default
setting in Threestudio. The ICL loss is applied in the first 1000 iterations
with probability 30% and weight 2000.

###  A.5 Failure Cases

Figure 9: Video-to-4D failure cases.

Figure 10: Text-to-3D failure cases.

Video-to-4D Since the video-to-4D task in this paper is very challenging, our
method actually has many failure cases. For example, we fail to generate the
dynamic object when the motion is complex or abrupt, as shown in Figure 10. In
Figure 10, the dog’s tail disappears in the second image because the tail is
occluded in the input image when $t=t_{2}$. The frog, which is jumping up and
down fast, gets blurry when $t=t_{1}$.

Text-to-3D When applying ICL in text-to-3D, we find some multi-face cases that
could not be alleviated, and we show them in Figure 10.

Figure 11: Ablation of Video enhancer. Please zoom in to view the details.