# Polynomial entropy of induced maps of circle and interval homeomorphisms

Maša -Dorić and Jelena Katić  Matematički institut SANU, Knez Mihailova 36,
11000 Belgrade, Serbia (Maša -Dorić<EMAIL_ADDRESS>Matematički
fakultet, Studentski trg 16, 11000 Belgrade, Serbia (Jelena Katić)
<EMAIL_ADDRESS>
###### Abstract.

We compute the polynomial entropy of the induced maps on hyperspace for a
homeomorphism $f$ of an interval or a circle with finitely many non-wandering
points.

2020 Mathematical subject classification: Primary 37B40, Secondary 54F16,
37A35
Keywords: Polynomial entropy, hyperspaces, interval homeomorphisms, circle
homeomorphisms, induced maps

##  1\. Introduction

Every continuous map on a compact metric space $X$ induces a continuous map
(called the induced map) on the hyperspace $2^{X}$ of all closed subsets. If
$X$ is connected, i.e. continuum, we consider the hyperspace $C(X)$ of
subcontinua of $X$, (which is also a continuum). One can also consider the
hyperspace $X^{*k}$ of all nonempty subsets with at most $k$ points (for
$k\in\mathbb{N}$). A natural question is to find some relations between the
given (individual) dynamics on $X$ and the induced one (collective dynamics)
on the hyperspace. Various results in this direction were obtained in the last
decades. Without attempting to give complete references, we mention just a
few: Borsuk and Ulam [7], Bauer and Sigmund [5], Román-Flores [21], Banks [4],
Acosta, Illanes and Méndez-Lango [1].

The topological entropy of the induced map was studied by Kwietniak and
Oprocha in [12], Lampart and Raith [17], Hernández and Méndez [10], Arbieto
and Bohorquez [2] and others.

In [17] the authors showed that, if $f$ is an interval or a circle
homeomorphism, the topological entropy of the induced map on the hyperspace of
subcontinua is zero. This is also obtained as a corollary in [2], for Morse-
Smale diffeomorphisms of a circle.

One of the measures of complexity of a system with zero topological entropy is
the polynomial entropy. The notion of the polynomial entropy was first
introduced by Marco in [18] and [19] in the context of Hamiltonian integrable
systems. It was further investigated in different contexts by Labrousse [13,
14, 15], Labrousse and Marco [16], Bernard and Labousse [6], Artigue,
Carrasco–Olivera and Monteverde [3], Haseaux and Le Roux [9], Roth, Roth and
Snoha [22], Correa and de Paula [8] etc.

As opposed to the topological entropy, which depends only on the dynamics
restricted to the non-wandering set, the wandering set is visible to the
polynomial entropy. In the case when non-wandering set is finite (for example
Morse-Smale systems), there is a technique for computing the polynomial
entropy developed in [9] by Hauseux and Le Roux. Originally, they invented a
simple coding procedure for homeomorphisms with only one non-wandering point,
where the polynomial entropy is particularly well adapted, since the growth of
the number of wandering orbits is always at least linear and at most
polynomial. Hauseux and Le Roux also proved that the polynomial entropy
localizes near a certain finite set (singular set), in order to compute the
polynomial entropy of Brouwer homeomorphisms. This method was slightly
generalized in [11], to the case of a map which is only continuous and the
non-wandering set is finite. The coding procedure was also used in [8] for the
computation of the polynomial entropy of Morse-Smale systems on surfaces.

In this paper we compute the polynomial entropy of the induced maps $C(f)$,
$f^{*k}$ and $2^{f}$ for a homeomorphism $f$ of a circle or an interval with a
finite non-wandering set. The polynomial entropy of such a homeomorphism of an
interval is known to be $1$, and of a circle is ether $0$ or $1$ (see [15]).
The hyperspace of subcontinua of an one-dimensional space is quite simple and
can be identified with a two-dimensional object. Our computation uses the
coding method and reduction to the singular sets.

Denote by $h_{pol}(\cdot)$ the polynomial entropy of a map and by $C(f)$,
$f^{*k}$ and $2^{f}$ the induced maps on $C(X)$, $X^{*k}$ and $2^{X}$,
respectively. These are the statements of our results.

######  Theorem A.

Let $f:[0,1]\to[0,1]$ be a homeomorphism with a finite non-wandering set. Then
$h_{pol}(C(f))=2$, $h_{pol}(f^{*k})=k$ and $h_{pol}(2^{f})=\infty$.∎

######  Theorem B.

Let $f:[0,1]\to[0,1]$ be a homeomorphism with a finite non-wandering set. Then
$h_{pol}(C(f))=2$, $h_{pol}(f^{*k})=k$ and $h_{pol}(2^{f})=\infty$.∎

##  2\. Preliminaries

###  2.1. Hyperspaces and induced maps

For a compact metric space $(X,d)$, the hyperspace $2^{X}$ is the set of all
nonempty closed subsets of $X$. The topology on $2^{X}$ is induced by the
Hausdroff metric

$d_{H}(A,B):=\inf\\{\varepsilon\mid A\subset U(B,\varepsilon),\;B\subset
U(A,\varepsilon)\\},$

where

$U(A,\varepsilon):=\\{x\in X\mid d(x,A)<\varepsilon\\}.$

The space $2^{X}$ is compact and the topology induced by $d_{H}$ is the same
as the Vietoris topology.

We will also consider two closed subspaces of $2^{X}$, with the induced
metric. The first one is $X^{*k}$, the space of all finite subsets of
cardinality at most $k$, with the same topology. The set $X^{*k}$ is called
$k$-fold symmetric product of $X$.

If $X$ is also connected (i.e. a continuum), then the set $C(X)$ of all
connected and closed nonempty subsets of $X$ is also a continuum. The set
$C(X)$ is called the hyperspace of subcontinua of $X$.

If $f:X\to X$ is continuous, then it induces continuous maps

$\displaystyle 2^{f}:2^{X}\to 2^{X},\quad$ $\displaystyle
2^{f}(A):=\\{f(x)\mid x\in A\\}$ $\displaystyle C(f):C(X)\to C(X),\quad$
$\displaystyle C(f)(A):=\\{f(x)\mid x\in A\\}$ $\displaystyle f^{*k}:X^{*k}\to
X^{*k},\quad$ $\displaystyle f^{*k}(A):=\\{f(x)\mid x\in A\\}.$

If $f$ is a homeomorphism, so are $2^{X}$, $C(f)$ and $f^{*k}$.

###  2.2. Polynomial entropy and coding

Suppose that $X$ is a compact metric space, and $f:X\rightarrow X$ is
continuos. Denote by $d_{n}^{f}(x,y)$ the dynamic metric (induced by $f$ and
$d$):

$d_{n}^{f}(x,y)=\max\limits_{0\leq k\leq n-1}d(f^{k}(x),f^{k}(y)).$

Fix $Y\subseteq X$. For $\varepsilon>0$, we say that a finite set $E\subset X$
is $(n,\varepsilon)$-separated if for every $x,y\in E$ it holds
$d_{n}^{f}(x,y)\geq\varepsilon$. Let $S(n,\varepsilon;Y)$ denotes the maximal
cardinality of an $(n,\varepsilon)$-separated set $E$, contained in $Y$.

The polynomial entropy of the map $f$ on the set $Y$ is defined by

$h_{pol}(f;Y)=\lim\limits_{\varepsilon\rightarrow
0}\limsup\limits_{n\rightarrow\infty}\frac{\log S(n,\varepsilon;Y)}{\log n}.$

If $X=Y$ we abbreviate $h_{pol}(f):=h_{pol}(f;X)$. The polynomial entropy, as
well as the topological entropy, can also be defined via coverings with sets
of $d^{n}_{f}$-diameters less than $\varepsilon$, or via coverings by balls of
$d_{n}^{f}$-radius less than $\varepsilon$, see [18]. We list some properties
of the polynomial entropy that are important for our computations:

  * •

$h_{pol}(f^{k})=h_{pol}(f)$, for $k\geq 1$

  * •

if $Y\subset X$ is a closed, $f$-invariant set, then
$h_{pol}(f;Y)=h_{pol}(f|_{Y})$

  * •

if $Y=\bigcup_{j=1}^{m}Y_{j}$ where $Y_{j}$ are $f$-invariant, then
$h_{pol}(f;Y)=\max\\{h_{pol}(f;Y_{j})\mid j=1,\ldots,m\\}$

  * •

If $f:X\to X$, $g:Y\to Y$ and $f\times g:X\times Y\to X\times Y$ is defined as
$f\times g(x,y):=(f(x),g(y))$, then $h_{pol}(f\times g)=h_{pol}(f)+h_{pol}(g)$

  * •

$h_{pol}(f)$ does not depend on a metric but only on the induced topology

  * •

$h_{pol}(\cdot)$ is a conjugacy invariant (meaning if $f:X\to X$,
$g:X^{\prime}\to X^{\prime}$, $\varphi:X\to X^{\prime}$ is a homeomorphism of
compact spaces with and $g\circ\varphi=\varphi\circ f$, then
$h_{pol}(f)=h_{pol}(g)$).

A point $p\in X$ is wandering if there exists a neighbourhood $U\ni p$ such
that $f^{n}(U)\cap U=\emptyset$, for $n\geq 1$.

A point that is not wandering is said to be non-wandering. We denote the set
of all non-wandering points by $NW(f)$. The set $NW(f)$ is closed and
$f$-invariant.

We now give a brief description of the computation of the polynomial entropy
for maps with a finite non-wandering set, by means of a coding and a local
polynomial entropy. This construction was first done in [9] for homeomorphisms
with only one non-wandering (hence fixed) point, and then modified in [11] for
continuous maps with finitely many non-wandering points. Let $Y$ be any
$f$-invariant subset of $X$.

We first define a coding relative to a family of sets $\mathcal{F}$. Let

$\mathcal{F}=\\{Y_{1},Y_{2},\ldots,Y_{L}\\}$

where $Y_{j}\subseteq X\setminus\mathrm{NW}(f)$ and

$Y_{\infty}:=Y\setminus\bigcup_{j=1}^{L}Y_{j}.$

Let $\underline{x}=(x_{0},\ldots,x_{n-1})$ be a finite sequence of elements in
$Y$. We say that a finite sequence $\underline{w}=(w_{0},\ldots,w_{n-1})$ of
elements in $\mathcal{F}\cup\\{Y_{\infty}\\}$ is a coding of $\underline{x}$
relative to $\mathcal{F}$ if $x_{j}\in w_{j}$. We will refer to
$\underline{w}$ as a word and to $w_{j}$ as a letter.

Let $\mathcal{A}_{n}(\mathcal{F};Y)$ be the set of all codings of all orbits

$(x,f(x),\ldots,f^{n-1}(x))$

of length $n$ relative to $\mathcal{F}$, for all $x\in Y$. If
$\sharp\mathcal{A}_{n}(\mathcal{F};Y)$ denotes the cardinality of
$\mathcal{A}_{n}(\mathcal{F};Y)$, we define the polynomial entropy of $f$, on
the set $Y$, relative to the family $\mathcal{F}$ as the number:

$h_{pol}(f,\mathcal{F};Y):=\limsup_{n\to\infty}\frac{\log\sharp\mathcal{A}_{n}(\mathcal{F};Y)}{\log
n}.$

We abbreviate $h_{pol}(\mathcal{F};Y):=h_{pol}(f,\mathcal{F};Y)$ whenever
there is no risk of confusion.

For $Z\subseteq X\setminus\operatorname{NW}(f)$, set

$M(Z):=\sup_{x\in X}\sharp\\{n\mid f^{n}(x)\in Z\\}.$

If $Z\subseteq X\setminus\operatorname{NW}(f)$ is compact, the number $M(Z)$
is finite, since $Z$ can be covered with a finite number of open wandering
sets, and every orbit can intersect a wandering set at most once.

We will use the following property of $h_{pol}(\mathcal{F};Y)$ in order to
localize our compuutation to a singular set.

######  Proposition 1.

[Proposition 3.2 in [11]] Let $\mathcal{F}$ and $\mathcal{F}^{\prime}$ be two
families of subsets of $X\setminus\mathrm{NW}(f)$ with
$M(\cup\mathcal{F})<+\infty$. Let $Y\subset X$ be an $f$-invariant subset with
exactly one non-wandering point. If for every
$Y_{j}^{\prime}\in\mathcal{F}^{\prime}$ there exists $Y_{j}\in\mathcal{F}$
such that $Y^{\prime}_{j}\subseteq Y_{j}$, then
$h_{pol}(\mathcal{F}^{\prime};Y)\leq h_{pol}(\mathcal{F};Y).$∎

Next we define the local polynomial entropy for a finite set

$\mathcal{S}=\left\\{x_{1},x_{2},\dots,x_{l}\right\\}\subset X\setminus
NW(f).$

We choose a decreasing sequence of neighbourhoods
$\\{U_{j,n}\\}_{n\in\mathbb{N}}$ of $x_{j}\in\mathcal{S}$ which form a basis
of neighbourhoods of $x_{j}$. It follows from Proposition 1 that the sequence

$\left\\{h_{pol}\left(\\{U_{1,n},U_{2,n},\dots,U_{l,n}\\};Y\right)\right\\}_{n\geq
1}$

is decreasing and converges, as well as that its limit does not depend on the
choice of neighbourhoods. Define:

$h_{pol}^{loc}(\mathcal{S};Y):=\lim\limits_{n\rightarrow\infty}h_{pol}\left(\\{U_{1,n},U_{2,n},\dots,U_{l,n}\\};Y\right).$

Finally, we relate the polynomial entropy to a singular set.

We say that the subsets $U_{1},\ldots,U_{L}$ of $X\setminus\mathrm{NW}(f)$ are
mutually singular if for every $M>0$, there exist $x$ and positive integers
$n_{1},\ldots,n_{L}$ such that

$f^{n_{j}}(x)\in U_{j},\quad|n_{i}-n_{j}|>M,\quad\text{for}\;i\neq j.$

The points $x_{1},\ldots,x_{L}$ are mutually singular if every family of
respective neighbourhoods $U_{1},\ldots,U_{L}$, $U_{j}\ni x_{j}$ is mutually
singular. We say that a finite set is singular if it consists of mutually
singular points.

######  Proposition 2.

[Propositions 3.2 and 3.3 in [11]] Let $Y\subseteq X$ be an $f$-invariant set
containing exactly one non-wandering point. Then it holds:

  * (a)

$h_{pol}(f;Y)=\sup\\{h_{pol}(\\{K\\};Y)\mid{K\subseteq
X\setminus\operatorname{NW}(f),\,K\,\text{compact}}\\}$

  * (b)

$h_{pol}(f;Y)=\sup\\{h_{pol}(\\{K\cap Y\\};Y)\mid{K\subseteq
X\setminus\operatorname{NW}(f),\,K\,\text{compact}}\\}$

  * (c)

$h_{pol}(f;Y)=\sup\big{\\{}h_{pol}^{loc}(\mathcal{S};Y)\mid\mathcal{S}\subset\overline{Y},\;\mathcal{S}\,\text{singular}\big{\\}}$.∎

The following corollary is of importance for our result, so we will prove it.

######  Corollary 3.

The polynomial entropy $h_{pol}(f;Y)$ is bounded from above by the maximal
cardinality of a singular set contained in $\overline{Y}$.

Proof. The local polynomial entropy of a finite set
$\\{x_{1},\ldots,x_{l}\\}\subset{X}\setminus\operatorname{NW}(f)$ is bounded
from above by its cardinality. Indeed, one can choose wandering neighbourhoods
$Y_{j}\ni x_{j}$, such that every letter $Y_{j}$ appears in the coding of any
orbit $(x,f(x),\ldots,f^{n-1}(x))$ at most once. Therefore there are at most
$n(n-1)\cdots(n-l+1)\leq n^{l}$ possible codings. We conclude

$h_{pol}^{loc}(\\{x_{1},\ldots,x_{l}\\};Y)\leq
h_{pol}(\\{Y_{1},\ldots,Y_{l}\\};Y)=\limsup_{n\to\infty}\frac{\log\;\sharp\;\mathcal{A}_{n}(\\{Y_{1},\ldots,Y_{l}\\};Y)}{\log
n}\leq l.$

∎

In particular, if the maximal cardinality of a singular set is two, we have a
more precise statement.

######  Proposition 4.

Let $Y$ be as above. Suppose that the maximal cardinality of a singular set
contained in $\overline{Y}$ equals 2. If for every singular set
$\mathcal{S}=\\{x_{1},x_{2}\\}\subset Y$ and every open $Y_{j}\ni x_{j}$,
$Y_{j}\subset X\setminus NW(f)$, there exists a positive integer $L$ such that
for all $n\geq L$ it holds $f^{n}(Y_{1})\cap Y_{2}\neq\emptyset$, then
$h_{pol}(f;Y)=2$.

Proof. It follows from Corollary 3 that $h_{pol}(f;Y)\leq 2$. To prove the
other inequality, note that the assumed condition on $Y_{1}$ and $Y_{2}$
implies that for all positive integer $m$ and for all $n\geq L$, there exists
$x$ with $f^{m}(x)\in Y_{1}$, $f^{m+n}(x)\in Y_{2}$. Therefore, for every
$n\geq L$ there exists a coding of a form

$(\underbrace{Y_{\infty},\ldots,Y_{\infty}}_{m},Y_{1},\underbrace{Y_{\infty},\ldots,Y_{\infty}}_{n},Y_{2},Y_{\infty},\ldots,Y_{\infty})$

where $m$ is any number and $n\geq L$. We get

$\sharp\;\mathcal{A}_{n}(\\{Y_{1},Y_{2}\\};Y)\geq\sum_{m=0}^{n-L-1}(n-L-m)\sim\frac{1}{2}n^{2},$

so $h_{pol}(\\{Y_{1},Y_{2}\\};Y)\geq 2$. Since this holds for all $Y_{j}$ with
$x_{j}\in Y_{j}\subset X\setminus NW(f)$, the same is true for
$h_{pol}(\\{x_{1},x_{2}\\};Y)$.∎

We will use the following notations:

$\mathcal{O}(x):=\\{f^{n}(x)\mid n\in\mathbb{Z}\\}$

for an orbit of a point $x$ and

$W^{s}(p):=\\{x\in X\mid f^{n}(x)\to p,n\to\infty\\}$

for the stable manifold of a (fixed) point $p$.

##  3\. Proofs

In order to use the methods described in Subsection 2.2 we need to establish
that the sets $NW(C(f))$ and $NW\left(f^{*k}\right)$ are finite. Since
$h_{pol}(f^{m})=h_{pol}(f)$, we can assume that $NW(f)=Fix(f)$.

######  Proposition 5.

Let $f:[0,1]\to[0,1]$ or $f:S^{1}\to S^{1}$ be a homeomorphism such that the
set $NW(f)=Fix(f)$ is finite. Then $NW(C(f))=Fix(C(f))$ and
$NW\left(f^{*k}\right)=Fix\left(f^{*k}\right)$ are also finite for every
$k\geq 1$.

Proof. Let us prove the proposition for $C(f)$, $f:I\to I$. The other case can
be proved analogously. The set $Fix(C(f))$ is finite as it consists of all
intervals $[a,b]\subseteq I$, where $a,b\in Fix(f)$. Note that the condition
$NW(f)=Fix(f)$ implies that $f$ is increasing. Suppose that there exists
$[x,y]\in NW(C(f))\setminus Fix(C(f))$ and that $x\notin Fix(f)$. Since
$NW(f)=Fix(f)$, there exist $a,b\in Fix(f)$ (possibly equal) with

$f^{n}(x)\to a,\quad f^{n}(y)\to b,\quad n\to\infty.$

Let $\varepsilon<d(x,a)/2$. For $n\geq n_{0}$ it holds
$d(f^{n}(x),a)<\varepsilon$, $d(f^{n}(y),b)<\varepsilon$ so we have

$\displaystyle d_{H}(C(f)^{n}([x,y]),[x,y])$ $\displaystyle\geq
d_{H}([a,b],[x,y])-d_{H}\left(C(f)^{n}([x,y]),[a,b]\right)$ $\displaystyle\geq
d(a,x)-\varepsilon>\varepsilon.$

∎

###  3.1. Proof of Theorem A

Let $I:=[0,1]$. We will first compute the polynomial entropy of $C(f)$. We can
identify the space $C(I)$ with the set

$\\{(x,y)\in[0,1]^{2}\mid 0\leq x\leq y\leq 1\\},$

which is the upper triangle $A$ in the square $[0,1]\times[0,1]$. The
homeomorphism $\varphi:C(I)\to A$ is given by $\varphi:[x,y]\to(x,y)$. The map
$C(f)$ is conjugated to $f\times f|_{A}$ via $\varphi$ (where $f\times
f:[0,1]\times[0,1]\to[0,1]\times[0,1]$, $f(x,y)=(f(x),f(y))$).

Denote the lower triangle in $[0,1]\times[0,1]$ by $B$. Since both $A$ and $B$
are closed and $f\times f$-invariant (if not, they are $(f\times
f)^{2}$-invariant), we have

$h_{pol}(f\times f)=\max\\{h_{pol}(f\times f|_{A}),h_{pol}(f\times
f|_{B})\\}.$

On the other hand

$h_{pol}(f\times f)=2h_{pol}(f).$

Since $f\times f|_{A}$ and $f\times f|_{B}$ are topologically equivalent
systems (the map $(x,y)\mapsto(y,x)$ realizes a conjugacy), we have

$h_{pol}(C(f))=2h_{pol}(f).$

It is not hard to see that $h_{pol}(C(f))=2$, because $h_{pol}(f)=1$. Indeed,
suppose that $f$ is increasing (if not, $f^{2}$ is). Let
$0=p_{0}<p_{1}<\ldots<p_{k}=1$ denote the fixed points. We apply again
$h_{pol}(f)=\max_{j}h_{pol}\left(f|_{[p_{j-1},p_{j}]}\right)$. Since
$f|_{[p_{j-1},p_{j}]}$ possesses a non-wandering point, it holds
$h_{pol}(f|_{[p_{j-1},p_{j}]})\geq 1$ (see, for example [15]). It is easy to
see that $f|_{[p_{j-1},p_{j}]}$ satisfies all the conditions in Proposition 2,
for $Y=(p_{j-1},p_{j}]$, therefore also in Corollary 3, as well as that $f$
has only one singular point in $(p_{j-1},p_{j}]$. So we obtain

$h_{pol}\left(f|_{[p_{j-1},p_{j}]}\right)=\max\left\\{h_{pol}\left(f|_{[p_{j-1},p_{j}]};(p_{j-1},p_{j}]\right),h_{pol}\left(f|_{\\{p_{j-1}\\}}\right)\right\\}\leq
1.$

(For a different, more explicit proof of $h_{pol}(f|_{[p_{j-1},p_{j}]})=1$ see
[15].)

Let us prove that $h_{pol}\left(f^{*k}\right)=k$. Since $f^{\times
k}:=f\times\ldots\times f$ and $f^{*k}$ are semi-conjugated via:

$\pi:I^{k}\to
I^{*k},\quad\pi:(x_{1},\ldots,x_{k})\mapsto\\{x_{1},\ldots,x_{k}\\},$

we have

(1) $h_{pol}\left(f^{*k}\right)\leq h_{pol}\left(f^{\times k}\right)=k.$

We want to prove the other inequality. For a permutation $\sigma$ of
$\\{1,\ldots,k\\}$, define

$A_{\sigma}:=\\{(x_{1},\ldots,x_{k})\in I^{k}\mid x_{\sigma(1)}\leq\ldots\leq
x_{\sigma(k)}\\}.$

As before, we see that $I^{k}=\bigcup_{\sigma}A_{\sigma}$, $A_{\sigma}$’s are
$f^{\times k}$-invariant, and all $f^{\times k}|_{A_{\sigma}}$ are mutually
conjugated, so

$h_{pol}\left(f^{\times k}\right)=\max\left\\{h_{pol}\left(f^{\times
k}|_{A_{\sigma}}\right)\right\\}.$

Therefore

$h_{pol}\left(f^{\times k}|_{A_{\sigma}}\right)=k$

for every $\sigma$.

Define:

$\displaystyle A(k)$ $\displaystyle:=\\{(x_{1},\ldots,x_{k})\in I^{k}\mid
x_{1}\leq\ldots\leq x_{k}\\}$ $\displaystyle\hat{A}(k,I)$
$\displaystyle:=\left\\{\\{x_{1},\ldots,x_{k}\\}\in I^{*k}\mid
x_{1}<\ldots<x_{k}\right\\}.$

Whenever there is no risk of confusion we will abbreviate
$\hat{A}(k)=\hat{A}(k,I)$. If $k=2$ we are done, since $\pi|_{A(2)}:A(2)\to
I^{*2}$ is a homeomorphism of compact sets, so
$h_{pol}\left(f^{*2}\right)=h_{pol}\left(f^{\times 2}|_{A(2)}\right)=2$.

For $k>2$ we need the following auxilirary fact.

######  Lemma 6.

Suppose that $Fix(f)=\\{0,1\\}$. Then
$h_{pol}\left(f^{*k};\hat{A}(k)\right)=k$.

The rest of the proof follows easily from Lemma 6. Indeed, if
$Fix(f)=\\{0,1\\}$ then we have:

$h_{pol}\left(f^{*k}\right)\geq
h_{pol}\left(f^{*k};{\hat{A}(k)})\right){=}k\;\stackrel{{\scriptstyle(\ref{eq:<=k})}}{{\Rightarrow}}\;h_{pol}\left(f^{*k}\right)=k.$

If $Fix(f)=\\{0,p_{1},\ldots,p_{k-1},1\\}$ with $0<p_{1}<\ldots<p_{k-1}<1$,
define

$B:=\left\\{\\{x_{1},\ldots,x_{j}\\}\mid
j\in\\{1,\ldots,k\\},\;x_{i}\in[0,p_{1}]\right\\}\subset I^{*k}$

and apply Lemma 6 to $f|_{[0,p_{1}]}$. Since $B$ is $f^{*k}$-invariant, we
conclude

$h_{pol}\left(f^{*k}\right)\geq
h_{pol}\left(f^{*k}|_{B}\right)=h_{pol}\left(\left(f|_{[0,p_{1}]}\right)^{*k}\right)\geqslant
h_{pol}\left(\left(f|_{[0,p_{1}]}\right)^{*k};\hat{A}(k,[0,p_{1}])\right)=k.$

From here and (1) we obtain $h_{pol}\left(f^{*k}\right)=k$.∎

Proof of Lemma 6. Consider the following (non-disjoint) partition of $A(k)$:

  * •

$\tilde{A}(k):=\\{(x_{1},\ldots,x_{k})\in I^{k}\mid x_{1}<\ldots<x_{k}\\}$

  * •

$A_{j}(k):=\\{(x_{1},\ldots,x_{k})\in I^{k}\mid x_{1}\leq\ldots\leq
x_{j}=x_{j+1}\leq\ldots\leq x_{k}\\}$, for $j=1,\ldots,k-1$.

It is obvious that $\tilde{A}(k)$ and $A_{j}(k)$ are $f^{\times k}$-invariant.
Therefore

$h_{pol}\left(f^{\times k}\right)=\max\left\\{h_{pol}\left(f^{\times
k};\tilde{A}(k)\right),h_{pol}\left(f^{\times k}|_{A_{j}(k)}\right)\right\\}.$

Notice that $f^{\times k}|_{A_{j}(k)}$ is conjugated to
$f^{\times(k-1)}|_{A(k-1)}$, so

$h_{pol}\left(f^{\times
k}|_{A_{j}(k)}\right)=h_{pol}\left(f^{\times(k-1)}\right)=k-1.$

Since $h_{pol}\left(f^{\times k}\right)=k$ we conclude that
$h_{pol}\left(f^{\times k};\tilde{A}(k)\right)=k$.

Notice that

$\pi|_{\tilde{A}(k)}:\tilde{A}(k)\to\hat{A}(k)$

is a homeomorphism that establishes a conjugacy between $f^{\times
k}|_{\tilde{A}(k)}$ and $f^{*k}|_{\hat{A}(k)}$. Although polynomial entropy is
a conjugacy invariant only when the domain is compact (which the sets
$\hat{A}(k)$ and $\tilde{A}(k)$ are not), we can indirectily prove that

$h_{pol}\left(f^{*k};{\hat{A}(k)})\right)=h_{pol}\left(f^{\times
k};\tilde{A}(k)\right)=k.$

We wish to apply the coding method from Proposition 2. Note that for $k>2$ the
sets $\tilde{A}(k)$ and $\hat{A}(k)$ do not contain any non-wandering points.
We can add the point $(0,\ldots,0)$ to the set $\tilde{A}(k)$ and $\\{0\\}$ to
$\hat{A}(k)$, keeping the same notations: this will not change the entropy and
$\pi$ will still be a homeomorphism between the two. In this way we achive
that the assumptions from Proposition 2 are fulfilled.

Note that $\pi$ induces a bijection between the sets

$\\{K\cap\tilde{A}(k)\mid K\subset I^{k}\setminus NW\left(f^{\times
k}\right),\;K\;\text{compact}\\}$

and

$\\{L\cap\hat{A}(k)\mid L\subset I^{*k}\setminus
NW\left(f^{*k}\right),\;L\;\text{compact}\\}.$

If $\underline{w}=(w_{0},\ldots,w_{n-1})$ is a coding of an orbit
$\left(x,f^{\times k}(x).\ldots,(f^{\times k})^{n-1}(x)\right)$ in
$\tilde{A}(k)$ (consisting of letters $K\cap\tilde{A}(k)$ and
$Y_{\infty}:=\tilde{A}(k)\setminus K$), then
$(\pi(w_{0}),\ldots,\pi(w_{n-1}))$ is the coding of an orbit
$\left(\pi(x),f^{*k}(\pi(x)),\ldots,(f^{*k})^{n-1}(\pi(x))\right)$ in
$\hat{A}(k)$ (consisting of letters $\pi(K\cap\tilde{A}(k))$ and
$Y_{\infty}:=\hat{A}(k)\setminus\pi(K\cap\tilde{A}(k))$), and vice versa.
Therefore, for a fixed compact $K$, the sets
$\mathcal{A}_{n}\left(\\{K\cap\tilde{A}(k)\\};\tilde{A}(k)\right)$ and
$\mathcal{A}_{n}\left(\\{\pi(K\cap\tilde{A}(k))\\};\hat{A}(k)\right)$ have the
same cardinality. Applying $(b)$ from Proposition 2 we finish the proof.

Finally, to prove that $h_{pol}\left(2^{f}\right)=\infty$, we notice that
$X^{*k}$ is a closed and $2^{f}$-invariant subset of $2^{X}$ and moreover
$f^{*k}=2^{f}|_{X^{*k}}$ so

$h_{pol}\left(2^{f}\right)\geq h_{pol}\left(f^{\times k}\right)=k$

for every $k\in\mathbb{N}$.∎

###  3.2. Proof of Theorem B

Let us first compute the polynomial entropy of $C(f)$. For that reason, we
will distinguish between the following possibilities:

  * (1)

the set $Fix(f)$ consists of only one point

  * (2)

the set $Fix(f)$ has at least three different points

  * (3)

the set $Fix(f)$ has exactly two points.

Case (1). Since $f$ has only one fixed point, $f$ preserves the orientation of
the circle. If $Fix(f)=\\{a\\}$, the continuum map $C(f)$ has only two non-
wandering points - $\\{a\\}$ and $S^{1}$. We will divide the set $C(S^{1})$
into two closed invariant subsets:

  * •

$A$ is the set of all $[x,y]$ such that the point $x$ is between points $a$
and $y$ counterclockwise, including degenerate cases when the two or all three
points are equal (meaning that $[x,a]$, $\\{x\\}$, $[a,y]$ and $S^{1}$ are in
$A$); notice that $[x,y]$ does not contain $a$ as an interior point, for
$x\neq y$

  * •

$B$ is the set of all $[x,y]$ such that the point $y$ is between points $a$
and $x$ counterclockwise, including degenerate cases when the two or all three
points are equal (meaning that $[x,a]$, $\\{a\\}$, $[a,y]$ and $S^{1}$ are in
$B$); notice that $[x,y]$ contains $a$

(see Figure 1). In this way we know that, for $[x,y],[z,w]\in A$ or
$[x,y],[z,w]\in B$ the following is true:

$d(x,z)<r\quad\mbox{and}\quad d(y,w)<r\;\Rightarrow\;d_{H}([x,y],[z,w])<r.$

(In general, this does not have to hold, since the arcs $[x,y]$ and $[z,w]$
may be ”at the oposite sides” of $S^{1}$.)

Figure 1. Elements of set $A$ (on the left) and $B$ (on the right)

Suppose that $f$ moves the points in $S^{1}$ in the positive direction, as in
Figure 2. The other case is treated in the same way.

We will first consider the map $C(f)|_{A}$. Its dynamics is depicted in Figure
2. We have the following four possibilities:

  * •

$x\neq a$ $\Rightarrow$ $C(f)^{n}([x,y])\stackrel{{\scriptstyle
n\to\pm\infty}}{{\longrightarrow}}\\{a\\}$

  * •

$[x,y]=\\{a\\}$ $\Rightarrow$ $C(f)^{n}([x,y])\stackrel{{\scriptstyle
n\to\pm\infty}}{{\longrightarrow}}\\{a\\}$

  * •

$x=a$, $y\neq a$ $\Rightarrow$ $C(f)^{n}([x,y])\stackrel{{\scriptstyle
n\to\infty}}{{\longrightarrow}}S^{1}$, $C(f)^{n}([x,y])\stackrel{{\scriptstyle
n\to-\infty}}{{\longrightarrow}}\\{a\\}$

  * •

$[x,y]=S^{1}$ $\Rightarrow$ $C(f)^{n}([x,y])\stackrel{{\scriptstyle
n\to\pm\infty}}{{\longrightarrow}}S^{1}$.

Figure 2. Direction of $f$ (on the left) and dynamics of $C(f)|_{A}$ (on the
right)

We can divide $A$ into the sets $\\{S^{1}\\}$ and $Y:=A\setminus\\{S^{1}\\}$
and compute

$h_{pol}(C(f)|_{A})=\max\left\\{h_{pol}(C(f)|_{A};Y),h_{pol}\left(C(f)|_{A};\\{S^{1}\\}\right)\right\\}=h_{pol}(C(f)|_{A};Y)$

since $h_{pol}(C(f)|_{A};\\{S^{1}\\})=h_{pol}(C(f)|_{\\{S^{1}\\}})=0$.

We claim that the arcs $[a,p]$, for $p\neq a$ and $[q,a]$ for $q\neq a$ are
two mutually singular points $x_{1}$ and $x_{2}$ satisfying the conditions
stated in Proposition 4.

Let us first prove that $[a,p]$ and $[q,a]$ are mutually singular. Fix an
$\varepsilon>0$ $\varepsilon<\min\\{d(a,p),d(a,q)\\}$ and $M>0$. Choose $y\in
B(p,\varepsilon)\subset S^{1}$ arbitrarly. Since $f^{n}(y)\to a$, when
$n\to\infty$, there exists a non-negative integer $n_{1}>M$ such that for
$n\geq n_{1}$ it holds $d(f^{n}(y),a)<\varepsilon$. Let $\alpha\in
B(q,\varepsilon)\subset S^{1}$ be any point. Since $f^{-n}(\alpha)\to a$, when
$n\to\infty$, there exsts $n_{2}\geq n_{1}$ with
$d(f^{-n_{2}}(\alpha),a)<\varepsilon$. We can increase $n_{2}$ if necessary to
obtain that the point $f^{-n_{2}}(\alpha)$ is between $a$ and $y$, and
$f^{-n_{2}}(\alpha)\neq y$. Choose $x:=f^{-n_{2}}(\alpha)$. Set $I:=[x,y]$. We
claim that the orbit of $I$ intersects the $\varepsilon$-balls around $[a,p]$
and $[q,a]$ in the times with difference greater than $M$. Indeed, we have:

$d_{H}(I,[a,p])\leq\max\\{d(x,a),d(y,p)\\}<\varepsilon,$

so $I\in B([a,p],\varepsilon)$ and

$d(f^{n_{2}}(I),[q,a])\leq\max\\{d(f^{n_{2}}(x),q),d(f^{n_{2}}(y),a\\}=\max\\{d(\alpha,q),d(f^{n_{2}}(y),a)\\}<\varepsilon$

therefore $f^{n_{2}}(I)\in B([q,a],\varepsilon)$.

The next step is to show that any two arcs except the ones of the form $[a,p]$
and $[q,a]$ cannot be mutually singular. Suppose that $[p_{1},q_{1}]$ and
$[p_{2},q_{2}]$ are two different arcs such that $p_{j}\neq a$, $q_{j}\neq a$.
If $[p_{1},q_{1}]$ and $[p_{2},q_{2}]$ are not on the same orbit of $C(f)$, it
is enough to show that there exist neighbourhoods $U\ni[p_{1},q_{1}]$ and
$V\ni[p_{2},q_{2}]$ such that

$C(f)^{n}(U)\cap V=\emptyset,\quad\mbox{for all}\;n\in\mathbb{Z}.$

Let $\varepsilon$ be the distance from $[p_{2},q_{2}]$ to the orbit
$\mathcal{O}([p_{1},q_{1}])$, since $C(f)^{n}([p_{1},q_{1}])$ converges to
$\\{a\\}$, when $n\to\pm\infty$, we have $\varepsilon>0$. The sets

$U:=B([p_{1},q_{1}]),\varepsilon/3),\quad V:=B([p_{2},q_{2}]),\varepsilon/3)$

have the desired properties, hence $[p_{1},q_{1}]$ and $[p_{2},q_{2}]$ are not
mutually singular.

If $[p_{1},q_{1}]$ and $[p_{2},q_{2}]$ are on the same orbit, it is enough to
show that there exist neighbourhoods $U\ni[p_{1},q_{1}]$ and
$V\ni[p_{2},q_{2}]$, $U,V\subset Y\setminus NW(C(f))$ and $M>0$, such that:

(2) $C(f)^{n}(U)\cap V\neq\emptyset\;\Rightarrow\;|n|<M,\quad\mbox{for
all}\;n\in\mathbb{Z}.$

Indeed, it follows from (2) that $U$ and $V$ are not mutually singular, so
neither are $[p_{1},q_{1}]$ and $[p_{2},q_{2}]$. So take $U\subset
W^{s}(\\{a\\})\cap Y$, $V$ and $U^{\prime}$ to be any three balls centered at
$[p_{1},q_{1}]$, $[p_{2},q_{2}]$ and $\\{a\\}$ respectively, such that $V\cap
U^{\prime}=\emptyset$. There exists a non-negative integer $n_{0}$ such that,
for all $n\in\mathbb{Z}$, $|n|\geq n_{0}$, $C(f)^{n}(U)\subset U^{\prime}$. We
see that for $|n|\geq n_{0}$ it holds $C(f)^{n}(U)\cap V=\emptyset$.

One can check in a similar way that for all other possibilities for
$[p_{1},q_{1}]$ and $[p_{2},q_{2}]$ (except $[a,p]$, for $p\neq a$ and $[q,a]$
for $q\neq a$), the arcs $[p_{1},q_{1}]$ and $[p_{2},q_{2}]$ can not be
mutually singular.

It remains to prove that for every two mutually singular points of the form
$[a,p]$ and $[q,a]$, where $p,q\neq a$ and every open $Y_{1}\ni[a,p]$,
$Y_{2}\ni[q,a]$, $Y_{j}\subset A\setminus NW(C(f))$, there exists a positive
integer $L$ such that for all $n\geq L$ it holds $C(f)^{n}(Y_{1})\cap
Y_{2}\neq\emptyset$. Then we are able to apply Proposition 4 and finish the
proof.

Fix $\varepsilon>0$ and set $Y_{1}:=B([a,p],\varepsilon)\subset A\setminus
NW(C(f))$ and $Y_{2}:=B([q,a],\varepsilon)\subset A\setminus NW(C(f))$.
Consider the line

$l:=\\{[x,p]\mid d(x,a)<\varepsilon\\}\subset Y_{1}.$

Notice that

$C(f)^{n}(l)=\\{[f^{n}(x),f^{n}(p)]\mid x\in B(a,\varepsilon)\\}.$

Since $f^{n}(p)\to a$ and $f^{n}(B(a,\varepsilon))\to S^{1}$, when
$n\to\infty$, there exists $n_{0}$ such that for all $n\geq n_{0}$ both
$d(f^{n}(p),a)<\varepsilon$ and $f^{n}(B(a,\varepsilon))\ni q$ hold. Denote by
$x_{1}\in B(a,\varepsilon)$ such that $f^{n}(x_{1})=q$. We conclude that
$d_{H}([f^{n}(p),f^{n}(x_{1})],[a,q])<\varepsilon$, therefore
$C(f)^{n}(Y_{1})\cap Y_{2}\neq\emptyset$.

The dynamics of $C(f)|_{B}$ is the following:

  * •

$y\neq a$ $\Rightarrow$ $C(f)^{n}([x,y])\stackrel{{\scriptstyle
n\to\pm\infty}}{{\longrightarrow}}S^{1}$

  * •

$[x,y]=\\{a\\}$ $\Rightarrow$ $C(f)^{n}([x,y])\stackrel{{\scriptstyle
n\to\pm\infty}}{{\longrightarrow}}\\{a\\}$

  * •

$y=a$, $x\neq a$ $\Rightarrow$ $C(f)^{n}([x,y])\stackrel{{\scriptstyle
n\to\infty}}{{\longrightarrow}}\\{a\\}$ and
$C(f)^{n}([x,y])\stackrel{{\scriptstyle n\to-\infty}}{{\longrightarrow}}S^{1}$

  * •

$[x,y]=S^{1}$ $\Rightarrow$ $C(f)^{n}([x,y])\stackrel{{\scriptstyle
n\to\pm\infty}}{{\longrightarrow}}S^{1}$.

The same reasoning applies to $C(f)|_{B}$, so the proof of Case (1) is done.

Case (2). Suppose $Fix(f)=\\{a_{1},\ldots,a_{m}\\}$ and there are no fixed
points between the points $a_{j}$ and $a_{j+1}$ . Denote by
$C_{j}:=[a_{j},a_{j+1}]$. It is obvious that the sets $C_{j}$ are
$f$-invariant, therefore the sets

$D_{ij}:=\\{[x,y]\mid x\in C_{i},y\in C_{j}\\}$

are $C(f)$-invariant. It is also easy to see that all $D_{ij}$ are closed as
well as $C(S^{1})=\bigcup_{i,j}D_{i,j}$. The proof of Theorem B is completed
if we prove that $h_{pol}(f|_{D_{ij}})=2$ for all $i,j$.

We see that $D_{i,j}$ can be identified with
$[a_{i},a_{i+1}]\times[a_{j},a_{j+1}]$ and since $f([x,y])=[f(x),f(y)]$,
$C(f)$ is conjugated to $f\times f$. Therefore we reduce the problem to the
computation of the polynomial entropy of

$C(f)=f\times
f:[a_{i},a_{i+1}]\times[a_{j},a_{j+1}]\to[a_{i},a_{i+1}]\times[a_{j},a_{j+1}],\quad
f\times f(x,y)=(f(x),f(y)).$

Since $h_{pol}(f\times g)=h_{pol}(f)+h_{pol}(g)$, we have
$h_{pol}(C(f)|_{D_{ij}})=h_{pol}(f|_{a_{i},a_{i+1}]})+h_{pol}(f|_{[a_{j},a_{j+1}]})$.
Similarly as in the proof of Theorem A, we have
$h_{pol}(f|_{[a_{i},a_{i+1}]})=h_{pol}(f|_{[a_{j},a_{j+1}]})=1$.

Case (3). Suppose $a:=a_{1}$ and $b:=a_{2}$ are the only two fixed points.
Then either $f$ maps both $[a,b]$ and $[b,a]$ to themselves, or one to
another. If the latter is the case, then $f^{2}$ maps both arcs to itself, so
we can assume that this is true, and apply the same argument as in (2).

Now we prove the statement for $f^{*k}$. Recall first that $h_{pol}(f)=1$.
This can be proved using the coding methods (it is easy to see that $f$
possesses no two mutually singular points), or, alternatively, by refering to
Theorem 2 in [15], which states that polynomial entropy of a circle
homeomorphism $f$ is $1$ if and only if $f$ is not conjugated to a rotation.

Define a relation $\leq$ on $S^{1}$ by identifying $S^{1}$ with $[0,1)$. We
can again assume that $f$ is orientation preserving, since if not, $f^{2}$ is
and

$h_{pol}\left(\left(f^{2})^{*k}\right)\right)=h_{pol}\left(\left(f^{*k}\right)^{2}\right)=h_{pol}\left(f^{*k}\right).$

So we can consider $f$ as an increasing homeomorphism of $[0,1]$ with $f(0)=0$
and $f(1)=1$ (we can assume that $0$ is a fixed point of $f$). As before, by
considering a semi-conjugacy
$\pi:\left(S^{1}\right)^{k}\to\left(S^{1}\right)^{*k}$, we derive
$h_{pol}\left(f^{*k}\right)\leq k$. The case $k=2$ is also covered in the same
way as for an interval.

If $f$ possesses at least two fixed points, $a$ and $b$, we can define $B$ as
a subset of $\left(S^{1}\right)^{*k}$ consisting of sets of points from the
interval $[a,b]$ and identify the map $f^{*k}|_{B}$ with
$\left(f|_{[a,b]}\right)^{*k}$. Therefore we have

$h_{pol}\left(f^{*k}\right)\geq
h_{pol}\left(f^{*k}|_{B}\right)=h_{pol}\left(\left(f|_{[a,b]}\right)^{*k}\right)=k$

(the last equality follows from the proof of Theorem A), so the proof is
finished.

If $f$ has only one fixed point, $0$, we can define the sets $\tilde{A}(k)$
and $\hat{A}(k)$ as in the proof of Theorem A, conclude that
$h_{pol}\left(f^{k};\tilde{A}(k)\right)=k$, and then prove that
$h_{pol}\left(f^{*k};\hat{A}(k)\right)=k$, as in the proof of Lemma 6.

The last statement, $h_{pol}\left(2^{f}\right)=\infty$, follows in the same
way as in the case of an interval. ∎

######  Remark 7.

If $(X,f)$ and $(Y,g)$ are two dynamical systems and there exists a semi-
conjugacy $\pi:X\to Y$ that is uniformly finite-to-one (meaning that there
exists $C$ such for any $y\in Y$ it holds
$\sharp\;\left(\pi^{-1}(y)\right)\leq C$), then the topological entropy
$h_{top}(f)$ and $h_{top}(g)$ coincide (see, for example [20]). It easily
follows from this that $h_{top}(f^{*k})=h_{top}(f^{\times k})$ (see [12]). An
analogous formula for the polynomial entropy is still not proved or disproved,
so we had to use the inequality relation, as a particularity of one-
dimensional sets.

## References

  * [1] G. Acosta, A. Illanes and H. Méndez-Lango, The transitivity of induced maps, Topology Appl. 156, 1013–1033, 2009.
  * [2] A. Arbieto, J. Bohorquez, Shadowing, topological entropy and recurrence of induced Morse-Smale diffeomorphisms, arXiv:2203.13356, 2022.
  * [3] A. Artigue, D. Carrasco–Olivera, I. Monteverde, Polynomial entropy and expansivity, Acta Math. Hungar. 152: 140–149, 2017.
  * [4] J. Banks, Chaos for induced hyperspace map, Chaos Solitons Fractals vol. 25, 681-685, 2005.
  * [5] W. Bauer and K. Sigmund, Topological dynamics of transformations induced on the space of probability measures, Monatsh. Math. vol. 79, no. 2, 81–92, 1975.
  * [6] P. Bernard and C. Labrousse, An entropic characterization on the flat metric on the two torus, Geom. Dedicata, 180, 187–201, 2016.
  * [7] K. Borsuk, S. Ulam S, On symmetric products of topological spaces, Bull Amer. Masth. Soc, 37, 875–882, 1931.
  * [8] J. Correa and H. de Paula, Polynomial entropy of Morse-Smale diffeomorphisms on surfaces, arXiv:2203.08336v2, 2022.
  * [9] L. Hauseux and F. Le Roux, Enropie polynomiale des homémorphsimes de Brouwer, Annales Henri Lebesgue 2, 39–57, 2019.
  * [10] P. Hernández, H. Méndez, Entropy of induced dendrite homeomorphisms, Topology proc., vol. 47, 191–205, 2016.
  * [11] J. Katić, M. Perić, On the polynomial entropy for Morse gradient systems, Mathematica Slovaca 69, No. 3, 611–624, 2019.
  * [12] D. Kwietniak and P. Oprocha, Topological entropy and chaos for maps induced on hyperspaces, Chaos Solitons Fractals vol. 33, no. 1, 76-86, 2007.
  * [13] C. Labrousse, Polynomial growth of the volume of balls for zero-entropy geodesic systems, Nonlinearity 25, 3049–3069, 2012.
  * [14] C. Labrousse, Flat Metrics are Strict Local Minimizers for the Polynomial Entropy, Regul. Chaotic Dyn. 17, 47–491, 2012.
  * [15] C. Labrousse, Polynomial entropy for the circle homeomorphisms and for $C^{1}$ nonvanishing vector fields on $\mathbb{T}^{2}$, arXiv:1311.0213, 2013.
  * [16] C. Labrousse and J. P. Marco, Polynomial Entropies for Bott Integrable Hamiltonian Systems, Regul. Chaotic Dyn. 19, 374–414, 2014.
  * [17] M. Lampart and P. Raith, Topological entropy for set valued maps, Nonlinear Anal. vol. 73, Issue 6, 1533-1537, 2010.
  * [18] J. P. Marco, Dynamical complexity and symplectic integrability, arXiv:0907.5363v1, 2009.
  * [19] J. P. Marco, Polynomial entropies and integrable Hamiltonian systems, Regul. Chaotic Dyn. 18(6), 623–655, 2013.
  * [20] C. Robinson, Dynamical systems, Studies in advanced mathematics. Boca Raton, FL: CRC Press; 1995.
  * [21] H. Román-Flores, A note on transitivity in set-valued discrete systems, Chaos Solitons Fractals, vol. 17, Issue 1, 99-104, 2003.
  * [22] S. Roth, Z. Roth and L’. Snoha, Rigidity and flexibility od polynomial enntropy, arXiv:2107.13695v1, 2021.


# Controlling Space and Time with Diffusion Models

Daniel Watson* Saurabh Saxena* Lala Li* Andrea Tagliasacchi David J. Fleet
Google DeepMind

###### Abstract

We present 4DiM, a cascaded diffusion model for 4D novel view synthesis (NVS),
conditioned on one or more images of a general scene, and a set of camera
poses and timestamps. To overcome challenges due to limited availability of 4D
training data, we advocate joint training on 3D (with camera pose), 4D
(pose+time) and video (time but no pose) data and propose a new architecture
that enables the same. We further advocate the calibration of SfM posed data
using monocular metric depth estimators for metric scale camera control. For
model evaluation, we introduce new metrics to enrich and overcome shortcomings
of current evaluation schemes, demonstrating state-of-the-art results in both
fidelity and pose control compared to existing diffusion models for 3D NVS,
while at the same time adding the ability to handle temporal dynamics. 4DiM is
also used for improved panorama stitching, pose-conditioned video to video
translation, and several other tasks.

Input |  $360\degree$ rotation
---|---
|
|
Input | Circular pan
|
|

Input 1 | Input 2 | Extrapolate time forward (waves approaching)
---|---|---
|  |
|  | Extrapolate time backward (waves receding)
|  |

Figure 1:  Zero-shot samples from 4DiM on LLFF (Mildenhall et al., 2019) and
an internal video dataset given one or two input images. We show samples on
different camera and time trajectories. 4DiM is a 32-frame model, so here we
evenly sub-sample outputs. Samples are best viewed as video in our website:
https://4d-diffusion.github.io

##  1 Introduction

Novel view synthesis (NVS) and 3D generative models have emerged as a new
frontier in generative models, enabling the synthesis of novel views of 3D
objects and scenes with control over camera pose. These models compliment
text-to-image/video models, with potential uses in the generation of 3D
assets, augmented reality, synthetic data for model training (e.g., in
robotics), and view interpolation. Notable recent examples using diffusion
models include 3DiM (Watson et al., 2022) and Zero-1-to-3 (Liu et al., 2023b).
They do not do generate explicit 3D scenes, rather, they leverage implicit 3D
knowledge for conditional view generation, as a form of image+pose conditioned
image generation.

To date, these models have focused mainly on objects with blank backgrounds,
rather than generic natural scenes, and on limited camera poses, like those
over the surface of the view-sphere fixated on objects centered at the origin.
While recent work (Wang et al., 2023a; Yu et al., 2023a) has attempted to
train models for generic scenes and 3D viewpoints, zero-shot performance on
out of distribution scenes remains fraught. The biggest challenge is that 3D
scene data is scarce. Much of the existing data also relies on COLMAP
(Schönberger and Frahm, 2016; Schönberger et al., 2016) for the estimation of
camera pose, yielding noisy poses with unknown scale. This is additionally
problematic for effective control in the single-image to 3D task because
models must sample from the distribution of plausible scales at inference
time.

The goal of this work is to extend NVS diffusion models in three respects: 1)
from objects to scenes; 2) to free-form camera poses specified in meaningful
physical units; and 3) to simultaneously allow spatial and temporal control
via camera pose and timestamp conditioning. To this end, we introduce 4DiM, a
diffusion model for NVS conditioned on (one or more) images of arbitrary
scenes, camera pose, and time. 4DiM is trained on a mixture of data sources,
including posed 3D images/video and unposed video, of both indoor and outdoor
scenes. We make this possible through various technical innovations that
enable training from missing data (e.g., images without pose or time
annotations) and sampling with separate guidance weights on different
conditioning images, poses or times. We also introduce calibrated versions of
video datasets posed via COLMAP. Calibrated data makes it easier to learn
metric regularities in the world, like the typical sizes of everyday objects
and spatial relations, and it enables one to specify camera poses in
meaningful physical units. As such, 4DiM generates 3D consistent, multi-view
images or video of dynamic scenes. Beyond sample generation 4DiM can be used
to unlock myriad applications, like video-to-video translation, improved
panoramic stitching, or training explicit 3D models with score-distillation
sampling (Poole et al., 2022).

To summarize, our main contributions include:

  * •

4DiM, a pixel-based diffusion model for novel view synthesis conditioned on
one or more images of arbitrary scenes, camera pose, and time. 4DiM comprises
a base model that generates 32 images at $64\\!\times\\!64$, and a multi-view
super resolution model that up-samples to $32\\!\times\\!256\\!\times\\!256$;

  * •

An effective data mixture for 4D models comprising posed and unposed video of
indoor and outdoor scenes, enabling zero-shot application with fine-grained
pose control and dynamics;

  * •

Novel architectural elements to facilitate time and pose conditioning as well
as training with incomplete data, and multi-guidance to simultaneously guide
on camera pose and timestamps;

  * •

A Calibrated version of RealEstate10K to improve model fidelity and enable
metric pose control;

  * •

Extensive evaluation and comparisons to prior work, including novel SfM-based
metrics, namely, SfM distances and keypoint distance to better quantify pose
alignment and dynamics.

##  2 Related work

We next provide a brief overview of prior work on camera control and 3D NVS
with diffusion models. We also briefly discuss some recent work on 3D
extraction using geometry-aware methods, which has emerged as an area of key
interest to the research community. Here, the use of geometry-free pose-
conditional diffusion models (like 4DiM) has been the key building block.

Unlike the typical setting of Neural Radiance Fields (Mildenhall et al., 2021)
(NeRF) where tens-to-hundreds of images are used as input for 3D
reconstruction, pose-conditional diffusion models for NVS aim to extrapolate
plausible, diverse, 3D consistent samples with as few as a single image input.
Conditioning diffusion models on an image and relative camera pose was
introduced by Watson et al. (2022) and Liu et al. (2023b) as an effective
alternative to prior few-view NVS methods, overcoming severe blur and floater
artifacts (Sitzmann et al., 2019; Yu et al., 2021; Niemeyer et al., 2022;
Sajjadi et al., 2022). However, they rely on neural architectures unsuitable
for jointly modeling more than two views, and consequently require heuristics
such as stochastic conditioning or Markovian sampling with a limited context
window (Yu et al., 2023a), for which it is hard to maintain 3D consistency.

Subsequent work proposed attention mechanisms leveraging epipolar geometry to
improve the 3D consistency of image-to-image diffusion models for NVS (Tseng
et al., 2023), and more recently, finetuning text-to-video models with
temporal attention layers or attention layers limited to overlapping regions
(He et al., 2024; Wang et al., 2023a) to model the joint distribution of
generated views and their camera extrinsics. These models however still suffer
from several issues: they have difficulty with static scenes due to persistent
dynamics from the underlying video models; they still suffer from 3D
inconsistencies and low fidelity; and they exhibit poor generalization to out-
of-distribution image inputs. Alternatives have been proposed to improve
fidelity in multi-view diffusion models of 3D scenes, albeit sacrificing the
ability to model free-form camera pose; e.g., see MVDiffusion and follow-ups
(Tang et al., 2023, 2024) where the data distribution is limited in its pose
trajectories.

A concurrent trajectory of closely related work on 3D extraction has recently
emerged following DreamFusion (Poole et al., 2022), where instead of directly
training diffusion models for NVS, new techniques for sampling views
parametrized as a NeRF with volume rendering are proposed such as Score
Distillation Sampling (SDS) and Variational Score Distillation (Wang et al.,
2024) (VSD). Here, a pre-existing diffusion model acts as a prior that drives
the generation process. This enables, for example, text-to-3D using a text-to-
image diffusion model. As demonstrated by MVDream (Shi et al., 2023),
ReconFusion (Wu et al., 2023) and CAT3D (Gao* et al., 2024), using a pose-
conditional diffusion model offers the unique advantage that the diffusion
model can produce samples at the exact viewpoint specified for volume
rendering during score distillation or NeRF postprocessing, which results in
dramatically improved sample quality. All of these works establish further
interest on improving pose-conditional diffusion models, which is the focus of
our work, thereby enabling improvements in 3D extraction methods that build
upon such models.

##  3 4D novel view synthesis models from limited data

4DiM uses a continuous-time diffusion model to learn the joint distribution
over multiple views:

$p(\bm{x}_{C+1:N}\,|\,\bm{x}_{1:C},\>\bm{p}_{1:N},\>t_{1:N})$ (1)

where $\bm{x}_{C+1:N}$ are generated images, $\bm{x}_{1:C}$ are conditioning
images, $\bm{p}_{1:N}$ are relative camera poses (extrinsics and intrinsics)
and $t_{1:N}$ are scalar, relative timestamps.1114DiM does not require
sequential temporal ordering as in video models as the architecture is
permutation-equivariant over frames. All $N$ images (conditioning and
generated) are processed by the diffusion model. Following Ho et al. (2020),
we choose our loss function to be the error between the predicted and actual
noise ($L_{\mathrm{simple}}$ in their paper), though we use the L1 rather than
L2 norm, because, like prior work (Saharia et al., 2022a; Saxena et al.,
2023), we found that it improves sample quality. Our models use the
“$v$-parametrization” (Salimans and Ho, 2022), which helps stabilize training,
and we adopt the noise schedules proposed by Kingma et al. (2021). Our current
models process $N=8$ or 32 images at resolution $256\\!\times\\!256$ ($N$ is
the number of conditioning plus generated frames). To this end, we decompose
the task into two models similar to prior work (Ho et al., 2022a, b); we first
generate images at $64\\!\times\\!64$, and then up-sample using a $4\times$
super-resolution model trained with noise conditioning augmentation (Saharia
et al., 2022c). We also finetuned the 32-frame model to condition on 2 and 8
frames to enable more comparisons and example applications of 4DiM models.

#### Training data

While 3D assets, multi-view image data, and 4D data are limited, video data is
available at scale and contains rich information about the 3D world, despite
not having camera poses. One of our key propositions is thus to train 4DiM on
a large-scale dataset of 30M videos without pose annotations, jointly with 3D
and 4D data. As shown in Section 5.1, video plays a key role in helping to
regularize the model. The 3D datasets used to train 4DiM include ScanNet++
(Yeshwanth et al., 2023) and Matterport3D (Chang et al., 2017), which have
metric scale, and more free-form camera poses compared to other common 3D
datasets in the literature (e.g., CO3D (Reizenstein et al., 2021) and MVImgNet
(Yu et al., 2023b)). We also use $1000$ scenes from Street View with
permission from Google, comprising posed panoramas with timestamps (i.e., it
is a “4D” dataset). During training, we randomly sample views from Street View
from the set of panorama images within $K$ consecutive timesteps ($K$=5 for
our 8-view models, and $K$=20 for 32-view models). We sample unposed videos
with probability 0.3 and views from posed datasets otherwise. 3D datasets are
sampled in proportion to the number of scenes in each dataset.

#### Calibrated RealEstate10K

One particularly rich dataset for training 3D models is RealEstate10K (Zhou et
al., 2018) (RE10K). It comprises 10,000 video segments of static scenes for
which SfM (Schönberger and Frahm, 2016) has been used to infer per-frame
camera pose, but only up to an unknown length scale. The lack of metric scale
makes training more difficult because metric regularities of the world are
lost, and it becomes more difficult for users to specify the target camera
poses or camera motions in any intuitive units. This is especially problematic
when conditioning on a single image, where scale itself otherwise becomes
ambiguous. We therefore created a calibrated version of RealEstate10K by
regressing the unknown metric scale from a monocular depth estimation model
(Saxena et al., 2023). (For details, see Supplementary Material A.) The
resulting dataset, cRE10K, has a major impact on model performance (see
Section 4 below). We follow the same procedure to calibrate the LLFF dataset
(Mildenhall et al., 2019) for evaluation purposes.

Figure 2: Architecture – We illustrate the 4DiM base model architecture,
including choices for attention blocks and our conditioning mechanism. The
4DiM super-resolution model follows the same choices modulo minor differences
to condition on low-resolution images. Outputs corresponding to the
conditioning frames are always discarded from the training objective.

#### Architecture

Relatively little training data has both time and camera pose annotations;
most 3D data represent static scenes, while video data rarely include camera
pose. Finding a way to effectively condition on both camera pose and time in a
way that allows for incomplete training data is essential. We thus propose to
chain “Masked FiLM” layers (Dumoulin et al., 2018) for (positional encodings
of) diffusion noise levels, per-pixel ray origins and directions, and video
timestamps. When any of these conditioning signals is missing (due to
incomplete training data or random dropout for classifier-free guidance (Ho
and Salimans, 2022)), the FiLM layers are designed to reduce to the identity
function, rather than simply setting missing values to zero. This avoids the
network from confusing a timestamp of zero with dropped or missing data. In
practice, we replace the FiLM shift with zeros and scale with ones. We
illustrate the overall choices in Fig. 2. (For details see Supplementary
Material B.)

#### Sampling

Steering 4DiM with the correct sampling hyperparameters is essential,
especially the guidance weights for classifier-free guidance (Ho and Salimans,
2022) (CFG). In its usual formulation, CFG treats all conditioning variables
as “one big variable”. In practice, placing a different weight on each
variable is important; e.g., text requires high guidance weights, but high
guidance weights on images can quickly lead to unwanted artifacts. We thus
propose multi-guidance, where we generalize CFG to do exactly this, without
making independence assumptions between conditioning variables. We start from
a classifier-guided formulation (Dhariwal and Nichol, 2021), where we wish to
sample with $k$ conditioning signals, $v_{1},v_{2},...v_{k}$, using the score

$\nabla_{\bm{x}}\log p(\bm{x})\,+\,w_{1}\nabla_{\bm{x}}\log
p(v_{1}|\bm{x})\,+\,\sum_{j=2...k}w_{j}\nabla_{\bm{x}}\log
p(v_{j}|v_{1:j-1},\bm{x})~{}.$ (2)

In the classifier-free formulation, this is equivalent to

$\begin{split}(1-w_{1})\,\nabla_{\bm{x}}\log
p(\bm{x})\,+\,\sum_{j=2...k}(w_{j-1}-w_{j})\,\nabla_{\bm{x}}\log
p(\bm{x}|v_{1:j-1})\,+\,w_{k}\nabla_{\bm{x}}\log
p(\bm{x}|v_{1:k})~{}.\end{split}$ (3)

If we train our model such that it drops out only $v_{k}$, or $v_{k-1}$ and
$v_{k}$, … , or all of $v_{1:k}$, we can sample with different guidance
weights $w_{i}$ on each conditioning variable $v_{i}$. 4DiM is trained to drop
out conditioning signals with probability 0.1, and it drops either $t_{1:N}$,
$t_{1:N}$ and $\bm{p}_{1:N}$, or all of $t_{1:N}$, $\bm{p}_{1:N}$ and
$\bm{x}_{1:C}$. This is a natural choice, since poses and timestamps without
corresponding conditioning images do not convey useful information. For best
results, 4DiM uses a guidance weight 2.0 on conditioning images, a weight of
4.0 on camera poses, and 1.0 for timestamps.

##  4 Evaluation

Evaluating 4D generative models is challenging. Typically, methods for NVS are
evaluated using generation quality. In addition, for pose-conditioned
generation it is desirable to find metrics that capture 3D consistency
(rendered views should be grounded in a consistent 3D scene) and pose
alignment (the camera should move as expected). Furthermore, evaluating
temporal conditioning requires some measure that captures the motion of
dynamic content. We leverage several existing metrics and propose new ones to
cover all these aspects. Below we cover each one in detail.

#### Image and Video fidelity

FID (Heusel et al., 2017) is a key metric for image quality, but if used in
isolation it can be uninformative and a poor objective for hyper-parameter
selection. For example, Saharia et al. (2022b) found that FID scores for text-
to-image models are optimal with low CFG weights (Ho and Salimans, 2022), but
text-image alignment suffers under his setting. In what follows we report FID
as well as the improved Frechet distance with DINOv2 features (FDD) (Oquab et
al., 2023), which appears to correlate with human judgements better than
InceptionV3 features (Szegedy et al., 2016). For video quality, we also report
FVD scores (Unterthiner et al., 2018).

#### 3D consistency

The work of Yu et al. (2023a) introduced the thresholded symmetric epipolar
distance (TSED) as a more computationally efficient alternative to training
separate NeRF models (Mildenhall et al., 2021) on every sample to evaluate 3D
consistency (Watson et al., 2022). First, SIFT is used to obtain keypoints
between a pair of views. For each keypoint in the first image, we can compute
the epipolar line in the second image and its minimum distance to the
corresponding keypoint. This is repeated for each keypoint in the second image
to obtain a SED score. A hyperparameter threshold is selected, and the
percentage of image pairs whose median SED is below the threshold is reported.
Below, we report TSED with a threshold of 2.0 and include the TSED score of
ground truth data, as poses in the data are noisy and do not achieve a perfect
score.

#### Pose alignment (new metric: SfM distances)

We propose a set of metrics which we name SfM distances. Here, we run COLMAP
pose estimation on the generated views, and compare the camera extrinsics
predicted by COLMAP to the target poses. To get best results from COLMAP, we
also feed it the input views. We report the relative error in camera positions
(SfMDpos) and the angular deviation of camera rotations in radians (SfMDrot).
Like we do for TSED scores, we also report the SfM distances for the ground
truth data. Due to the inherent scale and rotational ambiguity of COLMAP,
additional care must be taken to align the estimated poses to the original
ones before comparing their differences as described above. Please see
Supplementary Material D for more detail.

#### Metric scale pose alignment

The aforementioned metrics for 3D consistency and pose alignment are invariant
to the scale of the camera poses as they rely on epipolar distances and SfM.
In order to evaluate the metric scale alignment of 4DiM we report PSNR, SSIM
(Wang et al., 2004) and LPIPS (Zhang et al., 2018). These reconstruction-based
metrics are not generally suitable to measure sample quality, as generative
models can produce different but plausible samples. But for the same reason
(i.e., that reconstruction metrics favor content alignment), we instead find
them useful as an indirect indicators of metric scale alignment; i.e., given
metric-scale poses, we would like models to not over/undershoot in position
and rotation.

#### Dynamics (new metric: keypoint distance)

One common failure mode that we observed early in our work is a tendency for
certain models to copy input frames instead of generating temporal dynamics,
and prior work has noted that good FVD scores can still be obtained with
static video (Ge et al., 2024). We therefore propose a new metric called
keypoint distance (KD), where we compute the average motion of SIFT
keypoints(Lowe, 2004) across image pairs with $k\geq 10$ matches. We report
results on generated and reference views to assess whether generated images
have a similar motion distribution.

Figure 3: Qualitative comparison of 3D NVS for various diffusion models on
RE10K (in-distribution) and LLFF (OOD). Note, this is not our final 4DiM
model, but rather, an 8-frame 4DiM model trained only with RE10K for 3D data,
so it is trained with the same data as prior work for fair comparison. See our
website for more samples: https://4d-diffusion.github.io

##  5 Experiments

We consider both in-distribution and OOD evaluations for 3D NVS capabilities
in our ablations and comparisons to prior work. We use the RealEstate10K
dataset (Zhou et al., 2018) as a common in-distribution evaluation. To
maximize the amount of training data, we use 1% of the dataset as our
validation split. We run inference on all baselines ourselves for this split,
noting that they might instead be advantaged as our test data may exist in
their training data (for PNVS, all our test data is in fact part of their
training dataset, giving them a distinct advantage). For OOD evaluation, we
use the LLFF dataset (Mildenhall et al., 2019).

|  FID ($\downarrow$) |  FDD ($\downarrow$) |  FVD ($\downarrow$) |  TSEDt=2 ($\uparrow$) |  SfMDpos($\downarrow$) |  SfMDrot($\downarrow$) |  LPIPS ($\downarrow$) |  PSNR ($\uparrow$) |  SSIM ($\uparrow$)
---|---|---|---|---|---|---|---|---|---
cRE10k test |  |  |  |  |  |  |  |  |
PNVS | 51.41 | 1007. | 248.9 | 0.9961 (1.000) | 0.9773 (1.024) | 0.3068 (0.3638) | 0.3899 | 16.07 | 0.3878
MotionCtrl | 43.07 | 370.6 | 411.7 | 0.4193 (1.000) | 1.027 (1.032) | 0.2549 (0.3634) | 0.5003 | 12.74 | 0.2668
4DiM-R | 31.23 | 306.3 | 195.1 |  0.9974 (1.000) | 1.023 (1.075) | 0.3029 (0.3413) | 0.2630 | 18.09 | 0.5309
4DiM | 31.96 | 314.9 | 221.9 | 0.9935 (1.000) | 1.008 (1.034) | 0.3326 (0.3488) | 0.3016 | 17.08 | 0.4628
cLLFF zero-shot |  |  |  |  |  |  |  |  |
PNVS | 185.4 | 2197. | 1751. | 0.7235 (0.9962) | 0.9346 (0.8853) | 0.2213 (0.1857) | 0.5969 | 12.04 | 0.1311
MotionCtrl | 106.0 | 531.5 | 1754. | 0.1515 (0.9962) | 0.9148 (0.9415) | 0.1366 (0.2216) | 0.7016 | 9.722 | 0.06756
4DiM-R | 63.48 | 353.2 | 841.6 | 0.9659 (0.9962) | 0.9265 (0.8951) | 0.2011 (0.1934) | 0.5403 | 11.55 | 0.1444
4DiM | 63.78 | 356.8 | 864.4 | 0.9167 (0.9962) | 0.9131 (0.8960) | 0.1838 (0.1943) | 0.5415 | 11.58 | 0.1408

Table 1: Comparison to prior work in 3D NVS. We compare 8-frame 4DiM models
against prior work, one only using cRE10K (4DiM-R) for a more fair comparison,
and one trained with our full dataset mixture (4DiM). We evaluate in-
distribution and zero-shot on LLFF. For 3D consistency and pose alignment, we
include scores on real views (in parenthesis) to indicate plausible upper
bounds (which may be imperfect as poses are noisy and SfM may fail). We
highlight the best result in bold, and underline results where models score
better than real images. We do not report scores when SfM fails, denoted with
n/a. Our models substantially outperform baselines across all metrics.

|  FID ($\downarrow$) |  FDD ($\downarrow$) |  FVD ($\downarrow$) |  TSEDt=2 ($\uparrow$) |  SfMDpos($\downarrow$) |  SfMDrot($\downarrow$) |  LPIPS ($\downarrow$) |  PSNR ($\uparrow$) |  SSIM ($\uparrow$)
---|---|---|---|---|---|---|---|---|---
RE10k test |  |  |  |  |  |  |  |  |
4DiM-R (RE10k) | 32.40 | 335.9 | 212.0 |  1.000 (1.000) | 1.021 (1.022) | 0.3508 (0.3737) | 0.3002 | 16.64 | 0.4704
4DiM-R (cRE10k) | 31.23 | 306.3 | 195.1 | 0.9974 (1.000) | 1.023 (1.075) | 0.3029 (0.3413) | 0.2630 | 18.09 | 0.5309
LLFF zero-shot |  |  |  |  |  |  |  |  |
4DiM-R (RE10k) | 71.07 | 521.2 | 740.1 | 0.7727 (0.9962) |  0.9003 (0.8876) | 0.2108 (0.1915) | 0.4568 | 12.87 | 0.2016
4DiM-R (cRE10k) | 63.48 | 353.2 | 841.6 |  0.9659 (0.9962) | 0.9265 (0.8951) |  0.2011 (0.1934) | 0.5403 | 11.55 | 0.1444
ScanNet++ zero-shot |  |  |  |  |  |  |  |  |
4DiM-R (RE10k) | 23.68 | 248.0 | 137.9 | 0.9512 (0.9815) | 1.885 (1.858) |  1.534 (1.520) | 0.1938 | 20.74 | 0.6716
4DiM-R (cRE10k) | 22.89 | 243.2 | 130.6 |  0.9685 (0.9815) | 1.851 (1.917) | 1.541 (1.550) | 0.1809 | 21.25 | 0.6952

Table 2: Ablation showing the advantage of using calibrated data (metric scale
camera poses). We compare 8-frame 4DiM models trained on cRE10K vs
uncalibrated RE10K. For RE10K and LLFF, models trained on cRE10K are tested on
calibrated data, and models trained on uncalibrated RE10K are tested on
uncalibrated data for a fair comparison. The exception is ScanNet++, which is
already scale-calibrated. We find that training on cRE10K substantially
improves in-domain and OOD performance both in terms of alignment and image
fidelity. This is especially clear on reference-based image metrics such as
PSNR, LPIPS, SSIM (with the exception of LLFF, where scores may be too low to
be significant).

|  FID ($\downarrow$) |  FDD ($\downarrow$) |  FVD ($\downarrow$) |  TSEDt=2 ($\uparrow$) |  SfMDpos($\downarrow$) |  SfMDrot($\downarrow$) |  LPIPS ($\downarrow$) |  PSNR ($\uparrow$) |  SSIM ($\uparrow$)
---|---|---|---|---|---|---|---|---|---
cRE10k test |  |  |  |  |  |  |  |  |
4DiM (no video) | 32.45 | 326.5 | 207.7 |  0.9974 (1.000) | 1.023 (1.029) | 0.3356 (0.3692) | 0.2713 | 17.80 | 0.5209
4DiM | 31.96 | 314.9 | 221.9 | 0.9935 (1.000) | 1.008 (1.034) | 0.3326 (0.3488) | 0.3016 | 17.08 | 0.4628
ScanNet++ test |  |  |  |  |  |  |  |  |
4DiM (no video) | 24.61 | 250.7 | 129.2 | 0.9615 (0.9815) | 1.895 (1.764) |  1.446 (1.431) | 0.1824 | 21.34 | 0.6969
4DiM | 23.48 | 223.7 | 146.0 | n/a (0.9815) | 1.706 (1.889) | 1.550 (1.503) | 0.1965 | 20.67 | 0.6434
cLLFF zero-shot |  |  |  |  |  |  |  |  |
4DiM (no video) | 64.34 | 401.4 | 884.1 |  0.9886 (0.9962) | 0.9157 (0.9067) | 0.1872 (0.1965) | 0.5437 | 11.45 | 0.1430
4DiM | 63.78 | 356.8 | 864.4 | 0.9167 (0.9962) |  0.9131 (0.8960) | 0.1838 (0.1943) | 0.5415 | 11.58 | 0.1408

Table 3: Ablation showing the advantage of co-training with video data. We
train an identical 8-frame 4DiM model without video data to reveal the net-
positive impact of co-training with video. Co-training with video data
improves fidelity and generalization. Qualitatively, this is evident in 4DiM’s
ability to extrapolate realistic content.  Figure 4: Qualitative comparison of
4DiM trained on calibrated vs uncalibrated RealEstate10K. The latter tends to
‘overshoot’ along the trajectory.

We present our main results on 3D novel view synthesis conditioned on a single
image in Tab. 1 and Fig. 3, comparing the capabilities of our 4DiM cascade
against PhotoConsistentNVS222For PNVS, we follow Yu et al. (2023a) and use a
Markovian sliding window for sampling, as they find it is the stronger than
stochastic conditioning (Watson et al., 2022). (Yu et al., 2023a) (PNVS) and
MotionCtrl (Wang et al., 2023b). These two are the strongest 3D scene NVS
diffusion models with code and checkpoints available. For LLFF, we load the
input trajectory of views in order, subsample frames evenly, and then generate
7 views given a single input image. RealEstate10K trajectories are much
longer, so following PNVS, we subsample with a stride of 10. We use the 7-view
NVS task conditioned on a single image, in order to avoid weakening the
baselines: MotionCtrl can only predict up to 14 frames, and PNVS performance
degrades with the length of the sequence as it is an image-to-image model. We
trained versions of 4DiM that process 8-frames to this end (as opposed to
using the main 32-frame 4DiM model and subsampling the output) for a more
apples-to-apples comparison. Quantitative metrics are computed on 128 scenes
from our RealEstate10K test split, and on all 44 scenes in LLFF.

We find that 4DiM achieves superior results in fidelity (FID, FDD, FVD) and
metrc-scale pose alignment (LPIPS, PSNR, SSIM), by a wide margin compared to
the baseline diffusion models. Qualitatively, we find that MotionCtrl has
trouble aligning with the conditioning poses. We also observe that PNVS
exhibits more artifacts, with a substantial drop in quality in the out-of-
distribution setting. Surprisingly, PNVS achieves the best TSED in LLFF but
upon closer inspection, we find that the number of keypoint matches is
$\sim$3x less than for 4DiM. This suggests potential improvements for TSED,
such as a higher threshold for the number of minimum keypoint matches, or
requiring more spatial coverage of keypoints to discard examples for which the
matches are too localized.

###  5.1 3D datasets significantly affect NVS quality

One of the most critical ingredients for 3D NVS, especially in the out-of-
distribution setting, is the training dataset. We ablate this in three ways:

#### More diverse 3D data is helpful

Table 1 shows that adding more diverse 3D data beyond RealEstate10K to the
training mixture improves pose alignment (SfM distances) on LLFF (zero-shot).
The evaluation suggests some loss in fidelity when using the full mixture
which is expected since the mixture is more diverse and includes both indoor
and outdoor scenes.

#### Scale-calibrated data resolves ambiguity

To determine the impact of consistent, metric scale training data, we compare
a model trained with on the original RE10k data against one with calibrated
metric scale poses, with all else unchanged. Neither model includes the other
3D data sources in their training data used to train our best 4DiM model,
because these are scale-calibrated and would confound this study otherwise.
Quantitative results on 3D NVS from a single conditioning image are given in
Tab. 2 (including zero-shot performance on LLFF and ScanNet++), and
qualitative results are given in Fig. 4. We find that models trained on
uncalibrated data often overshoot or undershoot when scale is ambiguous, and
that models trained on scale-calibrated data solve this problem.

#### Large-scale video data improves generalization

We also show the importance of leveraging video as a training data source.
Table 3 presents results comparing 8-frame 4DiM trained on our final data
mixture to the otherwise identical model but trained without video data. We
find that including video data improves fidelity and generalization ability
which is expected since available 3D/4D data is not as diverse. The
improvement is more apparent qualitatively (see Supplementary Material F for
more samples).

###  5.2 Emergence of temporal dynamics

| 30M video dataset test split
---|---
|  FID ($\downarrow$) |  FDD ($\downarrow$) |  FVD ($\downarrow$) |  KD ($\uparrow$)
4DiM (1-frame conditioned) | 62.73 | 355.4 | 934.0 | 2.214 (11.64)
4DiM (2-frame conditioned) | 56.92 | 334.7 | 809.8 | 2.820 (11.51)
4DiM (8-frame conditioned) | 43.34 | 320.7 | 400.2 | 3.181 (6.269)
Table 4: Comparing different numbers of input frames for video extrapolation.
With 32-frame 4DiM models, we find that increasing the number of conditioning
frames greatly improves dynamics. Note that, because this is an extrapolation
task with constant video frame rate, the true keypoint distance decreases with
more conditioning frames (i.e., less frames to generate).

Early in our experiments, we observed that video from a single image can often
yield samples with little motion. While this can be remedied by lowering the
guidance weight on the conditioning image, this can entail a cost in fidelity.
Nevertheless, with two or more input frames, 4DiM can successfully interpolate
and extrapolate video. We quantitatively illustrate these results through our
keypoint distance metric (along with other standard metrics) in Tab. 4. We
suspect that future work including additional controls (e.g., text) on models
like 4DiM, as well as larger scale, will likely play a key role in breaking
modes for temporal dynamics, given that text+image-to-video has been
successful for large models (Brooks et al., 2024).

|  |  |  |  |  |
---|---|---|---|---|---|---
| Input
StreetView |  |  |  |  |  |
| Stitched using gamma adjustment and homography warp
|  |  |  |  |  |
| Generated
Figure 5: Stitching panoramas is non-trivial due to exposure differences,
which is hard to fix with simple gamma/gain adjustment. Here we show novel
views rendered by 4DiM conditioned on 6 frames covering 360 $\degree$ fov. We
use our 8-frame conditioned model - conditioning frames are randomly
replicated to achieve desired arity. (Note this is a subset of the 24
generated frames by 4DiM.)

|  |  |  |  |  |  |  |
---|---|---|---|---|---|---|---|---
| Input
Street View |  |  |  |  |  |  |  |
| Warped input
|  |  |  |  |  |  |  |
| Generated

Figure 6: Novel views conditioned on a space-time trajectory on the Street
View dataset. We condition on 8 consecutive time steps from the front-left
camera and generate views at the same time steps with a camera facing the
front. We find that the hallucinated regions (right half of the images) are
consistent across space-time. Warped input images are provided for reference.

###  5.3 Multiframe Conditioning

While we (and most baselines) focus on single frame conditioning, using more
than one frame unlocks interesting capabilities. Here we show two
applications. First, we consider 4DiM on the task of panorama stitching
(rendering novel viewpoints), given discrete images that collectively cover a
full 360$\degree$ FOV (see Fig. 5). For reference we stitch the images using a
naive homography warp and gamma adjustment (Brown and Lowe, 2007). We find
that outputs from 4DiM have higher fidelity than this baseline, and they avoid
artifacts like those arising from incorrect exposure adjustment.

# conditioning | Street View panoramas | Matterport3D $360\degree$ panoramas
---|---|---
frames | FID ($\downarrow$) | FDD ($\downarrow$) | PSNR ($\uparrow$) | SSIM ($\uparrow$) | LPIPS ($\downarrow$) | FID ($\downarrow$) | FDD ($\downarrow$) | PSNR ($\uparrow$) | SSIM ($\uparrow$) | LPIPS ($\downarrow$)
1 (60°FOV) | 119.7 | 1601 | 12.16 | 0.2925 | 0.5438 | 86.41 | 1080 | 9.366 | 0.1638 | 0.6467
6 (360°FOV) | 37.43 | 354.1 | 17.76 | 0.5249 | 0.2663 | 24.86 | 239.4 | 15.64 | 0.4974 | 0.2987

Table 5: Quantitative performance on generating panos in the extrapolation
(number of conditioning frames is 1) and interpolation (number of conditioning
frames is 6) regime. Using more conditioning frames improves image fidelity
which is to be expected. See Fig. 5 for samples.

We next consider the task of rendering an existing space-time trajectory from
novel viewpoints (i.e., video to video translation). We condition on images
from a single camera at 8 consecutive frames from a Street View sequence and
render a different view yielding 3D-consistent hallucinated regions in the
generated images. Results are shown in Fig. 6.

##  6 Discussion and future work

To the best of our knowledge, 4DiM is the first model capable of generating
multiple, approximately consistent views over simultaneous camera and time
control from as few as a single input image. 4DiM achieves state-of-the-art
pose alignment and much better generalization compared to prior work on 3D
NVS. This new class of models opens the door to myriad downstream
applications, some of which we demonstrate, e.g., changing camera pose in
videos and seamlessly stitching panoramas. While still not perfect, 4DiM also
achieves previously unseen quality in extremely challenging camera
trajectories such as $360\degree$ rotation from a single image in the wild,
indoor or outdoor. We expect 4DiM and follow-up works to have a major impact
on methods focused on 3D model extraction, which rely heavily on pose-
conditional diffusion models.

#### Limitations and future work

Results with 4DiM are promising, but there is room for improvement, with more
calibrated 3D/4D data and larger models, with which the improved capacity is
expected to improve image fidelity, 360∘ camera rotation, and dynamics with
single-frame conditioning.

#### Societal impact

Like all generative image and video models, it is important to develop and
release models with care. 4DiM is trained largely on data of scenes without
people (or anonymized where present), and does not condition on text prompts,
which mitigates many of the safety issues that might otherwise arise.

## References

  * Brooks et al. [2024] Tim Brooks, Bill Peebles, Connor Holmes, Will DePue, Yufei Guo, Li Jing, David Schnurr, Joe Taylor, Troy Luhman, Eric Luhman, Clarence Ng, Ricky Wang, and Aditya Ramesh.  Video generation models as world simulators.  2024\.  URL https://openai.com/research/video-generation-models-as-world-simulators.
  * Brown and Lowe [2007] Matthew A. Brown and David G. Lowe.  Automatic panoramic image stitching using invariant features.  _International Journal of Computer Vision_ , 74:59–73, 2007\.  URL https://api.semanticscholar.org/CorpusID:5328928.
  * Chang et al. [2017] Angel Chang, Angela Dai, Thomas Funkhouser, Maciej Halber, Matthias Niessner, Manolis Savva, Shuran Song, Andy Zeng, and Yinda Zhang.  Matterport3d: Learning from rgb-d data in indoor environments.  _arXiv preprint arXiv:1709.06158_ , 2017.
  * Dehghani et al. [2023] Mostafa Dehghani, Josip Djolonga, Basil Mustafa, Piotr Padlewski, Jonathan Heek, Justin Gilmer, Andreas Peter Steiner, Mathilde Caron, Robert Geirhos, Ibrahim Alabdulmohsin, et al.  Scaling vision transformers to 22 billion parameters.  In _International Conference on Machine Learning_ , pages 7480–7512. PMLR, 2023.
  * Dhariwal and Nichol [2021] Prafulla Dhariwal and Alexander Nichol.  Diffusion models beat gans on image synthesis.  _Advances in neural information processing systems_ , 34:8780–8794, 2021.
  * Dumoulin et al. [2018] Vincent Dumoulin, Ethan Perez, Nathan Schucher, Florian Strub, Harm de Vries, Aaron Courville, and Yoshua Bengio.  Feature-wise transformations.  _Distill_ , 3(7):e11, 2018.
  * Gao* et al. [2024] Ruiqi Gao*, Aleksander Holynski*, Philipp Henzler, Arthur Brussee, Ricardo Martin-Brualla, Pratul P. Srinivasan, Jonathan T. Barron, and Ben Poole*.  Cat3d: Create anything in 3d with multi-view diffusion models.  _arXiv_ , 2024.
  * Ge et al. [2024] Songwei Ge, Aniruddha Mahapatra, Gaurav Parmar, Jun-Yan Zhu, and Jia-Bin Huang.  On the content bias in fr$\backslash$’echet video distance.  _arXiv preprint arXiv:2404.12391_ , 2024.
  * [9] Justin Gilmer, Andrea Schioppa, and Jeremy Cohen.  Intriguing properties of transformer training instabilities, 2023.  _To appear_.
  * He et al. [2024] Hao He, Yinghao Xu, Yuwei Guo, Gordon Wetzstein, Bo Dai, Hongsheng Li, and Ceyuan Yang.  Cameractrl: Enabling camera control for text-to-video generation.  _arXiv preprint arXiv:2404.02101_ , 2024.
  * Heusel et al. [2017] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter.  Gans trained by a two time-scale update rule converge to a local nash equilibrium.  _Advances in neural information processing systems_ , 30, 2017.
  * Ho and Salimans [2022] Jonathan Ho and Tim Salimans.  Classifier-free diffusion guidance.  _arXiv preprint arXiv:2207.12598_ , 2022.
  * Ho et al. [2020] Jonathan Ho, Ajay Jain, and Pieter Abbeel.  Denoising diffusion probabilistic models.  _Advances in neural information processing systems_ , 33:6840–6851, 2020.
  * Ho et al. [2022a] Jonathan Ho, Chitwan Saharia, William Chan, David J Fleet, Mohammad Norouzi, and Tim Salimans.  Cascaded diffusion models for high fidelity image generation.  _Journal of Machine Learning Research_ , 23(47):1–33, 2022a.
  * Ho et al. [2022b] Jonathan Ho, Tim Salimans, Alexey Gritsenko, William Chan, Mohammad Norouzi, and David J Fleet.  Video diffusion models.  _Advances in Neural Information Processing Systems_ , 35:8633–8646, 2022b.
  * Hoogeboom et al. [2023] Emiel Hoogeboom, Jonathan Heek, and Tim Salimans.  simple diffusion: End-to-end diffusion for high resolution images.  In _International Conference on Machine Learning_ , pages 13213–13232. PMLR, 2023.
  * Kingma et al. [2021] Diederik Kingma, Tim Salimans, Ben Poole, and Jonathan Ho.  Variational diffusion models.  _Advances in neural information processing systems_ , 34:21696–21707, 2021.
  * Liu et al. [2023a] Hao Liu, Matei Zaharia, and Pieter Abbeel.  Ring attention with blockwise transformers for near-infinite context.  _arXiv preprint arXiv:2310.01889_ , 2023a.
  * Liu et al. [2023b] Ruoshi Liu, Rundi Wu, Basile Van Hoorick, Pavel Tokmakov, Sergey Zakharov, and Carl Vondrick.  Zero-1-to-3: Zero-shot one image to 3d object.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 9298–9309, 2023b.
  * Lowe [2004] David G Lowe.  Distinctive image features from scale-invariant keypoints.  _International journal of computer vision_ , 60:91–110, 2004.
  * Mildenhall et al. [2019] Ben Mildenhall, Pratul P. Srinivasan, Rodrigo Ortiz-Cayon, Nima Khademi Kalantari, Ravi Ramamoorthi, Ren Ng, and Abhishek Kar.  Local light field fusion: Practical view synthesis with prescriptive sampling guidelines.  _ACM Transactions on Graphics (TOG)_ , 2019.
  * Mildenhall et al. [2021] Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng.  Nerf: Representing scenes as neural radiance fields for view synthesis.  _Communications of the ACM_ , 65(1):99–106, 2021\.
  * Niemeyer et al. [2022] Michael Niemeyer, Jonathan T Barron, Ben Mildenhall, Mehdi SM Sajjadi, Andreas Geiger, and Noha Radwan.  Regnerf: Regularizing neural radiance fields for view synthesis from sparse inputs.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 5480–5490, 2022.
  * Oquab et al. [2023] Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et al.  Dinov2: Learning robust visual features without supervision.  _arXiv preprint arXiv:2304.07193_ , 2023.
  * Peebles and Xie [2023] William Peebles and Saining Xie.  Scalable diffusion models with transformers.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 4195–4205, 2023.
  * Poole et al. [2022] Ben Poole, Ajay Jain, Jonathan T. Barron, and Ben Mildenhall.  Dreamfusion: Text-to-3d using 2d diffusion.  _arXiv_ , 2022.
  * Rajbhandari et al. [2020] Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, and Yuxiong He.  Zero: Memory optimizations toward training trillion parameter models.  In _SC20: International Conference for High Performance Computing, Networking, Storage and Analysis_ , pages 1–16. IEEE, 2020.
  * Reizenstein et al. [2021] Jeremy Reizenstein, Roman Shapovalov, Philipp Henzler, Luca Sbordone, Patrick Labatut, and David Novotny.  Common objects in 3d: Large-scale learning and evaluation of real-life 3d category reconstruction.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 10901–10911, 2021.
  * Ronneberger et al. [2015] Olaf Ronneberger, Philipp Fischer, and Thomas Brox.  U-net: Convolutional networks for biomedical image segmentation.  In _Medical image computing and computer-assisted intervention–MICCAI 2015: 18th international conference, Munich, Germany, October 5-9, 2015, proceedings, part III 18_ , pages 234–241. Springer, 2015.
  * Saharia et al. [2022a] Chitwan Saharia, William Chan, Huiwen Chang, Chris Lee, Jonathan Ho, Tim Salimans, David Fleet, and Mohammad Norouzi.  Palette: Image-to-image diffusion models.  In _ACM SIGGRAPH 2022 conference proceedings_ , pages 1–10, 2022a.
  * Saharia et al. [2022b] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al.  Photorealistic text-to-image diffusion models with deep language understanding.  _Advances in neural information processing systems_ , 35:36479–36494, 2022b.
  * Saharia et al. [2022c] Chitwan Saharia, Jonathan Ho, William Chan, Tim Salimans, David J Fleet, and Mohammad Norouzi.  Image super-resolution via iterative refinement.  _IEEE transactions on pattern analysis and machine intelligence_ , 45(4):4713–4726, 2022c.
  * Sajjadi et al. [2022] Mehdi SM Sajjadi, Henning Meyer, Etienne Pot, Urs Bergmann, Klaus Greff, Noha Radwan, Suhani Vora, Mario Lučić, Daniel Duckworth, Alexey Dosovitskiy, et al.  Scene representation transformer: Geometry-free novel view synthesis through set-latent scene representations.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 6229–6238, 2022.
  * Salimans and Ho [2022] Tim Salimans and Jonathan Ho.  Progressive distillation for fast sampling of diffusion models.  _arXiv preprint arXiv:2202.00512_ , 2022.
  * Saxena et al. [2023] Saurabh Saxena, Junhwa Hur, Charles Herrmann, Deqing Sun, and David J. Fleet.  Zero-shot metric depth with a field-of-view conditioned diffusion model.  _arXiv:2312.13252_ , 2023.
  * Schönberger and Frahm [2016] Johannes Lutz Schönberger and Jan-Michael Frahm.  Structure-from-Motion Revisited.  In _Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2016.
  * Schönberger et al. [2016] Johannes Lutz Schönberger, Enliang Zheng, Marc Pollefeys, and Jan-Michael Frahm.  Pixelwise view selection for unstructured multi-view stereo.  In _European Conference on Computer Vision (ECCV)_ , 2016.
  * Shi et al. [2023] Yichun Shi, Peng Wang, Jianglong Ye, Mai Long, Kejie Li, and Xiao Yang.  Mvdream: Multi-view diffusion for 3d generation.  _arXiv preprint arXiv:2308.16512_ , 2023.
  * Sitzmann et al. [2019] Vincent Sitzmann, Michael Zollhöfer, and Gordon Wetzstein.  Scene representation networks: Continuous 3d-structure-aware neural scene representations.  _Advances in Neural Information Processing Systems_ , 32, 2019.
  * Szegedy et al. [2016] Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jon Shlens, and Zbigniew Wojna.  Rethinking the inception architecture for computer vision.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 2818–2826, 2016.
  * Tang et al. [2023] Shitao Tang, Fuyang Zhang, Jiacheng Chen, Peng Wang, and Yasutaka Furukawa.  Mvdiffusion: Enabling holistic multi-view image generation with correspondence-aware diffusion.  _arXiv_ , 2023.
  * Tang et al. [2024] Shitao Tang, Jiacheng Chen, Dilin Wang, Chengzhou Tang, Fuyang Zhang, Yuchen Fan, Vikas Chandra, Yasutaka Furukawa, and Rakesh Ranjan.  Mvdiffusion++: A dense high-resolution multi-view diffusion model for single or sparse-view 3d object reconstruction, 2024.
  * Tseng et al. [2023] Hung-Yu Tseng, Qinbo Li, Changil Kim, Suhib Alsisan, Jia-Bin Huang, and Johannes Kopf.  Consistent view synthesis with pose-guided diffusion models.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 16773–16783, 2023.
  * Unterthiner et al. [2018] Thomas Unterthiner, Sjoerd Van Steenkiste, Karol Kurach, Raphael Marinier, Marcin Michalski, and Sylvain Gelly.  Towards accurate generative models of video: A new metric & challenges.  _arXiv preprint arXiv:1812.01717_ , 2018.
  * Wang et al. [2024] Zhengyi Wang, Cheng Lu, Yikai Wang, Fan Bao, Chongxuan Li, Hang Su, and Jun Zhu.  Prolificdreamer: High-fidelity and diverse text-to-3d generation with variational score distillation.  _Advances in Neural Information Processing Systems_ , 36, 2024.
  * Wang et al. [2004] Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero P Simoncelli.  Image quality assessment: from error visibility to structural similarity.  _IEEE transactions on image processing_ , 13(4):600–612, 2004.
  * Wang et al. [2023a] Zhouxia Wang, Ziyang Yuan, Xintao Wang, Tianshui Chen, Menghan Xia, Ping Luo, and Ying Shan.  Motionctrl: A unified and flexible motion controller for video generation.  _arXiv preprint arXiv:2312.03641_ , 2023a.
  * Wang et al. [2023b] Zhouxia Wang, Ziyang Yuan, Xintao Wang, Tianshui Chen, Menghan Xia, Ping Luo, and Ying Shan.  Motionctrl: A unified and flexible motion controller for video generation.  _arXiv preprint arXiv:2312.03641_ , 2023b.
  * Watson et al. [2022] Daniel Watson, William Chan, Ricardo Martin-Brualla, Jonathan Ho, Andrea Tagliasacchi, and Mohammad Norouzi.  Novel view synthesis with diffusion models.  _arXiv preprint arXiv:2210.04628_ , 2022.
  * Wu et al. [2023] Rundi Wu, Ben Mildenhall, Philipp Henzler, Keunhong Park, Ruiqi Gao, Daniel Watson, Pratul P. Srinivasan, Dor Verbin, Jonathan T. Barron, Ben Poole, and Aleksander Holynski.  Reconfusion: 3d reconstruction with diffusion priors.  _arXiv_ , 2023.
  * Xiong et al. [2023] Yifeng Xiong, Haoyu Ma, Shanlin Sun, Kun Han, and Xiaohui Xie.  Light field diffusion for single-view novel view synthesis.  _arXiv preprint arXiv:2309.11525_ , 2023.
  * Yeshwanth et al. [2023] Chandan Yeshwanth, Yueh-Cheng Liu, Matthias Nießner, and Angela Dai.  Scannet++: A high-fidelity dataset of 3d indoor scenes.  In _Proceedings of the International Conference on Computer Vision (ICCV)_ , 2023.
  * Yu et al. [2021] Alex Yu, Vickie Ye, Matthew Tancik, and Angjoo Kanazawa.  pixelnerf: Neural radiance fields from one or few images.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 4578–4587, 2021.
  * Yu et al. [2023a] Jason Yu, Fereshteh Forghani, Konstantinos G Derpanis, and Marcus A Brubaker.  Long-term photometric consistent novel view synthesis with diffusion models.  In _2023 IEEE/CVF International Conference on Computer Vision (ICCV)_ , pages 7071–7081. IEEE, 2023a.
  * Yu et al. [2023b] Xianggang Yu, Mutian Xu, Yidan Zhang, Haolin Liu, Chongjie Ye, Yushuang Wu, Zizheng Yan, Chenming Zhu, Zhangyang Xiong, Tianyou Liang, et al.  Mvimgnet: A large-scale dataset of multi-view images.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pages 9150–9161, 2023b.
  * Zhang and Sennrich [2019] Biao Zhang and Rico Sennrich.  Root mean square layer normalization.  _Advances in Neural Information Processing Systems_ , 32, 2019.
  * Zhang et al. [2018] Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang.  The unreasonable effectiveness of deep features as a perceptual metric.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 586–595, 2018.
  * Zhou et al. [2018] Tinghui Zhou, Richard Tucker, John Flynn, Graham Fyffe, and Noah Snavely.  Stereo magnification: Learning view synthesis using multiplane images.  _arXiv preprint arXiv:1805.09817_ , 2018.

Controlling Space and Time with Diffusion Models
Supplementary Material

##  Appendix A Calibrated RealEstate10K

To calibrate RealEstate10K, we predict a single length scale per scene with
the help of a recent zero-shot model for monocular depth [Saxena et al., 2023]
that predicts metric depth from RGB and FOV. For each image we compute metric
depth using the FOV provided by COLMAP. The length-scale is then obtained by
regressing the 3D points obtained by COLMAP to metric depth at image locations
to which the COLMAP points project. An L1 loss provides robustness to outliers
in the depth map estimated by the metric depth models. The mean and variance
of the per-frame scales yields a per-sequence estimator. We use the variance
as a simple measure of confidence to identify scenes for which the scale
estimate may not be reliable, discarding $\sim$30% of scenes with the highest
variance. Table 6 shows the importance of filtering out scenes with less
reliable calibration. The calibrated dataset will be made public to facilitate
quantitative comparisons.

|  FID ($\downarrow$) |  FDD ($\downarrow$) |  FVD ($\downarrow$) |  TSEDt=2 ($\uparrow$) |  SfMDpos($\downarrow$) |  SfMDrot($\downarrow$) |  LPIPS ($\downarrow$) |  PSNR ($\uparrow$) |  SSIM ($\uparrow$)
---|---|---|---|---|---|---|---|---|---
cRE10k test |  |  |  |  |  |  |  |  |
4DiM-R (no filtering) | 31.81 | 313.8 | 237.6 | 0.9815 (1.000) | 1.035 (1.049) | 0.3328 (0.3529) | 0.3107 | 16.49 | 0.4465
4DiM-R (w/ filtering) | 31.23 | 306.3 | 195.1 |  0.9974 (1.000) | 1.023 (1.075) | 0.3029 (0.3413) | 0.2630 | 18.09 | 0.5309
ScanNet++ zero-shot |  |  |  |  |  |  |  |  |
4DiM-R (no filtering) | 24.35 | 232.0 | 143.1 | n/a (0.9815) | 1.881 (1.774) | 1.639 (1.512) | 0.1990 | 20.15 | 0.6436
4DiM-R (w/ filtering) | 22.89 | 243.2 | 130.6 | 0.9685 (0.9815) | 1.851 (1.917) | 1.541 (1.550) | 0.1809 | 21.25 | 0.6952
cLLFF zero-shot |  |  |  |  |  |  |  |  |
4DiM-R (no filtering) | 62.55 | 369.5 | 848.5 | 0.9167 (0.9962) |  0.9214 (0.8844) | 0.2194 (0.1932) | 0.5412 | 11.48 | 0.1389
4DiM-R (w/ filtering) | 63.48 | 353.2 | 841.6 |  0.9659 (0.9962) | 0.9265 (0.8951) |  0.2011 (0.1934) | 0.5403 | 11.55 | 0.1444

Table 6: Data filtering ablation. We show the importance of discarding scenes
where scale calibration is least reliable. Beyond the clear impact on
fidelity, metric scale alignment itself improves by a wide margin. This is
quantitatively shown through the reference-based image metrics (LPIPS, PSNR,
SSIM) which favor content alignment, as discussed in Section 4.

##  Appendix B Neural Architecture

#### UViT backbone

As hinted by Figure 2, 4DiM uses a UViT architecture following Hoogeboom et
al. [2023]. Compared to the commonly employed UNet architecture [Ronneberger
et al., 2015], UViT uses a transformer backbone at the bottleneck resolution
with no convolutions, leading to improved accelerator utilization. Information
across frames mixes exclusively in temporal attention blocks, following [Ho et
al., 2022b]. We find that this is key to keep computational costs within
reasonable limits compared to, say, 3D attention [Shi et al., 2023]. Because
the temporal attention blocks have a limited sequence length (32 frames), we
insert them at all UViT resolutions. To limit memory usage, we use per-frame
self-attention blocks only at the 16x16 bottleneck resolution where the
sequence length is the product of the current resolution’s dimensions.

#### Transformer block design

Our transformer blocks combine helpful choices from several prior work. We use
parallel attention and MLP blocks following Dehghani et al. [2023], as we
found in our early experiments that this leads to slightly better hardware
utilization while achieving almost identical sample quality. We also inject
conditioning information in a similar fashion to Peebles and Xie [2023],
though we use fewer conditioning blocks due to the use of the parallel
attention + MLP. We additionally emply query-key normalization following the
findings of Gilmer et al.  for improved training stability, but with RMSNorm
[Zhang and Sennrich, 2019] for simplicity.

#### Conditioning

We use the same positional encodings for diffusion noise levels and relative
timestamps as [Saharia et al., 2022b]. For relative poses, we follow 3DiM
[Watson et al., 2022], and condition generation with per-pixel ray origins and
directions, as originally proposed by SRT [Sajjadi et al., 2022] in a
regressive setting. Xiong et al. [2023] has compared this choice to
conditioning via encoded extrinsics and focal lengths a-la Zero-1-to-3 [Liu et
al., 2023b], finding it advantageous. While a thorough study of different
encodings is missing in the literature, we hypothesize that rays are a natural
choice as they encode camera intrinsics in a way that is independent of the
target resolution, yet gives the network precise, pixel-level information
about what contents of the scene are visible and which are outside the field
of view and therefore require extrapolation by the model. Unlike Plucker
coordinates, rays additionally preserve camera positions which is key to deal
with occlusions in the underlying 3D scene.

##  Appendix C Compute, training time, and sharding

We train 8-frame 4DiM models for $\sim$1M steps. Using 64 TPU v5e chips, we
achieve a throughput of approximately 1 step per second with a batch size of
128. The model has $\sim$2.6B parameters, and available HBM is maximized with
various strategies: first, we use bfloat16 activations (but still use float32
weights to avoid instabilities). We also use FSDP (i.e., zero-redundancy
sharding [Rajbhandari et al., 2020] with delayed and rematerialized all-
gathers). We then finetune this model to its final 32-frame version. This
strategy allows us to leverage large batch-size pre-training, which is not
possible with too many frames because the amount of activations stored for
backpropagation scales linearly respect the number of frames per training
example. The model is finetuned with the same number of chips, albeit only for
50,000 steps and at batch size of 32. This allows us to shard the frames of
the video and use more than one chip per example at batch size 1. When using
temporal attention, we simply all-gather the keys and values and keep the
queries sharded over frames (one of the key insights from Ring Attention [Liu
et al., 2023a]), though we don’t decompose the computation of attention
further as we find we have sufficient HBM to fully parallelize over all-
gathered keys and values. FSDP is also enabled on the frame axis to maximize
HBM savings so the number of shards is truly the number of chips (as opposed
to the number of batch-parallel towers). This is possible because frame
sharding requires an all-reduce by mean on the loss over the frame axis.
Identically to zero-redundancy sharding, this can be broken down into a
reduce-scatter followed by an all-gather.

##  Appendix D Further details on proposed metrics

We follow Yu et al. [2023a] and compute TSED only for contiguous pairs in each
view trajectory, and discard sequences where we find less than 10 two-view
keypoint matches. We use a threshold of 2.0 in all our experiments. For our
proposed SfM distances, in order to get a scale-invariant metric, we first
align the camera positions predicted by COLMAP by relativizing the predicted
and original poses with respect to the first conditioning frame. This should
resolve rotation ambiguity. Then, we resolve scale ambiguity by analytically
solving for the least squares optimization problem that aligns the original
positions to re-scaled COLMAP camera positions. Finally, we report the
relative error in positions under the $L_{2}$ norm, i.e., we normalize by the
norm of the original camera positions.

##  Appendix E Further details on baselines from prior work

For PNVS, the conditioning image is the largest square center crop of the
original conditioning frame, resized to $256\times 256$ pixels. This is the
same preprocessing procedure used by all our 4DiM models. PNVS generates
output frames sequentially (one at a time), using 2000 denoising steps for
each frame. All 4DiM samples are produced with 256 denoising steps.

For MotionCtrl, we use the checkpoint based on Stable Diffusion, which
generates 14 frames conditioning on one image and 14 poses. We set the
sampling hyper-parameter speed to 1, use 128 ddim denoising steps, and keep
the other hyper-parameters as default values in the released script from
MotionCtrl. To avoid any loss of quality due to an out-of-distribution
resolution or aspect ratio, we use the resolution $576\times 1024$, i.e. the
same resolution as MotionCtrl was trained on. RE10K images already have the
desired aspect ratio. LLFF images have resolution $756\times 1008$; we are
thus forced to crop them to $567\times 1008$ so they have the same aspect
ratio MotionCtrl was trained with. For comparability with 4DiM and PNVS, we
postprocess MotionCtrl samples at $567\times 1008$ by taking the largest
center crop and then resizing to $256\times 256$. Note that, in order to
preserve the aspect ratio of LLFF, the resulting square outputs are for a
smaller region than 4DiM and PNVS, hence the gray padding for MotionCtrl LLFF
samples in Figure 3.

##  Appendix F Samples

We include more 4DiM samples below, though we strongly encourage the reader to
browse our website: https://4d-diffusion.github.io

Input | Output
---|---
|
|
|
|
|
|
|
|
|
|
|

Figure 7: More 4DiM samples from the RealEstate10K dataset.

Input | Circular Pan
---|---
|
|
|
|
Input | Translate Forward
|
|
|
|
Input | Translate Backward
|
|
|
|

Figure 8: More 4DiM samples with custom trajectories from the LLFF dataset.

Input | Extrapolation
---|---
|
|
|
|
|
|
Input | Interpolation
|
|
|
|
|
|

Figure 9: More 4DiM samples of videos generated from 2 input images.