#  Bianchi’s Hazzidakis transformation for generic higher dimensional quadrics

Ion I. Dincă  Department of Applied Mathematics, Faculty of Applied Sciences,
University Politehnica of Bucharest 313 Spl. Independentei 060042 Bucharest,
Romania<EMAIL_ADDRESS>
###### Abstract.

We provide a generalization of Bianchi’s Hazzidakis transformation from
$2$-dimensional quadrics to generic higher dimensional quadrics.

###### Key words and phrases:

Bäcklund transformation, Hazzidakis transformation,(confocal) quadrics, common
conjugate system, isometric deformations in $\mathbb{C}^{2n-1}$ of quadrics in
$\mathbb{C}^{n+1}$.

###### 2010 Mathematics Subject Classification:

Primary 53 B25, Secondary 53 B99

###### Contents

  1. 1 Introduction
  2. 2 Proof of the Hazzidakis transformation

##  1\. Introduction

For more details on the history of isometric deformations of surfaces and
(higher dimensional) quadrics see Dincă [1].

All computations are local and assumed to be valid on their open domain of
validity without further details; all functions have the assumed order of
differentiability and are assumed to be invertible, non-zero, etc when
required (for all practical purposes we can assume all functions to be
analytic).

Consider the complexified Euclidean space

$(\mathbb{C}^{p},<.,.>),\ <x,y>:=x^{T}y,\ |x|^{2}:=x^{T}x,\
x,y\in\mathbb{C}^{p}$

with standard basis $\\{e_{j}\\}_{j=1,...,p},\ e_{j}^{T}e_{k}=\delta_{jk}$.

Isotropic (null) vectors are those vectors $v$ of length $0$ ($|v|^{2}=0$);
since most vectors are not isotropic we shall call a vector simply vector and
we shall only emphasize isotropic when the vector is assumed to be isotropic.
The same denomination will apply in other settings: for example we call
quadric a non-degenerate quadric (a quadric projectively equivalent to the
complex unit sphere).

A quadric $x\subset\mathbb{C}^{n+1}$ is given by the quadratic equation

$Q(x):=\begin{bmatrix}x\\\ 1\end{bmatrix}^{T}\begin{bmatrix}A&B\\\
B^{T}&C\end{bmatrix}\begin{bmatrix}x\\\ 1\end{bmatrix}=x^{T}(Ax+2B)+C=0,$
$A=A^{T}\in\mathbf{M}_{n+1}(\mathbb{C}),\ B\in\mathbb{C}^{n+1},\
C\in\mathbb{C},\ \begin{vmatrix}A&B\\\ B^{T}&C\end{vmatrix}\neq 0.$

The quadric $x\subset\mathbb{C}^{n+1}$ is degenerate if $\begin{vmatrix}A&B\\\
B^{T}&C\end{vmatrix}=0$.

There are many definitions of totally real (sub)spaces of $\mathbb{C}^{n+1}$,
some even involving a hermitian inner product, but all definitions coincide:
an $(n+1)$-totally real subspace of $\mathbb{C}^{n+1}$ is of the form
$(R,t)(\mathbb{R}^{k}\times(i\mathbb{R})^{n+1-k}),\ k=0,...,n+1$, where
$(R,t)\in\mathbf{O}_{n+1}(\mathbb{C})\ltimes\mathbb{C}^{n+1}$. Now a totally
real quadric is simply an $n$-dimensional quadric in an $(n+1)$-totally real
subspace of $\mathbb{C}^{n+1}$.

A metric classification of all (totally real) quadrics in $\mathbb{C}^{n+1}$
requires the notion of symmetric Jordan canonical form of asymmetric complex
matrix. The symmetric Jordan blocks are:

$J_{1}:=0=0_{1,1}\in\mathbf{M}_{1}{\mathbb{C}},\
J_{2}:=f_{1}f_{1}^{T}\in\mathbf{M}_{2}{\mathbb{C}},\
J_{3}:=f_{1}e_{3}^{T}+e_{3}f_{1}^{T}\in\mathbf{M}_{3}{\mathbb{C}},$
$J_{4}:=f_{1}\bar{f}_{2}^{T}+f_{2}f_{2}^{T}+\bar{f}_{2}f_{1}^{T}\in\mathbf{M}_{4}{\mathbb{C}},\
J_{5}:=f_{1}\bar{f}_{2}^{T}+f_{2}e_{5}^{T}+e_{5}f_{2}^{T}+\bar{f}_{2}f_{1}^{T}\in\mathbf{M}_{5}{\mathbb{C}},$
$J_{6}:=f_{1}\bar{f}_{2}^{T}+f_{2}\bar{f}_{3}^{T}+f_{3}f_{3}^{T}+\bar{f}_{3}f_{2}^{T}+\bar{f}_{2}f_{1}^{T}\in\mathbf{M}_{6}{\mathbb{C}},$

etc,where $f_{j}:=\frac{e_{2j-1}-ie_{2j}}{\sqrt{2}}$ are the standard
isotropic vectors (at least the blocks $J_{2},\ J_{3}$ were known to the
classical geometers).

Any symmetric complex matrix can be brought via conjugation with a complex
rotation to the symmetric Jordan canonical form, that is a matrix block
decomposition with blocks of the form $a_{j}I_{p}+J_{p}$; totally real
quadrics are obtained for eigenvalues $a_{j}$ of the quadratic part $A$
defining the quadric being real or coming in complex conjugate pairs $a_{j},\
\bar{a}_{j}$ with subjacent symmetric Jordan blocks of same dimension $p$.
Just as the usual Jordan block $\sum_{j=1}^{p}e_{j}e_{j+1}^{T}$ is nilpotent
with $e_{p+1}$ cyclic vector of order $p$, $J_{p}$ is nilpotent with
$\bar{f}_{1}$ cyclic vector of order $p$, so we can take square roots of
symmetric Jordan matrices without isotropic kernels
($\sqrt{aI_{p}+J_{p}}:=\sqrt{a}\sum_{j=0}^{p-1}(_{j}^{\frac{1}{2}})a^{-j}J_{p}^{j},\
a\in\mathbb{C}^{*},\ \sqrt{a}:=\sqrt{r}e^{i\theta}$ for $a=re^{2i\theta},\
0<r,\ -\pi<2\theta\leq\pi$), two matrices with same symmetric Jordan
decomposition type (that is $J_{p}$ is replaced with a polynomial in $J_{p}$)
commute, etc.

The confocal family $\\{x_{z}\\}_{z\in\mathbb{C}}$ of a quadric with center
$x_{0}\subset\mathbb{C}^{n+1}$ in canonical form (depending on as few
constants as possible) is given in the projective space $\mathbb{CP}^{n+1}$ by
the equation

$Q_{z}(x_{z}):=\begin{bmatrix}x_{z}\\\
1\end{bmatrix}^{T}(\begin{bmatrix}A&0\\\
0^{T}&-1\end{bmatrix}^{-1}-z\begin{bmatrix}I_{n+1}&0\\\
0^{T}&0\end{bmatrix})^{-1}\begin{bmatrix}x_{z}\\\ 1\end{bmatrix}=0,$

where $A=A^{T}\in\mathbf{GL}_{n+1}(\mathbb{C})$ is symmetric Jordan.

From the definition one can see that the family of quadrics confocal to
$x_{0}$ is the adjugate of the pencil generated by the adjugate of $x_{0}$ and
Cayley’s absolute $C(\infty)\subset\mathbb{CP}^{n}$ in the hyperplane at
infinity; since Cayley’s absolute encodes the Euclidean structure of
$\mathbb{C}^{n+1}$ (it is the set invariant under rigid motions and
homotheties of $\mathbb{C}^{n+1}:=\mathbb{CP}^{n+1}\setminus\mathbb{CP}^{n}$)
the mixed metric-projective character of the confocal family becomes clear.

With $R_{z}:=I_{n+1}-zA,\ z\in\mathbb{C}\setminus\mathrm{spec}(A)^{-1}$ the
family of quadrics $\\{x_{z}\\}_{z}$ confocal to the quadric with center
$x_{0}$ is given by $Q_{z}(x_{z})=x_{z}^{T}AR_{z}^{-1}x_{z}-1=0$.

The Ivory affine transformation for confocal quadrics with center is an affine
correspondence between confocal quadrics with center and having good metric
properties: it is given by

$x_{z}=\sqrt{R_{z}}x_{0}.$

For the specific computations of isometric deformations of quadrics with
center we shall use the convention $\mathbb{C}^{n}\subset\mathbb{C}^{n+1}$
with $0$ on the $(n+1)^{\mathrm{th}}$ component; thus for example we can
multiply $(n+1,n+1)$-matrices with $n$-column vectors and similarly one can
extend $(n,n)$-matrices to $(n+1,n+1)$-matrices with zeroes on the last column
and row. The converse is also valid: an $(n+1,n+1)$-matrix with zeroes on the
last column and row (or multiplied on the left with an $n$-row vector and on
the right with an $n$-column vector) will be considered as an $(n,n)$-matrix.

With $V:=\sum_{k=1}^{n}v^{k}e_{k}=[v^{1}\ ...\ v^{n}]^{T}$ consider the
complex unit sphere

$X=X(v^{1},...,v^{n})=\frac{2V+(|V|^{2}-1)e_{n+1}}{|V|^{2}+1}.$

We have

$dX=2\frac{dV+V^{T}dV(e_{n+1}-X)}{|V|^{2}+1},\ \mathrm{so}\
|dX|^{2}=\frac{4|dV|^{2}}{(|V|^{2}+1)^{2}}.$

With $A=A^{T}\in\mathbf{GL}_{n+1}(\mathbb{C})$ symmetric Jordan we have the
quadric with center $x_{0}=(\sqrt{A})^{-1}X$.

Consider the non-trivial isometric deformation $x\subset\mathbb{C}^{2n-1}$ of
the quadric with center $x_{0}\subset\mathbb{C}^{n+1}$ with common conjugate
system $(u^{1},...,u^{n})$.

Because $(v^{1},...,v^{n})$ are isothermal-conjugate and $(u^{1},...,u^{n})$
are conjugate on $x_{0}$, the Jacobian
$\frac{\partial(v^{1},...,v^{n})}{\partial(u^{1},...,u^{n})}$ has orthogonal
columns, so with

$\lambda_{j}:=|\partial_{u^{j}}V|\neq 0,\ \Lambda:=[\lambda_{1}\ ...\
\lambda_{n}]^{T},\ \delta:=\mathrm{diag}[du^{1}\ ...\ du^{n}]$

we have $dV=R\delta\Lambda,\ R\subset\mathbf{O}_{n}(\mathbb{C})$.

With $I_{1,n}:=I_{n+1}-e_{n+1}e_{n+1}^{T}$ we have the next theorem from Dincă
[1] on non-trivial isometric deformations of quadrics with center

######  Theorem 1.1.

Non-trivial isometric deformations $x\subset\mathbb{C}^{2n-1}$ of the quadric
with center $x_{0}\subset\mathbb{C}^{n+1}$ with the isothermal-conjugate
system $(v^{1},...,v^{n})$ and common conjugate system $(u^{1},...,u^{n})$ are
in bijective correspondence with solutions of the differential system (1) in
involution (that is no further conditions appear if one imposes $d$ conditions
and one uses the equations of the system itself).

$\displaystyle d\omega-\omega\wedge\omega=-4\delta
R^{T}(I_{1,n}+Ve_{n+1}^{T})A(I_{1,n}+e_{n+1}V^{T})R\wedge\delta,\
\omega\wedge\delta-\delta\wedge R^{T}dR=0,$ $\displaystyle dV=R\delta\Lambda,\
d\Lambda=\omega\Lambda-2\delta
R^{T}(I_{1,n}+Ve_{n+1}^{T})A(2V+(|V|^{2}-1)e_{n+1}),$
$\displaystyle\Lambda^{T}\Lambda=-(2V+(|V|^{2}-1)e_{n+1})^{T}A(2V+(|V|^{2}-1)e_{n+1}),$
(1.1)
$\displaystyle\omega:=\sum_{j=1}^{n}(e_{j}e_{j}^{T}R^{T}\partial_{u^{j}}R\delta+\delta
R^{T}\partial_{u^{j}}Re_{j}e_{j}^{T}),\ R\subset\mathbf{O}_{n}(\mathbb{C}).$

We have the next theorem from Dincă [1] on the Bäcklund transformation for
higher dimensional quadrics

######  Theorem 1.2.

I (existence and inversion of the Bäcklund transformation for quadrics and the
isometric correspondence provided by the Ivory affine transformation)

Any non-trivial isometric deformation $x^{0}\subset\mathbb{C}^{2n-1}$ of an
$n$-dimensional sub-manifold $x_{0}^{0}\subset x_{0}$
($x_{0}\subset\mathbb{C}^{n+1}\subset\mathbb{C}^{2n-1}$ being a quadric)
appears as a focal sub-manifold of an $(\frac{n(n-1)}{2}+1)$-dimensional
family of Weingarten congruences, whose other focal sub-manifolds
$x^{1}=B_{z}(x^{0})$ are isometric, via the Ivory affine transformation
between confocal quadrics, to sub-manifolds $x_{0}^{1}$ in the same quadric
$x_{0}$. The determination of these sub-manifolds requires the integration of
a family of Ricatti equations depending on the parameter $z$ (we ignore for
simplicity the dependence on the initial value data in the notation $B_{z}$).
Moreover, if we compose the inverse of the rigid motion provided by the Ivory
affine transformation $(R_{0}^{1},t_{0}^{1})$ with the rolling of $x_{0}^{0}$
on $x^{0}$, then we obtain the rolling of $x_{0}^{1}$ on $x^{1}$ and $x^{0}$
reveals itself as a $B_{z}$ transformation of $x^{1}$.

Recall that we have the Ivory affine transformation
$x_{z}=\sqrt{R_{z}}(\sqrt{A})^{-1}X,\
X=\frac{2V+(|V|^{2}-1)e_{n+1}}{|V|^{2}+1}.$

Consider two points $x_{0}^{0},\ x_{0}^{1}\in x_{0}$ such that $x_{0}^{0},\
x_{z}^{1}$ are in the symmetric tangency configuration

$\displaystyle(x_{z}^{1}-x_{0}^{0})^{T}N_{0}^{0}=0\Leftrightarrow
X_{1}^{T}\sqrt{R_{z}}X_{0}=1\Leftrightarrow
x_{z}^{1}=x_{0}^{0}+\frac{(|V_{0}|^{2}+1)^{2}}{4}\sum_{k=1}^{n}X_{1}^{T}\sqrt{R_{z}}\partial_{v_{0}^{k}}X_{0}\partial_{v_{0}^{k}}x_{0}^{0}\Leftrightarrow$
(1.2)
$\displaystyle(2V_{0}+(|V_{0}|^{2}-1)e_{n+1})^{T}\sqrt{R_{z}}(2V_{1}+(|V_{1}|^{2}-1)e_{n+1})=(|V_{0}|^{2}+1)(|V_{1}|^{2}+1).$

Thus among the $2n$ functionally independent variables
$\\{v_{0}^{j},v_{1}^{j}\\}_{j=1,...,n}$ a quadratic functional relationship is
established and only $2n-1$ among them remain functionally independent

(1.3) $\displaystyle
X_{1}^{T}\sqrt{R_{z}}dX_{0}=-X_{0}^{T}\sqrt{R_{z}}dX_{1}.$

Given a non-trivial isometric deformation $x^{0}\subset\mathbb{C}^{2n-1}$ of
$x_{0}^{0}$ (that is $|dx^{0}|^{2}=|dx_{0}^{0}|^{2}$) with orthonormal normal
frame $N^{0}:=[N_{n+1}^{0}\ ...\ N_{2n-1}^{0}]$ consider the $n$-dimensional
sub-manifold

(1.4) $\displaystyle
x^{1}=x^{0}+\frac{(|V_{0}|^{2}+1)^{2}}{4}\sum_{k=1}^{n}X_{1}^{T}\sqrt{R_{z}}\partial_{v_{0}^{k}}X_{0}\partial_{v_{0}^{k}}x^{0}\subset\mathbb{C}^{2n-1}$

(that is we restrict $\\{v_{1}^{j}\\}_{j=1,...,n}$ to depend only on the
functionally independent $\\{v_{0}^{j}\\}_{j=1,...,n}$ and constants in a
manner that will subsequently become clear when we shall impose the isometric
correspondence provided by the Ivory affine transformation).

We take advantage now of the conjugate system $(u^{1},...,u^{n})$ common to
$x_{0}^{0},\ x^{0}$ and of the non-degenerate joined second fundamental forms
property; according to the principle of symmetry $0\leftrightarrow 1$ we would
like $(u^{1},...,u^{n})$ to be conjugate system to both $x^{1}$ and
$x_{0}^{1}$ and also that the non-degenerate joined second fundamental forms
property holds for $x_{0}^{1},\ x^{1}$. We obtain
$dV_{1}=R_{1}\delta\Lambda_{1}$,

$\Lambda_{1}:=\frac{(|V_{1}|^{1}+1)(|V_{0}|^{2}+1)}{2\sqrt{z}}R_{1}^{T}[X_{1}^{T}\sqrt{R_{z}}\partial_{v_{0}^{1}}X_{0}\
...\ X_{1}^{T}\sqrt{R_{z}}\partial_{v_{0}^{n}}X_{0}]^{T}$

(note that the prime integral property
$|\Lambda_{1}|^{2}=-(2V_{1}+(|V_{1}|^{2}-1)e_{n+1})^{T}A(2V_{1}+(|V_{1}|^{2}-1)e_{n+1})$
is satisfied).

Thus

$\displaystyle\sqrt{z}R_{0}\Lambda_{1}=(I_{1,n}+V_{0}e_{n+1}^{T})\sqrt{R_{z}}(2V_{1}+(|V_{1}|^{2}-1)e_{n+1})-V_{0}(|V_{1}|^{2}+1),$
(1.5)
$\displaystyle-\sqrt{z}R_{1}\Lambda_{0}=(I_{1,n}+V_{1}e_{n+1}^{T})\sqrt{R_{z}}(2V_{0}+(|V_{0}|^{2}-1)e_{n+1})-V_{1}(|V_{0}|^{2}+1)$

(the symmetry $(0,\sqrt{z})\leftrightarrow(1,-\sqrt{z})$ follows from (1.3)).
From the second equation of (1) we get $V_{1}$ as an algebraic expression of
$R_{1},\ V_{0},\ \Lambda_{0}$; replacing this into the first relation of (1)
we get $\Lambda_{1}$ as an algebraic expression of $R_{0},\ R_{1},\ V_{0},\
\Lambda_{0}$:

$\displaystyle
V_{1}=-\frac{\sqrt{z}R_{1}\Lambda_{0}+I_{1,n}\sqrt{R_{z}}(2V_{0}+(|V_{0}|^{2}-1)e_{n+1})}{e_{n+1}^{T}\sqrt{R_{z}}(2V_{0}+(|V_{0}|^{2}-1)e_{n+1})-|V_{0}|^{2}-1},$
$\displaystyle\Lambda_{1}=2R_{0}^{T}\frac{(I_{1,n}+V_{0}e_{n+1}^{T})[\sqrt{z}A(2V_{0}+(|V_{0}|^{2}-1)e_{n+1})-\sqrt{R_{z}}(I_{1,n}+e_{n+1}V_{1}^{T})R_{1}\Lambda_{0}]+V_{0}V_{1}^{T}R_{1}\Lambda_{0}}{e_{n+1}^{T}\sqrt{R_{z}}(2V_{0}+(|V_{0}|^{2}-1)e_{n+1})-|V_{0}|^{2}-1}.$

Differentiating the second relation of (1) we get

$-\sqrt{z}dR_{1}\Lambda_{0}-\sqrt{z}R_{1}[\omega_{0}\Lambda_{0}-2\delta
R_{0}^{T}(I_{1,n}+V_{0}e_{n+1}^{T})A(2V_{0}+(|V_{0}|^{2}-1)e_{n+1})]-$
$2I_{1,n}\sqrt{R_{z}}(I_{1,n}+e_{n+1}V_{0}^{T})R_{0}\delta\Lambda_{0}=[e_{n+1}^{T}\sqrt{R_{z}}(2V_{0}+(|V_{0}|^{2}-1)e_{n+1})-|V_{0}|^{2}-1]R_{1}\delta\Lambda_{1}+$
$2V_{1}[e_{n+1}^{T}\sqrt{R_{z}}(I_{1,n}+e_{n+1}V_{0}^{T})-V_{0}^{T}]R_{0}\delta\Lambda_{0},$

or equivalently the Ricatti equation

$\displaystyle-
dR_{1}=R_{1}\omega_{0}+2I_{1,n}\frac{\sqrt{R_{z}}}{\sqrt{z}}(I_{1,n}+e_{n+1}V_{0}^{T})R_{0}\delta-2R_{1}\delta
R_{0}^{T}(I_{1,n}+V_{0}e_{n+1}^{T})\frac{\sqrt{R_{z}}}{\sqrt{z}}R_{1}$
$\displaystyle+2R_{1}\delta
R_{0}^{T}\frac{[(I_{1,n}+V_{0}e_{n+1}^{T})\sqrt{R_{z}}e_{n+1}-V_{0}][\Lambda_{0}^{T}+(2V_{0}^{T}+(|V_{0}|^{2}-1)e_{n+1}^{T})\frac{\sqrt{R_{z}}}{\sqrt{z}}R_{1}]}{e_{n+1}^{T}\sqrt{R_{z}}(2V_{0}+(|V_{0}|^{2}-1)e_{n+1})-|V_{0}|^{2}-1}$
(1.7)
$\displaystyle-2\frac{[R_{1}\Lambda_{0}+I_{1,n}\frac{\sqrt{R_{z}}}{\sqrt{z}}(2V_{0}+(|V_{0}|^{2}-1)e_{n+1})][e_{n+1}^{T}\sqrt{R_{z}}(I_{1,n}+e_{n+1}V_{0}^{T})-V_{0}^{T}]}{e_{n+1}^{T}\sqrt{R_{z}}(2V_{0}+(|V_{0}|^{2}-1)e_{n+1})-|V_{0}|^{2}-1}R_{0}\delta$

in $R_{1}$.

We have now from Dincă [1] the theorem partially corresponding to Theorem 1.2
for quadrics with center:

######  Theorem 1.3.

Given $(V_{0},\Lambda_{0},R_{0})$ solution of (1) that produces a non-trivial
isometric deformation $x^{0}\subset\mathbb{C}^{2n-1}$ of $x_{0}^{0}\subset
x_{0}$, $x_{0}$ being a quadric with center the Ricatti equation (1) is
completely integrable. If $R_{1}\subset\mathbf{M}_{n}(\mathbb{C})$ is a
solution of (1), then it remains orthogonal if initially it was orthogonal and
in this case together with $V_{1},\ \Lambda_{1}$ given by (1) it is another
solution of (1) (thus producing an $(\frac{n(n-1)}{2}+1)$-dimensional family
of leaves non-trivial isometric deformations $x^{1}\subset\mathbb{C}^{2n-1}$
of $x_{0}^{1}$) and we have the symmetry
$(0,\sqrt{z})\leftrightarrow(1,-\sqrt{z})$.

Note that with

$M_{0}(z):=I_{1,n}\frac{\sqrt{R_{z}}}{\sqrt{z}}(I_{1,n}+e_{n+1}V_{0}^{T}),\
P_{0}(z):=I_{1,n}\frac{\sqrt{R_{z}}}{\sqrt{z}}(2V_{0}+(|V_{0}|^{2}-1)e_{n+1}),$
$W_{0}(z):=(I_{1,n}+V_{0}e_{n+1}^{T})\sqrt{R_{z}}e_{n+1}-V_{0},\
U_{0}(z):=e_{n+1}^{T}\sqrt{R_{z}}(2V_{0}+(|V_{0}|^{2}-1)e_{n+1})-|V_{0}|^{2}-1$

we have

$dP_{0}(z)=2M_{0}(z)dV_{0},\ dU_{0}(z)=2W_{0}(z)^{T}dV_{0},\
dM_{0}(z)=C(z)dV_{0}^{T},\ dW_{0}(z)=c(z)dV_{0}$

and (1) can be written as

$\displaystyle-dR_{1}=R_{1}\omega_{0}+2M_{0}(z)R_{0}\delta-2R_{1}\delta
R_{0}^{T}M_{0}(z)^{T}R_{1}+\frac{2}{U_{0}(z)}R_{1}\delta
R_{0}^{T}W_{0}(z)(\Lambda_{0}^{T}+P_{0}(z)^{T}R_{1})$ (1.8)
$\displaystyle-\frac{2}{U_{0}(z)}(R_{1}\Lambda_{0}+P_{0}(z))W_{0}(z)^{T}R_{0}\delta.$

We have now the main theorem of this paper:

######  Theorem 1.4.

IV (Hazzidakis transformation) If an $n$-dimensional sub-manifold
$x^{0}\subset\mathbb{C}^{2n-1}$ is a non-trivial isometric deformation of a
sub-manifold $x_{0}^{0}\subseteq x_{0}$, $x_{0}\subset\mathbb{C}^{n+1}$ being
a generic quadric and the homography $H\in\mathbf{PGL}_{n+1}(\mathbb{C})$
takes the confocal family $x_{z}$ to another confocal family
$\tilde{x}_{\tilde{z}},\ \tilde{z}=\tilde{z}(z),\ \tilde{z}(0)=0$ of a generic
quadric $\tilde{x}_{0}\subset\mathbb{C}^{n+1}$, then one infinitesimally knows
a sub-manifold $\tilde{x}^{0}=H(x^{0})\subset\mathbb{C}^{2n-1}$ (that is one
knows the first and second fundamental forms) called the Hazzidakis transform
of $x^{0}$ and non-trivial isometric deformation of a sub-manifold
$\tilde{x}_{0}^{0}\subseteq\tilde{x}_{0}$. Moreover the Hazzidakis
transformation commutes with the Bäcklund transformation ($H\circ
B_{z}=B_{\tilde{z}}\circ H$) and the $B_{\tilde{z}}(\tilde{x}^{0})$ transforms
can be algebraically recovered from the knowledge of $\tilde{x}^{0}$ and
$B_{z}(x^{0})$.

In the next section we provide the proof of the main theorem of this paper.

##  2\. Proof of the Hazzidakis transformation

A generic quadric $x_{0}\subset\mathbb{C}^{n+1}$ is a quadric with center
given by $A=\mathrm{diag}[a_{1}^{-1}\ a_{2}^{-1}\ ...\ a_{n+1}^{-1}]$ with
$a_{j}\in\mathbb{C}\setminus\\{0\\},\ j=1,...,n+1$ distinct.

For a generic higher dimensional quadric with $A=\mathrm{diag}[a_{1}^{-1}\
a_{2}^{-1}\ ...\ a_{n+1}^{-1}]$ (1) becomes

$\displaystyle d\omega-\omega\wedge\omega=-4\delta
R^{T}(I_{1,n}AI_{1,n}+a_{n+1}^{-1}VV^{T})R\wedge\delta,\
\omega\wedge\delta-\delta\wedge R^{T}dR=0,$ $\displaystyle dV=R\delta\Lambda,\
d\Lambda=\omega\Lambda-2\delta
R^{T}(2I_{1,n}AI_{1,n}V+a_{n+1}^{-1}(|V|^{2}-1)V),$
$\displaystyle\Lambda^{T}\Lambda=-(4V^{T}I_{1,n}AI_{1,n}V+a_{n+1}^{-1}(|V|^{2}-1)^{2}),$
(2.1)
$\displaystyle\omega:=\sum_{j=1}^{n}(e_{j}e_{j}^{T}R^{T}\partial_{u^{j}}R\delta+\delta
R^{T}\partial_{u^{j}}Re_{j}e_{j}^{T}),\ R\subset\mathbf{O}_{n}(\mathbb{C})$

and (1) becomes

$\displaystyle\sqrt{z}R_{0}\Lambda_{1}=2I_{1,n}\sqrt{R_{z}}I_{1,n}V_{1}+\sqrt{1-za_{n+1}^{-1}}(|V_{1}|^{2}-1)V_{0}-V_{0}(|V_{1}|^{2}+1),$
(2.2)
$\displaystyle-\sqrt{z}R_{1}\Lambda_{0}=2I_{1,n}\sqrt{R_{z}}I_{1,n}V_{0}+\sqrt{1-za_{n+1}^{-1}}(|V_{0}|^{2}-1)V_{1}-V_{1}(|V_{0}|^{2}+1).$

In the following tilde quantities correspond to non-tilde quantities and
conversely.

The generic quadric $\tilde{x}_{0}\subset\mathbb{C}^{n+1}$ is a quadric with
center given by $\tilde{A}=\mathrm{diag}[\tilde{a}_{1}^{-1}\
\tilde{a}_{2}^{-1}\ ...\ \tilde{a}_{n+1}^{-1}]$, where
$\tilde{a}_{n+1}=a_{n+1}^{-1},\
\tilde{a}_{j}=\tilde{a}_{n+1}+(a_{j}-a_{n+1})^{-1},\ j=1,...,n,\
\tilde{z}=\tilde{a}_{n+1}+(z-a_{n+1})^{-1}$.

The homography $H$ which takes the confocal family $x_{z}$ of $x_{0}$ to the
confocal family $\tilde{x}_{\tilde{z}}$ of $\tilde{x}_{0}$ is given by
$\mathbb{C}^{n+1}\ni x\mapsto H(x)=\frac{H^{\prime}x+e_{n+1}}{e_{n+1}^{T}x},\
H^{\prime}=\sum_{j=1}^{n}\frac{1}{\sqrt{a_{n+1}-a_{j}}}e_{j}e_{j}^{T},\
\tilde{H}^{\prime}=\sum_{j=1}^{n}\sqrt{a_{n+1}-a_{j}}e_{j}e_{j}^{T}$.

We have
$x_{0}=(\sqrt{A})^{-1}\frac{2V_{0}+(|V_{0}|^{2}-1)e_{n+1}}{|V_{0}|^{2}+1},\
\tilde{x}_{0}=(\sqrt{\tilde{A}})^{-1}\frac{2\tilde{V}_{0}+(|\tilde{V}_{0}|^{2}-1)e_{n+1}}{|\tilde{V}_{0}|^{2}+1}$,
so $\tilde{V}_{0}=iV_{0}$ (in fact we have $\tilde{v}_{0}^{j}=\pm iv_{0}^{j},\
j=1,...,n$, but we assume simplifications of the form
$\sqrt{a}\sqrt{b}\simeq\sqrt{ab},\ a,b\in\mathbb{C}$ since the possible signs
are accounted by symmetries in the principal hyper-planes and disappear at the
level of the linear element for isometric deformations).

Note that $I_{1,n}\tilde{A}I_{1,n}=a_{n+1}I_{1,n}-a_{n+1}^{2}I_{1,n}AI_{1,n}$.

If we choose $\tilde{u}^{j}:=ia_{n+1}^{-1}u^{j},\ j=1,...,n$, then
$\partial_{\tilde{u}^{j}}=\frac{1}{ia_{n+1}^{-1}}\partial_{u^{j}},\
d\tilde{u}^{j}=ia_{n+1}^{-1}du^{j}$, so $\tilde{\delta}=ia_{n+1}^{-1}\delta$.

Choose $\tilde{R}_{0}:=R_{0}$; from
$d\tilde{V}_{0}=\tilde{R}_{0}\tilde{\delta}\tilde{\Lambda}_{0}$ we get
$\tilde{\Lambda}_{0}=a_{n+1}\Lambda_{0}$. From the definition we have
$\tilde{\omega}_{0}=\omega_{0}$ and (2) is satisfied with
$\tilde{R}_{0},\tilde{\omega}_{0},\tilde{\delta},\tilde{V}_{0},\tilde{\Lambda}_{0}.\tilde{A}$
replacing respectively $R,\omega,\delta,V,\Lambda,A$ if (2) is satisfied with
$R_{0},\omega_{0},\delta,V_{0},\Lambda_{0}.A$ replacing respectively
$R,\omega,\delta,V,\Lambda,A$. Thus we know infinitesimally an isometric
deformation $\tilde{x}^{0}$ of an $n$-dimensional sub-manifold
$\tilde{x}_{0}^{0}\subseteq\tilde{x}_{0}$ (that is we know the first and
second fundamental forms).

Assume now that we have a Bäcklund transform $x^{1}=B_{z}(x^{0})$ of $x^{0}$;
to get a Bäcklund transform $\tilde{x}^{1}=B_{\tilde{z}}(\tilde{x}^{0})$ (that
is such that $\tilde{x}^{1}$ is a Hazzidakis transform of $x^{1}$) that can be
algebraically recovered from the knowledge of $x^{1}$ and $\tilde{x}^{0}$ we
need to show that (2) is satisfied with tilde quantities replacing non-tilde
quantities if it is satisfied initially for non-tilde quantities(in this case
the equation (1) obtained by applying $d$ to the second equation of (2) needs
not to be checked for tilde quantities replacing non-tilde quantities).

We have $\tilde{z}=-\frac{z}{a_{n+1}^{2}}(1-za_{n+1})^{-1},\
1-\tilde{z}\tilde{a}_{n+1}^{-1}=(1-za_{n+1}^{-1})^{-1}$ and

$I_{1,n}\tilde{R}_{\tilde{z}}I_{1,n}=(1-za_{n+1}^{-1})^{-1}I_{1,n}R_{z}I_{1,n}$,
so (2) is satisfied with tilde quantities replacing non-tilde quantities if it
is satisfied initially for non-tilde quantities (the signs on the left hand of
(2) may need to exchange).

## References

  * [1] I. Dincă Bianchi’s Bäcklund transformation for higher dimensonal quadrics, Journal of Geometry and Physics, 110 (2016), 306-327.


# DepthGAN: GAN-based Depth Generation of Indoor Scenes from Semantic Layouts

Yidi Li1 Yiqun Wang2 Zhengda Lu1&Jun Xiao1 1School of Artificial Intelligence,
University of Chinese Academy of Sciences
2School of Computer Science, Chongqing University
<EMAIL_ADDRESS><EMAIL_ADDRESS>{luzhengda,
<EMAIL_ADDRESS>
###### Abstract

Limited by the computational efficiency and accuracy, generating complex 3D
scenes remains a challenging problem for existing generation networks. In this
work, we propose DepthGAN, a novel method of generating depth maps with only
semantic layouts as input. First, we introduce a well-designed cascade of
transformer blocks as our generator to capture the structural correlations in
depth maps, which makes a balance between global feature aggregation and local
attention. Meanwhile, we propose a cross-attention fusion module to guide edge
preservation efficiently in depth generation, which exploits additional
appearance supervision information. Finally, we conduct extensive experiments
on the perspective views of the Structured3d panorama dataset and demonstrate
that our DepthGAN achieves superior performance both on quantitative results
and visual effects in the depth generation task. Furthermore, 3D indoor scenes
can be reconstructed by our generated depth maps with reasonable structure and
spatial coherency.

##  1 Introduction

3D scene generation is an important task of computer vision, which can be used
in a variety of downstream applications, such as 3D scene modeling, AR and VR,
etc. However, existing 3D generation methods mainly focus on constructing
single objects Luo and Hu (2021); Xie et al. (2021) or optimizing the layout
of existing 3D models for scene construction Luo et al. (2020) due to the
limitation of computational resources and the complexity of object relations
in 3D scenes. Thus, generating complex 3D indoor scenes containing different
objects and stuff remains a challenging problem.

Rather than manually building 3D scenes with multiple objects, visual
designers typically prefer controlled and simple input, such as sketches or
layouts. However, due to insufficient input information, it is difficult to
directly construct 3D scenes. Inspired by 3D reconstruction tasks, the depth
map is a viable medium, which can be regarded as the transition from 2D
layouts to 3D scenes. In this work, we aim to generate 3D scenes in the form
of depth maps from the layout inputs with efficient semantic and relational
constraints. Since the depth map provides accurate geometric relations within
the scene, 3D scenes can be further constructed.

Figure 1: The construction results of our model. (a) input semantic layouts
with our generated depth maps and RGB images. (b) point clouds reconstructed
from our generated depth maps and RGB images. More details can be found in the
supplementary material.

For this purpose, we first test the depth generation on mainstream conditional
image generation models Isola et al. (2017); Park et al. (2019) based on CNNs.
However, convolution operations are proven to have the same effect as filters
with learnable parameters, and CNN-based generation models significantly
eliminate the components with high frequency as demonstrated in Jiang et al.
(2021a). Therefore, the multi-layer convolution architecture filters out the
boundaries of different objects and leads to the unclear hierarchical
structure and boundaries in the generated depth maps.

Meanwhile, the receptive field of CNNs is limited in the local scope Luo et
al. (2016). Hence most existing CNN-based methods for semantic image
generation can not accurately predict global correlations even with pre-
designed optimizations on global features. And the attention is paid to pixels
in the receptive field rather than those outside the sight, which makes the
generated results incoherent in terms of visual perception. Moreover, the
depth map emphasizes the structural relationship between different objects,
and this defect causes these methods to generate incorrect structural
correlations.

In this paper, we propose the DepthGAN to efficiently generate depth maps for
indoor scenes from the input of semantic layouts. First, we utilize a well-
designed cascade of transformer blocks with conditional normalization on
tokens in our generator to capture the structural correlation of depth maps.
Especially, we focus more on the global feature generation while along with
mimicking the receptive field of the convolutional architecture for local
features. Meanwhile, depth maps emphasize the smoothness in the interior of
the object more than color images. We use a learned residual connection with
2D convolutional normalization to smooth the depth generation of inside
objects. Furthermore, we propose a cross attention fusion (CAF) module to
preserve the semantic boundaries of depth maps, which facilitates the
consistency of depth structure with the supervision of RGB images. Finally, we
can also reconstruct the 3D indoor scenes by our generated depth maps with the
reasonable structure and spatial coherency as shown in Fig.1. The
contributions of this paper are summarized as follows:

  * 1.

We introduce a novel approach of generating 3D scenes with semantic layout
inputs using depth maps as intermediate guidance. It provides an effective
solution for 3D structures generation of complex scenes.

  * 2.

We propose a cascaded transformer-based generator for the pipeline from
semantic layouts to depth generation. Our model generates depth maps with
global structures and reasonable structural correlation, both in terms of
quantitative metrics and visual effects.

  * 3.

A cross attention fusion module is proposed to exploit the supervision
information of color images. The fusion guidance using color images can
effectively preserve semantic edges in the depth generation results.

##  2 Related Work

#### Depth Generation.

Existing depth generation methods mainly focus on regressing dense depth maps
from images such as monocular depth estimation Bhat et al. (2021); Lee et al.
(2019). Recent works have proposed to exploit assumptions about indoor scenes,
such as planarity constraintsLee et al. (2019), to guide the network in an
encoder-decoder architecture. Nevertheless, poor edge quality and the lack of
global information are common problems of CNN-based depth estimation models.
Zhu et al. (2020) explicitly uses pre-trained semantic segmentation to guide
depth boundaries due to the high quality of edges in the semantic map. Bhat et
al. (2021) performs a global statistical analysis of the output of encoder-
decoder architecture for refining the output depth at the highest resolution.
The regressed depth map can be further used to reconstruct the 3D scene, which
inspires us a time-saving choice to generate 3D scenes by means of depth maps.
However, depth estimation tasks typically require enough natural images as
input, which is hard to meet the visual designers’ requirements of the
simplicity of manipulation and variability. To solve this problem, we utilize
generative models with the input of semantic layouts for the depth generation
task.

#### Generative Models.

Generative models have been applied widely in various tasks such as
conditional image generation Isola et al. (2017); Park et al. (2019) and 3D
model generation Xie et al. (2021); Li et al. (2021). Conditional image
generation aims to generate photorealistic images using conditional input such
as texts and semantic layouts, which has been boosted to high resolution.
However, the depth maps generated by pix2pixHD and SPADE tend to be over-
smoothed, resulting in ambiguous depth boundaries. Meanwhile, the local
receptive fields of CNN lead to incorrect depth relationships. Liu et al.
(2019) predict depthwise separable convolution conditioned on the semantic
layouts to keep semantic consistence the generated results, which also helps
for edge information retention. On the other hand, existing 3D generation
models mainly focus on the small scale objects such as Chen et al. (2021)
generates 3D objects by 2D projection matching. But this is difficult to
directly apply to large-scale 3D scenes generation due to the limitation of
model resolution.

Therefore, we utilize the vision transformer for depth generation to make a
balance between global feature aggregation and local attention and preserve
the global depth structures.

#### Vision Transformer

Vision Transformer(ViT) Dosovitskiy et al. (2021) is a self-attention model
using image patches for local feature aggregation and extraction, which is
helpful to capture long-range dependencies in images and recover global
structure. Jiang et al. (2021b); Lee et al. (2021); Zhao et al. (2021)
proposed GAN models based on ViT for unconditional image generation. However,
the generation quality of these methods is not proportional to the time
consumption due to the low computing efficiency of ViT. Swin Transformer Liu
et al. (2021) improves the computational efficiency and accuracy of vision
transformer by the shifted window multi-head self-attention strategy, which
makes it possible to apply vision transformers for generation tasks. Zhang et
al. (2021) proves the promise of using swin transformer for high-resolution
image generation.

In this work, we focus on conditional generation tasks for converting the
input semantic layouts to a depth map which contains precise structural
information. To the best of our knowledge, this is the first work that
explores depth generation frameworks for 3D scenes that only use layouts as
input.

##  3 Method

Figure 2: Overview of the generator of our proposed DepthGAN. (a) pipeline of
generator; (b) swin transformer based generator block; (c) conditional token
normalization module.

Let $M\in\mathbb{L}^{H\times W}$ be a semantic layout where $\mathbb{L}$ is a
set of integers denoting the semantic label of each pixel, and $H,W$ are the
height and width of the semantic layout. As illustrated in Fig.2 (a), the
generator of our proposed DepthGAN, which takes the $M$ as input and gets the
depth map $D\in\mathbb{R}^{H\times W}$ as output. First, we encode the input
$M$ into a latent vector z and utilize a shared convolutional module to
generate the conjoint features of depth and RGB images. After that, we use the
cascaded swin transformer-based generator blocks with up-sampling to enable
the coarse-to-fine generation. Especially, we use two branches for generating
depth maps and RGB images simultaneously and we fuse the features of two
branches with the CAF module as the cross-domain guidance after the
penultimate stage.

###  3.1 Swin Transformer based Generator Block.

Generally, traditional CNN-based methods cannot effectively interact with the
overall depth structure. Therefore, we adopt the swin transformer Liu et al.
(2021) as the basic generator block compared with the ViTDosovitskiy et al.
(2021) with higher computational complexity. Our swin transformer based
generator block takes the semantic layout $M$ and feature map $F_{i}$ from the
previous stage in different scales as the inputs at each stage as shown in
Fig.2 (b).

Given an feature map $F_{i}\in\mathbb{R}^{C_{i}\times H_{i}\times W_{i}}$ at
the $i$-th stage, where $C_{i}$ is the number of channels and $H_{i}$ and
$W_{i}$ are width and height respectively. We first utilize a series of swin
transformer blocks to generate the details of tokens with multi-head self
attention MSA Liu et al. (2021) which uses the no-shifted or shifted window
partition strategy to advocate the information interaction among consecutive
windows. Meanwhile, we replace the layer normalization module in the swin
transformer with our Conditional Token Normalization(CTN) module which fuses
the semantic constraints to the tokenized feature maps. The transformer
architecture can be described as:

$F_{i,l+1}^{T}=\text{MLP}(\text{CTN}(\text{Attn}(\text{CTN}(F_{i,l}^{T},M)),M))\\\
$ (1)

where $F_{i,l}^{T}$ is the input tokens of the $l$-th layer at the $i$-th
stage. Attn denotes the attention blocks with the window-based multi-head
self-attention under regular (W-MSA) or shifted window partition (SW-MSA), and
always appears in pairs Liu et al. (2021). Note that the number of W-MSA and
SW-MSA pairs is set to be even.

After the consecutive attention operations, we reshape the sequence of tokens
$F^{T}_{i}$ back to a 2D feature map and then adopt a pixelshuffle module Shi
et al. (2016) to upsample its resolution and downsample the embedding
dimension if the feature map is patch embedded. The output 2D feature map is
calculated as:

$\hat{F}_{i}=\text{PixelShuffle}(\text{Reshape}(F^{T}_{i}))$ (2)

Then, we learn a residual connection between $F_{i}$ and $\hat{F}_{i}$. Due to
the inconsistency of channels, we simply apply the learned SPADE normalization
Park et al. (2019) for $F_{i}$ using the input semantic layout $M$. The
learned shortcut matches the channels of $F_{i}$ to $\hat{F}_{i}$ and provides
more 2D information apart from 1D tokens, which also helps smooth the depth
map.

Finally, the feature map is upsampled and fed to the next stage. Specially, we
gradually increase the patch size and window size at each stage along with the
increasing resolution of feature maps. Thus, our well-designed cascaded swin
transformer blocks enable the model to accurately generate structure features
and achieve a similar receptive field of CNNs. Moreover, we use more cascade
pairs of swin blocks in higher input resolution, which is beneficial for
generating more natural features. Therefore, in our generator module, we
achieve a trade-off between computational complexity with generating capacity
and the local attention with global feature aggregation.

###  3.2 Conditional Token Normalization.

Due to existing ViT based generation models are unconditional, we introduce
the Conditional Token Normalization(CTN) method to learn the affine
transformation with the input condition in the layer normalization.

With the input tokens $F^{T}$ with length $L$ and embedding dim $E$, our CTN
module is formulated as:

$\hat{F}^{T}=\frac{\gamma(M^{T})}{\sigma}\odot(F^{T}-\mu)+\beta(M^{T})$ (3)

where $\odot$ is the element-wise multiplication between two vectors. $\mu$
and $\sigma$ denote the mean and standard deviation of $F^{T}$:

$\displaystyle\mu=\frac{1}{EL}\sum\limits_{C,L}F^{T}$ (4)
$\displaystyle\sigma=\sqrt{\frac{1}{EL}\sum\limits_{C,L}(F^{T}-\mu)^{2}}$ (5)

As shown in Fig.2 (c), we apply the layer norm to $F^{T}$ without the affine
transformation and use a MLP to learn the hidden representation of tokenized
semantic layout $M^{T}$. After that, the vector $\gamma(M^{T})$ and
$\beta(M^{T})$ are learned from the hidden representation using MLPs, which
aims to formulate a conditional layer normalization in tokenized feature maps.
Finally, CTN make effective use of the input condition in the ViT for our
generator.

###  3.3 Cross Attention Fusion module

Due to the consistency of some properties between RGB and depth domains, such
as the semantic boundaries, we employ a CAF module to enhance the consistent
features by the supervision of RGB images, as shown in Fig.3.

Figure 3: Cross attention fusion module. $\odot$ is element-wise
multiplication.

First, we downsample and embed $M$ to the same spatial resolution and
dimension as the current feature map. Then we perform a semantic alignment by
the element-wise multiplication between two input feature maps and the
embedded semantic layout. Thus, we fuse the semantic information for the
further cross attention. For computational efficiency, we adopt patch
embedding for the aligned feature map. $F_{D}$ and $F_{R}$ denote the
tokenized feature maps in the depth and RGB branches, respectively. Our cross-
attention module is formulated as:

$\displaystyle\hat{F}_{D}=\text{Softmax}(\frac{Q_{D}K_{R}}{\sqrt{d}})V_{D}$
(6)
$\displaystyle\hat{F}_{R}=\text{Softmax}(\frac{Q_{R}K_{D}}{\sqrt{d}})V_{R}$
(7)

where $Q,K,V,d$ are projected query, key, value matrices and dimension in
attention operation. $\hat{F}_{D}$ and $\hat{F}_{R}$ are fused tokens after
cross attention in the two branches.

After the cross-attention, we reshape and upsample $\hat{F}_{D}$,
$\hat{F}_{R}$ to the same scale of the current input feature map. Moreover, we
utilize a learned weighted-sum in the input feature maps and fused attention
maps for preserving the original features. Generally, the feature fusion
module allows one domain to help guide the direction of another domain’s
generation with consistent features. The clear edges of RGB images can
efficiently help preserve semantic boundaries in the depth generation results.
Meanwhile, we also generate RGB images with a more visual depth hierarchy.

##  4 Experiments

In this section, we first introduce the details of our experiments, including
dataset, evaluation metrics, loss functions, and implementation details. Then,
we provide the qualitative and quantitative comparisons of our method with
state-of-the-art CNN-based models. Finally, we perform the ablation studies to
validate the effectiveness of our network structure.

###  4.1 Experiments Details

#### Dataset.

We conduct our experiments on the Structured3d dataset Zheng et al. (2020),
which contains 21835 RGB-D virtual indoor panoramic images with detailed
semantic and depth annotations. For our depth generation, we transform the
panoramic images from panorama to perspective views as shown in Fig.4 (a) and
(b). Then we choose the images except ceiling and floor in a cubemap with
their semantic layouts and depth maps as the compositions of our post-
processed dataset (see Fig.4 (c)). For other two views, they are meaningless
for the depth generation task as their depth are consistent in a perspective
view. Furthermore, we follow the official scene train-test split in
Structured3d and obtain 72996 images for training and 6748 images for testing.

Figure 4: Explanation of our data preparation process. (a) Panoramic image in
Structured3d dataset. (b) Our data in cubemap. (c) Semantic and depth map
pairs in the scene, viewed in color. The mapping between (a) and (b) with the
same color box represents the transformation from a spherical view to a cubic
view. Figure 5: Visual comparison of depth generation from semantic layouts on
the Structured3d dataset with 256$\times$256 resolution. We map the depth
value with turbo color mapping, the colors from blue to red mean the process
of depth increasing.

On the other hand, the depth maps in Structured3d are stored as 16-bit gray-
scale images while the data for the generation task is usually in the format
of an 8-bit image. In order to accurately describe the depth structure of each
view and maintain the numerical accuracy, we normalize the processed cubemap
by dividing its maximum depth value and rescale each view to the range of
(0,255), and we store the perspective depth map as a 32-bit float number.

#### Evaluation Metrics.

We adopt six standard evaluation metrics commonly used in depth estimation
tasks Eigen et al. (2014); Bhat et al. (2021), including mean absolute error
(MAE), root mean square error (RMSE), logarithmic root mean square error
(RMSElog), absolute relative error (Abs Rel), square relative error (SqRel)
and threshold percentage ($\delta^{n}$). Note that, we align the generated
depth with ground truth by the mean value during evaluation since the ill-
posed problem of monocular depth generation. Thus, we concentrate more on the
structure information of the depth map. The details of evaluation metrics are
provided in the supplementary.

#### Loss Functions.

Our DepthGAN uses a generator and two discriminators to generate and
discriminate two generating branches respectively. First, we choose the multi-
scale discriminator in Isola et al. (2017) for both depth map and RGB image
branches. Then, the generator and discriminators of our network are trained
alternatively. Our generator is optimized with multiple losses used in Park et
al. (2019), including hinge-based adversarial loss, GAN feature matching loss,
perceptual loss for RGB branch while structural similarity loss(SSIM) Wang et
al. (2004) for depth branch. And all discriminators adopt the GAN hinge loss
Lim and Ye (2017) for distinguishing real/fake images.

#### Implementation Details.

For the generator, we adopt synchronized batch normalization between different
GPUs for better estimating the batch statistics. For both discriminators, we
apply the Spectral Norm to all the layers. We adopt Adam optimizer with
learning rate 0.0001 for the generator and 0.0004 for both discriminators, and
we set $\beta_{1}=0$ and $\beta_{2}=0.999$. The weight for the perceptual loss
in RGB branch is 10 and the weight for the SSIM loss in depth branch is 20.
All experiments were conducted on 8 NVIDIA TITAN RTX 24GB GPUs. More details
are provided in the supplementary material.

###  4.2 Comparison Experiments

In this sub-section, we provide the quantitative and visual comparisons on the
post-processed Structured3d dataset to prove the effectiveness of our
DepthGAN. We compare with three leading CNN-based conditional image generation
models including pix2pixHD Isola et al. (2017), SPADE Park et al. (2019), and
CC-FPSE Liu et al. (2019) at the 128$\times$128 and 256$\times$256 resolutions
respectively. For a fair comparison, all performances are reported at 100
epochs with the same training strategy as ours. We also conduct our
experiments on both single depth branch and two branches with cross attention
fusion.

#### Qualitative Results.

As shown in Fig.5, our DepthGAN generates more accurate depth maps with
reasonable structure correlation and semantic boundaries. Furthermore, the
depth hierarchical structure of our results is closer to the ground truth.
Meanwhile, our transformer architecture generates depth maps with more
reasonable structural correlations (see the red boxes in the first two rows).
Pix2pixHD and SPADE always generate additional objects with incorrect depth
relationships, and the depth maps may be over-smoothed. Moreover, the table
and curtain in the first and second row are generated incorrectly for CC-FPSE.
Our results have boundaries that are neither blurred nor over-sharpened as
shown in the last two rows of Fig.5.

Resolution | Method | MAE$\downarrow$ | AbsRel$\downarrow$ | SqRel$\downarrow$ | RMSE$\downarrow$ | RMSElog$\downarrow$ | log10$\downarrow$ | $\delta_{1}\uparrow$ | $\delta_{2}\uparrow$ | $\delta_{3}\uparrow$
---|---|---|---|---|---|---|---|---|---|---
128 | Pix2pixHD | 0.1699 | 0.1561 | 0.1397 | 0.2352 | 0.0901 | 0.0610 | 0.8307 | 0.9203 | 0.9535
SPADE | 0.1508 | 0.1340 | 0.1075 | 0.1945 | 0.0698 | 0.0492 | 0.8667 | 0.9464 | 0.9740
CC-FPSE | 0.0963 | 0.0851 | 0.0365 | 0.1368 | 0.0527 | 0.0368 | 0.9158 | 0.9739 | 0.9874
Ours w/o fuse | 0.0939 | 0.0795 | 0.0348 | 0.1357 | 0.0490 | 0.0335 | 0.9196 | 0.9734 | 0.9888
Ours | 0.0873 | 0.0764 | 0.0336 | 0.1278 | 0.0467 | 0.0328 | 0.9205 | 0.9749 | 0.9903
256 | Pix2pixHD | 0.1701 | 0.1553 | 0.1207 | 0.2349 | 0.0524 | 0.0608 | 0.8332 | 0.9257 | 0.9589
SPADE | 0.1749 | 0.1497 | 0.1261 | 0.2405 | 0.0610 | 0.0507 | 0.8381 | 0.9203 | 0.9535
CC-FPSE | 0.1111 | 0.1018 | 0.0510 | 0.1594 | 0.0630 | 0.0418 | 0.9023 | 0.9672 | 0.9841
Ours w/o fuse | 0.1013 | 0.0905 | 0.0386 | 0.1426 | 0.0516 | 0.0374 | 0.9149 | 0.9763 | 0.9870
Ours | 0.0972 | 0.0855 | 0.0371 | 0.1387 | 0.0472 | 0.0343 | 0.9199 | 0.9773 | 0.9905
Table 1: Quantitative comparison results by the proposed and previous methods on our post-processed Structured3d dataset at different resolutions. For the last three metrics, higher is better. Method | MAE$\downarrow$ | AbsRel$\downarrow$ | SqRel$\downarrow$ | RMSE$\downarrow$ | RMSElog$\downarrow$ | log10$\downarrow$ | $\delta_{1}\uparrow$ | $\delta_{2}\uparrow$ | $\delta_{3}\uparrow$
---|---|---|---|---|---|---|---|---|---
w/o patch | 0.1107 | 0.0973 | 0.0530 | 0.1561 | 0.0549 | 0.0385 | 0.9047 | 0.9680 | 0.9855
Fixed patch | 0.1102 | 0.1002 | 0.0585 | 0.1552 | 0.0552 | 0.0387 | 0.9052 | 0.9659 | 0.9839
w/o fuse | 0.1036 | 0.0919 | 0.0437 | 0.1480 | 0.0538 | 0.0372 | 0.9119 | 0.9726 | 0.9879
w/o ssim | 0.1048 | 0.0907 | 0.0447 | 0.1522 | 0.0541 | 0.0370 | 0.9099 | 0.9731 | 0.9887
Increasing patch(Ours) | 0.0991 | 0.0860 | 0.0397 | 0.1402 | 0.0481 | 0.0354 | 0.9186 | 0.9778 | 0.9899
Table 2: Ablation studies on our post-processed Structured3d dataset at
resolution 256$\times$256.

#### Quantitative Results.

As shown in Tab.1, We have improved performance by more than 10% over the
state-of-the-art methods under our proposed depth generation settings,
especially for the second-order metrics such as SqRel and RMSE. This indicates
our results have fewer extreme depth values due to the perception of global
information by the transformer-based architecture. On the other hand, we also
achieve higher $\delta_{n}$ and lower first-order metrics. Meanwhile, a large
margin exists between the former two methods and other methods since there are
no enough constraints on global features or semantic consistency. In
particular, our method achieves more precedence at resolution 256$\times$256
than 128$\times$128 as the superiority of the transformer architecture in
larger resolutions.

###  4.3 Ablation Studies

To validate the effectiveness of our DepthGAN, we conduct ablation studies at
256$\times$256 resolution as shown in Tab.2. All performances are reported
using training results of 60 epochs.

#### Cross Attention Fusion.

We verify the effect of our attention fusion strategy by removing the color
image generation branch and the CAF module, respectively. As shown in Fig. 5,
compared to results without fusion, our results marked by the black boxes in
the third and fourth rows reveal that the appearance of RGB images can
efficiently help preserving edges in the depth maps especially for objects
with complicated semantics such as each individual pillow on sofa and complex
wall structure.

#### SSIM Loss Function.

We replace the loss of depth branch with perceptual loss in Park et al. (2019)
instead of the SSIM loss, which leads to a slight decrease of model
performance. The perceptual loss focuses more on the global features of the
image, while the SSIM loss cares about the difference between two images more
from the pixel level. Thus, using SSIM loss contributes to better results for
the generation of depth maps.

#### Patch Size in Generator.

We select three different strategies of patch size in our generator blocks:
(1) Without patch embedding; (2) Fixed patch size at each stage such as 4; and
(3) The patch size increases with the input feature resolution. We empirically
set the patch size to be ${1}/{32}$ of input resolution for the 128$\times$
128 task and ${1}/{64}$ of input resolution for the 256$\times$ 256 task while
the minimum value is 1.

As shown in Tab.2, the relative errors(AbsRel,SqRel) and $\delta_{n}$ of
strategy(1) are better than strategy (2), which indicates that the pixel
accuracy of the former is greater than that of the latter and the probability
of generating extreme depth values is smaller. Finally, our increasing
strategy generates the best results in numerical metrics. Since our patch size
increasing strategy embeds the feature map at different stages to the same
size with the enlarging window size, which mimics the receptive field in CNNs,
we can efficiently generate depth details of objects at different scales.

##  5 Conclusion

In this work, we first propose the DepthGAN to efficiently generate the depth
maps for indoor scenes from the input of semantic layouts, which provides an
effective solution for complex 3D scenes generation. Meanwhile, our depth
generation results outperform the SOTA CNN-based image generation approaches
in terms of structural correlations and depth edge preservation.

In addition, the results for fine components of objects are still relatively
coarse. In our future work, we will further improve the depth generation for
the fineness of objects. Furthermore, we will also consider simplifying the
input to scene graphs or texts and applying our method to outdoor scenes.

## References

  * Bhat et al. [2021] Shariq Farooq Bhat, Ibraheem Alhashim, and Peter Wonka.  Adabins: Depth estimation using adaptive bins.  In IEEE Conference on Computer Vision and Pattern Recognition, 2021.
  * Chen et al. [2021] Chao Chen, Zhizhong Han, Yu-Shen Liu, and Matthias Zwicker.  Unsupervised learning of fine structure generation for 3d point clouds by 2d projections matching.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021.
  * Dosovitskiy et al. [2021] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby.  An image is worth 16x16 words: Transformers for image recognition at scale.  In 9th International Conference on Learning Representations, 2021\.
  * Eigen et al. [2014] David Eigen, Christian Puhrsch, and Rob Fergus.  Depth map prediction from a single image using a multi-scale deep network.  In Advances in Neural Information Processing Systems, 2014.
  * Isola et al. [2017] Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, and Alexei A. Efros.  Image-to-image translation with conditional adversarial networks.  In 2017 IEEE Conference on Computer Vision and Pattern Recognition, CVPR, pages 5967–5976. IEEE Computer Society, 2017.
  * Jiang et al. [2021a] Liming Jiang, Bo Dai, Wayne Wu, and Chen Change Loy.  Focal frequency loss for image reconstruction and synthesis.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021.
  * Jiang et al. [2021b] Yifan Jiang, Shiyu Chang, and Zhangyang Wang.  Transgan: Two transformers can make one strong gan.  arXiv preprint arXiv:2102.07074, 2021.
  * Lee et al. [2019] Jin Han Lee, Myung-Kyu Han, Dong Wook Ko, and Il Hong Suh.  From big to small: Multi-scale local planar guidance for monocular depth estimation.  arXiv preprint arXiv:1907.10326, 2019.
  * Lee et al. [2021] Kwonjoon Lee, Huiwen Chang, Lu Jiang, Han Zhang, Zhuowen Tu, and Ce Liu.  Vitgan: Training gans with vision transformers.  arXiv preprint arXiv:2107.04589, 2021.
  * Li et al. [2021] Ruihui Li, Xianzhi Li, Ka-Hei Hui, and Chi-Wing Fu.  SP-GAN: sphere-guided 3d shape generation and manipulation.  ACM Trans. Graph., 2021.
  * Lim and Ye [2017] Jae Hyun Lim and Jong Chul Ye.  Geometric GAN.  CoRR, abs/1705.02894, 2017.
  * Liu et al. [2019] Xihui Liu, Guojun Yin, Jing Shao, Xiaogang Wang, and Hongsheng Li.  Learning to predict layout-to-image conditional convolutions for semantic image synthesis.  In Advances in Neural Information Processing Systems, 2019.
  * Liu et al. [2021] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo.  Swin transformer: Hierarchical vision transformer using shifted windows.  volume abs/2103.14030, 2021.
  * Luo and Hu [2021] Shitong Luo and Wei Hu.  Diffusion probabilistic models for 3d point cloud generation.  In IEEE Conference on Computer Vision and Pattern Recognition, 2021.
  * Luo et al. [2016] Wenjie Luo, Yujia Li, Raquel Urtasun, and Richard Zemel.  Understanding the effective receptive field in deep convolutional neural networks.  In Proceedings of the 30th International Conference on Neural Information Processing Systems, 2016.
  * Luo et al. [2020] Andrew Luo, Zhoutong Zhang, Jiajun Wu, and Joshua B. Tenenbaum.  End-to-end optimization of scene layout.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020.
  * Park et al. [2019] Taesung Park, Ming-Yu Liu, Ting-Chun Wang, and Jun-Yan Zhu.  Semantic image synthesis with spatially-adaptive normalization.  In IEEE Conference on Computer Vision and Pattern Recognition, pages 2337–2346. Computer Vision Foundation / IEEE, 2019.
  * Shi et al. [2016] Wenzhe Shi, Jose Caballero, Ferenc Huszar, Johannes Totz, Andrew P. Aitken, Rob Bishop, Daniel Rueckert, and Zehan Wang.  Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network.  In 2016 IEEE Conference on Computer Vision and Pattern Recognition, 2016.
  * Wang et al. [2004] Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero P Simoncelli.  Image quality assessment: from error visibility to structural similarity.  IEEE transactions on image processing, 2004.
  * Xie et al. [2021] Jianwen Xie, Yifei Xu, Zilong Zheng, Song-Chun Zhu, and Ying Nian Wu.  Generative pointnet: Deep energy-based learning on unordered point sets for 3d generation, reconstruction and classification.  In IEEE Conference on Computer Vision and Pattern Recognition, 2021.
  * Zhang et al. [2021] Bowen Zhang, Shuyang Gu, Bo Zhang, Jianmin Bao, Dong Chen, Fang Wen, Yong Wang, and Baining Guo.  Styleswin: Transformer-based gan for high-resolution image generation.  arXiv preprint arXiv:2112.10762, 2021.
  * Zhao et al. [2021] Long Zhao, Zizhao Zhang, Ting Chen, Dimitris N Metaxas, and Han Zhang.  Improved transformer for high-resolution gans.  arXiv preprint arXiv:2106.07631, 2021.
  * Zheng et al. [2020] Jia Zheng, Junfei Zhang, Jing Li, Rui Tang, Shenghua Gao, and Zihan Zhou.  Structured3d: A large photo-realistic dataset for structured 3d modeling.  In Computer Vision - ECCV 2020 - 16th European Conference, 2020\.
  * Zhu et al. [2020] Shengjie Zhu, Garrick Brazil, and Xiaoming Liu.  The edge of depth: Explicit constraints between segmentation and depth.  In IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020.