# On the Chromatic Vertex Stability Number of Graphs

Saieed Akbaria Email<EMAIL_ADDRESS>Arash Beikmohammadib Email:
<EMAIL_ADDRESS>Sandi Klavžarc,d,e Email<EMAIL_ADDRESS>lj.si Nazanin Movarraeif Email<EMAIL_ADDRESS>
###### Abstract

The chromatic vertex (resp. edge) stability number ${\rm vs}_{\chi}(G)$ (resp.
${\rm es}_{\chi}(G)$) of a graph $G$ is the minimum number of vertices (resp.
edges) whose deletion results in a graph $H$ with $\chi(H)=\chi(G)-1$. In the
main result it is proved that if $G$ is a graph with
$\chi(G)\in\\{\Delta(G),\Delta(G)+1\\}$, then ${\rm vs}_{\chi}(G)={\rm
ivs}_{\chi}(G)$, where ${\rm ivs}_{\chi}(G)$ is the independent chromatic
vertex stability number. The result need not hold for graphs $G$ with
$\chi(G)\leq\frac{\Delta(G)+1}{2}$. It is proved that if
$\chi(G)>\frac{\Delta(G)}{2}+1$, then ${\rm vs}_{\chi}(G)={\rm es}_{\chi}(G)$.
A Nordhaus-Gaddum-type result on the chromatic vertex stability number is also
given.

a Department of Mathematical Sciences, Sharif University of Technology, Iran

b Department of Computer Engineering, Sharif University of Technology, Iran

c Faculty of Mathematics and Physics, University of Ljubljana, Slovenia

d Faculty of Natural Sciences and Mathematics, University of Maribor, Slovenia

e Institute of Mathematics, Physics and Mechanics, Ljubljana, Slovenia

f Department of Mathematics, Yazd University, Iran

Keywords: chromatic number, chromatic vertex stability number; chromatic edge
stability number

MSC Subj. Class. (2010): 05C15

##  1 Introduction

Throughout this paper all graphs are finite, simple, and having at least one
edge. The chromatic edge stability number ${\rm es}_{\chi}(G)$ of a graph $G$
is the minimum number of edges whose deletion results in a graph $H$ with
$\chi(H)=\chi(G)-1$. This natural coloring concept was introduced in 1980 by
Staton [12], and independently rediscovered much later in [5]. Nevertheless,
this concept has become the subject of wider interest only recently. The paper
[10] gives, among other results, a characterization of graphs with equal
chromatic edge stability number and chromatic bondage number. In [7], edge-
stability critical graphs were studied. The paper [1] brings Nordhaus-Gaddum
type inequality for ${\rm es}_{\chi}(G)$ (stronger than a related result from
[5]), sharp upper bounds on ${\rm es}_{\chi}(G)$ in terms of size and of
maximum degree, and a characterization of graphs with ${\rm es}_{\chi}(G)=1$
among $k$-regular graphs for $k\leq 5$. In [8] progress on three open problems
from [1] are reported. The chromatic edge stability number has been
generalized to arbitrary graphical invariants in [9], where in particular it
was considered with respect to the chromatic index, see also [2, 3].

Like edge stability numbers, vertex stability numbers were introduced in the
1980s or earlier. In [6], the $\mu$-stability of a graph $G$, where $\mu$ is
an arbitrary graph invariant, is defined as the minimum number of vertices
whose removal changes $\mu$. The paper [6] then proceeds by investigating the
stability with respect to the domination number and the independence number,
which in turn led to a series papers investigation the stability with respect
to these two invariants. In this paper, however, we are interested in the
stability with respect to the chromatic number. At least as far as we know,
this concept has not yet been explored (in [6], the stability with respect to
the chromatic number is briefly mentioned only in one sentence) which we find
quite surprising since vertex versions are usually considered before edge
versions. The closest investigation we are aware of is the paper [4], where
the stability with respect to the distinguishing number is investigated.

Let $G$ be a graph. The _chromatic vertex stability number_ ${\rm
vs}_{\chi}(G)$ of $G$ is the minimum number of vertices of $G$ such that their
deletion results in a graph $H$ with $\chi(H)=\chi(G)-1$. For instance, it is
straightforward to see that ${\rm vs}_{\chi}(P)=3$, where $P$ is the Petersen
graph. Note that if $\chi(G)=3$, then ${\rm vs}_{\chi}(G)$ is just the minimum
cardinality of a set $X\subseteq V(G)$ such that the graph induced by
$V(G)\setminus X$ is bipartite.

We also introduce the _independent chromatic vertex stability number_ , ${\rm
ivs}_{\chi}(G)$, of $G$ as the minimum number of independent vertices such
that their deletion results in a graph $H$ with $\chi(H)=\chi(G)-1$. Then our
main result reads as follows.

######  Theorem 1.1.

If $G$ is a graph with $\chi(G)\in\\{\Delta(G),\Delta(G)+1\\}$, then ${\rm
vs}_{\chi}(G)={\rm ivs}_{\chi}(G)$.

The paper is structured as follows. In the rest of this section we recall
needed definitions and concepts. In Section 2 we prove Theorem 1.1. In the
subsequent section we show that Theorem 1.1 need not hold for graphs $G$ with
$\chi(G)\leq\frac{\Delta(G)+1}{2}$, and discuss a possible threshold function
$f(\Delta(G))$ that would guarantee that if $\chi(G)\geq f(\Delta(G))$, then
${\rm vs}_{\chi}(G)={\rm ivs}_{\chi}(G)$. In the final section we prove that
if $\chi(G)>\frac{\Delta(G)}{2}+1$, then ${\rm vs}_{\chi}(G)={\rm
es}_{\chi}(G)$, and give a Nordhaus-Gaddum-type result on the chromatic vertex
stability number.

Given a graph $G=(V(G),E(G))$, a function $c:V(G)\to[k]=\\{1,\ldots,k\\}$ with
$c(v)\neq c(u)$ for each edge $uv$ is a _proper $k$-coloring_ of $G$. The
minimum $k$ for which $G$ admits a proper $k$-coloring is the _chromatic
number_ $\chi(G)$ of $G$. If $c$ is a proper coloring of $G$, then the set of
all vertices of $G$ with color $i$, $i\in[\chi(G)]$, is a color class and will
be denoted by $C_{i}$. The _open neighborhood_ of a vertex $v$ in $G$ is the
set of neighbors of $v$, denoted by $N_{G}(v)$, whereas the _closed
neighborhood_ of $v$ is $N_{G}[v]=N_{G}(v)\cup\\{v\\}$. The _degree_ of a
vertex $v$ in $G$ is denoted by $d_{G}(v)$. The subgraph of $G$ induced by
$A\subseteq V(G)$ will be denoted by $G\left[A\right]$. The complete graph of
order $n$ is denoted by $K_{n}$ and the complement of a graph $G$ by
$\overline{G}$. Finally, the order of $G$ will be denoted by $n(G)$.

##  2 Proof of Theorem 1.1

The following lemma follows directly from Brooks’ Theorem (cf. [13, p.197]).

######  Lemma 2.1.

Let $G$ be a connected graph with $\chi(G)=\Delta(G)+1$. If $\Delta(G)\neq 2$,
then $G\cong K_{\Delta(G)+1}$, and if $\Delta(G)=2$, then $G\cong C_{n}$ for
some odd $n$.

For the proof of the theorem, we also need the following lemma.

######  Lemma 2.2.

Let $G$ be a connected graph with $\chi(G)=\Delta(G)$ and ${\rm
vs}_{\chi}(G)=1$. Then there exists $v\in V(G)$ such that $d_{G}(v)=\Delta(G)$
and $\chi(G-v)=\Delta(G)-1$.

###### Proof.

Let $S=\\{u\in V(G):\ \chi(G-u)=\Delta(G)-1\\}$. Since ${\rm vs}_{\chi}(G)=1$,
we have $S\neq\emptyset$. If $S=V(G)$, then there exists $u\in S$ such that
$d_{G}(u)=\Delta(G)$, as desired. Otherwise, $S\neq V(G)$ and by the
connectivity of $G$ there exists $xy\in E(G)$ such that $x\in S$ and $y\in
V(G)\setminus S$. Let $c$ be a proper $(\Delta(G)-1)$-coloring of $G-x$. We
now claim that $d_{G}(x)=\Delta(G)$. By contradiction, assume that
$d_{G}(x)<\Delta(G)$ and consider the following two cases.

Case 1. $d_{G}(x)<\Delta(G)-1$.
There is a color class $C_{i}$ such that $i\in[\Delta(G)-1]$ and $N_{G}(x)\cap
C_{i}=\emptyset$. So, we can color $x$ by $i$ to obtain a proper
$(\Delta(G)-1)$-coloring of $G$, a contradiction.

Case 2. $d_{G}(x)=\Delta(G)-1$.
We may assume that $|N_{G}(x)\cap C_{i}|=1$ for each $i\in[\Delta(G)-1]$, for
otherwise we may proceed as in Case 1 to get a contradiction. But then we can
color $x$ by $c(y)$ to obtain a proper $(\Delta(G)-1)$-coloring of $G-y$.
Hence, $y\in S$, a final contradiction. ∎

Note that in Lemma 2.2, the only graph that applies to the case
$\chi(G)=\Delta(G)=2$ and ${\rm vs}_{\chi}(G)=1$ is $P_{3}$. Also note that
the proof of Lemma 2.2 asserts that every vertex in $S$ that has a neighbor in
$V(G)\setminus S$ has maximum degree. We now proceed with the proof of the
theorem.

Proof of Theorem 1.1. If $\chi(G)=\Delta(G)+1$, then ${\rm vs}_{\chi}(G)$ is
the number of connected components $C$ of $G$ with $\chi(C)=\Delta(G)+1$ which
are complete graphs or odd cycles by Lemma 2.1 for which ${\rm
vs}_{\chi}(C)=1$ holds. We can remove one vertex from each of these components
to reduce the chromatic number of $G$ by $1$. The set of these removed
vertices is an independent set which means that ${\rm vs}_{\chi}(G)={\rm
ivs}_{\chi}(G)$, as desired.

Now, let $\chi(G)=\Delta(G)$. If $\Delta(G)\leq 2$, then it is easy to see
that ${\rm vs}_{\chi}(G)={\rm ivs}_{\chi}(G)$. Hence we may assume in the rest
of the proof that $\Delta(G)\geq 3$. We proceed by induction on ${\rm
vs}_{\chi}(G)$. Clearly, the assertion holds for ${\rm vs}_{\chi}(G)=1$.

Let $S\subseteq V(G)$ be a set of vertices such that $|S|={\rm
vs}_{\chi}(G)\geq 2$, $\chi(G\setminus S)=\Delta(G)-1$, and $|E(G[S])|$ is as
small as possible. We are going to prove that $S$ is an independent set. By
contradiction, suppose that $S$ is not an independent set. So, there are
$u,v\in S$ such that $uv\in E(G)$. Set $G^{\prime}=G\setminus
S=G[V(G)\setminus S]$ and let $c$ be a proper $(\Delta(G)-1)$-coloring of
$G^{\prime}$.

######  Claim 1.

Let $w\in S$. Then $N_{G}(w)\cap C_{i}\neq\emptyset$, for each
$i\in[\Delta(G)-1]$.

By contradiction, suppose that there is a color class $C_{i}$ such that
$N_{G}(w)\cap C_{i}=\emptyset$. Now, we can color $w$ by $i$ to obtain a
proper $(\Delta(G)-1)$-coloring of $G\setminus S^{\prime}$, where
$S^{\prime}=S\setminus\\{w\\}$. So, ${\rm
vs}_{\chi}(G)\leq|S^{\prime}|=|S|-1$, a contradiction. ($\Box$)

######  Claim 2.

$|N_{G}(u)\cap C_{i}|=1$, $|N_{G}(v)\cap C_{i}|=1$, for each
$i\in[\Delta(G)-1]$.

By Claim 1, $|N_{G}(u)\cap C_{i}|\geq 1$, $|N_{G}(v)\cap C_{i}|\geq 1$, for
each $i\in[\Delta(G)-1]$. If $|N_{G}(u)\cap C_{j}|>1$ for some
$j\in[\Delta(G)-1]$, then because $uv\in E(G)$, we get $d_{G}(u)>\Delta(G)$, a
contradiction. ($\Box$)

######  Claim 3.

Let $w\in S$. Then $|N_{G}(w)\cap C_{i}|\leq 2$, for each $i\in[\Delta(G)-1]$.

By Claim 1, $|N_{G}(w)\cap C_{i}|\geq 1$, for each $i\in[\Delta(G)-1]$. If
$|N_{G}(w)\cap C_{j}|>2$ for some $j\in[\Delta(G)-1]$, then we can conclude
that $d_{G}(w)>\Delta(G)$, a contradiction. ($\Box$)

######  Claim 4.

$\Delta(G[S])\leq 1$.

By Claim 1, if $w\in S$, then $|V(G^{\prime})\cap N_{G}(w)|\geq\Delta(G)-1$.
So, $d_{G[S]}(w)\leq 1$. ($\Box$)

######  Claim 5.

$E(G[S])=\\{uv\\}$.

Since ${\rm vs}_{\chi}(G)\geq 2$ we have $\chi(G-u)=\Delta(G)$. Moreover,
$\chi(G-u)\leq\Delta(G-u)+1$, so
$\Delta(G)\in\\{\Delta(G-u),\Delta(G-u)+1\\}$. Also, ${\rm
vs}_{\chi}(G-u)=|S|-1\geq 1$. By induction, there is an independent set
$S^{\prime}\subseteq V(G)\setminus\\{u\\}$ such that $\chi((G-u)\setminus
S^{\prime})=\Delta(G)-1$ and $|S^{\prime}|=|S|-1$. Since $S^{\prime}$ is an
independent set, similarly as in Claim 4, one can deduce that
$\Delta(G[S^{\prime}\cup\\{u\\}])\leq 1$. Thus,
$|E(G[S^{\prime}\cup\\{u\\}])|\leq 1$ and we get
$1\leq|E(G[S])|\leq|E(G[S^{\prime}\cup\\{u\\}])|\leq 1$, where the middle
inequality follows by the minimality assumption on the number of edges in
$G[S]$. Hence, $E(G[S])=\\{uv\\}$, as desired. ($\Box$)

Now, let $H=G[V(G^{\prime})\cup\\{u\\}]$ and let $H_{u}$ be the connected
component of $H$ containing $u$. Clearly, if $H$ is connected, then $H=H_{u}$.
Similarly, let $H^{\prime}=G[V(G^{\prime})\cup\\{v\\}]$ and let $H_{v}$ be the
connected component of $H^{\prime}$ containing $v$. Note that
$\chi(H)=\chi(H^{\prime})=\Delta(G)$.

######  Claim 6.

If $\Delta(G)>3$, then $H_{u}\cong K_{\Delta(G)}$ and $H_{v}\cong
K_{\Delta(G)}$. If $\Delta(G)=3$, then $H_{u}\cong C_{2a+1}$ and $H_{v}\cong
C_{2b+1}$ for some natural numbers $a$ and $b$.

Each connected component of $H$ except $H_{u}$ has chromatic number at most
$\Delta(G)-1$ because it is a subgraph of $G^{\prime}$, whose chromatic number
is $\Delta(G)-1$. Since $\chi(H)=\Delta(G)$, we can conclude that
$\chi(H_{u})=\Delta(G)\leq\Delta(H_{u})+1$, which implies
$\Delta(H_{u})\geq\Delta(G)-1$. If $\Delta(H_{u})=\Delta(G)-1$, then by Lemma
2.1, $H_{u}\cong K_{\Delta(H_{u})+1}=K_{\Delta(G)}$ if
$\Delta(H_{u})=\Delta(G)-1>2$ and $H_{u}\cong C_{2a+1}$ if
$\Delta(H_{u})=\Delta(G)-1=2$, as desired. Otherwise,
$\Delta(H_{u})=\Delta(G)$ and by Lemma 2.2, there is a vertex $x\in V(H_{u})$
such that $d_{H_{u}}(x)=\Delta(G)$ and $\chi(G\setminus
S^{\prime})<\Delta(G)$, where $S^{\prime}=(S\setminus\\{u\\})\cup\\{x\\}$. It
is easy to see that $S^{\prime}$ is an independent set because
$S\setminus\\{u\\}$ is an independent set and all neighbors of $x$ are in
$V(H_{u})$ and not in $S\setminus\\{u\\}$ (because $d_{H_{u}}(x)=\Delta(G)$
and $V(H_{u})\cap(S\setminus\\{u\\})=\emptyset$). Hence
$|E(G[S^{\prime}])|<|E(G(S])|$, a contradiction.

By the same method, $H_{v}\cong K_{\Delta(H_{v})+1}=K_{\Delta(G)}$ for
$\Delta(H_{v})=\Delta(G)-1>2$ and $H_{v}\cong C_{2b+1}$ for
$\Delta(H_{v})=\Delta(G)-1=2$. ($\Box$)

######  Claim 7.

$N_{G}(u)\cap N_{G}(v)=\emptyset$.

Case 1. $\Delta(G)>3$.
By Claim 6, $H_{u}\cong H_{v}\cong K_{\Delta(G)}$. By contradiction, suppose
that $N_{G}(u)\cap N_{G}(v)\neq\emptyset$ and let $x\in N_{G}(u)\cap
N_{G}(v)$. Clearly, $x\notin S$ and $x\in V(H_{u})\cap V(H_{v})$. Therefore,
$x$ is adjacent to all vertices of $N_{G}[u]\cup N_{G}[v]$. If $N_{G}[u]\neq
N_{G}[v]$, then $d_{G}(x)>\Delta(G)$, a contradiction. Otherwise,
$N_{G}[u]=N_{G}[v]$ and thus $G[N_{G}[u]]\cong K_{\Delta(G)+1}$ which means
$\chi(G)=\Delta(G)+1$, a contradiction.

Case 2. $\Delta(G)=3$.
By Claim 6, $H_{u}\cong C_{2a+1}$ and $H_{v}\cong C_{2b+1}$ for some natural
numbers $a$ and $b$. By contradiction, suppose that $N_{G}(u)\cap
N_{G}(v)\neq\emptyset$. Since $N_{G}(u)\cap N_{G}(v)\neq\emptyset$,
$H_{u}\cong C_{2a+1}$, $H_{v}\cong C_{2b+1}$, and $H_{u}$ and $H_{v}$ are
connected components of $H$ and $H^{\prime}$, we can conclude that
$H_{u}-u=H_{v}-v$. So, $u$ and $v$ have two common neighbors. Let $x$ and $y$
be those vertices. If $2a+1=2b+1=3$, then $u$, $v$, $x$, and $y$ form a
$K_{4}$, which means $\chi(G)>\Delta(G)=3$, a contradiction. Otherwise, $x$
and $y$ are not adjacent and since $c(x)\neq c(y)$ by Claim 2, we can color
$u$ and $v$ by $c(x)$ and $c(y)$, respectively, to obtain a proper
$(\Delta(G)-1)$-coloring of $G\setminus S^{\prime}$, where
$S^{\prime}=(S\setminus\\{u,v\\}))\cup\\{x,y\\}$. Since $d(x)\leq\Delta(G)=3$
and $d(y)\leq\Delta(G)=3$, we can conclude that $x$ and $y$ do not have
neighbors in $S^{\prime}$ which means $S^{\prime}$ is an independent set, a
contradiction. ($\Box$)

Now, among all proper $(\Delta(G)-1)$-coloring of $G^{\prime}$ let
$c^{\prime}$ be selected such that its smallest color class is as small as
possible. We may assume without loss of generality that $C^{\prime}_{1}$ is
such a color class.

######  Claim 8.

If $x\in C^{\prime}_{1}$, then $|N_{G}(x)\cap S|\leq 2$.

Indeed, otherwise there exists $i\in[\Delta(G)-1]\setminus\\{1\\}$ such that
$N_{G}(x)\cap C^{\prime}_{i}=\emptyset$. So, we can color $x$ by $i$ to obtain
a proper $(\Delta(G)-1)$-coloring of $G^{\prime}$. In this new coloring, the
number of vertices colored by $1$ is $|C^{\prime}_{1}|-1$. Therefore, we can
deduce that $|C^{\prime}_{1}|$ is not a smallest possible color class, a
contradiction. ($\Box$)

Now, let $P$ be the connected component of $G[S\cup C^{\prime}_{1}]$
containing $u$ and $v$. By Claims 2 and 5, $d_{P}(u)=d_{P}(v)=2$. By Claims 3
and 5, $d_{P}(w)\leq 2$, for each $w\in S\cap V(P)$. By Claim 8, $d_{P}(x)\leq
2$, for each $x\in C^{\prime}_{1}\cap V(P)$. Hence, $\Delta(P)=2$, which means
$P$ is a path or a cycle. Now, we have two cases.

Case 1. $P$ is a path.
Let $P=x_{k},\ldots,x_{1},u,v,y_{1},\ldots,y_{l}$ and let
$X_{1}=\\{x_{1},x_{3},\ldots\\}$ and $X_{2}=\\{x_{2},x_{4},\ldots\\}$.
Clearly, $X_{1}\subseteq C^{\prime}_{1}$ and $X_{2}\subseteq S$. Now, we can
color all vertices of $X_{2}\cup\\{u\\}$ by $1$ to obtain a proper
$(\Delta(G)-1)$-coloring of $G\setminus S^{\prime}$, where
$S^{\prime}=(S\setminus(X_{2}\cup\\{u\\}))\cup X_{1}$. Clearly,
$|S^{\prime}|\leq|S|$, and $S^{\prime}$ is an independent set since $X_{1}$
and $S\setminus(X_{2}\cup\\{u\\})$ are independent sets and by Claim 8 there
is no edge between $X_{1}$ and $S^{\prime}\setminus X_{1}$. Hence,
$0=|E(G[S^{\prime}])|<|E(G[S])|=1$, a contradiction.

Case 2. $P$ is a cycle.
It is easy to see that $P$ is an odd cycle. According to Claim 6, $H_{u}\cong
K_{\Delta(G)}$ or $H_{u}\cong C_{2a+1}$ for a natural number $a$. By Claim 7,
since $u$ and $v$ do not have a common neighbor, we can conclude that
$|V(P)|>3$. Now, we have two subcases.

Subcase 2.1. $H_{u}\cong K_{\Delta(G)}$.
In this subcase, $\Delta(G)>3$, or $\Delta(G)=3$ and $H_{u}\cong K_{3}$. Let
$P=u,v,x_{1},x_{2},\ldots,x_{2k-1},u$, where $k$ is a natural number greater
than 1, and let $X_{1}=\\{x_{1},x_{3},\ldots,x_{2k-1}\\}$ and
$X_{2}=\\{x_{2},x_{4},\ldots,x_{2k-2}\\}$. Clearly, $X_{1}\subseteq
C^{\prime}_{1}$ and $X_{2}\subseteq S$. Since $X_{2}\cup\\{v\\}$ is an
independent set, we can color all vertices of $X_{2}\cup\\{v\\}$ by $1$ to
obtain a proper $(\Delta(G)-1)$-coloring of $G\setminus S^{\prime}$, where
$S^{\prime}=(S\setminus(X_{2}\cup\\{v\\}))\cup X_{1}$. Clearly,
$|S^{\prime}|=|S|-|X_{2}\cup\\{v\\}|+|X_{1}|=|S|$. It is easy to see that
there is only one edge $ux_{2k-1}$ between $X_{1}$ and
$S\setminus(X_{2}\cup\\{v\\})$. In addition, $X_{1}$ and
$S\setminus(X_{2}\cup\\{v\\})$ are independent sets; hence,
$E(G[S^{\prime}])=\\{ux_{2k-1}\\}$. Clearly, $x_{2k-1}\in V(H_{u})$. Since
$H_{u}\cong K_{\Delta(G)}$ and $\Delta(G)\geq 3$, there is a vertex in
$V(H_{u})$ adjacent to both $u$ and $x_{2k-1}$ which means $N_{G}(u)\cap
N_{G}(x_{2k-1})\neq\emptyset$. Since $|S^{\prime}|=|S|$ and
$|E(G[S^{\prime}])|=1$, by the same method as in the proof of Claim 7, we can
show that $N_{G}(u)\cap N_{G}(x_{2k-1})=\emptyset$, a contradiction.

$u$$v$$x_{2k-1}$$x_{2k-2}$$t$$x_{1}$$x_{2}$$P$$H_{u}$$\ldots$$\vdots$ Figure
1: $\Delta(G)=3$, and $P$ and $H_{u}$ are odd cycles

Subcase 2.2. $H_{u}=C_{2r+1}$ for some $r>1$.
In this subcase $\Delta(G)=3$. With no loss of generality, assume that
$c^{\prime}$, the coloring described above, is a coloring where $|V(P)|$ is as
large as possible. We have $\\{u,x_{2k-1}\\}\subseteq V(P)\cap V(H_{u})$, cf.
Fig. 1. Moreover, it is also easy to see that $V(P)\cap
V(H_{u})=\\{u,x_{2k-1}\\}$ because if $w\in(V(P)\cap
V(H_{u}))\setminus\\{u,x_{2k-1}\\}$, then $c^{\prime}(w)=1$ and $w$ has two
neighbors with color $2$ and $w$ has two neighbors in $S$ which contradicts
$\Delta(G)=3$. Now, let $t$ be the other neighbor of $u$ in $H_{u}$ in the
proper $2$-coloring of $G^{\prime}$. Clearly, $c^{\prime}(x_{2k-1})=1$ and
$c^{\prime}(t)=2$. Let $c^{\prime\prime}$ be a proper $(\Delta(G)-1)$-coloring
($2$-coloring) of $G^{\prime}$ switching the color of vertices in $H_{u}-u$
(keeping the color of other vertices the same as $c^{\prime}$). Claim 8 still
holds for $c^{\prime\prime}$ because
$|C^{\prime}_{1}|=|C^{\prime\prime}_{1}|$. Now, let $P^{\prime}$ be the
connected component of $G[S\cup C^{\prime\prime}_{1}]$ containing $u$ and $v$.
If $P^{\prime}$ is a path, then simillar to the Case 1, we are done.
Otherwise, $P^{\prime}$ is a cycle. Obviously,
$V(P)\setminus\\{x_{2k-1}\\}\subseteq V(P^{\prime})$ and $t\in V(P^{\prime})$,
and since $|V(P)|\geq|V(P^{\prime})|$ and $P^{\prime}$ is a cycle, we can
conclude that $x_{2k-2}t\in E(G)$. Now, one can color $u$ and $x_{2k-2}$ by
$2$ to obtain a proper $2$-coloring of $G\setminus S^{\prime\prime}$, where
$S^{\prime\prime}=(S\setminus\\{u,x_{2k-2}\\})\cup\\{x_{2k-1}\\}$. Since
$|S^{\prime\prime}|<|S|$, we have a final contradiction. ∎

##  3 Discussion on graphs with smaller chromatic number

In the next remark we demonstrate that Theorem 1.1 does not extend to the case
when $\chi(G)\leq\frac{\Delta(G)}{2}$.

######  Remark 3.1.

Let $G_{n,k}$, $n\geq 2$, $k\geq 3$, be the graph obtained from the cycle
$C_{2n}$ as follows. For each vertex $u$ of the cycle take two disjoint
complete graphs $K_{k}$, select a fixed vertex in each of them, and identify
the two vertices with $u$. Then it is straightforward to verify that
$\chi(G_{n,k})=k$, $\Delta(G_{n,k})=2k$, ${\rm vs}_{\chi}(G_{n,k})=2n$, and
${\rm ivs}_{\chi}(G_{n,k})=3n$.

In view of Remark 3.1 we pose the next problem for which we feel the answer is
yes.

######  Problem 3.2.

Is it true that if $G$ is a graph with $\chi(G)\geq\frac{\Delta(G)}{2}+1$,
then ${\rm vs}_{\chi}(G)={\rm ivs}_{\chi}(G)$?

Remark 3.1 demonstrates that in Problem 3.2 the assumption
$\chi(G)\geq\frac{\Delta(G)}{2}+1$ cannot be weakened to
$\chi(G)\geq\frac{\Delta(G)}{2}$. We next demonstrate that furthermore the
assumption cannot be weakened to $\chi(G)\geq\frac{\Delta(G)+1}{2}$.

Let $k\geq 3$ be an odd positive number and let $H_{k}$ be the graph obtained
from two copies of $K_{\frac{k+1}{2}}$ by identifying a vertex from each of
the copies. Let $G_{k}$ be the graph obtained from two disjoint copies of
$H_{k}$ by adding an edge between the maximum degree vertices in the two
copies. Then we have $\Delta(G_{k})=k$,
$\chi(G_{k})=\frac{k+1}{2}=\frac{\Delta(G_{k})+1}{2}$, and $2={\rm
vs}_{\chi}(G_{k})<{\rm ivs}_{\chi}(G_{k})=3$.

Using a similar construction we next show that the ratio

$\frac{{\rm ivs}_{\chi}(G_{r})}{{\rm vs}_{\chi}(G_{r})}$

can be arbitrary large. Let $k$ be a natural number and let $H^{\prime}_{k}$
be the graph obtained from $k$ copies of $K_{3}$ by identifying a vertex from
each of the copies. Let further $G^{\prime}_{k}$ be the graph obtained from
two disjoint copies of $H^{\prime}_{k}$ by adding an edge between the two
vertices of maximum degrees in the copies. Then $\chi(G_{k}^{\prime})=3$,
${\rm vs}_{\chi}(G_{k}^{\prime})=2$, and ${\rm
ivs}_{\chi}(G_{k}^{\prime})=k+1$. This example shows that for any natural
number $r$, there exists a graph $G_{r}$ such that $\frac{{\rm
ivs}_{\chi}(G_{r})}{{\rm vs}_{\chi}(G_{r})}>r$.

##  4 Relation with ${\rm es}_{\chi}$ and a Nordhaus-Gaddum-type result

In this final section we first prove that if the chromatic number of a graph
is large, then its chromatic vertex stability number is equal to its chromatic
edge stability number. More precisely, we have the following result.

######  Theorem 4.1.

If $G$ is a graph with $\chi(G)>\frac{\Delta(G)}{2}+1$, then ${\rm
vs}_{\chi}(G)={\rm es}_{\chi}(G)$.

###### Proof.

Clearly, ${\rm vs}_{\chi}(G)\leq{\rm es}_{\chi}(G)$, hence we need to show
that ${\rm vs}_{\chi}(G)\geq{\rm es}_{\chi}(G)$. We proceed by induction on
${\rm vs}_{\chi}(G)$.

The base case is when ${\rm vs}_{\chi}(G)=1$. Let $u$ be a vertex such that
$\chi(G-u)=\chi(G)-1$. Let $c$ be a proper $(\chi(G)-1)$-coloring of $G-u$.
There is a color class $C_{i}$ such that $|N_{G}(u)\cap C_{i}|\leq 1$, because
otherwise $d_{G}(u)\geq 2(\chi(G)-1)>\Delta(G)$, a contradiction. If
$N_{G}(u)\cap C_{i}=\emptyset$ for some color class $C_{i}$, then we can color
$u$ by $i$ to obtain a proper $(\chi(G)-1)$-coloring of $G$, a contradiction.
Otherwise, $|N_{G}(u)\cap C_{i}|=1$ and let $e$ be the edge between $u$ and
$C_{i}$. So, we can color $u$ by $i$ to obtain a proper $(\chi(G)-1)$-coloring
of $G-e$. Thus, $1={\rm vs}_{\chi}(G)\geq{\rm es}_{\chi}(G)$, and the base
case is proved.

Now, suppose ${\rm vs}_{\chi}(G)=k>1$ and let
$S=\\{u_{1},\ldots,u_{k}\\}\subseteq V(G)$ be a subset such that
$\chi(G\setminus S)=\chi(G)-1$. Let $c$ be a proper $(\chi(G)-1)$-coloring of
$G\setminus S$. Similar to the base case of the induction, there is a color
class $C_{i}$ such that $|N_{G}(u_{1})\cap C_{i}|=1$. Now, suppose that $e$ is
the edge between $u_{1}$ and $C_{i}$ and define $G^{\prime}=G-e$. Clearly,
${\rm vs}_{\chi}(G^{\prime})=k-1$. By induction, ${\rm
vs}_{\chi}(G^{\prime})\geq{\rm es}_{\chi}(G^{\prime})$ and thus ${\rm
es}_{\chi}(G)\leq{\rm es}_{\chi}(G^{\prime})+|\\{e\\}|\leq k={\rm
vs}_{\chi}(G)$, which proves the induction step. ∎

The lower bound given in Theorem 4.1 is sharp. Let $k$ be an even positive
integer and $G$ be the graph obtained from two copies of $K_{\frac{k}{2}+1}$
which have only one vertex in common. Note that $\Delta(G)=k$,
$\chi(G)=\frac{k}{2}+1=\frac{\Delta(G)}{2}+1$, and ${\rm vs}_{\chi}(G)=1$,
${\rm es}_{\chi}(G)=2$.

To conclude the paper, we give the following Nordhaus-Gaddum-type result.

######  Proposition 4.2.

If $G$ is a not edgeless and not complete graph, then ${\rm vs}_{\chi}(G)+{\rm
vs}_{\chi}(\overline{G})\leq n(G)+1$.

###### Proof.

Considering a proper $\chi(G)$-coloring of $G$ and its smallest color class,
we see that ${\rm vs}_{\chi}(G)\leq\frac{n(G)}{\chi(G)}$. The clasical
Nordhaus-Gaddum inequalities [11] assert that $\chi(G)+\chi(\overline{G})\leq
n(G)+1$ and $\chi(G)\chi(\overline{G})\geq n(G)$, hence we can estimate as
follows:

${\rm vs}_{\chi}(G)+{\rm
vs}_{\chi}(\overline{G})\leq\frac{n(G)}{\chi(G)}+\frac{n(G)}{\chi(\overline{G})}=n(G)\frac{\chi(G)+\chi(\overline{G})}{\chi(G)\chi(\overline{G})}\leq
n(G)\frac{n(G)+1}{n(G)}=n(G)+1$

and we are done. ∎

## Acknowledgments

We thank one of the reviewers for careful reading of the article and for many
very helpful comments.

The research of the first author was supported by grant number G981202 from
Sharif University of Technology. Sandi Klavžar acknowledges the financial
support from the Slovenian Research Agency (research core funding P1-0297 and
projects N1-0095, J1-1693, J1-2452).

## References

  * [1] S. Akbari, S. Klavžar, N. Movarraei, M. Nahvi, Nordhaus-Gaddum and other bounds for the chromatic edge-stability number, European J. Combin. 84 (2020) 103042, 8 pp.
  * [2] S. Akbari, A. Beikmohammadi, B. Brešar, T. Dravec, M. Mahdi Habibollahi, N. Movarraei, On the chromatic edge stability index of graphs, arXiv:2108.10657 [math.CO]; also: European J. Combin., to appear.
  * [3] S. Alikhani, M. R. Piri, On the edge chromatic vertex stability number of graphs, arXiv:2004.10551 [math.CO].
  * [4] S. Alikhani, S. Soltani, Stabilizing the distinguishing number of a graph, Comm. Algebra 46 (2018) 5460–5468.
  * [5] S. Arumugam, I. Sahul Hamid, A. Muthukamatchi, Independent domination and graph colorings, Discrete mathematics, 195–203, Ramanujan Math. Soc. Lect. Notes Ser., 7, Ramanujan Math. Soc., Mysore, 2008.
  * [6] D. Bauer, F. Harary, J. Nieminen, C. L. Suffel, Domination alteration sets in graphs, Discrete Math. 47 (1983) 153-161.
  * [7] B. Brešar, S. Klavžar, N. Movarraei, Critical graphs for the chromatic edge-stability number, Discrete Math. 343 (2020) 111845, 7 pp.
  * [8] S. Huang, S. Klavžar, H. Lei, X. Lian, Y. Shi, Some extremal results on the chromatic-stability index, arXiv:2007.15368 [math.CO].
  * [9] A. Kemnitz, M. Marangio, On the-edge stability number of graphs, Discuss. Math. Graph Theory (2021) doi.org/10.7151/dmgt.2255.
  * [10] A. Kemnitz, M. Marangio, N. Movarraei, On the chromatic edge stability number of graphs, Graphs Combin. 34 (2018) 1539–1551.
  * [11] E.A. Nordhaus, J. Gaddum, On complementary graphs, Amer. Math. Monthly 63 (1956) 175–177.
  * [12] W. Staton, Edge deletions and the chromatic number, Ars Combin. 10 (1980) 103–106.
  * [13] D.B. West, Introduction to Graph Theory, 2nd ed., Prentice-Hall, NJ, 2001.


Sadra<EMAIL_ADDRESS>Fatma<EMAIL_ADDRESS>KUIS AI
Center
Koç University
Istanbul, Turkey DepthP+P

# DepthP+P: Metric Accurate Monocular Depth Estimation using Planar and
Parallax

###### Abstract

Current self-supervised monocular depth estimation methods are mostly based on
estimating a rigid-body motion representing camera motion. These methods
suffer from the well-known scale ambiguity problem in their predictions. We
propose DepthP+P, a method that learns to estimate outputs in metric scale by
following the traditional planar parallax paradigm. We first align the two
frames using a common ground plane which removes the effect of the rotation
component in the camera motion. With two neural networks, we predict the depth
and the camera translation, which is easier to predict alone compared to
predicting it together with rotation. By assuming a known camera height, we
can then calculate the induced 2D image motion of a 3D point and use it for
reconstructing the target image in a self-supervised monocular approach. We
perform experiments on the KITTI driving dataset and show that the planar
parallax approach, which only needs to predict camera translation, can be a
metrically accurate alternative to the current methods that rely on estimating
6DoF camera motion.

##  1 Introduction

Understanding the 3D structure of a scene is fairly easy for human beings. We
can easily reason about our surroundings and decompose them into different
objects. Having this ability is crucial for autonomous vehicles to be able to
drive in different environments. Training deep networks for estimating depth
has proven successful in computer vision research. However, many such methods
are supervised and require ground truth depth which is costly to achieve.
Another line of work uses a stereo setup that must be carefully calibrated.
Both of these approaches cannot use the vast amount of unlabeled videos that
are easily available for training. On the other hand, self-supervised
monocular depth estimation methods that do not rely on stereo supervision do
not suffer from these limitations and, in practice, have been closing the gap
with their supervised or stereo counterparts.

Current self-supervised monocular depth estimation approaches all follow the
same basic idea proposed in [Zhou et al.(2017)Zhou, Brown, Snavely, and Lowe].
They use a pose network to estimate the ego-motion between a source frame and
the target frame and a depth network to estimate the depth of the target
image. These estimations can then be used to sample pixels from the source
image to synthesize the target frame. The difference between the target frame
and the synthesized can be used as the source of supervision for training the
networks. In this paper, we propose another approach to synthesize the target
image. Our approach, DepthP+P, illustrated in Fig. 1, uses the traditional
planar parallax formulation [Sawhney(1994), Irani and Anandan(1996)], which
decomposes the motion into a planar homography and a residual parallax.
Consider a plane in the scene and its motion represented by a homography from
the source to the target image. By first warping the source image according to
this homography, the motion of the plane is canceled. Then the residual image
motion depends on two factors: (1) the deviations of the scene structure from
the plane, i.e. the depth of points and their perpendicular distance to the
plane and (2) only the translational motion of the camera. Autonomous driving
is a perfect use case for this approach because there is typically a planar
surface in front of the vehicle, i.e. the road. However, it is important to
note that the plane in the planar parallax formulation does not necessarily
have to be a real plane and can also be a virtual plane, but choosing the road
as the planar surface makes it easier to implement in practice. Moreover, our
approach does not rely on the availability of a plane to predict depth during
inference.

Figure 1: Overview of our Approach. Using the source image $\mathbf{I}_{s}$
and the target image $\mathbf{I}_{t}$, we first calculate the homography
$\mathbf{H}$ that aligns the road plane across these two images. We then warp
$\mathbf{I}_{s}$ according to $\mathbf{H}$ and obtain the aligned image
$\mathbf{I}_{w}$. The aligned image $\mathbf{I}_{w}$ and the target image
$\mathbf{I}_{t}$ are input to the pose network which estimates the camera
translation $\mathbf{t}$ only. The depth network takes the $\mathbf{I}_{t}$
and produces a metric accurate depth map $\hat{\mathbf{D}}$.

In this approach, we first align the road plane between the source and target
images. This is achieved by calculating the homography between the road
regions in two frames and then warping the source frame according to the
homography to obtain the aligned image. By doing so, the road regions in the
aligned image and target image match. The residual motion between the aligned
image and the target image can be explained as follows: We first estimate the
depth of each pixel with a monocular depth network and back-project them into
3D. Then using a known camera height, we can calculate the perpendicular
distance of each point to the road. In addition, we estimate the translation
between the camera origins. Note that this is different from the typical
monocular depth approach [Zhou et al.(2017)Zhou, Brown, Snavely, and Lowe,
Godard et al.(2019)Godard, Mac Aodha, Firman, and Brostow] which needs to
estimate both the rotation and translation components. Finally, the target
image can be synthesized from the aligned image using the calculated residual
parallax as shown in Fig. 2.

The planar parallax approach for self-supervised monocular estimations has a
number of advantages over the previous paradigm. Firstly, it is much easier to
optimize because it removes the ambiguities associated with predicting
rotational camera motion [Irani et al.(2002)Irani, Anandan, and Cohen].
Secondly, it can produce metric accurate outputs. Previous monocular depth
methods can estimate depth and motion up to a scale. Typically, during
inference, ground truth depth data is used to scale the predicted depth values
such that the median of the predicted depth is equal to that of ground truth
depth [Zhou et al.(2017)Zhou, Brown, Snavely, and Lowe]. Our approach is able
to predict metric accurate depth without needing ground truth depth data by
only assuming a known camera height.

##  2 Related Work

###  2.1 Self-Supervised Monocular Depth

View Synthesis: Garg et al. [Garg et al.(2016)Garg, Bg, Carneiro, and Reid]
were the first to propose a method that uses view synthesis as an objective
for depth estimation from single images. Monodepth [Godard et al.(2017)Godard,
Mac Aodha, and Brostow] uses Spatial Transformer Networks (STNs) [Jaderberg et
al.(2015)Jaderberg, Simonyan, Zisserman, and Kavukcuoglu] to synthesize the
images in a fully-differentiable way. SfmLearner [Zhou et al.(2017)Zhou,
Brown, Snavely, and Lowe] generalizes view synthesis to temporally consecutive
images by using another network to predict the relative pose between them.
Zhan et al. [Zhan et al.(2018)Zhan, Garg, Saroj Weerasekera, Li, Agarwal, and
Reid] use stereo sequences to perform view synthesis using temporally
consecutive pairs as well as the left-right pairs, enabling them to benefit
from both monocular and stereo supervision. In addition to image
reconstruction, they also use feature reconstruction as supervision.
Similarly, by going beyond pixel-wise reconstruction error, Mahjourian et al.
[Mahjourian et al.(2018)Mahjourian, Wicke, and Angelova] propose to use a 3D
point cloud alignment loss to enforce the estimated point clouds and the
camera pose to be consistent temporally. Wang et al. [Wang et al.(2018)Wang,
Miguel Buenaposada, Zhu, and Lucey] use direct visual odometry in a
differentiable manner to solve for ego-motion using the estimated depth.

In addition to depth and camera pose, several methods estimate optical flow
for residual motion. After predicting the camera motion, GeoNet [Yin and
Shi(2018)] estimates the remaining object motion using optical flow. In order
to prevent the errors of camera pose or depth predictions from propagating to
flow estimations, DF-Net [Zou et al.(2018)Zou, Luo, and Huang] enforces
consistency between optical flow and the flow induced by the depth and pose
predictions. GLNet [Chen et al.(2019)Chen, Schmid, and Sminchisescu] uses
epipolar constraint for optical flow, along with other geometric constraints,
further improving the performance. EPC++ [Luo et al.(2019)Luo, Yang, Wang,
Wang, Xu, Nevatia, and Yuille] proposes a holistic 3D motion parser that uses
predicted depth, pose, and optical flow to estimate segmentation masks for
dynamic objects and their motion as well as background motion. Ranjan et al.
[Ranjan et al.(2019)Ranjan, Jampani, Balles, Kim, Sun, Wulff, and Black]
jointly train networks for depth, pose, optical flow, and motion segmentation
so that they can use geometric constraints on the static regions and generic
optical flow on moving objects. MonoDepthSeg [Safadoust and Güney(2021)]
proposes to jointly estimate depth, independently moving regions, and their
motion with an efficient architecture.

Some approaches keep the original framework with a depth and a pose network
but improve the performance with better loss functions, improved network
architectures, and innovative design choices. When estimating depth at
multiple scales, Monodepth2 [Godard et al.(2019)Godard, Mac Aodha, Firman, and
Brostow] proposes to first upsample the estimated low-scale depths to the
input image size and then calculate the photometric loss at that scale.
Monodepth2 also proposes to calculate the minimum of reprojection errors per
pixel instead of averaging them when synthesizing the target image from
multiple views to prevent blurry depth estimations. PackNet [Guizilini et
al.(2020)Guizilini, Ambrus, Pillai, Raventos, and Gaidon] changes the
architecture of the depth network and uses 3D convolutions to learn to
preserve spatial information using symmetrical 3D packing and unpacking blocks
for predicting depth.

Scale Ambiguity: Self-supervised monocular depth estimation models suffer from
the scale ambiguity problem, and the depth and pose outputs of such models are
in an unknown scale. The median scaling technique used by many previous
methods does not actually solve this problem because it relies on ground truth
depth data during inference which is not always easily available. Bian et al.
[Bian et al.(2019)Bian, Li, Wang, Zhan, Shen, Cheng, and Reid] introduce a
loss to minimize normalized differences of depth maps across the entire
sequence. This makes the estimations globally scale-consistent. However,
although this means that the predictions are at the same scale, that specific
scale is still unknown, and the median scaling is still required during
evaluation.

There are a number of monocular methods that can output depth estimations in
absolute scale. Roussel et al. [Roussel et al.(2019)Roussel, Eycken, and
Tuytelaars] use a network that was pre-trained with stereo pairs on a dataset
and finetunes it on another dataset while maintaining the metric scale.
Guizilini et al. [Guizilini et al.(2020)Guizilini, Ambrus, Pillai, Raventos,
and Gaidon] propose a version of their PackNet that uses ground truth camera
velocity and the timestamps of images to enforce the estimations to be
metrically accurate. Bartoccioni et al. [Bartoccioni et al.(2021)Bartoccioni,
Zablocki, Pérez, Cord, and Alahari] supervise their depth predictions with a
sparse LiDAR. However, all of these approaches rely on ground truth data from
extra sensors during training.

There are a few other methods that do not require additional supervision and
only use the camera height to achieve depth estimations in metric units
similar to the proposed method. DNet [Xue et al.(2020)Xue, Zhuo, Huang, Fu,
Wu, and Ang] estimates the ground plane during inference and, using the real
height of the camera, recovers the scale of the predictions. However, it needs
a ground plane to be visible during the test time. In other words, they do not
train their depth outputs to be in absolute scale. Rather, they recover the
scale of the estimations with another module during test time. Wagstaff and
Kelly [Wagstaff and Kelly(2021)] train a network that learns the metric scale
during training using camera height. They introduce a plane segmentation
network and propose a three-staged training procedure for training the depth
estimation model in metric scale. First, they train an unscaled depth network
and then use it to train the plane segmentation network. Finally, they train a
new metrically accurate depth network using the pre-trained plane segmentation
network. Similar to [Wagstaff and Kelly(2021)], we also learn the metric scale
during training, but we do not need a multi-stage process, nor do we rely on
the existence of a ground plane during inference, differently from previous
work [Xue et al.(2020)Xue, Zhuo, Huang, Fu, Wu, and Ang].

###  2.2 Planar Parallax

The Planar Parallax paradigm, also called Plane + Parallax (P+P), has been
used to understand the 3D structure of a scene from multiple images by
decomposing the motion into a planar homography and a residual parallax.
Sawhney [Sawhney(1994)] proposes a formulation for the residual parallax that
uses depth and distance to the plane. Irani et al. [Irani and Anandan(1996)]
use this formulation to derive a rigidity constraint between pairs of points
over multiple images. Irani et al. [Irani et al.(1998)Irani, Anandan, and
Weinshall] derive trifocal constraints and use them to propose a simple method
for new view synthesis. In a follow-up work [Irani et al.(2002)Irani, Anandan,
and Cohen], they extend the planar parallax method to more than two
uncalibrated frames.

More recently, MR-Flow [Wulff et al.(2017)Wulff, Sevilla-Lara, and Black] uses
P+P to refine the optical flow estimations with rigidity constraints. Chaney
et al. [Chaney et al.(2019)Chaney, Zhu, and Daniilidis] use P+P to estimate
the height of points in the scene with event-based cameras. We propose a
method to use the P+P formulation within the view synthesis framework for
self-supervised monocular depth estimation.

##  3 Methodology

Figure 2: Visualization of the Planar Parallax. Left: The 3D point
$\mathbf{x}$ is projected to points $\mathbf{p}$ and $\mathbf{p}^{\prime}$ on
the target image $\mathbf{I}_{t}$ and the source image $\mathbf{I}_{s}$
respectively. Using the homography $\mathbf{H}$ induced by the plane
$\boldsymbol{\Pi}$, the point $\mathbf{p}^{\prime}$ will be transformed to
point $\mathbf{p}_{w}$ on the target image. Right: Calculating $\mathbf{h}$,
distance of $\mathbf{x}$ to the $\boldsymbol{\Pi}$ using the camera height
$\mathbf{d}_{c}$ and the normal vector $\mathbf{N}$ of the plane. $\mathbf{C}$
and $\mathbf{C}^{\prime}$ are the camera centers of $\mathbf{I}_{t}$ and
$\mathbf{I}_{s}$ and $\mathbf{Z}$ is the depth of the point.

Despite the success of current self-supervised monocular depth estimation
approaches, they suffer from scale ambiguity. i.e. the estimated depth values
are in an unknown scale. Therefore, in order to evaluate and compare these
methods, they are usually normalized using the median scaling approach [Zhou
et al.(2017)Zhou, Brown, Snavely, and Lowe]. Here, we propose an approach that
predicts depth maps in metric scale without using any ground truth depth
supervision.

###  3.1 DepthP+P

Our approach is based on the Planar Parallax decomposition which has been
studied in detail before [Sawhney(1994), Irani and Anandan(1996)]. We first
introduce it here to establish our notation and then build our method to
predict depth following that notation.

Notation: Let $\boldsymbol{\Pi}$ be a 3D plane and $\mathbf{H}$ be the
homography aligning $\boldsymbol{\Pi}$ between the target image
$\mathbf{I}_{t}$ and the source image $\mathbf{I}_{s}$. Let $\mathbf{p}$ and
$\mathbf{p}^{\prime}$ be the images of the 3D point
$\mathbf{x}=[\mathbf{X},\mathbf{Y},\mathbf{Z}]^{T}$ on the $\mathbf{I}_{t}$
and $\mathbf{I}_{s}$ respectively. As shown on the left in Fig. 2, we can warp
$\mathbf{p}^{\prime}$ by the homography $\mathbf{H}$ and obtain the image
point $\mathbf{p}_{w}$:

$\mathbf{p}_{w}\sim\mathbf{H}\mathbf{p}^{\prime}$ (1)

where we omit the conversion to the homogenous coordinates. Note that by
warping the source image $\mathbf{I}_{s}$, we obtain the aligned image
$\mathbf{I}_{w}$ such that the plane $\boldsymbol{\Pi}$ matches between them.
The displacement between $\mathbf{p}_{w}$ and $\mathbf{p}$ can be computed as
follows:

$\mathbf{p}_{w}-\mathbf{p}=\frac{\boldsymbol{\gamma}}{\mathbf{d}_{c}-\boldsymbol{\gamma}\mathbf{t}_{z}}(\mathbf{t}_{z}\mathbf{p}-\mathbf{K}\mathbf{t})$
(2)

where $\mathbf{K}$ is the camera intrinsic,
$\mathbf{t}=[\mathbf{t}_{x},\mathbf{t}_{y},\mathbf{t}_{z}]^{T}$ is the
translation vector between the $\mathbf{I}_{t}$ and $\mathbf{I}_{s}$, and
$\mathbf{d}_{c}$ is the distance between the camera for the source view to the
plane $\boldsymbol{\Pi}$. The structure is represented by
$\boldsymbol{\gamma}=\frac{\mathbf{h}}{\mathbf{Z}}$ where $\mathbf{h}$ is the
distance of $\mathbf{x}$ to $\boldsymbol{\Pi}$. Note that when $\mathbf{x}$
lies on the plane $\boldsymbol{\Pi}$, i.e. $\mathbf{h}=0$, we will have
$\mathbf{p}_{w}=\mathbf{p}$.

DepthP+P: Following the typical self-supervised monocular depth approach, our
framework has two networks, one for estimating depth and another for
estimating the translation between frames. Note that, unlike other methods, we
do not need to estimate the rotation between the two views. Precisely, our
pose network takes the source and images $\mathbf{I}_{s},\mathbf{I}_{w}$ and
outputs the translation vector $\mathbf{t}$. The depth network takes the
target image $\mathbf{I}_{t}$ and outputs the depth map $\hat{\mathbf{D}}$ for
$\mathbf{I}_{t}$.

For every pixel $\mathbf{p}=[x,y]$, let $\hat{\mathbf{D}}{(\mathbf{p})}$
denote its estimated depth. We backproject $\mathbf{p}$ using the camera
intrinsics and the estimated depth to obtain the corresponding 3D point
$\hat{\mathbf{x}}$ in the camera coordinate system as follows:

$\hat{\mathbf{x}}=\hat{\mathbf{D}}{(\mathbf{p})}~{}\mathbf{K}^{-1}~{}[x,y,1]^{T}.$
(3)

Therefore, as demonstrated on the right in Fig. 2, we have the following:

$\hat{\mathbf{h}}=\mathbf{d}_{c}-\mathbf{N}^{T}\hat{\mathbf{x}},\quad\quad\hat{\boldsymbol{\gamma}}=\frac{\hat{\mathbf{h}}}{\hat{\mathbf{D}}{(\mathbf{p})}}$
(4)

where $\mathbf{N}$ is the normal vector of the plane $\boldsymbol{\Pi}$,
$\hat{\mathbf{h}}$ is the estimated distance of the point $\hat{\mathbf{x}}$
to the plane $\boldsymbol{\Pi}$ and $\hat{\boldsymbol{\gamma}}$ is our
estimate of the structure variable $\boldsymbol{\gamma}$. As a result, we
obtain all the parameters required to use Eq. (2) to reconstruct the target
image $\mathbf{I}_{t}$ by warping the aligned image $\mathbf{I}_{w}$ resulting
in $\hat{\mathbf{I}}_{w}$. In other words, for each pixel $\mathbf{p}$ on the
$\mathbf{I}_{t}$, we calculate the $\mathbf{p}_{w}$ using (2) according to the
depth and translation predicted by our two networks and then inverse warp
$\mathbf{I}_{w}$ and obtain $\hat{\mathbf{I}}_{w}$ to reconstruct
$\mathbf{I}_{t}$:

$\mathbf{I}_{t}(\mathbf{p})\approx\hat{\mathbf{I}}_{w}(\mathbf{p})=\mathbf{I}_{w}(\mathbf{p}_{w})$
(5)

We minimize the difference between $\mathbf{I}_{t}$ and $\hat{\mathbf{I}}_{w}$
for supervision as explained in the Section 3.2.

In order to obtain the aligned images, we perform a pre-processing step on the
dataset. We calculate a homography for every consecutive frame by using the
road as the plane $\boldsymbol{\Pi}$ and warp the frames according to the
calculated homographies. In other words, we calculate a homography
$\mathbf{H}$ for every target image $\mathbf{I}_{t}$ and source image
$\mathbf{I}_{s}$ pair, and then warp $\mathbf{I}_{s}$ according to
$\mathbf{H}$ to obtain the warped source image $\mathbf{I}_{w}$. We explain
the details of this pre-processing step in Section 4.1.

###  3.2 Self-Supervised Training Loss

In our approach, we define our photometric loss function as the linear
combination of the L1 distance and the structural similarity (SSIM) [Wang et
al.(2004)Wang, Bovik, Sheikh, and Simoncelli] to minimize the difference
between the target image $\mathbf{I}_{t}$ and the reconstructed image
$\hat{\mathbf{I}}_{w}$. Our photometric loss is therefore defined as follows:

$\mathcal{L}_{\textrm{photo}}(\mathbf{p})=\left(1-\alpha\right)\lvert\mathbf{I}_{t}(\mathbf{p})-\hat{\mathbf{I}}_{w}(\mathbf{p})\rvert+\frac{\alpha}{2}\left(1-\textrm{SSIM}(\mathbf{I}_{t},\hat{\mathbf{I}}_{w})(\mathbf{p})\right)$
(6)

where we set $\alpha=0.85$. Note that for every target image we consider two
aligned images. One from warping the previous frame, and one from warping the
next frame. We use the per-pixel minimum reprojection error introduced in
[Godard et al.(2019)Godard, Mac Aodha, Firman, and Brostow] and calculate the
minimum of the $\mathcal{L}_{photo}$ for each pixel across the previous and
next aligned images. We also define $\mathcal{L}_{smooth}$ as an edge-aware
smoothness loss over the mean-normalized inverse depth estimates [Wang et
al.(2018)Wang, Miguel Buenaposada, Zhu, and Lucey] to encourage the depth
predictions to be locally smooth. Our total loss function is a combination of
$\mathcal{L}_{smooth}$ and $\mathcal{L}_{photo}$ averaged over all $N$ pixels:

$\mathcal{L}=\frac{1}{N}\sum_{\mathbf{p}}\lambda~{}\mathcal{L}_{smooth}(\mathbf{p})+\min_{w}(\mathcal{L}_{photo}(\mathbf{p}))$
(7)

where $\displaystyle\min_{w}$ calculates the minimum over the previous and
next aligned frames and $\lambda$ is a hyperparameter controlling the effect
of the loss terms.

###  3.3 Network Architecture

Our depth network is based on the U-Net architecture [Ronneberger et
al.(2015)Ronneberger, Fischer, and Brox]. We use a ResNet [He et al.(2016)He,
Zhang, Ren, and Sun] pre-trained on the ImageNet [Russakovsky et
al.(2015)Russakovsky, Deng, Su, Krause, Satheesh, Ma, Huang, Karpathy, Khosla,
Bernstein, et al.] as the encoder for our depth network, and for the decoder
we use the architecture similar to the one used by [Godard et al.(2019)Godard,
Mac Aodha, Firman, and Brostow]. The difference is that we directly estimate
depth by multiplying the output of the last sigmoid layer by 250, which is the
maximum depth value that can be predicted, instead of estimating disparity as
in [Godard et al.(2019)Godard, Mac Aodha, Firman, and Brostow]. The depth
network takes as input a single target image $\mathbf{I}_{t}$ and outputs the
per-pixel depth estimates. Note that the output of our depth decoder is in
metric scale.

In DepthP+P, our second network takes $\mathbf{I}_{w}$ and $\mathbf{I}_{t}$
and outputs only the translation vector between the views. The network is
similar to the pose network proposed in [Godard et al.(2019)Godard, Mac Aodha,
Firman, and Brostow], except that the output is a 3-element vector
representing the translation and is metric scale in our case.

##  4 Experiments

###  4.1 Dataset

KITTI: We use the Eigen split [Eigen et al.(2014)Eigen, Puhrsch, and Fergus]
of the KITTI dataset [Geiger et al.(2013)Geiger, Lenz, Stiller, and Urtasun,
Geiger et al.(2012)Geiger, Lenz, and Urtasun] to train and evaluate our model.
We use all of the images in the split for which we could accurately estimate
the homography for aligning the road between consecutive images as explained
in the next paragraph. This results in 45000 training and 1769 validation
samples. We evaluate our model on the 697 test images in the split using the
original ground truth provided by LiDAR. We also report results using the
improved ground truth for 652 test images provided by Uhrig et al [Uhrig et
al.(2017)Uhrig, Schneider, Schneider, Franke, Brox, and Geiger]. They use a
stereo-reconstruction method to remove the outliers in LiDAR points and
increase the ground truth density by accumulating laser scans which result in
high-quality ground truth data. The camera height in this dataset is
$\mathbf{d}_{c}=1.65$ and we assume that the road is completely horizontal,
i.e. $\mathbf{N}=[0,1,0]^{T}$.

Pre-processing the dataset for DepthP+P: In order to use our P+P approach, we
need to calculate the homography between the consecutive frames and warp the
source frame according to the estimated homographies. Since we work on the
driving scenarios on KITTI, we choose the “road” as our plane
$\boldsymbol{\Pi}$ which is visible in most of the frames. For calculating the
homography, we need to find a set of (at least 4) corresponding pairs of road
pixels between a source view $\mathbf{I}_{s}$ and the target view
$\mathbf{I}_{t}$, i.e. two consecutive images. For this purpose, we use the
optical flow between $\mathbf{I}_{s}$ and $\mathbf{I}_{t}$ using [Teed and
Deng(2020)] to find the corresponding pixels. We then use [Zhu et
al.(2019)Zhu, Sapra, Reda, Shih, Newsam, Tao, and Catanzaro] to select only
the pixels that belong to the semantic class “road”. Using the corresponding
pairs of road pixels, we estimate the homography $\mathbf{H}$ using OpenCV’s
RANSAC-based robust method. We do this to find the homography $\mathbf{H}$ for
all of the consecutive pairs of frames on KITTI. Note that for any consecutive
pair of frames $\mathbf{I}_{1},\mathbf{I}_{2}$ and the homography $\mathbf{H}$
between them, we use $\mathbf{H}$ to warp $\mathbf{I}_{1}$ towards
$\mathbf{I}_{2}$ and also use $\mathbf{H}^{-1}$ to warp $\mathbf{I}_{2}$
towards $\mathbf{I}_{1}$.

###  4.2 Depth Estimation Results

Method Scale Lower Better Higher Better Abs Rel Sq Rel RMSE RMSElog
$\delta<1.25$ $\delta<1.25^{2}$ $\delta<1.25^{3}$ Original Ground Truth Zhou
et al. [Zhou et al.(2017)Zhou, Brown, Snavely, and Lowe] ✗ 0.183 1.595 6.709
0.270 0.734 0.902 0.959 Yang et al. [Yang et al.(2018b)Yang, Wang, Xu, Zhao,
and Nevatia] ✗ 0.182 1.481 6.501 0.267 0.725 0.906 0.963 Mahjourian et al.
[Mahjourian et al.(2018)Mahjourian, Wicke, and Angelova] ✗ 0.163 1.240 6.220
0.250 0.762 0.916 0.968 Yin et al. [Yin and Shi(2018)] ✗ 0.149 1.060 5.567
2.226 0.796 0.935 0.975 Wang et al. [Wang et al.(2018)Wang, Miguel
Buenaposada, Zhu, and Lucey] ✗ 0.151 1.257 5.583 0.228 0.810 0.936 0.974 Zou
et al. [Zou et al.(2018)Zou, Luo, and Huang] ✗ 0.150 1.124 5.507 0.223 0.806
0.933 0.973 Yang et al. [Yang et al.(2018a)Yang, Wang, Wang, Xu, and Nevatia]
✗ 0.162 1.352 6.276 0.252 - - - Ranjan et al. [Ranjan et al.(2019)Ranjan,
Jampani, Balles, Kim, Sun, Wulff, and Black] ✗ 0.148 1.149 5.464 0.226 0.815
0.935 0.973 Luo et al. [Luo et al.(2019)Luo, Yang, Wang, Wang, Xu, Nevatia,
and Yuille] ✗ 0.141 1.029 5.350 0.216 0.816 0.941 0.976 Chen et al. [Chen et
al.(2019)Chen, Schmid, and Sminchisescu] ✗ 0.135 1.070 5.230 0.210 0.841 0.948
0.980 Godard et al. [Godard et al.(2019)Godard, Mac Aodha, Firman, and
Brostow] ✗ 0.110 0.831 4.642 0.187 0.883 0.962 0.982 Guizilini et al.
[Guizilini et al.(2020)Guizilini, Ambrus, Pillai, Raventos, and Gaidon] ✗
0.111 0.785 4.601 0.189 0.878 0.960 0.982 Safadoust et al. [Safadoust and
Güney(2021)] ✗ 0.110 0.792 4.700 0.189 0.881 0.960 0.982 Xue et al. [Xue et
al.(2020)Xue, Zhuo, Huang, Fu, Wu, and Ang] ✓ 0.118 0.925 4.918 0.199 0.862
0.953 0.979 Wagstaff and Kelly [Wagstaff and Kelly(2021)] ✓ 0.123 0.996 5.253
0.213 0.840 0.947 0.978 DepthP+P (Ours) ✓ 0.152 1.322 6.185 0.239 0.781 0.920
0.970 Improved GT [Uhrig et al.(2017)Uhrig, Schneider, Schneider, Franke,
Brox, and Geiger] Zhou et al. [Zhou et al.(2017)Zhou, Brown, Snavely, and
Lowe] ✗ 0.176 1.532 6.129 0.244 0.758 0.921 0.971 Mahjourian et al.
[Mahjourian et al.(2018)Mahjourian, Wicke, and Angelova] ✗ 0.134 0.983 5.501
0.203 0.827 0.944 0.981 Yin et al. [Yin and Shi(2018)] ✗ 0.132 0.994 5.240
0.193 0.833 0.953 0.985 Wang et al. [Wang et al.(2018)Wang, Miguel
Buenaposada, Zhu, and Lucey] ✗ 0.126 0.866 4.932 0.185 0.851 0.958 0.986
Ranjan et al. [Ranjan et al.(2019)Ranjan, Jampani, Balles, Kim, Sun, Wulff,
and Black] ✗ 0.123 0.881 4.834 0.181 0.860 0.959 0.985 Luo et al. [Luo et
al.(2019)Luo, Yang, Wang, Wang, Xu, Nevatia, and Yuille] ✗ 0.120 0.789 4.755
0.177 0.856 0.961 0.987 Godard et al. [Godard et al.(2019)Godard, Mac Aodha,
Firman, and Brostow] ✗ 0.085 0.468 3.672 0.128 0.921 0.985 0.995 Safadoust et
al. [Safadoust and Güney(2021)] ✗ 0.085 0.458 3.779 0.131 0.919 0.985 0.996
Guizilini et al. [Guizilini et al.(2020)Guizilini, Ambrus, Pillai, Raventos,
and Gaidon] ✗ 0.078 0.420 3.485 0.121 0.931 0.986 0.996 DepthP+P (Ours) ✓
0.134 1.042 5.566 0.199 0.820 0.946 0.983

Table 1: Quantitative Results for Monocular Training on KITTI. This table
compares our proposed approach, DepthP+P, to previous approaches on the KITTI
dataset that were trained only with monocular supervision. The scale column
specifies whether the method can estimate depth in metric scale. We provide
results with the original and improved ground truth. We show the results for
the input resolution $640\times 192$. The best method in each column is shown
in bold and the second best is underlined.

In Table 1, we report the depth estimation results of our method on the KITTI
Eigen split using both the original and the improved ground truth. To the best
of our knowledge, this is the first time that a deep learning model has been
trained with view synthesis through the planar parallax paradigm (Eq. (2)).
All of the previous methods are trained based on estimating the pose whereas
our method introduces a novel approach. We can see that our method achieves
significantly better results than the initial models by predicting the pose
and depth. After the initial proposal of SfMLearner by Zhou et al. [Zhou et
al.(2017)Zhou, Brown, Snavely, and Lowe], several improvements have been
proposed to improve its performance. Therefore, we believe that similar
improvements can follow our model as future work to make it perform better
than our initial proposal as well as the other state-of-the-art models that
are trained to estimate the full pose.

Note that [Xue et al.(2020)Xue, Zhuo, Huang, Fu, Wu, and Ang] is not trained
to estimate metrically accurate depth. Instead, its depth network outputs
depth in an unknown scale, and then during inference, it needs a ground plane
to be visible on the image to recover the scale of the network. When the
ground plane is not visible on the image, [Xue et al.(2020)Xue, Zhuo, Huang,
Fu, Wu, and Ang] fails completely as shown in Fig. 3. As can be seen in this
figure, this image from the KITTI dataset does not have a ground plane, and
[Xue et al.(2020)Xue, Zhuo, Huang, Fu, Wu, and Ang] cannot recover the scale
and produces completely wrong estimates. While our method needs a ground plane
during training, it does not rely on the availability of the ground plane
during inference, therefore it can still perform well. For reference, the
absolute relative (Abs Rel) error of [Xue et al.(2020)Xue, Zhuo, Huang, Fu,
Wu, and Ang] on Fig. 3 is 1.178, while our model achieves a 0.252 error.
[Wagstaff and Kelly(2021)] achieves better results by using a pre-trained
plane segmentation network in addition to the depth network, while our
approach can achieve comparable results without a separate segmentation
network.

Method Lower Better Higher Better Abs Rel Sq Rel RMSE RMSElog $\delta<1.25$
$\delta<1.25^{2}$ $\delta<1.25^{3}$ Original GT Li et al. [Li et al.(2018)Li,
Wang, Long, and Gu] 0.183 1.730 6.570 0.268 - - - Zhan et al. [Zhan et
al.(2018)Zhan, Garg, Saroj Weerasekera, Li, Agarwal, and Reid] 0.135 1.132
5.585 0.229 0.820 0.933 0.971 Luo et al. [Luo et al.(2019)Luo, Yang, Wang,
Wang, Xu, Nevatia, and Yuille] 0.128 0.935 5.011 0.209 0.831 0.945 0.979
Godard et al. [Godard et al.(2019)Godard, Mac Aodha, Firman, and Brostow]
0.106 0.818 4.750 0.196 0.874 0.957 0.979 DepthP+P (ResNet18) 0.110 0.907
4.888 0.199 0.867 0.954 0.979 DepthP+P (ResNet50) 0.106 0.900 4.828 0.198
0.871 0.954 0.979 Improved GT Zhan et al. [Zhan et al.(2018)Zhan, Garg, Saroj
Weerasekera, Li, Agarwal, and Reid] 0.130 1.520 5.184 0.205 0.859 0.955 0.981
Luo et al. [Luo et al.(2019)Luo, Yang, Wang, Wang, Xu, Nevatia, and Yuille]
0.123 0.754 4.453 0.172 0.863 0.964 0.989 Godard et al. [Godard et
al.(2019)Godard, Mac Aodha, Firman, and Brostow] 0.080 0.466 3.681 0.127 0.926
0.985 0.995 DepthP+P (ResNet18) 0.088 0.572 3.905 0.138 0.911 0.981 0.994
DepthP+P (ResNet50) 0.084 0.543 3.784 0.134 0.916 0.982 0.995

Table 2: Quantitative Results on KITTI with Additional Stereo Supervision. We
compare DepthP+P to previous approaches that use additional stereo supervision
on KITTI. Stereo supervision significantly improves the results of DepthP+P
model. By using ResNet50, our model performs on par with Monodepth2 [Godard et
al.(2019)Godard, Mac Aodha, Firman, and Brostow]. Figure 3: Qualitative
Comparison. We compare the absolute relative error of our depth estimation
method (left) with DNet [Xue et al.(2020)Xue, Zhuo, Huang, Fu, Wu, and Ang]
(right) on an image from the KITTI dataset without a ground plane. The
colorbar on the right shows the values of the absolute relative error metric.
We cap the max error at the value of 1.0 for visualization. We can see that
[Xue et al.(2020)Xue, Zhuo, Huang, Fu, Wu, and Ang] completely fails to
estimate the metric depth due to the wrong scale recovery because there is no
ground plane in the image, while our model does not have this issue and can
still perform well. The absolute relative error for [Xue et al.(2020)Xue,
Zhuo, Huang, Fu, Wu, and Ang] is 1.178 while it is 0.252 for our method.

DepthP+P can also be trained with additional stereo supervision. In the
proposed approach, we obtain monocular supervision from the P+P paradigm. In
addition, using the known camera baseline and the estimated depth, we can warp
the other image in the stereo setup to the input image for additional
supervision signal. In Table 2, we report the performances of the methods that
also use stereo supervision for training. Using stereo supervision
significantly improves the performance of our DepthP+P model, outperforming
all methods except for Monodepth2 [Godard et al.(2019)Godard, Mac Aodha,
Firman, and Brostow]. We show that by using a ResNet50 backbone instead of
ResNet18, DepthP+P can obtain comparable results to Monodepth2 [Godard et
al.(2019)Godard, Mac Aodha, Firman, and Brostow].

##  5 Conclusion and Future Work

In this paper, we presented a new approach to self-supervised monocular depth
estimation following the traditional planar parallax paradigm. We showed that
our approach is able to produce metrically accurate depth estimates by using a
known camera height. Unlike previous methods that rely on estimating the full
rigid-body motion of the camera, our method only needs to estimate the camera
translation. We discussed the advantage of our method compared to the other
scale-aware depth prediction methods. We see our approach as a first step to
unlocking the potential of the plane and parallax for efficient and metric-
accurate depth estimation. An exciting future direction can focus on detecting
moving foreground objects by checking the violations in the plane and parallax
constraints [Irani and Anandan(1996)].

## References

  * [Bartoccioni et al.(2021)Bartoccioni, Zablocki, Pérez, Cord, and Alahari] Florent Bartoccioni, Éloi Zablocki, Patrick Pérez, Matthieu Cord, and Karteek Alahari.  Lidartouch: Monocular metric depth estimation with a few-beam lidar.  _arXiv.org_ , 2109.03569, 2021.
  * [Bian et al.(2019)Bian, Li, Wang, Zhan, Shen, Cheng, and Reid] Jiawang Bian, Zhichao Li, Naiyan Wang, Huangying Zhan, Chunhua Shen, Ming-Ming Cheng, and Ian Reid.  Unsupervised scale-consistent depth and ego-motion learning from monocular video.  In _Advances in Neural Information Processing Systems (NeurIPS)_ , pages 35–45, 2019.
  * [Chaney et al.(2019)Chaney, Zhu, and Daniilidis] Kenneth Chaney, Alex Zihao Zhu, and Kostas Daniilidis.  Learning event-based height from plane and parallax.  In _Proc. IEEE International Conf. on Intelligent Robots and Systems (IROS)_ , pages 3690–3696, 2019.  10.1109/IROS40897.2019.8968223.
  * [Chen et al.(2019)Chen, Schmid, and Sminchisescu] Yuhua Chen, Cordelia Schmid, and Cristian Sminchisescu.  Self-supervised learning with geometric constraints in monocular video: Connecting flow, depth, and camera.  In _Proc. of the IEEE International Conf. on Computer Vision (ICCV)_ , pages 7063–7072, 2019.
  * [Eigen et al.(2014)Eigen, Puhrsch, and Fergus] David Eigen, Christian Puhrsch, and Rob Fergus.  Depth map prediction from a single image using a multi-scale deep network.  In _Advances in Neural Information Processing Systems (NeurIPS)_ , pages 2366–2374, 2014.
  * [Garg et al.(2016)Garg, Bg, Carneiro, and Reid] Ravi Garg, Vijay Kumar Bg, Gustavo Carneiro, and Ian Reid.  Unsupervised CNN for single view depth estimation: Geometry to the rescue.  In _Proc. of the European Conf. on Computer Vision (ECCV)_ , pages 740–756, 2016.
  * [Geiger et al.(2012)Geiger, Lenz, and Urtasun] Andreas Geiger, Philip Lenz, and Raquel Urtasun.  Are we ready for autonomous driving? the kitti vision benchmark suite.  In _Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR)_ , 2012.
  * [Geiger et al.(2013)Geiger, Lenz, Stiller, and Urtasun] Andreas Geiger, Philip Lenz, Christoph Stiller, and Raquel Urtasun.  Vision meets robotics: The KITTI dataset.  _International Journal of Robotics Research (IJRR)_ , 2013.
  * [Godard et al.(2017)Godard, Mac Aodha, and Brostow] Clément Godard, Oisin Mac Aodha, and Gabriel J Brostow.  Unsupervised monocular depth estimation with left-right consistency.  In _Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR)_ , pages 270–279, 2017.
  * [Godard et al.(2019)Godard, Mac Aodha, Firman, and Brostow] Clément Godard, Oisin Mac Aodha, Michael Firman, and Gabriel J Brostow.  Digging into self-supervised monocular depth estimation.  In _Proc. of the IEEE International Conf. on Computer Vision (ICCV)_ , 2019.
  * [Guizilini et al.(2020)Guizilini, Ambrus, Pillai, Raventos, and Gaidon] Vitor Guizilini, Rares Ambrus, Sudeep Pillai, Allan Raventos, and Adrien Gaidon.  3D packing for self-supervised monocular depth estimation.  In _Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR)_ , 2020.
  * [He et al.(2016)He, Zhang, Ren, and Sun] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In _Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR)_ , pages 770–778, 2016.
  * [Irani et al.(1998)Irani, Anandan, and Weinshall] M. Irani, P. Anandan, and D. Weinshall.  From reference frames to reference planes: Multi-view parallax geometry and applications.  In Hans Burkhardt and Bernd Neumann, editors, _Proc. of the European Conf. on Computer Vision (ECCV)_ , pages 829–845, Berlin, Heidelberg, 1998. Springer Berlin Heidelberg.  ISBN 978-3-540-69235-5.
  * [Irani et al.(2002)Irani, Anandan, and Cohen] M. Irani, P. Anandan, and M. Cohen.  Direct recovery of planar-parallax from multiple frames.  _IEEE Trans. on Pattern Analysis and Machine Intelligence (PAMI)_ , 24(11):1528–1534, 2002.  10.1109/TPAMI.2002.1046174.
  * [Irani and Anandan(1996)] Michal Irani and P. Anandan.  Parallax geometry of pairs of points for 3d scene analysis.  In Bernard Buxton and Roberto Cipolla, editors, _Proc. of the European Conf. on Computer Vision (ECCV)_ , pages 17–30, Berlin, Heidelberg, 1996\. Springer Berlin Heidelberg.  ISBN 978-3-540-49949-7.
  * [Jaderberg et al.(2015)Jaderberg, Simonyan, Zisserman, and Kavukcuoglu] Max Jaderberg, Karen Simonyan, Andrew Zisserman, and Koray Kavukcuoglu.  Spatial transformer networks.  In _Advances in Neural Information Processing Systems (NeurIPS)_ , pages 2017–2025, 2015.
  * [Li et al.(2018)Li, Wang, Long, and Gu] Ruihao Li, Sen Wang, Zhiqiang Long, and Dongbing Gu.  Undeepvo: Monocular visual odometry through unsupervised deep learning.  In _Proc. IEEE International Conf. on Robotics and Automation (ICRA)_ , pages 7286–7291. IEEE, 2018.
  * [Luo et al.(2019)Luo, Yang, Wang, Wang, Xu, Nevatia, and Yuille] Chenxu Luo, Zhenheng Yang, Peng Wang, Yang Wang, Wei Xu, Ram Nevatia, and Alan Yuille.  Every pixel counts++: Joint learning of geometry and motion with 3d holistic understanding.  _IEEE Trans. on Pattern Analysis and Machine Intelligence (PAMI)_ , 42(10):2624–2641, 2019.
  * [Mahjourian et al.(2018)Mahjourian, Wicke, and Angelova] Reza Mahjourian, Martin Wicke, and Anelia Angelova.  Unsupervised learning of depth and ego-motion from monocular video using 3d geometric constraints.  In _Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR)_ , pages 5667–5675, 2018.
  * [Ranjan et al.(2019)Ranjan, Jampani, Balles, Kim, Sun, Wulff, and Black] Anurag Ranjan, Varun Jampani, Lukas Balles, Kihwan Kim, Deqing Sun, Jonas Wulff, and Michael J Black.  Competitive collaboration: Joint unsupervised learning of depth, camera motion, optical flow and motion segmentation.  In _Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR)_ , pages 12240–12249, 2019.
  * [Ronneberger et al.(2015)Ronneberger, Fischer, and Brox] Olaf Ronneberger, Philipp Fischer, and Thomas Brox.  U-Net: Convolutional networks for biomedical image segmentation.  In _Medical Image Computing and Computer-Assisted Intervention (MICCAI)_ , pages 234–241, 2015.
  * [Roussel et al.(2019)Roussel, Eycken, and Tuytelaars] Tom Roussel, Luc Van Eycken, and Tinne Tuytelaars.  Monocular depth estimation in new environments with absolute scale.  In _Proc. IEEE International Conf. on Intelligent Robots and Systems (IROS)_ , pages 1735–1741, 2019.  10.1109/IROS40897.2019.8967677.
  * [Russakovsky et al.(2015)Russakovsky, Deng, Su, Krause, Satheesh, Ma, Huang, Karpathy, Khosla, Bernstein, et al.] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al.  ImageNet large scale visual recognition challenge.  _International Journal of Computer Vision (IJCV)_ , 115(3):211–252, 2015.
  * [Safadoust and Güney(2021)] Sadra Safadoust and Fatma Güney.  Self-supervised monocular scene decomposition and depth estimation.  In _International Conference on 3D Vision (3DV)_ , pages 627–636, 2021.
  * [Sawhney(1994)] Sawhney.  3d geometry from planar parallax.  In _Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR)_ , pages 929–934, 1994.  10.1109/CVPR.1994.323927.
  * [Teed and Deng(2020)] Zachary Teed and Jia Deng.  Raft: Recurrent all-pairs field transforms for optical flow.  In _Proc. of the European Conf. on Computer Vision (ECCV)_ , pages 402–419. Springer, 2020.
  * [Uhrig et al.(2017)Uhrig, Schneider, Schneider, Franke, Brox, and Geiger] Jonas Uhrig, Nick Schneider, Lukas Schneider, Uwe Franke, Thomas Brox, and Andreas Geiger.  Sparsity invariant CNNs.  In _Proc. of the International Conf. on 3D Vision (3DV)_ , 2017.
  * [Wagstaff and Kelly(2021)] Brandon Wagstaff and Jonathan Kelly.  Self-supervised scale recovery for monocular depth and egomotion estimation.  In _2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_ , pages 2620–2627. IEEE, 2021.
  * [Wang et al.(2018)Wang, Miguel Buenaposada, Zhu, and Lucey] Chaoyang Wang, José Miguel Buenaposada, Rui Zhu, and Simon Lucey.  Learning depth from monocular videos using direct methods.  In _Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR)_ , pages 2022–2030, 2018.
  * [Wang et al.(2004)Wang, Bovik, Sheikh, and Simoncelli] Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero P Simoncelli.  Image quality assessment: from error visibility to structural similarity.  _IEEE Trans. on Image Processing (TIP)_ , 13(4):600–612, 2004.
  * [Wulff et al.(2017)Wulff, Sevilla-Lara, and Black] Jonas Wulff, Laura Sevilla-Lara, and Michael J Black.  Optical flow in mostly rigid scenes.  In _Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR)_ , pages 4671–4680, 2017.
  * [Xue et al.(2020)Xue, Zhuo, Huang, Fu, Wu, and Ang] Feng Xue, Guirong Zhuo, Ziyuan Huang, Wufei Fu, Zhuoyue Wu, and Marcelo H Ang.  Toward hierarchical self-supervised monocular absolute depth estimation for autonomous driving applications.  In _Proc. IEEE International Conf. on Intelligent Robots and Systems (IROS)_ , pages 2330–2337. IEEE, 2020.
  * [Yang et al.(2018a)Yang, Wang, Wang, Xu, and Nevatia] Zhenheng Yang, Peng Wang, Yang Wang, Wei Xu, and Ram Nevatia.  LEGO: Learning edge with geometry all at once by watching videos.  In _Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR)_ , pages 225–234, 2018a.
  * [Yang et al.(2018b)Yang, Wang, Xu, Zhao, and Nevatia] Zhenheng Yang, Peng Wang, Wei Xu, Liang Zhao, and Ramakant Nevatia.  Unsupervised learning of geometry from videos with edge-aware depth-normal consistency.  In _Proc. of the Conf. on Artificial Intelligence (AAAI)_ , 2018b.
  * [Yin and Shi(2018)] Zhichao Yin and Jianping Shi.  GeoNet: Unsupervised learning of dense depth, optical flow and camera pose.  In _Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR)_ , pages 1983–1992, 2018.
  * [Zhan et al.(2018)Zhan, Garg, Saroj Weerasekera, Li, Agarwal, and Reid] Huangying Zhan, Ravi Garg, Chamara Saroj Weerasekera, Kejie Li, Harsh Agarwal, and Ian Reid.  Unsupervised learning of monocular depth estimation and visual odometry with deep feature reconstruction.  In _Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR)_ , pages 340–349, 2018.
  * [Zhou et al.(2017)Zhou, Brown, Snavely, and Lowe] Tinghui Zhou, Matthew Brown, Noah Snavely, and David G Lowe.  Unsupervised learning of depth and ego-motion from video.  In _Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR)_ , pages 1851–1858, 2017.
  * [Zhu et al.(2019)Zhu, Sapra, Reda, Shih, Newsam, Tao, and Catanzaro] Yi Zhu, Karan Sapra, Fitsum A Reda, Kevin J Shih, Shawn Newsam, Andrew Tao, and Bryan Catanzaro.  Improving semantic segmentation via video propagation and label relaxation.  In _Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR)_ , 2019.
  * [Zou et al.(2018)Zou, Luo, and Huang] Yuliang Zou, Zelun Luo, and Jia-Bin Huang.  DF-Net: Unsupervised joint learning of depth and flow using cross-task consistency.  In _Proc. of the European Conf. on Computer Vision (ECCV)_ , pages 36–53, 2018.

Supplementary Material for
DepthP+P: Metric Accurate Monocular Depth Estimation using Planar and Parallax
Sadra<EMAIL_ADDRESS>Fatma<EMAIL_ADDRESS>KUIS AI
Center
Koç University
Istanbul, Turkey

In Section A of this supplementary document, we provide the derivation of the
residual parallax (Equation 2 in the main paper) [Irani and Anandan(1996)]. In
Section B, we investigate the effect of using a more accurate estimation of
the normal vector of the road. Finally, in Section C, we provide additional
qualitative results of our model.

##  A Derivation

Let $\mathbf{C}$ and $\mathbf{C}^{\prime}$ be the camera centers of the target
view $\mathbf{I}_{t}$ and the source view $\mathbf{I}_{s}$, respectively.
Also, let $\mathbf{x}=[\mathbf{X},\mathbf{Y},\mathbf{Z}]^{T}$ and
$\mathbf{x}^{\prime}=[\mathbf{X}^{\prime},\mathbf{Y}^{\prime},\mathbf{Z}^{\prime}]^{T}$
be the coordinates of a 3D point with respect to $\mathbf{C}$ and
$\mathbf{C}^{\prime}$ respectively. We can relate $\mathbf{x}$ and
$\mathbf{x}^{\prime}$ as follows:

$\mathbf{x}=\mathbf{R}\mathbf{x}^{\prime}+\mathbf{t}$ (8)

where $\mathbf{R}$ is the rotation and
$\mathbf{t}=[\mathbf{t}_{x},\mathbf{t}_{y},\mathbf{t}_{z}]^{T}$ is the
translation between $\mathbf{C}$ and $\mathbf{C}^{\prime}$. According to the
Fig 2 (right), we can calculate $\mathbf{h}$, the perpendicular distance of
the 3D point to the plane $\boldsymbol{\Pi}$ as:

$\mathbf{h}=\mathbf{d}_{c}-\mathbf{N}^{T}\mathbf{x}^{\prime}$ (9)

Note that $\mathbf{d}_{c}$ is the height of $\mathbf{C}^{\prime}$, the camera
of the source view $\mathbf{I}_{s}$. However, $\mathbf{h}$ is invariant with
respect to the cameras. We can rewrite the above equation as:

$1=\frac{\mathbf{h}+\mathbf{N}^{T}\mathbf{x}^{\prime}}{\mathbf{d}_{c}}$ (10)

Therefore, by substituting this into Eq. (8), we have:

$\mathbf{x}=\mathbf{R}\mathbf{x}^{\prime}+\mathbf{t}\left(\frac{\mathbf{h}+\mathbf{N}^{T}\mathbf{x}^{\prime}}{\mathbf{d}_{c}}\right)=\left(\mathbf{R}+\frac{\mathbf{t}\mathbf{N}^{T}}{\mathbf{d}_{c}}\right)\mathbf{x}^{\prime}+\frac{\mathbf{h}}{\mathbf{d}_{c}}\mathbf{t}$
(11)

Let $\mathbf{K}$ and $\mathbf{K}^{\prime}$ denote the camera intrinsic for the
target view and the source view. Then
$\mathbf{p}=[x,y,1]^{T}=\frac{1}{\mathbf{Z}}\mathbf{K}\mathbf{x}$ and
$\mathbf{p}^{\prime}=[x^{\prime},y^{\prime},1]=\frac{1}{\mathbf{Z}^{\prime}}\mathbf{K}^{\prime}\mathbf{x}^{\prime}$
represent the pixel coordinates of our 3D point in the target and source view,
respectively. Therefore Eq. (11) can be written as:

$\mathbf{Z}\mathbf{K}^{-1}\mathbf{p}=\left(\mathbf{R}+\frac{\mathbf{t}\mathbf{N}^{T}}{\mathbf{d}_{c}}\right)\mathbf{Z}^{\prime}\mathbf{K}^{\prime-1}\mathbf{p}^{\prime}+\frac{\mathbf{h}}{\mathbf{d}_{c}}\mathbf{t}$
(12)

By multiplying both sides by $\frac{1}{\mathbf{Z}^{\prime}}\mathbf{K}$ we
obtain:

$\frac{\mathbf{Z}}{\mathbf{Z}^{\prime}}\mathbf{p}=\underbrace{\mathbf{K}\left(\mathbf{R}+\frac{\mathbf{t}\mathbf{N}^{T}}{\mathbf{d}_{c}}\right)\mathbf{K}^{\prime-1}}_{\mathbf{H}}\mathbf{p}^{\prime}+\frac{\mathbf{h}}{\mathbf{Z}^{\prime}\mathbf{d}_{c}}\mathbf{K}\mathbf{t}$
(13)

$\frac{\mathbf{Z}}{\mathbf{Z}^{\prime}}\mathbf{p}=\mathbf{H}\mathbf{p}^{\prime}+\frac{\mathbf{h}}{\mathbf{Z}^{\prime}\mathbf{d}_{c}}\mathbf{K}\mathbf{t}$
(14)

where
$\mathbf{H}=\mathbf{K}\left(\mathbf{R}+\frac{\mathbf{t}\mathbf{N}^{T}}{\mathbf{d}_{c}}\right)\mathbf{K}^{\prime-1}$
is the $3\times 3$ homography matrix associated with the plane
$\boldsymbol{\Pi}$ between the source and the target view. Note that in Eq.
(14), the third component of the vector in two sides of the equation should be
equal. Therefore:

$\frac{\mathbf{Z}}{\mathbf{Z}^{\prime}}=\mathbf{H}_{3}\mathbf{p}^{\prime}+\frac{\mathbf{h}}{\mathbf{Z}^{\prime}\mathbf{d}_{c}}\mathbf{K}_{3}\mathbf{t}$
(15)

where $\mathbf{H}_{3}$ and $\mathbf{K}_{3}$ are the third row of $\mathbf{H}$
and $\mathbf{K}$. Since $\mathbf{K}_{3}\mathbf{t}=\mathbf{t}_{z}$, we
therefore have:

$\frac{\mathbf{Z}}{\mathbf{Z}^{\prime}}=\mathbf{H}_{3}\mathbf{p}^{\prime}+\frac{\mathbf{h}}{\mathbf{Z}^{\prime}\mathbf{d}_{c}}\mathbf{t}_{z}$
(16)

By dividing each side of Eq. (14) by each side of Eq. (16) we obtain:

$\mathbf{p}=\frac{\mathbf{H}\mathbf{p}^{\prime}+\frac{\mathbf{h}}{\mathbf{Z}^{\prime}\mathbf{d}_{c}}\mathbf{K}\mathbf{t}}{\mathbf{H}_{3}\mathbf{p}^{\prime}+\frac{\mathbf{h}}{\mathbf{Z}^{\prime}\mathbf{d}_{c}}\mathbf{t}_{z}}$
(17)

Adding and subtracting
$\frac{\mathbf{H}\mathbf{p}^{\prime}}{\mathbf{H}_{3}\mathbf{p}^{\prime}}$ to
the right-hand side yields:

$\displaystyle\mathbf{p}$
$\displaystyle=\frac{\mathbf{H}\mathbf{p}^{\prime}}{\mathbf{H}_{3}\mathbf{p}^{\prime}}-\frac{\mathbf{H}\mathbf{p}^{\prime}}{\mathbf{H}_{3}\mathbf{p}^{\prime}}+\frac{\mathbf{H}\mathbf{p}^{\prime}+\frac{\mathbf{h}}{\mathbf{Z}^{\prime}\mathbf{d}_{c}}\mathbf{K}\mathbf{t}}{\mathbf{H}_{3}\mathbf{p}^{\prime}+\frac{\mathbf{h}}{\mathbf{Z}^{\prime}\mathbf{d}_{c}}\mathbf{t}_{z}}$
(18)
$\displaystyle=\frac{\mathbf{H}\mathbf{p}^{\prime}}{\mathbf{H}_{3}\mathbf{p}^{\prime}}-\frac{\mathbf{H}\mathbf{p}^{\prime}\left(\mathbf{H}_{3}\mathbf{p}^{\prime}+\frac{\mathbf{h}}{\mathbf{Z}^{\prime}\mathbf{d}_{c}}\mathbf{t}_{z}\right)}{\mathbf{H}_{3}\mathbf{p}^{\prime}\left(\mathbf{H}_{3}\mathbf{p}^{\prime}+\frac{\mathbf{h}}{\mathbf{Z}^{\prime}\mathbf{d}_{c}}\mathbf{t}_{z}\right)}+\frac{\mathbf{H}_{3}\mathbf{p}^{\prime}\left(\mathbf{H}\mathbf{p}^{\prime}+\frac{\mathbf{h}}{\mathbf{Z}^{\prime}\mathbf{d}_{c}}\mathbf{K}\mathbf{t}\right)}{\mathbf{H}_{3}\mathbf{p}^{\prime}\left(\mathbf{H}_{3}\mathbf{p}^{\prime}+\frac{\mathbf{h}}{\mathbf{Z}^{\prime}\mathbf{d}_{c}}\mathbf{t}_{z}\right)}$
(19)
$\displaystyle=\frac{\mathbf{H}\mathbf{p}^{\prime}}{\mathbf{H}_{3}\mathbf{p}^{\prime}}-\frac{\mathbf{H}\mathbf{p}^{\prime}\left(\frac{\mathbf{h}}{\mathbf{Z}^{\prime}\mathbf{d}_{c}}\mathbf{t}_{z}\right)}{\mathbf{H}_{3}\mathbf{p}^{\prime}\left(\mathbf{H}_{3}\mathbf{p}^{\prime}+\frac{\mathbf{h}}{\mathbf{Z}^{\prime}\mathbf{d}_{c}}\mathbf{t}_{z}\right)}+\frac{\mathbf{H}_{3}\mathbf{p}^{\prime}\left(\frac{\mathbf{h}}{\mathbf{Z}^{\prime}\mathbf{d}_{c}}\mathbf{K}\mathbf{t}\right)}{\mathbf{H}_{3}\mathbf{p}^{\prime}\left(\mathbf{H}_{3}\mathbf{p}^{\prime}+\frac{\mathbf{h}}{\mathbf{Z}^{\prime}\mathbf{d}_{c}}\mathbf{t}_{z}\right)}$
(20)

Substituting Eq. (16) into the denominators of Eq. (20) results in:

$\displaystyle\mathbf{p}$
$\displaystyle=\frac{\mathbf{H}\mathbf{p}^{\prime}}{\mathbf{H}_{3}\mathbf{p}^{\prime}}-\frac{\mathbf{H}\mathbf{p}^{\prime}\left(\frac{\mathbf{h}}{\mathbf{Z}^{\prime}\mathbf{d}_{c}}\mathbf{t}_{z}\right)}{\mathbf{H}_{3}\mathbf{p}^{\prime}\left(\frac{\mathbf{Z}}{\mathbf{Z}^{\prime}}\right)}+\frac{\mathbf{H}_{3}\mathbf{p}^{\prime}\left(\frac{\mathbf{h}}{\mathbf{Z}^{\prime}\mathbf{d}_{c}}\mathbf{K}\mathbf{t}\right)}{\mathbf{H}_{3}\mathbf{p}^{\prime}\left(\frac{\mathbf{Z}}{\mathbf{Z}^{\prime}}\right)}$
(21)
$\displaystyle=\frac{\mathbf{H}\mathbf{p}^{\prime}}{\mathbf{H}_{3}\mathbf{p}^{\prime}}-\frac{\mathbf{h}\mathbf{t}_{z}}{\mathbf{Z}\mathbf{d}_{c}}\frac{\mathbf{H}\mathbf{p}^{\prime}}{\mathbf{H}_{3}\mathbf{p}^{\prime}}+\frac{\mathbf{h}}{\mathbf{Z}\mathbf{d}_{c}}\mathbf{K}\mathbf{t}$
(22)

The point
$\mathbf{p}_{w}=[x_{w},y_{w},1]^{T}=\frac{\mathbf{H}\mathbf{p}^{\prime}}{\mathbf{H}_{3}\mathbf{p}^{\prime}}$
is the point $\mathbf{p}^{\prime}$ transformed by the homography matrix
$\mathbf{H}$. Thus, Eq. (22) can be simplified to:

$\mathbf{p}=\mathbf{p}_{w}-\frac{\mathbf{h}\mathbf{t}_{z}}{\mathbf{Z}\mathbf{d}_{c}}\mathbf{p}_{w}+\frac{\mathbf{h}}{\mathbf{Z}\mathbf{d}_{c}}\mathbf{K}\mathbf{t}$
(23)

Subtracting
$\frac{\mathbf{h}\mathbf{t}_{z}}{\mathbf{Z}\mathbf{d}_{c}}\mathbf{p}$ from
both sides, we obtain:

$\mathbf{p}-\frac{\mathbf{h}\mathbf{t}_{z}}{\mathbf{Z}\mathbf{d}_{c}}\mathbf{p}=\mathbf{p}_{w}-\frac{\mathbf{h}\mathbf{t}_{z}}{\mathbf{Z}\mathbf{d}_{c}}\mathbf{p}_{w}+\frac{\mathbf{h}}{\mathbf{Z}\mathbf{d}_{c}}\mathbf{K}\mathbf{t}-\frac{\mathbf{h}\mathbf{t}_{z}}{\mathbf{Z}\mathbf{d}_{c}}\mathbf{p}$
(24)
$\mathbf{p}\left(1-\frac{\mathbf{h}\mathbf{t}_{z}}{\mathbf{Z}\mathbf{d}_{c}}\right)=\mathbf{p}_{w}\left(1-\frac{\mathbf{h}\mathbf{t}_{z}}{\mathbf{Z}\mathbf{d}_{c}}\right)+\frac{\mathbf{h}}{\mathbf{Z}\mathbf{d}_{c}}\mathbf{K}\mathbf{t}-\frac{\mathbf{h}\mathbf{t}_{z}}{\mathbf{Z}\mathbf{d}_{c}}\mathbf{p}$
(25)

By dividing both sides by
$1-\frac{\mathbf{h}\mathbf{t}_{z}}{\mathbf{Z}\mathbf{d}_{c}}$ we get:

$\mathbf{p}=\mathbf{p}_{w}+\frac{\frac{\mathbf{h}}{\mathbf{Z}\mathbf{d}_{c}}\mathbf{K}\mathbf{t}-\frac{\mathbf{h}\mathbf{t}_{z}}{\mathbf{Z}\mathbf{d}_{c}}\mathbf{p}}{1-\frac{\mathbf{h}\mathbf{t}_{z}}{\mathbf{Z}\mathbf{d}_{c}}}$
(26)

By rearranging the terms and defining
$\boldsymbol{\gamma}=\frac{\mathbf{h}}{\mathbf{Z}}$ we obtain:

$\displaystyle\mathbf{p}_{w}-\mathbf{p}$
$\displaystyle=\frac{\frac{\mathbf{h}\mathbf{t}_{z}}{\mathbf{Z}\mathbf{d}_{c}}\mathbf{p}-\frac{\mathbf{h}}{\mathbf{Z}\mathbf{d}_{c}}\mathbf{K}\mathbf{t}}{1-\frac{\mathbf{h}\mathbf{t}_{z}}{\mathbf{Z}\mathbf{d}_{c}}}$
(27)
$\displaystyle=\frac{\frac{\mathbf{h}\mathbf{t}_{z}}{\mathbf{Z}}\mathbf{p}-\frac{\mathbf{h}}{\mathbf{Z}}\mathbf{K}\mathbf{t}}{\mathbf{d}_{c}-\frac{\mathbf{h}\mathbf{t}_{z}}{\mathbf{Z}}}$
(28)
$\displaystyle=\frac{\boldsymbol{\gamma}\mathbf{t}_{z}\mathbf{p}-\boldsymbol{\gamma}\mathbf{K}\mathbf{t}}{\mathbf{d}_{c}-\boldsymbol{\gamma}\mathbf{t}_{z}}$
(29)
$\displaystyle=\frac{\boldsymbol{\gamma}}{\mathbf{d}_{c}-\boldsymbol{\gamma}\mathbf{t}_{z}}\left(\mathbf{t}_{z}\mathbf{p}-\mathbf{K}\mathbf{t}\right)$
(30)

##  B Ablation Study on Normal Vector

Method Abs Rel Sq Rel RMSE RMSElog $\delta<1.25$ $\delta<1.25^{2}$
$\delta<1.25^{3}$ DepthP+P (Fixed Normal) 0.152 1.322 6.185 0.239 0.781 0.920
0.970 DepthP+P (GT Normal) 0.142 1.127 5.922 0.238 0.780 0.921 0.971

Table 3: Effect of Calculating Normal Vector of Road. This table analyzes the
effect of accurately predicting the normal vector $\mathbf{N}$ of the road on
the performance. Fixed Normal assumes that the ground is perfectly horizontal.
GT Normal uses the ground truth depth data, generates the point clouds for the
road region, fits a plane and calculates the normal vector. GT Normal performs
better than Fixed Normal, highlighting the fact that a more accurate normal
vector calculation will improve the performance.

In our experiments in the paper, we have assumed that the road plane is
horizontal with respect to the camera. In other words, we have assumed that
$\mathbf{N}=[0,1,0]^{T}$. However, the road planes are not always completely
flat and can be titled. For example, one side can be higher than the other. Or
consider an uphill where the road is sloping upwards. To analyze this, we
perform an experiment where we calculate the normal vector of the road using
ground truth depth. Concretely, during training, we back-project the road
pixels to 3D using their ground truth depth and fit a plane to the obtained 3D
points. We then use the normal vector of the fitted plane as our vector
$\mathbf{N}$ in Equation 4 in the main paper. We report the results in Table
3. It can be seen that by estimating $\mathbf{N}$ using this approach, we can
achieve much better results. In this experiment, the ground truth values were
only used to calculate the normal vector $\mathbf{N}$. Therefore, we conclude
that our method can benefit from a more accurate estimation of the normal
vector. A future study can focus on predicting $\mathbf{N}$ more accurately
during training, as was done in [Xue et al.(2020)Xue, Zhuo, Huang, Fu, Wu, and
Ang] in testing phase, in Planar Parallax framework.

##  C Qualitative Results

In Fig. 4, we provide qualitative results of our models on the KITTI dataset
in comparison to Monodepth2 [Godard et al.(2019)Godard, Mac Aodha, Firman, and
Brostow] and DNet [Xue et al.(2020)Xue, Zhuo, Huang, Fu, Wu, and Ang]. Our
stereo model produces sharp outputs and captures the boundaries of the objects
very well, and neither of our models suffer from artefacts such as the wrong
estimation for the road lane line in the last row.

Input Image |  Monodepth2 [Godard et al.(2019)Godard, Mac Aodha, Firman, and Brostow] |  DNet [Xue et al.(2020)Xue, Zhuo, Huang, Fu, Wu, and Ang] |  Ours-Mono |  Ours-Stereo
---|---|---|---|---

Figure 4: Qualitative Results. In each row, for an input image, we show the
results of Monodepth2 [Godard et al.(2019)Godard, Mac Aodha, Firman, and
Brostow] and DNet [Xue et al.(2020)Xue, Zhuo, Huang, Fu, Wu, and Ang] and
compare them to our models in the last two columns. Ours-Mono refers to our
ResNet18 model trained only on monocular sequences, and Ours-Stereo refers to
our ResNet50 model that was trained using additional stereo supervision.