#  Variational inequality method in stock loans

Zongxia Liang
Department of Mathematical Sciences, Tsinghua University,
Beijing 100084, China Email<EMAIL_ADDRESS>
Weiming Wu
Department of Mathematical Sciences, Tsinghua University,
Beijing 100084, China Email<EMAIL_ADDRESS>
###### Abstract.

In this paper we first introduce two new financial products: stock loan and
capped stock loan. Then we develop a pure variational inequality method to
establish explicitly the values of these stock loans. Finally, we work out
ranges of fair values of parameters associated with the loans. MSC(2000):
Primary 91B02, 91B70, 91B24; Secondary 60H10,91B28. Keywords: Capped stock
loan, variational inequality method, perpetual American option, generalized
Itô formula , Black-Scholes model.

##  1\. Introduction

Stock loan is a simple economy where a client(borrower), who owns one share of
a stock, borrows a loan of amount $q$ from a bank (lender) with one share of
stock as collateral. The bank charges amount $c(0\leq c\leq q)$ from the
client for the service. The client may regain the stock by repaying principal
and interest (that is, $qe^{\gamma t}$, here $\gamma$ is continuously
compounding loan interest rate ) to the bank, or surrender the stock instead
of repaying the loan at any time. It is a currently popular financial product.
It can create liquidity while overcoming the barrier of large block sales,
such as triggering tax events or control restrictions on sales of stocks. It
also can serve as a hedge against a market down: if the stock price goes down,
the client may forfeit the loan at initial time; if the stock price goes up,
the client keeps all the upside by repaying the principal and interest.
Therefore, the stock loan has unlimited liability. As a result, stock loan
transforms risk into the bank. To reduce the risk, the bank introduces cap $L$
feature in the stock loan because the cap adds a further incentive to exercise
early. Such a loan is called a capped stock loan throughout this paper. Since
the capped stock loan has limited liability and lower risk, it therefore will
be an attractive instrument to market for an issuer, or to hold short for an
investor like American call option with cap in financial market(cf. Broadie
and Detemple [2](1995)). The wide acceptance and popular feature of the capped
(uncapped) stock loan in the marketplace, however, greatly depend on how to
make successfully these kinds of financial products. More precisely, how to
work out right values of the parameters $(q,\gamma,c,L)$ is a natural and key
problem in negotiation between the client and the bank at initial time.
Unfortunately, to the authors’ best knowledge, it seems that few results on
the topic have been reported in existing literature. The main goal of the
present paper is to develop a pure variational inequality method to solve this
kind of problems. We explain major difficulty and main idea of solving the
problem as follows. We formulate the capped (uncapped) stock loan as a
perpetual American option with negative interest rate. We denote by $f(x)$ the
initial value of this option. The problem can be reduced to calculating the
function $f(x)$ for determining the ranges of fair values of the parameters
$(q,\gamma,c,L)$. According to the conventional variational inequality method,
the $f(x)$ must satisfy a variational inequality, we calculate $f(x)$ by
initial condition $f(0)=0$ and smooth-fit principle. However, because of
negative interest rate the initial condition $f(0)=0$ does not work, i.e., the
conventional variational inequality method can not solve this option with
negative interest rate. Moreover, the presence of the cap also complicates the
valuation procedure(if we focus on studying the capped stock loan). So we need
to develop the variational inequality method to deal with the case of negative
interest rate. As payoff process of the capped (uncapped) stock loan is a
Markov process, the optimal stopping time must be a hitting time( Remark 3.1
below) from which we guess that $f^{\prime}(0)=0$. In addition, we observe
that the condition do work in the case of negative interest rate but it has no
use to dealing with the case of non-negative interest rate. Based on the
conjecture and observation, we first establish explicitly the value $f(x)$ of
the capped stock loan by a new pure variational inequality method. Then we use
the expression of $f(x)$ to work out the ranges of fair values of parameters
associated with the loan. Finally, as a special case of our main result, we
also get the same conclusion on uncapped stock loan as in Xia and Zhou [13]
proved by a pure probability approach. The paper is organized as follows: In
Section 2, we formulate a mathematical model of capped stock loan and it is
considered as a perpetual American option with a possibly negative interest
rate. In Section 3, we extend standard variational inequality method to the
case of negative interest rate and calculate the initial value and the value
process of the capped(uncapped) stock loan. In Section 4, we work out the
ranges of fair values of parameters associated with the loan based on the
results in previous sections. In Section 5, we present two examples to explain
how the cap impacts on the initial value of uncapped stock loan.

##  2\. Mathematical model

In this section the standard Black-Scholes model in a continuous financial
market consists of two assets: a risky asset stock $S$ and a risk-less bond
$B$. The uncertainty is described by a standard Brownian motion
$\\{\mathbf{W}_{t},t\geq 0\\}$ on a risk-neutral complete probability space
$(\Omega,\mathcal{F},\\{\mathcal{F}_{t}\\}_{t\geq 0},P)$, where
$\\{\mathcal{F}_{t}\\}_{t\geq 0}$ is the filtration generated by $\mathbf{W}$,
$\mathcal{F}_{0}=\sigma\\{\Omega,\emptyset\\}$ and
$\mathcal{F}=\sigma\\{\bigcup_{t\geq 0}\mathcal{F}_{t}\\}$. The risk-less bond
$B$ evolves according to the following dynamic system,

$\displaystyle dB_{t}=rB_{t}dt,\ \ r>0,$

where $r$ is continuously compounding interest rate. The stock price $S$
follows a geometric Brownian motion,

$\displaystyle
S_{t}=S_{0}e^{(r-\delta-\frac{\sigma^{2}}{2})t+\sigma\mathbf{W}_{t}},$ (2.1)

where $S_{0}$ is initial stock price, $\delta\geq 0$ is dividend yield and
$\sigma>0$ is volatility. The discounted payoff process of the capped stock
loan is defined by

$\displaystyle Y(t)=e^{-rt}(S_{t}\wedge Le^{\gamma t}-qe^{\gamma t})_{+}.$

Since $Y(t)\geq 0,\ a.s.$ and $Y(t)>0$ with a positive probability, to avoid
arbitrage, throughout this paper we assume that

$\displaystyle S_{0}-q+c>0.$ (2.2)

According to theory of American contingent claim, the initial value of this
capped stock loan is

$\displaystyle f(x)$ $\displaystyle=$
$\displaystyle\sup\limits_{\tau\in\mathcal{T}_{0}}{\bf
E}\big{[}e^{-r\tau}(S_{\tau}\wedge
Le^{\gamma\tau}-qe^{\gamma\tau})_{+}\big{]}$ (2.3) $\displaystyle=$
$\displaystyle\sup\limits_{\tau\in\mathcal{T}_{0}}{\bf
E}\big{[}e^{-\tilde{r}\tau}(\tilde{S}_{\tau}\wedge L-q)_{+}\big{]},$

where $\tilde{r}=r-\gamma$, $\tilde{S}_{t}=e^{-\gamma
t}S_{t},\tilde{S}_{0}=S_{0}=x$, $L>S_{0}$ and $\mathcal{T}_{0}$ denotes all
$\\{\mathcal{F}_{t}\\}_{t\geq 0}$-stopping times. The value process of this
capped stock loan is

$\displaystyle V_{t}=\sup\limits_{\tau\in\mathcal{T}_{t}}{\bf
E}\big{[}e^{-{r}(\tau-t)}(S_{\tau}\wedge
Le^{\gamma\tau}-qe^{\gamma\tau})_{+}|\mathcal{F}_{t}\big{]},$ (2.4)

i.e.,

$\displaystyle e^{-rt}V_{t}=\sup\limits_{\tau\in\mathcal{T}_{t}}{\bf
E}\big{[}e^{-\tilde{r}\tau}(\tilde{S}_{\tau}\wedge
L-q)_{+}|\mathcal{F}_{t}\big{]},$

where $\mathcal{T}_{t}$ denotes all $\\{\mathcal{F}_{t}\\}_{t\geq 0}$-stopping
times $\tau$ with $\tau\geq t$ a.s.. Since the fair values of $q,\gamma,c,L$
should be such that $f(S_{0})=S_{0}-q+c$, the range of the fair values of the
parameters $(q,\gamma,c,L)$ reduce to calculating the $f(S_{0})$. Because
$\tilde{r}=r-\gamma\leq 0$, the problem is essentially to calculate the
initial value of a conventional perpetual American call option with a possibly
negative interest rate. We have the following.

######  Proposition 2.1.

$(x-q)_{+}\leq f(x)\leq x$ for $x\geq 0$. $f(x)$ is continuous and
nondecreasing on $(0,\infty)$

###### Proof.

Using the same way as in Xia and Zhou [13](2007), we have $(x\wedge
L-q)_{+}\leq f(x)\leq x\wedge L$ for $x\geq 0$. The $f(x)$ is continuous and
nondecreasing on $(0,\infty)$ can be proved by the optional sampling theorem.
∎

##  3\. Variational inequality method in stock loans

In this section we develop a pure variational inequality method in the case of
negative interest rate to establish explicitly the value $f(x)$ of the capped
(uncapped) stock loan. The key point is to replace the initial condition
$f(0)=0$ in the conventional case with a new one $f^{\prime}(0+)=0$. The
detailed observation will be given in Remark 3.1 below. We find that the
condition $f^{\prime}(0+)=0$ holds for the conventional perpetual American
call option but no use in determining free constants. Now we star with the
following.

######  Proposition 3.1.

Assume that $\delta>0$ and $\gamma-r+\delta\geq 0$ or $\delta=0$ and
$\gamma-r>\frac{\sigma^{2}}{2}$. Let $\lambda_{1}$, $\lambda_{2}$ and $\mu$ be
defined by

$\displaystyle\left\\{\begin{array}[]{l l
l}\lambda_{1}=\frac{-\mu+\sqrt{\mu^{2}-2(\gamma-r)}}{\sigma},\
\lambda_{2}=\frac{-\mu-\sqrt{\mu^{2}-2(\gamma-r)}}{\sigma},\\\
\mu=-(\frac{\sigma}{2}+\frac{\gamma-r+\delta}{\sigma}).\end{array}\right.$
(3.3)

(i)If $L\geq b$,
$h(x)\in\mathcal{C}\big{(}[0,\infty)\big{)}\cap\mathcal{C}^{1}\big{(}(0,\infty)\setminus\\{L\\}\big{)}\cap\mathcal{C}^{2}\big{(}(0,\infty)\setminus\\{b,L\\}\big{)}$
and solves the variational inequality

$\displaystyle\left\\{\begin{array}[]{l l l l
l}\frac{1}{2}\sigma^{2}x^{2}h^{{}^{\prime\prime}}+(\tilde{r}-\delta)xh^{{}^{\prime}}-\tilde{r}h=0,&x\in[0,b)\cup(L,\infty),\\\
h(x)=x-q,&x\in[b,L],\\\ h(x)>(x-q)_{+},&x<b,\\\ h(x)\leq x,&x\geq 0,\\\
h^{\prime}(0+)=0,\ h(b)=b-q,h^{{}^{\prime}}(b)=1,\end{array}\right.$ (3.9)

then

$\displaystyle h(x)=\left\\{\begin{array}[]{l l
l}(b-q)(\frac{x}{b})^{\lambda_{1}},&0<x\leq b,\\\ x-q,&b<x<L,\\\
(L-q)(\frac{x}{L})^{\lambda_{2}},&x\geq L.\end{array}\right.$ (3.13)

Moreover,

$\displaystyle b=\frac{q\lambda_{1}}{\lambda_{1}-1}.$ (3.14)

(ii) If $L<b$,
$h(x)\in\mathcal{C}\big{(}[0,\infty)\big{)}\cap\mathcal{C}^{1}\big{(}(0,\infty)\setminus\\{L\\}\big{)}\cap\mathcal{C}^{2}\big{(}(0,\infty)\setminus\\{L\\}\big{)}$
and solves the variational inequality

$\displaystyle\left\\{\begin{array}[]{l l l
l}\frac{1}{2}\sigma^{2}x^{2}h^{{}^{\prime\prime}}+(\tilde{r}-\delta)xh^{{}^{\prime}}-\tilde{r}h=0,&x\in[0,L)\cup(L,\infty),\\\
h(x)>(x-q)_{+},&x<L,\\\ h(x)\leq x,&x\geq 0,\\\
h^{\prime}(0+)=0,h(L)=L-q,\end{array}\right.$ (3.19)

then

$\displaystyle h(x)=\left\\{\begin{array}[]{l l
l}(L-q)(\frac{x}{L})^{\lambda_{1}},&0<x<L,\\\ L-q,&x=L,\\\
(L-q)(\frac{x}{L})^{\lambda_{2}},&x>L.\end{array}\right.$ (3.23)

###### Proof.

We only deal with the part (i). The part (ii) can be treated similarly.
Solving the first second-order differential equation of the(3.9), we get

$\displaystyle h(x)=\left\\{\begin{array}[]{l l
l}C_{1}x^{\lambda_{1}}+C_{2}x^{\lambda_{2}},&x\in[0,b),\\\ x-q,&x\in[b,L],\\\
C_{3}x^{\lambda_{1}}+C_{4}x^{\lambda_{2}},&x\in(L,\infty),\end{array}\right.$
(3.27)

where $C_{i}$, $i=1,2,3,4$, and $b$ are free constants to be determined,
$\lambda_{1}$ and $\lambda_{2}$ are defined by (3.3). Note that if $\delta>0$
then $\lambda_{1}>1>\lambda_{2}>0$, and if $\delta=0$ and
$\gamma-r>\frac{\sigma^{2}}{2}$ then
$\lambda_{1}=\frac{2(\gamma-r)}{\sigma^{2}}>1=\lambda_{2}$. Using
$h^{\prime}(0+)=0$ and $h(x)<x$, we have $C_{2}=C_{3}=0$. Substituting
$C_{2}=C_{3}=0$ into (3.27) and applying $h(b)=b-q$, $h^{\prime}(b)=1$ as well
as the principle of smooth fit in the resulting expression of $h(x)$ yields
that

$b-q=C_{1}b^{\lambda_{1}},\quad
L-q=C_{4}L^{\lambda_{2}},\quad\lambda_{1}C_{1}b^{\lambda_{1}-1}=1.$

Solving this system for $b$, $C_{1}$ and $C_{4}$, we get
$b=\frac{q\lambda_{1}}{\lambda_{1}-1}$, $C_{1}=\frac{b-q}{b^{\lambda_{1}}}$
and $C_{4}=\frac{L-q}{L^{\lambda_{2}}}$. Hence the equations (3.13) and (3.14)
follow. Since $\lambda_{1}>1\geq\lambda_{2}>0$ , the function $h(x)$ defined
by (3.13) obviously belongs to
$\mathcal{C}\big{(}[0,\infty)\big{)}\cap\mathcal{C}^{1}\big{(}(0,\infty)\setminus\\{L\\}\big{)}\cap\mathcal{C}^{2}\big{(}(0,\infty)\setminus\\{b,L\\}\big{)}.$
Thus we complete the proof. ∎

######  Remark 3.1.

From (3.3) we know that if $\widetilde{r}=r-\gamma>0$( the conventional case )
then $\lambda_{1}\geq 1>0$ and $\lambda_{2}<0$. Consequently, we see from
(3.27) that the function $h(x)=C_{2}x^{\lambda_{2}}$ is rejected by the
initial condition $h(0)=0$, i.e., $C_{2}=0$. So by the same way as in proof of
Proposition 3.1 above, we can determine other free constants $C_{1}$, $C_{3}$,
$C_{4}$ and $b$. If $\widetilde{r}=r-\gamma\leq 0$( the present case ) then
$\lambda_{1}>1\geq\lambda_{2}>0$. As opposed to the conventional case, we can
not deduce from $h(0)=0$ that $C_{2}=0$. This is the major difficulty
appearing in variational inequality method in the case of stock loan. However,
we note that in present case
$h^{\prime}(x)=\lambda_{1}C_{1}x^{\widetilde{\lambda_{1}}}+\lambda_{2}C_{2}x^{\widetilde{\lambda_{2}}}$
with $\widetilde{\lambda_{1}}=\lambda_{1}-1>0$ and
$\widetilde{\lambda_{2}}=\lambda_{2}-1<0$. So if $h^{\prime}(0+)=0$, the
present case can be reduced to the conventional case. On the other hand,
Markov property implies that the optimal stopping time must be a hitting time.
In view of the fact, we conjecture that $h^{\prime}(0+)=0$ is correct. The
proof of the conjecture will be given in Proposition 3.2 below. Comparing with
the conventional case, the $h^{\prime}(0+)=0$ will play an important role in
developing pure variational inequality method in the case of stock loan.

######  Proposition 3.2.

Assume that $\delta=0$, $\gamma-r>\frac{\sigma^{2}}{2}$ and $g(x)={\bf
E}\big{[}e^{-\tilde{r}(\tau\wedge\varsigma)}(\tilde{S}_{\tau\wedge\varsigma}-q)_{+}I_{\\{\varsigma<\infty\\}}\big{]}$
for $b\geq q$, $L\geq 0$, stoping times $\tau$ and $\varsigma$. Then
$g^{\prime}(0+)=0$.

###### Proof.

Using $g(0)=0$, we have for $x>0$

$\displaystyle 0$ $\displaystyle\leq$ $\displaystyle\frac{g(x)-g(0)}{x-0}$
(3.28) $\displaystyle=$ $\displaystyle{\bf
E}\large\\{\large(\exp\\{-\frac{1}{2}\sigma^{2}\tau\wedge\varsigma+\sigma\mathbf{W}_{\tau\wedge\varsigma}\\}$
$\displaystyle\qquad-\frac{q}{x}\exp\\{(\gamma-r)\tau\wedge\varsigma\\}\large)_{+}I_{\\{\varsigma<+\infty\\}}\large\\}$
$\displaystyle\leq$ $\displaystyle{\bf
E}\large\\{\large(\exp\\{-\frac{1}{2}\sigma^{2}\tau\wedge\varsigma+\sigma\mathbf{W}_{\tau\wedge\varsigma}\\}-\frac{q}{x}\large)_{+}I_{\\{\varsigma<+\infty\\}}\large\\}.$

Note that
$\large\\{\exp\\{-\frac{1}{2}\sigma^{2}t+\sigma\mathbf{W}_{t}\\},t\geq
0\large\\}$ is a strong martingale w.r.t. $\\{\mathcal{F}_{t}\\}_{t\geq 0}$.
It follows that

$\displaystyle{\bf
E}\large\\{\exp\\{-\frac{1}{2}\sigma^{2}\tau\wedge\varsigma+\sigma\mathbf{W}_{\tau\wedge\varsigma}\\}\large\\}=1.$
(3.29)

To let $x\rightarrow 0$ in (3.28), we need to dominate the right-hand side.
Because

$\displaystyle\Large(\exp\\{-\frac{1}{2}\sigma^{2}\tau\wedge\varsigma+\sigma\mathbf{W}_{\tau\wedge\varsigma}\\}-\frac{q}{x}\Large)_{+}I_{\\{\varsigma<+\infty\\}}\leq\exp\\{-\frac{1}{2}\sigma^{2}\tau\wedge\varsigma+\sigma\mathbf{W}{\tau\wedge\varsigma}\\},$
$\displaystyle\lim_{x\rightarrow
0+}\Large(\exp\\{-\frac{1}{2}\sigma^{2}\tau\wedge\varsigma+\sigma\mathbf{W}_{\tau\wedge\varsigma}\\}-\frac{q}{x}\Large)_{+}I_{\\{\varsigma<+\infty\\}}=0.\quad
a.s.,$

and (3.29), the dominated convergence theorem allows us to let $x\rightarrow
0$ in (3.28) and obtain

$\displaystyle 0\leq\lim_{x\rightarrow
0+}\frac{g(x)-g(0)}{x-0}\leq\lim_{x\rightarrow 0+}{\bf
E}\large\\{\big{(}\exp\\{-\frac{1}{2}\sigma^{2}\tau\wedge\varsigma+\sigma\mathbf{W}_{\tau\wedge\varsigma}\\}-\frac{q}{x}\big{)}_{+}I_{\\{\varsigma<+\infty\\}}\large\\}=0.$

Thus $g^{{}^{\prime}}(0+)=0$. ∎

In view of Proposition 3.1 and variational inequality method, $\tau_{b}$ must
be the optimal time, where $b$ is defined by (3.14). Now we show the fact.

######  Proposition 3.3.

Let $b$ and $h(x)$ be defined by (3.14), (3.13) and (3.23 ) respectively,
$E(x)\equiv{\bf
E}\big{[}e^{-\tilde{r}(\tau_{b}\wedge\tau_{L})}(\tilde{S}_{\tau_{b}\wedge\tau_{L}}\wedge
L-q)_{+}I_{\\{\tau_{L}<\infty\\}}\big{]}$ for $L\geq 0$, where $x=S_{0}$,
$\tau_{b}=\inf{\\{t\geq 0:e^{-\gamma t}S_{t}\geq b\\}}$ and
$\tau_{L}=\inf{\\{t\geq 0:e^{-\gamma t}S_{t}=L\\}}$. Then $E(x)=h(x)$.

###### Proof.

Using formula 2.20.3 in Section 9, Part II of [1](2000), we have

$\displaystyle{\bf
E}\big{[}e^{-\tilde{r}(\tau_{b}\wedge\tau_{L})}I_{\\{\tau_{b}\wedge\tau_{L}<\infty\\}}\big{]}=\left\\{\begin{array}[]{l
l}(\frac{x}{b})^{\lambda_{1}},x<b,\\\ (\frac{x}{L})^{\lambda_{2}},x\geq
L.\end{array}\right.$ (3.32)

Based on the (3.32), the proof of Proposition 3.3 will be accomplished in two
cases, namely, $L>b$ and $L\leq b$. Case of $L>b$.  We shall distinguish three
subcases, i.e., $x<b$, $L\geq x\geq b$ and $x\geq L$. If $x<b$ then
$\tau_{b}\leq\tau_{L}$. Using(3.32), we calculate

$\displaystyle E(x)$ $\displaystyle=$ $\displaystyle{\bf
E}\big{[}e^{-\tilde{r}(\tau_{b}\wedge\tau_{L})}(\tilde{S}_{\tau_{b}\wedge\tau_{L}}\wedge
L-q)_{+}I_{\\{\tau_{L}<\infty\\}}\big{]}$ (3.33) $\displaystyle=$
$\displaystyle{\bf
E}\big{[}e^{-\tilde{r}(\tau_{b}\wedge\tau_{L})}(\tilde{S}_{\tau_{b}}\wedge
L-q)_{+}I_{\\{\tau_{L}<\infty\\}}\big{]}$ $\displaystyle=$
$\displaystyle(b-q)(\frac{x}{b})^{\lambda_{1}}.$

If $L\geq x\geq b$ then $\tau_{b}=0$ and so

$\displaystyle E(x)$ $\displaystyle=$ $\displaystyle{\bf
E}\big{[}e^{-\tilde{r}(\tau_{b}\wedge\tau_{L})}(\tilde{S}_{\tau_{b}\wedge\tau_{L}}\wedge
L-q)_{+}I_{\\{\tau_{L}<\infty\\}}\big{]}$ (3.34) $\displaystyle=$
$\displaystyle{\bf E}\big{[}(\tilde{S}_{0}\wedge L-q)_{+}\big{]}=(x-q).$

If $x>L$ then $\tilde{S}_{\tau_{b}\wedge\tau_{L}}\geq L$. By using (3.32),

$\displaystyle E(x)$ $\displaystyle=$ $\displaystyle{\bf
E}\big{[}e^{-\tilde{r}(\tau_{b}\wedge\tau_{L})}(\tilde{S}_{\tau_{b}\wedge\tau_{L}}\wedge
L-q)_{+}I_{\\{\tau_{L}<\infty\\}}\big{]}$ (3.35) $\displaystyle=$
$\displaystyle(L-q){\bf
E}\big{[}e^{-\tilde{r}(\tau_{b}\wedge\tau_{L})}I_{\\{\tau_{L}<\infty\\}}\big{]}$
$\displaystyle=$ $\displaystyle(L-q)(\frac{x}{L})^{\lambda_{2}}.$

Comparing with (3.13), the equations (3.33), (3.34) and (3.35) yield that
$E(x)=h(x)$. Case of $b\geq L$. We shall distinguish two subcases, i.e.,$x<L$
and $x\geq L$. If $x<L$ then $\tau_{b}\geq\tau_{l}$ . Using (3.32), we have

$\displaystyle E(x)$ $\displaystyle=$ $\displaystyle{\bf
E}\big{[}e^{-\tilde{r}(\tau_{b}\wedge\tau_{L})}(\tilde{S}_{\tau_{b}\wedge\tau{L}}\wedge
L-q)_{+}I_{\\{\tau_{L}<\infty\\}}\big{]}$ (3.36) $\displaystyle=$
$\displaystyle{\bf
E}\big{[}e^{-\tilde{r}(\tau_{b}\wedge\tau_{L})}(\tilde{S}_{\tau{L}}\wedge
L-q)_{+}I_{\\{\tau_{L}<\infty\\}}\big{]}$ $\displaystyle=$
$\displaystyle(L-q)(\frac{x}{L})^{\lambda_{1}}.$

If $x\geq L$ then $\tilde{S}_{\tau_{b}\wedge\tau_{L}}\wedge L=L$. So by using
(3.32)

$\displaystyle E(x)$ $\displaystyle=$ $\displaystyle{\bf
E}\big{[}e^{-\tilde{r}(\tau_{b}\wedge\tau_{L})}(\tilde{S}_{\tau_{b}\wedge\tau_{L}}\wedge
L-q)_{+}I_{\\{\tau_{L}<\infty\\}}\big{]}$ (3.37) $\displaystyle=$
$\displaystyle(L-q){\bf
E}\big{[}e^{-\tilde{r}(\tau_{b}\wedge\tau_{L})}I_{\\{\tau_{L}<\infty\\}}\big{]}$
$\displaystyle=$ $\displaystyle(L-q)(\frac{x}{L})^{\lambda_{2}}.$

Comparing the equations (3.36) and (3.37) with the (3.23), we see that
$E(x)=h(x)$. ∎

We now return to main result of this section. It states that the initial value
of the capped stock loan is just $f(x)$ defined by (3.13) and (3.23)
respectively, where $x=S_{0}$.

######  Theorem 3.1.

Assume that $\delta>0$ and $\gamma-r+\delta\geq 0$ or $\delta=0$ and
$\gamma-r>\frac{\sigma^{2}}{2}$. Let $h(x)$ be defined by (3.13) and
(3.23)respectively, and $f(x)$ be defined by (2.3). Then $f(x)=h(x)$ for
$x\geq 0$, and $\tau_{b}\wedge\tau_{L}$ is the optimal stopping time.

###### Proof.

In view of Proposition 3.3,

$f(x)\equiv\sup\limits_{\tau\in\mathcal{T}_{0}}{\bf
E}\big{[}e^{-\tilde{r}\tau}(\tilde{S}_{\tau}\wedge
L-q)_{+}I_{\\{\tau_{L}<\infty\\}}\big{]}\geq h(x).$

Therefore we only need to prove that for any stopping time $\tau$

$\displaystyle h(x)\geq{\bf
E}\big{[}e^{-\tilde{r}(\tau\wedge\tau_{L})}(\tilde{S}_{\tau\wedge\tau_{L}}\wedge
L-q)_{+}I_{\\{\tau_{L}<\infty\\}}\big{]}.$ (3.38)

From Proposition 3.1 and the expression of $h(x)$ as well as
$\lambda_{1}>1\geq\lambda_{2}>0$, we know that
$h(x)\in\mathcal{C}\big{(}[0,\infty)\big{)}\cap\mathcal{C}^{1}\big{(}(0,\infty)\setminus\\{L\\}\big{)}\cap\mathcal{C}^{2}\big{(}(0,\infty)\setminus\\{b,L\\}\big{)}$
or
$h(x)\in\mathcal{C}\big{(}[0,\infty)\big{)}\cap\mathcal{C}^{1}\big{(}(0,\infty)\setminus\\{L\\}\big{)}\cap\mathcal{C}^{2}\big{(}(0,\infty)\setminus\\{L\\}\big{)}$
and $0<h^{\prime\prime}(b\pm)<+\infty$, $0<h^{\prime}(L\pm)<+\infty$ and
$0<h^{\prime\prime}(L\pm)<+\infty$. As a result, the generalized Itô’s
formula(cf.Karatzas and Shreve[5](1991) for Problem 6.24, p.215 ), the
inequalities (3.9) and (3.19) yield

$\displaystyle\int_{0}^{t}d(e^{-\tilde{r}s}h(\tilde{S}_{s}))$ $\displaystyle=$
$\displaystyle\int_{0}^{t}e^{-\tilde{r}s}\tilde{S}_{s}h^{{}^{\prime}}(\tilde{S}_{s})\sigma
d\mathcal{W}(s)$ (3.39)
$\displaystyle+\int_{0}^{t}e^{-\tilde{r}s}(\tilde{r}q-\delta\tilde{S}_{s})I_{\\{L>\tilde{S}_{s}\geq
b^{*}\\}}ds$
$\displaystyle+[h^{{}^{\prime}}(L+)-h^{{}^{\prime}}(L-)]\int_{0}^{t}e^{-\tilde{r}s}dL_{s}(L)$
$\displaystyle\equiv$ $\displaystyle\mathcal{M}(t)+A_{1}(t)+\Lambda(t),$

where $L_{t}(L)\geq 0$ is the local time of $\tilde{S}_{t}$ at the point $L$,
$\mathcal{M}(t)=\int_{0}^{t}e^{-\tilde{r}s}\tilde{S}_{s}h^{{}^{\prime}}(\tilde{S}_{s})\sigma
d\mathcal{W}(s)$ is a martingale,

$\displaystyle A_{1}(t)$ $\displaystyle=$
$\displaystyle\int_{0}^{t}e^{-\tilde{r}s}(\tilde{r}q-\delta\tilde{S}_{s})I_{\\{L>\tilde{S}_{s}\geq
b^{*}\\}}ds\qquad\mbox{and}$ $\displaystyle\Lambda(t)$ $\displaystyle=$
$\displaystyle\ \
[h^{{}^{\prime}}(L+)-h^{{}^{\prime}}(L-)]\int_{0}^{t}e^{-\tilde{r}s}dL_{s}(L).$

Define $T_{n}=t\wedge\tau\wedge\tau_{L}-\frac{1}{n}$ for $\forall t\geq 0$ ,
$n\geq 1$ and stopping time $\tau$. Then by using definition of $\tau_{L}$ and
equality $L_{t}(L)=\int^{t}_{0}I_{\\{s:\tilde{S}_{s}=L\\}}dL_{s}(L)$, we have
$\Lambda(T_{n})=0$. Moreover, since $\tilde{r}\leq 0$ and $\delta\geq 0$, we
see that $A_{1}(T_{n})\leq 0$. So by (3.39)

$\displaystyle h(\tilde{S}_{0})$ $\displaystyle=$ $\displaystyle{\bf
E}\big{[}e^{-\tilde{r}T_{n}}h(\tilde{S}_{T_{n}})I_{\\{\tau_{L}<\infty\\}}\big{]}+{\bf
E}\big{[}-A_{1}(T_{n})I_{\\{\tau_{L}<\infty\\}}\big{]}$ $\displaystyle\geq$
$\displaystyle{\bf
E}\big{[}e^{-\tilde{r}T_{n}}h(\tilde{S}_{T_{n}})I_{\\{\tau_{L}<\infty\\}}\big{]}$
$\displaystyle=$ $\displaystyle{\bf
E}\big{[}e^{-\tilde{r}T_{n}}h(\tilde{S}_{T_{n}})I_{\\{\tau_{L}<\infty\\}}\big{]}$
$\displaystyle\geq$ $\displaystyle{\bf
E}\big{[}e^{-\tilde{r}T_{n}}(\tilde{S}_{T_{n}}\wedge
L-q)_{+}I_{\\{\tau_{L}<\infty\\}}\big{]}.$

Note that

$\displaystyle e^{-\tilde{r}T_{n}}(\tilde{S}_{T_{n}}\wedge
L-q)_{+}I_{\\{\tau_{L}<\infty\\}}\leq\sup\limits_{0\leq
t<\infty}e^{-\tilde{r}t}(\tilde{S}_{t}-q)_{+}$

and

${\bf E}\big{[}\sup\limits_{0\leq
t<\infty}e^{-\tilde{r}t}(\tilde{S}_{t}-q)_{+}\big{]}<\infty.$

The dominated convergence theorem now implies

$\displaystyle h(x)=h(\tilde{S}_{0})$ $\displaystyle\geq$
$\displaystyle\lim_{n\rightarrow+\infty}{\bf
E}\big{[}e^{-\tilde{r}T_{n}}(\tilde{S}_{T_{n}}\wedge
L-q)_{+}I_{\\{\tau_{L}<\infty\\}}\big{]}$ $\displaystyle=$ $\displaystyle{\bf
E}\big{[}e^{-\tilde{r}(\tau\wedge\tau_{L})}(\tilde{S}_{\tau\wedge\tau_{L}}\wedge
L-q)_{+}I_{\\{\tau_{L}<\infty\\}}\big{]}.$

Thus $h(x)=f(x)$ and $\tau_{b}\wedge\tau_{L}$ is the optimal stopping time. ∎

As a direct consequence of Theorem 3.1, i.e., $L=+\infty$, we can get the
following result proved by a pure probability approach in Xia and Zhou[13].

######  Corollary 3.1.

Let $f(x)=\sup\limits_{\tau\in\mathcal{T}_{0}}{\bf
E}\big{[}e^{-\tilde{r}\tau}(\tilde{S}_{\tau}-q)_{+}\big{]}$, $b$ and
$\lambda_{1}$ be defined by (3.14) and (3.3) respectively. Then

$\displaystyle f(x)=\left\\{\begin{array}[]{l l
l}(b-q)(\frac{x}{b})^{\lambda_{1}},&0<x\leq b,\\\ x-q,&x>b\end{array}\right.$
(3.42)

and $\tau_{b}$ is the optimal stoping time.

######  Remark 3.2.

Comparing with the value of capped American option with non-negative interest
rate(cf.Broadie and Detemple [2](1995) ), we see that the value of capped
stock loan treated in the present paper has different behaviors. If the stock
price is bigger than $L$ then the capped American option with non-negative
interest rate should be exercise immediately. While the capped stock loan
treated in the present paper has no this kind of performances.

######  Remark 3.3.

Let $V_{t}$ be the value process for the capped stock loan defined by (2.4).
Then by using Markov property of the process $S_{t}$ (cf. Karatzas and
Shreve[6](1998)), we easily get

$\displaystyle e^{-\tilde{r}t}f(\tilde{S}_{t})$ $\displaystyle=$
$\displaystyle\sup\limits_{\tau\in\mathcal{T}_{t}}{\bf
E}\big{[}e^{-\tilde{r}(\tau\wedge\tau_{L})}(\tilde{S}_{\tau\wedge\tau_{L}}\wedge
L-q)_{+}I_{\\{\tau_{L}<\infty\\}}|\mathcal{F}_{t}\big{]}$ $\displaystyle=$
$\displaystyle e^{-rt}V_{t}$

and $V_{t}=e^{\gamma t}f(e^{-\gamma t}S_{t})$.

##  4\. Ranges of fair values of parameters

In this section we will work out the ranges of fair values of the parameters
$(q,\gamma,c,L)$ of the capped stock loan treated in this paper based on
Theorem 3.1 and equality $f(S_{0})=S_{0}-q+c$. In view of Proposition3.1,
there are two cases to be dealt with. The first is when $L>b$ and the second
case is when $L\leq b$. We only consider the first case, the second case can
be treated similarly. We shall distinguish three subcases, i.e., $S_{0}\geq
L$, $L>S_{0}\geq b$ and $0<S_{0}\leq b$. Case of $S_{0}\geq L$. By (3.13) and
$f(S_{0})=S_{0}-q+c$, we have
$f(S_{0})=(L-q)(\frac{S_{0}}{L})^{\lambda_{2}}=S_{0}-q+c$. So the parameters
$\gamma$, $q$, $c$ and $L$ must satisfy
$c=(L-q)(\frac{S_{0}}{L})^{\lambda_{2}}+q-S_{0}\leq 0$. This means that the
bank wants to earn maximal profit because the initial stock price is very
high. The bank gets the stock via paying the money $c$ to the client, the
client gives the stock to the bank due to the money $c$. Therefore both the
client and the bank have incentives to do the business. Actually, $\tau_{L}$
is the optimal stopping time for the client to terminate the capped stock
loans. Case of $L>S_{0}\geq b$. By (3.13) and $f(S_{0})=S_{0}-q+c$, we have
$f(S_{0})=S_{0}-q=S_{0}-q+c$, so $c=0$, i.e., the client has no incentive to
do the transaction. $\tau_{b}=0$ in this case is the optimal stopping time.
Case of $0<S_{0}\leq b$. In this case both the client and the bank have
incentives to do the business. The bank does because there is dividend payment
and so does the client because the initial stock price is neither very high
nor too low. By Theorem 3.1, the initial value is $f(S_{0})$. In this case the
bank charges an amount $c=f(S_{0})-S_{0}+q$ from the client for providing the
service. So the fair values of the parameters $\gamma,q$ and $c$ must satisfy

$\displaystyle S_{0}-q+c=(b-q)(\frac{x}{b})^{\lambda_{1}}$

and the optimal stopping time is $\tau_{b}$.

##  5\. Examples

In this section we will give two examples of capped stock loan as follows.

######  Example 5.1.

Let the risk free rate $r=0.05$, the loan rate $\gamma=0.07$, the volatility
$\sigma=0.15$, the dividend $\delta=0.01$, the principal $q=100$ and the cap
$L=240$. Then $b=147.8<L$. We compute the initial value $f(x)$ of capped stock
loan as in the following Figure 1. The graph obviously shows that the cap
greatly impacts on the initial value when the stock price is very large. The
cap reduces the value of the stock loan and the client can acquires more
liquidity.

Figure 1. $\gamma=0.07,r=0.05,\sigma=0.15,\delta=0.01,q=100,cap=240$

######  Example 5.2.

Let $r$, $\sigma$, $\sigma$, $\delta$ and $q$ be as the same as in Example
5.1, $L=120$. Then $L<b=147.8$. We compute the initial value $f(x)$ of capped
stock loan as in the following Figure 2. The graph explains that the initial
value of caped stock loan greatly decreases because the cap is less than $b$.
The client will acquire much more liquidity compared to the uncapped stock
loan.

Figure 2. $\gamma=0.07,r=0.05,\sigma=0.15,\delta=0.01,q=100,cap=120$

Acknowledgements. We are very grateful to Professor Jianming Xia for his
conversation with us and providing original paper of [13] for us. We also
express our deep thanks to Professor Xun Yu Zhou for providing power point
files of his talk on stock loan at Peking University for us. Special thanks
also go to the participants of the seminar stochastic analysis and finance at
Tsinghua University for their feedbacks and useful conversations. This work is
supported by Project 10771114 of NSFC, Project 20060003001 of SRFDP, and SRF
for ROCS, SEM, and the Korea Foundation for Advanced Studies. We would like to
thank the institutions for the generous financial support.

## References

  * [1] Borodin, A. N., Salminen, P., 2002. Handbook of Brownian Motion-Facts and Formulae. Probability and its Applications, Birkh$\ddot{a}$user Verlag, Basel, 2nd edition.
  * [2] Broadie, M., Detemple, J., 1995. American capped call options on dividend-paying asset . The Review of Finance Studies, Vol. 8, No. 1, pp. 161-191.
  * [3] Dayanik, S., Karatzas, I., 2003. On the optimal stopping problems for one-dimensional diffusions. Stochastic Process. Appl., 107, no. 2, 173-212.
  * [4] Jiang, S., Liang, Z., Wu, W., 2008. Stock loan with automatic termination clause. Preprint(11, 2008).
  * [5] Karatzas, I., Shreve,S. E., 1991. Brownian motion and stochastic calculus. Springer-Verlag, New York.
  * [6] Karatzas, I., Shreve,S. E., 1998. Methods of mathematical finance. Springer-Verlag, New York.
  * [7] Kifer,Y., 2000. Game Options. Finance and Stochastics, 4:443-463.
  * [8] Kyprianou, A.E., 2004. Some calculations for Israeli options. Finance and Stochastics, 8:73-86.
  * [9] Lions, P.-L., Sznitman, A.S.. 1984. Stochastic differential equations with reflecting boundary conditions. Comm.Pure Appl. Math.37, 511-537.
  * [10] McKean,H.P.JR., 1965. A free-boundary problem for the heat equation arising from a problem in mathematical economics. Industr. Manag. Rev., 6, 32-39. Appendix to Samuelson(1965a).
  * [11] $\varnothing$ksendal,B. and Sulem,A., 2005\. Applied Stochastic Control of Jump Diffusions. Springer.
  * [12] Shiryaev,A.N., Kabanov,Y.M., Kramkov, D.O. and Melnikov, A.V., 1994. Towards the theory of options of both European and American types II. Theory Prob.Appl, 39,61-102.
  * [13] Xia, J.M., Zhou,X.Y., 2007\. Stock loans. Mathematical Finance, Vol.17, No.2, 307-317.


: Self-supervised 3D Motion Estimation of Debris Flow

Liyuan Zhu1   Yuru Jia1   Shengyu Huang1   Nicholas Meyer1

Andreas Wieser1   Konrad Schindler1   Jordan Aaron2,3

1Institute of Geodesy and Photogrammetry, ETH Zurich

2Swiss Federal Institute for Forest, Snow and Landscape Research WSL

3Geological Institute, ETH Zurich

Acknowledgements Funding for this project was provided in part by an Swiss National Science Foundation grant to J. Aaron (p 193081).  The authors further thank Stefan Boss, Christoph Graf, Brian McArdell and Raffaele Spielmann for interesting discussions, as well as support in setting up and running the video camera and LiDAR systems.

§ APPENDIX

§.§ Dataset

In this section, we provide supplementary information and visualizations of the debris flow dataset. The representative shapes of the debris-flow surface are summarized in <ref>. The terrain of the monitoring site is shown in <ref>.

In the debris flow dataset, we release:

* 6000 high-resolution images (1920$\times$1080)

* 6000 high-accuracy point clouds

* Camera calibration matrices

* Rotation matrices between camera and LiDAR

* Translation vectors between camera and LiDAR

§.§ Evaluation Metrics

SSIM Structural similarity index <cit.> is used in our optical flow loss to assess the similarity between the target image and warped image:

\begin{equation}
\operatorname{SSIM}(x, y)=\frac{\left(2 \mu_x \mu_y+c_1\right)\left(2 \sigma_{x y}+c_2\right)}{\left(\mu_x^2+\mu_y^2+c_1\right)\left(\sigma_x^2+\sigma_y^2+c_2\right)},
\end{equation}

where $\mu_x$ and $\mu_y$ represent the mean pixel values of images $x$ and $y$, and $\sigma$ denotes the corresponding standard deviation.

Census Transform Loss We use ternary census transform loss <cit.> as the second metric to evaluate optical flow performance:

\begin{equation}
\mathbf{CT}\left(p, p^{\prime}\right)=
\begin{cases} -1 & \text { if } p^{\prime}-p\geq \epsilon \\
+1 & \text { if } p - p^{\prime} \geq \epsilon \\
0 & \text { if } |p - p^{\prime}|<\epsilon
\end{cases}
\end{equation}

Given two input images, we compute the corresponding census-transformed images and compute the average difference between them as the loss.

§.§ LiDAR to Range Image

Since point cloud-based networks <cit.> are computationally demanding and complex to train, we convert the 3D scan points to sparse range maps with the help of the camera-LiDAR transformation by projecting 3D points onto the image plane with known camera intrinsics $\K$ and camera pose parameters $\mathbf{R}$ and $\mathbf{t}$:

\begin{equation}
\mathbf{K}[\mathbf{R} \mid \mathbf{t}]
f & 0 & p_x \\
0 & f & p_y \\
0 & 0 & 1
\end{array}\right]\left[\begin{array}{llll}
r_1 & r_2 & r_3 & t_1 \\
r_4 & r_5 & r_6 & t_2 \\
r_7 & r_8 & r_9 & t_3
\end{array}\right].
\end{equation}

The 2D projection $\mathbf{p}=(u,v,1)^T$ of 3D point $\mathbf{P}=(x,y,z,1)^T$ is computed as

\begin{equation}
\mathbf{p} = \mathbf{K}[\mathbf{R} \mid \mathbf{t}]\mathbf{P}
\end{equation}

and rounded to the closest integer pixel coordinate.

§.§ Runtime and Model Size

We report the runtimes and model sizes of RAFT, DeFlow-Cam, and DeFlow-Fusion in <ref>.
Our camera-only baseline
is significantly smaller and faster,since we follow the lightweight design of PWC-Net <cit.> and Mono-SF <cit.>. Our fusion model has similar model size and runtime as RAFT. The increase in runtime and model size compared to the camera-only baseline  is caused by the additional depth encoder and the multi-level feature fusion, which in return offer marked performance gains.

§.§ Additional Qualitative Results

In <ref>, we present additional visualizations of optical flow results, and of the static pixel masks used to enforce a static sensor pose.
To see the qualitative behaviour of our temporal smoothing module, readers are encouraged to watch the video in the supplementary material.

Qualitative results of optical flow (c) and binary (static/dynamic) segmentation (d).

Flow and depth trade-off
We design our network to jointly estimate the depth and optical flow, which is a multi-task learning problem. We trade off the weights between $\lambda_{\text{flow}}$ and $\lambda_{\text{depth}}$ ($\lambda_{\text{flow}}+\lambda_{\text{depth}} = 1$), evaluate model performance accordingly and summarize the results in <ref>. As expected, reducing $\lambda_{\text{flow}}$ results in a corresponding decline in the performance of the optical flow component. This is not surprising given that the training of self-supervised optical flow is complex and time-consuming, and the optical flow branch dominates the encoder and decoder, while the depth branch is trained more stably with direct supervision from LiDAR. For future applications of our approach, the trade-off between these two components can be determined based on the specific analysis requirements, such as surface motion or flow structure.


Date of publication xxxx 00, 0000, date of current version xxxx 00, 0000.
10.1109/ACCESS.2017.DOI

# SemAttNet: Towards Attention-based Semantic Aware Guided Depth Completion

DANISH NAZIR1  MARCUS LIWICKI 2, DIDIER STRICKER 1, MUHAMMAD ZESHAN AFZAL 1
Deutsches Forschungszentrum für Künstliche Intelligenz (DFKI) Trippstadter
Str. 122, 67663 Kaiserslautern (e-mail<EMAIL_ADDRESS> Luleå University
of Technology (e-mail<EMAIL_ADDRESS>
###### Abstract

Depth completion involves recovering a dense depth map from a sparse map and
an RGB image. Recent approaches focus on utilizing color images as guidance
images to recover depth at invalid pixels. However, color images alone are not
enough to provide the necessary semantic understanding of the scene.
Consequently, the depth completion task suffers from sudden illumination
changes in RGB images (e.g., shadows). In this paper, we propose a novel
three-branch backbone comprising color-guided, semantic-guided, and depth-
guided branches. Specifically, the color-guided branch takes a sparse depth
map and RGB image as an input and generates color depth which includes color
cues (e.g., object boundaries) of the scene. The predicted dense depth map of
color-guided branch along-with semantic image and sparse depth map is passed
as input to semantic-guided branch for estimating semantic depth. The depth-
guided branch takes sparse, color, and semantic depths to generate the dense
depth map. The color depth, semantic depth, and guided depth are adaptively
fused to produce the output of our proposed three-branch backbone. In
addition, we also propose to apply semantic-aware multi-modal attention-based
fusion block (SAMMAFB) to fuse features between all three branches. We further
use CSPN++ with Atrous convolutions to refine the dense depth map produced by
our three-branch backbone. Extensive experiments show that our model achieves
state-of-the-art performance in the KITTI depth completion benchmark at the
time of submission.

###### Index Terms:

State-of-the-art Depth Completion

=-15pt

##  I Introduction

Figure 1: Block diagram of SemAttNet. In the first stage, the multi-modal data
consisting of RGB, semantic, and LiDAR sparse depth maps are passed to our
novel three-branch backbone, which outputs a fused dense depth map. Our
proposed backbone utilize Semantic-aware multi-model attention-based fusion
(SAMMAFB) to adaptively fuse features between RGB, semantic and sparse depth
branches. In the second stage, the output of the three-branch backbone is
further refined through CSPN++ with Atrous convolutions.

Estimation of dense depth measurements is crucial in various 3D vision and
robotics applications such as augmented and mixed reality [1, 2], scene
reconstruction [3, 4], autonomous driving [5, 6], and obstacle avoidance [7,
8]. For obtaining a reliable prediction of depth in outdoor scenes,
measurements from various sensors are used. Most commonly used sensors include
RGB cameras in a stereo setting, light detection and ranging (LiDAR), and time
of flight (ToF) cameras[9]. Among all the sensors, LiDAR is considered the
most reliable and efficient, and it produces accurate depth measurements in an
outdoor setting [8]. However, the density of LiDAR depth measurements is
sparse, with a considerable amount of missing depth data[10, 11]. For example,
the LiDAR sensor for mobile applications [8], Velodyne HDL-64e, which is also
used in the KITTI [10, 11] dataset produces depth maps consisting of valid
depth values only on $5.9\%$ pixels. Such sparse depth maps cannot be utilized
directly in the application areas mentioned above. Therefore, dense depth maps
estimation from sparse measurements is of utmost importance. It is considered
a challenging problem since the measured depth values only form $5.9\%$ of the
complete depth map.

Recent methods [12, 13, 14, 15, 16, 17, 18, 19, 20] employ the deep learning-
based approaches to dense depth completion. These methods utilize
convolutional neural networks and combine sparse LiDAR data with different
modalities, e.g. RGB images [13, 14], semantic maps [16, 21, 22], and surface
normal’s [12]. These modalities act as guidance and significantly help in the
recovery of missing depth values in the sparse maps. The idea is to actively
fuse features between different modalities. Most of the current methods adopt
two-branch networks architectures for feature fusion. For instance, DeepLiDAR
[12], FusionNet [23] and PENet [13] utilize an encoder-decoder architecture to
perform both early and late fusion between color images and LiDAR sparse depth
maps. The combination of early and late fusion between the modalities enhances
the depth completion performance. Recently, RigNet [14] proposed a repetitive
architecture, which repeats Hourglass network architecture for color images at
multiple levels to extract features and fuse them with depth generation branch
to create structure-detailed dense depth maps. These methods rely heavily only
on color images to extract color dominant information such as object
boundaries to complete sparse maps. However, color images truly do not provide
this information. It is because color images suffer from shadows and
reflections, which causes irregularities in pixel values [22].

To solve the problems in the methods mentioned earlier, we propose a novel
three-branch network inspired by PENet [13], FusionNet [23] and DeepLiDAR
[12]. It consists of color-guided (CG), semantic-guided (SG), and dense depth-
guided (DG) branches. In contrast to earlier methods [12, 13, 23], we add SG
branch to decrease the variance of depth values around object boundaries. The
CG branch produces a noisy depth estimate through the color image and LiDAR
sparse depth map. The output of the CG branch, LiDAR sparse depth and semantic
map of the color image is passed to the SG branch, which actively refines the
color depth and produces a depth based on semantic information. For ease of
understanding, we name it semantic depth in this paper. The semantic depth is
much more reliable around object boundaries than color depth. It is because
the pixel values of semantic maps are uniform and have fewer irregularities.
Furthermore, we also use the DG branch to refine further the depth produced by
the SG branch. It takes LiDAR sparse depth map, CG and SG depth as input and
produces a dense depth maps. Similar to earlier methods [13, 17], we also
combine depth information between CG, SG and DG depths with learned confidence
maps. The confidence map weights enables us to compare the reliability of the
predictions at each branch. In the end, we perform multi-modal depth fusion
between CG, SG and DG predicted dense depths to generate the final dense depth
map. The whole backbone model is trained in an end-to-end manner. Furthermore,
we apply CSPN++ with Atrous convolutions [24, 13, 19] on the final dense depth
map predicted by the three-branch backbone for further refinement.

Since we have branches dealing with multi-modal data, we perform both early
and late fusion between them for effective guidance of dense depth map
generation. Early fusion is performed by the depth-wise concatenation of
color, semantic or LiDAR sparse depth maps. The concatenation of the input is
fed to the respective branch. Late fusion is performed at the feature level
and involves the fusion of multi-modal data. The naive strategy of performing
late fusion is to apply simply concatenation or addition of features maps of
different modalities and it is used in many earlier works such as e.g. [8, 13,
25]. However, this is not an optimal way because each branch contains
different information. Therefore, we propose to apply semantic-aware multi-
modal attention-based fusion block (SAMMAFB) inspired by CBAM [26] and AFB
[27] to depth completion problem. SAMMAFB applies both channel and spatial
wise attention maps of the input feature maps and produces refined feature
maps. As demonstrated in Figure 1, we actively fuse features at multiple
stages through SAMMAFB in all of our branches.

The main contributions of our work is summarized as follows:

  * •

We propose a novel three-branch backbone for sparse depth completion, which
counters the sensitivity of image-guided methods to optical changes (e.g.,
shadows and reflections).

  * •

We present a novel SAMMAFB block to actively fuse the color, semantic, and
depth modalities at multiple stages in our three-branch backbone.

  * •

Extensive experimental results show that our model achieves state-of-the-art
results on the outdoor KITTI depth completion dataset.

##  II Related Work

###  II-A Sparse Depth Based Approaches

Earlier approaches [10, 28] based on convolutional neural networks (CNN)
utilized only sparse depth maps to generate dense depth maps. To counter the
sparsity of data in sparse depth maps, Depth-Net [29] performed nearest
neighbor interpolation in the sparse maps to fill out the holes (points with
no depth values). Later on, uncertainty aware CNN’s [30] proposed
probabilistic normalized convolutions to model the uncertainty in the sparse
depth maps. However, the obvious drawback of these approaches is that without
color or semantic image guidance, the predicted depth maps lack clear object
boundaries and also these methods are not suitable for real-time applications.

###  II-B Image-Guided Methods

Image-Guided methods [4, 8, 13, 14, 17, 18, 19, 31, 32, 33, 34] utilize multi-
modal information to facilitate dense depth completion. The multi-modal data
includes RGB images, semantic images, and surface normal’s, which act as
reference images for generating dense depth maps.

Methods such as SPN[34], CSPN[33], CSPN++[19], DSPN[31] and NLSPN [4] focus on
learning affinity matrix for high-level vision tasks including semantic
segmentation and depth completion. SPN [34] serially propagates the affinity
matrix, which is inefficient for real-time systems. CSPN [33] improved SPN
[34] by predicting affinity values of local neighbors and updating pixel
values simultaneously. Both of the methods suffer with the problem of fixed
local neighborhoods, which often have irrelevant information. To solve this
problem, CSPN++ [19], DSPN [31] and NLSPN [4] methods are introduced. CSPN++
[19] proposed adaptive learning of kernel sizes and the number of iterations
of propagation, which helped in reducing the computation time of CSPN [33].
DSPN [31] and NLSPN [4] learn deformable convolutional kernels to relax the
fixed local neighborhood of pixels, which enabled them to focus only on
relevant pixel neighbors for depth completion. All of the methods mentioned
above utilizes a single branch AutoEncoder (AE) [35] network architecture. The
input of the AE is the concatenation of RGB image and sparse depth map, which
outputs a dense depth map.

The two branch network architectures [12, 13, 14, 17, 18] consist of RGB and
sparse depth map branches. The RGB branch extracts color dominant information,
e.g., object boundaries, which is actively fused with a sparse depth map
branch at multiple stages. ACMNet [17] introduced a symmetric gated fusion
strategy for performing the fusion between two branches. FCFR-Net proposed a
channel shuffling and energy-based fusion between the two modalities. PENET
[13], utilized concatenation and addition operations to perform fusion at
multiple levels. Recently, RigNet [14] proposed an efficient guidance
algorithm to fuse and guide the sparse depth branch. In addition to RGB image
and sparse depth map, DeepLiDAR [12] introduced learning pixel-wise surface
normal’s of the scene, which is fused through addition with RGB branch to
generate dense depth maps. Similarly, Multi-Task GAN’s [16] generated semantic
maps of the RGB images and concatenated them with sparse and RGB images to
guide dense depth map completion. Compared to earlier approaches, the
performance of two branch methods are much better.

Unlike existing Image-guided methods, our approach consists of a semantic-
guided branch, which given the color and sparse dense depths, learns to
understand the semantic meaning of the scene. Consequently, it enables our
approach to perform consistently under different lighting conditions, which is
a challenging aspect of outdoor depth completion.

###  II-C Attention-based Multi-Modal Data Fusion

Attention-based multi-modal information fusion has been studied in various
computer vision applications, such as video description [36], human motion
estimation [37], emotion and sentiment classification [38] and many more. In
the context of dense depth completion, the attention mechanism enables the
model to focus on meaningful regions of multi-modal information, generating
depths with clear and sharp structural details. Moreover, given the sparsity
of the LiDAR depth data, the attention module is important for depth
completion task. Recently, ACM Net [17] proposed a Co-attention Guided Graph
Propagation (CGPM), which exploits the attention mechanism in the spatial
domain to propagate graphs at multiple scales. However, spatial attention only
capture local contexts within a fixed neighborhood. Therefore, to capture both
local and global attention, we propose to apply SAMMAFB between RGB, semantic,
and LiDAR sparse depth modalities. It produces both channel-wise and spatial-
wise attention weights, which are applied to multi-modal information to
generate refined fused feature maps.

Figure 2: The overview of proposed SemAttNet. It consists of a novel three-
branch backbone and a CSPN++ module with Atrous convolutions. Unlike earlier
image-guided methods, we design a separate branch for learning the semantic
information of the scene. Furthermore, we propose to apply attention based
fusion block (ABF) to perform semantic-aware fusion between RGB, depth, and
semantic modalities. Each branch outputs a depth map and a confidence map,
which are adaptively fused to produce a fused depth map. In the end, the fused
depth map are sent to CSPN++ module with Atrous convolutions for refinement.
Note, due to shortage of space, we use AFB to represent SAMMAFB block.

##  III Methodology

We formulate an image-guided depth completion problem as a two-stage task.
Figure 2 shows the pipeline of our approach. In the first stage, we design a
three-branch backbone network to produce a dense depth map, whereas, in the
second stage, we utilize CSPN++ [13, 19] with Atrous convolutions for further
refinement. The three-branch backbone consists of CG, SG and DG branches. The
purpose of the CG branch is to focus on color-dominant information, whereas
the SG branch aims to learn the semantics information of the scene. The
predicted dense depths of CG and SG branches are noisy depth estimates but
they contain important color and semantic cues of the scene. The DG branch
takes predicted dense depth maps of SG and CG branch along with aligned sparse
map as input to produce a dense depth map. DG branch focuses on learning depth
dominant features. All of the branches are complimentary to each other.
Therefore, we adaptively fuse them with learned confidence weights.

###  III-A Semantic-aware Multi-Modal Attention-based Fusion Block

In order to perform fusion between RGB, semantic, and depth features, we
propose to apply Semantic-aware Multi-Modal Attention-based Fusion Block
(SAMMAFB) on RGB, semantic, and depth features in our three-branch backbone.
SAMMAFB helps us refine the concatenated feature maps of modalities by
capturing salient feature maps while suppressing the unnecessary ones [27]. We
apply SAMMAFB in two different settings, first we apply it to fuse the
intermediate feature maps of RGB and semantic guided branches. Then, for
depth-guided branch, we perform the fusion between the intermediate feature
maps of all branches.

Figure 3 depicts the visual representation of SAMMAFB. It consists of a
channel and spatial attention sub-modules, which enables it to capture
channel-wise and spatial-wise attention. Specifically, channel-wise attention
aims to learn the important channels by assigning a weight to each channel
relative to their contribution towards increasing the overall performance.
Similarly, the spatial-wise attention module focuses on learning the location
of the important channels produced by the channel-wise attention sub-module.
Both of the modules complement each other and vital to producing refined fused
feature maps.

Figure 3: Architecture of SAMMAFB. The input to SAMMAFB is the concatenation
of RGB, semantic and sparse depth modalities. It applies channel-wise and
spatial-wise attention to the concatenated features to produce the refined
fused feature maps.

Let $F_{RGB}\in\mathbb{R}^{C\times H\times W}$, $F_{Sem}\in\mathbb{R}^{C\times
H\times W}$ and $F_{D}\in\mathbb{R}^{C\times H\times W}$ represent the
intermediate feature maps of RGB, semantic and depth guided branches at same
level, respectively and
$\mathbf{F}=\left[F_{RGB};F_{Sem};F_{D}\right]\in\mathbb{R}^{3C\times H\times
W}$ denotes their depth-wise concatenation. Channel-wise attention weights are
computed with Equation 1.

$\displaystyle\mathbf{{A}_{c}(F)}$
$\displaystyle=\sigma(MLP(AvgPool(\mathbf{F}))+MLP(MaxPool(\mathbf{F})))$ (1)
$\displaystyle=\sigma\left(\mathbf{W_{1}}\left(\mathbf{W_{0}}\left(\mathbf{F_{avg}^{c}}\right)\right)+\mathbf{W_{1}}\left(\mathbf{W_{0}}\left(\mathbf{F_{max}^{c}}\right)\right)\right)$

Where $\mathbf{{A}_{c}(F)}\in\mathbb{R}^{C\times 1\times 1}$ represents
channel-wise attention weights of $\mathbf{F}$, $\sigma$ denotes sigmoid
function, $\mathbf{W_{0}}\in\mathbb{R}^{C/r\times C}$,
$\mathbf{W_{1}}\in\mathbb{R}^{C\times C/r}$ represents the weights matrices
for multi-layer perceptron (MLP) layers. The parameter $r$ controls the number
of learnable parameters in MLP layer. $\mathbf{F_{max}^{c}}$ and
$\mathbf{F_{avg}^{c}}$ denotes the max-pooled and average-pooled features
respectively.

The channel-wise attentions weights are applied to $\mathbf{F}$ to obtain
$\mathbf{F^{\prime}}\in\mathbb{R}^{3C\times H\times W}$ such that
$\mathbf{F^{\prime}}=\mathbf{A_{c}(F)}\otimes\mathbf{F}$. The product
$\mathbf{F^{\prime}}$ is then sent to spatial-wise attention module, which is
given in Equation 2.

$\mathbf{A_{s}}\left(\mathbf{F^{\prime}}\right)=\sigma\left(Conv\left(\left[\mathbf{F_{avg}^{c}};\mathbf{F_{max}^{c}}\right]\right)\right)$
(2)

Where $\mathbf{{A}_{s}(F)}\in\mathbb{R}^{1\times H\times W}$ represents
spatial-wise attention weights of $\mathbf{F}$, $\sigma$ denotes sigmoid
function. $\mathbf{F_{max}^{c}}$ and $\mathbf{F_{avg}^{c}}$ denotes the max-
pooled and average-pooled features respectively.

The spatial-wise attentions weights are applied to $\mathbf{F^{\prime}}$ to
obtain final output of SAMMAFB i.e. refined fused feature maps
$\mathbf{F^{\prime\prime}}\in\mathbb{R}^{3C\times H\times W}$ such that
$\mathbf{F^{\prime\prime}}=\mathbf{A_{s}}(\mathbf{F^{\prime}})\otimes\mathbf{F^{\prime}}$.

###  III-B The Three-branch Backbone

The earlier two-branch methods [13, 14, 17, 18] relied heavily on the color
image to extract both semantic and color-dominant information. However, color
images alone might not be able to provide this information. Therefore, to
overcome this limitation, we propose a three-branch backbone consisting of CG,
SG and DG branches. The addition of the SG branch enables better learning of
semantic cues of the scene, which help predict depth maps with accurate and
sharp structure in the DG branch. The dataset comprises of color and semantic
images that are represented by $C=\mathbb{R}^{N\times 3\times H\times W}$ and
$S=\mathbb{R}^{N\times 3\times H\times W}$, whereas $D=\mathbb{R}^{N\times
1\times H\times W}$ and $D^{gt}=\mathbb{R}^{N\times 1\times H\times W}$
denotes sparse and ground truth depth maps, and $N$ denotes the number of
samples. Note, that we don’t have any supervision of confidence maps due the
unavailability of ground truth. Therefore, confidence maps are trained
indirectly based on the $\ell_{2}$ loss of each branch.

####  III-B1 Color-guided Branch

The CG branch aims to learn important color cues for dense depth completion.
It takes a color image concatenated with an aligned sparse depth map as input
and outputs a dense depth map. The concatenation of an aligned sparse depth
map with a color image helps in the prediction of a dense depth map [13]. The
CG branch follows an encoder-decoder network architecture with skip
connections. The encoder consists of ten ResNet [39] blocks, whereas the
decoder consists of one convolution and five transpose convolution layers for
upsampling. The output of this branch consist of dense depth map and a
confidence map. The resultant dense depth map of this branch is a noisy depth
estimate, but it provides a baseline for learning structural information of
the scene in other branches.

Let $\mathbf{X_{cg}}=\left[C_{B};D_{B}\right]\in\mathbb{R}^{B\times 4\times
H\times W}$ represent the concatenation of batch of size $B$ of color images
$C_{B}\in C$ and sparse depth maps $D_{B}\in C$. The CG branch takes
$\mathbf{X_{cg}}$ as input, and in the first step, it is encoded to a hidden
representation $\mathbf{\phi_{cg}}\in\mathbb{R}^{B\times 1024\times H\times
W}$ based on Equation 3. Then, $\mathbf{\phi_{cg}}$ is decoded into a
confidence map $C_{cg}\in\mathbb{R}^{B\times 1\times H\times W}$ and a color
depth map $D_{cg}\in\mathbb{R}^{B\times 1\times H\times W}$ based on Equation
4.

$\displaystyle\mathbf{\phi_{cg}}=f(\mathbf{W_{cg}}\mathbf{X_{cg}}+b_{cg})$ (3)
$\displaystyle D_{cg},C_{cg}=g(\mathbf{V_{cg}}\mathbf{\phi_{cg}}+c_{cg})$ (4)

Where $\mathbf{W_{cg}}$ and $\mathbf{V_{cg}}$ represents encoder and decoder
weight matrices. The variables $b_{cg}$ and $c_{cg}$ denote encoding and
decoding bias values. $f(.)$ and $g(.)$ represent activation functions.

$\displaystyle\mathbf{L_{cg}}=argmin_{D_{cg}}\ ||D^{gt}-D_{cg}||^{2}$ (5)

Equation 5 defines the $\ell_{2}$ loss for CG branch. Since the ground truth
contains invalid depth values, we only consider pixels with valid depth values
for computing loss.

####  III-B2 Semantic-guided Branch

Semantic cues help to understand the scene and are essential for depth
completion task [40]. However, CG branch alone is not enough to learn semantic
information. Therefore, we propose an SG branch to encourage learning
effective semantic cues and to complement RGB images for depth completion. The
SG branch takes the concatenation of color depth, semantic image and sparse
depth map as an input and outputs a dense depth map, which consist of both
color and semantic cues. The KITTI depth completion dataset does not provide
aligned semantic maps of RGB images. Therefore, we transform the RGB images to
semantic maps through a pre-trained WideResNet38 [41] model on KITTI semantic
segmentation benchmark [42]. Furthermore, Inspired by earlier works [8, 13],
we fuse the decoder features from the CG branch into the corresponding encoder
features of the SG branch. The features from CG and SG are sent to SAMMAFB,
which outputs refined fused depth feature maps.

Similar to CG branch, we define
$\mathbf{X_{sg}}=\left[D_{cg}^{B};S_{B};D_{B}\right]\in\mathbb{R}^{B\times
4\times H\times W}$ to represent the concatenation of batch size $B$ of color
depths $D_{cg}^{B}\in D_{cg}$, color images $C_{B}\in C$ and sparse depth maps
$D_{B}\in D$. The color depth consists of important color cues e.g. object
boundaries and its concatenation will complement in learning semantic cues.
The matrix $\mathbf{X_{sg}}$ is passed as an input to SG branch. In the first
step, the input is encoded to a hidden representation
$\mathbf{\phi_{sg}}\in\mathbb{R}^{B\times 1024\times H\times W}$ based on
Equation 6. In the second step, the encoded representation
$\mathbf{\phi_{sg}}$ is decoded into a confidence map
$C_{sg}\in\mathbb{R}^{B\times 1\times H\times W}$ and a semantic depth map
$D_{sg}\in\mathbb{R}^{B\times 1\times H\times W}$ based on Equation 7.

$\displaystyle\mathbf{\phi_{sg}}=f\mathbf{(W_{sg}^{T}.X_{sg}}+b_{sg})$ (6)
$\displaystyle D_{sg},C_{sg}=g\mathbf{(V_{sg}^{T}.\phi_{sg}}+c_{sg})$ (7)

Where $\mathbf{W_{sg}}$ and $\mathbf{V_{sg}}$ represents encoder and decoder
weight matrices. The variables $b_{sg}$ and $c_{sg}$ represents encoding and
decoding bias values. $f(.)$ and $g(.)$ denotes activation functions.

$\displaystyle\mathbf{L_{sg}}=argmin_{D_{sg}}\ ||\ D^{gt}-D_{sg}||^{2}$ (8)

Equation 8 defines the $\ell_{2}$ loss for SG branch. we only consider pixels
with valid depth values for loss calculation because the ground truth contains
invalid depth values.

####  III-B3 Depth-guided Branch

The DG branch focuses on learning depth-dominant features, which help generate
accurate dense depth maps. It takes the concatenation of CG, SG, and sparse
depth as input and outputs a dense depth map. Similar to the feature fusion
strategy in the CG and SG branch, we fuse decoder features from CG and SG
branches into the corresponding encoder features of the DG branch. However, in
DG branch, SAMMAFB take feature maps from three modalities as input and
outputs refined fused feature maps. The refined fused feature maps consists of
useful information from CG and SG branches, which guides the DG branch in
learning effective depth feature representations. The network architecture is
similar to CG and SG branches, but we add two more layers in encoder and
decoder architectures to accommodate CG and SG branch decoder features.

We define
$\mathbf{X_{dg}}=\left[D_{cg}^{B};D_{sg}^{B};D_{B}\right]\in\mathbb{R}^{B\times
3\times H\times W}$ to represent the concatenation of batch of size $B$ of
color depths $D_{cg}^{B}$, semantic depths $D_{sg}^{B}$ and sparse depth maps
$D_{B}$. The DD branch takes $\mathbf{X_{dg}}$ as an input and encodes it to a
hidden representation $\mathbf{\phi_{dg}}\in\mathbb{R}^{B\times 1984\times
H\times W}$ based on Equation 9. Afterwards, the encoded representation
$\mathbf{\phi_{dg}}$ is decoded into a confidence map
$C_{dg}\in\mathbb{R}^{B\times 1\times H\times W}$ and a dense depth map
$D_{dg}\in\mathbb{R}^{B\times 1\times H\times W}$ based on Equation 10.

$\displaystyle\mathbf{\phi_{dg}}=f\mathbf{(W_{dg}^{T}.X_{dg}}+b_{dg})$ (9)
$\displaystyle D_{dg},C_{dg}=g\mathbf{(V_{dg}^{T}.\phi_{dg}}+c_{dg})$ (10)

Where $\mathbf{W_{dg}}$ and $\mathbf{V_{dg}}$ represents encoder and decoder
weight matrices. The variables $b_{sg}$ and $c_{sg}$ represents encoding and
decoding bias values. $f(.)$ and $g(.)$ denotes activation functions.

$\displaystyle\mathbf{L_{dg}}=argmin_{D_{dg}}\ ||\ D^{gt}-D_{dg}||^{2}$ (11)

Equation 11 defines the $\ell_{2}$ loss for DD branch. Similar to SG and CG
branches, we only consider pixels with valid depth values in ground truth for
loss calculation.

####  III-B4 Multi-modal Depth Fusion

Similar to earlier approaches [13, 23], we fuse the predicted depth maps of
each branch by using learned confidence weights. Equation 12 shows the
mathematical representation of our depth map fusion strategy.

$\displaystyle D_{f}=\frac{e^{C_{cg}}\cdot D_{cg}+e^{C_{sg}}\cdot
D_{sg}+e^{C_{dg}}\cdot D_{dg}}{e^{C_{cg}}+e^{C_{sg}}+e^{C_{dg}}}$ (12)

Where $D_{f}$ denotes the fused depth map, $D_{cg}$, $D_{sg}$ and $D_{dg}$
represents the depth maps predicted by CG, SG and DG branches and
$e^{C_{cg}}$, $e^{C_{sg}}$, $e^{C_{dg}}$ denotes the learned confidence maps
of CG, SG and DG predicted depth maps.

Following the CG, SG and DG branches, we also calculate the loss on fused
depth map by taking $\ell_{2}$ norm between the fused depth map and ground
truth as depicted in Equation 13. We only consider pixels with valid depth
values for loss calculation of fused depth map.

$\displaystyle\mathbf{L_{fused}}=argmin_{D_{f}}\ ||\ D^{gt}-D_{f}||^{2}$ (13)

####  III-B5 Loss Function

The training loss of our three-branch backbone consists of the sum of CD, SD,
DD and fused depth losses. Equation 14 defines the training loss of our three-
branch backbone.

$\displaystyle\mathbf{L_{total}}=\lambda_{cg}L_{cg}+\lambda_{sg}L_{sg}+\lambda_{dg}L_{dg}+L_{fused}$
(14)

Where $\lambda_{cg},\ \lambda_{sg},\ \lambda_{dg}$ denotes the weights for
color-guided, semantic-guided and depth-guided branches.

###  III-C CSPN++ with Atrous Convolutions

The reason for applying CSPN++ for refining the dense depth maps produced by
our three-branch backbone is two folds. First, it recovers depth values at
valid pixels. Second, enabling a smooth transition between the neighbours of
our completed dense depth map. Furthermore, inspired by PENet [13], we use
Atrous convolutions [43] to enlarge the propagation neighborhood of pixels
[44].

Given a fused depth map $D_{f}\in\mathbb{R}^{h\times w\times 1}$, it is
embedded into a hidden representation $H\in\mathbb{R}^{h\times w\times c}$,
the one step propagation of CSPN++ with Atrous convolution is given in
Equation 15.

$\begin{gathered}H_{t+1}(x_{i})=\kappa_{x_{i}}\left(x_{i}\right)\odot
H_{t}\left(x_{i}\right)+\sum_{x_{j}\in\mathcal{N}_{k}(x_{i})}\kappa_{x_{i}}(x_{j})\odot
H_{t}(x_{j})\\\
\kappa_{x_{i}}\left(x_{j}\right)=\frac{\hat{\kappa}_{x_{i}}\left(x_{j}\right)}{\sum_{x_{j}\in\mathcal{N}}\left|\hat{\kappa}_{x_{i}}\left(x_{j}\right)\right|},\kappa_{x_{i}}\left(x_{i}\right)=1-\sum_{x_{j}\in\mathcal{N}}\kappa_{x_{i}}\left(x_{j}\right)\end{gathered}$
(15)

Where $\mathcal{N}_{k}\left(x\right)=\left\\{x_{n}+l\mid
l\in\mathbb{R}\right\\}$ denotes the neighborhood pixels of $x_{i}$,
$\hat{\kappa_{x_{i}}}\in\mathbb{R}^{k\times k\times c}$ represents the
affinity values produced by the network. During propagation, pixel $x_{i}$
receives information from enlarged neighbourhood $\mathcal{N}_{k}(x_{i})$.

##  IV Experiments

|  |  |
---|---|---|---
(a) RGB Image | (b) PENet | (c) RigNet | (d) SemAttNet(ours)

Figure 4: Qualitative results on KITTI depth completion test set. The results
are obtained by online KITTI depth completion leaderboard. From left-to-right
(a) RGB Image, (b) PENet [13], (c) RigNet [14], (d) SemAttNet(ours). The
correct estimates in errors maps are given in blue color, whereas wrong
estimate are depicted by warmer colors.

Method |  RMSE mm |  MAE mm |  iRMSE 1/km |  iMAE 1/km
---|---|---|---|---
TWISE [45] | 840.20 | 195.58 | 2.08 | 0.82
DSPN [31] | 766.74 | 220.36 | 2.47 | 1.03
DLiDAR [12] | 758.38 | 226.50 | 2.56 | 1.15
FuseNet [20] | 752.88 | 221.19 | 2.34 | 1.14
ACMNet [17] | 744.91 | 206.09 | 2.08 | 0.90
CSPN++ [19] | 743.69 | 209.28 | 2.07 | 0.90
NLSPN [4] | 741.68 | 199.59 | 1.99 | 0.84
GuideNet [8] | 736.24 | 218.83 | 2.25 | 0.99
FCFRNet [18] | 735.81 | 217.15 | 2.20 | 0.98
PENet [13] | 730.08 | 210.55 | 2.17 | 0.94
RigNet [14] | 713.44 | 204.55 | 2.16 | 0.92
SemAttNet | 709.41 | 205.49 | 2.03 | 0.90
Table I: Quantitative comparison with state-of-the-art results on KITTI depth
completion benchmark, ranked by RMSE. The results of other methods are
obtained from KITTI leaderboard. Our method i.e. SemAttNet achieves state-of-
the-art performance at the time of submission.

###  IV-A Dataset

We evaluate the performance of our method against different state-of-the-art
(SoTA) methods on KITTI depth completion benchmark [10, 11]. The KITTI dataset
is a large outdoor dataset for autonomous vehicles. It provides 85K sparse
depth maps with corresponding RGB images for training, 7K for validation, and
1K for testing. The sparse depth map in the KITTI dataset is generated by
using LiDAR HDL-64, which provides valid depth values on only $5.9\%$ of all
pixels [10, 11]. However, the ground-truth contains valid depth values on
$16\%$ of all the pixels. The ground-truth is generated by accumulating LiDAR
and stereo estimation of the scenes [10, 11]. For all of our experiments, we
use the official validation set, which consists of 1K frames [10, 11]. During
training, we bottom crop the semantic images, RGB images, and sparse depth map
from $375\times 1275$ to $352\times 1252$.

###  IV-B Evaluation Metrics

We use the official evaluation metrics of the KITTI depth completion benchmark
[10, 11] to evaluate the performance of our approach. The official evaluation
measures consist of root mean squared error (RMSE), mean absolute error (MAE),
root mean squared error of inverse depth (iRMSE), and mean absolute error of
the inverse depth (iMAE). Among the evaluation metrics, RMSE is chosen to rank
the submissions on the KITTI leaderboard.

Methods | Branches | Fusion |  CSPN++ Refinement |  RMSE mm
---|---|---|---|---
CG + DG | CG + SG + DG | add | concat | SAMMAFB
(a) | ✓ |  | ✓ |  |  |  | 782.89
(b) | ✓ |  |  | ✓ |  |  | 781.66
(c) | ✓ |  |  | ✓ |  | ✓ | 762.84
(d) |  | ✓ |  | ✓ |  |  | 755.16
(e) |  | ✓ |  | ✓ |  | ✓ | 750.06
(f) |  | ✓ |  |  | ✓ |  | 753.02
(g) |  | ✓ |  |  | ✓ | ✓ | 738.13
Table II: Ablation study on KITTI depth completion selected validation
dataset. CG, SG and DG stands for color-guided, semantic-guided and depth-
guided branches and concat, add refers to simple concatenation and addition of
multi-modal features.

###  IV-C Implementation Details

We used PyTorch [46] framework to implement the three-branch backbone and
CSPN++ with Atrous convolutions. The three-branch backbone is trained on two
NVIDIA A100 GPUs, whereas the full model is trained on three NVIDIA RTXA6000
GPUs. Both models use same setting of ADAM optimizer [47] with $\beta_{1}$ of
0.9, $\beta_{1}$ of 0.99 and weight decay of $10^{-6}$. For efficient
learning, we performed data prepossessing methods including random crop, flip
and color jitter [48] on the dataset. The random crops are performed at
$320\times 1216$ resolution.

The three-branch backbone is trained with a batch size of 8 and an initial
learning rate of 0.00128. The initial learning rate is decayed using
ReduceLROnPlateu scheduler. Furthermore, we assign an initial weight of 0.2 to
$\lambda_{cg}$, $\lambda_{sg}$ and $\lambda_{dg}$ coefficients, but we decay
the weight to 0 after few epochs. We train the backbone for 60 epochs. To
train the CSPN++ with Atrous convolutions, similar to PENet [13], we first
perform the initialization step and then fully train the models for 95 epochs.

###  IV-D Evaluation on KITTI dataset

Table I shows the quantitative comparison of our method with other state-of-
the-art methods. Our proposed method, SemAttNet, outperforms the previous
state-of-the-art methods under the primary evaluation metric RMSE at the time
of submission, and presents comparable performance on other evaluation
metrics. Specifically, in contrast to previous state-of-the-arts, i.e., RigNet
and PENet, we observe a difference of $4.03mm$ and $21.03mm$ to the
performance of SemAttNet. Moreover from the qualitative comparison of our
method given in Figure 4, it’s evident that our method produces much more
precise results than other methods in the areas affected by optical changes in
RGB images. Please refer to supplementary section for additional qualitative
results.

###  IV-E Ablation Studies

In this section, we investigate the impact of each component proposed by our
approach on final performance. All of the ablation studies are performed on
selected KITTI validation dataset. Table II depicts the results of our
experiments with different settings. Note, that we consider RMSE as our
primary metric to rank the performance of methods.

####  IV-E1 Effect of exploiting Semantic information.

From methods (b) and (d) listed in Table II, we can observe that by using the
proposed semantic-guided branch even with naive concatenation method for
fusion, there is a significant improvement of $26.5mm$ in the RMSE metric.
This shows the promise and importance of our proposed SG branch.

####  IV-E2 Effect of SAMMAFB

The methods (d) and (e) in Table II list the performance of the naive fusion
method, and SAMMAFB. It is quite evident that SAMMAFB, i.e., method (d),
outperforms the naive fusion method by achieving $2.14mm$ less error than
method (f). Moreover, if we consider methods (f) and (g) which employs CSPN++
refinement along with naive and SAMMAFB fusion methods, we see a significant
improvement of $11.93mm$ of RMSE in the method utilizing the SAMMAFB for
fusion. These improvement backs our claim that SAMMAFB is a better fusion
strategy than naive fusion methods.

####  IV-E3 Effect of CSPN++ refinement

Inspired by original CSPN++ [19], we also use 12 iterations for neighborhood
propagation. However, similar to PENet [13], we apply a dilation strategy to
the original CSPN++ in which for the first 6 iterations, we use a dilation
rate of 2, and then we decrease it to 1 for remaining iterations to propagate
the neighborhood of each pixel. From Table II, the method (g) that employs
CSPN++ refinement with the dilation strategy, outperforms method (f) with a
significant margin of $14.89mm$ in RMSE metric. This proves that CSPN++ is an
effective method for dense depth refinement.

##  V Conclusion

In this paper, we presented a novel three-branch backbone consisting of color-
guided, semantic-guided, and depth-guided branches for dense depth completion.
Existing image-guided methods focus only on RGB images to extract both color
and semantic cues of the scene. However, RGB images suffer from artifacts such
as varying contrast and shadows. To tackle this problem, in our three-branch
backbone, we introduced a novel semantic-guided branch, which, irrespective of
any optical changes in RGB images, learns to understand the semantics of the
scene and reduces the reliance on RGB images. Moreover, instead of fusing
features naively through addition or concatenation, we also utilized semantic-
aware multi-model fusion block (SAMMAFB) for feature fusion between all
branches. In addition, we also implement CSPN++ with Atrous convolutions to
further refine the dense depth maps. Extensive experiments show that our
results are superior both quantitatively and qualitatively compared to
previous state-of-the-art methods.

## References

  * [1] A. Holynski and J. Kopf, “Fast depth densification for occlusion-aware augmented reality,” ACM Transactions on Graphics (ToG), vol. 37, no. 6, pp. 1–11, 2018.
  * [2] M. Kalia, N. Navab, and T. Salcudean, “A real-time interactive augmented reality depth estimation technique for surgical robotics,” in 2019 International Conference on Robotics and Automation (ICRA), pp. 8291–8297, 2019\.
  * [3] T.-N. Nguyen, H.-H. Huynh, and J. Meunier, “3d reconstruction with time-of-flight depth camera and multiple mirrors,” IEEE Access, vol. 6, pp. 38106–38114, 2018.
  * [4] J. Park, K. Joo, Z. Hu, C.-K. Liu, and I. So Kweon, “Non-local spatial propagation network for depth completion,” in Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XIII 16, pp. 120–136, Springer, 2020.
  * [5] L. Bai, Y. Zhao, M. Elhousni, and X. Huang, “Depthnet: Real-time lidar point cloud depth completion for autonomous vehicles,” IEEE Access, vol. 8, pp. 227825–227833, 2020.
  * [6] Z. Cui, L. Heng, Y. C. Yeo, A. Geiger, M. Pollefeys, and T. Sattler, “Real-time dense mapping for self-driving vehicles using fisheye cameras,” in 2019 International Conference on Robotics and Automation (ICRA), pp. 6087–6093, 2019.
  * [7] H.-C. Huang, C.-T. Hsieh, and C.-H. Yeh, “An indoor obstacle detection system using depth information and region growth,” Sensors, vol. 15, no. 10, pp. 27116–27141, 2015.
  * [8] J. Tang, F.-P. Tian, W. Feng, J. Li, and P. Tan, “Learning guided convolutional network for depth completion,” IEEE Transactions on Image Processing, vol. 30, pp. 1116–1129, 2020.
  * [9] G. J. Iddan and G. Yahav, “Three-dimensional imaging in the studio and elsewhere,” in Three-Dimensional Image Capture and Applications IV (B. D. Corner, J. H. Nurre, and R. P. Pargas, eds.), vol. 4298, pp. 48 – 55, International Society for Optics and Photonics, SPIE, 2001.
  * [10] J. Uhrig, N. Schneider, L. Schneider, U. Franke, T. Brox, and A. Geiger, “Sparsity invariant cnns,” 2017.
  * [11] A. Geiger, P. Lenz, and R. Urtasun, “Are we ready for autonomous driving? the kitti vision benchmark suite,” in 2012 IEEE Conference on Computer Vision and Pattern Recognition, pp. 3354–3361, 2012.
  * [12] J. Qiu, Z. Cui, Y. Zhang, X. Zhang, S. Liu, B. Zeng, and M. Pollefeys, “Deeplidar: Deep surface normal guided depth prediction for outdoor scene from sparse lidar data and single color image,” in The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2019.
  * [13] M. Hu, S. Wang, B. Li, S. Ning, L. Fan, and X. Gong, “Towards precise and efficient image guided depth completion,” 2021.
  * [14] Z. Yan, K. Wang, X. Li, Z. Zhang, B. Xu, J. Li, and J. Yang, “Rignet: Repetitive image guided network for depth completion,” 2021.
  * [15] B.-U. Lee, K. Lee, and I. S. Kweon, “Depth completion using plane-residual representation,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 13916–13925, 2021.
  * [16] C. Zhang, Y. Tang, C. Zhao, Q. Sun, Z. Ye, and J. Kurths, “Multitask gans for semantic segmentation and depth completion with cycle consistency,” IEEE Transactions on Neural Networks and Learning Systems, pp. 1–12, 2021.
  * [17] S. Zhao, M. Gong, H. Fu, and D. Tao, “Adaptive context-aware multi-modal network for depth completion,” IEEE Transactions on Image Processing, 2021\.
  * [18] L. Liu, X. Song, X. Lyu, J. Diao, M. Wang, Y. Liu, and L. Zhang, “Fcfr-net: Feature fusion based coarse-to-fine residual learning for depth completion,” in Proceedings of the AAAI Conference on Artificial Intelligence, vol. 35, pp. 2136–2144, 2021.
  * [19] X. Cheng, P. Wang, C. Guan, and R. Yang, “Cspn++: Learning context and resource aware convolutional spatial propagation networks for depth completion,” in Proceedings of the AAAI Conference on Artificial Intelligence, vol. 34, pp. 10615–10622, 2020.
  * [20] Y. Chen, B. Yang, M. Liang, and R. Urtasun, “Learning joint 2d-3d representations for depth completion,” in Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 10023–10032, 2019.
  * [21] N. Schneider, L. Schneider, P. Pinggera, U. Franke, M. Pollefeys, and C. Stiller, “Semantically guided depth upsampling,” in German conference on pattern recognition, pp. 37–48, Springer, 2016.
  * [22] P.-Y. Chen, A. H. Liu, Y.-C. Liu, and Y.-C. F. Wang, “Towards scene understanding: Unsupervised monocular depth estimation with semantic-aware representation,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2624–2632, 2019.
  * [23] W. V. Gansbeke, D. Neven, B. D. Brabandere, and L. V. Gool, “Sparse and noisy lidar completion with rgb guidance and uncertainty,” 2019.
  * [24] L.-C. Chen, G. Papandreou, I. Kokkinos, K. Murphy, and A. L. Yuille, “Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs,” 2017.
  * [25] F. Ma, G. V. Cavalheiro, and S. Karaman, “Self-supervised sparse-to-dense: Self-supervised depth completion from lidar and monocular camera,” in 2019 International Conference on Robotics and Automation (ICRA), pp. 3288–3295, IEEE, 2019.
  * [26] S. Woo, J. Park, J.-Y. Lee, and I. S. Kweon, “Cbam: Convolutional block attention module,” in Proceedings of the European conference on computer vision (ECCV), pp. 3–19, 2018.
  * [27] F. Fooladgar and S. Kasaei, “Multi-modal attention-based fusion model for semantic segmentation of rgb-depth images,” arXiv preprint arXiv:1912.11691, 2019.
  * [28] N. Chodosh, C. Wang, and S. Lucey, “Deep convolutional compressed sensing for lidar depth completion,” 2018.
  * [29] L. Bai, Y. Zhao, M. Elhousni, and X. Huang, “Depthnet: Real-time lidar point cloud depth completion for autonomous vehicles,” IEEE Access, vol. 8, pp. 227825–227833, 2020.
  * [30] A. Eldesokey, M. Felsberg, K. Holmquist, and M. Persson, “Uncertainty-aware cnns for depth completion: Uncertainty from beginning to end,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12014–12023, 2020.
  * [31] Z. Xu, H. Yin, and J. Yao, “Deformable spatial propagation networks for depth completion,” in 2020 IEEE International Conference on Image Processing (ICIP), pp. 913–917, IEEE, 2020.
  * [32] J. Gu, Z. Xiang, Y. Ye, and L. Wang, “Denselidar: A real-time pseudo dense depth guided depth completion network,” IEEE Robotics and Automation Letters, vol. 6, no. 2, pp. 1808–1815, 2021.
  * [33] X. Cheng, P. Wang, and R. Yang, “Learning depth with convolutional spatial propagation network,” IEEE transactions on pattern analysis and machine intelligence, 2019.
  * [34] S. Liu, S. De Mello, J. Gu, G. Zhong, M.-H. Yang, and J. Kautz, “Learning affinity via spatial propagation networks,” arXiv preprint arXiv:1710.01020, 2017.
  * [35] C.-Y. Liou, W.-C. Cheng, J.-W. Liou, and D.-R. Liou, “Autoencoder for words,” Neurocomputing, vol. 139, pp. 84–96, 2014.
  * [36] C. Hori, T. Hori, T.-Y. Lee, Z. Zhang, B. Harsham, J. R. Hershey, T. K. Marks, and K. Sumi, “Attention-based multimodal fusion for video description,” in Proceedings of the IEEE international conference on computer vision, pp. 4193–4202, 2017.
  * [37] Y. Li, T. Zhao, and X. Shen, “Attention-based multimodal fusion for estimating human emotion in real-world hri,” in Companion of the 2020 ACM/IEEE International Conference on Human-Robot Interaction, pp. 340–342, 2020.
  * [38] M. G. Huddar, S. S. Sannakki, and V. S. Rajpurohit, “Attention-based multimodal contextual fusion for sentiment and emotion classification using bidirectional lstm,” Multimedia Tools and Applications, vol. 80, no. 9, pp. 13059–13076, 2021.
  * [39] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770–778, 2016.
  * [40] K. Lu, N. Barnes, S. Anwar, and L. Zheng, “From depth what can you see? depth completion via auxiliary image reconstruction,” in 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 11303–11312, 2020.
  * [41] Z. Wu, C. Shen, and A. van den Hengel, “Wider or deeper: Revisiting the resnet model for visual recognition,” 2016.
  * [42] H. Alhaija, S. Mustikovela, L. Mescheder, A. Geiger, and C. Rother, “Augmented reality meets computer vision: Efficient data generation for urban driving scenes,” International Journal of Computer Vision (IJCV), 2018.
  * [43] F. Yu and V. Koltun, “Multi-scale context aggregation by dilated convolutions,” 2016.
  * [44] Z. Xu, H. Yin, and J. Yao, “Deformable spatial propagation networks for depth completion,” in 2020 IEEE International Conference on Image Processing (ICIP), pp. 913–917, IEEE, 2020.
  * [45] S. Imran, X. Liu, and D. Morris, “Depth completion with twin surface extrapolation at occlusion boundaries,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2583–2592, 2021.
  * [46] A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga, A. Desmaison, A. Kopf, E. Yang, Z. DeVito, M. Raison, A. Tejani, S. Chilamkurthy, B. Steiner, L. Fang, J. Bai, and S. Chintala, “Pytorch: An imperative style, high-performance deep learning library,” in Advances in Neural Information Processing Systems 32 (H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, eds.), pp. 8024–8035, Curran Associates, Inc., 2019.
  * [47] D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” 2017.
  * [48] F. Ma and S. Karaman, “Sparse-to-dense: Depth prediction from sparse depth samples and a single image,” in 2018 IEEE international conference on robotics and automation (ICRA), pp. 4796–4803, IEEE, 2018.