# Model Selection in High-Dimensional Block-Sparse Linear Regression

Prakash B. Gohain, Magnus Jansson This research was supported in part by the
European Research Council (ERC) under the European Union’s Horizon 2020
research and innovation programme, grant agreement No. 742648. The authors are
with the Division of Information Science and Engineering, KTH Royal Institute
of Technology, Stockholm SE-10044, Sweden e-mail<EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

Model selection is an indispensable part of data analysis dealing very
frequently with fitting and prediction purposes. In this paper, we tackle the
problem of model selection in a general linear regression where the parameter
matrix possesses a block-sparse structure, i.e., the non-zero entries occur in
clusters or blocks and the number of such non-zero blocks is very small
compared to the parameter dimension. Furthermore, a high-dimensional setting
is considered where the parameter dimension is quite large compared to the
number of available measurements. To perform model selection in this setting,
we present an information criterion that is a generalization of the Extended
Bayesian Information Criterion-Robust (EBIC-R) and it takes into account both
the block structure and the high-dimensionality scenario. The analytical steps
for deriving the EBIC-R for this setting are provided. Simulation results show
that the proposed method performs considerably better than the existing state-
of-the-art methods and achieves empirical consistency at large sample sizes
and/or at high-SNR.

###### Index Terms:

Block-sparsity, general linear regression, model selection, sparse recovery,
OMP, EBIC

##  I Introduction

Selecting the best model/subset in the high-dimensional (HD) linear regression
has been an active research topic for a long time now. In this context,
methods based on Information Criterion (IC) have played a pivotal role ever
since Akaike proposed the famous Akaike IC [1]. Since then, IC-based model
selection has come a long way with the frequent appearance of more developed
and sophisticated methods [2, 3, 4]. However, classical methods do not fare
well in HD scenarios and quite often lead to overfitted models with false
parameters. In the present era, popular IC-based methods for model selection
in the HD setting include extended Bayesian IC (EBIC) [5], extended Fisher IC
(EFIC) [6], and extended BIC-Robust (EBIC${}_{\text{R}}$) [7, 8].

Apart from the methods mentioned above, there are other non-IC based model
selection approaches in the HD regime such as the Residual-Ratio-Thresholding
(RRT) [9] and the Multi-Beta-Test (MBT) [10], both of which are based on the
hypothesis testing framework used along with a greedy variable selection
method such as orthogonal matching pursuit (OMP) [11]. Some recent methods
also include significance test of the LASSO [12] and knock-off-filters [13].
Another popular method is cross-validation (CV) [14, 15]. However, CV-based
procedures can be computationally intensive, and their performance in the HD
setting is not satisfactory [16, 17, 18].

In this paper, we consider model selection in a general linear regression
where the nonzero coefficients in the parameter matrix occur in clusters (or
groups). Such signals are referred to as block-sparse [19, 20, 21]. Block-
sparsity inherently arises in a variety of scenarios. For example in multi-
band signals [22, 23], in the recovery of signals from compressed microarray
measurements [24], in the multiple measurement vector (MMV) problem [25, 26,
27, 28]. Furthermore, as shown in [19] and [20], the block-sparsity model can
be used to handle the issue of sampling signals that lie in a union of
subspaces [29, 30].

A recent method for model selection in block-sparse HD linear regression is
the Generalized RRT (GRRT) [31]. GRRT is an extension of RRT [9] developed to
treat the block-sparse structure in linear regression. The authors also
present a new approach that allows GRRT to perform model selection in non-
monotonic predictor sequences generated by LASSO [32]. However, the IC-based
methods are not designed to take into account the block structure during model
selection. Hence, in their current form, they cannot be applied directly
without tailoring them to incorporate the block nature of the underlying
linear model into the criterion. In this paper, the main goal is to develop an
IC-based model selection method for the general linear regression model
assuming a block-sparse structure and a HD setting.

In the paper, matrices and vectors are denoted by boldface letters. The
notation $(\cdot)^{T}$ stands for transpose. $\mathbf{I}_{N}$ is an $N\times
N$ identity matrix.
$\mathbf{\Pi}(\mathbf{A})=\mathbf{A}(\mathbf{A}^{T}\mathbf{A})^{-1}\mathbf{A}^{T}$
represents the orthogonal projection matrix on the column space of
$\mathbf{A}$ and
$\mathbf{\Pi}^{\perp}(\mathbf{A})=\mathbf{I}_{N}-\mathbf{\Pi}(\mathbf{A})$ the
orthogonal projection matrix on the null space of $\mathbf{A}^{T}$. The
notation $\big{\lvert}\mathbf{X}\big{\rvert}$ denotes the determinant of the
matrix $\mathbf{X}$, $\lVert\cdot\rVert_{2}$ denotes the Euclidean norm and
$\lVert\cdot\rVert_{F}$ the Frobenius norm. $X\sim\mathcal{N}(\mu,\sigma^{2})$
signifies a Gaussian distributed random variable with mean $\mu$ and variance
$\sigma^{2}$. The symbol $\otimes$ represents the Kronecker product and
vec$(\mathbf{A})$ signifies the vectorization of the matrix
$\mathbf{\mathbf{A}}$.

##  II Problem Statement

Technically there can be four different linear regression structures depending
on the configuration of the parameter matrix (or vector). They are (a) single
measurement vector (SMV), (b) block single measurement vector (BSMV), (c)
multiple measurement vector (MMV) and (d) block multiple measurement vector
(BMMV). For example, as mentioned in [31], SMV models are used in wireless
signal detection [33], MMV models in Electroencephalogram (EEG) [34], BSMV
models in multi pitch estimation [35] and BMMV models in face recognition
[36]. Here, we consider the BMMV model, since it is the general setting and
the rest of the models are special cases of BMMV. The BMMV model is as
follows:

$\mathbf{Y}=\mathbf{AX}+\mathbf{W},$ (1)

where, $\mathbf{Y}\in\mathbb{R}^{N\times L}$ is the observed response matrix,
$\mathbf{A}\in\mathbb{R}^{N\times p}$ is the design matrix,
$\mathbf{X}\in\mathbb{R}^{p\times L}$ is the unknown parameter matrix and
$\mathbf{W}\in\mathbb{R}^{N\times L}$ is the noise/error matrix, whose
elements are assumed to be i.i.d. and
$\mathbf{W}[i,j]\sim\mathcal{N}(0,\sigma^{2})$. The $p$ rows of $\mathbf{X}$
are divided into $p_{B}=p/L_{B}$ unique blocks of equal size $L_{B}$. Each of
these $p_{B}$ blocks of size $L_{B}\times L$ is non-zero or zero at once. The
block size $L_{B}$ is assumed to be known a-priori. The $j$th block consists
of the rows of $\mathbf{X}$ indexed by
$\mathcal{I}_{j}=\\{(j-1)L_{B}+1,(j-1)L_{B}+2,\ldots,jL_{B}\\}$. We denote the
true block support of $\mathbf{X}$ as
$\mathcal{S}_{B}=\left\\{j\in[p_{B}]:\mathbf{X}[\mathcal{I}_{j},:]\neq\mathbf{0}_{L_{B}\times
L}\right\\}$. Also, $\mathbf{X}$ is assumed to be block-sparse such that
$K_{B}=card(\mathcal{S}_{B})\ll p_{B}$. Table I shows the different linear
regression structures. The goal of model selection herein is estimating
$\mathcal{S}_{B}$ given $\mathbf{Y}$ and $\mathbf{A}$.

Type | Specifications |  $dim(\mathbf{Y})$, $dim(\mathbf{X})$
---|---|---
SMV |  $L=1$, $L_{B}=1$, $p_{B}=p$ |  $N\times 1$, $p\times 1$
MMV |  $L>1$, $L_{B}=1$, $p_{B}=p$ |  $N\times L$, $p\times L$
BSMV |  $L=1$, $L_{B}>1$, $p_{B}=p/L_{B}$ |  $N\times 1$, $p\times 1$
BMMV |  $L>1$, $L_{B}>1$, $p_{B}=p/L_{B}$ |  $N\times L$, $p\times L$
TABLE I: Types of linear regression structures.

The model selection procedure can be divided into two stages: (i) In the first
stage, we pick a competent set of candidate models using an appropriate
predictor/subset selection algorithm up to maximum cardinality $K$ under the
assumption that $K_{B}\leq K\ll N$. (ii) In the second stage, we estimate the
true model using a suitable model selection criterion. Let us denote
$\mathcal{I}_{B}$ as the block support of a candidate model such that
$card(\mathcal{I}_{B})=k_{B}$, where $k_{B}\in\\{1,2,\ldots,p_{B}\\}$. Then we
can reformulate the linear model in (1) as

$\mathcal{H}_{\mathcal{I}_{B}}:\mathbf{Y}=\mathbf{A}_{\mathcal{I}_{B}}\mathbf{X}_{\mathcal{I}_{B}}+\mathbf{W}_{\mathcal{I}_{B}},$
(2)

where $\mathcal{H}_{\mathcal{I}_{B}}$ signifies the hypothesis that the data
$\mathbf{Y}$ is actually produced in accordance with (2),
$\mathbf{A}_{\mathcal{I}_{B}}\in\mathbb{R}^{N\times(k_{B}L_{B})}$ is the sub-
matrix consisting of columns from the known matrix $\mathbf{A}$ with block
support $\mathcal{I}_{B}\subseteq\\{1,2,\ldots,p_{B}\\}$,
$\mathbf{X}_{\mathcal{I}_{B}}\in\mathbb{R}^{(k_{B}L_{B})\times L}$ is the
corresponding unknown parameter coefficient matrix, and
$\mathbf{W}_{\mathcal{I}_{B}}\in\mathbb{R}^{N\times L}$ is the associated
noise matrix.

##  III Proposed Method

In this section, we provide the necessary steps to derive the
EBIC${}_{\text{R}}$ to perform model selection in block-sparse general linear
regression for the BMMV scenario. The further analysis assumes the following
property of the design matrix $\mathbf{A}$ [37, 38, 39]

$\lim_{N\to\infty}\left\\{N^{-1}(\mathbf{A}^{T}_{\mathcal{I}_{B}}\mathbf{A}_{\mathcal{I}_{B}})\right\\}=\mathbf{M}_{\mathcal{I}_{B}}=\mathcal{O}(1),$
(3)

where $\mathbf{M}_{\mathcal{I}_{B}}$ is a $(k_{B}L_{B}\times k_{B}L_{B})$
positive definite matrix and bounded as $N\to\infty$. The assumption in (3)
holds true in many cases but not all (see [40, 37] for more details).

To arrive at the EBIC${}_{\text{R}}$ for the BMMV model, we first reformulate
the linear model in (2) into vector form as

$\text{vec}(\mathbf{Y})=\mathbf{I}_{L}\otimes\mathbf{A}_{\mathcal{I}_{B}}\text{vec}(\mathbf{X}_{\mathcal{I}_{B}})+\text{vec}(\mathbf{W}_{\mathcal{I}_{B}}).$
(4)

This step allows us to utilize the same derivation steps as in [7] without the
need to carry out the analysis from scratch. Also, (4) is technically
equivalent to (2), hence we do not alter the underlying original linear model
but just restructure it for our convenience. Now, let
$\mathbf{y}=\text{vec}(\mathbf{Y})\in\mathbb{R}^{NL\times 1}$,
$\breve{\mathbf{A}}_{\mathcal{I}}=\mathbf{I}_{L}\otimes\mathbf{A}_{\mathcal{I}_{B}}\in\mathbb{R}^{NL\times
k_{B}L_{B}L}$,
$\mathbf{x}_{\mathcal{I}}=\text{vec}(\mathbf{X}_{\mathcal{I}_{B}})\in\mathbb{R}^{k_{B}L_{B}L\times
1}$ and
$\mathbf{e}_{\mathcal{I}}=\text{vec}(\mathbf{W}_{\mathcal{I}_{B}})\in\mathbb{R}^{NL\times
1}$. The elements of $\mathbf{e}_{\mathcal{I}}$ are i.i.d. and
$\mathbf{e}_{\mathcal{I}}\sim\mathcal{N}(\mathbf{0},\sigma^{2}_{\mathcal{I}}\mathbf{I}_{NL})$.
Then, we can rewrite (4) as

$\mathcal{H}_{{\mathcal{I}}}:\mathbf{y}=\breve{\mathbf{A}}_{{\mathcal{I}}}\mathbf{x}_{{\mathcal{I}}}+\mathbf{e}_{{\mathcal{I}}},$
(5)

where $\mathcal{I}\subseteq\\{1,2,\ldots,pL\\}$. Then the pdf of $\mathbf{y}$
under $\mathcal{H}_{\mathcal{I}}$ is

$p\left(\mathbf{y}\lvert\boldsymbol{\theta}_{\mathcal{I}},\mathcal{H}_{\mathcal{I}}\right)=\frac{\exp\\{-\lVert\mathbf{y}-{\breve{\mathbf{A}}}_{\mathcal{I}}\mathbf{x}_{\mathcal{I}}\rVert^{2}_{2}/2\sigma^{2}_{\mathcal{I}}\\}}{(2\pi\sigma^{2}_{\mathcal{I}})^{NL/2}},$
(6)

where
$\boldsymbol{\theta}_{\mathcal{I}}=[\mathbf{x}^{T}_{\mathcal{I}},\sigma^{2}_{\mathcal{I}}]^{T}$
is the vector of all the unknown parameters of the model under
$\mathcal{H}_{\mathcal{I}}$. The maximum likelihood estimates (MLE)
$\boldsymbol{\hat{\theta}}_{\mathcal{I}}=[\hat{\mathbf{x}}^{T}_{\mathcal{I}},\hat{\sigma}^{2}_{\mathcal{I}}]^{T}$
are obtained as [41]

$\hat{\mathbf{x}}_{\mathcal{I}}=\left({\breve{\mathbf{A}}}^{T}_{\mathcal{I}}{\breve{\mathbf{A}}}_{\mathcal{I}}\right)^{-1}{\breve{\mathbf{A}}}^{T}_{\mathcal{I}}\mathbf{y}\quad\&\quad\hat{\sigma}^{2}_{\mathcal{I}}=\frac{\mathbf{y}^{T}\mathbf{\Pi}^{\perp}(\breve{\mathbf{A}}_{\mathcal{I}})\mathbf{y}}{NL}.$
(7)

EBIC${}_{\text{R}}$ is derived under the Bayesian framework of model
selection, which starts with deriving the maximum a-posteriori (MAP) criterion
and ending with the final EBIC${}_{\text{R}}$ after suitable modifications and
reasonable assumptions. We follow similar steps as in [7, 8], but incorporate
the multi-measurement and block structure into it. Let us denote the prior pdf
of the parameter vector $\boldsymbol{\theta}_{\mathcal{I}}$ as
$p(\boldsymbol{\theta}_{\mathcal{I}}|\mathcal{H}_{\mathcal{I}})$, the marginal
of $\mathbf{y}$ as $p(\mathbf{y}|\mathcal{H}_{\mathcal{I}})$ and the prior
probability of the model with support $\mathcal{I}$ as
$\Pr(\mathcal{H}_{\mathcal{I}})$. Then the MAP estimate of the true support
$\mathcal{S}\subseteq\\{1,2,\ldots pL\\}$ is equivalently given by [37, 39]

$\hat{\mathcal{S}}_{\text{MAP}}=\underset{\mathcal{I}}{\arg\max}\ \Big{\\{}\ln
p(\mathbf{y}|\mathcal{H}_{\mathcal{I}})+\ln\Pr\left(\mathcal{H}_{\mathcal{I}}\right)\Big{\\}}.$
(8)

Applying a second order Taylor series expansion, an approximation of $\ln
p(\mathbf{y}|\mathcal{H}_{\mathcal{I}})$ is obtained under the presumption
that $N$ is large or/and SNR is high (see [37, 39] for details)

$\begin{split}\ln p(\mathbf{y}|\mathcal{H}_{\mathcal{I}})\approx\ln
p(\mathbf{y}|\hat{\boldsymbol{\theta}}_{\mathcal{I}},\mathcal{H}_{\mathcal{I}})+\ln
p(\hat{\boldsymbol{\theta}}_{\mathcal{I}}|\mathcal{H}_{\mathcal{I}})\\\
+\frac{k_{B}L_{B}L+1}{2}\ln(2\pi)-\frac{1}{2}\ln\big{|}\hat{\mathbf{F}}_{\mathcal{I}}\big{|}.\end{split}$
(9)

$\hat{\mathbf{F}}_{\mathcal{I}}$ is the sample Fisher information matrix [41]
under $\mathcal{H}_{\mathcal{I}}$ evaluated at the MLE, hence (see [37, 39])

$\hat{\mathbf{F}}_{\mathcal{I}}=\begin{bmatrix}\frac{1}{\hat{\sigma}^{2}_{\mathcal{I}}}{\breve{\mathbf{A}}}_{\mathcal{I}}^{T}{\breve{\mathbf{A}}}_{\mathcal{I}}&\mathbf{0}\\\
\mathbf{0}&\frac{NL}{2\hat{\sigma}^{4}_{\mathcal{I}}}\end{bmatrix}.$ (10)

From the linear model in 5 we have

$-2\ln
p(\mathbf{y}|\boldsymbol{\hat{\theta}}_{\mathcal{I}},\mathcal{H}_{\mathcal{I}})=NL\ln\hat{\sigma}^{2}_{\mathcal{I}}+\text{const}.$
(11)

Now, using (11), it is possible to rewrite (9) as

$\begin{split}-2\ln p(\mathbf{y}|\mathcal{H}_{\mathcal{I}})\approx
NL\ln\hat{\sigma}^{2}_{\mathcal{I}}+\ln\big{|}\hat{\mathbf{F}}_{\mathcal{I}}\big{|}-2\ln
p(\hat{\boldsymbol{\theta}}_{\mathcal{I}}|\mathcal{H}_{\mathcal{I}})\\\
-k_{B}L_{B}L\ln 2\pi+\text{const.}\end{split}$ (12)

Furthermore, the prior term in (9), i.e., $\ln
p(\hat{\boldsymbol{\theta}}_{\mathcal{I}}|\mathcal{H}_{\mathcal{I}})$, is
ignored under the pretext that it is flat and uninformative. Thus, discarding
the constants and the terms not dependent on the block model dimension
$k_{B}$, we can equivalently reformulate the MAP-based model estimate as

$\displaystyle\hat{\mathcal{S}}_{\text{MAP}}=\underset{\mathcal{I}}{\arg\min}\Big{\\{}NL\ln\hat{\sigma}^{2}_{\mathcal{I}}+\ln$
$\displaystyle\big{|}\hat{\mathbf{F}}_{\mathcal{I}}\big{|}-k_{B}L_{B}L\ln
2\pi$ $\displaystyle-2\ln\Pr\left(\mathcal{H}_{\mathcal{I}}\right)\Big{\\}}.$
(13)

EBIC${}_{\text{R}}$ is derived from (13) with some further modifications and
approximations. The two key terms that require further analysis are
$\ln\lvert\hat{\mathbf{F}}_{\mathcal{I}}\rvert$ and the prior term
$\Pr(\mathcal{H}_{\mathcal{I}})$. First, we perform normalization of
$\hat{\mathbf{F}}_{\mathcal{I}}$ under both large-$N$ and high-SNR assumption.
For this we factorize the
$\ln\big{\lvert}\hat{\mathbf{F}}_{\mathcal{I}}\big{\rvert}$term in a similar
manner as performed in [7, 8, 39]

$\displaystyle\ln{\big{|}\hat{\mathbf{F}}_{\mathcal{I}}\big{|}}=$
$\displaystyle\ln\left[\big{|}\mathbf{Q}\big{|}\left\lvert\mathbf{Q}^{-1/2}\hat{\mathbf{F}}_{\mathcal{I}}\mathbf{Q}^{-1/2}\right\rvert\right]$
$\displaystyle=$
$\displaystyle\ln|\mathbf{Q}|+\ln{\Big{|}\mathbf{Q}^{-1/2}\mathbf{\hat{F}}_{\mathcal{I}}\mathbf{Q}^{-1/2}\Big{|}}.$
(14)

The objective here is to choose a suitable $\mathbf{Q}$ matrix that normalizes
$\hat{\mathbf{F}}_{\mathcal{I}}$ such that the second term in (14) is
$\mathcal{O}(1)$, i.e., it should be bounded as $N\to\infty$ and/or
$\sigma^{2}\to 0$. To achieve this purpose, we choose the following
$\mathbf{Q}^{-1/2}$ matrix [7]

$\mathbf{Q}^{-1/2}=\begin{bmatrix}\sqrt{\frac{L_{B}}{N}}\sqrt{\frac{\hat{\sigma}_{\mathcal{I}}^{2}}{\hat{\sigma}_{0}^{2}}}\mathbf{I}_{k_{B}L_{B}L}&\mathbf{0}\\\
\mathbf{0}&\sqrt{\frac{L_{B}}{N}}\left(\frac{\hat{\sigma}_{\mathcal{I}}^{2}}{\hat{\sigma}_{0}^{2}}\right)\end{bmatrix},$
(15)

where $\hat{\sigma}^{2}_{0}=\lVert\mathbf{y}\rVert^{2}_{2}/NL$. Also for the
considered generating model (5), $\hat{\sigma}^{2}_{0}\to$ const. as
$N\to\infty$ and/or $\sigma^{2}\to 0$ [38, 39]. Two important points to note
here regarding the choice of the $\mathbf{Q}^{-1/2}$ matrix are: (i) The ratio
$\left(\frac{\hat{\sigma}_{\mathcal{I}}^{2}}{\hat{\sigma}_{0}^{2}}\right)$ is
introduced to normalize the $\hat{\mathbf{F}}_{\mathcal{I}}$ w.r.t.
$\sigma^{2}$ where the factor $\hat{\sigma}^{2}_{0}$ is especially utilized to
counteract the data scaling problem (as discussed elaborately in [39, 7]).
(ii) The $\frac{1}{N}$ portion of the factor $\frac{L_{B}}{N}$ is used to
normalize the FIM w.r.t. $N$. However, $L_{B}$ is also included as part of the
normalizing term because for the mean-squared-error of $\hat{\sigma}^{2}$ to
approach the Cramér-Rao bound, we require that the number of measurements is
much larger than the number of parameters, i.e., $NL\gg K_{B}L_{B}L$ or in
other words $N/L_{B}\gg K_{B}$. Hence, we use the normalization factor
$L_{B}/N$ instead of just $1/N$ in (15). In this way, the penalty will be a
function of $N/L_{B}$ instead of $N$ alone (as will be seen in the subsequent
steps). This novel modification helps to counteract the effects of changing
$L_{B}$ on the performance of EBIC${}_{\text{R}}$.

Now, using (3), (10), and (15) we can show that

$\displaystyle\Big{\lvert}\mathbf{Q}^{-1/2}\mathbf{\hat{F}}_{\mathcal{I}}\mathbf{Q}^{-1/2}\Big{\rvert}=$
$\displaystyle\begin{vmatrix}{\frac{L_{B}}{\hat{\sigma}_{0}^{2}}}{\frac{\breve{{\mathbf{A}}}^{T}_{\mathcal{I}}{\breve{\mathbf{A}}}_{\mathcal{I}}}{N}}&\mathbf{0}\\\
\mathbf{0}&\frac{L_{B}L}{2\hat{\sigma}_{0}^{4}}\end{vmatrix}$ $\displaystyle=$
$\displaystyle\
\frac{L_{B}^{k_{B}L_{B}L+1}L}{2(\hat{\sigma}^{2}_{0})^{k_{B}L_{B}L+2}}\left\lvert\mathbf{I}_{L}\otimes{\frac{\mathbf{A}^{T}_{\mathcal{I}_{B}}\mathbf{A}_{\mathcal{I}_{B}}}{N}}\right\rvert$
$\displaystyle=$ $\displaystyle\
\text{const.}\times\left\lvert\mathbf{I}_{L}\right\rvert^{k_{B}\times
L_{B}}\left\lvert{\frac{\mathbf{A}^{T}_{\mathcal{I}_{B}}\mathbf{A}_{\mathcal{I}_{B}}}{N}}\right\rvert^{L}$
$\displaystyle=$ $\displaystyle\ \mathcal{O}(1)$ (16)

as $N$ grows large and/or $\sigma^{2}\to 0$. Hence, this term can be removed
without significantly affecting the criterion. Next, observe that the
$\ln\big{\lvert}\mathbf{Q}\big{\rvert}$ term in 14 can be expanded as follows

$\displaystyle\ln|\mathbf{Q}|$
$\displaystyle=\ln\begin{vmatrix}\left(\frac{N}{L_{B}}\right)\left(\frac{\hat{\sigma}_{0}^{2}}{\hat{\sigma}^{2}_{\mathcal{I}}}\right)\mathbf{I}_{k_{B}L_{B}L}&\mathbf{0}\\\
\mathbf{0}&\left(\frac{N}{L_{B}}\right)\left(\frac{\hat{\sigma}^{2}_{0}}{\hat{\sigma}^{2}_{\mathcal{I}}}\right)^{2}\end{vmatrix}$
$\displaystyle=(k_{B}L_{B}L+1)\ln\left(\frac{N}{L_{B}}\right)+(k_{B}L_{B}L+2)\ln\left(\frac{\hat{\sigma}_{0}^{2}}{\hat{\sigma}_{\mathcal{I}}^{2}}\right).$
(17)

Therefore, using (16) and (17) we can rewrite (14) as

$\begin{split}\ln{\big{|}\hat{\mathbf{F}}_{\mathcal{I}}\big{|}}=k_{B}L_{B}L\ln\left(\frac{N}{L_{B}}\right)+(k_{B}L_{B}L+2)\ln\left(\frac{\hat{\sigma}_{0}^{2}}{\hat{\sigma}_{\mathcal{I}}^{2}}\right)+\\\
\mathcal{O}(1)+\ln\left({N}/{L_{B}}\right).\end{split}$ (18)

Next, for the model prior probability term
$-2\ln\Pr(\mathcal{H}_{\mathcal{I}})$ in (13), a similar strategy is adopted
as in EBIC [5] such that $\Pr(\mathcal{H}_{\mathcal{I}})\propto{p_{B}\choose
k_{B}}^{-\zeta}$, where $\zeta\geq 0$ is a tuning parameter. If $p_{B}$ is
sufficiently large, the following approximation can be assumed
$\ln{p_{B}\choose k_{B}}\approx k_{B}\ln p_{B}$ [6]. This gives

$-2\ln\Pr(\mathcal{H}_{\mathcal{I}})=2\zeta k_{B}\ln p_{B}+\text{const}.$ (19)

Now, substituting (18), (19) in (13) and dropping the $\mathcal{O}(1)$, the
$\ln\left(N/L_{B}\right)$ term (since independent of $k_{B}$), the constant
and the $p(\hat{\boldsymbol{\theta}}_{\mathcal{I}}|\mathcal{H}_{\mathcal{I}})$
term we arrive at the EBIC${}_{\text{R}}$ for BMMV

$\begin{split}\text{EBIC}_{\text{R}}(\mathcal{I})=&NL\ln\hat{\sigma}^{2}_{\mathcal{I}}+k_{B}L_{B}L\ln\left(\frac{N}{2\pi
L_{B}}\right)\\\
&+(k_{B}L_{B}L+2)\ln\left(\frac{\hat{\sigma}_{0}^{2}}{\hat{\sigma}_{\mathcal{I}}^{2}}\right)+2k_{B}\zeta\ln
p_{B}.\end{split}$ (20)

In practice, we compute the EBIC${}_{\text{R}}$ score block-wise, i.e.,
EBIC${}_{\text{R}}(\mathcal{I}_{B})$ where
$\mathcal{I}_{B}\subseteq\\{1,\ldots,p_{B}\\}$. Then the
$\hat{\sigma}^{2}_{\mathcal{I}}$ can be replaced by
$\hat{\sigma}^{2}_{\mathcal{I}_{B}}=\lVert\mathbf{\Pi}^{\perp}(\mathbf{A}_{\mathcal{I}_{B}})\mathbf{Y}\rVert^{2}_{F}\big{/}NL$.
Finally, the true block support is estimated as

$\hat{\mathcal{S}}_{B}=\underset{\mathcal{I}_{B}}{\arg\min}\big{\\{}\text{EBIC}_{\text{R}}(\mathcal{I}_{B})\big{\\}}.$
(21)

Algorithm 1 BMMV-OMP with $K$ iterations

Inputs: Design matrix $\mathbf{A}$, measurement $\mathbf{Y}$.

Initialization: $\lVert\mathbf{a}_{j}\rVert_{2}=1\ \forall j$,
$\mathbf{R}^{0}=\mathbf{Y}$, $\mathcal{S}^{0}_{\text{B-OMP}}=\emptyset$

for $i=1\text{\ to\ }K$ do

Next block index:
$d^{i}=\underset{j=1,\ldots,p_{B}}{\arg\max}\left\lVert\mathbf{A}[:,\mathcal{I}_{j}]^{T}\mathbf{R}^{i-1}\right\rVert_{F}$

Add current index:
$\mathcal{S}^{i}_{\textrm{B-OMP}}=\mathcal{S}^{i-1}_{\text{B-OMP}}\cup\\{d^{i}\\}$

Update residual:
$\mathbf{R}^{i}=\mathbf{\Pi}^{\perp}(\mathbf{A}_{\mathcal{S}^{i}_{\textrm{B-OMP}}})\mathbf{Y}$

end for

Output: B-OMP generated block index sequence $\mathcal{S}^{K}_{\text{B-OMP}}$

Algorithm 2 Model selection combining EBIC${}_{\text{R}}$ with B-OMP

Run B-OMP for $K$ iterations to obtain $\mathcal{S}^{K}_{\text{B-OMP}}$

for $k_{B}=1\text{\ to\ }K$ do

$\mathcal{I}_{B}=\mathcal{S}^{k_{B}}_{\text{B-OMP}}$

Compute EBIC${}_{\text{R}}(\mathcal{I}_{B})$

end for

Estimated true block support:
$\hat{\mathcal{S}}_{B}=\underset{\mathcal{I}_{B}}{\arg\min}\\{$EBIC${}_{\text{R}}(\mathcal{I}_{B})\\}$

##  IV Simulation Results

In this section, we provide numerical simulations to highlight the performance
of EBIC${}_{\text{R}}$ for model selection in BMMV models. We consider the
linear model $\mathbf{Y}=\mathbf{A}\mathbf{X}+\mathbf{W}$, where the design
matrix $\mathbf{A}$ is generated with independent entries following normal
distribution $\mathcal{N}(0,1)$. The cardinality of the true block-support
$\mathcal{S}_{B}$ is chosen to be $K_{B}=4$. Also, without loss of generality,
we assume $\mathcal{S}_{B}=[1,2,3,4]$. The non-zero entries in $\mathbf{X}$
are randomly assigned $\pm 1$. The SNR in dB =
$10\log_{10}(\sigma_{s}^{2}/\sigma^{2})$, where $\sigma_{s}^{2}$ and
$\sigma^{2}$ denote signal and true noise power, respectively. The signal
power is computed as
$\sigma_{s}^{2}=\lVert\mathbf{A}\mathbf{X}\rVert^{2}_{F}/NL$. The chosen SNR
(dB) and $\sigma_{s}^{2}$ are then used to determine the noise power as
$\sigma^{2}=\sigma_{s}^{2}/10^{\textrm{SNR (dB)}/10}$. Using this
$\sigma^{2}$, the elements of the noise matrix $\mathbf{W}$ are generated
following
$\mathbf{W}[i,j]\overset{\text{i.i.d}}{\sim}\mathcal{N}(0,\sigma^{2})$. The
probability of correct model selection (PCMS), i.e.,
$\Pr(\hat{\mathcal{S}}_{B}=\mathcal{S}_{B})$ is evaluated over $1000$ Monte
Carlo trials. At each Monte Carlo trial, a new design matrix $\mathbf{A}$ is
generated in order to preserve the randomness in the data. For
predictor/subset selection, BMMV-OMP (B-OMP) [42, 31] (Algorithm 1) is
utilized because of its ease of use and broad application. The performance of
EBIC${}_{\text{R}}$ is compared with GRRT and the oracle, which is B-OMP with
a-priori knowledge of the block sparsity $K_{B}$. Hence, the oracle provides
the upper bound on the maximum achievable PCMS for any given setting. The
tuning parameters chosen are $\alpha=0.01$ for GRRT (as mentioned in [31]) and
$\zeta=1$ (EBIC${}_{\text{R}}$) [7, 8].

Fig. 1 shows the PCMS vs SNR (dB) with $N=150$ and $p=1000$. Since $L_{B}=10$,
hence, $p_{B}=p/L_{B}=100$. Additionally, the performance is shown for two
different settings of the $L$ parameter, viz. $L=5$ and $15$ to highlight the
influence of $L$ on the overall behaviour of the methods. The first clear
observation is that for the considered tuning parameter setting, both
EBIC${}_{\text{R}}$ and GRRT are empirically consistent in high-SNR, i.e.,
PCMS $\to 1$ as SNR $\to\infty$ (or inversely $\sigma^{2}\to 0$). Second,
compared to GRRT, the performance curve of EBIC${}_{\text{R}}$ is much closer
to the oracle, especially for low values of SNR. Furthermore, compared to
$L=5$, the oracle plot shifts toward the left when $L=15$. This indicates that
increasing $L$ improves the true support recovery ability of B-OMP, which
ultimately improves the model selection performance of the methods.

Fig. 2 presents the PCMS vs number of measurements $N$ plot. Here, a fixed
value of $p=5000$ is chosen. Additionally, the performance is shown for two
separate values of the $L_{B}$ variable, viz. $L_{B}=5$ and $20$ to highlight
the impact of $L_{B}$ on the overall model selection performance. A similar
trend is observed here as well. Both the methods achieve empirical consistency
(PCMS $\to 1$) as $N$ grows large. However, EBIC${}_{\text{R}}$ provides
slightly better performance compared to GRRT for smaller $N$ values, and is
much closer to the oracle performance. Furthermore, we also observe that
increasing $L_{B}$ lowers the support recovery performance of B-OMP, which is
obvious from the shift in the oracle performance towards the right. Thus, it
requires more samples to achieve the same PCMS for $L_{B}=20$ as compared to
$L_{B}=5$. This ultimately lowers the overall performance of all model
selection methods.

Figure 1: PCMS vs SNR (dB) for $N=150$, $p=1000$, $L=[5,15]$, $L_{B}=10$ and
$K_{B}=4$. Figure 2: PCMS vs $N$ for SNR = -4 dB, $p=5000$, $L=5$,
$L_{B}=[5,20]$ and $K_{B}=4$.

##  V Conclusion

In this paper, we have extended the EBIC${}_{\text{R}}$ to handle model
selection in the block-sparse HD linear regression. A generalized method is
developed that is applicable to all forms of the linear regression structure
such as SMV, BSMV, MMV, and BMMV. The steps to arrive at the criterion are
shown in detail. Simulation results show that EBIC${}_{\text{R}}$ is an
empirically consistent criterion as $N\to\infty$ and/or SNR $\to\infty$. Also.
Its performance for lower SNR and $N$ values is close to the oracle behaviour.
Furthermore, we also underline the manner in which the parameters $L$ and the
block length $L_{B}$ affect the model selection performance.

## References

  * [1] H. Akaike, “A new look at the statistical model identification,” _IEEE transactions on automatic control_ , vol. 19, no. 6, pp. 716–723, 1974.
  * [2] C. Rao, Y. Wu, S. Konishi, and R. Mukerjee, “On model selection,” _Lecture Notes-Monograph Series_ , pp. 1–64, 2001.
  * [3] P. Stoica and Y. Selen, “Model-order selection: a review of information criterion rules,” _IEEE Signal Processing Magazine_ , vol. 21, no. 4, pp. 36–47, 2004.
  * [4] A. Chakrabarti and J. K. Ghosh, “AIC, BIC and recent advances in model selection,” _Philosophy of statistics_ , pp. 583–605, 2011.
  * [5] J. Chen and Z. Chen, “Extended Bayesian information criteria for model selection with large model spaces,” _Biometrika_ , vol. 95, no. 3, pp. 759–771, 2008.
  * [6] A. Owrang and M. Jansson, “A model selection criterion for high-dimensional linear regression,” _IEEE Transactions on Signal Processing_ , vol. 66, no. 13, pp. 3436–3446, 2018.
  * [7] P. B. Gohain and M. Jansson, “New improved criterion for model selection in sparse high-dimensional linear regression models,” in _ICASSP IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , 2022, pp. 5692–5696.
  * [8] ——, “Robust information criterion for model selection in sparse high-dimensional linear regression models,” _arXiv preprint arXiv:2206.08731_ , 2022.
  * [9] S. Kallummil and S. Kalyani, “Signal and noise statistics oblivious orthogonal matching pursuit,” in _International Conference on Machine Learning_. PMLR, 2018, pp. 2429–2438.
  * [10] P. B. Gohain and M. Jansson, “Relative cost based model selection for sparse high-dimensional linear regression models,” in _ICASSP IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. IEEE, 2020, pp. 5515–5519.
  * [11] T. T. Cai and L. Wang, “Orthogonal matching pursuit for sparse signal recovery with noise,” _IEEE Transactions on Information theory_ , vol. 57, no. 7, pp. 4680–4688, 2011.
  * [12] R. Lockhart, J. Taylor, R. J. Tibshirani, and R. Tibshirani, “A significance test for the lasso,” _Annals of statistics_ , vol. 42, no. 2, p. 413, 2014\.
  * [13] R. F. Barber and E. J. Candès, “A knockoff filter for high-dimensional selective inference,” _The Annals of Statistics_ , vol. 47, no. 5, pp. 2504–2537, 2019.
  * [14] J. Shao, “Linear model selection by cross-validation,” _Journal of the American statistical Association_ , vol. 88, no. 422, pp. 486–494, 1993.
  * [15] R. R. Picard and R. D. Cook, “Cross-validation of regression models,” _Journal of the American Statistical Association_ , vol. 79, no. 387, pp. 575–583, 1984.
  * [16] L. de Torrenté and T. Hastie, “Does cross-validation work when $p\gg n$?” 2012\.
  * [17] M. Chichignoud, J. Lederer, and M. J. Wainwright, “A practical scheme and fast algorithm to tune the lasso with optimality guarantees,” _The Journal of Machine Learning Research_ , vol. 17, no. 1, pp. 8162–8181, 2016.
  * [18] S. Arlot and A. Celisse, “A survey of cross-validation procedures for model selection,” _Statistics surveys_ , vol. 4, pp. 40–79, 2010.
  * [19] Y. C. Eldar and M. Mishali, “Block sparsity and sampling over a union of subspaces,” in _2009 16th International Conference on Digital Signal Processing_. IEEE, 2009, pp. 1–8.
  * [20] ——, “Robust recovery of signals from a structured union of subspaces,” _IEEE Transactions on Information Theory_ , vol. 55, no. 11, pp. 5302–5316, 2009.
  * [21] Y. C. Eldar, P. Kuppinger, and H. Bolcskei, “Block-sparse signals: Uncertainty relations and efficient recovery,” _IEEE Transactions on Signal Processing_ , vol. 58, no. 6, pp. 3042–3054, 2010.
  * [22] M. Mishali and Y. C. Eldar, “Blind multiband signal reconstruction: Compressed sensing for analog signals,” _IEEE Transactions on signal processing_ , vol. 57, no. 3, pp. 993–1009, 2009.
  * [23] ——, “From theory to practice: Sub-nyquist sampling of sparse wideband analog signals,” _IEEE Journal of selected topics in signal processing_ , vol. 4, no. 2, pp. 375–391, 2010.
  * [24] F. Parvaresh, H. Vikalo, S. Misra, and B. Hassibi, “Recovering sparse signals using sparse measurement matrices in compressed dna microarrays,” _IEEE Journal of Selected Topics in Signal Processing_ , vol. 2, no. 3, pp. 275–285, 2008.
  * [25] S. F. Cotter, B. D. Rao, K. Engan, and K. Kreutz-Delgado, “Sparse solutions to linear inverse problems with multiple measurement vectors,” _IEEE Transactions on Signal Processing_ , vol. 53, no. 7, pp. 2477–2488, 2005.
  * [26] J. Chen and X. Huo, “Theoretical results on sparse representations of multiple-measurement vectors,” _IEEE Transactions on Signal processing_ , vol. 54, no. 12, pp. 4634–4643, 2006.
  * [27] A. Owrang and M. Jansson, “Weighted covariance matching based square root lasso,” in _2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. IEEE, 2015, pp. 3751–3755.
  * [28] Y. C. Eldar and H. Rauhut, “Average case analysis of multichannel sparse recovery using convex relaxation,” _IEEE Transactions on Information Theory_ , vol. 56, no. 1, pp. 505–519, 2009.
  * [29] M. Mishali and Y. C. Eldar, “Blind multiband signal reconstruction: Compressed sensing for analog signals,” _IEEE Transactions on signal processing_ , vol. 57, no. 3, pp. 993–1009, 2009.
  * [30] K. Gedalyahu and Y. C. Eldar, “Time-delay estimation from low-rate samples: A union of subspaces approach,” _IEEE Transactions on Signal Processing_ , vol. 58, no. 6, pp. 3017–3031, 2010.
  * [31] S. Kallummil and S. Kalyani, “Generalized residual ratio thresholding,” _Signal Processing_ , vol. 197, p. 108531, 2022.
  * [32] R. Tibshirani, “Regression shrinkage and selection via the lasso,” _Journal of the Royal Statistical Society: Series B (Methodological)_ , vol. 58, no. 1, pp. 267–288, 1996.
  * [33] J. W. Choi and B. Shim, “Detection of large-scale wireless systems via sparse error recovery,” _IEEE Transactions on Signal Processing_ , vol. 65, no. 22, pp. 6038–6052, 2017.
  * [34] S. Aviyente, “Compressed sensing framework for eeg compression,” in _2007 IEEE/SP 14th workshop on statistical signal processing_. IEEE, 2007, pp. 181–184.
  * [35] T. Kronvall, S. I. Adalbjörnsson, S. Nadig, and A. Jakobsson, “Group-sparse regression using the covariance fitting criterion,” _Signal Processing_ , vol. 139, pp. 116–130, 2017.
  * [36] I. Fedorov, R. Giri, B. D. Rao, and T. Q. Nguyen, “Robust bayesian method for simultaneous block sparse signal recovery with applications to face recognition,” in _2016 IEEE International Conference on Image Processing (ICIP)_. IEEE, 2016, pp. 3872–3876.
  * [37] P. Stoica and P. Babu, “On the proper forms of BIC for model order selection,” _IEEE Transactions on Signal Processing_ , vol. 60, no. 9, pp. 4956–4961, 2012.
  * [38] D. F. Schmidt and E. Makalic, “The consistency of MDL for linear regression models with increasing signal-to-noise ratio,” _IEEE transactions on signal processing_ , vol. 60, no. 3, pp. 1508–1510, 2011.
  * [39] P. B. Gohain and M. Jansson, “Scale-invariant and consistent Bayesian information criterion for order selection in linear regression models,” _Signal Processing_ , p. 108499, 2022.
  * [40] P. M. Djuric, “Asymptotic MAP criteria for model selection,” _IEEE Transactions on Signal Processing_ , vol. 46, no. 10, pp. 2726–2735, 1998.
  * [41] S. M. Kay, _Fundamentals of statistical signal processing: estimation theory_. Prentice Hall PTR, 1993.
  * [42] Y. Shi, L. Wang, and R. Luo, “Sparse recovery with block multiple measurement vectors algorithm,” _IEEE Access_ , vol. 7, pp. 9470–9475, 2019.


# Outdoor Monocular Depth Estimation: A Research Review

Pulkit Vyas, Chirag Saxena, Anwesh Badapanda, Anurag Goswami  School of
Computer Science and Technology
Bennett University
Greater Noida, India
{e19cse284, e19cse124, e19cse058, anurag.goswami} @bennett.edu.in

###### Abstract

Depth estimation is an important task, applied in various methods and
applications of computer vision. While the traditional methods of estimating
depth are based on depth cues and require specific equipment such as stereo
cameras and configuring input according to the approach being used, the focus
at the current time is on a single source, or monocular, depth estimation. The
recent developments in Convolution Neural Networks along with the integration
of classical methods in these deep learning approaches have led to a lot of
advancements in the depth estimation problem. The problem of outdoor depth
estimation, or depth estimation in wild, is a very scarcely researched field
of study. In this paper, we give an overview of the available datasets, depth
estimation methods, research work, trends, challenges, and opportunities that
exist for open research. To our knowledge, no openly available survey work
provides a comprehensive collection of outdoor depth estimation techniques and
research scope, making our work an essential contribution for people looking
to enter this field of study.

###### Index Terms:

Monocular Depth Estimation, Outdoor dataset, Deep learning

##  I Introduction

The ability to detect, classify, segment, and reconstruct outdoor and long-
range distance scenes is an important requirement for computer vision
techniques in application domains and use-cases of autonomous vehicles,
robotic systems, 3D architectural modeling, terrestrial surveys, and AR/VR
[2]. Extracting depth information from outdoor scenes becomes a task of utmost
importance since that provides a lot of context about the spatial and logical
relationship between the different entities present in them. Techniques such
as robust point-cloud-based methods or stereo-based methods might seem to be a
viable solution to this problem, and in fact, there has been a lot of research
being done on them [3][4]. The limiting factors of mass adoption of these
technologies are the requirement of specific equipment and the restriction of
usage of the input data from them. Most applications cannot account for the
need for sensors such as LiDAR needed by these methods, moreover, the
applications might include computer vision techniques such as object
detection, tracking, and segmentation that would require 2D images instead of
sensor data. Thus the requirement for monocular depth estimation systems that
work for outdoor data is quite evident.

The biggest challenge with outdoor monocular depth estimation is the lack of
perspective changes in the scenes and frames of input image data [18][23].
This happens because when the subject of the scene is much bigger than the
focal view size of the camera then the overall change between the elements
captured in a set of frames would be much less compared to the same camera
being used in an indoor setting. Classical techniques based on depth cues from
motion and texture thus fail because of the lack of dynamic nature [5][6][7].
Fig. 1 gives an overview of the evolution of techniques used for depth
estimation.

Figure 1: Evolution of techniques for depth estimation over the years.
Inspired from [8]

This issue is accompanied by the lack of publicly available outdoor datasets
for the training and evaluation of models [18]. Depth estimation datasets are
majorly used in research work in the fields of robotics and 3D reconstruction,
both of which are predominantly indoor-based activities in their current form
and industrial usage. Due to this, the number of outdoor datasets is very
less, with the ones targeting very long-range data being very few.

The current research in the direction of outdoor depth estimation is driven by
the need of capturing or synthetically generating outdoor scene and their
depth data, building deep learning-based models and integrating traditional
methods wherever applicable, and, applying various learning approaches and
improving upon them [8].

This paper concentrates on the research of monocular depth estimation in the
context of outdoor scenes, surveying the developments and trends in deep
learning-based approaches over the last few years. We also provide a look at
the limitations of current research work and highlight future research
directions. The rest of the paper is structured in the following way: Section
2 summarizes the datasets that are either built solely with outdoor scenes or
have some components of them, section 3 introduces the previous and
contemporary deep learning models proposed over the years, section 4 discusses
the variety of techniques used for training deep learning models for the task
of depth estimation, section 5 outlines the challenges and trends of outdoor
depth estimation, we thus conclude in section 6.

##  II Methodology

A literature review is supposed to integrate existing information, detect
issues related to bias or problematic trends, and identify gaps in the
literature of a particular field. Since our study aims to survey and summarize
the previous as well as contemporary work in the domain of outdoor depth
estimation, the methodology that we follow was to do a backward snowballing of
research work basis on the recent works accepted at popular conferences in
computer vision like CVPR and ICCV, rather than discovering the papers with
the arbitrary search of keywords. Instead, we first collected a suite of
relevant keywords and used them to conduct the survey search using Google
Scholar, Scopus, and Connected Paper’s search engines. This allowed us to find
the patterns of research, development, and industry adoption over the years
and the kind of approaches that were involved in them.

The “ill-posed” nature of the problem of monocular depth estimation is
especially highlighted when done in outdoor settings. This creates a
significant difference in the kind of methods used by indoor depth estimation
solutions and the outdoor ones. Due to this, we divide the datasets, deep
learning approaches as well as training paradigms in such a way to point out
the progress and limitations of current works in this context.

The inclusion and exclusion of papers in this work are based on the criteria
defined in Table 1.

TABLE I: Inclusion and Exclusion Criterion Criterion |  Included
---|---
Papers that are focused on outdoor depth estimation | Yes
Papers using monocular sources, stereo sources, synthetic data, or panoramic images as datasets | Yes
Papers that exclusively use indoor depth estimation datasets | No
Papers that only use private datasets | No
Papers that use depth estimation as the input rather than assisting or improving the results of an MDE system | No

##  III Datasets

The work of [9] gives a comprehensive review of the publicly available
datasets for monocular depth estimation. Still, the majority of datasets meant
for monocular depth estimation are created in indoor settings [10]. This is
primarily because it is much easier, both in terms of equipment cost and human
effort, to create a dataset indoors as compared to taking the various steps of
outdoor data collection, annotation, and pre-processing in consideration and
creating a novel dataset based on that. Yet, there are a few notable datasets
available for the outdoor depth estimation, here we discuss these datasets
into the following categories: generic outdoor datasets, panoramic datasets,
and, generative methods. There are a few other datasets like [62] that are
adapted from other domains such as semantic segmentation for the task of
monocular depth estimation but are not inherently made for these tasks and so
they don’t contain depth maps as GT. Depth is extracted in these using
artificial methods making the data not ideal for training MDE models but
still, a lot of research utilizes these. We do not discuss such datasets in
this section but reference them when discussing methods that use them in
subsequent sections.

###  III-A Generic outdoor datasets

The collection of data is generally done either using moving vehicles or
pictures of buildings and scenes with a limited focal view at a short distance
(less than 100 meters). This category of depth estimation datasets include the
KITTI dataset [10], Make3D dataset [11], Newer College dataset [12], Megadepth
dataset [13], DIODE [14] and DrivingStereo dataset [15]. The subsequent
subsections give a brief on the two of the most used of these datasets:

####  III-A1 KITTI

The KITTI dataset [10] is the most used reference with exterior scenes taken
from a moving vehicle. There are two main divisions used to estimate monocular
depth in outdoor environments. One is a training/test set with 23,488 pairs of
training images and 697 test images. The other is an official crash with
42,949 pairs of training images, 1000 validation images, and 500 test images.
For the official division, the true depth map for the test images is retained
with the benchmark reviewer to test models against new data. Fig. 2 provides a
few examples of the outdoor parts of the KITTI dataset.

Figure 2: Sections of the KITTI dataset where scenes are captured in an
outdoor setting

####  III-A2 Make3D

The Make3D dataset [11] is another outdoor dataset for deep learning-based
monocular depth estimation. The Make3D dataset includes cityscapes and natural
landscapes captured during the daytime, with the collection of depth maps done
by laser scanners. The dataset contains a total of 53 pairs of RGBD images, of
which 400 pairs are used for training and 13 pairs are used for testing. The
native RGB image resolution is 2272 x 170 and the depth map resolution is 55 x
305 pixels. Fig. 3 gives an overlook of the Make3D dataset.

Figure 3: The Make3D Laser + Image dataset

###  III-B Panoramic datasets

These datasets are collected using panoramic cameras and meant for usage by
large-focal length input devices as well as 3D cameras. Panoramic datasets
have been used in a lot of recent research work [16][17]. Still, the number of
outdoor panoramic datasets is quite low [18][19], resulting in a lack of
research on the depth estimation for the same. The noteworthy panoramic
datasets are Multi-FoV (Urban Canyon) dataset [20], ETH3D [21], and Forest
Virtual [22]. A few snippets from the dataset are shown in Fig. 4.

Figure 4: The ETH3D dataset with panoramic images and corresponding depth maps

###  III-C Generative methods

Depending upon the domain of the application that particular research work is
focusing on, the availability of the specific datasets might be a roadblock.
The works of [23] and [24] adapt existing datasets such as the KITTI dataset
or public image and 3D data from sources like Google Maps to generate novel
datasets that have very long-range scenes and 360 panoramic scenes,
respectively.

####  III-C1 FarSight

This work [23] is a strategy for generating very-long range outdoor images
along with annotated depth maps. They use a new strategy for generating
aggregate data of long-range ground truth depths. They used images from Google
Earth to recreate large-scale 3D models of different cities at the appropriate
scale. The acquired archive of 3D models and associated RGB views and their
long-range depth rendering is used as training data for depth prediction. Fig.
5 shows the resulting images and depth maps from following this process.

Figure 5: From left to right: the 3D models of the cityscapes extracted from
Google Earth, depth prediction using the paper’s GAN model, depth prediction
using the paper’s CNN model, ground truth depth maps

####  III-C2 KITTI to panoramic dataset adaption

In this approach [24], a two-step process is used where the images from the
source, the KITTI dataset, are transformed into a replicated version of their
target domain which requires images to be captured by 360-degree FOV cameras.
The first step is the style transfer of 360-degree images to the images in the
target dataset via a learning-based approach [25]. The second step involves
reprojection of the resulting images to the required format along with the
relevant annotations. The generated depth maps can be visualized in Fig. 6

Figure 6: Depth recovery from panoramic imagery using the approach described
in [24]

##  IV Deep Learning Approaches

###  IV-A Convolutional Neural Networks

Convolutional neural networks are primarily used on images with the major
components - convolutional layers, pooling layers (max pooling and average
pooling), and activation functions - which together allow these networks in
learning 2D spatial features of input images. In the context of depth
estimation, CNNs are used for extracting depth features from images, reducing
the size of these extractions using the pooling layers, and reconstructing the
depth maps using their activation functions and FC layers. CNN-based
approaches for outdoor depth estimation, mostly for KIITI dataset experiments,
include Convolutional Neural Fields [26] AdaBins [27], FarSight [23], and,
Unsupervised CNN [28], all of which use CNNs as the backend for the main depth
estimation network. Fig. 7 shows the steps involved in predicting depth maps
using convolutional layers.

Figure 7: The general pipeline of deep learning for monocular depth estimation
using CNNs. Source: [8]

###  IV-B Recurrent Neural Networks

RNNs are inter-sequence models [29][30] with memory storage capabilities, and
are introduced into monocular depth estimation to learn temporal features from
video sequences. The RNN contains an input unit, a hidden unit, and three
parts: input unit, output unit, and hidden unit. The input of a hidden unit
consists of the outputs of both the current input unit and the previously
hidden unit. LSTMs, are a specific type of RNNs where feedback connections
allow these networks to learn temporal dependencies between data points. This
can be exploited by using video-based datasets where the advancing frames can
be fed through the LSTM networks and depth maps are extracted. Fig. 8 gives an
overview of the general pipelines involved in RNN-based depth prediction
models where one uses only LSTM (or ConvLSTM), the other uses convolution and
LSTM (or ConvLSTM) layers.

RNN and LSTM based networks are employed in the works of [31] where spatio-
temporal consistencies in the datasets are exploited using a Convolutional
LSTM, [32] where attention mechanism of ConvLSTM and ConvGRU are compared,
[33] and [34].

Figure 8: The two major architectures of RNN based networks for predicting
monocular depth maps. (a) shows the architecture containing only LSTMs in the
encoder while (b) uses convolutions in addition to the LSTMs. Source: [8]

###  IV-C Segmentation models for MDE

Drawing similarities from the pixel-level nature of segmentation in computer
vision, monocular depth estimation is a great fit for applying those models
for the task of depth estimation. Models for semantic segmentation [35]
specifically work well as they can divide the images into different “stuff”
that can have different spatial positions within the field of view and based
on that the depth maps can be extracted. The concept of knowledge distillation
[36] about using a large pre-trained “teacher” network to train a smaller
“student” network is used in addition to semantic segmentation models like
U-Net [37] and RefineNet [38] by the works of [39] and [40]. The work of [41]
uses a novel segmentation-based learning network to estimate depth in
monocular 360-degree videos.

Furthermore, the advancements in panoptic segmentation [42] have a significant
potential of making an impact in the field with its segregation of background
and instances as “stuff” and “things”, making it intuitive to estimate depth
in scenes. [43] and [44] make contributions towards MDE using panoptic
segmentation. Fig. 9 shows the model architecture used by [43] for aiding
depth estimation using panoptic segmentation.

Figure 9: Architectural diagram of our panoptic-boosted monocular depth
architecture at training (top) and test time (bottom). During training, our
network takes both views as input and produces the corresponding left and
right disparity maps in the decoder’s output. At test time, the network uses
only a single-view RGB image for prediction. Source: [43]

##  V Deep Learning Training Paradigms

###  V-A Supervised Learning

Supervised learning networks for monocular depth estimation are trained using
the Ground Truth depth maps. The purpose of learning is to penalize the error
between the prediction and the ground truth depth map on a loss function which
is based on the log depth [45] and the inverse Huber function (Berhu). The
goal is for the depth model to converge when the predicted depth value is as
close to GT as possible. The generalized pipeline for supervised learning
using depth maps as GT is summarized in Fig. 10.

Figure 10: The general architecture of supervised learning for MDE, with the
RGB and GT depth images as inputs and depth maps as outputs. Source: [8]

Most methods discussed in this paper so-far use a supervised learning approach
where the annotated datasets with the GT depth maps are used by a
CNN/LSTM/CRF-based learning network for depth estimation, often using stereo
pairs for increased accuracy.

DenseNet [46] and ResNet [47] are widely used backbones for CNN-based
monocular depth estimation solutions like [48] and [49].

Solutions integrating Conditional Random Fields (CRFs) into supervised
learning networks have also yielded state-of-the-art results such as the work
of [50] that divides input images into small windows to make the computation
of fully-connected CRFs feasible via the help of a multi-head attention
mechanism.

###  V-B Unsupervised Learning

High-resolution, publicly labeled datasets still require a high amount of
human resources and intensive work. Therefore, researchers are investigating
unsupervised deep learning methods for monocular depth estimation without the
use of GT depth maps. Unsupervised monocular depth estimation is typically
trained in pairwise stereo images or monocular image sequences and tested in
monocular images.

The two most common ways of training unsupervised MDE models are stereo
matching and using monocular sequences. A well-known implementation of the
former is [51] where a traditional belief propagation approach is used to
build a depth estimation system. [52] propose a CNN-based architecture where
the model learns depth maps of the right and left views. The work of MonoDepth
[53] with a 2D CNN architecture using an unsupervised learning approach and a
combination of disparity smoothness Loss, appearance matching loss, and left-
right disparity consistency loss resulted in a significant improvement upon
the then SOTA models on the KITTI dataset in 2017. 3D CNNs have also been used
by researchers [6][55][56][57] to take into account the temporal consistency
of stereo sequences in addition to the spatial part that is handled by regular
CNNs. Fig. 11 provides the architecture of the general implementation of a
stereo matching unsupervised MDE pipeline.

Figure 11: General pipeline for stereo matching based unsupervised learning
for depth prediction on monocular sources. Source: [8]

The other approach to training unsupervised learning-based networks is the use
of monocular sequences. This is especially attractive as a research topic due
to the higher availability and easier collection process of monocular depth
estimation datasets. It also avoids getting into the issues posed by stereo
matching relating to projection and left-right source mapping. [58] and the
subsequent related work of [59] propose methodologies for training
unsupervised learning networks on unstructured monocular video sequences along
with other elements such as SLAM and optical flow. Fig. 12 shows the
generalized pipeline for this approach.

Figure 12: General pipeline for monocular video sequence based unsupervised
learning for depth prediction. Source: [8]

###  V-C Semi-supervised Learning

To effectively use unlabeled datasets available easily in the public domain as
well as easy enough to capture by small teams, for improving learning
performance, semi-supervised learning approaches are used. These methods can
also use other sensor and depth information from sources such as synthetic
data, LIDAR, and surface normals, to reduce the model’s need for ground truth
depth maps, improving the depth map accuracy. The work of [60] introduces a
learning network that works on sparse data along with the RGB data in a
stereo-aligned geometric constraint manner. The model then generates two depth
maps from these input sources, for which loss is calculated separately with
the experiments showing an improvement with this model over a supervised one.
[61] uses a mutual distillation-based loss function in a semi-supervised
learning network setting, showing very good performance on KITTI and
Cityscapes datasets [62].

###  V-D Self-supervised Learning

Since the depth value in real-world applications is much larger than the value
these neural networks can consistently generate, a proper depth representation
will improve the performance considerably. Therefore, the appropriate choice
of depth representation to facilitate feature representation learning plays an
important role in depth learning and self-supervised monocular motion. SSL
(self-supervised learning) is a machine learning technique. It gets its
information from unlabeled sample data. It’s a kind of learning that’s halfway
between supervised and unsupervised.

There are two stages to learning. The job is first solved using pseudo-labels,
which aid in the initialization of network weights. Second, either supervised
or unsupervised learning is used to complete the assignment. In recent years,
self-supervised learning has yielded encouraging outcomes. The main benefit of
SSL is that it allows training to take place with lower-quality data rather
than focusing on improving outcomes. In the case of outdoor depth estimation,
the availability of diverse and long-range datasets makes it hard to train
generalizable deep learning models. Here self-supervised learning comes into
the picture by using the sparse annotated part of the dataset and generating
new data to train on. Fig. 13 visualizes this architecture of SSL based depth
estimation.

Figure 13: Predicting depth maps using self-supervised learning, a general
pipeline

The popular work of MonoDepth2 [63] uses a fully convolutional U-Net for depth
prediction along with a pose network to account for temporal consistency in
video frames. To consider occlusions they utilize per-pixel re-projection with
a specific loss function and then upsample the depth maps. SuperDepth [64]
proposes a super-resolution-based depth estimation solution along with a novel
augmentation layer that improves prediction accuracy. [65] introduces a novel
technique for SSL-based depth estimation approaches that bring about the then
state-of-the-art results by involving uncertainty modeling in this training
paradigm. [66] uses an SSL-based network for depth estimation that is used
along with LiDAR data for the task of depth completion in outdoor scenes.
Other works such as [67] are also pushing the envelope of what is possible
with the self-supervised learning framework by integrating popular model
architectures like transformers that do not require a specific configuration
of camera that has to be used for capturing the RGB input.

##  VI Challenges and Trends

In the last few years, the focus of monocular depth estimation has slightly
shifted towards large-scale outdoor and landscape-based data utilization which
aligns with the applications they are required for. In this section, we
discuss the limitations as well as future research scope that come with them.

###  VI-A Collection of long-range datasets

The applications of autonomous UAVs, robotics, and landscape-level 3D
reconstruction require very-long range datasets (captured from 100+ meters
far). Most existing research is done for indoor, short-range outdoor, and
moving vehicle outdoor datasets. The lack of long-range datasets makes it
difficult to extrapolate or adapt the available data too, thus research
relating to the collection of such datasets would be an interesting avenue.

Synthetic datasets created using virtual worlds in simulators are the easiest
way to approach this problem but care has to be taken in including real-world
elements into these datasets such as natural conditions of lighting and haze,
occlusion from dynamic objects, and perspective geometry that might be “off”
in virtual environments.

###  VI-B Integration with semantic segmentation

There are pieces of work that include segmentation into the overall depth
estimation system of supervised and self-supervised learning but these two are
treated as independent modules rather than a co-dependent framework. This
results in higher computational requirements for training these models and
creating/tuning them according to the use case. Research in the direction of
integrating these models is a promising area of work.

###  VI-C Real-time inference

The exploitation of temporal consistencies done by 3D CNNs, LSTMs, and other
attention mechanisms is still prone to the static nature of per-frame changes
in real-world applications that render the task of outdoor depth estimation as
a problem of a single image depth estimation even when video sequences are
involved. The existing work that balances this issue makes the tradeoff of
using deep networks that require a lot of computation resources and time.
Research in the area of utilizing lightweight networks similar to what has
been done for segmentation is essential to the real-world adoption of these
techniques.

##  VII Conclusion

Outdoor monocular depth estimation is an important step toward the full
realization of applications in robotics and simulation. This paper surveys the
publicly available datasets, and deep learning methods and summarizes the
training approaches used by existing models. Moreover, this paper discusses
the performance of popular approaches in different scenarios and the
limitations associated with them. In the end, we identify and list the current
challenges and related open research opportunities for the task of outdoor
depth estimation using monocular vision sources.

## References

  * [1] G. Eason, B. Noble, and I. N. Sneddon, “On certain integrals of Lipschitz-Hankel type involving products of Bessel functions,” Phil. Trans. Roy. Soc. London, vol. A247, pp. 529–551, April 1955.
  * [2] Liu, Yang, et al. ”A survey of depth estimation based on computer vision.” 2020 IEEE Fifth International Conference on Data Science in Cyberspace (DSC). IEEE, 2020.
  * [3] Chen, Hui, et al. ”3D reconstruction approach for outdoor scene based on multiple point cloud fusion.” Journal of the Indian Society of Remote Sensing 47.10 (2019): 1761-1772.
  * [4] Lan, Ziquan, Zi Jian Yew, and Gim Hee Lee. ”Robust point cloud based reconstruction of large-scale outdoor scenes.” Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2019.
  * [5] Tsai, Yi-Min, Yu-Lin Chang, and Liang-Gee Chen. ”Block-based vanishing line and vanishing point detection for 3D scene reconstruction.” 2006 international symposium on intelligent signal processing and communications. IEEE, 2006.
  * [6] Tang, Chang, Chunping Hou, and Zhanjie Song. ”Depth recovery and refinement from a single image using defocus cues.” Journal of Modern Optics 62.6 (2015): 441-448.
  * [7] Zhang, Ping, et al. ”Stereoscopic video saliency detection based on spatiotemporal correlation and depth confidence optimization.” Neurocomputing 377 (2020): 256-268.
  * [8] Ming, Yue, et al. ”Deep learning for monocular depth estimation: A review.” Neurocomputing 438 (2021): 14-33.
  * [9] Lopes, Alexandre, Roberto Souza, and Helio Pedrini. ”A Survey on RGB-D Datasets.” arXiv preprint arXiv:2201.05761 (2022).
  * [10] Geiger, Andreas, et al. ”Vision meets robotics: The kitti dataset.” The International Journal of Robotics Research 32.11 (2013): 1231-1237.
  * [11] Saxena, Ashutosh, Min Sun, and Andrew Y. Ng. ”Make3d: Learning 3d scene structure from a single still image.” IEEE transactions on pattern analysis and machine intelligence 31.5 (2008): 824-840.
  * [12] Ramezani, Milad, et al. ”The newer college dataset: Handheld lidar, inertial and vision with ground truth.” 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). IEEE, 2020.
  * [13] Li, Zhengqi, and Noah Snavely. ”Megadepth: Learning single-view depth prediction from internet photos.” Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2018.
  * [14] Vasiljevic, Igor, et al. ”Diode: A dense indoor and outdoor depth dataset.” arXiv preprint arXiv:1908.00463 (2019).
  * [15] Yang, Guorun, et al. ”Drivingstereo: A large-scale dataset for stereo matching in autonomous driving scenarios.” Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2019.
  * [16] Tateno, Keisuke, Nassir Navab, and Federico Tombari. ”Distortion-aware convolutional filters for dense prediction in panoramic images.” Proceedings of the European Conference on Computer Vision (ECCV). 2018.
  * [17] Yuan, Weihao, et al. ”NeW CRFs: Neural Window Fully-connected CRFs for Monocular Depth Estimation.” arXiv preprint arXiv:2203.01502 (2022).
  * [18] Zhou, Keyang, Kaiwei Wang, and Kailun Yang. ”PADENet: An efficient and robust panoramic monocular depth estimation network for outdoor scenes.” 2020 IEEE 23rd International Conference on Intelligent Transportation Systems (ITSC). IEEE, 2020.
  * [19] Zhuang, Chuanqing, et al. ”ACDNet: Adaptively Combined Dilated Convolution for Monocular Panorama Depth Estimation.” arXiv preprint arXiv:2112.14440 (2021).
  * [20] Chao, Fang-Yi, et al. ”A Multi-FoV Viewport-Based Visual Saliency Model Using Adaptive Weighting Losses for 360∘ Images.” IEEE Transactions on Multimedia 23 (2020): 1811-1826.
  * [21] Schops, Thomas, et al. ”A multi-view stereo benchmark with high-resolution images and multi-camera videos.” Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2017.
  * [22] Mancini, Michele, et al. ”Toward domain independence for learning-based monocular depth estimation.” IEEE Robotics and Automation Letters 2.3 (2017): 1778-1785.
  * [23] Reza, Md Alimoor, Jana Kosecka, and Philip David. ”FarSight: Long-Range Depth Estimation from Outdoor Images.” 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). IEEE, 2018.
  * [24] de La Garanderie, Greire Payen, Amir Atapour Abarghouei, and Toby P. Breckon. ”Eliminating the blind spot: Adapting 3d object detection and monocular depth estimation to 360 panoramic imagery.” Proceedings of the European Conference on Computer Vision (ECCV). 2018.
  * [25] Zhu, Jun-Yan, et al. ”Unpaired image-to-image translation using cycle-consistent adversarial networks.” Proceedings of the IEEE international conference on computer vision. 2017.
  * [26] Liu, Fayao, Chunhua Shen, and Guosheng Lin. ”Deep convolutional neural fields for depth estimation from a single image.” Proceedings of the IEEE conference on computer vision and pattern recognition. 2015.
  * [27] Bhat, Shariq Farooq, Ibraheem Alhashim, and Peter Wonka. ”Adabins: Depth estimation using adaptive bins.” Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2021.
  * [28] Garg, Ravi, et al. ”Unsupervised cnn for single view depth estimation: Geometry to the rescue.” European conference on computer vision. Springer, Cham, 2016.
  * [29] Ceni, Andrea, Peter Ashwin, and Lorenzo Livi. ”Interpreting recurrent neural networks behaviour via excitable network attractors.” Cognitive Computation 12.2 (2020): 330-356.
  * [30] Gregor, Karol, et al. ”Draw: A recurrent neural network for image generation.” International Conference on Machine Learning. PMLR, 2015.
  * [31] Zhang, Haokui, et al. ”Exploiting temporal consistency for real-time video depth estimation.” Proceedings of the IEEE/CVF International Conference on Computer Vision. 2019.
  * [32] Maslov, Dmitrii, and Ilya Makarov. ”Online supervised attention-based recurrent depth estimation from monocular video.” PeerJ Computer Science 6 (2020): e317.
  * [33] CS Kumar, Arun, Suchendra M. Bhandarkar, and Mukta Prasad. ”Depthnet: A recurrent neural network architecture for monocular depth prediction.” Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops. 2018.
  * [34] Wang, Rui, Stephen M. Pizer, and Jan-Michael Frahm. ”Recurrent neural network for (un-) supervised learning of monocular video visual odometry and depth.” Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2019.
  * [35] Long, Jonathan, Evan Shelhamer, and Trevor Darrell. ”Fully convolutional networks for semantic segmentation.” Proceedings of the IEEE conference on computer vision and pattern recognition. 2015.
  * [36] Hinton, Geoffrey, Oriol Vinyals, and Jeff Dean. ”Distilling the knowledge in a neural network.” arXiv preprint arXiv:1503.02531 2.7 (2015).
  * [37] Ronneberger, Olaf, Philipp Fischer, and Thomas Brox. ”U-net: Convolutional networks for biomedical image segmentation.” International Conference on Medical image computing and computer-assisted intervention. Springer, Cham, 2015.
  * [38] Lin, Guosheng, et al. ”Refinenet: Multi-path refinement networks for high-resolution semantic segmentation.” Proceedings of the IEEE conference on computer vision and pattern recognition. 2017.
  * [39] Cho, Jaehoon, et al. ”Deep monocular depth estimation leveraging a large-scale outdoor stereo dataset.” Expert Systems with Applications 178 (2021): 114877.
  * [40] Nekrasov, Vladimir, et al. ”Real-time joint semantic segmentation and depth estimation using asymmetric annotations.” 2019 International Conference on Robotics and Automation (ICRA). IEEE, 2019.
  * [41] Feng, Qi, Hubert PH Shum, and Shigeo Morishima. ”360 Depth Estimation in the Wild-the Depth360 Dataset and the SegFuse Network.” 2022 IEEE Conference on Virtual Reality and 3D User Interfaces (VR). IEEE, 2022.
  * [42] Yuan, Haobo, et al. ”PolyphonicFormer: Unified Query Learning for Depth-aware Video Panoptic Segmentation.” arXiv preprint arXiv:2112.02582 (2021).
  * [43] Saeedan, Faraz, and Stefan Roth. ”Boosting monocular depth with panoptic segmentation maps.” Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision. 2021.
  * [44] Qiao, Siyuan, et al. ”Vip-deeplab: Learning visual perception with depth-aware video panoptic segmentation.” Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2021.
  * [45] Eigen, David, and Rob Fergus. ”Predicting depth, surface normals and semantic labels with a common multi-scale convolutional architecture.” Proceedings of the IEEE international conference on computer vision. 2015.
  * [46] Huang, Gao, et al. ”Densely connected convolutional networks.” Proceedings of the IEEE conference on computer vision and pattern recognition. 2017.
  * [47] He, Kaiming, et al. ”Deep residual learning for image recognition.” Proceedings of the IEEE conference on computer vision and pattern recognition. 2016.
  * [48] Laina, Iro, et al. ”Deeper depth prediction with fully convolutional residual networks.” 2016 Fourth international conference on 3D vision (3DV). IEEE, 2016.
  * [49] Lee, Jin Han, et al. ”From big to small: Multi-scale local planar guidance for monocular depth estimation.” arXiv preprint arXiv:1907.10326 (2019).
  * [50] Zhang, Zhenyu, et al. ”Progressive hard-mining network for monocular depth estimation.” IEEE Transactions on Image Processing 27.8 (2018): 3691-3702.
  * [51] Sun, Jian, Nan-Ning Zheng, and Heung-Yeung Shum. ”Stereo matching using belief propagation.” IEEE Transactions on pattern analysis and machine intelligence 25.7 (2003): 787-800.
  * [52] Garg, Ravi, et al. ”Unsupervised cnn for single view depth estimation: Geometry to the rescue.” European conference on computer vision. Springer, Cham, 2016.
  * [53] Godard, Clément, Oisin Mac Aodha, and Gabriel J. Brostow. ”Unsupervised monocular depth estimation with left-right consistency.” Proceedings of the IEEE conference on computer vision and pattern recognition. 2017.
  * [54] Chen, Chuangrong, Xiaozhi Chen, and Hui Cheng. ”On the over-smoothing problem of cnn based disparity estimation.” Proceedings of the IEEE/CVF International Conference on Computer Vision. 2019.
  * [55] Tulyakov, Stepan, Anton Ivanov, and Francois Fleuret. ”Practical deep stereo (pds): Toward applications-friendly deep stereo matching.” Advances in neural information processing systems 31 (2018).
  * [56] Kendall, Alex, et al. ”End-to-end learning of geometry and context for deep stereo regression.” Proceedings of the IEEE international conference on computer vision. 2017.
  * [57] Chang, Jia-Ren, and Yong-Sheng Chen. ”Pyramid stereo matching network.” Proceedings of the IEEE conference on computer vision and pattern recognition. 2018.
  * [58] A. Moreau, M. Mancas and T. Dutoit, ”Unsupervised depth prediction from monocular sequences: Improving performances through instance segmentation,” 2020 17th Conference on Computer and Robot Vision (CRV), 2020
  * [59] Wang, Guangming, et al. ”Unsupervised learning of depth, optical flow and pose with occlusion from 3d geometry.” IEEE Transactions on Intelligent Transportation Systems 23.1 (2020): 308-320.
  * [60] Kuznietsov, Yevhen, Jorg Stuckler, and Bastian Leibe. ”Semi-supervised deep learning for monocular depth map prediction.” Proceedings of the IEEE conference on computer vision and pattern recognition. 2017.
  * [61] Baek, Jongbeom, Gyeongnyeon Kim, and Seungryong Kim. ”Semi-Supervised Learning with Mutual Distillation for Monocular Depth Estimation.” arXiv preprint arXiv:2203.09737 (2022).
  * [62] Cordts, Marius, et al. ”The cityscapes dataset for semantic urban scene understanding.” Proceedings of the IEEE conference on computer vision and pattern recognition. 2016.
  * [63] Godard, Clément, et al. ”Digging into self-supervised monocular depth estimation.” Proceedings of the IEEE/CVF International Conference on Computer Vision. 2019.
  * [64] Pillai, Sudeep, Rareş Ambruş, and Adrien Gaidon. ”Superdepth: Self-supervised, super-resolved monocular depth estimation.” 2019 International Conference on Robotics and Automation (ICRA). IEEE, 2019.
  * [65] Poggi, Matteo, et al. ”On the uncertainty of self-supervised monocular depth estimation.” Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2020.
  * [66] Ma, Fangchang, Guilherme Venturelli Cavalheiro, and Sertac Karaman. ”Self-supervised sparse-to-dense: Self-supervised depth completion from lidar and monocular camera.” 2019 International Conference on Robotics and Automation (ICRA). IEEE, 2019.
  * [67] Varma, Arnav, et al. ”Transformers in Self-Supervised Monocular Depth Estimation with Unknown Camera Intrinsics.” arXiv preprint arXiv:2202.03131 (2022).