# Fusion of Range and Stereo Data for
High-Resolution Scene-Modeling††thanks: This work has received funding from
Agence Nationale de la Recherche under the MIXCAM project number
ANR-13-BS02-0010-01.

Georgios D. Evangelidis, Miles Hansard, and Radu Horaud G. D. Evangelidis and
R. Horaud are with Perception Team, INRIA Grenoble Rhône-Alpes, 655, avenue de
l’Europe, 38330 Montbonnot Saint-Martin, France,
<EMAIL_ADDRESS><EMAIL_ADDRESS>Hansard is with
the Vision Group, School of Electronic Engineering and Computer Science, Queen
Mary, University of London, Mile End Road, London E1 4NS, UK,
<EMAIL_ADDRESS>
###### Abstract

This paper addresses the problem of range-stereo fusion, for the construction
of high-resolution depth maps. In particular, we combine low-resolution depth
data with high-resolution stereo data, in a maximum a posteriori (MAP)
formulation. Unlike existing schemes that build on MRF optimizers, we infer
the disparity map from a series of local energy minimization problems that are
solved hierarchically, by growing sparse initial disparities obtained from the
depth data. The accuracy of the method is not compromised, owing to three
properties of the data-term in the energy function. Firstly, it incorporates a
new correlation function that is capable of providing refined correlations and
disparities, via subpixel correction. Secondly, the correlation scores rely on
an adaptive cost aggregation step, based on the depth data. Thirdly, the
stereo and depth likelihoods are adaptively fused, based on the scene texture
and camera geometry. These properties lead to a more selective growing process
which, unlike previous seed-growing methods, avoids the tendency to propagate
incorrect disparities. The proposed method gives rise to an intrinsically
efficient algorithm, which runs at 3FPS on 2.0MP images on a standard desktop
computer. The strong performance of the new method is established both by
quantitative comparisons with state-of-the-art methods, and by qualitative
comparisons using real depth-stereo data-sets.

###### Index Terms:

Stereo, range data, time-of-flight camera, sensor fusion, maximum a
posteriori, seed-growing.

##  I Introduction

Figure 1:  The pipeline of the proposed depth-stereo fusion method. The low-
resolution (LR) depth data are projected onto the color data and refined to
yield a high-resolution (HR) sparse disparity map. Starting from these
disparity seeds, an upsampling process provides an initial HR dense disparity
map. Both the HR seeds and the initial dense disparity map are then used by
the region-growing depth-stereo fusion to produce the final HR depth map. A
prominent feature of our method is that fusion takes place at several data
processing stages.

Many computer vision methodologies, including dense 3D reconstruction [1, 2],
gesture recognition [3, 4], and object detection [5] have benefited from
recently-developed depth sensors. These sensors rely on active-light
principles, including modulated-light and pulsed-light cameras, commonly
denoted time-of-flight (TOF) [6, 7], or projected-pattern triangulation
cameras [8]. Regardless of the working principle, however, these sensors
provide low-resolution (LR) or mid-resolution depth maps that are inadequate
for a number of applications such as 3DTV and film production. For example,
many tasks in the film production industry can greatly benefit from a high-
resolution (HR) and high-quality depth map [9].

While HR depth maps can be obtained from multiple-view matching and
reconstruction using standard color cameras, it is well known that stereo
matching is problematic when the scene contains weakly textured areas,
repetitive patterns, or occlusions; these situations are very common in both
indoor and outdoor environments. Active-light sensors do not suffer from these
limitations, although their own depth data are quite noisy in the presence of
scattering, non-Lambertian materials, and slanted surfaces. The complementary
nature of HR stereo and LR depth sensors leads to the design of mixed camera
systems [10, 11, 12, 13, 14, 15, 16, 17, 18], which seem to be the most
promising approach, at present, for high-quality 3D depth maps.

In this context, this paper addresses the problem of HR 3D reconstruction from
the combination of a photometric camera pair and an active-light camera,
provided that the multiple-camera setup is calibrated [19]. The combination of
a stereo matching algorithm and of an active-light sensor raises the central
question of devising a matching algorithm with the following features: (i) it
considerably increases the resolution of the depth data, e.g., by a factor of
ten, (ii) it eliminates depth-sensor errors wherever possible, (iii) it
overcomes the limitations of stereo algorithms in textureless areas, and (iv)
it is able to compete with a depth sensor in terms of speed. Hence, the
availability of an efficient and robust stereo algorithm that takes advantage
of LR depth sensors and that provides dense and accurate HR depth maps,
possibly with subpixel accuracy, is particularly desirable.

To this end, we propose a 3D reconstruction method that merges depth-sensor
measurements with photo-consistency stereo matching. We address the problem
from the perspective of seed-growing, starting from a small number of control
points whose disparities are then propagated to yield a dense disparity map.
We show that this can be cast into maximum a posteriori (MAP) formulation
(Sec. III), which leads, in turn, to a series of local optimization problems
that are solved hierarchically by a novel region-growing process (Sec. V).
While the proposed method may not reach the global optimum, it allows us to
devise an intrinsically efficient methodology that bridges the gap between
global optimizers based on Markov random fields (MRF) and locally-optimal
winner-take-all (WTA) strategies (Sec V-E).

Efficient stereo-only or stereo-depth fusion methods rely on control points,
by exploiting either feature correspondences in stereo [20, 21] or depth data
in fusion [14, 15], and they assume that these points are of very good
quality. A key contribution of this paper is that this requirement is relaxed
in order to devise a method tolerant to bad control points. We propose to
truly combine LR depth-sensor data with HR rich photometric information,
whenever and wherever possible, showing that fusion is helpful, even in the
early stage of depth initialization (Sec. IV-A & IV-B). The data term of the
proposed MAP formulation benefits from a new cross-correlation function, which
provides real-valued disparities via subpixel corrections computed in closed-
form (Sec.V-B1), and takes advantage of a depth-guided cross-product
aggregation (Sec. V-B2). Moreover, the data-term adaptively merges the stereo-
and depth-consistency terms guided by the scene texture and the camera
geometry (Sec. V-C). These advantages lead to a more selective growing-of-
disparities process, thus preventing the algorithm from propagating erroneous
depth-sensor data – a phenomenon that is often associated with propagation
techniques – as experimentally verified (Sec. VI). It is important to note
that the proposed method can be used ‘as is’ with any depth sensor, as it
requires neither sensor-dependent confidence maps nor a sensor-dependent
model. The proposed fusion pipeline is illustrated in Fig. 1.

Supplementary materials, in particular image datasets and Matlab code are
available online.111https://team.inria.fr/perception/research/dsfusion/

##  II Related Work

We review pure stereo matching methods with emphasis on local algorithms,
owing to their computational suitability for HR images. We also review
upsampling methods and depth-stereo fusion methods. More detailed surveys of
stereo matching algorithms and depth-stereo fusion methods can be found in
[22] and [9], respectively.

###  II-A Pure Stereo Matching

Stereo matching methods can be broadly classified into global and local [22].
Global algorithms [23] typically adopt an MRF formulation and solve a single
optimization problem based on a MAP criterion. Despite their superiority over
local methods, global algorithms are extremely time-consuming, and hence
unattractive for fast fusing of depth-sensor data with high-resolution images.
Local algorithms solve per-pixel optimization problems and the state-of-the-
art methods build on adaptive cost aggregation [24, 25, 26, 27]. Most methods,
however, must visit the entire cost volume to find an optimal disparity value
at each pixel. This volume grows rapidly with respect to the input, as the
width of the image typically multiplies the number of disparities. Therefore,
although they are able to provide LR disparity maps in real-time, they remain
slow and subject to memory issues in HR stereo. Note that global algorithms
need several approximations to obtain LR disparity maps in real-time [28].

More interestingly, algorithms that rely on control-point correspondences [29]
are drastically more efficient since they avoid visiting the whole cost
volume. Region-growing approaches start from reliable but sparse
correspondences (seeds) and propagate them in textured areas [20]. [30]
suggests a similar propagation scheme where orientation-consistent disparities
are propagated to neighbors at the cost of finding a plane equation per pixel.
[21] proposed a generative model, where the prior disparity comes from a 2D
triangulated mesh whose vertices (control points) are obtained from matches
between low-level features. As with [20], textureless areas remain intractable
and the final map is reliable only when the matches are dense and uniformly
distributed over the images. Notice that the proposed method relies on the
idea of control points that are transferred from a depth sensor, thus avoiding
the limitation owing to untextured areas.

To obtain continuous disparities that are required in many scenarios, e.g., 3D
reconstruction [31], local stereo algorithms typically employ two strategies:
(i) fitting a curve around the correlation peak [22, 14] or (ii) integrating
an intensity interpolation kernel into the (dis)similarity function [32] whose
optimization leads to subpixel correction. The latter is also the case in the
fusion framework of [17, 16] that inherently takes advantage of inter-pixel
depth estimations.

###  II-B Depth Upsampling and Depth-Stereo Fusion

Any prior depth information, even at low resolution, is likely to help dense
disparity estimation. Apart from a naive interpolation, the bilateral filter
[33] can post-process an interpolated map using color HR images for guidance
[31]. Alternatively, a joint bilateral filter applies spatial and range
kernels to the LR depth map and HR color image respectively, so that
upsampling is a by-product of filtering [34]. Upsampling methods, however, are
limited to clean depth LR data and cannot reconstruct accurate HR maps when LR
data are delivered by depth sensors.

The above limitation of the upsampling methods gives rise to fusion approaches
that can merge depth and stereo data in either early or late stages, once the
sensors are calibrated. Late fusion suggests merging two depth maps, one
obtained with stereo and one from the range sensor, possibly upsampled [35,
18]. The majority of fusion methods, however, merge the stereo and range-
sensor data at an earlier processing level. [10] estimates a LR TOF-based
disparity map, which initializes a coarse-to-fine stereo scheme [36]. A semi-
global dynamic programming approach is followed in [14] with the TOF-based
disparities, wherever available, being considered as error-free matches. In a
global framework, [11] produces mid-resolution depth maps by merging TOF and
stereo data within a graph-cut stereo-matching paradigm; each energy term
exploits both modalities. Likewise, MRF-based formulations have been proposed
[37, 12, 13]. In [37], ground control points reflect an extra regularization
term in the MRF energy function. The work described in [12] uses an MRF scheme
to merge depth distributions of each sensor alone, but the goal is a LR depth
map. [13] extends [12] by means of weighted fusion. A balanced fusion (based
on several confidence maps) within a total variation model is also proposed in
[17]. A similar variational model that infers the HR map in a coarse-to-fine
manner is adopted by [16]. Note that, unlike most fusion methods and similar
to the proposed one, [16] is not tuned to a specific depth sensor.

More closely to the present work, [15] fuses the data within the seed-growing
method of [20]. In particular, TOF-based disparities constitute seeds while a
triangulation-based interpolated TOF map regularizes the seed-growing process.
When the TOF data are noisy, however, this approach tends to produce incorrect
disparities, and to propagate false positives during the growing process. The
proposed method differs considerably from [15] in terms of depth
initialization, cost function, and fusion strategy. The proposed initial map
is robust to depth discontinuities while it also guides the cost aggregation
inside a window. Moreover, our likelihood term integrates functions that are
capable of providing sub-pixel corrections [32]. This turns out to be very
beneficial, not only for the continuous nature of the final map, but also for
the growing process itself, thus propagating high-confidence disparities only.
Note that the subpixel disparity correction is obtained from a closed-form
solution – an interesting feature for efficiency. Finally, our algorithm
benefits from an adaptive fusion scheme that better balances the contribution
of each modality (depth or color), and that results in fewer unmatched pixels.

##  III Problem Formulation

The main mathematical notations that are used throughout the paper are
summarized in Table I.

TABLE I: The main mathematical notations used in the paper. $p$, $q$: |  Pixel locations of the high-resolution grid
---|---
$p$$\downarrow$, $q$$\downarrow$: |  Pixel locations of the sparse grid
$d_{p}$: |  Unknown disparity of pixel $p$, initialized by $d_{p}^{0}$
$d_{p\downarrow}$: |  Known disparity of pixel $p$$\downarrow$ (observed)
$D$: |  Unknown HR disparity map, initialized by $D^{0}$
$D$$\downarrow$: |  Known sparse version of $D$ (observed)
$\mathcal{D},\mathcal{D}^{0}$, $\mathcal{D}$$\downarrow$: |  Sets of all random variables (disparities) associated with $D$, $D^{0}$ and $D$$\downarrow$ respectively, with $d_{p}\in\mathcal{D}$, $d_{p}^{0}\in\mathcal{D}^{0}$ and $d_{p\downarrow}\in$$\mathcal{D}$$\downarrow$
$t_{p}$: |  Subpixel disparity correction of $p$
$\mu_{p}=(d_{p};t_{p})$: |  Disparity-correction pair referred here to as _meta-disparity_ with $|t_{p}|<1$
$\mathcal{M}=\\{\mu_{p}\\}$: |  Set of meta-disparities
$\mathcal{S}=\\{s_{p}\\}$: |  Set of observed stereo pixel intensities
$I_{p}$: |  Intensity of pixel $p$
$\mathbf{u}(x,y)$: |  a vectorized (zero-mean) form of an intensity window centered at the 2D position $(x,y)$
$\mathcal{N}_{p}$: |  Neighborhood of pixel $p$
med: |  $2$-d median operator
$I_{R}$, $D_{R}$: |  Intensity and disparity maps defined on a sub-region $R$
$E_{S}(d_{p})$, $E_{D}(d_{p})$: |  Stereo-based and depth-based energy of $p$ for given disparity
$g(\xi;\gamma)\\!=\\!e^{-|\xi|/\gamma}$: |  Exponential mapping of $\xi$ with scale $\gamma$

As discussed, the direct upsampling of LR depth data suffers from limitations,
in particular when the upsampling factor is high.222In our experiments, the
upsampling factor is $10\times$ in each dimension, that is $100\times$ in the
number of pixels, e.g., from 0.02Mp to 2MP. Therefore, our goal is to build
$D$ by jointly taking advantage of both sensing modalities. Given a proper
calibration, (e.g., [38, 19]), the mapping of the LR depth image onto the
rectified HR color images will typically yield a _sparse_ disparity map
$D$$\downarrow$, which is almost evenly distributed across the HR grid. Note
that in [21, 20], the initial matches between the stereo images do not
correspond to a uniform sparse version of $D$, as they are unpredictably
distributed, due to the reliability of properly detecting interest points in
images. Instead, the sparse map obtained with a depth sensor can be used to
guide a stereo algorithm, regardless of the presence or absence of scene
texture.

We propose to model the estimation of $\mathcal{D}$, and therefore the map
$D$, as a _maximum a posteriori_ (MAP) problem, based on the available depth
and stereo observations. However, instead of immediately using
$\mathcal{D}$$\downarrow$, we first estimate a dense initial map $D^{0}$ and
its associated set $\mathcal{D}^{0}$. Then, we obtain the final disparity map
by solving the following optimization problem:

$\mathcal{D}^{*}=\arg\max_{\mathcal{D}}P(\mathcal{D}|\mathcal{S},\mathcal{D}^{0}),$
(1)

where $P(\mathcal{D}|\mathcal{S},\mathcal{D}^{0})$ is the posterior
distribution of disparities given the observations $\mathcal{S}$ and
$\mathcal{D}^{0}$. The proposed depth initialization method is described in
Sec. IV and the proposed solution to the MAP formulation (1) is described in
Sec. V. Note that a reliable estimation of $D^{0}$ is quite important, since
it guides several components of the fusion methodology.

|  |
---|---|---
(a) | (b) | (c)
Figure 2: (_Best viewed on screen_) (a) The mapping of depth data onto the
left image causes artifacts in the presence of depth discontinuities. A
cascade of (b) geometry-consistency and (c) color-consistency filters refines
the sparse disparity map. Depth values are color-coded from red (close) to
blue (far).

##  IV Depth Initialization

A two-step approach is proposed in order to obtain the initial disparity map
$D^{0}$. First, we refine $D$$\downarrow$ to deal with mapping errors. Second,
we upsample the refined sparse map in a novel way using color information to
obtain the initial dense map $D^{0}$. As shown below, this leads to an
initialization robust to depth discontinuities, which in turn helps the
growing.

###  IV-A Sparse-Depth Refinement

|  |
---|---|---
(a) | (b) | (c)
Figure 3: (_Best viewed on screen_) Depth upsampling results using (a)
triangulation-based interpolation [15] after cutting big triangles, (b) joint
bilateral filter [34] and (c) our method. The depth values are color-coded
from red (close) to blue (far), while black areas correspond to non-available
values. The white edges in close-ups show the color edges of the image.

We assume a camera setup with a depth camera mounted in between the two color
cameras; other depth-stereo setups are discussed in [9]. Regardless of the
depth sensor technology and type, the projection of the depth map onto the
left and right images implies a parallax effect, and hence occlusions.
Moreover, this causes gaps as well as areas with overlapping depth data close
to depth boundaries [15]. In the case of TOF cameras, these areas are further
contaminated from _jump-edge errors_ [39], or ‘flying pixels’ [40], while a
structured-light camera, e.g., Kinect, leaves more gaps due to the offset
between the position of the light projector and the position of the infrared
sensor. Fig. 2(a) illustrates the artifacts that we briefly discussed: flying
pixels and depth-data overlap in the top and bottom closeups, respectively. In
order to eliminate these artifacts we apply a geometry-consistency cascade of
two filters: the first one removes isolated pixels (mostly flying pixels) and
the second one keeps the foremost pixel inside a window to compensate for the
above mentioned overlap. An example of applying this filtering is shown in
Fig. 2(b). In practice this does not fully refine the sparse depth map. We
still observe mismatches near depth discontinuities, because of depth bias and
calibration errors. Therefore, a second filter that imposes color consistency
is applied, as described below.

We consider a window centered at $p$$\downarrow$ and split into four equally
sized sub-windows $W_{i}$, $i=\\{1\ldots 4\\}$, such that their intersection
is only the pixel $p$$\downarrow$ (see Fig. 4). The output, $d_{p\downarrow}$,
of the filter is:

$d_{p\downarrow}=\text{med}(D\\!\\!\downarrow_{W_{i^{*}}})$ (2)

with

$i^{*}=\arg\min_{i}\bigl{(}|I_{p\downarrow}-\text{med}(I_{W_{i}})|\bigr{)}.$
(3)

The output $d_{p\downarrow}$ is the median disparity of the adjacent sub-
window whose median intensity is closest to that at $p$$\downarrow$. For color
images, the term $|I_{p\downarrow}-\text{med}(I_{W_{i}})|$ can be replaced by
the average deviation from the median, over the color channels. This filter
leads to a further refinement near depth discontinuities which are
pathological areas for stereo algorithms. The result of this kind of filtering
is shown in Fig. 2(c). Note that both refinement filters apply to sparse
locations only so that their complexity is negligible.

Figure 4: The window split for the color-consistency filter. The pixel $p$ is
linked with the closest (shaded) sub-window in terms of the color consistency
(links represent color distances from $I_{p}$ to $\text{med}(I_{W_{i}})$).

###  IV-B Sparse-Depth Upsampling

While a naive upsampling of the sparse disparity map could be performed, e.g.,
[15], thereby producing an initial map, strong depth discontinuities are
likely to contaminate such an interpolation. Alternatively, cross-bilateral
filtering [41] or joint bilateral upsampling [34] may upsample the map using
the HR color image as a guide. The filter support of these methods jointly
counts on spatial and range (color) kernels. However, both methods apply a
linear smoothing once the filter support per pixel is computed. Instead, we
propose a non-linear upsampling strategy that builds on the median filter.

Specifically, the depth (or disparity) at a dense pixel location $p$ is
initialized by

$d^{0}_{p}=\text{med}(D\\!\\!\downarrow_{\mathcal{N}_{p}^{c}})$ (4)

where $\mathcal{N}_{p}^{c}$ is a constrained neighborhood of $p$, that is
$\mathcal{N}_{p}^{c}\subset\mathcal{N}_{p}$, which contains only sparse depth
measurements whose color is consistent with $I_{p}$:

$\mathcal{N}_{p}^{c}=\Bigl{\\{}q\\!\downarrow\,\,\big{|}\,\,g(I_{p}-I_{q\\!\downarrow};\gamma_{c})>\epsilon_{c}\Bigr{\\}}$
(5)

with $g(\xi;\gamma)$ being an exponential mapping (see Table I).

Unlike common bilateral filters, our upsampling process makes a more
definitive selection of pixels, thus preserving the depth discontinuities of
the scene, while filtering out some of the noise in the depth data. Once the
$\mathcal{N}_{p}^{c}$ is defined, one can optionally consider a spatial kernel
and compute a weighted average instead, in order to better deal with slanted
surfaces. Fig. 3 compares our initialization with the upsampling results
obtained by [15] and [34]; in the detailed views, the intensity edges are also
shown. The proposed method provides more discriminative depth boundaries that
coincide with intensity edges. Missing values may be observed in highly
textured areas, since $\mathcal{N}_{p}^{c}$ may be an empty set. In this
example, the radius of $\mathcal{N}_{p}$ is $20$, $\gamma_{c}=10$, and
$\epsilon_{c}=0.2$. The same radius is used for the method of [34] while the
scales for spatial and color kernel are $10$ and $20$ pixels, respectively.
The geometry-consistency filter reasonably applies in all cases while our
method benefits from our color-consistency refinement as well.

Since the median operator is chosen to account for outliers within a window,
the mean operator can be invoked instead when the depth variance almost
vanishes (homogeneous areas), thus drastically reducing the computational
burden of the upsampling process. Since the vast majority of pixels belong to
such areas, the complexity of our filter approaches that of joint upsampling
filter [34]. Note that the latter has been extended in [18] by integrating
color segmentation results.

##  V Depth-Stereo Fusion

Let $d_{p}\in\mathcal{D}$ have $N$ possible discrete states; the goal is to
estimate the disparity (state) of each HR image location through the MAP
formulation (1). Once $\mathcal{D}^{0}$ has been initialized, one can assume
that $\mathcal{S}$ and $\mathcal{D}^{0}$ are conditionally independent, so
that the posterior distribution of (1) can be decomposed as

$P(\mathcal{D}|\mathcal{S},\mathcal{D}^{0})\propto
P(\mathcal{S}|\mathcal{D})\,P(\mathcal{D}^{0}|\mathcal{D})\,P(\mathcal{D}).$
(6)

As mentioned, global solutions are prohibitively expensive for high-resolution
disparity maps. Therefore, we focus on approximate solutions that allow for
the decomposition of the global optimization problem into many local (per-
pixel) optimization problems.

The proposed method is based on the seeded region-growing framework [42, 20],
where the known message of a location (the parent) is propagated to its
neighbor (the child). This implies that the estimation of $d_{p}$ is also
conditioned by a parent _known_ disparity $d_{p}^{pa}$, hence dealing with the
principal graph of Fig. 5 for every pixel with unknown disparity (the visiting
order of pixels is made clear later). As a result, if $P(d_{p})$ is considered
uniform, the posterior probability of $d_{p}$ can be written as333Strictly
speaking, it is an approximation since $d_{p}^{pa}$ and $d_{p}^{0}$ may not be
fully independent.

$\displaystyle P(d_{p}|s_{p},d_{p}^{0},d^{pa}_{p})$ $\displaystyle=$
$\displaystyle P(d_{p}|d^{pa}_{p})P(d_{p}|s_{p},d_{p}^{0})$ (7)
$\displaystyle\propto$ $\displaystyle
P(d_{p}|d^{pa}_{p})P(s_{p}|d_{p})P(d_{p}^{0}|d_{p})$

where $d^{pa}_{p}$ is the parent of $d_{p}$ and the probability

$P(d_{p}|d^{pa}_{p})=\left\\{\begin{array}[]{l
l}\frac{1}{2r+1}&\quad|d^{pa}_{p}-d_{p}|\leq r\\\\[2.15277pt]
0&\quad\text{otherwise}\\\ \end{array}\right.$ (8)

has a uniform distribution. In other words, the disparity range for each node
is a function of the assigned disparity of his parent. We consider a narrow
support area, i.e., a constant low value for $r$, e.g., $1$ or $2$; node-
dependent parametrization of $r$ is left to future work.

Figure 5: The principal graph that is iteratively considered in our region-
growing fusion method.

###  V-A MAP as Energy Minimization

As is customary, likelihoods are chosen from the exponential family which
leads to an energy minimization framework. Our model assumes an energy-
dependent distribution (Boltzmann) for $P(s_{p}|d_{p})$ and a Laplacian one
for $P(d_{p}^{0}|d_{p})$:

$\displaystyle P(s_{p}|d_{p})$ $\displaystyle\propto$
$\displaystyle\exp\bigl{(}-E_{S}(d_{p})\big{/}\lambda_{S}\bigr{)}$ (9)
$\displaystyle P(d_{p}^{0}|d_{p})$ $\displaystyle\propto$
$\displaystyle\exp\bigl{(}-|d_{p}-d^{0}_{p}|\big{/}\lambda_{D}\bigr{)}.$ (10)

Based on (7-10), Fig. 6 shows an example with the distribution of $d_{p}$
being constrained by single or joint observations. Because of the exponential
terms, the pixel-wise maximization of the posterior distribution reduces to
the minimization of the _local_ energy

$E(d_{p})=E_{S}(d_{p})+E_{D}(d_{p})$ (11)

where $E_{D}(d_{p})=\lambda|d_{p}-d^{0}_{p}|$ is the regularization term,
$\lambda=\lambda_{S}/\lambda_{D}$ and $E_{S}(d_{p})$ is the (stereo) data-term
which is defined below. The term $E_{D}(d_{p})$ guides the inference in
textureless areas, while it penalizes mismatches due to depth discontinuities
when the latter are well preserved in $D^{0}$. Notice that a smoothness
constraint is implicitly enforced because of the prior term
$P(d_{p}|d^{pa}_{p})$ owing to the low value of $r$. In other words, the
support area of $E(d_{p})$ is truncated as shown in Fig. 6, and the pixel-wise
optimization problem becomes

$\displaystyle\min_{d_{p}}$ $\displaystyle E(d_{p})$ (12) subject to
$\displaystyle|d_{p}-d^{pa}_{p}|\leq r.$

The order of visiting pixels and solving (12) obeys a most-confident first-
solved rule, as discussed in Sec V-D. Below, once the data-term is defined, we
modify (11) to adaptively combine the data-term with the regularizer.

Figure 6: An example of the probability distribution of $d_{p}$ constrained by
different observations.

###  V-B The Data Term

The proposed data-term benefits from two properties: its ability to provide
energy for subpixel disparities and to adaptively consider pixel-wise
similarities within a window. Apart from the advantage of a locally continuous
map [31], it is more important to note that these properties lead to a more
selective growing in areas with texture (owing to the former) and varying
depth (owing to the latter) because of more accurate and reliable local
energies.

####  V-B1 Subpixel disparity

To obtain energy at interpixel locations, we make use of meta-disparities
$(d_{p};t_{p})$ (see Table I). The estimation of $t_{p}$ builds on [32], which
has been shown to be superior to the parabola-fitting around the peak
correlation: the pixel $p=(x,y)$ of the left image is matched with the
subpixel $p^{\prime}=(x+d_{p}+t_{p},y)$ of the right image by defining a
bounded correlation function $C_{d_{p}}(t_{p})$ ($|C_{d_{p}}(t_{p})|<1$) which
is maximized with respect to $t_{p}$ for a given $d_{p}$. This allows us to
define our data-term as

$E_{S}(d_{p};t_{p}^{*})=1-C_{d_{p}}(t_{p}^{*}),$ (13)

where

$t_{p}^{*}=\arg\max_{t}\,C_{d_{p}}(t).$ (14)

Consequently, the total energy (11) is parameterized by $t_{p}^{*}$ as well.
Note that if $t_{p}^{*}$ can be estimated via an analytic solution, the extra
computational cost of subpixel disparity estimation is negligible. It becomes
clear that, since $C_{d_{p}}(t_{p}^{*})\geq C_{d_{p}}(0)$ in (13), more
accurate energy values are estimated and more reliable messages are
propagated.

Since invariance to photometric distortions is important in stereo matching,
we propose to adopt the normalized correlation coefficient, and one of its
variants [43], for $C_{d_{p}}(t_{p})$. While the former has been already
extended to deal with subpixel disparities [32], the latter has never been
extended before. Both use zero-mean vectorized forms of the windows around $p$
and $p^{\prime}$, let $\mathbf{u}_{L}(x,y)$ and $\mathbf{u}_{R}(x+d+t,y)$,
with the latter being written via a first-order Taylor approximation as
$\mathbf{u}_{R}(x+d+t,y)\simeq\mathbf{u}_{R}(x+d,y)+t\Delta\mathbf{u}_{R}(x+d,y)$
where $\Delta$ is a difference operator along the $x-$axis. This is also the
case in variational framework [17, 16] where subpixel accuracy is obtained by
the early interpolation of the intensity, rather than the late interpolation
of correlation around the peak [22].

#### Enhanced Correlation Coefficient (ECC)

The ECC function [32] results from the integration of the above linear
interpolation kernel into Pearson coefficient:

$C_{d}^{P}(t)=\frac{\mathbf{u}_{L}^{\top}(\mathbf{u}_{R}+t\Delta\mathbf{u}_{R})}{\|\mathbf{u}_{L}\|~{}\|\mathbf{u}_{R}+t\Delta\mathbf{u}_{R}\|}.$
(15)

If the denominator of (15) is non-degenerate, then $C_{d}^{P}(t)$ is a quasi-
concave function of $t$ and its maximization results in a closed-form solution
[32].

#### Enhanced Moravec Correlation Coefficient (EMCC)

The Moravec coefficient [43] replaces the denominator of (15) with the mean of
the variances. This also allows us to introduce a left-right symmetry, thereby
estimating a left disparity $-t/2$ and a right disparity $t/2$ instead of $t$
(such a modification with ECC leads to a complex optimization problem). The
enhanced Moravec correlation coefficient (EMCC) is defined by

$C_{d}^{M}(t)=\frac{2(\mathbf{i}_{L}-t/2\Delta\mathbf{i}_{L})^{\top}(\mathbf{i}_{R}+t/2\Delta\mathbf{i}_{R})}{\|\mathbf{i}_{L}-t/2\Delta\mathbf{i}_{L}\|^{2}+\|\mathbf{i}_{R}+t/2\Delta\mathbf{i}_{R}\|^{2}}.$
(16)

Note that one can easily show that the integration of an interpolation kernel
into the cost function of [15] is equivalent with the EMCC scheme. Although
(16) is a rational function of $t$, the next proposition guarantees that the
maximizer has an analytic form. We refer the reader to the appendix for the
proof and the exact maximizer.

Proposition I: A rational function of two second-degree polynomials, as in
(16), attains at most one global maximum if the denominator is non-degenerate;
its maximizer is given by a closed-form solution.

Note that, the estimation of $t$ could be unreliable for strictly homogeneous
areas. The value of the window variance or entropy is a good criterion to
assess the reliability of subpixel correction, and to enable it accordingly.

####  V-B2 Adaptive similarity aggregation

The best-performing local stereo algorithms benefit from an adaptive cost
aggregation strategy [24]. This strategy is based on the assumption that depth
discontinuities are most likely to coincide with color discontinuities, so
that each pixel within a window contributes differently to the (dis)similarity
cost based on its spatial and color distance from the central pixel. However,
only a few color edges correspond to depth edges and the above assumption
should be followed only in the absence of any prior information about the
depth. Since in our scenario the prior depth information is available, the
spatial and color consistency can be replaced by a depth consistency term.

To be specific, we adopt here the exponential $g(\cdot)$ (see Table I) to
compute pixel-wise weights $w_{q}$:

$w_{q}=g\bigl{(}d^{0}_{p}-d^{0}_{q};\gamma_{d}\bigr{)},$ (17)

with $q\in\mathcal{N}_{p}$. The weights apply element-wise to
$\mathbf{u}_{L}$, $\mathbf{u}_{R}$, $\Delta\mathbf{u}_{R}$ and
$\Delta\mathbf{u}_{L}$ in (15) and (16). In other words, we compute the
subpixel correction and the optimum local energy after down-weighting pixels
in the window that belong to another surface compared to the one of the
central pixel. It becomes clear now that not only the term $E_{D}(d_{p})$ but
also the stereo term $E_{S}(d_{p})$ in (11) benefit from an upsampling method
that is robust to depth discontinuities. Note that, even if the initial depth
map is biased, it is sufficient enough to guide the aggregation step within
the window.

###  V-C Adaptive Fusion

While a constant fusion may be reasonable for specific types of scenes (e.g.
highly-textured scenes), an adaptive balance of the terms in (11) is usually
preferred. This implies that the less we count on the data term
$E_{S}(d_{p})$, the more we should count on the regularization term
$E_{D}(d_{p})$ during the inference.444Here we omit the disparity correction
$t$. This suggests a convex combination when the scene point of $p$ is viewed
by all cameras.

A summary of methods that perform weighted fusion is discussed in [17].
However, most of them consider TOF-based weights for the regularizer (e.g.,
[13]) which contradicts our goal of a sensor independent fusion. Moreover,
directly using the confidence map of a TOF image is not a good strategy [40].
Therefore, we only rely on stereo data to obtain the mixing coefficients. It
is well known in stereo or optical flow that the matching of a point is
reliable when its associated image patch contains sufficient texture [44, 45].
Since a good indicator for the texture presence is the image entropy [44], an
entropy filter provides us with am adequate reliability factor $e_{p}$ for
each window centered at $p$.

Let us now consider the left image as reference and compute the initial left-
to-right disparities. Likewise, we can build a right-to-left disparity map
based on the right image, and a cross-checking of these maps can provide an
estimation of the major occlusions due to strong depth discontinuities, with
respect to the reference image. We refer to these areas as stereo-occlusions
and we denote them as $\Omega_{SO}$. Recall now that some points in the left
image are not seen by the depth camera, and that this gives rise to gaps in
the initial disparity map which can be easily detected. We refer to these
areas as depth-occlusions and we denote them as $\Omega_{DO}$. It becomes
obvious that the evaluation of $E_{S}(d_{p})$ and $E_{D}(d_{p})$, in
$\Omega_{SO}$ and $\Omega_{DO}$ respectively, should be avoided. Hence, we
propose the following adaptive fusion

$E(d_{p})=\eta_{p}^{S}E_{S}(d_{p})+\eta_{p}^{D}E_{D}(d_{p})$ (18)

with the pair $(\eta_{p}^{S},\eta_{p}^{D})$ being defined as

$(\eta_{p}^{S},\eta_{p}^{D})\\!=\\!\begin{cases}(0,1)&\text{if}~{}p\in\Omega_{SO}\setminus\Omega_{DO}\\\
(1,0)&\text{if}~{}p\in\Omega_{DO}\setminus\Omega_{SO}\\\
(e_{p},1\\!-\\!e_{p})&\text{if}~{}p\in\Omega\setminus\Omega_{SO}\bigcup\Omega_{DO}\\\
(inf,inf)&\text{if}~{}p\in\Omega_{SO}\bigcap\Omega_{DO}\end{cases}$ (19)

where $\Omega$ defines the whole image area and $e_{p}$ is the normalized
output of the entropy filter. We intentionally add the last case in (19) which
shows that the fusion in $\Omega_{SO}\bigcap\Omega_{DO}$ is meaningless and a
post-filling method should be followed.

(a) (b) (c)

Figure 7: (a) The validity of ordering constraint implies
$\Omega_{DO}\subset\Omega_{SO}$ (depth camera is mounted between the color
cameras); the shorter the baseline is, the more coincident these regions are.
(b) $\Omega_{DO}$ and $\Omega_{SO}$ do not overlap when the scene contains
very thin foreground objects and the ordering constraint is invalid. In (c),
an estimation of $\Omega_{SO}$, $\Omega_{DO}$ for the example of Fig 2 is
shown.

It is important to note that the so-called _ordering constraint_ in stereo is
valid when large foreground objects appear in the scene, while it is violated
when very thin objects are close to the camera (see Fig. 7). The former
implies $\Omega_{DO}\bigcap\Omega_{SO}=\Omega_{DO}$ and the latter implies
$\Omega_{SO}\bigcap\Omega_{DO}=\emptyset$. While the area
$\Omega_{SO}\setminus\Omega_{DO}$ can always be predetected, the area
$\Omega_{DO}\setminus\Omega_{SO}$ is safely predetected only when the ordering
constraint is not valid.555It would be possible to detect all areas if one
knows a priori that the constraint is _everywhere_ valid or invalid. This is
because $\Omega_{SO}$ is detected from the cross-checking of disparity maps
that already suffer from depth-occlusions, since they are computed from the
depth-to-stereo mapping. Ideally, if the complexity is not an issue, a stereo-
occlusion detection scheme (e.g. [46]) based on stereo image pair could be
enabled beforehand. Therefore, we prefer to not grow the area of depth-
occlusions. Optionally, these areas can be filled in a post-processing step.
Fig. 7(c) shows the areas for the example of Fig 2. Points that have been
removed during the refinement step (Sec. IV-A) are also marked as depth-
occlusions, while outliers in the initial maps can produce false positives for
$\Omega_{SO}$, e.g. the red points on the right arm of the person. Our region-
growing method, however, is able to compensate for such errors.

###  V-D The region-growing algorithm

As has been explained, our method solves pixel-based optimization problems in
a region-growing manner, based on the seeds contained in
$\mathcal{D}$$\downarrow$. Since the initial disparities may be noisy and
biased, they do not reflect true matches as opposed to [15]. This means that
we exploit $d_{p\downarrow}$ to restrict the disparity range of its neighbor,
but once a disparity value is assigned to the latter, $p$$\downarrow$ is reset
to a pixel $p$ with unknown disparity. For our convenience, the set
$\mathcal{D}$$\downarrow$ is augmented to an initial set $\mathcal{M}$ of
meta-disparities $(d_{p};0)$ with the same cardinality. We also denote with
$\mathcal{N}(\mu_{p})=\\{\mu_{p}^{j}\\}_{j=1}^{4}$ the image-based neighbors
of $\mu_{p}$, that is, $\mu_{p}^{j}=(d_{\hat{p}};t_{\hat{p}})$ with $\hat{p}$
being any of the four immediate neighbors of $p$. Note that $\mu_{p}^{j}$ does
not necessarily belong to $\cal M$.

Algorithm 1 describes the growing process. The algorithm starts by sorting the
elements of $\cal M$ based on their energy value, while it initializes to
false the visit- and assignment-flag of all candidate meta-disparities. Next,
it considers the lowest-energy $\mu_{p}$ with false visit-flag and switches
this flag to true. Then, it assigns values at _each_
$\mu_{p}^{j}\in\mathcal{N}(\mu_{p})$ with false assignment-flag based on the
following minimization scheme

$(d^{*}_{\hat{p}};t^{*}_{\hat{p}})=\underset{t_{\hat{p}},\,|d_{\hat{p}}-d^{*}_{p}|\leq
r}{\arg\min}E(d_{\hat{p}};t_{\hat{p}}),$ (20)

where

$E(d_{\hat{p}};t_{\hat{p}})=\eta_{\hat{p}}^{S}E_{S}(d_{\hat{p}};t_{\hat{p}})+\eta_{\hat{p}}^{D}E_{D}(d_{\hat{p}})$
(21)

and $d^{*}_{p}$ is the optimum disparity of the parent node of $\hat{p}$. Note
that (21) extends (11) by adding the subpixel disparity parameter into the
stereo term, and making the fusion pixel-dependent. In other words, what we do
for each neighbor $\hat{p}$ with false assignment-flag is the following. For
each candidate integer value $d_{\hat{p}}\in[d_{p}^{*}-r,d_{p}^{*}+r]$ the
optimum $t_{\hat{p}}^{*}$ that minimizes the term
$E_{S}(d_{\hat{p}};t_{\hat{p}})$ is obtained based on (16) or (15) and the
local energy $E(d_{p};t_{p}^{*})$ is computed from (21). Among the $2r+1$
values, the disparity minimizer $d_{\hat{p}}^{*}$ is finally chosen and the
corresponding subpixel correction is assigned, as it is shown in (20). Recall
that $r$ has a low value in contrast to conventional stereo algorithms where
it equals the whole disparity range. If
$E(d_{\hat{p}}^{*};t_{\hat{p}}^{*})<T$, then the meta-disparity
$(d_{\hat{p}}^{*};t_{\hat{p}}^{*})$ is pushed into $\mathcal{M}$ with respect
to the sorting, and its assignment-flag becomes true. The above process is
repeated with unvisited meta-disparities until the cardinality of $\cal M$
remains fixed. The visiting order depends on local energies, since the lowest-
energy meta-disparity is always picked from the stack. The final set $\cal M$
corresponds to the final dense disparity map while a post-filling method can
deal with missing disparities; their number heavily depends on threshold $T$.
It is important to note that the algorithm cannot get stuck in a loop, because
it propagates disparities in a tree structure, and a true visit-flag can never
be reset to false.

Algorithm 1 Stereo-Depth Fusion

0: Image-pair $I_{L},I_{R}$, set $\mathcal{D}\\!\downarrow$, Threshold $T$.

1: Transform $\mathcal{D}\\!\downarrow$ into a set $\mathcal{M}$ of meta-
disparities with $t_{p}\\!\\!=\\!0$.

2: Compute the initial disparity map $D^{0}$.

3: Sort $\cal M$’s elements based on their energy $E(\mu)$.

4: Set both the visit and assignment flags $f_{v}(\mu)$, $f_{a}(\mu)$ to
false, for all candidate meta-disparities, including $\mathcal{M}$.

5: repeat

6: Consider $\mu_{p}$ with minimum energy and false $f_{v}(\mu_{p})$

7: Set $f_{v}(\mu_{p})=\text{true}$

8: for all $\mu_{p}^{j}\in\mathcal{N}(\mu_{p})$ with
$f_{a}(\mu_{p}^{j})=\text{false}$ do

9: $\mu_{p}^{j*}=\arg\min E(\mu_{p}^{j})~{}$ (Eq. 20)

10: if $E(\mu_{p}^{j*})<T$ then

11: Set $f_{a}(\mu_{p}^{j*})=\text{true}$

12: Push $\mu_{p}^{j*}$ in $\cal M$ w.r.t. sorting

13: end if

14: end for

15: until $\text{card}(\cal M)$ is fixed

16: Compute the dense disparity map $D$ from $\cal M$

17: return $D$.

###  V-E Comparison with Other Inference Formulations

State-of-the-art stereo or fusion methods adopt an MRF model that provides a
straightforward way to integrate multiple sensor data. If we recall equation
(6), the term $P(D)$ can be written as a Gibbs distribution whose energy is a
sum of potential functions over maximal cliques [47], hence a sum of pairwise
potentials when a 4-pixel neighborhood is considered. This in turn offers a
smoothness term in the global energy equation that leads to piecewise smooth
disparity map. There exist exact solutions for such models, under very
specific conditions [23]. Their computational complexity, however, becomes
more and more severe as the number of states increases, thus becoming
prohibitively expensive in the case of HR-stereo. Therefore, one has to focus
on feasible approximate inferences that decompose the global optimization
problem into a series of local optimization problems.

Figure 8: Dependency network for (a) WTA and (b) MRF models in an 1D example.
Initial and two final candidate graphs for the proposed scheme are shown in
(c), (d) and (e) respectively.

MRF stereo generalizes the winner-take-all (WTA) approach [22] which can be
seen as the simplest inference. Fig. 8(a) and (b) show an 1D-grid example of
the MRF and WTA networks. In essence, links between disparity nodes in WTA
disappear and each node is connected only to input data (observations). This
implies a uniform prior distribution $P(\mathcal{D})$ and the maximization of
$P(\mathcal{D}|\mathcal{S},\mathcal{D}^{0})$ reduces to a set of independent
pixel-wise maximization problems of type $P(d_{p}|s_{p},d_{p}^{0})$. As with
the proposed method, we tacitly assume, that ${s}_{p}$ are not intensities but
they represent the stereo data of local windows centered at $p$ and its
candidate correspondence. This is necessary in WTA since the likelihood cannot
count on single pixels only, while it can optionally used in global MRF
solutions as well. It is worth noticing that our algorithm switches to a WTA
solution when $r=N$, since the visiting order of nodes and their connectivity
becomes meaningless and the uniform distribution $P(d_{p}^{pa}|d_{p})$ is not
truncated anymore.

Another MRF approximation that finds pixel-wise optimizers without breaking
the connectivity between nodes is the iterated conditional modes (ICM) method
[48]. After a proper initialization of all disparity nodes, ICM-like schemes
visit one node (pixel) at a time and assign the disparity value that maximally
contributes to the global posterior distribution. On the contrary, our
approach requires the initialization of few nodes only, at least one in
principle. Moreover, the graph in our case is a set of directed _trees_ ,
i.e., a forest, so that each non-root node has only one parent. It is
important to note, however, that it is the output of our algorithm that
defines the final network since nodes may be re-ordered, links may be
cancelled and arrows may be reversed during the inference.

To be more specific, let us consider the graph of Fig. 8(c) and let us assume
the clamping of $d_{2}$ and $d_{5}$ to the observation $d_{2}^{0}$ and
$d_{5}^{0}$. Starting from $d_{2}$, we can ‘propagate’ $d_{2}^{0}$ to its
neighbor $d_{3}$ (and $d_{1}$), i.e., to look for the optimum value of $d_{3}$
but being strictly conditioned by the value $d_{2}^{0}$. Note that ICM would
look for the best assignment of $d_{3}$ value by taking into account both
initial values $d_{2}^{0}$ and $d_{4}^{0}$. Moreover, ICM would search, in
principle, among all (here $N$) states for the optimum $d_{3}$’s assignment,
while our scheme looks only around $d_{2}^{0}$, namely $d_{3}$ takes values in
$\\{d_{2}^{0}-r,...,d_{2}^{0}+r\\}$, with $r$ being a small integer. Once
$d_{3}$’s assignment is done, $d_{2}^{0}$ can be passed to $d_{1}$ in a
similar way. Next, another node is visited, here one of $d_{3}$ and $d_{5}$,
and its disparity is propagated to its neighbors. As a result, the principal
graph of Fig. 5 is iteratively considered. Note that ICM assigns a new value
to $d_{3}$ anyway, while our scheme invokes a criterion that validates the
assignment. If $d_{3}$’s assignment is not valid, our algorithm will possibly
assign a new disparity value to $d_{3}$ only after $d_{4}$’s assignment. Fig
8(d) and (e) show possible final graphs obtained by different realizations of
our algorithm. In the example of Fig 8(e), the initial disparity of $d_{5}$
was cancelled by the validation process, thus all nodes were filled due to
$d_{2}$.

It is now clear that, as opposed to WTA solution, the final inference we
obtain depends on the visiting order of nodes. The visiting order of the ICM
scheme is either fixed in advance (e.g. raster-scanning), or random. Inspired
by [47] and [20], however, we instead adopt a highest confidence first (HCF)
scheme that suggests visiting the nodes based on their local evidence
(energy). This means that we keep the nodes sorted with respect to their
energy, and we visit each time the least-energy node that has not been visited
yet. For instance, in Fig. 8(d), starting with $d_{2}$ implies that $d_{2}$ is
more confident than $d_{5}$. The above validation process relies on
thresholding the local energy as explained in Sec. V-D. Table II summarizes
some properties of MRF-based solutions (Graph-cut [49] and ICM [48]), WTA, and
our approach.

TABLE II: Properties of inference algorithms in stereo and/or depth-stereo
fusion.

| MRF (graph cuts) | MRF (ICM) | WTA | Proposed
---|---|---|---|---
Dependency network | undirected graph (MRF) | undirected graph (MRF) | independent minor graphs | independent directed trees (forest)
Inference | exact* | approximate | approximate | approximate
Prior distribution | Gibbs | Gibbs | uniform | truncated uniform
Invariance to visiting order | yes* | no | yes | no
Disparity search range | full ($r=N$) | full ($r=N$) | full ($r=N$) | narrow ($r\ll N$)
Complexity in HR stereo/fusion | too high | high | high | low

*under specific conditions [49, 23]

##  VI Experiments

In this section, we evaluate our algorithm, and quantitatively compare it to
the state-of-the-art, based on both simulated and real data-sets. We also test
our algorithm and provide qualitative comparison on a new and challenging
dataset.

###  VI-A Simulated Data

We use the Middlebury database, and focus on a challenging data-set which
contains $1.5$MP images ($1300\times 1100$) along with ground-truth maps (GTM)
[50]. To simulate an LR disparity map from another viewpoint, we proceed as
follows. Given the calibration parameters, we transform the ground-truth
disparities into 3D points, as viewed from the midpoint of the baseline. We
then apply a 3D rotation to the point-set and we downsample the points by a
factor of $10$. The rotation is such that the average disparity bias is more
than $2$ pixels. Finally, we back-translate the points into sparse biased
disparities and we add colored noise, that is, a 2D mid-frequency sinusoidal
signal with peak-to-peak distance equal to $2\sigma$, where $\sigma$ denotes
the noise deviation. Note that [34, 31, 15] only downsample the GTMs in their
experimental setup. Algorithm performance is quantified in terms of the so-
called _bad matching pixels_ (BMP) percentage in the non-occluded areas [22],
i.e., $(1/N_{o})\sum_{p}(|D_{p}-G_{p}|>\delta)$ where $G$ is the GTM and
$N_{o}$ is the number of the non-occluded pixels. While the threshold level
$\delta=1$ is mostly used for mid-resolution images, HR stereo justifies the
value $\delta=2$ as well [21]; a value $\delta<1$ is chosen when subpixel
accuracy is to be evaluated.

The same parameter settings are used for our method, in all of the
experiments. The radius of the upsampling filter is $20$ and the values
$\gamma_{c}$ and $e_{c}$ are $10$ and $0.2$ respectively. Because of the
propagation strategy, we choose a relatively small window, i.e., $9\times 9$.
The local energy in (11) and the weights in (17) are obtained with
$\lambda=0.01$ and $\gamma_{d}=5$, respectively. We enable subpixel correction
when the normalized entropy in the (left) window is above $0.4$. As mentioned,
we use a fixed (and strict) search range around the disparity parent, that is
$r=1$, which leads to the most efficient solution. As for the threshold, we
set $T=0.5$. Note that the energy validation threshold implies a tradeoff
between accuracy and density. We recommend setting a middle threshold value
and post-filling sparse missing disparities, e.g., with the upsampling filter,
rather than using a high threshold that incorporates erroneous disparities in
a fully dense map. The density obtained with this strategy is about $90\%$ in
HR images. As with all algorithms, large remaining gaps are filled with a
streak-based filling [21]. We refer to our methods as Fusion-ECC (F-ECC) and
Fusion-EMCC (F-EMCC).

Before comparing with the state-of-the-art, we show the performance gain in
terms of the new modules that are integrated compared to EPC method [15], thus
quantifying the contribution of the new data-term and the adaptive fusion.
Note that [15] (EPC) follows a seed-growing approach by using a quadratic
model for both stereo and depth consistency terms respectively. To better
evaluate the contribution of the enhanced correlation coefficients presented
in the data-terms, we use LR images ($450\times 375$ on an average) whose GTM
contains subpixel disparities. The noise deviation in the sparse map is
$\sigma=2$. We intentionally do not fill any remaining large gaps, in order to
assess the net contribution of each module, and we compute the error for the
filled area only ($85\%$ density). Table III shows the BMP error averaged over
eight images. We also evaluate our approach when none of the modules are
enabled, i.e. pure correlation is used along with a fixed fusion of the terms;
we refer to this method as _simple fusion_. All of the variants start from the
same initial depth map, obtained by our upsampling method. As can be seen,
both the data-term and the adaptive fusion process contribute to a better
reconstruction, compared to simple fusion. The use of the proposed energy
data-term leads to more accurate results, while the adaptive fusion eliminates
large errors (see the error with $\delta=2$). Even the simple fusion has a
lower BMP than EPC, owing to the different stereo- and depth-consistency terms
(we use a linear model for the latter). The two proposed criteria behave
similarly, with the F-ECC being slightly more accurate, since it achieves
higher correlation values (see also Table IV). Note that the results are
systematically worse if we use the initialization of [15].

TABLE III: Contribution of various modules of the proposed algorithm (BMP error averaged over eight LR images). | BMP ($\%$) for $\delta=0.5$ / $\delta=1$ / $\delta=2$
---|---
| F-ECC | F-EMCC | EPC [15]
Simple Fusion |  $22.4$ / $7.7$ / $3.4$ |  $23.3$ / $7.9$ / $3.5$ | $33.0$ / $11.6$ / $3.7$
Data-term |  $16.8$ / $6.4$ / $3.2$ |  $17.2$ / $6.7$ / $3.3$
Data-term+ Adap. fusion |  $14.6$ / $5.8$ / $2.4$ |  $15.0$ / $6.2$ / $2.7$

To compare with the state-of-the-art, we implemented the upsampling methods of
[34] and [31] (two-view version), as well as the MRF-based fusion [12] by
using the MRF-stereo toolbox of [23], referred here to as F-MRF. While [12]
uses belief propagation, we experimentally found that Graph-Cuts [49] perform
better. Specifically, we tried ten different parameter settings and we found
that the best performing algorithm is the expansion mode with Birchfield-
Tomasi cost [51] truncated at $7$, linear disparity differences truncated at
$5$ and quadratic cost for the smoothness-energy; the weights for the stereo-,
depth- and smoothness terms were $1$, $1.2$ and $10$. We refer here to [12]
instead of [13] for MRF-based fusion since the latter relies on a TOF-based
reliability fusion which cannot be implemented here.

Fig. 9 plots the BMP curves of the upsampling and fusion algorithms as a
function of the noise deviation for the challenging low-texture HR image
_Lampshade1_. We just add noise here in the down-sampled GTMs. Except for
[34], all schemes start from the same HR map, obtained by a _naive
interpolation_ , while its BMP curve is plotted as well. As can be seen, F-MRF
and EPC are more affected by the noisy prior disparity, in contrast to the
proposed algorithm, which is less sensitive to initialization. It is clear
that the pure up-sampling methods provide acceptable results only when the
initial LR disparity map is very accurate.

Figure 9: Left image with superimposed depth map (left) and BMP curves (right) for the HR stereo pair _Lampshade1_. TABLE IV: BMP for _high-resolution_ disparity maps of the Middlebury dataset with $\delta=1$. |  | Lampshade1 | Art | Monopoly | Rocks2 | Reindeer | Bowling2 | Baby1 | Moebius | Average
---|---|---|---|---|---|---|---|---|---|---
Stereo |  GCS [20] | 25.1 | 20.0 | 52.4 | 4.5 | 14.1 | 25.4 | 18.1 | 19.5 | 22.4
ELAS [21] | 15.2 | 12.2 | 38.1 | 2.6 | 5.6 | 14.6 | 10.4 | 14.0 | 14.1
GraphCuts [52] | 32.3 | 26.3 | 62.1 | 10.8 | 28.8 | 43.1 | 15.6 | 19.5 | 29.8
CSBP [28] | 39.4 | 26.6 | 61.8 | 8.3 | 20.3 | 31.9 | 16.4 | 20.0 | 28.1
FastAgg [25] | 26.8 | 10.8 | 44.6 | 2.0 | 8.1 | 15.6 | 11.5 | 14.9 | 16.8
NonLocalAgg [27] | 25.0 | 23.5 | 32.6 | 10.7 | 24.3 | 28.8 | 18.0 | 17.2 | 22.5
Upsamping |  Yang _et al._ [31] | 46.9 | 47.4 | 48.5 | 41.8 | 42.9 | 46.3 | 36.9 | 41.9 | 44.1
Kopf _et al._ [34] | 43.7 | 46.5 | 45.4 | 39.4 | 40.9 | 42.1 | 35.1 | 41.8 | 41.8
Fusion |  EPC [15] | 20.7 | 17.8 | 20.7 | 4.2 | 11.8 | 20.8 | 11.9 | 12.0 | 14.9
F-MRF (GC) [12] | 16.3 | 17.1 | 25.7 | 4.5 | 12.1 | 20.0 | 10.1 | 10.5 | 14.5
F-MRF (ICM) | 23.1 | 27.9 | 43.4 | 14.7 | 20.1 | 24.8 | 17.9 | 20.1 | 24.0
F-ECC | 8.4 | 14.0 | 7.6 | 2.9 | 8.7 | 8.5 | 4.4 | 9.0 | 7.9
F-EMCC | 8.2 | 14.9 | 7.5 | 3.1 | 9.0 | 8.5 | 4.2 | 10.8 | 8.2

We now proceed with a detailed comparison including efficient and well-known
stereo algorithms as well. Specifically, we include four recently proposed
methods, [21] (ELAS), [20] (CGS), [25] (FastAgg), [27] (NonLocalAgg) and two
MRF-based stereo algorithms, Graph Cuts (GC) [52] and constant-space belief
propagation (CSBP) [28]. The top-performing local algorithms, FastAgg and
NonLocalAgg, build cost volumes that depend on both the image size and the
disparity range. This leads to a huge memory footprint ($\sim 3$GB) in the
case of HR images, and the authors’ implementations could not be run as is. In
order to be able to run FastAgg, the cost volume has been split into slices
and cached on disk. The NonLocalAgg was run using the maximum allowed
resolution while the disparity map produced by the algorithm was finally
upsampled. Note that both methods invoke a left-right consistency checking,
combine color and gradient information and enable refinement steps. Authors’
implementations for ELAS, GCS, GraphCut, CSBP (local-minima
version$+$bilateral post-processing) were used in the comparisons, with the
default settings suggested by the authors. We also implemented the ICM
algorithm for the MRF-based fusion using the same parameters with GC. For a
fair comparison, all fusion schemes merge stereo data with the _same_ initial
disparity map, which is obtained here by our upsampling process. Due to the
simulated experimental setup, however, the error of the initial map obtained
from this process is very close to that of Kopf _et al._ ’s method [34] (the
average difference of their BMP error is below $0.5$) and is thus omitted.

Table IV provides the BMP error for eight HR images with error threshold
$\delta=1$, while the corresponding table for $\delta=2$ is given in the
appendix. Bold and underlined numbers mark the lowest and second lowest errors
per column. Weakly textured scenes (_Lampshade1_ , _Monopoly_) seem to be
problematic for conventional stereo algorithms, while cylindrical surfaces
(_Bowling2_ , _Baby1_) present another challenge. It is not surprising,
however, that stereo methods outperform fusion methods in highly textured
images (e.g. _Rocks2_), or in images with many thin objects (e.g. _Art_),
since the sparse noisy initial map negatively affects the fusion. ELAS and
FastAgg behave better than other stereo algorithms. Similar results would be
expected from the NonLocalAgg method, if we were able to run it at full
resolution.

Stereoscopic and depth data are better fused in general by the proposed
criteria than the other fusion methods. EPC and up-sampling methods verify the
sensitiveness to their initialization, with the former being more effective
due to the fusion process. As far as the MRF solution is concerned, the
benefit due to the depth data is verified from the results, i.e. the F-MRF
(GC) scheme behaves better than pure-stereo GC. Moreover, GC in fusion
provides better results than the simple ICM algorithm. Unlike the proposed
criteria, F-MRF deals better with thin objects, since it does not aggregate
costs in a window, hence the lower error in _Art_ and _Reindeer_. Recall that
[12] refines the TOF depth map based on stereo data, without increasing the
resolution. By putting aside the high complexity, it seems that MRF-based
solutions need to be reformulated for HR stereo-depth fusion, e.g. high-order
connectivity might be more helpful, semi-global solutions could be
investigated, and conditional random fields [50] might need to be extended to
the fusion framework.

Between the two proposed criteria in our fusion scheme, ECC and EMCC, it is
the image content that makes one outperform the other. While F-ECC may be
slightly better on average, F-EMCC deals better with images of very low
texture, e.g. _Monopoly_ , which supports Moravec’s argument for introducing
MCC [43]. Moreover, F-EMCC is more affected by the filling, as it provides
less dense maps than F-ECC, provided that the threshold is the same.

Fig. 10 shows the average performance of all fusion competitors as a function
of the error threshold $\delta$, while the stereo baseline of FastAgg is added
for reference. In essence, this figure reflects the distribution of errors.
Evidently, the contribution of the LR depth prior in the fusion schemes is
verified, as opposed to the stereo baseline whose performance is bounded. The
proposed schemes are more accurate compared to the fusion baselines. However,
F-MRF provides lower errors when the tolerance is not that strict ($\delta>2$)
owing to its global smoothness constraint. Note that the performance of [34]
approaches the performance of the fusion schemes as $\delta$ is increasing.
The other upsampling method of [31] seems to produce large errors.

Figure 10: The BMP error of fusion algorithms averaged over the eight test images as a function of the error tolerance $\delta$. TABLE V: Average execution times of algorithms for the HR Middlebury data-set [50] in a $2.6$GHz machine. | Time(sec) | Matlab | C/C++ | GPU | SSE
---|---|---|---|---|---
GCS [20] | $1.2$ | $\checkmark$ | $\checkmark$ |  |
ELAS [21] | $1.0$ |  | $\checkmark$ |  | $\checkmark$
GC [52] | $>10^{3}$ |  | $\checkmark$ |  |
CSBP [28] | $15$ |  | $\checkmark$ |  |
FastAgg [25] | $>10^{3}$ | $\checkmark$ |  |  |
NonLocalAgg [27] | $5.0$ |  | $\checkmark$ |  |
Yang et al. [31] | $>10^{2}$ | $\checkmark$ |  |  |
Kopf et al [34] | $88$ | $\checkmark$ |  |  |
EPC [15] | $1.5$ | $\checkmark$ | $\checkmark$ |  |
F-MRF (GC) [12] | $>10^{3}$ |  | $\checkmark$ |  |
F-MRF (ICM) [12] | $>10^{2}$ |  | $\checkmark$ |  |
Our upsampling | $95$ | $\checkmark$ |  |  |
F-ECC, F-EMCC | $2.0$ | $\checkmark$ | $\checkmark$ |  |
Upsampling of [15] | $<0.1*$ |  | $\checkmark$ | $\checkmark$ |
F-ECC, F-EMCC | $0.3*$ |  | $\checkmark$ | $\checkmark$ | $\checkmark$
* time needed for the 2MP images of our real data-set (see Fig.13)

Computational efficiency is an important feature of any depth-stereo fusion
method. Table V shows the execution times of the algorithms for the simulated
data, as well as some of their implementation details. A combined Matlab-C
version of our fusion algorithm requires $2.0$s per image triplet while it
takes $0.3$s (for 2MP images) when GPU hardware is used for some
initializations, and the SSE instruction set accounts for the online
computation of correlation values between windows. We also developed a GPU-
based implementation of a triangulation-based upsampling (interpolation) that
takes less than $50$ms. This allows one to envisage real-time execution of the
proposed depth-stereo fusion framework, despite the high resolution. Note that
it is the Matlab implementation that makes FastAgg and upsampling schemes
slow, while the time for the NonLocalAgg method is based on matching at half
resolution. We also point out that CSBP attains a solution in a reasonable
time, despite its MRF-based formulation.

###  VI-B Real Data

TABLE VI: Evaluation on the dataset of Dal Mutto  _et al._ [18] | MSE of disparity estimation
---|---
| Scene A | Scene B | Scene C | Average
Stereo [25] | $97.52$ | $5.78$ | $93.94$ | $65.74$
Initialization | $9.33$ | $6.34$ | $5.62$ | $7.09$
Our upsampling | $9.96$ | $6.54$ | $\underline{5.52}$ | $7.34$
Kopf _et al._ [34] | $10.23$ | $7.45$ | $5.68$ | $7.78$
Dal Mutto _et al._ [18] | $\bf{3.76}$ | $6.56$ | $8.69$ | $\underline{6.34}$
EPC [15] | $8.54$ | $6.61$ | $5.72$ | $6.95$
F-MRF [12] | $8.96$ | $\underline{4.67}$ | $6.18$ | $6.67$
F-ECC | $\underline{6.98}$ | $\bf{4.19}$ | $\bf{5.39}$ | $\bf{5.52}$

Dal Mutto  _et al._ [18] provide real TOF-stereo data along with ground truth
disparities, shown in Fig. 12. To be consistent with [18], we upsample the
depth in a similar way, using a bilateral filter, which benefits from color
segmentation. This procedure is used to initialize the depth in _all_ fusion
schemes. Table VI shows the mean square error (MSE) of the disparity
estimation for several algorithms. The contribution of the stereo data in
fusion is unquestionable in scenes A and B, where the depth varies locally.
All of the fusion methods obtain a more accurate map than the initial one.
However, scene C contains only planar objects, and the upsampling methods
provide good results. Although the proposed scheme does not perform best in
all examples, it always improves the initial estimate, which demonstrates the
advantage of adaptive fusion (very similar numbers are obtained with F-EMCC).
As expected, the use of stereo data only (e.g., [25]) performs well only with
the textured scene B. Our upsamping filter is more accurate than [34] and less
accurate than the filter proposed by [18]. Note that our implementation of
[34] achieves better results than those reported in [18].

We also assess the performance of the fusion methods on the HCIbox data-set
[17]. The scene shows the interior of a box that contains some objects (Fig.
11). Note that there is no texture, apart from some horizontal lines on the
stairs and the ramp, hence stereo methods tend to fail. We follow the
experimental setup of [17], thus evaluating the depth estimation based on some
statistics of the absolute error, after excluding inter-reflection areas (see
Fig. 11). We do not include the results of [17], since the authors provided a
different inter-reflection mask with larger support area than the one used in
[17].666Personal communication with R. Nair. Table VII shows the error
statistics of the algorithms. All of the fusion methods start from the same
initial map, obtained by our upsampling method. The proposed fusion method
achieves the lowest mean and median error (similar results are obtained with
F-EMCC). The variance of F-MRF is increased (a local bias was observed owing
to the global smoothness), while its median remains low. Because of the depth
discontinuities, [34] yields a less accurate result compared to our
upsampling.

Figure 11: The left image(left) and the mask that excludes inter-reflections
(right) of the HCIbox dataset [17].

(A) (B) (C)

Figure 12: The three scenes (cropped) of the dataset used in [18]. TABLE VII: Evaluation on HCIBox dataset  _et al._ [17] | Mean | St. D. |  $1^{st}$ Quart. | Median |  $3^{rd}$ Quart.
---|---|---|---|---|---
Kopf _et al._ [34] | $3.23$ | $\bf{4.00}$ | $0.93$ | $2.15$ | $3.85$
Initialization | $3.00$ | $\underline{4.21}$ | $0.84$ | $1.94$ | $3.61$
EPC [15] | $3.01$ | $4.32$ | $0.90$ | $1.93$ | $\underline{3.17}$
F-MRF [12] | $\underline{2.95}$ | $4.96$ | $\underline{0.83}$ | $\underline{1.85}$ | $\underline{3.17}$
F-ECC | $\bf{2.53}$ | $4.25$ | $\bf{0.62}$ | $\bf{1.38}$ | $\bf{2.64}$

Figure 13: HR images and disparity maps for MIXCAM dataset obtained by (from
left to right) ELAS, FastAgg, F-MRF (GC), F-ECC, and F-ECC after post-
processing.

We also captured our own challenging TOF-stereo data-set using a synchronized
camera setup, developed in collaboration with 4D View
Solutions777http://www.4dviews.com. Two HR ($1624\times 1224$) color cameras
and a MESA Imaging SR4000 TOF (range) camera ($176\times 144$) are mounted on
a rail. The stereo baseline is $50$cm approximately while the TOF camera is
mounted in the middle. The algorithm of [19] allows us to transform the depth-
sensor measurements into a very sparse stereo disparity map ($1\%$ sparsity),
with an average error of $1.0$ pixel. The sparse map is refined and upsampled
as discussed in Sec. IV (see also Fig.3). For a fully real-time upsampling, we
use our GPU-based implementation of the triangulation-based interpolation
[15]. Note that our stereo rectification puts the principal points in the same
position, rather than making the optical axes parallel; this maximizes the
overlap between the images, given the relatively wide baseline.

Our ‘MIXCAM’ data-set contains challenging cases, e.g., periodicities, weakly
textured areas, thin objects, depth discontinuities, and so on. The fusion
algorithm merges the stereo and depth data and the outcome is a dense HR
disparity map. We reuse our upsampling filter in a post-processing step to
fill missing disparities. A streak-based method fills any remaining gaps in
all algorithms. The results are shown in Fig. 13. White areas denote unmatched
pixels, while black areas mark the detected TOF-occlusions. The left column
shows the left image, with the TOF image shown in the bottom-right corner at
the true scale. Next columns show the results of ELAS, FastAgg, F-MRF (GC) and
F-ECC algorithms; the last column show the disparity maps of F-ECC after post-
processing.

ELAS fills local areas, surrounded by textured points, through an
interpolation scheme. We intentionally show the results of ELAS before the
streak-based filling; as opposed to FastAgg, where missing disparities after
the left-right consistency check are filled. Clearly, a pure stereo algorithm
cannot deal with large untextured areas, and the post-filling is unreliable.
F-MRF provides fully dense results. Note that we run F-MRF with half-
resolution images ($812\times 612$), due to its tremendous memory
requirements. Moreover, we set a fixed value along the disparity range for the
data-term of all TOF-occlusion points, so that the global inference becomes
independent of this area. F-MRF provides artifacts in stereo occlusions, that
are next to TOF-occlusions when the scene contains large foreground objects.
As with [13], the results of F-MRF scheme verify the lack of an adaptive
fusion of the depth- and stereo-consistency data terms, as opposed to our
methods. However, F-MRF seems to deal better with very thin objects (e.g. the
branch of the plant), as already discussed above. Note that the biased range
measurements of very slanted surfaces (e.g. the table-top) negatively affect
the fusion schemes, in particular when the table surface lacks texture (e.g.
first example). The proposed scheme provides very good results on average,
especially after the post-processing step, which fills the gaps and refines
the disparities. We obtain very similar results with the F-EMCC method, while
EPC provides results _visually_ close to ours, but with more gaps. The
bilateral upsampling of [34] provides visually good results, but with blurred
depth discontinuities (see also Fig. 3).

##  VII Conclusions

We have presented a high-resolution stereo matching algorithm that is guided
by low-resolution depth data, thus helping the algorithm to compensate for its
difficulty in estimating disparities over weakly textured areas. We cast the
problem into a MAP formulation whose inference is obtained through a series of
local optimization problems, solved hierarchically in a seed-growing manner.
The latter characteristic yields an intrinsically efficient solution that
allows for near real-time matching of 2.0MP images. The data-term of the
energy function benefits from a correlation function that is capable of
providing scores at subpixel disparities, from an adaptive cost aggregation
step inside the window based on the depth data, and from an adaptive fusion of
stereo- and depth-consistency terms based on the scene texture and the camera
geometry. These properties lead to a more selective growing process that
prevents the algorithm from propagating incorrect disparities. As a result, a
low-complexity method builds an accurate high-resolution disparity map. A
quantitative comparison against pure stereo and stereo-depth fusion
algorithms, as well as a qualitative assessment on real data, has validated
the strong performance of the proposed method. Future research will include
the optimum visiting order for seeds in the growing framework, as well as an
adaptive window size, based on the local surface orientation.

## References

  * [1] P. Henry, M. Krainin, E. Herbst, X. Ren, and D. Fox, “RGB-D mapping: Using kinect-style depth cameras for dense 3D modeling of indoor environments,” _IJRR_ , vol. 31, no. 5, 2012.
  * [2] Y. M. Kim et al., “Multi-view image and tof sensor fusion for dense 3D reconstruction,” in _ICCV Workshops (3DIM)_ , 2009.
  * [3] M. D. Bergh and L. V. Gool, “Combining RGB and ToF cameras for real-time 3D hand gesture interaction,” in _WACV_ , 2011.
  * [4] G. D. Evangelidis, G. Singh, and R. Horaud, “Continuous gesture recognition from articulated poses,” in _ECCV Workshops_ , 2014.
  * [5] J. Stückler and S. Behnke, “Combining depth and color cues for scale- and viewpoint-invariant object segmentation and recognition using random forests,” in _IROS_ , 2010.
  * [6] M. Hansard, S. Lee, O. Choi, and R. Horaud, _Time-of-flight cameras: principles, methods and applications_. Springer, 2012.
  * [7] F. Remondino and D. Stoppa, Eds., _TOF Range-Imaging Cameras_. Springer, 2013.
  * [8] J. Geng, “Structured-light 3D surface imaging: a tutorial,” _Advances in Optics and Photonics_ , vol. 3, no. 2, pp. 128–160, 2011.
  * [9] R. Nair et al., “A survey on time-of-flight stereo fusion,” in _Time-of-Flight and Depth Imaging. Sensors, Algorithms, and Applications_ , M. Grzegorzek et al., Ed. Springer, 2013, vol. 8200, pp. 105–127.
  * [10] S. A. Gudmundsson, H. Aanaes, and R. Larsen, “Fusion of stereo vision and time-of-flight imaging for improved 3D estimation,” _Int. J. Intell. Syst. Technol. Appl._ , vol. 5, no. 3/4, 2008.
  * [11] U. Hahne and M. Alexa, “Combining time-of-flight depth and stereo images without accurate extrinsic calibration.” _IJISTA_ , vol. 5, no. 3/4, pp. 325–333, 2008.
  * [12] J. Zhu, L. Wang, R. Yang, and J. Davis, “Fusion of time-of-flight depth and stereo for high accuracy depth maps,” in _CVPR_ , 2008.
  * [13] J. Zhu, L. Wang, R. Yang, J. Davis, and Z. Pan, “Reliability fusion of time-of-flight depth and stereo geometry for high quality depth maps,” _IEEE TPAMI_ , vol. 33, no. 7, pp. 1400–1414, 2011.
  * [14] J. Fischer, G. Arbeiter, and A. Verl, “Combination of time-of-flight depth and stereo using semiglobal optimization,” in _ICRA_ , 2011.
  * [15] V. Gandhi, J. Cech, and R. Horaud, “High-resolution depth maps based on tof-stereo fusion,” in _ICRA_ , 2012.
  * [16] K. Ruhl, F. Klose, C. Lipski, and M. Magnor, “Integrating approximate depth data into dense image correspondence estimation,” in _CVMP_ , 2012.
  * [17] R. Nair, F. Lenzen, S. Meister, H. Schäfer, C. Garbe, and D. Kondermann, “High accuracy tof and stereo sensor fusion at interactive rates,” in _ECCV Workshops_ , 2012.
  * [18] C. Dal Mutto, P. Zanuttigh, S. Mattoccia, and G. Cortelazzo, “Locally consistent tof and stereo data fusion,” in _ECCV Workshop (CDC4CV)_ , 2012\.
  * [19] M. Hansard, G. Evangelidis, Q. Pelorson, and R. Horaud, “Cross-calibration of time-of-flight and colour cameras,” _CVIU_ , 2014, to appear.
  * [20] J. Čech and R. Šára, “Efficient sampling of disparity space for fast and accurate matching,” in _BenCOS Workshop-CVPR_ , 2007.
  * [21] A. Geiger, M. Roser, and R. Urtasun, “Efficient large-scale stereo matching,” in _ACCV_ , 2010.
  * [22] D. Scharstein and R. Szeliski, “A taxonomy and evaluation of dense two-frame stereo correspondence algorithms,” _IJCV_ , vol. 47, no. 1-3, 2002.
  * [23] R. Szeliski et al., “A comparative study of energy minimization methods for markov random fields with smoothness-based priors,” _IEEE TPAMI_ , vol. 30, no. 6, 2008.
  * [24] K. J. Yoon and I. S. Kweon, “Adaptive support-weight approach for correspondence search,” _IEEE TPAMI_ , vol. 28, no. 4, 2006.
  * [25] C. Rhemann, A. Hosni, M. Bleyer, C. Rother, and M. Gelautz, “Fast cost-volume filtering for visual correspondence and beyond,” in _CVPR_ , 2011.
  * [26] L. De-Maeztu, S. Mattoccia, A. Villanueva, and R. Cabeza, “Linear stereo matching,” in _ICCV_ , 2011.
  * [27] Q. Yang, “A non-local cost aggregation method for stereo matching,” in _CVPR_ , 2012.
  * [28] Q. Yang, L. Wang, and N. Ahuja, “A constant-space belief propagation algorithm for stereo matching,” in _CVPR_ , 2010.
  * [29] A. F. Bobick and S. S. Intille, “Large occlusion stereo,” _IJCV_ , vol. 33, no. 3, 1999.
  * [30] M. Bleyer, C. Rhemann, and C. Rother, “PatchMatch stereo - stereo matching with slanted support windows,” in _BMVC_ , 2011.
  * [31] Q. Yang, R. Yang, J. Davis, and D. Nistér, “Spatial-depth super resolution for range images,” in _CVPR_ , 2007.
  * [32] E. Z. Psarakis and G. D. Evangelidis, “An enhanced correlation-based method for stereo correspondence with sub-pixel accuracy,” in _ICCV_ , 2005.
  * [33] C. Tomasi and R. Manduchi, “Bilateral filtering for gray and color images,” in _ICCV_ , 1998.
  * [34] J. Kopf, M. F. Cohen, D. Lischinski, and M. Uyttendaele, “Joint bilateral upsampling,” _SIGGRAPH_ , 2007.
  * [35] K. Kuhnert and M. Stommel, “Fusion of stereo-camera and pmd-camera data for real-time suited precise 3D environment reconstruction,” in _ICIRS_ , 2006\.
  * [36] G. Van Meerbergen, M. Vergauwen, M. Pollefeys, and L. Van Gool, “A hierarchical symmetric stereo algorithm using dynamic programming,” _IJCV_ , vol. 47, no. 1–3, pp. 275–285, 2002.
  * [37] L. Wang and R. Yang, “Global stereo matching leveraged by sparse ground control points,” in _CVPR_ , 2011.
  * [38] M. Hansard, R. Horaud, M. Amat, and S. Lee, “Projective alignment of range and parallax data,” in _CVPR_ , 2011.
  * [39] S. Foix, G. Alenya, and C. Torras, “Lock-in time-of-flight (tof) cameras: A survey,” _IEEE SENS J_ , vol. 11, no. 9, 2011.
  * [40] M. Reynolds, J. Doboš, L. Peel, T. Weyrich, and G. J. Brostow, “Capturing time-of-flight data with confidence,” in _CVPR_ , 2011.
  * [41] G. Petschnigg et al., “Digital photography with flash and no-flash image pairs.” _ACM T on Graphics_ , vol. 23, no. 3, pp. 664–672, 2004.
  * [42] R. Adams and L. Bischof, “Seeded region growing,” _IEEE TPAMI_ , vol. 16, no. 6, pp. 641–647, 1994.
  * [43] H. P. Moravec, “Toward automatic visual obstacle avoidance,” in _IJCAI_ , 1977\.
  * [44] G. Egnal, M. Mintz, and R. P. Wildes, “A stereo confidence metric using single view imagery with comparison to five alternative approaches,” _IVC_ , vol. 22, no. 12, pp. 943 – 957, 2004.
  * [45] O. Mac Aodha, A. Humayun, M. Pollefeys, and G. J. Brostow, “Learning a confidence measure for optical flow,” _IEEE TPAMI._ , vol. 35, no. 5, pp. 1107–1120, 2013.
  * [46] J. Sun, Y. Li, S. B. Kang, and H.-Y. Shum, “Symmetric stereo matching for occlusion handling,” in _CVPR_ , 2005.
  * [47] P. Chou and C. Brown, “The theory and practice of bayessian image labeling,” _IJCV_ , vol. 4, no. 3, pp. 185–210, 1990.
  * [48] J. Besag, “On the statistical analysis of dirty pictures,” _Royal Statistical Soc._ , vol. 48, no. 3, pp. 259–302, 1986.
  * [49] Y. Boykov, O. Veksler, and R. Zabih, “Fast approximate energy minimization via graph cuts,” in _IEEE TPAMI_ , vol. 23, no. 11, 2001, pp. 1222–1239.
  * [50] D. Scharstein and C. Pal, “Learning conditional random fields for stereo,” in _CVPR_ , 2007.
  * [51] S. Birchfield and C. Tomasi, “A pixel dissimilarity measure that is insensitive to image sampling,” _IEEE TPAMI_ , vol. 20, no. 4, 1998.
  * [52] V. Kolmogorov and R. Zabih, “Computing visual correspondence with occlusions using graph cuts,” in _ICCV_ , 2001.

|  Georgios D. Evangelidis received his BSc, MSc and PhD degree in computer
science in 2001, 2003 and 2008 respectively from the University of Patras,
Greece. From 2007 to 2009 he was an adjunct lecturer of the Technological
Institute of Larissa, Greece. During 2009-2010, he was an ERCIM (Alain
Bensoussan) Fellow and joined the Fraunhofer Institute for Intelligent
Analysis and Information Systems (IAIS) in Sankt Augustin, Germany, as a
postdoctoral researcher. Currently, he is a researcher at the Perception Team
of INRIA Grenoble, France. His research interests are in the area of computer
vision and include 3D reconstruction, Gesture recognition and Depth-Stereo
Fusion.
---|---
|  Miles Hansard is a lecturer in Computer Science at Queen Mary, University
of London. He is a member of the Vision Group, and of the QMUL Centre for
Intelligent Sensing. His research interests include 3D scene modelling, depth
cameras, and human vision. He has BSc, MRes and PhD degrees from University
College London.
---|---
|  Radu Horaud received the B.Sc. degree in electrical engineering, the M.Sc.
degree in control engineering, and the Ph.D. degree in computer science from
the Institut National Polytechnique de Grenoble, Grenoble, France. Currently
he holds a position of director of research with the Institut National de
Recherche en Informatique et Automatique (INRIA), Grenoble Rhône-Alpes,
Montbonnot, France, where he is the founder and head of the PERCEPTION team.
His research interests include computer vision, machine learning, audio signal
processing, audiovisual analysis, and robotics. He is an area editor of the
Elsevier Computer Vision and Image Understanding, a member of the advisory
board of the Sage International Journal of Robotics Research, and an associate
editor of the Kluwer International Journal of Computer Vision. He was Program
Cochair of the Eighth IEEE International Conference on Computer Vision (ICCV
2001). In 2013, Radu Horaud was awarded a five year ERC Advanced Grant for his
project Vision and Hearing in Action (VHIA).
---|---