# SparseDC: Depth Completion from sparse and non-uniform inputs

Chen Long1 Wenxiao Zhang211footnotemark: 1 Zhe Chen1 Haiping Wang1
Yuan Liu3 Zhen Cao1 Zhen Dong1 Bisheng Yang122footnotemark: 2
1Wuhan University 2Singapore University of Technology and Design 3The
University of Hong Kong
<EMAIL_ADDRESS>
wenxxiao.zhang<EMAIL_ADDRESS>These authors contribute
equally to this work.Corresponding authors.

###### Abstract

We propose SparseDC, a model for Depth Completion of Sparse and non-uniform
depth inputs. Unlike previous methods focusing on completing fixed
distributions on benchmark datasets (e.g., NYU with 500 points, KITTI with 64
lines), SparseDC is specifically designed to handle depth maps with poor
quality in real usage. The key contributions of SparseDC are two-fold. First,
we design a simple strategy, called SFFM, to improve the robustness under
sparse input by explicitly filling the unstable depth features with stable
image features. Second, we propose a two-branch feature embedder to predict
both the precise local geometry of regions with available depth values and
accurate structures in regions with no depth. The key of the embedder is an
uncertainty-based fusion module called UFFM to balance the local and long-term
information extracted by CNNs and ViTs. Extensive indoor and outdoor
experiments demonstrate the robustness of our framework when facing sparse and
non-uniform input depths. The pre-trained model and code are available at
https://github.com/WHU-USI3DV/SparseDC.

Figure 1: Depth Completion from sparse and non-uniform inputs. From top to
bottom, we show the completed depth of state-of-the-art methods and ours using
500 depth points, 5 depth points, and non-uniform depth points obtained by
SFM, respectively. Unlike the degradation of comparison methods, our framework
can achieve reliable prediction under the above settings.

##  1 Introduction

Acquiring dense and accurate depth is crucial in various fields, such as
autonomous driving[55, 29], 3D reconstruction[8, 26, 33], scene
understanding[7, 21], AR/VR[54, 35]. Although the commonly used depth sensors,
such as Microsoft Kinect[56], Intel RealSense[19], LIDAR[2], can obtain depth
within their effective field of view, it is still a great challenge to obtain
dense depth maps due to environment, equipment, and cost limitations. In
recent years, depth completion, serving as a cost-effective means to achieve
dense pixel-wise depth, has become a long-standing research endeavor that has
captivated the attention of many researchers.

Despite achieving impressive performance on benchmark datasets[32, 9],
existing state-of-the-art methods[46, 52, 51, 58] often fall short in some
challenging cases. In contrast to the fixed data distribution of the benchmark
dataset(e.g., NYU Depth[32] with 500 points, Kitti[9] with 64 lines), the
inputs in real scenarios may be non-uniform and sparser. For example,
researchers often employ depth sensors with lower resolutions (Velodyne
HDL-16E LiDAR, VCSEL ToF sensors) to mitigate costs. In addition,
environmental factors easily influence sensors, such as high-reflectivity
objects and multipath effects resulting in uneven distribution of effective
depth points. Regrettably, mainstream methods primarily focus on the sparse-
to-dense completion process without realizing the sparsity changes and the
non-uniform distribution of depth points. Consequently, they are difficult to
reconstruct the scene structure when the spatial pattern of the input
significantly deviates from the distribution observed in the benchmark dataset
during training, as illustrated in Fig. 1

The sparse and non-uniform input depth brings great challenges in extracting
stable and robust features for the completion task. In the regions with dense
input depth, the network needs to learn to trust the local dense input depth
values while in regions with sparse or even no depth, the network is supposed
to speculate the depth values from the RGB features and other long-term
context information. However, the changing input depth patterns shown in Fig.
1(a) make the network confusing in which kind of features should be extracted
even with data augmentation to simulate the changing depth in training [47,
10, 6, 43]. To solve this problem, we introduce Sparse Feature Filling Module
that explicitly fills the unstable depth features with stable image features
thereby improving the robustness of the network to various spatial
distributions of input depth.

Furthermore, we observed that CNNs and ViTs show different characteristics in
dealing with non-uniform input depths. In a simple experiment, we select
ResNet-18[12] and PvtV2-b1[44, 45] as two backbone networks for the depth
completion task and evaluate them on the NYU Depth V2[32] with input depth
maps with varying densities. As illustrated in Fig. 2, though the overall
performance of ResNet is better, as the density of input depth decreases, more
and more depth values predicted by the PvtV2-b1 are more accurate than those
predicted by the ResNet-18. The reason is that high-density depth points are
sufficient to provide rich local context and detail for depth completion,
while for low-density inputs, we need to understand the overall scene to fill
large areas. Correspondingly, CNNs are known for their ability to extract
local features with fine-grained details while ViTs focus on capturing the
global scene context to leverage global relationships among pixels. The long-
term context helps the ViT to account for the sparse input depth values. This
observation motivates us to design a two-branch network structure that
consists of both CNNs and ViTs to combine the advantages of both parts for a
better depth completion on the sparse and non-uniform input depth maps.

Figure 2: Pilot study results on NYU Depth V2 dataset. The horizontal axis
represents the number of effective depth points in each testing depth map. The
vertical axis illustrates the number of testing depth maps for which the
network outperforms the other in terms of RMSE.

An important problem here is how to fuse the two-branch features to get robust
features. As discussed before, the network needs to put different importance
on features from the CNNs and ViTs according to the local density of the input
depth maps. However, a naive fusion [52, 46], which simply concatenate or add
the features from two branches, actually treats features from two branches
equally and thus leads to degenerated performances. To address this problem,
we design an Uncertainty-Guided Feature Fusion Module(UFFM) to dynamically
fuse extracted features explicitly. The UFFM uses sparse depths to estimate
and correct the uncertainty of each pixel and utilizes this information to
guide the fusion process, enabling the network to adjust the relative
contribution of each branch based on the distribution patterns of the inputs.

Figure 3: SparseDC Architecture. Given a sparse depth map and an image, we
obtain a stable initial feature ${\boldsymbol{\rm F}}$ using SFFM. Then, we
adopt a two-branch feature extractor to get multi-scale features. After that,
the extracted features and sparse depth map are fed to UFFM to obtain the
initial depth and estimated uncertainty. Finally, a non-local propagation
network[34] is exploited for final refinement.

In summary, we propose a novel deep completion framework named SparseDC, Fig.
3 illustrates the whole pipeline with above new components. SparseDC exhibits
significant robustness improvement with better efficiency when dealing with
sparse and non-uniform depths. We conduct a comprehensive experimental
evaluation of our approach on NYU Depth[32], KITTI DC[9], and SUN RGB-D[41]
and demonstrate 17% improvements on REL metric, 7.8% improvements on RMSE
metric, respectively, over the State-of-the-art method CompletionFormer[52].

Overall, our contributions can be summarized as follows:

  * •

We analyze the limitations of mainstream methods when dealing with sparse and
non-uniform inputs and propose a novel depth completion framework called
SparseDC to tackle this. It consists of several modules which can help extract
stable and robust features for the completion task.

  * •

We access the performance between ours and state-of-the-art methods on some
simulated and real challenging cases using indoor[32, 41] and outdoor
datasets[9]. Extensive experiments show the robustness of SparseDC when
dealing with sparse and non-uniform depths.

##  2 Related Work

Depth Completion is a fundamental task in computer vision. It aims at
recovering dense pixel-level depth from sparse depth maps obtained from
sensors. In recent years, deep learning-based methods have emerged as the
leading approaches to this task. Uhrig et al.[43] introduced sparse
convolution as an alternative to traditional convolution and established
benchmark datasets based on Kitti for evaluating depth completion algorithms.
Later, Ma et al.[28] incorporated RGB data as an additional input,
significantly improving the performance for the deep completion task. Building
upon this[28], researchers have been growing interest in exploring multimodal
data fusion in their frameworks, such as images[30, 37, 17, 24, 57, 36],
normals[50, 11], semantics[31], and residual depth maps[23]. Then, Cheng et
al. proposed a convolutional spatial propagation network CSPN[3], the first to
apply spatial propagation techniques to depth completion. After that, methods
such as CSPN++[4], NLSPN[34], DySPN[22], GraphCSPN[25], etc. have been
proposed, they improve the accuracy and efficiency of depth completion by
designing different convolution strategies (deformable convolution, graph
convolution, etc.) to propagate the local information provided by the depth
points over image, and achieve SOTA performance on benchmark datasets. To push
the envelope of depth completion, recent approaches tended to use complex
network structures and sophisticated learning strategies[14, 58, 46, 51], such
as BEV@DC[52], which proposed a more efficient and powerful multi-modal
training scheme to boost the performance of image-guided depth completion.
However, these methods mainly emphasize sparse-to-dense conversion without
realizing the impact of sparsity changes and non-uniform depth points, which
limits their scope of applications in real scenarios.

Some researchers have considered this problem and given their solutions.
Andrea Conti et al. proposed SpAgNet[5]. The key idea is not to directly feed
sparse depth points to the convolutional layers, but to iteratively merge the
sparse input points with multiple depth maps predicted by the network during
the decoding process. This approach mitigates the effects of uneven inputs.
However, the receptive field of the depth points is very restricted, leading
to the final result that relies on the performance of the backbone network,
which increases the number of parameters of the model (51M $\gg$ 25.8M
NLSPN[34]) and leads to a domain bias, limiting its applicability. Another
related research is Sparse SPN[48], which propagates sparsely distributed
depth values to the rest of the image by designing a cross-shaped spatial
propagation network. However, this approach still relies on the local
inductive bias provided by the spatial propagation technique, and has limited
ability to model depth values over long ranges. In addition, Sparse SPN[48] is
designed only for keypoint sampling, which cannot handle other spatial
patterns such as uneven density and big holes.

##  3 Method

###  3.1 Overview

Given a sparse depth map $\mathcal{S}\in\mathbb{R}^{H\times W\times 1}$ and an
image $\mathcal{I}\in\mathbb{R}^{H\times W\times 3}$, our goal is to restore a
dense pixel-wise depth $\mathcal{D}\in\mathbb{R}^{H\times W\times 1}$.
Different from mainstream methods, we focus on dealing with sparse and non-
uniform inputs.

The whole pipeline of SparseDC is illustrated in Fig. 3. First, we feed the
depth map $\mathcal{S}$ and image $\mathcal{I}$ into the SFFM to obtain an
initial feature map ${\boldsymbol{\rm F}}\in\mathbb{R}^{H\times W\times C}$.
Then, we embed ${\boldsymbol{\rm F}}$ with two separate feature extractors,
composed by CNNs and ViTs, to respectively extract so-called local multi-scale
features ${\boldsymbol{\rm F}}_{local}=\\{{f}^{n}_{l}\\}_{n=0}^{N}$ and global
multi-scale features ${\boldsymbol{\rm
F}}_{global}=\\{{f}^{n}_{g}\\}_{n=0}^{N}$, where $f_{*}^{n}$ indicates the
$n^{th}$ scale feature map. After that, we design an UFFM to predict the
pixel-wise uncertainty to guide the fusion of local features ${\boldsymbol{\rm
F}}_{local}$ and global features ${\boldsymbol{\rm F}}_{global}$ at different
scales. We thus get the fused features ${\boldsymbol{\rm
F}}_{fuse}=\\{{f}^{n}\\}_{n=0}^{N}$ to estimate the initial depth map and
uncertainty. Finally, we apply the non-local spatial propagation module[34] to
refine the initial depth map and get the final depth $D$.

###  3.2 Sparse Feature Fill Module

Given an image and a depth map, SFFM aims to output a feature map mitigating
the feature instability[47] caused by non-uniform depths. As shown in Fig. 4,
existing methods struggle to maintain the geometry shape in the face of non-
uniform inputs, which can be alleviated by SFFM.

Figure 4: Visual comparison. Illustration of the effectiveness of our proposed
SFFM on CNN and ViT backbones with non-uniform depth inputs.

Fig. 5 depicts the detailed structure of the SFFM. Given the sparse and non-
uniform depth map $\mathcal{S}$ and the image $\mathcal{I}$, we first extract
the features of the image and depth with a stack of 2D convolution, batch
normalization[16], and LeakyReLU[1, 49] layer. Then, we concatenate the image
and depth features on the feature dimension. After that, we use gate
convolution module[53] with channel attention to fuse these features. It
learns the weights of the stable image features and sparse depth features
based on the distribution of input depths, and use the weight to dynamically
guide the feature filling to obtain ${\boldsymbol{\rm
F}}_{dep}^{{}^{\prime}}$. Finally, ${\boldsymbol{\rm F}}_{dep}^{{}^{\prime}}$
and ${\boldsymbol{\rm F}}_{rgb}$ are concatenated to obtain a stable feature
input $F$ by using another gated convolution layer. In addition, to control
the feature range of $F$ and accelerate the convergence of the model, we
predict a coarse depth map $D^{{}^{\prime}}$ by feeding $F$ to an extra depth
decoder head for intermediate supervision.

Figure 5: An illustration of SFFM. SFFM aims to output stable and robustness
features for depth completion.

###  3.3 Two Branch Feature Extraction

Motivated by the observations in Fig. 2, we introduced a two-branch feature
extraction architecture, one convolution-based branch focuses on extracting
the local information provided by the depth points, called Local Branch, and
the other branch is based on the vision transformer to provide indispensable
global information, called Global Branch.

Local Branch. We choose ResNet-18[12] pretrained on ImageNet[20] as the
backbone network of the convolution branch. Specifically, it downsamples the
feature map ${\boldsymbol{\rm F}}$ to scales
$\frac{1}{1},\frac{1}{2},\frac{1}{4},\frac{1}{8},\frac{1}{16}$. After that,
the features will be used in the UNet[38] decoding step as input to get
${\boldsymbol{\rm F}}_{local}$.

Global Branch. We chose the lightweight PvtV2-b1[45] also pre-trained on
ImageNet[20] as our global feature extract backbone. This branch first splits
the input Feature ${\boldsymbol{\rm F}}$ into $4\times 4$ patches by a
convolution filter with a kernel size of 7. After that, a linear embedding
layer is applied to project the downsampled feature to an arbitrary dimension.
Then, it uses 4 transformer layers to downsample the initial feature to
$\frac{1}{1},\frac{1}{4},\frac{1}{8},\frac{1}{16},\frac{1}{32}$, and these
features will also be used in decoding steps to get ${\boldsymbol{\rm
F}}_{global}$, like ${\boldsymbol{\rm F}}_{local}$.

###  3.4 Uncertainty-Guide Feature Fusion Module

Given non-uniform and sparse inputs, the network needs to learn to trust the
local dense input depth values in the regions with dense input depth. In
regions with sparse or even no depth, the network is supposed to speculate the
depth values from long-term context information.

Figure 6: Weight Map Visualization. We visualize the weight map with different
input patterns. The darker the color, the more the global branch contributes
to the whole model.

To achieve this, we propose a weighted fusion strategy that enables the
network to dynamically determine the confidence levels of local and global
branch features. Specifically, we predict the pixel-wise uncertainty of each
feature branch. Then, we use the input depth points to rectify the uncertainty
and finally use uncertainty to guide the explicit fusion of features. Fig. 6
illustrates the effect of UFFM in the face of sparse and non-uniform inputs.
UFFM can adjust the contributions of each branch according to the inputs’
spatial pattern, realizing the complementary advantages of global and local
information.

Figure 7: UFFM Architecture. UFFM aims to dynamically fuse the extracted
features and then estimate the initial depth and reliable uncertainty.

In practice, UFFM applies several Uncertainty-guided Feature Fusion Block
(UFFB) at different scales $\frac{1}{1},\frac{1}{2},\frac{1}{4},\frac{1}{8}$.
Fig. 7 illustrates the detailed structure of UFFM. Given the multi-scale local
features ${\boldsymbol{\rm F}}_{local}$ and global features ${\boldsymbol{\rm
F}}_{global}$, UFFB first uses two different convolutional layers (depicted as
UDM in Fig. 7) to compute an uncertainty and a depth map at this scale. These
will be fed to the UFFB of the next scale to correct the predicted results
iteratively. Since there is no ground truth for the uncertainty prediction, we
compute a pseudo ground truth uncertainty inspired by [40], formulated in Eq.
1.

$\hat{u}=1-{\operatorname{exp}}(-\frac{|d^{n}-g^{n}|}{b\times g^{n}})$ (1)

where, $\hat{u}$ denotes the defined pseudo ground truth uncertainty, $g^{n}$
is the ground truth depth map at scale $n^{th}$, the $d^{n}$ is predicted
depth maps. $b$ is a coefficient that controls the tolerance for error, we set
$b$ as 0.1 in this paper to amplify differences between features. In addition,
to further exploit the depth information, we downsample the sparse depth map
$\mathcal{S}$ by sparsity aware pooling to get multi-scale depth maps. After
that, we use Eq. 1 to calculate the accurate uncertainty and replace the
estimated uncertainty at positions that have valid depth points. Then, we use
the predicted and replaced uncertainty to guide the fusion of different
features as shown in Eq. 2.

$f^{n}=Gate((1-u^{n}_{l})\cdot f^{n}_{l}\oplus(1-u^{n}_{g})\cdot f^{n}_{g})$
(2)

where, $u^{n}_{l}$ denotes the local feature uncertainty, the $u^{n}_{g}$
denotes the global feature uncertainty, and the $f^{n}$ is the fused feature
which will be fed to non-local spatial propagation module[34] to compute the
final completed depth $D$.

###  3.5 Loss Fuction

We train our network by supervising all the outputs of each UFFM. First, we
downsample $G$ through average pooling to obtain multi-scale ground truths
dense depth maps $\\{g^{n}\\}_{n=0}^{N}$ and use Eq. 1 to compute pseudo
uncertainty. Then, we use $\mathcal{L}_{1}$ loss to supervise the predicted
uncertainty and $\mathcal{L}_{2}$ loss for depth at each scale, which is
described by Eq. 3.

$\begin{split}L_{g}^{n}&=0.5|u^{n}_{g}-\hat{u}^{n}_{g}|+\|d^{n}_{g}-g^{n}\|\\\
L_{l}^{n}&=0.5|u^{n}_{l}-\hat{u}^{n}_{l}|+\|d^{n}_{l}-g^{n}\|\\\
L^{n}&=0.5|u^{n}-\hat{u}^{n}|+\|d^{n}-g^{n}\|\\\
L_{rec}^{n}&=0.5(L_{g}^{n}+L_{l}^{n})+L^{n}\\\ \end{split}$ (3)

Then, we need to supervise the output of SFFM and non-local spatial
propagation module, the overall loss function can be formulated by Eq. 4

$L=\sum^{N}_{n=0}\gamma^{n}L_{rec}^{n}+0.05\|D^{{}^{\prime}}-G\|+|D-G|+\|D-G\|$
(4)

where $\gamma$ denotes the exponential decay factor, we set it as 0.8 in this
paper to make the network pay more attention to the high-resolution outputs.

##  4 Experiments

###  4.1 Experimental Settings

In our experiments, we included both indoor and outdoor datasets.

NYU Depth V2. The NYU Depth V2[32] is a widely used indoor dataset in depth
completion, which contains 40,800 RGBD images captured by Microsoft Kinect
with an original resolution of 640$\times$480\. Each image and depth map
should be down-sampled to 320$\times$240, then center-cropped to
304$\times$228\. We follow the previous work[28] to divide the dataset and set
the upper bound of depth to 10 meters. Unlike previous methods that randomly
sampled 500 points from the ground truth as sparse input, we sampled 5-500
random points during training.

SUN RGB-D The SUN RGB-D dataset[41] contains 10,335 raw RGB-D images captured
by four different sensors. This dataset has more challenging scenes and
sensors, which helps to evaluate model generalization efficiently.

KITTI depth completion. The KITTI dataset[9] contains over 93,000 raw sparse
LIDAR scans (retrieved by Velodyne HDL-64E LiDAR sensor) and corresponding RGB
images. The training, validation, and test set have 86,000, 7,000, and 1,000
samples, respectively[14], and the upper depth bound is 80 meters. For outdoor
acquisition scenarios, in addition to high-emissivity objects and noise, the
other factor is the different vertical resolution of the different sensors,
e.g., 64-line LIDAR vs. 16-line LIDAR. So, we devised a simple but useful
training strategy on KITTI that randomly masks out all depths from 0%-95% of
rows during training.

Evaluation metrics. We follow [13, 18] to use three metrics for the dense
depth prediction evaluation RMSE: Root mean squared error, MAE: Mean absolute
error, REL: Mean relative error. Performances using other metrics are reported
in the Appendix.

Baselines. We compared SparseDC with serval state-of-the-art depth completion
networks, including NLSPN (ECCV 2020)[34], GraphCSPN (ECCV 2022)[25],
CompletionFormer (CVPR 2023)[52], PENet (ICRA 2021)[15]. For a fair
comparison, we retrained all of them using their code and our training
strategy. Note that, we did not compare it with SpAgNet[5], SparseSPN[48] and
some other state-of-the-art methods[58, 46, 22, 51], et al. because they did
not release their code.

Figure 8: Sparse depth patterns. Including varying sparsity depth points,
shift grid patterns (VCSEL TOF sensors), keypoints-based sampling, uneven
density depth points, and big holes.

Implementation details. We trained our network with a batch size 16 for NYU (8
for each GPU) and 6 for KITTI (2 for each GPU) on the PyTorch platform,
respectively. The Adam optimizer is used with an initial learning rate of
$10^{-4}$, which is decreased in every unimproved five epochs by a decay rate
of $0.3$. Our training process will stop when there is no improvement for ten
consecutive epochs. We will release the source code and the trained models
upon acceptance.

Figure 9: Qualitative results on NYU Depth V2. Comparing completion results
with different patterns of inputs using different methods.

###  4.2 Results on NYU Depth V2

Unlike existing methods, which mainly focus on sparse-to-dense solutions, our
networks pay more attention to dealing with sparse and non-uniform inputs. We
trained our network on NYU Depth V2, and randomly sampled $5\sim 500$ depth
points as inputs during training. SpAgNet[5] has proved that existing methods
trained on fixed spatial patterns(500 random samples) are not comparable in
the face of some challenging cases. For example, with only 5 points, the RMSE
by NLSPN[34] is 1.033m. So, we retrained the comparison methods using their
released code and the same training strategy with us for a fair comparison.

Model | Samples |  RMSE (m)$\downarrow$ |  REL$\downarrow$
---|---|---|---
NLSPN[34] | 500 | 0.0984 | 0.0136
GraphCSPN[25] | 500 | 0.0990 | 0.0136
CompletionFormer[52] | 500 | 0.0984 | 0.0135
Ours | 500 | 0.0976 | 0.0128
NLSPN[34] | 200 | 0.1336 | 0.0201
GraphCSPN[25] | 200 | 0.1338 | 0.0197
CompletionFormer[52] | 200 | 0.1349 | 0.0205
Ours | 200 | 0.1346 | 0.0191
NLSPN[34] | 50 | 0.2123 | 0.0382
GraphCSPN[25] | 50 | 0.2119 | 0.0375
CompletionFormer[52] | 50 | 0.2183 | 0.0405
Ours | 50 | 0.2079 | 0.0350
NLSPN[34] | 10 | 0.3639 | 0.0827
GraphCSPN[25] | 10 | 0.3650 | 0.0832
CompletionFormer[52] | 10 | 0.3642 | 0.0874
Ours | 10 | 0.3226 | 0.0681
NLSPN[34] | 5 | 0.4439 | 0.1094
GraphCSPN[25] | 5 | 0.4555 | 0.1151
CompletionFormer[52] | 5 | 0.4377 | 0.1144
Ours | 5 | 0.3816 | 0.0877
NLSPN[34] | shift_grid | 0.4263 | 0.1043
GraphCSPN[25] | shift_grid | 0.4485 | 0.1083
CompletionFormer[52] | shift_grid | 0.4313 | 0.1149
Ours | shift_grid | 0.3736 | 0.0852
NLSPN[34] | uneven density | 0.3931 | 0.0909
GraphCSPN[25] | uneven density | 0.3991 | 0.0920
CompletionFormer[52] | uneven density | 0.3919 | 0.0961
Ours | uneven density | 0.3543 | 0.0763
NLSPN[34] | ORB keypoint | 0.2590 | 0.0625
GraphCSPN[25] | ORB keypoint | 0.2571 | 0.0603
CompletionFormer[52] | ORB keypoint | 0.2660 | 0.0672
Ours | ORB keypoint | 0.2441 | 0.0541
NLSPN[34] | big holes | 0.2368 | 0.0402
GraphCSPN[25] | big holes | 0.2443 | 0.0403
CompletionFormer[52] | big holes | 0.2493 | 0.0434
Ours | big holes | 0.2262 | 0.0362
Table 1: Results on NYU Depth V2 with different patterns. Quantitative
comparisons with state-of-the-art methods in different spatial patterns. The
best results are red and in bold.

We designed the following scenarios during testing to simulate varying
sparsity and non-uniform inputs as shown in Fig. 8. 1) Varying sparsity
inputs: Randomly sampling depth points included some extreme cases, such as
only 5 valid depth points. 2) VCSEL TOF sensors[27]: Sampling a shifting grid
from the ground truth. 3) Uneven density: Randomly selected two regions with
different sizes and different numbers of depth points from the ground truth.
4) Keypoint-based Sampling: Following the SpaseSPN[48], we use a keypoints-
based(ORB[39]) sample strategy to simulate the sparse depth values available
from keypoint triangulation. 5) Holes: Randomly select depth points, then
select a $200\times 200$ square grid, and remove all depth points inside the
grid.

The quantitative comparison results on NYU Depth V2 under various non-uniform
settings are shown in Tab. 7 and some visual comparisons are given in Fig. 9.
It can be seen that SparseDC consistently outperforms the baselines only
except the 200-sample case. Moreover, our method achieved 17% improvements on
REL and 7.8% improvements on RMSE, respectively, over the State-of-the-art
method CompletionFormer[52]. This is mainly because SparseDC can dynamically
adjust the reliance on both local and global features during the depth
completion process and thus can achieve a balance to retain the precise
geometry of regions with available depth values and accurate structures in
regions with no depth input.

Methods | Params |  RMSE(m)$\downarrow$ |  REL$\downarrow$
---|---|---|---
NLSPN[34] | 25.8M | 0.5966 | 0.1257
NLSPN*[34] | 25.8M | 3.5966 | 1.9789
GraphCSPN$*$[25] | 26M | 4.3420 | 2.2700
CompletionFormer[52] | 45M | 0.9613 | 0.4707
CompletionFormer$*$[52] | 45M | 0.6047 | 0.1574
Ours | 38.2M | 0.4536 | 0.1180
Table 2: Generalization Ability Test on SUN RGBD. $*$ denotes trained model
using our training strategy, without $*$ is their released pretrained model
trained using fixed patterns.

###  4.3 Generalization results on SUN RGB-D

To validate the generalization ability of our method, we use the model trained
on NYU[32] to conduct depth completion on SUN RGB-D directly. In the test set,
we used the noisy and incomplete raw depth image as the input without
sampling. As shown in Tab. 2, our proposed SparseDC performs best in most
metrics. The augment-based strategy makes [34, 25] overfit the training data.
Completionformer[52] and SparseDC fuse global and local information
simultaneously, resulting in improvements and robustness with unseen patterns.
We report the corresponding visual comparison results in the Appendix.

###  4.4 Results on KITTI Depth Completion

On the outdoor KITTI dataset, we randomly mask some rows in depth maps during
training as mentioned in Sec. 4.1. Due to resource constraints, we directly
use the metric in reported CompltionFormer[52] The specific results are shown
in Tab. 3. It can be seen that SparseDC consistently outperforms the baselines
by a large margin in MAE, and the gap becomes larger when fewer scanning lines
are kept, demonstrating the effectiveness of SparseDC under sparse input.

###  4.5 Ablation Study

To demonstrate the effectiveness of each component proposed in SparseDC, we
conducted corresponding ablation experiments on NYU Depth V2 dataset by
removing each component as follows.

Methods | Scanning Lines |  RMSE(m)$\downarrow$ |  MAE(m)$\downarrow$
---|---|---|---
NLSPN[34] | lines4 | 2.2648 | 0.7492
PENet[15] | lines4 | 2.2323 | 0.8132
CompletionFormer[52] | lines4 | 2.1500 | 0.7401
DySPN[22] | lines4 | 2.2858 | 0.8343
Ours | lines4 | 2.1029 | 0.7012
NLSPN[34] | lines8 | 1.5452 | 0.4593
PENet[15] | lines8 | 1.5983 | 0.5261
Ours | lines8 | 1.5286 | 0.4444
NLSPN[34] | lines16 | 1.1781 | 0.3274
PENet[15] | lines16 | 1.2231 | 0.3818
CompletionFormer[52] | lines16 | 1.2186 | 0.3374
DySPN[22] | lines16 | 1.2748 | 0.3664
Ours | lines16 | 1.1764 | 0.3178
NLSPN[34] | lines32 | 0.9575 | 0.2587
PENet[15] | lines32 | 0.9866 | 0.3019
Ours | lines32 | 0.9485 | 0.2488
NLSPN[34] | lines64 | 0.8125 | 0.2167
PENet[15] | lines64 | 0.8406 | 0.2472
CompletionFormer[52] | lines64 | 0.8487 | 0.2159
DySPN[22] | lines64 | 0.8785 | 0.2286
Ours | lines64 | 0.7966 | 0.2051
Table 3: Results on KITTI DC. Quantitative comparisons with state-of-the-art
methods on varying scanning lines.  Figure 10: Qualitative results on KITTI
DC. We report the completed depth and error map with varying scanning lines as
inputs.

  * •

Remove the Sparse Feature Filling Module, and replace it with two simple
convolution layers, as previous methods[34, 25].

  * •

Remove the Uncertainty-Guide Feature Fusion Module, and use Eq. 5 to fuse the
extracted features.

$f^{n}=Gate(f^{n}_{l}\oplus\cdot f^{n}_{g})$ (5)

  * •

Removing the two-branch feature extraction structure in favor of a single-
branch one, i.e., the overall model will degenerate into a combination of
ResNet[12] with a non-local spatial propagation module, so we use the
retrained NLSPN[34] as a replacement.

Model | samples |  RMSE(m)$\downarrow$ |  REL$\downarrow$
---|---|---|---
w/o UFFM | 500 | 0.0973 | 0.0128
w/o SFFM | 500 | 0.0974 | 0.0128
w/o Two-Branch | 500 | 0.0984 | 0.0136
Full Model | 500 | 0.0976 | 0.0128
w/o UFFM | 50 | 0.2089 | 0.0356
w/o SFFM | 50 | 0.2092 | 0.0354
w/o Two-Branch | 50 | 0.2123 | 0.0382
Full Model | 50 | 0.2079 | 0.0350
w/o UFFM | 5 | 0.3846 | 0.0904
w/o SFFM | 5 | 0.3854 | 0.0918
w/o Two-Branch | 5 | 0.4439 | 0.1094
Full Model | 5 | 0.3816 | 0.0877
w/o UFFM | shift_grid | 0.4618 | 0.1072
w/o SFFM | shift_grid | 0.4380 | 0.1042
w/o Two-Branch | shift_grid | 0.4263 | 0.1043
Full Model | shift_grid | 0.3736 | 0.0852
w/o UFFM | uneven density | 0.5391 | 0.1266
w/o SFFM | uneven density | 0.4124 | 0.0934
w/o Two-Branch | uneven density | 0.3931 | 0.0909
Full Model | uneven density | 0.3543 | 0.0763
Table 4: Ablation study. The evaluation results of different components of
SparseDC.

All ablation models are trained under the same settings as SparseDC, the
results are shown in Tab. 4.

Our full pipeline gets the minimum RMSE and REL value for most experiments,
removing any component will degrade the overall performance of the network. In
addition, the results show that removing the Two-Branch feature extraction
structure has the greatest impact on the network in the face of extremely
sparse inputs. This is because the Two-Branch structure provides both global
semantic and local geometric information, which is necessary for dealing with
non-uniform inputs. Furthermore, SFFM provides a stabilized feature space for
sparse depth maps, which makes the depth estimated by the network more
accurate. UFFM can predict the uncertainty based on the spatial pattern of the
inputs, and then explicitly guides the fusion process of features from
different branches by this uncertainty, as shown in Fig. 6. Both modules
benefit SparseDC with sparse and non-uniform input.

##  5 Conclusion

This paper focused on depth completion of sparse and non-uniform inputs. To
address this, We first propose an SFFM module to improve the feature stability
by explicitly filling the unstable depth features with stable image features.
Then, we introduce a two-branch feature embedder to extract local or long-term
information of the input depth map and RGB image by CNNs and ViTs. Here we
propose an uncertainty-based fusion module called UFFM to predict both precise
local geometries of regions with available depth values and accurate
structures in regions with no depth input. Extensive indoor and outdoor
experiments demonstrate our method’s effectiveness when facing sparse and non-
uniform inputs.

## References

  * Agarap [2018] Abien Fred Agarap.  Deep learning using rectified linear units (relu).  _arXiv preprint arXiv:1803.08375_ , 2018.
  * Atanacio-Jiménez et al. [2011] Gerardo Atanacio-Jiménez, José-Joel González-Barbosa, Juan B Hurtado-Ramos, Francisco J Ornelas-Rodríguez, Hugo Jiménez-Hernández, Teresa García-Ramirez, and Ricardo González-Barbosa.  Lidar velodyne hdl-64e calibration using pattern planes.  _International Journal of Advanced Robotic Systems_ , 8(5):59, 2011.
  * Cheng et al. [2018] Xinjing Cheng, Peng Wang, and Ruigang Yang.  Depth Estimation via Affinity Learned with Convolutional Spatial Propagation Network.  In _Computer Vision – ECCV 2018_ , pages 108–125. Springer International Publishing, Cham, 2018.
  * Cheng et al. [2019] Xinjing Cheng, Peng Wang, Chenye Guan, and Ruigang Yang.  CSPN++: Learning Context and Resource Aware Convolutional Spatial Propagation Networks for Depth Completion, 2019.
  * Conti et al. [2023] Andrea Conti, Matteo Poggi, and Stefano Mattoccia.  Sparsity agnostic depth completion.  In _Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision_ , pages 5871–5880, 2023.
  * Contributors [2022] Spconv Contributors.  Spconv: Spatially sparse convolution library.  https://github.com/traveller59/spconv, 2022.
  * Cordts et al. [2016] Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele.  The cityscapes dataset for semantic urban scene understanding.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 3213–3223, 2016.
  * Geiger et al. [2011] Andreas Geiger, Julius Ziegler, and Christoph Stiller.  Stereoscan: Dense 3d reconstruction in real-time.  In _2011 IEEE intelligent vehicles symposium (IV)_ , pages 963–968. Ieee, 2011.
  * Geiger et al. [2012] Andreas Geiger, Philip Lenz, and Raquel Urtasun.  Are we ready for autonomous driving? the kitti vision benchmark suite.  In _Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2012.
  * Graham and Van der Maaten [2017] Benjamin Graham and Laurens Van der Maaten.  Submanifold sparse convolutional networks.  _arXiv preprint arXiv:1706.01307_ , 2017.
  * Gu et al. [2021] Jiaqi Gu, Zhiyu Xiang, Yuwen Ye, and Lingxuan Wang.  DenseLiDAR: A Real-Time Pseudo Dense Depth Guided Depth Completion Network.  _IEEE Robotics and Automation Letters_ , 6(2):1808–1815, 2021.
  * He et al. [2016] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 770–778, 2016.
  * Hu et al. [2019] Junjie Hu, Mete Ozay, Yan Zhang, and Takayuki Okatani.  Revisiting single image depth estimation: Toward higher resolution maps with accurate object boundaries.  In _2019 IEEE winter conference on applications of computer vision (WACV)_ , pages 1043–1051. IEEE, 2019.
  * Hu et al. [2023] Junjie Hu, Chenyu Bao, Mete Ozay, Chenyou Fan, Qing Gao, Honghai Liu, and Tin Lun Lam.  Deep Depth Completion from Extremely Sparse Data: A Survey.  _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , 2023.
  * Hu et al. [2021] Mu Hu, Shuling Wang, Bin Li, Shiyu Ning, Li Fan, and Xiaojin Gong.  PENet: Towards Precise and Efficient Image Guided Depth Completion, 2021.
  * Ioffe and Szegedy [2015] Sergey Ioffe and Christian Szegedy.  Batch normalization: Accelerating deep network training by reducing internal covariate shift.  In _International conference on machine learning_ , pages 448–456. pmlr, 2015.
  * Jeon et al. [2022] Yurim Jeon, Hwichang Kim, and Seung-Woo Seo.  ABCD: Attentive Bilateral Convolutional Network for Robust Depth Completion.  _IEEE Robotics and Automation Letters_ , 7(1):81–87, 2022.
  * Jiang et al. [2021] Hualie Jiang, Laiyan Ding, Junjie Hu, and Rui Huang.  Plnet: Plane and line priors for unsupervised indoor depth estimation.  In _2021 International Conference on 3D Vision (3DV)_ , pages 741–750. IEEE, 2021.
  * Keselman et al. [2017] Leonid Keselman, John Iselin Woodfill, Anders Grunnet-Jepsen, and Achintya Bhowmik.  Intel realsense stereoscopic depth cameras.  In _Proceedings of the IEEE conference on computer vision and pattern recognition workshops_ , pages 1–10, 2017.
  * Krizhevsky et al. [2012] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton.  Imagenet classification with deep convolutional neural networks.  _Advances in neural information processing systems_ , 25, 2012.
  * Li et al. [2009] Li-Jia Li, Richard Socher, and Li Fei-Fei.  Towards total scene understanding: Classification, annotation and segmentation in an automatic framework.  In _2009 IEEE Conference on Computer Vision and Pattern Recognition_ , pages 2036–2043. IEEE, 2009.
  * Lin et al. [2022] Yuankai Lin, Tao Cheng, Qi Zhong, Wending Zhou, and Hua Yang.  Dynamic Spatial Propagation Network for Depth Completion.  _Proceedings of the AAAI Conference on Artificial Intelligence_ , 36(2):1638–1646, 2022.
  * Liu et al. [2020] Lina Liu, Xibin Song, Xiaoyang Lyu, Junwei Diao, Mengmeng Wang, Yong Liu, and Liangjun Zhang.  FCFR-Net: Feature Fusion based Coarse-to-Fine Residual Learning for Depth Completion.  page 9, 2020.
  * Liu et al. [2021] Lina Liu, Yiyi Liao, Yue Wang, Andreas Geiger, and Yong Liu.  Learning Steering Kernels for Guided Depth Completion.  _IEEE Transactions on Image Processing_ , 30:2850–2861, 2021.
  * Liu et al. [2022] Xin Liu, Xiaofei Shao, Bo Wang, Yali Li, and Shengjin Wang.  GraphCSPN: Geometry-Aware Depth Completion via Dynamic GCNs, 2022.
  * Long et al. [2022] Chen Long, WenXiao Zhang, Ruihui Li, Hao Wang, Zhen Dong, and Bisheng Yang.  Pc2-pu: Patch correlation and point correlation for effective point cloud upsampling.  In _Proceedings of the 30th ACM International Conference on Multimedia_ , pages 2191–2201, 2022.
  * Luetzenburg et al. [2021] Gregor Luetzenburg, Aart Kroon, and Anders A Bjørk.  Evaluation of the apple iphone 12 pro lidar for an application in geosciences.  _Scientific reports_ , 11(1):22221, 2021.
  * Ma and Karaman [2018] Fangchang Ma and Sertac Karaman.  Sparse-to-dense: Depth prediction from sparse depth samples and a single image.  In _2018 IEEE international conference on robotics and automation (ICRA)_ , pages 4796–4803. IEEE, 2018.
  * Maurer et al. [2016] Markus Maurer, J Christian Gerdes, Barbara Lenz, and Hermann Winner.  _Autonomous driving: technical, legal and social aspects_.  Springer Nature, 2016.
  * Metzger et al. [2022] Nando Metzger, Rodrigo Caye Daudt, and Konrad Schindler.  Guided Depth Super-Resolution by Deep Anisotropic Diffusion, 2022.
  * Märkert et al. [2022] Fabian Märkert, Martin Sunkel, Anselm Haselhoff, and Stefan Rudolph.  Segmentation-guided Domain Adaptation for Efficient Depth Completion, 2022.
  * Nathan Silberman and Fergus [2012] Pushmeet Kohli Nathan Silberman, Derek Hoiem and Rob Fergus.  Indoor segmentation and support inference from rgbd images.  In _ECCV_ , 2012.
  * Pan et al. [2016] Hailong Pan, Tao Guan, Yawei Luo, Liya Duan, Yuan Tian, Liu Yi, Yizhu Zhao, and Junqing Yu.  Dense 3d reconstruction combining depth and rgb information.  _Neurocomputing_ , 175:644–651, 2016.
  * Park et al. [2020] Jinsun Park, Kyungdon Joo, Zhe Hu, Chi-Kuei Liu, and In So Kweon.  Non-local Spatial Propagation Network for Depth Completion.  In _Computer Vision – ECCV 2020_ , pages 120–136. Springer International Publishing, Cham, 2020.
  * Park et al. [2019] Mi Jin Park, Dong Jun Kim, Unjoo Lee, Eun Jin Na, and Hong Jin Jeon.  A literature overview of virtual reality (vr) in treatment of psychiatric disorders: recent advances and limitations.  _Frontiers in psychiatry_ , 10:505, 2019.
  * Qiu et al. [2019] Jiaxiong Qiu, Zhaopeng Cui, Yinda Zhang, Xingdi Zhang, Shuaicheng Liu, Bing Zeng, and Marc Pollefeys.  Deeplidar: Deep surface normal guided depth prediction for outdoor scene from sparse lidar data and single color image.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 3313–3322, 2019.
  * Rho et al. [2020] Kyeongha Rho, Jinsung Ha, and Youngjung Kim.  GuideFormer: Transformers for Image Guided Depth Completion.  page 10, 2020.
  * Ronneberger et al. [2015] Olaf Ronneberger, Philipp Fischer, and Thomas Brox.  U-net: Convolutional networks for biomedical image segmentation.  In _Medical Image Computing and Computer-Assisted Intervention–MICCAI 2015: 18th International Conference, Munich, Germany, October 5-9, 2015, Proceedings, Part III 18_ , pages 234–241. Springer, 2015.
  * Rublee et al. [2011] Ethan Rublee, Vincent Rabaud, Kurt Konolige, and Gary Bradski.  Orb: An efficient alternative to sift or surf.  In _2011 International conference on computer vision_ , pages 2564–2571. Ieee, 2011.
  * Shao et al. [2023] Shuwei Shao, Zhongcai Pei, Weihai Chen, Ran Li, Zhong Liu, and Zhengguo Li.  Urcdc-depth: Uncertainty rectified cross-distillation with cutflip for monocular depth estimation.  _arXiv preprint arXiv:2302.08149_ , 2023.
  * Song et al. [2015] Shuran Song, Samuel P Lichtenberg, and Jianxiong Xiao.  Sun rgb-d: A rgb-d scene understanding benchmark suite.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 567–576, 2015.
  * Tang et al. [2019] Jie Tang, Fei-Peng Tian, Wei Feng, Jian Li, and Ping Tan.  Learning Guided Convolutional Network for Depth Completion, 2019\.
  * Uhrig et al. [2017] Jonas Uhrig, Nick Schneider, Lukas Schneider, Uwe Franke, Thomas Brox, and Andreas Geiger.  Sparsity invariant cnns.  In _2017 international conference on 3D Vision (3DV)_ , pages 11–20. IEEE, 2017.
  * Wang et al. [2021] Wenhai Wang, Enze Xie, Xiang Li, Deng-Ping Fan, Kaitao Song, Ding Liang, Tong Lu, Ping Luo, and Ling Shao.  Pyramid vision transformer: A versatile backbone for dense prediction without convolutions.  In _Proceedings of the IEEE/CVF international conference on computer vision_ , pages 568–578, 2021.
  * Wang et al. [2022] Wenhai Wang, Enze Xie, Xiang Li, Deng-Ping Fan, Kaitao Song, Ding Liang, Tong Lu, Ping Luo, and Ling Shao.  Pvtv2: Improved baselines with pyramid vision transformer.  _Computational Visual Media_ , 8(3):1–10, 2022\.
  * Wang et al. [2023] Yufei Wang, Bo Li, Ge Zhang, Qi Liu, Tao Gao, and Yuchao Dai.  Lrru: Long-short range recurrent updating networks for depth completion.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 9422–9432, 2023.
  * Woo et al. [2023] Sanghyun Woo, Shoubhik Debnath, Ronghang Hu, Xinlei Chen, Zhuang Liu, In So Kweon, and Saining Xie.  Convnext v2: Co-designing and scaling convnets with masked autoencoders.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 16133–16142, 2023.
  * Wu et al. [2022] Yuqun Wu, Jae Yong Lee, and Derek Hoiem.  Sparse spn: Depth completion from sparse keypoints.  _arXiv preprint arXiv:2212.00987_ , 2022.
  * Xu et al. [2020] Jin Xu, Zishan Li, Bowen Du, Miaomiao Zhang, and Jing Liu.  Reluplex made more practical: Leaky relu.  In _2020 IEEE Symposium on Computers and communications (ISCC)_ , pages 1–7. IEEE, 2020.
  * Xu et al. [2019] Yan Xu, Xinge Zhu, Jianping Shi, Guofeng Zhang, Hujun Bao, and Hongsheng Li.  Depth completion from sparse lidar data with depth-normal constraints.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 2811–2820, 2019.
  * Yan et al. [2022] Zhiqiang Yan, Kun Wang, Xiang Li, Zhenyu Zhang, Jun Li, and Jian Yang.  RigNet: Repetitive Image Guided Network for Depth Completion, 2022.
  * Youmin et al. [2023] Zhang Youmin, Guo Xianda, Poggi Matteo, Zhu Zheng, Huang Guan, and Mattoccia Stefano.  CompletionFormer: Depth Completion with Convolutions and Vision Transformers, 2023.
  * Yu et al. [2019] Jiahui Yu, Zhe Lin, Jimei Yang, Xiaohui Shen, Xin Lu, and Thomas S Huang.  Free-form image inpainting with gated convolution.  In _Proceedings of the IEEE/CVF international conference on computer vision_ , pages 4471–4480, 2019.
  * Yuen et al. [2011] Steve Chi-Yin Yuen, Gallayanee Yaoyuneyong, and Erik Johnson.  Augmented reality: An overview and five directions for ar in education.  _Journal of Educational Technology Development and Exchange (JETDE)_ , 4(1):11, 2011.
  * Yurtsever et al. [2020] Ekim Yurtsever, Jacob Lambert, Alexander Carballo, and Kazuya Takeda.  A survey of autonomous driving: Common practices and emerging technologies.  _IEEE access_ , 8:58443–58469, 2020.
  * Zhang [2012] Zhengyou Zhang.  Microsoft kinect sensor and its effect.  _IEEE multimedia_ , 19(2):4–10, 2012.
  * Zhao et al. [2021] Shanshan Zhao, Mingming Gong, Huan Fu, and Dacheng Tao.  Adaptive Context-Aware Multi-Modal Network for Depth Completion.  _IEEE Transactions on Image Processing_ , 30:5264–5276, 2021.
  * Zhou et al. [2023] Wending Zhou, Xu Yan, Yinghong Liao, Yuankai Lin, Jin Huang, Gangming Zhao, Shuguang Cui, and Zhen Li.  BEV@ DC: Bird’s-Eye View Assisted Training for Depth Completion.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 9233–9242, 2023.

Supplementary Material

Figure 11: Qualitative results on NYU Depth V2. Comparing completion results
with different patterns of inputs using different methods trained in fixed
distribution.

##  6 Appendix

In the following, we show additional experimental results.

###  6.1 More results on NYU Depth V2

We trained the comparison methods using their released code and the same
training strategy with us (described in Sec. 4.1), because the existing
methods trained on fixed spatial patterns (500 random samples) are not
comparable in the face of some challenging cases. To prove this point, we
report the results of their pre-trained models in Tab. 5.

Model | Samples |  RMSE(m)$\downarrow$ |  REL$\downarrow$
---|---|---|---
NLSPN[34] | 500 | 0.0917 | 0.0116
CompletionFormer[52] | 500 | 0.0901 | 0.0119
Ours | 500 | 0.0976 | 0.0128
NLSPN[34] | 5 | 1.0389 | 0.2647
CompletionFormer[52] | 5 | 1.1405 | 0.3073
Ours | 5 | 0.3816 | 0.0877
NLSPN[34] | ORB keypoint | 0.5997 | 0.1462
CompletionFormer[52] | ORB keypoint | 0.6222 | 0.2222
Ours | ORB keypoint | 0.2441 | 0.0541
Table 5: Results on NYU Depth V2. Quantitative comparisons with the state-of-
the-art methods trained in fixed distribution. The best results are red and in
bold.

The visualization results in Fig. 11 also illustrate that the existing methods
cannot reconstruct the overall scene structure when dealing with such
challenging cases.

Furthermore, we validated the performance of our method with a fixed
distribution. We used 500 sampling random depth points as inputs and retrained
our model, like the previous setting. Even though SparseDC is focused on depth
completion from sparse and non-uniform inputs, our method still exhibits
comparable performance as shown in the Tab. 6.

Methods | Params |  RMSE(m)$\downarrow$ |  REL$\downarrow$
---|---|---|---
GuideNet[42] | 74M | 0.101 | 0.015
DySPN[22] | 26M | 0.090 | 0.012
NLSPN[34] | 25.8M | 0.092 | 0.012
GraphCSPN[25] | 26M | 0.090 | 0.012
CompletionFormer[52] | 45M | 0.090 | 0.012
Ours | 38.2M | 0.093 | 0.012
Table 6: Results on NYU Depth V2 with fixed distribution. Quantitative
comparisons with state-of-the-art methods in a fixed spatial pattern.

Finally, we report the performance of comparison methods using other metrics
in Tab. 7, and show more visualizations in Fig. 12.

Methods | Samples |  RMSE (m)$\downarrow$ |  MAE (m)$\downarrow$ |  IRMSE ($\frac{1}{m}$)$\downarrow$ |  iMAE ($\frac{1}{m}$)$\downarrow$ |  REL $\downarrow$ |  $\delta_{1.25}$ $\uparrow$ |  $\delta_{1.25}^{2}$ $\uparrow$ |  $\delta_{1.25}^{3}$ $\uparrow$
---|---|---|---|---|---|---|---|---|---
NLSPN[34] | 500 | 0.0984 | 0.0395 | 0.0153 | 0.0058 | 0.0136 | 0.9949 | 0.9992 | 0.9998
GraphCSPN[25] | 500 | 0.0990 | 0.0399 | 0.0153 | 0.0059 | 0.0136 | 0.9948 | 0.9992 | 0.9998
CompletionFormer[52] | 500 | 0.0984 | 0.0392 | 0.0153 | 0.0058 | 0.0135 | 0.9948 | 0.9992 | 0.9998
Ours | 500 | 0.0976 | 0.0379 | 0.0148 | 0.0054 | 0.0128 | 0.9949 | 0.9992 | 0.9998
NLSPN[34] | 200 | 0.1336 | 0.0579 | 0.0209 | 0.0087 | 0.0201 | 0.9902 | 0.9983 | 0.9996
GraphCSPN[25] | 200 | 0.1338 | 0.0575 | 0.0207 | 0.0085 | 0.0197 | 0.9902 | 0.9982 | 0.9995
CompletionFormer[52] | 200 | 0.1349 | 0.0588 | 0.0214 | 0.0089 | 0.0205 | 0.9898 | 0.9982 | 0.9995
Ours | 200 | 0.1346 | 0,0565 | 0.0203 | 0.0081 | 0.0191 | 0.9899 | 0.9982 | 0,9995
NLSPN[34] | 50 | 0.2123 | 0.1078 | 0.0345 | 0.0168 | 0.0382 | 0.9743 | 0.9948 | 0.9986
GraphCSPN[25] | 50 | 0.2119 | 0.1069 | 0.0340 | 0.0165 | 0.0375 | 0.9745 | 0.9942 | 0.9984
CompletionFormer[52] | 50 | 0.2183 | 0.1124 | 0.0362 | 0.0178 | 0.0405 | 0.9717 | 0.9941 | 0.9984
Ours | 50 | 0.2079 | 0.1017 | 0.0323 | 0.0152 | 0.0350 | 0.9755 | 0.9946 | 0.9985
NLSPN[34] | 10 | 0.3639 | 0.2254 | 0.0624 | 0.0373 | 0.0827 | 0.9154 | 0.9810 | 0.9950
GraphCSPN[25] | 10 | 0.3650 | 0.2237 | 0.0639 | 0.0373 | 0.0832 | 0.9138 | 0.9796 | 0.9943
CompletionFormer[52] | 10 | 0.3642 | 0.2300 | 0.0650 | 0.0387 | 0.0874 | 0.9110 | 0.9802 | 0.9946
Ours | 10 | 0.3226 | 0.1923 | 0.0536 | 0.0308 | 0.0681 | 0.9358 | 0.9854 | 0.9960
NLSPN[34] | 5 | 0.4439 | 0.2953 | 0.0770 | 0.0497 | 0.1094 | 0.8674 | 0.9696 | 0.9925
GraphCSPN[25] | 5 | 0.4555 | 0.3001 | 0.0814 | 0.0512 | 0.1151 | 0.8604 | 0.9650 | 0.9899
CompletionFormer[52] | 5 | 0.4377 | 0.2961 | 0.0791 | 0.0507 | 0.1144 | 0.8657 | 0.9683 | 0.9917
Ours | 5 | 0.3816 | 0.2456 | 0.0649 | 0.0405 | 0.0877 | 0.9059 | 0.9795 | 0.9945
NLSPN[34] | shift_grid | 0.4263 | 0.2718 | 0.0791 | 0.0487 | 0.1043 | 0.8655 | 0.9671 | 0.9923
GraphCSPN[25] | shift_grid | 0.4485 | 0.2808 | 0.0833 | 0.0502 | 0.1083 | 0.8531 | 0.9614 | 0.9899
CompletionFormer[52] | shift_grid | 0.4313 | 0.2812 | 0.0834 | 0.0512 | 0.1149 | 0.8560 | 0.9635 | 0.9901
Ours | shift_grid | 0.3736 | 0.2331 | 0.0680 | 0.0409 | 0.0852 | 0.8987 | 0.9782 | 0.9947
NLSPN[34] | uneven density | 0.3931 | 0.2438 | 0.0710 | 0.0425 | 0.0909 | 0.8901 | 0.9741 | 0.9935
GraphCSPN[25] | uneven density | 0.3991 | 0.2420 | 0.0737 | 0.0427 | 0.0920 | 0.8855 | 0.9705 | 0.9922
CompletionFormer[52] | uneven density | 0.3919 | 0.2432 | 0.0724 | 0.0424 | 0.0961 | 0.8867 | 0.9719 | 0.9921
Ours | uneven density | 0.3543 | 0.2141 | 0.0632 | 0.0372 | 0.0763 | 0.9121 | 0.9809 | 0.9950
NLSPN[34] | ORB keypoint | 0.2590 | 0.1565 | 0.0542 | 0.0301 | 0.0625 | 0.9479 | 0.9890 | 0.9965
GraphCSPN[25] | ORB keypoint | 0.2571 | 0.1510 | 0.0540 | 0.0290 | 0.0603 | 0.9481 | 0.9880 | 0.9961
CompletionFormer[52] | ORB keypoint | 0.2660 | 0.1616 | 0.0576 | 0.0316 | 0.0672 | 0.9417 | 0.9873 | 0.9965
Ours | ORB keypoint | 0.2441 | 0.1404 | 0.0498 | 0.0265 | 0.0541 | 0.9539 | 0.9898 | 0.9970
NLSPN[34] | big holes | 0.2368 | 0.1205 | 0.0332 | 0.0171 | 0.0402 | 0.9695 | 0.9944 | 0.9986
GraphCSPN[25] | big holes | 0.2443 | 0.1224 | 0.0336 | 0.0171 | 0.0403 | 0.9675 | 0.9932 | 0.9983
CompletionFormer[52] | big holes | 0.2493 | 0.1273 | 0.0350 | 0.0180 | 0.0434 | 0.9648 | 0.9933 | 0.9983
Ours | big holes | 0.2262 | 0.1121 | 0.0307 | 0.0155 | 0.0362 | 0.9731 | 0.9944 | 0.9985
Table 7: Results on NYU Depth V2 in all Metrics. Quantitative comparisons with
state-of-the-art methods in different spatial patterns. The best results are
red and in bold.

###  6.2 More results on KITTI

Fig. 13 shows, on an image of the KITTI DC dataset and three different depth
inputs, the outcome of SparseDC. Our method can effectively maintain the
overall scene structure under different lines of depth inputs.

###  6.3 More results on SUN RGB-D

SUN RGB-D[41] contains 10,355 RGB-D images captured by four different sensors.
It was used to validate the generalization ability of our method. We used the
noisy and incomplete raw depth data as inputs without sampling, and used the
reﬁned depth maps based on multiple frames as the ground truths for
evaluation. The input images were resized to 320$\times$240 and randomly
cropped to 304$\times$228\. The Qualitative comparison results are shown in
Fig. 14. For such challenging cases (noisy, incomplete, unseen), existing
methods with or without augment-based strategy are difficult to obtain
satisfactory results, whereas our method achieved the best performance due to
its meticulous design.

Figure 12: Qualitative results on NYU Depth V2. Examples of using SparseDC to
complete sparse depth with different patterns. Figure 13: Qualitative results
on Kitti. We report completed results from SparseDC with 4, 16, and 64 line
depth inputs, respectively. Figure 14: Qualitative results on SUNRGBD. $*$
denotes trained model using our training strategy, without $*$ is their
released pretrained model trained using fixed patterns.