11institutetext: LIPN - CNRS UMR 7030, Université Sorbonne Paris Nord, France
22institutetext: LaMSN - La Maison des Sciences Numériques,
France33institutetext: LIMOS - CNRS UMR 6158, Université Clermont Auvergne,
France 33email: {Firstname.Lastname}@{sorbonne-paris-nord,uca}.fr

# Inductive Semi-supervised Learning Through Optimal Transport

Mourad El Hamri 1122 Younès Bennani 1122 Issam Falih 2233

###### Abstract

In this paper, we tackle the inductive semi-supervised learning problem that
aims to obtain label predictions for out-of-sample data. The proposed
approach, called Optimal Transport Induction (OTI), extends efficiently an
optimal transport based transductive algorithm (OTP) to inductive tasks for
both binary and multi-class settings. A series of experiments are conducted on
several datasets in order to compare the proposed approach with state-of-the-
art methods. Experiments demonstrate the effectiveness of our approach. We
make our code publicly available.111Code is available at:
https://github.com/MouradElHamri/OTI

###### Keywords:

Optimal transport Semi-supervised learning Label propagation.

##  1 Introduction

Supervised learning models, especially deep neural networks can achieve human
or superhuman level performances on a broad spectrum of learning tasks. This
big achievement, however, heavily relies on the availability of large-scale
labeled datasets, which usually come at a significant cost, since labeling
data often requires an extensive human labor. For instance, the work needed
for labeling manually sequential data (e.g. speech or video) is proportional
to their lengths. Moreover, some specific domain knowledge (e.g. medicine) is
often critical for labeling and requires an involvement of experts. This lack
of labeled data is often accompanied by an abundance of unlabeled instances,
which raises the question of how to take advantage of the big amount of
unlabeled samples to alleviate the need for large high quality labeled
datasets.

Semi-supervised learning (SSL) [2] has emerged as one of the most promising
paradigms to achieve this goal by leveraging both labeled and unlabeled data.
There are two categories of semi-supervised learning models [13]: inductive
learning models and transductive learning models. Inductive semi-supervised
learning aims to learn using both labeled and unlabeled data a predictive
model capable of making predictions both for unlabeled data already
encountered in the training phase and for out-of-sample data, i.e. previously
unseen instances. While transductive semi-supervised learning methods are only
concerned with predicted the labels of unlabeled data already presented during
the training.

Most of transductive learning models are graph-based approaches [12] (e.g.
label propagation methods), that aim to present each data, labeled or not by a
vertex in the graph, then, similarities between vertices are evaluated to
determine whether or not there should be an edge between each pair of
vertices, finally, edges are weighted to reflect the similarity degree between
vertices. After the graph construction, labels are inferred in several ways
(e.g. in the case of label propagation, labels are diffused from labeled to
unlabeled data through the graph edges). The common major inconvenience of
transductive methods is their inability to predict labels for out-of-sample
data, so when some previously unseen test data arrive, transductive learning
methods need to fusion these new samples into the previous data in our
disposal to reconstruct a new augmented graph based on merged data, and then
perform label propagation from scratch. This process is too costly, since the
presentation of even, a single new point require to rerun all the process
already done in their entirety, which is distasteful in many real-world
applications, where on-the-fly prediction for previously unseen instances is
highly requested.

Several works have attempted to bridge the gap between transductive and
inductive semi-supervised learning, by proposing to derive an inductive
function for out-of-sample data from the transductive predictions [4] [1].
However, these approaches cannot be generalized to all the transductive
methods, such as, Optimal Transport Propagation (OTP) [6]. This inability to
generalize is mainly due to the difference of this approach from the state-of-
the-art methods, since instead of a complete graph, OTP relies on a bipartite
graph, which modifies its objective function, that does not follow the
standard form of objective functions composed usually of two terms, a first
term to penalize predicted labels that do not match the true labels and
another term to penalize the difference in labels between similar samples.
Such a setting motivated the development of a simple way to extend OTP (and
other transductive semi-supervised approaches based on bipartite graphs) to
the inductive framework.

In this paper, we derive an inductive approach called Optimal Transport
Induction (OTI) from the modified objective function of OTP. In addition to
being applicable in the case of binary classification, OTI can be efficiently
extended to multi-class settings.

The rest of this paper is organized as follows: in Section 2, we present an
overview of optimal transport [14] [11]. Section 3 details the proposed
optimal transport induction method (OTI) and in Section 4, we provide
comparisons to state-of-the-art methods on several benchmark datasets.

##  2 Optimal Transport

The birth of optimal transport is dated back to 1781, with the following
problem introduced by Gaspard Monge [9]: Let $(\mathcal{X},\mu)$ and
$(\mathcal{Y},\nu)$ be two probability spaces and
$c:\mathcal{X}\times\mathcal{Y}\to\mathbb{R}^{+}$ a measurable cost function,
the problem of Monge aims at finding the transport map
$\mathcal{T}:\mathcal{X}\to\mathcal{Y}$, that transport the mass represented
by the measure $\mu$ to the mass represented by the measure $\nu$ and which
minimizes the total cost of this transportation, more formally:

$\quad\underset{\mathcal{T}}{\inf}\\{\int_{\mathcal{X}}c(x,\mathcal{T}(x))d\mu(x)|\mathcal{T}\\#\mu=\nu\\},$
(1)

where $\mathcal{T}\\#\mu$ denotes the push-forward operator of $\mu$ through
the map $\mathcal{T}$.

A long period of sleep followed Monge’s formulation until the relaxation of
Leonid Kantorovitch in 1942 [8]. The relaxed formulation of kantorovich, known
as the Monge-Kantorovich problem, can be formulated in the following way:

$\underset{\gamma}{\inf}\\{\,\int_{\mathcal{X}\times\mathcal{Y}}\,c(x,y)\,d\gamma(x,y)\,|\,\gamma\in\Pi(\mu,\nu)\,\\},$
(2)

where $\Pi(\mu,\nu)$ is the set of probability measures over the product space
$\mathcal{X}\times\mathcal{Y}$ such that both marginals of $\gamma$ are $\mu$
and $\nu$.

In several real world application, the access to the measures $\mu$ and $\nu$
is only available through finite samples
$X=(x_{1},...,x_{n})\subset\mathcal{X}$ and
$Y=(y_{1},...,y_{m})\subset\mathcal{Y}$, then, the measures $\mu$ and $\nu$
can be casted as the following discrete measures,
$\mu=\sum_{i=1}^{n}a_{i}\delta_{x_{i}}$ and
$\nu=\sum_{j=1}^{m}b_{j}\delta_{y_{j}}$, where $a\in\sum_{n}$ and
$b\in\sum_{m}$ are probability vectors of size $n$ and $m$ respectively. The
relaxation of Kantorovich becomes then the following linear program [10]:

$\underset{T\in U(a,b)}{\min}\langle{T},{C_{XY}}\rangle_{F}$ (3)

where $U(a,b)=\\{T\in\mathcal{M}_{n\times
m}(\mathbb{R}^{+})\,|\,T1_{m}=a\,\,\text{and}\,\,T^{\mathbf{T}}1_{n}=b\\}$ is
the transportation polytope which acts as a feasible set, $C_{XY}$ is the cost
matrix and $\langle T,C_{XY}\rangle_{F}\,=trace(T^{\mathrm{T}}C_{XY})$ is the
Frobenius dot-product of matrices.

This linear program, can be solved with the simplex algorithm or interior
point methods. However, optimal transport problem scales cubically on the
sample size, which is often too costly in practice, especially for machine
learning applications that involve massive datasets. Entropy-regularization
[3] has emerged as a solution to the computational burden of optimal
transport. The entropy-regularized discrete optimal transport problem reads:

$\underset{T\in
U(a,b)}{\min}\langle{T},{C_{XY}}\rangle_{F}-\varepsilon\mathcal{H}(T)$ (4)

where $\mathcal{H}(T)=-\sum_{i=1}^{n}\sum_{j=1}^{m}t_{ij}(\log(t_{ij})-1)$ is
the entropy of $T$. This regularized problem can be solved efficiently via an
iterative procedure: Sinkhorn-Knopp algorithm.

##  3 Optimal Transport Induction

In this section, we propose an efficient way to extend OTP 222which would be
too lengthy to detail here, please refer directly to this approach in the
following paper [6]. for out-of-sample data. The main underlying idea behind
this extension is the modification of the objective function of OTP, from
which we derive a novel algorithm called Optimal Transport Induction (OTI)
able to predict labels for previously unseen data.

The transductive method OTP can be casted as the minimization of the following
objective function $\mathcal{C}^{transduction}_{\mathcal{W},l_{u}}$ in terms
of the label function values at unlabeled samples $x_{j}\in X_{U}$:

$\mathcal{C}^{transduction}_{\mathcal{W},l_{u}}(f)=\sum_{x_{i}\in
X_{L}}\sum_{x_{j}\in X_{U}}w_{x_{i},x_{j}}l_{u}(y_{i},f(x_{j}))$ (5)

where $l_{u}$ is an unsupervised loss function (e.g. a lower-bounded
dissimilarity function applied on a pair of output values $y_{i}$ and
$f(x_{j})$), and $\mathcal{W}=(w_{x_{i},x_{j}})_{i,j}$ is the affinity matrix
derived from the optimal transport plan between labeled and unlabeled data.
The objective function in Eq(5) is a smoothness criterion that seeks to
penalize differences in the label predictions for similar data points in the
graph, which means that a good classifier should not change too much between
similar samples. The main difference between the objective function of OTP and
traditional transductive approaches is the absence of a second term to
penalize labels that do not match the correct ones, because OTP relies on a
bipartite edge weighted graph instead of a fully connected graph to leave the
labels $Y_{L}$ unchanged during the training. The absence of this term in the
objective function of OTP, makes the use of state-of-the-art induction
formulas unfeasible. which is the main motivation behind the conception of
OTI.

###  3.1 Problem setup

Let $X=\\{x_{1},...,x_{l+u}\\}$ be a set of $l+u$ data points in
$\mathbb{R}^{d}$ and $\mathcal{C}=\\{c_{1},...,c_{K}\\}$ a discrete label set
consisting of $K$ classes. The first $l$ points denoted by
$X_{L}=\\{x_{1},...,x_{l}\\}$ are labeled according to
$Y_{L}=\\{y_{1},...,y_{l}\\}$, where $y_{i}\in\mathcal{C}$ for every
$i\in\\{1,...,l\\}$, and the remaining data points denoted by
$X_{U}=\\{x_{l+1},...,x_{l+u}\\}$ are labeled using OTP according to
$Y_{U}=\\{y_{l+1},...,y_{l+u}\\}$. For inductive classification, unseen data
in the training phase are denoted by $X_{new}$. The aim of OTI is to predict
labels of $X_{new}$ using $X=X_{L}\cup X_{U}$ and $Y=Y_{L}\cup Y_{U}$, without
being obliged to perform OTP from scratch.

###  3.2 Induction formula: OTI

In order to extend the transductive algorithm OTP into function induction for
out-of-sample data points, the same smoothness criterion as in Eq(5) will be
used for new test instances $x_{new}\in X_{new}$ [1], and then we can minimize
the modified objective function with respect to only the predicted labels
$\tilde{f}(x_{new})$. The novel smoothness criterion for new test samples
$x_{new}$ becomes then:

$\mathcal{C}^{induction}_{\mathcal{W},l_{u}}(\tilde{f}(x_{new}))=\sum_{x_{i}\in
X_{L}\cup X_{U}}w_{x_{i},x_{new}}l_{u}(y_{i},\tilde{f}(x_{new}))$ (6)

If the loss function $l_{u}$ is convex, e.g.,
$l_{u}=(y_{i}-\tilde{f}(x_{new}))^{2}$, then the cost function
$\mathcal{C}^{induction}_{\mathcal{W},l_{u}}$ is also convex in
$\tilde{f}(x_{new})$. Thus the label assignment $\tilde{f}(x_{new})$
minimizing $\mathcal{C}^{induction}_{\mathcal{W},l_{u}}$ is given by:

$\tilde{f}(x_{new})=\frac{\sum_{x_{i}\in X_{L}\cup
X_{U}}w_{x_{i},x_{new}}y_{i}}{\sum_{x_{i}\in X_{L}\cup
X_{U}}w_{x_{i},x_{new}}}$ (7)

The similarities $w_{x_{i},x_{new}}$ are directly inferred using optimal
transport as follows:

$w_{x_{i},x_{new}}=\frac{\gamma^{*}_{\varepsilon_{x_{i},x_{new}}}}{\sum_{i}\gamma^{*}_{\varepsilon_{x_{i},x_{new}}}}\,\,\,\forall
x_{i}\in X=X_{L}\cup X_{U},$ (8)

where $\gamma^{*}_{\varepsilon}$ is the new optimal transport plan between the
empirical distributions of $X=X_{L}\cup X_{U}$ and $X_{new}$ defined by:

$\gamma^{*}_{\varepsilon}=\underset{\gamma\in
U(a,b)}{argmin}\,\,\langle{\gamma},{C}\rangle_{F}-\varepsilon\mathcal{H}(\gamma),$

and $C$ is the cost matrix between $X=X_{L}\cup X_{U}$ and $X_{new}$:

$C=[c_{x_{i},x_{new}}]\,\text{defined by}\,\,c_{x_{i},x_{new}}=\lVert
x_{i}-x_{new}\rVert^{2},\,\forall(x_{i},x_{new})\in X\times X_{new}$

###  3.3 Binary classification and multi-class settings

In binary classification, where $\mathcal{C}=\\{+1,-1\\}$, the classification
problem in Eq(7) can be casted as a regression problem, in the following way:

$\begin{cases}y_{new}=+1\,\,\,\text{if}\,\,sign(\tilde{f}(x_{new}))\geq 0\\\
y_{new}=-1\,\,\,\text{otherwise}\end{cases},$ (9)

Most of transductive approaches possess the ability to handle multiple
classes, while the inductive methods are usually limited to the binary
classification framework, where $\mathcal{C}=\\{+1,-1\\}$ as in Eq(9).
However, our proposed approach OTI can be adapted accurately for multi-class
settings, as follows: the label $\tilde{f}(x_{new})$ is given by a weighted
majority vote of the training samples in $X=X_{L}\cup X_{U}$:

$\tilde{f}(x_{new})=\underset{c_{k}\in\mathcal{C}}{argmax}\sum_{x_{i}\in
X_{L}\cup X_{U}/y_{i}=c_{k}}w_{x_{i},x_{new}}$ (10)

The predicted class of $x_{new}$ is then the class whose representatives have
the highest similarity with $x_{new}$.

Eq(9) can be seen as a special case of Eq(10) in the binary classification
settings, in fact, if $\mathcal{C}=\\{+1,-1\\}$, then choosing between the
class that maximizes $\sum_{x_{i}\in X_{L}\cup
X_{U}/y_{i}=c_{k}}w_{x_{i},x_{new}}$ is equivalent to choosing according to
the sign of $\tilde{f}(x_{new})$, since the term $\sum_{x_{i}\in X_{L}\cup
X_{U}}w_{x_{i},x_{new}}$ in $\tilde{f}(x_{new})$ is always positive.

The proposed algorithm OTI, is formally summarized in algorithm 1, where we
use the algorithm OTP for training and the induction formula for testing.

Input : $X_{new},X_{L},X_{U},Y_{L}$

Parameters : $\varepsilon$

(1) Training phase

Compute $Y_{U}$ by OTP

(2) Testing phase

for  _a new test point $x_{new}\in X_{new}$_ do

Compute $w_{x_{i},x_{new}}\,\,\,\forall x_{i}\in X=X_{L}\cup X_{U}$ by Eq(8)

Compute the label $\tilde{f}(x_{new})$ by Eq(9) or Eq(10)

end for

return _$\tilde{f}(x_{new})$_

Algorithm 1 OTI

##  4 Experiments

###  4.1 Experimental protocol

In this section, we provide empirical experimentation for the proposed
algorithm OTI. To thoroughly evaluate the performance of the proposed
approach, a total of five benchmark datasets (Iris, Ionosphere, Dermatology,
Digits, MNIST) have been employed for experimental studies 333The datasets are
publicly available at: https://archive.ics.uci.edu/ml/datasets.php. The
performance of OTI is compared with four state-of-the-art methods, including
three transductive and one inductive semi-supervised learning approaches: LP
[16], LNP [15], OTP [6] and SSI [4]. To evaluate the performance of our
approach, two widely-used evaluation measures were employed: the normalized
mutual information (NMI) [5], and the adjusted rand index (ARI) [7]. For each
dataset, we randomly sample $\zeta\times 100\%$ instances to form the labeled
set $X_{L}$. For the remaining examples, $40\%$ of them are randomly sampled
to form the unlabeled set $X_{U}$ and $60\%$ of them are randomly sampled to
form the out-of-sample data $X_{new}$. In the experiments, the training and
testing procedures are conducted for the compared algorithms as follows: for
the inductive semi-supervised learning methods OTI and SSI, both of them are
trained on $X=X_{L}\cup X_{U}$ and tested on $X_{new}$. The three transductive
semi-supervised learning methods LP, LNP and OTP can only make predictions on
unlabeled samples already encountered in the training, so, to ensure that all
the comparing algorithms are evaluated on the same out-of-sample set, the
three algorithms are trained on $\tilde{X}=X_{L}\cup X_{U}\cup X_{new}$ and
the evaluation is restricted to $X_{new}$. The sampling rate $\zeta$ for
labeled data is varied from 5% to 25% with a step-size of 10%. Under each
sampling rate, all the four compared algorithms were run with ten different
random sampling, the mean performance out of the ten runs is recorded for the
four algorithms.

###  4.2 Results

Table 1 reports the detailed experimental results of each algorithm on all the
datasets in terms of ARI and NMI.

Table 1: Inductive performances in terms of ARI and NMI | ARI | NMI
---|---|---
Datasets | $\zeta$ | LP | LNP | OTP | SSI | OTI | LP | LNP | OTP | SSI | OTI
| 5 | 0.767 | 0.728 | 0.772 | 0.720 | 0.753 | 0.743 | 0.682 | 0.749 | 0.691 | 0.723
Iris | 15 | 0.860 | 0.804 | 0.889 | 0.796 | 0.862 | 0.863 | 0.774 | 0.863 | 0.772 | 0.818
| 25 | 0.880 | 0.857 | 0.917 | 0.845 | 0.897 | 0.874 | 0.849 | 0.882 | 0.847 | 0.869
| 5 | 0.328 | 0.307 | 0.527 | 0.298 | 0.502 | 0.293 | 0.262 | 0.383 | 0.263 | 0.326
Ionosphere | 15 | 0.449 | 0.419 | 0.595 | 0.408 | 0.572 | 0.373 | 0.347 | 0.483 | 0.342 | 0.424
| 25 | 0.483 | 0.462 | 0.618 | 0.460 | 0.591 | 0.404 | 0.375 | 0.530 | 0.362 | 0.445
| 5 | 0.806 | 0.799 | 0.850 | 0.789 | 0.837 | 0.814 | 0.795 | 0.835 | 0.783 | 0.819
Dermatology | 15 | 0.902 | 0.869 | 0.922 | 0.862 | 0.901 | 0.893 | 0.854 | 0.914 | 0.853 | 0.889
| 25 | 0.918 | 0.894 | 0.945 | 0.892 | 0.919 | 0.918 | 0.882 | 0.921 | 0.869 | 0.908
| 5 | 0.842 | 0.820 | 0.887 | 0.827 | 0.868 | 0.851 | 0.803 | 0.872 | 0.809 | 0.853
Digits | 15 | 0.931 | 0.915 | 0.952 | 0.925 | 0.930 | 0.937 | 0.901 | 0.950 | 0.892 | 0.930
| 25 | 0.969 | 0.958 | 0.980 | 0.960 | 0.968 | 0.962 | 0.949 | 0.968 | 0.928 | 0.947
| 5 | 0.774 | 0.737 | 0.795 | 0.730 | 0.772 | 0.761 | 0.726 | 0.777 | 0.719 | 0.759
MNIST | 15 | 0.811 | 0.784 | 0.855 | 0.783 | 0.836 | 0.826 | 0.793 | 0.830 | 0.782 | 0.812
| 25 | 0.863 | 0.839 | 0.884 | 0.841 | 0.859 | 0.850 | 0.814 | 0.861 | 0.810 | 0.849
Average | 0.772 | 0.746 | 0.826 | 0.748 | 0.804 | 0.758 | 0.720 | 0.788 | 0.715 | 0.758

The first remark that can be made from this table is that the performance of
each algorithm on all the datasets grows in parallel with the growth of the
sampling rate $\zeta$, which is quite normal, because the amount of the
available labeled data is an important factor to improve the performance of
semi-supervised models, the more labeled data is available, the better the
model is performing. The table also shows that our method outperforms the
other inductive model SSI, in all the datasets and for all the sampling rates
$\zeta$, this achievement can be explained by the same reason why OTP
outperforms LP, i.e. the use of optimal transport to calculate similarities
between the data instead of the pairwise distances which only capture
information on a bilateral level. However, the performance of OTI is inferior
to that of OTP, which is reasonable because, on the one hand, in order to
allow to OTP to perform inductive tasks, the out-of-sample set $X_{new}$ was
merged with the sets $X_{L}$ and $X_{U}$ in the training, while only
predictions on $X_{new}$ were retained to evaluate its performance, and on the
other hand, OTI relies on OTP in the training, so if OTP makes errors in its
predictions on $Y_{U}$, this will directly affect the performance of OTI on
$X_{new}$. The same analysis can be extended to the case of SSI, that uses LP
in its training, which explains the better performance of the latter. The
difference in the average performances for both NMI and ARI, between OTP and
OTI, is less than the difference between LP and SSI, demonstrating that using
optimal transport to extend the label function to out-of-sample data can
significantly improve the accuracy of the predictions.

##  5 Conclusion

In this paper, the problem of inductive semi-supervised learning is addressed.
A new approach, named Optimal Transport Induction, is proposed to extend OTP
from transductive parameters to inductive ones. A modification in the
objective function of OTP was considered, from which an efficient algorithm
OTI was derived to allow us to extend predictions to previously unseen data,
by relying on optimal transport to compute the similarity between the out-of-
sample data and the data already encountered in the training phase.
Experimental studies have been conducted to show the effectiveness of our
approach.

## References

  * [1] Yoshua Bengio, Olivier Delalleau, and Nicolas Le Roux.  Label propagation and quadratic criterion.  2006\.
  * [2] Olivier Chapelle, Bernhard Scholkopf, and Alexander Zien.  Semi-supervised learning (chapelle, o. et al., eds.; 2006).  IEEE Transactions on Neural Networks, 2009.
  * [3] Marco Cuturi.  Sinkhorn distances: Lightspeed computation of optimal transport.  In Advances in neural information processing systems, pages 2292–2300, 2013.
  * [4] Olivier Delalleau, Yoshua Bengio, and Nicolas Le Roux.  Efficient non-parametric function induction in semi-supervised learning.  PMLR, 2005.
  * [5] Byron E Dom.  An information-theoretic external cluster-validity measure.  2012\.
  * [6] Mourad El Hamri, Younés Bennani, and Issam Falih.  Label propagation through optimal transport.  In 2021 International Joint Conference on Neural Networks.
  * [7] Lawrence Hubert and Phipps Arabie.  Comparing partitions.  Journal of classification, 2(1):193–218, 1985.
  * [8] Leonid Vitalievich Kantorovich.  On the translocation of masses.  In Dokl. Akad. Nauk. USSR (NS), volume 37, pages 199–201, 1942\.
  * [9] Gaspard Monge.  Mémoire sur la théorie des déblais et des remblais.  Histoire de l’Académie Royale des Sciences de Paris, 1781.
  * [10] Gabriel Peyré, Marco Cuturi, et al.  Computational optimal transport: With applications to data science.  Foundations and Trends® in Machine Learning, 2019\.
  * [11] Filippo Santambrogio.  Optimal transport for applied mathematicians.  Birkäuser, NY, 55(58-63):94, 2015.
  * [12] Amarnag Subramanya and Partha Pratim Talukdar.  Graph-based semi-supervised learning.  Synthesis Lectures on Artificial Intelligence and Machine Learning, 2014.
  * [13] Jesper E Van Engelen and Holger H Hoos.  A survey on semi-supervised learning.  Machine Learning, 109(2):373–440, 2020.
  * [14] Cédric Villani.  Optimal transport: old and new, volume 338.  Springer Science & Business Media, 2008.
  * [15] Fei Wang and Changshui Zhang.  Label propagation through linear neighborhoods.  IEEE Transactions on Knowledge and Data Engineering, 20(1):55–67, 2007.
  * [16] Xiaojin Zhu and Zoubin Ghahramani.  Learning from labeled and unlabeled data with label propagation.  2002\.


# Bridging Stereo Geometry and BEV Representation with
Reliable Mutual Interaction for Semantic Scene Completion

Bohan Li1,2 Yasheng SUN 3 Zhujin Liang 4 Dalong Du 4
Zhuanghui Zhang 4 Xiaofeng Wang 5 Yunnan Wang 1,2 Xin Jin 2,∗ &Wenjun Zeng 2
1Shanghai Jiao Tong University, 2 Eastern Institute of Technology,
3Tokyo Institute of Technology, 4PhiGent Robotics, 5University of Chinese
Academy of Sciences
<EMAIL_ADDRESS><EMAIL_ADDRESS>
{zhujin.liang, dalong.du<EMAIL_ADDRESS>
<EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

3D semantic scene completion (SSC) is an ill-posed perception task that
requires inferring a dense 3D scene from limited observations. Previous
camera-based methods struggle to predict accurate semantic scenes due to
inherent geometric ambiguity and incomplete observations. In this paper, we
resort to _stereo matching_ technique and _bird’s-eye-view (BEV)_
representation learning to address such issues in SSC. Complementary to each
other, stereo matching mitigates geometric ambiguity with epipolar constraint
while BEV representation enhances the hallucination ability for invisible
regions with global semantic context. However, due to the inherent
representation gap between stereo geometry and BEV features, it is non-trivial
to bridge them for _dense prediction task_ of SSC. Therefore, we further
develop a unified occupancy-based framework dubbed BRGScene, which effectively
bridges these two representations with dense 3D volumes for reliable semantic
scene completion. Specifically, we design a novel Mutual Interactive Ensemble
(MIE) block for pixel-level reliable aggregation of stereo geometry and BEV
features. Within the MIE block, a Bi-directional Reliable Interaction (BRI)
module, enhanced with confidence re-weighting, is employed to encourage fine-
grained interaction through mutual guidance. Besides, a Dual Volume Ensemble
(DVE) module is introduced to facilitate complementary aggregation through
channel-wise recalibration and multi-group voting. Our method outperforms all
published camera-based methods on SemanticKITTI for semantic scene completion.

$\blacksquare$bicycle $\blacksquare$car $\blacksquare$motorcycle
$\blacksquare$truck $\blacksquare$other vehicle $\blacksquare$person
$\blacksquare$bicyclist $\blacksquare$motorcyclist $\blacksquare$road
---
$\blacksquare$parking $\blacksquare$sidewalk $\blacksquare$other ground
$\blacksquare$building $\blacksquare$fence $\blacksquare$vegetation
$\blacksquare$trunk $\blacksquare$terrain $\blacksquare$pole
$\blacksquare$traffic sign

Figure 1:  Overview of the proposed BRGScene. The figure illustrates the
stereo inputs, SSC prediction results and ground truth from left to right. We
can see that our method shows promising performance in completing semantic
scenes, especially for those challenging distant small objects, as indicated
by the car highlighted with a red box.

##  1 Introduction

3D scene understanding is a fundamental task in computer vision Roberts
(1963), facilitating a variety of applications such as autonomous driving,
robotic navigation and augmented reality. Due to the limitations of real-world
sensors such as restricted field of view, measurement noise, or sparse
results, this task remains a challenging problem. To address this problem, 3D
Semantic Scene Completion (SSC) Roldao et al. (2022) is introduced to jointly
predict the geometry and semantic segmentation of a scene. Given its inherent
3D nature, most existing SSC solutions Garbade et al. (2019); Roldao et al.
(2020); Wu et al. (2020) employ 3D geometric signals, in the form of occupancy
grids, point clouds, or distance fields, as their model inputs. Although they
provide insightful geometric cues, it requires costly sensors (e.g. LiDAR)
alongside considerable manual labor entailed in their deployment. Hence, it is
worth exploring an efficient and effective approach for high-fidelity SSC
solely with cost-friendly portable cameras.

However, the absence of explicit 3D geometric information and incomplete
observation pose large challenges to accurate geometry acquisition and
reasonable hallucination in invisible regions Cao and de Charette (2022);
Huang et al. (2023); Li et al. (2023a). Thus, previous camera-based SSC
solutions Cao and de Charette (2022); Huang et al. (2023) tend to utilize
learning-based projection techniques to convert 2D image features into a 3D
dense space, but their predictions inevitably fall short of capturing accurate
geometry without explicit constraints. Later studies Li et al. (2023a) attempt
to introduce depth information to augment query for reliable geometry
prediction. But their results still struggle to hallucinate reasonable
invisible regions without ensembling global semantic context.

As a pivotal technique in 3D vision applications, stereo matching leverages
explicit epipolar constraint to establish pixel-level correspondence, which is
suitable for reconstructing dense 3D scene geometry Guo et al. (2019). On the
other hand, the remarkable global robustness and hallucination ability of
bird’s-eye-view (BEV) representation, coupled with rich context and global
semantic information have led to its widespread utilization in the 3D object
detection community Philion and Fidler (2020); Li et al. (2022b); Roddick and
Cipolla (2020). Inspired by the above two aspects, recent works have begun to
simultaneously use stereo matching and BEV features for 3D perception. For
instance, in the 3D object detection area, BEVStereo Li et al. (2022a) fuses
monocular and temporal stereo depth maps before generating BEV features.
BEVDepth Li et al. (2023b) improves geometry modeling with BVE representations
by introducing extra depth supervision. These strategies significantly enhance
the effectiveness of sparse perception tasks (e.g. 3D object detection) by
focusing on coarse-grained predictions at the region level for common visual
objects.

Despite remarkable advances in 3D object detection, it’s non-trivial in the
semantic scene completion (SSC) task to bridge the representation gap between
stereo geometry and BEV features within a unified framework for pixel-level
reliable prediction. This difficulty arises due to the structural variations
among similar objects and the insufficient region-level coarse information for
pixel-level semantics and geometry in complex real-world scenarios.

Given these concerns, we propose BRGScene, a framework that bridges stereo
matching technique and BEV representation for fine-grained reliable SSC, and
the results are shown in Figure 1. Our framework aims to fully exploit the
potential of vision inputs with explicit geometric constraint of stereo
matching Guo et al. (2019); Li et al. (2021) and global semantic context of
BEV representation Philion and Fidler (2020); Li et al. (2023b). Different
from previous methods that focus on 2D features or depth maps Li et al.
(2023a, b), we propose to employ the dense 3D volumes of stereo and BEV
representations for SSC.

Given the distinct nature of the two volumes, we devise a _Mutual Interactive
Ensemble (MIE)_ block to bridge the gap for fine-grained reliable perception.
Specifically, a _Bi-directional Reliable Interaction (BRI)_ module is designed
to guide each volume to retrieve pixel-level reliable information. A
confidence re-weighting strategy inspired by MVS Chen et al. (2020) is
incorporated on top of the BRI module to further enhance the performance.
Furthermore, a _Dual Volume Ensemble (DVE)_ module is introduced to facilitate
complementary aggregation with channel-wise recalibration and multi-group
feature voting. Our contributions are summarized as follows: 1) We propose a
novel framework that resorts to both stereo and BEV representations with dense
3D volumes for precise geometry modeling and hallucination ability enhancement
in SSC. 2) To bridge the representation gap for fine-grained reliable
perception, a _Mutual Interactive Ensemble_ block is designed to take
advantage of the complementary merits of the volumes in the two
representations. 3) Our proposed BRGScene outperforms state-of-the-art
VoxFormer-T with a 14.5% relative improvement on the SemanticKITTI
leaderboard.

##  2 Related Works

###  2.1 Semantic Scene Completion

Semantic scene completion is a dense 3D perception task that jointly estimates
semantic segmentation and scene completion Behley et al. (2019); Cai et al.
(2021). To provide additional texture or geometry information, some works Cai
et al. (2021); Li et al. (2019) exploit multi-modal inputs, such as RGB images
coupled with geometric cues. Another slew of studies Cao and de Charette
(2022); Li et al. (2023a) aims to achieve semantic scene completion solely
with camera-only inputs. For instance, MonoScene Cao and de Charette (2022)
lifts a monocular image using 2D-3D projections and leverages 2D and 3D UNets
for semantic scene completion. TPVFormer Huang et al. (2023) utilizes a tri-
perspective view representation and attention mechanism for 3D scene
understanding. VoxFormer Li et al. (2023a) employs a transformer-based
framework where a sparse set of depth-based voxel queries are devised for
scene structure reconstruction.

###  2.2 Stereo Matching Based 3D Perception

With the advances of deep convolution neural networks, the quality of depth
predictions from stereo images Poggi et al. (2022) has steadily improved and
led to a remarkable improvement in downstream 3D vision applications such as
object detection, surface reconstruction and augmented reality. GC-Net Kendall
et al. (2017) first proposes to employ 3D CNNs in the stereo matching
framework, where 2D corresponding features are mapped to a 3D cost volume
through a concatenation operation. RAFTStereo Deng (2021) and CREStereoLi et
al. (2021) utilize feature correlation to produce matching cost volume, which
is subsequently optimized through sequential refinement modules for depth
prediction. However, in conditions such as occlusion and large textureless
regions, stereo matching prediction performance drops significantly.

Figure 2:  Overall framework of our proposed BRGScene. Given input stereo
images, we employ 2D UNet to extract image features. The BEV latent volume and
stereo geometric volume are constructed by a _BEV Constructor_ and a _Stereo
Constructor_ , respectively. To bridge their representation gap for fine-
grained reliable perception, a _Mutual Interactive Ensemble_ block is proposed
to take advantage of complementary merits of the volumes.

###  2.3 Bird’s-Eye-View Representation

The bird’s-eye-view is a widely used representation in 3D object detection
since it provides a clear depiction of the layout and strong hallucination
ability from a top-down perspective Philion and Fidler (2020); Hu et al.
(2021). Lift-Splat Philion and Fidler (2020) extracts BEV representations from
an arbitrary number of cameras by implicitly unprojecting 2D visual inputs
based on estimated depth distribution. BEVDepth Li et al. (2023b) leverages a
camera-aware monocular depth estimation module to improve depth perception in
BEV-based 3D detection. However, region-level coarse perception in the object
detection task is not effective enough for the dense prediction task of SSC.
In this work, our objective is to investigate reliable pixel-level prediction
for high-quality semantic scene completion.

##  3 Methodology

We present our proposed BRGScene that aims to jointly infer dense 3D geometry
and semantics solely from RGB images. In this section, we first introduce a
hybrid occupancy-based SSC framework (Sec. 3.1), including problem formulation
and architecture overview. Then we provide detailed construction of _dual
volume_ representations (Sec. 3.2). To bridge their representation gap for
fine-grained reliable perception, we depict our devised ensemble block (Sec.
3.3). Finally, we introduce our SSC generator and training paradigm (Sec.
3.4).

###  3.1 The Proposed Framework of BRGScene

We propose BRGScene, a framework that bridges the stereo matching technique
and the BEV representation for fine-grained reliable SSC.

Problem Formulation. Given a set of stereo RGB images
$I^{rgb}_{Stereo}=\\{I^{rgb}_{l},I^{rgb}_{r}\\}$, our goal is to jointly infer
geometry and semantics of a 3D scene. The scene is represented as a voxel grid
$\textbf{Y}\in\mathbb{R}^{H\times W\times Z}$, where $H,W,Z$ denote the
height, width and depth in 3D space. Regarding each voxel, it will be assigned
to a unique semantic label belonging to $C\in\\{c_{0},c_{1},\cdots,c_{M}\\}$,
which either occupies empty space $c_{0}$ or falls on a specific semantic
class $\\{c_{1},c_{2},\cdots,c_{M}\\}$. Here $M$ denotes the total number of
semantic classes. We would like to learn a transformation
$\hat{\textbf{Y}}=\Theta(I^{rgb}_{Stereo})$ to approach ground truth 3D
semantics Y.

Architecture Overview. The overall architecture of our proposed framework is
illustrated in Figure 2. We follow a common paradigm Cao and de Charette
(2022) that employs successive 2D and 3D UNets as backbones. The input stereo
images $I^{rgb}_{Stereo}$ are separately encoded by a 2D UNet into paired
context-aware features ${\textbf{F}_{l}}$ and ${\textbf{F}_{r}}$. Then we
leverage a _Stereo Constructor_ to convert these features into a dense 3D
volume $\textbf{V}_{Stereo}\in\mathbb{R}^{D\times H\times W}$. In parallel, a
_BEV Constructor_ lift 2D features ${\textbf{F}_{l}}$ of the left image to a
latent BEV volume $\textbf{V}_{BEV}\in\mathbb{R}^{D\times H\times W}$
alongside its context feature $\textbf{C}_{BEV}\in\mathbb{R}^{C_{b}\times
H\times W}$ following standard protocol of Philion and Fidler (2020).
Sequentially, the two-stream built volumes are bridged and aggregated to a new
volume $\textbf{V}_{ens}$ by a _Mutual Interactive Ensemble_ block. Finally,
the context features $\textbf{C}_{BEV}$ splat along volume $\textbf{V}_{ens}$
by outer-product, which will be fed to a 3D UNet for semantic segmentation and
completion.

###  3.2 Dual Volume Construction

Unlike previous studies Li et al. (2023a, b) which focus on 2D
representations, we employ _3D volumetric representation_ to resolve dense
scene understanding. Specially, we introduce hybrid volumetric representations
with stereo and BEV volumes to take full advantage of camera inputs.

2D Feature Extraction Backbone. For image feature extraction, 2D UNet with
pre-trained EfficientNetB7 Tan and Le (2019) is leveraged to separately
process left and right input images. Note that we utilize shared weights to
encourage efficient correspondence feature learning.

Stereo Geometric Volume Constructor. With the obtained unary features
${\textbf{F}_{l}}$ and ${\textbf{F}_{r}}$ from the left and right images,
_Stereo Constructor_ targets to build a stereo depth volume
$\textbf{V}_{Stereo}$ by matching them with epipolar constraint. Specifically,
group-wise correlation Guo et al. (2019) is first adopted to generate
disparity cost volume. Formally,

$D_{gwc}(d,x,y,g)=\frac{1}{N_{c}/N_{g}}\left<f_{g}^{l}(x,y),f_{g}^{r}(x-d,y)\right>,$
(1)

where $\left<\cdot,\cdot\right>$represents the inner product, $N_{c}$ is the
channels of input features, $N_{g}$ is the number of groups, $f_{g}^{l}$ and
$f_{g}^{r}$ represent $g^{th}$ left and right feature group, respectively.
Afterward, the disparity volume is converted into a depth volume following You
et al. (2020), which is formulated as:

$z_{(u,v)}=\frac{f_{u}\times b}{D_{(u,v)}},x=\frac{(u-c_{u})\times
z}{f_{u}},y=\frac{(v-c_{v})\times z}{f_{v}},$ (2)

where $f_{u}$ and $f_{v}$ represent the horizontal and vertical focal length,
($c_{u},c_{v}$) is the camera center. Next, we employ 3D CNNs following Guo et
al. (2019) for dimension reduction and finally squeeze the channel dimension
to construct the dense stereo depth volume
$\textbf{V}_{Stereo}\in\mathbb{R}^{D\times H\times W}$.

BEV Latent Volume Constructor. Although the stereo constructor provides
accurate estimation in matched regions, it struggles in extreme conditions
where severe occlusion or high reflection happens. Unlike the stereo-based
approach relying on strict geometric matching, BEV representations are
obtained by lifting an image $I^{rgb}$ to a shared bird’s eye space through 3D
prior. Following Philion and Fidler (2020); Li et al. (2023b), we feed visual
features $\textbf{F}_{l}$ to a neural network and obtain a latent depth
distribution $\textbf{V}_{BEV}\in\mathbb{R}^{D\times H\times W}$ with its
associated context features $\textbf{C}_{BEV}\in\mathbb{R}^{C_{b}\times
H\times W}$. Since this distribution is essentially a voxel grid that stores
the probability of all possible depths, we denote it as BEV latent volume for
the sake of clarity.

###  3.3 Mutual Interactive Ensemble

To achieve fine-grained reliable perception, the Mutual Interactive Ensemble
(MIE) block is introduced to bridge the representation gap between stereo
volume $\textbf{V}_{Stereo}$ and BEV volume $\textbf{V}_{BEV}$ by mutually
reinforcing each other and integrating their respective potentials at pixel
level.

Figure 3:  The structure of the proposed Bi-directional Reliable Interaction
module, which is designed for pixel-level reliable geometry information
interaction.

####  3.3.1 Bi-directional Reliable Interaction.

For pixel-level reliable interaction, we propose an initial interactive stage
that selectively retrieves dependable information alongside its counterpart
volume. More specifically, a Bi-directional Reliable Interaction (BRI) module
as shown in Figure 3 is devised to interactively guide reliable predictions of
its contrary side through a cross-attention mechanism. For stereo volume
$\textbf{V}_{Stereo}$, we first obtain its query $Q_{S}$, key $K_{S}$ and
value $V_{S}$ by flattening in spatial and depth dimensions following standard
protocol Wang et al. (2018); Liao et al. (2022). Similarly, the BEV volume
$\textbf{V}_{BEV}$ is forwarded and its query, key and value are denoted as
$Q_{B},K_{B},V_{B}$, respectively.

Then we construct the interacted volume with the cross-attention operation. To
reduce computational and memory consumption, we follow Katharopoulos et al.
(2020); Kitaev et al. (2020) to compute linear cross-attention. Specially, the
interacted BEV volume $\textbf{V}_{BEV}^{\prime}$ is obtained by:

$\begin{split}\textbf{V}_{BEV}^{\prime}&=CrossAtt(Q_{S},K_{B},V_{B})\\\
&=\phi_{q}(Q_{S})\textbf{G}_{B}=\phi_{q}(Q_{S})(\phi_{k}{(K_{B})}^{T}V_{B}),\end{split}$
(3)

where $\phi_{q}$ and $\phi_{k}$ denote the softmax function along each row and
column of the input matrix, respectively. $\textbf{G}_{B}$ represents global
contextual vectors of the BEV representation. In this way,
$\textbf{V}_{BEV}^{\prime}$ retrieves the relevant aspects of the stereo
sides, thereby providing an alternative perspective on the feature importance
of the BEV side. Likewise, its opposite interacted volume
$\textbf{V}_{Stereo}^{\prime}$ is computed by $CrossAtt(Q_{B},K_{S},V_{S})$ to
encourage reliable geometry information exchange.

_Depth Confidence Filtering_. In order to further retrieve pixel-level
reliable information for dense prediction, we develop a depth confidence
filtering strategy, which explicitly takes advantage of the involved reliable
geometry information behind the volume. We aim to utilize its depth confidence
information to enforce the cross-attention operation similar to Chen et al.
(2020). Particularly, to project the volume to a confidence map
$\textbf{C}_{S}$ , we first adopt $softmax$ to convert depth cost value
$d_{i}$ into a probability form, and then take out the highest probability
value among all depth hypothesis planes along the depth dimension as the
prediction confidence. The process is formally written as:

${\textbf{C}_{S}=WTA(\phi(\textbf{V}_{Stereo}))=WTA\left\\{\frac{\exp(d_{i})}{\sum_{j=1}^{D_{max}}\exp(d_{j})}\right\\},}$
(4)

where the $softmax$ is applied across the depth dimension and $WTA$ represents
winner-takes-all operation. $D_{max}$ denotes the length of the depth
dimension.

Next, we revisit the cross-attention operation in Equation 3 and construct
pixel-level reliable retrieval with the confidence map $\textbf{C}_{S}$ to
identify the criteria for an optimized formulation:

${CrossAtt(Q_{S},K_{B},V_{B})=\phi_{q}(Q_{S})\odot\textbf{C}_{S}(\phi_{k}{(K_{B})}^{T}V_{B}),}$
(5)

where $\odot$ represents the element-wise product, through which the reliable
geometry information is preserved while low-confidence information is
suppressed.

Figure 4: The structure of the proposed Dual Volume Ensemble module, which is
devised for mutually beneficial aggregation.

####  3.3.2 Dual Volume Ensemble.

With the interacted volume representations
$\textbf{V}_{Stereo}^{\prime}\in\mathbb{R}^{1\times D\times H\times W}$ and
$\textbf{V}_{BEV}^{\prime}\in\mathbb{R}^{1\times D\times H\times W}$, the
primary objective of this module is to leverage their strengths and facilitate
mutually beneficial complementation.

As illustrated in Figure 4, the DVE takes as input concatenated features
$\textbf{V}^{\prime}_{cat}=[\textbf{V}_{Stereo}^{\prime},\textbf{V}_{BEV}^{\prime}]\in\mathbb{R}^{2\times
D\times H\times W}$ and outputs ensembled volume $\textbf{V}_{ens}$.
Especially, the input $\textbf{V}^{\prime}_{cat}$ is first fed into residual
3D CNNs for regularization and channel adjustment, which generates transformed
representation $\textbf{V}_{f}\in\mathbb{R}^{C_{f}\times D\times H\times W}$.
The transformed representation $\textbf{V}_{f}$ is further processed by
Channel-wise Recalibration and Multi-group Voting, which are described in
detail below.

_Channel-wise Recalibration._ To fully exploit its contextual information Hu
et al. (2018), we utilize average pooling to squeeze the information into a
channel descriptor $\textbf{z}_{c}$. More specifically, we shrink
$\textbf{V}_{f}$ along both the depth dimension $D$ and spatial dimension
$H\times W$:

$\textbf{z}_{c}=\frac{1}{D\times H\times
W}\sum_{d=1}^{D}\sum_{i=1,j=1}^{H,W}\textbf{V}_{f}(d,i,j),$ (6)

Subsequently, an excitation block Hu et al. (2018) is leveraged to capture its
channel-wise dependencies. Formally, the channel descriptor $\textbf{z}_{c}$
is updated by two stacked bottleneck-shape convolutions with non-linear
activation. Finally, the updated channel descriptor is employed to re-weight
the previous transformed feature $\textbf{V}_{f}$ along the channel dimension:

$\textbf{V}_{f}^{\prime}=\sigma(\textbf{W}_{2}\delta(\textbf{W}_{1}\textbf{z}_{c}))\cdot\textbf{V}_{f},$
(7)

where the ${W}_{1}$ and ${W}_{2}$ represent $1\times 1\times 1$ convolutions
with dimensionality-reduction. The $\delta$ denotes standard GELU activation
and the $\sigma$ indicates the sigmoid gate.

_Multi-group Voting._ To further capture multi-scale context, we split
$\textbf{V}_{f}^{\prime}$ into four groups and employ 3D atrous convolutions
Li et al. (2019); Cao and de Charette (2022) with different dilation rates.
The contextual information in different groups exhibits distinct receptive
fields, staying non-interfering with each other. Finally, we ensemble the
multi-scale context between different voting groups to construct
$\textbf{V}_{ens}$:

$\textbf{V}_{ens}=\mathbb{P}\left\\{\mathrm{Concat}\left(A_{g}^{i}(\mathrm{Split}_{channel}^{g1\sim
g4}(\textbf{V}_{f}^{\prime}))\right)\right\\},$ (8)

where $A_{g}^{i}$ denotes 3D atrous convolution employed in the $i^{th}$
context group. $\mathbb{P}$ is composed of point-wise convolution with GELU
activation and group normalization. The point-wise convolution encourages
channel-aware mixing by $1\times 1\times 1$ kernel size, through which the
ensembled volume $\textbf{V}_{ens}$ takes into consideration voting features
of different aspects.

###  3.4 Semantic Scene Completion

To make use of this high-quality volume $\textbf{V}_{ens}$ for semantic scene
completion, we augment it with its associated context information
$\textbf{C}_{BEV}$. The extracted context information from input images is
placed in a specific location of the bird-eye’s representation by an outer
product operation similar to Philion and Fidler (2020); Li et al. (2023b).
Formally, the ensembled voxel features
$\textbf{F}_{vox}\in\mathbb{R}^{C_{b}\times D\times H\times W}$ is computed
by:

$\textbf{F}_{vox}=\textbf{C}_{BEV}\otimes\textbf{V}_{ens},$ (9)

In this way, we are able to seamlessly blend the complementary benefits of
stereo representation of precise geometry and BEV features of rich semantic
context.

Semantic Segmentation Learning. Following Cao and de Charette (2022), we
leverage 3D UNet to regularize the ensembled voxel features. Its output
features are fed to a SSC head holding upsampling and a softmax layer for
semantic occupancy prediction $\hat{\textbf{Y}}$.

Network Training. We follow the basic learning objective of MonoScene Cao and
de Charette (2022) for semantic scene completion. Standard semantic loss
$\mathcal{L}_{\text{sem}}$ and geometry loss $\mathcal{L}_{\text{geo}}$ are
leveraged for semantic and geometry supervision, while an extra class
weighting loss $\mathcal{L}_{ce}$ is also added. To further enforce the
ensembled volume, we adopt a binary cross entropy loss $\mathcal{L}_{depth}$
to encourage the sparse depth distribution. The overall learning objective of
this framework is formulated as:

${\mathcal{L}=\mathcal{L}_{depth}+\mathcal{\lambda}_{ce}\mathcal{L}_{ce}+\mathcal{\lambda}_{sem}\mathcal{L}_{\text{sem}}+\mathcal{\lambda}_{geo}\mathcal{L}_{\text{geo}}.}$
(10)

where several $\lambda$s are balancing coefficients.

Methods | BRGScene(ours) | VoxFormer-T | VoxFormer-S | OccFormer | SurroundOcc | TPVFormer | MonoScene
---|---|---|---|---|---|---|---
Input | Stereo | Stereo-T | Stereo | Mono | Mono | Mono | Mono
IoU (%) $\uparrow$ | 43.34 | 43.21 | 42.95 | 34.53 | 34.72 | 34.25 | 34.16
mIoU (%) $\uparrow$ | 15.36 | 13.41 | 12.20 | 12.32 | 11.86 | 11.26 | 11.08
car (3.92%) | 22.80 | 21.70 | 20.80 | 21.60 | 20.60 | 19.20 | 18.80
bicycle (0.03%) | 3.40 | 1.90 | 1.00 | 1.50 | 1.60 | 1.00 | 0.50
motorcycle (0.03%) | 2.40 | 1.60 | 0.70 | 1.70 | 1.20 | 0.50 | 0.70
truck (0.16%) | 2.80 | 3.60 | 3.50 | 1.20 | 1.40 | 3.70 | 3.30
other-veh. (0.20%) | 6.10 | 4.10 | 3.70 | 3.20 | 4.40 | 2.30 | 4.40
person (0.07%) | 2.90 | 1.60 | 1.40 | 2.20 | 1.40 | 1.10 | 1.00
bicyclist (0.07%) | 2.20 | 1.10 | 2.60 | 1.10 | 2.00 | 2.40 | 1.40
motorcyclist (0.05%) | 0.50 | 0.00 | 0.20 | 0.20 | 0.10 | 0.30 | 0.40
road (15.30%) | 61.90 | 54.10 | 53.90 | 55.90 | 56.90 | 55.10 | 54.70
parking (1.12%) | 30.70 | 25.10 | 21.10 | 31.50 | 30.20 | 27.40 | 24.80
sidewalk (11.13%) | 31.20 | 26.90 | 25.30 | 30.30 | 28.30 | 27.20 | 27.10
other-grnd (0.56%) | 10.70 | 7.30 | 5.60 | 6.50 | 6.80 | 6.50 | 5.70
building (14.10%) | 24.20 | 23.50 | 19.80 | 15.70 | 15.20 | 14.80 | 14.40
fence (3.90%) | 16.50 | 13.10 | 11.10 | 11.90 | 11.30 | 11.00 | 11.10
vegetation (39.3%) | 23.80 | 24.40 | 22.40 | 16.80 | 14.90 | 13.90 | 14.90
trunk (0.51%) | 8.40 | 8.10 | 7.50 | 3.90 | 3.40 | 2.60 | 2.40
terrain (9.17%) | 27.00 | 24.20 | 21.30 | 21.30 | 19.30 | 20.40 | 19.50
pole (0.29%) | 7.00 | 6.60 | 5.10 | 3.80 | 3.90 | 2.90 | 3.30
traf.-sign (0.08%) | 7.20 | 5.70 | 4.90 | 3.70 | 2.40 | 1.50 | 2.10
Table 1: Quantitative results on the SemanticKITTI test set. The top two performers are marked bold. Our method outperforms temporal stereo-based (Stereo-T) VoxFormer-T in terms of mIoU. Methods | Input |  mIoU (%) $\uparrow$ |  Time (s) $\downarrow$
---|---|---|---
SSCNet∗ (2017) | Stereo-PTS | 10.31 | 0.187
LMSCNet∗ (2020) | Stereo-PTS | 10.45 | 0.214
MonoScene∗ (2022) | Stereo | 12.82 | 0.274
TPVFormer∗ (2023) | Stereo | 13.06 | 0.313
OccFormer∗ (2023) | Stereo | 13.57 | 0.338
VoxFormer-S (2023) | Stereo | 12.35 | 0.256
VoxFormer-T (2023) | Stereo-T | 13.35 | 0.307
BRGScene (ours) | Stereo | 15.43 | 0.285
Table 2:  Evaluation results of stereo variants. For MonoScene∗ and
TPVFormer∗, We employ stereo images as inputs. For SSCNet∗ and LMSCNet∗, we
leverage stereo depth net to generate pseudo point clouds (Stereo-PTS).

##  4 Experiments

###  4.1 Experimental Settings

Datasets. We evaluate the proposed BRGScene on SemanticKITTI Behley et al.
(2019) that is popularly used in a great number of previous studies. There are
22 driving outdoor scenes from the KITTI Odometry Benchmark Geiger et al.
(2012), covering diverse and challenging autonomous driving situations.
SemanticKITTI holds semantic annotations of LiDAR sweeps that are registered,
aggregated and voxelized as 256$\times$256$\times$32 grid of 0.2m voxels. The
target ground truth of each voxel grid is annotated as one of 21 classes (1
unknown, 1 free and 19 semantics). The SemanticKITTI benchmark provides both
voxelized LiDAR scans and RGB images as model input options. We solely utilize
RGB images since our main focus is to explore the portable camera-only signals
as did in MonoScene Cao and de Charette (2022).

Implementation Details. We set the 3D UNet input to 128$\times$128$\times$16
(1:2) for efficient memory usage, whose feature will be upscaled to 1:1 at
completion head by deconvolution operation. The $\lambda$s are empirically set
to 1. Our model is implemented on PyTorch. The model is trained for 30 epochs
using the AdamW optimizer Loshchilov and Hutter (2017) with a learning rate of
$1\times 10^{-4}$ and batch size set to 8.

Evaluation Metrics. Regarding quantitative evaluations, we conduct experiments
on the typical metrics Li et al. (2023a); Cao and de Charette (2022) that have
been widely employed in the field of SSC. Specially, we leverage IoU
(Intersection over Union) to account for the scene completion (SC) task and
mIoU (mean Intersection over Union) to measure the performance of the semantic
scene completion (SSC) task, respectively. For both of these two metrics,
higher values are desirable, where high IoU indicates accurate geometric
prediction and high mIoU implies precise semantic segmentation.

$\blacksquare$bicycle $\blacksquare$car $\blacksquare$motorcycle
$\blacksquare$truck $\blacksquare$other vehicle $\blacksquare$person
$\blacksquare$bicyclist $\blacksquare$motorcyclist $\blacksquare$road
---
$\blacksquare$parking $\blacksquare$sidewalk $\blacksquare$other ground
$\blacksquare$building $\blacksquare$fence $\blacksquare$vegetation
$\blacksquare$trunk $\blacksquare$terrain $\blacksquare$pole
$\blacksquare$traffic sign

Figure 5:  Qualitative results on the SemanticKITTI validation set. The overlay shadow areas at the bottom of semantic predictions denote unseen scenery out of the camera’s field of view (FOV).  Methods | Resolution |  mAP $\uparrow$ |  NDS $\uparrow$
---|---|---|---
BEVDet-Base (2021) |  1600$\times$ 640 | 0.397 | 0.477
BEVDet4D-Base (2021) |  1600$\times$ 640 | 0.426 | 0.552
PETR-R101 (2022) |  1408$\times$ 512 | 0.357 | 0.421
BEVDepth-R101 (2023) |  512$\times$ 1408 | 0.412 | 0.535
BRGScene (ours) | 1600$\times$ 640 | 0.451 | 0.563
Table 3:  Quantitative results of BEV Detection on the nuScenes validation
set. We conduct preliminary experiments by employing the detection head.
Figure 6: Visualization results of BEV detection on the nuScenes validation
set.

###  4.2 Performance

Quantitative Comparison. Table 1 reports the performance of our BRGScene and
other baselines on the SemanticKITTI test set. We compare our method with the
best models Li et al. (2023a); Zhang et al. (2023); Huang et al. (2023); Wei
et al. (2023); Cao and de Charette (2022) for semantic scene completion.
BRGScene surpasses temporal stereo-based (Stereo-T) VoxFormer-T Li et al.
(2023a) in terms of mIoU. Additionally, our approach outperforms stereo-based
VoxFormer-S by a large margin in terms of geometric completion
(42.95$\rightarrow$43.34) and semantic segmentation (12.20$\rightarrow$15.36).
It’s worth noting that our method demonstrates significant superiority in the
prediction of small moving objects compared to VoxFormer-S, including bicycle
(1.00$\rightarrow$3.40), motorcycle (0.70$\rightarrow$2.40), pole
(5.10$\rightarrow$7.00), etc. We ascribe such improvements to the ensemble of
dual volume, which is critical for 3D geometry awareness. Besides, although
VoxFormer-T employs up to 4 temporal stereo image pairs as inputs, our method
has a significant advantage in mIoU, with a 14.5% relative improvement in
terms of semantics.

Evaluation of Stereo Variants. To ensure fair comparisons, we also implement
stereo variants of the baselines as shown in Table 2. For MonoScene∗ Cao and
de Charette (2022) and TPVFormer∗ Huang et al. (2023), we employ left and
right images to generate stereo-based predictions. Other LiDAR-based baselines
including LMSCNet Roldao et al. (2020) and SSCNet Song et al. (2017) require
3D geometric inputs, so we adapt them with pseudo-3D inputs leveraging the
same GwcNet Guo et al. (2019) used in our framework. Our proposed BRGScene
efficiently outperforms all the other methods. For more details on model
complexity, please refer to the Supplementary Material.

Qualitative Comparison.  As shown in Figure 5, we compare the visualization
results of BRGScene and VoxFormer on the SemanticKITTI validation set. Due to
the complexity of the real-world scenes and the sparsity of the labels, it is
challenging to reconstruct the scenes accurately and completely. Compared to
VoxFormer-T and VoxFormer-S, our method evidently captures better geometric
representations for more complete and precise scene reconstruction (e.g.
crossroads in rows 1,2) and generates more proper hallucinations outside FOV
(e.g. shadow regions in rows 2,3).

BEV Detection Evaluation. We further conduct preliminary experimental results
for BEV 3D detection on nuScenes validation set. Specifically, we adopt BEVDet
Huang et al. (2021) as the baseline setting, and replace the BEVDet model with
our proposed BRGScene while maintaining the detection head. Note that we adopt
temporal inputs from current and previous images to construct the temporal
volume, which replaces the original stereo volume. The preliminary results are
in Table 3 and Figure 6, which show that our proposed method can also be
applied to a more wide range of downstream tasks.

Dual Volume | MIE | IoU (%) $\uparrow$ | mIoU (%) $\uparrow$
---|---|---|---
Stereo | BEV | BRI | DVE
|  |  |  | 33.87 | 9.92
✓ |  |  |  | 37.38 | 11.49
| ✓ |  |  | 36.34 | 11.06
✓ | ✓ |  |  | 39.68 | 12.53
✓ | ✓ | ✓ |  | 43.04 | 14.64
✓ | ✓ |  | ✓ | 42.92 | 14.59
✓ | ✓ | ✓ | ✓ | 43.85 | 15.43
Table 4:  Ablation study for architectural components.

###  4.3 Ablation Study

We ablate our BRGScene on the SemanticKITTI validation set for semantic scene
completion. The ablation study for the architectural components is shown in
Table 4, which includes the Dual Volume and the Mutual Interactive Ensemble
block. The baseline in the first row of the table is built by removing the
Dual Volume and the MIE block.

Effect of the Dual Volume. For the ablation study on the dual volume, we build
the framework with an individual volume to verify the effect. We find that
after adding the stereo volume, the geometric perception ability of the
framework improves obviously (+3.51 IoU), while the prediction performance of
semantic scene segmentation is enhanced as well (+1.57 mIoU). The introduction
of BEV volume also has an obvious impact on the semantic and the geometric
aspects , boosting IoU by 2.47 and mIoU by 1.14, respectively.

Effect of the Mutual Interactive Ensemble. We further conduct architectural
ablation to evaluate the impact of the Mutual Interactive Ensemble (MIE) block
as shown in Table 4. The BRI module can significantly improve the geometric
and semantic estimations (+3.36 IoU, +2.11 mIoU) with efficient mutual
interaction. Furthermore, we evaluate the DVE module by replacing the
alternative naive concatenation, which leads to significant improvements in
performance (+3.24 IoU, +2.06 mIoU). The aforementioned results validate that
our complementary mutual interaction has a significant performance improvement
compared to naive aggregation.

##  5 Conclusion

In this work, we propose BRGScene, a 3D Semantic Scene Completion framework
that leverages both stereo and BEV representations to produce reliable 3D
scene understanding results. To bridge the representation gap between the
stereo volume and BEV volume for fine-grained 3D perception, a Mutual
Interactive Ensemble block is proposed to incorporate complementary merits of
the two dense volumes. Our BRGScene outperforms existing camera-based state-
of-the-arts on the challenging SemanticKITTI dataset. We hope BRGScene could
inspire further research in camera-based SSC and its applications in 3D scene
understanding.

## References

  * Behley et al. [2019] Jens Behley, Martin Garbade, Andres Milioto, Jan Quenzel, Sven Behnke, Cyrill Stachniss, and Jurgen Gall.  Semantickitti: A dataset for semantic scene understanding of lidar sequences.  In ICCV, 2019.
  * Cai et al. [2021] Yingjie Cai, Xuesong Chen, Chao Zhang, Kwan-Yee Lin, Xiaogang Wang, and Hongsheng Li.  Semantic scene completion via integrating instances and scene in-the-loop.  In CVPR, 2021.
  * Cao and de Charette [2022] Anh-Quan Cao and Raoul de Charette.  Monoscene: Monocular 3d semantic scene completion.  In CVPR, 2022.
  * Chen et al. [2020] Po-Heng Chen, Hsiao-Chien Yang, Kuan-Wen Chen, and Yong-Sheng Chen.  Mvsnet++: Learning depth-based attention pyramid features for multi-view stereo.  IEEE Transactions on Image Processing, 29, 2020.
  * Deng [2021] Lahav Lipson;Zachary Teed;Jia Deng.  Raft-stereo: Multilevel recurrent field transforms for stereo matching.  3DV, 2021.
  * Garbade et al. [2019] Martin Garbade, Yueh-Tung Chen, Johann Sawatzky, and Juergen Gall.  Two stream 3d semantic scene completion.  In CVPRW, 2019.
  * Geiger et al. [2012] Andreas Geiger, Philip Lenz, and Raquel Urtasun.  Are we ready for autonomous driving? the kitti vision benchmark suite.  In CVPR, 2012.
  * Guo et al. [2019] Xiaoyang Guo, Kai Yang, Wukui Yang, Xiaogang Wang, and Hongsheng Li.  Group-wise correlation stereo network.  In CVPR, 2019.
  * Hu et al. [2018] Jie Hu, Li Shen, and Gang Sun.  Squeeze-and-excitation networks.  In CVPR, 2018.
  * Hu et al. [2021] Anthony Hu, Zak Murez, Nikhil Mohan, Sofía Dudas, Jeffrey Hawke, Vijay Badrinarayanan, Roberto Cipolla, and Alex Kendall.  Fiery: future instance prediction in bird’s-eye view from surround monocular cameras.  In ICCV, 2021.
  * Huang et al. [2021] Junjie Huang, Guan Huang, Zheng Zhu, Yun Ye, and Dalong Du.  Bevdet: High-performance multi-camera 3d object detection in bird-eye-view.  arXiv preprint arXiv:2112.11790, 2021.
  * Huang et al. [2023] Yuanhui Huang, Wenzhao Zheng, Yunpeng Zhang, Jie Zhou, and Jiwen Lu.  Tri-perspective view for vision-based 3d semantic occupancy prediction.  In CVPR, 2023.
  * Katharopoulos et al. [2020] Angelos Katharopoulos, Apoorv Vyas, Nikolaos Pappas, and François Fleuret.  Transformers are rnns: Fast autoregressive transformers with linear attention, 2020.
  * Kendall et al. [2017] Alex Kendall, Hayk Martirosyan, Saumitro Dasgupta, Peter Henry, Ryan Kennedy, Abraham Bachrach, and Adam Bry.  End-to-end learning of geometry and context for deep stereo regression.  In ICCV, 2017.
  * Kitaev et al. [2020] Nikita Kitaev, Łukasz Kaiser, and Anselm Levskaya.  Reformer: The efficient transformer, 2020.
  * Li et al. [2019] Jie Li, Yu Liu, Dong Gong, Qinfeng Shi, Xia Yuan, Chunxia Zhao, and Ian Reid.  Rgbd based dimensional decomposition residual network for 3d semantic scene completion.  In CVPR, 2019.
  * Li et al. [2021] Jiankun Li, Peisen Wang, Pengfei Xiong, Tao Cai, Ziwei Yan, Lei Yang, Jiangyu Liu, Haoqiang Fan, and Shuaicheng Liu.  Practical stereo matching via cascaded recurrent network with adaptive correlation.  CVPR, 2021.
  * Li et al. [2022a] Yinhao Li, Han Bao, Zheng Ge, Jinrong Yang, Jianjian Sun, and Zeming Li.  Bevstereo: Enhancing depth estimation in multi-view 3d object detection with dynamic temporal stereo.  arXiv preprint arXiv:2209.10248, 2022.
  * Li et al. [2022b] Zhiqi Li, Wenhai Wang, Hongyang Li, Enze Xie, Chonghao Sima, Tong Lu, Yu Qiao, and Jifeng Dai.  Bevformer: Learning bird’s-eye-view representation from multi-camera images via spatiotemporal transformers.  In ECCV, 2022.
  * Li et al. [2023a] Yiming Li, Zhiding Yu, Christopher Choy, Chaowei Xiao, Jose M Alvarez, Sanja Fidler, Chen Feng, and Anima Anandkumar.  Voxformer: Sparse voxel transformer for camera-based 3d semantic scene completion.  In CVPR, 2023.
  * Li et al. [2023b] Yinhao Li, Zheng Ge, Guanyi Yu, Jinrong Yang, Zengran Wang, Yukang Shi, Jianjian Sun, and Zeming Li.  Bevdepth: Acquisition of reliable depth for multi-view 3d object detection.  In AAAI, 2023.
  * Liao et al. [2022] Jinli Liao, Yikang Ding, Yoli Shavit, Dihe Huang, Shihao Ren, Jia Guo, Wensen Feng, and Kai Zhang.  Wt-mvsnet: window-based transformers for multi-view stereo.  NeurIPS, 2022.
  * Loshchilov and Hutter [2017] Ilya Loshchilov and Frank Hutter.  Decoupled weight decay regularization.  arXiv preprint arXiv:1711.05101, 2017.
  * Philion and Fidler [2020] Jonah Philion and Sanja Fidler.  Lift, splat, shoot: Encoding images from arbitrary camera rigs by implicitly unprojecting to 3d.  In ECCV, 2020.
  * Poggi et al. [2022] Matteo Poggi, Fabio Tosi, Konstantinos Batsos, Philippos Mordohai, and Stefano Mattoccia.  On the synergies between machine learning and binocular stereo for depth estimation from images: A survey.  IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022\.
  * Roberts [1963] Lawrence G Roberts.  Machine perception of three-dimensional solids.  PhD thesis, Massachusetts Institute of Technology, 1963.
  * Roddick and Cipolla [2020] Thomas Roddick and Roberto Cipolla.  Predicting semantic map representations from images using pyramid occupancy networks.  In CVPR, 2020.
  * Roldao et al. [2020] Luis Roldao, Raoul de Charette, and Anne Verroust-Blondet.  Lmscnet: Lightweight multiscale 3d semantic completion.  In 3DV, 2020.
  * Roldao et al. [2022] Luis Roldao, Raoul De Charette, and Anne Verroust-Blondet.  3d semantic scene completion: A survey.  International Journal of Computer Vision, 130(8), 2022.
  * Song et al. [2017] Shuran Song, Fisher Yu, Andy Zeng, Angel X Chang, Manolis Savva, and Thomas Funkhouser.  Semantic scene completion from a single depth image.  In CVPR, 2017.
  * Tan and Le [2019] Mingxing Tan and Quoc Le.  Efficientnet: Rethinking model scaling for convolutional neural networks.  In ICML, 2019.
  * Wang et al. [2018] Xiaolong Wang, Ross Girshick, Abhinav Gupta, and Kaiming He.  Non-local neural networks.  In CVPR, 2018.
  * Wei et al. [2023] Yi Wei, Linqing Zhao, Wenzhao Zheng, Zheng Zhu, Jie Zhou, and Jiwen Lu.  Surroundocc: Multi-camera 3d occupancy prediction for autonomous driving.  In ICCV, 2023.
  * Wu et al. [2020] Shun-Cheng Wu, Keisuke Tateno, Nassir Navab, and Federico Tombari.  Scfusion: Real-time incremental scene reconstruction with semantic completion.  In 3DV, 2020.
  * You et al. [2020] Yurong You, Yan Wang, Wei-Lun Chao, Divyansh Garg, Geoff Pleiss, Bharath Hariharan, Mark Campbell, and Kilian Q Weinberger.  Pseudo-lidar++: Accurate depth for 3d object detection in autonomous driving.  ICLR, 2020.
  * Zhang et al. [2023] Yunpeng Zhang, Zheng Zhu, and Dalong Du.  Occformer: Dual-path transformer for vision-based 3d semantic occupancy prediction.  ICCV, 2023.