# SOccDPT: Semi-Supervised 3D Semantic Occupancy from Dense Prediction
Transformers trained under memory constraints

Aditya Nalgunda Ganesh1 1Aditya Nalgunda Ganesh is with Department of Computer
Science and Engineering, PES University, India<EMAIL_ADDRESS>
###### Abstract

We present SOccDPT, a memory-efficient approach for 3D semantic occupancy
prediction from monocular image input using dense prediction transformers. To
address the limitations of existing methods trained on structured traffic
datasets, we train our model on unstructured datasets including the Indian
Driving Dataset and Bengaluru Driving Dataset. Our semi-supervised training
pipeline allows SOccDPT to learn from datasets with limited labels by reducing
the requirement for manual labelling by substituting it with pseudo-ground
truth labels to produce our Bengaluru Semantic Occupancy Dataset. This broader
training enhances our model’s ability to handle unstructured traffic scenarios
effectively. To overcome memory limitations during training, we introduce
patch-wise training where we select a subset of parameters to train each
epoch, reducing memory usage during auto-grad graph construction. In the
context of unstructured traffic and memory-constrained training and inference,
SOccDPT outperforms existing disparity estimation approaches as shown by the
RMSE score of 9.1473, achieves a semantic segmentation IoU score of 46.02% and
operates at a competitive frequency of 69.47 Hz. We make our code and semantic
occupancy dataset public.

##  I INTRODUCTION

Figure 1: Above are a few frames from our Bengaluru Semantic Occupancy Dataset
which is an extension of the Bengaluru Driving Dataset [1]. Each panel
consists of the RGB image with 2D semantic labels on the top left, the
disparity map on the bottom left and the 3D semantic occupancy on the right.
The vehicle and pedestrian classes are colored in blue and red respectively.
Objects without classes have been plotted as a height map for the sake of
visualization. The vehicle and its future trajectory have been plotted out in
grey and green respectively to aid the reader to have a better scene
understanding.

Autonomous navigation requires 3D semantic understanding of the environment at
a high frequency with a limited compute budget. The field of autonomous
driving has shown significant interest in vision-based 3D scene perception due
to its exceptional efficiency and abundant semantic information. When it comes
to choosing an architecture, works such as [2, 3, 4, 5, 6, 7] inspired from
ViT [8] have the domain agnostic learning capabilities of the transformer. The
transformer’s versatility comes at the cost of having no good inductive priors
for any domain, requiring large volumes of training data and a large volume of
GPU memory to train. To apply such models on a new domain, we must be
efficient in making use of transfer learning and pseudo-labelling to solve the
ground truth data scale problem.

In the context of 3D semantic occupancy from monocular vision, ground truth
data would refer to semantically labelled 3D point clouds with corresponding
RGB images acquired from a calibrated camera sensor as shown in Fig. 1 of our
Bengaluru Semantic Occupancy Dataset. While there exist datasets [9, 10, 11]
which have labelled 3D semantic occupancy data in the context of structured
traffic, the unstructured traffic scenarios remain largely underrepresented.
It may not be feasible to gather large volumes of training data considering
the fact that LiDAR sensors are expensive and labelling 3D semantic classes
can be tedious. Hence, we make use of a set of teacher models and boosting
techniques inspired from [12, 1, 13, 14] to produce labels for depth and
semantic on driving video footage which we use to supervise the training of
our model. We train our system on unstructured driving datasets such as the
Indian Driving Dataset [15] and Bengaluru Driving Dataset [1] to ensure that
our system generalizes well. Training such models requires large volume of GPU
memory. We overcome this hurdle with our PatchWise training approach which
keeps the GPU memory in check and this allowed us to explore higher batch
sizes without altering the back-propagation algorithm.

With the goal of designing a model, which is efficient during both training
and inference, we propose SOccDPT and our PatchWise training system. To ensure
SOccDPT performs well in unstructured traffic scenarios, we introduce semi-
supervision with our pseudo-labelling process for depth boosting and semantic
auto-labelling. We use a common backbone for image feature extraction and dual
heads to extract disparity and semantic information of the scene. Camera
intrinsics are used along with disparity to project the semantic information
into 3D space.

##  II RELATED WORK

Figure 2: SOccDPT uses the ViT family for backbone feature extraction which
allows us to carefully balance accuracy and compute requirements. SOccDPT
takes an RGB image input of shape $3\times 256\times 256$ produces image
features of shape 256x128x128. We then pass the extracted features to a
disparity head and a segmentation head. We apply the Scale and Shift Invariant
loss [16] and the Binary Cross Entropy loss for the disparity and segmentation
outputs respectively. With the known camera intrinsic, we project the
semantics into 3D space with the help of the disparity map and accumulate the
semantics into a 3D occupancy grid of size $256\times 256\times 32$, thus
producing a 3D semantic map from one backbone

Multi-Task Learning.  The domain of visual perception has extensively studied
the concept of Multi-Task Learning including tasks such as semantic
segmentation [17, 18] and jointly learning depth and semantics [19]. A
comprehensive discussion of deep multi-task learning can be found in [20].
Although there have been some advancements in semi-supervised multi-task
learning, as evidenced by previous studies [21, 22], these approaches do not
specifically tackle the difficult problem of training models across diverse
datasets in the absence of ground truth information. Our focus is to address
this gap by developing a solution for semi-supervised multi-task learning in
the domain of 3D semantic occupancy within unstructured traffic environments.

Semi-Supervised Learning and Self-Supervised Learning.  In the context of
disparity estimation, semi-supervised learning has become very important due
to the challenges involved in obtaining accurate depth information in diverse
real-world environments. Several self-supervised algorithms for perceiving
depth have been suggested [23, 24, 25, 26, 27]. These algorithms offer the
advantage of utilizing only a single camera, making them suitable for easy
deployment in real-world scenarios. However, they still face numerous
unresolved issues. One such problem is the generation of disparity maps that
lack local and temporal consistency. Watson et al. [27] addressed the temporal
inconsistency by incorporating multiple consecutive frames as input. Another
line of research in semi-supervised learning looks into using the existing
model to generate confident annotations on unlabelled data. Examples for such
approaches include pseudo-labelling [28] and entropy minimization [29]. Since
the degree of disparity is inversely related to depth, as demonstrated in Eq.
1, slight variations in disparity for distant objects lead to significant
variations in depth. Consequently, the resulting point clouds exhibit non-
uniform resolution, with closer objects represented by more points compared to
those farther away. There are broadly two approaches tot he disparity
estimation problem: monocular and stereoscopic.

Monocular and Stereo Disparity Estimation.  Diverse neural network
architectures, including variational auto-encoders, convolutional neural
networks, generative adversarial networks and recurrent neural networks, have
demonstrated their efficacy in tackling the task of depth estimation. Within
this framework, two methods are commonly employed: monocular, where depth is
estimated from a single input image, and stereoscopic depth estimation, where
depth is estimated from a pair of images provided as input to the system.
Monocular approaches such as [16, 23, 24, 27] take advantage of depth cues
such as occlusion boundaries, parallel lines and so on to understand the 3D
scene. Techniques based on binocular learning [30, 31, 32, 33], generate depth
maps by leveraging the epipolar constraints associated with feasible disparity
values. Although these approaches have enhanced accuracy, they come with a
trade-off in terms of computational time and/or hardware demands. Running such
systems in real-time becomes impractical on embedded devices that lack
sufficient power, especially without specialized hardware like FPGAs.

Bird’s Eye View (BEV) Architectures. Since disparity is inversely proportional
to depth, errors in depth estimation grow quadratically with disparity errors.
To address these errors, some approaches look to operate in the Bird’s Eye
View space which is directly proportional to depth. Obtaining a top-down view
of a scene offers a comprehensive understanding of the surrounding
environment, effectively capturing both static and dynamic elements. BEV
architectures, exemplified by [34, 35, 36], generate this top-down map, which
can be utilized for path planning purposes. This top down map is essentially a
segmentation map which would highlight the road, non-drivable space, parking
areas, vehicles, pedestrians and so on. The concept of predicting BEV from
multiple camera perspectives has demonstrated performance comparable to LiDAR-
centric methods [37, 38]. However, a limitation of this approach is the
absence of 3D information about the scene, such as unclassified objects,
potholes, and overhanging obstacles.

3D Occupancy Networks. Difficult to classify 3D obstacles become harder and
harder to catch as the research community and industry chases the long tail of
nines. Recent approaches look to build generic 3D object detection free of
ontology by the way of 3D Occupancy Networks. Achieving an effective
representation of a 3D scene is a fundamental objective in perceiving 3D
environments. One direct approach involves discretizing the 3D space into
voxels within an occupancy grid [39, 38]. The voxel-based representation is
advantageous for capturing intricate 3D structures, making it suitable for
tasks like LiDAR segmentation [40, 41, 42, 43, 44, 38] and 3D scene completion
[45, 46, 47, 48, 49]. A recent method, TPVFormer [50], addresses memory
optimization by representing the 3D space as projections on three orthogonal
planes. Despite the significant progress made by these approaches, they do not
specifically tackle the issue of existing dataset biases towards structured
traffic.

##  III PROPOSED WORK

RGB [] |  |
---|---|---
Ground Truth |  |
$SOccDPT$ |  |
$MiDaS$[51] |  |
Manydepth [27] |  |
PackNet [52] |  |
ZeroDepth [53] |  |
Figure 3: Qualitative results comparing frames in BDD to Midas versions,
monodepth, manydepth, ZeroDepth. As we can see, all the existing approaches do
not address the diversity that is seen in unstructured traffic

###  III-A SOccDPT Architecture

As described in Fig. 2, SOccDPT uses the Dense Prediction Transformer [51, 6]
backbones to efficiently extract image features. We then use independent heads
to produce the disparity and segmentation maps. Instead of penalizing the
model for generating the output in an inaccurate scale, we address the issue
of arbitrary scale in the disparity map by estimating the scale and shift
relative to the ground truth for every frame. This estimation process involves
aligning the prediction with the ground truth using a least-squares criterion.

Once the segmentation and disparity maps are computed, we make use of the
camera intrinsics to project the semantics into 3D space. Consider a point on
the image plane at position $(u,v)$ with disparity $D(u,v)$ and 2D semantics
$S_{2D}(u,v)$. This point corresponds to the 3D point $(x,y,z)$ as shown in
Eq. 1 from which we can assert the 3D semantics correspondence to be
$S_{3D}(x,y,z)\longleftarrow S_{2D}(u,v)$

$\begin{split}(x,y,z)=(\frac{b\cdot(u-o_{x})}{D(u,v)},\frac{b\cdot
f_{x}\cdot(v-o_{y})}{f_{y}\cdot D(u,v)},\frac{b\cdot
f_{x}}{D(u,v)})\end{split}$ (1)

In order to train our network, we started off by building a baseline model
$V1$ which consists of 2 separate backbones, one for disparity and the other
for segmentation. This informs us of the performance of the dense prediction
transformer on unstructured traffic datasets. We improve upon $V1$ by having a
common backbone in $V2$ which lead to optimizations in speed and memory
consumption. This came at the cost of the accuracy of both the segmentation
and disparity. This is due to the fact that the network would be learning the
features and intricacies of both the tasks from scratch simultaneously. To
address this, $V3$ makes a minor modification to $V2$ which allows us to load
in the disparity estimation backbone from $V1$. This allows $V3$ to have a
backbone which is proficient in the disparity estimation task. When starting
from this point, the backbone and segmentation head only have to learn the
task of image segmentation, without making any major alterations to the
existing disparity estimation. This provided an improvement in how much the
model was able to learn with the same data.

###  III-B PatchWise Training

Our PatchWise system, implemented in PyTorch [54], offers a solution to GPU
memory limitations during neural network training. Instead of updating all
weights simultaneously, which can lead to "out of memory" errors, PatchWise
updates a subset of the model’s weights at a time. This approach enables the
training of larger networks and the use of larger batch sizes on systems with
limited GPU memory, although it increases training time. The implementation
details are described in Algorithm 1.

PatchWise $(\text{net},\text{train\\_percentage},\text{train\\_step})$;

Input : PyTorch Module net, training percentage train_percentage, training
function train_step

Output : Trained neural network

$\text{N}\leftarrow\text{length}(\text{net.parameters})$;

$\text{M}\leftarrow\text{round}(\text{N}\times\text{train\\_percentage})$;

$\text{num\\_iterations}\leftarrow\lceil\text{N}/\text{M}\rceil$;

$\text{updated\\_weights}\leftarrow\\{\\}$;

$\text{saved\\_weights}\leftarrow\\{\\}$;

for  _$\text{index},\text{param}\text{ in }\text{net.parameters}$_ do

$\text{saved\\_weights[index]}\leftarrow\text{param}$;

end for

for  _$\text{net\\_patch\\_index}\text{ in range}(0,\text{num\\_iterations})$_
do

$\text{start\\_index}\leftarrow\text{net\\_patch\\_index}\times\text{M}$;

$\text{end\\_index}\leftarrow\min(\text{start\\_index}+\text{M},\text{N})$;

$\text{train\\_indices}\leftarrow\text{range}(\text{start\\_index},\text{end\\_index})\;$
for  _$\text{index},\text{param}\text{ in }\text{net.parameters}$_ do

$\text{param}\leftarrow\text{saved\\_weights[index]}$;

$\text{param.requires\\_grad}\leftarrow\text{bool}($

$\text{index}\in\text{train\\_indices}$

$)$;

end for

train_step(net);

$\text{save\\_indices}\leftarrow\text{range}(\text{start\\_index},\text{end\\_index})$;

for  _$\text{index},\text{param}\text{ in }\text{net.parameters}$_ do

if  _$\text{index}\in\text{save\\_indices}$_ then

$\text{updated\\_weights[index]}\leftarrow\text{param}$;

end if

end for

end for

for  _$\text{index},\text{param}\text{ in }\text{net.parameters}$_ do

$\text{param}\leftarrow\text{updated\\_weights[index]}$;

end for

return net;

Algorithm 1 PatchWise

###  III-C Pseudo-Ground Truth Labels for Semi-Supervision

|  |
---|---|---
|  |
|  |
|  |
|  |
|  |

Figure 4: We use Depth Boosting to generate depth labels for the Indian Driving Dataset. We have the RGB frames on the left, segmentation map in the middle and our depth labels on the right. We would like to highlight the detail in the automatically generated disparity maps |  |
---|---|---
|  |
|  |
|  |
|  |
|  |

Figure 5: We use Semantic Segmentation auto-labelling to generate semantic
labels for the Bengaluru Driving Dataset. We have the RGB frames on the left,
our segmentation maps in the middle and depth labels on the right. We would
like to highlight the accuracy in the automatically generated segmentation
maps

Figure 6: While training $SOccDPT_{V3}$ we start with the pre-trained depth
backbone. As a result, the initial disparity metrics (RMSE, a1, a2, a3) are
good while the initial IoU score is under 5%. Within the first few epochs the
IoU score starts growing steadily, and we observe a small spike in the
disparity metrics as the depth head is adjusting to the changes made to
accommodate the segmentation head

The ability of vision based networks to learn and accurately predict based on
image input is limited by their receptive field and their overall learning
capacity (number of parameters). We generate pseudo-labels using existing
disparity estimation and image segmentation models by feeding in segments of
the image which allows the model to focus on smaller regions. This approach
trades compute time for higher accuracy. We augment the Indian Driving Dataset
[15] with depth labels and the Bengaluru Driving Dataset [1] with 2D semantic
labels, enhancing their utility for training.

Depth Boosting.  Monocular depth estimation systems use a lot of the depth
cues used by humans including occlusion boundaries, parallel lines, edges,
vanishing points and the shape and size of objects. Altering the resolution of
the image affects the clarity of these depth cues. While increasing the
resolution can produce sharper results, feeding in smaller patches of the
image fails. This happens when the window size shrinks to the point where
there are no depth cues, which generates an inconsistent overall structure and
may introduce low frequency artefacts. Taking inspiration from the depth
boosting techniques [1, 14, 13], we select a content adaptive resolution
$R_{x}$, beyond which the low frequency artefacts begin to hurt the overall
structure of the image. By merging the disparity maps from the various
resolutions, we are able to generate a high resolution disparity map with
global consistency. As suggested by [14], we select $R_{20}$ as the high-
resolution upper bound $R_{x}$, as their work shows that using resolutions
higher than $R_{30}$ results in a decrease in performance due to the
aforementioned artefacts. We use this method to generate disparity labels for
the Indian Driving Dataset as shown in Fig. 4. The depth images on the left
are colored by inverse depth (or disparity), such that pixels representing
objects closer to the camera are brighter and those representing objects
further away are darker.

Semantic Segmentation auto-labelling.  To produce high resolution 2D semantic
labels, we take inspiration from PointRend [12]. We take an image as input and
produce a coarse intermediate segmentation map using an existing segmentation
approach MaskRCNN [55]. This coarse map is gradually up-sampled using bi-
linear interpolation and only the regions of the resized map with high
uncertainty are refined. The uncertain regions typically include the
boundaries of objects. The uncertain region is refined by a lightweight multi-
layered perceptron. Its input is a feature vector which is extracted through
interpolation from the feature maps, which intern has been computed by the
base model. As shown in Fig. 5 we have auto-labelled vehicles in red and
humans in blue.

##  IV EXPERIMENTS

###  IV-A Experimental Setup

Model | Dataset | $RMSE^{\downarrow}$ | $a1^{\uparrow}$ | $a2^{\uparrow}$ | $a3^{\uparrow}$ |  $FPS^{\uparrow}$ (Hz) | $Parameters^{\downarrow}$
---|---|---|---|---|---|---|---
$MiDaS_{v3.1}$ $Swin2_{T-256}$[6] | BDD | 23.325 | 0.5944 | 0.7816 | 0.8585 | 82.2656 | 14.84M
$MiDaS_{v3.0}$ $DPT_{H-384}$[51] | BDD | 21.861 | 0.5527 | 0.7712 | 0.8558 | 13.4394 | 123.1M
$MiDaS_{v3.0}$ $DPT_{L-384}$[51] | BDD | 13.36 | 0.6888 | 0.8514 | 0.9192 | 6.3142 | 42.3M
$manydepth$[27] | BDD | 31.3599 | 0.4508 | 0.6668 | 0.7850 | 20.3578 | 35.34M
$ZeroDepth$[53] | BDD | 36.3419 | 0.3882 | 0.6552 | 0.7875 | 2.363 | 232.59M
$PackNet$[52] | BDD | 50.6722 | 0.2257 | 0.4936 | 0.6870 | 8.7504 | 128.29M
$SOccDPT_{V1}$ | BDD | 13.3782 | 0.6854 | 0.8442 | 0.9172 | 39.1141 | 84.3M
$SOccDPT_{V2}$ | BDD | 26.2383 | 0.4879 | 0.7181 | 0.8309 | 69.6503 | 42.3M
$SOccDPT_{V3}$ | BDD | 12.4075 | 0.6935 | 0.8588 | 0.9265 | 69.4733 | 42.3M

TABLE I: We compare SOccDPT’s disparity metrics on the Bengaluru Driving Dataset, FPS and number of parameters with existing approaches. $SOccDPT_{V3}$ outperforms the models in terms of accuracy while maintaining a high FPS and small model size Method | Dataset | Hyperparameters | $RMSE^{\downarrow}$ | $a1^{\uparrow}$ | $a2^{\uparrow}$ | $a3^{\uparrow}$ | $IoU^{\uparrow}$ (%)
---|---|---|---|---|---|---|---
BS | PP | EP | LR | $\delta<1.25^{1}$ | $\delta<1.25^{2}$ | $\delta<1.25^{3}$
$SOccDPT_{V1}$ | IDD | 12* | 0.5 | 0.5 | 0.00001 | 11.2353 | 0.7717 | 0.8991 | 0.9211 | 42.48
BDD | 12* | 0.5 | 0.5 | 0.00001 | 13.3782 | 0.6854 | 0.8442 | 0.9172 | 41.73
$SOccDPT_{V2}$ | IDD | 6 | 0.5 | 0.95 | 0.00001 | 27.6473 | 0.5302 | 0.7084 | 0.8134 | 26.29
BDD | 6 | 0.5 | 0.95 | 0.00001 | 26.2383 | 0.4879 | 0.7181 | 0.8309 | 34.75
$SOccDPT_{V3}$ | IDD | 6 | 0.5 | 0.95 | 0.0001 | 9.1473 | 0.7807 | 0.9009 | 0.9416 | 43.50
BDD | 6 | 0.5 | 0.95 | 0.0001 | 12.4075 | 0.6935 | 0.8588 | 0.9265 | 46.02

TABLE II: Ablation Study SOccDPT’s hyper-parameters and the metrics achieved.
RMSE, a1, a2, a3 are disparity metrics and IoU is the segmentation metric. The
hyper-parameters are batch size (BS), patch-wise percentage (PP), Encoder
Percentage (EP) and learning rate (LR). Models with a * have had their two
heads and backbones trained separately

We train SOccDPT on a laptop with an Intel i7-12700H (20 threads) and NVIDIA
GeForce RTX 3070 Laptop GPU with 8 GB VRAM and utilize PyTorch 2.1.0 [54].
With the goal of focusing performance in unstructured traffic, our network has
been trained on the Indian Driving Dataset [15] and the Bengaluru Driving
Dataset [1]. In Tab. II, we present the set of hyper-parameters which produce
optimal results. We evaluate on the metrics Intersection over Union (IoU),
Root Mean Squared Error (RMSE), threshold errors ($a1$, $a2$, $a3$). Here,
$a1$ is the fraction of predictions where the threshold
${\text{gt}}/{\text{pred}}$ or ${\text{pred}}/{\text{gt}}$ is less than
$1.25$; $a2$ is the fraction of predictions where the threshold is less than
$1.25^{2}$ and $a3$ is the fraction of predictions where the threshold is less
than $1.25^{3}$.

###  IV-B Datasets

Indian Driving Dataset [15] . The IDD has a total of about 7974 frames with
6993 and 981 frames for training and testing respectively.

Bengaluru Driving Dataset [1]. The BDD has a total of about 3629 frames. We
split it to have 10% for testing and the remainder for training.

Bengaluru Semantic Occupancy Dataset (Ours). We extend BDD with 3D semantic
occupancy labels by picking a voxel size of 50 cm and applied a voting filter
to drop the voxels with fewer than 10 points. Since the disparity maps are in
arbitrary scale, we manually estimated the scale of each frame.

###  IV-C Ablation Study

In Tab. II, we present the set of hyper-parameters for SOccDPT’s $V1$, $V2$
and $V3$. We observe that $V1$ produces good disparity and segmentation
metrics while also being the largest network in terms of number of parameters
and the slowest to run as shown in Tab. I. $SOccDPT_{V1}$ has two independent
backbones which explains the larger number of parameters and increased
inference time. While $SOccDPT_{V2}$ shows an improvement in speed and
reduction in number of parameters, it takes a performance hit in terms of
disparity and segmentation accuracy, as this network is being trained from
scratch. $SOccDPT_{V2}$ introduces the common backbone which reduces the
compute requirements, but since this entire network is being trained from
scratch, it has no priors regarding either semantics or disparity estimation.
We introduce this prior into $SOccDPT_{V3}$ by changing the architecture of
$V2$ to allow us to load in pre-trained weights from the disparity backbone.
As seen in Fig. 6, $SOccDPT_{V3}$ starts off with good RMSE scores for
disparity estimation and poor IoU for segmentation, which is as expected.
Through the course of training, the IoU steadily climbs. Initially, we see a
spike in RMSE which comes back down over several epochs. $SOccDPT_{V3}$ has
similar timing and memory characteristics when compared to $SOccDPT_{V2}$ as
it is only a minor modification that allows us to load in the disparity
backbone. But this small change allows $SOccDPT_{V3}$ to vastly outperform
$SOccDPT_{V2}$ without requiring additional training data.

###  IV-D Comparison with Existing Methods

$SOccDPT_{V3}$’s performance exceeds existing disparity estimation approaches
on unstructured traffic scenarios presented from the Bengaluru Driving
Dataset. As shown in Tab. I, $SOccDPT_{V3}$ shows the best accuracy in
disparity estimation while also maintaining a high FPS and keeping compute
requirements low. As shown in Fig. 3, our model provides very detailed
disparity maps compared to existing approaches while performing in real time
and keeping memory requirements low.

##  V CONCLUSIONS

Existing disparity and segmentation approaches have come far, but do not
specifically address the challenge in the autonomous vehicle context in
unstructured traffic scenarios. We use depth boosting and semantic auto-
labelling to build a self-supervised training pipeline, which can take videos
as input and train a 3D semantic occupancy network. $SOccDPT$ uses a multi-
headed Dense Transformer based architecture to take advantage of this self-
supervised pipeline, to learn 3D semantic occupancy in the context of
autonomous navigation in unstructured traffic. Our PatchWise training system
allowed us to explore training with larger batch sizes which would not have
been possible with memory constrained hardware. These models show potential in
their ability to learn 3D semantic occupancy from monocular vision and operate
at real time.

## ACKNOWLEDGMENT

We thank Dhruval Pobbathi Badrinath for the fruitful discussions through the
course of the project.

## References

  * [1] A. N. Ganesh, D. Pobbathi Badrinath, H. M. Kumar, P. S, and S. Narayan, “Octran: 3d occupancy convolutional transformer network in unstructured traffic scenarios,” Spotlight Presentation at the Transformers for Vision Workshop, CVPR, 2023, transformers for Vision Workshop, CVPR 2023. [Online]. Available: https://sites.google.com/view/t4v-cvpr23/papers
  * [2] H. Bao, L. Dong, S. Piao, and F. Wei, “BEit: BERT pre-training of image transformers,” in _International Conference on Learning Representations_ , 2022. [Online]. Available: https://openreview.net/forum?id=p-BhZSz59o4
  * [3] B. Graham, A. El-Nouby, H. Touvron, P. Stock, A. Joulin, H. Jegou, and M. Douze, “Levit: A vision transformer in convnet’s clothing for faster inference,” in _Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)_ , October 2021, pp. 12 259–12 269.
  * [4] A. Jaegle, S. Borgeaud, J.-B. Alayrac, C. Doersch, C. Ionescu, D. Ding, S. Koppula, A. Brock, E. Shelhamer, O. Hénaff, M. M. Botvinick, A. Zisserman, O. Vinyals, and J. Carreira, “Perceiver io: A general architecture for structured inputs & outputs,” 2021.
  * [5] J. Li, X. Xia, W. Li, H. Li, X. Wang, X. Xiao, R. Wang, M. Zheng, and X. Pan, “Next-vit: Next generation vision transformer for efficient deployment in realistic industrial scenarios,” _arXiv preprint arXiv:2207.05501_ , 2022\.
  * [6] Z. Liu, H. Hu, Y. Lin, Z. Yao, Z. Xie, Y. Wei, J. Ning, Y. Cao, Z. Zhang, L. Dong, F. Wei, and B. Guo, “Swin transformer v2: Scaling up capacity and resolution,” in _International Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2022.
  * [7] Z. Liu, Y. Lin, Y. Cao, H. Hu, Y. Wei, Z. Zhang, S. Lin, and B. Guo, “Swin transformer: Hierarchical vision transformer using shifted windows,” in _Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)_ , 2021.
  * [8] A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly _et al._ , “An image is worth 16x16 words: Transformers for image recognition at scale,” _arXiv preprint arXiv:2010.11929_ , 2020.
  * [9] W. K. Fong, R. Mohan, J. V. Hurtado, L. Zhou, H. Caesar, O. Beijbom, and A. Valada, “Panoptic nuscenes: A large-scale benchmark for lidar panoptic segmentation and tracking,” _arXiv preprint arXiv:2109.03805_ , 2021.
  * [10] A. Geiger, P. Lenz, C. Stiller, and R. Urtasun, “Vision meets robotics: the kitti dataset,” _The International Journal of Robotics Research_ , vol. 32, pp. 1231–1237, 09 2013.
  * [11] Y. Liao, J. Xie, and A. Geiger, “KITTI-360: A novel dataset and benchmarks for urban scene understanding in 2d and 3d,” _CoRR_ , vol. abs/2109.13410, 2021. [Online]. Available: https://arxiv.org/abs/2109.13410
  * [12] B. Cheng, O. Parkhi, and A. Kirillov, “Pointly-supervised instance segmentation,” 2021.
  * [13] S. M. H. Miangoleh, “Boosting monocular depth estimation to high resolution,” Master’s thesis, Simon Fraser University, 2022.
  * [14] S. M. H. Miangoleh, S. Dille, L. Mai, S. Paris, and Y. Aksoy, “Boosting monocular depth estimation models to high-resolution via content-adaptive multi-resolution merging,” in _2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2021, pp. 9680–9689.
  * [15] G. Varma, A. Subramanian, A. M. Namboodiri, M. Chandraker, and C. V. Jawahar, “IDD: A dataset for exploring problems of autonomous navigation in unconstrained environments,” _CoRR_ , vol. abs/1811.10200, 2018. [Online]. Available: http://arxiv.org/abs/1811.10200
  * [16] R. Ranftl, K. Lasinger, D. Hafner, K. Schindler, and V. Koltun, “Towards robust monocular depth estimation: Mixing datasets for zero-shot cross-dataset transfer,” _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , vol. 44, no. 3, 2022.
  * [17] S. Liu, W. Ding, C. Liu, Y. Liu, and H. Li, “Ern: Edge loss reinforced semantic segmentation network for remote sensing images,” _Remote Sensing_ , vol. 10, no. 9, p. 1339, 2018.
  * [18] M. Zhen, J. Wang, L. Zhou, S. Li, T. Shen, S. Jiaxiang, T. Fang, and L. Quan, “Joint semantic segmentation and boundary detection using iterative pyramid contexts,” 06 2020, pp. 13 663–13 672.
  * [19] Y. Wang, Y.-H. Tsai, W.-C. Hung, W. Ding, S. Liu, and M.-H. Yang, “Semi-supervised multi-task learning for semantics and depth,” in _2022 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV)_ , 2022, pp. 2663–2672.
  * [20] S. Vandenhende, S. Georgoulis, W. Gansbeke, M. Proesmans, D. Dai, and L. Gool, “Multi-task learning for dense prediction tasks: A survey,” _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , vol. PP, pp. 1–1, 01 2021.
  * [21] Y. Cheng, X. Zhao, R. Cai, Z. Li, K. Huang, and Y. Rui, “Semi-supervised multimodal deep learning for rgb-d object recognition,” in _International Joint Conference on Artificial Intelligence_ , 2016.
  * [22] X. Lu, X. Li, and L. Mou, “Semi-supervised multitask learning for scene recognition,” _IEEE Transactions on Cybernetics_ , vol. 45, no. 9, pp. 1967–1976, 2015.
  * [23] C. Godard, O. Aodha, and G. Brostow, “Unsupervised monocular depth estimation with left-right consistency,” 07 2017.
  * [24] C. Godard, O. Aodha, M. Firman, and G. Brostow, “Digging into self-supervised monocular depth estimation,” 11 2019.
  * [25] Z. Li and N. Snavely, “Megadepth: Learning single-view depth prediction from internet photos,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , June 2018.
  * [26] A. Masoumian, H. A. Rashwan, S. Abdulwahab, J. Cristiano, M. S. Asif, and D. Puig, “Gcndepth: Self-supervised monocular depth estimation based on graph convolutional network,” _Neurocomputing_ , 2022.
  * [27] J. Watson, O. M. Aodha, V. Prisacariu, G. Brostow, and M. Firman, “The temporal opportunist: Self-supervised multi-frame monocular depth,” in _2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_. Los Alamitos, CA, USA: IEEE Computer Society, June 2021, pp. 1164–1174. [Online]. Available: https://doi.ieeecomputersociety.org/10.1109/CVPR46437.2021.00122
  * [28] W.-C. Hung, Y.-H. Tsai, Y.-T. Liou, Y.-Y. Lin, and M.-H. Yang, “Adversarial learning for semi-supervised semantic segmentation,” 2018. [Online]. Available: https://openreview.net/forum?id=SJQO7UJCW
  * [29] Y. Grandvalet and Y. Bengio, “Semi-supervised learning by entropy minimization,” in _Advances in Neural Information Processing Systems_ , L. Saul, Y. Weiss, and L. Bottou, Eds., vol. 17. MIT Press, 2004. [Online]. Available: https://proceedings.neurips.cc/paper_files/paper/2004/file/96f2b50b5d3613adf9c27049b2a888c7-Paper.pdf
  * [30] X. Cheng, Y. Zhong, M. Harandi, Y. Dai, X. Chang, H. Li, T. Drummond, and Z. Ge, “Hierarchical neural architecture search for deep stereo matching,” _Advances in Neural Information Processing Systems_ , vol. 33, 2020.
  * [31] Y. Mao, Z. Liu, W. Li, Y. Dai, Q. Wang, Y.-T. Kim, and H.-S. Lee, “Uasnet: Uncertainty adaptive sampling network for deep stereo matching,” in _2021 IEEE/CVF International Conference on Computer Vision (ICCV)_ , 2021, pp. 6291–6299.
  * [32] Z. Wu, X. Wu, X. Zhang, S. Wang, and L. Ju, “Semantic stereo matching with pyramid cost volumes,” in _2019 IEEE/CVF International Conference on Computer Vision (ICCV)_ , 2019, pp. 7483–7492.
  * [33] G. Xu, X. Wang, X. Ding, and X. Yang, “Iterative geometry encoding volume for stereo matching,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2023, pp. 21 919–21 928.
  * [34] Z. Li, W. Wang, H. Li, E. Xie, C. Sima, T. Lu, Y. Qiao, and J. Dai, “Bevformer: Learning bird’s-eye-view representation from multi-camera images via spatiotemporal transformers,” _arXiv preprint arXiv:2203.17270_ , 2022.
  * [35] L. Reiher, B. Lampe, and L. Eckstein, “A sim2real deep learning approach for the transformation of images from multiple vehicle-mounted cameras to a semantically segmented image in bird’s eye view,” in _2020 IEEE 23rd International Conference on Intelligent Transportation Systems (ITSC)_ , 2020, pp. 1–7.
  * [36] T. Roddick and R. Cipolla, “Predicting semantic map representations from images using pyramid occupancy networks,” 06 2020, pp. 11 135–11 144.
  * [37] Z. Liu, H. Tang, A. Amini, X. Yang, H. Mao, D. Rus, and S. Han, “Bevfusion: Multi-task multi-sensor fusion with unified bird’s-eye view representation,” in _IEEE International Conference on Robotics and Automation (ICRA)_ , 2023\.
  * [38] X. Zhu, H. Zhou, T. Wang, F. Hong, Y. Ma, W. Li, H. Li, and D. Lin, “Cylindrical and asymmetrical 3d convolution networks for lidar segmentation,” in _2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_. Los Alamitos, CA, USA: IEEE Computer Society, June 2021, pp. 9934–9943. [Online]. Available: https://doi.ieeecomputersociety.org/10.1109/CVPR46437.2021.00981
  * [39] Y. Zhou and O. Tuzel, “Voxelnet: End-to-end learning for point cloud based 3d object detection,” in _2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2018, pp. 4490–4499.
  * [40] R. Cheng, R. Razani, E. Taghavi, T. Li, and B. Liu, “(af) 2 -s3net: Attentive feature fusion with adaptive feature selection for sparse semantic segmentation network,” 06 2021, pp. 12 542–12 551.
  * [41] V. E. Liong, T. N. T. Nguyen, S. Widjaja, D. Sharma, and Z. J. Chong, “Amvnet: Assertion-based multi-view fusion network for lidar semantic segmentation,” 2020\.
  * [42] H. Tang, Z. Liu, S. Zhao, Y. Lin, J. Lin, H. Wang, and S. Han, “Searching efficient 3d architectures with sparse point-voxel convolution,” in _Computer Vision - ECCV 2020 - 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XXVIII_ , ser. Lecture Notes in Computer Science, A. Vedaldi, H. Bischof, T. Brox, and J. Frahm, Eds., vol. 12373\. Springer, 2020, pp. 685–702. [Online]. Available: https://doi.org/10.1007/978-3-030-58604-1_41
  * [43] D. Ye, Z. Zhou, W. Chen, Y. Xie, Y. Wang, P. Wang, and H. Foroosh, “Lidarmultinet: Towards a unified multi-task network for lidar perception,” _CoRR_ , vol. abs/2209.09385, 2022. [Online]. Available: https://doi.org/10.48550/arXiv.2209.09385
  * [44] M. Ye, R. Wan, S. Xu, T. Cao, and Q. Chen, “Drinet++: Efficient voxel-as-point point cloud segmentation,” 2021.
  * [45] A.-Q. Cao and R. de Charette, “Monoscene: Monocular 3d semantic scene completion,” in _CVPR_ , 2022.
  * [46] X. Chen, K.-Y. Lin, C. Qian, G. Zeng, and H. Li, “3d sketch-aware semantic scene completion via semi-supervised structure prior,” _2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pp. 4192–4201, 2020.
  * [47] J. Li, K. Han, P. Wang, Y. Liu, and X. Yuan, “Anisotropic convolutional networks for 3d semantic scene completion,” 06 2020, pp. 3348–3356.
  * [48] L. Roldão, R. de Charette, and A. Verroust-Blondet, “Lmscnet: Lightweight multiscale 3d semantic completion,” in _2020 International Conference on 3D Vision (3DV)_ , 2020, pp. 111–119.
  * [49] X. Yan, J. Gao, J. Li, R. Zhang, Z. Li, R. Huang, and S. Cui, “Sparse single sweep lidar point cloud segmentation via learning contextual shape priors from scene completion,” in _AAAI Conference on Artificial Intelligence_ , 2020.
  * [50] Y. Huang, W. Zheng, Y. Zhang, J. Zhou, and J. Lu, “Tri-perspective view for vision-based 3d semantic occupancy prediction,” _arXiv preprint arXiv:2302.07817_ , 2023.
  * [51] R. Ranftl, K. Lasinger, D. Hafner, K. Schindler, and V. Koltun, “Towards robust monocular depth estimation: Mixing datasets for zero-shot cross-dataset transfer,” _IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI)_ , 2020.
  * [52] V. Guizilini, R. Ambrus, S. Pillai, A. Raventos, and A. Gaidon, “3d packing for self-supervised monocular depth estimation,” in _Proceedings of the International Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2020\.
  * [53] V. Guizilini, I. Vasiljevic, D. Chen, R. Ambrus, and A. Gaidon, “Towards zero-shot scale-aware monocular depth estimation,” in _Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)_ , October 2023\.
  * [54] A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga, A. Desmaison, A. Kopf, E. Yang, Z. DeVito, M. Raison, A. Tejani, S. Chilamkurthy, B. Steiner, L. Fang, J. Bai, and S. Chintala, “Pytorch: An imperative style, high-performance deep learning library,” in _Advances in Neural Information Processing Systems 32_. Curran Associates, Inc., 2019, pp. 8024–8035.
  * [55] K. He, G. Gkioxari, P. Dollar, and R. Girshick, “Mask r-cnn,” 10 2017, pp. 2980–2988.