# 2D Supervised Monocular 3D Object Detection
by Global-to-Local 3D Reconstruction

Jiawei He1,2 Yuqi Wang1,2 Yuntao Chen3 Zhaoxiang Zhang1,2,3

1 CRIPAC, Institute of Automation, Chinese Academy of Sciences (CASIA)
2 School of Artificial Intelligence, University of Chinese Academy of Sciences
(UCAS)
3 Centre for Artificial Intelligence and Robotics, HKISI_CAS
{hejiawei2019, wangyuqi2020<EMAIL_ADDRESS><EMAIL_ADDRESS>

https://ba2det.site

###### Abstract

With the advent of the big model era, the demand for data has become more
important. Especially in monocular 3D object detection, expensive manual
annotations potentially limit further developments. Existing works have
investigated weakly supervised algorithms with the help of _LiDAR_ modality to
generate 3D pseudo labels, which cannot be applied to ordinary videos. In this
paper, we propose a novel paradigm, termed as BA2-Det, leveraging the idea of
global-to-local 3D reconstruction for _2D_ supervised monocular 3D object
detection. Specifically, we recover 3D structures from monocular videos by
scene-level global reconstruction with global bundle adjustment (BA) and
obtain object clusters by the DoubleClustering algorithm. Learning from
completely reconstructed objects in global BA, GBA-Learner predicts pseudo
labels for occluded objects. Finally, we train an LBA-Learner with object-
centric local BA to generalize the generated 3D pseudo labels to moving
objects. Experiments on the large-scale Waymo Open Dataset show that the
performance of BA2-Det is on par with the fully-supervised BA-Det trained with
10% videos and even outperforms some pioneer fully-supervised methods. We also
show the great potential of BA2-Det for detecting open-set 3D objects in
complex scenes. The code will be made available.

##  1 Introduction

3D object detection has gained increasing attention from researchers and has
become a fundamental task in real-world perception. Thanks to the efforts of
researchers, 3D object detection using LiDAR and RGBD input has been shown to
be practical in both traffic scenes [43, 56, 8] and indoor scenes [36, 25].
Autonomous vehicles and service robots increasingly rely on 3D object
detection to perceive and understand their environment. In recent years,
detecting 3D objects from images has emerged as a growing area of research.
Due to the cheap cost of the camera sensor, monocular and multi-camera 3D
object detection is assured of a proper place in 3D perception. The
performance of camera-only 3D object detector is also gradually catching up
with LiDAR-based methods.

However, with heavy and expensive manual annotations, the development of
camera-based 3D object detection methods is potentially limited. The existing
work has explored weakly supervised [60, 34] algorithms with the help of LiDAR
to unlock the potential of unlabeled images. However, the camera-only 3D
object label generation method has not been investigated much. In this paper,
we expect to explore camera-only 3D object detection without 3D annotations
and the auxiliary LiDAR data. Considering that 2D object detection and
segmentation are nearly free [17] to obtain, in this paper we assume that we
can get the 2D object labels and simplify the problem as 2D supervised
monocular 3D object detection.

In this paper, we propose a novel paradigm for 2D supervised monocular 3D
object detection, called BA2-Det, by 3D reconstruction from global (scene-
level) to local (object-level) using twice bundle adjustment (BA) technique,
as shown in Fig. 1. We face some core issues step-by-step in this process. The
first issue is: _how to recover the 3D location of each object only with
images._ Given a video taken by a moving camera, the whole scene can be
recovered in 3D by solving global BA. Further more, with 2D bounding boxes in
each frame, we group the reconstructed foreground 3D points into clusters
corresponding to each object. Specifically, we design a two-step clustering
algorithm called _DoubleCluster_ , including intra-frame Local Point
Clustering (LPC) to remove the background points and inter-frame Global Point
Clustering (GPC) to gather the same object into the same cluster.

After obtaining 3D object clusters, here is the next problem: _how to estimate
the 3D bounding boxes from the object clusters._ A simple solution is to fit a
tight 3D bounding box enclosing the object point cluster. However, the
estimated orientation is very unreliable since it is sensitive to the outlier
points. We design a better orientation optimization method by making sure more
reconstructed points are near the edge of the bounding box. Besides, due to
occlusion, the tight 3D bounding boxes are not always perfect. So we learn the
object’s 3D bounding box with a PointNet-like [37] neural network from
completely reconstructed objects, called _GBA-Learner_ , i.e., a learner from
the global BA.

However, many moving objects are not reconstructed in scene-level global 3D
reconstruction. So the last problem is: _how to generalize the generated
static 3D pseudo labels to dynamic objects._ Inspired by BA-Det [10], equipped
with object-centric local BA, we design a monocular 3D object detector (_LBA-
Learner_). We train LBA-Learner with iterative self-retraining by learning
from static objects. With the generalization ability of the object detector
and object-centric local bundle adjustment, we can predict the 3D bounding
boxes for both static and moving objects.

In summary, our main contributions are as follows: (1) We propose a novel
paradigm for 2D supervised monocular 3D object detection, called BA2-Det and
unify the learning of 3D pseudo label generation and monocular 3D object
detection from the perspective of temporal 3D reconstruction. (2) We try to
solve three core problems when learning the 3D object detector without 3D
labels and design three modules: DoubleCluster, GBA-Learner, and LBA-Learner.
(3) We conduct the experiments on the large-scale Waymo Open Dataset (WOD).
The main results and ablation study show the high quality of pseudo labels and
with learning a 3D object detector from global to local, the performance of
BA2-Det is on par with the fully-supervised BA-Det trained with 10% videos and
even outperforms some pioneer fully-supervised methods.

(a) BA-Det [10] (b) BA2-Det (Ours)

Figure 1: A brief introduction of BA2-Det. Compared with BA-Det, to learn
detector without 3D labels, our BA2-Det generates 3D pseudo labels from scene-
level reconstruction using global BA.

##  2 Related Work

###  2.1 Camera-based 3D Object Detection

#### Monocular 3D Object Detection.

Monocular 3D object detection [5, 2, 51, 62] has been explored for several
years, thanks to the ability of the neural network that can estimate
relatively accurate depth from the monocular images. The existing monocular 3D
object detection methods can be divided into three categories: regressing 3D
objects from the image directly, regressing on the depth map or lifted 3D
space, and regressing based on geometric constraints. CenterNet [63] and
FCOS3D [50] are the representing works to estimate the 3D objects with a 3D
regression branch based on the 2D object detectors CenterNet [63] and FCOS
[45], respectively. PL [51] and PL++ [58] use the off-the-shelf dense depth
estimator to project the scene in 3D space and detect objects from pseudo-
LiDAR. D4LCN [7] and PatchNet [28] use the image-aligned depth map to extract
features. E2E-PL [38] and CaDDN [39] jointly learn the depth estimator and 3D
object detector in an end-to-end manner. With the geometric constraints, the
3D object depth can be estimated by solving Perspective-n-Point (PnP) problem.
MonoFlex [62] solves PnP from the vertical lines of 3D bounding boxes.
AutoShape [26] uses semantic points sampled on the CAD model. DCD [18] uses
arbitrary point pairs to construct dense constraints. Recently, there has been
a surge in the development of temporal 3D object detection from monocular
images. DfM [49] and BA-Det [10] aggregate temporal information at the scene
level and in an object-centric manner, inspired by two-view and multi-view
geometry theory, respectively.

#### Multi-camera 3D Object Detection.

Multi-camera 3D object detection takes images from surrounding cameras as
input and mainly detects 3D objects in bird’s eye view (BEV). The methods to
construct BEV features are mainly by lifting image features to 3D, using BEV
grids to query image features, and constructing implicit BEV features on the
images. LSS [35] is the foundation for lifting image features to 3D. BEVDet
[13] lifts 2D image features to frustum features and transform to BEV features
by pooling in height. BEVDepth [20] utilizes LiDAR to explicitly supervise the
depth estimation for better feature lifting. BEVDet4D [12] is a simple
temporal fusion method based on BEVDet by BEV feature concatenation of
different frames. STS [54], BEVStereo [19] utilize temporal geometric
constraints to fuse temporal information. SOLOFusion [32] explores long term
temporal fusion. BEVFormer [21] is an exemplar using BEV grids to query image
features via deformable attention and it construct BEV grids to aggregate
multi-view image features. PolarFormer [15] further generates polar
representation instead of Cartesian coordinates and extract BEV features in
polar coordinates. Constructing implicit BEV features on the images means no
explicit grids in BEV. Inspired by DETR [4], DETR3D [52] constructs 3D object
queries in 3D space and and extract features by attention with reprojected
image features. PETR [23, 24] proposes implicit 3D position-aware features on
the image directly.

###  2.2 Detecting 3D Objects without 3D Labels

Since the 3D object detection task makes great progress in the past few years,
many researchers begin to explore using fewer 3D labels or even without 3D
labels to train a 3D object detector. For LiDAR-based 3D object segmentation,
clustering-based methods [46, 3, 31] are the mainstream methods. LSMOL [53]
and Najibi et al. [30] combine image and LiDAR to segment 2D and 3D objects.
However, only class-agnostic segmentation can be achieved in these methods.
MODEST [57] is an unsupervised 3D mobile object detection method to predict 3D
bounding box. Its key idea is that mobile objects are ephemeral members of a
scene. For image-based 3D object detection, SDFLabel [60] is a pioneer work
that can auto-label the 3D bounding boxes from a pre-trained 2D detector and
the corresponding LiDAR data by recovering the object shape with signed
distance fields (SDF). WeakM3D [34] is also a weakly supervised method and
needs additional LiDAR data. Yang et al. [55] first explore the image-only
weakly supervised without LiDAR. However, box size and orientation cannot be
estimated in this method and can only be learned by ground truth in the semi-
supervised setting. Unlike the above works, our BA2-Det _only uses images_
without LiDAR as an auxiliary modality and can estimate 3D _bounding boxes_
including center position, box size, and the orientation.

##  3 Preliminary: BA-Det

BA-Det [10] is a two-stage 3D object detector with a learnable object-centric
global optimization network. The BA-Det pipeline has two stages. The first
stage is a single-frame monocular 3D object detector. The second stage is an
object-centric temporal correspondence learning (OTCL) module. This module
uses feature-metric object bundle adjustment loss to learn temporal feature
correspondence. During inference, BA-Det uses object-centric bundle adjustment
(OBA) to optimize the object’s pose and 3D box size over time. It takes the
first-stage object prediction and temporal feature correspondence as input.

We notice that the moving objects cannot be generated 3D pseudo labels in the
scene-level global reconstruction. However, because BA-Det can handle both
static and moving objects in an object-centric style, we can design a training
strategy that solves this problem. We first learn the 3D object iteratively to
refine the 3D bounding box using the first stage of BA-Det, and with the help
of object-centric bundle adjustment, the moving objects that do not learn well
can be refined in the second stage of BA-Det.

##  4 Methodology

###  4.1 Problem Setup and Algorithm Overview

#### Problem Setup

In this paper, we present our BA2-Det for detecting 3D objects from monocular
images. The objects are represented as 7-DoF 3D bounding boxes with only yaw
rotation considered. Our detector does not require training with 3D ground
truth or the use of LiDAR data as an auxiliary modality. However, we use
additional 2D labels to separate instances and obtain class labels for each
object. Note that during inference, we do not use any types of labels.

#### Algorithm Overview

We introduce our framework BA2-Det in brief and explain how we design each
module. In general, BA2-Det is a pipeline learning reconstructed 3D objects
from scene-level global reconstruction to object-centric local reconstruction.
As shown in Fig. 2, BA2-Det mainly contains three modules, DoubleCluster for
obtaining 3D object clusters from scene reconstruction (Sec. 4.2), learning 3D
object bounding boxes from clusters (Sec. 4.3), and iterative learning a 3D
object detector based on object-centric local reconstruction (Sec. 4.4).

Figure 2: Pipeline of BA2-Det. We take the video sequence as input. The Global
BA stage is to generate 3D pseudo labels from scene-level global
reconstruction, including DoubleCluster and GBA-Learner. Then the labels are
sent to the Local BA stage, which is to learn a monocular 3D object detector
in an iterative way.

###  4.2 3D Object Clustering from Scene-level Global Reconstruction

With Structure-from-Motion, the 3D scene can be recovered from the ego-motion.
And from the reconstructed scene, with the help of 2D bounding boxes in each
frame, the 3D object point cluster can be constrained in the region of the
frustum intersection. In this section, we design the DoubleCluster algorithm
(Alg. 1) to cluster 3D objects from the 3D reconstructed scene.

We denote the video clip as $\mathcal{V}=\\{\mathbf{I}_{t}|t=1,2\cdots,T\\}$,
keypoints $\mathbf{p}_{t}^{i}=[u_{i},v_{i}]^{\top},(i=1,2,\cdots,n)$ and local
feature $\mathcal{F}_{t}=\\{\mathbf{f}_{t}^{i}\\}$. In this paper, the
keypoint extraction and local feature matching network are based on SuperPoint
[6] and SuperGlue [41]. Given camera intrinsic parameter $\mathbf{K}$ and the
camera extrinsic parameter $\mathbf{T}_{t}=[\mathbf{R}_{t}|\mathbf{t}_{t}]$ in
$t$-th frame, 3D keypoint $\mathbf{P}_{i}$ in the global frame can be
optimized by solving bundle adjustment with projection error

$\\{{\mathbf{P}}^{*}_{i}\\}_{i=1}^{n}=\underset{\\{\mathbf{P}_{i}\\}_{i=1}^{n}}{\arg\min}\frac{1}{2}\sum_{i=1}^{n}\sum_{t=1}^{T}||\mathbf{p}_{i}^{t}-\Pi(\mathbf{T}^{t},\mathbf{P}_{i},\mathbf{K})||^{2},$
(1)

where $\Pi(\cdot)$ is the function projecting the 3D points in the world frame
to the image with the camera’s intrinsic and extrinsic parameters. However,
when the ego-motion is small, the disparity between frames is also small. The
reconstruction can be influenced by the noise. So, when the camera moves at a
low speed, we ignore the video sequence and do not reconstruct the scene.

Algorithm 1 Generating 3D object clusters (DoubleCluster)

1:video clip $\mathcal{V}$, camera intrinsic $\\{\mathbf{K}\\}$, camera pose
$\\{\mathbf{T}_{t}\\}$, 2D bounding box with id
$\\{\mathbf{B}_{j}^{t}\\}_{j=1}^{n_{d}}$

2:3D object cluster $\\{\mathcal{P}_{j}^{*}\\}_{j=1}^{n_{d}}$

3:$\\{{\mathbf{P}}^{*}_{i}\\}_{i=1}^{n}\leftarrow\mathtt{SfM}(\mathcal{V},\\{\mathbf{K}\\},\\{\mathbf{T}_{t}\\})$;

4:$\mathcal{P}\leftarrow\varnothing$;

5:for $j\in[1,n_{d}]$ do $\triangleright$ LPC for each object in each frame

6: for $t\in[1,T]$ do

7:
$\mathcal{P}_{j}^{t}\leftarrow\mathtt{LPC}(\\{{\mathbf{P}}^{*}_{i}\\},\mathbf{B}_{j}^{t})$;

8: $\mathcal{P}\leftarrow\mathcal{P}\cup\mathcal{P}_{j}^{t}$;

9:$\\{\mathcal{P}_{j^{\prime}}\\}_{j^{\prime}=1}^{n^{\prime}_{d}}\leftarrow\mathtt{GPC}(\mathcal{P})$;
$\triangleright$ GPC from $\mathcal{P}$ to generate $n_{d}^{\prime}$ clusters

10:$\\{\mathcal{P}_{j}^{*}\\}_{j=1}^{n_{d}}\leftarrow\mathtt{Match}(\\{\mathcal{P}_{j^{\prime}}\\},\\{\mathbf{B}_{j}^{t}\\})$;
$\triangleright$ Match objects to the point clusters

11:return $\\{\mathcal{P}_{j}^{*}\\}_{j=1}^{n_{d}}$

After the scene-level global reconstruction, we choose the foreground 3D
points that can be reprojected within the 2D bounding boxes, and perform the
Local Point Clustering (LPC) in each frame via connected component algorithm
to choose the biggest 3D point cluster for the object $\mathbf{B}_{j}^{t}$:

$\mathcal{P}_{j}^{t}=\\{{\mathbf{P}}_{i}\\}_{i=1}^{m}=\mathtt{LPC}(\\{\mathbf{P}_{i}^{*}|\Pi(\mathbf{T}_{t},\mathbf{P}_{i}^{*},\mathbf{K})\in\mathbf{b}_{j}^{t}\\}),$
(2)

where we denote $\mathbf{b}_{j}^{t}$ as the region in the 2D bounding box
$\mathbf{B}_{j}^{t}$. The distance threshold of the connected component
algorithm is $\delta_{1}$.

Besides, we cluster the 3D points in the global frame by finding connected
components with distance threshold $\delta_{2}$, called Global Point
Clustering (GPC):

$\\{\mathcal{P}_{j^{\prime}}\\}_{j^{\prime}=1}^{n^{\prime}_{d}}=\mathtt{GPC}(\bigcup_{t,j}\mathcal{P}_{j}^{t}),$
(3)

where $n_{d}^{\prime}$ is the total object clusters. We ignores the cluster
whose point number is lower than the threshold $\theta$. Each cluster can be
matched to the 2D bounding box $\mathbf{B}_{j}^{t}$ by finding the maximum
number of reprojected points within the bounding box.

###  4.3 Generating 3D Object Labels from 3D Object Clusters

As we can separate the 3D objects from the reconstructed scene into object
clusters, then we need to generate 3D pseudo bounding boxes from the object
clusters and learn the 3D object labels, called GBA-Learner. Given the object
cluster $\mathcal{P}_{j}^{*}$ in the global frame, we first fit a tight 3D
bounding box. Since the 3D object detection task only takes into account the
yaw rotation angle, a simple solution would be finding the minimum enclosing
rectangle in the bird’s-eye view and calculating the object’s height based on
the points along the z-axis. However, this method cannot estimate the
orientation well, because the BEV bounding box cannot guarantee the
reconstructed 3D points are near the edges of the bounding box, and so the
orientation can be easily influenced by the noise. Supposed that the
reconstructed points are mainly on the surface of the object and inspired by
Zhang et al. [61], we optimize the orientation $r_{y}$ by minimizing the sum
of distances between the points and their nearest edge and then adjust the
width and length of the bird’s-eye view bounding box to achieve the minimum
area, as

$r_{y}^{*},\mathbf{B}_{bev}^{*}=\mathop{\arg\min}_{r_{y}\in[0,\pi),\mathbf{B}_{bev}\in\mathbb{R}^{2}}\sum_{i=1}^{m}\min_{l\in[1,4]}d(\mathbf{P}_{i},\mathbf{R}(r_{y})\mathbf{B}_{bev}^{l}),$
(4)

where $\mathbf{P}_{i}$ is the 3D point in the object cluster, and
$\mathbf{R}(r_{y})\mathbf{B}_{bev}^{l}$ is the edge of rotated BEV bounding
box with angle $r_{y}$. And the experiments in Sec. 5.4 also verify the
effectiveness of this bounding box fitting method, especially on the
orientation metrics.

However, not all fitted bounding boxes are good enough, because many objects
are occluded or influenced by the outliers and the sizes are smaller or larger
than the ground truth. So, we design a neural network $G_{\theta}$ to learn
and refine the initial 3D bounding box by learning from the completely
reconstructed objects. Network $G_{\theta}$ takes the object cluster as input,
and the coordinates of the 3D points are normalized with the center of the 3D
pseudo bounding box. The network is with a PointNet backbone and a head to
regress the 3D bounding box $[c_{x},c_{y},c_{z},w,h,l,r_{y}]$. To imitate the
points for partial objects caused by occlusion, we conduct cutting-off
augmentation from the complete object, i.e., we randomly cut off some regions
of 3D object points. Note that we only consider the length between
$[\sigma_{0},\sigma_{1}]$ as the complete object and take these 3D pseudo
bounding boxes to supervise $G_{\theta}$.

Note that we can only generate 3D object labels for the static object with
$G_{\theta}$ because the moving objects cannot be reconstructed in the scene-
level global reconstruction. So we need a neural network to generalize to the
moving objects by learning from static objects and to refine the 3D bounding
boxes with object-centric local reconstruction.

###  4.4 Learning 3D Object Detector with Object-centric Local Reconstruction

As 3D pseudo labels are generated and the size and orientation are refined in
the GBA-Learner stage, in this section, we mainly learn and refine the 3D
bounding boxes utilizing the object-centric local reconstruction for better
generalization and object temporal consistency, called LBA-Learner. The 3D
object detector $N_{\theta}$ is based on BA-Det [10]. The detector learning
contains _initial training_ and _iterative self-retraining_.

The initial training stage is to learn a 3D object detector with generated 3D
pseudo labels, which is different from fully supervised methods and the
iterative self-retraining stage. There are two main differences: distance
distribution of the labels and unlabeled 3D objects. 3D pseudo labels have a
wider distance distribution due to the ability of scene-level global
reconstruction to recover 3D object points from an entire video. However, the
ground truth labeled on LiDAR is only from 0 to 75m due to the limited
scanning range. We discuss this problem with experiments in Sec. 5.5. To
achieve better performance, we choose labels in the depth range
$[0.5\text{m},200\text{m}]$ for initial training and
$[0.5\text{m},75\text{m}]$ for iterative self-retraining. As for the problem
of unlabeled 3D objects, we assign labels with 2D GT labels and ignore their
3D losses if without 3D pseudo labels. So the unlabeled objects will not be
the negative samples. This strategy will keep the recall and will refine the
3D position in the iterative self-retraining stage. The verification
experiments are in Sec. 5.4.

Another important difference between our BA2-Det and fully-supervised
monocular 3D object detector is how to supervise the orientation estimation.
The orientation of the 3D pseudo label may have a deviation of $180^{\circ}$,
as it can be difficult to distinguish whether the object is facing forward or
backward. MultiBin Loss [29] is a common practice in fully supervised methods.
We use it with modification to alleviate this problem. We compute two losses
with the original pseudo label and $180^{\circ}$-reversed pseudo label, and
choose the minimum as the final orientation loss. As for the other losses, we
mainly follow BA-Det. Please refer to BA-Det [10] for the details.

Because the 3D pseudo labels are not precise enough and the camera-based 3D
object detector is hard to learn, we iteratively self-retrain the detector
with predictions as the updated pseudo labels. We design two retraining
strategies: keeping the initial 3D pseudo labels and supplementing the high
score predictions iteratively

$\mathcal{D}^{(l)}(X,Y)=N_{\theta}^{(l)}(\widetilde{\mathcal{D}}(X,Y)\cup\mathcal{D}^{(l-1)}(X,Y)),$
(5)

or using generated 3D pseudo labels for initial training and retraining only
with the predictions of the last iteration

$\left\\{\begin{aligned}
\mathcal{D}^{(0)}(X,Y)&=N_{\theta}^{(0)}(\widetilde{\mathcal{D}}(X,Y)),\\\
\mathcal{D}^{(l)}(X,Y)&=N_{\theta}^{(l)}(\mathcal{D}^{(l-1)}(X,Y)),\end{aligned}\right.$
(6)

where $\widetilde{\mathcal{D}}(X,Y)$ are the data with generated pseudo
labels, $\mathcal{D}^{(l)}(X,Y)$ are the data with predicted labels, $(l)$
means the $l$-th self-training iteration. Note that we do not keep the last
network parameters for each self-retraining iteration and train the network
$N_{\theta}$ for the same $\kappa$ epochs. We finally choose the second
retraining strategy for better performance (Sec. 5.5). Note that the
correspondence learning branch in BA-Det needn’t be trained along with the
first-stage object detector. We do not train this branch in the self-
retraining procedure.

##  5 Experiments

###  5.1 Datasets and metrics

To verify our proposed BA2-Det, we conduct our ablation studies and comparison
experiments with other methods on the large-scale autonomous driving dataset,
Waymo Open Dataset (WOD) [44]. WOD is the mainstream 3D object detection
benchmark, containing 1150 video sequences, 798 for training, 202 for
validation, and 150 for testing. Only objects within 75m that can be scanned
by LiDAR have 3D labels. To keep the same experiment settings as other
methods, we mainly report the results on WOD v1.2. The evaluation metrics for
camera-based 3D object detection are 3D AP and LET-3D AP [14]. 3D AP is a
common metric for both camera and LiDAR-based 3D object detection. LET-3D AP
is specifically designed for camera-only 3D object detection. Because the
camera-based 3D object detector has a natural weakness in depth estimation,
LET-3D AP is much looser for longitudinal localization and uses Longitudinal
Error Tolerant IoU (LET-IoU) instead of the original IoU as the criterion.
Following the existing camera-based 3D object detection methods, we mainly
report the results of the VEHICLE class on the FRONT camera. For 3D AP and 3D
APH, we choose a loose IoU threshold of 0.05 and a common one of 0.5, called
AP${}_{\text{5}}$ and AP${}_{\text{50}}$. For LET-3D metrics, we report the
results under the official IoU threshold of 0.5. The official MOT evaluation
metrics are CLEAR MOT metrics [1]. MOTA is the main metric to evaluate the
number of false positives, false negatives, and mismatches. The number of
mismatches is the most important to evaluate the data association performance.

###  5.2 Implementation Details

#### Architecture.

The main 3D object detector architecture we used follows BA-Det [10]. We use a
DLA-34 [59] as the backbone without an FPN neck, and the head is with 2 layers
of 3$\times$3 convolutions and MLP. The resolution of the input images is
1920$\times$1280\. If the input size is smaller than it, we will use zero
padding to complete the image.

#### Label generation and model training.

The scene reconstruction is based on hloc [40]
framework111https://github.com/cvg/Hierarchical-Localization. The distance
threshold $\delta_{1}$ and $\delta_{2}$ in DoubleCluster are 0.5 and 0.7. We
keep the object cluster for more than $\theta=100$ points. The size threshold
$\sigma_{0}=3m$ and $\sigma_{1}=10m$. Our implementation is based on the
PyTorch [33] framework. We train our model on 8 NVIDIA RTX 4090 GPUs. Adam
[16] optimizer is applied with $\beta_{1}=0.9$ and $\beta_{2}=0.999$. The
learning rate is 8$\times$10-5 and weight decay is 10-5. We train 1 epoch for
network $G_{\theta}$ and $\kappa=12$ epochs for the 3D object detector
$N_{\theta}$. The loss weights are the same as BA-Det. We finally iterative
self-retrain $N_{\theta}$ for 2 iterations.

###  5.3 Main Results

Table 1: The main results on WOD _val_ set. ‘3D Sup.’ means the ratio of video
sequences with 3D labels. ${\dagger}$: using 100% 3D labeled sequences but
only trained with 1/3 frames.

Method | 3D Sup. | 3D AP${}_{\text{5}}$ | 3D APH${}_{\text{5}}$ | 3D AP${}_{\text{50}}$ | 3D APH${}_{\text{50}}$ | LET APL${}_{\text{50}}$ | LET AP${}_{\text{50}}$ | LET APH${}_{\text{50}}$
---|---|---|---|---|---|---|---|---
PatchNet [28] | 100%† | - | - | 2.92 | 2.74 | - | - | -
M3D-RPN [2] | 100%† | - | - | 3.79 | 3.63 | - | - | -
PCT [47] | 100%† | - | - | 4.20 | 4.15 | - | - | -
MonoJSG [22] | 100%† | - | - | 5.65 | 5.47 | - | - | -
GUPNet [27] | 100%† | - | - | 10.02 | 9.94 | - | - | -
BA-Det [10] Stage 1 | 100% | 70.33 | 69.41 | 34.70 | 34.43 | 50.63 | 67.30 | 66.50
BA-Det [10] Stage 1 | 10% | 53.68 | 52.30 | 15.44 | 15.22 | 28.21 | 44.21 | 43.23
BA-Det [10] | 100% | 72.96 | 71.78 | 40.93 | 40.51 | 54.45 | 68.32 | 67.36
BA-Det [10] | 10% | 57.29 | 55.27 | 19.70 | 19.27 | 32.53 | 46.91 | 45.52
SfM [42]+BA-Det [10] | 0% | 27.84 | 8.80 | 2.89 | 0.75 | 7.34 | 10.75 | 3.31
BA2-Det(Ours) | 0% | 60.01 | 44.81 | 10.39 | 8.98 | 22.24 | 32.60 | 23.86

Table 2: The detailed results in different depth ranges (meters) on WOD _val_
set.

Method | 3D Sup. | 3D AP5 | 3D APH5 | LET APL${}_{\text{50}}$ | LET AP${}_{\text{50}}$
---|---|---|---|---|---
0-30 | 30-50 | 50-$\infty$ | 0-30 | 30-50 | 50-$\infty$ | 0-30 | 30-50 | 50-$\infty$ | 0-30 | 30-50 | 50-$\infty$
BA-Det [10] | 100% | 87.80 | 72.52 | 48.45 | 86.91 | 71.52 | 46.98 | 66.15 | 57.97 | 36.44 | 82.74 | 69.58 | 45.77
BA-Det [10] | 10% | 73.25 | 54.00 | 34.50 | 71.38 | 52.22 | 32.53 | 38.31 | 35.57 | 22.40 | 56.98 | 47.28 | 31.11
SfM [42]+BA-Det [10] | 0% | 46.87 | 25.88 | 9.09 | 14.26 | 8.86 | 2.84 | 11.35 | 7.74 | 2.60 | 17.59 | 10.12 | 3.48
BA2-Det (Ours) | 0% | 77.38 | 54.95 | 33.74 | 64.54 | 37.57 | 21.64 | 25.00 | 23.97 | 14.63 | 39.24 | 31.73 | 20.30

As shown in Table 1, we show the main results compared with fully-supervised
BA-Det, a simple solution combining SfM and BA-Det, and some other fully
supervised methods. Especially, to make a clear understanding of our results,
we also compare the results of BA-Det trained with fewer data. We find that
with a loose IoU threshold (0.05), our BA2-Det can outperform fully-supervised
BA-Det with 10% data by 2.7 AP and is close to the 100% data (with a ~12 AP
gap). As for the 0.5 IoU threshold, we can beat some other fully-supervised
methods, such as PCT and MonoJSG. Compared with our baseline method, using SfM
and clustering to fitting 3D labels and learning a BA-Det, our BA2-Det have
huge gains on all metrics. The detailed results in different depth ranges are
shown in Table 2. Compared with the baseline, we have a more significant gain
for the objects far off.

In Table 3, we show the 3D MOT results on WOD. We compare the proposed BA2-Det
with other SOTA monocular 3D MOT methods on WOD. All results are reported in
Vehicle LEVEL 2 difficulty. 50 and 30 in the metrics are for the IoU threshold
of 0.5 and 0.3. Note that the SOTA methods are both fully supervised, learning
with 3D ground truth. Our BA2-Det is a 2D supervised 3D multiple object
tracking method. Even though, BA2-Det is also comparable to these fully
supervised ones. Especially for MOTA50 metric, we outperform QD-3DT [11]. For
Mismatch, we only have less than 1/100 identity switches compared with the
SOTA methods. Compared with the baseline method, using SfM to generate pseudo
labels, BA-Det as the 3D object detector, and ImmortalTracker to track 3D
objects, our method BA2-Det improve the performance by a large margin.

Table 3: Comparisons with SOTA methods on WOD for 3D MOT.

| Fully Sup. | MOTA50 $\uparrow$ | Mismatch50 $\downarrow$ | MOTA30 $\uparrow$ | Mismatch30 $\downarrow$
---|---|---|---|---|---
QD-3DT [11] | ✓ | 0.0308 | 0.00550 | 0.1867 | 0.01340
CC-3DT [9] | ✓ | 0.0480 | 0.00180 | 0.2032 | 0.00690
SfM [42]+BA-Det [10]+Immortal [48] |  | 0.0011 |  $<$0.00001 | 0.0652 | 0.00038
BA2-Det (Ours) |  | 0.0352 | 0.00002 | 0.1522 | 0.00008

###  5.4 Ablation Study

Table 4: The ablation study about each component of BA2-Det.

$N_{\theta}$ w/ 3D |  $N_{\theta}$ w/ 2D | $G_{\theta}$ |  $r_{y}$ w/ $d$ | Iter. | OBA | 3D AP${}_{\text{5}}$ | 3D APH${}_{\text{5}}$ | LET APL${}_{\text{50}}$ | LET AP${}_{\text{50}}$
---|---|---|---|---|---|---|---|---|---
✓ |  |  |  |  |  | 20.97 | 6.70 | 4.27 | 7.28
| ✓ |  |  |  |  | 28.40 | 11.34 | 5.02 | 8.62
| ✓ | ✓ |  |  |  | 33.75 | 11.94 | 9.63 | 16.80
| ✓ | ✓ | ✓ |  |  | 41.17 | 28.73 | 12.23 | 21.41
| ✓ | ✓ | ✓ | ✓ |  | 56.33 | 42.05 | 17.87 | 29.62
| ✓ | ✓ | ✓ | ✓ | ✓ | 60.01 | 44.81 | 22.24 | 32.60

We conduct the ablation study on WOD val set. The results are shown in Table
4. ‘$N_{\theta}$ w/ 3D’ and ‘$N_{\theta}$ w/ 2D’ mean we assign labels based
on 3D or 2D labels (Sec. 4.4). We finally choose to assign labels with 2D
labels and ignore the 3D loss if the 3D labels cannot be generated, which is
7.4 AP higher than only assigning labels when the 3D label exists.
$G_{\theta}$ is the GBA-Learner, learning a complete 3D box from the partially
reconstructed object, that can obtain a 5.4 AP gain. ‘$r_{y}$ w/ $d$’ is the
orientation optimization method minimizing the distances from points to the
edge (Sec. 4.3), improving 16.8 APH by refining the orientation and box size.
iterative self-retraining (Iter.) has the greatest gain of 15.2 AP. More
experiments and discussions about iterative self-retraining are in Sec. 5.5.
With object-centric bundle adjustment (OBA), the 3D bounding boxes are refined
in temporal, which improves 3.7 AP.

###  5.5 Discussions

#### Self-retraining iteration.

Table 5: Discussion about self-retraining iterations.

Iter. | 3D AP${}_{\text{5}}$ | 3D APH${}_{\text{5}}$ | LET APL${}_{\text{50}}$ | LET AP${}_{\text{50}}$
---|---|---|---|---
0 | 41.17 | 28.73 | 12.23 | 21.41
1 | 51.81 | 38.66 | 17.42 | 28.57
2 | 56.33 | 42.05 | 17.87 | 29.62
3 | 56.70 | 42.27 | 16.87 | 28.06
4 | 57.12 | 42.42 | 16.64 | 27.73

We discuss how many iterations does self-retraining stage need. In Table 5, we
conduct the experiments of a maximum of 4 iterations. The first iteration can
bring the greatest benefit of 10.7 AP and the benefits are progressively
decreased. After the second iteration, the performance is near the highest
performance, which shows fast convergence of the self-retraining stage.

#### Retraining strategy.

Table 6: Ablation study of self-retraining strategy.

Strategy | 3D AP${}_{\text{5}}$ | 3D APH${}_{\text{5}}$ | LET APL${}_{\text{50}}$ | LET AP${}_{\text{50}}$
---|---|---|---|---
w/o iter. | 41.17 | 28.73 | 12.23 | 21.41
Eq. 5 | 39.71$\downarrow{1.5}$ | 29.79$\uparrow{1.0}$ | 12.75$\uparrow{0.5}$ | 21.08$\downarrow{0.3}$
Eq. 6 | 51.81$\uparrow{10.6}$ | 38.66$\uparrow{9.9}$ | 17.42$\uparrow{5.2}$ | 28.57$\uparrow{7.2}$

In Sec. 4.4, we propose two self-retraining strategies (Eq. 5 and Eq. 6). In
Table 6, we show the experiment results to retain 1 iteration. The latter is
better than the former strategy. The combination of pseudo-labels and
predictions may introduce two inconsistent data distributions, causing poor
results. The results in Table 4 and Table 5 also indicate the importance of
the chosen self-retraining strategy.

#### Depth threshold settings.

Table 7: Experiments on different depth threshold settings.

Iter. | Depth (m) | 3D AP${}_{\text{5}}$ | 3D APH${}_{\text{5}}$ | LET APL${}_{\text{50}}$ | LET AP${}_{\text{50}}$
---|---|---|---|---|---
0 | $[0.5,75]$ | 22.24 | 7.66 | 4.83 | 8.56
0 | all, $[0.5,200]$ | 33.75 | 11.94 | 9.63 | 16.80
1 | $[0.5,75]$ | 42.07 | 15.00 | 12.32 | 20.48
1 | all, $[0.5,200]$ | 40.60 | 14.49 | 12.18 | 20.05

By utilizing reconstruction, we can generate labels for objects that are
farther away, and thus the depth distribution is different from ground truth,
shown in Fig. 3. According to experiments (Table 7), we find that when we
initially train the object detector, we need these farther objects, and for
iteratively self-retraining, we can only train with the same depth ranges as
the ground truth (0-75m).

#### Generalization to unlabeled objects.

Figure 3: Depth distributions of ground truth and pseudo labels.

In Table 8, we validate the performance of generalizing 3D location for
unlabeled objects. The metrics follow depth estimation but on the object
level. We find that only 47.8% objects can be generated 3D pseudo labels
directly. The others are either heavily occluded or moving. The small gap
between labeled and unlabeled objects in the results shows the generalization
of our LBA-Learner module to the unlabeled dynamic objects.

Table 8: Genralization ability of 3D localization on WOD training set. ‘w/o
p.l.’: objects without pseudo label, ‘w/ p.l.’: objects with pseudo label.

| Ratio | $\delta<1.25\uparrow$ | $\delta<1.25^{2}\uparrow$ | $\delta<1.25^{3}\uparrow$ | Abs Rel$\downarrow$ | Sq Rel$\downarrow$ | RMSE$\downarrow$ | RMSE log$\downarrow$
---|---|---|---|---|---|---|---|---
w/o p.l. | 52.2% | 0.991 | 0.994 | 0.995 | 0.055 | 0.324 | 2.773 | 0.100
w/ p.l. | 47.8% | 0.995 | 0.996 | 0.997 | 0.049 | 0.117 | 1.726 | 0.077
All | 100% | 0.992 | 0.995 | 0.996 | 0.053 | 0.266 | 2.524 | 0.094

#### Open-set 3D object detection.

We also show the ability to detect open-set 3D objects in complex scenes with
SAM [17] instead of 2D ground truth. Please refer to the demos in the
supplementary materials.

##  6 Qualitative Results

We show some qualitative results about 3D object detection and tracking
(BA2-Det), open-set 3D object detection, and 2D MOT with auxiliary 3D
representation (BA2-Track). For more qualitative results and video demos,
please refer to the project page: https://ba2det.site.

###  6.1 3D Object Detection and MOT Results on WOD

In Fig. 4, we show the qualitative results of BA-Det (trained with 10% labeled
videos), the baseline method, and the proposed BA2-Det. Our method can achieve
comparable performance with fully supervised BA-Det, and even better in some
near cases. Compared with the baseline, a very obvious phenomenon is that our
recall can be much better than the baseline method, mainly due to the
iterative self-retraining design. The illustrations also show a typical
failure case of BA2-Det that on a distance of about 75m, there are some false
positives. This is because the 3D pseudo labels can be 0-200m and thus
somewhat affects the training process. If the annotations include some farther
objects, this problem may be alleviated.

(a) BA-Det (10% labeled videos). (b) SfM + BA-Det (Baseline).

(c) BA2-Det (Ours).

Figure 4: Qualitative results of 3D object detection and tracking shown in
BEV. Black boxes are the ground truth, cyan boxes are the tracking results
with id, green boxes are the detection results with scores, red boxes are the
false positives.

###  6.2 Open-Set 3D Object Detection with SAM

In fig. 5, we also show the ability to detect open-set 3D objects in complex
scenes with SAM [17] instead of 2D ground truth. We click the objects to
generate the 2D masks in SAM. Please refer to the detailed video demos from
the project page.

(a) Input image examples.

(b) Detected 3D boxes from the video.

Figure 5: Open-set 3D object detection from a video. For the video demos for
3D box generation, please refer to the project page.

##  7 Conclusion

In this paper, we propose BA2-Det, a novel paradigm for 2D supervised
monocular 3D object detection. The key idea of BA2-Det is to generate 3D
pseudo labels and learn a 3D object detector from scene-level global
reconstruction and object-centric local reconstruction. Specifically, the
pipeline of BA2-Det contains three parts: DoubleCluster algorithm to cluster
object clusters from the reconstructed 3D scene, 3D object label generation
with GBA-Learner, and 3D object detector GBA-Learner generalizing the pseudo
labels to unlabeled dynamic objects. Experiments on the large-scale Waymo Open
Dataset show that the performance of BA2-Det is on par with the fully-
supervised BA-Det trained with 10% videos and even outperforms some pioneer
fully-supervised methods. We also show the potential of BA2-Det for detecting
open-set 3D objects in complex scenes.

## References

  * [1] Keni Bernardin and Rainer Stiefelhagen.  Evaluating multiple object tracking performance: the clear mot metrics.  EURASIP Journal on Image and Video Processing, 2008:1–10, 2008\.
  * [2] Garrick Brazil and Xiaoming Liu.  M3d-rpn: Monocular 3d region proposal network for object detection.  In ICCV, 2019.
  * [3] Ricardo JGB Campello, Davoud Moulavi, and Jörg Sander.  Density-based clustering based on hierarchical density estimates.  In KDD, 2013.
  * [4] Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko.  End-to-end object detection with transformers.  In ECCV, 2020.
  * [5] Xiaozhi Chen, Kaustav Kundu, Ziyu Zhang, Huimin Ma, Sanja Fidler, and Raquel Urtasun.  Monocular 3d object detection for autonomous driving.  In CVPR, 2016.
  * [6] Daniel DeTone, Tomasz Malisiewicz, and Andrew Rabinovich.  Superpoint: Self-supervised interest point detection and description.  In CVPR Workshops, 2018.
  * [7] Mingyu Ding, Yuqi Huo, Hongwei Yi, Zhe Wang, Jianping Shi, Zhiwu Lu, and Ping Luo.  Learning depth-guided convolutions for monocular 3d object detection.  In CVPR, 2020.
  * [8] Lue Fan, Yuxue Yang, Feng Wang, Naiyan Wang, and Zhaoxiang Zhang.  Super sparse 3d object detection.  arXiv preprint arXiv:2301.02562, 2023.
  * [9] Tobias Fischer, Yung-Hsu Yang, Suryansh Kumar, Min Sun, and Fisher Yu.  CC-3DT: Panoramic 3d object tracking via cross-camera fusion.  In CoRL, 2022.
  * [10] Jiawei He, Yuntao Chen, Naiyan Wang, and Zhaoxiang Zhang.  3d video object detection with learnable object-centric global optimization.  In CVPR, 2023.
  * [11] Hou-Ning Hu, Yung-Hsu Yang, Tobias Fischer, Trevor Darrell, Fisher Yu, and Min Sun.  Monocular quasi-dense 3d object tracking.  IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(2):1992–2008, 2022.
  * [12] Junjie Huang and Guan Huang.  Bevdet4d: Exploit temporal cues in multi-camera 3d object detection.  arXiv preprint arXiv:2203.17054, 2022.
  * [13] Junjie Huang, Guan Huang, Zheng Zhu, Yun Ye, and Dalong Du.  Bevdet: High-performance multi-camera 3d object detection in bird-eye-view.  arXiv preprint arXiv:2112.11790, 2021.
  * [14] Wei-Chih Hung, Henrik Kretzschmar, Vincent Casser, Jyh-Jing Hwang, and Dragomir Anguelov.  Let-3d-ap: Longitudinal error tolerant 3d average precision for camera-only 3d detection.  arXiv preprint arXiv:2206.07705, 2022.
  * [15] Yanqin Jiang, Li Zhang, Zhenwei Miao, Xiatian Zhu, Jin Gao, Weiming Hu, and Yu-Gang Jiang.  Polarformer: Multi-camera 3d object detection with polar transformers.  In AAAI, 2023.
  * [16] Diederik P Kingma and Jimmy Ba.  Adam: A method for stochastic optimization.  arXiv preprint arXiv:1412.6980, 2014.
  * [17] Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan-Yen Lo, et al.  Segment anything.  arXiv preprint arXiv:2304.02643, 2023.
  * [18] Yingyan Li, Yuntao Chen, Jiawei He, and Zhaoxiang Zhang.  Densely constrained depth estimator for monocular 3d object detection.  In ECCV, 2022.
  * [19] Yinhao Li, Han Bao, Zheng Ge, Jinrong Yang, Jianjian Sun, and Zeming Li.  Bevstereo: Enhancing depth estimation in multi-view 3d object detection with dynamic temporal stereo.  In AAAI, 2023.
  * [20] Yinhao Li, Zheng Ge, Guanyi Yu, Jinrong Yang, Zengran Wang, Yukang Shi, Jianjian Sun, and Zeming Li.  Bevdepth: Acquisition of reliable depth for multi-view 3d object detection.  In AAAI, 2023.
  * [21] Zhiqi Li, Wenhai Wang, Hongyang Li, Enze Xie, Chonghao Sima, Tong Lu, Yu Qiao, and Jifeng Dai.  Bevformer: Learning bird’s-eye-view representation from multi-camera images via spatiotemporal transformers.  In ECCV, 2022.
  * [22] Qing Lian, Peiliang Li, and Xiaozhi Chen.  Monojsg: Joint semantic and geometric cost volume for monocular 3d object detection.  In CVPR, 2022.
  * [23] Yingfei Liu, Tiancai Wang, Xiangyu Zhang, and Jian Sun.  Petr: Position embedding transformation for multi-view 3d object detection.  In ECCV, 2022.
  * [24] Yingfei Liu, Junjie Yan, Fan Jia, Shuailin Li, Qi Gao, Tiancai Wang, Xiangyu Zhang, and Jian Sun.  Petrv2: A unified framework for 3d perception from multi-camera images.  arXiv preprint arXiv:2206.01256, 2022.
  * [25] Ze Liu, Zheng Zhang, Yue Cao, Han Hu, and Xin Tong.  Group-free 3d object detection via transformers.  In ICCV, 2021.
  * [26] Zongdai Liu, Dingfu Zhou, Feixiang Lu, Jin Fang, and Liangjun Zhang.  Autoshape: Real-time shape-aware monocular 3d object detection.  In ICCV, 2021.
  * [27] Yan Lu, Xinzhu Ma, Lei Yang, Tianzhu Zhang, Yating Liu, Qi Chu, Junjie Yan, and Wanli Ouyang.  Geometry uncertainty projection network for monocular 3d object detection.  In ICCV, 2021.
  * [28] Xinzhu Ma, Shinan Liu, Zhiyi Xia, Hongwen Zhang, Xingyu Zeng, and Wanli Ouyang.  Rethinking pseudo-lidar representation.  In ECCV, 2020.
  * [29] Arsalan Mousavian, Dragomir Anguelov, John Flynn, and Jana Kosecka.  3d bounding box estimation using deep learning and geometry.  In CVPR, 2017.
  * [30] Mahyar Najibi, Jingwei Ji, Yin Zhou, Charles R Qi, Xinchen Yan, Scott Ettinger, and Dragomir Anguelov.  Motion inspired unsupervised perception and prediction in autonomous driving.  In ECCV, 2022.
  * [31] Lucas Nunes, Xieyuanli Chen, Rodrigo Marcuzzi, Aljosa Osep, Laura Leal-Taixé, Cyrill Stachniss, and Jens Behley.  Unsupervised class-agnostic instance segmentation of 3d lidar data for autonomous vehicles.  IEEE Robotics and Automation Letters, 7(4):8713–8720, 2022.
  * [32] Jinhyung Park, Chenfeng Xu, Shijia Yang, Kurt Keutzer, Kris Kitani, Masayoshi Tomizuka, and Wei Zhan.  Time will tell: New outlooks and a baseline for temporal multi-view 3d object detection.  In ICLR, 2023.
  * [33] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al.  Pytorch: An imperative style, high-performance deep learning library.  NeurIPS, 2019.
  * [34] Liang Peng, Senbo Yan, Boxi Wu, Zheng Yang, Xiaofei He, and Deng Cai.  Weakm3d: Towards weakly supervised monocular 3d object detection.  arXiv preprint arXiv:2203.08332, 2022.
  * [35] Jonah Philion and Sanja Fidler.  Lift, splat, shoot: Encoding images from arbitrary camera rigs by implicitly unprojecting to 3d.  In ECCV, 2020.
  * [36] Charles R Qi, Or Litany, Kaiming He, and Leonidas J Guibas.  Deep hough voting for 3d object detection in point clouds.  In ICCV, 2019.
  * [37] Charles R Qi, Hao Su, Kaichun Mo, and Leonidas J Guibas.  Pointnet: Deep learning on point sets for 3d classification and segmentation.  In CVPR, 2017.
  * [38] Rui Qian, Divyansh Garg, Yan Wang, Yurong You, Serge Belongie, Bharath Hariharan, Mark Campbell, Kilian Q Weinberger, and Wei-Lun Chao.  End-to-end pseudo-lidar for image-based 3d object detection.  In CVPR, 2020.
  * [39] Cody Reading, Ali Harakeh, Julia Chae, and Steven L Waslander.  Categorical depth distribution network for monocular 3d object detection.  In CVPR, 2021.
  * [40] Paul-Edouard Sarlin, Cesar Cadena, Roland Siegwart, and Marcin Dymczyk.  From coarse to fine: Robust hierarchical localization at large scale.  In CVPR, 2019.
  * [41] Paul-Edouard Sarlin, Daniel DeTone, Tomasz Malisiewicz, and Andrew Rabinovich.  Superglue: Learning feature matching with graph neural networks.  In CVPR, 2020.
  * [42] Johannes L Schonberger and Jan-Michael Frahm.  Structure-from-motion revisited.  In CVPR, 2016.
  * [43] Shaoshuai Shi, Chaoxu Guo, Li Jiang, Zhe Wang, Jianping Shi, Xiaogang Wang, and Hongsheng Li.  Pv-rcnn: Point-voxel feature set abstraction for 3d object detection.  In CVPR, 2020.
  * [44] Pei Sun, Henrik Kretzschmar, Xerxes Dotiwalla, Aurelien Chouard, Vijaysai Patnaik, Paul Tsui, James Guo, Yin Zhou, Yuning Chai, Benjamin Caine, et al.  Scalability in perception for autonomous driving: Waymo open dataset.  In CVPR, 2020.
  * [45] Zhi Tian, Chunhua Shen, Hao Chen, and Tong He.  Fcos: Fully convolutional one-stage object detection.  In ICCV, 2019.
  * [46] Rudolph Triebel, Jiwon Shin, and Roland Siegwart.  Segmentation and unsupervised part-based discovery of repetitive objects.  In RSS, 2010.
  * [47] Li Wang, Li Zhang, Yi Zhu, Zhi Zhang, Tong He, Mu Li, and Xiangyang Xue.  Progressive coordinate transforms for monocular 3d object detection.  NeurIPS, 2021.
  * [48] Qitai Wang, Yuntao Chen, Ziqi Pang, Naiyan Wang, and Zhaoxiang Zhang.  Immortal tracker: Tracklet never dies.  arXiv preprint arXiv:2111.13672, 2021.
  * [49] Tai Wang, Jiangmiao Pang, and Dahua Lin.  Monocular 3d object detection with depth from motion.  In ECCV, 2022.
  * [50] Tai Wang, Xinge Zhu, Jiangmiao Pang, and Dahua Lin.  Fcos3d: Fully convolutional one-stage monocular 3d object detection.  In ICCV, 2021.
  * [51] Yan Wang, Wei-Lun Chao, Divyansh Garg, Bharath Hariharan, Mark Campbell, and Kilian Q Weinberger.  Pseudo-lidar from visual depth estimation: Bridging the gap in 3d object detection for autonomous driving.  In CVPR, 2019.
  * [52] Yue Wang, Vitor Campagnolo Guizilini, Tianyuan Zhang, Yilun Wang, Hang Zhao, and Justin Solomon.  Detr3d: 3d object detection from multi-view images via 3d-to-2d queries.  In CoRL, 2022.
  * [53] Yuqi Wang, Yuntao Chen, and ZHAO-XIANG ZHANG.  4d unsupervised object discovery.  In NeurIPS, 2022.
  * [54] Zengran Wang, Chen Min, Zheng Ge, Yinhao Li, Zeming Li, Hongyu Yang, and Di Huang.  Sts: Surround-view temporal stereo for multi-view 3d detection.  arXiv preprint arXiv:2208.10145, 2022.
  * [55] Jinrong Yang, Tiancai Wang, Zheng Ge, Weixin Mao, Xiaoping Li, and Xiangyu Zhang.  Towards 3d object detection with 2d supervision.  arXiv preprint arXiv:2211.08287, 2022.
  * [56] Tianwei Yin, Xingyi Zhou, and Philipp Krahenbuhl.  Center-based 3d object detection and tracking.  In CVPR, 2021.
  * [57] Yurong You, Katie Luo, Cheng Perng Phoo, Wei-Lun Chao, Wen Sun, Bharath Hariharan, Mark Campbell, and Kilian Q Weinberger.  Learning to detect mobile objects from lidar scans without labels.  In CVPR, 2022.
  * [58] Yurong You, Yan Wang, Wei-Lun Chao, Divyansh Garg, Geoff Pleiss, Bharath Hariharan, Mark Campbell, and Kilian Q. Weinberger.  Pseudo-lidar++: Accurate depth for 3d object detection in autonomous driving.  In ICLR, 2020.
  * [59] Fisher Yu, Dequan Wang, Evan Shelhamer, and Trevor Darrell.  Deep layer aggregation.  In CVPR, 2018.
  * [60] Sergey Zakharov, Wadim Kehl, Arjun Bhargava, and Adrien Gaidon.  Autolabeling 3d objects with differentiable rendering of sdf shape priors.  In CVPR, 2020.
  * [61] Xiao Zhang, Wenda Xu, Chiyu Dong, and John M Dolan.  Efficient l-shape fitting for vehicle detection using laser scanners.  In IV, 2017.
  * [62] Yunpeng Zhang, Jiwen Lu, and Jie Zhou.  Objects are different: Flexible monocular 3d object detection.  In CVPR, 2021.
  * [63] Xingyi Zhou, Dequan Wang, and Philipp Krähenbühl.  Objects as points.  arXiv preprint arXiv:1904.07850, 2019.