# ActiveAnno3D - An Active Learning Framework for
Multi-Modal 3D Object Detection

Ahmed Ghita1∗ ID , Bjørk Antoniussen2,3∗ ID , Walter Zimmer1∗ ID , Ross
Greer2∗ ID , Christian Creß1∗ ID ,
Andreas Møgelmose3 ID , Mohan M. Trivedi2 ID , and Alois C. Knoll1 ID 1A.
Ghita, W. Zimmer, C. Creß, and A. Knoll are with the Chair of Robotics and
Artificial Intelligence at the Technical University of Munich. Corresponding
author: W. Zimmer<EMAIL_ADDRESS>Antoniussen, R. Greer, and M.
Trivedi are with the Laboratory for Intelligent and Safe Automobiles at the
University of California San Diego.3B. Antoniussen and A. Møgelmose are with
the Visual Analysis and Perception Lab at Aalborg Universitet.∗Authors
contributed equally.

###### Abstract

The curation of large-scale datasets is still costly and requires much time
and resources. Data is often manually labeled, and the challenge of creating
high-quality datasets remains. In this work, we fill the research gap using
active learning for multi-modal 3D object detection. We propose ActiveAnno3D,
an active learning framework to select data samples for labeling that are of
maximum informativeness for training. We explore various continuous training
methods and integrate the most efficient method regarding computational demand
and detection performance. Furthermore, we perform extensive experiments and
ablation studies with BEVFusion and PV-RCNN on the nuScenes and TUM Traffic
Intersection dataset. We show that we can achieve almost the same performance
with PV-RCNN and the entropy-based query strategy when using only half of the
training data (77.25 mAP compared to 83.50 mAP) of the TUM Traffic
Intersection dataset. BEVFusion achieved an mAP of 64.31 when using half of
the training data and 75.0 mAP when using the complete nuScenes dataset. We
integrate our active learning framework into the proAnno labeling tool to
enable AI-assisted data selection and labeling and minimize the labeling
costs. Finally, we provide code, weights, and visualization results on our
website: https://active3d-framework.github.io/active3d-framework.

TABLE I: Comparison of previous research in Active Learning for 3D Object Detection in Autonomous Driving Datasets. Active Learning Methods |  Year |  Datasets |  Modalities |  Limitations
---|---|---|---|---
Entropy, Monte Carlo dropout, ensemble learning [1] |  2019 |  KITTI |  LiDAR, Camera† |  Image detector excluded from active learning loop
Uncertainty sampling [2] |  2019 |  Astyx [2] |  Radar, Camera, LiDAR |  Small dataset, little environmental variance
Semi-supervised co-training on prediction disagreement [3] |  2020 |  KITTI, Waymo |  Camera |  Single-mode approaches do not provide necessary accuracy for domain
Consensus score variation ratio, sequential region-of-interest matching [4] |  2020 |  KITTI |  LiDAR+Camera |  Hardware limitations prevent training backbone to convergence
Bayesian surprise (KL divergence) [5] |  2020 |  AGV Anomaly Dataset |  Camera |  Single-mode approaches do not provide necessary accuracy for domain
Augmentation, dropout, insertion, deletion [6] |  2021 |  KITTI |  LiDAR |  Sparse LiDAR points necessitate camera; pedestrians are challenging
Class Entropy and Spatial Uncertainty [7] |  2022 |  Private |  LiDAR |  Single-mode approaches do not provide necessary accuracy for domain
Uncertainty sampling [8] |  2022 |  Private |  LiDAR+Camera |  Detection of road damage; does not address object collision
Spatial and temporal diversity-based sampling [9] |  2022 |  nuScenes |  LiDAR |  Model uncertainty may improve AL for task performance; single sensing modality
Kernel coding rate [10] |  2023 |  KITTI, Waymo |  LiDAR |  Single-mode approaches do not provide necessary accuracy for domain
3D consistency of bounding box predictions in both semi-supervised and active learning [11] |  2023 |  KITTI |  LiDAR |  Plans to evaluate on nuScenes for robustness
Bi-domain active learning, diversity-based sampling [12] |  2023 |  KITTI, nuScenes, Waymo, Lyft |  LiDAR |  Single-mode approaches do not provide necessary accuracy for domain
Ego-pose distance-based sampling [13] |  2023 |  Navya3DSeg |  LiDAR |  Single-mode approaches do not provide necessary accuracy for domain
Sensor consistency-based selection score, LiDAR guidance as semi-supervision for monocular detection [14, 15] |  2024 |  KITTI, Waymo |  Camera |  Single-mode approaches do not provide necessary accuracy for domain
Classification Entropy Querying, T-CRB Criteria [16] (Ours) |  2024 |  nuScenes, TUMTraf-I* |  LiDAR+Camera, LiDAR |  Assumption of uniform label and point cloud distributions
∗TUMTraf-I Dataset.
†Camera is used for pre-trained region proposals and is not included in the
active learning loop.

##  I INTRODUCTION

The annotation process remains a challenge, especially for 3D point clouds.
According to [17], labeling a precise 3D box takes more than 100 seconds for
an annotator. This scales up to a labeling time of 7,937 hours for the
nuScenes dataset (see Fig. LABEL:fig:overview_figure). Since deep learning
models require a massive volume of labeled data, creating large-scale labeled
3D datasets is a challenge in developing robust 3D perception models. To
alleviate this challenge, active learning aims to reduce the labeling costs by
querying labels for a small fraction of unlabeled data, thus maximizing the
model performance while annotating the fewest possible samples. While active
learning significantly reduces the annotation costs by selecting the most
informative image and point cloud frames, it concurrently elevates the
computational expense due to the recurrent model training and evaluation
cycles. Hence, active learning is especially useful toward continual learning
methods that already facilitate these training recurrences, reducing the
computational burden by facilitating more efficient model updates, thereby
maintaining the gains of active learning without encountering prohibitive
computational expenses.

With this work, we fill the research gap in using active learning for multi-
modal 3D object detection. We adopt the work of [18] as our baseline active
learning framework and integrate it with the continuous training methods from
[19]. Existing work just focused on a single modality [20, 3, 5, 6] or applied
active learning for multi-task settings, like object detection and semantic
segmentation [14], but it is recognized that multi-modal approaches to
detection are significantly stronger toward the safety-critical 3D object
detection task.

Our contributions are the following:

  * •

We propose the ActiveAnno3D framework, which ensures that the selected data
samples for labeling are of maximum informativeness to our model training.

  * •

We explore various continuous training methods and integrate the most
efficient method regarding computational demand and detection performance.

  * •

We conduct extensive experiments of BEVFusion [21] and PV-RCNN [22] on the
nuScenes [23] and TUM Traffic Intersection (TUMTraf-I) dataset [24].

  * •

Finally, we integrate ActiveAnno3D into the Providentia Annotation (proAnno)
platform to enable AI-assisted features and minimize manual labeling efforts.

##  II Related Research

Active learning techniques can be broadly divided into two groups: those that
depend on a particular model for a particular task and those that operate more
generally on the data itself, independent of what problem the data will be
solving (sometimes called “diversity-based methods” [9]). Though these
techniques can be combined in creating a sampling strategy [20], the benefit
of using a task- or model-dependent active learning strategy is that the
selected data will be, in theory, optimal for improved performance of the
desired task. In our research, we specifically consider the task of 3D object
detection, which is critical for safety in autonomous driving. For this
reason, we consider only these task-dependent, model-specific, uncertainty-
driven approaches to active learning. We frame this as an episodic active
learning task [19], where some collection of data is gathered. Once a certain
accumulation of data has been made, the active learning algorithm re-samples
the available pool and uses an acquisition function to select samples to be
labeled. While general active learning is well-researched on demonstration ML
benchmark datasets, in the following sections, we describe the unique
challenges associated with active learning in the domain of real-world
autonomous driving.

Figure 1: The generalized active learning flow involves the selection of data
from an unlabeled pool according to an acquisition function, which, in the
case of uncertainty-driven AL, utilizes the trained model or, in the case of
diversity-driven AL, may be independent of the training. This selected data is
then annotated by an oracle and aggregated with previously labeled data.
Whether or not all data or just the new data is used in the next training step
is determined by the choice of training strategy. The variety of possible
acquisition and training techniques and unique domain challenges posed by
autonomous driving make active learning an opportune environment for
innovation toward safe and accurate learning.

Active Learning for Autonomous Driving must be:

####  II-1 Multitask

Identifying data optimal for one task is a good step. Still, at a practical
level, if the purpose of active learning is to reduce the annotation budget,
this does little good for an autonomous system that must use the same
carefully curated and sampled data to solve multiple tasks—trajectory
prediction [25, 26], sign and light detection and classification [27, 28, 29],
dynamic object detection and tracking [30, 31, 32, 33], lane detection [34],
vehicle landmark identification [35], and more—simultaneously. Fortunately,
multi-task consistency measurements can inform sampling in a way that
considers the performance of multiple goal tasks [36, 14].

####  II-2 Multi-modal

While autonomous vehicle testbeds may be configured with a variety of sensors,
the state of the art in 3D object detection, critical for collision-free
driving, utilizes both LiDAR (for accurate ranging) and image (for semantic
utility). At the time of this writing, the top 33 models on the nuScenes 3D
object detection leaderboard utilize both modalities (in fact, with two in the
middle of the pack, further including radar). Most active learning research
for 3D detection does not include the visual modality, and those that do leave
their image-related modules out of the learning loop. We induce active
learning on a state-of-the-art LiDAR+Camera architecture.

####  II-3 Open-set

We also note that data in the realm of autonomous driving is fundamentally
open-set, meaning that the items labeled in a dataset today may not fully
represent all items that might appear on the road when driving in the real
environment. Standard active learning methods cannot perform as effectively in
the open-set setting due to a class distribution mismatch between labeled and
unlabeled data [37], which also makes a strong case for the importance in the
classification stage of an active learning model, as uncertainty in
classification provides a means toward the identification of open-set class
instances. Safaei et al. [37] show that using a closed-set entropy on class
probabilities and distance entropy on learned features can be an effective
learning strategy to efficiently learn on an unknown open-set pool, though
this is yet to be applied to autonomous driving data, as their experimentation
is done over the CIFAR and TinyImageNet benchmarks. This further motivates our
exploration of entropy-based active learning within the autonomous driving
domain.

Related to the open-set environment, though driving environments may differ
regionally, there are fundamental similarities in the 3D detection problem
that may facilitate efficient cross-domain knowledge transfer for learned
models. Active learning strategies, and more specifically, active domain
adaptation, are useful in improving learning for a target domain by sampling
from a different source domain [12]. However, these techniques have currently
been evaluated only in the LiDAR learning setting, and there is room for
multi-modal adaptation to improve accuracy.

In Table I, we present an approximate chronological view of techniques that
have been researched and evaluated on autonomous driving-related datasets for
3D object detection. The first active learning application to 3D object
detection [1], evaluated on the KITTI dataset, explored uncertainty-based
techniques of Monte-Carlo dropout and deep ensembles; since then, the state of
the art in 3D object detection has advanced, and our research provides further
evidence that active learning methods which draw from model uncertainty are
still relevant, even with updated detection methods and on a further variety
of driving-related datasets. In developing active learning techniques that are
particular for 3D object detection, Luo et al. [18] define three mathematical
terms (“CRB”) to determine samples:

  1. 1.

Label Conciseness: aiming for the distribution of sampled labels to resemble
the uniform distribution,

  2. 2.

Bounding box Representativeness: aiming to select samples which the model
would learn similar patterns for as reflected in the model gradients, and

  3. 3.

Geometric Balance: aiming to learn from samples that fail to match expected
point densities relative to an object’s distance from the sensor.

In our empirical study, we examine the benefits of active learning on two
datasets using two methods: one which acquires LiDAR samples according to
these criteria of conciseness, representativeness, and geometric balance, and
one which acquires combined LiDAR+Camera samples using object class entropy.

Our research represents the first investigation of active learning for 3D
object detection (1) evaluating BEVFusion on the nuScenes dataset with an
uncertainty-driven query strategy, and (2) evaluating PV-RCNN using multiple
query strategies on the roadside infrastructure TUM Traffic Intersection
dataset [24].

##  III Methods

###  III-A Problem Statement

Consider the problem of 3D object detection from orderless LiDAR point clouds
represented as $\mathcal{P}=\\{(x,y,z,\varepsilon)\\}$, where $(x,y,z)$
denotes the 3D location and $\varepsilon$ represents the reflectance, and
synchronous images represented as $\mathcal{I}_{v}(r,c)$, where
$\mathcal{I}_{v}(r,c)=\\{R,G,B\\}$ is a 3-channel image of dimensions $r\times
c$ from view $v$, and the collection of images from all views $v$ is referred
to as $\mathcal{I}$. The objective is to localize objects of interest in the
form of a set of 3D bounding boxes.

Let $\Theta=\\{\mathbf{b}_{k}\\}_{k\in[N_{B}]}$ be the set of 3D bounding
boxes, where $N_{B}$ is the number of detected bounding boxes. Each bounding
box $\mathbf{b}_{k}$ is associated with a label $\mathbf{y}_{k}$ from a set
$\mathcal{Y}=\\{1,\ldots,C\\}$, where $C$ is the number of classes to predict.

The parameters of each bounding box $\mathbf{b}_{k}$ are given by
$(p_{x},p_{y},p_{z},l,w,h,\theta)$, representing the coordinates of the box’s
center ($p_{x},p_{y},p_{z}$), length ($l$), width ($w$), height ($h$), and
orientation angle ($\theta$).

At the training phase, a small set of labeled point clouds and associated
images
$\mathcal{D}_{L}=\\{(\mathcal{P},\mathcal{I},\mathbf{B},\mathbf{Y})_{i}\\}_{i\in[m]}$
is provided, where $m$ is the number of labeled point clouds. Additionally, an
unlabeled set of point clouds and associated images
$\mathcal{D}_{U}=\\{\mathcal{P}_{j},\mathcal{I}_{j}\\}_{j\in[n]}$ is
available.

In each active learning round $t$, guided by the active learning policy, a
subset of the unlabeled raw data is selected. Subsequently, the labels of the
3D bounding boxes for the selected subset are queried from an oracle to
construct $\mathcal{D}_{\text{query},t}$. The 3D object detection model is
initially trained using $\mathcal{D}_{L}$, then trained by $\mathcal{D}_{L}$
and $\mathcal{D}_{\text{query},t}$ for all following rounds $t$ until the
selected samples reach the final budget.

TABLE II: This table shows the mAP score for the random sampling baseline and
the entropy querying method using the LiDAR-only PV-RCNN model trained on the
TUM Traffic Intersection dataset, and the camera+LiDAR BEVFusion trained on
nuScenes. The results are compared to the respective 100% data usage results.

@cccccc@ gray!1011-111-6 Labeled Pool LiDAR-Only (PV-RCNN) LiDAR+Camera
(BEVFusion)
Round % Random Entropy Random Entropy
1 10 51.03 54.32 (+3.29) 30.95 31.06 (+0.11)
2 15 61.98 62.24 (+0.26) 34.19 36.39 (+2.20)
3 20 69.84 68.23 (-1.61) 38.00 40.41 (+2.41)
4 25 74.82 72.40 (-2.42) 42.36 42.17 (-0.19)
5 30 77.25 76.56 (-0.69) 44.94 45.57 (+0.63)
6 35 75.40 75.00 (-0.40) 44.74 46.76 (+2.02)
7 40 77.03 75.48 (-1.55) - 49.24
8 50 79.09 77.25 (-1.84) - 64.31
SOA (No AL) 100 83.50 75.00

###  III-B Active Learning for 3D Object Detection

3D object detection is crucial for autonomous driving and mobile robotics, yet
building precise 3D object detectors relies on the availability of extensive
labeled datasets, a task that is particularly challenging under constrained
annotation budgets. Active learning provides a promising solution to reduce
labeling costs by querying labels for only a small subset of the unlabeled
data pool. A criterion-based query selection process iteratively selects the
most informative samples for the subsequent model training until the labeling
budget is exhausted. Building on this, we deployed a pool-based active
learning framework to a two-stage LiDAR-only 3D object detector and a multi-
modal 3D object detector from the entropy querying strategies in [38]. We
examine the use of continuous training strategies to address the resource-
consuming nature of active learning.

###  III-C Entropy Querying

The query strategy, also known as the acquisition function, is the most
critical component of the active learning framework as it identifies the most
informative data samples according to a pre-defined criterion, and queries
labels from the human annotator. We employ entropy querying that depends on
the model’s uncertainty in predictions. The model’s uncertainty is measured
using the entropy formula:

$H(X)=-\sum_{i}(P(x_{i})\log P(x_{i}))$ (1)

This equation predicts the entropy $H(x)$, where $P(x)$ is the probability of
the model predicting class $x_{i}$.

Figure 2: The graph on left illustrates the mAP scores achieved by the PV-RCNN
model on the TUM Traffic Intersection dataset relative to the expanding size
of the training set in the active learning setting with random and entropy
queries separately. Similarly, the graph on right illustrates the mAP score
achieved by the BEVFusion model on the nuScenes dataset relative to the
expanding size of the training set.

###  III-D Active Learning for LiDAR-only 3D Object Detection

Initially, we employ the LiDAR-only model for 3D object detection known as the
two-stage PV-RCNN [22]. Subsequently, during the inference stage, we utilize
eight distinct query strategies, encompassing uncertainty-based, diversity-
based, and hybrid methods. Each strategy is employed separately for selecting
the most informative samples, guided by their individual selection criteria.
Moreover, we introduce a temporal modification to the AL-baseline CRB [39],
which we refer to as T-CRB. T-CRB [16] is designed to select the most
informative temporally-consistent set of point clouds from the unlabeled data
pool. This is particularly advantageous in scenarios involving point cloud
labeling, where annotators seek to label a sequence of frames sequentially,
aiming to maximize their contribution to the model’s performance.

###  III-E Active Learning for Multi-Modal 3D Object Detection

Second, we explore the BEVFusion model for 3D object detection, as outlined in
the work by Liu et al. [21]. They employ a multi-modal model combining camera
and LiDAR data. This modality is relevant to exploring and comparing the
results of our LiDAR-only approach. This approach is critical because, despite
the existence of various techniques for achieving a unified representation of
image and LiDAR data, LiDAR-to-Camera projections, as well as camera-only
methods, introduce significant geometric distortions, and camera-to-LiDAR
projections and LiDAR-only methods also face challenges in semantic-
orientation tasks. BEVFusion is designed to create a unified representation
that preserves geometric structure and semantic density.

The Swin-Transformer [40] serves as the image backbone, while VoxelNet [41]
acts as the LiDAR backbone. To generate bird’s-eye-view (BEV) features for
images, we employ a Feature Pyramid Network (FPN) [42] to fuse multi-scale
camera features, resulting in a feature map one-eighth of the original size.
Subsequently, images are down-sampled to 256x704, and LiDAR point clouds are
voxelized to 0.075 m to obtain the BEV features necessary for object
detection. These two modalities are fused using a convolution-based BEV
encoder with residual blocks, mitigating local misalignment between LiDAR-BEV
features and camera-BEV features. Especially from the uncertainty created in
the depth estimation from the camera encoder, as a result, is not compared
with ground truth i.e., this is a self-supervised learner.

We then follow the same Active Learning scheme previously used for the LiDAR-
only model.

###  III-F Continuous Training Strategies

The framework illustrated in Figure 1 follows an active learning approach,
wherein an initial 3D detector undergoes pre-training on an initially labeled
dataset. The query strategy identifies the most informative data samples
through successive training episodes and solicits labels from an oracle.
Subsequently, the initial model is updated using the augmented training set.
However, this process is characterized by its time and resource intensiveness,
as the model undergoes retraining from scratch in each episode on the
expanding set of labeled data. Additionally, this approach neglects the
knowledge accumulated in previous episodes. To address this challenge, we
enhance the active training cycle by incorporating multiple continuous
training strategies, as outlined in [43].

##  IV Experiments

We conduct a set of two experiments, integrating entropy-driven active
learning over the multi-modal BEVFusion model on two 3D object detection
datasets, nuScenes, and TUM Traffic Intersection, and the two-stage PV-RCNN
model on the TUM Traffic Intersection dataset.

###  IV-A Datasets

Large-scale datasets are essential for meaningful experiments. Therefore, we
conducted our experiments on nuScenes [23] and the TUM Trafic Intersection
dataset [24]. The popular nuScenes dataset provides 400k LiDAR point clouds
and 1.4M images from which the annotators carefully labeled 40k frames. The
dataset covers complex traffic scenarios, for example, at intersections or
construction sites, and groups traffic participants into 23 different object
classes. At the time of publication, nuScenes was the first dataset that
included night and rain scenarios. The data has been recorded by six cameras
and one LiDAR (Velodyne HDL32E) mounted on a vehicle. nuScenes contains 1000
scenes, 700 training scenes, 150 validation scenes, and 150 test scenes. We
start with an initial split of 100 training scenes and iteratively add 50
scenes for each subsequent active learning round until 35% of the data is used
resulting in six training rounds.

As mentioned above, nuScenes only covers the vehicle perspective. Therefore,
we also used the TUM Traffic Intersection dataset to experiment with further
perspectives, considering the roadside perspective from a gantry bridge at a
complex intersection. Two LiDARs (Ouster OS1-64, gen. 2) mounted at a height
of 7 m on the Providentia++ test field [44] near Munich generated the relevant
data for this work. The recorded traffic scenarios cover complex maneuvers
(left/right turns, overtaking, U-turns), with 25% at nighttime, including
heavy rain, and 75% during the day. In total, the dataset contains, 4,800
point clouds with more than 57.4k manually high-quality labeled 3D boxes
categorized in ten object classes, labeled using the Providentia Annotation
proAnno framework, an improved version of 3D BAT [45]. After point cloud
registration, the dataset contains 2,400 registered point clouds and 38,045
registered 3D boxes, which we used for this work. We split the data into
training (80%), validation (10% ), and test (10%) sets. Here, we performed our
active learning experiments on the train set with 1,920 samples. The initial
labeled set $D_{pre}$ contained approximately 5% (100 samples). We selected an
additional 100 samples every training episode until the labeling budged (50%
of the training data) was exhausted. Similar to our nuScenes experiments, we
trained an oracle model on the complete training data of the TUM Traffic
Intersection dataset to set a baseline for measuring the effectiveness of our
active learning approach.

###  IV-B Results

Figure 3 provides an insight into the performance linked with various query
strategies within active learning for 3D object detection. The evaluation
encompasses a variety of strategies such as BADGE [46], CoreSet [47], Monte-
Carlo sampling, confidence sampling, CRB [39], and the temporally-modified CRB
(T-CRB). Such analysis is useful towards pinpointing the most effective
approach tailored to the unique characteristics of the TUM Traffic
Intersection dataset. The results presented in Table III-A show that the PV-
RCNN detector using the random sampling query strategy trained on the TUM
Traffic Intersection dataset, outperformed the entropy query strategy as
illustrated in the left of Figure 2. However, this pattern does not persist in
the LiDAR+camera model. As observed in right of Figure 2, in every round
except round four, entropy querying outperforms random querying.

Figure 3: The graph illustrates the mAP scores achieved by the PV-RCNN model
on the TUM Traffic Intersection dataset relative to the expanding size of the
training set in the active learning setting with different query strategies.

For the qualitative analysis, we compare a pair of images from each dataset,
with the predicted annotations and the corresponding ground truth. For the TUM
Traffic Intersection dataset, illustrated in Figure 4, it can be seen that all
the eight classes are present in the image, and the corresponding bounding
boxes reflect the data well, as confirmed by the green bounding boxes seen in
the left image.

CRB

Oracle

Entropy

Oracle

Figure 4: Qualitative results are illustrated by two pairs of images. The left
pair is from the TUM Traffic Intersection dataset, and the right pair is from
nuScenes. For each pair, the left image shows the predicted labels for each
class, with each class represented by a different color. The right image of
each pair shows the predictions made by learning on the complete dataset. Both
results are quite similar, showing the efficiency of the active learning
technique.

For the nuScenes dataset illustrated in Figure 4 there are also many classes
present, though not all. On the right image, it is also possible to see that
all predictions are correct, except for one instance in the back where an
object is classified as a pedestrian, whereas in the image showing predictions
made by the model trained on the full dataset, there is no annotation to
represent this object.

###  IV-C Discussion

The variability in performance between datasets, where entropy querying
outperformed random querying in the scenario of BEVFusion with nuScenes, while
random querying approximately matches entropy querying in PV-RCNN with TUM
Traffic Intersection, highlights the importance of understanding the data
characteristics and intricacies. Random sampling, for instance, characterized
by its unselective and undiscriminating approach, often serves as a robust
baseline. Its lack of bias towards specific data characteristics ensures a
diverse and representative sample of the scenario. This characteristic proves
advantageous in scenarios with moderate-range objects, where the data achieves
an equilibrium between density and sparsity, as the case in the TUMTraf
Intersection Dataset. The ability of random sampling to provide a varied set
of frames contributes to its effectiveness in such contexts. On the other
hand, entropy sampling, designed to target the uncertainty of object
classification, may introduce bias towards certain object classes. This bias
has the potential to limit the selection of a diverse range of frames, which,
in turn, might negatively impact the overall learning performance for certain
datasets.

The unexpected superiority of random querying over entropy querying in this
case highlights the nuanced relationship between the acquisition function and
the dataset characteristics, emphasizing the need for careful consideration
when choosing active learning strategies tailored for specific scenarios.
Moreover, strategies such as CRB, random sampling, and confidence sampling
show high mAP scores, further highlighting the importance of strategy
selection based on specific data characteristics in active learning settings.
CRB achieves notable overall performance, attributed to its hierarchical and
hybrid nature, which combines uncertainty-based criteria, diversity-based
criteria, and geometric characteristics. This enables the query strategy to
balance between diverse detection scenarios defined by distance categories. In
contrast, confidence sampling, focusing on the uncertainty of objectness in
frames, attains the highest mAP performance due to its narrower focus. This
allows it to select frames that facilitate the model’s learning about various
objects. Unlike entropy sampling, which targets the uncertainty of object
classification and might introduce a bias towards certain object classes,
confidence sampling’s focused approach contributes to its effectiveness in
providing informative samples.

##  V CONCLUSIONS AND FUTURE WORK

To overcome the high labeling costs, we have proposed an active learning
framework termed ActiveAnno3D for multi-modal 3D object detection. We provide
a promising framework to reduce labeling costs by querying labels for only a
small subset of the unlabeled data pool. A criterion-based query selection
process iteratively selects the most informative samples for the subsequent
model training until the labeling budget is exhausted. We adopt [18] as our
baseline active learning framework, deploying a hierarchical, cost-effective
point cloud acquisition criteria at the 3D bounding box level, termed CRB. To
prevent retraining the model from scratch every time new data is selected, we
explore various continuous training methods and integrate the most efficient
method regarding computational demand and detection performance. Finally, we
show that our active learning framework achieves high performance (77.25 mAP
compared to 83.50 mAP) with reduced data utilization (50%) without placing
excessive demands on computational resources. The camera-LiDAR BEVFusion model
achieved an mAP of 64.31 when using 50% of the data for training that was
selected with the entropy-based sampling strategy, compared to 75.0 mAP when
using all the data.

In the future, we plan to improve the CRB query strategy. The current version
operates under the assumption that the unlabeled data pool possesses both a
uniform label distribution and a uniform point density distribution. By
incorporating a Bayesian approach, benefiting from the iterative process of
active training, and evolving the training set, it may become possible to
mirror the test set. Another future direction will be guided active selection.
The active query strategy can be tailored to suit the preferences of a human
annotator. If specific classes need to be labeled, the strategy can be
modified to evaluate the informativeness of these data samples. The framework
will then return the most informative samples about the desired object
classes. We also plan to make our ActiveAnno3D framework compatible with
multiple baselines (e.g., BEVFusion) and ease the integration of new
architectures. Finally, we plan to extend BEVFusion with different sampling
strategies and compare it with the results of the LiDAR-only PV-RCNN method.

Overall, the demonstrated sensitivity of 3D object detection performance to
active learning strategy is dependent on the nature of the available data,
both offering a clear value in adopting active learning in general for
efficient learning, and encouraging further research and analysis on optimal
strategies for individual and multiple tasks in safe autonomous driving.

## ACKNOWLEDGMENT

This research was supported by the Federal Ministry of Education and Research
in Germany within the AUTOtech.agil project, Grant Number: 01IS22088U. The
authors thank Qualcomm for the support of the Qualcomm Innovation Fellowship,
mentors Per Siden and Varun Ravi for their feedback, and Mathias V. Andersen
for assistance in algorithm implementations.

## References

  * [1] D. Feng, X. Wei, L. Rosenbaum, A. Maki, and K. Dietmayer, “Deep active learning for efficient training of a lidar 3d object detector,” in _2019 IEEE Intelligent Vehicles Symposium (IV)_. IEEE, 2019, pp. 667–674.
  * [2] M. Meyer and G. Kuschk, “Automotive radar dataset for deep learning based 3d object detection,” in _2019 16th european radar conference (EuRAD)_. IEEE, 2019, pp. 129–132.
  * [3] G. Villalonga and A. M. L. Pena, “Co-training for on-board deep object detection,” _IEEE Access_ , vol. 8, pp. 194 441–194 456, 2020.
  * [4] S. Schmidt, Q. Rao, J. Tatsch, and A. Knoll, “Advanced active learning strategies for object detection,” in _2020 IEEE Intelligent Vehicles Symposium (IV)_. IEEE, 2020, pp. 871–876.
  * [5] O. Çatal, S. Leroux, C. De Boom, T. Verbelen, and B. Dhoedt, “Anomaly detection for autonomous guided vehicles using bayesian surprise,” in _2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_. IEEE, 2020, pp. 8148–8153.
  * [6] Q. Meng, W. Wang, T. Zhou, J. Shen, Y. Jia, and L. Van Gool, “Towards a weakly supervised framework for 3d point cloud object detection and annotation,” _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , vol. 44, no. 8, pp. 4454–4468, 2021.
  * [7] A. Moses, S. Jakkampudi, C. Danner, and D. Biega, “Localization-based active learning (local) for object detection in 3d point clouds,” in _Geospatial Informatics XII_ , vol. 12099. SPIE, 2022, pp. 44–58.
  * [8] L. Chen, X. He, X. Zhao, H. Li, Y. Huang, B. Zhou, W. Chen, Y. Li, C. Wen, and C. Wang, “Gocomfort: Comfortable navigation for autonomous vehicles leveraging high-precision road damage crowdsensing,” _IEEE Transactions on Mobile Computing_ , 2022.
  * [9] Z. Liang, X. Xu, S. Deng, L. Cai, T. Jiang, and K. Jia, “Exploring diversity-based active learning for 3d object detection in autonomous driving,” _arXiv preprint arXiv:2205.07708_ , 2022.
  * [10] Y. Luo, Z. Chen, Z. Fang, Z. Zhang, M. Baktashmotlagh, and Z. Huang, “Kecor: Kernel coding rate maximization for active 3d object detection,” in _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 2023, pp. 18 279–18 290.
  * [11] S. Hwang, S. Kim, Y. Kim, and D. Kum, “Joint semi-supervised and active learning via 3d consistency for 3d object detection,” in _2023 IEEE International Conference on Robotics and Automation (ICRA)_. IEEE, 2023, pp. 4819–4825.
  * [12] J. Yuan, B. Zhang, X. Yan, T. Chen, B. Shi, Y. Li, and Y. Qiao, “Bi3d: Bi-domain active learning for cross-domain 3d object detection,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2023, pp. 15 599–15 608.
  * [13] A. Almin, L. Lemarié, A. Duong, and B. R. Kiran, “Navya3dseg-navya 3d semantic segmentation dataset design & split generation for autonomous vehicles,” _IEEE Robotics and Automation Letters_ , 2023.
  * [14] A. Hekimoglu, P. Friedrich, W. Zimmer, M. Schmidt, A. Marcos-Ramiro, and A. Knoll, “Multi-task consistency for active learning,” in _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 2023, pp. 3415–3424.
  * [15] A. Hekimoglu, M. Schmidt, and A. Marcos-Ramiro, “Monocular 3d object detection with lidar guided semi supervised active learning,” in _Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision_ , 2024, pp. 2346–2355.
  * [16] A. A. Ghita, “Active learning for 3d object detection and labeling,” Master’s Thesis, Technical University of Munich, TUM School of Computation, Information and Technology, Munich, December 2023. [Online]. Available: https://www.ce.cit.tum.de/fileadmin/w00cgn/air/Personal_Files/WalterZimmer/final_masters_thesis_ahmed_ghita_compressed_default.pdf
  * [17] S. Song, S. P. Lichtenberg, and J. Xiao, “Sun rgb-d: A rgb-d scene understanding benchmark suite,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2015, pp. 567–576.
  * [18] Y. Luo, Z. Chen, Z. Wang, X. Yu, Z. Huang, and M. Baktashmotlagh, “Exploring active 3d object detection from a generalization perspective,” in _The Eleventh International Conference on Learning Representations_ , 2022.
  * [19] F. Dayoub, N. Sunderhauf, and P. I. Corke, “Episode-based active learning with bayesian neural networks,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops_ , 2017, pp. 26–28.
  * [20] A. Hekimoglu, M. Schmidt, A. Marcos-Ramiro, and G. Rigoll, “Efficient active learning strategies for monocular 3d object detection,” in _2022 IEEE Intelligent Vehicles Symposium (IV)_. IEEE, 2022, pp. 295–302.
  * [21] Z. Liu, H. Tang, A. Amini, X. Yang, H. Mao, D. L. Rus, and S. Han, “Bevfusion: Multi-task multi-sensor fusion with unified bird’s-eye view representation,” in _2023 IEEE international conference on robotics and automation (ICRA)_. IEEE, 2023, pp. 2774–2781.
  * [22] S. Shi, C. Guo, L. Jiang, Z. Wang, J. Shi, X. Wang, and H. Li, “Pv-rcnn: Point-voxel feature set abstraction for 3d object detection,” in _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , 2020, pp. 10 529–10 538.
  * [23] H. Caesar, V. Bankiti, A. H. Lang, S. Vora, V. E. Liong, Q. Xu, A. Krishnan, Y. Pan, G. Baldan, and O. Beijbom, “nuscenes: A multimodal dataset for autonomous driving,” in _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , 2020, pp. 11 621–11 631.
  * [24] W. Zimmer, C. Creß, H. T. Nguyen, and A. C. Knoll, “Tumtraf intersection dataset: All you need for urban 3d camera-lidar roadside perception,” in _26th IEEE International Conference on Intelligent Transportation Systems (ITSC 2023)_. IEEE, 2023.
  * [25] R. Greer, N. Deo, and M. Trivedi, “Trajectory prediction in autonomous driving with a lane heading auxiliary loss,” _IEEE Robotics and Automation Letters_ , vol. 6, no. 3, pp. 4907–4914, 2021.
  * [26] S. Lefevre, A. Carvalho, and F. Borrelli, “A learning-based framework for velocity control in autonomous driving,” _IEEE Transactions on Automation Science and Engineering_ , vol. 13, no. 1, pp. 32–42, 2015.
  * [27] R. Greer, J. Isa, N. Deo, A. Rangesh, and M. M. Trivedi, “On salience-sensitive sign classification in autonomous vehicle path planning: Experimental explorations with a novel dataset,” in _2022 Winter Conference on Applications of Computer Vision (WACV)_.
  * [28] R. Greer, A. Gopalkrishnan, N. Deo, A. Rangesh, and M. Trivedi, “Salient sign detection in safe autonomous driving: Ai which reasons over full visual context,” _27th International Technical Symposium on the Enhanced Safety of Vehicles (ESV)_ , 2023.
  * [29] R. Greer, A. Gopalkrishnan, J. Landgren, L. Rakla, A. Gopalan, and M. Trivedi, “Robust traffic light detection using salience-sensitive loss: Computational framework and evaluations,” in _2023 IEEE Intelligent Vehicles Symposium (IV)_ , 2023, pp. 1–7.
  * [30] W. Zimmer, E. Ercelik, X. Zhou, X. J. D. Ortiz, and A. Knoll, “A survey of robust 3d object detection methods in point clouds,” _arXiv preprint arXiv:2204.00106_ , 2022.
  * [31] W. Zimmer, J. Wu, X. Zhou, and A. C. Knoll, “Real-time and robust 3d object detection with roadside lidars,” in _Proceedings of the 12th International Scientific Conference on Mobility and Transport: Mobility Innovations for Growing Megacities_. Springer, 2023, pp. 199–219.
  * [32] H.-k. Chiu, J. Li, R. Ambruş, and J. Bohg, “Probabilistic 3d multi-modal, multi-object tracking for autonomous driving,” in _2021 IEEE international conference on robotics and automation (ICRA)_. IEEE, 2021, pp. 14 227–14 233.
  * [33] W. Zimmer, J. Birkner, M. Brucker, H. T. Nguyen, S. Petrovski, B. Wang, and A. C. Knoll, “Infradet3d: Multi-modal 3d object detection based on roadside infrastructure camera and lidar sensors,” in _2023 IEEE Intelligent Vehicles Symposium (IV)_. IEEE, 2023.
  * [34] J. Huang, P. K. Choudhury, S. Yin, and L. Zhu, “Real-time road curb and lane detection for autonomous driving using lidar point clouds,” _IEEE Access_ , vol. 9, pp. 144 940–144 951, 2021.
  * [35] R. Greer, A. Gopalkrishnan, M. Keskar, and M. M. Trivedi, “Patterns of vehicle lights: Addressing complexities of camera-based vehicle light datasets and metrics,” _Pattern Recognition Letters_ , 2024. [Online]. Available: https://www.sciencedirect.com/science/article/pii/S0167865524000047
  * [36] Y. Xiao, Z. Chang, and B. Liu, “An efficient active learning method for multi-task learning,” _Knowledge-Based Systems_ , vol. 190, p. 105137, 2020.
  * [37] B. Safaei, V. VS, C. M. de Melo, and V. M. Patel, “Entropic open-set active learning,” _arXiv preprint arXiv:2312.14126_ , 2023.
  * [38] R. Greer, B. Antoniussen, M. V. Andersen, A. Møgelmose, and M. M. Trivedi, “The why, when, and how to use active learning in large-data-driven 3d object detection for safe autonomous driving: An empirical exploration,” _arXiv preprint arXiv:2401.16634_ , 2024.
  * [39] Y. Luo, Z. Chen, Z. Wang, X. Yu, Z. Huang, and M. Baktashmotlagh, “Exploring active 3d object detection from a generalization perspective,” in _The Eleventh International Conference on Learning Representations_ , 2023. [Online]. Available: https://openreview.net/forum?id=2RwXVje1rAh
  * [40] Z. Liu, Y. Lin, Y. Cao, H. Hu, Y. Wei, Z. Zhang, S. Lin, and B. Guo, “Swin transformer: Hierarchical vision transformer using shifted windows,” in _Proceedings of the IEEE/CVF international conference on computer vision_ , 2021, pp. 10 012–10 022.
  * [41] Y. Zhou and O. Tuzel, “Voxelnet: End-to-end learning for point cloud based 3d object detection,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2018, pp. 4490–4499.
  * [42] T.-Y. Lin, P. Dollár, R. Girshick, K. He, B. Hariharan, and S. Belongie, “Feature pyramid networks for object detection,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2017, pp. 2117–2125.
  * [43] F. Dayoub, N. Sünderhauf, and P. Corke, “Episode-based active learning with bayesian neural networks,” 2017.
  * [44] A. Krämmer, C. Schöller, D. Gulati, V. Lakshminarasimhan, F. Kurz, D. Rosenbaum, C. Lenz, and A. Knoll, “Providentia - a large-scale sensor system for the assistance of autonomous vehicles and its evaluation,” in _Journal of Field Robotics_. Wiley, 2022, pp. 1156–1176.
  * [45] W. Zimmer, A. Rangesh, and M. Trivedi, “3d bat: A semi-automatic, web-based 3d annotation toolbox for full-surround, multi-modal data streams,” in _2019 IEEE Intelligent Vehicles Symposium (IV)_. IEEE, 2019, pp. 1816–1821.
  * [46] J. T. Ash, C. Zhang, A. Krishnamurthy, J. Langford, and A. Agarwal, “Deep batch active learning by diverse, uncertain gradient lower bounds,” 2020.
  * [47] O. Sener and S. Savarese, “Active learning for convolutional neural networks: A core-set approach,” 2018.