# End-to-End Zero-Shot HOI Detection via
Vision and Language Knowledge Distillation

Mingrui Wu1,2${{\dagger}}$, Jiaxin Gu3${{\dagger}}$, Yunhang Shen2, Mingbao
Lin2,
Chao Chen2, Xiaoshuai Sun1,4,5${{\ddagger}}$
Intern at Youtu Lab, Tencent. ${{\dagger}}$ Equal Contribution. Works
completed at Tencent. ${{\ddagger}}$ Corresponding Author.

###### Abstract

Most existing Human-Object Interaction (HOI) Detection methods rely heavily on
full annotations with predefined HOI categories, which is limited in diversity
and costly to scale further. We aim at advancing zero-shot HOI detection to
detect both seen and unseen HOIs simultaneously. The fundamental challenges
are to discover potential human-object pairs and identify novel HOI
categories. To overcome the above challenges, we propose a novel End-to-end
zero-shot HOI Detection (EoID) framework via vision-language knowledge
distillation. We first design an _Interactive Score_ module combined with a
_Two-stage Bipartite Matching_ algorithm to achieve interaction
distinguishment for human-object pairs in an action-agnostic manner. Then we
transfer the distribution of action probability from the pretrained vision-
language teacher as well as the seen ground truth to the HOI model to attain
zero-shot HOI classification. Extensive experiments on HICO-Det dataset
demonstrate that our model discovers potential interactive pairs and enables
the recognition of unseen HOIs. Finally, our EoID outperforms the previous
SOTAs under various zero-shot settings. Moreover, our method is generalizable
to large-scale object detection data to further scale up the action sets. The
source code is available at: https://github.com/mrwu-mac/EoID.

##  1 Introduction

Figure 1: CLIP shows promising zero-shot ability in detecting unseen actions
(red) and unseen objects (_e.g._ tiger) with seen actions (blue). The unseen
and seen here denote the divisions under the zero-shot setting.

The task of Human-Object Interaction (HOI) detection aims to detect $\langle
human,$ $verb,object\rangle$ triplets, which simultaneously localizes human-
object pairs and identifies the corresponding interactive actions. HOI
detection plays an important role in many downstream visual understanding
tasks, especially for human-centric scenes, such as Image Captioning (Li et
al. 2017) and Visual Question Answering (Goyal et al. 2017).

Most of the existing works simply focus on improving action classification
performances for predefined HOI categories. However, they suffer from two
major weaknesses: 1) excessive cost for new HOI dataset construction and 2) a
lack of generalization for unseen actions. Previous works (Shen et al. 2018;
Bansal et al. 2020; Liu, Yuan, and Chen 2020; Hou et al. 2021b) attempted to
overcome the above drawbacks via zero-shot learning. Most of them are devoted
to improving human-object visual representation (Hou et al. 2021b) and
introducing language model (Bansal et al. 2020; Liu, Yuan, and Chen 2020),
ignoring the implicit relations between vision and language.

As a recent technological breakthrough, CLIP (Radford et al. 2021) performs
contrastive learning on $400$ million image-text pairs collected from the Web
and shows impressive zero-shot transferability on over $30$ classification
datasets. Some recent works also successfully transfer the pretrained CLIP
model to various downstream tasks such as object detection (Gu et al. 2021),
text-driven image manipulation (Patashnik et al. 2021) and semantic
segmentation (Zhou, Loy, and Dai 2021). Compared to text embeddings simply
extracted from pure language models, the text embeddings learned jointly with
visual images can better encode the visual similarity between concepts (Gu et
al. 2021). Inspired by this, we attempt to transfer vision and language
(V$\&$L) knowledge of CLIP into the zero-shot HOI task. As shown in Fig. 1,
CLIP has the ability to identify certain unseen actions (Fig. 1(a)), and also
discover unseen objects with seen action interaction (Fig. 1(b)). Recent GEN-
VLKT (Liao et al. 2022) applies CLIP to HOI detection task. However, GEN-VLKT
is limited in known human-object pairs and fails to deal with the potential
interactive pairs. In addition, its global image-level distillation inevitably
introduces noise when there exist multiple interactions in one image. It
requires local region-level distillation for accuracy and robustness.

To this end, we propose EoID, an end-to-end zero-shot HOI detection framework
to detect unseen HOI pairs by distilling the knowledge from CLIP. We first
design a novel _Interactive Score_ (IS) module with a _Two-stage Bipartite
Matching_ algorithm to discover potential action-agnostic interactive human-
object pairs. Then we distill interactive knowledge from CLIP to teach the HOI
model to identify unseen actions. Specifically, the probability distribution
of the actions is obtained by CLIP given the cropped union regions of each
human-object pairs, with predefined HOI prompts. Finally, the HOI model learns
from the distilled probability distribution as well as the ground truth
actions. We evaluate the proposed method on HICO-Det (Chao et al. 2018)
benchmark under unseen action (UA) and unseen action-object combination (UC)
settings. Extensive experiments validate that our framework can detect
potential interactive human-object pairs. And the results show that our
approach outperforms previous SOTAs under various zero-shot settings. In
addition, our method can generalize to object detection datasets and obtain
$47.15\%$ $m$AP on unseen actions of V-COCO (Gupta and Malik 2015) only with
the bounding boxes from MS-COCO (Lin et al. 2014).

To summarize, our contributions are:

  * •

We propose an end-to-end zero-shot HOI detection framework which attains zero-
shot HOI classification via V&L knowledge distillation.

  * •

We succeed in detecting potential action-agnostic interactive human-object
pairs by applying an _Interactive Score_ module combined with a _Two-stage
Bipartite Matching_ algorithm, the effectiveness of which has been validated
through extensive experiments.

  * •

Experiments show that EoID is capable of detecting HOIs with unseen HOI
categories and outperforms previous SOTA under zero-shot settings. Moreover,
our method is able to generalize to object detection datasets only with
bounding boxes, which further scales up the action sets.

##  2 Related Works

###  2.1 Human-Object Interaction Detection

Most previous works on HOI detection can be categorized into two groups: two-
stage (Chao et al. 2018; Gao, Zou, and Huang 2018; Li et al. 2019; Kim et al.
2020b) and one-stage (Gkioxari et al. 2018; Liao et al. 2020; Wang et al.
2020; Kim et al. 2020a, 2021; Zou et al. 2021; Tamura, Ohashi, and Yoshinaga
2021; Zhang et al. 2021) methods. Following this paradigm, we build our
framework on the transformer-based approach to achieve zero-shot HOI
detection. We also replace the one-stage Hungarian matching algorithm (Kuhn
1955; Carion et al. 2020; Tamura, Ohashi, and Yoshinaga 2021; Zhang et al.
2021) with a novel two-stage matching algorithm for detecting potential
interactive human-object pairs.

###  2.2 Knowledge Distillation from CLIP

CLIP (Radford et al. 2021) adopts contrastive learning to jointly train image-
text embedding models on large-scale image-text pairs collected from the
internet and has shown promising zero-shot transferability. It inspires
subsequent studies to transfer the vision and language knowledge to various
downstream tasks such as object detection (Gu et al. 2021), text-driven image
manipulation (Patashnik et al. 2021), video clip retrieval (Luo et al. 2021)
and semantic segmentation (Zhou, Loy, and Dai 2021; Rao et al. 2021). Recent
GEN-VLKT (Liao et al. 2022) for the first time applies CLIP to HOI detection
task, which transfers the knowledge of CLIP by image-level feature
distillation. However, it fails to deal with multiple human-object
interactions in one image. Different from previous attempts, we propose to use
region-level distillation by distilling soft action probability from CLIP.

###  2.3 Zero-Shot Learning on HOI detection

Zero-shot learning aims at classifying categories that are not seen during
training. Previous works implement zero-shot learning on HOI detection task
from three scenarios: unseen combination scenario (UC) (Shen et al. 2018;
Bansal et al. 2020; Hou et al. 2021b; Liu, Yuan, and Chen 2020), unseen object
scenario (UO) (Bansal et al. 2020; Hou et al. 2021b; Liu, Yuan, and Chen 2020)
and unseen action scenario (UA) (Liu, Yuan, and Chen 2020). ConsNet (Liu,
Yuan, and Chen 2020) performs zero-shot HOI detection for the three scenarios
by learning from a consistency graph along with word embeddings. (Wang et al.
2022) develop a transferable HOI detector via joint visual-and-text modeling.
RLIP (Yuan et al. 2022) propose relational Language- Image pre-training to
improve zero-shot, few-shot and fine-tuning HOI detection performance.
However, they show limited zero-shot capability due to limited HOI datasets.
GEN-VLKT (Liao et al. 2022) distills CLIP knowledge into known human-object
pairs to attain zero-shot learning. In contrast, our proposed method is able
to attain zero-shot learning on known human-object pairs as well as potential
interactive pairs.

Figure 2: Overview of our EoID. After getting $N$ predictions from the HOI
model, two-stage bipartite matching is used to select the best matched
predictions with ground truth human-object pairs. Then we train the model with
the selected predictions to learn the distribution of action probability from
the pretrained V&L teacher as well as the seen ground truth to achieve zero-
shot HOI classification.

##  3 Preliminary

Problem Formulation: Denote
$\mathcal{A}_{S}=\left\\{a_{1},\cdots,a_{k}\right\\}$ as a set of the seen
action categories, and $\mathcal{A}_{U}=\left\\{a_{k+1},\cdots,a_{n}\right\\}$
as a set of unseen actions. Let $I$ denote an input image, with corresponding
labels $\mathcal{T}=\\{\mathcal{B},\mathcal{Y}\\}$ where $\mathcal{B}$ is a
set of bounding boxes including human boxes $b_{h}$ and object boxes $b_{o}$,
and $\mathcal{Y}$ denote a set of known HOI triplets. Each $y=\langle
b_{h},b_{o},a\rangle$ in $\mathcal{Y}$ is an HOI triplet, where $b_{h}$ and
$b_{o}$ are the elements in the set $\mathcal{B}$, and $a\in\mathcal{A}_{S}$.

For our HOI model, the bounding boxes are trained in a paired manner. It
requires to construct all possible human-object pairs one by one between human
and objects in $\mathcal{B}$. The constructed pairs present in the annotated
HOI triplets set of $\mathcal{Y}$ denote seen pairs (or known pairs),
$y_{s}=\langle b_{h},b_{o},a\rangle$ , and the others absent in $\mathcal{Y}$
denote unknown pairs, $y_{u}=\langle b_{h},b_{o},\varnothing\rangle$. The
unknown pairs consist of the unseen pairs with potential interaction and the
non-interactive pairs. Finally, our goal is to detect all interactive seen and
unseen pairs, and also recognize their actions.

Transformer-based HOI models: Most of existing SOTA HOI models (Zou et al.
2021; Tamura, Ohashi, and Yoshinaga 2021; Zhang et al. 2021) are end-to-end
transformer-based models. First, the input image $I$ and learnable query
vectors $Q_{e}$ are fed to an HOI model to predict human-object bounding boxes
pairs and the corresponding actions. The paradigm is formulated as,
$\hat{y}=Transformer(I,Q_{e})$, where $\hat{y}$ is the prediction. During
training, a bipartite matching algorithm is adopted to match predictions with
the best ground truth by the Hungarian algorithm, as follows,

$\centering\hat{\sigma}=\underset{\sigma\in\Theta_{N}}{\arg\min}\sum_{i=1}^{N}\mathcal{H}_{\text{match
}}\left(y_{i},\hat{y}_{\sigma(i)}\right),\@add@centering$ (1)

where $y_{i}\in\mathcal{\widetilde{Y}}$,
$\mathcal{\widetilde{Y}}=\\{y_{1},\cdots,y_{M},\varnothing_{M+1},\cdots,\varnothing_{N}\\}$
denotes the $M$ ground truth pairs padded with $N-M$ no-pairs $\varnothing$,
$\\{\hat{y}_{i}\\}^{N}_{i=1}$ denotes the set of $N$ predictions, and
$\Theta_{N}$ is a search space for a permutation of $N$ elements.
$\mathcal{H}_{match}$ is the matching cost (Carion et al. 2020) between ground
truth $y_{i}$ and a prediction with index $\sigma(i)$, which consists of four
types of costs: the box-regression cost $\mathcal{H}_{b}$, intersection-over-
union (IoU) cost $\mathcal{H}_{u}$, object-class cost $\mathcal{H}_{c}$, and
action-class cost $\mathcal{H}_{a}$, as follows,

$\centering\mathcal{H}_{\text{match
}}=\mathcal{H}_{b}+\mathcal{H}_{u}+\mathcal{H}_{c}+\mathcal{H}_{a}.\@add@centering$
(2)

Finally, the losses of the matched pairs are optimized by a Hungarian loss,
which can be formulated as:

$\begin{split}\@add@centering\centering\mathcal{L}_{H}=\sum_{i=1}^{N}\sum_{\mathcal{L}\in\Omega}\\{\mathds{1}_{\\{y_{i}\neq\varnothing\\}}\mathcal{L}(\hat{y}_{\hat{\sigma}(i)},y_{i})+\mathds{1}_{\\{y_{i}=\varnothing\\}}\mathcal{L}(\hat{y}_{\hat{\sigma}(i)},\varnothing)\\},\end{split}$
(3)

where
$\Omega=\\{\mathcal{L}_{b},\mathcal{L}_{u},\mathcal{L}_{c},\mathcal{L}_{a}\\}$,
$\mathcal{L}_{b}=\mathcal{L}^{(h)}_{b}+\mathcal{L}^{(o)}_{b}$ is the box
regression loss, $\mathcal{L}_{u}=\mathcal{L}^{(h)}_{u}+\mathcal{L}^{(o)}_{u}$
is the intersection-over-union loss, $\mathcal{L}_{c}$ is the object-class
loss and $\mathcal{L}_{a}$ the action-class loss, following QPIC (Tamura,
Ohashi, and Yoshinaga 2021) and CDN (Zhang et al. 2021).

##  4 Method

Figure 3: The two-stage bipartite matching algorithm. Given $N$ predictions,
we first match all the predictions with $M$ seen pairs ($y_{s}$) as
interactive pairs in the first stage. In the second stage, we use the
remaining $N-M$ predictions to match the $K$ unknown pairs ($y_{u}$). And we
select $topk$ predictions that have interactive score greater than interactive
threshold $thres_{is}$ from the matched $K$ predictions as potential
interactive pairs, and the others as non-interactive pairs. The non-
interactive pairs and the remaining $N-M-K$ unmatched predictions will be
regarded as no-pairs.

###  4.1 Overall Architecture

Fig. 2 illustrates an overview of our EoID. We benchmark on transformer-based
model CDN (Zhang et al. 2021). Given an image $I$, the CDN first encodes $I$
into visual feature sequence $S$, then the cascaded human-object decoder and
interaction decoder are assigned to decode the visual feature sequence into
$N$ predictions, including human-object bounding boxes pairs ($b_{h}$,
$b_{o}$) and action vectors $\boldsymbol{a}$ with corresponding interactive
scores $s_{is}$ from a set of queries $Q_{e}$. For the vision and language
teacher, the union regions of human-object pairs are cropped to extract the
visual features, and prompt engineering is adopted before calculating the
semantic features of the HOI texts. The cosine similarity between them
combined with the prior knowledge represents the action probability for these
predictions. Then a two-stage bipartite matching algorithm is applied to
select the predictions that best match the ground truth human-object pairs for
the seen pairs and the unknown pairs. These selected predictions from the two-
stage bipartite matching algorithm are used to train bounding boxes regressive
branches, object classification branch, interactive score branch, and action
classification branch. Finally, we train the model to learn the distribution
of action probability from the pretrained vision and language teacher as well
as the seen ground truth to achieve zero-shot HOI classification.

###  4.2 Learning to Detect Potential Interactive Pairs

Exhaustively traversing all possible human-object pairs is computationally
infeasible and might introduce excessive training noise. As a result, the
first challenge in our work is to detect potential action-agnostic interactive
pairs in both training and inference. We address this issue by introducing an
interactive score module and a two-stage bipartite matching algorithm.

Interactive Score Module: To distinguish interactive and non-interactive pairs
in the predictions, it is a natural idea to adopt the naive implementation of
the interactive score module in CDN. However, it only considers the
predictions matching seen pairs $y_{s}$ as interactive pairs and the others as
non-interactive pairs, which suppresses the detection of unseen pairs.

Unlike CDN, we apply the interactive score head from the interaction decoder
instead of the human-object decoder to detect unseen pairs. Specifically,
given $N$ predictions, we first apply a two-stage bipartite matching algorithm
to select $M$+$topk$ predictions which best match the $M$ seen pairs and
$topk$ unknown pairs. Then we train the interactive score module by rewarding
the $M+topk$ matched predictions which have human and object IoUs with the
matched ground truth pairs are greater than $0.5$, and penalizing the others.
For the rest $N-M-topk$ predictions with any unknown pairs that satisfy the
above condition, their loss is omitted during the optimization.

Two-stage Bipartite Matching Algorithm: Similar to OW-DETR (Gupta et al.
2021), we apply a two-stage bipartite matching algorithm to match the seen
pairs and the unknown pairs respectively, as shown in Fig. 3. We first match
$M$ predictions with the seen pairs as interactive pairs based on the box-
regression cost $H_{b}$, IoU cost $H_{u}$, object-class cost $H_{c}$, and
action-class cost $H_{a}$, as follows,

$\centering\hat{\sigma}_{1}=\underset{\sigma_{1}\in\Theta_{N}}{\arg\min}\sum_{i=1}^{N}\mathcal{H}^{1}_{\text{match}}\left(y_{i},\hat{y}_{\sigma_{1}(i)}\right),\@add@centering$
(4)

where
$y_{i}\in\mathcal{\widetilde{Y}}_{1}=\\{y_{1},\cdots,y_{M},\varnothing_{M+1},\cdots,\varnothing_{N}\\}$,
and
$\mathcal{H}^{1}_{\text{match}}=\mathcal{H}_{b}+\mathcal{H}_{u}+\mathcal{H}_{c}+\mathcal{H}_{a}$.
Then the $N-M$ predictions not selected by the first stage matching will be
used for the second stage matching. Since ground truth actions are not
available for the unknown pairs, we implement the second stage matching only
based on the box-regression cost $H_{b}$, IoU cost $H_{u}$, object-class cost
$H_{c}$. The second stage matching process can be formulated as follows,

$\displaystyle\hat{\sigma}_{2}=\underset{\sigma_{2}\in\Theta_{N-M}}{\arg\min}\sum_{i=M+1}^{N}\mathcal{H}^{2}_{\text{match}}\left(y_{i},\hat{y}_{\sigma_{2}(i)}\right),$
(5)

where
$\mathcal{H}^{2}_{\text{match}}=\mathcal{H}_{b}+\mathcal{H}_{u}+\mathcal{H}_{c}$,
$y_{i}\in\mathcal{\widetilde{Y}}_{2}=\\{y_{M+1},\cdots,y_{M+K},\varnothing_{M+K+1},\cdots,\varnothing_{N}\\}$,
and $K$ is the number of unknown pairs. We select the $topk$ predictions which
have interactive scores $s_{is}$ greater than interactive threshold
$thres_{is}$ as potential interactive pairs, the others as non-interactive
pair. So we can get the $M+topk$ predictions with
$\hat{\sigma}=\hat{\sigma}_{1}\cup topk(\hat{\sigma}_{2})$. The non-
interactive pairs and the remaining $N-M-K$ unmatched predictions will be
regarded as no-pairs. The no-pairs and $M+topk$ predictions combined with
matched ground truth will be used to train bounding boxes regressive branch,
object classification branch, interactive score branch and action
classification branch.

Such a strategy will help the model learn from the seen pairs to discriminate
whether there exists interaction between human-object pairs at the early
training stage, and gradually introduce potential interactive pairs for
learning. Different from GEN-VLKT (Liao et al. 2022) that simply detects known
pairs, our method also additionally detects potential interactive pairs which
not exist in training set and can also be applied to scale up the action sets
on object detection dataset.

###  4.3 Knowledge Distillation from CLIP

After detecting potential interactive human-object pairs, we need to identify
the corresponding action happening between the human and object. For this
purpose, we transfer the interaction knowledge from the pretrained V&L model
CLIP (teacher) into the HOI model (student) via knowledge distillation similar
to ViLD (Gu et al. 2021) and GEN-VLKT (Liao et al. 2022). In contrast to
global image-level distillation in GEN-VLKT, we adopt local region-level
distillation to deal with multiple human-object pairs in one image. In order
to avoid the misalignment between the local feature of the teacher and global
feature of the student, we apply logits distillation instead of feature
distillation adopted by ViLD.

We first convert the HOI category texts, _e.g._ riding bicycle, into the
prompts by feeding them into prompt template a picture of person
$\left\\{verb\right\\}$ $\left\\{object\right\\}$. Then we encode these
prompts to generate the text embeddings $t_{e}$ offline by the CLIP text
encoder $T$. For the $M+topk$ matched pairs of $I$, we crop the human-object
union regions, and feed the preprocessed ones into the CLIP image encoder $V$
to generate the image embeddings $v_{e}$. Then, we compute cosine similarities
between the image and text embeddings, as
$s_{i}=v_{e}^{T}t_{e}^{i}\mathbin{/}(\|v_{e}\|\cdot\|t_{e}^{i}\|)$. According
to the prior knowledge (Chao et al. 2018), we select the valid actions which
is able to interact with the object for each union region. This makes the
model pay more attention to the learning of the current human-object pair, and
avoid the interference of other human-object interactions in the union-box. We
apply a softmax activation on similarities of these HOI categories to get the
probability distribution $\mathbf{p}$ of the actions in
$\mathcal{A}_{S}+\mathcal{A}_{U}$ for each of union regions. The process can
be formulated as follows,

$\displaystyle p_{i}$
$\displaystyle=\frac{e^{\gamma\,s_{i}m_{i}}}{\sum_{j=1}^{n}e^{\gamma\,s_{j}m_{j}}},$
(6) $\displaystyle\mathrm{s.t.}~{}m_{i}$ $\displaystyle=\begin{cases}1,&\text{
if }\textit{ valid }\\\ -\infty,&\text{ if }\textit{ invalid }\end{cases},$

where $p_{i}$ is the probability of the action, $\gamma$ is a scalar hyper-
parameter and $m_{i}$ is a correct coefficient to eliminate invalid HOI
categories (Chao et al. 2018). Finally, we train the model to fit this
probability distribution $\mathbf{p}$ of the actions in
$\mathcal{A}_{S}+\mathcal{A}_{U}$ as well as the ground truth actions in
$\mathcal{A}_{S}$.

###  4.4 Training and Inference

Training: We calculate the loss with extra interactive score loss and CLIP
distillation loss, as follows,

$\mathcal{L}_{total}=\mathcal{L}_{H}+\lambda_{is}\mathcal{L}_{is}+\lambda_{clip}\mathcal{L}_{clip},\\\
$ (7)

where $\mathcal{L}_{H}$ is computed by Eq. 3, $\mathcal{L}_{is}$ is the
interactive score loss, and $\mathcal{L}_{clip}$ is the CLIP distillation
loss. The $\mathcal{L}_{is}$ term adopts cross entropy loss, and the
$\mathcal{L}_{clip}$ term adopts binary cross entropy loss. $\lambda_{is}$ and
$\lambda_{clip}$ are the hyper-parameters.

Inference: After distilling the action knowledge from CLIP, we only keep the
learned CDN model for inference, avoiding extra computation cost. The post-
process of our method remains unchanged as CDN.

##  5 Experiments

###  5.1 Experimental Setup

Datasets and Evaluation Metrics: We perform our experiments on two HOI
detection benchmarks: HICO-DET (Chao et al. 2018) and V-COCO (Gupta and Malik
2015). We follow the standard evaluation (Chao et al. 2018) to use the mean
average precision ($m$AP) as the evaluation metric. A HOI triplet is
considered as a true positive when (1) the predicted object and action
categories are correct, and (2) both the predicted human and object bounding
boxes have intersection-over-union (IoU) with a ground truth greater than 0.5.

Table 1: The recall on unseen pairs (U-R). We compare the models trained with different interactive score modules and supervision sources. Our method achieves comparable recall for unseen pairs on HICO-Det test set. Results show that our method can efficiently detect potential interactive pairs. Method | Supervision | U-R@3 | U-R@5 | U-R@10
---|---|---|---|---
CDN | seen | 61.47 | 66.68 | 71.70
EoID | seen | 64.72 | 71.45 | 76.79
EoID | seen+$topk$ | 65.25 | 71.16 | 77.20
CDN | full | 67.03 | 73.41 | 78.64

Zero-shot Setups: We conduct experiments on HICO-Det: unseen combination
scenario (UC) (Bansal et al. 2020), Rare-first UC (RF-UC) (Hou et al. 2020),
Non-rare-first UC (NF-UC) (Hou et al. 2020), unseen action scenario (UA) (Liu,
Yuan, and Chen 2020) and UV (Liao et al. 2022). Details are shown in the
supplementary material.

Implementations: We benchmark on the CDN (Zhang et al. 2021) and use the same
settings for all models unless explicitly specified. The query number $N$ is
64. The loss weights $\lambda_{bbox},\lambda_{giou}$, $\lambda_{c}$,
$\lambda_{is}$, $\lambda_{a}$ and $\lambda_{clip}$ are set to 2.5, 1, 1, 1,
1.6 and 700 respectively. For simplicity, the decoupling dynamic re-weighting
in CDN is not used. For CLIP, we use the public pretrained model
111https://github.com/openai/CLIP., with an input size of $224\times 224$, and
$\gamma$=100. The cropped union regions are preprocessed by square padding and
resizing. We feed prompt engineered texts to the text encoder of CLIP with a
prompt template _a picture of person $\left\\{verb\right\\}$
$\left\\{object\right\\}$_. Experiments are conducted on 4 Tesla V100 GPUs,
with a batch size of 16.

###  5.2 Learning to Detect Unseen Pairs

We first show our framework can detect potential interactive pairs. We compare
the models trained with different interactive score (IS) modules and
supervision sources. Table 1 shows the top-k recall (U-R@K) of unseen pairs on
HICO-Det test set. Even though training only with seen pairs (seen), the
recall on unseen pairs of our model outperforms the CDN with naive IS module
by a large margin. After introducing the potential pairs ($topk$) for
training, the recall is further improved and shows comparable results with the
fully-supervised model. The result shows that our method can detect more
potential interactive pairs than the CDN.

We also compare the curves of the interactive score loss $\mathcal{L}_{is}$
and the overall $m$AP with IS branch from the interactive decoder
($IS_{inter}$) and human-object decoder ($IS_{ho}$), as shown in Fig. 4.
Compared to the model with $IS_{ho}$, the interactive score loss of the model
with $IS_{inter}$ converges faster, and also shows a better overall
performance. These results demonstrate the interactive decoder is more capable
of extracting interactive information.

Figure 4: The curves of the interactive score loss (left) and the $m$AP (right), compared between the models with IS branch from interactive decoder and human-object decoder. The model with IS branch from interactive decoder shows faster convergence and better performance. Table 2: We replace the action classifier of the converged CDN with CLIP to validate the zero-shot transferability of CLIP on HOI classification, with or without prior knowledge. Using CLIP for HOI classification shows the competitive performance with the full-supervised model on rare HOI categories of HICO-Det.  Method | Full | Rare | Non-rare
---|---|---|---
CLIP | 21.11 | 26.02 | 19.64
CLIP w/ prior | 21.45 | 26.42 | 19.97
Full-Supervised(seen + unseen) | 31.11 | 26.49 | 32.49

###  5.3 Zero-Shot Transferability of CLIP on HOI Classification

We replace the action classifier of the converged CDN model with CLIP to
validate the zero-shot transferability of CLIP on HOI classification. The
prior knowledge is also applied to avoid the dispersion of probability
distribution of CLIP to some invalid actions. The CLIP model is RN50x16 and
the CDN model is CDN-S. We report the $m$AP on the full set of HOI classes
(Full), a rare set of the HOI classes (Rare) that have less than 10 training
instances, and a non-rare set of the other HOI classes (Non-rare). As shown in
Table 2, the HOI model with a fixed CLIP classifier only has a small
degradation with the fully supervised model on Rare categories. This gap is
further reduced when prior knowledge is adopted, which shows the competitive
performance with the full-supervised model. However, the performances of Full
and Non-rare are not effective enough. Note that the CLIP classifies all the
predicted pairs without considering whether it is interactive or not. There is
still much improvement space for the overall performance and the inference
speed.

###  5.4 Ablation Studies

We perform ablation experiments in Sec. 5.4. Unless otherwise specified, the
CLIP model used here is RN50x16, the CDN model is CDN-S, the $topk$ and
$thres_{is}$ are set to 3 and 0.5 respectively. All ablation results are
evaluated on the HICO-Det test set.

Ablation of variants: As shown in Sec. 5.3, the degradation on Full and Non-
rare indicates the existence of noise from CLIP, which may lead to poor
performance on the Seen category. In addition, extra loss terms are introduced
into the framework which leads to the problem of convergence difficulties. We
study the methods to overcome the above challenges: 1) we only distill CLIP to
the HOI actions in $\mathcal{A}_{U}$ under UA setting, short as
$\mathcal{A}_{U}$ only; 2) we use detach technique to cut off the back-
propagation of gradients between the human-object decoder and interaction
decoder. As shown in Table 3, we can combine the $\mathcal{A}_{U}$ only to
alleviate the impact of the noise from CLIP, which improves the Seen and
Unseen by $1.05\%$ and $0.14\%$ $m$AP respectively. With the detach technique
adopted, the best Full performance is obtained to $29.22\%$ $m$AP.

Table 3: Ablation of variants. We study the methods to cope with the problems occurred in Sec. 5.3. $A_{U}$ only | detach | Full | Seen | Unseen
---|---|---|---|---
|  | 27.93 | 29.15 | 21.84
✓ |  | 28.83 | 30.20 | 21.98
| ✓ | 28.68 | 29.88 | 22.71
✓ | ✓ | 29.22 | 30.46 | 23.04
Table 4: Ablation studies. We perform ablation experiments to study the impact
of $topk$ and $thres_{is}$.

(a) Impact of $topk$. $topk$ | Full | Seen | Unseen
---|---|---|---
1 | 28.45 | 29.75 | 21.96
3 | 28.83 | 30.20 | 21.98
5 | 28.56 | 29.85 | 22.14
10 | 28.45 | 29.69 | 22.29

(b) Impact of $thres_{is}$. $thres_{is}$ | Full | Seen | Unseen
---|---|---|---
0.1 | 28.50 | 29.80 | 22.01
0.3 | 28.48 | 29.59 | 22.93
0.5 | 28.83 | 30.20 | 21.98
0.7 | 28.08 | 29.11 | 22.90
0.9 | 28.44 | 29.99 | 20.75

Impact of $topk$: We compare models with different $topk\in\\{1,3,5,10\\}$,
and results are shown in Table 4(a). The model with $topk$=3 obtains the best
performance on Full and Seen. With the growth of the $topk$, the model obtains
a better performance on Unseen while the overall $m$AP starts to drop for too
much noise.

Table 5: Zero-shot HOI Detection results on HICO-DET dataset. UC and UA(UV) denote unseen action-object combination and unseen action scenarios respectively, RF-UC and NF-UC denote rare-first and non-rare-first UC scenarios. Our method outperforms all the other methods by a large margin. The $*$ indicates that the model training without using $A_{U}$ only and detach technique. The ‡ denotes our implementation. Method | Type | Full | Seen | Unseen
---|---|---|---|---
VCL | RF-UC | 21.43 | 24.28 | 10.06
ATL | RF-UC | 21.57 | 24.67 | 9.18
FCL | RF-UC | 22.01 | 24.23 | 13.16
GEN-VLKT | RF-UC | 30.56 | 32.91 | 21.36
baseline | RF-UC | 28.46 | 30.80 | 19.10
EoID | RF-UC | 29.52 | 31.39 | 22.04
VCL | NF-UC | 18.06 | 18.52 | 16.22
ATL | NF-UC | 18.67 | 18.78 | 18.25
FCL | NF-UC | 19.37 | 19.55 | 18.66
GEN-VLKT | NF-UC | 23.71 | 23.38 | 25.05
baseline | NF-UC | 23.93 | 25.18 | 18.94
EoID | NF-UC | 26.69 | 26.66 | 26.77
Functional | UC | 12.45$\pm$0.16 | 12.74$\pm$0.34 | 11.31$\pm$1.03
ConsNet | UC | 19.81$\pm$0.32 | 20.51$\pm$0.62 | 16.99$\pm$1.67
GEN-VLKT‡ | UC | 25.23$\pm$0.59 | 27.16$\pm$0.88 | 20.64$\pm$0.89
baseline | UC | 26.57$\pm$0.43 | 28.65$\pm$0.58 | 18.24$\pm$1.02
EoID | UC | 28.91$\pm$0.27 | 30.39$\pm$0.40 | 23.01$\pm$1.54
ConsNet | UA | 19.04 | 20.02 | 14.12
GEN-VLKT‡ | UA | 26.28 | 28.72 | 20.85
baseline | UA | 26.53 | 28.77 | 15.30
EoID* | UA | 27.93 | 29.15 | 21.84
EoID | UA | 29.22 | 30.46 | 23.04
GEN-VLKT | UV | 28.74 | 30.23 | 20.96
EoID | UV | 29.61 | 30.73 | 22.71

Impact of $thres_{is}$: We compare models with different
$thres_{is}\in\\{0.1,0.3,0.5$, $0.7,0.9\\}$, and results are shown in Table
4(b). The models with a larger $thres_{is}$ will eliminate more non-
interactive pairs, which may results in a lager precision but a lower recall
of unseen pairs and vice versa. The best performance of Full and Seen are
obtained when adopting a proper $thres_{is}$=0.5.

###  5.5 Zero-Shot HOI Detection

We compare our method with state-of-the-art models on the HICO-Det test set
under RF-UC, NF-UC, UC, UA and UV settings in Table 5. The compared models
include: Functional (Bansal et al. 2020), VCL (Hou et al. 2020), ATL (Hou et
al. 2021a), FCL (Hou et al. 2021b), GEN-VLKT (Liao et al. 2022), and ConsNet
(Liu, Yuan, and Chen 2020). Besides, we introduce a new variant of CDN by
applying the consistency graph of ConsNet (CDN+ConsNet, details in
supplementary material) as our baseline to validate whether the better
performance is obtained by the better backbone. As shown in Table 5, our
method outperforms the previous SOTA on Unseen under five various zero-shot
settings, which valid the effectiveness of the proposed EoIDs. Compared to
GEN-VLKT, the slightly lower performance on Seen at the RF-UC setting may be
due to the different HOI models, which is evaluated in GEN-VLKT (Tab. 4(a)).
Even with a weaker HOI model CDN, our methods still outperform GEN-VLKT at the
remaining settings. Note that, our best Unseen performance is competitive with
the fully-supervised method (Zou et al. 2021) ($23.04\%$ v.s. $23.46\%$) and
these experiments indicate the effectiveness of our proposed zero-shot HOI
detection framework.

Table 6: Transfer to object detection datasets. We study the performance of our method on V-COCO with the bounding box annotations from COCO. Our method can transfer to datasets with only bounding boxes annotated to further scale up existing HOI categories. Training source | Method | Full | Seen | Unseen
---|---|---|---|---
HICO only | CDN | - | 35.15 | -
HICO+pseudo-V-COCO | CDN | - | 38.13 | -
HICO+pseudo-V-COCO | EoID | 40.39 | 38.13 | 47.15
V-COCO(full) | CDN | 56.43 | 54.56 | 62.05

###  5.6 Scaling up Action Sets via Object Detection Datasets

Current HOI detectors are limited in action size due to the small scale of HOI
detection datasets. Meanwhile, large-scale object detection datasets contain
potential unknown interaction pairs. As a result, we introduce a more
difficult but practical experiment, in which object detection datasets are
merged to further scale up the existing HOI categories. For this purpose, we
study the performance of our method on V-COCO with the bounding box
annotations from COCO. Specially, we compare the models trained on HICO-Det
dataset (HICO only), HICO-Det with pseudo-V-COCO and full V-COCO dataset,
where the pseudo-V-COCO consists of bounding boxes from COCO, pseudo-seen
action labels from the predictions of the CDN trained on HICO-Det dataset. We
test on V-COCO test set, and set the overlap actions between HICO-Det and
V-COCO as Seen and the others as Unseen. More details are elaborated in
Appendix. Results are shown in Table 6, and our method has an about $16\%$
overall $m$AP gap compared to the fully-supervised method, while a smaller gap
on Unseen categories. This experiment shows that our method can transfer to
bounding boxes annotated datasets to further scale up the existing HOI
categories.

##  6 Conclusions

In this work, we present EoID, an end-to-end zero-shot HOI detection framework
via knowledge distillation from multimodal vision-language embeddings. Our
method first detects potential action-agnostic interactive human-object pairs
by applying a two-stage bipartite matching algorithm and an interactive score
module. Then a zero-shot action classification is applied to identify novel
HOIs. The experiments demonstrate that our detector is able to detect unseen
pairs, which benefits the recognition of unseen HOIs. Our method outperforms
the previous SOTAs under four zero-shot settings and shows a promising
generalization to utilize large-scale detection datasets to scale up the
action sets.

## Acknowledgement

This work was supported by the National Science Fund for Distinguished Young
Scholars (No.62025603), the National Natural Science Foundation of China (No.
U21B2037, No. 62176222, No. 62176223, No. 62176226, No. 62072386, No.
62072387, No. 62072389, and No. 62002305), Guangdong Basic and Applied Basic
Research Foundation (No.2019B1515120049), and the Natural Science Foundation
of Fujian Province of China (No.2021J01002).

## References

  * Bansal et al. (2020) Bansal, A.; Rambhatla, S. S.; Shrivastava, A.; and Chellappa, R. 2020.  Detecting human-object interactions via functional generalization.  In _AAAI_.
  * Carion et al. (2020) Carion, N.; Massa, F.; Synnaeve, G.; Usunier, N.; Kirillov, A.; and Zagoruyko, S. 2020.  End-to-end object detection with transformers.  In _ECCV_.
  * Chao et al. (2018) Chao, Y.-W.; Liu, Y.; Liu, X.; Zeng, H.; and Deng, J. 2018.  Learning to detect human-object interactions.  In _WACV_.
  * Gao, Zou, and Huang (2018) Gao, C.; Zou, Y.; and Huang, J.-B. 2018.  iCAN: Instance-Centric Attention Network for Human-Object Interaction Detection.  In _BMVC_.
  * Gkioxari et al. (2018) Gkioxari, G.; Girshick, R.; Dollár, P.; and He, K. 2018.  Detecting and recognizing human-object interactions.  In _CVPR_.
  * Goyal et al. (2017) Goyal, Y.; Khot, T.; Summers-Stay, D.; Batra, D.; and Parikh, D. 2017.  Making the v in vqa matter: Elevating the role of image understanding in visual question answering.  In _CVPR_.
  * Gu et al. (2021) Gu, X.; Lin, T.-Y.; Kuo, W.; and Cui, Y. 2021.  Open-vocabulary Object Detection via Vision and Language Knowledge Distillation.  _arXiv:2104.13921_.
  * Gupta et al. (2021) Gupta, A.; Narayan, S.; Joseph, K.; Khan, S.; Khan, F. S.; and Shah, M. 2021.  OW-DETR: Open-world Detection Transformer.  _arXiv:2112.01513_.
  * Gupta and Malik (2015) Gupta, S.; and Malik, J. 2015.  Visual semantic role labeling.  _arXiv:1505.04474_.
  * Hou et al. (2020) Hou, Z.; Peng, X.; Qiao, Y.; and Tao, D. 2020.  Visual Compositional Learning for Human-Object Interaction Detection.  In _ECCV_.
  * Hou et al. (2021a) Hou, Z.; Yu, B.; Qiao, Y.; Peng, X.; and Tao, D. 2021a.  Affordance Transfer Learning for Human-Object Interaction Detection.  In _CVPR_.
  * Hou et al. (2021b) Hou, Z.; Yu, B.; Qiao, Y.; Peng, X.; and Tao, D. 2021b.  Detecting human-object interaction via fabricated compositional learning.  In _CVPR_.
  * Kim et al. (2020a) Kim, B.; Choi, T.; Kang, J.; and Kim, H. J. 2020a.  Uniondet: Union-level detector towards real-time human-object interaction detection.  In _ECCV_.
  * Kim et al. (2021) Kim, B.; Lee, J.; Kang, J.; Kim, E.-S.; and Kim, H. J. 2021.  HOTR: End-to-End Human-Object Interaction Detection with Transformers.  In _CVPR_.
  * Kim et al. (2020b) Kim, D.-J.; Sun, X.; Choi, J.; Lin, S.; and Kweon, I. S. 2020b.  Detecting human-object interactions with action co-occurrence priors.  In _ECCV_.
  * Kuhn (1955) Kuhn, H. W. 1955.  The Hungarian method for the assignment problem.  _Naval research logistics quarterly_.
  * Li et al. (2017) Li, Y.; Ouyang, W.; Zhou, B.; Wang, K.; and Wang, X. 2017.  Scene graph generation from objects, phrases and region captions.  In _ICCV_.
  * Li et al. (2019) Li, Y.-L.; Zhou, S.; Huang, X.; Xu, L.; Ma, Z.; Fang, H.-S.; Wang, Y.; and Lu, C. 2019.  Transferable interactiveness knowledge for human-object interaction detection.  In _CVPR_.
  * Liao et al. (2020) Liao, Y.; Liu, S.; Wang, F.; Chen, Y.; Qian, C.; and Feng, J. 2020.  Ppdm: Parallel point detection and matching for real-time human-object interaction detection.  In _CVPR_.
  * Liao et al. (2022) Liao, Y.; Zhang, A.; Lu, M.; Wang, Y.; Li, X.; and Liu, S. 2022.  GEN-VLKT: Simplify Association and Enhance Interaction Understanding for HOI Detection.  _arXiv:2203.13954_.
  * Lin et al. (2014) Lin, T.-Y.; Maire, M.; Belongie, S.; Hays, J.; Perona, P.; Ramanan, D.; Dollár, P.; and Zitnick, C. L. 2014.  Microsoft coco: Common objects in context.  In _ECCV_.
  * Liu, Yuan, and Chen (2020) Liu, Y.; Yuan, J.; and Chen, C. W. 2020.  Consnet: Learning consistency graph for zero-shot human-object interaction detection.  In _Proceedings of the 28th ACM International Conference on Multimedia_.
  * Luo et al. (2021) Luo, H.; Ji, L.; Zhong, M.; Chen, Y.; Lei, W.; Duan, N.; and Li, T. 2021.  Clip4clip: An empirical study of clip for end to end video clip retrieval.  _arXiv:2104.08860_.
  * Patashnik et al. (2021) Patashnik, O.; Wu, Z.; Shechtman, E.; Cohen-Or, D.; and Lischinski, D. 2021.  Styleclip: Text-driven manipulation of stylegan imagery.  In _ICCV_.
  * Radford et al. (2021) Radford, A.; Kim, J. W.; Hallacy, C.; Ramesh, A.; Goh, G.; Agarwal, S.; Sastry, G.; Askell, A.; Mishkin, P.; Clark, J.; et al. 2021.  Learning transferable visual models from natural language supervision.  In _ICCV_.
  * Rao et al. (2021) Rao, Y.; Zhao, W.; Chen, G.; Tang, Y.; Zhu, Z.; Huang, G.; Zhou, J.; and Lu, J. 2021\.  DenseCLIP: Language-Guided Dense Prediction with Context-Aware Prompting.  _arXiv:2112.01518_.
  * Shen et al. (2018) Shen, L.; Yeung, S.; Hoffman, J.; Mori, G.; and Fei-Fei, L. 2018.  Scaling human-object interaction recognition through zero-shot learning.  In _WACV_.
  * Tamura, Ohashi, and Yoshinaga (2021) Tamura, M.; Ohashi, H.; and Yoshinaga, T. 2021.  QPIC: Query-Based Pairwise Human-Object Interaction Detection with Image-Wide Contextual Information.  In _CVPR_.
  * Wang et al. (2022) Wang, S.; Duan, Y.; Ding, H.; Tan, Y.-P.; Yap, K.-H.; and Yuan, J. 2022.  Learning Transferable Human-Object Interaction Detector With Natural Language Supervision.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_.
  * Wang et al. (2020) Wang, T.; Yang, T.; Danelljan, M.; Khan, F. S.; Zhang, X.; and Sun, J. 2020.  Learning human-object interaction detection using interaction points.  In _CVPR_.
  * Yuan et al. (2022) Yuan, H.; Jiang, J.; Albanie, S.; Feng, T.; Huang, Z.; Ni, D.; and Tang, M. 2022\.  RLIP: Relational Language-Image Pre-training for Human-Object Interaction Detection.  _arXiv preprint arXiv:2209.01814_.
  * Zhang et al. (2021) Zhang, A.; Liao, Y.; Liu, S.; Lu, M.; Wang, Y.; Gao, C.; and Li, X. 2021.  Mining the Benefits of Two-stage and One-stage HOI Detection.  _NeurIPS_.
  * Zhou, Loy, and Dai (2021) Zhou, C.; Loy, C. C.; and Dai, B. 2021.  DenseCLIP: Extract Free Dense Labels from CLIP.  _arXiv:2112.01071_.
  * Zou et al. (2021) Zou, C.; Wang, B.; Hu, Y.; Liu, J.; Wu, Q.; Zhao, Y.; Li, B.; Zhang, C.; Zhang, C.; Wei, Y.; et al. 2021.  End-to-end human object interaction detection with hoi transformer.  In _CVPR_.


11institutetext: The University of Hong Kong, Pokfulam, Hong Kong
11email: {lcon7, qianxy10<EMAIL_ADDRESS>
11email<EMAIL_ADDRESS>
# Multimodal Transformer for Automatic 3D Annotation and Object Detection

Chang Liu  0000-0001-9791-4594 Xiaoyan Qian  0000-0003-4426-0667  Binxiao
Huang  0000-0001-5316-703X Xiaojuan Qi  0000-0002-4285-1626 Edmund Lam
0000-0001-6268-950X Siew-Chong Tan  0000-0001-9007-8749 Ngai Wong
0000-0002-3026-0108

###### Abstract

Despite a growing number of datasets being collected for training 3D object
detection models, significant human effort is still required to annotate 3D
boxes on LiDAR scans. To automate the annotation and facilitate the production
of various customized datasets, we propose an end-to-end multimodal
transformer (MTrans) autolabeler, which leverages both LiDAR scans and images
to generate precise 3D box annotations from weak 2D bounding boxes. To
alleviate the pervasive sparsity problem that hinders existing autolabelers,
MTrans densifies the sparse point clouds by generating new 3D points based on
2D image information. With a multi-task design, MTrans segments the
foreground/background, densifies LiDAR point clouds, and regresses 3D boxes
simultaneously. Experimental results verify the effectiveness of the MTrans
for improving the quality of the generated labels. By enriching the sparse
point clouds, our method achieves 4.48% and 4.03% better 3D AP on KITTI
moderate and hard samples, respectively, versus the state-of-the-art
autolabeler. MTrans can also be extended to improve the accuracy for 3D object
detection, resulting in a remarkable 89.45% AP on KITTI hard samples. Codes
are at https://github.com/Cliu2/MTrans.

###### Keywords:

3D Autolabeler, 3D Object Detection, Multimodal Vision, Self-attention, Self-
supervision, Transformer

##  1 Introduction

3D detection technologies have seen rapid development in recent years, with
considerable importance for tasks such as autonomous driving and robotics.
Large-scale datasets can often improve the performance and generality of deep
neural networks, hence new datasets such as nuScenes [1] and Waymo Open
Dataset [20] include millions of annotated objects and hundreds of thousands
of LiDAR scans. Nonetheless, the annotation procedure is extremely labor-
intensive, especially for 3D point clouds [12, 26]. Therefore, it is
imperative to explore ways to accelerate the annotation procedure.

Compared with 3D annotations, 2D bounding boxes are much easier to obtain.
Hence, we investigate the generation of 3D annotations from weak 2D boxes.
Despite its substantial practical value, only a few existing works study the
automatic annotation problem. Taking point clouds and images as the multimodal
inputs, FGR [26] converts 2D boxes to 3D boxes with a non-learning algorithm.
Alternatively, WS3D [11] leverages center clicks as weak annotations, and
directly regresses 3D boxes from cylindrical proposals centered by the clicks.
Both methods leverage the image information to reduce the problem space and
achieves state-of-the-art performances on the KITTI dataset [5]. Nevertheless,
their methods overlook the pervasive sparsity issue of point clouds. As shown
on the left of Fig. 1, even though the problem domain is narrowed down by the
weak 2D annotations, it is still challenging to generate a 3D box if the
points are too sparse, due to the ambiguity in orientation and scale.
Consequently, although these methods generate human-comparable annotations for
easy samples, they suffer obvious accuracy drops for hard samples that are
usually sparse, by 9.13% and 6.47% in 3D AP, respectively.

Figure 1: Left: The sparsity issue of point cloud for 3D box annotation.
Compared with the top object, the bottom is ambiguous in orientation and
scale. Both are from the Bird’s Eye View of LiDAR scans. Right: Interpolating
3D points by referring to context points based on 2D semantic and geometric
relationships. Point’s depth shown in color.

The sparsity problem remains a fundamental challenge for automatic
annotations. Besides the sparsity issue, point clouds lack color information.
Therefore, similar-shaped objects (e.g., traffic light post and pedestrian)
are difficult to distinguish, which further complicates the problem [23].
Fortunately, images provide dense fine-grid RGB information, which is an
effective complement to the sparse point clouds. Motivated by this, we
investigate point cloud enrichment with image information. As shown on the
right of Fig. 1, 3D LiDAR points can be projected onto the image, and
therefore an image pixel’s 3D coordinates can be estimated by referring to the
context LiDAR points, based on their 2D spatial and semantic relationships.
This way, new 3D points can be generated from 2D pixels, and hence densify the
sparse point cloud.

To this end, we propose the end-to-end multimodal transformer (MTrans) which
alleviates the sparsity problem and outputs accurate 3D box annotations given
weak 2D boxes. In order to leverage the dense image information to enrich
sparse point clouds, we present the multimodal self-attention module, which
efficiently extracts multimodal 2D and 3D features, modeling points’ geometric
and semantic relationships. To train MTrans, we also introduce a multi-task
design, so that it learns to generate new 3D points to reduce the sparsity of
point clouds while predicting 3D boxes. The multi-task design also includes a
foreground segmentation task to encourage MTrans to uncover point-wise
semantics. The multimodal self-attention and multi-task design work
collectively and enable the MTrans to efficiently exploit multimodal
information to address the sparsity issue, and hence facilitate more accurate
3D box prediction. Moreover, to exploit unlabelled data for the point
generation task, a _mask-and-predict_ self-supervision strategy can be carried
out. With a small amount of annotated data (e.g., 500 frames) and optional
unlabeled data, our method can generate high-quality 3D box annotations for
training other 3D object detectors. Comprehensive experiments are conducted,
and our MTrans outperforms all the other baseline methods significantly. Using
only 500 annotated frames, PointRCNN trained with MTrans can achieve 99.62% of
its original performance trained with full human annotations, which is 3.65%
higher in mAP than the state-of-the-art autolabeler. Furthermore, our point
generation approach for handling the sparsity issue also has direct value for
3D object detection tasks. Extended for object detection, MTrans surpasses
similar 2D-box-aided methods of F-PointNet [14] and F-ConvNet [25] by 19.6%
and 10.8% mAP, respectively. Our contributions are threefold:

  1. 1.

We propose a novel autolabeler, MTrans, which generates high-quality 3D boxes
from weak 2D bounding boxes, thus greatly saving manual work.

  2. 2.

MTrans effectively alleviates the pervasive sparsity issue for 3D point
clouds, and it can also be extended for 3D object detection tasks.

  3. 3.

Comprehensive experiments are conducted to justify our method. MTrans
outperforms existing state-of-the-art autolabelers by large margins.
Visualization results are also provided for interpretability.

##  2 Related Work

###  2.1 Multimodal 3D Object Detection

Several previous studies investigate multimodal networks, utilizing images and
LiDAR point clouds for a more comprehensive perception. Most of them extract
features from images and point clouds in different branches, followed by
fusing the information for final decision [2, 6, 33, 27]. MV3D [2] proposes
the deep fusion manner to repetitively exchange information from the different
views including the image, LiDAR’s front view and Bird’s Eye View (BEV),
during the inference. Later, EPNet [6] adopts Feature Pyramid Network to
encode LiDAR’s front view and RGB images, and fuses the two feature pyramids
with the gating mechanism. Similarly, ACMNet [33] and 3D-CVF [31] are based on
gating fusion as well. PI-RCNN [27] presents a hybrid PACF fusion module. To
align different views of the image and point clouds during fusion, AVOD [7]
introduces the 3D anchor grid to generate 3D proposals and applies fusion only
for the proposed region. Alternatively, Pi-RCNN [27] and PointPainting [23]
directly extract point cloud features with point-based backbones (e.g.,
PointNet [15] and PointNet++ [16]), and augment them with image features
through LiDAR-image calibration.

Rather than fusing different modalities, some other research employs images to
narrow down the problem space. PointPainting [23] performs image semantic
segmentation and highlights the points corresponding to the interested 2D
region. Similarly, Ref. [10] performs instance segmentation on images to
filter the 3D point clouds. F-PointNet [14] and F-Convnet [25] leverage 2D
bounding boxes on images to locate a frustum region of the point cloud, and
hence lowers the difficulty and improves accuracy. CLOCs [13] generates 2D box
proposals from images to filter 3D proposals, exploiting the consistency of
the two modalities to improve the overall detection accuracy.

The approaches above demonstrated the value of multiple modalities. The dense
image data can complement the sparse point clouds. Nevertheless, the number of
points remains unchanged, hence the sparsity problem is not substantially
solved. Also, the multi-branch design introduces extra computation and memory
overheads, leaving it still an open question on how to fully utilize
multimodal information [24].

###  2.2 Single-modal Detectors Addressing the Sparsity Problem

Noticing the sparsity problem, some researchers try to enrich the sparse
clouds to improve detection accuracy. SPG [29] firstly locates foreground
regions, where an object is likely to exist, and then generates semantic
points with a voxel-based point generation module. Specifically, the network
predicts whether each voxel is in the foreground or background, and then
generates a new point for each foreground voxel. BtcDet [28] densifies the
point clouds within voxel-based proposals as well. Instead of directly
generating new points, BtcDet finds a best matched prior object for each
proposal, and densifies the proposal by mirroring the original points as well
as aggregating the points from the prior object.

Both SPG and BtcDet alleviate the sparsity problem, resulting in state-of-the-
art performances on the KITTI detection dataset. The two works demonstrate the
value of enriching the sparse point cloud for the 3D object detection task.
Nonetheless, they are all trained with a large amount of annotated data, which
are not accessible for the automatic annotation task. Additionally, they only
leverage single modality, omitting the dense RGB information from images.

###  2.3 Autolabelers for 3D Object Detection

Although autolabelers have considerable potential for real-world applications,
only a few works have investigated this problem. FGR [26] is a non-learning
approach that takes 2D bounding boxes to locate frustum sub-point-clouds and
removes ground points with the RANSAC algorithm. After that, they
heuristically calculate the tightest 3D bounding box that can wrap all
remaining points for each object. In contrast, other deep-learning approaches
train their networks with a small amount of human annotations, and generate 3D
boxes from inexpensive weak annotations.

CC [21] extracts frustum sub-clouds in the same way as FGR, but adopts
PointNets [15] for segmenting and regressing 3D boxes. SDF [32] uses
predefined CAD models to estimate the 3D geometry of cars detected in 2D
images. VS3D [17] generates 3D proposals based on cloud density with an
unsupervised UPM module. Points in the proposals together with cropped images
are fed into a classification/regression network for the final 3D boxes.
Later, WS3D [12] and WS3D(2021) [11] utilize center clicks as weak 2D
annotations. Human annotators are asked to click on the objects’ centroids on
images, followed by adjusting the position in BEV of LiDAR scans. Points
within a cylindrical proposal centered by each click are processed by a
backbone network for the 3D pseudo box.

The above autolabelers utilize image modality and weak annotations to
automatically generate 3D boxes for point clouds. However, they mainly use
images to coarsely locate objects but do not address the sparsity problem of
point clouds. Consequently, their performances are still sub-optimal compared
with human annotations, having noticeable accuracy drops, especially for
moderate and hard samples, which are usually far and sparse.

##  3 MTrans for Automatic Annotation

In this section, we first define the automatic annotation problem, and then
introduce the proposed MTrans as an autolabeler, including multimodal self-
attention mechanism, multi-task design, and self-supervised point generation.

####  3.0.1 Automatic Annotation.

Given the LiDAR scan and its corresponding image, this research aims to
generate 3D bounding box annotations for interested objects (i.e., Cars) from
weak annotations of 2D bounding boxes. LiDAR points and image pixels are
connected by the LiDAR-image projection based on the calibration parameters.
Each point in the LiDAR point cloud can be projected to a pixel on the
corresponding image. The autolabeler is trained with a small amount of human
annotated data (e.g., 500 frames) and then used to label the remaining data.
Another object detection network can be trained with the auto-generated 3D
labels, saving manual work significantly.

Figure 2: Workflow of the MTrans. The network models both 3D and 2D
information from the LiDAR scans and images. The frustum points and the image
are fed into the network for simultaneous foreground segmentation, point
generation, and box regression. $\mathcal{F}_{f2d}$, $\mathcal{F}_{c2d}$ and
$\mathcal{F}_{c3d}$ are the embeddings for image patches, 2D coordinates and
3D coordinates. $\mathcal{F}_{s}$ is the global image feature derived by
averaging $\mathcal{F}_{f2d}$.

####  3.0.2 Overview.

As shown in Fig. 2, to address the sparsity problem and generate high quality
3D annotations, our MTrans takes multimodal inputs of LiDAR point clouds and
their corresponding images (Sec. 3.1). The multimodal inputs are encoded and
fused as point-level embedding vectors for further processing (Sec. 3.2). With
the proposed multimodal self-attention module, object features are extracted
based on the points’ semantic and geometric relationships (Sec. 3.3). Then the
extracted features are used for multiple tasks of foreground segmentation,
point generation, and 3D box regression, encouraging the MTrans to
comprehensively utilize the multimodal information and alleviate the sparsity
problem (Sec. 3.4). Additionally, a self-supervised training strategy is
proposed to leverage unlabeled data for the point generation task (Sec. 3.5).

###  3.1 Data Preparation

Our MTrans generates one 3D bounding box for each object in a weak 2D box.
Using the LiDAR-image calibration parameters, a 3D point $(x,y,z)$ can be
projected onto the image plane $(u,v)$ by the mapping function $f_{cal}$.
Therefore, we can extract the frustum sub-cloud $\mathcal{P}_{F}$
corresponding to a 2D box:

$\mathcal{P}_{F}=\\{(x,y,z)\;|\;f_{cal}(x,y,z)\in\mathcal{B}_{2D}\\},$ (1)

where $\mathcal{B}_{2D}$ is the region cropped by the 2D box. We use $n$ to
represent the frustum cloud size (i.e., the number of points). The points’ 2D
projections are defined as:

$\mathcal{C}_{2D}=\\{(u,v)\;|\;(u,v)=f_{cal}(x,y,z),\;(x,y,z)\in\mathcal{P}_{F}\\},$
(2)

A shown in Fig. 2, centered by the 2D projections, image patches of shape
$k\times k$ are extracted, denoted as $\mathcal{I}_{p}$. Thus, every LiDAR
point has three categories of features, $\mathcal{P}_{F}^{(i)}$,
$\mathcal{C}_{2D}^{(i)}$ and $\mathcal{I}_{p}^{(i)}$.

As mentioned in Sec. 1, in order to alleviate the sparsity problem, we
generate 3D points from the image pixels, by referring to the $n$ real LiDAR
points as _context_. Since the number of real LiDAR points, $n$, varies from
sample to sample, we randomly drop points or sample pixels from images as
_padding_ , resulting in fixed $n^{\prime}$ elements. Besides the padding
points, we further sample another $m$ pixels uniformly from the image, called
_target_ points. The padding and target points only have 2D features
($\mathcal{C}_{2D}^{(i)}$ and $\mathcal{I}_{p}^{(i)}$) without known 3D
coordinates ($\mathcal{P}_{F}^{(i)}$).

###  3.2 Feature Embeddings and Object Representation

Same as vanilla transformers [22, 4], we encode the object features as
embedding vectors. For the 2D position, we use sinusoidal position embedding
[22] to encode $u$ and $v$ into a half-length vector respectively, and then
concatenate them for the 2D position embedding $\mathcal{F}_{c2d}$. As the 3D
coordinates are continuous, a multilayer perceptron (MLP) is used to encode
the $(x,y,z)$ coordinates into 3D position embeddings $\mathcal{F}_{c3d}$.
Lastly, we extract image patch features $\mathcal{F}_{f2d}$ with a 3-layer
convolutional sub-network from $\mathcal{I}_{p}$. Specifically, we empirically
set the patch size as $7\times 7$ and kernel size as $3\times 3$. All of
$\mathcal{F}_{c2d}$, $\mathcal{F}_{c3d}$ and $\mathcal{F}_{f2d}$ have shapes
of $((n^{\prime}+m),d)$, where $d$ is the embedding vector length, for the
$n^{\prime}+m$ points. Note that for the padding and target points, they have
no known 3D coordinates. Therefore, their $\mathcal{F}_{c3d}$ features are
replaced by a trainable embedding vector $\mathcal{E}$.

Afterwards, we fuse the three kinds of features by concatenating them along
the $d$-axis, and decrease the channel from $3d$ back to $d$ by a fully-
connected layer. As shown in Fig. 2, we introduce an object-level token
$[CLS]$, which is a randomly-initialized trainable embedding vector. The total
$n^{\prime}+m+1$ element representations of the object,
$\mathcal{F}\in\mathbb{R}^{(n^{\prime}+m+1)\times d}$, are then input into the
MTrans for feature extraction with the proposed multimodal self-attention
mechanism.

Figure 3: Multimodal Self-attention. C-pts with known 3D coordinates attend to
each other, while each of the T-pts attends to itself as well as the C-pts.
$[CLS]$ attends to all available C-pts and T-pts for object-level
representation. For better visual effect, we show one T-pt doing self-
attention in the middle row. Best viewed in color.

###  3.3 Multimodal Self-attention

Following the vanilla transformer [22, 4], we manipulate the element
representations with the multi-head self-attention mechanism. The element
representations are transformed into query $Q$, key $K$, and value $V$. The
values $V$ are recombined based on attention scores calculated by multiplying
$Q$ and $K$. Owing to the space limitation, we do not repeat the technical
details of the multi-head self-attention mechanism, but only stress on the
modifications for our MTrans.

Different from the original transformer, under our problem setting, some
elements contain incomplete information (i.e., the padding and target points
have only 2D information but no 3D information). Therefore, we treat the
original frustum sub-cloud points with full information as the context points
(_C-pts_ in Fig. 3), and recover 3D coordinates for the padding and target
elements (_T-pts_). Specifically, the T-pts look up C-pts based on their
position and semantic relationships, and interpolate their 3D coordinates to
enrich the sparse point cloud. During our multimodal self-attention, the C-pts
only attend to each other but not T-pts, while each T-pt can also attend to
itself, and $[CLS]$ attends to all available points to aggregate object-level
information.

As shown in Fig. 3, the C-pts perform vanilla self-attention, and have
representations updated as $\text{C-pts}^{\prime}$. For each T-pts, a one-
element $Q$ is built from its element representation, while $K$ and $V$ are
from the concatenation of itself and the C-pts. Similarly, $[CLS]$ has $Q$
from itself, but $K$ and $V$ from the concatenation of $[CLS]$, T-pts, and
C-pts. With this asymmetric design, T-pts update themselves by absorbing 3D
information from C-pts, and $[CLS]$ encodes a global representation of the
object. For our MTrans, we stack 4 multimodal self-attention layers, and
vanilla feed-forward-layer [22] is adopted between each pair of the self-
attention layers.

###  3.4 Multi-task Prediction Heads

After the self-attention layers, the transformed element representations are
fed into multiple prediction heads for the multi-task supervision. Firstly, a
2-layer MLP $\mathcal{H}_{seg}$ performs binary segmentation for each point.
Points within the ground-truth 3D bounding box are regarded as foreground,
otherwise background. The segmentation loss $\mathcal{L}_{seg}$ is the
summation of Cross-Entropy loss and Dice loss. Since T-pts have no 3D
coordinates, only C-pts are supervised.

Another 2-layer MLP head $\mathcal{H}_{xyz}$ predicts the 3D coordinates for
the point generation task. Since there are no ground-truth values for the
padding and target points, we adopt a mask-and-predict strategy to train this
task with self-supervision, which will be introduced in detail in the next
section. Smooth $L_{1}$ loss is used for the point generation task as
$\mathcal{L}_{xyz}$. Intuitively, the foreground is more critical than
background, so we only calculate $\mathcal{L}_{xyz}$ for the foreground points
predicted by $\mathcal{H}_{seg}$. Moreover, we multiply the loss with a weight
of 0.1 if the point is a context point due to the leaked 3D coordinates,
otherwise 1.

The third prediction head $\mathcal{H}_{box}$ directly regresses 3D bounding
boxes from the object-level representations, which is derived by concatenating
the $[CLS]$ representation, the averaged element representations, and the
averaged image feature of $\mathcal{F}_{s}$, as shown in Fig. 2. The output
box is a 7-element vector for the location ($(x,y,z)$ coordinates), dimension
(length, width, height), and rotation along the z-axis. We use the dIoU loss
[35, 9, 34] for the box loss $\mathcal{L}_{box}$.

As the IoU (intersection over union) metric is direction-invariant (i.e.,
rotating a box for 180 degrees does not change the IoU), we further introduce
a direction head for the binary direction classification. Specifically,
orientation within $[-\pi/2,\pi/2)$ is regarded as the front, and
$[-\pi,-\pi/2)\cup[\pi/2,\pi]$ as the back. Cross Entropy loss is used for the
direction loss $\mathcal{L}_{dir}$.

All the heads are trained simultaneously, and the overall loss is given by:

$\mathcal{L}=\mathcal{L}_{seg}+\mathcal{L}_{xyz}+\lambda_{box}\mathcal{L}_{box}+\mathcal{L}_{dir},$
(3)

where the $\lambda_{box}$ is a coefficient and we empirically set it as 5.

###  3.5 Self-supervised Point Generation

One major motivation of the proposed MTrans is to enrich the sparse point
clouds with image information. As mentioned above, beyond the context points,
some extra T-pts (i.e., padding and target points) are appended with unknown
3D information. The point generation task, supervised by the loss
$\mathcal{L}_{xyz}$, aims to uncover the 3D coordinates for these T-pts based
on their 2D geometric and semantic relationships with the context points. Due
to the lack of ground truths, we adopt a self-supervised strategy for this
task. Specifically, we randomly mask a portion ($0\sim 95\%$) of the context
points, by replacing their $\mathcal{F}_{c3d}$ with the trainable embedding
vector $\mathcal{E}$. The masked points are then treated as T-pts and
supervised by the $\mathcal{L}_{xyz}$. With this partial supervision, we
managed to train the MTrans for the point generation task. During the
inference, no context points are masked and the model recovers the 3D
coordinates for the real T-pts.

##  4 Extension of MTrans to 3D Object Detection

Inspired by F-PointNet [14] and F-ConvNet [25], we can also convert the MTrans
into a 3D object detector. Similarly, our model lifts 2D detection results up
to 3D bounding boxes. The architecture and workflow remain the same as in Sec.
3. However, to be compatible for the detection metric of Average Precision
(AP), we add another prediction head $\mathcal{H}_{c}$ for the confidence of
the generated 3D box. The confidence is indicated by the estimated IoU score
of each generated 3D box. Instead of using a small amount of data for the
automatic annotation task, the detector version of the MTrans is trained with
full data. The overall loss is also updated as:

$\mathcal{L}=\mathcal{L}_{seg}+\mathcal{L}_{xyz}+\lambda_{box}\mathcal{L}_{iou}+\mathcal{L}_{dir}+\mathcal{L}_{c},$
(4)

where the new term $\mathcal{L}_{c}$ is the Smooth $L_{1}$ loss for confidence
head $\mathcal{H}_{c}$.

##  5 Experimental Evaluations

The proposed MTrans is evaluated for two tasks, the automatic annotation and
3D object detection. We also conduct ablation studies to justify the
contributions of each module.

####  5.0.1 KITTI Dataset.

KITTI [5] is a benchmark dataset for 3D detection for autonomous driving. We
follow the official practice to split the dataset into training and validation
sets with 3,712 and 3,769 frames. The standard evaluation metric of AP with
the IoU threshold being 0.7 is adopted. Same as FGR [26], we focus on the Car
class and filter out objects with less than 5 foreground LiDAR points, due to
some samples having too few points to possibly draw a 3D box.

###  5.1 Implementation Details

For the MTrans, 4 multimodal self-attention layers with the hidden size of 768
and 12 heads are used. As an autolabeler, the model is trained with 500 or 125
annotated frames for 300 epochs. The remaining frames are also used as
unlabeled data for self-supervision on the point generation task. Adam
optimizer is employed with a learning rate of $10^{-4}$. We also use standard
image augmentation methods of auto contrast, random sharpness and color
jittering, as well as point cloud augmentation methods of random translation,
scaling and mirroring. Due to some 2D boxes having overlaps, an overlap mask
of shape is added as a new channel of the image, where the overlapped areas
have values of 0, and otherwise 1. For each object, the cloud size
$n^{\prime}$ is set to 512 for context plus padding points, and $m$ is 784 for
the number of image-sampled target points.

Table 1: Automatic annotation results on KITTI _val_ set and _test_ set.

Method | Fully Supervised |  $\text{AP}_{3D}$ Val |  $\text{AP}_{3D}$ Test
---|---|---|---
Easy | Mod. | Hard | Easy | Mod. | Hard
PointPillars [8] | ✓ | 86.10 | 76.58 | 72.79 | 82.58 | 74.31 | 68.99
PointPillars [8] | 500f | 80.65 | 67.65 | 66.14 | - | - | -
PointPillars [8] | 125f | 71.36 | 58.29 | 55.11 | - | - | -
Ours + PointPillars | 500f | 86.69 | 76.56 | 72.38 | 77.65 | 67.48 | 62.38
Ours + PointPillars | 125f | 83.70 | 71.66 | 66.67 | - | - | -
PointRCNN [19] | ✓ | 88.99 | 78.71 | 78.21 | 86.96 | 75.64 | 70.70
PointRCNN [19] | 500f | 88.54 | 76.85 | 70.12 | - | - | -
PointRCNN [19] | 125f | 85.63 | 73.60 | 67.98 | - | - | -
Ours + PointRCNN | 500f | 88.72 | 78.84 | 77.43 | 83.42 | 75.07 | 68.26
Ours + PointRCNN | 125f | 87.64 | 77.31 | 74.32 | - | - | -
Compare with other autolabelers
WS3D [12] | BEV Centroid | 84.04 | 75.10 | 73.29 | 80.15 | 69.64 | 63.71
WS3D(2021) [11] | BEV Centroid | 85.04 | 75.94 | 74.38 | 80.99 | 70.59 | 64.23
FGR [26] | 2D Box | 86.67 | 73.55 | 67.90 | 80.26 | 68.47 | 61.57
Ours (PointRCNN) | 2D Box | 88.72 | 78.84 | 77.43 | 83.42 | 75.07 | 68.26

###  5.2 Automatic Annotation Results

To evaluate our MTrans as an autolabeler, we employ two popular backbone
object detection networks, PointRCNN [19] and PointPillars [18], and retrain
them with the auto-generated 3D annotations. To examine the influence of
different data scales, we also train them with all frames, 500 frames, and 125
frames of human-annotated data, respectively. As shown in Tab. 1, the accuracy
drops dramatically when the data are insufficient, especially for the moderate
and hard samples. With 125 frames of annotated data, PointPillars and
PointRCNN suffer mean accuracy drops of 16.9% and 6.23% in AP.

Meanwhile, the proposed MTrans is also trained with 500 frames and 125 frames
of annotated data, and then re-annotates the whole KITTI _train_ set. As shown
in Tab. 1, trained with the MTrans-generated annotations, the two networks
both significantly outperform their counterparts. For PointPillars, our method
brings 7.06% and 12.42% absolute average increases in AP on the _val_ set for
500 and 125 frames of data, respectively. Impressively, using only 500 and 125
frames, PointRCNN trained with MTrans can achieve 99.62% and 97.24% of the
original model trained with full data on the _val_ set. On the _test_ set,
MTrans-trained PointRCNN also yields 97.19% of the original model. The above
results demonstrate that our MTrans can generate high-quality 3D annotations,
producing object detection networks comparable to their counterparts trained
with human annotations. Automatically raising 2D boxes into 3D annotations,
our method greatly saves human efforts for the annotation procedure.

We also compare our method with existing state-of-the-art autolabelers, FGR
[26] and WS3D [11, 12]. Although they also generate 3D boxes from weak 2D
annotations, we want to stress the difference in task design for the three
approaches. FGR uses 2D boxes, same as in MTrans, but is a non-learning
algorithm, while WS3D uses center-clicks on the image and BEV, which might
provide extra 3D information. Using the same PointRCNN backbone, our method
surpasses all the above baselines significantly. Especially for the moderate
and hard samples, MTrans improves the 3D AP by 4.48% and 4.03% respectively.
This observation aligns with the motivation of enriching the sparse point
cloud for higher quality of generated labels, since moderate and hard samples
usually suffer more from the sparsity issue than easy samples.

Table 2: Detection results on KITTI _val_ set.

Method | Modality | Extra Ref. Signal | $\text{AP}_{3D}$ |  $\text{AP}_{3D}$ R40
---|---|---|---|---
Easy | Mod. | Hard | Easy | Mod. | Hard
MV3D [2] | LiDAR+RGB | ✗ | 71.29 | 62.68 | 56.56 | - | - | -
F-PointNet [14] | LiDAR+RGB | 2D box | 83.76 | 70.92 | 63.65 | - | - | -
F-ConvNet [25] | LiDAR+RGB | 2D box | 89.02 | 78.80 | 77.09 | - | - | -
VPFNet [36] | LiDAR+RGB | ✗ | - | - | - | 93.42 | 88.76 | 86.05
SECOND [30] | LiDAR | ✗ | 87.43 | 76.48 | 69.10 | 90.97 | 79.94 | 77.09
Voxel-RCNN [3] | LiDAR | ✗ | 89.41 | 84.52 | 78.93 | 92.38 | 85.29 | 82.86
PV-RCNN [18] | LiDAR | ✗ | 89.35 | 83.69 | 78.70 | 92.57 | 84.83 | 82.69
SPG [29] | LiDAR | ✗ | 89.81 | 84.45 | 79.14 | 92.53 | 85.31 | 82.82
BtcDet [28] | LiDAR | ✗ | - | - | - | 93.15 | 86.28 | 83.86
Ours | LiDAR+RGB | 2D box | 97.82 | 89.89 | 89.45 | 98.83 | 93.57 | 90.95

###  5.3 Detection Results

As mentioned in Sec. 4, with an extra prediction head for the box confidence,
the proposed MTrans can also be applied for 3D object detection task. Same as
F-PointNet [14] and F-ConvNet [25], we train our MTrans with full data, and
use ground-truth 2D boxes for frustum proposals. We evaluate our method on the
KITTI _val_ set, using metrics of $\text{AP}_{3D}$ and $\text{AP}_{3D}$ R40
(40 recall thresholds).

Shown in Tab. 2, our method achieves impressive performances of 92.38% and
94.46% for mAP and mAP(R40). The detection accuracy is significantly higher
than other state-of-the-art baselines. However, same as F-PointNet [14] and
F-ConvNet [25], our method utilizes the ground-truth 2D boxes as extra
reference signals, which reduces the problem difficulty. Nonetheless, in the
fair comparison with F-PointNet and F-ConvNet under identical task and
experiment settings, our method still boosts the mAP by 19.61% and 10.75%,
respectively.

Table 3: Ablation Results. | MM | Seg | Gen | ExtraPts | SelfSup | Metric
---|---|---|---|---|---|---
mIoU | Recall | mAP | $\text{mAP}_{R40}$
Vanilla | ✗ | ✗ | ✗ | ✗ | ✗ | 60.07 | 41.53 | 48.57 | 46.89
Multimodal | ✓ | ✗ | ✗ | ✗ | ✗ | 65.57 | 53.30 | 58.59 | 57.33
Seg Loss | ✓ | ✓ | ✗ | ✗ | ✗ | 69.45 | 61.53 | 67.84 | 69.42
XYZ Loss | ✓ | ✗ | ✓ | ✗ | ✗ | 67.61 | 57.76 | 63.62 | 63.82
Self-supervise | ✓ | ✗ | ✓ | ✗ | ✓ | 68.42 | 59.82 | 65.25 | 66.89
Densify | ✓ | ✗ | ✓ | ✓ | ✗ | 75.37 | 74.95 | 80.82 | 83.47
Full Generate | ✓ | ✗ | ✓ | ✓ | ✓ | 76.84 | 78.48 | 83.98 | 85.72
Ours | ✓ | ✓ | ✓ | ✓ | ✓ | 77.50 | 81.22 | 86.24 | 90.30
Ours(all frames) | ✓ | ✓ | ✓ | ✓ | ✓ | 81.01 | 87.65 | 92.36 | 94.43

###  5.4 Ablation Study

Ablation studies are conducted to verify the technical contributions of the
proposed method. We train the MTrans with 500 annotated frames of the KITTI
_train_ set, and evaluate the model on the _val_ set. The generated 3D boxes
are compared with the ground-truth human annotations, and three metrics are
employed, namely, the mean IoU (mIoU), recall with IoU threshold of 0.7, and
mean average precision (mAP, $\text{mAP}_{R40}$) over the Car class.

As shown in Tab. 3, the vanilla transformer, which directly processes 3D LiDAR
points $\mathcal{F}_{c3d}$, cannot model the problem well with the small
amount of data with only 500 frames. The _Multimodal_ variant introduces extra
modality of images, and improves the mIoU and recall significantly by 5.5% and
11.77%, respectively. _Seg Loss_ and _XYZ Loss_ introduce the multi-task
design, and again improve the accuracy significantly by 12.09 and 6.49%
$\text{mAP}_{R40}$, demonstrating the contribution of the auxiliary losses.
For _Densify_ , extra target points are evenly sampled from the image, and
their 3D coordinates are estimated to densify the point cloud a step further.
Alternatively, _Self-supervise_ utilize unlabeled data for the point
generation task, also improving the accuracy, resulting in 68.42% mIoU.
Combining all the point cloud densification techniques, _Full Generate_ ’s
mIoU and recall both surpass 70%. Leveraging all the modules, our MTrans
archives impressive 81.22% Recall and 86.24% mAP using only 500 frames of
annotated data, which is even comparable with state-of-the-art object
detectors trained with full annotations.

The ablation studies justify the contributions of our four innovations,
namely, the multimodal network, auxiliary multiple tasks, point generation,
and self-supervision. The multimodal inputs can effectively boost accuracy
with negligible modification on the vanilla transformer (_Multimodal_
variant). Moreover, the introduction of the auxiliary multi-task design (_Seg
Loss_ and _XYZ Loss_) raises the mIoU by 2.04% and 3.88%, showing that the
model is encouraged to understand the problem more comprehensively from
multiple aspects. The largest gain, however, comes from adding extra target
points from the image, which increases the mIoU by 7.76% and mAP by 20.35%
(_Densify_). This observation well supports our motivation that image-guided
interpolation can enrich the sparse point cloud for better annotations.
Combining all the techniques, our final MTrans yields a surprising 77.5% mIoU
and 81.22% recall, with only 500 annotated frames. Nonetheless, the gains from
these techniques tend to be saturated when combining all of the modules.
Compared with the mAP accuracy for all-data-trained MTrans at the bottom of
Tab. 1, we hypothesize that the main constraint is the amount of training
data.

Figure 4: Attention score visualization results. Context points and their
average attention scores are plotted. Each point’s color represents the amount
of attention it receives from other points. The attention scores sum to 1 for
each data sample. It is observed that the foreground points tend to receive
more attention than background points, which provides an intuitive
interpretation for the multimodal attention MTrans.

##  6 Visualization of Attention on Context Points

In Fig. 4 we visualize the attention distribution over context points.
Specifically, for each context point, we average the attention scores it
received from all other points. The attention scores of all context points sum
to 1. As shown in the figure, points that belong to the car tends to receive
more attention than background points, which aligns with our intuition, since
background points provide little reference value of labeling 3D boxes.
Moreover, the attention distribution also demonstrates a coherent spatial
pattern in both the 2D image contents and the 3D space. This observation
suggests that multimodal self-attention extracts object features based on 3D &
2D geometric, and 2D semantic relationships.

Another interesting phenomenon is that the foreground points closer to object
edges tend to receive more attention. As shown in the figures, yellow points
are presented near box corners for all four samples. This is also reasonable
since corner points usually provide more critical information for 3D box
prediction. The observation provides good interpretability for our MTrans.

##  7 Conclusion

This work has proposed the MTrans, an autolabeler that generates 3D boxes from
weak 2D box annotations, given the LiDAR point clouds and corresponding
images. Our method effectively alleviates the pervasive sparsity problem of
point clouds. Specifically, we introduce the multimodal self-attention
mechanism, together with auxiliary multi-task and self-supervision design,
which leverage image information to generate extra 3D points. Experimental
results demonstrate that our method can outperform existing autolabelers
significantly, generating 3D box annotations comparable to human annotations.
Moreover, the proposed MTrans can also be applied for the 3D object detection
task, achieving state-of-the-art performances. Beyond LiDAR + monocular image,
an interesting future direction would be to incorporate more modalities to
improve the detection accuracy and robustness, such as radar and/or stereo
images. It would also be valuable to investigate the domain transfer problem
for the autolabelers, which could save human effort further. We hope this
study can inspire future works to solve the sparsity problem as well as to
develop automatic annotation workflows to reduce manual workloads.

## References

  * [1] Caesar, H., Bankiti, V., Lang, A.H., Vora, S., Liong, V.E., Xu, Q., Krishnan, A., Pan, Y., Baldan, G., Beijbom, O.: nuscenes: A multimodal dataset for autonomous driving. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 11621–11631 (2020)
  * [2] Chen, X., Ma, H., Wan, J., Li, B., Xia, T.: Multi-view 3d object detection network for autonomous driving. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. pp. 1907–1915 (2017)
  * [3] Deng, J., Shi, S., Li, P., Zhou, W., Zhang, Y., Li, H.: Voxel r-cnn: Towards high performance voxel-based 3d object detection. In: Proceedings of the AAAI Conference on Artificial Intelligence. vol. 35, pp. 1201–1209 (2021)
  * [4] Devlin, J., Chang, M.W., Lee, K., Toutanova, K.: Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805 (2018)
  * [5] Geiger, A., Lenz, P., Urtasun, R.: Are we ready for autonomous driving? the kitti vision benchmark suite. In: Conference on Computer Vision and Pattern Recognition (CVPR) (2012)
  * [6] Huang, T., Liu, Z., Chen, X., Bai, X.: Epnet: Enhancing point features with image semantics for 3d object detection. In: European Conference on Computer Vision. pp. 35–52. Springer (2020)
  * [7] Ku, J., Mozifian, M., Lee, J., Harakeh, A., Waslander, S.L.: Joint 3d proposal generation and object detection from view aggregation. In: 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). pp. 1–8. IEEE (2018)
  * [8] Lang, A.H., Vora, S., Caesar, H., Zhou, L., Yang, J., Beijbom, O.: Pointpillars: Fast encoders for object detection from point clouds. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 12697–12705 (2019)
  * [9] lilanxiao: Differentiable iou of oriented boxes. https://github.com/lilanxiao/Rotated_IoU (2021)
  * [10] McCraith, R., Insafutdinov, E., Neumann, L., Vedaldi, A.: Lifting 2d object locations to 3d by discounting lidar outliers across objects and views. arXiv preprint arXiv:2109.07945 (2021)
  * [11] Meng, Q., Wang, W., Zhou, T., Shen, J., Jia, Y., Van Gool, L.: Towards a weakly supervised framework for 3d point cloud object detection and annotation. IEEE Transactions on Pattern Analysis and Machine Intelligence (2021)
  * [12] Meng, Q., Wang, W., Zhou, T., Shen, J., Van Gool, L., Dai, D.: Weakly supervised 3d object detection from lidar point cloud. In: European Conference on Computer Vision. pp. 515–531. Springer (2020)
  * [13] Pang, S., Morris, D., Radha, H.: Clocs: Camera-lidar object candidates fusion for 3d object detection. In: 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). pp. 10386–10393. IEEE (2020)
  * [14] Qi, C.R., Liu, W., Wu, C., Su, H., Guibas, L.J.: Frustum pointnets for 3d object detection from rgb-d data. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. pp. 918–927 (2018)
  * [15] Qi, C.R., Su, H., Mo, K., Guibas, L.J.: Pointnet: Deep learning on point sets for 3d classification and segmentation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. pp. 652–660 (2017)
  * [16] Qi, C.R., Yi, L., Su, H., Guibas, L.J.: Pointnet++: Deep hierarchical feature learning on point sets in a metric space. arXiv preprint arXiv:1706.02413 (2017)
  * [17] Qin, Z., Wang, J., Lu, Y.: Weakly supervised 3d object detection from point clouds. In: Proceedings of the 28th ACM International Conference on Multimedia. pp. 4144–4152 (2020)
  * [18] Shi, S., Guo, C., Jiang, L., Wang, Z., Shi, J., Wang, X., Li, H.: Pv-rcnn: Point-voxel feature set abstraction for 3d object detection. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 10529–10538 (2020)
  * [19] Shi, S., Wang, X., Li, H.: Pointrcnn: 3d object proposal generation and detection from point cloud. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 770–779 (2019)
  * [20] Sun, P., Kretzschmar, H., Dotiwalla, X., Chouard, A., Patnaik, V., Tsui, P., Guo, J., Zhou, Y., Chai, Y., Caine, B., et al.: Scalability in perception for autonomous driving: Waymo open dataset. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 2446–2454 (2020)
  * [21] Tang, Y.S., Lee, G.H.: Transferable semi-supervised 3d object detection from rgb-d data. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 1931–1940 (2019)
  * [22] Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, Ł., Polosukhin, I.: Attention is all you need. In: Advances in Neural Information Processing Systems. pp. 5998–6008 (2017)
  * [23] Vora, S., Lang, A.H., Helou, B., Beijbom, O.: Pointpainting: Sequential fusion for 3d object detection. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 4604–4612 (2020)
  * [24] Wang, W., Tran, D., Feiszli, M.: What makes training multi-modal classification networks hard? In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 12695–12705 (2020)
  * [25] Wang, Z., Jia, K.: Frustum convnet: Sliding frustums to aggregate local point-wise features for amodal 3d object detection. In: 2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). pp. 1742–1749. IEEE (2019)
  * [26] Wei, Y., Su, S., Lu, J., Zhou, J.: Fgr: Frustum-aware geometric reasoning for weakly supervised 3d vehicle detection. In: 2021 IEEE International Conference on Robotics and Automation (ICRA). pp. 4348–4354. IEEE (2021)
  * [27] Xie, L., Xiang, C., Yu, Z., Xu, G., Yang, Z., Cai, D., He, X.: Pi-rcnn: An efficient multi-sensor 3d object detector with point-based attentive cont-conv fusion module. In: Proceedings of the AAAI Conference on Artificial Intelligence. vol. 34, pp. 12460–12467 (2020)
  * [28] Xu, Q., Zhong, Y., Neumann, U.: Behind the curtain: Learning occluded shapes for 3d object detection. arXiv preprint arXiv:2112.02205 (2021)
  * [29] Xu, Q., Zhou, Y., Wang, W., Qi, C.R., Anguelov, D.: Spg: Unsupervised domain adaptation for 3d object detection via semantic point generation. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 15446–15456 (2021)
  * [30] Yan, Y., Mao, Y., Li, B.: Second: Sparsely embedded convolutional detection. Sensors 18(10), 3337 (2018)
  * [31] Yoo, J.H., Kim, Y., Kim, J., Choi, J.W.: 3d-cvf: Generating joint camera and lidar features using cross-view spatial feature fusion for 3d object detection. In: Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XXVII 16. pp. 720–736. Springer (2020)
  * [32] Zakharov, S., Kehl, W., Bhargava, A., Gaidon, A.: Autolabeling 3d objects with differentiable rendering of sdf shape priors. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 12224–12233 (2020)
  * [33] Zhao, S., Gong, M., Fu, H., Tao, D.: Adaptive context-aware multi-modal network for depth completion. IEEE Transactions on Image Processing (2021)
  * [34] Zheng, Z., Wang, P., Liu, W., Li, J., Ye, R., Ren, D.: Distance-iou loss: Faster and better learning for bounding box regression. In: Proceedings of the AAAI Conference on Artificial Intelligence. vol. 34, pp. 12993–13000 (2020)
  * [35] Zhou, D., Fang, J., Song, X., Guan, C., Yin, J., Dai, Y., Yang, R.: Iou loss for 2d/3d object detection. In: 2019 International Conference on 3D Vision (3DV). pp. 85–94 (2019)
  * [36] Zhu, H., Deng, J., Zhang, Y., Ji, J., Mao, Q., Li, H., Zhang, Y.: Vpfnet: Improving 3d object detection with virtual point based lidar and stereo data fusion. arXiv preprint arXiv:2111.14382 (2021)