# Detecting Human-Object Interaction with Mixed Supervision

Suresh Kirthi Kumaraswamy
Univ Le Mans, CNRS, IRISA
<EMAIL_ADDRESS>Miaojing Shi
King’s College London
<EMAIL_ADDRESS>Ewa Kijak
Univ Rennes, Inria, CNRS, IRISA
<EMAIL_ADDRESS>
###### Abstract

Human object interaction (HOI) detection is an important task in image
understanding and reasoning. It is in a form of HOI triplet ${\langle
human,verb,object\rangle}$, requiring bounding boxes for human and object, and
action between them for the task completion. In other words, this task
requires strong supervision for training that is however hard to procure. A
natural solution to overcome this is to pursue weakly-supervised learning,
where we only know the presence of certain HOI triplets in images but their
exact location is unknown. Most weakly-supervised learning methods do not make
provision for leveraging data with strong supervision, when they are
available; and indeed a naive combination of this two paradigms in HOI
detection fails to make contributions to each other. In this regard we propose
a mixed-supervised HOI detection pipeline: thanks to a specific design of
momentum-independent learning that learns seamlessly across these two types of
supervision. Moreover, in light of the annotation insufficiency in mixed
supervision, we introduce an HOI element swapping technique to synthesize
diverse and hard negatives across images and improve the robustness of the
model. Our method is evaluated on the challenging HICO-DET dataset. It
performs close to or even better than many fully-supervised methods by using a
mixed amount of strong and weak annotations; furthermore, it outperforms
representative state of the art weakly- and fully-supervised methods under the
same supervision.

##  1 Introduction

Figure 1: Human-object interaction detection with different levels of
supervision. Top: annotation cost increases from image-level ($x$) labels
(left) in weakly-supervised learning to region-level ($r$) bounding boxes
(middle) and their correspondences (right) in fully-supervised learning.
Bottom: our proposed mixed-supervised HOI detection pipeline (middle) enables
joint learning of weakly- and fully-supervised HOI detection (left and right).

The task of human-object interaction (HOI) detection is defined as a detection
of a triplet ${\langle human,verb,object\rangle}$, identifying not only the
bounding boxes of human and object but also their interaction [14, 49, 19, 36,
4, 57, 8, 10]. It is derived from visual relationship detection (VRD) of
triplet ${\langle object_{1},predicate,object_{2}\rangle}$ [31, 18, 16, 17,
37, 1, 55, 42, 56], but present different challenges: the predicates in VRD
can be verbs (e.g. “push”), spatial (e.g. “on top of”), prepositions (e.g.
“with”), comparative (e.g. “ taller than”), while in HOI they are mainly
verbs. On the other hand, _human-centric_ interactions are more diverse and
complicated, one person can easily interact with multiple objects in the
meantime, e.g. “person wearing a jacket and riding a bicycle”. This makes HOI
a much more fine-grained task than VRD.

Intensive attention has been drawn to HOI alongside the introduction of new
benchmarks, i.e. V-COCO [18], HICO-DET [5], featured with diverse and numerous
human-object interactions. For instance, in HICO-DET, there exist 600 HOI
classes and 80 common object classes in total. Despite that recent advances
have reported significant improvement [19, 36, 4, 57], the annotation cost is
exponentially increased in these datasets. Given $N$ humans and $M$ objects in
an image, the maximum number of HOI is $N\times M$, where we have to scan each
of them and provide instance-level annotations (e.g. bounding boxes) for the
real ones (see Fig. 1: top-right). To alleviate this manual labor, we could
provide only image-level HOI labels: a set of images are known to contain
triplets of a certain HOI class, but the location and correspondence of
objects are unknown in images (see Fig. 1: top-left); note the location of
objects are assumed known sometimes (top-middle). Both cases can be
conceptualized as weakly-supervised HOI detection.

There are few works that learn interactions from weak supervision. One
representative is for VRD [54], which designs a weakly-supervised predicate
prediction module inspired from the two-branch parallel structure in [3]. This
can be easily adapted to HOI detection, as shown in Fig. 1: bottom-left.
Nevertheless, in real application, instead of having only one type of labels,
we can have a mixture of them: fully-labeled (instance-level), weakly-labeled
(image-level), and even unlabeled.

A generalized HOI detection framework for mixed supervision thus becomes
necessary. An intuitive solution would be merging the weakly-supervised and
fully-supervised HOI detection as a multi-task job, which is nevertheless not
straightforward: the different quality of annotations between weakly-labeled
and fully-labeled data, as well as their imbalanced ratios should be
considered. Fig. 4 illustrates an example: in the HICO-DET dataset, when
adding different amounts of fully-labeled data (in red), results are either
only slightly better than or even worse than learning with only weakly-labeled
data (in grey). This simple combination, at best, does not exploit the full
potential that could be derived from fully-supervised data, at worst,
decreases the results obtained with weak supervision. This is the first
challenge that needs to be solved in the mixed-supervised setting.
Furthermore, HOI detection is a fine-grained task requiring the classification
of similar interactions such as ”eating”, ”drinking”, ”blowing”. To be able to
accurately distinguish them, diverse and hard negatives from similar
interactions are essential for the network training. Nevertheless, due to the
reason that many samples are weakly-labeled, interactions within them can not
be clearly discriminated on the region-level; plus, some interaction classes
are not even sufficiently collected. This poses another challenge for HOI
detection.

Contributions. We for the first time propose a generalized framework for
mixed-supervised HOI detection (MX-HOI):

  * •

We integrate two state-of-the-art pipelines [19] and [54] for fully- and
weakly-supervised HOI detection into a mixed-supervised pipeline.

  * •

To tackle the multi-task optimization in the mixed pipeline, we introduce a
momentum-independent learning strategy to tackle the adversarial effect
between full and weak supervision, by separating their gradient history in
momentum learning.

  * •

To tackle the annotation insufficiency in the mixed supervision, we introduce
an HOI element swapping strategy to specifically harvest hard negatives across
images for the weakly-labeled data.

By conducting our generalized HOI detection framework on the most challenging
HICO-DET dataset, we show our method enables HOI detection with a mixed amount
of supervision, e.g. with 30% fully labeled data and 70% weakly-labeled data,
we are able to retain 93.3% accuracy of the setting of 100% fully-labeled
data. Furthermore, 1) our model improves both the state of the art weakly- and
fully-supervised HOI detection methods [54, 19] under the same supervision; 2)
unlabeled data can also be leveraged into our pipeline following a ”pseudo
label” solution [23], where we can use the network trained on labeled data to
infer labels of HOI pairs on unlabeled data.

##  2 Related Work

Visual relationships were originally used to help improve object localization
[16, 22, 38], action recognition and pose estimation [9, 37] or semantic
segmentation [15]. Relationships that are often modelled between objects
include verbs, actions, spatial and prepositions [38, 52, 18, 16, 17, 37, 1,
20, 7]. [31] was the first work to formulate the detection of visual
relationships as a separate task. They propose to leverage language priors
from semantic word embedding to finetune the likelihood of a predicted
relationship. Subsequently, many researchers improved and generalized this
model [53, 27, 7]. Triplet metric learning is also adopted to optimize the
visual feature connections among semantically related objects in [24, 42].
Attention [20, 55] and spatial locations [56] are some other additive cues to
visual relationship detection.

Human-Object Interaction is a concept related to visual relationship. The
interactions between humans and objects are mainly focused on verbs, and are
much more fine grained (e.g. holding, hitting, throwing, touching) than
relationships between generic objects. The study of HOI dates back to [8, 10,
17, 35, 49], when most works were tested on small datasets. This issue was
addressed by Chao et al. [5] where they introduced a large dataset (HICO-DET)
covering 80 common object categories and 600 HOI categories in total. Many
recent works report their performance on this dataset and significant
improvement has been achieved [36, 4, 19, 29, 12, 25, 46, 47, 26, 45]. For
instance, Qi et al. [36] proposed a graph parsing neural network for HOI and
was later extended by [21, 48, 57]; Gupta et al. [19] showed that a simple
factorised model with appearance and layout encoding constructed from
pretrained object detectors outperforms more sophisticated approaches;
additional cues such as language features [12], parts based features [25, 46]
are also exploited. Our MX-HOI is built on the recent advance of [19].

Figure 2: Illustration of our mixed-supervised HOI detection pipeline (MX-
HOI). Human and object bounding boxes are obtained via an object detector.
Human-object pairs are randomly created within an image and also across
another image via the proposed HOI element swapping. HOI detection is realized
via a two-branch structure in parallel for interaction classification and
selection. Each branch consists of FC layer ($w_{c}$/$w_{s}$) for score
prediction and softmax layer ($\sigma_{c}$/$\sigma_{s}$) for score
normalization over rows or columns of the score matrix, respectively. The
score matrices in the two branches are of size $N$ (human-object pairs) and
$C$ (HOI classes) and are multiplied to produce the final matrix $P$. Training
data with full and weak supervision ($fs$, $ws$) are optimized with region-
level and image-level ground truth, respectively. We introduce a momentum-
independent strategy to enable the mixed-supervised learning with two momentum
$z^{fs}$ and $z^{ws}$. Human-object pairs from two images are optimized in one
batch.

Weakly-supervised learning in visual recognition is mostly focused on object
detection [3, 40, 44, 39, 43, 51]. One seminal work in weakly-supervised
object detection (WSOD) is [3], where they designed a popular two-branch
parallel structure followed by [44, 43, 51]. Weakly-supervised relationship
detection is more complex than WSOD as we need to detect individual objects
for specific relations. Pretrained object detectors are normally assumed in
this scenario [34, 54, 33]. Peyre et al. [34] proposed a weakly-supervised
discriminative clustering model to learn relations with only image-level
labels; later on, they developed another model for transfer by analogy to
obtain visual phrases of never seen relations [33]. Zhang et al. [54] adopted
the WSOD module in [3] to do weakly-supervised relationship detection and
achieved very good results. In this work, we also adopt the WSOD module
following [54] and adapt it to be part of our MX-HOI pipeline.

Mixed-supervised learning normally refers to learning from a mixture of
strongly labeled data and weakly-labeled data. For instance, Cinbis et al. [6]
considered mixed supervision in object detection where some images are
annotated with bounding boxes while some are only with image-level labels.
Papandreou et al. [32] studied the problem for semantic image segmentation
from a combination of few strongly labeled (pixel-level annotations) and many
weakly labeled (image-level labels or bounding boxes) images. Mixed-supervised
learning can also be realized as leveraging an existing dataset of fully-
labeled training images of non-target classes during the weakly-supervised
learning of a new object category, which is connected to transfer learning,
see e.g. [11, 41, 39, 50].

##  3 Mixed-supervised HOI detection

###  3.1 Preliminary

We build our MX-HOI framework on two state of the art HOI works with full
supervision [19] and weak supervision [54], respectively. [19] introduces a
no-frills model for HOI detection where they use appearance features from
pretrained object detectors, spatial features through box layout, and encode
human pose keypoints, as shown in Fig. 2: feature extraction. This is a no-
frills detection without relying on attention or graph-based message passing
[55, 36]. It uses a factorized multi-layer perceptrons (MLPs) and introduces
several new training techniques to improve the MLPs: eliminating a train-
inference mismatch, rejecting easy negatives using indicator terms, and
training with large negative to positive ratios.

[54] adopts the weakly-supervised object detection pipeline [3] for weakly-
supervised predicate prediction (WSPP): it is accomplished via the element-
wise multiplication of the predicate selection and classification branch (see
Fig. 2). The predicate score is softmax normalized over all candidate human-
object box pairs with respect to a predicate class for the selection branch,
and over all possible predicate classes with respect to one human-object pair
for the classification branch, respectively. The predicate score in [54] is
obtained from a position-role sensitive score maps with a pairwise ROI
pooling. To integrate it into the no-frills model above, we use the
conventional ROI pooling. Predicate scores are predicted from the FC layers of
the two branches, which is similar to the original structure in [3].

###  3.2 Overview

We introduce a mixed-supervised HOI detection framework (MX-HOI) as shown in
Fig. 2: the input for MX-HOI is region proposals output from a pretrained
object detector. We follow the same procedure as in [19] to extract both
appearance and layout features for the human and object bounding boxes in a
pair. Given human-object pairs, their region features are fed into the adapted
two-branch predicate prediction structure from [54] (Sec. 3.1). The output of
the two branches (matrices) are multiplied element-wise to produce one
$N\times C$ matrix $P$ over $N$ human-object pairs (in a batch) and $C$
interaction classes. Each element $p_{ij}$ indicates the probability of the
$i^{\text{th}}$ human-object pair having $j^{\text{th}}$ interaction type. For
fully-labeled data, the predicate prediction is optimized on the region-level
on matrix $P$, where a corresponding ground truth matrix is associated with
each element being 1 or 0 indicating the true or false for the human-object
interaction. For weakly-labeled data, the predicate optimization is on the
image-level: $P$ is accumulated over rows ($\sum_{i}p_{ij}$) to produce a
$C$-dimensional vector where each element signifies the probability of the
image containing certain HOI class. Similarly, an image-level ground truth
vector with elements 1 or 0 is associated.

This is a multi-task optimization defined jointly with full and weak
supervision. The learning is not straightforward: the optimization in the two
learning manners is different as one focuses on the image-level and the other
on the region-level ($x$ and $r$ in Fig. 1); the error surface toggles between
the gradients from weak supervision and full supervision across different
batches in the network. We therefore propose a momentum-independent learning
strategy (Sec. 3.3). Besides, for the weakly-labeled data in mixed-supervised
learning, we introduce an HOI element swapping strategy (Sec. 3.4) to further
augment the hard negatives. Loss function is given in the end (Sec. 3.5).

###  3.3 Momentum-independent Learning

In the context of mixed-supervised learning, network weights are updated by
either weak or full supervision, depending on the samples within the mini-
batch. Most recent gradient descent based optimizers use momentum-based
weights update. Let $w_{t}$ and $\nabla f(w_{t})$ be respectively the weight
and the gradient at iteration $t$, and $\alpha$ be the step size, the
momentum-based update rule is given by:

$\displaystyle\vspace{-3mm}w_{t}=w_{t-1}-z_{t};~{}~{}z_{t}=\beta
z_{t-1}+\alpha\nabla f(w_{t-1})\vspace{-3mm}$ (1)

where $\beta$ is the momentum parameter (usually $\beta\geq 0.9$) and $z_{t}$
is the momentum, which is dependent on all the previous gradient values.

Using momentum-based gradient descent can however be a problem in the mixed-
supervised learning. In the fully-supervised case, the ground truth is
directly given on the instance level such that the gradient of the loss
function will accordingly backpropagate to the specific regions of the human
and object in a pair. In the weakly supervised case, the ground truth is
instead only given on the image-level, and predictions on all possible human-
object pairs are aggregated together to the image-level for loss computation;
at the backpropagation time, the gradient is distributed among all the human-
object pairs. As a result, the gradients for full and weak supervision are
computed on different error surfaces and are not compatible. Using one
momentum to record both will make the network weight optimization toggles
between the two sources of gradients across mini-batches. This indeed leads to
an adversarial effect of the mixed-supervised learning (see the ablation study
in Sec. 4.2).

To mitigate this, we propose to bootstrap the mixed-supervised learning with
two independent momentum $z_{t}^{ws}$ and $z_{t}^{fs}$ to record the gradient
history of weak and full supervision separately. $z_{t}^{ws}$ will be used and
updated only with weakly-labeled samples in the mini-batch, while $z_{t}^{fs}$
will be instead used for the fully-labeled samples. $w_{t}$ however is
remained to be shared in the network such that the weakly- and fully-
supervised pipeline are jointly optimized:

$\begin{split}\vspace{-5mm}w_{t}=w_{t-1}-z_{t}^{ws};\;\;\;\;z_{t}^{ws}=\beta
z_{{t-1}}^{ws}+\alpha\nabla f(w_{t-1})\\\
w_{t}=w_{t-1}-z_{t}^{fs};\;\;\;\;z_{t}^{fs}=\beta z_{t-1}^{fs}+\alpha\nabla
f(w_{t-1})\end{split}$ (2)

###  3.4 HOI element swapping

Figure 3: Illustration of HOI element swapping. Top: Object classes from the
two images are different. Bottom: same object class in the two images. Swapped
pairs ($h_{i}$, $o_{j}$) are negatives due to wrong object class (top) or
wrong spatial layout (bottom).

HOI detection is a fine-grained task. To accurately classify similar
interactions, diverse and hard negatives are needed. In the fully-supervised
case, where region-level ground truth are available, this can be achieved via
choosing the false positive class of large confidence score or false positive
region of large intersection-over-union (IoU) with ground truth. While in the
weakly-supervised case, where only image-level ground truths are available,
conventional manners of finding negatives no longer apply. We instead
introduce an HOI element swapping way to collect diverse and hard negatives
across images.

Suppose that two images $im_{1}$ and $im_{2}$ contain one human $h$ and one
object $o$ inside each, respectively. The standard way to create the candidate
human-object pairs is to group ($h_{1}$, $o_{1}$) and ($h_{2}$, $o_{2}$)
within each image (see Fig. 3). To further augment negatives, a simple way is
to lower the threshold from the RPN to produce more proposals; but this is
inefficient as many proposals with low confidence scores are not good
detection of humans or objects, and we do not have ground truth bounding boxes
to distinguish them in the weakly-supervised setting. Hence, we propose to
keep a fair detection of humans and objects within each image and augment the
human-object pairs across images by swapping their HOI elements: given
($h_{1}$, $o_{1}$) from $im_{1}$ and ($h_{2}$, $o_{2}$) from $im_{2}$, we mix
the human proposal from the $im_{1}$ with the object proposal from $im_{2}$
and vice versa: ($h_{1}$, $o_{2}$) and ($h_{2}$, $o_{1}$); this gives us two
more mixed human-object pairs. One image may contain more than one human or
object. Considering there are $H_{1}$ humans and $O_{1}$ objects in $im_{1}$,
$H_{2}$ humans and $O_{2}$ objects in $im_{2}$, selecting all humans and
objects from the two images will produce $(H_{1}+H_{2})\times(O_{1}+O_{2})$
pairs in total for the two images. which is far too much. In practice, we
remove those easy negatives with low confidence scores such that the number of
human-object pairs is kept the same to that of the original number in two
images, i.e. $H_{1}\times O_{1}+H_{2}\times O2$.

By doing this, we can obtain more diverse combinations of HOI pairs, where
many swapped HOI pairs coming from two images might look like positive HOI
pairs playing the role of hard negatives. For instance, in Fig. 3: bottom, the
object class from two images is the same in particular, yet the swapped human-
object pairs ($h_{1}$, $o_{2}$) and ($h_{2}$, $o_{1}$) are still negatives due
to the wrong spatial layout.

The augmented human-object pairs as shown in Fig. 3 can be hard negatives for
both images. To efficiently optimize the learning on two images, we propose to
aggregate all the human-object pairs from two images to form one image-level
HOI label vector, where the corresponding ground truth is the HOI labels from
both images. Apart from efficiency, another benefit of doing this, compared to
optimizing the image separately, is that the positive human-object pairs from
one image could also serve as negatives for the other image if they are of
different HOI labels or as positives if they are of the same HOI label.

###  3.5 Loss function

Loss function is defined within each mini-batch depending on the supervision
$\mathbb{S}$ of the input samples, which can be full supervision
$(\mathbb{S=FS})$ or weak supervision $(\mathbb{S=WS})$:

$\begin{split}{\mathcal{L}_{\text{mini-
batch}}}&=\sum_{j=1}^{C}\bigg{(}\mathbf{1}(\mathbb{S=FS})\frac{1}{N}\sum_{i=1}^{N}BCE(y_{ij},p_{ij})\\\
&+\mathbf{1}(\mathbb{S=WS})BCE(y_{j},p_{j})\bigg{)}\\\ \end{split}$ (3)

BCE is the binary cross-entropy; $p_{ij}$ is the probability of
$j^{\text{th}}$ HOI class for the $i^{th}$ human-object pair, where there are
$N$ pairs and $C$ classes in total; $p_{j}$ is the probability of the
$j^{\text{th}}$ HOI class for the given images in the mini-batch. The former
is defined for the fully-labeled data with region-level ground truth, while
the latter is defined for the weakly-labeled data with image-level ground
truth only. In practice, we feed the features of human-object pairs from two
images in each mini-batch. Referring to Sec. 3.2, the ground truth for fully-
labeled data is given in a form of a matrix, where each element $y_{ij}$
indicates whether the $i^{th}$ human-object pair with $j^{\text{th}}$ HOI
class is true or false. $y_{ij}=1$ if the human and object boxes in the
$i^{th}$ pair have an IoU greater than 0.5 with a ground truth box-pair of the
$j^{\text{th}}$ HOI class. The ground truth for the weakly-labeled data is
given in a form of $C$-dimensional vector where its element $y_{j}=1$ if the
$j^{\text{th}}$ HOI class occurs in any of the two images.

##  4 Experiments

###  4.1 Experimental setup

Datasets. Ever since its introduction in [4], the HICO-DET has become the
defacto standard dataset for human object interaction detection. The dataset
has a total of 47,776 images: 38,118 (80%) are used for training and 9,658
(20%) for testing. Each image is provided with the $\langle
human,object,predicate\rangle$ triplets which include human and object
bounding boxes and HOI classes. It covers 80 object categories and 117
interactions, which result into 600 HOI classes in total. These classes are
subdivided into 138 rare ones, whose training samples are less than $10$
images; $462$ non-rare ones, whose training samples are more than $10$ images.
On average, 1.67 HOI triplets are annotated in each image. HICO-DET is a much
bigger dataset compared to the previous V-COCO dataset [18]. In line with
recent works on HOI detection [19, 2], we evaluate our method on the large-
scale HICO-DET to offer comprehensive study and in-depth analysis on it.

Data splitting.The training images are randomly split with different ratios of
weakly- and fully-labeled data (denoted as WS and FS). The default WS/FS ratio
is set to 70/30 and 30/70 where 70% (30%) data from the training set are
weakly-labeled and the rest are fully-labeled. We also evaluate different
WS/FS ratios ranging between 100/0 and 0/100 in the experiments. Some more
settings regarding unlabeled data and class-split are also presented in the
end.

Implementation details and evaluation protocol. Following [4], the human and
object detection results are taken from the top scoring output of a Faster-
RCNN pretrained on MS-COCO [28]. Each human is paired with all the objects
within an image. Faster-RCNN produces numerous candidate bounding boxes. For
each object, we filter the 30-top performing boxes depending on the detection
scores. For fully-labeled data, ground truth HOI triplets are provided with
human/object bounding boxes and their interaction. For weakly-labeled data,
only image-level HOI labels are provided meaning that the real correspondence
from a human detection to an object is not given in the image. For
unsupervised data, no HOI labels are provided. The network is trained with a
mini-batch containing the set of region proposal pairs in two images, which
are randomly selected from either the fully-labelled set or the weakly-labeled
set. This is done once before the training for efficiency. Two images from the
weakly-labeled set are applied with element swapping. The learning rate is
1e-3 and 1e-4 for weakly- and fully-labeled data, respectively. We train
40,000 iterations in total.

Evaluation of HOI detection employs the widely used mean average precision
(mAP) metric where a prediction is considered correct only if its HOI class
label is correct, and its human and object bounding boxes have an Intersection
over Union (IoU) larger than $0.5$ with their respective ground truth bounding
boxes.

###  4.2 Ablation study

In this section, we first justfiy the importance of our proposed new elements
MIL and HES in order to enable a meaningful mix-supervised HOI detection.
Next, we give the result of MX-HOI generalizing over different WS/FS ratios
from 0/100 to 100/0.

Using weak and strong annotations. To start with our ablation study, we first
train our HOI detector with weak annotations only; next, we train the detector
with both weak and strong annotations. We illustrate the mAP on the test set
in Fig. 4: by default 70% (30%) data are chosen as weakly-labeled and the rest
are as fully-labeled. The results show that without using our proposed MIL
(w/o MIL), adding FS data can perform even worse than using WS data only: for
example using 70% of the training data, all weakly-labeled (WS/FS = 70/0), it
(_weak only_ , grey) yields a mAP of 14.68; when adding fully-labeled data
(WS/FS = 70/30), the detector performs an even worse mAP 13.17 (red). This
illustrates well the adversarial effect between the weakly- and fully-labeled
data. In order to tackle the problem, we first tried a sequence training (ST)
strategy [40, 30], where all fully-labeled data (resp. all weakly-labeled
data) are presented in the network with some epochs before the weakly-labeled
data (resp. fully-labeled data) are added in. We denote the strategy as w/
ST-F when the fully-labeled data are trained first, or w/ ST-W when the
weakly-labeled data are trained first. The results are improved in this manner
but not too much (see Fig 4). Next, we introduce our momentum-independent
learning strategy to specifically enable the mixed-supervised HOI detection.

Momentum-independent learning (MIL). To tackle the inconsistency of gradients
in the network backpropagation, we introduce two independent momentum to
record the gradient history of full and weak supervision separately as they
are computed on different error surfaces and are functionally different in the
network. This is conceptualized as momentum-independent learning and is a key
element of our MX-HOI pipeline. Having a look at Fig. 4, ours (w/ MIL)
significantly increases the mAP to e.g. 16.63 for WS/FS = 70/30 and 17.47 for
WS/FS = 30/70. This demonstrates the importance of our proposed MIL to enable
a meaningful mixed-supervised HOI detection. Some more detailed comparisons
between our MIL and other variants on the rare and non-rare classes are shown
in Table 1.

Figure 4: Using weak, strong and mixed annotations for HOI detection. MIL: momentum-independent learning. ST-W: sequence training with weakly-labeled data first; ST-F: sequence training with fully-labeled data first;  Method | WS/FS | Rare | Non-Rare | WS/FS | Rare | Non-Rare
---|---|---|---|---|---|---
weak only | 70/0 | 11.84 | 15.72 | 30/0 | 8.68 | 14.11
w/o MIL | 70/30 | 8.88 | 14.45 | 30/70 | 9.17 | 15.74
w/ ST-F | 70/30 | 10.41 | 16.69 | 30/70 | 10.52 | 17.00
w/ ST-W | 70/30 | 10.17 | 16.71 | 30/70 | 11.03 | 16.43
with MIL (ours) | 70/30 | 12.36 | 17.91 | 30/70 | 12.79 | 18.80
Table 1: Ablation of momentum-independent learning (MIL) in MX-HOI on HICO-DET
dataset. mAP is reported.  Figure 5: Examples of correct detections (top) and
incorrect detections (bottom) with MX-HOI. The classes shown here are Carry-
Sheep, Set-Umbrella, Repair-Parking-meter, Make-Vase and Hold-Hairdryer.

HOI element swapping (HES). Referring to Sec. 3.4, HOI element swapping is
introduced for hard negative harvest on weakly-labeled data. To verify its
effectiveness, we ablate it in Table 2 by comparing with MX-HOI without HES.
We vary WS/FS from 100/0 to 30/70 and show that on different mixed levels, HES
always helps the HOI detection. For instance, when WS/FS = 70/30, MX-HOI
yields +0.81% improvement over MX-HOI (w/o HES).

Additionally, we also apply HES on fully-labeled data, i.e. WS/FS = 0/100, and
obtain the mAP 17.04, 13.35, 18.11 on full, rare and non-rare classes,
respectively, which actually harms the performance on non-rare classes while
helps a bit on rare classes comparing to 17.82, 12.91, and 19.17 in Table 3.
Rare classes do not have adequate training samples, HES can help provide hard
negatives; while for non-rare classes, hard negatives can be directly mined
via the given bounding box ground truth. Overall, we did not find HES to be
effective for fully-labeled data in general.

Method | WS/FS | Full | Rare | Non-Rare
---|---|---|---|---
ours (w/o HES) | 100/0 | 15.14 | 10.65 | 16.48
ours | 100/0 | 16.14 | 12.06 | 17.50
ours (w/o HES) | 70/30 | 15.82 | 10.39 | 17.41
ours | 70/30 | 16.63 | 12.36 | 17.91
ours (w/o HES) | 30/70 | 16.73 | 12.00 | 18.14
ours | 30/70 | 17.47 | 12.79 | 18.80
Table 2: Ablation of HOI element swapping (HES) in MX-HOI on HICO-DET dataset.
mAP is reported.

WS/FS variations. We offer the results of MX-HOI with different WS/FS: 100/0,
80/20, 70/30, 50/50, 30/70, 20/80 and 0/100 in Table 3. One can see that the
performance increases with an increase of FS for both rare and non-rare
classes, as more fully-labeled data are added into the training. The overall
full mAP increases from 16.14 to 17.82. In Table 3, we also show the result of
fixing either WS, or FS to 30% while varying the other: the performance
increases along with the training (sub-)set size; but the improvement margin
is bigger when fixing WS and increasing FS compared to fixing FS and
increasing WS. All these results make perfect sense for MX-HOI: adding more
labeled data, regardless WS or FS, increases its performance; FS data in
general provides more help than WS data. Examples of MX-HOI with WS/FS=70/30
are given in Fig. 5.

Supervision (WS/FS) | Full | Rare | Non-Rare
---|---|---|---
100/0 | 16.14 | 12.06 | 17.50
80/20 | 16.49 | 12.28 | 17.81
70/30 | 16.63 | 12.36 | 17.91
50/50 | 17.08 | 12.58 | 18.17
30/70 | 17.47 | 12.79 | 18.80
20/80 | 17.60 | 12.85 | 18.95
0/100 | 17.82 | 12.91 | 19.17
30/30 | 16.05 | 11.64 | 17.37
30/50 | 16.84 | 11.81 | 18.47
30/70 | 17.47 | 12.79 | 18.80
50/30 | 16.34 | 12.04 | 17.69
70/30 | 16.63 | 12.36 | 17.91
Table 3: Different ratios of WS/FS in MX-HOI on HICO-DET dataset (top). Fixing
WS (resp. FS) ratio and varying the other (bottom). mAP is reported.

###  4.3 Comparison to state of the art

In Table 4, we first compare our method to its lower and upper bounds denoted
respectively by WS-No-Frills and No-Frills in the following. WS-No-Frills is
an adaption of a representative weakly-supervised relationship detection
module [54] onto the SOTA HOI detection pipeline [19], which produces mAP
15.14, 10.65, and 16.48 on full, rare and non-rare classes. Our MX-HOI under
the same setting WS/FS=100/0 improves the result to 16.14, 12.06, 17.50 due to
the adoption of HOI element swapping. In fully-supervised setting (WS/FS =
0/100), our MX-HOI also improves the No-Frills [19] by +0.6%, this is
attributed to our two-branch softmax [3] in No-Frills (This softmax is applied
in subbranches before the final classification head). Despite MX-HOI is
introduced for mix-supervised HOI detection, it also improves the SOTA bounds
as side benefits.

In the mixed-supervised setting, MX-HOI retains 93.3 % accuracy of the SOTA
No-Frills by using a mixture of 30% fully-labeled data and 70% weakly-labeled
data. To compare with it, we implement a naive _multi-stage training pipeline_
: it first trains the model on 30% fully-labeled data, then infers the HOI
class probabilities on the human-object pairs in the rest 70% weakly-labeled
images; the HOI triplet with the largest probability is selected as pseudo
ground truth for each given HOI label on the image-level in weakly-labeled
data. These selected HOI triplets are mixed with existing fully-labeled data
to train the network again. The network remains a fully-supervised pipeline in
this manner. This process would repeat several rounds until the convergence of
the model. We obtain mAP 15.23, 10.63, and 16.61 under the setting of WS/FS =
70/30, which is much lower than our MX-HOI (16.63, 12.36 and 17.91).

We also compare MX-HOI with other recent arts [48, 13, 36, 45, 46, 26] using
100% supervision. One can see that with WS/FS = 70/30, MX-HOI performs very
close to the SOTA.

Methods | WS/FS | Full | Rare | Non-rare
---|---|---|---|---
WS-No-Frills | 100/0 | 15.14 | 10.65 | 16.48
MX-HOI | 16.14 | 12.06 | 17.50
MX-HOI | 70/30 | 16.63 | 12.36 | 17.91
MX-HOI | 30/70 | 17.47 | 12.79 | 18.80
MX-HOI | 0/100 | 17.82 | 12.91 | 19.17
No-Frills [19] | 17.18 | 12.17 | 18.68
VSGNet [45] | 19.80 | 16.05 | 20.91
PMFNet [46] | 17.46 | 15.65 | 18.00
TIN [26] | 17.22 | 13.51 | 18.32
GCN-HOI [48] | 14.70 | 13.26 | 15.13
GPNN [36] |  | 13.11 | 9.34 | 14.23
ICAN [13] |  | 12.80 | 8.53 | 14.07
Table 4: Comparison with the state-of-the-art methods on HICO-DET test set
(mAP).

###  4.4 More settings

Unlabeled data can also be added into the whole framework: we first obtain all
possible human-object pairs in an unlabeled image (US) from the detection
result. Given the trained model of the mix-supervised pipeline, we can
estimate the marginal HOI class probability for every human-object pair in the
unlabeled image. If the probability is larger than a threshold (e.g. 0.5), we
take the predicted HOI class as the pseudo ground truth for this human-object
pair and add it into the network training in the next cycle. The loss function
in (3) now includes another term for the unlabeled data, which is formulated
similarly to the fully-labeled data with pseudo ground truth. The loss weights
among the three terms remain 1. This process iterates for several cycles until
the convergence of the network.

Table 5 shows the result of WS/FS/US being 30/40/30, where 30%, 40% and 30%
percent data are respectively weakly-, fully- and un-supervised (see Sec.
4.1). Results using unlabeled data improves performance when compared to using
only WS/FS with 30/40 ratio.

| WS/FS/US | Full | Rare | Non-Rare
---|---|---|---|---
MX-HOI | 30/40/0 | 16.08 | 12.05 | 17.29
MX-HOI | 30/40/30 | 16.53 | 11.63 | 17.79
Table 5: Adding unlabeled data (US) into MX-HOI.

Class-split: Instead of randomly splitting the dataset images for weak and
full supervision, we can randomly split the whole HOI classes into 50% vs.
50%. Images from the first 50% classes are trained with full supervision, the
second 50% with weak supervision. If we train two models separately on the two
sets, we got mAP 13.3 and 11.6 on the test set of each own part of classes,
respectively. If we train one model over the two sets jointly using MX-HOI,
the mAP increases to 14.8 and 13.10. Despite the two sets are of different
classes, training them together with more data benefit the performance of both
in our pipeline.

##  5 Conclusion

We present a mixed-supervised HOI detection framework (MX-HOI) which employs
two state-of-the-art fully- and weakly-supervised pipelines. Within this
framework, we first introduce a momentum-independent strategy to tackle the
adversarial effect of full and weak supervision by separating their gradient
history in momentum learning. Second, we introduce an HOI element swapping
strategy to harvest hard negatives across images for weakly-labeled data.
Unlabeled data can also be leveraged using a “pseudo label” solution where
class labels on HOI pairs are provided by the trained mixed-supervised
pipeline. Extensive experiments on the large-scale HICO-DET dataset show that,
with only 30% fully-labeled data and 70% weakly-labeled data, our MX-HOI is
able to retain 93.3% accuracy of the setting of 100% fully-labeled data.
Future work will be focused on developing a stronger weakly-supervised HOI
detection pipeline to integrate it into our MX-HOI framework.

Acknowledgements: This work was partially supported by the READ-IT project,
funded by the JPI Cultural Heritage under the European Union Horizon 2020 R&I
program (grant agreement No. 699523). Miaojing Shi was supported by the
National Natural Science Foundation of China (NSFC) under Grant No. 61828602.

## References

  * [1] Yuval Atzmon, Jonathan Berant, Vahid Kezami, Amir Globerson, and Gal Chechik.  Learning to generalize to new compositions in image understanding.  arXiv preprint arXiv:1608.07639, 2016.
  * [2] Ankan Bansal, Sai Saketh Rambhatla, Abhinav Shrivastava, and Rama Chellappa.  Detecting human-object interactions via functional generalization.  In AAAI, 2020.
  * [3] Hakan Bilen and Andrea Vedaldi.  Weakly supervised deep detection networks.  In CVPR, 2016.
  * [4] Yu-Wei Chao, Yunfan Liu, Xieyang Liu, Huayi Zeng, and Jia Deng.  Learning to detect human-object interactions.  In WACV, 2018.
  * [5] Yu-Wei Chao, Zhan Wang, Yugeng He, Jiaxuan Wang, and Jia Deng.  Hico: A benchmark for recognizing human-object interactions in images.  In ICCV, 2015.
  * [6] Ramazan Gokberk Cinbis, Jakob Verbeek, and Cordelia Schmid.  Weakly supervised object localization with multi-fold multiple instance learning.  IEEE Transactions on Pattern Analysis and Machine Intelligence, 39(1):189–203, 2016.
  * [7] Zhen Cui, Chunyan Xu, Wenming Zheng, and Jian Yang.  Context-dependent diffusion network for visual relationship detection.  In ACM MM, 2018.
  * [8] Vincent Delaitre, Josef Sivic, and Ivan Laptev.  Learning person-object interactions for action recognition in still images.  In NeurIPS, 2011.
  * [9] Chaitanya Desai and Deva Ramanan.  Detecting actions, poses, and objects with relational phraselets.  In ECCV, 2012.
  * [10] Chaitanya Desai, Deva Ramanan, and Charless Fowlkes.  Discriminative models for static human-object interactions.  In CVPR Workshops, 2010.
  * [11] Thomas Deselaers, Alexe Bogdan, and Vittorio Ferrari.  Weakly supervised localization and learning with generic knowledge.  International Journal of Computer Vision, 100:275–293, 2012.
  * [12] Chen Gao, Jiarui Xu, Yuliang Zou, and Jia-Bin Huang.  Drg: Dual relation graph for human-object interaction detection.  In ECCV, 2020.
  * [13] Chen Gao, Yuliang Zou, and Jia-Bin Huang.  ican: Instance-centric attention network for human-object interaction detection.  In BMVC, 2018.
  * [14] Georgia Gkioxari, Ross Girshick, Piotr Dollár, and Kaiming He.  Detecting and recognizing human-object interactions.  In CVPR, 2018.
  * [15] Stephen Gould, Jim Rodgers, David Cohen, Gal Elidan, and Daphne Koller.  Multi-class segmentation with relative location prior.  International Journal of Computer Vision, 80(3):300–316, 2008.
  * [16] Abhinav Gupta and Larry S Davis.  Beyond nouns: Exploiting prepositions and comparative adjectives for learning visual classifiers.  In ECCV, 2008.
  * [17] Abhinav Gupta, Aniruddha Kembhavi, and Larry S Davis.  Observing human-object interactions: Using spatial and functional compatibility for recognition.  IEEE Transactions on Pattern Analysis and Machine Intelligence, 31(10):1775–1789, 2009.
  * [18] Saurabh Gupta and Jitendra Malik.  Visual semantic role labeling.  arXiv preprint arXiv:1505.04474, 2015.
  * [19] Tanmay Gupta, Alexander Schwing, and Derek Hoiem.  No-frills human-object interaction detection: Factorization, layout encodings, and training techniques.  In ICCV, 2019.
  * [20] Chaojun Han, Fumin Shen, Li Liu, Yang Yang, and Heng Tao Shen.  Visual spatial attention network for relationship detection.  In ACM MM, 2018.
  * [21] Ronghang Hu, Anna Rohrbach, Trevor Darrell, and Kate Saenko.  Language-conditioned graph networks for relational reasoning.  In ICCV, 2019.
  * [22] M Pawan Kumar and Daphne Koller.  Efficiently selecting regions for scene understanding.  In CVPR, 2010.
  * [23] Dong-Hyun Lee.  Pseudo-label: The simple and efficient semi-supervised learning method for deep neural networks.  In ICML Workshop, 2013.
  * [24] Yikang Li, Wanli Ouyang, Xiaogang Wang, and Xiao’ou Tang.  Vip-cnn: Visual phrase guided convolutional neural network.  In CVPR, 2017.
  * [25] Yong-Lu Li, Liang Xu, Xinpeng Liu, Xijie Huang, Yue Xu, Shiyi Wang, Hao-Shu Fang, Ze Ma, Mingyang Chen, and Cewu Lu.  Pastanet: Toward human activity knowledge engine.  In CVPR, 2020.
  * [26] Yong-Lu Li, Siyuan Zhou, Xijie Huang, Liang Xu, Ze Ma, Hao-Shu Fang, Yanfeng Wang, and Cewu Lu.  Transferable interactiveness knowledge for human-object interaction detection.  In CVPR, 2019.
  * [27] Xiaodan Liang, Lisa Lee, and Eric P Xing.  Deep variation-structured reinforcement learning for visual relationship and attribute detection.  In CVPR, 2017.
  * [28] Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick.  Microsoft coco: Common objects in context.  In ECCV, 2014.
  * [29] Y Liu, Q Chen, and A Zisserman.  Amplifying key cues for human-object-interaction detection.  ECCV, 2020.
  * [30] Yuting Liu, Miaojing Shi, Qijun Zhao, and Xiaofang Wang.  Point in, box out: Beyond counting persons in crowds.  In CVPR, 2019.
  * [31] Cewu Lu, Ranjay Krishna, Michael Bernstein, and Li Fei-Fei.  Visual relationship detection with language priors.  In ECCV, pages 852–869, 2016.
  * [32] George Papandreou, Liang-Chieh Chen, Kevin P Murphy, and Alan L Yuille.  Weakly-and semi-supervised learning of a deep convolutional network for semantic image segmentation.  In ICCV, 2015.
  * [33] Julia Peyre, Ivan Laptev, Cordelia Schmid, and Josef Sivic.  Detecting unseen visual relations using analogies.  In ICCV, 2019.
  * [34] Julia Peyre, Josef Sivic, Ivan Laptev, and Cordelia Schmid.  Weakly-supervised learning of visual relations.  In ICCV, 2017.
  * [35] Alessandro Prest, Cordelia Schmid, and Vittorio Ferrari.  Weakly supervised learning of interactions between humans and objects.  IEEE Transactions on Pattern Analysis and Machine Intelligence, 34(3):601–614, 2011.
  * [36] Siyuan Qi, Wenguan Wang, Baoxiong Jia, Jianbing Shen, and Song-Chun Zhu.  Learning human-object interactions by graph parsing neural networks.  In ECCV, 2018.
  * [37] Vignesh Ramanathan, Congcong Li, Jia Deng, Wei Han, Zhen Li, Kunlong Gu, Yang Song, Samy Bengio, Charles Rosenberg, and Li Fei-Fei.  Learning semantic relationships for better action retrieval in images.  In CVPR, 2015.
  * [38] Mohammad Amin Sadeghi and Ali Farhadi.  Recognition using visual phrases.  In CVPR, 2011.
  * [39] Miaojing Shi, Holger Caesar, and Vittorio Ferrari.  Weakly supervised object localization using things and stuff transfer.  In ICCV, 2017.
  * [40] Miaojing Shi and Vittorio Ferrari.  Weakly supervised object localization using size estimates.  In ECCV, 2016.
  * [41] Zhiyuan Shi, Parthipan Siva Siva, and Tao Xiang.  Transfer learning by ranking for weakly supervised object annotation.  In BMVC, 2015.
  * [42] Xu Sun, Yuan Zi, Tongwei Ren, Jinhui Tang, and Gangshan Wu.  Hierarchical visual relationship detection.  In ACM MM, 2019.
  * [43] Peng Tang, Xinggang Wang, Song Bai, Wei Shen, Xiang Bai, Wenyu Liu, and Alan Yuille.  Pcl: Proposal cluster learning for weakly supervised object detection.  IEEE Transactions on Pattern Analysis and Machine Intelligence, 42(1):176–191, 2018.
  * [44] Peng Tang, Xinggang Wang, Xiang Bai, and Wenyu Liu.  Multiple instance detection network with online instance classifier refinement.  In CVPR, 2017.
  * [45] Oytun Ulutan, ASM Iftekhar, and Bangalore S Manjunath.  Vsgnet: Spatial attention network for detecting human object interactions using graph convolutions.  In CVPR, 2020.
  * [46] Bo Wan, Desen Zhou, Yongfei Liu, Rongjie Li, and Xuming He.  Pose-aware multi-level feature network for human object interaction detection.  In ICCV, 2019.
  * [47] Tiancai Wang, Tong Yang, Martin Danelljan, Fahad Shahbaz Khan, Xiangyu Zhang, and Jian Sun.  Learning human-object interaction detection using interaction points.  In CVPR, 2020.
  * [48] Bingjie Xu, Yongkang Wong, Junnan Li, Qi Zhao, and Mohan S Kankanhalli.  Learning to detect human-object interactions with knowledge.  In CVPR, 2019.
  * [49] Wanru Xu, Jian Yu, Zhenjiang Miao, Lili Wan, and Qiang Ji.  Prediction-cgan: Human action prediction with conditional generative adversarial networks.  In ACM MM, 2019.
  * [50] Yukuan Yang, Fangyu Wei, Miaojing Shi, and Guoqi Li.  Restoring negative information in few-shot object detection.  In NeurIPS, 2020.
  * [51] Zhaohui Yang, Miaojing Shi, Yannis Avrithis, Chao Xu, and Vittorio Ferrari.  Training object detectors from few weakly-labeled and many unlabeled images.  arXiv preprint arXiv:1912.00384, 2019.
  * [52] Mark Yatskar, Luke Zettlemoyer, and Ali Farhadi.  Situation recognition: Visual semantic role labeling for image understanding.  In CVPR, 2016.
  * [53] Ruichi Yu, Ang Li, Vlad I Morariu, and Larry S Davis.  Visual relationship detection with internal and external linguistic knowledge distillation.  In ICCV, 2017.
  * [54] Hanwang Zhang, Zawlin Kyaw, Jinyang Yu, and Shih-Fu Chang.  PPR-FCN:: Weakly supervised visual relation detection via parallel pairwise R-FCN.  In ICCV, 2017.
  * [55] Sipeng Zheng, Shizhe Chen, and Qin Jin.  Visual relation detection with multi-level attention.  In ACM MM, 2019.
  * [56] Hao Zhou, Chongyang Zhang, and Chuanping Hu.  Visual relationship detection with relative location mining.  In ACM MM, 2019.
  * [57] Penghao Zhou and Mingmin Chi.  Relation parsing neural network for human-object interaction detection.  In ICCV, 2019.


# Cross-view Transformers for real-time Map-view Semantic Segmentation

Brady Zhou
UT Austin
<EMAIL_ADDRESS>Philipp Krähenbühl
UT Austin
<EMAIL_ADDRESS>
###### Abstract

We present cross-view transformers, an efficient attention-based model for
map-view semantic segmentation from multiple cameras. Our architecture
implicitly learns a mapping from individual camera views into a canonical map-
view representation using a camera-aware cross-view attention mechanism. Each
camera uses positional embeddings that depend on its intrinsic and extrinsic
calibration. These embeddings allow a transformer to learn the mapping across
different views without ever explicitly modeling it geometrically. The
architecture consists of a convolutional image encoder for each view and
cross-view transformer layers to infer a map-view semantic segmentation. Our
model is simple, easily parallelizable, and runs in real-time. The presented
architecture performs at state-of-the-art on the nuScenes dataset, with 4x
faster inference speeds. Code is available at
https://github.com/bradyz/cross_view_transformers.

##  1 Introduction

Autonomous vehicles depend on robust scene understanding and online mapping to
navigate the world. To drive safely, these systems not only reason about the
semantics of their surroundings, but also a spatial understanding due to the
geometric nature of navigation. Many prior approaches directly model geometry
and relationships between different view and a canonical map representation
[20, 15, 35, 2, 16, 29, 14]. They require an explicit [20, 15, 35, 2, 16] or
probabilistic [29, 14] estimate of depth in either image or map-view. However,
this explicit modeling can be hard. First, image-based depth estimates are
error-prone, as monocular depth estimates scale poorly with the distance to
the observer. Second, depth-based projections are a fairly inflexible and
rigid bottleneck to map between views. In this work, we take a different
approach.

We learn to map from camera-view to a canonical map-view representation using
a cross-view transformer architecture. The transformer does not perform any
explicit geometric reasoning but instead learns to map between views through a
geometry-aware positional embedding. Multi-head attention then learns to map
features from camera-view into a canonical map-view representation using a
learned map-view positional embedding. We learn a single map-view positional
embedding for all cameras and perform attention across all views. The model
thus learns to link up different map locations to both cameras and locations
within each camera. Our cross-view transformer refines the map-view embedding
through multiple attention and MLP blocks. The cross-view transformer allows
the network to learn any geometric transformation implicitly and directly from
data. It learns an implicit estimate of depth through the camera-dependent
map-view positional embedding by performing the downstream task as accurately
as possible.

Figure 1: We introduce an architecture for perception in a map-view frame from
multiple views. Our model builds a map-view representation by cross-attending
to image features. A camera-aware positional embedding can geometrically link
up the camera and map-views.

The simplicity of our model is a key strength. The model performs at state-of-
the-art on the nuScenes [3] dataset for vehicle and road segmentation in the
map-view and comfortably runs in real-time (35 FPS) on a single RTX 2080 Ti
GPU. The model is easy to implement and trains within 32 GPU hours. The
learned attention mechanism learns accurate correspondences between camera and
map-view directly from data.

##  2 Related Works

Map-view semantic segmentation lies at the intersection of 3D recognition,
depth estimation, and mapping.

#### Monocular 3D object detection.

Monocular detection aims to find objects in a scene, estimate their real-world
size, orientation, and placement in the 3D scene. Most common approaches
reduce the problem to 2D object detection and infer monocular depth [48, 25].
CenterNet [48] directly predicts depth for each image coordinate. ROI-10D [25]
lifts 2D detections into 3D using depth estimates then regresses 3D bounding
boxes. Psuedo-lidar based approaches [45, 24, 46, 44] project to 3D points
using a depth estimate and leverage 3D point based architectures (e.g. [30,
18, 43]) with 2D labels. This family of algorithms directly benefits from
advances in monocular depth estimation and 3D vision.

Monocular 3D object detection is both easier and harder than mapping from
multiple cameras. The overall problem setup deals with just a single camera
and does not need to merge multiple sources of inputs. However, it strongly
relies on a good explicit monocular depth estimate, which may be harder to
obtain.

#### Depth estimation.

Depth is a core ingredient in many multi-view mapping approaches. Classic
structure-from-motion approaches [20, 1, 38, 7, 35] leverage epipolar geometry
and triangulation to explicitly compute camera extrinsics and depth. Stereo
matching finds corresponding pixels, from which depth can be explicitly
computed [15]. Recent deep learning approaches directly regress depth from
images [6, 11, 12, 47, 31, 8].

While convenient, explicit depth is challenging to utilize for downstream
tasks. It is camera-dependent and requires an accurate calibration and fusion
of multiple noisy estimates. Our approach side-steps explicit depth estimation
and instead allows an attention mechanism with positional embedding to take
its place. Our cross-view transformer learns to reproject camera views into a
common map representation as part of training.

#### Semantic mapping in the map-view.

Driven by ever larger 3D recognition datasets [10, 5, 3, 39, 13], a number of
works have focused on perception in the map-view. This problem is particularly
challenging as the inputs and outputs lie in different coordinate frames.
Inputs are recorded in calibrated camera views, outputs are rasterized onto a
map. Most prior works differ in the way the transformation is modeled. One
common technique is to assume the scene is mostly planar and represent the
image to map-view transformation as a simple homography [9, 22, 16, 49, 36,
2]. A second family of methods directly produces map-view predictions from
input images, with no explicit geometric modeling. VED [23] uses a Variational
Auto Encoder [17] to produce a semantic occupancy grid from a single monocular
camera-view.

Closely related in spirit to our method, VPN [28] learns a common feature
representation across multiple views with their proposed view relation module
\- an MLP that outputs map-view features from inputs across all views. Both
VED and VPN show carefully-designed networks trained with sufficient training
data can jointly learn the map-view transformation and perform prediction.
However, these methods do suffer certain drawbacks as they do not model the
geometric structure of the scene. They forgo the inherit inductive biases
contained in a calibrated camera setup and instead need to learn an implicit
model of camera calibration baked into the network weights. Our cross-view
transformer instead uses positional embeddings derived from calibrated camera
intrinsics and extrinsics. The transformer can learn a camera-calibration-
dependent mapping akin to raw geometric transformations.

Most recently, top-performing methods returned back to explicit geometric
reasoning [33, 29, 32, 34, 27, 14]. Orthographic Feature Transform (OFT) [33]
creates a map-view intermediate representation from a monocular image by
average pooling image features from the 2D projection that corresponds with
the pillar in map-view. This pooling operation foregoes an explicit depth
estimate and instead averages all possible image locations a map-view object
could take. Lift-Splat-Shoot (LSS) [29] constructs an intermediate map-view
representation in a similar fashion. However, they allow the model to learn a
soft depth estimate and average across different bins using a learned depth-
estimate-dependent weight. Their downstream decoder can account for
uncertainty in depth. This weighted averaging operation closely mimics the
attention used in a transformer. However, their “attention weights” are
derived from geometric principles and not learned from data. The original
Lift-Splat-Shoot approach considers multiple views within a single timestep.
Recent methods have extended this further to take aggregate features from
previous timesteps [34], and use multi-view, multi-timestep observations to do
motion forecasting [14].

In this work, we show that implicit geometric reasoning performs as well as
explicit geometric models. The added benefit of our implicit handling of
geometry is an improvement in inference speed compared to explicit models. We
simply learn a set of positional embeddings, and attention will reproject the
camera to map-view.

Figure 2: An overview of our proposed architecture for map-view segmentation.
For each image, we extract image features across multiple scales. Using known
camera pose and intrinsics, we construct a camera-aware positional embedding.
We learn a map-view positional embedding that aggregates information from all
views through a series of cross attention layers. Each cross-attention refines
the map-view positional embedding and queries more accurate image locations.

##  3 Cross-view transformers

In this section, we introduce our proposed architecture for semantic
segmentation in the map-view from multiple camera views. In this task, we are
given a set of $n$ monocular views ${(I_{k},K_{k},R_{k},t_{k})}_{k=1}^{n}$
consisting of input image $I_{k}\in\mathbb{R}^{H\times W\times 3}$, camera
intrinsics $K_{k}\in\mathbb{R}^{3\times 3}$, and extrinsic rotation
$R_{k}\in\mathbb{R}^{3\times 3}$ and translation $t_{k}\in\mathbb{R}^{3}$
relative to the center of the ego-vehicle. Our goal is to learn an efficient
model to extract information from the multiple camera views in order to
predict a binary semantic segmentation mask $y\in{\\{0,1\\}}^{h\times w\times
C}$ in the orthographic map-view coordinate frame.

We design a simple, yet effective encoder-decoder architecture for map-view
semantic segmentation. An image-encoder produces a multi-scale feature
representation of each input image. A cross-view cross-attention mechanism
then aggregates multi-scale features into a shared map-view representation.
The cross-view attention relies on a positional embedding that is aware of the
geometric structure of the scene and learns to match up camera-view and map-
view locations. All cameras share the same image-encoder, but use a positional
embedding dependent on their individual camera calibration. Finally, a
lightweight convolutional decoder upsamples the refined map-view embedding and
produces the final segmentation output. The entire network is end-to-end
differentiable and learned jointly. Figure 2 shows an overview of the full
architecture.

In Section 3.1, we first present the core cross-view attention mechanism and
positional embedding that underlies our entire architecture. Section 3.2 then
combines multiple cross-view attention layers into the final map-view
segmentation model.

###  3.1 Cross-view attention

The goal of cross-view attention is to link up a map-view representation with
image-view features. For any world coordinate $x^{(W)}\in\mathbb{R}^{3}$, the
perspective transformation describes its corresponding image coordinate
$x^{(I)}\in\mathbb{R}^{3}$:

$x^{(I)}\simeq K_{k}R_{k}(x^{(W)}-t_{k}).$ (1)

Here, $\simeq$ describes equality up to a scale factor, and
$x^{(I)}=(\cdot,\cdot,1)$ uses homogeneous coordinates. However, without an
accurate depth estimate in camera view or height-above-ground estimate in map-
view, the world coordinate $x^{(W)}$ is ambiguous. We do not learn an explicit
estimate of depth but encode any depth ambiguity in the positional embeddings
and let a transformer learn a proxy for depth.

We start by rephrasing the geometric relationship between world and image
coordinates in Equation 1 as a cosine similarity for use in an attention
mechanism.

$sim_{k}(x^{(I)},x^{(W)})=\frac{\left(R_{k}^{-1}K_{k}^{-1}x^{(I)}\right)\cdot\left(x^{(W)}-t_{k}\right)}{\|R_{k}^{-1}K_{k}^{-1}x^{(I)})\|\|(x^{(W)}-t_{k}\|}.$
(2)

This similarity still relies on the exact world coordinate $w^{(W)}$. Next, we
replace all geometric components of this similarity with positional encodings
that can learn both geometric and appearance features.

#### Camera-aware positional encoding.

The camera-aware positional encoding starts from the unprojected image
coordinate $d_{k,i}=R_{k}^{-1}K_{k}^{-1}x^{(I)}_{i}$ for each image coordinate
$x^{(I)}_{i}$. The unprojected image coordinate $d_{k,i}$ describes a
direction vector from the origin $t_{k}$ of camera $k$ to the image plane at
depth $1$. The direction vector uses world coordinates.

We encode this direction vector $d_{k,i}$ using an MLP (shared across all $k$
views) into a $D$-dimensional positional embedding
$\delta_{k,i}\in\mathbb{R}^{D}$. We use $D=128$ in our experiments. We combine
this positional embedding with image features $\phi_{k,i}$ in the keys of our
cross-view attention mechanism. This allows cross-view attention to use both
appearance and geometric cues to reason about correspondences across the
different views.

Next, we show how to build an equivalent representation for the map-view
queries. This embedding can no longer rely on exact geometric inputs and
instead needs to learn geometric reasoning in consecutive layers of the
transformer.

#### Map-view latent embedding.

The map-view component of the geometric similarity metric in Equation 2
contains a world coordinate $x^{(W)}$ and camera location $t_{k}$. We encode
both in a separate positional embedding. We use an MLP to transform each
camera location $t_{k}$ into an embedding $\tau_{k}\in\mathbb{R}^{D}$. We
build the map-view representation up over multiple iterations in our
transformer. We start with a learned positional encoding
$c^{(0)}\in\mathbb{R}^{w\times h\times D}$. The goal of the map-view
positional encoding is to produce an estimate of the 3D location of each
element of the road. Initially, this estimate is shared across all scenes and
likely learns an average position and height above the ground plane for each
element of the scene. The transformer architecture then refines this estimate
through multiple rounds of computation, resulting in new latent embeddings
$c^{(1)},c^{(2)},\ldots$. Each positional embedding is better able to project
the map-view coordinates into a proxy of the 3D environment. Following the
geometric similarity measure in Equation 2, we use the difference between map-
view embeddings $c$ and camera-location embeddings $\tau_{k}$ as queries in
the transformer.

#### Cross-view attention.

Our cross-view transformer combines both positional encodings through a cross-
view attention mechanism. We allow each map-view coordinate to attend one or
more image locations. Crucially, not every map-view location has a
corresponding image patch in each view. Front-facing cameras do not see the
back, rear-facing cameras do not see the front. We allow the attention
mechanism to select both camera and location within each camera when
corresponding map-view and camera-view perspectives. To this end, we first
combine all camera-aware positional embeddings $\delta_{1},\delta_{2},\ldots$
from all views into a single key vector
$\delta=\left[\delta_{1},\delta_{2},\ldots\right]$. At the same time, we
combine all image features $\phi_{1},\phi_{2},\ldots$ into a single value
vector $\phi=\left[\phi_{1},\phi_{2},\ldots\right]$. We combine camera-aware
positional embeddings $\delta$ and image features $\phi$ to compute attention
keys. Finally, we perform softmax-cross-attention [42] between keys
$[\delta,\phi]$, values $\phi$, and map-view queries $c-\tau_{k}$.

The softmax attention uses a cosine similarity between keys and queries as a
basic building block

$sim(\delta_{k,i},\phi_{k,i},c_{j}^{(n)}\\!,\tau_{k})=\frac{\left(\delta_{k,i}+\phi_{k,i}\right)\\!\cdot\\!\left(c_{j}^{(n)}\\!-\tau_{k}\right)}{\|\delta_{k,i}+\phi_{k,i}\|\|c_{j}^{(n)}\\!-\tau_{k}\|}.$
(3)

This cosine similarity follows the geometric interpretation in Equation 2.
This cross-view attention forms the basic building block of our cross-view
transformer architecture.

###  3.2 A cross-view transformer architecture

The first stage of the network builds up a camera-view representation for each
input image. We feed each image $I_{i}$ into feature extractor
(EfficientNet-B4 [40]) and get a multi-resolution patch embedding
$\\{\phi_{1}^{1},\phi_{1}^{2},\ldots,\phi_{n}^{R}\\}$, where $R$ is the number
of resolutions we consider. We found $R=2$ resolutions to produce sufficiently
accurate results. We process each resolution separately. We start from the
lowest resolution and project all image features into map-view using cross-
view attention. We then refine the map-view embedding and repeat the process
for higher resolutions. Finally, we use three up-convolutional layers to
produce the full resolution output.

A detailed overview of this architecture is shown in Figure 2. The final
network is end-to-end trainable. We train all layers using ground truth
semantic map-view annotations and a focal loss [19].

##  4 Implementation Details

#### Architecture.

We use (and fine-tune) a pre-trained EfficientNet-B4 [40] to compute image
features at two different scales - (28, 60) and (14, 30), which correspond to
a 8x and 16x downscaling, respectively. The initial map-view positional
embedding is a tensor of learned parameters $w\times h\times D$, where
$D=128$. For computational efficiency, we choose $w=h=25$ as the cross-
attention function grows quadratically with grid size. The encoder consists of
two cross-attention blocks: one for each scale of patch features. We use
multi-head attention with $4$ heads and an embedding size $d_{head}=64$. The
decoder consists of three (bilinear upsample + conv) layers to upsample the
latent representation to the final output size. Each upsampling layer
increases the resolution by a factor of 2 up to a final output resolution of
$200\times 200$. This corresponds to a $100\times 100$ meter area centered
around the ego-vehicle.

#### Training.

We train all models using a focal loss [19], with a batch size of 4 (per GPU)
for 30 epochs. We optimize using the AdamW [21] optimizer with the one-cycle
learning rate scheduler [37]. Training converges within 8 hours on a 4 GPU
machine.

##  5 Results

| Setting 1 | Setting 2 | #Params (M) | FPS
---|---|---|---|---
PON [32] | 24.7 | - | 38 | 30
VPN [28] | 25.5 | - | 18 | -
STA [34] | 36.0 | - | - | -
Lift-Splat [29] | - | 32.1 | 14 | 25
FIERY [14] | 37.7 | 35.8 | 7 | 8
Ours | 37.5 | 36.0 | 5 | 35
Table 1: Vehicle map-view segmentation on nuScenes. Setting 1 refers to the
100m$\times$50m at 25cm resolution setting proposed by Roddick _et al_. [32].
Setting 2 refers to the 100m$\times$100m at 50cm resolution setting proposed
by Philion and Fidler [29]. Both settings evaluate the Intersection over Union
(IoU) metric. Higher is better. For a fair comparison, we use single-timestep
models only. In particular, we compare to FIERY static [14]. In both settings,
our cross-view transformer performs at the state-of-the-art with a smaller
model and runs $4.5\times$ faster during inference.

We evaluate our cross-view transformer on vehicle and road map-view semantic
segmentation on the nuScenes [3] and Argoverse [4] datasets.

#### Dataset.

The nuScenes [3] dataset is a collection of 1000 diverse scenes collected over
a variety of weathers, time-of-day, and traffic conditions. Each scene lasts
20 seconds and contains 40 frames for a total of 40k total samples in the
dataset. The recorded data captures a full 360° view around the ego-vehicle
and is composed of 6 camera views. Each camera view has calibrated intrinsics
$K$ and extrinsics $(R,t)$ at every timestep. We resize every image to
$224\times 448$ unless specified otherwise. The Argoverse [4] dataset contains
10k total frames.

Vehicles and other objects in the scene are tracked across frames and
annotated with 3D bounding boxes using LiDAR data. Using the pose of the ego-
vehicle, we generate the ground-truth labels $y$, a binary vehicle occupancy
mask rendered at a resolution of (200, 200) by orthographically projecting 3D
box annotations onto the ground plane, following standard practice [14, 29].

#### Evaluation.

There are two commonly used evaluation settings for map-view vehicle
segmentation. Setting 1 uses a 100m$\times$50m area around the vehicle and
samples a map at a 25cm resolution. The setting, popularized by Roddick _et
al_. [32], serves as the main comparison to prior work. Setting 2 [29] uses a
100m$\times$100m area around the vehicle, with a 50cm sampling resolution.
This setting was popularized by Philion and Fidler [29] and serves as a
comparison to Lift-Splat-Shoot [29] and FIERY [14]. We use Setting 2 for all
ablations. In both settings, we use the Intersection-over-Union (IoU) score
between the model predictions and the ground truth map-view labels as the main
performance measure. We additionally report inference speeds measured on an
RTX 2080 Ti GPU.

###  5.1 Comparison to prior work

| Vehicle | Driveable Area
---|---|---
OFT [33] | 30.1 | 71.7
Lift-Splat [29] | 32.1 | 72.9
Ours | 36.0 | 74.3
Monolayout [26] | 32.1 | 58.3
PON [32] | 31.4 | 65.4
Ours | 35.2 | 73.6
Table 2: Additional comparison with models that perform map-view segmentation for vehicles and driveable area. The top and bottom rows correspond to on nuScenes [3] setting 2 and Argoverse [4] dataset respectively. | IoU
---|---
No camera-aware embedding $\delta$ | 31.0
No image features $\phi$ in attention | 33.2
No map-view embedding refinement | 33.6
Full model | 36.0
Table 3: Ablations of the cross-view attention mechanism. The first row
compares to a model that does not use camera-aware positional embedding thus
only uses image features as attention keys. The second row does not use any
image features in the keys of the attention mechanism. The third row uses the
full attention computation in camera-view but does not refine the map-view
positional embedding. All partial models degrade reasonably and perform below
the full model.

We compare our model to the five most competitive prior approaches on online
mapping. For a fair comparison, we use single-timestep models only and do not
consider temporal models. We compare to Pyramid Occupancy Networks (PON) [32],
Orthographic Feature Transform (OFT) [33], View Parsing Network (VPN) [28],
Spatio-temporal Aggregation (STA) [34], Lift-Splat-Shoot [29], and FIERY [14].
PON, VPN, STA only report numbers in Setting 1, while Lift-Splat-Shoot only
uses Setting 2.

In both settings, our cross-view transformer and FIERY outperform all
alternative approaches by a significant margin. Our cross-view transformer and
FIERY perform comparably. We have a slight edge in Setting 2, FIERY in Setting
1. The main advantage of our model is simplicity and inference speed, along
with the accompanying edge in model size. Our model trains significantly
faster (32 GPU hours vs 96 GPU hours) and performs $4\times$ faster inference.
We intentionally use the same image feature extractor (EfficientNet-B4) [40]
and similar decoder architecture as FIERY. This suggests our cross-view
transformer is capable of combining features from multiple views in a more
efficient manner.

###  5.2 Ablations of cross-view attention

The core ingredient of our approach is the cross-view attention mechanism. It
combines camera-aware embeddings and image features as keys and learned map-
view positional embeddings as queries. The map-view embeddings are allowed to
update across multiple iterations, while the camera-aware embeddings contain
some geometric information. Table 3 compares the impact of each of the
components of the attention mechanism on the resulting map-view segmentation
system. For each ablation, we trained a model from scratch using equivalent
experimental settings, changing a single component at the time.

The most important component of our system is the camera-aware positional
embedding. It bestows the attention mechanism with the ability to reason about
the geometric layout of the scene. Without it, attention has to rely on the
image feature to reveal its own location. It is possible for the network to
learn this localization due to the size of the receptive field and zero
padding around the boundary of the image. However, an image feature alone
struggles to properly link up map-view and camera-view perspectives. It also
needs to explicitly infer the direction each image is facing to disambiguate
different views. On the other hand, a purely geometric camera-aware positional
embedding alone is also insufficient. The network likely uses both semantic
and geometric cues to align map-view and camera-view, especially after the
refinement of the map-view embedding. Finally, using a single fixed map-view
embedding also degrades the performance of the model. The final model performs
best with all its attention components.

Method | IoU
---|---
None | 31.0
Learned per camera | 34.4
Camera-aware + Random Fourier | 35.8
Camera-aware + Linear Projection | 36.0
Table 4: Ablations of the camera-aware positional embeddings. The first row
compares to a model that does not use camera-aware positional embedding thus
only uses image features as attention keys. The second row uses a learned
embedding for each camera. The third row uses a camera-aware positional
embedding with a random Fourier projection. The last row uses a camera-aware
positional embedding with a linear projection (default).

###  5.3 Camera-aware positional embeddings

As we have previously seen, the camera-aware positional embedding plays a
major role in the success of the cross-view transformer. Table 4 compares
different choices for this embedding. We ablate just the positional embedding
and keep all other model and training parameters fixed.

Not using any positional embedding performs poorly. The attention mechanism
has a hard time localizing features and identifying cameras. A learned
embedding per camera performs surprisingly well. This is likely because the
camera calibration stays mostly static and a learned embedding simply bakes in
all geometric information. A camera-aware embedding with either a linear or
Random Fourier projection [41] performs best. This should not come as a
surprise as both can learn a compact embedding that directly captures the
geometry of the scene.

###  5.4 Accuracy vs distance

Figure 3: A comparison of model performance vs distance to the camera. Each
entry shows the average intersection over union accuracy for annotations that
are at least distance $d$ away.  0123$20$$25$$30$$35$# Cameras DroppedIoU
Figure 4: Degradation of our model as we randomly drop out $m\in\\{0,1,2,3\\}$
cameras. The models performance drops linearly as the observed area shrinks
roughly linearly with the number of cameras removed.

Figure 5: Qualitative results on scenes with varying degrees of occlusion.
Left shows the six camera views surrounding the vehicle. The top 3 views are
front-facing, the bottom 3 views back-facing. On the right is our predicted
map-view segmentation for vehicles and driveable area. Second from the right
is the ground truth segmentation for reference. The ego-vehicle is located at
the center of the map.

Figure 6: Visualization of cross-view attention. We compute attention from a
point in map-view coordinates and visualize the corresponding attention values
of the front camera view. Note how the network learns geometric
correspondences through this attention mechanism.

Next, we evaluate how well our model performs as the distance to the ego-
vehicle increases. For this experiment, we measure the intersection-over-union
accuracy, but ignore all predictions that are closer than a certain distance
to the ego-vehicle. Figure 3 compares to our closest competitor FIERY.

Both models have close to identical error modes. As the distance to the camera
increases, the models get less accurate. This is easiest explained through
actual qualitative results in Figure 5. Farther away vehicles are often
(partially) occluded and thus much harder to detect and segment. Our approach
degrades slower for close-by distances, but slightly under-performs FIERY at
longer ranges.

Partially occluded far-away samples have fewer corresponding image features,
thus learning a mapping from map-view to camera-view directly is harder: There
is less training data and fewer geometric priors to rely upon for our model.
We anticipate more data to make up for this difference.

###  5.5 Robustness to sensor dropout

We take a model trained on all six inputs and evaluate the intersection over
union (IoU) metric by randomly dropping $m$ cameras for each sample in the
validation set. Figure 4 shows how the performance decreases linearly with the
number of cameras dropped. This is quite intuitive as different cameras only
overlap marginally. Thus each removed camera reduces the visible area linearly
and drops the performance in the unobserved area. Note that the transformer-
based model is generally quite robust to this camera dropout and the overall
performance does not degrade beyond unobserved parts of the scene.

###  5.6 Qualitative Results

Figure 5 shows qualitative results on a variety of scenes. For each row, we
show the six input camera views and the predicted map-view segmentation along
with the ground truth segmentation. Our presented method can accurately
segments nearby vehicles, but does not sense far away or occluded vehicles
well.

###  5.7 Geometric reasoning in cross-view attention

Our quantitative experiments indicate that cross-view attention can learn some
geometric reasoning. In Figure 6, we visualize the image-view attention for
several points in the map-view. Each point corresponds to part of a vehicle.
From qualitative evidence, the attention mechanism can highlight closely
corresponding map-view and camera-view locations.

##  6 Conclusion

We present a novel map-view segmentation approach based on a cross-view
transformer architecture built on top of camera-aware positional embeddings.
The proposed approach achieves state-of-the-art performance, is simple to
implement, and runs in real time.

#### Acknowledgements.

This material is based upon work supported by the National Science Foundation
under Grant No. IIS-1845485 and IIS-2006820.

## References

  * [1] Sameer Agarwal, Yasutaka Furukawa, Noah Snavely, Ian Simon, Brian Curless, Steven M Seitz, and Richard Szeliski.  Building rome in a day.  Communications of the ACM, 2011.
  * [2] Syed Ammar Abbas and Andrew Zisserman.  A geometric approach to obtain a bird’s eye view from an image.  In ICCV Workshops, 2019.
  * [3] Holger Caesar, Varun Bankiti, Alex H. Lang, Sourabh Vora, Venice Erin Liong, Qiang Xu, Anush Krishnan, Yu Pan, Giancarlo Baldan, and Oscar Beijbom.  nuScenes: A multimodal dataset for autonomous driving.  In CVPR, 2020.
  * [4] Ming-Fang Chang, John Lambert, Patsorn Sangkloy, Jagjeet Singh, Slawomir Bak, Andrew Hartnett, De Wang, Peter Carr, Simon Lucey, Deva Ramanan, et al.  Argoverse: 3d tracking and forecasting with rich maps.  In CVPR, 2019.
  * [5] Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Scharwächter, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele.  The cityscapes dataset.  In CVPRW, 2015.
  * [6] David Eigen, Christian Puhrsch, and Rob Fergus.  Depth map prediction from a single image using a multi-scale deep network.  NeurIPS, 2014.
  * [7] Jan-Michael Frahm, Pierre Fite-Georgel, David Gallup, Tim Johnson, Rahul Raguram, Changchang Wu, Yi-Hung Jen, Enrique Dunn, Brian Clipp, Svetlana Lazebnik, et al.  Building rome on a cloudless day.  In ECCV, 2010.
  * [8] Huan Fu, Mingming Gong, Chaohui Wang, Kayhan Batmanghelich, and Dacheng Tao.  Deep ordinal regression network for monocular depth estimation.  In CVPR, 2018.
  * [9] Noa Garnett, Rafi Cohen, Tomer Pe’er, Roee Lahav, and Dan Levi.  3d-lanenet: end-to-end 3d multiple lane detection.  In ICCV, 2019.
  * [10] Andreas Geiger, Philip Lenz, and Raquel Urtasun.  Are we ready for autonomous driving? The KITTI vision benchmark suite.  In CVPR, 2012.
  * [11] Clément Godard, Oisin Mac Aodha, and Gabriel J Brostow.  Unsupervised monocular depth estimation with left-right consistency.  In CVPR, 2017.
  * [12] Clément Godard, Oisin Mac Aodha, Michael Firman, and Gabriel J Brostow.  Digging into self-supervised monocular depth estimation.  In CVPR, 2019.
  * [13] John Houston, Guido Zuidhof, Luca Bergamini, Yawei Ye, Long Chen, Ashesh Jain, Sammy Omari, Vladimir Iglovikov, and Peter Ondruska.  One thousand and one hours: Self-driving motion prediction dataset.  In CoRL, 2021.
  * [14] Anthony Hu, Zak Murez, Nikhil Mohan, Sofía Dudas, Jeffrey Hawke, Vijay Badrinarayanan, Roberto Cipolla, and Alex Kendall.  FIERY: Future instance segmentation in bird’s-eye view from surround monocular cameras.  In ICCV, 2021.
  * [15] Takeo Kanade and Masatoshi Okutomi.  A stereo matching algorithm with an adaptive window: Theory and experiment.  TPAMI, 1994.
  * [16] Youngseok Kim and Dongsuk Kum.  Deep learning based vehicle position and orientation estimation via inverse perspective mapping image.  In IV, 2019.
  * [17] Diederik P Kingma and Max Welling.  Auto-encoding variational bayes.  ICLR, 2014.
  * [18] Alex H Lang, Sourabh Vora, Holger Caesar, Lubing Zhou, Jiong Yang, and Oscar Beijbom.  Pointpillars: Fast encoders for object detection from point clouds.  In CVPR, 2019.
  * [19] Tsung-Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár.  Focal loss for dense object detection.  In CVPR, 2017.
  * [20] H Christopher Longuet-Higgins.  A computer algorithm for reconstructing a scene from two projections.  Nature, 1981.
  * [21] Ilya Loshchilov and Frank Hutter.  Decoupled weight decay regularization.  In ICLR, 2017.
  * [22] Abdelhak Loukkal, Yves Grandvalet, Tom Drummond, and You Li.  Driving among flatmobiles: Bird-eye-view occupancy grids from a monocular camera for holistic trajectory planning.  In WACV, 2021.
  * [23] Chenyang Lu, Marinus Jacobus Gerardus van de Molengraft, and Gijs Dubbelman.  Monocular semantic occupancy grid mapping with convolutional variational encoder–decoder networks.  Robotics and Automation Letters, 2019.
  * [24] Xinzhu Ma, Zhihui Wang, Haojie Li, Pengbo Zhang, Wanli Ouyang, and Xin Fan.  Accurate monocular 3d object detection via color-embedded 3d reconstruction for autonomous driving.  In ICCV, 2019.
  * [25] Fabian Manhardt, Wadim Kehl, and Adrien Gaidon.  Roi-10d: Monocular lifting of 2d detection to 6d pose and metric shape.  In CVPR, 2019.
  * [26] Kaustubh Mani, Swapnil Daga, Shubhika Garg, Sai Shankar Narasimhan, Madhava Krishna, and Krishna Murthy Jatavallabhula.  Monolayout: Amodal scene layout from a single image.  In WACV, 2020.
  * [27] Zak Murez, Tarrence van As, James Bartolozzi, Ayan Sinha, Vijay Badrinarayanan, and Andrew Rabinovich.  Atlas: End-to-end 3d scene reconstruction from posed images.  In ECCV, 2020.
  * [28] Bowen Pan, Jiankai Sun, Ho Yin Tiga Leung, Alex Andonian, and Bolei Zhou.  Cross-view semantic segmentation for sensing surroundings.  Robotics and Automation Letters, 2020.
  * [29] Jonah Philion and Sanja Fidler.  Lift, splat, shoot: Encoding images from arbitrary camera rigs by implicitly unprojecting to 3d.  In ECCV, 2020.
  * [30] Charles R Qi, Hao Su, Kaichun Mo, and Leonidas J Guibas.  Pointnet: Deep learning on point sets for 3d classification and segmentation.  In CVPR, 2017.
  * [31] René Ranftl, Katrin Lasinger, David Hafner, Konrad Schindler, and Vladlen Koltun.  Towards robust monocular depth estimation: Mixing datasets for zero-shot cross-dataset transfer.  TPAMI, 2020.
  * [32] Thomas Roddick and Roberto Cipolla.  Predicting semantic map representations from images using pyramid occupancy networks.  In CVPR, 2020.
  * [33] Thomas Roddick, Alex Kendall, and Roberto Cipolla.  Orthographic feature transform for monocular 3d object detection.  BMVC, 2019.
  * [34] Avishkar Saha, Oscar Mendez, Chris Russell, and Richard Bowden.  Enabling spatio-temporal aggregation in birds-eye-view vehicle estimation.  In ICRA, 2021.
  * [35] Johannes Lutz Schönberger and Jan-Michael Frahm.  Structure-from-motion revisited.  In CVPR, 2016.
  * [36] Sunando Sengupta, Paul Sturgess, L’ubor Ladickỳ, and Philip HS Torr.  Automatic dense visual semantic mapping from street-level imagery.  In RSS, 2012.
  * [37] Leslie N Smith and Nicholay Topin.  Super-convergence: Very fast training of neural networks using large learning rates.  In Artificial intelligence and machine learning for multi-domain operations applications, volume 11006, page 1100612. International Society for Optics and Photonics, 2019.
  * [38] Noah Snavely, Steven M Seitz, and Richard Szeliski.  Photo tourism: exploring photo collections in 3d.  In SIGGRAPH, 2006.
  * [39] Pei Sun, Henrik Kretzschmar, Xerxes Dotiwalla, Aurelien Chouard, Vijaysai Patnaik, Paul Tsui, James Guo, Yin Zhou, Yuning Chai, Benjamin Caine, et al.  Scalability in perception for autonomous driving: Waymo open dataset.  In CVPR, 2020.
  * [40] Mingxing Tan and Quoc Le.  Efficientnet: Rethinking model scaling for convolutional neural networks.  In ICML, 2019.
  * [41] Matthew Tancik, Pratul P Srinivasan, Ben Mildenhall, Sara Fridovich-Keil, Nithin Raghavan, Utkarsh Singhal, Ravi Ramamoorthi, Jonathan T Barron, and Ren Ng.  Fourier features let networks learn high frequency functions in low dimensional domains.  NeurIPS, 2020.
  * [42] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  In NeurIPS, 2017.
  * [43] Sourabh Vora, Alex H Lang, Bassam Helou, and Oscar Beijbom.  Pointpainting: Sequential fusion for 3d object detection.  In CVPR, 2020.
  * [44] Yan Wang, Wei-Lun Chao, Divyansh Garg, Bharath Hariharan, Mark Campbell, and Kilian Q Weinberger.  Pseudo-lidar from visual depth estimation: Bridging the gap in 3d object detection for autonomous driving.  In CVPR, 2019.
  * [45] Xinshuo Weng and Kris Kitani.  Monocular 3d object detection with pseudo-lidar point cloud.  In ICCV Workshops, 2019.
  * [46] Bin Xu and Zhenzhong Chen.  Multi-level fusion based 3d object detection from monocular images.  In CVPR, 2018.
  * [47] Tinghui Zhou, Matthew Brown, Noah Snavely, and David G Lowe.  Unsupervised learning of depth and ego-motion from video.  In CVPR, 2017.
  * [48] Xingyi Zhou, Dequan Wang, and Philipp Krähenbühl.  Objects as points.  In arXiv preprint arXiv:1904.07850, 2019.
  * [49] Minghan Zhu, Songan Zhang, Yuanxin Zhong, Pingping Lu, Huei Peng, and John Lenneman.  Monocular 3d vehicle detection using uncalibrated traffic cameras through homography.  IROS, 2021.