# DualCross: Cross-Modality Cross-Domain Adaptation for Monocular BEV
Perception

Yunze Man, Liang-Yan Gui, Yu-Xiong Wang
University of Illinois at Urbana-Champaign Champaign, IL 61820, USA
{yunzem2, lgui<EMAIL_ADDRESS>Yunze Man
UIUC
<EMAIL_ADDRESS>Liang-Yan Gui
UIUC
<EMAIL_ADDRESS>Yu-Xiong Wang
UIUC
<EMAIL_ADDRESS>
###### Abstract

Closing the domain gap between training and deployment and incorporating
multiple sensor modalities are two challenging yet critical topics for self-
driving. Existing work only focuses on single one of the above topics,
overlooking the simultaneous domain and modality shift which pervasively
exists in real-world scenarios. A model trained with multi-sensor data
collected in Europe may need to run in Asia with a subset of input sensors
available. In this work, we propose DualCross, a cross-modality cross-domain
adaptation framework to facilitate the learning of a more robust monocular
bird’s-eye-view (BEV) perception model, which transfers the point cloud
knowledge from a LiDAR sensor in one domain during the training phase to the
camera-only testing scenario in a different domain. This work results in the
first open analysis of cross-domain cross-sensor perception and adaptation for
monocular 3D tasks in the wild. We benchmark our approach on large-scale
datasets under a wide range of domain shifts and show state-of-the-art results
against various baselines. Our project webpage is at
https://yunzeman.github.io/DualCross.

Figure 1: Left & Middle: Existing adaptation models assume either a fixed
modality or a fixed domain between training and testing phases. Right: A more
realistic setting considers both cross-modality and cross-domain shifts. We
propose DualCross to reduce the domain and modality discrepancy, and achieve
state-of-the-art performance.

##  1 Introduction

In recent years, multi-modality 3D perception has shown outstanding
performance and robustness over its single-modality counterpart, achieving
leading results for various 3D perception tasks [14, 23, 27, 31, 36, 42] on
large-scale multi-sensor 3D datasets [3, 15, 34]. Despite the superiority in
information coverage, the introduction of more sensor modalities also poses
additional challenges to the perception system. On one hand, generalizing the
model between datasets becomes hard, because each sensor has its unique
properties, such as field-of-view (FoV) for cameras, density for LiDAR, _etc_.
On the other hand, the operation of the model is conditioned on the presence
and function of more sensors, making it hard to work on autonomous agents with
less sensor types or under sensor failure scenarios.

More specifically, transferring knowledge among different data domains is
still an open problem for autonomous agents in the wild. In the self-driving
scenario, training the perception models offline in a source domain with
annotation while deploying the model in a different target domain without
annotation is very common in practice. As a result, a model has to consider
the domain gap between source and target environments or datasets, which
usually involves different running locations, different sensor specifications,
different illumination and weather conditions, _etc_.

Meanwhile, in addition to domain shift, modality shift is another factor which
challenges the successful deployment of models. The widely adopted assumption
that all sensors are available during training, validation, and deployment
time is not always true in reality. Due to the cost and efficiency trade-off,
or sensor missing and failure, in many scenarios we can have fewer sensors
available in the target domain during testing than what we have in the source
domain during training. A typical scenario is having camera and LiDAR sensors
in the large-scale training phase while only having cameras for testing, as
shown in Figure 1. It is not clear how to facilitate the camera-only 3D
inference with the help of a LiDAR sensor only in the source domain during
training.

The challenges above raise an important question: Can we achieve robust 3D
perception under both domain shift and sensor modality shift? Existing methods
either study cross-domain scenarios assuming consistent modality [10, 14, 17,
20, 22, 29, 32, 47, 49], or study cross-modality scenarios assuming the same
domain during training and validation [6, 8, 11, 13, 18, 19, 46]. However,
simultaneous domain and modality shift poses additional challenges of large
domain discrepancy and exacerbates the ill-pose nature of 3D inference from
monocular information due to the misaligned sensory data. As we will discuss
in Sec. 3.2, our new problem setting requires a novel methodology in using
LiDAR without increasing the domain discrepancy.

To tackle the above challenges, we propose DualCross, a cross-modality cross-
domain adaptation framework for bird’s-eye-view (BEV) perception. Our model
addresses the monocular 3D perception task between different domains, and
utilizes additional modalities in the source domain to facilitate the
evaluation performance. Motivated by the fact that image and BEV frames are
bridged with 3D representation, we first design an efficient backbone to
perform 3D depth estimation followed by a BEV projection. Then, to learn from
point clouds without explicitly taking them as model inputs, we propose an
implicit learning strategy, which distills 3D knowledge from a LiDAR-Teacher
to help the Camera-Student learn better 3D representation. Finally, in order
to address the visual domain shift, we introduce adversarial learning on the
student to align the features learned from source and target domains.
Supervision from the teacher and feature discriminators are designed at
multiple layers to ensure an effective knowledge transfer.

By considering the domain gap and effectively leveraging LiDAR point clouds in
the source domain, our proposed method is able to work reliably in more
complicated, uncommon, and even unseen environments. Our model achieves state-
of-the-art performance in four very different domain shift settings. Extensive
ablation studies are conducted to investigate the contribution of our proposed
components, the robustness under different changes, and other design choices.

The main contributions of this paper are as follows. (1) We introduce mixed
domain and modality mismatch, an overlooked but realistic problem setting in
3D domain adaptation in the wild, leading to a robust camera-only 3D model
that works in complicated and dynamic scenarios with minimum sensors
available. (2) We propose a novel LiDAR-Teacher and Camera-Student knowledge
distillation model, which considerably outperforms state-of-the-art LiDAR
supervision methods. (3) Extensive experiments in challenging domain shift
settings demonstrate the capability of our methods in leveraging source domain
point cloud information for accurate monocular 3D perception.

##  2 Related Work

Multi- and Cross-modality 3D Perception. Considerable research has examined
leveraging signals from multiple modalities, especially images and point
clouds, for 3D perception tasks. Early work [21] projects point clouds to the
BEV frame and fuses 2D RGB features to generate proposals and regress bounding
boxes. Later work [48, 51] explores deep fusion between points and images.
Under the umbrella of the cross-modality setting, 2DPASS [46] transfers
features learned from images to the LiDAR. BEVDepth [19] obtains reliable
depth estimation by exploiting camera parameters with image features during
training. More recently, a line of work explores knowledge distillation from
one sensor to another for 3D object detection [6, 8, 11, 13, 18, 19, 46]. On
the contrary, our method explores a more realistic yet challenging setting,
where we use LiDAR data in one domain (Boston/Sunny/Daylight) during training
to help the camera-only model during inference in another domain
(Singapore/Rainy/Night). As a result, we analyze and improve the actual
usefulness of additional sensors under domain shift settings.

Cross-domain 3D Perception. While extensive research has been conducted on
domain adaptation for 2D tasks, the field of domain adaptation for 3D
perception in the real world has received relatively less attention. Some
prior work adapts depth estimation from synthetic to real image domains [17,
50]. Working on point clouds, PointDAN [32] designs a multi-scale adaptation
model for 3D classification. For 3D semantic segmentation, SqueezeSeg [43]
projects point clouds to the 2D view, while other work [10, 14, 29] leverages
point clouds and images data together. Recent work [22, 47, 49] explores
cross-domain 3D object detection from point clouds. SRDAN [49] employs
adversarial learning to align the features between different domains. Although
prior work [14, 20] explores various domain adaptation techniques for
different sensor modalities, these methods only adopt the same modalities to
learn the domain shift between source and target data. In contrast, our
approach achieves robust 3D perception in a more general scenario, where the
model can perform accurate 3D inference in the target domain by adapting
information encoded in source-exclusive modalities.

3D Inference in Bird’s-Eye-View Frame. Inferring 3D scenes from the BEV
perspective has recently received a large amount of interest due to its
effectiveness. MonoLayout [24] estimates the layout of urban driving scenes
from images in the BEV frame and uses an adversarial loss to enhance the
learning of hidden objects. Another work [4] proposes to employ graphical
representation and temporal aggregation for better inference in the driving
scenarios using on-board cameras. Recently, using BEV representation to merge
images from multiple camera sensors has become a popular approach [12, 26].
Following the monocular feature projection proposed by Orthographic Feature
Transform (OFT) [33], Lift-Splat-Shoot [30] disentangles feature learning and
depth inference by learning a depth distribution over pixels to convert camera
image features into BEV. Unlike the above work performing BEV analysis in
settings with more controlled premises, we are the first to explore cross-
domain and cross-sensor settings, leading to a more robust and more realistic
3D inference methodology.

Figure 2: Overview of our DualCross framework. DualCross includes three
components. (1) LiDAR-Teacher uses voxelized LiDAR point clouds to transform
the image features to the BEV frame. It provides essential knowledge on how to
guide image learning given LiDAR information. (2) Camera-Student is supervised
by the teacher model as well as the LiDAR ground truth. (3) Discriminators are
used to align features from source and target domains.

##  3 Approach

In this work, we consider the task of learning BEV representation of scenes
with domain shift and modality mismatch. Specifically, the model will be given
annotated LiDAR point clouds and camera images in the source domain, but only
unannotated camera images in the target domain. And the model seeks to achieve
highest performance on the unsupervised target domain. This problem setting is
common and worthwhile, especially considering the existence of many existing
public multi-modality datasets and the rise of many camera-only vehicle
scenarios.

Formally, for the source domain, we are given labeled data with $N^{s}$ multi-
modality samples,
$\mathcal{D}^{s}=\\{(\bm{X}^{s}_{i},\bm{P}^{s}_{i},\bm{y}^{s}_{i})\\}_{i=1}^{N^{s}}$,
where $s$ represents the source domain. Here
$\bm{X}^{s}_{i}=\\{\bm{x}^{s}_{ik}\\}_{k=1}^{n}$ consists of $n$ camera images
$\bm{x}^{s}_{ik}\in\mathbb{R}^{3\times H\times W}$. The number of cameras $n$
can take any integer as small as one, depending on the dataset or cameras
deployed on the vehicle. In addition, each camera image has an intrinsic
matrix and an extrinsic matrix. $\bm{P}^{s}_{i}$ is a point cloud containing
multiple unordered points $\bm{p}\in\mathbb{R}^{3}$ represented by 3D
coordinate values. And label $\bm{y}^{s}_{i}$ represents rasterized
representation of the scenes in the BEV coordinate. For the target domain, we
are given unlabeled data with $N^{t}$ image samples,
$\mathcal{D}^{t}=\\{\bm{X}^{t}_{i}\\}_{i=1}^{N^{t}}$, where $t$ represents the
target domain, and we want to estimate $\\{\bm{y}^{t}_{i}\\}_{i=1}^{N^{t}}$,
the BEV representation of the scenes in the target domain.

An overview of our method DualCross is illustrated in Figure 2. DualCross is
designed to extract features from monocular images and project the features
into the BEV frame (Section 3.1), using estimated or ground truth 3D depth
information. The model is composed of a LiDAR-teacher and a Camera-student
(Section 3.2), where the teacher encodes how to learn better representation
given point clouds, and transfers that knowledge to the camera-only student
using multi-level teacher-student supervision. Finally, to bridge the domain
gap between source and target domains, we leverage adversarial discriminators
at different feature layers to align the distributions across two domains in
the camera-student model (Section 3.3). Finally, we describe the overall
learning objective and loss designs (Section 3.4).

###  3.1 Learning BEV from Images

In order to achieve 3D perception under the cross-modality setting, our first
challenge is to unify the image coordinates, point cloud coordinates, and BEV
coordinates into a joint space. We follow LSS [30] to transform the image
features from perspective view into the BEV view. Specifically, we tackle this
problem by constructing a 3D voxel representation of the scene for each input
image. We discretize the depth axis into $N_{d}$ bins and lift each pixel of
the images into multiple voxels (frustums), where each voxel is represented by
the 3D coordinate of its center location. For a given pixel
$\mathrm{px}=(h,w)$ on one of the camera image, it corresponds to a set of
$N_{d}$ voxels at different depth bins:

$\displaystyle
V_{\mathrm{px}}=\\{v_{i}=M^{-1}[d_{i}h,d_{i}w,d_{i}]^{T}|i\in\\{1,2,\cdots,N_{d}\\}\\},$
(1)

where $M$ is camera matrix and $d_{i}$ is the depth of the $i$-th depth bin.
The feature vector of each voxel $v_{i}$ in $V_{\mathrm{px}}$ is the base
feature $\bm{f}_{\mathrm{px}}$ of pixel $\mathrm{px}$ scaled by the depth
value $\alpha_{i}$. More specifically, $\bm{f}_{v_{i}\in
V_{\mathrm{px}}}=\alpha_{i}\cdot\bm{f}_{\mathrm{px}}$, where the pixel feature
$\bm{f}_{\mathrm{px}}$ is extracted by an image encoder. And the depth value
$\alpha_{i}$ is obtained either from LiDAR point clouds or by estimation, in
the teacher and student model, respectively. The acquirement of $\alpha_{i}$
is introduced in Sec. 3.2.

After getting the feature for each of the voxels, we project the voxels onto
the BEV and aggregate the features to get the BEV feature map. The BEV frame
is rasterized into $(X,Y)$ 2D grids, and for each grid, its feature is
constructed from the features of all the 3D voxels projected into it using
mean pooling. This projection allows us to transform an arbitrary number of
camera images into a unified BEV frame. Finally, we obtain an image-like BEV
feature embedding, which is used to estimate the final representation using a
convolutional neural network (CNN) decoder.

This architecture design bridges the image and LiDAR modalities through an
intermediate 3D voxelized representation. Hence, we can take LiDAR point
clouds as input into the model to directly guide the BEV projection without
having to change the overall pipeline. This further enables the distillation
of knowledge from the point clouds to images using a teacher-student model.

###  3.2 LiDAR-Teacher and Camera-Student

The co-existence of domain and modality gaps poses additional challenges to
the adaptation task. Although the LiDAR sensor in the source domain provides
3D knowledge to the model, it also increases the discrepancy between the two
domains, which hurts the model adaptation (as we will see in Sec. 4.2 and
Table 4). Hence, the unique difficulty of our work lies in exploiting the
LiDAR point clouds during training to guide the camera model for better 3D
estimation.

Depth Supervision by Point Clouds.  The main advantage of point clouds over
the image modality is the accurate 3D positional information coming from the
depth measurement. Due to the lack of LiDAR during evaluation, we cannot use
point clouds as direct input of the model. Hence, one alternative approach to
using point clouds is to supervise the depth estimation in the model. As in
Eq. 1, for each pixel, we calculate the features of its corresponding voxels
by multiplying the pixel feature with a depth value $\alpha_{i}$. We use
another head to predict a depth distribution
$\bm{\alpha}_{\mathrm{px}}=\\{\alpha_{1},\alpha_{2},\ldots,\alpha_{N_{d}}\\}$
over $N_{d}$ depth bins for each pixel $\mathrm{px}$.

The ground truth depth supervision for this estimation task is generated by
LiDAR point clouds as follows: When projected to the image frame, the points
corresponding to one pixel can have three conditions. If the pixel has, (1) no
point inside: the ground truth depth distribution of it is omitted; (2) only
one point inside: the ground truth depth distribution is a one-hot vector,
with value one being in the voxel that the point lies in; (3) multiple points
inside: the ground truth depth distribution $\alpha_{i}$ of this point is
calculated by counting the number of points in each depth bin, and dividing
them by $V_{\mathrm{px}}$, the total number of points in
$\bm{\alpha}_{\mathrm{px}}$: ${\alpha}_{i}=\frac{\text{ Number of points in
depth bin }v_{i}}{\text{ Total number of points in }V_{\mathrm{px}}}$.

Using a distribution-based depth representation effectively accounts for the
ambiguity when objects of different depth occur in one pixel. This happens at
the boundary of the objects, and becomes more severe during feature encoding
processing when images get down-sampled and each pixel represents larger
space. Moreover, a probabilistic depth representation considers uncertainty
during depth estimation, and degenerates to pseudo-LiDAR methods [41] if the
one-hot constraint is added.

Learning from LiDAR-Teacher.  Despite being intuitive and straightforward,
direct depth supervision is not optimal for two reasons. First, LiDAR
supervision is only on the intermediate feature layer, providing no
supervision on the second half of the model. Also, while LiDAR provides
accurate depth measurement, “depth estimation” is still different from our
overall objective on BEV representation. Motivated by this, as shown in Figure
2, we propose to use a pretrained LiDAR oracle model to supervise the image
model at the final BEV feature embedding, such that the supervision of LiDAR
is provided to the whole model and aligns better with the final objective. We
call the model using ground truth point cloud information “LiDAR-Teacher,” and
the model to be supervised “Camera-Student.” This boils down to a knowledge
distillation problem where the 3D inference knowledge of the LiDAR-teacher is
distilled to the camera-only student. Note that the classic problem of “better
teacher, worse student” [7, 25, 52] in knowledge distillation due to capacity
mismatch does not exist in this model, because the LiDAR-Teacher and Camera-
Student models in DualCross are almost identical.

Overall, this teacher-student mechanism allows the camera model to learn
better 3D representation from the point clouds, leading to better LiDAR
supervision at different stages, while still keeping the model image-centric
for image-only inference.

Table 1: DualCross leads to significant improvements under day-to-night and
dry-to-rain domain shift settings. Numbers reported in IoU. DA and CM denote
whether a model considers domain adaptation and cross-modality in design,
respectively.

Day $\rightarrow$ Night | DA | CM |  Vehicle |  Road |  Lane
---|---|---|---|---|---
MonoLayout [24] |  ✗ |  ✗ | 5.9 | 37.7 | 5.9
OFT [33] |  ✗ |  ✗ | 6.6 | 40.5 | 6.0
LSS [30] |  ✗ |  ✗ | 6.7 | 41.2 | 7.1
Wide-range Aug. |  ✓ |  ✗ | 10.3 | 46.0 | 10.4
Vanilla DA |  ✓ |  ✗ | 11.2 | 48.8 | 11.1
Depth-Supv DA |  ✓ |  ✓ | 15.7 | 50.5 | 14.2
Input-fusion Teacher |  ✓ |  ✓ | 14.9 | 48.8 | 13.1
DualCross (ours) |  ✓ |  ✓ | 17.0 | 51.8 | 16.9

Dry $\rightarrow$ Rain | DA | CM |  Vehicle |  Road |  Lane
---|---|---|---|---|---
MonoLayout [24] |  ✗ |  ✗ | 20.6 | 68.7 | 13.1
OFT [33] |  ✗ |  ✗ | 24.1 | 79.8 | 16.2
LSS [30] |  ✗ |  ✗ | 27.8 | 71.0 | 16.8
Wide-range Aug. |  ✓ |  ✗ | 28.2 | 71.2 | 17.2
Vanilla DA |  ✓ |  ✗ | 29.1 | 70.8 | 18.3
Depth-Supv DA |  ✓ |  ✓ | 29.6 | 71.8 | 19.1
Input-fusion Teacher |  ✓ |  ✓ | 29.5 | 71.0 | 18.8
DualCross (ours) |  ✓ |  ✓ | 29.6 | 71.9 | 19.5

Table 2: DualCross performs the best under city-to-city shift.

Boston $\rightarrow$ Singapore | DA | CM |  Vehicle |  Road |  Lane
---|---|---|---|---|---
MonoLayout [24] |  ✗ |  ✗ | 14.2 | 35.9 | 7.5
OFT [33] |  ✗ |  ✗ | 16.8 | 37.9 | 9.6
LSS [30] |  ✗ |  ✗ | 17.6 | 38.2 | 10.6
Wide-range Aug. |  ✓ |  ✗ | 17.9 | 40.5 | 12.4
Vanilla DA |  ✓ |  ✗ | 13.0 | 31.4 | 9.1
Depth-Supv DA |  ✓ |  ✓ | 19.0 | 42.8 | 14.9
Input-fusion Teacher |  ✓ |  ✓ | 18.6 | 42.7 | 14.1
DualCross (ours) |  ✓ |  ✓ | 20.5 | 43.1 | 15.6

Table 3: DualCross performs the best under dataset-to-dataset domain gaps in
IoU.

nuScenes $\rightarrow$ Lyft | DA | CM |  Vehicle
---|---|---|---
MonoLayout [24] |  ✗ |  ✗ | 11.8
OFT [33] |  ✗ |  ✗ | 16.5
LSS [30] |  ✗ |  ✗ | 19.9
Wide-range Aug. |  ✓∗ |  ✗ | 21.9
Vanilla DA |  ✓ |  ✗ | 22.5
Depth-Supv DA |  ✓ |  ✓ | 23.4
Input-fusion Teacher |  ✓ |  ✓ | 22.8
DualCross (ours) |  ✓ |  ✓ | 24.4

###  3.3 Cross-Domain Adaptation

Since the BEV annotations and the LiDAR ground truth are only available in the
source data, the model will be heavily biased to the source distribution
during teacher-student supervision. Hence, we bridge the target and source
domains using adversarial training. Specifically, we place one discriminator
$D_{1}$ at the BEV decoder CNN blocks, and another $D_{2}$ at the image
encoder CNN blocks, to align the features of two domains by optimizing over
discriminator losses. While the final-layer discriminator $D_{1}$ is
constantly useful to align features learned from the LiDAR-Teacher and final
ground truth, we find that the middle-layer discriminator $D_{2}$ is very
effective under certain domain gaps where images have great changes but LiDAR
remains robust.

To achieve adversarial learning, given a feature encoder $E$ and input sample
$X$, a domain discriminator $D$ is used to discriminate whether the feature
$E(X)$ comes from the source domain or the target domain. The target and
source domain samples are given the label $d=1$ and $d=0$, respectively. And
$D(E(X))$ outputs the probability of the sample $X$ belonging to the target
domain. Hence, the discriminator loss is formulated by a cross-entropy loss:

$\displaystyle\mathcal{L}_{\mathrm{dis}}=d\log D(E(X))+(1-d)\log(1-D(E(X))).$
(2)

Moreover, in order to learn domain-invariant features, our feature encoder $E$
should try to extract features that fool the discriminator $D$, while the
discriminator $D$ tries to distinguish the right domain label of the samples.
This adversarial strategy can be formulated as a “min-max” optimization
problem:
$\mathcal{L}_{\mathrm{D}}=\min_{E}\max_{D}\mathcal{L}_{\mathrm{dis}}$. The
“min-max” problem is achieved by a Gradient Reverse Layer (GRL) [9], which
produces reverse gradient from the discriminator $D$ to learn the domain-
invariant encoder $E$. The loss form is the same for both $D_{1}$ and $D_{2}$.

###  3.4 Full Objective and Inference

The overall objective of our model is composed of the supervision from the BEV
ground truth, the LiDAR-Teacher, and the domain alignment discriminators.
Given the output rasterized BEV representation map
$\bm{y}\in\mathbb{R}^{X\times Y\times C}$, the ground truth (GT) loss term
$\mathcal{L}_{\mathrm{GT}}$ can be formulated as a cross-entropy loss between
the estimated source domain BEV map $\bm{\tilde{y}}^{s}$ and the GT label
$\bm{y}^{s}$:

$\displaystyle\mathcal{L}_{\mathrm{GT}}(\bm{\tilde{y}}^{s},\bm{y}^{s})=-\sum_{i=1}^{X}\sum_{j=i}^{Y}\sum_{k=1}^{C}y_{(i,j,k)}^{s}\log\tilde{y}_{(i,j,k)}^{s}.$
(3)

The supervision from the LiDAR-Teacher is composed of a direct depth
estimation loss $\mathcal{L}_{\mathrm{dp}}$ and a teacher feature supervision
$\mathcal{L}_{\mathrm{T}}$. As described in Sec. 3.1, given the 3D depth
volume $\bm{\alpha}\in\mathbb{R}^{H\times W\times N_{d}}$, the direct depth
supervision term $\mathcal{L}_{\mathrm{dp}}$ is formulated as a cross-entropy
loss between the estimated 3D depth distribution volume
$\bm{\tilde{\alpha}}^{s}$ in the source domain, and the GT depth volume
$\bm{\alpha}^{s}$ calculated from LiDAR point clouds as described in Sec. 3.2:

$\displaystyle\mathcal{L}_{\mathrm{dp}}(\bm{\tilde{\alpha}}^{s},\bm{\alpha}^{s})=-\sum_{i=1}^{H}\sum_{j=i}^{W}\sum_{k=1}^{N_{d}}\alpha_{(i,j,k)}^{s}\log\tilde{\alpha}_{(i,j,k)}^{s}.$
(4)

And for the LiDAR-Teacher feature supervision:
$\mathcal{L}_{\mathrm{T}}(\bm{F}^{\mathrm{te}},\bm{F}^{\mathrm{st}})=\mathcal{L}_{2}(\bm{F}^{\mathrm{te}},\bm{F}^{\mathrm{st}})$
is an $\mathcal{L}_{\mathrm{2}}$ loss, where $\bm{F}^{\mathrm{te}}$ and
$\bm{F}^{\mathrm{st}}$ are the feature maps of teacher and student models,
respectively. Finally, the domain adaptation loss contains
$\mathcal{L}_{\mathrm{D_{1}}}$ and $\mathcal{L}_{\mathrm{D_{2}}}$ with the
form described in Eq. 2.

The final objective  is formulated as a multi-task optimization problem:

$\displaystyle\mathcal{L}_{\mathrm{DualCross}}=\mathcal{L}_{\mathrm{F}}+\mathcal{\lambda}_{\mathrm{T}}\mathcal{L}_{\mathrm{T}}+\mathcal{\lambda}_{\mathrm{dp}}\mathcal{L}_{\mathrm{dp}}+\mathcal{\lambda}_{\mathrm{D_{1}}}\mathcal{L}_{\mathrm{D_{1}}}+\mathcal{\lambda}_{\mathrm{D_{2}}}\mathcal{L}_{\mathrm{D_{2}}},$
(5)

where
$\mathcal{\lambda}_{\mathrm{T}},\mathcal{\lambda}_{\mathrm{dp}},\mathcal{\lambda}_{\mathrm{D_{1}}},\text{and
}\mathcal{\lambda}_{\mathrm{D_{2}}}$ are weights for the corresponding loss
terms. The DualCross model is trained end-to-end using the loss term in Eq. 5.
During inference, target samples are fed into the Camera-Student model to
output the final BEV representation. More training details are provided in
Sec. 4.

##  4 Experiments

Datasets and Domain Settings. We evaluate DualCross with four unique domain
shift settings constructed from two large-scale datasets, nuScenes [3] and
Lyft [15], following existing LiDAR-based domain adaptation work, including
SRDAN [49], ST3D [47], UDA3D [22], and xMUDA [14]. Specifically, for the day-
to-night, city-to-city, and dry-to-rain settings, we use the sentence in the
nuScenes dataset and filter the keywords to split the dataset into
corresponding subsets to create the intra-class adaptation scenarios. For the
dataset-to-dataset setting, we use the official split of the nuScenes dataset,
and the split provided in ST3D [47] for the Lyft dataset. All adaptation
settings follow the assumption that the source has access to cameras and LiDAR
sensors, while the target only has cameras. We use all six cameras provided by
the nuScenes dataset. We also analyze surprising observations on cross-
modality performance in the ablation study.

Implementation Details. Following [30], we use EfficientNet [35] pretrained on
ImageNet as our image encoder backbone. Two heads are applied to estimate
pixel features and pixel-wise depth distribution from the $8\times$ down-
sampled feature map. The 3D feature maps are projected to the BEV frame using
mean pooling. For the BEV decoder we use ResNet-18 as the backbone, and
upsample the features learned from the first three meta-layers of ResNet to
the final BEV output. The $D_{1}$ and $D_{2}$ domain discriminators are
applied to the output feature layers of EfficientNet and ResNet backbones,
respectively. We use a light-weight discriminator architecture, which is
composed of a global averaging pooling layer, followed by two fully-connected
layers, and outputs the domain label. For input, we resize and crop input
images to size $128\times 352$. For output, we consider a $100\text{
meters}\times 100\text{ meters}$ range centered at the ego-vehicle, with the
grid size set to be $0.5\text{ meters}\times 0.5\text{ meters}$. The depth bin
is set to be $1.0\text{ meter}$ between $4.0\text{ meters}$ and $45.0\text{
meters}$ range. The whole model is trained end-to-end, with
$\lambda_{\mathrm{T}}=1.0,\lambda_{\mathrm{dp}}=0.05,\lambda_{\mathrm{D_{1}}}=0.1,\lambda_{\mathrm{D_{2}}}=0.01$.
We train DualCross using the Adam [16] optimizer with learning rate $0.001$
and weight decay 1$e$-7 for $50$K steps for the teacher model, and $200$K for
the student model. We use horizontal flipping, random cropping, rotation, and
color jittering augmentation during training. The whole model is implemented
using the PyTorch framework [28].

Figure 3: Qualitative Results in Day $\mathbf{\rightarrow}$ Night setting
(model is trained with daytime data, and validated with night data). We notice
that DualCross performs significantly better than other baselines for
vehicles, drivable roads, and lane marking classes. From left to right: (1)
Vanilla adversarial learning; (2) LiDAR as depth supervision with adversarial
learning; (3) our DualCross model; (4) Ground Truth. Best viewed in color.

###  4.1 BEV Segmentation Results and Comparisons

Baselines.  We compare our method with state-of-the-art BEV 3D layout
perception work MonoLayout [24], OFT [33], LSS [30], as well as other baseline
methods in domain adaptation and cross-modality learning. Wide-range Aug.
means using a wide range of random scaling augmentation which potentially
includes the target domain scale. For Vanilla DA, we adapt camera-only DA-
Faster [5] to our BEV perception setting. Depth-Supv DA stands for depth-
supervised domain adaptation. We use source domain LiDAR as ground truth to
supervise the depth estimation during training, without LiDAR-Teacher
supervision (only $\mathcal{L}_{\mathrm{dp}}$ without
$\mathcal{L}_{\mathrm{T}}$). Input-fusion Teacher is an alternative way of
designing the LiDAR-Teacher, where we directly fuse point $(x,y,z)$
coordinates into their corresponding image pixels as additional channels in
the teacher model, similar to Pointpainting [36]. We use DA and CM to denote
whether a model considers domain adaptation and cross-modality in design,
respectively. Results are reported on vehicle, drivable roads, and lane
marking classes using intersection-over-union (IoU).

Day-to-Night Adaptation.  As shown in Table 1 on the left, we observe that our
DualCross model achieves the best performance on all classes. We notice that
the improvement under the Day $\rightarrow$ Night setting is exceptionally
high. This is because the initial domain gap between day and night scenarios
is very large in the camera modality space. Moreover, the LiDAR sensor is
robust under illumination changes, due to its active imaging mechanism as
opposed to camera’s passive one. Thus, incorporating LiDAR point cloud
information helps the model to learn a more robust, illumination-invariant
representation from the image inputs.

Dry-to-Rain Adaptation.  As shown in Table 1 on the right, under this setting
we also observe that our DualCross model achieves the best performance on all
classes. We notice that the improvement under the Dry $\rightarrow$ Rain
setting is not as big as the previous setting. This is because the domain gap
between dry and rain scenarios is not big in the image modality. Hence,
baseline methods OFT and LSS are already able to obtain decent results even
without domain adaptation. Furthermore, rainy weather is known to cause great
domain shift in the LiDAR modality [45]. As a result, the knowledge learned
from source-exclusive LiDAR suffers from an unknown domain shift which hinders
its usefulness. This can potentially cancel out the benefit of 3D information
learned from point clouds and explains for the smaller improvement.

Dataset-to-Dataset Adaptation.  As shown in Table 3, we also observe that our
DualCross model achieves the best performance in the nuScenes $\rightarrow$
Lyft setting. Following [30], because Lyft does not provide road segment and
lane marking information in the HD map, we report results on the vehicle
class. Compared with baselines with and without domain adaptation or cross-
modality learning, our DualCross demonstrates superior performance in
leveraging and adapting LiDAR information.

City-to-City Adaptation.  As shown in Table 3, we observe that our DualCross
model achieves the best performance on all classes for two inter-city transfer
settings. Without domain adaptation, baseline approaches MonoLayout, OFT, and
LSS all suffer from performance degradation. Direct depth supervision and
alternative input-fusion teacher models do not bring as much improvement as
DualCross. The results clearly demonstrate the effectiveness of our method by
distilling and aligning the LiDAR information for cross-modality 3D BEV
perception.

Qualitative Results.  As shown in Figure 3, under the Day
$\mathbf{\rightarrow}$ Night domain shift setting, our model achieves
significantly better monocular 3D perception than other baselines. We observe
that DualCross provides more clearly defined road boundaries and lane
markings. The depth and size of the vehicles and the road on the right side
are also predicted more accurately. DualCross only misses some vehicles that
are hardly visible in camera due to occlusion and distance. Overall, the
qualitative results validate the effectiveness of DualCross in closing the gap
between data domains and leveraging point clouds information for better 3D
inference.

Table 4: Our proposed components all contribute to the final performance. We
report results on the vehicle class under the day-to-night domain gap in IoU.
WA, AD, LS, LT stand for Wide Augmentation, Adversarial Discriminators, LiDAR
Supervision, and LiDAR-Teacher, respectively.

Baseline | WA | AD | LS | LT | Results | diff
---|---|---|---|---|---|---
✓ |  |  |  |  | 6.7 | $0$
✓ |  |  | ✓ |  | 6.4 | $-0.3$
✓ | ✓ |  |  |  | 10.3 | $+3.6$
✓ | ✓ | ✓ |  |  | 11.2 | $+4.5$
✓ | ✓ | ✓ | ✓ |  | 15.7 | $+9.0$
✓ | ✓ | ✓ | ✓ | ✓ | 17.0 | $+10.3$

###  4.2 Analysis and Ablation Study

Direct LiDAR Supervision Leads to Worse Performance. It is naturally believed
that introducing multiple sensors in the perception model is bound to increase
the model performance. Surprisingly, experiments shown in Table 4 negate this
naive intuition. When we introduce the LiDAR sensor in the source domain as
depth supervision, the result decreases by 0.3. As we described in Sec. 3.2,
the domain distribution divergence increases after introducing the sensor-
modality shift. As a result, we propose multiple components in DualCross to
account for the visual and sensor domain shifts. Experiments show that while
the wide augmentation strategy and adversarial discriminator both achieve
better results than the baseline $(11.2$ vs. $6.7$ in IoU$)$, our LiDAR-
Teacher further boosts the result to $17.0$ by leveraging effective LiDAR
knowledge distillation.

Figure 4: Results of DualCross improve as the number of LiDAR points
increases.

LiDAR Density & Comparison with Oracle Model.  As shown in Figure 4, we
validate that our model achieves higher performance when denser LiDAR is
available. This can be accomplished by grouping continuous scans of LiDAR
point clouds (from 1 to 5) into a single unit, to have a denser 3D
representation of the scene. We observe that other cross-modality baselines
including Input-Fusion Teacher and Depth-Supv models cannot effectively
leverage the LiDAR knowledge, even with dense point clouds available. We also
compare our model with the LiDAR oracle model (target domain also has the
LiDAR modality) and find that the gap between the upper bound result and the
No-LiDAR baseline is significantly reduced. The remaining performance gap is
caused by the unknown LiDAR domain gap which we hope to further reduce in
future work.

Dealing with Mixed Domain Shift.  Another common but under-explored question
we observe in the 3D domain adaption setting is the mixed domain shift
problem, where multiple types of gaps between source and target domains occur
concurrently. For example, in the nuScenes dataset, the Boston data are
collected exclusively during daytime, whereas the Singapore data encompass
both day and night captures. This leads to a mixture of city-wise and
lighting-wise domain shifts. As shown in Table 5, we find that directly
leveraging domain adaptation in this scenario leads to worse performance than
direct inference, because mixed domains in the target confuse the
discriminator. Hence, we propose a progressive learning mechanism, where we
first perform adaptation with city-wise data for $100K$ steps, and then train
the model on the full target domain dataset for another $150K$ steps. This
effectively alleviates the mixed domain shift problem, and helps DualCross
achieve leading results than other baselines.

Computational Complexity.  Table 6 summarizes the number of parameters and
inference speed for prior baselines and our model. Our Lidar-Teacher
distillation and multi-level adversarial learning modules do not affect the
inference efficiency of DualCross compared with the baseline. Our total number
of parameters is 15M, and our inference time is 33 Frame-per-Second (FPS) on a
V100 GPU, which is on par with the baseline LSS [30]. The training time for
our model is around 20 hours on 4$\times$V100 GPUs.

Table 5: The proposed progressive learning strategy effectively addresses the
challenge caused by the mixed domain gap scenario (Boston-to-Singapore mixed
with day-to-night) on nuScenes.

Mixed Domain Gap |  Vehicle |  Road |  Lane
---|---|---|---
Direct Inference | 17.6 | 38.2 | 10.6
Vanilla DA | 13.0 | 31.4 | 9.1
Progressive DA | 18.8 | 41.5 | 13.2
DualCross (ours) | 20.5 | 43.1 | 15.6

Table 6: DualCross achieves great perception results with efficient inference
time compared with the baselines.

| #Params (M) | Frame-per-Second (FPS)
---|---|---
OFT [33] | 22 | 25
LSS [30] | 14 | 35
DualCross (Ours) | 15 | 33

##  5 Conclusion

In this paper, we proposed DualCross to estimate 3D scene representation in
BEV under domain shift and modality change. To achieve this, we construct a
LiDAR-Teacher and distill knowledge from it into a Camera-Student by feature
supervision. And we further propose to align feature space between the domains
using multi-stage adversarial learning. Results on large-scale datasets with
various challenging domain gaps demonstrated the effectiveness of our
approach, which marks a significant step towards robust 3D scene perception in
the wild.

## References

  * [1] Shai Ben-David, John Blitzer, Koby Crammer, Alex Kulesza, Fernando Pereira, and Jennifer Wortman Vaughan.  A theory of learning from different domains.  Machine learning, 79(1):151–175, 2010.
  * [2] Garrick Brazil and Xiaoming Liu.  M3d-rpn: Monocular 3d region proposal network for object detection.  In ICCV, 2019.
  * [3] Holger Caesar, Varun Bankiti, Alex H Lang, Sourabh Vora, Venice Erin Liong, and Qiang Xu.  nuscenes: A multimodal dataset for autonomous driving.  In CVPR, 2020.
  * [4] Yigit Baran Can, Alexander Liniger, Danda Pani Paudel, and Luc Van Gool.  Structured bird’s-eye-view traffic scene understanding from onboard images.  In ICCV, 2021.
  * [5] Yuhua Chen, Wen Li, Christos Sakaridis, Dengxin Dai, and Luc Van Gool.  Domain adaptive faster r-cnn for object detection in the wild.  In CVPR, 2018.
  * [6] Zehui Chen, Zhenyu Li, Shiquan Zhang, Liangji Fang, et al.  Bevdistill: Cross-modal bev distillation for multi-view 3d object detection.  In ICLR, 2023.
  * [7] Jang Hyun Cho and Bharath Hariharan.  On the efficacy of knowledge distillation.  In ICCV, 2019.
  * [8] Zhiyu Chong, Xinzhu Ma, Hong Zhang, Yuxin Yue, Haojie Li, Zhihui Wang, and Wanli Ouyang.  Monodistill: Learning spatial features for monocular 3d object detection.  In ICLR, 2022.
  * [9] Yaroslav Ganin and Victor Lempitsky.  Unsupervised domain adaptation by backpropagation.  In ICML, 2015.
  * [10] Rui Gong, Dengxin Dai, Yuhua Chen, Wen Li, and Luc Van Gool.  mdalu: Multi-source domain adaptation and label unification with partial datasets.  In ICCV, 2021.
  * [11] Xiaoyang Guo, Shaoshuai Shi, Xiaogang Wang, et al.  Liga-stereo: Learning lidar geometry aware representations for stereo-based 3d detector.  In ICCV, 2021.
  * [12] Noureldin Hendy, Cooper Sloan, Feng Tian, Pengfei Duan, Nick Charchut, Yuesong Xie, Chuang Wang, and James Philbin.  Fishing net: Future inference of semantic heatmaps in grids.  In CVPR, 2020.
  * [13] Yu Hong, Hang Dai, and Yong Ding.  Cross-modality knowledge distillation network for monocular 3d object detection.  In ECCV, 2022.
  * [14] Maximilian Jaritz, Tuan-Hung Vu, Raoul de Charette, Emilie Wirbel, and Patrick Pérez.  xmuda: Cross-modal unsupervised domain adaptation for 3d semantic segmentation.  In CVPR, 2020.
  * [15] R. Kesten, M. Usman, J. Houston, T. Pandya, K. Nadhamuni, A. Ferreira, M. Yuan, B. Low, A. Jain, P. Ondruska, S. Omari, S. Shah, A. Kulkarni, A. Kazakova, C. Tao, L. Platinsky, W. Jiang, and V. Shet.  Level 5 perception dataset 2020.  https://level-5.global/level5/data/, 2019.
  * [16] Diederik P Kingma and Jimmy Ba.  Adam: A method for stochastic optimization.  arXiv preprint arXiv:1412.6980, 2014.
  * [17] Jogendra Nath Kundu, Phani Krishna Uppala, Anuj Pahuja, and R Venkatesh Babu.  Adadepth: Unsupervised content congruent adaptation for depth estimation.  In CVPR, 2018.
  * [18] Yanwei Li, Yilun Chen, Xiaojuan Qi, Zeming Li, Jian Sun, and Jiaya Jia.  Unifying voxel-based representation with transformer for 3d object detection.  2022\.
  * [19] Yinhao Li, Zheng Ge, Guanyi Yu, Jinrong Yang, Zengran Wang, Yukang Shi, Jianjian Sun, and Zeming Li.  Bevdepth: Acquisition of reliable depth for multi-view 3d object detection.  arXiv, 2022.
  * [20] Yu-Jhe Li, Xiaoliang Dai, Chih-Yao Ma, Yen-Cheng Liu, Kan Chen, Bichen Wu, Zijian He, Kris Kitani, and Peter Vadja.  Cross-domain object detection via adaptive self-training.  arXiv preprint arXiv:2111.13216, 2021.
  * [21] Ming Liang, Bin Yang, Yun Chen, Rui Hu, and Raquel Urtasun.  Multi-task multi-sensor fusion for 3d object detection.  In CVPR, 2019.
  * [22] Zhipeng Luo, Zhongang Cai, Changqing Zhou, Gongjie Zhang, Haiyu Zhao, Shuai Yi, Shijian Lu, Hongsheng Li, Shanghang Zhang, and Ziwei Liu.  Unsupervised domain adaptive 3d detection with multi-level consistency.  In ICCV, 2021.
  * [23] Yunze Man, Xinshuo Weng, Prasanna Kumar Sivakumar, Matthew O’Toole, and Kris M Kitani.  Multi-echo lidar for 3d object detection.  In ICCV, 2021.
  * [24] Kaustubh Mani, Swapnil Daga, Shubhika Garg, Sai Shankar Narasimhan, Madhava Krishna, and Krishna Murthy Jatavallabhula.  Monolayout: Amodal scene layout from a single image.  In WACV, 2020.
  * [25] Seyed Iman Mirzadeh, Mehrdad Farajtabar, Ang Li, Nir Levine, Akihiro Matsukawa, and Hassan Ghasemzadeh.  Improved knowledge distillation via teacher assistant.  In AAAI, 2020.
  * [26] Bowen Pan, Jiankai Sun, Ho Yin Tiga Leung, Alex Andonian, and Bolei Zhou.  Cross-view semantic segmentation for sensing surroundings.  RA-L, 2020.
  * [27] Jinhyung Park, Xinshuo Weng, Yunze Man, and Kris Kitani.  Multi-modality task cascade for 3d object detection.  In BMVC, 2021.
  * [28] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al.  Pytorch: An imperative style, high-performance deep learning library.  In NeurIPS, 2019.
  * [29] Duo Peng, Yinjie Lei, Wen Li, Pingping Zhang, and Yulan Guo.  Sparse-to-dense feature matching: Intra and inter domain cross-modal learning in domain adaptation for 3d semantic segmentation.  In ICCV, 2021.
  * [30] Jonah Philion and Sanja Fidler.  Lift, splat, shoot: Encoding images from arbitrary camera rigs by implicitly unprojecting to 3d.  In ECCV, 2020.
  * [31] Charles R Qi, Xinlei Chen, Or Litany, and Leonidas J Guibas.  Imvotenet: Boosting 3d object detection in point clouds with image votes.  In CVPR, 2020.
  * [32] Can Qin, Haoxuan You, Lichen Wang, C-C Jay Kuo, and Yun Fu.  Pointdan: A multi-scale 3d domain adaption network for point cloud representation.  In NeurIPS, 2019.
  * [33] Thomas Roddick, Alex Kendall, and Roberto Cipolla.  Orthographic feature transform for monocular 3d object detection.  arXiv preprint arXiv:1811.08188, 2018.
  * [34] Pei Sun, Henrik Kretzschmar, Xerxes Dotiwalla, Aurelien Chouard, Vijaysai Patnaik, Paul Tsui, James Guo, Yin Zhou, Yuning Chai, Benjamin Caine, et al.  Scalability in perception for autonomous driving: Waymo open dataset.  In CVPR, 2020.
  * [35] Mingxing Tan and Quoc Le.  Efficientnet: Rethinking model scaling for convolutional neural networks.  In ICML, 2019.
  * [36] Sourabh Vora, Alex H Lang, Bassam Helou, and Oscar Beijbom.  Pointpainting: Sequential fusion for 3d object detection.  In CVPR, 2020.
  * [37] Tai Wang, ZHU Xinge, Jiangmiao Pang, and Dahua Lin.  Probabilistic and geometric depth: Detecting objects in perspective.  In CoRL, 2022.
  * [38] Tai Wang, Xinge Zhu, Jiangmiao Pang, and Dahua Lin.  Fcos3d: Fully convolutional one-stage monocular 3d object detection.  In ICCV, 2021.
  * [39] Yan Wang, Wei-Lun Chao, Divyansh Garg, Bharath Hariharan, Mark Campbell, and Kilian Q Weinberger.  Pseudo-lidar from visual depth estimation: Bridging the gap in 3d object detection for autonomous driving.  In CVPR, 2019.
  * [40] Yue Wang and Justin M Solomon.  Object dgcnn: 3D object detection using dynamic graphs.  In NeurIPS, 2021.
  * [41] Xinshuo Weng and Kris Kitani.  Monocular 3d object detection with pseudo-lidar point cloud.  In ICCVW, 2019.
  * [42] Xinshuo Weng, Yongxin Wang, Yunze Man, and Kris M Kitani.  Gnn3dmot: Graph neural network for 3d multi-object tracking with 2d-3d multi-feature learning.  In CVPR, 2020.
  * [43] Bichen Wu, Xuanyu Zhou, Sicheng Zhao, Xiangyu Yue, and Kurt Keutzer.  Squeezesegv2: Improved model structure and unsupervised domain adaptation for road-object segmentation from a lidar point cloud.  In ICRA, 2019.
  * [44] Bin Xu and Zhenzhong Chen.  Multi-level fusion based 3d object detection from monocular images.  In CVPR, 2018.
  * [45] Qiangeng Xu, Yin Zhou, Weiyue Wang, Charles R Qi, and Dragomir Anguelov.  Spg: Unsupervised domain adaptation for 3d object detection via semantic point generation.  In ICCV, 2021.
  * [46] Xu Yan, Jiantao Gao, Chaoda Zheng, Chao Zheng, Ruimao Zhang, Shuguang Cui, and Zhen Li.  2dpass: 2d priors assisted semantic segmentation on lidar point clouds.  In ECCV, 2022.
  * [47] Jihan Yang, Shaoshuai Shi, Zhe Wang, Hongsheng Li, and Xiaojuan Qi.  St3d: Self-training for unsupervised domain adaptation on 3d object detection.  In CVPR, 2021.
  * [48] Jin Hyeok Yoo, Yeocheol Kim, Ji Song Kim, and Jun Won Choi.  3d-cvf: Generating joint camera and lidar features using cross-view spatial feature fusion for 3d object detection.  In ECCV, 2020.
  * [49] Weichen Zhang, Wen Li, and Dong Xu.  Srdan: Scale-aware and range-aware domain adaptation network for cross-dataset 3d object detection.  In CVPR, 2021.
  * [50] Shanshan Zhao, Huan Fu, Mingming Gong, and Dacheng Tao.  Geometry-aware symmetric domain adaptation for monocular depth estimation.  In CVPR, 2019.
  * [51] Ming Zhu, Chao Ma, Pan Ji, and Xiaokang Yang.  Cross-modality 3d object detection.  In WACV, 2021.
  * [52] Yichen Zhu and Yi Wang.  Student customized knowledge distillation: Bridging the gap between student and teacher.  In ICCV, 2021.

##  Appendix A Dataset Details

In this section, we explain our dataset split in more details. For our
experiments, we always split the target domain into two subsets for a fair
comparison. One of them can be accessed during training for adversarial
learning and domain alignment, and the other is held out exclusively for
validation.

City-to-City Adaptation.  For the city-to-city adaptation scenario, we sub-
sample the trainval split of the large-scale dataset nuScenes [3] captured in
Boston and Singapore cities. We treat one city as the source domain and the
other as the target domain. The Boston part of data has 467 scenes in total,
which is separated into 350 scenes for training and 117 scenes for validation.
And the Singapore part of data has 383 scenes in total, which is separated
into 287 scenes for training and 96 scenes for validation.

Day-to-Night Adaptation.  For the day-to-night adaptation scenario, we also
sub-sample the trainval split of the large-scale dataset nuScenes [3]. Every
scene in the nuScenes dataset has a sentence of description, which can be
parsed and used to categorize it into certain class. In this way, we create a
day scene subset and a night scene subset out of the whole dataset. We treat
day as the source domain and night as the target domain, because the night
subset has significantly fewer samples than the day subset. The day split has
751 scenes which are all used for training. And the night part has 99 scenes
in total, which is separated into 74 scenes for training and 25 scenes for
validation.

Dry-to-Rain Adaptation.  For the dry-to-rain adaptation scenario, we also sub-
sample the trainval split of the large-scale dataset nuScenes [3]. Similarly,
using the scene description sentence, we create a dry (non-rainy) scene subset
and a rainy scene subset out of the whole dataset. We treat dry as the source
domain and rain as the target domain, because the rain subset has
significantly fewer samples than the dry subset. The dry split has 685 scenes
which are all used for training. And the rain part of data has 165 scenes in
total, which is separated into 124 scenes for training and 41 scenes for
validation.

Dataset-to-Dataset Adaptation.  For the dataset-to-dataset adaptation
scenario, we use the two large-scale autonomous driving datasets nuScenes [3]
and Lyft [15]. We treat one dataset as the source domain and the other as the
target domain. For the nuScenes dataset, we use the original train and
validation split, which has 700 scenes and 150 scenes, respectively. The Lyft
dataset does not have an original split, so we sub-sample 132 scenes for
training and 48 scenes for validation.

##  Appendix B Additional Results and Analysis

###  B.1 Unique Challenge Due to Problem Formulation

We provide another perspective to understand the challenges brought by the co-
existence of cross-modality and cross-domain gaps, thus further motivating the
design of our architecture. It is proved that in domain adaptation, the target
domain error $\epsilon^{t}(h)$ for input $h$ can be upper-bounded by the
inequality [1]:

$\displaystyle\epsilon^{t}(h)\leq\epsilon^{s}(h)+d_{\mathcal{H}\Delta\mathcal{H}}\left(P_{X}^{s},P_{X}^{t}\right)+C,$
(6)

where the bound is composed of the source-domain error $\epsilon^{s}(h)$,
source-target distribution divergence
$d_{\mathcal{H}\Delta\mathcal{H}}\left(P_{X}^{s},P_{X}^{t}\right)$, and
another term which is considered constant in our case. Existing domain
adaptation work mostly focuses on reducing the source-domain error. However,
while the introduction of the LiDAR sensor reduces the first term, it
increases the second term, because the source and target domains have an
additional sensor-type gap. It is also demonstrated in Sec. 4 that naively
using source domain LiDAR can even hinder the target performance rather than
improve it. Hence, the unique difficulty of our work lies in leveraging LiDAR
to reduce the source-domain error $\epsilon^{s}(h)$ and, in the meanwhile,
preventing distribution divergence
$d_{\mathcal{H}\Delta\mathcal{H}}\left(P_{X}^{s},P_{X}^{t}\right)$ from
increasing too much.

Table 7: We validate the design choices of DualCross by comparing with various
depth GT generation methods and different choices of teacher supervision
signals. Results show that depth supervision generated by Point Number
Distribution (Eq. 1 in the main paper), and feature-level supervision from
LiDAR-Teacher helps the model achieve the best performance.

Design choices in Day $\rightarrow$ Night Setting |  Vehicle |  Road |  Lane
---|---|---|---
Depth Ground Truth Signal (From Point Clouds) | No LiDAR | 11.2 | 48.8 | 11.1
Majority Voting | 14.1 | 50.0 | 13.7
Random Selection | 13.8 | 49.5 | 12.6
Softmax Point Number Distribution | 13.1 | 49.9 | 11.9
Point Number Distribution (Ours) | 17.0 | 51.8 | 16.9
LiDAR-Teacher Supervision  | No LiDAR-Teacher | 15.7 | 50.5 | 14.2
Soft-label Supervision | 15.2 | 50.7 | 13.9
Feature Supervision (Ours) | 17.0 | 51.8 | 16.9

###  B.2 Different Depth GT Signal Generation Methods

As shown in Table 7, we validate DualCross’s design choice in depth ground
truth signal generation (Eq. 1 in the main paper) by comparing with different
alternative methods. Note that all the methods have the same output when there
is no point or only one point projected to a pixel. The difference only comes
from the behaviour when multiple points are projected to one pixel.

  * •

Majority Voting means to generate a one-hot GT vector by assigning “1” to the
depth bin with the most number of points inside, and randomly select one bin
when more than one bin with the most number of points.

  * •

Random Selection means to generate a one-hot GT vector by randomly selecting
one of all the projected points, and assigning “1” to the depth bin in which
the selected point lies.

  * •

Softmax Point Number Distribution means to count the number of points inside
every depth bin, and then use softmax to turn this vector into a distribution.

Finally, in DualCross we use direct Point Number Distribution, where we count
the number of points inside every depth bin, and divide the vector by the
total number of points to get a depth distribution. Results show that our
direct point number distribution outperforms other counterparts in ground
truth generation. One possible reason is because DualCross down-samples the
feature map $8\times$ when estimating the 3D depth volume. This makes it much
more common for multiple points with different depth values to fall in the
same pixel. By contrast, the other two methods, Majority Voting and Random
Selection, will lose valuable information in these cases. On the other hand,
softmax activation over-smooths the distribution, and also assigns small
values to zero bins (depth bins with no point inside). Hence, we find that
preserving the native depth distribution of the points is better to supervise
DualCross in 3D evaluation.

###  B.3 Different LiDAR-Teacher Supervision Design

As shown in Table 7, we also validate DualCross’s design choice of LiDAR-
Teacher Supervision by comparing with different alternative methods. In
addition to the feature-level supervision used in DualCross, another commonly
used teacher supervision is the soft label output. Specifically, Soft-label
Supervision means to use the class distribution output of the teacher model as
the supervision of the student model, as opposed to the one-hot vector from
the ground truth annotation. We find that the feature-level supervision
performs better than the soft-label supervision. One reason is because we use
a small number of classes, which makes the supervision from the soft labels
less informative. Moreover, because the teacher and student models in
DualCross have almost identical architecture and capacity, enforcing the
corresponding feature level similarity between the teacher and student models
provides a stronger supervision than the soft-label output, without harming
the model learning. As future work, we will explore whether soft labels may be
more useful when the number of classes in the model is larger.

Table 8: DualCross achieves the best performance on all classes under Lyft-to-
nuScenes domain gaps in IoU. DA and CM mean domain adaptation and cross
modality, respectively.

Lyft $\rightarrow$ nuScenes | DA | CM |  Vehicle
---|---|---|---
MonoLayout [24] |  ✗ |  ✗ | 7.1
OFT [33] |  ✗ |  ✗ | 11.9
LSS [30] |  ✗ |  ✗ | 13.8
Wide-range Aug. |  ✓ |  ✗ | 14.6
Vanilla DA |  ✓ |  ✗ | 15.1
Depth-Supv DA |  ✓ |  ✓ | 16.5
Input-fusion Teacher |  ✓ |  ✓ | 15.1
DualCross (ours) |  ✓ |  ✓ | 19.2

###  B.4 Lyft-to-nuScenes Adaptation

Due to page limit, Table 3 in the main paper only shows the results under the
nuScenes $\rightarrow$ Lyft setting. Here we also present the results under
the Lyft $\rightarrow$ nuScenes adaptation setting for completeness. As shown
in Table 8, we can also observe that our DualCross model achieves the best
performance. Like previous scenarios, the baseline approaches, MonoLayout,
OFT, and LSS without domain adaptation suffer from performance degradation due
to domain shift. And compared with other baselines with domain adaptation or
cross-modality learning, our DualCross performs better in leveraging and
adapting LiDAR information.

Table 9: Different types of domain gaps react unevenly to different model
designs. Direct depth supervision and the middle layer feature alignment block
provide a larger improvement under the Day $\rightarrow$ Night setting than
the Dry $\rightarrow$ Rain setting.

DualCross Designs |  Day $\rightarrow$ Night |  Dry $\rightarrow$ Rain
---|---|---
Image-only Baseline | 11.2 | 28.3
LiDAR Teacher Feature Supervision | 14.9 (+3.7) | 29.5 (+1.2)
LiDAR Teacher Feature + Depth Supervision | 17.0 (+5.8) | 29.6 (+1.3)
Without Domain Alignment | 7.1 | 28.1
Feature Alignment at Final layer | 12.2 (+5.1) | 29.6 (+1.5)
Feature Alignment at Mid + Final layer | 17.0 (+9.9) | 29.3 (+1.2)

Table 10: DualCross achieves the best performance under simultaneous modality
and domain shift for 3D object detection. Domain shift is Singapore
$\rightarrow$ Boston.

Modality-shift + Domain Shift | mAP $\uparrow$ | NDS $\uparrow$
---|---|---
LSS [30] | 16.0 | 20.3
MonoDistill [8] | 16.5 | 21.9
Depth-Supv DA | 19.1 | 23.5
DualCross | 22.5 | 26.1

Table 11: DualCross achieves great performance under only the modality shift
for 3D object detection. EfficientNet and ResNet50 are backbones for image
feature extraction.

Modality-shift Only |  mAP $\uparrow$ | NDS $\uparrow$
---|---|---
Set2Set [40] | 33.1 | 41.0
MonoDistill [8] | 34.3 | 41.2
DualCross-EfficientNet | 34.5 | 41.5
DualCross-ResNet50 | 35.2 | 42.4

###  B.5 Effect of Designs on Different Domain Gaps

We find in our experiments that different types of domain gaps react unevenly
to different model designs. As shown in Table 9, for the Day $\rightarrow$
Night setting which has less domain shift in the LiDAR point cloud modality
than the camera modality, the depth supervision and the middle layer feature
alignment block provide a large improvement on top of other modules. By
contrast, under the Dry $\rightarrow$ Rain setting, where domain gap is larger
for point clouds than images, no significant improvement is achieved by using
these two components. This further validates the necessity of using multiple
modalities under domain adaptation settings, which can effectively improve the
algorithm robustness under different domain shifts.

###  B.6 3D Detection Results and Comparisons

Baselines.  In addition to the BEV segmentation task, we compare DualCross
with state-of-the-art cross-modality 3D object detection models, MonoDistill
[8] and Set2Set [40]. The original MonoDistill was designed for the single-
camera setting on the KITTI dataset, so we extend it into a multi-camera
setting for a fair comparison. We evaluate with mean Average Precision (mAP)
and Nuscenes Detection Score (NDS) metrics [3].

Modality-Shift + Domain-Shift.  As shown in Table 11, under concurrent
modality and domain gaps, DualCross outperforms all previous baselines by a
large margin, showing that it is a more robust and trustworthy model in real-
world scenarios.

Modality-Shift Only.  As shown in Table 11 on the right, our DualCross model
achieves the best performance. By using ResNet-50, the same image feature
extractor backbone as MonoDistill, we can achieve more than 1% improvement in
mAP and NDS metrics. Moreover, our model still runs 42ms per frame during
inference, faster than MonoDistill which runs 80ms per frame.

###  B.7 Solving Scale Problem

Scale ambiguity is an inherent problem for monocular depth estimation. We
solve this problem by using the random cropping and scaling augmentation
strategy during training, and also by matching the FoV (Field-of-View) of two
domains using their intrinsic matrices. The augmentation increases the
robustness of the depth prediction model in scale difference. And the FoV
matching scales the images in the target domain to match the FoV of the source
domain. This ensures that one object looks approximately the same size across
images, if it is of the same distance to the ego vehicle in source and target
domains, thus reducing the scale ambiguity. In Table 12, we provide an
ablation study of scale augmentation and FoV matching in nuScenes-to-Lyft
adaptation. As we can see, they both improve our final model performance.

Table 12: DualCross achieves great performance with scaling augmentation and
FoV matching for nuScenes-to-Lyft domain gaps in vehicle IoU.

nuScenes $\rightarrow$ Lyft |  Scaling Aug. |  Match FoV |  Vehicle IoU
---|---|---|---
None |  |  | 23.5
with scaling aug. |  ✓ |  | 23.8
with both |  ✓ |  ✓ | 24.4

From Table 12, we also notice that even without the two methods mentioned
above, the model still performs fairly well, compared with the baseline
methods in Table 3. This is because in driving scenarios, the camera FoV, the
context in the images, and the depth distribution of the images have a
relatively strong prior – they do not have a strong discrepancy even across
different driving scenarios (domains). This is different from more general
depth estimation scenarios, where objects can have drastically different depth
distribution and the intrinsic matrix can have big differences from image to
image.

Table 13: Our proposed modules achieve great performance in both depth
estimation and semantic segmentation tasks.

Day $\rightarrow$ Night |  Depth Estimation (CEE) |  Semantic Segmentation (IoU)
---|---|---
Direct Inference | 2.56 | 27.5
Adversarial Learning (AL) | 1.97 | 31.8
Lidar-Teacher Supervision + AL | 1.41 | 32.1

Table 14: Naive LiDAR supervision cannot help the final perception under
cross-domain and cross-modality scenarios.

Method | Boston $\rightarrow$ Singapore | Dry $\rightarrow$ Rain | nuScenes$\rightarrow$Lyft
---|---|---|---
LSS [30] Baseline | 17.5 | 27.9 | 20.6
Baseline + naïve LiDAR supervision | 17.5 (-0.0) | 28.2 (+0.3) | 20.3 (-0.2)
DualCross (Full Model) | 20.5 (+3.0) | 29.6 (+1.7) | 24.4 (+3.9)

###  B.8 Ablation in Semantic Segmentation and Depth Prediction

From the task perspective, there are two domain gaps in this task, one in the
semantic segmentation task and the other in the depth prediction task. In
Table 13, we show the result of depth estimation and semantic segmentation
alone in the Day-to-Night scenario.

For depth estimation, we report the cross-entropy error (CEE) with the ground
truth we described in Sec 3.2, because each pixel will have multiple depth
values. For this task, the LiDAR teacher supervision refers to
$L_{\mathrm{dp}}$ in our main pipeline. We observe that our method
significantly improves the depth estimation metrics by 44.9%.

For semantic segmentation, we report the IoU for the vehicle class. To remove
the effect of depth estimation, we use a pre-trained depth estimation model
and fix its weight. For this task, the LiDAR teacher supervision refers to
$L_{\mathrm{T}}$ in our main pipeline. We observe that the DualCross
architecture also improves the baseline method by 16.7%.

###  B.9 Results of Naive LiDAR Supervision Under Other Adaptation Settings

In Table 14, we present the ablation study of direct LiDAR supervision under
different adaptation scenarios. We observe that in all scenarios, naive LiDAR
supervision cannot lead to better performance against the baseline.

When the source and target domains have larger visual gaps
(Day$\rightarrow$Night, nuScenes$\rightarrow$Lyft), the naive supervision
leads to worse results, and when the gap is smaller (Boston→Singapore,
Dry→Rain), the LiDAR supervision results in on-par or only slightly better
performance. The reason is that although the LiDAR sensor in the source domain
provides 3D knowledge to the model, it also increases the domain discrepancy
between the source and the target (the model has to adapt to the additional
modality shift), which hurts the model performance instead.

##  Appendix C Visualization on Failure Cases

As shown in Figure 5, we visualize the failure cases of our DualCross model.
We notice that most failure cases come from far distance, or heavy occlusion
(which are typical failure cases for baselines as well). Faraway objects are
known to be typically hard cases for monocular 3D perception [44, 39, 2, 38,
37], because the ambiguity of object depth from images becomes larger as the
distance increases. This can be potentially alleviated by using a smaller
downsampling rate when extracting the image features. As for objects inside
the red dashed boxes in Figure 5, they can be seen in the LiDAR sensor due to
the higher deployment position. But in cameras, the objects are almost
invisible due to the occlusion of vegetation, structures, or other vehicles.
The occlusion problem can be potentially addressed if we have access to
additional sensors during evaluation. As future work, we will also try to
alleviate the occlusion problem in monocular settings by leveraging temporal
information.

Figure 5: Visualization of failure cases of DualCross (model is trained with
daytime data, and validated with night data). Up: Input images. Left:
Prediction from DualCross. Right: Ground truth. We notice that major failure
cases of DualCross are far distance and occlusions of objects and regions.