# X-Align: Cross-Modal Cross-View Alignment for Bird’s-Eye-View Segmentation

Shubhankar Borse Qualcomm AI Research, an initiative of Qualcomm Technologies,
Inc. Marvin Klingner Automated Driving, Qualcomm Technologies International
GmbH Varun Ravi Kumar Automated Driving, Qualcomm Technologies, Inc. Hong Cai
11footnotemark: 1 Abdulaziz Almuzairee 11footnotemark: 1 Senthil Yogamani
Fatih Porikli 11footnotemark: 1
{sborse, mklingne, vravikum, hongcai, almuzair, syogaman,
<EMAIL_ADDRESS>
University of California, San Diego. Work done at Qualcomm.Automated Driving,
QT Technologies Ireland Limited

###### Abstract

Bird’s-eye-view (BEV) grid is a typical representation of the perception of
road components, e.g., drivable area, in autonomous driving. Most existing
approaches rely on cameras only to perform segmentation in BEV space, which is
fundamentally constrained by the absence of reliable depth information. The
latest works leverage both camera and LiDAR modalities but suboptimally fuse
their features using simple, concatenation-based mechanisms.

In this paper, we address these problems by enhancing the alignment of the
unimodal features in order to aid feature fusion, as well as enhancing the
alignment between the cameras’ perspective view (PV) and BEV representations.
We propose X-Align, a novel end-to-end cross-modal and cross-view learning
framework for BEV segmentation consisting of the following components: (i) a
novel Cross-Modal Feature Alignment (X-FA) loss, (ii) an attention-based
Cross-Modal Feature Fusion (X-FF) module to align multi-modal BEV features
implicitly, and (iii) an auxiliary PV segmentation branch with Cross-View
Segmentation Alignment (X-SA) losses to improve the PV-to-BEV transformation.
We evaluate our proposed method across two commonly used benchmark datasets,
i.e., nuScenes and KITTI-360. Notably, X-Align significantly outperforms the
state-of-the-art by 3 absolute mIoU points on nuScenes. We also provide
extensive ablation studies to demonstrate the effectiveness of the individual
components.

##  1 Introduction

Figure 1: Existing methods for cross-modal BEV segmentation (top) utilize
simple concatenation-based fusion (_e.g_., [34]), while our proposed X-Align
enforces cross-modal feature alignment together with attention-based feature
fusion, as well as cross-view segmentation alignment (bottom). These
contributions improve both feature aggregation and PV-to-BEV transformation,
leading to more accurate BEV segmentation.

Bird’s-eye-view (BEV) segmentation aims at classifying each cell in a BEV grid
around the ego position. As such, it is a key enabler for applications like
autonomous driving and robotics. For instance, the BEV segmentation map is a
prerequisite for current works on behavior prediction and trajectory planning
[17, 50, 57]. It is also an important input modality for learning end-to-end
controls (_e.g_., speed control, steering) in autonomous driving [9].

Given the ubiquity of camera sensors, existing BEV segmentation methods
predominantly focus on predicting BEV segmentation maps from camera images
[41, 58, 59, 67]. However, the lack of reliable 3D information significantly
limits the performance of these methods. A possible way to resolve this
challenge is to leverage a LiDAR sensor and fuse the measured sparse geometric
information with that contained in camera images. While camera-LiDAR fusion
has been extensively studied for the task of 3D object detection, such fusion
strategies are relatively unexplored for BEV segmentation. The concurrent work
of [34] provides the first baseline using a simple concatenation of LiDAR BEV
features and camera features projected from perspective view (PV) to BEV via
estimated depth and voxel pooling. However, the PV-to-BEV projection can be
inaccurate due to errors in depth estimation. As a result, in the
concatenation stage, the network may aggregate poorly aligned features across
the camera and the LiDAR branches, resulting in suboptimal fusion results.

In this paper, we propose a novel cross-modal, cross-view alignment strategy,
X-Align, which enforces feature alignment across features extracted from the
camera and LiDAR inputs as well as segmentation consistency across PV and BEV
to improve the overall BEV segmentation accuracy (cf. Fig. 1). First, we
propose a Cross-Modal Feature Alignment (X-FA) loss function that promotes the
correlation between projected camera features and LiDAR features, as measured
by cosine similarity. In addition, we incorporate attention to the Cross-Modal
Feature Fusion (X-FF) of these two sets of modality-specific features instead
of using simple concatenation as in [34]. This gives the network a more
substantial capability to properly align and aggregate features from the two
sensing modalities.

We further impose Cross-View Segmentation Alignment (X-SA) losses during
training. We introduce a trainable segmentation decoder based on the
intermediate PV camera features to generate PV segmentation. Next, we utilize
the same PV-to-BEV transformation [41] that converts PV camera features to BEV
to convert the PV segmentation map into a BEV segmentation map, which is then
supervised by the ground truth. We also supervise the intermediate PV
segmentation map using pseudo-labels generated by a high-quality, off-the-
shelf semantic segmentation model. In this way, the camera branch learns to
derive intermediate features containing useful PV semantic features, providing
richer information for BEV segmentation after being projected to BEV space.
Moreover, this provides additional supervision on the PV-to-BEV module,
allowing it to learn a more accurate transformation.

Our main contributions are summarized as follows:

  * •

We propose a novel framework, X-Align, that enables better feature alignment
and fusion across camera and LiDAR modalities and enforces segmentation
alignment across perspective view and bird’s eye view.

  * •

Specifically, we propose a Cross-Modal Feature Alignment (X-FA) loss to
enhance the correlation between the camera and LiDAR features. We also devise
an attention-based Cross-Modal Feature Fusion (X-FF).

  * •

We further propose to enforce Cross-View Segmentation Alignment (X-SA) across
the perspective view and bird’s eye view, which encourages the model to learn
richer semantic features and a more accurate PV-to-BEV projection.

  * •

We conduct extensive experiments on the nuScenes and KITTI-360 datasets with
comprehensive ablation studies that demonstrate the efficacy of X-Align. In
particular, on nuScenes, we surpass the state-of-the-art in BEV segmentation
by 3 absolute mIoU points.

##  2 Related Work

BEV Segmentation: The task of BEV segmentation has mostly been explored using
(multiple) camera images as input. Building on top of Perspective View (PV)
segmentation [4, 2, 3, 64, 18, 65], early works used the homography
transformation to convert camera images to BEV, subsequently estimating the
segmentation map [53, 13, 35, 68]. As the homography transformation introduces
strong artifacts, subsequent works moved towards depth estimation and
voxelization [41, 44] for the PV-to-BEV transformation as end-to-end learning
[36, 43]. This basic setup has been further explored in various directions:
VPN [39] explores domain adaptation, BEVerse [66] and M2BEV [58] explore
multi-task learning with 3D object detection, CoBEVT [59] explores fusion of
features from vehicles, Gosala  _et al_. explore panoptic BEV segmentation
[15], while several works explore incorporation of temporal context [17, 46].
Furthermore, CVT [67] uses a learned map embedding and an attention mechanism
between map queries and camera features. In contrast to these existing BEV
segmentation approaches that only use camera images, we explore the multi-
modal fusion of LiDAR point clouds and camera images.

Multi-modal fusion for BEV segmentation has been enabled by recently
introduced large-scale datasets providing time-synchronized data from multiple
sensors [5, 14, 48]. However, most works on these datasets focus on the 3D
object detection task [1, 25, 34, 62, 10, 47], while we focus on BEV
segmentation. The closest prior art to our work is BEVFusion [34]. While their
method also predicts BEV segmentation based on LiDAR point clouds and camera
images, they use a simple feature concatenation to fuse multi-modal features
such that the network implicitly has to connect information from misaligned
features. In contrast, we explicitly enforce alignment between multi-modal
features. Also, we enforce alignment between PV and BEV segmentation to
improve the PV-to-BEV transformation.

Camera-LiDAR Sensor Fusion: The vast majority of fusion methods have been
proposed for the 3D object detection task. Initially, two-stage approaches
have been proposed, lifting image bounding box proposals into 3D frustum view
[38, 42, 56] for fusion with LiDAR. However, research focus has shifted
towards end-to-end training, where approaches can roughly be divided into
three categories: point-/input-level decoration, feature-level fusion, and
proposal-level fusion. Point-level fusion includes methods such as
PointAugmenting [54], PointPainting [52], FusionPainting [60], AutoAlign [8],
and MVP [63], which extract camera features and use these features to enrich
the point-level information, which is subsequently processed by a LiDAR-based
detector. Recently proposed FocalSparseCNNs [7] similarly enriches the
features in the early feature extraction stage. For proposal-level fusion
usually the predicted bounding boxes are refined [28], often making use of an
attention mechanism such as in FUTR3D [6] and TransFusion [1]. However, these
two fusion types have downsides regarding generalization. While proposal-level
fusion is not easily generalizable to other tasks, input-level decoration is
not generically extendable to other sensor modalities.

Feature-level fusion aims at fusing extracted features from different sensors,
subsequently predicting outputs for one or several tasks [25, 26, 27, 34]. Our
approach also falls into this category. While [27, 34] use fusion via
concatenation, more recent approaches apply attention-based fusion [25, 26].
However, these approaches still try to implicitly learn the interconnection
between cross-modal features, while we explicitly encourage alignment between
features from different modalities. Also, none of the above-mentioned methods
uses attention-based cross-modal fusion for BEV segmentation or output-level
segmentation alignment of PV and BEV segmentation.

##  3 Proposed X-Align Framework

Figure 2: Our proposed X-Align framework: We present a cross-modal and cross-
view alignment algorithm for the task of BEV segmentation based on camera
images and LiDAR point clouds. We apply the Cross-View Segmentation Alignment
(X-SA) and Cross-Modal Feature Alignment (X-FA) losses during training. We
also propose a Cross-Modal Feature Fusion (X-FF) module to correct pixel
inconsistencies between multi-modal features. Our proposed contributions are
highlighted in red. During inference, we can remove the blocks which solely
contribute to computing loss functions, suggesting that the performance
enhancement comes with no added inference cost.

This section describes X-Align, our novel cross-modal and cross-view alignment
strategy. We first formally introduce the problem and describe a baseline
method in Section 3.1. We provide an overview of X-Align in Section 3.2 and
then discuss its components in detail, including Cross-Modal Feature Fusion
(X-FF), Cross-Modal Feature Alignment (X-FA), and Cross-View Segmentation
Alignment (X-SA) in Sections 3.3, 3.4, 3.5, respectively.

###  3.1 Problem Formulation and Baseline

Our goal is to develop a framework taking multi-modal sensor data
$\mathcal{X}$ as input and predicting a BEV segmentation map
$\hat{\bm{m}}\in\mathcal{S}^{H^{\mathrm{BEV}}\times W^{\mathrm{BEV}}}$, with
resolution $H^{\mathrm{BEV}}\times W^{\mathrm{BEV}}$, and the set of classes
$\mathcal{S}=\left\\{0,1,\dots,|\mathcal{S}|\right\\}$. As illustrated in Fig.
2, the set of inputs, $\mathcal{X}=\left\\{\bm{x},\bm{l}\right\\}$, contains
RGB camera images in PV, $\bm{x}\in\mathbb{R}^{N^{\mathrm{cam}}\times
H^{\mathrm{cam}}\times W^{\mathrm{cam}}\times 3}$, where $N^{\mathrm{cam}}$,
$H^{\mathrm{cam}}$, $W^{\mathrm{cam}}$ denote number of cameras, image height,
and image width, respectively, as well as a LiDAR point cloud,
$\bm{l}\in\mathbb{R}^{P\times 5}$, with number of points $P$. Each point
consists of its 3-dimensional coordinates, reflectivity, and ring index.

Baseline Method: We first establish a baseline method of fusion-based BEV
segmentation based on BEVFusion [34]. As shown in Fig. 2, initial features are
extracted from both sensor inputs. For camera images, a camera encoder,
$\bm{E}^{\mathrm{cam}}$, extracts features in PV, $\bm{f}^{\mathrm{cam}}$.
Subsequently, we use a feature pyramid network (FPN) and a PV-to-BEV
transformation based on [41] to obtain camera features in BEV space, following
BEVDet [19]. For the LiDAR points, we follow SECOND [61] in using voxelization
and a sparse LiDAR encoder, $\bm{E}^{\mathrm{LiDAR}}$. The LiDAR features are
projected to BEV space using a flattening operation as in [34].

These operations result in two sets of modality-specific BEV features,
$\bm{\tilde{f}}^{\mathrm{cam}}\in\mathbb{R}^{H^{\mathrm{lat}}\times
W^{\mathrm{lat}}\times C^{\mathrm{cam}}}$ and
$\bm{\tilde{f}}^{\mathrm{LiDAR}}\in\mathbb{R}^{H^{\mathrm{lat}}\times
W^{\mathrm{lat}}\times C^{\mathrm{LiDAR}}}$, with BEV space feature resolution
$H^{\mathrm{lat}}\times W^{\mathrm{lat}}$ and number of channels
$C^{\mathrm{cam}}$ and $C^{\mathrm{LiDAR}}$ for camera and LiDAR features,
respectively. The features are then combined (_e.g_., by simple concatenation
as in [24, 34, 16]), resulting in fused features
$\bm{\tilde{f}}^{\mathrm{fused}}\in\mathbb{R}^{H^{\mathrm{lat}}\times
W^{\mathrm{lat}}\times C^{\mathrm{fused}}}$, which are further processed by a
BEV Encoder and FPN as in SECOND [61]. Finally, the features are processed by
a segmentation head with the same architecture as in [34] to ensure
comparability. This baseline model is trained using the focal cross-entropy
loss [31]:

$\mathcal{L}^{\mathrm{BEV}}=\mathrm{FocalCE}\left(\hat{\bm{y}}^{\mathrm{BEV}},\bm{y}^{\mathrm{BEV}}\right),$
(1)

where $\hat{\bm{y}}^{\mathrm{BEV}}\in\mathbb{R}^{H^{\mathrm{BEV}}\times
W^{\mathrm{BEV}}\times S}$ are class probabilities and
$\bm{y}^{\mathrm{BEV}}\in\left\\{0,1\right\\}^{H^{\mathrm{BEV}}\times
W^{\mathrm{BEV}}\times S}$ denotes the one-hot encoded ground truth. We can
obtain the final classes $\hat{\bm{m}}$ by a pixel-wise argmax operation on
$\hat{\bm{y}}$ during inference.

###  3.2 X-Align Overview

Building on top of the previously described baseline, we present our novel
cross-modal and cross-view alignment strategy, X-Align (highlighted by red
boxes in Fig. 2). First, we improve the simple concatenation-based fusion with
a Cross-Modal Feature Fusion (X-FF) module, which leverages attention and
mitigates misalignment between features across modalities (Section 3.3).
Secondly, we propose a Cross-Modal Feature Alignment (X-FA) loss,
$\mathcal{L}^{\mathrm{X\text{-}FA}}$, which promotes the correlation between
features across modalities (Section 3.4). Finally, we propose losses to
enforce Cross-View Segmentation Alignment (X-SA) in Section 3.5, where we
supervise a PV segmentation predicted from the intermediate camera features
with a loss $\mathcal{L}^{\mathrm{PV}}$ and the PV-to-BEV-projected version of
this segmentation with a loss $\mathcal{L}^{\mathrm{PV2BEV}}$. These two
losses provide more direct training signals to the PV-to-BEV transformation
and encourage richer semantic features in PV before the transformation.
Overall, our total optimization objective is

$\mathcal{L}^{\mathrm{X\text{-}Align}}\\!=\\!\lambda_{1}\mathcal{L}^{\mathrm{BEV}}\\!+\\!\lambda_{2}\mathcal{L}^{\mathrm{X\text{-}FA}}\\!+\\!\lambda_{3}\mathcal{L}^{\mathrm{PV}}\\!+\\!\lambda_{4}\mathcal{L}^{\mathrm{PV2BEV}},$
(2)

where $\lambda_{i},\,i\in\left\\{1,2,3,4\right\\}$ are the loss weighting
factors.

###  3.3 Cross-Modal Feature Fusion (X-FF)

In recent BEV segmentation literature [34, 24, 16], it is a common approach to
utilize concatenation followed by convolution to combine features from multi-
modal inputs, $\bm{\tilde{f}}^{\mathrm{cam}}$ and
$\bm{\tilde{f}}^{\mathrm{LiDAR}}$, resulting in the aggregated features
$\bm{\tilde{f}}^{\mathrm{fused}}$. However, the lack of reliable depth
information can cause inaccurate PV-to-BEV transformation of features, which
subsequently results in suboptimal alignment and fusion of multi-modal
features. The convolution blocks utilized in existing approaches [34, 24, 16]
cannot rectify such misalignment due to their translation invariance. To
address this issue, we propose more powerful, Cross-Modal Feature Fusion
(X-FF) modules that can account for pixel-wise misalignment, as shown in Fig.
3. Next, we describe in detail our three proposed fusion designs.

Self-Attention: Our proposed X-FF using self-attention is shown in Fig. 3
(left). We first stack features
$\bm{\tilde{f}}^{\mathrm{cam}}\in\mathbb{R}^{H^{\mathrm{lat}}\times
W^{\mathrm{lat}}\times C^{\mathrm{cam}}}$ and
$\bm{\tilde{f}}^{\mathrm{LiDAR}}\in\mathbb{R}^{H^{\mathrm{lat}}\times
W^{\mathrm{lat}}\times C^{\mathrm{LiDAR}}}$, and tokenize them into $K\times
K$ patches with an embedding dimension of $L^{\mathrm{embed}}$. These patches
are fed into a multi-head self-attention module [51]. The output is then
projected back to the original resolution using a deconvolution block,
resulting in the final fused features
$\bm{\tilde{f}}^{\mathrm{fused}}\in\mathbb{R}^{H^{\mathrm{lat}}\times
W^{\mathrm{lat}}\times C^{\mathrm{fused}}}$. By using self-attention, our
proposed module can correspond to the camera and LiDAR features spatially,
accounting for potential misalignment.

Figure 3: Our three proposed Cross-Modal Feature Fusion (X-FF) strategies,
using standard self-attention (left), spatial-channel attention (middle), and
pose-driven deformable convolution (right), respectively.

Spatial-Channel Attention: In this option, we leverage the recently proposed
Split-Depth Transpose Attention (SDTA) [37], as shown in Fig. 3 (middle). It
first performs spatial and channel mixing of the stacked camera and LiDAR
features via depth-wise and point-wise convolutions. Then, it applies channel
attention followed by a lightweight MLP. The output is passed through a
deconvolution block to generate the fused features
$\bm{\tilde{f}}^{\mathrm{fused}}$. Spatial and channel mixing together with
channel attention provides powerful capacity for the module to better address
the misalignment between the camera and LiDAR features.

Pose-Driven Deformable Convolution: This design is illustrated in Fig. 3
(right). We know that the transformation between modalities is a function of
their relative poses to the ego vehicle. Hence, we apply an adaptive
transformation, _i.e_., Deformable Convolution (DCNv2) [69], to the stacked
multi-modal features $\bm{\tilde{f}}^{\mathrm{cam}}$ and
$\bm{\tilde{f}}^{\mathrm{LiDAR}}$, which can implicitly learn the cross-modal
alignment based on available pose information. More specifically, we process
the pose matrices with an MLP to generate a pose embedding
$\bm{\tilde{f}}^{\mathrm{pose}}\in\mathbb{R}^{H^{\mathrm{lat}}\times
W^{\mathrm{lat}}\times C^{\mathrm{pose}}}$, which is then concatenated with
$\bm{\tilde{f}}^{\mathrm{cam}}$ and $\bm{\tilde{f}}^{\mathrm{LiDAR}}$. They
are used to generate $K\times K$ offset vectors to be used by the DCNv2 block,
which produces the output fused features $\bm{\tilde{f}}^{\mathrm{fused}}$.

Our proposed X-FF designs provide the network with the suitable capacity to
properly align and fuse multi-modal features. While they introduce additional
computations, they show more superior accuracy-efficiency trade-offs as
compared to naively increasing the complexity of the baseline network, as we
shall see in our study in Section 4.3.

###  3.4 Cross-Modal Feature Alignment (X-FA)

While our proposed X-FF modules can improve feature alignment, they incur
additional computations, which may not always be feasible. As such, we propose
a second measure to improve feature alignment with a Cross-Modal Feature
Alignment (X-FA) loss $\mathcal{L}^{\mathrm{X\text{-}FA}}$ that is only
applied during training and does not introduce additional computations for
inference. It can also be used in conjunction with X-FF.

Consider extracted features in BEV space, $\bm{\tilde{f}}^{\mathrm{cam}}$ and
$\bm{\tilde{f}}^{\mathrm{LiDAR}}$, from camera and LiDAR branches,
respectively. We promote the correlation between the two sets of features by
imposing a cosine similarity loss between them:

$\mathcal{L}^{\mathrm{X\text{-}FA}}=\mathrm{CosineSim}\left(\bm{\tilde{f}}^{\mathrm{cam}},\bm{\tilde{f}}^{\mathrm{LiDAR}}\right).$
(3)

In order to apply this loss, there are two requirements. First, the camera
features $\bm{\tilde{f}}^{\mathrm{cam}}$ and LiDAR features
$\bm{\tilde{f}}^{\mathrm{LiDAR}}$ in BEV space need to have the same
resolution. In this work, we ensure that the network parameters are chosen
accordingly. In case both features have different resolutions, differentiable
grid sampling [20] can be used. Second, the channels $C^{\mathrm{cam}}$ and
$C^{\mathrm{LiDAR}}$ should match for Eq. (3) to be applicable. This, however,
is generally not the case. As such, we take all features from the lower-
dimensional branch and enforce their similarity to several subsets of features
from the higher-dimensional branch.

###  3.5 Cross-View Segmentation Alignment (X-SA)

In addition to encouraging feature alignment, we also impose alignment at the
output segmentation level across PV and BEV using our Cross-Modal Segmentation
Alignment (X-SA) losses. More specifically, for the camera branch, we take
intermediate PV features and feed them through an additional decoder to
generate a PV segmentation prediction
$\hat{\bm{y}}^{\mathrm{PV}}\in\mathbb{R}^{N^{\mathrm{cam}}\times
H^{\mathrm{cam}}\times W^{\mathrm{cam}}\times|\mathcal{S}|}$ (cf. Fig. 2, top
part). We further transform $\hat{\bm{y}}^{\mathrm{PV}}$ to BEV space by
utilizing the PV-to-BEV transformation as for the features, resulting in a
projected BEV segmentation map
$\hat{\bm{y}}^{\mathrm{PV}\rightarrow\mathrm{BEV}}\in\mathbb{R}^{H^{\mathrm{BEV}}\times
W^{\mathrm{BEV}}\times|\mathcal{S}|}$. The projected BEV segmentation map is
supervised w.r.t ground-truth BEV segmentation using a focal cross-entropy
loss:

$\mathcal{L}^{\mathrm{PV2BEV}}=\mathrm{FocalCE}\left(\hat{\bm{y}}^{\mathrm{PV}\rightarrow\mathrm{BEV}},\bm{y}^{\mathrm{BEV}}\right).$
(4)

As for the PV segmentation, since PV ground truth is not always available on
BEV perception datasets, we supervise it with a focal loss using pseudo-labels
generated by a state-of-the-art model pretrained on Cityscapes [12], as
follows:

$\mathcal{L}^{\mathrm{PV}}=\mathrm{FocalCE}\left(\hat{\bm{y}}^{\mathrm{PV}},\bm{y}^{\mathrm{PV}}\right),$
(5)

By introducing these two additional supervisions, we enforce that across PV
and BEV, the segmentations are accurate and aligned through the PV-to-BEV
transformation. This benefit is two-fold: First, the module used here is given
by the same PV-to-BEV transformation as on the feature level in the main
camera branch. Our X-SA loss provides additional supervision to more
accurately train this key module. Second, imposing a PV segmentation loss
$\mathcal{L}^{\mathrm{PV}}$ encourages the network to learn useful PV semantic
features, providing richer semantic information for the downstream BEV
features. Our X-SA components, including the additional decoder and the
losses, are only used during training and do not require overhead at test
time.

In summary, our complete X-Align framework X-Alignall proposes four additions
to the baseline: the X-FF feature fusion module, along with three additional
training losses: $\mathcal{L}^{\mathrm{X\text{-}FA}}$,
$\mathcal{L}^{\mathrm{PV}}$, and $\mathcal{L}^{\mathrm{PV2BEV}}$. In cases
where the network only takes camera inputs, we apply the two X-SA losses,
$\mathcal{L}^{\mathrm{PV}}$ and $\mathcal{L}^{\mathrm{PV2BEV}}$, giving us the
X-Alignview variant. In addition, in case extra computation is not allowed, we
apply all three X-Align losses when training the network, forming the
X-Alignlosses variant. We extensively evaluate these variants as well as
combinations of our proposed X-Align components in Section 4.3.

##  4 Experiments

In this section, we present comprehensive performance evaluations of X-Align
and compare it with baselines and the current state of the art. We further
conduct extensive ablation studies on various aspects of our proposed
approach.

Model | Backbone | Modality |  [HTML]96bbceDrivable |  [HTML]fb9a99Ped. Cross. |  [HTML]fc4538Walkway |  [HTML]fdbf6fStop Line |  [HTML]ff7f00Carpark |  [HTML]ab9ac0Divider |  [HTML]a5eb8dmIoU
---|---|---|---|---|---|---|---|---|---
OFT [44] | ResNet-18 | C | 74.0 | 35.3 | 45.9 | 27.5 | 35.9 | 33.9 | 42.1
LSS [41] | ResNet-18 | C | 75.4 | 38.8 | 46.3 | 30.3 | 39.1 | 36.5 | 44.4
CVT [67] | EfficientNet-B4 | C | 74.3 | 36.8 | 39.9 | 25.8 | 35.0 | 29.4 | 40.2
M2BEV [58] | ResNeXt-101 | C | 77.2 | ✗ | ✗ | ✗ | ✗ | 40.5 | ✗
BEVFusion [34] | Swin-T | C | 81.7 | 54.8 | 58.4 | 47.4 | 50.7 | 46.4 | 56.6
gray9 X-Alignview | Swin-T | C | 82.4 | 55.6 | 59.3 | 49.6 | 53.8 | 47.4 | 58.0
PointPillars [22] | VoxelNet | L | 72.0 | 43.1 | 53.1 | 29.7 | 27.7 | 37.5 | 43.8
CenterPoint [62] | VoxelNet | L | 75.6 | 48.4 | 57.5 | 36.5 | 31.7 | 41.9 | 48.6
PointPainting [52] | ResNet-101, PointPillars | C + L | 75.9 | 48.5 | 57.1 | 36.9 | 34.5 | 41.9 | 49.1
MVP [63] | ResNet-101, VoxelNet | C + L | 76.1 | 48.7 | 57.0 | 36.9 | 33.0 | 42.2 | 49.0
BEVFusion [34] | Swin-T, VoxelNet | C + L | 85.5 | 60.5 | 67.6 | 52.0 | 57.0 | 53.7 | 62.7
gray9 X-Alignlosses | Swin-T, VoxelNet | C + L | 85.8 | 63.1 | 68.6 | 53.6 | 57.9 | 56.7 | 64.3
gray8 X-Alignall | Swin-T, VoxelNet | C + L | 86.8 | 65.2 | 70.0 | 58.3 | 57.1 | 58.2 | 65.7
Table 1: Quantitative evaluation on the nuScenes validation set, in terms of single-class IoUs and the overall mIoU. We compare with existing methods from literature, where the numbers are taken from [34], as their reproduced results are better than the ones originally reported in the papers due to a higher number of trained classes. We also provide information on the backbones and input modalities in the table. Our proposed X-Align outperforms all existing approaches in both single-class IoUs and the overall mIoU by a significant margin. Model | Encoder | Modality | mIoU | PQ
---|---|---|---|---
PanopticBEV [15] | EffDet-D3 | C | 25.4 | 16.0
gray9 X-Alignview | EffDet-D3 | C | 27.8 | 16.9
Table 2: Quantitative evaluation on KITTI-360 in terms of mIoU and PQ, using
the camera-only modality.

###  4.1 Experimental Setup

Datasets: We evaluate performance on the large-scale nuScenes benchmark [5],
which provides ground-truth annotations to support BEV segmentation. It
contains 40,000 annotated keyframes captured by a 32-beam LiDAR scanner and
six monocular cameras providing a $360^{\circ}$ field of view. Following the
BEV map segmentation setup from [34], we predict six semantic classes:
drivable lanes, pedestrian crossings, walkways, stop lines, carparks, and lane
dividers. We further evaluate on KITTI-360 [29], a large-scale dataset with
83,000 annotated frames, including data collected using two fish-eye cameras
and a perspective stereo camera. KITTI-360 does not provide dense ground-truth
annotations for BEV segmentation. Hence, we use the BEV segmentation
annotations from [15] as ground truth. These contain both static classes such
as road and sidewalk, along with dynamic objects such as cars and trucks.

Evaluation Metrics: For BEV map segmentation, our primary evaluation metric is
the mean Intersection Over Union (mIoU). Because some classes may overlap, we
apply binary segmentation separately to each class and choose the highest IoU
over different thresholds. We then take the mean over all semantic classes to
produce the mIoU. This evaluation protocol was proposed in [34]. We
additionally use Panoptic Quality (PQ) [21] on KITTI-360 when evaluating
panoptic BEV segmentation.

Model | PV | X-FA | PV2BEV | X-FF |  [HTML]96bbceDrivable |  [HTML]fb9a99Ped. Cross. |  [HTML]fc4538Walkway |  [HTML]fdbf6fStop Line |  [HTML]ff7f00Carpark |  [HTML]ab9ac0Divider |  [HTML]a5eb8dmIoU | GFlops | fps
---|---|---|---|---|---|---|---|---|---|---|---|---|---
Baseline | ✗ | ✗ | ✗ | ✗ | 85.5 | 60.5 | 67.6 | 52.0 | 57.0 | 53.7 | 62.7 | 364.3 | 5.1
| ✓ | ✗ | ✗ | ✗ | 85.7 | 62.8 | 68.4 | 52.4 | 56.5 | 56.1 | 63.7 | 364.3 | 5.1
| ✗ | ✓ | ✗ | ✗ | 85.6 | 62.3 | 68.2 | 51.6 | 56.4 | 55.9 | 63.4 | 364.3 | 5.1
gray9 X-Alignview | ✓ | ✗ | ✓ | ✗ | 85.8 | 63.1 | 68.6 | 53.2 | 57.7 | 56.4 | 64.1 | 364.3 | 5.1
gray9 X-Alignlosses | ✓ | ✓ | ✓ | ✗ | 85.8 | 63.1 | 68.6 | 53.6 | 57.9 | 56.7 | 64.3 | 364.3 | 5.1
| ✗ | ✗ | ✗ | ✓ | 86.8 | 64.3 | 69.5 | 54.5 | 59.5 | 57.6 | 65.3 | 367.4 | 5.0
| ✗ | ✓ | ✗ | ✓ | 86.8 | 65.1 | 69.8 | 60.0 | 56.5 | 58.2 | 65.4 | 367.4 | 5.0
| ✓ | ✗ | ✓ | ✓ | 86.8 | 65.0 | 70.0 | 55.9 | 57.0 | 58.1 | 65.5 | 367.4 | 5.0
gray8 X-Alignall | ✓ | ✓ | ✓ | ✓ | 86.8 | 65.2 | 70.0 | 58.3 | 57.1 | 58.2 | 65.7 | 367.4 | 5.0
Table 3: Ablation study on the proposed X-Align components. In the top part,
we show the effects of our proposed losses, _i.e_., the Cross-Modal Feature
Alignment (X-FA) loss and Cross-View Segmentation Alignment (X-SA) comprised
of the PV and PV2BEV segmentation losses. In the bottom part, we further show
the improvements enabled by our Cross-Modal Feature Fusion (X-FF).

Network Architecture and Training: For evaluation on nuScenes, we build upon
BEVFusion [34] for the baseline and train our networks within mmdetection3d
[11]. In the camera branch, images are downsampled to 256$\times$704 before
going into a Swin-T [32] or ConvNeXt [33] backbone pretrained on ImageNet
[45]. The extracted features are fed into several FPN [30] layers and then
through a PV-to-BEV transformation based on LSS [41] to be mapped into the BEV
space. In the LiDAR branch, we voxelize the points with a grid size of 0.1m
and use a sparse convolution backbone [61] to extract the features, which are
then flattened onto the BEV space. Given the camera and LiDAR features in BEV
space, we utilize our proposed X-FF mechanism from Section 3.3 to fuse them.
We use the self-attention module in our main results, providing the best
accuracy-computation trade-off (see Fig. 4). The fused features are fed into a
BEV encoder and FPN layers similar to those in SECOND [61] and subsequently to
a segmentation head as in BEVFusion [34]. Since nuScenes does not provide
ground-truth PV segmentation labels, we utilize a SOTA model pre-trained on
Cityscapes to generate pseudo-labels for supervising our PV segmentation in
X-SA. Specifically, we use an HRNet-w48 [55] trained with InverseForm [4].

On KITTI-360, we take the camera-only PanopticBEV [15] as the baseline, which
we retrain using the code and hyperparameters released by the authors. Then,
we include our two proposed X-SA losses on top of this baseline to generate
the X-Alignview results on KITTI-360.

Additional hyperparameters and training details for all experiments can be
found in the Appendix.

###  4.2 Quantitative Evaluation

nuScenes Camera-LiDAR Fusion: We report segmentation results based on camera-
LiDAR fusion in the bottom section of Table 1. We evaluate in the region-bound
of [-50m, 50m]$\times$[-50m, 50m] around the ego car following the standard
evluation procedure on nuScenes [41, 67, 58, 26, 34]. We compare our complete
method X-Alignall with existing SOTA approaches on BEV segmentation and see
that X-Align significantly outperforms them. Specifically, X-Align achieves a
new record mIoU of 65.7% on nuScenes BEV segmentation and consistently
improves across all the classes, thanks to the proposed novel cross-modal and
cross-view alignment strategies. We used the Self-Attention block illustrated
in Figure 3 as our preferred X-FF strategy, as it provided the optimal trade-
off in Figure 4. For this strategy, the computational overhead (0.8%) and
increase in latency (2%) is minimal as observed in Table 1. We further show
the performance of X-Align but without using the more advanced fusion module,
_i.e_., X-Alignlosses, in the second last row of Table 1. X-Align still
significantly outperforms existing methods even without introducing additional
computational complexity during inference.

nuScenes Camera-Only: To demonstrate the efficacy of our X-SA scheme, we
evaluate an instance of X-Align, _i.e_., X-Alignview, using only the camera
branch for BEV map segmentation. The results and comparisons are shown in the
top part of Table 1. It can be seen that X-Alignview considerably outperforms
the existing best performance, achieving a record mIoU of 58.0%, surpassing
the prior camera-only SOTA by 1.4 points in mIoU. This shows the benefit of
X-SA, which enhances the intermediate semantic features and the PV-to-BEV
transformation without incurring computational overhead at inference.

KITTI-360 Camera-Only: We present additional camera-only results on KITTI-360
in Table 2, for the task of panoptic BEV segmentation. In this case, we use
PanopticBEV [15] as our baseline. Additionally, we include our two novel X-SA
losses to train our X-Alignview model.111The baseline scores are obtained by
training PanopticBEV using the authors’ code in https://github.com/robot-
learning-freiburg/PanopticBEV. It can be seen that our proposed approach
improves the baseline in terms of both mIoU and PQ scores.

Overall, our proposed X-Align consistently improves upon the existing methods
across modalities and classes on both nuScenes and KITTI-360, demonstrating
the efficacy of our proposed X-FF, X-FA, and X-SA components.

Figure 4: Accuracy-Computation Analysis: We compare our proposed Cross-Modal
Feature Fusion (X-FF) designs with simply scaling up the fusion mechanism
(adopted by existing methods [34, 24]) in terms of accuracy and computation
complexity. Figure 5: Qualitative results on nuScenes. We present a sample
scene from nuScenes: a) six camera inputs, b) LiDAR scan, c) ground-truth BEV
segmentation map, d) baseline BEV segmentation, e) BEV segmentation using
X-Alignview, and d) BEV segmentation X-Alignall. We observe that the baseline
model prediction is highly erroneous in the region highlighted in green. We
highlight this region of interest in the input views as well. By using the two
X-SA losses, X-Alignview can already correct substantial errors in the
baseline prediction, and the X-Alignall model further improves accuracy.

###  4.3 Ablation Study

We conduct an ablation study on the different X-Align components and summarize
our results in Table 3. We evaluate model variants using different
combinations of our proposed novel losses, including Cross-Modal Feature
Alignment (X-FA) loss and our Cross-View Segmentation Alignment (X-SA)
comprised of our PV and PV2BEV losses. Additionally, we investigate the effect
of our Cross-Modal Feature Fusion (X-FF).

Our X-Alignview variant, leveraging both PV and PV2BEV losses, improves the
mIoU from 62.7% to 64.1% when compared to the baseline. Specifically, the PV
loss alone contributes to a 1-point mIoU improvement. When using the X-FA
loss, we increase the baseline mIoU from 62.7% to 63.4%. Finally, adding all
the losses together, our X-Alignlosses variant boosts the mIoU score to 64.3%,
significantly surpassing the baseline’s mIoU. Notably, these improvements rely
on new training losses and have no computational overhead during inference.

Next, we study the effect of our X-FF module. Without any new losses, X-FF
enables a 2.6-point mIoU improvement over the baseline (with a minor increase
of computation, cf. Table 3 and Fig. 4). This shows that simple concatenation
is a key limitation in the baseline, which fails to fuse the camera and LiDAR
features properly. Finally, using all our novel loss functions together with
the X-FF module, we arrive at the full X-Alignall model, which achieves the
overall best performance of 65.7% mIoU, significantly higher than the
baseline’s  62.7% mIoU. Our extensive ablation study results show that each of
our proposed components in X-Align provides a meaningful contribution to
improving the SOTA BEV segmentation performance.

###  4.4 Accuracy-Computation Analysis

In Fig. 4, we report the accuracy-computation trade-off by utilizing our
different X-FF fusion strategies, including self-attention, spatial-channel
attention, and pose-driven deformable convolution (DCNv2). It can be seen that
when using spatial-channel attention, we achieve the highest accuracy
improvement at a higher computational cost, while pose-driven DCNv2 introduces
the least amount of additional cost but provides less performance gain. Using
self-attention, on the other hand, provides the best trade-off between
performance and complexity.

We further compare by naively scaling up the complexity of the baseline
fusion, _e.g_., by adding more layers and channels in the convolution blocks,
shown by the blue curve. It can be seen that the baseline’s performance
saturates, and all our proposed fusion methods achieve better trade-offs as
compared to the baseline. This again verifies that the baseline fusion using
simple concatenation and convolutions does not provide the suitable capacity
for the model to align and aggregate multi-modal features.

###  4.5 Qualitative Results

In Fig. 5, we present qualitative results on a sample test scene from
nuScenes, showing both LiDAR and camera inputs. We compare the BEV
segmentation maps of different models, including the baseline, X-Alignview
(only using the two X-SA losses), and the full X-Align, _i.e_., X-Alignall.

In this scene, the baseline wrongly predicts the building in the NW image as
part of a road in the BEV segmentation output due to the inaccurate PV-to-BEV
transformation, cf. Fig. 5(d). Since the building is not captured in the LiDAR
scan (see Fig. 5(b)), the LiDAR branch also cannot correct the camera
projection later in the fusion. However, by utilizing our Cross-View
Segmentation Alignment (X-SA), this erroneous projection can be largely
rectified, as shown in Fig. 5(e). The remnants of this error are then
completely removed when we apply our proposed alignment and fusion schemes,
X-FA and X-FF, which enables proper fusion of the visual information from the
camera and the geometric information from the LiDAR. We can see in Fig. 5(f)
that our full X-Align Model can accurately predict the BEV segmentation map.
We refer readers to the Appendix for more visual examples.

##  5 Conclusions

In this paper, we proposed a novel framework, X-Align, which addresses cross-
view and cross-modal alignment in BEV segmentation. It enhances the alignment
of unimodal features to aid feature fusion and the alignment between
perspective view and bird’s-eye-view representations. Our experiments show
that X-Align improves performance on nuScenes and KITTI-360 datasets, in
particular outperforming previous SOTA by 3 mIoU points on nuScenes. We also
verified the effectiveness of the X-Align components via an extensive ablation
study. As part of future work, we believe that X-Align can further benefit
other multi-modal perception tasks.

## References

  * [1] Xuyang Bai, Zeyu Hu, Xinge Zhu, Qingqiu Huang, Yilun Chen, Hongbo Fu, and Chiew-Lan Tai.  Transfusion: Robust Lidar-Camera Fusion for 3D Object Detection with Transformers.  In Proc. of CVPR, pages 1090–1099, 2022.
  * [2] Shubhankar Borse, Hong Cai, Yizhe Zhang, and Fatih Porikli.  Hs3: Learning with proper task complexity in hierarchically supervised semantic segmentation.  2021\.
  * [3] Shubhankar Borse, Hyojin Park, Hong Cai, Debasmit Das, Risheek Garrepalli, and Fatih Porikli.  Panoptic, instance and semantic relations: A relational context encoder to enhance panoptic segmentation.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1269–1279, 2022.
  * [4] Shubhankar Borse, Ying Wang, Yizhe Zhang, and Fatih Porikli.  Inverseform: A loss function for structured boundary-aware segmentation.  In Proc. of CVPR, pages 5901–5911, 2021.
  * [5] Holger Caesar, Varun Bankiti, Alex H Lang, Sourabh Vora, Venice Erin Liong, Qiang Xu, Anush Krishnan, Yu Pan, Giancarlo Baldan, and Oscar Beijbom.  nuScenes: A Multimodal Dataset for Autonomous Driving.  In Proc. of CVPR, pages 11621–11631, 2020.
  * [6] Xuanyao Chen, Tianyuan Zhang, Yue Wang, Yilun Wang, and Hang Zhao.  FUTR3D: A Unified Sensor Fusion Framework for 3D Detection.  arXiv preprint arXiv:2203.10642, 2022.
  * [7] Yukang Chen, Yanwei Li, Xiangyu Zhang, Jian Sun, and Jiaya Jia.  Focal Sparse Convolutional Networks for 3D Object Detection.  In Proc. of CVPR, pages 5428–5437, 2022.
  * [8] Zehui Chen, Zhenyu Li, Shiquan Zhang, Liangji Fang, Qinghong Jiang, Feng Zhao, Bolei Zhou, and Hang Zhao.  AutoAlign: Pixel-Instance Feature Aggregation for Multi-Modal 3D Object Detection.  arXiv preprint arXiv:2201.06493, 2022.
  * [9] Kashyap Chitta, Aditya Prakash, and Andreas Geiger.  NEAT: Neural Attention Fields for End-to-End Autonomous Driving.  In Proc. of ICCV, pages 15793–15803, 2021.
  * [10] Kashyap Chitta, Aditya Prakash, Bernhard Jaeger, Zehao Yu, Katrin Renz, and Andreas Geiger.  Transfuser: Imitation with transformer-based sensor fusion for autonomous driving.  arXiv preprint arXiv:2205.15997, 2022.
  * [11] MMDetection3D Contributors.  MMDetection3D: OpenMMLab next-generation platform for general 3D object detection.  https://github.com/open-mmlab/mmdetection3d, 2020.
  * [12] Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele.  The Cityscapes Dataset for Semantic Urban Scene Understanding.  In Proc. of CVPR, pages 3213–3223, 2016.
  * [13] Noa Garnett, Rafi Cohen, Tomer Pe’er, Roee Lahav, and Dan Levi.  3D-LaneNet: End-to-End 3D Multiple Lane Detection.  In Proc of ICCV, pages 2921–2930, 2019.
  * [14] Andreas Geiger, Philip Lenz, Christoph Stiller, and Raquel Urtasun.  Vision Meets Robotics: The KITTI Dataset.  The International Journal of Robotics Research, 32(11):1231–1237, 2013.
  * [15] Nikhil Gosala and Abhinav Valada.  Bird’s-Eye-View Panoptic Segmentation Using Monocular Frontal View Images.  IEEE RA-L, 7(2):1968–1975, 2022.
  * [16] Adam W Harley, Zhaoyuan Fang, Jie Li, Rares Ambrus, and Katerina Fragkiadaki.  A Simple Baseline for BEV Perception Without LiDAR.  arXiv preprint arXiv:2206.07959, 2022.
  * [17] Anthony Hu, Zak Murez, Nikhil Mohan, Sofía Dudas, Jeffrey Hawke, Vijay Badrinarayanan, Roberto Cipolla, and Alex Kendall.  FIERY: Future Instance Prediction in Bird’s-Eye View From Surround Monocular Cameras.  In Proc. of ICCV, pages 15273–15282, 2021.
  * [18] Hanzhe Hu, Yinbo Chen, Jiarui Xu, Shubhankar Borse, Hong Cai, Fatih Porikli, and Xiaolong Wang.  Learning implicit feature alignment function for semantic segmentation.  arXiv preprint arXiv:2206.08655, 2022.
  * [19] Junjie Huang, Guan Huang, Zheng Zhu, and Dalong Du.  BEVDet: High-performance Multi-camera 3D Object Detection in Bird-Eye-View.  arXiv preprint arXiv:2112.11790, 2021.
  * [20] Max Jaderberg, Karen Simonyan, Andrew Zisserman, et al.  Spatial Transformer Networks.  In Proc. of NIPS, pages 2017–2025, 2015.
  * [21] Alexander Kirillov, Kaiming He, Ross Girshick, Carsten Rother, and Piotr Dollár.  Panoptic segmentation.  In Proc. of CVPR, pages 9404–9413, 2019.
  * [22] Alex H Lang, Sourabh Vora, Holger Caesar, Lubing Zhou, Jiong Yang, and Oscar Beijbom.  Pointpillars: Fast encoders for object detection from point clouds.  In Proc. of CVPR, pages 12697–12705, 2019.
  * [23] Gustav Larsson, Michael Maire, and Gregory Shakhnarovich.  Fractalnet: Ultra-deep neural networks without residuals.  arXiv preprint arXiv:1605.07648, 2016.
  * [24] Qi Li, Yue Wang, Yilun Wang, and Hang Zhao.  Hdmapnet: A local semantic map learning and evaluation framework.  arXiv preprint arXiv:2107.06307, 2021.
  * [25] Yingwei Li, Adams Wei Yu, Tianjian Meng, Ben Caine, Jiquan Ngiam, Daiyi Peng, Junyang Shen, Yifeng Lu, Denny Zhou, Quoc V. Le, Alan Yuille, and Mingxing Tan.  Deepfusion: Lidar-camera deep fusion for multi-modal 3d object detection.  In Proc. of CVPR, pages 17182–17191, 2022.
  * [26] Zhiqi Li, Wenhai Wang, Hongyang Li, Enze Xie, Chonghao Sima, Tong Lu, Qiao Yu, and Jifeng Dai.  BEVFormer: Learning Bird’s-Eye-View Representation from Multi-Camera Images via Spatiotemporal Transformers.  In Proc. of ECCV, 2022.
  * [27] Ming Liang, Bin Yang, Shenlong Wang, and Raquel Urtasun.  Deep Continuous Fusion for Multi-sensor 3D Object Detection.  In Proc. of ECCV, pages 641–656, 2018.
  * [28] Tingting Liang, Hongwei Xie, Kaicheng Yu, Zhongyu Xia, Zhiwei Lin, Yongtao Wang, Tao Tang, Bing Wang, and Zhi Tang.  BEVFusion: A Simple and Robust LiDAR-Camera Fusion Framework.  arXiv preprint arXiv:2205.13790, 2022.
  * [29] Yiyi Liao, Jun Xie, and Andreas Geiger.  KITTI-360: A novel dataset and benchmarks for urban scene understanding in 2d and 3d.  Proc. of PAMI, 2022.
  * [30] Tsung-Yi Lin, Piotr Dollár, Ross Girshick, Kaiming He, Bharath Hariharan, and Serge Belongie.  Feature pyramid networks for object detection.  In Proc. of CVPR, pages 2117–2125, 2017.
  * [31] Tsung-Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár.  Focal Loss for Dense Object Detection.  In Proc. of ICCV, pages 2980–2988, 2017.
  * [32] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo.  Swin Transformer: Hierarchical Vision Transformer Using Shifted Windows.  In Proc. of ICCV, pages 10012–10022, 2021.
  * [33] Zhuang Liu, Hanzi Mao, Chao-Yuan Wu, Christoph Feichtenhofer, Trevor Darrell, and Saining Xie.  A ConvNet for the 2020s.  In Proc. of CVPR, pages 11976–11986, 2022.
  * [34] Zhijian Liu, Haotian Tang, Alexander Amini, Xinyu Yang, Huizi Mao, Daniela Rus, and Song Han.  BEVFusion: Multi-Task Multi-Sensor Fusion with Unified Bird’s-Eye View Representation.  arXiv preprint arXiv:2205.13542, 2022.
  * [35] Abdelhak Loukkal, Yves Grandvalet, Tom Drummond, and You Li.  Driving Among Flatmobiles: Bird-Eye-View Occupancy Grids From a Monocular Camera for Holistic Trajectory Planning.  In Proc. of WACV, pages 51–60, 2021.
  * [36] Chenyang Lu, Marinus Jacobus Gerardus van de Molengraft, and Gijs Dubbelman.  Monocular semantic occupancy grid mapping with convolutional variational encoder–decoder networks.  IEEE RA-L, 4(2):445–452, 2019.
  * [37] Muhammad Maaz, Abdelrahman Shaker, Hisham Cholakkal, Salman Khan, Syed Waqas Zamir, Rao Muhammad Anwer, and Fahad Shahbaz Khan.  EdgeNeXt: Efficiently Amalgamated CNN-Transformer Architecture for Mobile Vision Applications.  arXiv preprint arXiv:2206.10589, 2022.
  * [38] Ramin Nabati and Hairong Qi.  CenterFusion: Center-Based Radar and Camera Fusion for 3D Object Detection.  In Proc. of WACV, pages 1527–1536, 2021.
  * [39] Bowen Pan, Jiankai Sun, Ho Yin Tiga Leung, Alex Andonian, and Bolei Zhou.  Cross-view Semantic Segmentation for Sensing Surroundings.  IEEE Robotics and Automation Letters, 5(3):4867–4873, 2020.
  * [40] Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer.  Automatic differentiation in pytorch.  2017\.
  * [41] Jonah Philion and Sanja Fidler.  Lift, Splat, Shoot: Encoding Images from Arbitrary Camera Rigs by Implicitly Unprojecting to 3D.  In Proc. of ECCV, pages 194–210, 2020.
  * [42] Charles R. Qi, Wei Liu, Chenxia Wu, Hao Su, and Leonidas J. Guibas.  Frustum PointNets for 3D Object Detection From RGB-D Data.  In Proc. of CVPR, pages 918–927, 2018.
  * [43] Thomas Roddick and Roberto Cipolla.  Predicting Semantic Map Representations From Images Using Pyramid Occupancy Networks.  In Proc. of CVPR, pages 11138–11147, 2020.
  * [44] Thomas Roddick, Alex Kendall, and Roberto Cipolla.  Orthographic Feature Transform for Monocular 3D Object Detection.  In Proc. of BMVC, pages 1–10, 2018.
  * [45] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei.  Imagenet Large Scale Visual Recognition Challenge.  Proc. of IJCV, 115(3):211–252, 2015.
  * [46] Avishkar Saha, Oscar Mendez, Chris Russell, and Richard Bowden.  Enabling Spatio-temporal Aggregation in Birds-Eye-View Vehicle Estimation.  In Proc. of ICRA, pages 5133–5139, 2021.
  * [47] Hao Shao, Letian Wang, Ruobing Chen, Hongsheng Li, and Yu Liu.  Safety-enhanced autonomous driving using interpretable sensor fusion transformer.  arXiv preprint arXiv:2207.14024, 2022.
  * [48] Pei Sun, Henrik Kretzschmar, Xerxes Dotiwalla, Aurelien Chouard, Vijaysai Patnaik, Paul Tsui, James Guo, Yin Zhou, Yuning Chai, Benjamin Caine, Vijay Vasudevan, Wei Han, Jiquan Ngiam, Hang Zhao, and Aleksei Timofeev.  Scalability in Perception for Autonomous Driving: Waymo Open Dataset.  In Proc. of CVPR, 2020.
  * [49] Mingxing Tan, Ruoming Pang, and Quoc V Le.  Efficientdet: Scalable and efficient object detection.  In Proc. of CVPR, pages 10781–10790, 2020.
  * [50] Ömer Şahin Taş, Florian Kuhnt, J Marius Zöllner, and Christoph Stiller.  Functional System Architectures Towards Fully Automated Driving.  In Proc. of IV, pages 304–309, 2016.
  * [51] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin.  Attention is All You Need.  In Proc. of NIPS, pages 5998–6008, Dec. 2017.
  * [52] Sourabh Vora, Alex H Lang, Bassam Helou, and Oscar Beijbom.  Pointpainting: Sequential Fusion for 3D Object Detection.  In Proc. of CVPR, pages 4604–4612, 2020.
  * [53] Chunwei Wang, Chao Ma, Ming Zhu, and Xiaokang Yang.  A Geometric Approach to Obtain a Bird’s Eye View From an Image.  In Proc of ICCV - Workshops, pages 1–10, 2019.
  * [54] Chunwei Wang, Chao Ma, Ming Zhu, and Xiaokang Yang.  PointAugmenting: Cross-Modal Augmentation for 3D Object Detection.  In Proc of CVPR, pages 11794–11803, 2021.
  * [55] Jingdong Wang, Ke Sun, Tianheng Cheng, Borui Jiang, Chaorui Deng, Yang Zhao, Dong Liu, Yadong Mu, Mingkui Tan, Xinggang Wang, et al.  Deep High-resolution Representation Learning for Visual Recognition.  Proc. of PAMI, 43(10):3349–3364, 2020.
  * [56] Zhixin Wang and Kui Jia.  Frustum ConvNet: Sliding Frustums to Aggregate Local Point-Wise Features for Amodal 3D Object Detection.  In Proc. of IROS, pages 1742–1749, 2019.
  * [57] Pengxiang Wu, Siheng Chen, and Dimitris N. Metaxas.  MotionNet: Joint Perception and Motion Prediction for Autonomous Driving Based on Bird’s Eye View Maps.  In Proc. of CVPR, pages 11385–11395, 2020.
  * [58] Enze Xie, Zhiding Yu, Daquan Zhou, Jonah Philion, Anima Anandkumar, Sanja Fidler, Ping Luo, and Jose M. Alvarez.  M2̂BEV: Multi-Camera Joint 3D Detection and Segmentation with Unified Birds-Eye View Representation.  arXiv preprint arXiv:2204.05088, 2022.
  * [59] Runsheng Xu, Zhengzhong Tu, Hao Xiang, Wei Shao, Bolei Zhou, and Jiaqi Ma.  CoBEVT: Cooperative Bird’s Eye View Semantic Segmentation with Sparse Transformers.  arXiv preprint arXiv:2207.02202, 2022.
  * [60] Shaoqing Xu, Dingfu Zhou, Jin Fang, Junbo Yin, Zhou Bin, and Liangjun Zhang.  FusionPainting: Multimodal Fusion with Adaptive Attention for 3D Object Detection.  In Proc. of ITSC, pages 3047–3054, 2021.
  * [61] Yan Yan, Yuxing Mao, and Bo Li.  Second: Sparsely Embedded Convolutional Detection.  Sensors, 18(10):3337, 2018.
  * [62] Tianwei Yin, Xingyi Zhou, and Philipp Krahenbuhl.  Center-based 3D Object Detection and Tracking.  In Proc. of CVPR, pages 11784–11793, 2021.
  * [63] Tianwei Yin, Xingyi Zhou, and Philipp Krähenbühl.  Multimodal Virtual Point 3D Detection.  In Proc. of NeurIPS, pages 16494–16507, 2021.
  * [64] Yizhe Zhang, Shubhankar Borse, Hong Cai, and Fatih Porikli.  Auxadapt: Stable and efficient test-time adaptation for temporally consistent video semantic segmentation.  In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 2339–2348, 2022.
  * [65] Yizhe Zhang, Shubhankar Borse, Hong Cai, Ying Wang, Ning Bi, Xiaoyun Jiang, and Fatih Porikli.  Perceptual consistency in video segmentation.  In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 2564–2573, 2022.
  * [66] Yunpeng Zhang, Zheng Zhu, Wenzhao Zheng, Junjie Huang, Guan Huang, Jie Zhou, and Jiwen Lu.  BEVerse: Unified Perception and Prediction in Birds-Eye-View for Vision-Centric Autonomous Driving.  arXiv preprint arXiv:2205.09743, 2022.
  * [67] Brady Zhou and Philipp Krähenbühl.  Cross-View Transformers for Real-Time Map-View Semantic Segmentation.  In Proc. of CVPR, pages 13760–13769, 2022.
  * [68] Minghan Zhu, Songan Zhang, Yuanxin Zhong, Pingping Lu, Huei Peng, and John Lenneman.  Monocular 3D Vehicle Detection Using Uncalibrated Traffic Cameras through Homography.  In Proc. of IROS, pages 3814–3821, 2021.
  * [69] Xizhou Zhu, Han Hu, Stephen Lin, and Jifeng Dai.  Deformable Convnets V2: More Deformable, Better Results.  In Proc. of CVPR, pages 9308–9316, 2019.

##  Appendix A Introduction

As part of the supplementary materials for this paper, we present our hyper-
parameters and show more visual and quantitative results as an extension to
the ones shown in the paper. The supplementary materials contain:

  * •

An ablation study analyzing the performance of X-Align on various backbones.

  * •

An ablation study to measure the adaptability of X-Align to camera noise.

  * •

An ablation study to measure the impact of X-Align on different driving
conditions and its adaptability to noise.

  * •

Implementation details and training hyper-parameters for all our experiments.

  * •

Qualitative results on nuScenes as an addition to the example shown in Figure
5 of the paper, and visual comparison with the baseline network.

##  Appendix B Varying Backbones

Model | Encoder | Modality | mIoU
---|---|---|---
BEVFusion [34] | Swin-T | Camera, LiDAR | 62.7
gray9 X-Alignview | Swin-T | Camera, LiDAR | 64.1
BEVFusion | ConvneXt-T | Camera, LiDAR | 62.1
gray9 X-Alignview | ConvneXt-T | Camera, LiDAR | 63.8
BEVFusion | ConvneXt-S | Camera, LiDAR | 63.9
gray9 X-Alignview | ConvneXt-S | Camera, LiDAR | 64.7
Table B.1: Quantitative evaluation on nuScenes in terms of mIoU, varying the
camera encoder.

To verify the generalizability of our X-Align method, we present results for
three different encoders, _i.e_., SWin-T, ConvneXt-T, and ConvneXt-S, in Table
B.1. For this ablation study, we choose our X-Alignview variant because this
shows that a given model can be improved without adding additional
computational complexity during inference. Compared to the current state-of-
the-art result from BEVFusion [34], we can improve their result from 62.7 to
64.1 in terms of mIoU. Similar improvements can be observed for our
additionally investigated backbones ConvneXt-T and ConvneXt-S. These results
show that our method generalizes well to different backbones without
hyperparameter tuning.

##  Appendix C Adaptability to Noise

Model | Encoder | Modality | $\sigma=0.0$ | $\sigma=0.05$ | $\sigma=0.075$ | $\sigma=0.1$
---|---|---|---|---|---|---
BEVFusion [34] | Swin-T | Camera | 56.6 | 52.7 | 47.2 | 40.7
gray9 X-Alignview | Swin-T | Camera | 58.0 | 55.2 | 51.4 | 45.6
BEVFusion [34] | Swin-T | Camera, Lidar | 62.7 | 59.2 | 54.8 | 48.9
gray9 X-Alignview | Swin-T | Camera, Lidar | 64.1 | 62.1 | 58.3 | 54.1
gray8 X-Alignall | Swin-T | Camera, Lidar | 65.7 | 64.3 | 62.6 | 59.6
Table C.1: Quantitative evaluation on nuScenes, adding gaussian noise to the
input images. We observe that X-Align methods provide more consistent results
as they are lesser variant to noise.

Another interesting property of deep neural network-based methods is their
susceptibility to input perturbations because it gives insights into their
robustness in real environments. Therefore, we add Gaussian noise to the input
camera images and observe how the performance degrades in Table C.1. The noise
is zero-mean normalized, and $\sigma$ in Table C.1 is the standard deviation
of the Gaussian noise. From the table, we can deduce two main observations:
First, the performance of X-Align methods under input perturbations is
relatively stable compared to the baseline. We attribute this to the fact that
the input camera noise will make the extracted camera feature less reliable.
Due to our robust attention-based cross-modal feature fusion (X-FF), the
method can still correct the camera features using LiDAR predictions. Our
second interesting observation concerns the comparison to the baseline
BEVFusion. We observe that our performance loss at a higher noise-variance is
much smaller than for BEVFusion, e.g., the performance of our method
X-Alignall at $\sigma=0.1$ drops from 65.7 to 59.6. In contrast, BEVFusion
drops from 62.7 to 48.9. This shows superior properties of our method in terms
of robustness.

##  Appendix D Adverse Weather Conditions

Model | Encoder | Modality | nuScenes-Night | nuScenes-Rain | nuScenes
---|---|---|---|---|---
BEVFusion [34] | Swin-T | Camera | 30.8 | 50.5 | 56.6
gray9 X-Alignview | Swin-T | Camera | 33.1 | 51.1 | 58.0
BEVFusion [34] | Swin-T | Camera, LiDAR | 43.6 | 55.9 | 62.7
gray9 X-Alignview | Swin-T | Camera, LiDAR | 44.5 | 56.3 | 64.3
gray8 X-Alignall | Swin-T | Camera, LiDAR | 46.1 | 57.8 | 65.7
Table D.1: Quantitative evaluation on nuScenes-rainy and nuScenes-night. We
observe that X-Align provides consistent improvements in adverse weather
conditions.

To further investigate our method’s robustness, we show results of how our
method performs in adverse weather conditions in Table D.1. For this
experiment, we use the metadata provided by nuScenes, which indicates the
weather and light conditions of the recorded scene. With this information, we
filter the validation set to obtain two splits containing all images recorded
at night and all images recorded during rainy conditions. Our results show
consistent improvement in these adverse conditions by X-Alignview as well as
X-Alignall. This shows that our method improves on comparably easy samples and
difficult ones, which is an important property for future deployment of our
method.

##  Appendix E Training Details and Hyperparameter Analysis

In this subsection, we provide the hyper-parameters and training details for
all our experiments in the paper. All our models are trained using Pytorch
[40] on 4 Nvidia Tesla A100 GPUs.

nuScenes experiments: To reproduce the BEVFusion [34] baseline model, we adapt
the code provided by their authors222https://github.com/mit-han-lab/bevfusion
into the mmdetection3d [11] framework. This is because they do not provide
training code in the current version. In the camera pipeline, the images are
downsampled into a size of $256\times 704$ before being passed through a
Swin-T [32] or ConvNext [33] backbone pretrained on ImageNet [45]. The
intermediate features extracted from the camera are passed through a set of
FPN [30] layers to retain the salient low-level encoder features. These are
then passed to View-Transformers based on LSS [41]. The baseline is trained
using their reported hyper-parameters [34], with a learning schedule of 20
epochs and a cyclic learning rate, starting for $1e^{-4}$ and performing a
single cycle with target ratios $\\{10,1e^{-4}\\}$ and a step of $0.4$. For
our experiments, we implement our proposed blocks:

  * •

X-FF Modules: For the X-FF modules described in Section 3.3 of the paper, we
switch the naive convolutional fuser in the baseline model. To implement the
Self-Attention fuser, we tokenize the inputs into patches of size $3\times 3$
and a stride of 2. This step is followed by a Multi-Head-Self-Attention block
as described in [51], containing 8 heads and an embedding dimension of 256. To
implement the Spatial-Channel Attention module, we use the Split-depth
Transpose Attention (SDTA) block from EdgeNext [37].We use an embedding
dimension of 256, 2 scales, and 8 heads. We also use DropPath [23] of $0.1$. A
deconvolution block follows the SDTA fusion module to output 256 channels and
the fused feature resolution. Finally, to implement the pose-driven deformable
convolutional operation, we pass the input pose through a 2-layer MLP network,
whose output can is interpolated to the feature size. This pose channel is
concatenated with the two input modalities, then passes through a
convolutional block to obtain $18$ offset channels, input into a DCNv2 [69]
block of kernel size $3\times 3$ The output of the DCNv2 block is our fused
feature dimension.

  * •

X-FA Loss: As explained in Section 3.4 of the paper, we use cosine similarity
as a loss function to model the X-FA similarity. Using a sparse hyper-
parameter search, we set the best value of $\gamma_{2}$ in Equation (2) of the
main paper as $-0.002$. We reduce its negative value because we want to
increase the cosine similarity.

  * •

X-SA Module: As explained in Section 3.5 of the paper, we utilize a
perspective decoder to generate 2D perspective view semantic labels. The
decoder has two Convolution-BN2D-ReLU sequences with a hidden dimension of
$256$ channels, generating semantic probabilities at the output. As nuScenes
[5] does not contain ground truth segmentation labels; we make use of a state-
of-the-art (SOTA) model pre-trained on Cityscapes to generate pseudo labels
$\bm{y}^{\mathrm{PV}}$. This is an HRNet-w48 [55] checkpoint, trained with the
InverseForm [4] loss, as made public by the authors in their
codebase.333https://github.com/Qualcomm-AI-research/InverseForm We pick
$\gamma_{3}$ in Equation (2) of the main paper as $0.1$ following a sparse
hyper-parameter search. Once we have the 2D perspective features, we share
parameters with the LSS Transform from earlier to splat the probability maps
to BEV space. We use convolution blocks for channel aggregation at the input
and output and interpolate to equalize dimensions. The output is then
supervised with BEV GT as shown in Figure 3 of the paper. The loss weight
$\gamma_{4}$ is tuned to $0.1$ by observing the total loss value and setting
the weight to produce $\sim\\!20\%$ of the total loss.

KITTI360 experiments: To reproduce the PanopticBEV [15] scores, we use the
code made public by the authors and their default
settings.444https://github.com/robot-learning-freiburg/PanopticBEV The model
consists of an EfficientDet-d3 [49] camera encoder, which encodes multi-
resolution features. Next, a multi-scale transformer converts the perspective
view to BEV features. This is followed by decoders for both instance and
semantics. Finally, their outputs are combined to generate panoptic
predictions. The baseline uploaded by the authors reaches a lower value than
what was obtained in their paper. For our experiments with the X-SA module, we
added a 2D Perspective decoder to the multi-scale encoded features to predict
semantic labels in the perspective view. As KITTI360 contains 2d semantic
labels, we use them to supervise this decoder. Then, we reuse the view
transformer to map predictions to BEV space. We supervise our X-SA branch with
loss weights $\gamma_{3}$ set at $0.5$ and $\gamma_{4}$ set at $0.1$, adding
them to the total loss of the PanopticBEV baseline.

##  Appendix F Qualitative Analysis

In this Section, we present more sample scenes from nuScenes, as an extension
to the one shown in Figure 5 of the main paper. Each scene consists of 5
parts: a) six surround camera inputs b) LiDAR scan, c) ground-truth BEV
segmentation map, d) baseline BEV segmentation, e) BEV segmentation using
X-Alignview, and d) BEV segmentation X-Alignall. For each scenario, we observe
that the baseline model prediction is highly erroneous in the region
highlighted in green. We highlight this region of interest in the input views
as well. By using our proposed X-SA losses, X-Alignview can already correct
substantial errors in the baseline prediction, and the X-Alignall model
further improves accuracy.

Figure F.1: Scenario 1 on nuScenes: We present a scene where multiple road
occlusions are apparent in the N and NW camera images, which were caused by
pedestrians. The baseline model fails to fill in the gaps properly through the
LiDAR-camera fusion, producing an entangled segmentation representation of the
intersecting roads. Using the two X-SA losses, X-Align${}_{\text{view}}$
improves the segmentation map’s prediction. By adding all the components,
X-Align${}_{\text{all}}$ can address the occlusions and produce a more refined
semantic representation of the scene. Figure F.2: Scenario 2 on nuScenes: In
this scene from nuScenes, the road boundaries in the north west direction of
the vehicle appear vague in the LiDAR point cloud’s green box area. However,
in the N and NW camera images, they are apparent. The baseline model’s
boundary prediction is faulty due to misaligned camera features. Using the two
X-SA losses, X-Align${}_{\text{view}}$ rectifies some of the camera features.
By adding all the components, X-Align${}_{\text{all}}$ can predict a map where
the road boundaries are properly segmented. Figure F.3: Scenario 3 on
nuScenes: We present a scene where patterned occlusions like fences are
evident in the N and NW camera images. Due to these repetitive occlusions, the
baseline model has difficulty producing a clear segmentation map of the
intersection. By using the two X-SA losses, X-Align${}_{\text{view}}$ can
extract more BEV-segmentation-oriented features from the images. By adding all
the components, X-Align${}_{\text{all}}$ can properly rectify the camera’s
incomplete view of the road and align it properly with the LiDAR’s features,
producing a clear segmentation map of the road as shown in (f). Figure F.4:
Scenario 4 on nuScenes: We present a scene from nuScenes, where the back
intersection is dimly lighted with slightly occluded road boundaries, as shown
in the S camera image. The LiDAR point cloud input lacks salient features
capturing the road boundaries. We show the baseline’s erroneous boundary
prediction due to their simple concatenation-based fusion between the two
modalities. We also show that by using two X-SA losses,
X-Align${}_{\text{view}}$ can have a more accurate prediction of the road
boundaries, and by additionally using X-FF and X-FA, X-Align${}_{\text{all}}$
can output an accurate segmentation map of the intersection.