# DaF-BEVSeg: Distortion-aware Fisheye Camera based
Bird’s Eye View Segmentation with Occlusion Reasoning

Senthil Yogamani1, David Unger2, Venkatraman Narayanan3 and Varun Ravi Kumar3
1Automated Driving, QT Technologies Ireland Limited
2Internship at Qualcomm Technologies International GmbH
3Automated Driving, Qualcomm Technologies, Inc.

###### Abstract

Semantic segmentation is an effective way to perform scene understanding.
Recently, segmentation in 3D Bird’s Eye View (BEV) space has become popular as
its directly used by drive policy. However, there is limited work on BEV
segmentation for surround-view fisheye cameras, commonly used in commercial
vehicles. As this task has no real-world public dataset and existing synthetic
datasets do not handle amodal regions due to occlusion, we create a synthetic
dataset using the Cognata simulator comprising diverse road types, weather,
and lighting conditions. We generalize the BEV segmentation to work with any
camera model; this is useful for mixing diverse cameras. We implement a
baseline by applying cylindrical rectification on the fisheye images and using
a standard LSS-based BEV segmentation model. We demonstrate that we can
achieve better performance without undistortion, which has the adverse effects
of increased runtime due to pre-processing, reduced field-of-view, and
resampling artifacts. Further, we introduce a distortion-aware learnable BEV
pooling strategy that is more effective for the fisheye cameras. We extend the
model with an occlusion reasoning module, which is critical for estimating in
BEV space. Qualitative performance of DaF-BEVSeg is showcased in the video at
https://streamable.com/ge4v51.

##  I Introduction

In the pursuit of advancing Automated Driving (AD) and Robotics, Bird’s Eye
View (BEV) grid generation has emerged as a critical task, aiming to provide a
comprehensive understanding of the surrounding environment. As such, several
state-of-the-art works on AD tasks like 3D Object Detection [1, 2, 3, 4, 5, 6,
7], Semantic Segmentation [8, 9, 10, 11], Multi-task learning [12, 13, 14, 15,
16], operates on a BEV grid. Semantic BEV segmentation is the task of creating
a 2D grid that rasterizes the real world into equal-sized grid cells that each
contain semantic content about the object present at the corresponding real-
world location. An example of what the semantic BEV grid looks like can be
seen in Figure 1.

Enabling safe automated driving requires a diverse sensor set containing many
different cameras. Given the large Field of View (FOV) of the fisheye (FE)
Cameras, they are quickly becoming ubiquitous in the AD sensor setup. However,
academic research on automated driving focuses heavily on pinhole cameras
because of their predominance in open-source automated driving datasets [17,
18]. Furthermore, fisheye cameras introduce more substantial distortions to
the images. Hence, fisheye cameras require a different camera projection
model, and it is non-trivial to extend standard state-of-the-art works on
pinhole cameras to fisheye cameras.

Four wide angle fisheye cameras form the basic sensor set in automotive
systems for near-field sensing in combination with Ultrasonics sensor [19,
20]. The limited literature in various fisheye perception tasks such as object
detection [21, 22], depth estimation [23, 24] , localization [25, 26] , motion
estimation [27, 28, 29], semantic segmentation [30, 31], blockage and adverse
weather detection [32, 33], multi-task learning [34, 35, 36] and near-field
perception systems [37, 38, 19, 39] indicate that special attention and design
is needed to handle the large radial distortion.

The work on Lift-Splat-Shoot (LSS) [5] forms the baseline for generating a BEV
map from standard pinhole cameras. Many state-of-the-art works adapt LSS for
downstream AD tasks ([8, 12, 1, 40]). While pinhole camera-based BEV map
generation and multi-modal sensor fusion have been studied extensively, they
still need to be explored for fisheye cameras.

This work proposes a novel approach to extend LSS to fisheye cameras. It
further extends our capability to perform multi-sensor fusion in BEV space
with fisheye cameras. In addition, we propose a state-of-the-art architecture
that performs Semantic Segmentation in BEV Space for fisheye cameras. We
incorporated a multi-task head that makes classification and occlusion
predictions for each cell instead of a simple pixel-wise classification. It
enables us to circumvent network hallucinations in occluded regions, which is
predominant in fisheye camera images due to other non-linear optical
characteristics. Further, we also developed an efficient pooling mechanism to
improve the BEV grid generation from multiple sensors that cater to the
fisheye lens characteristics. We also perform an extensive evaluation of our
architecture on synthetic datasets.

|
---|---
Figure 1: Semantic encoding of the BEV space. Left: BEV camera image. Right:
Scene’s semantic representation.

Our main contributions include:

  * •

Creation of a fisheye BEV segmentation dataset with occlusion masks using a
commercial-grade simulator.

  * •

Design of a novel distortion-aware learnable pooling strategy using camera
intrinsics for adaptation.

  * •

Generalized framework for BEV semantic segmentation from raw images supporting
various camera models.

  * •

A end-to-end multi-task model that provides semantic classes and occlusion
reasoning in ambiguous scenarios.

##  II Related Work

###  II-A BEV Generation for Automated Driving Tasks

Recent literature explores BEV generation from camera images in 3 broad
approaches. The first category of BEV transformation architectures involves
Multilayer Perceptron (MLP)-based methods, where the network learns the
transformation from image to BEV space within its connections [41, 42, 43].
Attention-based methods use cross-attention, with queries in the BEV space for
keys and values in the image space. TIIM [44] leverages the symmetry between
an image column and a ray in BEV space, while CVT [45] treats the entire
image-to-BEV transformation as a translation problem. While attention-based
approaches have shown promise, they are computationally expensive and may not
be suitable for fisheye cameras due to specific transformations and limited
computational constraints.

Lastly, geometry-based methods utilize camera parameters such as position,
orientation, focal length, and sensor-shift. Inverse Perspective Mapping (IPM)
[46, 47] assumes a flat ground surface, projecting each pixel into BEV space.
While simpler, this approach introduces distortions, making it less suitable
for real-world scenarios. Some methods [48] use generative methods to reduce
the distortions. Cam2BEV [49] employs IPM but compensates for errors through
learning in the BEV encoder. Another method is to use depth estimates for
projecting image features accurately into 3D space [5, 1].

###  II-B Semantic Segmentation on Generic Camera

As mentioned in Section I, standard vision pipelines apply only for pinhole
cameras and do not scale well for generic camera applications such as fisheye
cameras. NVAutoNet [16] proposes a framework to process generic camera images
for automated driving perception applications using an MLP-based BEV
generation. However, semantic segmentation is not considered in the approach.
F2BEV [50] proposes a distortion-aware spatial cross-attention, using
normalized undistorted images in conjunction with distorted images.
FisheyePixPro [51] proposes a 2D self-supervised semantic segmentation based
on contrastive learning. Other fisheye camera-specific approaches propose
overlapping pyramid pooling [52] and deformable convolution [53] for semantic
segmentation. Dense depth/structure estimation methods on fisheye cameras,
like [54, 55] utilize specific camera projection models or use camera
parameters as input [56]

##  III Method

This section presents our approach to learning bird’s-eye-view representations
of scenes from image data captured by a surround fisheye camera rig. Our goal
is, given $n$ surround-view fisheye images ${X_{k}\in R^{3\times H\times W}}$
each with an extrinsic matrix $E_{k}\in R^{3\times 4}$ and an intrinsic matrix
$I_{k}\in R^{3\times 3}$, and we seek to find a rasterized representation of
the scene in the BEV coordinate frame $y\in R^{C\times X\times Y}$ with
semantic information for each cell. The end-to-end architecture of DaF-
BEVSegis outlined in Figure 3.

###  III-A Fisheye Camera BEV features

First, similar to the LSS [5] approach, we use the camera parameters to
project the image features from the image encoder into the 3D world. The
camera parameters map the image feature to the corresponding ray in the BEV
map, while the estimated depth determines the distance. For this to work in
Fisheye cameras, we convert the camera features into a direction vector for
each feature projected into 3D space. The camera direction vector is moved to
the vehicle coordinate system using projection geometry. Firstly, the camera
features in pixel coordinates, $u,v$, are converted to camera co-ordinates,
$x,y$, using the camera intrinsic matrix ($K$). As depicted in Equation 1,
using the Kannala-Brandt model [57], we generate spherical angles
($\theta,\varphi$) from the camera coordinate features.

$\begin{split}r=\sqrt{x^{2}+y^{2}}\\\
\theta=p_{1}.r+p_{2}.r^{2}+p_{3}.r^{3}+...+p_{9}.r^{9}\\\
\varphi=\arctan\big{(}\frac{y}{x}\big{)}\end{split}$ (1)

We use a generic equation for radial distortion angle ($\theta$) here. Our
method generalizes the inverse mappings of radial distortion models, such as
Polynomial, UCM, eUCM, Rectilinear, Stereographic, Double Sphere. based on the
derivation by [58].

Our method supports the inverse mappings of radial distortion models
summarized below:

  1. 1.

Polynomial: $r=a_{1}\theta+a_{2}\theta^{2}+a_{3}\theta^{3}+a_{4}\theta^{4}$

  2. 2.

UCM: $r=f\cdot\sin\theta/(\cos\theta+\xi)$

  3. 3.

eUCM:
$r=f\cdot\frac{\sin\theta}{\cos\theta+\alpha\left(\sqrt{\beta\cdot\sin^{2}\theta+\cos^{2}\theta}-\cos\theta\right)}$

  4. 4.

Rectilinear: $r=f\cdot\tan\theta$

  5. 5.

Stereographic: $r=2f\cdot\tan(\theta/2)$

  6. 6.

Double Sphere:
$r=f\cdot\frac{\sin\theta}{\alpha\sqrt{\sin^{2}\theta+(\xi+\cos\theta)^{2}}+(1-\alpha)(\xi+\cos\theta)}$

Where $f$ represents the focal length of the camera. The Rectilinear and
Stereographic models are unsuitable for fisheye cameras but are added for
completeness, whereas the Polynomial model is the most used. The spherical
angles ($\theta,\varphi$) are used to generate the direction vectors
($X,Y,Z$), following the equation 2. The direction vectors are multiplied with
depth estimates in a subsequent step to obtain the final 3D vectors.

$\displaystyle\begin{pmatrix}X\\\ Y\\\
Z\end{pmatrix}=r\begin{pmatrix}\cos{\varphi}.\sin{\theta}\\\
\sin{\varphi}.\sin{\theta}\\\ \cos{\theta}\end{pmatrix}$ (2)

The BEV grid is generated by Splatting [5] the depth estimates with the
direction vectors, ($X,Y,Z$). Unlike traditional cameras with a narrower field
of view, fisheye lenses capture a wide-angle view, resulting in significant
overlap between adjacent images. This overlap means that features from
multiple camera perspectives contribute to the same areas in the BEV grid, as
seen in Figure 2, creating a rich and redundant source of information. The
depth resolution of the BEV grid in the figure has been altered for
visualization. This redundancy can be leveraged to enhance the robustness and
accuracy of the BEV representation, as features from multiple viewpoints
provide complementary information about the scene. Therefore, in the fisheye
BEV grid, the challenge lies in handling distortion and effectively
integrating and leveraging the overlapping features to construct a
comprehensive representation of the environment. The architectural changes
needed for Fisheye cameras are explained in Section IV-B.

NuScenes - Pinhole |  Cognata - Fisheye
---|---
Figure 2: The visulization of BEV Grid space from each camera pixel. On the
Left is camera BEV grid sample from NuScenes dataset [59] color-coded as
FRONT, FRONT-RIGHT, BACK-RIGHT, BACK, BACK-LEFT, FRONT-LEFT. On the Right is
camera BEV grid sample from our Cognata dataset color-coded as FRONT, RIGHT,
BACK, LEFT.

###  III-B Distortion-aware Learnable Pooling

The BEV features from each camera sensor are pooled into a single BEV grid.
Most current methods use the BEV Pooling [12] strategy to associate the camera
BEV features to the BEV grid along with a symmetric function (e.g., mean, max,
and sum) to aggregate the features within each BEV grid. However, the features
in overlapping regions are treated equally and do not contextualize the sensor
parameters. Our experiments showcased that it does not translate well to
surround-view fisheye cameras because of the non-linear scaling and larger
FOV. Also, symmetric aggregation is disadvantageous when the surround fisheye
cameras do not share the same sensor configuration.

Hence, we experiment with several weighting and embedding strategies while
pooling the BEV features into a single BEV grid. Specifically, we convert the
pooling function into a learnable weighting function. The learnable pooling:

  1. 1.

primarily learns and focuses on the features relevant for an accurate BEV grid
generation and semantic segmentation.

  2. 2.

plays to the strengths of the camera sensor (within the FOV), either
implicitly or explicitly, while aggregating the camera BEV features.

  3. 3.

makes the final BEV grid resilient to differences in camera BEV features,
especially in cases of object boundaries and small objects.

In the following sections, we discuss the different BEV pooling strategies
employed.

Weighted Sum: In the first approach, represented in Equation 3, we add a
weighting parameter when summing the BEV features. The summing weights
($W_{i}\in R^{C\times X\times Y}$) are a learnable parameter for the BEV
feature ($F_{bev}^{i}\in R^{C\times X\times Y}$) from each camera sensor, that
is learned through the training process.

$F_{bev}^{total}=\sum\limits_{i=1}^{N}{W_{i}\cdot F_{bev}^{i}}$ (3)

Sensors per BEV cell: Secondly, we assign a set of learnable parameters for
each cell in the BEV grid. Specifically, we assign one parameter for each
camera contributing to the cell. These parameters weigh the features captured
by each camera within the cell. The learnable parameters are initialized as
one divided by the number of cameras projecting to the cell. This ensures that
the initial weights are evenly distributed among the contributing cameras. If
($w_{i,j}$) is the learnable weight for a single BEV grid cell, and
($N_{i,j}$) represents the number of cameras contributing to BEV grid cell
features, the weighting parameter is defined in Equation 4.

$w_{i,j}=\frac{1}{N_{i,j}}$ (4)

Camera intrinsic based embedding: Finally, we also experiment with another
pooling approach, shown in Equation 5, wherein we add a learnable embedding
$E_{i}\in R^{C\times X\times Y}$ per sensor while incorporating the sensor’s
intrinsic parameters $I_{K}^{i}\in R^{3\times 3}$. The BEV features from
individual camera sensors are also scaled with their mean $\mu_{i}$. We found
that using a learnable embedding in pooling the BEV features focuses the
network, specifically in our occlusion estimation module, thereby improving
our overall results. The results from our experiments on BEV pooling are
discussed in detail in Section V-A3.

$F_{bev}^{total}=\sum\limits_{k=1}^{N}{I_{k}\cdot(F_{bev}^{k}}-\mu_{k})+E_{k}$
(5)

###  III-C Occlusion reasoning in BEV Segmentation

BEV Segmentation task is designed to represent image information in BEV space,
focusing solely on grid cells perceivable by vehicle cameras. Occluded areas
should not contribute to predictions to avoid speculative inference. This
occlusion reasoning is particularly crucial for near-field sensing in urban
driving and parking scenarios, where occlusion is prevalent. To address this,
we propose a novel multi-task approach, which includes Occlusion reasoning,
wherein the model outputs an occlusion probability for each grid cell, a
feature included in our dataset. In synthetic engines, the BEV camera records
the class of the closest object, whereas the desired semantic BEV labels must
contain the class relevant for the ego vehicle to navigate the world. This
contrast leads to the misrepresentation of BEV semantic classes when trees or
bridges obstruct the BEV camera. On the contrary, there will also be label
mismatches when objects are occluded in the ego vehicle’s camera sensors but
not in the BEV camera.

To generate the occlusion map $p(o)$, a circular kernel is applied to the
camera BEV features to count local pixel occurrences within grid cells. The
occurrences are normalized with a threshold ($\tau$) to derive an occlusion
score $p(o)$. When vehicles are visible in multiple cameras, the entire
vehicle is designated as occupied and non-occluded in the BEV grid. This
allows the network to comprehend vehicle dimensions, including occlusion
effects. Figure 3 visualizes occluded areas as black regions. In practice, the
occupancy probability ($p^{{}^{\prime}}(o)=1-p(o)$) is used for ease of loss
assignment and visualization.

###  III-D Fisheye BEV Segmentation

Fisheye cameras are most desirable for near-field perception because of their
large FOV. Hence, we adapt the BEV segmentation to have a smaller resolution
and range. For the same reason, we incorporate a multi-task head that makes
classification and occupancy predictions for each cell. However, since heads
operate on BEV features, we keep the model architecture of heads unchanged
from state-of-the-art BEV segmentation techniques [12]. As illustrated in
Equation 6, we use weighted cross-entropy loss, based on [60], for training
the classification head. The occlusion head, depicted in Equation 7, is
trained using binary cross-entropy loss.

$\pounds_{semantic}(p,q)=-p(o)\sum_{c_{i}\in C}\alpha_{i}\cdot
p(c_{i})\cdot\log q(c_{i})$ (6)

The classification loss is a function of occupancy probability ($p(o)$). The
predicted ($p(c_{i})$) and ground truth ($q(c_{i}$) class-probabilities are
weighted by $\alpha_{i}$. The chosen weighting strategy for training is
explained in Section V-A1.

$\pounds_{occ}(p(o),p(\hat{o}))=-[p(o)\cdot\log{p(\hat{o})}+(1-p(o)\cdot\log{1-p(\hat{o})}]$
(7)

Figure 3: The network architecture converts images into image features,
transforms the features into BEV space using the calibration data, and then
calculates the semantic output in BEV space.

Where $p(o)$ is the occupancy probability predicted by the occupancy head and
$p(\hat{o})\in\\{0,1\\}$ is the ground truth occupancy label.

The combination of semantic classification loss ($\pounds_{semantic}$), and
occupancy loss ($\pounds_{occ}$), presented in Equation 8, is used to train
the BEV Segmentation model end-to-end. When combined with semantic loss, we
introduce a weighting parameter ($\lambda$) to occupancy oss. This focuses the
training on the BEV Segmentation task and helps in training stability and
convergence.

$\pounds_{total}=\pounds_{semantic}+\lambda\cdot\pounds_{occ}$ (8)

##  IV Implementation

###  IV-A Datasets

Figure 4: Row 1: Four different scenes from the Cognata training set. Rows
2-4: 12 different weather and traffic situations from the fifth training scene

For fisheye BEV segmentation, there are no real-world datasets, but there are
two possible synthetic datasets, namely SynWoodScape[61] and F2BEV (FB-SSEM
dataset) [50]. These datasets do not address the occlusion problem described
in the previous Occlusion reasoning section. Besides this issue, F2BEV is
confined to simulating just parking lots and is not suitable for general
driving. SynWoodScape has released only a small initial sample of the dataset.
Therefore, we generated a synthetic dataset using the Cognata simulation
platform. To overcome the occlusion problem, we enrich the information from
the BEV camera using additional sensor information from other vehicle cameras.
We project the perspective semantic labels from other camera sensors in the
vehicle to 3D space to rectify the BEV camera semantic labels. Our final
dataset comprises four fisheye cameras and six pinhole cameras with
$1920\times 1208$ resolution, BEV ground truth images ($400\times 400$) with
five semantic classes: invalid, vehicles, lane markings, street, and
background.

In addition, we generate an occlusion probability map indicating the
likelihood of occlusion in each grid cell. The training dataset comprises five
virtual scenes of about 90 seconds each from Pittsburgh, Munich, and HaSharon.
Each scene is simulated with different traffic, weather, and daylight
conditions. The whole dataset contains more than 12,000 different frames
corresponding to 50,000 fisheye images. Sample images from the dataset can be
seen in Figure 4.

TABLE I: Quantitative Performance of DaF-BEVSeg on different test dataset
slices. Input image size is $480\times 302$.

Method Image Type Dataset [HTML]a5eb8dIoU [HTML]00b050mIoU
[HTML]fb9a99Occlusions [HTML]96bbceVehicles [HTML]fc4538Markings
[HTML]fdbf6fStreet [HTML]ab9ac0Background LSS [5] Cyl. Rectified gray95Easy
gray950.659 gray95- gray950.607 gray950.379 gray950.773 gray950.875
gray9Medium gray90.540 gray9- gray90.539 gray90.214 gray90.645 gray90.763
gray85Hard gray850.314 gray85- gray850.317 gray850.114 gray850.523 gray850.303
DaF-BEVSeg Raw Distorted gray95Easy gray950.796 gray950.815 gray950.776
gray950.517 gray950.895 gray950.978 gray9Medium gray90.690 gray90.682
gray90.764 gray90.364 gray90.858 gray90.782 gray85Hard gray850.466 gray850.666
gray850.464 gray850.176 gray850.572 gray850.449

TABLE II: Ablation Study of DaF-BEVSeg on Medium test dataset. Under Class
Weight ablation, X-X-X-X represents weights for Vehicles, Lane Markings,
Street and Background .

[HTML]96bbceApproaches [HTML]a5eb8dIoU [HTML]00b050mIoU [HTML]fb9a99Occlusion
[HTML]96bbceVehicles [HTML]fc4538Markings [HTML]fdbf6fStreet
[HTML]ab9ac0Background [HTML]FFCCC9Class Weights 13-3-1-1 0.658 0.687 0.716
0.346 0.803 0.736 1-1-1-1 0.690 0.682 0.764 0.364 0.858 0.782 [HTML]FFCE93Loss
for Occluded areas ✓ 0.658 0.687 0.716 0.346 0.803 0.736 ✗ 0.631 0.675 0.724
0.342 0.752 0.662 [HTML]7d9ebfBEV Pooling methods Sum 0.666 0.665 0.719 0.308
0.862 0.776 Maxpool 0.656 0.602 0.748 0.309 0.857 0.764 Weighted Sum (ours)
0.667 0.607 0.755 0.325 0.860 0.769 Sensors per BEV Cell (ours) 0.679 0.644
0.759 0.363 0.861 0.774 Cam Intrin + Emb (ours) 0.690 0.682 0.764 0.364 0.858
0.782

###  IV-B Architecture Details

As described in Section III, we select the architectural approach from Lift-
Splat-Shoot [5] to implement the BEV segmentation using fisheye cameras. Given
the near-field applications of fisheye cameras, we choose a BEV grid cell size
of $0.25m$ instead of $0.5m$, which is prevalent in pinhole camera approaches.
Further, we set a coverage range of $25m$ from the ego vehicle longitudinally
and latitudinally instead of the commonly used $50m$ range.

Also, we analyzed the architecture for accuracy bottlenecks and found it to be
input resolution. Hence we increased it to $480\times 302$, from $352\times
128$, almost doubling the number of input pixels. Given the larger FOV of the
fisheye images, this also follows our intuitive reasoning. Finally, to address
the relative imbalance in the dataset volume between open-source pinhole
datasets and our simulated Cognata dataset, we test several combinations of
dropout layers in the network to prevent overfitting. Additionally, we
incorporate color jittering augmentation and other commonly used data
augmentation techniques such as scaling, rotation, and cropping. We use a
contrast variation of $0.5$, saturation and brightness variation of $0.3$
each, and a hue variation of $0.05$.

##  V Experiments and Results

We perform several training runs to evaluate implementation details and
improve the results. We use a GPU server with four Nvidia Quadro RTX5000 GPUs
with a batch size of 64. We use Adam optimizer [62] with a learning rate of
$10^{-4}$. All implementation is done in PyTorch [63].

We generate a test set of three scenes: easy, medium, and hard. The easy scene
is a traffic and weather variation of a scene that’s part of the training set.
Therefore, the road geometry is familiar to the model in an easy scene, while
the vehicle configuration is still unknown. The medium scene is a previously
unseen highway scene, and the hard scene is a previously unseen urban scene
with complex road geometry. We use Intersection over Union (IoU), defined in
Equation 9, as the primary metric for semantic classification. It is computed
with predicted semantic class ($y$) and ground truth semantic class ($\hat{y}$
). We track and report the IoU for each class separately to account for the
skew in classes (background vs pedestrians/vehicles). We also use IoU to
predict occupancy by converting it to binary masks.

$IoU(y,\hat{y})=\frac{|y\cap\hat{y}|}{|y\cup\hat{y}|}$ (9)

Finally, we report the mIoU, which is typically just the mean of all IoU
scores of the classification head. As we want a single metric to compare the
performance between different networks, we want to incorporate the score for
occlusion into the mIoU score. Therefore, the mIoU score we report in the
following is the average from the five IoU scores for occlusion, vehicles,
lane markings, street, and background. Table I shows the results of training
and test datasets. Naturally, the training performance scores highest.
Comparing the three test sets shows the performance drop with the increasing
difficulty level. The medium test set scores are significantly lower than the
easy test set, especially for the lane marking and the background. However,
the network learns to position the vehicles quite well.

We also compared against standard baseline [5] by applying a cylindrical
projection to rectify the fisheye images. Cylindrical projection preserves the
vertical lines from fisheye images. The results in Table I indicate that our
model performs better than the rectification process. It is an intuitive
outcome, given that rectification introduces unnecessary object stretching and
interpolation artifacts to the image in addition to losing some of the field
of view.

###  V-A Ablation Studies

####  V-A1 Class Weighting

As discussed in Section III-D, we apply a non-uniform class weighting to
improve the overall performance by focusing more on underrepresented classes.
During the experiment, we used a class weighting of
($13\times,3\times,1\times,1\times$) for vehicles, lane markings, streets, and
backgrounds, respectively. We analyze the effect of the class-weighted loss
and compare the configuration mentioned above with a uniformly weighted loss.
The model with uniformly weighted loss performs better for almost all classes.
The overall performance of the model with uniform class loss is $0.032$ mIoU
points better than the performance for the model with weighted loss, as stated
in Table II.

Our explanation for this counter-intuitive behavior is that it is a different
task than image classification. Most literature about dataset and loss
balancing studies the problem of image classification. There, the classes are
often equally challenging, and improving the performance of underrepresented
classes makes sense. For semantic BEV segmentation, though, not only the
occurrence of a class at the output of the network matters. Additionally, the
presence and detectability of a class in the input domain are essential.

####  V-A2 Occupancy Prediction

Given the geometry-based approach for Camera to BEV space, we hypothesized
that the network cannot estimate areas occluded in the image well. Therefore,
the network guesses occluded regions based on image statistics, deteriorating
the overall quality. We apply an occlusion-based loss function to circumvent
this guessing or hallucinating on occluded areas. When regions are occluded,
we do not assign a loss to them, so the network is not encouraged to
hallucinate. Loss removal for occluded regions avoids the network updating its
weights based on these occluded regions. Further, we compute the IoU only for
grid cells that are at least $50\%$ visible so that our metrics indicate the
performance on perceived regions and not on hallucinated ones. Table II
illustrates the impact of occupancy loss on the segmentation module.

####  V-A3 Comparing different BEV Pooling methods

Incorporating learnable parameters for each cell in the BEV grid enables
adaptive feature pooling, allowing us to consider each sensor’s contribution
to the BEV grid. This results in a more context-aware representation of
features. This adaptivity enhances the ability to capture relevant information
from different cameras and their varying characteristics. Further, it enables
the pooling operation to leverage the strengths of various sensors,
considering factors such as their field of view, resolution, or proximity to
the target object. Specifically, for the BEV segmentation task, the finer
cell-level adaptivity of BEV pooling yields a better semantic boundary in the
case of small objects. It is evidenced in our results in Table II. We see much
better results from our pooling method that factor in sensor intrinsics than
other naive pooling approaches. Using intrinsics and embedding for BEV pooling
also improves our occlusion IoU significantly, further strengthening our
claims.

Figure 5: DaF-BEVSeg results on Easy scene. Left: The 4x surround camera
images. Top right: The ground truth. Bottom right: Model Prediction. Figure 6:
DaF-BEVSeg results on Medium scene. Left: The 4x surround camera images. Top
right: The ground truth. Bottom right: Model Prediction. Figure 7: DaF-BEVSeg
results on Hard scene. Left: The 4x surround camera images. Top right: The
ground truth. Bottom right: Model Prediction.

###  V-B Qualitative Results

We provide qualitative results of DaF-BEVSeg on the test data. Figure 5 shows
the 4x fisheye images, ground truth, and prediction results for an easy scene.
We provide complete qualitative results of DaF-BEVSeg on the test data at
https://streamable.com/ge4v51. As a similar scene is part of the training
data, the network outputs an accurate semantic BEV segmentation. The network
performs an excellent fusion of the Fisheye images into a single BEV image.

In case of medium test scene, shown in Figure 6, the segmentation complexity
can seen in road class beyond 10m. However, vehicles and lane markings are
still detected accurately. On a hard test scene, as shown in Figure 7, DaF-
BEVSeg is challenged the most, with complicated lane markings. This is
intuitive considering out-of-distribution data is always challenging for
supervised methods.

##  VI Conclusion

In this work, we introduce a novel pipeline to perform semantic Bird’s Eye
View (BEV) segmentation using images from fisheye cameras. We perform our
experiments on a synthetic dataset due to the lack of public datasets with
fisheye cameras. Future directions for our work include bridging the reality
gap between synthetic and real-world data and extending to additional tasks
with fisheye cameras. We hope that this work encourages further research in
performing BEV perception for fisheye cameras.

## Acknowledgment

The synthetic data used in this paper were generated using a commercial
synthetic engine www.cognata.com. The authors thank our colleague Anamarija
Fofonjka for the detailed review and feedback.

## References

  * [1] J. Huang, G. Huang, Z. Zhu, Y. Yun _et al._ , “Bevdet: High-performance multi-camera 3d object detection in bird-eye-view,” _arXiv preprint arXiv:2112.11790_ , 2021.
  * [2] H. Rashed, M. Essam, M. Mohamed, A. E. Sallab _et al._ , “Bev-modnet: Monocular camera based bird’s eye view moving object detection for autonomous driving,” in _2021 IEEE International Intelligent Transportation Systems Conference (ITSC)_. IEEE, 2021, pp. 1503–1508.
  * [3] H. Liu, Y. Teng, T. Lu, H. Wang _et al._ , “Sparsebev: High-performance sparse 3d object detection from multi-camera videos,” in _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 2023, pp. 18 580–18 590.
  * [4] Z. Li, W. Wang, H. Li, E. Xie _et al._ , “Bevformer: Learning bird’s-eye-view representation from multi-camera images via spatiotemporal transformers,” _ArXiv_ , vol. abs/2203.17270, 2022.
  * [5] J. Philion and S. Fidler, “Lift, splat, shoot: Encoding images from arbitrary camera rigs by implicitly unprojecting to 3d,” in _European Conference on Computer Vision_ , 2020.
  * [6] J. Zhang, Y. Zhang, Q. Liu, and Y. Wang, “Sa-bev: Generating semantic-aware bird’s-eye-view feature for multi-view 3d object detection,” in _Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)_ , October 2023, pp. 3348–3357.
  * [7] M. Klingner, S. Borse, V. R. Kumar, B. Rezaei _et al._ , “X3kd: Knowledge distillation across modalities, tasks and stages for multi-camera 3d object detection,” _2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pp. 13 343–13 353, 2023.
  * [8] S. Borse, M. Klingner, V. R. Kumar, H. Cai _et al._ , “X-align++: cross-modal cross-view alignment for bird’s-eye-view segmentation,” _Machine Vision and Applications_ , vol. 34, pp. 1–16, 2022.
  * [9] M. Li, Y. Zhang, X. Ma, Y. Qu _et al._ , “Bev-dg: Cross-modal learning under bird’s-eye view for domain generalization of 3d semantic segmentation,” _2023 IEEE/CVF International Conference on Computer Vision (ICCV)_ , pp. 11 598–11 608, 2023.
  * [10] L. Peng, Z. Chen, Z.-H. Fu, P. Liang _et al._ , “Bevsegformer: Bird’s eye view semantic segmentation from arbitrary camera rigs,” _2023 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV)_ , pp. 5924–5932, 2022.
  * [11] C. Pan, Y. He, J. Peng, Q. Zhang _et al._ , “Baeformer: Bi-directional and early interaction transformers for bird’s eye view semantic segmentation,” _2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pp. 9590–9599, 2023.
  * [12] Z. Liu, H. Tang, A. Amini, X. Yang _et al._ , “Bevfusion: Multi-task multi-sensor fusion with unified bird’s-eye view representation,” _2023 IEEE International Conference on Robotics and Automation (ICRA)_ , pp. 2774–2781, 2022.
  * [13] B. Huang, Y. Li, E. Xie, F. Liang _et al._ , “Fast-bev: Towards real-time on-vehicle bird’s-eye view perception,” _ArXiv_ , vol. abs/2301.07870, 2023.
  * [14] E. Xie, Z. Yu, D. Zhou, J. Philion _et al._ , “M2bev: Multi-camera joint 3d detection and segmentation with unified birds-eye view representation,” _ArXiv_ , vol. abs/2204.05088, 2022.
  * [15] H. Wang, H. Tang, S. Shi, A. Li _et al._ , “Unitr: A unified and efficient multi-modal transformer for bird’s-eye-view representation,” _2023 IEEE/CVF International Conference on Computer Vision (ICCV)_ , pp. 6769–6779, 2023.
  * [16] T. Pham, M. Maghoumi, W. Jiang, B. S. S. Jujjavarapu, M. Sajjadi, X. Liu, H.-C. Lin, B.-J. Chen, G. Truong, C. Fang _et al._ , “Nvautonet: Fast and accurate 360deg 3d visual perception for self driving,” in _Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision_ , 2024, pp. 7376–7385.
  * [17] W. K. Fong, R. Mohan, J. V. Hurtado, L. Zhou _et al._ , “Panoptic nuscenes: A large-scale benchmark for lidar panoptic segmentation and tracking,” _arXiv preprint arXiv:2109.03805_ , 2021.
  * [18] P. Sun, H. Kretzschmar, X. Dotiwalla, A. Chouard _et al._ , “Scalability in perception for autonomous driving: Waymo open dataset,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , June 2020.
  * [19] V. R. Kumar, C. Eising, C. Witt, and S. Yogamani, “Surround-view fisheye camera perception for automated driving: Overview, survey & challenges,” _IEEE Transactions on Intelligent Transportation Systems_ , 2023.
  * [20] M. Pöpperli, R. Gulagundi, S. Yogamani, and S. Milz, “Capsule neural network based height classification using low-cost automotive ultrasonic sensors,” in _2019 IEEE Intelligent Vehicles Symposium (IV)_. IEEE, 2019, pp. 661–666.
  * [21] H. Rashed, E. Mohamed, G. Sistu, V. R. Kumar _et al._ , “Fisheyeyolo: Object detection on fisheye cameras for autonomous driving,” in _Proceedings of the Machine Learning for Autonomous Driving NeurIPS 2020 Virtual Workshop, Virtual_ , vol. 11, 2020.
  * [22] L. Yahiaoui, J. Horgan, B. Deegan, S. Yogamani _et al._ , “Overview and empirical analysis of isp parameter tuning for visual perception in autonomous driving,” _Journal of Imaging_ , vol. 5, no. 10, p. 78, 2019.
  * [23] V. R. Kumar, M. Klingner, S. Yogamani, M. Bach _et al._ , “Svdistnet: Self-supervised near-field distance estimation on surround view fisheye cameras,” _IEEE Transactions on Intelligent Transportation Systems_ , vol. 23, no. 8, pp. 10 252–10 261, 2021.
  * [24] V. R. Kumar, S. Milz, C. Witt, M. Simon _et al._ , “Near-field depth estimation using monocular fisheye camera: A semi-supervised learning approach using sparse lidar data,” in _CVPR Workshop_ , vol. 7, 2018, p. 2.
  * [25] N. Tripathi and S. Yogamani, “Trained trajectory based automated parking system using Visual SLAM,” in _Proceedings of the Computer Vision and Pattern Recognition Conference Workshops_ , 2021.
  * [26] A. Konrad, C. Eising, G. Sistu _et al._ , “FisheyeSuperPoint: Keypoint Detection and Description Network for Fisheye Images,” _Proceedings of the International Conference on Computer Vision Theory and Applications_ , vol. abs/2103.00191, 2021.
  * [27] S. Shen, L. Kerofsky, and S. Yogamani, “Optical flow for autonomous driving: Applications, challenges and improvements,” _Electronic Imaging_ , 2024.
  * [28] E. Mohamed, M. Ewaisha, M. Siam, H. Rashed _et al._ , “Monocular instance motion segmentation for autonomous driving: Kitti instancemotseg dataset and multi-task baseline,” in _2021 IEEE Intelligent Vehicles Symposium (IV)_. IEEE, 2021, pp. 114–121.
  * [29] H. Rashed, A. El Sallab, S. Yogamani, and M. ElHelw, “Motion and depth augmented semantic segmentation for autonomous navigation,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops_ , 2019, pp. 0–0.
  * [30] S. Ramachandran, G. Sistu, J. McDonald, and S. Yogamani, “Woodscape fisheye semantic segmentation for autonomous driving–cvpr 2021 omnicv workshop challenge,” _arXiv preprint arXiv:2107.08246_ , 2021.
  * [31] H. Rashed, S. Yogamani, A. El-Sallab, P. Krizek, and M. El-Helw, “Optical flow augmented semantic segmentation networks for automated driving,” in _Proceedings of the International Conference on Computer Vision Theory and Applications_ , 2019.
  * [32] M. Uricár, J. Ulicny, G. Sistu, H. Rashed _et al._ , “Desoiling dataset: Restoring soiled areas on automotive fisheye cameras,” in _Proceedings of the IEEE/CVF International Conference on Computer Vision Workshops_ , 2019, pp. 0–0.
  * [33] M. M. Dhananjaya, V. R. Kumar, and S. Yogamani, “Weather and light level classification for autonomous driving: Dataset, baseline and active learning,” in _2021 IEEE International Intelligent Transportation Systems Conference (ITSC)_ , 2021, pp. 2816–2821.
  * [34] G. Sistu, I. Leang, and S. Yogamani, “Real-time joint object detection and semantic segmentation network for automated driving,” _arXiv preprint arXiv:1901.03912_ , 2019.
  * [35] G. Sistu, I. Leang, S. Chennupati, S. Yogamani _et al._ , “Neurall: Towards a unified visual perception model for automated driving,” in _2019 IEEE Intelligent Transportation Systems Conference (ITSC)_. IEEE, 2019, pp. 796–803.
  * [36] I. Leang, G. Sistu, F. Bürger, A. Bursuc _et al._ , “Dynamic task weighting methods for multi-task networks in autonomous driving systems,” in _2020 IEEE 23rd International Conference on Intelligent Transportation Systems (ITSC)_. IEEE, 2020, pp. 1–8.
  * [37] V. R. Kumar, S. Yogamani, H. Rashed, G. Sitsu _et al._ , “Omnidet: Surround view cameras based multi-task visual perception network for autonomous driving,” _IEEE Robotics and Automation Letters_ , vol. 6, no. 2, pp. 2830–2837, 2021.
  * [38] C. Eising, J. Horgan, and S. Yogamani, “Near-field perception for low-speed vehicle automation using surround-view fisheye cameras,” _IEEE Transactions on Intelligent Transportation Systems_ , vol. 23, no. 9, pp. 13 976–13 993, 2021.
  * [39] M. Uricár, D. Hurych, P. Krizek, and S. Yogamani, “Challenges in designing datasets and validation for autonomous driving,” _Proceedings of the International Conference on Computer Vision Theory and Applications_ , 2019.
  * [40] S. Mohapatra, S. Yogamani, H. Gotzig, S. Milz _et al._ , “Bevdetnet: bird’s eye view lidar point cloud based real-time 3d object detection for autonomous driving,” in _2021 IEEE International Intelligent Transportation Systems Conference (ITSC)_. IEEE, 2021, pp. 2809–2815.
  * [41] C. Lu, M. J. G. Van De Molengraft, and G. Dubbelman, “Monocular semantic occupancy grid mapping with convolutional variational encoder–decoder networks,” _IEEE Robotics and Automation Letters_ , vol. 4, no. 2, pp. 445–452, 2019.
  * [42] B. Pan, J. Sun, H. Y. T. Leung, A. Andonian _et al._ , “Cross-view semantic segmentation for sensing surroundings,” _IEEE Robotics and Automation Letters_ , vol. 5, no. 3, pp. 4867–4873, 2020.
  * [43] T. Roddick and R. Cipolla, “Predicting semantic map representations from images using pyramid occupancy networks,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2020, pp. 11 138–11 147.
  * [44] A. Saha, O. Mendez, C. Russell, and R. Bowden, “Translating images into maps,” in _2022 International conference on robotics and automation (ICRA)_. IEEE, 2022, pp. 9200–9206.
  * [45] B. Zhou and P. Krähenbühl, “Cross-view transformers for real-time map-view semantic segmentation,” in _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , 2022, pp. 13 760–13 769.
  * [46] H. A. Mallot, H. H. Bülthoff, J. Little, and S. Bohrer, “Inverse perspective mapping simplifies optical flow computation and obstacle detection,” _Biological cybernetics_ , vol. 64, no. 3, pp. 177–185, 1991.
  * [47] Y. Kim and D. Kum, “Deep learning based vehicle position and orientation estimation via inverse perspective mapping image,” in _2019 IEEE Intelligent Vehicles Symposium (IV)_. IEEE, 2019, pp. 317–323.
  * [48] X. Zhu, Z. Yin, J. Shi, H. Li _et al._ , “Generative adversarial frontal view to bird view synthesis,” in _2018 International conference on 3D Vision (3DV)_. IEEE, 2018, pp. 454–463.
  * [49] L. Reiher, B. Lampe, and L. Eckstein, “A sim2real deep learning approach for the transformation of images from multiple vehicle-mounted cameras to a semantically segmented image in bird’s eye view,” in _2020 IEEE 23rd International Conference on Intelligent Transportation Systems (ITSC)_. IEEE, 2020, pp. 1–7.
  * [50] E. U. Samani, F. Tao, H. R. Dasari, S. Ding _et al._ , “F2bev: Bird’s eye view generation from surround-view fisheye camera images for automated driving,” in _2023 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_. IEEE, 2023, pp. 9367–9374.
  * [51] C. Ramchandra, S. Ganesh, E. Ciarán, R. K. Varun _et al._ , “Fisheyepixpro: Self-supervised pretraining using fisheye images for semantic segmentation,” _Electronic Imaging_ , vol. 34, pp. 1–6, 2022.
  * [52] L. Deng, M. Yang, Y. Qian, C. Wang _et al._ , “Cnn based semantic segmentation for urban traffic scenes using fisheye camera,” in _2017 IEEE Intelligent Vehicles Symposium (IV)_. IEEE, 2017, pp. 231–236.
  * [53] L. Deng, M. Yang, H. Li, T. Li _et al._ , “Restricted deformable convolution-based road scene semantic segmentation using surround view cameras,” _IEEE Transactions on Intelligent Transportation Systems_ , vol. 21, no. 10, pp. 4350–4362, 2019.
  * [54] C. Häne, L. Heng, G. H. Lee, A. Sizov _et al._ , “Real-time direct dense matching on fisheye images using plane-sweeping stereo,” in _2014 2nd International Conference on 3D Vision_ , vol. 1. IEEE, 2014, pp. 57–64.
  * [55] C. Won, J. Ryu, and J. Lim, “Omnimvs: End-to-end learning for omnidirectional stereo matching,” in _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 2019, pp. 8987–8996.
  * [56] V. R. Kumar, S. A. Hiremath, M. Bach, S. Milz _et al._ , “Fisheyedistancenet: Self-supervised scale-aware distance estimation using monocular fisheye camera for autonomous driving,” in _2020 IEEE International Conference on Robotics and Automation (ICRA)_ , 2020, pp. 574–581.
  * [57] J. Kannala and S. Brandt, “A generic camera model and calibration method for conventional, wide-angle, and fish-eye lenses,” _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , vol. 28, no. 8, pp. 1335–1340, 2006.
  * [58] V. R. Kumar, S. Yogamani, M. Bach _et al._ , “Unrectdepthnet: Self-supervised monocular depth estimation using a generic framework for handling common camera distortion models,” _Conference on Intelligent Robots & Systems (IROS)_, 2020.
  * [59] H. Caesar, V. Bankiti, A. H. Lang, S. Vora _et al._ , “nuScenes: A Multimodal Dataset for Autonomous Driving,” in _Proc. of CVPR_ , 2020, pp. 11 621–11 631.
  * [60] Z. Zhang and M. Sabuncu, “Generalized cross entropy loss for training deep neural networks with noisy labels,” _Advances in neural information processing systems_ , vol. 31, 2018.
  * [61] A. R. Sekkat, Y. Dupuis, V. R. Kumar, H. Rashed _et al._ , “Synwoodscape: Synthetic surround-view fisheye camera dataset for autonomous driving,” _IEEE Robotics and Automation Letters_ , vol. 7, no. 3, pp. 8502–8509, 2022.
  * [62] D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” _arXiv preprint arXiv:1412.6980_ , 2014.
  * [63] A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga _et al._ , “Pytorch: An imperative style, high-performance deep learning library,” _Advances in neural information processing systems_ , vol. 32, 2019.