2021

[2]Rui Tang

[1]Jian Pu

1]Institute of Science and Technology for Brain-Inspired Intelligence, Fudan
University, No. 220, Handan Road, Yangpu Area, Shanghai, 200433, Shanghai,
China

2]Computer Vision Perception Department, ZongMu Technology, Lane 55, Chuanhe
Road, Pudong New Area, Shanghai, 201203, Shanghai, China

3]Faculty of Electrical Engineering, Information Technology, Physics,
Technical University of Braunschweig, Universitätsplatz 2, Braunschweig,
38106, Lower Saxony, Germany

# LineMarkNet: Line Landmark Detection for Valet Parking

Zizhang Wu  Yuanzhu Gan  Tianhao Xu<EMAIL_ADDRESS><EMAIL_ADDRESS>[  [  [

###### Abstract

We aim for accurate and efficient line landmark detection for valet parking,
which is a long-standing yet unsolved problem in autonomous driving. To this
end, we present a deep line landmark detection system where we carefully
design the modules to be lightweight. Specifically, we first empirically
design four general line landmarks including three physical lines and one
novel mental line. The four line landmarks are effective for valet parking. We
then develop a deep network (LineMarkNet) to detect line landmarks from
surround-view cameras where we, via the pre-calibrated homography, fuse
context from four separate cameras into the unified bird-eye-view (BEV) space,
specifically we fuse the surround-view features and BEV features, then employ
the multi-task decoder to detect multiple line landmarks where we apply the
center-based strategy for object detection task, and design our graph
transformer to enhance the vision transformer with hierarchical level graph
reasoning for semantic segmentation task. At last, we further parameterize the
detected line landmarks (e.g., intercept-slope form) whereby a novel filtering
backend incorporates temporal and multi-view consistency to achieve smooth and
stable detection. Moreover, we annotate a large-scale dataset to validate our
method. Experimental results show that our framework achieves the enhanced
performance compared with several line detection methods and validate the
multi-task network’s efficiency about the real-time line landmark detection on
the Qualcomm 820A platform while meantime keeps superior accuracy, with our
deep line landmark detection system.

###### keywords:

Valet Parking, surround-view cameras, line landmark, BEV.

##  1 Introduction

Valet parking, aiming at driving the vehicle into the drop-off area such as
parking slots, is an important and challenging task in autonomous driving
song2019apollocar ; zhou2020joint ; wu2020motionnet . With the advances of
deep learning, valet parking achieves significant progress but is still
bottlenecked by the drop-off area’s highly complex environment resulting in
inaccurate perception. In this case, different from autonomous driving in
relatively simpler scenarios like highways and urban areas, valet parking
faces more challenges in terms of perception.

Many methods have been proposed to improve the perception of valet parking
lee2017real ; zhou2020joint ; wu2020motionnet . Among those works, the
detection of traffic landmarks is one of the most effective ways Lee2022robust
; hu2022sim ; chen2020collaborative ; wu2020psdet . For instance,
PSDetwu2020psdet  achieves impressive parking slot detection by recognizing a
variety of landmarks around the empty slots (e.g., four corners of a
rectangular parking slot.). In particular, line landmarks are essential for
valet parking, i.e., the traffic signs that are typically long lines. In some
sense, line landmark is one of the most informative traffic signs, which
dominate the information source in order to park vehicles. Even as human
beings, while parking cars in parking lots, we analyze and follow the line
landmarks (e.g., lane line) to recognize the drivable areas and then drive the
vehicles safely to the drop-off area. Thus, it’s crucial for autonomous
driving to take full advantage of these informative traffic signs, especially
in parking scenes with various parking cases under different illumination
conditions.

Nevertheless, existing works are still limited by the inability to accurately
detect a variety of line landmarks qin2022ultra ; wang2022keypoint ;
feng2022rethinking ; zheng2022clrnet . The key challenge is that – compared
with landmarks of other shapes such as circles, points, and rectangles – line
landmarks are typically long and ambiguous (two different line landmarks could
have a similar visual appearance). Thus, accurate detection of line landmarks
requires a long-range context which, however, is not trivial. On the other
hand, due to ambiguous appearance, line landmarks are also ambiguous in terms
of definition which results in unclear human annotations pauls2021automatic ;
wang2017landmarks . When collecting labels for a dataset, annotators, also
learners, don’t have paradigms to follow and thus give error-prone labels.
Furthermore, deep models may fail to converge without a clear definition of
labels due to the ambiguous labels in training data liu2021condlanenet ;
tabelini2021keep .

Figure 1:  Four Line landmark detection for valet parking: Inspired by
empirical observations from industrial applications, we summarize four general
line landmarks which are essential for valet parking. Specifically, we propose
three physical line landmarks (a: parking line, b: lane line, c: median line)
and one mental line (d: stationary car boundary line). Note our mental line is
a virtual line connecting stationary cars (parking in the slots) and serves as
the tight boundary between drivable and non-drivable areas. The other three
physical line landmarks play a key role in navigation, particularly in the
parking environments. We further elaborate on the motivation of the proposed
four line landmark in Sec. 3.1.

In light of this, we empirically hand-craft four common and essential types of
line landmarks by observing a variety of possible environments for valet
parking. Specifically, we collect and observe a large-scale dataset for valet
parking, assuring that the summarized four line landmarks are representative.
As shown in Fig. 1, the four line landmarks consist of three physical line
landmarks (visible lines such as lane lines) and one mental line landmarks
(i.e., a virtual line that connects the stationary vehicles parking along
lanes). It’s noteworthy that our novel mental line landmarks aim to define the
tight boundary between drivable and non-drivable areas. Moreover, we annotate
the collected dataset, allowing us to train our network for line landmark
detection.

Nevertheless, it’s still challenging to incorporate long-range context for the
detection of long-line landmarks. One key challenge from the hardware aspect
is that, due to the limited field of view, the observations of long line
landmarks via cameras are generally incomplete. To address this issue, we
leverage the surrounded-view camera system to achieve the complete observation
of line landmarks. It consists of four fisheye cameras whereby a large field
of view enables line landmarks to be fully visible. However, as the other
challenge from the software side, it’s still non-trivial for deep networks to
extract long-context since line landmarks distribute over multiple views. To
solve this problem, we develop the LineMarkNet that aims to detect four common
line landmarks from surround-view cameras. The core idea is that, via the pre-
calibrated homography of four cameras, we transform image planes into the BEV
space whereby context information from multiple views is fused. It adopts the
multi-task architecture, where we firstly fuse the surround-view features and
BEV features, then utilize the multi-task decoder (semantic segmentation and
object detection) and calculates the line parameters through the followed line
fitting module as shown in Fig. 3. Within the decoders, we apply the center-
based strategy for object detection and design our graph transformer network
to improve the semantic segmentation performance, which enhances the global
and local relation modeling with hierarchical level graph reasoning.

Our experiments show that LineMarkNet achieves impressive performance.
However, we observe that the detected line landmarks are still unsurprisingly
noisy due to occlusion. Note that while the noisy output is not surprising,
it’s not satisfying for valet parking where high precision is required. Thus,
to mitigate the noise, we refine the initial output of LineMarkNet through a
novel filtering backend. The idea of the filtering backend is to enforce
multi-view and temporal consistency where we encourage the line landmarks to
be consistent across multiple fisheye cameras and adjacent frames. Our
experiments show that our backend effectively removes noises and achieves
smooth and stable line landmark detection.

We observe in our experiments that our system achieves accurate line landmark
detection. Moreover, we carefully design modules of our system to be
lightweight – both memory and computationally efficient. This enables our line
detection system to run in real-time on the Qualcomm 820A platform.

We simply summarize our contributions as:

  * •

we define four general and informative line landmarks including three physical
lines and one novel mental line. The four line landmarks are effective and
essential for valet parking.

  * •

we propose LineMarkNet to detect line landmarks from the surround-view camera
system, which adopts the multi-task architecture. Within it, we fuse the
surround-view features and BEV features to the improved multi-task decoder
(semantic segmentation and object detection) and calculate the line parameters
through the following line fitting module.

  * •

we annotate a large-scale dataset which allows us to train our LineMarkNet,
which contains plenty of the long and ambiguous line landmarks in the valet
parking scenes.

  * •

we further propose a novel filtering backend to refine the initial output of
LineMarkNet where we filter out incorrect line landmarks by enforcing multi-
view and temporal consistency.

  * •

our line mark detection system achieves superior accuracy while being real-
time, compared with the previous line detection approaches.

The other parts of the paper are organized as follows. The second section of
the paper introduces the related methods of the paper. The third section
introduces the details of the method in this paper. The fourth section
describes the experimental comparison and analysis of the method proposed in
this paper. Finally, the fifth section is about the overall summary of the
paper.

##  2 Related Works

We discuss related works that utilize surround-view cameras, review the method
dealing with surround-view images, and briefly summarize the methods for line
landmark detection as below.

###  2.1 Surround-view camera system

Thanks to its large field-of-view 2018IVreal ; wu2021disentangling , the
surround-view camera system is a very popular imaging system for visual
perception 2018IROSend ; 2021arxivomnidet ; zhang2021avp . In particular, the
frameworks for autonomous driving kumar2020fisheyedistancenet ;
yahiaoui2019fisheyemodnet ; 2020IROSAccuratelow ; qin2020avp  are typical of
the surround-view camera system that consists of four fisheye cameras mounted
on the ego-vehicle ouyang2020online ; wang2020reliable . It’s thus capable of
capturing all objects nearby the ego-vehicle, allowing for reliable visual
perception. In this work, we utilize a similar camera system in our line
landmark detection system. Note that our focus is on the perception of valet
parking, which is more challenging than methods for autonomous driving.

###  2.2 Deep frameworks for surround-view camera system

Recently, lots of deep methods have been proposed for surround-view camera
systems with a variety of target tasks such as semantic segmentation
2018ECCVWsemantic ; kumar2021syndistnet ; ye2020universal , distance
estimation kumar2020fisheyedistancenet ; zaffaroni2019estimation ;
komatsu2020360  and object detection yahiaoui2019fisheyemodnet ;
plaut2020monocular ; li2020fisheyedet . The most relevant works are the
methods for semantic segmentation and detection as our system utilizes these
two tasks for line detection. Note that one of the most desiderata for these
methods is the strategy to fuse information from multiple viewpoints. To
address this, existing works wu2020psdet  typically fuse deep features in the
BEV space. In general, deep features are learned from multiple views
individually via the siamese network wu2020psdet . We follow a similar
strategy to fuse information from multiple views, with modifications (e.g.,
line-fitting module) specifically for line landmark detection. Moreover, in
our system, we have the filtering backend that further compresses the
potential noise or outliers during the fusion process.

###  2.3 Line landmarks detection

Line landmarks are kind of semantic lines, existing in many tasks 2020xiao ;
hu2019mapping . Note that we exclude the discussion about the low-level line
feature qin2018vins  as it’s quite different from our line landmarks by
definition. In some sense, by defining four line landmarks, we endow line
features with the semantics to be learned using deep networks. In this
context, the most relevant task is the lane detection 2020Marzonggui ;
2020xiao ; qin2020ultra ; huang2019dmpr ; Lee2022robust ; hu2022sim ;
wang2022keypoint ; feng2022rethinking  where the lane line is modeled as the
line landmarks. So far, previous works 2020xiao ; 2018Song ; qin2020ultra ;
zheng2022clrnet ; liu2021condlanenet ; tabelini2021keep  have achieved
promising lane detection. However, more types of line landmarks exist in the
environment for valet parking whereas, to our best knowledge, previous works
rarely investigate the general line landmark detection as in our benchmark. In
this case, our method bridges the gap between effective deep methods and
missing pipelines for line landmark detection in valet parking.

##  3 Methods

Figure 2: Line landmark detection system –  Our system consists of the
LineMarkNet – a multi-task architecture that detects multiple line landmarks
from the surround-view cameras – and a novel filtering backend to removes
detection errors in the output of lineMarksNet. Our system effectively detects
the line landmarks detection and meanwhile is real-time.

We introduce a deep line landmark detection system for valet parking. As
illustrated in Fig. 2, the system consists of a LineMarkNet that detects line
landmarks from surround-view cameras, and an efficient filtering backend where
we further enhance the results by filtering out noise and outliers in the
output of LineMarkNet. Note that, while having two complex components, our
system is very efficient such that we achieve the real-time line landmark
detection on the Qualcomm 820A platform.

Specifically, we detect line landmarks from surround-view cameras. As shown in
Fig. 4, the surround-view camera system consists of four fisheye cameras with
a large field of view whereby the long line landmarks are fully visible in
four images. To detect line landmarks from surrounding views, we propose
LineMarkNet as illustrated in Fig. 3 where we employ a multi-task architecture
to detect multiple line landmarks. Long-range contextual information is the
key to detecting long-line landmarks. Thus, we transform the information
extracted in 2D image planes into the BEV space via calibrated homography
zhang2000flexible . In some sense, we detect line landmarks in BEV space.

Additionally, we propose the filtering backend to further refine the results
from LineMarkNet. Although the initial detection from LineMarkNet surpasses
naive baselines, the line landmark detection is still noisy whereas valet
parking requires very high precision to order keep the safety of driving. To
mitigate the issue, we propose to optimize detection by enforcing the multi-
view and temporal consistency. Specifically, our filtering backend encourages
the line landmarks detection to be consistent across different views and
adjacent frames. As a result, our system achieves stable and consistent line
landmark detection which enables practical applications in valet parking.

We will detail our method below.

###  3.1 Definition of line landmarks

Our empirical experiences gained in industrial software development mainly
motivate the four line landmarks whereas our four line landmarks suffice to
navigate vehicles in part lots. Interestingly, although a variety of line
landmarks exist in the complex parking lots, merely a few common line
landmarks is sufficient for valet parking. The reason is that line landmarks
are human-designed for certain goals. In particular, one of the most important
goals is to navigate vehicles. For instance, the lane marks, by design, aim to
separate different lanes whereby vehicles are navigated to drive in the legal
lanes. In essence, all our line landmarks aim to navigate vehicles in the
parking environments except for the minor differences in functionality – e.g,
navigating in different areas, leading to our four line landmarks. Note that,
despite the possibility of neglecting useful line landmarks, it’s trivial to
adapt our line landmark detection system to new line landmarks, allowing our
system to generalize into new types of parking lots.

As shown in Fig. 1, we summarize four types of line landmarks. And we further
briefly describe the functions and pattern as

  * •

Parking line (Fig. 1 a) is the landmark that separates drivable lanes and
parking slots, essential for vehicle to localize the target parking slot. The
parking lines are typically long and continuous (i.e., the landmark is a
straight line.).

  * •

Lane line (Fig. 1 b) separates multiple drivable lanes, enable vehicle to
drive on a certain lane. In general, it’s the line landmarks closest to the
vehicle. And it can be continuous or discrete (i.e., the landmark is composed
of a set of line segments.).

  * •

Median line (Fig. 1 c) differentiate between lanes of two directions. It’s
critical to avoid collisions between vehicles from different directions. As
illustrated in Fig. 1, it’s generally a long and straight line.

  * •

Stationary vehicle boundary line (Fig. 1 d) is a mental line landmark – it
doesn’t physically exists. We define it as the line that connects stationary
vehicles (i.e., vehicles parked in slots). We connect the mid points of
stationary vehicles closest to the lane. Thus, our novel mental line serves as
the tight boundary between drivable and non-drivable areas. The motivation of
this mental line is very straightforward since as human-being, we mentally
draw some auxiliary lines to help driving.

The line landmarks of the same type share similarity in terms of visual
pattern. Thus, we easily collect labels according to the visual pattern which
enables the training of our pipeline to detect line landmarks.

###  3.2 The Framework of LineMarkNet

Figure 3: The framework of LineMarkNet – The network takes as input the
surround-view fisheye images, which allows for the line landmark detection
with the large field of view. Specifically, we fuse contextual information in
BEV space and performance multiple tasks where our system perceives line
landmarks via tasks of semantic segmentation and object detection and
calculates the line parameters through the followed line fitting module.

We introduce the LineMarkNet where we detect line landmarks from surround-view
images. To fuse information in different views (BEV and surrounding views), we
transform the information extracted in 2D image planes into the BEV space. The
multi-task decoders make full use of the fusion features to detect and
vectorize the four general and informative line landmarks. Finally, the
efficient filtering backend enhances the line landmarks to be more smooth and
more stable. Fig. 2 shows the overall structure of our proposed system.

####  3.2.1 BEV Images Generation

Our system needs to obtain BEV images of the surrounding environment.
Generally, the camera’s optical axis has a certain inclination angle with the
ground, so we need to conduct the perspective transformation to obtain the BEV
images. We choose the center of the front camera as the origin of the world
coordinate system. We change the perspective of the images according to the
linear transform of the homography matrix zhang2000flexible . We can compute
the coordinate ($x$, $y$) in the image coordinate system from the coordinate
($X$, $Y$, $Z$) in the world coordinate system using the following homography
transformation:

$\displaystyle x=\frac{G_{1}X+G_{2}Y+G_{3}Z+G_{4}}{G_{9}X+G_{10}Y+G_{11}Z+1}$
(1) $\displaystyle
y=\frac{G_{5}X+G_{6}Y+G_{7}Z+G_{8}}{G_{9}X+G_{10}Y+G_{11}Z+1}$ (2)

where $G_{1}$, $G_{2}$, $G_{3}$, …, $G_{11}$ are unknown parameters. When
projecting the object to the horizontal plane, $Z$ in the world coordinate
system becomes zero. So we select four sets of corresponding coordinate points
in the world coordinate system and the image coordinate system:
($X_{0}$,$Y_{0}$), ($x_{0}$,$y_{0}$); ($X_{1}$,$Y_{1}$), ($x_{1}$,$y_{1}$);
($X_{2}$,$Y_{2}$), ($x_{2}$,$y_{2}$); ($X_{3}$,$Y_{3}$), ($x_{3}$,$y_{3}$),
and substitute them into Equation (1) and (2) to solve the parameters (from
$G_{1}$ to $G_{8}$), and complete the transformation process.

Fig. 4 presents the illustration of four surround-view cameras and the
synthetic BEV image from four surround-view camera images. After obtaining the
parameters from $G_{1}$ to $G_{8}$, we can capture the physical coordinates
from pixel coordinates through the homography transformation.

Figure 4: Illustration of four surround-view cameras and the synthetic BEV
image from four surround-view camera images. Best view in color and zoom in.
Figure 5: The overall structure of our BEV encoder. The BEV encoder exploits
graph transformer, which enhances the vision transformer with hierarchical
level graph reasoning, including global and local relation modeling. Best view
in color and zoom in.

Our system is a multi-task architecture where various tasks share the image
features, and different decoders are designed for specific tasks, as shown in
Fig. 3.

####  3.2.2 Encoder

The encoder consists of the surround-view encoder and BEV encoder, extracting
informative image features from surround-view images and BEV images.

##### Surround-view encoder (CNNs)

Our surround-view encoder takes as input the four surround-view images and
outputs the surround-view features. We concatenate surround-view images as
batched input – a tensor with the batch size of 4. In other words, our encoder
acts as a siamese network that is weight-sharing among four views. The tensor
shape indicates $[4,3,H_{S},W_{S}]$, where $H_{S}$ and $W_{S}$ mean the
surround-view images’ height and width. We adopt the standard DLA-34 dla  as
the surround-view encoder, for a better speed-accuracy trade-off. Finally, the
surround-view encoder produces surround-view features:
$[4,N_{S},h_{m},w_{m}]$, where $N_{S}$ denotes the channel of feature maps,
$h_{m},w_{m}$ mean feature maps’ height and width.

##### BEV image encoder (graph transformer)

Unlike surround-view images, BEV images consist of long-range contexts and
need the feature extractor with a larger receptive field. In this case, we
propose to learn the BEV features with the graph transformer (i.e., our BEV
image encoder), which is an enhanced vision transformer with better long-range
contextual information.

We firstly state the input and output. Four surround-view images can create
one BEV image, denoting $[1,3,H_{B},W_{B}]$. To match the 4 batch-size
surround-view features when feature fusing, we need to copy the BEV image 4
times to form $[4,3,H_{B},W_{B}]$ as the BEV encoder’s inputs. Then the BEV
encoder can output $[4,N_{B},h_{m},w_{m}]$ features, with the same shape with
surround-view features.

Below we introduce our BEV encoder, as shown in Figure 5. We divide one image
into patches and define one window composed of some (like 7$\times$7) patches.
In other words, one image can be divided into some windows and one window
includes 7$\times$7 patches. Firstly, we propose the Global Relation (GR)
module to reflect the global modeling among different windows. Specifically,
we consider each window as one graph node and use the efficient graph relation
network to build the relation among the graph nodes. Then we update these
nodes using graph convolution bruna2014spectral . The GR module can obtain the
coarse relation of the objects in each image. Afterward, we conduct the Local
Relation (LR) module to build relations inside each window, meaning the patch-
level relations. Considering each patch as a node, the module builds the graph
relation network to model the local relation within each window.

The original Swin liu2021swin  employs the shifted window technique to
strengthen windows’ connection, but this interaction with other windows is
unfree with fixed orientation and fixed offset, where the long-distance
relationships between windows/patches are not thoroughly explored. We adopt
graph convolutions to establish relationships between windows and patches
inside each window, which enhances the block to address the above issues.
Actually, both the GR module and LR module can be regarded as one kind of
self-attention mechanism.

It’s noteworthy that the graph transformer block can be implemented by two
$1\times 1$ Conv, a normalization function (Softmax) and other basic elements
(LayerNorm and Multilayer Perceptron in Swin liu2021swin ). Specifically, we
first use $1\times 1$ Conv to reduce the computational complexity within the
block (saying X becomes X′). Similarly, another $1\times 1$ Conv is realized
to resume the channel dimension. To obtain the relation matrix A, we use a
matrix multiplication and a softmax function to normalize the results. W is
the learnable parameter of graph convolution. So the graph node update process
can be accomplished by two matrix multiplication simply:
A$\otimes$X′$\otimes$W. $\otimes$ means the matrix multiplication.

The LR module’s node update is the same as the GR’s, but with different
learnable parameters. At last, we fuse (add) these two relation modeling
modules to the following LayerNorm and MLP modules. In the end, the graph
transformer can extract more robust BEV features. After concatenation between
the surround-view features and BEV features, the produced fusion features can
contain long-range contextual information.

####  3.2.3 Decoder

Multi-task decoders contain the semantic segmentation task and object
detection task, which extract line landmarks’ pixels and perform vectorized
outputs after different post-processing.

Figure 6: The overall structure of our decoder, including object detection
decoder and semantic segmentation decoder. Best view in color and zoom in.

##### Semantic segmentation decoder

To better optimize the boundaries of objects, we add a boundary-aware
attention module in the decoder. We perceive the relation between local
pixels, based on which we learn to obtain the boundary-aware attention
coefficients of each surrounding pixel, as shown in Fig. 6(b). Concretely, we
first use two $1\times 1$ Conv to reduce and resume the channel dimension.
Then we adopt the local modeling function to learn the local relation, where
the pixels around the boundary of the object are adjusted. Here we simply use
$7\times 7$ Conv to implement the local relation modeling function. Besides,
we add the non-linear function GELU inside the boundary-aware module to learn
more robust relations. Finally, we choose Sigmoid as our normalization layer.
The boundary-aware attention module emphasizes on boundary-aware attention
with local relations, which results in a better segmentation performance.

##### Object detection decoder

The stationary vehicle boundary line is important for helping automatic
parking vehicles to determine the drivable area more accurately. To generate
it, we need to detect and locate surrounding vehicles from the fisheye images,
then compute their key points, as shown in Fig. 7.

So we perform 2D object detection on fisheye images and adopt the same decoder
as CenterNet 2019arxivobjects  for a trade-off between real-time performance
and accuracy, as shown in Fig. 6(a). The object detection decoder regresses
objects center points’ heatmap and offsets, as well as object’s height and
width, to fulfill the objects’ correct location. 3D object detection is
abandoned because of the difficulty of acquiring the large scale of 3D labeled
truth data and unsatisfactory real-time performance.

Figure 7: The structure of stationary vehicle boundary generation method. Best
view in color and zoom in.

##### Line fitting

From segmentation results, we detect three kinds of line landmarks, including
parking lines, lane lines, and median lines. The parking lines contain
horizontal and longitudinal lines, where longitudinal parking lines are along
the heading direction, beneficial to automatic parking controlling; while
horizontal lines are perpendicular to the car’s heading direction, which
contains little useful information for driving and parking. Line detection
should strive for both quality and speed. So we propose a simple and ingenious
line detection algorithm to improve performance on embedded devices.

Firstly we perform the lane line and median line’s detection according to
semantic segmentation results. Then horizontal scan lines with equal intervals
traverse these landmarks. Every horizontal line creates some intersection
points with the lane line or median line. We count every horizontal line’s
intersection points’ number and get the most occurrences of the intersection
points’ number. We choose the horizontal lines with the most occurrences of
intersection numbers and select their intersection points to fit the lane line
and median line by the least square method. At last, we calculate the
vectorized information (slope and intercept) of the fitted line landmarks, and
the rotation angle based on the longitudinal axis of the ego-vehicle.
Algorithm 1 shows the detailed steps.

Algorithm 1 Lane line and median line’s fitting

Input: Semantic segmentation results
Output: Vectorized information (slope and intercept)
Step 1: Applying horizontal scan lines $\\{l_{1},l_{2},\ldots,l_{n}\\}$ with
equal intervals to traverse the semantic segmentation results. Every
horizontal line returns $\\{c_{1},c_{2},\ldots,c_{n}\\}$ intersection points.
Step 2: Counting most occurrences of the intersection points’ number
$\\{c_{1},c_{2},\ldots,c_{n}\\}$, and getting $c_{*}$
Step 3: Reserving the horizontal scan lines with $c_{*}$ intersection points’:
$\\{{l_{1}}^{*},{l_{2}}^{*},\ldots,{l_{m}}^{*}\\}$ and abandoning the others.
Step 4: Obtaining the intersection points from
$\\{{l_{1}}^{*},{l_{2}}^{*},\ldots,{l_{m}}^{*}\\}$. Every ${l_{i}}^{*}$
($i\in\\{1,2,\ldots,m\\}$) can obtain $c_{*}$ intersection points:
$\\{{p_{i1}},{p_{i2}},\ldots,{p_{i(c_{*}-1)}},{p_{{ic_{*}}}}\\}$ in sequence.

Step 5: Fitting lines. Selecting $\\{{p_{i1}},{p_{i2}}\\}$ to fit the first
line’s vectorized information by the least square method, since one line could
create two adjacent intersection points. Completing the $c_{*}/2$ lines’
fitting using the same procedure.

Next, we carry out the detection of the longitudinal parking lines. We rotate
the image by the rotation angle and traverse the parking lines longitudinally
to detect longitudinal parking lines’ points without the vertical lines’
disturbance. Then we rotate back the images and use the points after rotation
to fit the longitudinal parking lines with the least square method and get the
longitudinal parking lines’ vectorized information.

For stationary vehicle boundary lines, after obtaining the bounding boxes of
the vehicles from the object detection decoder, we regard the midpoints of the
lower bottom of the vehicle boxes as the key points and acquire their pixel
coordinates in the image through post-processing. These points are special
since their coordinates in the ego-vehicle coordinate system are $P(x,y,0)$.
After obtaining the pixel coordinates of the key points, we can get these
physical coordinates in the real-world coordinates through the Fisheye IPM
algorithm mallot1991inverse .

With the help of the distance calibration table, we can obtain the coordinate
points of stationary vehicles closest to the ego-vehicle in the lateral
direction. Finally, the line passing through these points and parallel to the
ego-vehicles heading direction is the stationary vehicle’s boundary line. The
red line on the right side of Fig. 7 shows the visualization results.

###  3.3 Filtering backend

Our filtering backend endows the system with the ability of refining initial
line landmark detection. While our LineMarkNet effectively detects line
landmarks, there is still a non-negligible amount of noise and outliers
present in initial outputs. The basic idea behind our novel backend is that,
with a carefully designed filter, our filter backend smooths the initial
results by encouraging the consistency of line landmarks in two aspects: (1)
multiview consistency: Detection from multi-view images should be consistent.
and (2) temporal consistency:  Detection from temporal frames should be
consistent. To be more specific, for the former, our backend requires
consistent line landmarks detection from four fisheye cameras. For the latter,
it encourages the temporal consistency via Kalman filtering where we estimate
the inconsistency between Kalman’s prediction based on previously detected
line landmark and current detection.

In some sense, our filtering is similar to existing works kalman1 ; kalman2 ;
kalman3 ; qin2020avp . However, these existing works are targeted for their
specific tasks like odometry, which can’t be transferred directly to our line
landmark detection. We customize the filtering, aiming for the new problem in
valet parking – i.e., line landmarks detection.

Our experimental results show that, by enforcing these two consistencies, our
filtering backend achieves a more stable and smooth line landmarks detection.
Moreover, the filtering backend, thanks to its lightweight nature, is very
efficient and thus enables our entire line landmark detection system to be
real-time.

We will elaborate multi-view and temporal consistency as below.

####  3.3.1 Multi-view consistency

Our backend enforces the multi-view consistency by associating objects
observed in multiple view cameras. Thus, apart from the stationary vehicle
boundary line, the other three line landmarks estimated in the BEV space don’t
require this step. Nevertheless, it’s noteworthy that BEV based line landmarks
theoretically guarantee the multi-view consistency since multi-view
information is fused before the detection. Therefore, to enforce the multi-
view consistency for the stationary vehicle boundary line, we associate
vehicles detected from different cameras via the defined association scores.
Specifically, we estimate the consistency of two detections from different
cameras as the distance of midpoints projected to the ego-vehicle coordinate
system whereby two stationary vehicles are associated if its’ midpoints
distance is smaller than 25cm.

####  3.3.2 Temporal consistency

We base temporal consistency on the Kalman filter algorithm where, with motion
equations, our backend predicts line landmarks at the current time frame.
Following the spirit of the Kalman filter, we construct the vehicle motion
equation that allows us to predict the current multi-dimensional state of line
landmarks. Specifically, the multi-dimensional state ($s$) consist of line
landmarks’ center point (i.e., center pixels of $c$) and parameters (i.e.,
intercept $\theta$ and slope $\beta$). Denote the predicted multi-dimensional
state at time frame $t$ as

$\displaystyle\tilde{s}_{t}=[\tilde{c}_{t}\mathchar
24635\relax\;\tilde{\theta}_{t}\mathchar 24635\relax\;\tilde{\beta}_{t}]$ (3)

Thus, our backend, in some sense, tracks the line landmarks via multi-
dimensional information. Note that additional dimensions would help. But we
empirically observe that center points and line parameters are sufficient to
achieve stable and smooth line landmarks detection. Moreover, the state $s$
has a small footprint which is the key reason why our system is capable of
being real-time.

In the meantime, our network detects the line landmarks at time $t$ where we
obtain the state $s_{t}$ from the current given surround-view images.
Accordingly, we quantify the inconsistency ($\sigma$) as

$\displaystyle\sigma_{t}=\lambda_{1}\mid\mid
c_{t}-\tilde{c}_{t}\mid\mid+\lambda_{2}\mid\mid\theta_{t}-\tilde{\theta}_{t}\mid\mid+\lambda_{3}\mid\mid\beta_{t}-\tilde{\beta}_{t}\mid\mid$
(4)

where weights $\lambda_{1}$, $\lambda_{2}$ and $\lambda_{3}$ adjust the
contribution of different dimensions on the final inconsistency . By doing so,
our filtering backend removes the line landmarks detected at the current frame
via thresholding on the inconsistency. Specifically, for any line landmark, we
calculate its inconsistency and remove it from detection if its inconsistency
$\sigma$ is larger than a predefined threshold. In this way, our system can
filter out the incorrect line landmarks that are quite different from previous
detections.

Note that, slightly different from the typical object tracking that requires
the association process – i.e., new detection needs to be associated with old
detection before calculating the inconsistency, our backend takes the
advantage of the unique nature of line landmarks. That being said that we
mildly assume that our line landmarks, by design, have a single instance
present in the given scene and thus can eliminate the association process.
Even though, it’s noteworthy that it’s not nontrivial to add the association
process into our backend by leveraging our multi-dimensional state.

##  4 Experiments

In this section, we first detail our experimental setup. Then, we provide
quantitative and qualitative results to validate the effectiveness of our
method. We further demonstrate the ablation study on different components of
our method. The results show that our method achieves effective yet efficient
line landmark detection.

###  4.1 Experimental Setup

####  4.1.1 Dataset

To validate our method, we collect a large-scale dataset for line landmark
detection in valet parking. The dataset consists of 140k training samples and
400k testing samples. It’s noteworthy that, to handle a variety of parking
scenarios, we collect our data from over 400 parking lots distributed in
different places.

Figure 8: An example of stationary vehicles in surround-view images’ labelling
We annotate the bounding box for the stationary cars, by which we train the
modules for stationary vehicle boundary line detection.

Different from the general detection benchmark in which each sample is a
single image, we capture images using the surround-view camera system as
detailed in Fig. 4 where each sample consists of four fisheye images. And the
image resolution is $1280\times 960$. To train our system, we annotate our
dataset with two types of labels – i.e., the bounding box of stationary
vehicle Fig. 8 and pixel-wise label map of line landmarks in BEV space as
shown in Fig. 9. Note that the stationary vehicle boundary line’s detection
depends on the vehicle boxes’ key points (midpoints of the lower bottom), so
the annotation for the stationary vehicle boundary line is equal to the
annotation for the bounding box of stationary vehicles.

Figure 9: An example of line landmark’s labelling in BEV image. We annotate
the line landmark in the style of pixel-wise style.

####  4.1.2 Implementation details

We implement our system on the Qualcomm 820A platform with an Adreno 530 GPU
and a Hexagon 680 DSP. The platform provides the computation power of 1.2
trillion operations per second (TOPS, Tera Operations Per Second) which is
weaker than the popular NVIDIA Xavier (40 TOPS) and Tesla’s V3 Full self-
driving computer claims (144 TOPS). Even though, our system achieves real-time
line landmark detection. In addition, we reveal the settings of hyper-
parameters. We input the surround-view images with 960 pixels of $H_{S}$ and
1280 pixels of $W_{S}$. For the BEV images, the $H_{B}$ is 600 and $W_{B}$ is
480. The $h_{m}$ and $w_{m}$ contains four levels with 1/4, 1/8, 1/16 and 1/32
of the BEV image shape ($H_{B}$, $W_{B}$). We set the $\lambda_{1}$,
$\lambda_{2}$ and $\lambda_{3}$ with 1.0, 1.0 and 1.0.

###  4.2 Performance of our line landmark detection system

Our method achieves effective and efficient line landmark detection. We detail
the performance in terms of accuracy and efficiency.

####  4.2.1 Accuracy

We evaluate the accuracy of line landmark detection using false detection rate
(FD) and missed detection rate (MD). We define FD as the ratio of incorrect
detected line landmarks (e.g., line landmarks with the wrong line parameters.)
and MD as the ratio of missing detection. FD and MD are critical metrics to
measure the accuracy of line landmark detection systems. Typically, the valet
parking system requires very low FD and MD. We don’t test on real distance
error. Real distance might be a good metric for some tasks. Nevertheless, in
our case, we use the rate because we empirically observed that, in some cases,
the real error distance could be very large, which dominates the final number
and leads to a minor difference between methods. Thus, we choose to use the
ratio as our metric.

As shown in Tab. 1, our method achieves the very low MD and FD in the test
set. We achieve the low FD smaller than 3 percent and the lower MD smaller
than 1 percent for all types of line landmarks. The very small MD means that
we have a very high recall in terms of line landmark detection, which is
critical to assure safety. For instance, we achieve nearly zero MD (0.09) to
detect the stationary vehicle boundary, in which case, our system demonstrates
very high accuracy in localizing the boundary between drivable and non-
drivable areas. It’s the key to guaranteeing collision-free valet parking.

To further validate the effectiveness of our system, we adapt the existing
works to our benchmark such that we have baselines for our method including
UFLD qin2020ultra  and DMPR-PS huang2019dmpr . Note that, unlike our method,
those baselines don’t aim for the detection of various line landmarks. We thus
carefully adapt other methods to our benchmark. Nevertheless, all baselines,
as shown in Tab. 2, are still not applicable to line landmark detection. On
the other hand, thanks to the carefully designed modules, our method achieves
superior accuracy, where our multi-task architecture fully fuses the surround-
view features and BEV features to produce better line detection with improved
multi-task decoders and line fitting module.

Table 1: The line landmark detection results with our dataset. Type | FD (%) | MD (%)
---|---|---
Lane line | 1.69 | 0.17
Parking line | 2.83 | 0.17
Median line | 1.20 | 0.13
Stationary vehicle boundary line | 1.07 | 0.09
Table 2: Comparison in landmark detection using metrics of FD (%) / MD (%). Methods | UFLD qin2020ultra  | DMPR-PShuang2019dmpr  | Ours
---|---|---|---
Lane line | 0.15/98.41 | - | 1.69/0.17
Parking line | 0.17/99.51 | 2.53/0.88 | 2.83/0.17
Median line | 1.02/97.74 | - | 1.20/0.13
Stationary vehicle boundary line | - | - | 1.07/0.09

Specifically, we utilize UFLD qin2020ultra  to detect three physical lines by
replacing the instance segmentation with semantic segmentation (i.e., same as
line detection module). Regarding the DMPR-PS huang2019dmpr  – a method for
parking slot detection, we detect parking lines by applying our line fitting
module after the parking slot detection. Experimental results show that, not
surprisingly, our method achieves significantly better performance. It’s
noteworthy that, there are no existing works for stationary vehicle boundary
line detection since they propose mental line is novel. Therefore, our method,
with baselines being very worse or even inapplicable in line landmark
detection, is significant to our research community given the importance of
line landmark detection in valet parking.

Figure 10: Elapsed time of our system on Qualcomm 820A platform. We estimate
the time for different time-consuming parts in our system. The total elapsed
time of our system is less than 45 million seconds (ms) for each sample. Note
that 60 ms is the minimum requirement for real-time detection.

####  4.2.2 Efficiency

Our line detection system is very efficient. We perform real-time detection,
which is critical for valet parking. To quantify the efficiency, we estimate
the elapsed time in the Qualcomm 820A platform with weak computation ability.

As shown in Fig. 10, we estimate the elapsed time for the time-consuming parts
in our system including the input pipeline (input images captured by four-way
cameras, image stitching (i.e., BEV image generation), line landmark
detection, and result publish. The total elapsed time is less than 45 ms for
each sample, which enables real-time detection where 60 ms/sample is the upper
bound (minimum requirement) for time efficiency. It’s noteworthy that our line
landmark detection (i.e., our line landmark detection system) is only (27.3
ms/sample) since the process of images stitching is time-consuming.

###  4.3 Ablation study

We perform the ablation study to validate the core components of the system
including multi-task architecture, transformer-based network, and our
filtering backend.

####  4.3.1 Multi-task Architecture

As shown in Tab. 3, the multi-task architecture achieves a better performance
than the single-task architecture, where the multiple tasks are complementary
to each other when we perform the tasks together. We achieve the robust fused
features not only for the BEV-specific task (BEV semantic segmentation) but
also for the image-feature-specific task (object detection).  Note that, for
the sake of simplicity, we quantify the performance using accuracy (i.e.,
Accuracy = 1 - MD - FD).

Table 3: Accuracy(%) of single semantic segmentation task, single object
detection task and multi tasks.

Type of line landmarks | SS | OD | Multi tasks
---|---|---|---
Lane line | 97.86 | - | 98.14
Parking line | 96.59 | - | 97.00
Median line | 97.92 | - | 98.67
Stationary vehicle boundary line | - | 98.41 | 98.84

  * *

Note SS is semantic segmentation, OD is object detection.

####  4.3.2 Semantic Segmentation

As shown in Tab. 4, we compare different semantic segmentation decoder designs
for line landmarks, and it reveals that our transformer-based structure for
semantic segmentation task outperforms other variants including DeepLabV3+
chen2018encoder  and the recent Swin liu2021swin . Same as Tab. 3, we utilize
the metric of accuracy to evaluate the performance. The result shows that our
transformer-based network leads to the best line landmark detection, and
validates the effectiveness of our hierarchical level graph reasoning
transformer to enhance the global and local relation modeling, which
contributes to the long and ambiguous line contents detection.

Table 4: Accuracy(%) of different semantic segmentation decoder designs. ‘Time’ denotes the time consumption on the NVIDIA GeForce RTX 4080. Method | Lane line | Parking line | Median Line | Time(ms)
---|---|---|---|---
DeepLabv3+ chen2018encoder  | 94.84 | 94.19 | 93.96 |  $\sim$12
Swin liu2021swin  | 96.38 | 96.21 | 96.45 |  $\sim$40
Ours | 97.86 | 96.59 | 97.92 |  $\sim$48

To explore the effectiveness of our graph transformer, we conduct the
experiment on the public Cityscapes dataset cityscapes , as shown in Tab. 5.
To suit the large-scale dataset, we change the larger backbone together with
compared methods, and the performance validates our graph transformer’s
advancement.

Table 5: The performance of methods on the public Cityscapes dataset. Method | Backbone | val mIoU | test mIoU
---|---|---|---
DeepLabv3+ chen2018encoder  | ResNet-101 | 79.3 | 80.1
Swin liu2021swin  | Swin-L | 82.3 | 81.3
Ours | Swin-L | 82.9 | 81.9

####  4.3.3 Object Detection

We also evaluate the object detection task (though it’s an intermediate
result) on the surround-view valet parking dataset FPD wu2022surround , where
we select the IoU criterion of 0.7 for object detection metrics: Average
Precision (AP) and Average Recall (AR), indicating $AP_{2D}$ and $AR_{2D}$. As
shown in Tab. 6, for anchor-based RetinaNet article3 , it receives better
$AP_{2D}$, but our anchor-free CenterNet follows the center-based structure
duan2019centernet  which directly predict 2D bbox center and dimension, which
achieves well-matched $AP_{2D}$ and better $AR_{2D}$.

Table 6: The performance of methods on the FPD dataset. Highest result is marked with red. ‘Time’ denotes the time consumption on the NVIDIA GeForce RTX 4080. Method | $AP_{2D}$ | $AR_{2D}$ | Time(ms)
---|---|---|---
RetinaNet article3  | 45.6 | 43.6 |  $\sim$7
CenterNet(Ours) duan2019centernet  | 45.3 | 45.4 |  $\sim$5

####  4.3.4 Filtering Backend

We ablate our filtering backend. In order to benchmark the impact of the
filtering backend, we calculate the slope/intercept error ($\delta C_{0}$ and
$\delta C_{1}$) of a sequence of samples where the error is defined as the
offset between prediction and ground-truth parameters.

Figure 11:  Effectiveness of filtering backend: we show, with our filtering
backend (b), our system achieves a lower error than the system without
filtering backend (a).

As illustrated in Fig. 11, we sequentially sample 400 frames and plot the
error curves. We show that, through considering both temporal and multi-view
consistency, our system effectively filters out the outlier detection with
large errors. More importantly, we observe that, with our novel filtering
backend, our system achieves a more stable and reliable line landmark
detection given a sequence of surround-view frames. Note that, while achieving
better performance, our filtering backend is still very efficient as evidenced
in our efficiency experiments.

Figure 12:  Failure cases –  (a) Line landmarks are perpendicular to the
vehicle’s heading direction; (b) median line under poor light conditions; (c)
parking line and stationary vehicle’s boundary line under poor light
conditions; (d) lane line under poor light conditions.

###  4.4 Failure cases

Fig. 12 demonstrates some failure cases. Specifically, Fig. 12 (a) illustrates
that our method fails when line landmarks are perpendicular to the vehicle’s
heading direction. As shown in Fig. 12 (b), (c), and (d), our method fails to
detect the median line, parking line, lane line, and stationary vehicle’s
boundary line under extremely poor light conditions (e.g., dark environments).
Our model doesn’t work well under some special scenes like the above
perpendicular condition and the poor light conditions, thus in the future we
will collect more extreme pool conditions to improve the dataset’s diversity
and design more clever blocks to avoid failure cases. In addition, in future
work, we will upgrade the fitting algorithm to save more computation and
consider more special scenes.

##  5 DISCUSSION

In this section, we will systematize the experimental results and analyze the
causes. (i) For accuracy experiments, we exhibit our method’s performance
(Tab. 1) and its comparison with existing works (Tab. 2) on our benchmark. Our
approach achieves pretty great accuracy, especially the low missed detection
rate (MD), while the existing works reveal inapplicable in our line landmark
detection, thus receiving sub-optimal performance. (ii) For efficiency, we
detail the elapsed time of our system on the Qualcomm 820A platform (Fig.10),
which is less than 45 million seconds for each sample. Note that 60 ms is the
minimum requirement for real-time detection. (iii) For the multi-task
architecture ablation study (Tab. 3), we find that the multi-task architecture
achieves more robust performance than the single-task one since the multiple
tasks can promote each other by sharing helpful and meaningful features. (iv)
For the semantic segmentation task, we compare our approach with other
existing approaches in Tab. 4. Our method harvests the best line landmark
detection, for our hierarchical level graph reasoning transformer enhances the
global and local relation modeling. In addition, in public dataset evaluation
(Tab. 5), our approach still can make an advantage. (v) For the object
detection task, we evaluate our center-based structure with the anchor-based
RetinaNet (Tab. 6), which remains a well-matched performance with less time
consumption. (vi) The comparison with and without filtering (Fig. 11) succeeds
in confirming our filtering backend’s effectiveness, where the offset is
obviously inhibited with our beneficial filtering for multi-view consistency
and temporal consistency.

In total, our method fulfills the superiority on both accuracy and efficiency
compared with other existing approaches, which contributes to our elaborate
designs, i.e. the multi-task architecture, the graph transformer for semantic
segmentation task, the center-based CenterNet for object detection task and
filtering backend for multi-view and temporal consistency, where our ablation
studies have certified their effectiveness.

##  6 CONCLUSIONS

In this work, we present an effective yet efficient line landmark detection
system for valet parking. To validate our method, we further propose a
benchmark where we define four types of line landmarks, serving as the
cornerstone for self-driving in a variety of parking lots. Experimental
results show that our system achieves low error rates of detection while being
real-time on a low-cost computing platform.

While we achieve the promising result, the error rate is still not zero
whereas the perfect performance is critical for realistic valet parking. Thus,
in the future, we would like to further improve modules in our system – e.g.,
we further improve the generalization ability via enforcing inductive bias.
Additionally, we would like to extend our method for generic detection – apart
from line landmarks, other landmarks would be also important such as parking
slots, pedestrians, and road curbs to name a few. Note that it’s non-trivial
for those extensions even detection has been very successful in autonomous
driving since, as discussed aforementioned, visual perception is much more
challenging in valet parking environments than in autonomous driving areas.

## References

  *   * (1) Song, X., Wang, P., Zhou, D., Zhu, R., Guan, C., Dai, Y., Su, H., Li, H., Yang, R.: Apollocar3d: A large 3d car instance understanding benchmark for autonomous driving. In: Conference on Computer Vision and Pattern Recognition, pp. 5452–5462 (2019). IEEE
  * (2) Zhou, D., Fang, J., Song, X., Liu, L., Yin, J., Dai, Y., Li, H., Yang, R.: Joint 3d instance segmentation and object detection for autonomous driving. In: Conference on Computer Vision and Pattern Recognition, pp. 1839–1849 (2020)
  * (3) Wu, P., Chen, S., Metaxas, D.N.: Motionnet: Joint perception and motion prediction for autonomous driving based on bird’s eye view maps. In: Conference on Computer Vision and Pattern Recognition, pp. 11385–11395 (2020). IEEE
  * (4) Lee, D.-K., Shin, J.-S., Jung, J.-H., Park, S.-J., Oh, S.-J., Lee, I.-S.: Real-time lane detection and tracking system using simple filter and kalman filter. In: International Conference on Ubiquitous and Future Networks, pp. 275–277 (2017). IEEE
  * (5) Lee, M., Lee, J., Lee, D., Kim, W., Hwang, S., Lee, S.: Robust lane detection via expanded self attention. In: Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 533–542 (2022)
  * (6) Hu, C., Hudson, S., Ethier, M., Al-Sharman, M., Rayside, D., Melek, W.: Sim-to-real domain adaptation for lane detection and classification in autonomous driving. In: 2022 IEEE Intelligent Vehicles Symposium (IV), pp. 457–463 (2022). IEEE
  * (7) Chen, S., Zhang, N., Sun, H.: Collaborative localization based on traffic landmarks for autonomous driving. In: International Symposium on Circuits and Systems, pp. 1–5 (2020). IEEE
  * (8) Wu, Z., Sun, W., Wang, M., Wang, X., Ding, L., Wang, F.: Psdet: Efficient and universal parking slot detection. In: Intelligent Vehicles Symposium, pp. 290–297 (2020). IEEE
  * (9) Qin, Z., Zhang, P., Li, X.: Ultra fast deep lane detection with hybrid anchor driven ordinal classification. IEEE Transactions on Pattern Analysis and Machine Intelligence (2022)
  * (10) Wang, J., Ma, Y., Huang, S., Hui, T., Wang, F., Qian, C., Zhang, T.: A keypoint-based global association network for lane detection. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1392–1401 (2022)
  * (11) Feng, Z., Guo, S., Tan, X., Xu, K., Wang, M., Ma, L.: Rethinking efficient lane detection via curve modeling. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 17062–17070 (2022)
  * (12) Zheng, T., Huang, Y., Liu, Y., Tang, W., Yang, Z., Cai, D., He, X.: Clrnet: Cross layer refinement network for lane detection. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 898–907 (2022)
  * (13) Pauls, J.-H., Schmidt, B., Stiller, C.: Automatic mapping of tailored landmark representations for automated driving and map learning. In: International Conference on Robotics and Automation, pp. 6725–6731 (2021). IEEE
  * (14) Wang, B., Stafford-Fraser, Q., Robinson, P., Dias, E., Skrypchuk, L.: Landmarks based human-like guidance for driving navigation in an urban environment. In: International Conference on Intelligent Transportation Systems, pp. 1–6 (2017). IEEE
  * (15) Liu, L., Chen, X., Zhu, S., Tan, P.: Condlanenet: a top-to-down lane detection framework based on conditional convolution. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 3773–3782 (2021)
  * (16) Tabelini, L., Berriel, R., Paixao, T.M., Badue, C., De Souza, A.F., Oliveira-Santos, T.: Keep your eyes on the lane: Real-time attention-guided lane detection. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 294–302 (2021)
  * (17) Baek, I., Davies, A., Yan, G., Rajkumar, R.R.: Real-time detection, tracking, and classification of moving and stationary objects using multiple fisheye images. In: Intelligent Vehicles Symposium, pp. 447–452 (2018). IEEE
  * (18) Wu, Z., Zhang, W., Wang, J., Wang, M., Gan, Y., Gou, X., Fang, M., Song, J.: Disentangling and vectorization: A 3d visual perception approach for autonomous driving based on surround-view fisheye cameras. In: IEEE/RJS International Conference on Intelligent RObots and Systems, pp. 5576–5582 (2021). IEEE
  * (19) Toromanoff, M., Wirbel, E., Wilhelm, F., Vejarano, C., Perrotton, X., Moutarde, F.: End to end vehicle lateral control using a single fisheye camera. In: IEEE/RJS International Conference on Intelligent RObots and Systems, pp. 3613–3619 (2018). IEEE
  * (20) Kumar, V.R., Yogamani, S., Rashed, H., Sitsu, G., Witt, C., Leang, I., Milz, S., Mäder, P.: Omnidet: Surround view cameras based multi-task visual perception network for autonomous driving. IEEE Robotics and Automation Letters 6(2), 2830–2837 (2021)
  * (21) Zhang, C., Liu, H., Xie, Z., Yang, K., Guo, K., Cai, R., Li, Z.: Avp-loc: Surround view localization and relocalization based on hd vector map for automated valet parking. In: IEEE/RJS International Conference on Intelligent RObots and Systems, pp. 5552–5559 (2021). IEEE
  * (22) Kumar, V.R., Hiremath, S.A., et al.: Fisheyedistancenet: Self-supervised scale-aware distance estimation using monocular fisheye camera for autonomous driving. In: International Conference on Robotics and Automation, pp. 574–581 (2020). IEEE
  * (23) Yahiaoui, M., Rashed, H., Mariotti, L., Sistu, G., Clancy, I., Yahiaoui, L., Kumar, V.R., Yogamani, S.: Fisheyemodnet: Moving object detection on surround-view cameras for autonomous driving. In: International Conference on Computer Vision Workshop (2019)
  * (24) Strobel, K., Zhu, S., Chang, R., Koppula, S.: Accurate, low-latency visual perception for autonomous racing: Challenges, mechanisms, and practical solutions. In: 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 1969–1975 (2020). IEEE
  * (25) Qin, T., Chen, T., Chen, Y., Su, Q.: Avp-slam: Semantic visual mapping and localization for autonomous vehicles in the parking lot. In: IEEE/RJS International Conference on Intelligent RObots and Systems, pp. 5939–5945 (2020). IEEE
  * (26) Ouyang, Z., Hu, L., Lu, Y., Wang, Z., Peng, X., Kneip, L.: Online calibration of exterior orientations of a vehicle-mounted surround-view camera system. In: International Conference on Robotics and Automation, pp. 4990–4996 (2020). IEEE
  * (27) Wang, Y., Huang, K., Peng, X., Li, H., Kneip, L.: Reliable frame-to-frame motion estimation for vehicle-mounted surround-view camera systems. In: International Conference on Robotics and Automation, pp. 1660–1666 (2020). IEEE
  * (28) Blott, G., Takami, M., Heipke, C.: Semantic segmentation of fisheye images. In: European Conference on Computer Vision Workshop, pp. 181–196 (2018)
  * (29) Kumar, V.R., Klingner, M., Yogamani, S., Milz, S., Fingscheidt, T., Mader, P.: Syndistnet: Self-supervised monocular fisheye camera distance estimation synergized with semantic segmentation for autonomous driving. In: Winter Conference on Applications of Computer Vision, pp. 61–71 (2021). IEEE
  * (30) Ye, Y., Yang, K., Xiang, K., Wang, J., Wang, K.: Universal semantic segmentation for fisheye urban driving images. In: International Conference on Systems, Man, and Cybernetics, pp. 648–655 (2020). IEEE
  * (31) Zaffaroni, M., Grangetto, M., Farasin, A.: Estimation of speed and distance of surrounding vehicles from a single camera. In: International Conference on Image Analysis and Processing, pp. 388–398 (2019). Springer
  * (32) Komatsu, R., Fujii, H., Tamura, Y., Yamashita, A., Asama, H.: 360 depth estimation from multiple fisheye images with origami crown representation of icosahedron. In: IEEE/RJS International Conference on Intelligent RObots and Systems, pp. 10092–10099 (2020). IEEE
  * (33) Plaut, E., Yaacov, E.B., Shlomo, B.E.: Monocular 3d object detection in cylindrical images from fisheye cameras. arXiv preprint arXiv:2003.03759 (2020)
  * (34) Li, T., Tong, G., Tang, H., Li, B., Chen, B.: Fisheyedet: A self-study and contour-based object detector in fisheye images. IEEE Access 8, 71739–71751 (2020)
  * (35) Xiao, J., Xiong, W., Yao, Y., Li, L., Klette, R.: Lane detection algorithm based on road structure and extended kalman filter. International Journal of Digital Crime and Forensics 12(2), 1–20 (2020)
  * (36) Hu, J., Yang, M., Xu, H., He, Y., Wang, C.: Mapping and localization using semantic road marking with centimeter-level accuracy in indoor parking lots. In: International Conference on Intelligent Transportation Systems, pp. 4068–4073 (2019). IEEE
  * (37) Qin, T., Li, P., Shen, S.: Vins-mono: A robust and versatile monocular visual-inertial state estimator. IEEE Transactions on Robotics 34(4), 1004–1020 (2018)
  * (38) Marzougui, M., Alasiry, A., Kortli, Y., Baili, J.: A lane tracking method based on progressive probabilistic hough transform. IEEE access 8, 84893–84905 (2020)
  * (39) Qin, Z., Wang, H., Li, X.: Ultra fast structure-aware deep lane detection. In: European Conference on Computer Vision, pp. 276–291 (2020). Springer
  * (40) Huang, J., Zhang, L., Shen, Y., Zhang, H., Zhao, S., Yang, Y.: Dmpr-ps: a novel approach for parking-slot detection using directional marking-point regression. In: IEEE International Conference on Multimedia and Expo, pp. 212–217 (2019). IEEE
  * (41) Song, M., Kim, C., Kim, M., Yi, K.: Robust lane tracking algorithm for forward target detection of automated driving vehicles. Proceedings of the Institution of Mechanical Engineers, Part D: Journal of automobile engineering 233(7), 1930–1949 (2019)
  * (42) Zhang, Z.: A flexible new technique for camera calibration. IEEE Transactions on Pattern Analysis and Machine Intelligence 22(11), 1330–1334 (2000)
  * (43) Yu, F., Wang, D., Shelhamer, E., Darrell, T.: Deep layer aggregation. In: Computer Vision and Pattern Recognition, pp. 2403–2412 (2018). IEEE
  * (44) Bruna, J., Zaremba, W., Szlam, A., LeCun, Y.: Spectral networks and locally connected networks on graphs. In: International Conference on Learning Representations (2014). OpenReview.net
  * (45) Liu, Z., Lin, Y., Cao, Y., Hu, H., Wei, Y., Zhang, Z., Lin, S., Guo, B.: Swin transformer: Hierarchical vision transformer using shifted windows. In: IEEE International Conference on Computer Vision, pp. 10012–10022 (2021). IEEE
  * (46) Zhou, X., Wang, D., Krähenbühl, P.: Objects as points. arXiv preprint arXiv:1904.07850 (2019)
  * (47) Mallot, H.A., Bülthoff, H.H., Little, J., Bohrer, S.: Inverse perspective mapping simplifies optical flow computation and obstacle detection. Biological Cybernetics 64(3), 177–185 (1991)
  * (48) Cen, R., Jiang, T., Tan, Y., Su, X., Xue, F.: A low-cost visual inertial odometry for mobile vehicle based on double stage kalman filter. Signal Processing, 108537 (2022)
  * (49) Bloesch, M., Omari, S., Hutter, M., Siegwart, R.: Robust visual inertial odometry using a direct ekf-based approach. In: IEEE/RJS International Conference on Intelligent RObots and Systems, pp. 298–304 (2015). IEEE
  * (50) Faessler, M., Fontana, F., Forster, C., Mueggler, E., Pizzoli, M., Scaramuzza, D.: Autonomous, vision-based flight and live dense 3d mapping with a quadrotor micro aerial vehicle. Journal of Field Robotics 33(4), 431–450 (2016)
  * (51) Chen, L.-C., Zhu, Y., Papandreou, G., Schroff, F., Adam, H.: Encoder-decoder with atrous separable convolution for semantic image segmentation. In: European Conference on Computer Vision, pp. 833–851 (2018). Springer
  * (52) Cordts, M., Omran, M., Ramos, S., Rehfeld, T., Enzweiler, M., Benenson, R., Franke, U., Roth, S., Schiele, B.: The cityscapes dataset for semantic urban scene understanding. In: Computer Vision and Pattern Recognition, pp. 3213–3223 (2016). IEEE
  * (53) Wu, Z., Gan, Y., Li, X., Wu, Y., Wang, X., Xu, T., Wang, F.: Surround-view fisheye bev-perception for valet parking: Dataset, baseline and distortion-insensitive multi-task framework. IEEE Transactions on Intelligent Vehicles (2022)
  * (54) Lin, T.-Y., Goyal, P., Girshick, R., He, K., Dollár, P.: Focal loss for dense object detection. In: IEEE Int. Conf. on Computer Vision (ICCV), pp. 2980–2988 (2017). IEEE
  * (55) Duan, K., Bai, S., Xie, L., Qi, H., Huang, Q., Tian, Q.: CenterNet: Keypoint triplets for object detection. In: IEEE Int. Conf. on Computer Vision (ICCV), pp. 6569–6578 (2019). IEEE