# RONELDv2: A faster, improved lane tracking method

Zhe Ming Chng, Joseph Mun Hung Lew, Jimmy Addison Lee Z. M. Chng is currently
with the College of Computing, Georgia Institute of Technology, GA, USA.
Email<EMAIL_ADDRESS>M. H. Lew and J. A. Lee are currently with Aison
Pte. Ltd., Singapore. Email: {joseph.lew<EMAIL_ADDRESS>
###### Abstract

Lane detection is an integral part of control systems in autonomous vehicles
and lane departure warning systems as lanes are a key component of the
operating environment for road vehicles. In a previous paper, a robust neural
network output enhancement for active lane detection (RONELD) method
augmenting deep learning lane detection models to improve active, or ego, lane
accuracy performance was presented. This paper extends the work by further
investigating the lane tracking methods used to increase robustness of the
method to lane changes and different lane dimensions (e.g. lane marking
thickness) and proposes an improved, lighter weight lane detection method,
RONELDv2. It improves on the previous RONELD method by detecting the lane
point variance, merging lanes to find a more accurate set of lane parameters,
and using an exponential moving average method to calculate more robust lane
weights. Experiments using the proposed improvements show a consistent
increase in lane detection accuracy results across different datasets and deep
learning models, as well as a decrease in computational complexity observed
via an up to two-fold decrease in runtime, which enhances its suitability for
real-time use on autonomous vehicles and lane departure warning systems.

###### Index Terms:

lane tracking, lane detection, autonomous driving

##  I Introduction

Lane detection research has seen increased interest in recent years due to its
importance in autonomous vehicles and advanced driver assistance systems
(ADAS). Lane markings, in particular active or ego lane markings, serve as
important markers that indicate the road lane the vehicle is currently
travelling on and can be used to constrain the maneuver of the vehicle on
public roads. This prevents collisions with other road users and provides the
vehicle with a better understanding of its surroundings.

To detect these lane markings, there has been much research on using different
methods such as using cameras and light detection and ranging (LiDAR) sensors
[1]. Camera-based lane detection has been a popular method due to the
disadvantages of other lane detection methods, such as the high cost of LiDAR
sensors [2]. It also more closely mimics how humans detect lanes using visual
cues and has seen increasing performance with the use of deep learning lane
detection models, which further enhance its attractiveness for lane detection
on vehicles. However, despite these advantages and recent improvements,
vision-based lane detection faces significant challenges due to complicated
driving scenarios with heavy traffic that occludes lane markings, poor weather
conditions that blur important contours, as well as shadows and other road
markings that could be easily mistaken for lane markings, etc. However, lane
detection also necessitates a real-time, lightweight solution that can be
placed on edge devices in autonomous vehicles while being able to respond
rapidly to changes in the dynamic and high-speed driving environment. While
recent state-of-the-art lane detection methods have adopted new innovations,
such as the attention mechanism [3] and in particular the self-attention
mechanism [4, 5], to address some of these concerns, it is observed that many
of these models still face difficulty adapting to datasets that differ
significantly from their train sets. This covariate shift arising from the
different distribution of train and test set data can be attributed to reasons
such as varying road surface conditions and different lane markings. This is a
cause for concern as autonomous vehicles must be equipped to respond to
different driving environments and scenarios that might not have been
presented in their train sets, hence a more robust method is required to
improve lane detection results in such scenarios.

In our previous paper, we introduced the RONELD method [6] which aimed to
improve lane detection results by enhancing the outputs of deep learning lane
detection models through a four-part process involving adaptive lane point
extraction, curved lane detection, lane construction, and lane tracking using
preceding frames. In this paper, we further investigate the lane tracking step
of the method and focus on observed rooms for improvement on the RONELD
method’s lane tracking step, such as the inability of RONELD to rapidly adapt
to lane markings that appeared later on in the video feed if the original lane
markings are still present in the frame and the constant distance relative to
the frame width used to match lanes between frames which does not adjust for
new lane marking thickness on different roads. We aim to further improve the
ability of RONELD to obtain accurate detected active lanes, while remaining
robust to changes in the road condition and driving environment.

To this end, we introduce RONELDv2, which incorporates the use of lane point
variance, lane merging, and an exponentially weighted moving average method to
compute weights in order to strive for a more robust and lightweight solution
which further improves on the RONELD method and increases its suitability for
real-time use on autonomous vehicles and ADAS. To verify the usefulness of our
proposed changes to the method, we test the proposed changes on the TuSimple
[7] and CULane [8] datasets and compare accuracy results with those achieved
in [6] as well as the two state-of-the-art models used in the RONELD paper,
namely SCNN [8] and ENet-SAD [5]. Our experiments demonstrate the
effectiveness of the changes, with increased accuracy results above those
achieved by RONELD and an up to two-fold decrease in the runtime compared to
an optimized version of RONELD. We present two simple before and after
comparisons of applying RONELDv2 onto the CULane and TuSimple test set images
in Fig. 1.

((a)) Before

((b)) After

((c)) Before

((d)) After

Figure 1: Two SCNN lane detection results using a CULane-trained model on the
CULane [8] and TuSimple [7] test sets are shown in (a) and (c) respectively,
detected lane marking in red, while ground truth lane markings are highlighted
in blue. Corresponding results after RONELDv2 are shown in (b) and (d)
respectively, highlighted in green.

We organize the rest of our paper as follows. Section II discusses related
work on camera-based lane detection. Section III explains our proposed method
in three parts: Lane point variance, weighing tracked lanes, and lane merging.
Section IV provides empirical evidence of the usefulness of our method, and
section V concludes our work.

##  II Related Work

###  II-A Traditional lane detection

A common process workflow for traditional lane detection algorithms utilizes a
three-step process: pre-processing, lane detection, and lane tracking. To pre-
process the images, multiple methods have been used such as edge detection
algorithms (e.g. Canny edge detector and Sobel operator), adaptive
thresholding, and inverse perspective mapping [9]. For the lane detection
step, there is a reliance on hand-crafted features, such as color-based
features [10], steerable filters [11], Hough transform [12, 13], random sample
consensus (RANSAC) [14, 15], and ridge features [16, 17]. Many traditional
lane detection algorithms apply some combination of these hand-crafted
features in a lane detection step on the pre-processed road image, before
applying a final tracking step. For the final tracking step, some algorithms
utilize tracking algorithms, with particle [18] or Kalman [14] filter being a
popular choice in the literature, to reduce noise in the lane marking outputs
and obtain more accurate lane estimates from the noisy input data. While these
methods are able to achieve good performance on datasets with certain
characteristics such as good weather, straight lanes, clear road markings,
etc., they generally lack robustness and fail to perform well on images that
do not follow underlying assumptions, which restricts possible driving
environments and conditions out of the many diverse driving conditions that
autonomous vehicles would have to encounter while in use. This could include
road images with heavy traffic which obstructs a clear view of the lane
marking or poor weather conditions, which significantly reduce the performance
of these traditional lane detection methods.

###  II-B Deep learning lane detection

Kim et al. introduced a convolutional neural network (CNN) method combined
with RANSAC to detect lanes in complicated road scenes [19]. Following that, a
dual-view convolutional neural network (DVCNN) method [20] was proposed by He
et al. which uses front-view and top-view images simultaneously to improve
lane detection precision by eliminating false detections arising from moving
vehicles, arrows, words, etc. Recently, end-to-end deep learning models [21,
22, 23] have seen an increase in popularity in lane detection research after
achieving state-of-the-art results on other computer vision tasks [24, 25, 26]
and with greater availability of large-scale lane detection datasets [7, 8,
27]. One common approach to address the lane detection problem has been
treating it as a semantic segmentation task and using CNNs to formulate dense
predictions for the road image, i.e. predict whether each pixel in the road
image is a part of a lane marking [5, 8, 20]. Lee et al. proposed a vanishing
point guided network (VPGNet) [28], a multi-task network looking at road and
lane markings as well as the vanishing point to improve lane detection under
adverse weather conditions. Later on, Pan et al. proposed a Spatial CNN (SCNN)
[8] method which enabled message passing between pixels across rows and
columns of a layer which allowed the method to learn the strong spatial
relationships amongst pixels across rows and columns of an image and won 1
place in the TuSimple Benchmark Lane Detection Challenge. Subsequently, Hou et
al. proposed a self attention distillation (SAD) [5] method which showed
compelling results when combined with the lightweight ENet [29], ResNet-18
[25], and ResNet-32 [25] models, while running 10 times faster than SCNN for
the SAD incorporated ENet model. More recently, Qin et al. [21] introduced a
method which treats the lane detection process as a row-based selection
problem using global features, achieving compelling results with the ResNet-34
model on the CULane [8] dataset while running around 23 times faster than
SCNN. Other methods have also approached the lane detection problem as an
instance segmentation task [30, 31] or model lanes in three dimensions [23],
the latter which has achieved competitive results even on the TuSimple dataset
[7], an image-only lane detection dataset. Some Generative Adversarial
Networks (GANs) [32] have also been recently introduced to address the lane
detection problem [33, 34], where the network benefits from seeing both real
and generated fake predictions at the same time to improve lane detection
results.

While the aforementioned methods have shown great promise in addressing the
lane detection problem in a greater range of driving scenarios, they still
face significant challenges adapting to lane images that are from a different
distribution than their test set (e.g. different traffic density, lane
markings, or road surface conditions). To illustrate this issue, we have
included probability maps from the cross-dataset validation of CULane-trained
models on the TuSimple dataset in Fig. 2. There have been some methods using
different techniques to enhance the performance of these models such as the
use of an additional vanishing point prediction task added to guide VPGNet
under adverse weather conditions, but they are usually paired for use with
specific models or lack robustness.

((a)) Input

((b)) SCNN

((c)) ENet-SAD

((d)) Input

((e)) SCNN

((f)) ENet-SAD

Figure 2: Probability map outputs from the CULane-trained SCNN and ENet-SAD
deep learning models on sample images from the TuSimple test set.

##  III Proposed Method

We explain our proposed method in this section. We build our contributions on
top of the RONELD method, specifically focusing on improving the lane tracking
portion of the method by making better use of information inherent in lane
detection outputs from the deep learning models in previous frames.

###  III-A Lane point variance

To enhance robustness to different lane marking thickness and different
abilities of lane detection models to localize the lane marking, our method
calculates the standard deviation of lane point positions from the output
probability map of the deep learning model. To do this, we model the
confidence values around the detected lane point as a normal distribution
populated by points lying along the normal to the gradient of the proposed
lane marking based on lane points detected by the RONELD method. This is
achieved by using

$c(x)=\frac{1}{\sigma\sqrt{2\pi}}e^{-\frac{1}{2}(\frac{x-\mu}{\sigma})^{2}},$
(1)

where $c(x)$ is the confidence of the point at $x$ based on Fig. 3, $\mu$ is
the mean of the distribution which in this case is the position of the
detected lane point, and $\sigma$ is the standard deviation of the lane point
position which we are searching for. These confidence markings are derived
from the semantic segmentation output of the lane detection models. The
detected lane point, which is the highest confidence point in the search area
from the RONELD method [6], occurs at $x=\hat{\mu}$, where
$\hat{\mu}=\operatorname*{arg\,max}_{x}c(x)$. If another point along the
normal is found with a higher confidence, we would use that point as the
detected lane point instead. Based on this model, $c(x)=e^{-\frac{1}{2}}c_{d}$
when $x=\mu\,\pm\,\sigma$, which our method uses to find $\sigma$ by finding
the shortest distance to a point with confidence $c(x)\leq
e^{-\frac{1}{2}}c_{d}$, where $c_{d}$ is the confidence of the relevant
detected lane point, along the normal to the gradient of the lane. We do this
for both the left and right side of the point and take the average of the two
distances as the standard deviation $\sigma$ of that point, with the lane
point variance as $\sigma^{2}$. We use the root mean square (RMS) of the
standard deviation of the lane points in a lane to determine the standard
deviation for the lane as a whole, which we use subsequently for our method.

$c_{d}$$e^{-\frac{1}{2}}c_{d}$$\mu-\sigma$$\mu$$\mu+\sigma$lane point location
along normal, $x$confidence, $c(x)$ Figure 3: Gaussian distribution graph with
mean $\mu$ and variance $\sigma^{2}$, where the detected lane point is at the
highest confidence point where $x=\mu$

We use this when determining if a lane from a previous frame and a lane in the
current frame are the same lane. Similar to RONELD, we calculate the RMS
distance between the lanes. However, unlike RONELD which uses a fixed distance
threshold based on image width to determine if lanes in previous and current
frames should be matched, we set the threshold based on the standard deviation
of the lane points instead, which allows our method to adjust for thicker lane
markings that likely have greater variance in their lane points since more
points fall within the lane width, or lanes that are detected with greater
positional uncertainty by the deep learning model as seen in Fig. 4. This is
critical as the lane points of these lanes are likely to see greater shift in
their proposed location from frame to frame, leading to greater uncertainty in
the lane parameters and hence greater distance despite being the same lane.

We also include the lane point variance information in the weights matrix for
weighted ordinary least squares linear regression used in RONELD to determine
the gradient and y-intercept for straight lanes,

$\bm{C}=\begin{pmatrix}\frac{c_{1}}{{\sigma_{1}}^{2}}&0&\dots&0\\\
0&\frac{c_{2}}{{\sigma_{2}}^{2}}&\dots&0\\\ \vdots&\vdots&\ddots&\vdots\\\
0&0&\dots&\frac{c_{m}}{{\sigma_{m}}^{2}}\end{pmatrix},$ (2)

where $c_{i}$ is the confidence of the $i$-th detected lane point, and $m$ is
the number of detected lane points. This is in contrast to the weights matrix
used in RONELD which consisted solely of the confidence of the lane points as
their respective weights. By taking into account the variance of the lane
points in addition to the confidence values when computing the weight matrix
for the weighted ordinary least squares linear regression step, our method
adjusts for points that have greater positional uncertainty, represented by a
higher lane point variance and lower confidence values. This further reduces
the problem of heteroskedasticity due to differences in variance of lane
points arising from factors such as the accuracy of the model on the dataset
and different lane marking thickness.

Figure 4: Comparison of a (a) well-detected and defined lane marking and (b)
less well-detected and defined lane marking. The blue points represent the
detected lane points, the green line represents the weighted ordinary least
squares regression line for the points and the red line indicates the normal
to the regression line between the two points, $\mu-\sigma$ and $\mu+\sigma$,
used in the calculation of $\sigma$ for each detected lane point.

###  III-B Weighing tracked lanes

After matching the two lanes together based on the line distance with a
threshold based on $\sigma$, we assign weights to each lane and the lanes with
the highest weight in the left and right half of the frame are identified as
the final lane marking outputs. In RONELDv2, we propose an exponentially
weighted moving average (EWMA) method to replace RONELD’s linear weight
increment method. The benefits of this new method are two-fold. Firstly, it
increases the robustness of the system to new lanes by limiting the relative
weight of old lanes compared to new lanes, where old lanes are lanes that have
appeared in many consecutive frames while new lanes are lanes that have
appeared in comparatively fewer frames. Secondly, by reducing the ratio of
lane weights between old and new lanes, it allows for old lanes that are not
detected in the current frame to be cleared from the cache of previous lanes
more rapidly, reducing processing times needed to process the cache of
previous lanes.

We calculate a current weight, $\omega_{f}$, for the lane in the current frame
$f$ using

$\omega_{f}=\psi\,c_{f}N_{f},$ (3)

where $\psi$ is the weight increment factor which is higher for identified
potential active lane markings and lower for nonactive lane markings, $c_{f}$
is the RMS confidence of detected lane points, and $N_{f}$ is the number of
lane points in frame $f$. This is similar to the weight increment used in
RONELD. Using the $\omega_{f}$ of the lane in frame $f$ and the EWMA weight of
the lane in preceding frames, we then calculate the current EWMA lane weight,
$\Omega_{f}$, which we obtain using

$\Omega_{f}=\alpha\,\omega_{f}+(1-\alpha)\,\Omega_{f-1},$ (4)

where $\alpha$ is a smoothing coefficient that represents the weight placed on
the current frame when computing the EWMA weight. We use $\Omega_{f}$ as the
final lane weight when determining the final lane marking outputs. The
function can be seen as a piecewise function changing at each integer $f$
value (with $\omega_{f}$ changing at each integer $f$ as well) and with the
gradient at each point calculated as

$\frac{d\,\Omega}{df}=\alpha(\omega_{f}-\Omega).$ (5)

We set $\alpha=0.5$ in this paper to place equal weight between the current
lane weight and the previous EWMA lane weight. We use the EWMA function
without bias correction to provide a weight advantage for lanes that have
appeared in comparatively more frames. By using an EWMA function, it reduces
the total weight of the lanes and imposes max$(\omega_{f})$ as an upper bound
on the weight of the lane and thereby limiting the weight of lanes despite
appearing in many preceding frames. This reduces the weight of old lanes more
rapidly and allows new lanes to quickly supersede old lanes while
simultaneously allowing us to remove the old lanes from the stored cache of
previous lanes, thereby reducing the processing required to handle these lanes
as well. Assuming a constant lane weight $\omega_{f}$ in each frame, this can
also be modelled as a geometric sequence where

$\Omega_{f}=\omega[1-(1-\alpha)^{n}],\,\omega=\omega_{f},$ (6)

where $n$ is the number of frames that lane has appeared in. The benefits of
this EWMA function can also be seen through an example in Fig. 5 using lanes
with constant weights. In the example, the higher-weighted lane with constant
weight 1.5 appears in frame 3 and overtakes the lower-weighted lane with
constant weight 1 more rapidly, in frame 4 instead of frame 6, while
converging towards the weight of the lane in each frame.

Figure 5: Comparison of a 5 linear weights method used in [6] against an 5 exponentially weighted moving average method ($\alpha=0.5$) used in our method. The red line represents the weight of a higher-weighted lane that appears from frame 3 onwards while the blue line represents the weight of a lower-weighted lane that appears from frame 1. The dots represent the weight of each lane in each frame. Figure 6: Illustration of ($r$, $\theta$) lane parameters used in lane merge step TABLE I: Basic information of datasets (TuSimple and CULane) used in our experiments. Dataset | # Total | # Test | # Test Labelled | Resolution | Environment | Traffic Density
---|---|---|---|---|---|---
TuSimple | 6,408 | 55,640 | 2,782 |  1280$\times$720 | Highway | Low
CULane | 133,235 | 34,680 | 34,680 |  1640$\times$590 | Urban, rural, highway | Varies

###  III-C Lane merging

To further enhance the ability of our method to adjust to noise in the lane
detection model’s semantic segmentation output, we use lane merging to obtain
current lane parameters from the current and previous frames. As a pre-
processing step, we convert the lane parameters obtained from the weighted
least squares linear regression to Hesse’s normal form with the line
parameters ($r$, $\theta$), where $r$ is the length of the perpendicular from
the origin to the line and $\theta$ is the angle between that perpendicular
and the $x$-axis. An illustration of the lane parameters is provided in Fig.
6. We use this form as it prevents a line with a large gradient from
dominating the lane parameters when merging the lane parameters together. As
an example, a vertical line has a gradient tending to infinity, and hence the
weighted average lane would adopt a gradient tending to infinity as well,
regardless of the gradient of the other lane being measured.

Using these lane parameters, we implement a lane merging method that
calculates the lane parameters in the current frame by weighing the lane
parameters from the previous and current frame with
$\frac{\Omega_{f-1}}{\sigma_{f-1}}$ and $\frac{\omega_{f}}{\sigma_{f}}$ as the
weights respectively. Using these weights, the proportion of the current lane
parameters contributed by the lane in the current frame, $\zeta_{f}$, is equal
to

$\zeta_{f}=\frac{\omega_{f}\,\sigma_{f-1}}{\omega_{f}\,\sigma_{f-1}+\Omega_{f-1}\,\sigma_{f}}.$
(7)

Using these weights, we can obtain the predicted lane parameters for the
current frame, ($r_{f}$, $\theta_{f}$), from the detected lane parameters,
($\hat{r}_{f}$, $\hat{\theta}_{f}$) and lane parameters from previous frame,
($r_{f-1}$, $\theta_{f-1}$) through

$\begin{pmatrix}r_{f}\\\\[3.0pt]
\theta_{f}\end{pmatrix}=\begin{pmatrix}\hat{r}_{f}&r_{f-1}\\\\[3.0pt]
\hat{\theta}_{f}&\theta_{f-1}\end{pmatrix}\begin{pmatrix}\zeta_{f}\\\\[3.0pt]
1-\zeta_{f}\end{pmatrix}$ (8)

which allows us to use lane information over multiple frames to deduce more
accurate parameters for the lane in the current frame, similar to other
estimation algorithms such as the Kalman filter but without prior assumptions
on the distribution of process or observation noise, hence allowing the method
to remain a turnkey solution that can be easily added to deep learning models
for new datasets without fitting new parameters.

##  IV Experiments

To verify the usefulness of our proposed changes to RONELD and to check the
accuracy results of our proposed method, we run experiments on the test sets
of two popular lane detection datasets in the literature, namely TuSimple [7]
and CULane [8], which were also used in RONELD [6].

Figure 7: Sample images from the (a) CULane and (b) TuSimple test sets. Ground
truth active lane markings are highlighted in blue.

###  IV-A Dataset details

Table I summarizes the details of the two datasets and Fig. 7 shows some
sample frames from the datasets. Comparing the datasets, CULane is generally a
more complicated dataset with a variety of road conditions and traffic
densities, which present challenges due to wider variety of road image
contexts and more challenging driving scenarios (e.g. night scenes with poor
lighting conditions and occlusions of lane markings by other vehicles). In
contrast, TuSimple is a rather simple dataset consisting primarily of highway
roads in fair weather conditions with low traffic density, albeit containing
some variability in road surface color and conditions. TuSimple also has
ground truths labelled on the last frame of each twenty-frame clip, which
provides preceding frames for our lane tracking method and RONELD to use
before making predictions on a labelled image. On the other hand, CULane
provides ground truths on all road images within the dataset.

###  IV-B Evaluation metric

To identify true positive (TP) predicted lane markings, we use the
intersection over union (IoU) between ground truth and predicted lane
markings, similar to [5, 8, 6]. This method sets the line width of the ground
truth lane markings and predicted lane markings as 16 and 30 pixels
respectively. We use these line widths for output probability maps with widths
of 800 pixels and scale them accordingly for different output probability map
sizes for uniform comparison across models. Lane predictions with IoU values
above a certain threshold are marked as TP lanes for that threshold. We then
calculate the accuracy of each model for each threshold value using

$accuracy=\frac{N_{TP}}{N_{gt}},$ (9)

where ${N_{TP}}$ is the number of TP lanes for a particular IoU threshold
value and $N_{gt}$ is the number of ground truth lane markings. We record the
accuracy at IoU threshold values between 0.3 and 0.5 (inclusive) at 0.1
intervals. We apply this evaluation metric consistently across our experiments
for a uniform comparison across the original dataset and cross-dataset
validation tests.

###  IV-C Implementation details

For our experiments we exploit two state-of-the-art lane detection models,
SCNN [8] and ENet-SAD [5]. We use their probability map outputs with our
method for comparison with the RONELD [6] method and the original models. We
used models that are pre-trained on the CULane dataset, with no images from
the TuSimple dataset as we use the TuSimple dataset for our cross-dataset
validation tests to determine the performance of the methods on unseen
datasets. We select the CULane model as the train dataset and TuSimple as the
cross-dataset as CULane is a more complicated dataset and should ideally be
generalizable to the TuSimple test set given its greater complexity and wider
variety of driving scenarios.

We use the CULane-trained models to generate probabily map outputs on the lane
images from the CULane and TuSimple test sets and use the method outlined in
[8, 5, 6] to generate lane marking predictions for the SCNN and ENet-SAD
models respectively. For the SCNN+RONELD, ENet-SAD+RONELD, SCNN+RONELDv2, and
ENet-SAD+RONELDv2 methods, we run the corresponding method on the probability
map outputs of the respective models to generate lane marking predictions. We
compare the lane marking predictions with the ground truth lane marking
provided with the datasets to calculate the accuracy performance of the
methods which we list in Tables II and III and display using graphs in Fig. 8.

TABLE II: Comparative accuracy results and recorded runtimes using different
lane tracking methods presented together with the SCNN and ENet-SAD lane
detection models in [8, 5] respectively, separately in RONELD [6], and this
paper on the CULane test set. The highest accuracy measure on each model at
each IoU threshold and lowest runtime is presented in bold.

Model: | SCNN | ENet-SAD
---|---|---
IoU Threshold | Original | RONELD | RONELDv2 (Ours) | Original | RONELD | RONELDv2 (Ours)
0.3 | 0.812 | 0.826 | 0.832 | 0.823 | 0.832 | 0.835
0.4 | 0.762 | 0.789 | 0.797 | 0.778 | 0.799 | 0.804
0.5 | 0.629 | 0.703 | 0.714 | 0.655 | 0.729 | 0.737
Runtime (ms) | - | 2.82 (5.68*) | 1.48 | - | 3.59 (6.29*) | 2.16

* : original RONELD implementation from [6] using a different CPU (Intel Core i9-9900K)

TABLE III: Comparative accuracy results and recorded runtimes using different
lane tracking methods presented together with the SCNN and ENet-SAD lane
detection models in [8, 5] respectively, separately in RONELD [6], and this
paper on the TuSimple test set, which is a cross-dataset validation using
CULane-trained models. The highest accuracy measure on each model at each IoU
threshold and lowest runtime is presented in bold.

Model: | SCNN | ENet-SAD
---|---|---
IoU Threshold | Original | RONELD | RONELDv2 (Ours) | Original | RONELD | RONELDv2 (Ours)
0.3 | 0.625 | 0.869 | 0.879 | 0.608 | 0.825 | 0.845
0.4 | 0.470 | 0.796 | 0.815 | 0.502 | 0.753 | 0.770
0.5 | 0.238 | 0.549 | 0.580 | 0.341 | 0.530 | 0.552
Runtime (ms) | - | 2.07 (2.80*) | 1.00 | - | 2.76 (3.55*) | 1.57

* : original RONELD implementation from [6] using a different CPU (Intel Core i9-9900K)

((a)) Accuracy results on CULane test set using CULane-trained models.

((b)) Accuracy results on TuSimple test set, which is a cross-dataset
validation using CULane-trained models.

Figure 8: Comparative accuracy results for 8(a) CULane and 8(b) TuSimple test
sets using CULane-trained state-of-the-art deep learning models, RONELD and
RONELDv2, between 0.3 to 0.5 IoU thresholds at 0.01 IoU threshold intervals.

###  IV-D Results

####  IV-D1 Accuracy performance

We generally see an increase in the accuracy performance using our proposed
method beyond those achieved by RONELD. In particular, we see a greater
increase in accuracy performance on the TuSimple cross-dataset validation test
and at higher IoU thresholds as well, increasing from 0.549 to 0.580 at the
highest 0.5 IoU threshold using SCNN on the TuSimple test set. This greater
improvement in performance at higher IoU thresholds and on the TuSimple test
set can generally be explained by the ability of RONELDv2 to refine the lane
parameters using its improved use of preceding frame information. RONELDv2 was
able to detect these more accurate lane parameters through the use of its lane
merging step and lane point variance information to improve on the lanes
detected by the original RONELD method, however these two steps in particular
are unable to make changes to lanes which are far apart and hence not matched
together. As a result, these steps help RONELDv2 detect lanes which better
meet the higher IoU thresholds but have a less significant impact on poorly
detected lanes that are far from the ground truths and hence do not meet even
the lowest 0.3 IoU threshold. RONELDv2 also had a more significant improvement
in accuracy performance on the TuSimple dataset due to the greater reliance on
preceding frame information to obtain accurate lane information on this
dataset, where RONELDv2 is able to make better use of this preceding frame
information. This can be seen by the relatively low initial performance of the
deep learning lane detection models on the TuSimple cross-dataset validation
test and in our ablation studies on the effectiveness of RONELDv2 methods.

####  IV-D2 Runtime

To test the runtime for our proposed method, we recorded the mean runtime of
the method on all images on the dataset using a single Ryzen 7 4800H CPU. We
record runtime of our method in Tables II and III together with the runtimes
from [6] for comparison. In our experiments, we implemented some additional
optimizations on the RONELD method which led to a lower runtime using the
RONELD method despite the slower CPU (mean runtime for the SCNN CULane dataset
was 5.68ms for Intel Core i9-9900K [6] and 7.30ms for Ryzen 7 4800H using the
same original implementation of RONELD), which we recorded for RONELD as well.
These additional optimizations include better use of just-in-time compilation
and a closed form expression for the integral in the calculation of the lane
distance instead of using numerical integration methods for calculating the
distance between the predicted straight lanes. From our experiments, when
comparing between RONELD and RONELDv2 methods that use the new optimization,
it is observed that there is an up to two fold decrease in the runtime for
RONELDv2 when compared with RONELD. This can be attributed to the improved
weight system used in our new method which reduces the maximum weight of
stored lanes. This lower maximum weight allows these lanes to be cleared more
rapidly from the stored cache of previous lanes that current lanes are
compared to when searching for the matching previous lane, thereby reducing
the amount of processing needed for the lane merging and tracking steps of our
method. The greater absolute decrease in the runtime on the CULane dataset vis
a vis the TuSimple dataset can be attributed to the greater number of lanes
detected on the CULane dataset as the models used were trained on the CULane
train set. Comparing the runtimes on the SCNN and ENet-SAD models, the
difference in runtime can be attributed to the different sizes of the
probability map outputs from the model, with the ENet-SAD model used
outputting a $976\times 351$ probability map while the SCNN model used outputs
a $800\times 288$ probability map. Rescaling the ENet-SAD model output
probability map to size $800\times 288$ results in a runtime of 1.62ms on the
CULane dataset which is similar to the runtime of 1.48ms for the SCNN model on
the same dataset.

###  IV-E Ablation study

TABLE IV: Comparison of accuracy results of SCNN + RONELDv2 with different
$\alpha$ values at 0.3 and 0.5 IoU thresholds on the CULane and TuSimple test
sets.

CULane

IoU Threshold | $\alpha=0.25$ | $\alpha=0.5$ | $\alpha=0.75$ | $\alpha=1$
---|---|---|---|---
0.3 | 0.783 | 0.832 | 0.831 | 0.825
0.5 | 0.644 | 0.714 | 0.713 | 0.712

TuSimple

IoU Threshold | $\alpha=0.25$ | $\alpha=0.5$ | $\alpha=0.75$ | $\alpha=1$
---|---|---|---|---
0.3 | 0.887 | 0.879 | 0.869 | 0.809
0.5 | 0.582 | 0.580 | 0.571 | 0.544

####  IV-E1 $\alpha$ value

To measure the impact of the exponentially weighted moving average method for
calculating lane weights and to determine the impact of different $\alpha$
values for this step, we test different $\alpha$ values from 0.25 to 1
(inclusive) at 0.25 intervals, where an alpha value of 1 corresponds to no
weight placed on the previous $\Omega_{f-1}$ value, though we still retain the
lane merge step to merge lane parameters covered in section III-C if the lanes
are matched, with the results recorded in Table IV. It is observed that
performance on the TuSimple test set increases as $\alpha$ value decreases,
which reflects the positive impact that preceding frame tracking has in this
cross-dataset validation test by using information from preceding frames to
supplement the relatively weak performance in the current frame. On the other
hand, on the CULane test set, performance peaks at $\alpha=0.5$ which
corresponds to an equal weight placed on the weight of the lane in the current
frame and on the exponentially weighted average value from the previous frame.
$\alpha=0.5$ value was selected due to the maximum performance achieved on the
CULane test set and diminishing returns achieved on the TuSimple test set by
decreasing $\alpha$ further.

####  IV-E2 Lane match distance

To determine the optimal threshold for the length (measured in standard
deviation of the lane points) between lanes that are matched together in our
lane tracking process, we test different lane distance values based on the
standard deviation of the lane point locations calculated in section III-A and
list the results in Table V. We tested maximum lane match distances set at 0,
1$\sigma$, 2$\sigma$, and 3$\sigma$, with a lane match distance of 0
corresponding to lanes being matched only when their lane parameters match
exactly which is rare in practice. For this ablation study, the lanes that are
not matched are still stored for subsequent frames, with their weights
undergoing exponential decay in line with the exponentially weighted moving
average method used to calculate lane weights, and this is intended to isolate
the impact of the lane matching step. From our experiments, it is observed
that a lane match distance of $2\sigma$ provides an optimal combined result
across the CULane and TuSimple test sets, with minor decrease in results of
less than 0.01 accuracy compared to using a $1\sigma$ or $3\sigma$ lane match
distance at the 0.3 IoU threshold but a more significant increase in accuracy
compared to the other lane match distances at the 0.5 IoU threshold .

####  IV-E3 Lane merge

To verify the usefulness of merging the lane parameters in our method, we test
the difference in accuracy performance with and without the lane merge step
and list the results in Table VI. It is observed that the lane merge step
generally increases the accuracy performance of the method. In particular, it
has a greater impact at the higher 0.5 IoU threshold which is as expected as
the lane merge step helps to finetune the lane parameters of lanes that are
relatively close to each other due to the prior lane match step. As such, the
merging of the lane parameters generally makes small adjustments to lanes that
are already within a close vicinity to each other, which helps them reach a
higher IoU threshold but is unable to improve the result of a poorly detected
lane that is far from the ground truth lane marking as those lanes would not
have been merged together and have no impact on each other.

TABLE V: Comparison of accuracy results of SCNN + RONELDv2 with different lane
match distance values at 0.3 and 0.5 IoU thresholds on the CULane and TuSimple
test sets.

CULane

IoU Threshold | $0$ | $1\sigma$ | $2\sigma$ | $3\sigma$
---|---|---|---|---
0.3 | 0.832 | 0.832 | 0.832 | 0.832
0.5 | 0.712 | 0.713 | 0.714 | 0.712

TuSimple

IoU Threshold | $0$ | $1\sigma$ | $2\sigma$ | $3\sigma$
---|---|---|---|---
0.3 | 0.872 | 0.876 | 0.879 | 0.880
0.5 | 0.567 | 0.573 | 0.580 | 0.578

TABLE VI: Accuracy results of SCNN + RONELDv2 with and without lane merge (LM) step on the CULane and TuSimple test sets at 0.3 and 0.5 IoU thresholds. IoU Threshold | CULane | TuSimple
---|---|---
w/o LM | w/ LM | w/o LM | w/ LM
0.3 | 0.831 | 0.832 | 0.875 | 0.879
0.5 | 0.711 | 0.714 | 0.569 | 0.580

##  V Conclusion

In this paper, we have presented an improved lane tracking method that
improves on a previous method by finding the lane point variance, using an
exponentially weighted moving average method to weigh lanes, and merging lane
parameters. We have demonstrated the usefulness of our method on the CULane
and TuSimple test sets using the SCNN and ENet-SAD lane detection models with
an increase in accuracy performance and an up to two-fold decrease in the
runtime of our method. This shows the usefulness of lane tracking methods in
improving the accuracy performance of deep learning lane detection methods,
particularly on cross-dataset validation tests.

## References

  * [1] Q. Li, L. Chen, M. Li, S. Shaw, and A. Nüchter, “A sensor-fusion drivable-region and lane-detection system for autonomous vehicle navigation in challenging road scenarios,” _IEEE Transactions on Vehicular Technology_ , vol. 63, no. 2, pp. 540–555, 2014.
  * [2] Y. Xing, C. Lv, L. Chen, H. Wang, H. Wang, D. Cao, E. Velenis, and F. Wang, “Advances in vision-based lane detection: Algorithms, integration, assessment, and perspectives on acp-based parallel vision,” _IEEE/CAA Journal of Automatica Sinica_ , vol. 5, no. 3, pp. 645–661, 2018\.
  * [3] L. Tabelini, R. Berriel, T. M. Paixão, C. Badue, A. F. D. Souza, and T. Oliveira-Santos, “Keep your eyes on the lane: Real-time attention-guided lane detection,” 2020.
  * [4] R. Liu, Z. Yuan, T. Liu, and Z. Xiong, “End-to-end lane shape prediction with transformers,” in _WACV_ , 2021.
  * [5] Y. Hou, Z. Ma, C. Liu, and C. C. Loy, “Learning lightweight lane detection cnns by self attention distillation,” in _2019 IEEE/CVF International Conference on Computer Vision (ICCV)_ , 2019, pp. 1013–1021.
  * [6] Z. M. Chng, J. M. H. Lew, and J. A. Lee, “Roneld: Robust neural network output enhancement for active lane detection,” in _2020 25th International Conference on Pattern Recognition (ICPR)_ , 2021, pp. 6842–6849.
  * [7] TuSimple, “Tusimple competitions for cvpr2017,” https://github.com/TuSimple/tusimple-benchmark/, accessed: 2019-10-02.
  * [8] X. Pan, J. Shi, P. Luo, X. Wang, and X. Tang, “Spatial as deep: Spatial CNN for traffic scene understanding,” in _AAAI Conf. Artificial Intelligence_ , 2018, pp. 7276–7283.
  * [9] J. Wang, T. Mei, B. Kong, and H. Wei, “An approach of lane detection based on inverse perspective mapping,” in _17th International IEEE Conference on Intelligent Transportation Systems (ITSC)_ , 2014, pp. 35–38.
  * [10] K.-Y. Chiu and S.-F. Lin, “Lane detection using color-based segmentation,” in _Proc. IEEE Intelligent Vehicles Sympo. (IV)_ , 2005, pp. 706–711.
  * [11] J. C. McCall and M. M. Trivedi, “An integrated, robust approach to lane marking detection and lane tracking,” in _IEEE Intelligent Vehicles Symposium, 2004_ , 2004, pp. 533–537.
  * [12] X. Li, Q. Wu, Y. Kou, L. Hou, and H. Yang, “Lane detection based on spiking neural network and hough transform,” in _Proc. Int’l Congress. Image and Signal Processing (CISP)_ , 2015, pp. 626–630.
  * [13] M. Haloi and D. B. Jayagopi, “A robust lane detection and departure warning system,” in _2015 IEEE Intelligent Vehicles Symposium (IV)_ , 2015, pp. 126–131.
  * [14] A. Borkar, M. Hayes, and M. T. Smith, “Robust lane detection and tracking with ransac and kalman filter,” in _2009 16th IEEE International Conference on Image Processing (ICIP)_ , 2009, pp. 3261–3264.
  * [15] M. Aly, “Real time detection of lane markers in urban streets,” in _2008 IEEE Intelligent Vehicles Symposium_ , 2008, pp. 7–12.
  * [16] M. Beyeler, F. Mirus, and A. Verl, “Vision-based robust road lane detection in urban environments,” in _2014 IEEE International Conference on Robotics and Automation (ICRA)_ , 2014, pp. 4920–4925.
  * [17] A. M. López, J. Serrat, C. Cañero, F. Lumbreras, and T. Graf, “Robust lane markings detection and road geometry computation,” _Int’l J. Automotive Tech._ , vol. 11, no. 3, pp. 395–407, 2010.
  * [18] H. Loose, U. Franke, and C. Stiller, “Kalman particle filter for lane recognition on rural roads,” in _2009 IEEE Intelligent Vehicles Symposium_ , 2009, pp. 60–65.
  * [19] J. Kim and M. Lee, “Robust lane detection based on convolutional neural network and random sample consensus,” in _Proc. Int’l Conf. Neural Info. Processing (ICONIP)_ , 2014, pp. 454–461.
  * [20] B. He, R. Ai, Y. Yan, and X. Lang, “Accurate and robust lane detection based on dual-vew convolutional neutral network,” in _Proc. IEEE Intelligent Vehicles Sympo. (IV)_ , 2016, pp. 1041–1046.
  * [21] Z. Qin, H. Wang, and X. Li, “Ultra fast structure-aware deep lane detection,” in _The European Conference on Computer Vision (ECCV)_ , 2020.
  * [22] X. Li, J. Li, X. Hu, and J. Yang, “Line-cnn: End-to-end traffic line detection with line proposal unit,” _IEEE Transactions on Intelligent Transportation Systems_ , vol. 21, no. 1, pp. 248–258, 2020.
  * [23] N. Garnett, R. Cohen, T. Pe’er, R. Lahav, and D. Levi, “3d-lanenet: End-to-end 3d multiple lane detection,” in _2019 IEEE/CVF International Conference on Computer Vision (ICCV)_ , 2019, pp. 2921–2930.
  * [24] J. Redmon, S. Divvala, R. Girshick, and A. Farhadi, “You only look once: Unified, real-time object detection,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , June 2016.
  * [25] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in _2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2016, pp. 770–778.
  * [26] Q. Zou, H. Jiang, Q. Dai, Y. Yue, L. Chen, and Q. Wang, “Robust lane detection from continuous driving scenes using deep neural networks,” _IEEE Transactions on Vehicular Technology_ , vol. 69, no. 1, pp. 41–54, 2020\.
  * [27] H. Xu, S. Wang, X. Cai, W. Zhang, and X. Liang, “Curvelane-nas: Unifying lane-sensitive architecture search and adaptive point blending,” in _The European Conference on Computer Vision (ECCV)_ , 2020.
  * [28] S. Lee, J. Kim, J. S. Yoon, S. Shin, O. Bailo, N. Kim, T. Lee, H. S. Hong, S. Han, and I. S. Kweon, “Vpgnet: Vanishing point guided network for lane and road marking detection and recognition,” in _2017 IEEE International Conference on Computer Vision (ICCV)_ , 2017, pp. 1965–1973.
  * [29] A. Paszke, A. Chaurasia, S. Kim, and E. Culurciello, “Enet: A deep neural network architecture for real-time semantic segmentation,” _arXiv_ , vol. abs/1606.02147, 2016.
  * [30] D. Neven, B. D. Brabandere, S. Georgoulis, M. Proesmans, and L. V. Gool, “Towards end-to-end lane detection: an instance segmentation approach,” in _2018 IEEE Intelligent Vehicles Symposium (IV)_ , 2018, pp. 286–291.
  * [31] Y. Ko, J. Jun, D. Ko, and M. Jeon, “Key points estimation and point instance segmentation approach for lane detection,” _arXiv_ , vol. abs/2002.06604, 2020.
  * [32] I. J. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio, “Generative adversarial nets,” in _Proceedings of the 27th International Conference on Neural Information Processing Systems - Volume 2_ , ser. NIPS’14. Cambridge, MA, USA: MIT Press, 2014, p. 2672–2680.
  * [33] T. Liu, Z. Chen, Y. Yang, Z. Wu, and H. Li, “Lane detection in low-light conditions using an efficient data enhancement : Light conditions style transfer,” in _2020 IEEE Intelligent Vehicles Symposium (IV)_ , 2020.
  * [34] M. Ghafoorian, C. Nugteren, N. Baka, O. Booij, and M. Hofmann, “El-gan: Embedding loss driven generative adversarial networks for lane detection,” in _Computer Vision – ECCV 2018 Workshops_ , L. Leal-Taixé and S. Roth, Eds. Cham: Springer International Publishing, 2019, pp. 256–272.