# ContinuityLearner: Geometric Continuity Feature Learning for Lane
Segmentation

Haoyu Fang  NYU Tandon School of Engineering
New York, NY 11201
Email<EMAIL_ADDRESS>Jing Zhu  NYU Tandon School of Engineering
New York, NY 11201
Email<EMAIL_ADDRESS>Yi Fang  ECE Department
New York University Abu Dhabi
Email<EMAIL_ADDRESS>
###### Abstract

Lane segmentation is a challenging issue in the autonomous driving system
designing because lane marks show weak textural consistency due to occlusion
or extreme illumination but strong geometric continuity in traffic images,
from which general convolution neural networks (CNNs) are not capable of
learning semantic objects. To empower conventional CNNs in learning geometric
clues of lanes, we propose a deep network named ContinuityLearner to better
learn geometric prior within lane. Specifically, our proposed CNN-based
paradigm involves a novel Context-encoding image feature learning network to
generate class-dependent image feature maps and a new encoding layer to
exploit the geometric continuity feature representation by fusing both spatial
and visual information of lane together. The ContinuityLearner, performing on
the geometric continuity feature of lanes, is trained to directly predict the
lane in traffic scenarios with integrated and continuous instance semantic.
The experimental results on the CULane dataset and the Tusimple benchmark
demonstrate that our ContinuityLearner has superior performance over other
state-of-the-art techniques in lane segmentation.

##  I Introduction

Autonomous vehicles and driving assistant systems have been a major focus of
the computer vision research community. One of the key components of
autonomous driving is to perceive and understand the surrounding environment
of the vehicle. In particular, environment comprehension tasks, i.e. lane
detection, semantic segmentation, become essential as they assist the self-
driving car to locate itself between lane boundaries. Despite years of
research, lane segmentation is still a challenging task due to several reasons
(shown in Fig. 1): 1) complex traffic scenarios (e.g. self-occlusion and
vehicle-occlusion) reduce visual information for a vision-based algorithm to
recognize lanes; 2) various lane patterns, especially those lack essential
visual features, bring great difficulties to learn distinctive feature of road
lanes; 3) environmental interference like low illumination, reflection brings
unpredictable influence on captured images even in the same scenes.

Before neural networks prevailing in computer vision tasks, traditional
methods to segment lane boundaries are almost based on handcrafted features,
which can only work in limited scenarios under strict constraints such as
parallel lanes [1, 2, 3, 4] or close to straight lanes [5, 6]. The handcrafted
features, such as color [7], bar filter [8] and structure tensor [9] etc,
usually are not robust enough to handle all the challenging cases of diverse
real-world road scenarios. With neural networks, especially fast development
of deep-learning based approaches, works on deep learnable feature instead of
handcrafted features have become a new promising research direction.

Figure 1: An example of lane segmentation on CULane Dataset [10]. The fist row
is a traffic image and the second is corresponding annotations of the lane
boundaries. The example illustrates four common challenging cases that may
disable a deep learning network to segment perfect lane boundaries in
practical applications. From left to right, the challenges are occlusion,
noncontinuous visual information (dash line), no sufficient visual clues (no
line) and strong interference (optical reflection).

Although CNNs have shown impressive performance in lane segmentation, they
exploit mainly on visual information that might be insufficient in real-world
traffic scene. This challenge greatly limits the networks to provide a
satisfactory performance in practice. However, unlike many other traffic
objects, road lanes are observed to have strong structure prior but less
appearance clues. Exploitation on this structure prior or geometric feature of
lanes that has long continuous shape, shows importance on performance
improvement. Besides, geometric feature is not only conveyed by visual
knowledge but also contained in lane pixel locations (coordinates of pixels
that belongs to road lanes), which provides distinctive feature even if visual
knowledge is not perfect or sufficient. Triggered by the motivation of
cultivating geometric feature, we propose a novel deep CNN named
ContinuityLearner, which is the pioneering approach to directly learn
structural feature beneath lane-pixel locations and exploit visual information
of lane at the same time. The proposed ContinuityLearner, performing on a
novel geometric continuous feature representation that fuses visual feature
and pixel locations together, predicts lanes in traffic scenarios with
integrated and smooth instance semantics. Therefore, the major contributions
of the paper are as follows:

  * •

We firstly propose a lane segmentation paradigm that applies geometric
continuity prior of lanes’ shape as a global constraint to refine the CNN-
based semantic. The geometric continuity feature is a robust and comprehensive
representation for networks to learn the semantic of lanes in the traffic
image.

  * •

We specifically develop a Context-encoding Image Feature Learning Network to
encode class-dependent image featuremaps for various lanes in the image.

  * •

We propose a novel Geometric Continuity Feature Encoding Layer that fuses the
class-dependent image with pixels’ locations to jointly exploit textual
knowledge of lanes as well as their spatial information.

  * •

Our proposed model outperforms several state-of-the-art methods in experiment
conducted on Tusimple and CULane dataset, especially in the cases where visual
(textual) knowledge of lanes is missing due to occlusion and extreme
illumination.

##  II Related works

Development of deep learning techniques has stimulated a promising research
direction for lane detection. [5] uses a spiking neural network to first
extract information about lane edges and then employs Hough transform for lane
detection. However, this method is susceptible to occluded lane boundaries.
[11] using the front-view and top-view of the input image, proposes DVCNN
(Dual-View Convolutional Neural Network) for detecting lanes. [12] proposes a
deep neural network namely DeepLanes that estimates the position of the lanes.
[13] develops a multi-task CNN network that extracts geometric information
about the lanes and a Recurrent Neural Network (RNN) based model that detects
lane boundaries. [14] adopts an instance segmentation approach to lane
detection in which the proposed end-to-end network outputs an instance map.
Note that each lane pixel in the instance map indicates which lane it belongs
to. In this paper, in order to take into account changes in road plane, a
unique learned transformation (outputted by H-Net) is employed instead of
using the traditional fixed ”bird’s-eye view” transformation. [15] proposes
LaneNet which divides the lane detection task into two steps: lane edge
proposal and lane line localization. In the first stage, a lane edge proposal
network classifies each pixel of the input image and outputs lane edge
proposals which contain information about the edges of the lanes. These lane
proposals are then fed to the lane line localization networks to finally
output the detected lanes. [16] proposes a Spatial CNN which employs layer-by-
layer convolutions to learn the spatial information and detect lane lines.

Some other recent works, using global geometric constraints, have been
conducted in the realm of lane detection. One geometric feature that has been
considered important in lane detection is vanishing point [17, 18]. Lee et al.
propose VPGNet [18], a network that not only detects different types of lane
lines but also detects road markings under unfavourable weather conditions.
[19] proposes a multi-task model for lane segmentation and lane boundary
detection.

Figure 2: Pipeline of ContinuityLearner which consists two branch: A) the
rough segmentation framework based on a context-encode image feature learning
network; B) the refined segmentation framework that applies a geometric
continuity feature encoding layer and a PointNet-based [20] network. Note that
W, H and C denote width and height and channels of feature maps respectively.
$d_{i}$ refers to the channels of feature of object $i$. $2$ in (B) represents
the location of pixel in image plane (x,y).

##  III Method

The proposed model (depicted in Fig.2) consists of two branch: A) the rough
segmentation branch and B) the refined segmentation branch. The rough
segmentation branch is mainly formed by a context-encoding image feature
learning network that captures the feature statistics as a global semantic
context and generates a coarse semantic and class-dependent feature maps. The
refined segmentation branch involves a geometric continuity feature encoding
layer which converts the rough semantic into a 2D point cloud and then fuses
the point set conveying the spatial information with the image feature learned
in (A). The multilayer perceptron (MLP)-based network in the branch (B)
predicts possible missing key lane points to generate a more integrated and
continuous semantic of road lanes.

###  III-A Rough Segmentation Branch

The rough segmentation branch in Fig.3, fed with a color traffic image,
generates a coarse semantic of lanes and class-dependent feature maps which
encode rich information what lane objects are in the input image. This branch
is mainly formed by context-encoding feature learning network that predicts a
set of attention weights to highlight the class-dependent feature maps.

Figure 3: The context-encoding image feature learning network

Context-encoding module The proposed context-encoding feature learning network
takes advantages of an encoding module [21] that learns a dictionary to
describe the semantic context of dataset and generates attention weights with
rich contextual knowledge. The encoder module is fed with an input feature map
with the shape $C\times H\times W$ as a set of C-dimensional input feature
$X=\\{x_{1},...,x_{N}\\}$, where $N$ is total number of features given by
$H\times W$. The module learns an inherent $D=\\{d_{1},....,d_{k}\\}$
containing $K$ number of codewords and a set of smoothing factor of visual
centers $S=\\{s_{1},...,s_{k}\\}$. The context-encoding network outputs the
residual encoder by aggregating the residuals with soft-assignment weights
$e_{k}=\Sigma_{i=1}^{N}e_{ik}$, where

$\centering
e_{ik}=\frac{exp(-s_{k}||x_{i}-d_{k}||^{2})}{\Sigma_{j=1}^{K}exp(-s_{j}||x_{i}-d_{j}||^{2})}(x_{i}-d_{k}).\@add@centering$
(1)

We aggregate the encoders by $e=\Sigma_{k=1}^{K}\phi(e_{k})$, where $\phi$
denotes Batch Normalization with ReLU activation.

Class-dependent feature maps The utility of the encoders $e$ reduces the
dimension of the feature representations and we produce an attention weights
which highlight the independence of different lane objects in feature
dimension by $A=\delta(We)$, where $W$ represents convolution layers’ weights
and $\delta$ denotes an activation function (i.e. the sigmoid function in this
paper). The encoded image feature denoted by $Y$ is calculated by

$\centering Y=X\otimes A,\@add@centering$ (2)

where $\otimes$ is a channel-wise multiplication and $Y$ has the same
dimension as the input feature map $X$. To further fit the encoded image
feature map into a 2D point cloud format, a convolution layer denoted by
$conv(\cdot):R^{C\times H\times W}\xrightarrow{}R^{C^{*}\times 1\times 1}$ is
employed to extract a class-dependent feature, where $C^{*}=\Sigma_{M}c_{i}$
and $M$ is the number of detected lanes. $c_{i}$ represents the feature that
encodes image feature of the $i$th lane.

$\centering F=conv(Y),\@add@centering$ (3)

In the training process, a semantic encoding loss (SE-loss) [21] that makes
individual predictions for the presences of object categories in the scene and
learns with binary cross entropy loss is applied to force the network to
understand the global semantic information and calculate the class-dependent
feature maps according to the lane annotations (e.g. lane No.1, lane No.2 and
etc.).

###  III-B Refined Segmentation Branch

Although lanes demonstrate relatively weak textual coherence due to the lack
of visual information, they have stronger geometric consistency in the image.
The geometric continuity of lanes is reflected by the location of lane pixel
in the image plane. Therefore, the proposed refined segmentation branch
concentrates on exploiting spatial knowledge of lane pixels to construct the
geometric feature representation of lanes. This module converts the coarse
semantic into several incomplete 2D point clouds and appends class-dependent
image feature maps with points’ location in the corresponding point set.

Before converting, a filtering process with a fixed probability threshold will
perform on the rough segmentation to guarantee that only prediction with high
confidence will be regraded as lane pixels and further reduce the false
positive prediction in the rough segmentation. Then a MLP-based network [20]
is trained to fill possible key points into the incomplete point set to
reconstruct a continuous prediction which better reflects the geometric shape
of lanes.

Geometric feature encoding layer This module generates a geometric continuity
feature representation by concatenating the class-dependent image feature of a
specific lane with locations of all its lane pixels (in Fig. 4). We convert
the rough segmentation results $I_{rough}\in R^{W\times H}$ with lane
instances’ semantic into a 2D point cloud $P\in R^{N*2}$ by regarding each
lane pixel in the coarse segmentation as an isolated point in the image plane,
where $N$ is the total number of points. A specific order is taken to extract
the points (i.e. top to bottom and left to right in this paper) and the point
cloud contains all lanes’ pixels in the rough segmentation
$P=\\{P_{i}|i=1,2,\cdot\cdot\cdot,M\\}$, where $M$ denotes the number of
predicted lanes in the rough segmentation. For each points of $i$th lane
($P_{i}$), the geometric continuity feature representation $G_{i}$ is produced
by concatenating the point’s location with object’s independent image feature
$c_{i}$ and the point set $P_{i}$ is encoded into the high dimensional feature
space by $P_{i}\in R^{{M_{i}}\times 2}\xrightarrow{}G_{i}\in
R^{{M_{i}}\times(2+c_{i})}$.

Figure 4: Geometric continuity feature encoding layer.

Point set completion Considering the segmentation of an arbitrary lane as a
point set. The missing areas in rough segmentation of the lane result in some
key points that reflect the shape of the lane missing in the corresponding
point set. we use a MLP-based network [20] to predict those missing points and
reconstruct a completed point set which better represents the semantic of
lanes. The MLP-based network denoted by $MLP(\cdot)$ takes the geometric
continuity feature $G_{i}$ as input and predicts missing points of the
incomplete set by

$\centering P^{*}_{i}=MLP(G_{i}),\@add@centering$ (4)

where $P^{*}_{i}\in R^{M_{i}^{*}\times 2}$ is the completed point set and
$M_{i}^{*}\geq M_{i}$. Finally, the completed point set is converted into
image pixels by a quantization process $h:P^{*}_{i}\in R^{M_{i}^{*}\times
2}\xrightarrow{}P^{**}_{i}\in Z^{M_{i}^{*}\times 2}$. Those pixels are finally
deployed in image plane and form the lane semantics.

##  IV Experiments and Results

We test the proposed ContinuityLearner on the Tusimple dataset [22] and the
challenging CULane dataset [10], which are both large scale datasets
containing sufficient images for training, validation and testing in various
traffic conditions. The experiments will be analyzed in this section in
details. To conduct fair comparison, we employ the same evaluation metrics as
[10] and [23] to test performance on CULane and Tusimple testing sets
respectively.

(a) Input Images

(b) ResNet [24]

(c) Our Results

(d) Ground Truth

Figure 5: Experimental results on CULane dataset [10]. The first column is the
input image, the second is segmentation results by ResNet101 [24], the third
column is our results and the final column is the ground truth. In each sub-
figure, the three or four lanes are marked in different colors (blue, green,
red and yellow from left to right).

We train the ContinuityLearner on the Tusimple dataset. The model are trained
for 2300 episodes with batch size 8 using a standard SGD optimizer with
initial learn rate 0.01, momentum 0.9. The learning rate gets halved every
other epoch. To save computation and memory usage, we resize the images of
TuSimple to $368\times 640$. For CULane experiments, we use identical hyper-
parameters to the former settings but we train the model on the full CULane
training dataset for 60K episodes and resize the image of CULane to $288\times
800$. Moreover, we set the probability threshold of the filter performing on
rough segmentation is $70\%$ (regarding prediction with $70\%$ or higher
confidence scores as lane pixels) for all experiments.

A quantitative comparison between the proposed method and several state-of-
the-art (SOTA) approaches are given in Table I. While the ContinuityLearner
demonstrates the SOTA level instance segmentation accuracy, it also provides
the lowest $FN$ rate among all advanced approaches. The ContinuityLearner
learns the geometric feature of lanes so that it is still robust to recognise
lanes even in dash lane or no lane cases that lack textual clues and confuse
conventional algorithms which focus on exploiting visual information only
(i.e. SCNN [10] and FastDraw [25] etc.). Similarly, another quantitative
comparison on CULane [10] is provided in Table II, which clearly illustrates
the robust performance of the proposed method in various traffic scenes. In
the Table II, the proposed method achieves the highest F1 score in three most
challenging traffic scenes due to lack of visual information of lanes (i.e.
crowded, no lines and shadow traffic scenes.) and also outperforms the SOTA
methods in the average score.

In the qualitative comparison, Fig. 5 shows the robust performance of the
proposed method in instance segmentation of various lanes in CULane testing
dataset. We can observe that the proposed network segments different numbers
of lanes in various shapes (curve, dash or straight lines). The first row
illustrates the instance segmentation in a dazzle traffic scene. Three
instances of lane with one in curve shape are segmented more smoothly than the
ResNet-101’s results. In third row, we observe that the proposed network
accurately predicts the semantics of a curve lane and a dash line. In the same
row, the proposed method accurately predicts a lane (in blue) which is
occluded by a vehicle and has no visual information at most part of the image.
The fourth row illustrates the ContinuityLearner’s performance is not
influenced by the illumination condition and predicts all lanes in the image.

TABLE I: Comparison of experimental results of ContinuityLearner with other state-of-the-art methods on the Tusimple dataset [22]. $Acc$, $FP$ and $FN$ denotes accuracy, false positive score and false negative score respectively. Higher $Acc$ but lower $FP$ and $FN$ represent better performance. Approach | Acc (%) | FP | FN
---|---|---|---
M. Ghafoorian et al. [26] | 94.9 | 0.059 | 0.067
J. Philion et al. [25] | 95.2 | 0.076 | 0.045
D. Neven et al. [23] | 96.9 | 0.044 | 0.020
D. Neven et al. [23] | 96.5 | 0.085 | 0.027
D. Neven et al. [23] | 96.2 | 0.236 | 0.036
P. L et al. [10] | 96.5 | 0.062 | 0.018
Z. Wang et al. [15] | 96.4 | 0.078 | 0.024
Y. Hou et al. [27] | 96.6 | 0.060 | 0.021
ContinuityLearner | 96.2 | 0.069 | 0.016
TABLE II: Comparison of experimental results of ContinuityLearner with other state-of-the-art methods on the CULane [10] test set. Harmonic mean (F1 measure with an IoU threshold of 0.5) are reported. Higher F1 is better. Approach | Normal | Crowded | Night | No lines | Shadow | Arrow | Dazzle | Curved | Total
---|---|---|---|---|---|---|---|---|---
F. Vision et al. [28] | 83.3 | 60.5 | 56.3 | 34.5 | 55.0 | 74.1 | 48.2 | 59.9 | 62.9
Krahenbuhl et al. [29] | 81.3 | 58.8 | 54.2 | 31.9 | 56.3 | 71.2 | 46.2 | 57.8 | 61.0
C. Szegedy et al. [24] | 87.4 | 64.1 | 60.6 | 38.1 | 60.7 | 79.0 | 54.1 | 59.8 | 66.7
P.L. et al. [10] | 90.6 | 69.7 | 66.1 | 43.4 | 66.9 | 84.1 | 58.5 | 64.4 | 71.6
J. Phillion et al.[25] | 85.9 | 63.6 | 57.8 | 40.6 | 59.9 | 79.4 | 57.0 | 65.2 | -
Y. Hou et al. [27] | 90.7 | 70.0 | 66.3 | 43.5 | 67.0 | 84.4 | 59.9 | 65.7 | 71.8
Base model (ResNet-101) | 90.2 | 68.2 | 65.9 | 41.7 | 64.6 | 84.0 | 59.8 | 65.5 | 70.8
ContinuityLearner | 90.5 | 70.3 | 66.2 | 43.9 | 67.2 | 82.3 | 58.8 | 64.0 | 72.1
TABLE III: Ablation study on the CULane test set. F1 scores (with an IoU threshold of 0.5) are reported. The proposed ContinuityLearner equals to the aggregation of the base model and A) the rough segmentation branch and B) the refined segmentation branch. The blue texts represent the best score based on ResNet-50 as the base model and the red represent the best based on ResNet-101. Approach | Normal | Crowded | Night | No lines | Shadow | Arrow | Dazzle | Curved | Total
---|---|---|---|---|---|---|---|---|---
C. Szegedy et a [24] | 87.4 | 64.1 | 60.6 | 38.1 | 60.7 | 79.0 | 54.1 | 59.8 | 66.7
Base model (ResNet-50) | 85.9 | 64.5 | 60.4 | 38.9 | 59.5 | 76.2 | 51.9 | 58.9 | 66.2
Base model+(A) | 85.7 | 65.4 | 61.7 | 40.9 | 59.1 | 78.2 | 53.2 | 58.2 | 66.9
Base model+(A)+(B) | 86.2 | 64.3 | 62.7 | 41.9 | 61.2 | 77.3 | 54.7 | 58.5 | 67.3
C. Szegedy et al [24] | 90.2 | 68.2 | 65.9 | 41.7 | 64.6 | 84.0 | 59.8 | 65.5 | 70.8
Base model (ResNet-101) | 88.3 | 68.7 | 64.2 | 41.5 | 66.4 | 81.2 | 60.3 | 65.0 | 68.2
Base model+(A) | 89.9 | 69.3 | 66.7 | 42.2 | 66.1 | 80.8 | 61.7 | 64.3 | 70.3
Base model+(A)+(B) | 90.5 | 70.3 | 66.2 | 43.9 | 67.2 | 82.3 | 58.8 | 64.0 | 72.1

Figure 6: Illustration of ablation study. The first row is the input images,
the second row is the rough segmentation results in branch (A) and the third
row is the refined segmentation results after (B).

##  V Ablation study and discussion

We investigate the effect of all modules in the proposed framework including
the base model, (A) the rough segmentation branch and (B) the refined
segmentation branch to the segmentation results. Note that our base model
shares the same structure as well as experimental setting with [24] and thus
achieves similar performance.

Reported by [24], ResNet-101 [24] has better performance in lane segmentation
than ResNet-50 since image feature is more fully learned with deeper neural
network. In the Table III, we can summarize that it is better to select
network with effective image feature exploiting as the base model for the
proposed ContinuityLearner because almost all frameworks based on ResNet-101
achieve higher accuracy than their correspondences based on ResNet-50. When
the base model better comprehends image feature, it provides an relatively
accurate rough segmentation with less false positive prediction (e.g. To
recognise a background pixel as a lane pixel). Though (B) the refined
segmentation branch is capable of predicting missing key points of lanes
(false negative prediction), it is sensitive to the false positive in the
coarse semantics outputted by (A). Consequentially, We also observe that
reducing false positive prediction in rough segmentation results in higher
refinement performance.

In Table III, given the same base model, we analyze the contribution of branch
(A) and branch (B) and obverse that both modules improve prediction accuracy
in the average case. Branch (A) introduces global context as a constraint to
exploit image only and brings unstable improvement among various traffic
cases. For instance, based on ResNet-101, the branch (A) has lower accuracy in
shadow, arrow and curved cases compared with the base model but it achieves
better performance in the other cases. Compared with the coarse semantic
outputted by (A), branch (B) has higher accuracy almost in every case, which
verifies the geometric continuity feature is able to represent shape knowledge
of lanes more effectively than image feature. Fig. 6 provides a clear
illustration how branch (A) and branch (B) contributes to the final lane
segmentation.

##  VI Conclusion

In this paper, we propose the ContinuityLearner, a novel deep-learning-based
paradigm, that exploits geometric continuity feature representation conveying
in traffic images and generates smooth semantic for lanes. The geometric
continuity feature, as a compound of class-dependent image feature and pixel’s
spatial location, better represents objects that has weak textual coherency
but strong geometric continuity in images. Specifically, the paradigm consists
of a rough segmentation branch and a refined segmentation branch. In the rough
segmentation branch, coarse semantic of lanes and class-dependent feature maps
are calculated by a ResNet-based CNN with the context-encoding module. Fusing
the coarse semantic and the class-dependent feature maps together, the
geometric continuity feature is generated by an encoding layer. Finally, a
MLP-based network performs on the geometric continuity feature and completes
the course segmentation into a smooth and integrated semantic of lanes.
Experiments over Tusimple and CULane datasets show that our proposed method
delivers outstanding performance on lane instance segmentation and outperforms
other existing methods, especially in scenes that lack visual information of
lanes.

## References

  * [1] M. Aly, “Real time detection of lane markers in urban streets,” in _2008 IEEE Intelligent Vehicles Symposium_. IEEE, 2008, pp. 7–12.
  * [2] H. Deusch, J. Wiest, S. Reuter, M. Szczot, M. Konrad, and K. Dietmayer, “A random finite set approach to multiple lane detection,” in _2012 15th International IEEE Conference on Intelligent Transportation Systems_. IEEE, 2012, pp. 270–275.
  * [3] Y. Jiang, F. Gao, and G. Xu, “Computer vision-based multiple-lane detection on straight road and in a curve,” in _2010 International Conference on Image Analysis and Signal Processing_. IEEE, 2010, pp. 114–117.
  * [4] M. Nieto, L. Salgado, F. Jaureguizar, and J. Arróspide, “Robust multiple lane road modeling based on perspective analysis,” in _2008 15th IEEE International Conference on Image Processing_. IEEE, 2008, pp. 2396–2399.
  * [5] X. Li, Q. Wu, Y. Kou, L. Hou, and H. Yang, “Lane detection based on spiking neural network and hough transform,” in _2015 8th International Congress on Image and Signal Processing (CISP)_. IEEE, 2015, pp. 626–630.
  * [6] J. Niu, J. Lu, M. Xu, P. Lv, and X. Zhao, “Robust lane detection using two-stage feature extraction with curve fitting,” _Pattern Recognition_ , vol. 59, pp. 225–233, 2016.
  * [7] K.-Y. Chiu and S.-F. Lin, “Lane detection using color-based segmentation,” in _IEEE Proceedings. Intelligent Vehicles Symposium, 2005._ IEEE, 2005, pp. 706–711.
  * [8] Z. Teng, J.-H. Kim, and D.-J. Kang, “Real-time lane detection by using multiple cues,” in _ICCAS 2010_. IEEE, 2010, pp. 2334–2337.
  * [9] H. Loose, U. Franke, and C. Stiller, “Kalman particle filter for lane recognition on rural roads,” in _2009 IEEE Intelligent Vehicles Symposium_. IEEE, 2009, pp. 60–65.
  * [10] X. Pan, J. Shi, P. Luo, X. Wang, and X. Tang, “Spatial as deep: Spatial cnn for traffic scene understanding,” in _Thirty-Second AAAI Conference on Artificial Intelligence_ , 2018.
  * [11] B. He, R. Ai, Y. Yan, and X. Lang, “Accurate and robust lane detection based on dual-view convolutional neutral network,” in _2016 IEEE Intelligent Vehicles Symposium (IV)_. IEEE, 2016, pp. 1041–1046.
  * [12] A. Gurghian, T. Koduri, S. V. Bailur, K. J. Carey, and V. N. Murali, “Deeplanes: End-to-end lane position estimation using deep neural networksa,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops_ , 2016, pp. 38–45.
  * [13] J. Li, X. Mei, D. Prokhorov, and D. Tao, “Deep neural network for structural prediction and lane detection in traffic scene,” _IEEE transactions on neural networks and learning systems_ , vol. 28, no. 3, pp. 690–703, 2017.
  * [14] D. Neven, B. De Brabandere, S. Georgoulis, M. Proesmans, and L. Van Gool, “Towards end-to-end lane detection: an instance segmentation approach,” in _2018 IEEE Intelligent Vehicles Symposium (IV)_. IEEE, 2018, pp. 286–291.
  * [15] Z. Wang, W. Ren, and Q. Qiu, “Lanenet: Real-time lane detection networks for autonomous driving,” _arXiv preprint arXiv:1807.01726_ , 2018.
  * [16] X. Pan, J. Shi, P. Luo, X. Wang, and X. Tang, “Spatial as deep: Spatial cnn for traffic scene understanding,” in _Thirty-Second AAAI Conference on Artificial Intelligence_ , 2018.
  * [17] Y. Su, Y. Zhang, T. Lu, J. Yang, and H. Kong, “Vanishing point constrained lane detection with a stereo camera,” _IEEE Transactions on Intelligent Transportation Systems_ , vol. 19, no. 8, pp. 2739–2744, 2018.
  * [18] S. Lee, J. Kim, J. Shin Yoon, S. Shin, O. Bailo, N. Kim, T.-H. Lee, H. Seok Hong, S.-H. Han, and I. So Kweon, “Vpgnet: Vanishing point guided network for lane and road marking detection and recognition,” in _Proceedings of the IEEE International Conference on Computer Vision_ , 2017, pp. 1947–1955.
  * [19] J. Zhang, Y. Xu, B. Ni, and Z. Duan, “Geometric constrained joint lane segmentation and lane boundary detection,” in _Proceedings of the European Conference on Computer Vision (ECCV)_ , 2018, pp. 486–502.
  * [20] C. R. Qi, H. Su, K. Mo, and L. J. Guibas, “Pointnet: Deep learning on point sets for 3d classification and segmentation,” _CoRR_ , vol. abs/1612.00593, 2016. [Online]. Available: http://arxiv.org/abs/1612.00593
  * [21] H. Zhang, J. Xue, and K. Dana, “Deep ten: Texture encoding network,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2017, pp. 708–717.
  * [22] _Tusimple Benchmark_ , 2017, available: http://benchmark.tusimple.ai.
  * [23] D. Neven, B. De Brabandere, S. Georgoulis, M. Proesmans, and L. Van Gool, “Towards end-to-end lane detection: an instance segmentation approach,” in _2018 IEEE Intelligent Vehicles Symposium (IV)_. IEEE, 2018, pp. 286–291.
  * [24] C. Szegedy, S. Ioffe, V. Vanhoucke, and A. A. Alemi, “Inception-v4, inception-resnet and the impact of residual connections on learning,” in _Thirty-First AAAI Conference on Artificial Intelligence_ , 2017.
  * [25] J. Philion, “Fastdraw: Addressing the long tail of lane detection by adapting a sequential prediction network,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2019, pp. 11 582–11 591.
  * [26] M. Ghafoorian, C. Nugteren, N. Baka, O. Booij, and M. Hofmann, “El-gan: embedding loss driven generative adversarial networks for lane detection,” in _Proceedings of the European Conference on Computer Vision (ECCV)_ , 2018, pp. 0–0.
  * [27] Y. Hou, Z. Ma, C. Liu, and C. C. Loy, “Learning lightweight lane detection cnns by self attention distillation,” in _Proceedings of the IEEE International Conference on Computer Vision_ , 2019, pp. 1013–1021.
  * [28] F. Visin, K. Kastner, K. Cho, M. Matteucci, A. Courville, and Y. Bengio, “Renet: A recurrent neural network based alternative to convolutional networks,” _arXiv preprint arXiv:1505.00393_ , 2015.
  * [29] P. Krähenbühl and V. Koltun, “Efficient inference in fully connected crfs with gaussian edge potentials,” in _Advances in neural information processing systems_ , 2011, pp. 109–117.