# A Review of Driver Gaze Estimation and Application in Gaze Behavior
Understanding

Pavan Kumar Sharma  Pranamesh Chakraborty  Department of Civil Engineering,
Indian Institute of Technology Kanpur, Kanpur-208016, U.P, India

###### Abstract

Driver gaze plays a key role in different gaze-based applications, such as
driver attentiveness detection, visual distraction detection, gaze behavior
understanding, and building driver assistance system. The main objective of
this study is to perform a comprehensive summary of driver gaze fundamentals,
methods to estimate driver gaze using machine learning (ML) based technique,
and its applications in real world driving scenarios. We first discuss the
fundamentals related to driver gaze, involving head-mounted and remote setup
based gaze estimation and the terminologies used for driver gaze behavior
understanding. Next, we list out the existing benchmark driver gaze datasets,
highlighting the collection methodology and the equipment used for such data
collection. This is followed by a discussion of the algorithms used for driver
gaze estimation, which primarily involves traditional machine learning and
deep learning (DL) based techniques. The estimated driver gaze is then used
for understanding gaze behavior while maneuvering through intersections, on-
ramps, off-ramps, lane changing, determining the effect of roadside
advertising structures and also for developing driver gaze based applications
such as maneuver prediction, driver inattention, and distraction detection
systems, etc. Finally, we discuss the limitations in the existing literature,
challenges, and future scope in driver gaze estimation and gaze-based
applications.

###### keywords:

Driver gaze , gaze estimation , driver gaze datasets , driver gaze behavior
understanding

##  1 Introduction

Driver safety is one of the major global concerns due to the increasing number
of road crashes yearly. According to the Global status report on road safety
2018 by World Health Organization (WHO), approximately 1.3 million people die
every year from road crashes [Organization et al., 2018]. There are several
causes of road crashes, out of which distracted driving, drowsiness, and
inattentiveness of the driver from their surrounding traffic are also
significant. Driver gaze is an important clue to measuring distraction [Koay
et al., 2022, Misra et al., 2023] and attentiveness [Jha et al., 2023] to the
surroundings. Although research on driver gaze estimation has been mostly
carried out during the last two to three decades, however, the history of
human gaze estimation dates back to the nineteenth century. In the early 20th
century, it was limited to the medical field with invasive gaze-estimation
techniques [Mowrer et al., 1935]. Due to the advancement of technology over
the past few decades, gaze estimation has become one of the critical research
field. In addition to driver gaze estimation, estimation of human gaze is used
in many other applications, such as human-computer interaction [Majaranta &
Bulling, 2014, Shimata et al., 2015, Prabhakar & Biswas, 2021, Pathirana et
al., 2022a], health care and medical field [Hansen & Ji, 2009, König &
Buffalo, 2014, Harezlak & Kasprowski, 2018], education and e-learning [Rosch &
Vogel-Walcutt, 2013, Sun et al., 2017], consumer psychology and marketing
[Recarte et al., 2008, Tomas et al., 2021, Senarath et al., 2022, Pathirana et
al., 2022b], etc. In the driving context, the gaze can be estimated using
intrusive or non-intrusive methods. In an intrusive technique, drivers wear a
head-mountable eyeglass setup or eye trackers, some of which look like normal
eyeglasses. In contrast, non-intrusive techniques, remote gaze tracking
systems are used, which classify the driver gaze in several areas of interest
(AOI). In the literature, AOI is also known as gaze zone or gaze class. In
this paper, we will primarily use the term gaze zone in the case of remote
gaze tracking.

Several review papers exist on gaze tracking [Kar & Corcoran, 2017, Khan &
Lee, 2019, Akinyelu & Blignaut, 2020, Klaib et al., 2021, Shehu et al., 2021,
Pathirana et al., 2022a, Wang et al., 2021, Zheng et al., 2022] in different
fields such as manufacturing and logistics, consumer psychology and marketing,
human-computer interaction (HCI) etc. A few of these studies also focus on
driver gaze tracking [Kar & Corcoran, 2017, Klaib et al., 2021, Pathirana et
al., 2022a]. However, these studies have not done an in-depth analysis of
driver gaze tracking and did not discuss how it differs from gaze tracking in
other platforms, specifically in terms of metrics, feature extraction, and
algorithms used for driver gaze tracking. Prabhakar & Biswas [2021] reviewed
different interactive automotive user interfaces, including the gaze
controlled ones which can help drivers perform secondary tasks while driving.
However, this study didn’t cover driver gaze behavior understanding and other
driver gaze related application areas.

A few studies have focused on driver gaze tracking and its application in
inattention, distraction, and drowsiness detection. Koay et al. [2022]
primarily focused on different models of driver inattention measurement for
assistive and autonomous driving and discussed driver face and eye datasets.
However, this study focused on driver distraction detection using various
physiological (e.g., EEG, ECG) and external sensors (steering angle, GPS,
etc.) along with visual sensors for gaze tracking (camera types and eye
tracker). On the other hand, Khan & Lee [2019] and Kotseruba & Tsotsos [2022]
reviewed different driver eye-tracking techniques and attention models,
specifically related to driver gaze. However, the discussion was limited to
different inattention, distraction, and drowsiness detection techniques, which
can be used to measure attention levels and build driver monitoring systems
and advanced driver assistance systems (ADAS). They did not discuss regarding
the uses of driver gaze tracking in understanding driver behavior in complex
situations such as maneuvering through intersection, lane-changing and
overtaking behavior and its significance for safer road infrastructure design.
To fill these existing gaps, the present review study primarily makes the
following contributions:

  1. 1.

Discussion on metrics for driver gaze used in both head-mounted and remote
setup and their relationship to each other.

  2. 2.

In-depth analysis of the features and models specifically useful for driver
gaze estimation.

  3. 3.

Uses of driver gaze specifically in driver behavior understanding along with
application in different driving fields such as building driver maneuver
recognition and prediction systems, ADAS along with distraction detection and
inattention measurement system.

  4. 4.

Discuss different driver gaze estimation challenges and their possible
solutions and future scope.

The structure of the paper is illustrated in Fig. 1 and the remainder of the
paper is organized as follows. Section 2 describes the literature search and
selection criteria for review, followed by Section 3, which describes driver
gaze representation, driver gaze estimation setups, and different
terminologies related to driver gaze. Existing driver gaze benchmark datasets
with their pros and cons are covered in Section 4, while different gaze
estimation algorithms and models used in appearance-based and geometric model-
based methods are discussed in Section 5. Section 6 covers the uses of driver
gaze in gaze behavior understanding and different driver gaze-based
applications, such as building advanced driving assistance systems. Finally,
Section 7 provides a general discussion and future scope, including the
existing studies challenges and limitations, followed by a conclusion at the
end of the paper.

Fig. 1: Structure of the paper

##  2 Literature Search and Selection

In this review study, we have conducted a systematic literature review using
the Preferred Reporting Items for Systematic reviews and Meta-Analyses
(PRISMA) 2020 report [Page et al., 2021]. Following the PRISMA guidelines, we
have followed the search strategy, exclusion and inclusion criteria (see Fig.
2), details of which are discussed next.

Fig. 2: Literature selection criteria based on PRISMA technique

###  2.1 Search strategy

We have passed a thorough search on Google Scholar and Scopus using search
keywords, which can be categorized into three categories. The first category
is related to gaze estimation, predominantly for driver gaze. It includes the
keywords: _gaze estimation, driver gaze estimation, driver head and eye pose
estimation, gaze tracking, driver gaze, and eye tracking_. The second category
is related to driver gaze-based behavior understanding, which includes the
keywords: _driver gaze behavior, driver gaze behavior at intersections, lane
changing gaze behavior, and over passing gaze behavior_. The third category is
related to diver gaze-based applications, which includes the keywords: _driver
gaze distraction, driver inattentiveness, gaze-based advanced driving
assistance system, and gaze-based driver warning system._

###  2.2 Inclusion/Exclusion criteria

We have considered the papers which are published in scientific journals and
conference proceedings. Initially, literature samples included approximately
1100 papers from Google Scholar and Scopus using the search keywords discussed
above. After removing the duplicates, 621 remaining papers were considered for
screening (second stage). In the first stage of screening, the remaining
papers were screened based on the papers’ titles, abstract, keywords and
dissemination outlets. We have considered the following selection criteria for
this stage. 1) selected paper includes driver gaze estimation, 2) reports
results of driver gaze behavior understanding, 3) use of driver gaze in any
application, such as driver assistance, driver safety, warning or alert
system, 4) review papers on gaze estimation or gaze application in different
consumer platforms such as retail stores, manufacturing and logistics,
navigate menus in smart TVs, gaze-based input keyboards for laptops, etc. Out
of the above four criteria, if anyone is satisfied, the paper is considered
for the second screening stage. In second screening stage, selected papers
that fulfill first three criteria are further checked to find whether they
have considered passenger car or not. Only the passenger car-based study has
been considered for the refined list of paper analyses. Finally, 154 relevant
research papers published until June 30, 2023, have been considered in this
present study.

###  2.3 Included journals and conferences

The research papers reviewed in this study are from different reputed journals
such as Knowledge-Based Systems, Expert Systems with Applications, Engineering
Applications of Artificial Intelligence, Transportation Research Part A:
Policy and Practice, Transportation Research Part C: Emerging Technology,
Transportation Research Part F: Traffic Psychology and Behavior,
Transportation Research Interdisciplinary Perspectives, Transportation
Engineering (TRENG), Journal of Eye Movement Research, International Journal
of Human–Computer Interaction (IJHCI), Journal of Safety Research, Pattern
Analysis and Machine Intelligence, Transactions on Intelligent Transportation
Systems, Transactions on Intelligent Vehicles, Transactions on Vehicular
Technology, Transactions on Information Theory, IET Intelligent Transport
Systems, Spanish Journal of Psychology, International Journal of Robotics
Research, etc. We have also included conference papers such as Intelligent
Transportation Systems Conference, Intelligent Vehicles Symposium (IV),
Computer Vision and Pattern Recognition (CVPR), International Conference on
Computer Vision (ICCV), European Conference on Computer Vision (ECCV),
International Conference on Automatic Face and Gesture Recognition,
International Conference on Robotics and Biomimetics, International Conference
on Learning representations (ICLR), Association for Computing Machinery (ACM),
International Conference on Automotive User Interfaces and Interactive
Vehicular Applications (AutoUI), International Conference on Intelligent User
Interfaces (IUI), and Conference on Human Factors in Computing Systems (CHI).

##  3 Driver Gaze Tracking Fundamentals

In the literature, estimation of driver gaze is referred by different
terminologies, such as eye pose estimation, eye tracking [Liu et al., 2002],
gaze estimation, gaze detection [Naqvi et al., 2018, Yoon et al., 2019,
Rangesh et al., 2020], and gaze tracking. However, all these are similar and
often used interchangeably in different research papers. This section begins
with a discussion of how driver gaze output is represented in the driving
field. Subsequently, head-mounted and remote setups for driver gaze estimation
are explored, highlighting their advantages and disadvantages. The discussion
concludes with an overview of key terminologies used for driver gaze behavior
understanding and gaze-based different driving applications.

###  3.1 Representation of driver gaze

In the driving field, gaze estimation is a technique to determine where a
driver is looking. Existing literature on driver gaze provides insight into
three common methods of gaze measurement. Firstly, the zone-based approach
categorizes the driver gaze location into different discrete gaze zones. These
zones encompass areas such as the windshield, dashboard, forward, speedometer,
rearview mirror, left-wing mirror, right-wing mirror, etc. The output of this
method is the gaze zone class, offering a discrete type of driver gaze
estimation [Chuang et al., 2014, Tawari et al., 2014b, Diaz-Chito et al.,
2016, Vora et al., 2018, Ribeiro & Costa, 2019, Ortega et al., 2020, Ghosh et
al., 2021]. The second approach of driver gaze measurement is described using
the gaze direction, gaze vector, and point of gaze. The gaze direction is
defined as the ray that goes through the cornea centre and the fovea.  As
shown in Fig. 3, the deviation (Kappa) of the visual axis from the optical
axis determines the gaze direction. The point on the plane or the object
surface on which the gaze direction intersects is called the point of gaze
[Sun et al., 2016]. In a head-mounted setup, the point of gaze is the
intersecting point on the scene (object’s surface) captured by the scene
camera. In contrast, a gaze vector is the mathematical representation of gaze
direction in 3D space [Yang et al., 2019, Yuan et al., 2022]. It typically
includes three components: (x, y, z), where each component represents the gaze
direction along one of the coordinate axis, where x and y represent gaze in
horizontal and vertical directions while z represents the depth component of
gaze. The depth component z provides information about how far or near the
driver is looking. The third approach of gaze measurement is based on the gaze
object or entity they are looking at, such as a car, bus, truck, traffic
light, or pedestrian. In this case, the output of driver gaze estimation is
the class of the gaze traffic entity [Palazzi et al., 2018, Dua et al., 2020].
Traditionally, gaze zone-based driver gaze estimation has been mostly
prevalent in the literature. However, a recent shift in trend indicates a
preference for measuring driver gaze in terms of the gaze direction or point
of gaze, which is a more refined and precise method of gaze estimation.

Fig. 3: Geometric model of human eye [Pathirana et al., 2022a]

###  3.2 Setup for driver gaze estimation

Gaze estimation is done using different techniques depending on the
application. Initially, sensors attached to the facial skin, such as electrode
pairs were used to record potential differences during eye movements [Mowrer
et al., 1935] to understand cognitive behavior. This technique, though
accurate, is usually uncomfortable to the users. Due to the advancement of
computer vision-based technology, gaze estimation has seen widespread
application in different fields, including driver assistance and behavior
understanding. Head-mounted (wearable) and remote setup (non-wearable) are the
two commonly used setups in practice for driver gaze estimation, which are
briefly discussed next.

####  3.2.1 Head mounted gaze estimation

In head-mounted gaze estimation, driver mounts or wear a device on their head
called an eye tracker. Fig. 4a shows a sample eye-tracker glass, which looks
exactly similar to regular prescription glasses. Primarily head-mounted system
consists of near-eye cameras and infrared LED light for active illumination of
the eyes (Fig. 4b) and a scene camera (Fig. 4c). In this system, a near-eye
camera is present near each eye, recording the eye movements from close-up, as
shown in Fig. 4e. The scene camera records the frontal view, allowing
correlation of gaze data to the cues and stimuli present in the scene of the
driver (Fig. 4f).

Head-mounted eye trackers rely on detecting image features for the near eye
cameras, including the pupil, iris contours (Fig. 4d), and glints, i.e.,
reflections produced by the Infrared LEDs. Typically, eye trackers require
calibration of each driver before estimating the driver gaze [Dukic & Broberg,
2012, Scott et al., 2013, Lemonnier et al., 2015, 2020]. However, some latest
eye trackers have been developed which are calibration-free [Tonsen et al.,
2020]. Several studies have used eye trackers to understand the driver gaze
behavior in indoor (simulation-based studies) [Romoser et al., 2013, Scott et
al., 2013, Lemonnier et al., 2015] and outdoor traffic environments (real-
world driving studies) [Bao & Boyle, 2009, Dukic & Broberg, 2012, Li et al.,
2019, Lemonnier et al., 2020]. In these studies, driver gaze output, has been
measured in any of the three forms (gaze zones, gaze direction, gaze on
traffic entity), which are discussed in Section 3.1.

Head-mounted gaze estimation in the driving field offers significant insights
into driver gaze behavior understanding and gaze-based application. The
primary advantages of this gaze estimation setup are high accuracy, fine-
grained gaze data, and reduced environmental noise. Since the driver wears the
head-mounted gaze estimation device (see Fig. 5a), it captures the eyes and
head movement information from a very close distance. Because of this, the
gaze estimation accuracy is relatively high, and the output is typically gaze
direction or point of gaze. This system can capture gaze-related metrics such
as fixation, saccade, and pupil dilation of driver gaze, which will be
discussed next. This setup is also relatively less affected by environmental
factors such as adverse lighting conditions. However, the head-mounted gaze
estimation system has some common disadvantages, such as intrusiveness,
limited comfort and accessibility, and frequent calibration. Head-mounted gaze
estimation systems typically require the driver to wear a device on their
head, which can be uncomfortable and intrusive, especially for those who do
not wear regular eyeglasses. It is challenging to continuously use the system
for long drives. This can lead to discomfort, distraction, or reduced
situational awareness, which is counterproductive for safe driving. Further
proper calibration is crucial for the accuracy of head-mounted gaze estimation
systems. Calibrating the system correctly for each user can be time-consuming,
and inaccuracies in calibration can lead to misleading results.

Fig. 4: (a) Eye tracking glass (Pupil Invisible) (b) Near Eye camera (green
circle) and LED light (red circle) (c) Scene camera (d) Different parts of an
Eye (e) Left and Right Eye Infrared images captured by using Near Eye Cameras
(f) Driver frontal view recorded by Scene Camera (blue circles represent
fixations).

####  3.2.2 Remote setup gaze estimation

For remote setup-based gaze estimation, cameras are typically placed on the
dashboard [Chuang et al., 2014, Vicente et al., 2015, Naqvi et al., 2018, Yoon
et al., 2019, Wang et al., 2019] or sometimes mounted on the windshield [Choi
et al., 2016]. Single or multiple cameras can be used, depending on the nature
of gaze zone classification. Typically, single cameras are preferred when
dashboard and windshield areas are divided into coarser gaze zones, while in
finer gaze zones classification, multiple cameras are primarily used [Naqvi et
al., 2018, Yoon et al., 2019, Guo et al., 2019]. A detailed discussion on
coarser and finer gaze zones is given in Section 4.2. Multiple cameras capture
face images from different angles, such as left and right sides of the face
and eye [Yoon et al., 2019]. This helps to make gaze estimation robust even
for substantial head movement, which is a shortcoming of single camera-based
estimation. Typically, the output of remote setup gaze estimation is the gaze
zone class, but in very few studies [Yang et al., 2019, Yuan et al., 2022],
the output is the point of gaze.

The non-intrusive nature of the remote gaze estimation system eliminates some
of the limitations of head-mounted gaze estimation. The driver is not required
to wear any head-mounted and intrusive devices, which can lead to a more
natural and comfortable driving experience. However, these systems typically
have lower accuracy compared to the head-mounted gaze estimation technique.
Since they are placed on the dashboard or mounted on a windshield, it has
difficulty capturing detailed information about eye movements, such as the
position or movement of the pupil and cornea inside the eye, making it less
accurate than head-mounted device. Most remote setup systems do not capture
gaze metrics data such as fixation and pupil dilation, which is essential for
different driver applications. Further, variations in driver posture and seat
position also affect the gaze estimation accuracy.

###  3.3 Terminologies used for gaze estimation

This section discusses the different terminologies related to head-mounted and
remote setup based temporal characteristics of driver gaze. This involves
studying the duration for which a driver looks or focuses his/her attention on
a particular object or region. Typically, the terms fixation, saccades,
scanpath, blink rate, and pupil dilation are related to temporal gaze
estimation in head-mounted setup, where the driver gaze is estimated as the
point of gaze. On the other hand, in remote setup based gaze estimation, it is
challenging to capture the exact eye movement. Therefore, instead of the point
of gaze, gaze zone class is the estimated gaze output. In this case,
primarily, the term glance is related to driver gaze, which depends on the
driver’s head pose and eye pose.

  1. 1.

Fixation: It is the state of eyes in which a driver maintains the visual gaze
in an area of interest (AOI) for a certain period [Dukic & Broberg, 2012, Kar
& Corcoran, 2017]. Most fixations last between 100 to 600 milliseconds, but
this varies based on the context and other relevant factors [Pauszek, 2023].
Remote eye tracker manufacturers such as Smart Eye Pro and Tobii consider 200
ms as the minimum gaze period for a valid fixation in driver behavior studies
[Holmqvist et al., 2022]. The following terms further describe fixation.

    1. (a)

Dwell Time: It is the sum of the duration of fixations in a given area of
interest. In driver behavior studies, dwell time gives the proportion of time
spent by a driver gazing at an object in an interval of time. The object can
be dynamic objects such as moving vehicles or static objects such as
stationary vehicles, traffic lights, traffic signs, road markings, etc.
[Lemonnier et al., 2015, Chung et al., 2022]. Higher dwell time in a given
gaze zone represents a high level of interest of the driver in that particular
zone.

    2. (b)

Time to first fixation (TTFF): It measures how long a driver takes to start
looking at a specific stimulus in a gaze zone after the stimulus is presented.
TTFF indicates how much an aspect of the scene initially attracted attention.

    3. (c)

Number of fixations: It represents the total number of times a subject fixates
their gaze in a given gaze zone in a given time interval [Chung et al., 2022].

  2. 2.

Saccades: Saccades are defined as the rapid eye movement when the gaze sight
shifts from one fixation point to another [Imaoka et al., 2020]. The time
taken to shift the eyes from one fixation point to another is called saccadic
duration. The majority of saccades range in duration from 20 to 80
milliseconds, although this varies considerably depending on the situation
[Pauszek, 2023]. It is also measured using other parameters such as saccade
number, amplitude, and fixation-saccade ratio [Kar & Corcoran, 2017]. It shows
the dynamics of the driver gaze while driving.

  3. 3.

Scanpath: Scanpath represents the path followed by the driver’s eyes to reach
from one area of interest to another. It is also defined as the sequence of
fixations and saccades that an individual driver makes while performing a
task, such as lane changing, turning at an intersection, etc. Scanpath
includes scanpath direction, duration, and length [Kar & Corcoran, 2017,
Pauszek, 2023]. Scanpath duration refers to the total time taken to complete a
scanpath and it can be measured by summing the cumulative time spent on
fixations and saccades during the task. Scanpath length typically refers to
the spatial extent covered by the eyes during a scanpath. It represents the
total distance travelled by the eyes while performing a task and is often
measured in visual degrees or pixels.

  4. 4.

Glances: Glances are the coarser measurement of gaze, while fixation is a more
refined measurement. It measures the gaze over an area, while fixation is
measured at a point. In an interested gaze zone, glances may contain fixation
and saccades. Here, gaze zones are pre-defined areas inside the vehicle cabin,
such as the speedometer, center stack, left wing mirror, right wing mirror,
rearview mirror [Martin et al., 2018, Rangesh et al., 2020, Schindler &
Piccinini, 2021], etc. Glances are measured using glance duration, glance
frequency, glance transition, glance transition sequence, glance transition
length, and number of glance transitions, which are discussed next.

    1. (a)

Glance Duration: It is the time spent in each gaze zone in a given time
interval. It is similar to the dwell time used to define fixations. It is
typically measured in terms of minimum, maximum, and average glance duration
made by drivers in different gaze zones [Birrell & Fowkes, 2014, Martin et
al., 2018] to understand driver gaze behavior. Longer glances off the road can
be used to detect driver distraction.

    2. (b)

Glance Frequency: The number of glances a driver makes in a given gaze zone in
a unit time interval is glance frequency [Birrell & Fowkes, 2014]. Several
studies include glance duration and frequency in examining the driver
attention level while driving [Muñoz et al., 2016]. Longer glance duration and
higher glance frequency signify the higher task demand [Wang et al., 2014].

    3. (c)

Glance Transition: It denotes shifting of gaze from one gaze zone to another
while assessing the situation of the surrounding traffic environment [Scott et
al., 2013, Lemonnier et al., 2015]. Glance transitions reveal the flow of
attention between different gaze zones while driving. A higher correlation
exists between the two gaze zone when glance transitions occurring between the
two are more frequent [Fridman et al., 2017].

    4. (d)

Glance Transition Sequence: This is the sequence followed by the glances
shifting from one gaze zone to another in a given time duration. The
occurrence of unusual sequences of gaze patterns while driving contains more
rich information than usual.

    5. (e)

Glance Transition Length: This is defined as the time duration for shifting
glances from one specific gaze zone to another. It depends on the position of
the former and later gaze zone class. Higher glance transition length can be
observed when former and later classes are forward and leftwing mirror, as
compared to forward and right wing mirror (see Fig. 6a).

    6. (f)

Number of Glance Transition: It is the sum of glance transitions the driver
makes from one gaze zone to another in a given time interval. A higher number
of glance transitions typically indicates higher driver attentiveness to their
traffic surrounding.

  5. 5.

Pupil size (dilation/constriction) and blink rate: Pupil dilation refers to an
increase in pupil size, while constriction denotes a decrease in size. In a
real-world driving study [Heger, 1998], the mental workload of the driver has
been measured based on the blink rate. The findings of this study revealed
that as the road curvature increased, the eye blink rate reduced. Blink rate
has been found to decrease for visually demanding tasks [Marquart et al.,
2015], while shorter blink duration has been observed for mentally and
visually demanding tasks [Kramer, 2020].

  6. 6.

Entropy Rate: Entropy rate, inspired by the concept of information entropy, is
one of the critical metrics used in head-mounted and remote setup-based gaze
estimation to measure driver attentiveness to the surrounding traffic.
[Shannon, 1948]. In a driving scenario, stationary gaze entropy (SGE) and gaze
transition entropy (GTE) are two commonly used metrics for measuring driver
attention level.

Stationary gaze entropy [Bao & Boyle, 2009, Chung et al., 2022] describes the
information generated by the driver gaze dispersed across the gaze zones.
Stationary gaze transition is defined as:

$SGE=-\sum_{l=1}^{L}p_{l}\log_{2}p_{l}$ (1)

where $L$ is the total number of gaze zones. The probability of the driver
looking towards gaze zone $l$ is represented by $p_{l}$. SGE does not reflect
how drivers control and assess the situation from the surrounding traffic
environment. To overcome this limitation, gaze transition entropy [Krejtz et
al., 2015, Chung et al., 2022] is used to measure the complexity of the
different gaze transition patterns. For instance, when there are multiple
stimuli, such as intersections, the gaze pattern is complex compared to an
ordinary road; hence the GTE is higher at intersections.

$\text{ GTE
}=-\sum_{k=1}^{L}p_{k}\sum_{k,l=1}^{L}\left(p_{k,l}\right)\log_{2}\left(p_{k,l}\right)\quad\left\\{\begin{array}[]{c}k,l=1,2,3\ldots
L\\\ k\neq l\end{array}\right.$ (2)

Where $p_{k,l}$ is the occurrence of transition probability from gaze zone $k$
to gaze zone $l$.

Fig. 5: (a) Driver gaze estimation using wearable eye tracker: Driver wearing
eye tracking glasses (b) Photograph of instrumented vehicle for remote setup
of gaze estimation A:Capture gaze information, B:Infra-red lamp, C:Scene
camera [Lemonnier et al., 2020].

##  4 Benchmark Datasets and Collection Methodology

Good quality data is one of the crucial needs for computer vision-based gaze-
tracking applications. The quality of driver gaze datasets depends on the
precision and configuration of collection equipment, methodology, and
information level. Driver data must include possible driving scenarios and
conditions and a sufficiently large number of subjects. This section discusses
the different types of equipment used for data collection, methodology
adopted, and different open-source benchmark driver gaze datasets available
for gaze estimation model development.

###  4.1 Equipment used for data collection

Driver face data is generally collected using a remote setup, where cameras
are installed in front of the driver (on the dashboard or windshield) to
record the driver’s head movement. Different types of cameras are used to
collect driver face data, including traditional RGB (red, green, and blue),
RGB-Depth, and Infrared cameras. Traditional RGB cameras are used to capture
driver face in the visible wavelength spectrum [Chuang et al., 2014, Tawari et
al., 2014b, Diaz-Chito et al., 2016, Vora et al., 2018, Ribeiro & Costa, 2019,
Ortega et al., 2020, Ghosh et al., 2021]. However, the image quality
sufficiently degrades in low-light conditions, making it difficult to
understand driver gaze at night. To alleviate this problem, Infrared
(IR)[Ribeiro & Costa, 2019, Rangesh et al., 2020, Ortega et al., 2020]/Near
Infrared (NIR) cameras are also used for driver gaze data collection [Nuevo et
al., 2010, Naqvi et al., 2018, Yoon et al., 2019] due to its intrinsic
advantage to capture better features in night compared to traditional RGB
cameras. It provides a gray-scale image using infrared/near-infrared light.
Although NIR cameras are robust enough to capture images in low-light
conditions, however, prolonged use of the NIR camera may hurt the driver’s
eyes [Ou et al., 2021]. The basic challenge in driver gaze estimation is the
illumination vulnerability under poor environmental conditions where light and
shade bring negative effects. Standard RGB cameras have the advantage of color
information but are missing depth information. To overcome these challenges,
RGB-D cameras [Ribeiro & Costa, 2019, Wang et al., 2019, Ortega et al., 2020,
Liu et al., 2022] have been used to obtain RGB images and depth information
using point cloud-based sensors. RGB-D camera has unique features to merge
pixel-to-pixel information of depth and RGB information in a single image.
Depth information of the camera is provided by a 3D depth sensor, which can be
stereo, time of flight, structured light sensor, etc. In some studies, eye
trackers are also used to capture data such as pupil dilation, iris, and gaze
information, in terms of fixation and saccades [Palazzi et al., 2018].

###  4.2 Collection methodology

Driver gaze data can be collected using a vehicle in stationary state (parked
vehicle) or moving state. In the stationary state [Chuang et al., 2014, Naqvi
et al., 2018, Jha & Busso, 2018, Yoon et al., 2019, Ribeiro & Costa, 2019,
Ghosh et al., 2021], dashboard and windshield areas of the vehicle (car) are
divided into several zones by sticking stickers or by pointing with a marker.
The number of gaze zones can be broadly divided into coarser and finer
categories, as shown in Fig. 6a and Fig. 6b. In the coarser gaze zone
classification, the windshield and dashboard area are divided into fewer gaze
zones than the finer. Primarily coarser gaze zone classification includes
forward, speedometer, center-stack, left wing mirror, right wing mirror, and
rearview mirror [Nuevo et al., 2010, Chuang et al., 2014, Tawari et al.,
2014b, Fridman et al., 2016a, Martin et al., 2018, Vora et al., 2018, Rangesh
et al., 2020]. In finer gaze zone classification, the windshield, wing mirror,
and dashboard areas are subdivided into smaller gaze zones, resulting in more
gaze zones. Despite giving the specific class such as forward, rearview
mirror, left-wing mirror, right-wing mirror, etc., they divided each broader
gaze zone area into a smaller gaze zone with the class a numerical value such
as 1, 2, 3, etc. [Vicente et al., 2015, Naqvi et al., 2018, Yoon et al., 2019,
Ghosh et al., 2021, Wang et al., 2022b] as shown in Fig. 6b. In some studies,
instructions are given to the driver by a second person to the subjects to
look toward specific gaze zones [Ribeiro & Costa, 2019], while in some other
studies, drivers typically look toward the gaze zones by their own choice
[Chuang et al., 2014, Ortega et al., 2020]. The frontal face area of the
driver is recorded by installing the camera on the dashboard, rearview mirror,
or windshield (see Fig. 5b). One or more human annotators label the captured
frames and cross-verify their labels with each other. Speak2label [Ghosh et
al., 2021] is another method of annotations in which audio signals (gaze zone
class such as 1, 2, 3, etc.) are converted into text. In the moving state, we
can not instruct the driver to look toward the specific gaze zone due to the
risk of driver safety concerns. In this state, the driver drives the vehicle
actually on the road, and the labels are given either by the human annotators
[Fridman et al., 2016b, Vora et al., 2018, Wang et al., 2019], using eye-
trackers, or using unsupervised machine learning (ML) techniques [Chuang et
al., 2014].

Fig. 6: Gaze zone classification. (a) Coarser gaze zones (b) Finer gaze zones.

The primary advantage of stationary vehicle-based data collection methods is
the associated safety and more control during data collection since data is
collected on parked vehicles. Getting the gaze zone label using speak2label or
looking towards the specific gaze zone based on the instructions given by
another person is relatively smooth. The drawback of this method is the gaze
zone classifiers built using this data are not generalized satisfactorily from
a stationary to a moving vehicle. Also, some classes can be intermingled
thereby affecting the classifier results. The moving vehicle based data
collection method is more generalized since data is collected in actual road
driving, however, interpreting the correct gaze zone labels is challenging.

###  4.3 Driver gaze datasets

Driver gaze datasets can divided into two categories based on the details of
the information captured. The first category contains driver face information,
and the second category includes the eye pose information, such as the
position of the iris, pupil, and cornea reflection inside the eye. Typically,
driver face and eye data are collected using cameras [Fusek, 2018] while iris
and pupil data are collected using eye trackers [Palazzi et al., 2018].

####  4.3.1 Driver face benchmark datasets

Several open-source driver face datasets are available, which are collected
either inside parked vehicles or moving vehicles in real-world conditions.
These datasets can be downloaded from different open-source repositories or
available on request through a source generator. A detailed description of
these publicly available driver gaze datasets is discussed next, and a
comprehensive summary is provided in Table 1.

RS-DMV [Nuevo et al., 2010] contains grayscale face videos of ten drivers (see
Fig. 7a) in indoor (simulator) and on-campus outdoor driving. DriveFace [Diaz-
Chito et al., 2016] contains three classes: right, frontal, and left head pose
(see Fig. 7b). Brain4Cars [Jain et al., 2015a] consists of multi-sensor
synchronized data containing outside and inside views of car, vehicle speed,
and GPS coordinates. The data was recorded using ten drivers in natural
driving settings for up to two months. DriveAHEAD [Schwarz et al., 2017]
dataset is a wide-range head pose dataset containing depth and IR images. To
measure the head position (x, y, z coordinate) and orientation (yaw, pitch,
and roll), they used a 3D motion capture sensor. The DMD [Ortega et al., 2020]
dataset is a multimodal dataset containing images from 3 camera streams (RGB,
IR, Depth). This dataset contains head pose, body pose, blink rate, and hand
wheel interactions. DriveMVT [Othman et al., 2022] is a multi-purpose natural
driving data consisting of frame-by-frame information of driver health, such
as heart rate, mental fatigue, head pose (yaw, pitch, roll), drowsiness, etc.
The data was collected using USB cameras and smartphone cameras to capture
driver face, while heart rate recording was obtained uing Xiaomi Mi Band 3
sensor. All these datasets were collected when the vehicle was in movement
state.

Driver gaze datasets are also recorded inside parked vehicles. One such
dataset is LISA GAZE v2 [Vora et al., 2018], a large-scale driver face data
containing diverse driving conditions such as daylight, night light, harsh
illumination, and eyeglasses reflections (see Fig. 7c). DG-UNICAMP [Ribeiro &
Costa, 2019] is another extensive driver face dataset containing three camera
types RGB, IR (see Fig. 7d), and depth (see Fig. 7e) images. Driver Gaze in
Wild (DGW) dataset [Ghosh et al., 2021] is one of the largest datasets in
terms of the participants involved. This study used speak-to-label technique
to label data. It contains different challenging lighting conditions, such as
low light (see Fig. 7f), half/full face shadow, sunlight reflection, etc. The
drawback of the data is its intermingled gaze classes, which can reduce the
gaze classification accuracy.

Fig. 7: Open source benchmark driver’s face datasets image samples. (a) RS-DMV [Nuevo et al., 2010] (b) DriveFace [Diaz-Chito et al., 2016] (c) LISA GAZE v2 [Rangesh et al., 2020] (d)DG-UNICAMP [Ribeiro & Costa, 2019] (e) DG-UNICAMP [Ribeiro & Costa, 2019] (f) DGW [Dua et al., 2020].  Table 1: Open source driver gaze datasets. | Name
---
| Camera
---
type
Resolution |  | No. of
---
subjects
(F/M)
| Gaze
---
zones
Size | Merits | Demerits
| RS-DMV
---
[Nuevo et al., 2010]
Grayscale | 960×240 |  | 10
---
(2/8)
- | 21k |  | Both indoor
---
and outdoor
environments
| Only grayscale
---
images
| DrivFace
---
[Diaz-Chito et al., 2016]
RGB | 640×480 |  | 4
---
(2/2)
3 | 606 |  | Real driving
---
scenarios
| Limited number of
---
subjects, invariability
in lighting condition
| Brain4Cars
---
[Jain et al., 2015a]
RGB | - |  | 10
---
(3/7)
- | 2M |  | 1180 miles
---
city and freeway
real driving data,
with with GPS,
IMU, camera
| Limited number
---
of subjects
| DriveAHEAD
---
[Schwarz et al., 2017]
| Depth
---
/IR
512×424 |  | 24
---
(4/16)
- | 1M |  | Real driving
---
scenarios
| Only depth,
---
IR images
| DR(eye)VE
---
[Palazzi et al., 2018]
RGB | 1280×720 |  | 8
---
(1/7)
- | 555k |  | Real driving
---
scenarios with
different lighting
conditions
| Limited number
---
of subjects
| LISA GAZE v2
---
[Rangesh et al., 2020]
RGB | 2704×1524 |  | 10
---
(4/6)
7 | 47k |  | Different
---
lighting
conditions
Simulator driving
| DG-UNICAMP
---
[Ribeiro & Costa, 2019]
| RGB
---
/IR
/Depth
240×320 |  | 45
---
(10/35)
18 | 1M |  | First driver face
---
dataset which
contains all three
cameras type
(RGB, IR, depth)
Stationary vehicle
| DGW
---
[Ghosh et al., 2021]
RGB | 179×179 |  | 338
---
(91/247)
9 | 50k |  | Large number of
---
subjects with
different lighting
conditions
Stationary vehicle
| DMD
---
[Ortega et al., 2020]
| RGB
---
/IR
/Depth
1920×1080 |  | 37
---
(10/27)
9 | 41h |  | Both real world and
---
simulated driving
scenarios
| Subjects performed
---
predefined activities
such as drinking water,
texting etc.
| DGAZE
---
[Dua et al., 2020]
RGB | 1920×1080 |  | 20
---
(6/14)
- | 100k |  | Gaze dataset on the
---
vehicle entities
e.g. cars, motorcycles,
etc.
Simulator driving

####  4.3.2 Open source driver eyes tracking datasets

While the driver face dataset is the most common driver gaze dataset explored
in literature, a few studies have also used the driver eye dataset for gaze
estimation. Besides gaze estimation tasks, driver eye datasets are also used
for detecting drowsiness, pupil dilation, and blink frequency for cognitive
workload, etc.

DR(eye)VE [Palazzi et al., 2018], is a real-world driving dataset consisting
of 74 videos of different weather (sunny, cloudy, rainy) and light conditions
(day, evening, night). Driver gaze information and pupil dilation were
captured using eye trackers, and the gaze was mapped to the surrounding
traffic. Media research lab (MRL) [Fusek, 2018] dataset was recorded on actual
road driving using a NIR camera to reduce the low illumination light effect on
the eyes during evening and night. IR illuminator is used to create reflection
on eyes and eyeglasses to produce a wide range of lighting effects. The
benchmark driver face and eye datasets are used to build several state-of-the-
art gaze estimation models, which are discussed in the next section.

##  5 Algorithms and Models for Driver Gaze Estimation

Existing driver gaze estimation techniques can be divided into two groups:
Appearance-based methods and Geometric model-based methods. Predominantly,
appearance-based gaze estimation techniques have been used in driver gaze
estimation; however, some studies have also used geometric model-based
techniques. Appearance-based methods can be further subdivided into two
groups: conventional appearance-based methods and appearance-based methods
with deep learning (DL).

This section will discuss the appearance and geometric model-based methods,
along with their strengths and shortcomings. We will start with discussion of
different features and traditional machine learning classifiers (e.g., support
vector machine etc.) used in conventional appearance-based methods. Then, we
will discuss appearance-based deep learning methods, mainly focusing on
convolutional neural network (CNN) models used in driver gaze estimation.
Finally, we will highlight some studies that use geometrical models to
estimate driver gaze.

###  5.1 Appearance based methods

Appearance-based methods derive insights from the visual appearance of driver
face and eyes. Conventional appearance-based methods rely on feature
extractors to extract the features, while appearance-based deep learning
methods learn the features directly through the convolutional layers.

####  5.1.1 Conventional appearance based method

Conventional appearance-based gaze estimation methods rely on handcrafted
features and traditional machine learning models to estimate driver gaze. The
features, such as texture, shape, edges, etc. are extracted from the eyes and
face region.

In conventional appearance-based methods, head pose and eye pose are the two
important aspects of driver gaze estimation. Different features are extracted
from the driver facial image for head pose and eye pose. The common features
for head pose are Euler angle (Yaw, Pitch, and Roll) [Lee et al., 2011, Tawari
et al., 2014b, Tawari & Trivedi, 2014, Wang et al., 2019] or the facial
landmark (such as mouth and nose corner, nose tip, eyebrows, etc.) [Lee et
al., 2011, Chuang et al., 2014, Tawari et al., 2014b, Tawari & Trivedi, 2014].
On the other hand, the commonly used features extracted from the eye region
are eye corners, eye contours, pupil, iris center location, [Chuang et al.,
2014, Tawari et al., 2014b, Fridman et al., 2016b]. The extracted features
using different feature extractors are then fed to conventional machine
learning models for regression based gaze direction estimation or gaze zone
classification. The commonly used traditional machine learning models are
Support Vector Machine (SVM) [Lee et al., 2011, Chuang et al., 2014, Vicente
et al., 2015, Vasli et al., 2016], Random Forest (RF) [Tawari & Trivedi, 2014,
Tawari et al., 2014b, Fridman et al., 2016b, a, Wang et al., 2019], etc. We
will first discuss the driver gaze studies where the gaze output is gaze zone
class such as forward, rearview mirror, etc. (see Fig. 6), followed by the
studies where the gaze output is point of gaze.

For driver gaze estimation based on gaze zone class, Chuang et al. [2014]
estimated coarse head pose direction (frontal, left, and right) using Adaboost
cascade classifiers operating on Haar feature descriptors. Then, only the
frontal face region was fed to extract the location of the left iris, right
iris, mouth, and nose using Adaboost cascade classifier. Finally, the 14
feature vectors comprising of face part location were fed to the ML model (as
demonstrated in Table 2) for the gaze estimation. Study by Tawari & Trivedi
[2014] incorporated both static head pose features and dynamic head pose
features for gaze estimation. Static head pose features based on facial
landmarks and their 3D configuration were determined through a perspective
projection model. The corresponding yaw, pitch, and roll angles were then
obtained using POS (pose from orthography and scaling). For dynamic head pose
feature extraction, raw data from static features were first smoothed. From
the windowed time series (spanning W seconds prior to the current time t), min
and max values, positions of min and max, mean angle, and angular velocity
were extracted. The total 18 dynamic features were then fed into the
classifier to estimate the gaze zone class. However, since this study used
only the head pose, the trained model confused between some nearby classes,
such as forward and the speedometer, center console, and rearview mirror.
Therefore, in another study, Tawari et al. [2014a] added eye pose features
(iris position) to improve accuracy. Fridman et al. [2016b] tested how much
the gaze estimation accuracy increased when the eye pose was included along
with the head pose. For the head pose, 68 points of multi-PIE facial landmark
markup were extracted, while for the eye pose, the pupil center was extracted
by extracting points of iso-contour. Adding eye pose information with head
pose was found to increase 5.4 % accuracy of the gaze zone classification
system.

In the literature, when driver gaze output is a gaze zone class, RF is
preferred over SVM and KNN for several reasons. RF consistently exhibits
higher classification accuracy, as demonstrated in Table 2. Additionally, RF
provides prediction probabilities for each class and requires very few tuning
parameters.

While a significant number of studies have been conducted based on gaze zone
class, limited research have been done on gaze point location. Study by Yang
et al. [2019] used dual cameras for driver gaze mapping, as shown in Fig. 8.
Camera 1 was placed on the front of the driver on the dashboard to capture the
facial features, and camera 2 was placed at the driver’s top to capture the
driver scene images. A nonlinear finite impulse response (NFIR) model was
utilized to depict a system with several inputs and a single output where the
inputs were the face features and the output was the point of gaze on camera 2
image. To increase the accuracy of driver gaze estimation, Yuan et al. [2022]
presented a self-calibrated driver gaze estimation system. Features
extraction, gaze pattern learning, and gaze mapping model training comprised
of three stages. Using OpenFace, they initially extracted features like head
and eye poses and facial landmarks, which were tracked using unscented Kalman
filter. In the gaze pattern learning step, unaligned typical gaze patterns
with gaze features were automatically extracted from the time samples. In the
third step, the gaze was mapped using Gaussian Processes for Regression (GPR),
which builds the relationship between driver status and point of gaze.

Fig. 8: Systematic flow of driver gaze mapping [Yang et al., 2019] using dual
camera. Camera 1: Capture the facial features including eye gaze and head
movement. Camera 2: Used to mapped location of eye gaze in scene images.

####  5.1.2 Appearance based method with deep learning

Recently, deep learning-based driver gaze estimation has gained significant
attention. When contrasted with conventional appearance-based methods, these
methods exhibit several notable advantages. Deep learning based methods
directly learn the mapping functions from the face and eye appearance
characterized by pixel intensity (color intensity) statistics [LRD et al.,
2022, Pathirana et al., 2022a]. Typically, they do not require handcrafted
feature extractors. These methods work well in real-world settings, both in
head-mounted and remote setup-based gaze estimation. They are robust for head
movements and also improve cross-subject gaze estimation performance. Next, we
will discuss different deep learning-based driver gaze estimation models.
Initially, we focus on CNN models used in driver gaze estimation based on gaze
zone classification. Later, we will discuss the studies where estimated gaze
output is point of gaze.

Table 2: Gaze estimation using conventional appearance based method.

Reference |  | No. of
---
cameras
(type)
| Output
---
(no. of
zones)
Features |  | Feature
---
extractor
Model |  | Accuracy (% or °)
---
| Lee et al. [2011]
---
1 (NIR) |  | GZC1
---
(18)
| Head pose (yaw, pitch),
---
face features
(Left, right border and
center of drivers face)
| Head pose by
---
support vector
regressor (SVR)
SVM |  | 47.4 (SCER),
---
87.2 (LCER)
| Chuang et al. [2014]
---
1 (RGB) |  | GZC
---
(2)
| Head pose and eye pose
---
(location of left, right
iris, mouth and nose)
| Haar featur
---
descriptor,
haar cascade
classifier
SVM |  | 97.4
---
| Tawari & Trivedi [2014]
---
| 2 (RGB)
---
| GZC
---
(8)
| Facial land marks
---
(eye and nose corner,
nose tip), static features
of head pose (yaw,
pitch, roll)
| WPPM2 for
---
landmark tracking,
POS3 for determines
the rotation matrix
and corresponding
yaw, pitch and roll
RF |  | 94.7
---
| Tawari et al. [2014b]
---
2 (RGB) |  | GZC
---
(6)
| Head pose (yaw, pitch,
---
roll), eye pose (by pupil),
features such as eye
corners and contours
| WPPM for
---
landmark tracking,
POS for determines
the rotation matrix
and corresponding
yaw, pitch and roll
RF |  | 79.8 (H),
---
94.9 (H + E)4
| Vicente et al. [2015]
---
1 (NIR) |  | GZC
---
(18)
| Facial features (pupil
---
and eye corners)
Shift descriptor | SVM |  | 90
---
| Vasli et al. [2016]
---
1 (RGB) |  | GZC
---
(6)
| Head pose
---
(yaw, pitch, roll)
| Face detection
---
using AlexNet
model
SVM |  | 93.7
---
| Fridman et al. [2016b]
---
1 (NIR) |  | GZC
---
(6)
| 3D Head pose, eye pose
---
by pupil detection.
| DLIB HOG face
---
detection, pupil
center detection
based on CDF5
RF |  | 89.2 (H)
---
94.6 (H+E)
| Fridman et al. [2016a]
---
1 (NIR) |  | GZC
---
(6)
| 19 facial landmarks
---
e.g. nose tip, eye
corner etc.
| DLIB HOG face
---
detection, face
alignment using
68 Multi-PIE
facial landmark
RF |  | 91.4
---
| Wang et al. [2019]
---
1 (RGB-D) |  | PoG6
---
| Head pose (yaw, pitch,
---
roll), localized eye
region, eye pose
| Multi-zone ICP7
---
based head pose
and appearance
based gaze
RF |  | 8.0°
---
| Yang et al. [2019]
---
2 (RGB) |  | PoG
---
| 68 facial landmarks
---
head pose, eye pose
| OpenFace for face
---
detection, conditional
local neural fields
(CLNF) for facial
landmark detection
NFIR8 |  | RMS9
---
| Yuan et al. [2022]
---
1 (RGB) |  | PoG
---
(8)
| 68 facial landmarks
---
head pose, eye pose
| OpenFace,
---
Unscented Kalman filter
(for tracking the feature
over time)
GPR |  | 6.9°
---

  * 1

Gaze zone class;

  * 2

Weak perspective projection model;

  * 3

Pose from orthography and scaling;

  * 4

H-Head pose, E-Eye pose;

  * 5

Cumulative distribution function;

  * 6

Point of gaze;

  * 7

Iterative closet points;

  * 8

Non linear finite impulse response;

  * 9

Root mean square error used to measurement of gaze estimation error, in X and
Y direction in terms of pixels are ((7.80$\pm$5.99)) and (4.64$\pm$3.47).

##### CNN models for driver gaze estimation

Several state-of-the-art CNN-based models mentioned in Table 3 have been used
for driver gaze estimation. These models are generally pre-trained on the
large-scale ImageNet dataset [Deng et al., 2009] and applied to driver face
data, collected using RGB or NIR cameras. We first discuss studies that used
RGB camera data, followed by NIR camera based studies, where the estimated
gaze output is the gaze zone class. Choi et al. [2016] proposed a CNN model
inspired by Alexnet architecture [Krizhevsky et al., 2012], consisting of
three convolutional, three pooling, two fully connected, and one output layer
for nine gaze zone classifications. Similarly, several other studies have
focused on leveraging the advantage of large-scale CNN models by developing
gaze classification models that can generalize for different drivers, driver
position and perspective, lighting conditions, etc., as shown in Table 3. Vora
et al. [2018] collected gaze data of ten drivers in naturalistic driving
conditions during dry weather in the daytime to build a generalized driver
gaze classification model using CNN architecture. They achieved the highest
accuracy of 88.9 % and 93.4 % accuracy in half-face data type by using AlexNet
and VGG16, respectively. The most likely reason for getting higher accuracy on
the upper half face compared to the full face data was that the upper half
face images can extract finer features of the eyes, like the position and
shape of the iris and eyelid. One of the challenges in gaze classification is
reflection or low visibility due to eyeglasses. Rangesh et al. [2020]
attempted to overcome the eyeglass challenges by removing the eyeglasses in
the real driving environment using the Gaze Preserving CycleGAN (GPCycleGAN)
model.

Gaze estimation using RGB cameras discussed above suffers during low light and
sunlight reflection on the driver face. To solve this problem, researchers
collected driver face data using NIR camera. Naqvi et al. [2018] used three
separate VGG models, one each for the driver’s face, left eye, and right eye,
respectively to estimate driver gaze. The accuracy of the proposed system was
measured using two metrics: strictly correct estimation rate (SCER) and
loosely correct estimation rate (LCER). SCER refers to the ratio of the number
of strictly correct frames divided by the total number of frames. The strictly
correct frame represents the frame where the estimated gaze zone is equivalent
to the ground truth gaze zone. LCER refers to the ratio of the number of
loosely correct frames divided by the total number of frames. The loosely
correct frame represents the frame where the estimated gaze zone is placed
within the ground truth gaze zone or its surrounding zones. The system
achieved 92.8 % and 99.6 % accuracy in SCER and LCER, respectively. Using two
NIR cameras, Yoon et al. [2019] captured the drive frontal face and right side
face images. Each face image was further used to detect left and right eyes
using Dlib facial feature trackers. Finally, they combined six images and made
a single image, giving it as a input to the CNN model for gaze estimation.

All the above-discussed studies in this subsection estimated gaze zone class
as the gaze output. However, few studies have also estimated driver gaze in
terms of point of gaze. A study by LRD et al. [2022] developed a system to
estimate driver gaze using a modified Attention-based Gaze Estimation network.
The estimated gaze error across eight gaze regions was 15.61 ± 10.4 cm in the
x-direction and 15.13 ± 14.5 cm in the y-direction.

Overall, deep learning-based CNN models have been preferred for driver gaze
estimation because of their higher ability to incorporate large head movement
and eye movement detection, robustness to variations in vehicle type and
driver head [Kanade et al., 2021], different lighting conditions such as low
light, sunlight reflection on the face, and improved classification accuracy.
These models have been demonstrated to be efficient in handling large-scale
data, which helps in improving the model training to learn complex feature
distribution and thereby finer gaze zone classification.

Table 3: Gaze estimation using appearance based method with deep learning

Reference |  | No. of
---
cameras
(type)
| Output
---
(no. of
zones)
Description |  | Best CNN
---
model
| Accuracy (%)
---
| Choi et al. [2016]
---
1 (RGB) |  | GZC
---
(9)
| Face tracking using
---
MOSSE tracker,
compared with
Haar cascade classifier
AlexNet |  | 95.0
---
| Vora et al. [2018]
---
1 (RGB) |  | GZC
---
(7)
| Three separate data
---
with face, half face,
and face plus context
| SqueezeNet
---
| 92.7
---
| Naqvi et al. [2018]
---
1 (NIR) |  | GZC
---
(17)
| Trained three separate
---
VGG16 on face,
left and right eye
VGG16 |  | 64.8 (SCER),
---
91.1(LCER)
| Yoon et al. [2019]
---
2 (NIR) |  | GZC
---
(15)
| Generate single image
---
by combining 6 images of
frontal face, side face and
left and right eye of each face
| ResNet50
---
| 92.9(SCER),
---
99.5(LCER)
| Rangesh et al. [2020]
---
1 (IR) |  | GZC
---
(7)
| Include day and night time,
---
with and without eye glasses
images , remove eye glasses
using GPcycleGANE
SqueezeNet |  | 72.4
---
| Yang et al. [2021]
---
1 (RGB) |  | GZC
---
(7)
| Face detection using docker
---
face, ERW-Net10 for facial
information from eyes
and mouth, HP-Net11 based on
kronecker product mechanism
for fusing unbalanced features
of the face and head pose
HP-ERW |  | 72.11 (DGW)
---
98.26
(LISA GAZE v1)
| Shah et al. [2022]
---
1 (RGB) |  | GZC
---
(7)
| YOLOV4 face detector
---
Inception-v3, robust feature
learning, improved
face detection
InceptionResNet-v2 |  | 91.0
---

  * 10

Eye region weighted encoding network;

  * 11

Head pose fusion network

###  5.2 Geometric model based methods

Geometric model based gaze estimation requires geometric model of an eye (see
Fig. 3) with eye features such as the center of the cornea, corneal reflection
(glint), pupil center (dark pupil or bright pupil), optical and visual axes of
the eye [Akinyelu & Blignaut, 2020]. Geometric models use subject-specific
parameters of geometric 3d eye models such as kappa angle. Because of
variations between different drivers, kappa angle eye tracker requires
calibration before using. Kappa angle is defined as the angle between the
visual axis and the eye optical axis [Kar & Corcoran, 2017, Akinyelu &
Blignaut, 2020], which is used to estimate the gaze direction.

This method is typically used in studies based on head-mounted driver gaze
estimation setup, where the driver eye information is captured from a closeup.
However, some studies such as Ji & Yang [2001, 2002], Vicente et al. [2015],
Vasli et al. [2016] also used this method in remote setup. Studies by Ji &
Yang [2001, 2002] used eye pose and head pose for monitoring the driver by
considering bright pupil effects. The pupil was tracked using Kalman filter
with features around the pupil in conjunction with a KNN classifier for head
pose estimation. Driver gaze was estimated using displacement between the
center of the pupil to the glint with the help of a linear regression model to
map nine gaze directions, thereby producing the final output as the nine gaze
zone classes.

Geometric model-based methods offer notable advantages, including higher
accuracy in gaze estimation and ability to provide precise outputs in terms of
point of gaze. These methods excel in controlled environments, which is why
they are preferred in lab settings (simulation-based studies), delivering
reliable results and maintaining low-latency responses. However, their
application is tempered by certain limitations. Geometric models may exhibit
limited robustness to individual variability, struggling to adapt seamlessly
to differences in eye anatomy and facial characteristics among users.
Challenges arise due to eye occlusion, where obscured visual cues hinder
accurate gaze estimation. Sensitivity to calibration errors can also impact
performance, emphasizing the importance of meticulous calibration procedures.
In real-world scenarios, the complexity of environments, varying lighting
conditions, and dynamic situations can pose challenges for geometric models,
potentially compromising their accuracy.

##  6 Uses and Application of Driver Gaze

Driver gaze estimation is essential from several aspects. One critical aspect
of driver gaze is understanding driver gaze behavior at different road
sections, which helps to build safer road infrastructure and safety systems
for drivers. Driver gaze is also used to build driver distraction detection
system, inattention detection system, and advanced driver assistance system.
This section is divided into two parts: first, we will discuss the uses of
driver gaze in understanding driver behavior, followed by applications of
driver gaze in real-world systems such as driver attentiveness detection, etc.

###  6.1 Uses of driver gaze in driver behavior understanding

Driver gaze behavior shows awareness of the driver to the surrounding traffic,
such as vehicles coming from different traffic streams, road infrastructure
such as traffic signs or traffic lights, road markings, etc., and roadside
infrastructures such as buildings, trees, advertising hoardings, billboards,
etc. Analyzing driver gaze behavior is crucial for evaluating attention to
traffic signs and optimizing compliance with rules. Intersection design, stop
sign, and traffic light placement impact gaze focus, with analysis aiding in
enhancing design effectiveness. Gaze behavior assessment at intersections
identifies potential conflict areas, helping in infrastructure improvements
for increased safety. Road curvature, influenced by infrastructure design,
affects driver navigation through curves. Gaze behavior analysis reveals
drivers’ adaptation to road geometries, offering insights for road design
enhancements. Roadside elements like billboards can divert driver attention.
Monitoring gaze behavior helps understand driver interaction with these
elements, assessing their impact on attention and safety. Driver gaze behavior
analysis can be used by insurance companies to assess driver behavior and
provide more accurate risk assessments, potentially leading to more
personalized and fair insurance premiums. Therefore, it is critical to check
if the drivers are aware of their surroundings before and while performing
different traffic maneuvers such as lane changing, merging/diverging in on-
ramp/off-ramps, and left/right turning at intersections. A detailed discussion
of each aspect is provided in the following sections.

####  6.1.1 Gaze behavior at intersections

Intersections are known for their complex nature because of different
participants’ behaviors and interactions [Shirazi & Morris, 2016].
Interactions at intersection are vehicle-to-vehicle (V2V) [Harding et al.,
2014, Liu et al., 2018, Ye et al., 2019], vehicle-to-pedestrian (V2P) [Anaya
et al., 2014, Liu et al., 2018, Sewalkar & Seitz, 2019], vehicle-to-
infrastructure (V2I) [Milanés et al., 2012, Liu et al., 2018], and pedestrian-
to-infrastructure (P2I) [Liu et al., 2018]. In the literature, interactions
involving vehicles include drivers. So the first three interactions are
important from the driver’s perspective and hence included in the present
review.

This section will discuss how driver gaze is influenced when the driver
approaches, maneuvers (left turning, right turning, and going straight), and
leaves the intersection. Literature on driver gaze behavior at intersections
is broadly divided into three categories based on driver age or experience
(novice, young experienced, old experienced), intersection types (signalized,
unsignalized), and surrounding traffic environments (traffic density,
familiarity), mentioned in Table 4. A few studies compared driver glance
behavior based on their age or experience while approaching or negotiating
through the intersection [Bao & Boyle, 2009, Dukic & Broberg, 2012, Scott et
al., 2013, Romoser et al., 2013, Savage et al., 2020, Chung et al., 2022]. The
discussion will be based on left-hand driving to maintain uniformity in the
paper.

In real driving scenarios, Bao & Boyle [2009] measured driver gaze scanning
behavior and found that middle-aged drivers (35-55 years old) have higher
scanning randomness (i.e., a greater entropy rate) than older drivers (65-80
years old). In another study, Dukic & Broberg [2012] found that older drivers
looked more at road lines and markings to position themselves in surrounding
traffic. Some other simulation-based studies examine the age and experience
effect on driver gaze. While selecting a safe gap at an unsignalized
intersection (USI), Scott et al. [2013] compared glance transition patterns of
three groups of drivers, including novice (mean age 20.57 years), young
experienced (23.79 years), and older experienced (66.43 years) on a right
turn. They divided the intersection approach period into scanning phase (first
10 seconds before finding negotiable gaps) and decision phases (first 5
seconds after finding negotiable gaps), the results being shown in Table 4. A
study by Romoser et al. [2013] examined four hypotheses to determine why older
drivers fail to scan effectively at intersections compared to young drivers.
The four hypotheses were difficulty with head movements, decrease in working
memory capacity, increased distractibility, and failure to recall specific
scanning patterns. None of the hypotheses were able to fully explain the above
reason. Still, the research does support the alternative theory that some of
the issues older drivers experience when looking at junctions are due to
unique attentional weaknesses in the older drivers’ ability. The effect of age
and guidance type (lead car and GPS) Savage et al. [2020] slightly reduced the
gaze scanned when the driver is close to the intersection. Chung et al. [2022]
compared the static (dwell time) and dynamic (gaze transitions) gaze of novice
and experienced drivers. Static analysis of novice drivers shows higher dwell
time in an area of interest (AOI) than experienced.

The general observation from these studies was that older drivers, compared to
younger drivers, scan fewer right and left areas of interest and focus more
straight ahead or in the intended direction of vehicles. This behavior may
explain the fact that older drivers are more involved in angle crashes at
intersections and ”failure to yield” when involved in ”seen but not seen”
crashes and accidents with other vehicles [Stutts et al., 2009, Misra et al.,
2023].

In addition to studies focusing on driver scanning behavior in younger and
older drivers, researchers have also focused on driver gaze behavior at SI and
USI. Li et al. [2019] examined the influence of intersection types on driver
scanning measures. When performing right-turning at SI, drivers were found to
give more attention to the forward and right areas than at USI. Two different
studies, Lemonnier et al. [2015, 2020] examined the impact of the three-factor
priority rule (yield, priority, and stop), expected traffic density (no
traffic, light, and heavy), and familiarity while approaching SI. The dwell
time in intersecting road AOI was found to be higher in yield than in priority
condition and smallest in stop sign condition. They also found that the
horizontal gaze eccentricity is inversely related to traffic density.
Horizontal eccentricity is defined as the absolute value of the horizontal
component of the gaze (and head) direction angle. Visual information related
to the decision-making task starts later when the driver is slightly familiar
with the environment.

Table 4: Driver gaze behavior at intersections. Reference | Study area |  | Study type
---
and setup
Gaze behavior | Key findings
| Bao & Boyle [2009]
---
| USI: left turn,
---
right turn, and
going straight
| Real driving
---
(Remote
setup gaze
estimation)
| Age related visual
---
difference in
proportion of time
glanced in each
gaze zone
| During intersection negotiations,
---
older drivers had a significantly
smaller proportion of visual
sampling to the left and right.
| Dukic & Broberg [2012]
---
| T-USI:right turn,
---
SI: right turn,
and left turn
| Real driving
---
(Eye Tracker)
| Fixation (each gaze
---
point was associated
with one location
and one object) of
older and youger
drivers
| Older driver follow same
---
behavioural pattern as
younger. Difference found in
gaze zone where older driver looked
more on markings whereas younger
driver looked at dynamic objects.
| Romoser et al. [2013]
---
| USI:Right turn,
---
left turn, and
going straight
| Simulator
---
(Eye Tracker)
| Older versus
---
younger drivers,
glance frequency
pattern in each
gaze zone
| Compared to older drivers, younger
---
drivers spent more time looking at
the central area while turning left and
less time going straight.
| Scott et al. [2013]
---
| USI:Right
---
turn (gap
acceptance)
| Simulator
---
(Eye Tracker)
| Gaze transitions
---
made by
different drivers
group
| Young experienced drivers, who
---
are at ‘lower risk’ of accident,
showed a
more even distribution of gaze
compared to the ‘at risk’ groups.
| Lemonnier et al. [2015]
---
| USI: approaching
---
intersection
(effect of traffic
density and road
sign)
| Simulator
---
(Eye Tracker)
| Gaze accumulation
---
and gaze transitions
| Visual attention to intersecting roads
---
changed with the priority rule and
influenced the visual attention associa
-ted with the vehicle control sub-tasks.
| Li et al. [2019]
---
| SI (Green phase)
---
and USI:left turn,
right turn, and
going straight
| Real driving
---
(Remote
setup gaze
estimation)
| Glance duration,
---
glance frequency
and gaze transition
probabilities
| Visual scanning performance was
---
similar between SI and USI in
through and right turning
maneuvers
| Lemonnier et al. [2020]
---
| USI: approaching
---
intersection
(Effect of traffic
density, road sign
and familiarity)
| Real driving
---
(Eye Tracker)
| Driver visual
---
attention, head and
gaze horizontal
eccentricity
| The effect of expected traffic density,
---
priority rule, and familiarity increases
as the distance of the driver to the
intersection decreases and temporal
pressure increases.
| Savage et al. [2020]
---
| USI: approaching
---
intersection
| Simulator
---
(Eye Tracker)
| Effect of guidance
---
type (lead vehicle
and GPS) and age
on eye, head and
eye plus head scan
| Older driver had lower eye and head
---
scan compared to younger
drivers.
| Chung et al. [2022]
---
| SI: approaching
---
and performing
maneuvers
| Simulator
---
(VR driving
simulator )
| Static (Dwell time)
---
and dynamic (Gaze
transitions) analysis
of driver gaze
| Novice driver showed longer dwell
---
time and longer fixation duration on
AOI. Gaze transition of novice
drivers between AOI occur at close
distance, while experienced driver
check surrounding traffic conditions
for vehicle driving.
| Lyu et al. [2022]
---
| USI: approaching
---
and performing
maneuvers
| Simulator
---
(Eye Tracker)
| Fixation duration,
---
number of fixation,
gaze transition,
blink duration,
heart beats rate
| Both intersection types and priority
---
rule made differences in drivers’
scanning behavior and mental
workloads
| Pike et al. [2023]
---
| USI: approaching
---
(Effect of road
signs and cross
walk treatments)
| Real driving
---
(Eye Tracker)
| Gaze,
---
gaze transition
| No significant effect of road cross-
---
walk signs, significantly higher gaze
value for treatment compared to no
treatment crosswalk
| Ringhand et al. [2022]
---
| USI: approaching
---
(Effect of surroun-
ding traffic,
intersection type
driving maneuver,
secondary task
involvement)
| Simulator
---
(Eye Tracker)
| Relative frequency,
---
of fixation
| Effect of adjacent traffic, secondary
---
task involvement and the intended
traffic changed significantly depen-
ding on the right of way of the
driver.
| Girgis et al. [2023]
---
| SI: Right turn
---
(Left turn in Indian
driving), safety of
vulnerable road
user (pedestrian
and cyclist) at turns
| Real driving
---
(Eye Tracker)
| Glance duration,
---
glance frequency
| Glances were more towards the
---
pedestrian, rather than cyclist.
Driver attention was more towards
the leftward (rightward Indian driving)
traffic at red light of the driver.

####  6.1.2 Over taking/lane changing gaze behavior

Overtaking or lane-changing occurs in the traffic stream when all vehicles do
not move at the design speed [Chandra & Shukla, 2012]. In this study, we
consider lane-changing gaze behavior for the overtaking of vehicles. A leading
vehicle moving at a slow speed hinders the following fast-moving vehicle,
provoking the following vehicles to overtake, and lane changing occurs. A
sequence of different glance patterns can be seen before the lane change
starts to know the possible threats from the surrounding traffic. In lane
changing gaze behavior analysis, two kinds of study have been done: driver
gaze behavior understanding during a lane change and predicting lane changes
based on driver gaze. A detailed discussion of driver gaze-based lane change
prediction system is given in Section 6.2.1.

In a simulation study, Salvucci & Liu [2002] found that drivers began to
exhibit notably different gaze behavior about three seconds before the lane
change (independent of the vehicle speed), with an increase in the frequency
of glances in the rearview mirror at the expense of glances in the direction
of their current lane. As soon as the driver decides to change lane, their
eyes typically move from salient guiding features of the present lane (such as
the tangent point or the lead car) to salient guiding features of the
destination lane. Additionally, drivers increase their gazes at surrounding
vehicles during lane changes to help with situation awareness and decision-
making. Another simulation-based study by Lavalliere et al. [2011] tested the
influence of age differences in glances to the blind spot and mirrors when
changing lanes. Compared to younger drivers, older drivers were found to show
reduced glance frequency while checking towards the left side mirror, rearview
mirror, and blind spot. This behavior of drivers may explain the observations
made by Stutts et al. [2009] that older drivers are more likely to be involved
in collisions when changing lane.

####  6.1.3 Driver gaze behavior on curve, on-ramp and off-ramp sections

This section will cover driver gaze behavior while going through curves, on-
ramps, and off-ramps. A study by Lehtonen et al. [2014] examined the look
ahead fixation driving experience behavior when approaching and negotiating
the curve on a rural road. They found that experienced drivers spent more time
on look-ahead fixation than the road ahead while moving through the curve. To
obtain accurate foveal information from the rest of the curve, drivers need to
make an eccentric fixation towards the road further up, disengaging the gaze
from the visual guidance of online control of steering; these fixations have
been called look-ahead fixations. Compared to the entry phase, the driver
look-ahead fixation behavior is more on the approach phase because the turning
steering driver needs a higher visual demand at the curve [Tsimhoni & Green,
2001, Lehtonen et al., 2014]. Zwahlen et al. [2003] investigated the effect of
ground-mounted diagrammatic guide signs on driver eye scanning before entrance
of freeway ramps. The diagrammatic guide sign is a type of sign that indicates
the destination using large map-like figures of the road layout. The findings
of this study revealed that ground-mounted signs on multi-lane arterials do
not excessively distract drivers or influence eye-scanning behavior
detrimentally. In situations where placing overhead span type signs can not be
economically feasible although placing these signs is highly desirable for
driver guidance, these diagrammatic signs give unfamiliar drivers more
navigational information in advance (by identifying the correct lane to access
the desired entrance ramp).

In addition to real-world experimental studies, simulation studies have also
been done to understand driver gaze behavior in on-ramps and off-ramps.
Mecheri et al. [2022] checked how the presence of pavement shoulder influences
driver gaze in the right bend curve on a two-lane rural highway. The results
showed that driver gaze shifted towards the inside of the curve, followed by
the steering trajectory, irrespective of the width of the shoulder. They
suggested that the delineator on the curve is more useful to bring driver gaze
and vehicle back to the lane. Further, Zahabi et al. [2017] analyzed the
effect of driver age, specific service sign content and format, and
familiarity with the road sign on the performance and attention level of the
driver when exiting the freeways, i.e., on-ramps and off-ramps. Drivers were
found to identify six-panel signs more accurately than nine-panels and were
found to be more accurate when familiar with the road sign.

####  6.1.4 Influence of roadside advertising structure on driver gaze
behavior

A recurring finding in the literature is that there appears to be a link
between crashes [Sisiopiku et al., 2015, Wallace, 2003] and the presence of
roadside advertising. In literature, different studies have investigated the
effect of different advertisement characteristics on driver behavior. Some of
the attributes of the advertisement are its nature, placement, content, road
and traffic characteristics, type of area, and driver characteristics
[Anciaes, 2022].

Compared to traditional static road signs, electronic roadside advertising has
been usually found to have a more significant influence on driver attention,
which causes higher safety risk for the general public [Beijer et al., 2004,
Roberts et al., 2013, Dukic et al., 2013, Herrstedt et al., 2017, Oviedo-
Trespalacios et al., 2019]. Further, the brightness levels and illumination of
roadside advertising billboards also affect visual driver behavior. Driver
gaze is attracted when the luminance changes in the visual field [Roberts et
al., 2013, Oviedo-Trespalacios et al., 2019]. In a simulated study, Herrstedt
et al. [2017] examined the impact of LED advertising signs on driver gaze
behavior. They found that average glance duration was higher for LED-based
signs compared to other objects (e.g., non-Led signs, hoardings, rearview
mirrors, speedometer, etc.) In comparison to control parts of the road with no
billboards, drivers on the section of the road with billboards drove at lower
mean speeds, with more speed variability, lane position variability, time
spent at high-risk headway, and more visual fixations. The least detrimental
effects on driving outcomes were caused by billboards with simple (versus
complex) content presented for a longer dwell time (60 seconds versus 40 or 20
seconds). Regardless of dwell time, the billboards with complex content had
similar adverse effects on driving. Vickers [2017] observed that the drivers
confronting potential risks glanced more at street-level advertisements than
the ones raised three meters from the street light. When the vehicle speed is
low, drivers have been found to pay more attention to the electronic
billboards and other advertising at the junction than other road locations
[Abbas & Haghighi, 2020]. Further, the frequency of glances has been found to
be higher in retail areas [Mollu et al., 2018], while longer duration of
glances has been found in rural areas [Costa et al., 2019].

###  6.2 Application of driver gaze

Driver gaze is useful in different applications, such as building driver
maneuver recognition and prediction systems, inattention and distraction
detection systems, advanced driving assistance systems, etc. In the following
section, each of them will be discussed separately.

####  6.2.1 Driver maneuver recognition and prediction

Recognizing and predicting driver intent before the driver’s action is
important for driver monitoring system. Knowing the driver’s intent and
correlating it to predict the driver’s action is useful to warn the driver for
their safety. Driver gaze plays an important role in knowing the driver’s
intent.

Research on lane change prediction indicates that the driver’s glances can be
used as an early indicator before a lane change [Salvucci & Liu, 2002, Doshi &
Trivedi, 2009, Pech et al., 2014, Martin et al., 2018]. The sequence of lane
change on a highway has been defined by considering three parameters:
reference point, start time, and end time. The lane marking has been defined
as the reference point, while the start point is considered to be the time
just before touching the lane marking. Similarly, end time is taken as the
time when the tyre just crosses the reference lane marking. Since three
seconds is usually taken as the critical decision-making time for a lane
change [Mourant & Donohue, 1974], several studies considered 3-5 seconds
duration before the start time to analyze the driver gaze behavior [Salvucci &
Liu, 2002, Doshi & Trivedi, 2009, Martin et al., 2018, Long et al., 2022]. A
lane change prediction system helps drivers safely change lanes during
overtaking. Lane change prediction based on eye movements in a simulation-
based study proposed a 4DDTW (four-dimensional dynamic time warping) KNN-based
lane-changing prediction approach [Long et al., 2022]. They used a sliding-
space time algorithm to extract the scanpath of the left and right eye in time
series data. 4DDTW was used to find the similarity between the scanpaths and
then applied KNN on each sample to classify left lane change, right lane
change, and lane keeping. The classification accuracy of the system using KNN
classifier was 86.5 %. In another study, Martin et al. [2018] developed a
machine vision-based predictor for lane change behavior. They categorized
dynamic lane changes into three classes: left lane change, right lane change,
and lane keeping. The analysis focused on a 10-second time window, with 5
seconds before and after crossing the lane marking. Multiple scan paths were
examined for each class in each time window, extracting features like minimum,
maximum, and average glance duration, frequency, and gaze accumulation in
specific zones. The lane changes gaze predictor was modelled using a
multivariate normal distribution (MVN), achieving approximately 75.0 %
accuracy in predicting right and left lane changes.

Jain et al. [2015b] developed an Autoregressive input-output hidden Markov
model (AIO-HMM) to predict vehicle maneuvers. This model incorporates a
diverse range of input features, encompassing both inside and outside vehicle
information. Inside vehicle information, such as driver head movements
(facilitated by face detection, tracking, and feature extraction) were
combined with outside vehicle information, including road conditions, vehicle
dynamics, and GPS coordinates. The road-facing camera was used to extract
binary lane features (left lane, right lane) while integrating GPS coordinates
with street maps, which enables the identification of road artifacts within a
15-meter range. Notably, the model encodes the vehicle’s average, minimum, and
maximum speed over the previous five seconds as additional features. Finally,
the model outputs probabilities for four distinct maneuvers (left/right lane
change, left/right turn) and driving straight. Rahman et al. [2020] used a
similar AIO-HMM model with different input features to forecast vehicle
maneuvers at intersections. The study integrated five key features, namely 3D
driver gaze position, driver head position, traffic light state, steering
wheel position, and speed, into a unified feature vector for maneuver
prediction. The model was able to anticipate maneuvers (left turn, right turn,
going straight) 3.97 seconds before the actual maneuver occurred, achieving an
average precision and recall of 86.4% and 87.7% respectively. In a simulation
study Wu et al. [2019] proposed a driver intention prediction system with the
driver seated in a semi-autonomous (ego vehicle) vehicle. They proposed a
probabilistic Dynamic Time Warping-Hidden Markov Model (pDTW-HMM) to predict
the intention of the driver 3 seconds before the execution using fixation and
saccades.

####  6.2.2 Driver inattention and distraction detection system

Several studies show driver distraction and inattention are the leading cause
of vehicle crashes and incidents.  In a review on driver distraction and
inattention, Regan et al. [2011] defined driver inattention as “insufficient
or no attention to activities critical for safe driving” and driver diverted
attention (which is a synonym of driver distraction) is an alternate form of
driver inattention. For a more in-depth understanding, the reader can refer to
the studies [Regan et al., 2011, Koay et al., 2022, Kotseruba & Tsotsos, 2022]
related to driver distraction and inattention. In this section, we will first
discuss inattention-based studies followed by driver distraction-based
studies.

Driver inattention is generally used in literature than attention due to its
association with driver safety risk. In two different studies, Fletcher &
Zelinsky [2009] and Hu et al. [2021] developed a driver inattention detection
system using eye gaze and road events. Multiple road event inattention
detection systems were built, and one of them was road center inattention
detection system. A warning was given whenever the driver can be seen to be
diverted from the forward direction for a minimum specific time period.

One of the important measures of driver distraction is the measure of eyes off
the road [Vicente et al., 2015, Badgujar & Selmokar, 2023]. Eyes off the road
indicates that the driver is looking somewhere else rather than looking at the
road. This situation delays the reaction time of the driver and causes a
higher risk of accidents [Lamble et al., 1999, Bärgman et al., 2015, Dingus et
al., 2016]. Ahlstrom et al. [2013] developed an attention-based driver warning
system called AttenD that can help to mitigate driver distraction. This study
assumes that the driver attention is directed towards the gaze object. A 3D
car model was divided into several zones, such as mirrors, speedometer, etc.
The algorithm works according to the principle that frequent glances away from
Field Relevant for Driving (FRD) can be used as a sign of distraction. FRD is
defined as the intersection between a viewing cone of 90 degrees and vehicle
windows for gaze tracking. The authors considered 2 seconds as the buffer
time, which will depleted when the driver gazes away from the FRD. When the
buffer time reaches zero, a warning is given to the driver.

Apart from driver distraction systems, studies have also been performed to
measure the cognitive load of drivers. Prabhakar et al. [2020a] developed a
system to measure driver’s cognitive load in both simulated and real driving
scenarios. They classified the cognitive state into two classes: No Task
(driving without secondary task) and Task (driving with secondary task). Two
physiological parameters, gaze and pupil based metrics captured by head
mounted eye tracker were used as input to the machine learning model. The
input gaze parameters were saccade rate, fixation rate, and median velocity of
saccadic intrusions (SI), while the pupil parameters were the L1 norm of the
spectrum (L1NS), standard deviation of pupil (STDP), and low pass filter of
pupil (LPF) from pupil dilation. Two binary classifiers, support vector
classifier (SVC) and feed-forward neural network (NN) models were used to
define the cognitive state of the drivers. This achieved an accuracy of 75.0 %
using feed-forward NN model. Using the cognitive state of the driver and
detecting eyes-off-road distraction, they have also developed a multimodal
driver alert system.

####  6.2.3 Building advanced driver assistance system

An advanced driver assistance system (ADAS) is an automated system that
assists the driver when the driver fails or misses some events in nearby
traffic [Fletcher et al., 2005, Yang et al., 2021]. These systems use sensors,
cameras, radar, and other advanced technologies to collect and analyze data in
real-time, providing assistance to the driver in various aspects of driving.
There are some key features and ways in which ADAS helps drivers. These key
features are collision avoidance system, lane-keeping assistance system,
adaptive cruise control, blind spot detection, parking assistance, traffic
sign recognition, driver fatigue monitoring, etc. In this study, we have
discussed the studies that included the driver gaze in building ADAS.

Since driver gaze information is an important input variable to measure driver
attentiveness towards the surrounding traffic environment, it plays an
important role in building an advanced driver assistance system. In real-world
driving scenario, Fletcher et al. [2005] made a driver assistance system using
driver gaze direction and speed limit sign. When a traffic sign is recognized,
the system checks two things: (a) whether the driver looking at the sign and
(b) if the speed and acceleration of the vehicle are compliant with the sign.
If the vehicle state is not compliant and the driver has not seen the sign, a
high-priority warning is given to the driver. A study by Prabhakar et al.
[2020b] developed an intelligent interactive head-up (HUD) display on the
windscreen that does not require taking eyes off the road while doing
secondary tasks such as watching navigation map, operating music player, or
operating vent control. The system consists of a screen in which content is
displayed on the windscreen of the car, and the content on the screen either
changes by eye gaze or finger movement of drivers.

ADAS technologies collectively contribute to improved safety, reduced
accidents, and enhanced driving comfort. However, it is important for drivers
to remain attentive and not entirely rely on these systems, as they are
designed to assist rather than replacing human judgment and control.
Additionally, the effectiveness of ADAS may vary depending on factors such as
system quality, environmental conditions, and proper maintenance.

##  7 General Discussion and Future Scope

Over the past decade, significant progress has been made towards driver gaze
estimation by gaze zone classification and tracking. Driver gaze data
collection methods and equipment have also evolved over the years. Gaze
estimation has evolved from coarse gaze zone classification to finer gaze zone
classification, and gaze classification models have been developed using
traditional machine learning to deep learning algorithms. Detected driver gaze
is used to analyze the awareness and attentiveness of the drivers to the
surroundings, build advanced driving safety systems, and develop safer driving
guidelines for the drivers. However, there are still research gaps existing in
the current studies that can be handled in the future, thereby leading to
further evolution of this domain and helping to build a safer transportation
system.

###  7.1 Challenges in driver gaze estimation and possible solutions

Driver gaze estimation is a challenging task due to the complex and dynamic
nature of driving. Some of these challenges include varying lighting
conditions, occlusions and eyeglasses, dynamic road and traffic environments,
in-car distraction, real-time processing, safety concerns, user comfort and
acceptance, etc. There are some representative methods that aim to address
these challenges in driver gaze estimation. In this subsection, we discuss the
limitations and future scope of driver gaze estimation in three broad
categories: (a) inclusion of advanced sensing technologies, (b) generating
datasets that can better represent real-world conditions, and (c) advancements
of models for driver gaze estimation. Each of these is discussed in detail
next.

####  7.1.1 Sensors

The datasets mentioned in Table 1 typically captured the driver face using a
single camera. However, in case of large head movement, one side of the face
maybe be occluded by the other, thereby creating problems in detecting the
iris or pupil position inside the eye. This in turn affects the estimation of
the point of gaze of the driver. These limitations can be overcome using
multiple cameras and capturing the driver face at different angles or
positions [Yang et al., 2019, Wang et al., 2022a]. Varying light conditions,
either due to daytime or due to weather conditions, which impact the gaze
estimation accuracies, can be resolved by using NIR or depth camera [Naqvi et
al., 2018].

Also, driver gaze can be estimated in terms of object of interest e.g.
surrounding vehicles, pedestrians etc. This approach is different from the
traditional approach of estimating the gaze zone class or the point of gaze.
To estimate the object of interest of the driver, the scene camera of eye
tracker or dashboard camera has been used. However, it can be extended further
by including information from cameras placed at the top of the vehicles or
other sensors such as LiDAR (Light Detection and Ranging) or radar. This can
help in accurate estimation of objects of interest of drivers and correlate
them with driver actions. Also, monitoring additional cues such as hand
movements, facial expressions, or physiological signals can help to
differentiate between intentional gaze shifts from distractions [Wang et al.,
2010, Koay et al., 2022].

####  7.1.2 Dataset generation

As discussed in Section 4, benchmark open-source driver gaze data, discussed
in Table 1, have been typically collected in parked or moving vehicles.
Mostly, the parked vehicle participants have been college or university
students [Ribeiro & Costa, 2019, Dua et al., 2020, Ghosh et al., 2021], who
can have limited knowledge of how head movement or eye movements occur during
real driving scenarios. On the other hand, datasets collected in real driving
scenarios have been limited in terms of the number of participants (drivers)
[Nuevo et al., 2010, Diaz-Chito et al., 2016, Jain et al., 2015a, Schwarz et
al., 2017, Palazzi et al., 2018]. A limited number of participants in real-
world driving datasets therefore lack diversification in terms of experience,
age, lighting conditions, and traffic conditions. Therefore, large scale open
source driver gaze datasets can be created encompassing different
environmental and lighting conditions, which can help in the development of a
robust and generalized system of gaze estimation based on deep learning
techniques.

Further, there are inherent problems in classifying gaze zones inside a parked
vehicle compared to real-world driving scenarios. Driver gaze collection and
ground truth generation can be easier inside the parked vehicle and is also
safer for the driver. However, this method may affect the psychological
behavior of the drivers because, in this method, a second person is providing
instructions [Costa et al., 2019, Ghosh et al., 2021] to the driver to look
towards a particular gaze zone or the driver has been informed earlier to look
at predefined gaze zone. Chuang et al. [2014] revealed that the gaze zone
classifier created using parked vehicles could not successfully generalize to
a moving vehicle. On the other hand, while doing driver gaze data collection
in moving vehicles, drivers gaze naturally in and around the dashboard and
windshield area, but generating the ground truth labels of gaze zone is
difficult in this situation. Since, in this method, drivers can not look
towards the gaze zone by instructions due to accident risk, unsupervised
techniques can be used for estimating the gaze zone class [Chuang et al.,
2014].

Apart from data collection strategies, there are also limitations in ground
truth generation methodology for gaze datasets. Datasets collected in the
parked vehicle have been typically annotated by two or more human annotators
and cross-verified. However, these datasets do not typically mention any data
statistics on how much the difference or error was observed during ground
truth generation of the different annotators and how to handle such cases.
Also, gaze data annotations by humans cannot be assumed due to 100% accurate,
which can impact gaze classification algorithms too. Ground truth generation
by speak2label [Ghosh et al., 2021] is an automatic way of ground truth
generation, but it still has the limitations of data generation inside parked
vehicles, as discussed above. Ground truth labels of existing driver gaze
datasets are typically given based on the zone in which the driver is looking.
These labels are the predefined regions such as rearview mirror, forward, left
wing mirror, right wing mirror, center stack, speedometer, etc. These datasets
do not clearly mention the surrounding data collection conditions except
DGAZE. During recording the DGAZE datasets, moving traffic was shown on the
screen, and the gaze zones were based on the vehicle entities [Dua et al.,
2020]. Therefore, in future, gaze datasets can be classified not only based on
the gaze zone but can also denote the vehicle entities observed, which can
help in understanding driver gaze behavior better.

####  7.1.3 Models for gaze estimation

Driver gaze zone estimation using conventional appearance-based method and
appearance based method with deep learning has certain limitations. In
conventional appearance based method, the decision made by the traditional ML
model completely depends on the individual sub-model (face, pupil detection,
landmark estimation, feature extraction), which affects the accuracy of the
gaze estimation. The hand-crafted features designed from facial landmarks on
the eyes are not completely robust to variations across different drivers,
cars, seat positions, etc. On the other hand, gaze estimation using appearance
based method with deep learning, uses pre-trained CNN model which does not
require hand-crafted features because of inherent feature extraction quality.
However, these models require large-scale datasets and more computational
power for their training compared to traditional ML models. The availability
of large-scale open-source datasets with different illumination conditions and
more subjects can help build a robust classifier. On the other hand, geometric
model-based methods suffer in remote setup-based driver gaze estimation due to
varying light conditions and the difficulty of capturing geometric eye
parameters in non-constrained environments. However, in head-mounted setup
based gaze estimation, this method can be used to give more accurate results
compared to appearance-based methods. However, in both these methods,
attention can be given to developing calibration-free gaze estimation
techniques, which can help in large-scale implementation purposes [Tonsen et
al., 2020].

Gaze estimation models can also focus on estimating the exact point of gaze
instead of gaze zones in the windscreen, right and left-wing mirror only. This
can help to understand driver attention on different traffic entities, such as
vehicles, pedestrians, etc., and other surrounding objects, such as
billboards, traffic signs, etc. This can also help to understand driver
behavior during complex traffic manoeuvres such as intersections, on-ramps,
off-ramps, etc.

Finally, driver gaze estimation models can be further advanced based on the
techniques that have been successful in gaze estimation in other platforms,
such as logistics and manufacturing, gaming, virtual reality, and human-
computer interaction (HCI) systems. For example, gaze estimation based on
attention mechanisms has been an active area of research in the field of
computer vision and human-computer interaction. Recently, several studies
based on attention mechanisms developed human gaze estimation systems in
different fields [Murthy & Biswas, 2021, Fang et al., 2021, Senarath et al.,
2022]. These studies have mostly focused on developing depth-based dual
attention model for customer gaze estimation in retail stores [Senarath et
al., 2022] or human gaze estimation in the wild [Fang et al., 2021]. Similar
attention-based mechanism can be developed in future for 3D driver gaze
estimation in wild.

###  7.2 Future scope in uses and applications of driver gaze

There has been a significant number of studies on driver gaze behavior that
have focused on maneuvering through intersections, lane changing during
overpassing, etc. However, most of these studies are either based on comparing
different driver groups, such as younger versus older, or based on experience,
such as novice versus experienced. A few studies, based on real driving
[Lemonnier et al., 2020] and simulator [Lemonnier et al., 2015], evaluated the
impact of traffic density and the familiarity of the route at intersections on
the driver gaze. Further, gaze behavior studies on mixed traffic conditions
and unstructured driving environments are limited; only one study in real
driving compared the gaze pattern of drivers on SI and USI [Li et al., 2019].
Hence, more research is needed on driver gaze behavior in mixed-traffic
environments, which can include the effect of pedestrians, traffic density,
intersection type along with focusing on unstructured driving environments
too.

Eye tracker based driver gaze behavior studies use different terminologies
such as fixation, dwell time, and saccades; on the other hand, studies based
on the remote setup use glance duration, glance frequency, and gaze
transitions to define driver gaze behavior. They can explain the gaze behavior
based on statistics such as minimum, and maximum glance duration, glance
frequency, number of fixations, dwell time in each gaze zone, glance or gaze
transitions, saccades between the gaze zones, etc. However, these studies do
not consider the effect of the shape and size of different traffic entities on
driver gaze. For example, is the influence of small vehicles (cycles,
motorcycles) on driver gaze same or different than the large vehicles such as
trucks, buses, etc.? More in-depth analysis on understanding such gaze
behavior will be helpful in determining driver attentiveness [Yamaguchi et
al., 2022] and building a safer driving environment.

Driver gaze research can also be incorporated and extended with other
physiological sensors [Wang et al., 2010] such as heartbeat (fitbit) [Lu et
al., 2022, Othman et al., 2022], EEG (Electroencephalography) [Tuncer et al.,
2021, Arefnezhad et al., 2022] to understand driver gaze behavior
comprehensively. Monitoring the pulse rate, breathing rate, and other
physiological data might reveal the psychological state of the driver [Abou
Elassad et al., 2020, Zhou et al., 2023]. Since connected and autonomous
vehicles are the emerging future of intelligent transportation systems, more
attention is required to know how the driver gaze is influenced by surrounding
traffic, including autonomous or connected vehicles. Limited research has been
done on the use of driver gaze in building driver assistance systems. So there
is significant scope for researchers and industry experts to build a robust
and generalized driver assistance system that can help drivers perform safe
maneuvers at intersections, overpassing, etc. Driver gaze behavior
understanding can also help in taking steering control from manual to semi-
automatic when the driver is not fully attentive to the surrounding traffic
environment.

##  8 Conclusion

Driver gaze plays an important role in different driving gaze-based
applications, such as driver inattention detection, visual distraction
detection, and taking over automatic steering control. This study thoroughly
summarizes different terminologies used in driver gaze estimation and behavior
understanding based on head-mounted and remote setup-based techniques and
compiles the existing benchmark gaze estimation datasets. We have also
reviewed different algorithms and models used in driver gaze estimation based
on appearance and geometric model approaches. Compared to traditional machine
learning, deep learning-based approaches are more efficient in detecting
driver gaze in different lighting conditions and large head movements. This
study also explores uses of driver gaze in different driver gaze behavior
understanding such as negotiating at intersections, lane changing during
overpassing, moving on the curve, on-ramp and off-ramp, influence of roadside
advertising infrastructure and driver gaze-based applications such as driver
maneuver recognition and prediction, inattention and distraction detection,
and ADAS. Finally, we have highlighted the limitations of the existing studies
and the future scope in this domain, which can help the researchers and the
developers to build a more robust and generalized driver gaze estimation and
gaze-based driving assistance system.

## CRediT authorship contribution statement

Pavan Kumar Sharma: Conceptualization, Formal analysis, Investigation,
Methodology, Writing-Original Draft. Pranamesh Chakraborty: Conceptualization,
Funding acquisition, Investigation, Methodology, Supervision, Writing-Review &
Editing.

## Declaration of competing interest

The authors declare that they have no known competing financial interests or
personal relationships that could have appeared to influence the work reported
in this paper.

## Data availability

No data was used for the research described in the article.

## Acknowledgement

Our research results are based upon work supported by the Initiation Grant
scheme of Indian Institute of Technology Kanpur (IITK/CE/2019378). Any
opinions, findings, and conclusions or recommendations expressed in this
material are those of the author(s) and do not necessarily reflect the views
of the IITK.

## References

  * Abbas & Haghighi [2020] Abbas, S., & Haghighi, F. (2020).  Driver distraction by digital billboards. structural equation modelling based on naturalistic driving study data: a case study in iran.  J. Saf. Res, 72, 1–8.
  * Abou Elassad et al. [2020] Abou Elassad, Z. E., Mousannif, H., Al Moatassime, H., & Karkouch, A. (2020).  The application of machine learning techniques for driving behavior analysis: A conceptual framework and a systematic literature review.  Engineering Applications of Artificial Intelligence, 87, 103312.
  * Ahlstrom et al. [2013] Ahlstrom, C., Kircher, K., & Kircher, A. (2013).  A gaze-based driver distraction warning system and its effect on visual behavior.  IEEE Transactions on Intelligent Transportation Systems, 14, 965–973.
  * Akinyelu & Blignaut [2020] Akinyelu, A. A., & Blignaut, P. (2020).  Convolutional neural network-based methods for eye gaze estimation: A survey.  IEEE Access, 8, 142581–142605.
  * Anaya et al. [2014] Anaya, J. J., Merdrignac, P., Shagdar, O., Nashashibi, F., & Naranjo, J. E. (2014).  Vehicle to pedestrian communications for protection of vulnerable road users.  In 2014 IEEE Intelligent Vehicles Symposium Proceedings (pp. 1037–1042).  IEEE.
  * Anciaes [2022] Anciaes, P. (2022).  Effects of the roadside visual environment on driver wellbeing and behaviour–a systematic review.  Transport Reviews, (pp. 1–28).
  * Arefnezhad et al. [2022] Arefnezhad, S., Hamet, J., Eichberger, A., Frühwirth, M., Ischebeck, A., Koglbauer, I. V., Moser, M., & Yousefi, A. (2022).  Driver drowsiness estimation using eeg signals with a dynamical encoder–decoder modeling framework.  Scientific reports, 12, 1–18.
  * Badgujar & Selmokar [2023] Badgujar, P., & Selmokar, P. (2023).  Driver gaze tracking and eyes off the road detection.  Materials Today: Proceedings, 72, 1863–1868.
  * Bao & Boyle [2009] Bao, S., & Boyle, L. N. (2009).  Age-related differences in visual scanning at median-divided highway intersections in rural areas.  Accident Analysis & Prevention, 41, 146–152.
  * Bärgman et al. [2015] Bärgman, J., Lisovskaja, V., Victor, T., Flannagan, C., & Dozza, M. (2015).  How does glance behavior influence crash and injury risk? a ‘what-if’counterfactual simulation using crashes and near-crashes from shrp2.  Transportation Research Part F: Traffic Psychology and Behaviour, 35, 152–169.
  * Beijer et al. [2004] Beijer, D., Smiley, A., & Eizenman, M. (2004).  Observed driver glance behavior at roadside advertising signs.  Transportation Research Record, 1899, 96–103.
  * Birrell & Fowkes [2014] Birrell, S. A., & Fowkes, M. (2014).  Glance behaviours when using an in-vehicle smart driving aid: A real-world, on-road driving study.  Transportation research part F: traffic psychology and behaviour, 22, 113–125.
  * Chandra & Shukla [2012] Chandra, S., & Shukla, S. (2012).  Overtaking behavior on divided highways under mixed traffic conditions.  Procedia-Social and Behavioral Sciences, 43, 313–322.
  * Choi et al. [2016] Choi, I.-H., Hong, S. K., & Kim, Y.-G. (2016).  Real-time categorization of driver’s gaze zone using the deep learning techniques.  In 2016 International conference on big data and smart computing (BigComp) (pp. 143–148).  IEEE.
  * Chuang et al. [2014] Chuang, M.-C., Bala, R., Bernal, E. A., Paul, P., & Burry, A. (2014).  Estimating gaze direction of vehicle drivers using a smartphone camera.  In Proceedings of the IEEE conference on computer vision and pattern recognition workshops (pp. 165–170).
  * Chung et al. [2022] Chung, J., Lee, H., Moon, H., & Lee, E. (2022).  The static and dynamic analyses of drivers’ gaze movement using vr driving simulator.  Applied Sciences, 12, 2362.
  * Costa et al. [2019] Costa, M., Bonetti, L., Vignali, V., Bichicchi, A., Lantieri, C., & Simone, A. (2019).  Driver’s visual attention to different categories of roadside advertising signs.  Applied ergonomics, 78, 127–136.
  * Deng et al. [2009] Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., & Fei-Fei, L. (2009).  Imagenet: A large-scale hierarchical image database.  In 2009 IEEE conference on computer vision and pattern recognition (pp. 248–255).  Ieee.
  * Diaz-Chito et al. [2016] Diaz-Chito, K., Hernández-Sabaté, A., & López, A. M. (2016).  A reduced feature set for driver head pose estimation.  Applied Soft Computing, 45, 98–107.
  * Dingus et al. [2016] Dingus, T. A., Guo, F., Lee, S., Antin, J. F., Perez, M., Buchanan-King, M., & Hankey, J. (2016).  Driver crash risk factors and prevalence evaluation using naturalistic driving data.  Proceedings of the National Academy of Sciences, 113, 2636–2641.
  * Doshi & Trivedi [2009] Doshi, A., & Trivedi, M. M. (2009).  On the roles of eye gaze and head dynamics in predicting driver’s intent to change lanes.  IEEE Transactions on Intelligent Transportation Systems, 10, 453–462.
  * Dua et al. [2020] Dua, I., John, T. A., Gupta, R., & Jawahar, C. (2020).  Dgaze: Driver gaze mapping on road.  In 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) (pp. 5946–5953).  IEEE.
  * Dukic et al. [2013] Dukic, T., Ahlstrom, C., Patten, C., Kettwich, C., & Kircher, K. (2013).  Effects of electronic billboards on driver distraction.  Traffic injury prevention, 14, 469–476.
  * Dukic & Broberg [2012] Dukic, T., & Broberg, T. (2012).  Older drivers’ visual search behaviour at intersections.  Transportation research part F: traffic psychology and behaviour, 15, 462–470.
  * Fang et al. [2021] Fang, Y., Tang, J., Shen, W., Shen, W., Gu, X., Song, L., & Zhai, G. (2021).  Dual attention guided gaze target detection in the wild.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (pp. 11390–11399).
  * Fletcher et al. [2005] Fletcher, L., Loy, G., Barnes, N., & Zelinsky, A. (2005).  Correlating driver gaze with the road scene for driver assistance systems.  Robotics and Autonomous Systems, 52, 71–84.
  * Fletcher & Zelinsky [2009] Fletcher, L., & Zelinsky, A. (2009).  Driver inattention detection based on eye gaze—road event correlation.  The international journal of robotics research, 28, 774–801.
  * Fridman et al. [2016a] Fridman, L., Langhans, P., Lee, J., & Reimer, B. (2016a).  Driver gaze region estimation without use of eye movement.  IEEE Intelligent Systems, 31, 49–56.
  * Fridman et al. [2016b] Fridman, L., Lee, J., Reimer, B., & Victor, T. (2016b).  ‘owl’and ‘lizard’: Patterns of head pose and eye pose in driver gaze classification.  IET Computer Vision, 10, 308–314.
  * Fridman et al. [2017] Fridman, L., Toyoda, H., Seaman, S., Seppelt, B., Angell, L., Lee, J., Mehler, B., & Reimer, B. (2017).  What can be predicted from six seconds of driver glances?  In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems (pp. 2805–2813).
  * Fusek [2018] Fusek, R. (2018).  Pupil localization using geodesic distance.  Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), 11241 LNCS, 433–444.
  * Ghosh et al. [2021] Ghosh, S., Dhall, A., Sharma, G., Gupta, S., & Sebe, N. (2021).  Speak2label: Using domain knowledge for creating a large scale driver gaze zone estimation dataset.  In Proceedings of the IEEE/CVF International Conference on Computer Vision (pp. 2896–2905).
  * Girgis et al. [2023] Girgis, J., Powell, M., Donmez, B., Pratt, J., & Hess, P. (2023).  How do drivers allocate visual attention to vulnerable road users when turning at urban intersections?  Transportation research interdisciplinary perspectives, 19, 100822.
  * Guo et al. [2019] Guo, T., Liu, Y., Zhang, H., Liu, X., Kwak, Y., In Yoo, B., Han, J.-J., & Choi, C. (2019).  A generalized and robust method towards practical gaze estimation on smart phone.  In Proceedings of the IEEE/CVF International Conference on Computer Vision Workshops (pp. 0–0).
  * Hansen & Ji [2009] Hansen, D. W., & Ji, Q. (2009).  In the eye of the beholder: A survey of models for eyes and gaze.  IEEE transactions on pattern analysis and machine intelligence, 32, 478–500.
  * Harding et al. [2014] Harding, J., Powell, G., Yoon, R., Fikentscher, J., Doyle, C., Sade, D., Lukuc, M., Simons, J., Wang, J. et al. (2014).  Vehicle-to-vehicle communications: readiness of V2V technology for application..  Technical Report United States. National Highway Traffic Safety Administration.
  * Harezlak & Kasprowski [2018] Harezlak, K., & Kasprowski, P. (2018).  Application of eye tracking in medicine: A survey, research issues and challenges.  Computerized Medical Imaging and Graphics, 65, 176–190.
  * Heger [1998] Heger, R. (1998).  Driving behavior and driver mental workload as criteria of highway geometric design quality.  Transportation research circular, (pp. 43–1).
  * Herrstedt et al. [2017] Herrstedt, L., Greibe, P., Andersson, P., & la Cour Lund, B. (2017).  Do led-advertising signs affect driver attention.  In 5th International Driver Distraction and Innatention (DDI) Conference, Paris, France.
  * Holmqvist et al. [2022] Holmqvist, K., Örbom, S. L., Hooge, I. T., Niehorster, D. C., Alexander, R. G., Andersson, R., Benjamins, J. S., Blignaut, P., Brouwer, A.-M., Chuang, L. L. et al. (2022).  Eye tracking: empirical foundations for a minimal reporting guideline.  Behavior research methods, (pp. 1–53).
  * Hu et al. [2021] Hu, Z., Lv, C., Hang, P., Huang, C., & Xing, Y. (2021).  Data-driven estimation of driver attention using calibration-free eye gaze and scene features.  IEEE Transactions on Industrial Electronics, 69, 1800–1808.
  * Imaoka et al. [2020] Imaoka, Y., Flury, A., & De Bruin, E. D. (2020).  Assessing saccadic eye movements with head-mounted display virtual reality technology.  Frontiers in Psychiatry, 11, 572938.
  * Jain et al. [2015a] Jain, A., Koppula, H. S., Raghavan, B., Soh, S., & Saxena, A. (2015a).  Car that knows before you do: Anticipating maneuvers via learning temporal driving models.  In Proceedings of the IEEE International Conference on Computer Vision (ICCV).
  * Jain et al. [2015b] Jain, A., Koppula, H. S., Raghavan, B., Soh, S., & Saxena, A. (2015b).  Car that knows before you do: Anticipating maneuvers via learning temporal driving models.  In Proceedings of the IEEE International Conference on Computer Vision (pp. 3182–3190).
  * Jha et al. [2023] Jha, S., Al-Dhahir, N., & Busso, C. (2023).  Driver visual attention estimation using head pose and eye appearance information.  IEEE Open Journal of Intelligent Transportation Systems, 4, 216–231.
  * Jha & Busso [2018] Jha, S., & Busso, C. (2018).  Probabilistic estimation of the gaze region of the driver using dense classification.  In 2018 21st International Conference on Intelligent Transportation Systems (ITSC) (pp. 697–702).  IEEE.
  * Ji & Yang [2001] Ji, Q., & Yang, X. (2001).  Real time visual cues extraction for monitoring driver vigilance.  In International Conference on Computer Vision Systems (pp. 107–124).  Springer.
  * Ji & Yang [2002] Ji, Q., & Yang, X. (2002).  Real-time eye, gaze, and face pose tracking for monitoring driver vigilance.  Real-time imaging, 8, 357–377.
  * Kanade et al. [2021] Kanade, P., David, F., & Kanade, S. (2021).  Convolutional neural networks (cnn) based eye-gaze tracking system using machine learning algorithm.  European Journal of Electrical Engineering and Computer Science, 5, 36–40.
  * Kar & Corcoran [2017] Kar, A., & Corcoran, P. (2017).  A review and analysis of eye-gaze estimation systems, algorithms and performance evaluation methods in consumer platforms.  IEEE Access, 5, 16495–16519.
  * Khan & Lee [2019] Khan, M. Q., & Lee, S. (2019).  Gaze and eye tracking: Techniques and applications in adas.  Sensors, 19, 5540\.
  * Klaib et al. [2021] Klaib, A. F., Alsrehin, N. O., Melhem, W. Y., Bashtawi, H. O., & Magableh, A. A. (2021).  Eye tracking algorithms, techniques, tools, and applications with an emphasis on machine learning and internet of things technologies.  Expert Systems with Applications, 166, 114037.
  * Koay et al. [2022] Koay, H. V., Chuah, J. H., Chow, C.-O., & Chang, Y.-L. (2022).  Detecting and recognizing driver distraction through various data modality using machine learning: A review, recent advances, simplified framework and open challenges (2014–2021).  Engineering Applications of Artificial Intelligence, 115, 105309.
  * König & Buffalo [2014] König, S. D., & Buffalo, E. A. (2014).  A nonparametric method for detecting fixations and saccades using cluster analysis: Removing the need for arbitrary thresholds.  Journal of neuroscience methods, 227, 121–131.
  * Kotseruba & Tsotsos [2022] Kotseruba, I., & Tsotsos, J. K. (2022).  Attention for vision-based assistive and automated driving: A review of algorithms and datasets.  IEEE Transactions on Intelligent Transportation Systems, .
  * Kramer [2020] Kramer, A. F. (2020).  Physiological metrics of mental workload: A review of recent progress.  Multiple-task performance, (pp. 279–328).
  * Krejtz et al. [2015] Krejtz, K., Duchowski, A., Szmidt, T., Krejtz, I., González Perilli, F., Pires, A., Vilaro, A., & Villalobos, N. (2015).  Gaze transition entropy.  ACM Transactions on Applied Perception (TAP), 13, 1–20.
  * Krizhevsky et al. [2012] Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2012).  Imagenet classification with deep convolutional neural networks.  In F. Pereira, C. Burges, L. Bottou, & K. Weinberger (Eds.), Advances in Neural Information Processing Systems.  Curran Associates, Inc. volume 25.
  * Lamble et al. [1999] Lamble, D., Laakso, M., & Summala, H. (1999).  Detection thresholds in car following situations and peripheral vision: Implications for positioning of visually demanding in-car displays.  Ergonomics, 42, 807–815.
  * Lavalliere et al. [2011] Lavalliere, M., Laurendeau, D., Simoneau, M., & Teasdale, N. (2011).  Changing lanes in a simulator: effects of aging on the control of the vehicle and visual inspection of mirrors and blind spot.  Traffic injury prevention, 12, 191–200.
  * Lee et al. [2011] Lee, S. J., Jo, J., Jung, H. G., Park, K. R., & Kim, J. (2011).  Real-time gaze estimator based on driver’s head orientation for forward collision warning system.  IEEE Transactions on Intelligent Transportation Systems, 12, 254–267.
  * Lehtonen et al. [2014] Lehtonen, E., Lappi, O., Koirikivi, I., & Summala, H. (2014).  Effect of driving experience on anticipatory look-ahead fixations in real curve driving.  Accident Analysis & Prevention, 70, 195–208.
  * Lemonnier et al. [2015] Lemonnier, S., Brémond, R., & Baccino, T. (2015).  Gaze behavior when approaching an intersection: Dwell time distribution and comparison with a quantitative prediction.  Transportation research part F: traffic psychology and behaviour, 35, 60–74.
  * Lemonnier et al. [2020] Lemonnier, S., Désiré, L., Brémond, R., & Baccino, T. (2020).  Drivers’ visual attention: A field study at intersections.  Transportation research part F: traffic psychology and behaviour, 69, 206–221.
  * Li et al. [2019] Li, G., Wang, Y., Zhu, F., Sui, X., Wang, N., Qu, X., & Green, P. (2019).  Drivers’ visual scanning behavior at signalized and unsignalized intersections: A naturalistic driving study in china.  Journal of safety research, 71, 219–229.
  * Liu et al. [2022] Liu, F., Chen, D., Zhou, J., & Xu, F. (2022).  A review of driver fatigue detection and its advances on the use of rgb-d camera and deep learning.  Engineering Applications of Artificial Intelligence, 116, 105399.
  * Liu et al. [2018] Liu, W., Muramatsu, S., & Okubo, Y. (2018).  Cooperation of v2i/p2i communication and roadside radar perception for the safety of vulnerable road users.  In 2018 16th International Conference on Intelligent Transportation Systems Telecommunications (ITST) (pp. 1–7).  IEEE.
  * Liu et al. [2002] Liu, X., Xu, F., & Fujimura, K. (2002).  Real-time eye detection and tracking for driver observation under various light conditions.  In Intelligent Vehicle Symposium, 2002. IEEE (pp. 344–351).  IEEE volume 2.
  * Long et al. [2022] Long, Y., Huang, J., Zhao, X., & Li, Z. (2022).  Does lstm outperform 4ddtw-knn in lane change identification based on eye gaze data?  Transportation research part C: emerging technologies, 137, 103583.
  * LRD et al. [2022] LRD, M., Mukhopadhyay, A., & Biswas, P. (2022).  Distraction detection in automotive environment using appearance-based gaze estimation.  In 27th International Conference on Intelligent User Interfaces (pp. 38–41).
  * Lu et al. [2022] Lu, K., Dahlman, A. S., Karlsson, J., & Candefjord, S. (2022).  Detecting driver fatigue using heart rate variability: A systematic review.  Accident Analysis & Prevention, 178, 106830.
  * Lyu et al. [2022] Lyu, Z., Qi, C., Zhu, S., & Wang, H. (2022).  The visual scanning behavior and mental workload of drivers at prairie highway intersections with different characteristics.  IEEE Access, 10, 123043–123056.
  * Majaranta & Bulling [2014] Majaranta, P., & Bulling, A. (2014).  Eye tracking and eye-based human–computer interaction.  In Advances in physiological computing (pp. 39–65).  Springer.
  * Marquart et al. [2015] Marquart, G., Cabrall, C., & de Winter, J. (2015).  Review of eye-related measures of drivers’ mental workload.  Procedia Manufacturing, 3, 2854–2861.
  * Martin et al. [2018] Martin, S., Vora, S., Yuen, K., & Trivedi, M. M. (2018).  Dynamics of driver’s gaze: Explorations in behavior modeling and maneuver prediction.  IEEE Transactions on Intelligent Vehicles, 3, 141–150.
  * Mecheri et al. [2022] Mecheri, S., Mars, F., & Lobjois, R. (2022).  Gaze and steering strategies while driving around bends with shoulders.  Applied ergonomics, 103, 103798.
  * Milanés et al. [2012] Milanés, V., Villagrá, J., Godoy, J., Simó, J., Pérez, J., & Onieva, E. (2012).  An intelligent v2i-based traffic management system.  IEEE Transactions on Intelligent Transportation Systems, 13, 49–58.