Date of publication xxxx 00, 0000, date of current version xxxx 00, 0000.
10.1109/ACCESS.2017.DOI

© 2023 IEEE. Personal use of this material is permitted. Permission from IEEE
must be obtained for all other uses, in any current or future media, including
reprinting/republishing this material for advertising or promotional purposes,
creating new collective works, for resale or redistribution to servers or
lists, or reuse of any copyrighted component of this work in other works

Corresponding author: Sei Ikeda (e-mail: ikeda.sei.jp@ieee.org).

# Fixation-based Self-calibration for
Eye Tracking in VR Headsets

RYUSEI URAMUNE1  SEI IKEDA1 HIROKI ISHIZUKA1  AND OSAMU OSHIRO1  Graduate
School of Engineering Science, Osaka University, Toyonaka 5608531, Japan

###### Abstract

This study proposes a novel self-calibration method for eye tracking in a
virtual reality (VR) headset. The proposed method is based on the assumptions
that the user’s viewpoint can freely move and that the points of regard (PoRs)
from different viewpoints are distributed within a small area on an object
surface during visual fixation. In the method, fixations are first detected
from the time-series data of uncalibrated gaze directions using an extension
of the I-VDT (velocity and dispersion threshold identification) algorithm to a
three-dimensional (3D) scene. Then, the calibration parameters are optimized
by minimizing the sum of a dispersion metrics of the PoRs. The proposed method
can potentially identify the optimal calibration parameters representing the
user-dependent offset from the optical axis to the visual axis without
explicit user calibration, image processing, or marker-substitute objects. For
the gaze data of 18 participants walking in two VR environments with many
occlusions, the proposed method achieved an accuracy of 2.1∘, which was
significantly lower than the average offset. Our method is the first self-
calibration method with an average error lower than 3∘ in 3D environments.
Further, the accuracy of the proposed method can be improved by up to 1.2∘ by
refining the fixation detection or optimization algorithm.

###### Index Terms:

fixation detection, valuation metric, eye tracking, virtual reality

=-15pt

##  I Introduction

In virtual and augmented reality (VR/AR), eye tracking has been shown to be
effective not only for offline applications, such as psychological analysis
and usability evaluation, but also for various online applications, including
foveated rendering [1], precise ocular parallax displays [2, 3], depth of
field rendering [4], auto-focus displays [5], and object selection [6]. As of
2023, camera-based eye-tracking is still widely used as a standard feature in
VR/AR head-mounted displays (HMDs), as exemplified by FOVE, HTC Vive Pro Eye,
Magic Leap, and Microsoft HoloLens2. Most devices need to be calibrated before
use to obtain highly accurate line of sight. In the above applications, the
accuracy of gaze measurement can be one of the main factors that determine the
performance of the applications. For example, in foveated rendering, low
accuracy in gaze measurement increases rendering cost by enlarging the area of
detail to be rendered; in object selection, it becomes difficult to select
small objects.

The principle of eye tracking calibration is strongly based on the human
visual system. The human eyeball obtains a clear image by rotating to capture
a target at the fovea, a spot on the retina where the distribution density of
cones, the photo-receptors mainly related to bright vision, is the highest.
The line passing through the fovea and the center of the lens is called the
visual axis. In contrast, an eye tracker can directly observe some feature
related to the optical (or geometric) axis, which is the line passing through
the center of the eyeball rotation and the center of the pupil. The visual
axis is typically offset from the optical axis by about 4∘ to 7∘ [7].
According to literature [8], the offset has non-negligible individual
differences, with a maximum difference of 19.57∘ in the horizontal direction
and 16.25∘ in the vertical direction. Thus, only information about the optical
axis can be directly obtained from eye observations; the offset angle is
estimated in the calibration of eye trackers.

In traditional calibration methods, several reference markers are presented to
the user, and the user is asked to individually gaze at each marker, both
before and sometimes after eye tracking. Such calibration inhibits VR
experiences. After such calibration, many state-of-the-art eye trackers
maintain accuracy, even when the head is moved in relation to the device. This
ability depends not so much on the calibration method as on the gaze
estimation model. As summarized in the next section, various gaze estimation
models exist and can be divided into four categories: two-dimensional (2D)
regression models, three-dimensional (3D) eye model-based models, appearance-
based models, and head-correlation models [9, 10]. The former three models are
based on eye observations and require calibration. The head-correlation models
are challenging approach to estimate the visual axis from head motion [11, 12,
13] without the use of eye trackers. Although they do not require the
estimation of offsets, they are inaccurate.

Self-calibration (also called automatic or implicit calibration) can be
defined as a technology that estimates the individual offset angles from other
information, such as scene images or time-series data of the optical axis.
Self-calibration can reduce total experimental time in offline applications.
In online applications, the eye tracker can be calibrated at any time as
needed. Self-calibration methods can be divided into saliency-based [14, 15,
16, 17, 18], smooth pursuit-based [19, 20], and scene model-based methods [21,
22, 23, 24, 25, 26, 27], as described in detail in the next section. These
approaches vary in terms of finding the visual axis cues. Most methods compute
the offset angles from the cues obtained by analyzing scene images or scene
models over a set amount of time. Once the offsets are obtained, the optical
axis can be directly converted to the visual axis. In particular, scene model-
based methods have been reported to achieve higher accuracy than the other
methods. However, no cases of successful calibration using this type of method
in 3D environments in which the user can freely move have been reported to
date. Moreover, the conventional scene model-based methods are limited to flat
monitors [21, 22, 23, 24, 25], objects at infinity [26], or differentiable
curved surfaces [27].

In this study, we propose a self-calibration method for an eye-tracking VR
headset. The method has the following advantages:

  * •

No active gazing at markers or fingers is required; instead, it is only
necessary for a certain amount of passive movement such as freely looking or
walking in a VR environment.

  * •

No markers or specific objects to replace markers are required.

  * •

The scenes can contain multiple objects with various textures, shapes, and
occlusions.

  * •

The method does not rely on computationally expensive image processing, such
as saliency map generation, optical flow estimation, or object detection.

The proposed method assumes the head position and scene model as known
information and estimates the calibration parameters using the points of
regard (PoRs) during fixational eye movements. Therefore, this method can be
classified as a scene model-based method. In this study, as in previous
studies [28, 29], we define fixation as a state in which the PoRs are densely
packed in the object coordinates for a certain period of time. According to
this definition, even when the object being gazed at or the head is in motion,
the eyes track a single point on the object by smooth pursuit eye movements
during a fixation. Thus, in principle, our method is also similar to smooth
pursuit-based methods except that the object being tracked must be identified.

As shown in Fig. 1, our method is based on the assumptions that the PoRs in
each fixation are densely concentrated on an object surface if the eye tracker
is accurately calibrated and that the dispersion of the PoRs in each fixation
increases when the gaze measurement contains large errors.  Evaluating the
dispersion of PoRs is similar to evaluating whether the thinnest part of the
line-of-sight bundle is near the object surface. the proposed method is
inspired by multi-view geometry techniques such as multi-view stereo and
camera self-calibration, where the user’s view is represented by a pinhole
camera model. In such multi-view techniques, rays from multiple cameras to the
same point intersect on the object surface as a clue to estimate some
parameter. It is well known that the rotation of the camera is generally
irrelevant, and the baseline distance between different cameras affects the
accuracy. The same is true for our method, where simply rotating the user’s
head at the same point is not sufficient for calibration; the head must be
translating during the short period of time that fixation occurs.

The proposed method has two steps. First, it detects fixations from the time-
series data of uncalibrated gaze directions. We extend an existing I-VDT
(velocity and dispersion threshold identification) [30] algorithm suitable for
fixation detection in 3D scenes. Second, the method estimates calibration
parameters by minimizing a novel cost function. The cost function is designed
based on the reprojection errors of the PoRs that are inspired by multi-view
geometry in computer vision. The reprojection errors are defined on the image
plane of a scene camera, which is a pinhole camera model representing the
user’s field of view (See Fig. 1).

Figure 1: Principle of the proposed fixation-based calibration method. During
a fixation, human eyeballs move so that the PoRs are concentrated around a
single point, even if the head moves. We applied this characteristic and the
corresponding multi-view geometry to the self-calibration of an eye tracker.

##  II Related work

In Section II-A, we first briefly summarize the various models for gaze
estimation from sensing data and show the types of calibration parameters
required for each model. Then, in Section II-B, we explain the existing
methods for automatically estimating such parameters and clarify the
differences between these and the proposed method. Finally, in Section II-C,
we describe the existing fixation detection algorithms and their problems, as
the results of fixation detection significantly affect calibration accuracy
(see Section IV-D).

###  II-A Gaze Estimation Models

As mentioned above, existing gaze estimation models can be divided into four
types: 2D regression models, 3D eye models, appearance models [9], and head-
correlation models. This section discusses the conventional gaze estimation
models and how they are not capable of determining the visual axis with
sufficient accuracy without the provision of individual offset angles.

#### 2D Regression Models

2D regression models map 2D coordinates, including the pupil center and glint
positions (reflections of light sources), in an eye image to a two-degree-of-
freedom (DoF) gaze direction. As mapping functions, polynomials [31, 16, 32]
are often used as Taylor approximations of arbitrary transformation functions
[33]. In addition to polynomials, machine learning models such as Gaussian
process interpolation [34], neural networks [35], and support vector machines
[36] are used.

Calibration of a 2D regression models requires numerous pairs of pupil center,
approximately corresponding to the optical axis, and marker position,
corresponding to the visual axis. The regression coefficients in the
calibration are the coefficients of the mapping function. These coefficients
contain information on the pose of the eyeball relative to the camera and the
offset angles, which cannot be separated. Therefore, in 2D regression models,
recalibration is necessary when the user’s head moves relative to the eye
tracker, even if the same user is continuously using the same tracker. In
addition, calibration generally requires a sufficient number of markers to
cover the entire field of view (FOV). In our experiments, we used the
calibration results from a polynomial-based 2D regression as a control
condition.

#### 3D Eye Models

3D eye models determine the optical axis through feature detection on eye
images. Most of the eye model-based approaches approximate the surface shape
of the eyeball using a combination of two spheres [37, 38]; a few of them use
an ellipsoid [39] or a solid of revolution around the optical axis [40]. The
optical axis can be determined using one camera and a few point light sources,
allowing for head movement [41]. Our method adopts two DoF offset angles as
the calibration parameters of the sphere model.

For calibration, most 3D models use one [37, 42] or two [39, 38, 40, 15] DoF
offset angles between the optical and visual axes. In rare cases, models that
require the estimation of three parameters (one DoF offset between the optical
and visual axes, radius of corneal curvature, and distance between the pupil
center and the center of corneal curvature) also exist [40, 43]. In summary,
each model has different individual parameters, which must be estimated for
each user calibration. Our method adopts two DoF offset angles as the
calibration parameters.

#### Appearance Models

In appearance models, an eye image is used without manually designed feature
extraction and machine learning to directly output the gaze direction. Several
learning algorithms have been considered, including neural networks with three
layers [44, 45, 46], manifold learning [47], Gaussian processes [48], and deep
neural networks [49, 50]. A complete description of the state-of-the-art
convolutional neural network-based methods, summarized in [51], lies outside
the scope of this study.

The performance of machine learning depends on the training dataset. For these
techniques to be used as calibration-free gaze estimators, they must be
trained on a dataset that does not include the user’s data. However,
individual differences in the position of the fovea cannot be predicted from
the appearance of the eyeball. For high-accuracy estimation of the individual
visual axis, training with data specific to each individual user [45, 49] or
compensating for accuracy using an offset table [44, 46] that maps the amount
of correction for each user is necessary. Either of these requires at least
one explicit operation similar to user calibration. In general, appearance
models are less accurate than 3D or regression models [10].

#### Head-correlation Models

Head-correlation models are the challenging techniques that estimate the
visual axis without observation of eyes [11, 12, 13]. These models commonly
estimate the direction of the visual axis by combining the head–eye
coordination and the scene points that the user is likely to pay attention to.
The fingertip position [11], saliency map [12, 13], and moving objects [13]
are used as cues to identify such scene points. Calibration is not required
for these models, as the visual axis is estimated frame by frame. In
principle, the offset can be determined by combining the visual axis estimated
by these models and the optical axis obtained from the eye tracker. That is,
these models are applicable for self-calibration if the estimation accuracy is
sufficiently high. Unfortunately, the reported estimation errors exceed 7∘
[11, 12, 13], which is an angle as large as or larger than the typical offset.

###  II-B Self-calibration Methods

Self-calibration is a technique that estimates the offset angles without the
placement of any special objects or without the user being aware of the
process. Table I summarizes the features of conventional self-calibration
methods. Three main approaches to self-calibration exist: saliency-based,
smooth pursuit-based, and scene model-based methods.

TABLE I: Comparison of Self-calibration Methods

Methods1 | Base approaches | Target scene | Head pose | Scene image | Accuracy2
---|---|---|---|---|---
Sugano et al. (2013)[14] | saliency | flat monitor | fixed | necessary | 3.5∘
Chen et al. (2015)[52, 15] | saliency | flat monitor3 | fixed | necessary | 3.4∘
Wang et al. (2016)[16] | saliency | flat monitor3 | free | necessary | 1.0∘/1.4∘
Alnajar et al. (2017)[53] | saliency | flat monitor | free | necessary | 4.2∘
Hiroe et al. (2019)[54] | saliency | flat monitor4 | fixed | necessary | 1.71∘
Liu et al. (2020)[18] | saliency | arbitrary scene | free | necessary | 3.7∘/4.0∘
Shi et al. (2020)[17] | saliency | arbitrary scene | free | necessary | 3∘–4∘
Murauer et al. (2018)[19] | smooth pursuit | arbitrary scene5 | free | necessary | 6.17∘
Tamura et al. (2020)[20] | smooth pursuit | arbitrary scene5 | free | necessary | —
Nagamatsu et al. (2010)[21] | scene model | flat monitor | free | unnecessary | 1.58∘
Model et al. (2010)[22, 55] | scene model | flat monitor | free | unnecessary | 1.3∘
Maio et al. (2011)[23] | scene model | flat monitor | free | unnecessary | 2.4∘–8.9∘
Nagamatsu et al. (2013)[26] | scene model | infinity | free | unnecessary | 0.2∘
Miki et al. (2016)[24] | scene model | flat monitor | free | unnecessary | 0.5∘ 6
Wang et al. (2018)[25] | scene model | flat monitor | free | unnecessary | 1.3∘
Uramune et al. (2020)[27] | scene model | continuous surface | free | unnecessary | 1.29∘
Proposed method (optical axis)7 | scene model | arbitrary scene | free | unnecessary | 2.12∘
Proposed method (visual axis)8 | scene model | arbitrary scene | free | unnecessary | 1.20∘

1\. The self-calibration methods that do not require real or virtual markers
are listed.
2\. The accuracy evaluation methods are not standardized. Only a rough
comparison is meaningful.
3\. The scene is limited to a static scene image with low entropy.
4\. The face of a person must be displayed.
5\. Smooth pursuit-based methods rely on a few objects that move in
correlation with the eye.
6\. Real gaze data were not used in the experiments.
7\. The optical axis was used as the initial calibration parameter for
fixation detection.
8\. The visual axis was used as the initial calibration parameter for fixation
detection.

#### Saliency-based Methods

Saliency-based methods extract the global maximum position of the saliency in
the FOV as the visual axis. Because a saliency map generated from a single-
frame scene image is multi-modal, most saliency-based methods try to improve
accuracy by accumulating multiple frames of saliency maps. However, the
accuracy of most methods with simple accumulation is limited to about 3∘ [14,
52]. To further improve accuracy and efficiency, several methods limit the
scenes or frames to be accumulated to only those with low entropy [15, 16],
with faces [54], or with gaze data of many other people looking at the same
scene [53]. However, such approaches limit the applications or situations. In
all of the studies presented above, experiments were conducted with the
participants seated in front of a flat monitor.

For 3D scenes, two methods have been proposed to calibrate a head-mounted eye
tracker while the user is freely walking in a real [18] or VR [17]
environment. To more accurately detect the maximum saliency, these methods
adopt multi-frame accumulation considering the 3D environments. Liu et al.
[18] limited the area of saliency to the object area, and Shi et al. [17]
projected the saliency map of one eye onto the other eye via the 3D shape of
the scene. Despite these improvements, accuracy of about 3∘ or worse has been
reported [18, 17]. In addition, acquiring scene images to compute the saliency
map frequently incurs additional high computational costs in VR systems.

#### Smooth Pursuit-based Methods

Smooth pursuit-based methods have emerged as PoR estimation techniques without
user calibration. They can also be applied to self-calibration after PoRs with
high accuracy are obtained [56]. Such methods can identify the target of the
user’s gaze based on the similarity between the motion of the eye and the
motion of the target in the scene images. Once the target is determined, the
offset can be calculated by considering one point on the target object as the
direction of the visual axis [19, 20].

Smooth pursuit-based methods assume that identifying a single image point that
moves on a similar trajectory as the PoR is possible. This requires the target
object to be small, and no other object must have the same movement. The
accuracy of these methods has been reported to be around 4∘ to 6∘ [19]. These
methods are similar in principle to our proposed method, which utilizes the
PoRs during smooth pursuits caused by fixation; however, the proposed method
does not require the identification of the object at which the user is gazing.

#### Scene Model-based Methods

In scene model-based methods, the optical axis direction and the scene shape
are used to determine the visual axis direction. Nagamatsu et al. proposed a
method to calculate a single PoR as the midpoint between the two intersections
of the optical axes of both eyes on a monitor plane [21]. This method is
equivalent to estimating the horizontal component of the offset between the
optical and visual axes but cannot correct for the vertical component. Model
et al. generalized this method to estimate the two DoF offset angles for each
eye, minimizing the distance between the two intersections of the optical axes
of both eyes on the planar screen [22]. This generalized method has been
improved to make the solution more stable [24] and efficient [57]. In addition
to the constraint of coincidence of the intersections of the two eyes’ visual
axes [21], other constraints have been indicated. Examples include the
constraint that the PoR is within the screen and the range of the offset
angles [23], and the constraint that the results of the different methods
coincide [25]. In the methods mentioned above, however, the scene is limited
to a flat monitor [21, 22] or to infinity [26].

Our proposed method can be classified as a scene model-based method in the
sense that it assumes the 3D shape of the scene, which is inevitably available
for VR applications. Our previous work has also proposed a self-calibration
method using the variance of PoRs on the surface of a 3D scene model [27].
However, although this method is similar to the proposed method, it does not
support scenes with discontinuous depths or occlusions.

###  II-C Fixation Detection Algorithms for 3D Scenes

In this section, we first review the fundamentals of fixation detection; next,
we provide a overview of detection algorithms. During fixation, the gaze
remains stationary for at least 80–100 ms within 2∘– 5∘ of the central visual
field [9, 58]. When the target object or the head is moving, fixation is
maintained by smooth pursuit eye movements and the vestibulo-ocular reflex.
Rapid eye movements between two successive fixations are called saccades,
which have velocities of 100 – 700 ∘/s [10]. A detailed explanation of eye
movements is beyond the scope of this paper. Interested readers may refer to
[7] for a detailed description.

The results of fixation detection depend on the detection algorithm and
parameters such as threshold values [59, 60]. Previous studies [58, 61, 62]
reported that the number of fixations and the average fixation time are
significantly influenced by changes in the algorithm and threshold values. In
some cases, these changes affect the conclusions of later analyses [59]. The
optimization results of the proposed method also depend on the results of
fixation detection, as shown in Section IV-D. Thus, here we explain the need
for algorithms suitable for fixation detection in 3D scenes.

Fixation detection algorithms from time-series gaze direction data can be
divided into two main categories based on the reference coordinate systems:
scene camera coordinate system and object coordinate system [63]. The scene
camera coordinate system involves a coordinate system fixed to the scene
camera– that is, the user’s head. If the head is fixed, there is little
difference between the two systems; otherwise, their properties significantly
differ. The algorithms using scene camera coordinates takes advantage of the
fact that the angular velocity of the gaze during fixation is sufficiently
slow compared with that of the saccades. A typical algorithm is I-VT (velocity
threshold identification) [63], which identifies gaze directions as a fixation
if the angular velocity of successive gaze directions in the scene camera
coordinate system is below a certain threshold. This algorithm has a wide
range of applications because it does not require object information and is
almost independent of the initial calibration parameters. However, it tends to
produce more misdetections than algorithms using object coordinates [63]. This
may be even more likely when head rotation occurs.

The algorithms that use object coordinates are based on the fact that the
dispersion of PoRs during a fixation is small. A typical algorithm is I-DT
(dispersion threshold identification) [63], which identifies PoRs as a
fixation if the PoRs in the object coordinate system are within a certain
range for a certain period of time or longer. This algorithm is less prone to
false positives from other eye movements. Larsson et al. have tried to develop
an intermediate algorithm that detects fixations by thresholding the angular
velocity in the world coordinate system with the head pose without object
shapes [64]. This method allows head movements and arbitrary scenes. However,
a single velocity threshold can incorrectly detect fixation in situations
where head movement is large, such as in our VR environments. I-VDT [30]
simply combines a velocity threshold with a dispersion threshold to achieve
the intermediate properties of I-DT and I-VT. In principle, I-DT and I-VDT are
expected to be insensitive to both rotation and translation of the head, but
neither of them is applicable to arbitrary scenes, as the dispersion of PoRs
is defined only on a flat monitor.

Only few methods for detecting fixation for scenes with large head movements
and arbitrary shapes have been proposed [29, 65]. One is to use pattern
matching with the similarity of scene image patches around the PoRs to detect
fixations [29]. The disadvantages of this method are misdetections in texture-
less and repeated-pattern regions and bandwidth-consuming image capturing.
Another method is to define the dispersion on a 3D object [65]. This method is
similar to the algorithm we employ. Although this method has been recently
published, various studies have clarified the properties of I-DT and I-VT.
Therefore, in this study, we further extend the I-VDT method, which combines
both I-DT and I-VT, to a 3D environment. The details of the algorithm and its
relationship with I-DT and I-VT are explained in the next sections.

##  III Method

###  III-A Preliminary Information

We first present our assumptions and define the variables necessary to the
explanation of our proposed method. In this method, a single virtual scene
camera (pinhole camera) representing the user’s cyclopean eye is used for all
our gaze processing, although different stereoscopic images are presented to
both eyes on the HMD. Assuming that the focal length of the scene camera is 1,
an arbitrary gaze direction can be expressed as $\bm{g}=[u,v,1]^{T}$ using the
homogeneous representation of the 2D coordinate $[u,v]^{T}$ on the image
plane. If the position of a PoR represented as an object coordinate $\bm{X}$,
the projection onto the scene camera image yields the following equation:

$\displaystyle\bm{g}=\pi(\bm{X}),$ (1)

where $\pi$ is the mapping from the object coordinate $\bm{X}$ to the image
coordinate. $\pi$ can be obtained at the same acquisition rate as the gaze
measurement. Conversely, if $\bm{X}$ is the point where the extension of the
line of sight in the direction $\bm{g}$ first collides with the object
surface, it is expressed as $\bm{X}=\pi^{-1}(\bm{g})$ using the inverse
projection $\pi^{-1}$.

The inputs in this method include a set of gaze directions $\\{\bm{g}_{i}\\}$,
the position and orientation of the HMD (corresponding to the projection
$\\{\pi_{i}\\}$ or inverse projection $\\{\pi_{i}^{-1}\\}$) at $M$ discrete
time frames $\ (i=1,2,\cdots,M)$, and a 3D environment model. The model’s
texture is displayed but not used for the computation of the proposed method.
In the general 3D eye model described in Section II-A, the transformation from
the optical axis $\bm{g}_{\rm{opt}}$ to the visual axis $\bm{g}_{\rm{vis}}$ is
presented by the following equation:

$\displaystyle\bm{g}_{\rm{vis}}$ $\displaystyle\propto$ $\displaystyle
f(\bm{g}_{\rm{opt}}\,;\,{\bm{\theta}}_{\rm{offset}}),$ (2) $\displaystyle=$
$\displaystyle\begin{bmatrix}1&0&0\\\ 0&\hskip
4.2679pt\cos\beta_{0}&\sin\beta_{0}\\\
0&-\sin\beta_{0}&\cos\beta_{0}\end{bmatrix}\begin{bmatrix}\hskip
8.53581pt\cos\alpha_{0}&0&\sin\alpha_{0}\\\ 0&1&0\\\
-\sin\alpha_{0}&0&\cos\alpha_{0}\end{bmatrix}\bm{g}_{\rm{opt}},\hskip
11.38109pt$ (3)

where $\propto$ is the symbol for proportionality in the homogeneous
representation, indicating that both sides are equal in Cartesian
representation. $\bm{\theta}_{\rm{offset}}=[\alpha_{0},\beta_{0}]$ is the
offset angle addressed in Section 1. $\alpha_{0}$ and $\beta_{0}$ are the
horizontal and vertical components of the offset angle, respectively.  In this
model, the axis directly measured by the eye tracker is not necessarily the
optical axis, as long as it satisfies Eq. (3), i.e., the axis only needs to be
fixed to the eyeball. Although strictly speaking, the axis must pass through
the center of rotation of the eyeball, but if the scale of the environment is
large enough, a relatively small discrepancy between the center of rotation
and the axis can be ignored.

The calibration parameters to be obtained as the final output are the
estimated offset angles in the 3D eye model. The calibration function $f$
mapping $\bm{g}_{\rm{opt}}$ to an arbitrary direction $\bm{g}$ can be
expressed as the following equation:

$\displaystyle\bm{g}$ $\displaystyle\propto$ $\displaystyle
f(\bm{g}_{\rm{opt}}\,;\,{\bm{\theta}}),$ (4)

where $\bm{\theta}=[\alpha,\beta]$ is a set of two arbitrary calibration
parameters. $\alpha$ and $\beta$ are the horizontal and vertical rotation
angles of the gaze direction, respectively.

The proposed method estimates the calibration parameter $\bm{\theta}$ using
the above input information in two major steps: fixation detection and
calibration parameter optimization. In the next section, we explain in detail
the extended I-VDT algorithm for fixation detection. Then, in Section III-C,
we introduce the reprojection errors and describe the optimization method
using these errors.

###  III-B Fixation Detection

Fixations are detected using the I-VDT algorithm extended to a 3D environment.
Our extended I-VDT (referred to as 3D I-VDT) combines I-VT with the extended
I-DT (referred to as 3D I-DT) to 3D scenes; also, the original I-VDT is a
combination of I-VT and I-DT. Therefore, this section describes the I-VT, 3D
I-DT, and 3D I-VDT in order.

These algorithms generally adopt a moving window for successive data frames to
check for a fixation candidate. The window has minimum $m$ frames that must
satisfy certain conditions. $m$ can be determined by the minimum fixation time
$T_{\rm{th}}$ and the sampling rate $S_{\rm{r}}$:

$m=S_{\rm{r}}T_{\rm{th}}.$ (5)

In any algorithm, the movement of the window is as follows. Initially, the
window contains $m$ frames. Let ${\bm{G}}$ be an index set or the window,
which is initially set as ${\bm{G}}=\\{i-m+1,\cdots,i\\}$, where $i$ is the
current frame. If some criterion computed from the gaze data in the window
satisfies a certain condition, the window is expanded to include the next
frame, i.e., $\\{i-m+1,\cdots,i+1\\}$. The window continues to expand as many
times as the condition is satisfied. If the condition is no longer satisfied
owing to the window expansion, the gaze data in the window right before the
expansion is labeled as the same fixation. The window is shrunk to the
original size $m$ and jumped so that the index set is updated to
${\bm{G}=\\{i+1,\cdots,i+m\\}}$. If the condition is not satisfied without the
expansion, all the frames in the window are identified as not a fixation; the
window slides by one frame.

#### Algorithm of I-VT

The I-VT algorithm [63] detects fixations based on the angular velocity for
each gaze direction. In implementation, $\\{\bm{g}_{k}\\}$ is categorized as a
fixational gaze if any gaze direction $\bm{g}_{k}$ satisfies the following
equation.

$\displaystyle\frac{\bm{g}_{k-1}\cdot\bm{g}_{k}}{|\bm{g}_{k-1}||\bm{g}_{k}|}$
$\displaystyle>$ $\displaystyle\cos\phi_{\rm{th}}\ \ \ (k-1,k\in\bm{G}),$ (6)

where $\phi_{\rm{th}}$ is the angle threshold and ‘$\cdot$’ represents the
inner product.

#### Algorithm of 3D I-DT

The original I-DT algorithm [63] calculates a dispersion of the PoRs from the
gaze data contained in the window $\bm{G}$. Dispersion can be calculated using
several methods, such as the distance between the most distant PoRs, the
largest distance from the center of the PoRs to any PoR, and the standard
deviation [60]. The conventional I-DT is limited to a flat monitor, and the
dispersion of the PoRs is calculated on a 2D plane.

To extend the I-DT to 3D scenes with head movements, the 3D I-DT calculates a
2D dispersion after the PoRs are projected onto a scene camera. For the
projection, we select a center camera from the multiple scene cameras
corresponding to window $\bm{G}$. The center camera represents an average head
posture. Because the window size is about a few hundred milliseconds and the
change in head posture is small, a camera with an average gaze direction is
selected as the center camera. For gaze directions
$\\{\bm{g}_{k}|k\in\bm{G}\\}$, the selection criterion to determine the
$k_{c}$-th camera as the center camera is as follows:

$\displaystyle\bm{g}^{\prime}_{k}$ $\displaystyle\propto$
$\displaystyle\bm{\mathrm{R}}^{-1}_{k}\bm{g}_{k},$ (7) $\displaystyle k_{c}$
$\displaystyle=$
$\displaystyle\mathop{\rm{arg~{}min}}\limits_{k\in\bm{G}}|\bm{g}^{\prime}_{k}-\bar{\bm{g}}|,$
(8)

where $\bar{\bm{g}}$ is the average gaze direction defined as:

$\displaystyle\bar{\bm{g}}=\frac{1}{m}\sum_{k\in\bm{G}}\bm{g}^{\prime}_{k}.$
(9)

$\bm{g}_{k}$ and $\bm{g}^{\prime}_{k}$ are the gaze direction vectors
expressed in the scene camera and object coordinates, respectively, and both
are normalized so that their magnitude is 1. $\bm{\mathrm{R}}_{k}$ is the
rotation matrix from the scene camera coordinates to the object coordinates
and is the rotational component of the inverse projection function
$\pi_{k}^{-1}$.

The dispersion is evaluated with reprojection errors on the image plane of the
center camera in 3D I-DT. Fig. 2 shows a reprojection error as the 2D distance
between a projected PoR and the average point of all the PoRs on the center
camera image. The 3D I-DT algorithm first calculates the 3D position of the
PoR ${\bm{X}}_{k}$ as the inverse projection of the gaze direction
$\bm{g}_{k}$:

$\displaystyle{\bm{X}}_{k}$ $\displaystyle=$
$\displaystyle\pi_{k}^{-1}(\bm{g}_{k}).$ (10)

Next, the PoR ${\bm{X}}_{k}$ is projected to the center camera to obtain the
reprojection point ${\bm{x}}_{k}$ using the following equation:

$\displaystyle\bm{x}_{k}$ $\displaystyle=$
$\displaystyle\pi_{k_{c}}({\bm{X}}_{k}).$ (11)

This projection prevents extremely large evaluation values even for PoRs
scattered in the depth direction near the occlusion edge of the object. Then,
the average position $\bar{\bm{x}}$ of ${\bm{X}}_{k}$ can be calculated by

$\displaystyle\bar{\bm{x}}$ $\displaystyle=$
$\displaystyle\frac{1}{m}\sum_{k\in\bm{G}}\bm{x}_{k}.$ (12)

Finally, the gaze directions $\\{\bm{g}_{k}|k\in\bm{G}\\}$ are labeled as a
single fixation if the reprojected PoRs satisfy the following equation:

$\displaystyle\mathop{\rm{arg~{}max}}\limits_{k\in\bm{G}}|\bm{x}_{k}-\bm{\bar{x}}|^{2}<D_{\rm{th}},$
(13)

where $D_{\rm{th}}$ is the dispersion threshold.

#### Algorithm of 3D I-VDT

The I-VDT algorithm is a fixation detection algorithm that combines I-VT and
I-DT. 3D I-VDT uses both a velocity threshold $\phi_{\rm{th}}$ and a
dispersion threshold $D_{\rm{th}}$. The same moving window as in 3D I-DT is
used. If all the gaze directions in the window simultaneously satisfy both
Equations (6) and (13), I-VDT categorizes the gaze directions
$\\{\bm{g}_{k}|k\in\bm{G}\\}$ as a fixation.

Both 3D I-DT and 3D I-VDT algorithms work equivalently to the original I-DT
and I-VDT, respectively, under the fixed head condition. For simplicity, we
abbreviate below the 3D I-DT and 3D I-VDT algorithms as I-DT and I-VDT,
respectively, unless there is a risk of misunderstanding. In addition, let $N$
fixations be obtained using the abovementioned I-VDT algorithm and
$\bm{G}_{j}$ be the set of frame numbers corresponding to the $j$-th detected
fixation that comprises of $M_{j}$ frames. Hereafter, we call each set a
fixation cluster.

Figure 2: Reprojection errors. In the proposed method, the calibration
parameters are estimated by optimizing a cost function based on the
reprojection errors.

###  III-C Optimization of Reprojection Errors

In the first part of this section, we define the optimization metric,
reprojection error; in the second part, we describe the optimization method.
As mentioned in Section III-B, the reprojection error is defined on the center
camera that is representative of each fixation cluster, as in Fig. 2. Although
the definition is the same, and we refer to the same figure as in the
description of fixation detection, the projection and inverse projection in
the optimization are slightly different from those in I-DT: the parameters in
the process of optimization are used for both projection and back projection.

The optimization metric is determined as follows. First, we select the
$k_{c}$-th camera as the center camera from the $M_{j}$ scene cameras
belonging to the $j$-th fixation cluster using Equation (7) and (8). The
reprojection error function $E_{j}$ of the $j$-th fixation cluster
$\bm{G}_{j}$ is the mean of squares of the reprojection errors. $E_{j}$ is
defined by the following equation:

$\displaystyle E_{j}(\bm{\theta})$ $\displaystyle=$
$\displaystyle\frac{1}{M_{j}}\sum_{k\in\bm{G}_{j}}|{\bm{x}}_{k}-\bm{\bar{x}}|^{2},$
(14)

where:

$\displaystyle{\bm{x}}_{k}$ $\displaystyle=$
$\displaystyle\pi_{k_{c}}({\bm{X}}_{k}),$ (15) $\displaystyle\bar{\bm{x}}$
$\displaystyle=$
$\displaystyle\frac{1}{M_{j}}\sum_{k\in\bm{G}_{j}}{\bm{x}}_{k}.$ (16)

That is to say, $E_{j}$ is equivalent to the 2D variance of the set of the
PoRs $\\{{\bm{X}}_{i}\\}$ on the image plane. The 3D point ${\bm{X}}_{k}$ is
calculated as the inverse-projection point of the calibrated gaze direction
$\bm{g}_{k}$:

$\displaystyle{\bm{X}}_{k}$ $\displaystyle=$
$\displaystyle\pi_{k}^{-1}(\,f({\bm{g}}_{k}\,;\,{\bm{\theta}})\,).$ (17)

Thus, we can find $E_{j}$ of cluster $\bm{G}_{j}$ is a function of
$\bm{\theta}$.

Via optimization, the sum of the reprojection error functions for multiple
fixation clusters is minimized. The cost function simultaneously minimizes the
reprojection error function for $N$ fixation clusters to estimate the optimal
calibration parameters $\hat{\bm{\theta}}$:

$\hat{\bm{\theta}}=\mathop{\rm{arg~{}min}}\limits_{\bm{\theta}}\sum_{j=1}^{N}E_{j}(\bm{\theta}).$
(18)

This function has multiple local minima. Therefore, a global optimization
algorithm is required. In our method, differential evolution [66] is applied
as one possible approach for finding optimal parameters. This optimization
method is a type of evolutionary computation algorithm that does not require
differentiability or convexity assumptions for the objective function and can
search for the global optimal solution. According to the offset range of
approximately 4∘ to 7∘, as mentioned in Section 1, the method searches the
range $-5\leq\alpha\leq 5,-5\leq\beta\leq 5$ in degrees. In this range, the
maximum offset is 7.1∘. To improve search performance, the parameter space is
divided into 4 $\times$ 4 square regions, and differential evolution is
applied in each region. For the final result, the parameters with minimum cost
are adopted.

##  IV Experiments

This section demonstrates the performance of the proposed method by presenting
three experiments using a common gaze dataset acquired in two VR environments.
The common experimental setup is described in detail in Section IV-A. In
Section IV-B, we show the accuracy of the proposed method. Section IV-C
confirms the convergence performance with the walking distance. Finally, we
demonstrate the effect of the accuracy of the initial parameters on the
estimation accuracy in Section IV-D. All the experiments were approved by the
ethics committee of the Graduate School of Engineering Science, Osaka
University (R2-9) and, the participants gave informed consent.

###  IV-A Experimental Setup

In this section, we explain the setup for acquiring gaze data used in the
three experiments.

#### HMD and Eye Tracker

HTC Vive Pro Eye was used as an HMD. The resolution of the HMD is 1440
$\times$ 1600 pixels per eye (total 2880 $\times$ 1600 pixels), the refresh
rate is 90 Hz, and the viewing angle is 110∘. Rendering was implemented in the
game engine Unity on a PC (CPU: Intel Core i7-9700K, GPU: NVIDIA GeForce RTX
2080 SUPER).

Gaze data were acquired using the HMD’s built-in eye tracker (acquisition rate
as per specifications: 120 Hz), with an actual acquisition rate of
approximately 50 Hz. The head position and posture were acquired by the
Lighthouse tracking system attached to the HMD, and the acquisition rate
during the VR scene presentation was approximately 50 Hz. The acquisition
rates of the gaze measurement and the head position and posture are lower than
those mentioned in the specifications because the VR scene rendering rate is
prioritized.

We used the SRanipal SDK as a software library for acquiring gaze data. The
average of the gazes of both eyes starting from the midpoint of both pupil
centers obtained from this library was used as the user’s gaze. In addition,
using the SRanipal SDK, we can obtain the degree of eyelid opening and closing
(called openness) in the range from 0.0 (completely closed) to 1.0 (wide
open). To exclude blinks, we eliminated the gaze directions with the openness
below 0.5.

#### VR Environments

“Office Interior
Archviz111https://assetstore.unity.com/packages/3d/environments/urban/office-
interior-archviz-155701” and “Modern
Supermarket222https://assetstore.unity.com/packages/3d/environments/modern-
supermarket-186122,” which are available on the Unity Asset Store, were
partially modified for use as VR environments. Here, they are simply named
“Office” and “Supermarket,” respectively. Fig. 3 shows these two VR
environments. For safety and guiding purposes, a 2 m white square outline was
present on the floor, and no virtual objects were placed in the area around
the white outline. The real environment was a leveled floor with no objects
inside or around 1 m of the square.

(a) Office

(b) Supermarket

Figure 3: VR environments. Each participant was asked to walk along the white
outline, counterclockwise and clockwise, twice each, for a total of four
consecutive laps, starting from the upper right corner.

#### Participants

Twenty-two undergraduate and graduate students with decimal visual acuity of
0.5 or better (no other visual impairment), either with naked eyes or contact
lenses, participated in the experiment. The mean age was 22.5 years (SD: 1.8
years). They had experience wearing some HMDs but did not understand the
details of the proposed method. If, during or after the experiment, the
average openness of one eye was found to be less than 0.5 or the average error
of the control condition, described later, was found to be 1.0∘ or greater,
the data of that participant were not used. As a result, gaze data were
obtained for 18 participants.

#### Data Acquisition Procedure

Gaze data were acquired while the participants were walking and freely looking
around in VR environments. Each participant first received instruction on the
experiments and gave informed consent. In addition, we told the participants
that they would be given some simple quizzes about the objects in the VR
environments after experiencing these environments. Next, participants were
asked to gaze at the markers for the traditional calibration and then walk in
two different VR environments. Finally, the participant was asked to complete
short quizzes about the scenes and a questionnaire about their visual acuity.
These quizzes were dummy tasks. We did not use their answer data. The details
of each are described below.

#### Tasks

To counterbalance the order effect, the participants were divided into two
groups that experienced the two environments in reverse order. For each
environment, starting from the upper right corner in Fig. 3, each participant
was asked to walk along the white outline in the VR environments,
counterclockwise and clockwise, twice each, for a total of four consecutive
laps. The pace of the walk was sometimes instructed to the participant so that
the total duration would be around 180 s.

#### Thresholds for Fixation Detection

The threshold values for each fixation detection algorithm were as follows. In
I-VDT, the velocity threshold was set to 80 ∘/s (${\phi_{\rm{th}}=0.16}$∘,
${S_{\rm{r}}=50}$ Hz) and the dispersion threshold to 0.7∘
(${D_{\rm{th}}=1.5\times 10^{-4}}$), following previous studies [67] that
investigated the best threshold of I-VDT by grid search. Similarly, for I-VT
and I-DT, the thresholds were set to 80∘/s and 0.7∘, respectively. Because the
minimum fixation time is generally set to a value from 100 to 200 ms [63], we
set $T_{\rm{th}}$ to 160 ms for all three algorithms.

TABLE II: Combination of Fixation Detection Algorithms and Initial Parameters Methods | I-VT (opt) | I-VT (vis) | I-DT (opt) | I-DT (vis) | I-VDT (opt) | I-VDT (vis)
---|---|---|---|---|---|---
Fixation detection algorithms | I-VT | I-VT | I-DT | I-DT | I-VDT | I-VDT
Initial parameters for fixation detection | optical axis | visual axis | optical axis | visual axis | optical axis | visual axis
Initial parameters for optimization | optical axis | optical axis | optical axis | optical axis | optical axis | optical axis

#### Control Condition and Accuracy Evaluation

We regarded the parameters estimated by traditional marker-based calibration
with a 2D regression model as the control condition. The gaze direction
obtained after a built-in calibration of the Vive Pro Eye for each participant
was not treated as the ground truth because of its insufficient accuracy,
sometimes far exceeding 1∘. With the Vive Pro Eye, built-in calibration should
be performed at least once to acquire gaze data. Therefore, the built-in
calibration was performed only once for one of the authors, who is not
included in the participants, and we maintained this state while acquiring the
gaze data of all the participants.

The parameters of the control condition was estimated using 5 $\times$ 5 black
cross markers. The markers were displayed in pseudo random order on a gray-
colored plane fixed to the scene camera coordinate system and 1 m in front of
the user, with a FOV angle of 20∘. We asked each participant to gaze at each
marker for 3 s. This resulted in 25 sets of gaze data for each participant. We
used 16 of the these sets for the estimation of the control condition
parameters; the other 9 sets were used for confirming the accuracy of the
ground parameters $\hat{\bm{\theta}}$. Details on the methods for displaying
the markers and estimating the control condition are provided in the
supplemental material.

The Vive Pro Eye does not provide the optical axis, but it provides the gaze
direction corresponding to the visual axis estimated by the built-in
calibration process. Therefore, we generated arbitrary parameters from the
control condition, i.e., the estimated visual axis. The optical axis
$\bm{g}_{\rm{opt}}$ can be calculated from the visual axis $\bm{g}_{\rm{vis}}$
by the inverse transformation of Equation (2), as shown in the following
equation:

$\displaystyle\bm{g}_{\rm{opt}}$ $\displaystyle\propto$ $\displaystyle
f^{-1}(\bm{g}_{\rm{vis}}\,;\,{\bm{\theta}}_{\rm{offset}}).$ (19)

We determined the gaze direction $\bm{g}$ calibrated by arbitrary parameters
${\bm{\theta}}$ by the following redefined equation:

$\displaystyle\bm{g}$ $\displaystyle\propto$ $\displaystyle
f^{-1}(\bm{g}_{\rm{vis}}\,;\,{\bm{\theta}}).$ (20)

In this case, the control condition parameters can be represented as
$\bm{\theta}=[0,0]$. For the evaluation of the accuracy of both the proposed
method and the control condition, the estimated parameters were applied to the
same nine sets, and the average absolute error from the markers was evaluated
in degrees.

###  IV-B Accuracy Evaluation

#### Purpose

In this study, accuracy evaluation has two purposes. The first is to confirm
the feasibility of our self-calibration method. One straightforward criterion
for feasibility is that the accuracy of the visual axis estimated by the
proposed method is higher than that of the optical axis as the initial
parameters obtained by the 3D eye model. The second is to determine the impact
of the choice of fixation detection algorithm and the use of uncalibrated gaze
data for fixation detection on accuracy.

#### Method

We evaluated the accuracy of the variations of the proposed method and
compared the accuracy with that of the optical axis and the visual axis
(control condition). As variations of the proposed method, we prepared six
methods that are all possible combinations of three fixation detection
algorithms (I-VT, 3D I-DT, and 3D I-VDT) and two initial values (opt: optical
axis, vis: visual axis) given to each fixation detection algorithm. For all
the methods, the initial parameter for optimization is the optical axis. Table
II summarizes all the methods. For the optical axis, average offsets reported
in a previous study were used. The literature [8] reported the average offsets
of both eyes in a total of 1020 Brazilian, Chinese, and Italian participants.
In our experiment, the offset of the cyclopean eye was approximated
$\bm{\theta}_{\rm{offset}}=[1.02,3.30]$ by averaging each component of both
eyes’ offsets.

We applied each method to three sets of data containing fixations detected in
Office, Supermarket, and their combination (named Office+Supermarket). For
reference, the average absolute errors of the optical and visual axes (control
condition) of all the participants were also computed. In total, the average
absolute error of 20 conditions (3 detection algorithms $\times$ 2 initial
values $\times$ 3 environments + 2 references) was evaluated.

For each environment condition, we performed a multiple test on the seven
results of all the participants: the results of the six proposed methods and
the optical axis. Because the prior Shapiro-Wilk test [68] confirmed that the
absolute errors of each method were not normally distributed, we applied
Steel–Dwass’s method [69, 70], which is a non-parametric multiple comparison
test that assumes arbitrary probability distributions.

Figure 4: Absolute errors of the proposed method. “opt” indicates that the
initial calibration parameters corresponding to the optical axis were used in
the fixation detection. Similarly, “vis” refers to fixation detection using
the parameters of the visual axis. “Visual axis” represents the accuracy of
the control condition with 16 of the 25 markers. “Optical axis” shows the
accuracy of the optical axis generated by adding the average offset to the
control condition. All the errors were evaluated with the same gaze data of
the other nine markers. The error bars indicate standard errors for all
participants.

#### Results

Fig. 4 shows the average absolute errors under all the conditions. First, we
focus on the case “opt” and present the results of using the optical axis for
fixation detection. The absolute errors of I-DT (opt) and I-VDT (opt) were
significantly ($p<0.01$) lower than those of the optical axis in all
environmental conditions. I-VDT (opt) in the Office + Supermarket had the
highest accuracy with 2.12∘ among the “opt” methods. Meanwhile, I-VT (opt) was
not significantly different from the optical axis, except in the Office.

Next, we focus on the comparison of “opt” with “vis”, i.e., the difference
between the initial parameters. I-DT (vis) and I-VDT (vis) showed improved
accuracy of I-DT (opt) and I-VDT (opt), respectively, under all conditions. In
Office + Supermarket, I-DT (vis) and I-VDT (vis) were significantly better
($p<0.05$) for all “opt” methods. I-DT (vis) in the Office + Supermarket was
the most accurate with 1.20∘.

(a)

(c)

(b)

(d)

Figure 5: Absolute error and number of fixations. The cumulative distance is
the sum of the translational distances of the scene cameras between adjacent
frames.

#### Discussion

Some expected results were obtained. First, the proposed method with I-VDT
(opt) was more accurate than the optical axis. This means that the proposed
method is applicable to gaze estimation based on the 3D eye model. None of the
self-calibration methods applicable to arbitrary scenes shown in Section II
have been statistically shown to be more accurate in estimating calibration
parameters than the optical axis. Thus, the proposed method is the first whose
feasibility has been confirmed.

Second, the average absolute errors of I-VDT (opt) in any scenes were also
smaller than those of any conventional self-calibration methods for arbitrary
scenes [18, 17, 19], i.e., about 3∘ at best. This result is consistent with
the fact that scene model-based methods [40, 22, 55, 23, 24, 37] tend to be
more accurate than saliency-based methods [14, 15] or smooth pursuit-based
methods [19], as shown in Table I. In previous studies, participants were
asked to search for a specific object [17] or to see a scene in with one
salient object [20, 19] in their experiments. In our experiments, we have not
employed the tasks, which could potentially increase the accuracy of the
calibration. However, the statistical superiority of the proposed method
cannot be determined without the unification of experimental conditions. For
example, our method adopts a cyclopean eye, whereas in previous studies, the
parameters were independently estimated for both eyes. In terms of the
stability of parameter estimation, this clearly works in favor of the proposed
method. Comparative experiments under standardized conditions have not been
performed in previous studies. This is our future work.

In contrast, unexpected results were also obtained. First, the absolute error
of “vis” was smaller than that of “opt” in I-DT and I-VDT in Office +
Supermarket environments. In particular, the accuracy of “vis” in I-DT was
surprisingly improved to 1.2∘, which is fairly close to the accuracy of the
visual axis (green bar in Fig. 4). Note that we used the visual axis
parameters only for fixation detection and not for optimization, and the
effect of the different parameters on the fixation detection is only a slight
increase or decrease in the number of frames judged to be fixations. This
result indicates the little to no cause for inaccuracy in the optimization,
and that mainly fixation detection has room for improvement. In general,
calibrated gaze data must be used for fixation detection. To our knowledge,
previous studies have not discussed the effect of calibration accuracy on the
results of fixation detection. Therefore, in Section IV-D, we comprehensively
investigate the relationship between the calibration parameters for fixation
detection and the accuracy of the proposed method. The above trend did not
exist in the case with I-VT. We expected this to some extent because changes
in the offset of the gaze direction have little effect on the angular velocity
of the gaze direction used by I-VT as a criterion. In the future, development
of a fixation detection method that is less dependent on initial parameters
and the optimization of parameters for the entire method including fixation
detection are necessary.

Second, it was found that the accuracy of Supermarket tended to be lower and
the that of Office + Supermarket tended to be higher than that of the other
scenes. Although statistical significance was not confirmed, the Office scene
was more accurate than the Supermarket scene under any condition, and the same
trend was observed in Sections IV-C and IV-D. The cause of this scene
dependence could not be clarified in our experiments. A possible cause is the
variation of surface normals. The sensitivity of the reprojection error
depends on the scene distance and the surface inclination. The proposed method
was expected to average out the bias owing to such sensitivity using multiple
fixations because a realistic scene had surfaces with various normal
directions. The abovementioned scene dependence is consistent with the results
if interpreted as follows: Supermarket had a biased distribution of surface
normals and was less accurate than Office; Office + Supermarket had the best
accuracy owing to the enhanced averaging by the increased variations of
surface normals. Another possible cause could simply be an increase in the
number of fixations. This is contradicted by the experiments in the next
section.

###  IV-C Convergence Performance by Walking Distance

#### Purpose

This experiment aimed to confirm the relationship between the accuracy of the
estimated parameters and the cumulative translational distance of users. This
relationship indicates how far the user must move to complete the calibration.
The convergence speed of automatic calibration is often based on time or the
number of frames [14]. Because the proposed method requires fixations to occur
along with the translational movement of the user’s head, we confirmed the
accuracy and the number of fixations with respect to the translational
movement distance.

#### Method

For the two VR scenes, we obtained the average absolute errors in distance
intervals from 3 to 34 m in 1 m increments. A minimum distance of 3 m was
considered because in some cases, fixation is not observed at distances less
than 3 m. A maximum distance of 34 m was set as the shortest cumulative
distance among the 18 participants. Three algorithms, I-VT, I-DT, and I-VDT,
were used for fixation detection, with the optical axis as defined in Section
IV-B. The average absolute errors at 3, 18, and 34 m were compared for the
three fixation detection algorithm. The Steel–Dwass method was used to test
for differences.

#### Results

Fig. 5 (a) and (b) show the average absolute error versus cumulative distance
in Office and Supermarket, respectively. Fig. 5 (c) and (d) show the average
number of fixations versus cumulative distance in Office and Supermarket,
respectively. The error bands with light colors along the lines indicate
standard errors. Except for I-VT in Supermarket, the average errors at 18 and
34 m showed significantly higher accuracy than that at 3 m ($p<0.05$). No
significant difference was found between the average errors at 18 and 34 m.
I-VT in Supermarket showed no significant difference between any two of the
three distances.

The number of fixations decreased in the order of I-VT, I-DT, and I-VDT at any
distance. Significant differences between I-VT and I-VDT were confirmed at 3
($p<0.05$), 18 ($p<0.05$), and 34 m ($p<0.01$). For reference, the average
computation times of the two scenes were 10, 28, and 56 s at 3, 18, and 34 m,
respectively, in the case of I-VDT.

#### Discussion

The accuracy of the proposed method in I-DT and I-VDT improved at the
cumulative distance of 18 m compared with that at the start of walking, and
this value is not significantly different from the accuracy at 34 m.
Therefore, the parameters almost converge after a walking distance of about 20
m. The speed to reach 34 m in 180 s is approximately 0.7 km/h, which is
considerably slower than the typical walking speed of 4 km/h. This is because
we could only prepare a small environment, and we instructed them to walk at
such a speed. Considering that the proposed method utilizes the translation of
the head position, even faster convergence can possibly be achieved if the
user walks at normal speed.

Although the trend was for the error to decrease with distance, at some
distances, the error locally increased in both scenes even with I-DT and
I-VDT. We present four possible hypotheses regarding the cause of this result:
the head position was almost stationary, few fixations were detected, the
dispersion of the detected fixations was large, and the inclination of the
surface where the PoRs of fixations are distributed was biased. According to
the first hypothesis, fixations detected without head movement adversely
affect parameter estimation. However, in that case, the number of fixations at
that distance should rapidly increase because the cumulative distance does not
increase. As shown in Fig. 5 (c) and (b), no such increase is found; thus, the
first hypothesis does not hold. In contrast, if no fixation is detected as the
cumulative distance increases, the absolute error should remain constant.
Therefore, the second hypothesis is also rejected because the same is true.
The third hypothesis cannot be ruled out because the dispersion of the
fixation depends on psychological states [71]. We asked each participant to
pace themselves or reverse their walking direction, which may have led to a
loss of concentration. In our method, such psychological factors are not taken
into account. Regarding the fourth hypothesis, the possibility of a bias in
the inclination of the surface at some distances cannot be ruled out.

###  IV-D Dependence on Initial Parameters

#### Purpose

This experiment was performed to analyzed the dependence of the accuracy of
the proposed method on the initial calibration parameters used for fixation
detection. As mentioned in Section II, in general, fixation detection results
depend on the algorithm and the initial calibration parameters. However, the
results of fixation detection using pre-calibration parameters are unknown. In
this section, we confirmed the effect of the initial parameters of fixation
detection on the accuracy of the proposed method.

#### Method

The parameter space was divided into five regions, and each region was
referred to as Range 1–5. We define Range $l$ as ${l-1\leq|\alpha|\leq l}$ or
${l-1\leq|\beta|\leq l}$ ($l=1,2,\cdots,5$ in degrees). Fifty sets of random
calibration parameters $[\alpha,\beta]$ were generated with the uniform
distribution for each parameter range, and fixation detection was performed
using each set of parameters. We obtained the calibration accuracy for each
participant using the fixation detection results and computed the average
absolute error for each parameter range. Consequently, we acquired 900 results
($=50$ parameters $\times$ 18 participants). The same accuracy results were
obtained for six conditions: two scenes (Office, Supermarket) and three
algorithms (I-VT, I-DT, I-VDT). As in the experiments described above, the
Steel–Dwass method was applied for testing against a set of parameter ranges
for each condition.

#### Results

Fig. 6 shows the relationship between calibration accuracy and parameter
range. The error bands with light colors along the lines indicate standard
errors. In the cases of I-DT and I-VDT, any two of the parameter ranges were
significantly different ($p<0.01$) from each other in both scenes. For I-VT in
Office, significant differences ($p<0.01$) were observed between Range 1 and
any of Range 3–5 and between Range 2 and any of Range 4 and 5. For I-VT in
Supermarket, no significant difference was observed between any two parameter
ranges.

In Office, the absolute error of the three methods was comparable in Range 3.
In Supermarket, the absolute error of I-VT, I-DT, and I-VDT were comparable in
Range 4. Therefore, I-DT and I-VDT are superior for Range 1 and 2, whereas
I-VT is superior for Range 5. In additon to the results of the accuracy
evaluation shown in Fig. 4, the absolute error was lower in Supermarket than
in Office for all parameter ranges of any of the fixation detection
algorithms.

#### Discussion

Fig. 6 clearly shows that some fixation detection algorithms such as I-VT have
almost no parameter dependence on accuracy and that the initial calibration
parameters for fixation detection with I-DT and I-VDT have a strong influence
on the overall processing. The fact that accurate initial parameters are
needed to estimate accurate calibration parameters may seem to be a “chicken-
or-egg” situation; nevertheless, the proposed method was able to estimate
parameters with higher accuracy than the optical axis. To further improve
accuracy, two issues need to be addressed: providing accurate initial
parameters and developing an accurate fixation detection method with lower
parameter dependence.

The output of the conventional methods can be used to obtain accurate initial
parameters. For accurate fixation detection methods, fortunately, our
experiments allow a quantitative evaluation of the accuracy of a fixation
detection method itself. Studies on the objective and quantitative evaluation
of fixation detection methods are limited and do not deal with 3D scenes where
the head moves [60, 30, 72].

Figure 6: Dependence on parameters for fixation detection. Range $l$ indicates
${l-1\leq|\alpha|\leq l}$ or ${l-1\leq|\beta|\leq l}$ ($l=1,2,\cdots,5$ in
degrees). The bands with light colors indicate standard errors.

##  V Limitations

In this section, we clarify the limitations of the proposed method.

#### Statistical Comparison with Previous Studies

The qualitative comparisons in Table I show that the conventional methods
available for 3D environments [18, 17, 19, 20] are directly comparable to the
proposed method. However, we did not perform a quantitative comparison with
these conventional methods. The same is true of most of studies on these
conventional methods. The reason is probably the same as ours: the source code
is not open, and implementation details are not sufficiently clear. Therefore,
the standardization of evaluation methods and quantitative comparison under
the same conditions will be our focus in future work.

A comparison of their accuracy, although under different conditions, should
not be ignored for the following reasons. First, the methods for accuracy
evaluation are almost the same in most conventional studies as well as in
ours. In particular, in all methods, the angular difference in gaze directions
after calibration to markers is used as an evaluation index, although the
number, shape, and arrangement of markers are different; moreover the accuracy
of the control condition is not taken into account in previous studies.
Second, in general, every study presents experimental results in the
environment in which the method proposed in that study shows the highest
accuracy. In other words, comparing the upper limits of accuracy is
beneficial. Accordingly, it can be said that the potential of our method has
been sufficiently demonstrated, but its superiority over conventional methods
cannot be determined.

#### Moving & Deforming Objects

To avoid complications in implementation, we did not include moving objects in
the VR scenes. A smooth pursuit to gaze at a moving object is equivalent to a
moving head in the object coordinate system. Theoretically, the presence of
moving objects should work to the advantage of our method.

Moving or deforming objects complicate the calculation of the collision
position in the gaze direction. In particular, the collision point depends on
the time as well as the position and orientation of the scene camera.
Therefore, during optimization, the collision points must be calculated with
various parameters at various times. The implementation can be simplified
using only the line of sight to static objects.

#### Physiological & Psychological Factors

The proposed method does not take optokinetic response (OKR) and vestibulo-
ocular reflex (VOR) into account, although free head movement and rotation are
assumed. The accuracy of the proposed method could be improved using a
mathematical model of OKR or VOR gain, which causes gaze shift. In addition,
the proposed method does not take psychological effects into account. The
introduction of oculomotor and psychological models is a future challenge.

#### Cyclopean Eye

In many conventional scene model-based methods, both eyes are treated
independently. In contrast, in this study, we applied the scene camera at the
cyclopean eye, as mentioned in Section IV-B, because independently treating
both eyes was expected to have little effect. The scene depth in our
experimental environments was dominated by the region over 5 m, and we
expected that the change in convergence angle would not be stable compared
with the case of a planar monitor placed at about 1 m. Because many gaze
analyses use some method of averaging the gaze directions of both eyes [73,
74], we conclude that the proposed method is at least effective for such
applications.

#### Optimization Algorithm & Computation Time

Many local solutions to the optimization of the proposed method are available.
Differential evolution is less accurate than the brute force method. The
calculation of collision points and the projection of PoRs require most of the
computation time during optimization, which can be parallelized. Therefore,
there is room for improving the computation time.

#### AR Scenario

The proposed method is suitable for VR environments because of its assumptions
about scene models and head postures; theoretically, it can be also extended
to AR environments. The difference between AR and VR is the accuracy of the
scene model. The effects of the scene model and head pose accuracy on
calibration accuracy is a topic of great importance and interest.

##  VI Conclusion

In this study, we propose a self-calibration method for an eye-tracking VR HMD
based on fixation detection. Although the proposed method can be categorized
as a scene model-based method, it is applicable to arbitrary scenes with large
head movements and a large number of occlusions, unlike conventional methods.
The proposed method can automatically calibrate the eye tracker with an
accuracy of about 2∘ in about 30 s if the user walks about 20 m. The 3D I-VDT
algorithm is the best for fixation detection in terms of accuracy and
computational cost compared with the I-VT and 3D I-DT algorithm. The
evaluation function for optimization has little room for improvement. Further
improvement in accuracy can be achieved by developing a fixation detection
algorithm that is less dependent on the initial calibration accuracy or by
developing a method for the simultaneous optimization of both the calibration
parameters and the fixation detection results. Quantitative comparison with
conventional methods under uniform conditions is the main future research
avenue.

## References

  * [1] X. Meng, R. Du, M. Zwicker, and A. Varshney, “Kernel foveated rendering,” _Proceedings of the ACM on Computer Graphics and Interactive Techniques_ , vol. 1, no. 1, pp. 1–20, 2018.
  * [2] R. Konrad, A. Angelopoulos, and G. Wetzstein, “Gaze-contingent ocular parallax rendering for virtual reality,” _ACM Transactions on Graphics_ , vol. 39, no. 2, pp. 1–12, 2020.
  * [3] B. Krajancich, P. Kellnhofer, and G. Wetzstein, “Optimizing depth perception in virtual and augmented reality through gaze-contingent stereo rendering,” _ACM Transactions on Graphics_ , vol. 39, no. 6, pp. 1–10, 2020.
  * [4] T. Orisaka and T. Okatani, “A gaze-reactive display for simulating depth-of-field of eyes when viewing scenes with multiple depths,” _IEICE Transactions on Information and Systems_ , vol. E99.D, no. 3, pp. 739–746, 2016.
  * [5] N. Padmanaban, R. Konrad, and G. Wetzstein, “Autofocals: Evaluating gaze-contingent eyeglasses for presbyopes,” _Science Advances_ , vol. 5, no. 6, p. eaav6187, 2019.
  * [6] D. Mardanbegi, T. Langlotz, and H. Gellersen, “Resolving target ambiguity in 3D gaze interaction through VOR depth estimation,” in _Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems_ , 2019, pp. 1–12.
  * [7] R. Carpenter, _Movements of the eyes_. London: Pion, 1988.
  * [8] A. Abass, R. Vinciguerra, B. T. Lopes, F. Bao, P. Vinciguerra, R. Ambrósio, Jr., and A. Elsheikh, “Positions of ocular geometrical and visual axes in Brazilian, Chinese and Italian populations,” _Current Eye Research_ , vol. 43, no. 11, pp. 1404–1414, 2018.
  * [9] D. W. Hansen and Q. Ji, “In the eye of the beholder: A survey of models for eyes and gaze,” _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , vol. 32, no. 3, pp. 478–500, 2010.
  * [10] A. Kar and P. Corcoran, “A review and analysis of eye-gaze estimation systems, algorithms and performance evaluation methods in consumer platforms,” _IEEE Access_ , vol. 5, pp. 16 495–16 519, 2017.
  * [11] Y. Li, A. Fathi, and J. M. Rehg, “Learning to predict gaze in egocentric video,” in _2013 IEEE International Conference on Computer Vision_ , 2013, pp. 3216–3223.
  * [12] Z. Hu, C. Zhang, S. Li, G. Wang, and D. Manocha, “SGaze: A data-driven eye-head coordination model for realtime gaze prediction,” _IEEE Transactions on Visualization and Computer Graphics_ , vol. 25, no. 5, pp. 2002–2010, 2019.
  * [13] Z. Hu, S. Li, C. Zhang, K. Yi, G. Wang, and D. Manocha, “DGaze: CNN-based gaze prediction in dynamic scenes,” _IEEE Transactions on Visualization and Computer Graphics_ , vol. 26, no. 5, pp. 1902–1911, 2020.
  * [14] Y. Sugano, Y. Matsushita, and Y. Sato, “Appearance-based gaze estimation using visual saliency,” _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , vol. 35, no. 2, pp. 329–341, 2013.
  * [15] J. Chen and Q. Ji, “A probabilistic approach to online eye gaze tracking without explicit personal calibration,” _IEEE Transactions on Image Processing_ , vol. 24, no. 3, pp. 1076–1086, 2015.
  * [16] K. Wang, S. Wang, and Q. Ji, “Deep eye fixation map learning for calibration-free eye gaze tracking,” in _Proceedings of the Ninth Biennial ACM Symposium on Eye Tracking Research & Applications_, 2016, pp. 47–55.
  * [17] P. Shi, M. Billeter, and E. Eisemann, “SalientGaze: Saliency-based gaze correction in virtual reality,” _Computers & Graphics_, vol. 91, pp. 83–94, 2020.
  * [18] M. Liu, Y. Li, and H. Liu, “3D gaze estimation for head-mounted eye tracking system with auto-calibration method,” _IEEE Access_ , vol. 8, pp. 104 207–104 215, 2020.
  * [19] M. Murauer, M. Haslgrübler, and A. Ferscha, “Natural pursuits for eye tracker calibration,” in _Proceedings of the 5th international Workshop on Sensor-based Activity Recognition and Interaction_ , 2018, pp. 1–10.
  * [20] Y. Tamura and K. Takemura, “Estimating point-of-gaze using smooth pursuit eye movements without implicit and explicit user-calibration,” in _ACM Symposium on Eye Tracking Research & Applications_, 2020, pp. 1–4.
  * [21] T. Nagamatsu, R. Sugano, Y. Iwamoto, J. Kamahara, and N. Tanaka, “User-calibration-free gaze tracking with estimation of the horizontal angles between the visual and the optical axes of both eyes,” in _Proceedings of the 2010 Symposium on Eye-Tracking Research & Applications_, 2010, pp. 251–254.
  * [22] D. Model and M. Eizenman, “User-calibration-free remote gaze estimation system,” in _Proceedings of the 2010 Symposium on Eye-Tracking Research & Applications_, 2010, pp. 1–8.
  * [23] W. Maio, J. Chen, and Q. Ji, “Constraint-based gaze estimation without active calibration,” in _IEEE International Conference on Automatic Face & Gesture Recognition_, 2011, pp. 627–631.
  * [24] K. Miki, T. Nagamatsu, and D. W. Hansen, “Implicit user calibration for gaze-tracking systems using kernel density estimation,” in _Proceedings of the Ninth Biennial ACM Symposium on Eye Tracking Research & Applications_, 2016, pp. 249–252.
  * [25] K. Wang and Q. Ji, “3D gaze estimation without explicit personal calibration,” _Pattern Recognition_ , vol. 79, pp. 216–227, 2018.
  * [26] T. Nagamatsu, T. Ueki, and J. Kamahara, “Automatic user calibration for gaze-tracking systems by looking into the distance,” in _3rd International Workshop on Pervasive Eye Tracking and Mobile Eye-Based Interaction_ , 2013, pp. 1–6.
  * [27] R. Uramune, S. Ikeda, H. Ishizuka, and O. Oshiro, “Automatic calibration of eye trackers using 3D positions of fixations and virtual object surface shapes,” in _IPSJ SIG Technical Report_ , vol. 2021-CVIM-224, 2021, pp. 1–6, (in Japanese).
  * [28] B. Massé, S. Ba, and R. Horaud, “Tracking gaze and visual focus of attention of people involved in social interaction,” _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , vol. 40, no. 11, pp. 2711–2724, 2018.
  * [29] J. Steil, M. X. Huang, and A. Bulling, “Fixation detection for head-mounted eye tracking based on visual similarity of gaze targets,” in _Proceedings of the 2018 ACM Symposium on Eye Tracking Research & Applications_, 2018, pp. 1–9.
  * [30] O. V. Komogortsev and A. Karpov., “Automated classification and scoring of smooth pursuit eye movements in the presence of fixations and saccades,” _Behavior research methods_ , vol. 45, no. 1, pp. 203–215, 2013.
  * [31] K. P. White, T. E. Hutchinson, and J. M. Carley, “Spatially dynamic calibration of an eye-tracking system,” _IEEE Transactions on Systems, Man, and Cybernetics_ , vol. 23, no. 4, pp. 1162–1168, 1993.
  * [32] D. M. Stampe, “Heuristic filtering and reliable calibration methods for video-based pupil-tracking systems,” _Behavior Research Methods, Instruments, & Computers_, vol. 25, no. 2, pp. 137–142, 1993.
  * [33] P. Blignaut, “Mapping the pupil-glint vector to gaze coordinates in a simple video-based eye tracker,” _Journal of Eye Movement Research_ , vol. 7, no. 1, pp. 1–11, 2013.
  * [34] D. Hansen, J. Hansen, M. Nielsen, A. Johansen, and M. Stegmann, “Eye typing using Markov and active appearance models,” in _Sixth IEEE Workshop on Applications of Computer Vision_ , 2002, pp. 132–136.
  * [35] Z. Zhu and Q. Ji, “Eye and gaze tracking for interactive graphic display,” _Machine Vision and Applications_ , vol. 15, no. 3, pp. 79–85, 2004.
  * [36] Z. Zhu, Q. Ji, and K. P. Bennett, “Nonlinear eye gaze mapping function estimation via support vector regression,” in _18th International Conference on Pattern Recognition_ , 2006.
  * [37] J. G. Wang, E. Sung, and R. Venkateswarlu, “Estimating the eye gaze from one eye,” _Computer Vision and Image Understanding_ , vol. 98, no. 1, pp. 83–103, 2005.
  * [38] A. Villanueva, R. Cabeza, and S. Porta, “Gaze tracking system model based on physical parameters,” _International Journal of Pattern Recognition and Artificial Intelligence_ , vol. 21, no. 5, pp. 855–877, 2007.
  * [39] D. Beymer and M. Flickner, “Eye gaze tracking using an active stereo head,” in _2003 IEEE Computer Society Conference on Computer Vision and Pattern Recognition_ , vol. 2, 2003, pp. 451–458.
  * [40] T. Nagamatsu, Y. Iwamoto, J. Kamahara, N. Tanaka, and M. Yamamoto, “Gaze estimation method based on an aspherical model of the cornea: Surface of revolution about the optical axis of the eye,” in _Proceedings of the 2010 Symposium on Eye-Tracking Research & Applications_, 2010, pp. 255–258.
  * [41] E. Guestrin and M. Eizenman, “General theory of remote gaze estimation using the pupil center and corneal reflections,” _IEEE Transactions on Biomedical Engineering_ , vol. 53, no. 6, pp. 1124–1133, 2006.
  * [42] T. Nagamatsu, J. Kamahara, T. Iko, and N. Tanaka, “One-point calibration gaze tracking based on eyeball kinematics using stereo cameras,” in _Proceedings of the 2008 Symposium on Eye Tracking Research & Applications_, 2008, pp. 95–98.
  * [43] T. Nagamatsu, M. Hiroe, and H. Arai, “Extending the measurement angle of a gaze estimation method using an eye model expressed by a revolution about the optical axis of the eye,” _IEICE Transactions on Information and Systems_ , pp. 729–740, 2021.
  * [44] S. Baluja and D. Pomerleau, “Non-intrusive gaze tracking using artificial neural networks,” in _Proceedings of the 6th International Conference on Neural Information Processing Systems_ , 1993, pp. 753–760.
  * [45] R. Stiefelhagen, J. Yang, and A. Waibel, “Tracking eyes and monitoring eye gaze,” in _Proceedings of the Workshop on Perceptual User Interfaces_ , 1997, pp. 98–100.
  * [46] L. Q. Xu, D. Machin, and P. Sheppard, “A novel approach to real-time non-intrusive gaze finding,” in _Proceedings of the British Machine Conference_ , 1998, pp. 428–437.
  * [47] K. H. Tan, D. J. Kriegman, and N. Ahuja, “Appearance-based eye gaze estimation,” in _Proceedings of 6-th IEEE Workshop on Applications of Computer Vision_ , 2002.
  * [48] O. Williams, A. Blake, and R. Cipolla, “Sparse and semi-supervised visual mapping with the $\rm S^{3}GP$,” in _Proceedings of the 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition_ , vol. 1, 2006, pp. 230–237.
  * [49] X. Zhang, Y. Sugano, M. Fritz, and A. Bulling, “Appearance-based gaze estimation in the wild,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2015, pp. 4511–4520.
  * [50] K. Krafka, A. Khosla, P. Kellnhofer, H. Kannan, S. Bhandarkar, W. Matusik, and A. Torralba, “Eye tracking for everyone,” in _2016 IEEE Conference on Computer Vision and Pattern Recognition_ , 2016, pp. 2176–2184.
  * [51] A. A. Akinyelu and P. Blignaut, “Convolutional neural network-based methods for eye gaze estimation: A survey,” _IEEE Access_ , vol. 8, pp. 142 581–142 605, 2020.
  * [52] J. Chen and Q. Ji, “Probabilistic gaze estimation without active personal calibration,” in _Proceedings of the 2011 IEEE Conference on Computer Vision and Pattern Recognition_ , 2011, pp. 609–616.
  * [53] F. Alnajar, T. Gevers, R. Valenti, and S. Ghebreab, “Auto-calibrated gaze estimation using human gaze patterns,” _International Journal of Computer Vision_ , vol. 124, no. 2, pp. 223–236, 2017.
  * [54] M. Hiroe, S. Mitsunaga, and T. Nagamatsu, “Implicit user calibration for model-based gaze-tracking system using face detection around optical axis of eye,” in _Extended Abstracts of the 2019 CHI Conference on Human Factors in Computing Systems_ , 2019, pp. 1–6.
  * [55] D. Model and M. Eizenman, “An automatic personal calibration procedure for advanced gaze estimation systems,” _IEEE Transactions on Biomedical Engineering_ , vol. 57, no. 5, pp. 1031–1039, 2010.
  * [56] E. Velloso, M. Wirth, C. Weichel, A. Esteves, and H. Gellersen, “AmbiGaze: Direct control of ambient devices by gaze,” in _Proceedings of the 2016 ACM Conference on Designing Interactive Systems_ , 2016, pp. 1–6.
  * [57] S. Wang, J. Wang, H. Peng, S. Gao, and D. He, “A new calibration-free gaze tracking algorithm based on DE-SLFA,” in _2016 8th International Conference on Information Technology in Medicine and Education_ , 2016, pp. 380–384.
  * [58] B. R. Manor and E. Gordon, “Defining the temporal threshold for ocular fixation in free-viewing visuocognitive tasks,” _Journal of Neuroscience Methods_ , vol. 128, no. 1, pp. 85–93, 2003.
  * [59] F. Shic, B. Scassellati, and K. Chawarska, “The incomplete fixation measure,” in _Proceedings of the 2008 Symposium on Eye Tracking Research & Applications_, 2008, pp. 111–114.
  * [60] P. Blignaut, “Fixation identification: The optimum threshold for a dispersion algorithm,” _Attention, Perception & Psychophysics_, vol. 71, pp. 881–895, 2009.
  * [61] A. Poole and L. J. Ball, “Eye tracking in human-computer interaction and usability research: Current status and future prospects,” in _Encyclopedia of Human Computer Interaction_ , 2006, pp. 211–219.
  * [62] I. T. C. Hooge, D. C. Niehorster, M. Nyström, R. Andersson, and R. S. Hessels, “Fixation classification: How to merge and select fixation candidates,” _Behavior Research Methods_ , pp. 1–12, 2022.
  * [63] D. D. Salvucci and J. H. Goldberg, “Identifying fixations and saccades in eye-tracking protocols,” in _Proceedings of the 2000 Symposium on Eye Tracking Research & Applications_, 2000, pp. 71–78.
  * [64] L. Larsson, A. Schwaller, M. Nyström, and M. Stridh, “Head movement compensation and multi-modal event detection in eye-tracking data for unconstrained head movements,” _Journal of Neuroscience Methods_ , vol. 274, pp. 13–26, 2016.
  * [65] J. L. Jurado, J. M. Morales, J. Guixeres, and M. Alcañiz, “Development and calibration of an eye-tracking fixation identification algorithm for immersive virtual reality,” _Sensors_ , vol. 20, no. 17, pp. 1–15, 2020.
  * [66] R. Storn and K. Price, “Differential evolution – A simple and efficient heuristic for global optimization over continuous spaces,” _Journal of Global Optimization_ , vol. 11, no. 4, pp. 341–359, 1997.
  * [67] M. Startsev, I. Agtzidis, and M. Dorr, “1d CNN with BLSTM for automated classification of fixations,” _Behavior Research Methods_ , vol. 51, no. 2, pp. 556–572, 2019.
  * [68] S. S. Shapiro and M. B. Wilk, “An analysis of variance test for normality (complete samples),” _Biometrika_ , vol. 52, no. 3-4, pp. 591–611, 1965.
  * [69] R. G. D. Steel, “A rank sum test for comparing all pairs of treatments,” _Technometrics_ , vol. 2, no. 2, pp. 197–207, 1960.
  * [70] M. Dwass, “Some k-sample rank-order tests,” in _Contributions to Probability and Statistics_ , 1960, pp. 198–202.
  * [71] Z. M. Hafed and J. J. Clark, “Microsaccades as an overt measure of covert attention shifts,” _Vision Research_ , vol. 42, no. 22, pp. 2533–2545, 2002.
  * [72] O. V. Komogortsev and A. Karpov, “Automated classification and scoring of smooth pursuit eye movements in the presence of fixations and saccades,” _Behavior Research Methods_ , vol. 45, no. 1, pp. 203–215, 2012.
  * [73] S. Nakamura and S. Shimojo, “Sustained deviation of gaze direction can affect “inverted vection” induced by the foreground motion,” _Vision Research_ , vol. 43, no. 7, pp. 745–749, 2003.
  * [74] H. Wang, J. Pi, T. Qin, S. Shen, and B. E. Shi, “SLAM-based localization of 3D gaze using a mobile eye tracker,” in _Proceedings of the 2018 ACM Symposium on Eye Tracking Research & Applications_. New York, NY, USA: Association for Computing Machinery, 2018.

|  Ryusei Uramune received a B.E. degrees from Osaka University, Toyonaka,
Japan in 2022. His research interests include virtual reality technology.
---|---
|  Sei Ikeda received his Ph.D. degree from Nara Institute of Science and
Technology, Ikoma, Japan in 2006. He is an Associate Professor with the
Graduate School of Engineering Science of Osaka University. His research
interests include human-computer interaction, mixed reality, and computer
vision.
---|---
|  Hiroki Ishizuka received the B. S. , and M.S. from Meiji University, Japan
in 2011, and 2013. He received Ph.D. from Keio University in 2016. He was an
assistant professor with the faculty of engineering, Kagawa University from
2016 to 2019. He is currently an assistant professor with the graduate school
of engineering science of Osaka University. His research interests include
haptics and soft robotics.
---|---
|  Osamu Oshiro received a Ph.D. degree in Engineering from Osaka University,
Toyonaka, Japan, in 1990. He was a Research Engineer with Sumitomo Metal
Industries from 1990 to 1993 and was an Assistant/Associate Professor with the
Nara Institute of Science and Technology from 1993 to 2003. Since 2003, he has
been a Professor with the Graduate School of Engineering Science, Osaka
University. His research interests include PBR, facing technology, modeling of
distributed human system, and other areas.
---|---