11institutetext: Stony Brook University, NY, USA 22institutetext: UC Berkeley,
CA, USA 33institutetext: Waymo LLC 44institutetext: The University of
Adelaide, Adelaide, Australia

# Look Hear: Gaze Prediction for
Speech-directed Human Attention

Sounak Mondal 11 Seoyoung Ahn 22 Zhibo Yang 33 Niranjan Balasubramanian 11
Dimitris Samaras 11 Gregory Zelinsky 11 Minh Hoai 44

###### Abstract

For computer systems to effectively interact with humans using spoken
language, they need to understand how the words being generated affect the
users’ moment-by-moment attention. Our study focuses on the incremental
prediction of attention as a person is seeing an image and hearing a referring
expression defining the object in the scene that should be fixated by gaze. To
predict the gaze scanpaths in this incremental object referral task, we
developed the Attention in Referral Transformer model or ART, which predicts
the human fixations spurred by each word in a referring expression. ART uses a
multimodal transformer encoder to jointly learn gaze behavior and its
underlying grounding tasks, and an autoregressive transformer decoder to
predict, for each word, a variable number of fixations based on fixation
history. To train ART, we created RefCOCO-Gaze, a large-scale dataset of
19,738 human gaze scanpaths, corresponding to 2,094 unique image-expression
pairs, from 220 participants performing our referral task. In our quantitative
and qualitative analyses, ART not only outperforms existing methods in
scanpath prediction, but also appears to capture several human attention
patterns, such as waiting, scanning, and verification.

###### Keywords:

Scanpath Prediction Object Referral Human Attention

###  1 Introduction

Humans are unique in that we use language to direct each others’ attention in
visual tasks. For example, a customer telling a baker “I’d like the smallest
pastry on the left” communicates the desired object that needs to be selected.
Understanding the human capacity to use these referring expressions to
incrementally direct attention is an important problem in cognitive science
and has been studied for over half a century, with the more recent studies
adopting eye-tracking methods [17, 67, 3, 38]. Most relevant is work showing
the very tight link between a word in a referring expression and the very next
eye movements of the person hearing it [68, 34], suggesting that humans
incrementally integrate visual information and word-by-word linguistic
guidance in our attention control. However, these studies are limited in that
they used small numbers of simple objects (often line drawings) in arrays (not
scenes) and this constrained the linguistic complexity of the referring
expression. How spoken language guides another person’s attention in more
naturalistic and ecologically valid contexts is still an open question in
cognitive science.

As our interactions with computers, vehicles, and AR/VR devices deepen, human-
computer interaction (HCI) systems also need to give spoken guidance to users
that is similarly effective in directing their attention. But, to attain this
degree of synchrony with users, HCI systems must be able to integrate vision
and language inputs to predict human gaze. Applications with this predictive
ability will be highly time-sensitive and crucial for activities such as
voice-assisted VR driving, offering a streamlined and immersive user
experience where a person’s gaze can be directed by generated spoken language
as if the interaction were with another person. Being able to incrementally
predict how a user integrates their visual input with a spoken instruction to
direct their attention is a general advance in speech-assisted HCI, benefiting
a broad range of applications, including efficient foveated rendering [61], VR
sickness reduction [1], AR/VR eye–hand coordination analysis during human-
object interaction [42], VR skill training/assessment (_e.g_., driving [41],
surgery [6]), and user engagement analysis [36]. Using predicted gaze for each
word as guidance will enable speech-assisted HCI systems to incrementally
generate efficient and clear instructions, correctly guiding user attention.
Measuring gaze using eye-trackers instead of predicting it is more accurate,
but also costlier and has limited applicability to aforementioned scenarios
(such as time-crucial HCI) which require gaze prediction, and to situations
where eye-trackers are unavailable or prohibited.

In this context, we study the incremental object referral task, for which we
incrementally predict eye movements of humans searching for a target object in
an image as they are hearing a referring expression describing that target.
This task has not been studied previously in the context of human gaze
prediction. The standard object referral task [53, 86] requires localizing the
target object given an image and a referring expression. Our task is different
in that we aim to incrementally predict the attention of a human as they are
hearing the expression. Our task is also related to the categorical search
task [89], where humans direct their attention to a category of target object
in an image. However, our incremental object referral task differs from
categorical search in two key aspects. First, in incremental object referral,
the target is designated by a complex referring expression (_e.g_., “red
baseball glove on the desk”) since the image may contain other objects
belonging to the same category as the target. Referring expressions often
refer to a target by its attributes (“red”) or its spatial relationships to
other objects (“on the desk”), making it even more challenging to precisely
localize the target without the broader descriptive context. In categorical
search, the target is designated by only its category name (_e.g_., “baseball
glove”), which excludes the spatial terms and attributes commonly used by
humans to describe an object. This task is also less realistic in that images
depicting multiple instances of the target category are excluded (as spatial
terms and attributes are not used). Second, because categorical search studies
of attention use only an one or two word category name to designate a target
(_e.g_., “baseball glove”), most of the human search fixations occur only
after the complete referring “expression” is provided. This makes it an
impoverished example of the longer and more natural referring expressions that
we hope to study, ones requiring an incremental allocation of attention. Our
more ecologically valid incremental object referral task therefore contributes
to this cognitive science question by enabling exploration of natural
referential expressions in real-world image contexts and generating testable
hypotheses about how humans integrate language and vision.

Given its differences from related tasks that were studied previously, the
incremental object referral task presents several technical challenges that
necessitate more than mere updates to existing models. For instance, a
standard object referral model cannot be re-purposed for incremental
prediction. Although it is possible to sequentially input incomplete referring
expressions, one word at a time, and use the predicted positions as proxies
for fixation locations, this approach proves ineffective as existing object
referral models train on complete referring expressions in contrast to how
humans integrate visual and linguistic information in a word-by-word basis
[68, 34]. Alternatively, one can adapt an existing scanpath prediction model
for our incremental object referral task, but this approach is also unsuitable
because existing scanpath prediction models do not learn the object grounding
processes (for both partial and complete referring expressions) hypothesized
to underlie the gaze behavior observed in our task.

To address the aforementioned challenges of the incremental object referral
task, we introduce the Attention in Referral Transformer (ART) model. ART is
tailored to the multimodal demands of our task as it uses a multimodal
transformer encoder that jointly learns gaze prediction and object grounding
objectives. Furthermore, we integrate an autoregressive transformer decoder
that leverages fixation history to better predict the subsequent fixations
corresponding to each sequentially presented word from the referring
expression. This innovative decoder component flexibly adjusts both the count
and the parameters of predicted fixations in alignment with the evolving
input, thus mirroring the dynamic nature of human attention.

Figure 1: RefCOCO-Gaze Dataset. Sample image-expression pairs and
corresponding scanpaths under our incremental object referral task. Fixations
(denoted by circles numbered with fixation order) are color-coded to the
corresponding word in the referring expression (above each image). Fixations
color-coded to [BOT] occurred before the expression started, and fixations
color-coded to [EOT] occurred after the expression ended. Blue bounding boxes
indicating referred objects were not visible during trials.

To train ART, we collected RefCOCO-Gaze (Fig. 1), a large-scale dataset of
gaze behavior from 220 people performing the incremental object referral task
on 2,094 images and associated referring expressions from RefCOCO [86]
dataset. Compared to baselines [12, 78, 58], only ART was able to accurately
predict the dynamic changes in gaze of humans incrementally hearing and
shifting their attention in response to the words in the referring expression,
even with incomplete target descriptions. Ablation studies showed that pre-
training and training on auxiliary grounding tasks, such as object
localization and target category prediction, improves ART’s gaze prediction
performance. In qualitative analyses, ART was shown to capture several
fixation patterns of people performing the incremental object referral task,
such as waiting, scanning, and verification, suggesting that it learned to
strategically disambiguate vision-language ambiguities.

In summary, our contributions are: (1) Introducing the incremental object
referral task for gaze prediction that will lead to more user-responsive HCI
systems. (2) Creating RefCOCO-Gaze, a large-scale dataset of gaze behavior
during the incremental object referral task. (3) Developing ART, the first
gaze prediction model of incremental object referral that offers computational
solutions to the incremental and multimodal aspects of our task. (4) Bringing
RefCOCO-Gaze and ART into the toolboxes of researchers studying incremental
object referral, thereby enabling them to understand how humans dynamically
merge their visual and linguistic information in the real world to control
their attention.

###  2 Related Work

Interest in gaze prediction as a computer vision problem has been growing [82,
12, 58, 83], given that the anticipation of user attention would enable more
natural augmented/virtual reality systems [61, 16, 7, 57]. Most existing human
attention prediction models predict free-viewing behavior [30, 54, 8], but
fail to generalize to goal-directed behaviors, such as visual search [27, 39].
More related is the work predicting eye fixations during the search for a
target object [88, 90]. Another study [82] predicted fixation scanpaths during
search using COCO-Search18 [14], a dataset of search fixations. Using a
dataset [11] of fixation scanpaths from a Visual Question Answering (VQA)
task, Chen _et al_. [12] proposed a model that predicted both VQA and search
behavior, and both Chen _et al_. [15], Yang _et al_. [84] proposed models to
predict both target-present and target-absent search. Mondal _et al_. [58]
proposed a multimodal transformer model called Gazeformer, which achieves
state-of-the-art search prediction performance while generalizing well to
unknown targets. Gazeformer [58] and Chen _et al_. [12] can be adapted for our
multimodal task, and serve as baselines in Sec. 5.

Despite this increasing interest in gaze prediction as a computer vision
problem, no existing model effectively addresses the incremental object
referral task. Large vision-language foundation models [64, 31, 85, 78, 87, 2]
yield unprecedented performance in visual, lingual and cross-modal tasks and
effectively generalize to new concepts and tasks. Moreover, several tasks
simply require multimodal modeling, such as VQA [5, 56], image captioning [23,
40], and object referral [86, 53, 81]. Relatedly, object referral (also known
as visual/object grounding of referring expressions) localizes or grounds a
single unambiguous object in an image that is referred to in a natural
language referring expression. Thus, the input is an image-text pair and the
output is the referred object’s bounding box parameters. Recent object
referral models have adopted two-stage [28, 29, 49] and one-stage [13, 20, 47]
architectures, and to this end, several high-quality object referral
benchmarks have been curated, such as ReferItGame [35], RefCOCO [86], RefCOCO+
[86], and RefCOCOg [53]. Researchers have also studied human attention as
people view an image and concurrently describe it [74, 73]. One study [63]
collected a dataset of spoken image descriptions where each word was visually
grounded by a mouse trace. He _et al_. [26] collected a dataset containing
fixations (recorded by an eye-tracker) synchronized with concurrently spoken
image descriptions. However, these studies specifically focused on spoken
description of the entire image and not object referral. Vasudevan _et al_.
[76] explored object referral for previously spoken referring expressions, and
did not predict human attention. Another study [75] on spoken object referral
in videos used human gaze and spoken referring expression as inputs. Zhang _et
al_. [91] collected a dataset of static gaze estimation heatmaps for non-
incremental referral. To our knowledge, we are the first to computationally
model human gaze and explore its interactions with vision and language in a
realistic incremental referral task.

Figure 2: Behavioral data collection in our incremental object referral
paradigm.

###  3 RefCOCO-Gaze Dataset

RefCOCO-Gaze is the largest dataset for studying human gaze behavior during an
incremental object referral task. It consists of 19,738 scanpaths that were
recorded while 220 participants with normal or corrected-to-normal vision
viewed 2,094 COCO [48] images and listened to the associated referring
expressions from the RefCOCO dataset [86]. RefCOCO was collected using the
ReferItGame [35] where players must construct efficient referring expressions
for another player to locate the correct object. RefCOCO mirrors real-life
speech which is known to contain elliptical and unstructured expressions [71,
69]. The gaze data, recorded by an EyeLink 1000 eyetracker, includes
information about the location and duration of each fixation, the bounding box
of the search target, audio recordings of the referring expressions, the
timing of the target word, and the synchronization between the spoken words
and the sequence of fixations (tells us which word triggered which fixations).
RefCOCO-Gaze covers a diverse range of linguistic and visual complexity,
making it an ideal dataset for researchers studying human integration of
vision and language, and HCI researchers alike.

Fig. 2 depicts the incremental object referral paradigm used for human gaze
collection. We have selected 2,094 image-expression pairs from the larger
RefCOCO dataset based on the target object size, image ratio, and sentence
complexity. Each participant performed ${\sim}100$ trials, yielding 10-16
scanpaths per image-expression pair. Participants were instructed to move
their gaze as quickly as possible to the target object that is being referred
to in a language expression played through a speaker. Each trial began with
drift correction (for accurate eye tracking) and presentation of an image. The
image was displayed for 200ms before the audio onset (too short a time for an
eye movement). The image remained visible until the participant pressed a
button to indicate that they found the target or until five seconds elapsed
following completion of the auditorily presented referring expression. At the
end of each trial, the correct bounding box location of the target object was
provided as feedback, followed by a survey asking whether the participant
indeed found and recognized the target. A target was present in each image. We
used a forced aligner [55], a tool for aligning speech with text, to
synchronize gaze movements with individual words of a referring expression.
This study had IRB approval.

We divided RefCOCO-Gaze into disjoint training and evaluation sets that
preserve the approximate proportion of training to evaluation data in RefCOCO.
The training set consisted of 1799 image-expression pairs (selected only from
the original RefCOCO train split), corresponding to 16,982 scanpaths.
Scanpaths from image-expression pairs from validation and test splits of
RefCOCO were randomly shuffled and split (1:2 ratio) to create disjoint
validation (92 image-expression pairs, 869 scanpaths) and test (203 image-
expression pairs, 1887 scanpaths) sets – both having a balanced distribution
of target categories. Dataset details (_e.g_., stimuli selection, gaze
recording, pre-processing, comparison with related datasets, etc.) and
extensive dataset analyses are in the supplement.

Figure 3: Attention in Referral Transformer (ART) Architecture. On each pass
after comprehending a new word, the model takes an image $I$ and tokens
$T_{j}$ of prefix $R_{j}$ of the referring expression as input and generates a
possibly empty sub-sequence of fixations based on previous fixation history
encoded by a fixation context encoder.

###  4 Gaze Prediction for Incremental Object Referral

Our goal is to predict fixations as a person progressively receives
information about the referred object through each word of the referring
expression that they are hearing. We design a novel multimodal transformer
architecture called Attention in Referral Transformer or ART for this task.
ART solves multiple problems that arise when adapting previous gaze prediction
models [58, 12] for the incremental object referral task, owing to its several
novel features that these models [58, 12] lack: (1) ART integrates an object
referral network into our gaze prediction framework and trains it on partial
expressions, whereas previous baselines either extract task guidance from a
frozen object referral model [33] trained on complete expressions [12], or
lack object grounding capabilities [58]. (2) ART integrates a novel fixation
prediction framework (absent in previous methods) that accommodates the
autoregressive prediction of any number of fixations triggered by a word –
zero or one or several, based on previous fixations encoded by a novel
fixation context encoder (in contrast to non-autoregressive Gazeformer [58]
which predicts fixations with the knowledge of the first fixation alone). (3)
ART is an object referral network with a scanpath prediction decoder module,
which allows us to pre-train the object referral network on a large-scale
object referral dataset [86] using object grounding objectives, thus
generalizing better despite training on a much smaller gaze dataset. ART also
jointly trains on the object grounding objectives along with the primary
fixation prediction objective. On the other hand, previous baselines either
rely on frozen object referral models [12] or lack an object grounding
subnetwork altogether [58] and are therefore limited to only training on the
fixation prediction objective alone.

####  4.1 Architecture

The overall architecture of ART is shown in Fig. 3. Since the understanding of
the referred object changes with each incoming word, we design ART to output a
possibly empty sequence of fixations (which we dub a “pack” of fixations)
triggered by a new word $w_{j}$ from the referring expression. We follow
single-stream vision-language model architectures [20, 44, 45, 46] to design a
multimodal transformer encoder module that encodes visuo-linguistic context
for object referral. A transformer decoder module generates embeddings for
incremental scanpath prediction conditioned by visuo-linguistic context from
the encoder and fixation history encoded by a fixation context encoder module.

Visual and Language Encoders. We use separate encoders for the vision and
language modalities. As in DETR [10], the visual encoder consists of a
ResNet-50 [25] backbone followed by a standard transformer encoder module
[77]. Given an image $I\in\mathbb{R}^{3{\times}H{\times}W}$, the visual
encoder generates patch embeddings $g_{vis}\in\mathbb{R}^{d_{vis}{\times}hw}$.
A language encoder (RoBERTa [51]) encodes a prefix
$R_{j}=\\{w_{1},\ldots,w_{j}\\}$ upon utterance of the $j^{th}$ word $w_{j}$
in a referring expression. $R_{j}$ is tokenized [66] to obtain
$T_{j}=\\{\texttt{[BOT]},t_{1},...,\texttt{[EOT]}\\}$, which is then processed
by the language encoder to yield language embedding sequence
$g_{lang}\in\mathbb{R}^{d_{lang}{\times}l_{lang}}$. Here, $l_{lang}$ is the
maximum number of tokens that can be processed by the language model; [BOT]
and [EOT] are the beginning-of-text and end-of-text tokens, respectively. To
predict fixations triggered before the first word has been spoken, the input
tokens to the language encoder is the sequence
$\\{\texttt{[BOT]},\texttt{[EOT]}\\}$, and to predict fixations after the
expression has ended, we append an additional [EOT] token to the tokenized
input. Existing baselines [58, 12] use ResNet-50 and RoBERTa, hence we choose
them for fair comparison.

Visuo-linguistic Transformer Encoder. In contrast to Gazeformer [58], which
utilized pooled text encodings from a frozen language encoder for task
description and risked losing word-level details, our method integrates both
image and linguistic tokens into a unified sequence of multimodal tokens, thus
allowing fine-grained word-level interactions between image and linguistic
tokens. We process this sequence through a visuo-linguistic encoder, which is
designed as a standard transformer encoder [77]. Given the set of patch
embedding vectors $g_{vis}$ and the sequence of language embedding vectors
$g_{lang}$ with different dimensionalities, as described above, we first
project them to an embedding space of the same dimension $d$ using modality-
specific projections to obtain $f_{vis}\in\mathbb{R}^{d{\times}hw}$ and
$f_{lang}\in\mathbb{R}^{d{\times}l_{lang}}$. We also introduce two learnable
$d$-dimensional embeddings $\texttt{BBOX},\texttt{TGT}$ which correspond to
the object localization and target category prediction tasks, respectively.
The input to the visuo-linguistic encoder is the concatenation of BBOX, TGT,
$f_{vis}$, and $f_{lang}$. The corresponding output is tensor
$f_{vlg}\in\mathbb{R}^{d{\times}(hw+l_{lang}+2)}$. We project $f_{vlg}[0]$
(corresponding to BBOX) to $\\{x_{b},y_{b},w_{b},h_{b}\\}$ where $x_{b},y_{b}$
are the coordinates of the upper-left corner, and $w_{b}$ and $h_{b}$ are the
width and height of the bounding box. We also use a linear layer along with
softmax to project $f_{vlg}[1]$ (corresponding to TGT) to the probability
distribution $\mathbf{Pr}_{\textit{TgtCat}}$ over all possible target
categories.

Per-Word Fixation Prediction Framework. We predict fixations (or absence
thereof) triggered by a new word of a referring expression containing $L$
words. Let the scanpath $\mathcal{S}$ for incremental object referral be a
sequence of packs of fixations. A pack
$\mathcal{P}_{j}=\\{(x_{i}^{j},y_{i}^{j})|i=0,1,\ldots\\}$ is an ordered
sequence of 2D fixations, triggered by the change in knowledge about the
referred object due to a new word $w_{j}$, for $j\in\\{1,..,L\\}$. While a
pack is usually spurred by a word, a pack of fixations $\mathcal{P}_{0}$ can
be triggered before the first word is spoken, similar to free-viewing
behavior. A pack of fixations $\mathcal{P}_{L+1}$ can also occur after the
referring expression has ended. A word may not inspire any fixations at all,
yielding a null pack $\mathcal{P}_{\phi}=\phi$. We also define a terminal pack
$\mathcal{P}_{TERM}$ which, like the null pack, does not contain any valid
fixations, and denotes the end of scanpath (EOS).

Fixation context encoder. We parameterize a fixation $\mathbf{p}^{k}_{i}$
using four parameters: $x$-location $x^{k}_{i}$, $y$-location $y^{k}_{i}$, the
pack number $k$ (i.e., the index of the pack the fixation belongs to), and the
within-pack index $i$ (which we call order). We use this parameterization to
capture the fixation context, which refers to the information of previous
fixations in the ongoing scanpath. We use a fixed 2D sinusoidal positional
embedding [10] to encode the spatial $x,y$ location to
$XY_{i}^{k}\in\mathbb{R}^{2d}$ and two fixed 1D sinusoidal positional
embeddings [77] to encode the pack number $j$ and the order $i$ to
$n_{i}^{k},o_{i}^{k}\in\mathbb{R}^{d}$ respectively. $XY_{i}^{k}$,
$n_{i}^{k}$, and $o_{i}^{k}$ are concatenated and projected to fixation
encoding $\mathbf{c}_{i}^{k}\in\mathbb{R}^{d}$. Hence, for a new word $w_{j}$,
we can construct an ordered sequence of $\mathbf{c}_{i}^{k}$ ($k<j,i=0,1,...$)
and zero-pad to maximum length $L_{\mathcal{C}}$ to obtain the fixation
context tensor $\mathcal{C}_{j}\in\mathbb{R}^{d{\times}L_{\mathcal{C}}}$.

Pack decoder. To obtain the current pack of fixations, we use a transformer
decoder module [77]. Let the input
$\mathcal{Q}=\\{q_{k}|k=1,\ldots,L_{\mathcal{P}}\\}$ to the decoder be a
sequence of pack queries $q_{k}$, where $L_{\mathcal{P}}$ is the maximum
number of fixations in a pack and $q_{k}$’s are learnable vectors (similar to
fixation queries [58]). To help the model differentiate the nature of context
and pack embeddings, we further add two separate segment embeddings [21],
namely $SEG_{ctxt}$ and $SEG_{curr}$ to previous fixation context tensor
$\mathcal{C}_{j}$ and $\mathcal{Q}$, respectively. Next, we use the
concatenation of $\mathcal{C}_{j}$ and $\mathcal{Q}$ as input to the decoder.
The decoder also receives dynamic visuo-linguistic context through cross-
attention with $f_{vlg}$. The output from the decoder is tensor
$f_{decoder}\in\mathbb{R}^{d{\times}(L_{\mathcal{C}}+L_{\mathcal{P}})}$. The
last $L_{\mathcal{P}}$ $d$-dimensional slices of $f_{decoder}$ corresponding
to the $L_{\mathcal{P}}$ pack queries are denoted as
$f_{pack}\in\mathbb{R}^{d{\times}L_{\mathcal{P}}}$.

Fixation Prediction Module. Fixation prediction for incremental object
referral is challenging since a pack can have between zero and multiple
fixations, and a scanpath can be terminated before the end of a referring
expression. We account for these scenarios by making all packs be of length
$L_{\mathcal{P}}$ with the following parameterization. First, any valid
fixation in a pack is represented by a fixation token FIX. Second, null packs
and packs having less than $L_{\mathcal{P}}$ valid fixations are padded with
padding tokens PAD to maximum length $L_{\mathcal{P}}$. Third, we complete a
terminal pack $\mathcal{P}_{TERM}$ with $L_{\mathcal{P}}$ termination tokens
EOS. For each of the $L_{\mathcal{P}}$ slices of $f_{pack}$, we use a token
prediction MLP and a softmax layer to predict if that slice corresponds to one
of FIX, PAD, and EOS tokens. We use regression heads and Gaussian
distributions [58] to model the fixation locations. We also augment ART with
fixation duration modeling and detail it in the supplement.

####  4.2 Pre-training, Training, and Inference

Pre-training. Since object grounding is at the core of our task, we pre-train
the visual, language and visuo-linguistic encoder modules on the two
objectives that we hypothesize underlie the object grounding process: object
localization and target category prediction, using RefCOCO [86] training data.
Object localization is the estimation of the referred object bounding box. We
apply an $L_{1}$ regression loss $\mathcal{L}_{reg}$ and a generalized IoU
(GIoU) loss [65] $\mathcal{L}_{giou}$, between predicted and ground truth
bounding box parameters. The target category prediction task discerns the
object type from the expression (e.g., predict “car” in the expression “left
sedan next to the motorcycle”). We pre-train on this task using a cross-
entropy loss $\mathcal{L}_{target}$. Total pre-training loss is
$\mathcal{L}_{pretrain}=\mathcal{L}_{reg}+\mathcal{L}_{giou}+\mathcal{L}_{target}$.

Training & Inference. We train ART using the teacher-forcing algorithm [79],
i.e., we provide the ground truth fixations to construct the fixation context
and treat each pack in the training scanpaths as independent minibatch items.
To train ART on the gaze prediction task, we apply $\mathcal{L}_{1}$
regression loss (following [58]) on the predicted $x$ and $y$ locations. Let
the predicted pack of fixations
$\mathcal{P}_{k}=\\{(x_{i}^{k},y_{i}^{k})\\}_{i=1}^{L_{\mathcal{P}}}$, and
ground-truth pack of fixations
$\hat{\mathcal{P}}_{k}=\\{(\hat{x}_{i}^{k},\hat{y}_{i}^{k})\\}_{i=1}^{l^{k}}$
where $l^{k}$ is the length of the ground truth pack. Moreover, let
$\hat{v}_{i,t}^{k}$ be a binary scalar representing ground truth of the
$i^{th}$ token in $\mathcal{P}_{k}$ belonging to the token class $t\in T\text{
where }T=\\{\texttt{FIX},\texttt{PAD},\texttt{EOS}\\}$. Also let $v_{i,t}^{k}$
be the probability of that token belonging to token class $t$ as estimated by
our model. The multitask loss for a minibatch of size $M$ is

$\displaystyle\mathcal{L}_{gaze}=\frac{1}{M}\sum_{k=1}^{M}\left(\mathcal{L}_{xy}^{k}+\mathcal{L}_{token}^{k}\right).$
(1)

Here
$\mathcal{L}_{xy}^{k}=\frac{1}{l^{k}}\sum_{i=1}^{l^{k}}\left(|x_{i}^{k}-\hat{x}_{i}^{k}|+|y_{i}^{k}-\hat{y}_{i}^{k}|\right)$,
$\mathcal{L}_{token}^{k}=-\sum_{i=1}^{L_{\mathcal{P}}}\sum_{t\in
T}\hat{v}_{i,t}^{k}\log(v_{i,t}^{k})$. In addition to the gaze-prediction loss
$\mathcal{L}_{gaze}$, we also train on the object localization and target
category prediction tasks, but only after either of the following two events
has occurred: (1) the last word of the referring expression has been uttered,
(2) the ground truth scanpath has been terminated. Note that both events
ensure sufficient information in the referring expression comprehended thus
far for a human to localize the object. This multi-task grounding loss
$\mathcal{L}_{ground}$ is $\mathcal{L}_{bbox}+\mathcal{L}_{target}$, where
$\mathcal{L}_{bbox}=\mathcal{L}_{reg}+\mathcal{L}_{giou}$. Hence, the total
multi-task loss $\mathcal{L}$ that we use to train our ART model is
$\mathcal{L}=\mathcal{L}_{gaze}+\mathcal{L}_{ground}$ when the scanpath has
terminated or the referral audio has ended, and
$\mathcal{L}=\mathcal{L}_{gaze}$ otherwise. During inference, ART
autoregressively generates packs of fixations conditioned on the previous
fixations generated by the model and the scanpath is terminated upon
encountering the first termination token EOS in a predicted pack. The
fixations within a pack are efficiently generated in parallel.

###  5 Experiments

Here, we experimentally evaluate scanpath prediction capability for
incremental object referral. For the conventional scanpath prediction task,
accurately predicting the entire sequence of fixations is the main objective.
However, for our task, it is perhaps equally important, if not more, for the
predicted scanpath to be correct at the word-level granularity, i.e., packs
(including null packs) must be predicted accurately. Following previous work
[82, 84, 12, 58], we sample 10 scanpaths per image-expression pair for all
models. More details of ART, such as its design and implementation details,
are in the supplement.

####  5.1 Performance Metrics

We use a broad set of metrics to evaluate dynamic word-based scanpath
prediction for incremental object referral. Sequence Score metric [82]
converts predicted and ground truth scanpaths into strings of fixation cluster
IDs and compares them using a string matching algorithm [60]. Fixation Edit
Distance [58] measures scanpath dissimilarity using the Levenshtein algorithm
[43] after converting scanpaths to strings like Sequence Score does. We
measure Sequence Score and Fixation Edit Distance in two granularities: (1)
over the entire scanpath ($SS$ and $FED$); and (2) over a pack ($SS_{pack}$
and $FED_{pack}$), where $SS_{pack}$ and $FED_{pack}$ are the averages of
sequence scores and fixation edit distances, respectively, between the ground-
truth and predicted packs. We also introduce $CC_{pack}$ and $NSS_{pack}$, the
word-based versions of the Correlation Coefficient (CC) [32] and Normalized
Scanpath Saliency (NSS) [62] metrics. CC is the correlation between the
normalized model saliency map and a Gaussian-convolved human fixation map. NSS
averages the values of a model’s fixation map at the locations fixated by
humans [9], and is a discrete version of CC. $CC_{pack}$ and $NSS_{pack}$ are
the averages of $CC$ scores and $NSS$ scores, respectively, over all possible
packs. Higher $SS$, $SS_{pack}$, $NSS_{pack}$, and $CC_{pack}$ values signify
more similarity between model-generated and human scanpaths, whereas lower
$FED$ and $FED_{pack}$ scores indicate higher similarity. More details are in
the supplement.

####  5.2 Baselines

We compare ART with: (1) Random Scanpath: We uniformly sample a pack length
value $l_{p}$ and then uniformly sample $l_{p}$ fixation locations from the
image. (2) OFA: We use the state-of-the-art vision-language model OFA [78],
trained on several multimodal benchmarks. We uniformly sample pack length
$l_{p}$ and then sample $l_{p}$ fixation locations within the OFA-predicted
bounding box for each referring expression prefix. (3) Chen _et al_. [12]:
This model learns goal-directed human gaze through a dynamically updated
memory which is initialized by task guidance maps. To extend this model to our
task, we create task guidance maps using bounding boxes from the SOTA referral
model MDETR [33], trained only on RefCOCO. (4) Gazeformer-ref: This baseline,
based on Gazeformer [58], takes expression prefixes as target information and
generates packs of fixations. (5) Gazeformer-cat: Since target category
information might get lost in the pooled linguistic embedding used by
Gazeformer-ref, we evaluate another variant of Gazeformer [58] called
Gazeformer-cat which takes the target category name estimated for an
expression prefix as input and treats the problem as categorical visual
search. The target category estimation of a prefix is done by a pre-trained
RoBERTa-based classifier. Find more details of the baselines in the
supplement.

Table 1: Performance of ART and baselines on RefCOCO-Gaze test set.  | $SS\bm{\uparrow}$ | $SS_{pack}\bm{\uparrow}$ | $FED\bm{\downarrow}$ | $FED_{pack}\bm{\downarrow}$ | $CC_{pack}\bm{\uparrow}$ | $NSS_{pack}\bm{\uparrow}$
---|---|---|---|---|---|---
Human | 0.400 | 0.317 | 6.573 | 1.278 | 0.283 | 3.112
Random | 0.189 | 0.133 | 17.735 | 3.005 | 0.094 | 1.689
OFA [78] | 0.216 | 0.170 | 17.084 | 2.901 | 0.174 | 2.175
Chen _et al_. [12] | 0.299 | 0.188 | 8.309 | 1.507 | 0.159 | 1.557
Gazeformer-ref [58] | 0.269 | 0.194 | 6.788 | 1.286 | 0.208 | 3.006
Gazeformer-cat [58] | 0.269 | 0.189 | 6.841 | 1.327 | 0.204 | 2.932
ART (Proposed) | 0.359 | 0.292 | 6.371 | 1.143 | 0.280 | 3.478

Human ART (proposed) Chen _et al_. [12] Gazeformer-ref [58]

Human ART (proposed) Chen _et al_. [12] Gazeformer-ref [58]

Human ART (proposed) Chen _et al_. [12] Gazeformer-ref [58]

Figure 4: Qualitative results. Scanpaths from humans and three scanpath
prediction models on three trials exhibiting strategic fixation behavior.
Fixations (denoted by circles numbered with fixation order) are color-coded to
corresponding words in the referring expression (above each row). Fixations
color-coded to [BOT] occurred before the expression started, and those color-
coded to [EOT] occurred after the expression ended. Blue bounding boxes
indicating the referred objects are not visible during trials. Our model
generates the most human-like scanpaths for incremental object referral.

####  5.3 Results

We train ART and the baselines on the RefCOCO-Gaze training set and evaluate
them on the test set. Results are in Table 1. ART outperforms baselines on all
metrics by significant margins. We hypothesize that ART performs best because
it includes an object referral model and jointly trains on grounding and gaze
prediction objectives. In contrast, Chen _et al_. [12] use a frozen MDETR
model trained only on complete RefCOCO expressions, and the Gazeformer
variants (Gazeformer-ref and Gazeformer-cat) lack grounding subnetworks to
train on object grounding. Hence, both baselines are unable to learn the
auxiliary grounding tasks on partial expressions. Interestingly, despite using
no spatial and attribute information, Gazeformer-cat is almost as predictive
as Gazeformer-ref, underscoring the importance of target category estimation.
Note that these analyses focused on spatial attention, but in the supplement,
we show that ART also outperforms baselines in terms of fixation duration
prediction as well. We also show in the supplement that ART generalizes well
to categorical search when trained and evaluated using COCO-Search18 [14]
dataset.

Fig. 8 shows qualitative results comparing the sequences of fixations from ART
and the baselines to the behavioral data. In these three examples, ART finds
the referred object and generates efficient, human-like scanpaths. ART also
exhibits several strategic fixation patterns that we observe in the human
data. In the top row, ART waits near the center until after getting the word
“right”, which conveys information about the referred sheep. A scanning gaze
pattern appears in the second row, where both the person and ART scan multiple
bags, thereby enabling the correct one to be located when the disambiguating
information arrives at the end of the expression. The third row exemplifies
verification. ART successfully finds the correct target on fixation #3 after
input of the word “girl”, but then makes another fixation (#5) to the girl in
the center after getting the word “pink”, presumably to verify which of the
girls is pinker before returning to the one on the left on the next fixation
(#6).

####  5.4 Ablation Studies

Table 2: Ablation studies on ART model. If either $\mathcal{L}_{bbox}$ or $\mathcal{L}_{target}$ is included, the loss is applied in both pre-training and training phases. Ablation # | Pre-training | $\mathcal{L}_{bbox}$ | $\mathcal{L}_{target}$ | $SS\bm{\uparrow}$ | $SS_{pack}\bm{\uparrow}$ | $CC_{pack}\bm{\uparrow}$
---|---|---|---|---|---|---
1 | $\times$ | $\times$ | $\times$ | 0.309 | 0.257 | 0.222
2 | $\checkmark$ | $\checkmark$ | $\times$ | 0.321 | 0.279 | 0.239
3 | $\checkmark$ | $\times$ | $\checkmark$ | 0.292 | 0.260 | 0.216
4 | $\times$ | $\checkmark$ | $\checkmark$ | 0.304 | 0.257 | 0.215
5 | $\checkmark$ | $\checkmark$ | $\checkmark$ | 0.359 | 0.292 | 0.280

We performed a number of ablations (in Table 2) on ART to probe the effects of
pre-training and inclusion of grounding losses on its performance, which we
evaluate using the RefCOCO-Gaze test set. As evidenced by comparing Ablations
4 and 5, pre-training significantly improves performance. We also observe that
without pre-training, the grounding losses $\mathcal{L}_{bbox}$ and
$\mathcal{L}_{target}$ slightly degrade performance (compare Ablations 1 and
4). When trained from scratch on the small gaze dataset, these auxiliary tasks
introduce noise to the optimization of the gaze prediction objective using
$\mathcal{L}_{gaze}$. Including one of $\mathcal{L}_{bbox}$ (Ablation 2) and
$\mathcal{L}_{target}$ (Ablation 3) losses in pre-training and training does
not improve performance significantly (although $\mathcal{L}_{bbox}$ seems to
be more beneficial than $\mathcal{L}_{target}$), whereas including both
$\mathcal{L}_{bbox}$ and $\mathcal{L}_{target}$ (main model and Ablation 5)
yields the best performance. This demonstrates that both object localization
and target category estimation tasks are integral to the object referral
process. We note that even when ART is not pre-trained on RefCOCO (Ablations
1, 4), it still outperforms baselines like the two Gazeformer variants that
are also not pre-trained on RefCOCO. See the supplement for more metrics,
ablations and analyses.

###  6 Conclusions and Discussion

How do humans integrate vision and language information to guide their
attention to target goals? To study this question we introduced the
incremental object referral task, a naturalistic version of an object referral
task in which people must incrementally integrate the visual information that
they are actively collecting with each location that they fixate in the image,
with the language information that they are hearing about the target object.
Our task therefore provides an experimental context for studying how humans
use the spoken language of another to dynamically control the visual
information that they sample from the world. We also introduced a model that
we call ART that similarly generates sequences of gaze fixations that occur as
this vision-language integration is happening. ART has a multimodal
transformer architecture that it uses to learn how to incrementally generate
packs of fixations for each word in the referring expression. To provide the
human behavior needed to train ART, we collected a high-quality and large-
scale dataset called RefCOCO-Gaze. We trained and evaluated ART and several
competitive baselines on RefCOCO-Gaze and found that ART outperformed other
baselines by significant margins on multiple metrics. We also performed
extensive ablation analyses to show how pre-training ART on RefCOCO, and the
addition of auxiliary grounding losses, significantly contributed to its
superior performance. Qualitative analysis revealed that ART showed human-like
effects of visual and linguistic target ambiguity on its attention behavior
through higher-level strategic forms of integrating vision and language
information, expressed as distinct waiting, verification, and scanning
strategies. We believe that ART will be instrumental for predicting gaze in
time-sensitive, voice-assisted HCI applications (especially AR/VR) where
predicting future eye movements will enable seamless human-computer
interactions.

A current limitation of ART is that it treats the referring expression as text
and not audio, thereby ignoring the phonological factors influencing vision-
language disambiguation and attention control. Future work will explore
representing these phonological factors as well.

Acknowledgements. This project was supported by US National Science Foundation
Award IIS-1763981, IIS-2123920, DUE-2055406, and the SUNY2020 Infrastructure
Transportation Security Center, and a gift from Adobe.

## Supplementary Material

In the supplementary material provided below, we present additional
experiments, visualizations and details of our work on gaze prediction during
incremental object referral task using the Attention in Referral Transformer
(ART) model and RefCOCO-Gaze dataset. The specific sections are as follows:

  * •

We provide details about how we selected the stimuli, i.e. the images and
referring expressions, from RefCOCO dataset to create RefCOCO-Gaze dataset.
(Section I).

  * •

We discuss the gaze recording method we used to collect human fixations for
RefCOCO-Gaze dataset along with analysis of the collected gaze data (Section
II).

  * •

We provide a comparative analysis of our proposed RefCOCO-Gaze dataset and
other related gaze datasets discussed in the main text (Section III).

  * •

We provide details of various components of the ART model along with the pre-
training and training procedures (Section IV).

  * •

We provide implementation details of several scanpath metrics used for
evaluation in the main text (Section V).

  * •

We augment ART with fixation duration prediction capability and report the
experimental results for ART model and other baselines on RefCOCO-Gaze with
respect to both fixation location and fixation duration prediction (Section
VI).

  * •

We show that ART generalizes to categorical search task when trained and
evaluated using COCO-Search18 [14] dataset (Section VII).

  * •

We provide implementation details of the baselines - Random Scanpath, OFA
[78], Chen  _et al_. [12], Gazeformer-ref [58], and Gazeformer-cat [58]
(Section VIII).

  * •

We augment the experimental results for the ablation studies on ART model,
which are discussed in the main text, with additional metrics (Section IX).

  * •

We present additional ablation studies investigating the effects that the
components of ART model have on performance and also include additional
analysis on the ablations (Section X).

  * •

We explore the effects of next word token prediction task on the performance
of ART model (Section XI).

  * •

We present additional qualitative examples of scanpaths generated by human
participants, our ART model and other competitive baseline models. (Section
XII).

###  I Image and Referring Expression Selection Details

We utilized a subset of the RefCOCO dataset [86] (the original UNC split) to
create our dataset. RefCOCO dataset consists of referring expressions
collected for 50,000 target objects present in 19,994 COCO [48] training
images. RefCOCO was carefully curated such that each image contained at least
two objects of the same object category as the target object. To ensure the
reliability of our gaze data, we selected the longest referring expression
amongst the multiple referring expressions collected for each target object.
To eliminate stimuli that might produce inaccurate gaze patterns due to low
quality or extreme difficulty, we further refined our dataset by excluding
images and referring expressions that did not meet the following criteria. For
detailed examples of such exclusions, please refer to Fig. 5.

  * •

Target Size: We excluded data where the size of the target object, as measured
by the area of its bounding box, was larger than 10% of the total image area.

  * •

Image Ratio: We excluded data with images whose width-to-height ratios were
outside the range of 1.2-2.0 (based on a screen ratio of 1.6). We did this to
eliminate very elongated images, which might distort normal viewing behavior.

  * •

Sentence Complexity: We excluded data where the referring expression sentence
was either too simple or too complex. We measured sentence complexity using
the metric introduced by Liu et al. [50], which is correlated with sentence
length and frequency. Specifically, we excluded data where the referring
expression language complexity was below the 10th percentile (e.g., "the
girl") or above the 90th percentile (e.g., "second row from left to right
second one up from bottom…"). This ensured that the length of the remaining
referring expressions ranged from 2 to 10 words, with a median of 4. The
original dataset had a wider range of sentence lengths, from 1 to 39 words,
with a median of 4.

After applying the aforementioned exclusion criteria, we were left with 7,568
referring expressions from 72 categories. However, this number exceeded our
resources for gaze data collection. Therefore, we decided to further trim the
dataset while maintaining a balanced distribution of target categories. To do
so, we removed entire categories if the application of the exclusion criteria
left fewer than 100 referring expressions per category. Then, we randomly
selected up to 150 data points per category. This process resulted in a
dataset of 2,422 referring expressions from 18 categories. Finally, we
conducted a manual exclusion process to remove any referring expressions or
images containing obscene, inappropriate, or irrelevant content (e.g., blood,
nudity, slang). We also manually removed any data points with spelling
mistakes, or incorrect or poor target descriptions. In total, 328 data points
were removed during this manual exclusion process, yielding 2,094 image-
expression pairs for the dataset.

Figure 5: Excluded Samples Figure 6: Stimuli Statistics. A: Spatial
distribution of target locations B: Temporal distribution of target word onset
Figure 7: Descriptive statistics for RefCOCO-Gaze images and referral
expressions.

###  II Gaze Recording Methods

####  II.1 Participants

Our dataset was collected from 220 participants, consisting of 155 male, 63
female, and 2 non-binary individuals. Participants were undergraduate students
from our institution who were recruited for extra credit in a psychology class
and had normal or corrected-to-normal vision. The age range of participants
was between 18 and 33 years. Among the participants, 28 were non-native
English speakers but rated their fluency level as either very good or good.
This study had Institutional Review Board (IRB) approval.

####  II.2 Stimuli

Images were resized and padded to fit to the computer screen size and
resolution (1680$\times$1050 pixel resolution). Fig. 6A displays the spatial
distribution of target locations in the images, which are evenly distributed
but with slightly higher probability around the center-bottom area. Spoken
referring expression were generated using Google Text to Speech
API111https://pypi.org/project/google-cloud-texttospeech/ commonly known as
the gTTS available in Python. Fig. 6B shows the temporal distribution of
target word onset, which was measured by the timing of the target word from
generated audio. The original dataset did not provide the target word (i.e.,
the word in the sentence that refers to the target object). Therefore, we
manually annotated the target word for each referring expression using the
consensus of two annotators. Example target words are provided in Table 3. The
target word is usually referred to at the beginning of the sentence (median
time of 0.4 seconds), and the total duration of the audio mostly ranges from 1
to 3 seconds, with a median of 1.7 seconds.

Our dataset closely matches the distribution of the original RefCOCO dataset
(panels A and B of Fig. 7), capturing the distributions of target clutter
(i.e., the number of instances of the same target category in the image) and
sentence complexity (as measured by Liu _et al_. [50]). Fig. 7C shows the
category distribution of target objects in the dataset. These categories were
well-balanced and span a wide semantic range, from animate objects (e.g.,
person, sheep) and indoor objects (e.g., chair, cup) to outdoor vehicles
(e.g., car, truck).

Table 3: Target Names Category |  Target Names Used for Referral
---|---
person |  coach, figure, pic, jeans, player, girl, arm, row, thingy, shirt, girls, kid, hat, red, skier, picture, head, woman, left bottom, hand, slider, child, lady, man, boy, catcher, jacket, person, red and black, green, guy
elephant |  rump and tail, ear, elephant butt, butt, one, camera, animal, elephant, baby, corner, legs
sheep |  sheep, goat or sheep, cow, area, animal butt, animals face, leg, ship, one, lamb, sheep butt, animal, guy, face, goat, calf
cow |  brown, cow, leg, camel, one, band, animal, bull, cows, corner, critter, legs, goat, calf
bus |  trolley, double decker, van, phone, bus, ride, deck, decker, glass, train, truck, vehicle, thing, rectangle, car
car |  taxi, benz, van, reflection, area, car, screen, mirror, police, suv, truck, vehicle, cab, ford, black
truck |  hummer, area, van, semi, suv, door, car, vehicle, firetruck, bus, fedex, machine, thing, part, truck, item, fire truck, corner, rig, tarp, trailer
couch |  loveseat, armchair, frame, seat, corner, ottoman, chair, love seat, thingy, couch, orange, seat cushion, pillows, leg rest, cushion, table, furniture, footstool, chairs, sofa, bed, thing, pillow
chair |  woven, center, thing, chairs, tray, object, couch, jacket, item, bench, pattern, corner, chair, seat, lady, seat cushion
tv |  laptop, monitor, tablet, tv, poster, screen, tv screen, bruce lee, face, computer, monitor screen, desktop, girl, sign, spot, computer screen
suitcase |  case, box, container, briefcase, luggage, area, space, bag, black, item, suitcase, corner, chair, thing, trunk
bowl |  right, bananas, cup and spoon, pot, dish, corner, thing, row, container, cup, things, section, bowl, butter, sauce, pan, kiwi, plate, food, left bowl, grapes, tuna, chips, broccoli, pottery piece, hot dog, fruit slices, soup, stuff, apples, dip
cup |  tea, pot, whatever, one, juice, second, frosty, dish, candle, milk, blender, container, cup, section, jar, mug, beer, coffee, coke, drink, pitcher, toothbrush, glass, water, thing, stuff, bottle
donut |  plate, sprinkles, item, food stuff, donuts, food, cheerio, donut, chocolate ice, bun, pastry, dessert, corner, skewer, doughnut, thing, striped, row
cake |  muffin, pie, one, pastry, corner, thing, ice cream, row, item, pile, orange, hat, roll, plate, umbrella, food, dessert, brownie, cupcake, cake, fruit, cookie, frosting, chocolate, bread, center, biscuit, train car, cake slice
sandwich |  taco, waffle, burger, ball, pastry, wrap, sammy, toast, flower, sub, bowl, palte, roll, plate, sandwich, food, appetizer, bun, half, banana slices, bread, meat, thing, piece
orange |  one, slice, apple, left, thing, orange slice, bowl, oranges, pieces, lime, grapefruit, fruit, food, lemons, front, orange, lemon, stem, egg, row
broccoli |  broccoli piece, broccoli pieces, greens, spinach, food, blur, veggie, patch, thing, piece, green, goop, basket, piece of broccoli, broccoli

####  II.3 Procedure and Apparatus

Gaze was recorded using the EyeLink 1000 eye-tracker (SR Research Ltd.,
Ottawa, Ontario, Canada) and the data were exported using the EyeLink Data
Viewer software package (also from SR Research Ltd.). During the experiment,
the presentation of images was controlled using Experiment Builder software
(SR Research Ltd., Ottawa, Ontario, Canada). The stimuli were displayed on a
22-inch LCD monitor, positioned at a viewing distance of 47cm from the
participant, with the help of chin and head rests. This resulted in a
horizontal and vertical visual angle of 54∘ $\times$ 35∘, respectively. At the
beginning of each trial, participants were instructed to fixate on a central
point but were free to move their eyes while searching for the target. Eye
movements were recorded throughout the experiment using the EyeLink 1000 eye-
tracker in tower-mount configuration. Prior to each block or whenever
necessary, the eye-tracker was calibrated using a 9-point calibration method,
and the calibration was not accepted unless the average calibration error was
below 1.0∘ and the maximum error was below 1.5∘. The experiment was conducted
in a quiet laboratory room under dim lighting conditions. All responses were
recorded using Microsoft Game controller triggers. The following instructions
were provided to the participants prior to the gaze data collection process:

“We wish to observe your natural eye-movement behavior while searching for a
referred target. You will be shown 100 images with spoken referring
expressions describing the target’s location and appearance. Your job is to
find a target AS QUICKLY AND ACCURATELY AS POSSIBLE. When you find a target,
please press any button on the top side of the controller. We will analyze
your gaze later and measure accuracy by checking whether your gaze land on the
target correctly at the time you press. So please make sure you press the
button WHILE you are looking at the target. Please press the button as soon as
you find the referred target. You can browse each image up to 5 seconds after
the sound ends. There will be a break around halfway through the experiment,
but if you need an additional break during the experiment, let the
experimenter know anytime.”

####  II.4 Preprocessing

Fixations were detected from raw gaze samples using the EyeLink online parser,
which applied velocity and acceleration thresholds of 30${}^{\circ}/s$ and
8000${}^{\circ}/s^{2}$, respectively. Fixations with a duration lower than
60ms were filtered out, but all other fixations were retained. The initial raw
dataset consisted of 21,898 valid scanpaths. However, to ensure data
reliability, we removed trials where participants did not find the target
within a given time limit of 5 seconds or reported not finding the target in
the survey. We also eliminated trials where any of the participant’s fixations
did not land within the target bounding box, resulting in the removal of 10%
of the entire dataset and leaving us with 19,738 scanpaths. Additionally, we
observed that 6% of trials had the participant’s final fixation not within the
target area, which may have been due to them moving away from the target as
they pressed the button. To address this, we trimmed the fixations up to the
last fixation that landed on the target, thereby ensuring that only fixations
relevant to the target search were included in the analysis.

####  II.5 Gaze Data Analysis for RefCOCO-Gaze

RefCOCO-Gaze fixations are intention-driven. As can be seen by comparing the
top two rows in Table 1 from the main text, the inter-observer agreement
metrics (row 1, labeled “Human”) far exceed the Random baseline metrics (row
2). Based on this observation, and findings of previous behavioral work [70,
80] suggesting that high inter-observer agreement mark similar task-driven
attention allocation across individuals, we infer that the fixations in
RefCOCO-Gaze are not random but rather, intention-driven and under attention
control.

Target Localization analysis. Analysis of the gaze data collected in our
incremental object referral task revealed that on 9.76% of the trials,
participants failed to either fixate on the correct target or to localize the
target within the 5 second limit. Mean saccade amplitudes for successful and
failed localizations were 192.948 (standard deviation=75.86), and 191.662
(standard deviation=78.94), respectively. On the other hand, mean fixation
durations (in msecs) for successful and failure localizations were 280.741
(standard deviation=114.38) and 277.432 (standard deviation=126.68),
respectively. As is evident, the gaze statistics of average saccade amplitude
and average fixation duration did not significantly differ between the
successful localizations and failed localizations (T-test revealed
p-value$=$0.437 for average saccade amplitude, and p-value$=$0.247 for average
fixation duration – both not statistically significant since p-value$>$0.05).
However failure cases yielded statistically significant longer scanpaths
(average of 10 fixations for failure, 8 for success; p-value$<$0.05). Failure
cases also showed strong positive correlations with scene complexity (Pearson
correlation coefficient $r$=0.81) in terms of object instance count in a
scene, and referral language perplexity (Pearson correlation coefficient
$r$=0.76). These complexity scores tend to be higher for failure cases than
for successful ones, suggesting that search performance decreases with
increasing scene complexity and linguistic complexity of the referring
expressions.

We also note that in 12.52% of the trials, observers fixated on the target
during exposure time. Yet, for these trials, search ended after a median of 5
words, implying that observers required ample description for confident
localization.

###  III Comparison of RefCOCO-Gaze with other gaze datasets

Here, we compare related datasets discussed in Related Work section (Section 2
in the main text) in the table below. Our proposed dataset, RefCOCO-Gaze is
the only gaze scanpath dataset for the incremental object referral task.

Dataset |  Apparatus |  Task |  Gaze recorded [during/after] task description |  Stimuli |  No. of scanpaths |  No. of Subjects |  Relevant w.r.t. Object Referral |  Relevant w.r.t. Incremental Prediction
---|---|---|---|---|---|---|---|---
|  |  |  |  |  |  |  |
COCO-Search18 [14] |  Eye-tracker |  Categorical Visual Search |  After |  Images |  299037 |  10 |  No |  No
AiR [11] |  Eye-tracker |  VQA |  After |  Images |  13173 |  20 |  No |  No
Localized Narratives [63] |  Mouse proxy |  Image Captioning |  During |  Images |  848749 |  156 |  No |  Yes
He  _et al_. [26] |  Eye-tracker |  Image Captioning |  During |  Images |  14000 |  16 |  No |  Yes
SNAG [74, 73] |  Eye-tracker |  Image Description |  During |  Images |  3000 |  30 |  No |  Yes
OR [75] |  Face videos |  Object Referral |  After |  Videos |  30000 |  20 |  Yes |  No
Zhang  _et al_. [91] |  Gaze Following |  Object Referral |  After |  Images |  $-^{*}$ |  - |  Yes |  No
RefCOCO-Gaze(ours) |  Eye-tracker |  Incremental Object Referral |  During |  Images |  19738 |  220 |  Yes |  Yes

${*}$ Zhang  _et al_. [91] collect 40000 static gaze heatmaps, not
spatiotemporal gaze scanpaths

###  IV Additional Details of ART

In this section, we share additional details about the ART model, such as
details of implementation, architectural design, and hyperparameter choices
for our experiments on RefCOCO-Gaze.

####  IV.1 Visual Encoder and Language Encoder

For designing the visual encoder, we use an ImageNet [19] pre-trained
ResNet-50 [25] backbone followed by a transformer encoder consisting of 6
standard transformer encoder layers [77] with hidden size $d_{vis}=256$ and 8
attention heads. A dropout of 0.1 was applied to the transformer encoder
layers. The output of the visual encoder is patch embedding tensor
$g_{vis}\in\mathbb{R}^{d_{vis}{\times}hw}$, corresponding to $h\times w$ grid,
where $h=10,w=18$. For the language encoder, we use the RoBERTa-base variant
[51] which generates embeddings of dimension $d_{lang}=768$ for each token in
the tokenized text string. The hyperparameter $l_{lang}$ is set to 32. RoBERTa
encodes text tokenized using a Byte-Pair Encoding (BPE) [66]. RoBERTa is pre-
trained on a large corpus of English data (which includes the BookCorpus [92],
English Wikipedia data, the English portion of CommonCrawl News dataset [59]
called CC-News, OpenWebText [24] and STORIES [72]) using a Masked Language
Modeling (MLM) objective with a dynamic masking scheme. As mentioned in the
main text, we specifically use ResNet-50 and RoBERTa backbones for fair
comparison because they form the backbones of our baselines Chen _et al_. [12]
and Gazeformer [58] variants. Both visual and language encoders are trainable,
and not frozen as in Gazeformer [58].

####  IV.2 Visuo-linguistic Transformer Encoder

For our experiments on RefCOCO-Gaze, ART’s visuo-linguistic encoder consists
of 6 standard transformer encoder layers [77] with hidden size ($d$) 256 and 8
attention heads each. A dropout of 0.1 was applied to all transformer layers
in this module. For the bounding box regression and target category prediction
heads, a dropout of 0.3 was applied during the pre-training phase while a
dropout of 0.2 was applied during the training phase. To deal with scale
variation, we normalize the parameters of of ground truth bounding boxes and
consequently apply sigmoid activation to the bounding box regression head.

####  IV.3 Pack Decoder & Fixation Prediction

For our experiments on RefCOCO-Gaze, ART’s pack decoder module consists of 6
transformer decoder layers [77] with hidden size($d$) 256 and 8 attention
heads. A dropout of 0.2 was applied for all transformer decoder layers in this
module. For the fixation prediction heads in the fixation prediction module, a
dropout of 0.4 was applied. We choose hyperparameters $L_{\mathcal{P}}$ and
$L_{\mathcal{C}}$ to be 6 and 36 respectively. Spatial location estimation was
done by regressing parameters (i.e. mean and log-variance) of two separate
Gaussian distributions using 4 regression heads (two heads each for the two
Gaussian distrbutions - one head for estimating mean and the other head for
estimating log-variance) in the fixation prediction module. These Gaussian
distributions model the x and y co-ordinates (raw unnormalized pixel co-
ordinates) of fixations [58]. The spatial locations are sampled from the
Gaussian distributions using the reparameterization trick [37]. The range of
the predicted unnormalized fixation location (x and y) co-ordinates are the
respective image dimensions. We do not involve the pack decoder module and the
fixation prediction module in the pre-training stage.

####  IV.4 Pre-training & Training

To deal with scale variation, we normalize the parameters of the ground truth
bounding boxes during pre-training and training of the bounding box head. We
use AdamW [52] optimizers for our pre-training and training phases with weight
value 1e-4. During the pre-training process, the visual encoder, the language
encoder and the visuo-linguistic encoder are all assigned learning rates of
1e-5. During the training process, the visual encoder and the language encoder
are both assigned learning rates of 1e-7 while the visuolinguistic encoder is
assigned a learning rate of 1e-5, while the rest of the ART model is assigned
a learning rate of 1e-4. We pre-train on the RefCOCO training set for 200
epochs with a batch size of 128 and train on RefCOCO-Gaze training set with a
batch size of 64 for a maximum of 200 epochs. Note that the visual, language
and visuo-linguistic encoders are trainable (not frozen) during the pre-
training stage, and all components of the ART model are trainable (not frozen)
during the training stage. We ran our experiments on NVIDIA RTX A5000 GPUs.

###  V Additional Details of Metrics

In this section, we provide additional implementation details of the scanpath
metrics.

SS. This metric is the sequence score between the ground truth and predicted
scanpaths over the entire referring expression. Hence, this metric considers
only the valid 2D fixation locations.

$\textbf{SS}_{pack}$. We might encounter two edge cases while calculating
$SS_{pack}$ \- either (1) the predicted pack is a null pack or (2) the ground
truth pack is a null pack, with both scenarios resulting in empty strings
which hinder direct application of string matching algorithm [60]. We handle
the first scenario by duplicating the last fixation of previous non-null
predicted pack (initial central fixation point in case there are no previous
fixations) and handle the second scenario by duplicating the last fixation of
previous non-null ground truth pack (initial central fixation point in case
there are no previous fixations) - similar to the process for calculating
ScanMatch with duration [18]. Similarly, we also duplicate the last 2D
fixation when one of ground truth scanpath or predicted scanpath has
terminated and the other one has not.

$\textbf{CC}_{pack}$. For our implementation of $CC_{pack}$, we add a small
$\epsilon=1e^{-9}$ to the ground truth and predicted maps to avoid a divide-
by-zero error for cases where either the ground truth map or the prediction
map is a zero map due to a null pack.

$\textbf{NSS}_{pack}$. We disregard cases where either ground truth or
predicted pack is a null pack while calculating the average for $NSS_{pack}$.
This is because there is no theoretical upper or lower bound of NSS that can
be assigned to scenarios where either one or both of ground truth and
predicted packs are null packs (resulting in zero action/saliency maps).

###  VI Fixation Duration Prediction with ART

Table 4: Performance of ART and baselines on RefCOCO-Gaze test set when
trained and evaluated on both fixation location and fixation duration
prediction tasks.

(a) Duration-agnostic metrics
$SS\bm{\uparrow}$ $SS_{pack}\bm{\uparrow}$ $FED\bm{\downarrow}$
$FED_{pack}\bm{\downarrow}$ $CC_{pack}\bm{\uparrow}$ $NSS_{pack}\bm{\uparrow}$
Human 0.400 0.317 6.573 1.278 0.283 3.112 Random 0.189 0.133 17.735 3.005
0.094 1.689 OFA [78] 0.216 0.170 17.084 2.901 0.174 2.175 Chen _et al_. [12]
0.281 0.255 6.825 1.163 0.209 1.953 Gazeformer-ref [58] 0.261 0.187 6.833
1.307 0.197 2.882 Gazeformer-cat [58] 0.244 0.172 7.144 1.394 0.194 2.664 ART
(Proposed) 0.356 0.285 6.410 1.161 0.281 3.539

(b) Duration-aware metrics
$SS^{(t)}\bm{\uparrow}$ $SS_{pack}^{(t)}\bm{\uparrow}$
$FED^{(t)}\bm{\downarrow}$ $FED_{pack}^{(t)}\bm{\downarrow}$
$MM_{dur}\bm{\uparrow}$ Human 0.379 0.215 38.153 8.204 0.589 Random 0.169
0.097 108.296 18.395 0.688 OFA [78] 0.206 0.124 103.347 17.868 0.688 Chen _et
al_. [12] 0.272 0.157 42.058 8.224 0.633 Gazeformer-ref [58] 0.236 0.166
39.104 7.131 0.617 Gazeformer-cat [58] 0.224 0.161 39.937 7.216 0.519 ART
(Proposed) 0.332 0.199 35.997 7.120 0.696

ART is also capable of predicting the fixation durations of humans. We model
fixation durations as Gaussian distributions, similar to how we model fixation
locations. First, we reparameterize a fixation $\mathbf{p}^{k}_{i}$ using five
parameters: $x$-location $x^{k}_{i}$, $y$-location $y^{k}_{i}$, fixation
duration $d^{k}_{i}$, the pack number $k$ (i.e., the index of the pack the
fixation belongs to), and the within-pack index $i$ (which we call order). We
then add two fixation duration regression heads (along with the already
existing fixation location regression heads) to the fixation prediction module
to estimate parameters (i.e., mean and log-variance) of a Gaussian
distribution modeling fixation durations. Fixation durations $d_{i}^{k}$ are
sampled from this Gaussian distribution using the reparameterization trick
[37]. Let the predicted pack of fixations
$\mathcal{P}_{k}=\\{(x_{i}^{k},y_{i}^{k},d_{i}^{k})\\}_{i=1}^{L_{\mathcal{P}}}$,
and ground-truth pack of fixations
$\hat{\mathcal{P}}_{k}=\\{(\hat{x}_{i}^{k},\hat{y}_{i}^{k},\hat{d}_{i}^{k})\\}_{i=1}^{l^{k}}$
where $l^{k}$ is the length of the ground truth pack. Moreover, let
$\hat{v}_{i,t}^{k}$ be a binary scalar representing ground truth of the
$i^{th}$ token in $\mathcal{P}_{k}$ belonging to the token class $t\in T\text{
where }T=\\{\texttt{FIX},\texttt{PAD},\texttt{EOS}\\}$. Also let $v_{i,t}^{k}$
be the probability of that token belonging to token class $t$ as estimated by
our model. For accomodating the additional fixation duration prediction
objective, an $L_{1}$ loss $\mathcal{L}_{d}^{k}$ between ground-truth fixation
durations $\hat{d}_{i}^{k}$ and predicted fixation durations $d_{i}^{k}$ is
added to the formulation of $\mathcal{L}_{gaze}$ in Equation 1 of the main
paper. Hence, upon accounting for fixation duration prediction along with
fixation location prediction, $\mathcal{L}_{gaze}$ for a minibatch of size $M$
now becomes:

$\displaystyle\mathcal{L}_{gaze}=\frac{1}{M}\sum_{k=1}^{M}\left(\mathcal{L}_{xy}^{k}+\mathcal{L}_{token}^{k}+\mathcal{L}_{d}^{k}\right).$
(2)

Here
$\mathcal{L}_{xy}^{k}=\frac{1}{l^{k}}\sum_{i=1}^{l^{k}}\left(|x_{i}^{k}-\hat{x}_{i}^{k}|+|y_{i}^{k}-\hat{y}_{i}^{k}|\right)$,
$\mathcal{L}_{d}^{k}=\frac{1}{l^{k}}\sum_{i=1}^{l^{k}}\left(|d_{i}^{k}-\hat{d}_{i}^{k}|\right)$,
and $\mathcal{L}_{token}^{k}=-\sum_{i=1}^{L_{\mathcal{P}}}\sum_{t\in
T}\hat{v}_{i,t}^{k}\log(v_{i,t}^{k})$. Hence, the total multi-task loss
$\mathcal{L}$ that we use to train our ART model for both fixation location
prediction and fixation duration prediction is
$\mathcal{L}=\mathcal{L}_{gaze}+\mathcal{L}_{ground}$ when the scanpath has
terminated or the referral audio has ended, and
$\mathcal{L}=\mathcal{L}_{gaze}$ otherwise. Note that $\mathcal{L}_{ground}$
is the auxiliary multi-task grounding loss defined in Sec. 4.2 of the main
paper.

To evaluate fixation duration prediction of baselines and ART, we train them
on fixation duration prediction along with fixation location prediction. Along
with the duration-agnostic metrics we used in the main paper, we also report
$SS^{(t)}$, $FED^{(t)}$, $SS_{pack}^{(t)}$, and $FED_{pack}^{(t)}$, which are
duration-aware variants (as done in previous works [18, 58, 12]) of $SS$,
$FED$, $SS_{pack}$, and $FED_{pack}$, respectively. We also report the
duration component of MultiMatch [4, 22] ($MM_{dur}$). Higher $SS$,
$SS_{pack}$, $SS^{(t)}$, $SS^{(t)}_{pack}$, $CC_{pack}$, $NSS_{pack}$,
$MM_{dur}$ metrics signify higher scanpath similarity, whereas higher $FED$,
$FED_{pack}$, $FED_{pack}^{(t)}$, and $FED^{(t)}$ metrics denote lower
scanpath similarity. The results are in Table 4.

ART outperforms baselines on all metrics (both duration-agnostic and duration-
aware) when trained on and evaluated for fixation duration prediction and
fixation location prediction. The model hyperparameters, pre-training and
training processes remain as mentioned in Sec. IV. Additional details for
baselines endowed with fixation prediction can be found in Sec. VIII.

Note that $MM_{dur}$ reflects solely the duration component, in contrast to
the spatio-temporal metrics ($SS^{(t)}$, $FED^{(t)}$, $SS_{pack}^{(t)}$,
$FED_{pack}^{(t)}$) in Table 4(b), and by that metric, the random baseline
(whose generated fixation duration is set to the average training set fixation
duration, as detailed in Sec. VIII) scores higher than the human consistency
score. We believe this is because of very poor agreement among the behavioral
participants in their fixation duration in our incremental object referral
task, which makes the prediction of fixation duration less meaningful than the
prediction of fixation spatial locations (when we created scanpaths using
average fixation locations from the RefCOCO-Gaze training set, we observed
$SS=0.037$ and $SS_{pack}=0.044$, which are far lower than human consistency
scores ($SS=0.400$, $SS_{pack}=0.317$), signifying that the spatial attention
of humans for our task is meaningful and intention-driven).

Table 5: Performance of ART and baselines on COCO-Search18 [14] dataset.
Gazeformer and ART models are shown in two variants - one with fixation
prediction capability (“w/ dur.” in parenthesis) and one without fixation
prediction capability (“w/o dur.” in parenthesis). Metrics in bold are the
best performing metrics, while those underlined with a single dash are second-
best, and those underlined with a double dash are third-best (we do not
underline the third-best metric with double dash for duration-aware metrics
since there are only three models predicting fixation duration).

(a) Duration-agnostic metrics
$SS\bm{\uparrow}$ $SemSS\bm{\uparrow}$ $FED\bm{\downarrow}$
$SemFED\bm{\downarrow}$ Human 0.490 0.522 2.531 1.720 IRL [82] 0.405 0.441
2.781 2.393 Chen _et al_. [12] 0.398 0.425 2.376 2.064 FFM [84] 0.384 0.391
2.719 2.479 Gazeformer (w/o dur.) [58] 0.475 0.456 2.159 2.012 Gazeformer(w/
dur.) [58] 0.467 0.449 2.198 2.082 ART (w/o dur.) (Proposed) 0.454 0.461 2.251
1.995 ART(w/ dur.) (Proposed) 0.432 0.441 2.335 2.070
(b) Duration-aware metrics
$SS^{(t)}\bm{\uparrow}$ $SemSS^{(t)}\bm{\uparrow}$ $FED^{(t)}\bm{\downarrow}$
$SemFED^{(t)}\bm{\downarrow}$ $MM_{dur}\bm{\uparrow}$ Human 0.409 0.433 11.526
8.389 0.663 Chen _et al_. [12] 0.354 0.368 11.610 9.991 0.691 Gazeformer(w/
dur.) [58] 0.417 0.408 10.216 8.771 0.727 ART (w/ dur.) 0.373 0.394 11.127
9.089 0.725

###  VII ART generalizes to Categorical Search (COCO-Search18)

In this section, we extend ART to the related categorical search task. We do
this via providing a prefix in the form of the category name (_e.g_., “car” or
“potted plant”) to ART. We chose the large-scale categorical search fixation
prediction dataset, COCO-Search18 [14] to train and evaluate ART and other
baselines on its target-present trials. We use several competitive baselines,
such as IRL [82] and FFM [84] which are not trained on an additional fixation
duration prediction objective, along with Chen  _et al_. [12]’s model and
Gazeformer [58] which can be trained on the additional fixation duration
prediction objective. State-of-the-art baseline Gazeformer [58] and ART are
trained and evaluated in two variants - one which is trained on the additional
fixation duration prediction objective (“w/ dur.” in parenthesis) and another
one which is not trained on the additional fixation duration prediction
objective (“w/o dur.” in parenthesis).

To evaluate on this categorical search task embodied by COCO-Search18, we
follow previous methods [58, 84] and map all predictions to our input grid,
and then report Sequence Score [82] ($SS$), Semantic Sequence Score [84]
($SemSS$), Fixation Edit Distance [58] ($FED$), Semantic Fixation Edit
Distance [58] ($SemFED$) and duration component of MultiMatch [4, 22]
($MM_{dur}$). $SemSS$ and $SemFED$ differs from $SS$ and $FED$, respectively,
in that they convert scanpaths to strings of fixated scene object IDs instead
of cluster IDs. We also report $SS$, $SemSS$, $FED$, and $SemFED$ with
duration denoted by $SS^{(t)}$, $SemSS^{(t)}$, $FED^{(t)}$, and
$SemFED^{(t)}$, respectively, as done by previous works [18, 58, 12]. Results
are in Table 5. Higher $SS$, $SemSS$, $SS^{(t)}$, $SemSS^{(t)}$, $MM_{dur}$
metrics signify higher scanpath similarity, whereas higher $FED$, $SemFED$,
$FED^{(t)}$, and $SemFED^{(t)}$ metrics denote lower scanpath similarity.

Even though ART is designed for the incremental object referral task, in an
extension to the categorical search task, we found that its performance is on
par with Gazeformer [58], the state-of-the-art search fixation prediction
model. ART even outperforms Gazeformer on Semantic Sequence Score (SemSS) and
Semantic Fixation Edit Distance (SemFED) metrics. This generalization to the
categorical search task further demonstrates the strength of ART’s
architecture.

###  VIII Additional Details of Baseline Models

In this section, we provide additional implementation details of the baselines
used in the main paper.

Random Scanpath: We sample pack length $l_{p}$ uniformly from integers
[0,1,…,$L_{\mathcal{P}}$] where $L_{\mathcal{P}}$ is the hyperparameter for
maximum number of fixations in a pack. Since we chose $L_{\mathcal{P}}=6$ for
ART, we use the same value for this baseline for fair comparison. Then we
uniformly sample $l_{p}$ fixation locations within the entire image to obtain
a generated pack of fixations. For the variant with fixation duration (Sec.
VI), we use the average of all fixation durations in the RefCOCO-Gaze training
set.

OFA: We sample pack length $l_{p}$ uniformly from integers
[0,1,…,$L_{\mathcal{P}}$] where $L_{\mathcal{P}}$ is the hyperparameter for
maximum number of fixations in a pack. Since we chose $L_{\mathcal{P}}=6$ for
ART, we use the same value for this baseline for fair comparison. In order to
obtain a generated pack of fixations, we uniformly sample $l_{p}$ fixation
locations from within the bounding box predicted by the OFA [78] model for the
referring expression prefix corresponding to an incoming word within the
referring expression. For the variant with fixation duration (Sec. VI), we use
the average of all fixation durations in the RefCOCO-Gaze training set.

Chen  _et al_. We train model from Chen  _et al_. [12] using teacher-forcing
algorithm [79] in the same manner we have trained ART. To incorporate fixation
history, we construct a composite action map containing all previous
fixations. We subsequently initialize the dynamic memory of the model with the
sum of the task guidance map (from MDETR model which is pre-trained on RefCOCO
for fair comparison) and the composite action map. Maximum number of fixations
in a predicted pack is set to 6, identical to the value of pack length
$L_{\mathcal{P}}$ chosen for ART, for fair comparison. In the context of
experiments on RefCOCO-Gaze, we train Chen  _et al_.’s model with fixation
duration information for results in Sec. VI and without fixation duration
information for the rest, unless specified otherwise.

Gazeformer-ref. We train the Gazeformer [58] variant, that we named
Gazeformer-ref, using teacher-forcing algorithm [79] in the same manner we
have trained ART. We provide previous fixation information in the form of the
last fixation from the previous non-null pack, which is encoded using a 2D
positional encoding and added to the first fixation query as prescribed in
[58] for including initial fixation information. The validity prediction head
in the model is also extended to support the prediction of an additional end-
of-scanpath token. Maximum number of fixations in a predicted pack is set to
6, identical to the value of pack length $L_{\mathcal{P}}$ chosen for ART, for
fair comparison. In the context of experiments on RefCOCO-Gaze, we train
Gazeformer-ref with fixation duration information for results in Sec. VI and
without fixation duration information for the rest, unless specified
otherwise.

Gazeformer-cat. We train the Gazeformer [58] variant, that we named
Gazeformer-cat, using teacher-forcing algorithm [79] in the same manner we
have trained ART. The previous fixation history is conveyed in the same manner
as in the implementation of Gazeformer-ref (see above). We also extend the
validity prediction head to support the prediction of an additional end-of-
scanpath token. The target category estimation which is used to construct the
input category name comes from a RoBERTa-based classifier which is separately
trained on RefCOCO referring expressions and their corresponding target
categories. Specifically, the target category estimator is a RoBERTa-base
model with a classification head on top. This baseline should show how
important target category estimation is for gaze prediction. Maximum number of
fixations in a predicted pack is set to 6, identical to the value of pack
length $L_{\mathcal{P}}$ chosen for ART, for fair comparison. In the context
of experiments on RefCOCO-Gaze, we train Gazeformer-cat with fixation duration
information for results in Sec. VI and without fixation duration information
for the rest, unless specified otherwise.

###  IX Additional metrics for Ablation Studies

Table 6: Ablation studies on ART model (reported in Table 2 of the main paper) augmented with additional metrics. If either $\mathcal{L}_{bbox}$ or $\mathcal{L}_{target}$ is included, and the model undergoes pre-training, the loss is applied in both pre-training and gaze training phases. Ablation | Pre- | $\mathcal{L}_{bbox}$ | $\mathcal{L}_{target}$ | $SS$ | $SS_{pack}$ | $FED$ | $FED_{pack}$ | $CC_{pack}$ | $NSS_{pack}$
---|---|---|---|---|---|---|---|---|---
# | training |  |  | $\bm{\uparrow}$ | $\bm{\uparrow}$ | $\bm{\downarrow}$ | $\bm{\downarrow}$ | $\bm{\uparrow}$ | $\bm{\uparrow}$
1 | $\times$ | $\times$ | $\times$ | 0.309 | 0.257 | 6.873 | 1.203 | 0.222 | 3.032
2 | $\checkmark$ | $\checkmark$ | $\times$ | 0.321 | 0.279 | 7.341 | 1.348 | 0.239 | 2.769
3 | $\checkmark$ | $\times$ | $\checkmark$ | 0.292 | 0.260 | 6.713 | 1.162 | 0.216 | 2.967
4 | $\times$ | $\checkmark$ | $\checkmark$ | 0.304 | 0.257 | 7.104 | 1.245 | 0.215 | 2.953
5 | $\checkmark$ | $\checkmark$ | $\checkmark$ | 0.359 | 0.292 | 6.371 | 1.143 | 0.280 | 3.478

In Table 6, we augment Table 2 in the main paper with additional metrics
($FED$, $FED_{pack}$ and $NSS_{pack}$). The trends remain similar to the what
we observed for $SS$ and $SS_{pack}$ scores, thereby reaffirming our assertion
that both object localization and target category prediction tasks are
integral to the object referral process and that pre-training on these tasks
is instrumental for superior performance.

###  X Additional Ablation Studies and Analysis

Table 7: Additional ablation studies on ART model. If either $\mathcal{L}_{bbox}$ or $\mathcal{L}_{target}$ is included, and the model undergoes pre-training, the loss is applied in both pre-training and gaze training phases. Ablations #3, #4 and #5 are from Table 6 (also in Table 2 in the main paper).  Ablation | Pre- | $\mathcal{L}_{bbox}$ | $\mathcal{L}_{target}$ | $SS$ | $SS_{pack}$ | $FED$ | $FED_{pack}$ | $CC_{pack}$ | $NSS_{pack}$
---|---|---|---|---|---|---|---|---|---
# | training |  |  | $\bm{\uparrow}$ | $\bm{\uparrow}$ | $\bm{\downarrow}$ | $\bm{\downarrow}$ | $\bm{\uparrow}$ | $\bm{\uparrow}$
1 | $\times$ | $\checkmark$ | $\times$ | 0.319 | 0.270 | 6.736 | 1.193 | 0.228 | 3.135
2 | $\times$ | $\times$ | $\checkmark$ | 0.309 | 0.264 | 6.705 | 1.175 | 0.216 | 2.919
3 | $\checkmark$ | $\checkmark$ | $\times$ | 0.321 | 0.279 | 7.341 | 1.348 | 0.239 | 2.769
4 | $\checkmark$ | $\times$ | $\checkmark$ | 0.292 | 0.260 | 6.713 | 1.162 | 0.216 | 2.967
5 | $\checkmark$ | $\checkmark$ | $\checkmark$ | 0.359 | 0.292 | 6.371 | 1.143 | 0.280 | 3.478

We provide two additional ablations (in addition to the five ablations in
Table 2 of the main paper and Table 6 in Section IX) tabulated as ablation #1
and ablation #2 in Table 7. As it can be seen, addition of only one of the
auxiliary losses $\mathcal{L}_{bbox}$ and $\mathcal{L}_{target}$ results in
little to no boost in performance. It is evident that we need both
$\mathcal{L}_{bbox}$ and $\mathcal{L}_{target}$ in the objective function
along with pre-training for our model to achieve high performance. We posit
that ablation #4 in Table 7 fails to perform well because the target category
prediction task largely, if not completely, relies on the linguistic input
(i.e., the referring expression). Consequently, the sub-networks dedicated to
visual and visuo-linguistic processing (especially, the visual encoder) might
not benefit from pre-training only on this objective whereas the linguistic
subnetworks (i.e. the linguistic encoder) are greatly optimized. We speculate
that it is also perhaps hard for ART to adapt to object referral during
training after pre-training its parameters to significantly align with the
target-category estimation objective (which can be inadequate for our task
since there are multiple objects belonging to the target category) in ablation
#4 of Table 7. On the other hand, object localization seems to be much more
aligned with the object referral task, which is indeed shown by ablation #3 in
Table 7. In summary, the ablation studies validate our hypothesis that both
object localization and target category prediction tasks are integral to the
object referral process. Also note that Ablations 1 and 4 in Table 6 and
Ablations 1 and 2 in Table 7 show that even when ART is not pre-trained on
RefCOCO, it still outperforms baselines that are also not pre-trained on
RefCOCO, i.e., Gazeformer-ref and Gazeformer-cat (in Table 1 of main text).

Table 8: Additional ablation studies on ART model when trained with additional
Fixation Duration Prediction objective. If either $\mathcal{L}_{bbox}$ or
$\mathcal{L}_{target}$ is included, and the model undergoes pre-training, the
loss is applied in both pre-training and gaze training phases.

(a) Duration-agnostic Metrics  Ablation Pre- $\mathcal{L}_{bbox}$
$\mathcal{L}_{target}$ $SS$ $SS_{pack}$ $FED$ $FED_{pack}$ $CC_{pack}$
$NSS_{pack}$ # training $\bm{\uparrow}$ $\bm{\uparrow}$ $\bm{\downarrow}$
$\bm{\downarrow}$ $\bm{\uparrow}$ $\bm{\uparrow}$ 1 $\times$ $\times$ $\times$
0.206 0.169 10.840 2.073 0.167 2.331 2 $\times$ $\checkmark$ $\times$ 0.262
0.230 8.225 1.528 0.207 2.930 3 $\times$ $\times$ $\checkmark$ 0.269 0.201
9.626 1.616 0.196 2.396 4 $\times$ $\checkmark$ $\checkmark$ 0.296 0.252 7.049
1.271 0.209 2.914 5 $\checkmark$ $\checkmark$ $\times$ 0.309 0.278 7.306 1.339
0.257 3.307 6 $\checkmark$ $\times$ $\checkmark$ 0.284 0.210 7.172 1.351 0.179
2.460 7 $\checkmark$ $\checkmark$ $\checkmark$ 0.356 0.285 6.410 1.161 0.281
3.539
(b) Duration-aware Metrics  Ablation Pre- $\mathcal{L}_{bbox}$
$\mathcal{L}_{target}$ $SS^{(t)}$ $SS_{pack}^{(t)}$ $FED^{(t)}$
$FED_{pack}^{(t)}$ $MM_{dur}$ # training $\bm{\uparrow}$ $\bm{\uparrow}$
$\bm{\downarrow}$ $\bm{\downarrow}$ $\bm{\uparrow}$ 1 $\times$ $\times$
$\times$ 0.222 0.110 55.614 11.274 0.672 2 $\times$ $\checkmark$ $\times$
0.253 0.169 44.074 8.559 0.691 3 $\times$ $\times$ $\checkmark$ 0.277 0.159
47.876 8.928 0.675 4 $\times$ $\checkmark$ $\checkmark$ 0.253 0.164 38.722
7.232 0.652 5 $\checkmark$ $\checkmark$ $\times$ 0.282 0.181 37.922 7.389
0.685 6 $\checkmark$ $\times$ $\checkmark$ 0.252 0.165 38.934 7.206 0.683 7
$\checkmark$ $\checkmark$ $\checkmark$ 0.332 0.199 35.997 7.120 0.696

In Table 8, we tabulate the ablation studies for ART when equipped with
fixation duration prediction capability. As shown in Sec. VI through analysis
of $MM_{dur}$ metric values for random baseline and human consistency, there
is poor agreement between participants in their fixation durations and thus
training on such noisy supervision can be challenging. We interpret Table 8 as
being consistent with our findings from the ablation studies with ART without
fixation prediction (Sec. 5.4 in main text; Sec. IX, Sec. X in the supplement)
in supporting our assertion that both pre-training and training on both
auxiliary object localization and target category estimation objectives are
crucial for ART’s performance. We also observe that without pre-training and
training on auxiliary losses, ART struggles to generalize when trained with
noisy fixation durations, as seen in Ablation 1. Our proposed model (#7 in
Table 8) supports our assertion that ART’s pre-training and training on the
two object grounding tasks for complete/partial expressions underlying object
referral significantly contributes towards its SOTA performance when compared
to existing baselines that are unable to pre-train/train on object grounding
for partial/complete expressions. Our proposed model thus generalizes well
even when trained with a noisy supervision signal, such as the fixation
durations. We also note that when not pre-trained on RefCOCO (Ablations 2, 3,
and 4 in Table 8), ART still outperforms baselines that are not pre-trained on
RefCOCO, i.e. Gazeformer-ref and Gazeformer-cat (see Table 4).

###  XI Auxiliary Next Word Token Prediction Task

We also hypothesized that predicting the next linguistic token during search
also underlies incremental object referral process along with the object
localization and target category prediction tasks. So we added a
NEXT_WORD_TOKEN token along with BBOX and TGT tokens as input to the visuo-
linguistic encoder. The corresponding latent vector served as input to an MLP
which generated logits over a vocabulary of tokens in order to predict the
next word token. The loss imposed is a cross-entropy loss
$\mathcal{L}_{nextword}$. The results are tabulated in Table 9. As we can see,
adding $\mathcal{L}_{nextword}$ does not improve the performance significantly
– we achieve best performance without $\mathcal{L}_{nextword}$ (Ablation #1 in
Table 9). We hypothesize that this is because the next word token prediction
task is considerably more difficult than the object localization and target
category prediction tasks, and potentially introduces noise while training on
the gaze prediction objective.

Table 9: Effect of auxiliary next word token prediction task on ART model. Ablations #1 and #3 are from Table 2 in main text.  Sl.No# | Pre-training | $\mathcal{L}_{nextword}$ | $\mathcal{L}_{bbox}$ | $\mathcal{L}_{target}$ | $SS$ | $SS_{pack}$ | $CC_{pack}$ | $NSS_{pack}$
---|---|---|---|---|---|---|---|---
1 | ✓ | $\times$ | $\checkmark$ | $\checkmark$ | 0.359 | 0.292 | 0.280 | 3.478
2 | ✓ | $\checkmark$ | $\checkmark$ | $\checkmark$ | 0.355 | 0.281 | 0.269 | 3.388
3 | $\times$ | $\times$ | $\checkmark$ | $\checkmark$ | 0.304 | 0.257 | 0.215 | 2.953
4 | $\times$ | $\checkmark$ | $\checkmark$ | $\checkmark$ | 0.313 | 0.265 | 0.224 | 3.049

###  XII Qualitative Results

In this section, we present additional qualitative results of human behavior,
our model ART and other competitive baseline models in Fig. 8 and Fig. 9. We
see that ART efficiently finds the correct target through scanpaths that
closely resemble human behavior in all rows except the last row in Fig. 9 \-
where it fails to localize the “head” of the correct person. In the first row
of Fig. 8, we can see both the human participant and ART wait until the last
word “left” to localize the correct muffin. We see a similar waiting pattern
in second and third row of Fig. 8 where ART and the human participant wait
until disambiguation towards the end of the expression for localizing the
correct “bus” and “car” respectively. In the first row of Fig. 9, we see a
scanning behavior where ART and the human fixate on the kids in the center
after hearing the word “kid” until the contextual information “far right” is
provided in the end to locate the correct “kid”. In the second row of Fig. 9,
we observe that both ART and the human participant wait till the utterance of
the target category word “elephant” in order to localize the correct elephant.

Human ART (Proposed) Chen _et al_. Gazeformer-ref

Human ART (Proposed) Chen _et al_. Gazeformer-ref

Human ART (Proposed) Chen _et al_. Gazeformer-ref

Figure 8: Qualitative results [1/2]. Scanpaths from humans and three scanpath
prediction models on three trials exhibiting strategic fixation behavior.
Fixations (denoted by circles numbered with fixation order) are color-coded to
corresponding words in the referring expression (above each row). Fixations
color-coded to [BOT] occurred before the expression started, and those color-
coded to [EOT] occurred after the expression ended. Blue bounding boxes
indicating the referred objects are not visible during trials. Our model
generates the most human-like scanpaths for incremental object referral.

Human ART (Proposed) Chen _et al_. Gazeformer-ref

Human ART (Proposed) Chen _et al_. Gazeformer-ref

Human ART (Proposed) Chen _et al_. Gazeformer-ref

Figure 9: Qualitative results [2/2]. Scanpaths from humans and three scanpath
prediction models on three trials exhibiting strategic fixation behavior.
Fixations (denoted by circles numbered with fixation order) are color-coded to
corresponding words in the referring expression (above each row). Fixations
color-coded to [BOT] occurred before the expression started, and those color-
coded to [EOT] occurred after the expression ended. Blue bounding boxes
indicating the referred objects are not visible during trials. Our model
generates the most human-like scanpaths for incremental object referral.

## References

  * [1] Adhanom, I.B., Griffin, N.N., MacNeilage, P., Folmer, E.: The effect of a foveated field-of-view restrictor on vr sickness. In: 2020 IEEE Conference on Virtual Reality and 3D User Interfaces (VR). IEEE (2020)
  * [2] Alayrac, J.B., Donahue, J., Luc, P., Miech, A., Barr, I., Hasson, Y., Lenc, K., Mensch, A., Millican, K., Reynolds, M., et al.: Flamingo: a visual language model for few-shot learning. Advances in Neural Information Processing Systems (2022)
  * [3] Altmann, G.T.: Language can mediate eye movement control within 100milliseconds, regardless of whether there is anything to move the eyes to. Acta Psychologica 137(2), 190–200 (Jun 2011)
  * [4] Anderson, N.C., Anderson, F., Kingstone, A., Bischof, W.F.: A comparison of scanpath comparison methods. Behavior research methods 47(4), 1377–1392 (2015)
  * [5] Antol, S., Agrawal, A., Lu, J., Mitchell, M., Batra, D., Zitnick, C.L., Parikh, D.: VQA: Visual Question Answering. In: Proceedings of the IEEE/CVF International Conference on Computer Vision (2015)
  * [6] Bapna, T., Valles, J., Leng, S., Pacilli, M., Nataraja, R.M.: Eye-tracking in surgery: a systematic review. ANZ Journal of Surgery 93(11), 2600–2608 (2023)
  * [7] Bennett, C.R., Bex, P.J., Merabet, L.B.: Assessing visual search performance using a novel dynamic naturalistic scene. Journal of Vision 21(1), 5–5 (2021)
  * [8] Berg, D.J., Boehnke, S.E., Marino, R.A., Munoz, D.P., Itti, L.: Free viewing of dynamic stimuli by humans and monkeys. Journal of vision 9(5), 19–19 (2009)
  * [9] Bylinskii, Z., Judd, T., Oliva, A., Torralba, A., Durand, F.: What do different evaluation metrics tell us about saliency models? IEEE Transactions on Pattern Analysis and Machine Intelligence 41(3), 740–757 (2019)
  * [10] Carion, N., Massa, F., Synnaeve, G., Usunier, N., Kirillov, A., Zagoruyko, S.: End-to-end object detection with transformers. In: European Conference on Computer Vision (2020)
  * [11] Chen, S., Jiang, M., Yang, J., Zhao, Q.: Air: Attention with reasoning capability. In: European Conference on Computer Vision (2020)
  * [12] Chen, X., Jiang, M., Zhao, Q.: Predicting human scanpaths in visual question answering. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (2021)
  * [13] Chen, X., Ma, L., Chen, J., Jie, Z., Liu, W., Luo, J.: Real-time referring expression comprehension by single-stage grounding network. arXiv preprint arXiv:1812.03426 (2018)
  * [14] Chen, Y., Yang, Z., Ahn, S., Samaras, D., Hoai, M., Zelinsky, G.: Coco-search18 fixation dataset for predicting goal-directed attention control. Scientific reports 11(1), 8776 (2021)
  * [15] Chen, Y., Yang, Z., Chakraborty, S., Mondal, S., Ahn, S., Samaras, D., Hoai, M., Zelinsky, G.: Characterizing target-absent human attention. In: Proceedings of CVPR International Workshop on Gaze Estimation and Prediction in the Wild (2022)
  * [16] Chung, J., Lee, H., Moon, H., Lee, E.: The static and dynamic analyses of drivers’ gaze movement using vr driving simulator. Applied Sciences 12(5), 2362 (2022)
  * [17] Cooper, R.M.: The control of eye fixation by the meaning of spoken language: A new methodology for the real-time investigation of speech perception, memory, and language processing. Cognitive Psychology 6(1), 84–107 (1974)
  * [18] Cristino, F., Mathôt, S., Theeuwes, J., Gilchrist, I.D.: Scanmatch: A novel method for comparing fixation sequences. Behavior research methods 42(3), 692–700 (2010)
  * [19] Deng, J., Dong, W., Socher, R., Li, L.J., Li, K., Fei-Fei, L.: Imagenet: A large-scale hierarchical image database. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (2009)
  * [20] Deng, J., Yang, Z., Chen, T., Zhou, W., Li, H.: Transvg: End-to-end visual grounding with transformers. In: Proceedings of the IEEE/CVF International Conference on Computer Vision (2021)
  * [21] Devlin, J., Chang, M.W., Lee, K., Toutanova, K.: BERT: Pre-training of deep bidirectional transformers for language understanding. In: Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) (2019)
  * [22] Dewhurst, R., Nyström, M., Jarodzka, H., Foulsham, T., Johansson, R., Holmqvist, K.: It depends on how you look at it: Scanpath comparison in multiple dimensions with multimatch, a vector-based approach. Behavior research methods 44(4), 1079–1100 (2012)
  * [23] Fang, H., Gupta, S., Iandola, F., Srivastava, R.K., Deng, L., Dollár, P., Gao, J., He, X., Mitchell, M., Platt, J.C., et al.: From captions to visual concepts and back. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (2015)
  * [24] Gokaslan, A., Cohen, V.: Openwebtext corpus. http://Skylion007.github.io/OpenWebTextCorpus (2019)
  * [25] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (2016)
  * [26] He, S., Tavakoli, H.R., Borji, A., Pugeault, N.: Human attention in image captioning: Dataset and analysis. In: Proceedings of the IEEE/CVF International Conference on Computer Vision (2019)
  * [27] Henderson, J.M., Brockmole, J.R., Castelhano, M.S., Mack, M.: Visual saliency does not account for eye movements during visual search in real-world scenes. In: Eye movements, pp. 537–III. Elsevier (2007)
  * [28] Hong, R., Liu, D., Mo, X., He, X., Zhang, H.: Learning to compose and reason with language tree structures for visual grounding. IEEE Transactions on Pattern Analysis and Machine Intelligence 44(2), 684–696 (2019)
  * [29] Hu, R., Rohrbach, M., Andreas, J., Darrell, T., Saenko, K.: Modeling relationships in referential expressions with compositional modular networks. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (2017)
  * [30] Itti, L., Koch, C., Niebur, E.: A model of saliency-based visual attention for rapid scene analysis. IEEE Transactions on Pattern Analysis and Machine Intelligence 20(11), 1254–1259 (1998)
  * [31] Jia, C., Yang, Y., Xia, Y., Chen, Y.T., Parekh, Z., Pham, H., Le, Q., Sung, Y.H., Li, Z., Duerig, T.: Scaling up visual and vision-language representation learning with noisy text supervision. In: International Conference on Machine Learning (2021)
  * [32] Jost, T., Ouerhani, N., Von Wartburg, R., Müri, R., Hügli, H.: Assessing the contribution of color in visual attention. Computer Vision and Image Understanding 100(1-2), 107–123 (2005)
  * [33] Kamath, A., Singh, M., LeCun, Y., Synnaeve, G., Misra, I., Carion, N.: Mdetr-modulated detection for end-to-end multi-modal understanding. In: Proceedings of the IEEE/CVF International Conference on Computer Vision (2021)
  * [34] Kamide, Y., Altmann, G.T., Haywood, S.L.: The time-course of prediction in incremental sentence processing: Evidence from anticipatory eye movements. Journal of Memory and language 49(1), 133–156 (2003)
  * [35] Kazemzadeh, S., Ordonez, V., Matten, M., Berg, T.: Referitgame: Referring to objects in photographs of natural scenes. In: Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) (2014)
  * [36] Khokhar, A., Yoshimura, A., Borst, C.: Eye-gaze-triggered visual cues to restore attention in educational vr. In: 2019 IEEE conference on virtual reality and 3D user interfaces (VR), poster (2019)
  * [37] Kingma, D.P., Welling, M.: Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114 (2013)
  * [38] Knoeferle, P., Guerra, E.: Visually Situated Language Comprehension: Visually Situated Language Comprehension. Language and Linguistics Compass 10(2), 66–82 (Feb 2016)
  * [39] Koehler, K., Guo, F., Zhang, S., Eckstein, M.P.: What do saliency models predict? Journal of vision 14(3), 14–14 (2014)
  * [40] Kuo, C.W., Kira, Z.: Beyond a pre-trained object detector: Cross-modal textual and visual context for image captioning. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (2022)
  * [41] Lang, Y., Wei, L., Xu, F., Zhao, Y., Yu, L.F.: Synthesizing personalized training programs for improving driving habits via virtual reality. In: 2018 IEEE Conference on Virtual Reality and 3D User Interfaces (VR). IEEE (2018)
  * [42] Lavoie, E., Hebert, J.S., Chapman, C.S.: Comparing eye–hand coordination between controller-mediated virtual reality, and a real-world object interaction task. Journal of Vision 24(2), 9–9 (2024)
  * [43] Levenshtein, V.I.: Binary codes capable of correcting deletions, insertions, and reversals. Soviet physics. Doklady 10, 707–710 (1965)
  * [44] Li, L.H., Yatskar, M., Yin, D., Hsieh, C.J., Chang, K.W.: Visualbert: A simple and performant baseline for vision and language. arXiv preprint arXiv:1908.03557 (2019)
  * [45] Li, P., Tian, B., Shi, Y., Chen, X., Zhao, H., Zhou, G., Zhang, Y.Q.: Toist: Task oriented instance segmentation transformer with noun-pronoun distillation. Advances in Neural Information Processing Systems (2022)
  * [46] Li, Y., Chen, X., Zhao, H., Gong, J., Zhou, G., Rossano, F., Zhu, Y.: Understanding embodied reference with touch-line transformer. In: International Conference on Learning Representations (2023)
  * [47] Liao, Y., Liu, S., Li, G., Wang, F., Chen, Y., Qian, C., Li, B.: A real-time cross-modality correlation filtering method for referring expression comprehension. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (2020)
  * [48] Lin, T.Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., Zitnick, C.L.: Microsoft coco: Common objects in context. In: European Conference on Computer Vision. pp. 740–755. Springer (2014)
  * [49] Liu, D., Zhang, H., Wu, F., Zha, Z.J.: Learning to assemble neural module tree networks for visual grounding. In: Proceedings of the IEEE/CVF International Conference on Computer Vision (2019)
  * [50] Liu, X., Lai, H., Wong, D.F., Chao, L.S.: Norm-based curriculum learning for neural machine translation. In: Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics (2020)
  * [51] Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., Stoyanov, V.: Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692 (2019)
  * [52] Loshchilov, I., Hutter, F.: Decoupled weight decay regularization. In: International Conference on Learning Representations (2019)
  * [53] Mao, J., Huang, J., Toshev, A., Camburu, O., Yuille, A.L., Murphy, K.: Generation and comprehension of unambiguous object descriptions. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (2016)
  * [54] Masciocchi, C.M., Mihalas, S., Parkhurst, D., Niebur, E.: Everyone knows what is interesting: Salient locations which should be fixated. Journal of vision 9(11), 25–25 (2009)
  * [55] McAuliffe, M., Socolof, M., Mihuc, S., Wagner, M., Sonderegger, M.: Montreal forced aligner: Trainable text-speech alignment using kaldi. In: Interspeech (2017)
  * [56] Mensink, T., Uijlings, J., Castrejon, L., Goel, A., Cadar, F., Zhou, H., Sha, F., Araujo, A., Ferrari, V.: Encyclopedic vqa: Visual questions about detailed properties of fine-grained categories. In: Proceedings of the IEEE/CVF International Conference on Computer Vision (2023)
  * [57] Min, K., Corso, J.J.: Integrating human gaze into attention for egocentric activity recognition. In: Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (2021)
  * [58] Mondal, S., Yang, Z., Ahn, S., Samaras, D., Zelinsky, G., Hoai, M.: Gazeformer: Scalable, effective and fast prediction of goal-directed human attention. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (2023)
  * [59] Nagel, S.: Cc-news. http://commoncrawl.org/blog/news-dataset-available (2016)
  * [60] Needleman, S.B., Wunsch, C.D.: A general method applicable to the search for similarities in the amino acid sequence of two proteins. Journal of molecular biology 48(3), 443–453 (1970)
  * [61] Pai, Y.S., Tag, B., Outram, B., Vontin, N., Sugiura, K., Kunze, K.: Gazesim: Simulating foveated rendering using depth in eye gaze for vr. In: ACM SIGGRAPH 2016 Posters (2016)
  * [62] Peters, R.J., Iyer, A., Koch, C., Itti, L.: Components of bottom-up gaze allocation in natural scenes. Journal of Vision 5(8), 692–692 (2005)
  * [63] Pont-Tuset, J., Uijlings, J., Changpinyo, S., Soricut, R., Ferrari, V.: Connecting vision and language with localized narratives. In: European Conference on Computer Vision (2020)
  * [64] Radford, A., Kim, J.W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al.: Learning transferable visual models from natural language supervision. In: International conference on machine learning (2021)
  * [65] Rezatofighi, H., Tsoi, N., Gwak, J., Sadeghian, A., Reid, I., Savarese, S.: Generalized intersection over union: A metric and a loss for bounding box regression. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (2019)
  * [66] Sennrich, R., Haddow, B., Birch, A.: Neural machine translation of rare words with subword units. In: Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (2016)
  * [67] Tanenhaus, M.K., Spivey-Knowlton, M.J., Eberhard, K.M., Sedivy, J.C.: Integration of Visual and Linguistic Information in Spoken Language Comprehension. Science 268(5217), 1632–1634 (Jun 1995)
  * [68] Tanenhaus, M.K., Spivey-Knowlton, M.J., Eberhard, K.M., Sedivy, J.C.: Using eye movements to study spoken language comprehension: Evidence for visually mediated incremental interpretation. (1996)
  * [69] Thanh, N.C.: The differences between spoken and written grammar in english, in comparison with vietnamese (las diferencias entre la gramática oral y escrita del idioma inglés en comparación con el idioma vietnamita). Gist Education and Learning Research Journal 11, 138–153 (2015)
  * [70] Torralba, A., Oliva, A., Castelhano, M.S., Henderson, J.M.: Contextual guidance of eye movements and attention in real-world scenes: the role of global features in object search. Psychological review 113(4), 766 (2006)
  * [71] Townend, J., Walker, J.: Structure of Language: Spoken and Written English. Whurr Publishers (2006)
  * [72] Trinh, T.H., Le, Q.V.: A simple method for commonsense reasoning. arXiv preprint arXiv:1806.02847 (2018)
  * [73] Vaidyanathan, P., Prud’hommeaux, E., Alm, C.O., Pelz, J.B.: Computational framework for fusing eye movements and spoken narratives for image annotation. Journal of Vision 20(7), 13–13 (2020)
  * [74] Vaidyanathan, P., Prud’hommeaux, E., Pelz, J.B., Alm, C.O.: Snag: Spoken narratives and gaze dataset. In: Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (2018)
  * [75] Vasudevan, A.B., Dai, D., Van Gool, L.: Object referring in videos with language and human gaze. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (2018)
  * [76] Vasudevan, A.B., Dai, D., Van Gool, L.: Object referring in visual scene with spoken language. In: Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (2018)
  * [77] Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, L., Polosukhin, I.: Attention is all you need. In: Advances in Neural Information Processing Systems (2017)
  * [78] Wang, P., Yang, A., Men, R., Lin, J., Bai, S., Li, Z., Ma, J., Zhou, C., Zhou, J., Yang, H.: Ofa: Unifying architectures, tasks, and modalities through a simple sequence-to-sequence learning framework. In: International Conference on Machine Learning (2022)
  * [79] Williams, R.J., Zipser, D.: A learning algorithm for continually running fully recurrent neural networks. Neural computation 1(2), 270–280 (1989)
  * [80] Xu, J., Yue, S., Menchinelli, F., Guo, K.: What has been missed for predicting human attention in viewing driving clips? PeerJ 5, e2946 (2017)
  * [81] Yan, B., Jiang, Y., Wu, J., Wang, D., Luo, P., Yuan, Z., Lu, H.: Universal instance perception as object discovery and retrieval. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (2023)
  * [82] Yang, Z., Huang, L., Chen, Y., Wei, Z., Ahn, S., Zelinsky, G., Samaras, D., Hoai, M.: Predicting goal-directed human attention using inverse reinforcement learning. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (2020)
  * [83] Yang, Z., Mondal, S., Ahn, S., Xue, R., Zelinsky, G., Hoai, M., Samaras, D.: Unifying top-down and bottom-up scanpath prediction using transformers. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (2024)
  * [84] Yang, Z., Mondal, S., Ahn, S., Zelinsky, G., Hoai, M., Samaras, D.: Target-absent human attention. In: European Conference on Computer Vision (2022)
  * [85] Yu, J., Wang, Z., Vasudevan, V., Yeung, L., Seyedhosseini, M., Wu, Y.: Coca: Contrastive captioners are image-text foundation models. Transactions on Machine Learning Research (2022)
  * [86] Yu, L., Poirson, P., Yang, S., Berg, A.C., Berg, T.L.: Modeling context in referring expressions. In: European Conference on Computer Vision (2016)
  * [87] Yuan, L., Chen, D., Chen, Y.L., Codella, N., Dai, X., Gao, J., Hu, H., Huang, X., Li, B., Li, C., et al.: Florence: A new foundation model for computer vision. arXiv preprint arXiv:2111.11432 (2021)
  * [88] Zelinsky, G., Yang, Z., Huang, L., Chen, Y., Ahn, S., Wei, Z., Adeli, H., Samaras, D., Hoai, M.: Benchmarking gaze prediction for categorical visual search. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (2019)
  * [89] Zelinsky, G.J., Chen, Y., Ahn, S., Adeli, H.: Changing perspectives on goal-directed attention control: The past, present, and future of modeling fixations during visual search. In: Psychology of Learning and Motivation, vol. 73, pp. 231–286. Elsevier (2020)
  * [90] Zelinsky, G.J., Chen, Y., Ahn, S., Adeli, H., Yang, Z., Huang, L., Samaras, D., Hoai, M.: Predicting goal-directed attention control using inverse-reinforcement learning. Neurons, behavior, data analysis and theory (2020)
  * [91] Zhang, D., Tian, Y., Chen, K., Qian, K.: Gaze-directed visual grounding under object referring uncertainty. In: 2022 41st Chinese Control Conference (CCC). IEEE (2022)
  * [92] Zhu, Y., Kiros, R., Zemel, R., Salakhutdinov, R., Urtasun, R., Torralba, A., Fidler, S.: Aligning books and movies: Towards story-like visual explanations by watching movies and reading books. In: Proceedings of the IEEE/CVF International Conference on Computer Vision (2015)