11institutetext: Hong Kong University of Science and Technology, Clear Water
Bay, HK 22institutetext: University of Durham, Durham, UK
22email: {yfangba<EMAIL_ADDRESS>22email:
<EMAIL_ADDRESS>22email<EMAIL_ADDRESS>22email<EMAIL_ADDRESS>
# OAT: Object-Level Attention Transformer for Gaze Scanpath Prediction

Yini Fang *1*1 0009-0008-9478-5545 Jingling Yu *1*1 0000-0002-3076-1287
Haozheng Zhang 22 0000-0003-1312-4566 Ralf van der Lans 11 0000-0002-7726-8238
Bertram Shi 11 0000-0001-9167-7495

###### Abstract

Visual search is important in our daily life. The efficient allocation of
visual attention is critical to effectively complete visual search tasks.
Prior research has predominantly modelled the spatial allocation of visual
attention in images at the pixel level, e.g. using a saliency map. However,
emerging evidence shows that visual attention is guided by objects rather than
pixel intensities. This paper introduces the Object-level Attention
Transformer (OAT), which predicts human scanpaths as they search for a target
object within a cluttered scene of distractors. OAT uses an encoder-decoder
architecture. The encoder captures information about the position and
appearance of the objects within an image and about the target. The decoder
predicts the gaze scanpath as a sequence of object fixations, by integrating
output features from both the encoder and decoder. We also propose a new
positional encoding that better reflects spatial relationships between
objects. We evaluated OAT on the Amazon book cover dataset and a new dataset
for visual search that we collected. OAT’s predicted gaze scanpaths align more
closely with human gaze patterns, compared to predictions by algorithms based
on spatial attention on both established metrics and a novel behavioural-based
metric. Our results demonstrate the generalization ability of OAT, as it
accurately predicts human scanpaths for unseen layouts and target objects. The
code is available at: https://github.com/HKUST-NISL/oat_eccv24.

11footnotetext: The first two authors contributed equally in the paper.

##  1 Introduction

Figure 1: Illustration of object-level scanpath prediction.

Target search happens regularly in daily life. For example, parents may search
for their children in the crowd at an amusement park. Short-sighted people may
look for their glasses in the clutter of objects on their desk. Shoppers may
search for a desired product from among an array of similar products on a
shelf. Predicting people’s gaze behaviour in visual search is vital, as it
offers valuable implications across multiple domains, such as estimating user
intentions and subsequently facilitating robot guidance in hierarchical tasks
in HCI [29], enhancing user experience in virtual reality (VR) and augmented
reality (AR) environments [2, 14, 26], and increasing revenue through
advertising and marketing strategies [16, 33].

Because the high-resolution vision that facilitates target identification is
restricted to a small area (the fovea) of the eye, target search is often
executed as a sequence of saccades to different locations in the visual field.
This sequence of locations is referred to as the scanpath. Better
understanding of visual search can be obtained through models that predict
human scanpaths during visual search.

Scanpath prediction is closely related to visual attention, a set of processes
that identifies salient (important or task-relevant) areas in the visual
field. Early models of scanpath prediction during free-viewing of images were
based on the concept of two-dimensional saliency map, which assigns to each
pixel a saliency value. Most work has focused on modelling free-viewing, where
scanpaths are often generated by successively selecting pixels with high
salience with an inhibition-of-return mechanism to avoid the same point being
selected over and over again. Recently, models have begun to address scanpath
prediction in task-oriented or goal-directed behaviors, like visual search
[36, 34, 3, 21, 35]. These usually employ a spatial component for extracting
visual features of the stimuli and a temporal component for generating the
gaze sequence, but have retained a focus on pixel-level prediction.

However, increasing evidence suggests visual attention is not directed towards
pixels, but rather objects, especially in the case of goal-directed behavior
[8, 23, 25]. A wide range of psychophysical experiments have demonstrated the
importance of objects in understanding visual attention [12, 7, 24].

We hypothesize that modelling scanpaths as sequences of object fixations,
rather than pixel fixations, will result in more accurate models of human gaze
behavior, especially in tasks such as visual search. We anticipate that this
approach will be especially valuable in handling cluttered environments
containing many similar objects. As a step in this direction, we propose the
Object Attention Modelling Transformer OAT architecture for scanpath
prediction at the object level. It consists of four modules: object embedding,
encoder, decoder, and object attention. The object embedding module extracts
visual and geometrical features for each object. For visual search, the
transformer encoder combines information about the target and the objects and
their positions in the scene. The transformer decoder assigns to each object
the probability that it will be the next fixated object, based on cross
attention computed between the past sequence of fixated objects and the
encoder representation of the target and scene. Predictions of the next
fixation can be produced by sampling from this distribution. Figure 1
illustrates our framework using an example from the Amazon book cover dataset
[30], where users search for a target book cover in a 2D array of book covers.

Our contributions can be summarized as follows:

  1. 1.

To the best of our knowledge, we propose the first approach to predict gaze
scanpaths at the object level. We study visual search, but believe that this
architecture is a valuable starting point for object-level modelling of other
tasks.

  2. 2.

We propose two modifications to the transformer architecture to optimize
performance on this task. The first is a 2D positional encoding algorithm that
better captures the spatial relationships between objects. The second is a
scalable object attention module that enables OAT to generate predictions for
scenes containing varying numbers of objects without retraining.

  3. 3.

For model validation, we propose a new metric for comparing generated
scanpaths with human scanpaths, which compares the distributions of task-
relevant gaze behaviors based on psychological processes of visual attention.

  4. 4.

OAT achieves state-of-the-art performance compared to baselines using both
previously proposed standard metrics, as well as our new metric. Our predicted
scanpaths differ by only 3.1% from human scanpaths.

##  2 Related Work

While there is no work on gaze scanpath prediction at object level, there has
been extensive investigation into gaze scanpath prediction at the pixel level.
Early work by [9] on the visual attention system and the development of
saliency benchmarks [11, 10, 15] have paved the way for this research. Most of
the existing work in gaze scanpath modelling [27, 19, 20, 4] focuses on
predicting free-viewing behaviour, where human attention is driven solely by
visual input without a specific target. These models generate saliency maps to
capture spatial dependencies, but do not adequately consider temporal
dependencies in scanpaths.

In goal-directed scanpath prediction, attention is guided by a top-down goal,
and sequence modelling algorithms are used to generate the sequence while
capturing correlations between the target and the sequence. Zelinsky et al.
[36, 37] employed Inverse Reinforcement Learning (IRL) to learn a target-
specific reward function and evaluated their model on a self-collected dataset
based on the COCO dataset. Yang et al. [34] extended the IRL algorithm and
tested it on the larger-scale COCO-Search18 dataset. They later introduced a
more generalized scanpath prediction model that incorporates a transformer-
based architecture and a simulated foveated retina to emulate human dynamic
visual working memory [35]. Mondal et al. [21] also applied a transformer-
based architecture for goal-directed scanpath prediction, combining visual and
semantic features to decode fixations. Tononi et al. [31] propose a
Transformer-based architecture for gaze target detection, but focusing on
predicting a single point a person inside the image is looking at, rather than
predicting the scanpath of an external observer.

Previous models of goal-directed search have primarily focused on predicting
scanpaths at the pixel level, rather than at the object level, as we propose.
Pixel-based models may suffice for natural-scene images where objects are
distinct and viewers can rely on prior knowledge about likely spatial
locations. However, in more challenging scenarios such as scenes with random
clutter, disorganization, or similar-looking distractors, pixel-based models
may not be as appropriate. These difficult scenarios, as exemplified by the
Amazon book cover dataset, require more fixations (8.95) on average compared
to the COCO-Search18 dataset (3.5 fixations). The experiments conducted in
this study demonstrate that object-level modeling outperforms pixel-level
modeling in these challenging scenarios.

Our model is based on a transformer architecture similar to that proposed by
Mondal et al. [21]. However, OAT introduces distinct differences. Firstly, it
incorporates a target-dependent encoding of the input image by including the
target as an extra token in the encoder transformer. This allows objects
similar to the target to be encoded differently from dissimilar objects.
Secondly, OAT includes past fixations in the decoder input instead of
predicting all fixations at once. Lastly, OAT’s object attention module
predicts the fixated object, rather than the fixated pixel.

##  3 OAT Architecture

Figure 2: An overview of OAT. The input to the encoder is a target object, an
image containing the target object and other distractor objects, and the input
to the decoder is the sequence of previously and currently fixated objects.
The output is a probability distribution over objects being fixated next. The
model repeats the process until the next token is the end token <EOS>. This is
the process of predicting one gaze scanpath.

Figure 2 presents an overview of OAT. It consists of an object embedding
module, $N_{e}$ transformer encoder blocks and $N_{d}$ decoder blocks, and
$N_{c}$ object attention blocks. The encoder and decoder architecture follow
[32]. We first partition the image into objects and append the target object
to the beginning of the object sequence. The object embedding module generates
the embedding sequence $F_{e}$, which is then processed by encoder blocks to
obtain the encoder hidden states $H_{e}$. The decoder computes the cross
attention between $H_{e}$ and the embedding sequence from the decoder $F_{d}$,
and outputs the hidden states of the decoder $H_{d}$. $H_{e}$ and $H_{d}$ are
processed by the object attention module, which yields an output $O$. Finally,
this is converted to a probability distribution over the next fixated object
by the Softmax function.

#### Object Embedding and Encoder.

The object embedding module computes a $p$-dimensional visual descriptor for
each object. For the databases we consider here, objects are placed in regular
arrays. Each object corresponds to a rectangular patch. We use a Convolutional
Neural Network (CNN) followed by ReLU and feed-forward layers applied to the
patch to extract visual features. More complex layouts, where objects have
variable sizes and potentially overlap/occlude, can be handled by first
segmenting the image, then extracting object visual descriptors.

For each object, we also compute a $p$-dimensional positional encoding,
indicating its location in the image, as described in the next section. This
is concatenated with the object’s visual descriptor.

We also compute a visual descriptor and positional embedding for the target
object. If there are $m$ objects in the image and one target, we then have a
set of object embeddings $F_{e}\in\mathbb{R}^{(m+1)\times(2p)}$ after the
object embedding module.

Subsequently, the set of embeddings $F_{e}$ is fed into $n_{e}$ stacked
identical transformer encoder blocks. The encoder outputs the hidden states of
the sequence, $H_{e}\in\mathbb{R}^{(m+1)\times h}$, where $h$ is the dimension
of hidden states of encoder blocks.

#### Distance-based Positional Encoding.

Figure 3: (left) Cosine similarity of the PE at the centre position and the
PEs at other positions for the PE in [32]. (right) A Gaussian distribution.

Conventional fixed positional encoding (PE) algorithms [32, 6] use sinusoidal
functions to generate positional encodings. While these encodings can
differentiate between different locations, they are not designed to encode
information about the spatial distance between different locations. Figure 3
(left image) illustrates the cosine similarity between the PE of the centre
position and the PE of the other positions in a 2D space for the PE defined in
[32]. While there is some decay in similarity as the distance from the center
increases, this decay is slow. Thus, although there are differences between
PEs at locations close and far from the center, these are small. Spatial
positions would be better encoded if cosine similarity between PEs decayed
more rapidly, such as by following the Gaussian distribution as illustrated in
Figure 3 (right image). While learnable positional encodings might be able to
capture this information, they are not explicitly designed to do so and thus
would require inordinate amounts of training data to learn this if it is
relevant to the task.

Spatial distance is an important factor in determining saccades. Most saccades
are to close objects (see Figure 7). Large saccades are relatively rare. Given
the importance of spatial distance in determining scanpath behavior, we
propose learnable PE that more explicitly encodes spatial distance between
locations. We consider an encoding of three spatial dimensions: $(x,y)$ as
image locations, and $z\in\\{0,1\\}$ as a target indicator, constructed by
concatenating three 1D PEs.

For the 1D PE, we learn the set of PEs
$P_{i}\in\mathbb{R}^{L\times\frac{p}{3}}$ where $i$ indicates spatial location
by ensuring that cosine similarity between PEs at different locations decays
approximately following a zero-mean Gaussian distribution, by minimizing

$\mathbf{l_{PE}}=\frac{2}{L\times(L-1)}\sum_{i=0}^{L}\sum_{j=i+1}^{L}(\frac{P_{i}P_{j}}{|P_{i}||P_{j}|}-f(i,j))+\lambda
r(P),$ (1)

where $f(i,j)=e^{-\frac{1}{2}\left(\frac{|i-j|}{\sigma}\right)^{2}}$,
$r(\cdot)$ is the regularization term that constrains each vector from the
learned PE matrix to have a unit magnitude and $\lambda$ is the weighting
factor. $\sigma$ denotes the standard deviation of the Gaussian distribution.

We scale the PEs by $\alpha$, to make their magnitude similar to the magnitude
of the visual feature branch in the object embedding module. For an object
with 2D coordinates $(x,y)$, its positional embedding is $P(x)\oplus P(y)$,
where $\oplus$ stands for the concatenation. Since we have an extra dimension
$z$ representing the target, we also concatenate the embedding with $P(z)$.

#### Decoder.

These $n_{d}$ decoder blocks have a similar structure to the encoder blocks,
except that the decoder has cross-attention layers, where the keys and values
come from the encoder output and queries come from the decoder input. The
output of the decoder is the hidden states $H_{d}\in\mathbb{R}^{l\times h}$.
Then we input the hidden state of the last fixation
$H_{d}^{last}\in\mathbb{R}^{1\times h}$ and all the hidden states from the
encoder $H_{e}$ into the object attention modelling module.

#### Object Attention (OA)

In this final step, we model object attention based on the hidden states of
the encoder and decoder. $H_{e}$ serves as key and $H_{d}^{last}$ is query.
The feed-forward layer computes attention embedding of the encoder and
decoder, $A_{e}\in\mathbb{R}^{(m+1)\times k}$ and $A_{d}\in\mathbb{R}^{1\times
k}$, respectively, where $k$ is the attention embedding size. Then it outputs
the multiplication of two features: $O=\frac{A_{e}\times A_{d}.T}{\sqrt{h}}$.
We take the average of $n_{c}$ outputs from $n_{c}$ identical OA blocks.

Finally, we apply a Softmax to the output of the last module and obtain a
probability distribution $D\in\mathbb{R}^{m+1}$, where the extra probability
represents the end token. If the end token has the largest probability, the
generation is terminated. The next fixation can be either taken as the one
with the largest probability (greedy sampling) or taken by sampling randomly
from this distribution.

#### Training and Testing.

The OAT takes as input a target object and an image containing the target and
distractors. It outputs a gaze scanpath prediction.

During training, the input to the decoder is shifted because we append a start
token [BOS] to the beginning of the sequence. At each step, we compute the
cross-entropy loss between the output distribution and the ground truth
distribution at that step (a binary matrix where only one value is one). Due
to the nature of our task, we do not apply any pre-training or data
augmentation. For inference, the decoder starts from [BOS] and predicts
subsequent outputs iteratively, as it takes the newly generated output as the
next input.

##  4 Experimental Results

To assess the performance of OAT, we evaluated it on a public dataset and a
dataset we collected. To account for the longer and more variable scanpaths in
these datasets in comparison to datasets with less clutter, like COCO-
Search18, we introduce a new metric based on human behavior. We also report
performance using common metrics used previously to facilitate comparison with
other approaches. A series of experiments were conducted to validate the
interpretability and generalization of the proposed model.

###  4.1 Datasets

Table 1: Dataset Summary

|  | # Target
---
category
Shelf size | # Images | # Scanpaths | Mean/Std. length | Target area | Resolution
Amazon book cover | 5 | 6 * 14 | 100 | 585 | 12.9/9.1 | 1.2% | 1600 * 2554
Yogurt/wine | 40 | 3 * 9 / 2 * 11 | 196 | 858 | 7.7/3.3 | 3.2% / 3.9% | 1050 * 1680

Figure 4: Example product array images.

We trained and evaluated OAT on two datasets. The summary of the datasets and
examples images are shown in Table 1 and Figure 4, respectively. The first is
the Amazon book cover dataset [30]. It contains 102 images of 6 by 14 arrays
of randomly shuffled book covers. The dataset provides fixation data from six
participants on average per image searching for a target book cover. To study
the applicability of our model to other scenarios, we collected another
similar dataset, but with 2D arrays of yogurt and wine. The wine product
arrays have 2 $\times$ 11 packages from 21 brands. The yogurt product array
includes 3 $\times$ 9 packages from 13 brands. Each subject viewed around 30
shelf-target combinations. More details about the dataset collection are
provided in the supplementary materials.

We convert the fixations into their corresponding objects based on bounding
boxes placed around the objects.

###  4.2 Implementation Settings

We used four encoder and decoder blocks and two OA blocks. Each layer
contained four attention heads. The $p$ of the patch embedding module is 128.
The size of hidden states $h$ was 256. The $\mu$ and $\sigma$ parameters in
the PE loss function were 0 and 2, respectively. We set $\lambda$ equal to 1
and $L$ equal to 11, the largest dimension size of the shelf in our dataset.
The PE matrix was trained with a learning rate of 0.01 and 10000 iterations.
We trained OAT with a batch size of 20 and a learning rate of 1e-4. The data
was randomly split into 8:1:1 as training/validation/testing data.

###  4.3 Metrics

Human scanpaths for the same stimulus vary from trial to trial. Therefore, for
all the baselines and OAT, we generated 100 trajectories by sampling from the
output distribution and averaged the results across these trajectories. We
evaluated the models using the following four metrics.

(i) Sequence Score (SS) [34]: SS is commonly used in pixel-level scanpath
prediction. It clusters fixations and converts the output sequence of
fixations into cluster IDs, and then uses a string matching algorithm to
measure the similarity between two strings. Since we have object-level
outputs, we treat object output as the cluster ID and directly use the string-
matching algorithm.

(ii) Fixation Edit Distance (FED) [21]: Similar to the SS, the FED converts
scanpaths to strings but uses the Levenshtein distance [17] for string-
matching.

(iii) Behavioural measures: It has been observed that individuals often
revisit previously inspected items when searching for a specific target due to
the inherent limitations of working memory, necessitating the need for
rehearsal [1]. Commonly used string-matching algorithms, fail to adequately
account for the distinct behavioural patterns associated with the retrieval of
new and previously encountered objects. This aspect has been overlooked in
previous research and warrants comprehensive evaluation.

To this end, we propose a new metric to measure these behavioural patterns.
Gaze patterns involve three attention states: search (viewing a new object for
exploration), revisits (re-viewing an object), and successive re-fixations on
the same object to confirm its identity [22].We propose to quantify the extent
to which OAT conforms to human behaviour by comparing the percentages of
search, revisits, and refixations; the accuracy of finding the correct target
(Accuracy); the average length of the scanpaths in fixations (Avg. Length);
and the average absolute percentage difference between these values and human
scanpaths (Overall). Mathematically, for one predicted scanpath
$\mathcal{S}=\\{O_{1},O_{2},...,O_{n}\\}$ which consists of $n$ objects, we
calculate the length $n$, the target accuracy $\mathbbm{1}(O_{n}=T)$ where $T$
is the target object, search $\frac{a}{n-1}$, revisit $\frac{b}{n-1}$, and
refix $\frac{c}{n-1}$. For $n-1$ saccades in the scanpath, $a+b+c=n-1$. $a$ is
the number of fixations on objects that were not viewed before. $b$ is the
number of fixations on objects that have already been viewed but not in the
last fixation, and $c$ is the number of fixations on the same object as the
last fixated one.

(iv) Multi Match (MM) [5]: To assess the performance on pixel level with other
pixel-level based baselines, we convert the object-level prediction of OAT to
the pixel-level predictions, by assuming the fixations fall on the center
pixels of the objects, and compare this pixel sequence with the pixel-level
outputs of other baselines. MM measures the scanpath similarity from four
perspectives: shape, direction, length, and position scores.

###  4.4 Quantitative Performance Comparison

To assess the efficacy of OAT, we conducted a comparative analysis with four
baseline methods. We considered two types of baseline methods:

(i) IID random sampling: Fixations are randomly selected from one of two
distributions over the objects independently for each fixation. Stopping is
determined by a geometric random process where the stopping probability is
determined by the average sequence length in the training dataset. The two
distributions were: (1) Random (Uniform over all objects) (2) Center (Decaying
with distance from the centre following a Gaussian distribution [11].

(ii) Temporal modelling of fixation probabilities conditioned on previous
fixations: (1) WTA (Winner-take-all [13]). A saliency map is computed as in
[9], and the initial fixation is determined at the location of maximum
saliency. Inhibition-of-return is applied and subsequent fixations are
selected based on the maximum saliency within the updated map. This process is
repeated iteratively until the length reaches the average sequence length in
the training dataset. (2) IRL [34]: IRL applies inverse reinforcement learning
to scanpath prediction and learns a target-related reward function. (3)
Gazeformer [21]: The Gazeformer is a transformer-based architecture. It
applies a language model (LM) to generate an embedding of a textual
description of the target and fuses it with the global features from the
image. Since the targets in our datasets are specified as images, rather than
text, we applied image captioning to each target image using Blip [18] to
generate a textual description of each target.

The baselines output the sequences of pixel-level fixations. We convert pixel-
level fixations to object-level fixations according to the object bounding
boxes into which the pixel-level fixation fall into, except when calculating
the MM metric.

Table 2: Comparison with baselines in Amazon book cover dataset. Percentages
in parentheses indicate the percentage change relative to human behaviour.

| Behaviour Difference |  FED $\downarrow$ |  SS $\uparrow$
---|---|---|---
| Search (%) | Revisit (%) | Refix (%) |  | Accuracy
---
(%)
| Avg. Length
---
(# fixations)
Overall $\downarrow$
Human | 85.8 | 2.3 | 11.9 | 91.7 | 8.4 | 0 | 0 | 1
Random | 95.5 (+0.11) | 3.6 (+0.56) | 0.9 (-0.92) | 1.1 (-0.99) | 8.8 (+0.05) | 0.530 | 11.73 | 0.042
Center [11] | 86.5(+0.01) | 10.3(+3.48) | 3.2(-0.73) | 1.3 (-0.98) | 8.9(+0.06) | 1.053 | 11.73 | 0.059
WTA [13, 9] | 84.3(-0.02) | 12.0(+4.22) | 3.7(-0.69) | 1.2(-0.99) | 8.8(+0.05) | 1.186 | 11.60 | 0.056
IRL [34] | 83.4 (-0.03) | 14.1 (+5.13) | 2.5(-0.79) | 2.5(-0.97) | 12.3(+0.46) | 1.466 | 13.70 | 0.051
Gazeformer [21] | 81.2 (-0.05) | 3.2(+0.39) | 15.6(+0.31) | 1.7(-0.98) | 6.0(-0.29) | 0.404 | 8.65 | 0.098
OAT |  85.3(-0.01) |  3.0 (+0.30) |  11.7 (-0.02) |  89.4 (-0.03) |  8.5(+0.01) | 0.074 | 0.68 | 0.950

Table 3: Comparison with baselines in our collected dataset. Percentages in
parentheses indicate the percentage change relative to human behaviour.

| Behaviour Difference |  FED $\downarrow$ |  SS $\uparrow$
---|---|---|---
| Search (%) | Revisit (%) | Refix (%) |  | Accuracy
---
(%)
| Avg. Length
---
(# fixations)
Overall $\downarrow$
Human | 63.9 | 4.4 | 31.7 | 84.9 | 8.1 | 0 | 0 | 1
Random | 88.6(+0.39) | 8.6 (+0.95) | 2.8 (-0.91) | 4.3 (-0.95) | 7.6 (-0.06) | 0.657 | 10.09 | 0.093
Center [11] | 85.4 (+0.34) | 10.5 (+1.39) | 4.0 (-0.87) | 4.3 (-0.95) | 7.7 (-0.05) | 0.719 | 9.98 | 0.119
WTA [13, 9] | 72.0 (+0.13) | 18.8 (+3.27) | 9.2 (-0.71) | 3.3 (-0.96) | 7.7 (-0.06) | 1.032 | 10.05 | 0.097
IRL [34] | 83.1(+0.3) | 14.4(+2.27) | 2.5(-0.92) | 49.0(-0.42) | 12.4(+0.54) | 0.901 | 11.72 | 0.193
Gazeformer [21] | 29.9(-0.53) | 6.7(+0.52) | 63.4(1) | 9.3(-0.89) | 7.2(-0.11) | 0.614 | 7.69 | 0.177
OAT |  62.9 (-0.02) |  4.0 (-0.09) |  33.0 (+0.04) |  85.7 (+0.01) |  7.8 (-0.04) | 0.031 | 7.19 | 0.346

Table 4: Multi Match comparison in Amazon book cover/our collected dataset

| Vector | Direction | Length | Position | Average
---|---|---|---|---|---
Random | 0.846/0.866 | 0.603/0.593 | 0.786/0.802 | 0.675/0.687 | 0.727/0.737
Center | 0.906/0.882 | 0.621/0.601 | 0.886/0.830 | 0.711/0.708 | 0.781/0.755
WTA | 0.910/0.918 | 0.620/0.609 | 0.892/0.888 | 0.694/0.680 | 0.779/0.774
Gazeformer | 0.937/0.956 | 0.739/0.588 | 0.918/0.922 | 0.828/0.793 | 0.855/0.815
IRL | 0.861/0.881 | 0.641/0.645 | 0.809/0.830 | 0.731/0.744 | 0.761/0.775
OAT |  0.978/0.942 |  0.916/0.603 |  0.972/0.922 |  0.964/0.820 | 0.958/0.822

As shown in Table 2 (Amazon book cover dataset) and Table 3 (our collected
dataset), OAT outperforms all baseline methods across all metrics. Notably,
the behaviour difference between scanpaths generated by the OAT and human
scanpaths is only 3.12% in our collected dataset, underscoring its
effectiveness in emulating human attention patterns. IID random sampling shows
poor performance across all metrics, due to the absence of temporal
dependencies. Among the baseline methods that include temporal dependencies,
WTA only updates the distribution by zeroing out the neighbour area of the
last viewing point. Thus, it does not capture long-range spatiotemporal
dependencies. Because the Gazeformer appends the target to each patch token
independently, it does not enable the target representation to be influenced
by the global context, nor the integration of global context into local patch
representations. In addition, because it predicts all the fixations in one go,
it does not account for past gaze history when generating each fixation
sequence. While this may be effective when scanpaths are short, e.g., as in
COCO-Search18, it fails to capture the long-distance temporal dependencies
within individual scanpaths. Similarily, IRL leverages global image features
and fails to capture the informative features of the objects. In contrast, our
OAT outputs every fixation in a sequence, and incorporates memory
reinforcement through OA, allowing for accurate sequence modelling of gaze
scanpaths.

Table 4 shows the pixel-level MultiMatch comparison in two datasets. Despite
the relatively coarse object-level predictions, our method still outperforms
other baselines in average scanpath similarity, since it more accurately
estimates the objects being fixated, thus leading to closer pixel-level
predictions despite object-level quantization.

###  4.5 Scanpath Visualization in Spatial Dimension

Figure 5: Heatmap of scanpaths predicted by OAT and generated by human.

We generated object-level heatmaps for predicted and human scanpaths by
calculating the percentage of time each objects was viewed. For OAT, we
averaged over 100 predicted scanpaths. The results are visualized in Figure 5.
Predicted heatmaps closely align with the human heatmaps. Only minor
discrepancies in the probability distribution are observed between the
predicted and human scanpaths. This heatmap analysis shows that OAT can
effectively capture object-level attention dynamics.

###  4.6 Generalization to Unknown Categories

Figure 6: Generalization ability of OAT.

In order to assess OAT’s predictions on unseen data, we adopt a random
selection approach in our collected dataset whereby a single pair of layout
and target object is chosen as the testing dataset, while the remaining
dataset is utilized for training. The observed behaviours in Figure 6
demonstrate that OAT exhibits comparable patterns to those observed in human
scanpaths. This outcome not only validates the effectiveness of OAT in
accurately modeling consumer scanpaths but also underscores its ability to
generalize well to previously unseen layouts and target objects.

###  4.7 Parameter Analysis of Distance-based PE v.s. End-to-End Learnable PE
and Relative PE

Table 5: Parameter analysis of DPE, E2E and RPE.

| Behaviour Difference |  FED $\downarrow$ |  SS $\uparrow$
---|---|---|---
| Search (%) | Revisit (%) | Refix (%) |  | Accuracy
---
(%)
| Avg. Length
---
(# fixations)
Overall $\downarrow$
Human | 85.8 | 2.3 | 11.9 | 91.7 | 8.4 | 0 | 0 | 1
E2E PE | 85.2(-0.01) | 3.1(+0.35) | 11.7(-0.02) | 89.1(-0.03) | 8.7(+0.04) | 0.088 | 2.076 | 0.830
E2E PE + RPE | 85.4(-0.00) | 3.0(+0.30) | 11.6(-0.03) | 86.9(-0.05) | 8.9(+0.06) | 0.090 | 1.282 | 0.916
DPE + RPE | 85.3(-0.01) | 3.0(+0.30) | 11.7(-0.02) | 81.6(-0.11) | 8.4(0.00) | 0.088 | 2.076 | 0.830
DPE | 85.3(-0.01) | 3.0(+0.30) | 11.7(-0.02) | 89.4(-0.03) | 8.5(+0.01) | 0.074 | 0.681 | 0.950

(a)

(b)

(c)

Figure 7: Distribution of Manhatten distances of the saccades in the predicted
scanpaths. Distribution of DPE is closer to human than E2E PE.

We compared our proposed Distance-based PE (DPE) with an end-to-end (E2E)
learnable PE and Relative PE (RPE) in Amazon book cover dataset to examine the
impact of having prior geometric knowledge in the PE component. In the E2E PE
approach, we randomly initialized a PE matrix of equivalent size and
subsequently trained it in an end-to-end manner as described in [6]. RPE [28]
adds new parameter vectors on top of an existing absolute encoding to encode
relative positions. The results in Table 5 reveal that DPE outperforms the E2E
PE and RPE by 16% in behavioural metric. DPE requires fewer parameters, which
gives better performance since gaze training data is limited.The result
suggests that incorporating geometry-based priors helps OAT to mimic the human
behaviour of looking at closer objects due to foveated vision.

To further investigate this effect, we analyzed the distribution of Manhattan
distances of all saccades in the predicted scanpaths. In Figure 7, it can be
observed that the distribution for the DPE is closer to human behaviour than
for the E2E PE. E2E PE generates a larger number of longer saccades, than
observed in human data. This suggests that E2E PE does not encode spatial
distance as effectively as DPE. Understanding this behavior is crucial for
replicating human saccade patterns. Shorter saccades are more prevalent due to
energy constraints and the limited information available about peripheral
objects. While E2E PE might learn such encoding with sufficient training data,
for smaller datasets, incorporating prior knowledge about image geometry via
the DPE yields better performance.

###  4.8 Importance of object-centric modelling

To determine whether our model depends upon path tokens encoding single
objects, we increased the grid size in the Amazon book covers experiments from
6 $\times$ 14 to 12 $\times$ 28 (so that patches contained only parts of
objects) and decreased it to 3 $\times$ 7, so that (so that patches contained
four objects). We mapped fine-scale grid fixations to corresponding
objects/groups at coarser scales. The results are shown in Table 6. We see
large degradations in performance when patches correspond to parts of objects
(Grid 12 $\times$ 28), but relatively small degradations in performance when
we model at the object level but consider fixations on groups of objects.

Table 6: Analysis of object-centric attribute (mapped to grid size 6 $\times$
14 / 3 $\times$ 7)

| Behaviour Difference |  FED $\downarrow$ |  SS $\uparrow$
---|---|---|---
| Search (%) | Revisit (%) | Refix (%) |  | Acc
---
(%)
| Avg. Len
---
(# fixations)
Overall $\downarrow$
Human | 85.8/70.0 | 2.3/5.5 | 11.9/24.5 | 91.7/93.3 | 8.4/8.4 | 0 | 0 | 1
| Grid 12 $\times$ 28
---
78.7/64.1 | 11.6/19.8 | 9.6/16.1 | 36.2/39.1 | 15.2/15.2 | 1.14/0.89 | 13.10/12.39 | 0.31/0.35
OAT (6 $\times$ 14) | 85.3/73.3 | 3.0/5.5 | 11.7/21.2 | 89.4/91.7 | 8.5/8.2 |  0.07/0.05 |  0.68/0.81 |  0.95/0.90
Grid 3 $\times$ 7 | -/69.9 | -/5.5 | -/24.6 | -/92.9 | -/8.5 | -/0.01 | -/0.78 | -/0.93

###  4.9 Improving revisit/refix behaviour by the cross attention

We found that the cross-attention mechanism is critical in modelling revisits
and refixations. For 20 shelf-target pairs and over 100 generated trajectories
each, we looked at how the probabilities of fixating on the object fixated in
fixation 2 at subsequent fixations (3, 4, 5 and 6) changed when the object in
the fixation record at fixation 2 was replaced by a random object. the changes
were $-3.42\%$, $18.14\%$, $21.53\%$, and $33.22\%$, respectively. The
percentage change in probability for fixation 3 was negative ($-3.42\%$),
which is consistent with the increased percentage of re-fixations by OAT in
Table 2 and 3 compared to most baselines except Gazeformer. It was positive
for fixations 4 through 6, which is consistent with the decreased percentage
of re-visits by OAT compared to all baselines. This brought OAT’s behaviour
closer to human behaviour.

###  4.10 Ablation Study

Table 7: Ablation Study of two novel components. Percentages in parentheses
indicate the percentage change relative to human behaviour.

| Behaviour Difference |  FED $\downarrow$ |  SS $\uparrow$
---|---|---|---
| Search (%) | Revisit (%) | Refix (%) |  | Accuracy
---
(%)
| Avg. Length
---
(# fixations)
Overall $\downarrow$
| Human
---
63.9 | 4.4 | 31.7 | 84.9 | 8.1 | 0 | 0 | 1
| w/o Both
---
65.5 (+0.03) | 24.0 (+4.45) | 10.4 (-0.67) | 41.0 (-0.52) | 9.9 (+0.22) | 0.533 | 11.66 | 0.224
| w/o DPE
---
59.2 (-0.07) | 5.5(+0.25) | 35.3(+0.11) | 86.9(+0.02) | 9.1(+0.12) | 0.119 | 7.70 | 0.327
| w/o OA
---
61.1 (-4.4) | 8.2(+0.86) | 32.5(+0.03) | 74.9(-0.12) | 8.5(+0.05) | 0.169 | 7.58 | 0.316
OAT |  62.9(-0.02) |  4.0(-0.09) |  33.0(+0.04) |  85.7(+0.01) |  7.8(-0.04) | 0.031 | 7.19 | 0.346

We evaluated the impact of our novel components, DPE and OA, through ablation
experiments on our dataset in Table 7. Because models without the OA cannot
handle different layouts, we trained two separate models for wine and yogurt
and calculate the average outcomes. Removing DPE and OA significantly degraded
performance. The behavioral difference was 17 times worse. Removing the DPE
component alone led to 3.84 times degradation. Removal of OA resulted in a
5.45 times degradation. The DPE component improves the accuracy of the modeled
shelf geometry, while the OA component captures dependencies and enhances
model flexibility. Combining these components improves overall performance.

##  5 Conclusion

In this paper, we propose a transformer-based architecture OAT for modelling
the gaze scanpaths at the object level. OAT outperforms all prior baselines in
modelling human behaviour. It also demonstrates a remarkable ability to
generalize on unseen layouts and targets. Moving forward, potential extensions
to this work could involve applying to searching for objects with non-uniform
shapes, which is more common in real world search tasks.

## References

  * [1] Beck, M.R., Peterson, M.S., Vomela, M.: Memory for where, but not what, is used during visual search. Journal of Experimental Psychology: Human Perception and Performance 32(2), 235 (2006)
  * [2] Bennett, C.R., Bex, P.J., Merabet, L.B.: Assessing visual search performance using a novel dynamic naturalistic scene. Journal of Vision 21(1), 5–5 (2021)
  * [3] Chen, X., Jiang, M., Zhao, Q.: Predicting human scanpaths in visual question answering. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 10876–10885 (2021)
  * [4] Cornia, M., Baraldi, L., Serra, G., Cucchiara, R.: Predicting human eye fixations via an lstm-based saliency attentive model. IEEE Transactions on Image Processing 27(10), 5142–5154 (2018)
  * [5] Dewhurst, R., Nyström, M., Jarodzka, H., Foulsham, T., Johansson, R., Holmqvist, K.: It depends on how you look at it: Scanpath comparison in multiple dimensions with multimatch, a vector-based approach. Behavior research methods 44, 1079–1100 (2012)
  * [6] Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., et al.: An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929 (2020)
  * [7] Egly, R., Driver, J., Rafal, R.D.: Shifting visual attention between objects and locations: evidence from normal and parietal lesion subjects. Journal of Experimental Psychology: General 123(2), 161 (1994)
  * [8] Einhäuser, W., Spain, M., Perona, P.: Objects predict fixations better than early saliency. Journal of vision 8(14), 18–18 (2008)
  * [9] Itti, L., Koch, C., Niebur, E.: A model of saliency-based visual attention for rapid scene analysis. IEEE Transactions on pattern analysis and machine intelligence 20(11), 1254–1259 (1998)
  * [10] Judd, T., Durand, F., Torralba, A.: A benchmark of computational models of saliency to predict human fixations (2012)
  * [11] Judd, T., Ehinger, K., Durand, F., Torralba, A.: Learning to predict where humans look. In: 2009 IEEE 12th international conference on computer vision. pp. 2106–2113. IEEE (2009)
  * [12] Kahneman, D., Treisman, A., Gibbs, B.J.: The reviewing of object files: Object-specific integration of information. Cognitive psychology 24(2), 175–219 (1992)
  * [13] Koch, C., Ullman, S.: Shifts in selective visual attention: towards the underlying neural circuitry. Human neurobiology 4(4), 219–227 (1985)
  * [14] Krajancich, B., Kellnhofer, P., Wetzstein, G.: Optimizing depth perception in virtual and augmented reality through gaze-contingent stereo rendering. ACM Transactions on Graphics (TOG) 39(6), 1–10 (2020)
  * [15] Kummerer, M., Wallis, T.S., Bethge, M.: Saliency benchmarking made easy: Separating models, maps and metrics. In: Proceedings of the European Conference on Computer Vision (ECCV). pp. 770–787 (2018)
  * [16] Van der Lans, R., Pieters, R., Wedel, M.: Online advertising suppresses visual competition during planned purchases. Journal of Consumer Research 48(3), 374–393 (2021)
  * [17] Levenshtein, V.I., et al.: Binary codes capable of correcting deletions, insertions, and reversals. In: Soviet physics doklady. vol. 10, pp. 707–710. Soviet Union (1966)
  * [18] Li, J., Li, D., Xiong, C., Hoi, S.: Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation. In: International Conference on Machine Learning. pp. 12888–12900. PMLR (2022)
  * [19] Linardos, A., Kümmerer, M., Press, O., Bethge, M.: Deepgaze iie: Calibrated prediction in and out-of-domain for state-of-the-art saliency modeling. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 12919–12928 (2021)
  * [20] Lou, J., Lin, H., Marshall, D., Saupe, D., Liu, H.: Transalnet: Towards perceptually relevant visual saliency prediction. Neurocomputing 494, 455–467 (2022)
  * [21] Mondal, S., Yang, Z., Ahn, S., Samaras, D., Zelinsky, G., Hoai, M.: Gazeformer: Scalable, effective and fast prediction of goal-directed human attention. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 1441–1450 (2023)
  * [22] Niebur, E.: Computational architectures for attention. The attentive brain (1998)
  * [23] Nuthmann, A., Henderson, J.M.: Object-based attentional selection in scene viewing. Journal of vision 10(8), 20–20 (2010)
  * [24] O’Craven, K.M., Downing, P.E., Kanwisher, N.: fmri evidence for objects as the units of attentional selection. Nature 401(6753), 584–587 (1999)
  * [25] Pajak, M., Nuthmann, A.: Object-based saccadic selection during scene perception: Evidence from viewing position effects. Journal of vision 13(5), 2–2 (2013)
  * [26] Rivu, R., Abdrabou, Y., Pfeuffer, K., Esteves, A., Meitner, S., Alt, F.: Stare: gaze-assisted face-to-face communication in augmented reality. In: ACM Symposium on Eye Tracking Research and Applications. pp. 1–5 (2020)
  * [27] Schwinn, L., Precup, D., Eskofier, B., Zanca, D.: Behind the machine’s gaze: Neural networks with biologically-inspired constraints exhibit human-like visual attention. arXiv preprint arXiv:2204.09093 (2022)
  * [28] Shaw, P., Uszkoreit, J., Vaswani, A.: Self-attention with relative position representations. arXiv preprint arXiv:1803.02155 (2018)
  * [29] Shen, Y., Mo, X., Krisciunas, V., Hanson, D., Shi, B.E.: Intention estimation via gaze for robot guidance in hierarchical tasks. In: Annual Conference on Neural Information Processing Systems. pp. 140–164. PMLR (2023)
  * [30] Stauden, S., Barz, M., Sonntag, D.: Visual search target inference using bag of deep visual words. In: Trollmann, F., Turhan, A. (eds.) KI 2018: Advances in Artificial Intelligence - 41st German Conference on AI, Berlin, Germany, September 24-28, 2018, Proceedings. Lecture Notes in Computer Science, vol. 11117, pp. 297–304. Springer (2018). https://doi.org/10.1007/978-3-030-00111-7_25, https://doi.org/10.1007/978-3-030-00111-7_25
  * [31] Tonini, F., Dall’Asen, N., Beyan, C., Ricci, E.: Object-aware gaze target detection. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 21860–21869 (2023)
  * [32] Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, Ł., Polosukhin, I.: Attention is all you need. Advances in neural information processing systems 30 (2017)
  * [33] Wedel, M., Pieters, R., van der Lans, R.: Eye tracking methodology for research in consumer psychology. In: Handbook of Research Methods in Consumer Psychology, pp. 276–292. Routledge (2019)
  * [34] Yang, Z., Huang, L., Chen, Y., Wei, Z., Ahn, S., Zelinsky, G., Samaras, D., Hoai, M.: Predicting goal-directed human attention using inverse reinforcement learning. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 193–202 (2020)
  * [35] Yang, Z., Mondal, S., Ahn, S., Zelinsky, G., Hoai, M., Samaras, D.: Predicting human attention using computational attention. arXiv preprint arXiv:2303.09383 (2023)
  * [36] Zelinsky, G., Yang, Z., Huang, L., Chen, Y., Ahn, S., Wei, Z., Adeli, H., Samaras, D., Hoai, M.: Benchmarking gaze prediction for categorical visual search. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops. pp. 0–0 (2019)
  * [37] Zelinsky, G.J., Chen, Y., Ahn, S., Adeli, H., Yang, Z., Huang, L., Samaras, D., Hoai, M.: Predicting goal-directed attention control using inverse-reinforcement learning. Neurons, behavior, data analysis and theory 2021 (2021)


# Technical Opinion: From Animal Behaviour to Autonomous Robots

Chinedu Pascal Ezenkwu  School of Engineering, University of Aberdeen, United
Kingdom  Electrical/Electronic & Computer Engineering Department, University
of Uyo, Nigeria  Andrew Starkey  School of Engineering, University of
Aberdeen, United Kingdom

###### Abstract

With the rising applications of robots in unstructured real-world
environments, roboticists are increasingly concerned with the problems posed
by the complexity of such environments. One solution to these problems is
robot autonomy. Since nature has already solved the problem of autonomy it can
be a suitable model for developing autonomous robots. This paper presents a
concise review on robot autonomy from the perspective of animal behaviour. It
examines some state-of-the-art techniques as well as suggesting possible
research directions.

## Introduction

As the scope for robotic applications extends from structured to unstructured
and more complex environments, autonomy has become a desideratum for most of
today’s robots. The practice of handcrafting robots does not give them the
capability to cope with unforeseen situations. Although several research
contributions have been made towards robot autonomy, we are nowhere near the
level of autonomy that is exhibited by animals, even ones at the lowest
biological level of organisation. This is because animals are born with innate
capabilities, both in their body structure and intelligence, to survive and
develop in their milieus; their behaviours and sometimes their morphological
traits can evolve to adapt to persistent changes in their habitats. For
example, Corcoran et al [1] studied the co-evolutionary battle between the bat
and the moth. Bats rely on echolocation for finding their prey. They produce
high frequency sounds and examine the echoes for estimating the positions of
their prey. Since moths have been on the bat’s favourite menu for tens of
millions of years, they have evolved simple ears to detect possible attacks,
produce their own ultrasounds to jam the sound from the predator and initiate
amazing deflective strategies for manoeuvres. This ability of the moth to
develop skills that were not originally there in order to survive in its
environment is a fundamental product of autonomy. We posit that this idea of
autonomy should remain the same in robotics. An autonomous robot, without any
need for external change of the underlying algorithms, should learn to develop
novel skills in order to cope with unpredictable situations.

In our previous review, we defined what it means for an agent to be understood
as autonomous in terms of specified attributes [2]. The attributes are
classified into two categories: low-level attributes and high-level
attributes. The low-level attributes constitute the separating line between
autonomous and automatic or automated agents. They include perception,
actuation, learning, context-awareness and decision-making. We argue that any
system, whether biological or artificial, has to possess at least these low-
level attributes to be called autonomous. The high-level attributes of
autonomy are the advanced attributes of autonomy. They include domain-
independence, self-motivation, self-recovery and self-identification of goals.
While the low-level attributes have been extensively addressed in robotics and
artificial intelligence, the high-level attributes remain a challenge. In the
following section, we discuss why some popular AI techniques such as deep
learning and reinforcement learning are not suitable methods for achieving
high-level autonomy. Furthermore, some techniques that agree with our
definition of autonomous behaviour in robots are highlighted.

## Review of State-of-the-art Techniques

Inspired by animal behaviours or nature in general, AI is developing at a
mind-boggling speed. Some of the more sophisticated AI techniques have shown
promising results in different areas of applications. For example, in 2015
DeepMind Technologies developed AlphaGo, a deep reinforcement learning
algorithm that became the first to defeat a master in the game of Go; deep
learning has also proven to be the go-to method in different classes of AI
problems such as computer vision, natural language processing (NLP), self-
driving cars and so on. Despite the level of progress attained by these
methods in various application areas we believe that these methods on their
own are not suitable for high degree autonomy.

According to Goodfellow [3] “some artificial neural networks have nearly as
many connections per neuron as a cat, and it is quite common for other neural
networks to have as many connections per neuron as smaller mammals like mice”.
Revisiting the moth example: although moths have fewer connections per neuron
than some artificial neural networks, they posses a higher degree of autonomy
than the latter. A more radical example is the behaviour of the trichoplax, a
small disk-shaped marine metazoan which exhibits a sequence of coordinated
feeding and response behaviour despite having no nervous system [4]. This
arouses curiosity with regard to the nature of autonomy in animals. While many
improvements have been made in AI in recent times by creating neural networks
with massive interconnections of neurons in mimicry of the human brain, it can
be argued that these massive interconnections alone are not enough to realise
common sense and autonomous behaviours in robots. Although different classes
of animals have different neural complexities, they exhibit a high degree of
autonomy in their milieus. So, instead of focusing only on the anatomical
similarities between the AI and biological agents, it is also good to consider
the basic theories and attributes that underpin autonomous behaviours in
animals and seek to formalise those in artificial systems.

The emergence of deep learning has revolutionalised how AI is applied in
virtually all application domains especially in perceptual tasks such as
visual recognition, speech recognition and so on. However, the concept of deep
learning does not agree with our ideas of autonomous behaviour. Most
successful deep learning algorithms like convolutional neural networks (CNN)
and recurrent neural networks (RNN) are supervised learning algorithms, and
therefore they require a teacher to provide them with accurate labels of
sensory inputs. This makes them impracticable in unknown or totally new
environments. An autonomous agent should be driven by self-motivation to learn
and improve in unknown environments without a teacher. Moreover, deep learning
algorithms are data and resource hungry. For example, with 1.2 million
labelled training examples, trained on two GTX 580 3GB GPUs, AlexNet, a deep
convolutional neural network that won the 2010 ImageNet Large-Scale Visual
Recognition Challenge (ILSVRC), took between five and six days to learn how to
classify images into 1000 different categories [5]. Humans, and animals like
rats and primates, do not require so many examples to learn [6] and as a
result they can quickly adapt to changes in their environments without having
to wait for too many experiences.

One of the most applied AI techniques in autonomous decision-making is
reinforcement learning (RL). Amongst all machine learning paradigms, RL has
the most obvious similarity with goal-directed behaviour in animals. The
modern RL can trace its roots back to the psychology of animal learning,
specifically Thorndike’s classic theory of the law of effect which suggests
that animals tend to increase behaviours that are followed by pleasant
consequences and decrease behaviours that are followed by negative
consequences [7]. RL has been successful in a number of problems from computer
games to self-flying helicopters. However, a major challenge of applying RL in
autonomous robotic scenarios is that it requires handcrafting of the reward
function, which is often difficult to design for some application scenarios.
As such, RL depends on demonstration data from human experts for most real-
life applications; this data is non-existent in most cases. Moreover, reward
functions are often environment-dependent, causing the algorithms to find it
difficult to cope when environments change in ways that are not captured in
the reward functions. RL exists in two categories i.e. model-free and model-
based RL. This classification is dependent on whether the transition model of
the environment is required or not. Unlike model-based RL, model-free RL does
not require the model of the environment to learn. Although model-based RL has
proven to be sample efficient and has high generalisation, it is not often
applied in most practical problems because environment models are difficult to
design for unpredictable or unknown environments. Also, model-based RL is
strongly coupled with environment models and they perform worse than model-
free methods when these environment models no longer capture the current
situation, unless the designer manually updates them. As a result of this,
model-free RL is the most applied RL method in real-world contexts. However,
these methods are sample inefficient and have a very poor generalisation
performance; they take a long time to adapt to changing goals and/or
environments.

## The Future of Autonomous Learning

A number of research areas such as developmental robotics [8], enactive
learning [9] and unsupervised learning [10] seem to offer good alternatives
for autonomous robots. Developmental robotics is inspired by the cognitive
development in human infants. This is in line with Alan Turing’s idea that it
is easier to build an artificial baby and train it to maturity than trying to
build and simulate an adult mind [11]. Enactive learning seeks to develop a
self-motivated agent that has the capability to self-program itself while
engaging in active perception. In our ongoing research, we seek to represent
causation in unsupervised self-organising neural networks. This will help an
agent to learn the sensorimotor map of its environment for planning instead of
depending on the designer to formulate it as in model-based RL. While we
believe that these methods have the potential to drive autonomous robotic
research, we also agree that no single method has the capabilities to realise
a truly autonomous behaviour. However, a combination of two or more methods
will yield better results to the challenges of designing autonomous robots.

## Conclusions

This paper has reviewed robot autonomy in the context of animal behaviour. It
argues that the popular state-of-the-art AI methods, on their own, are not
capable of achieving high degree autonomy. Some AI methods that seem to agree
with our definition of autonomy are briefly explained. This paper does not
claim that any single method has the capability to achieve high degree
autonomy but understanding the strengths and weaknesses of different methods
can help in synergising them for autonomous robot applications.

## References

  * [1] Aaron J Corcoran, Jesse R Barber, Nickolay I Hristov, and William E Conner. How do tiger moths jam bat sonar? Journal of Experimental Biology, 214(14):24162425, 2011.
  * [2] Chinedu Pascal Ezenkwu and Andrew Starkey. Machine autonomy: Definition, approaches, challenges and research gaps. In Intelligent Computing-Proceedings of the Computing Conference, pages 335–358. Springer, 2019.
  * [3] Ian Goodfellow, Yoshua Bengio, and Aaron Courville. Deep learning. MIT press, 2016\.
  * [4] Carolyn L Smith, Natalia Pivovarova, and Thomas S Reese. Coordinated feeding behavior in trichoplax, an animal without synapses. PloS one, 10(9), 2015.
  * [5] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. In Advances in neural information processing systems, pages 1097–1105, 2012.
  * [6] JP Aggleton. One-trial object recognition by rats. The Quarterly Journal of Experimental Psychology, 37(4):279–294, 1985.
  * [7] EL Thorndike. Animal intelligence; experimental studies. on cover: the animal behavior series, 1911.
  * [8] Angelo Cangelosi and Matthew Schlesinger. Developmental robotics: From babies to robots. MIT press, 2015.
  * [9] Olivier L Georgeon, Christian Wolf, and Simon Gay. An enactive approach to autonomous agent and robot learning. In 2013 IEEE Third Joint International Conference on Development and Learning and Epigenetic Robotics (ICDL), pages 1–6. IEEE, 2013.
  * [10] Chinedu Pascal Ezenkwu and Andrew Starkey. Unsupervised temporospatial neural architecture for sensorimotor map learning. IEEE Transactions on Cognitive and Developmental Systems, 2019.
  * [11] Alan M Turing. Computing machinery and intelligence. In Parsing the Turing Test, pages 23–65. Springer, 2009.