# Hard Label Black Box Node Injection Attack on Graph Neural Network

Yu Zhou, Zihao Dong, Guofeng Zhang, Jingchen Tang
University of California, Los Angeles

(June 15th 2022)

###### Abstract

While graph neural networks have achieved state-of-the-art performances in
many real-world tasks including graph classification and node classification,
recent works have demonstrated they are also extremely vulnerable to
adversarial attacks. Most previous works have focused on attacking node
classification networks under impractical white-box scenarios. In this work,
we will propose a non-targeted Hard Label Black Box Node Injection Attack on
Graph Neural Networks, which to the best of our knowledge, is the first of its
kind. Under this setting, more real world tasks can be studied because our
attack assumes no prior knowledge about (1): the model architecture of the GNN
we are attacking; (2): the model’s gradients; (3): the output logits of the
target GNN model. Our attack is based on an existing edge perturbation attack,
from which we restrict the optimization process to formulate a node injection
attack. In the work, we will evaluate the performance of the attack using
three datasets, COIL-DEL ([1]), IMDB-BINARY ([3]), and NCI1 ([2]). Our code
and demo are publicly available for research purposes at:
https://github.com/bryanzhou008/Hard_Label_Black_Box_Attack_GNN.git 111This
project was completed during the CS249: Graph Neural Networks class at UCLA
advised by Prof. Yizhou Sun.

##  1 Introduction

Nowadays, Graph Neural Networks (GNN) have been applied to various tasks, and
demonstrated state-of-the-art performances. Among of the most popular tasks
for Graph Neural Networks are node classification ([4]), link prediction
([7]), and graph classification ([9, 6, 7]). Specifically, our work focus on
the graph classification tasks, in which a graph is given an associated label,
for example, a group of actors might be labeled as ”actors in Comedies,” and
the Graph Neural Network is trained to infer this label from the given graph.
This kind of graph classification is especially useful in fields like social
network applications, biomedics, chemistry, and object classification.

Previous studies also demonstrated that Graph Neural Networks are vulnerable
to adversarial attacks ([8, 10, 5]). However, the existing attack methods, as
we summarized, have two main drawbacks. First, most attacking methods focus on
node level prediction or edge level prediction, whereas only a few of them
mentioned attack on graph level, which is actually a very important
application of Graph Neural Networks. In a graph level attack, the adversarial
agent will try to add/remove edge/node from the original graph to create
adversarial instances to shift the model’s prediction on the graph. Such
attack can be very dangerous in real world scenarios. For example, when
predicting the chemical properties of a compound in a medicine, then a wrong
prediction can be deadly. And for user classification tasks in social network,
malicious users may be able to change the label of the user group it is in and
cause the model to give wrong recommendations to the users. Furthermore,
existing adversarial attacks on Graph Neural Networks are usually non-
practical in real-world scenarios. For example, there exist a portion of
white-box attacks that assume adversarial agent has access to all necessary
information about the target model, including its structure and gradients, and
other attacks are soft-label black box attacks that assume agent has the
output logits, all of these information is usually hidden from the public in
real world. There are also some off-the-shelf hard-label black box attacks on
Graph Neural Network, for example Jiaming et.al presented a hard label black
box attack that perturb the edges in the graph to influence the graph level
prediction ([12]). However, in applications like social network prediction, it
is impossible for adversarial agents to modify connections between existing
users. Therefore, in most cases, we believe node injection attack is more
practical in the real-world and thus worth study more comparing to edge
perturbing attacks.

Therefore, in our paper, we propose a node injection attack on graph
classification to better understand the robustness of Graph Neural Network in
graph level tasks, and we assume a hard-label black box setting, meaning we
only have access to the prediction label of the model, resembling most real
world scenarios. And we evaluate the performance of our method on three
datasets: COIL-DEL (100-class object classification) ([1]), IMDB-BINARY(binary
classification on social network) ([3]), and NCI1 (Binary classification of
chemical property) ([2]). For different datasets, we use different node
feature initialization methods to generate nodes that can fit into the graph
better (less noticable), and also provides a comparison between the
performance of using different initialization methods. We further
systematically evaluate the possible limitations and future works that can be
built upon our work. To summarize, our main contribution in the work includes:

  * •

We present the first hard-label black box node injection attack on Graph
Neural Networks, which we built upon an existing edge perturbing attack.

  * •

We evaluate the performance of the method on three real world datasets, COIL-
DEL ([1]), IMDB-BINARY ([3]), and NCI1 ([2]), using initialization methods
specific to each dataset, and give a comparison between different
initialization methods

  * •

We identify possible drawbacks or limitations of our current work and point
out future research directions

##  2 Related Work and Background

###  2.1 Adversarial Attack

Depending on the availability of knowledge about the GNN model structure and
output logits, we could categorize the attack into: White Box attack, where
the adversarial agent has access to all information of the model, including
gradient, model structure, and all outputs; Soft-Label Black Box Attack, where
the adversarial agent only has access to the logits output of the model, and
has no knowledge about the model’s gradient and structure; and the Hard-Label
Black Box Attack, which is the attack we are deploying, only has access to the
label output of the model.

###  2.2 Adversarial Attack on GNN

Multiple literature in the past have shown the vulnerability of Graph Neural
Network against Adversarial Attack ([8]). In general, there are two types of
tasks that are frequently being conducted through GNN nowadays, Graph-level
Classification and Node-level Classification, where we mainly focus on the
former one in this paper. As for the forms of Adversarial Attack, which could
be categorized into targeted attack and non-targeted attack. In our setting,
we focused on the non-targeted attack, which means we tries to mislead to
model to generate any prediction that deviates from the ground truth one
instead of a specific prediction. Moreover, The attack can happen either
during model training (poisoning) or during model testing (evasion) in our
case ([11]). For the specific method of adversarial attack, there are edge
attack and node injection attack. To be specific, An attacker can perform an
adversarial attack by perturbing one of three components in a graph: (i)
perturbing nodes, i.e., adding new nodes or deleting existing nodes; (ii)
perturbing node feature matrix, i.e., modifying nodes’ feature vectors; and
(iii) perturbing edges, i.e., adding new edges, deleting existing edges or
rewiring edges ([12]).

##  3 Problem Formulation

Figure 1: A Set of Example Graphs to Illustrate Our Node Injection Attack
Method

Given a target GNN model $f$ and a target victim graph $G$ with ground truth
class prediction label $y_{0}$ for that particular input graph $G$, the
attacker attempts to generate a non-targeted adversarial graph $G^{\prime}$ by
node injection with initialization on injected nodes’ feature. Moreover, the
attacker will initialize the connection of the $k$ injected nodes with $G$ to
be the adjacency matrix $A$, and perturb $A$ to be $A^{\prime}$ such that the
predicted label of $G^{\prime}$ will differ from $y_{0}$. As mentioned above
in our setting, we only allows attacker to perturb the edges related to the
$k$ injected nodes, but not the existing ones of the original graph, let the
adversarial perturbation be a binary matrix $\Theta\in\\{0,1\\}^{k\times N}$.
Specifically, $\Theta_{ij}=1$ means the attacker changes the edge status
between nodes i and j and $\Theta_{ij}=0$ means we remain the edge status
between nodes i and j unchanged. Hence we can define the perturbation function
$h$ that could generate the perturbed adjacency matrix $A^{\prime}$ by
$A^{\prime}=h(A,\Theta)$ as:

$\displaystyle\begin{split}h(A,\Theta)_{ij}=\begin{cases}A_{ij}&\Theta_{ij}=0\\\
\neg A_{ij}&\Theta_{ij}=1\end{cases}\end{split}$ (1)

Furthermore, we established a budget $b$ for to restrict the perturbation rate
$r$ and hence we could construct the adversarial attack as the following
optimization problem:

$\displaystyle\begin{split}\Theta^{*}=\underset{\Theta}{\arg\min}&\|A^{\prime}-A\|_{0},\\\
\text{subject to }&A^{\prime}=h(A,\Theta),\\\ &f(A^{\prime})\neq y_{0},\\\
&r\leq b,\end{split}$ (2)

where r is defined as $r=\|A^{\prime}-A\|_{0}/kN$ and $\|A^{\prime}-A\|_{0}$
is the $L_{0}$ norm that counts the number of nonzero entries, which is the
number of perturbation in our case.

However, we could show that the above optimization problem is intractable to
solve due to $l_{0}$ norm and $\Theta$, and we need to reformulate the
optimization problem by relaxing $\Theta$ to be continuous variables ranging
from 0 to 1 that could be treated as the probability that corresponding edge
between two nodes is changed, instead of binary entries 0,1, such that we
could approximate the gradients of the objective function. Therefore, the
perturbation function $h$ could be reformulated as:

$\displaystyle\begin{split}h(A,\Theta)_{ij}=\begin{cases}A_{ij}&\Theta_{ij}<0.5\\\
\neg A_{ij}&\Theta_{ij}\geq 0.5\end{cases}\end{split}$ (3)

We can then define a new objective function that replaces the $L_{0}$ norm
with the $L_{1}$ norm. Define distance function $g(\Theta)$ that measures the
distance from the target graph to the classification boundary as:

$\displaystyle
g(\Theta)=\underset{\lambda>0}{\arg\min}\\{f(h(A,\lambda\Theta_{norm})\neq
y_{0}\\},$ (4)

where $\Theta_{norm}$ is the normalized perturbation matrix such that
$\|\Theta_{norm}\|_{2}=1$. Therefore, $g(\Theta)$ gives the minimum distance
$\lambda$ starting from $A$ towards another class in direction $\Theta$ that
changes the prediction label. In addition, we denote
$\hat{g}(\Theta)=g(\Theta)\Theta_{norm}$ to be the distance vector.

Although it may seem reasonable to get optimal $\Theta^{*}$ by simply minimize
$g(\Theta)$, it is not effective since it does not take into account of the
search direction. Two perturbations with equal distance but different $\Theta$
might result in a different number of perturbations on edges ([12]). Hence, we
define the following objective function:
$p(\Theta)=\|clip(\hat{g}(\Theta)-0.5)\|_{0}$, where $clip(x)$ clips $x$ into
$[0,1]$, and this objective function counts the number of elements of
$\hat{g}(\Theta)$ that exceed 0.5. And to make the calculation of gradients
viable, we replace the $L_{0}$ norm with the $L_{1}$ norm as:

$\displaystyle p(\Theta)=\|clip(\hat{g}(\Theta)-0.5)\|_{1}.$ (5)

Finally, we could find the optimal matrix $\Theta^{*}$ by minimizing
$p(\Theta)$ and convert the original optimization problem in Eq.(2) into:

$\displaystyle\begin{split}\Theta^{*}&=\underset{\Theta}{\arg\min}\text{
}p(\Theta),\\\ &\text{subject to }r\leq b.\end{split}$ (6)

##  4 Experiments

For experiments, our target model will be a Graph Isomorphism Network (GIN)
model trained on COIL-DEL ([1]), IMDB-BINARY ([3]), or NCI1 ([2]). A summary
of these three datasets is listed in table [1]. For these three datasets, in
order to make the injected node less noticeable to model or defender, we will
use three different node feature initialization methods that are designed
specifically to each dataset. Moreover, we will compare the effect of using
different node connection initialization methods, namely randomly connect
injected node to an existing node (random initialization), or connect the
injected node to the node with highest degree (mode initialization).
Theoretically we expect a higher success rate when using mode initialization.
Note that in IMDB-BINARY dataset, we use an iterative approach because their
node feature initialization methods are deterministic (non-random). We will
first try to inject 1 node, and if the attack fails, we try to inject 2, so
on. Until we exhaust a preset budget on the number of nodes we can inject or
we succeeded in an attack.

Dataset | COIL | IMDB-B | NCI1
---|---|---|---
Num. of Graphs | 390 | 100 | 411
Num. of Classes | 100 | 2 | 2
Avg. Num. of Nodes | 19.77 | 21.54 | 29.87
Avg. Num. of Edges | 96.53 | 54.24 | 32.30
Table 1: Summary of datasets

The metrics in the tables we present the experiment results are as follow:

  * •

method: the method for connection initialization

  * •

budget: the maximum percentage of nodes allowed to inject

  * •

SR: success rate (percentage) of the attack, = (success + Pred Change) / (Num.
Graphs - No need)

  * •

success: number of instances that have prediction changed after the direction
search and binary search

  * •

Pred Change: number of instances that have prediction changed right after
initialization, in experiments we consider this as successful attack as well,
because theoretically we will be able to change the label after performing the
algorithm if the label is already changed after initialization

  * •

Injected: the percentage of instances that have adversarial nodes still
connected to the graph after perturbing the graph

  * •

No need: the number of instances that the GIN model originally predicted
wrong, thus no meaning to attack

  * •

Perturb Edge: average number of edges flipped in the attack of the whole
dataset

###  4.1 COIL-DEL

In the COIL-DEL dataset, the node features are $(x,y)$, $x,y\in\mathbb{N}$
pairs representing the coordinate of the node in the space. Therefore, when
initializing the feature of the node to be injected, denote the nodes in the
original graph as $(x_{i},y_{i})$ for $i\in[0,n-1]$, where n stands for the
number of nodes in the graph, we generate random pair $(x,y)$ with
$x\sim\mathcal{N}(mean(x_{0},...x_{n}),std(x_{0},...x_{n}))$,
$y\sim\mathcal{N}(mean(y_{0},...y_{n}),std(y_{0},...y_{n}))$, and round
$(x,y)$ to the nearest integer. The experiment results on COIL-DEL is shown in
table [2]. From the table, we can observe that when using mode connection
initialization, we indeed get higher success rate at both 10% and 15% node
injection budget as we expected. Moreover, if we look at the Pred Change
column, we can see that when using random connection initialization, Pred
Change decreased a lot, which makes sense because we are expecting less
noticeable changes after initialization when we use random connection.
Moreover, the experiments demonstrated that just initializing the connection
is sufficient for keeping the new nodes connected to the graph through out the
attack, without any further restriction during the attacking process.

method | budget | SR | success | Pred Change | Injected | No need | Perturb Edge
---|---|---|---|---|---|---|---
mode | 0.1 | 50.35 | 94 | 51 | 99.16 | 102 | 13.51
mode | 0.15 | 63.89 | 112 | 72 | 100 | 102 | 11.57
random | 0.1 | 48.68 | 100 | 40 | 100 | 102 | 15.51
random | 0.15 | 61.46 | 122 | 55 | 100 | 102 | 17.38
Table 2: Experiment Results for COIL-DEL dataset

###  4.2 IMDB-BINARY

For the IMDB-BINARY dataset, we observed that the node feature are just all
one’s in the original graph, so node feature initialization is very trivial in
this case. In table [3] we show the experiment results on this dataset. For
Success Rate and Pred Change, we observe similar trends as in COIL-DEL
dataset.

method | budget | SR | success | Pred Change | Injected | No need | Perturb Edge
---|---|---|---|---|---|---|---
mode | 0.1 | 27.63 | 7 | 14 | 100 | 24 | 167.15
mode | 0.15 | 53.95 | 20 | 21 | 100 | 24 | 104.4
random | 0.1 | 27.63 | 6 | 15 | 100 | 24 | 191.67
random | 0.15 | 51.32 | 20 | 19 | 100 | 24 | 111.58
Table 3: Experiment Results for IMDB-BINARY dataset

###  4.3 NCI1

For the NCI1 dataset, the node features are 37-dimensional one-hot encodings
for the atom type that the node represents. We randomly generate injected node
feature. For atom type x, we generate it with probability
$\frac{count(x)}{Num.Nodes}$. For SR and Pred Change, we observe similar
trends as the former 2 experiments.

method | budget | SR | success | Pred Change | Injected | No need | Perturb Edge
---|---|---|---|---|---|---|---
mode | 0.1 | 47.43 | 101 | 45 | 99.17 | 104 | 128.17
mode | 0.15 | 63.19 | 140 | 21 | 98.82 | 104 | 165.06
random | 0.1 | 41.04 | 101 | 25 | 100 | 104 | 143.63
random | 0.15 | 54.40 | 140 | 27 | 100 | 104 | 185.53
Table 4: Experiment Results for IMDB-BINARY dataset Figure 2: An Example Graph
From the NCI1 Dataset Successfully Attacked by Our Model

###  4.4 Further Observersation for IMDB-BINARY and NCI1 Dataset

In our experiments for IMDB-BINARY and NCI1 datasets, we can see from the
results that the Perturb Edge in these 2 sets of experiments are significantly
larger than that of the COIL-DEL dataset (10-20 VS 100+). We believe this is
because COIL-DEL dataset has 100 class labels, whereas IMDB-BINARY and NCI1
are binary classification datasets. As a result, if we only search for perturb
direction restricted on the injected nodes, we may need to go a larger
distance along the direction we find to arrive at the decision boundary
comparing to in the COIL-DEL dataset, leading to a much larger Perturbation
number. If we compare the Perturb Edge stats to the average number of edges in
IMDB-BINARY and NCI1 datasets, we will notice that these two sets of numbers
are somewhat similar. Therefore, we believe our method still needs
improvements on such datasets where the decision boundary can be very far away
from the original instance. For example, future research may seek to define a
better direction search algorithm to find a direction along which we walk to
decision boundary in a smaller distance. Alternatively, one may consider a
better connection initialization method to allow to adversarial nodes to have
larger impact on the graph classification.

##  5 Extended investigation on feature of injected nodes

###  5.1 Methodology

In all above investigations, we assume injected node features either to be
random or to be the same as the mean of all nodes in current graph. However,
such assumption does not take fully advantage of conducting attacks on graph
using node injections. We ignore the features of the injected nodes. Based on
intuition, if we can manipulate the features of the injected nodes to make it
has the biggest impact on the graph classification task, we can definitely
make attacks more successful and more efficient.

Since it is hard to conduct optimization process in a hard-label black box
attack setting, we only focus on manipulating the injected features at the
initialization phase. In a graph classification task, due to the fact that
graph neural network will aggregate information from its neighbors, we believe
the node with highest rank in the graph should have the greatest influence on
the final prediction of the graph, and we can such node as pivot node. So we
try to interfere this aggregation process by injecting nodes that have very
different features from the features of the pivot node and connect all
injecting nodes to the pivot node at initialization. In addition, to make sure
the features of the injected nodes make sense in all datasets and also make
the injection less noticeable. We choose features from the original graph and
slightly perturb the feature to make sure it is not same to any node in the
graph or to any other injected nodes. Since the selected features come from
original graph, the feature itself always makes sense in all datasets. Also,
since the features are not going to same as any other features in the graph,
we make the injection process less noticeable.

###  5.2 Experiments and analysis

For experiments on our manipulation of the injected nodes features, we decide
to only run the experiments on COIL-DEL dataset with above algorithm. We run
the both iterative version where we gradually increase the number of nodes
injected until certain budget and the normal version where we directly
injected fixed percentage of the nodes to the original graph. The following is
our final result.

method | budget | SR | Perturb Edge | Attack Time | Query Count
---|---|---|---|---|---
Normal | 0.1 | 51.61 | 10.84 | 9.06 | 544.2
Iterative | 0.1 | 51.95 | 8.25 | 11.1 | 580.8
Normal | 0.15 | 64.74 | 10.65 | 9.56 | 644.0
Iterative | 0.15 | 65.17 | 8.65 | 13.2 | 786.6
Normal | 0.2 | 71.18 | 11.88 | 9.74 | 632.0
Iterative | 0.2 | 71.53 | 9.39 | 15.7 | 822.8
Table 5: Feature manipulation experiment result on COIL-DEL dataset

As you can see, with feature manipulation using mode connection, we can
achieve a slightly higher success rate with a lower number of edge
perturbations. We prove this is indeed a smarter initialization that makes the
attack more successful and less noticeable with less changes to the original
graph.

##  6 Conclusion and Future Works

We believe there are three main contributions in our work. Firstly, we extend
on the original work to significantly improves its success rate with single-
node injection and perturbations to edges in the whole graph. Secondly, we
propose method to do node injections with random or mean feature
initialization work in a strict black box setting. To our knowledge, we are
the first who investigate in this area. Thirdly, we propose smarter ways to
initialize the feature of the injected nodes and we prove it does lead to a
higher success rate with smaller changes to the original graph.

There are definitely still a lot of room for further investigation based on
our current work. Most importantly, since currently our injected features are
all fixed after initialization, how to develop a concrete method to update and
find the optimal injected node feature is a very interesting topic. Some other
future works include finding a better initial search specific to node
injections as the original edge perturbation paper, improve our performance on
datasets that have small number of classes, and find better connection
initialization for injected nodes that is less noticeable. We hope our work
can inspire further researches in related fields.

## References

  * [1] Kaspar Riesen and Horst Bunke  “IAM graph database repository for graph based pattern recognition and machine learning”  In _Joint IAPR International Workshops on Statistical Techniques in Pattern Recognition (SPR) and Structural and Syntactic Pattern Recognition (SSPR)_ , 2008, pp. 287–297  Springer
  * [2] Nikil Wale, Ian A Watson and George Karypis  “Comparison of descriptor spaces for chemical compound retrieval and classification”  In _Knowledge and Information Systems_ 14.3 Springer, 2008, pp. 347–375
  * [3] Pinar Yanardag and SVN Vishwanathan  “Deep graph kernels”  In _Proceedings of the 21th ACM SIGKDD international conference on knowledge discovery and data mining_ , 2015, pp. 1365–1374
  * [4] Thomas N Kipf and Max Welling  “Semi-supervised classification with graph convolutional networks”  In _arXiv preprint arXiv:1609.02907_ , 2016
  * [5] Hanjun Dai et al.  “Adversarial attack on graph structured data”  In _International conference on machine learning_ , 2018, pp. 1115–1124  PMLR
  * [6] Zhitao Ying et al.  “Hierarchical graph representation learning with differentiable pooling”  In _Advances in neural information processing systems_ 31, 2018
  * [7] Muhan Zhang and Yixin Chen  “Link prediction based on graph neural networks”  In _Advances in neural information processing systems_ 31, 2018
  * [8] Daniel Zügner, Amir Akbarnejad and Stephan Günnemann  “Adversarial attacks on neural networks for graph data”  In _Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining_, 2018, pp. 2847–2856
  * [9] Fan Zhou et al.  “Meta-gnn: On few-shot node classification in graph meta-learning”  In _Proceedings of the 28th ACM International Conference on Information and Knowledge Management_ , 2019, pp. 2357–2360
  * [10] Wanyu Lin, Shengxiang Ji and Baochun Li  “Adversarial attacks on link prediction algorithms based on graph neural networks”  In _Proceedings of the 15th ACM Asia Conference on Computer and Communications Security_ , 2020, pp. 370–380
  * [11] Jiaqi Ma, Shuangrui Ding and Qiaozhu Mei  “Towards more practical adversarial attacks on graph neural networks”  In _Advances in neural information processing systems_ 33, 2020, pp. 4756–4766
  * [12] Jiaming Mu et al.  “A Hard Label Black-box Adversarial Attack Against Graph Neural Networks”  In _Proceedings of the 2021 ACM SIGSAC Conference on Computer and Communications Security_ , 2021, pp. 108–125

##  7 APPENDIX

###  7.1 Interactive Demo

We have created an interactive Google Colab Demo of our Node Injection Attack.
When using the demo, you should first change google drive paths to your own
and run all cells collapsed under the ”Environment” group to install all
dependencies. Then, you may continue to the ”Settings” block to set all
parameters you wish to try. Here dataset name should be one of ’NCI1’, ’COIL-
DEL’, and ’IMDB-BINARY’. Injection number specifies how many nodes you want to
inject to each graph. Injection percentage specifies what percentage of total
number of nodes in the graph you wish to inject. Only one field of the above
two should be non-zero. Initialization specifies how the node features will be
initialized. The supported methods include zero, one, random, and
node_mean(the mean value of all node features in the graph). Connection
specifies how and whether you want the injected nodes to be connected to the
graph. The available options include no_connection, random and mode(connect
all injected nodes to the node in the original graph with most connections).

Once you have specified all the parameters in ”Settings”, please run the
blocks collapsed in ”Utility Functions” to read in data and. Now, to visualize
the injection results, you should just run the first cell in ”Demo” and the
injection results should show up on your screen.

Figure 3: An example graph produced from the following settings: {dataset:
NCI1, injection number: 0, injection percentage: 0.15, initialization method:
node mean, connection method: random}


11institutetext: Centre for Intelligent Machines & ECE Department
McGill University, Montreal, QC, Canada

# Visual Attention in Imaginative Agents

Samrudhdhi B. Rangrej  James J. Clark

###### Abstract

We present a recurrent agent who perceives surroundings through a series of
discrete fixations. At each timestep, the agent imagines a variety of
plausible scenes consistent with the fixation history. The next fixation is
planned using uncertainty in the content of the imagined scenes. As time
progresses, the agent becomes more certain about the content of the
surrounding, and the variety in the imagined scenes reduces. The agent is
built using a variational autoencoder and normalizing flows, and trained in an
unsupervised manner on a proxy task of scene-reconstruction. The latent
representations of the imagined scenes are found to be useful for performing
pixel-level and scene-level tasks by higher-order modules. The agent is tested
on various 2D and 3D datasets.

##  1 Introduction

Attention is central to visual perception. When an agent is performing a task,
it is imprudent to dedicate resources for the detailed sensing of an
environment. Actively attending to the limited but most informative regions of
an environment is crucial for efficiency. An intelligent agent is one which
can make use of a few informative views to determine an entire surrounding.

Humans analyze scenes by paying attention to various areas of the visual field
one after another. The eyes successively fixate on discrete locations of a
scene and move very quickly between consecutive fixations. The details
captured from various fixations are progressively fused to build a mental
representation of a scene [58], which is subsequently used for higher-order
tasks.

To act confidently, an agent has to know its surroundings accurately. Complete
certainty may be achieved if an agent scans an entire scene. However, time-
critical tasks demand to learn about the surroundings without waiting for the
scanning to finish. But due to the regularities found in the world, it seems
that it should be possible to predict the surroundings from a limited set of
views.

Humans can ‘guess’ the surroundings from obstructed views [66]. Based on a
narrow view from a window showing a few trees, we can imagine a forest or a
garden outside. Similarly, from a limited interior view containing
bookshelves, we can imagine tables and chairs in the surroundings and can
predict that it is a scene of a study room or a library. Gathering information
from a visual scene is a recurrent process. Information gathered so far helps
in hypothesizing the content from as yet unseen regions. One can then shift
attention to a region that is deemed important based on these hypotheses. As
the sensing progresses, the hypotheses are refined. One can predict the entire
scene without attending to unimportant regions.

Hypothesizing plausible surroundings has several benefits. It helps in
reducing errors while making decisions with partial observations. A partially
observed scene may not contain the critical features required for making a
decision. Furthermore, it may contain distracting information resulting in the
wrong decision. Consider the task of ‘cat’ vs. ‘dog’ classification and
consider a test image of a dog containing a toy shaped like a cat. If a
glimpse is on the toy, the agent may wrongly classify the image as ‘cat’.
However, if the agent can hypothesize a dog in the surrounding, it may choose
to gather more evidence before making a decision.

Hypothesizing plausible surroundings also facilitates planning. An agent can
focus on likely scenarios while eliminating unlikely ones. While driving, if
the agent sees a house and hypothesizes that it is entering a residential
territory as opposed to a freeway, it can prepare to brake often. Similarly,
if a robotic agent can hypothesize a 3D structure of an object from a few
strategic 2D views, it can plan grasping and object manipulation. Furthermore,
these types of agents are more trustworthy compared to black-box agents as
they reveal the imagined hypotheses on which the decisions are based. Also,
hypothesizing complete surroundings from the past set of glimpses resists
forgetting.

We propose to build an imaginative agent which imagines the yet unseen
portions of the scene. Higher-order modules can use the imagination for making
task-specific decisions. Imagining surroundings from a limited number of
glimpses is an ill-posed problem. One can imagine several surrounding scenes
that are consistent with the seen glimpses. If an agent is optimized to make a
single prediction, it predicts a blurry scene - a numerical average of all
possibilities. Hence, we will allow our agents to predict many plausible
surroundings.

A problem of how to capture informative views remains. A naive approach is to
scan the scene in a sliding-window fashion, which is sub-optimal. An optimal
next view should be the one which maximally reduces the uncertainty in the
prediction of the surrounding. An agent should start with fixating on those
locations that help in learning the most about the unseen surrounding and
gradually move to the less informative locations. Informativeness of a view
can be decided based on its contribution to imagining the unseen surrounding.
A view is deemed informative if it reduces the uncertainty in the imagined
content. The eventual low uncertainty in the imagination obviates the
necessity of further fixations. This can improve the performance of an agent
in time-critical tasks [6].

###  1.1 Problem Description

We address two related problems of hypothesizing surrounding scenes from
limited views and deciding the next most informative view based on the
predicted surroundings. An agent captures a partial view of a scene and
hypothesizes a complete scene. Uncertainty due to the ill-posedness of the
problem is tackled by allowing the agent to make multiple simultaneous
predictions. The predictions are pseudo-realistic and interpretable. The agent
maintains a set of internal representations, each corresponding to a different
hypothesis. As new views become available, the representations are filtered to
exclude false hypotheses. The agent is trained in an unsupervised fashion.
Unsupervised training makes use of unlabelled data, which is available in
abundance and at a low cost. Once the training is finished, the imagined
scenes or their internal representations can be used for higher-order tasks.

The design of our agent is inspired by the biological phenomena of maintaining
a visual memory and guessing surroundings from glimpses based on memory. The
human brain compresses observed visual scenes into latent codes to remember
them efficiently. The compression retains only high-level information [65].
Semantically similar visual concepts are encoded with similar codes [74]. The
high-level information can be recalled by decoding the latent codes.
Additionally, the same latent codes are used for higher-order vision tasks.
The brain also possesses the ability to imagine new scenes that are similar to
past visual experiences [63]. Humans can also imagine the surroundings given
partial observation(s) of a scene [66]. The imagination is derived from visual
memory. It is easy to guess the surrounding if similar scenes are present in
the visual memory.

Our agent is based on a variational autoencoder, which recurrently generates
complete scenes conditioned on a sequence of partial observations. Scenes
generated by VAEs are associated with a distribution of latent
representations. To use the representations for a scene-level task, they
should contain the information of the entire scene independent of the history
of the partial observations. We develop a novel technique to warp and unwarp
the latent space of an unconditional VAE based on the observed views. Two
normalizing flows learn the warping and unwarping functions conditioned on the
observation history. During the prediction phase, samples from the warped
latent space are mapped to the unconditional latent space using a conditional
unwarping function. The unconditional decoder decodes the mapped samples to
generate complete scenes. Lastly, we demonstrate that the unconditional
representations can be used for higher-order scene-level tasks.

The next fixation is planned based on the uncertainty in the predicted scenes.
A next view is captured on the location where the uncertainty is highest.
Sensing in this manner reduces uncertainty in the subsequent predictions.

##  2 Literature Review

####  2.0.1 Visual Attention.

The computer vision community has used attention to solve many vision tasks.
Early traces of visual attention can be found in multi-scale edge detection
algorithms [35]. Fukushima [23] proposed neural networks that attend to and
recognize patterns in an image sequentially. Koch and Ullman [40] proposed a
winner-take-all algorithm for visual attention. Later, Itti and Koch [31]
extended the method by including image saliency. Clark and Ferrier [14] tuned
the weights of salient features based on feedback. Wixson and Ballard [73]
suggested paying attention in the direction of the gradient of the object
matching function. Swain and Ballard [70] developed a method called histogram
back-projection to identify the important regions to attend. Wilkes and
Tsotsos [72] planned fixations on the prominent edges to recognize origami
objects.

Several researchers used graphical models to describe visual attention. Rimey
and Brown [61] modeled eye-movements using an augmented Hidden Markov Model.
Butko and Movellan [9] learned eye-movement for visual search task by
minimizing the entropy of a belief state used for predicting the location of a
target. Butko and Movellan [10] extended the model for multiple targets.
Larochelle and Hinton [42] used third-order connections in Restricted
Boltzmann Machine(RBM) to learn a relationship between the context, the
location of fixations, and the internal representation of a scene.
Furthermore, an approach based on an autoregressive model was proposed by
Zheng et al. [81]. Alexe et al. [1] proposed a learning-free method based on
nearest neighbors.

Recently, the machine learning community developed ‘soft’ and ‘hard’ visual
attention. Soft attention allows the content of an entire scene to be
perceived at once, but with varying degrees of importance. A deterministic
attention map is predicted and multiplied with the input to suppress
unimportant content [77, 28]. Soft attention has been used for solving various
computer vision tasks such as image classification and segmentation [71, 12],
image captioning [77], optical character recognition [45], and pose estimation
and action recognition [13, 47]. Hard attention refers to a mechanism where an
agent perceives a scene through a series of fixations. The agent decides the
location of the next glimpse stochastically [51]. Hard attention has been
employed for solving various computer vision tasks such as image captioning
[77], object recognition and localization [3, 49, 11], and 3D scene
classification [32]. The REINFORCE optimization algorithm used for learning
hard attention has high variance and is very sensitive to the hyper-parameter
setting. As a solution, Ranzato [57] proposed optimization using an EM-like
algorithm, and Ba et al. [4] proposed a wake-sleep algorithm.

Active vision is closely related to hard attention. An active agent has more
degree of freedom to evaluate a scene from multiple viewpoints. A large body
of work is focused on actively perceiving a 3D environment for scene
recognition and localization [11], and manipulating a 3D object for
recognition [2]. Another line of work addresses active vision-based view-
planning [75], navigation and localization [17]. Fast and timely agents start
recognizing objects, though imperfectly, from the very first viewpoint and
update the decision as time progresses [34].

Similar to the paradigms of hard attention and active vision, our agent
perceives the surrounding through a series of glimpses. However, learning is
not dedicated to any supervised task. The agent learns to explore the
surroundings by solving an unsupervised proxy task of constructing complete
scenes from partial observations.

####  2.0.2 Prediction of the Surrounding.

The task of predicting surroundings is similar to the tasks of image in-
painting [79], context prediction [55], view-synthesis [21], predicting 3D
structure from 2D views [60], and predicting missing pixels [68]. These are
passive tasks where an active exploration of scenes is not performed. Our
agent explores the scene actively while learning from a proxy task of scene-
construction.

####  2.0.3 Proxy Tasks.

In computer vision, proxy tasks are often used to improve performance on the
main task. Examples are colorization for visual understanding [43], learning
depth maps by reconstructing stereo images [26], crowd counting by learning to
rank [48], cross channel prediction [80], and feature learning by predicting
image rotation [25], by solving jigsaw puzzle [36] or by learning to count
visual primitives [53], etc.

####  2.0.4 Prediction of Multiple Hypotheses.

Prediction of surroundings from a small set of glimpses is an ill-posed
problem as there exist several non-unique solutions. A model might learn to
predict an average of all possible solutions if the uncertainty is not
accounted for [22]. A widely used approach is to find a fixed number of
multiple simultaneous solutions. The method is used for several ill-posed
tasks such as future frame-prediction [62], optical flow estimation [30],
classification, segmentation and caption generation [46]. A variational
approach overcomes the constraint of predicting a limited number of
hypotheses. The approach allows unlimited sampling of hypotheses from a learnt
distribution. The method has been used to solve several ill-posed tasks such
as segmentation [41] and motion prediction [8].

Our problem is similar to the ones addressed in [33, 56, 64]. They also use
scene reconstruction as a proxy task for learning visual attention. However,
their agents predict a single scene from partial observations. Additionally,
they learn policies for scene exploration, whereas our approach uses a
predefined policy.

##  3 Background

The proposed imaginative agent is built using variational autoencoders(VAE)
and normalizing flows(NF). A brief review of these is provided next.

###  3.1 Variational Autoencoder

A variational autoencoder(VAE) is a latent-variable model that assumes that
the data is generated from a low dimensional latent representation. The data
$x$ and the latent representation $z$ are generated from the likelihood model
$p(x|z)$ and the prior $p(z)$, respectively. Learning the likelihood model is
challenging as the posterior $p(z|x)$ is intractable. Therefore, a surrogate
posterior $q(z|x)$ is inferred. The posterior $q(z|x)$ and the likelihood
$p(x|z)$ are modeled using an encoder and a decoder network, respectively.
Network parameters are optimized by maximizing the evidence lower bound (ELBO)
objective [37].

$\displaystyle L_{VAE}=\mathbb{E}_{z\sim q(z|x)}\log
p(x|z)-\mathbb{KL}(q(z|x)||p(z))$ (1)

The expectation in the above formula is approximated using samples $z$ drawn
from $q(z|x)$. Despite sampling, differentiability is preserved using the
reparameterization trick. The prior $p(z)$ is assumed to be a multivariate
Gaussian with unit variance and $q(z|x)$ is modeled as a factorized Gaussian
$\mathcal{N}(\mu(x),\Sigma(x))$.

The ELBO in Eq. 1 is tight when $q(z|x)\approx p(z|x)$. However, the
factorized Gaussian assumption limits the capacity of the posterior in making
quality inferences [16]. Alternate approaches are proposed to increase the
flexibility of the posterior such as using a mixture of Gaussians [24] and
normalizing flows [59] to model the posterior, building hierarchical latent
spaces [69] and using importance weighted sampling [7]. We use normalizing
flow, a scalable tool, to learn a flexible and arbitrarily complex posterior
$q(z|x)$ that fits the true posterior $p(z|x)$ more closely.

###  3.2 Normalizing Flow

Normalizing flow (NF) maps a primary posterior $q(z|x)\equiv q(z_{0}|x)$ to an
alternate posterior $q(z_{K}|x)$ using a series of $K$ invertible
transformations. A single invertible map $f_{1}$ transforms a primary latent
representation $z_{0}$ into an alternate latent representation $z_{1}$.
According to the change of variable formula,

$\displaystyle q(z_{1}|x)$
$\displaystyle=q(z_{0}|x)|\text{det}(J_{f_{1}})|^{-1}$ (2)

Here, $J_{f_{1}}$ is the Jacobian of $f_{1}$. The normalizing flow transforms
a factorized Gaussian posterior $q(z_{0}|x)$ to a more complex posterior
$q(z_{K}|x)$ using a composition of several invertible maps
$f=f_{K}\circ\dots\circ f_{2}\circ f_{1}$. The final posterior can be derived
by repeated application of Eq. 2.

$\displaystyle q(z_{K}|x)$
$\displaystyle=q(z_{0}|x)\prod_{\kappa=1}^{K}|\text{det}(J_{f_{\kappa}})|^{-1}$
(3)

The above posterior replaces the primary posterior in the ELBO objective
defined in Eq. 1. The law of the unconscious statistician (LOTUS) [16]
simplifies the computation of the expectation.

$\displaystyle L_{VAE}$ $\displaystyle=\mathbb{E}_{z_{0}\sim q(z_{0}|x)}\log
p(x|z_{K})-\mathbb{KL}(q(z_{K}|x)||p(z_{K}))$ (4)

Examples of neural architectures for normalizing flow are IAF [39], MAF [54],
NICE [19], Real NVP [20], Glow [38], NAF [29] and BNAF [18]. We adapt Block
Neural Autoregressive Flow (BNAF) in our implementation.

###  3.3 Conditional Variational Autoencoder

A conditional variational autoencoder(cVAE) [67] generates data $x$ from a
latent variable $z$ based on an input observation $h$. Similar to what we did
for unconditional VAEs, a NF can be used here as well to model a complex
posterior for more accurate inference. Eq. 4 can be modified to define the
conditional ELBO objective as follows.

$\displaystyle L_{cVAE}=\mathbb{E}_{z\sim q(z_{0}|x,h)}\log
p(x|z_{k},h)-\mathbb{KL}(q(z_{k}|x,h)||p(z_{k}|h))$ (5)

A conditional prior $p(z_{k}|h)$ is assumed to be a multivariate Gaussian with
unit variance. The posterior $q(z_{0}|x,h)$ is transformed to $q(z_{k}|x,h)$
using NF.

(a)

(b)

(c)

(d)

Figure 1: The proposed model. Unconditional and conditional modules are shown
in blue and red respectively. (a) Training phase. The encoder infers a latent
distribution $q(z_{0}|x)$ from an input $x$. The first BNAF warps $q(z_{0}|x)$
to $q(z_{k}|x,h_{t})\approx\mathcal{N}(0,I)$ based on features $h_{t}$
extracted from observation history $O_{t}$. The second BNAF unwarps
$q(z_{k}|x,h_{t})$ to $q(z_{K}|x)$. The decoder reconstructs $x$ from
$q(z_{K}|x)$. (b) The model viewed as an unconditional VAE. The space-warping
and unwarping BNAFs are grouped to form an auxiliary BNAF. (c) The model
viewed as a conditional VAE. The encoder and the space-warping BNAF are
grouped as an auxiliary encoder. The space-unwarping BNAF and the decoder are
grouped as an auxiliary decoder. (d) Prediction phase. An auxiliary decoder is
used to predict scenes from partial observations. The second BNAF unwarps
$\mathcal{N}(0,I)$ to $p(z_{K}|h_{t})$. The posterior $p(z_{K}|h_{t})$ is a
multi-modal distribution. Each mode gives rise to a scene with different
global structure when decoded using the decoder. The constructed scenes are
consistent with $O_{t}$.

##  4 Imaginative Agent

A recurrent agent sequentially observes glimpses covering small portions of a
scene and hypothesizes complete scenes that are coherent with the partial
observations. The model has four main building blocks as shown in Fig 1(a).

Unconditional encoder and decoder. An encoder network infers a posterior
$q(z_{0}|x)$ from a complete scene $x$. Then, a sample $z_{0}\sim q(z_{0}|x)$
is transformed to $z_{K}$ using BNAFs, as discussed next. A decoder network
reconstructs $x$ from $z_{K}$. Note that the encoder and the decoder are
unconditional. They are independent of the sensing process.

Sensor. A sensing function $s_{w}(x,l)$ extracts a square glimpse (space-
limited sample of the scene) of size $w\times w$ centered at location $l$ from
a given scene $x$. Let the observations until time $t$ be defined as
$O_{t}(x)=\\{(s_{w}(x,l_{\tau}),l_{\tau})|\tau\in{1,2,\dots,t}\\}$.

Recurrent feature extractor. A recurrent neural network(RNN) extracts fixed-
length features from observations $O_{t}(x)$. At time $\tau$, the RNN updates
its hidden state by processing and incorporating features extracted from a
tuple $(s_{w}(x,l_{\tau}),l_{\tau})$. The final hidden state after $t$
timesteps is output as the feature map $h_{t}$.

Conditional BNAFs. The original unconditional BNAF presented in [18] is made
conditional by connecting each hidden layer to an additional input $h_{t}$.
Conditional BNAFs learn to warp and unwarp a latent space between the pair of
unconditional encoder and decoder. The first BNAF warps an unconditional
posterior $q(z_{0}|x)$ to a Gaussian $\mathcal{N}(0,I)$, and the second BNAF
unwarps $\mathcal{N}(0,I)$ to an unconditional posterior $q(z_{K}|x)$.

###  4.1 Training Objective

The building blocks can be grouped to view the proposed system as either
unconditional or conditional VAEs. Grouping under the unconditional view is
shown in Fig 1(b), where a combination of the two BNAFs can be seen as a
single auxiliary BNAF. The encoder learns to make an inference from a complete
scene, and the decoder reconstructs the same scene. The auxiliary BNAF learns
a flexible posterior $q(z_{K}|x)$ while ignoring $O_{t}$ as noise. In this
setting, the system can be trained by optimizing the $L_{VAE}$ objective
defined in Eq. 4.

Grouping under the conditional view is shown in Fig 1(c). A pair of encoder
and space-warping BNAF is identified as an auxiliary encoder. Similarly, a
pair of space-unwarping BNAF and decoder is identified as an auxiliary
decoder. The auxiliary encoder and decoder are conditioned on $h_{t}$. The
auxiliary encoder performs inference on a complete scene $x$ and features
$h_{t}$. The auxiliary decoder reconstructs $x$ from a latent representation
$z_{k}$ and features $h_{t}$. The model can be optimized by using the
$L_{cVAE}$ objective defined in Eq. 5.

The agent is trained by optimizing a linear combination of the unconditional
and the conditional objectives. In practice, better performance is achieved by
modifying the conditional objective using a trick used in [67]. The idea is to
train auxiliary encoder and decoder using separate training objectives. The
auxiliary encoder is trained to minimize
$\mathbb{KL}(q(z_{k}|x,h_{t})||p(z_{k}|h_{t}))$ without the help of the
auxiliary decoder, i.e. it does not receive any gradient from $\log
p(x|z_{k},h_{t})$. Similarly, the auxiliary decoder should be trained to
predict the entire scene $x$ conditioned on $h_{t}$ without the help of the
remaining components. However, direct optimization of $\log p(x|h_{t})$ is not
possible as the posterior $p(z_{k}|x,h_{t})$ is not known to the auxiliary
decoder. Instead, we train the auxiliary decoder to reconstruct locally the
portion of the scene observed by the agent so far. This is done by optimizing
ELBO on masked $x$. The mask $m_{t}(\cdot)$ segregates pixels observed until
time $t$. The ELBO on $\log p(m_{t}(x))$ is given below.

$\displaystyle\log p(m_{t}(x))$
$\displaystyle\leq\mathbb{E}_{p(z_{k}|h_{t})}\eta\log
p(m_{t}(x)|z_{K})-\mathbb{KL}(p(z_{K}|h_{t})||p(z_{K}))$ (6)

The coefficient $\eta$ is the ratio of the number of pixels in $x$ to the
number of pixels retained by $m_{t}(\cdot)$. The final training objective at
time $t$ is as follows.

$\displaystyle L_{t}$ $\displaystyle=\mathbb{E}_{q(z_{0}|x)}\log
p(x|z_{K})-\mathbb{KL}(q(z_{K}|x)||p(z_{K}))-\mathbb{KL}(q(z_{k}|x,h_{t})||p(z_{k}|h_{t}))$
$\displaystyle+\mathbb{E}_{p(z_{k}|h_{t})}\eta\log
p(m_{t}(x)|z_{K})-\mathbb{KL}(p(z_{K}|h_{t})||p(z_{K}))$ (7)

The complete model is optimized by maximizing $\sum_{t}L_{t}$. The first two
terms in Eq. 7 help in creating visual memory by learning compact
representations of the complete scenes. The third term arranges $z_{k}$
corresponding to different scenes with a similar $O_{t}$ to follow a Gaussian
density. The Gaussian density unwarps to a multi-modal posterior
$p(z_{K}|h_{t})$ during the prediction phase (see Fig.2(b)). Each mode may
generate scenes with any realistic global structure, but the content on the
observed locations should be consistent with $O_{t}$. The last two terms
ensure local consistency and global realism, respectively.

###  4.2 Prediction Phase

Prediction of complete scenes from partial observations only requires the
auxiliary decoder. A detailed view of the auxiliary decoder is shown in
Fig.1(d). Note that the complete scene $x$ is not available to the agent
during the prediction phase. The agent extracts features $h_{t}$ from a set of
partial observations $O_{t}$. The features $h_{t}$ are used by the BNAF to
perform the conditional unwarping of a Gaussian latent space. Multiple samples
from the prior $p(z_{k}|h_{t})=\mathcal{N}(0,I)$ are unwarped to the
representations $z_{K}$s simultaneously. The decoder decodes the $z_{K}$s and
generates realistic scenes that have different global structures but are
consistent with the partial observations. As shown in Fig.1(d), when an agent
observes a portion of a scene containing an angular stroke on the upper right
corner, it generates digits ‘3’, ‘9’, and ‘7’, which have similar angular
strokes.

(a)

(b)

(c)

(d)

(e)

(f)

Figure 2: (a) A T-SNE projection [50] of unconditional latent space $q(z_{K})$
for MNIST dataset. (b-f) T-SNE projections of unwarped posteriors
$p(z_{K}|h_{t})$ for MNIST image from t=1 to 5. Partial observations are shown
in the upper left corner.

###  4.3 Uncertainty-based Sensing

We can quantify the variety in the scenes predicted at time $t$ using a pixel-
wise variance-map $V_{t}$. The conditional BNAF and the decoder are
represented as $f_{h_{t}}$ and $D$. A mean scene is defined as $\mu_{t}$.

$\displaystyle V_{t}$
$\displaystyle=\int||D(z_{K})-\mu_{t}||^{2}p(z_{K}|h_{t})dz_{K}=\mathbb{E}_{z_{k}\sim
p(z_{k}|h_{t})}||D\big{(}f_{h_{t}}(z_{k})\big{)}-\mu_{t}||^{2}$ (8)

The variety in the predicted scenes is high initially, as there are many
possible scenes with the same $O_{t}$. As time passes, $O_{t}$ covers a larger
area of the scene, and the variety in the possible distinct predictions
reduces. When all structure defining parts of the scene are observed, the
predicted scene is the same as the true scene. This is also evident from
Fig.2. The unwarped posterior at $t=1$ is multi-modal. Each mode corresponds
to a different hand-written digit. As the agent observes more regions of the
scene, it eliminates inconsistent possibilities, and the number of modes in
$p(z_{K}|h_{t})$ reduces. Eventually, $p(z_{K}|h_{t})$ becomes uni-modal,
representing a single possibility corresponding to the actual scene.

In most time-critical tasks, it is crucial to predict the actual surrounding
from a limited number of observations. This is possible if the consecutive
observations help in eliminating a maximum number of wrong hypotheses. After
the first random fixation, the consecutive fixations should sit on the
location where $V_{t}$ is highest since $V_{t}$ represents uncertainty in the
content predicted by the agent. Sensing on the locations where the uncertainty
is highest achieves two goals. First, the actual content on the most uncertain
location can be determined. Second, uncertainty in the content of the other
locations can be reduced due to structural dependence found in the natural
scenes. The observations in Fig.2 are captured using uncertainty-based
sensing. The agent achieves certainty from 3 observations covering less than
25% of the total area.

##  5 Experiments

The proposed system is validated on several 2D and 3D datasets. Refer to the
supplementary material for details about the implementation.

MNIST [44]. The dataset consists of gray-scale images of hand-written digits
of size $28\times 28$. The train and test sets contain 60K and 10K images,
respectively. An agent is trained for 7 time-steps using the observations of
size $8\times 8$.

SVHN [52]. The dataset is a collection of color images of street-view house
numbers of size $32\times 32$. There are approximately 0.6M train and 26K test
images. An agent is trained on the observations of size $11\times 11$ for 8
time-steps.

ModelNet40 [76]. The dataset is a collection of CAD models of 3D objects
belonging to 40 different categories. The train and test sets contain
approximately 10K and 2.5K objects. A 2D grid of 7$\times$12 views is
generated by capturing views of size $32\times 32$ from 7 camera elevations
$(0^{\circ}$, $30^{\circ}$, $60^{\circ}$, $90^{\circ}$, $120^{\circ}$,
$150^{\circ}$, $180^{\circ})$ and 12 azimuths $(0^{\circ}$, $\pm 30^{\circ}$,
$\pm 60^{\circ}$, $\pm 90^{\circ}$, $\pm 120^{\circ}$, $\pm 150^{\circ}$,
$180^{\circ})$. An agent is trained to predict the entire view-grid from up to
8 2D views.

Cityscapes [15] \+ BDD [78] These datasets provide urban scene images captured
by vehicle-mounted cameras. Train sets of the two datasets are combined to
make a large train set of approximately 10K images. The validation set of 1.5K
images from the Cityscape dataset is used as a test set. The images are
downsampled to the size of $320\times 640$ prior to the training. An agent is
trained on the observations of size $160\times 160$ for 6 timesteps. The agent
directly predicts segmentation maps of the surroundings instead of the visual
images. This requires supervision from human generated ground truth.

(a)

(b)

(c)

(d)

(e)

(f)

(g)

Figure 3: Samples of scenes predicted for 4 timesteps while observations are
captured using uncertainty-based sensing. (a-b) Agents are trained on MNIST
and SVHN. The first column shows the observation history, and the remaining
columns show 10 prediction samples. Rows correspond to increasing timesteps.
(c) An actual view-grid from ModelNet. An agent observes views highlighted in
red, green, blue, and yellow from $t=1$ to 4. (d) Two samples of predicted
view-grids from $t=1$ to 4 (left to right and top to bottom). (e) Actual scene
from Cityscape explored by the agent in (g). (f) The ground-truth segmentation
map of the scene. (g) An agent is exploring the scene shown in (e). The first
column shows observation history and the remaining columns show 4 samples of
predicted segmentation maps. Rows correspond to increasing timesteps. Zoom in
to view details.

###  5.1 Qualitative Analysis

Predictions by the imaginative agents are shown in Fig.3. The agents trained
on MNIST and SVHN (Fig.3(a-b)) predict different digits after looking at the
first random glimpse. Predicted images have the same content at the fixated
location. The successive glimpses are captured using the uncertainty-based
sensing mechanism. The varieties in the predicted digits reduce as the agents
receive further cues. The behavior of an agent trained on ModelNet is
demonstrated on an example view-grid of a chair (Fig.3(c)). After looking at
one random view, the agent predicts view-grids which seem to represent objects
from the desk and the chair categories (Fig.3(d)). With the help of the second
glimpse, the agent determines the global structure of the object. The
following observations resolve uncertainty in the local details. For the
cityscape dataset, an agent captures a new glimpse from an actual surrounding
and hypothesizes complete segmentation maps. Instead of the variance-map, the
next fixation is planned based on the pixel-wise entropy in the predicted
segmentation maps. In Fig.3(g), the agent looks at a cycle-stand and a car in
the first glimpse, and guesses a scene of an intersection. It guesses traffic
lights and road signs on the intersection, and sidewalks and pedestrians on
either side of the road. In the actual scene, there are cars parked on the
right side of the road. Instead, the agent predicts a sidewalk, which is more
likely in general. Through the following glimpses, the agent explores the
configuration of the intersection point. It discovers a building on the left
and the cars parked on the right. The agent strengthens it’s belief and
updates the imaginations with the next couple of glimpses. The segmentation
maps computed using variational inference lack details but are useful for
planning based on approximate locations of the various entities on the road.

###  5.2 Quantitative Analysis

The analysis is conducted using 1K, 1K, 100, and 50 predictions per-timestep
for MNIST, SVHN, ModelNet, and Cityscape, respectively. The analysis is
performed 10 times and the average results are presented next.

First, we compare the hypothesized scenes and an actual scene using an average
of pixel-wise distances computed for increasing timesteps. The distance
metrics used for comparison are binary cross-entropy for MNIST, mean squared
error for SVHN and ModelNet, and categorical cross-entropy with median
frequency balancing [5] for Cityscape. The decreasing trends in Fig.4 suggest
that the predicted scenes converge to the actual scene as time progresses.
Also, the rate of convergence is higher with uncertainty-based sensing than
sensing at random locations. The gain achieved using uncertainty-based sensing
is lower for ModelNet and Cityscape as it is easy to determine the global
structure of the scene from the very first glimpse. The remaining glimpses
only resolve uncertainty in the local details. We also analyze how quickly an
agent can reduce uncertainty in the content of the predicted scenes. As shown
in Fig.5, maximum values in the variance and entropy maps fall with time. The
variety in the prediction reduces rapidly with uncertainty-based sensing
compared to random sensing.

(a)

(b)

(c)

(c)

Figure 4: Average of pixel-wise distances between actual scenes and the
hypothesized scenes for increasing timesteps. Uncertainty-based sensing
achieves better performance than random sensing. (a) MNIST (b) SVHN (c)
ModelNet (d) Cityscape.

(a)

(b)

(c)

(c)

Figure 5: Maximum values in the variance and entropy maps for increasing
timesteps. The variance and the entropy reduces faster using uncertainty-based
sensing compared to random sensing. (a) MNIST (b) SVHN (c) ModelNet (d)
Cityscape.

Next, we train classifiers to predict classes of the scenes hypothesized by
the agents trained on MNIST, SVHN, and ModelNet. The classifiers predict
class-labels from the latent representations of the hypotheses. Note that we
do not aim at achieving the state of the art results in image classification.
The main goal of this analysis is to demonstrate the effectiveness of the
learnt representations for a scene-level task.

We can analyze how quickly an agent concludes on the category of the scene.
The agent may predict scenes from different categories provided that the
content on the observed locations is consistent with the observations. As time
progresses, the agent eliminates unlikely hypotheses, and the entropy in the
categories of the predicted scenes reduces (see Fig.6(a-c)). Again,
uncertainty-based sensing outperforms random sensing. We also analyze how
quickly the category of an actual scene is identified. Fig.7(a-c) show
increasing trend in the classification accuracy. Higher performance can be
achieved with a small number of glimpses captured using uncertainty-based
sensing. The performance of the uncertainty-based sensing and the random
sensing aligns when nearly the entire scene has been seen. Similar trends can
be observed for entropy and average class accuracy of segmentation maps
predicted for Cityscape (see Fig.6(d) and Fig.7(d)). The class accuracy is the
mean of the per-class accuracy.

(a)

(b)

(c)

(d)

Figure 6: Entropy in the categories of the scenes predicted at increasing
timesteps. Uncertainty-based sensing reduces entropy faster compared to random
sensing. (a) MNIST (b) SVHN (c) ModelNet (d) Cityscape.

(a)

(b)

(c)

(d)

Figure 7: Classification accuracy achieved at increasing timesteps.
Uncertainty-based sensing outperforms random sensing. (a) MNIST (b) SVHN (c)
ModelNet (d) Cityscape.

###  5.3 Comparison to Related Work

Jayaraman and Grauman [33], Seifi and Tuytelaars [64] also proposed methods
for unsupervised exploration of the scene. They hypothesize a single scene
from the observation history. The predicted scene is blurry and non-realistic
due to uncertainty. Ramakrishnan et al. [56] use a generative adversarial
network [27] to make the predicted scene realistic. However, this approach
still chooses a single hypothesis, which may not replicate the actual scene.
We resolve the uncertainty by predicting multiple realistic scenes. The
predictions and their latent representations can be used for planning by the
higher-order processes. Additionally, our exploration policy is defined using
the uncertainty in the predicted content. Unlike the previous works, the
presented scheme is interpretable and does not require policy-learning.

A limitation of the proposed method is that it requires large datasets to
learn posterior and likelihood densities accurately. Furthermore, optimization
of the existing flow architectures become difficult for high dimensional
latent spaces. Consequently, a laborious training procedure is required for
large panoramic scenes.

##  6 Conclusion

We proposed a recurrent visual agent which perceives its surroundings through
a series of partial observations. The imaginative agent hypothesizes
unobserved portions of the scene and plans the next fixation using uncertainty
in the hypotheses. The uncertainty in the predicted content reduces as the
agent gathers additional cues. The uncertainty-based sensing helps in
determining the actual surroundings faster than random sensing. The agent is
probabilistic and is designed using variational autoencoders and normalizing
flows. The presented architecture can be interpreted as conditional and
unconditional VAE simultaneously. The internal representations of the
predicted scenes are found to be useful for performing scene-level (e.g.,
classification) and pixel-level (e.g., segmentation) tasks. The agent is
tested on multiple datasets with 2D and 3D scenes. The results display great
potential in imagination-based attention.

## References

  * Alexe et al. [2012] Bogdan Alexe, Nicolas Heess, Yee W Teh, and Vittorio Ferrari.  Searching for objects driven by context.  In _Advances in Neural Information Processing Systems_ , pages 881–889, 2012.
  * Ammirato et al. [2017] Phil Ammirato, Patrick Poirson, Eunbyung Park, Jana Košecká, and Alexander C Berg.  A dataset for developing and benchmarking active vision.  In _2017 IEEE International Conference on Robotics and Automation (ICRA)_ , pages 1378–1385. IEEE, 2017.
  * Ba et al. [2014] Jimmy Ba, Volodymyr Mnih, and Koray Kavukcuoglu.  Multiple object recognition with visual attention.  _arXiv preprint arXiv:1412.7755_ , 2014.
  * Ba et al. [2015] Jimmy Ba, Ruslan R Salakhutdinov, Roger B Grosse, and Brendan J Frey.  Learning wake-sleep recurrent attention models.  In _Advances in Neural Information Processing Systems_ , pages 2593–2601, 2015.
  * Badrinarayanan et al. [2017] Vijay Badrinarayanan, Alex Kendall, and Roberto Cipolla.  Segnet: A deep convolutional encoder-decoder architecture for image segmentation.  _IEEE transactions on pattern analysis and machine intelligence_ , 39(12):2481–2495, 2017.
  * Bajcsy [1988] Ruzena Bajcsy.  Active perception.  _Proceedings of the IEEE_ , 76(8):966–1005, 1988\.
  * Burda et al. [2015] Yuri Burda, Roger B. Grosse, and Ruslan Salakhutdinov.  Importance weighted autoencoders.  _ICLR_ , 2015.
  * Bütepage et al. [2017] Judith Bütepage, Hedvig Kjellström, and Danica Kragic.  Anticipating many futures: Online human motion prediction and synthesis for human-robot collaboration.  _arXiv preprint arXiv:1702.08212_ , 2017.
  * Butko and Movellan [2008] Nicholas J Butko and Javier R Movellan.  I-pomdp: An infomax model of eye movement.  In _2008 7th IEEE International Conference on Development and Learning_ , pages 139–144. IEEE, 2008.
  * Butko and Movellan [2009] Nicholas J Butko and Javier R Movellan.  Optimal scanning for faster object detection.  In _2009 IEEE Conference on Computer Vision and Pattern Recognition_ , pages 2751–2758. IEEE, 2009.
  * Caicedo and Lazebnik [2015] Juan C Caicedo and Svetlana Lazebnik.  Active object localization with deep reinforcement learning.  In _Proceedings of the IEEE International Conference on Computer Vision_ , pages 2488–2496, 2015.
  * Chen et al. [2016] Liang-Chieh Chen, Yi Yang, Jiang Wang, Wei Xu, and Alan L Yuille.  Attention to scale: Scale-aware semantic image segmentation.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 3640–3649, 2016.
  * Chu et al. [2017] Xiao Chu, Wei Yang, Wanli Ouyang, Cheng Ma, Alan L Yuille, and Xiaogang Wang.  Multi-context attention for human pose estimation.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , pages 1831–1840, 2017.
  * [14] James J Clark and Nicola J Ferrier.  Modal control of an attentive vision system.  In _1988 Second International Conference on Computer Vision_ , pages 514–523. IEEE.
  * Cordts et al. [2016] Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele.  The cityscapes dataset for semantic urban scene understanding.  In _Proc. of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2016.
  * Cremer et al. [2018] Chris Cremer, Xuechen Li, and David Duvenaud.  Inference suboptimality in variational autoencoders.  In _International Conference on Machine Learning_ , pages 1078–1086, 2018.
  * Davison and Murray [2002] Andrew J Davison and David W Murray.  Simultaneous localization and map-building using active vision.  _IEEE transactions on pattern analysis and machine intelligence_ , 24(7):865–880, 2002.
  * De Cao et al. [2019] Nicola De Cao, Ivan Titov, and Wilker Aziz.  Block neural autoregressive flow.  _arXiv preprint arXiv:1904.04676_ , 2019.
  * Dinh et al. [2014] Laurent Dinh, David Krueger, and Yoshua Bengio.  Nice: Non-linear independent components estimation.  _arXiv preprint arXiv:1410.8516_ , 2014.
  * Dinh et al. [2016] Laurent Dinh, Jascha Sohl-Dickstein, and Samy Bengio.  Density estimation using real nvp.  _arXiv preprint arXiv:1605.08803_ , 2016.
  * Eslami et al. [2018] SM Ali Eslami, Danilo Jimenez Rezende, Frederic Besse, Fabio Viola, Ari S Morcos, Marta Garnelo, Avraham Ruderman, Andrei A Rusu, Ivo Danihelka, Karol Gregor, et al.  Neural scene representation and rendering.  _Science_ , 360(6394):1204–1210, 2018.
  * Finn et al. [2016] Chelsea Finn, Ian Goodfellow, and Sergey Levine.  Unsupervised learning for physical interaction through video prediction.  In _Advances in neural information processing systems_ , pages 64–72, 2016.
  * Fukushima [1986] Kunihiko Fukushima.  A neural network model for selective attention in visual pattern recognition.  _Biological Cybernetics_ , 55(1):5–15, 1986.
  * Gershman et al. [2012] Samuel J Gershman, Matthew D Hoffman, and David M Blei.  Nonparametric variational inference.  _ICML_ , 2012.
  * Gidaris et al. [2018] Spyros Gidaris, Praveer Singh, and Nikos Komodakis.  Unsupervised representation learning by predicting image rotations.  In _ICLR 2018_ , 2018.
  * Godard et al. [2017] Clément Godard, Oisin Mac Aodha, and Gabriel J Brostow.  Unsupervised monocular depth estimation with left-right consistency.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , pages 270–279, 2017.
  * Goodfellow et al. [2014] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio.  Generative adversarial nets.  In _Advances in neural information processing systems_ , pages 2672–2680, 2014.
  * Gregor et al. [2015] Karol Gregor, Ivo Danihelka, Alex Graves, Danilo Jimenez Rezende, and Daan Wierstra.  Draw: A recurrent neural network for image generation.  _arXiv preprint arXiv:1502.04623_ , 2015.
  * Huang et al. [2018] Chin-Wei Huang, David Krueger, Alexandre Lacoste, and Aaron Courville.  Neural autoregressive flows.  _arXiv preprint arXiv:1804.00779_ , 2018.
  * Ilg et al. [2018] Eddy Ilg, Ozgun Cicek, Silvio Galesso, Aaron Klein, Osama Makansi, Frank Hutter, and Thomas Brox.  Uncertainty estimates and multi-hypotheses networks for optical flow.  In _Proceedings of the European Conference on Computer Vision (ECCV)_ , pages 652–667, 2018.
  * Itti and Koch [2000] Laurent Itti and Christof Koch.  A saliency-based search mechanism for overt and covert shifts of visual attention.  _Vision research_ , 40(10-12):1489–1506, 2000\.
  * Jayaraman and Grauman [2016] Dinesh Jayaraman and Kristen Grauman.  Look-ahead before you leap: end-to-end active recognition by forecasting the effect of motion.  In _European Conference on Computer Vision_ , pages 489–505. Springer, 2016.
  * Jayaraman and Grauman [2018] Dinesh Jayaraman and Kristen Grauman.  Learning to look around: Intelligently exploring unseen environments for unknown tasks.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , pages 1238–1247, 2018.
  * Karayev et al. [2012] Sergey Karayev, Tobias Baumgartner, Mario Fritz, and Trevor Darrell.  Timely object recognition.  In _Advances in Neural Information Processing Systems_ , pages 890–898, 2012.
  * Kelly [1970] Michael D Kelly.  Edge detection in pictures by computer using planning.  Technical report, STANFORD UNIV CALIF DEPT OF COMPUTER SCIENCE, 1970.
  * Kim et al. [2018] Dahun Kim, Donghyeon Cho, Donggeun Yoo, and In So Kweon.  Learning image representations by completing damaged jigsaw puzzles.  In _2018 IEEE Winter Conference on Applications of Computer Vision (WACV)_ , pages 793–802. IEEE, 2018.
  * Kingma and Welling [2013] Diederik P Kingma and Max Welling.  Auto-encoding variational bayes.  _arXiv preprint arXiv:1312.6114_ , 2013.
  * Kingma and Dhariwal [2018] Durk P Kingma and Prafulla Dhariwal.  Glow: Generative flow with invertible 1x1 convolutions.  In _Advances in Neural Information Processing Systems_ , pages 10215–10224, 2018.
  * Kingma et al. [2016] Durk P Kingma, Tim Salimans, Rafal Jozefowicz, Xi Chen, Ilya Sutskever, and Max Welling.  Improved variational inference with inverse autoregressive flow.  In _Advances in neural information processing systems_ , pages 4743–4751, 2016.
  * Koch and Ullman [1987] Christof Koch and Shimon Ullman.  Shifts in selective visual attention: towards the underlying neural circuitry.  In _Matters of intelligence_ , pages 115–141. Springer, 1987.
  * Kohl et al. [2018] Simon Kohl, Bernardino Romera-Paredes, Clemens Meyer, Jeffrey De Fauw, Joseph R Ledsam, Klaus Maier-Hein, SM Ali Eslami, Danilo Jimenez Rezende, and Olaf Ronneberger.  A probabilistic u-net for segmentation of ambiguous images.  In _Advances in Neural Information Processing Systems_ , pages 6965–6975, 2018.
  * Larochelle and Hinton [2010] Hugo Larochelle and Geoffrey E Hinton.  Learning to combine foveal glimpses with a third-order boltzmann machine.  In _Advances in neural information processing systems_ , pages 1243–1251, 2010.
  * Larsson et al. [2017] Gustav Larsson, Michael Maire, and Gregory Shakhnarovich.  Colorization as a proxy task for visual understanding.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , pages 6874–6883, 2017.
  * LeCun et al. [1998] Yann LeCun, Léon Bottou, Yoshua Bengio, Patrick Haffner, et al.  Gradient-based learning applied to document recognition.  _Proceedings of the IEEE_ , 86(11):2278–2324, 1998.
  * Lee and Osindero [2016] Chen-Yu Lee and Simon Osindero.  Recursive recurrent nets with attention modeling for ocr in the wild.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , pages 2231–2239, 2016.
  * Lee et al. [2016] Stefan Lee, Senthil Purushwalkam Shiva Prakash, Michael Cogswell, Viresh Ranjan, David Crandall, and Dhruv Batra.  Stochastic multiple choice learning for training diverse deep ensembles.  In _Advances in Neural Information Processing Systems_ , pages 2119–2127, 2016.
  * Liu et al. [2017a] Jun Liu, Gang Wang, Ping Hu, Ling-Yu Duan, and Alex C Kot.  Global context-aware attention lstm networks for 3d action recognition.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , pages 1647–1656, 2017a.
  * Liu et al. [2018] Xialei Liu, Joost van de Weijer, and Andrew D Bagdanov.  Leveraging unlabeled data for crowd counting by learning to rank.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , pages 7661–7669, 2018.
  * Liu et al. [2017b] Xiao Liu, Jiang Wang, Shilei Wen, Errui Ding, and Yuanqing Lin.  Localizing by describing: Attribute-guided attention localization for fine-grained recognition.  In _Thirty-First AAAI Conference on Artificial Intelligence_ , 2017b.
  * Maaten and Hinton [2008] Laurens van der Maaten and Geoffrey Hinton.  Visualizing data using t-sne.  _Journal of machine learning research_ , 9(Nov):2579–2605, 2008.
  * Mnih et al. [2014] Volodymyr Mnih, Nicolas Heess, Alex Graves, et al.  Recurrent models of visual attention.  In _Advances in neural information processing systems_ , pages 2204–2212, 2014.
  * Netzer et al. [2011] Yuval Netzer, Tiejie Wang, Adam Coates, Alessandro Bissacco, Baolin Wu, and Andrew Y. Ng.  Reading digits in natural images with unsupervised feature learning.  2011\.
  * Noroozi et al. [2017] Mehdi Noroozi, Hamed Pirsiavash, and Paolo Favaro.  Representation learning by learning to count.  In _Proceedings of the IEEE International Conference on Computer Vision_ , pages 5898–5906, 2017.
  * Papamakarios et al. [2017] George Papamakarios, Theo Pavlakou, and Iain Murray.  Masked autoregressive flow for density estimation.  In _Advances in Neural Information Processing Systems_ , pages 2338–2347, 2017.
  * Qiao et al. [2019] Xiaotian Qiao, Quanlong Zheng, Ying Cao, and Rynson WH Lau.  Tell me where i am: Object-level scene context prediction.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , pages 2633–2641, 2019.
  * Ramakrishnan et al. [2019] Santhosh K Ramakrishnan, Dinesh Jayaraman, and Kristen Grauman.  Emergence of exploratory look-around behaviors through active observation completion.  _Science Robotics_ , 4(30):eaaw6326, 2019.
  * Ranzato [2014] Marc’Aurelio Ranzato.  On learning where to look.  _arXiv preprint arXiv:1405.5488_ , 2014.
  * Rensink [2000] Ronald A Rensink.  The dynamic representation of scenes.  _Visual cognition_ , 7(1-3):17–42, 2000.
  * Rezende and Mohamed [2015] Danilo Jimenez Rezende and Shakir Mohamed.  Variational inference with normalizing flows.  _arXiv preprint arXiv:1505.05770_ , 2015.
  * Rezende et al. [2016] Danilo Jimenez Rezende, SM Ali Eslami, Shakir Mohamed, Peter Battaglia, Max Jaderberg, and Nicolas Heess.  Unsupervised learning of 3d structure from images.  In _Advances in Neural Information Processing Systems_ , pages 4996–5004, 2016.
  * Rimey and Brown [1990] Raymond Rimey and Christopher Brown.  Selective attention as sequential behavior: Modeling eye movements with an augmented hidden markov model.  1990\.
  * Rupprecht et al. [2017] Christian Rupprecht, Iro Laina, Robert DiPietro, Maximilian Baust, Federico Tombari, Nassir Navab, and Gregory D Hager.  Learning in an uncertain world: Representing ambiguity through multiple hypotheses.  In _Proceedings of the IEEE International Conference on Computer Vision_ , pages 3591–3600, 2017.
  * Schacter and Madore [2016] Daniel L Schacter and Kevin P Madore.  Remembering the past and imagining the future: Identifying and enhancing the contribution of episodic memory.  _Memory Studies_ , 9(3):245–255, 2016.
  * Seifi and Tuytelaars [2019] Soroush Seifi and Tinne Tuytelaars.  Where to look next: Unsupervised active visual exploration on $360^{\circ}$ input.  _arXiv preprint arXiv:1909.10304_ , 2019.
  * Simons and Levin [1997] Daniel J Simons and Daniel T Levin.  Change blindness.  _Trends in cognitive sciences_ , 1(7):261–267, 1997.
  * Snodgrass and Feenan [1990] Joan G Snodgrass and Kelly Feenan.  Priming effects in picture fragment completion: support for the perceptual closure hypothesis.  _Journal of Experimental Psychology: General_ , 119(3):276, 1990.
  * Sohn et al. [2015a] Kihyuk Sohn, Honglak Lee, and Xinchen Yan.  Learning structured output representation using deep conditional generative models.  In _Advances in neural information processing systems_ , pages 3483–3491, 2015a.
  * Sohn et al. [2015b] Kihyuk Sohn, Honglak Lee, and Xinchen Yan.  Learning structured output representation using deep conditional generative models.  In _Advances in neural information processing systems_ , pages 3483–3491, 2015b.
  * Sønderby et al. [2016] Casper Kaae Sønderby, Tapani Raiko, Lars Maaløe, Søren Kaae Sønderby, and Ole Winther.  Ladder variational autoencoders.  In _Advances in neural information processing systems_ , pages 3738–3746, 2016.
  * Swain and Ballard [1991] Michael J Swain and Dana H Ballard.  Color indexing.  _International journal of computer vision_ , 7(1):11–32, 1991.
  * Wang et al. [2017] Fei Wang, Mengqing Jiang, Chen Qian, Shuo Yang, Cheng Li, Honggang Zhang, Xiaogang Wang, and Xiaoou Tang.  Residual attention network for image classification.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , pages 3156–3164, 2017.
  * Wilkes and Tsotsos [1992] David Wilkes and John K Tsotsos.  Active object recognition.  In _Proceedings 1992 IEEE Computer Society Conference on Computer Vision and Pattern Recognition_ , pages 136–141. IEEE, 1992.
  * Wixson and Ballard [1989] LE Wixson and DH Ballard.  Color histograms for real-time object search.  In _Proc. SPIE Sensor Fusion II: Human and Machine Strategies Workshop, Philadelphia, PA_ , 1989.
  * Woodman et al. [2003] Geoffrey F Woodman, Shaun P Vecera, and Steven J Luck.  Perceptual organization influences visual working memory.  _Psychonomic bulletin & review_, 10(1):80–87, 2003.
  * Wu et al. [2014] Zhirong Wu, Shuran Song, Aditya Khosla, Xiaoou Tang, and Jianxiong Xiao.  3d shapenets for 2.5 d object recognition and next-best-view prediction.  _arXiv preprint arXiv:1406.5670_ , 2(4), 2014.
  * Wu et al. [2015] Zhirong Wu, Shuran Song, Aditya Khosla, Fisher Yu, Linguang Zhang, Xiaoou Tang, and Jianxiong Xiao.  3d shapenets: A deep representation for volumetric shapes.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 1912–1920, 2015.
  * Xu et al. [2015] Kelvin Xu, Jimmy Ba, Ryan Kiros, Kyunghyun Cho, Aaron Courville, Ruslan Salakhudinov, Rich Zemel, and Yoshua Bengio.  Show, attend and tell: Neural image caption generation with visual attention.  In _International conference on machine learning_ , pages 2048–2057, 2015.
  * Yu et al. [2018a] Fisher Yu, Wenqi Xian, Yingying Chen, Fangchen Liu, Mike Liao, Vashisht Madhavan, and Trevor Darrell.  Bdd100k: A diverse driving video database with scalable annotation tooling.  _arXiv preprint arXiv:1805.04687_ , 2018a.
  * Yu et al. [2018b] Jiahui Yu, Zhe Lin, Jimei Yang, Xiaohui Shen, Xin Lu, and Thomas S Huang.  Generative image inpainting with contextual attention.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , pages 5505–5514, 2018b.
  * Zhang et al. [2017] Richard Zhang, Phillip Isola, and Alexei A Efros.  Split-brain autoencoders: Unsupervised learning by cross-channel prediction.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , pages 1058–1067, 2017.
  * Zheng et al. [2015] Yin Zheng, Richard S Zemel, Yu-Jin Zhang, and Hugo Larochelle.  A neural autoregressive approach to attention-based recognition.  _International Journal of Computer Vision_ , 113(1):67–79, 2015.